2022-04-22T17:06:31.336 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-22T17:06:31.337 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/upgrade/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} --owner scheduled_yuriw@teuthology --name yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi/6800380 --verbose -- /home/teuthworker/archive/yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi/6800380/orig.config.yaml 2022-04-22T17:06:31.389 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-22T17:06:31.461 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi/6800380 branch: wip-yuri2-testing-2022-04-22-0500 description: rados/upgrade/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6800380' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run run randomized correctness test for rados operations on an erasure-coded pool ' - desc: 'object class functional tests ' - desc: 'generate read/write load with rados objects ranging from 1MB to 25MB ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'librbd python api tests ' name: yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2022-04-22-0500 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 720674591667bc34cb7086618bbed8722a1f429d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 720674591667bc34cb7086618bbed8722a1f429d kernel: hwe: true workunit: branch: wip-yuri2-testing-2022-04-22-0500 sha1: 720674591667bc34cb7086618bbed8722a1f429d owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mds.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b sha1: 720674591667bc34cb7086618bbed8722a1f429d sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2022-04-22-0500 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_720674591667bc34cb7086618bbed8722a1f429d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 720674591667bc34cb7086618bbed8722a1f429d targets: smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRbkxo2NQ9MkQpnpR5ybV3XgWJMx4iOrZlDb5E67pNYG06t4Xx7GIG2M9smRj6D2tF/Gb3t1vbkaT9cDsndV6/gkWcwXy6Y9SyDhXseGvHno0zYsi5KEX4JpJXxzb5nVaJKktJ1uMA3HJDgVBIS2pP7DtT2Cr7uuA1PsoglCdrLWM3yHx7BVbC+xJ5YFLnluO2QQp4EVXoGi7y1qZ0K0Ta4ViGWWA811rCe9+vKTsKeISNOmZRqbW+j2dWCrNes6UtmlnguY7nojyg2ymAwfJKMbCutaL2L/Vp6oNnMKINokuKSzpEo2xbdRJ9tXyzvpmSsQ9DxC5Fk054Q7UrC7H2D9H2/JRrMAoMwl6A4Lj+DTjGFWlM1CavW8ffvnU1hBkqs+XkV6YULyMuyVOQai87Ct7j2IsM7E1qImikM8F1Nw5i6Fh+Knb/OWTL7BeFr8k/iBBelDqM9W0aA9HwyVKFscf01rDvodczLe6OZ991uLwD2dAfckkhl/pQ2KUceWE= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7bQrCL+PPIT7X2sy6Jj94LbNlHTKnDceRPGqPrTw7btWJIwjekGNLS5AkHN+PP1FtrtgY+ltYiSevAdi1jOekFI6UhWI+e0TO9EZXehOjqXUcjIEKTALj37C0eGUP60Q0DetIgI+1+G93Aa8XN6x2BQ1r4gZkQ166tJ2YeEQFHqIgp4lXUvSVtJ3bbN2GVc5S4TOKfzP6TMVa+NEJWoWjtMye7cbTIYaJOfujtssxk+/AFWit3Zq1My5dqJ0t43bjzQSSdQwV2uGEkW8lGUbdudhRZHhDW9s9fFci+fFH3PGqIWgb9BxPS5kv3r5C0d4eyTj3RdNczjrGQuSEKgzm7Bybxuy5G7WGhNVRjWE7497tpX03HWYJmbYSGQNfaIqsTbESpd8HaTjM+fnVD/fd7jNjErQmIhEYc0kvRiXrFHdJuKpmzDvC78JWSvEQLgz/QtWQYY3I4+GGnkDrA7pBYSDKAFUrCm9Ol1x22Wqdem+L0nAVOIOl6UsUY5WO7wc= tasks: - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - print: '**** done start installing pacific cephadm ...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific - print: '**** done end installing pacific cephadm ...' - print: '**** done start cephadm.shell ceph config set mgr...' - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - print: '**** done start parallel' - parallel: - workload - upgrade-sequence - print: '**** done end parallel' teuthology_branch: master teuthology_sha1: 1b11219b9af33884aae5eb66700541df4b9620ed timestamp: 2022-04-22_13:56:48 tube: smithi upgrade-sequence: sequential: - print: '**** done start upgrade, wait' - cephadm.shell: env: - sha1 mon.a: - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - print: '**** done end upgrade, wait...' user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.12483 workload: full_sequential: - print: '**** done start ec-rados-default.yaml' - rados: clients: - client.0 ec_pool: true objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false - print: '**** done end ec-rados-default.yaml' - print: '**** done start rados_api.yaml' - workunit: branch: pacific clients: client.0: - cls - print: '**** done end rados_api.yaml' - print: '**** done start rados_loadgenbig.yaml' - workunit: branch: pacific clients: client.0: - rados/load-gen-big.sh - print: '**** done end rados_loadgenbig.yaml' - print: '**** done start rbd_import_export.yaml' - workunit: branch: pacific clients: client.1: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done end rbd_import_export.yaml' - print: '**** done start test_rbd_api.yaml' - workunit: branch: pacific clients: client.0: - rbd/test_librbd.sh - print: '**** done end test_rbd_api.yaml' - print: '**** done start test_rbd_python.yaml' - workunit: branch: pacific clients: client.0: - rbd/test_librbd_python.sh - print: '**** done end test_rbd_python.yaml' 2022-04-22T17:06:31.461 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_720674591667bc34cb7086618bbed8722a1f429d/qa; will attempt to use it 2022-04-22T17:06:31.462 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_720674591667bc34cb7086618bbed8722a1f429d/qa/tasks 2022-04-22T17:06:31.462 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-22T17:06:31.464 INFO:teuthology.task.internal:Checking packages... 2022-04-22T17:06:31.487 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '720674591667bc34cb7086618bbed8722a1f429d' 2022-04-22T17:06:31.487 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-22T17:06:31.487 INFO:teuthology.packaging:ref: None 2022-04-22T17:06:31.487 INFO:teuthology.packaging:tag: None 2022-04-22T17:06:31.487 INFO:teuthology.packaging:branch: wip-yuri2-testing-2022-04-22-0500 2022-04-22T17:06:31.488 INFO:teuthology.packaging:sha1: 720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:06:31.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2022-04-22-0500 2022-04-22T17:06:31.698 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11772-g72067459-1focal 2022-04-22T17:06:31.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-22T17:06:31.712 INFO:teuthology.task.internal:no buildpackages task found 2022-04-22T17:06:31.712 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-22T17:06:31.762 INFO:teuthology.task.internal:Saving configuration 2022-04-22T17:06:31.780 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-22T17:06:31.798 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-22T17:06:31.820 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi/6800380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-22 17:01:10.626503', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILfnnEjtICNYciah8QIlOSEHQ2te8eEMOAf8VjPFsk6M'} 2022-04-22T17:06:31.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-04-22_13:56:48-rados-wip-yuri2-testing-2022-04-22-0500-distro-default-smithi/6800380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-22 17:01:10.625104', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHm/Y8lRkdBAA1UHzJ/ZRzXxDJh1JTxVG3QaHU18xxNh'} 2022-04-22T17:06:31.843 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-22T17:06:31.853 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2022-04-22T17:06:31.853 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2022-04-22T17:06:31.853 INFO:teuthology.run_tasks:Running task console_log... 2022-04-22T17:06:31.958 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb286d34510>, signals=[15]) 2022-04-22T17:06:31.958 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-22T17:06:31.968 INFO:teuthology.task.internal:Opening connections... 2022-04-22T17:06:31.969 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2022-04-22T17:06:31.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:06:32.023 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2022-04-22T17:06:32.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:06:32.074 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-22T17:06:32.084 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2022-04-22T17:06:32.102 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2022-04-22T17:06:32.103 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2022-04-22T17:06:32.150 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Ubuntu" 2022-04-22T17:06:32.175 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:ID=ubuntu 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE=debian 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="20.04" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-22T17:06:32.176 INFO:teuthology.orchestra.run.smithi115.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-22T17:06:32.177 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_CODENAME=focal 2022-04-22T17:06:32.177 INFO:teuthology.orchestra.run.smithi115.stdout:UBUNTU_CODENAME=focal 2022-04-22T17:06:32.177 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-04-22T17:06:32.196 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2022-04-22T17:06:32.204 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2022-04-22T17:06:32.204 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2022-04-22T17:06:32.250 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="20.04" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-22T17:06:32.251 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-22T17:06:32.252 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=focal 2022-04-22T17:06:32.252 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=focal 2022-04-22T17:06:32.252 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-04-22T17:06:32.270 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-22T17:06:32.285 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-22T17:06:32.311 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-22T17:06:32.311 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2022-04-22T17:06:32.314 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2022-04-22T17:06:32.320 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-22T17:06:32.345 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-22T17:06:32.345 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2022-04-22T17:06:32.347 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2022-04-22T17:06:32.373 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-22T17:06:32.430 INFO:teuthology.run_tasks:Running task kernel... 2022-04-22T17:06:32.445 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-04-22T17:06:32.445 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-04-22T17:06:32.446 DEBUG:teuthology.task.kernel:normalized overrides {'mon.a': {'hwe': True}, 'mon.c': {'hwe': True}, 'mgr.y': {'hwe': True}, 'mds.a': {'hwe': True}, 'osd.0': {'hwe': True}, 'osd.1': {'hwe': True}, 'osd.2': {'hwe': True}, 'osd.3': {'hwe': True}, 'client.0': {'hwe': True}, 'node-exporter.a': {'hwe': True}, 'alertmanager.a': {'hwe': True}, 'mon.b': {'hwe': True}, 'mds.b': {'hwe': True}, 'mgr.x': {'hwe': True}, 'osd.4': {'hwe': True}, 'osd.5': {'hwe': True}, 'osd.6': {'hwe': True}, 'osd.7': {'hwe': True}, 'client.1': {'hwe': True}, 'prometheus.a': {'hwe': True}, 'grafana.a': {'hwe': True}, 'node-exporter.b': {'hwe': True}} 2022-04-22T17:06:32.446 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'mon.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-04-22T17:06:32.446 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-22T17:06:32.447 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-22T17:06:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T17:06:32.483 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2022-04-22T17:06:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T17:06:32.484 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-04-22T17:06:32.533 INFO:teuthology.orchestra.run.smithi115.stdout:5.4.0-107-generic 2022-04-22T17:06:32.590 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.4.0-107-generic 2022-04-22T17:06:32.590 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get clean 2022-04-22T17:06:32.591 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-107-generic 2022-04-22T17:06:32.591 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.4.0-107-generic 2022-04-22T17:06:32.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-04-22T17:06:32.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-04-22T17:06:32.679 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2022-04-22T17:06:32.942 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:06:32.948 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-22T17:06:32.953 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:06:32.958 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-22T17:06:33.009 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-22T17:06:33.021 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-22T17:06:33.352 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-22T17:06:33.392 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-22T17:06:33.423 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [425 kB] 2022-04-22T17:06:33.447 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,417 kB] 2022-04-22T17:06:33.570 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,745 kB] 2022-04-22T17:06:33.587 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [635 kB] 2022-04-22T17:06:33.712 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,417 kB] 2022-04-22T17:06:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [425 kB] 2022-04-22T17:06:33.869 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,745 kB] 2022-04-22T17:06:33.879 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [245 kB] 2022-04-22T17:06:33.882 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [245 kB] 2022-04-22T17:06:33.889 INFO:teuthology.orchestra.run.smithi115.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.0 kB] 2022-04-22T17:06:33.889 INFO:teuthology.orchestra.run.smithi115.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [22.0 kB] 2022-04-22T17:06:33.890 INFO:teuthology.orchestra.run.smithi115.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [886 kB] 2022-04-22T17:06:33.895 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.0 kB] 2022-04-22T17:06:33.895 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [22.0 kB] 2022-04-22T17:06:33.896 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [886 kB] 2022-04-22T17:06:33.943 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [126 kB] 2022-04-22T17:06:33.947 INFO:teuthology.orchestra.run.smithi115.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [126 kB] 2022-04-22T17:06:33.949 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] 2022-04-22T17:06:33.950 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [700 kB] 2022-04-22T17:06:33.952 INFO:teuthology.orchestra.run.smithi115.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] 2022-04-22T17:06:33.952 INFO:teuthology.orchestra.run.smithi115.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [549 kB] 2022-04-22T17:06:33.965 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [635 kB] 2022-04-22T17:06:33.972 INFO:teuthology.orchestra.run.smithi115.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [700 kB] 2022-04-22T17:06:33.974 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [549 kB] 2022-04-22T17:06:33.983 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [324 kB] 2022-04-22T17:06:33.989 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [124 kB] 2022-04-22T17:06:33.993 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB] 2022-04-22T17:06:33.994 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB] 2022-04-22T17:06:33.994 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [946 kB] 2022-04-22T17:06:33.999 INFO:teuthology.orchestra.run.smithi115.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [124 kB] 2022-04-22T17:06:34.001 INFO:teuthology.orchestra.run.smithi115.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB] 2022-04-22T17:06:34.023 INFO:teuthology.orchestra.run.smithi115.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [324 kB] 2022-04-22T17:06:34.039 INFO:teuthology.orchestra.run.smithi115.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB] 2022-04-22T17:06:34.140 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [24.3 kB] 2022-04-22T17:06:34.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [135 kB] 2022-04-22T17:06:34.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] 2022-04-22T17:06:34.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-04-22T17:06:34.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [918 kB] 2022-04-22T17:06:34.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.6 kB] 2022-04-22T17:06:34.142 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-04-22T17:06:34.142 INFO:teuthology.orchestra.run.smithi115.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [946 kB] 2022-04-22T17:06:34.142 INFO:teuthology.orchestra.run.smithi115.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [24.3 kB] 2022-04-22T17:06:34.142 INFO:teuthology.orchestra.run.smithi115.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [135 kB] 2022-04-22T17:06:34.143 INFO:teuthology.orchestra.run.smithi115.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] 2022-04-22T17:06:34.143 INFO:teuthology.orchestra.run.smithi115.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [918 kB] 2022-04-22T17:06:34.143 INFO:teuthology.orchestra.run.smithi115.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-04-22T17:06:34.143 INFO:teuthology.orchestra.run.smithi115.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.6 kB] 2022-04-22T17:06:34.143 INFO:teuthology.orchestra.run.smithi115.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-04-22T17:06:34.144 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-04-22T17:06:34.145 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-04-22T17:06:34.186 INFO:teuthology.orchestra.run.smithi115.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-04-22T17:06:34.201 INFO:teuthology.orchestra.run.smithi115.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-04-22T17:06:37.135 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 10.3 MB in 2s (4,798 kB/s) 2022-04-22T17:06:37.230 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 10.3 MB in 2s (4,627 kB/s) 2022-04-22T17:06:38.352 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:06:38.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-22T17:06:38.415 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:06:38.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-22T17:06:38.447 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:06:38.501 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:06:38.628 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2022-04-22T17:06:38.628 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2022-04-22T17:06:38.677 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-04-22T17:06:38.685 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-04-22T17:06:38.780 INFO:teuthology.orchestra.run.smithi115.stdout:The following additional packages will be installed: 2022-04-22T17:06:38.781 INFO:teuthology.orchestra.run.smithi115.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-22T17:06:38.781 INFO:teuthology.orchestra.run.smithi115.stdout: libusbmuxd6 linux-image-5.13.0-40-generic linux-modules-5.13.0-40-generic 2022-04-22T17:06:38.781 INFO:teuthology.orchestra.run.smithi115.stdout: linux-modules-extra-5.13.0-40-generic thermald upower usbmuxd 2022-04-22T17:06:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:Suggested packages: 2022-04-22T17:06:38.781 INFO:teuthology.orchestra.run.smithi115.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-22T17:06:38.782 INFO:teuthology.orchestra.run.smithi115.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-40-generic 2022-04-22T17:06:38.826 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-04-22T17:06:38.826 INFO:teuthology.orchestra.run.smithi121.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-22T17:06:38.826 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd6 linux-image-5.13.0-40-generic linux-modules-5.13.0-40-generic 2022-04-22T17:06:38.826 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-5.13.0-40-generic thermald upower usbmuxd 2022-04-22T17:06:38.827 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-04-22T17:06:38.827 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-22T17:06:38.827 INFO:teuthology.orchestra.run.smithi121.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-40-generic 2022-04-22T17:06:38.841 INFO:teuthology.orchestra.run.smithi115.stdout:The following NEW packages will be installed: 2022-04-22T17:06:38.842 INFO:teuthology.orchestra.run.smithi115.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-22T17:06:38.842 INFO:teuthology.orchestra.run.smithi115.stdout: libusbmuxd6 linux-image-5.13.0-40-generic linux-image-generic-hwe-20.04 2022-04-22T17:06:38.842 INFO:teuthology.orchestra.run.smithi115.stdout: linux-modules-5.13.0-40-generic linux-modules-extra-5.13.0-40-generic 2022-04-22T17:06:38.842 INFO:teuthology.orchestra.run.smithi115.stdout: thermald upower usbmuxd 2022-04-22T17:06:38.877 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-04-22T17:06:38.877 INFO:teuthology.orchestra.run.smithi121.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-22T17:06:38.878 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd6 linux-image-5.13.0-40-generic linux-image-generic-hwe-20.04 2022-04-22T17:06:38.878 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-5.13.0-40-generic linux-modules-extra-5.13.0-40-generic 2022-04-22T17:06:38.878 INFO:teuthology.orchestra.run.smithi121.stdout: thermald upower usbmuxd 2022-04-22T17:06:38.899 INFO:teuthology.orchestra.run.smithi115.stdout:0 upgraded, 13 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:06:38.899 INFO:teuthology.orchestra.run.smithi115.stdout:Need to get 79.9 MB of archives. 2022-04-22T17:06:38.899 INFO:teuthology.orchestra.run.smithi115.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-22T17:06:38.899 INFO:teuthology.orchestra.run.smithi115.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-22T17:06:38.954 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 13 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:06:38.954 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 79.9 MB of archives. 2022-04-22T17:06:38.955 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-22T17:06:38.955 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-22T17:06:38.990 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-22T17:06:38.997 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-22T17:06:38.998 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-22T17:06:39.042 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-22T17:06:39.057 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [18.5 MB] 2022-04-22T17:06:39.069 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-22T17:06:39.074 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-22T17:06:39.077 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-22T17:06:39.132 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-22T17:06:39.151 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [18.5 MB] 2022-04-22T17:06:39.907 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [10.0 MB] 2022-04-22T17:06:39.949 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [10.0 MB] 2022-04-22T17:06:40.111 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [50.8 MB] 2022-04-22T17:06:40.177 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-40-generic amd64 5.13.0-40.45~20.04.1 [50.8 MB] 2022-04-22T17:06:41.690 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.40.45~20.04.25 [2,612 B] 2022-04-22T17:06:41.690 INFO:teuthology.orchestra.run.smithi115.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-04-22T17:06:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-22T17:06:41.692 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.40.45~20.04.25 [2,612 B] 2022-04-22T17:06:41.692 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-04-22T17:06:41.693 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-22T17:06:41.696 INFO:teuthology.orchestra.run.smithi115.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-22T17:06:41.698 INFO:teuthology.orchestra.run.smithi115.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-22T17:06:41.699 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-22T17:06:41.701 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-22T17:06:42.083 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 79.9 MB in 3s (28.4 MB/s) 2022-04-22T17:06:42.100 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 79.9 MB in 3s (28.0 MB/s) 2022-04-22T17:06:42.253 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-22T17:06:42.323 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-22T17:06:43.943 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102172 files and directories currently installed.) 2022-04-22T17:06:44.022 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-22T17:06:44.040 INFO:teuthology.orchestra.run.smithi115.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102172 files and directories currently installed.) 2022-04-22T17:06:44.045 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-22T17:06:44.115 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-22T17:06:44.200 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-22T17:06:44.490 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-22T17:06:44.502 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-22T17:06:44.541 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-22T17:06:44.562 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-22T17:06:44.576 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-22T17:06:44.612 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-22T17:06:44.874 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-22T17:06:44.886 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-22T17:06:44.925 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-22T17:06:44.926 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-22T17:06:44.926 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-22T17:06:44.963 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-22T17:06:45.229 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-22T17:06:45.242 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-22T17:06:45.271 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-22T17:06:45.279 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-22T17:06:45.283 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-22T17:06:45.314 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-22T17:06:45.615 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-22T17:06:45.621 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-22T17:06:45.628 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-22T17:06:45.636 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-22T17:06:45.674 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-22T17:06:45.682 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-22T17:06:45.994 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-modules-5.13.0-40-generic. 2022-04-22T17:06:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-5.13.0-40-generic. 2022-04-22T17:06:46.008 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../05-linux-modules-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:46.015 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../05-linux-modules-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:46.052 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-modules-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:46.068 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:48.235 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-image-5.13.0-40-generic. 2022-04-22T17:06:48.249 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../06-linux-image-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:48.288 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-5.13.0-40-generic. 2022-04-22T17:06:48.301 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../06-linux-image-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:48.348 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:48.410 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:48.830 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-40-generic. 2022-04-22T17:06:48.842 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:48.879 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-modules-extra-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:48.941 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-40-generic. 2022-04-22T17:06:48.953 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-40-generic_5.13.0-40.45~20.04.1_amd64.deb ... 2022-04-22T17:06:48.990 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:54.628 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-22T17:06:54.671 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.40.45~20.04.25_amd64.deb ... 2022-04-22T17:06:54.686 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.40.45~20.04.25) ... 2022-04-22T17:06:54.763 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-22T17:06:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.40.45~20.04.25_amd64.deb ... 2022-04-22T17:06:54.821 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.40.45~20.04.25) ... 2022-04-22T17:06:55.005 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-22T17:06:55.018 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-04-22T17:06:55.047 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-04-22T17:06:55.154 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-22T17:06:55.170 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-04-22T17:06:55.212 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-04-22T17:06:55.394 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package thermald. 2022-04-22T17:06:55.408 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-22T17:06:55.453 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-22T17:06:55.562 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package thermald. 2022-04-22T17:06:55.575 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-22T17:06:55.612 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-22T17:06:55.998 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package upower. 2022-04-22T17:06:56.012 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-22T17:06:56.076 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-22T17:06:56.147 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package upower. 2022-04-22T17:06:56.161 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-22T17:06:56.214 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-22T17:06:56.481 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package usbmuxd. 2022-04-22T17:06:56.496 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-22T17:06:56.539 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-22T17:06:56.597 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package usbmuxd. 2022-04-22T17:06:56.612 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-22T17:06:56.655 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-22T17:06:56.843 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-22T17:06:56.954 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-22T17:06:56.967 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-22T17:06:57.060 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-22T17:06:57.094 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-22T17:06:57.219 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-22T17:06:57.220 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-22T17:06:57.312 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-22T17:06:57.347 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-22T17:06:57.438 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-22T17:06:57.472 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-04-22T17:06:57.564 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-04-22T17:06:57.599 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-22T17:06:57.715 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-22T17:06:58.033 INFO:teuthology.orchestra.run.smithi115.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-22T17:06:58.091 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-22T17:06:58.153 INFO:teuthology.orchestra.run.smithi121.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-22T17:06:58.248 INFO:teuthology.orchestra.run.smithi115.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-22T17:06:58.254 INFO:teuthology.orchestra.run.smithi115.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-22T17:06:58.254 INFO:teuthology.orchestra.run.smithi115.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-22T17:06:58.260 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-22T17:06:58.717 INFO:teuthology.orchestra.run.smithi115.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-22T17:06:58.814 INFO:teuthology.orchestra.run.smithi121.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-22T17:06:58.822 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-22T17:06:58.822 INFO:teuthology.orchestra.run.smithi121.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-22T17:06:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-22T17:06:59.005 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-22T17:06:59.308 INFO:teuthology.orchestra.run.smithi121.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-22T17:06:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-22T17:06:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-22T17:06:59.547 INFO:teuthology.orchestra.run.smithi121.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-22T17:06:59.587 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-22T17:06:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:06:59.895 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-22T17:06:59.895 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-22T17:07:00.366 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:01.063 INFO:teuthology.orchestra.run.smithi115.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-40-generic 2022-04-22T17:07:01.063 INFO:teuthology.orchestra.run.smithi115.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-40-generic 2022-04-22T17:07:01.240 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-modules-extra-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:01.642 INFO:teuthology.orchestra.run.smithi121.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-40-generic 2022-04-22T17:07:01.666 INFO:teuthology.orchestra.run.smithi121.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-40-generic 2022-04-22T17:07:01.812 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:02.574 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-modules-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:03.179 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.40.45~20.04.25) ... 2022-04-22T17:07:03.991 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-22T17:07:04.514 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.40.45~20.04.25) ... 2022-04-22T17:07:04.615 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-22T17:07:08.122 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-22T17:07:08.480 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-22T17:07:08.783 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-22T17:07:08.883 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:09.085 INFO:teuthology.orchestra.run.smithi115.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-22T17:07:09.085 INFO:teuthology.orchestra.run.smithi115.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:09.823 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-22T17:07:10.123 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-5.13.0-40-generic (5.13.0-40.45~20.04.1) ... 2022-04-22T17:07:10.304 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-22T17:07:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:32.495 INFO:teuthology.orchestra.run.smithi115.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-22T17:07:32.495 INFO:teuthology.orchestra.run.smithi115.stdout:Sourcing file `/etc/default/grub' 2022-04-22T17:07:32.500 INFO:teuthology.orchestra.run.smithi115.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-22T17:07:32.522 INFO:teuthology.orchestra.run.smithi115.stdout:Generating grub configuration file ... 2022-04-22T17:07:32.829 INFO:teuthology.orchestra.run.smithi115.stdout:Found linux image: /boot/vmlinuz-5.13.0-40-generic 2022-04-22T17:07:32.854 INFO:teuthology.orchestra.run.smithi115.stdout:Found initrd image: /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:33.245 INFO:teuthology.orchestra.run.smithi115.stdout:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-22T17:07:33.255 INFO:teuthology.orchestra.run.smithi115.stdout:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-22T17:07:33.949 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-22T17:07:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2022-04-22T17:07:33.958 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-22T17:07:33.979 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2022-04-22T17:07:34.259 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.13.0-40-generic 2022-04-22T17:07:34.287 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:34.701 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-22T17:07:34.744 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-22T17:07:35.836 INFO:teuthology.orchestra.run.smithi115.stdout:done 2022-04-22T17:07:36.158 DEBUG:teuthology.orchestra.run.smithi115:> dpkg -s linux-image-generic-hwe-20.04 2022-04-22T17:07:36.179 INFO:teuthology.orchestra.run.smithi115.stdout:Package: linux-image-generic-hwe-20.04 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Status: install ok installed 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Priority: optional 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Section: kernel 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Installed-Size: 19 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Maintainer: Ubuntu Kernel Team 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: amd64 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Source: linux-meta-hwe-5.13 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Version: 5.13.0.40.45~20.04.25 2022-04-22T17:07:36.180 INFO:teuthology.orchestra.run.smithi115.stdout:Provides: virtualbox-guest-modules (= 5.13.0-40), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-22T17:07:36.181 INFO:teuthology.orchestra.run.smithi115.stdout:Depends: linux-image-5.13.0-40-generic, linux-modules-extra-5.13.0-40-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-22T17:07:36.181 INFO:teuthology.orchestra.run.smithi115.stdout:Recommends: thermald 2022-04-22T17:07:36.181 INFO:teuthology.orchestra.run.smithi115.stdout:Description: Generic Linux kernel image 2022-04-22T17:07:36.181 INFO:teuthology.orchestra.run.smithi115.stdout: This package will always depend on the latest generic kernel image 2022-04-22T17:07:36.181 INFO:teuthology.orchestra.run.smithi115.stdout: available. 2022-04-22T17:07:36.182 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-107-generic Expected: 5.13.0-40-generic 2022-04-22T17:07:36.182 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-22T17:07:36.182 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-22T17:07:36.182 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2022-04-22T17:07:36.183 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2022-04-22T17:07:36.183 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get clean 2022-04-22T17:07:36.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2022-04-22T17:07:36.381 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:07:36.389 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-22T17:07:36.390 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-22T17:07:36.409 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-22T17:07:37.297 INFO:teuthology.orchestra.run.smithi121.stdout:done 2022-04-22T17:07:37.647 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic-hwe-20.04 2022-04-22T17:07:37.674 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic-hwe-20.04 2022-04-22T17:07:37.674 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta-hwe-5.13 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.13.0.40.45~20.04.25 2022-04-22T17:07:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 5.13.0-40), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-22T17:07:37.676 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.13.0-40-generic, linux-modules-extra-5.13.0-40-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-22T17:07:37.676 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-04-22T17:07:37.676 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-04-22T17:07:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-04-22T17:07:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-04-22T17:07:37.677 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-107-generic Expected: 5.13.0-40-generic 2022-04-22T17:07:37.677 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-22T17:07:37.677 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-22T17:07:37.677 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2022-04-22T17:07:37.677 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2022-04-22T17:07:37.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-04-22T17:07:37.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-04-22T17:07:37.881 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-22T17:07:37.924 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:07:37.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-22T17:07:37.989 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:07:38.014 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:07:38.070 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-22T17:07:38.190 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2022-04-22T17:07:38.190 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-22T17:07:38.190 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2022-04-22T17:07:38.373 INFO:teuthology.orchestra.run.smithi115.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.40.45~20.04.25). 2022-04-22T17:07:38.373 INFO:teuthology.orchestra.run.smithi115.stdout:0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:07:38.376 DEBUG:teuthology.orchestra.run.smithi115:> dpkg -s linux-image-generic-hwe-20.04 2022-04-22T17:07:38.395 INFO:teuthology.orchestra.run.smithi115.stdout:Package: linux-image-generic-hwe-20.04 2022-04-22T17:07:38.395 INFO:teuthology.orchestra.run.smithi115.stdout:Status: install ok installed 2022-04-22T17:07:38.395 INFO:teuthology.orchestra.run.smithi115.stdout:Priority: optional 2022-04-22T17:07:38.395 INFO:teuthology.orchestra.run.smithi115.stdout:Section: kernel 2022-04-22T17:07:38.395 INFO:teuthology.orchestra.run.smithi115.stdout:Installed-Size: 19 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Maintainer: Ubuntu Kernel Team 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: amd64 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Source: linux-meta-hwe-5.13 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Version: 5.13.0.40.45~20.04.25 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Provides: virtualbox-guest-modules (= 5.13.0-40), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Depends: linux-image-5.13.0-40-generic, linux-modules-extra-5.13.0-40-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Recommends: thermald 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout:Description: Generic Linux kernel image 2022-04-22T17:07:38.396 INFO:teuthology.orchestra.run.smithi115.stdout: This package will always depend on the latest generic kernel image 2022-04-22T17:07:38.397 INFO:teuthology.orchestra.run.smithi115.stdout: available. 2022-04-22T17:07:38.397 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2022-04-22T17:07:38.445 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.L36aHyAq1H 2022-04-22T17:07:38.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub/grub.cfg /tmp/tmp.L36aHyAq1H 2022-04-22T17:07:38.459 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.L36aHyAq1H 2022-04-22T17:07:38.507 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.L36aHyAq1H is 9KB 2022-04-22T17:07:38.522 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.L36aHyAq1H 2022-04-22T17:07:38.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-22T17:07:38.591 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:07:38.591 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-22T17:07:38.591 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-22T17:07:38.623 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-40-generic 2022-04-22T17:07:38.624 DEBUG:teuthology.orchestra.run.smithi115:> sudo update-grub 2022-04-22T17:07:39.426 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:07:39.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-22T17:07:39.512 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:07:39.669 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-04-22T17:07:39.670 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-04-22T17:07:39.834 INFO:teuthology.orchestra.run.smithi115.stderr:Sourcing file `/etc/default/grub' 2022-04-22T17:07:39.840 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.40.45~20.04.25). 2022-04-22T17:07:39.840 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:07:39.843 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic-hwe-20.04 2022-04-22T17:07:39.845 INFO:teuthology.orchestra.run.smithi115.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-22T17:07:39.855 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2022-04-22T17:07:39.863 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic-hwe-20.04 2022-04-22T17:07:39.864 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-04-22T17:07:39.864 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-04-22T17:07:39.864 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-04-22T17:07:39.864 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-04-22T17:07:39.864 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-04-22T17:07:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-04-22T17:07:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta-hwe-5.13 2022-04-22T17:07:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.13.0.40.45~20.04.25 2022-04-22T17:07:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 5.13.0-40), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-22T17:07:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.13.0-40-generic, linux-modules-extra-5.13.0-40-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-22T17:07:39.866 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-04-22T17:07:39.866 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-04-22T17:07:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-04-22T17:07:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-04-22T17:07:39.867 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2022-04-22T17:07:39.914 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.pinI0kwDdF 2022-04-22T17:07:39.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.pinI0kwDdF 2022-04-22T17:07:39.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.pinI0kwDdF 2022-04-22T17:07:39.970 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.pinI0kwDdF is 9KB 2022-04-22T17:07:39.982 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.pinI0kwDdF 2022-04-22T17:07:39.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-22T17:07:40.047 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:07:40.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-22T17:07:40.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-22T17:07:40.071 INFO:teuthology.orchestra.run.smithi115.stderr:Found linux image: /boot/vmlinuz-5.13.0-40-generic 2022-04-22T17:07:40.075 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-40-generic 2022-04-22T17:07:40.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2022-04-22T17:07:40.083 INFO:teuthology.orchestra.run.smithi115.stderr:Found initrd image: /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:40.424 INFO:teuthology.orchestra.run.smithi115.stderr:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-22T17:07:40.435 INFO:teuthology.orchestra.run.smithi115.stderr:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-22T17:07:41.251 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2022-04-22T17:07:41.253 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-22T17:07:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2022-04-22T17:07:41.486 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.13.0-40-generic 2022-04-22T17:07:41.498 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.13.0-40-generic 2022-04-22T17:07:41.875 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-22T17:07:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-22T17:07:42.023 INFO:teuthology.orchestra.run.smithi115.stderr:done 2022-04-22T17:07:42.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2022-04-22T17:07:43.440 INFO:teuthology.orchestra.run.smithi121.stderr:done 2022-04-22T17:07:43.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2022-04-22T17:08:12.055 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2022-04-22T17:08:12.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-04-22T17:08:12.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:08:13.446 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2022-04-22T17:08:13.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-04-22T17:08:13.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:08:30.539 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2022-04-22T17:08:31.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-04-22T17:09:00.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-04-22T17:09:00.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:09:01.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-04-22T17:09:01.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:09:18.923 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2022-04-22T17:09:20.203 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-04-22T17:09:48.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-04-22T17:09:48.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:09:49.127 DEBUG:teuthology.orchestra.run.smithi115:> true 2022-04-22T17:09:49.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-04-22T17:09:49.644 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.13.0-40-generic"... 2022-04-22T17:09:49.644 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2022-04-22T17:09:49.691 INFO:teuthology.orchestra.run.smithi115.stdout:5.13.0-40-generic 2022-04-22T17:09:49.692 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-40-generic vs 5.13.0-40-generic 2022-04-22T17:09:49.692 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-22T17:09:49.692 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-22T17:09:50.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-04-22T17:09:50.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:09:50.435 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-04-22T17:09:50.693 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-04-22T17:09:50.693 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-22T17:09:50.774 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2022-04-22T17:09:50.819 DEBUG:teuthology.parallel:result is None 2022-04-22T17:09:50.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-04-22T17:09:50.936 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.13.0-40-generic"... 2022-04-22T17:09:50.937 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-04-22T17:09:50.941 INFO:teuthology.orchestra.run.smithi121.stdout:5.13.0-40-generic 2022-04-22T17:09:50.941 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-40-generic vs 5.13.0-40-generic 2022-04-22T17:09:50.941 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-22T17:09:50.941 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-22T17:09:51.942 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-04-22T17:09:51.943 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-22T17:09:52.017 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2022-04-22T17:09:52.048 DEBUG:teuthology.parallel:result is None 2022-04-22T17:09:52.049 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-22T17:09:52.059 INFO:teuthology.task.internal:Creating test directory... 2022-04-22T17:09:52.059 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-22T17:09:52.062 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-22T17:09:52.069 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-22T17:09:52.118 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-22T17:09:52.186 INFO:teuthology.task.internal:Creating archive directory... 2022-04-22T17:09:52.186 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-22T17:09:52.189 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-22T17:09:52.211 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-22T17:09:52.224 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-22T17:09:52.225 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-22T17:09:52.227 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-22T17:09:52.248 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-22T17:09:52.258 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-22T17:09:52.267 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-22T17:09:52.275 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-22T17:09:52.276 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-22T17:09:52.283 INFO:teuthology.task.internal:Configuring sudo... 2022-04-22T17:09:52.283 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-22T17:09:52.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-22T17:09:52.301 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-22T17:09:52.342 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-22T17:09:52.343 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-22T17:09:52.346 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-22T17:09:52.353 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-22T17:09:52.359 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-22T17:09:52.364 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:09:52.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-22T17:09:52.384 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-22T17:09:52.401 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-22T17:09:52.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:09:52.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-22T17:09:52.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2022-04-22T17:09:52.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2022-04-22T17:09:52.515 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-22T17:09:52.528 INFO:teuthology.task.internal:Starting timer... 2022-04-22T17:09:52.528 INFO:teuthology.run_tasks:Running task pcp... 2022-04-22T17:09:52.558 INFO:teuthology.run_tasks:Running task selinux... 2022-04-22T17:09:52.605 DEBUG:teuthology.task.selinux:Excluding smithi115: OS 'ubuntu' does not support SELinux 2022-04-22T17:09:52.605 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2022-04-22T17:09:52.605 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-22T17:09:52.606 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-04-22T17:09:52.606 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-22T17:09:52.606 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-22T17:09:52.698 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-04-22T17:09:52.704 INFO:teuthology.repo_utils:Fetching master from origin 2022-04-22T17:09:52.738 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-22T17:09:52.755 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-22T17:09:52.756 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-22T17:12:44.797 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2022-04-22T17:12:44.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-04-22T17:12:44.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:12:44.855 DEBUG:teuthology.orchestra.run.smithi115:> true 2022-04-22T17:12:44.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-04-22T17:12:45.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-22T17:12:45.091 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-04-22T17:12:45.208 INFO:teuthology.run_tasks:Running task clock... 2022-04-22T17:12:45.224 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-22T17:12:45.225 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-22T17:12:45.226 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-22T17:12:45.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-22T17:12:45.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-22T17:12:45.257 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-22T17:12:45.257 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Command line: ntpd -gq 2022-04-22T17:12:45.258 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: proto: precision = 0.070 usec (-24) 2022-04-22T17:12:45.258 INFO:teuthology.orchestra.run.smithi115.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-22T17:12:45.258 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-22T17:12:45.259 INFO:teuthology.orchestra.run.smithi115.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: restrict ::: KOD does nothing without LIMITED. 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen and drop on 0 v6wildcard [::]:123 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen normally on 2 lo 127.0.0.1:123 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen normally on 3 enp3s0f1 172.21.15.115:123 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen normally on 4 lo [::1]:123 2022-04-22T17:12:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce8f%5]:123 2022-04-22T17:12:45.261 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:45 ntpd[7839]: Listening on routing socket on fd #22 for interface updates 2022-04-22T17:12:45.261 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-22T17:12:45.261 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Command line: ntpd -gq 2022-04-22T17:12:45.262 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: proto: precision = 0.066 usec (-24) 2022-04-22T17:12:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-22T17:12:45.262 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-22T17:12:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-22T17:12:45.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: restrict ::: KOD does nothing without LIMITED. 2022-04-22T17:12:45.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen and drop on 0 v6wildcard [::]:123 2022-04-22T17:12:45.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-22T17:12:45.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen normally on 2 lo 127.0.0.1:123 2022-04-22T17:12:45.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2022-04-22T17:12:45.265 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen normally on 4 lo [::1]:123 2022-04-22T17:12:45.265 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2022-04-22T17:12:45.265 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:45 ntpd[7857]: Listening on routing socket on fd #22 for interface updates 2022-04-22T17:12:46.257 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:46.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:46.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:46.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:46.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:46.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:46 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:47.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:47 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:47 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:48.257 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:48.290 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:48.290 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:48.290 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:48.291 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:48.291 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:48 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:48.292 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:48 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.261 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.262 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:49 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.267 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:49.268 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:49 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:50.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:50.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:50.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:50.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:50.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:50.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:50 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:50 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.303 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.304 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.305 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:51 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:51 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:52.257 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:52.258 INFO:teuthology.orchestra.run.smithi115.stdout:22 Apr 17:12:52 ntpd[7839]: ntpd: time slew -0.000252 s 2022-04-22T17:12:52.259 INFO:teuthology.orchestra.run.smithi115.stdout:ntpd: time slew -0.000252s 2022-04-22T17:12:52.259 INFO:teuthology.orchestra.run.smithi115.stderr:22 Apr 17:12:52 ntpd[7839]: can't open /var/log/ntpstats/loopstats.20220422: Permission denied 2022-04-22T17:12:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:52 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: remote refid st t when poll reach delay offset jitter 2022-04-22T17:12:52.321 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================== 2022-04-22T17:12:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:53.262 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/rawstats.20220422: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-22T17:12:53.263 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/peerstats.20220422: Permission denied 2022-04-22T17:12:53.264 INFO:teuthology.orchestra.run.smithi121.stdout:22 Apr 17:12:53 ntpd[7857]: ntpd: time slew -0.001090 s 2022-04-22T17:12:53.264 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew -0.001090s 2022-04-22T17:12:53.264 INFO:teuthology.orchestra.run.smithi121.stderr:22 Apr 17:12:53 ntpd[7857]: can't open /var/log/ntpstats/loopstats.20220422: Permission denied 2022-04-22T17:12:53.386 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-04-22T17:12:53.387 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-04-22T17:12:53.387 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:53.387 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:53.387 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:53.387 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-22T17:12:53.388 INFO:teuthology.run_tasks:Running task install... 2022-04-22T17:12:53.398 DEBUG:teuthology.task.install:project ceph 2022-04-22T17:12:53.398 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '720674591667bc34cb7086618bbed8722a1f429d'}} 2022-04-22T17:12:53.398 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '720674591667bc34cb7086618bbed8722a1f429d'} 2022-04-22T17:12:53.398 INFO:teuthology.task.install:Using flavor: default 2022-04-22T17:12:53.406 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-04-22T17:12:53.406 INFO:teuthology.task.install:extra packages: [] 2022-04-22T17:12:53.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-04-22T17:12:53.454 INFO:teuthology.orchestra.run.smithi115.stdout:Ubuntu 2022-04-22T17:12:53.460 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:12:53.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-04-22T17:12:53.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-key list | grep Ceph 2022-04-22T17:12:53.504 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-04-22T17:12:53.508 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:12:53.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-key list | grep Ceph 2022-04-22T17:12:53.559 INFO:teuthology.orchestra.run.smithi115.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-04-22T17:12:53.604 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-04-22T17:12:53.617 INFO:teuthology.orchestra.run.smithi115.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-04-22T17:12:53.618 INFO:teuthology.orchestra.run.smithi115.stdout:uid [ unknown] Ceph.com (release key) 2022-04-22T17:12:53.619 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-04-22T17:12:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-04-22T17:12:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph.com (release key) 2022-04-22T17:12:53.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-04-22T17:12:53.662 INFO:teuthology.orchestra.run.smithi115.stdout:Ubuntu 2022-04-22T17:12:53.668 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:12:53.668 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-04-22T17:12:53.669 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-22T17:12:53.669 INFO:teuthology.packaging:ref: None 2022-04-22T17:12:53.669 INFO:teuthology.packaging:tag: None 2022-04-22T17:12:53.669 INFO:teuthology.packaging:branch: pacific 2022-04-22T17:12:53.669 INFO:teuthology.packaging:sha1: 720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:12:53.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=pacific 2022-04-22T17:12:53.710 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-04-22T17:12:53.716 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:12:53.716 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-04-22T17:12:53.717 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-22T17:12:53.717 INFO:teuthology.packaging:ref: None 2022-04-22T17:12:53.717 INFO:teuthology.packaging:tag: None 2022-04-22T17:12:53.717 INFO:teuthology.packaging:branch: pacific 2022-04-22T17:12:53.717 INFO:teuthology.packaging:sha1: 720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:12:53.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=pacific 2022-04-22T17:12:53.954 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default/ 2022-04-22T17:12:53.954 INFO:teuthology.task.install.deb:Package version is 16.2.7-924-gb42bd433-1focal 2022-04-22T17:12:53.971 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default/ 2022-04-22T17:12:53.972 INFO:teuthology.task.install.deb:Package version is 16.2.7-924-gb42bd433-1focal 2022-04-22T17:12:54.229 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:12:54.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-04-22T17:12:54.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-04-22T17:12:54.273 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:12:54.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-04-22T17:12:54.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2022-04-22T17:12:54.435 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-22T17:12:54.492 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-22T17:12:54.568 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:12:54.569 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-22T17:12:54.648 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-22T17:12:54.648 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-22T17:12:54.662 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal InRelease 2022-04-22T17:12:54.663 INFO:teuthology.orchestra.run.smithi115.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal InRelease 2022-04-22T17:12:54.725 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-04-22T17:12:54.728 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-22T17:12:54.728 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-04-22T17:12:54.731 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-22T17:12:54.791 INFO:teuthology.orchestra.run.smithi115.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal Release.gpg 2022-04-22T17:12:54.792 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal Release.gpg 2022-04-22T17:12:54.856 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 Packages [17.1 kB] 2022-04-22T17:12:54.857 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main i386 Packages [3,097 B] 2022-04-22T17:12:54.941 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main i386 Packages [3,097 B] 2022-04-22T17:12:54.942 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 Packages [17.1 kB] 2022-04-22T17:12:55.060 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 27.8 kB in 1s (45.6 kB/s) 2022-04-22T17:12:55.067 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 27.8 kB in 1s (45.6 kB/s) 2022-04-22T17:12:55.916 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:12:55.933 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:12:55.941 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.7-924-gb42bd433-1focal cephadm=16.2.7-924-gb42bd433-1focal ceph-mds=16.2.7-924-gb42bd433-1focal ceph-mgr=16.2.7-924-gb42bd433-1focal ceph-common=16.2.7-924-gb42bd433-1focal ceph-fuse=16.2.7-924-gb42bd433-1focal ceph-test=16.2.7-924-gb42bd433-1focal radosgw=16.2.7-924-gb42bd433-1focal python3-rados=16.2.7-924-gb42bd433-1focal python3-rgw=16.2.7-924-gb42bd433-1focal python3-cephfs=16.2.7-924-gb42bd433-1focal python3-rbd=16.2.7-924-gb42bd433-1focal libcephfs2=16.2.7-924-gb42bd433-1focal libcephfs-dev=16.2.7-924-gb42bd433-1focal librados2=16.2.7-924-gb42bd433-1focal librbd1=16.2.7-924-gb42bd433-1focal rbd-fuse=16.2.7-924-gb42bd433-1focal 2022-04-22T17:12:55.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.7-924-gb42bd433-1focal cephadm=16.2.7-924-gb42bd433-1focal ceph-mds=16.2.7-924-gb42bd433-1focal ceph-mgr=16.2.7-924-gb42bd433-1focal ceph-common=16.2.7-924-gb42bd433-1focal ceph-fuse=16.2.7-924-gb42bd433-1focal ceph-test=16.2.7-924-gb42bd433-1focal radosgw=16.2.7-924-gb42bd433-1focal python3-rados=16.2.7-924-gb42bd433-1focal python3-rgw=16.2.7-924-gb42bd433-1focal python3-cephfs=16.2.7-924-gb42bd433-1focal python3-rbd=16.2.7-924-gb42bd433-1focal libcephfs2=16.2.7-924-gb42bd433-1focal libcephfs-dev=16.2.7-924-gb42bd433-1focal librados2=16.2.7-924-gb42bd433-1focal librbd1=16.2.7-924-gb42bd433-1focal rbd-fuse=16.2.7-924-gb42bd433-1focal 2022-04-22T17:12:56.018 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2022-04-22T17:12:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-04-22T17:12:56.195 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2022-04-22T17:12:56.196 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2022-04-22T17:12:56.216 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-04-22T17:12:56.216 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-04-22T17:12:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:The following packages were automatically installed and are no longer required: 2022-04-22T17:12:56.333 INFO:teuthology.orchestra.run.smithi115.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-04-22T17:12:56.333 INFO:teuthology.orchestra.run.smithi115.stdout:Use 'sudo apt autoremove' to remove them. 2022-04-22T17:12:56.333 INFO:teuthology.orchestra.run.smithi115.stdout:The following additional packages will be installed: 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-04-22T17:12:56.334 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-04-22T17:12:56.335 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-04-22T17:12:56.336 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2022-04-22T17:12:56.336 INFO:teuthology.orchestra.run.smithi115.stdout:Suggested packages: 2022-04-22T17:12:56.336 INFO:teuthology.orchestra.run.smithi115.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-04-22T17:12:56.336 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-04-22T17:12:56.336 INFO:teuthology.orchestra.run.smithi115.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout:Recommended packages: 2022-04-22T17:12:56.337 INFO:teuthology.orchestra.run.smithi115.stdout: btrfs-tools 2022-04-22T17:12:56.365 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-04-22T17:12:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-04-22T17:12:56.367 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2022-04-22T17:12:56.367 INFO:teuthology.orchestra.run.smithi121.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-04-22T17:12:56.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-04-22T17:12:56.369 INFO:teuthology.orchestra.run.smithi121.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-04-22T17:12:56.370 INFO:teuthology.orchestra.run.smithi121.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-04-22T17:12:56.370 INFO:teuthology.orchestra.run.smithi121.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-04-22T17:12:56.370 INFO:teuthology.orchestra.run.smithi121.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-04-22T17:12:56.370 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2022-04-22T17:12:56.370 INFO:teuthology.orchestra.run.smithi121.stdout: btrfs-tools 2022-04-22T17:12:56.444 INFO:teuthology.orchestra.run.smithi115.stdout:The following NEW packages will be installed: 2022-04-22T17:12:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-04-22T17:12:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2022-04-22T17:12:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2022-04-22T17:12:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-04-22T17:12:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-04-22T17:12:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-04-22T17:12:56.448 INFO:teuthology.orchestra.run.smithi115.stdout:The following packages will be upgraded: 2022-04-22T17:12:56.449 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 librbd1 2022-04-22T17:12:56.474 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-04-22T17:12:56.474 INFO:teuthology.orchestra.run.smithi121.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-04-22T17:12:56.474 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2022-04-22T17:12:56.475 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2022-04-22T17:12:56.475 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-04-22T17:12:56.475 INFO:teuthology.orchestra.run.smithi121.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-04-22T17:12:56.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-04-22T17:12:56.477 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-04-22T17:12:56.478 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 librbd1 2022-04-22T17:12:56.631 INFO:teuthology.orchestra.run.smithi115.stdout:2 upgraded, 99 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:12:56.631 INFO:teuthology.orchestra.run.smithi115.stdout:Need to get 173 MB of archives. 2022-04-22T17:12:56.631 INFO:teuthology.orchestra.run.smithi115.stdout:After this operation, 799 MB of additional disk space will be used. 2022-04-22T17:12:56.631 INFO:teuthology.orchestra.run.smithi115.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-04-22T17:12:56.663 INFO:teuthology.orchestra.run.smithi121.stdout:2 upgraded, 99 newly installed, 0 to remove and 9 not upgraded. 2022-04-22T17:12:56.664 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 173 MB of archives. 2022-04-22T17:12:56.664 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 799 MB of additional disk space will be used. 2022-04-22T17:12:56.664 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-04-22T17:12:56.730 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.7-924-gb42bd433-1focal [3,844 B] 2022-04-22T17:12:56.762 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.7-924-gb42bd433-1focal [3,844 B] 2022-04-22T17:12:56.890 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.7-924-gb42bd433-1focal [3,352 kB] 2022-04-22T17:12:56.921 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.7-924-gb42bd433-1focal [3,352 kB] 2022-04-22T17:12:57.028 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-04-22T17:12:57.036 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-04-22T17:12:57.063 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-04-22T17:12:57.070 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-04-22T17:12:57.079 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-04-22T17:12:57.108 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-04-22T17:12:57.113 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-04-22T17:12:57.142 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-04-22T17:12:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-04-22T17:12:57.238 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-04-22T17:12:57.245 INFO:teuthology.orchestra.run.smithi115.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-04-22T17:12:57.250 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-04-22T17:12:57.272 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-04-22T17:12:57.276 INFO:teuthology.orchestra.run.smithi115.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-04-22T17:12:57.279 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-04-22T17:12:57.280 INFO:teuthology.orchestra.run.smithi115.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-04-22T17:12:57.283 INFO:teuthology.orchestra.run.smithi115.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-04-22T17:12:57.298 INFO:teuthology.orchestra.run.smithi115.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-04-22T17:12:57.298 INFO:teuthology.orchestra.run.smithi115.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-04-22T17:12:57.300 INFO:teuthology.orchestra.run.smithi115.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-04-22T17:12:57.309 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-04-22T17:12:57.314 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-04-22T17:12:57.317 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-04-22T17:12:57.332 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-04-22T17:12:57.333 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-04-22T17:12:57.335 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-04-22T17:12:57.350 INFO:teuthology.orchestra.run.smithi115.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-04-22T17:12:57.353 INFO:teuthology.orchestra.run.smithi115.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-04-22T17:12:57.354 INFO:teuthology.orchestra.run.smithi115.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-04-22T17:12:57.384 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-04-22T17:12:57.387 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-04-22T17:12:57.388 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-04-22T17:12:57.430 INFO:teuthology.orchestra.run.smithi115.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-04-22T17:12:57.433 INFO:teuthology.orchestra.run.smithi115.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-04-22T17:12:57.434 INFO:teuthology.orchestra.run.smithi115.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-04-22T17:12:57.438 INFO:teuthology.orchestra.run.smithi115.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-04-22T17:12:57.441 INFO:teuthology.orchestra.run.smithi115.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-04-22T17:12:57.441 INFO:teuthology.orchestra.run.smithi115.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-04-22T17:12:57.462 INFO:teuthology.orchestra.run.smithi115.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-04-22T17:12:57.465 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-04-22T17:12:57.468 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-04-22T17:12:57.469 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-04-22T17:12:57.473 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-04-22T17:12:57.476 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-04-22T17:12:57.476 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-04-22T17:12:57.496 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-04-22T17:12:57.509 INFO:teuthology.orchestra.run.smithi115.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-04-22T17:12:57.510 INFO:teuthology.orchestra.run.smithi115.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-04-22T17:12:57.511 INFO:teuthology.orchestra.run.smithi115.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-04-22T17:12:57.545 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-04-22T17:12:57.545 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-04-22T17:12:57.546 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-04-22T17:12:57.589 INFO:teuthology.orchestra.run.smithi115.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-04-22T17:12:57.595 INFO:teuthology.orchestra.run.smithi115.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-04-22T17:12:57.598 INFO:teuthology.orchestra.run.smithi115.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-04-22T17:12:57.599 INFO:teuthology.orchestra.run.smithi115.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-04-22T17:12:57.603 INFO:teuthology.orchestra.run.smithi115.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-04-22T17:12:57.606 INFO:teuthology.orchestra.run.smithi115.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-04-22T17:12:57.609 INFO:teuthology.orchestra.run.smithi115.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-04-22T17:12:57.625 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-04-22T17:12:57.630 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-04-22T17:12:57.634 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-04-22T17:12:57.634 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-04-22T17:12:57.638 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-04-22T17:12:57.641 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-04-22T17:12:57.644 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-04-22T17:12:57.669 INFO:teuthology.orchestra.run.smithi115.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-04-22T17:12:57.670 INFO:teuthology.orchestra.run.smithi115.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-04-22T17:12:57.705 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-04-22T17:12:57.705 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-04-22T17:12:57.734 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.7-924-gb42bd433-1focal [3,372 kB] 2022-04-22T17:12:58.034 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-04-22T17:12:58.090 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-04-22T17:12:58.091 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-04-22T17:12:58.132 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-04-22T17:12:58.132 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-04-22T17:12:58.136 INFO:teuthology.orchestra.run.smithi121.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-04-22T17:12:58.137 INFO:teuthology.orchestra.run.smithi121.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-04-22T17:12:58.144 INFO:teuthology.orchestra.run.smithi115.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-04-22T17:12:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-04-22T17:12:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-04-22T17:12:58.159 INFO:teuthology.orchestra.run.smithi121.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-04-22T17:12:58.169 INFO:teuthology.orchestra.run.smithi115.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-04-22T17:12:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-04-22T17:12:58.197 INFO:teuthology.orchestra.run.smithi121.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-04-22T17:12:58.197 INFO:teuthology.orchestra.run.smithi121.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-04-22T17:12:58.206 INFO:teuthology.orchestra.run.smithi121.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.7-924-gb42bd433-1focal [722 kB] 2022-04-22T17:12:58.249 INFO:teuthology.orchestra.run.smithi115.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-04-22T17:12:58.250 INFO:teuthology.orchestra.run.smithi115.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-04-22T17:12:58.251 INFO:teuthology.orchestra.run.smithi115.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-04-22T17:12:58.251 INFO:teuthology.orchestra.run.smithi115.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-04-22T17:12:58.252 INFO:teuthology.orchestra.run.smithi115.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-04-22T17:12:58.252 INFO:teuthology.orchestra.run.smithi115.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-04-22T17:12:58.253 INFO:teuthology.orchestra.run.smithi115.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-04-22T17:12:58.278 INFO:teuthology.orchestra.run.smithi121.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-04-22T17:12:58.282 INFO:teuthology.orchestra.run.smithi121.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-04-22T17:12:58.285 INFO:teuthology.orchestra.run.smithi121.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-04-22T17:12:58.286 INFO:teuthology.orchestra.run.smithi121.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.7-924-gb42bd433-1focal [408 kB] 2022-04-22T17:12:58.288 INFO:teuthology.orchestra.run.smithi121.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-04-22T17:12:58.289 INFO:teuthology.orchestra.run.smithi121.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-22T17:12:58.292 INFO:teuthology.orchestra.run.smithi121.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-04-22T17:12:58.293 INFO:teuthology.orchestra.run.smithi121.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-04-22T17:12:58.295 INFO:teuthology.orchestra.run.smithi121.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-04-22T17:12:58.305 INFO:teuthology.orchestra.run.smithi115.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-04-22T17:12:58.305 INFO:teuthology.orchestra.run.smithi115.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-04-22T17:12:58.319 INFO:teuthology.orchestra.run.smithi121.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.7-924-gb42bd433-1focal [22.0 kB] 2022-04-22T17:12:58.319 INFO:teuthology.orchestra.run.smithi121.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.7-924-gb42bd433-1focal [212 kB] 2022-04-22T17:12:58.342 INFO:teuthology.orchestra.run.smithi121.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.7-924-gb42bd433-1focal [36.4 kB] 2022-04-22T17:12:58.343 INFO:teuthology.orchestra.run.smithi121.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.7-924-gb42bd433-1focal [385 kB] 2022-04-22T17:12:58.357 INFO:teuthology.orchestra.run.smithi121.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-04-22T17:12:58.357 INFO:teuthology.orchestra.run.smithi121.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-04-22T17:12:58.384 INFO:teuthology.orchestra.run.smithi115.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-04-22T17:12:58.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.7-924-gb42bd433-1focal [3,701 kB] 2022-04-22T17:12:58.389 INFO:teuthology.orchestra.run.smithi115.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-04-22T17:12:58.392 INFO:teuthology.orchestra.run.smithi115.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-04-22T17:12:58.394 INFO:teuthology.orchestra.run.smithi115.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-04-22T17:12:58.395 INFO:teuthology.orchestra.run.smithi115.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-22T17:12:58.397 INFO:teuthology.orchestra.run.smithi115.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-04-22T17:12:58.399 INFO:teuthology.orchestra.run.smithi115.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-04-22T17:12:58.401 INFO:teuthology.orchestra.run.smithi115.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-04-22T17:12:58.435 INFO:teuthology.orchestra.run.smithi115.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.7-924-gb42bd433-1focal [3,372 kB] 2022-04-22T17:12:58.437 INFO:teuthology.orchestra.run.smithi121.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-04-22T17:12:58.437 INFO:teuthology.orchestra.run.smithi121.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-04-22T17:12:58.439 INFO:teuthology.orchestra.run.smithi121.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-04-22T17:12:58.458 INFO:teuthology.orchestra.run.smithi121.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-04-22T17:12:58.458 INFO:teuthology.orchestra.run.smithi121.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-04-22T17:12:58.458 INFO:teuthology.orchestra.run.smithi121.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-04-22T17:12:58.459 INFO:teuthology.orchestra.run.smithi121.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-04-22T17:12:58.460 INFO:teuthology.orchestra.run.smithi121.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-04-22T17:12:58.463 INFO:teuthology.orchestra.run.smithi115.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-04-22T17:12:58.464 INFO:teuthology.orchestra.run.smithi115.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-04-22T17:12:58.517 INFO:teuthology.orchestra.run.smithi121.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-04-22T17:12:58.517 INFO:teuthology.orchestra.run.smithi121.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-04-22T17:12:58.543 INFO:teuthology.orchestra.run.smithi115.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-04-22T17:12:58.544 INFO:teuthology.orchestra.run.smithi115.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-04-22T17:12:58.546 INFO:teuthology.orchestra.run.smithi115.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-04-22T17:12:58.597 INFO:teuthology.orchestra.run.smithi121.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-04-22T17:12:58.600 INFO:teuthology.orchestra.run.smithi121.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-04-22T17:12:58.697 INFO:teuthology.orchestra.run.smithi115.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-04-22T17:12:58.698 INFO:teuthology.orchestra.run.smithi115.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-04-22T17:12:58.699 INFO:teuthology.orchestra.run.smithi115.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-04-22T17:12:58.699 INFO:teuthology.orchestra.run.smithi115.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-04-22T17:12:58.700 INFO:teuthology.orchestra.run.smithi115.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-04-22T17:12:58.701 INFO:teuthology.orchestra.run.smithi115.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-04-22T17:12:58.701 INFO:teuthology.orchestra.run.smithi115.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-04-22T17:12:58.702 INFO:teuthology.orchestra.run.smithi115.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-04-22T17:12:58.713 INFO:teuthology.orchestra.run.smithi121.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.7-924-gb42bd433-1focal [116 kB] 2022-04-22T17:12:58.728 INFO:teuthology.orchestra.run.smithi121.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.7-924-gb42bd433-1focal [447 kB] 2022-04-22T17:12:58.762 INFO:teuthology.orchestra.run.smithi121.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.7-924-gb42bd433-1focal [22.6 MB] 2022-04-22T17:12:58.769 INFO:teuthology.orchestra.run.smithi115.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-04-22T17:12:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.7-924-gb42bd433-1focal [722 kB] 2022-04-22T17:12:59.267 INFO:teuthology.orchestra.run.smithi115.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.7-924-gb42bd433-1focal [408 kB] 2022-04-22T17:12:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.7-924-gb42bd433-1focal [22.0 kB] 2022-04-22T17:12:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.7-924-gb42bd433-1focal [212 kB] 2022-04-22T17:12:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.7-924-gb42bd433-1focal [36.4 kB] 2022-04-22T17:12:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.7-924-gb42bd433-1focal [385 kB] 2022-04-22T17:12:59.426 INFO:teuthology.orchestra.run.smithi115.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.7-924-gb42bd433-1focal [3,701 kB] 2022-04-22T17:13:00.022 INFO:teuthology.orchestra.run.smithi115.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.7-924-gb42bd433-1focal [116 kB] 2022-04-22T17:13:00.023 INFO:teuthology.orchestra.run.smithi115.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.7-924-gb42bd433-1focal [447 kB] 2022-04-22T17:13:00.028 INFO:teuthology.orchestra.run.smithi115.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.7-924-gb42bd433-1focal [22.6 MB] 2022-04-22T17:13:00.214 INFO:teuthology.orchestra.run.smithi121.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.7-924-gb42bd433-1focal [5,901 kB] 2022-04-22T17:13:00.365 INFO:teuthology.orchestra.run.smithi121.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.7-924-gb42bd433-1focal [197 kB] 2022-04-22T17:13:00.367 INFO:teuthology.orchestra.run.smithi121.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.7-924-gb42bd433-1focal [1,521 kB] 2022-04-22T17:13:00.440 INFO:teuthology.orchestra.run.smithi121.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.7-924-gb42bd433-1focal [6,958 kB] 2022-04-22T17:13:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.7-924-gb42bd433-1focal [26.1 MB] 2022-04-22T17:13:02.036 INFO:teuthology.orchestra.run.smithi121.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.7-924-gb42bd433-1focal [3,940 B] 2022-04-22T17:13:02.036 INFO:teuthology.orchestra.run.smithi121.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.7-924-gb42bd433-1focal [828 kB] 2022-04-22T17:13:02.042 INFO:teuthology.orchestra.run.smithi121.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.7-924-gb42bd433-1focal [2,337 kB] 2022-04-22T17:13:02.169 INFO:teuthology.orchestra.run.smithi121.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.7-924-gb42bd433-1focal [73.2 kB] 2022-04-22T17:13:02.207 INFO:teuthology.orchestra.run.smithi121.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.7-924-gb42bd433-1focal [93.2 kB] 2022-04-22T17:13:02.237 INFO:teuthology.orchestra.run.smithi121.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.7-924-gb42bd433-1focal [2,566 kB] 2022-04-22T17:13:02.367 INFO:teuthology.orchestra.run.smithi121.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.7-924-gb42bd433-1focal [7,890 kB] 2022-04-22T17:13:02.618 INFO:teuthology.orchestra.run.smithi121.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.7-924-gb42bd433-1focal [4,056 B] 2022-04-22T17:13:02.619 INFO:teuthology.orchestra.run.smithi121.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.7-924-gb42bd433-1focal [126 kB] 2022-04-22T17:13:02.621 INFO:teuthology.orchestra.run.smithi121.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.7-924-gb42bd433-1focal [49.6 MB] 2022-04-22T17:13:03.992 INFO:teuthology.orchestra.run.smithi115.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.7-924-gb42bd433-1focal [5,901 kB] 2022-04-22T17:13:04.753 INFO:teuthology.orchestra.run.smithi115.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.7-924-gb42bd433-1focal [197 kB] 2022-04-22T17:13:04.758 INFO:teuthology.orchestra.run.smithi115.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.7-924-gb42bd433-1focal [1,521 kB] 2022-04-22T17:13:04.945 INFO:teuthology.orchestra.run.smithi115.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.7-924-gb42bd433-1focal [6,958 kB] 2022-04-22T17:13:04.979 INFO:teuthology.orchestra.run.smithi121.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.7-924-gb42bd433-1focal [21.0 kB] 2022-04-22T17:13:04.999 INFO:teuthology.orchestra.run.smithi121.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.7-924-gb42bd433-1focal [11.3 MB] 2022-04-22T17:13:05.326 INFO:teuthology.orchestra.run.smithi121.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.7-924-gb42bd433-1focal [78.0 kB] 2022-04-22T17:13:05.931 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 173 MB in 9s (19.6 MB/s) 2022-04-22T17:13:05.981 INFO:teuthology.orchestra.run.smithi115.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.7-924-gb42bd433-1focal [26.1 MB] 2022-04-22T17:13:06.027 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjaeger. 2022-04-22T17:13:06.071 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109370 files and directories currently installed.) 2022-04-22T17:13:06.074 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../000-libjaeger_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:06.109 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjaeger (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:06.428 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-04-22T17:13:06.442 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-04-22T17:13:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-22T17:13:06.897 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../002-librbd1_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:07.024 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librbd1 (16.2.7-924-gb42bd433-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-22T17:13:07.682 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../003-librados2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:07.803 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librados2 (16.2.7-924-gb42bd433-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-22T17:13:08.398 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs2. 2022-04-22T17:13:08.411 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../004-libcephfs2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:08.440 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:08.759 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rados. 2022-04-22T17:13:08.772 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../005-python3-rados_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:08.808 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rados (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:09.101 INFO:teuthology.orchestra.run.smithi115.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.7-924-gb42bd433-1focal [3,940 B] 2022-04-22T17:13:09.101 INFO:teuthology.orchestra.run.smithi115.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.7-924-gb42bd433-1focal [828 kB] 2022-04-22T17:13:09.112 INFO:teuthology.orchestra.run.smithi115.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.7-924-gb42bd433-1focal [2,337 kB] 2022-04-22T17:13:09.153 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-04-22T17:13:09.165 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:09.200 INFO:teuthology.orchestra.run.smithi115.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.7-924-gb42bd433-1focal [73.2 kB] 2022-04-22T17:13:09.202 INFO:teuthology.orchestra.run.smithi115.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.7-924-gb42bd433-1focal [93.2 kB] 2022-04-22T17:13:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-argparse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:09.214 INFO:teuthology.orchestra.run.smithi115.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.7-924-gb42bd433-1focal [2,566 kB] 2022-04-22T17:13:09.312 INFO:teuthology.orchestra.run.smithi115.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.7-924-gb42bd433-1focal [7,890 kB] 2022-04-22T17:13:09.539 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cephfs. 2022-04-22T17:13:09.552 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../007-python3-cephfs_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:09.597 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cephfs (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:09.645 INFO:teuthology.orchestra.run.smithi115.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.7-924-gb42bd433-1focal [4,056 B] 2022-04-22T17:13:09.645 INFO:teuthology.orchestra.run.smithi115.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.7-924-gb42bd433-1focal [126 kB] 2022-04-22T17:13:09.646 INFO:teuthology.orchestra.run.smithi115.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.7-924-gb42bd433-1focal [49.6 MB] 2022-04-22T17:13:09.917 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-common. 2022-04-22T17:13:09.930 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../008-python3-ceph-common_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:09.975 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:10.282 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-prettytable. 2022-04-22T17:13:10.296 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2022-04-22T17:13:10.332 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-04-22T17:13:10.586 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rbd. 2022-04-22T17:13:10.598 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../010-python3-rbd_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rbd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:10.930 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-04-22T17:13:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-04-22T17:13:10.987 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-22T17:13:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-04-22T17:13:11.345 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2022-04-22T17:13:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-04-22T17:13:11.727 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-04-22T17:13:11.741 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-04-22T17:13:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-22T17:13:11.945 INFO:teuthology.orchestra.run.smithi115.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.7-924-gb42bd433-1focal [21.0 kB] 2022-04-22T17:13:11.945 INFO:teuthology.orchestra.run.smithi115.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.7-924-gb42bd433-1focal [11.3 MB] 2022-04-22T17:13:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librgw2. 2022-04-22T17:13:12.136 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../014-librgw2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librgw2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:12.363 INFO:teuthology.orchestra.run.smithi115.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/pacific/b42bd433b8c85da129c1c277665f5ddab82dd5fb/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.7-924-gb42bd433-1focal [78.0 kB] 2022-04-22T17:13:12.775 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rgw. 2022-04-22T17:13:12.788 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../015-python3-rgw_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:12.989 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 173 MB in 16s (10.9 MB/s) 2022-04-22T17:13:13.076 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjaeger. 2022-04-22T17:13:13.121 INFO:teuthology.orchestra.run.smithi115.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109370 files and directories currently installed.) 2022-04-22T17:13:13.124 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../000-libjaeger_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-04-22T17:13:13.158 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjaeger (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:13.159 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-04-22T17:13:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-04-22T17:13:13.526 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-04-22T17:13:13.541 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-04-22T17:13:13.747 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liboath0. 2022-04-22T17:13:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2022-04-22T17:13:13.803 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-22T17:13:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-04-22T17:13:14.117 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libradosstriper1. 2022-04-22T17:13:14.131 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../018-libradosstriper1_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:14.171 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../002-librbd1_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:14.175 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libradosstriper1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librbd1 (16.2.7-924-gb42bd433-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-22T17:13:14.628 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-common. 2022-04-22T17:13:14.642 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../019-ceph-common_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:14.686 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:14.965 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../003-librados2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:15.069 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librados2 (16.2.7-924-gb42bd433-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-22T17:13:15.739 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libcephfs2. 2022-04-22T17:13:15.754 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../004-libcephfs2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libcephfs2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:16.133 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rados. 2022-04-22T17:13:16.148 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../005-python3-rados_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:16.191 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rados (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:16.503 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-04-22T17:13:16.516 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:16.544 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-ceph-argparse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:16.855 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cephfs. 2022-04-22T17:13:16.939 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-base. 2022-04-22T17:13:16.940 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../020-ceph-base_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:16.940 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../007-python3-cephfs_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:16.940 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cephfs (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:16.986 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-base (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:17.221 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-ceph-common. 2022-04-22T17:13:17.235 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../008-python3-ceph-common_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:17.280 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:17.599 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-prettytable. 2022-04-22T17:13:17.612 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2022-04-22T17:13:17.666 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-04-22T17:13:17.876 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rbd. 2022-04-22T17:13:17.891 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../010-python3-rbd_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:17.927 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rbd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:17.951 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cryptography. 2022-04-22T17:13:17.964 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-04-22T17:13:18.001 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-22T17:13:18.288 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-04-22T17:13:18.300 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-04-22T17:13:18.338 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-22T17:13:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-openssl. 2022-04-22T17:13:18.353 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2022-04-22T17:13:18.397 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-04-22T17:13:18.646 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-04-22T17:13:18.672 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:18.704 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-modules-core (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:18.716 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-04-22T17:13:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2022-04-22T17:13:18.765 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-04-22T17:13:19.088 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-04-22T17:13:19.093 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bcrypt. 2022-04-22T17:13:19.102 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-04-22T17:13:19.106 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-04-22T17:13:19.143 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-22T17:13:19.145 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-22T17:13:19.446 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cherrypy3. 2022-04-22T17:13:19.461 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2022-04-22T17:13:19.504 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-04-22T17:13:19.506 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librgw2. 2022-04-22T17:13:19.519 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../014-librgw2_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:19.556 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librgw2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:19.832 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jwt. 2022-04-22T17:13:19.845 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-04-22T17:13:19.891 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-04-22T17:13:20.167 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rgw. 2022-04-22T17:13:20.180 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../015-python3-rgw_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:20.202 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-logutils. 2022-04-22T17:13:20.215 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2022-04-22T17:13:20.225 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:20.252 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-04-22T17:13:20.537 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-04-22T17:13:20.549 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-04-22T17:13:20.555 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-markupsafe. 2022-04-22T17:13:20.569 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-04-22T17:13:20.587 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-04-22T17:13:20.618 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-04-22T17:13:20.916 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-mako. 2022-04-22T17:13:20.930 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-04-22T17:13:20.965 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liboath0. 2022-04-22T17:13:20.974 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-22T17:13:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2022-04-22T17:13:21.023 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-04-22T17:13:21.294 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplegeneric. 2022-04-22T17:13:21.307 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2022-04-22T17:13:21.334 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libradosstriper1. 2022-04-22T17:13:21.344 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-04-22T17:13:21.347 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../018-libradosstriper1_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:21.409 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libradosstriper1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:21.648 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-singledispatch. 2022-04-22T17:13:21.661 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-04-22T17:13:21.698 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-04-22T17:13:21.862 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-common. 2022-04-22T17:13:21.875 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../019-ceph-common_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:21.913 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:22.002 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webob. 2022-04-22T17:13:22.013 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2022-04-22T17:13:22.052 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-04-22T17:13:22.320 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-soupsieve. 2022-04-22T17:13:22.333 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-04-22T17:13:22.362 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-22T17:13:22.622 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bs4. 2022-04-22T17:13:22.631 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2022-04-22T17:13:22.672 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-04-22T17:13:23.001 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-waitress. 2022-04-22T17:13:23.010 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-04-22T17:13:23.053 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-22T17:13:23.379 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-tempita. 2022-04-22T17:13:23.391 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2022-04-22T17:13:23.428 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-04-22T17:13:23.715 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-paste. 2022-04-22T17:13:23.727 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-04-22T17:13:23.764 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-04-22T17:13:24.086 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-04-22T17:13:24.099 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-04-22T17:13:24.143 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-04-22T17:13:24.164 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-base. 2022-04-22T17:13:24.178 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../020-ceph-base_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:24.220 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-base (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:24.445 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastedeploy. 2022-04-22T17:13:24.461 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2022-04-22T17:13:24.503 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-04-22T17:13:24.823 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webtest. 2022-04-22T17:13:24.838 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2022-04-22T17:13:24.881 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-04-22T17:13:25.184 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pecan. 2022-04-22T17:13:25.185 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cryptography. 2022-04-22T17:13:25.197 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-04-22T17:13:25.199 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-04-22T17:13:25.234 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-04-22T17:13:25.235 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-22T17:13:25.558 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-werkzeug. 2022-04-22T17:13:25.568 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-04-22T17:13:25.597 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-22T17:13:25.654 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-openssl. 2022-04-22T17:13:25.680 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2022-04-22T17:13:25.704 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-04-22T17:13:25.917 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr. 2022-04-22T17:13:25.930 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../043-ceph-mgr_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:25.967 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-04-22T17:13:25.976 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:26.024 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-modules-core (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:26.387 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-bcrypt. 2022-04-22T17:13:26.402 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-04-22T17:13:26.437 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-22T17:13:26.478 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mon. 2022-04-22T17:13:26.485 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../044-ceph-mon_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:26.528 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mon (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:26.723 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cherrypy3. 2022-04-22T17:13:26.737 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2022-04-22T17:13:26.773 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-04-22T17:13:27.143 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-jwt. 2022-04-22T17:13:27.158 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-04-22T17:13:27.201 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-04-22T17:13:27.512 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-logutils. 2022-04-22T17:13:27.526 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2022-04-22T17:13:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-04-22T17:13:27.615 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-osd. 2022-04-22T17:13:27.628 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../045-ceph-osd_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:27.673 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-osd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:27.849 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-markupsafe. 2022-04-22T17:13:27.864 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-04-22T17:13:27.907 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-04-22T17:13:28.234 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-mako. 2022-04-22T17:13:28.249 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-04-22T17:13:28.293 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-22T17:13:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-simplegeneric. 2022-04-22T17:13:28.586 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2022-04-22T17:13:28.629 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-04-22T17:13:28.952 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-singledispatch. 2022-04-22T17:13:28.967 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-04-22T17:13:29.010 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-04-22T17:13:29.937 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webob. 2022-04-22T17:13:29.950 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2022-04-22T17:13:29.987 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-04-22T17:13:30.324 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-soupsieve. 2022-04-22T17:13:30.336 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-04-22T17:13:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-22T17:13:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph. 2022-04-22T17:13:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../046-ceph_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:30.642 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-bs4. 2022-04-22T17:13:30.700 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2022-04-22T17:13:30.700 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-04-22T17:13:30.839 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-fuse. 2022-04-22T17:13:30.853 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../047-ceph-fuse_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:30.889 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:31.012 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-waitress. 2022-04-22T17:13:31.024 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-04-22T17:13:31.065 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-22T17:13:31.309 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mds. 2022-04-22T17:13:31.333 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../048-ceph-mds_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:31.367 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mds (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:31.454 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-tempita. 2022-04-22T17:13:31.456 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2022-04-22T17:13:31.457 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-04-22T17:13:31.734 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-paste. 2022-04-22T17:13:31.747 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-04-22T17:13:31.784 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-04-22T17:13:31.904 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephadm. 2022-04-22T17:13:31.916 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../049-cephadm_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:31.953 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:32.096 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-04-22T17:13:32.110 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-04-22T17:13:32.146 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-04-22T17:13:32.231 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jinja2. 2022-04-22T17:13:32.245 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2022-04-22T17:13:32.299 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-04-22T17:13:32.448 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pastedeploy. 2022-04-22T17:13:32.462 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2022-04-22T17:13:32.499 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-04-22T17:13:32.593 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-04-22T17:13:32.607 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:32.643 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:32.801 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webtest. 2022-04-22T17:13:32.815 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2022-04-22T17:13:32.859 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-04-22T17:13:32.962 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-repoze.lru. 2022-04-22T17:13:32.979 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2022-04-22T17:13:33.020 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-04-22T17:13:33.204 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pecan. 2022-04-22T17:13:33.216 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-04-22T17:13:33.254 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-04-22T17:13:33.257 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-underscore. 2022-04-22T17:13:33.270 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-04-22T17:13:33.315 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-22T17:13:33.567 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-04-22T17:13:33.582 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-04-22T17:13:33.625 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-werkzeug. 2022-04-22T17:13:33.626 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-22T17:13:33.638 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-04-22T17:13:33.675 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-22T17:13:33.945 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-routes. 2022-04-22T17:13:33.958 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2022-04-22T17:13:34.003 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-04-22T17:13:34.029 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr. 2022-04-22T17:13:34.042 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../043-ceph-mgr_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:34.087 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:34.324 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-04-22T17:13:34.360 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:34.372 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-dashboard (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mon. 2022-04-22T17:13:34.536 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../044-ceph-mon_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:34.581 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mon (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:35.272 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-decorator. 2022-04-22T17:13:35.287 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-04-22T17:13:35.330 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-04-22T17:13:35.633 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-04-22T17:13:35.646 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-04-22T17:13:35.652 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-osd. 2022-04-22T17:13:35.666 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../045-ceph-osd_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:35.691 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-22T17:13:35.735 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-osd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:35.994 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-scipy. 2022-04-22T17:13:36.009 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-04-22T17:13:36.052 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-04-22T17:13:37.336 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-04-22T17:13:37.349 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-04-22T17:13:37.394 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-22T17:13:37.923 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-joblib. 2022-04-22T17:13:37.937 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2022-04-22T17:13:37.981 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-04-22T17:13:38.318 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn. 2022-04-22T17:13:38.331 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-04-22T17:13:38.365 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph. 2022-04-22T17:13:38.367 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-22T17:13:38.379 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../046-ceph_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:38.457 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:38.776 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-fuse. 2022-04-22T17:13:38.809 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../047-ceph-fuse_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:38.834 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:39.026 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-04-22T17:13:39.041 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:39.296 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mds. 2022-04-22T17:13:39.310 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../048-ceph-mds_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:39.354 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mds (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:39.919 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package cephadm. 2022-04-22T17:13:39.929 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../049-cephadm_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:39.965 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:40.243 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-jinja2. 2022-04-22T17:13:40.284 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2022-04-22T17:13:40.284 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cachetools. 2022-04-22T17:13:40.284 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2022-04-22T17:13:40.293 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-04-22T17:13:40.320 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-04-22T17:13:40.613 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-04-22T17:13:40.623 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1. 2022-04-22T17:13:40.627 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:40.637 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-04-22T17:13:40.663 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:40.682 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-04-22T17:13:40.983 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-repoze.lru. 2022-04-22T17:13:40.996 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2022-04-22T17:13:41.027 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-04-22T17:13:41.033 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-04-22T17:13:41.040 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-04-22T17:13:41.077 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-22T17:13:41.302 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjs-underscore. 2022-04-22T17:13:41.315 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-04-22T17:13:41.360 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-22T17:13:41.363 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rsa. 2022-04-22T17:13:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2022-04-22T17:13:41.413 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rsa (4.0-3) ... 2022-04-22T17:13:41.604 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-04-22T17:13:41.648 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-04-22T17:13:41.649 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-22T17:13:41.707 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-google-auth. 2022-04-22T17:13:41.721 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2022-04-22T17:13:41.765 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-04-22T17:13:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-routes. 2022-04-22T17:13:42.004 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2022-04-22T17:13:42.052 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-04-22T17:13:42.085 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-blinker. 2022-04-22T17:13:42.100 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-04-22T17:13:42.144 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-22T17:13:42.368 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-04-22T17:13:42.381 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:42.422 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-oauthlib. 2022-04-22T17:13:42.437 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-04-22T17:13:42.443 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-dashboard (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-22T17:13:42.783 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-04-22T17:13:42.797 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-04-22T17:13:42.833 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-22T17:13:43.136 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-websocket. 2022-04-22T17:13:43.150 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-04-22T17:13:43.194 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-04-22T17:13:43.393 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-decorator. 2022-04-22T17:13:43.406 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-04-22T17:13:43.451 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-04-22T17:13:43.505 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-kubernetes. 2022-04-22T17:13:43.521 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-04-22T17:13:43.564 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-04-22T17:13:43.737 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-04-22T17:13:43.751 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-04-22T17:13:43.804 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-22T17:13:44.081 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-scipy. 2022-04-22T17:13:44.128 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-04-22T17:13:44.128 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:44.129 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-04-22T17:13:44.129 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-04-22T17:13:44.146 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-k8sevents (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:44.423 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libonig5:amd64. 2022-04-22T17:13:44.437 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../075-libonig5_6.9.4-1_amd64.deb ... 2022-04-22T17:13:44.473 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-04-22T17:13:44.826 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjq1:amd64. 2022-04-22T17:13:44.839 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../076-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-22T17:13:44.901 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:13:45.470 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package jq. 2022-04-22T17:13:45.485 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../077-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-22T17:13:45.528 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:13:45.674 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-04-22T17:13:45.690 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-04-22T17:13:45.724 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-22T17:13:45.764 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2022-04-22T17:13:45.777 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../078-socat_1.7.3.3-2_amd64.deb ... 2022-04-22T17:13:45.814 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-22T17:13:46.201 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-04-22T17:13:46.271 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../079-libxslt1.1_1.1.34-4_amd64.deb ... 2022-04-22T17:13:46.271 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-04-22T17:13:46.272 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-joblib. 2022-04-22T17:13:46.272 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2022-04-22T17:13:46.272 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-04-22T17:13:46.537 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package xmlstarlet. 2022-04-22T17:13:46.549 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-04-22T17:13:46.587 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-04-22T17:13:46.606 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-sklearn. 2022-04-22T17:13:46.619 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-04-22T17:13:46.685 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-22T17:13:47.043 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-04-22T17:13:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../081-libsqlite3-mod-ceph_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:47.101 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libsqlite3-mod-ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:47.379 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-test. 2022-04-22T17:13:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../082-ceph-test_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:47.414 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-04-22T17:13:47.428 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:47.437 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-test (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:47.472 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:48.575 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cachetools. 2022-04-22T17:13:48.591 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2022-04-22T17:13:48.633 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-04-22T17:13:48.936 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyasn1. 2022-04-22T17:13:48.949 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-04-22T17:13:48.978 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-04-22T17:13:49.306 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-04-22T17:13:49.320 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-04-22T17:13:49.398 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-22T17:13:49.717 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rsa. 2022-04-22T17:13:49.732 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2022-04-22T17:13:49.778 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rsa (4.0-3) ... 2022-04-22T17:13:50.112 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-google-auth. 2022-04-22T17:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2022-04-22T17:13:50.170 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-04-22T17:13:50.507 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-blinker. 2022-04-22T17:13:50.522 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-04-22T17:13:50.565 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-22T17:13:50.868 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-oauthlib. 2022-04-22T17:13:50.883 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-04-22T17:13:50.926 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-22T17:13:51.246 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-04-22T17:13:51.260 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-04-22T17:13:51.304 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-22T17:13:51.658 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-websocket. 2022-04-22T17:13:51.674 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-04-22T17:13:51.717 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-04-22T17:13:51.741 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs-dev. 2022-04-22T17:13:51.756 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../083-libcephfs-dev_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:51.799 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs-dev (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:52.003 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-kubernetes. 2022-04-22T17:13:52.018 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-04-22T17:13:52.077 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-04-22T17:13:52.193 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package nvme-cli. 2022-04-22T17:13:52.209 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../084-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-04-22T17:13:52.269 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-04-22T17:13:52.607 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-04-22T17:13:52.621 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_16.2.7-924-gb42bd433-1focal_all.deb ... 2022-04-22T17:13:52.658 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-atomicwrites. 2022-04-22T17:13:52.665 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-k8sevents (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:52.671 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../085-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-04-22T17:13:52.715 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-04-22T17:13:52.985 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libonig5:amd64. 2022-04-22T17:13:53.076 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../075-libonig5_6.9.4-1_amd64.deb ... 2022-04-22T17:13:53.076 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-04-22T17:13:53.076 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-attr. 2022-04-22T17:13:53.076 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../086-python3-attr_19.3.0-2_all.deb ... 2022-04-22T17:13:53.076 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-04-22T17:13:53.363 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webencodings. 2022-04-22T17:13:53.376 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../087-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-04-22T17:13:53.421 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-22T17:13:53.427 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjq1:amd64. 2022-04-22T17:13:53.434 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../076-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-22T17:13:53.479 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:13:53.724 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-html5lib. 2022-04-22T17:13:53.738 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../088-python3-html5lib_1.0.1-2_all.deb ... 2022-04-22T17:13:53.765 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package jq. 2022-04-22T17:13:53.783 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-04-22T17:13:53.783 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../077-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-22T17:13:53.823 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:13:54.101 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package socat. 2022-04-22T17:13:54.115 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../078-socat_1.7.3.3-2_amd64.deb ... 2022-04-22T17:13:54.127 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-04-22T17:13:54.142 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../089-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-04-22T17:13:54.159 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-22T17:13:54.185 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-22T17:13:54.521 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-04-22T17:13:54.535 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../079-libxslt1.1_1.1.34-4_amd64.deb ... 2022-04-22T17:13:54.579 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-04-22T17:13:54.630 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-packaging. 2022-04-22T17:13:54.644 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../090-python3-packaging_20.3-1_all.deb ... 2022-04-22T17:13:54.680 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-packaging (20.3-1) ... 2022-04-22T17:13:54.857 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package xmlstarlet. 2022-04-22T17:13:54.871 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-04-22T17:13:54.915 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-04-22T17:13:54.941 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastescript. 2022-04-22T17:13:54.955 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../091-python3-pastescript_2.0.2-4_all.deb ... 2022-04-22T17:13:54.999 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-04-22T17:13:55.311 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pluggy. 2022-04-22T17:13:55.326 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../092-python3-pluggy_0.13.0-2_all.deb ... 2022-04-22T17:13:55.388 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-04-22T17:13:55.394 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-04-22T17:13:55.401 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../081-libsqlite3-mod-ceph_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:55.447 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libsqlite3-mod-ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:55.705 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-psutil. 2022-04-22T17:13:55.720 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../093-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-04-22T17:13:55.750 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-test. 2022-04-22T17:13:55.755 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-04-22T17:13:55.763 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../082-ceph-test_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:55.807 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-test (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-py. 2022-04-22T17:13:56.058 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../094-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-04-22T17:13:56.091 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-04-22T17:13:56.378 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyinotify. 2022-04-22T17:13:56.392 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../095-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-04-22T17:13:56.430 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-22T17:13:56.731 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-wcwidth. 2022-04-22T17:13:56.746 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../096-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-04-22T17:13:56.789 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-22T17:13:57.092 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pytest. 2022-04-22T17:13:57.106 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../097-python3-pytest_4.6.9-1_all.deb ... 2022-04-22T17:13:57.150 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-04-22T17:13:57.487 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplejson. 2022-04-22T17:13:57.507 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../098-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-04-22T17:13:57.545 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-22T17:13:57.965 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package radosgw. 2022-04-22T17:13:57.978 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../099-radosgw_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:58.015 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking radosgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:59.226 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package rbd-fuse. 2022-04-22T17:13:59.241 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../100-rbd-fuse_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:13:59.284 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking rbd-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:13:59.592 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-attr (19.3.0-2) ... 2022-04-22T17:13:59.914 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-04-22T17:14:00.023 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-04-22T17:14:00.036 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libcephfs-dev. 2022-04-22T17:14:00.050 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../083-libcephfs-dev_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:14:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libcephfs-dev (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:00.149 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-04-22T17:14:00.342 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package nvme-cli. 2022-04-22T17:14:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../084-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-04-22T17:14:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-04-22T17:14:00.929 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user cephadm....done 2022-04-22T17:14:00.989 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-atomicwrites. 2022-04-22T17:14:01.001 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../085-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-04-22T17:14:01.046 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-04-22T17:14:01.091 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-22T17:14:01.308 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-attr. 2022-04-22T17:14:01.322 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../086-python3-attr_19.3.0-2_all.deb ... 2022-04-22T17:14:01.358 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-04-22T17:14:01.438 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-04-22T17:14:01.693 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webencodings. 2022-04-22T17:14:01.705 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-04-22T17:14:01.707 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../087-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-04-22T17:14:01.751 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-22T17:14:01.815 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-04-22T17:14:02.062 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-html5lib. 2022-04-22T17:14:02.076 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../088-python3-html5lib_1.0.1-2_all.deb ... 2022-04-22T17:14:02.120 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-04-22T17:14:02.141 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-04-22T17:14:02.474 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-04-22T17:14:02.487 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../089-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-04-22T17:14:02.523 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-22T17:14:02.526 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-04-22T17:14:02.821 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-04-22T17:14:03.011 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-packaging. 2022-04-22T17:14:03.024 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../090-python3-packaging_20.3-1_all.deb ... 2022-04-22T17:14:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-packaging (20.3-1) ... 2022-04-22T17:14:03.105 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-22T17:14:03.354 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pastescript. 2022-04-22T17:14:03.368 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../091-python3-pastescript_2.0.2-4_all.deb ... 2022-04-22T17:14:03.373 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-22T17:14:03.412 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-04-22T17:14:03.649 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-argparse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:03.775 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pluggy. 2022-04-22T17:14:03.790 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../092-python3-pluggy_0.13.0-2_all.deb ... 2022-04-22T17:14:03.832 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-04-22T17:14:03.925 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-22T17:14:04.051 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-04-22T17:14:04.127 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-psutil. 2022-04-22T17:14:04.142 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../093-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-04-22T17:14:04.185 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-04-22T17:14:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-22T17:14:04.538 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-py. 2022-04-22T17:14:04.552 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../094-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-04-22T17:14:04.596 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-04-22T17:14:04.612 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-04-22T17:14:04.908 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyinotify. 2022-04-22T17:14:04.923 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../095-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-04-22T17:14:04.983 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-22T17:14:04.988 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-04-22T17:14:05.256 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-22T17:14:05.294 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-wcwidth. 2022-04-22T17:14:05.322 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../096-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-04-22T17:14:05.352 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-22T17:14:05.599 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-04-22T17:14:05.671 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pytest. 2022-04-22T17:14:05.686 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../097-python3-pytest_4.6.9-1_all.deb ... 2022-04-22T17:14:05.729 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-04-22T17:14:05.891 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjaeger (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:06.017 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-04-22T17:14:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-simplejson. 2022-04-22T17:14:06.072 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../098-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-04-22T17:14:06.116 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-22T17:14:06.352 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-04-22T17:14:06.580 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package radosgw. 2022-04-22T17:14:06.594 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../099-radosgw_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:14:06.636 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking radosgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:06.645 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-packaging (20.3-1) ... 2022-04-22T17:14:06.938 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-04-22T17:14:07.069 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-22T17:14:07.366 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-04-22T17:14:07.758 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-22T17:14:07.890 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package rbd-fuse. 2022-04-22T17:14:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../100-rbd-fuse_16.2.7-924-gb42bd433-1focal_amd64.deb ... 2022-04-22T17:14:07.947 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking rbd-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:08.101 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-22T17:14:08.240 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-attr (19.3.0-2) ... 2022-04-22T17:14:08.461 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-22T17:14:08.560 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-04-22T17:14:08.587 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:08.686 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-04-22T17:14:08.812 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-04-22T17:14:08.897 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-04-22T17:14:09.030 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:09.240 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-22T17:14:09.350 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-04-22T17:14:09.603 INFO:teuthology.orchestra.run.smithi115.stdout:Adding system user cephadm....done 2022-04-22T17:14:09.625 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-22T17:14:09.727 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-04-22T17:14:09.797 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-22T17:14:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-04-22T17:14:10.134 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-04-22T17:14:10.181 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-04-22T17:14:10.410 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-04-22T17:14:10.480 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-04-22T17:14:10.536 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-04-22T17:14:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-22T17:14:10.870 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-04-22T17:14:10.901 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-04-22T17:14:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-04-22T17:14:11.264 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-04-22T17:14:11.467 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-22T17:14:11.557 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-04-22T17:14:11.751 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-04-22T17:14:11.841 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-22T17:14:12.109 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-22T17:14:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-04-22T17:14:12.403 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-ceph-argparse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:12.446 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-04-22T17:14:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-22T17:14:12.686 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-22T17:14:12.812 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-04-22T17:14:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-04-22T17:14:13.097 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-22T17:14:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-22T17:14:13.342 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-22T17:14:13.372 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-04-22T17:14:13.443 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-04-22T17:14:13.765 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-04-22T17:14:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-22T17:14:14.050 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-22T17:14:14.079 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librados2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:14.205 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-04-22T17:14:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-04-22T17:14:14.694 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjaeger (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:14.820 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-04-22T17:14:15.154 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-04-22T17:14:15.447 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-packaging (20.3-1) ... 2022-04-22T17:14:15.669 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librgw2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:15.795 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libsqlite3-mod-ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:15.921 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:14:16.047 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-04-22T17:14:16.115 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-04-22T17:14:16.224 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-22T17:14:16.500 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-04-22T17:14:16.623 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-04-22T17:14:16.893 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-22T17:14:16.916 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-04-22T17:14:17.217 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-04-22T17:14:17.245 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-22T17:14:17.544 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:17.613 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-22T17:14:17.670 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-22T17:14:17.739 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:17.988 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libradosstriper1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:18.065 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-04-22T17:14:18.139 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-22T17:14:18.417 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-22T17:14:18.474 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-22T17:14:18.576 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-04-22T17:14:18.869 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-22T17:14:18.995 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-04-22T17:14:19.146 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-04-22T17:14:19.457 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-04-22T17:14:19.571 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librbd1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:19.671 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-22T17:14:19.768 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-04-22T17:14:19.781 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-04-22T17:14:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-22T17:14:20.075 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-22T17:14:20.209 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-04-22T17:14:20.216 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:14:20.342 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rsa (4.0-3) ... 2022-04-22T17:14:20.510 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-04-22T17:14:20.644 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-04-22T17:14:20.811 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-22T17:14:20.945 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-modules-core (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:21.054 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:21.133 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-04-22T17:14:21.250 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-22T17:14:21.250 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-22T17:14:21.486 INFO:teuthology.orchestra.run.smithi115.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-04-22T17:14:21.731 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-04-22T17:14:21.739 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs-dev (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:21.865 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-22T17:14:21.882 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-22T17:14:22.141 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-04-22T17:14:22.200 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-04-22T17:14:22.492 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-04-22T17:14:22.560 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-22T17:14:22.686 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-22T17:14:22.803 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-04-22T17:14:22.878 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rados (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:22.988 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rbd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:23.088 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up rbd-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:23.130 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-22T17:14:23.189 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-routes (2.4.1-2) ... 2022-04-22T17:14:23.439 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librados2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:23.558 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-04-22T17:14:23.565 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-04-22T17:14:23.862 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:23.988 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cephfs (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:24.281 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-04-22T17:14:25.029 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librgw2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:25.175 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libsqlite3-mod-ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:25.218 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:25.310 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:14:25.432 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-04-22T17:14:25.763 INFO:teuthology.orchestra.run.smithi121.stdout:Adding group ceph....done 2022-04-22T17:14:25.808 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-04-22T17:14:26.126 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-04-22T17:14:26.131 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user ceph....done 2022-04-22T17:14:26.307 INFO:teuthology.orchestra.run.smithi121.stdout:Setting system user ceph properties....done 2022-04-22T17:14:26.326 INFO:teuthology.orchestra.run.smithi121.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-04-22T17:14:26.427 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-04-22T17:14:26.471 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-04-22T17:14:26.770 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libcephfs2 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:26.809 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-04-22T17:14:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-22T17:14:27.231 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libradosstriper1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:27.311 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-test (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:27.366 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-22T17:14:27.445 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up radosgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:27.700 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-22T17:14:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-22T17:14:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-22T17:14:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-base (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:28.747 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librbd1 (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:28.830 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-04-22T17:14:28.931 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-22T17:14:28.999 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-04-22T17:14:29.126 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-22T17:14:29.325 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mds (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:29.477 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-04-22T17:14:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-22T17:14:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-22T17:14:29.602 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rsa (4.0-3) ... 2022-04-22T17:14:29.912 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-04-22T17:14:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:30.196 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-modules-core (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-22T17:14:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-22T17:14:30.314 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:30.537 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-22T17:14:30.538 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-22T17:14:30.769 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-osd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:31.041 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libcephfs-dev (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:31.167 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-22T17:14:31.350 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-22T17:14:31.350 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-22T17:14:31.508 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-04-22T17:14:31.828 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-04-22T17:14:31.846 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-k8sevents (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:31.955 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-local (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.098 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mon (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.196 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rados (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.322 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-22T17:14:32.322 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-22T17:14:32.323 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rbd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.457 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up rbd-fuse (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.608 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-routes (2.4.1-2) ... 2022-04-22T17:14:32.824 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.959 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:32.977 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-04-22T17:14:33.068 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-dashboard (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for systemd (245.4-4ubuntu3.16) ... 2022-04-22T17:14:33.292 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:33.404 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cephfs (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:33.522 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-22T17:14:33.684 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-04-22T17:14:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-common (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:35.195 INFO:teuthology.orchestra.run.smithi115.stdout:Adding group ceph....done 2022-04-22T17:14:35.597 INFO:teuthology.orchestra.run.smithi115.stdout:Adding system user ceph....done 2022-04-22T17:14:35.781 INFO:teuthology.orchestra.run.smithi115.stdout:Setting system user ceph properties....done 2022-04-22T17:14:35.804 INFO:teuthology.orchestra.run.smithi115.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-04-22T17:14:35.957 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-04-22T17:14:36.267 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-04-22T17:14:36.400 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-22T17:14:36.752 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-test (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:36.822 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-04-22T17:14:36.827 DEBUG:teuthology.parallel:result is None 2022-04-22T17:14:36.878 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up radosgw (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:37.148 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-22T17:14:37.175 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-22T17:14:37.839 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-base (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:38.219 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-04-22T17:14:38.707 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mds (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:38.921 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-22T17:14:38.921 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-22T17:14:39.408 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:39.628 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-22T17:14:39.628 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-22T17:14:40.117 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-osd (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:40.672 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-22T17:14:40.672 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-22T17:14:41.135 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-k8sevents (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:41.244 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-diskprediction-local (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:41.395 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mon (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:41.623 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-22T17:14:41.662 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-22T17:14:42.114 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-cephadm (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:42.257 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:42.383 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-dashboard (16.2.7-924-gb42bd433-1focal) ... 2022-04-22T17:14:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for systemd (245.4-4ubuntu3.16) ... 2022-04-22T17:14:42.811 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-22T17:14:45.713 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-22T17:14:46.146 INFO:teuthology.orchestra.run.smithi115.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-04-22T17:14:46.149 DEBUG:teuthology.parallel:result is None 2022-04-22T17:14:46.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-04-22T17:14:46.199 INFO:teuthology.orchestra.run.smithi115.stdout:Ubuntu 2022-04-22T17:14:46.204 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:14:46.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-22T17:14:46.205 INFO:teuthology.packaging:ref: None 2022-04-22T17:14:46.205 INFO:teuthology.packaging:tag: None 2022-04-22T17:14:46.205 INFO:teuthology.packaging:branch: pacific 2022-04-22T17:14:46.205 INFO:teuthology.packaging:sha1: 720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:14:46.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=pacific 2022-04-22T17:14:46.457 DEBUG:teuthology.orchestra.run.smithi115:> dpkg-query -W -f '${Version}' ceph 2022-04-22T17:14:46.478 INFO:teuthology.orchestra.run.smithi115.stdout:16.2.7-924-gb42bd433-1focal 2022-04-22T17:14:46.479 INFO:teuthology.packaging:The installed version of ceph is 16.2.7-924-gb42bd433-1focal 2022-04-22T17:14:46.479 INFO:teuthology.task.install:The correct ceph version 16.2.7-924-gb42bd433-1focal is installed. 2022-04-22T17:14:46.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-04-22T17:14:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-04-22T17:14:46.535 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-22T17:14:46.535 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-22T17:14:46.536 INFO:teuthology.packaging:ref: None 2022-04-22T17:14:46.536 INFO:teuthology.packaging:tag: None 2022-04-22T17:14:46.536 INFO:teuthology.packaging:branch: pacific 2022-04-22T17:14:46.536 INFO:teuthology.packaging:sha1: 720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:14:46.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=pacific 2022-04-22T17:14:46.769 DEBUG:teuthology.orchestra.run.smithi121:> dpkg-query -W -f '${Version}' ceph 2022-04-22T17:14:46.792 INFO:teuthology.orchestra.run.smithi121.stdout:16.2.7-924-gb42bd433-1focal 2022-04-22T17:14:46.792 INFO:teuthology.packaging:The installed version of ceph is 16.2.7-924-gb42bd433-1focal 2022-04-22T17:14:46.792 INFO:teuthology.task.install:The correct ceph version 16.2.7-924-gb42bd433-1focal is installed. 2022-04-22T17:14:46.793 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-04-22T17:14:46.793 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:14:46.793 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-22T17:14:46.809 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:14:46.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-22T17:14:46.824 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-04-22T17:14:46.825 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:14:46.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2022-04-22T17:14:46.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-22T17:14:46.902 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:14:46.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2022-04-22T17:14:46.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-22T17:14:46.930 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-04-22T17:14:46.931 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:14:46.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-22T17:14:46.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-22T17:14:47.009 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:14:47.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-22T17:14:47.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-22T17:14:47.040 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:14:47.060 INFO:teuthology.task.print:**** done install task... 2022-04-22T17:14:47.060 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:14:47.118 INFO:teuthology.task.print:**** done start installing pacific cephadm ... 2022-04-22T17:14:47.118 INFO:teuthology.run_tasks:Running task cephadm... 2022-04-22T17:14:47.248 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '720674591667bc34cb7086618bbed8722a1f429d'} 2022-04-22T17:14:47.317 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2022-04-22T17:14:47.318 INFO:tasks.cephadm:Cluster fsid is b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:14:47.318 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-04-22T17:14:47.318 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.115', 'mon.c': '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]', 'mon.b': '172.21.15.121'} 2022-04-22T17:14:47.318 INFO:tasks.cephadm:First mon is mon.a on smithi115 2022-04-22T17:14:47.318 INFO:tasks.cephadm:First mgr is y 2022-04-22T17:14:47.319 INFO:tasks.cephadm:Normalizing hostnames... 2022-04-22T17:14:47.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2022-04-22T17:14:47.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2022-04-22T17:14:47.392 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2022-04-22T17:14:47.392 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.601 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r-- 1 ubuntu ubuntu 336708 Apr 22 17:14 /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.601 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.723 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r-- 1 ubuntu ubuntu 336708 Apr 22 17:14 /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.724 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.733 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-04-22T17:14:47.757 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2022-04-22T17:14:47.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2022-04-22T17:14:47.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2022-04-22T17:14:48.019 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-04-22T17:14:48.056 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-04-22T17:15:16.495 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-04-22T17:15:16.623 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2022-04-22T17:15:16.623 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2022-04-22T17:15:16.623 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2022-04-22T17:15:16.623 INFO:teuthology.orchestra.run.smithi121.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2022-04-22T17:15:16.623 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2022-04-22T17:15:16.624 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-04-22T17:15:17.084 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2022-04-22T17:15:17.104 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:15:17.109 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2022-04-22T17:15:17.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2022-04-22T17:15:17.139 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2022-04-22T17:15:17.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2022-04-22T17:15:17.198 INFO:tasks.cephadm:Writing seed config... 2022-04-22T17:15:17.199 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-04-22T17:15:17.200 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-04-22T17:15:17.201 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-04-22T17:15:17.201 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:15:17.201 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-04-22T17:15:17.208 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b6061a0a-c25f-11ec-8c39-001a4aab830c mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2022-04-22T17:15:17.209 DEBUG:teuthology.orchestra.run.smithi115:mon.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service 2022-04-22T17:15:17.211 DEBUG:teuthology.orchestra.run.smithi115:mgr.y> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service 2022-04-22T17:15:17.239 INFO:journalctl@ceph.mon.a.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:15:17.240 INFO:tasks.cephadm:Bootstrapping... 2022-04-22T17:15:17.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.115 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-04-22T17:15:17.258 INFO:journalctl@ceph.mgr.y.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:15:17.430 INFO:teuthology.orchestra.run.smithi115.stderr:-------------------------------------------------------------------------------- 2022-04-22T17:15:17.430 INFO:teuthology.orchestra.run.smithi115.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', 'b6061a0a-c25f-11ec-8c39-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.115', '--skip-admin-label'] 2022-04-22T17:15:17.430 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying podman|docker is present... 2022-04-22T17:15:17.430 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying lvm2 is present... 2022-04-22T17:15:17.430 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying time synchronization is in place... 2022-04-22T17:15:17.436 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-04-22T17:15:17.442 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.449 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2022-04-22T17:15:17.455 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.462 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: masked 2022-04-22T17:15:17.468 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.475 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2022-04-22T17:15:17.482 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.490 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: enabled 2022-04-22T17:15:17.496 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: active 2022-04-22T17:15:17.498 INFO:teuthology.orchestra.run.smithi115.stderr:Unit ntp.service is enabled and running 2022-04-22T17:15:17.498 INFO:teuthology.orchestra.run.smithi115.stderr:Repeating the final host check... 2022-04-22T17:15:17.498 INFO:teuthology.orchestra.run.smithi115.stderr:docker (/usr/bin/docker) is present 2022-04-22T17:15:17.499 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl is present 2022-04-22T17:15:17.499 INFO:teuthology.orchestra.run.smithi115.stderr:lvcreate is present 2022-04-22T17:15:17.503 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-04-22T17:15:17.510 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.517 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2022-04-22T17:15:17.524 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.531 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: masked 2022-04-22T17:15:17.537 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.544 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2022-04-22T17:15:17.550 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2022-04-22T17:15:17.559 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: enabled 2022-04-22T17:15:17.566 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: active 2022-04-22T17:15:17.567 INFO:teuthology.orchestra.run.smithi115.stderr:Unit ntp.service is enabled and running 2022-04-22T17:15:17.567 INFO:teuthology.orchestra.run.smithi115.stderr:Host looks OK 2022-04-22T17:15:17.567 INFO:teuthology.orchestra.run.smithi115.stderr:Cluster fsid: b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:17.568 INFO:teuthology.orchestra.run.smithi115.stderr:Acquiring lock 139968946220816 on /run/cephadm/b6061a0a-c25f-11ec-8c39-001a4aab830c.lock 2022-04-22T17:15:17.568 INFO:teuthology.orchestra.run.smithi115.stderr:Lock 139968946220816 acquired on /run/cephadm/b6061a0a-c25f-11ec-8c39-001a4aab830c.lock 2022-04-22T17:15:17.568 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying IP 172.21.15.115 port 3300 ... 2022-04-22T17:15:17.568 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying IP 172.21.15.115 port 6789 ... 2022-04-22T17:15:17.569 INFO:teuthology.orchestra.run.smithi115.stderr:Base mon IP(s) is [172.21.15.115:3300, 172.21.15.115:6789], mon addrv is [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2022-04-22T17:15:17.573 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2022-04-22T17:15:17.573 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.115 metric 100 2022-04-22T17:15:17.573 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2022-04-22T17:15:17.574 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.115 2022-04-22T17:15:17.574 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.115 metric 100 2022-04-22T17:15:17.577 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2022-04-22T17:15:17.577 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2022-04-22T17:15:17.577 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1788sec pref medium 2022-04-22T17:15:17.581 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2022-04-22T17:15:17.582 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2022-04-22T17:15:17.582 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2022-04-22T17:15:17.582 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2022-04-22T17:15:17.582 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce8f/64 scope link 2022-04-22T17:15:17.582 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2022-04-22T17:15:17.584 INFO:teuthology.orchestra.run.smithi115.stderr:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2022-04-22T17:15:17.584 INFO:teuthology.orchestra.run.smithi115.stderr:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2022-04-22T17:15:17.584 INFO:teuthology.orchestra.run.smithi115.stderr:Mon IP `172.21.15.115` is in CIDR network `172.21.15.254` 2022-04-22T17:15:17.584 INFO:teuthology.orchestra.run.smithi115.stderr:Mon IP `172.21.15.115` is in CIDR network `172.21.15.254` 2022-04-22T17:15:17.585 INFO:teuthology.orchestra.run.smithi115.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.15.254', '172.21.15.254'] 2022-04-22T17:15:17.585 INFO:teuthology.orchestra.run.smithi115.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-04-22T17:15:17.585 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-04-22T17:15:18.047 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/docker: latest-pacific: Pulling from ceph/daemon-base 2022-04-22T17:15:18.146 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/docker: Digest: sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684 2022-04-22T17:15:18.146 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/docker: Status: Image is up to date for ceph/daemon-base:latest-pacific 2022-04-22T17:15:18.146 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/docker: docker.io/ceph/daemon-base:latest-pacific 2022-04-22T17:15:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-04-22T17:15:19.973 INFO:teuthology.orchestra.run.smithi115.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-04-22T17:15:19.974 INFO:teuthology.orchestra.run.smithi115.stderr:Extracting ceph user uid/gid from container image... 2022-04-22T17:15:21.045 INFO:teuthology.orchestra.run.smithi115.stderr:stat: 167 167 2022-04-22T17:15:21.778 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial keys... 2022-04-22T17:15:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQCq4mJigghSMRAA3QX5nnBSK7ZD2VsR8/pdYQ== 2022-04-22T17:15:24.533 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQCs4mJiBM+rHxAABFUa2YU5A7T6NZKuNGPhtg== 2022-04-22T17:15:26.248 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQCu4mJiHw7FDhAAd2YHmKEct2Mbg0WJZsdfHw== 2022-04-22T17:15:26.940 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial monmap... 2022-04-22T17:15:27.987 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2022-04-22T17:15:27.988 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:27.988 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-04-22T17:15:28.696 INFO:teuthology.orchestra.run.smithi115.stderr:monmaptool for a [v2:172.21.15.115:3300,v1:172.21.15.115:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2022-04-22T17:15:28.740 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: set fsid to b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:28.741 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-04-22T17:15:28.741 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:15:28.741 INFO:teuthology.orchestra.run.smithi115.stderr:Creating mon... 2022-04-22T17:15:29.860 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.824+0000 7f46e9b83700 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:15:29.860 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.824+0000 7f46e9b83700 1 imported monmap: 2022-04-22T17:15:29.860 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: epoch 0 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: last_changed 2022-04-22T17:15:27.986542+0000 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: created 2022-04-22T17:15:27.986542+0000 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: election_strategy: 1 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.824+0000 7f46e9b83700 0 /usr/bin/ceph-mon: set fsid to b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: RocksDB version: 6.8.1 2022-04-22T17:15:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Compile date Jul 8 2021 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: DB SUMMARY 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:15:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.create_if_missing: 1 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.env: 0x5622199a21c0 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.info_log: 0x56221b432f00 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:15:29.863 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:15:29.864 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:15:29.864 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:15:29.864 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:15:29.864 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:15:29.869 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:15:29.869 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.db_log_dir: 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:15:29.870 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:15:29.871 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.write_buffer_manager: 0x56221b43b500 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:15:29.872 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:15:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.row_cache: None 2022-04-22T17:15:29.874 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.wal_filter: None 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:15:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:15:29.876 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:15:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Compression algorithms supported: 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kZSTD supported: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:15:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.merge_operator: 2022-04-22T17:15:29.879 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:15:29.880 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:15:29.880 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:15:29.880 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.828+0000 7f46e9b83700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:15:29.880 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56221b34a160) 2022-04-22T17:15:29.881 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2022-04-22T17:15:29.881 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:15:29.881 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:15:29.881 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2022-04-22T17:15:29.881 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: index_type: 0 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: index_shortening: 1 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: checksum: 1 2022-04-22T17:15:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2022-04-22T17:15:29.883 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_cache: 0x56221b382d10 2022-04-22T17:15:29.883 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2022-04-22T17:15:29.883 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_cache_options: 2022-04-22T17:15:29.883 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: capacity : 536870912 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_size: 4096 2022-04-22T17:15:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: partition_filters: 0 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: verify_compression: 0 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2022-04-22T17:15:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: format_version: 2 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: block_align: 0 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:15:29.886 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:15:29.888 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:15:29.888 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.num_levels: 7 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:15:29.889 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:15:29.890 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:15:29.891 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:15:29.892 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:15:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:15:29.894 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:15:29.895 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:15:29.895 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:15:29.895 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:15:29.895 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:15:29.895 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:15:29.896 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.force_consistency_checks: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.897 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: DB pointer 0x56221b447800 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46d2a0c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46d2a0c700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:15:29.898 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:15:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:15:29.900 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:29.902 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: 2022-04-22T17:15:29.903 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.832+0000 7f46e9b83700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2022-04-22T17:15:29.904 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.836+0000 7f46e9b83700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2022-04-22T17:15:29.904 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-mon: debug 2022-04-22T17:15:29.836+0000 7f46e9b83700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2022-04-22T17:15:30.487 INFO:teuthology.orchestra.run.smithi115.stderr:create mon.a on 2022-04-22T17:15:30.515 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2022-04-22T17:15:30.708 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2022-04-22T17:15:30.943 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target -> /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target. 2022-04-22T17:15:30.944 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target -> /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target. 2022-04-22T17:15:31.345 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to reset failed state of unit ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service: Unit ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service not loaded. 2022-04-22T17:15:31.353 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target.wants/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service -> /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@.service. 2022-04-22T17:15:31.570 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld does not appear to be present 2022-04-22T17:15:31.570 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to enable service . firewalld.service is not available 2022-04-22T17:15:31.571 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mon to start... 2022-04-22T17:15:31.571 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mon... 2022-04-22T17:15:31.615 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:31 smithi115 systemd[1]: Started Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: cluster: 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: id: b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: health: HEALTH_OK 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: services: 2022-04-22T17:15:33.519 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.208088s) 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr: no daemons active 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: data: 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: pgs: 2022-04-22T17:15:33.520 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:33.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:33 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.310343+0000 mon.a (mon.0) 0 : [INF] mkfs b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:33.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:33 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.280844+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2022-04-22T17:15:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:mon is available 2022-04-22T17:15:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:Assimilating anything we can from ceph.conf... 2022-04-22T17:15:34.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.308390+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2022-04-22T17:15:34.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.308786+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-04-22T17:15:34.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.314193+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2022-04-22T17:15:34.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.318334+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2022-04-22T17:15:34.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: cluster 2022-04-22T17:15:33.318991+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2022-04-22T17:15:34.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:34 smithi115 bash[12216]: audit 2022-04-22T17:15:33.516400+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.115:0/2356133021' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-04-22T17:15:35.670 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:35.670 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [global] 2022-04-22T17:15:35.670 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: fsid = b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:35.670 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: log_to_journald = false 2022-04-22T17:15:35.672 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2022-04-22T17:15:35.672 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-04-22T17:15:35.672 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [mgr] 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [osd] 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-04-22T17:15:35.673 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-04-22T17:15:36.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:35 smithi115 bash[12216]: audit 2022-04-22T17:15:35.664691+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.115:0/2343942385' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-04-22T17:15:36.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:35 smithi115 bash[12216]: audit 2022-04-22T17:15:35.668789+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.115:0/2343942385' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-04-22T17:15:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:Generating new minimal ceph.conf... 2022-04-22T17:15:38.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:37 smithi115 bash[12216]: audit 2022-04-22T17:15:37.836535+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.115:0/2261515525' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:15:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:Restarting the monitor... 2022-04-22T17:15:38.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:38 smithi115 systemd[1]: Stopping Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:15:38.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:38 smithi115 bash[12625]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.a 2022-04-22T17:15:38.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:38 smithi115 bash[12216]: debug 2022-04-22T17:15:38.620+0000 7f3e0e2c7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-04-22T17:15:38.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:38 smithi115 bash[12216]: debug 2022-04-22T17:15:38.620+0000 7f3e0e2c7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-04-22T17:15:39.485 INFO:teuthology.orchestra.run.smithi115.stderr:Setting mon public_network to 172.21.0.0/20,172.21.15.254 2022-04-22T17:15:39.700 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:39 smithi115 bash[12639]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon-a 2022-04-22T17:15:39.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:39 smithi115 bash[12681]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.a 2022-04-22T17:15:39.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:39 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service: Succeeded. 2022-04-22T17:15:39.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:39 smithi115 systemd[1]: Stopped Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:15:39.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:39 smithi115 systemd[1]: Started Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:15:41.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.952+0000 7fb56f722700 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:15:41.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.952+0000 7fb56f722700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 8 2022-04-22T17:15:41.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.952+0000 7fb56f722700 0 pidfile_write: ignore empty --pid-file 2022-04-22T17:15:41.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 0 load: jerasure load: lrc load: isa 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: RocksDB version: 6.8.1 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Compile date Jul 8 2021 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: DB SUMMARY 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: CURRENT file: CURRENT 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: IDENTITY file: IDENTITY 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 86586 ; 2022-04-22T17:15:41.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.create_if_missing: 0 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.env: 0x55b5a9fa21c0 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.info_log: 0x55b5ac7d5e20 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:15:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:15:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:15:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:15:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:15:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:15:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.db_log_dir: 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:15:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.write_buffer_manager: 0x55b5ac86df20 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:15:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:15:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.row_cache: None 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.wal_filter: None 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:15:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Compression algorithms supported: 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:15:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kZSTD supported: 0 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:15:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.merge_operator: 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b5ac77c170) 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: cache_index_and_filter_blocks: 1 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: pin_top_level_index_and_filter: 1 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: index_type: 0 2022-04-22T17:15:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: data_block_index_type: 0 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: index_shortening: 1 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: hash_index_allow_collision: 1 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: checksum: 1 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: no_block_cache: 0 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_cache: 0x55b5ac7b4f10 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_cache_name: BinnedLRUCache 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_cache_options: 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: capacity : 536870912 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: num_shard_bits : 4 2022-04-22T17:15:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: strict_capacity_limit : 0 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: high_pri_pool_ratio: 0.000 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_cache_compressed: (nil) 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: persistent_cache: (nil) 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_size: 4096 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_size_deviation: 10 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_restart_interval: 16 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: index_block_restart_interval: 1 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: metadata_block_size: 4096 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: partition_filters: 0 2022-04-22T17:15:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: use_delta_encoding: 1 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: whole_key_filtering: 1 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: verify_compression: 0 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: read_amp_bytes_per_bit: 0 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: format_version: 2 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: enable_index_compression: 1 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: block_align: 0 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:15:41.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.num_levels: 7 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:15:41.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:15:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:15:41.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:15:41.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:15:41.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.force_consistency_checks: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:15:41.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.956+0000 7fb56f722700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647740963735, "job": 1, "event": "recovery_started", "log_files": [6]} 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647740965177, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 82946, "table_properties": {"data_size": 81215, "index_size": 336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9730, "raw_average_key_size": 47, "raw_value_size": 75628, "raw_average_value_size": 368, "num_data_blocks": 11, "num_entries": 205, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1650647740, "oldest_key_time": 3, "file_creation_time": 0}} 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647740965859, "job": 1, "event": "recovery_finished"} 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb56f722700 4 rocksdb: DB pointer 0x55b5ac879800 2022-04-22T17:15:41.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb55652a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.960+0000 7fb55652a700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** DB Stats ** 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:15:41.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** Compaction Stats [default] ** 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: L0 2/0 82.66 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Sum 2/0 82.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** Compaction Stats [default] ** 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Cumulative compaction: 0.00 GB write, 18.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Interval compaction: 0.00 GB write, 18.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** Compaction Stats [default] ** 2022-04-22T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: L0 2/0 82.66 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Sum 2/0 82.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** Compaction Stats [default] ** 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 82.3 0.00 0.00 1 0.001 0 0 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:41.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Cumulative compaction: 0.00 GB write, 18.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 1 mon.a@-1(???) e1 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).mds e1 new map 2022-04-22T17:15:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).mds e1 print_map 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: e1 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: legacy client fscid: -1 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: No filesystems configured 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:15:41.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:15:41.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: audit 2022-04-22T17:15:37.836535+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.115:0/2261515525' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:15:41.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-04-22T17:15:41.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expand map: {default=false} 2022-04-22T17:15:41.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta from 'false' to 'false' 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expanded map: {default=false} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expand map: {default=info} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta from 'info' to 'info' 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expanded map: {default=info} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expand map: {default=daemon} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta from 'daemon' to 'daemon' 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expanded map: {default=daemon} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expand map: {default=debug} 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta from 'debug' to 'debug' 2022-04-22T17:15:41.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:40 smithi115 bash[12737]: debug 2022-04-22T17:15:40.964+0000 7fb56f722700 20 expand_channel_meta expanded map: {default=debug} 2022-04-22T17:15:41.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:41 smithi115 bash[12737]: cluster 2022-04-22T17:15:40.992428+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2022-04-22T17:15:41.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:41 smithi115 bash[12737]: cluster 2022-04-22T17:15:40.992550+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-04-22T17:15:41.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:41 smithi115 bash[12737]: cluster 2022-04-22T17:15:40.992640+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2022-04-22T17:15:41.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:41 smithi115 bash[12737]: cluster 2022-04-22T17:15:40.992707+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2022-04-22T17:15:41.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:41 smithi115 bash[12737]: cluster 2022-04-22T17:15:40.993504+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2022-04-22T17:15:41.882 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote config to /etc/ceph/ceph.conf 2022-04-22T17:15:41.882 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-04-22T17:15:41.882 INFO:teuthology.orchestra.run.smithi115.stderr:Creating mgr... 2022-04-22T17:15:41.883 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying port 9283 ... 2022-04-22T17:15:42.118 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to reset failed state of unit ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service: Unit ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service not loaded. 2022-04-22T17:15:42.125 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c.target.wants/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service -> /etc/systemd/system/ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@.service. 2022-04-22T17:15:42.326 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld does not appear to be present 2022-04-22T17:15:42.326 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to enable service . firewalld.service is not available 2022-04-22T17:15:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld does not appear to be present 2022-04-22T17:15:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2022-04-22T17:15:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr to start... 2022-04-22T17:15:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr... 2022-04-22T17:15:42.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:42 smithi115 bash[12737]: audit 2022-04-22T17:15:41.267731+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.115:0/3045407829' entity='client.admin' 2022-04-22T17:15:44.082 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:44.082 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:15:44.082 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "b6061a0a-c25f-11ec-8c39-001a4aab830c", 2022-04-22T17:15:44.082 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2022-04-22T17:15:44.083 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 3, 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:44.084 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2022-04-22T17:15:44.085 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-04-22T17:15:44.085 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-04-22T17:15:44.086 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-04-22T17:15:44.086 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-04-22T17:15:44.086 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-04-22T17:15:44.086 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.086 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2022-04-22T17:15:44.087 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-04-22T17:15:44.087 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-04-22T17:15:44.087 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2022-04-22T17:15:44.087 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2022-04-22T17:15:44.087 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2022-04-22T17:15:44.090 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": false, 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:44.091 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2022-04-22T17:15:33.311631+0000", 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2022-04-22T17:15:44.092 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:15:44.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:44 smithi115 bash[12737]: audit 2022-04-22T17:15:44.078402+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.115:0/1936121973' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-04-22T17:15:44.747 INFO:teuthology.orchestra.run.smithi115.stderr:mgr not available, waiting (1/15)... 2022-04-22T17:15:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:15:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "b6061a0a-c25f-11ec-8c39-001a4aab830c", 2022-04-22T17:15:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2022-04-22T17:15:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-04-22T17:15:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2022-04-22T17:15:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2022-04-22T17:15:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-04-22T17:15:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2022-04-22T17:15:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2022-04-22T17:15:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2022-04-22T17:15:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 6, 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2022-04-22T17:15:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-04-22T17:15:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-04-22T17:15:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2022-04-22T17:15:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2022-04-22T17:15:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-04-22T17:15:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-04-22T17:15:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-04-22T17:15:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2022-04-22T17:15:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2022-04-22T17:15:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2022-04-22T17:15:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": false, 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2022-04-22T17:15:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2022-04-22T17:15:33.311631+0000", 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2022-04-22T17:15:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:15:48.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:48 smithi115 bash[12737]: audit 2022-04-22T17:15:47.976825+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.115:0/1871015230' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-04-22T17:15:48.620 INFO:teuthology.orchestra.run.smithi115.stderr:mgr not available, waiting (2/15)... 2022-04-22T17:15:50.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: cluster 2022-04-22T17:15:50.133723+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2022-04-22T17:15:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: cluster 2022-04-22T17:15:50.139678+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00618137s) 2022-04-22T17:15:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: audit 2022-04-22T17:15:50.142143+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:15:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: audit 2022-04-22T17:15:50.142649+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:15:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: audit 2022-04-22T17:15:50.142982+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:15:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: audit 2022-04-22T17:15:50.143330+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:15:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: audit 2022-04-22T17:15:50.143758+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:15:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:50 smithi115 bash[12737]: cluster 2022-04-22T17:15:50.171633+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2022-04-22T17:15:51.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: audit 2022-04-22T17:15:50.185327+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' 2022-04-22T17:15:51.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: audit 2022-04-22T17:15:50.194846+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:15:51.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: audit 2022-04-22T17:15:50.196569+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:15:51.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: audit 2022-04-22T17:15:50.203735+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' 2022-04-22T17:15:51.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: audit 2022-04-22T17:15:50.206805+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.115:0/1672037849' entity='mgr.y' 2022-04-22T17:15:51.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:51 smithi115 bash[12737]: cluster 2022-04-22T17:15:51.144994+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.0115s) 2022-04-22T17:15:52.259 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "b6061a0a-c25f-11ec-8c39-001a4aab830c", 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-04-22T17:15:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 11, 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-04-22T17:15:52.261 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.262 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2022-04-22T17:15:52.263 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2022-04-22T17:15:52.264 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2022-04-22T17:15:33.311631+0000", 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2022-04-22T17:15:52.265 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:15:52.893 INFO:teuthology.orchestra.run.smithi115.stderr:mgr is available 2022-04-22T17:15:53.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:53 smithi115 bash[12737]: cluster 2022-04-22T17:15:52.193396+0000 mon.a (mon.0) 23 : cluster [DBG] mgrmap e4: y(active, since 2s) 2022-04-22T17:15:53.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:53 smithi115 bash[12737]: audit 2022-04-22T17:15:52.246733+0000 mon.a (mon.0) 24 : audit [DBG] from='client.? 172.21.15.115:0/2450626790' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [global] 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: fsid = b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: log_to_journald = false 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-04-22T17:15:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [mgr] 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [osd] 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-04-22T17:15:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-04-22T17:15:54.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:54 smithi115 bash[12737]: audit 2022-04-22T17:15:54.352680+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.115:0/1931128209' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-04-22T17:15:54.982 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling cephadm module... 2022-04-22T17:15:57.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:56 smithi115 bash[12737]: audit 2022-04-22T17:15:56.563017+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.115:0/208042771' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-04-22T17:15:57.338 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:15:57 smithi115 bash[13018]: ignoring --setgroup ceph since I am not root 2022-04-22T17:15:58.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:57 smithi115 bash[12737]: audit 2022-04-22T17:15:56.980553+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.115:0/208042771' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-04-22T17:15:58.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:57 smithi115 bash[12737]: cluster 2022-04-22T17:15:56.980625+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 6s) 2022-04-22T17:15:59.147 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:15:59.152 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 5, 2022-04-22T17:15:59.152 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2022-04-22T17:15:59.152 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "active_name": "y", 2022-04-22T17:15:59.152 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standby": 0 2022-04-22T17:15:59.152 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:15:59.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:15:59 smithi115 bash[12737]: audit 2022-04-22T17:15:59.146266+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.115:0/2485814698' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-04-22T17:15:59.778 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for the mgr to restart... 2022-04-22T17:15:59.778 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr epoch 5... 2022-04-22T17:16:04.290 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:16:04.290 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2022-04-22T17:16:04.290 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "initialized": true 2022-04-22T17:16:04.290 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:16:04.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: cluster 2022-04-22T17:16:03.213514+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2022-04-22T17:16:04.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: cluster 2022-04-22T17:16:03.214664+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2022-04-22T17:16:04.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: cluster 2022-04-22T17:16:03.218405+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2022-04-22T17:16:04.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: cluster 2022-04-22T17:16:03.273172+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0588056s) 2022-04-22T17:16:04.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.275534+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.277194+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.278380+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.278756+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.279072+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: cluster 2022-04-22T17:16:03.303007+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.329199+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.333206+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.336947+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.467505+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:04.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.468810+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:04.501 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.472423+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:04.501 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.475152+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:04.501 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.494216+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:16:04.501 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:04 smithi115 bash[12737]: audit 2022-04-22T17:16:03.496143+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:16:04.941 INFO:teuthology.orchestra.run.smithi115.stderr:mgr epoch 5 is available 2022-04-22T17:16:04.942 INFO:teuthology.orchestra.run.smithi115.stderr:Setting orchestrator backend to cephadm... 2022-04-22T17:16:05.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:05 smithi115 bash[12737]: cluster 2022-04-22T17:16:04.273986+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.05963s) 2022-04-22T17:16:05.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:05 smithi115 bash[12737]: audit 2022-04-22T17:16:04.275077+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-04-22T17:16:05.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:05 smithi115 bash[12737]: audit 2022-04-22T17:16:04.288956+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-04-22T17:16:06.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:06 smithi115 bash[12737]: cluster 2022-04-22T17:16:05.278301+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2022-04-22T17:16:08.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:07 smithi115 bash[12737]: audit 2022-04-22T17:16:06.550722+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:08.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:07 smithi115 bash[12737]: audit 2022-04-22T17:16:06.560923+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:08.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:07 smithi115 bash[12737]: audit 2022-04-22T17:16:06.566642+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:08.633 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: value unchanged 2022-04-22T17:16:09.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:09 smithi115 bash[12737]: audit 2022-04-22T17:16:08.631780+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:09.304 INFO:teuthology.orchestra.run.smithi115.stderr:Generating ssh key... 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: Generating public/private rsa key pair. 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: Your identification has been saved in /tmp/tmpbep8kf2s/key. 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: Your public key has been saved in /tmp/tmpbep8kf2s/key.pub. 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: The key fingerprint is: 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: SHA256:1PUiE50SD4V9ZyjxjHJvvyPjP7FJzt0SElwGrgVPmEI ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: The key's randomart image is: 2022-04-22T17:16:11.131 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: +---[RSA 3072]----+ 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | .E =O*+ . | 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | ..=X=== o| 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | ..=oO=+o | 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | . Boo | 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | S . .o | 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | ...+ | 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | .+.B| 2022-04-22T17:16:11.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | o.B+| 2022-04-22T17:16:11.133 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: | ..++o| 2022-04-22T17:16:11.133 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:10 smithi115 bash[13018]: +----[SHA256]-----+ 2022-04-22T17:16:12.036 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:11 smithi115 bash[12737]: cephadm 2022-04-22T17:16:10.796027+0000 mgr.y (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2022-04-22T17:16:12.036 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:11 smithi115 bash[12737]: audit 2022-04-22T17:16:10.885686+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:12.036 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:11 smithi115 bash[12737]: audit 2022-04-22T17:16:10.890213+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:13.101 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQ4wYO+RP3BglwqtmEBGzUbOe3Fif5NXEUiGRu1QbQWeuT0dgorbws7DVnFzcc1vT3sK+lTQEm7vb9se0C20w/V1LmVvdbQEck9MSrwXeVNukAesCDWP3NXj96CKYfwFjv1GULZswQq8XhVY20QaO90urpUWiYK5agj2KxVF2ECnBPvxLCpUpK62/j0PtXfYCV9ZiACsbn8vUV2s5uLU7HaoGQSl6gHu4pxzNUGCL69x/ikKpcDJgOHgTdKKs4wPityWtyh/8QO5rCga3/ScLCRfTix+ljlJ1Sdf73KHwAJgnnRNjB2DoiZjVBJitJauC5CtUW923hU/ycZMZv1I02nws5DQf42F6Jk5YZFa+4+ltO/fCrNLp74O5lAiPSfgP/jh1C/OQo41kC0h1rYhk+ZLMz32yNnTocpDUr/7WS8Ak3uGlWFok1qf8F2vcgPt7rpD+VRaB4Zx+q/nAXsGzWUAVJAt0+ctKOTFj5cnanZ+s+q3ACo9LX7wuJv7feBE8= ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:16:13.777 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-04-22T17:16:13.777 INFO:teuthology.orchestra.run.smithi115.stderr:Adding key to root@localhost authorized_keys... 2022-04-22T17:16:13.778 INFO:teuthology.orchestra.run.smithi115.stderr:Adding host smithi115... 2022-04-22T17:16:14.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:14 smithi115 bash[12737]: audit 2022-04-22T17:16:13.102161+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:15.784 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:15 smithi115 bash[13018]: Warning: Permanently added '172.21.15.115' (ECDSA) to the list of known hosts. 2022-04-22T17:16:16.074 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Added host 'smithi115' with addr '172.21.15.115' 2022-04-22T17:16:17.006 INFO:teuthology.orchestra.run.smithi115.stderr:Deploying unmanaged mon service... 2022-04-22T17:16:17.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:16 smithi115 bash[12737]: audit 2022-04-22T17:16:15.315287+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "addr": "172.21.15.115", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:17.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:16 smithi115 bash[12737]: audit 2022-04-22T17:16:16.076358+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:17.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:16 smithi115 bash[12737]: audit 2022-04-22T17:16:16.077589+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:18.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:17 smithi115 bash[12737]: cephadm 2022-04-22T17:16:16.077096+0000 mgr.y (mgr.14116) 9 : cephadm [INF] Added host smithi115 2022-04-22T17:16:19.255 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Scheduled mon update... 2022-04-22T17:16:20.138 INFO:teuthology.orchestra.run.smithi115.stderr:Deploying unmanaged mgr service... 2022-04-22T17:16:20.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:20 smithi115 bash[12737]: audit 2022-04-22T17:16:19.254984+0000 mgr.y (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:20.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:20 smithi115 bash[12737]: cephadm 2022-04-22T17:16:19.256775+0000 mgr.y (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2022-04-22T17:16:20.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:20 smithi115 bash[12737]: audit 2022-04-22T17:16:19.259794+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:20.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:20 smithi115 bash[12737]: audit 2022-04-22T17:16:19.881343+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:20.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:20 smithi115 bash[12737]: audit 2022-04-22T17:16:20.123915+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:21.719 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Scheduled mgr update... 2022-04-22T17:16:22.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:22 smithi115 bash[12737]: audit 2022-04-22T17:16:21.718024+0000 mgr.y (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:22.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:22 smithi115 bash[12737]: cephadm 2022-04-22T17:16:21.719005+0000 mgr.y (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2022-04-22T17:16:22.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:22 smithi115 bash[12737]: audit 2022-04-22T17:16:21.724033+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:24.402 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:24 smithi115 bash[12737]: audit 2022-04-22T17:16:23.129860+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.115:0/3906857093' entity='mgr.y' 2022-04-22T17:16:24.402 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:24 smithi115 bash[12737]: cluster 2022-04-22T17:16:23.280004+0000 mgr.y (mgr.14116) 14 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:16:25.281 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:25 smithi115 bash[12737]: audit 2022-04-22T17:16:24.140480+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.115:0/3689306922' entity='client.admin' 2022-04-22T17:16:27.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:26 smithi115 bash[12737]: cluster 2022-04-22T17:16:25.280521+0000 mgr.y (mgr.14116) 15 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:16:27.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:26 smithi115 bash[12737]: audit 2022-04-22T17:16:26.851374+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.115:0/3102911537' entity='client.admin' 2022-04-22T17:16:27.557 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling the dashboard module... 2022-04-22T17:16:29.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:28 smithi115 bash[12737]: cluster 2022-04-22T17:16:27.280978+0000 mgr.y (mgr.14116) 16 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:16:30.281 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:30 smithi115 bash[13018]: ignoring --setuser ceph since I am not root 2022-04-22T17:16:30.281 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:30 smithi115 bash[13018]: ignoring --setgroup ceph since I am not root 2022-04-22T17:16:30.281 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:29 smithi115 bash[12737]: audit 2022-04-22T17:16:29.165298+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.115:0/2787429926' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-04-22T17:16:31.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:30 smithi115 bash[12737]: audit 2022-04-22T17:16:29.995976+0000 mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.115:0/2787429926' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-04-22T17:16:31.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:30 smithi115 bash[12737]: cluster 2022-04-22T17:16:29.996048+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 26s) 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 9, 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "active_name": "y", 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standby": 0 2022-04-22T17:16:32.280 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:16:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:32 smithi115 bash[12737]: audit 2022-04-22T17:16:32.285550+0000 mon.a (mon.0) 67 : audit [DBG] from='client.? 172.21.15.115:0/3767763197' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-04-22T17:16:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for the mgr to restart... 2022-04-22T17:16:32.991 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr epoch 9... 2022-04-22T17:16:37.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:36 smithi115 bash[12737]: cluster 2022-04-22T17:16:36.269691+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2022-04-22T17:16:37.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:36 smithi115 bash[12737]: cluster 2022-04-22T17:16:36.270658+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2022-04-22T17:16:37.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:36 smithi115 bash[12737]: cluster 2022-04-22T17:16:36.274384+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2022-04-22T17:16:37.530 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:16:37 smithi115 bash[13018]: Warning: Permanently added '172.21.15.115' (ECDSA) to the list of known hosts. 2022-04-22T17:16:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2022-04-22T17:16:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2022-04-22T17:16:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "initialized": true 2022-04-22T17:16:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2022-04-22T17:16:38.249 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: cluster 2022-04-22T17:16:36.996163+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.725686s) 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:36.998193+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:36.999447+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.001172+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.001518+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.001802+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: cluster 2022-04-22T17:16:37.023097+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.174356+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:38.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.177952+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:16:38.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.214224+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:16:38.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:38 smithi115 bash[12737]: audit 2022-04-22T17:16:37.217342+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:16:38.882 INFO:teuthology.orchestra.run.smithi115.stderr:mgr epoch 9 is available 2022-04-22T17:16:38.882 INFO:teuthology.orchestra.run.smithi115.stderr:Generating a dashboard self-signed certificate... 2022-04-22T17:16:39.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:39 smithi115 bash[12737]: audit 2022-04-22T17:16:37.999226+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-04-22T17:16:39.281 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:39 smithi115 bash[12737]: cluster 2022-04-22T17:16:37.999965+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.72947s) 2022-04-22T17:16:39.281 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:39 smithi115 bash[12737]: audit 2022-04-22T17:16:38.014374+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-04-22T17:16:40.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:40 smithi115 bash[12737]: cluster 2022-04-22T17:16:39.009491+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e12: y(active, since 2s) 2022-04-22T17:16:40.499 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Self-signed certificate created 2022-04-22T17:16:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial admin user... 2022-04-22T17:16:41.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:41 smithi115 bash[12737]: audit 2022-04-22T17:16:40.416071+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:41.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:41 smithi115 bash[12737]: audit 2022-04-22T17:16:40.502484+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:16:41.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:41 smithi115 bash[12737]: audit 2022-04-22T17:16:40.506129+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:16:41.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:41 smithi115 bash[12737]: audit 2022-04-22T17:16:40.542395+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:16:43.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:43 smithi115 bash[12737]: cluster 2022-04-22T17:16:42.504138+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e13: y(active, since 6s) 2022-04-22T17:16:43.865 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$BViagbPsCZcroWk2O.0F.uAuWSTnXwPz3dXlvKuTV5/iYSdZJIhPO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1650647803, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2022-04-22T17:16:44.617 INFO:teuthology.orchestra.run.smithi115.stderr:Fetching dashboard port number... 2022-04-22T17:16:45.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:44 smithi115 bash[12737]: audit 2022-04-22T17:16:43.633362+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:16:45.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:44 smithi115 bash[12737]: audit 2022-04-22T17:16:43.871122+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:16:46.124 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 8443 2022-04-22T17:16:46.858 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld does not appear to be present 2022-04-22T17:16:46.858 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2022-04-22T17:16:46.861 INFO:teuthology.orchestra.run.smithi115.stderr:Ceph Dashboard is now available at: 2022-04-22T17:16:46.861 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:46.861 INFO:teuthology.orchestra.run.smithi115.stderr: URL: https://smithi115.front.sepia.ceph.com:8443/ 2022-04-22T17:16:46.861 INFO:teuthology.orchestra.run.smithi115.stderr: User: admin 2022-04-22T17:16:46.861 INFO:teuthology.orchestra.run.smithi115.stderr: Password: k3mw9nmcxc 2022-04-22T17:16:46.862 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:46.862 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling autotune for osd_memory_target 2022-04-22T17:16:47.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:46 smithi115 bash[12737]: audit 2022-04-22T17:16:46.130352+0000 mon.a (mon.0) 89 : audit [DBG] from='client.? 172.21.15.115:0/1126950427' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-04-22T17:16:50.736 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:Or, if you are only running a single cluster on this host: 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.479 INFO:teuthology.orchestra.run.smithi115.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr:Please consider enabling telemetry to help improve Ceph: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: ceph telemetry on 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr:For more information see: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2022-04-22T17:16:51.480 INFO:teuthology.orchestra.run.smithi115.stderr:Bootstrap complete. 2022-04-22T17:16:51.516 INFO:tasks.cephadm:Fetching config... 2022-04-22T17:16:51.517 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:16:51.517 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-04-22T17:16:51.522 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-04-22T17:16:51.522 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:16:51.523 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-04-22T17:16:51.528 INFO:tasks.cephadm:Fetching mon keyring... 2022-04-22T17:16:51.528 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:16:51.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-04-22T17:16:51.543 INFO:tasks.cephadm:Fetching pub ssh key... 2022-04-22T17:16:51.543 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:16:51.543 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-04-22T17:16:51.548 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-04-22T17:16:51.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQ4wYO+RP3BglwqtmEBGzUbOe3Fif5NXEUiGRu1QbQWeuT0dgorbws7DVnFzcc1vT3sK+lTQEm7vb9se0C20w/V1LmVvdbQEck9MSrwXeVNukAesCDWP3NXj96CKYfwFjv1GULZswQq8XhVY20QaO90urpUWiYK5agj2KxVF2ECnBPvxLCpUpK62/j0PtXfYCV9ZiACsbn8vUV2s5uLU7HaoGQSl6gHu4pxzNUGCL69x/ikKpcDJgOHgTdKKs4wPityWtyh/8QO5rCga3/ScLCRfTix+ljlJ1Sdf73KHwAJgnnRNjB2DoiZjVBJitJauC5CtUW923hU/ycZMZv1I02nws5DQf42F6Jk5YZFa+4+ltO/fCrNLp74O5lAiPSfgP/jh1C/OQo41kC0h1rYhk+ZLMz32yNnTocpDUr/7WS8Ak3uGlWFok1qf8F2vcgPt7rpD+VRaB4Zx+q/nAXsGzWUAVJAt0+ctKOTFj5cnanZ+s+q3ACo9LX7wuJv7feBE8= ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-04-22T17:16:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQ4wYO+RP3BglwqtmEBGzUbOe3Fif5NXEUiGRu1QbQWeuT0dgorbws7DVnFzcc1vT3sK+lTQEm7vb9se0C20w/V1LmVvdbQEck9MSrwXeVNukAesCDWP3NXj96CKYfwFjv1GULZswQq8XhVY20QaO90urpUWiYK5agj2KxVF2ECnBPvxLCpUpK62/j0PtXfYCV9ZiACsbn8vUV2s5uLU7HaoGQSl6gHu4pxzNUGCL69x/ikKpcDJgOHgTdKKs4wPityWtyh/8QO5rCga3/ScLCRfTix+ljlJ1Sdf73KHwAJgnnRNjB2DoiZjVBJitJauC5CtUW923hU/ycZMZv1I02nws5DQf42F6Jk5YZFa+4+ltO/fCrNLp74O5lAiPSfgP/jh1C/OQo41kC0h1rYhk+ZLMz32yNnTocpDUr/7WS8Ak3uGlWFok1qf8F2vcgPt7rpD+VRaB4Zx+q/nAXsGzWUAVJAt0+ctKOTFj5cnanZ+s+q3ACo9LX7wuJv7feBE8= ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:16:51.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQ4wYO+RP3BglwqtmEBGzUbOe3Fif5NXEUiGRu1QbQWeuT0dgorbws7DVnFzcc1vT3sK+lTQEm7vb9se0C20w/V1LmVvdbQEck9MSrwXeVNukAesCDWP3NXj96CKYfwFjv1GULZswQq8XhVY20QaO90urpUWiYK5agj2KxVF2ECnBPvxLCpUpK62/j0PtXfYCV9ZiACsbn8vUV2s5uLU7HaoGQSl6gHu4pxzNUGCL69x/ikKpcDJgOHgTdKKs4wPityWtyh/8QO5rCga3/ScLCRfTix+ljlJ1Sdf73KHwAJgnnRNjB2DoiZjVBJitJauC5CtUW923hU/ycZMZv1I02nws5DQf42F6Jk5YZFa+4+ltO/fCrNLp74O5lAiPSfgP/jh1C/OQo41kC0h1rYhk+ZLMz32yNnTocpDUr/7WS8Ak3uGlWFok1qf8F2vcgPt7rpD+VRaB4Zx+q/nAXsGzWUAVJAt0+ctKOTFj5cnanZ+s+q3ACo9LX7wuJv7feBE8= ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-04-22T17:16:51.602 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQ4wYO+RP3BglwqtmEBGzUbOe3Fif5NXEUiGRu1QbQWeuT0dgorbws7DVnFzcc1vT3sK+lTQEm7vb9se0C20w/V1LmVvdbQEck9MSrwXeVNukAesCDWP3NXj96CKYfwFjv1GULZswQq8XhVY20QaO90urpUWiYK5agj2KxVF2ECnBPvxLCpUpK62/j0PtXfYCV9ZiACsbn8vUV2s5uLU7HaoGQSl6gHu4pxzNUGCL69x/ikKpcDJgOHgTdKKs4wPityWtyh/8QO5rCga3/ScLCRfTix+ljlJ1Sdf73KHwAJgnnRNjB2DoiZjVBJitJauC5CtUW923hU/ycZMZv1I02nws5DQf42F6Jk5YZFa+4+ltO/fCrNLp74O5lAiPSfgP/jh1C/OQo41kC0h1rYhk+ZLMz32yNnTocpDUr/7WS8Ak3uGlWFok1qf8F2vcgPt7rpD+VRaB4Zx+q/nAXsGzWUAVJAt0+ctKOTFj5cnanZ+s+q3ACo9LX7wuJv7feBE8= ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:16:51.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-04-22T17:16:51.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:51 smithi115 bash[12737]: audit 2022-04-22T17:16:50.737262+0000 mon.a (mon.0) 90 : audit [INF] from='client.? 172.21.15.115:0/3068108012' entity='client.admin' 2022-04-22T17:16:57.468 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-04-22T17:16:57.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-04-22T17:16:57.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:57 smithi115 bash[12737]: audit 2022-04-22T17:16:56.400367+0000 mon.a (mon.0) 91 : audit [INF] from='client.? 172.21.15.115:0/3139336869' entity='client.admin' 2022-04-22T17:16:57.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:57 smithi115 bash[12737]: cluster 2022-04-22T17:16:57.003000+0000 mgr.y (mgr.14146) 5 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:16:59.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:16:59 smithi115 bash[12737]: cluster 2022-04-22T17:16:59.003371+0000 mgr.y (mgr.14146) 6 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:01.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:00.543206+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:01.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:00.836122+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:17:01.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:00.842288+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:01.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:00.844984+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:01.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:00.850276+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:01.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: cluster 2022-04-22T17:17:01.003786+0000 mgr.y (mgr.14146) 7 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.473933+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.475090+0000 mon.a (mon.0) 98 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.477649+0000 mon.a (mon.0) 99 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.479381+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.536074+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:01 smithi115 bash[12737]: audit 2022-04-22T17:17:01.539761+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:02.304 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi121 2022-04-22T17:17:02.304 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:17:02.304 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.conf 2022-04-22T17:17:02.312 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:17:02.312 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:17:02.362 INFO:tasks.cephadm:Adding host smithi121 to orchestrator... 2022-04-22T17:17:02.362 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch host add smithi121 2022-04-22T17:17:03.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:02 smithi115 bash[12737]: cephadm 2022-04-22T17:17:01.482139+0000 mgr.y (mgr.14146) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:03.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:02 smithi115 bash[12737]: cephadm 2022-04-22T17:17:01.507143+0000 mgr.y (mgr.14146) 10 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:17:04.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:03 smithi115 bash[12737]: cluster 2022-04-22T17:17:03.004139+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:05.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:05 smithi115 bash[12737]: cluster 2022-04-22T17:17:05.004540+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:06.280 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:17:05 smithi115 bash[13018]: Warning: Permanently added 'smithi121,172.21.15.121' (ECDSA) to the list of known hosts. 2022-04-22T17:17:06.709 INFO:teuthology.orchestra.run.smithi115.stdout:Added host 'smithi121' with addr '172.21.15.121' 2022-04-22T17:17:07.030 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[13018]: Warning: Permanently added '172.21.15.121' (ECDSA) to the list of known hosts. 2022-04-22T17:17:07.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[12737]: audit 2022-04-22T17:17:05.921427+0000 mgr.y (mgr.14146) 13 : audit [DBG] from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:17:07.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[12737]: audit 2022-04-22T17:17:06.715086+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:07.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[12737]: audit 2022-04-22T17:17:06.716520+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:07.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[12737]: audit 2022-04-22T17:17:06.720093+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:07.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:06 smithi115 bash[12737]: audit 2022-04-22T17:17:06.722428+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:07.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch host ls --format=json 2022-04-22T17:17:08.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:07 smithi115 bash[12737]: cephadm 2022-04-22T17:17:06.715849+0000 mgr.y (mgr.14146) 14 : cephadm [INF] Added host smithi121 2022-04-22T17:17:08.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:07 smithi115 bash[12737]: cluster 2022-04-22T17:17:07.004972+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:08.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:07 smithi115 bash[12737]: audit 2022-04-22T17:17:07.212702+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:08.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:07 smithi115 bash[12737]: audit 2022-04-22T17:17:07.953042+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:09.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:09 smithi115 bash[12737]: cluster 2022-04-22T17:17:09.005440+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:10.995 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:17:10.996 INFO:teuthology.orchestra.run.smithi115.stdout:[{"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}] 2022-04-22T17:17:11.737 INFO:tasks.cephadm:Setting crush tunables to default 2022-04-22T17:17:11.738 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd crush tunables default 2022-04-22T17:17:12.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:11 smithi115 bash[12737]: audit 2022-04-22T17:17:11.002390+0000 mgr.y (mgr.14146) 17 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:17:12.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:11 smithi115 bash[12737]: cluster 2022-04-22T17:17:11.005792+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:13.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:13 smithi115 bash[12737]: cluster 2022-04-22T17:17:13.006131+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:14.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:14 smithi115 bash[12737]: audit 2022-04-22T17:17:13.224838+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:14.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:14 smithi115 bash[12737]: audit 2022-04-22T17:17:13.528885+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:14.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:14 smithi115 bash[12737]: audit 2022-04-22T17:17:13.529934+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:17:14.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:14 smithi115 bash[12737]: audit 2022-04-22T17:17:13.535674+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:15.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:15 smithi115 bash[12737]: cluster 2022-04-22T17:17:15.006560+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:16.236 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2022-04-22T17:17:16.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:16 smithi115 bash[12737]: audit 2022-04-22T17:17:15.283719+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.115:0/445345803' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-04-22T17:17:17.022 INFO:tasks.cephadm:Adding mon.a on smithi115 2022-04-22T17:17:17.022 INFO:tasks.cephadm:Adding mon.c on smithi115 2022-04-22T17:17:17.022 INFO:tasks.cephadm:Adding mon.b on smithi121 2022-04-22T17:17:17.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply mon '3;smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi121:172.21.15.121=b' 2022-04-22T17:17:17.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:17 smithi115 bash[12737]: audit 2022-04-22T17:17:16.239239+0000 mon.a (mon.0) 114 : audit [INF] from='client.? 172.21.15.115:0/445345803' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-04-22T17:17:17.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:17 smithi115 bash[12737]: cluster 2022-04-22T17:17:16.239320+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:17.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:17 smithi115 bash[12737]: cluster 2022-04-22T17:17:17.006951+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:19.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:19 smithi115 bash[12737]: cluster 2022-04-22T17:17:19.007379+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:20.275 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled mon update... 2022-04-22T17:17:20.977 DEBUG:teuthology.orchestra.run.smithi115:mon.c> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.c.service 2022-04-22T17:17:20.980 DEBUG:teuthology.orchestra.run.smithi121:mon.b> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.b.service 2022-04-22T17:17:20.982 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-04-22T17:17:20.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph mon dump -f json 2022-04-22T17:17:21.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:17:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:17:21.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.274347+0000 mgr.y (mgr.14146) 23 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi121:172.21.15.121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:17:21.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: cephadm 2022-04-22T17:17:20.277749+0000 mgr.y (mgr.14146) 24 : cephadm [INF] Saving service mon spec with placement smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi121:172.21.15.121=b;count:3 2022-04-22T17:17:21.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.282306+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:21.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.283663+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:21.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.286103+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.287718+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: cephadm 2022-04-22T17:17:20.289633+0000 mgr.y (mgr.14146) 25 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: cephadm 2022-04-22T17:17:20.316743+0000 mgr.y (mgr.14146) 26 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.346649+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.351519+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.353836+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: audit 2022-04-22T17:17:20.355262+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: cephadm 2022-04-22T17:17:20.356710+0000 mgr.y (mgr.14146) 27 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:17:21.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:21 smithi115 bash[12737]: cluster 2022-04-22T17:17:21.007802+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:23.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:23 smithi115 bash[12737]: cluster 2022-04-22T17:17:23.008258+0000 mgr.y (mgr.14146) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-04-22T17:17:24.293 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":1,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","modified":"2022-04-22T17:15:27.986542Z","created":"2022-04-22T17:15:27.986542Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-04-22T17:17:24.296 INFO:teuthology.orchestra.run.smithi121.stderr:dumped monmap epoch 1 2022-04-22T17:17:24.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:24 smithi115 bash[12737]: audit 2022-04-22T17:17:24.300569+0000 mon.a (mon.0) 124 : audit [DBG] from='client.? 172.21.15.121:0/2205314531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:25.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:25 smithi115 bash[12737]: cluster 2022-04-22T17:17:25.008635+0000 mgr.y (mgr.14146) 30 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:25.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:25 smithi115 bash[12737]: audit 2022-04-22T17:17:25.096618+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:25.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:25 smithi115 bash[12737]: audit 2022-04-22T17:17:25.097867+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:25.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:25 smithi115 bash[12737]: audit 2022-04-22T17:17:25.098835+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:25.924 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-04-22T17:17:25.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph mon dump -f json 2022-04-22T17:17:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:26.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2022-04-22T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Cumulative compaction: 0.00 GB write, 0.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Interval compaction: 0.00 GB write, 0.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ** Compaction Stats [default] ** 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2022-04-22T17:17:26.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ** Compaction Stats [default] ** 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:17:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Cumulative compaction: 0.00 GB write, 0.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.290+0000 7f1c0f938700 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.290+0000 7f1c0f938700 0 using public_addrv [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.294+0000 7f1c0f938700 0 starting mon.c rank -1 at public addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] at bind addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.294+0000 7f1c0f938700 1 mon.c@-1(???) e0 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).mds e1 new map 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).mds e1 print_map 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: e1 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: legacy client fscid: -1 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: No filesystems configured 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-04-22T17:17:26.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: cluster 2022-04-22T17:17:25.008635+0000 mgr.y (mgr.14146) 30 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: audit 2022-04-22T17:17:25.096618+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: audit 2022-04-22T17:17:25.097867+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: audit 2022-04-22T17:17:25.098835+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2022-04-22T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expand map: {default=false} 2022-04-22T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta from 'false' to 'false' 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expanded map: {default=false} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expand map: {default=info} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta from 'info' to 'info' 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expanded map: {default=info} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expand map: {default=daemon} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta from 'daemon' to 'daemon' 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expanded map: {default=daemon} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expand map: {default=debug} 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta from 'debug' to 'debug' 2022-04-22T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:26 smithi115 bash[18070]: debug 2022-04-22T17:17:26.318+0000 7f1bf8f45700 20 expand_channel_meta expanded map: {default=debug} 2022-04-22T17:17:31.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:26.329574+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:17:31.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:26.329949+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:26.331512+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:27.009063+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:27.326049+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:28.325961+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:28.328288+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:29.009492+0000 mgr.y (mgr.14146) 33 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:29.326016+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:30.326227+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.009890+0000 mgr.y (mgr.14146) 34 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: audit 2022-04-22T17:17:31.326348+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.353277+0000 mon.a (mon.0) 137 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.359001+0000 mon.a (mon.0) 138 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.359143+0000 mon.a (mon.0) 139 : cluster [DBG] fsmap 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.359225+0000 mon.a (mon.0) 140 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.359630+0000 mon.a (mon.0) 141 : cluster [DBG] mgrmap e13: y(active, since 55s) 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[18070]: cluster 2022-04-22T17:17:31.364892+0000 mon.a (mon.0) 142 : cluster [INF] overall HEALTH_OK 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:26.329574+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:17:31.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:26.329949+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:31.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:26.331512+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:27.009063+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:27.326049+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:28.325961+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:28.328288+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:29.009492+0000 mgr.y (mgr.14146) 33 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:29.326016+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:30.326227+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.009890+0000 mgr.y (mgr.14146) 34 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: audit 2022-04-22T17:17:31.326348+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.353277+0000 mon.a (mon.0) 137 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-04-22T17:17:31.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.359001+0000 mon.a (mon.0) 138 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:17:31.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.359143+0000 mon.a (mon.0) 139 : cluster [DBG] fsmap 2022-04-22T17:17:31.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.359225+0000 mon.a (mon.0) 140 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:31.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.359630+0000 mon.a (mon.0) 141 : cluster [DBG] mgrmap e13: y(active, since 55s) 2022-04-22T17:17:31.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:31 smithi115 bash[12737]: cluster 2022-04-22T17:17:31.364892+0000 mon.a (mon.0) 142 : cluster [INF] overall HEALTH_OK 2022-04-22T17:17:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-04-22T17:17:31.939 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":2,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","modified":"2022-04-22T17:17:26.326238Z","created":"2022-04-22T17:15:27.986542Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3301","nonce":0},{"type":"v1","addr":"172.21.15.115:6790","nonce":0}]},"addr":"172.21.15.115:6790/0","public_addr":"172.21.15.115:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-04-22T17:17:31.943 INFO:teuthology.orchestra.run.smithi121.stderr:dumped monmap epoch 2 2022-04-22T17:17:31.992 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:31 smithi121 systemd[1]: Started Ceph mon.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:17:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:31.946836+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? 172.21.15.121:0/3019011728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.004378+0000 mon.a (mon.0) 144 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.006579+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.008897+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.010743+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.207921+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[12737]: audit 2022-04-22T17:17:32.326974+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:31.946836+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? 172.21.15.121:0/3019011728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.004378+0000 mon.a (mon.0) 144 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.006579+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.008897+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.010743+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.207921+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:32.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:32 smithi115 bash[18070]: audit 2022-04-22T17:17:32.326974+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.505+0000 7fea5f6e1700 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.505+0000 7fea5f6e1700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.505+0000 7fea5f6e1700 0 pidfile_write: ignore empty --pid-file 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.509+0000 7fea5f6e1700 0 load: jerasure load: lrc load: isa 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: RocksDB version: 6.8.1 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Compile date Jul 8 2021 2022-04-22T17:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: DB SUMMARY 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: CURRENT file: CURRENT 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: IDENTITY file: IDENTITY 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 511 ; 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.create_if_missing: 0 2022-04-22T17:17:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.env: 0x556c3a5a21c0 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.info_log: 0x556c3ba8de20 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:17:33.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.db_log_dir: 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-04-22T17:17:33.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:17:33.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.write_buffer_manager: 0x556c3bb25f20 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:17:33.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:17:33.773 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.row_cache: None 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.wal_filter: None 2022-04-22T17:17:33.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:17:33.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:17:33.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Compression algorithms supported: 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:17:33.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kZSTD supported: 0 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2022-04-22T17:17:33.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.merge_operator: 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556c3ba34170) 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: cache_index_and_filter_blocks: 1 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:17:33.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: pin_top_level_index_and_filter: 1 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: index_type: 0 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: data_block_index_type: 0 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: index_shortening: 1 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: hash_index_allow_collision: 1 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: checksum: 1 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: no_block_cache: 0 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_cache: 0x556c3ba6cf10 2022-04-22T17:17:33.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_cache_name: BinnedLRUCache 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_cache_options: 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: capacity : 536870912 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: num_shard_bits : 4 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: strict_capacity_limit : 0 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: high_pri_pool_ratio: 0.000 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_cache_compressed: (nil) 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: persistent_cache: (nil) 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_size: 4096 2022-04-22T17:17:33.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_size_deviation: 10 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_restart_interval: 16 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: index_block_restart_interval: 1 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: metadata_block_size: 4096 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: partition_filters: 0 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: use_delta_encoding: 1 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:17:33.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: whole_key_filtering: 1 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: verify_compression: 0 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: read_amp_bytes_per_bit: 0 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: format_version: 2 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: enable_index_compression: 1 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: block_align: 0 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:17:33.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.num_levels: 7 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:17:33.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:17:33.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:17:33.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:17:33.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:17:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:17:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.force_consistency_checks: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:17:33.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647853516125, "job": 1, "event": "recovery_started", "log_files": [3]} 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647853516682, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1650647853, "oldest_key_time": 3, "file_creation_time": 0}} 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650647853517161, "job": 1, "event": "recovery_finished"} 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea5f6e1700 4 rocksdb: DB pointer 0x556c3bb31800 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.513+0000 7fea464e9700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-04-22T17:17:33.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.517+0000 7fea5f6e1700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.517+0000 7fea464e9700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** DB Stats ** 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:17:33.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** Compaction Stats [default] ** 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** Compaction Stats [default] ** 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:33.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** Compaction Stats [default] ** 2022-04-22T17:17:33.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** Compaction Stats [default] ** 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.000 0 0 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:17:33.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Flush(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.517+0000 7fea5f6e1700 0 using public_addr v2:172.21.15.121:0/0 -> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.517+0000 7fea5f6e1700 0 starting mon.b rank -1 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.517+0000 7fea5f6e1700 1 mon.b@-1(???) e0 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).mds e1 new map 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).mds e1 print_map 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: e1 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: legacy client fscid: -1 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: No filesystems configured 2022-04-22T17:17:33.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-04-22T17:17:33.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: cluster 2022-04-22T17:17:33.010339+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 1 mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expand map: {default=false} 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta from 'false' to 'false' 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expanded map: {default=false} 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expand map: {default=info} 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta from 'info' to 'info' 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expanded map: {default=info} 2022-04-22T17:17:33.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expand map: {default=daemon} 2022-04-22T17:17:33.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta from 'daemon' to 'daemon' 2022-04-22T17:17:33.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expanded map: {default=daemon} 2022-04-22T17:17:33.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expand map: {default=debug} 2022-04-22T17:17:33.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta from 'debug' to 'debug' 2022-04-22T17:17:33.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:33 smithi121 bash[12679]: debug 2022-04-22T17:17:33.545+0000 7fea48cee700 20 expand_channel_meta expanded map: {default=debug} 2022-04-22T17:17:33.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:33 smithi115 bash[12737]: cluster 2022-04-22T17:17:33.010339+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:33.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:33 smithi115 bash[18070]: cluster 2022-04-22T17:17:33.010339+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:34.113 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-04-22T17:17:34.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph mon dump -f json 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:33.569536+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:33.569830+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:33.570062+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:33.572548+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:33.588828+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:34.562926+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:35.010705+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:35.555602+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:35.562824+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:36.562970+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:37.011108+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:37.216788+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:37.219358+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:37.563119+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.563260+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.574773+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.596492+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.610663+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.610764+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.610818+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.611097+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 62s) 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: cluster 2022-04-22T17:17:38.617689+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.622126+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.625969+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.645206+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.653282+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.657889+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.659639+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:38 smithi121 bash[12679]: audit 2022-04-22T17:17:38.660864+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:33.569536+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:33.569830+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:33.570062+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:33.572548+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:33.588828+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:39.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:34.562926+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:35.010705+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:35.555602+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:35.562824+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:36.562970+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:37.011108+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:37.216788+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:17:39.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:37.219358+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:17:39.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:37.563119+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.563260+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.574773+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.596492+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:17:39.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.610663+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:17:39.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.610764+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2022-04-22T17:17:39.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.610818+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:39.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.611097+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 62s) 2022-04-22T17:17:39.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:33.569536+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:17:39.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:33.569830+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:33.570062+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:33.572548+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:33.588828+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:34.562926+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:35.010705+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:35.555602+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:35.562824+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:36.562970+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:37.011108+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:39.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:37.216788+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:37.219358+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:37.563119+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.563260+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.574773+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.596492+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:17:39.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.610663+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.610764+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.610818+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.611097+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 62s) 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: cluster 2022-04-22T17:17:38.617689+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.622126+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.625969+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.645206+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.653282+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.657889+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.659639+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[12737]: audit 2022-04-22T17:17:38.660864+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:39.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: cluster 2022-04-22T17:17:38.617689+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2022-04-22T17:17:39.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.622126+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.625969+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.645206+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.653282+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.657889+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.659639+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:38 smithi115 bash[18070]: audit 2022-04-22T17:17:38.660864+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:39.059 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-04-22T17:17:39.059 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":3,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","modified":"2022-04-22T17:17:33.562682Z","created":"2022-04-22T17:15:27.986542Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3301","nonce":0},{"type":"v1","addr":"172.21.15.115:6790","nonce":0}]},"addr":"172.21.15.115:6790/0","public_addr":"172.21.15.115:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2022-04-22T17:17:39.062 INFO:teuthology.orchestra.run.smithi121.stderr:dumped monmap epoch 3 2022-04-22T17:17:39.806 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-04-22T17:17:39.806 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph config generate-minimal-conf 2022-04-22T17:17:40.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: cephadm 2022-04-22T17:17:38.622996+0000 mgr.y (mgr.14146) 38 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: cephadm 2022-04-22T17:17:38.626884+0000 mgr.y (mgr.14146) 39 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: cephadm 2022-04-22T17:17:38.659272+0000 mgr.y (mgr.14146) 40 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2022-04-22T17:17:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: audit 2022-04-22T17:17:38.661960+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: cephadm 2022-04-22T17:17:38.663361+0000 mgr.y (mgr.14146) 41 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:17:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: cluster 2022-04-22T17:17:39.011686+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: audit 2022-04-22T17:17:39.066033+0000 mon.a (mon.0) 177 : audit [DBG] from='client.? 172.21.15.121:0/2629322286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[18070]: audit 2022-04-22T17:17:39.563567+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: cephadm 2022-04-22T17:17:38.622996+0000 mgr.y (mgr.14146) 38 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: cephadm 2022-04-22T17:17:38.626884+0000 mgr.y (mgr.14146) 39 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: cephadm 2022-04-22T17:17:38.659272+0000 mgr.y (mgr.14146) 40 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: audit 2022-04-22T17:17:38.661960+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: cephadm 2022-04-22T17:17:38.663361+0000 mgr.y (mgr.14146) 41 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: cluster 2022-04-22T17:17:39.011686+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: audit 2022-04-22T17:17:39.066033+0000 mon.a (mon.0) 177 : audit [DBG] from='client.? 172.21.15.121:0/2629322286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:40.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:39 smithi115 bash[12737]: audit 2022-04-22T17:17:39.563567+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: cephadm 2022-04-22T17:17:38.622996+0000 mgr.y (mgr.14146) 38 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: cephadm 2022-04-22T17:17:38.626884+0000 mgr.y (mgr.14146) 39 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: cephadm 2022-04-22T17:17:38.659272+0000 mgr.y (mgr.14146) 40 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: audit 2022-04-22T17:17:38.661960+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: cephadm 2022-04-22T17:17:38.663361+0000 mgr.y (mgr.14146) 41 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:17:40.044 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: cluster 2022-04-22T17:17:39.011686+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:40.045 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: audit 2022-04-22T17:17:39.066033+0000 mon.a (mon.0) 177 : audit [DBG] from='client.? 172.21.15.121:0/2629322286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-04-22T17:17:40.045 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:39 smithi121 bash[12679]: audit 2022-04-22T17:17:39.563567+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:17:42.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[18070]: cluster 2022-04-22T17:17:41.012074+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[18070]: audit 2022-04-22T17:17:41.560828+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[18070]: audit 2022-04-22T17:17:41.562226+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[18070]: audit 2022-04-22T17:17:41.563248+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[18070]: audit 2022-04-22T17:17:41.564257+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[12737]: cluster 2022-04-22T17:17:41.012074+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[12737]: audit 2022-04-22T17:17:41.560828+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[12737]: audit 2022-04-22T17:17:41.562226+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:42.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[12737]: audit 2022-04-22T17:17:41.563248+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:42.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:42 smithi115 bash[12737]: audit 2022-04-22T17:17:41.564257+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:41 smithi121 bash[12679]: cluster 2022-04-22T17:17:41.012074+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:41 smithi121 bash[12679]: audit 2022-04-22T17:17:41.560828+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:42 smithi121 bash[12679]: audit 2022-04-22T17:17:41.562226+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:42 smithi121 bash[12679]: audit 2022-04-22T17:17:41.563248+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:42 smithi121 bash[12679]: audit 2022-04-22T17:17:41.564257+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:43 smithi121 bash[12679]: cephadm 2022-04-22T17:17:41.561858+0000 mgr.y (mgr.14146) 44 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:17:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:43 smithi121 bash[12679]: cephadm 2022-04-22T17:17:41.565435+0000 mgr.y (mgr.14146) 45 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:17:43.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:43 smithi115 bash[12737]: cephadm 2022-04-22T17:17:41.561858+0000 mgr.y (mgr.14146) 44 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:17:43.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:43 smithi115 bash[12737]: cephadm 2022-04-22T17:17:41.565435+0000 mgr.y (mgr.14146) 45 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:17:43.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:43 smithi115 bash[18070]: cephadm 2022-04-22T17:17:41.561858+0000 mgr.y (mgr.14146) 44 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:17:43.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:43 smithi115 bash[18070]: cephadm 2022-04-22T17:17:41.565435+0000 mgr.y (mgr.14146) 45 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:17:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:44 smithi121 bash[12679]: cluster 2022-04-22T17:17:43.012490+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:44.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:44 smithi115 bash[12737]: cluster 2022-04-22T17:17:43.012490+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:44.273 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:44 smithi115 bash[18070]: cluster 2022-04-22T17:17:43.012490+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:45.556 INFO:teuthology.orchestra.run.smithi115.stdout:# minimal ceph.conf for b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:45.639 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2022-04-22T17:17:45.642 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:17:45.642 INFO:teuthology.orchestra.run.smithi115.stdout: mon_host = [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: cluster 2022-04-22T17:17:45.013039+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: audit 2022-04-22T17:17:45.246490+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: audit 2022-04-22T17:17:45.247603+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: audit 2022-04-22T17:17:45.248619+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: audit 2022-04-22T17:17:45.249751+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:46 smithi121 bash[12679]: audit 2022-04-22T17:17:45.564389+0000 mon.a (mon.0) 187 : audit [DBG] from='client.? 172.21.15.115:0/396567403' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: cluster 2022-04-22T17:17:45.013039+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: audit 2022-04-22T17:17:45.246490+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: audit 2022-04-22T17:17:45.247603+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: audit 2022-04-22T17:17:45.248619+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: audit 2022-04-22T17:17:45.249751+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[12737]: audit 2022-04-22T17:17:45.564389+0000 mon.a (mon.0) 187 : audit [DBG] from='client.? 172.21.15.115:0/396567403' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: cluster 2022-04-22T17:17:45.013039+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:46.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: audit 2022-04-22T17:17:45.246490+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:46.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: audit 2022-04-22T17:17:45.247603+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:17:46.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: audit 2022-04-22T17:17:45.248619+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:17:46.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: audit 2022-04-22T17:17:45.249751+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:46 smithi115 bash[18070]: audit 2022-04-22T17:17:45.564389+0000 mon.a (mon.0) 187 : audit [DBG] from='client.? 172.21.15.115:0/396567403' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:46.364 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-04-22T17:17:46.364 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:17:46.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2022-04-22T17:17:46.377 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:17:46.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:17:46.389 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:17:46.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2022-04-22T17:17:46.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:17:46.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:17:46.463 INFO:tasks.cephadm:Adding mgr.y on smithi115 2022-04-22T17:17:46.463 INFO:tasks.cephadm:Adding mgr.x on smithi121 2022-04-22T17:17:46.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply mgr '2;smithi115=y;smithi121=x' 2022-04-22T17:17:47.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:47 smithi115 bash[12737]: cephadm 2022-04-22T17:17:45.247299+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:17:47.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:47 smithi115 bash[12737]: cephadm 2022-04-22T17:17:45.251037+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:17:47.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:47 smithi115 bash[18070]: cephadm 2022-04-22T17:17:45.247299+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:17:47.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:47 smithi115 bash[18070]: cephadm 2022-04-22T17:17:45.251037+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:17:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:47 smithi121 bash[12679]: cephadm 2022-04-22T17:17:45.247299+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:17:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:47 smithi121 bash[12679]: cephadm 2022-04-22T17:17:45.251037+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:17:48.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: cluster 2022-04-22T17:17:47.013479+0000 mgr.y (mgr.14146) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.886707+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.888309+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.890791+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.892644+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.917562+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.943186+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[12737]: audit 2022-04-22T17:17:47.950992+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: cluster 2022-04-22T17:17:47.013479+0000 mgr.y (mgr.14146) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.886707+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.888309+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.890791+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.892644+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.917562+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.943186+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:48 smithi115 bash[18070]: audit 2022-04-22T17:17:47.950992+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: cluster 2022-04-22T17:17:47.013479+0000 mgr.y (mgr.14146) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:48.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.886707+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.888309+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:48.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.890791+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:48.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.892644+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:48.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.917562+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.943186+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:48.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:48 smithi121 bash[12679]: audit 2022-04-22T17:17:47.950992+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:49 smithi121 bash[12679]: cephadm 2022-04-22T17:17:47.894636+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:49 smithi121 bash[12679]: cephadm 2022-04-22T17:17:47.895010+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:49.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:49 smithi115 bash[12737]: cephadm 2022-04-22T17:17:47.894636+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:49.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:49 smithi115 bash[12737]: cephadm 2022-04-22T17:17:47.895010+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:49.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:49 smithi115 bash[18070]: cephadm 2022-04-22T17:17:47.894636+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:17:49.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:49 smithi115 bash[18070]: cephadm 2022-04-22T17:17:47.895010+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:17:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:50 smithi121 bash[12679]: cluster 2022-04-22T17:17:49.013902+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:50.287 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled mgr update... 2022-04-22T17:17:50.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:50 smithi115 bash[12737]: cluster 2022-04-22T17:17:49.013902+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:50.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:50 smithi115 bash[18070]: cluster 2022-04-22T17:17:49.013902+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:51.354 DEBUG:teuthology.orchestra.run.smithi121:mgr.x> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.x.service 2022-04-22T17:17:51.357 INFO:tasks.cephadm:Deploying OSDs... 2022-04-22T17:17:51.357 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:17:51.357 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2022-04-22T17:17:51.367 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-22T17:17:51.367 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-22T17:17:51.424 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-22T17:17:51.424 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.424 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-04-22T17:17:51.424 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.425 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-04-22 17:12:15.342873280 +0000 2022-04-22T17:17:51.425 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-04-22 17:12:15.126873285 +0000 2022-04-22T17:17:51.425 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-04-22 17:12:15.126873285 +0000 2022-04-22T17:17:51.425 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-04-22T17:17:51.434 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-04-22T17:17:51.435 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-04-22T17:17:51.435 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000440559 s, 1.2 MB/s 2022-04-22T17:17:51.437 INFO:journalctl@ceph.mgr.x.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:17:51.440 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-04-22 17:12:15.786873270 +0000 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-04-22 17:12:15.558873275 +0000 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-04-22 17:12:15.558873275 +0000 2022-04-22T17:17:51.452 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-04-22T17:17:51.462 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-04-22T17:17:51.463 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-04-22T17:17:51.463 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000443357 s, 1.2 MB/s 2022-04-22T17:17:51.465 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-04-22 17:12:16.246873260 +0000 2022-04-22T17:17:51.525 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-04-22 17:12:16.034873265 +0000 2022-04-22T17:17:51.526 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-04-22 17:12:16.034873265 +0000 2022-04-22T17:17:51.526 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-04-22T17:17:51.533 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-04-22T17:17:51.533 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-04-22T17:17:51.533 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00050738 s, 1.0 MB/s 2022-04-22T17:17:51.537 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-22T17:17:51.592 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-22T17:17:51.592 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.592 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-04-22T17:17:51.593 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.593 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-04-22 17:12:16.662873251 +0000 2022-04-22T17:17:51.593 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-04-22 17:12:16.454873255 +0000 2022-04-22T17:17:51.593 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-04-22 17:12:16.454873255 +0000 2022-04-22T17:17:51.593 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-04-22T17:17:51.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.285844+0000 mgr.y (mgr.14146) 54 : audit [DBG] from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:17:51.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: cephadm 2022-04-22T17:17:50.287522+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Saving service mgr spec with placement smithi115=y;smithi121=x;count:2 2022-04-22T17:17:51.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.293761+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.295209+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:51.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.297634+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.299500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.305176+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.307127+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.310926+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.312248+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: audit 2022-04-22T17:17:50.313777+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: cephadm 2022-04-22T17:17:50.315064+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:17:51.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[12737]: cluster 2022-04-22T17:17:51.014326+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.285844+0000 mgr.y (mgr.14146) 54 : audit [DBG] from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: cephadm 2022-04-22T17:17:50.287522+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Saving service mgr spec with placement smithi115=y;smithi121=x;count:2 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.293761+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.295209+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.297634+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.299500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:51.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.305176+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.307127+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.310926+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.312248+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: audit 2022-04-22T17:17:50.313777+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: cephadm 2022-04-22T17:17:50.315064+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:17:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:51 smithi115 bash[18070]: cluster 2022-04-22T17:17:51.014326+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:51.604 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-04-22T17:17:51.604 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-04-22T17:17:51.604 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000552105 s, 927 kB/s 2022-04-22T17:17:51.606 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:17:51.607 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2022-04-22T17:17:51.615 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-22T17:17:51.615 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-04-22 17:12:16.974280976 +0000 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-04-22 17:12:16.770280978 +0000 2022-04-22T17:17:51.621 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-04-22 17:12:16.770280978 +0000 2022-04-22T17:17:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.285844+0000 mgr.y (mgr.14146) 54 : audit [DBG] from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: cephadm 2022-04-22T17:17:50.287522+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Saving service mgr spec with placement smithi115=y;smithi121=x;count:2 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.293761+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.295209+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.297634+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.299500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.305176+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.307127+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.310926+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.312248+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: audit 2022-04-22T17:17:50.313777+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: cephadm 2022-04-22T17:17:50.315064+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:17:51.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:51 smithi121 bash[12679]: cluster 2022-04-22T17:17:51.014326+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:51.629 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-04-22T17:17:51.629 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-04-22T17:17:51.630 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000557414 s, 919 kB/s 2022-04-22T17:17:51.632 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-04-22 17:12:17.390280971 +0000 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-04-22 17:12:17.186280973 +0000 2022-04-22T17:17:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-04-22 17:12:17.186280973 +0000 2022-04-22T17:17:51.640 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-04-22T17:17:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-04-22T17:17:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-04-22T17:17:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00044092 s, 1.2 MB/s 2022-04-22T17:17:51.650 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-04-22 17:12:17.814280967 +0000 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-04-22 17:12:17.614280969 +0000 2022-04-22T17:17:51.658 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-04-22 17:12:17.614280969 +0000 2022-04-22T17:17:51.659 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-04-22T17:17:51.664 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-04-22T17:17:51.664 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-04-22T17:17:51.664 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000536096 s, 955 kB/s 2022-04-22T17:17:51.668 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-04-22 17:12:18.238280962 +0000 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-04-22 17:12:18.038280964 +0000 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-04-22 17:12:18.038280964 +0000 2022-04-22T17:17:51.721 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-04-22T17:17:51.729 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-04-22T17:17:51.729 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-04-22T17:17:51.729 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000500686 s, 1.0 MB/s 2022-04-22T17:17:51.732 INFO:tasks.cephadm:Deploying osd.0 on smithi115 with /dev/vg_nvme/lv_4... 2022-04-22T17:17:51.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-04-22T17:17:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:53 smithi121 bash[12679]: cluster 2022-04-22T17:17:53.014817+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:53.263 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:17:53 smithi121 systemd[1]: Started Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:17:53.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:53 smithi115 bash[18070]: cluster 2022-04-22T17:17:53.014817+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:53.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:53 smithi115 bash[12737]: cluster 2022-04-22T17:17:53.014817+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:54.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[12737]: audit 2022-04-22T17:17:53.214769+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:54.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[12737]: audit 2022-04-22T17:17:53.217547+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[12737]: audit 2022-04-22T17:17:53.219449+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[12737]: audit 2022-04-22T17:17:53.220740+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[18070]: audit 2022-04-22T17:17:53.214769+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[18070]: audit 2022-04-22T17:17:53.217547+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[18070]: audit 2022-04-22T17:17:53.219449+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:54.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:54 smithi115 bash[18070]: audit 2022-04-22T17:17:53.220740+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:54.515 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:17:54 smithi121 bash[13536]: debug 2022-04-22T17:17:54.369+0000 7fe13c7cb700 1 -- 172.21.15.121:0/3500202070 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a7794004e0 con 0x55a7794ee800 2022-04-22T17:17:54.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:54 smithi121 bash[12679]: audit 2022-04-22T17:17:53.214769+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:54.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:54 smithi121 bash[12679]: audit 2022-04-22T17:17:53.217547+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:17:54.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:54 smithi121 bash[12679]: audit 2022-04-22T17:17:53.219449+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:54.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:54 smithi121 bash[12679]: audit 2022-04-22T17:17:53.220740+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:17:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:55 smithi121 bash[12679]: cluster 2022-04-22T17:17:55.015303+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:55.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:55 smithi115 bash[12737]: cluster 2022-04-22T17:17:55.015303+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:55.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:55 smithi115 bash[18070]: cluster 2022-04-22T17:17:55.015303+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:55.565 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-04-22T17:17:55.565 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-04-22T17:17:55.565 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-04-22T17:17:55.566 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-04-22T17:17:55.566 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0166734 s, 629 MB/s 2022-04-22T17:17:55.566 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-04-22T17:17:56.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2022-04-22T17:17:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:56.459714+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:56.464197+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: cephadm 2022-04-22T17:17:56.465938+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2022-04-22T17:17:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:56.466357+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:56.467779+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:56.468906+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: cephadm 2022-04-22T17:17:56.470007+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:17:57.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: cluster 2022-04-22T17:17:57.015794+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:57.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:57 smithi121 bash[12679]: audit 2022-04-22T17:17:57.215778+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:56.459714+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:56.464197+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: cephadm 2022-04-22T17:17:56.465938+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:56.466357+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:56.467779+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:56.468906+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:57.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: cephadm 2022-04-22T17:17:56.470007+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: cluster 2022-04-22T17:17:57.015794+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[18070]: audit 2022-04-22T17:17:57.215778+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:56.459714+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:56.464197+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: cephadm 2022-04-22T17:17:56.465938+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:56.466357+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:56.467779+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:17:57.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:56.468906+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:17:57.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: cephadm 2022-04-22T17:17:56.470007+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:17:57.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: cluster 2022-04-22T17:17:57.015794+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:57.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:57 smithi115 bash[12737]: audit 2022-04-22T17:17:57.215778+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:17:59.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:17:59 smithi115 bash[18070]: cluster 2022-04-22T17:17:59.016161+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:59.236 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:17:59 smithi115 bash[12737]: cluster 2022-04-22T17:17:59.016161+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:17:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:17:59 smithi121 bash[12679]: cluster 2022-04-22T17:17:59.016161+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:00 smithi121 bash[12679]: audit 2022-04-22T17:17:59.156647+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:00 smithi121 bash[12679]: audit 2022-04-22T17:17:59.157970+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:00 smithi121 bash[12679]: audit 2022-04-22T17:17:59.159393+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:00 smithi121 bash[12679]: audit 2022-04-22T17:17:59.160889+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:00 smithi121 bash[12679]: audit 2022-04-22T17:17:59.166929+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[12737]: audit 2022-04-22T17:17:59.156647+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[12737]: audit 2022-04-22T17:17:59.157970+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[12737]: audit 2022-04-22T17:17:59.159393+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[12737]: audit 2022-04-22T17:17:59.160889+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[12737]: audit 2022-04-22T17:17:59.166929+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[18070]: audit 2022-04-22T17:17:59.156647+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[18070]: audit 2022-04-22T17:17:59.157970+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:00.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[18070]: audit 2022-04-22T17:17:59.159393+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:00.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[18070]: audit 2022-04-22T17:17:59.160889+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:00.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:00 smithi115 bash[18070]: audit 2022-04-22T17:17:59.166929+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:01.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.616369+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:18:01.281 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.617063+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:18:01.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.618952+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:18:01.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.619665+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:18:01.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: cluster 2022-04-22T17:18:00.623194+0000 mon.a (mon.0) 219 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:18:01.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.728603+0000 mgr.y (mgr.14146) 64 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:01.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.729713+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:01.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.732282+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:01.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: audit 2022-04-22T17:18:00.733832+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:01.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[12737]: cluster 2022-04-22T17:18:01.016576+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:01.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.616369+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:18:01.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.617063+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:18:01.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.618952+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:18:01.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.619665+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:18:01.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: cluster 2022-04-22T17:18:00.623194+0000 mon.a (mon.0) 219 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:18:01.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.728603+0000 mgr.y (mgr.14146) 64 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:01.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.729713+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:01.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.732282+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: audit 2022-04-22T17:18:00.733832+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:01 smithi115 bash[18070]: cluster 2022-04-22T17:18:01.016576+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.616369+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:18:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.617063+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:18:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.618952+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:18:01.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.619665+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.121:0/3255568931' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:18:01.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: cluster 2022-04-22T17:18:00.623194+0000 mon.a (mon.0) 219 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:18:01.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.728603+0000 mgr.y (mgr.14146) 64 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:01.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.729713+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:01.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.732282+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:01.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: audit 2022-04-22T17:18:00.733832+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:01.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:01 smithi121 bash[12679]: cluster 2022-04-22T17:18:01.016576+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:02.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:02 smithi115 bash[12737]: cluster 2022-04-22T17:18:01.171214+0000 mon.a (mon.0) 223 : cluster [DBG] mgrmap e14: y(active, since 84s), standbys: x 2022-04-22T17:18:02.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:02 smithi115 bash[12737]: audit 2022-04-22T17:18:01.171839+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:18:02.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:02 smithi115 bash[18070]: cluster 2022-04-22T17:18:01.171214+0000 mon.a (mon.0) 223 : cluster [DBG] mgrmap e14: y(active, since 84s), standbys: x 2022-04-22T17:18:02.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:02 smithi115 bash[18070]: audit 2022-04-22T17:18:01.171839+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:18:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:02 smithi121 bash[12679]: cluster 2022-04-22T17:18:01.171214+0000 mon.a (mon.0) 223 : cluster [DBG] mgrmap e14: y(active, since 84s), standbys: x 2022-04-22T17:18:02.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:02 smithi121 bash[12679]: audit 2022-04-22T17:18:01.171839+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:18:03.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:03 smithi115 bash[12737]: cluster 2022-04-22T17:18:03.016911+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:03.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:03 smithi115 bash[18070]: cluster 2022-04-22T17:18:03.016911+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:03 smithi121 bash[12679]: cluster 2022-04-22T17:18:03.016911+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:04 smithi121 bash[12679]: audit 2022-04-22T17:18:04.913942+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.115:0/4015503490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:04 smithi121 bash[12679]: audit 2022-04-22T17:18:04.914618+0000 mon.a (mon.0) 225 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:04 smithi121 bash[12679]: audit 2022-04-22T17:18:04.921976+0000 mon.a (mon.0) 226 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]': finished 2022-04-22T17:18:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:04 smithi121 bash[12679]: cluster 2022-04-22T17:18:04.922163+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2022-04-22T17:18:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:04 smithi121 bash[12679]: audit 2022-04-22T17:18:04.937327+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[12737]: audit 2022-04-22T17:18:04.913942+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.115:0/4015503490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[12737]: audit 2022-04-22T17:18:04.914618+0000 mon.a (mon.0) 225 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[12737]: audit 2022-04-22T17:18:04.921976+0000 mon.a (mon.0) 226 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]': finished 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[12737]: cluster 2022-04-22T17:18:04.922163+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[12737]: audit 2022-04-22T17:18:04.937327+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:05.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[18070]: audit 2022-04-22T17:18:04.913942+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.115:0/4015503490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[18070]: audit 2022-04-22T17:18:04.914618+0000 mon.a (mon.0) 225 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]: dispatch 2022-04-22T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[18070]: audit 2022-04-22T17:18:04.921976+0000 mon.a (mon.0) 226 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc5fef33-43b5-4c7a-bbf5-cc37de953e22"}]': finished 2022-04-22T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[18070]: cluster 2022-04-22T17:18:04.922163+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2022-04-22T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:04 smithi115 bash[18070]: audit 2022-04-22T17:18:04.937327+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:05 smithi121 bash[12679]: cluster 2022-04-22T17:18:05.017300+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:05 smithi121 bash[12679]: audit 2022-04-22T17:18:05.578400+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.115:0/1205992376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:06.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:05 smithi115 bash[12737]: cluster 2022-04-22T17:18:05.017300+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:06.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:05 smithi115 bash[12737]: audit 2022-04-22T17:18:05.578400+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.115:0/1205992376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:06.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:05 smithi115 bash[18070]: cluster 2022-04-22T17:18:05.017300+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:06.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:05 smithi115 bash[18070]: audit 2022-04-22T17:18:05.578400+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.115:0/1205992376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:07 smithi121 bash[12679]: cluster 2022-04-22T17:18:07.017816+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:07.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:07 smithi115 bash[12737]: cluster 2022-04-22T17:18:07.017816+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:07.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:07 smithi115 bash[18070]: cluster 2022-04-22T17:18:07.017816+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:09 smithi121 bash[12679]: cluster 2022-04-22T17:18:09.018340+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:09.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:09 smithi115 bash[12737]: cluster 2022-04-22T17:18:09.018340+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:09.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:09 smithi115 bash[18070]: cluster 2022-04-22T17:18:09.018340+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:12 smithi121 bash[12679]: cluster 2022-04-22T17:18:11.018756+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:12.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:12 smithi115 bash[12737]: cluster 2022-04-22T17:18:11.018756+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:12.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:12 smithi115 bash[18070]: cluster 2022-04-22T17:18:11.018756+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:13.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:13 smithi115 bash[18070]: cluster 2022-04-22T17:18:13.019145+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:13 smithi115 bash[12737]: cluster 2022-04-22T17:18:13.019145+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:13.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:13 smithi121 bash[12679]: cluster 2022-04-22T17:18:13.019145+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:14.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[12737]: audit 2022-04-22T17:18:13.234635+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-04-22T17:18:14.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[12737]: audit 2022-04-22T17:18:13.236355+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:14.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[12737]: cephadm 2022-04-22T17:18:13.237800+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:18:14.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[18070]: audit 2022-04-22T17:18:13.234635+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-04-22T17:18:14.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[18070]: audit 2022-04-22T17:18:13.236355+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:14.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:14 smithi115 bash[18070]: cephadm 2022-04-22T17:18:13.237800+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:18:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:14 smithi121 bash[12679]: audit 2022-04-22T17:18:13.234635+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-04-22T17:18:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:14 smithi121 bash[12679]: audit 2022-04-22T17:18:13.236355+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:14 smithi121 bash[12679]: cephadm 2022-04-22T17:18:13.237800+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:18:15.415 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:15 smithi115 bash[12737]: cluster 2022-04-22T17:18:15.019607+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:15.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:15 smithi115 bash[18070]: cluster 2022-04-22T17:18:15.019607+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:15 smithi121 bash[12679]: cluster 2022-04-22T17:18:15.019607+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:15.927 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 0 on host 'smithi115' 2022-04-22T17:18:16.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[12737]: audit 2022-04-22T17:18:15.918138+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:16.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[12737]: audit 2022-04-22T17:18:15.921268+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:16.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[12737]: audit 2022-04-22T17:18:15.922118+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:16.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[12737]: audit 2022-04-22T17:18:15.927336+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:16 smithi121 bash[12679]: audit 2022-04-22T17:18:15.918138+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:16.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:16 smithi121 bash[12679]: audit 2022-04-22T17:18:15.921268+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:16.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:16 smithi121 bash[12679]: audit 2022-04-22T17:18:15.922118+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:16.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:16 smithi121 bash[12679]: audit 2022-04-22T17:18:15.927336+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:16.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:16 smithi121 bash[12679]: audit 2022-04-22T17:18:15.934494+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:16.612 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[12737]: audit 2022-04-22T17:18:15.934494+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[18070]: audit 2022-04-22T17:18:15.918138+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[18070]: audit 2022-04-22T17:18:15.921268+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[18070]: audit 2022-04-22T17:18:15.922118+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[18070]: audit 2022-04-22T17:18:15.927336+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:16 smithi115 bash[18070]: audit 2022-04-22T17:18:15.934494+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:17 smithi115 bash[12737]: cluster 2022-04-22T17:18:17.019908+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:17 smithi115 bash[18070]: cluster 2022-04-22T17:18:17.019908+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:17.442 DEBUG:teuthology.orchestra.run.smithi115:osd.0> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.0.service 2022-04-22T17:18:17.445 INFO:tasks.cephadm:Deploying osd.1 on smithi115 with /dev/vg_nvme/lv_3... 2022-04-22T17:18:17.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-04-22T17:18:17.486 INFO:journalctl@ceph.osd.0.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:18:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:17 smithi121 bash[12679]: cluster 2022-04-22T17:18:17.019908+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-04-22T17:18:18.529 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:18 smithi115 bash[20585]: --> ceph-volume lvm activate successful for osd ID: 0 2022-04-22T17:18:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:19 smithi121 bash[12679]: cluster 2022-04-22T17:18:19.020319+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:19.521 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:19 smithi115 bash[12737]: cluster 2022-04-22T17:18:19.020319+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:19.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:19 smithi115 bash[18070]: cluster 2022-04-22T17:18:19.020319+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:22.007 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:21 smithi115 bash[20761]: debug 2022-04-22T17:18:21.682+0000 7fac1dc60700 1 -- 172.21.15.115:0/2637967440 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562db33444e0 con 0x562db33bac00 2022-04-22T17:18:22.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:22 smithi115 bash[12737]: cluster 2022-04-22T17:18:21.020670+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:22.287 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:22 smithi115 bash[20761]: debug 2022-04-22T17:18:22.258+0000 7fac256f9080 -1 Falling back to public interface 2022-04-22T17:18:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:22 smithi115 bash[18070]: cluster 2022-04-22T17:18:21.020670+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:22.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:22 smithi121 bash[12679]: cluster 2022-04-22T17:18:21.020670+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0172713 s, 607 MB/s 2022-04-22T17:18:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-04-22T17:18:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:23 smithi121 bash[12679]: audit 2022-04-22T17:18:22.068777+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:23 smithi121 bash[12679]: cluster 2022-04-22T17:18:23.021036+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:23 smithi115 bash[12737]: audit 2022-04-22T17:18:22.068777+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:23 smithi115 bash[12737]: cluster 2022-04-22T17:18:23.021036+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:23 smithi115 bash[18070]: audit 2022-04-22T17:18:22.068777+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:23 smithi115 bash[18070]: cluster 2022-04-22T17:18:23.021036+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:24.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2022-04-22T17:18:25.507 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:25 smithi115 bash[12737]: cluster 2022-04-22T17:18:25.021419+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:25.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:25 smithi115 bash[18070]: cluster 2022-04-22T17:18:25.021419+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:25.507 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:25 smithi115 bash[20761]: debug 2022-04-22T17:18:25.278+0000 7fac256f9080 -1 osd.0 0 log_to_monitors {default=true} 2022-04-22T17:18:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:25 smithi121 bash[12679]: cluster 2022-04-22T17:18:25.021419+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:26 smithi121 bash[12679]: audit 2022-04-22T17:18:25.287267+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:26 smithi121 bash[12679]: audit 2022-04-22T17:18:25.287775+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:26.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:26 smithi115 bash[12737]: audit 2022-04-22T17:18:25.287267+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:26.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:26 smithi115 bash[12737]: audit 2022-04-22T17:18:25.287775+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:26.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:26 smithi115 bash[18070]: audit 2022-04-22T17:18:25.287267+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:26.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:26 smithi115 bash[18070]: audit 2022-04-22T17:18:25.287775+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:18:27.278 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[20761]: debug 2022-04-22T17:18:27.138+0000 7fac1bc5c700 -1 osd.0 0 waiting for initial osdmap 2022-04-22T17:18:27.279 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[20761]: debug 2022-04-22T17:18:27.138+0000 7fac14bd1700 -1 osd.0 7 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:18:27.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: audit 2022-04-22T17:18:26.128701+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:18:27.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: cluster 2022-04-22T17:18:26.128800+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2022-04-22T17:18:27.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: audit 2022-04-22T17:18:26.129001+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:27.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: audit 2022-04-22T17:18:26.129567+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: audit 2022-04-22T17:18:26.129962+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[12737]: cluster 2022-04-22T17:18:27.021668+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: audit 2022-04-22T17:18:26.128701+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: cluster 2022-04-22T17:18:26.128800+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: audit 2022-04-22T17:18:26.129001+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:27.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: audit 2022-04-22T17:18:26.129567+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: audit 2022-04-22T17:18:26.129962+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:27 smithi115 bash[18070]: cluster 2022-04-22T17:18:27.021668+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: audit 2022-04-22T17:18:26.128701+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:18:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: cluster 2022-04-22T17:18:26.128800+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2022-04-22T17:18:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: audit 2022-04-22T17:18:26.129001+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: audit 2022-04-22T17:18:26.129567+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: audit 2022-04-22T17:18:26.129962+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:27 smithi121 bash[12679]: cluster 2022-04-22T17:18:27.021668+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:28 smithi121 bash[12679]: audit 2022-04-22T17:18:27.131689+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:28 smithi121 bash[12679]: cluster 2022-04-22T17:18:27.131869+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2022-04-22T17:18:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:28 smithi121 bash[12679]: audit 2022-04-22T17:18:27.132612+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:28 smithi121 bash[12679]: audit 2022-04-22T17:18:27.140138+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:28.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[12737]: audit 2022-04-22T17:18:27.131689+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:28.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[12737]: cluster 2022-04-22T17:18:27.131869+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2022-04-22T17:18:28.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[12737]: audit 2022-04-22T17:18:27.132612+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:28.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[12737]: audit 2022-04-22T17:18:27.140138+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:28.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[18070]: audit 2022-04-22T17:18:27.131689+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:28.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[18070]: cluster 2022-04-22T17:18:27.131869+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2022-04-22T17:18:28.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[18070]: audit 2022-04-22T17:18:27.132612+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:28.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:28 smithi115 bash[18070]: audit 2022-04-22T17:18:27.140138+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:29.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: cluster 2022-04-22T17:18:26.300030+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:29.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: cluster 2022-04-22T17:18:26.300194+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:29.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: cluster 2022-04-22T17:18:28.141348+0000 mon.a (mon.0) 246 : cluster [INF] osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491] boot 2022-04-22T17:18:29.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: cluster 2022-04-22T17:18:28.141474+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2022-04-22T17:18:29.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: audit 2022-04-22T17:18:28.142017+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:29.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[18070]: cluster 2022-04-22T17:18:29.022070+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:29.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: cluster 2022-04-22T17:18:26.300030+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:29.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: cluster 2022-04-22T17:18:26.300194+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:29.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: cluster 2022-04-22T17:18:28.141348+0000 mon.a (mon.0) 246 : cluster [INF] osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491] boot 2022-04-22T17:18:29.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: cluster 2022-04-22T17:18:28.141474+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2022-04-22T17:18:29.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: audit 2022-04-22T17:18:28.142017+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:29.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:29 smithi115 bash[12737]: cluster 2022-04-22T17:18:29.022070+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:29.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: cluster 2022-04-22T17:18:26.300030+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:29.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: cluster 2022-04-22T17:18:26.300194+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:29.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: cluster 2022-04-22T17:18:28.141348+0000 mon.a (mon.0) 246 : cluster [INF] osd.0 [v2:172.21.15.115:6802/1141629491,v1:172.21.15.115:6803/1141629491] boot 2022-04-22T17:18:29.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: cluster 2022-04-22T17:18:28.141474+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2022-04-22T17:18:29.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: audit 2022-04-22T17:18:28.142017+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:18:29.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:29 smithi121 bash[12679]: cluster 2022-04-22T17:18:29.022070+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: cluster 2022-04-22T17:18:29.154199+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: cephadm 2022-04-22T17:18:29.737379+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:29.745961+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:29.888357+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:29.889679+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:29.891694+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:29.892799+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.011785+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: cephadm 2022-04-22T17:18:30.012608+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 16251M 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.018843+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.024000+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.026881+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.028351+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.029235+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:30 smithi121 bash[12679]: audit 2022-04-22T17:18:30.035037+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: cluster 2022-04-22T17:18:29.154199+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: cephadm 2022-04-22T17:18:29.737379+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:29.745961+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:29.888357+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:29.889679+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:29.891694+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:29.892799+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.011785+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: cephadm 2022-04-22T17:18:30.012608+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 16251M 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.018843+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.024000+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.026881+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.028351+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.029235+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[18070]: audit 2022-04-22T17:18:30.035037+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: cluster 2022-04-22T17:18:29.154199+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: cephadm 2022-04-22T17:18:29.737379+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:29.745961+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:29.888357+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:29.889679+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:29.891694+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:29.892799+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.011785+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: cephadm 2022-04-22T17:18:30.012608+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 16251M 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.018843+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.024000+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.026881+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.028351+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:30.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.029235+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:30.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:30 smithi115 bash[12737]: audit 2022-04-22T17:18:30.035037+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:32 smithi121 bash[12679]: cluster 2022-04-22T17:18:31.022591+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:32.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:32 smithi115 bash[12737]: cluster 2022-04-22T17:18:31.022591+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:32.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:32 smithi115 bash[18070]: cluster 2022-04-22T17:18:31.022591+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:33 smithi121 bash[12679]: cluster 2022-04-22T17:18:33.023136+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:33.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:33 smithi115 bash[12737]: cluster 2022-04-22T17:18:33.023136+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:33.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:33 smithi115 bash[18070]: cluster 2022-04-22T17:18:33.023136+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:34 smithi121 bash[12679]: audit 2022-04-22T17:18:34.146250+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]: dispatch 2022-04-22T17:18:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:34 smithi121 bash[12679]: audit 2022-04-22T17:18:34.155760+0000 mon.a (mon.0) 262 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]': finished 2022-04-22T17:18:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:34 smithi121 bash[12679]: cluster 2022-04-22T17:18:34.155954+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2022-04-22T17:18:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:34 smithi121 bash[12679]: audit 2022-04-22T17:18:34.156209+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:34.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[12737]: audit 2022-04-22T17:18:34.146250+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]: dispatch 2022-04-22T17:18:34.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[12737]: audit 2022-04-22T17:18:34.155760+0000 mon.a (mon.0) 262 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]': finished 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[12737]: cluster 2022-04-22T17:18:34.155954+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[12737]: audit 2022-04-22T17:18:34.156209+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[18070]: audit 2022-04-22T17:18:34.146250+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]: dispatch 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[18070]: audit 2022-04-22T17:18:34.155760+0000 mon.a (mon.0) 262 : audit [INF] from='client.? 172.21.15.115:0/1284861527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094"}]': finished 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[18070]: cluster 2022-04-22T17:18:34.155954+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2022-04-22T17:18:34.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:34 smithi115 bash[18070]: audit 2022-04-22T17:18:34.156209+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:35 smithi121 bash[12679]: audit 2022-04-22T17:18:34.817027+0000 mon.a (mon.0) 265 : audit [DBG] from='client.? 172.21.15.115:0/4189226503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:35 smithi121 bash[12679]: cluster 2022-04-22T17:18:35.023746+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:35.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:35 smithi115 bash[12737]: audit 2022-04-22T17:18:34.817027+0000 mon.a (mon.0) 265 : audit [DBG] from='client.? 172.21.15.115:0/4189226503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:35.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:35 smithi115 bash[12737]: cluster 2022-04-22T17:18:35.023746+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:35.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:35 smithi115 bash[18070]: audit 2022-04-22T17:18:34.817027+0000 mon.a (mon.0) 265 : audit [DBG] from='client.? 172.21.15.115:0/4189226503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:18:35.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:35 smithi115 bash[18070]: cluster 2022-04-22T17:18:35.023746+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:37.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:37 smithi115 bash[12737]: cluster 2022-04-22T17:18:37.024244+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:37 smithi115 bash[18070]: cluster 2022-04-22T17:18:37.024244+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:37 smithi121 bash[12679]: cluster 2022-04-22T17:18:37.024244+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:38 smithi121 bash[12679]: audit 2022-04-22T17:18:38.628252+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:18:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:38 smithi121 bash[12679]: audit 2022-04-22T17:18:38.628800+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:18:39.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:38 smithi115 bash[12737]: audit 2022-04-22T17:18:38.628252+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:18:39.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:38 smithi115 bash[12737]: audit 2022-04-22T17:18:38.628800+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:18:39.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:38 smithi115 bash[18070]: audit 2022-04-22T17:18:38.628252+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:18:39.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:38 smithi115 bash[18070]: audit 2022-04-22T17:18:38.628800+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:18:39.867 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:39 smithi115 bash[12737]: cluster 2022-04-22T17:18:39.024843+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:39 smithi121 bash[12679]: cluster 2022-04-22T17:18:39.024843+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:40.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:39 smithi115 bash[18070]: cluster 2022-04-22T17:18:39.024843+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:42 smithi121 bash[12679]: cluster 2022-04-22T17:18:41.025289+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:42.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:42 smithi115 bash[12737]: cluster 2022-04-22T17:18:41.025289+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:42.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:42 smithi115 bash[18070]: cluster 2022-04-22T17:18:41.025289+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:43 smithi121 bash[12679]: audit 2022-04-22T17:18:42.578488+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-04-22T17:18:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:43 smithi121 bash[12679]: audit 2022-04-22T17:18:42.580058+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:43.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:43 smithi115 bash[18070]: audit 2022-04-22T17:18:42.578488+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-04-22T17:18:43.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:43 smithi115 bash[18070]: audit 2022-04-22T17:18:42.580058+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:43.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:43 smithi115 bash[12737]: audit 2022-04-22T17:18:42.578488+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-04-22T17:18:43.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:43 smithi115 bash[12737]: audit 2022-04-22T17:18:42.580058+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:44.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:44 smithi115 bash[12737]: cephadm 2022-04-22T17:18:42.581314+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:18:44.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:44 smithi115 bash[12737]: cluster 2022-04-22T17:18:43.025581+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:44.264 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:44 smithi115 bash[18070]: cephadm 2022-04-22T17:18:42.581314+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:18:44.264 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:44 smithi115 bash[18070]: cluster 2022-04-22T17:18:43.025581+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:44 smithi121 bash[12679]: cephadm 2022-04-22T17:18:42.581314+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:18:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:44 smithi121 bash[12679]: cluster 2022-04-22T17:18:43.025581+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:45.392 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 1 on host 'smithi115' 2022-04-22T17:18:46.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: cluster 2022-04-22T17:18:45.026045+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: audit 2022-04-22T17:18:45.389254+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: audit 2022-04-22T17:18:45.395083+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: audit 2022-04-22T17:18:45.395372+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: audit 2022-04-22T17:18:45.397804+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[12737]: audit 2022-04-22T17:18:45.401051+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: cluster 2022-04-22T17:18:45.026045+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: audit 2022-04-22T17:18:45.389254+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: audit 2022-04-22T17:18:45.395083+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: audit 2022-04-22T17:18:45.395372+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: audit 2022-04-22T17:18:45.397804+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:46 smithi115 bash[18070]: audit 2022-04-22T17:18:45.401051+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: cluster 2022-04-22T17:18:45.026045+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: audit 2022-04-22T17:18:45.389254+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: audit 2022-04-22T17:18:45.395083+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: audit 2022-04-22T17:18:45.395372+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: audit 2022-04-22T17:18:45.397804+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:46 smithi121 bash[12679]: audit 2022-04-22T17:18:45.401051+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:46.568 DEBUG:teuthology.orchestra.run.smithi115:osd.1> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.1.service 2022-04-22T17:18:46.571 INFO:tasks.cephadm:Deploying osd.2 on smithi115 with /dev/vg_nvme/lv_2... 2022-04-22T17:18:46.571 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-04-22T17:18:46.635 INFO:journalctl@ceph.osd.1.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:18:48.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:48 smithi115 bash[12737]: cluster 2022-04-22T17:18:47.026601+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:48.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:48 smithi115 bash[18070]: cluster 2022-04-22T17:18:47.026601+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2022-04-22T17:18:48.231 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-04-22T17:18:48.232 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:47 smithi115 bash[23405]: --> ceph-volume lvm activate successful for osd ID: 1 2022-04-22T17:18:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:48 smithi121 bash[12679]: cluster 2022-04-22T17:18:47.026601+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:49 smithi121 bash[12679]: audit 2022-04-22T17:18:48.768452+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:49 smithi121 bash[12679]: cluster 2022-04-22T17:18:49.027099+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:50.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:49 smithi115 bash[12737]: audit 2022-04-22T17:18:48.768452+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:50.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:49 smithi115 bash[12737]: cluster 2022-04-22T17:18:49.027099+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:50.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:49 smithi115 bash[18070]: audit 2022-04-22T17:18:48.768452+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:50.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:49 smithi115 bash[18070]: cluster 2022-04-22T17:18:49.027099+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:52.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:52 smithi115 bash[12737]: cluster 2022-04-22T17:18:51.027630+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:52.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:52 smithi115 bash[18070]: cluster 2022-04-22T17:18:51.027630+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:52.029 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:51 smithi115 bash[23816]: debug 2022-04-22T17:18:51.826+0000 7fa489d04700 1 -- 172.21.15.115:0/2475683054 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b46de4c4e0 con 0x55b46dec2c00 2022-04-22T17:18:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:52 smithi121 bash[12679]: cluster 2022-04-22T17:18:51.027630+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:52.778 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:52 smithi115 bash[23816]: debug 2022-04-22T17:18:52.370+0000 7fa49179d080 -1 Falling back to public interface 2022-04-22T17:18:53.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:53 smithi115 bash[12737]: cluster 2022-04-22T17:18:53.027953+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:53.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:53 smithi115 bash[18070]: cluster 2022-04-22T17:18:53.027953+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:53 smithi121 bash[12679]: cluster 2022-04-22T17:18:53.027953+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-04-22T17:18:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-04-22T17:18:53.615 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-04-22T17:18:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-04-22T17:18:53.615 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0194691 s, 539 MB/s 2022-04-22T17:18:53.616 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-04-22T17:18:54.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2022-04-22T17:18:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:55 smithi121 bash[12679]: cluster 2022-04-22T17:18:55.028265+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:55.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:55 smithi115 bash[12737]: cluster 2022-04-22T17:18:55.028265+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:55.528 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:55 smithi115 bash[23816]: debug 2022-04-22T17:18:55.362+0000 7fa49179d080 -1 osd.1 0 log_to_monitors {default=true} 2022-04-22T17:18:55.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:55 smithi115 bash[18070]: cluster 2022-04-22T17:18:55.028265+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:56.411 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:56 smithi115 bash[12737]: audit 2022-04-22T17:18:55.368107+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:18:56.412 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:56 smithi115 bash[18070]: audit 2022-04-22T17:18:55.368107+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:18:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:56 smithi121 bash[12679]: audit 2022-04-22T17:18:55.368107+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:57 smithi121 bash[12679]: audit 2022-04-22T17:18:56.137219+0000 mon.a (mon.0) 277 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:57 smithi121 bash[12679]: cluster 2022-04-22T17:18:56.137375+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2022-04-22T17:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:57 smithi121 bash[12679]: audit 2022-04-22T17:18:56.137593+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:57 smithi121 bash[12679]: audit 2022-04-22T17:18:56.138106+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:57 smithi121 bash[12679]: cluster 2022-04-22T17:18:57.028655+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:57.528 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[23816]: debug 2022-04-22T17:18:57.142+0000 7fa487d00700 -1 osd.1 0 waiting for initial osdmap 2022-04-22T17:18:57.528 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[23816]: debug 2022-04-22T17:18:57.146+0000 7fa482478700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:18:57.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[12737]: audit 2022-04-22T17:18:56.137219+0000 mon.a (mon.0) 277 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:18:57.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[12737]: cluster 2022-04-22T17:18:56.137375+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[12737]: audit 2022-04-22T17:18:56.137593+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[12737]: audit 2022-04-22T17:18:56.138106+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[12737]: cluster 2022-04-22T17:18:57.028655+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[18070]: audit 2022-04-22T17:18:56.137219+0000 mon.a (mon.0) 277 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[18070]: cluster 2022-04-22T17:18:56.137375+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[18070]: audit 2022-04-22T17:18:56.137593+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[18070]: audit 2022-04-22T17:18:56.138106+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:18:57.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:57 smithi115 bash[18070]: cluster 2022-04-22T17:18:57.028655+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:58.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[12737]: audit 2022-04-22T17:18:57.136992+0000 mon.a (mon.0) 281 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:58.141 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[12737]: cluster 2022-04-22T17:18:57.137121+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2022-04-22T17:18:58.141 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[12737]: audit 2022-04-22T17:18:57.137768+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:58.141 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[12737]: audit 2022-04-22T17:18:57.145012+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:58 smithi121 bash[12679]: audit 2022-04-22T17:18:57.136992+0000 mon.a (mon.0) 281 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:58 smithi121 bash[12679]: cluster 2022-04-22T17:18:57.137121+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2022-04-22T17:18:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:58 smithi121 bash[12679]: audit 2022-04-22T17:18:57.137768+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:58 smithi121 bash[12679]: audit 2022-04-22T17:18:57.145012+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[18070]: audit 2022-04-22T17:18:57.136992+0000 mon.a (mon.0) 281 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[18070]: cluster 2022-04-22T17:18:57.137121+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2022-04-22T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[18070]: audit 2022-04-22T17:18:57.137768+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:58 smithi115 bash[18070]: audit 2022-04-22T17:18:57.145012+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:59.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cluster 2022-04-22T17:18:56.327139+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:59.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cluster 2022-04-22T17:18:56.327284+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cluster 2022-04-22T17:18:58.146643+0000 mon.a (mon.0) 285 : cluster [INF] osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474] boot 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cluster 2022-04-22T17:18:58.146717+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.147428+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cephadm 2022-04-22T17:18:58.166410+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.174847+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.176424+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.177836+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cephadm 2022-04-22T17:18:58.178653+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 8125M 2022-04-22T17:18:59.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.183662+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.191086+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.193389+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.194872+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.195917+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: audit 2022-04-22T17:18:58.204250+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[18070]: cluster 2022-04-22T17:18:59.029137+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cluster 2022-04-22T17:18:56.327139+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:59.468 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cluster 2022-04-22T17:18:56.327284+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cluster 2022-04-22T17:18:58.146643+0000 mon.a (mon.0) 285 : cluster [INF] osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474] boot 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cluster 2022-04-22T17:18:58.146717+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.147428+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cephadm 2022-04-22T17:18:58.166410+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.174847+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.176424+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.177836+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cephadm 2022-04-22T17:18:58.178653+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 8125M 2022-04-22T17:18:59.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.183662+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.191086+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.193389+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.194872+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.195917+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: audit 2022-04-22T17:18:58.204250+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.470 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:18:59 smithi115 bash[12737]: cluster 2022-04-22T17:18:59.029137+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:18:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cluster 2022-04-22T17:18:56.327139+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cluster 2022-04-22T17:18:56.327284+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cluster 2022-04-22T17:18:58.146643+0000 mon.a (mon.0) 285 : cluster [INF] osd.1 [v2:172.21.15.115:6810/245673474,v1:172.21.15.115:6811/245673474] boot 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cluster 2022-04-22T17:18:58.146717+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.147428+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cephadm 2022-04-22T17:18:58.166410+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.174847+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.176424+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.177836+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:18:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cephadm 2022-04-22T17:18:58.178653+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 8125M 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.183662+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.191086+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.193389+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.194872+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.195917+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: audit 2022-04-22T17:18:58.204250+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:18:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:18:59 smithi121 bash[12679]: cluster 2022-04-22T17:18:59.029137+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-04-22T17:19:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:00 smithi121 bash[12679]: cluster 2022-04-22T17:18:59.218057+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2022-04-22T17:19:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:00 smithi121 bash[12679]: audit 2022-04-22T17:18:59.275712+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:00 smithi121 bash[12679]: audit 2022-04-22T17:18:59.277077+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:00 smithi121 bash[12679]: audit 2022-04-22T17:18:59.279035+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:00 smithi121 bash[12679]: audit 2022-04-22T17:18:59.280011+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:00.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[12737]: cluster 2022-04-22T17:18:59.218057+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2022-04-22T17:19:00.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[12737]: audit 2022-04-22T17:18:59.275712+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:00.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[12737]: audit 2022-04-22T17:18:59.277077+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:00.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[12737]: audit 2022-04-22T17:18:59.279035+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[12737]: audit 2022-04-22T17:18:59.280011+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[18070]: cluster 2022-04-22T17:18:59.218057+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[18070]: audit 2022-04-22T17:18:59.275712+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[18070]: audit 2022-04-22T17:18:59.277077+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[18070]: audit 2022-04-22T17:18:59.279035+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:00.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:00 smithi115 bash[18070]: audit 2022-04-22T17:18:59.280011+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:02.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:02 smithi115 bash[12737]: cluster 2022-04-22T17:19:01.029518+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:02.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:02 smithi115 bash[18070]: cluster 2022-04-22T17:19:01.029518+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:02 smithi121 bash[12679]: cluster 2022-04-22T17:19:01.029518+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:03 smithi121 bash[12679]: cluster 2022-04-22T17:19:03.029928+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:03.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:03 smithi115 bash[12737]: cluster 2022-04-22T17:19:03.029928+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:03.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:03 smithi115 bash[18070]: cluster 2022-04-22T17:19:03.029928+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:04 smithi121 bash[12679]: audit 2022-04-22T17:19:03.507039+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.115:0/1743119205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:04 smithi121 bash[12679]: audit 2022-04-22T17:19:03.507872+0000 mon.a (mon.0) 301 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:04 smithi121 bash[12679]: audit 2022-04-22T17:19:03.518650+0000 mon.a (mon.0) 302 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]': finished 2022-04-22T17:19:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:04 smithi121 bash[12679]: cluster 2022-04-22T17:19:03.518866+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2022-04-22T17:19:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:04 smithi121 bash[12679]: audit 2022-04-22T17:19:03.519074+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:04.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[12737]: audit 2022-04-22T17:19:03.507039+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.115:0/1743119205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[12737]: audit 2022-04-22T17:19:03.507872+0000 mon.a (mon.0) 301 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[12737]: audit 2022-04-22T17:19:03.518650+0000 mon.a (mon.0) 302 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]': finished 2022-04-22T17:19:04.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[12737]: cluster 2022-04-22T17:19:03.518866+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2022-04-22T17:19:04.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[12737]: audit 2022-04-22T17:19:03.519074+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:04.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[18070]: audit 2022-04-22T17:19:03.507039+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.115:0/1743119205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[18070]: audit 2022-04-22T17:19:03.507872+0000 mon.a (mon.0) 301 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]: dispatch 2022-04-22T17:19:04.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[18070]: audit 2022-04-22T17:19:03.518650+0000 mon.a (mon.0) 302 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3aecb00-afa4-4473-abf6-aba50f8b2b3d"}]': finished 2022-04-22T17:19:04.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[18070]: cluster 2022-04-22T17:19:03.518866+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2022-04-22T17:19:04.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:04 smithi115 bash[18070]: audit 2022-04-22T17:19:03.519074+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:05 smithi121 bash[12679]: audit 2022-04-22T17:19:04.196388+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.115:0/2369266546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:05.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:05 smithi121 bash[12679]: cluster 2022-04-22T17:19:05.030452+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:05.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:05 smithi115 bash[12737]: audit 2022-04-22T17:19:04.196388+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.115:0/2369266546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:05.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:05 smithi115 bash[12737]: cluster 2022-04-22T17:19:05.030452+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:05 smithi115 bash[18070]: audit 2022-04-22T17:19:04.196388+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.115:0/2369266546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:05 smithi115 bash[18070]: cluster 2022-04-22T17:19:05.030452+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:07 smithi121 bash[12679]: cluster 2022-04-22T17:19:07.030965+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:07.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:07 smithi115 bash[12737]: cluster 2022-04-22T17:19:07.030965+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:07.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:07 smithi115 bash[18070]: cluster 2022-04-22T17:19:07.030965+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:09 smithi121 bash[12679]: cluster 2022-04-22T17:19:09.031412+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:09.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:09 smithi115 bash[12737]: cluster 2022-04-22T17:19:09.031412+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:09.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:09 smithi115 bash[18070]: cluster 2022-04-22T17:19:09.031412+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:12.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:12 smithi115 bash[12737]: cluster 2022-04-22T17:19:11.031907+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:12.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:12 smithi115 bash[18070]: cluster 2022-04-22T17:19:11.031907+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:12.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:12 smithi121 bash[12679]: cluster 2022-04-22T17:19:11.031907+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:13.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[18070]: audit 2022-04-22T17:19:12.053984+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-04-22T17:19:13.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[18070]: audit 2022-04-22T17:19:12.055587+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:13.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[18070]: cephadm 2022-04-22T17:19:12.057015+0000 mgr.y (mgr.14146) 109 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:19:13.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[12737]: audit 2022-04-22T17:19:12.053984+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-04-22T17:19:13.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[12737]: audit 2022-04-22T17:19:12.055587+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:13.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:13 smithi115 bash[12737]: cephadm 2022-04-22T17:19:12.057015+0000 mgr.y (mgr.14146) 109 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:19:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:13 smithi121 bash[12679]: audit 2022-04-22T17:19:12.053984+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-04-22T17:19:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:13 smithi121 bash[12679]: audit 2022-04-22T17:19:12.055587+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:13 smithi121 bash[12679]: cephadm 2022-04-22T17:19:12.057015+0000 mgr.y (mgr.14146) 109 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:19:14.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:14 smithi115 bash[12737]: cluster 2022-04-22T17:19:13.032431+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:14.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:14 smithi115 bash[18070]: cluster 2022-04-22T17:19:13.032431+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:14 smithi121 bash[12679]: cluster 2022-04-22T17:19:13.032431+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:14.915 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 2 on host 'smithi115' 2022-04-22T17:19:15.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[12737]: audit 2022-04-22T17:19:14.907103+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:15.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[12737]: audit 2022-04-22T17:19:14.910122+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[12737]: audit 2022-04-22T17:19:14.916053+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[12737]: audit 2022-04-22T17:19:14.916335+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[12737]: audit 2022-04-22T17:19:14.923839+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[18070]: audit 2022-04-22T17:19:14.907103+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[18070]: audit 2022-04-22T17:19:14.910122+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[18070]: audit 2022-04-22T17:19:14.916053+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[18070]: audit 2022-04-22T17:19:14.916335+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:15 smithi115 bash[18070]: audit 2022-04-22T17:19:14.923839+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:15 smithi121 bash[12679]: audit 2022-04-22T17:19:14.907103+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:15 smithi121 bash[12679]: audit 2022-04-22T17:19:14.910122+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:15 smithi121 bash[12679]: audit 2022-04-22T17:19:14.916053+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:15 smithi121 bash[12679]: audit 2022-04-22T17:19:14.916335+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:15 smithi121 bash[12679]: audit 2022-04-22T17:19:14.923839+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:16.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:16 smithi115 bash[18070]: cluster 2022-04-22T17:19:15.032953+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:16.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:16 smithi115 bash[12737]: cluster 2022-04-22T17:19:15.032953+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:16.337 DEBUG:teuthology.orchestra.run.smithi115:osd.2> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.2.service 2022-04-22T17:19:16.339 INFO:tasks.cephadm:Deploying osd.3 on smithi115 with /dev/vg_nvme/lv_1... 2022-04-22T17:19:16.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-04-22T17:19:16.387 INFO:journalctl@ceph.osd.2.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:19:16.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:16 smithi121 bash[12679]: cluster 2022-04-22T17:19:15.032953+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:17.972 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-04-22T17:19:17.972 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2022-04-22T17:19:17.973 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2022-04-22T17:19:17.973 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2022-04-22T17:19:17.973 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2022-04-22T17:19:17.973 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-04-22T17:19:17.973 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:17 smithi115 bash[26248]: --> ceph-volume lvm activate successful for osd ID: 2 2022-04-22T17:19:18.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:18 smithi115 bash[12737]: cluster 2022-04-22T17:19:17.033366+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:18.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:18 smithi115 bash[18070]: cluster 2022-04-22T17:19:17.033366+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:18 smithi121 bash[12679]: cluster 2022-04-22T17:19:17.033366+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:19 smithi121 bash[12679]: audit 2022-04-22T17:19:18.463031+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:19 smithi121 bash[12679]: cluster 2022-04-22T17:19:19.033692+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:19.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:19 smithi115 bash[12737]: audit 2022-04-22T17:19:18.463031+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:19.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:19 smithi115 bash[12737]: cluster 2022-04-22T17:19:19.033692+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:19.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:19 smithi115 bash[18070]: audit 2022-04-22T17:19:18.463031+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:19.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:19 smithi115 bash[18070]: cluster 2022-04-22T17:19:19.033692+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:22.277 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:22 smithi115 bash[26675]: debug 2022-04-22T17:19:22.082+0000 7faa2d687080 -1 Falling back to public interface 2022-04-22T17:19:22.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:22 smithi115 bash[12737]: cluster 2022-04-22T17:19:21.034021+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:22.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:22 smithi115 bash[18070]: cluster 2022-04-22T17:19:21.034021+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:22 smithi121 bash[12679]: cluster 2022-04-22T17:19:21.034021+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0171614 s, 611 MB/s 2022-04-22T17:19:22.614 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-04-22T17:19:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:23 smithi121 bash[12679]: cluster 2022-04-22T17:19:23.034591+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:23 smithi115 bash[12737]: cluster 2022-04-22T17:19:23.034591+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:23 smithi115 bash[18070]: cluster 2022-04-22T17:19:23.034591+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:23.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2022-04-22T17:19:25.423 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:25 smithi115 bash[12737]: cluster 2022-04-22T17:19:25.035241+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:25.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:25 smithi115 bash[18070]: cluster 2022-04-22T17:19:25.035241+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:25.424 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:25 smithi115 bash[26675]: debug 2022-04-22T17:19:25.122+0000 7faa2d687080 -1 osd.2 0 log_to_monitors {default=true} 2022-04-22T17:19:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:25 smithi121 bash[12679]: cluster 2022-04-22T17:19:25.035241+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:26.389 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:26 smithi115 bash[12737]: audit 2022-04-22T17:19:25.132126+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:19:26.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:26 smithi115 bash[18070]: audit 2022-04-22T17:19:25.132126+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:19:26.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:26 smithi121 bash[12679]: audit 2022-04-22T17:19:25.132126+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:19:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:27 smithi121 bash[12679]: audit 2022-04-22T17:19:26.145395+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:19:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:27 smithi121 bash[12679]: cluster 2022-04-22T17:19:26.145485+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2022-04-22T17:19:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:27 smithi121 bash[12679]: audit 2022-04-22T17:19:26.145627+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:27 smithi121 bash[12679]: audit 2022-04-22T17:19:26.146035+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:27 smithi121 bash[12679]: cluster 2022-04-22T17:19:27.035738+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:27.527 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[26675]: debug 2022-04-22T17:19:27.154+0000 7faa23bea700 -1 osd.2 0 waiting for initial osdmap 2022-04-22T17:19:27.527 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[26675]: debug 2022-04-22T17:19:27.158+0000 7faa1db61700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[12737]: audit 2022-04-22T17:19:26.145395+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[12737]: cluster 2022-04-22T17:19:26.145485+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[12737]: audit 2022-04-22T17:19:26.145627+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[12737]: audit 2022-04-22T17:19:26.146035+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[12737]: cluster 2022-04-22T17:19:27.035738+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[18070]: audit 2022-04-22T17:19:26.145395+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:19:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[18070]: cluster 2022-04-22T17:19:26.145485+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2022-04-22T17:19:27.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[18070]: audit 2022-04-22T17:19:26.145627+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:27.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[18070]: audit 2022-04-22T17:19:26.146035+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:27.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:27 smithi115 bash[18070]: cluster 2022-04-22T17:19:27.035738+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: cluster 2022-04-22T17:19:26.121998+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: cluster 2022-04-22T17:19:26.122123+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.150338+0000 mon.a (mon.0) 318 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: cluster 2022-04-22T17:19:27.150522+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2022-04-22T17:19:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.151044+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.159009+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: cephadm 2022-04-22T17:19:27.300812+0000 mgr.y (mgr.14146) 118 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.308000+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.309103+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.310197+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.311203+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: cephadm 2022-04-22T17:19:27.311791+0000 mgr.y (mgr.14146) 119 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 5417M 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.316315+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.323566+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.325587+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.326970+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.327994+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:28 smithi121 bash[12679]: audit 2022-04-22T17:19:27.581691+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: cluster 2022-04-22T17:19:26.121998+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: cluster 2022-04-22T17:19:26.122123+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.150338+0000 mon.a (mon.0) 318 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: cluster 2022-04-22T17:19:27.150522+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.151044+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.159009+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: cephadm 2022-04-22T17:19:27.300812+0000 mgr.y (mgr.14146) 118 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.308000+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.309103+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.310197+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.311203+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: cephadm 2022-04-22T17:19:27.311791+0000 mgr.y (mgr.14146) 119 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 5417M 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.316315+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.323566+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.325587+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.326970+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.327994+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:29.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[18070]: audit 2022-04-22T17:19:27.581691+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: cluster 2022-04-22T17:19:26.121998+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: cluster 2022-04-22T17:19:26.122123+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.150338+0000 mon.a (mon.0) 318 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: cluster 2022-04-22T17:19:27.150522+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.151044+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.159009+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: cephadm 2022-04-22T17:19:27.300812+0000 mgr.y (mgr.14146) 118 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.308000+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.309103+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.310197+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.311203+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: cephadm 2022-04-22T17:19:27.311791+0000 mgr.y (mgr.14146) 119 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 5417M 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.316315+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.323566+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.325587+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.326970+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.327994+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:29.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:28 smithi115 bash[12737]: audit 2022-04-22T17:19:27.581691+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:28.156590+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: cluster 2022-04-22T17:19:28.164850+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147] boot 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: cluster 2022-04-22T17:19:28.164900+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:28.165882+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:28.166351+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:28.828922+0000 mgr.y (mgr.14146) 120 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: cluster 2022-04-22T17:19:29.036247+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:29.166867+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: cluster 2022-04-22T17:19:29.166986+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2022-04-22T17:19:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:29 smithi121 bash[12679]: audit 2022-04-22T17:19:29.168536+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2022-04-22T17:19:30.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:28.156590+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: cluster 2022-04-22T17:19:28.164850+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147] boot 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: cluster 2022-04-22T17:19:28.164900+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:28.165882+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:28.166351+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:28.828922+0000 mgr.y (mgr.14146) 120 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: cluster 2022-04-22T17:19:29.036247+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:29.166867+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: cluster 2022-04-22T17:19:29.166986+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2022-04-22T17:19:30.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[12737]: audit 2022-04-22T17:19:29.168536+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2022-04-22T17:19:30.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:28.156590+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: cluster 2022-04-22T17:19:28.164850+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 [v2:172.21.15.115:6818/2968343147,v1:172.21.15.115:6819/2968343147] boot 2022-04-22T17:19:30.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: cluster 2022-04-22T17:19:28.164900+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2022-04-22T17:19:30.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:28.165882+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:19:30.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:28.166351+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2022-04-22T17:19:30.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:28.828922+0000 mgr.y (mgr.14146) 120 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:30.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: cluster 2022-04-22T17:19:29.036247+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-04-22T17:19:30.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:29.166867+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2022-04-22T17:19:30.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: cluster 2022-04-22T17:19:29.166986+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2022-04-22T17:19:30.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:29 smithi115 bash[18070]: audit 2022-04-22T17:19:29.168536+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2022-04-22T17:19:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: audit 2022-04-22T17:19:30.167198+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2022-04-22T17:19:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: cluster 2022-04-22T17:19:30.167456+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2022-04-22T17:19:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: audit 2022-04-22T17:19:30.177866+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: audit 2022-04-22T17:19:30.179805+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: audit 2022-04-22T17:19:30.180845+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:31 smithi121 bash[12679]: cluster 2022-04-22T17:19:31.036840+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:31.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: audit 2022-04-22T17:19:30.167198+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2022-04-22T17:19:31.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: cluster 2022-04-22T17:19:30.167456+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: audit 2022-04-22T17:19:30.177866+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: audit 2022-04-22T17:19:30.179805+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: audit 2022-04-22T17:19:30.180845+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[12737]: cluster 2022-04-22T17:19:31.036840+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: audit 2022-04-22T17:19:30.167198+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: cluster 2022-04-22T17:19:30.167456+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: audit 2022-04-22T17:19:30.177866+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: audit 2022-04-22T17:19:30.179805+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: audit 2022-04-22T17:19:30.180845+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:31 smithi115 bash[18070]: cluster 2022-04-22T17:19:31.036840+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:32.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:32 smithi115 bash[12737]: cluster 2022-04-22T17:19:31.168595+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2022-04-22T17:19:32.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:32 smithi115 bash[18070]: cluster 2022-04-22T17:19:31.168595+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2022-04-22T17:19:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:32 smithi121 bash[12679]: cluster 2022-04-22T17:19:31.168595+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2022-04-22T17:19:33.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:33 smithi121 bash[12679]: cluster 2022-04-22T17:19:33.037395+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v99: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:33.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:33 smithi115 bash[12737]: cluster 2022-04-22T17:19:33.037395+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v99: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:33.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:33 smithi115 bash[18070]: cluster 2022-04-22T17:19:33.037395+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v99: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:34 smithi121 bash[12679]: audit 2022-04-22T17:19:34.784699+0000 mon.a (mon.0) 346 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]: dispatch 2022-04-22T17:19:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:34 smithi121 bash[12679]: audit 2022-04-22T17:19:34.791042+0000 mon.a (mon.0) 347 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]': finished 2022-04-22T17:19:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:34 smithi121 bash[12679]: cluster 2022-04-22T17:19:34.791103+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2022-04-22T17:19:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:34 smithi121 bash[12679]: audit 2022-04-22T17:19:34.791204+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:35.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[12737]: audit 2022-04-22T17:19:34.784699+0000 mon.a (mon.0) 346 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]: dispatch 2022-04-22T17:19:35.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[12737]: audit 2022-04-22T17:19:34.791042+0000 mon.a (mon.0) 347 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]': finished 2022-04-22T17:19:35.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[12737]: cluster 2022-04-22T17:19:34.791103+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2022-04-22T17:19:35.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[12737]: audit 2022-04-22T17:19:34.791204+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:35.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[18070]: audit 2022-04-22T17:19:34.784699+0000 mon.a (mon.0) 346 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]: dispatch 2022-04-22T17:19:35.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[18070]: audit 2022-04-22T17:19:34.791042+0000 mon.a (mon.0) 347 : audit [INF] from='client.? 172.21.15.115:0/1590262925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96c0386b-e1ba-44a5-8daf-f76b1303b81d"}]': finished 2022-04-22T17:19:35.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[18070]: cluster 2022-04-22T17:19:34.791103+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2022-04-22T17:19:35.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:34 smithi115 bash[18070]: audit 2022-04-22T17:19:34.791204+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:35 smithi121 bash[12679]: cluster 2022-04-22T17:19:35.037871+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:35 smithi121 bash[12679]: audit 2022-04-22T17:19:35.452416+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/1612600601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:36.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:35 smithi115 bash[12737]: cluster 2022-04-22T17:19:35.037871+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:36.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:35 smithi115 bash[12737]: audit 2022-04-22T17:19:35.452416+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/1612600601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:36.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:35 smithi115 bash[18070]: cluster 2022-04-22T17:19:35.037871+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:36.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:35 smithi115 bash[18070]: audit 2022-04-22T17:19:35.452416+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/1612600601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:19:37.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:37 smithi121 bash[12679]: cluster 2022-04-22T17:19:37.038382+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:37.630 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:37 smithi115 bash[12737]: cluster 2022-04-22T17:19:37.038382+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:37.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:37 smithi115 bash[18070]: cluster 2022-04-22T17:19:37.038382+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:38 smithi121 bash[12679]: audit 2022-04-22T17:19:38.630214+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:19:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:38 smithi121 bash[12679]: audit 2022-04-22T17:19:38.630850+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:19:39.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:38 smithi115 bash[12737]: audit 2022-04-22T17:19:38.630214+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:19:39.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:38 smithi115 bash[12737]: audit 2022-04-22T17:19:38.630850+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:19:39.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:38 smithi115 bash[18070]: audit 2022-04-22T17:19:38.630214+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:19:39.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:38 smithi115 bash[18070]: audit 2022-04-22T17:19:38.630850+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:19:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:39 smithi121 bash[12679]: cluster 2022-04-22T17:19:39.038936+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:40.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:39 smithi115 bash[12737]: cluster 2022-04-22T17:19:39.038936+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:40.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:39 smithi115 bash[18070]: cluster 2022-04-22T17:19:39.038936+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:42.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:42 smithi115 bash[12737]: cluster 2022-04-22T17:19:41.039497+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:42.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:42 smithi115 bash[18070]: cluster 2022-04-22T17:19:41.039497+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:42 smithi121 bash[12679]: cluster 2022-04-22T17:19:41.039497+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:43.332 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:43 smithi115 bash[12737]: cluster 2022-04-22T17:19:43.039919+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:43.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:43 smithi115 bash[18070]: cluster 2022-04-22T17:19:43.039919+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:43.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:43 smithi121 bash[12679]: cluster 2022-04-22T17:19:43.039919+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:44.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[12737]: audit 2022-04-22T17:19:43.659181+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-04-22T17:19:44.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[12737]: audit 2022-04-22T17:19:43.660922+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:44.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[12737]: cephadm 2022-04-22T17:19:43.662225+0000 mgr.y (mgr.14146) 129 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:19:44.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[18070]: audit 2022-04-22T17:19:43.659181+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-04-22T17:19:44.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[18070]: audit 2022-04-22T17:19:43.660922+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:44.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:44 smithi115 bash[18070]: cephadm 2022-04-22T17:19:43.662225+0000 mgr.y (mgr.14146) 129 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:19:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:44 smithi121 bash[12679]: audit 2022-04-22T17:19:43.659181+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-04-22T17:19:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:44 smithi121 bash[12679]: audit 2022-04-22T17:19:43.660922+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:44 smithi121 bash[12679]: cephadm 2022-04-22T17:19:43.662225+0000 mgr.y (mgr.14146) 129 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:19:45.407 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:45 smithi115 bash[12737]: cluster 2022-04-22T17:19:45.040366+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:45.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:45 smithi115 bash[18070]: cluster 2022-04-22T17:19:45.040366+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:45 smithi121 bash[12679]: cluster 2022-04-22T17:19:45.040366+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:46.585 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 3 on host 'smithi115' 2022-04-22T17:19:47.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[12737]: audit 2022-04-22T17:19:46.579760+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[12737]: audit 2022-04-22T17:19:46.582097+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[12737]: audit 2022-04-22T17:19:46.586764+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[12737]: audit 2022-04-22T17:19:46.587135+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[12737]: audit 2022-04-22T17:19:46.594992+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:47.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[18070]: audit 2022-04-22T17:19:46.579760+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:47.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[18070]: audit 2022-04-22T17:19:46.582097+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:47.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[18070]: audit 2022-04-22T17:19:46.586764+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:47.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[18070]: audit 2022-04-22T17:19:46.587135+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:47.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:47 smithi115 bash[18070]: audit 2022-04-22T17:19:46.594992+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:47.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:47 smithi121 bash[12679]: audit 2022-04-22T17:19:46.579760+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:47 smithi121 bash[12679]: audit 2022-04-22T17:19:46.582097+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:47 smithi121 bash[12679]: audit 2022-04-22T17:19:46.586764+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:47 smithi121 bash[12679]: audit 2022-04-22T17:19:46.587135+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:47 smithi121 bash[12679]: audit 2022-04-22T17:19:46.594992+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:48.111 DEBUG:teuthology.orchestra.run.smithi115:osd.3> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.3.service 2022-04-22T17:19:48.155 INFO:tasks.cephadm:Deploying osd.4 on smithi121 with /dev/vg_nvme/lv_4... 2022-04-22T17:19:48.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-04-22T17:19:48.178 INFO:journalctl@ceph.osd.3.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:19:48.334 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[12737]: cluster 2022-04-22T17:19:47.040683+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:48.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[18070]: cluster 2022-04-22T17:19:47.040683+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:48.371 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:48 smithi121 bash[12679]: cluster 2022-04-22T17:19:47.040683+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2022-04-22T17:19:49.031 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-04-22T17:19:49.032 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:48 smithi115 bash[29112]: --> ceph-volume lvm activate successful for osd ID: 3 2022-04-22T17:19:49.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:49 smithi115 bash[18070]: cluster 2022-04-22T17:19:49.041178+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:49.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:49 smithi115 bash[12737]: cluster 2022-04-22T17:19:49.041178+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:49.401 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:49 smithi121 bash[12679]: cluster 2022-04-22T17:19:49.041178+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:51.027 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:50 smithi115 bash[29268]: debug 2022-04-22T17:19:50.874+0000 7ff81e0da700 1 -- 172.21.15.115:0/527895535 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de044644e0 con 0x55de044dac00 2022-04-22T17:19:51.777 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:51 smithi115 bash[29268]: debug 2022-04-22T17:19:51.422+0000 7ff825b73080 -1 Falling back to public interface 2022-04-22T17:19:51.796 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-04-22T17:19:51.797 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-04-22T17:19:51.797 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2022-04-22T17:19:51.797 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-04-22T17:19:51.797 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0166122 s, 631 MB/s 2022-04-22T17:19:51.797 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2022-04-22T17:19:52.233 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:51 smithi115 bash[12737]: audit 2022-04-22T17:19:50.948965+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:52.233 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:51 smithi115 bash[12737]: cluster 2022-04-22T17:19:51.041664+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:52.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:51 smithi115 bash[18070]: audit 2022-04-22T17:19:50.948965+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:52.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:51 smithi115 bash[18070]: cluster 2022-04-22T17:19:51.041664+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:51 smithi121 bash[12679]: audit 2022-04-22T17:19:50.948965+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:51 smithi121 bash[12679]: cluster 2022-04-22T17:19:51.041664+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:52.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2022-04-22T17:19:53.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:53 smithi121 bash[12679]: cluster 2022-04-22T17:19:53.042198+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:53.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:53 smithi115 bash[12737]: cluster 2022-04-22T17:19:53.042198+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:53.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:53 smithi115 bash[18070]: cluster 2022-04-22T17:19:53.042198+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:54 smithi121 bash[12679]: audit 2022-04-22T17:19:54.442751+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:19:54.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:54 smithi115 bash[18070]: audit 2022-04-22T17:19:54.442751+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:19:54.777 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:54 smithi115 bash[29268]: debug 2022-04-22T17:19:54.434+0000 7ff825b73080 -1 osd.3 0 log_to_monitors {default=true} 2022-04-22T17:19:54.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:54 smithi115 bash[12737]: audit 2022-04-22T17:19:54.442751+0000 mon.a (mon.0) 360 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:19:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: audit 2022-04-22T17:19:54.503094+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: cluster 2022-04-22T17:19:54.503239+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: audit 2022-04-22T17:19:54.504061+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: audit 2022-04-22T17:19:54.505895+0000 mon.a (mon.0) 364 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: cluster 2022-04-22T17:19:55.042811+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: audit 2022-04-22T17:19:55.501725+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: cluster 2022-04-22T17:19:55.501806+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2022-04-22T17:19:55.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:55 smithi121 bash[12679]: audit 2022-04-22T17:19:55.501999+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:55.790 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[29268]: debug 2022-04-22T17:19:55.502+0000 7ff81c0d6700 -1 osd.3 0 waiting for initial osdmap 2022-04-22T17:19:55.790 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[29268]: debug 2022-04-22T17:19:55.510+0000 7ff81704f700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: audit 2022-04-22T17:19:54.503094+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: cluster 2022-04-22T17:19:54.503239+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: audit 2022-04-22T17:19:54.504061+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: audit 2022-04-22T17:19:54.505895+0000 mon.a (mon.0) 364 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: cluster 2022-04-22T17:19:55.042811+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: audit 2022-04-22T17:19:55.501725+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: cluster 2022-04-22T17:19:55.501806+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2022-04-22T17:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[12737]: audit 2022-04-22T17:19:55.501999+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: audit 2022-04-22T17:19:54.503094+0000 mon.a (mon.0) 361 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: cluster 2022-04-22T17:19:54.503239+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: audit 2022-04-22T17:19:54.504061+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: audit 2022-04-22T17:19:54.505895+0000 mon.a (mon.0) 364 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: cluster 2022-04-22T17:19:55.042811+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: audit 2022-04-22T17:19:55.501725+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: cluster 2022-04-22T17:19:55.501806+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2022-04-22T17:19:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:55 smithi115 bash[18070]: audit 2022-04-22T17:19:55.501999+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:56.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:55.506927+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:56.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:55.924159+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:56.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:55.925656+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:56.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:55.929352+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:56.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:55.930879+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[18070]: audit 2022-04-22T17:19:56.507115+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:55.506927+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:55.924159+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:55.925656+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:55.929352+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:55.930879+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:56.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:56 smithi115 bash[12737]: audit 2022-04-22T17:19:56.507115+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:55.506927+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:55.924159+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:19:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:55.925656+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:19:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:55.929352+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:19:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:55.930879+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:56 smithi121 bash[12679]: audit 2022-04-22T17:19:56.507115+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:58.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cluster 2022-04-22T17:19:55.465226+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:58.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cluster 2022-04-22T17:19:55.465439+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:58.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cluster 2022-04-22T17:19:56.531802+0000 mon.a (mon.0) 373 : cluster [INF] osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144] boot 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cluster 2022-04-22T17:19:56.531859+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:56.532695+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cephadm 2022-04-22T17:19:56.841629+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:56.852653+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: cluster 2022-04-22T17:19:57.043346+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.118574+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.120172+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.121600+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.122775+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.132853+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.140416+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.142671+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.144405+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.145338+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:57 smithi121 bash[12679]: audit 2022-04-22T17:19:57.153068+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cluster 2022-04-22T17:19:55.465226+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:58.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cluster 2022-04-22T17:19:55.465439+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cluster 2022-04-22T17:19:56.531802+0000 mon.a (mon.0) 373 : cluster [INF] osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144] boot 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cluster 2022-04-22T17:19:56.531859+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:56.532695+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cephadm 2022-04-22T17:19:56.841629+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:56.852653+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: cluster 2022-04-22T17:19:57.043346+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.118574+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.120172+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.121600+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.122775+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.132853+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.140416+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.142671+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.144405+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.145338+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[12737]: audit 2022-04-22T17:19:57.153068+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cluster 2022-04-22T17:19:55.465226+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:19:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cluster 2022-04-22T17:19:55.465439+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cluster 2022-04-22T17:19:56.531802+0000 mon.a (mon.0) 373 : cluster [INF] osd.3 [v2:172.21.15.115:6826/543187144,v1:172.21.15.115:6827/543187144] boot 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cluster 2022-04-22T17:19:56.531859+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:56.532695+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cephadm 2022-04-22T17:19:56.841629+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:56.852653+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: cluster 2022-04-22T17:19:57.043346+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.118574+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.120172+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.121600+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.122775+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.132853+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.140416+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.142671+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.144405+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.145338+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:19:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:57 smithi115 bash[18070]: audit 2022-04-22T17:19:57.153068+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:58 smithi121 bash[12679]: cephadm 2022-04-22T17:19:57.123659+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 4062M 2022-04-22T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:58 smithi121 bash[12679]: cluster 2022-04-22T17:19:58.164515+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2022-04-22T17:19:59.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:58 smithi115 bash[12737]: cephadm 2022-04-22T17:19:57.123659+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 4062M 2022-04-22T17:19:59.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:58 smithi115 bash[12737]: cluster 2022-04-22T17:19:58.164515+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2022-04-22T17:19:59.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:58 smithi115 bash[18070]: cephadm 2022-04-22T17:19:57.123659+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 4062M 2022-04-22T17:19:59.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:58 smithi115 bash[18070]: cluster 2022-04-22T17:19:58.164515+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2022-04-22T17:20:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:19:59 smithi121 bash[12679]: cluster 2022-04-22T17:19:59.043910+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:00.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:19:59 smithi115 bash[12737]: cluster 2022-04-22T17:19:59.043910+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:00.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:19:59 smithi115 bash[18070]: cluster 2022-04-22T17:19:59.043910+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: audit 2022-04-22T17:19:59.914458+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.121:0/1802458244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: audit 2022-04-22T17:19:59.925438+0000 mon.a (mon.0) 388 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: audit 2022-04-22T17:19:59.932201+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]': finished 2022-04-22T17:20:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: cluster 2022-04-22T17:19:59.932315+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2022-04-22T17:20:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: audit 2022-04-22T17:19:59.932502+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: cluster 2022-04-22T17:20:00.000175+0000 mon.a (mon.0) 392 : cluster [INF] overall HEALTH_OK 2022-04-22T17:20:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:00 smithi121 bash[12679]: audit 2022-04-22T17:20:00.569011+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.121:0/2336144673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: audit 2022-04-22T17:19:59.914458+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.121:0/1802458244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: audit 2022-04-22T17:19:59.925438+0000 mon.a (mon.0) 388 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: audit 2022-04-22T17:19:59.932201+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]': finished 2022-04-22T17:20:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: cluster 2022-04-22T17:19:59.932315+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2022-04-22T17:20:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: audit 2022-04-22T17:19:59.932502+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: cluster 2022-04-22T17:20:00.000175+0000 mon.a (mon.0) 392 : cluster [INF] overall HEALTH_OK 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[18070]: audit 2022-04-22T17:20:00.569011+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.121:0/2336144673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: audit 2022-04-22T17:19:59.914458+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.121:0/1802458244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: audit 2022-04-22T17:19:59.925438+0000 mon.a (mon.0) 388 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]: dispatch 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: audit 2022-04-22T17:19:59.932201+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0f6ab0c-14ee-4800-86bc-daa9f68b116c"}]': finished 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: cluster 2022-04-22T17:19:59.932315+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: audit 2022-04-22T17:19:59.932502+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: cluster 2022-04-22T17:20:00.000175+0000 mon.a (mon.0) 392 : cluster [INF] overall HEALTH_OK 2022-04-22T17:20:01.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:00 smithi115 bash[12737]: audit 2022-04-22T17:20:00.569011+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.121:0/2336144673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:02 smithi121 bash[12679]: cluster 2022-04-22T17:20:01.044360+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:02.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:02 smithi115 bash[12737]: cluster 2022-04-22T17:20:01.044360+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:02.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:02 smithi115 bash[18070]: cluster 2022-04-22T17:20:01.044360+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:03.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:03 smithi121 bash[12679]: cluster 2022-04-22T17:20:03.044863+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:03.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:03 smithi115 bash[12737]: cluster 2022-04-22T17:20:03.044863+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:03.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:03 smithi115 bash[18070]: cluster 2022-04-22T17:20:03.044863+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:05.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:05 smithi121 bash[12679]: cluster 2022-04-22T17:20:05.045417+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:05.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:05 smithi115 bash[12737]: cluster 2022-04-22T17:20:05.045417+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:05.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:05 smithi115 bash[18070]: cluster 2022-04-22T17:20:05.045417+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:07.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:07 smithi121 bash[12679]: cluster 2022-04-22T17:20:07.045951+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:07.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:07 smithi115 bash[12737]: cluster 2022-04-22T17:20:07.045951+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:07.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:07 smithi115 bash[18070]: cluster 2022-04-22T17:20:07.045951+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:08 smithi121 bash[12679]: audit 2022-04-22T17:20:07.762609+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-04-22T17:20:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:08 smithi121 bash[12679]: audit 2022-04-22T17:20:07.764435+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:08 smithi121 bash[12679]: cephadm 2022-04-22T17:20:07.765870+0000 mgr.y (mgr.14146) 145 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[12737]: audit 2022-04-22T17:20:07.762609+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[12737]: audit 2022-04-22T17:20:07.764435+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[12737]: cephadm 2022-04-22T17:20:07.765870+0000 mgr.y (mgr.14146) 145 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[18070]: audit 2022-04-22T17:20:07.762609+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[18070]: audit 2022-04-22T17:20:07.764435+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:08.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:08 smithi115 bash[18070]: cephadm 2022-04-22T17:20:07.765870+0000 mgr.y (mgr.14146) 145 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:20:09.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:09 smithi121 bash[12679]: cluster 2022-04-22T17:20:09.046508+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:09.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:09 smithi115 bash[12737]: cluster 2022-04-22T17:20:09.046508+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:09.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:09 smithi115 bash[18070]: cluster 2022-04-22T17:20:09.046508+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 4 on host 'smithi121' 2022-04-22T17:20:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:10 smithi121 bash[12679]: audit 2022-04-22T17:20:10.276847+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:10 smithi121 bash[12679]: audit 2022-04-22T17:20:10.285350+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:10 smithi121 bash[12679]: audit 2022-04-22T17:20:10.285762+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:10 smithi121 bash[12679]: audit 2022-04-22T17:20:10.287582+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:10 smithi121 bash[12679]: audit 2022-04-22T17:20:10.291288+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:10.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[12737]: audit 2022-04-22T17:20:10.276847+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[12737]: audit 2022-04-22T17:20:10.285350+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[12737]: audit 2022-04-22T17:20:10.285762+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[12737]: audit 2022-04-22T17:20:10.287582+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[12737]: audit 2022-04-22T17:20:10.291288+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[18070]: audit 2022-04-22T17:20:10.276847+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[18070]: audit 2022-04-22T17:20:10.285350+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:10.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[18070]: audit 2022-04-22T17:20:10.285762+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:10.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[18070]: audit 2022-04-22T17:20:10.287582+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:10.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:10 smithi115 bash[18070]: audit 2022-04-22T17:20:10.291288+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:11.543 DEBUG:teuthology.orchestra.run.smithi121:osd.4> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.4.service 2022-04-22T17:20:11.545 INFO:tasks.cephadm:Deploying osd.5 on smithi121 with /dev/vg_nvme/lv_3... 2022-04-22T17:20:11.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-04-22T17:20:11.583 INFO:journalctl@ceph.osd.4.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:20:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[12679]: cluster 2022-04-22T17:20:11.046934+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:12.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:12 smithi115 bash[12737]: cluster 2022-04-22T17:20:11.046934+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:12.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:12 smithi115 bash[18070]: cluster 2022-04-22T17:20:11.046934+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:12.763 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-04-22T17:20:12.763 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2022-04-22T17:20:12.763 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2022-04-22T17:20:12.763 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2022-04-22T17:20:12.763 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2022-04-22T17:20:12.764 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-04-22T17:20:12.764 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:12 smithi121 bash[15570]: --> ceph-volume lvm activate successful for osd ID: 4 2022-04-22T17:20:14.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:14 smithi121 bash[12679]: cluster 2022-04-22T17:20:13.047497+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:14.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:14 smithi115 bash[12737]: cluster 2022-04-22T17:20:13.047497+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:14.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:14 smithi115 bash[18070]: cluster 2022-04-22T17:20:13.047497+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:15.685 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:15 smithi121 bash[12679]: audit 2022-04-22T17:20:14.401088+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:15.685 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:15 smithi121 bash[12679]: cluster 2022-04-22T17:20:15.048062+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:15.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:15 smithi115 bash[18070]: audit 2022-04-22T17:20:14.401088+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:15.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:15 smithi115 bash[18070]: cluster 2022-04-22T17:20:15.048062+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:15.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:15 smithi115 bash[12737]: audit 2022-04-22T17:20:14.401088+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:15.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:15 smithi115 bash[12737]: cluster 2022-04-22T17:20:15.048062+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:16.013 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:15 smithi121 bash[15740]: debug 2022-04-22T17:20:15.681+0000 7fef0ff41700 1 -- 172.21.15.121:0/1745838571 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560f2a6244e0 con 0x560f2a69b000 2022-04-22T17:20:16.513 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:16 smithi121 bash[15740]: debug 2022-04-22T17:20:16.241+0000 7fef179da080 -1 Falling back to public interface 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156205 s, 671 MB/s 2022-04-22T17:20:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2022-04-22T17:20:17.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:17 smithi121 bash[12679]: cluster 2022-04-22T17:20:17.048557+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:17.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:17 smithi115 bash[12737]: cluster 2022-04-22T17:20:17.048557+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:17.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:17 smithi115 bash[18070]: cluster 2022-04-22T17:20:17.048557+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:18.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2022-04-22T17:20:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:19 smithi121 bash[12679]: cluster 2022-04-22T17:20:19.049084+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:19.513 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:19 smithi121 bash[15740]: debug 2022-04-22T17:20:19.297+0000 7fef179da080 -1 osd.4 0 log_to_monitors {default=true} 2022-04-22T17:20:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:19 smithi115 bash[12737]: cluster 2022-04-22T17:20:19.049084+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:19.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:19 smithi115 bash[18070]: cluster 2022-04-22T17:20:19.049084+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:20 smithi121 bash[12679]: audit 2022-04-22T17:20:19.305411+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:20 smithi121 bash[12679]: audit 2022-04-22T17:20:19.316628+0000 mon.a (mon.0) 401 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:20.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:20 smithi115 bash[12737]: audit 2022-04-22T17:20:19.305411+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:20.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:20 smithi115 bash[12737]: audit 2022-04-22T17:20:19.316628+0000 mon.a (mon.0) 401 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:20.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:20 smithi115 bash[18070]: audit 2022-04-22T17:20:19.305411+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:20.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:20 smithi115 bash[18070]: audit 2022-04-22T17:20:19.316628+0000 mon.a (mon.0) 401 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:20:21.514 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[15740]: debug 2022-04-22T17:20:21.153+0000 7fef0df3d700 -1 osd.4 0 waiting for initial osdmap 2022-04-22T17:20:21.514 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[15740]: debug 2022-04-22T17:20:21.161+0000 7fef0a6b9700 -1 osd.4 29 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: audit 2022-04-22T17:20:20.146392+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: audit 2022-04-22T17:20:20.155067+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: cluster 2022-04-22T17:20:20.155320+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: audit 2022-04-22T17:20:20.155558+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: audit 2022-04-22T17:20:20.157330+0000 mon.a (mon.0) 405 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:21 smithi121 bash[12679]: cluster 2022-04-22T17:20:21.049657+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: audit 2022-04-22T17:20:20.146392+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: audit 2022-04-22T17:20:20.155067+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: cluster 2022-04-22T17:20:20.155320+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: audit 2022-04-22T17:20:20.155558+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: audit 2022-04-22T17:20:20.157330+0000 mon.a (mon.0) 405 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[12737]: cluster 2022-04-22T17:20:21.049657+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:21.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: audit 2022-04-22T17:20:20.146392+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: audit 2022-04-22T17:20:20.155067+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: cluster 2022-04-22T17:20:20.155320+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2022-04-22T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: audit 2022-04-22T17:20:20.155558+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: audit 2022-04-22T17:20:20.157330+0000 mon.a (mon.0) 405 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:21 smithi115 bash[18070]: cluster 2022-04-22T17:20:21.049657+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-04-22T17:20:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.159536+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:22.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: cluster 2022-04-22T17:20:21.159754+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2022-04-22T17:20:22.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.160467+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.167840+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: cephadm 2022-04-22T17:20:21.527756+0000 mgr.y (mgr.14146) 153 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:22.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.536727+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.538639+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: cephadm 2022-04-22T17:20:21.539783+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17275M 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.545133+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.550688+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.553163+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.554788+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.555982+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:22.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:22 smithi121 bash[12679]: audit 2022-04-22T17:20:21.562448+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.159536+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:22.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: cluster 2022-04-22T17:20:21.159754+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2022-04-22T17:20:22.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.160467+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.167840+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: cephadm 2022-04-22T17:20:21.527756+0000 mgr.y (mgr.14146) 153 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.536727+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.538639+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: cephadm 2022-04-22T17:20:21.539783+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17275M 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.545133+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.550688+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.553163+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.554788+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.555982+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:22.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[12737]: audit 2022-04-22T17:20:21.562448+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.159536+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: cluster 2022-04-22T17:20:21.159754+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.160467+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.167840+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: cephadm 2022-04-22T17:20:21.527756+0000 mgr.y (mgr.14146) 153 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.536727+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.538639+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: cephadm 2022-04-22T17:20:21.539783+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17275M 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.545133+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.550688+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.553163+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.554788+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.555982+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:22.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:22 smithi115 bash[18070]: audit 2022-04-22T17:20:21.562448+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:20.334380+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:20.334595+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:22.162219+0000 mon.a (mon.0) 418 : cluster [INF] osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480] boot 2022-04-22T17:20:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:22.162416+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2022-04-22T17:20:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: audit 2022-04-22T17:20:22.162806+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: audit 2022-04-22T17:20:22.592227+0000 mgr.y (mgr.14146) 155 : audit [DBG] from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: audit 2022-04-22T17:20:22.593822+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: audit 2022-04-22T17:20:22.597261+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: audit 2022-04-22T17:20:22.598920+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:23.050238+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:23 smithi121 bash[12679]: cluster 2022-04-22T17:20:23.163138+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2022-04-22T17:20:23.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:20.334380+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:20.334595+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:22.162219+0000 mon.a (mon.0) 418 : cluster [INF] osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480] boot 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:22.162416+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: audit 2022-04-22T17:20:22.162806+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: audit 2022-04-22T17:20:22.592227+0000 mgr.y (mgr.14146) 155 : audit [DBG] from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: audit 2022-04-22T17:20:22.593822+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: audit 2022-04-22T17:20:22.597261+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:23.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: audit 2022-04-22T17:20:22.598920+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:23.050238+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[12737]: cluster 2022-04-22T17:20:23.163138+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:20.334380+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:20.334595+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:22.162219+0000 mon.a (mon.0) 418 : cluster [INF] osd.4 [v2:172.21.15.121:6800/826605480,v1:172.21.15.121:6801/826605480] boot 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:22.162416+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2022-04-22T17:20:23.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: audit 2022-04-22T17:20:22.162806+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: audit 2022-04-22T17:20:22.592227+0000 mgr.y (mgr.14146) 155 : audit [DBG] from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: audit 2022-04-22T17:20:22.593822+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: audit 2022-04-22T17:20:22.597261+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: audit 2022-04-22T17:20:22.598920+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:23.050238+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:23 smithi115 bash[18070]: cluster 2022-04-22T17:20:23.163138+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2022-04-22T17:20:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:25 smithi121 bash[12679]: cluster 2022-04-22T17:20:24.166672+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2022-04-22T17:20:25.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:25 smithi121 bash[12679]: cluster 2022-04-22T17:20:25.050825+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v136: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:25.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:25 smithi115 bash[12737]: cluster 2022-04-22T17:20:24.166672+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2022-04-22T17:20:25.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:25 smithi115 bash[12737]: cluster 2022-04-22T17:20:25.050825+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v136: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:25.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:25 smithi115 bash[18070]: cluster 2022-04-22T17:20:24.166672+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2022-04-22T17:20:25.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:25 smithi115 bash[18070]: cluster 2022-04-22T17:20:25.050825+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v136: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:27 smithi121 bash[12679]: audit 2022-04-22T17:20:26.577936+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.121:0/1349054109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:27 smithi121 bash[12679]: audit 2022-04-22T17:20:26.589179+0000 mon.a (mon.0) 426 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:27 smithi121 bash[12679]: audit 2022-04-22T17:20:26.602766+0000 mon.a (mon.0) 427 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]': finished 2022-04-22T17:20:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:27 smithi121 bash[12679]: cluster 2022-04-22T17:20:26.602906+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2022-04-22T17:20:27.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:27 smithi121 bash[12679]: audit 2022-04-22T17:20:26.603131+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:27.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[12737]: audit 2022-04-22T17:20:26.577936+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.121:0/1349054109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[12737]: audit 2022-04-22T17:20:26.589179+0000 mon.a (mon.0) 426 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[12737]: audit 2022-04-22T17:20:26.602766+0000 mon.a (mon.0) 427 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]': finished 2022-04-22T17:20:27.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[12737]: cluster 2022-04-22T17:20:26.602906+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2022-04-22T17:20:27.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[12737]: audit 2022-04-22T17:20:26.603131+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[18070]: audit 2022-04-22T17:20:26.577936+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.121:0/1349054109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[18070]: audit 2022-04-22T17:20:26.589179+0000 mon.a (mon.0) 426 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]: dispatch 2022-04-22T17:20:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[18070]: audit 2022-04-22T17:20:26.602766+0000 mon.a (mon.0) 427 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b85a231-7194-42e3-ae6c-f61741e030fa"}]': finished 2022-04-22T17:20:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[18070]: cluster 2022-04-22T17:20:26.602906+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2022-04-22T17:20:27.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:27 smithi115 bash[18070]: audit 2022-04-22T17:20:26.603131+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:28 smithi121 bash[12679]: cluster 2022-04-22T17:20:27.051278+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v138: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:28 smithi121 bash[12679]: audit 2022-04-22T17:20:27.250449+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.121:0/3763173636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:28.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:28 smithi115 bash[12737]: cluster 2022-04-22T17:20:27.051278+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v138: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:28.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:28 smithi115 bash[12737]: audit 2022-04-22T17:20:27.250449+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.121:0/3763173636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:28.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:28 smithi115 bash[18070]: cluster 2022-04-22T17:20:27.051278+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v138: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:28.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:28 smithi115 bash[18070]: audit 2022-04-22T17:20:27.250449+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.121:0/3763173636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:30.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:30 smithi121 bash[12679]: cluster 2022-04-22T17:20:29.051834+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v139: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:30.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:30 smithi115 bash[12737]: cluster 2022-04-22T17:20:29.051834+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v139: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:30.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:30 smithi115 bash[18070]: cluster 2022-04-22T17:20:29.051834+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v139: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:32 smithi121 bash[12679]: cluster 2022-04-22T17:20:31.052392+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:32.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:32 smithi115 bash[12737]: cluster 2022-04-22T17:20:31.052392+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:32.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:32 smithi115 bash[18070]: cluster 2022-04-22T17:20:31.052392+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:33.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:33 smithi121 bash[12679]: cluster 2022-04-22T17:20:33.052928+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:33.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:33 smithi115 bash[12737]: cluster 2022-04-22T17:20:33.052928+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:33.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:33 smithi115 bash[18070]: cluster 2022-04-22T17:20:33.052928+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:35.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:34 smithi121 bash[12679]: audit 2022-04-22T17:20:34.820584+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-04-22T17:20:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:34 smithi121 bash[12679]: audit 2022-04-22T17:20:34.822534+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:35.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:34 smithi115 bash[12737]: audit 2022-04-22T17:20:34.820584+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-04-22T17:20:35.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:34 smithi115 bash[12737]: audit 2022-04-22T17:20:34.822534+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:35.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:34 smithi115 bash[18070]: audit 2022-04-22T17:20:34.820584+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-04-22T17:20:35.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:34 smithi115 bash[18070]: audit 2022-04-22T17:20:34.822534+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:36.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:35 smithi121 bash[12679]: cephadm 2022-04-22T17:20:34.824230+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:20:36.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:35 smithi121 bash[12679]: cluster 2022-04-22T17:20:35.053568+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:36.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:35 smithi115 bash[12737]: cephadm 2022-04-22T17:20:34.824230+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:20:36.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:35 smithi115 bash[12737]: cluster 2022-04-22T17:20:35.053568+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:36.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:35 smithi115 bash[18070]: cephadm 2022-04-22T17:20:34.824230+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:20:36.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:35 smithi115 bash[18070]: cluster 2022-04-22T17:20:35.053568+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:37.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:37 smithi121 bash[12679]: cluster 2022-04-22T17:20:37.054121+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:37.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:37 smithi115 bash[12737]: cluster 2022-04-22T17:20:37.054121+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:37.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:37 smithi115 bash[18070]: cluster 2022-04-22T17:20:37.054121+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:37.548 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 5 on host 'smithi121' 2022-04-22T17:20:38.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:38 smithi121 bash[12679]: audit 2022-04-22T17:20:37.542500+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:38.206 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:38 smithi121 bash[12679]: audit 2022-04-22T17:20:37.546153+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:38.206 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:38 smithi121 bash[12679]: audit 2022-04-22T17:20:37.551627+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:38.206 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:38 smithi121 bash[12679]: audit 2022-04-22T17:20:37.551957+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:38.206 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:38 smithi121 bash[12679]: audit 2022-04-22T17:20:37.558128+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:38.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[12737]: audit 2022-04-22T17:20:37.542500+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:38.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[12737]: audit 2022-04-22T17:20:37.546153+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[12737]: audit 2022-04-22T17:20:37.551627+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[12737]: audit 2022-04-22T17:20:37.551957+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[12737]: audit 2022-04-22T17:20:37.558128+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[18070]: audit 2022-04-22T17:20:37.542500+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[18070]: audit 2022-04-22T17:20:37.546153+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[18070]: audit 2022-04-22T17:20:37.551627+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[18070]: audit 2022-04-22T17:20:37.551957+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:38 smithi115 bash[18070]: audit 2022-04-22T17:20:37.558128+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:39.010 DEBUG:teuthology.orchestra.run.smithi121:osd.5> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.5.service 2022-04-22T17:20:39.030 INFO:tasks.cephadm:Deploying osd.6 on smithi121 with /dev/vg_nvme/lv_2... 2022-04-22T17:20:39.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-04-22T17:20:39.092 INFO:journalctl@ceph.osd.5.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:20:39.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[12679]: audit 2022-04-22T17:20:38.632737+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:20:39.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[12679]: audit 2022-04-22T17:20:38.633490+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:20:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[12679]: cluster 2022-04-22T17:20:39.054668+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:39.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[12737]: audit 2022-04-22T17:20:38.632737+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:20:39.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[12737]: audit 2022-04-22T17:20:38.633490+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:20:39.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[12737]: cluster 2022-04-22T17:20:39.054668+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:39.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[18070]: audit 2022-04-22T17:20:38.632737+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:20:39.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[18070]: audit 2022-04-22T17:20:38.633490+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:20:39.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:39 smithi115 bash[18070]: cluster 2022-04-22T17:20:39.054668+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:40.005 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-04-22T17:20:40.073 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2022-04-22T17:20:40.073 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2022-04-22T17:20:40.074 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2022-04-22T17:20:40.074 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2022-04-22T17:20:40.074 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-04-22T17:20:40.074 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:39 smithi121 bash[18381]: --> ceph-volume lvm activate successful for osd ID: 5 2022-04-22T17:20:42.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:42 smithi121 bash[12679]: cluster 2022-04-22T17:20:41.055150+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:42.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:42 smithi115 bash[12737]: cluster 2022-04-22T17:20:41.055150+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:42.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:42 smithi115 bash[18070]: cluster 2022-04-22T17:20:41.055150+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:42.743 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:42 smithi121 bash[18604]: debug 2022-04-22T17:20:42.449+0000 7faa353db700 1 -- 172.21.15.121:0/2839377801 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be25fd24e0 con 0x55be26049000 2022-04-22T17:20:43.012 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:42 smithi121 bash[18604]: debug 2022-04-22T17:20:42.989+0000 7faa3ce74080 -1 Falling back to public interface 2022-04-22T17:20:44.012 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:43 smithi121 bash[12679]: audit 2022-04-22T17:20:42.561755+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:44.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:43 smithi121 bash[12679]: cluster 2022-04-22T17:20:43.055679+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:44.026 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:43 smithi115 bash[12737]: audit 2022-04-22T17:20:42.561755+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:44.026 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:43 smithi115 bash[12737]: cluster 2022-04-22T17:20:43.055679+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:44.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:43 smithi115 bash[18070]: audit 2022-04-22T17:20:42.561755+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:44.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:43 smithi115 bash[18070]: cluster 2022-04-22T17:20:43.055679+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-04-22T17:20:45.397 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-04-22T17:20:45.397 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2022-04-22T17:20:45.397 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-04-22T17:20:45.397 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161252 s, 650 MB/s 2022-04-22T17:20:45.397 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2022-04-22T17:20:45.405 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:45 smithi121 bash[12679]: cluster 2022-04-22T17:20:45.056304+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:45.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:45 smithi115 bash[12737]: cluster 2022-04-22T17:20:45.056304+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:45.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:45 smithi115 bash[18070]: cluster 2022-04-22T17:20:45.056304+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:46.012 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:45 smithi121 bash[18604]: debug 2022-04-22T17:20:45.989+0000 7faa3ce74080 -1 osd.5 0 log_to_monitors {default=true} 2022-04-22T17:20:46.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:46 smithi121 bash[12679]: audit 2022-04-22T17:20:46.007453+0000 mon.c (mon.1) 10 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:46 smithi121 bash[12679]: audit 2022-04-22T17:20:46.008312+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:46 smithi115 bash[12737]: audit 2022-04-22T17:20:46.007453+0000 mon.c (mon.1) 10 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:46 smithi115 bash[12737]: audit 2022-04-22T17:20:46.008312+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:46 smithi115 bash[18070]: audit 2022-04-22T17:20:46.007453+0000 mon.c (mon.1) 10 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:46 smithi115 bash[18070]: audit 2022-04-22T17:20:46.008312+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:20:46.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2022-04-22T17:20:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: audit 2022-04-22T17:20:46.161465+0000 mon.a (mon.0) 441 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:20:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: cluster 2022-04-22T17:20:46.161572+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2022-04-22T17:20:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: audit 2022-04-22T17:20:46.162052+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: audit 2022-04-22T17:20:46.163360+0000 mon.c (mon.1) 11 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: audit 2022-04-22T17:20:46.163922+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[12679]: cluster 2022-04-22T17:20:47.056846+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:47.512 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[18604]: debug 2022-04-22T17:20:47.165+0000 7faa333d7700 -1 osd.5 0 waiting for initial osdmap 2022-04-22T17:20:47.513 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:20:47 smithi121 bash[18604]: debug 2022-04-22T17:20:47.173+0000 7faa2cb4d700 -1 osd.5 35 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:20:47.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: audit 2022-04-22T17:20:46.161465+0000 mon.a (mon.0) 441 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:20:47.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: cluster 2022-04-22T17:20:46.161572+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2022-04-22T17:20:47.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: audit 2022-04-22T17:20:46.162052+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:47.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: audit 2022-04-22T17:20:46.163360+0000 mon.c (mon.1) 11 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: audit 2022-04-22T17:20:46.163922+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[12737]: cluster 2022-04-22T17:20:47.056846+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: audit 2022-04-22T17:20:46.161465+0000 mon.a (mon.0) 441 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: cluster 2022-04-22T17:20:46.161572+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: audit 2022-04-22T17:20:46.162052+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: audit 2022-04-22T17:20:46.163360+0000 mon.c (mon.1) 11 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: audit 2022-04-22T17:20:46.163922+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:20:47.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:47 smithi115 bash[18070]: cluster 2022-04-22T17:20:47.056846+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: audit 2022-04-22T17:20:47.169401+0000 mon.a (mon.0) 445 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: cluster 2022-04-22T17:20:47.169549+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: audit 2022-04-22T17:20:47.170262+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: audit 2022-04-22T17:20:47.180476+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: cluster 2022-04-22T17:20:48.167080+0000 mon.a (mon.0) 449 : cluster [INF] osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089] boot 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: cluster 2022-04-22T17:20:48.167182+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2022-04-22T17:20:48.458 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:48 smithi121 bash[12679]: audit 2022-04-22T17:20:48.167433+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: audit 2022-04-22T17:20:47.169401+0000 mon.a (mon.0) 445 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:48.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: cluster 2022-04-22T17:20:47.169549+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2022-04-22T17:20:48.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: audit 2022-04-22T17:20:47.170262+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: audit 2022-04-22T17:20:47.180476+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: cluster 2022-04-22T17:20:48.167080+0000 mon.a (mon.0) 449 : cluster [INF] osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089] boot 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: cluster 2022-04-22T17:20:48.167182+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[18070]: audit 2022-04-22T17:20:48.167433+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: audit 2022-04-22T17:20:47.169401+0000 mon.a (mon.0) 445 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: cluster 2022-04-22T17:20:47.169549+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: audit 2022-04-22T17:20:47.170262+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: audit 2022-04-22T17:20:47.180476+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:48.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: cluster 2022-04-22T17:20:48.167080+0000 mon.a (mon.0) 449 : cluster [INF] osd.5 [v2:172.21.15.121:6808/1865718089,v1:172.21.15.121:6809/1865718089] boot 2022-04-22T17:20:48.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: cluster 2022-04-22T17:20:48.167182+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2022-04-22T17:20:48.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:48 smithi115 bash[12737]: audit 2022-04-22T17:20:48.167433+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:20:49.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:49 smithi121 bash[12679]: cluster 2022-04-22T17:20:47.028264+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:49.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:49 smithi121 bash[12679]: cluster 2022-04-22T17:20:47.028404+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:49 smithi121 bash[12679]: cluster 2022-04-22T17:20:49.057368+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:49 smithi121 bash[12679]: cluster 2022-04-22T17:20:49.169261+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2022-04-22T17:20:49.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[12737]: cluster 2022-04-22T17:20:47.028264+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:49.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[12737]: cluster 2022-04-22T17:20:47.028404+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:49.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[12737]: cluster 2022-04-22T17:20:49.057368+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:49.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[12737]: cluster 2022-04-22T17:20:49.169261+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2022-04-22T17:20:49.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[18070]: cluster 2022-04-22T17:20:47.028264+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:20:49.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[18070]: cluster 2022-04-22T17:20:47.028404+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:20:49.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[18070]: cluster 2022-04-22T17:20:49.057368+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:49.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:49 smithi115 bash[18070]: cluster 2022-04-22T17:20:49.169261+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2022-04-22T17:20:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: cluster 2022-04-22T17:20:50.173519+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2022-04-22T17:20:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: cephadm 2022-04-22T17:20:50.293519+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.303539+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.548525+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.550685+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: cephadm 2022-04-22T17:20:50.551940+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8637M 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.560686+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.569160+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.571923+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.573686+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.574840+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.583222+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.881698+0000 mgr.y (mgr.14146) 173 : audit [DBG] from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.883298+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.886619+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: audit 2022-04-22T17:20:50.888277+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:51 smithi121 bash[12679]: cluster 2022-04-22T17:20:51.057940+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v155: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:51.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: cluster 2022-04-22T17:20:50.173519+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2022-04-22T17:20:51.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: cephadm 2022-04-22T17:20:50.293519+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.303539+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.548525+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.550685+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: cephadm 2022-04-22T17:20:50.551940+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8637M 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.560686+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.569160+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.571923+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.573686+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.574840+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:51.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.583222+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.881698+0000 mgr.y (mgr.14146) 173 : audit [DBG] from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.883298+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.886619+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: audit 2022-04-22T17:20:50.888277+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[12737]: cluster 2022-04-22T17:20:51.057940+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v155: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: cluster 2022-04-22T17:20:50.173519+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: cephadm 2022-04-22T17:20:50.293519+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:20:51.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.303539+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.548525+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.550685+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: cephadm 2022-04-22T17:20:50.551940+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8637M 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.560686+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.569160+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.571923+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.573686+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.574840+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.583222+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.881698+0000 mgr.y (mgr.14146) 173 : audit [DBG] from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:20:51.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.883298+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:20:51.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.886619+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:20:51.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: audit 2022-04-22T17:20:50.888277+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:20:51.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:51 smithi115 bash[18070]: cluster 2022-04-22T17:20:51.057940+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v155: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:53.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:53 smithi121 bash[12679]: cluster 2022-04-22T17:20:53.058559+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v156: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:53.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:53 smithi115 bash[12737]: cluster 2022-04-22T17:20:53.058559+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v156: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:53.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:53 smithi115 bash[18070]: cluster 2022-04-22T17:20:53.058559+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v156: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:55.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:54 smithi121 bash[12679]: audit 2022-04-22T17:20:54.887281+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.121:0/622964381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:54 smithi121 bash[12679]: audit 2022-04-22T17:20:54.898704+0000 mon.a (mon.0) 466 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:54 smithi121 bash[12679]: audit 2022-04-22T17:20:54.907657+0000 mon.a (mon.0) 467 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]': finished 2022-04-22T17:20:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:54 smithi121 bash[12679]: cluster 2022-04-22T17:20:54.907859+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2022-04-22T17:20:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:54 smithi121 bash[12679]: audit 2022-04-22T17:20:54.908155+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[18070]: audit 2022-04-22T17:20:54.887281+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.121:0/622964381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[18070]: audit 2022-04-22T17:20:54.898704+0000 mon.a (mon.0) 466 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[18070]: audit 2022-04-22T17:20:54.907657+0000 mon.a (mon.0) 467 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]': finished 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[18070]: cluster 2022-04-22T17:20:54.907859+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[18070]: audit 2022-04-22T17:20:54.908155+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[12737]: audit 2022-04-22T17:20:54.887281+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.121:0/622964381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[12737]: audit 2022-04-22T17:20:54.898704+0000 mon.a (mon.0) 466 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]: dispatch 2022-04-22T17:20:55.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[12737]: audit 2022-04-22T17:20:54.907657+0000 mon.a (mon.0) 467 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0e12e76-ac56-4f67-85c0-c945cecd1bd5"}]': finished 2022-04-22T17:20:55.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[12737]: cluster 2022-04-22T17:20:54.907859+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2022-04-22T17:20:55.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:54 smithi115 bash[12737]: audit 2022-04-22T17:20:54.908155+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:20:56.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:55 smithi121 bash[12679]: cluster 2022-04-22T17:20:55.059149+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:56.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:55 smithi121 bash[12679]: audit 2022-04-22T17:20:55.532102+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/239855732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:56.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:55 smithi115 bash[18070]: cluster 2022-04-22T17:20:55.059149+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:56.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:55 smithi115 bash[18070]: audit 2022-04-22T17:20:55.532102+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/239855732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:56.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:55 smithi115 bash[12737]: cluster 2022-04-22T17:20:55.059149+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:56.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:55 smithi115 bash[12737]: audit 2022-04-22T17:20:55.532102+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/239855732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:20:57.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:57 smithi121 bash[12679]: cluster 2022-04-22T17:20:57.059739+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:57.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:57 smithi115 bash[18070]: cluster 2022-04-22T17:20:57.059739+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:57.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:57 smithi115 bash[12737]: cluster 2022-04-22T17:20:57.059739+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:59.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:20:59 smithi121 bash[12679]: cluster 2022-04-22T17:20:59.060454+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:59.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:20:59 smithi115 bash[18070]: cluster 2022-04-22T17:20:59.060454+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:20:59.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:20:59 smithi115 bash[12737]: cluster 2022-04-22T17:20:59.060454+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:02.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:02 smithi121 bash[12679]: cluster 2022-04-22T17:21:01.061031+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:02.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:02 smithi115 bash[18070]: cluster 2022-04-22T17:21:01.061031+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:02 smithi115 bash[12737]: cluster 2022-04-22T17:21:01.061031+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:03.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:03 smithi121 bash[12679]: cluster 2022-04-22T17:21:03.061599+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:03.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:03 smithi115 bash[12737]: cluster 2022-04-22T17:21:03.061599+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:03.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:03 smithi115 bash[18070]: cluster 2022-04-22T17:21:03.061599+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:04.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:04 smithi121 bash[12679]: audit 2022-04-22T17:21:03.231274+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-04-22T17:21:04.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:04 smithi121 bash[12679]: audit 2022-04-22T17:21:03.233210+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:04.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:04 smithi121 bash[12679]: cephadm 2022-04-22T17:21:03.234624+0000 mgr.y (mgr.14146) 181 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:21:04.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[12737]: audit 2022-04-22T17:21:03.231274+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-04-22T17:21:04.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[12737]: audit 2022-04-22T17:21:03.233210+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:04.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[12737]: cephadm 2022-04-22T17:21:03.234624+0000 mgr.y (mgr.14146) 181 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:21:04.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[18070]: audit 2022-04-22T17:21:03.231274+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-04-22T17:21:04.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[18070]: audit 2022-04-22T17:21:03.233210+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:04.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:04 smithi115 bash[18070]: cephadm 2022-04-22T17:21:03.234624+0000 mgr.y (mgr.14146) 181 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:21:05.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:05 smithi121 bash[12679]: cluster 2022-04-22T17:21:05.062205+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:05.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:05 smithi115 bash[12737]: cluster 2022-04-22T17:21:05.062205+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:05.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:05 smithi115 bash[18070]: cluster 2022-04-22T17:21:05.062205+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:05.890 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 6 on host 'smithi121' 2022-04-22T17:21:06.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:06 smithi121 bash[12679]: audit 2022-04-22T17:21:05.883727+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:06.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:06 smithi121 bash[12679]: audit 2022-04-22T17:21:05.891381+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:06.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:06 smithi121 bash[12679]: audit 2022-04-22T17:21:05.891865+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:06.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:06 smithi121 bash[12679]: audit 2022-04-22T17:21:05.895093+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:06.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:06 smithi121 bash[12679]: audit 2022-04-22T17:21:05.900991+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[12737]: audit 2022-04-22T17:21:05.883727+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[12737]: audit 2022-04-22T17:21:05.891381+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[12737]: audit 2022-04-22T17:21:05.891865+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[12737]: audit 2022-04-22T17:21:05.895093+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[12737]: audit 2022-04-22T17:21:05.900991+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[18070]: audit 2022-04-22T17:21:05.883727+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:06.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[18070]: audit 2022-04-22T17:21:05.891381+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:06.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[18070]: audit 2022-04-22T17:21:05.891865+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:06.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[18070]: audit 2022-04-22T17:21:05.895093+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:06.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:06 smithi115 bash[18070]: audit 2022-04-22T17:21:05.900991+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:07.303 DEBUG:teuthology.orchestra.run.smithi121:osd.6> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.6.service 2022-04-22T17:21:07.347 INFO:tasks.cephadm:Deploying osd.7 on smithi121 with /dev/vg_nvme/lv_1... 2022-04-22T17:21:07.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-04-22T17:21:07.390 INFO:journalctl@ceph.osd.6.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:21:07.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:07 smithi121 bash[12679]: cluster 2022-04-22T17:21:07.062762+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:07.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:07 smithi115 bash[12737]: cluster 2022-04-22T17:21:07.062762+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:07.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:07 smithi115 bash[18070]: cluster 2022-04-22T17:21:07.062762+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:08.512 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-04-22T17:21:08.513 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:08 smithi121 bash[21246]: --> ceph-volume lvm activate successful for osd ID: 6 2022-04-22T17:21:09.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:09 smithi121 bash[12679]: cluster 2022-04-22T17:21:09.063333+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:09.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:09 smithi115 bash[12737]: cluster 2022-04-22T17:21:09.063333+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:09.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:09 smithi115 bash[18070]: cluster 2022-04-22T17:21:09.063333+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:11.762 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:11 smithi121 bash[12679]: audit 2022-04-22T17:21:10.380667+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:11.762 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:11 smithi121 bash[12679]: cluster 2022-04-22T17:21:11.063919+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:11.763 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:11 smithi121 bash[21417]: debug 2022-04-22T17:21:11.449+0000 7fcc1d628700 1 -- 172.21.15.121:0/4213156431 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5583669584e0 con 0x5583669cf000 2022-04-22T17:21:11.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:11 smithi115 bash[12737]: audit 2022-04-22T17:21:10.380667+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:11.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:11 smithi115 bash[12737]: cluster 2022-04-22T17:21:11.063919+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:11.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:11 smithi115 bash[18070]: audit 2022-04-22T17:21:10.380667+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:11.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:11 smithi115 bash[18070]: cluster 2022-04-22T17:21:11.063919+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:12.488 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:12 smithi121 bash[21417]: debug 2022-04-22T17:21:12.089+0000 7fcc250c1080 -1 Falling back to public interface 2022-04-22T17:21:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-04-22T17:21:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-04-22T17:21:13.472 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2022-04-22T17:21:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-04-22T17:21:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180595 s, 581 MB/s 2022-04-22T17:21:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2022-04-22T17:21:13.511 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:13 smithi121 bash[12679]: cluster 2022-04-22T17:21:13.064520+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:13.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:13 smithi115 bash[12737]: cluster 2022-04-22T17:21:13.064520+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:13.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:13 smithi115 bash[18070]: cluster 2022-04-22T17:21:13.064520+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:14.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2022-04-22T17:21:15.512 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:15 smithi121 bash[21417]: debug 2022-04-22T17:21:15.137+0000 7fcc250c1080 -1 osd.6 0 log_to_monitors {default=true} 2022-04-22T17:21:15.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:15 smithi121 bash[12679]: cluster 2022-04-22T17:21:15.065143+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:15 smithi115 bash[12737]: cluster 2022-04-22T17:21:15.065143+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:15 smithi115 bash[18070]: cluster 2022-04-22T17:21:15.065143+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:16.430 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:16 smithi121 bash[12679]: audit 2022-04-22T17:21:15.147954+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:16.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:16 smithi121 bash[12679]: audit 2022-04-22T17:21:15.159243+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:16.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:16 smithi115 bash[12737]: audit 2022-04-22T17:21:15.147954+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:16.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:16 smithi115 bash[12737]: audit 2022-04-22T17:21:15.159243+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:16.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:16 smithi115 bash[18070]: audit 2022-04-22T17:21:15.147954+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:16.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:16 smithi115 bash[18070]: audit 2022-04-22T17:21:15.159243+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:21:17.512 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[21417]: debug 2022-04-22T17:21:17.165+0000 7fcc1b624700 -1 osd.6 0 waiting for initial osdmap 2022-04-22T17:21:17.512 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[21417]: debug 2022-04-22T17:21:17.173+0000 7fcc15d9c700 -1 osd.6 41 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:21:17.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: audit 2022-04-22T17:21:16.157945+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: audit 2022-04-22T17:21:16.166677+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:21:17.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: cluster 2022-04-22T17:21:16.166786+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2022-04-22T17:21:17.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: audit 2022-04-22T17:21:16.166967+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:17.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: audit 2022-04-22T17:21:16.169248+0000 mon.a (mon.0) 482 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:17 smithi121 bash[12679]: cluster 2022-04-22T17:21:17.065760+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:17.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: audit 2022-04-22T17:21:16.157945+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: audit 2022-04-22T17:21:16.166677+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: cluster 2022-04-22T17:21:16.166786+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: audit 2022-04-22T17:21:16.166967+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: audit 2022-04-22T17:21:16.169248+0000 mon.a (mon.0) 482 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[18070]: cluster 2022-04-22T17:21:17.065760+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: audit 2022-04-22T17:21:16.157945+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: audit 2022-04-22T17:21:16.166677+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:21:17.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: cluster 2022-04-22T17:21:16.166786+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2022-04-22T17:21:17.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: audit 2022-04-22T17:21:16.166967+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:17.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: audit 2022-04-22T17:21:16.169248+0000 mon.a (mon.0) 482 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:17.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:17 smithi115 bash[12737]: cluster 2022-04-22T17:21:17.065760+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-04-22T17:21:18.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: cluster 2022-04-22T17:21:16.138403+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:18.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: cluster 2022-04-22T17:21:16.138559+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:18.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.172744+0000 mon.a (mon.0) 483 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:18.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: cluster 2022-04-22T17:21:17.172892+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2022-04-22T17:21:18.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.173731+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.179166+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: cephadm 2022-04-22T17:21:17.732124+0000 mgr.y (mgr.14146) 189 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:18.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.741801+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.743565+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.745195+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.746674+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: cephadm 2022-04-22T17:21:17.747722+0000 mgr.y (mgr.14146) 190 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5758M 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.753548+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.759268+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.762132+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.763686+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.765020+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:18.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:18 smithi121 bash[12679]: audit 2022-04-22T17:21:17.771226+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: cluster 2022-04-22T17:21:16.138403+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: cluster 2022-04-22T17:21:16.138559+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.172744+0000 mon.a (mon.0) 483 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: cluster 2022-04-22T17:21:17.172892+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.173731+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.179166+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: cephadm 2022-04-22T17:21:17.732124+0000 mgr.y (mgr.14146) 189 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.741801+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.743565+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.745195+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.746674+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: cephadm 2022-04-22T17:21:17.747722+0000 mgr.y (mgr.14146) 190 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5758M 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.753548+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.759268+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.762132+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.763686+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.765020+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[12737]: audit 2022-04-22T17:21:17.771226+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: cluster 2022-04-22T17:21:16.138403+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: cluster 2022-04-22T17:21:16.138559+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.172744+0000 mon.a (mon.0) 483 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: cluster 2022-04-22T17:21:17.172892+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2022-04-22T17:21:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.173731+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.179166+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: cephadm 2022-04-22T17:21:17.732124+0000 mgr.y (mgr.14146) 189 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.741801+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.743565+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.745195+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.746674+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: cephadm 2022-04-22T17:21:17.747722+0000 mgr.y (mgr.14146) 190 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5758M 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.753548+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.759268+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:18.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.762132+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:18.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.763686+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:18.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.765020+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:18.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:18 smithi115 bash[18070]: audit 2022-04-22T17:21:17.771226+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: cluster 2022-04-22T17:21:18.174521+0000 mon.a (mon.0) 497 : cluster [INF] osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083] boot 2022-04-22T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: cluster 2022-04-22T17:21:18.174579+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2022-04-22T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: audit 2022-04-22T17:21:18.174807+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: audit 2022-04-22T17:21:18.875702+0000 mgr.y (mgr.14146) 191 : audit [DBG] from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:21:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: audit 2022-04-22T17:21:18.877408+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:21:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: audit 2022-04-22T17:21:18.880941+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:21:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: audit 2022-04-22T17:21:18.882539+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:19 smithi121 bash[12679]: cluster 2022-04-22T17:21:19.066420+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:19.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: cluster 2022-04-22T17:21:18.174521+0000 mon.a (mon.0) 497 : cluster [INF] osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083] boot 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: cluster 2022-04-22T17:21:18.174579+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: audit 2022-04-22T17:21:18.174807+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: audit 2022-04-22T17:21:18.875702+0000 mgr.y (mgr.14146) 191 : audit [DBG] from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: audit 2022-04-22T17:21:18.877408+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: audit 2022-04-22T17:21:18.880941+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: audit 2022-04-22T17:21:18.882539+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:19.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[12737]: cluster 2022-04-22T17:21:19.066420+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: cluster 2022-04-22T17:21:18.174521+0000 mon.a (mon.0) 497 : cluster [INF] osd.6 [v2:172.21.15.121:6816/1033820083,v1:172.21.15.121:6817/1033820083] boot 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: cluster 2022-04-22T17:21:18.174579+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: audit 2022-04-22T17:21:18.174807+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: audit 2022-04-22T17:21:18.875702+0000 mgr.y (mgr.14146) 191 : audit [DBG] from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: audit 2022-04-22T17:21:18.877408+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-04-22T17:21:19.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: audit 2022-04-22T17:21:18.880941+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-04-22T17:21:19.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: audit 2022-04-22T17:21:18.882539+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:19.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:19 smithi115 bash[18070]: cluster 2022-04-22T17:21:19.066420+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:20.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:20 smithi121 bash[12679]: cluster 2022-04-22T17:21:19.189710+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2022-04-22T17:21:20.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:20 smithi115 bash[12737]: cluster 2022-04-22T17:21:19.189710+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2022-04-22T17:21:20.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:20 smithi115 bash[18070]: cluster 2022-04-22T17:21:19.189710+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2022-04-22T17:21:21.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:21 smithi121 bash[12679]: cluster 2022-04-22T17:21:20.192609+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2022-04-22T17:21:21.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:21 smithi121 bash[12679]: cluster 2022-04-22T17:21:21.067098+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:21.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:21 smithi115 bash[12737]: cluster 2022-04-22T17:21:20.192609+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2022-04-22T17:21:21.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:21 smithi115 bash[12737]: cluster 2022-04-22T17:21:21.067098+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:21.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:21 smithi115 bash[18070]: cluster 2022-04-22T17:21:20.192609+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2022-04-22T17:21:21.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:21 smithi115 bash[18070]: cluster 2022-04-22T17:21:21.067098+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:22.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:22 smithi121 bash[12679]: cluster 2022-04-22T17:21:22.058759+0000 mon.a (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:22.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:22 smithi115 bash[18070]: cluster 2022-04-22T17:21:22.058759+0000 mon.a (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:22.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:22 smithi115 bash[12737]: cluster 2022-04-22T17:21:22.058759+0000 mon.a (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:23.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: cluster 2022-04-22T17:21:23.067758+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:23.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: audit 2022-04-22T17:21:23.147270+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.121:0/739654132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: audit 2022-04-22T17:21:23.158755+0000 mon.a (mon.0) 506 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: audit 2022-04-22T17:21:23.167314+0000 mon.a (mon.0) 507 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]': finished 2022-04-22T17:21:23.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: cluster 2022-04-22T17:21:23.167570+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2022-04-22T17:21:23.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:23 smithi121 bash[12679]: audit 2022-04-22T17:21:23.167848+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: cluster 2022-04-22T17:21:23.067758+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:23.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: audit 2022-04-22T17:21:23.147270+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.121:0/739654132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: audit 2022-04-22T17:21:23.158755+0000 mon.a (mon.0) 506 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: audit 2022-04-22T17:21:23.167314+0000 mon.a (mon.0) 507 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]': finished 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: cluster 2022-04-22T17:21:23.167570+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[12737]: audit 2022-04-22T17:21:23.167848+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: cluster 2022-04-22T17:21:23.067758+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: audit 2022-04-22T17:21:23.147270+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.121:0/739654132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: audit 2022-04-22T17:21:23.158755+0000 mon.a (mon.0) 506 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]: dispatch 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: audit 2022-04-22T17:21:23.167314+0000 mon.a (mon.0) 507 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4a829b3-bc21-4e29-aa3e-b1e4b15938b5"}]': finished 2022-04-22T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: cluster 2022-04-22T17:21:23.167570+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2022-04-22T17:21:23.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:23 smithi115 bash[18070]: audit 2022-04-22T17:21:23.167848+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:24.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:24 smithi121 bash[12679]: audit 2022-04-22T17:21:23.815229+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.121:0/3244094340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:21:24.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:24 smithi115 bash[18070]: audit 2022-04-22T17:21:23.815229+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.121:0/3244094340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:21:24.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:24 smithi115 bash[12737]: audit 2022-04-22T17:21:23.815229+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.121:0/3244094340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-04-22T17:21:25.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:25 smithi121 bash[12679]: cluster 2022-04-22T17:21:25.068447+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:25.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:25 smithi115 bash[12737]: cluster 2022-04-22T17:21:25.068447+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:25.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:25 smithi115 bash[18070]: cluster 2022-04-22T17:21:25.068447+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:26.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:26 smithi121 bash[12679]: cluster 2022-04-22T17:21:25.213783+0000 mon.a (mon.0) 510 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:26.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:26 smithi121 bash[12679]: cluster 2022-04-22T17:21:25.213851+0000 mon.a (mon.0) 511 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:26.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:26 smithi115 bash[12737]: cluster 2022-04-22T17:21:25.213783+0000 mon.a (mon.0) 510 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:26.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:26 smithi115 bash[12737]: cluster 2022-04-22T17:21:25.213851+0000 mon.a (mon.0) 511 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:26.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:26 smithi115 bash[18070]: cluster 2022-04-22T17:21:25.213783+0000 mon.a (mon.0) 510 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:26.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:26 smithi115 bash[18070]: cluster 2022-04-22T17:21:25.213851+0000 mon.a (mon.0) 511 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:27.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:27 smithi121 bash[12679]: cluster 2022-04-22T17:21:27.069086+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:27.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:27 smithi115 bash[12737]: cluster 2022-04-22T17:21:27.069086+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:27.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:27 smithi115 bash[18070]: cluster 2022-04-22T17:21:27.069086+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:29.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:29 smithi121 bash[12679]: cluster 2022-04-22T17:21:29.069796+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:29.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:29 smithi115 bash[12737]: cluster 2022-04-22T17:21:29.069796+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:29.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:29 smithi115 bash[18070]: cluster 2022-04-22T17:21:29.069796+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:32.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:32 smithi121 bash[12679]: cluster 2022-04-22T17:21:31.070326+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:32.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:32 smithi121 bash[12679]: audit 2022-04-22T17:21:31.712787+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-04-22T17:21:32.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:32 smithi121 bash[12679]: audit 2022-04-22T17:21:31.714374+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:32.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[12737]: cluster 2022-04-22T17:21:31.070326+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:32.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[12737]: audit 2022-04-22T17:21:31.712787+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-04-22T17:21:32.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[12737]: audit 2022-04-22T17:21:31.714374+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:32.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[18070]: cluster 2022-04-22T17:21:31.070326+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:32.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[18070]: audit 2022-04-22T17:21:31.712787+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-04-22T17:21:32.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:32 smithi115 bash[18070]: audit 2022-04-22T17:21:31.714374+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:33.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:33 smithi121 bash[12679]: cephadm 2022-04-22T17:21:31.715674+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:21:33.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:33 smithi115 bash[12737]: cephadm 2022-04-22T17:21:31.715674+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:21:33.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:33 smithi115 bash[18070]: cephadm 2022-04-22T17:21:31.715674+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:21:34.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:34 smithi121 bash[12679]: cluster 2022-04-22T17:21:33.070880+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:34.473 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 7 on host 'smithi121' 2022-04-22T17:21:34.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:34 smithi115 bash[12737]: cluster 2022-04-22T17:21:33.070880+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:34.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:34 smithi115 bash[18070]: cluster 2022-04-22T17:21:33.070880+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:35.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:35 smithi121 bash[12679]: audit 2022-04-22T17:21:34.467351+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:35.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:35 smithi121 bash[12679]: audit 2022-04-22T17:21:34.475371+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:35 smithi121 bash[12679]: audit 2022-04-22T17:21:34.475884+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:35 smithi121 bash[12679]: audit 2022-04-22T17:21:34.478718+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:35 smithi121 bash[12679]: audit 2022-04-22T17:21:34.483822+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:35.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[12737]: audit 2022-04-22T17:21:34.467351+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:35.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[12737]: audit 2022-04-22T17:21:34.475371+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[12737]: audit 2022-04-22T17:21:34.475884+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[12737]: audit 2022-04-22T17:21:34.478718+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[12737]: audit 2022-04-22T17:21:34.483822+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[18070]: audit 2022-04-22T17:21:34.467351+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[18070]: audit 2022-04-22T17:21:34.475371+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[18070]: audit 2022-04-22T17:21:34.475884+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:35.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[18070]: audit 2022-04-22T17:21:34.478718+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:35.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:35 smithi115 bash[18070]: audit 2022-04-22T17:21:34.483822+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:36.060 DEBUG:teuthology.orchestra.run.smithi121:osd.7> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.7.service 2022-04-22T17:21:36.077 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2022-04-22T17:21:36.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd stat -f json 2022-04-22T17:21:36.111 INFO:journalctl@ceph.osd.7.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:21:36.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[12679]: cluster 2022-04-22T17:21:35.071538+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:36.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:36 smithi115 bash[12737]: cluster 2022-04-22T17:21:35.071538+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:36.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:36 smithi115 bash[18070]: cluster 2022-04-22T17:21:35.071538+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:37.012 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-04-22T17:21:37.026 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2022-04-22T17:21:37.027 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2022-04-22T17:21:37.027 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2022-04-22T17:21:37.027 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2022-04-22T17:21:37.027 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-04-22T17:21:37.028 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:36 smithi121 bash[24063]: --> ceph-volume lvm activate successful for osd ID: 7 2022-04-22T17:21:38.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:38 smithi115 bash[18070]: cluster 2022-04-22T17:21:37.072194+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:38.383 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:38 smithi115 bash[12737]: cluster 2022-04-22T17:21:37.072194+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:38.511 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:38 smithi121 bash[12679]: cluster 2022-04-22T17:21:37.072194+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:38.920 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:38 smithi121 bash[24219]: debug 2022-04-22T17:21:38.629+0000 7f1e14f84700 1 -- 172.21.15.121:0/1365976343 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ac69a84e0 con 0x557ac6a1ec00 2022-04-22T17:21:39.262 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:39 smithi121 bash[24219]: debug 2022-04-22T17:21:39.169+0000 7f1e1ca1d080 -1 Falling back to public interface 2022-04-22T17:21:39.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:39 smithi121 bash[12679]: audit 2022-04-22T17:21:38.635510+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:21:39.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:39 smithi121 bash[12679]: audit 2022-04-22T17:21:38.636267+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:21:39.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:39 smithi121 bash[12679]: audit 2022-04-22T17:21:38.725944+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:39.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[12737]: audit 2022-04-22T17:21:38.635510+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:21:39.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[12737]: audit 2022-04-22T17:21:38.636267+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:21:39.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[12737]: audit 2022-04-22T17:21:38.725944+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:39.395 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[18070]: audit 2022-04-22T17:21:38.635510+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:21:39.395 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[18070]: audit 2022-04-22T17:21:38.636267+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:21:39.395 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:39 smithi115 bash[18070]: audit 2022-04-22T17:21:38.725944+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:39.783 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:21:40.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:40 smithi115 bash[12737]: cluster 2022-04-22T17:21:39.072786+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:40.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:40 smithi115 bash[12737]: audit 2022-04-22T17:21:39.796119+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.115:0/2185544407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:40.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:40 smithi115 bash[18070]: cluster 2022-04-22T17:21:39.072786+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:40.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:40 smithi115 bash[18070]: audit 2022-04-22T17:21:39.796119+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.115:0/2185544407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:40.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:40 smithi121 bash[12679]: cluster 2022-04-22T17:21:39.072786+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:40.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:40 smithi121 bash[12679]: audit 2022-04-22T17:21:39.796119+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.115:0/2185544407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:40.595 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":45,"num_osds":8,"num_up_osds":7,"osd_up_since":1650648078,"num_in_osds":8,"osd_in_since":1650648083,"num_remapped_pgs":0} 2022-04-22T17:21:41.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd stat -f json 2022-04-22T17:21:42.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:42 smithi115 bash[12737]: cluster 2022-04-22T17:21:41.073320+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:42.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:42 smithi115 bash[18070]: cluster 2022-04-22T17:21:41.073320+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:42.512 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:42 smithi121 bash[24219]: debug 2022-04-22T17:21:42.149+0000 7f1e1ca1d080 -1 osd.7 0 log_to_monitors {default=true} 2022-04-22T17:21:42.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:42 smithi121 bash[12679]: cluster 2022-04-22T17:21:41.073320+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:43.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:43 smithi115 bash[12737]: audit 2022-04-22T17:21:42.160308+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:43.272 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:43 smithi115 bash[12737]: audit 2022-04-22T17:21:42.171804+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:43.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:43 smithi115 bash[18070]: audit 2022-04-22T17:21:42.160308+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:43.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:43 smithi115 bash[18070]: audit 2022-04-22T17:21:42.171804+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:43.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:43 smithi121 bash[12679]: audit 2022-04-22T17:21:42.160308+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:43.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:43 smithi121 bash[12679]: audit 2022-04-22T17:21:42.171804+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:21:44.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: cluster 2022-04-22T17:21:43.073895+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: audit 2022-04-22T17:21:43.078235+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: audit 2022-04-22T17:21:43.088208+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: cluster 2022-04-22T17:21:43.088314+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: audit 2022-04-22T17:21:43.088506+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[12737]: audit 2022-04-22T17:21:43.089883+0000 mon.a (mon.0) 527 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: cluster 2022-04-22T17:21:43.073895+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:44.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: audit 2022-04-22T17:21:43.078235+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:44.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: audit 2022-04-22T17:21:43.088208+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:21:44.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: cluster 2022-04-22T17:21:43.088314+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2022-04-22T17:21:44.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: audit 2022-04-22T17:21:43.088506+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:44.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:44 smithi115 bash[18070]: audit 2022-04-22T17:21:43.089883+0000 mon.a (mon.0) 527 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:44.512 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[24219]: debug 2022-04-22T17:21:44.080+0000 7f1e12f80700 -1 osd.7 0 waiting for initial osdmap 2022-04-22T17:21:44.512 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[24219]: debug 2022-04-22T17:21:44.088+0000 7f1e0bef5700 -1 osd.7 47 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: cluster 2022-04-22T17:21:43.073895+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: audit 2022-04-22T17:21:43.078235+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: audit 2022-04-22T17:21:43.088208+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: cluster 2022-04-22T17:21:43.088314+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: audit 2022-04-22T17:21:43.088506+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:44 smithi121 bash[12679]: audit 2022-04-22T17:21:43.089883+0000 mon.a (mon.0) 527 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.087704+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: cluster 2022-04-22T17:21:44.087859+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.088490+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.094900+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.514449+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.516287+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.517820+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.519795+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.521585+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.531551+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.539653+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.542278+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.543833+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.545407+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:45.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[18070]: audit 2022-04-22T17:21:44.555215+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.087704+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:45.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: cluster 2022-04-22T17:21:44.087859+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2022-04-22T17:21:45.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.088490+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.094900+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.514449+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.516287+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.517820+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.519795+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.521585+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.531551+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.539653+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.542278+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:45.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.543833+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:45.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.545407+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:45.280 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:45 smithi115 bash[12737]: audit 2022-04-22T17:21:44.555215+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.352 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:21:45.512 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.087704+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: cluster 2022-04-22T17:21:44.087859+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.088490+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.094900+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.514449+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.516287+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.517820+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.519795+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.521585+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.531551+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.539653+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.542278+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.543833+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.545407+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:21:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:45 smithi121 bash[12679]: audit 2022-04-22T17:21:44.555215+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:21:46.163 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1650648105,"num_in_osds":8,"osd_in_since":1650648083,"num_remapped_pgs":1} 2022-04-22T17:21:46.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd dump --format=json 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:43.159817+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:43.159930+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cephadm 2022-04-22T17:21:44.504589+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cephadm 2022-04-22T17:21:44.522827+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4318M 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:45.074522+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v191: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:45.100018+0000 mon.a (mon.0) 543 : cluster [INF] osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072] boot 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:45.100096+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2022-04-22T17:21:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: audit 2022-04-22T17:21:45.100998+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: audit 2022-04-22T17:21:45.363946+0000 mon.a (mon.0) 546 : audit [DBG] from='client.? 172.21.15.115:0/1563442056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:45.551835+0000 mon.a (mon.0) 547 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:46 smithi121 bash[12679]: cluster 2022-04-22T17:21:46.069089+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2022-04-22T17:21:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:43.159817+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:43.159930+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cephadm 2022-04-22T17:21:44.504589+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cephadm 2022-04-22T17:21:44.522827+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4318M 2022-04-22T17:21:46.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:45.074522+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v191: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:45.100018+0000 mon.a (mon.0) 543 : cluster [INF] osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072] boot 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:45.100096+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: audit 2022-04-22T17:21:45.100998+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: audit 2022-04-22T17:21:45.363946+0000 mon.a (mon.0) 546 : audit [DBG] from='client.? 172.21.15.115:0/1563442056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:45.551835+0000 mon.a (mon.0) 547 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:46.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[12737]: cluster 2022-04-22T17:21:46.069089+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:43.159817+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:43.159930+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cephadm 2022-04-22T17:21:44.504589+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cephadm 2022-04-22T17:21:44.522827+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4318M 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:45.074522+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v191: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:45.100018+0000 mon.a (mon.0) 543 : cluster [INF] osd.7 [v2:172.21.15.121:6824/493226072,v1:172.21.15.121:6825/493226072] boot 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:45.100096+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2022-04-22T17:21:46.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: audit 2022-04-22T17:21:45.100998+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:21:46.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: audit 2022-04-22T17:21:45.363946+0000 mon.a (mon.0) 546 : audit [DBG] from='client.? 172.21.15.115:0/1563442056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-04-22T17:21:46.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:45.551835+0000 mon.a (mon.0) 547 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-04-22T17:21:46.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:46 smithi115 bash[18070]: cluster 2022-04-22T17:21:46.069089+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2022-04-22T17:21:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:48 smithi121 bash[12679]: cluster 2022-04-22T17:21:47.075153+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v195: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:48.570 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:48 smithi121 bash[12679]: cluster 2022-04-22T17:21:47.080133+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2022-04-22T17:21:48.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:48 smithi115 bash[18070]: cluster 2022-04-22T17:21:47.075153+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v195: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:48.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:48 smithi115 bash[18070]: cluster 2022-04-22T17:21:47.080133+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2022-04-22T17:21:48.572 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:48 smithi115 bash[12737]: cluster 2022-04-22T17:21:47.075153+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v195: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:48.572 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:48 smithi115 bash[12737]: cluster 2022-04-22T17:21:47.080133+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2022-04-22T17:21:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:21:50.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:50 smithi115 bash[12737]: cluster 2022-04-22T17:21:49.075768+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v196: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:50.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:50 smithi115 bash[18070]: cluster 2022-04-22T17:21:49.075768+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v196: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:50 smithi121 bash[12679]: cluster 2022-04-22T17:21:49.075768+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v196: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:52.442 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:21:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":50,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:21:47.071614+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:21:52.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:52 smithi115 bash[12737]: cluster 2022-04-22T17:21:51.076224+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v197: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:52.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:52 smithi115 bash[18070]: cluster 2022-04-22T17:21:51.076224+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v197: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:52 smithi121 bash[12679]: cluster 2022-04-22T17:21:51.076224+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v197: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:53.243 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-04-22T17:19:28.627096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2022-04-22T17:21:53.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2022-04-22T17:21:53.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:53 smithi121 bash[12679]: audit 2022-04-22T17:21:52.450303+0000 mon.a (mon.0) 550 : audit [DBG] from='client.? 172.21.15.115:0/2349756934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:21:53.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:53 smithi115 bash[12737]: audit 2022-04-22T17:21:52.450303+0000 mon.a (mon.0) 550 : audit [DBG] from='client.? 172.21.15.115:0/2349756934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:21:53.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:53 smithi115 bash[18070]: audit 2022-04-22T17:21:52.450303+0000 mon.a (mon.0) 550 : audit [DBG] from='client.? 172.21.15.115:0/2349756934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:21:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:54 smithi121 bash[12679]: cluster 2022-04-22T17:21:53.076796+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:54 smithi121 bash[12679]: cluster 2022-04-22T17:21:53.127991+0000 mon.a (mon.0) 551 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:54 smithi121 bash[12679]: cluster 2022-04-22T17:21:53.128055+0000 mon.a (mon.0) 552 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:54.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[12737]: cluster 2022-04-22T17:21:53.076796+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:54.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[12737]: cluster 2022-04-22T17:21:53.127991+0000 mon.a (mon.0) 551 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:54.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[12737]: cluster 2022-04-22T17:21:53.128055+0000 mon.a (mon.0) 552 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:54.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[18070]: cluster 2022-04-22T17:21:53.076796+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:54.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[18070]: cluster 2022-04-22T17:21:53.127991+0000 mon.a (mon.0) 551 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-04-22T17:21:54.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:54 smithi115 bash[18070]: cluster 2022-04-22T17:21:53.128055+0000 mon.a (mon.0) 552 : cluster [INF] Cluster is now healthy 2022-04-22T17:21:56.012 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:21:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:56 smithi121 bash[12679]: cluster 2022-04-22T17:21:55.077418+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:56.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:56 smithi115 bash[12737]: cluster 2022-04-22T17:21:55.077418+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:56.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:56 smithi115 bash[18070]: cluster 2022-04-22T17:21:55.077418+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:58.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:21:58 smithi121 bash[12679]: cluster 2022-04-22T17:21:57.078120+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:58.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:21:58 smithi115 bash[12737]: cluster 2022-04-22T17:21:57.078120+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:58.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:21:58 smithi115 bash[18070]: cluster 2022-04-22T17:21:57.078120+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:21:59.656 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2022-04-22T17:22:00.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:00 smithi115 bash[18070]: cluster 2022-04-22T17:21:59.078763+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:00 smithi115 bash[18070]: audit 2022-04-22T17:21:59.663187+0000 mon.a (mon.0) 553 : audit [DBG] from='client.? 172.21.15.115:0/488696103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:00.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:00 smithi115 bash[12737]: cluster 2022-04-22T17:21:59.078763+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:00.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:00 smithi115 bash[12737]: audit 2022-04-22T17:21:59.663187+0000 mon.a (mon.0) 553 : audit [DBG] from='client.? 172.21.15.115:0/488696103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:00.435 INFO:tasks.cephadm:Adding mds.a on smithi115 2022-04-22T17:22:00.436 INFO:tasks.cephadm:Adding mds.b on smithi121 2022-04-22T17:22:00.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply mds all '2;smithi115=a;smithi121=b' 2022-04-22T17:22:00.451 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:00 smithi121 bash[12679]: cluster 2022-04-22T17:21:59.078763+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:00.451 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:00 smithi121 bash[12679]: audit 2022-04-22T17:21:59.663187+0000 mon.a (mon.0) 553 : audit [DBG] from='client.? 172.21.15.115:0/488696103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:02 smithi121 bash[12679]: cluster 2022-04-22T17:22:01.079292+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:02.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:02 smithi115 bash[12737]: cluster 2022-04-22T17:22:01.079292+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:02.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:02 smithi115 bash[18070]: cluster 2022-04-22T17:22:01.079292+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:03 smithi121 bash[12679]: cluster 2022-04-22T17:22:03.079957+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:03.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:03 smithi115 bash[12737]: cluster 2022-04-22T17:22:03.079957+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:03.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:03 smithi115 bash[18070]: cluster 2022-04-22T17:22:03.079957+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled mds.all update... 2022-04-22T17:22:04.927 DEBUG:teuthology.orchestra.run.smithi115:mds.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mds.a.service 2022-04-22T17:22:04.946 DEBUG:teuthology.orchestra.run.smithi121:mds.b> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mds.b.service 2022-04-22T17:22:04.972 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-04-22T17:22:04.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:22:04.985 INFO:journalctl@ceph.mds.a.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:05.007 INFO:journalctl@ceph.mds.b.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.122504+0000 mgr.y (mgr.14146) 218 : audit [DBG] from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: cephadm 2022-04-22T17:22:04.124162+0000 mgr.y (mgr.14146) 219 : cephadm [INF] Saving service mds.all spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.131406+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.132872+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.134941+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.136494+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.385323+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.393724+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.395186+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.399454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: audit 2022-04-22T17:22:04.400701+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: cephadm 2022-04-22T17:22:04.402143+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:22:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:05 smithi121 bash[12679]: cluster 2022-04-22T17:22:05.080555+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:05.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.122504+0000 mgr.y (mgr.14146) 218 : audit [DBG] from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:05.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: cephadm 2022-04-22T17:22:04.124162+0000 mgr.y (mgr.14146) 219 : cephadm [INF] Saving service mds.all spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.131406+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.132872+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.134941+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.136494+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.385323+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.393724+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.395186+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:05.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.399454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: audit 2022-04-22T17:22:04.400701+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: cephadm 2022-04-22T17:22:04.402143+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[18070]: cluster 2022-04-22T17:22:05.080555+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.122504+0000 mgr.y (mgr.14146) 218 : audit [DBG] from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: cephadm 2022-04-22T17:22:04.124162+0000 mgr.y (mgr.14146) 219 : cephadm [INF] Saving service mds.all spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.131406+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.132872+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.134941+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.136494+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.385323+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.393724+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.395186+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.399454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: audit 2022-04-22T17:22:04.400701+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: cephadm 2022-04-22T17:22:04.402143+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:22:05.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:05 smithi115 bash[12737]: cluster 2022-04-22T17:22:05.080555+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:05.367 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:05.367 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":50,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:21:47.071614+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:22:05.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-04-22T17:19:28.627096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2022-04-22T17:22:05.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-04-22T17:22:05.646 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2022-04-22T17:22:05.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:06.055 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:06.055 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-04-22T17:22:06.059 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 1 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[18070]: audit 2022-04-22T17:22:05.371888+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/1504476230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[18070]: audit 2022-04-22T17:22:05.652030+0000 mon.a (mon.0) 563 : audit [DBG] from='client.? 172.21.15.115:0/1816726717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[18070]: audit 2022-04-22T17:22:06.060512+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.115:0/4105252434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[12737]: audit 2022-04-22T17:22:05.371888+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/1504476230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[12737]: audit 2022-04-22T17:22:05.652030+0000 mon.a (mon.0) 563 : audit [DBG] from='client.? 172.21.15.115:0/1816726717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:06.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:06 smithi115 bash[12737]: audit 2022-04-22T17:22:06.060512+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.115:0/4105252434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:06 smithi121 bash[12679]: audit 2022-04-22T17:22:05.371888+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/1504476230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:06 smithi121 bash[12679]: audit 2022-04-22T17:22:05.652030+0000 mon.a (mon.0) 563 : audit [DBG] from='client.? 172.21.15.115:0/1816726717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:06 smithi121 bash[12679]: audit 2022-04-22T17:22:06.060512+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.115:0/4105252434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:07.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:07 smithi115 bash[12737]: cluster 2022-04-22T17:22:07.081104+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:07.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:07 smithi115 bash[18070]: cluster 2022-04-22T17:22:07.081104+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:07 smithi121 bash[12679]: cluster 2022-04-22T17:22:07.081104+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:08.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:08.370 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:08.370 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-04-22T17:22:08.372 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 1 2022-04-22T17:22:08.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: audit 2022-04-22T17:22:07.455564+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: audit 2022-04-22T17:22:07.457231+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: audit 2022-04-22T17:22:07.462416+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: audit 2022-04-22T17:22:07.463554+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: cephadm 2022-04-22T17:22:07.464437+0000 mgr.y (mgr.14146) 223 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[18070]: audit 2022-04-22T17:22:08.375207+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.115:0/903453803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:08.640 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: audit 2022-04-22T17:22:07.455564+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:08.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: audit 2022-04-22T17:22:07.457231+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:08.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: audit 2022-04-22T17:22:07.462416+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:08.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: audit 2022-04-22T17:22:07.463554+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:08.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: cephadm 2022-04-22T17:22:07.464437+0000 mgr.y (mgr.14146) 223 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:22:08.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[12737]: audit 2022-04-22T17:22:08.375207+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.115:0/903453803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: audit 2022-04-22T17:22:07.455564+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: audit 2022-04-22T17:22:07.457231+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: audit 2022-04-22T17:22:07.462416+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-04-22T17:22:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: audit 2022-04-22T17:22:07.463554+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: cephadm 2022-04-22T17:22:07.464437+0000 mgr.y (mgr.14146) 223 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:22:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:08 smithi121 bash[12679]: audit 2022-04-22T17:22:08.375207+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.115:0/903453803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:09.032 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:22:08 smithi115 bash[31291]: starting mds.a at 2022-04-22T17:22:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:09 smithi121 bash[12679]: cluster 2022-04-22T17:22:09.081679+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:09.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:09 smithi115 bash[18070]: cluster 2022-04-22T17:22:09.081679+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:09.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:09 smithi115 bash[12737]: cluster 2022-04-22T17:22:09.081679+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:10.265 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:22:10 smithi121 systemd[1]: Started Ceph mds.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:22:10.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:10.677 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:10.677 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-04-22T17:22:10.680 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 1 2022-04-22T17:22:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: audit 2022-04-22T17:22:10.245691+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: audit 2022-04-22T17:22:10.247779+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: audit 2022-04-22T17:22:10.249716+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: audit 2022-04-22T17:22:10.251084+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: audit 2022-04-22T17:22:10.682086+0000 mon.a (mon.0) 572 : audit [DBG] from='client.? 172.21.15.115:0/4130791349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[12679]: cluster 2022-04-22T17:22:11.082137+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:11.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: audit 2022-04-22T17:22:10.245691+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:11.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: audit 2022-04-22T17:22:10.247779+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: audit 2022-04-22T17:22:10.249716+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:11.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: audit 2022-04-22T17:22:10.251084+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: audit 2022-04-22T17:22:10.682086+0000 mon.a (mon.0) 572 : audit [DBG] from='client.? 172.21.15.115:0/4130791349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[12737]: cluster 2022-04-22T17:22:11.082137+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: audit 2022-04-22T17:22:10.245691+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: audit 2022-04-22T17:22:10.247779+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: audit 2022-04-22T17:22:10.249716+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: audit 2022-04-22T17:22:10.251084+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: audit 2022-04-22T17:22:10.682086+0000 mon.a (mon.0) 572 : audit [DBG] from='client.? 172.21.15.115:0/4130791349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:11.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:11 smithi115 bash[18070]: cluster 2022-04-22T17:22:11.082137+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:12.014 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:22:11 smithi121 bash[25529]: starting mds.b at 2022-04-22T17:22:12.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:12.985 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:12.985 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2211373279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2211373279},{"type":"v1","addr":"172.21.15.121:6833","nonce":2211373279}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[]} 2022-04-22T17:22:12.988 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 2 2022-04-22T17:22:12.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[12737]: cluster 2022-04-22T17:22:11.576589+0000 mon.a (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:boot 2022-04-22T17:22:12.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[12737]: cluster 2022-04-22T17:22:11.576658+0000 mon.a (mon.0) 574 : cluster [DBG] fsmap 1 up:standby 2022-04-22T17:22:12.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[12737]: audit 2022-04-22T17:22:11.576933+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:22:12.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[12737]: audit 2022-04-22T17:22:12.233736+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:12.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[18070]: cluster 2022-04-22T17:22:11.576589+0000 mon.a (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:boot 2022-04-22T17:22:12.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[18070]: cluster 2022-04-22T17:22:11.576658+0000 mon.a (mon.0) 574 : cluster [DBG] fsmap 1 up:standby 2022-04-22T17:22:12.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[18070]: audit 2022-04-22T17:22:11.576933+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:22:12.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:12 smithi115 bash[18070]: audit 2022-04-22T17:22:12.233736+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:12 smithi121 bash[12679]: cluster 2022-04-22T17:22:11.576589+0000 mon.a (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:boot 2022-04-22T17:22:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:12 smithi121 bash[12679]: cluster 2022-04-22T17:22:11.576658+0000 mon.a (mon.0) 574 : cluster [DBG] fsmap 1 up:standby 2022-04-22T17:22:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:12 smithi121 bash[12679]: audit 2022-04-22T17:22:11.576933+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:22:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:12 smithi121 bash[12679]: audit 2022-04-22T17:22:12.233736+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:13 smithi121 bash[12679]: audit 2022-04-22T17:22:12.989012+0000 mon.a (mon.0) 577 : audit [DBG] from='client.? 172.21.15.115:0/1176629954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:13 smithi121 bash[12679]: cluster 2022-04-22T17:22:13.082646+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:13 smithi121 bash[12679]: cluster 2022-04-22T17:22:13.580545+0000 mon.a (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] up:boot 2022-04-22T17:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:13 smithi121 bash[12679]: cluster 2022-04-22T17:22:13.580638+0000 mon.a (mon.0) 579 : cluster [DBG] fsmap 2 up:standby 2022-04-22T17:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:13 smithi121 bash[12679]: audit 2022-04-22T17:22:13.580872+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:22:14.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[18070]: audit 2022-04-22T17:22:12.989012+0000 mon.a (mon.0) 577 : audit [DBG] from='client.? 172.21.15.115:0/1176629954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:14.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[18070]: cluster 2022-04-22T17:22:13.082646+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:14.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[18070]: cluster 2022-04-22T17:22:13.580545+0000 mon.a (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] up:boot 2022-04-22T17:22:14.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[18070]: cluster 2022-04-22T17:22:13.580638+0000 mon.a (mon.0) 579 : cluster [DBG] fsmap 2 up:standby 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[18070]: audit 2022-04-22T17:22:13.580872+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[12737]: audit 2022-04-22T17:22:12.989012+0000 mon.a (mon.0) 577 : audit [DBG] from='client.? 172.21.15.115:0/1176629954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[12737]: cluster 2022-04-22T17:22:13.082646+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[12737]: cluster 2022-04-22T17:22:13.580545+0000 mon.a (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] up:boot 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[12737]: cluster 2022-04-22T17:22:13.580638+0000 mon.a (mon.0) 579 : cluster [DBG] fsmap 2 up:standby 2022-04-22T17:22:14.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:13 smithi115 bash[12737]: audit 2022-04-22T17:22:13.580872+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:22:15.001 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:13.865541+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.801915+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.808212+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.813566+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:15.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.815290+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:15.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.816525+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:15.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:14 smithi121 bash[12679]: audit 2022-04-22T17:22:14.823296+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:13.865541+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.801915+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.808212+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.813566+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.815290+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.816525+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[18070]: audit 2022-04-22T17:22:14.823296+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:13.865541+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.801915+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.808212+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.813566+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:15.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.815290+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:15.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.816525+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:15.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:14 smithi115 bash[12737]: audit 2022-04-22T17:22:14.823296+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:15.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:15.301 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.115:6835/3475839169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3475839169},{"type":"v1","addr":"172.21.15.115:6835","nonce":3475839169}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3},{"gid":24386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2211373279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2211373279},{"type":"v1","addr":"172.21.15.121:6833","nonce":2211373279}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[]} 2022-04-22T17:22:15.303 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 3 2022-04-22T17:22:15.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:22:15.572 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:15.572 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":50,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:21:47.071614+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:22:15.587 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-04-22T17:19:28.627096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2022-04-22T17:22:15.587 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-04-22T17:22:15.866 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2022-04-22T17:22:15.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[12737]: cluster 2022-04-22T17:22:15.083279+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:15.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[12737]: audit 2022-04-22T17:22:15.304593+0000 mon.a (mon.0) 588 : audit [DBG] from='client.? 172.21.15.115:0/2199635995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:15.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[12737]: audit 2022-04-22T17:22:15.574931+0000 mon.a (mon.0) 589 : audit [DBG] from='client.? 172.21.15.115:0/1435725225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:15.881 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-04-22T17:22:16.137 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout:[ 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout: "poolnum": 1, 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout: "poolname": "device_health_metrics" 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:22:16.138 INFO:teuthology.orchestra.run.smithi115.stdout:] 2022-04-22T17:22:16.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[18070]: cluster 2022-04-22T17:22:15.083279+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:16.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[18070]: audit 2022-04-22T17:22:15.304593+0000 mon.a (mon.0) 588 : audit [DBG] from='client.? 172.21.15.115:0/2199635995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:16.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:15 smithi115 bash[18070]: audit 2022-04-22T17:22:15.574931+0000 mon.a (mon.0) 589 : audit [DBG] from='client.? 172.21.15.115:0/1435725225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:16.152 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-04-22T17:22:16.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-04-22T17:22:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:15 smithi121 bash[12679]: cluster 2022-04-22T17:22:15.083279+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:15 smithi121 bash[12679]: audit 2022-04-22T17:22:15.304593+0000 mon.a (mon.0) 588 : audit [DBG] from='client.? 172.21.15.115:0/2199635995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:15 smithi121 bash[12679]: audit 2022-04-22T17:22:15.574931+0000 mon.a (mon.0) 589 : audit [DBG] from='client.? 172.21.15.115:0/1435725225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:17.089 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_metadata' created 2022-04-22T17:22:17.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-04-22T17:22:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:16 smithi121 bash[12679]: audit 2022-04-22T17:22:15.863759+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/326145771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:16 smithi121 bash[12679]: audit 2022-04-22T17:22:16.141464+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/3495066243' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-04-22T17:22:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:16 smithi121 bash[12679]: audit 2022-04-22T17:22:16.421723+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.115:0/1190386242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:16 smithi121 bash[12679]: audit 2022-04-22T17:22:16.422492+0000 mon.a (mon.0) 590 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:17.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[18070]: audit 2022-04-22T17:22:15.863759+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/326145771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:17.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[18070]: audit 2022-04-22T17:22:16.141464+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/3495066243' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-04-22T17:22:17.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[18070]: audit 2022-04-22T17:22:16.421723+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.115:0/1190386242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:17.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[18070]: audit 2022-04-22T17:22:16.422492+0000 mon.a (mon.0) 590 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:17.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[12737]: audit 2022-04-22T17:22:15.863759+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/326145771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:22:17.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[12737]: audit 2022-04-22T17:22:16.141464+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/3495066243' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-04-22T17:22:17.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[12737]: audit 2022-04-22T17:22:16.421723+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.115:0/1190386242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:17.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:16 smithi115 bash[12737]: audit 2022-04-22T17:22:16.422492+0000 mon.a (mon.0) 590 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-04-22T17:22:18.105 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data' created 2022-04-22T17:22:18.123 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-04-22T17:22:18.436 INFO:teuthology.orchestra.run.smithi115.stderr:new fs with metadata pool 2 and data pool 3 2022-04-22T17:22:18.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[12737]: audit 2022-04-22T17:22:17.081677+0000 mon.a (mon.0) 591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-04-22T17:22:18.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[12737]: cluster 2022-04-22T17:22:17.081830+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2022-04-22T17:22:18.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[12737]: cluster 2022-04-22T17:22:17.083894+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v211: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:18.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[12737]: audit 2022-04-22T17:22:17.379864+0000 mon.a (mon.0) 593 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-04-22T17:22:18.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:22:18.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[18070]: audit 2022-04-22T17:22:17.081677+0000 mon.a (mon.0) 591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-04-22T17:22:18.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[18070]: cluster 2022-04-22T17:22:17.081830+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2022-04-22T17:22:18.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[18070]: cluster 2022-04-22T17:22:17.083894+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v211: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:18.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:18 smithi115 bash[18070]: audit 2022-04-22T17:22:17.379864+0000 mon.a (mon.0) 593 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-04-22T17:22:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:18 smithi121 bash[12679]: audit 2022-04-22T17:22:17.081677+0000 mon.a (mon.0) 591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-04-22T17:22:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:18 smithi121 bash[12679]: cluster 2022-04-22T17:22:17.081830+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2022-04-22T17:22:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:18 smithi121 bash[12679]: cluster 2022-04-22T17:22:17.083894+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v211: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:18 smithi121 bash[12679]: audit 2022-04-22T17:22:17.379864+0000 mon.a (mon.0) 593 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-04-22T17:22:18.696 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:18.696 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":53,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:22:18.412946+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:22:18.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:22:18.970 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:18.970 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":53,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:22:18.412946+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:22:18.984 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-04-22T17:22:19.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.094036+0000 mon.a (mon.0) 594 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-04-22T17:22:19.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: cluster 2022-04-22T17:22:18.094141+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2022-04-22T17:22:19.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.412104+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.115:0/997855505' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.412720+0000 mon.a (mon.0) 596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: cluster 2022-04-22T17:22:18.413748+0000 mon.a (mon.0) 597 : cluster [INF] daemon mds.b assigned to filesystem cephfs as rank 0 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: cluster 2022-04-22T17:22:18.422981+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.423476+0000 mon.a (mon.0) 599 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: cluster 2022-04-22T17:22:18.423655+0000 mon.a (mon.0) 600 : cluster [DBG] fsmap cephfs:1 {0=b=up:creating} 1 up:standby 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: cluster 2022-04-22T17:22:18.465000+0000 mon.a (mon.0) 601 : cluster [INF] daemon mds.b is now active in filesystem cephfs as rank 0 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.697272+0000 mon.a (mon.0) 602 : audit [DBG] from='client.? 172.21.15.115:0/3595171425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:19.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[12737]: audit 2022-04-22T17:22:18.971671+0000 mon.a (mon.0) 603 : audit [DBG] from='client.? 172.21.15.115:0/460635656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.094036+0000 mon.a (mon.0) 594 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: cluster 2022-04-22T17:22:18.094141+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.412104+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.115:0/997855505' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.412720+0000 mon.a (mon.0) 596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: cluster 2022-04-22T17:22:18.413748+0000 mon.a (mon.0) 597 : cluster [INF] daemon mds.b assigned to filesystem cephfs as rank 0 2022-04-22T17:22:19.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: cluster 2022-04-22T17:22:18.422981+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2022-04-22T17:22:19.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.423476+0000 mon.a (mon.0) 599 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-04-22T17:22:19.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: cluster 2022-04-22T17:22:18.423655+0000 mon.a (mon.0) 600 : cluster [DBG] fsmap cephfs:1 {0=b=up:creating} 1 up:standby 2022-04-22T17:22:19.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: cluster 2022-04-22T17:22:18.465000+0000 mon.a (mon.0) 601 : cluster [INF] daemon mds.b is now active in filesystem cephfs as rank 0 2022-04-22T17:22:19.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.697272+0000 mon.a (mon.0) 602 : audit [DBG] from='client.? 172.21.15.115:0/3595171425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:19.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:19 smithi115 bash[18070]: audit 2022-04-22T17:22:18.971671+0000 mon.a (mon.0) 603 : audit [DBG] from='client.? 172.21.15.115:0/460635656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:19.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2022-04-22T17:22:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.094036+0000 mon.a (mon.0) 594 : audit [INF] from='client.? 172.21.15.115:0/3273804310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: cluster 2022-04-22T17:22:18.094141+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.412104+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.115:0/997855505' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.412720+0000 mon.a (mon.0) 596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: cluster 2022-04-22T17:22:18.413748+0000 mon.a (mon.0) 597 : cluster [INF] daemon mds.b assigned to filesystem cephfs as rank 0 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: cluster 2022-04-22T17:22:18.422981+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.423476+0000 mon.a (mon.0) 599 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-04-22T17:22:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: cluster 2022-04-22T17:22:18.423655+0000 mon.a (mon.0) 600 : cluster [DBG] fsmap cephfs:1 {0=b=up:creating} 1 up:standby 2022-04-22T17:22:19.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: cluster 2022-04-22T17:22:18.465000+0000 mon.a (mon.0) 601 : cluster [INF] daemon mds.b is now active in filesystem cephfs as rank 0 2022-04-22T17:22:19.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.697272+0000 mon.a (mon.0) 602 : audit [DBG] from='client.? 172.21.15.115:0/3595171425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:19.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:19 smithi121 bash[12679]: audit 2022-04-22T17:22:18.971671+0000 mon.a (mon.0) 603 : audit [DBG] from='client.? 172.21.15.115:0/460635656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: cluster 2022-04-22T17:22:19.084573+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v214: 97 pgs: 23 creating+peering, 11 active+clean, 63 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: audit 2022-04-22T17:22:19.275512+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.115:0/570789632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: audit 2022-04-22T17:22:19.276111+0000 mon.a (mon.0) 604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: cluster 2022-04-22T17:22:19.426456+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2022-04-22T17:22:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: cluster 2022-04-22T17:22:19.430113+0000 mon.a (mon.0) 606 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:active 2022-04-22T17:22:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: audit 2022-04-22T17:22:19.430211+0000 mon.a (mon.0) 607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-04-22T17:22:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: cluster 2022-04-22T17:22:19.430332+0000 mon.a (mon.0) 608 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[18070]: audit 2022-04-22T17:22:19.752427+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2022-04-22T17:22:20.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: cluster 2022-04-22T17:22:19.084573+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v214: 97 pgs: 23 creating+peering, 11 active+clean, 63 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:20.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: audit 2022-04-22T17:22:19.275512+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.115:0/570789632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: audit 2022-04-22T17:22:19.276111+0000 mon.a (mon.0) 604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: cluster 2022-04-22T17:22:19.426456+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2022-04-22T17:22:20.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: cluster 2022-04-22T17:22:19.430113+0000 mon.a (mon.0) 606 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:active 2022-04-22T17:22:20.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: audit 2022-04-22T17:22:19.430211+0000 mon.a (mon.0) 607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-04-22T17:22:20.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: cluster 2022-04-22T17:22:19.430332+0000 mon.a (mon.0) 608 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:20.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:20 smithi115 bash[12737]: audit 2022-04-22T17:22:19.752427+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2022-04-22T17:22:20.456 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:22:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: cluster 2022-04-22T17:22:19.084573+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v214: 97 pgs: 23 creating+peering, 11 active+clean, 63 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:20.749 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: audit 2022-04-22T17:22:19.275512+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.115:0/570789632' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: audit 2022-04-22T17:22:19.276111+0000 mon.a (mon.0) 604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: cluster 2022-04-22T17:22:19.426456+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: cluster 2022-04-22T17:22:19.430113+0000 mon.a (mon.0) 606 : cluster [DBG] mds.? [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] up:active 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: audit 2022-04-22T17:22:19.430211+0000 mon.a (mon.0) 607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: cluster 2022-04-22T17:22:19.430332+0000 mon.a (mon.0) 608 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:20.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:20 smithi121 bash[12679]: audit 2022-04-22T17:22:19.752427+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2022-04-22T17:22:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:20.773 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.115:6835/3475839169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3475839169},{"type":"v1","addr":"172.21.15.115:6835","nonce":3475839169}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-22T17:22:18.413732+0000","modified":"2022-04-22T17:22:20.428010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24386},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24386":{"gid":24386,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6833/2211373279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2211373279},{"type":"v1","addr":"172.21.15.121:6833","nonce":2211373279}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-22T17:22:20.781 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 6 2022-04-22T17:22:20.782 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:22:21.042 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:21.057 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":54,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:22:19.416154+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:22:21.061 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-04-22T17:22:21.062 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-22T17:22:21.335 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:21.337 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:22:21.347 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2022-04-22T17:22:21.084838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":192,"acting":192,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":44156,"kb_used_data":3900,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":749917060,"statfs":{"total":767960285184,"available":767915069440,"internally_reserved":0,"allocated":3993600,"data_stored":1590190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.663445"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.443213+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:22:18.443213+0000","last_peered":"2022-04-22T17:22:18.443213+0000","last_clean":"2022-04-22T17:22:18.443213+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:22:18.443213+0000","last_undegraded":"2022-04-22T17:22:18.443213+0000","last_fullsized":"2022-04-22T17:22:18.443213+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456670+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:22:18.456670+0000","last_peered":"2022-04-22T17:22:18.456670+0000","last_clean":"2022-04-22T17:22:18.456670+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:22:18.456670+0000","last_undegraded":"2022-04-22T17:22:18.456670+0000","last_fullsized":"2022-04-22T17:22:18.456670+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103745+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103209+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090959+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.092429+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.447868+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:22:18.447868+0000","last_peered":"2022-04-22T17:22:18.447868+0000","last_clean":"2022-04-22T17:22:18.447868+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:22:18.447868+0000","last_undegraded":"2022-04-22T17:22:18.447868+0000","last_fullsized":"2022-04-22T17:22:18.447868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451210+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:22:18.451210+0000","last_peered":"2022-04-22T17:22:18.451210+0000","last_clean":"2022-04-22T17:22:18.451210+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:22:18.451210+0000","last_undegraded":"2022-04-22T17:22:18.451210+0000","last_fullsized":"2022-04-22T17:22:18.451210+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102678+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102095+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.442140+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:22:18.442140+0000","last_peered":"2022-04-22T17:22:18.442140+0000","last_clean":"2022-04-22T17:22:18.442140+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:22:18.442140+0000","last_undegraded":"2022-04-22T17:22:18.442140+0000","last_fullsized":"2022-04-22T17:22:18.442140+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.463153+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:22:18.463153+0000","last_peered":"2022-04-22T17:22:18.463153+0000","last_clean":"2022-04-22T17:22:18.463153+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:22:18.463153+0000","last_undegraded":"2022-04-22T17:22:18.463153+0000","last_fullsized":"2022-04-22T17:22:18.463153+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086285+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449275+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:22:18.449275+0000","last_peered":"2022-04-22T17:22:18.449275+0000","last_clean":"2022-04-22T17:22:18.449275+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:22:18.449275+0000","last_undegraded":"2022-04-22T17:22:18.449275+0000","last_fullsized":"2022-04-22T17:22:18.449275+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100151+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100854+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086116+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110705+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:22:18.110705+0000","last_peered":"2022-04-22T17:22:18.110705+0000","last_clean":"2022-04-22T17:22:18.110705+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:22:18.110705+0000","last_undegraded":"2022-04-22T17:22:18.110705+0000","last_fullsized":"2022-04-22T17:22:18.110705+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460074+0000","last_change":"2022-04-22T17:22:18.460074+0000","last_active":"2022-04-22T17:22:18.460074+0000","last_peered":"2022-04-22T17:22:18.460074+0000","last_clean":"2022-04-22T17:22:18.460074+0000","last_became_active":"2022-04-22T17:22:18.459771+0000","last_became_peered":"2022-04-22T17:22:18.459771+0000","last_unstale":"2022-04-22T17:22:18.460074+0000","last_undegraded":"2022-04-22T17:22:18.460074+0000","last_fullsized":"2022-04-22T17:22:18.460074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086019+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.111677+0000","last_change":"2022-04-22T17:22:18.111677+0000","last_active":"2022-04-22T17:22:18.111677+0000","last_peered":"2022-04-22T17:22:18.111677+0000","last_clean":"2022-04-22T17:22:18.111677+0000","last_became_active":"2022-04-22T17:22:18.111341+0000","last_became_peered":"2022-04-22T17:22:18.111341+0000","last_unstale":"2022-04-22T17:22:18.111677+0000","last_undegraded":"2022-04-22T17:22:18.111677+0000","last_fullsized":"2022-04-22T17:22:18.111677+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451206+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:22:18.451206+0000","last_peered":"2022-04-22T17:22:18.451206+0000","last_clean":"2022-04-22T17:22:18.451206+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:22:18.451206+0000","last_undegraded":"2022-04-22T17:22:18.451206+0000","last_fullsized":"2022-04-22T17:22:18.451206+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.098820+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:22:18.098820+0000","last_peered":"2022-04-22T17:22:18.098820+0000","last_clean":"2022-04-22T17:22:18.098820+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:22:18.098820+0000","last_undegraded":"2022-04-22T17:22:18.098820+0000","last_fullsized":"2022-04-22T17:22:18.098820+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099775+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103533+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:22:18.103533+0000","last_peered":"2022-04-22T17:22:18.103533+0000","last_clean":"2022-04-22T17:22:18.103533+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:22:18.103533+0000","last_undegraded":"2022-04-22T17:22:18.103533+0000","last_fullsized":"2022-04-22T17:22:18.103533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454571+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:22:18.454571+0000","last_peered":"2022-04-22T17:22:18.454571+0000","last_clean":"2022-04-22T17:22:18.454571+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:22:18.454571+0000","last_undegraded":"2022-04-22T17:22:18.454571+0000","last_fullsized":"2022-04-22T17:22:18.454571+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457018+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:22:18.457018+0000","last_peered":"2022-04-22T17:22:18.457018+0000","last_clean":"2022-04-22T17:22:18.457018+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:22:18.457018+0000","last_undegraded":"2022-04-22T17:22:18.457018+0000","last_fullsized":"2022-04-22T17:22:18.457018+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103415+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:22:18.103415+0000","last_peered":"2022-04-22T17:22:18.103415+0000","last_clean":"2022-04-22T17:22:18.103415+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:22:18.103415+0000","last_undegraded":"2022-04-22T17:22:18.103415+0000","last_fullsized":"2022-04-22T17:22:18.103415+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461078+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:22:18.461078+0000","last_peered":"2022-04-22T17:22:18.461078+0000","last_clean":"2022-04-22T17:22:18.461078+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:22:18.461078+0000","last_undegraded":"2022-04-22T17:22:18.461078+0000","last_fullsized":"2022-04-22T17:22:18.461078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460589+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:22:18.460589+0000","last_peered":"2022-04-22T17:22:18.460589+0000","last_clean":"2022-04-22T17:22:18.460589+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:22:18.460589+0000","last_undegraded":"2022-04-22T17:22:18.460589+0000","last_fullsized":"2022-04-22T17:22:18.460589+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.116687+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:22:18.116687+0000","last_peered":"2022-04-22T17:22:18.116687+0000","last_clean":"2022-04-22T17:22:18.116687+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:22:18.116687+0000","last_undegraded":"2022-04-22T17:22:18.116687+0000","last_fullsized":"2022-04-22T17:22:18.116687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.113792+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:22:18.113792+0000","last_peered":"2022-04-22T17:22:18.113792+0000","last_clean":"2022-04-22T17:22:18.113792+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:22:18.113792+0000","last_undegraded":"2022-04-22T17:22:18.113792+0000","last_fullsized":"2022-04-22T17:22:18.113792+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090974+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449939+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:22:18.449939+0000","last_peered":"2022-04-22T17:22:18.449939+0000","last_clean":"2022-04-22T17:22:18.449939+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:22:18.449939+0000","last_undegraded":"2022-04-22T17:22:18.449939+0000","last_fullsized":"2022-04-22T17:22:18.449939+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.084450+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.122198+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:22:18.122198+0000","last_peered":"2022-04-22T17:22:18.122198+0000","last_clean":"2022-04-22T17:22:18.122198+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:22:18.122198+0000","last_undegraded":"2022-04-22T17:22:18.122198+0000","last_fullsized":"2022-04-22T17:22:18.122198+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.098821+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.107633+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:22:18.107633+0000","last_peered":"2022-04-22T17:22:18.107633+0000","last_clean":"2022-04-22T17:22:18.107633+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:22:18.107633+0000","last_undegraded":"2022-04-22T17:22:18.107633+0000","last_fullsized":"2022-04-22T17:22:18.107633+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.431535+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:22:18.431535+0000","last_peered":"2022-04-22T17:22:18.431535+0000","last_clean":"2022-04-22T17:22:18.431535+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:22:18.431535+0000","last_undegraded":"2022-04-22T17:22:18.431535+0000","last_fullsized":"2022-04-22T17:22:18.431535+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.109388+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:22:18.109388+0000","last_peered":"2022-04-22T17:22:18.109388+0000","last_clean":"2022-04-22T17:22:18.109388+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:22:18.109388+0000","last_undegraded":"2022-04-22T17:22:18.109388+0000","last_fullsized":"2022-04-22T17:22:18.109388+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110776+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:22:18.110776+0000","last_peered":"2022-04-22T17:22:18.110776+0000","last_clean":"2022-04-22T17:22:18.110776+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:22:18.110776+0000","last_undegraded":"2022-04-22T17:22:18.110776+0000","last_fullsized":"2022-04-22T17:22:18.110776+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":50,"state":"active+clean","last_fresh":"2022-04-22T17:21:47.152297+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:21:47.152297+0000","last_peered":"2022-04-22T17:21:47.152297+0000","last_clean":"2022-04-22T17:21:47.152297+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:21:47.152297+0000","last_undegraded":"2022-04-22T17:21:47.152297+0000","last_fullsized":"2022-04-22T17:21:47.152297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450291+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:22:18.450291+0000","last_peered":"2022-04-22T17:22:18.450291+0000","last_clean":"2022-04-22T17:22:18.450291+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:22:18.450291+0000","last_undegraded":"2022-04-22T17:22:18.450291+0000","last_fullsized":"2022-04-22T17:22:18.450291+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457429+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:22:18.457429+0000","last_peered":"2022-04-22T17:22:18.457429+0000","last_clean":"2022-04-22T17:22:18.457429+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:22:18.457429+0000","last_undegraded":"2022-04-22T17:22:18.457429+0000","last_fullsized":"2022-04-22T17:22:18.457429+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.105155+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:22:18.105155+0000","last_peered":"2022-04-22T17:22:18.105155+0000","last_clean":"2022-04-22T17:22:18.105155+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:22:18.105155+0000","last_undegraded":"2022-04-22T17:22:18.105155+0000","last_fullsized":"2022-04-22T17:22:18.105155+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.112844+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:22:18.112844+0000","last_peered":"2022-04-22T17:22:18.112844+0000","last_clean":"2022-04-22T17:22:18.112844+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:22:18.112844+0000","last_undegraded":"2022-04-22T17:22:18.112844+0000","last_fullsized":"2022-04-22T17:22:18.112844+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.458267+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:22:18.458267+0000","last_peered":"2022-04-22T17:22:18.458267+0000","last_clean":"2022-04-22T17:22:18.458267+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:22:18.458267+0000","last_undegraded":"2022-04-22T17:22:18.458267+0000","last_fullsized":"2022-04-22T17:22:18.458267+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117892+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:22:18.117892+0000","last_peered":"2022-04-22T17:22:18.117892+0000","last_clean":"2022-04-22T17:22:18.117892+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:22:18.117892+0000","last_undegraded":"2022-04-22T17:22:18.117892+0000","last_fullsized":"2022-04-22T17:22:18.117892+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117219+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:22:18.117219+0000","last_peered":"2022-04-22T17:22:18.117219+0000","last_clean":"2022-04-22T17:22:18.117219+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:22:18.117219+0000","last_undegraded":"2022-04-22T17:22:18.117219+0000","last_fullsized":"2022-04-22T17:22:18.117219+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091917+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.115843+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:22:18.115843+0000","last_peered":"2022-04-22T17:22:18.115843+0000","last_clean":"2022-04-22T17:22:18.115843+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:22:18.115843+0000","last_undegraded":"2022-04-22T17:22:18.115843+0000","last_fullsized":"2022-04-22T17:22:18.115843+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.420716+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:22:18.420716+0000","last_peered":"2022-04-22T17:22:18.420716+0000","last_clean":"2022-04-22T17:22:18.420716+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:22:18.420716+0000","last_undegraded":"2022-04-22T17:22:18.420716+0000","last_fullsized":"2022-04-22T17:22:18.420716+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099615+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441302+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:22:18.441302+0000","last_peered":"2022-04-22T17:22:18.441302+0000","last_clean":"2022-04-22T17:22:18.441302+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:22:18.441302+0000","last_undegraded":"2022-04-22T17:22:18.441302+0000","last_fullsized":"2022-04-22T17:22:18.441302+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091622+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438590+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:22:18.438590+0000","last_peered":"2022-04-22T17:22:18.438590+0000","last_clean":"2022-04-22T17:22:18.438590+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:22:18.438590+0000","last_undegraded":"2022-04-22T17:22:18.438590+0000","last_fullsized":"2022-04-22T17:22:18.438590+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100804+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.459327+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:22:18.459327+0000","last_peered":"2022-04-22T17:22:18.459327+0000","last_clean":"2022-04-22T17:22:18.459327+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:22:18.459327+0000","last_undegraded":"2022-04-22T17:22:18.459327+0000","last_fullsized":"2022-04-22T17:22:18.459327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434698+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:22:18.434698+0000","last_peered":"2022-04-22T17:22:18.434698+0000","last_clean":"2022-04-22T17:22:18.434698+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:22:18.434698+0000","last_undegraded":"2022-04-22T17:22:18.434698+0000","last_fullsized":"2022-04-22T17:22:18.434698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091545+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101640+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101281+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460372+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:22:18.460372+0000","last_peered":"2022-04-22T17:22:18.460372+0000","last_clean":"2022-04-22T17:22:18.460372+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:22:18.460372+0000","last_undegraded":"2022-04-22T17:22:18.460372+0000","last_fullsized":"2022-04-22T17:22:18.460372+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091441+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":111,"acting":111,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":78,"acting":78,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5212,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]}]},{"osd":6,"up_from":42,"seq":180388626446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5348,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":487424,"data_stored":194843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5764,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739388,"statfs":{"total":95995035648,"available":95989133312,"internally_reserved":0,"allocated":520192,"data_stored":208745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.55600000000000005,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63600000000000001,"15min":0.57699999999999996},"min":{"1min":0.53700000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.94499999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.74399999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.68100000000000005},"min":{"1min":0.61399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":1.016},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.159},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.95499999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.754},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5904,"kb_used_data":520,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739248,"statfs":{"total":95995035648,"available":95988989952,"internally_reserved":0,"allocated":532480,"data_stored":209600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:22:12 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.55500000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:21:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.64400000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.95599999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Apr 22 17:21:58 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79200000000000004,"15min":0.79400000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.434},{"interface":"front","average":{"1min":0.873,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Apr 22 17:21:52 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5692,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":512000,"data_stored":208238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.2609999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.51000000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.622}]},{"osd":1,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.442,"15min":0.442},"max":{"1min":1.1739999999999999,"5min":1.294,"15min":1.294},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.78600000000000003,"15min":0.77500000000000002},"min":{"1min":0.66400000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Apr 22 17:22:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.57199999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0680000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.83499999999999996}]},{"osd":4,"last update":"Fri Apr 22 17:21:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.92200000000000004}]},{"osd":5,"last update":"Fri Apr 22 17:21:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.089},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182431,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.006,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.745,"15min":0.745},"min":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1060000000000001,"5min":1.133,"15min":1.133},"last":0.85799999999999998}]},{"osd":1,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.628,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1459999999999999,"5min":1.276,"15min":1.276},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.099,"5min":1.202,"15min":1.202},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.194,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.105,"5min":1.224,"15min":1.224},"last":1.1459999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":1.0249999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.1639999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":1.216},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":1.196}]},{"osd":3,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":1.0760000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5428,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739724,"statfs":{"total":95995035648,"available":95989477376,"internally_reserved":0,"allocated":503808,"data_stored":201779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94999999999999996}]},{"osd":1,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.91100000000000003}]},{"osd":2,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0389999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:22:21.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-22T17:22:21.668 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:21.672 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:22:21.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[18070]: audit 2022-04-22T17:22:20.435828+0000 mon.a (mon.0) 610 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2022-04-22T17:22:21.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[18070]: cluster 2022-04-22T17:22:20.436008+0000 mon.a (mon.0) 611 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:21.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[18070]: audit 2022-04-22T17:22:20.749346+0000 mon.a (mon.0) 612 : audit [DBG] from='client.? 172.21.15.115:0/3768726905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[18070]: audit 2022-04-22T17:22:21.044197+0000 mon.a (mon.0) 613 : audit [DBG] from='client.? 172.21.15.115:0/2898216989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[18070]: cluster 2022-04-22T17:22:21.085101+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v216: 97 pgs: 23 creating+peering, 41 active+clean, 33 unknown; 748 B data, 43 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:21.684 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[12737]: audit 2022-04-22T17:22:20.435828+0000 mon.a (mon.0) 610 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2022-04-22T17:22:21.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[12737]: cluster 2022-04-22T17:22:20.436008+0000 mon.a (mon.0) 611 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:21.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[12737]: audit 2022-04-22T17:22:20.749346+0000 mon.a (mon.0) 612 : audit [DBG] from='client.? 172.21.15.115:0/3768726905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[12737]: audit 2022-04-22T17:22:21.044197+0000 mon.a (mon.0) 613 : audit [DBG] from='client.? 172.21.15.115:0/2898216989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:21 smithi115 bash[12737]: cluster 2022-04-22T17:22:21.085101+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v216: 97 pgs: 23 creating+peering, 41 active+clean, 33 unknown; 748 B data, 43 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:21.686 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2022-04-22T17:22:21.084838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":192,"acting":192,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":44156,"kb_used_data":3900,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":749917060,"statfs":{"total":767960285184,"available":767915069440,"internally_reserved":0,"allocated":3993600,"data_stored":1590190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.663445"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.443213+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:22:18.443213+0000","last_peered":"2022-04-22T17:22:18.443213+0000","last_clean":"2022-04-22T17:22:18.443213+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:22:18.443213+0000","last_undegraded":"2022-04-22T17:22:18.443213+0000","last_fullsized":"2022-04-22T17:22:18.443213+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456670+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:22:18.456670+0000","last_peered":"2022-04-22T17:22:18.456670+0000","last_clean":"2022-04-22T17:22:18.456670+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:22:18.456670+0000","last_undegraded":"2022-04-22T17:22:18.456670+0000","last_fullsized":"2022-04-22T17:22:18.456670+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103745+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103209+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090959+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.092429+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.447868+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:22:18.447868+0000","last_peered":"2022-04-22T17:22:18.447868+0000","last_clean":"2022-04-22T17:22:18.447868+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:22:18.447868+0000","last_undegraded":"2022-04-22T17:22:18.447868+0000","last_fullsized":"2022-04-22T17:22:18.447868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451210+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:22:18.451210+0000","last_peered":"2022-04-22T17:22:18.451210+0000","last_clean":"2022-04-22T17:22:18.451210+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:22:18.451210+0000","last_undegraded":"2022-04-22T17:22:18.451210+0000","last_fullsized":"2022-04-22T17:22:18.451210+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102678+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102095+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.442140+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:22:18.442140+0000","last_peered":"2022-04-22T17:22:18.442140+0000","last_clean":"2022-04-22T17:22:18.442140+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:22:18.442140+0000","last_undegraded":"2022-04-22T17:22:18.442140+0000","last_fullsized":"2022-04-22T17:22:18.442140+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.463153+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:22:18.463153+0000","last_peered":"2022-04-22T17:22:18.463153+0000","last_clean":"2022-04-22T17:22:18.463153+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:22:18.463153+0000","last_undegraded":"2022-04-22T17:22:18.463153+0000","last_fullsized":"2022-04-22T17:22:18.463153+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086285+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449275+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:22:18.449275+0000","last_peered":"2022-04-22T17:22:18.449275+0000","last_clean":"2022-04-22T17:22:18.449275+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:22:18.449275+0000","last_undegraded":"2022-04-22T17:22:18.449275+0000","last_fullsized":"2022-04-22T17:22:18.449275+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100151+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100854+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086116+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110705+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:22:18.110705+0000","last_peered":"2022-04-22T17:22:18.110705+0000","last_clean":"2022-04-22T17:22:18.110705+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:22:18.110705+0000","last_undegraded":"2022-04-22T17:22:18.110705+0000","last_fullsized":"2022-04-22T17:22:18.110705+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460074+0000","last_change":"2022-04-22T17:22:18.460074+0000","last_active":"2022-04-22T17:22:18.460074+0000","last_peered":"2022-04-22T17:22:18.460074+0000","last_clean":"2022-04-22T17:22:18.460074+0000","last_became_active":"2022-04-22T17:22:18.459771+0000","last_became_peered":"2022-04-22T17:22:18.459771+0000","last_unstale":"2022-04-22T17:22:18.460074+0000","last_undegraded":"2022-04-22T17:22:18.460074+0000","last_fullsized":"2022-04-22T17:22:18.460074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086019+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.111677+0000","last_change":"2022-04-22T17:22:18.111677+0000","last_active":"2022-04-22T17:22:18.111677+0000","last_peered":"2022-04-22T17:22:18.111677+0000","last_clean":"2022-04-22T17:22:18.111677+0000","last_became_active":"2022-04-22T17:22:18.111341+0000","last_became_peered":"2022-04-22T17:22:18.111341+0000","last_unstale":"2022-04-22T17:22:18.111677+0000","last_undegraded":"2022-04-22T17:22:18.111677+0000","last_fullsized":"2022-04-22T17:22:18.111677+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451206+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:22:18.451206+0000","last_peered":"2022-04-22T17:22:18.451206+0000","last_clean":"2022-04-22T17:22:18.451206+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:22:18.451206+0000","last_undegraded":"2022-04-22T17:22:18.451206+0000","last_fullsized":"2022-04-22T17:22:18.451206+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.098820+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:22:18.098820+0000","last_peered":"2022-04-22T17:22:18.098820+0000","last_clean":"2022-04-22T17:22:18.098820+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:22:18.098820+0000","last_undegraded":"2022-04-22T17:22:18.098820+0000","last_fullsized":"2022-04-22T17:22:18.098820+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099775+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103533+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:22:18.103533+0000","last_peered":"2022-04-22T17:22:18.103533+0000","last_clean":"2022-04-22T17:22:18.103533+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:22:18.103533+0000","last_undegraded":"2022-04-22T17:22:18.103533+0000","last_fullsized":"2022-04-22T17:22:18.103533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454571+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:22:18.454571+0000","last_peered":"2022-04-22T17:22:18.454571+0000","last_clean":"2022-04-22T17:22:18.454571+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:22:18.454571+0000","last_undegraded":"2022-04-22T17:22:18.454571+0000","last_fullsized":"2022-04-22T17:22:18.454571+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457018+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:22:18.457018+0000","last_peered":"2022-04-22T17:22:18.457018+0000","last_clean":"2022-04-22T17:22:18.457018+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:22:18.457018+0000","last_undegraded":"2022-04-22T17:22:18.457018+0000","last_fullsized":"2022-04-22T17:22:18.457018+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103415+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:22:18.103415+0000","last_peered":"2022-04-22T17:22:18.103415+0000","last_clean":"2022-04-22T17:22:18.103415+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:22:18.103415+0000","last_undegraded":"2022-04-22T17:22:18.103415+0000","last_fullsized":"2022-04-22T17:22:18.103415+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461078+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:22:18.461078+0000","last_peered":"2022-04-22T17:22:18.461078+0000","last_clean":"2022-04-22T17:22:18.461078+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:22:18.461078+0000","last_undegraded":"2022-04-22T17:22:18.461078+0000","last_fullsized":"2022-04-22T17:22:18.461078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460589+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:22:18.460589+0000","last_peered":"2022-04-22T17:22:18.460589+0000","last_clean":"2022-04-22T17:22:18.460589+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:22:18.460589+0000","last_undegraded":"2022-04-22T17:22:18.460589+0000","last_fullsized":"2022-04-22T17:22:18.460589+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.116687+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:22:18.116687+0000","last_peered":"2022-04-22T17:22:18.116687+0000","last_clean":"2022-04-22T17:22:18.116687+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:22:18.116687+0000","last_undegraded":"2022-04-22T17:22:18.116687+0000","last_fullsized":"2022-04-22T17:22:18.116687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.113792+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:22:18.113792+0000","last_peered":"2022-04-22T17:22:18.113792+0000","last_clean":"2022-04-22T17:22:18.113792+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:22:18.113792+0000","last_undegraded":"2022-04-22T17:22:18.113792+0000","last_fullsized":"2022-04-22T17:22:18.113792+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090974+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449939+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:22:18.449939+0000","last_peered":"2022-04-22T17:22:18.449939+0000","last_clean":"2022-04-22T17:22:18.449939+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:22:18.449939+0000","last_undegraded":"2022-04-22T17:22:18.449939+0000","last_fullsized":"2022-04-22T17:22:18.449939+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.084450+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.122198+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:22:18.122198+0000","last_peered":"2022-04-22T17:22:18.122198+0000","last_clean":"2022-04-22T17:22:18.122198+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:22:18.122198+0000","last_undegraded":"2022-04-22T17:22:18.122198+0000","last_fullsized":"2022-04-22T17:22:18.122198+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.098821+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.107633+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:22:18.107633+0000","last_peered":"2022-04-22T17:22:18.107633+0000","last_clean":"2022-04-22T17:22:18.107633+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:22:18.107633+0000","last_undegraded":"2022-04-22T17:22:18.107633+0000","last_fullsized":"2022-04-22T17:22:18.107633+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.431535+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:22:18.431535+0000","last_peered":"2022-04-22T17:22:18.431535+0000","last_clean":"2022-04-22T17:22:18.431535+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:22:18.431535+0000","last_undegraded":"2022-04-22T17:22:18.431535+0000","last_fullsized":"2022-04-22T17:22:18.431535+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.109388+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:22:18.109388+0000","last_peered":"2022-04-22T17:22:18.109388+0000","last_clean":"2022-04-22T17:22:18.109388+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:22:18.109388+0000","last_undegraded":"2022-04-22T17:22:18.109388+0000","last_fullsized":"2022-04-22T17:22:18.109388+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110776+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:22:18.110776+0000","last_peered":"2022-04-22T17:22:18.110776+0000","last_clean":"2022-04-22T17:22:18.110776+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:22:18.110776+0000","last_undegraded":"2022-04-22T17:22:18.110776+0000","last_fullsized":"2022-04-22T17:22:18.110776+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":50,"state":"active+clean","last_fresh":"2022-04-22T17:21:47.152297+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:21:47.152297+0000","last_peered":"2022-04-22T17:21:47.152297+0000","last_clean":"2022-04-22T17:21:47.152297+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:21:47.152297+0000","last_undegraded":"2022-04-22T17:21:47.152297+0000","last_fullsized":"2022-04-22T17:21:47.152297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450291+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:22:18.450291+0000","last_peered":"2022-04-22T17:22:18.450291+0000","last_clean":"2022-04-22T17:22:18.450291+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:22:18.450291+0000","last_undegraded":"2022-04-22T17:22:18.450291+0000","last_fullsized":"2022-04-22T17:22:18.450291+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457429+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:22:18.457429+0000","last_peered":"2022-04-22T17:22:18.457429+0000","last_clean":"2022-04-22T17:22:18.457429+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:22:18.457429+0000","last_undegraded":"2022-04-22T17:22:18.457429+0000","last_fullsized":"2022-04-22T17:22:18.457429+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.105155+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:22:18.105155+0000","last_peered":"2022-04-22T17:22:18.105155+0000","last_clean":"2022-04-22T17:22:18.105155+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:22:18.105155+0000","last_undegraded":"2022-04-22T17:22:18.105155+0000","last_fullsized":"2022-04-22T17:22:18.105155+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.112844+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:22:18.112844+0000","last_peered":"2022-04-22T17:22:18.112844+0000","last_clean":"2022-04-22T17:22:18.112844+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:22:18.112844+0000","last_undegraded":"2022-04-22T17:22:18.112844+0000","last_fullsized":"2022-04-22T17:22:18.112844+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.458267+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:22:18.458267+0000","last_peered":"2022-04-22T17:22:18.458267+0000","last_clean":"2022-04-22T17:22:18.458267+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:22:18.458267+0000","last_undegraded":"2022-04-22T17:22:18.458267+0000","last_fullsized":"2022-04-22T17:22:18.458267+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117892+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:22:18.117892+0000","last_peered":"2022-04-22T17:22:18.117892+0000","last_clean":"2022-04-22T17:22:18.117892+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:22:18.117892+0000","last_undegraded":"2022-04-22T17:22:18.117892+0000","last_fullsized":"2022-04-22T17:22:18.117892+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117219+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:22:18.117219+0000","last_peered":"2022-04-22T17:22:18.117219+0000","last_clean":"2022-04-22T17:22:18.117219+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:22:18.117219+0000","last_undegraded":"2022-04-22T17:22:18.117219+0000","last_fullsized":"2022-04-22T17:22:18.117219+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091917+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.115843+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:22:18.115843+0000","last_peered":"2022-04-22T17:22:18.115843+0000","last_clean":"2022-04-22T17:22:18.115843+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:22:18.115843+0000","last_undegraded":"2022-04-22T17:22:18.115843+0000","last_fullsized":"2022-04-22T17:22:18.115843+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.420716+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:22:18.420716+0000","last_peered":"2022-04-22T17:22:18.420716+0000","last_clean":"2022-04-22T17:22:18.420716+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:22:18.420716+0000","last_undegraded":"2022-04-22T17:22:18.420716+0000","last_fullsized":"2022-04-22T17:22:18.420716+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099615+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441302+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:22:18.441302+0000","last_peered":"2022-04-22T17:22:18.441302+0000","last_clean":"2022-04-22T17:22:18.441302+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:22:18.441302+0000","last_undegraded":"2022-04-22T17:22:18.441302+0000","last_fullsized":"2022-04-22T17:22:18.441302+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091622+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438590+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:22:18.438590+0000","last_peered":"2022-04-22T17:22:18.438590+0000","last_clean":"2022-04-22T17:22:18.438590+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:22:18.438590+0000","last_undegraded":"2022-04-22T17:22:18.438590+0000","last_fullsized":"2022-04-22T17:22:18.438590+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100804+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.459327+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:22:18.459327+0000","last_peered":"2022-04-22T17:22:18.459327+0000","last_clean":"2022-04-22T17:22:18.459327+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:22:18.459327+0000","last_undegraded":"2022-04-22T17:22:18.459327+0000","last_fullsized":"2022-04-22T17:22:18.459327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434698+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:22:18.434698+0000","last_peered":"2022-04-22T17:22:18.434698+0000","last_clean":"2022-04-22T17:22:18.434698+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:22:18.434698+0000","last_undegraded":"2022-04-22T17:22:18.434698+0000","last_fullsized":"2022-04-22T17:22:18.434698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091545+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101640+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101281+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460372+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:22:18.460372+0000","last_peered":"2022-04-22T17:22:18.460372+0000","last_clean":"2022-04-22T17:22:18.460372+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:22:18.460372+0000","last_undegraded":"2022-04-22T17:22:18.460372+0000","last_fullsized":"2022-04-22T17:22:18.460372+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091441+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":111,"acting":111,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":78,"acting":78,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5212,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]}]},{"osd":6,"up_from":42,"seq":180388626446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5348,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":487424,"data_stored":194843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5764,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739388,"statfs":{"total":95995035648,"available":95989133312,"internally_reserved":0,"allocated":520192,"data_stored":208745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.55600000000000005,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63600000000000001,"15min":0.57699999999999996},"min":{"1min":0.53700000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.94499999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.74399999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.68100000000000005},"min":{"1min":0.61399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":1.016},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.159},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.95499999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.754},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5904,"kb_used_data":520,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739248,"statfs":{"total":95995035648,"available":95988989952,"internally_reserved":0,"allocated":532480,"data_stored":209600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:22:12 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.55500000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:21:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.64400000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.95599999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Apr 22 17:21:58 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79200000000000004,"15min":0.79400000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.434},{"interface":"front","average":{"1min":0.873,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Apr 22 17:21:52 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5692,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":512000,"data_stored":208238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.2609999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.51000000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.622}]},{"osd":1,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.442,"15min":0.442},"max":{"1min":1.1739999999999999,"5min":1.294,"15min":1.294},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.78600000000000003,"15min":0.77500000000000002},"min":{"1min":0.66400000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Apr 22 17:22:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.57199999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0680000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.83499999999999996}]},{"osd":4,"last update":"Fri Apr 22 17:21:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.92200000000000004}]},{"osd":5,"last update":"Fri Apr 22 17:21:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.089},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182431,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.006,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.745,"15min":0.745},"min":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1060000000000001,"5min":1.133,"15min":1.133},"last":0.85799999999999998}]},{"osd":1,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.628,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1459999999999999,"5min":1.276,"15min":1.276},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.099,"5min":1.202,"15min":1.202},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.194,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.105,"5min":1.224,"15min":1.224},"last":1.1459999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":1.0249999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.1639999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":1.216},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":1.196}]},{"osd":3,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":1.0760000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5428,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739724,"statfs":{"total":95995035648,"available":95989477376,"internally_reserved":0,"allocated":503808,"data_stored":201779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94999999999999996}]},{"osd":1,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.91100000000000003}]},{"osd":2,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0389999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:22:21.725 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[25529]: debug 2022-04-22T17:22:21.573+0000 7f1480038700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[12679]: audit 2022-04-22T17:22:20.435828+0000 mon.a (mon.0) 610 : audit [INF] from='client.? 172.21.15.115:0/2660046004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[12679]: cluster 2022-04-22T17:22:20.436008+0000 mon.a (mon.0) 611 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[12679]: audit 2022-04-22T17:22:20.749346+0000 mon.a (mon.0) 612 : audit [DBG] from='client.? 172.21.15.115:0/3768726905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[12679]: audit 2022-04-22T17:22:21.044197+0000 mon.a (mon.0) 613 : audit [DBG] from='client.? 172.21.15.115:0/2898216989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:22:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:21 smithi121 bash[12679]: cluster 2022-04-22T17:22:21.085101+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v216: 97 pgs: 23 creating+peering, 41 active+clean, 33 unknown; 748 B data, 43 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:22.033 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:22.033 INFO:teuthology.orchestra.run.smithi115.stdout:{"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":283,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1650648105,"num_in_osds":8,"osd_in_since":1650648083,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":63},{"state_name":"creating+peering","count":23},{"state_name":"active+clean","count":11}],"num_pgs":97,"num_pools":3,"num_objects":0,"data_bytes":0,"bytes_used":44425216,"bytes_avail":767915859968,"bytes_total":767960285184,"unknown_pgs_ratio":0.64948451519012451,"inactive_pgs_ratio":0.23711340129375458},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24386}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.115:8443/"}},"servicemap":{"epoch":12,"modified":"2022-04-22T17:22:19.084250+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2022-04-22T17:22:22.046 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2022-04-22T17:22:22.047 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-22T17:22:22.324 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:22.326 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:22:22.335 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2022-04-22T17:22:21.084838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":192,"acting":192,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":44156,"kb_used_data":3900,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":749917060,"statfs":{"total":767960285184,"available":767915069440,"internally_reserved":0,"allocated":3993600,"data_stored":1590190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.663445"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.443213+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:22:18.443213+0000","last_peered":"2022-04-22T17:22:18.443213+0000","last_clean":"2022-04-22T17:22:18.443213+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:22:18.443213+0000","last_undegraded":"2022-04-22T17:22:18.443213+0000","last_fullsized":"2022-04-22T17:22:18.443213+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456670+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:22:18.456670+0000","last_peered":"2022-04-22T17:22:18.456670+0000","last_clean":"2022-04-22T17:22:18.456670+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:22:18.456670+0000","last_undegraded":"2022-04-22T17:22:18.456670+0000","last_fullsized":"2022-04-22T17:22:18.456670+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103745+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.103209+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090959+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.092429+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.447868+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:22:18.447868+0000","last_peered":"2022-04-22T17:22:18.447868+0000","last_clean":"2022-04-22T17:22:18.447868+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:22:18.447868+0000","last_undegraded":"2022-04-22T17:22:18.447868+0000","last_fullsized":"2022-04-22T17:22:18.447868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451210+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:22:18.451210+0000","last_peered":"2022-04-22T17:22:18.451210+0000","last_clean":"2022-04-22T17:22:18.451210+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:22:18.451210+0000","last_undegraded":"2022-04-22T17:22:18.451210+0000","last_fullsized":"2022-04-22T17:22:18.451210+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102678+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.102095+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.442140+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:22:18.442140+0000","last_peered":"2022-04-22T17:22:18.442140+0000","last_clean":"2022-04-22T17:22:18.442140+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:22:18.442140+0000","last_undegraded":"2022-04-22T17:22:18.442140+0000","last_fullsized":"2022-04-22T17:22:18.442140+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.463153+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:22:18.463153+0000","last_peered":"2022-04-22T17:22:18.463153+0000","last_clean":"2022-04-22T17:22:18.463153+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:22:18.463153+0000","last_undegraded":"2022-04-22T17:22:18.463153+0000","last_fullsized":"2022-04-22T17:22:18.463153+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086285+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449275+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:22:18.449275+0000","last_peered":"2022-04-22T17:22:18.449275+0000","last_clean":"2022-04-22T17:22:18.449275+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:22:18.449275+0000","last_undegraded":"2022-04-22T17:22:18.449275+0000","last_fullsized":"2022-04-22T17:22:18.449275+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100151+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100854+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086116+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110705+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:22:18.110705+0000","last_peered":"2022-04-22T17:22:18.110705+0000","last_clean":"2022-04-22T17:22:18.110705+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:22:18.110705+0000","last_undegraded":"2022-04-22T17:22:18.110705+0000","last_fullsized":"2022-04-22T17:22:18.110705+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460074+0000","last_change":"2022-04-22T17:22:18.460074+0000","last_active":"2022-04-22T17:22:18.460074+0000","last_peered":"2022-04-22T17:22:18.460074+0000","last_clean":"2022-04-22T17:22:18.460074+0000","last_became_active":"2022-04-22T17:22:18.459771+0000","last_became_peered":"2022-04-22T17:22:18.459771+0000","last_unstale":"2022-04-22T17:22:18.460074+0000","last_undegraded":"2022-04-22T17:22:18.460074+0000","last_fullsized":"2022-04-22T17:22:18.460074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.086019+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.111677+0000","last_change":"2022-04-22T17:22:18.111677+0000","last_active":"2022-04-22T17:22:18.111677+0000","last_peered":"2022-04-22T17:22:18.111677+0000","last_clean":"2022-04-22T17:22:18.111677+0000","last_became_active":"2022-04-22T17:22:18.111341+0000","last_became_peered":"2022-04-22T17:22:18.111341+0000","last_unstale":"2022-04-22T17:22:18.111677+0000","last_undegraded":"2022-04-22T17:22:18.111677+0000","last_fullsized":"2022-04-22T17:22:18.111677+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451206+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:22:18.451206+0000","last_peered":"2022-04-22T17:22:18.451206+0000","last_clean":"2022-04-22T17:22:18.451206+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:22:18.451206+0000","last_undegraded":"2022-04-22T17:22:18.451206+0000","last_fullsized":"2022-04-22T17:22:18.451206+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.098820+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:22:18.098820+0000","last_peered":"2022-04-22T17:22:18.098820+0000","last_clean":"2022-04-22T17:22:18.098820+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:22:18.098820+0000","last_undegraded":"2022-04-22T17:22:18.098820+0000","last_fullsized":"2022-04-22T17:22:18.098820+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099775+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103533+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:22:18.103533+0000","last_peered":"2022-04-22T17:22:18.103533+0000","last_clean":"2022-04-22T17:22:18.103533+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:22:18.103533+0000","last_undegraded":"2022-04-22T17:22:18.103533+0000","last_fullsized":"2022-04-22T17:22:18.103533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454571+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:22:18.454571+0000","last_peered":"2022-04-22T17:22:18.454571+0000","last_clean":"2022-04-22T17:22:18.454571+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:22:18.454571+0000","last_undegraded":"2022-04-22T17:22:18.454571+0000","last_fullsized":"2022-04-22T17:22:18.454571+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457018+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:22:18.457018+0000","last_peered":"2022-04-22T17:22:18.457018+0000","last_clean":"2022-04-22T17:22:18.457018+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:22:18.457018+0000","last_undegraded":"2022-04-22T17:22:18.457018+0000","last_fullsized":"2022-04-22T17:22:18.457018+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.103415+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:22:18.103415+0000","last_peered":"2022-04-22T17:22:18.103415+0000","last_clean":"2022-04-22T17:22:18.103415+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:22:18.103415+0000","last_undegraded":"2022-04-22T17:22:18.103415+0000","last_fullsized":"2022-04-22T17:22:18.103415+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461078+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:22:18.461078+0000","last_peered":"2022-04-22T17:22:18.461078+0000","last_clean":"2022-04-22T17:22:18.461078+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:22:18.461078+0000","last_undegraded":"2022-04-22T17:22:18.461078+0000","last_fullsized":"2022-04-22T17:22:18.461078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460589+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:22:18.460589+0000","last_peered":"2022-04-22T17:22:18.460589+0000","last_clean":"2022-04-22T17:22:18.460589+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:22:18.460589+0000","last_undegraded":"2022-04-22T17:22:18.460589+0000","last_fullsized":"2022-04-22T17:22:18.460589+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.116687+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:22:18.116687+0000","last_peered":"2022-04-22T17:22:18.116687+0000","last_clean":"2022-04-22T17:22:18.116687+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:22:18.116687+0000","last_undegraded":"2022-04-22T17:22:18.116687+0000","last_fullsized":"2022-04-22T17:22:18.116687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.113792+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:22:18.113792+0000","last_peered":"2022-04-22T17:22:18.113792+0000","last_clean":"2022-04-22T17:22:18.113792+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:22:18.113792+0000","last_undegraded":"2022-04-22T17:22:18.113792+0000","last_fullsized":"2022-04-22T17:22:18.113792+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.090974+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449939+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:22:18.449939+0000","last_peered":"2022-04-22T17:22:18.449939+0000","last_clean":"2022-04-22T17:22:18.449939+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:22:18.449939+0000","last_undegraded":"2022-04-22T17:22:18.449939+0000","last_fullsized":"2022-04-22T17:22:18.449939+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.084450+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.122198+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:22:18.122198+0000","last_peered":"2022-04-22T17:22:18.122198+0000","last_clean":"2022-04-22T17:22:18.122198+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:22:18.122198+0000","last_undegraded":"2022-04-22T17:22:18.122198+0000","last_fullsized":"2022-04-22T17:22:18.122198+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.098821+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.107633+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:22:18.107633+0000","last_peered":"2022-04-22T17:22:18.107633+0000","last_clean":"2022-04-22T17:22:18.107633+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:22:18.107633+0000","last_undegraded":"2022-04-22T17:22:18.107633+0000","last_fullsized":"2022-04-22T17:22:18.107633+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.431535+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:22:18.431535+0000","last_peered":"2022-04-22T17:22:18.431535+0000","last_clean":"2022-04-22T17:22:18.431535+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:22:18.431535+0000","last_undegraded":"2022-04-22T17:22:18.431535+0000","last_fullsized":"2022-04-22T17:22:18.431535+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.109388+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:22:18.109388+0000","last_peered":"2022-04-22T17:22:18.109388+0000","last_clean":"2022-04-22T17:22:18.109388+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:22:18.109388+0000","last_undegraded":"2022-04-22T17:22:18.109388+0000","last_fullsized":"2022-04-22T17:22:18.109388+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.110776+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:22:18.110776+0000","last_peered":"2022-04-22T17:22:18.110776+0000","last_clean":"2022-04-22T17:22:18.110776+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:22:18.110776+0000","last_undegraded":"2022-04-22T17:22:18.110776+0000","last_fullsized":"2022-04-22T17:22:18.110776+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":50,"state":"active+clean","last_fresh":"2022-04-22T17:21:47.152297+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:21:47.152297+0000","last_peered":"2022-04-22T17:21:47.152297+0000","last_clean":"2022-04-22T17:21:47.152297+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:21:47.152297+0000","last_undegraded":"2022-04-22T17:21:47.152297+0000","last_fullsized":"2022-04-22T17:21:47.152297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450291+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:22:18.450291+0000","last_peered":"2022-04-22T17:22:18.450291+0000","last_clean":"2022-04-22T17:22:18.450291+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:22:18.450291+0000","last_undegraded":"2022-04-22T17:22:18.450291+0000","last_fullsized":"2022-04-22T17:22:18.450291+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457429+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:22:18.457429+0000","last_peered":"2022-04-22T17:22:18.457429+0000","last_clean":"2022-04-22T17:22:18.457429+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:22:18.457429+0000","last_undegraded":"2022-04-22T17:22:18.457429+0000","last_fullsized":"2022-04-22T17:22:18.457429+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.105155+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:22:18.105155+0000","last_peered":"2022-04-22T17:22:18.105155+0000","last_clean":"2022-04-22T17:22:18.105155+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:22:18.105155+0000","last_undegraded":"2022-04-22T17:22:18.105155+0000","last_fullsized":"2022-04-22T17:22:18.105155+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.112844+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:22:18.112844+0000","last_peered":"2022-04-22T17:22:18.112844+0000","last_clean":"2022-04-22T17:22:18.112844+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:22:18.112844+0000","last_undegraded":"2022-04-22T17:22:18.112844+0000","last_fullsized":"2022-04-22T17:22:18.112844+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.458267+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:22:18.458267+0000","last_peered":"2022-04-22T17:22:18.458267+0000","last_clean":"2022-04-22T17:22:18.458267+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:22:18.458267+0000","last_undegraded":"2022-04-22T17:22:18.458267+0000","last_fullsized":"2022-04-22T17:22:18.458267+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117892+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:22:18.117892+0000","last_peered":"2022-04-22T17:22:18.117892+0000","last_clean":"2022-04-22T17:22:18.117892+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:22:18.117892+0000","last_undegraded":"2022-04-22T17:22:18.117892+0000","last_fullsized":"2022-04-22T17:22:18.117892+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.117219+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:22:18.117219+0000","last_peered":"2022-04-22T17:22:18.117219+0000","last_clean":"2022-04-22T17:22:18.117219+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:22:18.117219+0000","last_undegraded":"2022-04-22T17:22:18.117219+0000","last_fullsized":"2022-04-22T17:22:18.117219+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:17.074592+0000","last_change":"2022-04-22T17:22:17.074592+0000","last_active":"2022-04-22T17:22:17.074592+0000","last_peered":"2022-04-22T17:22:17.074592+0000","last_clean":"2022-04-22T17:22:17.074592+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:17.074592+0000","last_undegraded":"2022-04-22T17:22:17.074592+0000","last_fullsized":"2022-04-22T17:22:17.074592+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091917+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":52,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.115843+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:22:18.115843+0000","last_peered":"2022-04-22T17:22:18.115843+0000","last_clean":"2022-04-22T17:22:18.115843+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:22:18.115843+0000","last_undegraded":"2022-04-22T17:22:18.115843+0000","last_fullsized":"2022-04-22T17:22:18.115843+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.420716+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:22:18.420716+0000","last_peered":"2022-04-22T17:22:18.420716+0000","last_clean":"2022-04-22T17:22:18.420716+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:22:18.420716+0000","last_undegraded":"2022-04-22T17:22:18.420716+0000","last_fullsized":"2022-04-22T17:22:18.420716+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.099615+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441302+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:22:18.441302+0000","last_peered":"2022-04-22T17:22:18.441302+0000","last_clean":"2022-04-22T17:22:18.441302+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:22:18.441302+0000","last_undegraded":"2022-04-22T17:22:18.441302+0000","last_fullsized":"2022-04-22T17:22:18.441302+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091622+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438590+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:22:18.438590+0000","last_peered":"2022-04-22T17:22:18.438590+0000","last_clean":"2022-04-22T17:22:18.438590+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:22:18.438590+0000","last_undegraded":"2022-04-22T17:22:18.438590+0000","last_fullsized":"2022-04-22T17:22:18.438590+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.100804+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.459327+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:22:18.459327+0000","last_peered":"2022-04-22T17:22:18.459327+0000","last_clean":"2022-04-22T17:22:18.459327+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:22:18.459327+0000","last_undegraded":"2022-04-22T17:22:18.459327+0000","last_fullsized":"2022-04-22T17:22:18.459327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434698+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:22:18.434698+0000","last_peered":"2022-04-22T17:22:18.434698+0000","last_clean":"2022-04-22T17:22:18.434698+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:22:18.434698+0000","last_undegraded":"2022-04-22T17:22:18.434698+0000","last_fullsized":"2022-04-22T17:22:18.434698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091545+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101640+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.078190+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.101281+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460372+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:22:18.460372+0000","last_peered":"2022-04-22T17:22:18.460372+0000","last_clean":"2022-04-22T17:22:18.460372+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:22:18.460372+0000","last_undegraded":"2022-04-22T17:22:18.460372+0000","last_fullsized":"2022-04-22T17:22:18.460372+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2022-04-22T17:22:18.078190+0000","last_change":"2022-04-22T17:22:18.091441+0000","last_active":"2022-04-22T17:22:18.078190+0000","last_peered":"2022-04-22T17:22:18.078190+0000","last_clean":"2022-04-22T17:22:18.078190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-04-22T17:22:18.078190+0000","last_undegraded":"2022-04-22T17:22:18.078190+0000","last_fullsized":"2022-04-22T17:22:18.078190+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":111,"acting":111,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":748,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":78,"acting":78,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5212,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]}]},{"osd":6,"up_from":42,"seq":180388626446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5348,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":487424,"data_stored":194843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5764,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739388,"statfs":{"total":95995035648,"available":95989133312,"internally_reserved":0,"allocated":520192,"data_stored":208745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.55600000000000005,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63600000000000001,"15min":0.57699999999999996},"min":{"1min":0.53700000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.94499999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.74399999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.68100000000000005},"min":{"1min":0.61399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":1.016},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.159},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.95499999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.754},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5904,"kb_used_data":520,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739248,"statfs":{"total":95995035648,"available":95988989952,"internally_reserved":0,"allocated":532480,"data_stored":209600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:22:12 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.55500000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:21:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.64400000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.95599999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Apr 22 17:21:58 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79200000000000004,"15min":0.79400000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.434},{"interface":"front","average":{"1min":0.873,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Apr 22 17:21:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Apr 22 17:21:52 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5692,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":512000,"data_stored":208238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.2609999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.51000000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.622}]},{"osd":1,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.442,"15min":0.442},"max":{"1min":1.1739999999999999,"5min":1.294,"15min":1.294},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.78600000000000003,"15min":0.77500000000000002},"min":{"1min":0.66400000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Apr 22 17:22:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.57199999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0680000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.83499999999999996}]},{"osd":4,"last update":"Fri Apr 22 17:21:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.92200000000000004}]},{"osd":5,"last update":"Fri Apr 22 17:21:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.089},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182431,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.006,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.745,"15min":0.745},"min":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1060000000000001,"5min":1.133,"15min":1.133},"last":0.85799999999999998}]},{"osd":1,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.628,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1459999999999999,"5min":1.276,"15min":1.276},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.099,"5min":1.202,"15min":1.202},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.194,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.105,"5min":1.224,"15min":1.224},"last":1.1459999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":1.0249999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.1639999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":1.216},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":1.196}]},{"osd":3,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":1.0760000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5428,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739724,"statfs":{"total":95995035648,"available":95989477376,"internally_reserved":0,"allocated":503808,"data_stored":201779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94999999999999996}]},{"osd":1,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.91100000000000003}]},{"osd":2,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0389999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:22 smithi121 bash[12679]: audit 2022-04-22T17:22:21.318004+0000 mgr.y (mgr.14146) 231 : audit [DBG] from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:22 smithi121 bash[12679]: audit 2022-04-22T17:22:21.658916+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:22 smithi121 bash[12679]: audit 2022-04-22T17:22:22.035633+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/2388443376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-04-22T17:22:22.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[12737]: audit 2022-04-22T17:22:21.318004+0000 mgr.y (mgr.14146) 231 : audit [DBG] from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[12737]: audit 2022-04-22T17:22:21.658916+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[12737]: audit 2022-04-22T17:22:22.035633+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/2388443376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-04-22T17:22:22.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[18070]: audit 2022-04-22T17:22:21.318004+0000 mgr.y (mgr.14146) 231 : audit [DBG] from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[18070]: audit 2022-04-22T17:22:21.658916+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:22.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:22 smithi115 bash[18070]: audit 2022-04-22T17:22:22.035633+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/2388443376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-04-22T17:22:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:23 smithi121 bash[12679]: audit 2022-04-22T17:22:22.306588+0000 mgr.y (mgr.14146) 233 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:23.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:23 smithi121 bash[12679]: cluster 2022-04-22T17:22:23.085945+0000 mgr.y (mgr.14146) 234 : cluster [DBG] pgmap v217: 97 pgs: 19 creating+peering, 78 active+clean; 770 B data, 44 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 2 op/s 2022-04-22T17:22:23.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:23 smithi115 bash[12737]: audit 2022-04-22T17:22:22.306588+0000 mgr.y (mgr.14146) 233 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:23 smithi115 bash[12737]: cluster 2022-04-22T17:22:23.085945+0000 mgr.y (mgr.14146) 234 : cluster [DBG] pgmap v217: 97 pgs: 19 creating+peering, 78 active+clean; 770 B data, 44 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 2 op/s 2022-04-22T17:22:23.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:23 smithi115 bash[18070]: audit 2022-04-22T17:22:22.306588+0000 mgr.y (mgr.14146) 233 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:23.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:23 smithi115 bash[18070]: cluster 2022-04-22T17:22:23.085945+0000 mgr.y (mgr.14146) 234 : cluster [DBG] pgmap v217: 97 pgs: 19 creating+peering, 78 active+clean; 770 B data, 44 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 2 op/s 2022-04-22T17:22:25.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-22T17:22:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:25 smithi121 bash[12679]: cluster 2022-04-22T17:22:25.086989+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v218: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 723 B/s wr, 3 op/s 2022-04-22T17:22:25.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:25 smithi115 bash[12737]: cluster 2022-04-22T17:22:25.086989+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v218: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 723 B/s wr, 3 op/s 2022-04-22T17:22:25.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:25 smithi115 bash[18070]: cluster 2022-04-22T17:22:25.086989+0000 mgr.y (mgr.14146) 235 : cluster [DBG] pgmap v218: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 723 B/s wr, 3 op/s 2022-04-22T17:22:25.667 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:22:25.670 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:22:25.677 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2022-04-22T17:22:25.086255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":45428,"kb_used_data":4212,"kb_used_omap":0,"kb_used_meta":41152,"kb_avail":749915788,"statfs":{"total":767960285184,"available":767913766912,"internally_reserved":0,"allocated":4313088,"data_stored":1729994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42139648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1558,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.664861"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456527+0000","last_change":"2022-04-22T17:22:18.455566+0000","last_active":"2022-04-22T17:22:18.456527+0000","last_peered":"2022-04-22T17:22:18.456527+0000","last_clean":"2022-04-22T17:22:18.456527+0000","last_became_active":"2022-04-22T17:22:18.455256+0000","last_became_peered":"2022-04-22T17:22:18.455256+0000","last_unstale":"2022-04-22T17:22:18.456527+0000","last_undegraded":"2022-04-22T17:22:18.456527+0000","last_fullsized":"2022-04-22T17:22:18.456527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455614+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:22:18.455614+0000","last_peered":"2022-04-22T17:22:18.455614+0000","last_clean":"2022-04-22T17:22:18.455614+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:22:18.455614+0000","last_undegraded":"2022-04-22T17:22:18.455614+0000","last_fullsized":"2022-04-22T17:22:18.455614+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457500+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:22:18.457500+0000","last_peered":"2022-04-22T17:22:18.457500+0000","last_clean":"2022-04-22T17:22:18.457500+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:22:18.457500+0000","last_undegraded":"2022-04-22T17:22:18.457500+0000","last_fullsized":"2022-04-22T17:22:18.457500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449232+0000","last_change":"2022-04-22T17:22:18.448571+0000","last_active":"2022-04-22T17:22:18.449232+0000","last_peered":"2022-04-22T17:22:18.449232+0000","last_clean":"2022-04-22T17:22:18.449232+0000","last_became_active":"2022-04-22T17:22:18.448283+0000","last_became_peered":"2022-04-22T17:22:18.448283+0000","last_unstale":"2022-04-22T17:22:18.449232+0000","last_undegraded":"2022-04-22T17:22:18.449232+0000","last_fullsized":"2022-04-22T17:22:18.449232+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451186+0000","last_change":"2022-04-22T17:22:18.450409+0000","last_active":"2022-04-22T17:22:18.451186+0000","last_peered":"2022-04-22T17:22:18.451186+0000","last_clean":"2022-04-22T17:22:18.451186+0000","last_became_active":"2022-04-22T17:22:18.449933+0000","last_became_peered":"2022-04-22T17:22:18.449933+0000","last_unstale":"2022-04-22T17:22:18.451186+0000","last_undegraded":"2022-04-22T17:22:18.451186+0000","last_fullsized":"2022-04-22T17:22:18.451186+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.445477+0000","last_change":"2022-04-22T17:22:18.443897+0000","last_active":"2022-04-22T17:22:18.445477+0000","last_peered":"2022-04-22T17:22:18.445477+0000","last_clean":"2022-04-22T17:22:18.445477+0000","last_became_active":"2022-04-22T17:22:18.443606+0000","last_became_peered":"2022-04-22T17:22:18.443606+0000","last_unstale":"2022-04-22T17:22:18.445477+0000","last_undegraded":"2022-04-22T17:22:18.445477+0000","last_fullsized":"2022-04-22T17:22:18.445477+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454804+0000","last_change":"2022-04-22T17:22:18.453280+0000","last_active":"2022-04-22T17:22:18.454804+0000","last_peered":"2022-04-22T17:22:18.454804+0000","last_clean":"2022-04-22T17:22:18.454804+0000","last_became_active":"2022-04-22T17:22:18.452955+0000","last_became_peered":"2022-04-22T17:22:18.452955+0000","last_unstale":"2022-04-22T17:22:18.454804+0000","last_undegraded":"2022-04-22T17:22:18.454804+0000","last_fullsized":"2022-04-22T17:22:18.454804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450016+0000","last_change":"2022-04-22T17:22:18.447447+0000","last_active":"2022-04-22T17:22:18.450016+0000","last_peered":"2022-04-22T17:22:18.450016+0000","last_clean":"2022-04-22T17:22:18.450016+0000","last_became_active":"2022-04-22T17:22:18.447040+0000","last_became_peered":"2022-04-22T17:22:18.447040+0000","last_unstale":"2022-04-22T17:22:18.450016+0000","last_undegraded":"2022-04-22T17:22:18.450016+0000","last_fullsized":"2022-04-22T17:22:18.450016+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454886+0000","last_change":"2022-04-22T17:22:18.453951+0000","last_active":"2022-04-22T17:22:18.454886+0000","last_peered":"2022-04-22T17:22:18.454886+0000","last_clean":"2022-04-22T17:22:18.454886+0000","last_became_active":"2022-04-22T17:22:18.453648+0000","last_became_peered":"2022-04-22T17:22:18.453648+0000","last_unstale":"2022-04-22T17:22:18.454886+0000","last_undegraded":"2022-04-22T17:22:18.454886+0000","last_fullsized":"2022-04-22T17:22:18.454886+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.436120+0000","last_change":"2022-04-22T17:22:18.435236+0000","last_active":"2022-04-22T17:22:18.436120+0000","last_peered":"2022-04-22T17:22:18.436120+0000","last_clean":"2022-04-22T17:22:18.436120+0000","last_became_active":"2022-04-22T17:22:18.434908+0000","last_became_peered":"2022-04-22T17:22:18.434908+0000","last_unstale":"2022-04-22T17:22:18.436120+0000","last_undegraded":"2022-04-22T17:22:18.436120+0000","last_fullsized":"2022-04-22T17:22:18.436120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455388+0000","last_change":"2022-04-22T17:22:18.453847+0000","last_active":"2022-04-22T17:22:18.455388+0000","last_peered":"2022-04-22T17:22:18.455388+0000","last_clean":"2022-04-22T17:22:18.455388+0000","last_became_active":"2022-04-22T17:22:18.453590+0000","last_became_peered":"2022-04-22T17:22:18.453590+0000","last_unstale":"2022-04-22T17:22:18.455388+0000","last_undegraded":"2022-04-22T17:22:18.455388+0000","last_fullsized":"2022-04-22T17:22:18.455388+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457061+0000","last_change":"2022-04-22T17:22:18.451896+0000","last_active":"2022-04-22T17:22:18.457061+0000","last_peered":"2022-04-22T17:22:18.457061+0000","last_clean":"2022-04-22T17:22:18.457061+0000","last_became_active":"2022-04-22T17:22:18.451579+0000","last_became_peered":"2022-04-22T17:22:18.451579+0000","last_unstale":"2022-04-22T17:22:18.457061+0000","last_undegraded":"2022-04-22T17:22:18.457061+0000","last_fullsized":"2022-04-22T17:22:18.457061+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441823+0000","last_change":"2022-04-22T17:22:18.439501+0000","last_active":"2022-04-22T17:22:18.441823+0000","last_peered":"2022-04-22T17:22:18.441823+0000","last_clean":"2022-04-22T17:22:18.441823+0000","last_became_active":"2022-04-22T17:22:18.439170+0000","last_became_peered":"2022-04-22T17:22:18.439170+0000","last_unstale":"2022-04-22T17:22:18.441823+0000","last_undegraded":"2022-04-22T17:22:18.441823+0000","last_fullsized":"2022-04-22T17:22:18.441823+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449558+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:22:18.449558+0000","last_peered":"2022-04-22T17:22:18.449558+0000","last_clean":"2022-04-22T17:22:18.449558+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:22:18.449558+0000","last_undegraded":"2022-04-22T17:22:18.449558+0000","last_fullsized":"2022-04-22T17:22:18.449558+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456813+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:22:18.456813+0000","last_peered":"2022-04-22T17:22:18.456813+0000","last_clean":"2022-04-22T17:22:18.456813+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:22:18.456813+0000","last_undegraded":"2022-04-22T17:22:18.456813+0000","last_fullsized":"2022-04-22T17:22:18.456813+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449211+0000","last_change":"2022-04-22T17:22:18.447051+0000","last_active":"2022-04-22T17:22:18.449211+0000","last_peered":"2022-04-22T17:22:18.449211+0000","last_clean":"2022-04-22T17:22:18.449211+0000","last_became_active":"2022-04-22T17:22:18.446733+0000","last_became_peered":"2022-04-22T17:22:18.446733+0000","last_unstale":"2022-04-22T17:22:18.449211+0000","last_undegraded":"2022-04-22T17:22:18.449211+0000","last_fullsized":"2022-04-22T17:22:18.449211+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451295+0000","last_change":"2022-04-22T17:22:18.447963+0000","last_active":"2022-04-22T17:22:18.451295+0000","last_peered":"2022-04-22T17:22:18.451295+0000","last_clean":"2022-04-22T17:22:18.451295+0000","last_became_active":"2022-04-22T17:22:18.447352+0000","last_became_peered":"2022-04-22T17:22:18.447352+0000","last_unstale":"2022-04-22T17:22:18.451295+0000","last_undegraded":"2022-04-22T17:22:18.451295+0000","last_fullsized":"2022-04-22T17:22:18.451295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.442525+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:22:18.442525+0000","last_peered":"2022-04-22T17:22:18.442525+0000","last_clean":"2022-04-22T17:22:18.442525+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:22:18.442525+0000","last_undegraded":"2022-04-22T17:22:18.442525+0000","last_fullsized":"2022-04-22T17:22:18.442525+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457285+0000","last_change":"2022-04-22T17:22:18.456452+0000","last_active":"2022-04-22T17:22:18.457285+0000","last_peered":"2022-04-22T17:22:18.457285+0000","last_clean":"2022-04-22T17:22:18.457285+0000","last_became_active":"2022-04-22T17:22:18.456060+0000","last_became_peered":"2022-04-22T17:22:18.456060+0000","last_unstale":"2022-04-22T17:22:18.457285+0000","last_undegraded":"2022-04-22T17:22:18.457285+0000","last_fullsized":"2022-04-22T17:22:18.457285+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.463249+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:22:18.463249+0000","last_peered":"2022-04-22T17:22:18.463249+0000","last_clean":"2022-04-22T17:22:18.463249+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:22:18.463249+0000","last_undegraded":"2022-04-22T17:22:18.463249+0000","last_fullsized":"2022-04-22T17:22:18.463249+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441714+0000","last_change":"2022-04-22T17:22:18.440558+0000","last_active":"2022-04-22T17:22:18.441714+0000","last_peered":"2022-04-22T17:22:18.441714+0000","last_clean":"2022-04-22T17:22:18.441714+0000","last_became_active":"2022-04-22T17:22:18.440197+0000","last_became_peered":"2022-04-22T17:22:18.440197+0000","last_unstale":"2022-04-22T17:22:18.441714+0000","last_undegraded":"2022-04-22T17:22:18.441714+0000","last_fullsized":"2022-04-22T17:22:18.441714+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"53'1","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.446923+0000","last_change":"2022-04-22T17:22:18.117767+0000","last_active":"2022-04-22T17:22:18.446923+0000","last_peered":"2022-04-22T17:22:18.446923+0000","last_clean":"2022-04-22T17:22:18.446923+0000","last_became_active":"2022-04-22T17:22:18.117374+0000","last_became_peered":"2022-04-22T17:22:18.117374+0000","last_unstale":"2022-04-22T17:22:18.446923+0000","last_undegraded":"2022-04-22T17:22:18.446923+0000","last_fullsized":"2022-04-22T17:22:18.446923+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450041+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:22:18.450041+0000","last_peered":"2022-04-22T17:22:18.450041+0000","last_clean":"2022-04-22T17:22:18.450041+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:22:18.450041+0000","last_undegraded":"2022-04-22T17:22:18.450041+0000","last_fullsized":"2022-04-22T17:22:18.450041+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457734+0000","last_change":"2022-04-22T17:22:18.118589+0000","last_active":"2022-04-22T17:22:18.457734+0000","last_peered":"2022-04-22T17:22:18.457734+0000","last_clean":"2022-04-22T17:22:18.457734+0000","last_became_active":"2022-04-22T17:22:18.118118+0000","last_became_peered":"2022-04-22T17:22:18.118118+0000","last_unstale":"2022-04-22T17:22:18.457734+0000","last_undegraded":"2022-04-22T17:22:18.457734+0000","last_fullsized":"2022-04-22T17:22:18.457734+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.445805+0000","last_change":"2022-04-22T17:22:18.444605+0000","last_active":"2022-04-22T17:22:18.445805+0000","last_peered":"2022-04-22T17:22:18.445805+0000","last_clean":"2022-04-22T17:22:18.445805+0000","last_became_active":"2022-04-22T17:22:18.444288+0000","last_became_peered":"2022-04-22T17:22:18.444288+0000","last_unstale":"2022-04-22T17:22:18.445805+0000","last_undegraded":"2022-04-22T17:22:18.445805+0000","last_fullsized":"2022-04-22T17:22:18.445805+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456206+0000","last_change":"2022-04-22T17:22:18.119939+0000","last_active":"2022-04-22T17:22:18.456206+0000","last_peered":"2022-04-22T17:22:18.456206+0000","last_clean":"2022-04-22T17:22:18.456206+0000","last_became_active":"2022-04-22T17:22:18.119090+0000","last_became_peered":"2022-04-22T17:22:18.119090+0000","last_unstale":"2022-04-22T17:22:18.456206+0000","last_undegraded":"2022-04-22T17:22:18.456206+0000","last_fullsized":"2022-04-22T17:22:18.456206+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450258+0000","last_change":"2022-04-22T17:22:18.446755+0000","last_active":"2022-04-22T17:22:18.450258+0000","last_peered":"2022-04-22T17:22:18.450258+0000","last_clean":"2022-04-22T17:22:18.450258+0000","last_became_active":"2022-04-22T17:22:18.445791+0000","last_became_peered":"2022-04-22T17:22:18.445791+0000","last_unstale":"2022-04-22T17:22:18.450258+0000","last_undegraded":"2022-04-22T17:22:18.450258+0000","last_fullsized":"2022-04-22T17:22:18.450258+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"53'1","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.447504+0000","last_change":"2022-04-22T17:22:18.115931+0000","last_active":"2022-04-22T17:22:18.447504+0000","last_peered":"2022-04-22T17:22:18.447504+0000","last_clean":"2022-04-22T17:22:18.447504+0000","last_became_active":"2022-04-22T17:22:18.115583+0000","last_became_peered":"2022-04-22T17:22:18.115583+0000","last_unstale":"2022-04-22T17:22:18.447504+0000","last_undegraded":"2022-04-22T17:22:18.447504+0000","last_fullsized":"2022-04-22T17:22:18.447504+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.433101+0000","last_change":"2022-04-22T17:22:18.429737+0000","last_active":"2022-04-22T17:22:18.433101+0000","last_peered":"2022-04-22T17:22:18.433101+0000","last_clean":"2022-04-22T17:22:18.433101+0000","last_became_active":"2022-04-22T17:22:18.429404+0000","last_became_peered":"2022-04-22T17:22:18.429404+0000","last_unstale":"2022-04-22T17:22:18.433101+0000","last_undegraded":"2022-04-22T17:22:18.433101+0000","last_fullsized":"2022-04-22T17:22:18.433101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"53'1","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.459948+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:22:18.459948+0000","last_peered":"2022-04-22T17:22:18.459948+0000","last_clean":"2022-04-22T17:22:18.459948+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:22:18.459948+0000","last_undegraded":"2022-04-22T17:22:18.459948+0000","last_fullsized":"2022-04-22T17:22:18.459948+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461233+0000","last_change":"2022-04-22T17:22:18.460074+0000","last_active":"2022-04-22T17:22:18.461233+0000","last_peered":"2022-04-22T17:22:18.461233+0000","last_clean":"2022-04-22T17:22:18.461233+0000","last_became_active":"2022-04-22T17:22:18.459771+0000","last_became_peered":"2022-04-22T17:22:18.459771+0000","last_unstale":"2022-04-22T17:22:18.461233+0000","last_undegraded":"2022-04-22T17:22:18.461233+0000","last_fullsized":"2022-04-22T17:22:18.461233+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.429476+0000","last_change":"2022-04-22T17:22:18.115369+0000","last_active":"2022-04-22T17:22:18.429476+0000","last_peered":"2022-04-22T17:22:18.429476+0000","last_clean":"2022-04-22T17:22:18.429476+0000","last_became_active":"2022-04-22T17:22:18.115038+0000","last_became_peered":"2022-04-22T17:22:18.115038+0000","last_unstale":"2022-04-22T17:22:18.429476+0000","last_undegraded":"2022-04-22T17:22:18.429476+0000","last_fullsized":"2022-04-22T17:22:18.429476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.444031+0000","last_change":"2022-04-22T17:22:18.443167+0000","last_active":"2022-04-22T17:22:18.444031+0000","last_peered":"2022-04-22T17:22:18.444031+0000","last_clean":"2022-04-22T17:22:18.444031+0000","last_became_active":"2022-04-22T17:22:18.442849+0000","last_became_peered":"2022-04-22T17:22:18.442849+0000","last_unstale":"2022-04-22T17:22:18.444031+0000","last_undegraded":"2022-04-22T17:22:18.444031+0000","last_fullsized":"2022-04-22T17:22:18.444031+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.425106+0000","last_change":"2022-04-22T17:22:18.111677+0000","last_active":"2022-04-22T17:22:18.425106+0000","last_peered":"2022-04-22T17:22:18.425106+0000","last_clean":"2022-04-22T17:22:18.425106+0000","last_became_active":"2022-04-22T17:22:18.111341+0000","last_became_peered":"2022-04-22T17:22:18.111341+0000","last_unstale":"2022-04-22T17:22:18.425106+0000","last_undegraded":"2022-04-22T17:22:18.425106+0000","last_fullsized":"2022-04-22T17:22:18.425106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.456793+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:22:18.456793+0000","last_peered":"2022-04-22T17:22:18.456793+0000","last_clean":"2022-04-22T17:22:18.456793+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:22:18.456793+0000","last_undegraded":"2022-04-22T17:22:18.456793+0000","last_fullsized":"2022-04-22T17:22:18.456793+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.423385+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:22:18.423385+0000","last_peered":"2022-04-22T17:22:18.423385+0000","last_clean":"2022-04-22T17:22:18.423385+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:22:18.423385+0000","last_undegraded":"2022-04-22T17:22:18.423385+0000","last_fullsized":"2022-04-22T17:22:18.423385+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.447829+0000","last_change":"2022-04-22T17:22:18.446536+0000","last_active":"2022-04-22T17:22:18.447829+0000","last_peered":"2022-04-22T17:22:18.447829+0000","last_clean":"2022-04-22T17:22:18.447829+0000","last_became_active":"2022-04-22T17:22:18.446213+0000","last_became_peered":"2022-04-22T17:22:18.446213+0000","last_unstale":"2022-04-22T17:22:18.447829+0000","last_undegraded":"2022-04-22T17:22:18.447829+0000","last_fullsized":"2022-04-22T17:22:18.447829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.426912+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:22:18.426912+0000","last_peered":"2022-04-22T17:22:18.426912+0000","last_clean":"2022-04-22T17:22:18.426912+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:22:18.426912+0000","last_undegraded":"2022-04-22T17:22:18.426912+0000","last_fullsized":"2022-04-22T17:22:18.426912+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454798+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:22:18.454798+0000","last_peered":"2022-04-22T17:22:18.454798+0000","last_clean":"2022-04-22T17:22:18.454798+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:22:18.454798+0000","last_undegraded":"2022-04-22T17:22:18.454798+0000","last_fullsized":"2022-04-22T17:22:18.454798+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457202+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:22:18.457202+0000","last_peered":"2022-04-22T17:22:18.457202+0000","last_clean":"2022-04-22T17:22:18.457202+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:22:18.457202+0000","last_undegraded":"2022-04-22T17:22:18.457202+0000","last_fullsized":"2022-04-22T17:22:18.457202+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.443909+0000","last_change":"2022-04-22T17:22:18.443482+0000","last_active":"2022-04-22T17:22:18.443909+0000","last_peered":"2022-04-22T17:22:18.443909+0000","last_clean":"2022-04-22T17:22:18.443909+0000","last_became_active":"2022-04-22T17:22:18.443179+0000","last_became_peered":"2022-04-22T17:22:18.443179+0000","last_unstale":"2022-04-22T17:22:18.443909+0000","last_undegraded":"2022-04-22T17:22:18.443909+0000","last_fullsized":"2022-04-22T17:22:18.443909+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"53'6","reported_seq":17,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.459371+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:22:18.459371+0000","last_peered":"2022-04-22T17:22:18.459371+0000","last_clean":"2022-04-22T17:22:18.459371+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:22:18.459371+0000","last_undegraded":"2022-04-22T17:22:18.459371+0000","last_fullsized":"2022-04-22T17:22:18.459371+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450424+0000","last_change":"2022-04-22T17:22:18.449910+0000","last_active":"2022-04-22T17:22:18.450424+0000","last_peered":"2022-04-22T17:22:18.450424+0000","last_clean":"2022-04-22T17:22:18.450424+0000","last_became_active":"2022-04-22T17:22:18.449563+0000","last_became_peered":"2022-04-22T17:22:18.449563+0000","last_unstale":"2022-04-22T17:22:18.450424+0000","last_undegraded":"2022-04-22T17:22:18.450424+0000","last_fullsized":"2022-04-22T17:22:18.450424+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461178+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:22:18.461178+0000","last_peered":"2022-04-22T17:22:18.461178+0000","last_clean":"2022-04-22T17:22:18.461178+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:22:18.461178+0000","last_undegraded":"2022-04-22T17:22:18.461178+0000","last_fullsized":"2022-04-22T17:22:18.461178+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461785+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:22:18.461785+0000","last_peered":"2022-04-22T17:22:18.461785+0000","last_clean":"2022-04-22T17:22:18.461785+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:22:18.461785+0000","last_undegraded":"2022-04-22T17:22:18.461785+0000","last_fullsized":"2022-04-22T17:22:18.461785+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.426434+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:22:18.426434+0000","last_peered":"2022-04-22T17:22:18.426434+0000","last_clean":"2022-04-22T17:22:18.426434+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:22:18.426434+0000","last_undegraded":"2022-04-22T17:22:18.426434+0000","last_fullsized":"2022-04-22T17:22:18.426434+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451134+0000","last_change":"2022-04-22T17:22:18.450219+0000","last_active":"2022-04-22T17:22:18.451134+0000","last_peered":"2022-04-22T17:22:18.451134+0000","last_clean":"2022-04-22T17:22:18.451134+0000","last_became_active":"2022-04-22T17:22:18.449516+0000","last_became_peered":"2022-04-22T17:22:18.449516+0000","last_unstale":"2022-04-22T17:22:18.451134+0000","last_undegraded":"2022-04-22T17:22:18.451134+0000","last_fullsized":"2022-04-22T17:22:18.451134+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.421311+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:22:18.421311+0000","last_peered":"2022-04-22T17:22:18.421311+0000","last_clean":"2022-04-22T17:22:18.421311+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:22:18.421311+0000","last_undegraded":"2022-04-22T17:22:18.421311+0000","last_fullsized":"2022-04-22T17:22:18.421311+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454415+0000","last_change":"2022-04-22T17:22:18.452569+0000","last_active":"2022-04-22T17:22:18.454415+0000","last_peered":"2022-04-22T17:22:18.454415+0000","last_clean":"2022-04-22T17:22:18.454415+0000","last_became_active":"2022-04-22T17:22:18.452287+0000","last_became_peered":"2022-04-22T17:22:18.452287+0000","last_unstale":"2022-04-22T17:22:18.454415+0000","last_undegraded":"2022-04-22T17:22:18.454415+0000","last_fullsized":"2022-04-22T17:22:18.454415+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450073+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:22:18.450073+0000","last_peered":"2022-04-22T17:22:18.450073+0000","last_clean":"2022-04-22T17:22:18.450073+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:22:18.450073+0000","last_undegraded":"2022-04-22T17:22:18.450073+0000","last_fullsized":"2022-04-22T17:22:18.450073+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449613+0000","last_change":"2022-04-22T17:22:18.448940+0000","last_active":"2022-04-22T17:22:18.449613+0000","last_peered":"2022-04-22T17:22:18.449613+0000","last_clean":"2022-04-22T17:22:18.449613+0000","last_became_active":"2022-04-22T17:22:18.448633+0000","last_became_peered":"2022-04-22T17:22:18.448633+0000","last_unstale":"2022-04-22T17:22:18.449613+0000","last_undegraded":"2022-04-22T17:22:18.449613+0000","last_fullsized":"2022-04-22T17:22:18.449613+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"53'1","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454864+0000","last_change":"2022-04-22T17:22:18.112339+0000","last_active":"2022-04-22T17:22:18.454864+0000","last_peered":"2022-04-22T17:22:18.454864+0000","last_clean":"2022-04-22T17:22:18.454864+0000","last_became_active":"2022-04-22T17:22:18.112055+0000","last_became_peered":"2022-04-22T17:22:18.112055+0000","last_unstale":"2022-04-22T17:22:18.454864+0000","last_undegraded":"2022-04-22T17:22:18.454864+0000","last_fullsized":"2022-04-22T17:22:18.454864+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455761+0000","last_change":"2022-04-22T17:22:18.454678+0000","last_active":"2022-04-22T17:22:18.455761+0000","last_peered":"2022-04-22T17:22:18.455761+0000","last_clean":"2022-04-22T17:22:18.455761+0000","last_became_active":"2022-04-22T17:22:18.454221+0000","last_became_peered":"2022-04-22T17:22:18.454221+0000","last_unstale":"2022-04-22T17:22:18.455761+0000","last_undegraded":"2022-04-22T17:22:18.455761+0000","last_fullsized":"2022-04-22T17:22:18.455761+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.420509+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:22:18.420509+0000","last_peered":"2022-04-22T17:22:18.420509+0000","last_clean":"2022-04-22T17:22:18.420509+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:22:18.420509+0000","last_undegraded":"2022-04-22T17:22:18.420509+0000","last_fullsized":"2022-04-22T17:22:18.420509+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438149+0000","last_change":"2022-04-22T17:22:18.437505+0000","last_active":"2022-04-22T17:22:18.438149+0000","last_peered":"2022-04-22T17:22:18.438149+0000","last_clean":"2022-04-22T17:22:18.438149+0000","last_became_active":"2022-04-22T17:22:18.437197+0000","last_became_peered":"2022-04-22T17:22:18.437197+0000","last_unstale":"2022-04-22T17:22:18.438149+0000","last_undegraded":"2022-04-22T17:22:18.438149+0000","last_fullsized":"2022-04-22T17:22:18.438149+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.425652+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:22:18.425652+0000","last_peered":"2022-04-22T17:22:18.425652+0000","last_clean":"2022-04-22T17:22:18.425652+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:22:18.425652+0000","last_undegraded":"2022-04-22T17:22:18.425652+0000","last_fullsized":"2022-04-22T17:22:18.425652+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455637+0000","last_change":"2022-04-22T17:22:18.451520+0000","last_active":"2022-04-22T17:22:18.455637+0000","last_peered":"2022-04-22T17:22:18.455637+0000","last_clean":"2022-04-22T17:22:18.455637+0000","last_became_active":"2022-04-22T17:22:18.451231+0000","last_became_peered":"2022-04-22T17:22:18.451231+0000","last_unstale":"2022-04-22T17:22:18.455637+0000","last_undegraded":"2022-04-22T17:22:18.455637+0000","last_fullsized":"2022-04-22T17:22:18.455637+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.432323+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:22:18.432323+0000","last_peered":"2022-04-22T17:22:18.432323+0000","last_clean":"2022-04-22T17:22:18.432323+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:22:18.432323+0000","last_undegraded":"2022-04-22T17:22:18.432323+0000","last_fullsized":"2022-04-22T17:22:18.432323+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.425666+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:22:18.425666+0000","last_peered":"2022-04-22T17:22:18.425666+0000","last_clean":"2022-04-22T17:22:18.425666+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:22:18.425666+0000","last_undegraded":"2022-04-22T17:22:18.425666+0000","last_fullsized":"2022-04-22T17:22:18.425666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.448882+0000","last_change":"2022-04-22T17:22:18.444926+0000","last_active":"2022-04-22T17:22:18.448882+0000","last_peered":"2022-04-22T17:22:18.448882+0000","last_clean":"2022-04-22T17:22:18.448882+0000","last_became_active":"2022-04-22T17:22:18.444610+0000","last_became_peered":"2022-04-22T17:22:18.444610+0000","last_unstale":"2022-04-22T17:22:18.448882+0000","last_undegraded":"2022-04-22T17:22:18.448882+0000","last_fullsized":"2022-04-22T17:22:18.448882+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.426381+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:22:18.426381+0000","last_peered":"2022-04-22T17:22:18.426381+0000","last_clean":"2022-04-22T17:22:18.426381+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:22:18.426381+0000","last_undegraded":"2022-04-22T17:22:18.426381+0000","last_fullsized":"2022-04-22T17:22:18.426381+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434734+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:22:18.434734+0000","last_peered":"2022-04-22T17:22:18.434734+0000","last_clean":"2022-04-22T17:22:18.434734+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:22:18.434734+0000","last_undegraded":"2022-04-22T17:22:18.434734+0000","last_fullsized":"2022-04-22T17:22:18.434734+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.450782+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:22:18.450782+0000","last_peered":"2022-04-22T17:22:18.450782+0000","last_clean":"2022-04-22T17:22:18.450782+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:22:18.450782+0000","last_undegraded":"2022-04-22T17:22:18.450782+0000","last_fullsized":"2022-04-22T17:22:18.450782+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434794+0000","last_change":"2022-04-22T17:22:18.115452+0000","last_active":"2022-04-22T17:22:18.434794+0000","last_peered":"2022-04-22T17:22:18.434794+0000","last_clean":"2022-04-22T17:22:18.434794+0000","last_became_active":"2022-04-22T17:22:18.114179+0000","last_became_peered":"2022-04-22T17:22:18.114179+0000","last_unstale":"2022-04-22T17:22:18.434794+0000","last_undegraded":"2022-04-22T17:22:18.434794+0000","last_fullsized":"2022-04-22T17:22:18.434794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.440995+0000","last_change":"2022-04-22T17:22:18.115678+0000","last_active":"2022-04-22T17:22:18.440995+0000","last_peered":"2022-04-22T17:22:18.440995+0000","last_clean":"2022-04-22T17:22:18.440995+0000","last_became_active":"2022-04-22T17:22:18.115400+0000","last_became_peered":"2022-04-22T17:22:18.115400+0000","last_unstale":"2022-04-22T17:22:18.440995+0000","last_undegraded":"2022-04-22T17:22:18.440995+0000","last_fullsized":"2022-04-22T17:22:18.440995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457740+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:22:18.457740+0000","last_peered":"2022-04-22T17:22:18.457740+0000","last_clean":"2022-04-22T17:22:18.457740+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:22:18.457740+0000","last_undegraded":"2022-04-22T17:22:18.457740+0000","last_fullsized":"2022-04-22T17:22:18.457740+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"54'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2022-04-22T17:22:20.135150+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:22:20.135150+0000","last_peered":"2022-04-22T17:22:20.135150+0000","last_clean":"2022-04-22T17:22:20.135150+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:22:20.135150+0000","last_undegraded":"2022-04-22T17:22:20.135150+0000","last_fullsized":"2022-04-22T17:22:20.135150+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.451727+0000","last_change":"2022-04-22T17:22:18.451070+0000","last_active":"2022-04-22T17:22:18.451727+0000","last_peered":"2022-04-22T17:22:18.451727+0000","last_clean":"2022-04-22T17:22:18.451727+0000","last_became_active":"2022-04-22T17:22:18.450715+0000","last_became_peered":"2022-04-22T17:22:18.450715+0000","last_unstale":"2022-04-22T17:22:18.451727+0000","last_undegraded":"2022-04-22T17:22:18.451727+0000","last_fullsized":"2022-04-22T17:22:18.451727+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460004+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:22:18.460004+0000","last_peered":"2022-04-22T17:22:18.460004+0000","last_clean":"2022-04-22T17:22:18.460004+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:22:18.460004+0000","last_undegraded":"2022-04-22T17:22:18.460004+0000","last_fullsized":"2022-04-22T17:22:18.460004+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.458553+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:22:18.458553+0000","last_peered":"2022-04-22T17:22:18.458553+0000","last_clean":"2022-04-22T17:22:18.458553+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:22:18.458553+0000","last_undegraded":"2022-04-22T17:22:18.458553+0000","last_fullsized":"2022-04-22T17:22:18.458553+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.434649+0000","last_change":"2022-04-22T17:22:18.114125+0000","last_active":"2022-04-22T17:22:18.434649+0000","last_peered":"2022-04-22T17:22:18.434649+0000","last_clean":"2022-04-22T17:22:18.434649+0000","last_became_active":"2022-04-22T17:22:18.113799+0000","last_became_peered":"2022-04-22T17:22:18.113799+0000","last_unstale":"2022-04-22T17:22:18.434649+0000","last_undegraded":"2022-04-22T17:22:18.434649+0000","last_fullsized":"2022-04-22T17:22:18.434649+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441852+0000","last_change":"2022-04-22T17:22:18.440746+0000","last_active":"2022-04-22T17:22:18.441852+0000","last_peered":"2022-04-22T17:22:18.441852+0000","last_clean":"2022-04-22T17:22:18.441852+0000","last_became_active":"2022-04-22T17:22:18.440481+0000","last_became_peered":"2022-04-22T17:22:18.440481+0000","last_unstale":"2022-04-22T17:22:18.441852+0000","last_undegraded":"2022-04-22T17:22:18.441852+0000","last_fullsized":"2022-04-22T17:22:18.441852+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.419873+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:22:18.419873+0000","last_peered":"2022-04-22T17:22:18.419873+0000","last_clean":"2022-04-22T17:22:18.419873+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:22:18.419873+0000","last_undegraded":"2022-04-22T17:22:18.419873+0000","last_fullsized":"2022-04-22T17:22:18.419873+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.448919+0000","last_change":"2022-04-22T17:22:18.445409+0000","last_active":"2022-04-22T17:22:18.448919+0000","last_peered":"2022-04-22T17:22:18.448919+0000","last_clean":"2022-04-22T17:22:18.448919+0000","last_became_active":"2022-04-22T17:22:18.445036+0000","last_became_peered":"2022-04-22T17:22:18.445036+0000","last_unstale":"2022-04-22T17:22:18.448919+0000","last_undegraded":"2022-04-22T17:22:18.448919+0000","last_fullsized":"2022-04-22T17:22:18.448919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.420038+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:22:18.420038+0000","last_peered":"2022-04-22T17:22:18.420038+0000","last_clean":"2022-04-22T17:22:18.420038+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:22:18.420038+0000","last_undegraded":"2022-04-22T17:22:18.420038+0000","last_fullsized":"2022-04-22T17:22:18.420038+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449836+0000","last_change":"2022-04-22T17:22:18.446200+0000","last_active":"2022-04-22T17:22:18.449836+0000","last_peered":"2022-04-22T17:22:18.449836+0000","last_clean":"2022-04-22T17:22:18.449836+0000","last_became_active":"2022-04-22T17:22:18.445848+0000","last_became_peered":"2022-04-22T17:22:18.445848+0000","last_unstale":"2022-04-22T17:22:18.449836+0000","last_undegraded":"2022-04-22T17:22:18.449836+0000","last_fullsized":"2022-04-22T17:22:18.449836+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455549+0000","last_change":"2022-04-22T17:22:18.118345+0000","last_active":"2022-04-22T17:22:18.455549+0000","last_peered":"2022-04-22T17:22:18.455549+0000","last_clean":"2022-04-22T17:22:18.455549+0000","last_became_active":"2022-04-22T17:22:18.118019+0000","last_became_peered":"2022-04-22T17:22:18.118019+0000","last_unstale":"2022-04-22T17:22:18.455549+0000","last_undegraded":"2022-04-22T17:22:18.455549+0000","last_fullsized":"2022-04-22T17:22:18.455549+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.435543+0000","last_change":"2022-04-22T17:22:18.434578+0000","last_active":"2022-04-22T17:22:18.435543+0000","last_peered":"2022-04-22T17:22:18.435543+0000","last_clean":"2022-04-22T17:22:18.435543+0000","last_became_active":"2022-04-22T17:22:18.434258+0000","last_became_peered":"2022-04-22T17:22:18.434258+0000","last_unstale":"2022-04-22T17:22:18.435543+0000","last_undegraded":"2022-04-22T17:22:18.435543+0000","last_fullsized":"2022-04-22T17:22:18.435543+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.426126+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:22:18.426126+0000","last_peered":"2022-04-22T17:22:18.426126+0000","last_clean":"2022-04-22T17:22:18.426126+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:22:18.426126+0000","last_undegraded":"2022-04-22T17:22:18.426126+0000","last_fullsized":"2022-04-22T17:22:18.426126+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.457670+0000","last_change":"2022-04-22T17:22:18.457205+0000","last_active":"2022-04-22T17:22:18.457670+0000","last_peered":"2022-04-22T17:22:18.457670+0000","last_clean":"2022-04-22T17:22:18.457670+0000","last_became_active":"2022-04-22T17:22:18.456866+0000","last_became_peered":"2022-04-22T17:22:18.456866+0000","last_unstale":"2022-04-22T17:22:18.457670+0000","last_undegraded":"2022-04-22T17:22:18.457670+0000","last_fullsized":"2022-04-22T17:22:18.457670+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.455730+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:22:18.455730+0000","last_peered":"2022-04-22T17:22:18.455730+0000","last_clean":"2022-04-22T17:22:18.455730+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:22:18.455730+0000","last_undegraded":"2022-04-22T17:22:18.455730+0000","last_fullsized":"2022-04-22T17:22:18.455730+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438194+0000","last_change":"2022-04-22T17:22:18.436889+0000","last_active":"2022-04-22T17:22:18.438194+0000","last_peered":"2022-04-22T17:22:18.438194+0000","last_clean":"2022-04-22T17:22:18.438194+0000","last_became_active":"2022-04-22T17:22:18.436544+0000","last_became_peered":"2022-04-22T17:22:18.436544+0000","last_unstale":"2022-04-22T17:22:18.438194+0000","last_undegraded":"2022-04-22T17:22:18.438194+0000","last_fullsized":"2022-04-22T17:22:18.438194+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":13,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441431+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:22:18.441431+0000","last_peered":"2022-04-22T17:22:18.441431+0000","last_clean":"2022-04-22T17:22:18.441431+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:22:18.441431+0000","last_undegraded":"2022-04-22T17:22:18.441431+0000","last_fullsized":"2022-04-22T17:22:18.441431+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454868+0000","last_change":"2022-04-22T17:22:18.453658+0000","last_active":"2022-04-22T17:22:18.454868+0000","last_peered":"2022-04-22T17:22:18.454868+0000","last_clean":"2022-04-22T17:22:18.454868+0000","last_became_active":"2022-04-22T17:22:18.453161+0000","last_became_peered":"2022-04-22T17:22:18.453161+0000","last_unstale":"2022-04-22T17:22:18.454868+0000","last_undegraded":"2022-04-22T17:22:18.454868+0000","last_fullsized":"2022-04-22T17:22:18.454868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.452650+0000","last_change":"2022-04-22T17:22:18.451935+0000","last_active":"2022-04-22T17:22:18.452650+0000","last_peered":"2022-04-22T17:22:18.452650+0000","last_clean":"2022-04-22T17:22:18.452650+0000","last_became_active":"2022-04-22T17:22:18.451653+0000","last_became_peered":"2022-04-22T17:22:18.451653+0000","last_unstale":"2022-04-22T17:22:18.452650+0000","last_undegraded":"2022-04-22T17:22:18.452650+0000","last_fullsized":"2022-04-22T17:22:18.452650+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438829+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:22:18.438829+0000","last_peered":"2022-04-22T17:22:18.438829+0000","last_clean":"2022-04-22T17:22:18.438829+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:22:18.438829+0000","last_undegraded":"2022-04-22T17:22:18.438829+0000","last_fullsized":"2022-04-22T17:22:18.438829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.439853+0000","last_change":"2022-04-22T17:22:18.439502+0000","last_active":"2022-04-22T17:22:18.439853+0000","last_peered":"2022-04-22T17:22:18.439853+0000","last_clean":"2022-04-22T17:22:18.439853+0000","last_became_active":"2022-04-22T17:22:18.439094+0000","last_became_peered":"2022-04-22T17:22:18.439094+0000","last_unstale":"2022-04-22T17:22:18.439853+0000","last_undegraded":"2022-04-22T17:22:18.439853+0000","last_fullsized":"2022-04-22T17:22:18.439853+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.460275+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:22:18.460275+0000","last_peered":"2022-04-22T17:22:18.460275+0000","last_clean":"2022-04-22T17:22:18.460275+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:22:18.460275+0000","last_undegraded":"2022-04-22T17:22:18.460275+0000","last_fullsized":"2022-04-22T17:22:18.460275+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.442582+0000","last_change":"2022-04-22T17:22:18.442367+0000","last_active":"2022-04-22T17:22:18.442582+0000","last_peered":"2022-04-22T17:22:18.442582+0000","last_clean":"2022-04-22T17:22:18.442582+0000","last_became_active":"2022-04-22T17:22:18.442081+0000","last_became_peered":"2022-04-22T17:22:18.442081+0000","last_unstale":"2022-04-22T17:22:18.442582+0000","last_undegraded":"2022-04-22T17:22:18.442582+0000","last_fullsized":"2022-04-22T17:22:18.442582+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.436463+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:22:18.436463+0000","last_peered":"2022-04-22T17:22:18.436463+0000","last_clean":"2022-04-22T17:22:18.436463+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:22:18.436463+0000","last_undegraded":"2022-04-22T17:22:18.436463+0000","last_fullsized":"2022-04-22T17:22:18.436463+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438698+0000","last_change":"2022-04-22T17:22:18.436856+0000","last_active":"2022-04-22T17:22:18.438698+0000","last_peered":"2022-04-22T17:22:18.438698+0000","last_clean":"2022-04-22T17:22:18.438698+0000","last_became_active":"2022-04-22T17:22:18.436561+0000","last_became_peered":"2022-04-22T17:22:18.436561+0000","last_unstale":"2022-04-22T17:22:18.438698+0000","last_undegraded":"2022-04-22T17:22:18.438698+0000","last_fullsized":"2022-04-22T17:22:18.438698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438387+0000","last_change":"2022-04-22T17:22:18.438089+0000","last_active":"2022-04-22T17:22:18.438387+0000","last_peered":"2022-04-22T17:22:18.438387+0000","last_clean":"2022-04-22T17:22:18.438387+0000","last_became_active":"2022-04-22T17:22:18.437829+0000","last_became_peered":"2022-04-22T17:22:18.437829+0000","last_unstale":"2022-04-22T17:22:18.438387+0000","last_undegraded":"2022-04-22T17:22:18.438387+0000","last_fullsized":"2022-04-22T17:22:18.438387+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.438540+0000","last_change":"2022-04-22T17:22:18.437041+0000","last_active":"2022-04-22T17:22:18.438540+0000","last_peered":"2022-04-22T17:22:18.438540+0000","last_clean":"2022-04-22T17:22:18.438540+0000","last_became_active":"2022-04-22T17:22:18.436782+0000","last_became_peered":"2022-04-22T17:22:18.436782+0000","last_unstale":"2022-04-22T17:22:18.438540+0000","last_undegraded":"2022-04-22T17:22:18.438540+0000","last_fullsized":"2022-04-22T17:22:18.438540+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.441050+0000","last_change":"2022-04-22T17:22:18.438738+0000","last_active":"2022-04-22T17:22:18.441050+0000","last_peered":"2022-04-22T17:22:18.441050+0000","last_clean":"2022-04-22T17:22:18.441050+0000","last_became_active":"2022-04-22T17:22:18.438520+0000","last_became_peered":"2022-04-22T17:22:18.438520+0000","last_unstale":"2022-04-22T17:22:18.441050+0000","last_undegraded":"2022-04-22T17:22:18.441050+0000","last_fullsized":"2022-04-22T17:22:18.441050+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.449239+0000","last_change":"2022-04-22T17:22:18.448646+0000","last_active":"2022-04-22T17:22:18.449239+0000","last_peered":"2022-04-22T17:22:18.449239+0000","last_clean":"2022-04-22T17:22:18.449239+0000","last_became_active":"2022-04-22T17:22:18.448263+0000","last_became_peered":"2022-04-22T17:22:18.448263+0000","last_unstale":"2022-04-22T17:22:18.449239+0000","last_undegraded":"2022-04-22T17:22:18.449239+0000","last_fullsized":"2022-04-22T17:22:18.449239+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.461753+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:22:18.461753+0000","last_peered":"2022-04-22T17:22:18.461753+0000","last_clean":"2022-04-22T17:22:18.461753+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:22:18.461753+0000","last_undegraded":"2022-04-22T17:22:18.461753+0000","last_fullsized":"2022-04-22T17:22:18.461753+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":53,"state":"active+clean","last_fresh":"2022-04-22T17:22:18.454854+0000","last_change":"2022-04-22T17:22:18.453245+0000","last_active":"2022-04-22T17:22:18.454854+0000","last_peered":"2022-04-22T17:22:18.454854+0000","last_clean":"2022-04-22T17:22:18.454854+0000","last_became_active":"2022-04-22T17:22:18.452960+0000","last_became_peered":"2022-04-22T17:22:18.452960+0000","last_unstale":"2022-04-22T17:22:18.454854+0000","last_undegraded":"2022-04-22T17:22:18.454854+0000","last_fullsized":"2022-04-22T17:22:18.454854+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430218,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5524,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739628,"statfs":{"total":95995035648,"available":95989379072,"internally_reserved":0,"allocated":536576,"data_stored":216122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]}]},{"osd":6,"up_from":42,"seq":180388626447,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5588,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739564,"statfs":{"total":95995035648,"available":95989313536,"internally_reserved":0,"allocated":536576,"data_stored":216767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.018}]},{"osd":1,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":1.0529999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.90900000000000003}]},{"osd":3,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":1.101},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":1.1519999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5776,"kb_used_data":520,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739376,"statfs":{"total":95995035648,"available":95989121024,"internally_reserved":0,"allocated":532480,"data_stored":215391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.55600000000000005,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63600000000000001,"15min":0.57699999999999996},"min":{"1min":0.53700000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.94499999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.74399999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.68100000000000005},"min":{"1min":0.61399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.68400000000000005}]},{"osd":3,"last update":"Fri Apr 22 17:22:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001}]},{"osd":4,"last update":"Fri Apr 22 17:22:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0049999999999999,"15min":0.995},"min":{"1min":0.85799999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.96599999999999997,"15min":0.95399999999999996},"min":{"1min":0.77400000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":1.1160000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.161},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.143}]},{"osd":6,"last update":"Fri Apr 22 17:22:22 2022","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":1.25}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739236,"statfs":{"total":95995035648,"available":95988977664,"internally_reserved":0,"allocated":544768,"data_stored":216246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:22:12 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.55500000000000005,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82499999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.98999999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:21:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.749},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.64400000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.95599999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.72099999999999997}]},{"osd":3,"last update":"Fri Apr 22 17:21:58 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79200000000000004,"15min":0.79400000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.874},{"interface":"front","average":{"1min":0.873,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.96499999999999997}]},{"osd":4,"last update":"Fri Apr 22 17:22:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97199999999999998,"15min":0.97099999999999997},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.298,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.77400000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.516,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.054}]},{"osd":5,"last update":"Fri Apr 22 17:21:52 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.212}]},{"osd":6,"last update":"Fri Apr 22 17:22:19 2022","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.1890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5704,"kb_used_data":512,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":524288,"data_stored":214884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.2609999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.51000000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":1.1839999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:21:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.442,"15min":0.442},"max":{"1min":1.1739999999999999,"5min":1.294,"15min":1.294},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.78600000000000003,"15min":0.77500000000000002},"min":{"1min":0.66400000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":1.262}]},{"osd":3,"last update":"Fri Apr 22 17:22:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.57199999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0680000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.79600000000000004}]},{"osd":4,"last update":"Fri Apr 22 17:21:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":1.004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.88}]},{"osd":5,"last update":"Fri Apr 22 17:21:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.3129999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.111}]},{"osd":6,"last update":"Fri Apr 22 17:22:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.419}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]}]},{"osd":3,"up_from":25,"seq":107374182432,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5664,"kb_used_data":536,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":548864,"data_stored":217186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.006,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.745,"15min":0.745},"min":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1060000000000001,"5min":1.133,"15min":1.133},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.628,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1459999999999999,"5min":1.276,"15min":1.276},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.099,"5min":1.202,"15min":1.202},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Apr 22 17:22:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.194,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.105,"5min":1.224,"15min":1.224},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Apr 22 17:21:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":1.123},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.80800000000000005}]},{"osd":5,"last update":"Fri Apr 22 17:21:50 2022","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.028}]},{"osd":6,"last update":"Fri Apr 22 17:22:20 2022","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018906,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5660,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":544768,"data_stored":216226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.218},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.84199999999999997}]},{"osd":1,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.276}]},{"osd":2,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":1.115},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:21:27 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":1.532}]},{"osd":5,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.258}]},{"osd":6,"last update":"Fri Apr 22 17:22:20 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":1.3720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822676,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739556,"statfs":{"total":95995035648,"available":95989305344,"internally_reserved":0,"allocated":544768,"data_stored":217172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":1.03},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.94899999999999995}]},{"osd":2,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Apr 22 17:21:51 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":1.042},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.1819999999999999}]},{"osd":6,"last update":"Fri Apr 22 17:22:20 2022","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":1.1339999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:22:25.722 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-04-22T17:22:25.723 INFO:tasks.cephadm:Adding prometheus.a on smithi121 2022-04-22T17:22:25.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply prometheus '1;smithi121=a' 2022-04-22T17:22:27.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:27 smithi121 bash[12679]: audit 2022-04-22T17:22:25.648229+0000 mgr.y (mgr.14146) 236 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:27.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:27 smithi115 bash[12737]: audit 2022-04-22T17:22:25.648229+0000 mgr.y (mgr.14146) 236 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:27.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:27 smithi115 bash[18070]: audit 2022-04-22T17:22:25.648229+0000 mgr.y (mgr.14146) 236 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:22:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:28 smithi121 bash[12679]: cluster 2022-04-22T17:22:27.087659+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v219: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 534 B/s wr, 2 op/s 2022-04-22T17:22:28.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:28 smithi115 bash[12737]: cluster 2022-04-22T17:22:27.087659+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v219: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 534 B/s wr, 2 op/s 2022-04-22T17:22:28.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:28 smithi115 bash[18070]: cluster 2022-04-22T17:22:27.087659+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v219: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 534 B/s wr, 2 op/s 2022-04-22T17:22:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled prometheus update... 2022-04-22T17:22:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:29 smithi121 bash[12679]: cluster 2022-04-22T17:22:29.088607+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v220: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 423 B/s wr, 1 op/s 2022-04-22T17:22:29.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:29 smithi115 bash[18070]: cluster 2022-04-22T17:22:29.088607+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v220: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 423 B/s wr, 1 op/s 2022-04-22T17:22:29.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:29 smithi115 bash[12737]: cluster 2022-04-22T17:22:29.088607+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v220: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 423 B/s wr, 1 op/s 2022-04-22T17:22:30.333 DEBUG:teuthology.orchestra.run.smithi121:prometheus.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@prometheus.a.service 2022-04-22T17:22:30.335 INFO:tasks.cephadm:Adding node-exporter.a on smithi115 2022-04-22T17:22:30.335 INFO:tasks.cephadm:Adding node-exporter.b on smithi121 2022-04-22T17:22:30.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply node-exporter '2;smithi115=a;smithi121=b' 2022-04-22T17:22:30.353 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.505045+0000 mgr.y (mgr.14146) 239 : audit [DBG] from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: cephadm 2022-04-22T17:22:29.506375+0000 mgr.y (mgr.14146) 240 : cephadm [INF] Saving service prometheus spec with placement smithi121=a;count:1 2022-04-22T17:22:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.513093+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.514468+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.516531+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.518236+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: audit 2022-04-22T17:22:29.525316+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:30 smithi121 bash[12679]: cephadm 2022-04-22T17:22:29.543254+0000 mgr.y (mgr.14146) 241 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:22:30.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.505045+0000 mgr.y (mgr.14146) 239 : audit [DBG] from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:30.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: cephadm 2022-04-22T17:22:29.506375+0000 mgr.y (mgr.14146) 240 : cephadm [INF] Saving service prometheus spec with placement smithi121=a;count:1 2022-04-22T17:22:30.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.513093+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.514468+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:30.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.516531+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.518236+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: audit 2022-04-22T17:22:29.525316+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[12737]: cephadm 2022-04-22T17:22:29.543254+0000 mgr.y (mgr.14146) 241 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.505045+0000 mgr.y (mgr.14146) 239 : audit [DBG] from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: cephadm 2022-04-22T17:22:29.506375+0000 mgr.y (mgr.14146) 240 : cephadm [INF] Saving service prometheus spec with placement smithi121=a;count:1 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.513093+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.514468+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:30.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.516531+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:30.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.518236+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:30.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: audit 2022-04-22T17:22:29.525316+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:30.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:30 smithi115 bash[18070]: cephadm 2022-04-22T17:22:29.543254+0000 mgr.y (mgr.14146) 241 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:22:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:32 smithi121 bash[12679]: cluster 2022-04-22T17:22:31.089404+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v221: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 351 B/s wr, 1 op/s 2022-04-22T17:22:32.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:32 smithi115 bash[12737]: cluster 2022-04-22T17:22:31.089404+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v221: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 351 B/s wr, 1 op/s 2022-04-22T17:22:32.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:32 smithi115 bash[18070]: cluster 2022-04-22T17:22:31.089404+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v221: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 351 B/s wr, 1 op/s 2022-04-22T17:22:33.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:33 smithi121 bash[12679]: cluster 2022-04-22T17:22:33.090364+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v222: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 1 op/s 2022-04-22T17:22:33.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:33 smithi115 bash[12737]: cluster 2022-04-22T17:22:33.090364+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v222: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 1 op/s 2022-04-22T17:22:33.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:33 smithi115 bash[18070]: cluster 2022-04-22T17:22:33.090364+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v222: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 1 op/s 2022-04-22T17:22:35.460 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:35 smithi121 bash[12679]: cluster 2022-04-22T17:22:35.091413+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v223: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T17:22:35.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:35 smithi115 bash[12737]: cluster 2022-04-22T17:22:35.091413+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v223: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T17:22:35.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:35 smithi115 bash[18070]: cluster 2022-04-22T17:22:35.091413+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v223: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T17:22:35.831 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled node-exporter update... 2022-04-22T17:22:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:36 smithi121 bash[12679]: audit 2022-04-22T17:22:35.823456+0000 mgr.y (mgr.14146) 245 : audit [DBG] from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:36 smithi121 bash[12679]: cephadm 2022-04-22T17:22:35.825010+0000 mgr.y (mgr.14146) 246 : cephadm [INF] Saving service node-exporter spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:37.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:36 smithi121 bash[12679]: audit 2022-04-22T17:22:35.831949+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:37.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[12737]: audit 2022-04-22T17:22:35.823456+0000 mgr.y (mgr.14146) 245 : audit [DBG] from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:37.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[12737]: cephadm 2022-04-22T17:22:35.825010+0000 mgr.y (mgr.14146) 246 : cephadm [INF] Saving service node-exporter spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:37.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[12737]: audit 2022-04-22T17:22:35.831949+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[18070]: audit 2022-04-22T17:22:35.823456+0000 mgr.y (mgr.14146) 245 : audit [DBG] from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi121=b", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[18070]: cephadm 2022-04-22T17:22:35.825010+0000 mgr.y (mgr.14146) 246 : cephadm [INF] Saving service node-exporter spec with placement smithi115=a;smithi121=b;count:2 2022-04-22T17:22:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:36 smithi115 bash[18070]: audit 2022-04-22T17:22:35.831949+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:37.362 DEBUG:teuthology.orchestra.run.smithi115:node-exporter.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.a.service 2022-04-22T17:22:37.365 DEBUG:teuthology.orchestra.run.smithi121:node-exporter.b> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.b.service 2022-04-22T17:22:37.395 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:37.407 INFO:tasks.cephadm:Adding alertmanager.a on smithi115 2022-04-22T17:22:37.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply alertmanager '1;smithi115=a' 2022-04-22T17:22:37.429 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.045854+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2022-04-22T17:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.046559+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2022-04-22T17:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.047122+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.047558+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]: dispatch 2022-04-22T17:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.047940+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: audit 2022-04-22T17:22:37.048317+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]: dispatch 2022-04-22T17:22:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:37 smithi121 bash[12679]: cluster 2022-04-22T17:22:37.092219+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v224: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.045854+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.046559+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.047122+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.047558+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]: dispatch 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.047940+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: audit 2022-04-22T17:22:37.048317+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[12737]: cluster 2022-04-22T17:22:37.092219+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v224: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.045854+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.046559+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.047122+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.047558+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.047940+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2022-04-22T17:22:38.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: audit 2022-04-22T17:22:37.048317+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]: dispatch 2022-04-22T17:22:38.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:37 smithi115 bash[18070]: cluster 2022-04-22T17:22:37.092219+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v224: 97 pgs: 97 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846157+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2022-04-22T17:22:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846260+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846344+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846445+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]': finished 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846536+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:37.846640+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]': finished 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: cluster 2022-04-22T17:22:37.846713+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:38.638781+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:22:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:38 smithi121 bash[12679]: audit 2022-04-22T17:22:38.641535+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:22:39.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846157+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2022-04-22T17:22:39.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846260+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2022-04-22T17:22:39.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846344+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846445+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]': finished 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846536+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:37.846640+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]': finished 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: cluster 2022-04-22T17:22:37.846713+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:38.638781+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[12737]: audit 2022-04-22T17:22:38.641535+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846157+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846260+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2022-04-22T17:22:39.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846344+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846445+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [7, 1]}]': finished 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846536+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:37.846640+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [4, 3]}]': finished 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: cluster 2022-04-22T17:22:37.846713+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:38.638781+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:22:39.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:38 smithi115 bash[18070]: audit 2022-04-22T17:22:38.641535+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:22:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:39 smithi121 bash[12679]: cluster 2022-04-22T17:22:38.857013+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2022-04-22T17:22:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:39 smithi121 bash[12679]: cluster 2022-04-22T17:22:39.093014+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v227: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:40.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:39 smithi115 bash[12737]: cluster 2022-04-22T17:22:38.857013+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2022-04-22T17:22:40.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:39 smithi115 bash[12737]: cluster 2022-04-22T17:22:39.093014+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v227: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:40.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:39 smithi115 bash[18070]: cluster 2022-04-22T17:22:38.857013+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2022-04-22T17:22:40.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:39 smithi115 bash[18070]: cluster 2022-04-22T17:22:39.093014+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v227: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:42.496 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:42 smithi121 bash[12679]: cluster 2022-04-22T17:22:41.093848+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v228: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:42.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:42 smithi115 bash[12737]: cluster 2022-04-22T17:22:41.093848+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v228: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:42.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:42 smithi115 bash[18070]: cluster 2022-04-22T17:22:41.093848+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v228: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled alertmanager update... 2022-04-22T17:22:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:44 smithi121 bash[12679]: audit 2022-04-22T17:22:43.075451+0000 mgr.y (mgr.14146) 250 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:44 smithi121 bash[12679]: cephadm 2022-04-22T17:22:43.077059+0000 mgr.y (mgr.14146) 251 : cephadm [INF] Saving service alertmanager spec with placement smithi115=a;count:1 2022-04-22T17:22:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:44 smithi121 bash[12679]: audit 2022-04-22T17:22:43.083967+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:44 smithi121 bash[12679]: cluster 2022-04-22T17:22:43.094517+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v229: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:44.450 DEBUG:teuthology.orchestra.run.smithi115:alertmanager.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@alertmanager.a.service 2022-04-22T17:22:44.453 INFO:tasks.cephadm:Adding grafana.a on smithi121 2022-04-22T17:22:44.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph orch apply grafana '1;smithi121=a' 2022-04-22T17:22:44.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[12737]: audit 2022-04-22T17:22:43.075451+0000 mgr.y (mgr.14146) 250 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:44.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[12737]: cephadm 2022-04-22T17:22:43.077059+0000 mgr.y (mgr.14146) 251 : cephadm [INF] Saving service alertmanager spec with placement smithi115=a;count:1 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[12737]: audit 2022-04-22T17:22:43.083967+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[12737]: cluster 2022-04-22T17:22:43.094517+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v229: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[18070]: audit 2022-04-22T17:22:43.075451+0000 mgr.y (mgr.14146) 250 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[18070]: cephadm 2022-04-22T17:22:43.077059+0000 mgr.y (mgr.14146) 251 : cephadm [INF] Saving service alertmanager spec with placement smithi115=a;count:1 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[18070]: audit 2022-04-22T17:22:43.083967+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:44.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:44 smithi115 bash[18070]: cluster 2022-04-22T17:22:43.094517+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v229: 97 pgs: 1 peering, 96 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:44.483 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:45.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:45 smithi115 bash[12737]: cluster 2022-04-22T17:22:45.095652+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v230: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:45.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:45 smithi115 bash[18070]: cluster 2022-04-22T17:22:45.095652+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v230: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:45.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:45 smithi121 bash[12679]: cluster 2022-04-22T17:22:45.095652+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v230: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:47 smithi121 bash[12679]: cluster 2022-04-22T17:22:47.096542+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v231: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:47.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:47 smithi115 bash[12737]: cluster 2022-04-22T17:22:47.096542+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v231: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:47.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:47 smithi115 bash[18070]: cluster 2022-04-22T17:22:47.096542+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v231: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:48.416 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:48 smithi121 systemd[1]: Started Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:22:49.711 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:49 smithi121 bash[12679]: audit 2022-04-22T17:22:48.315953+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:49.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:49 smithi121 bash[12679]: audit 2022-04-22T17:22:48.318407+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:49.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:49 smithi121 bash[12679]: audit 2022-04-22T17:22:48.320670+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:49.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:49 smithi121 bash[12679]: audit 2022-04-22T17:22:48.322208+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:49.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:49 smithi121 bash[12679]: cluster 2022-04-22T17:22:49.097567+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v232: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[12737]: audit 2022-04-22T17:22:48.315953+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[12737]: audit 2022-04-22T17:22:48.318407+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[12737]: audit 2022-04-22T17:22:48.320670+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[12737]: audit 2022-04-22T17:22:48.322208+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[12737]: cluster 2022-04-22T17:22:49.097567+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v232: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:49.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[18070]: audit 2022-04-22T17:22:48.315953+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:49.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[18070]: audit 2022-04-22T17:22:48.318407+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:22:49.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[18070]: audit 2022-04-22T17:22:48.320670+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:22:49.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[18070]: audit 2022-04-22T17:22:48.322208+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:22:49.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:49 smithi115 bash[18070]: cluster 2022-04-22T17:22:49.097567+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v232: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:50.217 INFO:teuthology.orchestra.run.smithi121.stdout:Scheduled grafana update... 2022-04-22T17:22:50.440 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2022-04-22T17:22:50.441 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2022-04-22T17:22:50.441 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2022-04-22T17:22:50.441 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:339 host_details="(Linux 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 smithi121 (none))" 2022-04-22T17:22:50.441 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2022-04-22T17:22:50.551 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.070Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.072Z caller=main.go:678 msg="Starting TSDB ..." 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.072Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.076Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.076Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.076Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=339.706µs 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.077Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.077Z caller=main.go:695 msg="TSDB started" 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.077Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.088Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:22:50.552 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:22:50 smithi121 bash[26962]: level=info ts=2022-04-22T17:22:50.088Z caller=main.go:646 msg="Server is ready to receive web requests." 2022-04-22T17:22:51.116 DEBUG:teuthology.orchestra.run.smithi121:grafana.a> sudo journalctl -f -n 0 -u ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@grafana.a.service 2022-04-22T17:22:51.119 INFO:tasks.cephadm:Setting up client nodes... 2022-04-22T17:22:51.119 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-22T17:22:51.147 INFO:journalctl@ceph.grafana.a.smithi121.stdout:-- Logs begin at Sun 2022-04-17 12:19:38 UTC. -- 2022-04-22T17:22:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:51 smithi121 bash[12679]: audit 2022-04-22T17:22:50.209734+0000 mgr.y (mgr.14146) 256 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:51.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:51 smithi121 bash[12679]: cephadm 2022-04-22T17:22:50.211165+0000 mgr.y (mgr.14146) 257 : cephadm [INF] Saving service grafana spec with placement smithi121=a;count:1 2022-04-22T17:22:51.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:51 smithi121 bash[12679]: audit 2022-04-22T17:22:50.216656+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:51.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:51 smithi121 bash[12679]: cluster 2022-04-22T17:22:51.098477+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v233: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:51.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[12737]: audit 2022-04-22T17:22:50.209734+0000 mgr.y (mgr.14146) 256 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:51.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[12737]: cephadm 2022-04-22T17:22:50.211165+0000 mgr.y (mgr.14146) 257 : cephadm [INF] Saving service grafana spec with placement smithi121=a;count:1 2022-04-22T17:22:51.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[12737]: audit 2022-04-22T17:22:50.216656+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:51.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[12737]: cluster 2022-04-22T17:22:51.098477+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v233: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:51.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[18070]: audit 2022-04-22T17:22:50.209734+0000 mgr.y (mgr.14146) 256 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:22:51.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[18070]: cephadm 2022-04-22T17:22:50.211165+0000 mgr.y (mgr.14146) 257 : cephadm [INF] Saving service grafana spec with placement smithi121=a;count:1 2022-04-22T17:22:51.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[18070]: audit 2022-04-22T17:22:50.216656+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:51.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:51 smithi115 bash[18070]: cluster 2022-04-22T17:22:51.098477+0000 mgr.y (mgr.14146) 258 : cluster [DBG] pgmap v233: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:53 smithi121 bash[12679]: audit 2022-04-22T17:22:52.243719+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:53 smithi121 bash[12679]: cluster 2022-04-22T17:22:53.099213+0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v234: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:53.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:53 smithi115 bash[12737]: audit 2022-04-22T17:22:52.243719+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:53.585 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:53 smithi115 bash[12737]: cluster 2022-04-22T17:22:53.099213+0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v234: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:53.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:53 smithi115 bash[18070]: audit 2022-04-22T17:22:52.243719+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:53.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:53 smithi115 bash[18070]: cluster 2022-04-22T17:22:53.099213+0000 mgr.y (mgr.14146) 259 : cluster [DBG] pgmap v234: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:54.681 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:54 smithi115 systemd[1]: Started Ceph node-exporter.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:22:54.681 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[32345]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2022-04-22T17:22:54.935 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:54 smithi121 bash[12679]: audit 2022-04-22T17:22:53.680044+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:54.936 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:54 smithi121 bash[12679]: audit 2022-04-22T17:22:53.687022+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:54.936 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:54 smithi121 bash[12679]: cephadm 2022-04-22T17:22:53.690359+0000 mgr.y (mgr.14146) 260 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:22:54.936 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:54 smithi121 bash[12679]: audit 2022-04-22T17:22:54.431077+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[18070]: audit 2022-04-22T17:22:53.680044+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[18070]: audit 2022-04-22T17:22:53.687022+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[18070]: cephadm 2022-04-22T17:22:53.690359+0000 mgr.y (mgr.14146) 260 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[18070]: audit 2022-04-22T17:22:54.431077+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[12737]: audit 2022-04-22T17:22:53.680044+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[12737]: audit 2022-04-22T17:22:53.687022+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[12737]: cephadm 2022-04-22T17:22:53.690359+0000 mgr.y (mgr.14146) 260 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:22:55.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:54 smithi115 bash[12737]: audit 2022-04-22T17:22:54.431077+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:55.203 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2022-04-22T17:22:55.203 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQBv5GJiH6yuCxAAkTgR5+3EJQAMCpZ5k5o/aQ== 2022-04-22T17:22:55.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 systemd[1]: Started Ceph node-exporter.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:22:55.392 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: v0.18.1: Pulling from prom/node-exporter 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 49a2d53aa1af: Pulling fs layer 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 3589a6efd9ce: Pulling fs layer 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 190160031744: Pulling fs layer 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 3589a6efd9ce: Download complete 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 49a2d53aa1af: Verifying Checksum 2022-04-22T17:22:55.393 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 49a2d53aa1af: Download complete 2022-04-22T17:22:55.685 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 190160031744: Verifying Checksum 2022-04-22T17:22:55.686 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[32345]: 190160031744: Download complete 2022-04-22T17:22:55.686 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[27480]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2022-04-22T17:22:55.939 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[27480]: v0.18.1: Pulling from prom/node-exporter 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: cephadm 2022-04-22T17:22:54.432283+0000 mgr.y (mgr.14146) 261 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: cluster 2022-04-22T17:22:55.099846+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v235: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: audit 2022-04-22T17:22:55.194886+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.115:0/2770831798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: audit 2022-04-22T17:22:55.195730+0000 mon.a (mon.0) 646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: audit 2022-04-22T17:22:55.202257+0000 mon.a (mon.0) 647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:22:55.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[12679]: audit 2022-04-22T17:22:55.216251+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: cephadm 2022-04-22T17:22:54.432283+0000 mgr.y (mgr.14146) 261 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: cluster 2022-04-22T17:22:55.099846+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v235: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: audit 2022-04-22T17:22:55.194886+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.115:0/2770831798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: audit 2022-04-22T17:22:55.195730+0000 mon.a (mon.0) 646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: audit 2022-04-22T17:22:55.202257+0000 mon.a (mon.0) 647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:22:56.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[12737]: audit 2022-04-22T17:22:55.216251+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: cephadm 2022-04-22T17:22:54.432283+0000 mgr.y (mgr.14146) 261 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: cluster 2022-04-22T17:22:55.099846+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v235: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: audit 2022-04-22T17:22:55.194886+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.115:0/2770831798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: audit 2022-04-22T17:22:55.195730+0000 mon.a (mon.0) 646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: audit 2022-04-22T17:22:55.202257+0000 mon.a (mon.0) 647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:22:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:55 smithi115 bash[18070]: audit 2022-04-22T17:22:55.216251+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[27480]: 49a2d53aa1af: Pulling fs layer 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[27480]: 3589a6efd9ce: Pulling fs layer 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:55 smithi121 bash[27480]: 190160031744: Pulling fs layer 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 3589a6efd9ce: Verifying Checksum 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 3589a6efd9ce: Download complete 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 49a2d53aa1af: Verifying Checksum 2022-04-22T17:22:56.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 49a2d53aa1af: Download complete 2022-04-22T17:22:56.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 190160031744: Verifying Checksum 2022-04-22T17:22:56.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 190160031744: Download complete 2022-04-22T17:22:56.517 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 49a2d53aa1af: Pull complete 2022-04-22T17:22:56.778 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:56 smithi115 bash[32345]: 49a2d53aa1af: Pull complete 2022-04-22T17:22:56.779 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:22:56.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-04-22T17:22:56.780 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-04-22T17:22:56.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-22T17:22:56.966 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 3589a6efd9ce: Pull complete 2022-04-22T17:22:57.037 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:56 smithi115 bash[32345]: 3589a6efd9ce: Pull complete 2022-04-22T17:22:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:57 smithi121 bash[12679]: cephadm 2022-04-22T17:22:55.225871+0000 mgr.y (mgr.14146) 263 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:22:57.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:56 smithi121 bash[27480]: 190160031744: Pull complete 2022-04-22T17:22:57.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:57 smithi115 bash[12737]: cephadm 2022-04-22T17:22:55.225871+0000 mgr.y (mgr.14146) 263 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:22:57.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:57 smithi115 bash[18070]: cephadm 2022-04-22T17:22:55.225871+0000 mgr.y (mgr.14146) 263 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:22:57.740 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:57 smithi115 bash[32345]: 190160031744: Pull complete 2022-04-22T17:22:57.740 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:57 smithi115 bash[32345]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2022-04-22T17:22:57.740 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:22:57 smithi115 bash[32345]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2022-04-22T17:22:58.016 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:57 smithi121 bash[27480]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2022-04-22T17:22:58.517 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:58 smithi121 bash[27480]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2022-04-22T17:22:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:58 smithi121 bash[12679]: cluster 2022-04-22T17:22:57.100403+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v236: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:58 smithi121 bash[12679]: audit 2022-04-22T17:22:57.249040+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:58.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:58 smithi115 bash[12737]: cluster 2022-04-22T17:22:57.100403+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v236: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:58.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:58 smithi115 bash[12737]: audit 2022-04-22T17:22:57.249040+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:58.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:58 smithi115 bash[18070]: cluster 2022-04-22T17:22:57.100403+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v236: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:58.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:58 smithi115 bash[18070]: audit 2022-04-22T17:22:57.249040+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:22:59.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[12679]: cluster 2022-04-22T17:22:59.101397+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v237: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:59.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:22:59 smithi115 bash[12737]: cluster 2022-04-22T17:22:59.101397+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v237: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:59.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:22:59 smithi115 bash[18070]: cluster 2022-04-22T17:22:59.101397+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v237: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:22:59.815 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2022-04-22T17:22:59.815 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2022-04-22T17:22:59.816 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2022-04-22T17:22:59.816 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - arp" source="node_exporter.go:104" 2022-04-22T17:22:59.816 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - bcache" source="node_exporter.go:104" 2022-04-22T17:22:59.816 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - bonding" source="node_exporter.go:104" 2022-04-22T17:22:59.817 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2022-04-22T17:22:59.817 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - cpu" source="node_exporter.go:104" 2022-04-22T17:22:59.817 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2022-04-22T17:22:59.817 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2022-04-22T17:22:59.817 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - edac" source="node_exporter.go:104" 2022-04-22T17:22:59.818 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - entropy" source="node_exporter.go:104" 2022-04-22T17:22:59.818 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - filefd" source="node_exporter.go:104" 2022-04-22T17:22:59.818 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2022-04-22T17:22:59.818 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2022-04-22T17:22:59.819 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2022-04-22T17:22:59.819 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2022-04-22T17:22:59.819 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2022-04-22T17:22:59.819 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2022-04-22T17:22:59.819 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2022-04-22T17:22:59.820 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - netclass" source="node_exporter.go:104" 2022-04-22T17:22:59.820 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - netdev" source="node_exporter.go:104" 2022-04-22T17:22:59.820 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - netstat" source="node_exporter.go:104" 2022-04-22T17:22:59.820 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - nfs" source="node_exporter.go:104" 2022-04-22T17:22:59.821 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2022-04-22T17:22:59.821 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - pressure" source="node_exporter.go:104" 2022-04-22T17:22:59.822 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2022-04-22T17:22:59.822 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - stat" source="node_exporter.go:104" 2022-04-22T17:22:59.823 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - textfile" source="node_exporter.go:104" 2022-04-22T17:22:59.823 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - time" source="node_exporter.go:104" 2022-04-22T17:22:59.823 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - uname" source="node_exporter.go:104" 2022-04-22T17:22:59.823 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2022-04-22T17:22:59.823 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - xfs" source="node_exporter.go:104" 2022-04-22T17:22:59.824 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg=" - zfs" source="node_exporter.go:104" 2022-04-22T17:22:59.824 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:22:59 smithi121 bash[27480]: time="2022-04-22T17:22:59Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2022-04-22T17:23:00.788 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - arp" source="node_exporter.go:104" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - bcache" source="node_exporter.go:104" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - bonding" source="node_exporter.go:104" 2022-04-22T17:23:00.789 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - cpu" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - edac" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - entropy" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - filefd" source="node_exporter.go:104" 2022-04-22T17:23:00.790 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2022-04-22T17:23:00.791 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - netclass" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - netdev" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - netstat" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - nfs" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - pressure" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - stat" source="node_exporter.go:104" 2022-04-22T17:23:00.792 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - textfile" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - time" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - uname" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - xfs" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg=" - zfs" source="node_exporter.go:104" 2022-04-22T17:23:00.793 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:23:00 smithi115 bash[32345]: time="2022-04-22T17:23:00Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2022-04-22T17:23:01.968 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2022-04-22T17:23:02.007 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB15GJiI/5eORAApEucVAVQsGaxqIndNktwLA== 2022-04-22T17:23:02.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[12737]: cluster 2022-04-22T17:23:01.101993+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v238: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:02.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[12737]: audit 2022-04-22T17:23:01.962203+0000 mon.a (mon.0) 650 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:23:02.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[12737]: audit 2022-04-22T17:23:01.967729+0000 mon.a (mon.0) 651 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:23:02.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[18070]: cluster 2022-04-22T17:23:01.101993+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v238: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:02.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[18070]: audit 2022-04-22T17:23:01.962203+0000 mon.a (mon.0) 650 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:23:02.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:02 smithi115 bash[18070]: audit 2022-04-22T17:23:01.967729+0000 mon.a (mon.0) 651 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:23:02.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:02 smithi121 bash[12679]: cluster 2022-04-22T17:23:01.101993+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v238: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:02.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:02 smithi121 bash[12679]: audit 2022-04-22T17:23:01.962203+0000 mon.a (mon.0) 650 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-04-22T17:23:02.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:02 smithi121 bash[12679]: audit 2022-04-22T17:23:01.967729+0000 mon.a (mon.0) 651 : audit [INF] from='client.? 172.21.15.121:0/2024465878' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-04-22T17:23:02.769 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T17:23:02.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2022-04-22T17:23:02.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-04-22T17:23:02.792 INFO:tasks.cephadm:Waiting for OSDs to come up 2022-04-22T17:23:02.793 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-04-22T17:23:03.053 INFO:teuthology.misc.health.smithi115.stdout: 2022-04-22T17:23:03.053 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":56,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:22:38.841979+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.3b","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:23:03.066 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-04-22T17:23:03.066 INFO:tasks.cephadm:Creating RBD pool 2022-04-22T17:23:03.067 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- sudo ceph --cluster ceph osd pool create rbd 8 2022-04-22T17:23:03.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:03 smithi121 bash[12679]: audit 2022-04-22T17:23:03.051432+0000 mon.a (mon.0) 652 : audit [DBG] from='client.? 172.21.15.115:0/2021097896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:03.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:03 smithi115 bash[18070]: audit 2022-04-22T17:23:03.051432+0000 mon.a (mon.0) 652 : audit [DBG] from='client.? 172.21.15.115:0/2021097896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:03.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:03 smithi115 bash[12737]: audit 2022-04-22T17:23:03.051432+0000 mon.a (mon.0) 652 : audit [DBG] from='client.? 172.21.15.115:0/2021097896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:04.488 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:04 smithi115 bash[12737]: cluster 2022-04-22T17:23:03.102574+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v239: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:04.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:04 smithi115 bash[18070]: cluster 2022-04-22T17:23:03.102574+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v239: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:04 smithi121 bash[12679]: cluster 2022-04-22T17:23:03.102574+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v239: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:06.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:06 smithi115 bash[18070]: cluster 2022-04-22T17:23:05.103240+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v240: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:06.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:06 smithi115 bash[12737]: cluster 2022-04-22T17:23:05.103240+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v240: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:06.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:06 smithi121 bash[12679]: cluster 2022-04-22T17:23:05.103240+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v240: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:08.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:08 smithi115 systemd[1]: Started Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:08 smithi121 bash[12679]: cluster 2022-04-22T17:23:07.103870+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v241: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:08 smithi121 bash[12679]: audit 2022-04-22T17:23:08.038997+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:08.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:08 smithi115 bash[18070]: cluster 2022-04-22T17:23:07.103870+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v241: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:08.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:08 smithi115 bash[18070]: audit 2022-04-22T17:23:08.038997+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:08.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:08 smithi115 bash[12737]: cluster 2022-04-22T17:23:07.103870+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v241: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:08.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:08 smithi115 bash[12737]: audit 2022-04-22T17:23:08.038997+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:09 smithi121 bash[12679]: audit 2022-04-22T17:23:08.117911+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:09 smithi121 bash[12679]: audit 2022-04-22T17:23:08.128468+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:09 smithi121 bash[12679]: audit 2022-04-22T17:23:08.129188+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:09 smithi121 bash[12679]: audit 2022-04-22T17:23:08.138054+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[12737]: audit 2022-04-22T17:23:08.117911+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[12737]: audit 2022-04-22T17:23:08.128468+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[12737]: audit 2022-04-22T17:23:08.129188+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:09.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[12737]: audit 2022-04-22T17:23:08.138054+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[18070]: audit 2022-04-22T17:23:08.117911+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[18070]: audit 2022-04-22T17:23:08.128468+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:09.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[18070]: audit 2022-04-22T17:23:08.129188+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:09.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[18070]: audit 2022-04-22T17:23:08.138054+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:10.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.814Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.814Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.816Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.816Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.850Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.850Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:23:10.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:09 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:09.855Z caller=main.go:497 msg=Listening address=:9093 2022-04-22T17:23:10.152 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'rbd' created 2022-04-22T17:23:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:10 smithi121 bash[12679]: audit 2022-04-22T17:23:08.129612+0000 mgr.y (mgr.14146) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:10 smithi121 bash[12679]: cephadm 2022-04-22T17:23:08.147382+0000 mgr.y (mgr.14146) 271 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:23:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:10 smithi121 bash[12679]: cluster 2022-04-22T17:23:09.104780+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v242: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:10 smithi121 bash[12679]: audit 2022-04-22T17:23:09.544506+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.115:0/1109257170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:10 smithi121 bash[12679]: audit 2022-04-22T17:23:09.545008+0000 mon.a (mon.0) 658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:10.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[12737]: audit 2022-04-22T17:23:08.129612+0000 mgr.y (mgr.14146) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:10.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[12737]: cephadm 2022-04-22T17:23:08.147382+0000 mgr.y (mgr.14146) 271 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:23:10.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[12737]: cluster 2022-04-22T17:23:09.104780+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v242: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:10.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[12737]: audit 2022-04-22T17:23:09.544506+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.115:0/1109257170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:10.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[12737]: audit 2022-04-22T17:23:09.545008+0000 mon.a (mon.0) 658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:10.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[18070]: audit 2022-04-22T17:23:08.129612+0000 mgr.y (mgr.14146) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-04-22T17:23:10.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[18070]: cephadm 2022-04-22T17:23:08.147382+0000 mgr.y (mgr.14146) 271 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:23:10.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[18070]: cluster 2022-04-22T17:23:09.104780+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v242: 97 pgs: 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:10.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[18070]: audit 2022-04-22T17:23:09.544506+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.115:0/1109257170' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:10.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:10 smithi115 bash[18070]: audit 2022-04-22T17:23:09.545008+0000 mon.a (mon.0) 658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2022-04-22T17:23:11.000 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-04-22T17:23:11.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[12737]: audit 2022-04-22T17:23:10.146872+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2022-04-22T17:23:11.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[12737]: cluster 2022-04-22T17:23:10.146978+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2022-04-22T17:23:11.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[12737]: cluster 2022-04-22T17:23:11.105458+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v244: 105 pgs: 8 unknown, 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:11.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[18070]: audit 2022-04-22T17:23:10.146872+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2022-04-22T17:23:11.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[18070]: cluster 2022-04-22T17:23:10.146978+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2022-04-22T17:23:11.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[18070]: cluster 2022-04-22T17:23:11.105458+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v244: 105 pgs: 8 unknown, 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:11.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:11 smithi121 bash[12679]: audit 2022-04-22T17:23:10.146872+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2022-04-22T17:23:11.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:11 smithi121 bash[12679]: cluster 2022-04-22T17:23:10.146978+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2022-04-22T17:23:11.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:11 smithi121 bash[12679]: cluster 2022-04-22T17:23:11.105458+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v244: 105 pgs: 8 unknown, 97 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:12.158 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:11 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:11.817Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000098897s 2022-04-22T17:23:12.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:12 smithi115 bash[18070]: cluster 2022-04-22T17:23:11.148639+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2022-04-22T17:23:12.497 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:12 smithi115 bash[12737]: cluster 2022-04-22T17:23:11.148639+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2022-04-22T17:23:12.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:12 smithi121 bash[12679]: cluster 2022-04-22T17:23:11.148639+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2022-04-22T17:23:13.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:13 smithi121 bash[12679]: cluster 2022-04-22T17:23:12.167852+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2022-04-22T17:23:13.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:13 smithi121 bash[12679]: audit 2022-04-22T17:23:12.258847+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:13.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:13 smithi121 bash[12679]: cluster 2022-04-22T17:23:13.106139+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v247: 105 pgs: 6 unknown, 99 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:13.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[18070]: cluster 2022-04-22T17:23:12.167852+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2022-04-22T17:23:13.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[18070]: audit 2022-04-22T17:23:12.258847+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:13.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[18070]: cluster 2022-04-22T17:23:13.106139+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v247: 105 pgs: 6 unknown, 99 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:13.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[12737]: cluster 2022-04-22T17:23:12.167852+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2022-04-22T17:23:13.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[12737]: audit 2022-04-22T17:23:12.258847+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:13 smithi115 bash[12737]: cluster 2022-04-22T17:23:13.106139+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v247: 105 pgs: 6 unknown, 99 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:14.912 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rbd' on pool 'rbd' 2022-04-22T17:23:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:14 smithi121 bash[12679]: audit 2022-04-22T17:23:14.847943+0000 mon.a (mon.0) 664 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:23:15.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:14 smithi115 bash[12737]: audit 2022-04-22T17:23:14.847943+0000 mon.a (mon.0) 664 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:23:15.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:14 smithi115 bash[18070]: audit 2022-04-22T17:23:14.847943+0000 mon.a (mon.0) 664 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:23:15.792 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-04-22T17:23:15.793 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-04-22T17:23:15.793 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph mgr dump --format=json 2022-04-22T17:23:16.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:15 smithi121 bash[12679]: audit 2022-04-22T17:23:14.907392+0000 mon.a (mon.0) 665 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:23:16.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:15 smithi121 bash[12679]: cluster 2022-04-22T17:23:14.907696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2022-04-22T17:23:16.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:15 smithi121 bash[12679]: cluster 2022-04-22T17:23:15.106648+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v249: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:16.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[12737]: audit 2022-04-22T17:23:14.907392+0000 mon.a (mon.0) 665 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:23:16.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[12737]: cluster 2022-04-22T17:23:14.907696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2022-04-22T17:23:16.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[12737]: cluster 2022-04-22T17:23:15.106648+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v249: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:16.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[18070]: audit 2022-04-22T17:23:14.907392+0000 mon.a (mon.0) 665 : audit [INF] from='client.? 172.21.15.115:0/2915647092' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:23:16.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[18070]: cluster 2022-04-22T17:23:14.907696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2022-04-22T17:23:16.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:15 smithi115 bash[18070]: cluster 2022-04-22T17:23:15.106648+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v249: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:17.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:17 smithi121 bash[12679]: cluster 2022-04-22T17:23:17.107579+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v250: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:17.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:17 smithi115 bash[12737]: cluster 2022-04-22T17:23:17.107579+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v250: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:17.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:17 smithi115 bash[18070]: cluster 2022-04-22T17:23:17.107579+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v250: 105 pgs: 105 active+clean; 2.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:18.585 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:19.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:19 smithi121 bash[12679]: cluster 2022-04-22T17:23:19.108690+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:19.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:19 smithi115 bash[12737]: cluster 2022-04-22T17:23:19.108690+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:19.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:19 smithi115 bash[18070]: cluster 2022-04-22T17:23:19.108690+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v251: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:20.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:19 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:19.817Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000994366s 2022-04-22T17:23:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:23:22.466 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:22 smithi115 bash[12737]: cluster 2022-04-22T17:23:21.109381+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v252: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:22.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:22 smithi115 bash[18070]: cluster 2022-04-22T17:23:21.109381+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v252: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:22.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:22 smithi121 bash[12679]: cluster 2022-04-22T17:23:21.109381+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v252: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:23.064 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":14,"active_gid":14146,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":567188723},{"type":"v1","addr":"172.21.15.115:6801","nonce":567188723}]},"active_addr":"172.21.15.115:6801/567188723","active_change":"2022-04-22T17:16:36.270461+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.115:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1876895505}]},{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2286932496}]}]}} 2022-04-22T17:23:23.086 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-04-22T17:23:23.086 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-04-22T17:23:23.086 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd dump --format=json 2022-04-22T17:23:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:23 smithi121 bash[12679]: audit 2022-04-22T17:23:22.247344+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? 172.21.15.115:0/3144318725' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-04-22T17:23:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:23 smithi115 bash[12737]: audit 2022-04-22T17:23:22.247344+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? 172.21.15.115:0/3144318725' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-04-22T17:23:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:23 smithi115 bash[18070]: audit 2022-04-22T17:23:22.247344+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? 172.21.15.115:0/3144318725' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-04-22T17:23:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:24 smithi121 bash[12679]: cluster 2022-04-22T17:23:23.110284+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v253: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:24.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:24 smithi115 bash[12737]: cluster 2022-04-22T17:23:23.110284+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v253: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:24.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:24 smithi115 bash[18070]: cluster 2022-04-22T17:23:23.110284+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v253: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:25.136 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:25 smithi121 systemd[1]: Started Ceph grafana.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:25.871 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:26.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[12737]: audit 2022-04-22T17:23:25.092747+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:26.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[12737]: audit 2022-04-22T17:23:25.094763+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:26.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[12737]: audit 2022-04-22T17:23:25.096639+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[12737]: audit 2022-04-22T17:23:25.097921+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[12737]: cluster 2022-04-22T17:23:25.111302+0000 mgr.y (mgr.14146) 280 : cluster [DBG] pgmap v254: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[18070]: audit 2022-04-22T17:23:25.092747+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[18070]: audit 2022-04-22T17:23:25.094763+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[18070]: audit 2022-04-22T17:23:25.096639+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[18070]: audit 2022-04-22T17:23:25.097921+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:26.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:26 smithi115 bash[18070]: cluster 2022-04-22T17:23:25.111302+0000 mgr.y (mgr.14146) 280 : cluster [DBG] pgmap v254: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[12679]: audit 2022-04-22T17:23:25.092747+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:26.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[12679]: audit 2022-04-22T17:23:25.094763+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:26.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[12679]: audit 2022-04-22T17:23:25.096639+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:26.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[12679]: audit 2022-04-22T17:23:25.097921+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:26.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[12679]: cluster 2022-04-22T17:23:25.111302+0000 mgr.y (mgr.14146) 280 : cluster [DBG] pgmap v254: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:26.377 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2022-04-22T17:23:26.378 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2022-04-22T17:23:26.379 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2022-04-22T17:23:26.379 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2022-04-22T17:23:26.379 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2022-04-22T17:23:26.379 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="App mode production" logger=settings 2022-04-22T17:23:26.379 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Initializing SqlStore" logger=server 2022-04-22T17:23:26.380 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2022-04-22T17:23:26.380 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Starting DB migration" logger=migrator 2022-04-22T17:23:26.380 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2022-04-22T17:23:26.380 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2022-04-22T17:23:26.629 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2022-04-22T17:23:26.890 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2022-04-22T17:23:26.890 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2022-04-22T17:23:26.890 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2022-04-22T17:23:26.890 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2022-04-22T17:23:26.890 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2022-04-22T17:23:26.891 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2022-04-22T17:23:26.891 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2022-04-22T17:23:26.891 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:26 smithi121 bash[28066]: t=2022-04-22T17:23:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2022-04-22T17:23:27.177 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2022-04-22T17:23:27.178 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2022-04-22T17:23:27.178 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2022-04-22T17:23:27.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:27 smithi115 bash[18070]: cluster 2022-04-22T17:23:27.111996+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v255: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:27.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:27 smithi115 bash[12737]: cluster 2022-04-22T17:23:27.111996+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v255: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:27.438 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[12679]: cluster 2022-04-22T17:23:27.111996+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v255: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:27.438 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2022-04-22T17:23:27.438 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2022-04-22T17:23:27.438 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2022-04-22T17:23:27.439 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2022-04-22T17:23:27.439 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2022-04-22T17:23:27.439 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2022-04-22T17:23:27.699 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2022-04-22T17:23:27.699 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2022-04-22T17:23:27.699 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2022-04-22T17:23:27.699 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2022-04-22T17:23:27.700 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2022-04-22T17:23:27.951 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2022-04-22T17:23:27.952 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2022-04-22T17:23:27.952 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2022-04-22T17:23:27.952 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2022-04-22T17:23:27.953 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2022-04-22T17:23:27.953 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2022-04-22T17:23:27.953 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2022-04-22T17:23:28.230 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2022-04-22T17:23:28.230 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2022-04-22T17:23:28.230 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:27 smithi121 bash[28066]: t=2022-04-22T17:23:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2022-04-22T17:23:28.230 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2022-04-22T17:23:28.230 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2022-04-22T17:23:28.231 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2022-04-22T17:23:28.231 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2022-04-22T17:23:28.231 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2022-04-22T17:23:28.231 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2022-04-22T17:23:28.490 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[12679]: audit 2022-04-22T17:23:27.264882+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.491 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[12679]: audit 2022-04-22T17:23:28.133475+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2022-04-22T17:23:28.491 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2022-04-22T17:23:28.492 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2022-04-22T17:23:28.492 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2022-04-22T17:23:28.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:28 smithi115 bash[12737]: audit 2022-04-22T17:23:27.264882+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:28 smithi115 bash[12737]: audit 2022-04-22T17:23:28.133475+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:28 smithi115 bash[18070]: audit 2022-04-22T17:23:27.264882+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:28 smithi115 bash[18070]: audit 2022-04-22T17:23:28.133475+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2022-04-22T17:23:28.743 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2022-04-22T17:23:29.003 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:28 smithi121 bash[28066]: t=2022-04-22T17:23:28+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2022-04-22T17:23:29.268 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2022-04-22T17:23:29.531 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:23:29.531 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":60,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:23:14.898360+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"rbd","create_time":"2022-04-22T17:23:10.004854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.3b","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:23:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:29 smithi115 bash[12737]: cluster 2022-04-22T17:23:29.113083+0000 mgr.y (mgr.14146) 282 : cluster [DBG] pgmap v256: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:29.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:29 smithi115 bash[18070]: cluster 2022-04-22T17:23:29.113083+0000 mgr.y (mgr.14146) 282 : cluster [DBG] pgmap v256: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:29.541 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[12679]: cluster 2022-04-22T17:23:29.113083+0000 mgr.y (mgr.14146) 282 : cluster [DBG] pgmap v256: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2022-04-22T17:23:29.541 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2022-04-22T17:23:29.819 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2022-04-22T17:23:30.088 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2022-04-22T17:23:30.088 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2022-04-22T17:23:30.089 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2022-04-22T17:23:30.089 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:29 smithi121 bash[28066]: t=2022-04-22T17:23:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2022-04-22T17:23:30.089 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2022-04-22T17:23:30.089 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2022-04-22T17:23:30.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[12679]: audit 2022-04-22T17:23:29.527980+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.115:0/277511845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:30.357 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2022-04-22T17:23:30.357 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2022-04-22T17:23:30.357 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2022-04-22T17:23:30.358 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2022-04-22T17:23:30.358 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2022-04-22T17:23:30.371 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-04-22T17:23:30.371 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd dump --format=json 2022-04-22T17:23:30.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:30 smithi115 bash[18070]: audit 2022-04-22T17:23:29.527980+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.115:0/277511845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:30.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:30 smithi115 bash[12737]: audit 2022-04-22T17:23:29.527980+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.115:0/277511845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:30.617 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2022-04-22T17:23:30.618 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2022-04-22T17:23:30.619 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2022-04-22T17:23:30.895 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2022-04-22T17:23:30.933 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:30 smithi121 bash[28066]: t=2022-04-22T17:23:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2022-04-22T17:23:31.157 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2022-04-22T17:23:31.409 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2022-04-22T17:23:31.409 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2022-04-22T17:23:31.409 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2022-04-22T17:23:31.409 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2022-04-22T17:23:31.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[12679]: audit 2022-04-22T17:23:30.335162+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[12679]: audit 2022-04-22T17:23:30.612618+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[12679]: cephadm 2022-04-22T17:23:30.617994+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2022-04-22T17:23:31.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[12679]: cephadm 2022-04-22T17:23:30.622181+0000 mgr.y (mgr.14146) 284 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi115 2022-04-22T17:23:31.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[12679]: cluster 2022-04-22T17:23:31.113902+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v257: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:31.670 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2022-04-22T17:23:31.766 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2022-04-22T17:23:31.766 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2022-04-22T17:23:31.766 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2022-04-22T17:23:31.766 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2022-04-22T17:23:31.767 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2022-04-22T17:23:31.767 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2022-04-22T17:23:31.767 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2022-04-22T17:23:31.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[12737]: audit 2022-04-22T17:23:30.335162+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[12737]: audit 2022-04-22T17:23:30.612618+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[12737]: cephadm 2022-04-22T17:23:30.617994+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2022-04-22T17:23:31.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[12737]: cephadm 2022-04-22T17:23:30.622181+0000 mgr.y (mgr.14146) 284 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi115 2022-04-22T17:23:31.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[12737]: cluster 2022-04-22T17:23:31.113902+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v257: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[18070]: audit 2022-04-22T17:23:30.335162+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[18070]: audit 2022-04-22T17:23:30.612618+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[18070]: cephadm 2022-04-22T17:23:30.617994+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2022-04-22T17:23:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[18070]: cephadm 2022-04-22T17:23:30.622181+0000 mgr.y (mgr.14146) 284 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi115 2022-04-22T17:23:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:31 smithi115 bash[18070]: cluster 2022-04-22T17:23:31.113902+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v257: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2022-04-22T17:23:31.923 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2022-04-22T17:23:31.924 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:31 smithi121 bash[28066]: t=2022-04-22T17:23:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2022-04-22T17:23:32.183 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2022-04-22T17:23:32.184 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2022-04-22T17:23:32.452 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2022-04-22T17:23:32.712 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2022-04-22T17:23:32.712 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2022-04-22T17:23:32.713 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2022-04-22T17:23:32.973 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2022-04-22T17:23:32.974 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2022-04-22T17:23:32.974 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:32 smithi121 bash[28066]: t=2022-04-22T17:23:32+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2022-04-22T17:23:33.242 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[12679]: cluster 2022-04-22T17:23:33.114805+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v258: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2022-04-22T17:23:33.243 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2022-04-22T17:23:33.244 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2022-04-22T17:23:33.244 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2022-04-22T17:23:33.351 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:33.503 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2022-04-22T17:23:33.504 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2022-04-22T17:23:33.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:33 smithi115 bash[12737]: cluster 2022-04-22T17:23:33.114805+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v258: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:33.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:33 smithi115 bash[18070]: cluster 2022-04-22T17:23:33.114805+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v258: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:33.756 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2022-04-22T17:23:33.799 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2022-04-22T17:23:34.018 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2022-04-22T17:23:34.018 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2022-04-22T17:23:34.018 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2022-04-22T17:23:34.018 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2022-04-22T17:23:34.019 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2022-04-22T17:23:34.019 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2022-04-22T17:23:34.019 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2022-04-22T17:23:34.019 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:33 smithi121 bash[28066]: t=2022-04-22T17:23:33+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2022-04-22T17:23:34.322 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing HTTPServer" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing PluginManager" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Starting plugin search" logger=plugins 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing HooksService" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2022-04-22T17:23:34.323 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing RemoteCache" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing RenderingService" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing AlertEngine" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing QuotaService" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing ServerLockService" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2022-04-22T17:23:34.324 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing LoginService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing SearchService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing TracingService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing CleanUpService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing NotificationService" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2022-04-22T17:23:34.325 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2022-04-22T17:23:34.768 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2022-04-22T17:23:34.768 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="Initializing Stream Manager" 2022-04-22T17:23:34.768 INFO:journalctl@ceph.grafana.a.smithi121.stdout:Apr 22 17:23:34 smithi121 bash[28066]: t=2022-04-22T17:23:34+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2022-04-22T17:23:35.185 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:35 smithi115 bash[12737]: cluster 2022-04-22T17:23:35.115725+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v259: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:35.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:35 smithi121 bash[12679]: cluster 2022-04-22T17:23:35.115725+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v259: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:35.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:35 smithi115 bash[18070]: cluster 2022-04-22T17:23:35.115725+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v259: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:36.538 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:36 smithi115 systemd[1]: Stopping Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:23:36.539 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:36 smithi115 bash[33044]: level=info ts=2022-04-22T17:23:36.283Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2022-04-22T17:23:37.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:37 smithi115 bash[12737]: cluster 2022-04-22T17:23:37.116521+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v260: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:37.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:37 smithi115 bash[18070]: cluster 2022-04-22T17:23:37.116521+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v260: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:37.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:37 smithi121 bash[12679]: cluster 2022-04-22T17:23:37.116521+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v260: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:38.022 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:37 smithi115 docker[34498]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-alertmanager.a 2022-04-22T17:23:38.023 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:37 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@alertmanager.a.service: Succeeded. 2022-04-22T17:23:38.023 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:37 smithi115 systemd[1]: Stopped Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:38.023 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:37 smithi115 systemd[1]: Started Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:38.385 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:23:38.385 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":60,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:23:14.898360+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"rbd","create_time":"2022-04-22T17:23:10.004854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.3b","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:23:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: audit 2022-04-22T17:23:37.836211+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: cephadm 2022-04-22T17:23:37.837679+0000 mgr.y (mgr.14146) 289 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:23:39.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: cephadm 2022-04-22T17:23:37.840542+0000 mgr.y (mgr.14146) 290 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:23:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: audit 2022-04-22T17:23:38.382896+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.115:0/1012278776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: audit 2022-04-22T17:23:38.641764+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:23:39.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:38 smithi121 bash[12679]: audit 2022-04-22T17:23:38.645309+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: audit 2022-04-22T17:23:37.836211+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: cephadm 2022-04-22T17:23:37.837679+0000 mgr.y (mgr.14146) 289 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: cephadm 2022-04-22T17:23:37.840542+0000 mgr.y (mgr.14146) 290 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: audit 2022-04-22T17:23:38.382896+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.115:0/1012278776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: audit 2022-04-22T17:23:38.641764+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:23:39.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[12737]: audit 2022-04-22T17:23:38.645309+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: audit 2022-04-22T17:23:37.836211+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: cephadm 2022-04-22T17:23:37.837679+0000 mgr.y (mgr.14146) 289 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: cephadm 2022-04-22T17:23:37.840542+0000 mgr.y (mgr.14146) 290 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: audit 2022-04-22T17:23:38.382896+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.115:0/1012278776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: audit 2022-04-22T17:23:38.641764+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:23:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:38 smithi115 bash[18070]: audit 2022-04-22T17:23:38.645309+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:23:39.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-04-22T17:23:39.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-04-22T17:23:39.690 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-04-22T17:23:39.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-04-22T17:23:39.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-04-22T17:23:39.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-04-22T17:23:39.693 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-04-22T17:23:39.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-04-22T17:23:39.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:39 smithi115 bash[18070]: cluster 2022-04-22T17:23:39.117602+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v261: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:40.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:39 smithi115 bash[12737]: cluster 2022-04-22T17:23:39.117602+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v261: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:40.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:39 smithi121 bash[12679]: cluster 2022-04-22T17:23:39.117602+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v261: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:40.288 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.057Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.057Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.061Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.063Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.091Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.091Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:23:40.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:40 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:40.094Z caller=main.go:497 msg=Listening address=:9093 2022-04-22T17:23:42.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[12679]: cluster 2022-04-22T17:23:41.118459+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v262: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:42.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:42 smithi115 bash[12737]: cluster 2022-04-22T17:23:41.118459+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v262: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:42.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:42 smithi115 bash[18070]: cluster 2022-04-22T17:23:41.118459+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v262: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:42.539 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:42 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:42.064Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000376113s 2022-04-22T17:23:42.626 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.627 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.632 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.632 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.633 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.633 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.633 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.634 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:23:42.904 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 systemd[1]: Stopping Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=warn ts=2022-04-22T17:23:42.746Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.746Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:561 msg="Stopping notify discovery manager..." 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:583 msg="Stopping scrape manager..." 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:543 msg="Scrape discovery manager stopped" 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:557 msg="Notify discovery manager stopped" 2022-04-22T17:23:42.912 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:577 msg="Scrape manager stopped" 2022-04-22T17:23:42.913 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2022-04-22T17:23:42.913 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2022-04-22T17:23:42.913 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2022-04-22T17:23:42.913 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.747Z caller=main.go:749 msg="Notifier manager stopped" 2022-04-22T17:23:42.913 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:42 smithi121 bash[26962]: level=info ts=2022-04-22T17:23:42.748Z caller=main.go:761 msg="See you next time!" 2022-04-22T17:23:43.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:43 smithi121 bash[12679]: cluster 2022-04-22T17:23:43.119081+0000 mgr.y (mgr.14146) 293 : cluster [DBG] pgmap v263: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:43.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:43 smithi115 bash[12737]: cluster 2022-04-22T17:23:43.119081+0000 mgr.y (mgr.14146) 293 : cluster [DBG] pgmap v263: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:43.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:43 smithi115 bash[18070]: cluster 2022-04-22T17:23:43.119081+0000 mgr.y (mgr.14146) 293 : cluster [DBG] pgmap v263: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:44.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:43 smithi121 docker[28733]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-prometheus.a 2022-04-22T17:23:44.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:43 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@prometheus.a.service: Succeeded. 2022-04-22T17:23:44.019 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:43 smithi121 systemd[1]: Stopped Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:44.019 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:43 smithi121 systemd[1]: Started Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:23:45.019 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2022-04-22T17:23:45.058 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2022-04-22T17:23:45.058 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2022-04-22T17:23:45.058 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:339 host_details="(Linux 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 smithi121 (none))" 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.811Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.813Z caller=main.go:678 msg="Starting TSDB ..." 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.813Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.816Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.817Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2022-04-22T17:23:45.059 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.817Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.818Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.668622ms 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.819Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.819Z caller=main.go:695 msg="TSDB started" 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.819Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.827Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:23:45.060 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[28790]: level=info ts=2022-04-22T17:23:44.827Z caller=main.go:646 msg="Server is ready to receive web requests." 2022-04-22T17:23:45.060 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.591928+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.597048+0000 mon.a (mon.0) 682 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.597825+0000 mgr.y (mgr.14146) 294 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.599348+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.599779+0000 mgr.y (mgr.14146) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.607804+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.613377+0000 mon.a (mon.0) 685 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.613892+0000 mgr.y (mgr.14146) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.620759+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.621163+0000 mgr.y (mgr.14146) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.628047+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.635643+0000 mon.a (mon.0) 688 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.636100+0000 mgr.y (mgr.14146) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.642600+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.642984+0000 mgr.y (mgr.14146) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.649913+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.653179+0000 mon.a (mon.0) 691 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.654691+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:45.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:44 smithi121 bash[12679]: audit 2022-04-22T17:23:43.655821+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:45.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.591928+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.597048+0000 mon.a (mon.0) 682 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.597825+0000 mgr.y (mgr.14146) 294 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.599348+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.599779+0000 mgr.y (mgr.14146) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.607804+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.613377+0000 mon.a (mon.0) 685 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.613892+0000 mgr.y (mgr.14146) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.620759+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.621163+0000 mgr.y (mgr.14146) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.628047+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.635643+0000 mon.a (mon.0) 688 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.636100+0000 mgr.y (mgr.14146) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.642600+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.642984+0000 mgr.y (mgr.14146) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.649913+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.653179+0000 mon.a (mon.0) 691 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.654691+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[18070]: audit 2022-04-22T17:23:43.655821+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.591928+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.597048+0000 mon.a (mon.0) 682 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.597825+0000 mgr.y (mgr.14146) 294 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:23:45.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.599348+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.599779+0000 mgr.y (mgr.14146) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.607804+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.613377+0000 mon.a (mon.0) 685 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.613892+0000 mgr.y (mgr.14146) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.620759+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.621163+0000 mgr.y (mgr.14146) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.121:9095"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.628047+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.635643+0000 mon.a (mon.0) 688 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.636100+0000 mgr.y (mgr.14146) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.642600+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.642984+0000 mgr.y (mgr.14146) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.121:3000"}]: dispatch 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.649913+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.653179+0000 mon.a (mon.0) 691 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.654691+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:45.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:44 smithi115 bash[12737]: audit 2022-04-22T17:23:43.655821+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:45.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:45 smithi115 bash[12737]: cluster 2022-04-22T17:23:45.119924+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v264: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:45.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:45 smithi115 bash[18070]: cluster 2022-04-22T17:23:45.119924+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v264: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:46.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:45 smithi121 bash[12679]: cluster 2022-04-22T17:23:45.119924+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v264: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:47 smithi115 bash[12737]: cluster 2022-04-22T17:23:47.120397+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v265: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:47 smithi115 bash[18070]: cluster 2022-04-22T17:23:47.120397+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v265: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:47 smithi121 bash[12679]: cluster 2022-04-22T17:23:47.120397+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v265: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:49 smithi121 bash[12679]: audit 2022-04-22T17:23:48.013132+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:49.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:49 smithi115 bash[12737]: audit 2022-04-22T17:23:48.013132+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:49.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:49 smithi115 bash[18070]: audit 2022-04-22T17:23:48.013132+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:50.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:50 smithi115 bash[18070]: cluster 2022-04-22T17:23:49.121206+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v266: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:50.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:23:50 smithi115 bash[34566]: level=info ts=2022-04-22T17:23:50.065Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001205156s 2022-04-22T17:23:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:50 smithi115 bash[12737]: cluster 2022-04-22T17:23:49.121206+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v266: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:50.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:50 smithi121 bash[12679]: cluster 2022-04-22T17:23:49.121206+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v266: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:52.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:52 smithi115 bash[12737]: cluster 2022-04-22T17:23:51.121757+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v267: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:52.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:52 smithi115 bash[18070]: cluster 2022-04-22T17:23:51.121757+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v267: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:52.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:52 smithi121 bash[12679]: cluster 2022-04-22T17:23:51.121757+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v267: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:53 smithi121 bash[12679]: cluster 2022-04-22T17:23:53.122358+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v268: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:53 smithi115 bash[12737]: cluster 2022-04-22T17:23:53.122358+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v268: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:53 smithi115 bash[18070]: cluster 2022-04-22T17:23:53.122358+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v268: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:55.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:55 smithi121 bash[12679]: cluster 2022-04-22T17:23:55.123558+0000 mgr.y (mgr.14146) 305 : cluster [DBG] pgmap v269: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:55.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:55 smithi115 bash[12737]: cluster 2022-04-22T17:23:55.123558+0000 mgr.y (mgr.14146) 305 : cluster [DBG] pgmap v269: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:55.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:55 smithi115 bash[18070]: cluster 2022-04-22T17:23:55.123558+0000 mgr.y (mgr.14146) 305 : cluster [DBG] pgmap v269: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:57.390 INFO:teuthology.orchestra.run.smithi115.stdout:154618822695 2022-04-22T17:23:57.407 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-04-22T17:23:58.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: audit 2022-04-22T17:23:56.776299+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: audit 2022-04-22T17:23:56.780363+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: audit 2022-04-22T17:23:56.784463+0000 mon.a (mon.0) 697 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: audit 2022-04-22T17:23:56.785669+0000 mon.a (mon.0) 698 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: audit 2022-04-22T17:23:56.786618+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[12737]: cluster 2022-04-22T17:23:57.124576+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v270: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: audit 2022-04-22T17:23:56.776299+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: audit 2022-04-22T17:23:56.780363+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: audit 2022-04-22T17:23:56.784463+0000 mon.a (mon.0) 697 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:58.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: audit 2022-04-22T17:23:56.785669+0000 mon.a (mon.0) 698 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:58.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: audit 2022-04-22T17:23:56.786618+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:58.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:57 smithi115 bash[18070]: cluster 2022-04-22T17:23:57.124576+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v270: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:58.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: audit 2022-04-22T17:23:56.776299+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: audit 2022-04-22T17:23:56.780363+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:23:58.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: audit 2022-04-22T17:23:56.784463+0000 mon.a (mon.0) 697 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:23:58.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: audit 2022-04-22T17:23:56.785669+0000 mon.a (mon.0) 698 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:23:58.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: audit 2022-04-22T17:23:56.786618+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:23:58.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:57 smithi121 bash[12679]: cluster 2022-04-22T17:23:57.124576+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v270: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:59.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:23:59 smithi115 bash[18070]: cluster 2022-04-22T17:23:59.125517+0000 mgr.y (mgr.14146) 307 : cluster [DBG] pgmap v271: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:59.369 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:23:59 smithi115 bash[12737]: cluster 2022-04-22T17:23:59.125517+0000 mgr.y (mgr.14146) 307 : cluster [DBG] pgmap v271: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:23:59.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:23:59 smithi121 bash[12679]: cluster 2022-04-22T17:23:59.125517+0000 mgr.y (mgr.14146) 307 : cluster [DBG] pgmap v271: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:00.904 INFO:teuthology.orchestra.run.smithi115.stdout:107374182451 2022-04-22T17:24:00.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-04-22T17:24:01.310 INFO:teuthology.orchestra.run.smithi115.stdout:128849018926 2022-04-22T17:24:01.311 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-04-22T17:24:01.535 INFO:teuthology.orchestra.run.smithi115.stdout:34359738437 2022-04-22T17:24:01.536 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-04-22T17:24:01.653 INFO:teuthology.orchestra.run.smithi115.stdout:77309411385 2022-04-22T17:24:01.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-04-22T17:24:02.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:02 smithi121 bash[12679]: audit 2022-04-22T17:24:01.117731+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:02.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:02 smithi121 bash[12679]: cluster 2022-04-22T17:24:01.126093+0000 mgr.y (mgr.14146) 308 : cluster [DBG] pgmap v272: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:02.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:02 smithi115 bash[18070]: audit 2022-04-22T17:24:01.117731+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:02.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:02 smithi115 bash[18070]: cluster 2022-04-22T17:24:01.126093+0000 mgr.y (mgr.14146) 308 : cluster [DBG] pgmap v272: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:02 smithi115 bash[12737]: audit 2022-04-22T17:24:01.117731+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:02 smithi115 bash[12737]: cluster 2022-04-22T17:24:01.126093+0000 mgr.y (mgr.14146) 308 : cluster [DBG] pgmap v272: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:02.578 INFO:teuthology.orchestra.run.smithi115.stdout:180388626467 2022-04-22T17:24:02.579 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-04-22T17:24:03.070 INFO:teuthology.orchestra.run.smithi115.stdout:55834574910 2022-04-22T17:24:03.071 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-04-22T17:24:03.167 INFO:teuthology.orchestra.run.smithi115.stdout:206158430238 2022-04-22T17:24:03.168 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-04-22T17:24:03.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:03 smithi121 bash[12679]: cluster 2022-04-22T17:24:03.126591+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v273: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:03.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:03 smithi115 bash[12737]: cluster 2022-04-22T17:24:03.126591+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v273: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:03.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:03 smithi115 bash[18070]: cluster 2022-04-22T17:24:03.126591+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v273: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:03.642 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:03.714 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:03.716 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:03.716 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:03.718 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:04.740 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:05.761 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:05.763 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:06.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:05 smithi121 bash[12679]: audit 2022-04-22T17:24:04.958456+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:05 smithi121 bash[12679]: audit 2022-04-22T17:24:04.966215+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:05 smithi121 bash[12679]: cluster 2022-04-22T17:24:05.127680+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v274: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:06.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[12737]: audit 2022-04-22T17:24:04.958456+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.412 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[12737]: audit 2022-04-22T17:24:04.966215+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.413 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[12737]: cluster 2022-04-22T17:24:05.127680+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v274: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:06.414 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[18070]: audit 2022-04-22T17:24:04.958456+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.414 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[18070]: audit 2022-04-22T17:24:04.966215+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:06.414 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:05 smithi115 bash[18070]: cluster 2022-04-22T17:24:05.127680+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v274: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:07.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:07 smithi121 bash[12679]: cluster 2022-04-22T17:24:07.128578+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v275: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:07.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:07 smithi115 bash[12737]: cluster 2022-04-22T17:24:07.128578+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v275: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:07.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:07 smithi115 bash[18070]: cluster 2022-04-22T17:24:07.128578+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v275: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:09.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:09 smithi115 bash[18070]: cluster 2022-04-22T17:24:09.129348+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v276: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:09.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:09 smithi121 bash[12679]: cluster 2022-04-22T17:24:09.129348+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v276: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:09.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:09 smithi115 bash[12737]: cluster 2022-04-22T17:24:09.129348+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v276: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:12.418 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:12 smithi115 bash[12737]: cluster 2022-04-22T17:24:11.130243+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v277: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:12.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:12 smithi115 bash[18070]: cluster 2022-04-22T17:24:11.130243+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v277: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:12.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:12 smithi121 bash[12679]: cluster 2022-04-22T17:24:11.130243+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v277: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:13.490 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:13 smithi115 bash[12737]: cluster 2022-04-22T17:24:13.131073+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v278: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:13.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:13 smithi115 bash[18070]: cluster 2022-04-22T17:24:13.131073+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v278: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:13.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:13 smithi121 bash[12679]: cluster 2022-04-22T17:24:13.131073+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v278: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:15.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:15 smithi121 bash[12679]: cluster 2022-04-22T17:24:15.132183+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v279: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:15.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:15 smithi115 bash[18070]: cluster 2022-04-22T17:24:15.132183+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v279: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:15.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:15 smithi115 bash[12737]: cluster 2022-04-22T17:24:15.132183+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v279: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:17.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:17 smithi115 bash[12737]: cluster 2022-04-22T17:24:17.132694+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v280: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:17.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:17 smithi115 bash[18070]: cluster 2022-04-22T17:24:17.132694+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v280: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:17.400 INFO:teuthology.orchestra.run.smithi115.stdout:154618822699 2022-04-22T17:24:17.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:17 smithi121 bash[12679]: cluster 2022-04-22T17:24:17.132694+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v280: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:17.949 INFO:teuthology.orchestra.run.smithi115.stdout:128849018929 2022-04-22T17:24:18.345 INFO:teuthology.orchestra.run.smithi115.stdout:55834574914 2022-04-22T17:24:18.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:18 smithi121 bash[12679]: audit 2022-04-22T17:24:17.398201+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3451902196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-04-22T17:24:18.565 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:18 smithi121 bash[12679]: audit 2022-04-22T17:24:17.947166+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/3051599667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-04-22T17:24:18.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:18 smithi115 bash[12737]: audit 2022-04-22T17:24:17.398201+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3451902196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-04-22T17:24:18.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:18 smithi115 bash[12737]: audit 2022-04-22T17:24:17.947166+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/3051599667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-04-22T17:24:18.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:18 smithi115 bash[18070]: audit 2022-04-22T17:24:17.398201+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3451902196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-04-22T17:24:18.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:18 smithi115 bash[18070]: audit 2022-04-22T17:24:17.947166+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/3051599667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-04-22T17:24:19.270 INFO:teuthology.orchestra.run.smithi115.stdout:107374182456 2022-04-22T17:24:19.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:19 smithi121 bash[12679]: audit 2022-04-22T17:24:18.342970+0000 mon.a (mon.0) 703 : audit [DBG] from='client.? 172.21.15.115:0/40636013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-04-22T17:24:19.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:19 smithi121 bash[12679]: cluster 2022-04-22T17:24:19.133629+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v281: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:19.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:19 smithi115 bash[18070]: audit 2022-04-22T17:24:18.342970+0000 mon.a (mon.0) 703 : audit [DBG] from='client.? 172.21.15.115:0/40636013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-04-22T17:24:19.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:19 smithi115 bash[18070]: cluster 2022-04-22T17:24:19.133629+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v281: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:19 smithi115 bash[12737]: audit 2022-04-22T17:24:18.342970+0000 mon.a (mon.0) 703 : audit [DBG] from='client.? 172.21.15.115:0/40636013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-04-22T17:24:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:19 smithi115 bash[12737]: cluster 2022-04-22T17:24:19.133629+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v281: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:20.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:20 smithi115 bash[12737]: audit 2022-04-22T17:24:19.268688+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? 172.21.15.115:0/3666210260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-04-22T17:24:20.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:20 smithi121 bash[12679]: audit 2022-04-22T17:24:19.268688+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? 172.21.15.115:0/3666210260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-04-22T17:24:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:20 smithi115 bash[18070]: audit 2022-04-22T17:24:19.268688+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? 172.21.15.115:0/3666210260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-04-22T17:24:20.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018926 got 128849018929 for osd.4 2022-04-22T17:24:20.740 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:21.522 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822695 got 154618822699 for osd.5 2022-04-22T17:24:21.522 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:21.660 INFO:teuthology.orchestra.run.smithi115.stdout:180388626471 2022-04-22T17:24:21.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574914 for osd.1 2022-04-22T17:24:21.736 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:21.740 INFO:teuthology.orchestra.run.smithi115.stdout:77309411389 2022-04-22T17:24:21.834 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182451 got 107374182456 for osd.3 2022-04-22T17:24:21.834 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:22.033 INFO:teuthology.orchestra.run.smithi115.stdout:206158430242 2022-04-22T17:24:22.046 INFO:teuthology.orchestra.run.smithi115.stdout:34359738441 2022-04-22T17:24:22.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[12737]: cluster 2022-04-22T17:24:21.134463+0000 mgr.y (mgr.14146) 318 : cluster [DBG] pgmap v282: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:22.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[12737]: audit 2022-04-22T17:24:21.658024+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/78399934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[12737]: audit 2022-04-22T17:24:21.738425+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.115:0/703303714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[12737]: audit 2022-04-22T17:24:22.030844+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.115:0/490191162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[12737]: audit 2022-04-22T17:24:22.044855+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.115:0/866855586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[18070]: cluster 2022-04-22T17:24:21.134463+0000 mgr.y (mgr.14146) 318 : cluster [DBG] pgmap v282: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[18070]: audit 2022-04-22T17:24:21.658024+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/78399934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[18070]: audit 2022-04-22T17:24:21.738425+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.115:0/703303714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[18070]: audit 2022-04-22T17:24:22.030844+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.115:0/490191162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-04-22T17:24:22.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:22 smithi115 bash[18070]: audit 2022-04-22T17:24:22.044855+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.115:0/866855586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-04-22T17:24:22.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:22 smithi121 bash[12679]: cluster 2022-04-22T17:24:21.134463+0000 mgr.y (mgr.14146) 318 : cluster [DBG] pgmap v282: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:22.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:22 smithi121 bash[12679]: audit 2022-04-22T17:24:21.658024+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/78399934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-04-22T17:24:22.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:22 smithi121 bash[12679]: audit 2022-04-22T17:24:21.738425+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.115:0/703303714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-04-22T17:24:22.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:22 smithi121 bash[12679]: audit 2022-04-22T17:24:22.030844+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.115:0/490191162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-04-22T17:24:22.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:22 smithi121 bash[12679]: audit 2022-04-22T17:24:22.044855+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.115:0/866855586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-04-22T17:24:23.473 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:23 smithi115 bash[18070]: cluster 2022-04-22T17:24:23.135304+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v283: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:23.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:23 smithi115 bash[12737]: cluster 2022-04-22T17:24:23.135304+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v283: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:23.508 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626467 got 180388626471 for osd.6 2022-04-22T17:24:23.508 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:23 smithi121 bash[12679]: cluster 2022-04-22T17:24:23.135304+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v283: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:23.658 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411385 got 77309411389 for osd.2 2022-04-22T17:24:23.658 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:23.950 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430238 got 206158430242 for osd.7 2022-04-22T17:24:23.951 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:24.289 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738441 for osd.0 2022-04-22T17:24:24.289 DEBUG:teuthology.parallel:result is None 2022-04-22T17:24:24.290 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-04-22T17:24:24.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph pg dump --format=json 2022-04-22T17:24:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:25 smithi121 bash[12679]: cluster 2022-04-22T17:24:25.136596+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v284: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:25.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:25 smithi115 bash[12737]: cluster 2022-04-22T17:24:25.136596+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v284: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:25.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:25 smithi115 bash[18070]: cluster 2022-04-22T17:24:25.136596+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v284: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:27.063 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:27.376 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:27 smithi115 bash[12737]: cluster 2022-04-22T17:24:27.137229+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v285: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:27.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:27 smithi121 bash[12679]: cluster 2022-04-22T17:24:27.137229+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v285: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:27.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:27 smithi115 bash[18070]: cluster 2022-04-22T17:24:27.137229+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v285: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:29 smithi121 bash[12679]: cluster 2022-04-22T17:24:29.138245+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v286: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:29 smithi115 bash[12737]: cluster 2022-04-22T17:24:29.138245+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v286: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:29.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:29 smithi115 bash[18070]: cluster 2022-04-22T17:24:29.138245+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v286: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:30.723 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:24:30.726 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:24:31.557 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2022-04-22T17:24:29.137518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":315,"acting":315,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":315,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":46772,"kb_used_data":4788,"kb_used_omap":0,"kb_used_meta":41920,"kb_avail":749914444,"statfs":{"total":767960285184,"available":767912390656,"internally_reserved":0,"allocated":4902912,"data_stored":2059866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42926080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005090"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227334+0000","last_change":"2022-04-22T17:22:18.455566+0000","last_active":"2022-04-22T17:23:12.227334+0000","last_peered":"2022-04-22T17:23:12.227334+0000","last_clean":"2022-04-22T17:23:12.227334+0000","last_became_active":"2022-04-22T17:22:18.455256+0000","last_became_peered":"2022-04-22T17:22:18.455256+0000","last_unstale":"2022-04-22T17:23:12.227334+0000","last_undegraded":"2022-04-22T17:23:12.227334+0000","last_fullsized":"2022-04-22T17:23:12.227334+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229728+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:23:12.229728+0000","last_peered":"2022-04-22T17:23:12.229728+0000","last_clean":"2022-04-22T17:23:12.229728+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:23:12.229728+0000","last_undegraded":"2022-04-22T17:23:12.229728+0000","last_fullsized":"2022-04-22T17:23:12.229728+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231582+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:23:12.231582+0000","last_peered":"2022-04-22T17:23:12.231582+0000","last_clean":"2022-04-22T17:23:12.231582+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:23:12.231582+0000","last_undegraded":"2022-04-22T17:23:12.231582+0000","last_fullsized":"2022-04-22T17:23:12.231582+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167968+0000","last_change":"2022-04-22T17:22:18.448571+0000","last_active":"2022-04-22T17:23:12.167968+0000","last_peered":"2022-04-22T17:23:12.167968+0000","last_clean":"2022-04-22T17:23:12.167968+0000","last_became_active":"2022-04-22T17:22:18.448283+0000","last_became_peered":"2022-04-22T17:22:18.448283+0000","last_unstale":"2022-04-22T17:23:12.167968+0000","last_undegraded":"2022-04-22T17:23:12.167968+0000","last_fullsized":"2022-04-22T17:23:12.167968+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227043+0000","last_change":"2022-04-22T17:22:38.862851+0000","last_active":"2022-04-22T17:23:12.227043+0000","last_peered":"2022-04-22T17:23:12.227043+0000","last_clean":"2022-04-22T17:23:12.227043+0000","last_became_active":"2022-04-22T17:22:38.862422+0000","last_became_peered":"2022-04-22T17:22:38.862422+0000","last_unstale":"2022-04-22T17:23:12.227043+0000","last_undegraded":"2022-04-22T17:23:12.227043+0000","last_fullsized":"2022-04-22T17:23:12.227043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167613+0000","last_change":"2022-04-22T17:22:18.443897+0000","last_active":"2022-04-22T17:23:12.167613+0000","last_peered":"2022-04-22T17:23:12.167613+0000","last_clean":"2022-04-22T17:23:12.167613+0000","last_became_active":"2022-04-22T17:22:18.443606+0000","last_became_peered":"2022-04-22T17:22:18.443606+0000","last_unstale":"2022-04-22T17:23:12.167613+0000","last_undegraded":"2022-04-22T17:23:12.167613+0000","last_fullsized":"2022-04-22T17:23:12.167613+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225632+0000","last_change":"2022-04-22T17:22:18.453280+0000","last_active":"2022-04-22T17:23:12.225632+0000","last_peered":"2022-04-22T17:23:12.225632+0000","last_clean":"2022-04-22T17:23:12.225632+0000","last_became_active":"2022-04-22T17:22:18.452955+0000","last_became_peered":"2022-04-22T17:22:18.452955+0000","last_unstale":"2022-04-22T17:23:12.225632+0000","last_undegraded":"2022-04-22T17:23:12.225632+0000","last_fullsized":"2022-04-22T17:23:12.225632+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166597+0000","last_change":"2022-04-22T17:22:18.447447+0000","last_active":"2022-04-22T17:23:12.166597+0000","last_peered":"2022-04-22T17:23:12.166597+0000","last_clean":"2022-04-22T17:23:12.166597+0000","last_became_active":"2022-04-22T17:22:18.447040+0000","last_became_peered":"2022-04-22T17:22:18.447040+0000","last_unstale":"2022-04-22T17:23:12.166597+0000","last_undegraded":"2022-04-22T17:23:12.166597+0000","last_fullsized":"2022-04-22T17:23:12.166597+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227281+0000","last_change":"2022-04-22T17:22:18.453951+0000","last_active":"2022-04-22T17:23:12.227281+0000","last_peered":"2022-04-22T17:23:12.227281+0000","last_clean":"2022-04-22T17:23:12.227281+0000","last_became_active":"2022-04-22T17:22:18.453648+0000","last_became_peered":"2022-04-22T17:22:18.453648+0000","last_unstale":"2022-04-22T17:23:12.227281+0000","last_undegraded":"2022-04-22T17:23:12.227281+0000","last_fullsized":"2022-04-22T17:23:12.227281+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165475+0000","last_change":"2022-04-22T17:22:18.435236+0000","last_active":"2022-04-22T17:23:12.165475+0000","last_peered":"2022-04-22T17:23:12.165475+0000","last_clean":"2022-04-22T17:23:12.165475+0000","last_became_active":"2022-04-22T17:22:18.434908+0000","last_became_peered":"2022-04-22T17:22:18.434908+0000","last_unstale":"2022-04-22T17:23:12.165475+0000","last_undegraded":"2022-04-22T17:23:12.165475+0000","last_fullsized":"2022-04-22T17:23:12.165475+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165279+0000","last_change":"2022-04-22T17:22:18.453847+0000","last_active":"2022-04-22T17:23:12.165279+0000","last_peered":"2022-04-22T17:23:12.165279+0000","last_clean":"2022-04-22T17:23:12.165279+0000","last_became_active":"2022-04-22T17:22:18.453590+0000","last_became_peered":"2022-04-22T17:22:18.453590+0000","last_unstale":"2022-04-22T17:23:12.165279+0000","last_undegraded":"2022-04-22T17:23:12.165279+0000","last_fullsized":"2022-04-22T17:23:12.165279+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228808+0000","last_change":"2022-04-22T17:22:18.451896+0000","last_active":"2022-04-22T17:23:12.228808+0000","last_peered":"2022-04-22T17:23:12.228808+0000","last_clean":"2022-04-22T17:23:12.228808+0000","last_became_active":"2022-04-22T17:22:18.451579+0000","last_became_peered":"2022-04-22T17:22:18.451579+0000","last_unstale":"2022-04-22T17:23:12.228808+0000","last_undegraded":"2022-04-22T17:23:12.228808+0000","last_fullsized":"2022-04-22T17:23:12.228808+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168272+0000","last_change":"2022-04-22T17:22:18.439501+0000","last_active":"2022-04-22T17:23:12.168272+0000","last_peered":"2022-04-22T17:23:12.168272+0000","last_clean":"2022-04-22T17:23:12.168272+0000","last_became_active":"2022-04-22T17:22:18.439170+0000","last_became_peered":"2022-04-22T17:22:18.439170+0000","last_unstale":"2022-04-22T17:23:12.168272+0000","last_undegraded":"2022-04-22T17:23:12.168272+0000","last_fullsized":"2022-04-22T17:23:12.168272+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230605+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:23:12.230605+0000","last_peered":"2022-04-22T17:23:12.230605+0000","last_clean":"2022-04-22T17:23:12.230605+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:23:12.230605+0000","last_undegraded":"2022-04-22T17:23:12.230605+0000","last_fullsized":"2022-04-22T17:23:12.230605+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228352+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:23:12.228352+0000","last_peered":"2022-04-22T17:23:12.228352+0000","last_clean":"2022-04-22T17:23:12.228352+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:23:12.228352+0000","last_undegraded":"2022-04-22T17:23:12.228352+0000","last_fullsized":"2022-04-22T17:23:12.228352+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166621+0000","last_change":"2022-04-22T17:22:18.447051+0000","last_active":"2022-04-22T17:23:12.166621+0000","last_peered":"2022-04-22T17:23:12.166621+0000","last_clean":"2022-04-22T17:23:12.166621+0000","last_became_active":"2022-04-22T17:22:18.446733+0000","last_became_peered":"2022-04-22T17:22:18.446733+0000","last_unstale":"2022-04-22T17:23:12.166621+0000","last_undegraded":"2022-04-22T17:23:12.166621+0000","last_fullsized":"2022-04-22T17:23:12.166621+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168149+0000","last_change":"2022-04-22T17:22:18.447963+0000","last_active":"2022-04-22T17:23:12.168149+0000","last_peered":"2022-04-22T17:23:12.168149+0000","last_clean":"2022-04-22T17:23:12.168149+0000","last_became_active":"2022-04-22T17:22:18.447352+0000","last_became_peered":"2022-04-22T17:22:18.447352+0000","last_unstale":"2022-04-22T17:23:12.168149+0000","last_undegraded":"2022-04-22T17:23:12.168149+0000","last_fullsized":"2022-04-22T17:23:12.168149+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230036+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:23:12.230036+0000","last_peered":"2022-04-22T17:23:12.230036+0000","last_clean":"2022-04-22T17:23:12.230036+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:23:12.230036+0000","last_undegraded":"2022-04-22T17:23:12.230036+0000","last_fullsized":"2022-04-22T17:23:12.230036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226327+0000","last_change":"2022-04-22T17:22:18.456452+0000","last_active":"2022-04-22T17:23:12.226327+0000","last_peered":"2022-04-22T17:23:12.226327+0000","last_clean":"2022-04-22T17:23:12.226327+0000","last_became_active":"2022-04-22T17:22:18.456060+0000","last_became_peered":"2022-04-22T17:22:18.456060+0000","last_unstale":"2022-04-22T17:23:12.226327+0000","last_undegraded":"2022-04-22T17:23:12.226327+0000","last_fullsized":"2022-04-22T17:23:12.226327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231225+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:23:12.231225+0000","last_peered":"2022-04-22T17:23:12.231225+0000","last_clean":"2022-04-22T17:23:12.231225+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:23:12.231225+0000","last_undegraded":"2022-04-22T17:23:12.231225+0000","last_fullsized":"2022-04-22T17:23:12.231225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169472+0000","last_change":"2022-04-22T17:22:18.440558+0000","last_active":"2022-04-22T17:23:12.169472+0000","last_peered":"2022-04-22T17:23:12.169472+0000","last_clean":"2022-04-22T17:23:12.169472+0000","last_became_active":"2022-04-22T17:22:18.440197+0000","last_became_peered":"2022-04-22T17:22:18.440197+0000","last_unstale":"2022-04-22T17:23:12.169472+0000","last_undegraded":"2022-04-22T17:23:12.169472+0000","last_fullsized":"2022-04-22T17:23:12.169472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169494+0000","last_change":"2022-04-22T17:22:18.117767+0000","last_active":"2022-04-22T17:23:12.169494+0000","last_peered":"2022-04-22T17:23:12.169494+0000","last_clean":"2022-04-22T17:23:12.169494+0000","last_became_active":"2022-04-22T17:22:18.117374+0000","last_became_peered":"2022-04-22T17:22:18.117374+0000","last_unstale":"2022-04-22T17:23:12.169494+0000","last_undegraded":"2022-04-22T17:23:12.169494+0000","last_fullsized":"2022-04-22T17:23:12.169494+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231228+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:23:12.231228+0000","last_peered":"2022-04-22T17:23:12.231228+0000","last_clean":"2022-04-22T17:23:12.231228+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:23:12.231228+0000","last_undegraded":"2022-04-22T17:23:12.231228+0000","last_fullsized":"2022-04-22T17:23:12.231228+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226793+0000","last_change":"2022-04-22T17:22:18.118589+0000","last_active":"2022-04-22T17:23:12.226793+0000","last_peered":"2022-04-22T17:23:12.226793+0000","last_clean":"2022-04-22T17:23:12.226793+0000","last_became_active":"2022-04-22T17:22:18.118118+0000","last_became_peered":"2022-04-22T17:22:18.118118+0000","last_unstale":"2022-04-22T17:23:12.226793+0000","last_undegraded":"2022-04-22T17:23:12.226793+0000","last_fullsized":"2022-04-22T17:23:12.226793+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168113+0000","last_change":"2022-04-22T17:22:18.444605+0000","last_active":"2022-04-22T17:23:12.168113+0000","last_peered":"2022-04-22T17:23:12.168113+0000","last_clean":"2022-04-22T17:23:12.168113+0000","last_became_active":"2022-04-22T17:22:18.444288+0000","last_became_peered":"2022-04-22T17:22:18.444288+0000","last_unstale":"2022-04-22T17:23:12.168113+0000","last_undegraded":"2022-04-22T17:23:12.168113+0000","last_fullsized":"2022-04-22T17:23:12.168113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226409+0000","last_change":"2022-04-22T17:22:18.119939+0000","last_active":"2022-04-22T17:23:12.226409+0000","last_peered":"2022-04-22T17:23:12.226409+0000","last_clean":"2022-04-22T17:23:12.226409+0000","last_became_active":"2022-04-22T17:22:18.119090+0000","last_became_peered":"2022-04-22T17:22:18.119090+0000","last_unstale":"2022-04-22T17:23:12.226409+0000","last_undegraded":"2022-04-22T17:23:12.226409+0000","last_fullsized":"2022-04-22T17:23:12.226409+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168055+0000","last_change":"2022-04-22T17:22:18.446755+0000","last_active":"2022-04-22T17:23:12.168055+0000","last_peered":"2022-04-22T17:23:12.168055+0000","last_clean":"2022-04-22T17:23:12.168055+0000","last_became_active":"2022-04-22T17:22:18.445791+0000","last_became_peered":"2022-04-22T17:22:18.445791+0000","last_unstale":"2022-04-22T17:23:12.168055+0000","last_undegraded":"2022-04-22T17:23:12.168055+0000","last_fullsized":"2022-04-22T17:23:12.168055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168754+0000","last_change":"2022-04-22T17:22:18.115931+0000","last_active":"2022-04-22T17:23:12.168754+0000","last_peered":"2022-04-22T17:23:12.168754+0000","last_clean":"2022-04-22T17:23:12.168754+0000","last_became_active":"2022-04-22T17:22:18.115583+0000","last_became_peered":"2022-04-22T17:22:18.115583+0000","last_unstale":"2022-04-22T17:23:12.168754+0000","last_undegraded":"2022-04-22T17:23:12.168754+0000","last_fullsized":"2022-04-22T17:23:12.168754+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226298+0000","last_change":"2022-04-22T17:22:18.429737+0000","last_active":"2022-04-22T17:23:12.226298+0000","last_peered":"2022-04-22T17:23:12.226298+0000","last_clean":"2022-04-22T17:23:12.226298+0000","last_became_active":"2022-04-22T17:22:18.429404+0000","last_became_peered":"2022-04-22T17:22:18.429404+0000","last_unstale":"2022-04-22T17:23:12.226298+0000","last_undegraded":"2022-04-22T17:23:12.226298+0000","last_fullsized":"2022-04-22T17:23:12.226298+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164537+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:23:12.164537+0000","last_peered":"2022-04-22T17:23:12.164537+0000","last_clean":"2022-04-22T17:23:12.164537+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:23:12.164537+0000","last_undegraded":"2022-04-22T17:23:12.164537+0000","last_fullsized":"2022-04-22T17:23:12.164537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228798+0000","last_change":"2022-04-22T17:22:39.037886+0000","last_active":"2022-04-22T17:23:12.228798+0000","last_peered":"2022-04-22T17:23:12.228798+0000","last_clean":"2022-04-22T17:23:12.228798+0000","last_became_active":"2022-04-22T17:22:39.037184+0000","last_became_peered":"2022-04-22T17:22:39.037184+0000","last_unstale":"2022-04-22T17:23:12.228798+0000","last_undegraded":"2022-04-22T17:23:12.228798+0000","last_fullsized":"2022-04-22T17:23:12.228798+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167072+0000","last_change":"2022-04-22T17:22:18.115369+0000","last_active":"2022-04-22T17:23:12.167072+0000","last_peered":"2022-04-22T17:23:12.167072+0000","last_clean":"2022-04-22T17:23:12.167072+0000","last_became_active":"2022-04-22T17:22:18.115038+0000","last_became_peered":"2022-04-22T17:22:18.115038+0000","last_unstale":"2022-04-22T17:23:12.167072+0000","last_undegraded":"2022-04-22T17:23:12.167072+0000","last_fullsized":"2022-04-22T17:23:12.167072+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166923+0000","last_change":"2022-04-22T17:22:18.443167+0000","last_active":"2022-04-22T17:23:12.166923+0000","last_peered":"2022-04-22T17:23:12.166923+0000","last_clean":"2022-04-22T17:23:12.166923+0000","last_became_active":"2022-04-22T17:22:18.442849+0000","last_became_peered":"2022-04-22T17:22:18.442849+0000","last_unstale":"2022-04-22T17:23:12.166923+0000","last_undegraded":"2022-04-22T17:23:12.166923+0000","last_fullsized":"2022-04-22T17:23:12.166923+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166781+0000","last_change":"2022-04-22T17:22:39.032854+0000","last_active":"2022-04-22T17:23:12.166781+0000","last_peered":"2022-04-22T17:23:12.166781+0000","last_clean":"2022-04-22T17:23:12.166781+0000","last_became_active":"2022-04-22T17:22:39.032538+0000","last_became_peered":"2022-04-22T17:22:39.032538+0000","last_unstale":"2022-04-22T17:23:12.166781+0000","last_undegraded":"2022-04-22T17:23:12.166781+0000","last_fullsized":"2022-04-22T17:23:12.166781+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228521+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:23:12.228521+0000","last_peered":"2022-04-22T17:23:12.228521+0000","last_clean":"2022-04-22T17:23:12.228521+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:23:12.228521+0000","last_undegraded":"2022-04-22T17:23:12.228521+0000","last_fullsized":"2022-04-22T17:23:12.228521+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165658+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:23:12.165658+0000","last_peered":"2022-04-22T17:23:12.165658+0000","last_clean":"2022-04-22T17:23:12.165658+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:23:12.165658+0000","last_undegraded":"2022-04-22T17:23:12.165658+0000","last_fullsized":"2022-04-22T17:23:12.165658+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167761+0000","last_change":"2022-04-22T17:22:18.446536+0000","last_active":"2022-04-22T17:23:12.167761+0000","last_peered":"2022-04-22T17:23:12.167761+0000","last_clean":"2022-04-22T17:23:12.167761+0000","last_became_active":"2022-04-22T17:22:18.446213+0000","last_became_peered":"2022-04-22T17:22:18.446213+0000","last_unstale":"2022-04-22T17:23:12.167761+0000","last_undegraded":"2022-04-22T17:23:12.167761+0000","last_fullsized":"2022-04-22T17:23:12.167761+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167834+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:23:12.167834+0000","last_peered":"2022-04-22T17:23:12.167834+0000","last_clean":"2022-04-22T17:23:12.167834+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:23:12.167834+0000","last_undegraded":"2022-04-22T17:23:12.167834+0000","last_fullsized":"2022-04-22T17:23:12.167834+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231119+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:23:12.231119+0000","last_peered":"2022-04-22T17:23:12.231119+0000","last_clean":"2022-04-22T17:23:12.231119+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:23:12.231119+0000","last_undegraded":"2022-04-22T17:23:12.231119+0000","last_fullsized":"2022-04-22T17:23:12.231119+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229495+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:23:12.229495+0000","last_peered":"2022-04-22T17:23:12.229495+0000","last_clean":"2022-04-22T17:23:12.229495+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:23:12.229495+0000","last_undegraded":"2022-04-22T17:23:12.229495+0000","last_fullsized":"2022-04-22T17:23:12.229495+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228716+0000","last_change":"2022-04-22T17:22:18.443482+0000","last_active":"2022-04-22T17:23:12.228716+0000","last_peered":"2022-04-22T17:23:12.228716+0000","last_clean":"2022-04-22T17:23:12.228716+0000","last_became_active":"2022-04-22T17:22:18.443179+0000","last_became_peered":"2022-04-22T17:22:18.443179+0000","last_unstale":"2022-04-22T17:23:12.228716+0000","last_undegraded":"2022-04-22T17:23:12.228716+0000","last_fullsized":"2022-04-22T17:23:12.228716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"53'6","reported_seq":23,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165018+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:23:12.165018+0000","last_peered":"2022-04-22T17:23:12.165018+0000","last_clean":"2022-04-22T17:23:12.165018+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:23:12.165018+0000","last_undegraded":"2022-04-22T17:23:12.165018+0000","last_fullsized":"2022-04-22T17:23:12.165018+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226568+0000","last_change":"2022-04-22T17:22:18.449910+0000","last_active":"2022-04-22T17:23:12.226568+0000","last_peered":"2022-04-22T17:23:12.226568+0000","last_clean":"2022-04-22T17:23:12.226568+0000","last_became_active":"2022-04-22T17:22:18.449563+0000","last_became_peered":"2022-04-22T17:22:18.449563+0000","last_unstale":"2022-04-22T17:23:12.226568+0000","last_undegraded":"2022-04-22T17:23:12.226568+0000","last_fullsized":"2022-04-22T17:23:12.226568+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231392+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:23:12.231392+0000","last_peered":"2022-04-22T17:23:12.231392+0000","last_clean":"2022-04-22T17:23:12.231392+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:23:12.231392+0000","last_undegraded":"2022-04-22T17:23:12.231392+0000","last_fullsized":"2022-04-22T17:23:12.231392+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230503+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:23:12.230503+0000","last_peered":"2022-04-22T17:23:12.230503+0000","last_clean":"2022-04-22T17:23:12.230503+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:23:12.230503+0000","last_undegraded":"2022-04-22T17:23:12.230503+0000","last_fullsized":"2022-04-22T17:23:12.230503+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167201+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:23:12.167201+0000","last_peered":"2022-04-22T17:23:12.167201+0000","last_clean":"2022-04-22T17:23:12.167201+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:23:12.167201+0000","last_undegraded":"2022-04-22T17:23:12.167201+0000","last_fullsized":"2022-04-22T17:23:12.167201+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228643+0000","last_change":"2022-04-22T17:22:39.037942+0000","last_active":"2022-04-22T17:23:12.228643+0000","last_peered":"2022-04-22T17:23:12.228643+0000","last_clean":"2022-04-22T17:23:12.228643+0000","last_became_active":"2022-04-22T17:22:39.037091+0000","last_became_peered":"2022-04-22T17:22:39.037091+0000","last_unstale":"2022-04-22T17:23:12.228643+0000","last_undegraded":"2022-04-22T17:23:12.228643+0000","last_fullsized":"2022-04-22T17:23:12.228643+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164264+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:23:12.164264+0000","last_peered":"2022-04-22T17:23:12.164264+0000","last_clean":"2022-04-22T17:23:12.164264+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:23:12.164264+0000","last_undegraded":"2022-04-22T17:23:12.164264+0000","last_fullsized":"2022-04-22T17:23:12.164264+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164378+0000","last_change":"2022-04-22T17:22:18.452569+0000","last_active":"2022-04-22T17:23:12.164378+0000","last_peered":"2022-04-22T17:23:12.164378+0000","last_clean":"2022-04-22T17:23:12.164378+0000","last_became_active":"2022-04-22T17:22:18.452287+0000","last_became_peered":"2022-04-22T17:22:18.452287+0000","last_unstale":"2022-04-22T17:23:12.164378+0000","last_undegraded":"2022-04-22T17:23:12.164378+0000","last_fullsized":"2022-04-22T17:23:12.164378+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229231+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:23:12.229231+0000","last_peered":"2022-04-22T17:23:12.229231+0000","last_clean":"2022-04-22T17:23:12.229231+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:23:12.229231+0000","last_undegraded":"2022-04-22T17:23:12.229231+0000","last_fullsized":"2022-04-22T17:23:12.229231+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225859+0000","last_change":"2022-04-22T17:22:18.448940+0000","last_active":"2022-04-22T17:23:12.225859+0000","last_peered":"2022-04-22T17:23:12.225859+0000","last_clean":"2022-04-22T17:23:12.225859+0000","last_became_active":"2022-04-22T17:22:18.448633+0000","last_became_peered":"2022-04-22T17:22:18.448633+0000","last_unstale":"2022-04-22T17:23:12.225859+0000","last_undegraded":"2022-04-22T17:23:12.225859+0000","last_fullsized":"2022-04-22T17:23:12.225859+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229137+0000","last_change":"2022-04-22T17:22:18.112339+0000","last_active":"2022-04-22T17:23:12.229137+0000","last_peered":"2022-04-22T17:23:12.229137+0000","last_clean":"2022-04-22T17:23:12.229137+0000","last_became_active":"2022-04-22T17:22:18.112055+0000","last_became_peered":"2022-04-22T17:22:18.112055+0000","last_unstale":"2022-04-22T17:23:12.229137+0000","last_undegraded":"2022-04-22T17:23:12.229137+0000","last_fullsized":"2022-04-22T17:23:12.229137+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226950+0000","last_change":"2022-04-22T17:22:18.454678+0000","last_active":"2022-04-22T17:23:12.226950+0000","last_peered":"2022-04-22T17:23:12.226950+0000","last_clean":"2022-04-22T17:23:12.226950+0000","last_became_active":"2022-04-22T17:22:18.454221+0000","last_became_peered":"2022-04-22T17:22:18.454221+0000","last_unstale":"2022-04-22T17:23:12.226950+0000","last_undegraded":"2022-04-22T17:23:12.226950+0000","last_fullsized":"2022-04-22T17:23:12.226950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228732+0000","last_change":"2022-04-22T17:23:11.156205+0000","last_active":"2022-04-22T17:23:12.228732+0000","last_peered":"2022-04-22T17:23:12.228732+0000","last_clean":"2022-04-22T17:23:12.228732+0000","last_became_active":"2022-04-22T17:23:11.155917+0000","last_became_peered":"2022-04-22T17:23:11.155917+0000","last_unstale":"2022-04-22T17:23:12.228732+0000","last_undegraded":"2022-04-22T17:23:12.228732+0000","last_fullsized":"2022-04-22T17:23:12.228732+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229006+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:23:12.229006+0000","last_peered":"2022-04-22T17:23:12.229006+0000","last_clean":"2022-04-22T17:23:12.229006+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:23:12.229006+0000","last_undegraded":"2022-04-22T17:23:12.229006+0000","last_fullsized":"2022-04-22T17:23:12.229006+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167817+0000","last_change":"2022-04-22T17:22:18.437505+0000","last_active":"2022-04-22T17:23:12.167817+0000","last_peered":"2022-04-22T17:23:12.167817+0000","last_clean":"2022-04-22T17:23:12.167817+0000","last_became_active":"2022-04-22T17:22:18.437197+0000","last_became_peered":"2022-04-22T17:22:18.437197+0000","last_unstale":"2022-04-22T17:23:12.167817+0000","last_undegraded":"2022-04-22T17:23:12.167817+0000","last_fullsized":"2022-04-22T17:23:12.167817+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228861+0000","last_change":"2022-04-22T17:23:11.271950+0000","last_active":"2022-04-22T17:23:12.228861+0000","last_peered":"2022-04-22T17:23:12.228861+0000","last_clean":"2022-04-22T17:23:12.228861+0000","last_became_active":"2022-04-22T17:23:11.271542+0000","last_became_peered":"2022-04-22T17:23:11.271542+0000","last_unstale":"2022-04-22T17:23:12.228861+0000","last_undegraded":"2022-04-22T17:23:12.228861+0000","last_fullsized":"2022-04-22T17:23:12.228861+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167245+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:23:12.167245+0000","last_peered":"2022-04-22T17:23:12.167245+0000","last_clean":"2022-04-22T17:23:12.167245+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:23:12.167245+0000","last_undegraded":"2022-04-22T17:23:12.167245+0000","last_fullsized":"2022-04-22T17:23:12.167245+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226978+0000","last_change":"2022-04-22T17:22:18.451520+0000","last_active":"2022-04-22T17:23:12.226978+0000","last_peered":"2022-04-22T17:23:12.226978+0000","last_clean":"2022-04-22T17:23:12.226978+0000","last_became_active":"2022-04-22T17:22:18.451231+0000","last_became_peered":"2022-04-22T17:22:18.451231+0000","last_unstale":"2022-04-22T17:23:12.226978+0000","last_undegraded":"2022-04-22T17:23:12.226978+0000","last_fullsized":"2022-04-22T17:23:12.226978+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229730+0000","last_change":"2022-04-22T17:23:11.157136+0000","last_active":"2022-04-22T17:23:12.229730+0000","last_peered":"2022-04-22T17:23:12.229730+0000","last_clean":"2022-04-22T17:23:12.229730+0000","last_became_active":"2022-04-22T17:23:11.156936+0000","last_became_peered":"2022-04-22T17:23:11.156936+0000","last_unstale":"2022-04-22T17:23:12.229730+0000","last_undegraded":"2022-04-22T17:23:12.229730+0000","last_fullsized":"2022-04-22T17:23:12.229730+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228800+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:23:12.228800+0000","last_peered":"2022-04-22T17:23:12.228800+0000","last_clean":"2022-04-22T17:23:12.228800+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:23:12.228800+0000","last_undegraded":"2022-04-22T17:23:12.228800+0000","last_fullsized":"2022-04-22T17:23:12.228800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229069+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:23:12.229069+0000","last_peered":"2022-04-22T17:23:12.229069+0000","last_clean":"2022-04-22T17:23:12.229069+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:23:12.229069+0000","last_undegraded":"2022-04-22T17:23:12.229069+0000","last_fullsized":"2022-04-22T17:23:12.229069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164884+0000","last_change":"2022-04-22T17:23:11.268526+0000","last_active":"2022-04-22T17:23:12.164884+0000","last_peered":"2022-04-22T17:23:12.164884+0000","last_clean":"2022-04-22T17:23:12.164884+0000","last_became_active":"2022-04-22T17:23:11.268160+0000","last_became_peered":"2022-04-22T17:23:11.268160+0000","last_unstale":"2022-04-22T17:23:12.164884+0000","last_undegraded":"2022-04-22T17:23:12.164884+0000","last_fullsized":"2022-04-22T17:23:12.164884+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167466+0000","last_change":"2022-04-22T17:22:18.444926+0000","last_active":"2022-04-22T17:23:12.167466+0000","last_peered":"2022-04-22T17:23:12.167466+0000","last_clean":"2022-04-22T17:23:12.167466+0000","last_became_active":"2022-04-22T17:22:18.444610+0000","last_became_peered":"2022-04-22T17:22:18.444610+0000","last_unstale":"2022-04-22T17:23:12.167466+0000","last_undegraded":"2022-04-22T17:23:12.167466+0000","last_fullsized":"2022-04-22T17:23:12.167466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167403+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:23:12.167403+0000","last_peered":"2022-04-22T17:23:12.167403+0000","last_clean":"2022-04-22T17:23:12.167403+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:23:12.167403+0000","last_undegraded":"2022-04-22T17:23:12.167403+0000","last_fullsized":"2022-04-22T17:23:12.167403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225530+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:23:12.225530+0000","last_peered":"2022-04-22T17:23:12.225530+0000","last_clean":"2022-04-22T17:23:12.225530+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:23:12.225530+0000","last_undegraded":"2022-04-22T17:23:12.225530+0000","last_fullsized":"2022-04-22T17:23:12.225530+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230812+0000","last_change":"2022-04-22T17:23:11.271877+0000","last_active":"2022-04-22T17:23:12.230812+0000","last_peered":"2022-04-22T17:23:12.230812+0000","last_clean":"2022-04-22T17:23:12.230812+0000","last_became_active":"2022-04-22T17:23:11.271455+0000","last_became_peered":"2022-04-22T17:23:11.271455+0000","last_unstale":"2022-04-22T17:23:12.230812+0000","last_undegraded":"2022-04-22T17:23:12.230812+0000","last_fullsized":"2022-04-22T17:23:12.230812+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231055+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:23:12.231055+0000","last_peered":"2022-04-22T17:23:12.231055+0000","last_clean":"2022-04-22T17:23:12.231055+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:23:12.231055+0000","last_undegraded":"2022-04-22T17:23:12.231055+0000","last_fullsized":"2022-04-22T17:23:12.231055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225734+0000","last_change":"2022-04-22T17:22:18.115452+0000","last_active":"2022-04-22T17:23:12.225734+0000","last_peered":"2022-04-22T17:23:12.225734+0000","last_clean":"2022-04-22T17:23:12.225734+0000","last_became_active":"2022-04-22T17:22:18.114179+0000","last_became_peered":"2022-04-22T17:22:18.114179+0000","last_unstale":"2022-04-22T17:23:12.225734+0000","last_undegraded":"2022-04-22T17:23:12.225734+0000","last_fullsized":"2022-04-22T17:23:12.225734+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230406+0000","last_change":"2022-04-22T17:23:11.272541+0000","last_active":"2022-04-22T17:23:12.230406+0000","last_peered":"2022-04-22T17:23:12.230406+0000","last_clean":"2022-04-22T17:23:12.230406+0000","last_became_active":"2022-04-22T17:23:11.271738+0000","last_became_peered":"2022-04-22T17:23:11.271738+0000","last_unstale":"2022-04-22T17:23:12.230406+0000","last_undegraded":"2022-04-22T17:23:12.230406+0000","last_fullsized":"2022-04-22T17:23:12.230406+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226661+0000","last_change":"2022-04-22T17:22:18.115678+0000","last_active":"2022-04-22T17:23:12.226661+0000","last_peered":"2022-04-22T17:23:12.226661+0000","last_clean":"2022-04-22T17:23:12.226661+0000","last_became_active":"2022-04-22T17:22:18.115400+0000","last_became_peered":"2022-04-22T17:22:18.115400+0000","last_unstale":"2022-04-22T17:23:12.226661+0000","last_undegraded":"2022-04-22T17:23:12.226661+0000","last_fullsized":"2022-04-22T17:23:12.226661+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229078+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:23:12.229078+0000","last_peered":"2022-04-22T17:23:12.229078+0000","last_clean":"2022-04-22T17:23:12.229078+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:23:12.229078+0000","last_undegraded":"2022-04-22T17:23:12.229078+0000","last_fullsized":"2022-04-22T17:23:12.229078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166814+0000","last_change":"2022-04-22T17:23:11.272174+0000","last_active":"2022-04-22T17:23:12.166814+0000","last_peered":"2022-04-22T17:23:12.166814+0000","last_clean":"2022-04-22T17:23:12.166814+0000","last_became_active":"2022-04-22T17:23:11.271734+0000","last_became_peered":"2022-04-22T17:23:11.271734+0000","last_unstale":"2022-04-22T17:23:12.166814+0000","last_undegraded":"2022-04-22T17:23:12.166814+0000","last_fullsized":"2022-04-22T17:23:12.166814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"54'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229898+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:23:12.229898+0000","last_peered":"2022-04-22T17:23:12.229898+0000","last_clean":"2022-04-22T17:23:12.229898+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:23:12.229898+0000","last_undegraded":"2022-04-22T17:23:12.229898+0000","last_fullsized":"2022-04-22T17:23:12.229898+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169514+0000","last_change":"2022-04-22T17:22:18.451070+0000","last_active":"2022-04-22T17:23:12.169514+0000","last_peered":"2022-04-22T17:23:12.169514+0000","last_clean":"2022-04-22T17:23:12.169514+0000","last_became_active":"2022-04-22T17:22:18.450715+0000","last_became_peered":"2022-04-22T17:22:18.450715+0000","last_unstale":"2022-04-22T17:23:12.169514+0000","last_undegraded":"2022-04-22T17:23:12.169514+0000","last_fullsized":"2022-04-22T17:23:12.169514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227256+0000","last_change":"2022-04-22T17:23:11.157347+0000","last_active":"2022-04-22T17:23:12.227256+0000","last_peered":"2022-04-22T17:23:12.227256+0000","last_clean":"2022-04-22T17:23:12.227256+0000","last_became_active":"2022-04-22T17:23:11.157000+0000","last_became_peered":"2022-04-22T17:23:11.157000+0000","last_unstale":"2022-04-22T17:23:12.227256+0000","last_undegraded":"2022-04-22T17:23:12.227256+0000","last_fullsized":"2022-04-22T17:23:12.227256+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165505+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:23:12.165505+0000","last_peered":"2022-04-22T17:23:12.165505+0000","last_clean":"2022-04-22T17:23:12.165505+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:23:12.165505+0000","last_undegraded":"2022-04-22T17:23:12.165505+0000","last_fullsized":"2022-04-22T17:23:12.165505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231282+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:23:12.231282+0000","last_peered":"2022-04-22T17:23:12.231282+0000","last_clean":"2022-04-22T17:23:12.231282+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:23:12.231282+0000","last_undegraded":"2022-04-22T17:23:12.231282+0000","last_fullsized":"2022-04-22T17:23:12.231282+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225411+0000","last_change":"2022-04-22T17:22:18.114125+0000","last_active":"2022-04-22T17:23:12.225411+0000","last_peered":"2022-04-22T17:23:12.225411+0000","last_clean":"2022-04-22T17:23:12.225411+0000","last_became_active":"2022-04-22T17:22:18.113799+0000","last_became_peered":"2022-04-22T17:22:18.113799+0000","last_unstale":"2022-04-22T17:23:12.225411+0000","last_undegraded":"2022-04-22T17:23:12.225411+0000","last_fullsized":"2022-04-22T17:23:12.225411+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227789+0000","last_change":"2022-04-22T17:22:18.440746+0000","last_active":"2022-04-22T17:23:12.227789+0000","last_peered":"2022-04-22T17:23:12.227789+0000","last_clean":"2022-04-22T17:23:12.227789+0000","last_became_active":"2022-04-22T17:22:18.440481+0000","last_became_peered":"2022-04-22T17:22:18.440481+0000","last_unstale":"2022-04-22T17:23:12.227789+0000","last_undegraded":"2022-04-22T17:23:12.227789+0000","last_fullsized":"2022-04-22T17:23:12.227789+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228485+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:23:12.228485+0000","last_peered":"2022-04-22T17:23:12.228485+0000","last_clean":"2022-04-22T17:23:12.228485+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:23:12.228485+0000","last_undegraded":"2022-04-22T17:23:12.228485+0000","last_fullsized":"2022-04-22T17:23:12.228485+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166778+0000","last_change":"2022-04-22T17:22:18.445409+0000","last_active":"2022-04-22T17:23:12.166778+0000","last_peered":"2022-04-22T17:23:12.166778+0000","last_clean":"2022-04-22T17:23:12.166778+0000","last_became_active":"2022-04-22T17:22:18.445036+0000","last_became_peered":"2022-04-22T17:22:18.445036+0000","last_unstale":"2022-04-22T17:23:12.166778+0000","last_undegraded":"2022-04-22T17:23:12.166778+0000","last_fullsized":"2022-04-22T17:23:12.166778+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229303+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:23:12.229303+0000","last_peered":"2022-04-22T17:23:12.229303+0000","last_clean":"2022-04-22T17:23:12.229303+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:23:12.229303+0000","last_undegraded":"2022-04-22T17:23:12.229303+0000","last_fullsized":"2022-04-22T17:23:12.229303+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168983+0000","last_change":"2022-04-22T17:22:39.036987+0000","last_active":"2022-04-22T17:23:12.168983+0000","last_peered":"2022-04-22T17:23:12.168983+0000","last_clean":"2022-04-22T17:23:12.168983+0000","last_became_active":"2022-04-22T17:22:39.036130+0000","last_became_peered":"2022-04-22T17:22:39.036130+0000","last_unstale":"2022-04-22T17:23:12.168983+0000","last_undegraded":"2022-04-22T17:23:12.168983+0000","last_fullsized":"2022-04-22T17:23:12.168983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226760+0000","last_change":"2022-04-22T17:22:18.118345+0000","last_active":"2022-04-22T17:23:12.226760+0000","last_peered":"2022-04-22T17:23:12.226760+0000","last_clean":"2022-04-22T17:23:12.226760+0000","last_became_active":"2022-04-22T17:22:18.118019+0000","last_became_peered":"2022-04-22T17:22:18.118019+0000","last_unstale":"2022-04-22T17:23:12.226760+0000","last_undegraded":"2022-04-22T17:23:12.226760+0000","last_fullsized":"2022-04-22T17:23:12.226760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229408+0000","last_change":"2022-04-22T17:22:39.036769+0000","last_active":"2022-04-22T17:23:12.229408+0000","last_peered":"2022-04-22T17:23:12.229408+0000","last_clean":"2022-04-22T17:23:12.229408+0000","last_became_active":"2022-04-22T17:22:39.035610+0000","last_became_peered":"2022-04-22T17:22:39.035610+0000","last_unstale":"2022-04-22T17:23:12.229408+0000","last_undegraded":"2022-04-22T17:23:12.229408+0000","last_fullsized":"2022-04-22T17:23:12.229408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229911+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:23:12.229911+0000","last_peered":"2022-04-22T17:23:12.229911+0000","last_clean":"2022-04-22T17:23:12.229911+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:23:12.229911+0000","last_undegraded":"2022-04-22T17:23:12.229911+0000","last_fullsized":"2022-04-22T17:23:12.229911+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226651+0000","last_change":"2022-04-22T17:22:18.457205+0000","last_active":"2022-04-22T17:23:12.226651+0000","last_peered":"2022-04-22T17:23:12.226651+0000","last_clean":"2022-04-22T17:23:12.226651+0000","last_became_active":"2022-04-22T17:22:18.456866+0000","last_became_peered":"2022-04-22T17:22:18.456866+0000","last_unstale":"2022-04-22T17:23:12.226651+0000","last_undegraded":"2022-04-22T17:23:12.226651+0000","last_fullsized":"2022-04-22T17:23:12.226651+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229384+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:23:12.229384+0000","last_peered":"2022-04-22T17:23:12.229384+0000","last_clean":"2022-04-22T17:23:12.229384+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:23:12.229384+0000","last_undegraded":"2022-04-22T17:23:12.229384+0000","last_fullsized":"2022-04-22T17:23:12.229384+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167820+0000","last_change":"2022-04-22T17:22:18.436889+0000","last_active":"2022-04-22T17:23:12.167820+0000","last_peered":"2022-04-22T17:23:12.167820+0000","last_clean":"2022-04-22T17:23:12.167820+0000","last_became_active":"2022-04-22T17:22:18.436544+0000","last_became_peered":"2022-04-22T17:22:18.436544+0000","last_unstale":"2022-04-22T17:23:12.167820+0000","last_undegraded":"2022-04-22T17:23:12.167820+0000","last_fullsized":"2022-04-22T17:23:12.167820+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229934+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:23:12.229934+0000","last_peered":"2022-04-22T17:23:12.229934+0000","last_clean":"2022-04-22T17:23:12.229934+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:23:12.229934+0000","last_undegraded":"2022-04-22T17:23:12.229934+0000","last_fullsized":"2022-04-22T17:23:12.229934+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227224+0000","last_change":"2022-04-22T17:22:18.453658+0000","last_active":"2022-04-22T17:23:12.227224+0000","last_peered":"2022-04-22T17:23:12.227224+0000","last_clean":"2022-04-22T17:23:12.227224+0000","last_became_active":"2022-04-22T17:22:18.453161+0000","last_became_peered":"2022-04-22T17:22:18.453161+0000","last_unstale":"2022-04-22T17:23:12.227224+0000","last_undegraded":"2022-04-22T17:23:12.227224+0000","last_fullsized":"2022-04-22T17:23:12.227224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164177+0000","last_change":"2022-04-22T17:22:18.451935+0000","last_active":"2022-04-22T17:23:12.164177+0000","last_peered":"2022-04-22T17:23:12.164177+0000","last_clean":"2022-04-22T17:23:12.164177+0000","last_became_active":"2022-04-22T17:22:18.451653+0000","last_became_peered":"2022-04-22T17:22:18.451653+0000","last_unstale":"2022-04-22T17:23:12.164177+0000","last_undegraded":"2022-04-22T17:23:12.164177+0000","last_fullsized":"2022-04-22T17:23:12.164177+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228922+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:23:12.228922+0000","last_peered":"2022-04-22T17:23:12.228922+0000","last_clean":"2022-04-22T17:23:12.228922+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:23:12.228922+0000","last_undegraded":"2022-04-22T17:23:12.228922+0000","last_fullsized":"2022-04-22T17:23:12.228922+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167348+0000","last_change":"2022-04-22T17:22:18.439502+0000","last_active":"2022-04-22T17:23:12.167348+0000","last_peered":"2022-04-22T17:23:12.167348+0000","last_clean":"2022-04-22T17:23:12.167348+0000","last_became_active":"2022-04-22T17:22:18.439094+0000","last_became_peered":"2022-04-22T17:22:18.439094+0000","last_unstale":"2022-04-22T17:23:12.167348+0000","last_undegraded":"2022-04-22T17:23:12.167348+0000","last_fullsized":"2022-04-22T17:23:12.167348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230654+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:23:12.230654+0000","last_peered":"2022-04-22T17:23:12.230654+0000","last_clean":"2022-04-22T17:23:12.230654+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:23:12.230654+0000","last_undegraded":"2022-04-22T17:23:12.230654+0000","last_fullsized":"2022-04-22T17:23:12.230654+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169310+0000","last_change":"2022-04-22T17:22:18.442367+0000","last_active":"2022-04-22T17:23:12.169310+0000","last_peered":"2022-04-22T17:23:12.169310+0000","last_clean":"2022-04-22T17:23:12.169310+0000","last_became_active":"2022-04-22T17:22:18.442081+0000","last_became_peered":"2022-04-22T17:22:18.442081+0000","last_unstale":"2022-04-22T17:23:12.169310+0000","last_undegraded":"2022-04-22T17:23:12.169310+0000","last_fullsized":"2022-04-22T17:23:12.169310+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229061+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:23:12.229061+0000","last_peered":"2022-04-22T17:23:12.229061+0000","last_clean":"2022-04-22T17:23:12.229061+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:23:12.229061+0000","last_undegraded":"2022-04-22T17:23:12.229061+0000","last_fullsized":"2022-04-22T17:23:12.229061+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165693+0000","last_change":"2022-04-22T17:22:18.436856+0000","last_active":"2022-04-22T17:23:12.165693+0000","last_peered":"2022-04-22T17:23:12.165693+0000","last_clean":"2022-04-22T17:23:12.165693+0000","last_became_active":"2022-04-22T17:22:18.436561+0000","last_became_peered":"2022-04-22T17:22:18.436561+0000","last_unstale":"2022-04-22T17:23:12.165693+0000","last_undegraded":"2022-04-22T17:23:12.165693+0000","last_fullsized":"2022-04-22T17:23:12.165693+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169624+0000","last_change":"2022-04-22T17:22:18.438089+0000","last_active":"2022-04-22T17:23:12.169624+0000","last_peered":"2022-04-22T17:23:12.169624+0000","last_clean":"2022-04-22T17:23:12.169624+0000","last_became_active":"2022-04-22T17:22:18.437829+0000","last_became_peered":"2022-04-22T17:22:18.437829+0000","last_unstale":"2022-04-22T17:23:12.169624+0000","last_undegraded":"2022-04-22T17:23:12.169624+0000","last_fullsized":"2022-04-22T17:23:12.169624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166755+0000","last_change":"2022-04-22T17:22:18.437041+0000","last_active":"2022-04-22T17:23:12.166755+0000","last_peered":"2022-04-22T17:23:12.166755+0000","last_clean":"2022-04-22T17:23:12.166755+0000","last_became_active":"2022-04-22T17:22:18.436782+0000","last_became_peered":"2022-04-22T17:22:18.436782+0000","last_unstale":"2022-04-22T17:23:12.166755+0000","last_undegraded":"2022-04-22T17:23:12.166755+0000","last_fullsized":"2022-04-22T17:23:12.166755+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166914+0000","last_change":"2022-04-22T17:22:18.438738+0000","last_active":"2022-04-22T17:23:12.166914+0000","last_peered":"2022-04-22T17:23:12.166914+0000","last_clean":"2022-04-22T17:23:12.166914+0000","last_became_active":"2022-04-22T17:22:18.438520+0000","last_became_peered":"2022-04-22T17:22:18.438520+0000","last_unstale":"2022-04-22T17:23:12.166914+0000","last_undegraded":"2022-04-22T17:23:12.166914+0000","last_fullsized":"2022-04-22T17:23:12.166914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167394+0000","last_change":"2022-04-22T17:22:18.448646+0000","last_active":"2022-04-22T17:23:12.167394+0000","last_peered":"2022-04-22T17:23:12.167394+0000","last_clean":"2022-04-22T17:23:12.167394+0000","last_became_active":"2022-04-22T17:22:18.448263+0000","last_became_peered":"2022-04-22T17:22:18.448263+0000","last_unstale":"2022-04-22T17:23:12.167394+0000","last_undegraded":"2022-04-22T17:23:12.167394+0000","last_fullsized":"2022-04-22T17:23:12.167394+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230964+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:23:12.230964+0000","last_peered":"2022-04-22T17:23:12.230964+0000","last_clean":"2022-04-22T17:23:12.230964+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:23:12.230964+0000","last_undegraded":"2022-04-22T17:23:12.230964+0000","last_fullsized":"2022-04-22T17:23:12.230964+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164877+0000","last_change":"2022-04-22T17:22:18.453245+0000","last_active":"2022-04-22T17:23:12.164877+0000","last_peered":"2022-04-22T17:23:12.164877+0000","last_clean":"2022-04-22T17:23:12.164877+0000","last_became_active":"2022-04-22T17:22:18.452960+0000","last_became_peered":"2022-04-22T17:22:18.452960+0000","last_unstale":"2022-04-22T17:23:12.164877+0000","last_undegraded":"2022-04-22T17:23:12.164877+0000","last_fullsized":"2022-04-22T17:23:12.164877+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":257356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1359999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.034},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.86199999999999999,"15min":0.87},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0940000000000001,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.77700000000000002}]},{"osd":1,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.91800000000000004,"15min":0.90800000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":0.996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.85699999999999998,"15min":0.84699999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.82099999999999995}]},{"osd":2,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.89900000000000002},"min":{"1min":0.70299999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.2150000000000001,"5min":1.528,"15min":1.528},"last":1.137},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":1.1559999999999999}]},{"osd":3,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97399999999999998,"15min":0.98599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1419999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.98399999999999999,"15min":0.98199999999999998},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2889999999999999,"5min":1.375,"15min":1.375},"last":0.93500000000000005}]},{"osd":4,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.84599999999999997,"15min":0.83999999999999997},"min":{"1min":0.53000000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.97599999999999998,"15min":0.98099999999999998},"min":{"1min":0.56899999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.843,"15min":1.843},"last":1.1060000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.028,"15min":1.0229999999999999},"min":{"1min":0.64900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.343,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.98799999999999999,"15min":0.98299999999999998},"min":{"1min":0.60199999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.383,"5min":1.389,"15min":1.389},"last":1.0740000000000001}]},{"osd":6,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0620000000000001,"15min":1.0589999999999999},"min":{"1min":0.76000000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4470000000000001,"5min":1.661,"15min":1.661},"last":1.258},{"interface":"front","average":{"1min":1.052,"5min":1.1000000000000001,"15min":1.1080000000000001},"min":{"1min":0.64800000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3400000000000001,"5min":1.73,"15min":1.73},"last":1.2190000000000001}]}]},{"osd":6,"up_from":42,"seq":180388626473,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":258001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.93000000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.212,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.0569999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.52700000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2430000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.748},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.91100000000000003,"15min":0.92900000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1830000000000001,"5min":1.343,"15min":1.343},"last":1.026}]},{"osd":2,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.91700000000000004,"15min":0.93000000000000005},"min":{"1min":0.55700000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.94599999999999995,"15min":0.96799999999999997},"min":{"1min":0.67400000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1910000000000001,"5min":1.266,"15min":1.266},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.89700000000000002,"15min":0.91500000000000004},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.083,"5min":1.262,"15min":1.262},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1,"15min":1.0169999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.145}]},{"osd":4,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.53500000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.65300000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.90800000000000003}]},{"osd":5,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.997,"15min":1.002},"min":{"1min":0.79800000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.371,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.155},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.024,"15min":1.054},"min":{"1min":0.56499999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3320000000000001,"5min":1.504,"15min":1.504},"last":1}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.999,"15min":0.997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.075},"min":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.329,"5min":1.458,"15min":1.458},"last":1.1180000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5976,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739176,"statfs":{"total":95995035648,"available":95988916224,"internally_reserved":0,"allocated":606208,"data_stored":256625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.53500000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.628},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70099999999999996,"15min":0.59899999999999998},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0900000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.78300000000000003,"15min":0.75},"min":{"1min":0.54800000000000004,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.163,"5min":1.347,"15min":1.347},"last":0.504},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.72399999999999998,"15min":0.69199999999999995},"min":{"1min":0.56200000000000006,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:24:09 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.81000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Apr 22 17:23:24 2022","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0169999999999999,"15min":0.999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.335,"5min":1.542,"15min":1.542},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.996,"5min":0.97599999999999998,"15min":0.95699999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.212,"5min":1.462,"15min":1.462},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:23:52 2022","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98199999999999998,"15min":0.97799999999999998},"min":{"1min":0.67500000000000004,"5min":0.626,"15min":0.626},"max":{"1min":1.3149999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.05,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Apr 22 17:23:24 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0349999999999999,"15min":1.0309999999999999},"min":{"1min":0.85299999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.3440000000000001,"5min":1.395,"15min":1.395},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.069,"5min":1.036,"15min":1.03},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.01}]},{"osd":7,"last update":"Fri Apr 22 17:23:45 2022","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1439999999999999,"15min":1.1579999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.415,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.149,"15min":1.1539999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.75}]}]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":618496,"data_stored":257480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:24:17 2022","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.60399999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":0.47999999999999998}]},{"osd":2,"last update":"Fri Apr 22 17:23:37 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.75700000000000001,"15min":0.72099999999999997},"min":{"1min":0.52300000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.623}]},{"osd":3,"last update":"Fri Apr 22 17:24:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76600000000000001,"15min":0.78500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.82199999999999995,"15min":0.81599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.296,"5min":1.335,"15min":1.335},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Apr 22 17:23:24 2022","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.97699999999999998,"15min":0.97299999999999998},"min":{"1min":0.80100000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3069999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":0.98899999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.4350000000000001,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.95999999999999996}]},{"osd":5,"last update":"Fri Apr 22 17:23:55 2022","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.048,"15min":1.038},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.577,"5min":1.627,"15min":1.627},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.048,"15min":1.038},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.4259999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.75800000000000001}]},{"osd":6,"last update":"Fri Apr 22 17:24:22 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.0740000000000001,"15min":1.0720000000000001},"min":{"1min":0.68100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.369,"5min":5.226,"15min":5.226},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.101,"15min":1.145},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.272,"5min":1.671,"15min":1.671},"last":0.83499999999999996}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.1970000000000001,"15min":1.1870000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":5.2590000000000003,"5min":5.2590000000000003,"15min":5.2590000000000003},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.3,"5min":1.1819999999999999,"15min":1.1619999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":5.2880000000000003,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":1.091}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5840,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739312,"statfs":{"total":95995035648,"available":95989055488,"internally_reserved":0,"allocated":598016,"data_stored":256118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:35 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.45000000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75900000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.68500000000000005,"15min":0.65200000000000002},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.68600000000000005}]},{"osd":1,"last update":"Fri Apr 22 17:23:35 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.69799999999999995},"min":{"1min":0.53000000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.036,"5min":1.294,"15min":1.294},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82199999999999995,"15min":0.78700000000000003},"min":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0620000000000001,"5min":1.262,"15min":1.262},"last":1.0720000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:24:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94899999999999995,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81299999999999994,"15min":0.83699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97499999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.0600000000000001}]},{"osd":4,"last update":"Fri Apr 22 17:23:30 2022","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.95699999999999996,"15min":0.93000000000000005},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.603,"5min":1.923,"15min":1.923},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1,"15min":0.98399999999999999},"min":{"1min":0.81699999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.524,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Apr 22 17:23:54 2022","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0409999999999999,"15min":1.0309999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.008,"15min":0.99399999999999999},"min":{"1min":0.78300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.3939999999999999,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.028}]},{"osd":6,"last update":"Fri Apr 22 17:24:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.036,"15min":1.0509999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":1.113},{"interface":"front","average":{"1min":1.127,"5min":1.111,"15min":1.113},"min":{"1min":0.755,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":1.2869999999999999}]},{"osd":7,"last update":"Fri Apr 22 17:23:48 2022","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.151,"15min":1.161},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.155,"15min":1.161},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.51,"5min":1.782,"15min":1.782},"last":1.2589999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5800,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739352,"statfs":{"total":95995035648,"available":95989096448,"internally_reserved":0,"allocated":622592,"data_stored":258420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71399999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.089,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72699999999999998,"15min":0.73899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0649999999999999,"5min":1.133,"15min":1.133},"last":0.83699999999999997}]},{"osd":1,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76400000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.97499999999999998,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.75900000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.202},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82799999999999996,"15min":0.83699999999999997},"min":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1870000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.876},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79500000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Apr 22 17:23:27 2022","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95399999999999996,"15min":0.93400000000000005},"min":{"1min":0.68899999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.218,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.97199999999999998,"15min":0.97399999999999998},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.347,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.002}]},{"osd":5,"last update":"Fri Apr 22 17:23:57 2022","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.008,"15min":1.0129999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.046,"15min":1.0409999999999999},"min":{"1min":0.77900000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.0129999999999999}]},{"osd":6,"last update":"Fri Apr 22 17:24:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.042,"15min":1.0429999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0800000000000001,"15min":1.0860000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.125}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1519999999999999,"15min":1.1579999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.083},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1100000000000001,"15min":1.1060000000000001},"min":{"1min":0.85899999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.153}]}]},{"osd":4,"up_from":30,"seq":128849018932,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":618496,"data_stored":257460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0780000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83799999999999997,"15min":0.84399999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.077,"5min":1.292,"15min":1.292},"last":0.83999999999999997}]},{"osd":1,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86399999999999999,"15min":0.84099999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.85099999999999998,"15min":0.82599999999999996},"min":{"1min":0.65400000000000003,"5min":0.626,"15min":0.626},"max":{"1min":1.145,"5min":1.276,"15min":1.276},"last":1.117}]},{"osd":2,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85199999999999998,"15min":0.83599999999999997},"min":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1220000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.90400000000000003,"15min":0.91300000000000003},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.93600000000000005,"15min":0.90600000000000003},"min":{"1min":0.73599999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.242,"5min":1.532,"15min":1.532},"last":1.3220000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:23:56 2022","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.495,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.91600000000000004,"15min":0.89600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.3580000000000001}]},{"osd":6,"last update":"Fri Apr 22 17:24:25 2022","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.016,"15min":1},"min":{"1min":0.748,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.423,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.002,"15min":1.02},"min":{"1min":0.83199999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.2170000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:23:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98799999999999999,"15min":0.98999999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.143,"15min":1.151},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.472,"5min":1.603,"15min":1.603},"last":1.4159999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":258406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86799999999999999,"15min":0.86199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85099999999999998,"15min":0.85299999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.167,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.96899999999999997}]},{"osd":1,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.83999999999999997,"15min":0.81699999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.623},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.90000000000000002,"15min":0.89800000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.84699999999999998}]},{"osd":2,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.88,"15min":0.89200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.98399999999999999,"5min":1.147,"15min":1.147},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.92900000000000005,"15min":0.92800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.88700000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.202,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.96099999999999997,"15min":0.95099999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.0109999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.82999999999999996,"15min":0.78800000000000003},"min":{"1min":0.51000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.032},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.82699999999999996},"min":{"1min":0.497,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":1.24}]},{"osd":6,"last update":"Fri Apr 22 17:24:26 2022","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.038,"15min":1.0469999999999999},"min":{"1min":0.66800000000000004,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4610000000000001,"5min":1.482,"15min":1.482},"last":1.097},{"interface":"front","average":{"1min":1.113,"5min":1.0089999999999999,"15min":0.99099999999999999},"min":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.4350000000000001,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.4350000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.004,"15min":1.016},"min":{"1min":0.65800000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.323,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.355},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.129,"15min":1.145},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5,"5min":1.754,"15min":1.754},"last":1.1619999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:24:31.598 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph pg dump --format=json 2022-04-22T17:24:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:32 smithi121 bash[12679]: audit 2022-04-22T17:24:30.694458+0000 mgr.y (mgr.14146) 323 : audit [DBG] from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:32.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:32 smithi121 bash[12679]: cluster 2022-04-22T17:24:31.139221+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v287: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:32 smithi115 bash[12737]: audit 2022-04-22T17:24:30.694458+0000 mgr.y (mgr.14146) 323 : audit [DBG] from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:32 smithi115 bash[12737]: cluster 2022-04-22T17:24:31.139221+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v287: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:32.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:32 smithi115 bash[18070]: audit 2022-04-22T17:24:30.694458+0000 mgr.y (mgr.14146) 323 : audit [DBG] from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:32.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:32 smithi115 bash[18070]: cluster 2022-04-22T17:24:31.139221+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v287: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:33.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:33 smithi121 bash[12679]: cluster 2022-04-22T17:24:33.140108+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v288: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:33.577 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:33 smithi115 bash[12737]: cluster 2022-04-22T17:24:33.140108+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v288: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:33.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:33 smithi115 bash[18070]: cluster 2022-04-22T17:24:33.140108+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v288: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:34.384 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:35.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:35 smithi121 bash[12679]: cluster 2022-04-22T17:24:35.141228+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v289: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:35 smithi115 bash[12737]: cluster 2022-04-22T17:24:35.141228+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v289: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:35.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:35 smithi115 bash[18070]: cluster 2022-04-22T17:24:35.141228+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v289: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:37 smithi121 bash[12679]: cluster 2022-04-22T17:24:37.142099+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v290: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:37 smithi115 bash[12737]: cluster 2022-04-22T17:24:37.142099+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v290: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:37 smithi115 bash[18070]: cluster 2022-04-22T17:24:37.142099+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v290: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:38.164 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:24:38.170 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:24:38.965 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:38 smithi115 bash[12737]: audit 2022-04-22T17:24:38.647710+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:24:38.966 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:38 smithi115 bash[12737]: audit 2022-04-22T17:24:38.650278+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:24:38.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:38 smithi115 bash[18070]: audit 2022-04-22T17:24:38.647710+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:24:38.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:38 smithi115 bash[18070]: audit 2022-04-22T17:24:38.650278+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:24:38.967 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2022-04-22T17:24:37.141585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":315,"acting":315,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":315,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":46772,"kb_used_data":4788,"kb_used_omap":0,"kb_used_meta":41920,"kb_avail":749914444,"statfs":{"total":767960285184,"available":767912390656,"internally_reserved":0,"allocated":4902912,"data_stored":2059866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42926080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005849"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227334+0000","last_change":"2022-04-22T17:22:18.455566+0000","last_active":"2022-04-22T17:23:12.227334+0000","last_peered":"2022-04-22T17:23:12.227334+0000","last_clean":"2022-04-22T17:23:12.227334+0000","last_became_active":"2022-04-22T17:22:18.455256+0000","last_became_peered":"2022-04-22T17:22:18.455256+0000","last_unstale":"2022-04-22T17:23:12.227334+0000","last_undegraded":"2022-04-22T17:23:12.227334+0000","last_fullsized":"2022-04-22T17:23:12.227334+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229728+0000","last_change":"2022-04-22T17:22:18.443213+0000","last_active":"2022-04-22T17:23:12.229728+0000","last_peered":"2022-04-22T17:23:12.229728+0000","last_clean":"2022-04-22T17:23:12.229728+0000","last_became_active":"2022-04-22T17:22:18.442874+0000","last_became_peered":"2022-04-22T17:22:18.442874+0000","last_unstale":"2022-04-22T17:23:12.229728+0000","last_undegraded":"2022-04-22T17:23:12.229728+0000","last_fullsized":"2022-04-22T17:23:12.229728+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231582+0000","last_change":"2022-04-22T17:22:18.456670+0000","last_active":"2022-04-22T17:23:12.231582+0000","last_peered":"2022-04-22T17:23:12.231582+0000","last_clean":"2022-04-22T17:23:12.231582+0000","last_became_active":"2022-04-22T17:22:18.456365+0000","last_became_peered":"2022-04-22T17:22:18.456365+0000","last_unstale":"2022-04-22T17:23:12.231582+0000","last_undegraded":"2022-04-22T17:23:12.231582+0000","last_fullsized":"2022-04-22T17:23:12.231582+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167968+0000","last_change":"2022-04-22T17:22:18.448571+0000","last_active":"2022-04-22T17:23:12.167968+0000","last_peered":"2022-04-22T17:23:12.167968+0000","last_clean":"2022-04-22T17:23:12.167968+0000","last_became_active":"2022-04-22T17:22:18.448283+0000","last_became_peered":"2022-04-22T17:22:18.448283+0000","last_unstale":"2022-04-22T17:23:12.167968+0000","last_undegraded":"2022-04-22T17:23:12.167968+0000","last_fullsized":"2022-04-22T17:23:12.167968+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227043+0000","last_change":"2022-04-22T17:22:38.862851+0000","last_active":"2022-04-22T17:23:12.227043+0000","last_peered":"2022-04-22T17:23:12.227043+0000","last_clean":"2022-04-22T17:23:12.227043+0000","last_became_active":"2022-04-22T17:22:38.862422+0000","last_became_peered":"2022-04-22T17:22:38.862422+0000","last_unstale":"2022-04-22T17:23:12.227043+0000","last_undegraded":"2022-04-22T17:23:12.227043+0000","last_fullsized":"2022-04-22T17:23:12.227043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167613+0000","last_change":"2022-04-22T17:22:18.443897+0000","last_active":"2022-04-22T17:23:12.167613+0000","last_peered":"2022-04-22T17:23:12.167613+0000","last_clean":"2022-04-22T17:23:12.167613+0000","last_became_active":"2022-04-22T17:22:18.443606+0000","last_became_peered":"2022-04-22T17:22:18.443606+0000","last_unstale":"2022-04-22T17:23:12.167613+0000","last_undegraded":"2022-04-22T17:23:12.167613+0000","last_fullsized":"2022-04-22T17:23:12.167613+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225632+0000","last_change":"2022-04-22T17:22:18.453280+0000","last_active":"2022-04-22T17:23:12.225632+0000","last_peered":"2022-04-22T17:23:12.225632+0000","last_clean":"2022-04-22T17:23:12.225632+0000","last_became_active":"2022-04-22T17:22:18.452955+0000","last_became_peered":"2022-04-22T17:22:18.452955+0000","last_unstale":"2022-04-22T17:23:12.225632+0000","last_undegraded":"2022-04-22T17:23:12.225632+0000","last_fullsized":"2022-04-22T17:23:12.225632+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166597+0000","last_change":"2022-04-22T17:22:18.447447+0000","last_active":"2022-04-22T17:23:12.166597+0000","last_peered":"2022-04-22T17:23:12.166597+0000","last_clean":"2022-04-22T17:23:12.166597+0000","last_became_active":"2022-04-22T17:22:18.447040+0000","last_became_peered":"2022-04-22T17:22:18.447040+0000","last_unstale":"2022-04-22T17:23:12.166597+0000","last_undegraded":"2022-04-22T17:23:12.166597+0000","last_fullsized":"2022-04-22T17:23:12.166597+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227281+0000","last_change":"2022-04-22T17:22:18.453951+0000","last_active":"2022-04-22T17:23:12.227281+0000","last_peered":"2022-04-22T17:23:12.227281+0000","last_clean":"2022-04-22T17:23:12.227281+0000","last_became_active":"2022-04-22T17:22:18.453648+0000","last_became_peered":"2022-04-22T17:22:18.453648+0000","last_unstale":"2022-04-22T17:23:12.227281+0000","last_undegraded":"2022-04-22T17:23:12.227281+0000","last_fullsized":"2022-04-22T17:23:12.227281+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165475+0000","last_change":"2022-04-22T17:22:18.435236+0000","last_active":"2022-04-22T17:23:12.165475+0000","last_peered":"2022-04-22T17:23:12.165475+0000","last_clean":"2022-04-22T17:23:12.165475+0000","last_became_active":"2022-04-22T17:22:18.434908+0000","last_became_peered":"2022-04-22T17:22:18.434908+0000","last_unstale":"2022-04-22T17:23:12.165475+0000","last_undegraded":"2022-04-22T17:23:12.165475+0000","last_fullsized":"2022-04-22T17:23:12.165475+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165279+0000","last_change":"2022-04-22T17:22:18.453847+0000","last_active":"2022-04-22T17:23:12.165279+0000","last_peered":"2022-04-22T17:23:12.165279+0000","last_clean":"2022-04-22T17:23:12.165279+0000","last_became_active":"2022-04-22T17:22:18.453590+0000","last_became_peered":"2022-04-22T17:22:18.453590+0000","last_unstale":"2022-04-22T17:23:12.165279+0000","last_undegraded":"2022-04-22T17:23:12.165279+0000","last_fullsized":"2022-04-22T17:23:12.165279+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228808+0000","last_change":"2022-04-22T17:22:18.451896+0000","last_active":"2022-04-22T17:23:12.228808+0000","last_peered":"2022-04-22T17:23:12.228808+0000","last_clean":"2022-04-22T17:23:12.228808+0000","last_became_active":"2022-04-22T17:22:18.451579+0000","last_became_peered":"2022-04-22T17:22:18.451579+0000","last_unstale":"2022-04-22T17:23:12.228808+0000","last_undegraded":"2022-04-22T17:23:12.228808+0000","last_fullsized":"2022-04-22T17:23:12.228808+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168272+0000","last_change":"2022-04-22T17:22:18.439501+0000","last_active":"2022-04-22T17:23:12.168272+0000","last_peered":"2022-04-22T17:23:12.168272+0000","last_clean":"2022-04-22T17:23:12.168272+0000","last_became_active":"2022-04-22T17:22:18.439170+0000","last_became_peered":"2022-04-22T17:22:18.439170+0000","last_unstale":"2022-04-22T17:23:12.168272+0000","last_undegraded":"2022-04-22T17:23:12.168272+0000","last_fullsized":"2022-04-22T17:23:12.168272+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230605+0000","last_change":"2022-04-22T17:22:18.447868+0000","last_active":"2022-04-22T17:23:12.230605+0000","last_peered":"2022-04-22T17:23:12.230605+0000","last_clean":"2022-04-22T17:23:12.230605+0000","last_became_active":"2022-04-22T17:22:18.447564+0000","last_became_peered":"2022-04-22T17:22:18.447564+0000","last_unstale":"2022-04-22T17:23:12.230605+0000","last_undegraded":"2022-04-22T17:23:12.230605+0000","last_fullsized":"2022-04-22T17:23:12.230605+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228352+0000","last_change":"2022-04-22T17:22:18.451210+0000","last_active":"2022-04-22T17:23:12.228352+0000","last_peered":"2022-04-22T17:23:12.228352+0000","last_clean":"2022-04-22T17:23:12.228352+0000","last_became_active":"2022-04-22T17:22:18.450942+0000","last_became_peered":"2022-04-22T17:22:18.450942+0000","last_unstale":"2022-04-22T17:23:12.228352+0000","last_undegraded":"2022-04-22T17:23:12.228352+0000","last_fullsized":"2022-04-22T17:23:12.228352+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166621+0000","last_change":"2022-04-22T17:22:18.447051+0000","last_active":"2022-04-22T17:23:12.166621+0000","last_peered":"2022-04-22T17:23:12.166621+0000","last_clean":"2022-04-22T17:23:12.166621+0000","last_became_active":"2022-04-22T17:22:18.446733+0000","last_became_peered":"2022-04-22T17:22:18.446733+0000","last_unstale":"2022-04-22T17:23:12.166621+0000","last_undegraded":"2022-04-22T17:23:12.166621+0000","last_fullsized":"2022-04-22T17:23:12.166621+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168149+0000","last_change":"2022-04-22T17:22:18.447963+0000","last_active":"2022-04-22T17:23:12.168149+0000","last_peered":"2022-04-22T17:23:12.168149+0000","last_clean":"2022-04-22T17:23:12.168149+0000","last_became_active":"2022-04-22T17:22:18.447352+0000","last_became_peered":"2022-04-22T17:22:18.447352+0000","last_unstale":"2022-04-22T17:23:12.168149+0000","last_undegraded":"2022-04-22T17:23:12.168149+0000","last_fullsized":"2022-04-22T17:23:12.168149+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230036+0000","last_change":"2022-04-22T17:22:18.442140+0000","last_active":"2022-04-22T17:23:12.230036+0000","last_peered":"2022-04-22T17:23:12.230036+0000","last_clean":"2022-04-22T17:23:12.230036+0000","last_became_active":"2022-04-22T17:22:18.441709+0000","last_became_peered":"2022-04-22T17:22:18.441709+0000","last_unstale":"2022-04-22T17:23:12.230036+0000","last_undegraded":"2022-04-22T17:23:12.230036+0000","last_fullsized":"2022-04-22T17:23:12.230036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226327+0000","last_change":"2022-04-22T17:22:18.456452+0000","last_active":"2022-04-22T17:23:12.226327+0000","last_peered":"2022-04-22T17:23:12.226327+0000","last_clean":"2022-04-22T17:23:12.226327+0000","last_became_active":"2022-04-22T17:22:18.456060+0000","last_became_peered":"2022-04-22T17:22:18.456060+0000","last_unstale":"2022-04-22T17:23:12.226327+0000","last_undegraded":"2022-04-22T17:23:12.226327+0000","last_fullsized":"2022-04-22T17:23:12.226327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"53'4","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231225+0000","last_change":"2022-04-22T17:22:18.118715+0000","last_active":"2022-04-22T17:23:12.231225+0000","last_peered":"2022-04-22T17:23:12.231225+0000","last_clean":"2022-04-22T17:23:12.231225+0000","last_became_active":"2022-04-22T17:22:18.117766+0000","last_became_peered":"2022-04-22T17:22:18.117766+0000","last_unstale":"2022-04-22T17:23:12.231225+0000","last_undegraded":"2022-04-22T17:23:12.231225+0000","last_fullsized":"2022-04-22T17:23:12.231225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169472+0000","last_change":"2022-04-22T17:22:18.440558+0000","last_active":"2022-04-22T17:23:12.169472+0000","last_peered":"2022-04-22T17:23:12.169472+0000","last_clean":"2022-04-22T17:23:12.169472+0000","last_became_active":"2022-04-22T17:22:18.440197+0000","last_became_peered":"2022-04-22T17:22:18.440197+0000","last_unstale":"2022-04-22T17:23:12.169472+0000","last_undegraded":"2022-04-22T17:23:12.169472+0000","last_fullsized":"2022-04-22T17:23:12.169472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169494+0000","last_change":"2022-04-22T17:22:18.117767+0000","last_active":"2022-04-22T17:23:12.169494+0000","last_peered":"2022-04-22T17:23:12.169494+0000","last_clean":"2022-04-22T17:23:12.169494+0000","last_became_active":"2022-04-22T17:22:18.117374+0000","last_became_peered":"2022-04-22T17:22:18.117374+0000","last_unstale":"2022-04-22T17:23:12.169494+0000","last_undegraded":"2022-04-22T17:23:12.169494+0000","last_fullsized":"2022-04-22T17:23:12.169494+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231228+0000","last_change":"2022-04-22T17:22:18.449275+0000","last_active":"2022-04-22T17:23:12.231228+0000","last_peered":"2022-04-22T17:23:12.231228+0000","last_clean":"2022-04-22T17:23:12.231228+0000","last_became_active":"2022-04-22T17:22:18.448971+0000","last_became_peered":"2022-04-22T17:22:18.448971+0000","last_unstale":"2022-04-22T17:23:12.231228+0000","last_undegraded":"2022-04-22T17:23:12.231228+0000","last_fullsized":"2022-04-22T17:23:12.231228+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226793+0000","last_change":"2022-04-22T17:22:18.118589+0000","last_active":"2022-04-22T17:23:12.226793+0000","last_peered":"2022-04-22T17:23:12.226793+0000","last_clean":"2022-04-22T17:23:12.226793+0000","last_became_active":"2022-04-22T17:22:18.118118+0000","last_became_peered":"2022-04-22T17:22:18.118118+0000","last_unstale":"2022-04-22T17:23:12.226793+0000","last_undegraded":"2022-04-22T17:23:12.226793+0000","last_fullsized":"2022-04-22T17:23:12.226793+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168113+0000","last_change":"2022-04-22T17:22:18.444605+0000","last_active":"2022-04-22T17:23:12.168113+0000","last_peered":"2022-04-22T17:23:12.168113+0000","last_clean":"2022-04-22T17:23:12.168113+0000","last_became_active":"2022-04-22T17:22:18.444288+0000","last_became_peered":"2022-04-22T17:22:18.444288+0000","last_unstale":"2022-04-22T17:23:12.168113+0000","last_undegraded":"2022-04-22T17:23:12.168113+0000","last_fullsized":"2022-04-22T17:23:12.168113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226409+0000","last_change":"2022-04-22T17:22:18.119939+0000","last_active":"2022-04-22T17:23:12.226409+0000","last_peered":"2022-04-22T17:23:12.226409+0000","last_clean":"2022-04-22T17:23:12.226409+0000","last_became_active":"2022-04-22T17:22:18.119090+0000","last_became_peered":"2022-04-22T17:22:18.119090+0000","last_unstale":"2022-04-22T17:23:12.226409+0000","last_undegraded":"2022-04-22T17:23:12.226409+0000","last_fullsized":"2022-04-22T17:23:12.226409+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168055+0000","last_change":"2022-04-22T17:22:18.446755+0000","last_active":"2022-04-22T17:23:12.168055+0000","last_peered":"2022-04-22T17:23:12.168055+0000","last_clean":"2022-04-22T17:23:12.168055+0000","last_became_active":"2022-04-22T17:22:18.445791+0000","last_became_peered":"2022-04-22T17:22:18.445791+0000","last_unstale":"2022-04-22T17:23:12.168055+0000","last_undegraded":"2022-04-22T17:23:12.168055+0000","last_fullsized":"2022-04-22T17:23:12.168055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168754+0000","last_change":"2022-04-22T17:22:18.115931+0000","last_active":"2022-04-22T17:23:12.168754+0000","last_peered":"2022-04-22T17:23:12.168754+0000","last_clean":"2022-04-22T17:23:12.168754+0000","last_became_active":"2022-04-22T17:22:18.115583+0000","last_became_peered":"2022-04-22T17:22:18.115583+0000","last_unstale":"2022-04-22T17:23:12.168754+0000","last_undegraded":"2022-04-22T17:23:12.168754+0000","last_fullsized":"2022-04-22T17:23:12.168754+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226298+0000","last_change":"2022-04-22T17:22:18.429737+0000","last_active":"2022-04-22T17:23:12.226298+0000","last_peered":"2022-04-22T17:23:12.226298+0000","last_clean":"2022-04-22T17:23:12.226298+0000","last_became_active":"2022-04-22T17:22:18.429404+0000","last_became_peered":"2022-04-22T17:22:18.429404+0000","last_unstale":"2022-04-22T17:23:12.226298+0000","last_undegraded":"2022-04-22T17:23:12.226298+0000","last_fullsized":"2022-04-22T17:23:12.226298+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164537+0000","last_change":"2022-04-22T17:22:18.110705+0000","last_active":"2022-04-22T17:23:12.164537+0000","last_peered":"2022-04-22T17:23:12.164537+0000","last_clean":"2022-04-22T17:23:12.164537+0000","last_became_active":"2022-04-22T17:22:18.110365+0000","last_became_peered":"2022-04-22T17:22:18.110365+0000","last_unstale":"2022-04-22T17:23:12.164537+0000","last_undegraded":"2022-04-22T17:23:12.164537+0000","last_fullsized":"2022-04-22T17:23:12.164537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228798+0000","last_change":"2022-04-22T17:22:39.037886+0000","last_active":"2022-04-22T17:23:12.228798+0000","last_peered":"2022-04-22T17:23:12.228798+0000","last_clean":"2022-04-22T17:23:12.228798+0000","last_became_active":"2022-04-22T17:22:39.037184+0000","last_became_peered":"2022-04-22T17:22:39.037184+0000","last_unstale":"2022-04-22T17:23:12.228798+0000","last_undegraded":"2022-04-22T17:23:12.228798+0000","last_fullsized":"2022-04-22T17:23:12.228798+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167072+0000","last_change":"2022-04-22T17:22:18.115369+0000","last_active":"2022-04-22T17:23:12.167072+0000","last_peered":"2022-04-22T17:23:12.167072+0000","last_clean":"2022-04-22T17:23:12.167072+0000","last_became_active":"2022-04-22T17:22:18.115038+0000","last_became_peered":"2022-04-22T17:22:18.115038+0000","last_unstale":"2022-04-22T17:23:12.167072+0000","last_undegraded":"2022-04-22T17:23:12.167072+0000","last_fullsized":"2022-04-22T17:23:12.167072+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166923+0000","last_change":"2022-04-22T17:22:18.443167+0000","last_active":"2022-04-22T17:23:12.166923+0000","last_peered":"2022-04-22T17:23:12.166923+0000","last_clean":"2022-04-22T17:23:12.166923+0000","last_became_active":"2022-04-22T17:22:18.442849+0000","last_became_peered":"2022-04-22T17:22:18.442849+0000","last_unstale":"2022-04-22T17:23:12.166923+0000","last_undegraded":"2022-04-22T17:23:12.166923+0000","last_fullsized":"2022-04-22T17:23:12.166923+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166781+0000","last_change":"2022-04-22T17:22:39.032854+0000","last_active":"2022-04-22T17:23:12.166781+0000","last_peered":"2022-04-22T17:23:12.166781+0000","last_clean":"2022-04-22T17:23:12.166781+0000","last_became_active":"2022-04-22T17:22:39.032538+0000","last_became_peered":"2022-04-22T17:22:39.032538+0000","last_unstale":"2022-04-22T17:23:12.166781+0000","last_undegraded":"2022-04-22T17:23:12.166781+0000","last_fullsized":"2022-04-22T17:23:12.166781+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228521+0000","last_change":"2022-04-22T17:22:18.451206+0000","last_active":"2022-04-22T17:23:12.228521+0000","last_peered":"2022-04-22T17:23:12.228521+0000","last_clean":"2022-04-22T17:23:12.228521+0000","last_became_active":"2022-04-22T17:22:18.450879+0000","last_became_peered":"2022-04-22T17:22:18.450879+0000","last_unstale":"2022-04-22T17:23:12.228521+0000","last_undegraded":"2022-04-22T17:23:12.228521+0000","last_fullsized":"2022-04-22T17:23:12.228521+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165658+0000","last_change":"2022-04-22T17:22:18.098820+0000","last_active":"2022-04-22T17:23:12.165658+0000","last_peered":"2022-04-22T17:23:12.165658+0000","last_clean":"2022-04-22T17:23:12.165658+0000","last_became_active":"2022-04-22T17:22:18.098367+0000","last_became_peered":"2022-04-22T17:22:18.098367+0000","last_unstale":"2022-04-22T17:23:12.165658+0000","last_undegraded":"2022-04-22T17:23:12.165658+0000","last_fullsized":"2022-04-22T17:23:12.165658+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167761+0000","last_change":"2022-04-22T17:22:18.446536+0000","last_active":"2022-04-22T17:23:12.167761+0000","last_peered":"2022-04-22T17:23:12.167761+0000","last_clean":"2022-04-22T17:23:12.167761+0000","last_became_active":"2022-04-22T17:22:18.446213+0000","last_became_peered":"2022-04-22T17:22:18.446213+0000","last_unstale":"2022-04-22T17:23:12.167761+0000","last_undegraded":"2022-04-22T17:23:12.167761+0000","last_fullsized":"2022-04-22T17:23:12.167761+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167834+0000","last_change":"2022-04-22T17:22:18.103533+0000","last_active":"2022-04-22T17:23:12.167834+0000","last_peered":"2022-04-22T17:23:12.167834+0000","last_clean":"2022-04-22T17:23:12.167834+0000","last_became_active":"2022-04-22T17:22:18.103185+0000","last_became_peered":"2022-04-22T17:22:18.103185+0000","last_unstale":"2022-04-22T17:23:12.167834+0000","last_undegraded":"2022-04-22T17:23:12.167834+0000","last_fullsized":"2022-04-22T17:23:12.167834+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231119+0000","last_change":"2022-04-22T17:22:18.454571+0000","last_active":"2022-04-22T17:23:12.231119+0000","last_peered":"2022-04-22T17:23:12.231119+0000","last_clean":"2022-04-22T17:23:12.231119+0000","last_became_active":"2022-04-22T17:22:18.454281+0000","last_became_peered":"2022-04-22T17:22:18.454281+0000","last_unstale":"2022-04-22T17:23:12.231119+0000","last_undegraded":"2022-04-22T17:23:12.231119+0000","last_fullsized":"2022-04-22T17:23:12.231119+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229495+0000","last_change":"2022-04-22T17:22:18.121199+0000","last_active":"2022-04-22T17:23:12.229495+0000","last_peered":"2022-04-22T17:23:12.229495+0000","last_clean":"2022-04-22T17:23:12.229495+0000","last_became_active":"2022-04-22T17:22:18.120753+0000","last_became_peered":"2022-04-22T17:22:18.120753+0000","last_unstale":"2022-04-22T17:23:12.229495+0000","last_undegraded":"2022-04-22T17:23:12.229495+0000","last_fullsized":"2022-04-22T17:23:12.229495+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228716+0000","last_change":"2022-04-22T17:22:18.443482+0000","last_active":"2022-04-22T17:23:12.228716+0000","last_peered":"2022-04-22T17:23:12.228716+0000","last_clean":"2022-04-22T17:23:12.228716+0000","last_became_active":"2022-04-22T17:22:18.443179+0000","last_became_peered":"2022-04-22T17:22:18.443179+0000","last_unstale":"2022-04-22T17:23:12.228716+0000","last_undegraded":"2022-04-22T17:23:12.228716+0000","last_fullsized":"2022-04-22T17:23:12.228716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"53'6","reported_seq":23,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165018+0000","last_change":"2022-04-22T17:22:18.103415+0000","last_active":"2022-04-22T17:23:12.165018+0000","last_peered":"2022-04-22T17:23:12.165018+0000","last_clean":"2022-04-22T17:23:12.165018+0000","last_became_active":"2022-04-22T17:22:18.103059+0000","last_became_peered":"2022-04-22T17:22:18.103059+0000","last_unstale":"2022-04-22T17:23:12.165018+0000","last_undegraded":"2022-04-22T17:23:12.165018+0000","last_fullsized":"2022-04-22T17:23:12.165018+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226568+0000","last_change":"2022-04-22T17:22:18.449910+0000","last_active":"2022-04-22T17:23:12.226568+0000","last_peered":"2022-04-22T17:23:12.226568+0000","last_clean":"2022-04-22T17:23:12.226568+0000","last_became_active":"2022-04-22T17:22:18.449563+0000","last_became_peered":"2022-04-22T17:22:18.449563+0000","last_unstale":"2022-04-22T17:23:12.226568+0000","last_undegraded":"2022-04-22T17:23:12.226568+0000","last_fullsized":"2022-04-22T17:23:12.226568+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231392+0000","last_change":"2022-04-22T17:22:18.121931+0000","last_active":"2022-04-22T17:23:12.231392+0000","last_peered":"2022-04-22T17:23:12.231392+0000","last_clean":"2022-04-22T17:23:12.231392+0000","last_became_active":"2022-04-22T17:22:18.121628+0000","last_became_peered":"2022-04-22T17:22:18.121628+0000","last_unstale":"2022-04-22T17:23:12.231392+0000","last_undegraded":"2022-04-22T17:23:12.231392+0000","last_fullsized":"2022-04-22T17:23:12.231392+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230503+0000","last_change":"2022-04-22T17:22:18.460589+0000","last_active":"2022-04-22T17:23:12.230503+0000","last_peered":"2022-04-22T17:23:12.230503+0000","last_clean":"2022-04-22T17:23:12.230503+0000","last_became_active":"2022-04-22T17:22:18.460083+0000","last_became_peered":"2022-04-22T17:22:18.460083+0000","last_unstale":"2022-04-22T17:23:12.230503+0000","last_undegraded":"2022-04-22T17:23:12.230503+0000","last_fullsized":"2022-04-22T17:23:12.230503+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167201+0000","last_change":"2022-04-22T17:22:18.116687+0000","last_active":"2022-04-22T17:23:12.167201+0000","last_peered":"2022-04-22T17:23:12.167201+0000","last_clean":"2022-04-22T17:23:12.167201+0000","last_became_active":"2022-04-22T17:22:18.116424+0000","last_became_peered":"2022-04-22T17:22:18.116424+0000","last_unstale":"2022-04-22T17:23:12.167201+0000","last_undegraded":"2022-04-22T17:23:12.167201+0000","last_fullsized":"2022-04-22T17:23:12.167201+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228643+0000","last_change":"2022-04-22T17:22:39.037942+0000","last_active":"2022-04-22T17:23:12.228643+0000","last_peered":"2022-04-22T17:23:12.228643+0000","last_clean":"2022-04-22T17:23:12.228643+0000","last_became_active":"2022-04-22T17:22:39.037091+0000","last_became_peered":"2022-04-22T17:22:39.037091+0000","last_unstale":"2022-04-22T17:23:12.228643+0000","last_undegraded":"2022-04-22T17:23:12.228643+0000","last_fullsized":"2022-04-22T17:23:12.228643+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164264+0000","last_change":"2022-04-22T17:22:18.113792+0000","last_active":"2022-04-22T17:23:12.164264+0000","last_peered":"2022-04-22T17:23:12.164264+0000","last_clean":"2022-04-22T17:23:12.164264+0000","last_became_active":"2022-04-22T17:22:18.113453+0000","last_became_peered":"2022-04-22T17:22:18.113453+0000","last_unstale":"2022-04-22T17:23:12.164264+0000","last_undegraded":"2022-04-22T17:23:12.164264+0000","last_fullsized":"2022-04-22T17:23:12.164264+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164378+0000","last_change":"2022-04-22T17:22:18.452569+0000","last_active":"2022-04-22T17:23:12.164378+0000","last_peered":"2022-04-22T17:23:12.164378+0000","last_clean":"2022-04-22T17:23:12.164378+0000","last_became_active":"2022-04-22T17:22:18.452287+0000","last_became_peered":"2022-04-22T17:22:18.452287+0000","last_unstale":"2022-04-22T17:23:12.164378+0000","last_undegraded":"2022-04-22T17:23:12.164378+0000","last_fullsized":"2022-04-22T17:23:12.164378+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229231+0000","last_change":"2022-04-22T17:22:18.106792+0000","last_active":"2022-04-22T17:23:12.229231+0000","last_peered":"2022-04-22T17:23:12.229231+0000","last_clean":"2022-04-22T17:23:12.229231+0000","last_became_active":"2022-04-22T17:22:18.106008+0000","last_became_peered":"2022-04-22T17:22:18.106008+0000","last_unstale":"2022-04-22T17:23:12.229231+0000","last_undegraded":"2022-04-22T17:23:12.229231+0000","last_fullsized":"2022-04-22T17:23:12.229231+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225859+0000","last_change":"2022-04-22T17:22:18.448940+0000","last_active":"2022-04-22T17:23:12.225859+0000","last_peered":"2022-04-22T17:23:12.225859+0000","last_clean":"2022-04-22T17:23:12.225859+0000","last_became_active":"2022-04-22T17:22:18.448633+0000","last_became_peered":"2022-04-22T17:22:18.448633+0000","last_unstale":"2022-04-22T17:23:12.225859+0000","last_undegraded":"2022-04-22T17:23:12.225859+0000","last_fullsized":"2022-04-22T17:23:12.225859+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229137+0000","last_change":"2022-04-22T17:22:18.112339+0000","last_active":"2022-04-22T17:23:12.229137+0000","last_peered":"2022-04-22T17:23:12.229137+0000","last_clean":"2022-04-22T17:23:12.229137+0000","last_became_active":"2022-04-22T17:22:18.112055+0000","last_became_peered":"2022-04-22T17:22:18.112055+0000","last_unstale":"2022-04-22T17:23:12.229137+0000","last_undegraded":"2022-04-22T17:23:12.229137+0000","last_fullsized":"2022-04-22T17:23:12.229137+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226950+0000","last_change":"2022-04-22T17:22:18.454678+0000","last_active":"2022-04-22T17:23:12.226950+0000","last_peered":"2022-04-22T17:23:12.226950+0000","last_clean":"2022-04-22T17:23:12.226950+0000","last_became_active":"2022-04-22T17:22:18.454221+0000","last_became_peered":"2022-04-22T17:22:18.454221+0000","last_unstale":"2022-04-22T17:23:12.226950+0000","last_undegraded":"2022-04-22T17:23:12.226950+0000","last_fullsized":"2022-04-22T17:23:12.226950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228732+0000","last_change":"2022-04-22T17:23:11.156205+0000","last_active":"2022-04-22T17:23:12.228732+0000","last_peered":"2022-04-22T17:23:12.228732+0000","last_clean":"2022-04-22T17:23:12.228732+0000","last_became_active":"2022-04-22T17:23:11.155917+0000","last_became_peered":"2022-04-22T17:23:11.155917+0000","last_unstale":"2022-04-22T17:23:12.228732+0000","last_undegraded":"2022-04-22T17:23:12.228732+0000","last_fullsized":"2022-04-22T17:23:12.228732+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229006+0000","last_change":"2022-04-22T17:22:18.121884+0000","last_active":"2022-04-22T17:23:12.229006+0000","last_peered":"2022-04-22T17:23:12.229006+0000","last_clean":"2022-04-22T17:23:12.229006+0000","last_became_active":"2022-04-22T17:22:18.121570+0000","last_became_peered":"2022-04-22T17:22:18.121570+0000","last_unstale":"2022-04-22T17:23:12.229006+0000","last_undegraded":"2022-04-22T17:23:12.229006+0000","last_fullsized":"2022-04-22T17:23:12.229006+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167817+0000","last_change":"2022-04-22T17:22:18.437505+0000","last_active":"2022-04-22T17:23:12.167817+0000","last_peered":"2022-04-22T17:23:12.167817+0000","last_clean":"2022-04-22T17:23:12.167817+0000","last_became_active":"2022-04-22T17:22:18.437197+0000","last_became_peered":"2022-04-22T17:22:18.437197+0000","last_unstale":"2022-04-22T17:23:12.167817+0000","last_undegraded":"2022-04-22T17:23:12.167817+0000","last_fullsized":"2022-04-22T17:23:12.167817+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228861+0000","last_change":"2022-04-22T17:23:11.271950+0000","last_active":"2022-04-22T17:23:12.228861+0000","last_peered":"2022-04-22T17:23:12.228861+0000","last_clean":"2022-04-22T17:23:12.228861+0000","last_became_active":"2022-04-22T17:23:11.271542+0000","last_became_peered":"2022-04-22T17:23:11.271542+0000","last_unstale":"2022-04-22T17:23:12.228861+0000","last_undegraded":"2022-04-22T17:23:12.228861+0000","last_fullsized":"2022-04-22T17:23:12.228861+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167245+0000","last_change":"2022-04-22T17:22:18.107633+0000","last_active":"2022-04-22T17:23:12.167245+0000","last_peered":"2022-04-22T17:23:12.167245+0000","last_clean":"2022-04-22T17:23:12.167245+0000","last_became_active":"2022-04-22T17:22:18.107272+0000","last_became_peered":"2022-04-22T17:22:18.107272+0000","last_unstale":"2022-04-22T17:23:12.167245+0000","last_undegraded":"2022-04-22T17:23:12.167245+0000","last_fullsized":"2022-04-22T17:23:12.167245+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226978+0000","last_change":"2022-04-22T17:22:18.451520+0000","last_active":"2022-04-22T17:23:12.226978+0000","last_peered":"2022-04-22T17:23:12.226978+0000","last_clean":"2022-04-22T17:23:12.226978+0000","last_became_active":"2022-04-22T17:22:18.451231+0000","last_became_peered":"2022-04-22T17:22:18.451231+0000","last_unstale":"2022-04-22T17:23:12.226978+0000","last_undegraded":"2022-04-22T17:23:12.226978+0000","last_fullsized":"2022-04-22T17:23:12.226978+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229730+0000","last_change":"2022-04-22T17:23:11.157136+0000","last_active":"2022-04-22T17:23:12.229730+0000","last_peered":"2022-04-22T17:23:12.229730+0000","last_clean":"2022-04-22T17:23:12.229730+0000","last_became_active":"2022-04-22T17:23:11.156936+0000","last_became_peered":"2022-04-22T17:23:11.156936+0000","last_unstale":"2022-04-22T17:23:12.229730+0000","last_undegraded":"2022-04-22T17:23:12.229730+0000","last_fullsized":"2022-04-22T17:23:12.229730+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228800+0000","last_change":"2022-04-22T17:22:18.431535+0000","last_active":"2022-04-22T17:23:12.228800+0000","last_peered":"2022-04-22T17:23:12.228800+0000","last_clean":"2022-04-22T17:23:12.228800+0000","last_became_active":"2022-04-22T17:22:18.430379+0000","last_became_peered":"2022-04-22T17:22:18.430379+0000","last_unstale":"2022-04-22T17:23:12.228800+0000","last_undegraded":"2022-04-22T17:23:12.228800+0000","last_fullsized":"2022-04-22T17:23:12.228800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229069+0000","last_change":"2022-04-22T17:22:18.108722+0000","last_active":"2022-04-22T17:23:12.229069+0000","last_peered":"2022-04-22T17:23:12.229069+0000","last_clean":"2022-04-22T17:23:12.229069+0000","last_became_active":"2022-04-22T17:22:18.107328+0000","last_became_peered":"2022-04-22T17:22:18.107328+0000","last_unstale":"2022-04-22T17:23:12.229069+0000","last_undegraded":"2022-04-22T17:23:12.229069+0000","last_fullsized":"2022-04-22T17:23:12.229069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164884+0000","last_change":"2022-04-22T17:23:11.268526+0000","last_active":"2022-04-22T17:23:12.164884+0000","last_peered":"2022-04-22T17:23:12.164884+0000","last_clean":"2022-04-22T17:23:12.164884+0000","last_became_active":"2022-04-22T17:23:11.268160+0000","last_became_peered":"2022-04-22T17:23:11.268160+0000","last_unstale":"2022-04-22T17:23:12.164884+0000","last_undegraded":"2022-04-22T17:23:12.164884+0000","last_fullsized":"2022-04-22T17:23:12.164884+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167466+0000","last_change":"2022-04-22T17:22:18.444926+0000","last_active":"2022-04-22T17:23:12.167466+0000","last_peered":"2022-04-22T17:23:12.167466+0000","last_clean":"2022-04-22T17:23:12.167466+0000","last_became_active":"2022-04-22T17:22:18.444610+0000","last_became_peered":"2022-04-22T17:22:18.444610+0000","last_unstale":"2022-04-22T17:23:12.167466+0000","last_undegraded":"2022-04-22T17:23:12.167466+0000","last_fullsized":"2022-04-22T17:23:12.167466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167403+0000","last_change":"2022-04-22T17:22:18.110776+0000","last_active":"2022-04-22T17:23:12.167403+0000","last_peered":"2022-04-22T17:23:12.167403+0000","last_clean":"2022-04-22T17:23:12.167403+0000","last_became_active":"2022-04-22T17:22:18.110451+0000","last_became_peered":"2022-04-22T17:22:18.110451+0000","last_unstale":"2022-04-22T17:23:12.167403+0000","last_undegraded":"2022-04-22T17:23:12.167403+0000","last_fullsized":"2022-04-22T17:23:12.167403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225530+0000","last_change":"2022-04-22T17:21:47.152297+0000","last_active":"2022-04-22T17:23:12.225530+0000","last_peered":"2022-04-22T17:23:12.225530+0000","last_clean":"2022-04-22T17:23:12.225530+0000","last_became_active":"2022-04-22T17:21:47.151498+0000","last_became_peered":"2022-04-22T17:21:47.151498+0000","last_unstale":"2022-04-22T17:23:12.225530+0000","last_undegraded":"2022-04-22T17:23:12.225530+0000","last_fullsized":"2022-04-22T17:23:12.225530+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230812+0000","last_change":"2022-04-22T17:23:11.271877+0000","last_active":"2022-04-22T17:23:12.230812+0000","last_peered":"2022-04-22T17:23:12.230812+0000","last_clean":"2022-04-22T17:23:12.230812+0000","last_became_active":"2022-04-22T17:23:11.271455+0000","last_became_peered":"2022-04-22T17:23:11.271455+0000","last_unstale":"2022-04-22T17:23:12.230812+0000","last_undegraded":"2022-04-22T17:23:12.230812+0000","last_fullsized":"2022-04-22T17:23:12.230812+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231055+0000","last_change":"2022-04-22T17:22:18.450291+0000","last_active":"2022-04-22T17:23:12.231055+0000","last_peered":"2022-04-22T17:23:12.231055+0000","last_clean":"2022-04-22T17:23:12.231055+0000","last_became_active":"2022-04-22T17:22:18.449992+0000","last_became_peered":"2022-04-22T17:22:18.449992+0000","last_unstale":"2022-04-22T17:23:12.231055+0000","last_undegraded":"2022-04-22T17:23:12.231055+0000","last_fullsized":"2022-04-22T17:23:12.231055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225734+0000","last_change":"2022-04-22T17:22:18.115452+0000","last_active":"2022-04-22T17:23:12.225734+0000","last_peered":"2022-04-22T17:23:12.225734+0000","last_clean":"2022-04-22T17:23:12.225734+0000","last_became_active":"2022-04-22T17:22:18.114179+0000","last_became_peered":"2022-04-22T17:22:18.114179+0000","last_unstale":"2022-04-22T17:23:12.225734+0000","last_undegraded":"2022-04-22T17:23:12.225734+0000","last_fullsized":"2022-04-22T17:23:12.225734+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230406+0000","last_change":"2022-04-22T17:23:11.272541+0000","last_active":"2022-04-22T17:23:12.230406+0000","last_peered":"2022-04-22T17:23:12.230406+0000","last_clean":"2022-04-22T17:23:12.230406+0000","last_became_active":"2022-04-22T17:23:11.271738+0000","last_became_peered":"2022-04-22T17:23:11.271738+0000","last_unstale":"2022-04-22T17:23:12.230406+0000","last_undegraded":"2022-04-22T17:23:12.230406+0000","last_fullsized":"2022-04-22T17:23:12.230406+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226661+0000","last_change":"2022-04-22T17:22:18.115678+0000","last_active":"2022-04-22T17:23:12.226661+0000","last_peered":"2022-04-22T17:23:12.226661+0000","last_clean":"2022-04-22T17:23:12.226661+0000","last_became_active":"2022-04-22T17:22:18.115400+0000","last_became_peered":"2022-04-22T17:22:18.115400+0000","last_unstale":"2022-04-22T17:23:12.226661+0000","last_undegraded":"2022-04-22T17:23:12.226661+0000","last_fullsized":"2022-04-22T17:23:12.226661+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229078+0000","last_change":"2022-04-22T17:22:18.457429+0000","last_active":"2022-04-22T17:23:12.229078+0000","last_peered":"2022-04-22T17:23:12.229078+0000","last_clean":"2022-04-22T17:23:12.229078+0000","last_became_active":"2022-04-22T17:22:18.456547+0000","last_became_peered":"2022-04-22T17:22:18.456547+0000","last_unstale":"2022-04-22T17:23:12.229078+0000","last_undegraded":"2022-04-22T17:23:12.229078+0000","last_fullsized":"2022-04-22T17:23:12.229078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166814+0000","last_change":"2022-04-22T17:23:11.272174+0000","last_active":"2022-04-22T17:23:12.166814+0000","last_peered":"2022-04-22T17:23:12.166814+0000","last_clean":"2022-04-22T17:23:12.166814+0000","last_became_active":"2022-04-22T17:23:11.271734+0000","last_became_peered":"2022-04-22T17:23:11.271734+0000","last_unstale":"2022-04-22T17:23:12.166814+0000","last_undegraded":"2022-04-22T17:23:12.166814+0000","last_fullsized":"2022-04-22T17:23:12.166814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"54'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229898+0000","last_change":"2022-04-22T17:22:18.103842+0000","last_active":"2022-04-22T17:23:12.229898+0000","last_peered":"2022-04-22T17:23:12.229898+0000","last_clean":"2022-04-22T17:23:12.229898+0000","last_became_active":"2022-04-22T17:22:18.103440+0000","last_became_peered":"2022-04-22T17:22:18.103440+0000","last_unstale":"2022-04-22T17:23:12.229898+0000","last_undegraded":"2022-04-22T17:23:12.229898+0000","last_fullsized":"2022-04-22T17:23:12.229898+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169514+0000","last_change":"2022-04-22T17:22:18.451070+0000","last_active":"2022-04-22T17:23:12.169514+0000","last_peered":"2022-04-22T17:23:12.169514+0000","last_clean":"2022-04-22T17:23:12.169514+0000","last_became_active":"2022-04-22T17:22:18.450715+0000","last_became_peered":"2022-04-22T17:22:18.450715+0000","last_unstale":"2022-04-22T17:23:12.169514+0000","last_undegraded":"2022-04-22T17:23:12.169514+0000","last_fullsized":"2022-04-22T17:23:12.169514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227256+0000","last_change":"2022-04-22T17:23:11.157347+0000","last_active":"2022-04-22T17:23:12.227256+0000","last_peered":"2022-04-22T17:23:12.227256+0000","last_clean":"2022-04-22T17:23:12.227256+0000","last_became_active":"2022-04-22T17:23:11.157000+0000","last_became_peered":"2022-04-22T17:23:11.157000+0000","last_unstale":"2022-04-22T17:23:12.227256+0000","last_undegraded":"2022-04-22T17:23:12.227256+0000","last_fullsized":"2022-04-22T17:23:12.227256+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165505+0000","last_change":"2022-04-22T17:22:18.112844+0000","last_active":"2022-04-22T17:23:12.165505+0000","last_peered":"2022-04-22T17:23:12.165505+0000","last_clean":"2022-04-22T17:23:12.165505+0000","last_became_active":"2022-04-22T17:22:18.112502+0000","last_became_peered":"2022-04-22T17:22:18.112502+0000","last_unstale":"2022-04-22T17:23:12.165505+0000","last_undegraded":"2022-04-22T17:23:12.165505+0000","last_fullsized":"2022-04-22T17:23:12.165505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.231282+0000","last_change":"2022-04-22T17:22:18.458267+0000","last_active":"2022-04-22T17:23:12.231282+0000","last_peered":"2022-04-22T17:23:12.231282+0000","last_clean":"2022-04-22T17:23:12.231282+0000","last_became_active":"2022-04-22T17:22:18.457959+0000","last_became_peered":"2022-04-22T17:22:18.457959+0000","last_unstale":"2022-04-22T17:23:12.231282+0000","last_undegraded":"2022-04-22T17:23:12.231282+0000","last_fullsized":"2022-04-22T17:23:12.231282+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.225411+0000","last_change":"2022-04-22T17:22:18.114125+0000","last_active":"2022-04-22T17:23:12.225411+0000","last_peered":"2022-04-22T17:23:12.225411+0000","last_clean":"2022-04-22T17:23:12.225411+0000","last_became_active":"2022-04-22T17:22:18.113799+0000","last_became_peered":"2022-04-22T17:22:18.113799+0000","last_unstale":"2022-04-22T17:23:12.225411+0000","last_undegraded":"2022-04-22T17:23:12.225411+0000","last_fullsized":"2022-04-22T17:23:12.225411+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227789+0000","last_change":"2022-04-22T17:22:18.440746+0000","last_active":"2022-04-22T17:23:12.227789+0000","last_peered":"2022-04-22T17:23:12.227789+0000","last_clean":"2022-04-22T17:23:12.227789+0000","last_became_active":"2022-04-22T17:22:18.440481+0000","last_became_peered":"2022-04-22T17:22:18.440481+0000","last_unstale":"2022-04-22T17:23:12.227789+0000","last_undegraded":"2022-04-22T17:23:12.227789+0000","last_fullsized":"2022-04-22T17:23:12.227789+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228485+0000","last_change":"2022-04-22T17:22:18.117407+0000","last_active":"2022-04-22T17:23:12.228485+0000","last_peered":"2022-04-22T17:23:12.228485+0000","last_clean":"2022-04-22T17:23:12.228485+0000","last_became_active":"2022-04-22T17:22:18.117071+0000","last_became_peered":"2022-04-22T17:22:18.117071+0000","last_unstale":"2022-04-22T17:23:12.228485+0000","last_undegraded":"2022-04-22T17:23:12.228485+0000","last_fullsized":"2022-04-22T17:23:12.228485+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166778+0000","last_change":"2022-04-22T17:22:18.445409+0000","last_active":"2022-04-22T17:23:12.166778+0000","last_peered":"2022-04-22T17:23:12.166778+0000","last_clean":"2022-04-22T17:23:12.166778+0000","last_became_active":"2022-04-22T17:22:18.445036+0000","last_became_peered":"2022-04-22T17:22:18.445036+0000","last_unstale":"2022-04-22T17:23:12.166778+0000","last_undegraded":"2022-04-22T17:23:12.166778+0000","last_fullsized":"2022-04-22T17:23:12.166778+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229303+0000","last_change":"2022-04-22T17:22:18.116564+0000","last_active":"2022-04-22T17:23:12.229303+0000","last_peered":"2022-04-22T17:23:12.229303+0000","last_clean":"2022-04-22T17:23:12.229303+0000","last_became_active":"2022-04-22T17:22:18.116020+0000","last_became_peered":"2022-04-22T17:22:18.116020+0000","last_unstale":"2022-04-22T17:23:12.229303+0000","last_undegraded":"2022-04-22T17:23:12.229303+0000","last_fullsized":"2022-04-22T17:23:12.229303+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.168983+0000","last_change":"2022-04-22T17:22:39.036987+0000","last_active":"2022-04-22T17:23:12.168983+0000","last_peered":"2022-04-22T17:23:12.168983+0000","last_clean":"2022-04-22T17:23:12.168983+0000","last_became_active":"2022-04-22T17:22:39.036130+0000","last_became_peered":"2022-04-22T17:22:39.036130+0000","last_unstale":"2022-04-22T17:23:12.168983+0000","last_undegraded":"2022-04-22T17:23:12.168983+0000","last_fullsized":"2022-04-22T17:23:12.168983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226760+0000","last_change":"2022-04-22T17:22:18.118345+0000","last_active":"2022-04-22T17:23:12.226760+0000","last_peered":"2022-04-22T17:23:12.226760+0000","last_clean":"2022-04-22T17:23:12.226760+0000","last_became_active":"2022-04-22T17:22:18.118019+0000","last_became_peered":"2022-04-22T17:22:18.118019+0000","last_unstale":"2022-04-22T17:23:12.226760+0000","last_undegraded":"2022-04-22T17:23:12.226760+0000","last_fullsized":"2022-04-22T17:23:12.226760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229408+0000","last_change":"2022-04-22T17:22:39.036769+0000","last_active":"2022-04-22T17:23:12.229408+0000","last_peered":"2022-04-22T17:23:12.229408+0000","last_clean":"2022-04-22T17:23:12.229408+0000","last_became_active":"2022-04-22T17:22:39.035610+0000","last_became_peered":"2022-04-22T17:22:39.035610+0000","last_unstale":"2022-04-22T17:23:12.229408+0000","last_undegraded":"2022-04-22T17:23:12.229408+0000","last_fullsized":"2022-04-22T17:23:12.229408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229911+0000","last_change":"2022-04-22T17:22:18.111604+0000","last_active":"2022-04-22T17:23:12.229911+0000","last_peered":"2022-04-22T17:23:12.229911+0000","last_clean":"2022-04-22T17:23:12.229911+0000","last_became_active":"2022-04-22T17:22:18.111208+0000","last_became_peered":"2022-04-22T17:22:18.111208+0000","last_unstale":"2022-04-22T17:23:12.229911+0000","last_undegraded":"2022-04-22T17:23:12.229911+0000","last_fullsized":"2022-04-22T17:23:12.229911+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.226651+0000","last_change":"2022-04-22T17:22:18.457205+0000","last_active":"2022-04-22T17:23:12.226651+0000","last_peered":"2022-04-22T17:23:12.226651+0000","last_clean":"2022-04-22T17:23:12.226651+0000","last_became_active":"2022-04-22T17:22:18.456866+0000","last_became_peered":"2022-04-22T17:22:18.456866+0000","last_unstale":"2022-04-22T17:23:12.226651+0000","last_undegraded":"2022-04-22T17:23:12.226651+0000","last_fullsized":"2022-04-22T17:23:12.226651+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229384+0000","last_change":"2022-04-22T17:22:18.115181+0000","last_active":"2022-04-22T17:23:12.229384+0000","last_peered":"2022-04-22T17:23:12.229384+0000","last_clean":"2022-04-22T17:23:12.229384+0000","last_became_active":"2022-04-22T17:22:18.114874+0000","last_became_peered":"2022-04-22T17:22:18.114874+0000","last_unstale":"2022-04-22T17:23:12.229384+0000","last_undegraded":"2022-04-22T17:23:12.229384+0000","last_fullsized":"2022-04-22T17:23:12.229384+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167820+0000","last_change":"2022-04-22T17:22:18.436889+0000","last_active":"2022-04-22T17:23:12.167820+0000","last_peered":"2022-04-22T17:23:12.167820+0000","last_clean":"2022-04-22T17:23:12.167820+0000","last_became_active":"2022-04-22T17:22:18.436544+0000","last_became_peered":"2022-04-22T17:22:18.436544+0000","last_unstale":"2022-04-22T17:23:12.167820+0000","last_undegraded":"2022-04-22T17:23:12.167820+0000","last_fullsized":"2022-04-22T17:23:12.167820+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229934+0000","last_change":"2022-04-22T17:22:18.118329+0000","last_active":"2022-04-22T17:23:12.229934+0000","last_peered":"2022-04-22T17:23:12.229934+0000","last_clean":"2022-04-22T17:23:12.229934+0000","last_became_active":"2022-04-22T17:22:18.117977+0000","last_became_peered":"2022-04-22T17:22:18.117977+0000","last_unstale":"2022-04-22T17:23:12.229934+0000","last_undegraded":"2022-04-22T17:23:12.229934+0000","last_fullsized":"2022-04-22T17:23:12.229934+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.227224+0000","last_change":"2022-04-22T17:22:18.453658+0000","last_active":"2022-04-22T17:23:12.227224+0000","last_peered":"2022-04-22T17:23:12.227224+0000","last_clean":"2022-04-22T17:23:12.227224+0000","last_became_active":"2022-04-22T17:22:18.453161+0000","last_became_peered":"2022-04-22T17:22:18.453161+0000","last_unstale":"2022-04-22T17:23:12.227224+0000","last_undegraded":"2022-04-22T17:23:12.227224+0000","last_fullsized":"2022-04-22T17:23:12.227224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164177+0000","last_change":"2022-04-22T17:22:18.451935+0000","last_active":"2022-04-22T17:23:12.164177+0000","last_peered":"2022-04-22T17:23:12.164177+0000","last_clean":"2022-04-22T17:23:12.164177+0000","last_became_active":"2022-04-22T17:22:18.451653+0000","last_became_peered":"2022-04-22T17:22:18.451653+0000","last_unstale":"2022-04-22T17:23:12.164177+0000","last_undegraded":"2022-04-22T17:23:12.164177+0000","last_fullsized":"2022-04-22T17:23:12.164177+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.228922+0000","last_change":"2022-04-22T17:22:18.438590+0000","last_active":"2022-04-22T17:23:12.228922+0000","last_peered":"2022-04-22T17:23:12.228922+0000","last_clean":"2022-04-22T17:23:12.228922+0000","last_became_active":"2022-04-22T17:22:18.438280+0000","last_became_peered":"2022-04-22T17:22:18.438280+0000","last_unstale":"2022-04-22T17:23:12.228922+0000","last_undegraded":"2022-04-22T17:23:12.228922+0000","last_fullsized":"2022-04-22T17:23:12.228922+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167348+0000","last_change":"2022-04-22T17:22:18.439502+0000","last_active":"2022-04-22T17:23:12.167348+0000","last_peered":"2022-04-22T17:23:12.167348+0000","last_clean":"2022-04-22T17:23:12.167348+0000","last_became_active":"2022-04-22T17:22:18.439094+0000","last_became_peered":"2022-04-22T17:22:18.439094+0000","last_unstale":"2022-04-22T17:23:12.167348+0000","last_undegraded":"2022-04-22T17:23:12.167348+0000","last_fullsized":"2022-04-22T17:23:12.167348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230654+0000","last_change":"2022-04-22T17:22:18.459327+0000","last_active":"2022-04-22T17:23:12.230654+0000","last_peered":"2022-04-22T17:23:12.230654+0000","last_clean":"2022-04-22T17:23:12.230654+0000","last_became_active":"2022-04-22T17:22:18.459090+0000","last_became_peered":"2022-04-22T17:22:18.459090+0000","last_unstale":"2022-04-22T17:23:12.230654+0000","last_undegraded":"2022-04-22T17:23:12.230654+0000","last_fullsized":"2022-04-22T17:23:12.230654+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169310+0000","last_change":"2022-04-22T17:22:18.442367+0000","last_active":"2022-04-22T17:23:12.169310+0000","last_peered":"2022-04-22T17:23:12.169310+0000","last_clean":"2022-04-22T17:23:12.169310+0000","last_became_active":"2022-04-22T17:22:18.442081+0000","last_became_peered":"2022-04-22T17:22:18.442081+0000","last_unstale":"2022-04-22T17:23:12.169310+0000","last_undegraded":"2022-04-22T17:23:12.169310+0000","last_fullsized":"2022-04-22T17:23:12.169310+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.229061+0000","last_change":"2022-04-22T17:22:18.434698+0000","last_active":"2022-04-22T17:23:12.229061+0000","last_peered":"2022-04-22T17:23:12.229061+0000","last_clean":"2022-04-22T17:23:12.229061+0000","last_became_active":"2022-04-22T17:22:18.432597+0000","last_became_peered":"2022-04-22T17:22:18.432597+0000","last_unstale":"2022-04-22T17:23:12.229061+0000","last_undegraded":"2022-04-22T17:23:12.229061+0000","last_fullsized":"2022-04-22T17:23:12.229061+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.165693+0000","last_change":"2022-04-22T17:22:18.436856+0000","last_active":"2022-04-22T17:23:12.165693+0000","last_peered":"2022-04-22T17:23:12.165693+0000","last_clean":"2022-04-22T17:23:12.165693+0000","last_became_active":"2022-04-22T17:22:18.436561+0000","last_became_peered":"2022-04-22T17:22:18.436561+0000","last_unstale":"2022-04-22T17:23:12.165693+0000","last_undegraded":"2022-04-22T17:23:12.165693+0000","last_fullsized":"2022-04-22T17:23:12.165693+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.169624+0000","last_change":"2022-04-22T17:22:18.438089+0000","last_active":"2022-04-22T17:23:12.169624+0000","last_peered":"2022-04-22T17:23:12.169624+0000","last_clean":"2022-04-22T17:23:12.169624+0000","last_became_active":"2022-04-22T17:22:18.437829+0000","last_became_peered":"2022-04-22T17:22:18.437829+0000","last_unstale":"2022-04-22T17:23:12.169624+0000","last_undegraded":"2022-04-22T17:23:12.169624+0000","last_fullsized":"2022-04-22T17:23:12.169624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166755+0000","last_change":"2022-04-22T17:22:18.437041+0000","last_active":"2022-04-22T17:23:12.166755+0000","last_peered":"2022-04-22T17:23:12.166755+0000","last_clean":"2022-04-22T17:23:12.166755+0000","last_became_active":"2022-04-22T17:22:18.436782+0000","last_became_peered":"2022-04-22T17:22:18.436782+0000","last_unstale":"2022-04-22T17:23:12.166755+0000","last_undegraded":"2022-04-22T17:23:12.166755+0000","last_fullsized":"2022-04-22T17:23:12.166755+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.166914+0000","last_change":"2022-04-22T17:22:18.438738+0000","last_active":"2022-04-22T17:23:12.166914+0000","last_peered":"2022-04-22T17:23:12.166914+0000","last_clean":"2022-04-22T17:23:12.166914+0000","last_became_active":"2022-04-22T17:22:18.438520+0000","last_became_peered":"2022-04-22T17:22:18.438520+0000","last_unstale":"2022-04-22T17:23:12.166914+0000","last_undegraded":"2022-04-22T17:23:12.166914+0000","last_fullsized":"2022-04-22T17:23:12.166914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.167394+0000","last_change":"2022-04-22T17:22:18.448646+0000","last_active":"2022-04-22T17:23:12.167394+0000","last_peered":"2022-04-22T17:23:12.167394+0000","last_clean":"2022-04-22T17:23:12.167394+0000","last_became_active":"2022-04-22T17:22:18.448263+0000","last_became_peered":"2022-04-22T17:22:18.448263+0000","last_unstale":"2022-04-22T17:23:12.167394+0000","last_undegraded":"2022-04-22T17:23:12.167394+0000","last_fullsized":"2022-04-22T17:23:12.167394+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.230964+0000","last_change":"2022-04-22T17:22:18.460372+0000","last_active":"2022-04-22T17:23:12.230964+0000","last_peered":"2022-04-22T17:23:12.230964+0000","last_clean":"2022-04-22T17:23:12.230964+0000","last_became_active":"2022-04-22T17:22:18.460088+0000","last_became_peered":"2022-04-22T17:22:18.460088+0000","last_unstale":"2022-04-22T17:23:12.230964+0000","last_undegraded":"2022-04-22T17:23:12.230964+0000","last_fullsized":"2022-04-22T17:23:12.230964+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2022-04-22T17:23:12.164877+0000","last_change":"2022-04-22T17:22:18.453245+0000","last_active":"2022-04-22T17:23:12.164877+0000","last_peered":"2022-04-22T17:23:12.164877+0000","last_clean":"2022-04-22T17:23:12.164877+0000","last_became_active":"2022-04-22T17:22:18.452960+0000","last_became_peered":"2022-04-22T17:22:18.452960+0000","last_unstale":"2022-04-22T17:23:12.164877+0000","last_undegraded":"2022-04-22T17:23:12.164877+0000","last_fullsized":"2022-04-22T17:23:12.164877+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430246,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":257356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1359999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.86199999999999999,"15min":0.87},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0940000000000001,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.91800000000000004,"15min":0.90800000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.85699999999999998,"15min":0.84699999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.89900000000000002},"min":{"1min":0.70299999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.2150000000000001,"5min":1.528,"15min":1.528},"last":1.095},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":1.0609999999999999}]},{"osd":3,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97399999999999998,"15min":0.98599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1419999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.98399999999999999,"15min":0.98199999999999998},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2889999999999999,"5min":1.375,"15min":1.375},"last":0.84899999999999998}]},{"osd":4,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.84599999999999997,"15min":0.83999999999999997},"min":{"1min":0.53000000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.97599999999999998,"15min":0.98099999999999998},"min":{"1min":0.56899999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.843,"15min":1.843},"last":0.93999999999999995}]},{"osd":5,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.028,"15min":1.0229999999999999},"min":{"1min":0.64900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.343,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.98799999999999999,"15min":0.98299999999999998},"min":{"1min":0.60199999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.383,"5min":1.389,"15min":1.389},"last":1.254}]},{"osd":6,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0620000000000001,"15min":1.0589999999999999},"min":{"1min":0.76000000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4470000000000001,"5min":1.661,"15min":1.661},"last":1.05},{"interface":"front","average":{"1min":1.052,"5min":1.1000000000000001,"15min":1.1080000000000001},"min":{"1min":0.64800000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3400000000000001,"5min":1.73,"15min":1.73},"last":1.319}]}]},{"osd":6,"up_from":42,"seq":180388626474,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":258001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.93000000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.212,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.52700000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2430000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.91100000000000003,"15min":0.92900000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1830000000000001,"5min":1.343,"15min":1.343},"last":0.77500000000000002}]},{"osd":2,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.91700000000000004,"15min":0.93000000000000005},"min":{"1min":0.55700000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.94599999999999995,"15min":0.96799999999999997},"min":{"1min":0.67400000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1910000000000001,"5min":1.266,"15min":1.266},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.89700000000000002,"15min":0.91500000000000004},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.083,"5min":1.262,"15min":1.262},"last":1.222},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1,"15min":1.0169999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.1850000000000001}]},{"osd":4,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.53500000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":1.149},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.65300000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":1.04}]},{"osd":5,"last update":"Fri Apr 22 17:24:24 2022","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.997,"15min":1.002},"min":{"1min":0.79800000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.371,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.024,"15min":1.054},"min":{"1min":0.56499999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3320000000000001,"5min":1.504,"15min":1.504},"last":1.1180000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.999,"15min":0.997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.079},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.075},"min":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.329,"5min":1.458,"15min":1.458},"last":1.3500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5976,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739176,"statfs":{"total":95995035648,"available":95988916224,"internally_reserved":0,"allocated":606208,"data_stored":256625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.53500000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70099999999999996,"15min":0.59899999999999998},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0900000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Fri Apr 22 17:24:33 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76800000000000002,"15min":0.745},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.0620000000000001,"5min":1.347,"15min":1.347},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.91300000000000003,"5min":1.446,"15min":1.446},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Apr 22 17:24:09 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":1.254},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.81000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.92700000000000005}]},{"osd":4,"last update":"Fri Apr 22 17:24:26 2022","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.008,"15min":0.996},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.153,"5min":1.542,"15min":1.542},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.97299999999999998,"15min":0.95599999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.044}]},{"osd":5,"last update":"Fri Apr 22 17:23:52 2022","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98199999999999998,"15min":0.97799999999999998},"min":{"1min":0.67500000000000004,"5min":0.626,"15min":0.626},"max":{"1min":1.3149999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.135},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.05,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.85599999999999998}]},{"osd":6,"last update":"Fri Apr 22 17:24:26 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.01,"15min":1.0229999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0209999999999999,"15min":1.0249999999999999},"min":{"1min":0.77100000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.2190000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:23:45 2022","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1439999999999999,"15min":1.1579999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.415,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.149,"15min":1.1539999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.27}]}]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":618496,"data_stored":257480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:24:17 2022","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.60399999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":0.873}]},{"osd":2,"last update":"Fri Apr 22 17:23:37 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.75700000000000001,"15min":0.72099999999999997},"min":{"1min":0.52300000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Apr 22 17:24:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76600000000000001,"15min":0.78500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.82199999999999995,"15min":0.81599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.296,"5min":1.335,"15min":1.335},"last":0.79300000000000004}]},{"osd":4,"last update":"Fri Apr 22 17:24:28 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.96599999999999997,"15min":0.96899999999999997},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.198},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.99099999999999999,"15min":0.98399999999999999},"min":{"1min":0.67200000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.248,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Fri Apr 22 17:23:55 2022","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.048,"15min":1.038},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.577,"5min":1.627,"15min":1.627},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.048,"15min":1.038},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.4259999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.026}]},{"osd":6,"last update":"Fri Apr 22 17:24:22 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.0740000000000001,"15min":1.0720000000000001},"min":{"1min":0.68100000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.369,"5min":5.226,"15min":5.226},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.101,"15min":1.145},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.272,"5min":1.671,"15min":1.671},"last":1.0720000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.1970000000000001,"15min":1.1870000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":5.2590000000000003,"5min":5.2590000000000003,"15min":5.2590000000000003},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.1819999999999999,"15min":1.1619999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":5.2880000000000003,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":0.99099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5840,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739312,"statfs":{"total":95995035648,"available":95989055488,"internally_reserved":0,"allocated":598016,"data_stored":256118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:35 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.45000000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75900000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.68500000000000005,"15min":0.65200000000000002},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.70799999999999996}]},{"osd":1,"last update":"Fri Apr 22 17:23:35 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.69799999999999995},"min":{"1min":0.53000000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.036,"5min":1.294,"15min":1.294},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82199999999999995,"15min":0.78700000000000003},"min":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0620000000000001,"5min":1.262,"15min":1.262},"last":0.93100000000000005}]},{"osd":3,"last update":"Fri Apr 22 17:24:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94899999999999995,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81299999999999994,"15min":0.83699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97499999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.85999999999999999}]},{"osd":4,"last update":"Fri Apr 22 17:24:30 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95499999999999996,"15min":0.92900000000000005},"min":{"1min":0.749,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.256,"5min":1.923,"15min":1.923},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":0.98599999999999999},"min":{"1min":0.78900000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.3740000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.093}]},{"osd":5,"last update":"Fri Apr 22 17:23:54 2022","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0409999999999999,"15min":1.0309999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.071},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.008,"15min":0.99399999999999999},"min":{"1min":0.78300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.3939999999999999,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.22}]},{"osd":6,"last update":"Fri Apr 22 17:24:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.036,"15min":1.0509999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.111,"15min":1.113},"min":{"1min":0.755,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":1.256}]},{"osd":7,"last update":"Fri Apr 22 17:23:48 2022","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.151,"15min":1.161},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.155,"15min":1.161},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.51,"5min":1.782,"15min":1.782},"last":1.1240000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5800,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739352,"statfs":{"total":95995035648,"available":95989096448,"internally_reserved":0,"allocated":622592,"data_stored":258420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71399999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.089,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72699999999999998,"15min":0.73899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0649999999999999,"5min":1.133,"15min":1.133},"last":0.46700000000000003}]},{"osd":1,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76400000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.97499999999999998,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.75900000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.202},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:24:11 2022","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82799999999999996,"15min":0.83699999999999997},"min":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1870000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79500000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.66600000000000004}]},{"osd":4,"last update":"Fri Apr 22 17:24:29 2022","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.97899999999999998,"15min":0.94299999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.97299999999999998,"15min":0.97499999999999998},"min":{"1min":0.68300000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5740000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Fri Apr 22 17:23:57 2022","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.008,"15min":1.0129999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.042},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.046,"15min":1.0409999999999999},"min":{"1min":0.77900000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.90300000000000002}]},{"osd":6,"last update":"Fri Apr 22 17:24:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.042,"15min":1.0429999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0800000000000001,"15min":1.0860000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.008}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1519999999999999,"15min":1.1579999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1100000000000001,"15min":1.1060000000000001},"min":{"1min":0.85899999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.85999999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018933,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":618496,"data_stored":257460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0780000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83799999999999997,"15min":0.84399999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.077,"5min":1.292,"15min":1.292},"last":0.81799999999999995}]},{"osd":1,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86399999999999999,"15min":0.84099999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.85099999999999998,"15min":0.82599999999999996},"min":{"1min":0.65400000000000003,"5min":0.626,"15min":0.626},"max":{"1min":1.145,"5min":1.276,"15min":1.276},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85199999999999998,"15min":0.83599999999999997},"min":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1220000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.90400000000000003,"15min":0.91300000000000003},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.61699999999999999}]},{"osd":3,"last update":"Fri Apr 22 17:23:32 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.748},{"interface":"front","average":{"1min":1.004,"5min":0.93600000000000005,"15min":0.90600000000000003},"min":{"1min":0.73599999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.242,"5min":1.532,"15min":1.532},"last":0.64700000000000002}]},{"osd":5,"last update":"Fri Apr 22 17:23:56 2022","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.85199999999999998,"15min":0.81200000000000006},"min":{"1min":0.495,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.91600000000000004,"15min":0.89600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":0.84899999999999998}]},{"osd":6,"last update":"Fri Apr 22 17:24:25 2022","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.016,"15min":1},"min":{"1min":0.748,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.423,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.073,"5min":1.002,"15min":1.02},"min":{"1min":0.83199999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.028}]},{"osd":7,"last update":"Fri Apr 22 17:23:50 2022","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98799999999999999,"15min":0.98999999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.071},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.143,"15min":1.151},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.472,"5min":1.603,"15min":1.603},"last":1.1200000000000001}]}]},{"osd":5,"up_from":36,"seq":154618822703,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":258406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86799999999999999,"15min":0.86199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85099999999999998,"15min":0.85299999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.167,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94199999999999995}]},{"osd":1,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.83999999999999997,"15min":0.81699999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.88},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.90000000000000002,"15min":0.89800000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.0720000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.88,"15min":0.89200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.98399999999999999,"5min":1.147,"15min":1.147},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.92900000000000005,"15min":0.92800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":1.054}]},{"osd":3,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.202,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.96099999999999997,"15min":0.95099999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Apr 22 17:23:58 2022","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.82999999999999996,"15min":0.78800000000000003},"min":{"1min":0.51000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.026},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.82699999999999996},"min":{"1min":0.497,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":1.1299999999999999}]},{"osd":6,"last update":"Fri Apr 22 17:24:26 2022","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.038,"15min":1.0469999999999999},"min":{"1min":0.66800000000000004,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4610000000000001,"5min":1.482,"15min":1.482},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.0089999999999999,"15min":0.99099999999999999},"min":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.4350000000000001,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.2}]},{"osd":7,"last update":"Fri Apr 22 17:23:46 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.004,"15min":1.016},"min":{"1min":0.65800000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.323,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.26},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.129,"15min":1.145},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5,"5min":1.754,"15min":1.754},"last":1.1799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:24:39.011 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-04-22T17:24:39.011 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-04-22T17:24:39.011 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-04-22T17:24:39.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph health --format=json 2022-04-22T17:24:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:38 smithi121 bash[12679]: audit 2022-04-22T17:24:38.647710+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:24:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:38 smithi121 bash[12679]: audit 2022-04-22T17:24:38.650278+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:24:40.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:39 smithi121 bash[12679]: audit 2022-04-22T17:24:38.143921+0000 mgr.y (mgr.14146) 328 : audit [DBG] from='client.24691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:40.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:39 smithi121 bash[12679]: cluster 2022-04-22T17:24:39.143084+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v291: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:39 smithi115 bash[12737]: audit 2022-04-22T17:24:38.143921+0000 mgr.y (mgr.14146) 328 : audit [DBG] from='client.24691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:39 smithi115 bash[12737]: cluster 2022-04-22T17:24:39.143084+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v291: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:39 smithi115 bash[18070]: audit 2022-04-22T17:24:38.143921+0000 mgr.y (mgr.14146) 328 : audit [DBG] from='client.24691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:24:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:39 smithi115 bash[18070]: cluster 2022-04-22T17:24:39.143084+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v291: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:41.805 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:43 smithi115 bash[12737]: cluster 2022-04-22T17:24:41.144168+0000 mgr.y (mgr.14146) 330 : cluster [DBG] pgmap v292: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:43.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:43 smithi115 bash[18070]: cluster 2022-04-22T17:24:41.144168+0000 mgr.y (mgr.14146) 330 : cluster [DBG] pgmap v292: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:43.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:43 smithi121 bash[12679]: cluster 2022-04-22T17:24:41.144168+0000 mgr.y (mgr.14146) 330 : cluster [DBG] pgmap v292: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:45.298 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:24:45.385 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:45 smithi115 bash[12737]: cluster 2022-04-22T17:24:43.144897+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v293: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:45.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:45 smithi115 bash[18070]: cluster 2022-04-22T17:24:43.144897+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v293: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:45 smithi121 bash[12679]: cluster 2022-04-22T17:24:43.144897+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v293: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:46.139 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-22T17:24:46.139 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-04-22T17:24:46.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-22T17:24:46.404 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:24:46.404 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":60,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:23:14.898360+0000","last_up_change":"2022-04-22T17:21:45.090307+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"rbd","create_time":"2022-04-22T17:23:10.004854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6803","nonce":1141629491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6805","nonce":1141629491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6809","nonce":1141629491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1141629491},{"type":"v1","addr":"172.21.15.115:6807","nonce":1141629491}]},"public_addr":"172.21.15.115:6803/1141629491","cluster_addr":"172.21.15.115:6805/1141629491","heartbeat_back_addr":"172.21.15.115:6809/1141629491","heartbeat_front_addr":"172.21.15.115:6807/1141629491","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6811","nonce":245673474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6813","nonce":245673474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6817","nonce":245673474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":245673474},{"type":"v1","addr":"172.21.15.115:6815","nonce":245673474}]},"public_addr":"172.21.15.115:6811/245673474","cluster_addr":"172.21.15.115:6813/245673474","heartbeat_back_addr":"172.21.15.115:6817/245673474","heartbeat_front_addr":"172.21.15.115:6815/245673474","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6819","nonce":2968343147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6821","nonce":2968343147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6825","nonce":2968343147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2968343147},{"type":"v1","addr":"172.21.15.115:6823","nonce":2968343147}]},"public_addr":"172.21.15.115:6819/2968343147","cluster_addr":"172.21.15.115:6821/2968343147","heartbeat_back_addr":"172.21.15.115:6825/2968343147","heartbeat_front_addr":"172.21.15.115:6823/2968343147","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6827","nonce":543187144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6829","nonce":543187144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6833","nonce":543187144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":543187144},{"type":"v1","addr":"172.21.15.115:6831","nonce":543187144}]},"public_addr":"172.21.15.115:6827/543187144","cluster_addr":"172.21.15.115:6829/543187144","heartbeat_back_addr":"172.21.15.115:6833/543187144","heartbeat_front_addr":"172.21.15.115:6831/543187144","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6801","nonce":826605480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6803","nonce":826605480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6807","nonce":826605480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":826605480},{"type":"v1","addr":"172.21.15.121:6805","nonce":826605480}]},"public_addr":"172.21.15.121:6801/826605480","cluster_addr":"172.21.15.121:6803/826605480","heartbeat_back_addr":"172.21.15.121:6807/826605480","heartbeat_front_addr":"172.21.15.121:6805/826605480","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6809","nonce":1865718089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6811","nonce":1865718089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6815","nonce":1865718089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1865718089},{"type":"v1","addr":"172.21.15.121:6813","nonce":1865718089}]},"public_addr":"172.21.15.121:6809/1865718089","cluster_addr":"172.21.15.121:6811/1865718089","heartbeat_back_addr":"172.21.15.121:6815/1865718089","heartbeat_front_addr":"172.21.15.121:6813/1865718089","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6817","nonce":1033820083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6819","nonce":1033820083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6823","nonce":1033820083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1033820083},{"type":"v1","addr":"172.21.15.121:6821","nonce":1033820083}]},"public_addr":"172.21.15.121:6817/1033820083","cluster_addr":"172.21.15.121:6819/1033820083","heartbeat_back_addr":"172.21.15.121:6823/1033820083","heartbeat_front_addr":"172.21.15.121:6821/1033820083","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6825","nonce":493226072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6827","nonce":493226072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6831","nonce":493226072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":493226072},{"type":"v1","addr":"172.21.15.121:6829","nonce":493226072}]},"public_addr":"172.21.15.121:6825/493226072","cluster_addr":"172.21.15.121:6827/493226072","heartbeat_back_addr":"172.21.15.121:6831/493226072","heartbeat_front_addr":"172.21.15.121:6829/493226072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.3b","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:24:46.416 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:46 smithi115 bash[12737]: audit 2022-04-22T17:24:45.295113+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.115:0/1578325086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-04-22T17:24:46.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:46 smithi115 bash[18070]: audit 2022-04-22T17:24:45.295113+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.115:0/1578325086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-04-22T17:24:46.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-04-22T17:19:28.627096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-04-22T17:22:16.858039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-04-22T17:22:17.796936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'rbd', 'create_time': '2022-04-22T17:23:10.004854+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-04-22T17:24:46.419 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-04-22T17:24:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:46 smithi121 bash[12679]: audit 2022-04-22T17:24:45.295113+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.115:0/1578325086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-04-22T17:24:46.685 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2022-04-22T17:24:46.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-04-22T17:24:46.971 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2022-04-22T17:24:46.985 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-04-22T17:24:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[18070]: cluster 2022-04-22T17:24:45.145533+0000 mgr.y (mgr.14146) 332 : cluster [DBG] pgmap v294: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[18070]: audit 2022-04-22T17:24:46.400184+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.115:0/471673548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:24:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[18070]: audit 2022-04-22T17:24:46.682913+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.115:0/70255351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[18070]: audit 2022-04-22T17:24:46.968083+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? 172.21.15.115:0/2399793806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[12737]: cluster 2022-04-22T17:24:45.145533+0000 mgr.y (mgr.14146) 332 : cluster [DBG] pgmap v294: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[12737]: audit 2022-04-22T17:24:46.400184+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.115:0/471673548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:24:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[12737]: audit 2022-04-22T17:24:46.682913+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.115:0/70255351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:47 smithi115 bash[12737]: audit 2022-04-22T17:24:46.968083+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? 172.21.15.115:0/2399793806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.299 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2022-04-22T17:24:47.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-22T17:24:47.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:47 smithi121 bash[12679]: cluster 2022-04-22T17:24:45.145533+0000 mgr.y (mgr.14146) 332 : cluster [DBG] pgmap v294: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:47 smithi121 bash[12679]: audit 2022-04-22T17:24:46.400184+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.115:0/471673548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:24:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:47 smithi121 bash[12679]: audit 2022-04-22T17:24:46.682913+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.115:0/70255351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:47 smithi121 bash[12679]: audit 2022-04-22T17:24:46.968083+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? 172.21.15.115:0/2399793806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-04-22T17:24:47.580 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 8 2022-04-22T17:24:47.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-22T17:24:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:24:47.898 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.115:6835/3475839169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3475839169},{"type":"v1","addr":"172.21.15.115:6835","nonce":3475839169}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-22T17:22:18.413732+0000","modified":"2022-04-22T17:22:20.428010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24386},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24386":{"gid":24386,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6833/2211373279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2211373279},{"type":"v1","addr":"172.21.15.121:6833","nonce":2211373279}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-22T17:24:47.901 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 6 2022-04-22T17:24:47.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-04-22T17:22:18.413732+0000', 'modified': '2022-04-22T17:22:20.428010+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24386}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24386': {'gid': 24386, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6833/2211373279', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6832', 'nonce': 2211373279}, {'type': 'v1', 'addr': '172.21.15.121:6833', 'nonce': 2211373279}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-04-22T17:24:47.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-04-22T17:24:47.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2022-04-22T17:24:48.035 INFO:teuthology.orchestra.run.smithi115.stderr:2022-04-22T17:24:48.028+0000 7f9b817fa700 0 client.24745 ms_handle_reset on v2:172.21.15.121:6832/2211373279 2022-04-22T17:24:48.062 INFO:teuthology.orchestra.run.smithi115.stderr:2022-04-22T17:24:48.056+0000 7f9b817fa700 0 client.14844 ms_handle_reset on v2:172.21.15.121:6832/2211373279 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "b6061a0a-c25f-11ec-8c39-001a4aab830c", 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 0, 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24386, 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 149.64473654899999, 2022-04-22T17:24:48.063 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 6, 2022-04-22T17:24:48.064 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 60, 2022-04-22T17:24:48.064 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 54, 2022-04-22T17:24:48.064 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 156.52622257100001 2022-04-22T17:24:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:24:48.079 INFO:tasks.cephadm:Setup complete, yielding 2022-04-22T17:24:48.079 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:24:48.092 INFO:teuthology.task.print:**** done end installing pacific cephadm ... 2022-04-22T17:24:48.092 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:24:48.139 INFO:teuthology.task.print:**** done start cephadm.shell ceph config set mgr... 2022-04-22T17:24:48.139 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-04-22T17:24:48.206 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2022-04-22T17:24:48.207 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2022-04-22T17:24:48.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[18070]: audit 2022-04-22T17:24:47.296575+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.115:0/639379979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[18070]: audit 2022-04-22T17:24:47.577303+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.115:0/749589491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[18070]: audit 2022-04-22T17:24:47.895411+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.115:0/2506211391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:24:48.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[12737]: audit 2022-04-22T17:24:47.296575+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.115:0/639379979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[12737]: audit 2022-04-22T17:24:47.577303+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.115:0/749589491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:48 smithi115 bash[12737]: audit 2022-04-22T17:24:47.895411+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.115:0/2506211391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:24:48.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:48 smithi121 bash[12679]: audit 2022-04-22T17:24:47.296575+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.115:0/639379979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:48 smithi121 bash[12679]: audit 2022-04-22T17:24:47.577303+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.115:0/749589491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]: dispatch 2022-04-22T17:24:48.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:48 smithi121 bash[12679]: audit 2022-04-22T17:24:47.895411+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.115:0/2506211391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-04-22T17:24:49.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:49 smithi121 bash[12679]: cluster 2022-04-22T17:24:47.146037+0000 mgr.y (mgr.14146) 333 : cluster [DBG] pgmap v295: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:49 smithi115 bash[12737]: cluster 2022-04-22T17:24:47.146037+0000 mgr.y (mgr.14146) 333 : cluster [DBG] pgmap v295: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:49 smithi115 bash[18070]: cluster 2022-04-22T17:24:47.146037+0000 mgr.y (mgr.14146) 333 : cluster [DBG] pgmap v295: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:50 smithi121 bash[12679]: cluster 2022-04-22T17:24:49.146985+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v296: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:50 smithi115 bash[12737]: cluster 2022-04-22T17:24:49.146985+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v296: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:50 smithi115 bash[18070]: cluster 2022-04-22T17:24:49.146985+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v296: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:52.914 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:24:52.927 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2022-04-22T17:24:52.927 INFO:teuthology.run_tasks:Running task print... 2022-04-22T17:24:52.973 INFO:teuthology.task.print:**** done start parallel 2022-04-22T17:24:52.973 INFO:teuthology.run_tasks:Running task parallel... 2022-04-22T17:24:53.024 INFO:teuthology.task.parallel:starting parallel... 2022-04-22T17:24:53.024 INFO:teuthology.task.parallel:In parallel, running task full_sequential... 2022-04-22T17:24:53.025 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T17:24:53.025 INFO:teuthology.task.print:**** done start ec-rados-default.yaml 2022-04-22T17:24:53.025 INFO:teuthology.task.full_sequential:In full_sequential, running task rados... 2022-04-22T17:24:53.027 INFO:tasks.rados:Beginning rados... 2022-04-22T17:24:53.028 INFO:tasks.rados:joining rados 2022-04-22T17:24:53.028 INFO:teuthology.task.parallel:In parallel, running task sequential... 2022-04-22T17:24:53.028 INFO:teuthology.task.sequential:In sequential, running task print... 2022-04-22T17:24:53.028 INFO:teuthology.task.print:**** done start upgrade, wait 2022-04-22T17:24:53.029 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2022-04-22T17:24:53.029 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2022-04-22T17:24:53.029 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -e sha1=720674591667bc34cb7086618bbed8722a1f429d -- bash -c 'ceph config set global log_to_journald false --force' 2022-04-22T17:24:53.030 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2022-04-22T17:24:53.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd erasure-code-profile set teuthologyprofile k=2 m=1 crush-failure-domain=osd 2022-04-22T17:24:53.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: cluster 2022-04-22T17:24:51.148113+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v297: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:53.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: audit 2022-04-22T17:24:52.084851+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: audit 2022-04-22T17:24:52.092771+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: audit 2022-04-22T17:24:52.094478+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: audit 2022-04-22T17:24:52.095616+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[12737]: audit 2022-04-22T17:24:52.386243+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: cluster 2022-04-22T17:24:51.148113+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v297: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: audit 2022-04-22T17:24:52.084851+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: audit 2022-04-22T17:24:52.092771+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:53.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: audit 2022-04-22T17:24:52.094478+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:53.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: audit 2022-04-22T17:24:52.095616+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:53.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:53 smithi115 bash[18070]: audit 2022-04-22T17:24:52.386243+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:53.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: cluster 2022-04-22T17:24:51.148113+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v297: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: audit 2022-04-22T17:24:52.084851+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T17:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: audit 2022-04-22T17:24:52.092771+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: audit 2022-04-22T17:24:52.094478+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: audit 2022-04-22T17:24:52.095616+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:53 smithi121 bash[12679]: audit 2022-04-22T17:24:52.386243+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:55.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:55 smithi115 bash[18070]: cluster 2022-04-22T17:24:53.148678+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v298: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:55.381 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:55 smithi115 bash[12737]: cluster 2022-04-22T17:24:53.148678+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v298: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:55 smithi121 bash[12679]: cluster 2022-04-22T17:24:53.148678+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v298: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:55.822 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:24:57.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:57 smithi115 bash[18070]: cluster 2022-04-22T17:24:55.149732+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v299: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:57.387 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:57 smithi115 bash[12737]: cluster 2022-04-22T17:24:55.149732+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v299: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:57.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:57 smithi121 bash[12679]: cluster 2022-04-22T17:24:55.149732+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v299: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:58.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -e sha1=720674591667bc34cb7086618bbed8722a1f429d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: cluster 2022-04-22T17:24:57.150418+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v300: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: audit 2022-04-22T17:24:57.584729+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.115:0/9072455' entity='client.admin' 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: audit 2022-04-22T17:24:57.589583+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: audit 2022-04-22T17:24:57.591119+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: audit 2022-04-22T17:24:57.593902+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:58.925 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[12737]: audit 2022-04-22T17:24:57.603648+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: cluster 2022-04-22T17:24:57.150418+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v300: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: audit 2022-04-22T17:24:57.584729+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.115:0/9072455' entity='client.admin' 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: audit 2022-04-22T17:24:57.589583+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: audit 2022-04-22T17:24:57.591119+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: audit 2022-04-22T17:24:57.593902+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:58.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:24:58 smithi115 bash[18070]: audit 2022-04-22T17:24:57.603648+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:24:59.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: cluster 2022-04-22T17:24:57.150418+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v300: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:24:59.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: audit 2022-04-22T17:24:57.584729+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.115:0/9072455' entity='client.admin' 2022-04-22T17:24:59.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: audit 2022-04-22T17:24:57.589583+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:24:59.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: audit 2022-04-22T17:24:57.591119+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:24:59.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: audit 2022-04-22T17:24:57.593902+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:24:59.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:24:58 smithi121 bash[12679]: audit 2022-04-22T17:24:57.603648+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:01.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:00 smithi121 bash[12679]: cluster 2022-04-22T17:24:59.151668+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v301: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:00 smithi115 bash[12737]: cluster 2022-04-22T17:24:59.151668+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v301: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:01.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:00 smithi115 bash[18070]: cluster 2022-04-22T17:24:59.151668+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v301: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:01.691 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:01 smithi115 bash[12737]: audit 2022-04-22T17:25:00.726330+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/1681783821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:01.691 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:01 smithi115 bash[12737]: audit 2022-04-22T17:25:00.726939+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:01.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:01 smithi115 bash[18070]: audit 2022-04-22T17:25:00.726330+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/1681783821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:01.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:01 smithi115 bash[18070]: audit 2022-04-22T17:25:00.726939+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:01 smithi121 bash[12679]: audit 2022-04-22T17:25:00.726330+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/1681783821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:01 smithi121 bash[12679]: audit 2022-04-22T17:25:00.726939+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2022-04-22T17:25:02.786 INFO:tasks.rados:starting run 0 out of 1 2022-04-22T17:25:02.786 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2022-04-22T17:25:02.786 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd pool create unique_pool_0 16 16 erasure teuthologyprofile 2022-04-22T17:25:02.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[12737]: cluster 2022-04-22T17:25:01.152740+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v302: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:02.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[12737]: audit 2022-04-22T17:25:01.601829+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2022-04-22T17:25:02.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[12737]: cluster 2022-04-22T17:25:01.601928+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2022-04-22T17:25:02.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[18070]: cluster 2022-04-22T17:25:01.152740+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v302: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:02.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[18070]: audit 2022-04-22T17:25:01.601829+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2022-04-22T17:25:02.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:02 smithi115 bash[18070]: cluster 2022-04-22T17:25:01.601928+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2022-04-22T17:25:03.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:02 smithi121 bash[12679]: cluster 2022-04-22T17:25:01.152740+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v302: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:03.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:02 smithi121 bash[12679]: audit 2022-04-22T17:25:01.601829+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "teuthologyprofile", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2022-04-22T17:25:03.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:02 smithi121 bash[12679]: cluster 2022-04-22T17:25:01.601928+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2022-04-22T17:25:03.358 INFO:teuthology.orchestra.run.smithi115.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.355 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -e sha1=720674591667bc34cb7086618bbed8722a1f429d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done' 2022-04-22T17:25:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: cluster 2022-04-22T17:25:03.153275+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v304: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.347982+0000 mgr.y (mgr.14146) 342 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: cephadm 2022-04-22T17:25:03.349214+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.354279+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.355351+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.358002+0000 mon.a (mon.0) 728 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.359600+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: audit 2022-04-22T17:25:03.367173+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[12737]: cephadm 2022-04-22T17:25:03.373172+0000 mgr.y (mgr.14146) 344 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: cluster 2022-04-22T17:25:03.153275+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v304: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.347982+0000 mgr.y (mgr.14146) 342 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:04.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: cephadm 2022-04-22T17:25:03.349214+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.354279+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.355351+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.358002+0000 mon.a (mon.0) 728 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.359600+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: audit 2022-04-22T17:25:03.367173+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:04 smithi115 bash[18070]: cephadm 2022-04-22T17:25:03.373172+0000 mgr.y (mgr.14146) 344 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: cluster 2022-04-22T17:25:03.153275+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v304: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.347982+0000 mgr.y (mgr.14146) 342 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: cephadm 2022-04-22T17:25:03.349214+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.354279+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.355351+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.358002+0000 mon.a (mon.0) 728 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.359600+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: audit 2022-04-22T17:25:03.367173+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:04.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:04 smithi121 bash[12679]: cephadm 2022-04-22T17:25:03.373172+0000 mgr.y (mgr.14146) 344 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:720674591667bc34cb7086618bbed8722a1f429d 2022-04-22T17:25:05.874 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:25:07.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:07 smithi121 bash[12679]: cluster 2022-04-22T17:25:05.154229+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v305: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:07 smithi115 bash[12737]: cluster 2022-04-22T17:25:05.154229+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v305: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:07 smithi115 bash[18070]: cluster 2022-04-22T17:25:05.154229+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v305: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:09 smithi121 bash[12679]: cluster 2022-04-22T17:25:07.154834+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v306: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:09 smithi115 bash[12737]: cluster 2022-04-22T17:25:07.154834+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v306: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:09 smithi115 bash[18070]: cluster 2022-04-22T17:25:07.154834+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v306: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:11.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:11 smithi121 bash[12679]: cluster 2022-04-22T17:25:09.155515+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v307: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:11 smithi115 bash[12737]: cluster 2022-04-22T17:25:09.155515+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v307: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:11 smithi115 bash[18070]: cluster 2022-04-22T17:25:09.155515+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v307: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:13 smithi115 bash[12737]: cluster 2022-04-22T17:25:11.156054+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v308: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:13 smithi115 bash[18070]: cluster 2022-04-22T17:25:11.156054+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v308: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:13.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:13 smithi121 bash[12679]: cluster 2022-04-22T17:25:11.156054+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v308: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:15.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:15 smithi115 bash[12737]: cluster 2022-04-22T17:25:13.156488+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v309: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:15.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:15 smithi115 bash[18070]: cluster 2022-04-22T17:25:13.156488+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v309: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:15 smithi121 bash[12679]: cluster 2022-04-22T17:25:13.156488+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v309: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:17.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:17 smithi121 bash[12679]: cluster 2022-04-22T17:25:15.157215+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v310: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:17.621 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:17 smithi115 bash[12737]: cluster 2022-04-22T17:25:15.157215+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v310: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:17.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:17 smithi115 bash[18070]: cluster 2022-04-22T17:25:15.157215+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v310: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:19.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:19 smithi121 bash[12679]: cluster 2022-04-22T17:25:17.157714+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v311: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:19 smithi115 bash[12737]: cluster 2022-04-22T17:25:17.157714+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v311: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:19 smithi115 bash[18070]: cluster 2022-04-22T17:25:17.157714+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v311: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:20 smithi115 bash[12737]: cluster 2022-04-22T17:25:19.158403+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v312: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:20 smithi115 bash[18070]: cluster 2022-04-22T17:25:19.158403+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v312: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:20.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:20 smithi121 bash[12679]: cluster 2022-04-22T17:25:19.158403+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v312: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:23 smithi121 bash[12679]: cluster 2022-04-22T17:25:21.158966+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v313: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:23 smithi115 bash[18070]: cluster 2022-04-22T17:25:21.158966+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v313: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:23 smithi115 bash[12737]: cluster 2022-04-22T17:25:21.158966+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v313: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:25 smithi115 bash[12737]: cluster 2022-04-22T17:25:23.159642+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v314: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:25 smithi115 bash[18070]: cluster 2022-04-22T17:25:23.159642+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v314: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:25.418 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:25:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:25 smithi121 bash[12679]: cluster 2022-04-22T17:25:23.159642+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v314: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:25.751 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:25:25.751 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (107s) 80s ago 2m 8767k - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:25:25.751 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2m) 84s ago 2m 22.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:25:25.751 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3m) 80s ago 3m 11.3M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3m) 84s ago 3m 13.1M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (7m) 84s ago 7m 357M - 16.2.5 46cf6318c64e c89d6897ce72 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running 80s ago 9m - - 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running 80s ago 9m - 2048M 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (7m) 84s ago 7m 38.0M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 80s ago 8m 38.1M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 80s ago 2m 6232k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2m) 84s ago 2m 6448k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 80s ago 7m 38.5M 4062M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6m) 80s ago 6m 38.4M 4062M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:25:25.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6m) 80s ago 6m 36.6M 4062M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5m) 80s ago 5m 37.7M 4062M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5m) 84s ago 5m 37.5M 4318M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4m) 84s ago 4m 38.0M 4318M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4m) 84s ago 4m 38.3M 4318M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3m) 84s ago 3m 38.6M 4318M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:25:25.753 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (102s) 84s ago 2m 23.8M - 2.18.1 de242295e225 6fd3ae0aa84b 2022-04-22T17:25:26.118 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:25:26.118 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:25:26.118 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:25:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:26.120 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:25:26.120 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 15 2022-04-22T17:25:26.120 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:25:26.120 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:25:26.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:26 smithi121 bash[12679]: audit 2022-04-22T17:25:26.113945+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.115:0/269254370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:26 smithi115 bash[12737]: audit 2022-04-22T17:25:26.113945+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.115:0/269254370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:26 smithi115 bash[18070]: audit 2022-04-22T17:25:26.113945+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.115:0/269254370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:27.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:27 smithi121 bash[12679]: cluster 2022-04-22T17:25:25.160269+0000 mgr.y (mgr.14146) 355 : cluster [DBG] pgmap v315: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:27 smithi121 bash[12679]: audit 2022-04-22T17:25:25.398801+0000 mgr.y (mgr.14146) 356 : audit [DBG] from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:27 smithi121 bash[12679]: audit 2022-04-22T17:25:25.734142+0000 mgr.y (mgr.14146) 357 : audit [DBG] from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[12737]: cluster 2022-04-22T17:25:25.160269+0000 mgr.y (mgr.14146) 355 : cluster [DBG] pgmap v315: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[12737]: audit 2022-04-22T17:25:25.398801+0000 mgr.y (mgr.14146) 356 : audit [DBG] from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[12737]: audit 2022-04-22T17:25:25.734142+0000 mgr.y (mgr.14146) 357 : audit [DBG] from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[18070]: cluster 2022-04-22T17:25:25.160269+0000 mgr.y (mgr.14146) 355 : cluster [DBG] pgmap v315: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[18070]: audit 2022-04-22T17:25:25.398801+0000 mgr.y (mgr.14146) 356 : audit [DBG] from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:27.633 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:27 smithi115 bash[18070]: audit 2022-04-22T17:25:25.734142+0000 mgr.y (mgr.14146) 357 : audit [DBG] from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:29 smithi121 bash[12679]: cluster 2022-04-22T17:25:27.160758+0000 mgr.y (mgr.14146) 358 : cluster [DBG] pgmap v316: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:29 smithi115 bash[18070]: cluster 2022-04-22T17:25:27.160758+0000 mgr.y (mgr.14146) 358 : cluster [DBG] pgmap v316: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:29 smithi115 bash[12737]: cluster 2022-04-22T17:25:27.160758+0000 mgr.y (mgr.14146) 358 : cluster [DBG] pgmap v316: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:30.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:30 smithi121 bash[12679]: cluster 2022-04-22T17:25:29.161250+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v317: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:30.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:30 smithi115 bash[12737]: cluster 2022-04-22T17:25:29.161250+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v317: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:30.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:30 smithi115 bash[18070]: cluster 2022-04-22T17:25:29.161250+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v317: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:31 smithi115 bash[12737]: audit 2022-04-22T17:25:30.934292+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/995038059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:31 smithi115 bash[12737]: audit 2022-04-22T17:25:30.934796+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:31 smithi115 bash[18070]: audit 2022-04-22T17:25:30.934292+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/995038059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:31 smithi115 bash[18070]: audit 2022-04-22T17:25:30.934796+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:31.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:31 smithi121 bash[12679]: audit 2022-04-22T17:25:30.934292+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/995038059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:31.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:31 smithi121 bash[12679]: audit 2022-04-22T17:25:30.934796+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]: dispatch 2022-04-22T17:25:32.246 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'unique_pool_0' created 2022-04-22T17:25:32.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: cluster 2022-04-22T17:25:31.162160+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v318: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:32.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: cluster 2022-04-22T17:25:31.659966+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2022-04-22T17:25:32.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: audit 2022-04-22T17:25:31.928914+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:32.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: cephadm 2022-04-22T17:25:31.929798+0000 mgr.y (mgr.14146) 361 : cephadm [INF] Upgrade: Target is version 17.0.0-11772-g72067459 (unknown) 2022-04-22T17:25:32.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: cephadm 2022-04-22T17:25:31.929997+0000 mgr.y (mgr.14146) 362 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760, digests ['quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760'] 2022-04-22T17:25:32.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: audit 2022-04-22T17:25:31.931883+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:32.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:32 smithi121 bash[12679]: cephadm 2022-04-22T17:25:31.933377+0000 mgr.y (mgr.14146) 363 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:25:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: cluster 2022-04-22T17:25:31.162160+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v318: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: cluster 2022-04-22T17:25:31.659966+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2022-04-22T17:25:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: audit 2022-04-22T17:25:31.928914+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: cephadm 2022-04-22T17:25:31.929798+0000 mgr.y (mgr.14146) 361 : cephadm [INF] Upgrade: Target is version 17.0.0-11772-g72067459 (unknown) 2022-04-22T17:25:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: cephadm 2022-04-22T17:25:31.929997+0000 mgr.y (mgr.14146) 362 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760, digests ['quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760'] 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: audit 2022-04-22T17:25:31.931883+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[12737]: cephadm 2022-04-22T17:25:31.933377+0000 mgr.y (mgr.14146) 363 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: cluster 2022-04-22T17:25:31.162160+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v318: 105 pgs: 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: cluster 2022-04-22T17:25:31.659966+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: audit 2022-04-22T17:25:31.928914+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: cephadm 2022-04-22T17:25:31.929798+0000 mgr.y (mgr.14146) 361 : cephadm [INF] Upgrade: Target is version 17.0.0-11772-g72067459 (unknown) 2022-04-22T17:25:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: cephadm 2022-04-22T17:25:31.929997+0000 mgr.y (mgr.14146) 362 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760, digests ['quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760'] 2022-04-22T17:25:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: audit 2022-04-22T17:25:31.931883+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:25:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:32 smithi115 bash[18070]: cephadm 2022-04-22T17:25:31.933377+0000 mgr.y (mgr.14146) 363 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:25:33.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-04-22T17:25:33.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[18070]: cephadm 2022-04-22T17:25:32.169413+0000 mgr.y (mgr.14146) 364 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760 on smithi121 2022-04-22T17:25:33.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[18070]: audit 2022-04-22T17:25:32.230792+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2022-04-22T17:25:33.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[18070]: cluster 2022-04-22T17:25:32.230893+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2022-04-22T17:25:33.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[12737]: cephadm 2022-04-22T17:25:32.169413+0000 mgr.y (mgr.14146) 364 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760 on smithi121 2022-04-22T17:25:33.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[12737]: audit 2022-04-22T17:25:32.230792+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2022-04-22T17:25:33.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[12737]: cluster 2022-04-22T17:25:32.230893+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2022-04-22T17:25:33.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:33 smithi121 bash[12679]: cephadm 2022-04-22T17:25:32.169413+0000 mgr.y (mgr.14146) 364 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:3a930cc85adbceef689a98c803a50f144a0e5601d5969b12e61bca28a63a5760 on smithi121 2022-04-22T17:25:33.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:33 smithi121 bash[12679]: audit 2022-04-22T17:25:32.230792+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16, "pgp_num": 16, "pool_type": "erasure", "erasure_code_profile": "teuthologyprofile"}]': finished 2022-04-22T17:25:33.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:33 smithi121 bash[12679]: cluster 2022-04-22T17:25:32.230893+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2022-04-22T17:25:33.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:33.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:33.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:33.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:25:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:34 smithi121 bash[12679]: cluster 2022-04-22T17:25:33.167804+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v321: 121 pgs: 1 creating+peering, 15 unknown, 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:34 smithi121 bash[12679]: cluster 2022-04-22T17:25:33.239728+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2022-04-22T17:25:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:34 smithi115 bash[12737]: cluster 2022-04-22T17:25:33.167804+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v321: 121 pgs: 1 creating+peering, 15 unknown, 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:34 smithi115 bash[12737]: cluster 2022-04-22T17:25:33.239728+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2022-04-22T17:25:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:34 smithi115 bash[18070]: cluster 2022-04-22T17:25:33.167804+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v321: 121 pgs: 1 creating+peering, 15 unknown, 105 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:34 smithi115 bash[18070]: cluster 2022-04-22T17:25:33.239728+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2022-04-22T17:25:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:35 smithi121 bash[12679]: cluster 2022-04-22T17:25:34.247363+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2022-04-22T17:25:35.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:35 smithi115 bash[12737]: cluster 2022-04-22T17:25:34.247363+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2022-04-22T17:25:35.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:35 smithi115 bash[18070]: cluster 2022-04-22T17:25:34.247363+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2022-04-22T17:25:35.923 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:25:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:37 smithi121 bash[12679]: cluster 2022-04-22T17:25:35.168459+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v324: 121 pgs: 4 creating+peering, 4 unknown, 113 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:37.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:37 smithi115 bash[18070]: cluster 2022-04-22T17:25:35.168459+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v324: 121 pgs: 4 creating+peering, 4 unknown, 113 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:37.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:37 smithi115 bash[12737]: cluster 2022-04-22T17:25:35.168459+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v324: 121 pgs: 4 creating+peering, 4 unknown, 113 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:39.423 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[12737]: cluster 2022-04-22T17:25:37.169218+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v325: 121 pgs: 4 creating+peering, 2 unknown, 115 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:39.423 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[12737]: audit 2022-04-22T17:25:38.651728+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:25:39.423 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[12737]: audit 2022-04-22T17:25:38.655032+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:25:39.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[18070]: cluster 2022-04-22T17:25:37.169218+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v325: 121 pgs: 4 creating+peering, 2 unknown, 115 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:39.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[18070]: audit 2022-04-22T17:25:38.651728+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:25:39.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:39 smithi115 bash[18070]: audit 2022-04-22T17:25:38.655032+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:25:39.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:39 smithi121 bash[12679]: cluster 2022-04-22T17:25:37.169218+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v325: 121 pgs: 4 creating+peering, 2 unknown, 115 active+clean; 2.3 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:39.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:39 smithi121 bash[12679]: audit 2022-04-22T17:25:38.651728+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:25:39.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:39 smithi121 bash[12679]: audit 2022-04-22T17:25:38.655032+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:25:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-04-22T17:25:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:40 smithi121 bash[12679]: audit 2022-04-22T17:25:39.775855+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/602734522' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:40.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:40 smithi121 bash[12679]: audit 2022-04-22T17:25:39.776617+0000 mon.a (mon.0) 741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:40 smithi115 bash[18070]: audit 2022-04-22T17:25:39.775855+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/602734522' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:40 smithi115 bash[18070]: audit 2022-04-22T17:25:39.776617+0000 mon.a (mon.0) 741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:40 smithi115 bash[12737]: audit 2022-04-22T17:25:39.775855+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/602734522' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:40 smithi115 bash[12737]: audit 2022-04-22T17:25:39.776617+0000 mon.a (mon.0) 741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:25:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:41 smithi121 bash[12679]: cluster 2022-04-22T17:25:39.170042+0000 mgr.y (mgr.14146) 368 : cluster [DBG] pgmap v326: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:41.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:41 smithi121 bash[12679]: audit 2022-04-22T17:25:40.163253+0000 mon.a (mon.0) 742 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:25:41.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:41 smithi121 bash[12679]: cluster 2022-04-22T17:25:40.163388+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2022-04-22T17:25:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[12737]: cluster 2022-04-22T17:25:39.170042+0000 mgr.y (mgr.14146) 368 : cluster [DBG] pgmap v326: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[12737]: audit 2022-04-22T17:25:40.163253+0000 mon.a (mon.0) 742 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:25:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[12737]: cluster 2022-04-22T17:25:40.163388+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2022-04-22T17:25:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[18070]: cluster 2022-04-22T17:25:39.170042+0000 mgr.y (mgr.14146) 368 : cluster [DBG] pgmap v326: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[18070]: audit 2022-04-22T17:25:40.163253+0000 mon.a (mon.0) 742 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:25:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:41 smithi115 bash[18070]: cluster 2022-04-22T17:25:40.163388+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2022-04-22T17:25:42.091 DEBUG:teuthology.orchestra.run.smithi115:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2022-04-22T17:25:42.110 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight read -> 100 2022-04-22T17:25:42.110 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight delete -> 50 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight snap_create -> 50 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight snap_remove -> 50 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight rollback -> 50 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight setattr -> 25 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight rmattr -> 25 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight copy_from -> 50 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:adding op weight append -> 100 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:ceph version 16.2.7-924-gb42bd433 (b42bd433b8c85da129c1c277665f5ddab82dd5fb) pacific (stable) 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout:Configuration: 2022-04-22T17:25:42.111 INFO:tasks.rados.rados.0.smithi115.stdout: Number of operations: 4000 2022-04-22T17:25:42.112 INFO:tasks.rados.rados.0.smithi115.stdout: Number of objects: 50 2022-04-22T17:25:42.112 INFO:tasks.rados.rados.0.smithi115.stdout: Max in flight operations: 16 2022-04-22T17:25:42.112 INFO:tasks.rados.rados.0.smithi115.stdout: Object size (in bytes): 4000000 2022-04-22T17:25:42.112 INFO:tasks.rados.rados.0.smithi115.stdout: Write stride min: 400000 2022-04-22T17:25:42.112 INFO:tasks.rados.rados.0.smithi115.stdout: Write stride max: 800000 2022-04-22T17:25:42.180 INFO:tasks.rados.rados.0.smithi115.stdout:1: write initial oid 1 2022-04-22T17:25:42.180 INFO:tasks.rados.rados.0.smithi115.stdout:1: seq_num 0 ranges {0=24576} 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:1: writing smithi11541303-1 from 0 to 24576 tid 1 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:2: write initial oid 2 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:2: seq_num 1 ranges {0=24576} 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:2: writing smithi11541303-2 from 0 to 24576 tid 1 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:3: write initial oid 3 2022-04-22T17:25:42.181 INFO:tasks.rados.rados.0.smithi115.stdout:3: seq_num 2 ranges {0=40960} 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:3: writing smithi11541303-3 from 0 to 40960 tid 1 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:4: write initial oid 4 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:4: seq_num 3 ranges {0=57344} 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:4: writing smithi11541303-4 from 0 to 57344 tid 1 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:5: write initial oid 5 2022-04-22T17:25:42.182 INFO:tasks.rados.rados.0.smithi115.stdout:5: seq_num 4 ranges {0=73728} 2022-04-22T17:25:42.183 INFO:tasks.rados.rados.0.smithi115.stdout:5: writing smithi11541303-5 from 0 to 73728 tid 1 2022-04-22T17:25:42.183 INFO:tasks.rados.rados.0.smithi115.stdout:6: write initial oid 6 2022-04-22T17:25:42.183 INFO:tasks.rados.rados.0.smithi115.stdout:6: seq_num 5 ranges {0=90112} 2022-04-22T17:25:42.184 INFO:tasks.rados.rados.0.smithi115.stdout:6: writing smithi11541303-6 from 0 to 90112 tid 1 2022-04-22T17:25:42.184 INFO:tasks.rados.rados.0.smithi115.stdout:7: write initial oid 7 2022-04-22T17:25:42.184 INFO:tasks.rados.rados.0.smithi115.stdout:7: seq_num 6 ranges {0=106496} 2022-04-22T17:25:42.185 INFO:tasks.rados.rados.0.smithi115.stdout:7: writing smithi11541303-7 from 0 to 106496 tid 1 2022-04-22T17:25:42.185 INFO:tasks.rados.rados.0.smithi115.stdout:8: write initial oid 8 2022-04-22T17:25:42.186 INFO:tasks.rados.rados.0.smithi115.stdout:8: seq_num 7 ranges {0=122880} 2022-04-22T17:25:42.187 INFO:tasks.rados.rados.0.smithi115.stdout:8: writing smithi11541303-8 from 0 to 122880 tid 1 2022-04-22T17:25:42.187 INFO:tasks.rados.rados.0.smithi115.stdout:9: write initial oid 9 2022-04-22T17:25:42.187 INFO:tasks.rados.rados.0.smithi115.stdout:9: seq_num 8 ranges {0=139264} 2022-04-22T17:25:42.188 INFO:tasks.rados.rados.0.smithi115.stdout:9: writing smithi11541303-9 from 0 to 139264 tid 1 2022-04-22T17:25:42.188 INFO:tasks.rados.rados.0.smithi115.stdout:10: write initial oid 10 2022-04-22T17:25:42.188 INFO:tasks.rados.rados.0.smithi115.stdout:10: seq_num 9 ranges {0=155648} 2022-04-22T17:25:42.189 INFO:tasks.rados.rados.0.smithi115.stdout:10: writing smithi11541303-10 from 0 to 155648 tid 1 2022-04-22T17:25:42.190 INFO:tasks.rados.rados.0.smithi115.stdout:11: write initial oid 11 2022-04-22T17:25:42.190 INFO:tasks.rados.rados.0.smithi115.stdout:11: seq_num 10 ranges {0=172032} 2022-04-22T17:25:42.191 INFO:tasks.rados.rados.0.smithi115.stdout:11: writing smithi11541303-11 from 0 to 172032 tid 1 2022-04-22T17:25:42.192 INFO:tasks.rados.rados.0.smithi115.stdout:4: finishing write tid 1 to smithi11541303-4 2022-04-22T17:25:42.192 INFO:tasks.rados.rados.0.smithi115.stdout:12: write initial oid 12 2022-04-22T17:25:42.192 INFO:tasks.rados.rados.0.smithi115.stdout:12: seq_num 11 ranges {0=188416} 2022-04-22T17:25:42.194 INFO:tasks.rados.rados.0.smithi115.stdout:12: writing smithi11541303-12 from 0 to 188416 tid 1 2022-04-22T17:25:42.194 INFO:tasks.rados.rados.0.smithi115.stdout:1: finishing write tid 1 to smithi11541303-1 2022-04-22T17:25:42.194 INFO:tasks.rados.rados.0.smithi115.stdout:2: finishing write tid 1 to smithi11541303-2 2022-04-22T17:25:42.195 INFO:tasks.rados.rados.0.smithi115.stdout:3: finishing write tid 1 to smithi11541303-3 2022-04-22T17:25:42.195 INFO:tasks.rados.rados.0.smithi115.stdout:10: finishing write tid 1 to smithi11541303-10 2022-04-22T17:25:42.195 INFO:tasks.rados.rados.0.smithi115.stdout:4: finishing write tid 2 to smithi11541303-4 2022-04-22T17:25:42.195 INFO:tasks.rados.rados.0.smithi115.stdout:13: write initial oid 13 2022-04-22T17:25:42.195 INFO:tasks.rados.rados.0.smithi115.stdout:13: seq_num 12 ranges {0=204800} 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:13: writing smithi11541303-13 from 0 to 204800 tid 1 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:2: finishing write tid 2 to smithi11541303-2 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:7: finishing write tid 1 to smithi11541303-7 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:4: finishing write tid 3 to smithi11541303-4 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2022-04-22T17:25:42.196 INFO:tasks.rados.rados.0.smithi115.stdout:3: finishing write tid 2 to smithi11541303-3 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:1: finishing write tid 2 to smithi11541303-1 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:11: finishing write tid 1 to smithi11541303-11 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:8: finishing write tid 1 to smithi11541303-8 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:2: finishing write tid 3 to smithi11541303-2 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:2: done (12 left) 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:4: done (11 left) 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:14: write initial oid 14 2022-04-22T17:25:42.197 INFO:tasks.rados.rados.0.smithi115.stdout:14: seq_num 13 ranges {0=221184} 2022-04-22T17:25:42.198 INFO:tasks.rados.rados.0.smithi115.stdout:14: writing smithi11541303-14 from 0 to 221184 tid 1 2022-04-22T17:25:42.198 INFO:tasks.rados.rados.0.smithi115.stdout:10: finishing write tid 2 to smithi11541303-10 2022-04-22T17:25:42.198 INFO:tasks.rados.rados.0.smithi115.stdout:7: finishing write tid 2 to smithi11541303-7 2022-04-22T17:25:42.198 INFO:tasks.rados.rados.0.smithi115.stdout:6: finishing write tid 1 to smithi11541303-6 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:1: finishing write tid 3 to smithi11541303-1 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:3: finishing write tid 3 to smithi11541303-3 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:5: finishing write tid 1 to smithi11541303-5 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:6: finishing write tid 2 to smithi11541303-6 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:10: finishing write tid 3 to smithi11541303-10 2022-04-22T17:25:42.199 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:1: done (11 left) 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:3: done (10 left) 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:10: done (9 left) 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:15: write initial oid 15 2022-04-22T17:25:42.200 INFO:tasks.rados.rados.0.smithi115.stdout:15: seq_num 14 ranges {0=237568} 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:15: writing smithi11541303-15 from 0 to 237568 tid 1 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:9: finishing write tid 1 to smithi11541303-9 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:11: finishing write tid 2 to smithi11541303-11 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:6: finishing write tid 3 to smithi11541303-6 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2022-04-22T17:25:42.201 INFO:tasks.rados.rados.0.smithi115.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2022-04-22T17:25:42.202 INFO:tasks.rados.rados.0.smithi115.stdout:8: finishing write tid 2 to smithi11541303-8 2022-04-22T17:25:42.202 INFO:tasks.rados.rados.0.smithi115.stdout:6: done (9 left) 2022-04-22T17:25:42.202 INFO:tasks.rados.rados.0.smithi115.stdout:16: write initial oid 16 2022-04-22T17:25:42.202 INFO:tasks.rados.rados.0.smithi115.stdout:16: seq_num 15 ranges {0=253952} 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:16: writing smithi11541303-16 from 0 to 253952 tid 1 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:11: finishing write tid 3 to smithi11541303-11 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:13: finishing write tid 1 to smithi11541303-13 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:12: finishing write tid 1 to smithi11541303-12 2022-04-22T17:25:42.203 INFO:tasks.rados.rados.0.smithi115.stdout:7: finishing write tid 3 to smithi11541303-7 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 5 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:8: finishing write tid 3 to smithi11541303-8 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:5: finishing write tid 2 to smithi11541303-5 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:9: finishing write tid 2 to smithi11541303-9 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:12: finishing write tid 2 to smithi11541303-12 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:7: done (9 left) 2022-04-22T17:25:42.204 INFO:tasks.rados.rados.0.smithi115.stdout:8: done (8 left) 2022-04-22T17:25:42.205 INFO:tasks.rados.rados.0.smithi115.stdout:11: done (7 left) 2022-04-22T17:25:42.205 INFO:tasks.rados.rados.0.smithi115.stdout:17: write initial oid 17 2022-04-22T17:25:42.205 INFO:tasks.rados.rados.0.smithi115.stdout:17: seq_num 16 ranges {0=270336} 2022-04-22T17:25:42.205 INFO:tasks.rados.rados.0.smithi115.stdout:17: writing smithi11541303-17 from 0 to 270336 tid 1 2022-04-22T17:25:42.206 INFO:tasks.rados.rados.0.smithi115.stdout:18: write initial oid 18 2022-04-22T17:25:42.206 INFO:tasks.rados.rados.0.smithi115.stdout:14: finishing write tid 1 to smithi11541303-14 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:13: finishing write tid 2 to smithi11541303-13 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:5: finishing write tid 3 to smithi11541303-5 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:12: finishing write tid 3 to smithi11541303-12 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:15: finishing write tid 1 to smithi11541303-15 2022-04-22T17:25:42.207 INFO:tasks.rados.rados.0.smithi115.stdout:18: seq_num 17 ranges {0=286720} 2022-04-22T17:25:42.208 INFO:tasks.rados.rados.0.smithi115.stdout:18: writing smithi11541303-18 from 0 to 286720 tid 1 2022-04-22T17:25:42.209 INFO:tasks.rados.rados.0.smithi115.stdout:5: done (8 left) 2022-04-22T17:25:42.209 INFO:tasks.rados.rados.0.smithi115.stdout:12: done (7 left) 2022-04-22T17:25:42.209 INFO:tasks.rados.rados.0.smithi115.stdout:19: write initial oid 19 2022-04-22T17:25:42.209 INFO:tasks.rados.rados.0.smithi115.stdout:19: seq_num 18 ranges {0=303104} 2022-04-22T17:25:42.211 INFO:tasks.rados.rados.0.smithi115.stdout:19: writing smithi11541303-19 from 0 to 303104 tid 1 2022-04-22T17:25:42.211 INFO:tasks.rados.rados.0.smithi115.stdout:20: write initial oid 20 2022-04-22T17:25:42.211 INFO:tasks.rados.rados.0.smithi115.stdout:20: seq_num 19 ranges {0=319488} 2022-04-22T17:25:42.212 INFO:tasks.rados.rados.0.smithi115.stdout:20: writing smithi11541303-20 from 0 to 319488 tid 1 2022-04-22T17:25:42.213 INFO:tasks.rados.rados.0.smithi115.stdout:21: write initial oid 21 2022-04-22T17:25:42.213 INFO:tasks.rados.rados.0.smithi115.stdout:21: seq_num 20 ranges {0=344064} 2022-04-22T17:25:42.216 INFO:tasks.rados.rados.0.smithi115.stdout:21: writing smithi11541303-21 from 0 to 344064 tid 1 2022-04-22T17:25:42.216 INFO:tasks.rados.rados.0.smithi115.stdout:22: write initial oid 22 2022-04-22T17:25:42.216 INFO:tasks.rados.rados.0.smithi115.stdout:22: seq_num 21 ranges {0=360448} 2022-04-22T17:25:42.218 INFO:tasks.rados.rados.0.smithi115.stdout:22: writing smithi11541303-22 from 0 to 360448 tid 1 2022-04-22T17:25:42.218 INFO:tasks.rados.rados.0.smithi115.stdout:23: write initial oid 23 2022-04-22T17:25:42.218 INFO:tasks.rados.rados.0.smithi115.stdout:23: seq_num 22 ranges {0=376832} 2022-04-22T17:25:42.220 INFO:tasks.rados.rados.0.smithi115.stdout:23: writing smithi11541303-23 from 0 to 376832 tid 1 2022-04-22T17:25:42.220 INFO:tasks.rados.rados.0.smithi115.stdout:24: write initial oid 24 2022-04-22T17:25:42.221 INFO:tasks.rados.rados.0.smithi115.stdout:24: seq_num 23 ranges {0=393216} 2022-04-22T17:25:42.223 INFO:tasks.rados.rados.0.smithi115.stdout:24: writing smithi11541303-24 from 0 to 393216 tid 1 2022-04-22T17:25:42.223 INFO:tasks.rados.rados.0.smithi115.stdout:25: write initial oid 25 2022-04-22T17:25:42.223 INFO:tasks.rados.rados.0.smithi115.stdout:25: seq_num 24 ranges {0=409600} 2022-04-22T17:25:42.226 INFO:tasks.rados.rados.0.smithi115.stdout:25: writing smithi11541303-25 from 0 to 409600 tid 1 2022-04-22T17:25:42.226 INFO:tasks.rados.rados.0.smithi115.stdout:26: write initial oid 26 2022-04-22T17:25:42.226 INFO:tasks.rados.rados.0.smithi115.stdout:26: seq_num 25 ranges {0=425984} 2022-04-22T17:25:42.228 INFO:tasks.rados.rados.0.smithi115.stdout:26: writing smithi11541303-26 from 0 to 425984 tid 1 2022-04-22T17:25:42.228 INFO:tasks.rados.rados.0.smithi115.stdout:27: write initial oid 27 2022-04-22T17:25:42.228 INFO:tasks.rados.rados.0.smithi115.stdout:27: seq_num 26 ranges {0=442368} 2022-04-22T17:25:42.231 INFO:tasks.rados.rados.0.smithi115.stdout:27: writing smithi11541303-27 from 0 to 442368 tid 1 2022-04-22T17:25:42.231 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:15: finishing write tid 2 to smithi11541303-15 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:14: finishing write tid 2 to smithi11541303-14 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:13: finishing write tid 3 to smithi11541303-13 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:16: finishing write tid 1 to smithi11541303-16 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:13: done (15 left) 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:28: write initial oid 28 2022-04-22T17:25:42.232 INFO:tasks.rados.rados.0.smithi115.stdout:28: seq_num 27 ranges {0=458752} 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:28: writing smithi11541303-28 from 0 to 458752 tid 1 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:15: finishing write tid 3 to smithi11541303-15 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2022-04-22T17:25:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:15: done (15 left) 2022-04-22T17:25:42.235 INFO:tasks.rados.rados.0.smithi115.stdout:29: write initial oid 29 2022-04-22T17:25:42.235 INFO:tasks.rados.rados.0.smithi115.stdout:29: seq_num 28 ranges {0=475136} 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:29: writing smithi11541303-29 from 0 to 475136 tid 1 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:14: finishing write tid 3 to smithi11541303-14 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:9: finishing write tid 3 to smithi11541303-9 2022-04-22T17:25:42.238 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:17: finishing write tid 1 to smithi11541303-17 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:16: finishing write tid 2 to smithi11541303-16 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:16: finishing write tid 3 to smithi11541303-16 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:19: finishing write tid 1 to smithi11541303-19 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:18: finishing write tid 1 to smithi11541303-18 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:17: finishing write tid 2 to smithi11541303-17 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:19: finishing write tid 2 to smithi11541303-19 2022-04-22T17:25:42.239 INFO:tasks.rados.rados.0.smithi115.stdout:20: finishing write tid 1 to smithi11541303-20 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:18: finishing write tid 2 to smithi11541303-18 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:17: finishing write tid 3 to smithi11541303-17 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:20: finishing write tid 2 to smithi11541303-20 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:19: finishing write tid 3 to smithi11541303-19 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:18: finishing write tid 3 to smithi11541303-18 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2022-04-22T17:25:42.240 INFO:tasks.rados.rados.0.smithi115.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:20: finishing write tid 3 to smithi11541303-20 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:21: finishing write tid 1 to smithi11541303-21 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:22: finishing write tid 1 to smithi11541303-22 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:9: done (15 left) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:14: done (14 left) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:16: done (13 left) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:17: done (12 left) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:18: done (11 left) 2022-04-22T17:25:42.241 INFO:tasks.rados.rados.0.smithi115.stdout:19: done (10 left) 2022-04-22T17:25:42.242 INFO:tasks.rados.rados.0.smithi115.stdout:20: done (9 left) 2022-04-22T17:25:42.242 INFO:tasks.rados.rados.0.smithi115.stdout:30: write initial oid 30 2022-04-22T17:25:42.242 INFO:tasks.rados.rados.0.smithi115.stdout:30: seq_num 29 ranges {0=491520} 2022-04-22T17:25:42.242 INFO:tasks.rados.rados.0.smithi115.stdout:30: writing smithi11541303-30 from 0 to 491520 tid 1 2022-04-22T17:25:42.242 INFO:tasks.rados.rados.0.smithi115.stdout:31: write initial oid 31 2022-04-22T17:25:42.243 INFO:tasks.rados.rados.0.smithi115.stdout:31: seq_num 30 ranges {0=507904} 2022-04-22T17:25:42.244 INFO:tasks.rados.rados.0.smithi115.stdout:31: writing smithi11541303-31 from 0 to 507904 tid 1 2022-04-22T17:25:42.244 INFO:tasks.rados.rados.0.smithi115.stdout:32: write initial oid 32 2022-04-22T17:25:42.244 INFO:tasks.rados.rados.0.smithi115.stdout:32: seq_num 31 ranges {0=524288} 2022-04-22T17:25:42.247 INFO:tasks.rados.rados.0.smithi115.stdout:32: writing smithi11541303-32 from 0 to 524288 tid 1 2022-04-22T17:25:42.248 INFO:tasks.rados.rados.0.smithi115.stdout:33: write initial oid 33 2022-04-22T17:25:42.248 INFO:tasks.rados.rados.0.smithi115.stdout:33: seq_num 32 ranges {0=540672} 2022-04-22T17:25:42.251 INFO:tasks.rados.rados.0.smithi115.stdout:33: writing smithi11541303-33 from 0 to 540672 tid 1 2022-04-22T17:25:42.251 INFO:tasks.rados.rados.0.smithi115.stdout:34: write initial oid 34 2022-04-22T17:25:42.251 INFO:tasks.rados.rados.0.smithi115.stdout:34: seq_num 33 ranges {0=557056} 2022-04-22T17:25:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:34: writing smithi11541303-34 from 0 to 557056 tid 1 2022-04-22T17:25:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:35: write initial oid 35 2022-04-22T17:25:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:21: finishing write tid 2 to smithi11541303-21 2022-04-22T17:25:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:35: seq_num 34 ranges {0=573440} 2022-04-22T17:25:42.258 INFO:tasks.rados.rados.0.smithi115.stdout:35: writing smithi11541303-35 from 0 to 573440 tid 1 2022-04-22T17:25:42.258 INFO:tasks.rados.rados.0.smithi115.stdout:36: write initial oid 36 2022-04-22T17:25:42.258 INFO:tasks.rados.rados.0.smithi115.stdout:36: seq_num 35 ranges {0=589824} 2022-04-22T17:25:42.263 INFO:tasks.rados.rados.0.smithi115.stdout:36: writing smithi11541303-36 from 0 to 589824 tid 1 2022-04-22T17:25:42.263 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.263 INFO:tasks.rados.rados.0.smithi115.stdout:23: finishing write tid 1 to smithi11541303-23 2022-04-22T17:25:42.263 INFO:tasks.rados.rados.0.smithi115.stdout:22: finishing write tid 2 to smithi11541303-22 2022-04-22T17:25:42.263 INFO:tasks.rados.rados.0.smithi115.stdout:23: finishing write tid 2 to smithi11541303-23 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:21: finishing write tid 3 to smithi11541303-21 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:22: finishing write tid 3 to smithi11541303-22 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:23: finishing write tid 3 to smithi11541303-23 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2022-04-22T17:25:42.264 INFO:tasks.rados.rados.0.smithi115.stdout:24: finishing write tid 1 to smithi11541303-24 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:25: finishing write tid 1 to smithi11541303-25 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:21: done (15 left) 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:22: done (14 left) 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:23: done (13 left) 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:37: write initial oid 37 2022-04-22T17:25:42.265 INFO:tasks.rados.rados.0.smithi115.stdout:37: seq_num 36 ranges {0=606208} 2022-04-22T17:25:42.266 INFO:tasks.rados.rados.0.smithi115.stdout:37: writing smithi11541303-37 from 0 to 606208 tid 1 2022-04-22T17:25:42.267 INFO:tasks.rados.rados.0.smithi115.stdout:38: write initial oid 38 2022-04-22T17:25:42.267 INFO:tasks.rados.rados.0.smithi115.stdout:38: seq_num 37 ranges {0=622592} 2022-04-22T17:25:42.271 INFO:tasks.rados.rados.0.smithi115.stdout:38: writing smithi11541303-38 from 0 to 622592 tid 1 2022-04-22T17:25:42.271 INFO:tasks.rados.rados.0.smithi115.stdout:39: write initial oid 39 2022-04-22T17:25:42.271 INFO:tasks.rados.rados.0.smithi115.stdout:39: seq_num 38 ranges {0=638976} 2022-04-22T17:25:42.276 INFO:tasks.rados.rados.0.smithi115.stdout:39: writing smithi11541303-39 from 0 to 638976 tid 1 2022-04-22T17:25:42.276 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.276 INFO:tasks.rados.rados.0.smithi115.stdout:24: finishing write tid 2 to smithi11541303-24 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:25: finishing write tid 2 to smithi11541303-25 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:26: finishing write tid 1 to smithi11541303-26 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:24: finishing write tid 3 to smithi11541303-24 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:25: finishing write tid 3 to smithi11541303-25 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:26: finishing write tid 2 to smithi11541303-26 2022-04-22T17:25:42.277 INFO:tasks.rados.rados.0.smithi115.stdout:26: finishing write tid 3 to smithi11541303-26 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:27: finishing write tid 1 to smithi11541303-27 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:24: done (15 left) 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:25: done (14 left) 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:26: done (13 left) 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:40: write initial oid 40 2022-04-22T17:25:42.278 INFO:tasks.rados.rados.0.smithi115.stdout:40: seq_num 39 ranges {0=663552} 2022-04-22T17:25:42.281 INFO:tasks.rados.rados.0.smithi115.stdout:40: writing smithi11541303-40 from 0 to 663552 tid 1 2022-04-22T17:25:42.281 INFO:tasks.rados.rados.0.smithi115.stdout:41: write initial oid 41 2022-04-22T17:25:42.281 INFO:tasks.rados.rados.0.smithi115.stdout:41: seq_num 40 ranges {0=679936} 2022-04-22T17:25:42.285 INFO:tasks.rados.rados.0.smithi115.stdout:41: writing smithi11541303-41 from 0 to 679936 tid 1 2022-04-22T17:25:42.285 INFO:tasks.rados.rados.0.smithi115.stdout:42: write initial oid 42 2022-04-22T17:25:42.285 INFO:tasks.rados.rados.0.smithi115.stdout:42: seq_num 41 ranges {0=696320} 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:42: writing smithi11541303-42 from 0 to 696320 tid 1 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:28: finishing write tid 1 to smithi11541303-28 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:27: finishing write tid 2 to smithi11541303-27 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:27: finishing write tid 3 to smithi11541303-27 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:25:42.290 INFO:tasks.rados.rados.0.smithi115.stdout:28: finishing write tid 2 to smithi11541303-28 2022-04-22T17:25:42.291 INFO:tasks.rados.rados.0.smithi115.stdout:29: finishing write tid 1 to smithi11541303-29 2022-04-22T17:25:42.291 INFO:tasks.rados.rados.0.smithi115.stdout:27: done (15 left) 2022-04-22T17:25:42.291 INFO:tasks.rados.rados.0.smithi115.stdout:43: write initial oid 43 2022-04-22T17:25:42.291 INFO:tasks.rados.rados.0.smithi115.stdout:43: seq_num 42 ranges {0=712704} 2022-04-22T17:25:42.294 INFO:tasks.rados.rados.0.smithi115.stdout:43: writing smithi11541303-43 from 0 to 712704 tid 1 2022-04-22T17:25:42.294 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.294 INFO:tasks.rados.rados.0.smithi115.stdout:28: finishing write tid 3 to smithi11541303-28 2022-04-22T17:25:42.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2022-04-22T17:25:42.295 INFO:tasks.rados.rados.0.smithi115.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2022-04-22T17:25:42.295 INFO:tasks.rados.rados.0.smithi115.stdout:28: done (15 left) 2022-04-22T17:25:42.295 INFO:tasks.rados.rados.0.smithi115.stdout:44: write initial oid 44 2022-04-22T17:25:42.295 INFO:tasks.rados.rados.0.smithi115.stdout:44: seq_num 43 ranges {0=729088} 2022-04-22T17:25:42.300 INFO:tasks.rados.rados.0.smithi115.stdout:44: writing smithi11541303-44 from 0 to 729088 tid 1 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:29: finishing write tid 2 to smithi11541303-29 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:30: finishing write tid 1 to smithi11541303-30 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:29: finishing write tid 3 to smithi11541303-29 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:30: finishing write tid 2 to smithi11541303-30 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:29: done (15 left) 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:45: write initial oid 45 2022-04-22T17:25:42.301 INFO:tasks.rados.rados.0.smithi115.stdout:45: seq_num 44 ranges {0=745472} 2022-04-22T17:25:42.307 INFO:tasks.rados.rados.0.smithi115.stdout:45: writing smithi11541303-45 from 0 to 745472 tid 1 2022-04-22T17:25:42.307 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:25:42.308 INFO:tasks.rados.rados.0.smithi115.stdout:31: finishing write tid 1 to smithi11541303-31 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:30: finishing write tid 3 to smithi11541303-30 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:31: finishing write tid 2 to smithi11541303-31 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:31: finishing write tid 3 to smithi11541303-31 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:32: finishing write tid 1 to smithi11541303-32 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:32: finishing write tid 2 to smithi11541303-32 2022-04-22T17:25:42.309 INFO:tasks.rados.rados.0.smithi115.stdout:33: finishing write tid 1 to smithi11541303-33 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:32: finishing write tid 3 to smithi11541303-32 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:33: finishing write tid 2 to smithi11541303-33 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:34: finishing write tid 1 to smithi11541303-34 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:33: finishing write tid 3 to smithi11541303-33 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:34: finishing write tid 2 to smithi11541303-34 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:34: finishing write tid 3 to smithi11541303-34 2022-04-22T17:25:42.310 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:35: finishing write tid 1 to smithi11541303-35 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:35: finishing write tid 2 to smithi11541303-35 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:35: finishing write tid 3 to smithi11541303-35 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:30: done (15 left) 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:31: done (14 left) 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:32: done (13 left) 2022-04-22T17:25:42.311 INFO:tasks.rados.rados.0.smithi115.stdout:33: done (12 left) 2022-04-22T17:25:42.312 INFO:tasks.rados.rados.0.smithi115.stdout:34: done (11 left) 2022-04-22T17:25:42.312 INFO:tasks.rados.rados.0.smithi115.stdout:35: done (10 left) 2022-04-22T17:25:42.312 INFO:tasks.rados.rados.0.smithi115.stdout:46: write initial oid 46 2022-04-22T17:25:42.312 INFO:tasks.rados.rados.0.smithi115.stdout:46: seq_num 45 ranges {0=761856} 2022-04-22T17:25:42.315 INFO:tasks.rados.rados.0.smithi115.stdout:46: writing smithi11541303-46 from 0 to 761856 tid 1 2022-04-22T17:25:42.315 INFO:tasks.rados.rados.0.smithi115.stdout:47: write initial oid 47 2022-04-22T17:25:42.315 INFO:tasks.rados.rados.0.smithi115.stdout:47: seq_num 46 ranges {0=778240} 2022-04-22T17:25:42.321 INFO:tasks.rados.rados.0.smithi115.stdout:47: writing smithi11541303-47 from 0 to 778240 tid 1 2022-04-22T17:25:42.321 INFO:tasks.rados.rados.0.smithi115.stdout:48: write initial oid 48 2022-04-22T17:25:42.321 INFO:tasks.rados.rados.0.smithi115.stdout:48: seq_num 47 ranges {0=794624} 2022-04-22T17:25:42.326 INFO:tasks.rados.rados.0.smithi115.stdout:48: writing smithi11541303-48 from 0 to 794624 tid 1 2022-04-22T17:25:42.326 INFO:tasks.rados.rados.0.smithi115.stdout:49: write initial oid 49 2022-04-22T17:25:42.326 INFO:tasks.rados.rados.0.smithi115.stdout:49: seq_num 48 ranges {0=409600,409600=401408} 2022-04-22T17:25:42.328 INFO:tasks.rados.rados.0.smithi115.stdout:49: writing smithi11541303-49 from 0 to 409600 tid 1 2022-04-22T17:25:42.331 INFO:tasks.rados.rados.0.smithi115.stdout:49: writing smithi11541303-49 from 409600 to 811008 tid 2 2022-04-22T17:25:42.331 INFO:tasks.rados.rados.0.smithi115.stdout:50: write initial oid 50 2022-04-22T17:25:42.331 INFO:tasks.rados.rados.0.smithi115.stdout:50: seq_num 49 ranges {0=425984,425984=401408} 2022-04-22T17:25:42.333 INFO:tasks.rados.rados.0.smithi115.stdout:50: writing smithi11541303-50 from 0 to 425984 tid 1 2022-04-22T17:25:42.337 INFO:tasks.rados.rados.0.smithi115.stdout:50: writing smithi11541303-50 from 425984 to 827392 tid 2 2022-04-22T17:25:42.337 INFO:tasks.rados.rados.0.smithi115.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2022-04-22T17:25:42.337 INFO:tasks.rados.rados.0.smithi115.stdout:36: finishing write tid 1 to smithi11541303-36 2022-04-22T17:25:42.337 INFO:tasks.rados.rados.0.smithi115.stdout:36: finishing write tid 2 to smithi11541303-36 2022-04-22T17:25:42.337 INFO:tasks.rados.rados.0.smithi115.stdout:36: finishing write tid 3 to smithi11541303-36 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:37: finishing write tid 1 to smithi11541303-37 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:37: finishing write tid 2 to smithi11541303-37 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:37: finishing write tid 3 to smithi11541303-37 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:38: finishing write tid 1 to smithi11541303-38 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:38: finishing write tid 2 to smithi11541303-38 2022-04-22T17:25:42.338 INFO:tasks.rados.rados.0.smithi115.stdout:36: done (15 left) 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:37: done (14 left) 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 0 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:38: finishing write tid 3 to smithi11541303-38 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:25:42.339 INFO:tasks.rados.rados.0.smithi115.stdout:52: seq_num 50 ranges {286720=442368,729088=401408} 2022-04-22T17:25:42.341 INFO:tasks.rados.rados.0.smithi115.stdout:52: writing smithi11541303-18 from 286720 to 729088 tid 1 2022-04-22T17:25:42.345 INFO:tasks.rados.rados.0.smithi115.stdout:52: writing smithi11541303-18 from 729088 to 1130496 tid 2 2022-04-22T17:25:42.345 INFO:tasks.rados.rados.0.smithi115.stdout:38: done (14 left) 2022-04-22T17:25:42.345 INFO:tasks.rados.rados.0.smithi115.stdout:53: rmattr oid 6 current snap is 0 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:53: done (14 left) 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:54: delete oid 6 current snap is 0 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:39: finishing write tid 1 to smithi11541303-39 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:39: finishing write tid 2 to smithi11541303-39 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:39: finishing write tid 3 to smithi11541303-39 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:40: finishing write tid 1 to smithi11541303-40 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:40: finishing write tid 2 to smithi11541303-40 2022-04-22T17:25:42.346 INFO:tasks.rados.rados.0.smithi115.stdout:40: finishing write tid 3 to smithi11541303-40 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 4 (ObjNum 39 snap 0 seq_num 39) dirty exists 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:41: finishing write tid 1 to smithi11541303-41 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:41: finishing write tid 2 to smithi11541303-41 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:41: finishing write tid 3 to smithi11541303-41 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:42: finishing write tid 1 to smithi11541303-42 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:42: finishing write tid 2 to smithi11541303-42 2022-04-22T17:25:42.347 INFO:tasks.rados.rados.0.smithi115.stdout:42: finishing write tid 3 to smithi11541303-42 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 10 (ObjNum 41 snap 0 seq_num 41) dirty exists 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:43: finishing write tid 1 to smithi11541303-43 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:43: finishing write tid 2 to smithi11541303-43 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:43: finishing write tid 3 to smithi11541303-43 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:44: finishing write tid 1 to smithi11541303-44 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:44: finishing write tid 2 to smithi11541303-44 2022-04-22T17:25:42.348 INFO:tasks.rados.rados.0.smithi115.stdout:45: finishing write tid 1 to smithi11541303-45 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:44: finishing write tid 3 to smithi11541303-44 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2 (ObjNum 43 snap 0 seq_num 43) dirty exists 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:45: finishing write tid 2 to smithi11541303-45 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:45: finishing write tid 3 to smithi11541303-45 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:46: finishing write tid 1 to smithi11541303-46 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:46: finishing write tid 2 to smithi11541303-46 2022-04-22T17:25:42.349 INFO:tasks.rados.rados.0.smithi115.stdout:46: finishing write tid 3 to smithi11541303-46 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:47: finishing write tid 1 to smithi11541303-47 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:47: finishing write tid 2 to smithi11541303-47 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:47: finishing write tid 3 to smithi11541303-47 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:48: finishing write tid 1 to smithi11541303-48 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:48: finishing write tid 2 to smithi11541303-48 2022-04-22T17:25:42.350 INFO:tasks.rados.rados.0.smithi115.stdout:49: finishing write tid 1 to smithi11541303-49 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:48: finishing write tid 3 to smithi11541303-48 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 10 (ObjNum 47 snap 0 seq_num 47) dirty exists 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:49: finishing write tid 2 to smithi11541303-49 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:49: finishing write tid 3 to smithi11541303-49 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:49: finishing write tid 4 to smithi11541303-49 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 13 (ObjNum 48 snap 0 seq_num 48) dirty exists 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:50: finishing write tid 1 to smithi11541303-50 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:50: finishing write tid 2 to smithi11541303-50 2022-04-22T17:25:42.351 INFO:tasks.rados.rados.0.smithi115.stdout:51: finishing copy_from to smithi11541303-2 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:50: finishing write tid 3 to smithi11541303-50 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:51: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:52: finishing write tid 1 to smithi11541303-18 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:50: finishing write tid 4 to smithi11541303-50 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 7 (ObjNum 49 snap 0 seq_num 49) dirty exists 2022-04-22T17:25:42.352 INFO:tasks.rados.rados.0.smithi115.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:39: done (14 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:40: done (13 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:41: done (12 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:42: done (11 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:43: done (10 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:44: done (9 left) 2022-04-22T17:25:42.354 INFO:tasks.rados.rados.0.smithi115.stdout:45: done (8 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:46: done (7 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:47: done (6 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:48: done (5 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:49: done (4 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:50: done (3 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:51: done (2 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:54: done (1 left) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:55: read oid 15 snap -1 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:55: expect (ObjNum 14 snap 0 seq_num 14) 2022-04-22T17:25:42.355 INFO:tasks.rados.rados.0.smithi115.stdout:56: rollback oid 31 current snap is 0 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:56: done (2 left) 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:57: read oid 28 snap -1 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:58: copy_from oid 29 from oid 47 current snap is 0 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:59: rollback oid 45 current snap is 0 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:59: done (4 left) 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:60: copy_from oid 23 from oid 48 current snap is 0 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:61: delete oid 1 current snap is 0 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:52: finishing write tid 2 to smithi11541303-18 2022-04-22T17:25:42.356 INFO:tasks.rados.rados.0.smithi115.stdout:61: done (5 left) 2022-04-22T17:25:42.357 INFO:tasks.rados.rados.0.smithi115.stdout:62: read oid 17 snap -1 2022-04-22T17:25:42.357 INFO:tasks.rados.rados.0.smithi115.stdout:62: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:25:42.357 INFO:tasks.rados.rados.0.smithi115.stdout:63: snap_create 2022-04-22T17:25:42.357 INFO:tasks.rados.rados.0.smithi115.stdout:52: finishing write tid 3 to smithi11541303-18 2022-04-22T17:25:42.357 INFO:tasks.rados.rados.0.smithi115.stdout:55: expect (ObjNum 14 snap 0 seq_num 14) 2022-04-22T17:25:42.359 INFO:tasks.rados.rados.0.smithi115.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2022-04-22T17:25:42.371 INFO:tasks.rados.rados.0.smithi115.stdout:52: finishing write tid 4 to smithi11541303-18 2022-04-22T17:25:42.372 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2022-04-22T17:25:42.372 INFO:tasks.rados.rados.0.smithi115.stdout:52: left oid 18 (ObjNum 50 snap 0 seq_num 50) 2022-04-22T17:25:42.372 INFO:tasks.rados.rados.0.smithi115.stdout:62: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:25:42.381 INFO:tasks.rados.rados.0.smithi115.stdout:60: finishing copy_from to smithi11541303-23 2022-04-22T17:25:42.381 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 10 (ObjNum 47 snap 0 seq_num 47) dirty exists 2022-04-22T17:25:42.381 INFO:tasks.rados.rados.0.smithi115.stdout:60: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:25:42.381 INFO:tasks.rados.rados.0.smithi115.stdout:58: finishing copy_from to smithi11541303-29 2022-04-22T17:25:42.382 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:25:42.383 INFO:tasks.rados.rados.0.smithi115.stdout:58: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:25:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:42 smithi121 bash[12679]: cluster 2022-04-22T17:25:41.170670+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:42.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:42 smithi121 bash[12679]: audit 2022-04-22T17:25:42.120550+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2022-04-22T17:25:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:42 smithi115 bash[12737]: cluster 2022-04-22T17:25:41.170670+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:42 smithi115 bash[12737]: audit 2022-04-22T17:25:42.120550+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2022-04-22T17:25:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:42 smithi115 bash[18070]: cluster 2022-04-22T17:25:41.170670+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 2.3 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:25:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:42 smithi115 bash[18070]: audit 2022-04-22T17:25:42.120550+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2022-04-22T17:25:43.180 INFO:tasks.rados.rados.0.smithi115.stdout:52: done (6 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:55: done (5 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:57: done (4 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:58: done (3 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:60: done (2 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:62: done (1 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:63: done (0 left) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:64: copy_from oid 15 from oid 2 current snap is 1 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:65: read oid 30 snap -1 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:65: expect (ObjNum 29 snap 0 seq_num 29) 2022-04-22T17:25:43.181 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 1 2022-04-22T17:25:43.182 INFO:tasks.rados.rados.0.smithi115.stdout:66: seq_num 53 ranges {794624=491520,1286144=401408} 2022-04-22T17:25:43.191 INFO:tasks.rados.rados.0.smithi115.stdout:66: writing smithi11541303-23 from 794624 to 1286144 tid 1 2022-04-22T17:25:43.194 INFO:tasks.rados.rados.0.smithi115.stdout:66: writing smithi11541303-23 from 1286144 to 1687552 tid 2 2022-04-22T17:25:43.194 INFO:tasks.rados.rados.0.smithi115.stdout:67: rollback oid 10 current snap is 1 2022-04-22T17:25:43.195 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 0 2022-04-22T17:25:43.195 INFO:tasks.rados.rados.0.smithi115.stdout:65: expect (ObjNum 29 snap 0 seq_num 29) 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:64: finishing copy_from to smithi11541303-15 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:64: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:66: finishing write tid 1 to smithi11541303-23 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:66: finishing write tid 2 to smithi11541303-23 2022-04-22T17:25:43.213 INFO:tasks.rados.rados.0.smithi115.stdout:67: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:67: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:67: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 12 (ObjNum 9 snap 0 seq_num 9) dirty exists 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:66: finishing write tid 3 to smithi11541303-23 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:64: done (3 left) 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:65: done (2 left) 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:67: done (1 left) 2022-04-22T17:25:43.214 INFO:tasks.rados.rados.0.smithi115.stdout:68: setattr oid 31 current snap is 1 2022-04-22T17:25:43.215 INFO:tasks.rados.rados.0.smithi115.stdout:66: finishing write tid 4 to smithi11541303-23 2022-04-22T17:25:43.215 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 13 (ObjNum 53 snap 1 seq_num 53) dirty exists 2022-04-22T17:25:43.215 INFO:tasks.rados.rados.0.smithi115.stdout:66: left oid 23 (ObjNum 53 snap 1 seq_num 53) 2022-04-22T17:25:43.216 INFO:tasks.rados.rados.0.smithi115.stdout:66: done (1 left) 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:69: copy_from oid 45 from oid 34 current snap is 1 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:70: read oid 23 snap 0 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:70: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:71: copy_from oid 36 from oid 27 current snap is 1 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:72: rollback oid 49 current snap is 1 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 0 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:73: read oid 4 snap 0 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2022-04-22T17:25:43.217 INFO:tasks.rados.rados.0.smithi115.stdout:74: rmattr oid 7 current snap is 1 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:74: done (6 left) 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:75: copy_from oid 20 from oid 12 current snap is 1 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:76: read oid 39 snap -1 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:76: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:77: copy_from oid 6 from oid 37 current snap is 1 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:78: read oid 2 snap -1 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:78: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:43.218 INFO:tasks.rados.rados.0.smithi115.stdout:79: setattr oid 41 current snap is 1 2022-04-22T17:25:43.219 INFO:tasks.rados.rados.0.smithi115.stdout:80: snap_create 2022-04-22T17:25:43.226 INFO:tasks.rados.rados.0.smithi115.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2022-04-22T17:25:43.226 INFO:tasks.rados.rados.0.smithi115.stdout:70: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:43.249 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 18 (ObjNum 30 snap 0 seq_num 30) dirty exists 2022-04-22T17:25:43.249 INFO:tasks.rados.rados.0.smithi115.stdout:72: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:25:43.249 INFO:tasks.rados.rados.0.smithi115.stdout:76: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:25:43.266 INFO:tasks.rados.rados.0.smithi115.stdout:78: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:43.271 INFO:tasks.rados.rados.0.smithi115.stdout:75: finishing copy_from to smithi11541303-20 2022-04-22T17:25:43.271 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2022-04-22T17:25:43.271 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 14 (ObjNum 40 snap 0 seq_num 40) dirty exists 2022-04-22T17:25:43.271 INFO:tasks.rados.rados.0.smithi115.stdout:75: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:71: finishing copy_from to smithi11541303-36 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:71: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:72: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:72: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 22 (ObjNum 48 snap 0 seq_num 48) dirty exists 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:77: finishing copy_from to smithi11541303-6 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:77: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:69: finishing copy_from to smithi11541303-45 2022-04-22T17:25:43.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2022-04-22T17:25:43.273 INFO:tasks.rados.rados.0.smithi115.stdout:69: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:25:43.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[18070]: audit 2022-04-22T17:25:42.174592+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2022-04-22T17:25:43.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[18070]: cluster 2022-04-22T17:25:42.174661+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2022-04-22T17:25:43.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[12737]: audit 2022-04-22T17:25:42.174592+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2022-04-22T17:25:43.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[12737]: cluster 2022-04-22T17:25:42.174661+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2022-04-22T17:25:43.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:43 smithi121 bash[12679]: audit 2022-04-22T17:25:42.174592+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.115:0/2237573246' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2022-04-22T17:25:43.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:43 smithi121 bash[12679]: cluster 2022-04-22T17:25:42.174661+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2022-04-22T17:25:43.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:43.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:43.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:43.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:43.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:43.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:25:44.197 INFO:tasks.rados.rados.0.smithi115.stdout:68: done (11 left) 2022-04-22T17:25:44.197 INFO:tasks.rados.rados.0.smithi115.stdout:69: done (10 left) 2022-04-22T17:25:44.197 INFO:tasks.rados.rados.0.smithi115.stdout:70: done (9 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:71: done (8 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:72: done (7 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:73: done (6 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:75: done (5 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:76: done (4 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:77: done (3 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:78: done (2 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:79: done (1 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:80: done (0 left) 2022-04-22T17:25:44.198 INFO:tasks.rados.rados.0.smithi115.stdout:81: rollback oid 40 current snap is 2 2022-04-22T17:25:44.199 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 1 2022-04-22T17:25:44.199 INFO:tasks.rados.rados.0.smithi115.stdout:82: snap_create 2022-04-22T17:25:44.205 INFO:tasks.rados.rados.0.smithi115.stdout:81: finishing rollback tid 0 to smithi11541303-40 2022-04-22T17:25:44.205 INFO:tasks.rados.rados.0.smithi115.stdout:81: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:25:44.206 INFO:tasks.rados.rados.0.smithi115.stdout:81: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:25:44.206 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 18 (ObjNum 39 snap 0 seq_num 39) dirty exists 2022-04-22T17:25:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:44 smithi121 bash[12679]: cluster 2022-04-22T17:25:43.172529+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 442 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 73 KiB/s wr, 2 op/s 2022-04-22T17:25:44.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:44 smithi121 bash[12679]: cluster 2022-04-22T17:25:43.172710+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2022-04-22T17:25:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:44 smithi115 bash[12737]: cluster 2022-04-22T17:25:43.172529+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 442 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 73 KiB/s wr, 2 op/s 2022-04-22T17:25:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:44 smithi115 bash[12737]: cluster 2022-04-22T17:25:43.172710+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2022-04-22T17:25:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:44 smithi115 bash[18070]: cluster 2022-04-22T17:25:43.172529+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 442 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 73 KiB/s wr, 2 op/s 2022-04-22T17:25:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:44 smithi115 bash[18070]: cluster 2022-04-22T17:25:43.172710+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2022-04-22T17:25:45.208 INFO:tasks.rados.rados.0.smithi115.stdout:81: done (1 left) 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:82: done (0 left) 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:83: rollback oid 9 current snap is 3 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 0 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:84: read oid 20 snap 1 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:84: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:85: read oid 41 snap -1 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2022-04-22T17:25:45.243 INFO:tasks.rados.rados.0.smithi115.stdout:86: snap_create 2022-04-22T17:25:45.244 INFO:tasks.rados.rados.0.smithi115.stdout:83: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:25:45.245 INFO:tasks.rados.rados.0.smithi115.stdout:83: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:25:45.245 INFO:tasks.rados.rados.0.smithi115.stdout:83: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:25:45.245 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 26 (ObjNum 8 snap 0 seq_num 8) dirty exists 2022-04-22T17:25:45.245 INFO:tasks.rados.rados.0.smithi115.stdout:84: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:25:45.245 INFO:tasks.rados.rados.0.smithi115.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2022-04-22T17:25:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:45 smithi121 bash[12679]: cluster 2022-04-22T17:25:44.190980+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2022-04-22T17:25:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:45 smithi115 bash[12737]: cluster 2022-04-22T17:25:44.190980+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2022-04-22T17:25:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:45 smithi115 bash[18070]: cluster 2022-04-22T17:25:44.190980+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2022-04-22T17:25:46.210 INFO:tasks.rados.rados.0.smithi115.stdout:83: done (3 left) 2022-04-22T17:25:46.210 INFO:tasks.rados.rados.0.smithi115.stdout:84: done (2 left) 2022-04-22T17:25:46.210 INFO:tasks.rados.rados.0.smithi115.stdout:85: done (1 left) 2022-04-22T17:25:46.210 INFO:tasks.rados.rados.0.smithi115.stdout:86: done (0 left) 2022-04-22T17:25:46.211 INFO:tasks.rados.rados.0.smithi115.stdout:87: snap_remove snap 0 2022-04-22T17:25:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:46 smithi121 bash[12679]: cluster 2022-04-22T17:25:45.173344+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v333: 121 pgs: 121 active+clean; 15 MiB data, 72 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 35 op/s 2022-04-22T17:25:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:46 smithi121 bash[12679]: cluster 2022-04-22T17:25:45.201975+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2022-04-22T17:25:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:46 smithi115 bash[12737]: cluster 2022-04-22T17:25:45.173344+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v333: 121 pgs: 121 active+clean; 15 MiB data, 72 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 35 op/s 2022-04-22T17:25:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:46 smithi115 bash[12737]: cluster 2022-04-22T17:25:45.201975+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2022-04-22T17:25:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:46 smithi115 bash[18070]: cluster 2022-04-22T17:25:45.173344+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v333: 121 pgs: 121 active+clean; 15 MiB data, 72 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 35 op/s 2022-04-22T17:25:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:46 smithi115 bash[18070]: cluster 2022-04-22T17:25:45.201975+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2022-04-22T17:25:47.212 INFO:tasks.rados.rados.0.smithi115.stdout:87: done (0 left) 2022-04-22T17:25:47.213 INFO:tasks.rados.rados.0.smithi115.stdout:88: read oid 29 snap 3 2022-04-22T17:25:47.213 INFO:tasks.rados.rados.0.smithi115.stdout:88: expect (ObjNum 46 snap 0 seq_num 46) 2022-04-22T17:25:47.213 INFO:tasks.rados.rados.0.smithi115.stdout:89: read oid 46 snap -1 2022-04-22T17:25:47.213 INFO:tasks.rados.rados.0.smithi115.stdout:89: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:25:47.214 INFO:tasks.rados.rados.0.smithi115.stdout:90: copy_from oid 36 from oid 2 current snap is 4 2022-04-22T17:25:47.214 INFO:tasks.rados.rados.0.smithi115.stdout:91: rmattr oid 49 current snap is 4 2022-04-22T17:25:47.214 INFO:tasks.rados.rados.0.smithi115.stdout:91: done (3 left) 2022-04-22T17:25:47.214 INFO:tasks.rados.rados.0.smithi115.stdout:92: snap_remove snap 2 2022-04-22T17:25:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:47 smithi121 bash[12679]: cluster 2022-04-22T17:25:46.204142+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2022-04-22T17:25:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:47 smithi121 bash[12679]: cluster 2022-04-22T17:25:47.205598+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2022-04-22T17:25:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:47 smithi115 bash[12737]: cluster 2022-04-22T17:25:46.204142+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2022-04-22T17:25:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:47 smithi115 bash[12737]: cluster 2022-04-22T17:25:47.205598+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2022-04-22T17:25:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:47 smithi115 bash[18070]: cluster 2022-04-22T17:25:46.204142+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2022-04-22T17:25:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:47 smithi115 bash[18070]: cluster 2022-04-22T17:25:47.205598+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:92: done (3 left) 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:93: rmattr oid 30 current snap is 4 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:93: done (3 left) 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:94: read oid 42 snap 1 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:94: expect (ObjNum 41 snap 0 seq_num 41) 2022-04-22T17:25:48.219 INFO:tasks.rados.rados.0.smithi115.stdout:95: rmattr oid 14 current snap is 4 2022-04-22T17:25:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:95: done (4 left) 2022-04-22T17:25:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:96: read oid 9 snap 1 2022-04-22T17:25:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:96: expect (ObjNum 8 snap 0 seq_num 8) 2022-04-22T17:25:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 4 2022-04-22T17:25:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:97: seq_num 54 ranges {393216=507904,901120=401408} 2022-04-22T17:25:48.225 INFO:tasks.rados.rados.0.smithi115.stdout:97: writing smithi11541303-24 from 393216 to 901120 tid 1 2022-04-22T17:25:48.227 INFO:tasks.rados.rados.0.smithi115.stdout:97: writing smithi11541303-24 from 901120 to 1302528 tid 2 2022-04-22T17:25:48.227 INFO:tasks.rados.rados.0.smithi115.stdout:98: copy_from oid 31 from oid 6 current snap is 4 2022-04-22T17:25:48.227 INFO:tasks.rados.rados.0.smithi115.stdout:89: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:25:48.250 INFO:tasks.rados.rados.0.smithi115.stdout:88: expect (ObjNum 46 snap 0 seq_num 46) 2022-04-22T17:25:48.271 INFO:tasks.rados.rados.0.smithi115.stdout:90: finishing copy_from to smithi11541303-36 2022-04-22T17:25:48.271 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2022-04-22T17:25:48.271 INFO:tasks.rados.rados.0.smithi115.stdout:90: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:25:48.271 INFO:tasks.rados.rados.0.smithi115.stdout:94: expect (ObjNum 41 snap 0 seq_num 41) 2022-04-22T17:25:48.290 INFO:tasks.rados.rados.0.smithi115.stdout:96: expect (ObjNum 8 snap 0 seq_num 8) 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:97: finishing write tid 1 to smithi11541303-24 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:97: finishing write tid 2 to smithi11541303-24 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:97: finishing write tid 3 to smithi11541303-24 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:97: finishing write tid 4 to smithi11541303-24 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 10 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:97: left oid 24 (ObjNum 54 snap 4 seq_num 54) 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:98: finishing copy_from to smithi11541303-31 2022-04-22T17:25:48.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:98: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:88: done (6 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:89: done (5 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:90: done (4 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:94: done (3 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:96: done (2 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:97: done (1 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:98: done (0 left) 2022-04-22T17:25:48.295 INFO:tasks.rados.rados.0.smithi115.stdout:99: snap_remove snap 3 2022-04-22T17:25:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[12737]: cluster 2022-04-22T17:25:47.174077+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 20 MiB data, 77 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 59 op/s 2022-04-22T17:25:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[12737]: audit 2022-04-22T17:25:47.276363+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[12737]: cluster 2022-04-22T17:25:48.212506+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2022-04-22T17:25:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[18070]: cluster 2022-04-22T17:25:47.174077+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 20 MiB data, 77 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 59 op/s 2022-04-22T17:25:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[18070]: audit 2022-04-22T17:25:47.276363+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:48 smithi115 bash[18070]: cluster 2022-04-22T17:25:48.212506+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2022-04-22T17:25:48.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:48 smithi121 bash[12679]: cluster 2022-04-22T17:25:47.174077+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 20 MiB data, 77 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 59 op/s 2022-04-22T17:25:48.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:48 smithi121 bash[12679]: audit 2022-04-22T17:25:47.276363+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:25:48.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:48 smithi121 bash[12679]: cluster 2022-04-22T17:25:48.212506+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:99: done (0 left) 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:100: copy_from oid 42 from oid 39 current snap is 4 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:101: read oid 14 snap 1 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:101: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 4 2022-04-22T17:25:49.290 INFO:tasks.rados.rados.0.smithi115.stdout:102: seq_num 55 ranges {811008=524288,1335296=401408} 2022-04-22T17:25:49.300 INFO:tasks.rados.rados.0.smithi115.stdout:102: writing smithi11541303-49 from 811008 to 1335296 tid 1 2022-04-22T17:25:49.302 INFO:tasks.rados.rados.0.smithi115.stdout:102: writing smithi11541303-49 from 1335296 to 1736704 tid 2 2022-04-22T17:25:49.304 INFO:tasks.rados.rados.0.smithi115.stdout:103: delete oid 43 current snap is 4 2022-04-22T17:25:49.304 INFO:tasks.rados.rados.0.smithi115.stdout:101: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:25:49.311 INFO:tasks.rados.rados.0.smithi115.stdout:101: done (3 left) 2022-04-22T17:25:49.311 INFO:tasks.rados.rados.0.smithi115.stdout:103: done (2 left) 2022-04-22T17:25:49.312 INFO:tasks.rados.rados.0.smithi115.stdout:104: read oid 20 snap -1 2022-04-22T17:25:49.312 INFO:tasks.rados.rados.0.smithi115.stdout:104: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:25:49.312 INFO:tasks.rados.rados.0.smithi115.stdout:105: rollback oid 43 current snap is 4 2022-04-22T17:25:49.312 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 1 2022-04-22T17:25:49.312 INFO:tasks.rados.rados.0.smithi115.stdout:106: delete oid 15 current snap is 4 2022-04-22T17:25:49.313 INFO:tasks.rados.rados.0.smithi115.stdout:102: finishing write tid 1 to smithi11541303-49 2022-04-22T17:25:49.313 INFO:tasks.rados.rados.0.smithi115.stdout:102: finishing write tid 2 to smithi11541303-49 2022-04-22T17:25:49.315 INFO:tasks.rados.rados.0.smithi115.stdout:106: done (4 left) 2022-04-22T17:25:49.315 INFO:tasks.rados.rados.0.smithi115.stdout:107: read oid 2 snap 1 2022-04-22T17:25:49.316 INFO:tasks.rados.rados.0.smithi115.stdout:107: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:49.316 INFO:tasks.rados.rados.0.smithi115.stdout:108: rollback oid 27 current snap is 4 2022-04-22T17:25:49.316 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 1 2022-04-22T17:25:49.316 INFO:tasks.rados.rados.0.smithi115.stdout:109: delete oid 33 current snap is 4 2022-04-22T17:25:49.317 INFO:tasks.rados.rados.0.smithi115.stdout:100: finishing copy_from to smithi11541303-42 2022-04-22T17:25:49.317 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:25:49.317 INFO:tasks.rados.rados.0.smithi115.stdout:100: finishing copy_from racing read to smithi11541303-42 2022-04-22T17:25:49.318 INFO:tasks.rados.rados.0.smithi115.stdout:102: finishing write tid 3 to smithi11541303-49 2022-04-22T17:25:49.318 INFO:tasks.rados.rados.0.smithi115.stdout:105: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:25:49.318 INFO:tasks.rados.rados.0.smithi115.stdout:107: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:49.323 INFO:tasks.rados.rados.0.smithi115.stdout:104: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:105: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 18 (ObjNum 42 snap 0 seq_num 42) dirty exists 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:102: finishing write tid 4 to smithi11541303-49 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 44 (ObjNum 55 snap 4 seq_num 55) dirty exists 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:102: left oid 49 (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:25:49.329 INFO:tasks.rados.rados.0.smithi115.stdout:108: finishing rollback tid 0 to smithi11541303-27 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:108: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:108: finishing rollback tid 2 to smithi11541303-27 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:100: done (6 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:102: done (5 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:104: done (4 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:105: done (3 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:107: done (2 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:108: done (1 left) 2022-04-22T17:25:49.330 INFO:tasks.rados.rados.0.smithi115.stdout:109: done (0 left) 2022-04-22T17:25:49.331 INFO:tasks.rados.rados.0.smithi115.stdout:110: setattr oid 35 current snap is 4 2022-04-22T17:25:49.331 INFO:tasks.rados.rados.0.smithi115.stdout:111: snap_create 2022-04-22T17:25:49.333 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 11 (ObjNum 34 snap 0 seq_num 34) dirty exists 2022-04-22T17:25:50.300 INFO:tasks.rados.rados.0.smithi115.stdout:110: done (1 left) 2022-04-22T17:25:50.300 INFO:tasks.rados.rados.0.smithi115.stdout:111: done (0 left) 2022-04-22T17:25:50.300 INFO:tasks.rados.rados.0.smithi115.stdout:112: copy_from oid 25 from oid 6 current snap is 5 2022-04-22T17:25:50.300 INFO:tasks.rados.rados.0.smithi115.stdout:113: setattr oid 23 current snap is 5 2022-04-22T17:25:50.301 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 5 2022-04-22T17:25:50.302 INFO:tasks.rados.rados.0.smithi115.stdout:114: seq_num 59 ranges {0=598016,598016=401408} 2022-04-22T17:25:50.306 INFO:tasks.rados.rados.0.smithi115.stdout:114: writing smithi11541303-15 from 0 to 598016 tid 1 2022-04-22T17:25:50.309 INFO:tasks.rados.rados.0.smithi115.stdout:114: writing smithi11541303-15 from 598016 to 999424 tid 2 2022-04-22T17:25:50.309 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 5 2022-04-22T17:25:50.309 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 17 (ObjNum 53 snap 1 seq_num 53) dirty exists 2022-04-22T17:25:50.309 INFO:tasks.rados.rados.0.smithi115.stdout:115: seq_num 60 ranges {188416=614400,802816=401408} 2022-04-22T17:25:50.314 INFO:tasks.rados.rados.0.smithi115.stdout:115: writing smithi11541303-20 from 188416 to 802816 tid 1 2022-04-22T17:25:50.318 INFO:tasks.rados.rados.0.smithi115.stdout:115: writing smithi11541303-20 from 802816 to 1204224 tid 2 2022-04-22T17:25:50.318 INFO:tasks.rados.rados.0.smithi115.stdout:113: done (3 left) 2022-04-22T17:25:50.318 INFO:tasks.rados.rados.0.smithi115.stdout:116: rollback oid 14 current snap is 5 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 1 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:114: finishing write tid 1 to smithi11541303-15 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:114: finishing write tid 2 to smithi11541303-15 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:114: finishing write tid 3 to smithi11541303-15 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:112: finishing copy_from to smithi11541303-25 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:112: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:112: done (3 left) 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:117: read oid 22 snap -1 2022-04-22T17:25:50.319 INFO:tasks.rados.rados.0.smithi115.stdout:117: expect (ObjNum 21 snap 0 seq_num 21) 2022-04-22T17:25:50.320 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 5 2022-04-22T17:25:50.320 INFO:tasks.rados.rados.0.smithi115.stdout:118: seq_num 61 ranges {663552=630784,1294336=401408} 2022-04-22T17:25:50.325 INFO:tasks.rados.rados.0.smithi115.stdout:118: writing smithi11541303-40 from 663552 to 1294336 tid 1 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:118: writing smithi11541303-40 from 1294336 to 1695744 tid 2 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:119: snap_create 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:114: finishing write tid 4 to smithi11541303-15 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 11 (ObjNum 59 snap 5 seq_num 59) dirty exists 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:114: left oid 15 (ObjNum 59 snap 5 seq_num 59) 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:115: finishing write tid 1 to smithi11541303-20 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:115: finishing write tid 2 to smithi11541303-20 2022-04-22T17:25:50.328 INFO:tasks.rados.rados.0.smithi115.stdout:115: finishing write tid 3 to smithi11541303-20 2022-04-22T17:25:50.329 INFO:tasks.rados.rados.0.smithi115.stdout:115: finishing write tid 4 to smithi11541303-20 2022-04-22T17:25:50.329 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 24 (ObjNum 60 snap 5 seq_num 60) dirty exists 2022-04-22T17:25:50.329 INFO:tasks.rados.rados.0.smithi115.stdout:115: left oid 20 (ObjNum 60 snap 5 seq_num 60) 2022-04-22T17:25:50.329 INFO:tasks.rados.rados.0.smithi115.stdout:117: expect (ObjNum 21 snap 0 seq_num 21) 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:116: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:116: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:116: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 11 (ObjNum 13 snap 0 seq_num 13) dirty exists 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:118: finishing write tid 1 to smithi11541303-40 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:118: finishing write tid 2 to smithi11541303-40 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:118: finishing write tid 3 to smithi11541303-40 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:118: finishing write tid 4 to smithi11541303-40 2022-04-22T17:25:50.341 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 28 (ObjNum 61 snap 5 seq_num 61) dirty exists 2022-04-22T17:25:50.342 INFO:tasks.rados.rados.0.smithi115.stdout:118: left oid 40 (ObjNum 61 snap 5 seq_num 61) 2022-04-22T17:25:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:50 smithi115 bash[12737]: cluster 2022-04-22T17:25:49.175063+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v339: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 84 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 34 op/s 2022-04-22T17:25:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:50 smithi115 bash[12737]: cluster 2022-04-22T17:25:49.283667+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2022-04-22T17:25:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:50 smithi115 bash[18070]: cluster 2022-04-22T17:25:49.175063+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v339: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 84 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 34 op/s 2022-04-22T17:25:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:50 smithi115 bash[18070]: cluster 2022-04-22T17:25:49.283667+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2022-04-22T17:25:50.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:50 smithi121 bash[12679]: cluster 2022-04-22T17:25:49.175063+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v339: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 84 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 34 op/s 2022-04-22T17:25:50.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:50 smithi121 bash[12679]: cluster 2022-04-22T17:25:49.283667+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:114: done (5 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:115: done (4 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:116: done (3 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:117: done (2 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:118: done (1 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:119: done (0 left) 2022-04-22T17:25:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:120: read oid 48 snap 4 2022-04-22T17:25:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:120: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:121: read oid 49 snap 5 2022-04-22T17:25:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:121: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:25:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:122: delete oid 44 current snap is 6 2022-04-22T17:25:51.313 INFO:tasks.rados.rados.0.smithi115.stdout:122: done (2 left) 2022-04-22T17:25:51.313 INFO:tasks.rados.rados.0.smithi115.stdout:123: rollback oid 23 current snap is 6 2022-04-22T17:25:51.313 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 4 2022-04-22T17:25:51.313 INFO:tasks.rados.rados.0.smithi115.stdout:124: read oid 30 snap -1 2022-04-22T17:25:51.314 INFO:tasks.rados.rados.0.smithi115.stdout:124: expect (ObjNum 29 snap 0 seq_num 29) 2022-04-22T17:25:51.314 INFO:tasks.rados.rados.0.smithi115.stdout:125: setattr oid 35 current snap is 6 2022-04-22T17:25:51.315 INFO:tasks.rados.rados.0.smithi115.stdout:126: delete oid 25 current snap is 6 2022-04-22T17:25:51.317 INFO:tasks.rados.rados.0.smithi115.stdout:120: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:51.341 INFO:tasks.rados.rados.0.smithi115.stdout:123: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:25:51.341 INFO:tasks.rados.rados.0.smithi115.stdout:123: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:25:51.341 INFO:tasks.rados.rados.0.smithi115.stdout:123: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:25:51.341 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 21 (ObjNum 53 snap 1 seq_num 53) dirty exists 2022-04-22T17:25:51.341 INFO:tasks.rados.rados.0.smithi115.stdout:121: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:25:51.400 INFO:tasks.rados.rados.0.smithi115.stdout:124: expect (ObjNum 29 snap 0 seq_num 29) 2022-04-22T17:25:51.413 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 13 (ObjNum 34 snap 0 seq_num 34) dirty exists 2022-04-22T17:25:51.413 INFO:tasks.rados.rados.0.smithi115.stdout:120: done (5 left) 2022-04-22T17:25:51.413 INFO:tasks.rados.rados.0.smithi115.stdout:121: done (4 left) 2022-04-22T17:25:51.414 INFO:tasks.rados.rados.0.smithi115.stdout:123: done (3 left) 2022-04-22T17:25:51.414 INFO:tasks.rados.rados.0.smithi115.stdout:124: done (2 left) 2022-04-22T17:25:51.414 INFO:tasks.rados.rados.0.smithi115.stdout:125: done (1 left) 2022-04-22T17:25:51.414 INFO:tasks.rados.rados.0.smithi115.stdout:126: done (0 left) 2022-04-22T17:25:51.414 INFO:tasks.rados.rados.0.smithi115.stdout:127: snap_remove snap 4 2022-04-22T17:25:51.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:51 smithi121 bash[12679]: cluster 2022-04-22T17:25:50.294386+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2022-04-22T17:25:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:51 smithi115 bash[12737]: cluster 2022-04-22T17:25:50.294386+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2022-04-22T17:25:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:51 smithi115 bash[18070]: cluster 2022-04-22T17:25:50.294386+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2022-04-22T17:25:52.319 INFO:tasks.rados.rados.0.smithi115.stdout:127: done (0 left) 2022-04-22T17:25:52.319 INFO:tasks.rados.rados.0.smithi115.stdout:128: setattr oid 28 current snap is 6 2022-04-22T17:25:52.322 INFO:tasks.rados.rados.0.smithi115.stdout:129: snap_remove snap 1 2022-04-22T17:25:52.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:52 smithi121 bash[12679]: cluster 2022-04-22T17:25:51.175925+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v342: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2022-04-22T17:25:52.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:52 smithi121 bash[12679]: cluster 2022-04-22T17:25:51.296731+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2022-04-22T17:25:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:52 smithi115 bash[12737]: cluster 2022-04-22T17:25:51.175925+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v342: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2022-04-22T17:25:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:52 smithi115 bash[12737]: cluster 2022-04-22T17:25:51.296731+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2022-04-22T17:25:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:52 smithi115 bash[18070]: cluster 2022-04-22T17:25:51.175925+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v342: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 24 MiB data, 86 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2022-04-22T17:25:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:52 smithi115 bash[18070]: cluster 2022-04-22T17:25:51.296731+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2022-04-22T17:25:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:129: done (1 left) 2022-04-22T17:25:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:130: delete oid 6 current snap is 6 2022-04-22T17:25:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2022-04-22T17:25:53.334 INFO:tasks.rados.rados.0.smithi115.stdout:128: done (1 left) 2022-04-22T17:25:53.335 INFO:tasks.rados.rados.0.smithi115.stdout:130: done (0 left) 2022-04-22T17:25:53.335 INFO:tasks.rados.rados.0.smithi115.stdout:131: snap_remove snap 5 2022-04-22T17:25:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:53 smithi121 bash[12679]: cluster 2022-04-22T17:25:52.314096+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2022-04-22T17:25:53.767 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:25:53 smithi121 bash[15740]: debug 2022-04-22T17:25:53.329+0000 7feeed403700 -1 osd.4 pg_epoch: 78 pg[5.es0( v 74'8 (0'0,74'8] local-lis/les=63/64 n=4 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [4,5,0]p4(0) r=0 lpr=63 crt=74'8 lcod 67'6 mlcod 67'6 active+clean+snaptrim trimq=[3~1] ps=[2~1,4~2,9~1]] removing snap head 2022-04-22T17:25:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:53 smithi115 bash[18070]: cluster 2022-04-22T17:25:52.314096+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2022-04-22T17:25:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:53 smithi115 bash[12737]: cluster 2022-04-22T17:25:52.314096+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2022-04-22T17:25:53.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:53.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:53.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:53.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:25:53.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:25:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:25:53.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:25:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:131: done (0 left) 2022-04-22T17:25:54.337 INFO:tasks.rados.rados.0.smithi115.stdout:132: rollback oid 17 current snap is 6 2022-04-22T17:25:54.337 INFO:tasks.rados.rados.0.smithi115.stdout:132: done (0 left) 2022-04-22T17:25:54.337 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 6 2022-04-22T17:25:54.337 INFO:tasks.rados.rados.0.smithi115.stdout:133: seq_num 65 ranges {712704=696320,1409024=401408} 2022-04-22T17:25:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:133: writing smithi11541303-43 from 712704 to 1409024 tid 1 2022-04-22T17:25:54.349 INFO:tasks.rados.rados.0.smithi115.stdout:133: writing smithi11541303-43 from 1409024 to 1810432 tid 2 2022-04-22T17:25:54.349 INFO:tasks.rados.rados.0.smithi115.stdout:134: read oid 11 snap -1 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:134: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:135: read oid 6 snap -1 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:135: expect deleted 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:136: copy_from oid 18 from oid 46 current snap is 6 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:137: read oid 41 snap -1 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:137: expect (ObjNum 40 snap 0 seq_num 40) 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:138: rmattr oid 37 current snap is 6 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:138: done (5 left) 2022-04-22T17:25:54.350 INFO:tasks.rados.rados.0.smithi115.stdout:139: read oid 1 snap -1 2022-04-22T17:25:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:139: expect deleted 2022-04-22T17:25:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:140: rmattr oid 30 current snap is 6 2022-04-22T17:25:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:140: done (6 left) 2022-04-22T17:25:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:141: setattr oid 47 current snap is 6 2022-04-22T17:25:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:135: done (6 left) 2022-04-22T17:25:54.352 INFO:tasks.rados.rados.0.smithi115.stdout:139: done (5 left) 2022-04-22T17:25:54.352 INFO:tasks.rados.rados.0.smithi115.stdout:142: copy_from oid 23 from oid 25 current snap is 6 2022-04-22T17:25:54.352 INFO:tasks.rados.rados.0.smithi115.stdout:143: setattr oid 24 current snap is 6 2022-04-22T17:25:54.353 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 6 2022-04-22T17:25:54.353 INFO:tasks.rados.rados.0.smithi115.stdout:144: seq_num 66 ranges {303104=712704,1015808=401408} 2022-04-22T17:25:54.360 INFO:tasks.rados.rados.0.smithi115.stdout:144: writing smithi11541303-19 from 303104 to 1015808 tid 1 2022-04-22T17:25:54.363 INFO:tasks.rados.rados.0.smithi115.stdout:144: writing smithi11541303-19 from 1015808 to 1417216 tid 2 2022-04-22T17:25:54.363 INFO:tasks.rados.rados.0.smithi115.stdout:145: setattr oid 33 current snap is 6 2022-04-22T17:25:54.363 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 19 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:25:54.363 INFO:tasks.rados.rados.0.smithi115.stdout:134: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:25:54.369 INFO:tasks.rados.rados.0.smithi115.stdout:133: finishing write tid 1 to smithi11541303-43 2022-04-22T17:25:54.369 INFO:tasks.rados.rados.0.smithi115.stdout:142: finishing copy_from to smithi11541303-23 2022-04-22T17:25:54.369 INFO:tasks.rados.rados.0.smithi115.stdout:142: got expected ENOENT (src dne) 2022-04-22T17:25:54.369 INFO:tasks.rados.rados.0.smithi115.stdout:137: expect (ObjNum 40 snap 0 seq_num 40) 2022-04-22T17:25:54.392 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 33 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:142: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:133: finishing write tid 2 to smithi11541303-43 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:133: finishing write tid 3 to smithi11541303-43 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:133: finishing write tid 4 to smithi11541303-43 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 41 (ObjNum 65 snap 6 seq_num 65) dirty exists 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:133: left oid 43 (ObjNum 65 snap 6 seq_num 65) 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:144: finishing write tid 1 to smithi11541303-19 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:144: finishing write tid 2 to smithi11541303-19 2022-04-22T17:25:54.393 INFO:tasks.rados.rados.0.smithi115.stdout:144: finishing write tid 3 to smithi11541303-19 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:144: finishing write tid 4 to smithi11541303-19 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 9 (ObjNum 66 snap 6 seq_num 66) dirty exists 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:144: left oid 19 (ObjNum 66 snap 6 seq_num 66) 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:136: finishing copy_from to smithi11541303-18 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:136: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:133: done (8 left) 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:134: done (7 left) 2022-04-22T17:25:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:136: done (6 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:137: done (5 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:141: done (4 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:142: done (3 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:143: done (2 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:144: done (1 left) 2022-04-22T17:25:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:146: snap_create 2022-04-22T17:25:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 11 (ObjNum -1116569408 snap 22075 seq_num 3178397888) dirty exists 2022-04-22T17:25:54.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:54 smithi121 bash[12679]: cluster 2022-04-22T17:25:53.176770+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v345: 121 pgs: 121 active+clean; 25 MiB data, 87 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 11 op/s 2022-04-22T17:25:54.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:54 smithi121 bash[12679]: cluster 2022-04-22T17:25:53.324461+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2022-04-22T17:25:54.767 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:25:54 smithi121 bash[21417]: debug 2022-04-22T17:25:54.349+0000 7fcbfaaea700 -1 osd.6 pg_epoch: 79 pg[5.2s0( v 79'37 (0'0,79'37] local-lis/les=63/64 n=6 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [6,0,5]p6(0) r=0 lpr=63 luod=78'36 crt=78'36 lcod 78'34 mlcod 78'34 active+clean+snaptrim trimq=[a~1] ps=[2~4,9~1]] removing snap head 2022-04-22T17:25:54.793 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[26675]: debug 2022-04-22T17:25:54.340+0000 7faa020ae700 -1 osd.2 pg_epoch: 79 pg[5.bs0( v 78'8 (0'0,78'8] local-lis/les=63/64 n=2 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,0,5]p2(0) r=0 lpr=63 crt=78'8 lcod 77'6 mlcod 77'6 active+clean+snaptrim trimq=[a~1] ps=[2~4,9~1]] removing snap head 2022-04-22T17:25:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[12737]: cluster 2022-04-22T17:25:53.176770+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v345: 121 pgs: 121 active+clean; 25 MiB data, 87 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 11 op/s 2022-04-22T17:25:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[12737]: cluster 2022-04-22T17:25:53.324461+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2022-04-22T17:25:54.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[18070]: cluster 2022-04-22T17:25:53.176770+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v345: 121 pgs: 121 active+clean; 25 MiB data, 87 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 11 op/s 2022-04-22T17:25:54.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[18070]: cluster 2022-04-22T17:25:53.324461+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2022-04-22T17:25:54.794 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:25:54 smithi115 bash[23816]: debug 2022-04-22T17:25:54.324+0000 7fa4661c4700 -1 osd.1 pg_epoch: 79 pg[5.cs0( v 78'16 (0'0,78'16] local-lis/les=63/64 n=3 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [1,4,0]p1(0) r=0 lpr=63 crt=78'16 lcod 78'14 mlcod 78'14 active+clean+snaptrim trimq=[a~1] ps=[2~4,9~1]] removing snap head 2022-04-22T17:25:55.360 INFO:tasks.rados.rados.0.smithi115.stdout:145: done (1 left) 2022-04-22T17:25:55.360 INFO:tasks.rados.rados.0.smithi115.stdout:146: done (0 left) 2022-04-22T17:25:55.361 INFO:tasks.rados.rados.0.smithi115.stdout:147: snap_remove snap 6 2022-04-22T17:25:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:55 smithi121 bash[12679]: cluster 2022-04-22T17:25:54.328960+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2022-04-22T17:25:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:55 smithi115 bash[12737]: cluster 2022-04-22T17:25:54.328960+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2022-04-22T17:25:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:55 smithi115 bash[18070]: cluster 2022-04-22T17:25:54.328960+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2022-04-22T17:25:56.153 INFO:tasks.rados.rados.0.smithi115.stdout:147: done (0 left) 2022-04-22T17:25:56.153 INFO:tasks.rados.rados.0.smithi115.stdout:148: delete oid 1 current snap is 7 2022-04-22T17:25:56.162 INFO:tasks.rados.rados.0.smithi115.stdout:148: done (0 left) 2022-04-22T17:25:56.162 INFO:tasks.rados.rados.0.smithi115.stdout:149: snap_create 2022-04-22T17:25:56.476 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:25:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:56 smithi121 bash[12679]: cluster 2022-04-22T17:25:55.177503+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v348: 121 pgs: 121 active+clean; 29 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 35 op/s 2022-04-22T17:25:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:56 smithi121 bash[12679]: cluster 2022-04-22T17:25:55.354441+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2022-04-22T17:25:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:56 smithi121 bash[12679]: cluster 2022-04-22T17:25:56.146493+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2022-04-22T17:25:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[18070]: cluster 2022-04-22T17:25:55.177503+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v348: 121 pgs: 121 active+clean; 29 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 35 op/s 2022-04-22T17:25:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[18070]: cluster 2022-04-22T17:25:55.354441+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2022-04-22T17:25:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[18070]: cluster 2022-04-22T17:25:56.146493+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2022-04-22T17:25:56.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[12737]: cluster 2022-04-22T17:25:55.177503+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v348: 121 pgs: 121 active+clean; 29 MiB data, 94 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 35 op/s 2022-04-22T17:25:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[12737]: cluster 2022-04-22T17:25:55.354441+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2022-04-22T17:25:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:56 smithi115 bash[12737]: cluster 2022-04-22T17:25:56.146493+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2022-04-22T17:25:56.805 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:25:56.805 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 111s ago 2m 8767k - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:25:56.805 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2m) 115s ago 2m 22.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:25:56.805 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3m) 111s ago 3m 11.3M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:25:56.805 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3m) 115s ago 3m 13.1M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (8m) 115s ago 8m 357M - 16.2.5 46cf6318c64e c89d6897ce72 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running 111s ago 10m - - 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running 111s ago 10m - 2048M 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (8m) 115s ago 8m 38.0M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 111s ago 8m 38.1M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 111s ago 3m 6232k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2m) 115s ago 3m 6448k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 111s ago 7m 38.5M 4062M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 111s ago 7m 38.4M 4062M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:25:56.806 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6m) 111s ago 6m 36.6M 4062M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 111s ago 6m 37.7M 4062M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5m) 115s ago 5m 37.5M 4318M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5m) 115s ago 5m 38.0M 4318M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4m) 115s ago 4m 38.3M 4318M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4m) 115s ago 4m 38.6M 4318M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:25:56.807 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 115s ago 3m 23.8M - 2.18.1 de242295e225 6fd3ae0aa84b 2022-04-22T17:25:57.167 INFO:tasks.rados.rados.0.smithi115.stdout:149: done (0 left) 2022-04-22T17:25:57.167 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 8 2022-04-22T17:25:57.167 INFO:tasks.rados.rados.0.smithi115.stdout:150: seq_num 68 ranges {606208=745472,1351680=401408} 2022-04-22T17:25:57.179 INFO:tasks.rados.rados.0.smithi115.stdout:150: writing smithi11541303-31 from 606208 to 1351680 tid 1 2022-04-22T17:25:57.182 INFO:tasks.rados.rados.0.smithi115.stdout:150: writing smithi11541303-31 from 1351680 to 1753088 tid 2 2022-04-22T17:25:57.182 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 8 2022-04-22T17:25:57.182 INFO:tasks.rados.rados.0.smithi115.stdout:151: seq_num 69 ranges {253952=761856,1015808=401408} 2022-04-22T17:25:57.188 INFO:tasks.rados.rados.0.smithi115.stdout:151: writing smithi11541303-16 from 253952 to 1015808 tid 1 2022-04-22T17:25:57.191 INFO:tasks.rados.rados.0.smithi115.stdout:151: writing smithi11541303-16 from 1015808 to 1417216 tid 2 2022-04-22T17:25:57.191 INFO:tasks.rados.rados.0.smithi115.stdout:152: copy_from oid 10 from oid 39 current snap is 8 2022-04-22T17:25:57.191 INFO:tasks.rados.rados.0.smithi115.stdout:150: finishing write tid 1 to smithi11541303-31 2022-04-22T17:25:57.191 INFO:tasks.rados.rados.0.smithi115.stdout:153: read oid 48 snap 7 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:153: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:154: copy_from oid 50 from oid 1 current snap is 8 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:155: rollback oid 29 current snap is 8 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 7 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:156: copy_from oid 47 from oid 13 current snap is 8 2022-04-22T17:25:57.192 INFO:tasks.rados.rados.0.smithi115.stdout:157: snap_create 2022-04-22T17:25:57.195 INFO:tasks.rados.rados.0.smithi115.stdout:151: finishing write tid 1 to smithi11541303-16 2022-04-22T17:25:57.195 INFO:tasks.rados.rados.0.smithi115.stdout:155: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:25:57.195 INFO:tasks.rados.rados.0.smithi115.stdout:150: finishing write tid 2 to smithi11541303-31 2022-04-22T17:25:57.195 INFO:tasks.rados.rados.0.smithi115.stdout:151: finishing write tid 2 to smithi11541303-16 2022-04-22T17:25:57.195 INFO:tasks.rados.rados.0.smithi115.stdout:150: finishing write tid 3 to smithi11541303-31 2022-04-22T17:25:57.196 INFO:tasks.rados.rados.0.smithi115.stdout:151: finishing write tid 3 to smithi11541303-16 2022-04-22T17:25:57.196 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:25:57.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:25:57.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:25:57.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:57.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 15 2022-04-22T17:25:57.197 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:25:57.198 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:25:57.198 INFO:tasks.rados.rados.0.smithi115.stdout:153: expect (ObjNum 47 snap 0 seq_num 47) 2022-04-22T17:25:57.226 INFO:tasks.rados.rados.0.smithi115.stdout:155: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:155: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 25 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:150: finishing write tid 4 to smithi11541303-31 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 57 (ObjNum 68 snap 8 seq_num 68) dirty exists 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:150: left oid 31 (ObjNum 68 snap 8 seq_num 68) 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:154: finishing copy_from to smithi11541303-50 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:154: got expected ENOENT (src dne) 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:154: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:151: finishing write tid 4 to smithi11541303-16 2022-04-22T17:25:57.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 11 (ObjNum 69 snap 8 seq_num 69) dirty exists 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:151: left oid 16 (ObjNum 69 snap 8 seq_num 69) 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:152: finishing copy_from to smithi11541303-10 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:152: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:156: finishing copy_from to smithi11541303-47 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2022-04-22T17:25:57.232 INFO:tasks.rados.rados.0.smithi115.stdout:156: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:25:58.179 INFO:tasks.rados.rados.0.smithi115.stdout:150: done (7 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:151: done (6 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:152: done (5 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:153: done (4 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:154: done (3 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:155: done (2 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:156: done (1 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:157: done (0 left) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:158: read oid 46 snap -1 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:158: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:25:58.180 INFO:tasks.rados.rados.0.smithi115.stdout:159: read oid 7 snap 7 2022-04-22T17:25:58.181 INFO:tasks.rados.rados.0.smithi115.stdout:159: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:25:58.181 INFO:tasks.rados.rados.0.smithi115.stdout:160: snap_create 2022-04-22T17:25:58.182 INFO:tasks.rados.rados.0.smithi115.stdout:159: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:25:58.187 INFO:tasks.rados.rados.0.smithi115.stdout:158: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:25:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:58 smithi121 bash[12679]: audit 2022-04-22T17:25:56.456902+0000 mgr.y (mgr.14146) 377 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:58 smithi121 bash[12679]: audit 2022-04-22T17:25:56.780819+0000 mgr.y (mgr.14146) 378 : audit [DBG] from='client.24823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:58 smithi121 bash[12679]: cluster 2022-04-22T17:25:57.160581+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2022-04-22T17:25:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:58 smithi121 bash[12679]: audit 2022-04-22T17:25:57.189763+0000 mon.a (mon.0) 763 : audit [DBG] from='client.? 172.21.15.115:0/24523092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[12737]: audit 2022-04-22T17:25:56.456902+0000 mgr.y (mgr.14146) 377 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[12737]: audit 2022-04-22T17:25:56.780819+0000 mgr.y (mgr.14146) 378 : audit [DBG] from='client.24823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[12737]: cluster 2022-04-22T17:25:57.160581+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2022-04-22T17:25:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[12737]: audit 2022-04-22T17:25:57.189763+0000 mon.a (mon.0) 763 : audit [DBG] from='client.? 172.21.15.115:0/24523092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[18070]: audit 2022-04-22T17:25:56.456902+0000 mgr.y (mgr.14146) 377 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[18070]: audit 2022-04-22T17:25:56.780819+0000 mgr.y (mgr.14146) 378 : audit [DBG] from='client.24823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:25:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[18070]: cluster 2022-04-22T17:25:57.160581+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2022-04-22T17:25:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:58 smithi115 bash[18070]: audit 2022-04-22T17:25:57.189763+0000 mon.a (mon.0) 763 : audit [DBG] from='client.? 172.21.15.115:0/24523092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:25:59.181 INFO:tasks.rados.rados.0.smithi115.stdout:158: done (2 left) 2022-04-22T17:25:59.218 INFO:tasks.rados.rados.0.smithi115.stdout:159: done (1 left) 2022-04-22T17:25:59.218 INFO:tasks.rados.rados.0.smithi115.stdout:160: done (0 left) 2022-04-22T17:25:59.218 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 10 2022-04-22T17:25:59.218 INFO:tasks.rados.rados.0.smithi115.stdout:161: seq_num 70 ranges {204800=778240,983040=401408} 2022-04-22T17:25:59.220 INFO:tasks.rados.rados.0.smithi115.stdout:161: writing smithi11541303-13 from 204800 to 983040 tid 1 2022-04-22T17:25:59.220 INFO:tasks.rados.rados.0.smithi115.stdout:161: writing smithi11541303-13 from 983040 to 1384448 tid 2 2022-04-22T17:25:59.220 INFO:tasks.rados.rados.0.smithi115.stdout:162: copy_from oid 15 from oid 5 current snap is 10 2022-04-22T17:25:59.220 INFO:tasks.rados.rados.0.smithi115.stdout:163: rollback oid 19 current snap is 10 2022-04-22T17:25:59.220 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 8 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:164: copy_from oid 21 from oid 24 current snap is 10 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:165: copy_from oid 3 from oid 7 current snap is 10 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:166: read oid 33 snap -1 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:166: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:167: rollback oid 18 current snap is 10 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 7 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:168: copy_from oid 22 from oid 35 current snap is 10 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:169: read oid 31 snap -1 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:169: expect (ObjNum 68 snap 8 seq_num 68) 2022-04-22T17:25:59.221 INFO:tasks.rados.rados.0.smithi115.stdout:170: snap_remove snap 9 2022-04-22T17:25:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:59 smithi121 bash[12679]: cluster 2022-04-22T17:25:57.178572+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 29 MiB data, 95 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 28 op/s 2022-04-22T17:25:59.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:25:59 smithi121 bash[12679]: cluster 2022-04-22T17:25:58.173411+0000 mon.a (mon.0) 764 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2022-04-22T17:25:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:59 smithi115 bash[12737]: cluster 2022-04-22T17:25:57.178572+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 29 MiB data, 95 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 28 op/s 2022-04-22T17:25:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:25:59 smithi115 bash[12737]: cluster 2022-04-22T17:25:58.173411+0000 mon.a (mon.0) 764 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2022-04-22T17:25:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:59 smithi115 bash[18070]: cluster 2022-04-22T17:25:57.178572+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 29 MiB data, 95 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 28 op/s 2022-04-22T17:25:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:25:59 smithi115 bash[18070]: cluster 2022-04-22T17:25:58.173411+0000 mon.a (mon.0) 764 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2022-04-22T17:26:00.197 INFO:tasks.rados.rados.0.smithi115.stdout:170: done (9 left) 2022-04-22T17:26:00.197 INFO:tasks.rados.rados.0.smithi115.stdout:171: rollback oid 20 current snap is 10 2022-04-22T17:26:00.197 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 7 2022-04-22T17:26:00.198 INFO:tasks.rados.rados.0.smithi115.stdout:166: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:26:00.200 INFO:tasks.rados.rados.0.smithi115.stdout:163: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:26:00.200 INFO:tasks.rados.rados.0.smithi115.stdout:163: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:26:00.200 INFO:tasks.rados.rados.0.smithi115.stdout:161: finishing write tid 1 to smithi11541303-13 2022-04-22T17:26:00.200 INFO:tasks.rados.rados.0.smithi115.stdout:161: finishing write tid 2 to smithi11541303-13 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:161: finishing write tid 3 to smithi11541303-13 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:163: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 13 (ObjNum 66 snap 6 seq_num 66) dirty exists 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:161: finishing write tid 4 to smithi11541303-13 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 45 (ObjNum 70 snap 10 seq_num 70) dirty exists 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:161: left oid 13 (ObjNum 70 snap 10 seq_num 70) 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:165: finishing copy_from to smithi11541303-3 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 5 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:165: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:167: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:26:00.201 INFO:tasks.rados.rados.0.smithi115.stdout:167: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:26:00.202 INFO:tasks.rados.rados.0.smithi115.stdout:167: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:26:00.202 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 61 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:26:00.202 INFO:tasks.rados.rados.0.smithi115.stdout:169: expect (ObjNum 68 snap 8 seq_num 68) 2022-04-22T17:26:00.262 INFO:tasks.rados.rados.0.smithi115.stdout:162: finishing copy_from to smithi11541303-15 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:162: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:168: finishing copy_from to smithi11541303-22 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 13 (ObjNum 34 snap 0 seq_num 34) dirty exists 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:168: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:164: finishing copy_from to smithi11541303-21 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 19 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:164: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:26:00.263 INFO:tasks.rados.rados.0.smithi115.stdout:161: done (9 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:162: done (8 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:163: done (7 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:164: done (6 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:165: done (5 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:166: done (4 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:167: done (3 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:168: done (2 left) 2022-04-22T17:26:00.264 INFO:tasks.rados.rados.0.smithi115.stdout:169: done (1 left) 2022-04-22T17:26:00.265 INFO:tasks.rados.rados.0.smithi115.stdout:172: delete oid 39 current snap is 10 2022-04-22T17:26:00.265 INFO:tasks.rados.rados.0.smithi115.stdout:171: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:26:00.265 INFO:tasks.rados.rados.0.smithi115.stdout:171: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:26:00.265 INFO:tasks.rados.rados.0.smithi115.stdout:171: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:26:00.265 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 49 (ObjNum 60 snap 5 seq_num 60) dirty exists 2022-04-22T17:26:00.267 INFO:tasks.rados.rados.0.smithi115.stdout:171: done (1 left) 2022-04-22T17:26:00.267 INFO:tasks.rados.rados.0.smithi115.stdout:172: done (0 left) 2022-04-22T17:26:00.267 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 10 2022-04-22T17:26:00.267 INFO:tasks.rados.rados.0.smithi115.stdout:173: seq_num 72 ranges {0=409600,409600=524288,933888=278528} 2022-04-22T17:26:00.271 INFO:tasks.rados.rados.0.smithi115.stdout:173: writing smithi11541303-44 from 0 to 409600 tid 1 2022-04-22T17:26:00.276 INFO:tasks.rados.rados.0.smithi115.stdout:173: writing smithi11541303-44 from 409600 to 933888 tid 2 2022-04-22T17:26:00.279 INFO:tasks.rados.rados.0.smithi115.stdout:173: writing smithi11541303-44 from 933888 to 1212416 tid 3 2022-04-22T17:26:00.279 INFO:tasks.rados.rados.0.smithi115.stdout:174: snap_create 2022-04-22T17:26:00.279 INFO:tasks.rados.rados.0.smithi115.stdout:173: finishing write tid 1 to smithi11541303-44 2022-04-22T17:26:00.281 INFO:tasks.rados.rados.0.smithi115.stdout:173: finishing write tid 2 to smithi11541303-44 2022-04-22T17:26:00.282 INFO:tasks.rados.rados.0.smithi115.stdout:173: finishing write tid 3 to smithi11541303-44 2022-04-22T17:26:00.282 INFO:tasks.rados.rados.0.smithi115.stdout:173: finishing write tid 4 to smithi11541303-44 2022-04-22T17:26:00.284 INFO:tasks.rados.rados.0.smithi115.stdout:173: finishing write tid 5 to smithi11541303-44 2022-04-22T17:26:00.284 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 14 (ObjNum 72 snap 10 seq_num 72) dirty exists 2022-04-22T17:26:00.284 INFO:tasks.rados.rados.0.smithi115.stdout:173: left oid 44 (ObjNum 72 snap 10 seq_num 72) 2022-04-22T17:26:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:00 smithi121 bash[12679]: cluster 2022-04-22T17:25:59.174463+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2022-04-22T17:26:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:00 smithi121 bash[12679]: cluster 2022-04-22T17:25:59.179088+0000 mgr.y (mgr.14146) 380 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 27 MiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 767 KiB/s wr, 17 op/s 2022-04-22T17:26:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:00 smithi115 bash[12737]: cluster 2022-04-22T17:25:59.174463+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2022-04-22T17:26:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:00 smithi115 bash[12737]: cluster 2022-04-22T17:25:59.179088+0000 mgr.y (mgr.14146) 380 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 27 MiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 767 KiB/s wr, 17 op/s 2022-04-22T17:26:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:00 smithi115 bash[18070]: cluster 2022-04-22T17:25:59.174463+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2022-04-22T17:26:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:00 smithi115 bash[18070]: cluster 2022-04-22T17:25:59.179088+0000 mgr.y (mgr.14146) 380 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 27 MiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 767 KiB/s wr, 17 op/s 2022-04-22T17:26:01.158 INFO:tasks.rados.rados.0.smithi115.stdout:173: done (1 left) 2022-04-22T17:26:01.158 INFO:tasks.rados.rados.0.smithi115.stdout:174: done (0 left) 2022-04-22T17:26:01.158 INFO:tasks.rados.rados.0.smithi115.stdout:175: snap_remove snap 8 2022-04-22T17:26:01.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:01 smithi121 bash[12679]: cluster 2022-04-22T17:26:00.191056+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2022-04-22T17:26:01.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:01 smithi121 bash[12679]: cluster 2022-04-22T17:26:01.151698+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2022-04-22T17:26:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:01 smithi115 bash[12737]: cluster 2022-04-22T17:26:00.191056+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2022-04-22T17:26:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:01 smithi115 bash[12737]: cluster 2022-04-22T17:26:01.151698+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2022-04-22T17:26:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:01 smithi115 bash[18070]: cluster 2022-04-22T17:26:00.191056+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2022-04-22T17:26:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:01 smithi115 bash[18070]: cluster 2022-04-22T17:26:01.151698+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2022-04-22T17:26:02.166 INFO:tasks.rados.rados.0.smithi115.stdout:175: done (0 left) 2022-04-22T17:26:02.166 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 11 2022-04-22T17:26:02.166 INFO:tasks.rados.rados.0.smithi115.stdout:176: seq_num 73 ranges {573440=425984,999424=409600,1409024=393216} 2022-04-22T17:26:02.172 INFO:tasks.rados.rados.0.smithi115.stdout:176: writing smithi11541303-22 from 573440 to 999424 tid 1 2022-04-22T17:26:02.175 INFO:tasks.rados.rados.0.smithi115.stdout:176: writing smithi11541303-22 from 999424 to 1409024 tid 2 2022-04-22T17:26:02.178 INFO:tasks.rados.rados.0.smithi115.stdout:176: writing smithi11541303-22 from 1409024 to 1802240 tid 3 2022-04-22T17:26:02.178 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 11 2022-04-22T17:26:02.178 INFO:tasks.rados.rados.0.smithi115.stdout:176: finishing write tid 1 to smithi11541303-22 2022-04-22T17:26:02.178 INFO:tasks.rados.rados.0.smithi115.stdout:177: seq_num 74 ranges {638976=442368,1081344=696320,1777664=106496} 2022-04-22T17:26:02.184 INFO:tasks.rados.rados.0.smithi115.stdout:177: writing smithi11541303-10 from 638976 to 1081344 tid 1 2022-04-22T17:26:02.188 INFO:tasks.rados.rados.0.smithi115.stdout:177: writing smithi11541303-10 from 1081344 to 1777664 tid 2 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:177: writing smithi11541303-10 from 1777664 to 1884160 tid 3 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:178: setattr oid 6 current snap is 11 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:176: finishing write tid 2 to smithi11541303-22 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:176: finishing write tid 3 to smithi11541303-22 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:176: finishing write tid 4 to smithi11541303-22 2022-04-22T17:26:02.189 INFO:tasks.rados.rados.0.smithi115.stdout:176: finishing write tid 5 to smithi11541303-22 2022-04-22T17:26:02.190 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 23 (ObjNum 73 snap 11 seq_num 73) dirty exists 2022-04-22T17:26:02.190 INFO:tasks.rados.rados.0.smithi115.stdout:176: left oid 22 (ObjNum 73 snap 11 seq_num 73) 2022-04-22T17:26:02.190 INFO:tasks.rados.rados.0.smithi115.stdout:177: finishing write tid 1 to smithi11541303-10 2022-04-22T17:26:02.190 INFO:tasks.rados.rados.0.smithi115.stdout:176: done (2 left) 2022-04-22T17:26:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:179: read oid 12 snap 7 2022-04-22T17:26:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:179: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 11 2022-04-22T17:26:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:180: seq_num 75 ranges {106496=458752,565248=581632,1146880=221184} 2022-04-22T17:26:02.194 INFO:tasks.rados.rados.0.smithi115.stdout:180: writing smithi11541303-3 from 106496 to 565248 tid 1 2022-04-22T17:26:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:180: writing smithi11541303-3 from 565248 to 1146880 tid 2 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:180: writing smithi11541303-3 from 1146880 to 1368064 tid 3 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:181: snap_create 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:177: finishing write tid 2 to smithi11541303-10 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:177: finishing write tid 3 to smithi11541303-10 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:177: finishing write tid 4 to smithi11541303-10 2022-04-22T17:26:02.199 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 20 (ObjNum -1116466592 snap 22075 seq_num 3178500704) dirty exists 2022-04-22T17:26:02.200 INFO:tasks.rados.rados.0.smithi115.stdout:179: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:02.207 INFO:tasks.rados.rados.0.smithi115.stdout:177: finishing write tid 5 to smithi11541303-10 2022-04-22T17:26:02.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 42 (ObjNum 74 snap 11 seq_num 74) dirty exists 2022-04-22T17:26:02.207 INFO:tasks.rados.rados.0.smithi115.stdout:177: left oid 10 (ObjNum 74 snap 11 seq_num 74) 2022-04-22T17:26:02.207 INFO:tasks.rados.rados.0.smithi115.stdout:180: finishing write tid 1 to smithi11541303-3 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:180: finishing write tid 2 to smithi11541303-3 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:180: finishing write tid 3 to smithi11541303-3 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:180: finishing write tid 4 to smithi11541303-3 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:180: finishing write tid 5 to smithi11541303-3 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 38 (ObjNum 75 snap 11 seq_num 75) dirty exists 2022-04-22T17:26:02.208 INFO:tasks.rados.rados.0.smithi115.stdout:180: left oid 3 (ObjNum 75 snap 11 seq_num 75) 2022-04-22T17:26:03.175 INFO:tasks.rados.rados.0.smithi115.stdout:177: done (4 left) 2022-04-22T17:26:03.176 INFO:tasks.rados.rados.0.smithi115.stdout:178: done (3 left) 2022-04-22T17:26:03.176 INFO:tasks.rados.rados.0.smithi115.stdout:179: done (2 left) 2022-04-22T17:26:03.176 INFO:tasks.rados.rados.0.smithi115.stdout:180: done (1 left) 2022-04-22T17:26:03.176 INFO:tasks.rados.rados.0.smithi115.stdout:181: done (0 left) 2022-04-22T17:26:03.176 INFO:tasks.rados.rados.0.smithi115.stdout:182: snap_create 2022-04-22T17:26:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:03 smithi121 bash[12679]: cluster 2022-04-22T17:26:01.179901+0000 mgr.y (mgr.14146) 381 : cluster [DBG] pgmap v358: 121 pgs: 121 active+clean; 29 MiB data, 100 MiB used, 715 GiB / 715 GiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:26:03.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:03 smithi121 bash[12679]: cluster 2022-04-22T17:26:02.156312+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2022-04-22T17:26:03.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[18070]: cluster 2022-04-22T17:26:01.179901+0000 mgr.y (mgr.14146) 381 : cluster [DBG] pgmap v358: 121 pgs: 121 active+clean; 29 MiB data, 100 MiB used, 715 GiB / 715 GiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:26:03.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[18070]: cluster 2022-04-22T17:26:02.156312+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2022-04-22T17:26:03.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[12737]: cluster 2022-04-22T17:26:01.179901+0000 mgr.y (mgr.14146) 381 : cluster [DBG] pgmap v358: 121 pgs: 121 active+clean; 29 MiB data, 100 MiB used, 715 GiB / 715 GiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:26:03.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[12737]: cluster 2022-04-22T17:26:02.156312+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2022-04-22T17:26:03.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:03.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:03.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:03.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:26:04.186 INFO:tasks.rados.rados.0.smithi115.stdout:182: done (0 left) 2022-04-22T17:26:04.186 INFO:tasks.rados.rados.0.smithi115.stdout:183: read oid 45 snap -1 2022-04-22T17:26:04.187 INFO:tasks.rados.rados.0.smithi115.stdout:183: expect (ObjNum 33 snap 0 seq_num 33) 2022-04-22T17:26:04.187 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 13 2022-04-22T17:26:04.187 INFO:tasks.rados.rados.0.smithi115.stdout:184: seq_num 76 ranges {1753088=475136,2228224=458752,2686976=344064} 2022-04-22T17:26:04.200 INFO:tasks.rados.rados.0.smithi115.stdout:184: writing smithi11541303-31 from 1753088 to 2228224 tid 1 2022-04-22T17:26:04.202 INFO:tasks.rados.rados.0.smithi115.stdout:184: writing smithi11541303-31 from 2228224 to 2686976 tid 2 2022-04-22T17:26:04.205 INFO:tasks.rados.rados.0.smithi115.stdout:184: writing smithi11541303-31 from 2686976 to 3031040 tid 3 2022-04-22T17:26:04.205 INFO:tasks.rados.rados.0.smithi115.stdout:185: read oid 49 snap -1 2022-04-22T17:26:04.205 INFO:tasks.rados.rados.0.smithi115.stdout:185: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:26:04.205 INFO:tasks.rados.rados.0.smithi115.stdout:186: read oid 38 snap -1 2022-04-22T17:26:04.206 INFO:tasks.rados.rados.0.smithi115.stdout:186: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:26:04.206 INFO:tasks.rados.rados.0.smithi115.stdout:183: expect (ObjNum 33 snap 0 seq_num 33) 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:184: finishing write tid 1 to smithi11541303-31 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:184: finishing write tid 2 to smithi11541303-31 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:187: read oid 40 snap -1 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:187: expect (ObjNum 61 snap 5 seq_num 61) 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:183: done (4 left) 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:188: snap_create 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:184: finishing write tid 3 to smithi11541303-31 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:184: finishing write tid 4 to smithi11541303-31 2022-04-22T17:26:04.228 INFO:tasks.rados.rados.0.smithi115.stdout:184: finishing write tid 5 to smithi11541303-31 2022-04-22T17:26:04.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 70 (ObjNum 76 snap 13 seq_num 76) dirty exists 2022-04-22T17:26:04.229 INFO:tasks.rados.rados.0.smithi115.stdout:184: left oid 31 (ObjNum 76 snap 13 seq_num 76) 2022-04-22T17:26:04.229 INFO:tasks.rados.rados.0.smithi115.stdout:185: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:26:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:04 smithi121 bash[12679]: cluster 2022-04-22T17:26:03.169420+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2022-04-22T17:26:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:04 smithi121 bash[12679]: cluster 2022-04-22T17:26:03.180356+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v361: 121 pgs: 121 active+clean; 31 MiB data, 103 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2022-04-22T17:26:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:186: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:26:04.305 INFO:tasks.rados.rados.0.smithi115.stdout:187: expect (ObjNum 61 snap 5 seq_num 61) 2022-04-22T17:26:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:04 smithi115 bash[12737]: cluster 2022-04-22T17:26:03.169420+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2022-04-22T17:26:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:04 smithi115 bash[12737]: cluster 2022-04-22T17:26:03.180356+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v361: 121 pgs: 121 active+clean; 31 MiB data, 103 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2022-04-22T17:26:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:04 smithi115 bash[18070]: cluster 2022-04-22T17:26:03.169420+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2022-04-22T17:26:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:04 smithi115 bash[18070]: cluster 2022-04-22T17:26:03.180356+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v361: 121 pgs: 121 active+clean; 31 MiB data, 103 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2022-04-22T17:26:05.196 INFO:tasks.rados.rados.0.smithi115.stdout:184: done (4 left) 2022-04-22T17:26:05.196 INFO:tasks.rados.rados.0.smithi115.stdout:185: done (3 left) 2022-04-22T17:26:05.196 INFO:tasks.rados.rados.0.smithi115.stdout:186: done (2 left) 2022-04-22T17:26:05.197 INFO:tasks.rados.rados.0.smithi115.stdout:187: done (1 left) 2022-04-22T17:26:05.197 INFO:tasks.rados.rados.0.smithi115.stdout:188: done (0 left) 2022-04-22T17:26:05.197 INFO:tasks.rados.rados.0.smithi115.stdout:189: snap_remove snap 11 2022-04-22T17:26:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: cluster 2022-04-22T17:26:04.178745+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2022-04-22T17:26:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: cephadm 2022-04-22T17:26:04.760980+0000 mgr.y (mgr.14146) 383 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:26:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: audit 2022-04-22T17:26:04.770220+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: audit 2022-04-22T17:26:04.771993+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: audit 2022-04-22T17:26:04.773480+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: audit 2022-04-22T17:26:04.774648+0000 mon.a (mon.0) 774 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:05 smithi121 bash[12679]: cephadm 2022-04-22T17:26:04.775833+0000 mgr.y (mgr.14146) 384 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: cluster 2022-04-22T17:26:04.178745+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: cephadm 2022-04-22T17:26:04.760980+0000 mgr.y (mgr.14146) 383 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: audit 2022-04-22T17:26:04.770220+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: audit 2022-04-22T17:26:04.771993+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: audit 2022-04-22T17:26:04.773480+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: audit 2022-04-22T17:26:04.774648+0000 mon.a (mon.0) 774 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[12737]: cephadm 2022-04-22T17:26:04.775833+0000 mgr.y (mgr.14146) 384 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: cluster 2022-04-22T17:26:04.178745+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: cephadm 2022-04-22T17:26:04.760980+0000 mgr.y (mgr.14146) 383 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: audit 2022-04-22T17:26:04.770220+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: audit 2022-04-22T17:26:04.771993+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: audit 2022-04-22T17:26:04.773480+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: audit 2022-04-22T17:26:04.774648+0000 mon.a (mon.0) 774 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:05 smithi115 bash[18070]: cephadm 2022-04-22T17:26:04.775833+0000 mgr.y (mgr.14146) 384 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:26:06.162 INFO:tasks.rados.rados.0.smithi115.stdout:189: done (0 left) 2022-04-22T17:26:06.162 INFO:tasks.rados.rados.0.smithi115.stdout:190: read oid 36 snap -1 2022-04-22T17:26:06.162 INFO:tasks.rados.rados.0.smithi115.stdout:190: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:26:06.162 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 14 2022-04-22T17:26:06.162 INFO:tasks.rados.rados.0.smithi115.stdout:191: seq_num 77 ranges {0=491520,491520=794624,1286144=8192} 2022-04-22T17:26:06.166 INFO:tasks.rados.rados.0.smithi115.stdout:191: writing smithi11541303-1 from 0 to 491520 tid 1 2022-04-22T17:26:06.172 INFO:tasks.rados.rados.0.smithi115.stdout:191: writing smithi11541303-1 from 491520 to 1286144 tid 2 2022-04-22T17:26:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:191: writing smithi11541303-1 from 1286144 to 1294336 tid 3 2022-04-22T17:26:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 14 2022-04-22T17:26:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:192: seq_num 78 ranges {139264=516096,655360=679936,1335296=122880} 2022-04-22T17:26:06.177 INFO:tasks.rados.rados.0.smithi115.stdout:192: writing smithi11541303-9 from 139264 to 655360 tid 1 2022-04-22T17:26:06.181 INFO:tasks.rados.rados.0.smithi115.stdout:192: writing smithi11541303-9 from 655360 to 1335296 tid 2 2022-04-22T17:26:06.182 INFO:tasks.rados.rados.0.smithi115.stdout:192: writing smithi11541303-9 from 1335296 to 1458176 tid 3 2022-04-22T17:26:06.182 INFO:tasks.rados.rados.0.smithi115.stdout:193: rmattr oid 11 current snap is 14 2022-04-22T17:26:06.182 INFO:tasks.rados.rados.0.smithi115.stdout:193: done (3 left) 2022-04-22T17:26:06.182 INFO:tasks.rados.rados.0.smithi115.stdout:194: snap_remove snap 13 2022-04-22T17:26:06.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:06 smithi121 bash[12679]: cluster 2022-04-22T17:26:05.181378+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v363: 121 pgs: 121 active+clean; 36 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:26:06.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:06 smithi121 bash[12679]: cluster 2022-04-22T17:26:05.190047+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2022-04-22T17:26:06.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:06 smithi121 bash[12679]: cluster 2022-04-22T17:26:06.154871+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2022-04-22T17:26:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[12737]: cluster 2022-04-22T17:26:05.181378+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v363: 121 pgs: 121 active+clean; 36 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:26:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[12737]: cluster 2022-04-22T17:26:05.190047+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2022-04-22T17:26:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[12737]: cluster 2022-04-22T17:26:06.154871+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2022-04-22T17:26:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[18070]: cluster 2022-04-22T17:26:05.181378+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v363: 121 pgs: 121 active+clean; 36 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:26:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[18070]: cluster 2022-04-22T17:26:05.190047+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2022-04-22T17:26:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:06 smithi115 bash[18070]: cluster 2022-04-22T17:26:06.154871+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2022-04-22T17:26:07.176 INFO:tasks.rados.rados.0.smithi115.stdout:190: expect (ObjNum 10 snap 0 seq_num 10) 2022-04-22T17:26:07.182 INFO:tasks.rados.rados.0.smithi115.stdout:190: done (3 left) 2022-04-22T17:26:07.182 INFO:tasks.rados.rados.0.smithi115.stdout:194: done (2 left) 2022-04-22T17:26:07.182 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 14 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: finishing write tid 1 to smithi11541303-1 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: finishing write tid 2 to smithi11541303-1 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: finishing write tid 3 to smithi11541303-1 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: finishing write tid 4 to smithi11541303-1 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: finishing write tid 5 to smithi11541303-1 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 26 (ObjNum 77 snap 14 seq_num 77) dirty exists 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:191: left oid 1 (ObjNum 77 snap 14 seq_num 77) 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:192: finishing write tid 1 to smithi11541303-9 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:192: finishing write tid 2 to smithi11541303-9 2022-04-22T17:26:07.183 INFO:tasks.rados.rados.0.smithi115.stdout:192: finishing write tid 3 to smithi11541303-9 2022-04-22T17:26:07.184 INFO:tasks.rados.rados.0.smithi115.stdout:195: seq_num 79 ranges {1695744=532480,2228224=565248,2793472=237568} 2022-04-22T17:26:07.195 INFO:tasks.rados.rados.0.smithi115.stdout:195: writing smithi11541303-40 from 1695744 to 2228224 tid 1 2022-04-22T17:26:07.199 INFO:tasks.rados.rados.0.smithi115.stdout:195: writing smithi11541303-40 from 2228224 to 2793472 tid 2 2022-04-22T17:26:07.201 INFO:tasks.rados.rados.0.smithi115.stdout:195: writing smithi11541303-40 from 2793472 to 3031040 tid 3 2022-04-22T17:26:07.201 INFO:tasks.rados.rados.0.smithi115.stdout:191: done (2 left) 2022-04-22T17:26:07.201 INFO:tasks.rados.rados.0.smithi115.stdout:196: copy_from oid 13 from oid 43 current snap is 14 2022-04-22T17:26:07.201 INFO:tasks.rados.rados.0.smithi115.stdout:192: finishing write tid 4 to smithi11541303-9 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:192: finishing write tid 5 to smithi11541303-9 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 77 (ObjNum 78 snap 14 seq_num 78) dirty exists 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:192: left oid 9 (ObjNum 78 snap 14 seq_num 78) 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:192: done (2 left) 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:197: rollback oid 37 current snap is 14 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 7 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:198: rmattr oid 8 current snap is 14 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:198: done (3 left) 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:199: rmattr oid 24 current snap is 14 2022-04-22T17:26:07.202 INFO:tasks.rados.rados.0.smithi115.stdout:200: rmattr oid 11 current snap is 14 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:200: done (4 left) 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:201: read oid 4 snap -1 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:201: expect (ObjNum 3 snap 0 seq_num 3) 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:202: read oid 19 snap -1 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:202: expect (ObjNum 66 snap 6 seq_num 66) 2022-04-22T17:26:07.203 INFO:tasks.rados.rados.0.smithi115.stdout:203: setattr oid 26 current snap is 14 2022-04-22T17:26:07.204 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 14 2022-04-22T17:26:07.204 INFO:tasks.rados.rados.0.smithi115.stdout:204: seq_num 80 ranges {0=548864,548864=450560,999424=352256} 2022-04-22T17:26:07.207 INFO:tasks.rados.rados.0.smithi115.stdout:204: writing smithi11541303-6 from 0 to 548864 tid 1 2022-04-22T17:26:07.210 INFO:tasks.rados.rados.0.smithi115.stdout:204: writing smithi11541303-6 from 548864 to 999424 tid 2 2022-04-22T17:26:07.213 INFO:tasks.rados.rados.0.smithi115.stdout:204: writing smithi11541303-6 from 999424 to 1351680 tid 3 2022-04-22T17:26:07.213 INFO:tasks.rados.rados.0.smithi115.stdout:205: copy_from oid 47 from oid 17 current snap is 14 2022-04-22T17:26:07.213 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 22 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:07.213 INFO:tasks.rados.rados.0.smithi115.stdout:201: expect (ObjNum 3 snap 0 seq_num 3) 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:204: finishing write tid 1 to smithi11541303-6 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:197: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:197: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:197: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 27 (ObjNum 36 snap 0 seq_num 36) dirty exists 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 13 (ObjNum 25 snap 0 seq_num 25) dirty exists 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:195: finishing write tid 1 to smithi11541303-40 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:195: finishing write tid 2 to smithi11541303-40 2022-04-22T17:26:07.216 INFO:tasks.rados.rados.0.smithi115.stdout:195: finishing write tid 3 to smithi11541303-40 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:195: finishing write tid 4 to smithi11541303-40 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:197: done (8 left) 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:199: done (7 left) 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:201: done (6 left) 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:203: done (5 left) 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:206: snap_create 2022-04-22T17:26:07.217 INFO:tasks.rados.rados.0.smithi115.stdout:204: finishing write tid 2 to smithi11541303-6 2022-04-22T17:26:07.218 INFO:tasks.rados.rados.0.smithi115.stdout:204: finishing write tid 3 to smithi11541303-6 2022-04-22T17:26:07.218 INFO:tasks.rados.rados.0.smithi115.stdout:195: finishing write tid 5 to smithi11541303-40 2022-04-22T17:26:07.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 47 (ObjNum 79 snap 14 seq_num 79) dirty exists 2022-04-22T17:26:07.218 INFO:tasks.rados.rados.0.smithi115.stdout:195: left oid 40 (ObjNum 79 snap 14 seq_num 79) 2022-04-22T17:26:07.218 INFO:tasks.rados.rados.0.smithi115.stdout:204: finishing write tid 4 to smithi11541303-6 2022-04-22T17:26:07.219 INFO:tasks.rados.rados.0.smithi115.stdout:204: finishing write tid 5 to smithi11541303-6 2022-04-22T17:26:07.219 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 27 (ObjNum 80 snap 14 seq_num 80) dirty exists 2022-04-22T17:26:07.219 INFO:tasks.rados.rados.0.smithi115.stdout:204: left oid 6 (ObjNum 80 snap 14 seq_num 80) 2022-04-22T17:26:07.220 INFO:tasks.rados.rados.0.smithi115.stdout:202: expect (ObjNum 66 snap 6 seq_num 66) 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:205: finishing copy_from to smithi11541303-47 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:205: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:196: finishing copy_from to smithi11541303-13 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 41 (ObjNum 65 snap 6 seq_num 65) dirty exists 2022-04-22T17:26:07.264 INFO:tasks.rados.rados.0.smithi115.stdout:196: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:26:07.767 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:07 smithi121 systemd[1]: Stopping Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:26:08.183 INFO:tasks.rados.rados.0.smithi115.stdout:195: done (5 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:196: done (4 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:202: done (3 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:204: done (2 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:205: done (1 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:206: done (0 left) 2022-04-22T17:26:08.234 INFO:tasks.rados.rados.0.smithi115.stdout:207: delete oid 31 current snap is 15 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:207: done (0 left) 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:208: delete oid 26 current snap is 15 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:208: done (0 left) 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:209: copy_from oid 20 from oid 42 current snap is 15 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:210: delete oid 5 current snap is 15 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:210: done (1 left) 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 15 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:211: seq_num 84 ranges {1810432=614400,2424832=434176,2859008=368640} 2022-04-22T17:26:08.236 INFO:tasks.rados.rados.0.smithi115.stdout:211: writing smithi11541303-13 from 1810432 to 2424832 tid 1 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:211: writing smithi11541303-13 from 2424832 to 2859008 tid 2 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:211: writing smithi11541303-13 from 2859008 to 3227648 tid 3 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:209: finishing copy_from to smithi11541303-20 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:212: read oid 19 snap -1 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:212: expect (ObjNum 66 snap 6 seq_num 66) 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 15 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:209: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:213: seq_num 85 ranges {1302528=630784,1933312=720896,2654208=81920} 2022-04-22T17:26:08.237 INFO:tasks.rados.rados.0.smithi115.stdout:213: writing smithi11541303-21 from 1302528 to 1933312 tid 1 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:213: writing smithi11541303-21 from 1933312 to 2654208 tid 2 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:213: writing smithi11541303-21 from 2654208 to 2736128 tid 3 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:209: done (3 left) 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:214: delete oid 11 current snap is 15 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:211: finishing write tid 1 to smithi11541303-13 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:211: finishing write tid 2 to smithi11541303-13 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:211: finishing write tid 3 to smithi11541303-13 2022-04-22T17:26:08.240 INFO:tasks.rados.rados.0.smithi115.stdout:211: finishing write tid 4 to smithi11541303-13 2022-04-22T17:26:08.241 INFO:tasks.rados.rados.0.smithi115.stdout:211: finishing write tid 5 to smithi11541303-13 2022-04-22T17:26:08.241 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 64 (ObjNum 84 snap 15 seq_num 84) dirty exists 2022-04-22T17:26:08.241 INFO:tasks.rados.rados.0.smithi115.stdout:211: left oid 13 (ObjNum 84 snap 15 seq_num 84) 2022-04-22T17:26:08.241 INFO:tasks.rados.rados.0.smithi115.stdout:212: expect (ObjNum 66 snap 6 seq_num 66) 2022-04-22T17:26:08.283 INFO:tasks.rados.rados.0.smithi115.stdout:213: finishing write tid 1 to smithi11541303-21 2022-04-22T17:26:08.283 INFO:tasks.rados.rados.0.smithi115.stdout:213: finishing write tid 2 to smithi11541303-21 2022-04-22T17:26:08.283 INFO:tasks.rados.rados.0.smithi115.stdout:213: finishing write tid 3 to smithi11541303-21 2022-04-22T17:26:08.283 INFO:tasks.rados.rados.0.smithi115.stdout:213: finishing write tid 4 to smithi11541303-21 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:213: finishing write tid 5 to smithi11541303-21 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 27 (ObjNum 85 snap 15 seq_num 85) dirty exists 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:213: left oid 21 (ObjNum 85 snap 15 seq_num 85) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:211: done (3 left) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:212: done (2 left) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:213: done (1 left) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:214: done (0 left) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:215: read oid 38 snap -1 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:215: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:26:08.284 INFO:tasks.rados.rados.0.smithi115.stdout:216: copy_from oid 41 from oid 6 current snap is 15 2022-04-22T17:26:08.285 INFO:tasks.rados.rados.0.smithi115.stdout:217: copy_from oid 30 from oid 19 current snap is 15 2022-04-22T17:26:08.285 INFO:tasks.rados.rados.0.smithi115.stdout:218: read oid 14 snap 14 2022-04-22T17:26:08.285 INFO:tasks.rados.rados.0.smithi115.stdout:218: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:08.285 INFO:tasks.rados.rados.0.smithi115.stdout:219: delete oid 4 current snap is 15 2022-04-22T17:26:08.288 INFO:tasks.rados.rados.0.smithi115.stdout:218: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:08.297 INFO:tasks.rados.rados.0.smithi115.stdout:215: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:217: finishing copy_from to smithi11541303-30 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 13 (ObjNum 66 snap 6 seq_num 66) dirty exists 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:217: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:216: finishing copy_from to smithi11541303-41 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 27 (ObjNum 80 snap 14 seq_num 80) dirty exists 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:216: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:215: done (4 left) 2022-04-22T17:26:08.313 INFO:tasks.rados.rados.0.smithi115.stdout:216: done (3 left) 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:217: done (2 left) 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:218: done (1 left) 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:219: done (0 left) 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:220: rollback oid 14 current snap is 15 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 10 2022-04-22T17:26:08.314 INFO:tasks.rados.rados.0.smithi115.stdout:221: delete oid 44 current snap is 15 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:221: done (1 left) 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:222: copy_from oid 41 from oid 8 current snap is 15 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:223: rollback oid 34 current snap is 15 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 7 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 15 2022-04-22T17:26:08.317 INFO:tasks.rados.rados.0.smithi115.stdout:224: seq_num 89 ranges {3227648=696320,3923968=655360,4579328=147456} 2022-04-22T17:26:08.342 INFO:tasks.rados.rados.0.smithi115.stdout:224: writing smithi11541303-13 from 3227648 to 3923968 tid 1 2022-04-22T17:26:08.345 INFO:tasks.rados.rados.0.smithi115.stdout:224: writing smithi11541303-13 from 3923968 to 4579328 tid 2 2022-04-22T17:26:08.346 INFO:tasks.rados.rados.0.smithi115.stdout:224: writing smithi11541303-13 from 4579328 to 4726784 tid 3 2022-04-22T17:26:08.346 INFO:tasks.rados.rados.0.smithi115.stdout:225: rollback oid 29 current snap is 15 2022-04-22T17:26:08.347 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 10 2022-04-22T17:26:08.347 INFO:tasks.rados.rados.0.smithi115.stdout:220: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:26:08.347 INFO:tasks.rados.rados.0.smithi115.stdout:226: snap_remove snap 14 2022-04-22T17:26:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:08 smithi121 bash[12679]: cluster 2022-04-22T17:26:07.168990+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2022-04-22T17:26:08.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:08 smithi121 bash[12679]: cluster 2022-04-22T17:26:07.182023+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v367: 121 pgs: 121 active+clean; 37 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2022-04-22T17:26:08.522 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:08 smithi121 docker[30057]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr.x 2022-04-22T17:26:08.522 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:08 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2022-04-22T17:26:08.522 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:08 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2022-04-22T17:26:08.522 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:08 smithi121 systemd[1]: Stopped Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:26:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:08 smithi115 bash[12737]: cluster 2022-04-22T17:26:07.168990+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2022-04-22T17:26:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:08 smithi115 bash[12737]: cluster 2022-04-22T17:26:07.182023+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v367: 121 pgs: 121 active+clean; 37 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2022-04-22T17:26:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:08 smithi115 bash[18070]: cluster 2022-04-22T17:26:07.168990+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2022-04-22T17:26:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:08 smithi115 bash[18070]: cluster 2022-04-22T17:26:07.182023+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v367: 121 pgs: 121 active+clean; 37 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2022-04-22T17:26:09.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:08 smithi121 systemd[1]: Started Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:26:09.196 INFO:tasks.rados.rados.0.smithi115.stdout:226: done (5 left) 2022-04-22T17:26:09.196 INFO:tasks.rados.rados.0.smithi115.stdout:227: snap_remove snap 12 2022-04-22T17:26:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:09 smithi121 bash[12679]: cluster 2022-04-22T17:26:08.176139+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2022-04-22T17:26:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:09 smithi121 bash[12679]: audit 2022-04-22T17:26:08.621629+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:09 smithi121 bash[12679]: audit 2022-04-22T17:26:08.622944+0000 mon.a (mon.0) 780 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:09 smithi121 bash[12679]: audit 2022-04-22T17:26:08.625035+0000 mon.a (mon.0) 781 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:09.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:09 smithi121 bash[12679]: audit 2022-04-22T17:26:08.626476+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[12737]: cluster 2022-04-22T17:26:08.176139+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[12737]: audit 2022-04-22T17:26:08.621629+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[12737]: audit 2022-04-22T17:26:08.622944+0000 mon.a (mon.0) 780 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[12737]: audit 2022-04-22T17:26:08.625035+0000 mon.a (mon.0) 781 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[12737]: audit 2022-04-22T17:26:08.626476+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[18070]: cluster 2022-04-22T17:26:08.176139+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2022-04-22T17:26:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[18070]: audit 2022-04-22T17:26:08.621629+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[18070]: audit 2022-04-22T17:26:08.622944+0000 mon.a (mon.0) 780 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[18070]: audit 2022-04-22T17:26:08.625035+0000 mon.a (mon.0) 781 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:09 smithi115 bash[18070]: audit 2022-04-22T17:26:08.626476+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:227: done (5 left) 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:228: rollback oid 6 current snap is 15 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 7 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:229: copy_from oid 3 from oid 24 current snap is 15 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 15 2022-04-22T17:26:10.198 INFO:tasks.rados.rados.0.smithi115.stdout:230: seq_num 90 ranges {827392=712704,1540096=540672,2080768=262144} 2022-04-22T17:26:10.206 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:10 smithi121 bash[30140]: debug 2022-04-22T17:26:10.093+0000 7f15d3e95700 1 -- 172.21.15.121:0/1702981497 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f49b2c6340 con 0x55f49b3c0400 2022-04-22T17:26:10.209 INFO:tasks.rados.rados.0.smithi115.stdout:230: writing smithi11541303-50 from 827392 to 1540096 tid 1 2022-04-22T17:26:10.213 INFO:tasks.rados.rados.0.smithi115.stdout:230: writing smithi11541303-50 from 1540096 to 2080768 tid 2 2022-04-22T17:26:10.215 INFO:tasks.rados.rados.0.smithi115.stdout:230: writing smithi11541303-50 from 2080768 to 2342912 tid 3 2022-04-22T17:26:10.216 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 15 2022-04-22T17:26:10.216 INFO:tasks.rados.rados.0.smithi115.stdout:220: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:26:10.216 INFO:tasks.rados.rados.0.smithi115.stdout:220: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:26:10.216 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 29 (ObjNum 13 snap 0 seq_num 13) dirty exists 2022-04-22T17:26:10.216 INFO:tasks.rados.rados.0.smithi115.stdout:223: finishing rollback tid 0 to smithi11541303-34 2022-04-22T17:26:10.217 INFO:tasks.rados.rados.0.smithi115.stdout:231: seq_num 91 ranges {0=729088,729088=425984,1155072=376832} 2022-04-22T17:26:10.220 INFO:tasks.rados.rados.0.smithi115.stdout:231: writing smithi11541303-5 from 0 to 729088 tid 1 2022-04-22T17:26:10.223 INFO:tasks.rados.rados.0.smithi115.stdout:231: writing smithi11541303-5 from 729088 to 1155072 tid 2 2022-04-22T17:26:10.226 INFO:tasks.rados.rados.0.smithi115.stdout:231: writing smithi11541303-5 from 1155072 to 1531904 tid 3 2022-04-22T17:26:10.226 INFO:tasks.rados.rados.0.smithi115.stdout:220: done (8 left) 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:232: setattr oid 43 current snap is 15 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:223: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:223: finishing rollback tid 2 to smithi11541303-34 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:222: finishing copy_from to smithi11541303-41 2022-04-22T17:26:10.227 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2022-04-22T17:26:10.228 INFO:tasks.rados.rados.0.smithi115.stdout:222: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:26:10.228 INFO:tasks.rados.rados.0.smithi115.stdout:224: finishing write tid 1 to smithi11541303-13 2022-04-22T17:26:10.228 INFO:tasks.rados.rados.0.smithi115.stdout:225: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:26:10.228 INFO:tasks.rados.rados.0.smithi115.stdout:225: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:26:10.229 INFO:tasks.rados.rados.0.smithi115.stdout:225: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:26:10.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 33 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:26:10.229 INFO:tasks.rados.rados.0.smithi115.stdout:224: finishing write tid 2 to smithi11541303-13 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:224: finishing write tid 3 to smithi11541303-13 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:224: finishing write tid 4 to smithi11541303-13 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:224: finishing write tid 5 to smithi11541303-13 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 68 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:224: left oid 13 (ObjNum 89 snap 15 seq_num 89) 2022-04-22T17:26:10.230 INFO:tasks.rados.rados.0.smithi115.stdout:228: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:26:10.231 INFO:tasks.rados.rados.0.smithi115.stdout:228: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:26:10.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 33 (ObjNum -1116202336 snap 22075 seq_num 3178765328) dirty dne 2022-04-22T17:26:10.231 INFO:tasks.rados.rados.0.smithi115.stdout:230: finishing write tid 1 to smithi11541303-50 2022-04-22T17:26:10.231 INFO:tasks.rados.rados.0.smithi115.stdout:229: finishing copy_from to smithi11541303-3 2022-04-22T17:26:10.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 22 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:229: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:230: finishing write tid 2 to smithi11541303-50 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:230: finishing write tid 3 to smithi11541303-50 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:230: finishing write tid 4 to smithi11541303-50 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:230: finishing write tid 5 to smithi11541303-50 2022-04-22T17:26:10.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 24 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:26:10.233 INFO:tasks.rados.rados.0.smithi115.stdout:230: left oid 50 (ObjNum 90 snap 15 seq_num 90) 2022-04-22T17:26:10.233 INFO:tasks.rados.rados.0.smithi115.stdout:231: finishing write tid 1 to smithi11541303-5 2022-04-22T17:26:10.233 INFO:tasks.rados.rados.0.smithi115.stdout:231: finishing write tid 2 to smithi11541303-5 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:222: done (8 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:223: done (7 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:224: done (6 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:225: done (5 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:228: done (4 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:229: done (3 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:230: done (2 left) 2022-04-22T17:26:10.234 INFO:tasks.rados.rados.0.smithi115.stdout:233: snap_remove snap 7 2022-04-22T17:26:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:10 smithi121 bash[12679]: cluster 2022-04-22T17:26:09.183056+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v369: 121 pgs: 121 active+clean; 43 MiB data, 122 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2022-04-22T17:26:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:10 smithi121 bash[12679]: cluster 2022-04-22T17:26:09.188428+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2022-04-22T17:26:10.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:10 smithi121 bash[30140]: debug 2022-04-22T17:26:10.281+0000 7f15dcb0f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:26:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:10 smithi115 bash[12737]: cluster 2022-04-22T17:26:09.183056+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v369: 121 pgs: 121 active+clean; 43 MiB data, 122 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2022-04-22T17:26:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:10 smithi115 bash[12737]: cluster 2022-04-22T17:26:09.188428+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2022-04-22T17:26:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:10 smithi115 bash[18070]: cluster 2022-04-22T17:26:09.183056+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v369: 121 pgs: 121 active+clean; 43 MiB data, 122 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2022-04-22T17:26:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:10 smithi115 bash[18070]: cluster 2022-04-22T17:26:09.188428+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:233: done (2 left) 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 15 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:231: finishing write tid 3 to smithi11541303-5 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:231: finishing write tid 4 to smithi11541303-5 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:231: finishing write tid 5 to smithi11541303-5 2022-04-22T17:26:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 97 (ObjNum 91 snap 15 seq_num 91) dirty exists 2022-04-22T17:26:11.163 INFO:tasks.rados.rados.0.smithi115.stdout:231: left oid 5 (ObjNum 91 snap 15 seq_num 91) 2022-04-22T17:26:11.163 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 78 (ObjNum 65 snap 6 seq_num 65) dirty exists 2022-04-22T17:26:11.163 INFO:tasks.rados.rados.0.smithi115.stdout:234: seq_num 92 ranges {0=745472,745472=761856,1507328=40960} 2022-04-22T17:26:11.167 INFO:tasks.rados.rados.0.smithi115.stdout:234: writing smithi11541303-11 from 0 to 745472 tid 1 2022-04-22T17:26:11.174 INFO:tasks.rados.rados.0.smithi115.stdout:234: writing smithi11541303-11 from 745472 to 1507328 tid 2 2022-04-22T17:26:11.175 INFO:tasks.rados.rados.0.smithi115.stdout:234: writing smithi11541303-11 from 1507328 to 1548288 tid 3 2022-04-22T17:26:11.175 INFO:tasks.rados.rados.0.smithi115.stdout:231: done (2 left) 2022-04-22T17:26:11.175 INFO:tasks.rados.rados.0.smithi115.stdout:232: done (1 left) 2022-04-22T17:26:11.175 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 15 2022-04-22T17:26:11.175 INFO:tasks.rados.rados.0.smithi115.stdout:235: seq_num 93 ranges {638976=761856,1400832=647168,2048000=155648} 2022-04-22T17:26:11.184 INFO:tasks.rados.rados.0.smithi115.stdout:235: writing smithi11541303-20 from 638976 to 1400832 tid 1 2022-04-22T17:26:11.188 INFO:tasks.rados.rados.0.smithi115.stdout:235: writing smithi11541303-20 from 1400832 to 2048000 tid 2 2022-04-22T17:26:11.189 INFO:tasks.rados.rados.0.smithi115.stdout:235: writing smithi11541303-20 from 2048000 to 2203648 tid 3 2022-04-22T17:26:11.189 INFO:tasks.rados.rados.0.smithi115.stdout:236: copy_from oid 28 from oid 27 current snap is 15 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: finishing write tid 1 to smithi11541303-11 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: finishing write tid 2 to smithi11541303-11 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: finishing write tid 3 to smithi11541303-11 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: finishing write tid 4 to smithi11541303-11 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:237: snap_create 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: finishing write tid 5 to smithi11541303-11 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 53 (ObjNum 92 snap 15 seq_num 92) dirty exists 2022-04-22T17:26:11.190 INFO:tasks.rados.rados.0.smithi115.stdout:234: left oid 11 (ObjNum 92 snap 15 seq_num 92) 2022-04-22T17:26:11.193 INFO:tasks.rados.rados.0.smithi115.stdout:235: finishing write tid 1 to smithi11541303-20 2022-04-22T17:26:11.194 INFO:tasks.rados.rados.0.smithi115.stdout:235: finishing write tid 2 to smithi11541303-20 2022-04-22T17:26:11.194 INFO:tasks.rados.rados.0.smithi115.stdout:235: finishing write tid 3 to smithi11541303-20 2022-04-22T17:26:11.194 INFO:tasks.rados.rados.0.smithi115.stdout:235: finishing write tid 4 to smithi11541303-20 2022-04-22T17:26:11.196 INFO:tasks.rados.rados.0.smithi115.stdout:236: finishing copy_from to smithi11541303-28 2022-04-22T17:26:11.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:26:11.196 INFO:tasks.rados.rados.0.smithi115.stdout:235: finishing write tid 5 to smithi11541303-20 2022-04-22T17:26:11.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 88 (ObjNum 93 snap 15 seq_num 93) dirty exists 2022-04-22T17:26:11.197 INFO:tasks.rados.rados.0.smithi115.stdout:235: left oid 20 (ObjNum 93 snap 15 seq_num 93) 2022-04-22T17:26:11.197 INFO:tasks.rados.rados.0.smithi115.stdout:236: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:26:11.385 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[30140]: debug 2022-04-22T17:26:11.021+0000 7f15dcb0f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:26:11.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[12679]: cluster 2022-04-22T17:26:10.190674+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2022-04-22T17:26:11.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[12679]: cluster 2022-04-22T17:26:11.154963+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2022-04-22T17:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:11 smithi115 bash[18070]: cluster 2022-04-22T17:26:10.190674+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2022-04-22T17:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:11 smithi115 bash[18070]: cluster 2022-04-22T17:26:11.154963+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2022-04-22T17:26:11.545 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:26:11 smithi115 bash[26675]: debug 2022-04-22T17:26:11.179+0000 7faa010ac700 -1 osd.2 pg_epoch: 96 pg[5.ds0( v 95'29 (0'0,95'29] local-lis/les=63/64 n=6 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 crt=95'29 lcod 92'27 mlcod 92'27 active+clean+snaptrim trimq=[10~1] ps=[16~3,1b~1]] removing snap head 2022-04-22T17:26:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:11 smithi115 bash[12737]: cluster 2022-04-22T17:26:10.190674+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2022-04-22T17:26:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:11 smithi115 bash[12737]: cluster 2022-04-22T17:26:11.154963+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2022-04-22T17:26:11.664 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[30140]: debug 2022-04-22T17:26:11.377+0000 7f15dcb0f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:26:11.664 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[30140]: debug 2022-04-22T17:26:11.541+0000 7f15dcb0f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:26:11.960 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[30140]: debug 2022-04-22T17:26:11.656+0000 7f15dcb0f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:26:12.175 INFO:tasks.rados.rados.0.smithi115.stdout:234: done (3 left) 2022-04-22T17:26:12.175 INFO:tasks.rados.rados.0.smithi115.stdout:235: done (2 left) 2022-04-22T17:26:12.175 INFO:tasks.rados.rados.0.smithi115.stdout:236: done (1 left) 2022-04-22T17:26:12.175 INFO:tasks.rados.rados.0.smithi115.stdout:237: done (0 left) 2022-04-22T17:26:12.176 INFO:tasks.rados.rados.0.smithi115.stdout:238: snap_create 2022-04-22T17:26:12.266 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:11 smithi121 bash[30140]: debug 2022-04-22T17:26:11.952+0000 7f15dcb0f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:26:12.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:12 smithi121 bash[30140]: debug 2022-04-22T17:26:12.072+0000 7f15dcb0f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:26:12.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:12 smithi121 bash[30140]: debug 2022-04-22T17:26:12.180+0000 7f15dcb0f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:26:12.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:12 smithi121 bash[30140]: debug 2022-04-22T17:26:12.292+0000 7f15dcb0f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:26:13.075 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:12 smithi121 bash[30140]: debug 2022-04-22T17:26:12.804+0000 7f15dcb0f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:26:13.075 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:12 smithi121 bash[30140]: debug 2022-04-22T17:26:12.952+0000 7f15dcb0f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:26:13.178 INFO:tasks.rados.rados.0.smithi115.stdout:238: done (0 left) 2022-04-22T17:26:13.178 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 17 2022-04-22T17:26:13.178 INFO:tasks.rados.rados.0.smithi115.stdout:239: seq_num 94 ranges {270336=778240,1048576=532480,1581056=270336} 2022-04-22T17:26:13.184 INFO:tasks.rados.rados.0.smithi115.stdout:239: writing smithi11541303-17 from 270336 to 1048576 tid 1 2022-04-22T17:26:13.187 INFO:tasks.rados.rados.0.smithi115.stdout:239: writing smithi11541303-17 from 1048576 to 1581056 tid 2 2022-04-22T17:26:13.189 INFO:tasks.rados.rados.0.smithi115.stdout:239: writing smithi11541303-17 from 1581056 to 1851392 tid 3 2022-04-22T17:26:13.189 INFO:tasks.rados.rados.0.smithi115.stdout:240: rollback oid 8 current snap is 17 2022-04-22T17:26:13.189 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 15 2022-04-22T17:26:13.189 INFO:tasks.rados.rados.0.smithi115.stdout:241: rmattr oid 26 current snap is 17 2022-04-22T17:26:13.190 INFO:tasks.rados.rados.0.smithi115.stdout:241: done (2 left) 2022-04-22T17:26:13.190 INFO:tasks.rados.rados.0.smithi115.stdout:242: delete oid 2 current snap is 17 2022-04-22T17:26:13.195 INFO:tasks.rados.rados.0.smithi115.stdout:240: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:26:13.196 INFO:tasks.rados.rados.0.smithi115.stdout:240: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:26:13.196 INFO:tasks.rados.rados.0.smithi115.stdout:240: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:26:13.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 30 (ObjNum 7 snap 0 seq_num 7) dirty exists 2022-04-22T17:26:13.197 INFO:tasks.rados.rados.0.smithi115.stdout:240: done (2 left) 2022-04-22T17:26:13.197 INFO:tasks.rados.rados.0.smithi115.stdout:242: done (1 left) 2022-04-22T17:26:13.198 INFO:tasks.rados.rados.0.smithi115.stdout:243: rmattr oid 10 current snap is 17 2022-04-22T17:26:13.198 INFO:tasks.rados.rados.0.smithi115.stdout:243: done (1 left) 2022-04-22T17:26:13.198 INFO:tasks.rados.rados.0.smithi115.stdout:244: snap_create 2022-04-22T17:26:13.198 INFO:tasks.rados.rados.0.smithi115.stdout:239: finishing write tid 1 to smithi11541303-17 2022-04-22T17:26:13.198 INFO:tasks.rados.rados.0.smithi115.stdout:239: finishing write tid 2 to smithi11541303-17 2022-04-22T17:26:13.199 INFO:tasks.rados.rados.0.smithi115.stdout:239: finishing write tid 3 to smithi11541303-17 2022-04-22T17:26:13.199 INFO:tasks.rados.rados.0.smithi115.stdout:239: finishing write tid 4 to smithi11541303-17 2022-04-22T17:26:13.200 INFO:tasks.rados.rados.0.smithi115.stdout:239: finishing write tid 5 to smithi11541303-17 2022-04-22T17:26:13.200 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 93 (ObjNum 94 snap 17 seq_num 94) dirty exists 2022-04-22T17:26:13.200 INFO:tasks.rados.rados.0.smithi115.stdout:239: left oid 17 (ObjNum 94 snap 17 seq_num 94) 2022-04-22T17:26:13.436 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[30140]: debug 2022-04-22T17:26:13.068+0000 7f15dcb0f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:26:13.436 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[30140]: debug 2022-04-22T17:26:13.300+0000 7f15dcb0f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:26:13.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[12679]: cluster 2022-04-22T17:26:11.183628+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v373: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 50 MiB data, 136 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 30 op/s 2022-04-22T17:26:13.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[12679]: cluster 2022-04-22T17:26:12.168773+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2022-04-22T17:26:13.518 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[12737]: cluster 2022-04-22T17:26:11.183628+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v373: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 50 MiB data, 136 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 30 op/s 2022-04-22T17:26:13.519 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[12737]: cluster 2022-04-22T17:26:12.168773+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2022-04-22T17:26:13.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[18070]: cluster 2022-04-22T17:26:11.183628+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v373: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 50 MiB data, 136 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 30 op/s 2022-04-22T17:26:13.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[18070]: cluster 2022-04-22T17:26:12.168773+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2022-04-22T17:26:13.734 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[30140]: debug 2022-04-22T17:26:13.428+0000 7f15dcb0f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:26:13.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:13.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:26:13.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:13.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:13.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:13.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:26:14.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[30140]: debug 2022-04-22T17:26:13.728+0000 7f15dcb0f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:26:14.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:13 smithi121 bash[30140]: debug 2022-04-22T17:26:13.904+0000 7f15dcb0f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:26:14.198 INFO:tasks.rados.rados.0.smithi115.stdout:239: done (1 left) 2022-04-22T17:26:14.198 INFO:tasks.rados.rados.0.smithi115.stdout:244: done (0 left) 2022-04-22T17:26:14.198 INFO:tasks.rados.rados.0.smithi115.stdout:245: copy_from oid 48 from oid 47 current snap is 18 2022-04-22T17:26:14.198 INFO:tasks.rados.rados.0.smithi115.stdout:246: read oid 34 snap -1 2022-04-22T17:26:14.199 INFO:tasks.rados.rados.0.smithi115.stdout:246: expect (ObjNum 33 snap 0 seq_num 33) 2022-04-22T17:26:14.199 INFO:tasks.rados.rados.0.smithi115.stdout:247: snap_create 2022-04-22T17:26:14.205 INFO:tasks.rados.rados.0.smithi115.stdout:246: expect (ObjNum 33 snap 0 seq_num 33) 2022-04-22T17:26:14.220 INFO:tasks.rados.rados.0.smithi115.stdout:245: finishing copy_from to smithi11541303-48 2022-04-22T17:26:14.221 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:26:14.221 INFO:tasks.rados.rados.0.smithi115.stdout:245: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:26:14.486 INFO:tasks.rados.rados.0.smithi115.stdout:245: done (2 left) 2022-04-22T17:26:14.486 INFO:tasks.rados.rados.0.smithi115.stdout:246: done (1 left) 2022-04-22T17:26:14.487 INFO:tasks.rados.rados.0.smithi115.stdout:247: done (0 left) 2022-04-22T17:26:14.487 INFO:tasks.rados.rados.0.smithi115.stdout:248: read oid 47 snap -1 2022-04-22T17:26:14.487 INFO:tasks.rados.rados.0.smithi115.stdout:248: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:26:14.487 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 19 2022-04-22T17:26:14.487 INFO:tasks.rados.rados.0.smithi115.stdout:249: seq_num 96 ranges {557056=409600,966656=696320,1662976=507904} 2022-04-22T17:26:14.493 INFO:tasks.rados.rados.0.smithi115.stdout:249: writing smithi11541303-45 from 557056 to 966656 tid 1 2022-04-22T17:26:14.498 INFO:tasks.rados.rados.0.smithi115.stdout:249: writing smithi11541303-45 from 966656 to 1662976 tid 2 2022-04-22T17:26:14.502 INFO:tasks.rados.rados.0.smithi115.stdout:249: writing smithi11541303-45 from 1662976 to 2170880 tid 3 2022-04-22T17:26:14.502 INFO:tasks.rados.rados.0.smithi115.stdout:250: read oid 49 snap 18 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:250: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:251: read oid 28 snap -1 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:251: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:252: read oid 46 snap 16 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:252: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:253: snap_remove snap 15 2022-04-22T17:26:14.503 INFO:tasks.rados.rados.0.smithi115.stdout:248: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:26:14.514 INFO:tasks.rados.rados.0.smithi115.stdout:249: finishing write tid 1 to smithi11541303-45 2022-04-22T17:26:14.515 INFO:tasks.rados.rados.0.smithi115.stdout:249: finishing write tid 2 to smithi11541303-45 2022-04-22T17:26:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[12679]: cluster 2022-04-22T17:26:13.170713+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2022-04-22T17:26:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[12679]: cluster 2022-04-22T17:26:13.184306+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v376: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 51 MiB data, 140 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2022-04-22T17:26:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[12679]: audit 2022-04-22T17:26:14.164034+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:14.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[30140]: debug 2022-04-22T17:26:14.160+0000 7f15dcb0f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:26:14.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[30140]: debug 2022-04-22T17:26:14.408+0000 7f15dcb0f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:26:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[12737]: cluster 2022-04-22T17:26:13.170713+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2022-04-22T17:26:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[12737]: cluster 2022-04-22T17:26:13.184306+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v376: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 51 MiB data, 140 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2022-04-22T17:26:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[12737]: audit 2022-04-22T17:26:14.164034+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[18070]: cluster 2022-04-22T17:26:13.170713+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2022-04-22T17:26:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[18070]: cluster 2022-04-22T17:26:13.184306+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v376: 121 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 117 active+clean; 51 MiB data, 140 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2022-04-22T17:26:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:14 smithi115 bash[18070]: audit 2022-04-22T17:26:14.164034+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:14.914 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[30140]: debug 2022-04-22T17:26:14.652+0000 7f15dcb0f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:26:15.191 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:14 smithi121 bash[30140]: debug 2022-04-22T17:26:14.908+0000 7f15dcb0f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:26:15.191 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[30140]: debug 2022-04-22T17:26:15.148+0000 7f15dcb0f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:26:15.484 INFO:tasks.rados.rados.0.smithi115.stdout:248: done (5 left) 2022-04-22T17:26:15.484 INFO:tasks.rados.rados.0.smithi115.stdout:253: done (4 left) 2022-04-22T17:26:15.484 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 19 2022-04-22T17:26:15.484 INFO:tasks.rados.rados.0.smithi115.stdout:254: seq_num 97 ranges {122880=434176,557056=581632,1138688=540672,1679360=81920} 2022-04-22T17:26:15.490 INFO:tasks.rados.rados.0.smithi115.stdout:254: writing smithi11541303-8 from 122880 to 557056 tid 1 2022-04-22T17:26:15.493 INFO:tasks.rados.rados.0.smithi115.stdout:254: writing smithi11541303-8 from 557056 to 1138688 tid 2 2022-04-22T17:26:15.497 INFO:tasks.rados.rados.0.smithi115.stdout:254: writing smithi11541303-8 from 1138688 to 1679360 tid 3 2022-04-22T17:26:15.498 INFO:tasks.rados.rados.0.smithi115.stdout:254: writing smithi11541303-8 from 1679360 to 1761280 tid 4 2022-04-22T17:26:15.499 INFO:tasks.rados.rados.0.smithi115.stdout:255: read oid 12 snap -1 2022-04-22T17:26:15.499 INFO:tasks.rados.rados.0.smithi115.stdout:255: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:15.499 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 19 2022-04-22T17:26:15.499 INFO:tasks.rados.rados.0.smithi115.stdout:256: seq_num 98 ranges {1687552=450560,2138112=466944,2605056=737280} 2022-04-22T17:26:15.503 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: cluster 2022-04-22T17:26:14.191817+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2022-04-22T17:26:15.503 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: audit 2022-04-22T17:26:14.452354+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:15.503 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: audit 2022-04-22T17:26:14.461927+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: cephadm 2022-04-22T17:26:14.463869+0000 mgr.y (mgr.14146) 390 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: cephadm 2022-04-22T17:26:14.466966+0000 mgr.y (mgr.14146) 391 : cephadm [INF] Failing over to other MGR 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: audit 2022-04-22T17:26:14.467644+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: cluster 2022-04-22T17:26:14.477453+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:15 smithi121 bash[12679]: cluster 2022-04-22T17:26:14.945379+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: cluster 2022-04-22T17:26:14.191817+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: audit 2022-04-22T17:26:14.452354+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: audit 2022-04-22T17:26:14.461927+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: cephadm 2022-04-22T17:26:14.463869+0000 mgr.y (mgr.14146) 390 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: cephadm 2022-04-22T17:26:14.466966+0000 mgr.y (mgr.14146) 391 : cephadm [INF] Failing over to other MGR 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: audit 2022-04-22T17:26:14.467644+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: cluster 2022-04-22T17:26:14.477453+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2022-04-22T17:26:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[18070]: cluster 2022-04-22T17:26:14.945379+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: cluster 2022-04-22T17:26:14.191817+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: audit 2022-04-22T17:26:14.452354+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: audit 2022-04-22T17:26:14.461927+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: cephadm 2022-04-22T17:26:14.463869+0000 mgr.y (mgr.14146) 390 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: cephadm 2022-04-22T17:26:14.466966+0000 mgr.y (mgr.14146) 391 : cephadm [INF] Failing over to other MGR 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: audit 2022-04-22T17:26:14.467644+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: cluster 2022-04-22T17:26:14.477453+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2022-04-22T17:26:15.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[12737]: cluster 2022-04-22T17:26:14.945379+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:15.511 INFO:tasks.rados.rados.0.smithi115.stdout:256: writing smithi11541303-23 from 1687552 to 2138112 tid 1 2022-04-22T17:26:15.514 INFO:tasks.rados.rados.0.smithi115.stdout:256: writing smithi11541303-23 from 2138112 to 2605056 tid 2 2022-04-22T17:26:15.518 INFO:tasks.rados.rados.0.smithi115.stdout:256: writing smithi11541303-23 from 2605056 to 3342336 tid 3 2022-04-22T17:26:15.518 INFO:tasks.rados.rados.0.smithi115.stdout:257: setattr oid 50 current snap is 19 2022-04-22T17:26:15.519 INFO:tasks.rados.rados.0.smithi115.stdout:251: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:15.534 INFO:tasks.rados.rados.0.smithi115.stdout:249: finishing write tid 3 to smithi11541303-45 2022-04-22T17:26:15.534 INFO:tasks.rados.rados.0.smithi115.stdout:249: finishing write tid 4 to smithi11541303-45 2022-04-22T17:26:15.534 INFO:tasks.rados.rados.0.smithi115.stdout:249: finishing write tid 5 to smithi11541303-45 2022-04-22T17:26:15.535 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 43 (ObjNum 96 snap 19 seq_num 96) dirty exists 2022-04-22T17:26:15.535 INFO:tasks.rados.rados.0.smithi115.stdout:249: left oid 45 (ObjNum 96 snap 19 seq_num 96) 2022-04-22T17:26:15.535 INFO:tasks.rados.rados.0.smithi115.stdout:252: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:15.554 INFO:tasks.rados.rados.0.smithi115.stdout:250: expect (ObjNum 55 snap 4 seq_num 55) 2022-04-22T17:26:15.617 INFO:tasks.rados.rados.0.smithi115.stdout:255: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:15.622 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 1 to smithi11541303-8 2022-04-22T17:26:15.622 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 2 to smithi11541303-8 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 3 to smithi11541303-8 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 4 to smithi11541303-8 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 5 to smithi11541303-8 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:254: finishing write tid 6 to smithi11541303-8 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 38 (ObjNum 97 snap 19 seq_num 97) dirty exists 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:254: left oid 8 (ObjNum 97 snap 19 seq_num 97) 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:256: finishing write tid 1 to smithi11541303-23 2022-04-22T17:26:15.623 INFO:tasks.rados.rados.0.smithi115.stdout:256: finishing write tid 2 to smithi11541303-23 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:256: finishing write tid 3 to smithi11541303-23 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:256: finishing write tid 4 to smithi11541303-23 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:256: finishing write tid 5 to smithi11541303-23 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 58 (ObjNum 98 snap 19 seq_num 98) dirty exists 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:256: left oid 23 (ObjNum 98 snap 19 seq_num 98) 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:249: done (7 left) 2022-04-22T17:26:15.624 INFO:tasks.rados.rados.0.smithi115.stdout:250: done (6 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:251: done (5 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:252: done (4 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:254: done (3 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:255: done (2 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:256: done (1 left) 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 19 2022-04-22T17:26:15.625 INFO:tasks.rados.rados.0.smithi115.stdout:258: seq_num 99 ranges {1417216=466944,1884160=802816,2686976=401408} 2022-04-22T17:26:15.636 INFO:tasks.rados.rados.0.smithi115.stdout:258: writing smithi11541303-19 from 1417216 to 1884160 tid 1 2022-04-22T17:26:15.648 INFO:tasks.rados.rados.0.smithi115.stdout:258: writing smithi11541303-19 from 1884160 to 2686976 tid 2 2022-04-22T17:26:15.654 INFO:tasks.rados.rados.0.smithi115.stdout:258: writing smithi11541303-19 from 2686976 to 3088384 tid 3 2022-04-22T17:26:15.654 INFO:tasks.rados.rados.0.smithi115.stdout:259: read oid 1 snap 18 2022-04-22T17:26:15.654 INFO:tasks.rados.rados.0.smithi115.stdout:259: expect (ObjNum 77 snap 14 seq_num 77) 2022-04-22T17:26:15.654 INFO:tasks.rados.rados.0.smithi115.stdout:260: snap_remove snap 16 2022-04-22T17:26:15.794 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[13018]: ignoring --setuser ceph since I am not root 2022-04-22T17:26:15.794 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[13018]: ignoring --setgroup ceph since I am not root 2022-04-22T17:26:15.794 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:26:15 smithi115 bash[13018]: debug 2022-04-22T17:26:15.510+0000 7f5fec56d700 1 -- 172.21.15.115:0/3615791950 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b6761a4e0 con 0x559b6770ac00 2022-04-22T17:26:16.167 INFO:tasks.rados.rados.0.smithi115.stdout:260: done (3 left) 2022-04-22T17:26:16.167 INFO:tasks.rados.rados.0.smithi115.stdout:261: snap_remove snap 10 2022-04-22T17:26:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:16 smithi121 bash[12679]: audit 2022-04-22T17:26:15.462693+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2022-04-22T17:26:16.840 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:16 smithi121 bash[12679]: cluster 2022-04-22T17:26:15.462768+0000 mon.a (mon.0) 796 : cluster [DBG] mgrmap e15: x(active, starting, since 0.993829s), standbys: y 2022-04-22T17:26:16.840 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:16 smithi121 bash[12679]: cluster 2022-04-22T17:26:15.477301+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2022-04-22T17:26:16.840 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:16 smithi121 bash[12679]: cluster 2022-04-22T17:26:16.160300+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2022-04-22T17:26:16.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[12737]: audit 2022-04-22T17:26:15.462693+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2022-04-22T17:26:16.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[12737]: cluster 2022-04-22T17:26:15.462768+0000 mon.a (mon.0) 796 : cluster [DBG] mgrmap e15: x(active, starting, since 0.993829s), standbys: y 2022-04-22T17:26:16.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[12737]: cluster 2022-04-22T17:26:15.477301+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2022-04-22T17:26:16.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[12737]: cluster 2022-04-22T17:26:16.160300+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2022-04-22T17:26:16.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[18070]: audit 2022-04-22T17:26:15.462693+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14146 172.21.15.115:0/218914741' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2022-04-22T17:26:16.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[18070]: cluster 2022-04-22T17:26:15.462768+0000 mon.a (mon.0) 796 : cluster [DBG] mgrmap e15: x(active, starting, since 0.993829s), standbys: y 2022-04-22T17:26:16.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[18070]: cluster 2022-04-22T17:26:15.477301+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2022-04-22T17:26:16.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:16 smithi115 bash[18070]: cluster 2022-04-22T17:26:16.160300+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2022-04-22T17:26:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:261: done (3 left) 2022-04-22T17:26:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 19 2022-04-22T17:26:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 32 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:26:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:258: finishing write tid 1 to smithi11541303-19 2022-04-22T17:26:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:258: finishing write tid 2 to smithi11541303-19 2022-04-22T17:26:17.168 INFO:tasks.rados.rados.0.smithi115.stdout:258: finishing write tid 3 to smithi11541303-19 2022-04-22T17:26:17.168 INFO:tasks.rados.rados.0.smithi115.stdout:262: seq_num 100 ranges {3342336=483328,3825664=688128,4513792=516096} 2022-04-22T17:26:17.190 INFO:tasks.rados.rados.0.smithi115.stdout:262: writing smithi11541303-23 from 3342336 to 3825664 tid 1 2022-04-22T17:26:17.194 INFO:tasks.rados.rados.0.smithi115.stdout:262: writing smithi11541303-23 from 3825664 to 4513792 tid 2 2022-04-22T17:26:17.197 INFO:tasks.rados.rados.0.smithi115.stdout:262: writing smithi11541303-23 from 4513792 to 5029888 tid 3 2022-04-22T17:26:17.197 INFO:tasks.rados.rados.0.smithi115.stdout:257: done (3 left) 2022-04-22T17:26:17.198 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 19 2022-04-22T17:26:17.198 INFO:tasks.rados.rados.0.smithi115.stdout:258: finishing write tid 4 to smithi11541303-19 2022-04-22T17:26:17.198 INFO:tasks.rados.rados.0.smithi115.stdout:263: seq_num 101 ranges {1736704=499712,2236416=573440,2809856=524288,3334144=106496} 2022-04-22T17:26:17.210 INFO:tasks.rados.rados.0.smithi115.stdout:263: writing smithi11541303-49 from 1736704 to 2236416 tid 1 2022-04-22T17:26:17.213 INFO:tasks.rados.rados.0.smithi115.stdout:263: writing smithi11541303-49 from 2236416 to 2809856 tid 2 2022-04-22T17:26:17.217 INFO:tasks.rados.rados.0.smithi115.stdout:263: writing smithi11541303-49 from 2809856 to 3334144 tid 3 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:263: writing smithi11541303-49 from 3334144 to 3440640 tid 4 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:258: finishing write tid 5 to smithi11541303-19 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 24 (ObjNum 99 snap 19 seq_num 99) dirty exists 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:258: left oid 19 (ObjNum 99 snap 19 seq_num 99) 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:264: read oid 12 snap -1 2022-04-22T17:26:17.218 INFO:tasks.rados.rados.0.smithi115.stdout:264: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:17.219 INFO:tasks.rados.rados.0.smithi115.stdout:258: done (4 left) 2022-04-22T17:26:17.219 INFO:tasks.rados.rados.0.smithi115.stdout:265: read oid 37 snap 18 2022-04-22T17:26:17.219 INFO:tasks.rados.rados.0.smithi115.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2022-04-22T17:26:17.219 INFO:tasks.rados.rados.0.smithi115.stdout:266: setattr oid 4 current snap is 19 2022-04-22T17:26:17.219 INFO:tasks.rados.rados.0.smithi115.stdout:259: expect (ObjNum 77 snap 14 seq_num 77) 2022-04-22T17:26:17.258 INFO:tasks.rados.rados.0.smithi115.stdout:262: finishing write tid 1 to smithi11541303-23 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:262: finishing write tid 2 to smithi11541303-23 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:262: finishing write tid 3 to smithi11541303-23 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:262: finishing write tid 4 to smithi11541303-23 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:262: finishing write tid 5 to smithi11541303-23 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 70 (ObjNum 100 snap 19 seq_num 100) dirty exists 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:262: left oid 23 (ObjNum 100 snap 19 seq_num 100) 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 1 to smithi11541303-49 2022-04-22T17:26:17.259 INFO:tasks.rados.rados.0.smithi115.stdout:264: expect (ObjNum 11 snap 0 seq_num 11) 2022-04-22T17:26:17.263 INFO:tasks.rados.rados.0.smithi115.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2022-04-22T17:26:17.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:17 smithi121 bash[30140]: debug 2022-04-22T17:26:17.000+0000 7f15dcb0f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:26:17.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:17 smithi121 bash[30140]: debug 2022-04-22T17:26:17.108+0000 7f15dcb0f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:26:17.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:26:17 smithi121 bash[18604]: debug 2022-04-22T17:26:17.164+0000 7faa0f096700 -1 osd.5 pg_epoch: 103 pg[5.7s0( v 103'107 (0'0,103'107] local-lis/les=63/64 n=8 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [5,1,0]p5(0) r=0 lpr=63 luod=96'105 crt=96'105 lcod 96'103 mlcod 96'103 active+clean+snaptrim trimq=[14~1] ps=[10~1,17~1,1b~1,1f~2]] removing snap head 2022-04-22T17:26:17.268 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:26:17 smithi121 bash[24219]: debug 2022-04-22T17:26:17.164+0000 7f1df1444700 -1 osd.7 pg_epoch: 103 pg[5.9s0( v 103'53 (0'0,103'53] local-lis/les=63/64 n=9 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [7,6,1]p7(0) r=0 lpr=63 luod=102'51 crt=102'51 lcod 102'49 mlcod 102'49 active+clean+snaptrim trimq=[14~1] ps=[10~1,17~1,1b~1,1f~2]] removing snap head 2022-04-22T17:26:17.268 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:26:17 smithi121 bash[15740]: debug 2022-04-22T17:26:17.160+0000 7feef140b700 -1 osd.4 pg_epoch: 103 pg[5.es0( v 96'21 (0'0,96'21] local-lis/les=63/64 n=4 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [4,5,0]p4(0) r=0 lpr=63 crt=96'21 lcod 95'19 mlcod 95'19 active+clean+snaptrim trimq=[14~1] ps=[10~1,17~1,1b~1,1f~2]] removing snap head 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 2 to smithi11541303-49 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 3 to smithi11541303-49 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 4 to smithi11541303-49 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 5 to smithi11541303-49 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:263: finishing write tid 6 to smithi11541303-49 2022-04-22T17:26:17.281 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 117 (ObjNum 101 snap 19 seq_num 101) dirty exists 2022-04-22T17:26:17.282 INFO:tasks.rados.rados.0.smithi115.stdout:263: left oid 49 (ObjNum 101 snap 19 seq_num 101) 2022-04-22T17:26:17.282 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:26:17.282 INFO:tasks.rados.rados.0.smithi115.stdout:259: done (5 left) 2022-04-22T17:26:17.282 INFO:tasks.rados.rados.0.smithi115.stdout:262: done (4 left) 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:263: done (3 left) 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:264: done (2 left) 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:265: done (1 left) 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:266: done (0 left) 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:267: rollback oid 32 current snap is 19 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 18 2022-04-22T17:26:17.283 INFO:tasks.rados.rados.0.smithi115.stdout:268: delete oid 49 current snap is 19 2022-04-22T17:26:17.284 INFO:tasks.rados.rados.0.smithi115.stdout:268: done (1 left) 2022-04-22T17:26:17.284 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 19 2022-04-22T17:26:17.284 INFO:tasks.rados.rados.0.smithi115.stdout:269: seq_num 103 ranges {573440=532480,1105920=737280,1843200=466944} 2022-04-22T17:26:17.295 INFO:tasks.rados.rados.0.smithi115.stdout:269: writing smithi11541303-35 from 573440 to 1105920 tid 1 2022-04-22T17:26:17.299 INFO:tasks.rados.rados.0.smithi115.stdout:269: writing smithi11541303-35 from 1105920 to 1843200 tid 2 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:269: writing smithi11541303-35 from 1843200 to 2310144 tid 3 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:270: setattr oid 42 current snap is 19 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:267: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:267: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:267: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 27 (ObjNum 31 snap 0 seq_num 31) dirty exists 2022-04-22T17:26:17.303 INFO:tasks.rados.rados.0.smithi115.stdout:269: finishing write tid 1 to smithi11541303-35 2022-04-22T17:26:17.304 INFO:tasks.rados.rados.0.smithi115.stdout:267: done (2 left) 2022-04-22T17:26:17.304 INFO:tasks.rados.rados.0.smithi115.stdout:271: setattr oid 3 current snap is 19 2022-04-22T17:26:17.305 INFO:tasks.rados.rados.0.smithi115.stdout:272: snap_remove snap 17 2022-04-22T17:26:17.544 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:26:17 smithi115 bash[26675]: debug 2022-04-22T17:26:17.166+0000 7faa060b6700 -1 osd.2 pg_epoch: 103 pg[5.bs0( v 95'20 (0'0,95'20] local-lis/les=63/64 n=2 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,0,5]p2(0) r=0 lpr=63 crt=95'20 lcod 94'18 mlcod 94'18 active+clean+snaptrim trimq=[14~1] ps=[10~1,17~1,1b~1,1f~2]] removing snap head 2022-04-22T17:26:17.544 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:26:17 smithi115 bash[26675]: debug 2022-04-22T17:26:17.178+0000 7faa010ac700 -1 osd.2 pg_epoch: 103 pg[5.ds0( v 103'41 (0'0,103'41] local-lis/les=63/64 n=4 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 luod=102'39 crt=102'39 lcod 101'37 mlcod 101'37 active+clean+snaptrim trimq=[14~1] ps=[10~1,17~1,1b~1,1f~2]] removing snap head 2022-04-22T17:26:18.171 INFO:tasks.rados.rados.0.smithi115.stdout:272: done (3 left) 2022-04-22T17:26:18.172 INFO:tasks.rados.rados.0.smithi115.stdout:273: read oid 49 snap -1 2022-04-22T17:26:18.172 INFO:tasks.rados.rados.0.smithi115.stdout:273: expect deleted 2022-04-22T17:26:18.172 INFO:tasks.rados.rados.0.smithi115.stdout:274: read oid 43 snap -1 2022-04-22T17:26:18.172 INFO:tasks.rados.rados.0.smithi115.stdout:274: expect (ObjNum 65 snap 6 seq_num 65) 2022-04-22T17:26:18.172 INFO:tasks.rados.rados.0.smithi115.stdout:275: snap_create 2022-04-22T17:26:18.173 INFO:tasks.rados.rados.0.smithi115.stdout:269: finishing write tid 2 to smithi11541303-35 2022-04-22T17:26:18.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 120 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:26:18.173 INFO:tasks.rados.rados.0.smithi115.stdout:269: finishing write tid 3 to smithi11541303-35 2022-04-22T17:26:18.173 INFO:tasks.rados.rados.0.smithi115.stdout:269: finishing write tid 4 to smithi11541303-35 2022-04-22T17:26:18.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 72 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:18.174 INFO:tasks.rados.rados.0.smithi115.stdout:269: finishing write tid 5 to smithi11541303-35 2022-04-22T17:26:18.174 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 48 (ObjNum 103 snap 19 seq_num 103) dirty exists 2022-04-22T17:26:18.174 INFO:tasks.rados.rados.0.smithi115.stdout:269: left oid 35 (ObjNum 103 snap 19 seq_num 103) 2022-04-22T17:26:18.191 INFO:tasks.rados.rados.0.smithi115.stdout:274: expect (ObjNum 65 snap 6 seq_num 65) 2022-04-22T17:26:18.450 INFO:tasks.rados.rados.0.smithi115.stdout:269: done (5 left) 2022-04-22T17:26:18.450 INFO:tasks.rados.rados.0.smithi115.stdout:270: done (4 left) 2022-04-22T17:26:18.451 INFO:tasks.rados.rados.0.smithi115.stdout:271: done (3 left) 2022-04-22T17:26:18.451 INFO:tasks.rados.rados.0.smithi115.stdout:273: done (2 left) 2022-04-22T17:26:18.451 INFO:tasks.rados.rados.0.smithi115.stdout:274: done (1 left) 2022-04-22T17:26:18.452 INFO:tasks.rados.rados.0.smithi115.stdout:275: done (0 left) 2022-04-22T17:26:18.452 INFO:tasks.rados.rados.0.smithi115.stdout:276: rollback oid 30 current snap is 20 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 18 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:277: copy_from oid 9 from oid 25 current snap is 20 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:278: copy_from oid 12 from oid 28 current snap is 20 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:279: read oid 1 snap -1 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:279: expect (ObjNum 77 snap 14 seq_num 77) 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 20 2022-04-22T17:26:18.453 INFO:tasks.rados.rados.0.smithi115.stdout:280: seq_num 104 ranges {524288=548864,1073152=622592,1695744=581632} 2022-04-22T17:26:18.459 INFO:tasks.rados.rados.0.smithi115.stdout:280: writing smithi11541303-32 from 524288 to 1073152 tid 1 2022-04-22T17:26:18.464 INFO:tasks.rados.rados.0.smithi115.stdout:280: writing smithi11541303-32 from 1073152 to 1695744 tid 2 2022-04-22T17:26:18.467 INFO:tasks.rados.rados.0.smithi115.stdout:280: writing smithi11541303-32 from 1695744 to 2277376 tid 3 2022-04-22T17:26:18.468 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 20 2022-04-22T17:26:18.468 INFO:tasks.rados.rados.0.smithi115.stdout:281: seq_num 105 ranges {172032=565248,737280=507904,1245184=507904,1753088=188416} 2022-04-22T17:26:18.473 INFO:tasks.rados.rados.0.smithi115.stdout:281: writing smithi11541303-36 from 172032 to 737280 tid 1 2022-04-22T17:26:18.475 INFO:tasks.rados.rados.0.smithi115.stdout:281: writing smithi11541303-36 from 737280 to 1245184 tid 2 2022-04-22T17:26:18.478 INFO:tasks.rados.rados.0.smithi115.stdout:281: writing smithi11541303-36 from 1245184 to 1753088 tid 3 2022-04-22T17:26:18.479 INFO:tasks.rados.rados.0.smithi115.stdout:281: writing smithi11541303-36 from 1753088 to 1941504 tid 4 2022-04-22T17:26:18.480 INFO:tasks.rados.rados.0.smithi115.stdout:282: rollback oid 2 current snap is 20 2022-04-22T17:26:18.480 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 2 to 18 2022-04-22T17:26:18.480 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 20 2022-04-22T17:26:18.480 INFO:tasks.rados.rados.0.smithi115.stdout:283: seq_num 106 ranges {761856=581632,1343488=794624,2138112=409600} 2022-04-22T17:26:18.487 INFO:tasks.rados.rados.0.smithi115.stdout:283: writing smithi11541303-18 from 761856 to 1343488 tid 1 2022-04-22T17:26:18.491 INFO:tasks.rados.rados.0.smithi115.stdout:283: writing smithi11541303-18 from 1343488 to 2138112 tid 2 2022-04-22T17:26:18.494 INFO:tasks.rados.rados.0.smithi115.stdout:283: writing smithi11541303-18 from 2138112 to 2547712 tid 3 2022-04-22T17:26:18.494 INFO:tasks.rados.rados.0.smithi115.stdout:284: read oid 50 snap -1 2022-04-22T17:26:18.494 INFO:tasks.rados.rados.0.smithi115.stdout:284: expect (ObjNum 90 snap 15 seq_num 90) 2022-04-22T17:26:18.494 INFO:tasks.rados.rados.0.smithi115.stdout:285: snap_remove snap 19 2022-04-22T17:26:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:18 smithi121 bash[12679]: cluster 2022-04-22T17:26:17.159555+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2022-04-22T17:26:18.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:18 smithi121 bash[30140]: debug 2022-04-22T17:26:18.412+0000 7f15dcb0f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:26:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:18 smithi115 bash[12737]: cluster 2022-04-22T17:26:17.159555+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2022-04-22T17:26:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:18 smithi115 bash[18070]: cluster 2022-04-22T17:26:17.159555+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2022-04-22T17:26:19.436 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: cluster 2022-04-22T17:26:18.160769+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2022-04-22T17:26:19.436 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: cluster 2022-04-22T17:26:18.434626+0000 mon.a (mon.0) 801 : cluster [INF] Active manager daemon x restarted 2022-04-22T17:26:19.436 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: cluster 2022-04-22T17:26:18.435760+0000 mon.a (mon.0) 802 : cluster [INF] Activating manager daemon x 2022-04-22T17:26:19.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: audit 2022-04-22T17:26:18.439487+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:26:19.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: cluster 2022-04-22T17:26:18.441521+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2022-04-22T17:26:19.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: audit 2022-04-22T17:26:18.447399+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:19.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: audit 2022-04-22T17:26:18.448689+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:26:19.438 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[12679]: audit 2022-04-22T17:26:18.449085+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:19.438 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154dc01700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.438 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154dc01700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.438 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154dc01700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154dc01700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154dc01700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154abfb700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154abfb700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154abfb700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154abfb700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:19 smithi121 bash[30140]: debug 2022-04-22T17:26:19.368+0000 7f154abfb700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:26:19.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: cluster 2022-04-22T17:26:18.160769+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2022-04-22T17:26:19.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: cluster 2022-04-22T17:26:18.434626+0000 mon.a (mon.0) 801 : cluster [INF] Active manager daemon x restarted 2022-04-22T17:26:19.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: cluster 2022-04-22T17:26:18.435760+0000 mon.a (mon.0) 802 : cluster [INF] Activating manager daemon x 2022-04-22T17:26:19.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: audit 2022-04-22T17:26:18.439487+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:26:19.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: cluster 2022-04-22T17:26:18.441521+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: audit 2022-04-22T17:26:18.447399+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: audit 2022-04-22T17:26:18.448689+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[12737]: audit 2022-04-22T17:26:18.449085+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: cluster 2022-04-22T17:26:18.160769+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: cluster 2022-04-22T17:26:18.434626+0000 mon.a (mon.0) 801 : cluster [INF] Active manager daemon x restarted 2022-04-22T17:26:19.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: cluster 2022-04-22T17:26:18.435760+0000 mon.a (mon.0) 802 : cluster [INF] Activating manager daemon x 2022-04-22T17:26:19.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: audit 2022-04-22T17:26:18.439487+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:26:19.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: cluster 2022-04-22T17:26:18.441521+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2022-04-22T17:26:19.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: audit 2022-04-22T17:26:18.447399+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:19.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: audit 2022-04-22T17:26:18.448689+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:26:19.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:19 smithi115 bash[18070]: audit 2022-04-22T17:26:18.449085+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:19.451 INFO:tasks.rados.rados.0.smithi115.stdout:285: done (9 left) 2022-04-22T17:26:19.452 INFO:tasks.rados.rados.0.smithi115.stdout:286: copy_from oid 17 from oid 13 current snap is 20 2022-04-22T17:26:19.452 INFO:tasks.rados.rados.0.smithi115.stdout:279: expect (ObjNum 77 snap 14 seq_num 77) 2022-04-22T17:26:19.491 INFO:tasks.rados.rados.0.smithi115.stdout:278: finishing copy_from to smithi11541303-12 2022-04-22T17:26:19.504 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:278: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:277: finishing copy_from to smithi11541303-9 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:277: got expected ENOENT (src dne) 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:278: done (9 left) 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:279: done (8 left) 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:287: rollback oid 8 current snap is 20 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 18 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:280: finishing write tid 1 to smithi11541303-32 2022-04-22T17:26:19.505 INFO:tasks.rados.rados.0.smithi115.stdout:277: finishing copy_from racing read to smithi11541303-9 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:280: finishing write tid 2 to smithi11541303-32 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:276: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:276: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:276: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 68 (ObjNum 66 snap 6 seq_num 66) dirty exists 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:280: finishing write tid 3 to smithi11541303-32 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:280: finishing write tid 4 to smithi11541303-32 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:280: finishing write tid 5 to smithi11541303-32 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 34 (ObjNum 104 snap 20 seq_num 104) dirty exists 2022-04-22T17:26:19.506 INFO:tasks.rados.rados.0.smithi115.stdout:280: left oid 32 (ObjNum 104 snap 20 seq_num 104) 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 1 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 2 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:282: finishing rollback tid 1 to smithi11541303-2 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 0 (ObjNum 1349149548 snap 1566724693 seq_num 1919577200) dirty dne 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 3 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 4 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 5 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: finishing write tid 6 to smithi11541303-36 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 50 (ObjNum 105 snap 20 seq_num 105) dirty exists 2022-04-22T17:26:19.507 INFO:tasks.rados.rados.0.smithi115.stdout:281: left oid 36 (ObjNum 105 snap 20 seq_num 105) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:283: finishing write tid 1 to smithi11541303-18 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:283: finishing write tid 2 to smithi11541303-18 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:283: finishing write tid 3 to smithi11541303-18 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:276: done (8 left) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:277: done (7 left) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:280: done (6 left) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:281: done (5 left) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:282: done (4 left) 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:288: rollback oid 12 current snap is 20 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 18 2022-04-22T17:26:19.508 INFO:tasks.rados.rados.0.smithi115.stdout:283: finishing write tid 4 to smithi11541303-18 2022-04-22T17:26:19.509 INFO:tasks.rados.rados.0.smithi115.stdout:283: finishing write tid 5 to smithi11541303-18 2022-04-22T17:26:19.509 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 129 (ObjNum 106 snap 20 seq_num 106) dirty exists 2022-04-22T17:26:19.509 INFO:tasks.rados.rados.0.smithi115.stdout:283: left oid 18 (ObjNum 106 snap 20 seq_num 106) 2022-04-22T17:26:19.509 INFO:tasks.rados.rados.0.smithi115.stdout:284: expect (ObjNum 90 snap 15 seq_num 90) 2022-04-22T17:26:19.571 INFO:tasks.rados.rados.0.smithi115.stdout:286: finishing copy_from to smithi11541303-17 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 68 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:286: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:288: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:288: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:288: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 35 (ObjNum 11 snap 0 seq_num 11) dirty exists 2022-04-22T17:26:19.572 INFO:tasks.rados.rados.0.smithi115.stdout:287: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:26:19.573 INFO:tasks.rados.rados.0.smithi115.stdout:287: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:26:19.573 INFO:tasks.rados.rados.0.smithi115.stdout:287: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:26:19.573 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 53 (ObjNum 7 snap 0 seq_num 7) dirty exists 2022-04-22T17:26:19.573 INFO:tasks.rados.rados.0.smithi115.stdout:283: done (4 left) 2022-04-22T17:26:19.573 INFO:tasks.rados.rados.0.smithi115.stdout:284: done (3 left) 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:286: done (2 left) 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:287: done (1 left) 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:288: done (0 left) 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:289: rmattr oid 30 current snap is 20 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:289: done (0 left) 2022-04-22T17:26:19.574 INFO:tasks.rados.rados.0.smithi115.stdout:290: rollback oid 3 current snap is 20 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 18 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:291: rmattr oid 47 current snap is 20 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:291: done (1 left) 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:292: copy_from oid 18 from oid 29 current snap is 20 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:293: rollback oid 38 current snap is 20 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 18 2022-04-22T17:26:19.575 INFO:tasks.rados.rados.0.smithi115.stdout:294: snap_create 2022-04-22T17:26:19.579 INFO:tasks.rados.rados.0.smithi115.stdout:293: finishing rollback tid 0 to smithi11541303-38 2022-04-22T17:26:19.579 INFO:tasks.rados.rados.0.smithi115.stdout:290: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:26:19.579 INFO:tasks.rados.rados.0.smithi115.stdout:293: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:26:19.580 INFO:tasks.rados.rados.0.smithi115.stdout:293: finishing rollback tid 2 to smithi11541303-38 2022-04-22T17:26:19.580 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 41 (ObjNum 37 snap 0 seq_num 37) dirty exists 2022-04-22T17:26:19.581 INFO:tasks.rados.rados.0.smithi115.stdout:290: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:26:19.581 INFO:tasks.rados.rados.0.smithi115.stdout:290: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:26:19.581 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 79 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:19.582 INFO:tasks.rados.rados.0.smithi115.stdout:292: finishing copy_from to smithi11541303-18 2022-04-22T17:26:19.582 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 33 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:26:19.582 INFO:tasks.rados.rados.0.smithi115.stdout:292: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:26:20.439 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[30140]: [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:26:20.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: cluster 2022-04-22T17:26:19.182065+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e16: x(active, starting, since 0.74663s), standbys: y 2022-04-22T17:26:20.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.189551+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:20.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.189856+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:20.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.190110+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:20.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.191371+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:26:20.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.191682+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:26:20.442 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.192230+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:26:20.442 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.192483+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:26:20.442 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.192753+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:26:20.442 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.193102+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:26:20.443 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.193406+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:26:20.443 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.193695+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:26:20.443 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.193971+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:26:20.443 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.194258+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:26:20.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.194553+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:26:20.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.194850+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:26:20.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.196082+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:26:20.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.196578+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:26:20.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.197294+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:26:20.445 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: cluster 2022-04-22T17:26:19.247045+0000 mon.a (mon.0) 805 : cluster [INF] Manager daemon x is now available 2022-04-22T17:26:20.445 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.282047+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.445 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.286926+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.445 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.288884+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:20.446 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.291114+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.446 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.293730+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.446 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.294980+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:20.446 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.310739+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.313357+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.321080+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.323769+0000 mon.b (mon.2) 48 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.448 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.349116+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.445080+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: cluster 2022-04-22T17:26:19.445205+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.452466+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: audit 2022-04-22T17:26:19.455187+0000 mon.b (mon.2) 49 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: cephadm 2022-04-22T17:26:19.774926+0000 mgr.x (mgr.24749) 1 : cephadm [INF] Deploying cephadm binary to smithi115 2022-04-22T17:26:20.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[12679]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:26:20.458 INFO:tasks.rados.rados.0.smithi115.stdout:290: done (3 left) 2022-04-22T17:26:20.459 INFO:tasks.rados.rados.0.smithi115.stdout:292: done (2 left) 2022-04-22T17:26:20.459 INFO:tasks.rados.rados.0.smithi115.stdout:293: done (1 left) 2022-04-22T17:26:20.459 INFO:tasks.rados.rados.0.smithi115.stdout:294: done (0 left) 2022-04-22T17:26:20.459 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 21 2022-04-22T17:26:20.459 INFO:tasks.rados.rados.0.smithi115.stdout:295: seq_num 107 ranges {1417216=598016,2015232=729088,2744320=466944,3211264=8192} 2022-04-22T17:26:20.473 INFO:tasks.rados.rados.0.smithi115.stdout:295: writing smithi11541303-16 from 1417216 to 2015232 tid 1 2022-04-22T17:26:20.477 INFO:tasks.rados.rados.0.smithi115.stdout:295: writing smithi11541303-16 from 2015232 to 2744320 tid 2 2022-04-22T17:26:20.481 INFO:tasks.rados.rados.0.smithi115.stdout:295: writing smithi11541303-16 from 2744320 to 3211264 tid 3 2022-04-22T17:26:20.481 INFO:tasks.rados.rados.0.smithi115.stdout:295: writing smithi11541303-16 from 3211264 to 3219456 tid 4 2022-04-22T17:26:20.481 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 21 2022-04-22T17:26:20.481 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 1 to smithi11541303-16 2022-04-22T17:26:20.481 INFO:tasks.rados.rados.0.smithi115.stdout:296: seq_num 108 ranges {778240=614400,1392640=606208,1998848=598016} 2022-04-22T17:26:20.489 INFO:tasks.rados.rados.0.smithi115.stdout:296: writing smithi11541303-18 from 778240 to 1392640 tid 1 2022-04-22T17:26:20.492 INFO:tasks.rados.rados.0.smithi115.stdout:296: writing smithi11541303-18 from 1392640 to 1998848 tid 2 2022-04-22T17:26:20.496 INFO:tasks.rados.rados.0.smithi115.stdout:296: writing smithi11541303-18 from 1998848 to 2596864 tid 3 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:297: setattr oid 45 current snap is 21 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 2 to smithi11541303-16 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 3 to smithi11541303-16 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 4 to smithi11541303-16 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 5 to smithi11541303-16 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: finishing write tid 6 to smithi11541303-16 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 46 (ObjNum 107 snap 21 seq_num 107) dirty exists 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:295: left oid 16 (ObjNum 107 snap 21 seq_num 107) 2022-04-22T17:26:20.497 INFO:tasks.rados.rados.0.smithi115.stdout:296: finishing write tid 1 to smithi11541303-18 2022-04-22T17:26:20.498 INFO:tasks.rados.rados.0.smithi115.stdout:296: finishing write tid 2 to smithi11541303-18 2022-04-22T17:26:20.498 INFO:tasks.rados.rados.0.smithi115.stdout:295: done (2 left) 2022-04-22T17:26:20.499 INFO:tasks.rados.rados.0.smithi115.stdout:298: snap_remove snap 18 2022-04-22T17:26:20.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: cluster 2022-04-22T17:26:19.182065+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e16: x(active, starting, since 0.74663s), standbys: y 2022-04-22T17:26:20.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.189551+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:20.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.189856+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.190110+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.191371+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.191682+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.192230+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.192483+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.192753+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.193102+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.193406+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:26:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.193695+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.193971+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.194258+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.194553+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.194850+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.196082+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.196578+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:26:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: cluster 2022-04-22T17:26:19.182065+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e16: x(active, starting, since 0.74663s), standbys: y 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.189551+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.189856+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.190110+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.191371+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.191682+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.192230+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.192483+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.192753+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.193102+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:26:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.193406+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.193695+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.193971+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.194258+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.194553+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.194850+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.196082+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.196578+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:26:20.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.197294+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: cluster 2022-04-22T17:26:19.247045+0000 mon.a (mon.0) 805 : cluster [INF] Manager daemon x is now available 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.282047+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.286926+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.288884+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.291114+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.293730+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.294980+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.310739+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.313357+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.321080+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.323769+0000 mon.b (mon.2) 48 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.349116+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.445080+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: cluster 2022-04-22T17:26:19.445205+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.452466+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: audit 2022-04-22T17:26:19.455187+0000 mon.b (mon.2) 49 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: cephadm 2022-04-22T17:26:19.774926+0000 mgr.x (mgr.24749) 1 : cephadm [INF] Deploying cephadm binary to smithi115 2022-04-22T17:26:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[12737]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.197294+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: cluster 2022-04-22T17:26:19.247045+0000 mon.a (mon.0) 805 : cluster [INF] Manager daemon x is now available 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.282047+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.286926+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.288884+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.291114+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.293730+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.294980+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.310739+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.313357+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.321080+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.323769+0000 mon.b (mon.2) 48 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.349116+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.445080+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: cluster 2022-04-22T17:26:19.445205+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.452466+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: audit 2022-04-22T17:26:19.455187+0000 mon.b (mon.2) 49 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:26:20.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: cephadm 2022-04-22T17:26:19.774926+0000 mgr.x (mgr.24749) 1 : cephadm [INF] Deploying cephadm binary to smithi115 2022-04-22T17:26:20.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:20 smithi115 bash[18070]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:26:20.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[30140]: [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:26:20.767 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:26:20 smithi121 bash[30140]: [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:26:21.471 INFO:tasks.rados.rados.0.smithi115.stdout:298: done (2 left) 2022-04-22T17:26:21.471 INFO:tasks.rados.rados.0.smithi115.stdout:299: setattr oid 15 current snap is 21 2022-04-22T17:26:21.471 INFO:tasks.rados.rados.0.smithi115.stdout:296: finishing write tid 3 to smithi11541303-18 2022-04-22T17:26:21.471 INFO:tasks.rados.rados.0.smithi115.stdout:296: finishing write tid 4 to smithi11541303-18 2022-04-22T17:26:21.472 INFO:tasks.rados.rados.0.smithi115.stdout:296: finishing write tid 5 to smithi11541303-18 2022-04-22T17:26:21.472 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 137 (ObjNum 108 snap 21 seq_num 108) dirty exists 2022-04-22T17:26:21.472 INFO:tasks.rados.rados.0.smithi115.stdout:296: left oid 18 (ObjNum 108 snap 21 seq_num 108) 2022-04-22T17:26:21.472 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 72 (ObjNum 96 snap 19 seq_num 96) dirty exists 2022-04-22T17:26:21.473 INFO:tasks.rados.rados.0.smithi115.stdout:296: done (2 left) 2022-04-22T17:26:21.473 INFO:tasks.rados.rados.0.smithi115.stdout:297: done (1 left) 2022-04-22T17:26:21.473 INFO:tasks.rados.rados.0.smithi115.stdout:300: snap_create 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cluster 2022-04-22T17:26:20.196203+0000 mon.a (mon.0) 813 : cluster [DBG] mgrmap e17: x(active, since 1.76077s), standbys: y 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cluster 2022-04-22T17:26:20.227920+0000 mgr.x (mgr.24749) 3 : cluster [DBG] pgmap v4: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: audit 2022-04-22T17:26:20.442790+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: audit 2022-04-22T17:26:20.451516+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:21.480 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[12679]: cluster 2022-04-22T17:26:20.451624+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2022-04-22T17:26:21.483 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 23 (ObjNum 4 snap 0 seq_num 4) dirty exists 2022-04-22T17:26:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cluster 2022-04-22T17:26:20.196203+0000 mon.a (mon.0) 813 : cluster [DBG] mgrmap e17: x(active, since 1.76077s), standbys: y 2022-04-22T17:26:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cluster 2022-04-22T17:26:20.227920+0000 mgr.x (mgr.24749) 3 : cluster [DBG] pgmap v4: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:26:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:26:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: audit 2022-04-22T17:26:20.442790+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: audit 2022-04-22T17:26:20.451516+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[12737]: cluster 2022-04-22T17:26:20.451624+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cluster 2022-04-22T17:26:20.196203+0000 mon.a (mon.0) 813 : cluster [DBG] mgrmap e17: x(active, since 1.76077s), standbys: y 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cluster 2022-04-22T17:26:20.227920+0000 mgr.x (mgr.24749) 3 : cluster [DBG] pgmap v4: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:26:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:26:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: audit 2022-04-22T17:26:20.442790+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: audit 2022-04-22T17:26:20.451516+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-04-22T17:26:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:21 smithi115 bash[18070]: cluster 2022-04-22T17:26:20.451624+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2022-04-22T17:26:21.766 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:26:21 smithi121 bash[18604]: debug 2022-04-22T17:26:21.472+0000 7faa0f096700 -1 osd.5 pg_epoch: 108 pg[5.7s0( v 108'139 (0'0,108'139] local-lis/les=63/64 n=8 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [5,1,0]p5(0) r=0 lpr=63 luod=107'137 crt=107'137 lcod 107'136 mlcod 107'136 active+clean+snaptrim trimq=[22~1] ps=[10~1,14~1,17~1,1b~1,1f~3,27~1]] removing snap head 2022-04-22T17:26:22.472 INFO:tasks.rados.rados.0.smithi115.stdout:299: done (1 left) 2022-04-22T17:26:22.473 INFO:tasks.rados.rados.0.smithi115.stdout:300: done (0 left) 2022-04-22T17:26:22.473 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 22 2022-04-22T17:26:22.473 INFO:tasks.rados.rados.0.smithi115.stdout:301: seq_num 109 ranges {0=630784,630784=491520,1122304=483328,1605632=229376} 2022-04-22T17:26:22.479 INFO:tasks.rados.rados.0.smithi115.stdout:301: writing smithi11541303-33 from 0 to 630784 tid 1 2022-04-22T17:26:22.482 INFO:tasks.rados.rados.0.smithi115.stdout:301: writing smithi11541303-33 from 630784 to 1122304 tid 2 2022-04-22T17:26:22.485 INFO:tasks.rados.rados.0.smithi115.stdout:301: writing smithi11541303-33 from 1122304 to 1605632 tid 3 2022-04-22T17:26:22.487 INFO:tasks.rados.rados.0.smithi115.stdout:301: writing smithi11541303-33 from 1605632 to 1835008 tid 4 2022-04-22T17:26:22.487 INFO:tasks.rados.rados.0.smithi115.stdout:302: copy_from oid 18 from oid 13 current snap is 22 2022-04-22T17:26:22.487 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 1 to smithi11541303-33 2022-04-22T17:26:22.487 INFO:tasks.rados.rados.0.smithi115.stdout:303: snap_create 2022-04-22T17:26:22.490 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 2 to smithi11541303-33 2022-04-22T17:26:22.493 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 3 to smithi11541303-33 2022-04-22T17:26:22.493 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 4 to smithi11541303-33 2022-04-22T17:26:22.493 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 5 to smithi11541303-33 2022-04-22T17:26:22.495 INFO:tasks.rados.rados.0.smithi115.stdout:301: finishing write tid 6 to smithi11541303-33 2022-04-22T17:26:22.495 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 44 (ObjNum 109 snap 22 seq_num 109) dirty exists 2022-04-22T17:26:22.495 INFO:tasks.rados.rados.0.smithi115.stdout:301: left oid 33 (ObjNum 109 snap 22 seq_num 109) 2022-04-22T17:26:22.518 INFO:tasks.rados.rados.0.smithi115.stdout:302: finishing copy_from to smithi11541303-18 2022-04-22T17:26:22.519 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 68 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:22.519 INFO:tasks.rados.rados.0.smithi115.stdout:302: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: cluster 2022-04-22T17:26:21.196529+0000 mgr.x (mgr.24749) 7 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: cluster 2022-04-22T17:26:21.458082+0000 mon.a (mon.0) 817 : cluster [DBG] mgrmap e18: x(active, since 3s), standbys: y 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: cluster 2022-04-22T17:26:21.464491+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: audit 2022-04-22T17:26:21.569218+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: cluster 2022-04-22T17:26:21.921602+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: cluster 2022-04-22T17:26:21.921804+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: audit 2022-04-22T17:26:21.924148+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:26:22.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: audit 2022-04-22T17:26:21.929512+0000 mon.a (mon.0) 823 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:22.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: audit 2022-04-22T17:26:21.931201+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:26:22.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:22 smithi121 bash[12679]: audit 2022-04-22T17:26:21.931813+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: cluster 2022-04-22T17:26:21.196529+0000 mgr.x (mgr.24749) 7 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: cluster 2022-04-22T17:26:21.458082+0000 mon.a (mon.0) 817 : cluster [DBG] mgrmap e18: x(active, since 3s), standbys: y 2022-04-22T17:26:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: cluster 2022-04-22T17:26:21.464491+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: audit 2022-04-22T17:26:21.569218+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: cluster 2022-04-22T17:26:21.921602+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: cluster 2022-04-22T17:26:21.921804+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: audit 2022-04-22T17:26:21.924148+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: audit 2022-04-22T17:26:21.929512+0000 mon.a (mon.0) 823 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: audit 2022-04-22T17:26:21.931201+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[18070]: audit 2022-04-22T17:26:21.931813+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: cluster 2022-04-22T17:26:21.196529+0000 mgr.x (mgr.24749) 7 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: cluster 2022-04-22T17:26:21.458082+0000 mon.a (mon.0) 817 : cluster [DBG] mgrmap e18: x(active, since 3s), standbys: y 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: cluster 2022-04-22T17:26:21.464491+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: audit 2022-04-22T17:26:21.569218+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: cluster 2022-04-22T17:26:21.921602+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: cluster 2022-04-22T17:26:21.921804+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: audit 2022-04-22T17:26:21.924148+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: audit 2022-04-22T17:26:21.929512+0000 mon.a (mon.0) 823 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: audit 2022-04-22T17:26:21.931201+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:26:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:22 smithi115 bash[12737]: audit 2022-04-22T17:26:21.931813+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.? 172.21.15.115:0/173345360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:26:23.478 INFO:tasks.rados.rados.0.smithi115.stdout:301: done (2 left) 2022-04-22T17:26:23.478 INFO:tasks.rados.rados.0.smithi115.stdout:302: done (1 left) 2022-04-22T17:26:23.478 INFO:tasks.rados.rados.0.smithi115.stdout:303: done (0 left) 2022-04-22T17:26:23.478 INFO:tasks.rados.rados.0.smithi115.stdout:304: snap_create 2022-04-22T17:26:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:23 smithi121 bash[12679]: cluster 2022-04-22T17:26:22.466108+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2022-04-22T17:26:23.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:23 smithi121 bash[12679]: cluster 2022-04-22T17:26:22.579402+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e19: x(active, since 4s), standbys: y 2022-04-22T17:26:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[12737]: cluster 2022-04-22T17:26:22.466108+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2022-04-22T17:26:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[12737]: cluster 2022-04-22T17:26:22.579402+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e19: x(active, since 4s), standbys: y 2022-04-22T17:26:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[18070]: cluster 2022-04-22T17:26:22.466108+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2022-04-22T17:26:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[18070]: cluster 2022-04-22T17:26:22.579402+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e19: x(active, since 4s), standbys: y 2022-04-22T17:26:23.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:26:23.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:23.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:23.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:26:24.479 INFO:tasks.rados.rados.0.smithi115.stdout:304: done (0 left) 2022-04-22T17:26:24.480 INFO:tasks.rados.rados.0.smithi115.stdout:305: read oid 15 snap -1 2022-04-22T17:26:24.480 INFO:tasks.rados.rados.0.smithi115.stdout:305: expect (ObjNum 4 snap 0 seq_num 4) 2022-04-22T17:26:24.480 INFO:tasks.rados.rados.0.smithi115.stdout:306: snap_create 2022-04-22T17:26:24.489 INFO:tasks.rados.rados.0.smithi115.stdout:305: expect (ObjNum 4 snap 0 seq_num 4) 2022-04-22T17:26:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:24 smithi121 bash[12679]: cluster 2022-04-22T17:26:23.197313+0000 mgr.x (mgr.24749) 8 : cluster [DBG] pgmap v9: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:24.806 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:24 smithi121 bash[12679]: cluster 2022-04-22T17:26:23.470913+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2022-04-22T17:26:24.807 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:24 smithi121 bash[12679]: cluster 2022-04-22T17:26:24.471221+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2022-04-22T17:26:24.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[12737]: cluster 2022-04-22T17:26:23.197313+0000 mgr.x (mgr.24749) 8 : cluster [DBG] pgmap v9: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:24.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[12737]: cluster 2022-04-22T17:26:23.470913+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2022-04-22T17:26:24.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[12737]: cluster 2022-04-22T17:26:24.471221+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2022-04-22T17:26:24.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[18070]: cluster 2022-04-22T17:26:23.197313+0000 mgr.x (mgr.24749) 8 : cluster [DBG] pgmap v9: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:24.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[18070]: cluster 2022-04-22T17:26:23.470913+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2022-04-22T17:26:24.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:24 smithi115 bash[18070]: cluster 2022-04-22T17:26:24.471221+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2022-04-22T17:26:25.485 INFO:tasks.rados.rados.0.smithi115.stdout:305: done (1 left) 2022-04-22T17:26:25.485 INFO:tasks.rados.rados.0.smithi115.stdout:306: done (0 left) 2022-04-22T17:26:25.485 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 25 2022-04-22T17:26:25.485 INFO:tasks.rados.rados.0.smithi115.stdout:307: seq_num 110 ranges {5029888=647168,5677056=778240,6455296=425984} 2022-04-22T17:26:25.518 INFO:tasks.rados.rados.0.smithi115.stdout:307: writing smithi11541303-23 from 5029888 to 5677056 tid 1 2022-04-22T17:26:25.523 INFO:tasks.rados.rados.0.smithi115.stdout:307: writing smithi11541303-23 from 5677056 to 6455296 tid 2 2022-04-22T17:26:25.525 INFO:tasks.rados.rados.0.smithi115.stdout:307: writing smithi11541303-23 from 6455296 to 6881280 tid 3 2022-04-22T17:26:25.526 INFO:tasks.rados.rados.0.smithi115.stdout:308: rollback oid 9 current snap is 25 2022-04-22T17:26:25.526 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 24 2022-04-22T17:26:25.526 INFO:tasks.rados.rados.0.smithi115.stdout:309: snap_remove snap 22 2022-04-22T17:26:26.177 INFO:tasks.rados.rados.0.smithi115.stdout:309: done (2 left) 2022-04-22T17:26:26.177 INFO:tasks.rados.rados.0.smithi115.stdout:310: read oid 18 snap 23 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:310: expect (ObjNum 89 snap 15 seq_num 89) 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:311: snap_create 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:307: finishing write tid 1 to smithi11541303-23 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:307: finishing write tid 2 to smithi11541303-23 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:307: finishing write tid 3 to smithi11541303-23 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:308: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:307: finishing write tid 4 to smithi11541303-23 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:308: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:26:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:308: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:26:26.179 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 149 (ObjNum 78 snap 14 seq_num 78) dirty exists 2022-04-22T17:26:26.179 INFO:tasks.rados.rados.0.smithi115.stdout:307: finishing write tid 5 to smithi11541303-23 2022-04-22T17:26:26.179 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 88 (ObjNum 110 snap 25 seq_num 110) dirty exists 2022-04-22T17:26:26.179 INFO:tasks.rados.rados.0.smithi115.stdout:307: left oid 23 (ObjNum 110 snap 25 seq_num 110) 2022-04-22T17:26:26.205 INFO:tasks.rados.rados.0.smithi115.stdout:310: expect (ObjNum 89 snap 15 seq_num 89) 2022-04-22T17:26:26.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: audit 2022-04-22T17:26:25.115244+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: cluster 2022-04-22T17:26:25.197917+0000 mgr.x (mgr.24749) 9 : cluster [DBG] pgmap v12: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:26.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: audit 2022-04-22T17:26:25.262662+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: cluster 2022-04-22T17:26:25.476936+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2022-04-22T17:26:26.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: audit 2022-04-22T17:26:25.718480+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:26 smithi121 bash[12679]: audit 2022-04-22T17:26:25.724456+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: audit 2022-04-22T17:26:25.115244+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: cluster 2022-04-22T17:26:25.197917+0000 mgr.x (mgr.24749) 9 : cluster [DBG] pgmap v12: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: audit 2022-04-22T17:26:25.262662+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: cluster 2022-04-22T17:26:25.476936+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: audit 2022-04-22T17:26:25.718480+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[12737]: audit 2022-04-22T17:26:25.724456+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: audit 2022-04-22T17:26:25.115244+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: cluster 2022-04-22T17:26:25.197917+0000 mgr.x (mgr.24749) 9 : cluster [DBG] pgmap v12: 121 pgs: 121 active+clean; 62 MiB data, 187 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: audit 2022-04-22T17:26:25.262662+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: cluster 2022-04-22T17:26:25.476936+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2022-04-22T17:26:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: audit 2022-04-22T17:26:25.718480+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:26 smithi115 bash[18070]: audit 2022-04-22T17:26:25.724456+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:27.187 INFO:tasks.rados.rados.0.smithi115.stdout:307: done (3 left) 2022-04-22T17:26:27.187 INFO:tasks.rados.rados.0.smithi115.stdout:308: done (2 left) 2022-04-22T17:26:27.187 INFO:tasks.rados.rados.0.smithi115.stdout:310: done (1 left) 2022-04-22T17:26:27.188 INFO:tasks.rados.rados.0.smithi115.stdout:311: done (0 left) 2022-04-22T17:26:27.188 INFO:tasks.rados.rados.0.smithi115.stdout:312: read oid 23 snap -1 2022-04-22T17:26:27.188 INFO:tasks.rados.rados.0.smithi115.stdout:312: expect (ObjNum 110 snap 25 seq_num 110) 2022-04-22T17:26:27.188 INFO:tasks.rados.rados.0.smithi115.stdout:313: snap_remove snap 20 2022-04-22T17:26:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:27 smithi121 bash[12679]: cluster 2022-04-22T17:26:26.167604+0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2022-04-22T17:26:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:27 smithi121 bash[12679]: audit 2022-04-22T17:26:26.306236+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:27 smithi115 bash[18070]: cluster 2022-04-22T17:26:26.167604+0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2022-04-22T17:26:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:27 smithi115 bash[18070]: audit 2022-04-22T17:26:26.306236+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:27 smithi115 bash[12737]: cluster 2022-04-22T17:26:26.167604+0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2022-04-22T17:26:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:27 smithi115 bash[12737]: audit 2022-04-22T17:26:26.306236+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:27.563 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 2s ago 3m 12.2M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3m) 2s ago 3m 22.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4m) 2s ago 4m 12.5M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4m) 2s ago 4m 14.8M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (19s) 2s ago 8m 434M - 17.0.0-11772-g72067459 6c95b0749680 988b90e88450 2022-04-22T17:26:27.919 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (10m) 2s ago 10m 355M - 16.2.5 46cf6318c64e d7208c0f97cf 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (10m) 2s ago 10m 49.9M 2048M 16.2.5 46cf6318c64e c6b4ba941f5c 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (8m) 2s ago 8m 44.4M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 2s ago 9m 42.7M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 2s ago 3m 8636k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3m) 2s ago 3m 8599k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (8m) 2s ago 8m 72.6M 4062M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 2s ago 7m 63.3M 4062M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 2s ago 7m 69.5M 4062M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 2s ago 6m 50.6M 4062M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:26:27.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6m) 2s ago 6m 61.9M 4318M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:26:27.921 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5m) 2s ago 5m 63.6M 4318M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:26:27.921 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5m) 2s ago 5m 64.0M 4318M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:26:27.921 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4m) 2s ago 4m 58.6M 4318M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:26:27.921 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 2s ago 3m 32.5M - 2.18.1 de242295e225 6fd3ae0aa84b 2022-04-22T17:26:28.323 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:26:28.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 14, 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:26:28.325 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:26:29.898 INFO:tasks.rados.rados.0.smithi115.stdout:313: done (1 left) 2022-04-22T17:26:29.907 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 26 2022-04-22T17:26:29.907 INFO:tasks.rados.rados.0.smithi115.stdout:314: seq_num 111 ranges {2170880=663552,2834432=663552,3497984=450560,3948544=90112} 2022-04-22T17:26:29.914 INFO:tasks.rados.rados.0.smithi115.stdout:314: writing smithi11541303-45 from 2170880 to 2834432 tid 1 2022-04-22T17:26:29.918 INFO:tasks.rados.rados.0.smithi115.stdout:314: writing smithi11541303-45 from 2834432 to 3497984 tid 2 2022-04-22T17:26:29.920 INFO:tasks.rados.rados.0.smithi115.stdout:314: writing smithi11541303-45 from 3497984 to 3948544 tid 3 2022-04-22T17:26:29.921 INFO:tasks.rados.rados.0.smithi115.stdout:314: writing smithi11541303-45 from 3948544 to 4038656 tid 4 2022-04-22T17:26:29.921 INFO:tasks.rados.rados.0.smithi115.stdout:315: rollback oid 35 current snap is 26 2022-04-22T17:26:29.921 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 21 2022-04-22T17:26:29.921 INFO:tasks.rados.rados.0.smithi115.stdout:312: expect (ObjNum 110 snap 25 seq_num 110) 2022-04-22T17:26:30.249 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:29 smithi115 bash[12737]: cluster 2022-04-22T17:26:27.179927+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2022-04-22T17:26:30.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:29 smithi115 bash[12737]: audit 2022-04-22T17:26:28.074826+0000 mon.a (mon.0) 838 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:30.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:29 smithi115 bash[18070]: cluster 2022-04-22T17:26:27.179927+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2022-04-22T17:26:30.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:29 smithi115 bash[18070]: audit 2022-04-22T17:26:28.074826+0000 mon.a (mon.0) 838 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:29 smithi121 bash[12679]: cluster 2022-04-22T17:26:27.179927+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2022-04-22T17:26:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:29 smithi121 bash[12679]: audit 2022-04-22T17:26:28.074826+0000 mon.a (mon.0) 838 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 1 to smithi11541303-45 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 2 to smithi11541303-45 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 3 to smithi11541303-45 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 4 to smithi11541303-45 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 5 to smithi11541303-45 2022-04-22T17:26:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:314: finishing write tid 6 to smithi11541303-45 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 84 (ObjNum 111 snap 26 seq_num 111) dirty exists 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:314: left oid 45 (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:315: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:312: done (2 left) 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:314: done (1 left) 2022-04-22T17:26:30.291 INFO:tasks.rados.rados.0.smithi115.stdout:316: snap_create 2022-04-22T17:26:30.292 INFO:tasks.rados.rados.0.smithi115.stdout:315: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:26:30.292 INFO:tasks.rados.rados.0.smithi115.stdout:315: finishing rollback tid 2 to smithi11541303-35 2022-04-22T17:26:30.292 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 56 (ObjNum 103 snap 19 seq_num 103) dirty exists 2022-04-22T17:26:30.912 INFO:tasks.rados.rados.0.smithi115.stdout:315: done (1 left) 2022-04-22T17:26:30.912 INFO:tasks.rados.rados.0.smithi115.stdout:316: done (0 left) 2022-04-22T17:26:30.912 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 27 2022-04-22T17:26:30.912 INFO:tasks.rados.rados.0.smithi115.stdout:317: seq_num 112 ranges {4726784=679936,5406720=548864,5955584=655360} 2022-04-22T17:26:30.944 INFO:tasks.rados.rados.0.smithi115.stdout:317: writing smithi11541303-18 from 4726784 to 5406720 tid 1 2022-04-22T17:26:30.947 INFO:tasks.rados.rados.0.smithi115.stdout:317: writing smithi11541303-18 from 5406720 to 5955584 tid 2 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:317: writing smithi11541303-18 from 5955584 to 6610944 tid 3 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:318: read oid 31 snap -1 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:318: expect deleted 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:319: read oid 5 snap -1 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:319: expect (ObjNum 91 snap 15 seq_num 91) 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:320: copy_from oid 21 from oid 32 current snap is 27 2022-04-22T17:26:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:321: snap_create 2022-04-22T17:26:30.957 INFO:tasks.rados.rados.0.smithi115.stdout:317: finishing write tid 1 to smithi11541303-18 2022-04-22T17:26:30.957 INFO:tasks.rados.rados.0.smithi115.stdout:317: finishing write tid 2 to smithi11541303-18 2022-04-22T17:26:30.958 INFO:tasks.rados.rados.0.smithi115.stdout:317: finishing write tid 3 to smithi11541303-18 2022-04-22T17:26:30.959 INFO:tasks.rados.rados.0.smithi115.stdout:317: finishing write tid 4 to smithi11541303-18 2022-04-22T17:26:30.967 INFO:tasks.rados.rados.0.smithi115.stdout:317: finishing write tid 5 to smithi11541303-18 2022-04-22T17:26:30.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 160 (ObjNum 112 snap 27 seq_num 112) dirty exists 2022-04-22T17:26:30.967 INFO:tasks.rados.rados.0.smithi115.stdout:317: left oid 18 (ObjNum 112 snap 27 seq_num 112) 2022-04-22T17:26:30.968 INFO:tasks.rados.rados.0.smithi115.stdout:319: expect (ObjNum 91 snap 15 seq_num 91) 2022-04-22T17:26:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:320: finishing copy_from to smithi11541303-21 2022-04-22T17:26:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 34 (ObjNum 104 snap 20 seq_num 104) dirty exists 2022-04-22T17:26:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:320: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:26:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: cluster 2022-04-22T17:26:27.198372+0000 mgr.x (mgr.24749) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 3.0 MiB/s wr, 75 op/s 2022-04-22T17:26:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:27.544635+0000 mgr.x (mgr.24749) 11 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:27.886882+0000 mgr.x (mgr.24749) 12 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.078954+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.079401+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.079765+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.316412+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.115:0/1705408087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.987758+0000 mon.a (mon.0) 839 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.992916+0000 mon.c (mon.1) 35 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.998800+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.999085+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:28.999738+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: cluster 2022-04-22T17:26:29.198874+0000 mgr.x (mgr.24749) 13 : cluster [DBG] pgmap v17: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 59 op/s 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:29.886258+0000 mon.c (mon.1) 36 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: cluster 2022-04-22T17:26:29.891448+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:29.905447+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:29.905937+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:29.906334+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:29.906566+0000 mon.b (mon.2) 59 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:30 smithi121 bash[12679]: audit 2022-04-22T17:26:30.790443+0000 mon.b (mon.2) 60 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: cluster 2022-04-22T17:26:27.198372+0000 mgr.x (mgr.24749) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 3.0 MiB/s wr, 75 op/s 2022-04-22T17:26:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:27.544635+0000 mgr.x (mgr.24749) 11 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:27.886882+0000 mgr.x (mgr.24749) 12 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.078954+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.079401+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.079765+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.316412+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.115:0/1705408087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.987758+0000 mon.a (mon.0) 839 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.992916+0000 mon.c (mon.1) 35 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.998800+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.999085+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:28.999738+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: cluster 2022-04-22T17:26:29.198874+0000 mgr.x (mgr.24749) 13 : cluster [DBG] pgmap v17: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 59 op/s 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:29.886258+0000 mon.c (mon.1) 36 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: cluster 2022-04-22T17:26:29.891448+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:29.905447+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:29.905937+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:29.906334+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:29.906566+0000 mon.b (mon.2) 59 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[18070]: audit 2022-04-22T17:26:30.790443+0000 mon.b (mon.2) 60 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: cluster 2022-04-22T17:26:27.198372+0000 mgr.x (mgr.24749) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 3.0 MiB/s wr, 75 op/s 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:27.544635+0000 mgr.x (mgr.24749) 11 : audit [DBG] from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:27.886882+0000 mgr.x (mgr.24749) 12 : audit [DBG] from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.078954+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.079401+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.079765+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.316412+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.115:0/1705408087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.987758+0000 mon.a (mon.0) 839 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.992916+0000 mon.c (mon.1) 35 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.998800+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.999085+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:28.999738+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: cluster 2022-04-22T17:26:29.198874+0000 mgr.x (mgr.24749) 13 : cluster [DBG] pgmap v17: 121 pgs: 121 active+clean; 68 MiB data, 201 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 2.4 MiB/s wr, 59 op/s 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:29.886258+0000 mon.c (mon.1) 36 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: cluster 2022-04-22T17:26:29.891448+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:29.905447+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:29.905937+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:29.906334+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:29.906566+0000 mon.b (mon.2) 59 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-04-22T17:26:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:30 smithi115 bash[12737]: audit 2022-04-22T17:26:30.790443+0000 mon.b (mon.2) 60 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-04-22T17:26:31.918 INFO:tasks.rados.rados.0.smithi115.stdout:317: done (4 left) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:318: done (3 left) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:319: done (2 left) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:320: done (1 left) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:321: done (0 left) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:322: read oid 41 snap -1 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:322: expect (ObjNum 7 snap 0 seq_num 7) 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 28 2022-04-22T17:26:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:323: seq_num 113 ranges {3219456=696320,3915776=434176,4349952=466944,4816896=303104} 2022-04-22T17:26:31.940 INFO:tasks.rados.rados.0.smithi115.stdout:323: writing smithi11541303-16 from 3219456 to 3915776 tid 1 2022-04-22T17:26:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:323: writing smithi11541303-16 from 3915776 to 4349952 tid 2 2022-04-22T17:26:31.946 INFO:tasks.rados.rados.0.smithi115.stdout:323: writing smithi11541303-16 from 4349952 to 4816896 tid 3 2022-04-22T17:26:31.948 INFO:tasks.rados.rados.0.smithi115.stdout:323: writing smithi11541303-16 from 4816896 to 5120000 tid 4 2022-04-22T17:26:31.948 INFO:tasks.rados.rados.0.smithi115.stdout:324: setattr oid 13 current snap is 28 2022-04-22T17:26:31.948 INFO:tasks.rados.rados.0.smithi115.stdout:322: expect (ObjNum 7 snap 0 seq_num 7) 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 1 to smithi11541303-16 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 2 to smithi11541303-16 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 3 to smithi11541303-16 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 4 to smithi11541303-16 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 5 to smithi11541303-16 2022-04-22T17:26:31.957 INFO:tasks.rados.rados.0.smithi115.stdout:322: done (2 left) 2022-04-22T17:26:31.958 INFO:tasks.rados.rados.0.smithi115.stdout:325: delete oid 37 current snap is 28 2022-04-22T17:26:31.958 INFO:tasks.rados.rados.0.smithi115.stdout:323: finishing write tid 6 to smithi11541303-16 2022-04-22T17:26:31.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 58 (ObjNum 113 snap 28 seq_num 113) dirty exists 2022-04-22T17:26:31.958 INFO:tasks.rados.rados.0.smithi115.stdout:323: left oid 16 (ObjNum 113 snap 28 seq_num 113) 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:323: done (2 left) 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:325: done (1 left) 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:326: read oid 21 snap -1 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:326: expect (ObjNum 104 snap 20 seq_num 104) 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:327: read oid 28 snap 21 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:327: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:31.960 INFO:tasks.rados.rados.0.smithi115.stdout:328: delete oid 8 current snap is 28 2022-04-22T17:26:31.961 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 111 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:31.964 INFO:tasks.rados.rados.0.smithi115.stdout:324: done (3 left) 2022-04-22T17:26:31.964 INFO:tasks.rados.rados.0.smithi115.stdout:328: done (2 left) 2022-04-22T17:26:31.964 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 28 2022-04-22T17:26:31.964 INFO:tasks.rados.rados.0.smithi115.stdout:329: seq_num 116 ranges {73728=745472,819200=532480,1351680=638976,1990656=32768} 2022-04-22T17:26:31.970 INFO:tasks.rados.rados.0.smithi115.stdout:329: writing smithi11541303-15 from 73728 to 819200 tid 1 2022-04-22T17:26:31.974 INFO:tasks.rados.rados.0.smithi115.stdout:329: writing smithi11541303-15 from 819200 to 1351680 tid 2 2022-04-22T17:26:31.979 INFO:tasks.rados.rados.0.smithi115.stdout:329: writing smithi11541303-15 from 1351680 to 1990656 tid 3 2022-04-22T17:26:31.979 INFO:tasks.rados.rados.0.smithi115.stdout:329: writing smithi11541303-15 from 1990656 to 2023424 tid 4 2022-04-22T17:26:31.979 INFO:tasks.rados.rados.0.smithi115.stdout:330: delete oid 26 current snap is 28 2022-04-22T17:26:31.979 INFO:tasks.rados.rados.0.smithi115.stdout:326: expect (ObjNum 104 snap 20 seq_num 104) 2022-04-22T17:26:32.049 INFO:tasks.rados.rados.0.smithi115.stdout:327: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 1 to smithi11541303-15 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 2 to smithi11541303-15 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 3 to smithi11541303-15 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 4 to smithi11541303-15 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 5 to smithi11541303-15 2022-04-22T17:26:32.062 INFO:tasks.rados.rados.0.smithi115.stdout:329: finishing write tid 6 to smithi11541303-15 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 31 (ObjNum 116 snap 28 seq_num 116) dirty exists 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:329: left oid 15 (ObjNum 116 snap 28 seq_num 116) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:326: done (3 left) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:327: done (2 left) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:329: done (1 left) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:330: done (0 left) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:331: copy_from oid 32 from oid 23 current snap is 28 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:332: read oid 7 snap -1 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:332: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:333: read oid 19 snap -1 2022-04-22T17:26:32.063 INFO:tasks.rados.rados.0.smithi115.stdout:333: expect (ObjNum 99 snap 19 seq_num 99) 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:334: copy_from oid 5 from oid 44 current snap is 28 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:335: rmattr oid 36 current snap is 28 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:335: done (4 left) 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:336: read oid 45 snap -1 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:336: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:26:32.064 INFO:tasks.rados.rados.0.smithi115.stdout:337: setattr oid 29 current snap is 28 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:338: rollback oid 31 current snap is 28 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 23 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:339: snap_create 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:334: finishing copy_from to smithi11541303-5 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:334: got expected ENOENT (src dne) 2022-04-22T17:26:32.065 INFO:tasks.rados.rados.0.smithi115.stdout:334: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:26:32.066 INFO:tasks.rados.rados.0.smithi115.stdout:338: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:26:32.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:26:32.066 INFO:tasks.rados.rados.0.smithi115.stdout:332: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:32.074 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 55 (ObjNum 46 snap 0 seq_num 46) dirty exists 2022-04-22T17:26:32.076 INFO:tasks.rados.rados.0.smithi115.stdout:333: expect (ObjNum 99 snap 19 seq_num 99) 2022-04-22T17:26:32.196 INFO:tasks.rados.rados.0.smithi115.stdout:336: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:26:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:31 smithi121 bash[12679]: cluster 2022-04-22T17:26:30.905222+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2022-04-22T17:26:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:31 smithi115 bash[18070]: cluster 2022-04-22T17:26:30.905222+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2022-04-22T17:26:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:31 smithi115 bash[12737]: cluster 2022-04-22T17:26:30.905222+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2022-04-22T17:26:32.353 INFO:tasks.rados.rados.0.smithi115.stdout:331: finishing copy_from to smithi11541303-32 2022-04-22T17:26:32.354 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 88 (ObjNum 110 snap 25 seq_num 110) dirty exists 2022-04-22T17:26:32.354 INFO:tasks.rados.rados.0.smithi115.stdout:331: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:26:32.930 INFO:tasks.rados.rados.0.smithi115.stdout:331: done (7 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:332: done (6 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:333: done (5 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:334: done (4 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:336: done (3 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:337: done (2 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:338: done (1 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:339: done (0 left) 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:340: rollback oid 50 current snap is 29 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 27 2022-04-22T17:26:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:341: rmattr oid 38 current snap is 29 2022-04-22T17:26:32.932 INFO:tasks.rados.rados.0.smithi115.stdout:341: done (1 left) 2022-04-22T17:26:32.932 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 29 2022-04-22T17:26:32.932 INFO:tasks.rados.rados.0.smithi115.stdout:342: seq_num 118 ranges {2277376=786432,3063808=704512,3768320=499712} 2022-04-22T17:26:32.953 INFO:tasks.rados.rados.0.smithi115.stdout:342: writing smithi11541303-21 from 2277376 to 3063808 tid 1 2022-04-22T17:26:32.957 INFO:tasks.rados.rados.0.smithi115.stdout:342: writing smithi11541303-21 from 3063808 to 3768320 tid 2 2022-04-22T17:26:32.961 INFO:tasks.rados.rados.0.smithi115.stdout:342: writing smithi11541303-21 from 3768320 to 4268032 tid 3 2022-04-22T17:26:32.961 INFO:tasks.rados.rados.0.smithi115.stdout:343: copy_from oid 37 from oid 14 current snap is 29 2022-04-22T17:26:32.961 INFO:tasks.rados.rados.0.smithi115.stdout:340: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:26:32.961 INFO:tasks.rados.rados.0.smithi115.stdout:344: rmattr oid 34 current snap is 29 2022-04-22T17:26:32.961 INFO:tasks.rados.rados.0.smithi115.stdout:344: done (3 left) 2022-04-22T17:26:32.962 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 29 2022-04-22T17:26:32.962 INFO:tasks.rados.rados.0.smithi115.stdout:340: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:26:32.962 INFO:tasks.rados.rados.0.smithi115.stdout:340: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:26:32.962 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 62 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:26:32.962 INFO:tasks.rados.rados.0.smithi115.stdout:345: seq_num 119 ranges {188416=802816,991232=589824,1581056=409600,1990656=204800} 2022-04-22T17:26:32.967 INFO:tasks.rados.rados.0.smithi115.stdout:345: writing smithi11541303-12 from 188416 to 991232 tid 1 2022-04-22T17:26:32.971 INFO:tasks.rados.rados.0.smithi115.stdout:345: writing smithi11541303-12 from 991232 to 1581056 tid 2 2022-04-22T17:26:32.973 INFO:tasks.rados.rados.0.smithi115.stdout:345: writing smithi11541303-12 from 1581056 to 1990656 tid 3 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:345: writing smithi11541303-12 from 1990656 to 2195456 tid 4 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:340: done (3 left) 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 29 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:342: finishing write tid 1 to smithi11541303-21 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:342: finishing write tid 2 to smithi11541303-21 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:342: finishing write tid 3 to smithi11541303-21 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:342: finishing write tid 4 to smithi11541303-21 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:343: finishing copy_from to smithi11541303-37 2022-04-22T17:26:32.976 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 29 (ObjNum 13 snap 0 seq_num 13) dirty exists 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:342: finishing write tid 5 to smithi11541303-21 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 45 (ObjNum 118 snap 29 seq_num 118) dirty exists 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:342: left oid 21 (ObjNum 118 snap 29 seq_num 118) 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:343: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 1 to smithi11541303-12 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 2 to smithi11541303-12 2022-04-22T17:26:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:346: seq_num 120 ranges {1802240=417792,2220032=475136,2695168=622592,3317760=507904} 2022-04-22T17:26:32.988 INFO:tasks.rados.rados.0.smithi115.stdout:346: writing smithi11541303-22 from 1802240 to 2220032 tid 1 2022-04-22T17:26:32.991 INFO:tasks.rados.rados.0.smithi115.stdout:346: writing smithi11541303-22 from 2220032 to 2695168 tid 2 2022-04-22T17:26:32.994 INFO:tasks.rados.rados.0.smithi115.stdout:346: writing smithi11541303-22 from 2695168 to 3317760 tid 3 2022-04-22T17:26:32.998 INFO:tasks.rados.rados.0.smithi115.stdout:346: writing smithi11541303-22 from 3317760 to 3825664 tid 4 2022-04-22T17:26:32.998 INFO:tasks.rados.rados.0.smithi115.stdout:342: done (3 left) 2022-04-22T17:26:32.998 INFO:tasks.rados.rados.0.smithi115.stdout:343: done (2 left) 2022-04-22T17:26:32.998 INFO:tasks.rados.rados.0.smithi115.stdout:347: copy_from oid 27 from oid 35 current snap is 29 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 3 to smithi11541303-12 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 4 to smithi11541303-12 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 5 to smithi11541303-12 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: finishing write tid 6 to smithi11541303-12 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 53 (ObjNum 119 snap 29 seq_num 119) dirty exists 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: left oid 12 (ObjNum 119 snap 29 seq_num 119) 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 1 to smithi11541303-22 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 2 to smithi11541303-22 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:345: done (2 left) 2022-04-22T17:26:32.999 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 29 2022-04-22T17:26:33.000 INFO:tasks.rados.rados.0.smithi115.stdout:348: seq_num 121 ranges {1548288=434176,1982464=753664,2736128=425984,3162112=425984} 2022-04-22T17:26:33.009 INFO:tasks.rados.rados.0.smithi115.stdout:348: writing smithi11541303-11 from 1548288 to 1982464 tid 1 2022-04-22T17:26:33.013 INFO:tasks.rados.rados.0.smithi115.stdout:348: writing smithi11541303-11 from 1982464 to 2736128 tid 2 2022-04-22T17:26:33.016 INFO:tasks.rados.rados.0.smithi115.stdout:348: writing smithi11541303-11 from 2736128 to 3162112 tid 3 2022-04-22T17:26:33.019 INFO:tasks.rados.rados.0.smithi115.stdout:348: writing smithi11541303-11 from 3162112 to 3588096 tid 4 2022-04-22T17:26:33.019 INFO:tasks.rados.rados.0.smithi115.stdout:349: snap_remove snap 24 2022-04-22T17:26:33.019 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 3 to smithi11541303-22 2022-04-22T17:26:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:32 smithi121 bash[12679]: cluster 2022-04-22T17:26:31.199531+0000 mgr.x (mgr.24749) 14 : cluster [DBG] pgmap v20: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 825 KiB/s wr, 10 op/s 2022-04-22T17:26:33.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:32 smithi121 bash[12679]: cluster 2022-04-22T17:26:31.910879+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2022-04-22T17:26:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:32 smithi115 bash[12737]: cluster 2022-04-22T17:26:31.199531+0000 mgr.x (mgr.24749) 14 : cluster [DBG] pgmap v20: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 825 KiB/s wr, 10 op/s 2022-04-22T17:26:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:32 smithi115 bash[12737]: cluster 2022-04-22T17:26:31.910879+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2022-04-22T17:26:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:32 smithi115 bash[18070]: cluster 2022-04-22T17:26:31.199531+0000 mgr.x (mgr.24749) 14 : cluster [DBG] pgmap v20: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 825 KiB/s wr, 10 op/s 2022-04-22T17:26:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:32 smithi115 bash[18070]: cluster 2022-04-22T17:26:31.910879+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2022-04-22T17:26:33.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:33.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:33.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:33.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:26:33.947 INFO:tasks.rados.rados.0.smithi115.stdout:349: done (3 left) 2022-04-22T17:26:34.034 INFO:tasks.rados.rados.0.smithi115.stdout:350: copy_from oid 16 from oid 44 current snap is 29 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:351: copy_from oid 2 from oid 46 current snap is 29 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 4 to smithi11541303-22 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 5 to smithi11541303-22 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:346: finishing write tid 6 to smithi11541303-22 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 51 (ObjNum 120 snap 29 seq_num 120) dirty exists 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:346: left oid 22 (ObjNum 120 snap 29 seq_num 120) 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 1 to smithi11541303-11 2022-04-22T17:26:34.035 INFO:tasks.rados.rados.0.smithi115.stdout:347: finishing copy_from to smithi11541303-27 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 56 (ObjNum 103 snap 19 seq_num 103) dirty exists 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:347: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 2 to smithi11541303-11 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 3 to smithi11541303-11 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 4 to smithi11541303-11 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 5 to smithi11541303-11 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: finishing write tid 6 to smithi11541303-11 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 98 (ObjNum 121 snap 29 seq_num 121) dirty exists 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:348: left oid 11 (ObjNum 121 snap 29 seq_num 121) 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:346: done (4 left) 2022-04-22T17:26:34.036 INFO:tasks.rados.rados.0.smithi115.stdout:347: done (3 left) 2022-04-22T17:26:34.037 INFO:tasks.rados.rados.0.smithi115.stdout:348: done (2 left) 2022-04-22T17:26:34.037 INFO:tasks.rados.rados.0.smithi115.stdout:352: snap_create 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:350: finishing copy_from to smithi11541303-16 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:350: got expected ENOENT (src dne) 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:350: finishing copy_from racing read to smithi11541303-16 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:351: finishing copy_from to smithi11541303-2 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:26:34.038 INFO:tasks.rados.rados.0.smithi115.stdout:351: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:26:34.199 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:33 smithi115 bash[12737]: cluster 2022-04-22T17:26:32.923102+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2022-04-22T17:26:34.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:33 smithi115 bash[18070]: cluster 2022-04-22T17:26:32.923102+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2022-04-22T17:26:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:33 smithi121 bash[12679]: cluster 2022-04-22T17:26:32.923102+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2022-04-22T17:26:34.949 INFO:tasks.rados.rados.0.smithi115.stdout:350: done (2 left) 2022-04-22T17:26:34.949 INFO:tasks.rados.rados.0.smithi115.stdout:351: done (1 left) 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:352: done (0 left) 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:353: read oid 15 snap -1 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:353: expect (ObjNum 116 snap 28 seq_num 116) 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:354: copy_from oid 39 from oid 19 current snap is 30 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 30 2022-04-22T17:26:34.950 INFO:tasks.rados.rados.0.smithi115.stdout:355: seq_num 122 ranges {1458176=450560,1908736=688128,2596864=630784,3227648=286720} 2022-04-22T17:26:34.963 INFO:tasks.rados.rados.0.smithi115.stdout:355: writing smithi11541303-9 from 1458176 to 1908736 tid 1 2022-04-22T17:26:34.967 INFO:tasks.rados.rados.0.smithi115.stdout:355: writing smithi11541303-9 from 1908736 to 2596864 tid 2 2022-04-22T17:26:34.971 INFO:tasks.rados.rados.0.smithi115.stdout:355: writing smithi11541303-9 from 2596864 to 3227648 tid 3 2022-04-22T17:26:34.973 INFO:tasks.rados.rados.0.smithi115.stdout:355: writing smithi11541303-9 from 3227648 to 3514368 tid 4 2022-04-22T17:26:34.973 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 30 2022-04-22T17:26:34.974 INFO:tasks.rados.rados.0.smithi115.stdout:353: expect (ObjNum 116 snap 28 seq_num 116) 2022-04-22T17:26:35.040 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 1 to smithi11541303-9 2022-04-22T17:26:35.143 INFO:tasks.rados.rados.0.smithi115.stdout:354: finishing copy_from to smithi11541303-39 2022-04-22T17:26:35.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 24 (ObjNum 99 snap 19 seq_num 99) dirty exists 2022-04-22T17:26:35.143 INFO:tasks.rados.rados.0.smithi115.stdout:354: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:26:35.143 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 2 to smithi11541303-9 2022-04-22T17:26:35.143 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 3 to smithi11541303-9 2022-04-22T17:26:35.144 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 4 to smithi11541303-9 2022-04-22T17:26:35.144 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 5 to smithi11541303-9 2022-04-22T17:26:35.144 INFO:tasks.rados.rados.0.smithi115.stdout:355: finishing write tid 6 to smithi11541303-9 2022-04-22T17:26:35.145 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 170 (ObjNum 122 snap 30 seq_num 122) dirty exists 2022-04-22T17:26:35.145 INFO:tasks.rados.rados.0.smithi115.stdout:355: left oid 9 (ObjNum 122 snap 30 seq_num 122) 2022-04-22T17:26:35.145 INFO:tasks.rados.rados.0.smithi115.stdout:356: seq_num 123 ranges {4038656=466944,4505600=573440,5079040=794624,5873664=237568} 2022-04-22T17:26:35.147 INFO:tasks.rados.rados.0.smithi115.stdout:356: writing smithi11541303-45 from 4038656 to 4505600 tid 1 2022-04-22T17:26:35.147 INFO:tasks.rados.rados.0.smithi115.stdout:356: writing smithi11541303-45 from 4505600 to 5079040 tid 2 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:356: writing smithi11541303-45 from 5079040 to 5873664 tid 3 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:356: writing smithi11541303-45 from 5873664 to 6111232 tid 4 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:353: done (3 left) 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:354: done (2 left) 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:355: done (1 left) 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:357: snap_create 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 1 to smithi11541303-45 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 2 to smithi11541303-45 2022-04-22T17:26:35.148 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 3 to smithi11541303-45 2022-04-22T17:26:35.149 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 4 to smithi11541303-45 2022-04-22T17:26:35.149 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 5 to smithi11541303-45 2022-04-22T17:26:35.149 INFO:tasks.rados.rados.0.smithi115.stdout:356: finishing write tid 6 to smithi11541303-45 2022-04-22T17:26:35.149 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 96 (ObjNum 123 snap 30 seq_num 123) dirty exists 2022-04-22T17:26:35.149 INFO:tasks.rados.rados.0.smithi115.stdout:356: left oid 45 (ObjNum 123 snap 30 seq_num 123) 2022-04-22T17:26:35.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: cluster 2022-04-22T17:26:33.200258+0000 mgr.x (mgr.24749) 15 : cluster [DBG] pgmap v23: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2022-04-22T17:26:35.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: cluster 2022-04-22T17:26:33.934725+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2022-04-22T17:26:35.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.906732+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.911900+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.914130+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.915251+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.916304+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.917533+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.918519+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.919766+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.922026+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[18070]: audit 2022-04-22T17:26:34.926659+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: cluster 2022-04-22T17:26:33.200258+0000 mgr.x (mgr.24749) 15 : cluster [DBG] pgmap v23: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: cluster 2022-04-22T17:26:33.934725+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.906732+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.911900+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.914130+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.915251+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.916304+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.917533+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.918519+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.164 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.919766+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.164 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.922026+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.164 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:34 smithi115 bash[12737]: audit 2022-04-22T17:26:34.926659+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.255 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: cluster 2022-04-22T17:26:33.200258+0000 mgr.x (mgr.24749) 15 : cluster [DBG] pgmap v23: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2022-04-22T17:26:35.255 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: cluster 2022-04-22T17:26:33.934725+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2022-04-22T17:26:35.255 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.906732+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.911900+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.914130+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.915251+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.916304+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.917533+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.918519+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.919766+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.922026+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:35.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:34 smithi121 bash[12679]: audit 2022-04-22T17:26:34.926659+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:35.970 INFO:tasks.rados.rados.0.smithi115.stdout:356: done (1 left) 2022-04-22T17:26:35.970 INFO:tasks.rados.rados.0.smithi115.stdout:357: done (0 left) 2022-04-22T17:26:35.970 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 31 2022-04-22T17:26:35.970 INFO:tasks.rados.rados.0.smithi115.stdout:358: seq_num 124 ranges {1531904=483328,2015232=458752,2473984=598016,3072000=548864} 2022-04-22T17:26:35.982 INFO:tasks.rados.rados.0.smithi115.stdout:358: writing smithi11541303-5 from 1531904 to 2015232 tid 1 2022-04-22T17:26:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:358: writing smithi11541303-5 from 2015232 to 2473984 tid 2 2022-04-22T17:26:35.989 INFO:tasks.rados.rados.0.smithi115.stdout:358: writing smithi11541303-5 from 2473984 to 3072000 tid 3 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:358: writing smithi11541303-5 from 3072000 to 3620864 tid 4 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:359: rollback oid 31 current snap is 31 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 29 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 31 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 1 to smithi11541303-5 2022-04-22T17:26:35.994 INFO:tasks.rados.rados.0.smithi115.stdout:360: seq_num 125 ranges {0=499712,499712=745472,1245184=409600,1654784=450560} 2022-04-22T17:26:35.997 INFO:tasks.rados.rados.0.smithi115.stdout:360: writing smithi11541303-49 from 0 to 499712 tid 1 2022-04-22T17:26:36.003 INFO:tasks.rados.rados.0.smithi115.stdout:360: writing smithi11541303-49 from 499712 to 1245184 tid 2 2022-04-22T17:26:36.005 INFO:tasks.rados.rados.0.smithi115.stdout:360: writing smithi11541303-49 from 1245184 to 1654784 tid 3 2022-04-22T17:26:36.008 INFO:tasks.rados.rados.0.smithi115.stdout:360: writing smithi11541303-49 from 1654784 to 2105344 tid 4 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:361: rmattr oid 41 current snap is 31 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:361: done (3 left) 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:362: read oid 7 snap -1 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:362: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:363: snap_create 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 2 to smithi11541303-5 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 3 to smithi11541303-5 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 4 to smithi11541303-5 2022-04-22T17:26:36.009 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 5 to smithi11541303-5 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:359: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:358: finishing write tid 6 to smithi11541303-5 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 176 (ObjNum 124 snap 31 seq_num 124) dirty exists 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:358: left oid 5 (ObjNum 124 snap 31 seq_num 124) 2022-04-22T17:26:36.010 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 1 to smithi11541303-49 2022-04-22T17:26:36.011 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 2 to smithi11541303-49 2022-04-22T17:26:36.011 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 3 to smithi11541303-49 2022-04-22T17:26:36.012 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 4 to smithi11541303-49 2022-04-22T17:26:36.012 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 5 to smithi11541303-49 2022-04-22T17:26:36.013 INFO:tasks.rados.rados.0.smithi115.stdout:360: finishing write tid 6 to smithi11541303-49 2022-04-22T17:26:36.014 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 182 (ObjNum 125 snap 31 seq_num 125) dirty exists 2022-04-22T17:26:36.014 INFO:tasks.rados.rados.0.smithi115.stdout:360: left oid 49 (ObjNum 125 snap 31 seq_num 125) 2022-04-22T17:26:36.014 INFO:tasks.rados.rados.0.smithi115.stdout:362: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:36.178 INFO:tasks.rados.rados.0.smithi115.stdout:358: done (4 left) 2022-04-22T17:26:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:359: done (3 left) 2022-04-22T17:26:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:360: done (2 left) 2022-04-22T17:26:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:362: done (1 left) 2022-04-22T17:26:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:363: done (0 left) 2022-04-22T17:26:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:364: delete oid 49 current snap is 32 2022-04-22T17:26:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:364: done (0 left) 2022-04-22T17:26:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:365: rollback oid 18 current snap is 32 2022-04-22T17:26:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 30 2022-04-22T17:26:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:366: snap_create 2022-04-22T17:26:36.184 INFO:tasks.rados.rados.0.smithi115.stdout:365: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:26:36.184 INFO:tasks.rados.rados.0.smithi115.stdout:365: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:26:36.184 INFO:tasks.rados.rados.0.smithi115.stdout:365: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:26:36.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 188 (ObjNum 112 snap 27 seq_num 112) dirty exists 2022-04-22T17:26:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:26:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:26:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:26:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: cluster 2022-04-22T17:26:34.941515+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2022-04-22T17:26:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.270231+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.274145+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.276724+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.277430+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.279517+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.280105+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.282064+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.282878+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.285529+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.293825+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.412140+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.778863+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.787794+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.797035+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.811989+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:35 smithi121 bash[12679]: audit 2022-04-22T17:26:35.828030+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: cluster 2022-04-22T17:26:34.941515+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.270231+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.274145+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.276724+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.277430+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.279517+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.280105+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.282064+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.282878+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.285529+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.293825+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.412140+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.778863+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.787794+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.797035+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.811989+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[12737]: audit 2022-04-22T17:26:35.828030+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: cluster 2022-04-22T17:26:34.941515+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.270231+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.274145+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.276724+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.277430+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.279517+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.280105+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.282064+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.282878+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.285529+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.293825+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.412140+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.778863+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.787794+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.797035+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.811989+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:36.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:35 smithi115 bash[18070]: audit 2022-04-22T17:26:35.828030+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:37.188 INFO:tasks.rados.rados.0.smithi115.stdout:365: done (1 left) 2022-04-22T17:26:37.188 INFO:tasks.rados.rados.0.smithi115.stdout:366: done (0 left) 2022-04-22T17:26:37.188 INFO:tasks.rados.rados.0.smithi115.stdout:367: rollback oid 41 current snap is 33 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 31 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:368: rollback oid 18 current snap is 33 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 26 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:369: rmattr oid 23 current snap is 33 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:369: done (2 left) 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:370: copy_from oid 43 from oid 10 current snap is 33 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 33 2022-04-22T17:26:37.189 INFO:tasks.rados.rados.0.smithi115.stdout:371: seq_num 127 ranges {3088384=532480,3620864=507904,4128768=778240,4907008=319488} 2022-04-22T17:26:37.220 INFO:tasks.rados.rados.0.smithi115.stdout:371: writing smithi11541303-19 from 3088384 to 3620864 tid 1 2022-04-22T17:26:37.222 INFO:tasks.rados.rados.0.smithi115.stdout:371: writing smithi11541303-19 from 3620864 to 4128768 tid 2 2022-04-22T17:26:37.227 INFO:tasks.rados.rados.0.smithi115.stdout:371: writing smithi11541303-19 from 4128768 to 4907008 tid 3 2022-04-22T17:26:37.229 INFO:tasks.rados.rados.0.smithi115.stdout:371: writing smithi11541303-19 from 4907008 to 5226496 tid 4 2022-04-22T17:26:37.229 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 33 2022-04-22T17:26:37.229 INFO:tasks.rados.rados.0.smithi115.stdout:367: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:26:37.229 INFO:tasks.rados.rados.0.smithi115.stdout:372: seq_num 128 ranges {4726784=548864,5275648=794624,6070272=581632,6651904=229376} 2022-04-22T17:26:37.259 INFO:tasks.rados.rados.0.smithi115.stdout:372: writing smithi11541303-13 from 4726784 to 5275648 tid 1 2022-04-22T17:26:37.264 INFO:tasks.rados.rados.0.smithi115.stdout:372: writing smithi11541303-13 from 5275648 to 6070272 tid 2 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cluster 2022-04-22T17:26:35.201015+0000 mgr.x (mgr.24749) 20 : cluster [DBG] pgmap v26: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:26:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:26:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:26:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cluster 2022-04-22T17:26:35.961258+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2022-04-22T17:26:37.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:36 smithi121 bash[12679]: cluster 2022-04-22T17:26:36.169574+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2022-04-22T17:26:37.271 INFO:tasks.rados.rados.0.smithi115.stdout:372: writing smithi11541303-13 from 6070272 to 6651904 tid 3 2022-04-22T17:26:37.271 INFO:tasks.rados.rados.0.smithi115.stdout:372: writing smithi11541303-13 from 6651904 to 6881280 tid 4 2022-04-22T17:26:37.271 INFO:tasks.rados.rados.0.smithi115.stdout:373: setattr oid 26 current snap is 33 2022-04-22T17:26:37.271 INFO:tasks.rados.rados.0.smithi115.stdout:367: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:26:37.271 INFO:tasks.rados.rados.0.smithi115.stdout:368: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:368: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:368: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 192 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:367: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 80 (ObjNum 7 snap 0 seq_num 7) dirty exists 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:370: finishing copy_from to smithi11541303-43 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 42 (ObjNum 74 snap 11 seq_num 74) dirty exists 2022-04-22T17:26:37.272 INFO:tasks.rados.rados.0.smithi115.stdout:370: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 1 to smithi11541303-19 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 2 to smithi11541303-19 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 3 to smithi11541303-19 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 4 to smithi11541303-19 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 5 to smithi11541303-19 2022-04-22T17:26:37.273 INFO:tasks.rados.rados.0.smithi115.stdout:371: finishing write tid 6 to smithi11541303-19 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 61 (ObjNum 127 snap 33 seq_num 127) dirty exists 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:371: left oid 19 (ObjNum 127 snap 33 seq_num 127) 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 1 to smithi11541303-13 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:367: done (5 left) 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:368: done (4 left) 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:370: done (3 left) 2022-04-22T17:26:37.274 INFO:tasks.rados.rados.0.smithi115.stdout:371: done (2 left) 2022-04-22T17:26:37.275 INFO:tasks.rados.rados.0.smithi115.stdout:374: snap_create 2022-04-22T17:26:37.276 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 2 to smithi11541303-13 2022-04-22T17:26:37.277 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 3 to smithi11541303-13 2022-04-22T17:26:37.277 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 4 to smithi11541303-13 2022-04-22T17:26:37.277 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 89 (ObjNum -1116710992 snap 22075 seq_num 3178500720) dirty exists 2022-04-22T17:26:37.277 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 5 to smithi11541303-13 2022-04-22T17:26:37.277 INFO:tasks.rados.rados.0.smithi115.stdout:372: finishing write tid 6 to smithi11541303-13 2022-04-22T17:26:37.278 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 121 (ObjNum 128 snap 33 seq_num 128) dirty exists 2022-04-22T17:26:37.278 INFO:tasks.rados.rados.0.smithi115.stdout:372: left oid 13 (ObjNum 128 snap 33 seq_num 128) 2022-04-22T17:26:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cluster 2022-04-22T17:26:35.201015+0000 mgr.x (mgr.24749) 20 : cluster [DBG] pgmap v26: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:26:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:26:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:26:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:26:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:26:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cluster 2022-04-22T17:26:35.961258+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2022-04-22T17:26:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[12737]: cluster 2022-04-22T17:26:36.169574+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cluster 2022-04-22T17:26:35.201015+0000 mgr.x (mgr.24749) 20 : cluster [DBG] pgmap v26: 121 pgs: 121 active+clean; 71 MiB data, 211 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:26:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:26:37.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:26:37.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:26:37.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cluster 2022-04-22T17:26:35.961258+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2022-04-22T17:26:37.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:36 smithi115 bash[18070]: cluster 2022-04-22T17:26:36.169574+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2022-04-22T17:26:38.195 INFO:tasks.rados.rados.0.smithi115.stdout:372: done (2 left) 2022-04-22T17:26:38.196 INFO:tasks.rados.rados.0.smithi115.stdout:373: done (1 left) 2022-04-22T17:26:38.196 INFO:tasks.rados.rados.0.smithi115.stdout:374: done (0 left) 2022-04-22T17:26:38.196 INFO:tasks.rados.rados.0.smithi115.stdout:375: copy_from oid 34 from oid 46 current snap is 34 2022-04-22T17:26:38.196 INFO:tasks.rados.rados.0.smithi115.stdout:376: snap_remove snap 33 2022-04-22T17:26:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:38 smithi121 bash[12679]: cluster 2022-04-22T17:26:37.180547+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2022-04-22T17:26:38.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:38 smithi121 bash[12679]: cluster 2022-04-22T17:26:37.201643+0000 mgr.x (mgr.24749) 27 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2022-04-22T17:26:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:38 smithi115 bash[12737]: cluster 2022-04-22T17:26:37.180547+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2022-04-22T17:26:38.638 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:38 smithi115 bash[12737]: cluster 2022-04-22T17:26:37.201643+0000 mgr.x (mgr.24749) 27 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2022-04-22T17:26:38.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:38 smithi115 bash[18070]: cluster 2022-04-22T17:26:37.180547+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2022-04-22T17:26:38.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:38 smithi115 bash[18070]: cluster 2022-04-22T17:26:37.201643+0000 mgr.x (mgr.24749) 27 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:375: finishing copy_from to smithi11541303-34 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:375: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:375: done (1 left) 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:376: done (0 left) 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:377: rollback oid 46 current snap is 34 2022-04-22T17:26:39.211 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 26 2022-04-22T17:26:39.212 INFO:tasks.rados.rados.0.smithi115.stdout:378: read oid 28 snap -1 2022-04-22T17:26:39.212 INFO:tasks.rados.rados.0.smithi115.stdout:378: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:39.212 INFO:tasks.rados.rados.0.smithi115.stdout:379: read oid 36 snap -1 2022-04-22T17:26:39.212 INFO:tasks.rados.rados.0.smithi115.stdout:379: expect (ObjNum 105 snap 20 seq_num 105) 2022-04-22T17:26:39.212 INFO:tasks.rados.rados.0.smithi115.stdout:380: delete oid 12 current snap is 34 2022-04-22T17:26:39.218 INFO:tasks.rados.rados.0.smithi115.stdout:377: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:26:39.219 INFO:tasks.rados.rados.0.smithi115.stdout:377: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:26:39.219 INFO:tasks.rados.rados.0.smithi115.stdout:377: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:26:39.219 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 106 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:26:39.222 INFO:tasks.rados.rados.0.smithi115.stdout:378: expect (ObjNum 26 snap 0 seq_num 26) 2022-04-22T17:26:39.234 INFO:tasks.rados.rados.0.smithi115.stdout:379: expect (ObjNum 105 snap 20 seq_num 105) 2022-04-22T17:26:39.290 INFO:tasks.rados.rados.0.smithi115.stdout:377: done (3 left) 2022-04-22T17:26:39.290 INFO:tasks.rados.rados.0.smithi115.stdout:378: done (2 left) 2022-04-22T17:26:39.291 INFO:tasks.rados.rados.0.smithi115.stdout:379: done (1 left) 2022-04-22T17:26:39.291 INFO:tasks.rados.rados.0.smithi115.stdout:380: done (0 left) 2022-04-22T17:26:39.291 INFO:tasks.rados.rados.0.smithi115.stdout:381: rollback oid 5 current snap is 34 2022-04-22T17:26:39.291 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 32 2022-04-22T17:26:39.292 INFO:tasks.rados.rados.0.smithi115.stdout:382: snap_remove snap 23 2022-04-22T17:26:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:39 smithi121 bash[12679]: cluster 2022-04-22T17:26:38.187850+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2022-04-22T17:26:39.561 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:39 smithi115 bash[12737]: cluster 2022-04-22T17:26:38.187850+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2022-04-22T17:26:39.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:39 smithi115 bash[18070]: cluster 2022-04-22T17:26:38.187850+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2022-04-22T17:26:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:382: done (1 left) 2022-04-22T17:26:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 34 2022-04-22T17:26:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:381: finishing rollback tid 0 to smithi11541303-5 2022-04-22T17:26:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:383: seq_num 130 ranges {1835008=581632,2416640=614400,3031040=598016,3629056=393216} 2022-04-22T17:26:40.233 INFO:tasks.rados.rados.0.smithi115.stdout:383: writing smithi11541303-33 from 1835008 to 2416640 tid 1 2022-04-22T17:26:40.236 INFO:tasks.rados.rados.0.smithi115.stdout:383: writing smithi11541303-33 from 2416640 to 3031040 tid 2 2022-04-22T17:26:40.240 INFO:tasks.rados.rados.0.smithi115.stdout:383: writing smithi11541303-33 from 3031040 to 3629056 tid 3 2022-04-22T17:26:40.243 INFO:tasks.rados.rados.0.smithi115.stdout:383: writing smithi11541303-33 from 3629056 to 4022272 tid 4 2022-04-22T17:26:40.243 INFO:tasks.rados.rados.0.smithi115.stdout:384: read oid 7 snap -1 2022-04-22T17:26:40.243 INFO:tasks.rados.rados.0.smithi115.stdout:384: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:40.243 INFO:tasks.rados.rados.0.smithi115.stdout:385: setattr oid 25 current snap is 34 2022-04-22T17:26:40.243 INFO:tasks.rados.rados.0.smithi115.stdout:381: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:26:40.244 INFO:tasks.rados.rados.0.smithi115.stdout:381: finishing rollback tid 2 to smithi11541303-5 2022-04-22T17:26:40.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 196 (ObjNum 124 snap 31 seq_num 124) dirty exists 2022-04-22T17:26:40.244 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 1 to smithi11541303-33 2022-04-22T17:26:40.244 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 2 to smithi11541303-33 2022-04-22T17:26:40.245 INFO:tasks.rados.rados.0.smithi115.stdout:381: done (3 left) 2022-04-22T17:26:40.245 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 34 2022-04-22T17:26:40.245 INFO:tasks.rados.rados.0.smithi115.stdout:386: seq_num 131 ranges {1302528=598016,1900544=499712,2400256=753664,3153920=352256} 2022-04-22T17:26:40.256 INFO:tasks.rados.rados.0.smithi115.stdout:386: writing smithi11541303-24 from 1302528 to 1900544 tid 1 2022-04-22T17:26:40.259 INFO:tasks.rados.rados.0.smithi115.stdout:386: writing smithi11541303-24 from 1900544 to 2400256 tid 2 2022-04-22T17:26:40.263 INFO:tasks.rados.rados.0.smithi115.stdout:386: writing smithi11541303-24 from 2400256 to 3153920 tid 3 2022-04-22T17:26:40.265 INFO:tasks.rados.rados.0.smithi115.stdout:386: writing smithi11541303-24 from 3153920 to 3506176 tid 4 2022-04-22T17:26:40.266 INFO:tasks.rados.rados.0.smithi115.stdout:387: snap_create 2022-04-22T17:26:40.266 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 3 to smithi11541303-33 2022-04-22T17:26:40.266 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 4 to smithi11541303-33 2022-04-22T17:26:40.266 INFO:tasks.rados.rados.0.smithi115.stdout:384: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 5 to smithi11541303-33 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 126 (ObjNum -1116263472 snap 22075 seq_num 3178343136) dirty exists 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:383: finishing write tid 6 to smithi11541303-33 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 94 (ObjNum 130 snap 34 seq_num 130) dirty exists 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:383: left oid 33 (ObjNum 130 snap 34 seq_num 130) 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 1 to smithi11541303-24 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 2 to smithi11541303-24 2022-04-22T17:26:40.271 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 3 to smithi11541303-24 2022-04-22T17:26:40.272 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 4 to smithi11541303-24 2022-04-22T17:26:40.272 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 5 to smithi11541303-24 2022-04-22T17:26:40.273 INFO:tasks.rados.rados.0.smithi115.stdout:386: finishing write tid 6 to smithi11541303-24 2022-04-22T17:26:40.273 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 43 (ObjNum 131 snap 34 seq_num 131) dirty exists 2022-04-22T17:26:40.274 INFO:tasks.rados.rados.0.smithi115.stdout:386: left oid 24 (ObjNum 131 snap 34 seq_num 131) 2022-04-22T17:26:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:40 smithi121 bash[12679]: cluster 2022-04-22T17:26:39.202934+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2022-04-22T17:26:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:40 smithi121 bash[12679]: cluster 2022-04-22T17:26:39.203115+0000 mgr.x (mgr.24749) 28 : cluster [DBG] pgmap v33: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:40 smithi115 bash[18070]: cluster 2022-04-22T17:26:39.202934+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2022-04-22T17:26:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:40 smithi115 bash[18070]: cluster 2022-04-22T17:26:39.203115+0000 mgr.x (mgr.24749) 28 : cluster [DBG] pgmap v33: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:40.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:40 smithi115 bash[12737]: cluster 2022-04-22T17:26:39.202934+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2022-04-22T17:26:40.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:40 smithi115 bash[12737]: cluster 2022-04-22T17:26:39.203115+0000 mgr.x (mgr.24749) 28 : cluster [DBG] pgmap v33: 121 pgs: 121 active+clean; 103 MiB data, 271 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:383: done (4 left) 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:384: done (3 left) 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:385: done (2 left) 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:386: done (1 left) 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:387: done (0 left) 2022-04-22T17:26:41.223 INFO:tasks.rados.rados.0.smithi115.stdout:388: read oid 25 snap 26 2022-04-22T17:26:41.224 INFO:tasks.rados.rados.0.smithi115.stdout:388: expect deleted 2022-04-22T17:26:41.224 INFO:tasks.rados.rados.0.smithi115.stdout:389: snap_remove snap 32 2022-04-22T17:26:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:41 smithi121 bash[12679]: cluster 2022-04-22T17:26:40.206972+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2022-04-22T17:26:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:41 smithi115 bash[12737]: cluster 2022-04-22T17:26:40.206972+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2022-04-22T17:26:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:41 smithi115 bash[18070]: cluster 2022-04-22T17:26:40.206972+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2022-04-22T17:26:42.224 INFO:tasks.rados.rados.0.smithi115.stdout:388: done (1 left) 2022-04-22T17:26:42.225 INFO:tasks.rados.rados.0.smithi115.stdout:389: done (0 left) 2022-04-22T17:26:42.225 INFO:tasks.rados.rados.0.smithi115.stdout:390: delete oid 11 current snap is 35 2022-04-22T17:26:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:390: done (0 left) 2022-04-22T17:26:42.234 INFO:tasks.rados.rados.0.smithi115.stdout:391: snap_remove snap 25 2022-04-22T17:26:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:42 smithi121 bash[12679]: cluster 2022-04-22T17:26:41.204354+0000 mgr.x (mgr.24749) 29 : cluster [DBG] pgmap v35: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:26:42.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:42 smithi121 bash[12679]: cluster 2022-04-22T17:26:41.214071+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2022-04-22T17:26:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:42 smithi115 bash[12737]: cluster 2022-04-22T17:26:41.204354+0000 mgr.x (mgr.24749) 29 : cluster [DBG] pgmap v35: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:26:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:42 smithi115 bash[12737]: cluster 2022-04-22T17:26:41.214071+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2022-04-22T17:26:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:42 smithi115 bash[18070]: cluster 2022-04-22T17:26:41.204354+0000 mgr.x (mgr.24749) 29 : cluster [DBG] pgmap v35: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:26:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:42 smithi115 bash[18070]: cluster 2022-04-22T17:26:41.214071+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2022-04-22T17:26:43.242 INFO:tasks.rados.rados.0.smithi115.stdout:391: done (0 left) 2022-04-22T17:26:43.242 INFO:tasks.rados.rados.0.smithi115.stdout:392: snap_remove snap 34 2022-04-22T17:26:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:43 smithi121 bash[12679]: cluster 2022-04-22T17:26:42.216502+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2022-04-22T17:26:43.520 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:43 smithi115 bash[12737]: cluster 2022-04-22T17:26:42.216502+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2022-04-22T17:26:43.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:43 smithi115 bash[18070]: cluster 2022-04-22T17:26:42.216502+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2022-04-22T17:26:43.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:43.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:43.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:43.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:43.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:43.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:26:44.251 INFO:tasks.rados.rados.0.smithi115.stdout:392: done (0 left) 2022-04-22T17:26:44.251 INFO:tasks.rados.rados.0.smithi115.stdout:393: copy_from oid 43 from oid 3 current snap is 35 2022-04-22T17:26:44.252 INFO:tasks.rados.rados.0.smithi115.stdout:394: read oid 37 snap -1 2022-04-22T17:26:44.252 INFO:tasks.rados.rados.0.smithi115.stdout:394: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:44.252 INFO:tasks.rados.rados.0.smithi115.stdout:395: delete oid 12 current snap is 35 2022-04-22T17:26:44.258 INFO:tasks.rados.rados.0.smithi115.stdout:395: done (2 left) 2022-04-22T17:26:44.258 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 35 2022-04-22T17:26:44.259 INFO:tasks.rados.rados.0.smithi115.stdout:396: seq_num 134 ranges {6881280=647168,7528448=548864,8077312=573440,8650752=483328} 2022-04-22T17:26:44.311 INFO:tasks.rados.rados.0.smithi115.stdout:396: writing smithi11541303-13 from 6881280 to 7528448 tid 1 2022-04-22T17:26:44.314 INFO:tasks.rados.rados.0.smithi115.stdout:396: writing smithi11541303-13 from 7528448 to 8077312 tid 2 2022-04-22T17:26:44.317 INFO:tasks.rados.rados.0.smithi115.stdout:396: writing smithi11541303-13 from 8077312 to 8650752 tid 3 2022-04-22T17:26:44.320 INFO:tasks.rados.rados.0.smithi115.stdout:396: writing smithi11541303-13 from 8650752 to 9134080 tid 4 2022-04-22T17:26:44.320 INFO:tasks.rados.rados.0.smithi115.stdout:397: delete oid 9 current snap is 35 2022-04-22T17:26:44.320 INFO:tasks.rados.rados.0.smithi115.stdout:394: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:393: finishing copy_from to smithi11541303-43 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 79 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:393: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 1 to smithi11541303-13 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 2 to smithi11541303-13 2022-04-22T17:26:44.332 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 3 to smithi11541303-13 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 4 to smithi11541303-13 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 5 to smithi11541303-13 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:396: finishing write tid 6 to smithi11541303-13 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 140 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:396: left oid 13 (ObjNum 134 snap 35 seq_num 134) 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:393: done (3 left) 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:394: done (2 left) 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:396: done (1 left) 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:397: done (0 left) 2022-04-22T17:26:44.333 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 35 2022-04-22T17:26:44.334 INFO:tasks.rados.rados.0.smithi115.stdout:398: seq_num 136 ranges {221184=688128,909312=720896,1630208=540672,2170880=344064} 2022-04-22T17:26:44.340 INFO:tasks.rados.rados.0.smithi115.stdout:398: writing smithi11541303-37 from 221184 to 909312 tid 1 2022-04-22T17:26:44.347 INFO:tasks.rados.rados.0.smithi115.stdout:398: writing smithi11541303-37 from 909312 to 1630208 tid 2 2022-04-22T17:26:44.350 INFO:tasks.rados.rados.0.smithi115.stdout:398: writing smithi11541303-37 from 1630208 to 2170880 tid 3 2022-04-22T17:26:44.353 INFO:tasks.rados.rados.0.smithi115.stdout:398: writing smithi11541303-37 from 2170880 to 2514944 tid 4 2022-04-22T17:26:44.353 INFO:tasks.rados.rados.0.smithi115.stdout:399: copy_from oid 21 from oid 32 current snap is 35 2022-04-22T17:26:44.353 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 1 to smithi11541303-37 2022-04-22T17:26:44.353 INFO:tasks.rados.rados.0.smithi115.stdout:400: setattr oid 43 current snap is 35 2022-04-22T17:26:44.353 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 2 to smithi11541303-37 2022-04-22T17:26:44.355 INFO:tasks.rados.rados.0.smithi115.stdout:401: read oid 34 snap -1 2022-04-22T17:26:44.356 INFO:tasks.rados.rados.0.smithi115.stdout:401: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:44.356 INFO:tasks.rados.rados.0.smithi115.stdout:402: snap_create 2022-04-22T17:26:44.356 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 3 to smithi11541303-37 2022-04-22T17:26:44.358 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 4 to smithi11541303-37 2022-04-22T17:26:44.359 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 5 to smithi11541303-37 2022-04-22T17:26:44.360 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 141 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:26:44.362 INFO:tasks.rados.rados.0.smithi115.stdout:401: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:44.389 INFO:tasks.rados.rados.0.smithi115.stdout:398: finishing write tid 6 to smithi11541303-37 2022-04-22T17:26:44.389 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 79 (ObjNum 136 snap 35 seq_num 136) dirty exists 2022-04-22T17:26:44.389 INFO:tasks.rados.rados.0.smithi115.stdout:398: left oid 37 (ObjNum 136 snap 35 seq_num 136) 2022-04-22T17:26:44.397 INFO:tasks.rados.rados.0.smithi115.stdout:399: finishing copy_from to smithi11541303-21 2022-04-22T17:26:44.397 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 88 (ObjNum 110 snap 25 seq_num 110) dirty exists 2022-04-22T17:26:44.398 INFO:tasks.rados.rados.0.smithi115.stdout:399: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:26:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:44 smithi121 bash[12679]: cluster 2022-04-22T17:26:43.204995+0000 mgr.x (mgr.24749) 30 : cluster [DBG] pgmap v38: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2022-04-22T17:26:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:44 smithi121 bash[12679]: cluster 2022-04-22T17:26:43.234797+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2022-04-22T17:26:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:44 smithi115 bash[12737]: cluster 2022-04-22T17:26:43.204995+0000 mgr.x (mgr.24749) 30 : cluster [DBG] pgmap v38: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2022-04-22T17:26:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:44 smithi115 bash[12737]: cluster 2022-04-22T17:26:43.234797+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2022-04-22T17:26:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:44 smithi115 bash[18070]: cluster 2022-04-22T17:26:43.204995+0000 mgr.x (mgr.24749) 30 : cluster [DBG] pgmap v38: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2022-04-22T17:26:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:44 smithi115 bash[18070]: cluster 2022-04-22T17:26:43.234797+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2022-04-22T17:26:45.265 INFO:tasks.rados.rados.0.smithi115.stdout:398: done (4 left) 2022-04-22T17:26:45.265 INFO:tasks.rados.rados.0.smithi115.stdout:399: done (3 left) 2022-04-22T17:26:45.265 INFO:tasks.rados.rados.0.smithi115.stdout:400: done (2 left) 2022-04-22T17:26:45.265 INFO:tasks.rados.rados.0.smithi115.stdout:401: done (1 left) 2022-04-22T17:26:45.265 INFO:tasks.rados.rados.0.smithi115.stdout:402: done (0 left) 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:403: read oid 31 snap -1 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:403: expect deleted 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:404: rollback oid 22 current snap is 36 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 22 to 31 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:405: read oid 48 snap -1 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:405: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:26:45.266 INFO:tasks.rados.rados.0.smithi115.stdout:406: read oid 3 snap 29 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:406: expect (ObjNum 54 snap 4 seq_num 54) 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:407: read oid 34 snap -1 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:407: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:408: read oid 7 snap -1 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:408: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:45.267 INFO:tasks.rados.rados.0.smithi115.stdout:409: snap_remove snap 30 2022-04-22T17:26:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:45 smithi121 bash[12679]: cluster 2022-04-22T17:26:44.243574+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2022-04-22T17:26:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:45 smithi115 bash[12737]: cluster 2022-04-22T17:26:44.243574+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2022-04-22T17:26:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:45 smithi115 bash[18070]: cluster 2022-04-22T17:26:44.243574+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2022-04-22T17:26:46.179 INFO:tasks.rados.rados.0.smithi115.stdout:409: done (6 left) 2022-04-22T17:26:46.179 INFO:tasks.rados.rados.0.smithi115.stdout:410: read oid 14 snap -1 2022-04-22T17:26:46.179 INFO:tasks.rados.rados.0.smithi115.stdout:410: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:46.180 INFO:tasks.rados.rados.0.smithi115.stdout:411: read oid 44 snap -1 2022-04-22T17:26:46.180 INFO:tasks.rados.rados.0.smithi115.stdout:411: expect deleted 2022-04-22T17:26:46.180 INFO:tasks.rados.rados.0.smithi115.stdout:412: delete oid 9 current snap is 36 2022-04-22T17:26:46.180 INFO:tasks.rados.rados.0.smithi115.stdout:408: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:46.183 INFO:tasks.rados.rados.0.smithi115.stdout:405: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:26:46.195 INFO:tasks.rados.rados.0.smithi115.stdout:406: expect (ObjNum 54 snap 4 seq_num 54) 2022-04-22T17:26:46.242 INFO:tasks.rados.rados.0.smithi115.stdout:407: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:26:46.264 INFO:tasks.rados.rados.0.smithi115.stdout:404: finishing rollback tid 0 to smithi11541303-22 2022-04-22T17:26:46.264 INFO:tasks.rados.rados.0.smithi115.stdout:404: finishing rollback tid 1 to smithi11541303-22 2022-04-22T17:26:46.264 INFO:tasks.rados.rados.0.smithi115.stdout:403: done (8 left) 2022-04-22T17:26:46.264 INFO:tasks.rados.rados.0.smithi115.stdout:405: done (7 left) 2022-04-22T17:26:46.264 INFO:tasks.rados.rados.0.smithi115.stdout:406: done (6 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:407: done (5 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:408: done (4 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:412: done (3 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:413: rmattr oid 37 current snap is 36 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:413: done (3 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:414: rmattr oid 18 current snap is 36 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:414: done (3 left) 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:415: rmattr oid 45 current snap is 36 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:404: finishing rollback tid 2 to smithi11541303-22 2022-04-22T17:26:46.265 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 91 (ObjNum 120 snap 29 seq_num 120) dirty exists 2022-04-22T17:26:46.266 INFO:tasks.rados.rados.0.smithi115.stdout:404: done (3 left) 2022-04-22T17:26:46.266 INFO:tasks.rados.rados.0.smithi115.stdout:411: done (2 left) 2022-04-22T17:26:46.266 INFO:tasks.rados.rados.0.smithi115.stdout:416: copy_from oid 12 from oid 11 current snap is 36 2022-04-22T17:26:46.266 INFO:tasks.rados.rados.0.smithi115.stdout:410: expect (ObjNum 13 snap 0 seq_num 13) 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:416: finishing copy_from to smithi11541303-12 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:416: got expected ENOENT (src dne) 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:416: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 106 (ObjNum 123 snap 30 seq_num 123) dirty exists 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:410: done (2 left) 2022-04-22T17:26:46.270 INFO:tasks.rados.rados.0.smithi115.stdout:415: done (1 left) 2022-04-22T17:26:46.271 INFO:tasks.rados.rados.0.smithi115.stdout:416: done (0 left) 2022-04-22T17:26:46.271 INFO:tasks.rados.rados.0.smithi115.stdout:417: delete oid 43 current snap is 36 2022-04-22T17:26:46.273 INFO:tasks.rados.rados.0.smithi115.stdout:417: done (0 left) 2022-04-22T17:26:46.274 INFO:tasks.rados.rados.0.smithi115.stdout:418: rollback oid 30 current snap is 36 2022-04-22T17:26:46.274 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 28 2022-04-22T17:26:46.274 INFO:tasks.rados.rados.0.smithi115.stdout:419: read oid 19 snap -1 2022-04-22T17:26:46.274 INFO:tasks.rados.rados.0.smithi115.stdout:419: expect (ObjNum 127 snap 33 seq_num 127) 2022-04-22T17:26:46.274 INFO:tasks.rados.rados.0.smithi115.stdout:420: snap_remove snap 35 2022-04-22T17:26:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:46 smithi121 bash[12679]: cluster 2022-04-22T17:26:45.205557+0000 mgr.x (mgr.24749) 31 : cluster [DBG] pgmap v41: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:46 smithi121 bash[12679]: cluster 2022-04-22T17:26:45.254667+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2022-04-22T17:26:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:46 smithi121 bash[12679]: cluster 2022-04-22T17:26:46.167940+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2022-04-22T17:26:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[12737]: cluster 2022-04-22T17:26:45.205557+0000 mgr.x (mgr.24749) 31 : cluster [DBG] pgmap v41: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[12737]: cluster 2022-04-22T17:26:45.254667+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2022-04-22T17:26:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[12737]: cluster 2022-04-22T17:26:46.167940+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2022-04-22T17:26:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[18070]: cluster 2022-04-22T17:26:45.205557+0000 mgr.x (mgr.24749) 31 : cluster [DBG] pgmap v41: 121 pgs: 2 active+clean+snaptrim_wait, 112 active+clean, 7 active+clean+snaptrim; 114 MiB data, 304 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[18070]: cluster 2022-04-22T17:26:45.254667+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2022-04-22T17:26:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:46 smithi115 bash[18070]: cluster 2022-04-22T17:26:46.167940+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2022-04-22T17:26:47.275 INFO:tasks.rados.rados.0.smithi115.stdout:420: done (2 left) 2022-04-22T17:26:47.275 INFO:tasks.rados.rados.0.smithi115.stdout:421: delete oid 12 current snap is 36 2022-04-22T17:26:47.275 INFO:tasks.rados.rados.0.smithi115.stdout:418: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:26:47.276 INFO:tasks.rados.rados.0.smithi115.stdout:418: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:26:47.276 INFO:tasks.rados.rados.0.smithi115.stdout:418: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:26:47.276 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 110 (ObjNum 66 snap 6 seq_num 66) dirty exists 2022-04-22T17:26:47.276 INFO:tasks.rados.rados.0.smithi115.stdout:419: expect (ObjNum 127 snap 33 seq_num 127) 2022-04-22T17:26:47.496 INFO:tasks.rados.rados.0.smithi115.stdout:418: done (2 left) 2022-04-22T17:26:47.497 INFO:tasks.rados.rados.0.smithi115.stdout:419: done (1 left) 2022-04-22T17:26:47.497 INFO:tasks.rados.rados.0.smithi115.stdout:421: done (0 left) 2022-04-22T17:26:47.497 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 36 2022-04-22T17:26:47.497 INFO:tasks.rados.rados.0.smithi115.stdout:422: seq_num 140 ranges {0=753664,753664=704512,1458176=524288,1982464=376832} 2022-04-22T17:26:47.505 INFO:tasks.rados.rados.0.smithi115.stdout:422: writing smithi11541303-12 from 0 to 753664 tid 1 2022-04-22T17:26:47.511 INFO:tasks.rados.rados.0.smithi115.stdout:422: writing smithi11541303-12 from 753664 to 1458176 tid 2 2022-04-22T17:26:47.514 INFO:tasks.rados.rados.0.smithi115.stdout:422: writing smithi11541303-12 from 1458176 to 1982464 tid 3 2022-04-22T17:26:47.516 INFO:tasks.rados.rados.0.smithi115.stdout:422: writing smithi11541303-12 from 1982464 to 2359296 tid 4 2022-04-22T17:26:47.517 INFO:tasks.rados.rados.0.smithi115.stdout:423: snap_remove snap 26 2022-04-22T17:26:48.284 INFO:tasks.rados.rados.0.smithi115.stdout:423: done (1 left) 2022-04-22T17:26:48.285 INFO:tasks.rados.rados.0.smithi115.stdout:424: snap_remove snap 31 2022-04-22T17:26:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:48 smithi115 bash[12737]: cluster 2022-04-22T17:26:47.206353+0000 mgr.x (mgr.24749) 32 : cluster [DBG] pgmap v44: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:48 smithi115 bash[12737]: cluster 2022-04-22T17:26:47.267459+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2022-04-22T17:26:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:48 smithi115 bash[18070]: cluster 2022-04-22T17:26:47.206353+0000 mgr.x (mgr.24749) 32 : cluster [DBG] pgmap v44: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:48 smithi115 bash[18070]: cluster 2022-04-22T17:26:47.267459+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2022-04-22T17:26:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:48 smithi121 bash[12679]: cluster 2022-04-22T17:26:47.206353+0000 mgr.x (mgr.24749) 32 : cluster [DBG] pgmap v44: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:48 smithi121 bash[12679]: cluster 2022-04-22T17:26:47.267459+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2022-04-22T17:26:49.296 INFO:tasks.rados.rados.0.smithi115.stdout:424: done (1 left) 2022-04-22T17:26:49.297 INFO:tasks.rados.rados.0.smithi115.stdout:425: snap_remove snap 29 2022-04-22T17:26:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:49 smithi115 bash[12737]: cluster 2022-04-22T17:26:48.274431+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2022-04-22T17:26:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:49 smithi115 bash[18070]: cluster 2022-04-22T17:26:48.274431+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2022-04-22T17:26:49.766 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:26:49 smithi121 bash[18604]: debug 2022-04-22T17:26:49.296+0000 7faa0f096700 -1 osd.5 pg_epoch: 135 pg[5.7s0( v 135'231 (0'0,135'231] local-lis/les=63/64 n=12 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [5,1,0]p5(0) r=0 lpr=63 crt=135'231 lcod 135'229 mlcod 135'229 active+clean+snaptrim trimq=[38~1] ps=[2d~1,30~1,32~1,37~1,39~1,3d~1,41~1]] removing snap head 2022-04-22T17:26:49.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:49 smithi121 bash[12679]: cluster 2022-04-22T17:26:48.274431+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2022-04-22T17:26:50.296 INFO:tasks.rados.rados.0.smithi115.stdout:425: done (1 left) 2022-04-22T17:26:50.296 INFO:tasks.rados.rados.0.smithi115.stdout:426: snap_remove snap 27 2022-04-22T17:26:50.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:50 smithi121 bash[12679]: cluster 2022-04-22T17:26:49.207057+0000 mgr.x (mgr.24749) 33 : cluster [DBG] pgmap v47: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:50.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:50 smithi121 bash[12679]: cluster 2022-04-22T17:26:49.284984+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2022-04-22T17:26:50.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:50 smithi115 bash[12737]: cluster 2022-04-22T17:26:49.207057+0000 mgr.x (mgr.24749) 33 : cluster [DBG] pgmap v47: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:50.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:50 smithi115 bash[12737]: cluster 2022-04-22T17:26:49.284984+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2022-04-22T17:26:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:50 smithi115 bash[18070]: cluster 2022-04-22T17:26:49.207057+0000 mgr.x (mgr.24749) 33 : cluster [DBG] pgmap v47: 121 pgs: 121 active+clean; 128 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 799 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2022-04-22T17:26:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:50 smithi115 bash[18070]: cluster 2022-04-22T17:26:49.284984+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:426: done (1 left) 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:427: setattr oid 21 current snap is 36 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 1 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 2 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 3 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 4 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 5 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:422: finishing write tid 6 to smithi11541303-12 2022-04-22T17:26:51.179 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 86 (ObjNum 140 snap 36 seq_num 140) dirty exists 2022-04-22T17:26:51.180 INFO:tasks.rados.rados.0.smithi115.stdout:422: left oid 12 (ObjNum 140 snap 36 seq_num 140) 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:422: done (1 left) 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:428: read oid 27 snap -1 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:428: expect (ObjNum 103 snap 19 seq_num 103) 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:429: read oid 37 snap -1 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:429: expect (ObjNum 136 snap 35 seq_num 136) 2022-04-22T17:26:51.182 INFO:tasks.rados.rados.0.smithi115.stdout:430: snap_remove snap 21 2022-04-22T17:26:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:51 smithi121 bash[12679]: cluster 2022-04-22T17:26:50.288290+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2022-04-22T17:26:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:51 smithi121 bash[12679]: cluster 2022-04-22T17:26:51.170282+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2022-04-22T17:26:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:51 smithi115 bash[12737]: cluster 2022-04-22T17:26:50.288290+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2022-04-22T17:26:51.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:51 smithi115 bash[12737]: cluster 2022-04-22T17:26:51.170282+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2022-04-22T17:26:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:51 smithi115 bash[18070]: cluster 2022-04-22T17:26:50.288290+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2022-04-22T17:26:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:51 smithi115 bash[18070]: cluster 2022-04-22T17:26:51.170282+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2022-04-22T17:26:52.196 INFO:tasks.rados.rados.0.smithi115.stdout:430: done (3 left) 2022-04-22T17:26:52.196 INFO:tasks.rados.rados.0.smithi115.stdout:431: read oid 3 snap -1 2022-04-22T17:26:52.197 INFO:tasks.rados.rados.0.smithi115.stdout:431: expect (ObjNum 54 snap 4 seq_num 54) 2022-04-22T17:26:52.197 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 92 (ObjNum 110 snap 25 seq_num 110) dirty exists 2022-04-22T17:26:52.197 INFO:tasks.rados.rados.0.smithi115.stdout:429: expect (ObjNum 136 snap 35 seq_num 136) 2022-04-22T17:26:52.277 INFO:tasks.rados.rados.0.smithi115.stdout:428: expect (ObjNum 103 snap 19 seq_num 103) 2022-04-22T17:26:52.356 INFO:tasks.rados.rados.0.smithi115.stdout:431: expect (ObjNum 54 snap 4 seq_num 54) 2022-04-22T17:26:52.397 INFO:tasks.rados.rados.0.smithi115.stdout:427: done (3 left) 2022-04-22T17:26:52.398 INFO:tasks.rados.rados.0.smithi115.stdout:428: done (2 left) 2022-04-22T17:26:52.398 INFO:tasks.rados.rados.0.smithi115.stdout:429: done (1 left) 2022-04-22T17:26:52.398 INFO:tasks.rados.rados.0.smithi115.stdout:431: done (0 left) 2022-04-22T17:26:52.398 INFO:tasks.rados.rados.0.smithi115.stdout:432: delete oid 43 current snap is 36 2022-04-22T17:26:52.399 INFO:tasks.rados.rados.0.smithi115.stdout:432: done (0 left) 2022-04-22T17:26:52.399 INFO:tasks.rados.rados.0.smithi115.stdout:433: snap_create 2022-04-22T17:26:52.545 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:26:52 smithi115 bash[26675]: debug 2022-04-22T17:26:52.200+0000 7faa068b7700 -1 osd.2 pg_epoch: 138 pg[5.as0( v 137'71 (0'0,137'71] local-lis/les=63/64 n=5 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,4,3]p2(0) r=0 lpr=63 crt=137'71 lcod 134'69 mlcod 134'69 active+clean+snaptrim trimq=[2b~1] ps=[32~2,36~3,41~1]] removing snap head 2022-04-22T17:26:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:52 smithi115 bash[12737]: cluster 2022-04-22T17:26:51.207561+0000 mgr.x (mgr.24749) 34 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 585 KiB/s wr, 10 op/s 2022-04-22T17:26:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:52 smithi115 bash[12737]: cluster 2022-04-22T17:26:52.187451+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2022-04-22T17:26:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:52 smithi115 bash[18070]: cluster 2022-04-22T17:26:51.207561+0000 mgr.x (mgr.24749) 34 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 585 KiB/s wr, 10 op/s 2022-04-22T17:26:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:52 smithi115 bash[18070]: cluster 2022-04-22T17:26:52.187451+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2022-04-22T17:26:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:52 smithi121 bash[12679]: cluster 2022-04-22T17:26:51.207561+0000 mgr.x (mgr.24749) 34 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 585 KiB/s wr, 10 op/s 2022-04-22T17:26:52.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:52 smithi121 bash[12679]: cluster 2022-04-22T17:26:52.187451+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2022-04-22T17:26:53.325 INFO:tasks.rados.rados.0.smithi115.stdout:433: done (0 left) 2022-04-22T17:26:53.325 INFO:tasks.rados.rados.0.smithi115.stdout:434: read oid 45 snap -1 2022-04-22T17:26:53.325 INFO:tasks.rados.rados.0.smithi115.stdout:434: expect (ObjNum 123 snap 30 seq_num 123) 2022-04-22T17:26:53.325 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 37 2022-04-22T17:26:53.326 INFO:tasks.rados.rados.0.smithi115.stdout:435: seq_num 142 ranges {4022272=786432,4808704=475136,5283840=540672,5824512=589824} 2022-04-22T17:26:53.351 INFO:tasks.rados.rados.0.smithi115.stdout:435: writing smithi11541303-33 from 4022272 to 4808704 tid 1 2022-04-22T17:26:53.354 INFO:tasks.rados.rados.0.smithi115.stdout:435: writing smithi11541303-33 from 4808704 to 5283840 tid 2 2022-04-22T17:26:53.357 INFO:tasks.rados.rados.0.smithi115.stdout:435: writing smithi11541303-33 from 5283840 to 5824512 tid 3 2022-04-22T17:26:53.361 INFO:tasks.rados.rados.0.smithi115.stdout:435: writing smithi11541303-33 from 5824512 to 6414336 tid 4 2022-04-22T17:26:53.361 INFO:tasks.rados.rados.0.smithi115.stdout:436: delete oid 2 current snap is 37 2022-04-22T17:26:53.361 INFO:tasks.rados.rados.0.smithi115.stdout:434: expect (ObjNum 123 snap 30 seq_num 123) 2022-04-22T17:26:53.639 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 1 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 2 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 3 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 4 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 5 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: finishing write tid 6 to smithi11541303-33 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 99 (ObjNum 142 snap 37 seq_num 142) dirty exists 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:435: left oid 33 (ObjNum 142 snap 37 seq_num 142) 2022-04-22T17:26:53.640 INFO:tasks.rados.rados.0.smithi115.stdout:434: done (2 left) 2022-04-22T17:26:53.641 INFO:tasks.rados.rados.0.smithi115.stdout:435: done (1 left) 2022-04-22T17:26:53.641 INFO:tasks.rados.rados.0.smithi115.stdout:436: done (0 left) 2022-04-22T17:26:53.641 INFO:tasks.rados.rados.0.smithi115.stdout:437: snap_create 2022-04-22T17:26:53.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:53.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:26:53.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:26:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:26:53.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:437: done (0 left) 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:438: rmattr oid 48 current snap is 38 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:438: done (0 left) 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:439: rmattr oid 31 current snap is 38 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:439: done (0 left) 2022-04-22T17:26:54.335 INFO:tasks.rados.rados.0.smithi115.stdout:440: read oid 37 snap -1 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:440: expect (ObjNum 136 snap 35 seq_num 136) 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:441: rollback oid 45 current snap is 38 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 28 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:442: read oid 6 snap -1 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:442: expect deleted 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:443: rollback oid 50 current snap is 38 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 37 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:444: rollback oid 17 current snap is 38 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 17 to 36 2022-04-22T17:26:54.336 INFO:tasks.rados.rados.0.smithi115.stdout:445: delete oid 1 current snap is 38 2022-04-22T17:26:54.344 INFO:tasks.rados.rados.0.smithi115.stdout:444: finishing rollback tid 0 to smithi11541303-17 2022-04-22T17:26:54.344 INFO:tasks.rados.rados.0.smithi115.stdout:444: finishing rollback tid 1 to smithi11541303-17 2022-04-22T17:26:54.344 INFO:tasks.rados.rados.0.smithi115.stdout:441: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:26:54.345 INFO:tasks.rados.rados.0.smithi115.stdout:441: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:26:54.345 INFO:tasks.rados.rados.0.smithi115.stdout:444: finishing rollback tid 2 to smithi11541303-17 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 174 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:441: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 144 (ObjNum 111 snap 26 seq_num 111) dirty exists 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:441: done (5 left) 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:442: done (4 left) 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:444: done (3 left) 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:445: done (2 left) 2022-04-22T17:26:54.346 INFO:tasks.rados.rados.0.smithi115.stdout:446: delete oid 29 current snap is 38 2022-04-22T17:26:54.347 INFO:tasks.rados.rados.0.smithi115.stdout:443: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:26:54.347 INFO:tasks.rados.rados.0.smithi115.stdout:443: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:26:54.347 INFO:tasks.rados.rados.0.smithi115.stdout:443: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:26:54.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 112 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:26:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:443: done (2 left) 2022-04-22T17:26:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:446: done (1 left) 2022-04-22T17:26:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:447: read oid 26 snap 37 2022-04-22T17:26:54.351 INFO:tasks.rados.rados.0.smithi115.stdout:447: expect (ObjNum -1116464352 snap 22075 seq_num 3178369120) 2022-04-22T17:26:54.352 INFO:tasks.rados.rados.0.smithi115.stdout:448: snap_create 2022-04-22T17:26:54.352 INFO:tasks.rados.rados.0.smithi115.stdout:447: expect (ObjNum -1116464352 snap 22075 seq_num 3178369120) 2022-04-22T17:26:54.354 INFO:tasks.rados.rados.0.smithi115.stdout:440: expect (ObjNum 136 snap 35 seq_num 136) 2022-04-22T17:26:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[12679]: cluster 2022-04-22T17:26:53.208014+0000 mgr.x (mgr.24749) 35 : cluster [DBG] pgmap v53: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 576 KiB/s wr, 10 op/s 2022-04-22T17:26:54.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[12679]: cluster 2022-04-22T17:26:53.316919+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2022-04-22T17:26:54.517 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 systemd[1]: Stopping Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:26:54.517 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=warn ts=2022-04-22T17:26:54.213Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2022-04-22T17:26:54.517 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2022-04-22T17:26:54.517 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:561 msg="Stopping notify discovery manager..." 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:583 msg="Stopping scrape manager..." 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:543 msg="Scrape discovery manager stopped" 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:557 msg="Notify discovery manager stopped" 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.213Z caller=main.go:577 msg="Scrape manager stopped" 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.214Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2022-04-22T17:26:54.518 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.214Z caller=main.go:749 msg="Notifier manager stopped" 2022-04-22T17:26:54.519 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:54 smithi121 bash[28790]: level=info ts=2022-04-22T17:26:54.214Z caller=main.go:761 msg="See you next time!" 2022-04-22T17:26:54.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:54 smithi115 bash[12737]: cluster 2022-04-22T17:26:53.208014+0000 mgr.x (mgr.24749) 35 : cluster [DBG] pgmap v53: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 576 KiB/s wr, 10 op/s 2022-04-22T17:26:54.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:54 smithi115 bash[12737]: cluster 2022-04-22T17:26:53.316919+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2022-04-22T17:26:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:54 smithi115 bash[18070]: cluster 2022-04-22T17:26:53.208014+0000 mgr.x (mgr.24749) 35 : cluster [DBG] pgmap v53: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 576 KiB/s wr, 10 op/s 2022-04-22T17:26:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:54 smithi115 bash[18070]: cluster 2022-04-22T17:26:53.316919+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2022-04-22T17:26:55.341 INFO:tasks.rados.rados.0.smithi115.stdout:440: done (2 left) 2022-04-22T17:26:55.341 INFO:tasks.rados.rados.0.smithi115.stdout:447: done (1 left) 2022-04-22T17:26:55.341 INFO:tasks.rados.rados.0.smithi115.stdout:448: done (0 left) 2022-04-22T17:26:55.341 INFO:tasks.rados.rados.0.smithi115.stdout:449: delete oid 49 current snap is 39 2022-04-22T17:26:55.342 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:55 smithi121 docker[32382]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-prometheus.a 2022-04-22T17:26:55.342 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:55 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@prometheus.a.service: Succeeded. 2022-04-22T17:26:55.343 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:55 smithi121 systemd[1]: Stopped Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:26:55.343 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:55 smithi121 systemd[1]: Started Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:26:55.347 INFO:tasks.rados.rados.0.smithi115.stdout:449: done (0 left) 2022-04-22T17:26:55.347 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 39 2022-04-22T17:26:55.347 INFO:tasks.rados.rados.0.smithi115.stdout:450: seq_num 147 ranges {3620864=65536} 2022-04-22T17:26:55.369 INFO:tasks.rados.rados.0.smithi115.stdout:450: writing smithi11541303-5 from 3620864 to 3686400 tid 1 2022-04-22T17:26:55.369 INFO:tasks.rados.rados.0.smithi115.stdout:451: setattr oid 28 current snap is 39 2022-04-22T17:26:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:452: read oid 15 snap -1 2022-04-22T17:26:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:452: expect (ObjNum 116 snap 28 seq_num 116) 2022-04-22T17:26:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:453: rmattr oid 32 current snap is 39 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:453: done (3 left) 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:454: rollback oid 7 current snap is 39 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 7 to 36 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:455: rollback oid 45 current snap is 39 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 38 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:456: read oid 44 snap -1 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:456: expect deleted 2022-04-22T17:26:55.373 INFO:tasks.rados.rados.0.smithi115.stdout:457: snap_create 2022-04-22T17:26:55.374 INFO:tasks.rados.rados.0.smithi115.stdout:450: finishing write tid 1 to smithi11541303-5 2022-04-22T17:26:55.374 INFO:tasks.rados.rados.0.smithi115.stdout:450: finishing write tid 2 to smithi11541303-5 2022-04-22T17:26:55.377 INFO:tasks.rados.rados.0.smithi115.stdout:450: finishing write tid 3 to smithi11541303-5 2022-04-22T17:26:55.377 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 257 (ObjNum 147 snap 39 seq_num 147) dirty exists 2022-04-22T17:26:55.377 INFO:tasks.rados.rados.0.smithi115.stdout:450: left oid 5 (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:26:55.381 INFO:tasks.rados.rados.0.smithi115.stdout:455: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:26:55.382 INFO:tasks.rados.rados.0.smithi115.stdout:452: expect (ObjNum 116 snap 28 seq_num 116) 2022-04-22T17:26:55.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: cluster 2022-04-22T17:26:54.326579+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2022-04-22T17:26:55.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.073565+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.085069+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.085853+0000 mgr.x (mgr.24749) 36 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.087453+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.088358+0000 mgr.x (mgr.24749) 37 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.090629+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.100769+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[12737]: audit 2022-04-22T17:26:55.105058+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: cluster 2022-04-22T17:26:54.326579+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.073565+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.085069+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.085853+0000 mgr.x (mgr.24749) 36 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.087453+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.088358+0000 mgr.x (mgr.24749) 37 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.090629+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.100769+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:55.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:55 smithi115 bash[18070]: audit 2022-04-22T17:26:55.105058+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.455 INFO:tasks.rados.rados.0.smithi115.stdout:455: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:26:55.455 INFO:tasks.rados.rados.0.smithi115.stdout:455: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 148 (ObjNum 111 snap 26 seq_num 111) dirty exists 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:454: finishing rollback tid 0 to smithi11541303-7 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:454: finishing rollback tid 1 to smithi11541303-7 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:454: finishing rollback tid 2 to smithi11541303-7 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 79 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:26:55.456 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 71 (ObjNum 26 snap 0 seq_num 26) dirty exists 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: cluster 2022-04-22T17:26:54.326579+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.073565+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.085069+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.085853+0000 mgr.x (mgr.24749) 36 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.087453+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.088358+0000 mgr.x (mgr.24749) 37 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.090629+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.100769+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:26:55.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:55 smithi121 bash[12679]: audit 2022-04-22T17:26:55.105058+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:56.359 INFO:tasks.rados.rados.0.smithi115.stdout:450: done (6 left) 2022-04-22T17:26:56.359 INFO:tasks.rados.rados.0.smithi115.stdout:451: done (5 left) 2022-04-22T17:26:56.359 INFO:tasks.rados.rados.0.smithi115.stdout:452: done (4 left) 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:454: done (3 left) 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:455: done (2 left) 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:456: done (1 left) 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:457: done (0 left) 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:458: copy_from oid 1 from oid 7 current snap is 40 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 40 2022-04-22T17:26:56.360 INFO:tasks.rados.rados.0.smithi115.stdout:459: seq_num 148 ranges {270336=81920} 2022-04-22T17:26:56.363 INFO:tasks.rados.rados.0.smithi115.stdout:459: writing smithi11541303-48 from 270336 to 352256 tid 1 2022-04-22T17:26:56.363 INFO:tasks.rados.rados.0.smithi115.stdout:460: rmattr oid 50 current snap is 40 2022-04-22T17:26:56.363 INFO:tasks.rados.rados.0.smithi115.stdout:461: delete oid 14 current snap is 40 2022-04-22T17:26:56.366 INFO:tasks.rados.rados.0.smithi115.stdout:458: finishing copy_from to smithi11541303-1 2022-04-22T17:26:56.366 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 79 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:26:56.367 INFO:tasks.rados.rados.0.smithi115.stdout:458: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:26:56.368 INFO:tasks.rados.rados.0.smithi115.stdout:458: done (3 left) 2022-04-22T17:26:56.368 INFO:tasks.rados.rados.0.smithi115.stdout:461: done (2 left) 2022-04-22T17:26:56.368 INFO:tasks.rados.rados.0.smithi115.stdout:462: read oid 45 snap -1 2022-04-22T17:26:56.368 INFO:tasks.rados.rados.0.smithi115.stdout:462: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:26:56.368 INFO:tasks.rados.rados.0.smithi115.stdout:463: setattr oid 9 current snap is 40 2022-04-22T17:26:56.370 INFO:tasks.rados.rados.0.smithi115.stdout:459: finishing write tid 1 to smithi11541303-48 2022-04-22T17:26:56.370 INFO:tasks.rados.rados.0.smithi115.stdout:459: finishing write tid 2 to smithi11541303-48 2022-04-22T17:26:56.371 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 114 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:26:56.371 INFO:tasks.rados.rados.0.smithi115.stdout:460: done (3 left) 2022-04-22T17:26:56.372 INFO:tasks.rados.rados.0.smithi115.stdout:464: read oid 6 snap -1 2022-04-22T17:26:56.372 INFO:tasks.rados.rados.0.smithi115.stdout:464: expect deleted 2022-04-22T17:26:56.372 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 40 2022-04-22T17:26:56.372 INFO:tasks.rados.rados.0.smithi115.stdout:465: seq_num 150 ranges {0=114688} 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:465: writing smithi11541303-31 from 0 to 114688 tid 1 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:466: read oid 7 snap -1 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:466: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:467: read oid 16 snap -1 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:467: expect (ObjNum 113 snap 28 seq_num 113) 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:468: rmattr oid 46 current snap is 40 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:468: done (7 left) 2022-04-22T17:26:56.373 INFO:tasks.rados.rados.0.smithi115.stdout:469: setattr oid 25 current snap is 40 2022-04-22T17:26:56.374 INFO:tasks.rados.rados.0.smithi115.stdout:459: finishing write tid 3 to smithi11541303-48 2022-04-22T17:26:56.374 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 151 (ObjNum 148 snap 40 seq_num 148) dirty exists 2022-04-22T17:26:56.374 INFO:tasks.rados.rados.0.smithi115.stdout:459: left oid 48 (ObjNum 148 snap 40 seq_num 148) 2022-04-22T17:26:56.375 INFO:tasks.rados.rados.0.smithi115.stdout:459: done (7 left) 2022-04-22T17:26:56.375 INFO:tasks.rados.rados.0.smithi115.stdout:464: done (6 left) 2022-04-22T17:26:56.375 INFO:tasks.rados.rados.0.smithi115.stdout:470: rmattr oid 35 current snap is 40 2022-04-22T17:26:56.375 INFO:tasks.rados.rados.0.smithi115.stdout:471: setattr oid 12 current snap is 40 2022-04-22T17:26:56.377 INFO:tasks.rados.rados.0.smithi115.stdout:472: snap_create 2022-04-22T17:26:56.377 INFO:tasks.rados.rados.0.smithi115.stdout:466: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:26:56.381 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 258 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:26:56.382 INFO:tasks.rados.rados.0.smithi115.stdout:465: finishing write tid 1 to smithi11541303-31 2022-04-22T17:26:56.382 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 97 (ObjNum 103 snap 19 seq_num 103) dirty exists 2022-04-22T17:26:56.383 INFO:tasks.rados.rados.0.smithi115.stdout:465: finishing write tid 2 to smithi11541303-31 2022-04-22T17:26:56.384 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 108 (ObjNum 140 snap 36 seq_num 140) dirty exists 2022-04-22T17:26:56.385 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 176 (ObjNum 1 snap 1297622361 seq_num 3179372400) dirty exists 2022-04-22T17:26:56.385 INFO:tasks.rados.rados.0.smithi115.stdout:465: finishing write tid 3 to smithi11541303-31 2022-04-22T17:26:56.385 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 260 (ObjNum 150 snap 40 seq_num 150) dirty exists 2022-04-22T17:26:56.385 INFO:tasks.rados.rados.0.smithi115.stdout:465: left oid 31 (ObjNum 150 snap 40 seq_num 150) 2022-04-22T17:26:56.392 INFO:tasks.rados.rados.0.smithi115.stdout:462: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:26:56.549 INFO:tasks.rados.rados.0.smithi115.stdout:467: expect (ObjNum 113 snap 28 seq_num 113) 2022-04-22T17:26:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[12679]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:26:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[12679]: cluster 2022-04-22T17:26:55.208564+0000 mgr.x (mgr.24749) 39 : cluster [DBG] pgmap v56: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[12679]: cluster 2022-04-22T17:26:55.332952+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2022-04-22T17:26:56.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2022-04-22T17:26:56.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2022-04-22T17:26:56.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2022-04-22T17:26:56.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:339 host_details="(Linux 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 smithi121 (none))" 2022-04-22T17:26:56.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.495Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.497Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.497Z caller=main.go:678 msg="Starting TSDB ..." 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.500Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.500Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.512Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.512Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.512Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=12.403942ms 2022-04-22T17:26:56.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.515Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2022-04-22T17:26:56.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.515Z caller=main.go:695 msg="TSDB started" 2022-04-22T17:26:56.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.515Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:26:56.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.543Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:26:56.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:26:56 smithi121 bash[32438]: level=info ts=2022-04-22T17:26:56.543Z caller=main.go:646 msg="Server is ready to receive web requests." 2022-04-22T17:26:56.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[12737]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:26:56.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[12737]: cluster 2022-04-22T17:26:55.208564+0000 mgr.x (mgr.24749) 39 : cluster [DBG] pgmap v56: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:56.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[12737]: cluster 2022-04-22T17:26:55.332952+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2022-04-22T17:26:56.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[18070]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:26:56.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[18070]: cluster 2022-04-22T17:26:55.208564+0000 mgr.x (mgr.24749) 39 : cluster [DBG] pgmap v56: 121 pgs: 121 active+clean; 125 MiB data, 350 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:26:56.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:56 smithi115 bash[18070]: cluster 2022-04-22T17:26:55.332952+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2022-04-22T17:26:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:462: done (8 left) 2022-04-22T17:26:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:463: done (7 left) 2022-04-22T17:26:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:465: done (6 left) 2022-04-22T17:26:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:466: done (5 left) 2022-04-22T17:26:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:467: done (4 left) 2022-04-22T17:26:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:469: done (3 left) 2022-04-22T17:26:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:470: done (2 left) 2022-04-22T17:26:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:471: done (1 left) 2022-04-22T17:26:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:472: done (0 left) 2022-04-22T17:26:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:473: setattr oid 39 current snap is 41 2022-04-22T17:26:57.369 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 41 2022-04-22T17:26:57.369 INFO:tasks.rados.rados.0.smithi115.stdout:474: seq_num 151 ranges {3825664=131072} 2022-04-22T17:26:57.394 INFO:tasks.rados.rados.0.smithi115.stdout:474: writing smithi11541303-22 from 3825664 to 3956736 tid 1 2022-04-22T17:26:57.395 INFO:tasks.rados.rados.0.smithi115.stdout:475: snap_remove snap 40 2022-04-22T17:26:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:57 smithi115 bash[18070]: cluster 2022-04-22T17:26:56.350042+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2022-04-22T17:26:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:57 smithi115 bash[12737]: cluster 2022-04-22T17:26:56.350042+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2022-04-22T17:26:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:57 smithi121 bash[12679]: cluster 2022-04-22T17:26:56.350042+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2022-04-22T17:26:58.365 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: cluster 2022-04-22T17:26:57.209297+0000 mgr.x (mgr.24749) 40 : cluster [DBG] pgmap v59: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:26:58.365 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: cluster 2022-04-22T17:26:57.357492+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2022-04-22T17:26:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:475: done (2 left) 2022-04-22T17:26:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:476: rmattr oid 7 current snap is 41 2022-04-22T17:26:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:476: done (2 left) 2022-04-22T17:26:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:477: copy_from oid 19 from oid 37 current snap is 41 2022-04-22T17:26:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 99 (ObjNum 99 snap 19 seq_num 99) dirty exists 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:474: finishing write tid 1 to smithi11541303-22 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:474: finishing write tid 2 to smithi11541303-22 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:474: finishing write tid 3 to smithi11541303-22 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:474: left oid 22 (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:473: done (2 left) 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:474: done (1 left) 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:478: copy_from oid 48 from oid 31 current snap is 41 2022-04-22T17:26:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:479: delete oid 16 current snap is 41 2022-04-22T17:26:58.385 INFO:tasks.rados.rados.0.smithi115.stdout:479: done (2 left) 2022-04-22T17:26:58.385 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 41 2022-04-22T17:26:58.385 INFO:tasks.rados.rados.0.smithi115.stdout:480: seq_num 153 ranges {6881280=163840} 2022-04-22T17:26:58.428 INFO:tasks.rados.rados.0.smithi115.stdout:480: writing smithi11541303-21 from 6881280 to 7045120 tid 1 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:481: delete oid 39 current snap is 41 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:478: finishing copy_from to smithi11541303-48 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 260 (ObjNum 150 snap 40 seq_num 150) dirty exists 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:478: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:477: finishing copy_from to smithi11541303-19 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 79 (ObjNum 136 snap 35 seq_num 136) dirty exists 2022-04-22T17:26:58.429 INFO:tasks.rados.rados.0.smithi115.stdout:477: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:26:58.431 INFO:tasks.rados.rados.0.smithi115.stdout:477: done (3 left) 2022-04-22T17:26:58.432 INFO:tasks.rados.rados.0.smithi115.stdout:478: done (2 left) 2022-04-22T17:26:58.432 INFO:tasks.rados.rados.0.smithi115.stdout:481: done (1 left) 2022-04-22T17:26:58.432 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 41 2022-04-22T17:26:58.432 INFO:tasks.rados.rados.0.smithi115.stdout:482: seq_num 155 ranges {2023424=204800} 2022-04-22T17:26:58.444 INFO:tasks.rados.rados.0.smithi115.stdout:482: writing smithi11541303-15 from 2023424 to 2228224 tid 1 2022-04-22T17:26:58.445 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 41 2022-04-22T17:26:58.445 INFO:tasks.rados.rados.0.smithi115.stdout:480: finishing write tid 1 to smithi11541303-21 2022-04-22T17:26:58.445 INFO:tasks.rados.rados.0.smithi115.stdout:483: seq_num 156 ranges {0=221184} 2022-04-22T17:26:58.446 INFO:tasks.rados.rados.0.smithi115.stdout:483: writing smithi11541303-29 from 0 to 221184 tid 1 2022-04-22T17:26:58.446 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 41 2022-04-22T17:26:58.446 INFO:tasks.rados.rados.0.smithi115.stdout:484: seq_num 157 ranges {1417216=237568} 2022-04-22T17:26:58.455 INFO:tasks.rados.rados.0.smithi115.stdout:484: writing smithi11541303-30 from 1417216 to 1654784 tid 1 2022-04-22T17:26:58.456 INFO:tasks.rados.rados.0.smithi115.stdout:485: read oid 41 snap -1 2022-04-22T17:26:58.456 INFO:tasks.rados.rados.0.smithi115.stdout:485: expect (ObjNum 7 snap 0 seq_num 7) 2022-04-22T17:26:58.456 INFO:tasks.rados.rados.0.smithi115.stdout:486: snap_remove snap 37 2022-04-22T17:26:58.697 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: cluster 2022-04-22T17:26:57.209297+0000 mgr.x (mgr.24749) 40 : cluster [DBG] pgmap v59: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: cluster 2022-04-22T17:26:57.357492+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: audit 2022-04-22T17:26:58.059575+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: audit 2022-04-22T17:26:58.062190+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: audit 2022-04-22T17:26:58.066377+0000 mon.b (mon.2) 72 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: audit 2022-04-22T17:26:58.068837+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: audit 2022-04-22T17:26:58.070358+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:58.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:58 smithi121 bash[12679]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:26:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: cluster 2022-04-22T17:26:57.209297+0000 mgr.x (mgr.24749) 40 : cluster [DBG] pgmap v59: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:26:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: cluster 2022-04-22T17:26:57.357492+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2022-04-22T17:26:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: audit 2022-04-22T17:26:58.059575+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: audit 2022-04-22T17:26:58.062190+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: audit 2022-04-22T17:26:58.066377+0000 mon.b (mon.2) 72 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: audit 2022-04-22T17:26:58.068837+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: audit 2022-04-22T17:26:58.070358+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[18070]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:26:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: audit 2022-04-22T17:26:58.059575+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:26:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: audit 2022-04-22T17:26:58.062190+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: audit 2022-04-22T17:26:58.066377+0000 mon.b (mon.2) 72 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:26:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: audit 2022-04-22T17:26:58.068837+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:26:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: audit 2022-04-22T17:26:58.070358+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:26:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:58 smithi115 bash[12737]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:26:59.029 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 33s ago 3m 12.2M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3m) 33s ago 3m 22.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4m) 33s ago 4m 12.5M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4m) 33s ago 4m 14.8M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (50s) 33s ago 9m 434M - 17.0.0-11772-g72067459 6c95b0749680 988b90e88450 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (11m) 33s ago 11m 355M - 16.2.5 46cf6318c64e d7208c0f97cf 2022-04-22T17:26:59.030 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (11m) 33s ago 11m 49.9M 2048M 16.2.5 46cf6318c64e c6b4ba941f5c 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (9m) 33s ago 9m 44.4M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 33s ago 9m 42.7M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 33s ago 4m 8636k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4m) 33s ago 4m 8599k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (8m) 33s ago 8m 72.6M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 33s ago 8m 63.3M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 33s ago 7m 69.5M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 33s ago 7m 50.6M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:26:59.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6m) 33s ago 6m 61.9M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:26:59.032 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6m) 33s ago 6m 63.6M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:26:59.032 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5m) 33s ago 5m 64.0M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:26:59.032 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5m) 33s ago 5m 58.6M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:26:59.032 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 starting - - - - 2022-04-22T17:26:59.384 INFO:tasks.rados.rados.0.smithi115.stdout:486: done (5 left) 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:487: rollback oid 25 current snap is 41 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 36 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:480: finishing write tid 2 to smithi11541303-21 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:480: finishing write tid 3 to smithi11541303-21 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 104 (ObjNum 153 snap 41 seq_num 153) dirty exists 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:480: left oid 21 (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:482: finishing write tid 1 to smithi11541303-15 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:483: finishing write tid 1 to smithi11541303-29 2022-04-22T17:26:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:482: finishing write tid 2 to smithi11541303-15 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:483: finishing write tid 2 to smithi11541303-29 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:482: finishing write tid 3 to smithi11541303-15 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 46 (ObjNum 155 snap 41 seq_num 155) dirty exists 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:482: left oid 15 (ObjNum 155 snap 41 seq_num 155) 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:483: finishing write tid 3 to smithi11541303-29 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 75 (ObjNum 156 snap 41 seq_num 156) dirty exists 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:483: left oid 29 (ObjNum 156 snap 41 seq_num 156) 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:484: finishing write tid 1 to smithi11541303-30 2022-04-22T17:26:59.386 INFO:tasks.rados.rados.0.smithi115.stdout:485: expect (ObjNum 7 snap 0 seq_num 7) 2022-04-22T17:26:59.391 INFO:tasks.rados.rados.0.smithi115.stdout:484: finishing write tid 2 to smithi11541303-30 2022-04-22T17:26:59.391 INFO:tasks.rados.rados.0.smithi115.stdout:484: finishing write tid 3 to smithi11541303-30 2022-04-22T17:26:59.391 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 262 (ObjNum 157 snap 41 seq_num 157) dirty exists 2022-04-22T17:26:59.391 INFO:tasks.rados.rados.0.smithi115.stdout:484: left oid 30 (ObjNum 157 snap 41 seq_num 157) 2022-04-22T17:26:59.392 INFO:tasks.rados.rados.0.smithi115.stdout:480: done (5 left) 2022-04-22T17:26:59.392 INFO:tasks.rados.rados.0.smithi115.stdout:482: done (4 left) 2022-04-22T17:26:59.392 INFO:tasks.rados.rados.0.smithi115.stdout:483: done (3 left) 2022-04-22T17:26:59.392 INFO:tasks.rados.rados.0.smithi115.stdout:484: done (2 left) 2022-04-22T17:26:59.392 INFO:tasks.rados.rados.0.smithi115.stdout:485: done (1 left) 2022-04-22T17:26:59.393 INFO:tasks.rados.rados.0.smithi115.stdout:488: delete oid 32 current snap is 41 2022-04-22T17:26:59.393 INFO:tasks.rados.rados.0.smithi115.stdout:487: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:26:59.394 INFO:tasks.rados.rados.0.smithi115.stdout:488: done (1 left) 2022-04-22T17:26:59.394 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 41 2022-04-22T17:26:59.394 INFO:tasks.rados.rados.0.smithi115.stdout:489: seq_num 159 ranges {0=270336} 2022-04-22T17:26:59.396 INFO:tasks.rados.rados.0.smithi115.stdout:489: writing smithi11541303-4 from 0 to 270336 tid 1 2022-04-22T17:26:59.396 INFO:tasks.rados.rados.0.smithi115.stdout:490: snap_remove snap 38 2022-04-22T17:26:59.440 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:26:59.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 14, 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:26:59.442 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:26:59.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[12737]: cluster 2022-04-22T17:26:58.367913+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2022-04-22T17:26:59.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[12737]: audit 2022-04-22T17:26:58.679326+0000 mgr.x (mgr.24749) 43 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:59.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[12737]: audit 2022-04-22T17:26:59.006062+0000 mgr.x (mgr.24749) 44 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:59.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[18070]: cluster 2022-04-22T17:26:58.367913+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2022-04-22T17:26:59.673 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[18070]: audit 2022-04-22T17:26:58.679326+0000 mgr.x (mgr.24749) 43 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:59.673 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:26:59 smithi115 bash[18070]: audit 2022-04-22T17:26:59.006062+0000 mgr.x (mgr.24749) 44 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:59 smithi121 bash[12679]: cluster 2022-04-22T17:26:58.367913+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2022-04-22T17:26:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:59 smithi121 bash[12679]: audit 2022-04-22T17:26:58.679326+0000 mgr.x (mgr.24749) 43 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:26:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:26:59 smithi121 bash[12679]: audit 2022-04-22T17:26:59.006062+0000 mgr.x (mgr.24749) 44 : audit [DBG] from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:490: done (2 left) 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:491: copy_from oid 13 from oid 43 current snap is 41 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:487: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:487: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 179 (ObjNum 0 snap 32692 seq_num 3178927024) dirty exists 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:489: finishing write tid 1 to smithi11541303-4 2022-04-22T17:27:00.393 INFO:tasks.rados.rados.0.smithi115.stdout:489: finishing write tid 2 to smithi11541303-4 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:489: finishing write tid 3 to smithi11541303-4 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 264 (ObjNum 159 snap 41 seq_num 159) dirty exists 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:489: left oid 4 (ObjNum 159 snap 41 seq_num 159) 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:487: done (2 left) 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:489: done (1 left) 2022-04-22T17:27:00.394 INFO:tasks.rados.rados.0.smithi115.stdout:492: setattr oid 27 current snap is 41 2022-04-22T17:27:00.395 INFO:tasks.rados.rados.0.smithi115.stdout:493: rollback oid 3 current snap is 41 2022-04-22T17:27:00.395 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 36 2022-04-22T17:27:00.395 INFO:tasks.rados.rados.0.smithi115.stdout:494: setattr oid 40 current snap is 41 2022-04-22T17:27:00.398 INFO:tasks.rados.rados.0.smithi115.stdout:495: snap_create 2022-04-22T17:27:00.403 INFO:tasks.rados.rados.0.smithi115.stdout:491: finishing copy_from to smithi11541303-13 2022-04-22T17:27:00.403 INFO:tasks.rados.rados.0.smithi115.stdout:491: got expected ENOENT (src dne) 2022-04-22T17:27:00.405 INFO:tasks.rados.rados.0.smithi115.stdout:491: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:27:00.408 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 265 (ObjNum 103 snap 19 seq_num 103) dirty exists 2022-04-22T17:27:00.408 INFO:tasks.rados.rados.0.smithi115.stdout:493: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:27:00.409 INFO:tasks.rados.rados.0.smithi115.stdout:493: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:27:00.409 INFO:tasks.rados.rados.0.smithi115.stdout:493: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:27:00.409 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 152 (ObjNum 54 snap 4 seq_num 54) dirty exists 2022-04-22T17:27:00.409 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 100 (ObjNum 79 snap 14 seq_num 79) dirty exists 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[12737]: cluster 2022-04-22T17:26:59.210040+0000 mgr.x (mgr.24749) 45 : cluster [DBG] pgmap v62: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[12737]: cluster 2022-04-22T17:26:59.375719+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[12737]: audit 2022-04-22T17:26:59.432547+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.115:0/2758062997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[18070]: cluster 2022-04-22T17:26:59.210040+0000 mgr.x (mgr.24749) 45 : cluster [DBG] pgmap v62: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[18070]: cluster 2022-04-22T17:26:59.375719+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2022-04-22T17:27:00.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[18070]: audit 2022-04-22T17:26:59.432547+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.115:0/2758062997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:00 smithi121 bash[12679]: cluster 2022-04-22T17:26:59.210040+0000 mgr.x (mgr.24749) 45 : cluster [DBG] pgmap v62: 121 pgs: 121 active+clean; 114 MiB data, 345 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 584 KiB/s wr, 15 op/s 2022-04-22T17:27:00.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:00 smithi121 bash[12679]: cluster 2022-04-22T17:26:59.375719+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2022-04-22T17:27:00.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:00 smithi121 bash[12679]: audit 2022-04-22T17:26:59.432547+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.115:0/2758062997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:491: done (4 left) 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:492: done (3 left) 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:493: done (2 left) 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:494: done (1 left) 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:495: done (0 left) 2022-04-22T17:27:01.181 INFO:tasks.rados.rados.0.smithi115.stdout:496: snap_create 2022-04-22T17:27:01.201 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:00 smithi115 systemd[1]: Stopping Ceph mgr.y for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:27:01.201 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:00 smithi115 bash[42786]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr.y 2022-04-22T17:27:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:01 smithi115 bash[12737]: cluster 2022-04-22T17:27:00.384613+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2022-04-22T17:27:01.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:01 smithi115 bash[12737]: cluster 2022-04-22T17:27:01.172346+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2022-04-22T17:27:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:01 smithi115 bash[18070]: cluster 2022-04-22T17:27:00.384613+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2022-04-22T17:27:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:01 smithi115 bash[18070]: cluster 2022-04-22T17:27:01.172346+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2022-04-22T17:27:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:01 smithi121 bash[12679]: cluster 2022-04-22T17:27:00.384613+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2022-04-22T17:27:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:01 smithi121 bash[12679]: cluster 2022-04-22T17:27:01.172346+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2022-04-22T17:27:02.185 INFO:tasks.rados.rados.0.smithi115.stdout:496: done (0 left) 2022-04-22T17:27:02.185 INFO:tasks.rados.rados.0.smithi115.stdout:497: setattr oid 4 current snap is 43 2022-04-22T17:27:02.187 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 43 2022-04-22T17:27:02.188 INFO:tasks.rados.rados.0.smithi115.stdout:498: seq_num 160 ranges {0=286720} 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:498: writing smithi11541303-14 from 0 to 286720 tid 1 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:499: read oid 16 snap 42 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:499: expect deleted 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:500: rmattr oid 10 current snap is 43 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:500: done (3 left) 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:501: copy_from oid 41 from oid 17 current snap is 43 2022-04-22T17:27:02.191 INFO:tasks.rados.rados.0.smithi115.stdout:502: delete oid 32 current snap is 43 2022-04-22T17:27:02.195 INFO:tasks.rados.rados.0.smithi115.stdout:502: done (4 left) 2022-04-22T17:27:02.196 INFO:tasks.rados.rados.0.smithi115.stdout:503: rmattr oid 2 current snap is 43 2022-04-22T17:27:02.196 INFO:tasks.rados.rados.0.smithi115.stdout:503: done (4 left) 2022-04-22T17:27:02.196 INFO:tasks.rados.rados.0.smithi115.stdout:504: rollback oid 30 current snap is 43 2022-04-22T17:27:02.196 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 42 2022-04-22T17:27:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:505: copy_from oid 19 from oid 46 current snap is 43 2022-04-22T17:27:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:506: read oid 45 snap -1 2022-04-22T17:27:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:506: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:27:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:507: snap_remove snap 36 2022-04-22T17:27:02.378 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[42799]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr-y 2022-04-22T17:27:02.378 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2022-04-22T17:27:02.379 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[42842]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr.y 2022-04-22T17:27:02.379 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.y.service: Failed with result 'exit-code'. 2022-04-22T17:27:02.379 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 systemd[1]: Stopped Ceph mgr.y for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:27:02.729 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[12737]: cluster 2022-04-22T17:27:01.211008+0000 mgr.x (mgr.24749) 46 : cluster [DBG] pgmap v66: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.0 MiB/s wr, 45 op/s 2022-04-22T17:27:02.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[12737]: cluster 2022-04-22T17:27:02.176199+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2022-04-22T17:27:02.762 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:02 smithi115 systemd[1]: Started Ceph mgr.y for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:27:02.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[18070]: cluster 2022-04-22T17:27:01.211008+0000 mgr.x (mgr.24749) 46 : cluster [DBG] pgmap v66: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.0 MiB/s wr, 45 op/s 2022-04-22T17:27:02.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:02 smithi115 bash[18070]: cluster 2022-04-22T17:27:02.176199+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2022-04-22T17:27:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:02 smithi121 bash[12679]: cluster 2022-04-22T17:27:01.211008+0000 mgr.x (mgr.24749) 46 : cluster [DBG] pgmap v66: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.0 MiB/s wr, 45 op/s 2022-04-22T17:27:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:02 smithi121 bash[12679]: cluster 2022-04-22T17:27:02.176199+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2022-04-22T17:27:03.210 INFO:tasks.rados.rados.0.smithi115.stdout:507: done (7 left) 2022-04-22T17:27:03.210 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 43 2022-04-22T17:27:03.210 INFO:tasks.rados.rados.0.smithi115.stdout:508: seq_num 162 ranges {6414336=319488} 2022-04-22T17:27:03.247 INFO:tasks.rados.rados.0.smithi115.stdout:508: writing smithi11541303-33 from 6414336 to 6733824 tid 1 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:509: delete oid 8 current snap is 43 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:498: finishing write tid 1 to smithi11541303-14 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:498: finishing write tid 2 to smithi11541303-14 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:498: finishing write tid 3 to smithi11541303-14 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 87 (ObjNum 160 snap 43 seq_num 160) dirty exists 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:498: left oid 14 (ObjNum 160 snap 43 seq_num 160) 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:505: finishing copy_from to smithi11541303-19 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 106 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:03.248 INFO:tasks.rados.rados.0.smithi115.stdout:505: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:27:03.249 INFO:tasks.rados.rados.0.smithi115.stdout:506: expect (ObjNum 111 snap 26 seq_num 111) 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 266 (ObjNum 159 snap 41 seq_num 159) dirty exists 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:504: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:504: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:504: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 269 (ObjNum 157 snap 41 seq_num 157) dirty exists 2022-04-22T17:27:03.412 INFO:tasks.rados.rados.0.smithi115.stdout:501: finishing copy_from to smithi11541303-41 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 174 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:501: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:508: finishing write tid 1 to smithi11541303-33 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:508: finishing write tid 2 to smithi11541303-33 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:497: done (8 left) 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:498: done (7 left) 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:499: done (6 left) 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:501: done (5 left) 2022-04-22T17:27:03.413 INFO:tasks.rados.rados.0.smithi115.stdout:504: done (4 left) 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:505: done (3 left) 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:506: done (2 left) 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:509: done (1 left) 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:510: snap_create 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:508: finishing write tid 3 to smithi11541303-33 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 102 (ObjNum 162 snap 43 seq_num 162) dirty exists 2022-04-22T17:27:03.414 INFO:tasks.rados.rados.0.smithi115.stdout:508: left oid 33 (ObjNum 162 snap 43 seq_num 162) 2022-04-22T17:27:03.522 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[26675]: debug 2022-04-22T17:27:03.204+0000 7faa010ac700 -1 osd.2 pg_epoch: 149 pg[5.ds0( v 149'114 (0'0,149'114] local-lis/les=63/64 n=7 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 crt=149'114 lcod 149'112 mlcod 149'112 active+clean+snaptrim trimq=[49~1] ps=[4a~2,4d~1]] removing snap head 2022-04-22T17:27:03.522 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: audit 2022-04-22T17:27:02.414856+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.522 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: audit 2022-04-22T17:27:02.422554+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: audit 2022-04-22T17:27:02.424581+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: audit 2022-04-22T17:27:02.426009+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: audit 2022-04-22T17:27:02.739643+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[12737]: cluster 2022-04-22T17:27:03.196230+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: audit 2022-04-22T17:27:02.414856+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: audit 2022-04-22T17:27:02.422554+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:03.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: audit 2022-04-22T17:27:02.424581+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:03.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: audit 2022-04-22T17:27:02.426009+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:03.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: audit 2022-04-22T17:27:02.739643+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[18070]: cluster 2022-04-22T17:27:03.196230+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2022-04-22T17:27:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: audit 2022-04-22T17:27:02.414856+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: audit 2022-04-22T17:27:02.422554+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: audit 2022-04-22T17:27:02.424581+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: audit 2022-04-22T17:27:02.426009+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: audit 2022-04-22T17:27:02.739643+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:03 smithi121 bash[12679]: cluster 2022-04-22T17:27:03.196230+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2022-04-22T17:27:03.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:03.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:03.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:03.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:03.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:03.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:27:04.210 INFO:tasks.rados.rados.0.smithi115.stdout:508: done (1 left) 2022-04-22T17:27:04.210 INFO:tasks.rados.rados.0.smithi115.stdout:510: done (0 left) 2022-04-22T17:27:04.210 INFO:tasks.rados.rados.0.smithi115.stdout:511: snap_create 2022-04-22T17:27:04.238 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[42914]: debug 2022-04-22T17:27:04.055+0000 7f137b3db700 1 -- 172.21.15.115:0/3775936934 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b57e82a340 con 0x55b57e924400 2022-04-22T17:27:04.545 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[42914]: debug 2022-04-22T17:27:04.223+0000 7f1384055000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:27:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[18070]: cluster 2022-04-22T17:27:03.211641+0000 mgr.x (mgr.24749) 47 : cluster [DBG] pgmap v69: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2022-04-22T17:27:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[18070]: cluster 2022-04-22T17:27:04.201655+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2022-04-22T17:27:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[12737]: cluster 2022-04-22T17:27:03.211641+0000 mgr.x (mgr.24749) 47 : cluster [DBG] pgmap v69: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2022-04-22T17:27:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[12737]: cluster 2022-04-22T17:27:04.201655+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2022-04-22T17:27:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:04 smithi121 bash[12679]: cluster 2022-04-22T17:27:03.211641+0000 mgr.x (mgr.24749) 47 : cluster [DBG] pgmap v69: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2022-04-22T17:27:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:04 smithi121 bash[12679]: cluster 2022-04-22T17:27:04.201655+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2022-04-22T17:27:05.214 INFO:tasks.rados.rados.0.smithi115.stdout:511: done (0 left) 2022-04-22T17:27:05.214 INFO:tasks.rados.rados.0.smithi115.stdout:512: snap_remove snap 28 2022-04-22T17:27:05.295 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:04 smithi115 bash[42914]: debug 2022-04-22T17:27:04.971+0000 7f1384055000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:27:05.586 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:05 smithi115 bash[42914]: debug 2022-04-22T17:27:05.307+0000 7f1384055000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:27:05.586 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:05 smithi115 bash[42914]: debug 2022-04-22T17:27:05.455+0000 7f1384055000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:27:05.863 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:05 smithi115 bash[42914]: debug 2022-04-22T17:27:05.571+0000 7f1384055000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:27:06.177 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:05 smithi115 bash[42914]: debug 2022-04-22T17:27:05.847+0000 7f1384055000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:27:06.177 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:05 smithi115 bash[42914]: debug 2022-04-22T17:27:05.959+0000 7f1384055000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:27:06.177 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[42914]: debug 2022-04-22T17:27:06.063+0000 7f1384055000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:27:06.222 INFO:tasks.rados.rados.0.smithi115.stdout:512: done (0 left) 2022-04-22T17:27:06.222 INFO:tasks.rados.rados.0.smithi115.stdout:513: read oid 21 snap -1 2022-04-22T17:27:06.222 INFO:tasks.rados.rados.0.smithi115.stdout:513: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:06.222 INFO:tasks.rados.rados.0.smithi115.stdout:514: snap_remove snap 39 2022-04-22T17:27:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:06 smithi121 bash[12679]: cluster 2022-04-22T17:27:05.206045+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2022-04-22T17:27:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:06 smithi121 bash[12679]: cluster 2022-04-22T17:27:05.212470+0000 mgr.x (mgr.24749) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:06.517 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:27:06 smithi121 bash[21417]: debug 2022-04-22T17:27:06.228+0000 7fcbfaaea700 -1 osd.6 pg_epoch: 152 pg[5.2s0( v 149'189 (0'0,149'189] local-lis/les=63/64 n=9 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [6,0,5]p6(0) r=0 lpr=63 crt=149'189 lcod 149'187 mlcod 149'187 active+clean+snaptrim trimq=[34~1] ps=[49~3,4d~1]] removing snap head 2022-04-22T17:27:06.517 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:27:06 smithi121 bash[15740]: debug 2022-04-22T17:27:06.216+0000 7feeefc08700 -1 osd.4 pg_epoch: 152 pg[5.1s0( v 149'154 (0'0,149'154] local-lis/les=63/64 n=7 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [4,3,7]p4(0) r=0 lpr=63 crt=149'154 lcod 146'152 mlcod 146'152 active+clean+snaptrim trimq=[34~1] ps=[49~3,4d~1]] removing snap head 2022-04-22T17:27:06.545 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[42914]: debug 2022-04-22T17:27:06.163+0000 7f1384055000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:27:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[12737]: cluster 2022-04-22T17:27:05.206045+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2022-04-22T17:27:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[12737]: cluster 2022-04-22T17:27:05.212470+0000 mgr.x (mgr.24749) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[18070]: cluster 2022-04-22T17:27:05.206045+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2022-04-22T17:27:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[18070]: cluster 2022-04-22T17:27:05.212470+0000 mgr.x (mgr.24749) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 118 MiB data, 364 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:06.908 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[42914]: debug 2022-04-22T17:27:06.639+0000 7f1384055000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:27:06.908 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[42914]: debug 2022-04-22T17:27:06.779+0000 7f1384055000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:07.236 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:06 smithi115 bash[42914]: debug 2022-04-22T17:27:06.895+0000 7f1384055000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:27:07.255 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[42914]: debug 2022-04-22T17:27:07.111+0000 7f1384055000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:27:07.256 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[20761]: debug 2022-04-22T17:27:07.211+0000 7fabf911e700 -1 osd.0 pg_epoch: 153 pg[5.3s0( v 152'134 (0'0,152'134] local-lis/les=63/64 n=5 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [0,3,6]p0(0) r=0 lpr=63 crt=152'134 lcod 152'132 mlcod 152'132 active+clean+snaptrim trimq=[4c~1] ps=[34~1,49~3,4d~1]] removing snap head 2022-04-22T17:27:07.256 INFO:tasks.rados.rados.0.smithi115.stdout:514: done (1 left) 2022-04-22T17:27:07.256 INFO:tasks.rados.rados.0.smithi115.stdout:515: snap_remove snap 43 2022-04-22T17:27:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:07 smithi121 bash[12679]: cluster 2022-04-22T17:27:06.213343+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2022-04-22T17:27:07.517 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:27:07 smithi121 bash[15740]: debug 2022-04-22T17:27:07.216+0000 7feef140b700 -1 osd.4 pg_epoch: 153 pg[5.es0( v 152'106 (0'0,152'106] local-lis/les=63/64 n=5 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [4,5,0]p4(0) r=0 lpr=63 crt=152'106 lcod 152'104 mlcod 152'104 active+clean+snaptrim trimq=[4c~1] ps=[34~1,49~3,4d~1]] removing snap head 2022-04-22T17:27:07.540 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[42914]: debug 2022-04-22T17:27:07.239+0000 7f1384055000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:27:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[12737]: cluster 2022-04-22T17:27:06.213343+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2022-04-22T17:27:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[18070]: cluster 2022-04-22T17:27:06.213343+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2022-04-22T17:27:07.541 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[26675]: debug 2022-04-22T17:27:07.223+0000 7faa010ac700 -1 osd.2 pg_epoch: 153 pg[5.ds0( v 153'120 (0'0,153'120] local-lis/les=63/64 n=4 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 luod=152'118 crt=152'118 lcod 149'116 mlcod 149'116 active+clean+snaptrim trimq=[4c~1] ps=[34~1,49~3,4d~1]] removing snap head 2022-04-22T17:27:07.796 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[42914]: debug 2022-04-22T17:27:07.527+0000 7f1384055000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:27:07.796 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[42914]: debug 2022-04-22T17:27:07.711+0000 7f1384055000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:27:08.230 INFO:tasks.rados.rados.0.smithi115.stdout:515: done (1 left) 2022-04-22T17:27:08.230 INFO:tasks.rados.rados.0.smithi115.stdout:516: copy_from oid 40 from oid 22 current snap is 45 2022-04-22T17:27:08.230 INFO:tasks.rados.rados.0.smithi115.stdout:513: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:08.251 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:07 smithi115 bash[42914]: debug 2022-04-22T17:27:07.983+0000 7f1384055000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:27:08.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[12737]: cluster 2022-04-22T17:27:07.213012+0000 mgr.x (mgr.24749) 49 : cluster [DBG] pgmap v74: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2022-04-22T17:27:08.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[12737]: cluster 2022-04-22T17:27:07.214923+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2022-04-22T17:27:08.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[12737]: audit 2022-04-22T17:27:07.677776+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:08.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[12737]: cluster 2022-04-22T17:27:08.221334+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2022-04-22T17:27:08.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[18070]: cluster 2022-04-22T17:27:07.213012+0000 mgr.x (mgr.24749) 49 : cluster [DBG] pgmap v74: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2022-04-22T17:27:08.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[18070]: cluster 2022-04-22T17:27:07.214923+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2022-04-22T17:27:08.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[18070]: audit 2022-04-22T17:27:07.677776+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:08.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[18070]: cluster 2022-04-22T17:27:08.221334+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2022-04-22T17:27:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:08 smithi121 bash[12679]: cluster 2022-04-22T17:27:07.213012+0000 mgr.x (mgr.24749) 49 : cluster [DBG] pgmap v74: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2022-04-22T17:27:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:08 smithi121 bash[12679]: cluster 2022-04-22T17:27:07.214923+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2022-04-22T17:27:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:08 smithi121 bash[12679]: audit 2022-04-22T17:27:07.677776+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:08 smithi121 bash[12679]: cluster 2022-04-22T17:27:08.221334+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2022-04-22T17:27:08.529 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[42914]: debug 2022-04-22T17:27:08.235+0000 7f1384055000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:27:08.529 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[42914]: debug 2022-04-22T17:27:08.483+0000 7f1384055000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:08.730 INFO:tasks.rados.rados.0.smithi115.stdout:516: finishing copy_from to smithi11541303-40 2022-04-22T17:27:08.730 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:08.730 INFO:tasks.rados.rados.0.smithi115.stdout:516: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:27:08.730 INFO:tasks.rados.rados.0.smithi115.stdout:513: done (1 left) 2022-04-22T17:27:08.731 INFO:tasks.rados.rados.0.smithi115.stdout:516: done (0 left) 2022-04-22T17:27:08.731 INFO:tasks.rados.rados.0.smithi115.stdout:517: rollback oid 48 current snap is 45 2022-04-22T17:27:08.731 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 48 to 44 2022-04-22T17:27:08.731 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 45 2022-04-22T17:27:08.732 INFO:tasks.rados.rados.0.smithi115.stdout:518: seq_num 164 ranges {2203648=352256} 2022-04-22T17:27:08.752 INFO:tasks.rados.rados.0.smithi115.stdout:518: writing smithi11541303-20 from 2203648 to 2555904 tid 1 2022-04-22T17:27:08.753 INFO:tasks.rados.rados.0.smithi115.stdout:519: snap_create 2022-04-22T17:27:08.753 INFO:tasks.rados.rados.0.smithi115.stdout:517: finishing rollback tid 0 to smithi11541303-48 2022-04-22T17:27:08.753 INFO:tasks.rados.rados.0.smithi115.stdout:517: finishing rollback tid 1 to smithi11541303-48 2022-04-22T17:27:08.753 INFO:tasks.rados.rados.0.smithi115.stdout:517: finishing rollback tid 2 to smithi11541303-48 2022-04-22T17:27:08.753 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 272 (ObjNum 150 snap 40 seq_num 150) dirty exists 2022-04-22T17:27:08.759 INFO:tasks.rados.rados.0.smithi115.stdout:518: finishing write tid 1 to smithi11541303-20 2022-04-22T17:27:08.760 INFO:tasks.rados.rados.0.smithi115.stdout:518: finishing write tid 2 to smithi11541303-20 2022-04-22T17:27:08.763 INFO:tasks.rados.rados.0.smithi115.stdout:518: finishing write tid 3 to smithi11541303-20 2022-04-22T17:27:08.763 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 200 (ObjNum 164 snap 45 seq_num 164) dirty exists 2022-04-22T17:27:08.763 INFO:tasks.rados.rados.0.smithi115.stdout:518: left oid 20 (ObjNum 164 snap 45 seq_num 164) 2022-04-22T17:27:08.795 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:08 smithi115 bash[42914]: debug 2022-04-22T17:27:08.747+0000 7f1384055000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:27:09.106 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:09 smithi115 bash[42914]: debug 2022-04-22T17:27:09.007+0000 7f1384055000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:09.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:09 smithi115 bash[12737]: audit 2022-04-22T17:27:08.509369+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:09.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:09 smithi115 bash[12737]: audit 2022-04-22T17:27:09.116234+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:09.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:09 smithi115 bash[18070]: audit 2022-04-22T17:27:08.509369+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:09.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:09 smithi115 bash[18070]: audit 2022-04-22T17:27:09.116234+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:09.527 INFO:tasks.rados.rados.0.smithi115.stdout:517: done (2 left) 2022-04-22T17:27:09.527 INFO:tasks.rados.rados.0.smithi115.stdout:518: done (1 left) 2022-04-22T17:27:09.527 INFO:tasks.rados.rados.0.smithi115.stdout:519: done (0 left) 2022-04-22T17:27:09.527 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 46 2022-04-22T17:27:09.527 INFO:tasks.rados.rados.0.smithi115.stdout:520: seq_num 165 ranges {4726784=368640} 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:520: writing smithi11541303-17 from 4726784 to 5095424 tid 1 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:521: copy_from oid 28 from oid 42 current snap is 46 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:522: rollback oid 15 current snap is 46 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 44 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:523: rollback oid 33 current snap is 46 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 45 2022-04-22T17:27:09.559 INFO:tasks.rados.rados.0.smithi115.stdout:524: delete oid 46 current snap is 46 2022-04-22T17:27:09.567 INFO:tasks.rados.rados.0.smithi115.stdout:520: finishing write tid 1 to smithi11541303-17 2022-04-22T17:27:09.567 INFO:tasks.rados.rados.0.smithi115.stdout:522: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:27:09.567 INFO:tasks.rados.rados.0.smithi115.stdout:520: finishing write tid 2 to smithi11541303-17 2022-04-22T17:27:09.567 INFO:tasks.rados.rados.0.smithi115.stdout:522: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:27:09.568 INFO:tasks.rados.rados.0.smithi115.stdout:522: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:27:09.568 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 60 (ObjNum 155 snap 41 seq_num 155) dirty exists 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:522: done (4 left) 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:524: done (3 left) 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:525: read oid 1 snap 41 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:525: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:526: read oid 12 snap -1 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:526: expect (ObjNum 140 snap 36 seq_num 140) 2022-04-22T17:27:09.570 INFO:tasks.rados.rados.0.smithi115.stdout:527: rmattr oid 25 current snap is 46 2022-04-22T17:27:09.571 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 46 2022-04-22T17:27:09.571 INFO:tasks.rados.rados.0.smithi115.stdout:528: seq_num 167 ranges {114688=401408} 2022-04-22T17:27:09.576 INFO:tasks.rados.rados.0.smithi115.stdout:528: writing smithi11541303-31 from 114688 to 516096 tid 1 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:529: read oid 7 snap -1 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:529: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 46 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:523: finishing rollback tid 0 to smithi11541303-33 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:520: finishing write tid 3 to smithi11541303-17 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 203 (ObjNum 165 snap 46 seq_num 165) dirty exists 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:520: left oid 17 (ObjNum 165 snap 46 seq_num 165) 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 205 (ObjNum 0 snap 0 seq_num 3178813160) dirty exists 2022-04-22T17:27:09.577 INFO:tasks.rados.rados.0.smithi115.stdout:525: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:27:09.582 INFO:tasks.rados.rados.0.smithi115.stdout:523: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:27:09.582 INFO:tasks.rados.rados.0.smithi115.stdout:523: finishing rollback tid 2 to smithi11541303-33 2022-04-22T17:27:09.583 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 114 (ObjNum 162 snap 43 seq_num 162) dirty exists 2022-04-22T17:27:09.583 INFO:tasks.rados.rados.0.smithi115.stdout:528: finishing write tid 1 to smithi11541303-31 2022-04-22T17:27:09.584 INFO:tasks.rados.rados.0.smithi115.stdout:528: finishing write tid 2 to smithi11541303-31 2022-04-22T17:27:09.584 INFO:tasks.rados.rados.0.smithi115.stdout:530: seq_num 168 ranges {0=417792} 2022-04-22T17:27:09.588 INFO:tasks.rados.rados.0.smithi115.stdout:530: writing smithi11541303-46 from 0 to 417792 tid 1 2022-04-22T17:27:09.588 INFO:tasks.rados.rados.0.smithi115.stdout:520: done (8 left) 2022-04-22T17:27:09.588 INFO:tasks.rados.rados.0.smithi115.stdout:523: done (7 left) 2022-04-22T17:27:09.588 INFO:tasks.rados.rados.0.smithi115.stdout:525: done (6 left) 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:527: done (5 left) 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 46 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:528: finishing write tid 3 to smithi11541303-31 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 275 (ObjNum 167 snap 46 seq_num 167) dirty exists 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:528: left oid 31 (ObjNum 167 snap 46 seq_num 167) 2022-04-22T17:27:09.589 INFO:tasks.rados.rados.0.smithi115.stdout:531: seq_num 169 ranges {270336=434176} 2022-04-22T17:27:09.596 INFO:tasks.rados.rados.0.smithi115.stdout:531: writing smithi11541303-4 from 270336 to 704512 tid 1 2022-04-22T17:27:09.597 INFO:tasks.rados.rados.0.smithi115.stdout:528: done (5 left) 2022-04-22T17:27:09.597 INFO:tasks.rados.rados.0.smithi115.stdout:532: snap_remove snap 44 2022-04-22T17:27:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:09 smithi121 bash[12679]: audit 2022-04-22T17:27:08.509369+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:10.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:09 smithi121 bash[12679]: audit 2022-04-22T17:27:09.116234+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:10.535 INFO:tasks.rados.rados.0.smithi115.stdout:521: finishing copy_from to smithi11541303-28 2022-04-22T17:27:10.536 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 120 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:10.536 INFO:tasks.rados.rados.0.smithi115.stdout:521: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:27:10.536 INFO:tasks.rados.rados.0.smithi115.stdout:526: expect (ObjNum 140 snap 36 seq_num 140) 2022-04-22T17:27:10.608 INFO:tasks.rados.rados.0.smithi115.stdout:529: expect (ObjNum 6 snap 0 seq_num 6) 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:530: finishing write tid 1 to smithi11541303-46 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:530: finishing write tid 2 to smithi11541303-46 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:530: finishing write tid 3 to smithi11541303-46 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 166 (ObjNum 168 snap 46 seq_num 168) dirty exists 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:530: left oid 46 (ObjNum 168 snap 46 seq_num 168) 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:531: finishing write tid 1 to smithi11541303-4 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:531: finishing write tid 2 to smithi11541303-4 2022-04-22T17:27:10.611 INFO:tasks.rados.rados.0.smithi115.stdout:531: finishing write tid 3 to smithi11541303-4 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 274 (ObjNum 169 snap 46 seq_num 169) dirty exists 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:531: left oid 4 (ObjNum 169 snap 46 seq_num 169) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:521: done (5 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:526: done (4 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:529: done (3 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:530: done (2 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:531: done (1 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:532: done (0 left) 2022-04-22T17:27:10.612 INFO:tasks.rados.rados.0.smithi115.stdout:533: snap_create 2022-04-22T17:27:10.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:10 smithi115 bash[12737]: cluster 2022-04-22T17:27:09.213704+0000 mgr.x (mgr.24749) 50 : cluster [DBG] pgmap v77: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:27:10.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:10 smithi115 bash[12737]: cluster 2022-04-22T17:27:09.516568+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2022-04-22T17:27:10.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:10 smithi115 bash[18070]: cluster 2022-04-22T17:27:09.213704+0000 mgr.x (mgr.24749) 50 : cluster [DBG] pgmap v77: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:27:10.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:10 smithi115 bash[18070]: cluster 2022-04-22T17:27:09.516568+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2022-04-22T17:27:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:10 smithi121 bash[12679]: cluster 2022-04-22T17:27:09.213704+0000 mgr.x (mgr.24749) 50 : cluster [DBG] pgmap v77: 121 pgs: 121 active+clean; 121 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2022-04-22T17:27:11.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:10 smithi121 bash[12679]: cluster 2022-04-22T17:27:09.516568+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2022-04-22T17:27:11.188 INFO:tasks.rados.rados.0.smithi115.stdout:533: done (0 left) 2022-04-22T17:27:11.188 INFO:tasks.rados.rados.0.smithi115.stdout:534: snap_remove snap 41 2022-04-22T17:27:11.295 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:10 smithi115 bash[42914]: debug 2022-04-22T17:27:10.935+0000 7f1384055000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:27:11.296 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:11 smithi115 bash[42914]: debug 2022-04-22T17:27:11.055+0000 7f1384055000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:11 smithi115 bash[12737]: cluster 2022-04-22T17:27:10.527043+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2022-04-22T17:27:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:11 smithi115 bash[12737]: cluster 2022-04-22T17:27:11.179311+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2022-04-22T17:27:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:11 smithi115 bash[18070]: cluster 2022-04-22T17:27:10.527043+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2022-04-22T17:27:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:11 smithi115 bash[18070]: cluster 2022-04-22T17:27:11.179311+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2022-04-22T17:27:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:11 smithi121 bash[12679]: cluster 2022-04-22T17:27:10.527043+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2022-04-22T17:27:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:11 smithi121 bash[12679]: cluster 2022-04-22T17:27:11.179311+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2022-04-22T17:27:12.192 INFO:tasks.rados.rados.0.smithi115.stdout:534: done (0 left) 2022-04-22T17:27:12.192 INFO:tasks.rados.rados.0.smithi115.stdout:535: read oid 16 snap 46 2022-04-22T17:27:12.193 INFO:tasks.rados.rados.0.smithi115.stdout:535: expect deleted 2022-04-22T17:27:12.196 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 47 2022-04-22T17:27:12.196 INFO:tasks.rados.rados.0.smithi115.stdout:536: seq_num 170 ranges {0=450560} 2022-04-22T17:27:12.196 INFO:tasks.rados.rados.0.smithi115.stdout:536: writing smithi11541303-2 from 0 to 450560 tid 1 2022-04-22T17:27:12.197 INFO:tasks.rados.rados.0.smithi115.stdout:537: setattr oid 49 current snap is 47 2022-04-22T17:27:12.198 INFO:tasks.rados.rados.0.smithi115.stdout:535: done (2 left) 2022-04-22T17:27:12.198 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 47 2022-04-22T17:27:12.199 INFO:tasks.rados.rados.0.smithi115.stdout:538: seq_num 171 ranges {6733824=466944} 2022-04-22T17:27:12.240 INFO:tasks.rados.rados.0.smithi115.stdout:538: writing smithi11541303-33 from 6733824 to 7200768 tid 1 2022-04-22T17:27:12.240 INFO:tasks.rados.rados.0.smithi115.stdout:539: read oid 29 snap -1 2022-04-22T17:27:12.240 INFO:tasks.rados.rados.0.smithi115.stdout:539: expect (ObjNum 156 snap 41 seq_num 156) 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:540: read oid 5 snap -1 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:540: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:541: copy_from oid 1 from oid 7 current snap is 47 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 280 (ObjNum 1952209264 snap 1633448270 seq_num 1717333595) dirty exists 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:537: done (5 left) 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:542: rollback oid 36 current snap is 47 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 42 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:536: finishing write tid 1 to smithi11541303-2 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:536: finishing write tid 2 to smithi11541303-2 2022-04-22T17:27:12.241 INFO:tasks.rados.rados.0.smithi115.stdout:536: finishing write tid 3 to smithi11541303-2 2022-04-22T17:27:12.242 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 124 (ObjNum 170 snap 47 seq_num 170) dirty exists 2022-04-22T17:27:12.242 INFO:tasks.rados.rados.0.smithi115.stdout:536: left oid 2 (ObjNum 170 snap 47 seq_num 170) 2022-04-22T17:27:12.242 INFO:tasks.rados.rados.0.smithi115.stdout:536: done (5 left) 2022-04-22T17:27:12.242 INFO:tasks.rados.rados.0.smithi115.stdout:543: snap_remove snap 46 2022-04-22T17:27:12.796 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[42914]: debug 2022-04-22T17:27:12.467+0000 7f1384055000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:27:12.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: cluster 2022-04-22T17:27:11.214573+0000 mgr.x (mgr.24749) 51 : cluster [DBG] pgmap v81: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:27:12.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: cluster 2022-04-22T17:27:12.184078+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: cluster 2022-04-22T17:27:12.493957+0000 mon.a (mon.0) 912 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: cluster 2022-04-22T17:27:12.494163+0000 mon.a (mon.0) 913 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: audit 2022-04-22T17:27:12.496952+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: audit 2022-04-22T17:27:12.498158+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: audit 2022-04-22T17:27:12.499275+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[12737]: audit 2022-04-22T17:27:12.499644+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: cluster 2022-04-22T17:27:11.214573+0000 mgr.x (mgr.24749) 51 : cluster [DBG] pgmap v81: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:27:12.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: cluster 2022-04-22T17:27:12.184078+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2022-04-22T17:27:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: cluster 2022-04-22T17:27:12.493957+0000 mon.a (mon.0) 912 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:27:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: cluster 2022-04-22T17:27:12.494163+0000 mon.a (mon.0) 913 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:27:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: audit 2022-04-22T17:27:12.496952+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:27:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: audit 2022-04-22T17:27:12.498158+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: audit 2022-04-22T17:27:12.499275+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:27:12.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:12 smithi115 bash[18070]: audit 2022-04-22T17:27:12.499644+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: cluster 2022-04-22T17:27:11.214573+0000 mgr.x (mgr.24749) 51 : cluster [DBG] pgmap v81: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: cluster 2022-04-22T17:27:12.184078+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: cluster 2022-04-22T17:27:12.493957+0000 mon.a (mon.0) 912 : cluster [DBG] Standby manager daemon y restarted 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: cluster 2022-04-22T17:27:12.494163+0000 mon.a (mon.0) 913 : cluster [DBG] Standby manager daemon y started 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: audit 2022-04-22T17:27:12.496952+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: audit 2022-04-22T17:27:12.498158+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: audit 2022-04-22T17:27:12.499275+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-04-22T17:27:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:12 smithi121 bash[12679]: audit 2022-04-22T17:27:12.499644+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:543: done (5 left) 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:544: read oid 30 snap -1 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:544: expect (ObjNum 157 snap 41 seq_num 157) 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:545: rollback oid 11 current snap is 47 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 42 2022-04-22T17:27:13.219 INFO:tasks.rados.rados.0.smithi115.stdout:539: expect (ObjNum 156 snap 41 seq_num 156) 2022-04-22T17:27:13.228 INFO:tasks.rados.rados.0.smithi115.stdout:542: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:27:13.228 INFO:tasks.rados.rados.0.smithi115.stdout:542: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:542: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 99 (ObjNum 105 snap 20 seq_num 105) dirty exists 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:541: finishing copy_from to smithi11541303-1 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 79 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:541: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:538: finishing write tid 1 to smithi11541303-33 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:538: finishing write tid 2 to smithi11541303-33 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:538: finishing write tid 3 to smithi11541303-33 2022-04-22T17:27:13.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 121 (ObjNum 171 snap 47 seq_num 171) dirty exists 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:538: left oid 33 (ObjNum 171 snap 47 seq_num 171) 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:538: done (6 left) 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:539: done (5 left) 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:541: done (4 left) 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:542: done (3 left) 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:546: delete oid 4 current snap is 47 2022-04-22T17:27:13.230 INFO:tasks.rados.rados.0.smithi115.stdout:540: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:27:13.416 INFO:tasks.rados.rados.0.smithi115.stdout:545: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:27:13.416 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 0 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty dne 2022-04-22T17:27:13.416 INFO:tasks.rados.rados.0.smithi115.stdout:544: expect (ObjNum 157 snap 41 seq_num 157) 2022-04-22T17:27:13.488 INFO:tasks.rados.rados.0.smithi115.stdout:540: done (3 left) 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:544: done (2 left) 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:545: done (1 left) 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:546: done (0 left) 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:547: rmattr oid 4 current snap is 47 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:547: done (0 left) 2022-04-22T17:27:13.489 INFO:tasks.rados.rados.0.smithi115.stdout:548: snap_create 2022-04-22T17:27:13.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:13.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:27:13.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:13.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:13.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:13.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:27:14.227 INFO:tasks.rados.rados.0.smithi115.stdout:548: done (0 left) 2022-04-22T17:27:14.227 INFO:tasks.rados.rados.0.smithi115.stdout:549: delete oid 37 current snap is 48 2022-04-22T17:27:14.234 INFO:tasks.rados.rados.0.smithi115.stdout:549: done (0 left) 2022-04-22T17:27:14.234 INFO:tasks.rados.rados.0.smithi115.stdout:550: read oid 28 snap 47 2022-04-22T17:27:14.234 INFO:tasks.rados.rados.0.smithi115.stdout:550: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:14.234 INFO:tasks.rados.rados.0.smithi115.stdout:551: read oid 6 snap -1 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:551: expect deleted 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:552: read oid 24 snap 42 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:552: expect (ObjNum 131 snap 34 seq_num 131) 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:553: rollback oid 12 current snap is 48 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 47 2022-04-22T17:27:14.235 INFO:tasks.rados.rados.0.smithi115.stdout:554: snap_remove snap 45 2022-04-22T17:27:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:14 smithi121 bash[12679]: cluster 2022-04-22T17:27:13.201003+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e20: x(active, since 54s), standbys: y 2022-04-22T17:27:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:14 smithi121 bash[12679]: cluster 2022-04-22T17:27:13.209437+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2022-04-22T17:27:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:14 smithi121 bash[12679]: cluster 2022-04-22T17:27:13.215089+0000 mgr.x (mgr.24749) 52 : cluster [DBG] pgmap v84: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:27:14.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[12737]: cluster 2022-04-22T17:27:13.201003+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e20: x(active, since 54s), standbys: y 2022-04-22T17:27:14.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[12737]: cluster 2022-04-22T17:27:13.209437+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2022-04-22T17:27:14.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[12737]: cluster 2022-04-22T17:27:13.215089+0000 mgr.x (mgr.24749) 52 : cluster [DBG] pgmap v84: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:27:14.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[18070]: cluster 2022-04-22T17:27:13.201003+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e20: x(active, since 54s), standbys: y 2022-04-22T17:27:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[18070]: cluster 2022-04-22T17:27:13.209437+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2022-04-22T17:27:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:14 smithi115 bash[18070]: cluster 2022-04-22T17:27:13.215089+0000 mgr.x (mgr.24749) 52 : cluster [DBG] pgmap v84: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail; 7.5 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:27:15.240 INFO:tasks.rados.rados.0.smithi115.stdout:554: done (4 left) 2022-04-22T17:27:15.240 INFO:tasks.rados.rados.0.smithi115.stdout:555: rollback oid 47 current snap is 48 2022-04-22T17:27:15.240 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 47 to 42 2022-04-22T17:27:15.241 INFO:tasks.rados.rados.0.smithi115.stdout:556: copy_from oid 36 from oid 25 current snap is 48 2022-04-22T17:27:15.241 INFO:tasks.rados.rados.0.smithi115.stdout:557: rollback oid 16 current snap is 48 2022-04-22T17:27:15.241 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 42 2022-04-22T17:27:15.241 INFO:tasks.rados.rados.0.smithi115.stdout:558: snap_create 2022-04-22T17:27:15.241 INFO:tasks.rados.rados.0.smithi115.stdout:550: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:15.263 INFO:tasks.rados.rados.0.smithi115.stdout:553: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:27:15.264 INFO:tasks.rados.rados.0.smithi115.stdout:553: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:27:15.264 INFO:tasks.rados.rados.0.smithi115.stdout:553: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:27:15.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 138 (ObjNum 140 snap 36 seq_num 140) dirty exists 2022-04-22T17:27:15.264 INFO:tasks.rados.rados.0.smithi115.stdout:552: expect (ObjNum 131 snap 34 seq_num 131) 2022-04-22T17:27:15.386 INFO:tasks.rados.rados.0.smithi115.stdout:555: finishing rollback tid 0 to smithi11541303-47 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:555: finishing rollback tid 1 to smithi11541303-47 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:557: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:555: finishing rollback tid 2 to smithi11541303-47 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 177 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:556: finishing copy_from to smithi11541303-36 2022-04-22T17:27:15.387 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 205 (ObjNum 1 snap 32692 seq_num 3178537408) dirty exists 2022-04-22T17:27:15.388 INFO:tasks.rados.rados.0.smithi115.stdout:556: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:27:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:15 smithi121 bash[12679]: cluster 2022-04-22T17:27:14.218412+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2022-04-22T17:27:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:15 smithi115 bash[12737]: cluster 2022-04-22T17:27:14.218412+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2022-04-22T17:27:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:15 smithi115 bash[18070]: cluster 2022-04-22T17:27:14.218412+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2022-04-22T17:27:16.188 INFO:tasks.rados.rados.0.smithi115.stdout:550: done (7 left) 2022-04-22T17:27:16.188 INFO:tasks.rados.rados.0.smithi115.stdout:551: done (6 left) 2022-04-22T17:27:16.188 INFO:tasks.rados.rados.0.smithi115.stdout:552: done (5 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:553: done (4 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:555: done (3 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:556: done (2 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:557: done (1 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:558: done (0 left) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:559: read oid 47 snap -1 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:559: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:27:16.189 INFO:tasks.rados.rados.0.smithi115.stdout:560: delete oid 33 current snap is 49 2022-04-22T17:27:16.198 INFO:tasks.rados.rados.0.smithi115.stdout:559: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:27:16.211 INFO:tasks.rados.rados.0.smithi115.stdout:559: done (1 left) 2022-04-22T17:27:16.211 INFO:tasks.rados.rados.0.smithi115.stdout:560: done (0 left) 2022-04-22T17:27:16.212 INFO:tasks.rados.rados.0.smithi115.stdout:561: rollback oid 46 current snap is 49 2022-04-22T17:27:16.212 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 42 2022-04-22T17:27:16.212 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 49 2022-04-22T17:27:16.212 INFO:tasks.rados.rados.0.smithi115.stdout:562: seq_num 175 ranges {0=540672} 2022-04-22T17:27:16.216 INFO:tasks.rados.rados.0.smithi115.stdout:562: writing smithi11541303-49 from 0 to 540672 tid 1 2022-04-22T17:27:16.216 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 49 2022-04-22T17:27:16.217 INFO:tasks.rados.rados.0.smithi115.stdout:563: seq_num 176 ranges {1884160=557056} 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:563: writing smithi11541303-10 from 1884160 to 2441216 tid 1 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:564: rollback oid 39 current snap is 49 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 42 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:561: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:561: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:561: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 176 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:16.234 INFO:tasks.rados.rados.0.smithi115.stdout:562: finishing write tid 1 to smithi11541303-49 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:562: finishing write tid 2 to smithi11541303-49 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:562: finishing write tid 3 to smithi11541303-49 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 285 (ObjNum 175 snap 49 seq_num 175) dirty exists 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:562: left oid 49 (ObjNum 175 snap 49 seq_num 175) 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:561: done (3 left) 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:562: done (2 left) 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:565: read oid 38 snap -1 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:565: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 49 2022-04-22T17:27:16.235 INFO:tasks.rados.rados.0.smithi115.stdout:566: seq_num 177 ranges {0=573440} 2022-04-22T17:27:16.237 INFO:tasks.rados.rados.0.smithi115.stdout:566: writing smithi11541303-32 from 0 to 573440 tid 1 2022-04-22T17:27:16.237 INFO:tasks.rados.rados.0.smithi115.stdout:567: snap_create 2022-04-22T17:27:16.237 INFO:tasks.rados.rados.0.smithi115.stdout:564: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:27:16.237 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 0 (ObjNum -1116387936 snap 22075 seq_num 3179018784) dirty dne 2022-04-22T17:27:16.240 INFO:tasks.rados.rados.0.smithi115.stdout:563: finishing write tid 1 to smithi11541303-10 2022-04-22T17:27:16.240 INFO:tasks.rados.rados.0.smithi115.stdout:565: expect (ObjNum 37 snap 0 seq_num 37) 2022-04-22T17:27:16.258 INFO:tasks.rados.rados.0.smithi115.stdout:563: finishing write tid 2 to smithi11541303-10 2022-04-22T17:27:16.258 INFO:tasks.rados.rados.0.smithi115.stdout:563: finishing write tid 3 to smithi11541303-10 2022-04-22T17:27:16.258 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 179 (ObjNum 176 snap 49 seq_num 176) dirty exists 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:563: left oid 10 (ObjNum 176 snap 49 seq_num 176) 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:566: finishing write tid 1 to smithi11541303-32 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:566: finishing write tid 2 to smithi11541303-32 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:566: finishing write tid 3 to smithi11541303-32 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 129 (ObjNum 177 snap 49 seq_num 177) dirty exists 2022-04-22T17:27:16.259 INFO:tasks.rados.rados.0.smithi115.stdout:566: left oid 32 (ObjNum 177 snap 49 seq_num 177) 2022-04-22T17:27:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:16 smithi121 bash[12679]: cluster 2022-04-22T17:27:15.215754+0000 mgr.x (mgr.24749) 53 : cluster [DBG] pgmap v86: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:16.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:16 smithi121 bash[12679]: cluster 2022-04-22T17:27:15.225742+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2022-04-22T17:27:16.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:16 smithi121 bash[12679]: cluster 2022-04-22T17:27:16.179102+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2022-04-22T17:27:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[12737]: cluster 2022-04-22T17:27:15.215754+0000 mgr.x (mgr.24749) 53 : cluster [DBG] pgmap v86: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[12737]: cluster 2022-04-22T17:27:15.225742+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2022-04-22T17:27:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[12737]: cluster 2022-04-22T17:27:16.179102+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2022-04-22T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[18070]: cluster 2022-04-22T17:27:15.215754+0000 mgr.x (mgr.24749) 53 : cluster [DBG] pgmap v86: 121 pgs: 121 active+clean; 96 MiB data, 359 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[18070]: cluster 2022-04-22T17:27:15.225742+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2022-04-22T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:16 smithi115 bash[18070]: cluster 2022-04-22T17:27:16.179102+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:563: done (4 left) 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:564: done (3 left) 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:565: done (2 left) 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:566: done (1 left) 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:567: done (0 left) 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 50 2022-04-22T17:27:17.245 INFO:tasks.rados.rados.0.smithi115.stdout:568: seq_num 178 ranges {6881280=589824} 2022-04-22T17:27:17.293 INFO:tasks.rados.rados.0.smithi115.stdout:568: writing smithi11541303-23 from 6881280 to 7471104 tid 1 2022-04-22T17:27:17.294 INFO:tasks.rados.rados.0.smithi115.stdout:569: copy_from oid 26 from oid 22 current snap is 50 2022-04-22T17:27:17.294 INFO:tasks.rados.rados.0.smithi115.stdout:570: snap_create 2022-04-22T17:27:17.302 INFO:tasks.rados.rados.0.smithi115.stdout:568: finishing write tid 1 to smithi11541303-23 2022-04-22T17:27:17.302 INFO:tasks.rados.rados.0.smithi115.stdout:568: finishing write tid 2 to smithi11541303-23 2022-04-22T17:27:17.304 INFO:tasks.rados.rados.0.smithi115.stdout:568: finishing write tid 3 to smithi11541303-23 2022-04-22T17:27:17.304 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 179 (ObjNum 178 snap 50 seq_num 178) dirty exists 2022-04-22T17:27:17.304 INFO:tasks.rados.rados.0.smithi115.stdout:568: left oid 23 (ObjNum 178 snap 50 seq_num 178) 2022-04-22T17:27:17.324 INFO:tasks.rados.rados.0.smithi115.stdout:569: finishing copy_from to smithi11541303-26 2022-04-22T17:27:17.325 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:17.325 INFO:tasks.rados.rados.0.smithi115.stdout:569: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:27:18.043 INFO:tasks.rados.rados.0.smithi115.stdout:568: done (2 left) 2022-04-22T17:27:18.043 INFO:tasks.rados.rados.0.smithi115.stdout:569: done (1 left) 2022-04-22T17:27:18.043 INFO:tasks.rados.rados.0.smithi115.stdout:570: done (0 left) 2022-04-22T17:27:18.044 INFO:tasks.rados.rados.0.smithi115.stdout:571: snap_create 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cluster 2022-04-22T17:27:17.216549+0000 mgr.x (mgr.24749) 54 : cluster [DBG] pgmap v89: 121 pgs: 121 active+clean; 96 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 250 KiB/s wr, 13 op/s 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cluster 2022-04-22T17:27:17.236025+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: audit 2022-04-22T17:27:17.996877+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: audit 2022-04-22T17:27:18.006972+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: audit 2022-04-22T17:27:18.023032+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: audit 2022-04-22T17:27:18.025894+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:27:18.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:27:18.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: audit 2022-04-22T17:27:18.030862+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:18 smithi121 bash[12679]: cluster 2022-04-22T17:27:18.034597+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2022-04-22T17:27:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cluster 2022-04-22T17:27:17.216549+0000 mgr.x (mgr.24749) 54 : cluster [DBG] pgmap v89: 121 pgs: 121 active+clean; 96 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 250 KiB/s wr, 13 op/s 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cluster 2022-04-22T17:27:17.236025+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: audit 2022-04-22T17:27:17.996877+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: audit 2022-04-22T17:27:18.006972+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: audit 2022-04-22T17:27:18.023032+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: audit 2022-04-22T17:27:18.025894+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:27:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: audit 2022-04-22T17:27:18.030862+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[12737]: cluster 2022-04-22T17:27:18.034597+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cluster 2022-04-22T17:27:17.216549+0000 mgr.x (mgr.24749) 54 : cluster [DBG] pgmap v89: 121 pgs: 121 active+clean; 96 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 250 KiB/s wr, 13 op/s 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cluster 2022-04-22T17:27:17.236025+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: audit 2022-04-22T17:27:17.996877+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: audit 2022-04-22T17:27:18.006972+0000 mon.a (mon.0) 921 : audit [INF] from='mgr.24749 ' entity='mgr.x' 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: audit 2022-04-22T17:27:18.023032+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: audit 2022-04-22T17:27:18.025894+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:27:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:27:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: audit 2022-04-22T17:27:18.030862+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.24749 172.21.15.121:0/331551077' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2022-04-22T17:27:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:18 smithi115 bash[18070]: cluster 2022-04-22T17:27:18.034597+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2022-04-22T17:27:19.042 INFO:tasks.rados.rados.0.smithi115.stdout:571: done (0 left) 2022-04-22T17:27:19.042 INFO:tasks.rados.rados.0.smithi115.stdout:572: delete oid 4 current snap is 52 2022-04-22T17:27:19.049 INFO:tasks.rados.rados.0.smithi115.stdout:572: done (0 left) 2022-04-22T17:27:19.049 INFO:tasks.rados.rados.0.smithi115.stdout:573: delete oid 15 current snap is 52 2022-04-22T17:27:19.055 INFO:tasks.rados.rados.0.smithi115.stdout:573: done (0 left) 2022-04-22T17:27:19.055 INFO:tasks.rados.rados.0.smithi115.stdout:574: snap_create 2022-04-22T17:27:19.304 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: debug 2022-04-22T17:27:19.020+0000 7f15d1690700 -1 mgr handle_mgr_map I was active but no longer am 2022-04-22T17:27:19.357 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: ignoring --setuser ceph since I am not root 2022-04-22T17:27:19.357 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: ignoring --setgroup ceph since I am not root 2022-04-22T17:27:19.357 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: debug 2022-04-22T17:27:19.076+0000 7f977934e700 1 -- 172.21.15.121:0/911802837 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e69924340 con 0x562e69a1ec00 2022-04-22T17:27:19.357 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: debug 2022-04-22T17:27:19.248+0000 7f9781db5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:27:19.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: cluster 2022-04-22T17:27:18.956313+0000 mon.a (mon.0) 924 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.018179+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: cluster 2022-04-22T17:27:19.018296+0000 mon.a (mon.0) 926 : cluster [DBG] mgrmap e21: y(active, starting, since 0.991188s), standbys: x 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.020395+0000 mon.a (mon.0) 927 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.020556+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.020699+0000 mon.a (mon.0) 929 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:27:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.020926+0000 mon.a (mon.0) 930 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.021197+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.022418+0000 mon.a (mon.0) 932 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.022604+0000 mon.a (mon.0) 933 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.022946+0000 mon.a (mon.0) 934 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.023171+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:27:19.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.023395+0000 mon.a (mon.0) 936 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.023643+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.023844+0000 mon.a (mon.0) 938 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.024064+0000 mon.a (mon.0) 939 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.024402+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.024788+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.025073+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:27:19.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.025313+0000 mon.a (mon.0) 943 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.025791+0000 mon.a (mon.0) 944 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: cluster 2022-04-22T17:27:19.033954+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: cluster 2022-04-22T17:27:19.076393+0000 mon.a (mon.0) 946 : cluster [INF] Manager daemon y is now available 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.108668+0000 mon.a (mon.0) 947 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.114201+0000 mon.a (mon.0) 948 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.118837+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:19.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.121594+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:19.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.155264+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:27:19.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[12679]: audit 2022-04-22T17:27:19.163416+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:27:19.492 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12ee93a700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.492 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12ee93a700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12ee93a700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12ee93a700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12ee93a700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12f1940700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12f1940700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12f1940700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12f1940700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.493 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: debug 2022-04-22T17:27:19.211+0000 7f12f1940700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:27:19.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: cluster 2022-04-22T17:27:18.956313+0000 mon.a (mon.0) 924 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:19.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.018179+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2022-04-22T17:27:19.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: cluster 2022-04-22T17:27:19.018296+0000 mon.a (mon.0) 926 : cluster [DBG] mgrmap e21: y(active, starting, since 0.991188s), standbys: x 2022-04-22T17:27:19.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.020395+0000 mon.a (mon.0) 927 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.020556+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.020699+0000 mon.a (mon.0) 929 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.020926+0000 mon.a (mon.0) 930 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.021197+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.022418+0000 mon.a (mon.0) 932 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.022604+0000 mon.a (mon.0) 933 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.022946+0000 mon.a (mon.0) 934 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.023171+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.023395+0000 mon.a (mon.0) 936 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:27:19.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.023643+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.023844+0000 mon.a (mon.0) 938 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.024064+0000 mon.a (mon.0) 939 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.024402+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.024788+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.025073+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: cluster 2022-04-22T17:27:18.956313+0000 mon.a (mon.0) 924 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:19.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.018179+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24749 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: cluster 2022-04-22T17:27:19.018296+0000 mon.a (mon.0) 926 : cluster [DBG] mgrmap e21: y(active, starting, since 0.991188s), standbys: x 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.020395+0000 mon.a (mon.0) 927 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.020556+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.020699+0000 mon.a (mon.0) 929 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.020926+0000 mon.a (mon.0) 930 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.021197+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.022418+0000 mon.a (mon.0) 932 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.022604+0000 mon.a (mon.0) 933 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.022946+0000 mon.a (mon.0) 934 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.023171+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.023395+0000 mon.a (mon.0) 936 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:27:19.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.023643+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.023844+0000 mon.a (mon.0) 938 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.024064+0000 mon.a (mon.0) 939 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.024402+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.024788+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.025073+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.025313+0000 mon.a (mon.0) 943 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:27:19.498 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.025791+0000 mon.a (mon.0) 944 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: cluster 2022-04-22T17:27:19.033954+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: cluster 2022-04-22T17:27:19.076393+0000 mon.a (mon.0) 946 : cluster [INF] Manager daemon y is now available 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.108668+0000 mon.a (mon.0) 947 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.114201+0000 mon.a (mon.0) 948 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.118837+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.121594+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.155264+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[12737]: audit 2022-04-22T17:27:19.163416+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:27:19.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.025313+0000 mon.a (mon.0) 943 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.025791+0000 mon.a (mon.0) 944 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: cluster 2022-04-22T17:27:19.033954+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: cluster 2022-04-22T17:27:19.076393+0000 mon.a (mon.0) 946 : cluster [INF] Manager daemon y is now available 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.108668+0000 mon.a (mon.0) 947 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.114201+0000 mon.a (mon.0) 948 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.118837+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.121594+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.155264+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:27:19.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[18070]: audit 2022-04-22T17:27:19.163416+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:27:20.049 INFO:tasks.rados.rados.0.smithi115.stdout:574: done (0 left) 2022-04-22T17:27:20.049 INFO:tasks.rados.rados.0.smithi115.stdout:575: setattr oid 19 current snap is 53 2022-04-22T17:27:20.059 INFO:tasks.rados.rados.0.smithi115.stdout:576: rmattr oid 42 current snap is 53 2022-04-22T17:27:20.059 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 53 2022-04-22T17:27:20.059 INFO:tasks.rados.rados.0.smithi115.stdout:577: seq_num 181 ranges {622592=638976} 2022-04-22T17:27:20.065 INFO:tasks.rados.rados.0.smithi115.stdout:577: writing smithi11541303-38 from 622592 to 1261568 tid 1 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:578: read oid 26 snap -1 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:578: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 287 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:576: done (3 left) 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:579: read oid 17 snap 48 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:579: expect (ObjNum 165 snap 46 seq_num 165) 2022-04-22T17:27:20.066 INFO:tasks.rados.rados.0.smithi115.stdout:580: rollback oid 46 current snap is 53 2022-04-22T17:27:20.067 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 50 2022-04-22T17:27:20.067 INFO:tasks.rados.rados.0.smithi115.stdout:581: snap_create 2022-04-22T17:27:20.072 INFO:tasks.rados.rados.0.smithi115.stdout:580: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:27:20.074 INFO:tasks.rados.rados.0.smithi115.stdout:580: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:27:20.074 INFO:tasks.rados.rados.0.smithi115.stdout:580: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:27:20.074 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 183 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 142 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:20.077 INFO:tasks.rados.rados.0.smithi115.stdout:577: finishing write tid 1 to smithi11541303-38 2022-04-22T17:27:20.077 INFO:tasks.rados.rados.0.smithi115.stdout:577: finishing write tid 2 to smithi11541303-38 2022-04-22T17:27:20.078 INFO:tasks.rados.rados.0.smithi115.stdout:577: finishing write tid 3 to smithi11541303-38 2022-04-22T17:27:20.079 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 145 (ObjNum 181 snap 53 seq_num 181) dirty exists 2022-04-22T17:27:20.079 INFO:tasks.rados.rados.0.smithi115.stdout:577: left oid 38 (ObjNum 181 snap 53 seq_num 181) 2022-04-22T17:27:20.106 INFO:tasks.rados.rados.0.smithi115.stdout:579: expect (ObjNum 165 snap 46 seq_num 165) 2022-04-22T17:27:20.266 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:19 smithi121 bash[30140]: debug 2022-04-22T17:27:19.980+0000 7f9781db5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:27:20.296 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:19 smithi115 bash[42914]: [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:27:20.296 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[42914]: [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:27:20.296 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[42914]: [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:27:20.417 INFO:tasks.rados.rados.0.smithi115.stdout:578: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:20.576 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[30140]: debug 2022-04-22T17:27:20.312+0000 7f9781db5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:27:20.576 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[30140]: debug 2022-04-22T17:27:20.456+0000 7f9781db5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:27:21.059 INFO:tasks.rados.rados.0.smithi115.stdout:575: done (5 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:577: done (4 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:578: done (3 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:579: done (2 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:580: done (1 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:581: done (0 left) 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:582: rollback oid 42 current snap is 54 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 42 2022-04-22T17:27:21.060 INFO:tasks.rados.rados.0.smithi115.stdout:583: copy_from oid 3 from oid 1 current snap is 54 2022-04-22T17:27:21.061 INFO:tasks.rados.rados.0.smithi115.stdout:584: snap_remove snap 50 2022-04-22T17:27:21.163 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[12679]: audit 2022-04-22T17:27:19.822243+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:21.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[12679]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:27:21.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[12679]: cluster 2022-04-22T17:27:20.037285+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2022-04-22T17:27:21.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[12679]: cluster 2022-04-22T17:27:20.051756+0000 mon.a (mon.0) 955 : cluster [DBG] mgrmap e22: y(active, since 2s), standbys: x 2022-04-22T17:27:21.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[12679]: audit 2022-04-22T17:27:20.099093+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:21.165 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[30140]: debug 2022-04-22T17:27:20.568+0000 7f9781db5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:27:21.165 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[30140]: debug 2022-04-22T17:27:20.840+0000 7f9781db5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:27:21.165 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:20 smithi121 bash[30140]: debug 2022-04-22T17:27:20.948+0000 7f9781db5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:27:21.165 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:21 smithi121 bash[30140]: debug 2022-04-22T17:27:21.048+0000 7f9781db5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:27:21.166 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:21 smithi121 bash[30140]: debug 2022-04-22T17:27:21.156+0000 7f9781db5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:27:21.191 INFO:tasks.rados.rados.0.smithi115.stdout:584: done (2 left) 2022-04-22T17:27:21.191 INFO:tasks.rados.rados.0.smithi115.stdout:585: snap_create 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:582: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:582: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:582: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 291 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:583: finishing copy_from to smithi11541303-3 2022-04-22T17:27:21.192 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 79 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:27:21.193 INFO:tasks.rados.rados.0.smithi115.stdout:583: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:27:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[18070]: audit 2022-04-22T17:27:19.822243+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[18070]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:27:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[18070]: cluster 2022-04-22T17:27:20.037285+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2022-04-22T17:27:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[18070]: cluster 2022-04-22T17:27:20.051756+0000 mon.a (mon.0) 955 : cluster [DBG] mgrmap e22: y(active, since 2s), standbys: x 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[18070]: audit 2022-04-22T17:27:20.099093+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[12737]: audit 2022-04-22T17:27:19.822243+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[12737]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[12737]: cluster 2022-04-22T17:27:20.037285+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[12737]: cluster 2022-04-22T17:27:20.051756+0000 mon.a (mon.0) 955 : cluster [DBG] mgrmap e22: y(active, since 2s), standbys: x 2022-04-22T17:27:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:20 smithi115 bash[12737]: audit 2022-04-22T17:27:20.099093+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:22.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:21 smithi121 bash[30140]: debug 2022-04-22T17:27:21.612+0000 7f9781db5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:27:22.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:21 smithi121 bash[30140]: debug 2022-04-22T17:27:21.740+0000 7f9781db5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:22.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:21 smithi121 bash[30140]: debug 2022-04-22T17:27:21.848+0000 7f9781db5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:27:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:582: done (2 left) 2022-04-22T17:27:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:583: done (1 left) 2022-04-22T17:27:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:585: done (0 left) 2022-04-22T17:27:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:586: read oid 46 snap -1 2022-04-22T17:27:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:586: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:27:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:587: rollback oid 42 current snap is 55 2022-04-22T17:27:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 52 2022-04-22T17:27:22.200 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 55 2022-04-22T17:27:22.200 INFO:tasks.rados.rados.0.smithi115.stdout:588: seq_num 182 ranges {3956736=655360} 2022-04-22T17:27:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:588: writing smithi11541303-26 from 3956736 to 4612096 tid 1 2022-04-22T17:27:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:589: copy_from oid 41 from oid 40 current snap is 55 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:587: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:587: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:587: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 297 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:587: done (3 left) 2022-04-22T17:27:22.226 INFO:tasks.rados.rados.0.smithi115.stdout:590: rollback oid 39 current snap is 55 2022-04-22T17:27:22.227 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 53 2022-04-22T17:27:22.227 INFO:tasks.rados.rados.0.smithi115.stdout:586: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:590: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:588: finishing write tid 1 to smithi11541303-26 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:588: finishing write tid 2 to smithi11541303-26 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:588: finishing write tid 3 to smithi11541303-26 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 134 (ObjNum 182 snap 55 seq_num 182) dirty exists 2022-04-22T17:27:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:588: left oid 26 (ObjNum 182 snap 55 seq_num 182) 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:586: done (3 left) 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:588: done (2 left) 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:590: done (1 left) 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:591: copy_from oid 1 from oid 19 current snap is 55 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:592: rollback oid 25 current snap is 55 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 47 2022-04-22T17:27:22.252 INFO:tasks.rados.rados.0.smithi115.stdout:593: delete oid 22 current snap is 55 2022-04-22T17:27:22.255 INFO:tasks.rados.rados.0.smithi115.stdout:589: finishing copy_from to smithi11541303-41 2022-04-22T17:27:22.255 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:22.256 INFO:tasks.rados.rados.0.smithi115.stdout:589: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:27:22.256 INFO:tasks.rados.rados.0.smithi115.stdout:589: done (3 left) 2022-04-22T17:27:22.256 INFO:tasks.rados.rados.0.smithi115.stdout:593: done (2 left) 2022-04-22T17:27:22.256 INFO:tasks.rados.rados.0.smithi115.stdout:594: delete oid 15 current snap is 55 2022-04-22T17:27:22.257 INFO:tasks.rados.rados.0.smithi115.stdout:592: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:27:22.258 INFO:tasks.rados.rados.0.smithi115.stdout:592: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:27:22.258 INFO:tasks.rados.rados.0.smithi115.stdout:594: done (2 left) 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:595: read oid 9 snap -1 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:595: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:596: snap_create 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:592: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 225 (ObjNum 1 snap 32692 seq_num 3178612072) dirty exists 2022-04-22T17:27:22.259 INFO:tasks.rados.rados.0.smithi115.stdout:595: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:22.265 INFO:tasks.rados.rados.0.smithi115.stdout:591: finishing copy_from to smithi11541303-1 2022-04-22T17:27:22.265 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 142 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:22.265 INFO:tasks.rados.rados.0.smithi115.stdout:591: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:27:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:27:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:27:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cluster 2022-04-22T17:27:20.299928+0000 mgr.y (mgr.24919) 4 : cluster [DBG] pgmap v5: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cluster 2022-04-22T17:27:21.024566+0000 mgr.y (mgr.24919) 5 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cluster 2022-04-22T17:27:21.050123+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2022-04-22T17:27:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cluster 2022-04-22T17:27:21.110263+0000 mon.a (mon.0) 958 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2022-04-22T17:27:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[12679]: cluster 2022-04-22T17:27:21.181093+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2022-04-22T17:27:22.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[30140]: debug 2022-04-22T17:27:22.044+0000 7f9781db5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:27:22.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[30140]: debug 2022-04-22T17:27:22.160+0000 7f9781db5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:27:22.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[30140]: debug 2022-04-22T17:27:22.440+0000 7f9781db5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:27:22.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cluster 2022-04-22T17:27:20.299928+0000 mgr.y (mgr.24919) 4 : cluster [DBG] pgmap v5: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cluster 2022-04-22T17:27:21.024566+0000 mgr.y (mgr.24919) 5 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cluster 2022-04-22T17:27:21.050123+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cluster 2022-04-22T17:27:21.110263+0000 mon.a (mon.0) 958 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[12737]: cluster 2022-04-22T17:27:21.181093+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2022-04-22T17:27:22.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cluster 2022-04-22T17:27:20.299928+0000 mgr.y (mgr.24919) 4 : cluster [DBG] pgmap v5: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cluster 2022-04-22T17:27:21.024566+0000 mgr.y (mgr.24919) 5 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cluster 2022-04-22T17:27:21.050123+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cluster 2022-04-22T17:27:21.110263+0000 mon.a (mon.0) 958 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2022-04-22T17:27:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:22 smithi115 bash[18070]: cluster 2022-04-22T17:27:21.181093+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2022-04-22T17:27:22.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[30140]: debug 2022-04-22T17:27:22.608+0000 7f9781db5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:27:23.201 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:22 smithi121 bash[30140]: debug 2022-04-22T17:27:22.872+0000 7f9781db5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:27:23.201 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[30140]: debug 2022-04-22T17:27:23.112+0000 7f9781db5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:27:23.204 INFO:tasks.rados.rados.0.smithi115.stdout:591: done (3 left) 2022-04-22T17:27:23.204 INFO:tasks.rados.rados.0.smithi115.stdout:592: done (2 left) 2022-04-22T17:27:23.204 INFO:tasks.rados.rados.0.smithi115.stdout:595: done (1 left) 2022-04-22T17:27:23.204 INFO:tasks.rados.rados.0.smithi115.stdout:596: done (0 left) 2022-04-22T17:27:23.204 INFO:tasks.rados.rados.0.smithi115.stdout:597: read oid 35 snap 53 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:597: expect (ObjNum 103 snap 19 seq_num 103) 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:598: rollback oid 12 current snap is 56 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 55 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:599: rollback oid 49 current snap is 56 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 49 2022-04-22T17:27:23.205 INFO:tasks.rados.rados.0.smithi115.stdout:600: read oid 4 snap -1 2022-04-22T17:27:23.206 INFO:tasks.rados.rados.0.smithi115.stdout:600: expect deleted 2022-04-22T17:27:23.206 INFO:tasks.rados.rados.0.smithi115.stdout:601: rollback oid 39 current snap is 56 2022-04-22T17:27:23.206 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 49 2022-04-22T17:27:23.206 INFO:tasks.rados.rados.0.smithi115.stdout:602: snap_create 2022-04-22T17:27:23.213 INFO:tasks.rados.rados.0.smithi115.stdout:601: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:27:23.213 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 0 (ObjNum -1117898912 snap 22075 seq_num 3178684960) dirty dne 2022-04-22T17:27:23.215 INFO:tasks.rados.rados.0.smithi115.stdout:598: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:27:23.222 INFO:tasks.rados.rados.0.smithi115.stdout:597: expect (ObjNum 103 snap 19 seq_num 103) 2022-04-22T17:27:23.301 INFO:tasks.rados.rados.0.smithi115.stdout:598: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:598: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 153 (ObjNum 140 snap 36 seq_num 140) dirty exists 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:599: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:599: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:599: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:27:23.313 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 301 (ObjNum 175 snap 49 seq_num 175) dirty exists 2022-04-22T17:27:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[12679]: cluster 2022-04-22T17:27:22.185345+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2022-04-22T17:27:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[12679]: cluster 2022-04-22T17:27:23.025229+0000 mgr.y (mgr.24919) 6 : cluster [DBG] pgmap v10: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:23.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[30140]: debug 2022-04-22T17:27:23.356+0000 7f9781db5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:23.519 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[12737]: cluster 2022-04-22T17:27:22.185345+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2022-04-22T17:27:23.519 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[12737]: cluster 2022-04-22T17:27:23.025229+0000 mgr.y (mgr.24919) 6 : cluster [DBG] pgmap v10: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:23.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[18070]: cluster 2022-04-22T17:27:22.185345+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2022-04-22T17:27:23.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[18070]: cluster 2022-04-22T17:27:23.025229+0000 mgr.y (mgr.24919) 6 : cluster [DBG] pgmap v10: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:23.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:23.879 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:27:23.879 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" 2022-04-22T17:27:23.879 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:27:23.880 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:23.880 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:23.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:27:23.895 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[30140]: debug 2022-04-22T17:27:23.632+0000 7f9781db5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:27:24.203 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:23 smithi121 bash[30140]: debug 2022-04-22T17:27:23.888+0000 7f9781db5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:24.214 INFO:tasks.rados.rados.0.smithi115.stdout:597: done (5 left) 2022-04-22T17:27:24.214 INFO:tasks.rados.rados.0.smithi115.stdout:598: done (4 left) 2022-04-22T17:27:24.214 INFO:tasks.rados.rados.0.smithi115.stdout:599: done (3 left) 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:600: done (2 left) 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:601: done (1 left) 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:602: done (0 left) 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:603: rollback oid 44 current snap is 57 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 52 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 57 2022-04-22T17:27:24.215 INFO:tasks.rados.rados.0.smithi115.stdout:604: seq_num 185 ranges {1654784=704512} 2022-04-22T17:27:24.228 INFO:tasks.rados.rados.0.smithi115.stdout:604: writing smithi11541303-30 from 1654784 to 2359296 tid 1 2022-04-22T17:27:24.228 INFO:tasks.rados.rados.0.smithi115.stdout:605: rollback oid 40 current snap is 57 2022-04-22T17:27:24.228 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 53 2022-04-22T17:27:24.229 INFO:tasks.rados.rados.0.smithi115.stdout:603: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:27:24.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 0 (ObjNum -1116378576 snap 22075 seq_num 3178270032) dirty dne 2022-04-22T17:27:24.229 INFO:tasks.rados.rados.0.smithi115.stdout:603: done (2 left) 2022-04-22T17:27:24.229 INFO:tasks.rados.rados.0.smithi115.stdout:606: snap_remove snap 56 2022-04-22T17:27:24.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:24 smithi115 bash[18070]: cluster 2022-04-22T17:27:23.194704+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2022-04-22T17:27:24.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:24 smithi115 bash[12737]: cluster 2022-04-22T17:27:23.194704+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2022-04-22T17:27:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:24 smithi121 bash[12679]: cluster 2022-04-22T17:27:23.194704+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2022-04-22T17:27:25.231 INFO:tasks.rados.rados.0.smithi115.stdout:606: done (2 left) 2022-04-22T17:27:25.231 INFO:tasks.rados.rados.0.smithi115.stdout:607: snap_create 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:605: finishing rollback tid 0 to smithi11541303-40 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:604: finishing write tid 1 to smithi11541303-30 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:604: finishing write tid 2 to smithi11541303-30 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:605: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:605: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 182 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:604: finishing write tid 3 to smithi11541303-30 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 277 (ObjNum 185 snap 57 seq_num 185) dirty exists 2022-04-22T17:27:25.232 INFO:tasks.rados.rados.0.smithi115.stdout:604: left oid 30 (ObjNum 185 snap 57 seq_num 185) 2022-04-22T17:27:25.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[12679]: cluster 2022-04-22T17:27:24.204790+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2022-04-22T17:27:25.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[12679]: audit 2022-04-22T17:27:24.459159+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[12679]: audit 2022-04-22T17:27:24.858409+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[12679]: cluster 2022-04-22T17:27:25.025797+0000 mgr.y (mgr.24919) 7 : cluster [DBG] pgmap v13: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:25.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[12679]: audit 2022-04-22T17:27:25.029715+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[18070]: cluster 2022-04-22T17:27:24.204790+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[18070]: audit 2022-04-22T17:27:24.459159+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[18070]: audit 2022-04-22T17:27:24.858409+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[18070]: cluster 2022-04-22T17:27:25.025797+0000 mgr.y (mgr.24919) 7 : cluster [DBG] pgmap v13: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[18070]: audit 2022-04-22T17:27:25.029715+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[12737]: cluster 2022-04-22T17:27:24.204790+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[12737]: audit 2022-04-22T17:27:24.459159+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[12737]: audit 2022-04-22T17:27:24.858409+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[12737]: cluster 2022-04-22T17:27:25.025797+0000 mgr.y (mgr.24919) 7 : cluster [DBG] pgmap v13: 121 pgs: 121 active+clean; 104 MiB data, 389 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:25 smithi115 bash[12737]: audit 2022-04-22T17:27:25.029715+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[30140]: debug 2022-04-22T17:27:25.792+0000 7f9781db5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:27:26.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:25 smithi121 bash[30140]: debug 2022-04-22T17:27:25.904+0000 7f9781db5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:26.233 INFO:tasks.rados.rados.0.smithi115.stdout:604: done (2 left) 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:605: done (1 left) 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:607: done (0 left) 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:608: rmattr oid 40 current snap is 58 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:609: read oid 48 snap 51 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:609: expect (ObjNum 150 snap 40 seq_num 150) 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:610: read oid 13 snap -1 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:610: expect (ObjNum 134 snap 35 seq_num 134) 2022-04-22T17:27:26.234 INFO:tasks.rados.rados.0.smithi115.stdout:611: snap_remove snap 53 2022-04-22T17:27:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:26 smithi121 bash[12679]: cluster 2022-04-22T17:27:25.222076+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2022-04-22T17:27:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:26 smithi121 bash[12679]: audit 2022-04-22T17:27:25.429776+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:26 smithi121 bash[12679]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:27:26.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:26 smithi121 bash[12679]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:26.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:26 smithi121 bash[12679]: audit 2022-04-22T17:27:25.940711+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[12737]: cluster 2022-04-22T17:27:25.222076+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[12737]: audit 2022-04-22T17:27:25.429776+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[12737]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[12737]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[12737]: audit 2022-04-22T17:27:25.940711+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[18070]: cluster 2022-04-22T17:27:25.222076+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2022-04-22T17:27:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[18070]: audit 2022-04-22T17:27:25.429776+0000 mon.a (mon.0) 967 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[18070]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:27:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[18070]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:26 smithi115 bash[18070]: audit 2022-04-22T17:27:25.940711+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:27.236 INFO:tasks.rados.rados.0.smithi115.stdout:611: done (3 left) 2022-04-22T17:27:27.236 INFO:tasks.rados.rados.0.smithi115.stdout:612: rollback oid 4 current snap is 58 2022-04-22T17:27:27.236 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 47 2022-04-22T17:27:27.236 INFO:tasks.rados.rados.0.smithi115.stdout:609: expect (ObjNum 150 snap 40 seq_num 150) 2022-04-22T17:27:27.242 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 183 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:27.242 INFO:tasks.rados.rados.0.smithi115.stdout:610: expect (ObjNum 134 snap 35 seq_num 134) 2022-04-22T17:27:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:27 smithi121 bash[12679]: cluster 2022-04-22T17:27:26.222981+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2022-04-22T17:27:27.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:27 smithi121 bash[12679]: cluster 2022-04-22T17:27:27.026492+0000 mgr.y (mgr.24919) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:27.551 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:27 smithi121 bash[30140]: debug 2022-04-22T17:27:27.272+0000 7f9781db5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:27:27.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:27 smithi115 bash[18070]: cluster 2022-04-22T17:27:26.222981+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2022-04-22T17:27:27.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:27 smithi115 bash[18070]: cluster 2022-04-22T17:27:27.026492+0000 mgr.y (mgr.24919) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:27.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:27 smithi115 bash[12737]: cluster 2022-04-22T17:27:26.222981+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2022-04-22T17:27:27.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:27 smithi115 bash[12737]: cluster 2022-04-22T17:27:27.026492+0000 mgr.y (mgr.24919) 10 : cluster [DBG] pgmap v16: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:27.766 INFO:tasks.rados.rados.0.smithi115.stdout:612: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:27:27.766 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 275 (ObjNum 0 snap 22075 seq_num 3178659568) dirty dne 2022-04-22T17:27:27.766 INFO:tasks.rados.rados.0.smithi115.stdout:608: done (3 left) 2022-04-22T17:27:27.766 INFO:tasks.rados.rados.0.smithi115.stdout:609: done (2 left) 2022-04-22T17:27:27.767 INFO:tasks.rados.rados.0.smithi115.stdout:610: done (1 left) 2022-04-22T17:27:27.768 INFO:tasks.rados.rados.0.smithi115.stdout:612: done (0 left) 2022-04-22T17:27:27.768 INFO:tasks.rados.rados.0.smithi115.stdout:613: delete oid 17 current snap is 58 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:613: done (0 left) 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:614: read oid 42 snap -1 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:614: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:615: rollback oid 45 current snap is 58 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 49 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:616: rmattr oid 40 current snap is 58 2022-04-22T17:27:27.774 INFO:tasks.rados.rados.0.smithi115.stdout:617: copy_from oid 44 from oid 6 current snap is 58 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:618: rollback oid 23 current snap is 58 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 55 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:619: copy_from oid 32 from oid 24 current snap is 58 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:620: copy_from oid 4 from oid 13 current snap is 58 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 58 2022-04-22T17:27:27.775 INFO:tasks.rados.rados.0.smithi115.stdout:621: seq_num 187 ranges {761856=737280} 2022-04-22T17:27:27.788 INFO:tasks.rados.rados.0.smithi115.stdout:621: writing smithi11541303-34 from 761856 to 1499136 tid 1 2022-04-22T17:27:27.789 INFO:tasks.rados.rados.0.smithi115.stdout:622: snap_remove snap 47 2022-04-22T17:27:28.254 INFO:tasks.rados.rados.0.smithi115.stdout:622: done (8 left) 2022-04-22T17:27:28.254 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 58 2022-04-22T17:27:28.255 INFO:tasks.rados.rados.0.smithi115.stdout:617: finishing copy_from to smithi11541303-44 2022-04-22T17:27:28.255 INFO:tasks.rados.rados.0.smithi115.stdout:617: got expected ENOENT (src dne) 2022-04-22T17:27:28.255 INFO:tasks.rados.rados.0.smithi115.stdout:623: seq_num 188 ranges {1261568=753664} 2022-04-22T17:27:28.268 INFO:tasks.rados.rados.0.smithi115.stdout:623: writing smithi11541303-38 from 1261568 to 2015232 tid 1 2022-04-22T17:27:28.268 INFO:tasks.rados.rados.0.smithi115.stdout:617: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:27:28.269 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 184 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:28.269 INFO:tasks.rados.rados.0.smithi115.stdout:615: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:27:28.269 INFO:tasks.rados.rados.0.smithi115.stdout:614: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:28.293 INFO:tasks.rados.rados.0.smithi115.stdout:618: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:27:28.293 INFO:tasks.rados.rados.0.smithi115.stdout:615: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:27:28.293 INFO:tasks.rados.rados.0.smithi115.stdout:618: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:27:28.293 INFO:tasks.rados.rados.0.smithi115.stdout:615: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 280 (ObjNum 111 snap 26 seq_num 111) dirty exists 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:618: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 195 (ObjNum 178 snap 50 seq_num 178) dirty exists 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:621: finishing write tid 1 to smithi11541303-34 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:621: finishing write tid 2 to smithi11541303-34 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:619: finishing copy_from to smithi11541303-32 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 43 (ObjNum 131 snap 34 seq_num 131) dirty exists 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:619: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:624: read oid 20 snap -1 2022-04-22T17:27:28.294 INFO:tasks.rados.rados.0.smithi115.stdout:624: expect (ObjNum 164 snap 45 seq_num 164) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:614: done (9 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:615: done (8 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:616: done (7 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:617: done (6 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:618: done (5 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:619: done (4 left) 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 58 2022-04-22T17:27:28.295 INFO:tasks.rados.rados.0.smithi115.stdout:621: finishing write tid 3 to smithi11541303-34 2022-04-22T17:27:28.296 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 141 (ObjNum 187 snap 58 seq_num 187) dirty exists 2022-04-22T17:27:28.296 INFO:tasks.rados.rados.0.smithi115.stdout:621: left oid 34 (ObjNum 187 snap 58 seq_num 187) 2022-04-22T17:27:28.296 INFO:tasks.rados.rados.0.smithi115.stdout:620: finishing copy_from to smithi11541303-4 2022-04-22T17:27:28.296 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 140 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:27:28.296 INFO:tasks.rados.rados.0.smithi115.stdout:625: seq_num 189 ranges {2310144=770048} 2022-04-22T17:27:28.316 INFO:tasks.rados.rados.0.smithi115.stdout:625: writing smithi11541303-27 from 2310144 to 3080192 tid 1 2022-04-22T17:27:28.316 INFO:tasks.rados.rados.0.smithi115.stdout:621: done (4 left) 2022-04-22T17:27:28.316 INFO:tasks.rados.rados.0.smithi115.stdout:626: snap_create 2022-04-22T17:27:28.317 INFO:tasks.rados.rados.0.smithi115.stdout:620: finishing copy_from racing read to smithi11541303-4 2022-04-22T17:27:28.317 INFO:tasks.rados.rados.0.smithi115.stdout:623: finishing write tid 1 to smithi11541303-38 2022-04-22T17:27:28.317 INFO:tasks.rados.rados.0.smithi115.stdout:623: finishing write tid 2 to smithi11541303-38 2022-04-22T17:27:28.318 INFO:tasks.rados.rados.0.smithi115.stdout:623: finishing write tid 3 to smithi11541303-38 2022-04-22T17:27:28.318 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 162 (ObjNum 188 snap 58 seq_num 188) dirty exists 2022-04-22T17:27:28.319 INFO:tasks.rados.rados.0.smithi115.stdout:623: left oid 38 (ObjNum 188 snap 58 seq_num 188) 2022-04-22T17:27:28.319 INFO:tasks.rados.rados.0.smithi115.stdout:624: expect (ObjNum 164 snap 45 seq_num 164) 2022-04-22T17:27:28.427 INFO:tasks.rados.rados.0.smithi115.stdout:625: finishing write tid 1 to smithi11541303-27 2022-04-22T17:27:28.428 INFO:tasks.rados.rados.0.smithi115.stdout:625: finishing write tid 2 to smithi11541303-27 2022-04-22T17:27:28.428 INFO:tasks.rados.rados.0.smithi115.stdout:625: finishing write tid 3 to smithi11541303-27 2022-04-22T17:27:28.428 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 282 (ObjNum 189 snap 58 seq_num 189) dirty exists 2022-04-22T17:27:28.428 INFO:tasks.rados.rados.0.smithi115.stdout:625: left oid 27 (ObjNum 189 snap 58 seq_num 189) 2022-04-22T17:27:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: cluster 2022-04-22T17:27:27.225896+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2022-04-22T17:27:28.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: cluster 2022-04-22T17:27:27.291922+0000 mon.a (mon.0) 971 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:28.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: cluster 2022-04-22T17:27:27.292072+0000 mon.a (mon.0) 972 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:28.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: audit 2022-04-22T17:27:27.298975+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:28.602 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: audit 2022-04-22T17:27:27.302529+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:28.602 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: audit 2022-04-22T17:27:27.303902+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:28.602 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:28 smithi121 bash[12679]: audit 2022-04-22T17:27:27.304536+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:28.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: cluster 2022-04-22T17:27:27.225896+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: cluster 2022-04-22T17:27:27.291922+0000 mon.a (mon.0) 971 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: cluster 2022-04-22T17:27:27.292072+0000 mon.a (mon.0) 972 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: audit 2022-04-22T17:27:27.298975+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: audit 2022-04-22T17:27:27.302529+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: audit 2022-04-22T17:27:27.303902+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[12737]: audit 2022-04-22T17:27:27.304536+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:28.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: cluster 2022-04-22T17:27:27.225896+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: cluster 2022-04-22T17:27:27.291922+0000 mon.a (mon.0) 971 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: cluster 2022-04-22T17:27:27.292072+0000 mon.a (mon.0) 972 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: audit 2022-04-22T17:27:27.298975+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: audit 2022-04-22T17:27:27.302529+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: audit 2022-04-22T17:27:27.303902+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:28 smithi115 bash[18070]: audit 2022-04-22T17:27:27.304536+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.121:0/349856013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:29.267 INFO:tasks.rados.rados.0.smithi115.stdout:620: done (4 left) 2022-04-22T17:27:29.268 INFO:tasks.rados.rados.0.smithi115.stdout:623: done (3 left) 2022-04-22T17:27:29.268 INFO:tasks.rados.rados.0.smithi115.stdout:624: done (2 left) 2022-04-22T17:27:29.268 INFO:tasks.rados.rados.0.smithi115.stdout:625: done (1 left) 2022-04-22T17:27:29.268 INFO:tasks.rados.rados.0.smithi115.stdout:626: done (0 left) 2022-04-22T17:27:29.268 INFO:tasks.rados.rados.0.smithi115.stdout:627: delete oid 2 current snap is 59 2022-04-22T17:27:29.272 INFO:tasks.rados.rados.0.smithi115.stdout:627: done (0 left) 2022-04-22T17:27:29.273 INFO:tasks.rados.rados.0.smithi115.stdout:628: delete oid 5 current snap is 59 2022-04-22T17:27:29.278 INFO:tasks.rados.rados.0.smithi115.stdout:628: done (0 left) 2022-04-22T17:27:29.278 INFO:tasks.rados.rados.0.smithi115.stdout:629: read oid 43 snap -1 2022-04-22T17:27:29.278 INFO:tasks.rados.rados.0.smithi115.stdout:629: expect deleted 2022-04-22T17:27:29.279 INFO:tasks.rados.rados.0.smithi115.stdout:630: read oid 28 snap 54 2022-04-22T17:27:29.279 INFO:tasks.rados.rados.0.smithi115.stdout:630: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:29.279 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 59 2022-04-22T17:27:29.279 INFO:tasks.rados.rados.0.smithi115.stdout:631: seq_num 192 ranges {0=417792,417792=401408} 2022-04-22T17:27:29.282 INFO:tasks.rados.rados.0.smithi115.stdout:631: writing smithi11541303-11 from 0 to 417792 tid 1 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:631: writing smithi11541303-11 from 417792 to 819200 tid 2 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:632: read oid 39 snap -1 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:632: expect deleted 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:633: rollback oid 18 current snap is 59 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 42 2022-04-22T17:27:29.287 INFO:tasks.rados.rados.0.smithi115.stdout:634: setattr oid 6 current snap is 59 2022-04-22T17:27:29.288 INFO:tasks.rados.rados.0.smithi115.stdout:630: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:29.307 INFO:tasks.rados.rados.0.smithi115.stdout:631: finishing write tid 1 to smithi11541303-11 2022-04-22T17:27:29.307 INFO:tasks.rados.rados.0.smithi115.stdout:631: finishing write tid 2 to smithi11541303-11 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:633: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:631: finishing write tid 3 to smithi11541303-11 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:633: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:633: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 315 (ObjNum 89 snap 15 seq_num 89) dirty exists 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:631: finishing write tid 4 to smithi11541303-11 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 204 (ObjNum 192 snap 59 seq_num 192) dirty exists 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:631: left oid 11 (ObjNum 192 snap 59 seq_num 192) 2022-04-22T17:27:29.308 INFO:tasks.rados.rados.0.smithi115.stdout:629: done (5 left) 2022-04-22T17:27:29.309 INFO:tasks.rados.rados.0.smithi115.stdout:630: done (4 left) 2022-04-22T17:27:29.309 INFO:tasks.rados.rados.0.smithi115.stdout:631: done (3 left) 2022-04-22T17:27:29.309 INFO:tasks.rados.rados.0.smithi115.stdout:632: done (2 left) 2022-04-22T17:27:29.309 INFO:tasks.rados.rados.0.smithi115.stdout:633: done (1 left) 2022-04-22T17:27:29.309 INFO:tasks.rados.rados.0.smithi115.stdout:635: delete oid 27 current snap is 59 2022-04-22T17:27:29.310 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 62 (ObjNum -1116205376 snap 22075 seq_num 0) dirty exists 2022-04-22T17:27:29.312 INFO:tasks.rados.rados.0.smithi115.stdout:634: done (1 left) 2022-04-22T17:27:29.312 INFO:tasks.rados.rados.0.smithi115.stdout:635: done (0 left) 2022-04-22T17:27:29.312 INFO:tasks.rados.rados.0.smithi115.stdout:636: rollback oid 15 current snap is 59 2022-04-22T17:27:29.312 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 42 2022-04-22T17:27:29.313 INFO:tasks.rados.rados.0.smithi115.stdout:637: setattr oid 42 current snap is 59 2022-04-22T17:27:29.313 INFO:tasks.rados.rados.0.smithi115.stdout:638: setattr oid 27 current snap is 59 2022-04-22T17:27:29.315 INFO:tasks.rados.rados.0.smithi115.stdout:639: snap_remove snap 57 2022-04-22T17:27:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:29 smithi121 bash[12679]: cluster 2022-04-22T17:27:28.243341+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2022-04-22T17:27:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:29 smithi121 bash[12679]: cluster 2022-04-22T17:27:28.258483+0000 mon.a (mon.0) 974 : cluster [DBG] mgrmap e24: y(active, since 10s), standbys: x 2022-04-22T17:27:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:29 smithi121 bash[12679]: cluster 2022-04-22T17:27:29.027353+0000 mgr.y (mgr.24919) 11 : cluster [DBG] pgmap v19: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[18070]: cluster 2022-04-22T17:27:28.243341+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2022-04-22T17:27:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[18070]: cluster 2022-04-22T17:27:28.258483+0000 mon.a (mon.0) 974 : cluster [DBG] mgrmap e24: y(active, since 10s), standbys: x 2022-04-22T17:27:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[18070]: cluster 2022-04-22T17:27:29.027353+0000 mgr.y (mgr.24919) 11 : cluster [DBG] pgmap v19: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[12737]: cluster 2022-04-22T17:27:28.243341+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2022-04-22T17:27:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[12737]: cluster 2022-04-22T17:27:28.258483+0000 mon.a (mon.0) 974 : cluster [DBG] mgrmap e24: y(active, since 10s), standbys: x 2022-04-22T17:27:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:29 smithi115 bash[12737]: cluster 2022-04-22T17:27:29.027353+0000 mgr.y (mgr.24919) 11 : cluster [DBG] pgmap v19: 121 pgs: 121 active+clean; 110 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2022-04-22T17:27:29.798 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:27:30.121 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:27:30.121 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 5s ago 4m 12.6M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4m) 5s ago 4m 23.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5m) 5s ago 5m 12.8M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5m) 5s ago 5m 15.8M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (81s) 5s ago 9m 184M - 17.0.0-11772-g72067459 6c95b0749680 988b90e88450 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (27s) 5s ago 11m 433M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (11m) 5s ago 11m 52.3M 2048M 16.2.5 46cf6318c64e c6b4ba941f5c 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (9m) 5s ago 9m 44.1M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 5s ago 10m 41.8M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:27:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 5s ago 4m 8699k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4m) 5s ago 4m 8779k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 5s ago 9m 113M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 5s ago 8m 101M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 5s ago 8m 82.6M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 5s ago 7m 67.4M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (7m) 5s ago 7m 93.3M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6m) 5s ago 6m 92.8M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6m) 5s ago 6m 86.8M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5m) 5s ago 5m 90.2M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:27:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (34s) 5s ago 4m 29.4M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:27:30.278 INFO:tasks.rados.rados.0.smithi115.stdout:639: done (3 left) 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:640: delete oid 35 current snap is 59 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:636: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:636: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 75 (ObjNum 155 snap 41 seq_num 155) dirty exists 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 284 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:27:30.279 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 317 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:30.289 INFO:tasks.rados.rados.0.smithi115.stdout:636: done (3 left) 2022-04-22T17:27:30.289 INFO:tasks.rados.rados.0.smithi115.stdout:637: done (2 left) 2022-04-22T17:27:30.289 INFO:tasks.rados.rados.0.smithi115.stdout:638: done (1 left) 2022-04-22T17:27:30.289 INFO:tasks.rados.rados.0.smithi115.stdout:640: done (0 left) 2022-04-22T17:27:30.289 INFO:tasks.rados.rados.0.smithi115.stdout:641: read oid 42 snap -1 2022-04-22T17:27:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:641: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:30.290 INFO:tasks.rados.rados.0.smithi115.stdout:642: snap_create 2022-04-22T17:27:30.295 INFO:tasks.rados.rados.0.smithi115.stdout:641: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:27:30.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13, 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:27:30.541 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:27:30.542 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:27:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:30 smithi115 bash[12737]: cluster 2022-04-22T17:27:29.258344+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2022-04-22T17:27:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:30 smithi115 bash[12737]: audit 2022-04-22T17:27:29.774187+0000 mgr.y (mgr.24919) 12 : audit [DBG] from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:30.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:30 smithi115 bash[18070]: cluster 2022-04-22T17:27:29.258344+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2022-04-22T17:27:30.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:30 smithi115 bash[18070]: audit 2022-04-22T17:27:29.774187+0000 mgr.y (mgr.24919) 12 : audit [DBG] from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:30.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:30 smithi121 bash[12679]: cluster 2022-04-22T17:27:29.258344+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2022-04-22T17:27:30.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:30 smithi121 bash[12679]: audit 2022-04-22T17:27:29.774187+0000 mgr.y (mgr.24919) 12 : audit [DBG] from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:31.195 INFO:tasks.rados.rados.0.smithi115.stdout:641: done (1 left) 2022-04-22T17:27:31.195 INFO:tasks.rados.rados.0.smithi115.stdout:642: done (0 left) 2022-04-22T17:27:31.196 INFO:tasks.rados.rados.0.smithi115.stdout:643: copy_from oid 2 from oid 28 current snap is 60 2022-04-22T17:27:31.196 INFO:tasks.rados.rados.0.smithi115.stdout:644: snap_remove snap 48 2022-04-22T17:27:31.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[18070]: audit 2022-04-22T17:27:30.092307+0000 mgr.y (mgr.24919) 13 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:31.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[18070]: cluster 2022-04-22T17:27:30.267043+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2022-04-22T17:27:31.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[18070]: audit 2022-04-22T17:27:30.530764+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/1305291140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:31.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[18070]: cluster 2022-04-22T17:27:31.028394+0000 mgr.y (mgr.24919) 14 : cluster [DBG] pgmap v22: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:31.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[18070]: cluster 2022-04-22T17:27:31.185852+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2022-04-22T17:27:31.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[12737]: audit 2022-04-22T17:27:30.092307+0000 mgr.y (mgr.24919) 13 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:31.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[12737]: cluster 2022-04-22T17:27:30.267043+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2022-04-22T17:27:31.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[12737]: audit 2022-04-22T17:27:30.530764+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/1305291140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:31.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[12737]: cluster 2022-04-22T17:27:31.028394+0000 mgr.y (mgr.24919) 14 : cluster [DBG] pgmap v22: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:31.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:31 smithi115 bash[12737]: cluster 2022-04-22T17:27:31.185852+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2022-04-22T17:27:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:31 smithi121 bash[12679]: audit 2022-04-22T17:27:30.092307+0000 mgr.y (mgr.24919) 13 : audit [DBG] from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:27:31.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:31 smithi121 bash[12679]: cluster 2022-04-22T17:27:30.267043+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2022-04-22T17:27:31.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:31 smithi121 bash[12679]: audit 2022-04-22T17:27:30.530764+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/1305291140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:31.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:31 smithi121 bash[12679]: cluster 2022-04-22T17:27:31.028394+0000 mgr.y (mgr.24919) 14 : cluster [DBG] pgmap v22: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:31.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:31 smithi121 bash[12679]: cluster 2022-04-22T17:27:31.185852+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2022-04-22T17:27:32.199 INFO:tasks.rados.rados.0.smithi115.stdout:644: done (1 left) 2022-04-22T17:27:32.199 INFO:tasks.rados.rados.0.smithi115.stdout:645: delete oid 12 current snap is 60 2022-04-22T17:27:32.199 INFO:tasks.rados.rados.0.smithi115.stdout:643: finishing copy_from to smithi11541303-2 2022-04-22T17:27:32.199 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 120 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:32.199 INFO:tasks.rados.rados.0.smithi115.stdout:643: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:27:32.216 INFO:tasks.rados.rados.0.smithi115.stdout:643: done (1 left) 2022-04-22T17:27:32.216 INFO:tasks.rados.rados.0.smithi115.stdout:645: done (0 left) 2022-04-22T17:27:32.216 INFO:tasks.rados.rados.0.smithi115.stdout:646: read oid 24 snap -1 2022-04-22T17:27:32.216 INFO:tasks.rados.rados.0.smithi115.stdout:646: expect (ObjNum 131 snap 34 seq_num 131) 2022-04-22T17:27:32.219 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 60 2022-04-22T17:27:32.220 INFO:tasks.rados.rados.0.smithi115.stdout:647: seq_num 196 ranges {0=491520,491520=401408} 2022-04-22T17:27:32.220 INFO:tasks.rados.rados.0.smithi115.stdout:647: writing smithi11541303-12 from 0 to 491520 tid 1 2022-04-22T17:27:32.222 INFO:tasks.rados.rados.0.smithi115.stdout:647: writing smithi11541303-12 from 491520 to 892928 tid 2 2022-04-22T17:27:32.222 INFO:tasks.rados.rados.0.smithi115.stdout:648: snap_remove snap 58 2022-04-22T17:27:33.206 INFO:tasks.rados.rados.0.smithi115.stdout:648: done (2 left) 2022-04-22T17:27:33.206 INFO:tasks.rados.rados.0.smithi115.stdout:649: snap_remove snap 52 2022-04-22T17:27:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[12737]: cluster 2022-04-22T17:27:32.188772+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2022-04-22T17:27:33.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[12737]: cluster 2022-04-22T17:27:33.029209+0000 mgr.y (mgr.24919) 15 : cluster [DBG] pgmap v25: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[18070]: cluster 2022-04-22T17:27:32.188772+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2022-04-22T17:27:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[18070]: cluster 2022-04-22T17:27:33.029209+0000 mgr.y (mgr.24919) 15 : cluster [DBG] pgmap v25: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:33 smithi121 bash[12679]: cluster 2022-04-22T17:27:32.188772+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2022-04-22T17:27:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:33 smithi121 bash[12679]: cluster 2022-04-22T17:27:33.029209+0000 mgr.y (mgr.24919) 15 : cluster [DBG] pgmap v25: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:33.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:33.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:649: done (2 left) 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 60 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:647: finishing write tid 1 to smithi11541303-12 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:647: finishing write tid 2 to smithi11541303-12 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:647: finishing write tid 3 to smithi11541303-12 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:647: finishing write tid 4 to smithi11541303-12 2022-04-22T17:27:34.209 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 177 (ObjNum 196 snap 60 seq_num 196) dirty exists 2022-04-22T17:27:34.210 INFO:tasks.rados.rados.0.smithi115.stdout:647: left oid 12 (ObjNum 196 snap 60 seq_num 196) 2022-04-22T17:27:34.210 INFO:tasks.rados.rados.0.smithi115.stdout:646: expect (ObjNum 131 snap 34 seq_num 131) 2022-04-22T17:27:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[18070]: cluster 2022-04-22T17:27:33.196945+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2022-04-22T17:27:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[18070]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[18070]: audit 2022-04-22T17:27:33.781613+0000 mon.a (mon.0) 980 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[18070]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:27:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[18070]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[12737]: cluster 2022-04-22T17:27:33.196945+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2022-04-22T17:27:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[12737]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[12737]: audit 2022-04-22T17:27:33.781613+0000 mon.a (mon.0) 980 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[12737]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:27:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:34 smithi115 bash[12737]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:34.346 INFO:tasks.rados.rados.0.smithi115.stdout:650: seq_num 197 ranges {761856=507904,1269760=401408} 2022-04-22T17:27:34.358 INFO:tasks.rados.rados.0.smithi115.stdout:650: writing smithi11541303-1 from 761856 to 1269760 tid 1 2022-04-22T17:27:34.369 INFO:tasks.rados.rados.0.smithi115.stdout:650: writing smithi11541303-1 from 1269760 to 1671168 tid 2 2022-04-22T17:27:34.369 INFO:tasks.rados.rados.0.smithi115.stdout:646: done (2 left) 2022-04-22T17:27:34.369 INFO:tasks.rados.rados.0.smithi115.stdout:647: done (1 left) 2022-04-22T17:27:34.370 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 60 2022-04-22T17:27:34.370 INFO:tasks.rados.rados.0.smithi115.stdout:651: seq_num 198 ranges {3506176=524288,4030464=401408} 2022-04-22T17:27:34.385 INFO:tasks.rados.rados.0.smithi115.stdout:651: writing smithi11541303-24 from 3506176 to 4030464 tid 1 2022-04-22T17:27:34.388 INFO:tasks.rados.rados.0.smithi115.stdout:651: writing smithi11541303-24 from 4030464 to 4431872 tid 2 2022-04-22T17:27:34.388 INFO:tasks.rados.rados.0.smithi115.stdout:652: snap_create 2022-04-22T17:27:34.388 INFO:tasks.rados.rados.0.smithi115.stdout:650: finishing write tid 1 to smithi11541303-1 2022-04-22T17:27:34.388 INFO:tasks.rados.rados.0.smithi115.stdout:650: finishing write tid 2 to smithi11541303-1 2022-04-22T17:27:34.389 INFO:tasks.rados.rados.0.smithi115.stdout:650: finishing write tid 3 to smithi11541303-1 2022-04-22T17:27:34.389 INFO:tasks.rados.rados.0.smithi115.stdout:650: finishing write tid 4 to smithi11541303-1 2022-04-22T17:27:34.389 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 208 (ObjNum 197 snap 60 seq_num 197) dirty exists 2022-04-22T17:27:34.389 INFO:tasks.rados.rados.0.smithi115.stdout:650: left oid 1 (ObjNum 197 snap 60 seq_num 197) 2022-04-22T17:27:34.392 INFO:tasks.rados.rados.0.smithi115.stdout:651: finishing write tid 1 to smithi11541303-24 2022-04-22T17:27:34.393 INFO:tasks.rados.rados.0.smithi115.stdout:651: finishing write tid 2 to smithi11541303-24 2022-04-22T17:27:34.393 INFO:tasks.rados.rados.0.smithi115.stdout:651: finishing write tid 3 to smithi11541303-24 2022-04-22T17:27:34.395 INFO:tasks.rados.rados.0.smithi115.stdout:651: finishing write tid 4 to smithi11541303-24 2022-04-22T17:27:34.395 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 66 (ObjNum 198 snap 60 seq_num 198) dirty exists 2022-04-22T17:27:34.395 INFO:tasks.rados.rados.0.smithi115.stdout:651: left oid 24 (ObjNum 198 snap 60 seq_num 198) 2022-04-22T17:27:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:34 smithi121 bash[12679]: cluster 2022-04-22T17:27:33.196945+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2022-04-22T17:27:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:34 smithi121 bash[12679]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:27:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:34 smithi121 bash[12679]: audit 2022-04-22T17:27:33.781613+0000 mon.a (mon.0) 980 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:34 smithi121 bash[12679]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:27:34.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:34 smithi121 bash[12679]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:27:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:650: done (2 left) 2022-04-22T17:27:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:651: done (1 left) 2022-04-22T17:27:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:652: done (0 left) 2022-04-22T17:27:35.301 INFO:tasks.rados.rados.0.smithi115.stdout:653: rollback oid 13 current snap is 61 2022-04-22T17:27:35.301 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 59 2022-04-22T17:27:35.301 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 61 2022-04-22T17:27:35.301 INFO:tasks.rados.rados.0.smithi115.stdout:654: seq_num 199 ranges {0=540672,540672=401408} 2022-04-22T17:27:35.303 INFO:tasks.rados.rados.0.smithi115.stdout:654: writing smithi11541303-35 from 0 to 540672 tid 1 2022-04-22T17:27:35.306 INFO:tasks.rados.rados.0.smithi115.stdout:654: writing smithi11541303-35 from 540672 to 942080 tid 2 2022-04-22T17:27:35.306 INFO:tasks.rados.rados.0.smithi115.stdout:655: read oid 1 snap -1 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:655: expect (ObjNum 197 snap 60 seq_num 197) 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:656: rollback oid 26 current snap is 61 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 54 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:657: rollback oid 5 current snap is 61 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 60 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:658: setattr oid 22 current snap is 61 2022-04-22T17:27:35.307 INFO:tasks.rados.rados.0.smithi115.stdout:653: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:27:35.308 INFO:tasks.rados.rados.0.smithi115.stdout:659: delete oid 39 current snap is 61 2022-04-22T17:27:35.310 INFO:tasks.rados.rados.0.smithi115.stdout:653: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:27:35.310 INFO:tasks.rados.rados.0.smithi115.stdout:654: finishing write tid 1 to smithi11541303-35 2022-04-22T17:27:35.311 INFO:tasks.rados.rados.0.smithi115.stdout:657: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:27:35.311 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 311 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:27:35.312 INFO:tasks.rados.rados.0.smithi115.stdout:653: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:27:35.312 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 247 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:27:35.313 INFO:tasks.rados.rados.0.smithi115.stdout:654: finishing write tid 2 to smithi11541303-35 2022-04-22T17:27:35.314 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 135 (ObjNum -1116086464 snap 22075 seq_num 3178807104) dirty exists 2022-04-22T17:27:35.315 INFO:tasks.rados.rados.0.smithi115.stdout:654: finishing write tid 3 to smithi11541303-35 2022-04-22T17:27:35.316 INFO:tasks.rados.rados.0.smithi115.stdout:656: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:27:35.317 INFO:tasks.rados.rados.0.smithi115.stdout:656: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:27:35.317 INFO:tasks.rados.rados.0.smithi115.stdout:653: done (6 left) 2022-04-22T17:27:35.317 INFO:tasks.rados.rados.0.smithi115.stdout:657: done (5 left) 2022-04-22T17:27:35.317 INFO:tasks.rados.rados.0.smithi115.stdout:658: done (4 left) 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:659: done (3 left) 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:660: delete oid 13 current snap is 61 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:654: finishing write tid 4 to smithi11541303-35 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 154 (ObjNum 199 snap 61 seq_num 199) dirty exists 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:654: left oid 35 (ObjNum 199 snap 61 seq_num 199) 2022-04-22T17:27:35.318 INFO:tasks.rados.rados.0.smithi115.stdout:656: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:27:35.319 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 156 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:35.319 INFO:tasks.rados.rados.0.smithi115.stdout:655: expect (ObjNum 197 snap 60 seq_num 197) 2022-04-22T17:27:35.380 INFO:tasks.rados.rados.0.smithi115.stdout:654: done (3 left) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:655: done (2 left) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:656: done (1 left) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:660: done (0 left) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:661: rmattr oid 13 current snap is 61 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:661: done (0 left) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:662: read oid 1 snap -1 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:662: expect (ObjNum 197 snap 60 seq_num 197) 2022-04-22T17:27:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:663: delete oid 8 current snap is 61 2022-04-22T17:27:35.390 INFO:tasks.rados.rados.0.smithi115.stdout:662: expect (ObjNum 197 snap 60 seq_num 197) 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:662: done (1 left) 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:663: done (0 left) 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:664: rollback oid 9 current snap is 61 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 42 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:665: read oid 6 snap -1 2022-04-22T17:27:35.457 INFO:tasks.rados.rados.0.smithi115.stdout:665: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:35.458 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 61 2022-04-22T17:27:35.458 INFO:tasks.rados.rados.0.smithi115.stdout:666: seq_num 203 ranges {3506176=606208,4112384=401408} 2022-04-22T17:27:35.486 INFO:tasks.rados.rados.0.smithi115.stdout:666: writing smithi11541303-32 from 3506176 to 4112384 tid 1 2022-04-22T17:27:35.488 INFO:tasks.rados.rados.0.smithi115.stdout:666: writing smithi11541303-32 from 4112384 to 4513792 tid 2 2022-04-22T17:27:35.488 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 61 2022-04-22T17:27:35.488 INFO:tasks.rados.rados.0.smithi115.stdout:667: seq_num 204 ranges {9134080=622592,9756672=401408} 2022-04-22T17:27:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:35 smithi121 bash[12679]: cluster 2022-04-22T17:27:34.197125+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2022-04-22T17:27:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:35 smithi121 bash[12679]: audit 2022-04-22T17:27:34.274230+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:35 smithi121 bash[12679]: audit 2022-04-22T17:27:34.279887+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:35 smithi121 bash[12679]: audit 2022-04-22T17:27:34.287032+0000 mon.a (mon.0) 984 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:35.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:35 smithi121 bash[12679]: cluster 2022-04-22T17:27:35.029878+0000 mgr.y (mgr.24919) 19 : cluster [DBG] pgmap v28: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:35.543 INFO:tasks.rados.rados.0.smithi115.stdout:667: writing smithi11541303-4 from 9134080 to 9756672 tid 1 2022-04-22T17:27:35.545 INFO:tasks.rados.rados.0.smithi115.stdout:667: writing smithi11541303-4 from 9756672 to 10158080 tid 2 2022-04-22T17:27:35.546 INFO:tasks.rados.rados.0.smithi115.stdout:668: snap_create 2022-04-22T17:27:35.546 INFO:tasks.rados.rados.0.smithi115.stdout:665: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[12737]: cluster 2022-04-22T17:27:34.197125+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[12737]: audit 2022-04-22T17:27:34.274230+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[12737]: audit 2022-04-22T17:27:34.279887+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[12737]: audit 2022-04-22T17:27:34.287032+0000 mon.a (mon.0) 984 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[12737]: cluster 2022-04-22T17:27:35.029878+0000 mgr.y (mgr.24919) 19 : cluster [DBG] pgmap v28: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[18070]: cluster 2022-04-22T17:27:34.197125+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[18070]: audit 2022-04-22T17:27:34.274230+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[18070]: audit 2022-04-22T17:27:34.279887+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[18070]: audit 2022-04-22T17:27:34.287032+0000 mon.a (mon.0) 984 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:35 smithi115 bash[18070]: cluster 2022-04-22T17:27:35.029878+0000 mgr.y (mgr.24919) 19 : cluster [DBG] pgmap v28: 121 pgs: 121 active+clean; 127 MiB data, 453 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:664: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:664: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:664: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 350 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty exists 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:666: finishing write tid 1 to smithi11541303-32 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:666: finishing write tid 2 to smithi11541303-32 2022-04-22T17:27:35.549 INFO:tasks.rados.rados.0.smithi115.stdout:666: finishing write tid 3 to smithi11541303-32 2022-04-22T17:27:35.550 INFO:tasks.rados.rados.0.smithi115.stdout:666: finishing write tid 4 to smithi11541303-32 2022-04-22T17:27:35.550 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 160 (ObjNum 203 snap 61 seq_num 203) dirty exists 2022-04-22T17:27:35.550 INFO:tasks.rados.rados.0.smithi115.stdout:666: left oid 32 (ObjNum 203 snap 61 seq_num 203) 2022-04-22T17:27:35.553 INFO:tasks.rados.rados.0.smithi115.stdout:667: finishing write tid 1 to smithi11541303-4 2022-04-22T17:27:35.554 INFO:tasks.rados.rados.0.smithi115.stdout:667: finishing write tid 2 to smithi11541303-4 2022-04-22T17:27:35.554 INFO:tasks.rados.rados.0.smithi115.stdout:667: finishing write tid 3 to smithi11541303-4 2022-04-22T17:27:35.556 INFO:tasks.rados.rados.0.smithi115.stdout:667: finishing write tid 4 to smithi11541303-4 2022-04-22T17:27:35.556 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 287 (ObjNum 204 snap 61 seq_num 204) dirty exists 2022-04-22T17:27:35.557 INFO:tasks.rados.rados.0.smithi115.stdout:667: left oid 4 (ObjNum 204 snap 61 seq_num 204) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:664: done (4 left) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:665: done (3 left) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:666: done (2 left) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:667: done (1 left) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:668: done (0 left) 2022-04-22T17:27:36.201 INFO:tasks.rados.rados.0.smithi115.stdout:669: delete oid 35 current snap is 62 2022-04-22T17:27:36.206 INFO:tasks.rados.rados.0.smithi115.stdout:669: done (0 left) 2022-04-22T17:27:36.206 INFO:tasks.rados.rados.0.smithi115.stdout:670: snap_create 2022-04-22T17:27:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:36 smithi121 bash[12679]: cluster 2022-04-22T17:27:35.290929+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2022-04-22T17:27:36.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:36 smithi121 bash[12679]: cluster 2022-04-22T17:27:36.191201+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2022-04-22T17:27:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:36 smithi115 bash[12737]: cluster 2022-04-22T17:27:35.290929+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2022-04-22T17:27:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:36 smithi115 bash[12737]: cluster 2022-04-22T17:27:36.191201+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2022-04-22T17:27:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:36 smithi115 bash[18070]: cluster 2022-04-22T17:27:35.290929+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2022-04-22T17:27:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:36 smithi115 bash[18070]: cluster 2022-04-22T17:27:36.191201+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2022-04-22T17:27:37.212 INFO:tasks.rados.rados.0.smithi115.stdout:670: done (0 left) 2022-04-22T17:27:37.212 INFO:tasks.rados.rados.0.smithi115.stdout:671: snap_create 2022-04-22T17:27:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: audit 2022-04-22T17:27:36.887655+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: audit 2022-04-22T17:27:36.889870+0000 mon.a (mon.0) 988 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: audit 2022-04-22T17:27:36.891606+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: cluster 2022-04-22T17:27:37.030629+0000 mgr.y (mgr.24919) 22 : cluster [DBG] pgmap v31: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:37.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:37 smithi121 bash[12679]: cluster 2022-04-22T17:27:37.202185+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2022-04-22T17:27:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:27:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: audit 2022-04-22T17:27:36.887655+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:27:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: audit 2022-04-22T17:27:36.889870+0000 mon.a (mon.0) 988 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:27:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: audit 2022-04-22T17:27:36.891606+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: cluster 2022-04-22T17:27:37.030629+0000 mgr.y (mgr.24919) 22 : cluster [DBG] pgmap v31: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[12737]: cluster 2022-04-22T17:27:37.202185+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: audit 2022-04-22T17:27:36.887655+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: audit 2022-04-22T17:27:36.889870+0000 mon.a (mon.0) 988 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: audit 2022-04-22T17:27:36.891606+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:27:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: cluster 2022-04-22T17:27:37.030629+0000 mgr.y (mgr.24919) 22 : cluster [DBG] pgmap v31: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:37 smithi115 bash[18070]: cluster 2022-04-22T17:27:37.202185+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2022-04-22T17:27:38.216 INFO:tasks.rados.rados.0.smithi115.stdout:671: done (0 left) 2022-04-22T17:27:38.216 INFO:tasks.rados.rados.0.smithi115.stdout:672: read oid 22 snap -1 2022-04-22T17:27:38.217 INFO:tasks.rados.rados.0.smithi115.stdout:672: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:38.217 INFO:tasks.rados.rados.0.smithi115.stdout:673: copy_from oid 40 from oid 5 current snap is 64 2022-04-22T17:27:38.217 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 64 2022-04-22T17:27:38.217 INFO:tasks.rados.rados.0.smithi115.stdout:674: seq_num 206 ranges {2555904=655360,3211264=401408} 2022-04-22T17:27:38.238 INFO:tasks.rados.rados.0.smithi115.stdout:674: writing smithi11541303-20 from 2555904 to 3211264 tid 1 2022-04-22T17:27:38.240 INFO:tasks.rados.rados.0.smithi115.stdout:674: writing smithi11541303-20 from 3211264 to 3612672 tid 2 2022-04-22T17:27:38.240 INFO:tasks.rados.rados.0.smithi115.stdout:675: snap_remove snap 55 2022-04-22T17:27:39.220 INFO:tasks.rados.rados.0.smithi115.stdout:675: done (3 left) 2022-04-22T17:27:39.220 INFO:tasks.rados.rados.0.smithi115.stdout:676: delete oid 24 current snap is 64 2022-04-22T17:27:39.221 INFO:tasks.rados.rados.0.smithi115.stdout:673: finishing copy_from to smithi11541303-40 2022-04-22T17:27:39.221 INFO:tasks.rados.rados.0.smithi115.stdout:673: got expected ENOENT (src dne) 2022-04-22T17:27:39.221 INFO:tasks.rados.rados.0.smithi115.stdout:672: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:39.224 INFO:tasks.rados.rados.0.smithi115.stdout:673: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:27:39.224 INFO:tasks.rados.rados.0.smithi115.stdout:674: finishing write tid 1 to smithi11541303-20 2022-04-22T17:27:39.224 INFO:tasks.rados.rados.0.smithi115.stdout:674: finishing write tid 2 to smithi11541303-20 2022-04-22T17:27:39.224 INFO:tasks.rados.rados.0.smithi115.stdout:674: finishing write tid 3 to smithi11541303-20 2022-04-22T17:27:39.224 INFO:tasks.rados.rados.0.smithi115.stdout:674: finishing write tid 4 to smithi11541303-20 2022-04-22T17:27:39.225 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 252 (ObjNum 206 snap 64 seq_num 206) dirty exists 2022-04-22T17:27:39.225 INFO:tasks.rados.rados.0.smithi115.stdout:674: left oid 20 (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:27:39.236 INFO:tasks.rados.rados.0.smithi115.stdout:672: done (3 left) 2022-04-22T17:27:39.277 INFO:tasks.rados.rados.0.smithi115.stdout:673: done (2 left) 2022-04-22T17:27:39.277 INFO:tasks.rados.rados.0.smithi115.stdout:674: done (1 left) 2022-04-22T17:27:39.278 INFO:tasks.rados.rados.0.smithi115.stdout:676: done (0 left) 2022-04-22T17:27:39.278 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 64 2022-04-22T17:27:39.278 INFO:tasks.rados.rados.0.smithi115.stdout:677: seq_num 208 ranges {0=688128,688128=401408} 2022-04-22T17:27:39.279 INFO:tasks.rados.rados.0.smithi115.stdout:677: writing smithi11541303-24 from 0 to 688128 tid 1 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:677: writing smithi11541303-24 from 688128 to 1089536 tid 2 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:678: copy_from oid 49 from oid 22 current snap is 64 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:679: delete oid 35 current snap is 64 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:679: done (2 left) 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:680: read oid 11 snap -1 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:680: expect (ObjNum 192 snap 59 seq_num 192) 2022-04-22T17:27:39.280 INFO:tasks.rados.rados.0.smithi115.stdout:681: snap_remove snap 62 2022-04-22T17:27:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:39 smithi121 bash[12679]: cluster 2022-04-22T17:27:38.204404+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2022-04-22T17:27:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:39 smithi121 bash[12679]: cluster 2022-04-22T17:27:39.031440+0000 mgr.y (mgr.24919) 23 : cluster [DBG] pgmap v34: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:39.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:39 smithi115 bash[18070]: cluster 2022-04-22T17:27:38.204404+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2022-04-22T17:27:39.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:39 smithi115 bash[18070]: cluster 2022-04-22T17:27:39.031440+0000 mgr.y (mgr.24919) 23 : cluster [DBG] pgmap v34: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:39 smithi115 bash[12737]: cluster 2022-04-22T17:27:38.204404+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2022-04-22T17:27:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:39 smithi115 bash[12737]: cluster 2022-04-22T17:27:39.031440+0000 mgr.y (mgr.24919) 23 : cluster [DBG] pgmap v34: 121 pgs: 121 active+clean; 128 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 822 KiB/s wr, 4 op/s 2022-04-22T17:27:39.852 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:39 smithi121 systemd[1]: Stopping Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:27:40.233 INFO:tasks.rados.rados.0.smithi115.stdout:681: done (3 left) 2022-04-22T17:27:40.234 INFO:tasks.rados.rados.0.smithi115.stdout:682: snap_remove snap 42 2022-04-22T17:27:40.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:40 smithi121 bash[12679]: cluster 2022-04-22T17:27:39.210508+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2022-04-22T17:27:40.473 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 bash[34285]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr.x 2022-04-22T17:27:40.474 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2022-04-22T17:27:40.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:40 smithi115 bash[18070]: cluster 2022-04-22T17:27:39.210508+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2022-04-22T17:27:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:40 smithi115 bash[12737]: cluster 2022-04-22T17:27:39.210508+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2022-04-22T17:27:40.768 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 bash[34329]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr-x 2022-04-22T17:27:40.815 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 bash[34342]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mgr.x 2022-04-22T17:27:40.815 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2022-04-22T17:27:40.816 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 systemd[1]: Stopped Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:27:41.230 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:40 smithi121 systemd[1]: Started Ceph mgr.x for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:27:41.242 INFO:tasks.rados.rados.0.smithi115.stdout:682: done (3 left) 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:683: setattr oid 12 current snap is 64 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:677: finishing write tid 1 to smithi11541303-24 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:677: finishing write tid 2 to smithi11541303-24 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:677: finishing write tid 3 to smithi11541303-24 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:677: finishing write tid 4 to smithi11541303-24 2022-04-22T17:27:41.243 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 73 (ObjNum 208 snap 64 seq_num 208) dirty exists 2022-04-22T17:27:41.244 INFO:tasks.rados.rados.0.smithi115.stdout:677: left oid 24 (ObjNum 208 snap 64 seq_num 208) 2022-04-22T17:27:41.244 INFO:tasks.rados.rados.0.smithi115.stdout:678: finishing copy_from to smithi11541303-49 2022-04-22T17:27:41.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 135 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:27:41.244 INFO:tasks.rados.rados.0.smithi115.stdout:678: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:27:41.244 INFO:tasks.rados.rados.0.smithi115.stdout:680: expect (ObjNum 192 snap 59 seq_num 192) 2022-04-22T17:27:41.276 INFO:tasks.rados.rados.0.smithi115.stdout:677: done (3 left) 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:678: done (2 left) 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:680: done (1 left) 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:684: copy_from oid 15 from oid 10 current snap is 64 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:685: rollback oid 19 current snap is 64 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 63 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:686: read oid 36 snap -1 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:686: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:687: read oid 9 snap 54 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:687: expect (ObjNum -1116565472 snap 22075 seq_num 3178180048) 2022-04-22T17:27:41.277 INFO:tasks.rados.rados.0.smithi115.stdout:688: rollback oid 49 current snap is 64 2022-04-22T17:27:41.278 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 51 2022-04-22T17:27:41.278 INFO:tasks.rados.rados.0.smithi115.stdout:689: delete oid 24 current snap is 64 2022-04-22T17:27:41.278 INFO:tasks.rados.rados.0.smithi115.stdout:687: expect (ObjNum -1116565472 snap 22075 seq_num 3178180048) 2022-04-22T17:27:41.282 INFO:tasks.rados.rados.0.smithi115.stdout:686: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:688: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:688: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:688: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 377 (ObjNum 175 snap 49 seq_num 175) dirty exists 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 197 (ObjNum 196 snap 60 seq_num 196) dirty exists 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:685: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:27:41.286 INFO:tasks.rados.rados.0.smithi115.stdout:683: done (6 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:686: done (5 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:687: done (4 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:688: done (3 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:689: done (2 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:690: rmattr oid 47 current snap is 64 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:690: done (2 left) 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 64 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:685: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:27:41.287 INFO:tasks.rados.rados.0.smithi115.stdout:691: seq_num 211 ranges {2015232=737280,2752512=401408} 2022-04-22T17:27:41.304 INFO:tasks.rados.rados.0.smithi115.stdout:691: writing smithi11541303-38 from 2015232 to 2752512 tid 1 2022-04-22T17:27:41.306 INFO:tasks.rados.rados.0.smithi115.stdout:691: writing smithi11541303-38 from 2752512 to 3153920 tid 2 2022-04-22T17:27:41.306 INFO:tasks.rados.rados.0.smithi115.stdout:692: snap_remove snap 51 2022-04-22T17:27:41.516 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[15740]: debug 2022-04-22T17:27:41.235+0000 7feeed403700 -1 osd.4 pg_epoch: 188 pg[5.es0( v 186'164 (0'0,186'164] local-lis/les=63/64 n=9 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [4,5,0]p4(0) r=0 lpr=63 crt=186'164 lcod 186'162 mlcod 186'162 active+clean+snaptrim trimq=[52~1] ps=[60~1,64~2,68~2,6b~1,6e~1,76~1]] removing snap head 2022-04-22T17:27:41.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: cluster 2022-04-22T17:27:40.223801+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2022-04-22T17:27:41.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: audit 2022-04-22T17:27:40.825980+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:41.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: audit 2022-04-22T17:27:40.827936+0000 mon.a (mon.0) 995 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:41.602 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: audit 2022-04-22T17:27:40.829730+0000 mon.a (mon.0) 996 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:41.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: audit 2022-04-22T17:27:40.831000+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:41.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:41 smithi121 bash[12679]: cluster 2022-04-22T17:27:41.032371+0000 mgr.y (mgr.24919) 24 : cluster [DBG] pgmap v37: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:41.603 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[26675]: debug 2022-04-22T17:27:41.249+0000 7faa050b4700 -1 osd.2 pg_epoch: 188 pg[5.ds0( v 188'164 (0'0,188'164] local-lis/les=63/64 n=7 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 luod=186'162 crt=186'161 lcod 186'161 mlcod 186'161 active+clean+snaptrim trimq=[52~1] ps=[60~1,64~2,68~2,6b~1,6e~1,76~1]] removing snap head 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: cluster 2022-04-22T17:27:40.223801+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: audit 2022-04-22T17:27:40.825980+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: audit 2022-04-22T17:27:40.827936+0000 mon.a (mon.0) 995 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: audit 2022-04-22T17:27:40.829730+0000 mon.a (mon.0) 996 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: audit 2022-04-22T17:27:40.831000+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:41.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[18070]: cluster 2022-04-22T17:27:41.032371+0000 mgr.y (mgr.24919) 24 : cluster [DBG] pgmap v37: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: cluster 2022-04-22T17:27:40.223801+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: audit 2022-04-22T17:27:40.825980+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: audit 2022-04-22T17:27:40.827936+0000 mon.a (mon.0) 995 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: audit 2022-04-22T17:27:40.829730+0000 mon.a (mon.0) 996 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: audit 2022-04-22T17:27:40.831000+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:27:41.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:41 smithi115 bash[12737]: cluster 2022-04-22T17:27:41.032371+0000 mgr.y (mgr.24919) 24 : cluster [DBG] pgmap v37: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:692: done (3 left) 2022-04-22T17:27:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:693: copy_from oid 28 from oid 44 current snap is 64 2022-04-22T17:27:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:685: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:27:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 201 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:42.254 INFO:tasks.rados.rados.0.smithi115.stdout:684: finishing copy_from to smithi11541303-15 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 179 (ObjNum 176 snap 49 seq_num 176) dirty exists 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:684: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:691: finishing write tid 1 to smithi11541303-38 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:691: finishing write tid 2 to smithi11541303-38 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:691: finishing write tid 3 to smithi11541303-38 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:691: finishing write tid 4 to smithi11541303-38 2022-04-22T17:27:42.255 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 205 (ObjNum 211 snap 64 seq_num 211) dirty exists 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:691: left oid 38 (ObjNum 211 snap 64 seq_num 211) 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:684: done (3 left) 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:685: done (2 left) 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:691: done (1 left) 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:694: rollback oid 1 current snap is 64 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 1 to 49 2022-04-22T17:27:42.256 INFO:tasks.rados.rados.0.smithi115.stdout:695: setattr oid 6 current snap is 64 2022-04-22T17:27:42.257 INFO:tasks.rados.rados.0.smithi115.stdout:696: delete oid 37 current snap is 64 2022-04-22T17:27:42.266 INFO:tasks.rados.rados.0.smithi115.stdout:693: finishing copy_from to smithi11541303-28 2022-04-22T17:27:42.267 INFO:tasks.rados.rados.0.smithi115.stdout:693: got expected ENOENT (src dne) 2022-04-22T17:27:42.267 INFO:tasks.rados.rados.0.smithi115.stdout:693: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:27:42.267 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 82 (ObjNum 1 snap 32692 seq_num 3178655088) dirty exists 2022-04-22T17:27:42.272 INFO:tasks.rados.rados.0.smithi115.stdout:694: finishing rollback tid 0 to smithi11541303-1 2022-04-22T17:27:42.272 INFO:tasks.rados.rados.0.smithi115.stdout:694: finishing rollback tid 1 to smithi11541303-1 2022-04-22T17:27:42.272 INFO:tasks.rados.rados.0.smithi115.stdout:694: finishing rollback tid 2 to smithi11541303-1 2022-04-22T17:27:42.273 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 211 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:27:42.273 INFO:tasks.rados.rados.0.smithi115.stdout:693: done (3 left) 2022-04-22T17:27:42.273 INFO:tasks.rados.rados.0.smithi115.stdout:694: done (2 left) 2022-04-22T17:27:42.273 INFO:tasks.rados.rados.0.smithi115.stdout:695: done (1 left) 2022-04-22T17:27:42.273 INFO:tasks.rados.rados.0.smithi115.stdout:696: done (0 left) 2022-04-22T17:27:42.274 INFO:tasks.rados.rados.0.smithi115.stdout:697: snap_create 2022-04-22T17:27:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:42 smithi121 bash[12679]: cluster 2022-04-22T17:27:41.231972+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2022-04-22T17:27:42.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:42 smithi121 bash[34414]: debug 2022-04-22T17:27:42.387+0000 7f159276f700 1 -- 172.21.15.121:0/1619048320 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563bc47de340 con 0x563bc48d8c00 2022-04-22T17:27:42.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:42 smithi115 bash[12737]: cluster 2022-04-22T17:27:41.231972+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2022-04-22T17:27:42.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:42 smithi115 bash[18070]: cluster 2022-04-22T17:27:41.231972+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2022-04-22T17:27:43.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:42 smithi121 bash[34414]: debug 2022-04-22T17:27:42.559+0000 7f159b3e9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:27:43.262 INFO:tasks.rados.rados.0.smithi115.stdout:697: done (0 left) 2022-04-22T17:27:43.262 INFO:tasks.rados.rados.0.smithi115.stdout:698: delete oid 9 current snap is 65 2022-04-22T17:27:43.268 INFO:tasks.rados.rados.0.smithi115.stdout:698: done (0 left) 2022-04-22T17:27:43.269 INFO:tasks.rados.rados.0.smithi115.stdout:699: snap_create 2022-04-22T17:27:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[12679]: cluster 2022-04-22T17:27:42.243754+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2022-04-22T17:27:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[12679]: cluster 2022-04-22T17:27:43.033167+0000 mgr.y (mgr.24919) 25 : cluster [DBG] pgmap v40: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:43.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[34414]: debug 2022-04-22T17:27:43.267+0000 7f159b3e9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:27:43.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[18070]: cluster 2022-04-22T17:27:42.243754+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2022-04-22T17:27:43.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[18070]: cluster 2022-04-22T17:27:43.033167+0000 mgr.y (mgr.24919) 25 : cluster [DBG] pgmap v40: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:43.521 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[12737]: cluster 2022-04-22T17:27:42.243754+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2022-04-22T17:27:43.521 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[12737]: cluster 2022-04-22T17:27:43.033167+0000 mgr.y (mgr.24919) 25 : cluster [DBG] pgmap v40: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2022-04-22T17:27:43.522 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:27:43.522 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.510Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:43.522 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.510Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:27:43.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:43.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:27:43.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:43.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:27:43.845 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[34414]: debug 2022-04-22T17:27:43.583+0000 7f159b3e9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:27:43.874 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[34414]: debug 2022-04-22T17:27:43.727+0000 7f159b3e9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:27:44.121 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:43 smithi121 bash[34414]: debug 2022-04-22T17:27:43.835+0000 7f159b3e9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:27:44.270 INFO:tasks.rados.rados.0.smithi115.stdout:699: done (0 left) 2022-04-22T17:27:44.270 INFO:tasks.rados.rados.0.smithi115.stdout:700: rmattr oid 13 current snap is 66 2022-04-22T17:27:44.271 INFO:tasks.rados.rados.0.smithi115.stdout:700: done (0 left) 2022-04-22T17:27:44.271 INFO:tasks.rados.rados.0.smithi115.stdout:701: snap_create 2022-04-22T17:27:44.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[12679]: cluster 2022-04-22T17:27:43.252701+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2022-04-22T17:27:44.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[34414]: debug 2022-04-22T17:27:44.111+0000 7f159b3e9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:27:44.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[34414]: debug 2022-04-22T17:27:44.227+0000 7f159b3e9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:27:44.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[34414]: debug 2022-04-22T17:27:44.335+0000 7f159b3e9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:27:44.440 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[34414]: debug 2022-04-22T17:27:44.431+0000 7f159b3e9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:27:44.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:44 smithi115 bash[12737]: cluster 2022-04-22T17:27:43.252701+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2022-04-22T17:27:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:44 smithi115 bash[18070]: cluster 2022-04-22T17:27:43.252701+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2022-04-22T17:27:45.186 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:44 smithi121 bash[34414]: debug 2022-04-22T17:27:44.899+0000 7f159b3e9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:27:45.186 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[34414]: debug 2022-04-22T17:27:45.051+0000 7f159b3e9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:45.272 INFO:tasks.rados.rados.0.smithi115.stdout:701: done (0 left) 2022-04-22T17:27:45.273 INFO:tasks.rados.rados.0.smithi115.stdout:702: copy_from oid 43 from oid 39 current snap is 67 2022-04-22T17:27:45.273 INFO:tasks.rados.rados.0.smithi115.stdout:703: delete oid 49 current snap is 67 2022-04-22T17:27:45.278 INFO:tasks.rados.rados.0.smithi115.stdout:703: done (1 left) 2022-04-22T17:27:45.278 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 67 2022-04-22T17:27:45.278 INFO:tasks.rados.rados.0.smithi115.stdout:704: seq_num 215 ranges {7471104=409600,7880704=532480,8413184=270336} 2022-04-22T17:27:45.327 INFO:tasks.rados.rados.0.smithi115.stdout:704: writing smithi11541303-23 from 7471104 to 7880704 tid 1 2022-04-22T17:27:45.330 INFO:tasks.rados.rados.0.smithi115.stdout:704: writing smithi11541303-23 from 7880704 to 8413184 tid 2 2022-04-22T17:27:45.332 INFO:tasks.rados.rados.0.smithi115.stdout:704: writing smithi11541303-23 from 8413184 to 8683520 tid 3 2022-04-22T17:27:45.332 INFO:tasks.rados.rados.0.smithi115.stdout:705: rollback oid 3 current snap is 67 2022-04-22T17:27:45.332 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 54 2022-04-22T17:27:45.332 INFO:tasks.rados.rados.0.smithi115.stdout:702: finishing copy_from to smithi11541303-43 2022-04-22T17:27:45.332 INFO:tasks.rados.rados.0.smithi115.stdout:702: got expected ENOENT (src dne) 2022-04-22T17:27:45.333 INFO:tasks.rados.rados.0.smithi115.stdout:706: snap_create 2022-04-22T17:27:45.333 INFO:tasks.rados.rados.0.smithi115.stdout:702: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:27:45.339 INFO:tasks.rados.rados.0.smithi115.stdout:704: finishing write tid 1 to smithi11541303-23 2022-04-22T17:27:45.340 INFO:tasks.rados.rados.0.smithi115.stdout:704: finishing write tid 2 to smithi11541303-23 2022-04-22T17:27:45.342 INFO:tasks.rados.rados.0.smithi115.stdout:704: finishing write tid 3 to smithi11541303-23 2022-04-22T17:27:45.342 INFO:tasks.rados.rados.0.smithi115.stdout:704: finishing write tid 4 to smithi11541303-23 2022-04-22T17:27:45.343 INFO:tasks.rados.rados.0.smithi115.stdout:705: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:27:45.343 INFO:tasks.rados.rados.0.smithi115.stdout:705: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:27:45.344 INFO:tasks.rados.rados.0.smithi115.stdout:704: finishing write tid 5 to smithi11541303-23 2022-04-22T17:27:45.344 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 237 (ObjNum 215 snap 67 seq_num 215) dirty exists 2022-04-22T17:27:45.344 INFO:tasks.rados.rados.0.smithi115.stdout:704: left oid 23 (ObjNum 215 snap 67 seq_num 215) 2022-04-22T17:27:45.345 INFO:tasks.rados.rados.0.smithi115.stdout:705: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:27:45.345 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 241 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:27:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[12679]: cluster 2022-04-22T17:27:44.260570+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2022-04-22T17:27:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[12679]: cluster 2022-04-22T17:27:45.033892+0000 mgr.y (mgr.24919) 26 : cluster [DBG] pgmap v43: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[12679]: cluster 2022-04-22T17:27:45.262006+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2022-04-22T17:27:45.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[34414]: debug 2022-04-22T17:27:45.179+0000 7f159b3e9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:27:45.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[34414]: debug 2022-04-22T17:27:45.407+0000 7f159b3e9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[12737]: cluster 2022-04-22T17:27:44.260570+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[12737]: cluster 2022-04-22T17:27:45.033892+0000 mgr.y (mgr.24919) 26 : cluster [DBG] pgmap v43: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[12737]: cluster 2022-04-22T17:27:45.262006+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[18070]: cluster 2022-04-22T17:27:44.260570+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[18070]: cluster 2022-04-22T17:27:45.033892+0000 mgr.y (mgr.24919) 26 : cluster [DBG] pgmap v43: 121 pgs: 121 active+clean; 133 MiB data, 496 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:45 smithi115 bash[18070]: cluster 2022-04-22T17:27:45.262006+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2022-04-22T17:27:45.837 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[34414]: debug 2022-04-22T17:27:45.539+0000 7f159b3e9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:27:46.203 INFO:tasks.rados.rados.0.smithi115.stdout:702: done (3 left) 2022-04-22T17:27:46.203 INFO:tasks.rados.rados.0.smithi115.stdout:704: done (2 left) 2022-04-22T17:27:46.203 INFO:tasks.rados.rados.0.smithi115.stdout:705: done (1 left) 2022-04-22T17:27:46.203 INFO:tasks.rados.rados.0.smithi115.stdout:706: done (0 left) 2022-04-22T17:27:46.204 INFO:tasks.rados.rados.0.smithi115.stdout:707: read oid 42 snap -1 2022-04-22T17:27:46.204 INFO:tasks.rados.rados.0.smithi115.stdout:707: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:46.204 INFO:tasks.rados.rados.0.smithi115.stdout:708: delete oid 38 current snap is 68 2022-04-22T17:27:46.212 INFO:tasks.rados.rados.0.smithi115.stdout:708: done (1 left) 2022-04-22T17:27:46.212 INFO:tasks.rados.rados.0.smithi115.stdout:709: read oid 9 snap -1 2022-04-22T17:27:46.212 INFO:tasks.rados.rados.0.smithi115.stdout:709: expect deleted 2022-04-22T17:27:46.213 INFO:tasks.rados.rados.0.smithi115.stdout:710: read oid 40 snap 54 2022-04-22T17:27:46.213 INFO:tasks.rados.rados.0.smithi115.stdout:710: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:46.213 INFO:tasks.rados.rados.0.smithi115.stdout:711: snap_create 2022-04-22T17:27:46.219 INFO:tasks.rados.rados.0.smithi115.stdout:707: expect (ObjNum 38 snap 0 seq_num 38) 2022-04-22T17:27:46.246 INFO:tasks.rados.rados.0.smithi115.stdout:710: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:46.266 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:45 smithi121 bash[34414]: debug 2022-04-22T17:27:45.827+0000 7f159b3e9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:27:46.266 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:46 smithi121 bash[34414]: debug 2022-04-22T17:27:46.007+0000 7f159b3e9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:27:46.553 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:46 smithi121 bash[34414]: debug 2022-04-22T17:27:46.283+0000 7f159b3e9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:27:46.553 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:46 smithi121 bash[34414]: debug 2022-04-22T17:27:46.543+0000 7f159b3e9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:27:46.823 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:46 smithi121 bash[34414]: debug 2022-04-22T17:27:46.815+0000 7f159b3e9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:47.216 INFO:tasks.rados.rados.0.smithi115.stdout:707: done (3 left) 2022-04-22T17:27:47.216 INFO:tasks.rados.rados.0.smithi115.stdout:709: done (2 left) 2022-04-22T17:27:47.216 INFO:tasks.rados.rados.0.smithi115.stdout:710: done (1 left) 2022-04-22T17:27:47.216 INFO:tasks.rados.rados.0.smithi115.stdout:711: done (0 left) 2022-04-22T17:27:47.216 INFO:tasks.rados.rados.0.smithi115.stdout:712: snap_remove snap 64 2022-04-22T17:27:47.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:47 smithi121 bash[12679]: cluster 2022-04-22T17:27:46.192698+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2022-04-22T17:27:47.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:47 smithi121 bash[12679]: audit 2022-04-22T17:27:46.845814+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:47.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:47 smithi121 bash[12679]: cluster 2022-04-22T17:27:47.034662+0000 mgr.y (mgr.24919) 27 : cluster [DBG] pgmap v46: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:47.349 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:47 smithi121 bash[34414]: debug 2022-04-22T17:27:47.075+0000 7f159b3e9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:27:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[18070]: cluster 2022-04-22T17:27:46.192698+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2022-04-22T17:27:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[18070]: audit 2022-04-22T17:27:46.845814+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[18070]: cluster 2022-04-22T17:27:47.034662+0000 mgr.y (mgr.24919) 27 : cluster [DBG] pgmap v46: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[12737]: cluster 2022-04-22T17:27:46.192698+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2022-04-22T17:27:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[12737]: audit 2022-04-22T17:27:46.845814+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:47 smithi115 bash[12737]: cluster 2022-04-22T17:27:47.034662+0000 mgr.y (mgr.24919) 27 : cluster [DBG] pgmap v46: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:47.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:47 smithi121 bash[34414]: debug 2022-04-22T17:27:47.339+0000 7f159b3e9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:27:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:712: done (0 left) 2022-04-22T17:27:48.220 INFO:tasks.rados.rados.0.smithi115.stdout:713: delete oid 7 current snap is 69 2022-04-22T17:27:48.230 INFO:tasks.rados.rados.0.smithi115.stdout:713: done (0 left) 2022-04-22T17:27:48.231 INFO:tasks.rados.rados.0.smithi115.stdout:714: read oid 11 snap -1 2022-04-22T17:27:48.231 INFO:tasks.rados.rados.0.smithi115.stdout:714: expect (ObjNum 192 snap 59 seq_num 192) 2022-04-22T17:27:48.231 INFO:tasks.rados.rados.0.smithi115.stdout:715: snap_remove snap 49 2022-04-22T17:27:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:48 smithi121 bash[12679]: cluster 2022-04-22T17:27:47.205918+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2022-04-22T17:27:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:48 smithi121 bash[12679]: audit 2022-04-22T17:27:47.420226+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:48 smithi115 bash[18070]: cluster 2022-04-22T17:27:47.205918+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2022-04-22T17:27:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:48 smithi115 bash[18070]: audit 2022-04-22T17:27:47.420226+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:48 smithi115 bash[12737]: cluster 2022-04-22T17:27:47.205918+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2022-04-22T17:27:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:48 smithi115 bash[12737]: audit 2022-04-22T17:27:47.420226+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:49.234 INFO:tasks.rados.rados.0.smithi115.stdout:715: done (1 left) 2022-04-22T17:27:49.234 INFO:tasks.rados.rados.0.smithi115.stdout:716: read oid 12 snap -1 2022-04-22T17:27:49.235 INFO:tasks.rados.rados.0.smithi115.stdout:716: expect (ObjNum 196 snap 60 seq_num 196) 2022-04-22T17:27:49.235 INFO:tasks.rados.rados.0.smithi115.stdout:717: rmattr oid 22 current snap is 69 2022-04-22T17:27:49.235 INFO:tasks.rados.rados.0.smithi115.stdout:718: snap_create 2022-04-22T17:27:49.236 INFO:tasks.rados.rados.0.smithi115.stdout:714: expect (ObjNum 192 snap 59 seq_num 192) 2022-04-22T17:27:49.267 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 141 (ObjNum 0 snap 0 seq_num 3176620048) dirty exists 2022-04-22T17:27:49.267 INFO:tasks.rados.rados.0.smithi115.stdout:716: expect (ObjNum 196 snap 60 seq_num 196) 2022-04-22T17:27:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:49 smithi121 bash[12679]: cluster 2022-04-22T17:27:48.210231+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2022-04-22T17:27:49.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:49 smithi121 bash[12679]: cluster 2022-04-22T17:27:49.035429+0000 mgr.y (mgr.24919) 28 : cluster [DBG] pgmap v49: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:49.592 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:49 smithi121 bash[34414]: debug 2022-04-22T17:27:49.171+0000 7f159b3e9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:27:49.592 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:49 smithi121 bash[34414]: debug 2022-04-22T17:27:49.287+0000 7f159b3e9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:27:49.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:49 smithi115 bash[12737]: cluster 2022-04-22T17:27:48.210231+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2022-04-22T17:27:49.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:49 smithi115 bash[12737]: cluster 2022-04-22T17:27:49.035429+0000 mgr.y (mgr.24919) 28 : cluster [DBG] pgmap v49: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:49.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:49 smithi115 bash[18070]: cluster 2022-04-22T17:27:48.210231+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2022-04-22T17:27:49.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:49 smithi115 bash[18070]: cluster 2022-04-22T17:27:49.035429+0000 mgr.y (mgr.24919) 28 : cluster [DBG] pgmap v49: 121 pgs: 121 active+clean; 119 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 874 KiB/s wr, 16 op/s 2022-04-22T17:27:50.239 INFO:tasks.rados.rados.0.smithi115.stdout:714: done (3 left) 2022-04-22T17:27:50.239 INFO:tasks.rados.rados.0.smithi115.stdout:716: done (2 left) 2022-04-22T17:27:50.239 INFO:tasks.rados.rados.0.smithi115.stdout:717: done (1 left) 2022-04-22T17:27:50.239 INFO:tasks.rados.rados.0.smithi115.stdout:718: done (0 left) 2022-04-22T17:27:50.239 INFO:tasks.rados.rados.0.smithi115.stdout:719: read oid 24 snap -1 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:719: expect deleted 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:720: rollback oid 34 current snap is 70 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 67 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:721: copy_from oid 29 from oid 25 current snap is 70 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:722: read oid 17 snap -1 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:722: expect deleted 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:723: rollback oid 4 current snap is 70 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 59 2022-04-22T17:27:50.240 INFO:tasks.rados.rados.0.smithi115.stdout:724: snap_create 2022-04-22T17:27:50.248 INFO:tasks.rados.rados.0.smithi115.stdout:720: finishing rollback tid 0 to smithi11541303-34 2022-04-22T17:27:50.249 INFO:tasks.rados.rados.0.smithi115.stdout:721: finishing copy_from to smithi11541303-29 2022-04-22T17:27:50.249 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 225 (ObjNum -1116280704 snap 22075 seq_num 3176620048) dirty exists 2022-04-22T17:27:50.250 INFO:tasks.rados.rados.0.smithi115.stdout:723: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:27:50.250 INFO:tasks.rados.rados.0.smithi115.stdout:723: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:27:50.250 INFO:tasks.rados.rados.0.smithi115.stdout:721: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:27:50.250 INFO:tasks.rados.rados.0.smithi115.stdout:723: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:27:50.250 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 306 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:27:50.251 INFO:tasks.rados.rados.0.smithi115.stdout:720: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:27:50.251 INFO:tasks.rados.rados.0.smithi115.stdout:720: finishing rollback tid 2 to smithi11541303-34 2022-04-22T17:27:50.251 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 161 (ObjNum 187 snap 58 seq_num 187) dirty exists 2022-04-22T17:27:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:50 smithi121 bash[12679]: cluster 2022-04-22T17:27:49.224176+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2022-04-22T17:27:50.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:50 smithi115 bash[12737]: cluster 2022-04-22T17:27:49.224176+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2022-04-22T17:27:50.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:50 smithi115 bash[18070]: cluster 2022-04-22T17:27:49.224176+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2022-04-22T17:27:51.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:27:50 smithi121 bash[34414]: debug 2022-04-22T17:27:50.607+0000 7f159b3e9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:27:51.206 INFO:tasks.rados.rados.0.smithi115.stdout:719: done (5 left) 2022-04-22T17:27:51.206 INFO:tasks.rados.rados.0.smithi115.stdout:720: done (4 left) 2022-04-22T17:27:51.206 INFO:tasks.rados.rados.0.smithi115.stdout:721: done (3 left) 2022-04-22T17:27:51.206 INFO:tasks.rados.rados.0.smithi115.stdout:722: done (2 left) 2022-04-22T17:27:51.206 INFO:tasks.rados.rados.0.smithi115.stdout:723: done (1 left) 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:724: done (0 left) 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:725: copy_from oid 36 from oid 21 current snap is 71 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:726: read oid 12 snap -1 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:726: expect (ObjNum 196 snap 60 seq_num 196) 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:727: rmattr oid 13 current snap is 71 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:727: done (2 left) 2022-04-22T17:27:51.207 INFO:tasks.rados.rados.0.smithi115.stdout:728: setattr oid 42 current snap is 71 2022-04-22T17:27:51.208 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 71 2022-04-22T17:27:51.208 INFO:tasks.rados.rados.0.smithi115.stdout:729: seq_num 218 ranges {8683520=458752,9142272=581632,9723904=221184} 2022-04-22T17:27:51.266 INFO:tasks.rados.rados.0.smithi115.stdout:729: writing smithi11541303-23 from 8683520 to 9142272 tid 1 2022-04-22T17:27:51.270 INFO:tasks.rados.rados.0.smithi115.stdout:729: writing smithi11541303-23 from 9142272 to 9723904 tid 2 2022-04-22T17:27:51.272 INFO:tasks.rados.rados.0.smithi115.stdout:729: writing smithi11541303-23 from 9723904 to 9945088 tid 3 2022-04-22T17:27:51.272 INFO:tasks.rados.rados.0.smithi115.stdout:730: rollback oid 5 current snap is 71 2022-04-22T17:27:51.272 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 54 2022-04-22T17:27:51.272 INFO:tasks.rados.rados.0.smithi115.stdout:726: expect (ObjNum 196 snap 60 seq_num 196) 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 407 (ObjNum 38 snap 0 seq_num 38) dirty exists 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:725: finishing copy_from to smithi11541303-36 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 104 (ObjNum 153 snap 41 seq_num 153) dirty exists 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:725: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:729: finishing write tid 1 to smithi11541303-23 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:730: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:27:51.303 INFO:tasks.rados.rados.0.smithi115.stdout:730: finishing rollback tid 2 to smithi11541303-5 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 409 (ObjNum 147 snap 39 seq_num 147) dirty exists 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:729: finishing write tid 2 to smithi11541303-23 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:729: finishing write tid 3 to smithi11541303-23 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:729: finishing write tid 4 to smithi11541303-23 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:729: finishing write tid 5 to smithi11541303-23 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 256 (ObjNum 218 snap 71 seq_num 218) dirty exists 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:729: left oid 23 (ObjNum 218 snap 71 seq_num 218) 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:725: done (4 left) 2022-04-22T17:27:51.304 INFO:tasks.rados.rados.0.smithi115.stdout:726: done (3 left) 2022-04-22T17:27:51.305 INFO:tasks.rados.rados.0.smithi115.stdout:728: done (2 left) 2022-04-22T17:27:51.305 INFO:tasks.rados.rados.0.smithi115.stdout:729: done (1 left) 2022-04-22T17:27:51.305 INFO:tasks.rados.rados.0.smithi115.stdout:730: done (0 left) 2022-04-22T17:27:51.305 INFO:tasks.rados.rados.0.smithi115.stdout:731: snap_remove snap 67 2022-04-22T17:27:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: cluster 2022-04-22T17:27:50.228207+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: cluster 2022-04-22T17:27:50.627110+0000 mon.a (mon.0) 1010 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: cluster 2022-04-22T17:27:50.627367+0000 mon.a (mon.0) 1011 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: audit 2022-04-22T17:27:50.636203+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: audit 2022-04-22T17:27:50.637091+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: audit 2022-04-22T17:27:50.638675+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:51.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: audit 2022-04-22T17:27:50.639226+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:51.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: cluster 2022-04-22T17:27:51.036369+0000 mgr.y (mgr.24919) 29 : cluster [DBG] pgmap v52: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:51.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:51 smithi121 bash[12679]: cluster 2022-04-22T17:27:51.195430+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2022-04-22T17:27:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: cluster 2022-04-22T17:27:50.228207+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: cluster 2022-04-22T17:27:50.627110+0000 mon.a (mon.0) 1010 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: cluster 2022-04-22T17:27:50.627367+0000 mon.a (mon.0) 1011 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: audit 2022-04-22T17:27:50.636203+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: audit 2022-04-22T17:27:50.637091+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: audit 2022-04-22T17:27:50.638675+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: audit 2022-04-22T17:27:50.639226+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: cluster 2022-04-22T17:27:51.036369+0000 mgr.y (mgr.24919) 29 : cluster [DBG] pgmap v52: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[12737]: cluster 2022-04-22T17:27:51.195430+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: cluster 2022-04-22T17:27:50.228207+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: cluster 2022-04-22T17:27:50.627110+0000 mon.a (mon.0) 1010 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: cluster 2022-04-22T17:27:50.627367+0000 mon.a (mon.0) 1011 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: audit 2022-04-22T17:27:50.636203+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: audit 2022-04-22T17:27:50.637091+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: audit 2022-04-22T17:27:50.638675+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: audit 2022-04-22T17:27:50.639226+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.121:0/41612945' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: cluster 2022-04-22T17:27:51.036369+0000 mgr.y (mgr.24919) 29 : cluster [DBG] pgmap v52: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:51 smithi115 bash[18070]: cluster 2022-04-22T17:27:51.195430+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2022-04-22T17:27:52.276 INFO:tasks.rados.rados.0.smithi115.stdout:731: done (0 left) 2022-04-22T17:27:52.276 INFO:tasks.rados.rados.0.smithi115.stdout:732: copy_from oid 42 from oid 36 current snap is 71 2022-04-22T17:27:52.276 INFO:tasks.rados.rados.0.smithi115.stdout:733: copy_from oid 46 from oid 16 current snap is 71 2022-04-22T17:27:52.276 INFO:tasks.rados.rados.0.smithi115.stdout:734: delete oid 28 current snap is 71 2022-04-22T17:27:52.283 INFO:tasks.rados.rados.0.smithi115.stdout:734: done (2 left) 2022-04-22T17:27:52.284 INFO:tasks.rados.rados.0.smithi115.stdout:735: snap_remove snap 60 2022-04-22T17:27:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:52 smithi115 bash[12737]: cluster 2022-04-22T17:27:51.262671+0000 mon.a (mon.0) 1013 : cluster [DBG] mgrmap e25: y(active, since 33s), standbys: x 2022-04-22T17:27:52.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:52 smithi115 bash[18070]: cluster 2022-04-22T17:27:51.262671+0000 mon.a (mon.0) 1013 : cluster [DBG] mgrmap e25: y(active, since 33s), standbys: x 2022-04-22T17:27:52.615 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:52 smithi121 bash[12679]: cluster 2022-04-22T17:27:51.262671+0000 mon.a (mon.0) 1013 : cluster [DBG] mgrmap e25: y(active, since 33s), standbys: x 2022-04-22T17:27:53.291 INFO:tasks.rados.rados.0.smithi115.stdout:735: done (2 left) 2022-04-22T17:27:53.291 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 71 2022-04-22T17:27:53.291 INFO:tasks.rados.rados.0.smithi115.stdout:736: seq_num 220 ranges {819200=491520,1310720=753664,2064384=49152} 2022-04-22T17:27:53.302 INFO:tasks.rados.rados.0.smithi115.stdout:736: writing smithi11541303-11 from 819200 to 1310720 tid 1 2022-04-22T17:27:53.306 INFO:tasks.rados.rados.0.smithi115.stdout:736: writing smithi11541303-11 from 1310720 to 2064384 tid 2 2022-04-22T17:27:53.307 INFO:tasks.rados.rados.0.smithi115.stdout:736: writing smithi11541303-11 from 2064384 to 2113536 tid 3 2022-04-22T17:27:53.307 INFO:tasks.rados.rados.0.smithi115.stdout:737: delete oid 21 current snap is 71 2022-04-22T17:27:53.307 INFO:tasks.rados.rados.0.smithi115.stdout:733: finishing copy_from to smithi11541303-46 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:733: got expected ENOENT (src dne) 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:733: finishing copy_from racing read to smithi11541303-46 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:732: finishing copy_from to smithi11541303-42 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 104 (ObjNum 153 snap 41 seq_num 153) dirty exists 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:732: finishing copy_from racing read to smithi11541303-42 2022-04-22T17:27:53.308 INFO:tasks.rados.rados.0.smithi115.stdout:736: finishing write tid 1 to smithi11541303-11 2022-04-22T17:27:53.311 INFO:tasks.rados.rados.0.smithi115.stdout:732: done (3 left) 2022-04-22T17:27:53.311 INFO:tasks.rados.rados.0.smithi115.stdout:733: done (2 left) 2022-04-22T17:27:53.311 INFO:tasks.rados.rados.0.smithi115.stdout:737: done (1 left) 2022-04-22T17:27:53.311 INFO:tasks.rados.rados.0.smithi115.stdout:738: snap_remove snap 70 2022-04-22T17:27:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[18070]: cluster 2022-04-22T17:27:52.261045+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2022-04-22T17:27:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[18070]: cluster 2022-04-22T17:27:53.037167+0000 mgr.y (mgr.24919) 30 : cluster [DBG] pgmap v55: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[12737]: cluster 2022-04-22T17:27:52.261045+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2022-04-22T17:27:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[12737]: cluster 2022-04-22T17:27:53.037167+0000 mgr.y (mgr.24919) 30 : cluster [DBG] pgmap v55: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:53.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.499Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.499Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.499Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:27:53.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:27:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:27:53.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:27:53.695 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:53 smithi121 bash[12679]: cluster 2022-04-22T17:27:52.261045+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2022-04-22T17:27:53.695 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:53 smithi121 bash[12679]: cluster 2022-04-22T17:27:53.037167+0000 mgr.y (mgr.24919) 30 : cluster [DBG] pgmap v55: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 296 KiB/s wr, 8 op/s 2022-04-22T17:27:54.299 INFO:tasks.rados.rados.0.smithi115.stdout:738: done (1 left) 2022-04-22T17:27:54.299 INFO:tasks.rados.rados.0.smithi115.stdout:739: read oid 9 snap -1 2022-04-22T17:27:54.299 INFO:tasks.rados.rados.0.smithi115.stdout:739: expect deleted 2022-04-22T17:27:54.299 INFO:tasks.rados.rados.0.smithi115.stdout:740: snap_remove snap 65 2022-04-22T17:27:54.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:54 smithi115 bash[12737]: cluster 2022-04-22T17:27:53.280529+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2022-04-22T17:27:54.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:54 smithi115 bash[18070]: cluster 2022-04-22T17:27:53.280529+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2022-04-22T17:27:54.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:54 smithi121 bash[12679]: cluster 2022-04-22T17:27:53.280529+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2022-04-22T17:27:55.313 INFO:tasks.rados.rados.0.smithi115.stdout:740: done (2 left) 2022-04-22T17:27:55.313 INFO:tasks.rados.rados.0.smithi115.stdout:741: read oid 47 snap -1 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:741: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:742: rollback oid 33 current snap is 71 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 59 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:736: finishing write tid 2 to smithi11541303-11 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:736: finishing write tid 3 to smithi11541303-11 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:736: finishing write tid 4 to smithi11541303-11 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:736: finishing write tid 5 to smithi11541303-11 2022-04-22T17:27:55.314 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 267 (ObjNum 220 snap 71 seq_num 220) dirty exists 2022-04-22T17:27:55.315 INFO:tasks.rados.rados.0.smithi115.stdout:736: left oid 11 (ObjNum 220 snap 71 seq_num 220) 2022-04-22T17:27:55.315 INFO:tasks.rados.rados.0.smithi115.stdout:736: done (3 left) 2022-04-22T17:27:55.315 INFO:tasks.rados.rados.0.smithi115.stdout:739: done (2 left) 2022-04-22T17:27:55.315 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 71 2022-04-22T17:27:55.315 INFO:tasks.rados.rados.0.smithi115.stdout:743: seq_num 222 ranges {0=524288,524288=565248,1089536=237568} 2022-04-22T17:27:55.316 INFO:tasks.rados.rados.0.smithi115.stdout:743: writing smithi11541303-43 from 0 to 524288 tid 1 2022-04-22T17:27:55.320 INFO:tasks.rados.rados.0.smithi115.stdout:743: writing smithi11541303-43 from 524288 to 1089536 tid 2 2022-04-22T17:27:55.323 INFO:tasks.rados.rados.0.smithi115.stdout:743: writing smithi11541303-43 from 1089536 to 1327104 tid 3 2022-04-22T17:27:55.323 INFO:tasks.rados.rados.0.smithi115.stdout:744: read oid 24 snap -1 2022-04-22T17:27:55.323 INFO:tasks.rados.rados.0.smithi115.stdout:744: expect deleted 2022-04-22T17:27:55.324 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 71 2022-04-22T17:27:55.324 INFO:tasks.rados.rados.0.smithi115.stdout:742: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:27:55.324 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 0 (ObjNum 0 snap 0 seq_num 3176620048) dirty dne 2022-04-22T17:27:55.324 INFO:tasks.rados.rados.0.smithi115.stdout:745: seq_num 223 ranges {892928=540672,1433600=450560,1884160=352256} 2022-04-22T17:27:55.333 INFO:tasks.rados.rados.0.smithi115.stdout:745: writing smithi11541303-12 from 892928 to 1433600 tid 1 2022-04-22T17:27:55.336 INFO:tasks.rados.rados.0.smithi115.stdout:745: writing smithi11541303-12 from 1433600 to 1884160 tid 2 2022-04-22T17:27:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:745: writing smithi11541303-12 from 1884160 to 2236416 tid 3 2022-04-22T17:27:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:742: done (4 left) 2022-04-22T17:27:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:746: rollback oid 50 current snap is 71 2022-04-22T17:27:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 63 2022-04-22T17:27:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:741: expect (ObjNum 16 snap 0 seq_num 16) 2022-04-22T17:27:55.349 INFO:tasks.rados.rados.0.smithi115.stdout:743: finishing write tid 1 to smithi11541303-43 2022-04-22T17:27:55.349 INFO:tasks.rados.rados.0.smithi115.stdout:743: finishing write tid 2 to smithi11541303-43 2022-04-22T17:27:55.349 INFO:tasks.rados.rados.0.smithi115.stdout:743: finishing write tid 3 to smithi11541303-43 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:743: finishing write tid 4 to smithi11541303-43 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:743: finishing write tid 5 to smithi11541303-43 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 292 (ObjNum 222 snap 71 seq_num 222) dirty exists 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:743: left oid 43 (ObjNum 222 snap 71 seq_num 222) 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:741: done (4 left) 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:743: done (3 left) 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:744: done (2 left) 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:747: read oid 18 snap -1 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:747: expect (ObjNum 89 snap 15 seq_num 89) 2022-04-22T17:27:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:748: copy_from oid 29 from oid 38 current snap is 71 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: finishing write tid 1 to smithi11541303-12 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: finishing write tid 2 to smithi11541303-12 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: finishing write tid 3 to smithi11541303-12 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: finishing write tid 4 to smithi11541303-12 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: finishing write tid 5 to smithi11541303-12 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 236 (ObjNum 223 snap 71 seq_num 223) dirty exists 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:745: left oid 12 (ObjNum 223 snap 71 seq_num 223) 2022-04-22T17:27:55.351 INFO:tasks.rados.rados.0.smithi115.stdout:746: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:746: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:746: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 171 (ObjNum 90 snap 15 seq_num 90) dirty exists 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:745: done (3 left) 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:746: done (2 left) 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:749: rmattr oid 28 current snap is 71 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:749: done (2 left) 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:750: rollback oid 8 current snap is 71 2022-04-22T17:27:55.352 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 66 2022-04-22T17:27:55.353 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 71 2022-04-22T17:27:55.353 INFO:tasks.rados.rados.0.smithi115.stdout:751: seq_num 224 ranges {2441216=557056,2998272=737280,3735552=65536} 2022-04-22T17:27:55.365 INFO:tasks.rados.rados.0.smithi115.stdout:751: writing smithi11541303-10 from 2441216 to 2998272 tid 1 2022-04-22T17:27:55.369 INFO:tasks.rados.rados.0.smithi115.stdout:751: writing smithi11541303-10 from 2998272 to 3735552 tid 2 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:751: writing smithi11541303-10 from 3735552 to 3801088 tid 3 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:752: read oid 34 snap -1 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:752: expect (ObjNum 187 snap 58 seq_num 187) 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 71 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:750: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:27:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:748: finishing copy_from to smithi11541303-29 2022-04-22T17:27:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:748: got expected ENOENT (src dne) 2022-04-22T17:27:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:748: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:27:55.372 INFO:tasks.rados.rados.0.smithi115.stdout:753: seq_num 225 ranges {0=573440,573440=622592,1196032=180224} 2022-04-22T17:27:55.375 INFO:tasks.rados.rados.0.smithi115.stdout:753: writing smithi11541303-27 from 0 to 573440 tid 1 2022-04-22T17:27:55.379 INFO:tasks.rados.rados.0.smithi115.stdout:753: writing smithi11541303-27 from 573440 to 1196032 tid 2 2022-04-22T17:27:55.380 INFO:tasks.rados.rados.0.smithi115.stdout:753: writing smithi11541303-27 from 1196032 to 1376256 tid 3 2022-04-22T17:27:55.380 INFO:tasks.rados.rados.0.smithi115.stdout:748: done (5 left) 2022-04-22T17:27:55.381 INFO:tasks.rados.rados.0.smithi115.stdout:750: done (4 left) 2022-04-22T17:27:55.381 INFO:tasks.rados.rados.0.smithi115.stdout:754: setattr oid 43 current snap is 71 2022-04-22T17:27:55.381 INFO:tasks.rados.rados.0.smithi115.stdout:751: finishing write tid 1 to smithi11541303-10 2022-04-22T17:27:55.381 INFO:tasks.rados.rados.0.smithi115.stdout:747: expect (ObjNum 89 snap 15 seq_num 89) 2022-04-22T17:27:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:55 smithi121 bash[12679]: cluster 2022-04-22T17:27:54.289257+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2022-04-22T17:27:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:55 smithi121 bash[12679]: cluster 2022-04-22T17:27:55.037875+0000 mgr.y (mgr.24919) 31 : cluster [DBG] pgmap v58: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:751: finishing write tid 2 to smithi11541303-10 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:751: finishing write tid 3 to smithi11541303-10 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:751: finishing write tid 4 to smithi11541303-10 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:751: finishing write tid 5 to smithi11541303-10 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 188 (ObjNum 224 snap 71 seq_num 224) dirty exists 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:751: left oid 10 (ObjNum 224 snap 71 seq_num 224) 2022-04-22T17:27:55.600 INFO:tasks.rados.rados.0.smithi115.stdout:752: expect (ObjNum 187 snap 58 seq_num 187) 2022-04-22T17:27:55.652 INFO:tasks.rados.rados.0.smithi115.stdout:753: finishing write tid 1 to smithi11541303-27 2022-04-22T17:27:55.652 INFO:tasks.rados.rados.0.smithi115.stdout:753: finishing write tid 2 to smithi11541303-27 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:753: finishing write tid 3 to smithi11541303-27 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:753: finishing write tid 4 to smithi11541303-27 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:753: finishing write tid 5 to smithi11541303-27 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 317 (ObjNum 225 snap 71 seq_num 225) dirty exists 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:753: left oid 27 (ObjNum 225 snap 71 seq_num 225) 2022-04-22T17:27:55.653 INFO:tasks.rados.rados.0.smithi115.stdout:747: done (4 left) 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:751: done (3 left) 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:752: done (2 left) 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:753: done (1 left) 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:755: read oid 21 snap -1 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:755: expect deleted 2022-04-22T17:27:55.654 INFO:tasks.rados.rados.0.smithi115.stdout:756: snap_remove snap 61 2022-04-22T17:27:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:55 smithi115 bash[12737]: cluster 2022-04-22T17:27:54.289257+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2022-04-22T17:27:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:55 smithi115 bash[12737]: cluster 2022-04-22T17:27:55.037875+0000 mgr.y (mgr.24919) 31 : cluster [DBG] pgmap v58: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:55 smithi115 bash[18070]: cluster 2022-04-22T17:27:54.289257+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2022-04-22T17:27:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:55 smithi115 bash[18070]: cluster 2022-04-22T17:27:55.037875+0000 mgr.y (mgr.24919) 31 : cluster [DBG] pgmap v58: 121 pgs: 121 active+clean; 116 MiB data, 503 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:756: done (2 left) 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:757: rollback oid 19 current snap is 71 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 68 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 293 (ObjNum 222 snap 71 seq_num 222) dirty exists 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:754: done (2 left) 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:755: done (1 left) 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:758: read oid 8 snap -1 2022-04-22T17:27:56.317 INFO:tasks.rados.rados.0.smithi115.stdout:758: expect deleted 2022-04-22T17:27:56.318 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 71 2022-04-22T17:27:56.318 INFO:tasks.rados.rados.0.smithi115.stdout:759: seq_num 226 ranges {0=589824,589824=507904,1097728=294912} 2022-04-22T17:27:56.332 INFO:tasks.rados.rados.0.smithi115.stdout:759: writing smithi11541303-24 from 0 to 589824 tid 1 2022-04-22T17:27:56.336 INFO:tasks.rados.rados.0.smithi115.stdout:759: writing smithi11541303-24 from 589824 to 1097728 tid 2 2022-04-22T17:27:56.346 INFO:tasks.rados.rados.0.smithi115.stdout:759: writing smithi11541303-24 from 1097728 to 1392640 tid 3 2022-04-22T17:27:56.346 INFO:tasks.rados.rados.0.smithi115.stdout:760: setattr oid 9 current snap is 71 2022-04-22T17:27:56.346 INFO:tasks.rados.rados.0.smithi115.stdout:758: done (3 left) 2022-04-22T17:27:56.346 INFO:tasks.rados.rados.0.smithi115.stdout:761: rmattr oid 4 current snap is 71 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:762: setattr oid 26 current snap is 71 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:757: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:759: finishing write tid 1 to smithi11541303-24 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:759: finishing write tid 2 to smithi11541303-24 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:763: delete oid 18 current snap is 71 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 322 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:759: finishing write tid 3 to smithi11541303-24 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:759: finishing write tid 4 to smithi11541303-24 2022-04-22T17:27:56.347 INFO:tasks.rados.rados.0.smithi115.stdout:757: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:27:56.348 INFO:tasks.rados.rados.0.smithi115.stdout:757: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:27:56.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 240 (ObjNum 45 snap 0 seq_num 45) dirty exists 2022-04-22T17:27:56.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 431 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:757: done (5 left) 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:760: done (4 left) 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:761: done (3 left) 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:763: done (2 left) 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:764: copy_from oid 19 from oid 47 current snap is 71 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:765: read oid 46 snap -1 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:765: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:27:56.349 INFO:tasks.rados.rados.0.smithi115.stdout:766: setattr oid 7 current snap is 71 2022-04-22T17:27:56.350 INFO:tasks.rados.rados.0.smithi115.stdout:759: finishing write tid 5 to smithi11541303-24 2022-04-22T17:27:56.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 96 (ObjNum 226 snap 71 seq_num 226) dirty exists 2022-04-22T17:27:56.350 INFO:tasks.rados.rados.0.smithi115.stdout:759: left oid 24 (ObjNum 226 snap 71 seq_num 226) 2022-04-22T17:27:56.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 182 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:56.352 INFO:tasks.rados.rados.0.smithi115.stdout:759: done (4 left) 2022-04-22T17:27:56.352 INFO:tasks.rados.rados.0.smithi115.stdout:762: done (3 left) 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:767: rollback oid 13 current snap is 71 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 54 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:768: copy_from oid 45 from oid 20 current snap is 71 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:769: read oid 36 snap -1 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:769: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:770: copy_from oid 32 from oid 41 current snap is 71 2022-04-22T17:27:56.353 INFO:tasks.rados.rados.0.smithi115.stdout:771: snap_remove snap 66 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: cluster 2022-04-22T17:27:55.302475+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.939807+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.946948+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.955226+0000 mon.a (mon.0) 1020 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.958298+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:56.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.964581+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.966313+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.971323+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.972634+0000 mon.a (mon.0) 1025 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.976698+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.978390+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: audit 2022-04-22T17:27:55.979603+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:27:56.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:56 smithi121 bash[12679]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:27:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: cluster 2022-04-22T17:27:55.302475+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.939807+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.946948+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.955226+0000 mon.a (mon.0) 1020 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.958298+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.964581+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.966313+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2022-04-22T17:27:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.971323+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.972634+0000 mon.a (mon.0) 1025 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.976698+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.978390+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: audit 2022-04-22T17:27:55.979603+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[12737]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: cluster 2022-04-22T17:27:55.302475+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:27:56.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.939807+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.946948+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.955226+0000 mon.a (mon.0) 1020 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.958298+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.964581+0000 mon.a (mon.0) 1022 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.966313+0000 mon.a (mon.0) 1023 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.971323+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.972634+0000 mon.a (mon.0) 1025 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.976698+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.978390+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:27:56.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: audit 2022-04-22T17:27:55.979603+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:27:56.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:56 smithi115 bash[18070]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:27:57.332 INFO:tasks.rados.rados.0.smithi115.stdout:771: done (7 left) 2022-04-22T17:27:57.332 INFO:tasks.rados.rados.0.smithi115.stdout:772: setattr oid 48 current snap is 71 2022-04-22T17:27:57.333 INFO:tasks.rados.rados.0.smithi115.stdout:765: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:27:57.366 INFO:tasks.rados.rados.0.smithi115.stdout:764: finishing copy_from to smithi11541303-19 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 177 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:764: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:767: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:767: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 297 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 213 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:768: finishing copy_from to smithi11541303-45 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 252 (ObjNum 206 snap 64 seq_num 206) dirty exists 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:768: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:27:57.367 INFO:tasks.rados.rados.0.smithi115.stdout:769: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:57 smithi115 bash[12737]: cluster 2022-04-22T17:27:56.305584+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2022-04-22T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:57 smithi115 bash[12737]: cluster 2022-04-22T17:27:57.038429+0000 mgr.y (mgr.24919) 35 : cluster [DBG] pgmap v61: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:27:57.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:57 smithi115 bash[18070]: cluster 2022-04-22T17:27:56.305584+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2022-04-22T17:27:57.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:57 smithi115 bash[18070]: cluster 2022-04-22T17:27:57.038429+0000 mgr.y (mgr.24919) 35 : cluster [DBG] pgmap v61: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:27:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:57 smithi121 bash[12679]: cluster 2022-04-22T17:27:56.305584+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2022-04-22T17:27:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:57 smithi121 bash[12679]: cluster 2022-04-22T17:27:57.038429+0000 mgr.y (mgr.24919) 35 : cluster [DBG] pgmap v61: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:27:57.858 INFO:tasks.rados.rados.0.smithi115.stdout:770: finishing copy_from to smithi11541303-32 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:770: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 332 (ObjNum 150 snap 40 seq_num 150) dirty exists 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:764: done (7 left) 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:765: done (6 left) 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:766: done (5 left) 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:767: done (4 left) 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:768: done (3 left) 2022-04-22T17:27:57.875 INFO:tasks.rados.rados.0.smithi115.stdout:769: done (2 left) 2022-04-22T17:27:57.876 INFO:tasks.rados.rados.0.smithi115.stdout:770: done (1 left) 2022-04-22T17:27:57.876 INFO:tasks.rados.rados.0.smithi115.stdout:772: done (0 left) 2022-04-22T17:27:57.876 INFO:tasks.rados.rados.0.smithi115.stdout:773: read oid 36 snap -1 2022-04-22T17:27:57.876 INFO:tasks.rados.rados.0.smithi115.stdout:773: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:57.876 INFO:tasks.rados.rados.0.smithi115.stdout:774: copy_from oid 23 from oid 35 current snap is 71 2022-04-22T17:27:57.878 INFO:tasks.rados.rados.0.smithi115.stdout:775: delete oid 34 current snap is 71 2022-04-22T17:27:57.878 INFO:tasks.rados.rados.0.smithi115.stdout:774: finishing copy_from to smithi11541303-23 2022-04-22T17:27:57.879 INFO:tasks.rados.rados.0.smithi115.stdout:774: got expected ENOENT (src dne) 2022-04-22T17:27:57.879 INFO:tasks.rados.rados.0.smithi115.stdout:774: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:27:57.879 INFO:tasks.rados.rados.0.smithi115.stdout:774: done (2 left) 2022-04-22T17:27:57.879 INFO:tasks.rados.rados.0.smithi115.stdout:775: done (1 left) 2022-04-22T17:27:57.879 INFO:tasks.rados.rados.0.smithi115.stdout:776: snap_create 2022-04-22T17:27:57.885 INFO:tasks.rados.rados.0.smithi115.stdout:773: expect (ObjNum 153 snap 41 seq_num 153) 2022-04-22T17:27:58.375 INFO:tasks.rados.rados.0.smithi115.stdout:773: done (1 left) 2022-04-22T17:27:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:776: done (0 left) 2022-04-22T17:27:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:777: read oid 28 snap -1 2022-04-22T17:27:58.376 INFO:tasks.rados.rados.0.smithi115.stdout:777: expect deleted 2022-04-22T17:27:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 72 2022-04-22T17:27:58.377 INFO:tasks.rados.rados.0.smithi115.stdout:778: seq_num 229 ranges {638976=638976,1277952=606208,1884160=196608} 2022-04-22T17:27:58.383 INFO:tasks.rados.rados.0.smithi115.stdout:778: writing smithi11541303-2 from 638976 to 1277952 tid 1 2022-04-22T17:27:58.387 INFO:tasks.rados.rados.0.smithi115.stdout:778: writing smithi11541303-2 from 1277952 to 1884160 tid 2 2022-04-22T17:27:58.389 INFO:tasks.rados.rados.0.smithi115.stdout:778: writing smithi11541303-2 from 1884160 to 2080768 tid 3 2022-04-22T17:27:58.389 INFO:tasks.rados.rados.0.smithi115.stdout:779: delete oid 38 current snap is 72 2022-04-22T17:27:58.392 INFO:tasks.rados.rados.0.smithi115.stdout:778: finishing write tid 1 to smithi11541303-2 2022-04-22T17:27:58.392 INFO:tasks.rados.rados.0.smithi115.stdout:777: done (2 left) 2022-04-22T17:27:58.392 INFO:tasks.rados.rados.0.smithi115.stdout:779: done (1 left) 2022-04-22T17:27:58.392 INFO:tasks.rados.rados.0.smithi115.stdout:780: snap_create 2022-04-22T17:27:58.393 INFO:tasks.rados.rados.0.smithi115.stdout:778: finishing write tid 2 to smithi11541303-2 2022-04-22T17:27:58.393 INFO:tasks.rados.rados.0.smithi115.stdout:778: finishing write tid 3 to smithi11541303-2 2022-04-22T17:27:58.393 INFO:tasks.rados.rados.0.smithi115.stdout:778: finishing write tid 4 to smithi11541303-2 2022-04-22T17:27:58.395 INFO:tasks.rados.rados.0.smithi115.stdout:778: finishing write tid 5 to smithi11541303-2 2022-04-22T17:27:58.395 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 182 (ObjNum 229 snap 72 seq_num 229) dirty exists 2022-04-22T17:27:58.396 INFO:tasks.rados.rados.0.smithi115.stdout:778: left oid 2 (ObjNum 229 snap 72 seq_num 229) 2022-04-22T17:27:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:58 smithi115 bash[12737]: cluster 2022-04-22T17:27:57.322398+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2022-04-22T17:27:58.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:58 smithi115 bash[18070]: cluster 2022-04-22T17:27:57.322398+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2022-04-22T17:27:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:58 smithi121 bash[12679]: cluster 2022-04-22T17:27:57.322398+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2022-04-22T17:27:59.356 INFO:tasks.rados.rados.0.smithi115.stdout:778: done (1 left) 2022-04-22T17:27:59.356 INFO:tasks.rados.rados.0.smithi115.stdout:780: done (0 left) 2022-04-22T17:27:59.356 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 73 2022-04-22T17:27:59.357 INFO:tasks.rados.rados.0.smithi115.stdout:781: seq_num 231 ranges {1376256=671744,2048000=778240,2826240=24576} 2022-04-22T17:27:59.369 INFO:tasks.rados.rados.0.smithi115.stdout:781: writing smithi11541303-27 from 1376256 to 2048000 tid 1 2022-04-22T17:27:59.374 INFO:tasks.rados.rados.0.smithi115.stdout:781: writing smithi11541303-27 from 2048000 to 2826240 tid 2 2022-04-22T17:27:59.374 INFO:tasks.rados.rados.0.smithi115.stdout:781: writing smithi11541303-27 from 2826240 to 2850816 tid 3 2022-04-22T17:27:59.374 INFO:tasks.rados.rados.0.smithi115.stdout:782: read oid 26 snap -1 2022-04-22T17:27:59.374 INFO:tasks.rados.rados.0.smithi115.stdout:782: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:59.375 INFO:tasks.rados.rados.0.smithi115.stdout:783: rollback oid 45 current snap is 73 2022-04-22T17:27:59.375 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 71 2022-04-22T17:27:59.375 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 73 2022-04-22T17:27:59.375 INFO:tasks.rados.rados.0.smithi115.stdout:784: seq_num 232 ranges {270336=688128,958464=663552,1622016=139264} 2022-04-22T17:27:59.382 INFO:tasks.rados.rados.0.smithi115.stdout:784: writing smithi11541303-19 from 270336 to 958464 tid 1 2022-04-22T17:27:59.385 INFO:tasks.rados.rados.0.smithi115.stdout:784: writing smithi11541303-19 from 958464 to 1622016 tid 2 2022-04-22T17:27:59.387 INFO:tasks.rados.rados.0.smithi115.stdout:784: writing smithi11541303-19 from 1622016 to 1761280 tid 3 2022-04-22T17:27:59.387 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 73 2022-04-22T17:27:59.387 INFO:tasks.rados.rados.0.smithi115.stdout:785: seq_num 233 ranges {9945088=712704,10657792=548864,11206656=253952} 2022-04-22T17:27:59.447 INFO:tasks.rados.rados.0.smithi115.stdout:785: writing smithi11541303-23 from 9945088 to 10657792 tid 1 2022-04-22T17:27:59.450 INFO:tasks.rados.rados.0.smithi115.stdout:785: writing smithi11541303-23 from 10657792 to 11206656 tid 2 2022-04-22T17:27:59.452 INFO:tasks.rados.rados.0.smithi115.stdout:785: writing smithi11541303-23 from 11206656 to 11460608 tid 3 2022-04-22T17:27:59.452 INFO:tasks.rados.rados.0.smithi115.stdout:786: setattr oid 21 current snap is 73 2022-04-22T17:27:59.453 INFO:tasks.rados.rados.0.smithi115.stdout:781: finishing write tid 1 to smithi11541303-27 2022-04-22T17:27:59.453 INFO:tasks.rados.rados.0.smithi115.stdout:781: finishing write tid 2 to smithi11541303-27 2022-04-22T17:27:59.453 INFO:tasks.rados.rados.0.smithi115.stdout:781: finishing write tid 3 to smithi11541303-27 2022-04-22T17:27:59.453 INFO:tasks.rados.rados.0.smithi115.stdout:781: finishing write tid 4 to smithi11541303-27 2022-04-22T17:27:59.453 INFO:tasks.rados.rados.0.smithi115.stdout:783: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:27:59.454 INFO:tasks.rados.rados.0.smithi115.stdout:784: finishing write tid 1 to smithi11541303-19 2022-04-22T17:27:59.454 INFO:tasks.rados.rados.0.smithi115.stdout:783: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:27:59.454 INFO:tasks.rados.rados.0.smithi115.stdout:784: finishing write tid 2 to smithi11541303-19 2022-04-22T17:27:59.454 INFO:tasks.rados.rados.0.smithi115.stdout:784: finishing write tid 3 to smithi11541303-19 2022-04-22T17:27:59.454 INFO:tasks.rados.rados.0.smithi115.stdout:781: finishing write tid 5 to smithi11541303-27 2022-04-22T17:27:59.455 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 337 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:27:59.455 INFO:tasks.rados.rados.0.smithi115.stdout:781: left oid 27 (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:27:59.455 INFO:tasks.rados.rados.0.smithi115.stdout:783: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:27:59.455 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 341 (ObjNum 206 snap 64 seq_num 206) dirty exists 2022-04-22T17:27:59.455 INFO:tasks.rados.rados.0.smithi115.stdout:784: finishing write tid 4 to smithi11541303-19 2022-04-22T17:27:59.456 INFO:tasks.rados.rados.0.smithi115.stdout:784: finishing write tid 5 to smithi11541303-19 2022-04-22T17:27:59.456 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 259 (ObjNum 232 snap 73 seq_num 232) dirty exists 2022-04-22T17:27:59.456 INFO:tasks.rados.rados.0.smithi115.stdout:784: left oid 19 (ObjNum 232 snap 73 seq_num 232) 2022-04-22T17:27:59.456 INFO:tasks.rados.rados.0.smithi115.stdout:782: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:27:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: cluster 2022-04-22T17:27:58.335302+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2022-04-22T17:27:59.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:27:59.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: audit 2022-04-22T17:27:58.842565+0000 mon.a (mon.0) 1032 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:59.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: audit 2022-04-22T17:27:58.844319+0000 mon.a (mon.0) 1033 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:27:59.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: audit 2022-04-22T17:27:58.845635+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: audit 2022-04-22T17:27:58.846711+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[12737]: cluster 2022-04-22T17:27:59.039122+0000 mgr.y (mgr.24919) 38 : cluster [DBG] pgmap v64: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: cluster 2022-04-22T17:27:58.335302+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: audit 2022-04-22T17:27:58.842565+0000 mon.a (mon.0) 1032 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:59.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: audit 2022-04-22T17:27:58.844319+0000 mon.a (mon.0) 1033 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:27:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: audit 2022-04-22T17:27:58.845635+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:27:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: audit 2022-04-22T17:27:58.846711+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:27:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:27:59 smithi115 bash[18070]: cluster 2022-04-22T17:27:59.039122+0000 mgr.y (mgr.24919) 38 : cluster [DBG] pgmap v64: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: finishing write tid 1 to smithi11541303-23 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: finishing write tid 2 to smithi11541303-23 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: finishing write tid 3 to smithi11541303-23 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: finishing write tid 4 to smithi11541303-23 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: finishing write tid 5 to smithi11541303-23 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 294 (ObjNum 233 snap 73 seq_num 233) dirty exists 2022-04-22T17:27:59.672 INFO:tasks.rados.rados.0.smithi115.stdout:785: left oid 23 (ObjNum 233 snap 73 seq_num 233) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:781: done (5 left) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:782: done (4 left) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:783: done (3 left) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:784: done (2 left) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:785: done (1 left) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:787: read oid 29 snap -1 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:787: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:59.673 INFO:tasks.rados.rados.0.smithi115.stdout:788: snap_create 2022-04-22T17:27:59.674 INFO:tasks.rados.rados.0.smithi115.stdout:787: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:27:59.680 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 162 (ObjNum -1116472784 snap 22075 seq_num 2684356816) dirty exists 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: cluster 2022-04-22T17:27:58.335302+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: audit 2022-04-22T17:27:58.842565+0000 mon.a (mon.0) 1032 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: audit 2022-04-22T17:27:58.844319+0000 mon.a (mon.0) 1033 : audit [INF] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: audit 2022-04-22T17:27:58.845635+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: audit 2022-04-22T17:27:58.846711+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:27:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:27:59 smithi121 bash[12679]: cluster 2022-04-22T17:27:59.039122+0000 mgr.y (mgr.24919) 38 : cluster [DBG] pgmap v64: 121 pgs: 121 active+clean; 136 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 675 KiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:28:00.365 INFO:tasks.rados.rados.0.smithi115.stdout:786: done (2 left) 2022-04-22T17:28:00.404 INFO:tasks.rados.rados.0.smithi115.stdout:787: done (1 left) 2022-04-22T17:28:00.404 INFO:tasks.rados.rados.0.smithi115.stdout:788: done (0 left) 2022-04-22T17:28:00.404 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 74 2022-04-22T17:28:00.404 INFO:tasks.rados.rados.0.smithi115.stdout:789: seq_num 234 ranges {9134080=729088,9863168=434176,10297344=368640} 2022-04-22T17:28:00.427 INFO:tasks.rados.rados.0.smithi115.stdout:789: writing smithi11541303-13 from 9134080 to 9863168 tid 1 2022-04-22T17:28:00.429 INFO:tasks.rados.rados.0.smithi115.stdout:789: writing smithi11541303-13 from 9863168 to 10297344 tid 2 2022-04-22T17:28:00.431 INFO:tasks.rados.rados.0.smithi115.stdout:789: writing smithi11541303-13 from 10297344 to 10665984 tid 3 2022-04-22T17:28:00.431 INFO:tasks.rados.rados.0.smithi115.stdout:790: delete oid 36 current snap is 74 2022-04-22T17:28:00.439 INFO:tasks.rados.rados.0.smithi115.stdout:790: done (1 left) 2022-04-22T17:28:00.439 INFO:tasks.rados.rados.0.smithi115.stdout:791: rmattr oid 20 current snap is 74 2022-04-22T17:28:00.440 INFO:tasks.rados.rados.0.smithi115.stdout:791: done (1 left) 2022-04-22T17:28:00.440 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 74 2022-04-22T17:28:00.440 INFO:tasks.rados.rados.0.smithi115.stdout:792: seq_num 236 ranges {1392640=761856,2154496=647168,2801664=155648} 2022-04-22T17:28:00.451 INFO:tasks.rados.rados.0.smithi115.stdout:792: writing smithi11541303-24 from 1392640 to 2154496 tid 1 2022-04-22T17:28:00.455 INFO:tasks.rados.rados.0.smithi115.stdout:792: writing smithi11541303-24 from 2154496 to 2801664 tid 2 2022-04-22T17:28:00.457 INFO:tasks.rados.rados.0.smithi115.stdout:792: writing smithi11541303-24 from 2801664 to 2957312 tid 3 2022-04-22T17:28:00.457 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 74 2022-04-22T17:28:00.457 INFO:tasks.rados.rados.0.smithi115.stdout:789: finishing write tid 1 to smithi11541303-13 2022-04-22T17:28:00.457 INFO:tasks.rados.rados.0.smithi115.stdout:793: seq_num 237 ranges {0=778240,778240=532480,1310720=270336} 2022-04-22T17:28:00.461 INFO:tasks.rados.rados.0.smithi115.stdout:793: writing smithi11541303-37 from 0 to 778240 tid 1 2022-04-22T17:28:00.464 INFO:tasks.rados.rados.0.smithi115.stdout:793: writing smithi11541303-37 from 778240 to 1310720 tid 2 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:793: writing smithi11541303-37 from 1310720 to 1581056 tid 3 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:794: delete oid 43 current snap is 74 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:789: finishing write tid 2 to smithi11541303-13 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:789: finishing write tid 3 to smithi11541303-13 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:789: finishing write tid 4 to smithi11541303-13 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:789: finishing write tid 5 to smithi11541303-13 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 302 (ObjNum 234 snap 74 seq_num 234) dirty exists 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:789: left oid 13 (ObjNum 234 snap 74 seq_num 234) 2022-04-22T17:28:00.467 INFO:tasks.rados.rados.0.smithi115.stdout:792: finishing write tid 1 to smithi11541303-24 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:792: finishing write tid 2 to smithi11541303-24 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:792: finishing write tid 3 to smithi11541303-24 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:792: finishing write tid 4 to smithi11541303-24 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:792: finishing write tid 5 to smithi11541303-24 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 101 (ObjNum 236 snap 74 seq_num 236) dirty exists 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:792: left oid 24 (ObjNum 236 snap 74 seq_num 236) 2022-04-22T17:28:00.468 INFO:tasks.rados.rados.0.smithi115.stdout:793: finishing write tid 1 to smithi11541303-37 2022-04-22T17:28:00.469 INFO:tasks.rados.rados.0.smithi115.stdout:793: finishing write tid 2 to smithi11541303-37 2022-04-22T17:28:00.469 INFO:tasks.rados.rados.0.smithi115.stdout:793: finishing write tid 3 to smithi11541303-37 2022-04-22T17:28:00.470 INFO:tasks.rados.rados.0.smithi115.stdout:793: finishing write tid 4 to smithi11541303-37 2022-04-22T17:28:00.470 INFO:tasks.rados.rados.0.smithi115.stdout:789: done (3 left) 2022-04-22T17:28:00.471 INFO:tasks.rados.rados.0.smithi115.stdout:792: done (2 left) 2022-04-22T17:28:00.471 INFO:tasks.rados.rados.0.smithi115.stdout:794: done (1 left) 2022-04-22T17:28:00.471 INFO:tasks.rados.rados.0.smithi115.stdout:795: snap_create 2022-04-22T17:28:00.472 INFO:tasks.rados.rados.0.smithi115.stdout:793: finishing write tid 5 to smithi11541303-37 2022-04-22T17:28:00.472 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 186 (ObjNum 237 snap 74 seq_num 237) dirty exists 2022-04-22T17:28:00.472 INFO:tasks.rados.rados.0.smithi115.stdout:793: left oid 37 (ObjNum 237 snap 74 seq_num 237) 2022-04-22T17:28:00.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:00 smithi115 bash[12737]: cluster 2022-04-22T17:27:59.344662+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2022-04-22T17:28:00.670 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:00 smithi115 bash[18070]: cluster 2022-04-22T17:27:59.344662+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2022-04-22T17:28:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:00 smithi121 bash[12679]: cluster 2022-04-22T17:27:59.344662+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2022-04-22T17:28:00.904 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:28:01.210 INFO:tasks.rados.rados.0.smithi115.stdout:793: done (1 left) 2022-04-22T17:28:01.210 INFO:tasks.rados.rados.0.smithi115.stdout:795: done (0 left) 2022-04-22T17:28:01.210 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 75 2022-04-22T17:28:01.210 INFO:tasks.rados.rados.0.smithi115.stdout:796: seq_num 239 ranges {10665984=409600,11075584=704512,11780096=499712} 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 36s ago 4m 12.6M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4m) 14s ago 4m 23.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5m) 36s ago 5m 12.8M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5m) 14s ago 5m 16.1M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (20s) 14s ago 10m 125M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:28:01.234 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (58s) 36s ago 12m 433M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (12m) 36s ago 12m 52.3M 2048M 16.2.5 46cf6318c64e c6b4ba941f5c 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (10m) 14s ago 10m 47.0M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 36s ago 10m 41.8M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5m) 36s ago 5m 8699k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5m) 14s ago 5m 8783k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 36s ago 9m 113M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 36s ago 9m 101M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 36s ago 8m 82.6M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 36s ago 8m 67.4M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:28:01.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (7m) 14s ago 7m 104M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:28:01.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (7m) 14s ago 7m 111M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:28:01.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6m) 14s ago 6m 99.5M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:28:01.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6m) 14s ago 6m 103M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:28:01.236 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (66s) 14s ago 5m 31.9M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:28:01.274 INFO:tasks.rados.rados.0.smithi115.stdout:796: writing smithi11541303-13 from 10665984 to 11075584 tid 1 2022-04-22T17:28:01.277 INFO:tasks.rados.rados.0.smithi115.stdout:796: writing smithi11541303-13 from 11075584 to 11780096 tid 2 2022-04-22T17:28:01.280 INFO:tasks.rados.rados.0.smithi115.stdout:796: writing smithi11541303-13 from 11780096 to 12279808 tid 3 2022-04-22T17:28:01.280 INFO:tasks.rados.rados.0.smithi115.stdout:797: delete oid 50 current snap is 75 2022-04-22T17:28:01.288 INFO:tasks.rados.rados.0.smithi115.stdout:797: done (1 left) 2022-04-22T17:28:01.288 INFO:tasks.rados.rados.0.smithi115.stdout:798: rollback oid 21 current snap is 75 2022-04-22T17:28:01.288 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 72 2022-04-22T17:28:01.289 INFO:tasks.rados.rados.0.smithi115.stdout:799: snap_remove snap 73 2022-04-22T17:28:01.645 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[18070]: cluster 2022-04-22T17:28:00.355312+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2022-04-22T17:28:01.645 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[18070]: audit 2022-04-22T17:28:00.880387+0000 mgr.y (mgr.24919) 39 : audit [DBG] from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[18070]: cluster 2022-04-22T17:28:01.039885+0000 mgr.y (mgr.24919) 40 : cluster [DBG] pgmap v67: 121 pgs: 121 active+clean; 162 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.9 MiB/s wr, 55 op/s 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[18070]: cluster 2022-04-22T17:28:01.197168+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: cluster 2022-04-22T17:28:00.355312+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: audit 2022-04-22T17:28:00.880387+0000 mgr.y (mgr.24919) 39 : audit [DBG] from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: cluster 2022-04-22T17:28:01.039885+0000 mgr.y (mgr.24919) 40 : cluster [DBG] pgmap v67: 121 pgs: 121 active+clean; 162 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.9 MiB/s wr, 55 op/s 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: cluster 2022-04-22T17:28:01.197168+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2022-04-22T17:28:01.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 systemd[1]: Stopping Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:28:01.647 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:28:01.647 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:28:01.648 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13, 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:28:01.649 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:28:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:01 smithi121 bash[12679]: cluster 2022-04-22T17:28:00.355312+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2022-04-22T17:28:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:01 smithi121 bash[12679]: audit 2022-04-22T17:28:00.880387+0000 mgr.y (mgr.24919) 39 : audit [DBG] from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:01 smithi121 bash[12679]: cluster 2022-04-22T17:28:01.039885+0000 mgr.y (mgr.24919) 40 : cluster [DBG] pgmap v67: 121 pgs: 121 active+clean; 162 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.9 MiB/s wr, 55 op/s 2022-04-22T17:28:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:01 smithi121 bash[12679]: cluster 2022-04-22T17:28:01.197168+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2022-04-22T17:28:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[45369]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.a 2022-04-22T17:28:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: debug 2022-04-22T17:28:01.748+0000 7fb55f53c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-04-22T17:28:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:01 smithi115 bash[12737]: debug 2022-04-22T17:28:01.748+0000 7fb55f53c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2022-04-22T17:28:03.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:02 smithi115 bash[45385]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon-a 2022-04-22T17:28:03.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:02 smithi115 bash[45427]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.a 2022-04-22T17:28:03.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:02 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.a.service: Succeeded. 2022-04-22T17:28:03.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:02 smithi115 systemd[1]: Stopped Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:28:03.510 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:03 smithi115 systemd[1]: Started Ceph mon.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.500Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:03.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:04.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.652+0000 7fe1f48bb880 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:28:04.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.652+0000 7fe1f48bb880 0 ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev), process ceph-mon, pid 7 2022-04-22T17:28:04.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.652+0000 7fe1f48bb880 0 pidfile_write: ignore empty --pid-file 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 0 load: jerasure load: lrc 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: RocksDB version: 6.15.5 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Compile date Apr 22 2022 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: DB SUMMARY 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: DB Session ID: Q0S6A5QQ2WN1FSIYHJRR 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: CURRENT file: CURRENT 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: IDENTITY file: IDENTITY 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 972 Bytes 2022-04-22T17:28:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000023.sst 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000021.log size: 6761968 ; 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.create_if_missing: 0 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.env: 0x558fc10e0860 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:28:04.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.info_log: 0x558fc321be00 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:28:04.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:28:04.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:28:04.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:28:04.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:28:04.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:28:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:28:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:28:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:28:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:28:04.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.db_log_dir: 2022-04-22T17:28:04.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-04-22T17:28:04.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:28:04.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:28:04.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:28:04.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:28:04.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:28:04.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:28:04.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:28:04.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:28:04.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.write_buffer_manager: 0x558fc32e2420 2022-04-22T17:28:04.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:28:04.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:28:04.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:28:04.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:28:04.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:28:04.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:28:04.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:28:04.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:28:04.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:28:04.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:28:04.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:28:04.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:28:04.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:28:04.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:28:04.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.row_cache: None 2022-04-22T17:28:04.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.wal_filter: None 2022-04-22T17:28:04.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:28:04.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:28:04.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:28:04.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:28:04.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:28:04.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:28:04.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:28:04.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:28:04.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:28:04.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:28:04.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2022-04-22T17:28:04.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.best_efforts_recovery: 0 2022-04-22T17:28:04.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-04-22T17:28:04.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-04-22T17:28:04.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.allow_data_in_errors: 0 2022-04-22T17:28:04.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.db_host_id: __hostname__ 2022-04-22T17:28:04.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:28:04.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:28:04.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:28:04.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:28:04.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:28:04.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:28:04.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:28:04.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:28:04.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:28:04.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:28:04.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:28:04.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:28:04.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:28:04.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:28:04.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:28:04.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:28:04.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:28:04.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Compression algorithms supported: 2022-04-22T17:28:04.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:28:04.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kZSTD supported: 0 2022-04-22T17:28:04.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:28:04.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:28:04.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:28:04.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:28:04.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:28:04.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:28:04.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:28:04.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2022-04-22T17:28:04.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-04-22T17:28:04.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:28:04.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.merge_operator: 2022-04-22T17:28:04.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:28:04.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:28:04.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.sst_partitioner_factory: None 2022-04-22T17:28:04.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:28:04.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:28:04.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558fc31e9cb0) 2022-04-22T17:28:04.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: cache_index_and_filter_blocks: 1 2022-04-22T17:28:04.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:28:04.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:28:04.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: pin_top_level_index_and_filter: 1 2022-04-22T17:28:04.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: index_type: 0 2022-04-22T17:28:04.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: data_block_index_type: 0 2022-04-22T17:28:04.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: index_shortening: 1 2022-04-22T17:28:04.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:28:04.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: hash_index_allow_collision: 1 2022-04-22T17:28:04.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: checksum: 1 2022-04-22T17:28:04.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: no_block_cache: 0 2022-04-22T17:28:04.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_cache: 0x558fc3250170 2022-04-22T17:28:04.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_cache_name: BinnedLRUCache 2022-04-22T17:28:04.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_cache_options: 2022-04-22T17:28:04.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: capacity : 536870912 2022-04-22T17:28:04.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: num_shard_bits : 4 2022-04-22T17:28:04.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: strict_capacity_limit : 0 2022-04-22T17:28:04.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: high_pri_pool_ratio: 0.000 2022-04-22T17:28:04.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_cache_compressed: (nil) 2022-04-22T17:28:04.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: persistent_cache: (nil) 2022-04-22T17:28:04.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_size: 4096 2022-04-22T17:28:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_size_deviation: 10 2022-04-22T17:28:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_restart_interval: 16 2022-04-22T17:28:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: index_block_restart_interval: 1 2022-04-22T17:28:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: metadata_block_size: 4096 2022-04-22T17:28:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: partition_filters: 0 2022-04-22T17:28:04.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: use_delta_encoding: 1 2022-04-22T17:28:04.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:28:04.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: whole_key_filtering: 1 2022-04-22T17:28:04.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: verify_compression: 0 2022-04-22T17:28:04.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: read_amp_bytes_per_bit: 0 2022-04-22T17:28:04.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: format_version: 4 2022-04-22T17:28:04.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: enable_index_compression: 1 2022-04-22T17:28:04.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: block_align: 0 2022-04-22T17:28:04.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:28:04.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:28:04.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:28:04.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:28:04.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:28:04.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:28:04.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.num_levels: 7 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-04-22T17:28:04.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:28:04.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:28:04.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:28:04.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:28:04.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:28:04.841 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.force_consistency_checks: 1 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.enable_blob_files: false 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.min_blob_size: 0 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.blob_file_size: 268435456 2022-04-22T17:28:04.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.blob_compression_type: NoCompression 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.enable_blob_garbage_collection: false 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.656+0000 7fe1f48bb880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 25, last_sequence is 11699, log_number is 21,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 21 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 25 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648484668754, "job": 1, "event": "recovery_started", "wal_files": [21]} 2022-04-22T17:28:04.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.664+0000 7fe1f48bb880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #21 mode 2 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.672+0000 7fe1f48bb880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.684+0000 7fe1f48bb880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648484689489, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 6555577, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 6544945, "index_size": 8219, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 12500, "raw_average_key_size": 22, "raw_value_size": 6528585, "raw_average_value_size": 11575, "num_data_blocks": 353, "num_entries": 564, "num_deletions": 33, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1650648484, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f2956c30-0eb5-4497-9635-18ed62046ef1", "db_session_id": "Q0S6A5QQ2WN1FSIYHJRR"}} 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.684+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 27 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.684+0000 7fe1f48bb880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648484691784, "job": 1, "event": "recovery_finished"} 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000021.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x558fc3236a80 2022-04-22T17:28:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 4 rocksdb: DB pointer 0x558fc3247800 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1e4639700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1e4639700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** DB Stats ** 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:28:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** Compaction Stats [default] ** 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: L0 1/0 6.25 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Sum 2/0 29.06 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** Compaction Stats [default] ** 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Flush(GB): cumulative 0.006, interval 0.006 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Cumulative compaction: 0.01 GB write, 204.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Interval compaction: 0.01 GB write, 204.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:28:04.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** Compaction Stats [default] ** 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: L0 1/0 6.25 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Sum 2/0 29.06 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** Compaction Stats [default] ** 2022-04-22T17:28:04.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 609.9 0.01 0.00 1 0.010 0 0 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Flush(GB): cumulative 0.006, interval 0.000 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Cumulative compaction: 0.01 GB write, 204.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:04.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 0 starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 1 mon.a@-1(???) e3 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 0 mon.a@-1(???).mds e6 new map 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.688+0000 7fe1f48bb880 0 mon.a@-1(???).mds e6 print_map 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: e6 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: legacy client fscid: 1 2022-04-22T17:28:04.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Filesystem 'cephfs' (1) 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: fs_name cephfs 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: epoch 6 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: flags 12 joinable allow_snaps allow_multimds_snaps 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: created 2022-04-22T17:22:18.413732+0000 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: modified 2022-04-22T17:22:20.428010+0000 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: tableserver 0 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: root 0 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: session_timeout 60 2022-04-22T17:28:04.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: session_autoclose 300 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: max_file_size 1099511627776 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: required_client_features {} 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: last_failure 0 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: last_failure_osd_epoch 0 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: max_mds 1 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: in 0 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: up {0=24386} 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: failed 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: damaged 2022-04-22T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: stopped 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: data_pools [3] 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: metadata_pool 2 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: inline_data disabled 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: balancer 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: standby_count_wanted 0 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: [mds.b{0:24386} state up:active seq 3 addr [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: Standby daemons: 2022-04-22T17:28:04.861 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: [mds.a{-1:24385} state up:standby seq 2 addr [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.692+0000 7fe1f48bb880 0 mon.a@-1(???).osd e208 crush map has features 3314933069573799936, adjusting msgr requires 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.692+0000 7fe1f48bb880 0 mon.a@-1(???).osd e208 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.692+0000 7fe1f48bb880 0 mon.a@-1(???).osd e208 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.692+0000 7fe1f48bb880 0 mon.a@-1(???).osd e208 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:04.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:28:04 smithi115 bash[45495]: debug 2022-04-22T17:28:04.692+0000 7fe1f48bb880 1 mon.a@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:799: done (2 left) 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:800: snap_create 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:796: finishing write tid 1 to smithi11541303-13 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:796: finishing write tid 2 to smithi11541303-13 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:798: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:28:05.043 INFO:tasks.rados.rados.0.smithi115.stdout:796: finishing write tid 3 to smithi11541303-13 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:796: finishing write tid 4 to smithi11541303-13 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:798: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 165 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:796: finishing write tid 5 to smithi11541303-13 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 309 (ObjNum 239 snap 75 seq_num 239) dirty exists 2022-04-22T17:28:05.044 INFO:tasks.rados.rados.0.smithi115.stdout:796: left oid 13 (ObjNum 239 snap 75 seq_num 239) 2022-04-22T17:28:05.297 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[42914]: ignoring --setuser ceph since I am not root 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[42914]: ignoring --setgroup ceph since I am not root 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[42914]: debug 2022-04-22T17:28:05.036+0000 7f8f205b4700 1 -- 172.21.15.115:0/767606369 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555676308340 con 0x555677084800 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[42914]: debug 2022-04-22T17:28:05.192+0000 7f8f2901b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: audit 2022-04-22T17:28:01.760018+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: audit 2022-04-22T17:28:01.760991+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:28:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: audit 2022-04-22T17:28:01.761911+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:03.040765+0000 mgr.y (mgr.24919) 42 : cluster [DBG] pgmap v69: 121 pgs: 121 active+clean; 162 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 47 op/s 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.909200+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.933907+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.942256+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.960682+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.960812+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.962859+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e25: y(active, since 46s), standbys: x 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.976967+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e26: y(active, since 46s), standbys: x 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: cluster 2022-04-22T17:28:04.977015+0000 mon.a (mon.0) 8 : cluster [INF] overall HEALTH_OK 2022-04-22T17:28:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[18070]: audit 2022-04-22T17:28:04.977193+0000 mon.a (mon.0) 9 : audit [INF] from='mgr.24919 ' entity='' 2022-04-22T17:28:05.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[34414]: ignoring --setuser ceph since I am not root 2022-04-22T17:28:05.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[34414]: ignoring --setgroup ceph since I am not root 2022-04-22T17:28:05.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[34414]: debug 2022-04-22T17:28:05.035+0000 7fc2ddcad700 1 -- 172.21.15.121:0/2314199519 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564508622340 con 0x56450871cc00 2022-04-22T17:28:05.517 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[34414]: debug 2022-04-22T17:28:05.203+0000 7fc2e6714000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: audit 2022-04-22T17:28:01.760018+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: audit 2022-04-22T17:28:01.760991+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: audit 2022-04-22T17:28:01.761911+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.24919 172.21.15.115:0/2725406046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:03.040765+0000 mgr.y (mgr.24919) 42 : cluster [DBG] pgmap v69: 121 pgs: 121 active+clean; 162 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 47 op/s 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.909200+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.933907+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.942256+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:28:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.960682+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:28:05.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.960812+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2022-04-22T17:28:05.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.962859+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e25: y(active, since 46s), standbys: x 2022-04-22T17:28:05.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.976967+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e26: y(active, since 46s), standbys: x 2022-04-22T17:28:05.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: cluster 2022-04-22T17:28:04.977015+0000 mon.a (mon.0) 8 : cluster [INF] overall HEALTH_OK 2022-04-22T17:28:05.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[12679]: audit 2022-04-22T17:28:04.977193+0000 mon.a (mon.0) 9 : audit [INF] from='mgr.24919 ' entity='' 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:796: done (2 left) 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:798: done (1 left) 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:800: done (0 left) 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:801: read oid 46 snap -1 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:801: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:802: read oid 12 snap -1 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:802: expect (ObjNum 223 snap 71 seq_num 223) 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:803: copy_from oid 19 from oid 41 current snap is 76 2022-04-22T17:28:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:804: copy_from oid 49 from oid 18 current snap is 76 2022-04-22T17:28:06.052 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 76 2022-04-22T17:28:06.052 INFO:tasks.rados.rados.0.smithi115.stdout:805: seq_num 241 ranges {0=442368,442368=466944,909312=688128,1597440=49152} 2022-04-22T17:28:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:805: writing smithi11541303-35 from 0 to 442368 tid 1 2022-04-22T17:28:06.056 INFO:tasks.rados.rados.0.smithi115.stdout:805: writing smithi11541303-35 from 442368 to 909312 tid 2 2022-04-22T17:28:06.062 INFO:tasks.rados.rados.0.smithi115.stdout:805: writing smithi11541303-35 from 909312 to 1597440 tid 3 2022-04-22T17:28:06.062 INFO:tasks.rados.rados.0.smithi115.stdout:805: writing smithi11541303-35 from 1597440 to 1646592 tid 4 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:806: rollback oid 15 current snap is 76 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 59 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:804: finishing copy_from to smithi11541303-49 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:804: got expected ENOENT (src dne) 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:804: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:28:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:801: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:06.089 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 1 to smithi11541303-35 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 2 to smithi11541303-35 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:801: done (5 left) 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:804: done (4 left) 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:807: rollback oid 17 current snap is 76 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 17 to 72 2022-04-22T17:28:06.090 INFO:tasks.rados.rados.0.smithi115.stdout:802: expect (ObjNum 223 snap 71 seq_num 223) 2022-04-22T17:28:06.166 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 3 to smithi11541303-35 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 4 to smithi11541303-35 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 5 to smithi11541303-35 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:806: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:802: done (4 left) 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:808: rollback oid 25 current snap is 76 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 68 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:806: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:805: finishing write tid 6 to smithi11541303-35 2022-04-22T17:28:06.167 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 191 (ObjNum 241 snap 76 seq_num 241) dirty exists 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:805: left oid 35 (ObjNum 241 snap 76 seq_num 241) 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:806: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 182 (ObjNum 155 snap 41 seq_num 155) dirty exists 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:803: finishing copy_from to smithi11541303-19 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:803: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:807: finishing rollback tid 1 to smithi11541303-17 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 229 (ObjNum -1116632800 snap 22075 seq_num 3176620048) dirty dne 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:803: done (4 left) 2022-04-22T17:28:06.168 INFO:tasks.rados.rados.0.smithi115.stdout:805: done (3 left) 2022-04-22T17:28:06.169 INFO:tasks.rados.rados.0.smithi115.stdout:806: done (2 left) 2022-04-22T17:28:06.169 INFO:tasks.rados.rados.0.smithi115.stdout:807: done (1 left) 2022-04-22T17:28:06.169 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 76 2022-04-22T17:28:06.169 INFO:tasks.rados.rados.0.smithi115.stdout:809: seq_num 242 ranges {2359296=458752,2818048=753664,3571712=450560} 2022-04-22T17:28:06.183 INFO:tasks.rados.rados.0.smithi115.stdout:809: writing smithi11541303-30 from 2359296 to 2818048 tid 1 2022-04-22T17:28:06.189 INFO:tasks.rados.rados.0.smithi115.stdout:809: writing smithi11541303-30 from 2818048 to 3571712 tid 2 2022-04-22T17:28:06.192 INFO:tasks.rados.rados.0.smithi115.stdout:809: writing smithi11541303-30 from 3571712 to 4022272 tid 3 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:810: rollback oid 13 current snap is 76 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:808: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 72 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:808: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:808: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 318 (ObjNum -1116203520 snap 22075 seq_num 2571903024) dirty exists 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:809: finishing write tid 1 to smithi11541303-30 2022-04-22T17:28:06.193 INFO:tasks.rados.rados.0.smithi115.stdout:808: done (2 left) 2022-04-22T17:28:06.194 INFO:tasks.rados.rados.0.smithi115.stdout:811: read oid 11 snap -1 2022-04-22T17:28:06.194 INFO:tasks.rados.rados.0.smithi115.stdout:811: expect (ObjNum 220 snap 71 seq_num 220) 2022-04-22T17:28:06.194 INFO:tasks.rados.rados.0.smithi115.stdout:812: snap_create 2022-04-22T17:28:06.197 INFO:tasks.rados.rados.0.smithi115.stdout:809: finishing write tid 2 to smithi11541303-30 2022-04-22T17:28:06.200 INFO:tasks.rados.rados.0.smithi115.stdout:810: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:28:06.201 INFO:tasks.rados.rados.0.smithi115.stdout:810: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:28:06.201 INFO:tasks.rados.rados.0.smithi115.stdout:810: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:28:06.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 322 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:28:06.202 INFO:tasks.rados.rados.0.smithi115.stdout:809: finishing write tid 3 to smithi11541303-30 2022-04-22T17:28:06.202 INFO:tasks.rados.rados.0.smithi115.stdout:809: finishing write tid 4 to smithi11541303-30 2022-04-22T17:28:06.203 INFO:tasks.rados.rados.0.smithi115.stdout:809: finishing write tid 5 to smithi11541303-30 2022-04-22T17:28:06.203 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 346 (ObjNum 242 snap 76 seq_num 242) dirty exists 2022-04-22T17:28:06.203 INFO:tasks.rados.rados.0.smithi115.stdout:809: left oid 30 (ObjNum 242 snap 76 seq_num 242) 2022-04-22T17:28:06.204 INFO:tasks.rados.rados.0.smithi115.stdout:811: expect (ObjNum 220 snap 71 seq_num 220) 2022-04-22T17:28:06.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[12679]: cluster 2022-04-22T17:28:05.029512+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2022-04-22T17:28:06.223 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:05 smithi121 bash[34414]: debug 2022-04-22T17:28:05.895+0000 7fc2e6714000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:28:06.297 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:05 smithi115 bash[42914]: debug 2022-04-22T17:28:05.920+0000 7f8f2901b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-22T17:28:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:06 smithi115 bash[18070]: cluster 2022-04-22T17:28:05.029512+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2022-04-22T17:28:06.481 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.215+0000 7fc2e6714000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:28:06.481 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.367+0000 7fc2e6714000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:28:06.615 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:06 smithi115 bash[42914]: debug 2022-04-22T17:28:06.296+0000 7f8f2901b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-22T17:28:06.704 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:06 smithi115 bash[42914]: debug 2022-04-22T17:28:06.480+0000 7f8f2901b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-22T17:28:06.909 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:06 smithi115 bash[42914]: debug 2022-04-22T17:28:06.600+0000 7f8f2901b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:28:07.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.471+0000 7fc2e6714000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-22T17:28:07.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.739+0000 7fc2e6714000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:28:07.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.851+0000 7fc2e6714000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:28:07.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:06 smithi121 bash[34414]: debug 2022-04-22T17:28:06.955+0000 7fc2e6714000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:28:07.053 INFO:tasks.rados.rados.0.smithi115.stdout:809: done (3 left) 2022-04-22T17:28:07.053 INFO:tasks.rados.rados.0.smithi115.stdout:810: done (2 left) 2022-04-22T17:28:07.053 INFO:tasks.rados.rados.0.smithi115.stdout:811: done (1 left) 2022-04-22T17:28:07.053 INFO:tasks.rados.rados.0.smithi115.stdout:812: done (0 left) 2022-04-22T17:28:07.054 INFO:tasks.rados.rados.0.smithi115.stdout:813: snap_remove snap 63 2022-04-22T17:28:07.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[18070]: cluster 2022-04-22T17:28:06.038887+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2022-04-22T17:28:07.235 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:06 smithi115 bash[42914]: debug 2022-04-22T17:28:06.892+0000 7f8f2901b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-22T17:28:07.236 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.012+0000 7f8f2901b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-22T17:28:07.236 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.120+0000 7f8f2901b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-22T17:28:07.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[12679]: cluster 2022-04-22T17:28:06.038887+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2022-04-22T17:28:07.509 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[34414]: debug 2022-04-22T17:28:07.055+0000 7fc2e6714000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:28:07.548 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.220+0000 7f8f2901b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-22T17:28:07.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[34414]: debug 2022-04-22T17:28:07.499+0000 7fc2e6714000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:28:07.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[34414]: debug 2022-04-22T17:28:07.635+0000 7fc2e6714000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:28:07.767 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[34414]: debug 2022-04-22T17:28:07.739+0000 7fc2e6714000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:28:08.048 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.668+0000 7f8f2901b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-22T17:28:08.048 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.800+0000 7f8f2901b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-22T17:28:08.048 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:07 smithi115 bash[42914]: debug 2022-04-22T17:28:07.904+0000 7f8f2901b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-22T17:28:08.065 INFO:tasks.rados.rados.0.smithi115.stdout:813: done (0 left) 2022-04-22T17:28:08.065 INFO:tasks.rados.rados.0.smithi115.stdout:814: rmattr oid 41 current snap is 77 2022-04-22T17:28:08.066 INFO:tasks.rados.rados.0.smithi115.stdout:815: snap_remove snap 68 2022-04-22T17:28:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[12679]: cluster 2022-04-22T17:28:07.040125+0000 mon.a (mon.0) 12 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2022-04-22T17:28:08.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:07 smithi121 bash[34414]: debug 2022-04-22T17:28:07.931+0000 7fc2e6714000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:28:08.267 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[34414]: debug 2022-04-22T17:28:08.043+0000 7fc2e6714000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:28:08.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[18070]: cluster 2022-04-22T17:28:07.040125+0000 mon.a (mon.0) 12 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2022-04-22T17:28:08.546 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[42914]: debug 2022-04-22T17:28:08.108+0000 7f8f2901b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-22T17:28:08.546 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[42914]: debug 2022-04-22T17:28:08.212+0000 7f8f2901b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-22T17:28:08.711 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[34414]: debug 2022-04-22T17:28:08.311+0000 7fc2e6714000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:28:08.711 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[34414]: debug 2022-04-22T17:28:08.467+0000 7fc2e6714000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:28:08.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[42914]: debug 2022-04-22T17:28:08.472+0000 7f8f2901b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-22T17:28:08.810 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[42914]: debug 2022-04-22T17:28:08.628+0000 7f8f2901b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-22T17:28:09.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[34414]: debug 2022-04-22T17:28:08.703+0000 7fc2e6714000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:28:09.016 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:08 smithi121 bash[34414]: debug 2022-04-22T17:28:08.915+0000 7fc2e6714000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:28:09.069 INFO:tasks.rados.rados.0.smithi115.stdout:815: done (1 left) 2022-04-22T17:28:09.069 INFO:tasks.rados.rados.0.smithi115.stdout:816: setattr oid 12 current snap is 77 2022-04-22T17:28:09.069 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 195 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:09.072 INFO:tasks.rados.rados.0.smithi115.stdout:814: done (1 left) 2022-04-22T17:28:09.072 INFO:tasks.rados.rados.0.smithi115.stdout:817: rmattr oid 31 current snap is 77 2022-04-22T17:28:09.072 INFO:tasks.rados.rados.0.smithi115.stdout:817: done (1 left) 2022-04-22T17:28:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:818: rollback oid 42 current snap is 77 2022-04-22T17:28:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 75 2022-04-22T17:28:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:819: rollback oid 38 current snap is 77 2022-04-22T17:28:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 69 2022-04-22T17:28:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:820: snap_create 2022-04-22T17:28:09.087 INFO:tasks.rados.rados.0.smithi115.stdout:818: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:28:09.087 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 269 (ObjNum 223 snap 71 seq_num 223) dirty exists 2022-04-22T17:28:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:819: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:28:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 213 (ObjNum -1479265424 snap 32692 seq_num 2815701872) dirty dne 2022-04-22T17:28:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:818: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:28:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:818: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:28:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 451 (ObjNum 153 snap 41 seq_num 153) dirty exists 2022-04-22T17:28:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:09 smithi115 bash[18070]: cluster 2022-04-22T17:28:08.053266+0000 mon.a (mon.0) 13 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2022-04-22T17:28:09.298 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:08 smithi115 bash[42914]: debug 2022-04-22T17:28:08.852+0000 7f8f2901b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-22T17:28:09.298 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:09 smithi115 bash[42914]: debug 2022-04-22T17:28:09.076+0000 7f8f2901b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-22T17:28:09.393 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:09 smithi121 bash[12679]: cluster 2022-04-22T17:28:08.053266+0000 mon.a (mon.0) 13 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2022-04-22T17:28:09.393 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:09 smithi121 bash[34414]: debug 2022-04-22T17:28:09.143+0000 7fc2e6714000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:28:09.767 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:09 smithi121 bash[34414]: debug 2022-04-22T17:28:09.383+0000 7fc2e6714000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:28:09.767 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:09 smithi121 bash[34414]: debug 2022-04-22T17:28:09.607+0000 7fc2e6714000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:28:09.769 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:09 smithi115 bash[42914]: debug 2022-04-22T17:28:09.296+0000 7f8f2901b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:28:09.769 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:09 smithi115 bash[42914]: debug 2022-04-22T17:28:09.528+0000 7f8f2901b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-22T17:28:10.048 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:09 smithi115 bash[42914]: debug 2022-04-22T17:28:09.752+0000 7f8f2901b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-22T17:28:10.087 INFO:tasks.rados.rados.0.smithi115.stdout:816: done (3 left) 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:818: done (2 left) 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:819: done (1 left) 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:820: done (0 left) 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:821: rollback oid 43 current snap is 78 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 59 2022-04-22T17:28:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:822: snap_create 2022-04-22T17:28:10.094 INFO:tasks.rados.rados.0.smithi115.stdout:821: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:28:10.094 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 304 (ObjNum 288 snap 0 seq_num 273) dirty dne 2022-04-22T17:28:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:10 smithi121 bash[12679]: cluster 2022-04-22T17:28:09.057879+0000 mon.a (mon.0) 14 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2022-04-22T17:28:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:10 smithi115 bash[18070]: cluster 2022-04-22T17:28:09.057879+0000 mon.a (mon.0) 14 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2022-04-22T17:28:11.091 INFO:tasks.rados.rados.0.smithi115.stdout:821: done (1 left) 2022-04-22T17:28:11.092 INFO:tasks.rados.rados.0.smithi115.stdout:822: done (0 left) 2022-04-22T17:28:11.092 INFO:tasks.rados.rados.0.smithi115.stdout:823: rmattr oid 29 current snap is 79 2022-04-22T17:28:11.092 INFO:tasks.rados.rados.0.smithi115.stdout:824: snap_create 2022-04-22T17:28:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 227 (ObjNum 1835491670 snap 1768317019 seq_num 1734435432) dirty exists 2022-04-22T17:28:11.435 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:11 smithi121 bash[12679]: cluster 2022-04-22T17:28:10.076357+0000 mon.a (mon.0) 15 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2022-04-22T17:28:11.436 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:11 smithi121 bash[34414]: debug 2022-04-22T17:28:11.319+0000 7fc2e6714000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:28:11.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:11 smithi115 bash[18070]: cluster 2022-04-22T17:28:10.076357+0000 mon.a (mon.0) 15 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2022-04-22T17:28:11.766 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:11 smithi121 bash[34414]: debug 2022-04-22T17:28:11.427+0000 7fc2e6714000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:28:11.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:11 smithi115 bash[42914]: debug 2022-04-22T17:28:11.448+0000 7f8f2901b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-22T17:28:11.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:11 smithi115 bash[42914]: debug 2022-04-22T17:28:11.564+0000 7f8f2901b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-22T17:28:12.105 INFO:tasks.rados.rados.0.smithi115.stdout:823: done (1 left) 2022-04-22T17:28:12.105 INFO:tasks.rados.rados.0.smithi115.stdout:824: done (0 left) 2022-04-22T17:28:12.105 INFO:tasks.rados.rados.0.smithi115.stdout:825: delete oid 33 current snap is 80 2022-04-22T17:28:12.110 INFO:tasks.rados.rados.0.smithi115.stdout:825: done (0 left) 2022-04-22T17:28:12.110 INFO:tasks.rados.rados.0.smithi115.stdout:826: setattr oid 48 current snap is 80 2022-04-22T17:28:12.113 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 80 2022-04-22T17:28:12.113 INFO:tasks.rados.rados.0.smithi115.stdout:827: seq_num 244 ranges {2080768=491520,2572288=573440,3145728=458752,3604480=172032} 2022-04-22T17:28:12.130 INFO:tasks.rados.rados.0.smithi115.stdout:827: writing smithi11541303-2 from 2080768 to 2572288 tid 1 2022-04-22T17:28:12.133 INFO:tasks.rados.rados.0.smithi115.stdout:827: writing smithi11541303-2 from 2572288 to 3145728 tid 2 2022-04-22T17:28:12.136 INFO:tasks.rados.rados.0.smithi115.stdout:827: writing smithi11541303-2 from 3145728 to 3604480 tid 3 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:827: writing smithi11541303-2 from 3604480 to 3776512 tid 4 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:828: read oid 37 snap -1 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:828: expect (ObjNum 237 snap 74 seq_num 237) 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:829: copy_from oid 40 from oid 33 current snap is 80 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 368 (ObjNum 150 snap 40 seq_num 150) dirty exists 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:826: done (3 left) 2022-04-22T17:28:12.138 INFO:tasks.rados.rados.0.smithi115.stdout:830: snap_remove snap 75 2022-04-22T17:28:12.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:12 smithi121 bash[12679]: cluster 2022-04-22T17:28:11.079861+0000 mon.a (mon.0) 16 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2022-04-22T17:28:12.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:12 smithi115 bash[18070]: cluster 2022-04-22T17:28:11.079861+0000 mon.a (mon.0) 16 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2022-04-22T17:28:12.893 INFO:tasks.rados.rados.0.smithi115.stdout:830: done (3 left) 2022-04-22T17:28:12.939 INFO:tasks.rados.rados.0.smithi115.stdout:831: setattr oid 27 current snap is 80 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:829: finishing copy_from to smithi11541303-40 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:829: got expected ENOENT (src dne) 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 1 to smithi11541303-2 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:829: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 2 to smithi11541303-2 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 3 to smithi11541303-2 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 4 to smithi11541303-2 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 5 to smithi11541303-2 2022-04-22T17:28:12.940 INFO:tasks.rados.rados.0.smithi115.stdout:827: finishing write tid 6 to smithi11541303-2 2022-04-22T17:28:12.941 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 201 (ObjNum 244 snap 80 seq_num 244) dirty exists 2022-04-22T17:28:12.941 INFO:tasks.rados.rados.0.smithi115.stdout:827: left oid 2 (ObjNum 244 snap 80 seq_num 244) 2022-04-22T17:28:12.941 INFO:tasks.rados.rados.0.smithi115.stdout:828: expect (ObjNum 237 snap 74 seq_num 237) 2022-04-22T17:28:12.941 INFO:tasks.rados.rados.0.smithi115.stdout:827: done (3 left) 2022-04-22T17:28:12.941 INFO:tasks.rados.rados.0.smithi115.stdout:828: done (2 left) 2022-04-22T17:28:12.942 INFO:tasks.rados.rados.0.smithi115.stdout:829: done (1 left) 2022-04-22T17:28:12.942 INFO:tasks.rados.rados.0.smithi115.stdout:832: snap_create 2022-04-22T17:28:12.947 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:28:13.017 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 22 17:28:12 smithi121 bash[34414]: debug 2022-04-22T17:28:12.703+0000 7fc2e6714000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:28:13.122 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:12 smithi115 bash[42914]: debug 2022-04-22T17:28:12.844+0000 7f8f2901b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-22T17:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.094374+0000 mon.a (mon.0) 17 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2022-04-22T17:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.722010+0000 mon.a (mon.0) 18 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.722106+0000 mon.a (mon.0) 19 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.094374+0000 mon.a (mon.0) 17 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.722010+0000 mon.a (mon.0) 18 : cluster [DBG] Standby manager daemon x restarted 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.722106+0000 mon.a (mon.0) 19 : cluster [DBG] Standby manager daemon x started 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: audit 2022-04-22T17:28:12.732133+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: audit 2022-04-22T17:28:12.733585+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:28:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: audit 2022-04-22T17:28:12.735447+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:28:13.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: audit 2022-04-22T17:28:12.736332+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:28:13.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.869757+0000 mon.a (mon.0) 20 : cluster [INF] Active manager daemon y restarted 2022-04-22T17:28:13.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.870977+0000 mon.a (mon.0) 21 : cluster [INF] Activating manager daemon y 2022-04-22T17:28:13.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:13 smithi121 bash[12679]: cluster 2022-04-22T17:28:12.879907+0000 mon.a (mon.0) 22 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: audit 2022-04-22T17:28:12.732133+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: audit 2022-04-22T17:28:12.733585+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: audit 2022-04-22T17:28:12.735447+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: audit 2022-04-22T17:28:12.736332+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.121:0/1811415527' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.869757+0000 mon.a (mon.0) 20 : cluster [INF] Active manager daemon y restarted 2022-04-22T17:28:13.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.870977+0000 mon.a (mon.0) 21 : cluster [INF] Activating manager daemon y 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[18070]: cluster 2022-04-22T17:28:12.879907+0000 mon.a (mon.0) 22 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.280+0000 7f8e98109700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.280+0000 7f8e98109700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.280+0000 7f8e98109700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.280+0000 7f8e98109700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.280+0000 7f8e98109700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.284+0000 7f8e928fe700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.512 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.284+0000 7f8e928fe700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.513 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.284+0000 7f8e928fe700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.513 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.284+0000 7f8e928fe700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.513 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: debug 2022-04-22T17:28:13.284+0000 7f8e928fe700 -1 client.0 error registering admin socket command: (17) File exists 2022-04-22T17:28:13.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:28:13.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:28:13.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[42914]: [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:28:13.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.500Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused; Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.511Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.511Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" context_err="context deadline exceeded" 2022-04-22T17:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:13.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: dial tcp 172.21.15.121:8443: connect: connection refused; Post https://172.21.15.115:8443/api/prometheus_receiver: dial tcp 172.21.15.115:8443: connect: connection refused" 2022-04-22T17:28:13.892 INFO:tasks.rados.rados.0.smithi115.stdout:831: done (1 left) 2022-04-22T17:28:13.892 INFO:tasks.rados.rados.0.smithi115.stdout:832: done (0 left) 2022-04-22T17:28:13.892 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 81 2022-04-22T17:28:13.892 INFO:tasks.rados.rados.0.smithi115.stdout:833: seq_num 245 ranges {1581056=507904,2088960=458752,2547712=671744,3219456=73728} 2022-04-22T17:28:13.907 INFO:tasks.rados.rados.0.smithi115.stdout:833: writing smithi11541303-37 from 1581056 to 2088960 tid 1 2022-04-22T17:28:13.910 INFO:tasks.rados.rados.0.smithi115.stdout:833: writing smithi11541303-37 from 2088960 to 2547712 tid 2 2022-04-22T17:28:13.914 INFO:tasks.rados.rados.0.smithi115.stdout:833: writing smithi11541303-37 from 2547712 to 3219456 tid 3 2022-04-22T17:28:13.914 INFO:tasks.rados.rados.0.smithi115.stdout:833: writing smithi11541303-37 from 3219456 to 3293184 tid 4 2022-04-22T17:28:13.915 INFO:tasks.rados.rados.0.smithi115.stdout:834: snap_create 2022-04-22T17:28:13.920 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 1 to smithi11541303-37 2022-04-22T17:28:13.921 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 2 to smithi11541303-37 2022-04-22T17:28:13.921 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 3 to smithi11541303-37 2022-04-22T17:28:13.922 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 4 to smithi11541303-37 2022-04-22T17:28:13.922 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 5 to smithi11541303-37 2022-04-22T17:28:13.923 INFO:tasks.rados.rados.0.smithi115.stdout:833: finishing write tid 6 to smithi11541303-37 2022-04-22T17:28:13.924 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 209 (ObjNum 245 snap 81 seq_num 245) dirty exists 2022-04-22T17:28:13.924 INFO:tasks.rados.rados.0.smithi115.stdout:833: left oid 37 (ObjNum 245 snap 81 seq_num 245) 2022-04-22T17:28:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cluster 2022-04-22T17:28:13.105432+0000 mon.a (mon.0) 23 : cluster [DBG] mgrmap e27: y(active, starting, since 0.234662s), standbys: x 2022-04-22T17:28:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.110473+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.110696+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.110874+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.114970+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.115287+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.115565+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.115748+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:28:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.115926+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.116181+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.116397+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.116595+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.116804+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.117021+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.117225+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.117434+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:28:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.117645+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.117891+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.118384+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cluster 2022-04-22T17:28:13.148210+0000 mon.a (mon.0) 42 : cluster [INF] Manager daemon y is now available 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.187411+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.191434+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.194435+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.196645+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.222264+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.233152+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:28:14.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cephadm 2022-04-22T17:28:13.583710+0000 mgr.y (mgr.34107) 1 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:28:14.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cephadm 2022-04-22T17:28:13.710907+0000 mgr.y (mgr.34107) 2 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:28:14.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cephadm 2022-04-22T17:28:13.711621+0000 mgr.y (mgr.34107) 3 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:28:14.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: audit 2022-04-22T17:28:13.719771+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:14.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:14 smithi121 bash[12679]: cluster 2022-04-22T17:28:13.880758+0000 mon.a (mon.0) 50 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cluster 2022-04-22T17:28:13.105432+0000 mon.a (mon.0) 23 : cluster [DBG] mgrmap e27: y(active, starting, since 0.234662s), standbys: x 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.110473+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.110696+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.110874+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.114970+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:28:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.115287+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.115565+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.115748+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.115926+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.116181+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.116397+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.116595+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.116804+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:28:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.117021+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.117225+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.117434+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.117645+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.117891+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.118384+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cluster 2022-04-22T17:28:13.148210+0000 mon.a (mon.0) 42 : cluster [INF] Manager daemon y is now available 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.187411+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.191434+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.194435+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.196645+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.222264+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.233152+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cephadm 2022-04-22T17:28:13.583710+0000 mgr.y (mgr.34107) 1 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cephadm 2022-04-22T17:28:13.710907+0000 mgr.y (mgr.34107) 2 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cephadm 2022-04-22T17:28:13.711621+0000 mgr.y (mgr.34107) 3 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: audit 2022-04-22T17:28:13.719771+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:14 smithi115 bash[18070]: cluster 2022-04-22T17:28:13.880758+0000 mon.a (mon.0) 50 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2022-04-22T17:28:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:833: done (1 left) 2022-04-22T17:28:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:834: done (0 left) 2022-04-22T17:28:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:835: read oid 11 snap 79 2022-04-22T17:28:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:835: expect (ObjNum 220 snap 71 seq_num 220) 2022-04-22T17:28:14.897 INFO:tasks.rados.rados.0.smithi115.stdout:836: snap_create 2022-04-22T17:28:14.908 INFO:tasks.rados.rados.0.smithi115.stdout:835: expect (ObjNum 220 snap 71 seq_num 220) 2022-04-22T17:28:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:15 smithi121 bash[12679]: cluster 2022-04-22T17:28:14.131628+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e28: y(active, since 1.26087s), standbys: x 2022-04-22T17:28:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:15 smithi121 bash[12679]: cluster 2022-04-22T17:28:14.141827+0000 mgr.y (mgr.34107) 4 : cluster [DBG] pgmap v4: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:15 smithi121 bash[12679]: cluster 2022-04-22T17:28:14.883802+0000 mon.a (mon.0) 52 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2022-04-22T17:28:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:15 smithi115 bash[18070]: cluster 2022-04-22T17:28:14.131628+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e28: y(active, since 1.26087s), standbys: x 2022-04-22T17:28:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:15 smithi115 bash[18070]: cluster 2022-04-22T17:28:14.141827+0000 mgr.y (mgr.34107) 4 : cluster [DBG] pgmap v4: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:15 smithi115 bash[18070]: cluster 2022-04-22T17:28:14.883802+0000 mon.a (mon.0) 52 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2022-04-22T17:28:15.913 INFO:tasks.rados.rados.0.smithi115.stdout:835: done (1 left) 2022-04-22T17:28:15.913 INFO:tasks.rados.rados.0.smithi115.stdout:836: done (0 left) 2022-04-22T17:28:15.913 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 83 2022-04-22T17:28:15.913 INFO:tasks.rados.rados.0.smithi115.stdout:837: seq_num 246 ranges {3801088=524288,4325376=745472,5070848=425984,5496832=32768} 2022-04-22T17:28:15.938 INFO:tasks.rados.rados.0.smithi115.stdout:837: writing smithi11541303-10 from 3801088 to 4325376 tid 1 2022-04-22T17:28:15.942 INFO:tasks.rados.rados.0.smithi115.stdout:837: writing smithi11541303-10 from 4325376 to 5070848 tid 2 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:837: writing smithi11541303-10 from 5070848 to 5496832 tid 3 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:837: writing smithi11541303-10 from 5496832 to 5529600 tid 4 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:838: copy_from oid 5 from oid 16 current snap is 83 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:839: copy_from oid 36 from oid 3 current snap is 83 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:840: read oid 6 snap -1 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:840: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:841: rollback oid 23 current snap is 83 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 82 2022-04-22T17:28:15.945 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 83 2022-04-22T17:28:15.946 INFO:tasks.rados.rados.0.smithi115.stdout:842: seq_num 247 ranges {2113536=540672,2654208=622592,3276800=581632} 2022-04-22T17:28:15.960 INFO:tasks.rados.rados.0.smithi115.stdout:842: writing smithi11541303-11 from 2113536 to 2654208 tid 1 2022-04-22T17:28:15.964 INFO:tasks.rados.rados.0.smithi115.stdout:842: writing smithi11541303-11 from 2654208 to 3276800 tid 2 2022-04-22T17:28:15.967 INFO:tasks.rados.rados.0.smithi115.stdout:842: writing smithi11541303-11 from 3276800 to 3858432 tid 3 2022-04-22T17:28:15.968 INFO:tasks.rados.rados.0.smithi115.stdout:843: snap_remove snap 71 2022-04-22T17:28:16.913 INFO:tasks.rados.rados.0.smithi115.stdout:843: done (6 left) 2022-04-22T17:28:16.913 INFO:tasks.rados.rados.0.smithi115.stdout:844: setattr oid 4 current snap is 83 2022-04-22T17:28:16.913 INFO:tasks.rados.rados.0.smithi115.stdout:840: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:16.928 INFO:tasks.rados.rados.0.smithi115.stdout:840: done (6 left) 2022-04-22T17:28:16.928 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 83 2022-04-22T17:28:16.928 INFO:tasks.rados.rados.0.smithi115.stdout:845: seq_num 248 ranges {4022272=557056,4579328=507904,5087232=442368,5529600=253952} 2022-04-22T17:28:16.956 INFO:tasks.rados.rados.0.smithi115.stdout:845: writing smithi11541303-30 from 4022272 to 4579328 tid 1 2022-04-22T17:28:16.959 INFO:tasks.rados.rados.0.smithi115.stdout:845: writing smithi11541303-30 from 4579328 to 5087232 tid 2 2022-04-22T17:28:16.961 INFO:tasks.rados.rados.0.smithi115.stdout:845: writing smithi11541303-30 from 5087232 to 5529600 tid 3 2022-04-22T17:28:16.966 INFO:tasks.rados.rados.0.smithi115.stdout:845: writing smithi11541303-30 from 5529600 to 5783552 tid 4 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:846: delete oid 39 current snap is 83 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 1 to smithi11541303-10 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:841: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:838: finishing copy_from to smithi11541303-5 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:838: got expected ENOENT (src dne) 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:838: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:841: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:841: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:28:16.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 308 (ObjNum 233 snap 73 seq_num 233) dirty exists 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 2 to smithi11541303-10 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 3 to smithi11541303-10 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 4 to smithi11541303-10 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 5 to smithi11541303-10 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:839: finishing copy_from to smithi11541303-36 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 241 (ObjNum 6 snap 0 seq_num 6) dirty exists 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:839: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: finishing write tid 6 to smithi11541303-10 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 207 (ObjNum 246 snap 83 seq_num 246) dirty exists 2022-04-22T17:28:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:837: left oid 10 (ObjNum 246 snap 83 seq_num 246) 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: finishing write tid 1 to smithi11541303-11 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: finishing write tid 2 to smithi11541303-11 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: finishing write tid 3 to smithi11541303-11 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: finishing write tid 4 to smithi11541303-11 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: finishing write tid 5 to smithi11541303-11 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 313 (ObjNum 247 snap 83 seq_num 247) dirty exists 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:842: left oid 11 (ObjNum 247 snap 83 seq_num 247) 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 384 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 1 to smithi11541303-30 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:837: done (7 left) 2022-04-22T17:28:16.969 INFO:tasks.rados.rados.0.smithi115.stdout:838: done (6 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:839: done (5 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:841: done (4 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:842: done (3 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:844: done (2 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:846: done (1 left) 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 83 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:847: seq_num 250 ranges {114688=589824,704512=679936,1384448=409600,1794048=114688} 2022-04-22T17:28:16.970 INFO:tasks.rados.rados.0.smithi115.stdout:847: writing smithi11541303-48 from 114688 to 704512 tid 1 2022-04-22T17:28:16.974 INFO:tasks.rados.rados.0.smithi115.stdout:847: writing smithi11541303-48 from 704512 to 1384448 tid 2 2022-04-22T17:28:16.976 INFO:tasks.rados.rados.0.smithi115.stdout:847: writing smithi11541303-48 from 1384448 to 1794048 tid 3 2022-04-22T17:28:16.977 INFO:tasks.rados.rados.0.smithi115.stdout:847: writing smithi11541303-48 from 1794048 to 1908736 tid 4 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:848: snap_create 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 2 to smithi11541303-30 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 3 to smithi11541303-30 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 4 to smithi11541303-30 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 5 to smithi11541303-30 2022-04-22T17:28:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 1 to smithi11541303-48 2022-04-22T17:28:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:845: finishing write tid 6 to smithi11541303-30 2022-04-22T17:28:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 390 (ObjNum 248 snap 83 seq_num 248) dirty exists 2022-04-22T17:28:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:845: left oid 30 (ObjNum 248 snap 83 seq_num 248) 2022-04-22T17:28:16.982 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 2 to smithi11541303-48 2022-04-22T17:28:16.984 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 3 to smithi11541303-48 2022-04-22T17:28:16.984 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 4 to smithi11541303-48 2022-04-22T17:28:16.985 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 5 to smithi11541303-48 2022-04-22T17:28:16.986 INFO:tasks.rados.rados.0.smithi115.stdout:847: finishing write tid 6 to smithi11541303-48 2022-04-22T17:28:16.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 396 (ObjNum 250 snap 83 seq_num 250) dirty exists 2022-04-22T17:28:16.986 INFO:tasks.rados.rados.0.smithi115.stdout:847: left oid 48 (ObjNum 250 snap 83 seq_num 250) 2022-04-22T17:28:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:16 smithi121 bash[12679]: cluster 2022-04-22T17:28:15.114859+0000 mgr.y (mgr.34107) 5 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:17.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:16 smithi121 bash[12679]: cluster 2022-04-22T17:28:15.889312+0000 mon.a (mon.0) 53 : cluster [DBG] mgrmap e29: y(active, since 3s), standbys: x 2022-04-22T17:28:17.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:16 smithi121 bash[12679]: cluster 2022-04-22T17:28:15.901958+0000 mon.a (mon.0) 54 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2022-04-22T17:28:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:16 smithi115 bash[18070]: cluster 2022-04-22T17:28:15.114859+0000 mgr.y (mgr.34107) 5 : cluster [DBG] pgmap v6: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:16 smithi115 bash[18070]: cluster 2022-04-22T17:28:15.889312+0000 mon.a (mon.0) 53 : cluster [DBG] mgrmap e29: y(active, since 3s), standbys: x 2022-04-22T17:28:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:16 smithi115 bash[18070]: cluster 2022-04-22T17:28:15.901958+0000 mon.a (mon.0) 54 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2022-04-22T17:28:17.923 INFO:tasks.rados.rados.0.smithi115.stdout:845: done (2 left) 2022-04-22T17:28:17.923 INFO:tasks.rados.rados.0.smithi115.stdout:847: done (1 left) 2022-04-22T17:28:17.923 INFO:tasks.rados.rados.0.smithi115.stdout:848: done (0 left) 2022-04-22T17:28:17.923 INFO:tasks.rados.rados.0.smithi115.stdout:849: snap_remove snap 54 2022-04-22T17:28:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:17 smithi121 bash[12679]: cluster 2022-04-22T17:28:16.902197+0000 mon.a (mon.0) 55 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2022-04-22T17:28:18.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:17 smithi115 bash[18070]: cluster 2022-04-22T17:28:16.902197+0000 mon.a (mon.0) 55 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2022-04-22T17:28:18.928 INFO:tasks.rados.rados.0.smithi115.stdout:849: done (0 left) 2022-04-22T17:28:18.928 INFO:tasks.rados.rados.0.smithi115.stdout:850: rmattr oid 46 current snap is 84 2022-04-22T17:28:18.929 INFO:tasks.rados.rados.0.smithi115.stdout:850: done (0 left) 2022-04-22T17:28:18.929 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 84 2022-04-22T17:28:18.929 INFO:tasks.rados.rados.0.smithi115.stdout:851: seq_num 251 ranges {0=606208,606208=614400,1220608=589824} 2022-04-22T17:28:18.930 INFO:tasks.rados.rados.0.smithi115.stdout:851: writing smithi11541303-21 from 0 to 606208 tid 1 2022-04-22T17:28:18.934 INFO:tasks.rados.rados.0.smithi115.stdout:851: writing smithi11541303-21 from 606208 to 1220608 tid 2 2022-04-22T17:28:18.939 INFO:tasks.rados.rados.0.smithi115.stdout:851: writing smithi11541303-21 from 1220608 to 1810432 tid 3 2022-04-22T17:28:18.940 INFO:tasks.rados.rados.0.smithi115.stdout:852: delete oid 39 current snap is 84 2022-04-22T17:28:18.946 INFO:tasks.rados.rados.0.smithi115.stdout:851: finishing write tid 1 to smithi11541303-21 2022-04-22T17:28:18.949 INFO:tasks.rados.rados.0.smithi115.stdout:851: finishing write tid 2 to smithi11541303-21 2022-04-22T17:28:18.952 INFO:tasks.rados.rados.0.smithi115.stdout:852: done (1 left) 2022-04-22T17:28:18.952 INFO:tasks.rados.rados.0.smithi115.stdout:853: read oid 49 snap -1 2022-04-22T17:28:18.952 INFO:tasks.rados.rados.0.smithi115.stdout:853: expect deleted 2022-04-22T17:28:18.952 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 84 2022-04-22T17:28:18.952 INFO:tasks.rados.rados.0.smithi115.stdout:854: seq_num 253 ranges {0=647168,647168=786432,1433600=417792} 2022-04-22T17:28:18.955 INFO:tasks.rados.rados.0.smithi115.stdout:854: writing smithi11541303-28 from 0 to 647168 tid 1 2022-04-22T17:28:18.961 INFO:tasks.rados.rados.0.smithi115.stdout:854: writing smithi11541303-28 from 647168 to 1433600 tid 2 2022-04-22T17:28:18.964 INFO:tasks.rados.rados.0.smithi115.stdout:854: writing smithi11541303-28 from 1433600 to 1851392 tid 3 2022-04-22T17:28:18.964 INFO:tasks.rados.rados.0.smithi115.stdout:851: finishing write tid 3 to smithi11541303-21 2022-04-22T17:28:18.964 INFO:tasks.rados.rados.0.smithi115.stdout:851: finishing write tid 4 to smithi11541303-21 2022-04-22T17:28:18.964 INFO:tasks.rados.rados.0.smithi115.stdout:854: finishing write tid 1 to smithi11541303-28 2022-04-22T17:28:18.964 INFO:tasks.rados.rados.0.smithi115.stdout:851: finishing write tid 5 to smithi11541303-21 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 183 (ObjNum 251 snap 84 seq_num 251) dirty exists 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:851: left oid 21 (ObjNum 251 snap 84 seq_num 251) 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:851: done (2 left) 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:853: done (1 left) 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:855: read oid 8 snap -1 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:855: expect deleted 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:856: rmattr oid 21 current snap is 84 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:856: done (2 left) 2022-04-22T17:28:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:857: delete oid 7 current snap is 84 2022-04-22T17:28:18.967 INFO:tasks.rados.rados.0.smithi115.stdout:854: finishing write tid 2 to smithi11541303-28 2022-04-22T17:28:18.968 INFO:tasks.rados.rados.0.smithi115.stdout:855: done (2 left) 2022-04-22T17:28:18.968 INFO:tasks.rados.rados.0.smithi115.stdout:857: done (1 left) 2022-04-22T17:28:18.968 INFO:tasks.rados.rados.0.smithi115.stdout:858: copy_from oid 32 from oid 19 current snap is 84 2022-04-22T17:28:18.968 INFO:tasks.rados.rados.0.smithi115.stdout:854: finishing write tid 3 to smithi11541303-28 2022-04-22T17:28:18.969 INFO:tasks.rados.rados.0.smithi115.stdout:859: snap_remove snap 72 2022-04-22T17:28:19.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:18 smithi115 bash[18070]: cluster 2022-04-22T17:28:17.115631+0000 mgr.y (mgr.34107) 6 : cluster [DBG] pgmap v9: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:19.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:18 smithi115 bash[18070]: cluster 2022-04-22T17:28:17.912263+0000 mon.a (mon.0) 56 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2022-04-22T17:28:19.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:18 smithi115 bash[18070]: audit 2022-04-22T17:28:18.450121+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:19.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:18 smithi115 bash[18070]: audit 2022-04-22T17:28:18.844925+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:19.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:18 smithi121 bash[12679]: cluster 2022-04-22T17:28:17.115631+0000 mgr.y (mgr.34107) 6 : cluster [DBG] pgmap v9: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:19.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:18 smithi121 bash[12679]: cluster 2022-04-22T17:28:17.912263+0000 mon.a (mon.0) 56 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2022-04-22T17:28:19.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:18 smithi121 bash[12679]: audit 2022-04-22T17:28:18.450121+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:19.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:18 smithi121 bash[12679]: audit 2022-04-22T17:28:18.844925+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:19.179 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:28:18 smithi121 bash[21417]: debug 2022-04-22T17:28:18.943+0000 7fcbfeaf2700 -1 osd.6 pg_epoch: 223 pg[5.2s0( v 223'345 (0'0,223'345] local-lis/les=63/64 n=12 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [6,0,5]p6(0) r=0 lpr=63 luod=223'343 crt=223'343 lcod 223'341 mlcod 223'341 active+clean+snaptrim trimq=[67~1] ps=[75~1,77~1,7d~2,80~1,8b~1,8d~1,90~1]] removing snap head 2022-04-22T17:28:19.723 INFO:tasks.rados.rados.0.smithi115.stdout:859: done (2 left) 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:860: snap_create 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:854: finishing write tid 4 to smithi11541303-28 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:854: finishing write tid 5 to smithi11541303-28 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 231 (ObjNum 253 snap 84 seq_num 253) dirty exists 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:854: left oid 28 (ObjNum 253 snap 84 seq_num 253) 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:858: finishing copy_from to smithi11541303-32 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 99 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:858: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:28:20.016 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[21417]: debug 2022-04-22T17:28:19.727+0000 7fcbfeaf2700 -1 osd.6 pg_epoch: 224 pg[5.2s0( v 224'349 (0'0,224'349] local-lis/les=63/64 n=10 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [6,0,5]p6(0) r=0 lpr=63 luod=223'347 crt=223'347 lcod 223'345 mlcod 223'345 active+clean+snaptrim trimq=[8c~1] ps=[67~1,8b~1]] removing snap head 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: cluster 2022-04-22T17:28:18.913690+0000 mon.a (mon.0) 59 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: audit 2022-04-22T17:28:19.082315+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: audit 2022-04-22T17:28:19.136160+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: audit 2022-04-22T17:28:19.430402+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: audit 2022-04-22T17:28:19.561515+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:19 smithi121 bash[12679]: cluster 2022-04-22T17:28:19.710181+0000 mon.a (mon.0) 64 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2022-04-22T17:28:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: cluster 2022-04-22T17:28:18.913690+0000 mon.a (mon.0) 59 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2022-04-22T17:28:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: audit 2022-04-22T17:28:19.082315+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: audit 2022-04-22T17:28:19.136160+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: audit 2022-04-22T17:28:19.430402+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: audit 2022-04-22T17:28:19.561515+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:19 smithi115 bash[18070]: cluster 2022-04-22T17:28:19.710181+0000 mon.a (mon.0) 64 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2022-04-22T17:28:20.740 INFO:tasks.rados.rados.0.smithi115.stdout:854: done (2 left) 2022-04-22T17:28:20.740 INFO:tasks.rados.rados.0.smithi115.stdout:858: done (1 left) 2022-04-22T17:28:20.741 INFO:tasks.rados.rados.0.smithi115.stdout:860: done (0 left) 2022-04-22T17:28:20.741 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 85 2022-04-22T17:28:20.741 INFO:tasks.rados.rados.0.smithi115.stdout:861: seq_num 255 ranges {1810432=679936,2490368=548864,3039232=598016,3637248=57344} 2022-04-22T17:28:20.755 INFO:tasks.rados.rados.0.smithi115.stdout:861: writing smithi11541303-21 from 1810432 to 2490368 tid 1 2022-04-22T17:28:20.758 INFO:tasks.rados.rados.0.smithi115.stdout:861: writing smithi11541303-21 from 2490368 to 3039232 tid 2 2022-04-22T17:28:20.761 INFO:tasks.rados.rados.0.smithi115.stdout:861: writing smithi11541303-21 from 3039232 to 3637248 tid 3 2022-04-22T17:28:20.762 INFO:tasks.rados.rados.0.smithi115.stdout:861: writing smithi11541303-21 from 3637248 to 3694592 tid 4 2022-04-22T17:28:20.762 INFO:tasks.rados.rados.0.smithi115.stdout:862: rollback oid 42 current snap is 85 2022-04-22T17:28:20.762 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 76 2022-04-22T17:28:20.763 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 85 2022-04-22T17:28:20.763 INFO:tasks.rados.rados.0.smithi115.stdout:863: seq_num 256 ranges {270336=696320,966656=434176,1400832=770048} 2022-04-22T17:28:20.767 INFO:tasks.rados.rados.0.smithi115.stdout:863: writing smithi11541303-47 from 270336 to 966656 tid 1 2022-04-22T17:28:20.770 INFO:tasks.rados.rados.0.smithi115.stdout:863: writing smithi11541303-47 from 966656 to 1400832 tid 2 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:863: writing smithi11541303-47 from 1400832 to 2170880 tid 3 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:864: copy_from oid 13 from oid 7 current snap is 85 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 1 to smithi11541303-21 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 2 to smithi11541303-21 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 3 to smithi11541303-21 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 4 to smithi11541303-21 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 5 to smithi11541303-21 2022-04-22T17:28:20.776 INFO:tasks.rados.rados.0.smithi115.stdout:861: finishing write tid 6 to smithi11541303-21 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 189 (ObjNum 255 snap 85 seq_num 255) dirty exists 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:861: left oid 21 (ObjNum 255 snap 85 seq_num 255) 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:862: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:861: done (3 left) 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:865: snap_create 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:862: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:28:20.777 INFO:tasks.rados.rados.0.smithi115.stdout:862: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:28:20.778 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 475 (ObjNum 153 snap 41 seq_num 153) dirty exists 2022-04-22T17:28:20.779 INFO:tasks.rados.rados.0.smithi115.stdout:864: finishing copy_from to smithi11541303-13 2022-04-22T17:28:20.779 INFO:tasks.rados.rados.0.smithi115.stdout:864: got expected ENOENT (src dne) 2022-04-22T17:28:20.780 INFO:tasks.rados.rados.0.smithi115.stdout:864: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:28:20.780 INFO:tasks.rados.rados.0.smithi115.stdout:863: finishing write tid 1 to smithi11541303-47 2022-04-22T17:28:20.780 INFO:tasks.rados.rados.0.smithi115.stdout:863: finishing write tid 2 to smithi11541303-47 2022-04-22T17:28:20.781 INFO:tasks.rados.rados.0.smithi115.stdout:863: finishing write tid 3 to smithi11541303-47 2022-04-22T17:28:20.781 INFO:tasks.rados.rados.0.smithi115.stdout:863: finishing write tid 4 to smithi11541303-47 2022-04-22T17:28:20.782 INFO:tasks.rados.rados.0.smithi115.stdout:863: finishing write tid 5 to smithi11541303-47 2022-04-22T17:28:20.782 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 226 (ObjNum 256 snap 85 seq_num 256) dirty exists 2022-04-22T17:28:20.782 INFO:tasks.rados.rados.0.smithi115.stdout:863: left oid 47 (ObjNum 256 snap 85 seq_num 256) 2022-04-22T17:28:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:20 smithi121 bash[12679]: cephadm 2022-04-22T17:28:19.084862+0000 mgr.y (mgr.34107) 7 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:28:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:20 smithi121 bash[12679]: cluster 2022-04-22T17:28:19.116107+0000 mgr.y (mgr.34107) 8 : cluster [DBG] pgmap v12: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:21.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:20 smithi121 bash[12679]: cephadm 2022-04-22T17:28:19.327028+0000 mgr.y (mgr.34107) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:28:21.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:20 smithi121 bash[12679]: cluster 2022-04-22T17:28:20.728333+0000 mon.a (mon.0) 65 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2022-04-22T17:28:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:20 smithi115 bash[18070]: cephadm 2022-04-22T17:28:19.084862+0000 mgr.y (mgr.34107) 7 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:28:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:20 smithi115 bash[18070]: cluster 2022-04-22T17:28:19.116107+0000 mgr.y (mgr.34107) 8 : cluster [DBG] pgmap v12: 121 pgs: 121 active+clean; 177 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:20 smithi115 bash[18070]: cephadm 2022-04-22T17:28:19.327028+0000 mgr.y (mgr.34107) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:28:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:20 smithi115 bash[18070]: cluster 2022-04-22T17:28:20.728333+0000 mon.a (mon.0) 65 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2022-04-22T17:28:21.741 INFO:tasks.rados.rados.0.smithi115.stdout:862: done (3 left) 2022-04-22T17:28:21.741 INFO:tasks.rados.rados.0.smithi115.stdout:863: done (2 left) 2022-04-22T17:28:21.741 INFO:tasks.rados.rados.0.smithi115.stdout:864: done (1 left) 2022-04-22T17:28:21.742 INFO:tasks.rados.rados.0.smithi115.stdout:865: done (0 left) 2022-04-22T17:28:21.742 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 86 2022-04-22T17:28:21.742 INFO:tasks.rados.rados.0.smithi115.stdout:866: seq_num 257 ranges {2228224=712704,2940928=720896,3661824=483328} 2022-04-22T17:28:21.761 INFO:tasks.rados.rados.0.smithi115.stdout:866: writing smithi11541303-15 from 2228224 to 2940928 tid 1 2022-04-22T17:28:21.766 INFO:tasks.rados.rados.0.smithi115.stdout:866: writing smithi11541303-15 from 2940928 to 3661824 tid 2 2022-04-22T17:28:21.769 INFO:tasks.rados.rados.0.smithi115.stdout:866: writing smithi11541303-15 from 3661824 to 4145152 tid 3 2022-04-22T17:28:21.769 INFO:tasks.rados.rados.0.smithi115.stdout:867: delete oid 31 current snap is 86 2022-04-22T17:28:21.770 INFO:tasks.rados.rados.0.smithi115.stdout:866: finishing write tid 1 to smithi11541303-15 2022-04-22T17:28:21.775 INFO:tasks.rados.rados.0.smithi115.stdout:867: done (1 left) 2022-04-22T17:28:21.776 INFO:tasks.rados.rados.0.smithi115.stdout:868: snap_create 2022-04-22T17:28:21.776 INFO:tasks.rados.rados.0.smithi115.stdout:866: finishing write tid 2 to smithi11541303-15 2022-04-22T17:28:21.776 INFO:tasks.rados.rados.0.smithi115.stdout:866: finishing write tid 3 to smithi11541303-15 2022-04-22T17:28:21.777 INFO:tasks.rados.rados.0.smithi115.stdout:866: finishing write tid 4 to smithi11541303-15 2022-04-22T17:28:21.780 INFO:tasks.rados.rados.0.smithi115.stdout:866: finishing write tid 5 to smithi11541303-15 2022-04-22T17:28:21.780 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 186 (ObjNum 257 snap 86 seq_num 257) dirty exists 2022-04-22T17:28:21.780 INFO:tasks.rados.rados.0.smithi115.stdout:866: left oid 15 (ObjNum 257 snap 86 seq_num 257) 2022-04-22T17:28:22.751 INFO:tasks.rados.rados.0.smithi115.stdout:866: done (1 left) 2022-04-22T17:28:22.751 INFO:tasks.rados.rados.0.smithi115.stdout:868: done (0 left) 2022-04-22T17:28:22.751 INFO:tasks.rados.rados.0.smithi115.stdout:869: copy_from oid 33 from oid 44 current snap is 87 2022-04-22T17:28:22.751 INFO:tasks.rados.rados.0.smithi115.stdout:870: delete oid 3 current snap is 87 2022-04-22T17:28:22.756 INFO:tasks.rados.rados.0.smithi115.stdout:869: finishing copy_from to smithi11541303-33 2022-04-22T17:28:22.756 INFO:tasks.rados.rados.0.smithi115.stdout:869: got expected ENOENT (src dne) 2022-04-22T17:28:22.756 INFO:tasks.rados.rados.0.smithi115.stdout:869: finishing copy_from racing read to smithi11541303-33 2022-04-22T17:28:22.758 INFO:tasks.rados.rados.0.smithi115.stdout:869: done (1 left) 2022-04-22T17:28:22.758 INFO:tasks.rados.rados.0.smithi115.stdout:870: done (0 left) 2022-04-22T17:28:22.758 INFO:tasks.rados.rados.0.smithi115.stdout:871: snap_remove snap 84 2022-04-22T17:28:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:22 smithi121 bash[12679]: cluster 2022-04-22T17:28:21.116878+0000 mgr.y (mgr.34107) 10 : cluster [DBG] pgmap v15: 121 pgs: 121 active+clean; 169 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 17 op/s 2022-04-22T17:28:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:22 smithi121 bash[12679]: cluster 2022-04-22T17:28:21.728938+0000 mon.a (mon.0) 66 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2022-04-22T17:28:23.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:22 smithi115 bash[18070]: cluster 2022-04-22T17:28:21.116878+0000 mgr.y (mgr.34107) 10 : cluster [DBG] pgmap v15: 121 pgs: 121 active+clean; 169 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 17 op/s 2022-04-22T17:28:23.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:22 smithi115 bash[18070]: cluster 2022-04-22T17:28:21.728938+0000 mon.a (mon.0) 66 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2022-04-22T17:28:23.758 INFO:tasks.rados.rados.0.smithi115.stdout:871: done (0 left) 2022-04-22T17:28:23.758 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 87 2022-04-22T17:28:23.758 INFO:tasks.rados.rados.0.smithi115.stdout:872: seq_num 260 ranges {0=761856,761856=417792,1179648=786432} 2022-04-22T17:28:23.767 INFO:tasks.rados.rados.0.smithi115.stdout:872: writing smithi11541303-34 from 0 to 761856 tid 1 2022-04-22T17:28:23.770 INFO:tasks.rados.rados.0.smithi115.stdout:872: writing smithi11541303-34 from 761856 to 1179648 tid 2 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:872: writing smithi11541303-34 from 1179648 to 1966080 tid 3 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:873: read oid 25 snap -1 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:873: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:874: rollback oid 4 current snap is 87 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 77 2022-04-22T17:28:23.776 INFO:tasks.rados.rados.0.smithi115.stdout:875: snap_remove snap 85 2022-04-22T17:28:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[18070]: cluster 2022-04-22T17:28:22.739704+0000 mon.a (mon.0) 67 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2022-04-22T17:28:23.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.500Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.500Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:23.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:23 smithi121 bash[12679]: cluster 2022-04-22T17:28:22.739704+0000 mon.a (mon.0) 67 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2022-04-22T17:28:24.726 INFO:tasks.rados.rados.0.smithi115.stdout:875: done (3 left) 2022-04-22T17:28:24.727 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 87 2022-04-22T17:28:24.727 INFO:tasks.rados.rados.0.smithi115.stdout:876: seq_num 261 ranges {7045120=778240,7823360=704512,8527872=499712} 2022-04-22T17:28:24.776 INFO:tasks.rados.rados.0.smithi115.stdout:876: writing smithi11541303-42 from 7045120 to 7823360 tid 1 2022-04-22T17:28:24.780 INFO:tasks.rados.rados.0.smithi115.stdout:876: writing smithi11541303-42 from 7823360 to 8527872 tid 2 2022-04-22T17:28:24.784 INFO:tasks.rados.rados.0.smithi115.stdout:876: writing smithi11541303-42 from 8527872 to 9027584 tid 3 2022-04-22T17:28:24.784 INFO:tasks.rados.rados.0.smithi115.stdout:877: read oid 5 snap -1 2022-04-22T17:28:24.784 INFO:tasks.rados.rados.0.smithi115.stdout:877: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:24.784 INFO:tasks.rados.rados.0.smithi115.stdout:878: setattr oid 32 current snap is 87 2022-04-22T17:28:24.784 INFO:tasks.rados.rados.0.smithi115.stdout:873: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:24.791 INFO:tasks.rados.rados.0.smithi115.stdout:872: finishing write tid 1 to smithi11541303-34 2022-04-22T17:28:24.791 INFO:tasks.rados.rados.0.smithi115.stdout:872: finishing write tid 2 to smithi11541303-34 2022-04-22T17:28:24.791 INFO:tasks.rados.rados.0.smithi115.stdout:872: finishing write tid 3 to smithi11541303-34 2022-04-22T17:28:24.791 INFO:tasks.rados.rados.0.smithi115.stdout:872: finishing write tid 4 to smithi11541303-34 2022-04-22T17:28:24.791 INFO:tasks.rados.rados.0.smithi115.stdout:872: finishing write tid 5 to smithi11541303-34 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 204 (ObjNum 260 snap 87 seq_num 260) dirty exists 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:872: left oid 34 (ObjNum 260 snap 87 seq_num 260) 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:874: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:874: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:874: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 418 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:876: finishing write tid 1 to smithi11541303-42 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:876: finishing write tid 2 to smithi11541303-42 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:876: finishing write tid 3 to smithi11541303-42 2022-04-22T17:28:24.792 INFO:tasks.rados.rados.0.smithi115.stdout:872: done (5 left) 2022-04-22T17:28:24.793 INFO:tasks.rados.rados.0.smithi115.stdout:873: done (4 left) 2022-04-22T17:28:24.793 INFO:tasks.rados.rados.0.smithi115.stdout:874: done (3 left) 2022-04-22T17:28:24.793 INFO:tasks.rados.rados.0.smithi115.stdout:879: delete oid 12 current snap is 87 2022-04-22T17:28:24.793 INFO:tasks.rados.rados.0.smithi115.stdout:876: finishing write tid 4 to smithi11541303-42 2022-04-22T17:28:24.796 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 207 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:24.801 INFO:tasks.rados.rados.0.smithi115.stdout:878: done (3 left) 2022-04-22T17:28:24.801 INFO:tasks.rados.rados.0.smithi115.stdout:879: done (2 left) 2022-04-22T17:28:24.801 INFO:tasks.rados.rados.0.smithi115.stdout:880: rollback oid 4 current snap is 87 2022-04-22T17:28:24.801 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 86 2022-04-22T17:28:24.801 INFO:tasks.rados.rados.0.smithi115.stdout:881: rollback oid 21 current snap is 87 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 77 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:882: rmattr oid 30 current snap is 87 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:882: done (4 left) 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:883: read oid 20 snap -1 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:883: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:28:24.802 INFO:tasks.rados.rados.0.smithi115.stdout:884: snap_create 2022-04-22T17:28:24.803 INFO:tasks.rados.rados.0.smithi115.stdout:876: finishing write tid 5 to smithi11541303-42 2022-04-22T17:28:24.803 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 488 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:28:24.804 INFO:tasks.rados.rados.0.smithi115.stdout:876: left oid 42 (ObjNum 261 snap 87 seq_num 261) 2022-04-22T17:28:24.806 INFO:tasks.rados.rados.0.smithi115.stdout:881: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:28:24.806 INFO:tasks.rados.rados.0.smithi115.stdout:880: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:28:24.807 INFO:tasks.rados.rados.0.smithi115.stdout:881: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:28:24.807 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 194 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:28:24.807 INFO:tasks.rados.rados.0.smithi115.stdout:880: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:28:24.808 INFO:tasks.rados.rados.0.smithi115.stdout:880: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:28:24.808 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 423 (ObjNum 134 snap 35 seq_num 134) dirty exists 2022-04-22T17:28:24.817 INFO:tasks.rados.rados.0.smithi115.stdout:877: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:25.008 INFO:tasks.rados.rados.0.smithi115.stdout:883: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:28:25.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:24 smithi121 bash[12679]: cluster 2022-04-22T17:28:23.117645+0000 mgr.y (mgr.34107) 11 : cluster [DBG] pgmap v18: 121 pgs: 121 active+clean; 169 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 17 op/s 2022-04-22T17:28:25.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:24 smithi121 bash[12679]: cluster 2022-04-22T17:28:23.745147+0000 mon.a (mon.0) 68 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2022-04-22T17:28:25.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:24 smithi121 bash[12679]: audit 2022-04-22T17:28:24.139542+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:25.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:24 smithi121 bash[12679]: cluster 2022-04-22T17:28:24.714157+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2022-04-22T17:28:25.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:24 smithi115 bash[18070]: cluster 2022-04-22T17:28:23.117645+0000 mgr.y (mgr.34107) 11 : cluster [DBG] pgmap v18: 121 pgs: 121 active+clean; 169 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 17 op/s 2022-04-22T17:28:25.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:24 smithi115 bash[18070]: cluster 2022-04-22T17:28:23.745147+0000 mon.a (mon.0) 68 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2022-04-22T17:28:25.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:24 smithi115 bash[18070]: audit 2022-04-22T17:28:24.139542+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:25.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:24 smithi115 bash[18070]: cluster 2022-04-22T17:28:24.714157+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2022-04-22T17:28:25.769 INFO:tasks.rados.rados.0.smithi115.stdout:876: done (5 left) 2022-04-22T17:28:25.769 INFO:tasks.rados.rados.0.smithi115.stdout:877: done (4 left) 2022-04-22T17:28:25.769 INFO:tasks.rados.rados.0.smithi115.stdout:880: done (3 left) 2022-04-22T17:28:25.769 INFO:tasks.rados.rados.0.smithi115.stdout:881: done (2 left) 2022-04-22T17:28:25.770 INFO:tasks.rados.rados.0.smithi115.stdout:883: done (1 left) 2022-04-22T17:28:25.770 INFO:tasks.rados.rados.0.smithi115.stdout:884: done (0 left) 2022-04-22T17:28:25.770 INFO:tasks.rados.rados.0.smithi115.stdout:885: snap_remove snap 80 2022-04-22T17:28:26.776 INFO:tasks.rados.rados.0.smithi115.stdout:885: done (0 left) 2022-04-22T17:28:26.776 INFO:tasks.rados.rados.0.smithi115.stdout:886: read oid 47 snap -1 2022-04-22T17:28:26.776 INFO:tasks.rados.rados.0.smithi115.stdout:886: expect (ObjNum 256 snap 85 seq_num 256) 2022-04-22T17:28:26.776 INFO:tasks.rados.rados.0.smithi115.stdout:887: read oid 35 snap -1 2022-04-22T17:28:26.777 INFO:tasks.rados.rados.0.smithi115.stdout:887: expect (ObjNum 241 snap 76 seq_num 241) 2022-04-22T17:28:26.777 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 88 2022-04-22T17:28:26.777 INFO:tasks.rados.rados.0.smithi115.stdout:888: seq_num 263 ranges {9134080=409600,9543680=475136,10018816=557056,10575872=573440} 2022-04-22T17:28:26.833 INFO:tasks.rados.rados.0.smithi115.stdout:888: writing smithi11541303-4 from 9134080 to 9543680 tid 1 2022-04-22T17:28:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:888: writing smithi11541303-4 from 9543680 to 10018816 tid 2 2022-04-22T17:28:26.838 INFO:tasks.rados.rados.0.smithi115.stdout:888: writing smithi11541303-4 from 10018816 to 10575872 tid 3 2022-04-22T17:28:26.842 INFO:tasks.rados.rados.0.smithi115.stdout:888: writing smithi11541303-4 from 10575872 to 11149312 tid 4 2022-04-22T17:28:26.842 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 88 2022-04-22T17:28:26.842 INFO:tasks.rados.rados.0.smithi115.stdout:889: seq_num 264 ranges {106496=425984,532480=761856,1294336=770048,2064384=73728} 2022-04-22T17:28:26.845 INFO:tasks.rados.rados.0.smithi115.stdout:889: writing smithi11541303-36 from 106496 to 532480 tid 1 2022-04-22T17:28:26.849 INFO:tasks.rados.rados.0.smithi115.stdout:889: writing smithi11541303-36 from 532480 to 1294336 tid 2 2022-04-22T17:28:26.854 INFO:tasks.rados.rados.0.smithi115.stdout:889: writing smithi11541303-36 from 1294336 to 2064384 tid 3 2022-04-22T17:28:26.854 INFO:tasks.rados.rados.0.smithi115.stdout:889: writing smithi11541303-36 from 2064384 to 2138112 tid 4 2022-04-22T17:28:26.854 INFO:tasks.rados.rados.0.smithi115.stdout:890: copy_from oid 8 from oid 27 current snap is 88 2022-04-22T17:28:26.854 INFO:tasks.rados.rados.0.smithi115.stdout:886: expect (ObjNum 256 snap 85 seq_num 256) 2022-04-22T17:28:26.926 INFO:tasks.rados.rados.0.smithi115.stdout:887: expect (ObjNum 241 snap 76 seq_num 241) 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 1 to smithi11541303-4 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 2 to smithi11541303-4 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 3 to smithi11541303-4 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 4 to smithi11541303-4 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 5 to smithi11541303-4 2022-04-22T17:28:26.969 INFO:tasks.rados.rados.0.smithi115.stdout:888: finishing write tid 6 to smithi11541303-4 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 435 (ObjNum 263 snap 88 seq_num 263) dirty exists 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:888: left oid 4 (ObjNum 263 snap 88 seq_num 263) 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 1 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 2 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 3 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 4 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 5 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:889: finishing write tid 6 to smithi11541303-36 2022-04-22T17:28:26.970 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 247 (ObjNum 264 snap 88 seq_num 264) dirty exists 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:889: left oid 36 (ObjNum 264 snap 88 seq_num 264) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:890: finishing copy_from to smithi11541303-8 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:890: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:886: done (4 left) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:887: done (3 left) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:888: done (2 left) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:889: done (1 left) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:890: done (0 left) 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 88 2022-04-22T17:28:26.971 INFO:tasks.rados.rados.0.smithi115.stdout:891: seq_num 265 ranges {5529600=442368,5971968=647168,6619136=573440,7192576=385024} 2022-04-22T17:28:27.009 INFO:tasks.rados.rados.0.smithi115.stdout:891: writing smithi11541303-10 from 5529600 to 5971968 tid 1 2022-04-22T17:28:27.013 INFO:tasks.rados.rados.0.smithi115.stdout:891: writing smithi11541303-10 from 5971968 to 6619136 tid 2 2022-04-22T17:28:27.016 INFO:tasks.rados.rados.0.smithi115.stdout:891: writing smithi11541303-10 from 6619136 to 7192576 tid 3 2022-04-22T17:28:27.019 INFO:tasks.rados.rados.0.smithi115.stdout:891: writing smithi11541303-10 from 7192576 to 7577600 tid 4 2022-04-22T17:28:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:892: rollback oid 41 current snap is 88 2022-04-22T17:28:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 74 2022-04-22T17:28:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 1 to smithi11541303-10 2022-04-22T17:28:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:893: delete oid 38 current snap is 88 2022-04-22T17:28:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 2 to smithi11541303-10 2022-04-22T17:28:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:893: done (2 left) 2022-04-22T17:28:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 88 2022-04-22T17:28:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:894: seq_num 267 ranges {0=475136,475136=458752,933888=540672,1474560=606208} 2022-04-22T17:28:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:894: writing smithi11541303-7 from 0 to 475136 tid 1 2022-04-22T17:28:27.026 INFO:tasks.rados.rados.0.smithi115.stdout:894: writing smithi11541303-7 from 475136 to 933888 tid 2 2022-04-22T17:28:27.030 INFO:tasks.rados.rados.0.smithi115.stdout:894: writing smithi11541303-7 from 933888 to 1474560 tid 3 2022-04-22T17:28:27.034 INFO:tasks.rados.rados.0.smithi115.stdout:894: writing smithi11541303-7 from 1474560 to 2080768 tid 4 2022-04-22T17:28:27.034 INFO:tasks.rados.rados.0.smithi115.stdout:895: read oid 21 snap -1 2022-04-22T17:28:27.034 INFO:tasks.rados.rados.0.smithi115.stdout:895: expect deleted 2022-04-22T17:28:27.034 INFO:tasks.rados.rados.0.smithi115.stdout:896: snap_remove snap 77 2022-04-22T17:28:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:26 smithi115 bash[18070]: cluster 2022-04-22T17:28:25.118379+0000 mgr.y (mgr.34107) 12 : cluster [DBG] pgmap v21: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:28:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:26 smithi115 bash[18070]: cluster 2022-04-22T17:28:25.757818+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2022-04-22T17:28:27.063 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:26 smithi121 bash[12679]: cluster 2022-04-22T17:28:25.118379+0000 mgr.y (mgr.34107) 12 : cluster [DBG] pgmap v21: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:28:27.064 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:26 smithi121 bash[12679]: cluster 2022-04-22T17:28:25.757818+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2022-04-22T17:28:27.794 INFO:tasks.rados.rados.0.smithi115.stdout:896: done (4 left) 2022-04-22T17:28:27.795 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 88 2022-04-22T17:28:27.795 INFO:tasks.rados.rados.0.smithi115.stdout:897: seq_num 268 ranges {11460608=491520,11952128=745472,12697600=745472,13443072=114688} 2022-04-22T17:28:27.864 INFO:tasks.rados.rados.0.smithi115.stdout:897: writing smithi11541303-23 from 11460608 to 11952128 tid 1 2022-04-22T17:28:27.868 INFO:tasks.rados.rados.0.smithi115.stdout:897: writing smithi11541303-23 from 11952128 to 12697600 tid 2 2022-04-22T17:28:27.872 INFO:tasks.rados.rados.0.smithi115.stdout:897: writing smithi11541303-23 from 12697600 to 13443072 tid 3 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:897: writing smithi11541303-23 from 13443072 to 13557760 tid 4 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:898: snap_create 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 3 to smithi11541303-10 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 4 to smithi11541303-10 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 5 to smithi11541303-10 2022-04-22T17:28:27.874 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 1 to smithi11541303-7 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:892: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:892: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:891: finishing write tid 6 to smithi11541303-10 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 238 (ObjNum 265 snap 88 seq_num 265) dirty exists 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:891: left oid 10 (ObjNum 265 snap 88 seq_num 265) 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:892: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 242 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 2 to smithi11541303-7 2022-04-22T17:28:27.875 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 3 to smithi11541303-7 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 4 to smithi11541303-7 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 5 to smithi11541303-7 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:894: finishing write tid 6 to smithi11541303-7 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 375 (ObjNum 267 snap 88 seq_num 267) dirty exists 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:894: left oid 7 (ObjNum 267 snap 88 seq_num 267) 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 1 to smithi11541303-23 2022-04-22T17:28:27.876 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 2 to smithi11541303-23 2022-04-22T17:28:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 3 to smithi11541303-23 2022-04-22T17:28:27.879 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 4 to smithi11541303-23 2022-04-22T17:28:27.879 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 5 to smithi11541303-23 2022-04-22T17:28:27.881 INFO:tasks.rados.rados.0.smithi115.stdout:897: finishing write tid 6 to smithi11541303-23 2022-04-22T17:28:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 349 (ObjNum 268 snap 88 seq_num 268) dirty exists 2022-04-22T17:28:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:897: left oid 23 (ObjNum 268 snap 88 seq_num 268) 2022-04-22T17:28:28.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:27 smithi121 bash[12679]: cluster 2022-04-22T17:28:26.761893+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2022-04-22T17:28:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:27 smithi115 bash[18070]: cluster 2022-04-22T17:28:26.761893+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:891: done (5 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:892: done (4 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:894: done (3 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:895: done (2 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:897: done (1 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:898: done (0 left) 2022-04-22T17:28:28.850 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 89 2022-04-22T17:28:28.851 INFO:tasks.rados.rados.0.smithi115.stdout:899: seq_num 269 ranges {3956736=507904,4464640=630784,5095424=557056,5652480=417792} 2022-04-22T17:28:28.878 INFO:tasks.rados.rados.0.smithi115.stdout:899: writing smithi11541303-19 from 3956736 to 4464640 tid 1 2022-04-22T17:28:28.882 INFO:tasks.rados.rados.0.smithi115.stdout:899: writing smithi11541303-19 from 4464640 to 5095424 tid 2 2022-04-22T17:28:28.885 INFO:tasks.rados.rados.0.smithi115.stdout:899: writing smithi11541303-19 from 5095424 to 5652480 tid 3 2022-04-22T17:28:28.888 INFO:tasks.rados.rados.0.smithi115.stdout:899: writing smithi11541303-19 from 5652480 to 6070272 tid 4 2022-04-22T17:28:28.888 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 89 2022-04-22T17:28:28.888 INFO:tasks.rados.rados.0.smithi115.stdout:900: seq_num 270 ranges {0=524288,524288=516096,1040384=761856,1802240=327680} 2022-04-22T17:28:28.891 INFO:tasks.rados.rados.0.smithi115.stdout:900: writing smithi11541303-25 from 0 to 524288 tid 1 2022-04-22T17:28:28.894 INFO:tasks.rados.rados.0.smithi115.stdout:900: writing smithi11541303-25 from 524288 to 1040384 tid 2 2022-04-22T17:28:28.900 INFO:tasks.rados.rados.0.smithi115.stdout:900: writing smithi11541303-25 from 1040384 to 1802240 tid 3 2022-04-22T17:28:28.902 INFO:tasks.rados.rados.0.smithi115.stdout:900: writing smithi11541303-25 from 1802240 to 2129920 tid 4 2022-04-22T17:28:28.902 INFO:tasks.rados.rados.0.smithi115.stdout:901: rmattr oid 42 current snap is 89 2022-04-22T17:28:28.902 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 1 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 2 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 3 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 4 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 5 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: finishing write tid 6 to smithi11541303-19 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 305 (ObjNum 269 snap 89 seq_num 269) dirty exists 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: left oid 19 (ObjNum 269 snap 89 seq_num 269) 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:899: done (2 left) 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:902: delete oid 35 current snap is 89 2022-04-22T17:28:28.903 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 1 to smithi11541303-25 2022-04-22T17:28:28.904 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 2 to smithi11541303-25 2022-04-22T17:28:28.906 INFO:tasks.rados.rados.0.smithi115.stdout:902: done (2 left) 2022-04-22T17:28:28.906 INFO:tasks.rados.rados.0.smithi115.stdout:903: copy_from oid 21 from oid 40 current snap is 89 2022-04-22T17:28:28.906 INFO:tasks.rados.rados.0.smithi115.stdout:904: snap_create 2022-04-22T17:28:28.907 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 498 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:28:28.909 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 3 to smithi11541303-25 2022-04-22T17:28:28.909 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 4 to smithi11541303-25 2022-04-22T17:28:28.910 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 5 to smithi11541303-25 2022-04-22T17:28:28.912 INFO:tasks.rados.rados.0.smithi115.stdout:900: finishing write tid 6 to smithi11541303-25 2022-04-22T17:28:28.912 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 359 (ObjNum 270 snap 89 seq_num 270) dirty exists 2022-04-22T17:28:28.912 INFO:tasks.rados.rados.0.smithi115.stdout:900: left oid 25 (ObjNum 270 snap 89 seq_num 270) 2022-04-22T17:28:28.930 INFO:tasks.rados.rados.0.smithi115.stdout:903: finishing copy_from to smithi11541303-21 2022-04-22T17:28:28.930 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 184 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:28.931 INFO:tasks.rados.rados.0.smithi115.stdout:903: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:28:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: cluster 2022-04-22T17:28:27.119136+0000 mgr.y (mgr.34107) 13 : cluster [DBG] pgmap v24: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: cluster 2022-04-22T17:28:27.777079+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: cephadm 2022-04-22T17:28:27.821615+0000 mgr.y (mgr.34107) 14 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:27.832434+0000 mon.a (mon.0) 74 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: cephadm 2022-04-22T17:28:27.834316+0000 mgr.y (mgr.34107) 15 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.347281+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.356483+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.364195+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.367543+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:28:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.369355+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:28:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:28 smithi115 bash[18070]: audit 2022-04-22T17:28:28.370809+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2022-04-22T17:28:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: cluster 2022-04-22T17:28:27.119136+0000 mgr.y (mgr.34107) 13 : cluster [DBG] pgmap v24: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: cluster 2022-04-22T17:28:27.777079+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: cephadm 2022-04-22T17:28:27.821615+0000 mgr.y (mgr.34107) 14 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:27.832434+0000 mon.a (mon.0) 74 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: cephadm 2022-04-22T17:28:27.834316+0000 mgr.y (mgr.34107) 15 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.347281+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.356483+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.364195+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.367543+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:28:29.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.369355+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:28:29.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:28 smithi121 bash[12679]: audit 2022-04-22T17:28:28.370809+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:900: done (3 left) 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:901: done (2 left) 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:903: done (1 left) 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:904: done (0 left) 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:905: rollback oid 44 current snap is 90 2022-04-22T17:28:29.860 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 74 2022-04-22T17:28:29.861 INFO:tasks.rados.rados.0.smithi115.stdout:906: copy_from oid 18 from oid 38 current snap is 90 2022-04-22T17:28:29.861 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 90 2022-04-22T17:28:29.861 INFO:tasks.rados.rados.0.smithi115.stdout:907: seq_num 272 ranges {0=565248,565248=688128,1253376=729088,1982464=188416} 2022-04-22T17:28:29.862 INFO:tasks.rados.rados.0.smithi115.stdout:907: writing smithi11541303-3 from 0 to 565248 tid 1 2022-04-22T17:28:29.867 INFO:tasks.rados.rados.0.smithi115.stdout:907: writing smithi11541303-3 from 565248 to 1253376 tid 2 2022-04-22T17:28:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:907: writing smithi11541303-3 from 1253376 to 1982464 tid 3 2022-04-22T17:28:29.873 INFO:tasks.rados.rados.0.smithi115.stdout:907: writing smithi11541303-3 from 1982464 to 2170880 tid 4 2022-04-22T17:28:29.873 INFO:tasks.rados.rados.0.smithi115.stdout:908: snap_remove snap 76 2022-04-22T17:28:30.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:29 smithi115 bash[18070]: cephadm 2022-04-22T17:28:28.084488+0000 mgr.y (mgr.34107) 16 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:28:30.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:29 smithi115 bash[18070]: cephadm 2022-04-22T17:28:28.371906+0000 mgr.y (mgr.34107) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2022-04-22T17:28:30.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:29 smithi115 bash[18070]: cluster 2022-04-22T17:28:28.837405+0000 mon.a (mon.0) 81 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2022-04-22T17:28:30.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:29 smithi115 bash[18070]: audit 2022-04-22T17:28:29.142987+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:29 smithi121 bash[12679]: cephadm 2022-04-22T17:28:28.084488+0000 mgr.y (mgr.34107) 16 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:28:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:29 smithi121 bash[12679]: cephadm 2022-04-22T17:28:28.371906+0000 mgr.y (mgr.34107) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2022-04-22T17:28:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:29 smithi121 bash[12679]: cluster 2022-04-22T17:28:28.837405+0000 mon.a (mon.0) 81 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2022-04-22T17:28:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:29 smithi121 bash[12679]: audit 2022-04-22T17:28:29.142987+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:908: done (3 left) 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:909: snap_create 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:906: finishing copy_from to smithi11541303-18 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:906: got expected ENOENT (src dne) 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:906: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:28:30.867 INFO:tasks.rados.rados.0.smithi115.stdout:905: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 0 (ObjNum -1116782512 snap 22075 seq_num 2550139088) dirty dne 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 1 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 2 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 3 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 4 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 5 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: finishing write tid 6 to smithi11541303-3 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 354 (ObjNum 272 snap 90 seq_num 272) dirty exists 2022-04-22T17:28:30.868 INFO:tasks.rados.rados.0.smithi115.stdout:907: left oid 3 (ObjNum 272 snap 90 seq_num 272) 2022-04-22T17:28:31.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:30 smithi115 bash[18070]: cluster 2022-04-22T17:28:29.119942+0000 mgr.y (mgr.34107) 18 : cluster [DBG] pgmap v27: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:31.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:30 smithi115 bash[18070]: cluster 2022-04-22T17:28:29.846914+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2022-04-22T17:28:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:30 smithi121 bash[12679]: cluster 2022-04-22T17:28:29.119942+0000 mgr.y (mgr.34107) 18 : cluster [DBG] pgmap v27: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:28:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:30 smithi121 bash[12679]: cluster 2022-04-22T17:28:29.846914+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2022-04-22T17:28:31.878 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:31.616Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:31.878 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:31.616Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:31.878 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:31.616Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:31.880 INFO:tasks.rados.rados.0.smithi115.stdout:905: done (3 left) 2022-04-22T17:28:31.881 INFO:tasks.rados.rados.0.smithi115.stdout:906: done (2 left) 2022-04-22T17:28:31.881 INFO:tasks.rados.rados.0.smithi115.stdout:907: done (1 left) 2022-04-22T17:28:31.881 INFO:tasks.rados.rados.0.smithi115.stdout:909: done (0 left) 2022-04-22T17:28:31.881 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 91 2022-04-22T17:28:31.881 INFO:tasks.rados.rados.0.smithi115.stdout:910: seq_num 273 ranges {106496=581632,688128=565248,1253376=532480,1785856=507904} 2022-04-22T17:28:31.885 INFO:tasks.rados.rados.0.smithi115.stdout:910: writing smithi11541303-1 from 106496 to 688128 tid 1 2022-04-22T17:28:31.889 INFO:tasks.rados.rados.0.smithi115.stdout:910: writing smithi11541303-1 from 688128 to 1253376 tid 2 2022-04-22T17:28:31.893 INFO:tasks.rados.rados.0.smithi115.stdout:910: writing smithi11541303-1 from 1253376 to 1785856 tid 3 2022-04-22T17:28:31.896 INFO:tasks.rados.rados.0.smithi115.stdout:910: writing smithi11541303-1 from 1785856 to 2293760 tid 4 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:911: read oid 35 snap -1 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:911: expect deleted 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:912: snap_create 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 1 to smithi11541303-1 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 2 to smithi11541303-1 2022-04-22T17:28:31.897 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 3 to smithi11541303-1 2022-04-22T17:28:31.899 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 4 to smithi11541303-1 2022-04-22T17:28:31.900 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 5 to smithi11541303-1 2022-04-22T17:28:31.901 INFO:tasks.rados.rados.0.smithi115.stdout:910: finishing write tid 6 to smithi11541303-1 2022-04-22T17:28:31.901 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 380 (ObjNum 273 snap 91 seq_num 273) dirty exists 2022-04-22T17:28:31.902 INFO:tasks.rados.rados.0.smithi115.stdout:910: left oid 1 (ObjNum 273 snap 91 seq_num 273) 2022-04-22T17:28:32.006 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:28:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:31 smithi121 bash[12679]: cluster 2022-04-22T17:28:30.855490+0000 mon.a (mon.0) 84 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2022-04-22T17:28:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:31 smithi121 bash[12679]: audit 2022-04-22T17:28:31.303891+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:31 smithi121 bash[12679]: audit 2022-04-22T17:28:31.305450+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:28:32.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:31 smithi121 bash[12679]: audit 2022-04-22T17:28:31.306744+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:28:32.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:31 smithi121 bash[12679]: audit 2022-04-22T17:28:31.308009+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[18070]: cluster 2022-04-22T17:28:30.855490+0000 mon.a (mon.0) 84 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2022-04-22T17:28:32.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[18070]: audit 2022-04-22T17:28:31.303891+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:32.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[18070]: audit 2022-04-22T17:28:31.305450+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:28:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[18070]: audit 2022-04-22T17:28:31.306744+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:28:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:31 smithi115 bash[18070]: audit 2022-04-22T17:28:31.308009+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:32.349 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:28:32.350 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 13s ago 5m 12.7M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:28:32.350 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5m) 13s ago 5m 23.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:28:32.350 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6m) 13s ago 6m 13.1M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:28:32.351 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6m) 13s ago 6m 16.5M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:28:32.351 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (51s) 13s ago 10m 402M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:28:32.351 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (89s) 13s ago 12m 431M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:28:32.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (28s) 13s ago 13m 33.2M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:28:32.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (11m) 13s ago 11m 54.0M 2048M 16.2.5 46cf6318c64e 1f6c98b4c5b2 2022-04-22T17:28:32.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (11m) 13s ago 11m 52.0M 2048M 16.2.5 46cf6318c64e b7836dd7ce4a 2022-04-22T17:28:32.353 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5m) 13s ago 5m 8924k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:28:32.353 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5m) 13s ago 5m 9111k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:28:32.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 13s ago 10m 144M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:28:32.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 13s ago 9m 141M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:28:32.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 13s ago 9m 117M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:28:32.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 13s ago 8m 82.7M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:28:32.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (8m) 13s ago 8m 118M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:28:32.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (7m) 13s ago 7m 131M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:28:32.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (7m) 13s ago 7m 110M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:28:32.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6m) 13s ago 6m 114M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:28:32.355 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (97s) 13s ago 5m 32.5M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:28:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2, 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:28:32.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:28:32.733 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:910: done (2 left) 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:911: done (1 left) 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:912: done (0 left) 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:913: rollback oid 37 current snap is 92 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 91 2022-04-22T17:28:32.882 INFO:tasks.rados.rados.0.smithi115.stdout:914: delete oid 44 current snap is 92 2022-04-22T17:28:32.890 INFO:tasks.rados.rados.0.smithi115.stdout:914: done (1 left) 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:915: copy_from oid 39 from oid 8 current snap is 92 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:916: read oid 46 snap 90 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:916: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:917: rollback oid 29 current snap is 92 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 69 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 92 2022-04-22T17:28:32.891 INFO:tasks.rados.rados.0.smithi115.stdout:918: seq_num 275 ranges {0=614400,614400=786432,1400832=499712,1900544=319488} 2022-04-22T17:28:32.894 INFO:tasks.rados.rados.0.smithi115.stdout:918: writing smithi11541303-50 from 0 to 614400 tid 1 2022-04-22T17:28:32.900 INFO:tasks.rados.rados.0.smithi115.stdout:918: writing smithi11541303-50 from 614400 to 1400832 tid 2 2022-04-22T17:28:32.903 INFO:tasks.rados.rados.0.smithi115.stdout:918: writing smithi11541303-50 from 1400832 to 1900544 tid 3 2022-04-22T17:28:32.907 INFO:tasks.rados.rados.0.smithi115.stdout:918: writing smithi11541303-50 from 1900544 to 2220032 tid 4 2022-04-22T17:28:32.907 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 92 2022-04-22T17:28:32.907 INFO:tasks.rados.rados.0.smithi115.stdout:913: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:28:32.907 INFO:tasks.rados.rados.0.smithi115.stdout:919: seq_num 276 ranges {2957312=630784,3588096=671744,4259840=704512,4964352=229376} 2022-04-22T17:28:32.926 INFO:tasks.rados.rados.0.smithi115.stdout:919: writing smithi11541303-24 from 2957312 to 3588096 tid 1 2022-04-22T17:28:32.931 INFO:tasks.rados.rados.0.smithi115.stdout:919: writing smithi11541303-24 from 3588096 to 4259840 tid 2 2022-04-22T17:28:32.935 INFO:tasks.rados.rados.0.smithi115.stdout:919: writing smithi11541303-24 from 4259840 to 4964352 tid 3 2022-04-22T17:28:32.936 INFO:tasks.rados.rados.0.smithi115.stdout:919: writing smithi11541303-24 from 4964352 to 5193728 tid 4 2022-04-22T17:28:32.937 INFO:tasks.rados.rados.0.smithi115.stdout:920: snap_remove snap 87 2022-04-22T17:28:33.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: cluster 2022-04-22T17:28:31.120698+0000 mgr.y (mgr.34107) 19 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 190 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 28 op/s 2022-04-22T17:28:33.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: cephadm 2022-04-22T17:28:31.296655+0000 mgr.y (mgr.34107) 20 : cephadm [INF] Upgrade: Updating mon.c 2022-04-22T17:28:33.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: cephadm 2022-04-22T17:28:31.309467+0000 mgr.y (mgr.34107) 21 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:28:33.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: cluster 2022-04-22T17:28:31.869824+0000 mon.a (mon.0) 89 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2022-04-22T17:28:33.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: audit 2022-04-22T17:28:31.978163+0000 mgr.y (mgr.34107) 22 : audit [DBG] from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:33.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:32 smithi115 bash[18070]: audit 2022-04-22T17:28:32.720113+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? 172.21.15.115:0/181693675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:28:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: cluster 2022-04-22T17:28:31.120698+0000 mgr.y (mgr.34107) 19 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 190 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 28 op/s 2022-04-22T17:28:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: cephadm 2022-04-22T17:28:31.296655+0000 mgr.y (mgr.34107) 20 : cephadm [INF] Upgrade: Updating mon.c 2022-04-22T17:28:33.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: cephadm 2022-04-22T17:28:31.309467+0000 mgr.y (mgr.34107) 21 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:28:33.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: cluster 2022-04-22T17:28:31.869824+0000 mon.a (mon.0) 89 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2022-04-22T17:28:33.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: audit 2022-04-22T17:28:31.978163+0000 mgr.y (mgr.34107) 22 : audit [DBG] from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:33.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:32 smithi121 bash[12679]: audit 2022-04-22T17:28:32.720113+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? 172.21.15.115:0/181693675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:28:33.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.501Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.501Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.501Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:33.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:920: done (6 left) 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:921: rmattr oid 28 current snap is 92 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:921: done (6 left) 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:922: setattr oid 44 current snap is 92 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:913: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:913: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 237 (ObjNum 245 snap 81 seq_num 245) dirty exists 2022-04-22T17:28:33.913 INFO:tasks.rados.rados.0.smithi115.stdout:917: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:28:33.914 INFO:tasks.rados.rados.0.smithi115.stdout:917: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:28:33.914 INFO:tasks.rados.rados.0.smithi115.stdout:917: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:28:33.914 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 234 (ObjNum 156 snap 41 seq_num 156) dirty exists 2022-04-22T17:28:33.914 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 1 to smithi11541303-50 2022-04-22T17:28:33.914 INFO:tasks.rados.rados.0.smithi115.stdout:916: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:33.937 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 2 to smithi11541303-50 2022-04-22T17:28:33.937 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 3 to smithi11541303-50 2022-04-22T17:28:33.937 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 4 to smithi11541303-50 2022-04-22T17:28:33.937 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 5 to smithi11541303-50 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:915: finishing copy_from to smithi11541303-39 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:918: finishing write tid 6 to smithi11541303-50 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 209 (ObjNum 275 snap 92 seq_num 275) dirty exists 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:918: left oid 50 (ObjNum 275 snap 92 seq_num 275) 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:915: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 1 to smithi11541303-24 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 2 to smithi11541303-24 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 3 to smithi11541303-24 2022-04-22T17:28:33.938 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 4 to smithi11541303-24 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 5 to smithi11541303-24 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:919: finishing write tid 6 to smithi11541303-24 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 119 (ObjNum 276 snap 92 seq_num 276) dirty exists 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:919: left oid 24 (ObjNum 276 snap 92 seq_num 276) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:913: done (6 left) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:915: done (5 left) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:916: done (4 left) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:917: done (3 left) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:918: done (2 left) 2022-04-22T17:28:33.939 INFO:tasks.rados.rados.0.smithi115.stdout:919: done (1 left) 2022-04-22T17:28:33.940 INFO:tasks.rados.rados.0.smithi115.stdout:923: snap_remove snap 79 2022-04-22T17:28:34.141 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[18070]: audit 2022-04-22T17:28:32.315445+0000 mgr.y (mgr.34107) 23 : audit [DBG] from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:34.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:33 smithi115 bash[18070]: cluster 2022-04-22T17:28:32.868608+0000 mon.a (mon.0) 91 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2022-04-22T17:28:34.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:34 smithi115 systemd[1]: Stopping Ceph mon.c for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:28:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:33 smithi121 bash[12679]: audit 2022-04-22T17:28:32.315445+0000 mgr.y (mgr.34107) 23 : audit [DBG] from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:28:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:33 smithi121 bash[12679]: cluster 2022-04-22T17:28:32.868608+0000 mon.a (mon.0) 91 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2022-04-22T17:28:34.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:34 smithi115 bash[18070]: debug 2022-04-22T17:28:34.263+0000 7f1bff752700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-04-22T17:28:34.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:34 smithi115 bash[18070]: debug 2022-04-22T17:28:34.263+0000 7f1bff752700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2022-04-22T17:28:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 bash[46811]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.c 2022-04-22T17:28:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 bash[46853]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon-c 2022-04-22T17:28:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 bash[46866]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.c 2022-04-22T17:28:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.c.service: Succeeded. 2022-04-22T17:28:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 systemd[1]: Stopped Ceph mon.c for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:28:36.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:35 smithi115 systemd[1]: Started Ceph mon.c for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:28:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.983+0000 7fc621f5f880 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.983+0000 7fc621f5f880 0 ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev), process ceph-mon, pid 7 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.983+0000 7fc621f5f880 0 pidfile_write: ignore empty --pid-file 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 0 load: jerasure load: lrc 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: RocksDB version: 6.15.5 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Compile date Apr 22 2022 2022-04-22T17:28:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: DB SUMMARY 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: DB Session ID: FKK5WKK3RQJT67T9422X 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: CURRENT file: CURRENT 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: IDENTITY file: IDENTITY 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000020.sst 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000018.log size: 15560234 ; 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.create_if_missing: 0 2022-04-22T17:28:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.env: 0x5619a0ee0860 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.info_log: 0x5619a20ffe00 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:28:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:28:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.db_log_dir: 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:28:37.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.write_buffer_manager: 0x5619a21c6420 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:28:37.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.row_cache: None 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.wal_filter: None 2022-04-22T17:28:37.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:28:37.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.best_efforts_recovery: 0 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.allow_data_in_errors: 0 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.db_host_id: __hostname__ 2022-04-22T17:28:37.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:28:37.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Compression algorithms supported: 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kZSTD supported: 0 2022-04-22T17:28:37.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2022-04-22T17:28:37.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.merge_operator: 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.sst_partitioner_factory: None 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:28:37.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5619a20cdcb0) 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: cache_index_and_filter_blocks: 1 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: pin_top_level_index_and_filter: 1 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: index_type: 0 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: data_block_index_type: 0 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: index_shortening: 1 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:28:37.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: hash_index_allow_collision: 1 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: checksum: 1 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: no_block_cache: 0 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_cache: 0x5619a2134170 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_cache_name: BinnedLRUCache 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_cache_options: 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: capacity : 536870912 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: num_shard_bits : 4 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: strict_capacity_limit : 0 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: high_pri_pool_ratio: 0.000 2022-04-22T17:28:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_cache_compressed: (nil) 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: persistent_cache: (nil) 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_size: 4096 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_size_deviation: 10 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: block_restart_interval: 16 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:36 smithi115 bash[46934]: index_block_restart_interval: 1 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: metadata_block_size: 4096 2022-04-22T17:28:37.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: partition_filters: 0 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: use_delta_encoding: 1 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: whole_key_filtering: 1 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: verify_compression: 0 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: read_amp_bytes_per_bit: 0 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: format_version: 4 2022-04-22T17:28:37.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: enable_index_compression: 1 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: block_align: 0 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:28:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.num_levels: 7 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:28:37.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2022-04-22T17:28:37.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:28:37.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:28:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:28:37.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:28:37.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:28:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:28:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:28:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:28:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:28:37.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.force_consistency_checks: 1 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.enable_blob_files: false 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.min_blob_size: 0 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.blob_file_size: 268435456 2022-04-22T17:28:37.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.blob_compression_type: NoCompression 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.enable_blob_garbage_collection: false 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.987+0000 7fc621f5f880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12278, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 18 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 22 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:37.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648516992918, "job": 1, "event": "recovery_started", "wal_files": [18]} 2022-04-22T17:28:37.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:36.991+0000 7fc621f5f880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #18 mode 2 2022-04-22T17:28:37.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.011+0000 7fc621f5f880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-04-22T17:28:37.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.027+0000 7fc621f5f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648517028488, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 9976289, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 9960490, "index_size": 12104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2757, "raw_key_size": 22013, "raw_average_key_size": 20, "raw_value_size": 9933674, "raw_average_value_size": 9344, "num_data_blocks": 519, "num_entries": 1063, "num_deletions": 294, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1650648517, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c14cfe38-d287-4dca-bae6-f2488aedb80a", "db_session_id": "FKK5WKK3RQJT67T9422X"}} 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.027+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 24 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.027+0000 7fc621f5f880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.027+0000 7fc621f5f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648517030407, "job": 1, "event": "recovery_finished"} 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5619a211aa80 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 4 rocksdb: DB pointer 0x5619a212b800 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc611cdd700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-04-22T17:28:37.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc611cdd700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** DB Stats ** 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** Compaction Stats [default] ** 2022-04-22T17:28:37.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:37.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: L0 1/0 9.51 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Sum 2/0 32.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** Compaction Stats [default] ** 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:37.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Flush(GB): cumulative 0.009, interval 0.009 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Cumulative compaction: 0.01 GB write, 227.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Interval compaction: 0.01 GB write, 227.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** Compaction Stats [default] ** 2022-04-22T17:28:37.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: L0 1/0 9.51 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Sum 2/0 32.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** Compaction Stats [default] ** 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 615.0 0.02 0.00 1 0.015 0 0 2022-04-22T17:28:37.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Flush(GB): cumulative 0.009, interval 0.000 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Cumulative compaction: 0.01 GB write, 227.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:28:37.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 starting mon.c rank 1 at public addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] at bind addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 1 mon.c@-1(???) e3 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).mds e6 new map 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).mds e6 print_map 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: e6 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: legacy client fscid: 1 2022-04-22T17:28:37.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Filesystem 'cephfs' (1) 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: fs_name cephfs 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: epoch 6 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: flags 12 joinable allow_snaps allow_multimds_snaps 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: created 2022-04-22T17:22:18.413732+0000 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: modified 2022-04-22T17:22:20.428010+0000 2022-04-22T17:28:37.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: tableserver 0 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: root 0 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: session_timeout 60 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: session_autoclose 300 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: max_file_size 1099511627776 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: required_client_features {} 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: last_failure 0 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: last_failure_osd_epoch 0 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:28:37.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: max_mds 1 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: in 0 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: up {0=24386} 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: failed 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: damaged 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: stopped 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: data_pools [3] 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: metadata_pool 2 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: inline_data disabled 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: balancer 2022-04-22T17:28:37.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: standby_count_wanted 0 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: [mds.b{0:24386} state up:active seq 3 addr [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: Standby daemons: 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: [mds.a{-1:24385} state up:standby seq 2 addr [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).osd e238 crush map has features 3314933069573799936, adjusting msgr requires 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).osd e238 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).osd e238 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:37.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.031+0000 7fc621f5f880 0 mon.c@-1(???).osd e238 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:28:37.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:28:37 smithi115 bash[46934]: debug 2022-04-22T17:28:37.035+0000 7fc621f5f880 1 mon.c@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2022-04-22T17:28:37.784 INFO:tasks.rados.rados.0.smithi115.stdout:923: done (1 left) 2022-04-22T17:28:37.784 INFO:tasks.rados.rados.0.smithi115.stdout:924: rollback oid 16 current snap is 92 2022-04-22T17:28:37.784 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 83 2022-04-22T17:28:37.785 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 25 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty exists 2022-04-22T17:28:37.785 INFO:tasks.rados.rados.0.smithi115.stdout:922: done (1 left) 2022-04-22T17:28:37.785 INFO:tasks.rados.rados.0.smithi115.stdout:925: snap_remove snap 86 2022-04-22T17:28:38.802 INFO:tasks.rados.rados.0.smithi115.stdout:925: done (1 left) 2022-04-22T17:28:38.802 INFO:tasks.rados.rados.0.smithi115.stdout:926: read oid 46 snap -1 2022-04-22T17:28:38.802 INFO:tasks.rados.rados.0.smithi115.stdout:926: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:38.802 INFO:tasks.rados.rados.0.smithi115.stdout:927: copy_from oid 21 from oid 11 current snap is 92 2022-04-22T17:28:38.802 INFO:tasks.rados.rados.0.smithi115.stdout:928: snap_remove snap 82 2022-04-22T17:28:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:35.122983+0000 mgr.y (mgr.34107) 25 : cluster [DBG] pgmap v35: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 524 KiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.123827+0000 mgr.y (mgr.34107) 26 : cluster [DBG] pgmap v36: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.245647+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.730793+0000 mon.a (mon.0) 94 : cluster [INF] mon.a calling monitor election 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.753258+0000 mon.a (mon.0) 95 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.767154+0000 mon.a (mon.0) 96 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.768572+0000 mon.a (mon.0) 97 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.768653+0000 mon.a (mon.0) 98 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2022-04-22T17:28:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.769833+0000 mon.a (mon.0) 99 : cluster [DBG] mgrmap e29: y(active, since 24s), standbys: x 2022-04-22T17:28:39.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: cluster 2022-04-22T17:28:37.776592+0000 mon.a (mon.0) 100 : cluster [INF] overall HEALTH_OK 2022-04-22T17:28:39.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: audit 2022-04-22T17:28:37.782516+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:39.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: audit 2022-04-22T17:28:37.784165+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:39.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: audit 2022-04-22T17:28:37.793877+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:39.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:38 smithi121 bash[12679]: audit 2022-04-22T17:28:37.796543+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:28:39.809 INFO:tasks.rados.rados.0.smithi115.stdout:928: done (3 left) 2022-04-22T17:28:39.809 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 92 2022-04-22T17:28:39.809 INFO:tasks.rados.rados.0.smithi115.stdout:924: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:28:39.810 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 0 (ObjNum -1479266896 snap 32692 seq_num 3178843120) dirty dne 2022-04-22T17:28:39.810 INFO:tasks.rados.rados.0.smithi115.stdout:929: seq_num 277 ranges {0=647168,647168=557056,1204224=516096,1720320=532480} 2022-04-22T17:28:39.815 INFO:tasks.rados.rados.0.smithi115.stdout:929: writing smithi11541303-18 from 0 to 647168 tid 1 2022-04-22T17:28:39.818 INFO:tasks.rados.rados.0.smithi115.stdout:929: writing smithi11541303-18 from 647168 to 1204224 tid 2 2022-04-22T17:28:39.821 INFO:tasks.rados.rados.0.smithi115.stdout:929: writing smithi11541303-18 from 1204224 to 1720320 tid 3 2022-04-22T17:28:39.825 INFO:tasks.rados.rados.0.smithi115.stdout:929: writing smithi11541303-18 from 1720320 to 2252800 tid 4 2022-04-22T17:28:39.825 INFO:tasks.rados.rados.0.smithi115.stdout:924: done (3 left) 2022-04-22T17:28:39.825 INFO:tasks.rados.rados.0.smithi115.stdout:930: delete oid 29 current snap is 92 2022-04-22T17:28:39.834 INFO:tasks.rados.rados.0.smithi115.stdout:926: expect (ObjNum 45 snap 0 seq_num 45) 2022-04-22T17:28:39.853 INFO:tasks.rados.rados.0.smithi115.stdout:927: finishing copy_from to smithi11541303-21 2022-04-22T17:28:39.900 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 313 (ObjNum 247 snap 83 seq_num 247) dirty exists 2022-04-22T17:28:39.900 INFO:tasks.rados.rados.0.smithi115.stdout:927: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:28:39.900 INFO:tasks.rados.rados.0.smithi115.stdout:926: done (3 left) 2022-04-22T17:28:39.900 INFO:tasks.rados.rados.0.smithi115.stdout:927: done (2 left) 2022-04-22T17:28:39.900 INFO:tasks.rados.rados.0.smithi115.stdout:930: done (1 left) 2022-04-22T17:28:39.901 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 92 2022-04-22T17:28:39.901 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 1 to smithi11541303-18 2022-04-22T17:28:39.901 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 2 to smithi11541303-18 2022-04-22T17:28:39.901 INFO:tasks.rados.rados.0.smithi115.stdout:931: seq_num 279 ranges {3776512=679936,4456448=720896,5177344=483328,5660672=401408} 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:931: writing smithi11541303-2 from 3776512 to 4456448 tid 1 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:931: writing smithi11541303-2 from 4456448 to 5177344 tid 2 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:931: writing smithi11541303-2 from 5177344 to 5660672 tid 3 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:931: writing smithi11541303-2 from 5660672 to 6062080 tid 4 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:932: rollback oid 36 current snap is 92 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 88 2022-04-22T17:28:39.903 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 3 to smithi11541303-18 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 4 to smithi11541303-18 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 5 to smithi11541303-18 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:929: finishing write tid 6 to smithi11541303-18 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 519 (ObjNum 277 snap 92 seq_num 277) dirty exists 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:929: left oid 18 (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 1 to smithi11541303-2 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 2 to smithi11541303-2 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:929: done (2 left) 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:933: snap_create 2022-04-22T17:28:39.904 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 3 to smithi11541303-2 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:932: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:932: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 4 to smithi11541303-2 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:932: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 383 (ObjNum 264 snap 88 seq_num 264) dirty exists 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 5 to smithi11541303-2 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:931: finishing write tid 6 to smithi11541303-2 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 375 (ObjNum 279 snap 92 seq_num 279) dirty exists 2022-04-22T17:28:39.905 INFO:tasks.rados.rados.0.smithi115.stdout:931: left oid 2 (ObjNum 279 snap 92 seq_num 279) 2022-04-22T17:28:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:39 smithi121 bash[12679]: cluster 2022-04-22T17:28:38.788840+0000 mon.a (mon.0) 105 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2022-04-22T17:28:40.823 INFO:tasks.rados.rados.0.smithi115.stdout:931: done (2 left) 2022-04-22T17:28:40.823 INFO:tasks.rados.rados.0.smithi115.stdout:932: done (1 left) 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:933: done (0 left) 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:934: rmattr oid 32 current snap is 93 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:935: read oid 14 snap -1 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:935: expect (ObjNum 160 snap 43 seq_num 160) 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:936: rollback oid 42 current snap is 93 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 90 2022-04-22T17:28:40.824 INFO:tasks.rados.rados.0.smithi115.stdout:937: delete oid 12 current snap is 93 2022-04-22T17:28:40.833 INFO:tasks.rados.rados.0.smithi115.stdout:935: expect (ObjNum 160 snap 43 seq_num 160) 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 221 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:936: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:936: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:936: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 523 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:934: done (3 left) 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:935: done (2 left) 2022-04-22T17:28:40.843 INFO:tasks.rados.rados.0.smithi115.stdout:936: done (1 left) 2022-04-22T17:28:40.844 INFO:tasks.rados.rados.0.smithi115.stdout:937: done (0 left) 2022-04-22T17:28:40.844 INFO:tasks.rados.rados.0.smithi115.stdout:938: delete oid 45 current snap is 93 2022-04-22T17:28:40.853 INFO:tasks.rados.rados.0.smithi115.stdout:938: done (0 left) 2022-04-22T17:28:40.853 INFO:tasks.rados.rados.0.smithi115.stdout:939: copy_from oid 25 from oid 31 current snap is 93 2022-04-22T17:28:40.853 INFO:tasks.rados.rados.0.smithi115.stdout:940: read oid 49 snap 91 2022-04-22T17:28:40.853 INFO:tasks.rados.rados.0.smithi115.stdout:940: expect deleted 2022-04-22T17:28:40.854 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 93 2022-04-22T17:28:40.854 INFO:tasks.rados.rados.0.smithi115.stdout:941: seq_num 282 ranges {13557760=729088,14286848=425984,14712832=704512,15417344=475136} 2022-04-22T17:28:40.936 INFO:tasks.rados.rados.0.smithi115.stdout:941: writing smithi11541303-23 from 13557760 to 14286848 tid 1 2022-04-22T17:28:40.939 INFO:tasks.rados.rados.0.smithi115.stdout:941: writing smithi11541303-23 from 14286848 to 14712832 tid 2 2022-04-22T17:28:40.943 INFO:tasks.rados.rados.0.smithi115.stdout:941: writing smithi11541303-23 from 14712832 to 15417344 tid 3 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:941: writing smithi11541303-23 from 15417344 to 15892480 tid 4 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:942: copy_from oid 32 from oid 42 current snap is 93 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:939: finishing copy_from to smithi11541303-25 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:939: got expected ENOENT (src dne) 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:939: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 1 to smithi11541303-23 2022-04-22T17:28:40.947 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 2 to smithi11541303-23 2022-04-22T17:28:40.948 INFO:tasks.rados.rados.0.smithi115.stdout:939: done (3 left) 2022-04-22T17:28:40.948 INFO:tasks.rados.rados.0.smithi115.stdout:940: done (2 left) 2022-04-22T17:28:40.948 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 93 2022-04-22T17:28:40.948 INFO:tasks.rados.rados.0.smithi115.stdout:943: seq_num 283 ranges {7577600=745472,8323072=712704,9035776=458752,9494528=434176} 2022-04-22T17:28:40.994 INFO:tasks.rados.rados.0.smithi115.stdout:943: writing smithi11541303-10 from 7577600 to 8323072 tid 1 2022-04-22T17:28:40.999 INFO:tasks.rados.rados.0.smithi115.stdout:943: writing smithi11541303-10 from 8323072 to 9035776 tid 2 2022-04-22T17:28:41.001 INFO:tasks.rados.rados.0.smithi115.stdout:943: writing smithi11541303-10 from 9035776 to 9494528 tid 3 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:943: writing smithi11541303-10 from 9494528 to 9928704 tid 4 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:944: read oid 13 snap 92 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:944: expect (ObjNum 134 snap 35 seq_num 134) 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 93 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 3 to smithi11541303-23 2022-04-22T17:28:41.004 INFO:tasks.rados.rados.0.smithi115.stdout:945: seq_num 284 ranges {11149312=761856,11911168=598016,12509184=671744,13180928=335872} 2022-04-22T17:28:41.072 INFO:tasks.rados.rados.0.smithi115.stdout:945: writing smithi11541303-4 from 11149312 to 11911168 tid 1 2022-04-22T17:28:41.075 INFO:tasks.rados.rados.0.smithi115.stdout:945: writing smithi11541303-4 from 11911168 to 12509184 tid 2 2022-04-22T17:28:41.079 INFO:tasks.rados.rados.0.smithi115.stdout:945: writing smithi11541303-4 from 12509184 to 13180928 tid 3 2022-04-22T17:28:41.081 INFO:tasks.rados.rados.0.smithi115.stdout:945: writing smithi11541303-4 from 13180928 to 13516800 tid 4 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:946: copy_from oid 34 from oid 39 current snap is 93 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 4 to smithi11541303-23 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 5 to smithi11541303-23 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:941: finishing write tid 6 to smithi11541303-23 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 384 (ObjNum 282 snap 93 seq_num 282) dirty exists 2022-04-22T17:28:41.082 INFO:tasks.rados.rados.0.smithi115.stdout:941: left oid 23 (ObjNum 282 snap 93 seq_num 282) 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:942: finishing copy_from to smithi11541303-32 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 523 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:942: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:941: done (5 left) 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:942: done (4 left) 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:947: read oid 3 snap -1 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:947: expect (ObjNum 272 snap 90 seq_num 272) 2022-04-22T17:28:41.083 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 93 2022-04-22T17:28:41.084 INFO:tasks.rados.rados.0.smithi115.stdout:948: seq_num 285 ranges {286720=778240,1064960=475136,1540096=475136,2015232=655360} 2022-04-22T17:28:41.088 INFO:tasks.rados.rados.0.smithi115.stdout:948: writing smithi11541303-14 from 286720 to 1064960 tid 1 2022-04-22T17:28:41.091 INFO:tasks.rados.rados.0.smithi115.stdout:948: writing smithi11541303-14 from 1064960 to 1540096 tid 2 2022-04-22T17:28:41.094 INFO:tasks.rados.rados.0.smithi115.stdout:948: writing smithi11541303-14 from 1540096 to 2015232 tid 3 2022-04-22T17:28:41.100 INFO:tasks.rados.rados.0.smithi115.stdout:948: writing smithi11541303-14 from 2015232 to 2670592 tid 4 2022-04-22T17:28:41.100 INFO:tasks.rados.rados.0.smithi115.stdout:949: read oid 19 snap -1 2022-04-22T17:28:41.100 INFO:tasks.rados.rados.0.smithi115.stdout:949: expect (ObjNum 269 snap 89 seq_num 269) 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:950: snap_create 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 1 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 2 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 3 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 4 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 5 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: finishing write tid 6 to smithi11541303-10 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 280 (ObjNum 283 snap 93 seq_num 283) dirty exists 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:943: left oid 10 (ObjNum 283 snap 93 seq_num 283) 2022-04-22T17:28:41.101 INFO:tasks.rados.rados.0.smithi115.stdout:944: expect (ObjNum 134 snap 35 seq_num 134) 2022-04-22T17:28:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:40 smithi121 bash[12679]: cluster 2022-04-22T17:28:39.124554+0000 mgr.y (mgr.34107) 27 : cluster [DBG] pgmap v39: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:28:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:40 smithi121 bash[12679]: cluster 2022-04-22T17:28:39.797273+0000 mon.a (mon.0) 106 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 1 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 2 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 3 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 4 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 5 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:945: finishing write tid 6 to smithi11541303-4 2022-04-22T17:28:41.657 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 477 (ObjNum 284 snap 93 seq_num 284) dirty exists 2022-04-22T17:28:41.658 INFO:tasks.rados.rados.0.smithi115.stdout:945: left oid 4 (ObjNum 284 snap 93 seq_num 284) 2022-04-22T17:28:41.658 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 1 to smithi11541303-14 2022-04-22T17:28:41.658 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 2 to smithi11541303-14 2022-04-22T17:28:41.658 INFO:tasks.rados.rados.0.smithi115.stdout:947: expect (ObjNum 272 snap 90 seq_num 272) 2022-04-22T17:28:41.714 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 3 to smithi11541303-14 2022-04-22T17:28:41.714 INFO:tasks.rados.rados.0.smithi115.stdout:946: finishing copy_from to smithi11541303-34 2022-04-22T17:28:41.714 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:28:41.714 INFO:tasks.rados.rados.0.smithi115.stdout:946: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 4 to smithi11541303-14 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 5 to smithi11541303-14 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:948: finishing write tid 6 to smithi11541303-14 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 240 (ObjNum 285 snap 93 seq_num 285) dirty exists 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:948: left oid 14 (ObjNum 285 snap 93 seq_num 285) 2022-04-22T17:28:41.715 INFO:tasks.rados.rados.0.smithi115.stdout:949: expect (ObjNum 269 snap 89 seq_num 269) 2022-04-22T17:28:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:41.616Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:41.617Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:42.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:41.617Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:42.059 INFO:tasks.rados.rados.0.smithi115.stdout:943: done (7 left) 2022-04-22T17:28:42.059 INFO:tasks.rados.rados.0.smithi115.stdout:944: done (6 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:945: done (5 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:946: done (4 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:947: done (3 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:948: done (2 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:949: done (1 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:950: done (0 left) 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:951: copy_from oid 2 from oid 18 current snap is 94 2022-04-22T17:28:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:952: snap_create 2022-04-22T17:28:42.088 INFO:tasks.rados.rados.0.smithi115.stdout:951: finishing copy_from to smithi11541303-2 2022-04-22T17:28:42.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 519 (ObjNum 277 snap 92 seq_num 277) dirty exists 2022-04-22T17:28:42.089 INFO:tasks.rados.rados.0.smithi115.stdout:951: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:28:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:41 smithi121 bash[12679]: cluster 2022-04-22T17:28:40.810003+0000 mon.a (mon.0) 107 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2022-04-22T17:28:42.527 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:28:42 smithi115 bash[42914]: debug 2022-04-22T17:28:42.038+0000 7f8eea62c700 -1 mgr.server handle_report got status from non-daemon mon.c 2022-04-22T17:28:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:951: done (1 left) 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:952: done (0 left) 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:953: read oid 19 snap -1 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:953: expect (ObjNum 269 snap 89 seq_num 269) 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:954: read oid 26 snap 88 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:954: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:955: read oid 22 snap -1 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:955: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:956: read oid 45 snap -1 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:956: expect deleted 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:957: read oid 1 snap 94 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:957: expect (ObjNum 273 snap 91 seq_num 273) 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:958: rollback oid 48 current snap is 95 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 48 to 89 2022-04-22T17:28:43.065 INFO:tasks.rados.rados.0.smithi115.stdout:959: snap_remove snap 69 2022-04-22T17:28:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:42 smithi121 bash[12679]: cluster 2022-04-22T17:28:41.125017+0000 mgr.y (mgr.34107) 28 : cluster [DBG] pgmap v42: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:28:43.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:42 smithi121 bash[12679]: cluster 2022-04-22T17:28:41.815343+0000 mon.a (mon.0) 108 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2022-04-22T17:28:43.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:42 smithi121 bash[12679]: audit 2022-04-22T17:28:42.530600+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.501Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.501Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.501Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.512Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.512Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:43.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:44.073 INFO:tasks.rados.rados.0.smithi115.stdout:959: done (6 left) 2022-04-22T17:28:44.074 INFO:tasks.rados.rados.0.smithi115.stdout:960: read oid 40 snap -1 2022-04-22T17:28:44.074 INFO:tasks.rados.rados.0.smithi115.stdout:960: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:44.078 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 95 2022-04-22T17:28:44.078 INFO:tasks.rados.rados.0.smithi115.stdout:961: seq_num 286 ranges {2138112=794624,2932736=761856,3694592=679936,4374528=163840} 2022-04-22T17:28:44.092 INFO:tasks.rados.rados.0.smithi115.stdout:961: writing smithi11541303-36 from 2138112 to 2932736 tid 1 2022-04-22T17:28:44.097 INFO:tasks.rados.rados.0.smithi115.stdout:961: writing smithi11541303-36 from 2932736 to 3694592 tid 2 2022-04-22T17:28:44.101 INFO:tasks.rados.rados.0.smithi115.stdout:961: writing smithi11541303-36 from 3694592 to 4374528 tid 3 2022-04-22T17:28:44.103 INFO:tasks.rados.rados.0.smithi115.stdout:961: writing smithi11541303-36 from 4374528 to 4538368 tid 4 2022-04-22T17:28:44.103 INFO:tasks.rados.rados.0.smithi115.stdout:962: copy_from oid 4 from oid 33 current snap is 95 2022-04-22T17:28:44.103 INFO:tasks.rados.rados.0.smithi115.stdout:955: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:44.104 INFO:tasks.rados.rados.0.smithi115.stdout:958: finishing rollback tid 0 to smithi11541303-48 2022-04-22T17:28:44.104 INFO:tasks.rados.rados.0.smithi115.stdout:958: finishing rollback tid 1 to smithi11541303-48 2022-04-22T17:28:44.104 INFO:tasks.rados.rados.0.smithi115.stdout:958: finishing rollback tid 2 to smithi11541303-48 2022-04-22T17:28:44.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 481 (ObjNum 250 snap 83 seq_num 250) dirty exists 2022-04-22T17:28:44.104 INFO:tasks.rados.rados.0.smithi115.stdout:957: expect (ObjNum 273 snap 91 seq_num 273) 2022-04-22T17:28:44.166 INFO:tasks.rados.rados.0.smithi115.stdout:954: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:44.376 INFO:tasks.rados.rados.0.smithi115.stdout:953: expect (ObjNum 269 snap 89 seq_num 269) 2022-04-22T17:28:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:44 smithi121 bash[12679]: cluster 2022-04-22T17:28:43.051015+0000 mon.a (mon.0) 110 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2022-04-22T17:28:44.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:44 smithi121 bash[12679]: audit 2022-04-22T17:28:43.134927+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:44.706 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 1 to smithi11541303-36 2022-04-22T17:28:44.707 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 2 to smithi11541303-36 2022-04-22T17:28:44.707 INFO:tasks.rados.rados.0.smithi115.stdout:960: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:962: finishing copy_from to smithi11541303-4 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:962: got expected ENOENT (src dne) 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:962: finishing copy_from racing read to smithi11541303-4 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 3 to smithi11541303-36 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 4 to smithi11541303-36 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 5 to smithi11541303-36 2022-04-22T17:28:44.913 INFO:tasks.rados.rados.0.smithi115.stdout:961: finishing write tid 6 to smithi11541303-36 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 388 (ObjNum 286 snap 95 seq_num 286) dirty exists 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:961: left oid 36 (ObjNum 286 snap 95 seq_num 286) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:953: done (8 left) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:954: done (7 left) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:955: done (6 left) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:956: done (5 left) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:957: done (4 left) 2022-04-22T17:28:44.914 INFO:tasks.rados.rados.0.smithi115.stdout:958: done (3 left) 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:960: done (2 left) 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:961: done (1 left) 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:962: done (0 left) 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:963: read oid 26 snap -1 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:963: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:964: read oid 43 snap -1 2022-04-22T17:28:44.915 INFO:tasks.rados.rados.0.smithi115.stdout:964: expect deleted 2022-04-22T17:28:44.916 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 95 2022-04-22T17:28:44.916 INFO:tasks.rados.rados.0.smithi115.stdout:965: seq_num 287 ranges {0=8192} 2022-04-22T17:28:44.916 INFO:tasks.rados.rados.0.smithi115.stdout:965: writing smithi11541303-31 from 0 to 8192 tid 1 2022-04-22T17:28:44.916 INFO:tasks.rados.rados.0.smithi115.stdout:966: snap_create 2022-04-22T17:28:44.919 INFO:tasks.rados.rados.0.smithi115.stdout:965: finishing write tid 1 to smithi11541303-31 2022-04-22T17:28:44.919 INFO:tasks.rados.rados.0.smithi115.stdout:965: finishing write tid 2 to smithi11541303-31 2022-04-22T17:28:44.924 INFO:tasks.rados.rados.0.smithi115.stdout:965: finishing write tid 3 to smithi11541303-31 2022-04-22T17:28:44.924 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 531 (ObjNum 287 snap 95 seq_num 287) dirty exists 2022-04-22T17:28:44.924 INFO:tasks.rados.rados.0.smithi115.stdout:965: left oid 31 (ObjNum 287 snap 95 seq_num 287) 2022-04-22T17:28:44.933 INFO:tasks.rados.rados.0.smithi115.stdout:963: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:28:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:45 smithi121 bash[12679]: cluster 2022-04-22T17:28:43.125765+0000 mgr.y (mgr.34107) 29 : cluster [DBG] pgmap v45: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:28:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:45 smithi121 bash[12679]: cluster 2022-04-22T17:28:44.057851+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2022-04-22T17:28:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:45 smithi121 bash[12679]: cluster 2022-04-22T17:28:44.722748+0000 mon.a (mon.0) 113 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2022-04-22T17:28:45.745 INFO:tasks.rados.rados.0.smithi115.stdout:963: done (3 left) 2022-04-22T17:28:45.745 INFO:tasks.rados.rados.0.smithi115.stdout:964: done (2 left) 2022-04-22T17:28:45.745 INFO:tasks.rados.rados.0.smithi115.stdout:965: done (1 left) 2022-04-22T17:28:45.745 INFO:tasks.rados.rados.0.smithi115.stdout:966: done (0 left) 2022-04-22T17:28:45.745 INFO:tasks.rados.rados.0.smithi115.stdout:967: copy_from oid 12 from oid 23 current snap is 96 2022-04-22T17:28:45.746 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 96 2022-04-22T17:28:45.746 INFO:tasks.rados.rados.0.smithi115.stdout:968: seq_num 288 ranges {6070272=24576} 2022-04-22T17:28:45.785 INFO:tasks.rados.rados.0.smithi115.stdout:968: writing smithi11541303-19 from 6070272 to 6094848 tid 1 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:969: read oid 2 snap -1 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:969: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:970: read oid 14 snap -1 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:970: expect (ObjNum 285 snap 93 seq_num 285) 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:971: copy_from oid 31 from oid 17 current snap is 96 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:972: read oid 5 snap -1 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:972: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:45.786 INFO:tasks.rados.rados.0.smithi115.stdout:973: snap_remove snap 81 2022-04-22T17:28:46.753 INFO:tasks.rados.rados.0.smithi115.stdout:973: done (6 left) 2022-04-22T17:28:46.753 INFO:tasks.rados.rados.0.smithi115.stdout:974: snap_create 2022-04-22T17:28:46.754 INFO:tasks.rados.rados.0.smithi115.stdout:969: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:28:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:970: expect (ObjNum 285 snap 93 seq_num 285) 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:968: finishing write tid 1 to smithi11541303-19 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:968: finishing write tid 2 to smithi11541303-19 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:971: finishing copy_from to smithi11541303-31 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:971: got expected ENOENT (src dne) 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:971: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:968: finishing write tid 3 to smithi11541303-19 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 331 (ObjNum 288 snap 96 seq_num 288) dirty exists 2022-04-22T17:28:46.893 INFO:tasks.rados.rados.0.smithi115.stdout:968: left oid 19 (ObjNum 288 snap 96 seq_num 288) 2022-04-22T17:28:46.894 INFO:tasks.rados.rados.0.smithi115.stdout:972: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:46 smithi121 bash[12679]: cluster 2022-04-22T17:28:45.126474+0000 mgr.y (mgr.34107) 30 : cluster [DBG] pgmap v48: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:28:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:46 smithi121 bash[12679]: cluster 2022-04-22T17:28:45.732625+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2022-04-22T17:28:47.062 INFO:tasks.rados.rados.0.smithi115.stdout:967: finishing copy_from to smithi11541303-12 2022-04-22T17:28:47.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 384 (ObjNum 282 snap 93 seq_num 282) dirty exists 2022-04-22T17:28:47.063 INFO:tasks.rados.rados.0.smithi115.stdout:967: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:28:47.767 INFO:tasks.rados.rados.0.smithi115.stdout:967: done (6 left) 2022-04-22T17:28:47.767 INFO:tasks.rados.rados.0.smithi115.stdout:968: done (5 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:969: done (4 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:970: done (3 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:971: done (2 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:972: done (1 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:974: done (0 left) 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:975: read oid 29 snap -1 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:975: expect deleted 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 97 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:976: seq_num 289 ranges {0=40960} 2022-04-22T17:28:47.768 INFO:tasks.rados.rados.0.smithi115.stdout:976: writing smithi11541303-6 from 0 to 40960 tid 1 2022-04-22T17:28:47.769 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 97 2022-04-22T17:28:47.769 INFO:tasks.rados.rados.0.smithi115.stdout:977: seq_num 290 ranges {5193728=57344} 2022-04-22T17:28:47.797 INFO:tasks.rados.rados.0.smithi115.stdout:977: writing smithi11541303-24 from 5193728 to 5251072 tid 1 2022-04-22T17:28:47.831 INFO:tasks.rados.rados.0.smithi115.stdout:978: delete oid 33 current snap is 97 2022-04-22T17:28:47.831 INFO:tasks.rados.rados.0.smithi115.stdout:976: finishing write tid 1 to smithi11541303-6 2022-04-22T17:28:47.832 INFO:tasks.rados.rados.0.smithi115.stdout:976: finishing write tid 2 to smithi11541303-6 2022-04-22T17:28:47.832 INFO:tasks.rados.rados.0.smithi115.stdout:976: finishing write tid 3 to smithi11541303-6 2022-04-22T17:28:47.832 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 132 (ObjNum 289 snap 97 seq_num 289) dirty exists 2022-04-22T17:28:47.832 INFO:tasks.rados.rados.0.smithi115.stdout:976: left oid 6 (ObjNum 289 snap 97 seq_num 289) 2022-04-22T17:28:47.833 INFO:tasks.rados.rados.0.smithi115.stdout:975: done (3 left) 2022-04-22T17:28:47.833 INFO:tasks.rados.rados.0.smithi115.stdout:976: done (2 left) 2022-04-22T17:28:47.833 INFO:tasks.rados.rados.0.smithi115.stdout:978: done (1 left) 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:979: rollback oid 38 current snap is 97 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 78 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:980: delete oid 3 current snap is 97 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:977: finishing write tid 1 to smithi11541303-24 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:977: finishing write tid 2 to smithi11541303-24 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:979: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 213 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:977: finishing write tid 3 to smithi11541303-24 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 135 (ObjNum 290 snap 97 seq_num 290) dirty exists 2022-04-22T17:28:47.834 INFO:tasks.rados.rados.0.smithi115.stdout:977: left oid 24 (ObjNum 290 snap 97 seq_num 290) 2022-04-22T17:28:47.835 INFO:tasks.rados.rados.0.smithi115.stdout:977: done (2 left) 2022-04-22T17:28:47.835 INFO:tasks.rados.rados.0.smithi115.stdout:979: done (1 left) 2022-04-22T17:28:47.835 INFO:tasks.rados.rados.0.smithi115.stdout:980: done (0 left) 2022-04-22T17:28:47.835 INFO:tasks.rados.rados.0.smithi115.stdout:981: snap_create 2022-04-22T17:28:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:47 smithi121 bash[12679]: cluster 2022-04-22T17:28:46.740735+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2022-04-22T17:28:48.773 INFO:tasks.rados.rados.0.smithi115.stdout:981: done (0 left) 2022-04-22T17:28:48.773 INFO:tasks.rados.rados.0.smithi115.stdout:982: read oid 12 snap -1 2022-04-22T17:28:48.773 INFO:tasks.rados.rados.0.smithi115.stdout:982: expect (ObjNum 282 snap 93 seq_num 282) 2022-04-22T17:28:48.774 INFO:tasks.rados.rados.0.smithi115.stdout:983: rollback oid 47 current snap is 98 2022-04-22T17:28:48.774 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 47 to 74 2022-04-22T17:28:48.774 INFO:tasks.rados.rados.0.smithi115.stdout:984: copy_from oid 22 from oid 35 current snap is 98 2022-04-22T17:28:48.774 INFO:tasks.rados.rados.0.smithi115.stdout:985: delete oid 3 current snap is 98 2022-04-22T17:28:48.776 INFO:tasks.rados.rados.0.smithi115.stdout:984: finishing copy_from to smithi11541303-22 2022-04-22T17:28:48.776 INFO:tasks.rados.rados.0.smithi115.stdout:984: got expected ENOENT (src dne) 2022-04-22T17:28:48.777 INFO:tasks.rados.rados.0.smithi115.stdout:984: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:28:48.781 INFO:tasks.rados.rados.0.smithi115.stdout:984: done (3 left) 2022-04-22T17:28:48.781 INFO:tasks.rados.rados.0.smithi115.stdout:985: done (2 left) 2022-04-22T17:28:48.781 INFO:tasks.rados.rados.0.smithi115.stdout:986: rmattr oid 34 current snap is 98 2022-04-22T17:28:48.781 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 98 2022-04-22T17:28:48.781 INFO:tasks.rados.rados.0.smithi115.stdout:987: seq_num 294 ranges {40960=131072} 2022-04-22T17:28:48.782 INFO:tasks.rados.rados.0.smithi115.stdout:987: writing smithi11541303-6 from 40960 to 172032 tid 1 2022-04-22T17:28:48.782 INFO:tasks.rados.rados.0.smithi115.stdout:988: snap_create 2022-04-22T17:28:48.783 INFO:tasks.rados.rados.0.smithi115.stdout:983: finishing rollback tid 0 to smithi11541303-47 2022-04-22T17:28:48.783 INFO:tasks.rados.rados.0.smithi115.stdout:983: finishing rollback tid 1 to smithi11541303-47 2022-04-22T17:28:48.783 INFO:tasks.rados.rados.0.smithi115.stdout:983: finishing rollback tid 2 to smithi11541303-47 2022-04-22T17:28:48.783 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 296 (ObjNum 16 snap 0 seq_num 16) dirty exists 2022-04-22T17:28:48.786 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 371 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:28:48.790 INFO:tasks.rados.rados.0.smithi115.stdout:987: finishing write tid 1 to smithi11541303-6 2022-04-22T17:28:48.791 INFO:tasks.rados.rados.0.smithi115.stdout:987: finishing write tid 2 to smithi11541303-6 2022-04-22T17:28:48.792 INFO:tasks.rados.rados.0.smithi115.stdout:987: finishing write tid 3 to smithi11541303-6 2022-04-22T17:28:48.792 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 138 (ObjNum 294 snap 98 seq_num 294) dirty exists 2022-04-22T17:28:48.792 INFO:tasks.rados.rados.0.smithi115.stdout:987: left oid 6 (ObjNum 294 snap 98 seq_num 294) 2022-04-22T17:28:48.839 INFO:tasks.rados.rados.0.smithi115.stdout:982: expect (ObjNum 282 snap 93 seq_num 282) 2022-04-22T17:28:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:48 smithi121 bash[12679]: cluster 2022-04-22T17:28:47.127211+0000 mgr.y (mgr.34107) 31 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:28:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:48 smithi121 bash[12679]: cluster 2022-04-22T17:28:47.752168+0000 mon.a (mon.0) 116 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2022-04-22T17:28:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:49 smithi121 bash[12679]: cluster 2022-04-22T17:28:48.759403+0000 mon.a (mon.0) 117 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2022-04-22T17:28:50.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:49 smithi121 bash[12679]: cluster 2022-04-22T17:28:49.727308+0000 mon.a (mon.0) 118 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2022-04-22T17:28:50.389 INFO:tasks.rados.rados.0.smithi115.stdout:982: done (4 left) 2022-04-22T17:28:50.390 INFO:tasks.rados.rados.0.smithi115.stdout:983: done (3 left) 2022-04-22T17:28:50.390 INFO:tasks.rados.rados.0.smithi115.stdout:986: done (2 left) 2022-04-22T17:28:50.390 INFO:tasks.rados.rados.0.smithi115.stdout:987: done (1 left) 2022-04-22T17:28:50.390 INFO:tasks.rados.rados.0.smithi115.stdout:988: done (0 left) 2022-04-22T17:28:50.390 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 99 2022-04-22T17:28:50.391 INFO:tasks.rados.rados.0.smithi115.stdout:989: seq_num 295 ranges {2850816=147456} 2022-04-22T17:28:50.407 INFO:tasks.rados.rados.0.smithi115.stdout:989: writing smithi11541303-39 from 2850816 to 2998272 tid 1 2022-04-22T17:28:50.407 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 99 2022-04-22T17:28:50.407 INFO:tasks.rados.rados.0.smithi115.stdout:990: seq_num 296 ranges {0=163840} 2022-04-22T17:28:50.408 INFO:tasks.rados.rados.0.smithi115.stdout:990: writing smithi11541303-43 from 0 to 163840 tid 1 2022-04-22T17:28:50.408 INFO:tasks.rados.rados.0.smithi115.stdout:991: read oid 2 snap -1 2022-04-22T17:28:50.408 INFO:tasks.rados.rados.0.smithi115.stdout:991: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:28:50.408 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 99 2022-04-22T17:28:50.408 INFO:tasks.rados.rados.0.smithi115.stdout:992: seq_num 297 ranges {3858432=180224} 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:992: writing smithi11541303-11 from 3858432 to 4038656 tid 1 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:993: rmattr oid 14 current snap is 99 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:993: done (4 left) 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:994: rmattr oid 24 current snap is 99 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:990: finishing write tid 1 to smithi11541303-43 2022-04-22T17:28:50.430 INFO:tasks.rados.rados.0.smithi115.stdout:995: snap_remove snap 94 2022-04-22T17:28:50.795 INFO:tasks.rados.rados.0.smithi115.stdout:995: done (5 left) 2022-04-22T17:28:50.795 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 99 2022-04-22T17:28:50.795 INFO:tasks.rados.rados.0.smithi115.stdout:996: seq_num 298 ranges {9134080=196608} 2022-04-22T17:28:50.848 INFO:tasks.rados.rados.0.smithi115.stdout:996: writing smithi11541303-13 from 9134080 to 9330688 tid 1 2022-04-22T17:28:50.848 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 99 2022-04-22T17:28:50.848 INFO:tasks.rados.rados.0.smithi115.stdout:990: finishing write tid 2 to smithi11541303-43 2022-04-22T17:28:50.848 INFO:tasks.rados.rados.0.smithi115.stdout:989: finishing write tid 1 to smithi11541303-39 2022-04-22T17:28:50.848 INFO:tasks.rados.rados.0.smithi115.stdout:990: finishing write tid 3 to smithi11541303-43 2022-04-22T17:28:50.849 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 375 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:28:50.849 INFO:tasks.rados.rados.0.smithi115.stdout:990: left oid 43 (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:28:50.849 INFO:tasks.rados.rados.0.smithi115.stdout:989: finishing write tid 2 to smithi11541303-39 2022-04-22T17:28:50.849 INFO:tasks.rados.rados.0.smithi115.stdout:997: seq_num 299 ranges {8192=212992} 2022-04-22T17:28:50.850 INFO:tasks.rados.rados.0.smithi115.stdout:997: writing smithi11541303-31 from 8192 to 221184 tid 1 2022-04-22T17:28:50.850 INFO:tasks.rados.rados.0.smithi115.stdout:990: done (6 left) 2022-04-22T17:28:50.850 INFO:tasks.rados.rados.0.smithi115.stdout:998: snap_create 2022-04-22T17:28:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:991: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:28:50.914 INFO:tasks.rados.rados.0.smithi115.stdout:989: finishing write tid 3 to smithi11541303-39 2022-04-22T17:28:50.914 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 521 (ObjNum 295 snap 99 seq_num 295) dirty exists 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:989: left oid 39 (ObjNum 295 snap 99 seq_num 295) 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 0 (ObjNum 290 snap 97 seq_num 290) dirty exists 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:992: finishing write tid 1 to smithi11541303-11 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:992: finishing write tid 2 to smithi11541303-11 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:992: finishing write tid 3 to smithi11541303-11 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 402 (ObjNum 297 snap 99 seq_num 297) dirty exists 2022-04-22T17:28:50.915 INFO:tasks.rados.rados.0.smithi115.stdout:992: left oid 11 (ObjNum 297 snap 99 seq_num 297) 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:996: finishing write tid 1 to smithi11541303-13 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:996: finishing write tid 2 to smithi11541303-13 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:996: finishing write tid 3 to smithi11541303-13 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 378 (ObjNum 298 snap 99 seq_num 298) dirty exists 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:996: left oid 13 (ObjNum 298 snap 99 seq_num 298) 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:997: finishing write tid 1 to smithi11541303-31 2022-04-22T17:28:50.916 INFO:tasks.rados.rados.0.smithi115.stdout:997: finishing write tid 2 to smithi11541303-31 2022-04-22T17:28:50.917 INFO:tasks.rados.rados.0.smithi115.stdout:997: finishing write tid 3 to smithi11541303-31 2022-04-22T17:28:50.917 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 538 (ObjNum 299 snap 99 seq_num 299) dirty exists 2022-04-22T17:28:50.917 INFO:tasks.rados.rados.0.smithi115.stdout:997: left oid 31 (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:28:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:50 smithi121 bash[12679]: cluster 2022-04-22T17:28:49.127686+0000 mgr.y (mgr.34107) 32 : cluster [DBG] pgmap v54: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:989: done (6 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:991: done (5 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:992: done (4 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:994: done (3 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:996: done (2 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:997: done (1 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:998: done (0 left) 2022-04-22T17:28:51.808 INFO:tasks.rados.rados.0.smithi115.stdout:999: read oid 5 snap 96 2022-04-22T17:28:51.809 INFO:tasks.rados.rados.0.smithi115.stdout:999: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:51.809 INFO:tasks.rados.rados.0.smithi115.stdout:1000: read oid 47 snap 96 2022-04-22T17:28:51.809 INFO:tasks.rados.rados.0.smithi115.stdout:1000: expect (ObjNum 256 snap 85 seq_num 256) 2022-04-22T17:28:51.809 INFO:tasks.rados.rados.0.smithi115.stdout:1001: copy_from oid 23 from oid 15 current snap is 100 2022-04-22T17:28:51.809 INFO:tasks.rados.rados.0.smithi115.stdout:1002: delete oid 9 current snap is 100 2022-04-22T17:28:51.832 INFO:tasks.rados.rados.0.smithi115.stdout:1000: expect (ObjNum 256 snap 85 seq_num 256) 2022-04-22T17:28:51.898 INFO:tasks.rados.rados.0.smithi115.stdout:999: expect (ObjNum 147 snap 39 seq_num 147) 2022-04-22T17:28:51.944 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: cluster 2022-04-22T17:28:50.782260+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2022-04-22T17:28:51.944 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.576741+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:51.944 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.582903+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:51.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.592040+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:28:51.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.594864+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:28:51.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.596369+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:28:51.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:51 smithi121 bash[12679]: audit 2022-04-22T17:28:51.597474+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:28:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:51.617Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:51.617Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:51.617Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:52.080 INFO:tasks.rados.rados.0.smithi115.stdout:1001: finishing copy_from to smithi11541303-23 2022-04-22T17:28:52.080 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 186 (ObjNum 257 snap 86 seq_num 257) dirty exists 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:1001: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:999: done (3 left) 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:1000: done (2 left) 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:1001: done (1 left) 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:1002: done (0 left) 2022-04-22T17:28:52.081 INFO:tasks.rados.rados.0.smithi115.stdout:1003: setattr oid 40 current snap is 100 2022-04-22T17:28:52.083 INFO:tasks.rados.rados.0.smithi115.stdout:1004: rollback oid 26 current snap is 100 2022-04-22T17:28:52.083 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 96 2022-04-22T17:28:52.083 INFO:tasks.rados.rados.0.smithi115.stdout:1005: delete oid 42 current snap is 100 2022-04-22T17:28:52.087 INFO:tasks.rados.rados.0.smithi115.stdout:1005: done (2 left) 2022-04-22T17:28:52.087 INFO:tasks.rados.rados.0.smithi115.stdout:1006: copy_from oid 49 from oid 42 current snap is 100 2022-04-22T17:28:52.087 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 100 2022-04-22T17:28:52.087 INFO:tasks.rados.rados.0.smithi115.stdout:1007: seq_num 302 ranges {3686400=262144} 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:1007: writing smithi11541303-5 from 3686400 to 3948544 tid 1 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:1008: delete oid 48 current snap is 100 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:1006: finishing copy_from to smithi11541303-49 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:1006: got expected ENOENT (src dne) 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:1006: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:28:52.113 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 300 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:52.115 INFO:tasks.rados.rados.0.smithi115.stdout:1004: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:28:52.115 INFO:tasks.rados.rados.0.smithi115.stdout:1004: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:28:52.115 INFO:tasks.rados.rados.0.smithi115.stdout:1004: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:28:52.115 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 526 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:28:52.118 INFO:tasks.rados.rados.0.smithi115.stdout:1003: done (4 left) 2022-04-22T17:28:52.118 INFO:tasks.rados.rados.0.smithi115.stdout:1004: done (3 left) 2022-04-22T17:28:52.118 INFO:tasks.rados.rados.0.smithi115.stdout:1006: done (2 left) 2022-04-22T17:28:52.118 INFO:tasks.rados.rados.0.smithi115.stdout:1008: done (1 left) 2022-04-22T17:28:52.118 INFO:tasks.rados.rados.0.smithi115.stdout:1009: snap_create 2022-04-22T17:28:52.119 INFO:tasks.rados.rados.0.smithi115.stdout:1007: finishing write tid 1 to smithi11541303-5 2022-04-22T17:28:52.119 INFO:tasks.rados.rados.0.smithi115.stdout:1007: finishing write tid 2 to smithi11541303-5 2022-04-22T17:28:52.120 INFO:tasks.rados.rados.0.smithi115.stdout:1007: finishing write tid 3 to smithi11541303-5 2022-04-22T17:28:52.120 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 545 (ObjNum 302 snap 100 seq_num 302) dirty exists 2022-04-22T17:28:52.120 INFO:tasks.rados.rados.0.smithi115.stdout:1007: left oid 5 (ObjNum 302 snap 100 seq_num 302) 2022-04-22T17:28:52.817 INFO:tasks.rados.rados.0.smithi115.stdout:1007: done (1 left) 2022-04-22T17:28:52.817 INFO:tasks.rados.rados.0.smithi115.stdout:1009: done (0 left) 2022-04-22T17:28:52.817 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 101 2022-04-22T17:28:52.817 INFO:tasks.rados.rados.0.smithi115.stdout:1010: seq_num 304 ranges {270336=294912} 2022-04-22T17:28:52.822 INFO:tasks.rados.rados.0.smithi115.stdout:1010: writing smithi11541303-47 from 270336 to 565248 tid 1 2022-04-22T17:28:52.822 INFO:tasks.rados.rados.0.smithi115.stdout:1011: snap_remove snap 74 2022-04-22T17:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:52 smithi121 bash[12679]: cluster 2022-04-22T17:28:51.128359+0000 mgr.y (mgr.34107) 33 : cluster [DBG] pgmap v57: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:52 smithi121 bash[12679]: cephadm 2022-04-22T17:28:51.566824+0000 mgr.y (mgr.34107) 34 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:52 smithi121 bash[12679]: cephadm 2022-04-22T17:28:51.598239+0000 mgr.y (mgr.34107) 35 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2022-04-22T17:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:52 smithi121 bash[12679]: cluster 2022-04-22T17:28:51.794355+0000 mon.a (mon.0) 126 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2022-04-22T17:28:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.502Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.502Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.502Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:28:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:28:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:28:53.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:28:53.824 INFO:tasks.rados.rados.0.smithi115.stdout:1011: done (1 left) 2022-04-22T17:28:53.824 INFO:tasks.rados.rados.0.smithi115.stdout:1012: delete oid 9 current snap is 101 2022-04-22T17:28:53.825 INFO:tasks.rados.rados.0.smithi115.stdout:1010: finishing write tid 1 to smithi11541303-47 2022-04-22T17:28:53.825 INFO:tasks.rados.rados.0.smithi115.stdout:1010: finishing write tid 2 to smithi11541303-47 2022-04-22T17:28:53.825 INFO:tasks.rados.rados.0.smithi115.stdout:1010: finishing write tid 3 to smithi11541303-47 2022-04-22T17:28:53.825 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 303 (ObjNum 304 snap 101 seq_num 304) dirty exists 2022-04-22T17:28:53.825 INFO:tasks.rados.rados.0.smithi115.stdout:1010: left oid 47 (ObjNum 304 snap 101 seq_num 304) 2022-04-22T17:28:53.844 INFO:tasks.rados.rados.0.smithi115.stdout:1010: done (1 left) 2022-04-22T17:28:53.844 INFO:tasks.rados.rados.0.smithi115.stdout:1012: done (0 left) 2022-04-22T17:28:53.844 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 101 2022-04-22T17:28:53.844 INFO:tasks.rados.rados.0.smithi115.stdout:1013: seq_num 306 ranges {761856=327680} 2022-04-22T17:28:53.850 INFO:tasks.rados.rados.0.smithi115.stdout:1013: writing smithi11541303-46 from 761856 to 1089536 tid 1 2022-04-22T17:28:53.850 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 101 2022-04-22T17:28:53.850 INFO:tasks.rados.rados.0.smithi115.stdout:1014: seq_num 307 ranges {4145152=344064} 2022-04-22T17:28:53.875 INFO:tasks.rados.rados.0.smithi115.stdout:1014: writing smithi11541303-15 from 4145152 to 4489216 tid 1 2022-04-22T17:28:53.876 INFO:tasks.rados.rados.0.smithi115.stdout:1015: read oid 31 snap -1 2022-04-22T17:28:53.876 INFO:tasks.rados.rados.0.smithi115.stdout:1015: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:28:53.876 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 101 2022-04-22T17:28:53.876 INFO:tasks.rados.rados.0.smithi115.stdout:1013: finishing write tid 1 to smithi11541303-46 2022-04-22T17:28:53.876 INFO:tasks.rados.rados.0.smithi115.stdout:1016: seq_num 308 ranges {4145152=360448} 2022-04-22T17:28:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1016: writing smithi11541303-23 from 4145152 to 4505600 tid 1 2022-04-22T17:28:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1017: snap_remove snap 93 2022-04-22T17:28:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:53 smithi121 bash[12679]: cluster 2022-04-22T17:28:52.802734+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2022-04-22T17:28:54.828 INFO:tasks.rados.rados.0.smithi115.stdout:1017: done (4 left) 2022-04-22T17:28:54.869 INFO:tasks.rados.rados.0.smithi115.stdout:1018: rmattr oid 47 current snap is 101 2022-04-22T17:28:54.870 INFO:tasks.rados.rados.0.smithi115.stdout:1018: done (4 left) 2022-04-22T17:28:54.870 INFO:tasks.rados.rados.0.smithi115.stdout:1019: snap_remove snap 88 2022-04-22T17:28:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: cluster 2022-04-22T17:28:53.129141+0000 mgr.y (mgr.34107) 36 : cluster [DBG] pgmap v60: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:28:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: cluster 2022-04-22T17:28:53.811619+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2022-04-22T17:28:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: audit 2022-04-22T17:28:54.273786+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:28:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: audit 2022-04-22T17:28:54.275437+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:28:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: audit 2022-04-22T17:28:54.276653+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:28:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:54 smithi121 bash[12679]: audit 2022-04-22T17:28:54.277867+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:28:55.847 INFO:tasks.rados.rados.0.smithi115.stdout:1019: done (4 left) 2022-04-22T17:28:55.847 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 101 2022-04-22T17:28:55.847 INFO:tasks.rados.rados.0.smithi115.stdout:1020: seq_num 309 ranges {0=376832} 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1020: writing smithi11541303-45 from 0 to 376832 tid 1 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1021: read oid 44 snap -1 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1021: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1013: finishing write tid 2 to smithi11541303-46 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1013: finishing write tid 3 to smithi11541303-46 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 417 (ObjNum 306 snap 101 seq_num 306) dirty exists 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1013: left oid 46 (ObjNum 306 snap 101 seq_num 306) 2022-04-22T17:28:55.850 INFO:tasks.rados.rados.0.smithi115.stdout:1015: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1014: finishing write tid 1 to smithi11541303-15 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1014: finishing write tid 2 to smithi11541303-15 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1013: done (5 left) 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1015: done (4 left) 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1022: read oid 4 snap 90 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1022: expect (ObjNum 263 snap 88 seq_num 263) 2022-04-22T17:28:55.859 INFO:tasks.rados.rados.0.smithi115.stdout:1023: delete oid 40 current snap is 101 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:1014: finishing write tid 3 to smithi11541303-15 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 188 (ObjNum 307 snap 101 seq_num 307) dirty exists 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:1014: left oid 15 (ObjNum 307 snap 101 seq_num 307) 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:1016: finishing write tid 1 to smithi11541303-23 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:1016: finishing write tid 2 to smithi11541303-23 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:1016: finishing write tid 3 to smithi11541303-23 2022-04-22T17:28:55.860 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 420 (ObjNum 308 snap 101 seq_num 308) dirty exists 2022-04-22T17:28:55.861 INFO:tasks.rados.rados.0.smithi115.stdout:1016: left oid 23 (ObjNum 308 snap 101 seq_num 308) 2022-04-22T17:28:55.864 INFO:tasks.rados.rados.0.smithi115.stdout:1021: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:28:55.871 INFO:tasks.rados.rados.0.smithi115.stdout:1020: finishing write tid 1 to smithi11541303-45 2022-04-22T17:28:55.871 INFO:tasks.rados.rados.0.smithi115.stdout:1020: finishing write tid 2 to smithi11541303-45 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1014: done (5 left) 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1016: done (4 left) 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1021: done (3 left) 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1023: done (2 left) 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1024: read oid 13 snap -1 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1024: expect (ObjNum 298 snap 99 seq_num 298) 2022-04-22T17:28:55.873 INFO:tasks.rados.rados.0.smithi115.stdout:1025: snap_create 2022-04-22T17:28:55.943 INFO:tasks.rados.rados.0.smithi115.stdout:1024: expect (ObjNum 298 snap 99 seq_num 298) 2022-04-22T17:28:56.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:55 smithi121 bash[12679]: cephadm 2022-04-22T17:28:54.265336+0000 mgr.y (mgr.34107) 37 : cephadm [INF] Upgrade: Updating mon.b 2022-04-22T17:28:56.096 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:55 smithi121 bash[12679]: cephadm 2022-04-22T17:28:54.279127+0000 mgr.y (mgr.34107) 38 : cephadm [INF] Deploying daemon mon.b on smithi121 2022-04-22T17:28:56.096 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:55 smithi121 bash[12679]: cluster 2022-04-22T17:28:54.815727+0000 mon.a (mon.0) 133 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2022-04-22T17:28:56.631 INFO:tasks.rados.rados.0.smithi115.stdout:1022: expect (ObjNum 263 snap 88 seq_num 263) 2022-04-22T17:28:57.168 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:56 smithi121 bash[12679]: cluster 2022-04-22T17:28:55.129981+0000 mgr.y (mgr.34107) 39 : cluster [DBG] pgmap v63: 121 pgs: 121 active+clean; 245 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2022-04-22T17:28:57.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:56 smithi121 bash[12679]: cluster 2022-04-22T17:28:55.834419+0000 mon.a (mon.0) 134 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2022-04-22T17:28:57.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:57 smithi121 systemd[1]: Stopping Ceph mon.b for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:28:57.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:57 smithi121 bash[12679]: debug 2022-04-22T17:28:57.159+0000 7fea4f4fb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-04-22T17:28:57.323 INFO:tasks.rados.rados.0.smithi115.stdout:1020: finishing write tid 3 to smithi11541303-45 2022-04-22T17:28:57.323 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 517 (ObjNum 309 snap 101 seq_num 309) dirty exists 2022-04-22T17:28:57.323 INFO:tasks.rados.rados.0.smithi115.stdout:1020: left oid 45 (ObjNum 309 snap 101 seq_num 309) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1020: done (3 left) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1022: done (2 left) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1024: done (1 left) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1025: done (0 left) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1026: read oid 23 snap -1 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1026: expect (ObjNum 308 snap 101 seq_num 308) 2022-04-22T17:28:57.324 INFO:tasks.rados.rados.0.smithi115.stdout:1027: snap_create 2022-04-22T17:28:57.354 INFO:tasks.rados.rados.0.smithi115.stdout:1026: expect (ObjNum 308 snap 101 seq_num 308) 2022-04-22T17:28:57.492 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:57 smithi121 bash[12679]: debug 2022-04-22T17:28:57.159+0000 7fea4f4fb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2022-04-22T17:28:58.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 bash[36779]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.b 2022-04-22T17:28:58.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 bash[36821]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon-b 2022-04-22T17:28:58.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 bash[36834]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mon.b 2022-04-22T17:28:58.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mon.b.service: Succeeded. 2022-04-22T17:28:58.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 systemd[1]: Stopped Ceph mon.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:28:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:58 smithi121 systemd[1]: Started Ceph mon.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:29:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.882+0000 7f81fa7e7880 0 set uid:gid to 167:167 (ceph:ceph) 2022-04-22T17:29:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.882+0000 7f81fa7e7880 0 ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev), process ceph-mon, pid 7 2022-04-22T17:29:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.882+0000 7f81fa7e7880 0 pidfile_write: ignore empty --pid-file 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 0 load: jerasure load: lrc 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: RocksDB version: 6.15.5 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Compile date Apr 22 2022 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: DB SUMMARY 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: DB Session ID: BM14QEQSV5Z6XMUIAJWY 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: CURRENT file: CURRENT 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: IDENTITY file: IDENTITY 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2022-04-22T17:29:00.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000020.sst 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000018.log size: 18268358 ; 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.error_if_exists: 0 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.create_if_missing: 0 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.paranoid_checks: 1 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.env: 0x55dd8fee0860 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.fs: Posix File System 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.info_log: 0x55dd91617e00 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_file_opening_threads: 16 2022-04-22T17:29:00.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.statistics: (nil) 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.use_fsync: 0 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_log_file_size: 0 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.log_file_time_to_roll: 0 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.keep_log_file_num: 1000 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.recycle_log_file_num: 0 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_fallocate: 1 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_mmap_reads: 0 2022-04-22T17:29:00.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_mmap_writes: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.use_direct_reads: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.create_missing_column_families: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.db_log_dir: 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.table_cache_numshardbits: 6 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-04-22T17:29:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.advise_random_on_open: 1 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.db_write_buffer_size: 0 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.write_buffer_manager: 0x55dd916de420 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.use_adaptive_mutex: 0 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.rate_limiter: (nil) 2022-04-22T17:29:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-04-22T17:29:00.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.wal_recovery_mode: 2 2022-04-22T17:29:00.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.enable_thread_tracking: 0 2022-04-22T17:29:00.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.enable_pipelined_write: 0 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.unordered_write: 0 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.row_cache: None 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.wal_filter: None 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-04-22T17:29:00.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_ingest_behind: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.preserve_deletes: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.two_write_queues: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.manual_wal_flush: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.atomic_flush: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.persist_stats_to_disk: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.log_readahead_size: 0 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2022-04-22T17:29:00.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.best_efforts_recovery: 0 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.allow_data_in_errors: 0 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.db_host_id: __hostname__ 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_background_jobs: 2 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_background_compactions: -1 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_subcompactions: 1 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-04-22T17:29:00.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.delayed_write_rate : 16777216 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_total_wal_size: 0 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.stats_dump_period_sec: 600 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.stats_persist_period_sec: 600 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_open_files: -1 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bytes_per_sync: 0 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-04-22T17:29:00.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_readahead_size: 0 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_background_flushes: -1 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Compression algorithms supported: 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kZSTD supported: 0 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kXpressCompression supported: 0 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kLZ4HCCompression supported: 1 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kLZ4Compression supported: 1 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kBZip2Compression supported: 0 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kZlibCompression supported: 1 2022-04-22T17:29:00.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: kSnappyCompression supported: 1 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.merge_operator: 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_filter: None 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_filter_factory: None 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.sst_partitioner_factory: None 2022-04-22T17:29:00.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.table_factory: BlockBasedTable 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dd915e5cb0) 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: cache_index_and_filter_blocks: 1 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: cache_index_and_filter_blocks_with_high_priority: 0 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: pin_top_level_index_and_filter: 1 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: index_type: 0 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: data_block_index_type: 0 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: index_shortening: 1 2022-04-22T17:29:00.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: data_block_hash_table_util_ratio: 0.750000 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: hash_index_allow_collision: 1 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: checksum: 1 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: no_block_cache: 0 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_cache: 0x55dd9164c170 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_cache_name: BinnedLRUCache 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_cache_options: 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: capacity : 536870912 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: num_shard_bits : 4 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: strict_capacity_limit : 0 2022-04-22T17:29:00.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: high_pri_pool_ratio: 0.000 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_cache_compressed: (nil) 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: persistent_cache: (nil) 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_size: 4096 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_size_deviation: 10 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_restart_interval: 16 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: index_block_restart_interval: 1 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: metadata_block_size: 4096 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: partition_filters: 0 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: use_delta_encoding: 1 2022-04-22T17:29:00.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: filter_policy: rocksdb.BuiltinBloomFilter 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: whole_key_filtering: 1 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: verify_compression: 0 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: read_amp_bytes_per_bit: 0 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: format_version: 4 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: enable_index_compression: 1 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: block_align: 0 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.write_buffer_size: 33554432 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_write_buffer_number: 2 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression: NoCompression 2022-04-22T17:29:00.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression: Disabled 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.prefix_extractor: nullptr 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.num_levels: 7 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-04-22T17:29:00.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.window_bits: -14 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.level: 32767 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.strategy: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-04-22T17:29:00.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compression_opts.enabled: false 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.target_file_size_base: 67108864 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.target_file_size_multiplier: 1 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-04-22T17:29:00.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.arena_block_size: 4194304 2022-04-22T17:29:00.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.disable_auto_compactions: 0 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-04-22T17:29:00.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.table_properties_collectors: 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.inplace_update_support: 0 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-04-22T17:29:00.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.memtable_huge_page_size: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.bloom_locality: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.max_successive_merges: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.paranoid_file_checks: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.force_consistency_checks: 1 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.report_bg_io_stats: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.ttl: 2592000 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.enable_blob_files: false 2022-04-22T17:29:00.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.min_blob_size: 0 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.blob_file_size: 268435456 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.blob_compression_type: NoCompression 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.enable_blob_garbage_collection: false 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12232, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 18 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 22 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.886+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:29:00.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.890+0000 7f81fa7e7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648539894696, "job": 1, "event": "recovery_started", "wal_files": [18]} 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.890+0000 7f81fa7e7880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #18 mode 2 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.910+0000 7f81fa7e7880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.926+0000 7f81fa7e7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648539934179, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 11717018, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 11698372, "index_size": 14567, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3141, "raw_key_size": 25619, "raw_average_key_size": 20, "raw_value_size": 11666835, "raw_average_value_size": 9492, "num_data_blocks": 623, "num_entries": 1229, "num_deletions": 307, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1650648539, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3ccab8c4-cea5-4776-9478-b7d6899d342a", "db_session_id": "BM14QEQSV5Z6XMUIAJWY"}} 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.926+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 24 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.930+0000 7f81fa7e7880 4 rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.930+0000 7f81fa7e7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1650648539936465, "job": 1, "event": "recovery_finished"} 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.930+0000 7f81fa7e7880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-04-22T17:29:00.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55dd91632a80 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 4 rocksdb: DB pointer 0x55dd91643800 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81ea565700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81ea565700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** DB Stats ** 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:29:00.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** Compaction Stats [default] ** 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: L0 1/0 11.17 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Sum 2/0 33.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** Compaction Stats [default] ** 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Flush(GB): cumulative 0.011, interval 0.011 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:29:00.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Cumulative compaction: 0.01 GB write, 242.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Interval compaction: 0.01 GB write, 242.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** Compaction Stats [default] ** 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: L0 1/0 11.17 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: L6 1/0 22.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Sum 2/0 33.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** Compaction Stats [default] ** 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 592.7 0.02 0.00 1 0.019 0 0 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Uptime(secs): 0.0 total, 0.0 interval 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Flush(GB): cumulative 0.011, interval 0.000 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(GB): cumulative 0.000, interval 0.000 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(Total Files): cumulative 0, interval 0 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(L0 Files): cumulative 0, interval 0 2022-04-22T17:29:00.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: AddFile(Keys): cumulative 0, interval 0 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Cumulative compaction: 0.01 GB write, 242.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: ** File Read Latency Histogram By Level [default] ** 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 starting mon.b rank 2 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 1 mon.b@-1(???) e3 preinit fsid b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).mds e6 new map 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).mds e6 print_map 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: e6 2022-04-22T17:29:00.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: enable_multiple, ever_enabled_multiple: 1,1 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: legacy client fscid: 1 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Filesystem 'cephfs' (1) 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: fs_name cephfs 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: epoch 6 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: flags 12 joinable allow_snaps allow_multimds_snaps 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: created 2022-04-22T17:22:18.413732+0000 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: modified 2022-04-22T17:22:20.428010+0000 2022-04-22T17:29:00.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: tableserver 0 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: root 0 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: session_timeout 60 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: session_autoclose 300 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: max_file_size 1099511627776 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: required_client_features {} 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: last_failure 0 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: last_failure_osd_epoch 0 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: max_mds 1 2022-04-22T17:29:00.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: in 0 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: up {0=24386} 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: failed 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: damaged 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: stopped 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: data_pools [3] 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: metadata_pool 2 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: inline_data disabled 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: balancer 2022-04-22T17:29:00.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: standby_count_wanted 0 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: [mds.b{0:24386} state up:active seq 3 addr [v2:172.21.15.121:6832/2211373279,v1:172.21.15.121:6833/2211373279] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: Standby daemons: 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: [mds.a{-1:24385} state up:standby seq 2 addr [v2:172.21.15.115:6834/3475839169,v1:172.21.15.115:6835/3475839169] compat {c=[1],r=[1],i=[7ff]}] 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).osd e258 crush map has features 3314933069573799936, adjusting msgr requires 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).osd e258 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).osd e258 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:29:00.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 0 mon.b@-1(???).osd e258 crush map has features 432629308056666112, adjusting msgr requires 2022-04-22T17:29:00.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:28:59 smithi121 bash[36902]: debug 2022-04-22T17:28:59.934+0000 7f81fa7e7880 1 mon.b@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2022-04-22T17:29:02.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:01.617Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:01.617Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:01.617Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.502Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.502Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.502Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:03.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:06.109 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:29:06.453 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:29:06.453 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5m) 23s ago 5m 12.9M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:29:06.453 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5m) 47s ago 5m 23.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:29:06.454 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6m) 23s ago 6m 13.2M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:29:06.454 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6m) 47s ago 6m 16.5M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:29:06.454 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (85s) 47s ago 11m 402M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:29:06.454 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2m) 23s ago 13m 438M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (63s) 23s ago 13m 37.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 starting - - - 2048M 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (30s) 23s ago 11m 25.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 23s ago 6m 9039k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6m) 47s ago 6m 9111k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 23s ago 10m 151M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 23s ago 10m 161M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:29:06.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 23s ago 9m 129M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 23s ago 9m 90.3M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (8m) 47s ago 8m 118M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (8m) 47s ago 8m 131M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (7m) 47s ago 8m 110M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (7m) 47s ago 7m 114M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:29:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 47s ago 6m 32.5M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:29:10.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:29:09 smithi115 bash[42914]: debug 2022-04-22T17:29:09.937+0000 7f8eea62c700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-04-22T17:29:11.753 INFO:tasks.rados.rados.0.smithi115.stdout:1026: done (1 left) 2022-04-22T17:29:11.753 INFO:tasks.rados.rados.0.smithi115.stdout:1027: done (0 left) 2022-04-22T17:29:11.754 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 103 2022-04-22T17:29:11.754 INFO:tasks.rados.rados.0.smithi115.stdout:1028: seq_num 311 ranges {9330688=417792} 2022-04-22T17:29:11.812 INFO:tasks.rados.rados.0.smithi115.stdout:1028: writing smithi11541303-13 from 9330688 to 9748480 tid 1 2022-04-22T17:29:11.813 INFO:tasks.rados.rados.0.smithi115.stdout:1029: snap_create 2022-04-22T17:29:11.824 INFO:tasks.rados.rados.0.smithi115.stdout:1028: finishing write tid 1 to smithi11541303-13 2022-04-22T17:29:11.824 INFO:tasks.rados.rados.0.smithi115.stdout:1028: finishing write tid 2 to smithi11541303-13 2022-04-22T17:29:11.826 INFO:tasks.rados.rados.0.smithi115.stdout:1028: finishing write tid 3 to smithi11541303-13 2022-04-22T17:29:11.826 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 391 (ObjNum 311 snap 103 seq_num 311) dirty exists 2022-04-22T17:29:11.826 INFO:tasks.rados.rados.0.smithi115.stdout:1028: left oid 13 (ObjNum 311 snap 103 seq_num 311) 2022-04-22T17:29:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:15:33.310343+0000 mon.a (mon.0) 0 : [INF] mkfs b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:29:12.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.117225+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.117434+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.117645+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:29:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.117891+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.118384+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.187411+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.191434+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.194435+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.196645+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.222264+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:12.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.233152+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:12.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:13.583710+0000 mgr.y (mgr.34107) 1 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:29:12.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:13.710907+0000 mgr.y (mgr.34107) 2 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:13.711621+0000 mgr.y (mgr.34107) 3 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:29:12.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:13.719771+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:18.450121+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:18.844925+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:19.082315+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:19.084862+0000 mgr.y (mgr.34107) 7 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:19.136160+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:19.327028+0000 mgr.y (mgr.34107) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:19.430402+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:19.561515+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:24.139542+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:25.118379+0000 mgr.y (mgr.34107) 12 : cluster [DBG] pgmap v21: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:25.757818+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:26.761893+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:27.119136+0000 mgr.y (mgr.34107) 13 : cluster [DBG] pgmap v24: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:27.777079+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:27.821615+0000 mgr.y (mgr.34107) 14 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:27.832434+0000 mon.a (mon.0) 74 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:27.834316+0000 mgr.y (mgr.34107) 15 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:28.084488+0000 mgr.y (mgr.34107) 16 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.347281+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.356483+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.364195+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.367543+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.369355+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:28.370809+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:28.371906+0000 mgr.y (mgr.34107) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:28.837405+0000 mon.a (mon.0) 81 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:29.119942+0000 mgr.y (mgr.34107) 18 : cluster [DBG] pgmap v27: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:29.142987+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:29.846914+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:30.855490+0000 mon.a (mon.0) 84 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:31.120698+0000 mgr.y (mgr.34107) 19 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 190 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 28 op/s 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:31.296655+0000 mgr.y (mgr.34107) 20 : cephadm [INF] Upgrade: Updating mon.c 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:31.303891+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:31.305450+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:31.306744+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:31.308009+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:31.309467+0000 mgr.y (mgr.34107) 21 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:31.869824+0000 mon.a (mon.0) 89 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2022-04-22T17:29:12.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:31.978163+0000 mgr.y (mgr.34107) 22 : audit [DBG] from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:32.315445+0000 mgr.y (mgr.34107) 23 : audit [DBG] from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:32.720113+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? 172.21.15.115:0/181693675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:32.868608+0000 mon.a (mon.0) 91 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:35.122983+0000 mgr.y (mgr.34107) 25 : cluster [DBG] pgmap v35: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 524 KiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.123827+0000 mgr.y (mgr.34107) 26 : cluster [DBG] pgmap v36: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.245647+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.730793+0000 mon.a (mon.0) 94 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.753258+0000 mon.a (mon.0) 95 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.767154+0000 mon.a (mon.0) 96 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.768572+0000 mon.a (mon.0) 97 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.768653+0000 mon.a (mon.0) 98 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2022-04-22T17:29:12.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.769833+0000 mon.a (mon.0) 99 : cluster [DBG] mgrmap e29: y(active, since 24s), standbys: x 2022-04-22T17:29:12.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:37.776592+0000 mon.a (mon.0) 100 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:37.782516+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:37.784165+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:37.793877+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:37.796543+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:38.788840+0000 mon.a (mon.0) 105 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:39.124554+0000 mgr.y (mgr.34107) 27 : cluster [DBG] pgmap v39: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:39.797273+0000 mon.a (mon.0) 106 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:40.810003+0000 mon.a (mon.0) 107 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:41.125017+0000 mgr.y (mgr.34107) 28 : cluster [DBG] pgmap v42: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:41.815343+0000 mon.a (mon.0) 108 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:42.530600+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:43.051015+0000 mon.a (mon.0) 110 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:43.125765+0000 mgr.y (mgr.34107) 29 : cluster [DBG] pgmap v45: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:43.134927+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:44.057851+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:44.722748+0000 mon.a (mon.0) 113 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:45.126474+0000 mgr.y (mgr.34107) 30 : cluster [DBG] pgmap v48: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:45.732625+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:46.740735+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2022-04-22T17:29:12.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:47.127211+0000 mgr.y (mgr.34107) 31 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:47.752168+0000 mon.a (mon.0) 116 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:48.759403+0000 mon.a (mon.0) 117 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:49.127686+0000 mgr.y (mgr.34107) 32 : cluster [DBG] pgmap v54: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:49.727308+0000 mon.a (mon.0) 118 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:50.782260+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:51.128359+0000 mgr.y (mgr.34107) 33 : cluster [DBG] pgmap v57: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:51.566824+0000 mgr.y (mgr.34107) 34 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.576741+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.582903+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.592040+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.594864+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.596369+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:51.597474+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:51.598239+0000 mgr.y (mgr.34107) 35 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:51.794355+0000 mon.a (mon.0) 126 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:52.802734+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:53.129141+0000 mgr.y (mgr.34107) 36 : cluster [DBG] pgmap v60: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:53.811619+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2022-04-22T17:29:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:54.265336+0000 mgr.y (mgr.34107) 37 : cephadm [INF] Upgrade: Updating mon.b 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:54.273786+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:54.275437+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:54.276653+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:28:54.277867+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cephadm 2022-04-22T17:28:54.279127+0000 mgr.y (mgr.34107) 38 : cephadm [INF] Deploying daemon mon.b on smithi121 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:54.815727+0000 mon.a (mon.0) 133 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:55.129981+0000 mgr.y (mgr.34107) 39 : cluster [DBG] pgmap v63: 121 pgs: 121 active+clean; 245 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:55.834419+0000 mon.a (mon.0) 134 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:28:56.840665+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:00.556567+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:06.078425+0000 mgr.y (mgr.34107) 45 : audit [DBG] from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:06.427928+0000 mgr.y (mgr.34107) 46 : audit [DBG] from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.617732+0000 mon.a (mon.0) 154 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.618061+0000 mon.a (mon.0) 155 : cluster [DBG] mgrmap e29: y(active, since 53s), standbys: x 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.618241+0000 mon.a (mon.0) 156 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.618276+0000 mon.a (mon.0) 157 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.643890+0000 mon.a (mon.0) 158 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.643914+0000 mon.a (mon.0) 159 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.643931+0000 mon.a (mon.0) 160 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:06.644220+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.644810+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.646489+0000 mon.c (mon.1) 4 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:06.646613+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:06.646862+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:06.654145+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:07.136069+0000 mgr.y (mgr.34107) 47 : cluster [DBG] pgmap v71: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 36 KiB/s wr, 2 op/s 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:09.136861+0000 mgr.y (mgr.34107) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.646482+0000 mon.a (mon.0) 165 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,2) 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.649569+0000 mon.a (mon.0) 166 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.649620+0000 mon.a (mon.0) 167 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.649643+0000 mon.a (mon.0) 168 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.649731+0000 mon.a (mon.0) 169 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.671904+0000 mon.a (mon.0) 170 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.678336+0000 mon.a (mon.0) 171 : cluster [DBG] monmap e4: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.679425+0000 mon.a (mon.0) 172 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.679557+0000 mon.a (mon.0) 173 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.680028+0000 mon.a (mon.0) 174 : cluster [DBG] mgrmap e29: y(active, since 58s), standbys: x 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.680245+0000 mon.a (mon.0) 175 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.680274+0000 mon.a (mon.0) 176 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.699447+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.037 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:11 smithi121 bash[36902]: audit 2022-04-22T17:29:11.703375+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:11.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:11.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:11.618Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:12.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:15:33.310343+0000 mon.a (mon.0) 0 : [INF] mkfs b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:29:12.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:29:12.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:29:12.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:29:12.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:29:12.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:29:12.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:15:33.310343+0000 mon.a (mon.0) 0 : [INF] mkfs b6061a0a-c25f-11ec-8c39-001a4aab830c 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:19.788662+0000 mgr.x (mgr.24749) 2 : cephadm [INF] Deploying cephadm binary to smithi121 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:20.270034+0000 mgr.x (mgr.24749) 4 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTING 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:20.436259+0000 mgr.x (mgr.24749) 5 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Serving on https://172.21.15.121:7150 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:20.436623+0000 mgr.x (mgr.24749) 6 : cephadm [INF] [22/Apr/2022:17:26:20] ENGINE Bus STARTED 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:34.901371+0000 mgr.x (mgr.24749) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:34.923687+0000 mgr.x (mgr.24749) 17 : cephadm [INF] Adjusting osd_memory_target on smithi115 to 2526M 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:34.931635+0000 mgr.x (mgr.24749) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.165020+0000 mgr.x (mgr.24749) 19 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.261788+0000 mgr.x (mgr.24749) 21 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.287398+0000 mgr.x (mgr.24749) 22 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 2526M 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.299887+0000 mgr.x (mgr.24749) 23 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.521178+0000 mgr.x (mgr.24749) 24 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.842787+0000 mgr.x (mgr.24749) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:35.862420+0000 mgr.x (mgr.24749) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi121 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:55.112124+0000 mgr.x (mgr.24749) 38 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:58.054901+0000 mgr.x (mgr.24749) 41 : cephadm [INF] Upgrade: Updating mgr.y 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:26:58.071759+0000 mgr.x (mgr.24749) 42 : cephadm [INF] Deploying daemon mgr.y on smithi115 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:17.988348+0000 mgr.x (mgr.24749) 55 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:18.027291+0000 mgr.x (mgr.24749) 56 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:18.030461+0000 mgr.x (mgr.24749) 57 : cephadm [INF] Failing over to other MGR 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:19.952157+0000 mgr.y (mgr.24919) 1 : cephadm [INF] [22/Apr/2022:17:27:19] ENGINE Bus STARTING 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:20.089191+0000 mgr.y (mgr.24919) 2 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:20.089497+0000 mgr.y (mgr.24919) 3 : cephadm [INF] [22/Apr/2022:17:27:20] ENGINE Bus STARTED 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:25.431654+0000 mgr.y (mgr.24919) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:29:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.117225+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.117434+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.117645+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.117891+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.118384+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.187411+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.191434+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.194435+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.196645+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.222264+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:25.681904+0000 mgr.y (mgr.24919) 9 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:33.772821+0000 mgr.y (mgr.24919) 16 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:33.783724+0000 mgr.y (mgr.24919) 17 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:34.029605+0000 mgr.y (mgr.24919) 18 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:36.886715+0000 mgr.y (mgr.24919) 20 : cephadm [INF] Upgrade: Updating mgr.x 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:36.893287+0000 mgr.y (mgr.24919) 21 : cephadm [INF] Deploying daemon mgr.x on smithi121 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:55.928582+0000 mgr.y (mgr.24919) 32 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:55.959427+0000 mgr.y (mgr.24919) 33 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:55.980554+0000 mgr.y (mgr.24919) 34 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:58.834608+0000 mgr.y (mgr.24919) 36 : cephadm [INF] Upgrade: Updating mon.a 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:27:58.847985+0000 mgr.y (mgr.24919) 37 : cephadm [INF] Deploying daemon mon.a on smithi115 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.117225+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.117434+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.117645+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.117891+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.118384+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.187411+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.191434+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.194435+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.196645+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.222264+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:12.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.233152+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:13.583710+0000 mgr.y (mgr.34107) 1 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:13.710907+0000 mgr.y (mgr.34107) 2 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:13.711621+0000 mgr.y (mgr.34107) 3 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:13.719771+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:18.450121+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:18.844925+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:19.082315+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:19.084862+0000 mgr.y (mgr.34107) 7 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:19.136160+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:19.327028+0000 mgr.y (mgr.34107) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:19.430402+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:19.561515+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:24.139542+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:25.118379+0000 mgr.y (mgr.34107) 12 : cluster [DBG] pgmap v21: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:25.757818+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:26.761893+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:27.119136+0000 mgr.y (mgr.34107) 13 : cluster [DBG] pgmap v24: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:27.777079+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:27.821615+0000 mgr.y (mgr.34107) 14 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:27.832434+0000 mon.a (mon.0) 74 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:27.834316+0000 mgr.y (mgr.34107) 15 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:28.084488+0000 mgr.y (mgr.34107) 16 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.347281+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.233152+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:12.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:13.583710+0000 mgr.y (mgr.34107) 1 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTING 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:13.710907+0000 mgr.y (mgr.34107) 2 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Serving on https://172.21.15.115:7150 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:13.711621+0000 mgr.y (mgr.34107) 3 : cephadm [INF] [22/Apr/2022:17:28:13] ENGINE Bus STARTED 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:13.719771+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:18.450121+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:18.844925+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:19.082315+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:19.084862+0000 mgr.y (mgr.34107) 7 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.conf 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:19.136160+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:19.327028+0000 mgr.y (mgr.34107) 9 : cephadm [INF] Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:19.430402+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:19.561515+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:24.139542+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:25.118379+0000 mgr.y (mgr.34107) 12 : cluster [DBG] pgmap v21: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:25.757818+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:26.761893+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:27.119136+0000 mgr.y (mgr.34107) 13 : cluster [DBG] pgmap v24: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:27.777079+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:27.821615+0000 mgr.y (mgr.34107) 14 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:27.832434+0000 mon.a (mon.0) 74 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:27.834316+0000 mgr.y (mgr.34107) 15 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:28.084488+0000 mgr.y (mgr.34107) 16 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.347281+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.356483+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.364195+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.367543+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.369355+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:28.370809+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:28.371906+0000 mgr.y (mgr.34107) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:28.837405+0000 mon.a (mon.0) 81 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:29.119942+0000 mgr.y (mgr.34107) 18 : cluster [DBG] pgmap v27: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:29.142987+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:29.846914+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:30.855490+0000 mon.a (mon.0) 84 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:31.120698+0000 mgr.y (mgr.34107) 19 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 190 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 28 op/s 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:31.296655+0000 mgr.y (mgr.34107) 20 : cephadm [INF] Upgrade: Updating mon.c 2022-04-22T17:29:12.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:31.303891+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:31.305450+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:31.306744+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:31.308009+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:31.309467+0000 mgr.y (mgr.34107) 21 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:31.869824+0000 mon.a (mon.0) 89 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:31.978163+0000 mgr.y (mgr.34107) 22 : audit [DBG] from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:32.315445+0000 mgr.y (mgr.34107) 23 : audit [DBG] from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.070 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.356483+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.364195+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.367543+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.369355+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:28.370809+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:28.371906+0000 mgr.y (mgr.34107) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:28.837405+0000 mon.a (mon.0) 81 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:29.119942+0000 mgr.y (mgr.34107) 18 : cluster [DBG] pgmap v27: 121 pgs: 121 active+clean; 168 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:29.142987+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:29.846914+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2022-04-22T17:29:12.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:30.855490+0000 mon.a (mon.0) 84 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:31.120698+0000 mgr.y (mgr.34107) 19 : cluster [DBG] pgmap v30: 121 pgs: 121 active+clean; 190 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 28 op/s 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:31.296655+0000 mgr.y (mgr.34107) 20 : cephadm [INF] Upgrade: Updating mon.c 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:31.303891+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:31.305450+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:31.306744+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:31.308009+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:31.309467+0000 mgr.y (mgr.34107) 21 : cephadm [INF] Deploying daemon mon.c on smithi115 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:31.869824+0000 mon.a (mon.0) 89 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:31.978163+0000 mgr.y (mgr.34107) 22 : audit [DBG] from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:32.315445+0000 mgr.y (mgr.34107) 23 : audit [DBG] from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:32.720113+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? 172.21.15.115:0/181693675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:32.868608+0000 mon.a (mon.0) 91 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2022-04-22T17:29:12.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:35.122983+0000 mgr.y (mgr.34107) 25 : cluster [DBG] pgmap v35: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 524 KiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.123827+0000 mgr.y (mgr.34107) 26 : cluster [DBG] pgmap v36: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.245647+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.730793+0000 mon.a (mon.0) 94 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.753258+0000 mon.a (mon.0) 95 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.767154+0000 mon.a (mon.0) 96 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.768572+0000 mon.a (mon.0) 97 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.768653+0000 mon.a (mon.0) 98 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.769833+0000 mon.a (mon.0) 99 : cluster [DBG] mgrmap e29: y(active, since 24s), standbys: x 2022-04-22T17:29:12.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:37.776592+0000 mon.a (mon.0) 100 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:37.782516+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:37.784165+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:37.793877+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:37.796543+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:38.788840+0000 mon.a (mon.0) 105 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:39.124554+0000 mgr.y (mgr.34107) 27 : cluster [DBG] pgmap v39: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:39.797273+0000 mon.a (mon.0) 106 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:40.810003+0000 mon.a (mon.0) 107 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:41.125017+0000 mgr.y (mgr.34107) 28 : cluster [DBG] pgmap v42: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:41.815343+0000 mon.a (mon.0) 108 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:32.720113+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? 172.21.15.115:0/181693675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:32.868608+0000 mon.a (mon.0) 91 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:35.122983+0000 mgr.y (mgr.34107) 25 : cluster [DBG] pgmap v35: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 524 KiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.123827+0000 mgr.y (mgr.34107) 26 : cluster [DBG] pgmap v36: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.245647+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.730793+0000 mon.a (mon.0) 94 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.753258+0000 mon.a (mon.0) 95 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.767154+0000 mon.a (mon.0) 96 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.768572+0000 mon.a (mon.0) 97 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.768653+0000 mon.a (mon.0) 98 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.769833+0000 mon.a (mon.0) 99 : cluster [DBG] mgrmap e29: y(active, since 24s), standbys: x 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:37.776592+0000 mon.a (mon.0) 100 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:37.782516+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:37.784165+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:37.793877+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:37.796543+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:38.788840+0000 mon.a (mon.0) 105 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2022-04-22T17:29:12.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:39.124554+0000 mgr.y (mgr.34107) 27 : cluster [DBG] pgmap v39: 121 pgs: 121 active+clean; 201 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 373 KiB/s rd, 1.9 MiB/s wr, 8 op/s 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:39.797273+0000 mon.a (mon.0) 106 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:40.810003+0000 mon.a (mon.0) 107 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:41.125017+0000 mgr.y (mgr.34107) 28 : cluster [DBG] pgmap v42: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:41.815343+0000 mon.a (mon.0) 108 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:42.530600+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:43.051015+0000 mon.a (mon.0) 110 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2022-04-22T17:29:12.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:43.125765+0000 mgr.y (mgr.34107) 29 : cluster [DBG] pgmap v45: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:43.134927+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:44.057851+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:44.722748+0000 mon.a (mon.0) 113 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:45.126474+0000 mgr.y (mgr.34107) 30 : cluster [DBG] pgmap v48: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:45.732625+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:46.740735+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:47.127211+0000 mgr.y (mgr.34107) 31 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:47.752168+0000 mon.a (mon.0) 116 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:48.759403+0000 mon.a (mon.0) 117 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2022-04-22T17:29:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:49.127686+0000 mgr.y (mgr.34107) 32 : cluster [DBG] pgmap v54: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:49.727308+0000 mon.a (mon.0) 118 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:50.782260+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:51.128359+0000 mgr.y (mgr.34107) 33 : cluster [DBG] pgmap v57: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:51.566824+0000 mgr.y (mgr.34107) 34 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.576741+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.582903+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.592040+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.594864+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.596369+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:42.530600+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:43.051015+0000 mon.a (mon.0) 110 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:43.125765+0000 mgr.y (mgr.34107) 29 : cluster [DBG] pgmap v45: 121 pgs: 121 active+clean; 202 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 558 KiB/s rd, 942 KiB/s wr, 1 op/s 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:43.134927+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:44.057851+0000 mon.a (mon.0) 112 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:44.722748+0000 mon.a (mon.0) 113 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:45.126474+0000 mgr.y (mgr.34107) 30 : cluster [DBG] pgmap v48: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:45.732625+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2022-04-22T17:29:12.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:46.740735+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:47.127211+0000 mgr.y (mgr.34107) 31 : cluster [DBG] pgmap v51: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 7.3 MiB/s wr, 28 op/s 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:47.752168+0000 mon.a (mon.0) 116 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:48.759403+0000 mon.a (mon.0) 117 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:49.127686+0000 mgr.y (mgr.34107) 32 : cluster [DBG] pgmap v54: 121 pgs: 121 active+clean; 229 MiB data, 835 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:49.727308+0000 mon.a (mon.0) 118 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:50.782260+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:51.128359+0000 mgr.y (mgr.34107) 33 : cluster [DBG] pgmap v57: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:51.566824+0000 mgr.y (mgr.34107) 34 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.576741+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.582903+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.083 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.592040+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:12.083 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.594864+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:12.083 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.596369+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:51.597474+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:51.598239+0000 mgr.y (mgr.34107) 35 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:51.794355+0000 mon.a (mon.0) 126 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:52.802734+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:53.129141+0000 mgr.y (mgr.34107) 36 : cluster [DBG] pgmap v60: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:53.811619+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2022-04-22T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:54.265336+0000 mgr.y (mgr.34107) 37 : cephadm [INF] Upgrade: Updating mon.b 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:54.273786+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:54.275437+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:54.276653+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:28:54.277867+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cephadm 2022-04-22T17:28:54.279127+0000 mgr.y (mgr.34107) 38 : cephadm [INF] Deploying daemon mon.b on smithi121 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:54.815727+0000 mon.a (mon.0) 133 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:55.129981+0000 mgr.y (mgr.34107) 39 : cluster [DBG] pgmap v63: 121 pgs: 121 active+clean; 245 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:55.834419+0000 mon.a (mon.0) 134 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:28:56.840665+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:00.556567+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:06.078425+0000 mgr.y (mgr.34107) 45 : audit [DBG] from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:06.427928+0000 mgr.y (mgr.34107) 46 : audit [DBG] from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.617732+0000 mon.a (mon.0) 154 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.618061+0000 mon.a (mon.0) 155 : cluster [DBG] mgrmap e29: y(active, since 53s), standbys: x 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.618241+0000 mon.a (mon.0) 156 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.618276+0000 mon.a (mon.0) 157 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.643890+0000 mon.a (mon.0) 158 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.643914+0000 mon.a (mon.0) 159 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.643931+0000 mon.a (mon.0) 160 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:06.644220+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.644810+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.646489+0000 mon.c (mon.1) 4 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:06.646613+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:06.646862+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:06.654145+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:07.136069+0000 mgr.y (mgr.34107) 47 : cluster [DBG] pgmap v71: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 36 KiB/s wr, 2 op/s 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:09.136861+0000 mgr.y (mgr.34107) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.646482+0000 mon.a (mon.0) 165 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,2) 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.649569+0000 mon.a (mon.0) 166 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.649620+0000 mon.a (mon.0) 167 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.649643+0000 mon.a (mon.0) 168 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.649731+0000 mon.a (mon.0) 169 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.671904+0000 mon.a (mon.0) 170 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.678336+0000 mon.a (mon.0) 171 : cluster [DBG] monmap e4: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.679425+0000 mon.a (mon.0) 172 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.679557+0000 mon.a (mon.0) 173 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.680028+0000 mon.a (mon.0) 174 : cluster [DBG] mgrmap e29: y(active, since 58s), standbys: x 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.680245+0000 mon.a (mon.0) 175 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.680274+0000 mon.a (mon.0) 176 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.699447+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[46934]: audit 2022-04-22T17:29:11.703375+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:51.597474+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:51.598239+0000 mgr.y (mgr.34107) 35 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:51.794355+0000 mon.a (mon.0) 126 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:52.802734+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:53.129141+0000 mgr.y (mgr.34107) 36 : cluster [DBG] pgmap v60: 121 pgs: 121 active+clean; 244 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:12.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:53.811619+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:54.265336+0000 mgr.y (mgr.34107) 37 : cephadm [INF] Upgrade: Updating mon.b 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:54.273786+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:54.275437+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:54.276653+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:28:54.277867+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cephadm 2022-04-22T17:28:54.279127+0000 mgr.y (mgr.34107) 38 : cephadm [INF] Deploying daemon mon.b on smithi121 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:54.815727+0000 mon.a (mon.0) 133 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:55.129981+0000 mgr.y (mgr.34107) 39 : cluster [DBG] pgmap v63: 121 pgs: 121 active+clean; 245 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:55.834419+0000 mon.a (mon.0) 134 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2022-04-22T17:29:12.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:28:56.840665+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:00.556567+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:06.078425+0000 mgr.y (mgr.34107) 45 : audit [DBG] from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:06.427928+0000 mgr.y (mgr.34107) 46 : audit [DBG] from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.617732+0000 mon.a (mon.0) 154 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.618061+0000 mon.a (mon.0) 155 : cluster [DBG] mgrmap e29: y(active, since 53s), standbys: x 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.618241+0000 mon.a (mon.0) 156 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.618276+0000 mon.a (mon.0) 157 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.643890+0000 mon.a (mon.0) 158 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.643914+0000 mon.a (mon.0) 159 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.643931+0000 mon.a (mon.0) 160 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:06.644220+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.644810+0000 mon.a (mon.0) 162 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.646489+0000 mon.c (mon.1) 4 : cluster [INF] mon.c calling monitor election 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:06.646613+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:06.646862+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:06.654145+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2022-04-22T17:29:12.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:07.136069+0000 mgr.y (mgr.34107) 47 : cluster [DBG] pgmap v71: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 36 KiB/s wr, 2 op/s 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:09.136861+0000 mgr.y (mgr.34107) 48 : cluster [DBG] pgmap v72: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.646482+0000 mon.a (mon.0) 165 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,2) 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.649569+0000 mon.a (mon.0) 166 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.649620+0000 mon.a (mon.0) 167 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.649643+0000 mon.a (mon.0) 168 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.649731+0000 mon.a (mon.0) 169 : cluster [INF] mon.a calling monitor election 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.671904+0000 mon.a (mon.0) 170 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.678336+0000 mon.a (mon.0) 171 : cluster [DBG] monmap e4: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2022-04-22T17:29:12.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.679425+0000 mon.a (mon.0) 172 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.679557+0000 mon.a (mon.0) 173 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.680028+0000 mon.a (mon.0) 174 : cluster [DBG] mgrmap e29: y(active, since 58s), standbys: x 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.680245+0000 mon.a (mon.0) 175 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.680274+0000 mon.a (mon.0) 176 : cluster [INF] Cluster is now healthy 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.699447+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2022-04-22T17:29:12.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:11 smithi115 bash[45495]: audit 2022-04-22T17:29:11.703375+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:12.770 INFO:tasks.rados.rados.0.smithi115.stdout:1028: done (1 left) 2022-04-22T17:29:12.786 INFO:tasks.rados.rados.0.smithi115.stdout:1029: done (0 left) 2022-04-22T17:29:12.786 INFO:tasks.rados.rados.0.smithi115.stdout:1030: rollback oid 26 current snap is 104 2022-04-22T17:29:12.786 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 89 2022-04-22T17:29:12.786 INFO:tasks.rados.rados.0.smithi115.stdout:1031: copy_from oid 48 from oid 14 current snap is 104 2022-04-22T17:29:12.786 INFO:tasks.rados.rados.0.smithi115.stdout:1032: snap_create 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:1030: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:1030: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:1030: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 529 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:1031: finishing copy_from to smithi11541303-48 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 240 (ObjNum 285 snap 93 seq_num 285) dirty exists 2022-04-22T17:29:12.788 INFO:tasks.rados.rados.0.smithi115.stdout:1031: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:29:12.873 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:29:12.873 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:29:12.873 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:29:12.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:29:12.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:12.875 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:29:12.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 10, 2022-04-22T17:29:12.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 5 2022-04-22T17:29:12.875 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:29:12.875 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:29:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:12 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.138562+0000 mgr.y (mgr.34107) 49 : cluster [DBG] pgmap v73: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:12 smithi121 bash[36902]: cluster 2022-04-22T17:29:11.739652+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2022-04-22T17:29:13.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:12 smithi121 bash[36902]: audit 2022-04-22T17:29:12.292365+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:13.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.138562+0000 mgr.y (mgr.34107) 49 : cluster [DBG] pgmap v73: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:13.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[45495]: cluster 2022-04-22T17:29:11.739652+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2022-04-22T17:29:13.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[45495]: audit 2022-04-22T17:29:12.292365+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:13.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.138562+0000 mgr.y (mgr.34107) 49 : cluster [DBG] pgmap v73: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 31 KiB/s wr, 1 op/s 2022-04-22T17:29:13.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[46934]: cluster 2022-04-22T17:29:11.739652+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2022-04-22T17:29:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:12 smithi115 bash[46934]: audit 2022-04-22T17:29:12.292365+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:13.773 INFO:tasks.rados.rados.0.smithi115.stdout:1030: done (2 left) 2022-04-22T17:29:13.773 INFO:tasks.rados.rados.0.smithi115.stdout:1031: done (1 left) 2022-04-22T17:29:13.773 INFO:tasks.rados.rados.0.smithi115.stdout:1032: done (0 left) 2022-04-22T17:29:13.774 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 105 2022-04-22T17:29:13.774 INFO:tasks.rados.rados.0.smithi115.stdout:1033: seq_num 312 ranges {4505600=434176} 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.503Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.513Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.775 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.513Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:13.776 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.776 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:13.776 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:13.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[45495]: cluster 2022-04-22T17:29:12.755010+0000 mon.a (mon.0) 181 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2022-04-22T17:29:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[45495]: audit 2022-04-22T17:29:12.860912+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.115:0/1260746760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[45495]: audit 2022-04-22T17:29:13.231379+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[45495]: audit 2022-04-22T17:29:13.237991+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:13.806 INFO:tasks.rados.rados.0.smithi115.stdout:1033: writing smithi11541303-23 from 4505600 to 4939776 tid 1 2022-04-22T17:29:13.806 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 105 2022-04-22T17:29:13.806 INFO:tasks.rados.rados.0.smithi115.stdout:1034: seq_num 313 ranges {4538368=450560} 2022-04-22T17:29:13.835 INFO:tasks.rados.rados.0.smithi115.stdout:1034: writing smithi11541303-36 from 4538368 to 4988928 tid 1 2022-04-22T17:29:13.835 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 105 2022-04-22T17:29:13.835 INFO:tasks.rados.rados.0.smithi115.stdout:1035: seq_num 314 ranges {15892480=466944} 2022-04-22T17:29:13.927 INFO:tasks.rados.rados.0.smithi115.stdout:1035: writing smithi11541303-12 from 15892480 to 16359424 tid 1 2022-04-22T17:29:13.966 INFO:tasks.rados.rados.0.smithi115.stdout:1036: snap_create 2022-04-22T17:29:13.966 INFO:tasks.rados.rados.0.smithi115.stdout:1033: finishing write tid 1 to smithi11541303-23 2022-04-22T17:29:13.966 INFO:tasks.rados.rados.0.smithi115.stdout:1033: finishing write tid 2 to smithi11541303-23 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1033: finishing write tid 3 to smithi11541303-23 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 437 (ObjNum 312 snap 105 seq_num 312) dirty exists 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1033: left oid 23 (ObjNum 312 snap 105 seq_num 312) 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1034: finishing write tid 1 to smithi11541303-36 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1034: finishing write tid 2 to smithi11541303-36 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1034: finishing write tid 3 to smithi11541303-36 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 390 (ObjNum 313 snap 105 seq_num 313) dirty exists 2022-04-22T17:29:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1034: left oid 36 (ObjNum 313 snap 105 seq_num 313) 2022-04-22T17:29:13.968 INFO:tasks.rados.rados.0.smithi115.stdout:1035: finishing write tid 1 to smithi11541303-12 2022-04-22T17:29:13.969 INFO:tasks.rados.rados.0.smithi115.stdout:1035: finishing write tid 2 to smithi11541303-12 2022-04-22T17:29:13.969 INFO:tasks.rados.rados.0.smithi115.stdout:1035: finishing write tid 3 to smithi11541303-12 2022-04-22T17:29:13.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 386 (ObjNum 314 snap 105 seq_num 314) dirty exists 2022-04-22T17:29:13.969 INFO:tasks.rados.rados.0.smithi115.stdout:1035: left oid 12 (ObjNum 314 snap 105 seq_num 314) 2022-04-22T17:29:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[46934]: cluster 2022-04-22T17:29:12.755010+0000 mon.a (mon.0) 181 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2022-04-22T17:29:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[46934]: audit 2022-04-22T17:29:12.860912+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.115:0/1260746760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[46934]: audit 2022-04-22T17:29:13.231379+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:13 smithi115 bash[46934]: audit 2022-04-22T17:29:13.237991+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:13 smithi121 bash[36902]: cluster 2022-04-22T17:29:12.755010+0000 mon.a (mon.0) 181 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2022-04-22T17:29:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:13 smithi121 bash[36902]: audit 2022-04-22T17:29:12.860912+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.115:0/1260746760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:13 smithi121 bash[36902]: audit 2022-04-22T17:29:13.231379+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:29:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:13 smithi121 bash[36902]: audit 2022-04-22T17:29:13.237991+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:29:14.745 INFO:tasks.rados.rados.0.smithi115.stdout:1033: done (3 left) 2022-04-22T17:29:14.745 INFO:tasks.rados.rados.0.smithi115.stdout:1034: done (2 left) 2022-04-22T17:29:14.745 INFO:tasks.rados.rados.0.smithi115.stdout:1035: done (1 left) 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1036: done (0 left) 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1037: read oid 16 snap -1 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1037: expect deleted 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1038: read oid 47 snap -1 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1038: expect (ObjNum 304 snap 101 seq_num 304) 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1039: read oid 29 snap -1 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1039: expect deleted 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:1040: rollback oid 14 current snap is 106 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 96 2022-04-22T17:29:14.746 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 106 2022-04-22T17:29:14.747 INFO:tasks.rados.rados.0.smithi115.stdout:1041: seq_num 315 ranges {9748480=483328} 2022-04-22T17:29:14.805 INFO:tasks.rados.rados.0.smithi115.stdout:1041: writing smithi11541303-13 from 9748480 to 10231808 tid 1 2022-04-22T17:29:14.805 INFO:tasks.rados.rados.0.smithi115.stdout:1042: read oid 9 snap -1 2022-04-22T17:29:14.805 INFO:tasks.rados.rados.0.smithi115.stdout:1042: expect deleted 2022-04-22T17:29:14.805 INFO:tasks.rados.rados.0.smithi115.stdout:1043: snap_create 2022-04-22T17:29:14.805 INFO:tasks.rados.rados.0.smithi115.stdout:1038: expect (ObjNum 304 snap 101 seq_num 304) 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1040: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1040: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1040: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 243 (ObjNum 285 snap 93 seq_num 285) dirty exists 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1041: finishing write tid 1 to smithi11541303-13 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1041: finishing write tid 2 to smithi11541303-13 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:1041: finishing write tid 3 to smithi11541303-13 2022-04-22T17:29:14.833 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 394 (ObjNum 315 snap 106 seq_num 315) dirty exists 2022-04-22T17:29:14.834 INFO:tasks.rados.rados.0.smithi115.stdout:1041: left oid 13 (ObjNum 315 snap 106 seq_num 315) 2022-04-22T17:29:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[45495]: cluster 2022-04-22T17:29:13.139382+0000 mgr.y (mgr.34107) 50 : cluster [DBG] pgmap v76: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[45495]: cluster 2022-04-22T17:29:13.759145+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2022-04-22T17:29:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[45495]: cluster 2022-04-22T17:29:14.731128+0000 mon.a (mon.0) 185 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2022-04-22T17:29:15.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[46934]: cluster 2022-04-22T17:29:13.139382+0000 mgr.y (mgr.34107) 50 : cluster [DBG] pgmap v76: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:15.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[46934]: cluster 2022-04-22T17:29:13.759145+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2022-04-22T17:29:15.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:14 smithi115 bash[46934]: cluster 2022-04-22T17:29:14.731128+0000 mon.a (mon.0) 185 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2022-04-22T17:29:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:14 smithi121 bash[36902]: cluster 2022-04-22T17:29:13.139382+0000 mgr.y (mgr.34107) 50 : cluster [DBG] pgmap v76: 121 pgs: 121 active+clean; 245 MiB data, 920 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:14 smithi121 bash[36902]: cluster 2022-04-22T17:29:13.759145+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2022-04-22T17:29:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:14 smithi121 bash[36902]: cluster 2022-04-22T17:29:14.731128+0000 mon.a (mon.0) 185 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1037: done (6 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1038: done (5 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1039: done (4 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1040: done (3 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1041: done (2 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1042: done (1 left) 2022-04-22T17:29:15.786 INFO:tasks.rados.rados.0.smithi115.stdout:1043: done (0 left) 2022-04-22T17:29:15.787 INFO:tasks.rados.rados.0.smithi115.stdout:1044: rmattr oid 40 current snap is 107 2022-04-22T17:29:15.787 INFO:tasks.rados.rados.0.smithi115.stdout:1044: done (0 left) 2022-04-22T17:29:15.787 INFO:tasks.rados.rados.0.smithi115.stdout:1045: snap_remove snap 98 2022-04-22T17:29:16.790 INFO:tasks.rados.rados.0.smithi115.stdout:1045: done (0 left) 2022-04-22T17:29:16.790 INFO:tasks.rados.rados.0.smithi115.stdout:1046: snap_remove snap 103 2022-04-22T17:29:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:16 smithi121 bash[36902]: cluster 2022-04-22T17:29:15.140265+0000 mgr.y (mgr.34107) 51 : cluster [DBG] pgmap v79: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:16 smithi121 bash[36902]: cluster 2022-04-22T17:29:15.773001+0000 mon.a (mon.0) 186 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2022-04-22T17:29:17.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:16 smithi115 bash[46934]: cluster 2022-04-22T17:29:15.140265+0000 mgr.y (mgr.34107) 51 : cluster [DBG] pgmap v79: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:17.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:16 smithi115 bash[46934]: cluster 2022-04-22T17:29:15.773001+0000 mon.a (mon.0) 186 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2022-04-22T17:29:17.132 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:16 smithi115 bash[45495]: cluster 2022-04-22T17:29:15.140265+0000 mgr.y (mgr.34107) 51 : cluster [DBG] pgmap v79: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:17.132 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:16 smithi115 bash[45495]: cluster 2022-04-22T17:29:15.773001+0000 mon.a (mon.0) 186 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2022-04-22T17:29:17.791 INFO:tasks.rados.rados.0.smithi115.stdout:1046: done (0 left) 2022-04-22T17:29:17.792 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 107 2022-04-22T17:29:17.792 INFO:tasks.rados.rados.0.smithi115.stdout:1047: seq_num 316 ranges {2670592=499712} 2022-04-22T17:29:17.816 INFO:tasks.rados.rados.0.smithi115.stdout:1047: writing smithi11541303-14 from 2670592 to 3170304 tid 1 2022-04-22T17:29:17.817 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 107 2022-04-22T17:29:17.817 INFO:tasks.rados.rados.0.smithi115.stdout:1048: seq_num 317 ranges {2998272=516096} 2022-04-22T17:29:17.837 INFO:tasks.rados.rados.0.smithi115.stdout:1048: writing smithi11541303-39 from 2998272 to 3514368 tid 1 2022-04-22T17:29:17.837 INFO:tasks.rados.rados.0.smithi115.stdout:1049: read oid 9 snap -1 2022-04-22T17:29:17.837 INFO:tasks.rados.rados.0.smithi115.stdout:1049: expect deleted 2022-04-22T17:29:17.837 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 107 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:1047: finishing write tid 1 to smithi11541303-14 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:1047: finishing write tid 2 to smithi11541303-14 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:1047: finishing write tid 3 to smithi11541303-14 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 245 (ObjNum 316 snap 107 seq_num 316) dirty exists 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:1047: left oid 14 (ObjNum 316 snap 107 seq_num 316) 2022-04-22T17:29:17.838 INFO:tasks.rados.rados.0.smithi115.stdout:1050: seq_num 318 ranges {1851392=532480} 2022-04-22T17:29:17.850 INFO:tasks.rados.rados.0.smithi115.stdout:1050: writing smithi11541303-28 from 1851392 to 2383872 tid 1 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1047: done (3 left) 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1051: snap_create 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1048: finishing write tid 1 to smithi11541303-39 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1048: finishing write tid 2 to smithi11541303-39 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1048: finishing write tid 3 to smithi11541303-39 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 523 (ObjNum 317 snap 107 seq_num 317) dirty exists 2022-04-22T17:29:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1048: left oid 39 (ObjNum 317 snap 107 seq_num 317) 2022-04-22T17:29:17.855 INFO:tasks.rados.rados.0.smithi115.stdout:1050: finishing write tid 1 to smithi11541303-28 2022-04-22T17:29:17.856 INFO:tasks.rados.rados.0.smithi115.stdout:1050: finishing write tid 2 to smithi11541303-28 2022-04-22T17:29:17.858 INFO:tasks.rados.rados.0.smithi115.stdout:1050: finishing write tid 3 to smithi11541303-28 2022-04-22T17:29:17.858 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 247 (ObjNum 318 snap 107 seq_num 318) dirty exists 2022-04-22T17:29:17.858 INFO:tasks.rados.rados.0.smithi115.stdout:1050: left oid 28 (ObjNum 318 snap 107 seq_num 318) 2022-04-22T17:29:18.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:17 smithi115 bash[45495]: cluster 2022-04-22T17:29:16.776647+0000 mon.a (mon.0) 187 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2022-04-22T17:29:18.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:17 smithi115 bash[45495]: cluster 2022-04-22T17:29:17.778799+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2022-04-22T17:29:18.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:17 smithi115 bash[46934]: cluster 2022-04-22T17:29:16.776647+0000 mon.a (mon.0) 187 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2022-04-22T17:29:18.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:17 smithi115 bash[46934]: cluster 2022-04-22T17:29:17.778799+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2022-04-22T17:29:18.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:17 smithi121 bash[36902]: cluster 2022-04-22T17:29:16.776647+0000 mon.a (mon.0) 187 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2022-04-22T17:29:18.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:17 smithi121 bash[36902]: cluster 2022-04-22T17:29:17.778799+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2022-04-22T17:29:18.807 INFO:tasks.rados.rados.0.smithi115.stdout:1048: done (3 left) 2022-04-22T17:29:18.808 INFO:tasks.rados.rados.0.smithi115.stdout:1049: done (2 left) 2022-04-22T17:29:18.808 INFO:tasks.rados.rados.0.smithi115.stdout:1050: done (1 left) 2022-04-22T17:29:18.808 INFO:tasks.rados.rados.0.smithi115.stdout:1051: done (0 left) 2022-04-22T17:29:18.808 INFO:tasks.rados.rados.0.smithi115.stdout:1052: snap_create 2022-04-22T17:29:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:18 smithi121 bash[36902]: cluster 2022-04-22T17:29:17.141033+0000 mgr.y (mgr.34107) 52 : cluster [DBG] pgmap v82: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:19.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:18 smithi115 bash[46934]: cluster 2022-04-22T17:29:17.141033+0000 mgr.y (mgr.34107) 52 : cluster [DBG] pgmap v82: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:19.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:18 smithi115 bash[45495]: cluster 2022-04-22T17:29:17.141033+0000 mgr.y (mgr.34107) 52 : cluster [DBG] pgmap v82: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 3 op/s 2022-04-22T17:29:19.810 INFO:tasks.rados.rados.0.smithi115.stdout:1052: done (0 left) 2022-04-22T17:29:19.811 INFO:tasks.rados.rados.0.smithi115.stdout:1053: delete oid 18 current snap is 109 2022-04-22T17:29:19.822 INFO:tasks.rados.rados.0.smithi115.stdout:1053: done (0 left) 2022-04-22T17:29:19.822 INFO:tasks.rados.rados.0.smithi115.stdout:1054: read oid 32 snap -1 2022-04-22T17:29:19.822 INFO:tasks.rados.rados.0.smithi115.stdout:1054: expect (ObjNum 261 snap 87 seq_num 261) 2022-04-22T17:29:19.822 INFO:tasks.rados.rados.0.smithi115.stdout:1055: read oid 48 snap -1 2022-04-22T17:29:19.822 INFO:tasks.rados.rados.0.smithi115.stdout:1055: expect (ObjNum 285 snap 93 seq_num 285) 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1056: read oid 14 snap 108 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1056: expect (ObjNum 316 snap 107 seq_num 316) 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1057: copy_from oid 30 from oid 43 current snap is 109 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1058: read oid 39 snap -1 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1058: expect (ObjNum 317 snap 107 seq_num 317) 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 109 2022-04-22T17:29:19.823 INFO:tasks.rados.rados.0.smithi115.stdout:1059: seq_num 320 ranges {16359424=565248} 2022-04-22T17:29:19.924 INFO:tasks.rados.rados.0.smithi115.stdout:1059: writing smithi11541303-12 from 16359424 to 16924672 tid 1 2022-04-22T17:29:19.925 INFO:tasks.rados.rados.0.smithi115.stdout:1060: rollback oid 40 current snap is 109 2022-04-22T17:29:19.925 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 99 2022-04-22T17:29:19.925 INFO:tasks.rados.rados.0.smithi115.stdout:1056: expect (ObjNum 316 snap 107 seq_num 316) 2022-04-22T17:29:20.063 INFO:tasks.rados.rados.0.smithi115.stdout:1055: expect (ObjNum 285 snap 93 seq_num 285) 2022-04-22T17:29:20.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:19 smithi121 bash[36902]: cluster 2022-04-22T17:29:18.794494+0000 mon.a (mon.0) 189 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2022-04-22T17:29:20.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:19 smithi121 bash[36902]: audit 2022-04-22T17:29:19.150106+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:20.137 INFO:tasks.rados.rados.0.smithi115.stdout:1057: finishing copy_from to smithi11541303-30 2022-04-22T17:29:20.137 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 375 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:29:20.137 INFO:tasks.rados.rados.0.smithi115.stdout:1057: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:29:20.137 INFO:tasks.rados.rados.0.smithi115.stdout:1058: expect (ObjNum 317 snap 107 seq_num 317) 2022-04-22T17:29:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:19 smithi115 bash[45495]: cluster 2022-04-22T17:29:18.794494+0000 mon.a (mon.0) 189 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2022-04-22T17:29:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:19 smithi115 bash[45495]: audit 2022-04-22T17:29:19.150106+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:19 smithi115 bash[46934]: cluster 2022-04-22T17:29:18.794494+0000 mon.a (mon.0) 189 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2022-04-22T17:29:20.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:19 smithi115 bash[46934]: audit 2022-04-22T17:29:19.150106+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:20.351 INFO:tasks.rados.rados.0.smithi115.stdout:1054: expect (ObjNum 261 snap 87 seq_num 261) 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1060: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1060: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 329 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1059: finishing write tid 1 to smithi11541303-12 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1054: done (6 left) 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1055: done (5 left) 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1056: done (4 left) 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1057: done (3 left) 2022-04-22T17:29:20.967 INFO:tasks.rados.rados.0.smithi115.stdout:1058: done (2 left) 2022-04-22T17:29:20.968 INFO:tasks.rados.rados.0.smithi115.stdout:1060: done (1 left) 2022-04-22T17:29:20.968 INFO:tasks.rados.rados.0.smithi115.stdout:1061: snap_remove snap 104 2022-04-22T17:29:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: cluster 2022-04-22T17:29:19.141765+0000 mgr.y (mgr.34107) 53 : cluster [DBG] pgmap v85: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: cluster 2022-04-22T17:29:19.797317+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: audit 2022-04-22T17:29:20.735851+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: audit 2022-04-22T17:29:20.741498+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: audit 2022-04-22T17:29:20.746580+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: audit 2022-04-22T17:29:20.747515+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:21.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:20 smithi121 bash[36902]: audit 2022-04-22T17:29:20.748361+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:21.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: cluster 2022-04-22T17:29:19.141765+0000 mgr.y (mgr.34107) 53 : cluster [DBG] pgmap v85: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:21.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: cluster 2022-04-22T17:29:19.797317+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2022-04-22T17:29:21.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: audit 2022-04-22T17:29:20.735851+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: audit 2022-04-22T17:29:20.741498+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: audit 2022-04-22T17:29:20.746580+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: audit 2022-04-22T17:29:20.747515+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[45495]: audit 2022-04-22T17:29:20.748361+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: cluster 2022-04-22T17:29:19.141765+0000 mgr.y (mgr.34107) 53 : cluster [DBG] pgmap v85: 121 pgs: 121 active+clean; 249 MiB data, 928 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: cluster 2022-04-22T17:29:19.797317+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: audit 2022-04-22T17:29:20.735851+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: audit 2022-04-22T17:29:20.741498+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: audit 2022-04-22T17:29:20.746580+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:21.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: audit 2022-04-22T17:29:20.747515+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:21.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:20 smithi115 bash[46934]: audit 2022-04-22T17:29:20.748361+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:21.826 INFO:tasks.rados.rados.0.smithi115.stdout:1061: done (1 left) 2022-04-22T17:29:21.826 INFO:tasks.rados.rados.0.smithi115.stdout:1062: read oid 20 snap -1 2022-04-22T17:29:21.826 INFO:tasks.rados.rados.0.smithi115.stdout:1062: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:29:21.826 INFO:tasks.rados.rados.0.smithi115.stdout:1063: snap_remove snap 59 2022-04-22T17:29:22.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[45495]: cephadm 2022-04-22T17:29:20.728476+0000 mgr.y (mgr.34107) 54 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:22.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[45495]: cephadm 2022-04-22T17:29:20.746170+0000 mgr.y (mgr.34107) 55 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2022-04-22T17:29:22.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[45495]: cephadm 2022-04-22T17:29:20.749258+0000 mgr.y (mgr.34107) 56 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:29:22.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[46934]: cephadm 2022-04-22T17:29:20.728476+0000 mgr.y (mgr.34107) 54 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:22.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[46934]: cephadm 2022-04-22T17:29:20.746170+0000 mgr.y (mgr.34107) 55 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2022-04-22T17:29:22.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[46934]: cephadm 2022-04-22T17:29:20.749258+0000 mgr.y (mgr.34107) 56 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:29:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:21.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:21.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:22.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:21.618Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:21 smithi121 bash[36902]: cephadm 2022-04-22T17:29:20.728476+0000 mgr.y (mgr.34107) 54 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:29:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:21 smithi121 bash[36902]: cephadm 2022-04-22T17:29:20.746170+0000 mgr.y (mgr.34107) 55 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2022-04-22T17:29:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:21 smithi121 bash[36902]: cephadm 2022-04-22T17:29:20.749258+0000 mgr.y (mgr.34107) 56 : cephadm [INF] Reconfiguring daemon mon.a on smithi115 2022-04-22T17:29:22.833 INFO:tasks.rados.rados.0.smithi115.stdout:1063: done (2 left) 2022-04-22T17:29:22.833 INFO:tasks.rados.rados.0.smithi115.stdout:1064: rollback oid 32 current snap is 109 2022-04-22T17:29:22.833 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 95 2022-04-22T17:29:22.833 INFO:tasks.rados.rados.0.smithi115.stdout:1059: finishing write tid 2 to smithi11541303-12 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:1059: finishing write tid 3 to smithi11541303-12 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 388 (ObjNum 320 snap 109 seq_num 320) dirty exists 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:1059: left oid 12 (ObjNum 320 snap 109 seq_num 320) 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:1059: done (2 left) 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:1065: snap_create 2022-04-22T17:29:22.834 INFO:tasks.rados.rados.0.smithi115.stdout:1062: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:29:23.029 INFO:tasks.rados.rados.0.smithi115.stdout:1064: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:29:23.030 INFO:tasks.rados.rados.0.smithi115.stdout:1064: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:29:23.030 INFO:tasks.rados.rados.0.smithi115.stdout:1064: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:29:23.030 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 532 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:29:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:22 smithi121 bash[36902]: cluster 2022-04-22T17:29:21.142569+0000 mgr.y (mgr.34107) 57 : cluster [DBG] pgmap v87: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 454 KiB/s wr, 4 op/s 2022-04-22T17:29:23.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:22 smithi121 bash[36902]: cluster 2022-04-22T17:29:21.813191+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2022-04-22T17:29:23.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:29:22 smithi121 bash[18604]: debug 2022-04-22T17:29:22.838+0000 7faa1309e700 -1 osd.5 pg_epoch: 269 pg[5.7s0( v 269'584 (0'0,269'584] local-lis/les=63/64 n=16 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [5,1,0]p5(0) r=0 lpr=63 luod=269'582 crt=269'582 lcod 269'580 mlcod 269'580 active+clean+snaptrim trimq=[70~1] ps=[b8~1,c1~2]] removing snap head 2022-04-22T17:29:23.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:22 smithi115 bash[45495]: cluster 2022-04-22T17:29:21.142569+0000 mgr.y (mgr.34107) 57 : cluster [DBG] pgmap v87: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 454 KiB/s wr, 4 op/s 2022-04-22T17:29:23.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:22 smithi115 bash[45495]: cluster 2022-04-22T17:29:21.813191+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2022-04-22T17:29:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:22 smithi115 bash[46934]: cluster 2022-04-22T17:29:21.142569+0000 mgr.y (mgr.34107) 57 : cluster [DBG] pgmap v87: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 380 KiB/s rd, 454 KiB/s wr, 4 op/s 2022-04-22T17:29:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:22 smithi115 bash[46934]: cluster 2022-04-22T17:29:21.813191+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2022-04-22T17:29:23.301 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:29:22 smithi115 bash[26675]: debug 2022-04-22T17:29:22.836+0000 7faa048b3700 -1 osd.2 pg_epoch: 269 pg[5.6s0( v 269'361 (0'0,269'361] local-lis/les=63/64 n=10 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,5,1]p2(0) r=0 lpr=63 luod=268'359 crt=268'359 lcod 267'357 mlcod 267'357 active+clean+snaptrim trimq=[70~1] ps=[b8~1,c1~2]] removing snap head 2022-04-22T17:29:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.503Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:23.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:23.833 INFO:tasks.rados.rados.0.smithi115.stdout:1062: done (2 left) 2022-04-22T17:29:23.833 INFO:tasks.rados.rados.0.smithi115.stdout:1064: done (1 left) 2022-04-22T17:29:23.833 INFO:tasks.rados.rados.0.smithi115.stdout:1065: done (0 left) 2022-04-22T17:29:23.834 INFO:tasks.rados.rados.0.smithi115.stdout:1066: snap_create 2022-04-22T17:29:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: cluster 2022-04-22T17:29:22.819714+0000 mon.a (mon.0) 198 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2022-04-22T17:29:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: audit 2022-04-22T17:29:23.585641+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: audit 2022-04-22T17:29:23.587976+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: audit 2022-04-22T17:29:23.589435+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: audit 2022-04-22T17:29:23.590665+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:23 smithi121 bash[36902]: cluster 2022-04-22T17:29:23.818649+0000 mon.a (mon.0) 203 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2022-04-22T17:29:24.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: cluster 2022-04-22T17:29:22.819714+0000 mon.a (mon.0) 198 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2022-04-22T17:29:24.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: audit 2022-04-22T17:29:23.585641+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:24.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: audit 2022-04-22T17:29:23.587976+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: audit 2022-04-22T17:29:23.589435+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: audit 2022-04-22T17:29:23.590665+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[45495]: cluster 2022-04-22T17:29:23.818649+0000 mon.a (mon.0) 203 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: cluster 2022-04-22T17:29:22.819714+0000 mon.a (mon.0) 198 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: audit 2022-04-22T17:29:23.585641+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: audit 2022-04-22T17:29:23.587976+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:24.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: audit 2022-04-22T17:29:23.589435+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:24.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: audit 2022-04-22T17:29:23.590665+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:24.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:23 smithi115 bash[46934]: cluster 2022-04-22T17:29:23.818649+0000 mon.a (mon.0) 203 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2022-04-22T17:29:24.746 INFO:tasks.rados.rados.0.smithi115.stdout:1066: done (0 left) 2022-04-22T17:29:24.746 INFO:tasks.rados.rados.0.smithi115.stdout:1067: delete oid 39 current snap is 111 2022-04-22T17:29:24.758 INFO:tasks.rados.rados.0.smithi115.stdout:1067: done (0 left) 2022-04-22T17:29:24.758 INFO:tasks.rados.rados.0.smithi115.stdout:1068: rollback oid 33 current snap is 111 2022-04-22T17:29:24.758 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 108 2022-04-22T17:29:24.758 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 111 2022-04-22T17:29:24.758 INFO:tasks.rados.rados.0.smithi115.stdout:1069: seq_num 322 ranges {16924672=598016} 2022-04-22T17:29:24.859 INFO:tasks.rados.rados.0.smithi115.stdout:1069: writing smithi11541303-12 from 16924672 to 17522688 tid 1 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:1070: rollback oid 42 current snap is 111 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 90 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:1068: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 0 (ObjNum -1479266752 snap 32692 seq_num 3178613920) dirty dne 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:1068: done (2 left) 2022-04-22T17:29:24.876 INFO:tasks.rados.rados.0.smithi115.stdout:1071: snap_remove snap 102 2022-04-22T17:29:25.214 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[45495]: cluster 2022-04-22T17:29:23.143258+0000 mgr.y (mgr.34107) 58 : cluster [DBG] pgmap v90: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 382 KiB/s rd, 456 KiB/s wr, 4 op/s 2022-04-22T17:29:25.214 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[45495]: cephadm 2022-04-22T17:29:23.587414+0000 mgr.y (mgr.34107) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2022-04-22T17:29:25.214 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[45495]: cephadm 2022-04-22T17:29:23.591849+0000 mgr.y (mgr.34107) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:29:25.214 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[45495]: audit 2022-04-22T17:29:24.152629+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[45495]: cluster 2022-04-22T17:29:24.732671+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[46934]: cluster 2022-04-22T17:29:23.143258+0000 mgr.y (mgr.34107) 58 : cluster [DBG] pgmap v90: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 382 KiB/s rd, 456 KiB/s wr, 4 op/s 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[46934]: cephadm 2022-04-22T17:29:23.587414+0000 mgr.y (mgr.34107) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[46934]: cephadm 2022-04-22T17:29:23.591849+0000 mgr.y (mgr.34107) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[46934]: audit 2022-04-22T17:29:24.152629+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:25.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:24 smithi115 bash[46934]: cluster 2022-04-22T17:29:24.732671+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2022-04-22T17:29:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:24 smithi121 bash[36902]: cluster 2022-04-22T17:29:23.143258+0000 mgr.y (mgr.34107) 58 : cluster [DBG] pgmap v90: 121 pgs: 121 active+clean; 251 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 382 KiB/s rd, 456 KiB/s wr, 4 op/s 2022-04-22T17:29:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:24 smithi121 bash[36902]: cephadm 2022-04-22T17:29:23.587414+0000 mgr.y (mgr.34107) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2022-04-22T17:29:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:24 smithi121 bash[36902]: cephadm 2022-04-22T17:29:23.591849+0000 mgr.y (mgr.34107) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi115 2022-04-22T17:29:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:24 smithi121 bash[36902]: audit 2022-04-22T17:29:24.152629+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:24 smithi121 bash[36902]: cluster 2022-04-22T17:29:24.732671+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2022-04-22T17:29:25.862 INFO:tasks.rados.rados.0.smithi115.stdout:1071: done (2 left) 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1072: read oid 8 snap -1 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1072: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1073: read oid 31 snap -1 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1073: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1074: setattr oid 11 current snap is 111 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1069: finishing write tid 1 to smithi11541303-12 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1070: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:29:25.877 INFO:tasks.rados.rados.0.smithi115.stdout:1070: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:29:25.878 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 590 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:29:25.878 INFO:tasks.rados.rados.0.smithi115.stdout:1069: finishing write tid 2 to smithi11541303-12 2022-04-22T17:29:25.878 INFO:tasks.rados.rados.0.smithi115.stdout:1069: finishing write tid 3 to smithi11541303-12 2022-04-22T17:29:25.878 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 390 (ObjNum 322 snap 111 seq_num 322) dirty exists 2022-04-22T17:29:25.878 INFO:tasks.rados.rados.0.smithi115.stdout:1069: left oid 12 (ObjNum 322 snap 111 seq_num 322) 2022-04-22T17:29:25.879 INFO:tasks.rados.rados.0.smithi115.stdout:1069: done (4 left) 2022-04-22T17:29:25.879 INFO:tasks.rados.rados.0.smithi115.stdout:1070: done (3 left) 2022-04-22T17:29:25.879 INFO:tasks.rados.rados.0.smithi115.stdout:1075: snap_create 2022-04-22T17:29:25.883 INFO:tasks.rados.rados.0.smithi115.stdout:1073: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:29:25.884 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 457 (ObjNum 297 snap 99 seq_num 297) dirty exists 2022-04-22T17:29:25.884 INFO:tasks.rados.rados.0.smithi115.stdout:1072: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:26.870 INFO:tasks.rados.rados.0.smithi115.stdout:1072: done (3 left) 2022-04-22T17:29:26.871 INFO:tasks.rados.rados.0.smithi115.stdout:1073: done (2 left) 2022-04-22T17:29:26.871 INFO:tasks.rados.rados.0.smithi115.stdout:1074: done (1 left) 2022-04-22T17:29:26.871 INFO:tasks.rados.rados.0.smithi115.stdout:1075: done (0 left) 2022-04-22T17:29:26.871 INFO:tasks.rados.rados.0.smithi115.stdout:1076: setattr oid 40 current snap is 112 2022-04-22T17:29:26.871 INFO:tasks.rados.rados.0.smithi115.stdout:1077: read oid 8 snap -1 2022-04-22T17:29:26.872 INFO:tasks.rados.rados.0.smithi115.stdout:1077: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:26.872 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 112 2022-04-22T17:29:26.872 INFO:tasks.rados.rados.0.smithi115.stdout:1078: seq_num 323 ranges {9928704=614400} 2022-04-22T17:29:26.931 INFO:tasks.rados.rados.0.smithi115.stdout:1078: writing smithi11541303-10 from 9928704 to 10543104 tid 1 2022-04-22T17:29:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:1079: delete oid 26 current snap is 112 2022-04-22T17:29:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 339 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:29:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:1077: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:27.040 INFO:tasks.rados.rados.0.smithi115.stdout:1078: finishing write tid 1 to smithi11541303-10 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1078: finishing write tid 2 to smithi11541303-10 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1078: finishing write tid 3 to smithi11541303-10 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 342 (ObjNum 323 snap 112 seq_num 323) dirty exists 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1078: left oid 10 (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1076: done (3 left) 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1077: done (2 left) 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1078: done (1 left) 2022-04-22T17:29:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1079: done (0 left) 2022-04-22T17:29:27.042 INFO:tasks.rados.rados.0.smithi115.stdout:1080: read oid 28 snap -1 2022-04-22T17:29:27.042 INFO:tasks.rados.rados.0.smithi115.stdout:1080: expect (ObjNum 318 snap 107 seq_num 318) 2022-04-22T17:29:27.042 INFO:tasks.rados.rados.0.smithi115.stdout:1081: snap_remove snap 110 2022-04-22T17:29:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: cluster 2022-04-22T17:29:25.143892+0000 mgr.y (mgr.34107) 61 : cluster [DBG] pgmap v93: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: cluster 2022-04-22T17:29:25.848923+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: audit 2022-04-22T17:29:26.028960+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: cephadm 2022-04-22T17:29:26.030452+0000 mgr.y (mgr.34107) 62 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: audit 2022-04-22T17:29:26.030911+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: audit 2022-04-22T17:29:26.032217+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: audit 2022-04-22T17:29:26.033395+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:27.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:26 smithi121 bash[36902]: cephadm 2022-04-22T17:29:26.034858+0000 mgr.y (mgr.34107) 63 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:29:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: cluster 2022-04-22T17:29:25.143892+0000 mgr.y (mgr.34107) 61 : cluster [DBG] pgmap v93: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: cluster 2022-04-22T17:29:25.848923+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2022-04-22T17:29:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: audit 2022-04-22T17:29:26.028960+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: cephadm 2022-04-22T17:29:26.030452+0000 mgr.y (mgr.34107) 62 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: audit 2022-04-22T17:29:26.030911+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: audit 2022-04-22T17:29:26.032217+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: audit 2022-04-22T17:29:26.033395+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[45495]: cephadm 2022-04-22T17:29:26.034858+0000 mgr.y (mgr.34107) 63 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: cluster 2022-04-22T17:29:25.143892+0000 mgr.y (mgr.34107) 61 : cluster [DBG] pgmap v93: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: cluster 2022-04-22T17:29:25.848923+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2022-04-22T17:29:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: audit 2022-04-22T17:29:26.028960+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: cephadm 2022-04-22T17:29:26.030452+0000 mgr.y (mgr.34107) 62 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2022-04-22T17:29:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: audit 2022-04-22T17:29:26.030911+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: audit 2022-04-22T17:29:26.032217+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: audit 2022-04-22T17:29:26.033395+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:26 smithi115 bash[46934]: cephadm 2022-04-22T17:29:26.034858+0000 mgr.y (mgr.34107) 63 : cephadm [INF] Reconfiguring daemon mon.c on smithi115 2022-04-22T17:29:27.877 INFO:tasks.rados.rados.0.smithi115.stdout:1081: done (1 left) 2022-04-22T17:29:27.877 INFO:tasks.rados.rados.0.smithi115.stdout:1082: snap_remove snap 100 2022-04-22T17:29:27.877 INFO:tasks.rados.rados.0.smithi115.stdout:1080: expect (ObjNum 318 snap 107 seq_num 318) 2022-04-22T17:29:28.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:27 smithi115 bash[45495]: cluster 2022-04-22T17:29:26.856005+0000 mon.a (mon.0) 211 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2022-04-22T17:29:28.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:27 smithi115 bash[46934]: cluster 2022-04-22T17:29:26.856005+0000 mon.a (mon.0) 211 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2022-04-22T17:29:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:27 smithi121 bash[36902]: cluster 2022-04-22T17:29:26.856005+0000 mon.a (mon.0) 211 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2022-04-22T17:29:28.880 INFO:tasks.rados.rados.0.smithi115.stdout:1080: done (1 left) 2022-04-22T17:29:28.889 INFO:tasks.rados.rados.0.smithi115.stdout:1082: done (0 left) 2022-04-22T17:29:28.889 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 112 2022-04-22T17:29:28.889 INFO:tasks.rados.rados.0.smithi115.stdout:1083: seq_num 325 ranges {4939776=647168} 2022-04-22T17:29:28.915 INFO:tasks.rados.rados.0.smithi115.stdout:1083: writing smithi11541303-23 from 4939776 to 5586944 tid 1 2022-04-22T17:29:28.915 INFO:tasks.rados.rados.0.smithi115.stdout:1084: copy_from oid 40 from oid 35 current snap is 112 2022-04-22T17:29:28.915 INFO:tasks.rados.rados.0.smithi115.stdout:1085: snap_create 2022-04-22T17:29:28.916 INFO:tasks.rados.rados.0.smithi115.stdout:1084: finishing copy_from to smithi11541303-40 2022-04-22T17:29:28.916 INFO:tasks.rados.rados.0.smithi115.stdout:1084: got expected ENOENT (src dne) 2022-04-22T17:29:28.916 INFO:tasks.rados.rados.0.smithi115.stdout:1084: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:29:28.921 INFO:tasks.rados.rados.0.smithi115.stdout:1083: finishing write tid 1 to smithi11541303-23 2022-04-22T17:29:28.921 INFO:tasks.rados.rados.0.smithi115.stdout:1083: finishing write tid 2 to smithi11541303-23 2022-04-22T17:29:28.923 INFO:tasks.rados.rados.0.smithi115.stdout:1083: finishing write tid 3 to smithi11541303-23 2022-04-22T17:29:28.924 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 470 (ObjNum 325 snap 112 seq_num 325) dirty exists 2022-04-22T17:29:28.924 INFO:tasks.rados.rados.0.smithi115.stdout:1083: left oid 23 (ObjNum 325 snap 112 seq_num 325) 2022-04-22T17:29:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[46934]: cluster 2022-04-22T17:29:27.144656+0000 mgr.y (mgr.34107) 64 : cluster [DBG] pgmap v96: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[46934]: cluster 2022-04-22T17:29:27.864389+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2022-04-22T17:29:29.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[46934]: audit 2022-04-22T17:29:28.366139+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:29.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[45495]: cluster 2022-04-22T17:29:27.144656+0000 mgr.y (mgr.34107) 64 : cluster [DBG] pgmap v96: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:29.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[45495]: cluster 2022-04-22T17:29:27.864389+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2022-04-22T17:29:29.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:28 smithi115 bash[45495]: audit 2022-04-22T17:29:28.366139+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:28 smithi121 bash[36902]: cluster 2022-04-22T17:29:27.144656+0000 mgr.y (mgr.34107) 64 : cluster [DBG] pgmap v96: 121 pgs: 121 active+clean; 211 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 178 KiB/s wr, 7 op/s 2022-04-22T17:29:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:28 smithi121 bash[36902]: cluster 2022-04-22T17:29:27.864389+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2022-04-22T17:29:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:28 smithi121 bash[36902]: audit 2022-04-22T17:29:28.366139+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:29.749 INFO:tasks.rados.rados.0.smithi115.stdout:1083: done (2 left) 2022-04-22T17:29:29.750 INFO:tasks.rados.rados.0.smithi115.stdout:1084: done (1 left) 2022-04-22T17:29:29.750 INFO:tasks.rados.rados.0.smithi115.stdout:1085: done (0 left) 2022-04-22T17:29:29.750 INFO:tasks.rados.rados.0.smithi115.stdout:1086: delete oid 3 current snap is 113 2022-04-22T17:29:29.755 INFO:tasks.rados.rados.0.smithi115.stdout:1086: done (0 left) 2022-04-22T17:29:29.756 INFO:tasks.rados.rados.0.smithi115.stdout:1087: read oid 6 snap -1 2022-04-22T17:29:29.756 INFO:tasks.rados.rados.0.smithi115.stdout:1087: expect (ObjNum 294 snap 98 seq_num 294) 2022-04-22T17:29:29.756 INFO:tasks.rados.rados.0.smithi115.stdout:1088: setattr oid 21 current snap is 113 2022-04-22T17:29:29.758 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 113 2022-04-22T17:29:29.758 INFO:tasks.rados.rados.0.smithi115.stdout:1089: seq_num 327 ranges {3293184=679936} 2022-04-22T17:29:29.782 INFO:tasks.rados.rados.0.smithi115.stdout:1089: writing smithi11541303-37 from 3293184 to 3973120 tid 1 2022-04-22T17:29:29.783 INFO:tasks.rados.rados.0.smithi115.stdout:1090: snap_create 2022-04-22T17:29:29.783 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 314 (ObjNum 247 snap 83 seq_num 247) dirty exists 2022-04-22T17:29:29.783 INFO:tasks.rados.rados.0.smithi115.stdout:1087: expect (ObjNum 294 snap 98 seq_num 294) 2022-04-22T17:29:29.796 INFO:tasks.rados.rados.0.smithi115.stdout:1089: finishing write tid 1 to smithi11541303-37 2022-04-22T17:29:29.796 INFO:tasks.rados.rados.0.smithi115.stdout:1089: finishing write tid 2 to smithi11541303-37 2022-04-22T17:29:29.796 INFO:tasks.rados.rados.0.smithi115.stdout:1089: finishing write tid 3 to smithi11541303-37 2022-04-22T17:29:29.796 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 526 (ObjNum 327 snap 113 seq_num 327) dirty exists 2022-04-22T17:29:29.796 INFO:tasks.rados.rados.0.smithi115.stdout:1089: left oid 37 (ObjNum 327 snap 113 seq_num 327) 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[46934]: cephadm 2022-04-22T17:29:28.367750+0000 mgr.y (mgr.34107) 65 : cephadm [INF] Reconfiguring osd.0 (monmap changed)... 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[46934]: cephadm 2022-04-22T17:29:28.369133+0000 mgr.y (mgr.34107) 66 : cephadm [INF] Reconfiguring daemon osd.0 on smithi115 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[46934]: cluster 2022-04-22T17:29:28.864670+0000 mon.a (mon.0) 214 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[46934]: audit 2022-04-22T17:29:29.154846+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[46934]: cluster 2022-04-22T17:29:29.735529+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[45495]: cephadm 2022-04-22T17:29:28.367750+0000 mgr.y (mgr.34107) 65 : cephadm [INF] Reconfiguring osd.0 (monmap changed)... 2022-04-22T17:29:30.135 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[45495]: cephadm 2022-04-22T17:29:28.369133+0000 mgr.y (mgr.34107) 66 : cephadm [INF] Reconfiguring daemon osd.0 on smithi115 2022-04-22T17:29:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[45495]: cluster 2022-04-22T17:29:28.864670+0000 mon.a (mon.0) 214 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2022-04-22T17:29:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[45495]: audit 2022-04-22T17:29:29.154846+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:29 smithi115 bash[45495]: cluster 2022-04-22T17:29:29.735529+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2022-04-22T17:29:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:29 smithi121 bash[36902]: cephadm 2022-04-22T17:29:28.367750+0000 mgr.y (mgr.34107) 65 : cephadm [INF] Reconfiguring osd.0 (monmap changed)... 2022-04-22T17:29:30.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:29 smithi121 bash[36902]: cephadm 2022-04-22T17:29:28.369133+0000 mgr.y (mgr.34107) 66 : cephadm [INF] Reconfiguring daemon osd.0 on smithi115 2022-04-22T17:29:30.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:29 smithi121 bash[36902]: cluster 2022-04-22T17:29:28.864670+0000 mon.a (mon.0) 214 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2022-04-22T17:29:30.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:29 smithi121 bash[36902]: audit 2022-04-22T17:29:29.154846+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:30.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:29 smithi121 bash[36902]: cluster 2022-04-22T17:29:29.735529+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2022-04-22T17:29:30.759 INFO:tasks.rados.rados.0.smithi115.stdout:1087: done (3 left) 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:1088: done (2 left) 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:1089: done (1 left) 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:1090: done (0 left) 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:1091: copy_from oid 6 from oid 49 current snap is 114 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:1092: rollback oid 33 current snap is 114 2022-04-22T17:29:30.763 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 99 2022-04-22T17:29:30.764 INFO:tasks.rados.rados.0.smithi115.stdout:1093: copy_from oid 38 from oid 1 current snap is 114 2022-04-22T17:29:30.764 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 114 2022-04-22T17:29:30.764 INFO:tasks.rados.rados.0.smithi115.stdout:1094: seq_num 328 ranges {0=696320} 2022-04-22T17:29:30.769 INFO:tasks.rados.rados.0.smithi115.stdout:1094: writing smithi11541303-16 from 0 to 696320 tid 1 2022-04-22T17:29:30.770 INFO:tasks.rados.rados.0.smithi115.stdout:1095: snap_create 2022-04-22T17:29:30.773 INFO:tasks.rados.rados.0.smithi115.stdout:1091: finishing copy_from to smithi11541303-6 2022-04-22T17:29:30.774 INFO:tasks.rados.rados.0.smithi115.stdout:1091: got expected ENOENT (src dne) 2022-04-22T17:29:30.774 INFO:tasks.rados.rados.0.smithi115.stdout:1092: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:29:30.774 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 0 (ObjNum 1 snap 32692 seq_num 3179405808) dirty dne 2022-04-22T17:29:30.774 INFO:tasks.rados.rados.0.smithi115.stdout:1091: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:29:30.778 INFO:tasks.rados.rados.0.smithi115.stdout:1094: finishing write tid 1 to smithi11541303-16 2022-04-22T17:29:30.779 INFO:tasks.rados.rados.0.smithi115.stdout:1094: finishing write tid 2 to smithi11541303-16 2022-04-22T17:29:30.782 INFO:tasks.rados.rados.0.smithi115.stdout:1093: finishing copy_from to smithi11541303-38 2022-04-22T17:29:30.782 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 380 (ObjNum 273 snap 91 seq_num 273) dirty exists 2022-04-22T17:29:30.783 INFO:tasks.rados.rados.0.smithi115.stdout:1093: finishing copy_from racing read to smithi11541303-38 2022-04-22T17:29:30.785 INFO:tasks.rados.rados.0.smithi115.stdout:1094: finishing write tid 3 to smithi11541303-16 2022-04-22T17:29:30.785 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 373 (ObjNum 328 snap 114 seq_num 328) dirty exists 2022-04-22T17:29:30.785 INFO:tasks.rados.rados.0.smithi115.stdout:1094: left oid 16 (ObjNum 328 snap 114 seq_num 328) 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[46934]: cluster 2022-04-22T17:29:29.145578+0000 mgr.y (mgr.34107) 67 : cluster [DBG] pgmap v99: 121 pgs: 121 active+clean; 212 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 308 KiB/s wr, 12 op/s 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[46934]: cluster 2022-04-22T17:29:30.745788+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[46934]: audit 2022-04-22T17:29:30.833040+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[45495]: cluster 2022-04-22T17:29:29.145578+0000 mgr.y (mgr.34107) 67 : cluster [DBG] pgmap v99: 121 pgs: 121 active+clean; 212 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 308 KiB/s wr, 12 op/s 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[45495]: cluster 2022-04-22T17:29:30.745788+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2022-04-22T17:29:31.143 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:30 smithi115 bash[45495]: audit 2022-04-22T17:29:30.833040+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:30 smithi121 bash[36902]: cluster 2022-04-22T17:29:29.145578+0000 mgr.y (mgr.34107) 67 : cluster [DBG] pgmap v99: 121 pgs: 121 active+clean; 212 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 308 KiB/s wr, 12 op/s 2022-04-22T17:29:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:30 smithi121 bash[36902]: cluster 2022-04-22T17:29:30.745788+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2022-04-22T17:29:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:30 smithi121 bash[36902]: audit 2022-04-22T17:29:30.833040+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:31.767 INFO:tasks.rados.rados.0.smithi115.stdout:1091: done (4 left) 2022-04-22T17:29:31.767 INFO:tasks.rados.rados.0.smithi115.stdout:1092: done (3 left) 2022-04-22T17:29:31.767 INFO:tasks.rados.rados.0.smithi115.stdout:1093: done (2 left) 2022-04-22T17:29:31.767 INFO:tasks.rados.rados.0.smithi115.stdout:1094: done (1 left) 2022-04-22T17:29:31.767 INFO:tasks.rados.rados.0.smithi115.stdout:1095: done (0 left) 2022-04-22T17:29:31.768 INFO:tasks.rados.rados.0.smithi115.stdout:1096: copy_from oid 31 from oid 47 current snap is 115 2022-04-22T17:29:31.768 INFO:tasks.rados.rados.0.smithi115.stdout:1097: read oid 12 snap 109 2022-04-22T17:29:31.768 INFO:tasks.rados.rados.0.smithi115.stdout:1097: expect (ObjNum 320 snap 109 seq_num 320) 2022-04-22T17:29:31.768 INFO:tasks.rados.rados.0.smithi115.stdout:1098: delete oid 32 current snap is 115 2022-04-22T17:29:31.777 INFO:tasks.rados.rados.0.smithi115.stdout:1098: done (2 left) 2022-04-22T17:29:31.777 INFO:tasks.rados.rados.0.smithi115.stdout:1099: snap_create 2022-04-22T17:29:31.779 INFO:tasks.rados.rados.0.smithi115.stdout:1096: finishing copy_from to smithi11541303-31 2022-04-22T17:29:31.780 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 303 (ObjNum 304 snap 101 seq_num 304) dirty exists 2022-04-22T17:29:31.782 INFO:tasks.rados.rados.0.smithi115.stdout:1096: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:29:31.832 INFO:tasks.rados.rados.0.smithi115.stdout:1097: expect (ObjNum 320 snap 109 seq_num 320) 2022-04-22T17:29:31.906 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:31.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:31.906 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:31.618Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:31.906 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:31.618Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:31.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[45495]: cephadm 2022-04-22T17:29:30.834556+0000 mgr.y (mgr.34107) 68 : cephadm [INF] Reconfiguring osd.1 (monmap changed)... 2022-04-22T17:29:31.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[45495]: cephadm 2022-04-22T17:29:30.835873+0000 mgr.y (mgr.34107) 69 : cephadm [INF] Reconfiguring daemon osd.1 on smithi115 2022-04-22T17:29:31.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[45495]: cluster 2022-04-22T17:29:31.753394+0000 mon.a (mon.0) 219 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2022-04-22T17:29:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:31 smithi121 bash[36902]: cephadm 2022-04-22T17:29:30.834556+0000 mgr.y (mgr.34107) 68 : cephadm [INF] Reconfiguring osd.1 (monmap changed)... 2022-04-22T17:29:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:31 smithi121 bash[36902]: cephadm 2022-04-22T17:29:30.835873+0000 mgr.y (mgr.34107) 69 : cephadm [INF] Reconfiguring daemon osd.1 on smithi115 2022-04-22T17:29:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:31 smithi121 bash[36902]: cluster 2022-04-22T17:29:31.753394+0000 mon.a (mon.0) 219 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2022-04-22T17:29:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[46934]: cephadm 2022-04-22T17:29:30.834556+0000 mgr.y (mgr.34107) 68 : cephadm [INF] Reconfiguring osd.1 (monmap changed)... 2022-04-22T17:29:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[46934]: cephadm 2022-04-22T17:29:30.835873+0000 mgr.y (mgr.34107) 69 : cephadm [INF] Reconfiguring daemon osd.1 on smithi115 2022-04-22T17:29:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:31 smithi115 bash[46934]: cluster 2022-04-22T17:29:31.753394+0000 mon.a (mon.0) 219 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2022-04-22T17:29:33.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:32 smithi115 bash[45495]: cluster 2022-04-22T17:29:31.146274+0000 mgr.y (mgr.34107) 70 : cluster [DBG] pgmap v102: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 454 KiB/s wr, 17 op/s 2022-04-22T17:29:33.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:32 smithi115 bash[45495]: cluster 2022-04-22T17:29:32.753330+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2022-04-22T17:29:33.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:32 smithi115 bash[46934]: cluster 2022-04-22T17:29:31.146274+0000 mgr.y (mgr.34107) 70 : cluster [DBG] pgmap v102: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 454 KiB/s wr, 17 op/s 2022-04-22T17:29:33.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:32 smithi115 bash[46934]: cluster 2022-04-22T17:29:32.753330+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2022-04-22T17:29:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:32 smithi121 bash[36902]: cluster 2022-04-22T17:29:31.146274+0000 mgr.y (mgr.34107) 70 : cluster [DBG] pgmap v102: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 454 KiB/s wr, 17 op/s 2022-04-22T17:29:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:32 smithi121 bash[36902]: cluster 2022-04-22T17:29:32.753330+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2022-04-22T17:29:33.778 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.503Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.503Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:33.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:33.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:33.893 INFO:tasks.rados.rados.0.smithi115.stdout:1096: done (2 left) 2022-04-22T17:29:33.900 INFO:tasks.rados.rados.0.smithi115.stdout:1097: done (1 left) 2022-04-22T17:29:33.901 INFO:tasks.rados.rados.0.smithi115.stdout:1099: done (0 left) 2022-04-22T17:29:33.901 INFO:tasks.rados.rados.0.smithi115.stdout:1100: copy_from oid 38 from oid 8 current snap is 116 2022-04-22T17:29:33.901 INFO:tasks.rados.rados.0.smithi115.stdout:1101: delete oid 42 current snap is 116 2022-04-22T17:29:33.901 INFO:tasks.rados.rados.0.smithi115.stdout:1101: done (1 left) 2022-04-22T17:29:33.901 INFO:tasks.rados.rados.0.smithi115.stdout:1102: delete oid 29 current snap is 116 2022-04-22T17:29:33.902 INFO:tasks.rados.rados.0.smithi115.stdout:1102: done (1 left) 2022-04-22T17:29:33.902 INFO:tasks.rados.rados.0.smithi115.stdout:1103: rollback oid 27 current snap is 116 2022-04-22T17:29:33.902 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 108 2022-04-22T17:29:33.902 INFO:tasks.rados.rados.0.smithi115.stdout:1104: snap_remove snap 78 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1104: done (2 left) 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1105: copy_from oid 3 from oid 44 current snap is 116 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1103: finishing rollback tid 0 to smithi11541303-27 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1103: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1103: finishing rollback tid 2 to smithi11541303-27 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 548 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:29:34.524 INFO:tasks.rados.rados.0.smithi115.stdout:1100: finishing copy_from to smithi11541303-38 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1100: finishing copy_from racing read to smithi11541303-38 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1100: done (2 left) 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1103: done (1 left) 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1106: rollback oid 23 current snap is 116 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 91 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1107: rollback oid 49 current snap is 116 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 106 2022-04-22T17:29:34.525 INFO:tasks.rados.rados.0.smithi115.stdout:1108: snap_remove snap 83 2022-04-22T17:29:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:34 smithi121 bash[36902]: cluster 2022-04-22T17:29:33.146975+0000 mgr.y (mgr.34107) 71 : cluster [DBG] pgmap v105: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 146 KiB/s wr, 5 op/s 2022-04-22T17:29:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:34 smithi121 bash[36902]: audit 2022-04-22T17:29:33.500321+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:34 smithi121 bash[36902]: cephadm 2022-04-22T17:29:33.501629+0000 mgr.y (mgr.34107) 72 : cephadm [INF] Reconfiguring osd.2 (monmap changed)... 2022-04-22T17:29:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:34 smithi121 bash[36902]: cephadm 2022-04-22T17:29:33.502836+0000 mgr.y (mgr.34107) 73 : cephadm [INF] Reconfiguring daemon osd.2 on smithi115 2022-04-22T17:29:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:34 smithi121 bash[36902]: audit 2022-04-22T17:29:34.157170+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:34.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[46934]: cluster 2022-04-22T17:29:33.146975+0000 mgr.y (mgr.34107) 71 : cluster [DBG] pgmap v105: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 146 KiB/s wr, 5 op/s 2022-04-22T17:29:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[46934]: audit 2022-04-22T17:29:33.500321+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[46934]: cephadm 2022-04-22T17:29:33.501629+0000 mgr.y (mgr.34107) 72 : cephadm [INF] Reconfiguring osd.2 (monmap changed)... 2022-04-22T17:29:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[46934]: cephadm 2022-04-22T17:29:33.502836+0000 mgr.y (mgr.34107) 73 : cephadm [INF] Reconfiguring daemon osd.2 on smithi115 2022-04-22T17:29:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[46934]: audit 2022-04-22T17:29:34.157170+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:34.883 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[45495]: cluster 2022-04-22T17:29:33.146975+0000 mgr.y (mgr.34107) 71 : cluster [DBG] pgmap v105: 121 pgs: 121 active+clean; 222 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 146 KiB/s wr, 5 op/s 2022-04-22T17:29:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[45495]: audit 2022-04-22T17:29:33.500321+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[45495]: cephadm 2022-04-22T17:29:33.501629+0000 mgr.y (mgr.34107) 72 : cephadm [INF] Reconfiguring osd.2 (monmap changed)... 2022-04-22T17:29:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[45495]: cephadm 2022-04-22T17:29:33.502836+0000 mgr.y (mgr.34107) 73 : cephadm [INF] Reconfiguring daemon osd.2 on smithi115 2022-04-22T17:29:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:34 smithi115 bash[45495]: audit 2022-04-22T17:29:34.157170+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:35.533 INFO:tasks.rados.rados.0.smithi115.stdout:1108: done (3 left) 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:1109: copy_from oid 19 from oid 11 current snap is 116 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:1107: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 0 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty dne 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:1106: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:1106: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:1106: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:29:35.534 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 479 (ObjNum 268 snap 88 seq_num 268) dirty exists 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1105: finishing copy_from to smithi11541303-3 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 25 (ObjNum -1116758960 snap 22075 seq_num 3178117952) dirty exists 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1105: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1105: done (3 left) 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1106: done (2 left) 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1107: done (1 left) 2022-04-22T17:29:35.535 INFO:tasks.rados.rados.0.smithi115.stdout:1110: snap_create 2022-04-22T17:29:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:35 smithi115 bash[46934]: cluster 2022-04-22T17:29:34.510397+0000 mon.a (mon.0) 223 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2022-04-22T17:29:35.576 INFO:tasks.rados.rados.0.smithi115.stdout:1109: finishing copy_from to smithi11541303-19 2022-04-22T17:29:35.577 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 457 (ObjNum 297 snap 99 seq_num 297) dirty exists 2022-04-22T17:29:35.578 INFO:tasks.rados.rados.0.smithi115.stdout:1109: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:29:35.800 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:29:35 smithi115 bash[26675]: debug 2022-04-22T17:29:35.540+0000 7faa050b4700 -1 osd.2 pg_epoch: 281 pg[5.ds0( v 280'315 (0'0,280'315] local-lis/les=63/64 n=12 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 crt=280'315 lcod 280'313 mlcod 280'313 active+clean+snaptrim trimq=[9c~1] ps=[95~1,bb~1,c0~1,c2~1,cc~1]] removing snap head 2022-04-22T17:29:35.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:35 smithi115 bash[45495]: cluster 2022-04-22T17:29:34.510397+0000 mon.a (mon.0) 223 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2022-04-22T17:29:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:35 smithi121 bash[36902]: cluster 2022-04-22T17:29:34.510397+0000 mon.a (mon.0) 223 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2022-04-22T17:29:36.534 INFO:tasks.rados.rados.0.smithi115.stdout:1109: done (1 left) 2022-04-22T17:29:36.535 INFO:tasks.rados.rados.0.smithi115.stdout:1110: done (0 left) 2022-04-22T17:29:36.535 INFO:tasks.rados.rados.0.smithi115.stdout:1111: copy_from oid 35 from oid 22 current snap is 117 2022-04-22T17:29:36.535 INFO:tasks.rados.rados.0.smithi115.stdout:1112: rollback oid 20 current snap is 117 2022-04-22T17:29:36.535 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 116 2022-04-22T17:29:36.535 INFO:tasks.rados.rados.0.smithi115.stdout:1113: delete oid 27 current snap is 117 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: cluster 2022-04-22T17:29:35.147751+0000 mgr.y (mgr.34107) 74 : cluster [DBG] pgmap v107: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: cluster 2022-04-22T17:29:35.519379+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: audit 2022-04-22T17:29:35.839603+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: cephadm 2022-04-22T17:29:35.841316+0000 mgr.y (mgr.34107) 75 : cephadm [INF] Reconfiguring osd.3 (monmap changed)... 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: cephadm 2022-04-22T17:29:35.842417+0000 mgr.y (mgr.34107) 76 : cephadm [INF] Reconfiguring daemon osd.3 on smithi115 2022-04-22T17:29:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[45495]: cluster 2022-04-22T17:29:36.519036+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2022-04-22T17:29:36.545 INFO:tasks.rados.rados.0.smithi115.stdout:1112: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:29:36.545 INFO:tasks.rados.rados.0.smithi115.stdout:1112: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:29:36.545 INFO:tasks.rados.rados.0.smithi115.stdout:1111: finishing copy_from to smithi11541303-35 2022-04-22T17:29:36.545 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 141 (ObjNum 1 snap 22075 seq_num 3178731984) dirty exists 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:1113: done (2 left) 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:1114: rollback oid 36 current snap is 117 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 90 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:1112: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 422 (ObjNum 206 snap 64 seq_num 206) dirty exists 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:1112: done (2 left) 2022-04-22T17:29:36.546 INFO:tasks.rados.rados.0.smithi115.stdout:1115: delete oid 11 current snap is 117 2022-04-22T17:29:36.547 INFO:tasks.rados.rados.0.smithi115.stdout:1111: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:29:36.549 INFO:tasks.rados.rados.0.smithi115.stdout:1114: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:29:36.549 INFO:tasks.rados.rados.0.smithi115.stdout:1114: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:29:36.550 INFO:tasks.rados.rados.0.smithi115.stdout:1111: done (2 left) 2022-04-22T17:29:36.550 INFO:tasks.rados.rados.0.smithi115.stdout:1115: done (1 left) 2022-04-22T17:29:36.550 INFO:tasks.rados.rados.0.smithi115.stdout:1116: copy_from oid 1 from oid 48 current snap is 117 2022-04-22T17:29:36.550 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 117 2022-04-22T17:29:36.550 INFO:tasks.rados.rados.0.smithi115.stdout:1117: seq_num 334 ranges {3948544=802816} 2022-04-22T17:29:36.576 INFO:tasks.rados.rados.0.smithi115.stdout:1117: writing smithi11541303-5 from 3948544 to 4751360 tid 1 2022-04-22T17:29:36.576 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 117 2022-04-22T17:29:36.577 INFO:tasks.rados.rados.0.smithi115.stdout:1118: seq_num 335 ranges {0=417792,417792=401408} 2022-04-22T17:29:36.579 INFO:tasks.rados.rados.0.smithi115.stdout:1118: writing smithi11541303-29 from 0 to 417792 tid 1 2022-04-22T17:29:36.581 INFO:tasks.rados.rados.0.smithi115.stdout:1118: writing smithi11541303-29 from 417792 to 819200 tid 2 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:1119: rollback oid 13 current snap is 117 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:1114: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 393 (ObjNum 264 snap 88 seq_num 264) dirty exists 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:1116: finishing copy_from to smithi11541303-1 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 240 (ObjNum 285 snap 93 seq_num 285) dirty exists 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:1116: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 99 2022-04-22T17:29:36.582 INFO:tasks.rados.rados.0.smithi115.stdout:1114: done (4 left) 2022-04-22T17:29:36.583 INFO:tasks.rados.rados.0.smithi115.stdout:1116: done (3 left) 2022-04-22T17:29:36.583 INFO:tasks.rados.rados.0.smithi115.stdout:1120: read oid 10 snap -1 2022-04-22T17:29:36.583 INFO:tasks.rados.rados.0.smithi115.stdout:1120: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:36.583 INFO:tasks.rados.rados.0.smithi115.stdout:1121: snap_remove snap 92 2022-04-22T17:29:36.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: cluster 2022-04-22T17:29:35.147751+0000 mgr.y (mgr.34107) 74 : cluster [DBG] pgmap v107: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: cluster 2022-04-22T17:29:35.519379+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2022-04-22T17:29:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: audit 2022-04-22T17:29:35.839603+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: cephadm 2022-04-22T17:29:35.841316+0000 mgr.y (mgr.34107) 75 : cephadm [INF] Reconfiguring osd.3 (monmap changed)... 2022-04-22T17:29:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: cephadm 2022-04-22T17:29:35.842417+0000 mgr.y (mgr.34107) 76 : cephadm [INF] Reconfiguring daemon osd.3 on smithi115 2022-04-22T17:29:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:36 smithi115 bash[46934]: cluster 2022-04-22T17:29:36.519036+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2022-04-22T17:29:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: cluster 2022-04-22T17:29:35.147751+0000 mgr.y (mgr.34107) 74 : cluster [DBG] pgmap v107: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: cluster 2022-04-22T17:29:35.519379+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2022-04-22T17:29:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: audit 2022-04-22T17:29:35.839603+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: cephadm 2022-04-22T17:29:35.841316+0000 mgr.y (mgr.34107) 75 : cephadm [INF] Reconfiguring osd.3 (monmap changed)... 2022-04-22T17:29:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: cephadm 2022-04-22T17:29:35.842417+0000 mgr.y (mgr.34107) 76 : cephadm [INF] Reconfiguring daemon osd.3 on smithi115 2022-04-22T17:29:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:36 smithi121 bash[36902]: cluster 2022-04-22T17:29:36.519036+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2022-04-22T17:29:37.550 INFO:tasks.rados.rados.0.smithi115.stdout:1121: done (4 left) 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1122: delete oid 41 current snap is 117 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1118: finishing write tid 1 to smithi11541303-29 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1118: finishing write tid 2 to smithi11541303-29 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1117: finishing write tid 1 to smithi11541303-5 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1117: finishing write tid 2 to smithi11541303-5 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1119: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1118: finishing write tid 3 to smithi11541303-29 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1119: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:29:37.551 INFO:tasks.rados.rados.0.smithi115.stdout:1119: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 426 (ObjNum 298 snap 99 seq_num 298) dirty exists 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:1118: finishing write tid 4 to smithi11541303-29 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 250 (ObjNum 335 snap 117 seq_num 335) dirty exists 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:1118: left oid 29 (ObjNum 335 snap 117 seq_num 335) 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:1117: finishing write tid 3 to smithi11541303-5 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 617 (ObjNum 334 snap 117 seq_num 334) dirty exists 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:1117: left oid 5 (ObjNum 334 snap 117 seq_num 334) 2022-04-22T17:29:37.552 INFO:tasks.rados.rados.0.smithi115.stdout:1120: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:38.366 INFO:tasks.rados.rados.0.smithi115.stdout:1117: done (4 left) 2022-04-22T17:29:38.437 INFO:tasks.rados.rados.0.smithi115.stdout:1118: done (3 left) 2022-04-22T17:29:38.437 INFO:tasks.rados.rados.0.smithi115.stdout:1119: done (2 left) 2022-04-22T17:29:38.437 INFO:tasks.rados.rados.0.smithi115.stdout:1120: done (1 left) 2022-04-22T17:29:38.438 INFO:tasks.rados.rados.0.smithi115.stdout:1122: done (0 left) 2022-04-22T17:29:38.438 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 117 2022-04-22T17:29:38.438 INFO:tasks.rados.rados.0.smithi115.stdout:1123: seq_num 337 ranges {4038656=450560,4489216=401408} 2022-04-22T17:29:38.439 INFO:tasks.rados.rados.0.smithi115.stdout:1123: writing smithi11541303-19 from 4038656 to 4489216 tid 1 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: writing smithi11541303-19 from 4489216 to 4890624 tid 2 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1124: snap_create 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: finishing write tid 1 to smithi11541303-19 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: finishing write tid 2 to smithi11541303-19 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: finishing write tid 3 to smithi11541303-19 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: finishing write tid 4 to smithi11541303-19 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 460 (ObjNum 337 snap 117 seq_num 337) dirty exists 2022-04-22T17:29:38.440 INFO:tasks.rados.rados.0.smithi115.stdout:1123: left oid 19 (ObjNum 337 snap 117 seq_num 337) 2022-04-22T17:29:38.554 INFO:tasks.rados.rados.0.smithi115.stdout:1123: done (1 left) 2022-04-22T17:29:38.554 INFO:tasks.rados.rados.0.smithi115.stdout:1124: done (0 left) 2022-04-22T17:29:38.554 INFO:tasks.rados.rados.0.smithi115.stdout:1125: snap_create 2022-04-22T17:29:38.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[46934]: cluster 2022-04-22T17:29:37.148461+0000 mgr.y (mgr.34107) 77 : cluster [DBG] pgmap v110: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:38.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[46934]: cluster 2022-04-22T17:29:37.536295+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2022-04-22T17:29:38.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[46934]: audit 2022-04-22T17:29:38.389974+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:38.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[46934]: audit 2022-04-22T17:29:38.392250+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:38.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[46934]: audit 2022-04-22T17:29:38.393391+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:38.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[45495]: cluster 2022-04-22T17:29:37.148461+0000 mgr.y (mgr.34107) 77 : cluster [DBG] pgmap v110: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:38.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[45495]: cluster 2022-04-22T17:29:37.536295+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2022-04-22T17:29:38.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[45495]: audit 2022-04-22T17:29:38.389974+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:38.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[45495]: audit 2022-04-22T17:29:38.392250+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:38.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:38 smithi115 bash[45495]: audit 2022-04-22T17:29:38.393391+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:38 smithi121 bash[36902]: cluster 2022-04-22T17:29:37.148461+0000 mgr.y (mgr.34107) 77 : cluster [DBG] pgmap v110: 121 pgs: 121 active+clean; 228 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:29:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:38 smithi121 bash[36902]: cluster 2022-04-22T17:29:37.536295+0000 mon.a (mon.0) 227 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2022-04-22T17:29:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:38 smithi121 bash[36902]: audit 2022-04-22T17:29:38.389974+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:38 smithi121 bash[36902]: audit 2022-04-22T17:29:38.392250+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:38 smithi121 bash[36902]: audit 2022-04-22T17:29:38.393391+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:39.554 INFO:tasks.rados.rados.0.smithi115.stdout:1125: done (0 left) 2022-04-22T17:29:39.554 INFO:tasks.rados.rados.0.smithi115.stdout:1126: rollback oid 44 current snap is 119 2022-04-22T17:29:39.554 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 106 2022-04-22T17:29:39.555 INFO:tasks.rados.rados.0.smithi115.stdout:1127: read oid 36 snap -1 2022-04-22T17:29:39.555 INFO:tasks.rados.rados.0.smithi115.stdout:1127: expect (ObjNum 264 snap 88 seq_num 264) 2022-04-22T17:29:39.555 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 119 2022-04-22T17:29:39.555 INFO:tasks.rados.rados.0.smithi115.stdout:1128: seq_num 338 ranges {2670592=466944,3137536=401408} 2022-04-22T17:29:39.575 INFO:tasks.rados.rados.0.smithi115.stdout:1128: writing smithi11541303-1 from 2670592 to 3137536 tid 1 2022-04-22T17:29:39.577 INFO:tasks.rados.rados.0.smithi115.stdout:1128: writing smithi11541303-1 from 3137536 to 3538944 tid 2 2022-04-22T17:29:39.578 INFO:tasks.rados.rados.0.smithi115.stdout:1129: delete oid 50 current snap is 119 2022-04-22T17:29:39.584 INFO:tasks.rados.rados.0.smithi115.stdout:1127: expect (ObjNum 264 snap 88 seq_num 264) 2022-04-22T17:29:39.641 INFO:tasks.rados.rados.0.smithi115.stdout:1126: finishing rollback tid 0 to smithi11541303-44 2022-04-22T17:29:39.641 INFO:tasks.rados.rados.0.smithi115.stdout:1126: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1126: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 29 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1128: finishing write tid 1 to smithi11541303-1 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1128: finishing write tid 2 to smithi11541303-1 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1128: finishing write tid 3 to smithi11541303-1 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1128: finishing write tid 4 to smithi11541303-1 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 396 (ObjNum 338 snap 119 seq_num 338) dirty exists 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1128: left oid 1 (ObjNum 338 snap 119 seq_num 338) 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1126: done (3 left) 2022-04-22T17:29:39.642 INFO:tasks.rados.rados.0.smithi115.stdout:1127: done (2 left) 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1128: done (1 left) 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1129: done (0 left) 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1130: read oid 15 snap -1 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1130: expect (ObjNum 307 snap 101 seq_num 307) 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1131: rmattr oid 49 current snap is 119 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1131: done (1 left) 2022-04-22T17:29:39.643 INFO:tasks.rados.rados.0.smithi115.stdout:1132: delete oid 4 current snap is 119 2022-04-22T17:29:39.647 INFO:tasks.rados.rados.0.smithi115.stdout:1132: done (1 left) 2022-04-22T17:29:39.647 INFO:tasks.rados.rados.0.smithi115.stdout:1133: snap_create 2022-04-22T17:29:39.660 INFO:tasks.rados.rados.0.smithi115.stdout:1130: expect (ObjNum 307 snap 101 seq_num 307) 2022-04-22T17:29:39.933 INFO:tasks.rados.rados.0.smithi115.stdout:1130: done (1 left) 2022-04-22T17:29:39.934 INFO:tasks.rados.rados.0.smithi115.stdout:1133: done (0 left) 2022-04-22T17:29:39.934 INFO:tasks.rados.rados.0.smithi115.stdout:1134: read oid 10 snap -1 2022-04-22T17:29:39.934 INFO:tasks.rados.rados.0.smithi115.stdout:1134: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:39.934 INFO:tasks.rados.rados.0.smithi115.stdout:1135: delete oid 19 current snap is 120 2022-04-22T17:29:39.939 INFO:tasks.rados.rados.0.smithi115.stdout:1135: done (1 left) 2022-04-22T17:29:39.939 INFO:tasks.rados.rados.0.smithi115.stdout:1136: delete oid 5 current snap is 120 2022-04-22T17:29:39.944 INFO:tasks.rados.rados.0.smithi115.stdout:1136: done (1 left) 2022-04-22T17:29:39.945 INFO:tasks.rados.rados.0.smithi115.stdout:1137: read oid 9 snap -1 2022-04-22T17:29:39.945 INFO:tasks.rados.rados.0.smithi115.stdout:1137: expect deleted 2022-04-22T17:29:39.945 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 120 2022-04-22T17:29:39.945 INFO:tasks.rados.rados.0.smithi115.stdout:1138: seq_num 343 ranges {0=548864,548864=401408} 2022-04-22T17:29:39.947 INFO:tasks.rados.rados.0.smithi115.stdout:1138: writing smithi11541303-50 from 0 to 548864 tid 1 2022-04-22T17:29:39.950 INFO:tasks.rados.rados.0.smithi115.stdout:1138: writing smithi11541303-50 from 548864 to 950272 tid 2 2022-04-22T17:29:39.951 INFO:tasks.rados.rados.0.smithi115.stdout:1137: done (2 left) 2022-04-22T17:29:39.951 INFO:tasks.rados.rados.0.smithi115.stdout:1139: rmattr oid 21 current snap is 120 2022-04-22T17:29:39.951 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 120 2022-04-22T17:29:39.951 INFO:tasks.rados.rados.0.smithi115.stdout:1140: seq_num 344 ranges {5251072=565248,5816320=401408} 2022-04-22T17:29:39.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[45495]: cephadm 2022-04-22T17:29:38.391883+0000 mgr.y (mgr.34107) 78 : cephadm [INF] Reconfiguring mds.a (monmap changed)... 2022-04-22T17:29:39.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[45495]: cephadm 2022-04-22T17:29:38.394388+0000 mgr.y (mgr.34107) 79 : cephadm [INF] Reconfiguring daemon mds.a on smithi115 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[45495]: cluster 2022-04-22T17:29:38.539015+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[45495]: audit 2022-04-22T17:29:39.159601+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[45495]: cluster 2022-04-22T17:29:39.540661+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[46934]: cephadm 2022-04-22T17:29:38.391883+0000 mgr.y (mgr.34107) 78 : cephadm [INF] Reconfiguring mds.a (monmap changed)... 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[46934]: cephadm 2022-04-22T17:29:38.394388+0000 mgr.y (mgr.34107) 79 : cephadm [INF] Reconfiguring daemon mds.a on smithi115 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[46934]: cluster 2022-04-22T17:29:38.539015+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2022-04-22T17:29:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[46934]: audit 2022-04-22T17:29:39.159601+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:39 smithi115 bash[46934]: cluster 2022-04-22T17:29:39.540661+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2022-04-22T17:29:39.987 INFO:tasks.rados.rados.0.smithi115.stdout:1140: writing smithi11541303-24 from 5251072 to 5816320 tid 1 2022-04-22T17:29:39.990 INFO:tasks.rados.rados.0.smithi115.stdout:1140: writing smithi11541303-24 from 5816320 to 6217728 tid 2 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 120 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:1138: finishing write tid 1 to smithi11541303-50 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 315 (ObjNum 247 snap 83 seq_num 247) dirty exists 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:1138: finishing write tid 2 to smithi11541303-50 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:1138: finishing write tid 3 to smithi11541303-50 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:1138: finishing write tid 4 to smithi11541303-50 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 377 (ObjNum 343 snap 120 seq_num 343) dirty exists 2022-04-22T17:29:39.991 INFO:tasks.rados.rados.0.smithi115.stdout:1138: left oid 50 (ObjNum 343 snap 120 seq_num 343) 2022-04-22T17:29:39.992 INFO:tasks.rados.rados.0.smithi115.stdout:1141: seq_num 345 ranges {172032=581632,753664=401408} 2022-04-22T17:29:39.996 INFO:tasks.rados.rados.0.smithi115.stdout:1141: writing smithi11541303-6 from 172032 to 753664 tid 1 2022-04-22T17:29:39.999 INFO:tasks.rados.rados.0.smithi115.stdout:1141: writing smithi11541303-6 from 753664 to 1155072 tid 2 2022-04-22T17:29:39.999 INFO:tasks.rados.rados.0.smithi115.stdout:1138: done (4 left) 2022-04-22T17:29:39.999 INFO:tasks.rados.rados.0.smithi115.stdout:1139: done (3 left) 2022-04-22T17:29:40.001 INFO:tasks.rados.rados.0.smithi115.stdout:1142: snap_remove snap 119 2022-04-22T17:29:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:39 smithi121 bash[36902]: cephadm 2022-04-22T17:29:38.391883+0000 mgr.y (mgr.34107) 78 : cephadm [INF] Reconfiguring mds.a (monmap changed)... 2022-04-22T17:29:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:39 smithi121 bash[36902]: cephadm 2022-04-22T17:29:38.394388+0000 mgr.y (mgr.34107) 79 : cephadm [INF] Reconfiguring daemon mds.a on smithi115 2022-04-22T17:29:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:39 smithi121 bash[36902]: cluster 2022-04-22T17:29:38.539015+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2022-04-22T17:29:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:39 smithi121 bash[36902]: audit 2022-04-22T17:29:39.159601+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:39 smithi121 bash[36902]: cluster 2022-04-22T17:29:39.540661+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2022-04-22T17:29:40.763 INFO:tasks.rados.rados.0.smithi115.stdout:1142: done (3 left) 2022-04-22T17:29:40.763 INFO:tasks.rados.rados.0.smithi115.stdout:1143: snap_remove snap 99 2022-04-22T17:29:40.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: cluster 2022-04-22T17:29:39.149128+0000 mgr.y (mgr.34107) 80 : cluster [DBG] pgmap v113: 121 pgs: 121 active+clean; 238 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 196 KiB/s wr, 4 op/s 2022-04-22T17:29:40.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: cluster 2022-04-22T17:29:39.738552+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2022-04-22T17:29:40.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: audit 2022-04-22T17:29:40.587608+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:40.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: audit 2022-04-22T17:29:40.590339+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:40.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: audit 2022-04-22T17:29:40.591864+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:40.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:40 smithi121 bash[36902]: audit 2022-04-22T17:29:40.593291+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:41.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: cluster 2022-04-22T17:29:39.149128+0000 mgr.y (mgr.34107) 80 : cluster [DBG] pgmap v113: 121 pgs: 121 active+clean; 238 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 196 KiB/s wr, 4 op/s 2022-04-22T17:29:41.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: cluster 2022-04-22T17:29:39.738552+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2022-04-22T17:29:41.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: audit 2022-04-22T17:29:40.587608+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:41.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: audit 2022-04-22T17:29:40.590339+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:41.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: audit 2022-04-22T17:29:40.591864+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[45495]: audit 2022-04-22T17:29:40.593291+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: cluster 2022-04-22T17:29:39.149128+0000 mgr.y (mgr.34107) 80 : cluster [DBG] pgmap v113: 121 pgs: 121 active+clean; 238 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 196 KiB/s wr, 4 op/s 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: cluster 2022-04-22T17:29:39.738552+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: audit 2022-04-22T17:29:40.587608+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: audit 2022-04-22T17:29:40.590339+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: audit 2022-04-22T17:29:40.591864+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-04-22T17:29:41.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:40 smithi115 bash[46934]: audit 2022-04-22T17:29:40.593291+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:41.774 INFO:tasks.rados.rados.0.smithi115.stdout:1143: done (3 left) 2022-04-22T17:29:41.774 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 120 2022-04-22T17:29:41.775 INFO:tasks.rados.rados.0.smithi115.stdout:1134: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:42.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[45495]: cephadm 2022-04-22T17:29:40.589811+0000 mgr.y (mgr.34107) 81 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:29:42.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[45495]: cephadm 2022-04-22T17:29:40.594779+0000 mgr.y (mgr.34107) 82 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:29:42.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[45495]: cluster 2022-04-22T17:29:40.748421+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2022-04-22T17:29:42.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[46934]: cephadm 2022-04-22T17:29:40.589811+0000 mgr.y (mgr.34107) 81 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:29:42.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[46934]: cephadm 2022-04-22T17:29:40.594779+0000 mgr.y (mgr.34107) 82 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:29:42.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[46934]: cluster 2022-04-22T17:29:40.748421+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2022-04-22T17:29:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:41.619Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:41.619Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:41.619Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:42.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:41 smithi121 bash[36902]: cephadm 2022-04-22T17:29:40.589811+0000 mgr.y (mgr.34107) 81 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2022-04-22T17:29:42.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:41 smithi121 bash[36902]: cephadm 2022-04-22T17:29:40.594779+0000 mgr.y (mgr.34107) 82 : cephadm [INF] Reconfiguring daemon mon.b on smithi121 2022-04-22T17:29:42.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:41 smithi121 bash[36902]: cluster 2022-04-22T17:29:40.748421+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2022-04-22T17:29:42.586 INFO:tasks.rados.rados.0.smithi115.stdout:1140: finishing write tid 1 to smithi11541303-24 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1140: finishing write tid 2 to smithi11541303-24 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1140: finishing write tid 3 to smithi11541303-24 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1140: finishing write tid 4 to smithi11541303-24 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 174 (ObjNum 344 snap 120 seq_num 344) dirty exists 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1140: left oid 24 (ObjNum 344 snap 120 seq_num 344) 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1141: finishing write tid 1 to smithi11541303-6 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1141: finishing write tid 2 to smithi11541303-6 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1141: finishing write tid 3 to smithi11541303-6 2022-04-22T17:29:42.587 INFO:tasks.rados.rados.0.smithi115.stdout:1141: finishing write tid 4 to smithi11541303-6 2022-04-22T17:29:42.588 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 178 (ObjNum 345 snap 120 seq_num 345) dirty exists 2022-04-22T17:29:42.588 INFO:tasks.rados.rados.0.smithi115.stdout:1141: left oid 6 (ObjNum 345 snap 120 seq_num 345) 2022-04-22T17:29:42.588 INFO:tasks.rados.rados.0.smithi115.stdout:1144: seq_num 346 ranges {950272=598016,1548288=401408} 2022-04-22T17:29:42.594 INFO:tasks.rados.rados.0.smithi115.stdout:1144: writing smithi11541303-50 from 950272 to 1548288 tid 1 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1144: writing smithi11541303-50 from 1548288 to 1949696 tid 2 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1134: done (3 left) 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1140: done (2 left) 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1141: done (1 left) 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1145: read oid 40 snap 111 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1145: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1146: copy_from oid 21 from oid 2 current snap is 120 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1147: copy_from oid 35 from oid 44 current snap is 120 2022-04-22T17:29:42.597 INFO:tasks.rados.rados.0.smithi115.stdout:1148: snap_create 2022-04-22T17:29:42.603 INFO:tasks.rados.rados.0.smithi115.stdout:1144: finishing write tid 1 to smithi11541303-50 2022-04-22T17:29:42.603 INFO:tasks.rados.rados.0.smithi115.stdout:1147: finishing copy_from to smithi11541303-35 2022-04-22T17:29:42.603 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 29 (ObjNum 1 snap 32692 seq_num 3178731584) dirty exists 2022-04-22T17:29:42.604 INFO:tasks.rados.rados.0.smithi115.stdout:1144: finishing write tid 2 to smithi11541303-50 2022-04-22T17:29:42.604 INFO:tasks.rados.rados.0.smithi115.stdout:1147: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:29:42.604 INFO:tasks.rados.rados.0.smithi115.stdout:1144: finishing write tid 3 to smithi11541303-50 2022-04-22T17:29:42.606 INFO:tasks.rados.rados.0.smithi115.stdout:1144: finishing write tid 4 to smithi11541303-50 2022-04-22T17:29:42.606 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 380 (ObjNum 346 snap 120 seq_num 346) dirty exists 2022-04-22T17:29:42.606 INFO:tasks.rados.rados.0.smithi115.stdout:1144: left oid 50 (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:29:42.612 INFO:tasks.rados.rados.0.smithi115.stdout:1146: finishing copy_from to smithi11541303-21 2022-04-22T17:29:42.612 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 519 (ObjNum 277 snap 92 seq_num 277) dirty exists 2022-04-22T17:29:42.613 INFO:tasks.rados.rados.0.smithi115.stdout:1146: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:29:42.621 INFO:tasks.rados.rados.0.smithi115.stdout:1145: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:29:42.840 INFO:tasks.rados.rados.0.smithi115.stdout:1144: done (4 left) 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1145: done (3 left) 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1146: done (2 left) 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1147: done (1 left) 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1148: done (0 left) 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1149: rollback oid 26 current snap is 121 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 95 2022-04-22T17:29:42.841 INFO:tasks.rados.rados.0.smithi115.stdout:1150: rmattr oid 24 current snap is 121 2022-04-22T17:29:42.842 INFO:tasks.rados.rados.0.smithi115.stdout:1150: done (1 left) 2022-04-22T17:29:42.842 INFO:tasks.rados.rados.0.smithi115.stdout:1151: setattr oid 27 current snap is 121 2022-04-22T17:29:42.842 INFO:tasks.rados.rados.0.smithi115.stdout:1152: snap_remove snap 105 2022-04-22T17:29:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:42 smithi121 bash[36902]: cluster 2022-04-22T17:29:41.149841+0000 mgr.y (mgr.34107) 83 : cluster [DBG] pgmap v117: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2022-04-22T17:29:43.050 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:42 smithi121 bash[36902]: cluster 2022-04-22T17:29:41.759314+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2022-04-22T17:29:43.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:42 smithi115 bash[45495]: cluster 2022-04-22T17:29:41.149841+0000 mgr.y (mgr.34107) 83 : cluster [DBG] pgmap v117: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2022-04-22T17:29:43.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:42 smithi115 bash[45495]: cluster 2022-04-22T17:29:41.759314+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2022-04-22T17:29:43.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:42 smithi115 bash[46934]: cluster 2022-04-22T17:29:41.149841+0000 mgr.y (mgr.34107) 83 : cluster [DBG] pgmap v117: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2022-04-22T17:29:43.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:42 smithi115 bash[46934]: cluster 2022-04-22T17:29:41.759314+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2022-04-22T17:29:43.249 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:29:43.590 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:29:43.590 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 61s ago 6m 12.9M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:29:43.590 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6m) 32s ago 6m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:29:43.590 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (7m) 61s ago 7m 13.2M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (7m) 32s ago 7m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2m) 32s ago 11m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2m) 61s ago 14m 438M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (100s) 61s ago 14m 37.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (44s) 32s ago 12m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (67s) 61s ago 12m 25.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 61s ago 6m 9039k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6m) 32s ago 6m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:29:43.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (11m) 61s ago 11m 151M 2526M 16.2.5 46cf6318c64e 90bfe612abdf 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 61s ago 10m 161M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 61s ago 10m 129M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 61s ago 9m 90.3M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (9m) 32s ago 9m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (9m) 32s ago 9m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (8m) 32s ago 8m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (8m) 32s ago 8m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:29:43.592 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 32s ago 6m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:29:43.796 INFO:tasks.rados.rados.0.smithi115.stdout:1152: done (2 left) 2022-04-22T17:29:43.796 INFO:tasks.rados.rados.0.smithi115.stdout:1153: rmattr oid 50 current snap is 121 2022-04-22T17:29:43.796 INFO:tasks.rados.rados.0.smithi115.stdout:1153: done (2 left) 2022-04-22T17:29:43.796 INFO:tasks.rados.rados.0.smithi115.stdout:1154: snap_remove snap 115 2022-04-22T17:29:43.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:43.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:43.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:43.514Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:43.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:43.514Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:29:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[46934]: cluster 2022-04-22T17:29:42.763791+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2022-04-22T17:29:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[46934]: audit 2022-04-22T17:29:43.136193+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[46934]: audit 2022-04-22T17:29:43.137894+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[46934]: audit 2022-04-22T17:29:43.139205+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[46934]: audit 2022-04-22T17:29:43.140205+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:44.008 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:29:44.008 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:29:44.008 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:29:44.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:44.008 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 8 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:29:44.009 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:29:44.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 10, 2022-04-22T17:29:44.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 5 2022-04-22T17:29:44.010 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:29:44.010 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:29:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[45495]: cluster 2022-04-22T17:29:42.763791+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2022-04-22T17:29:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[45495]: audit 2022-04-22T17:29:43.136193+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[45495]: audit 2022-04-22T17:29:43.137894+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[45495]: audit 2022-04-22T17:29:43.139205+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:43 smithi115 bash[45495]: audit 2022-04-22T17:29:43.140205+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:43 smithi121 bash[36902]: cluster 2022-04-22T17:29:42.763791+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2022-04-22T17:29:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:43 smithi121 bash[36902]: audit 2022-04-22T17:29:43.136193+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:44.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:43 smithi121 bash[36902]: audit 2022-04-22T17:29:43.137894+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-04-22T17:29:44.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:43 smithi121 bash[36902]: audit 2022-04-22T17:29:43.139205+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2022-04-22T17:29:44.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:43 smithi121 bash[36902]: audit 2022-04-22T17:29:43.140205+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:1154: done (2 left) 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 121 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:1149: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:1149: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 536 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 591 (ObjNum -1116008432 snap 22075 seq_num 0) dirty exists 2022-04-22T17:29:44.756 INFO:tasks.rados.rados.0.smithi115.stdout:1155: seq_num 347 ranges {2383872=614400,2998272=401408} 2022-04-22T17:29:44.775 INFO:tasks.rados.rados.0.smithi115.stdout:1155: writing smithi11541303-28 from 2383872 to 2998272 tid 1 2022-04-22T17:29:44.778 INFO:tasks.rados.rados.0.smithi115.stdout:1155: writing smithi11541303-28 from 2998272 to 3399680 tid 2 2022-04-22T17:29:44.778 INFO:tasks.rados.rados.0.smithi115.stdout:1149: done (2 left) 2022-04-22T17:29:44.778 INFO:tasks.rados.rados.0.smithi115.stdout:1151: done (1 left) 2022-04-22T17:29:44.778 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 121 2022-04-22T17:29:44.778 INFO:tasks.rados.rados.0.smithi115.stdout:1156: seq_num 348 ranges {0=630784,630784=401408} 2022-04-22T17:29:44.782 INFO:tasks.rados.rados.0.smithi115.stdout:1156: writing smithi11541303-3 from 0 to 630784 tid 1 2022-04-22T17:29:44.785 INFO:tasks.rados.rados.0.smithi115.stdout:1156: writing smithi11541303-3 from 630784 to 1032192 tid 2 2022-04-22T17:29:44.785 INFO:tasks.rados.rados.0.smithi115.stdout:1157: rollback oid 12 current snap is 121 2022-04-22T17:29:44.785 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 97 2022-04-22T17:29:44.785 INFO:tasks.rados.rados.0.smithi115.stdout:1155: finishing write tid 1 to smithi11541303-28 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1155: finishing write tid 2 to smithi11541303-28 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1155: finishing write tid 3 to smithi11541303-28 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1155: finishing write tid 4 to smithi11541303-28 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 260 (ObjNum 347 snap 121 seq_num 347) dirty exists 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1155: left oid 28 (ObjNum 347 snap 121 seq_num 347) 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1155: done (2 left) 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1158: rollback oid 25 current snap is 121 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 112 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1159: rmattr oid 46 current snap is 121 2022-04-22T17:29:44.786 INFO:tasks.rados.rados.0.smithi115.stdout:1160: rmattr oid 18 current snap is 121 2022-04-22T17:29:44.787 INFO:tasks.rados.rados.0.smithi115.stdout:1160: done (4 left) 2022-04-22T17:29:44.787 INFO:tasks.rados.rados.0.smithi115.stdout:1161: delete oid 13 current snap is 121 2022-04-22T17:29:44.790 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 0 (ObjNum 306 snap 101 seq_num 306) dirty exists 2022-04-22T17:29:44.790 INFO:tasks.rados.rados.0.smithi115.stdout:1156: finishing write tid 1 to smithi11541303-3 2022-04-22T17:29:44.790 INFO:tasks.rados.rados.0.smithi115.stdout:1156: finishing write tid 2 to smithi11541303-3 2022-04-22T17:29:44.791 INFO:tasks.rados.rados.0.smithi115.stdout:1156: finishing write tid 3 to smithi11541303-3 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1157: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1158: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1158: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1157: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1158: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 446 (ObjNum 270 snap 89 seq_num 270) dirty exists 2022-04-22T17:29:44.793 INFO:tasks.rados.rados.0.smithi115.stdout:1158: done (4 left) 2022-04-22T17:29:44.794 INFO:tasks.rados.rados.0.smithi115.stdout:1159: done (3 left) 2022-04-22T17:29:44.794 INFO:tasks.rados.rados.0.smithi115.stdout:1161: done (2 left) 2022-04-22T17:29:44.794 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 121 2022-04-22T17:29:44.794 INFO:tasks.rados.rados.0.smithi115.stdout:1162: seq_num 350 ranges {565248=671744,1236992=401408} 2022-04-22T17:29:44.801 INFO:tasks.rados.rados.0.smithi115.stdout:1162: writing smithi11541303-47 from 565248 to 1236992 tid 1 2022-04-22T17:29:44.804 INFO:tasks.rados.rados.0.smithi115.stdout:1162: writing smithi11541303-47 from 1236992 to 1638400 tid 2 2022-04-22T17:29:44.804 INFO:tasks.rados.rados.0.smithi115.stdout:1163: snap_remove snap 91 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cephadm 2022-04-22T17:29:43.137448+0000 mgr.y (mgr.34107) 84 : cephadm [INF] Reconfiguring mgr.x (monmap changed)... 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cephadm 2022-04-22T17:29:43.141363+0000 mgr.y (mgr.34107) 85 : cephadm [INF] Reconfiguring daemon mgr.x on smithi121 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cluster 2022-04-22T17:29:43.150480+0000 mgr.y (mgr.34107) 86 : cluster [DBG] pgmap v120: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: audit 2022-04-22T17:29:43.219380+0000 mgr.y (mgr.34107) 87 : audit [DBG] from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: audit 2022-04-22T17:29:43.562670+0000 mgr.y (mgr.34107) 88 : audit [DBG] from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cluster 2022-04-22T17:29:43.773652+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: audit 2022-04-22T17:29:43.994357+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.115:0/895278883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:44.908 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cluster 2022-04-22T17:29:44.155806+0000 mon.a (mon.0) 247 : cluster [DBG] mgrmap e30: y(active, since 91s), standbys: x 2022-04-22T17:29:44.909 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: audit 2022-04-22T17:29:44.161939+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:44.909 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:44 smithi121 bash[36902]: cluster 2022-04-22T17:29:44.741691+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cephadm 2022-04-22T17:29:43.137448+0000 mgr.y (mgr.34107) 84 : cephadm [INF] Reconfiguring mgr.x (monmap changed)... 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cephadm 2022-04-22T17:29:43.141363+0000 mgr.y (mgr.34107) 85 : cephadm [INF] Reconfiguring daemon mgr.x on smithi121 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cluster 2022-04-22T17:29:43.150480+0000 mgr.y (mgr.34107) 86 : cluster [DBG] pgmap v120: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: audit 2022-04-22T17:29:43.219380+0000 mgr.y (mgr.34107) 87 : audit [DBG] from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: audit 2022-04-22T17:29:43.562670+0000 mgr.y (mgr.34107) 88 : audit [DBG] from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:45.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cluster 2022-04-22T17:29:43.773652+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: audit 2022-04-22T17:29:43.994357+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.115:0/895278883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cluster 2022-04-22T17:29:44.155806+0000 mon.a (mon.0) 247 : cluster [DBG] mgrmap e30: y(active, since 91s), standbys: x 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: audit 2022-04-22T17:29:44.161939+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[45495]: cluster 2022-04-22T17:29:44.741691+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cephadm 2022-04-22T17:29:43.137448+0000 mgr.y (mgr.34107) 84 : cephadm [INF] Reconfiguring mgr.x (monmap changed)... 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cephadm 2022-04-22T17:29:43.141363+0000 mgr.y (mgr.34107) 85 : cephadm [INF] Reconfiguring daemon mgr.x on smithi121 2022-04-22T17:29:45.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cluster 2022-04-22T17:29:43.150480+0000 mgr.y (mgr.34107) 86 : cluster [DBG] pgmap v120: 121 pgs: 121 active+clean; 240 MiB data, 949 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: audit 2022-04-22T17:29:43.219380+0000 mgr.y (mgr.34107) 87 : audit [DBG] from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: audit 2022-04-22T17:29:43.562670+0000 mgr.y (mgr.34107) 88 : audit [DBG] from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cluster 2022-04-22T17:29:43.773652+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: audit 2022-04-22T17:29:43.994357+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.115:0/895278883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cluster 2022-04-22T17:29:44.155806+0000 mon.a (mon.0) 247 : cluster [DBG] mgrmap e30: y(active, since 91s), standbys: x 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: audit 2022-04-22T17:29:44.161939+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:45.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:44 smithi115 bash[46934]: cluster 2022-04-22T17:29:44.741691+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2022-04-22T17:29:45.812 INFO:tasks.rados.rados.0.smithi115.stdout:1163: done (3 left) 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1164: read oid 30 snap -1 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1164: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1165: copy_from oid 25 from oid 43 current snap is 121 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1166: delete oid 36 current snap is 121 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1157: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 464 (ObjNum 282 snap 93 seq_num 282) dirty exists 2022-04-22T17:29:45.813 INFO:tasks.rados.rados.0.smithi115.stdout:1156: finishing write tid 4 to smithi11541303-3 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 520 (ObjNum 348 snap 121 seq_num 348) dirty exists 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1156: left oid 3 (ObjNum 348 snap 121 seq_num 348) 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1162: finishing write tid 1 to smithi11541303-47 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1162: finishing write tid 2 to smithi11541303-47 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1162: finishing write tid 3 to smithi11541303-47 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1162: finishing write tid 4 to smithi11541303-47 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 394 (ObjNum 350 snap 121 seq_num 350) dirty exists 2022-04-22T17:29:45.814 INFO:tasks.rados.rados.0.smithi115.stdout:1162: left oid 47 (ObjNum 350 snap 121 seq_num 350) 2022-04-22T17:29:45.822 INFO:tasks.rados.rados.0.smithi115.stdout:1156: done (5 left) 2022-04-22T17:29:45.822 INFO:tasks.rados.rados.0.smithi115.stdout:1157: done (4 left) 2022-04-22T17:29:45.823 INFO:tasks.rados.rados.0.smithi115.stdout:1162: done (3 left) 2022-04-22T17:29:45.823 INFO:tasks.rados.rados.0.smithi115.stdout:1166: done (2 left) 2022-04-22T17:29:45.823 INFO:tasks.rados.rados.0.smithi115.stdout:1167: copy_from oid 23 from oid 26 current snap is 121 2022-04-22T17:29:45.823 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 121 2022-04-22T17:29:45.824 INFO:tasks.rados.rados.0.smithi115.stdout:1168: seq_num 352 ranges {3956736=704512,4661248=401408} 2022-04-22T17:29:45.850 INFO:tasks.rados.rados.0.smithi115.stdout:1168: writing smithi11541303-40 from 3956736 to 4661248 tid 1 2022-04-22T17:29:45.852 INFO:tasks.rados.rados.0.smithi115.stdout:1168: writing smithi11541303-40 from 4661248 to 5062656 tid 2 2022-04-22T17:29:45.852 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 121 2022-04-22T17:29:45.852 INFO:tasks.rados.rados.0.smithi115.stdout:1169: seq_num 353 ranges {3973120=720896,4694016=401408} 2022-04-22T17:29:45.884 INFO:tasks.rados.rados.0.smithi115.stdout:1169: writing smithi11541303-37 from 3973120 to 4694016 tid 1 2022-04-22T17:29:45.886 INFO:tasks.rados.rados.0.smithi115.stdout:1169: writing smithi11541303-37 from 4694016 to 5095424 tid 2 2022-04-22T17:29:45.886 INFO:tasks.rados.rados.0.smithi115.stdout:1170: read oid 18 snap -1 2022-04-22T17:29:45.886 INFO:tasks.rados.rados.0.smithi115.stdout:1170: expect deleted 2022-04-22T17:29:45.886 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 121 2022-04-22T17:29:45.886 INFO:tasks.rados.rados.0.smithi115.stdout:1171: seq_num 354 ranges {0=737280,737280=401408} 2022-04-22T17:29:45.890 INFO:tasks.rados.rados.0.smithi115.stdout:1171: writing smithi11541303-33 from 0 to 737280 tid 1 2022-04-22T17:29:45.892 INFO:tasks.rados.rados.0.smithi115.stdout:1171: writing smithi11541303-33 from 737280 to 1138688 tid 2 2022-04-22T17:29:45.893 INFO:tasks.rados.rados.0.smithi115.stdout:1172: rollback oid 10 current snap is 121 2022-04-22T17:29:45.893 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 97 2022-04-22T17:29:45.893 INFO:tasks.rados.rados.0.smithi115.stdout:1164: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:29:45.901 INFO:tasks.rados.rados.0.smithi115.stdout:1165: finishing copy_from to smithi11541303-25 2022-04-22T17:29:45.901 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 375 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1165: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1164: done (7 left) 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1165: done (6 left) 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1173: copy_from oid 6 from oid 22 current snap is 121 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1167: finishing copy_from to smithi11541303-23 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 536 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1168: finishing write tid 1 to smithi11541303-40 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1168: finishing write tid 2 to smithi11541303-40 2022-04-22T17:29:45.902 INFO:tasks.rados.rados.0.smithi115.stdout:1167: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:1168: finishing write tid 3 to smithi11541303-40 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:1168: finishing write tid 4 to smithi11541303-40 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 406 (ObjNum 352 snap 121 seq_num 352) dirty exists 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:1168: left oid 40 (ObjNum 352 snap 121 seq_num 352) 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:1167: done (6 left) 2022-04-22T17:29:45.903 INFO:tasks.rados.rados.0.smithi115.stdout:1168: done (5 left) 2022-04-22T17:29:45.904 INFO:tasks.rados.rados.0.smithi115.stdout:1174: snap_remove snap 108 2022-04-22T17:29:46.819 INFO:tasks.rados.rados.0.smithi115.stdout:1174: done (5 left) 2022-04-22T17:29:46.819 INFO:tasks.rados.rados.0.smithi115.stdout:1175: copy_from oid 12 from oid 29 current snap is 121 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1169: finishing write tid 1 to smithi11541303-37 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1169: finishing write tid 2 to smithi11541303-37 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1169: finishing write tid 3 to smithi11541303-37 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1169: finishing write tid 4 to smithi11541303-37 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 529 (ObjNum 353 snap 121 seq_num 353) dirty exists 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1169: left oid 37 (ObjNum 353 snap 121 seq_num 353) 2022-04-22T17:29:46.820 INFO:tasks.rados.rados.0.smithi115.stdout:1172: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1171: finishing write tid 1 to smithi11541303-33 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1172: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1172: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 410 (ObjNum 283 snap 93 seq_num 283) dirty exists 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1169: done (5 left) 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1170: done (4 left) 2022-04-22T17:29:46.821 INFO:tasks.rados.rados.0.smithi115.stdout:1172: done (3 left) 2022-04-22T17:29:46.822 INFO:tasks.rados.rados.0.smithi115.stdout:1176: snap_create 2022-04-22T17:29:46.822 INFO:tasks.rados.rados.0.smithi115.stdout:1171: finishing write tid 2 to smithi11541303-33 2022-04-22T17:29:46.822 INFO:tasks.rados.rados.0.smithi115.stdout:1171: finishing write tid 3 to smithi11541303-33 2022-04-22T17:29:46.822 INFO:tasks.rados.rados.0.smithi115.stdout:1171: finishing write tid 4 to smithi11541303-33 2022-04-22T17:29:46.822 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 539 (ObjNum 354 snap 121 seq_num 354) dirty exists 2022-04-22T17:29:46.823 INFO:tasks.rados.rados.0.smithi115.stdout:1171: left oid 33 (ObjNum 354 snap 121 seq_num 354) 2022-04-22T17:29:46.823 INFO:tasks.rados.rados.0.smithi115.stdout:1173: finishing copy_from to smithi11541303-6 2022-04-22T17:29:46.823 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 141 (ObjNum 1 snap 32692 seq_num 3179010160) dirty exists 2022-04-22T17:29:46.823 INFO:tasks.rados.rados.0.smithi115.stdout:1173: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:29:46.833 INFO:tasks.rados.rados.0.smithi115.stdout:1175: finishing copy_from to smithi11541303-12 2022-04-22T17:29:46.833 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 250 (ObjNum 335 snap 117 seq_num 335) dirty exists 2022-04-22T17:29:46.833 INFO:tasks.rados.rados.0.smithi115.stdout:1175: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:46 smithi121 bash[36902]: cluster 2022-04-22T17:29:45.151301+0000 mgr.y (mgr.34107) 89 : cluster [DBG] pgmap v123: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:46 smithi121 bash[36902]: audit 2022-04-22T17:29:45.580735+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:46 smithi121 bash[36902]: cephadm 2022-04-22T17:29:45.581971+0000 mgr.y (mgr.34107) 90 : cephadm [INF] Reconfiguring osd.4 (monmap changed)... 2022-04-22T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:46 smithi121 bash[36902]: cephadm 2022-04-22T17:29:45.583095+0000 mgr.y (mgr.34107) 91 : cephadm [INF] Reconfiguring daemon osd.4 on smithi121 2022-04-22T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:46 smithi121 bash[36902]: cluster 2022-04-22T17:29:45.798426+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2022-04-22T17:29:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[45495]: cluster 2022-04-22T17:29:45.151301+0000 mgr.y (mgr.34107) 89 : cluster [DBG] pgmap v123: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[45495]: audit 2022-04-22T17:29:45.580735+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[45495]: cephadm 2022-04-22T17:29:45.581971+0000 mgr.y (mgr.34107) 90 : cephadm [INF] Reconfiguring osd.4 (monmap changed)... 2022-04-22T17:29:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[45495]: cephadm 2022-04-22T17:29:45.583095+0000 mgr.y (mgr.34107) 91 : cephadm [INF] Reconfiguring daemon osd.4 on smithi121 2022-04-22T17:29:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[45495]: cluster 2022-04-22T17:29:45.798426+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2022-04-22T17:29:47.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[46934]: cluster 2022-04-22T17:29:45.151301+0000 mgr.y (mgr.34107) 89 : cluster [DBG] pgmap v123: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:47.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[46934]: audit 2022-04-22T17:29:45.580735+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:47.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[46934]: cephadm 2022-04-22T17:29:45.581971+0000 mgr.y (mgr.34107) 90 : cephadm [INF] Reconfiguring osd.4 (monmap changed)... 2022-04-22T17:29:47.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[46934]: cephadm 2022-04-22T17:29:45.583095+0000 mgr.y (mgr.34107) 91 : cephadm [INF] Reconfiguring daemon osd.4 on smithi121 2022-04-22T17:29:47.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:46 smithi115 bash[46934]: cluster 2022-04-22T17:29:45.798426+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2022-04-22T17:29:47.830 INFO:tasks.rados.rados.0.smithi115.stdout:1171: done (3 left) 2022-04-22T17:29:47.890 INFO:tasks.rados.rados.0.smithi115.stdout:1173: done (2 left) 2022-04-22T17:29:47.890 INFO:tasks.rados.rados.0.smithi115.stdout:1175: done (1 left) 2022-04-22T17:29:47.890 INFO:tasks.rados.rados.0.smithi115.stdout:1176: done (0 left) 2022-04-22T17:29:47.890 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 122 2022-04-22T17:29:47.890 INFO:tasks.rados.rados.0.smithi115.stdout:1177: seq_num 355 ranges {0=753664,753664=401408} 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1177: writing smithi11541303-11 from 0 to 753664 tid 1 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1177: writing smithi11541303-11 from 753664 to 1155072 tid 2 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1178: rmattr oid 30 current snap is 122 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1178: done (1 left) 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1179: delete oid 19 current snap is 122 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1179: done (1 left) 2022-04-22T17:29:47.893 INFO:tasks.rados.rados.0.smithi115.stdout:1180: rollback oid 30 current snap is 122 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 120 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1181: copy_from oid 23 from oid 17 current snap is 122 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 122 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1182: seq_num 357 ranges {6217728=786432,7004160=401408} 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1182: writing smithi11541303-24 from 6217728 to 7004160 tid 1 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1182: writing smithi11541303-24 from 7004160 to 7405568 tid 2 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1183: read oid 34 snap 101 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1183: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1184: read oid 10 snap 113 2022-04-22T17:29:47.894 INFO:tasks.rados.rados.0.smithi115.stdout:1184: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1185: read oid 20 snap -1 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1185: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1186: rollback oid 43 current snap is 122 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 113 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1177: finishing write tid 1 to smithi11541303-11 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1177: finishing write tid 2 to smithi11541303-11 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1177: finishing write tid 3 to smithi11541303-11 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1187: rollback oid 13 current snap is 122 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 101 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1181: finishing copy_from to smithi11541303-23 2022-04-22T17:29:47.895 INFO:tasks.rados.rados.0.smithi115.stdout:1181: got expected ENOENT (src dne) 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1181: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1180: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1177: finishing write tid 4 to smithi11541303-11 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 539 (ObjNum 355 snap 122 seq_num 355) dirty exists 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1177: left oid 11 (ObjNum 355 snap 122 seq_num 355) 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1180: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1180: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 621 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1177: done (8 left) 2022-04-22T17:29:47.896 INFO:tasks.rados.rados.0.smithi115.stdout:1180: done (7 left) 2022-04-22T17:29:47.897 INFO:tasks.rados.rados.0.smithi115.stdout:1181: done (6 left) 2022-04-22T17:29:47.897 INFO:tasks.rados.rados.0.smithi115.stdout:1188: copy_from oid 35 from oid 30 current snap is 122 2022-04-22T17:29:47.897 INFO:tasks.rados.rados.0.smithi115.stdout:1189: snap_create 2022-04-22T17:29:47.900 INFO:tasks.rados.rados.0.smithi115.stdout:1182: finishing write tid 1 to smithi11541303-24 2022-04-22T17:29:47.900 INFO:tasks.rados.rados.0.smithi115.stdout:1182: finishing write tid 2 to smithi11541303-24 2022-04-22T17:29:47.903 INFO:tasks.rados.rados.0.smithi115.stdout:1182: finishing write tid 3 to smithi11541303-24 2022-04-22T17:29:47.904 INFO:tasks.rados.rados.0.smithi115.stdout:1182: finishing write tid 4 to smithi11541303-24 2022-04-22T17:29:47.904 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 208 (ObjNum 357 snap 122 seq_num 357) dirty exists 2022-04-22T17:29:47.904 INFO:tasks.rados.rados.0.smithi115.stdout:1182: left oid 24 (ObjNum 357 snap 122 seq_num 357) 2022-04-22T17:29:47.909 INFO:tasks.rados.rados.0.smithi115.stdout:1183: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:29:48.019 INFO:tasks.rados.rados.0.smithi115.stdout:1186: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:29:48.020 INFO:tasks.rados.rados.0.smithi115.stdout:1186: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:29:48.020 INFO:tasks.rados.rados.0.smithi115.stdout:1186: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:29:48.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 465 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:29:48.020 INFO:tasks.rados.rados.0.smithi115.stdout:1187: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:29:48.020 INFO:tasks.rados.rados.0.smithi115.stdout:1185: expect (ObjNum 206 snap 64 seq_num 206) 2022-04-22T17:29:48.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:47 smithi121 bash[36902]: cluster 2022-04-22T17:29:46.804525+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2022-04-22T17:29:48.203 INFO:tasks.rados.rados.0.smithi115.stdout:1187: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:29:48.204 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 467 (ObjNum 298 snap 99 seq_num 298) dirty exists 2022-04-22T17:29:48.204 INFO:tasks.rados.rados.0.smithi115.stdout:1188: finishing copy_from to smithi11541303-35 2022-04-22T17:29:48.204 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 621 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:29:48.204 INFO:tasks.rados.rados.0.smithi115.stdout:1188: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:29:48.204 INFO:tasks.rados.rados.0.smithi115.stdout:1184: expect (ObjNum 323 snap 112 seq_num 323) 2022-04-22T17:29:48.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:47 smithi115 bash[45495]: cluster 2022-04-22T17:29:46.804525+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2022-04-22T17:29:48.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:47 smithi115 bash[46934]: cluster 2022-04-22T17:29:46.804525+0000 mon.a (mon.0) 252 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1182: done (7 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1183: done (6 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1184: done (5 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1185: done (4 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1186: done (3 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1187: done (2 left) 2022-04-22T17:29:48.973 INFO:tasks.rados.rados.0.smithi115.stdout:1188: done (1 left) 2022-04-22T17:29:48.974 INFO:tasks.rados.rados.0.smithi115.stdout:1189: done (0 left) 2022-04-22T17:29:48.974 INFO:tasks.rados.rados.0.smithi115.stdout:1190: read oid 31 snap 113 2022-04-22T17:29:48.974 INFO:tasks.rados.rados.0.smithi115.stdout:1190: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:29:48.974 INFO:tasks.rados.rados.0.smithi115.stdout:1191: delete oid 4 current snap is 123 2022-04-22T17:29:48.976 INFO:tasks.rados.rados.0.smithi115.stdout:1191: done (1 left) 2022-04-22T17:29:48.976 INFO:tasks.rados.rados.0.smithi115.stdout:1192: snap_remove snap 117 2022-04-22T17:29:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:48 smithi121 bash[36902]: cluster 2022-04-22T17:29:47.152080+0000 mgr.y (mgr.34107) 92 : cluster [DBG] pgmap v126: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:48 smithi121 bash[36902]: cluster 2022-04-22T17:29:47.816764+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2022-04-22T17:29:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:48 smithi121 bash[36902]: audit 2022-04-22T17:29:48.013094+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:48 smithi121 bash[36902]: cephadm 2022-04-22T17:29:48.014493+0000 mgr.y (mgr.34107) 93 : cephadm [INF] Reconfiguring osd.5 (monmap changed)... 2022-04-22T17:29:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:48 smithi121 bash[36902]: cephadm 2022-04-22T17:29:48.015922+0000 mgr.y (mgr.34107) 94 : cephadm [INF] Reconfiguring daemon osd.5 on smithi121 2022-04-22T17:29:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[46934]: cluster 2022-04-22T17:29:47.152080+0000 mgr.y (mgr.34107) 92 : cluster [DBG] pgmap v126: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[46934]: cluster 2022-04-22T17:29:47.816764+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2022-04-22T17:29:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[46934]: audit 2022-04-22T17:29:48.013094+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[46934]: cephadm 2022-04-22T17:29:48.014493+0000 mgr.y (mgr.34107) 93 : cephadm [INF] Reconfiguring osd.5 (monmap changed)... 2022-04-22T17:29:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[46934]: cephadm 2022-04-22T17:29:48.015922+0000 mgr.y (mgr.34107) 94 : cephadm [INF] Reconfiguring daemon osd.5 on smithi121 2022-04-22T17:29:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[45495]: cluster 2022-04-22T17:29:47.152080+0000 mgr.y (mgr.34107) 92 : cluster [DBG] pgmap v126: 121 pgs: 121 active+clean; 250 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2022-04-22T17:29:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[45495]: cluster 2022-04-22T17:29:47.816764+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2022-04-22T17:29:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[45495]: audit 2022-04-22T17:29:48.013094+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[45495]: cephadm 2022-04-22T17:29:48.014493+0000 mgr.y (mgr.34107) 93 : cephadm [INF] Reconfiguring osd.5 (monmap changed)... 2022-04-22T17:29:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:48 smithi115 bash[45495]: cephadm 2022-04-22T17:29:48.015922+0000 mgr.y (mgr.34107) 94 : cephadm [INF] Reconfiguring daemon osd.5 on smithi121 2022-04-22T17:29:49.842 INFO:tasks.rados.rados.0.smithi115.stdout:1192: done (1 left) 2022-04-22T17:29:49.842 INFO:tasks.rados.rados.0.smithi115.stdout:1193: read oid 9 snap -1 2022-04-22T17:29:49.842 INFO:tasks.rados.rados.0.smithi115.stdout:1193: expect deleted 2022-04-22T17:29:49.842 INFO:tasks.rados.rados.0.smithi115.stdout:1194: delete oid 47 current snap is 123 2022-04-22T17:29:49.842 INFO:tasks.rados.rados.0.smithi115.stdout:1190: expect (ObjNum 299 snap 99 seq_num 299) 2022-04-22T17:29:49.855 INFO:tasks.rados.rados.0.smithi115.stdout:1190: done (2 left) 2022-04-22T17:29:49.855 INFO:tasks.rados.rados.0.smithi115.stdout:1194: done (1 left) 2022-04-22T17:29:49.856 INFO:tasks.rados.rados.0.smithi115.stdout:1195: snap_remove snap 96 2022-04-22T17:29:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:49 smithi121 bash[36902]: cluster 2022-04-22T17:29:48.819045+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2022-04-22T17:29:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:49 smithi121 bash[36902]: audit 2022-04-22T17:29:49.164224+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:50.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:49 smithi115 bash[45495]: cluster 2022-04-22T17:29:48.819045+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2022-04-22T17:29:50.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:49 smithi115 bash[45495]: audit 2022-04-22T17:29:49.164224+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:50.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:49 smithi115 bash[46934]: cluster 2022-04-22T17:29:48.819045+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2022-04-22T17:29:50.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:49 smithi115 bash[46934]: audit 2022-04-22T17:29:49.164224+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1195: done (1 left) 2022-04-22T17:29:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1196: read oid 39 snap 109 2022-04-22T17:29:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1196: expect (ObjNum 317 snap 107 seq_num 317) 2022-04-22T17:29:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 123 2022-04-22T17:29:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1197: seq_num 360 ranges {7405568=434176,7839744=704512,8544256=98304} 2022-04-22T17:29:50.897 INFO:tasks.rados.rados.0.smithi115.stdout:1197: writing smithi11541303-24 from 7405568 to 7839744 tid 1 2022-04-22T17:29:50.901 INFO:tasks.rados.rados.0.smithi115.stdout:1197: writing smithi11541303-24 from 7839744 to 8544256 tid 2 2022-04-22T17:29:50.902 INFO:tasks.rados.rados.0.smithi115.stdout:1197: writing smithi11541303-24 from 8544256 to 8642560 tid 3 2022-04-22T17:29:50.902 INFO:tasks.rados.rados.0.smithi115.stdout:1193: done (2 left) 2022-04-22T17:29:50.903 INFO:tasks.rados.rados.0.smithi115.stdout:1198: read oid 15 snap -1 2022-04-22T17:29:50.903 INFO:tasks.rados.rados.0.smithi115.stdout:1198: expect (ObjNum 307 snap 101 seq_num 307) 2022-04-22T17:29:50.903 INFO:tasks.rados.rados.0.smithi115.stdout:1199: copy_from oid 6 from oid 17 current snap is 123 2022-04-22T17:29:50.903 INFO:tasks.rados.rados.0.smithi115.stdout:1196: expect (ObjNum 317 snap 107 seq_num 317) 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1197: finishing write tid 1 to smithi11541303-24 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1197: finishing write tid 2 to smithi11541303-24 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1197: finishing write tid 3 to smithi11541303-24 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1197: finishing write tid 4 to smithi11541303-24 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1199: finishing copy_from to smithi11541303-6 2022-04-22T17:29:51.128 INFO:tasks.rados.rados.0.smithi115.stdout:1199: got expected ENOENT (src dne) 2022-04-22T17:29:51.129 INFO:tasks.rados.rados.0.smithi115.stdout:1199: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:29:51.129 INFO:tasks.rados.rados.0.smithi115.stdout:1197: finishing write tid 5 to smithi11541303-24 2022-04-22T17:29:51.129 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 221 (ObjNum 360 snap 123 seq_num 360) dirty exists 2022-04-22T17:29:51.129 INFO:tasks.rados.rados.0.smithi115.stdout:1197: left oid 24 (ObjNum 360 snap 123 seq_num 360) 2022-04-22T17:29:51.129 INFO:tasks.rados.rados.0.smithi115.stdout:1198: expect (ObjNum 307 snap 101 seq_num 307) 2022-04-22T17:29:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:50 smithi121 bash[36902]: cluster 2022-04-22T17:29:49.153325+0000 mgr.y (mgr.34107) 95 : cluster [DBG] pgmap v129: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:50 smithi121 bash[36902]: cluster 2022-04-22T17:29:49.828228+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2022-04-22T17:29:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:50 smithi121 bash[36902]: audit 2022-04-22T17:29:50.537104+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:51.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[45495]: cluster 2022-04-22T17:29:49.153325+0000 mgr.y (mgr.34107) 95 : cluster [DBG] pgmap v129: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:51.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[45495]: cluster 2022-04-22T17:29:49.828228+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2022-04-22T17:29:51.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[45495]: audit 2022-04-22T17:29:50.537104+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:51.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[46934]: cluster 2022-04-22T17:29:49.153325+0000 mgr.y (mgr.34107) 95 : cluster [DBG] pgmap v129: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[46934]: cluster 2022-04-22T17:29:49.828228+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2022-04-22T17:29:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:50 smithi115 bash[46934]: audit 2022-04-22T17:29:50.537104+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:51.389 INFO:tasks.rados.rados.0.smithi115.stdout:1196: done (3 left) 2022-04-22T17:29:51.439 INFO:tasks.rados.rados.0.smithi115.stdout:1197: done (2 left) 2022-04-22T17:29:51.439 INFO:tasks.rados.rados.0.smithi115.stdout:1198: done (1 left) 2022-04-22T17:29:51.439 INFO:tasks.rados.rados.0.smithi115.stdout:1199: done (0 left) 2022-04-22T17:29:51.439 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 123 2022-04-22T17:29:51.440 INFO:tasks.rados.rados.0.smithi115.stdout:1200: seq_num 361 ranges {819200=450560,1269760=581632,1851392=221184} 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:1200: writing smithi11541303-12 from 819200 to 1269760 tid 1 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:1200: writing smithi11541303-12 from 1269760 to 1851392 tid 2 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:1200: writing smithi11541303-12 from 1851392 to 2072576 tid 3 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:1201: rollback oid 39 current snap is 123 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 95 2022-04-22T17:29:51.441 INFO:tasks.rados.rados.0.smithi115.stdout:1202: delete oid 31 current snap is 123 2022-04-22T17:29:51.442 INFO:tasks.rados.rados.0.smithi115.stdout:1200: finishing write tid 1 to smithi11541303-12 2022-04-22T17:29:51.442 INFO:tasks.rados.rados.0.smithi115.stdout:1201: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:29:51.442 INFO:tasks.rados.rados.0.smithi115.stdout:1202: done (2 left) 2022-04-22T17:29:51.442 INFO:tasks.rados.rados.0.smithi115.stdout:1203: snap_remove snap 120 2022-04-22T17:29:51.862 INFO:tasks.rados.rados.0.smithi115.stdout:1203: done (2 left) 2022-04-22T17:29:51.862 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 123 2022-04-22T17:29:51.862 INFO:tasks.rados.rados.0.smithi115.stdout:1201: finishing rollback tid 2 to smithi11541303-39 2022-04-22T17:29:51.863 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 623 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:29:51.863 INFO:tasks.rados.rados.0.smithi115.stdout:1204: seq_num 363 ranges {2080768=483328,2564096=753664,3317760=49152} 2022-04-22T17:29:51.877 INFO:tasks.rados.rados.0.smithi115.stdout:1204: writing smithi11541303-7 from 2080768 to 2564096 tid 1 2022-04-22T17:29:51.883 INFO:tasks.rados.rados.0.smithi115.stdout:1204: writing smithi11541303-7 from 2564096 to 3317760 tid 2 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1204: writing smithi11541303-7 from 3317760 to 3366912 tid 3 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1201: done (2 left) 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1205: rmattr oid 45 current snap is 123 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1205: done (2 left) 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 123 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1200: finishing write tid 2 to smithi11541303-12 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1200: finishing write tid 3 to smithi11541303-12 2022-04-22T17:29:51.884 INFO:tasks.rados.rados.0.smithi115.stdout:1200: finishing write tid 4 to smithi11541303-12 2022-04-22T17:29:51.885 INFO:tasks.rados.rados.0.smithi115.stdout:1200: finishing write tid 5 to smithi11541303-12 2022-04-22T17:29:51.885 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 468 (ObjNum 361 snap 123 seq_num 361) dirty exists 2022-04-22T17:29:51.885 INFO:tasks.rados.rados.0.smithi115.stdout:1200: left oid 12 (ObjNum 361 snap 123 seq_num 361) 2022-04-22T17:29:51.885 INFO:tasks.rados.rados.0.smithi115.stdout:1204: finishing write tid 1 to smithi11541303-7 2022-04-22T17:29:51.885 INFO:tasks.rados.rados.0.smithi115.stdout:1206: seq_num 364 ranges {696320=499712,1196032=638976,1835008=163840} 2022-04-22T17:29:51.889 INFO:tasks.rados.rados.0.smithi115.stdout:1206: writing smithi11541303-16 from 696320 to 1196032 tid 1 2022-04-22T17:29:51.893 INFO:tasks.rados.rados.0.smithi115.stdout:1206: writing smithi11541303-16 from 1196032 to 1835008 tid 2 2022-04-22T17:29:51.895 INFO:tasks.rados.rados.0.smithi115.stdout:1206: writing smithi11541303-16 from 1835008 to 1998848 tid 3 2022-04-22T17:29:51.895 INFO:tasks.rados.rados.0.smithi115.stdout:1200: done (2 left) 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1207: copy_from oid 40 from oid 36 current snap is 123 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1204: finishing write tid 2 to smithi11541303-7 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1204: finishing write tid 3 to smithi11541303-7 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1204: finishing write tid 4 to smithi11541303-7 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1204: finishing write tid 5 to smithi11541303-7 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 401 (ObjNum 363 snap 123 seq_num 363) dirty exists 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1204: left oid 7 (ObjNum 363 snap 123 seq_num 363) 2022-04-22T17:29:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:1206: finishing write tid 1 to smithi11541303-16 2022-04-22T17:29:51.897 INFO:tasks.rados.rados.0.smithi115.stdout:1204: done (2 left) 2022-04-22T17:29:51.897 INFO:tasks.rados.rados.0.smithi115.stdout:1208: read oid 43 snap -1 2022-04-22T17:29:51.897 INFO:tasks.rados.rados.0.smithi115.stdout:1208: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:29:51.897 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 123 2022-04-22T17:29:51.897 INFO:tasks.rados.rados.0.smithi115.stdout:1209: seq_num 365 ranges {0=516096,516096=573440,1089536=229376} 2022-04-22T17:29:51.899 INFO:tasks.rados.rados.0.smithi115.stdout:1209: writing smithi11541303-19 from 0 to 516096 tid 1 2022-04-22T17:29:51.902 INFO:tasks.rados.rados.0.smithi115.stdout:1209: writing smithi11541303-19 from 516096 to 1089536 tid 2 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1209: writing smithi11541303-19 from 1089536 to 1318912 tid 3 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1210: snap_create 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1207: finishing copy_from to smithi11541303-40 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1207: got expected ENOENT (src dne) 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1207: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1206: finishing write tid 2 to smithi11541303-16 2022-04-22T17:29:51.905 INFO:tasks.rados.rados.0.smithi115.stdout:1206: finishing write tid 3 to smithi11541303-16 2022-04-22T17:29:51.906 INFO:tasks.rados.rados.0.smithi115.stdout:1206: finishing write tid 4 to smithi11541303-16 2022-04-22T17:29:51.906 INFO:tasks.rados.rados.0.smithi115.stdout:1208: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:1206: finishing write tid 5 to smithi11541303-16 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 384 (ObjNum 364 snap 123 seq_num 364) dirty exists 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:1206: left oid 16 (ObjNum 364 snap 123 seq_num 364) 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:1209: finishing write tid 1 to smithi11541303-19 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:1209: finishing write tid 2 to smithi11541303-19 2022-04-22T17:29:51.911 INFO:tasks.rados.rados.0.smithi115.stdout:1209: finishing write tid 3 to smithi11541303-19 2022-04-22T17:29:51.912 INFO:tasks.rados.rados.0.smithi115.stdout:1209: finishing write tid 4 to smithi11541303-19 2022-04-22T17:29:51.912 INFO:tasks.rados.rados.0.smithi115.stdout:1209: finishing write tid 5 to smithi11541303-19 2022-04-22T17:29:51.912 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 472 (ObjNum 365 snap 123 seq_num 365) dirty exists 2022-04-22T17:29:51.912 INFO:tasks.rados.rados.0.smithi115.stdout:1209: left oid 19 (ObjNum 365 snap 123 seq_num 365) 2022-04-22T17:29:52.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[45495]: cephadm 2022-04-22T17:29:50.538677+0000 mgr.y (mgr.34107) 96 : cephadm [INF] Reconfiguring osd.6 (monmap changed)... 2022-04-22T17:29:52.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[45495]: cephadm 2022-04-22T17:29:50.540666+0000 mgr.y (mgr.34107) 97 : cephadm [INF] Reconfiguring daemon osd.6 on smithi121 2022-04-22T17:29:52.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[45495]: cluster 2022-04-22T17:29:50.835809+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2022-04-22T17:29:52.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:51.619Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:52.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:51.619Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:52.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:51.619Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:52.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[46934]: cephadm 2022-04-22T17:29:50.538677+0000 mgr.y (mgr.34107) 96 : cephadm [INF] Reconfiguring osd.6 (monmap changed)... 2022-04-22T17:29:52.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[46934]: cephadm 2022-04-22T17:29:50.540666+0000 mgr.y (mgr.34107) 97 : cephadm [INF] Reconfiguring daemon osd.6 on smithi121 2022-04-22T17:29:52.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:51 smithi115 bash[46934]: cluster 2022-04-22T17:29:50.835809+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2022-04-22T17:29:52.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:51 smithi121 bash[36902]: cephadm 2022-04-22T17:29:50.538677+0000 mgr.y (mgr.34107) 96 : cephadm [INF] Reconfiguring osd.6 (monmap changed)... 2022-04-22T17:29:52.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:51 smithi121 bash[36902]: cephadm 2022-04-22T17:29:50.540666+0000 mgr.y (mgr.34107) 97 : cephadm [INF] Reconfiguring daemon osd.6 on smithi121 2022-04-22T17:29:52.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:51 smithi121 bash[36902]: cluster 2022-04-22T17:29:50.835809+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2022-04-22T17:29:52.872 INFO:tasks.rados.rados.0.smithi115.stdout:1206: done (4 left) 2022-04-22T17:29:52.872 INFO:tasks.rados.rados.0.smithi115.stdout:1207: done (3 left) 2022-04-22T17:29:52.872 INFO:tasks.rados.rados.0.smithi115.stdout:1208: done (2 left) 2022-04-22T17:29:52.872 INFO:tasks.rados.rados.0.smithi115.stdout:1209: done (1 left) 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:1210: done (0 left) 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:1211: copy_from oid 49 from oid 24 current snap is 124 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:1212: rollback oid 15 current snap is 124 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 114 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 124 2022-04-22T17:29:52.873 INFO:tasks.rados.rados.0.smithi115.stdout:1213: seq_num 366 ranges {163840=532480,696320=458752,1155072=344064} 2022-04-22T17:29:52.877 INFO:tasks.rados.rados.0.smithi115.stdout:1213: writing smithi11541303-35 from 163840 to 696320 tid 1 2022-04-22T17:29:52.880 INFO:tasks.rados.rados.0.smithi115.stdout:1213: writing smithi11541303-35 from 696320 to 1155072 tid 2 2022-04-22T17:29:52.883 INFO:tasks.rados.rados.0.smithi115.stdout:1213: writing smithi11541303-35 from 1155072 to 1499136 tid 3 2022-04-22T17:29:52.883 INFO:tasks.rados.rados.0.smithi115.stdout:1214: snap_create 2022-04-22T17:29:52.883 INFO:tasks.rados.rados.0.smithi115.stdout:1212: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:29:52.884 INFO:tasks.rados.rados.0.smithi115.stdout:1212: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:29:52.884 INFO:tasks.rados.rados.0.smithi115.stdout:1212: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:29:52.884 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 191 (ObjNum 307 snap 101 seq_num 307) dirty exists 2022-04-22T17:29:52.886 INFO:tasks.rados.rados.0.smithi115.stdout:1213: finishing write tid 1 to smithi11541303-35 2022-04-22T17:29:52.886 INFO:tasks.rados.rados.0.smithi115.stdout:1213: finishing write tid 2 to smithi11541303-35 2022-04-22T17:29:52.887 INFO:tasks.rados.rados.0.smithi115.stdout:1213: finishing write tid 3 to smithi11541303-35 2022-04-22T17:29:52.887 INFO:tasks.rados.rados.0.smithi115.stdout:1213: finishing write tid 4 to smithi11541303-35 2022-04-22T17:29:52.889 INFO:tasks.rados.rados.0.smithi115.stdout:1213: finishing write tid 5 to smithi11541303-35 2022-04-22T17:29:52.889 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 627 (ObjNum 366 snap 124 seq_num 366) dirty exists 2022-04-22T17:29:52.889 INFO:tasks.rados.rados.0.smithi115.stdout:1213: left oid 35 (ObjNum 366 snap 124 seq_num 366) 2022-04-22T17:29:52.942 INFO:tasks.rados.rados.0.smithi115.stdout:1211: finishing copy_from to smithi11541303-49 2022-04-22T17:29:52.942 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 221 (ObjNum 360 snap 123 seq_num 360) dirty exists 2022-04-22T17:29:52.943 INFO:tasks.rados.rados.0.smithi115.stdout:1211: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:29:53.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:52 smithi121 bash[36902]: cluster 2022-04-22T17:29:51.153834+0000 mgr.y (mgr.34107) 98 : cluster [DBG] pgmap v132: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:53.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:52 smithi121 bash[36902]: cluster 2022-04-22T17:29:51.848083+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2022-04-22T17:29:53.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:52 smithi115 bash[45495]: cluster 2022-04-22T17:29:51.153834+0000 mgr.y (mgr.34107) 98 : cluster [DBG] pgmap v132: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:53.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:52 smithi115 bash[45495]: cluster 2022-04-22T17:29:51.848083+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2022-04-22T17:29:53.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:52 smithi115 bash[46934]: cluster 2022-04-22T17:29:51.153834+0000 mgr.y (mgr.34107) 98 : cluster [DBG] pgmap v132: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:29:53.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:52 smithi115 bash[46934]: cluster 2022-04-22T17:29:51.848083+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2022-04-22T17:29:53.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:53.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:53.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:53.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:29:53.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:29:53.515Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:29:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1211: done (3 left) 2022-04-22T17:29:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1212: done (2 left) 2022-04-22T17:29:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1213: done (1 left) 2022-04-22T17:29:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1214: done (0 left) 2022-04-22T17:29:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1215: rollback oid 1 current snap is 125 2022-04-22T17:29:53.884 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 1 to 114 2022-04-22T17:29:53.884 INFO:tasks.rados.rados.0.smithi115.stdout:1216: rollback oid 36 current snap is 125 2022-04-22T17:29:53.884 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 95 2022-04-22T17:29:53.884 INFO:tasks.rados.rados.0.smithi115.stdout:1217: snap_remove snap 116 2022-04-22T17:29:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:53 smithi121 bash[36902]: cluster 2022-04-22T17:29:52.858138+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2022-04-22T17:29:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:53 smithi121 bash[36902]: audit 2022-04-22T17:29:52.966975+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:54.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:53 smithi121 bash[36902]: cephadm 2022-04-22T17:29:52.968573+0000 mgr.y (mgr.34107) 99 : cephadm [INF] Reconfiguring osd.7 (monmap changed)... 2022-04-22T17:29:54.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:53 smithi121 bash[36902]: cephadm 2022-04-22T17:29:52.970107+0000 mgr.y (mgr.34107) 100 : cephadm [INF] Reconfiguring daemon osd.7 on smithi121 2022-04-22T17:29:54.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[45495]: cluster 2022-04-22T17:29:52.858138+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2022-04-22T17:29:54.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[45495]: audit 2022-04-22T17:29:52.966975+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:54.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[45495]: cephadm 2022-04-22T17:29:52.968573+0000 mgr.y (mgr.34107) 99 : cephadm [INF] Reconfiguring osd.7 (monmap changed)... 2022-04-22T17:29:54.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[45495]: cephadm 2022-04-22T17:29:52.970107+0000 mgr.y (mgr.34107) 100 : cephadm [INF] Reconfiguring daemon osd.7 on smithi121 2022-04-22T17:29:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[46934]: cluster 2022-04-22T17:29:52.858138+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2022-04-22T17:29:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[46934]: audit 2022-04-22T17:29:52.966975+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[46934]: cephadm 2022-04-22T17:29:52.968573+0000 mgr.y (mgr.34107) 99 : cephadm [INF] Reconfiguring osd.7 (monmap changed)... 2022-04-22T17:29:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:53 smithi115 bash[46934]: cephadm 2022-04-22T17:29:52.970107+0000 mgr.y (mgr.34107) 100 : cephadm [INF] Reconfiguring daemon osd.7 on smithi121 2022-04-22T17:29:54.763 INFO:tasks.rados.rados.0.smithi115.stdout:1217: done (2 left) 2022-04-22T17:29:54.764 INFO:tasks.rados.rados.0.smithi115.stdout:1218: snap_remove snap 107 2022-04-22T17:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:54 smithi121 bash[36902]: cluster 2022-04-22T17:29:53.154538+0000 mgr.y (mgr.34107) 101 : cluster [DBG] pgmap v135: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:54 smithi121 bash[36902]: cluster 2022-04-22T17:29:53.869194+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2022-04-22T17:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:54 smithi121 bash[36902]: audit 2022-04-22T17:29:54.166610+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:54 smithi121 bash[36902]: cluster 2022-04-22T17:29:54.749191+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2022-04-22T17:29:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[45495]: cluster 2022-04-22T17:29:53.154538+0000 mgr.y (mgr.34107) 101 : cluster [DBG] pgmap v135: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[45495]: cluster 2022-04-22T17:29:53.869194+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2022-04-22T17:29:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[45495]: audit 2022-04-22T17:29:54.166610+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[45495]: cluster 2022-04-22T17:29:54.749191+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2022-04-22T17:29:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[46934]: cluster 2022-04-22T17:29:53.154538+0000 mgr.y (mgr.34107) 101 : cluster [DBG] pgmap v135: 121 pgs: 121 active+clean; 273 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2022-04-22T17:29:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[46934]: cluster 2022-04-22T17:29:53.869194+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2022-04-22T17:29:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[46934]: audit 2022-04-22T17:29:54.166610+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:54 smithi115 bash[46934]: cluster 2022-04-22T17:29:54.749191+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2022-04-22T17:29:55.765 INFO:tasks.rados.rados.0.smithi115.stdout:1218: done (2 left) 2022-04-22T17:29:55.781 INFO:tasks.rados.rados.0.smithi115.stdout:1219: snap_remove snap 112 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: cluster 2022-04-22T17:29:55.155286+0000 mgr.y (mgr.34107) 102 : cluster [DBG] pgmap v138: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: audit 2022-04-22T17:29:55.290554+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: cephadm 2022-04-22T17:29:55.292357+0000 mgr.y (mgr.34107) 103 : cephadm [INF] Reconfiguring mds.b (monmap changed)... 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: audit 2022-04-22T17:29:55.292932+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: audit 2022-04-22T17:29:55.294325+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: cephadm 2022-04-22T17:29:55.295725+0000 mgr.y (mgr.34107) 104 : cephadm [INF] Reconfiguring daemon mds.b on smithi121 2022-04-22T17:29:56.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:56 smithi121 bash[36902]: cluster 2022-04-22T17:29:55.751124+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2022-04-22T17:29:56.769 INFO:tasks.rados.rados.0.smithi115.stdout:1219: done (2 left) 2022-04-22T17:29:56.790 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 125 2022-04-22T17:29:56.790 INFO:tasks.rados.rados.0.smithi115.stdout:1215: finishing rollback tid 0 to smithi11541303-1 2022-04-22T17:29:56.790 INFO:tasks.rados.rados.0.smithi115.stdout:1220: seq_num 367 ranges {819200=548864,1368064=737280,2105344=65536} 2022-04-22T17:29:56.791 INFO:tasks.rados.rados.0.smithi115.stdout:1220: writing smithi11541303-29 from 819200 to 1368064 tid 1 2022-04-22T17:29:56.792 INFO:tasks.rados.rados.0.smithi115.stdout:1220: writing smithi11541303-29 from 1368064 to 2105344 tid 2 2022-04-22T17:29:56.792 INFO:tasks.rados.rados.0.smithi115.stdout:1220: writing smithi11541303-29 from 2105344 to 2170880 tid 3 2022-04-22T17:29:56.792 INFO:tasks.rados.rados.0.smithi115.stdout:1221: snap_remove snap 113 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: cluster 2022-04-22T17:29:55.155286+0000 mgr.y (mgr.34107) 102 : cluster [DBG] pgmap v138: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: audit 2022-04-22T17:29:55.290554+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: cephadm 2022-04-22T17:29:55.292357+0000 mgr.y (mgr.34107) 103 : cephadm [INF] Reconfiguring mds.b (monmap changed)... 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: audit 2022-04-22T17:29:55.292932+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: audit 2022-04-22T17:29:55.294325+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: cephadm 2022-04-22T17:29:55.295725+0000 mgr.y (mgr.34107) 104 : cephadm [INF] Reconfiguring daemon mds.b on smithi121 2022-04-22T17:29:56.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[45495]: cluster 2022-04-22T17:29:55.751124+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: cluster 2022-04-22T17:29:55.155286+0000 mgr.y (mgr.34107) 102 : cluster [DBG] pgmap v138: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: audit 2022-04-22T17:29:55.290554+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: cephadm 2022-04-22T17:29:55.292357+0000 mgr.y (mgr.34107) 103 : cephadm [INF] Reconfiguring mds.b (monmap changed)... 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: audit 2022-04-22T17:29:55.292932+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: audit 2022-04-22T17:29:55.294325+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: cephadm 2022-04-22T17:29:55.295725+0000 mgr.y (mgr.34107) 104 : cephadm [INF] Reconfiguring daemon mds.b on smithi121 2022-04-22T17:29:56.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:56 smithi115 bash[46934]: cluster 2022-04-22T17:29:55.751124+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1221: done (3 left) 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1222: copy_from oid 11 from oid 39 current snap is 125 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1215: finishing rollback tid 1 to smithi11541303-1 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1215: finishing rollback tid 2 to smithi11541303-1 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 404 (ObjNum 273 snap 91 seq_num 273) dirty exists 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1216: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:29:57.775 INFO:tasks.rados.rados.0.smithi115.stdout:1216: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 406 (ObjNum 286 snap 95 seq_num 286) dirty exists 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: finishing write tid 1 to smithi11541303-29 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: finishing write tid 2 to smithi11541303-29 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: finishing write tid 3 to smithi11541303-29 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: finishing write tid 4 to smithi11541303-29 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: finishing write tid 5 to smithi11541303-29 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 287 (ObjNum 367 snap 125 seq_num 367) dirty exists 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1220: left oid 29 (ObjNum 367 snap 125 seq_num 367) 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1215: done (3 left) 2022-04-22T17:29:57.776 INFO:tasks.rados.rados.0.smithi115.stdout:1216: done (2 left) 2022-04-22T17:29:57.777 INFO:tasks.rados.rados.0.smithi115.stdout:1220: done (1 left) 2022-04-22T17:29:57.777 INFO:tasks.rados.rados.0.smithi115.stdout:1223: snap_remove snap 118 2022-04-22T17:29:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: cluster 2022-04-22T17:29:56.753988+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.718934+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.723797+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.727054+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.728741+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.736141+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.742086+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2022-04-22T17:29:58.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:57 smithi121 bash[36902]: audit 2022-04-22T17:29:57.747080+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2022-04-22T17:29:58.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: cluster 2022-04-22T17:29:56.753988+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.718934+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.723797+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.727054+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.728741+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.736141+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.742086+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2022-04-22T17:29:58.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[45495]: audit 2022-04-22T17:29:57.747080+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: cluster 2022-04-22T17:29:56.753988+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.718934+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.723797+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.727054+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.728741+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.736141+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.742086+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2022-04-22T17:29:58.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:57 smithi115 bash[46934]: audit 2022-04-22T17:29:57.747080+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2022-04-22T17:29:58.781 INFO:tasks.rados.rados.0.smithi115.stdout:1223: done (1 left) 2022-04-22T17:29:58.781 INFO:tasks.rados.rados.0.smithi115.stdout:1224: snap_create 2022-04-22T17:29:58.781 INFO:tasks.rados.rados.0.smithi115.stdout:1222: finishing copy_from to smithi11541303-11 2022-04-22T17:29:58.781 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 623 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:29:58.781 INFO:tasks.rados.rados.0.smithi115.stdout:1222: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:29:59.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: cluster 2022-04-22T17:29:57.156049+0000 mgr.y (mgr.34107) 105 : cluster [DBG] pgmap v141: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:59.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: cephadm 2022-04-22T17:29:57.729941+0000 mgr.y (mgr.34107) 106 : cephadm [INF] Upgrade: Setting container_image for all mon 2022-04-22T17:29:59.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.748930+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2022-04-22T17:29:59.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: cluster 2022-04-22T17:29:57.760204+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2022-04-22T17:29:59.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.760340+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.761933+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.777777+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.780287+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: cephadm 2022-04-22T17:29:57.782126+0000 mgr.y (mgr.34107) 107 : cephadm [INF] Upgrade: Setting container_image for all crash 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.790501+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.793104+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: audit 2022-04-22T17:29:57.793302+0000 mgr.y (mgr.34107) 108 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[45495]: cephadm 2022-04-22T17:29:57.794801+0000 mgr.y (mgr.34107) 109 : cephadm [INF] Upgrade: osd.0 is safe to restart 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: cluster 2022-04-22T17:29:57.156049+0000 mgr.y (mgr.34107) 105 : cluster [DBG] pgmap v141: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: cephadm 2022-04-22T17:29:57.729941+0000 mgr.y (mgr.34107) 106 : cephadm [INF] Upgrade: Setting container_image for all mon 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.748930+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: cluster 2022-04-22T17:29:57.760204+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.760340+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.761933+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.777777+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.780287+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: cephadm 2022-04-22T17:29:57.782126+0000 mgr.y (mgr.34107) 107 : cephadm [INF] Upgrade: Setting container_image for all crash 2022-04-22T17:29:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.790501+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:59.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.793104+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: audit 2022-04-22T17:29:57.793302+0000 mgr.y (mgr.34107) 108 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:58 smithi115 bash[46934]: cephadm 2022-04-22T17:29:57.794801+0000 mgr.y (mgr.34107) 109 : cephadm [INF] Upgrade: osd.0 is safe to restart 2022-04-22T17:29:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: cluster 2022-04-22T17:29:57.156049+0000 mgr.y (mgr.34107) 105 : cluster [DBG] pgmap v141: 121 pgs: 121 active+clean; 296 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 3.9 MiB/s wr, 13 op/s 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: cephadm 2022-04-22T17:29:57.729941+0000 mgr.y (mgr.34107) 106 : cephadm [INF] Upgrade: Setting container_image for all mon 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.748930+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: cluster 2022-04-22T17:29:57.760204+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.760340+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.761933+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.777777+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.780287+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:29:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: cephadm 2022-04-22T17:29:57.782126+0000 mgr.y (mgr.34107) 107 : cephadm [INF] Upgrade: Setting container_image for all crash 2022-04-22T17:29:59.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.790501+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:29:59.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.793104+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: audit 2022-04-22T17:29:57.793302+0000 mgr.y (mgr.34107) 108 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2022-04-22T17:29:59.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:58 smithi121 bash[36902]: cephadm 2022-04-22T17:29:57.794801+0000 mgr.y (mgr.34107) 109 : cephadm [INF] Upgrade: osd.0 is safe to restart 2022-04-22T17:29:59.789 INFO:tasks.rados.rados.0.smithi115.stdout:1222: done (1 left) 2022-04-22T17:29:59.789 INFO:tasks.rados.rados.0.smithi115.stdout:1224: done (0 left) 2022-04-22T17:29:59.789 INFO:tasks.rados.rados.0.smithi115.stdout:1225: read oid 47 snap -1 2022-04-22T17:29:59.790 INFO:tasks.rados.rados.0.smithi115.stdout:1225: expect deleted 2022-04-22T17:29:59.790 INFO:tasks.rados.rados.0.smithi115.stdout:1226: read oid 50 snap -1 2022-04-22T17:29:59.790 INFO:tasks.rados.rados.0.smithi115.stdout:1226: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:29:59.790 INFO:tasks.rados.rados.0.smithi115.stdout:1227: snap_create 2022-04-22T17:29:59.799 INFO:tasks.rados.rados.0.smithi115.stdout:1226: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:29:59.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:59 smithi115 bash[45495]: cluster 2022-04-22T17:29:58.766658+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2022-04-22T17:29:59.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:29:59 smithi115 bash[45495]: audit 2022-04-22T17:29:59.169108+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:29:59.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:59 smithi115 bash[46934]: cluster 2022-04-22T17:29:58.766658+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2022-04-22T17:29:59.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:29:59 smithi115 bash[46934]: audit 2022-04-22T17:29:59.169108+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:59 smithi121 bash[36902]: cluster 2022-04-22T17:29:58.766658+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2022-04-22T17:30:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:29:59 smithi121 bash[36902]: audit 2022-04-22T17:29:59.169108+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:00.791 INFO:tasks.rados.rados.0.smithi115.stdout:1225: done (2 left) 2022-04-22T17:30:00.792 INFO:tasks.rados.rados.0.smithi115.stdout:1226: done (1 left) 2022-04-22T17:30:00.792 INFO:tasks.rados.rados.0.smithi115.stdout:1227: done (0 left) 2022-04-22T17:30:00.792 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 127 2022-04-22T17:30:00.792 INFO:tasks.rados.rados.0.smithi115.stdout:1228: seq_num 368 ranges {0=573440,573440=622592,1196032=180224} 2022-04-22T17:30:00.797 INFO:tasks.rados.rados.0.smithi115.stdout:1228: writing smithi11541303-17 from 0 to 573440 tid 1 2022-04-22T17:30:00.801 INFO:tasks.rados.rados.0.smithi115.stdout:1228: writing smithi11541303-17 from 573440 to 1196032 tid 2 2022-04-22T17:30:00.803 INFO:tasks.rados.rados.0.smithi115.stdout:1228: writing smithi11541303-17 from 1196032 to 1376256 tid 3 2022-04-22T17:30:00.803 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 127 2022-04-22T17:30:00.803 INFO:tasks.rados.rados.0.smithi115.stdout:1229: seq_num 369 ranges {1499136=589824,2088960=507904,2596864=294912} 2022-04-22T17:30:00.817 INFO:tasks.rados.rados.0.smithi115.stdout:1229: writing smithi11541303-35 from 1499136 to 2088960 tid 1 2022-04-22T17:30:00.821 INFO:tasks.rados.rados.0.smithi115.stdout:1229: writing smithi11541303-35 from 2088960 to 2596864 tid 2 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1229: writing smithi11541303-35 from 2596864 to 2891776 tid 3 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1230: snap_create 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: finishing write tid 1 to smithi11541303-17 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: finishing write tid 2 to smithi11541303-17 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: finishing write tid 3 to smithi11541303-17 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: finishing write tid 4 to smithi11541303-17 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: finishing write tid 5 to smithi11541303-17 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 527 (ObjNum 368 snap 127 seq_num 368) dirty exists 2022-04-22T17:30:00.823 INFO:tasks.rados.rados.0.smithi115.stdout:1228: left oid 17 (ObjNum 368 snap 127 seq_num 368) 2022-04-22T17:30:00.827 INFO:tasks.rados.rados.0.smithi115.stdout:1229: finishing write tid 1 to smithi11541303-35 2022-04-22T17:30:00.827 INFO:tasks.rados.rados.0.smithi115.stdout:1229: finishing write tid 2 to smithi11541303-35 2022-04-22T17:30:00.828 INFO:tasks.rados.rados.0.smithi115.stdout:1229: finishing write tid 3 to smithi11541303-35 2022-04-22T17:30:00.829 INFO:tasks.rados.rados.0.smithi115.stdout:1229: finishing write tid 4 to smithi11541303-35 2022-04-22T17:30:00.831 INFO:tasks.rados.rados.0.smithi115.stdout:1229: finishing write tid 5 to smithi11541303-35 2022-04-22T17:30:00.831 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 631 (ObjNum 369 snap 127 seq_num 369) dirty exists 2022-04-22T17:30:00.831 INFO:tasks.rados.rados.0.smithi115.stdout:1229: left oid 35 (ObjNum 369 snap 127 seq_num 369) 2022-04-22T17:30:00.995 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[45495]: cluster 2022-04-22T17:29:59.157283+0000 mgr.y (mgr.34107) 110 : cluster [DBG] pgmap v144: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:01.001 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[45495]: cluster 2022-04-22T17:29:59.774557+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2022-04-22T17:30:01.001 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[45495]: cluster 2022-04-22T17:30:00.000146+0000 mon.a (mon.0) 289 : cluster [INF] overall HEALTH_OK 2022-04-22T17:30:01.001 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[45495]: audit 2022-04-22T17:30:00.689056+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:01.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[46934]: cluster 2022-04-22T17:29:59.157283+0000 mgr.y (mgr.34107) 110 : cluster [DBG] pgmap v144: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:01.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[46934]: cluster 2022-04-22T17:29:59.774557+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2022-04-22T17:30:01.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[46934]: cluster 2022-04-22T17:30:00.000146+0000 mon.a (mon.0) 289 : cluster [INF] overall HEALTH_OK 2022-04-22T17:30:01.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:00 smithi115 bash[46934]: audit 2022-04-22T17:30:00.689056+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:00 smithi121 bash[36902]: cluster 2022-04-22T17:29:59.157283+0000 mgr.y (mgr.34107) 110 : cluster [DBG] pgmap v144: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:00 smithi121 bash[36902]: cluster 2022-04-22T17:29:59.774557+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2022-04-22T17:30:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:00 smithi121 bash[36902]: cluster 2022-04-22T17:30:00.000146+0000 mon.a (mon.0) 289 : cluster [INF] overall HEALTH_OK 2022-04-22T17:30:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:00 smithi121 bash[36902]: audit 2022-04-22T17:30:00.689056+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:01.808 INFO:tasks.rados.rados.0.smithi115.stdout:1228: done (2 left) 2022-04-22T17:30:01.857 INFO:tasks.rados.rados.0.smithi115.stdout:1229: done (1 left) 2022-04-22T17:30:01.857 INFO:tasks.rados.rados.0.smithi115.stdout:1230: done (0 left) 2022-04-22T17:30:01.857 INFO:tasks.rados.rados.0.smithi115.stdout:1231: delete oid 44 current snap is 128 2022-04-22T17:30:01.858 INFO:tasks.rados.rados.0.smithi115.stdout:1231: done (0 left) 2022-04-22T17:30:01.858 INFO:tasks.rados.rados.0.smithi115.stdout:1232: rmattr oid 21 current snap is 128 2022-04-22T17:30:01.858 INFO:tasks.rados.rados.0.smithi115.stdout:1232: done (0 left) 2022-04-22T17:30:01.858 INFO:tasks.rados.rados.0.smithi115.stdout:1233: read oid 50 snap -1 2022-04-22T17:30:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1233: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1234: read oid 4 snap -1 2022-04-22T17:30:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1234: expect deleted 2022-04-22T17:30:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1235: snap_create 2022-04-22T17:30:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1233: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:02.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[46934]: cephadm 2022-04-22T17:30:00.682597+0000 mgr.y (mgr.34107) 111 : cephadm [INF] Upgrade: Updating osd.0 2022-04-22T17:30:02.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[46934]: cephadm 2022-04-22T17:30:00.691936+0000 mgr.y (mgr.34107) 112 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:30:02.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[46934]: cluster 2022-04-22T17:30:00.777298+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2022-04-22T17:30:02.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[45495]: cephadm 2022-04-22T17:30:00.682597+0000 mgr.y (mgr.34107) 111 : cephadm [INF] Upgrade: Updating osd.0 2022-04-22T17:30:02.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[45495]: cephadm 2022-04-22T17:30:00.691936+0000 mgr.y (mgr.34107) 112 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:30:02.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[45495]: cluster 2022-04-22T17:30:00.777298+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2022-04-22T17:30:02.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:01.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:02.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:01.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:02.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:01.620Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:01 smithi121 bash[36902]: cephadm 2022-04-22T17:30:00.682597+0000 mgr.y (mgr.34107) 111 : cephadm [INF] Upgrade: Updating osd.0 2022-04-22T17:30:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:01 smithi121 bash[36902]: cephadm 2022-04-22T17:30:00.691936+0000 mgr.y (mgr.34107) 112 : cephadm [INF] Deploying daemon osd.0 on smithi115 2022-04-22T17:30:02.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:01 smithi121 bash[36902]: cluster 2022-04-22T17:30:00.777298+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1233: done (2 left) 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1234: done (1 left) 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1235: done (0 left) 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1236: read oid 40 snap 89 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1236: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1237: copy_from oid 13 from oid 3 current snap is 129 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 129 2022-04-22T17:30:02.814 INFO:tasks.rados.rados.0.smithi115.stdout:1238: seq_num 371 ranges {2850816=622592,3473408=679936,4153344=122880} 2022-04-22T17:30:02.837 INFO:tasks.rados.rados.0.smithi115.stdout:1238: writing smithi11541303-34 from 2850816 to 3473408 tid 1 2022-04-22T17:30:02.841 INFO:tasks.rados.rados.0.smithi115.stdout:1238: writing smithi11541303-34 from 3473408 to 4153344 tid 2 2022-04-22T17:30:02.842 INFO:tasks.rados.rados.0.smithi115.stdout:1238: writing smithi11541303-34 from 4153344 to 4276224 tid 3 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:1239: rollback oid 42 current snap is 129 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 111 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:1237: finishing copy_from to smithi11541303-13 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 520 (ObjNum 348 snap 121 seq_num 348) dirty exists 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:1237: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:30:02.843 INFO:tasks.rados.rados.0.smithi115.stdout:1236: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1238: finishing write tid 1 to smithi11541303-34 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1239: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1239: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 706 (ObjNum 261 snap 87 seq_num 261) dirty exists 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1238: finishing write tid 2 to smithi11541303-34 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1238: finishing write tid 3 to smithi11541303-34 2022-04-22T17:30:03.054 INFO:tasks.rados.rados.0.smithi115.stdout:1238: finishing write tid 4 to smithi11541303-34 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1238: finishing write tid 5 to smithi11541303-34 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 388 (ObjNum 371 snap 129 seq_num 371) dirty exists 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1238: left oid 34 (ObjNum 371 snap 129 seq_num 371) 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1236: done (3 left) 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1237: done (2 left) 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1238: done (1 left) 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1239: done (0 left) 2022-04-22T17:30:03.055 INFO:tasks.rados.rados.0.smithi115.stdout:1240: setattr oid 36 current snap is 129 2022-04-22T17:30:03.058 INFO:tasks.rados.rados.0.smithi115.stdout:1241: copy_from oid 6 from oid 41 current snap is 129 2022-04-22T17:30:03.058 INFO:tasks.rados.rados.0.smithi115.stdout:1242: snap_remove snap 97 2022-04-22T17:30:03.178 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:02 smithi115 bash[45495]: cluster 2022-04-22T17:30:01.158019+0000 mgr.y (mgr.34107) 113 : cluster [DBG] pgmap v147: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:03.178 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:02 smithi115 bash[45495]: cluster 2022-04-22T17:30:01.794114+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2022-04-22T17:30:03.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:02 smithi115 bash[46934]: cluster 2022-04-22T17:30:01.158019+0000 mgr.y (mgr.34107) 113 : cluster [DBG] pgmap v147: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:03.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:02 smithi115 bash[46934]: cluster 2022-04-22T17:30:01.794114+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2022-04-22T17:30:03.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:02 smithi121 bash[36902]: cluster 2022-04-22T17:30:01.158019+0000 mgr.y (mgr.34107) 113 : cluster [DBG] pgmap v147: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.0 MiB/s wr, 1 op/s 2022-04-22T17:30:03.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:02 smithi121 bash[36902]: cluster 2022-04-22T17:30:01.794114+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2022-04-22T17:30:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:03.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:03.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:03.515Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:30:03.825 INFO:tasks.rados.rados.0.smithi115.stdout:1242: done (2 left) 2022-04-22T17:30:03.825 INFO:tasks.rados.rados.0.smithi115.stdout:1243: snap_remove snap 106 2022-04-22T17:30:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:03 smithi121 bash[36902]: cluster 2022-04-22T17:30:02.799448+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2022-04-22T17:30:04.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:03 smithi115 bash[45495]: cluster 2022-04-22T17:30:02.799448+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2022-04-22T17:30:04.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:03 smithi115 bash[46934]: cluster 2022-04-22T17:30:02.799448+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2022-04-22T17:30:04.769 INFO:tasks.rados.rados.0.smithi115.stdout:1243: done (2 left) 2022-04-22T17:30:04.769 INFO:tasks.rados.rados.0.smithi115.stdout:1244: copy_from oid 5 from oid 46 current snap is 129 2022-04-22T17:30:04.769 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 129 2022-04-22T17:30:04.769 INFO:tasks.rados.rados.0.smithi115.stdout:1245: seq_num 372 ranges {1138688=638976,1777664=565248,2342912=237568} 2022-04-22T17:30:04.776 INFO:tasks.rados.rados.0.smithi115.stdout:1245: writing smithi11541303-33 from 1138688 to 1777664 tid 1 2022-04-22T17:30:04.780 INFO:tasks.rados.rados.0.smithi115.stdout:1245: writing smithi11541303-33 from 1777664 to 2342912 tid 2 2022-04-22T17:30:04.782 INFO:tasks.rados.rados.0.smithi115.stdout:1245: writing smithi11541303-33 from 2342912 to 2580480 tid 3 2022-04-22T17:30:04.783 INFO:tasks.rados.rados.0.smithi115.stdout:1241: finishing copy_from to smithi11541303-6 2022-04-22T17:30:04.783 INFO:tasks.rados.rados.0.smithi115.stdout:1241: got expected ENOENT (src dne) 2022-04-22T17:30:04.783 INFO:tasks.rados.rados.0.smithi115.stdout:1241: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:30:04.783 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 407 (ObjNum 286 snap 95 seq_num 286) dirty exists 2022-04-22T17:30:04.783 INFO:tasks.rados.rados.0.smithi115.stdout:1240: done (3 left) 2022-04-22T17:30:04.784 INFO:tasks.rados.rados.0.smithi115.stdout:1241: done (2 left) 2022-04-22T17:30:04.784 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 129 2022-04-22T17:30:04.784 INFO:tasks.rados.rados.0.smithi115.stdout:1246: seq_num 373 ranges {376832=655360,1032192=499712,1531904=303104} 2022-04-22T17:30:04.789 INFO:tasks.rados.rados.0.smithi115.stdout:1246: writing smithi11541303-45 from 376832 to 1032192 tid 1 2022-04-22T17:30:04.793 INFO:tasks.rados.rados.0.smithi115.stdout:1246: writing smithi11541303-45 from 1032192 to 1531904 tid 2 2022-04-22T17:30:04.796 INFO:tasks.rados.rados.0.smithi115.stdout:1246: writing smithi11541303-45 from 1531904 to 1835008 tid 3 2022-04-22T17:30:04.796 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 129 2022-04-22T17:30:04.796 INFO:tasks.rados.rados.0.smithi115.stdout:1245: finishing write tid 1 to smithi11541303-33 2022-04-22T17:30:04.796 INFO:tasks.rados.rados.0.smithi115.stdout:1245: finishing write tid 2 to smithi11541303-33 2022-04-22T17:30:04.797 INFO:tasks.rados.rados.0.smithi115.stdout:1245: finishing write tid 3 to smithi11541303-33 2022-04-22T17:30:04.797 INFO:tasks.rados.rados.0.smithi115.stdout:1245: finishing write tid 4 to smithi11541303-33 2022-04-22T17:30:04.797 INFO:tasks.rados.rados.0.smithi115.stdout:1245: finishing write tid 5 to smithi11541303-33 2022-04-22T17:30:04.797 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 543 (ObjNum 372 snap 129 seq_num 372) dirty exists 2022-04-22T17:30:04.798 INFO:tasks.rados.rados.0.smithi115.stdout:1245: left oid 33 (ObjNum 372 snap 129 seq_num 372) 2022-04-22T17:30:04.798 INFO:tasks.rados.rados.0.smithi115.stdout:1247: seq_num 374 ranges {2850816=671744,3522560=778240,4300800=24576} 2022-04-22T17:30:04.816 INFO:tasks.rados.rados.0.smithi115.stdout:1247: writing smithi11541303-8 from 2850816 to 3522560 tid 1 2022-04-22T17:30:04.820 INFO:tasks.rados.rados.0.smithi115.stdout:1247: writing smithi11541303-8 from 3522560 to 4300800 tid 2 2022-04-22T17:30:04.821 INFO:tasks.rados.rados.0.smithi115.stdout:1247: writing smithi11541303-8 from 4300800 to 4325376 tid 3 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1245: done (3 left) 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1248: rmattr oid 1 current snap is 129 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1248: done (3 left) 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1249: delete oid 28 current snap is 129 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1246: finishing write tid 1 to smithi11541303-45 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1246: finishing write tid 2 to smithi11541303-45 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1244: finishing copy_from to smithi11541303-5 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 417 (ObjNum 306 snap 101 seq_num 306) dirty exists 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1244: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:30:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1246: finishing write tid 3 to smithi11541303-45 2022-04-22T17:30:04.823 INFO:tasks.rados.rados.0.smithi115.stdout:1246: finishing write tid 4 to smithi11541303-45 2022-04-22T17:30:04.823 INFO:tasks.rados.rados.0.smithi115.stdout:1246: finishing write tid 5 to smithi11541303-45 2022-04-22T17:30:04.823 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 683 (ObjNum 373 snap 129 seq_num 373) dirty exists 2022-04-22T17:30:04.823 INFO:tasks.rados.rados.0.smithi115.stdout:1246: left oid 45 (ObjNum 373 snap 129 seq_num 373) 2022-04-22T17:30:04.827 INFO:tasks.rados.rados.0.smithi115.stdout:1244: done (3 left) 2022-04-22T17:30:04.827 INFO:tasks.rados.rados.0.smithi115.stdout:1246: done (2 left) 2022-04-22T17:30:04.827 INFO:tasks.rados.rados.0.smithi115.stdout:1249: done (1 left) 2022-04-22T17:30:04.827 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 129 2022-04-22T17:30:04.827 INFO:tasks.rados.rados.0.smithi115.stdout:1250: seq_num 376 ranges {8642560=704512,9347072=548864,9895936=253952} 2022-04-22T17:30:04.880 INFO:tasks.rados.rados.0.smithi115.stdout:1250: writing smithi11541303-49 from 8642560 to 9347072 tid 1 2022-04-22T17:30:04.883 INFO:tasks.rados.rados.0.smithi115.stdout:1250: writing smithi11541303-49 from 9347072 to 9895936 tid 2 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1250: writing smithi11541303-49 from 9895936 to 10149888 tid 3 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1247: finishing write tid 1 to smithi11541303-8 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1247: finishing write tid 2 to smithi11541303-8 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1247: finishing write tid 3 to smithi11541303-8 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1247: finishing write tid 4 to smithi11541303-8 2022-04-22T17:30:04.885 INFO:tasks.rados.rados.0.smithi115.stdout:1247: finishing write tid 5 to smithi11541303-8 2022-04-22T17:30:04.886 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 411 (ObjNum 374 snap 129 seq_num 374) dirty exists 2022-04-22T17:30:04.886 INFO:tasks.rados.rados.0.smithi115.stdout:1247: left oid 8 (ObjNum 374 snap 129 seq_num 374) 2022-04-22T17:30:04.886 INFO:tasks.rados.rados.0.smithi115.stdout:1247: done (1 left) 2022-04-22T17:30:04.886 INFO:tasks.rados.rados.0.smithi115.stdout:1251: delete oid 20 current snap is 129 2022-04-22T17:30:04.889 INFO:tasks.rados.rados.0.smithi115.stdout:1251: done (1 left) 2022-04-22T17:30:04.889 INFO:tasks.rados.rados.0.smithi115.stdout:1252: setattr oid 17 current snap is 129 2022-04-22T17:30:04.890 INFO:tasks.rados.rados.0.smithi115.stdout:1250: finishing write tid 1 to smithi11541303-49 2022-04-22T17:30:04.890 INFO:tasks.rados.rados.0.smithi115.stdout:1250: finishing write tid 2 to smithi11541303-49 2022-04-22T17:30:04.892 INFO:tasks.rados.rados.0.smithi115.stdout:1250: finishing write tid 3 to smithi11541303-49 2022-04-22T17:30:04.892 INFO:tasks.rados.rados.0.smithi115.stdout:1253: read oid 21 snap -1 2022-04-22T17:30:04.892 INFO:tasks.rados.rados.0.smithi115.stdout:1253: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:30:04.892 INFO:tasks.rados.rados.0.smithi115.stdout:1254: snap_remove snap 127 2022-04-22T17:30:05.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[45495]: cluster 2022-04-22T17:30:03.158503+0000 mgr.y (mgr.34107) 114 : cluster [DBG] pgmap v150: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[45495]: cluster 2022-04-22T17:30:03.809395+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[45495]: audit 2022-04-22T17:30:04.171411+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[45495]: cluster 2022-04-22T17:30:04.750484+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[46934]: cluster 2022-04-22T17:30:03.158503+0000 mgr.y (mgr.34107) 114 : cluster [DBG] pgmap v150: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[46934]: cluster 2022-04-22T17:30:03.809395+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2022-04-22T17:30:05.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[46934]: audit 2022-04-22T17:30:04.171411+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:05.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:04 smithi115 bash[46934]: cluster 2022-04-22T17:30:04.750484+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2022-04-22T17:30:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:04 smithi121 bash[36902]: cluster 2022-04-22T17:30:03.158503+0000 mgr.y (mgr.34107) 114 : cluster [DBG] pgmap v150: 121 pgs: 121 active+clean; 282 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:30:05.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:04 smithi121 bash[36902]: cluster 2022-04-22T17:30:03.809395+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2022-04-22T17:30:05.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:04 smithi121 bash[36902]: audit 2022-04-22T17:30:04.171411+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:05.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:04 smithi121 bash[36902]: cluster 2022-04-22T17:30:04.750484+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2022-04-22T17:30:05.834 INFO:tasks.rados.rados.0.smithi115.stdout:1254: done (3 left) 2022-04-22T17:30:05.834 INFO:tasks.rados.rados.0.smithi115.stdout:1255: snap_remove snap 95 2022-04-22T17:30:05.934 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:05 smithi115 systemd[1]: Stopping Ceph osd.0 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:30:06.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:05 smithi115 bash[20761]: debug 2022-04-22T17:30:05.918+0000 7fac1cc5e700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:30:06.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:05 smithi115 bash[20761]: debug 2022-04-22T17:30:05.918+0000 7fac1cc5e700 -1 osd.0 312 *** Got signal Terminated *** 2022-04-22T17:30:06.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:05 smithi115 bash[20761]: debug 2022-04-22T17:30:05.918+0000 7fac1cc5e700 -1 osd.0 312 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:30:06.840 INFO:tasks.rados.rados.0.smithi115.stdout:1255: done (3 left) 2022-04-22T17:30:06.840 INFO:tasks.rados.rados.0.smithi115.stdout:1256: snap_create 2022-04-22T17:30:06.840 INFO:tasks.rados.rados.0.smithi115.stdout:1250: finishing write tid 4 to smithi11541303-49 2022-04-22T17:30:06.841 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 547 (ObjNum 368 snap 127 seq_num 368) dirty exists 2022-04-22T17:30:06.841 INFO:tasks.rados.rados.0.smithi115.stdout:1250: finishing write tid 5 to smithi11541303-49 2022-04-22T17:30:06.841 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 728 (ObjNum 376 snap 129 seq_num 376) dirty exists 2022-04-22T17:30:06.841 INFO:tasks.rados.rados.0.smithi115.stdout:1250: left oid 49 (ObjNum 376 snap 129 seq_num 376) 2022-04-22T17:30:06.841 INFO:tasks.rados.rados.0.smithi115.stdout:1253: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:05.159208+0000 mgr.y (mgr.34107) 115 : cluster [DBG] pgmap v153: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:05.818035+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.203926+0000 mon.a (mon.0) 298 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.203992+0000 mon.a (mon.0) 299 : cluster [INF] osd.0 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.204261+0000 mon.a (mon.0) 300 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.204470+0000 mon.a (mon.0) 301 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.204666+0000 mon.a (mon.0) 302 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.204837+0000 mon.a (mon.0) 303 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.204958+0000 mon.a (mon.0) 304 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205077+0000 mon.a (mon.0) 305 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205192+0000 mon.a (mon.0) 306 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205322+0000 mon.a (mon.0) 307 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205480+0000 mon.a (mon.0) 308 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205628+0000 mon.a (mon.0) 309 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205753+0000 mon.a (mon.0) 310 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205875+0000 mon.a (mon.0) 311 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.205992+0000 mon.a (mon.0) 312 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.608038+0000 mon.a (mon.0) 313 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.608327+0000 mon.a (mon.0) 314 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.608492+0000 mon.a (mon.0) 315 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.608647+0000 mon.a (mon.0) 316 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.608795+0000 mon.a (mon.0) 317 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609037+0000 mon.a (mon.0) 318 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609246+0000 mon.a (mon.0) 319 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609414+0000 mon.a (mon.0) 320 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609563+0000 mon.a (mon.0) 321 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609726+0000 mon.a (mon.0) 322 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.609886+0000 mon.a (mon.0) 323 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.610033+0000 mon.a (mon.0) 324 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.610245+0000 mon.a (mon.0) 325 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:06 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.610442+0000 mon.a (mon.0) 326 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[49113]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.0 2022-04-22T17:30:07.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[49156]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-0 2022-04-22T17:30:07.302 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[49169]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.0 2022-04-22T17:30:07.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:05.159208+0000 mgr.y (mgr.34107) 115 : cluster [DBG] pgmap v153: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:05.818035+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.203926+0000 mon.a (mon.0) 298 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.203992+0000 mon.a (mon.0) 299 : cluster [INF] osd.0 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.204261+0000 mon.a (mon.0) 300 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.204470+0000 mon.a (mon.0) 301 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.204666+0000 mon.a (mon.0) 302 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.204837+0000 mon.a (mon.0) 303 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.204958+0000 mon.a (mon.0) 304 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205077+0000 mon.a (mon.0) 305 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205192+0000 mon.a (mon.0) 306 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205322+0000 mon.a (mon.0) 307 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205480+0000 mon.a (mon.0) 308 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205628+0000 mon.a (mon.0) 309 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205753+0000 mon.a (mon.0) 310 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205875+0000 mon.a (mon.0) 311 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.205992+0000 mon.a (mon.0) 312 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.608038+0000 mon.a (mon.0) 313 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.608327+0000 mon.a (mon.0) 314 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.608492+0000 mon.a (mon.0) 315 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.608647+0000 mon.a (mon.0) 316 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.608795+0000 mon.a (mon.0) 317 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609037+0000 mon.a (mon.0) 318 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609246+0000 mon.a (mon.0) 319 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609414+0000 mon.a (mon.0) 320 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:05.159208+0000 mgr.y (mgr.34107) 115 : cluster [DBG] pgmap v153: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:07.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:05.818035+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2022-04-22T17:30:07.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.203926+0000 mon.a (mon.0) 298 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.203992+0000 mon.a (mon.0) 299 : cluster [INF] osd.0 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:07.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.204261+0000 mon.a (mon.0) 300 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.204470+0000 mon.a (mon.0) 301 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.204666+0000 mon.a (mon.0) 302 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.204837+0000 mon.a (mon.0) 303 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.204958+0000 mon.a (mon.0) 304 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205077+0000 mon.a (mon.0) 305 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205192+0000 mon.a (mon.0) 306 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205322+0000 mon.a (mon.0) 307 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205480+0000 mon.a (mon.0) 308 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205628+0000 mon.a (mon.0) 309 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205753+0000 mon.a (mon.0) 310 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205875+0000 mon.a (mon.0) 311 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.205992+0000 mon.a (mon.0) 312 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.608038+0000 mon.a (mon.0) 313 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.608327+0000 mon.a (mon.0) 314 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.608492+0000 mon.a (mon.0) 315 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.608647+0000 mon.a (mon.0) 316 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.608795+0000 mon.a (mon.0) 317 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609037+0000 mon.a (mon.0) 318 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609246+0000 mon.a (mon.0) 319 : cluster [DBG] osd.0 reported immediately failed by osd.1 2022-04-22T17:30:07.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609414+0000 mon.a (mon.0) 320 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609563+0000 mon.a (mon.0) 321 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609726+0000 mon.a (mon.0) 322 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.609886+0000 mon.a (mon.0) 323 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.610033+0000 mon.a (mon.0) 324 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.610245+0000 mon.a (mon.0) 325 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.610442+0000 mon.a (mon.0) 326 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609563+0000 mon.a (mon.0) 321 : cluster [DBG] osd.0 reported immediately failed by osd.2 2022-04-22T17:30:07.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609726+0000 mon.a (mon.0) 322 : cluster [DBG] osd.0 reported immediately failed by osd.6 2022-04-22T17:30:07.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.609886+0000 mon.a (mon.0) 323 : cluster [DBG] osd.0 reported immediately failed by osd.4 2022-04-22T17:30:07.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.610033+0000 mon.a (mon.0) 324 : cluster [DBG] osd.0 reported immediately failed by osd.3 2022-04-22T17:30:07.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.610245+0000 mon.a (mon.0) 325 : cluster [DBG] osd.0 reported immediately failed by osd.7 2022-04-22T17:30:07.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:06 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.610442+0000 mon.a (mon.0) 326 : cluster [DBG] osd.0 reported immediately failed by osd.5 2022-04-22T17:30:07.846 INFO:tasks.rados.rados.0.smithi115.stdout:1250: done (3 left) 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:1252: done (2 left) 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:1253: done (1 left) 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:1256: done (0 left) 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:1257: rollback oid 14 current snap is 130 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 121 2022-04-22T17:30:07.847 INFO:tasks.rados.rados.0.smithi115.stdout:1258: snap_remove snap 122 2022-04-22T17:30:08.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.816089+0000 mon.a (mon.0) 327 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:08.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[45495]: cluster 2022-04-22T17:30:06.826435+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e313: 8 total, 7 up, 8 in 2022-04-22T17:30:08.222 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[45495]: cluster 2022-04-22T17:30:07.831017+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e314: 8 total, 7 up, 8 in 2022-04-22T17:30:08.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.816089+0000 mon.a (mon.0) 327 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:08.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[46934]: cluster 2022-04-22T17:30:06.826435+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e313: 8 total, 7 up, 8 in 2022-04-22T17:30:08.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:07 smithi115 bash[46934]: cluster 2022-04-22T17:30:07.831017+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e314: 8 total, 7 up, 8 in 2022-04-22T17:30:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:07 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.816089+0000 mon.a (mon.0) 327 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:07 smithi121 bash[36902]: cluster 2022-04-22T17:30:06.826435+0000 mon.a (mon.0) 328 : cluster [DBG] osdmap e313: 8 total, 7 up, 8 in 2022-04-22T17:30:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:07 smithi121 bash[36902]: cluster 2022-04-22T17:30:07.831017+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e314: 8 total, 7 up, 8 in 2022-04-22T17:30:08.847 INFO:tasks.rados.rados.0.smithi115.stdout:1258: done (1 left) 2022-04-22T17:30:08.847 INFO:tasks.rados.rados.0.smithi115.stdout:1259: snap_remove snap 125 2022-04-22T17:30:09.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:08 smithi115 bash[45495]: cluster 2022-04-22T17:30:07.159910+0000 mgr.y (mgr.34107) 116 : cluster [DBG] pgmap v156: 121 pgs: 15 stale+active+clean, 106 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:09.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:08 smithi115 bash[45495]: cluster 2022-04-22T17:30:08.832839+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e315: 8 total, 7 up, 8 in 2022-04-22T17:30:09.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:08 smithi115 bash[46934]: cluster 2022-04-22T17:30:07.159910+0000 mgr.y (mgr.34107) 116 : cluster [DBG] pgmap v156: 121 pgs: 15 stale+active+clean, 106 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:09.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:08 smithi115 bash[46934]: cluster 2022-04-22T17:30:08.832839+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e315: 8 total, 7 up, 8 in 2022-04-22T17:30:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:08 smithi121 bash[36902]: cluster 2022-04-22T17:30:07.159910+0000 mgr.y (mgr.34107) 116 : cluster [DBG] pgmap v156: 121 pgs: 15 stale+active+clean, 106 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2022-04-22T17:30:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:08 smithi121 bash[36902]: cluster 2022-04-22T17:30:08.832839+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e315: 8 total, 7 up, 8 in 2022-04-22T17:30:09.796 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:09 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.0.service: Succeeded. 2022-04-22T17:30:09.796 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:09 smithi115 systemd[1]: Stopped Ceph osd.0 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:30:09.850 INFO:tasks.rados.rados.0.smithi115.stdout:1259: done (1 left) 2022-04-22T17:30:09.850 INFO:tasks.rados.rados.0.smithi115.stdout:1260: snap_create 2022-04-22T17:30:09.850 INFO:tasks.rados.rados.0.smithi115.stdout:1257: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:30:09.851 INFO:tasks.rados.rados.0.smithi115.stdout:1257: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:30:09.851 INFO:tasks.rados.rados.0.smithi115.stdout:1257: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:30:09.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 309 (ObjNum 316 snap 107 seq_num 316) dirty exists 2022-04-22T17:30:10.052 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:09 smithi115 systemd[1]: Started Ceph osd.0 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:30:10.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: audit 2022-04-22T17:30:09.173581+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:10.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: audit 2022-04-22T17:30:09.819091+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:10.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: audit 2022-04-22T17:30:09.820487+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:10.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: audit 2022-04-22T17:30:09.822544+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:10.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: audit 2022-04-22T17:30:09.823671+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:10.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[45495]: cluster 2022-04-22T17:30:09.833129+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e316: 8 total, 7 up, 8 in 2022-04-22T17:30:10.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: audit 2022-04-22T17:30:09.173581+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: audit 2022-04-22T17:30:09.819091+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: audit 2022-04-22T17:30:09.820487+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: audit 2022-04-22T17:30:09.822544+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: audit 2022-04-22T17:30:09.823671+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:10.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:09 smithi115 bash[46934]: cluster 2022-04-22T17:30:09.833129+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e316: 8 total, 7 up, 8 in 2022-04-22T17:30:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: audit 2022-04-22T17:30:09.173581+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: audit 2022-04-22T17:30:09.819091+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: audit 2022-04-22T17:30:09.820487+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: audit 2022-04-22T17:30:09.822544+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: audit 2022-04-22T17:30:09.823671+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:09 smithi121 bash[36902]: cluster 2022-04-22T17:30:09.833129+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e316: 8 total, 7 up, 8 in 2022-04-22T17:30:10.849 INFO:tasks.rados.rados.0.smithi115.stdout:1257: done (1 left) 2022-04-22T17:30:10.850 INFO:tasks.rados.rados.0.smithi115.stdout:1260: done (0 left) 2022-04-22T17:30:10.850 INFO:tasks.rados.rados.0.smithi115.stdout:1261: snap_create 2022-04-22T17:30:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:10 smithi121 bash[36902]: cluster 2022-04-22T17:30:09.160803+0000 mgr.y (mgr.34107) 117 : cluster [DBG] pgmap v159: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:10 smithi121 bash[36902]: cluster 2022-04-22T17:30:09.857508+0000 mon.a (mon.0) 337 : cluster [WRN] Health check failed: Degraded data redundancy: 91/543 objects degraded (16.759%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:10 smithi121 bash[36902]: cluster 2022-04-22T17:30:10.835044+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e317: 8 total, 7 up, 8 in 2022-04-22T17:30:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[45495]: cluster 2022-04-22T17:30:09.160803+0000 mgr.y (mgr.34107) 117 : cluster [DBG] pgmap v159: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[45495]: cluster 2022-04-22T17:30:09.857508+0000 mon.a (mon.0) 337 : cluster [WRN] Health check failed: Degraded data redundancy: 91/543 objects degraded (16.759%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[45495]: cluster 2022-04-22T17:30:10.835044+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e317: 8 total, 7 up, 8 in 2022-04-22T17:30:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[46934]: cluster 2022-04-22T17:30:09.160803+0000 mgr.y (mgr.34107) 117 : cluster [DBG] pgmap v159: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[46934]: cluster 2022-04-22T17:30:09.857508+0000 mon.a (mon.0) 337 : cluster [WRN] Health check failed: Degraded data redundancy: 91/543 objects degraded (16.759%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:10 smithi115 bash[46934]: cluster 2022-04-22T17:30:10.835044+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e317: 8 total, 7 up, 8 in 2022-04-22T17:30:11.857 INFO:tasks.rados.rados.0.smithi115.stdout:1261: done (0 left) 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1262: read oid 4 snap -1 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1262: expect deleted 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1263: read oid 46 snap -1 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1263: expect (ObjNum 306 snap 101 seq_num 306) 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1264: rollback oid 24 current snap is 132 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 89 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1265: read oid 20 snap -1 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1265: expect deleted 2022-04-22T17:30:11.858 INFO:tasks.rados.rados.0.smithi115.stdout:1266: rmattr oid 3 current snap is 132 2022-04-22T17:30:11.859 INFO:tasks.rados.rados.0.smithi115.stdout:1267: copy_from oid 1 from oid 19 current snap is 132 2022-04-22T17:30:11.859 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 132 2022-04-22T17:30:11.859 INFO:tasks.rados.rados.0.smithi115.stdout:1268: seq_num 378 ranges {5062656=737280,5799936=720896,6520832=81920} 2022-04-22T17:30:11.889 INFO:tasks.rados.rados.0.smithi115.stdout:1268: writing smithi11541303-40 from 5062656 to 5799936 tid 1 2022-04-22T17:30:11.894 INFO:tasks.rados.rados.0.smithi115.stdout:1268: writing smithi11541303-40 from 5799936 to 6520832 tid 2 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1268: writing smithi11541303-40 from 6520832 to 6602752 tid 3 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1269: copy_from oid 41 from oid 35 current snap is 132 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1264: finishing rollback tid 0 to smithi11541303-24 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1264: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1262: done (7 left) 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1270: read oid 18 snap -1 2022-04-22T17:30:11.896 INFO:tasks.rados.rados.0.smithi115.stdout:1270: expect deleted 2022-04-22T17:30:11.897 INFO:tasks.rados.rados.0.smithi115.stdout:1271: rollback oid 5 current snap is 132 2022-04-22T17:30:11.897 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 129 2022-04-22T17:30:11.897 INFO:tasks.rados.rados.0.smithi115.stdout:1264: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:30:11.897 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 257 (ObjNum 236 snap 74 seq_num 236) dirty exists 2022-04-22T17:30:11.897 INFO:tasks.rados.rados.0.smithi115.stdout:1263: expect (ObjNum 306 snap 101 seq_num 306) 2022-04-22T17:30:11.899 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-04-22T17:30:11.899 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2022-04-22T17:30:11.899 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2022-04-22T17:30:11.899 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2022-04-22T17:30:11.900 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2022-04-22T17:30:11.900 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-04-22T17:30:11.900 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[49357]: --> ceph-volume raw activate successful for osd ID: 0 2022-04-22T17:30:11.901 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:11.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:11.902 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:11.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:11.902 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:11.620Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:30:11.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 624 (ObjNum 348 snap 121 seq_num 348) dirty exists 2022-04-22T17:30:11.944 INFO:tasks.rados.rados.0.smithi115.stdout:1267: finishing copy_from to smithi11541303-1 2022-04-22T17:30:11.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 472 (ObjNum 365 snap 123 seq_num 365) dirty exists 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1267: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1271: finishing rollback tid 0 to smithi11541303-5 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1271: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1271: finishing rollback tid 2 to smithi11541303-5 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 734 (ObjNum 306 snap 101 seq_num 306) dirty exists 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1268: finishing write tid 1 to smithi11541303-40 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1263: done (8 left) 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1264: done (7 left) 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1265: done (6 left) 2022-04-22T17:30:11.945 INFO:tasks.rados.rados.0.smithi115.stdout:1266: done (5 left) 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1267: done (4 left) 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1270: done (3 left) 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1271: done (2 left) 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1272: rollback oid 44 current snap is 132 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 124 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1268: finishing write tid 2 to smithi11541303-40 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1268: finishing write tid 3 to smithi11541303-40 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1268: finishing write tid 4 to smithi11541303-40 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:1268: finishing write tid 5 to smithi11541303-40 2022-04-22T17:30:11.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 503 (ObjNum 378 snap 132 seq_num 378) dirty exists 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1268: left oid 40 (ObjNum 378 snap 132 seq_num 378) 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1269: finishing copy_from to smithi11541303-41 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 631 (ObjNum 369 snap 127 seq_num 369) dirty exists 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1269: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1268: done (2 left) 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1269: done (1 left) 2022-04-22T17:30:11.947 INFO:tasks.rados.rados.0.smithi115.stdout:1273: snap_create 2022-04-22T17:30:11.948 INFO:tasks.rados.rados.0.smithi115.stdout:1272: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:30:11.948 INFO:tasks.rados.rados.0.smithi115.stdout:1272: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:30:11.948 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 61 (ObjNum 1768451675 snap 1802331763 seq_num 1851745366) dirty exists 2022-04-22T17:30:12.896 INFO:tasks.rados.rados.0.smithi115.stdout:1272: done (1 left) 2022-04-22T17:30:12.896 INFO:tasks.rados.rados.0.smithi115.stdout:1273: done (0 left) 2022-04-22T17:30:12.896 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 133 2022-04-22T17:30:12.896 INFO:tasks.rados.rados.0.smithi115.stdout:1274: seq_num 379 ranges {1089536=753664,1843200=606208,2449408=196608} 2022-04-22T17:30:12.906 INFO:tasks.rados.rados.0.smithi115.stdout:1274: writing smithi11541303-5 from 1089536 to 1843200 tid 1 2022-04-22T17:30:12.910 INFO:tasks.rados.rados.0.smithi115.stdout:1274: writing smithi11541303-5 from 1843200 to 2449408 tid 2 2022-04-22T17:30:12.911 INFO:tasks.rados.rados.0.smithi115.stdout:1274: writing smithi11541303-5 from 2449408 to 2646016 tid 3 2022-04-22T17:30:12.911 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 133 2022-04-22T17:30:12.911 INFO:tasks.rados.rados.0.smithi115.stdout:1275: seq_num 380 ranges {9928704=770048,10698752=483328,11182080=319488} 2022-04-22T17:30:12.974 INFO:tasks.rados.rados.0.smithi115.stdout:1275: writing smithi11541303-10 from 9928704 to 10698752 tid 1 2022-04-22T17:30:12.977 INFO:tasks.rados.rados.0.smithi115.stdout:1275: writing smithi11541303-10 from 10698752 to 11182080 tid 2 2022-04-22T17:30:12.981 INFO:tasks.rados.rados.0.smithi115.stdout:1275: writing smithi11541303-10 from 11182080 to 11501568 tid 3 2022-04-22T17:30:12.981 INFO:tasks.rados.rados.0.smithi115.stdout:1276: rmattr oid 30 current snap is 133 2022-04-22T17:30:12.981 INFO:tasks.rados.rados.0.smithi115.stdout:1276: done (2 left) 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1277: copy_from oid 33 from oid 50 current snap is 133 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: finishing write tid 1 to smithi11541303-5 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: finishing write tid 2 to smithi11541303-5 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: finishing write tid 3 to smithi11541303-5 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: finishing write tid 4 to smithi11541303-5 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: finishing write tid 5 to smithi11541303-5 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 739 (ObjNum 379 snap 133 seq_num 379) dirty exists 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: left oid 5 (ObjNum 379 snap 133 seq_num 379) 2022-04-22T17:30:12.982 INFO:tasks.rados.rados.0.smithi115.stdout:1274: done (2 left) 2022-04-22T17:30:12.983 INFO:tasks.rados.rados.0.smithi115.stdout:1278: read oid 28 snap -1 2022-04-22T17:30:12.983 INFO:tasks.rados.rados.0.smithi115.stdout:1278: expect deleted 2022-04-22T17:30:12.983 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 133 2022-04-22T17:30:12.983 INFO:tasks.rados.rados.0.smithi115.stdout:1279: seq_num 381 ranges {0=786432,786432=417792,1204224=385024} 2022-04-22T17:30:12.986 INFO:tasks.rados.rados.0.smithi115.stdout:1279: writing smithi11541303-20 from 0 to 786432 tid 1 2022-04-22T17:30:12.989 INFO:tasks.rados.rados.0.smithi115.stdout:1279: writing smithi11541303-20 from 786432 to 1204224 tid 2 2022-04-22T17:30:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:1279: writing smithi11541303-20 from 1204224 to 1589248 tid 3 2022-04-22T17:30:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:1280: copy_from oid 36 from oid 38 current snap is 133 2022-04-22T17:30:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:1275: finishing write tid 1 to smithi11541303-10 2022-04-22T17:30:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:1281: snap_remove snap 132 2022-04-22T17:30:13.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:12 smithi115 bash[46934]: cluster 2022-04-22T17:30:11.161678+0000 mgr.y (mgr.34107) 118 : cluster [DBG] pgmap v162: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:12 smithi115 bash[46934]: cluster 2022-04-22T17:30:11.836545+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e318: 8 total, 7 up, 8 in 2022-04-22T17:30:13.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:12 smithi115 bash[45495]: cluster 2022-04-22T17:30:11.161678+0000 mgr.y (mgr.34107) 118 : cluster [DBG] pgmap v162: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:13.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:12 smithi115 bash[45495]: cluster 2022-04-22T17:30:11.836545+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e318: 8 total, 7 up, 8 in 2022-04-22T17:30:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:12 smithi121 bash[36902]: cluster 2022-04-22T17:30:11.161678+0000 mgr.y (mgr.34107) 118 : cluster [DBG] pgmap v162: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.7 MiB/s wr, 11 op/s; 91/543 objects degraded (16.759%) 2022-04-22T17:30:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:12 smithi121 bash[36902]: cluster 2022-04-22T17:30:11.836545+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e318: 8 total, 7 up, 8 in 2022-04-22T17:30:13.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:13.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:13.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:13.515Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:13.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:13.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:13.902 INFO:tasks.rados.rados.0.smithi115.stdout:1281: done (5 left) 2022-04-22T17:30:13.902 INFO:tasks.rados.rados.0.smithi115.stdout:1282: snap_remove snap 123 2022-04-22T17:30:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:13 smithi121 bash[36902]: cluster 2022-04-22T17:30:12.881248+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e319: 8 total, 7 up, 8 in 2022-04-22T17:30:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:13 smithi121 bash[36902]: audit 2022-04-22T17:30:13.237498+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:30:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:13 smithi121 bash[36902]: audit 2022-04-22T17:30:13.242145+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:30:14.301 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[49534]: debug 2022-04-22T17:30:13.814+0000 7ff521ce4700 1 -- 172.21.15.115:0/1702863342 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b96c52c340 con 0x55b96c5b0c00 2022-04-22T17:30:14.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[46934]: cluster 2022-04-22T17:30:12.881248+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e319: 8 total, 7 up, 8 in 2022-04-22T17:30:14.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[46934]: audit 2022-04-22T17:30:13.237498+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:30:14.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[46934]: audit 2022-04-22T17:30:13.242145+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:30:14.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[45495]: cluster 2022-04-22T17:30:12.881248+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e319: 8 total, 7 up, 8 in 2022-04-22T17:30:14.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[45495]: audit 2022-04-22T17:30:13.237498+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:30:14.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:13 smithi115 bash[45495]: audit 2022-04-22T17:30:13.242145+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:30:14.471 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:30:14.768 INFO:tasks.rados.rados.0.smithi115.stdout:1282: done (5 left) 2022-04-22T17:30:14.768 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 133 2022-04-22T17:30:14.768 INFO:tasks.rados.rados.0.smithi115.stdout:1275: finishing write tid 2 to smithi11541303-10 2022-04-22T17:30:14.768 INFO:tasks.rados.rados.0.smithi115.stdout:1275: finishing write tid 3 to smithi11541303-10 2022-04-22T17:30:14.768 INFO:tasks.rados.rados.0.smithi115.stdout:1275: finishing write tid 4 to smithi11541303-10 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1275: finishing write tid 5 to smithi11541303-10 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 635 (ObjNum 380 snap 133 seq_num 380) dirty exists 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1275: left oid 10 (ObjNum 380 snap 133 seq_num 380) 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1279: finishing write tid 1 to smithi11541303-20 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1277: finishing copy_from to smithi11541303-33 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 380 (ObjNum 346 snap 120 seq_num 346) dirty exists 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1279: finishing write tid 2 to smithi11541303-20 2022-04-22T17:30:14.769 INFO:tasks.rados.rados.0.smithi115.stdout:1283: seq_num 382 ranges {2170880=802816,2973696=704512,3678208=98304} 2022-04-22T17:30:14.787 INFO:tasks.rados.rados.0.smithi115.stdout:1283: writing smithi11541303-29 from 2170880 to 2973696 tid 1 2022-04-22T17:30:14.795 INFO:tasks.rados.rados.0.smithi115.stdout:1283: writing smithi11541303-29 from 2973696 to 3678208 tid 2 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1283: writing smithi11541303-29 from 3678208 to 3776512 tid 3 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1275: done (5 left) 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1278: done (4 left) 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 133 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1277: finishing copy_from racing read to smithi11541303-33 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1279: finishing write tid 3 to smithi11541303-20 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1279: finishing write tid 4 to smithi11541303-20 2022-04-22T17:30:14.797 INFO:tasks.rados.rados.0.smithi115.stdout:1279: finishing write tid 5 to smithi11541303-20 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 557 (ObjNum 381 snap 133 seq_num 381) dirty exists 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:1279: left oid 20 (ObjNum 381 snap 133 seq_num 381) 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:1280: finishing copy_from to smithi11541303-36 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:1280: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:30:14.798 INFO:tasks.rados.rados.0.smithi115.stdout:1284: seq_num 383 ranges {9027584=417792,9445376=589824,10035200=417792,10452992=196608} 2022-04-22T17:30:14.801 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[49534]: debug 2022-04-22T17:30:14.366+0000 7ff5292db3c0 -1 Falling back to public interface 2022-04-22T17:30:14.852 INFO:tasks.rados.rados.0.smithi115.stdout:1284: writing smithi11541303-42 from 9027584 to 9445376 tid 1 2022-04-22T17:30:14.855 INFO:tasks.rados.rados.0.smithi115.stdout:1284: writing smithi11541303-42 from 9445376 to 10035200 tid 2 2022-04-22T17:30:14.858 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:30:14.858 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 0s ago 7m 12.9M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:30:14.858 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6m) 64s ago 6m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:30:14.858 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (8m) 0s ago 8m 13.7M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:30:14.858 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (8m) 64s ago 8m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2m) 64s ago 12m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3m) 0s ago 14m 444M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2m) 0s ago 14m 51.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (76s) 64s ago 12m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (99s) 0s ago 12m 43.6M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 0s ago 7m 9184k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (7m) 64s ago 7m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2s) 0s ago 11m - 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 0s ago 11m 203M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:30:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 0s ago 10m 177M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (10m) 0s ago 10m 116M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (10m) 64s ago 10m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (9m) 64s ago 9m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (9m) 64s ago 9m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (8m) 64s ago 8m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:30:14.860 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3m) 64s ago 7m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:30:14.860 INFO:tasks.rados.rados.0.smithi115.stdout:1284: writing smithi11541303-42 from 10035200 to 10452992 tid 3 2022-04-22T17:30:14.862 INFO:tasks.rados.rados.0.smithi115.stdout:1284: writing smithi11541303-42 from 10452992 to 10649600 tid 4 2022-04-22T17:30:14.862 INFO:tasks.rados.rados.0.smithi115.stdout:1277: done (4 left) 2022-04-22T17:30:14.862 INFO:tasks.rados.rados.0.smithi115.stdout:1279: done (3 left) 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1280: done (2 left) 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 133 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: finishing write tid 1 to smithi11541303-29 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: finishing write tid 2 to smithi11541303-29 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: finishing write tid 3 to smithi11541303-29 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: finishing write tid 4 to smithi11541303-29 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: finishing write tid 5 to smithi11541303-29 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 314 (ObjNum 382 snap 133 seq_num 382) dirty exists 2022-04-22T17:30:14.863 INFO:tasks.rados.rados.0.smithi115.stdout:1283: left oid 29 (ObjNum 382 snap 133 seq_num 382) 2022-04-22T17:30:14.864 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 1 to smithi11541303-42 2022-04-22T17:30:14.864 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 2 to smithi11541303-42 2022-04-22T17:30:14.864 INFO:tasks.rados.rados.0.smithi115.stdout:1285: seq_num 384 ranges {2670592=434176,3104768=475136,3579904=630784,4210688=98304} 2022-04-22T17:30:14.881 INFO:tasks.rados.rados.0.smithi115.stdout:1285: writing smithi11541303-48 from 2670592 to 3104768 tid 1 2022-04-22T17:30:14.886 INFO:tasks.rados.rados.0.smithi115.stdout:1285: writing smithi11541303-48 from 3104768 to 3579904 tid 2 2022-04-22T17:30:14.893 INFO:tasks.rados.rados.0.smithi115.stdout:1285: writing smithi11541303-48 from 3579904 to 4210688 tid 3 2022-04-22T17:30:14.894 INFO:tasks.rados.rados.0.smithi115.stdout:1285: writing smithi11541303-48 from 4210688 to 4308992 tid 4 2022-04-22T17:30:14.894 INFO:tasks.rados.rados.0.smithi115.stdout:1283: done (2 left) 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1286: copy_from oid 6 from oid 41 current snap is 133 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 3 to smithi11541303-42 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 4 to smithi11541303-42 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 5 to smithi11541303-42 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1284: finishing write tid 6 to smithi11541303-42 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 745 (ObjNum 383 snap 133 seq_num 383) dirty exists 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1284: left oid 42 (ObjNum 383 snap 133 seq_num 383) 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 1 to smithi11541303-48 2022-04-22T17:30:14.895 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 2 to smithi11541303-48 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:1284: done (2 left) 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:1287: copy_from oid 45 from oid 36 current snap is 133 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:1288: rollback oid 34 current snap is 133 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 89 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:1289: read oid 4 snap -1 2022-04-22T17:30:14.896 INFO:tasks.rados.rados.0.smithi115.stdout:1289: expect deleted 2022-04-22T17:30:14.897 INFO:tasks.rados.rados.0.smithi115.stdout:1290: snap_remove snap 114 2022-04-22T17:30:15.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[46934]: cluster 2022-04-22T17:30:13.162576+0000 mgr.y (mgr.34107) 119 : cluster [DBG] pgmap v165: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 91/543 objects degraded (16.759%) 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[46934]: cluster 2022-04-22T17:30:13.887927+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e320: 8 total, 7 up, 8 in 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[46934]: audit 2022-04-22T17:30:14.176007+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[46934]: cluster 2022-04-22T17:30:14.753294+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e321: 8 total, 7 up, 8 in 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[46934]: audit 2022-04-22T17:30:14.817914+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[45495]: cluster 2022-04-22T17:30:13.162576+0000 mgr.y (mgr.34107) 119 : cluster [DBG] pgmap v165: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 91/543 objects degraded (16.759%) 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[45495]: cluster 2022-04-22T17:30:13.887927+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e320: 8 total, 7 up, 8 in 2022-04-22T17:30:15.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[45495]: audit 2022-04-22T17:30:14.176007+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:15.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[45495]: cluster 2022-04-22T17:30:14.753294+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e321: 8 total, 7 up, 8 in 2022-04-22T17:30:15.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:14 smithi115 bash[45495]: audit 2022-04-22T17:30:14.817914+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:14 smithi121 bash[36902]: cluster 2022-04-22T17:30:13.162576+0000 mgr.y (mgr.34107) 119 : cluster [DBG] pgmap v165: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 300 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 91/543 objects degraded (16.759%) 2022-04-22T17:30:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:14 smithi121 bash[36902]: cluster 2022-04-22T17:30:13.887927+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e320: 8 total, 7 up, 8 in 2022-04-22T17:30:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:14 smithi121 bash[36902]: audit 2022-04-22T17:30:14.176007+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:14 smithi121 bash[36902]: cluster 2022-04-22T17:30:14.753294+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e321: 8 total, 7 up, 8 in 2022-04-22T17:30:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:14 smithi121 bash[36902]: audit 2022-04-22T17:30:14.817914+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:15.285 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 7 2022-04-22T17:30:15.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 9, 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 5 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:30:15.287 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:30:15.839 INFO:tasks.rados.rados.0.smithi115.stdout:1290: done (5 left) 2022-04-22T17:30:15.839 INFO:tasks.rados.rados.0.smithi115.stdout:1291: copy_from oid 11 from oid 17 current snap is 133 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1292: read oid 29 snap 130 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1292: expect (ObjNum 367 snap 125 seq_num 367) 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1293: rollback oid 47 current snap is 133 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 47 to 111 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1294: read oid 23 snap -1 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1294: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1295: rmattr oid 38 current snap is 133 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1296: read oid 32 snap -1 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:1296: expect deleted 2022-04-22T17:30:15.840 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 133 2022-04-22T17:30:15.841 INFO:tasks.rados.rados.0.smithi115.stdout:1297: seq_num 385 ranges {2957312=450560,3407872=761856,4169728=442368} 2022-04-22T17:30:15.860 INFO:tasks.rados.rados.0.smithi115.stdout:1297: writing smithi11541303-24 from 2957312 to 3407872 tid 1 2022-04-22T17:30:15.865 INFO:tasks.rados.rados.0.smithi115.stdout:1297: writing smithi11541303-24 from 3407872 to 4169728 tid 2 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1297: writing smithi11541303-24 from 4169728 to 4612096 tid 3 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1298: setattr oid 7 current snap is 133 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1288: finishing rollback tid 0 to smithi11541303-34 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 3 to smithi11541303-48 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1288: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1288: finishing rollback tid 2 to smithi11541303-34 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 391 (ObjNum 260 snap 87 seq_num 260) dirty exists 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 4 to smithi11541303-48 2022-04-22T17:30:15.869 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 5 to smithi11541303-48 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1285: finishing write tid 6 to smithi11541303-48 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 705 (ObjNum 384 snap 133 seq_num 384) dirty exists 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1285: left oid 48 (ObjNum 384 snap 133 seq_num 384) 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1287: finishing copy_from to smithi11541303-45 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1287: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1286: finishing copy_from to smithi11541303-6 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 631 (ObjNum 369 snap 127 seq_num 369) dirty exists 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:1286: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:30:15.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 474 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:15.871 INFO:tasks.rados.rados.0.smithi115.stdout:1293: finishing rollback tid 1 to smithi11541303-47 2022-04-22T17:30:15.871 INFO:tasks.rados.rados.0.smithi115.stdout:1293: finishing rollback tid 2 to smithi11541303-47 2022-04-22T17:30:15.871 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 637 (ObjNum 304 snap 101 seq_num 304) dirty exists 2022-04-22T17:30:15.871 INFO:tasks.rados.rados.0.smithi115.stdout:1292: expect (ObjNum 367 snap 125 seq_num 367) 2022-04-22T17:30:15.961 INFO:tasks.rados.rados.0.smithi115.stdout:1297: finishing write tid 1 to smithi11541303-24 2022-04-22T17:30:15.961 INFO:tasks.rados.rados.0.smithi115.stdout:1297: finishing write tid 2 to smithi11541303-24 2022-04-22T17:30:15.961 INFO:tasks.rados.rados.0.smithi115.stdout:1297: finishing write tid 3 to smithi11541303-24 2022-04-22T17:30:15.961 INFO:tasks.rados.rados.0.smithi115.stdout:1297: finishing write tid 4 to smithi11541303-24 2022-04-22T17:30:15.961 INFO:tasks.rados.rados.0.smithi115.stdout:1294: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1291: finishing copy_from to smithi11541303-11 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 547 (ObjNum 368 snap 127 seq_num 368) dirty exists 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1291: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1285: done (12 left) 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1286: done (11 left) 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1287: done (10 left) 2022-04-22T17:30:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:1288: done (9 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1289: done (8 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1291: done (7 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1292: done (6 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1293: done (5 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1294: done (4 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1295: done (3 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1296: done (2 left) 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1299: snap_create 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:1297: finishing write tid 5 to smithi11541303-24 2022-04-22T17:30:16.174 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 635 (ObjNum 385 snap 133 seq_num 385) dirty exists 2022-04-22T17:30:16.175 INFO:tasks.rados.rados.0.smithi115.stdout:1297: left oid 24 (ObjNum 385 snap 133 seq_num 385) 2022-04-22T17:30:16.175 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 473 (ObjNum 363 snap 123 seq_num 363) dirty exists 2022-04-22T17:30:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:15 smithi121 bash[36902]: audit 2022-04-22T17:30:14.438237+0000 mgr.y (mgr.34107) 120 : audit [DBG] from='client.44140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:15 smithi121 bash[36902]: audit 2022-04-22T17:30:14.822104+0000 mgr.y (mgr.34107) 121 : audit [DBG] from='client.44146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:15 smithi121 bash[36902]: audit 2022-04-22T17:30:15.271117+0000 mon.a (mon.0) 347 : audit [DBG] from='client.? 172.21.15.115:0/684841802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:15 smithi121 bash[36902]: audit 2022-04-22T17:30:15.414864+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:15 smithi121 bash[36902]: cluster 2022-04-22T17:30:15.823800+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e322: 8 total, 7 up, 8 in 2022-04-22T17:30:16.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[45495]: audit 2022-04-22T17:30:14.438237+0000 mgr.y (mgr.34107) 120 : audit [DBG] from='client.44140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[45495]: audit 2022-04-22T17:30:14.822104+0000 mgr.y (mgr.34107) 121 : audit [DBG] from='client.44146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[45495]: audit 2022-04-22T17:30:15.271117+0000 mon.a (mon.0) 347 : audit [DBG] from='client.? 172.21.15.115:0/684841802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[45495]: audit 2022-04-22T17:30:15.414864+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[45495]: cluster 2022-04-22T17:30:15.823800+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e322: 8 total, 7 up, 8 in 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[46934]: audit 2022-04-22T17:30:14.438237+0000 mgr.y (mgr.34107) 120 : audit [DBG] from='client.44140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[46934]: audit 2022-04-22T17:30:14.822104+0000 mgr.y (mgr.34107) 121 : audit [DBG] from='client.44146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[46934]: audit 2022-04-22T17:30:15.271117+0000 mon.a (mon.0) 347 : audit [DBG] from='client.? 172.21.15.115:0/684841802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:16.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[46934]: audit 2022-04-22T17:30:15.414864+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:16.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:15 smithi115 bash[46934]: cluster 2022-04-22T17:30:15.823800+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e322: 8 total, 7 up, 8 in 2022-04-22T17:30:16.922 INFO:tasks.rados.rados.0.smithi115.stdout:1297: done (2 left) 2022-04-22T17:30:16.922 INFO:tasks.rados.rados.0.smithi115.stdout:1298: done (1 left) 2022-04-22T17:30:16.922 INFO:tasks.rados.rados.0.smithi115.stdout:1299: done (0 left) 2022-04-22T17:30:16.922 INFO:tasks.rados.rados.0.smithi115.stdout:1300: rmattr oid 41 current snap is 134 2022-04-22T17:30:16.923 INFO:tasks.rados.rados.0.smithi115.stdout:1300: done (0 left) 2022-04-22T17:30:16.923 INFO:tasks.rados.rados.0.smithi115.stdout:1301: snap_remove snap 128 2022-04-22T17:30:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:16 smithi121 bash[36902]: cluster 2022-04-22T17:30:15.163279+0000 mgr.y (mgr.34107) 122 : cluster [DBG] pgmap v168: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:16 smithi121 bash[36902]: cluster 2022-04-22T17:30:15.883203+0000 mon.a (mon.0) 350 : cluster [WRN] Health check update: Degraded data redundancy: 95/570 objects degraded (16.667%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:17.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:16 smithi115 bash[46934]: cluster 2022-04-22T17:30:15.163279+0000 mgr.y (mgr.34107) 122 : cluster [DBG] pgmap v168: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:17.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:16 smithi115 bash[46934]: cluster 2022-04-22T17:30:15.883203+0000 mon.a (mon.0) 350 : cluster [WRN] Health check update: Degraded data redundancy: 95/570 objects degraded (16.667%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:16 smithi115 bash[45495]: cluster 2022-04-22T17:30:15.163279+0000 mgr.y (mgr.34107) 122 : cluster [DBG] pgmap v168: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:16 smithi115 bash[45495]: cluster 2022-04-22T17:30:15.883203+0000 mon.a (mon.0) 350 : cluster [WRN] Health check update: Degraded data redundancy: 95/570 objects degraded (16.667%), 15 pgs degraded (PG_DEGRADED) 2022-04-22T17:30:17.935 INFO:tasks.rados.rados.0.smithi115.stdout:1301: done (0 left) 2022-04-22T17:30:17.935 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 134 2022-04-22T17:30:17.935 INFO:tasks.rados.rados.0.smithi115.stdout:1302: seq_num 386 ranges {2850816=466944,3317760=638976,3956736=565248} 2022-04-22T17:30:17.956 INFO:tasks.rados.rados.0.smithi115.stdout:1302: writing smithi11541303-45 from 2850816 to 3317760 tid 1 2022-04-22T17:30:17.960 INFO:tasks.rados.rados.0.smithi115.stdout:1302: writing smithi11541303-45 from 3317760 to 3956736 tid 2 2022-04-22T17:30:17.964 INFO:tasks.rados.rados.0.smithi115.stdout:1302: writing smithi11541303-45 from 3956736 to 4521984 tid 3 2022-04-22T17:30:17.964 INFO:tasks.rados.rados.0.smithi115.stdout:1303: delete oid 26 current snap is 134 2022-04-22T17:30:17.965 INFO:tasks.rados.rados.0.smithi115.stdout:1302: finishing write tid 1 to smithi11541303-45 2022-04-22T17:30:17.969 INFO:tasks.rados.rados.0.smithi115.stdout:1303: done (1 left) 2022-04-22T17:30:17.969 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 134 2022-04-22T17:30:17.969 INFO:tasks.rados.rados.0.smithi115.stdout:1304: seq_num 388 ranges {0=507904,507904=458752,966656=606208,1572864=139264} 2022-04-22T17:30:17.972 INFO:tasks.rados.rados.0.smithi115.stdout:1304: writing smithi11541303-18 from 0 to 507904 tid 1 2022-04-22T17:30:17.975 INFO:tasks.rados.rados.0.smithi115.stdout:1304: writing smithi11541303-18 from 507904 to 966656 tid 2 2022-04-22T17:30:17.980 INFO:tasks.rados.rados.0.smithi115.stdout:1304: writing smithi11541303-18 from 966656 to 1572864 tid 3 2022-04-22T17:30:17.981 INFO:tasks.rados.rados.0.smithi115.stdout:1304: writing smithi11541303-18 from 1572864 to 1712128 tid 4 2022-04-22T17:30:17.981 INFO:tasks.rados.rados.0.smithi115.stdout:1305: delete oid 1 current snap is 134 2022-04-22T17:30:17.982 INFO:tasks.rados.rados.0.smithi115.stdout:1302: finishing write tid 2 to smithi11541303-45 2022-04-22T17:30:17.982 INFO:tasks.rados.rados.0.smithi115.stdout:1302: finishing write tid 3 to smithi11541303-45 2022-04-22T17:30:17.983 INFO:tasks.rados.rados.0.smithi115.stdout:1302: finishing write tid 4 to smithi11541303-45 2022-04-22T17:30:17.984 INFO:tasks.rados.rados.0.smithi115.stdout:1302: finishing write tid 5 to smithi11541303-45 2022-04-22T17:30:17.984 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 726 (ObjNum 386 snap 134 seq_num 386) dirty exists 2022-04-22T17:30:17.984 INFO:tasks.rados.rados.0.smithi115.stdout:1302: left oid 45 (ObjNum 386 snap 134 seq_num 386) 2022-04-22T17:30:17.985 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 1 to smithi11541303-18 2022-04-22T17:30:17.985 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 2 to smithi11541303-18 2022-04-22T17:30:17.986 INFO:tasks.rados.rados.0.smithi115.stdout:1302: done (2 left) 2022-04-22T17:30:17.986 INFO:tasks.rados.rados.0.smithi115.stdout:1305: done (1 left) 2022-04-22T17:30:17.986 INFO:tasks.rados.rados.0.smithi115.stdout:1306: copy_from oid 4 from oid 1 current snap is 134 2022-04-22T17:30:17.986 INFO:tasks.rados.rados.0.smithi115.stdout:1307: snap_create 2022-04-22T17:30:17.987 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 3 to smithi11541303-18 2022-04-22T17:30:17.987 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 4 to smithi11541303-18 2022-04-22T17:30:17.987 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 5 to smithi11541303-18 2022-04-22T17:30:17.989 INFO:tasks.rados.rados.0.smithi115.stdout:1304: finishing write tid 6 to smithi11541303-18 2022-04-22T17:30:17.989 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 750 (ObjNum 388 snap 134 seq_num 388) dirty exists 2022-04-22T17:30:17.989 INFO:tasks.rados.rados.0.smithi115.stdout:1304: left oid 18 (ObjNum 388 snap 134 seq_num 388) 2022-04-22T17:30:17.989 INFO:tasks.rados.rados.0.smithi115.stdout:1306: finishing copy_from to smithi11541303-4 2022-04-22T17:30:17.990 INFO:tasks.rados.rados.0.smithi115.stdout:1306: got expected ENOENT (src dne) 2022-04-22T17:30:17.990 INFO:tasks.rados.rados.0.smithi115.stdout:1306: finishing copy_from racing read to smithi11541303-4 2022-04-22T17:30:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:17 smithi121 bash[36902]: cluster 2022-04-22T17:30:16.907824+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e323: 8 total, 7 up, 8 in 2022-04-22T17:30:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:17 smithi121 bash[36902]: audit 2022-04-22T17:30:17.826197+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:17 smithi121 bash[36902]: audit 2022-04-22T17:30:17.826656+0000 mon.a (mon.0) 352 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[46934]: cluster 2022-04-22T17:30:16.907824+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e323: 8 total, 7 up, 8 in 2022-04-22T17:30:18.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[46934]: audit 2022-04-22T17:30:17.826197+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[46934]: audit 2022-04-22T17:30:17.826656+0000 mon.a (mon.0) 352 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.302 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[49534]: debug 2022-04-22T17:30:17.818+0000 7ff5292db3c0 -1 osd.0 312 log_to_monitors true 2022-04-22T17:30:18.302 INFO:journalctl@ceph.osd.0.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[49534]: debug 2022-04-22T17:30:17.962+0000 7ff51b678700 -1 osd.0 312 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:30:18.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[45495]: cluster 2022-04-22T17:30:16.907824+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e323: 8 total, 7 up, 8 in 2022-04-22T17:30:18.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[45495]: audit 2022-04-22T17:30:17.826197+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:17 smithi115 bash[45495]: audit 2022-04-22T17:30:17.826656+0000 mon.a (mon.0) 352 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-04-22T17:30:18.948 INFO:tasks.rados.rados.0.smithi115.stdout:1304: done (2 left) 2022-04-22T17:30:18.948 INFO:tasks.rados.rados.0.smithi115.stdout:1306: done (1 left) 2022-04-22T17:30:18.948 INFO:tasks.rados.rados.0.smithi115.stdout:1307: done (0 left) 2022-04-22T17:30:18.948 INFO:tasks.rados.rados.0.smithi115.stdout:1308: setattr oid 25 current snap is 135 2022-04-22T17:30:18.950 INFO:tasks.rados.rados.0.smithi115.stdout:1309: setattr oid 47 current snap is 135 2022-04-22T17:30:18.950 INFO:tasks.rados.rados.0.smithi115.stdout:1310: snap_create 2022-04-22T17:30:18.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 638 (ObjNum 304 snap 101 seq_num 304) dirty exists 2022-04-22T17:30:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:18 smithi121 bash[36902]: cluster 2022-04-22T17:30:17.164261+0000 mgr.y (mgr.34107) 123 : cluster [DBG] pgmap v171: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:18 smithi121 bash[36902]: audit 2022-04-22T17:30:17.920613+0000 mon.a (mon.0) 353 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:30:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:18 smithi121 bash[36902]: cluster 2022-04-22T17:30:17.920693+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e324: 8 total, 7 up, 8 in 2022-04-22T17:30:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:18 smithi121 bash[36902]: audit 2022-04-22T17:30:17.922442+0000 mon.c (mon.1) 8 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:18 smithi121 bash[36902]: audit 2022-04-22T17:30:17.922907+0000 mon.a (mon.0) 355 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[45495]: cluster 2022-04-22T17:30:17.164261+0000 mgr.y (mgr.34107) 123 : cluster [DBG] pgmap v171: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[45495]: audit 2022-04-22T17:30:17.920613+0000 mon.a (mon.0) 353 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:30:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[45495]: cluster 2022-04-22T17:30:17.920693+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e324: 8 total, 7 up, 8 in 2022-04-22T17:30:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[45495]: audit 2022-04-22T17:30:17.922442+0000 mon.c (mon.1) 8 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[45495]: audit 2022-04-22T17:30:17.922907+0000 mon.a (mon.0) 355 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[46934]: cluster 2022-04-22T17:30:17.164261+0000 mgr.y (mgr.34107) 123 : cluster [DBG] pgmap v171: 121 pgs: 15 active+undersized+degraded, 38 active+undersized, 68 active+clean; 315 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 799 KiB/s rd, 3.6 MiB/s wr, 10 op/s; 95/570 objects degraded (16.667%) 2022-04-22T17:30:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[46934]: audit 2022-04-22T17:30:17.920613+0000 mon.a (mon.0) 353 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-04-22T17:30:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[46934]: cluster 2022-04-22T17:30:17.920693+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e324: 8 total, 7 up, 8 in 2022-04-22T17:30:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[46934]: audit 2022-04-22T17:30:17.922442+0000 mon.c (mon.1) 8 : audit [INF] from='osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:18 smithi115 bash[46934]: audit 2022-04-22T17:30:17.922907+0000 mon.a (mon.0) 355 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:30:19.965 INFO:tasks.rados.rados.0.smithi115.stdout:1309: done (2 left) 2022-04-22T17:30:19.965 INFO:tasks.rados.rados.0.smithi115.stdout:1310: done (1 left) 2022-04-22T17:30:19.965 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 136 2022-04-22T17:30:19.965 INFO:tasks.rados.rados.0.smithi115.stdout:1311: seq_num 390 ranges {5095424=540672,5636096=630784,6266880=573440} 2022-04-22T17:30:19.996 INFO:tasks.rados.rados.0.smithi115.stdout:1311: writing smithi11541303-37 from 5095424 to 5636096 tid 1 2022-04-22T17:30:20.000 INFO:tasks.rados.rados.0.smithi115.stdout:1311: writing smithi11541303-37 from 5636096 to 6266880 tid 2 2022-04-22T17:30:20.004 INFO:tasks.rados.rados.0.smithi115.stdout:1311: writing smithi11541303-37 from 6266880 to 6840320 tid 3 2022-04-22T17:30:20.004 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 136 2022-04-22T17:30:20.005 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 559 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:30:20.005 INFO:tasks.rados.rados.0.smithi115.stdout:1312: seq_num 391 ranges {1318912=557056,1875968=516096,2392064=688128} 2022-04-22T17:30:20.016 INFO:tasks.rados.rados.0.smithi115.stdout:1312: writing smithi11541303-19 from 1318912 to 1875968 tid 1 2022-04-22T17:30:20.019 INFO:tasks.rados.rados.0.smithi115.stdout:1312: writing smithi11541303-19 from 1875968 to 2392064 tid 2 2022-04-22T17:30:20.024 INFO:tasks.rados.rados.0.smithi115.stdout:1312: writing smithi11541303-19 from 2392064 to 3080192 tid 3 2022-04-22T17:30:20.024 INFO:tasks.rados.rados.0.smithi115.stdout:1308: done (2 left) 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 136 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: finishing write tid 1 to smithi11541303-37 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: finishing write tid 2 to smithi11541303-37 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: finishing write tid 3 to smithi11541303-37 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: finishing write tid 4 to smithi11541303-37 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: finishing write tid 5 to smithi11541303-37 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 635 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1311: left oid 37 (ObjNum 390 snap 136 seq_num 390) 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1312: finishing write tid 1 to smithi11541303-19 2022-04-22T17:30:20.025 INFO:tasks.rados.rados.0.smithi115.stdout:1312: finishing write tid 2 to smithi11541303-19 2022-04-22T17:30:20.026 INFO:tasks.rados.rados.0.smithi115.stdout:1313: seq_num 392 ranges {1966080=573440,2539520=802816,3342336=401408} 2022-04-22T17:30:20.039 INFO:tasks.rados.rados.0.smithi115.stdout:1313: writing smithi11541303-34 from 1966080 to 2539520 tid 1 2022-04-22T17:30:20.044 INFO:tasks.rados.rados.0.smithi115.stdout:1313: writing smithi11541303-34 from 2539520 to 3342336 tid 2 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1313: writing smithi11541303-34 from 3342336 to 3743744 tid 3 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1311: done (2 left) 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1314: snap_create 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1312: finishing write tid 3 to smithi11541303-19 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1312: finishing write tid 4 to smithi11541303-19 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:1312: finishing write tid 5 to smithi11541303-19 2022-04-22T17:30:20.047 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 487 (ObjNum 391 snap 136 seq_num 391) dirty exists 2022-04-22T17:30:20.048 INFO:tasks.rados.rados.0.smithi115.stdout:1312: left oid 19 (ObjNum 391 snap 136 seq_num 391) 2022-04-22T17:30:20.048 INFO:tasks.rados.rados.0.smithi115.stdout:1313: finishing write tid 1 to smithi11541303-34 2022-04-22T17:30:20.049 INFO:tasks.rados.rados.0.smithi115.stdout:1313: finishing write tid 2 to smithi11541303-34 2022-04-22T17:30:20.050 INFO:tasks.rados.rados.0.smithi115.stdout:1313: finishing write tid 3 to smithi11541303-34 2022-04-22T17:30:20.050 INFO:tasks.rados.rados.0.smithi115.stdout:1313: finishing write tid 4 to smithi11541303-34 2022-04-22T17:30:20.052 INFO:tasks.rados.rados.0.smithi115.stdout:1313: finishing write tid 5 to smithi11541303-34 2022-04-22T17:30:20.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 395 (ObjNum 392 snap 136 seq_num 392) dirty exists 2022-04-22T17:30:20.052 INFO:tasks.rados.rados.0.smithi115.stdout:1313: left oid 34 (ObjNum 392 snap 136 seq_num 392) 2022-04-22T17:30:20.147 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[45495]: cluster 2022-04-22T17:30:18.917557+0000 mon.a (mon.0) 356 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[45495]: cluster 2022-04-22T17:30:18.932029+0000 mon.a (mon.0) 357 : cluster [INF] osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629] boot 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[45495]: cluster 2022-04-22T17:30:18.932095+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[45495]: audit 2022-04-22T17:30:18.932489+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[45495]: audit 2022-04-22T17:30:19.178766+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[46934]: cluster 2022-04-22T17:30:18.917557+0000 mon.a (mon.0) 356 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:30:20.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[46934]: cluster 2022-04-22T17:30:18.932029+0000 mon.a (mon.0) 357 : cluster [INF] osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629] boot 2022-04-22T17:30:20.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[46934]: cluster 2022-04-22T17:30:18.932095+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2022-04-22T17:30:20.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[46934]: audit 2022-04-22T17:30:18.932489+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:30:20.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:19 smithi115 bash[46934]: audit 2022-04-22T17:30:19.178766+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:19 smithi121 bash[36902]: cluster 2022-04-22T17:30:18.917557+0000 mon.a (mon.0) 356 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:30:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:19 smithi121 bash[36902]: cluster 2022-04-22T17:30:18.932029+0000 mon.a (mon.0) 357 : cluster [INF] osd.0 [v2:172.21.15.115:6802/3548478629,v1:172.21.15.115:6803/3548478629] boot 2022-04-22T17:30:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:19 smithi121 bash[36902]: cluster 2022-04-22T17:30:18.932095+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2022-04-22T17:30:20.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:19 smithi121 bash[36902]: audit 2022-04-22T17:30:18.932489+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-04-22T17:30:20.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:19 smithi121 bash[36902]: audit 2022-04-22T17:30:19.178766+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:20.964 INFO:tasks.rados.rados.0.smithi115.stdout:1312: done (2 left) 2022-04-22T17:30:20.965 INFO:tasks.rados.rados.0.smithi115.stdout:1313: done (1 left) 2022-04-22T17:30:20.965 INFO:tasks.rados.rados.0.smithi115.stdout:1314: done (0 left) 2022-04-22T17:30:20.965 INFO:tasks.rados.rados.0.smithi115.stdout:1315: delete oid 40 current snap is 137 2022-04-22T17:30:20.969 INFO:tasks.rados.rados.0.smithi115.stdout:1315: done (0 left) 2022-04-22T17:30:20.969 INFO:tasks.rados.rados.0.smithi115.stdout:1316: snap_remove snap 124 2022-04-22T17:30:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:20 smithi121 bash[36902]: cluster 2022-04-22T17:30:19.165469+0000 mgr.y (mgr.34107) 124 : cluster [DBG] pgmap v174: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:20 smithi121 bash[36902]: cluster 2022-04-22T17:30:19.940171+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2022-04-22T17:30:21.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:20 smithi115 bash[45495]: cluster 2022-04-22T17:30:19.165469+0000 mgr.y (mgr.34107) 124 : cluster [DBG] pgmap v174: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:21.334 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:20 smithi115 bash[45495]: cluster 2022-04-22T17:30:19.940171+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2022-04-22T17:30:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:20 smithi115 bash[46934]: cluster 2022-04-22T17:30:19.165469+0000 mgr.y (mgr.34107) 124 : cluster [DBG] pgmap v174: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:20 smithi115 bash[46934]: cluster 2022-04-22T17:30:19.940171+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2022-04-22T17:30:21.952 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:21.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:21.952 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:21.620Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:21.953 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:21.620Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:21.972 INFO:tasks.rados.rados.0.smithi115.stdout:1316: done (0 left) 2022-04-22T17:30:21.972 INFO:tasks.rados.rados.0.smithi115.stdout:1317: snap_remove snap 90 2022-04-22T17:30:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:21 smithi121 bash[36902]: cluster 2022-04-22T17:30:20.949915+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2022-04-22T17:30:22.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:21 smithi115 bash[45495]: cluster 2022-04-22T17:30:20.949915+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2022-04-22T17:30:22.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:21 smithi115 bash[46934]: cluster 2022-04-22T17:30:20.949915+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2022-04-22T17:30:22.973 INFO:tasks.rados.rados.0.smithi115.stdout:1317: done (0 left) 2022-04-22T17:30:23.054 INFO:tasks.rados.rados.0.smithi115.stdout:1318: snap_remove snap 89 2022-04-22T17:30:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:22 smithi121 bash[36902]: cluster 2022-04-22T17:30:21.166334+0000 mgr.y (mgr.34107) 125 : cluster [DBG] pgmap v177: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:22 smithi121 bash[36902]: cluster 2022-04-22T17:30:21.957275+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2022-04-22T17:30:23.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:22 smithi115 bash[45495]: cluster 2022-04-22T17:30:21.166334+0000 mgr.y (mgr.34107) 125 : cluster [DBG] pgmap v177: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:23.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:22 smithi115 bash[45495]: cluster 2022-04-22T17:30:21.957275+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2022-04-22T17:30:23.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:22 smithi115 bash[46934]: cluster 2022-04-22T17:30:21.166334+0000 mgr.y (mgr.34107) 125 : cluster [DBG] pgmap v177: 121 pgs: 38 peering, 4 active+undersized+degraded, 11 active+undersized, 68 active+clean; 328 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 17/588 objects degraded (2.891%) 2022-04-22T17:30:23.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:22 smithi115 bash[46934]: cluster 2022-04-22T17:30:21.957275+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2022-04-22T17:30:23.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:23.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:23.871 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:23.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:23.871 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:23.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:23.978 INFO:tasks.rados.rados.0.smithi115.stdout:1318: done (0 left) 2022-04-22T17:30:23.979 INFO:tasks.rados.rados.0.smithi115.stdout:1319: delete oid 34 current snap is 137 2022-04-22T17:30:23.992 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:23 smithi115 bash[45495]: cluster 2022-04-22T17:30:22.958919+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2022-04-22T17:30:24.005 INFO:tasks.rados.rados.0.smithi115.stdout:1319: done (0 left) 2022-04-22T17:30:24.006 INFO:tasks.rados.rados.0.smithi115.stdout:1320: read oid 33 snap -1 2022-04-22T17:30:24.006 INFO:tasks.rados.rados.0.smithi115.stdout:1320: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:24.006 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 137 2022-04-22T17:30:24.007 INFO:tasks.rados.rados.0.smithi115.stdout:1321: seq_num 395 ranges {4308992=622592,4931584=450560,5382144=753664} 2022-04-22T17:30:24.032 INFO:tasks.rados.rados.0.smithi115.stdout:1321: writing smithi11541303-48 from 4308992 to 4931584 tid 1 2022-04-22T17:30:24.034 INFO:tasks.rados.rados.0.smithi115.stdout:1321: writing smithi11541303-48 from 4931584 to 5382144 tid 2 2022-04-22T17:30:24.039 INFO:tasks.rados.rados.0.smithi115.stdout:1321: writing smithi11541303-48 from 5382144 to 6135808 tid 3 2022-04-22T17:30:24.039 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 137 2022-04-22T17:30:24.039 INFO:tasks.rados.rados.0.smithi115.stdout:1322: seq_num 396 ranges {2646016=638976,3284992=786432,4071424=417792} 2022-04-22T17:30:24.057 INFO:tasks.rados.rados.0.smithi115.stdout:1322: writing smithi11541303-5 from 2646016 to 3284992 tid 1 2022-04-22T17:30:24.061 INFO:tasks.rados.rados.0.smithi115.stdout:1322: writing smithi11541303-5 from 3284992 to 4071424 tid 2 2022-04-22T17:30:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:1322: writing smithi11541303-5 from 4071424 to 4489216 tid 3 2022-04-22T17:30:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:1323: read oid 42 snap -1 2022-04-22T17:30:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:1323: expect (ObjNum 383 snap 133 seq_num 383) 2022-04-22T17:30:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:1324: delete oid 15 current snap is 137 2022-04-22T17:30:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:1320: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:24.142 INFO:tasks.rados.rados.0.smithi115.stdout:1321: finishing write tid 1 to smithi11541303-48 2022-04-22T17:30:24.143 INFO:tasks.rados.rados.0.smithi115.stdout:1321: finishing write tid 2 to smithi11541303-48 2022-04-22T17:30:24.144 INFO:tasks.rados.rados.0.smithi115.stdout:1320: done (4 left) 2022-04-22T17:30:24.144 INFO:tasks.rados.rados.0.smithi115.stdout:1324: done (3 left) 2022-04-22T17:30:24.145 INFO:tasks.rados.rados.0.smithi115.stdout:1325: rollback oid 31 current snap is 137 2022-04-22T17:30:24.145 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 121 2022-04-22T17:30:24.145 INFO:tasks.rados.rados.0.smithi115.stdout:1321: finishing write tid 3 to smithi11541303-48 2022-04-22T17:30:24.146 INFO:tasks.rados.rados.0.smithi115.stdout:1321: finishing write tid 4 to smithi11541303-48 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1321: finishing write tid 5 to smithi11541303-48 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 755 (ObjNum 395 snap 137 seq_num 395) dirty exists 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1321: left oid 48 (ObjNum 395 snap 137 seq_num 395) 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: finishing write tid 1 to smithi11541303-5 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: finishing write tid 2 to smithi11541303-5 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: finishing write tid 3 to smithi11541303-5 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: finishing write tid 4 to smithi11541303-5 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: finishing write tid 5 to smithi11541303-5 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 791 (ObjNum 396 snap 137 seq_num 396) dirty exists 2022-04-22T17:30:24.147 INFO:tasks.rados.rados.0.smithi115.stdout:1322: left oid 5 (ObjNum 396 snap 137 seq_num 396) 2022-04-22T17:30:24.148 INFO:tasks.rados.rados.0.smithi115.stdout:1323: expect (ObjNum 383 snap 133 seq_num 383) 2022-04-22T17:30:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:23 smithi121 bash[36902]: cluster 2022-04-22T17:30:22.958919+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2022-04-22T17:30:24.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:30:23 smithi121 bash[18604]: debug 2022-04-22T17:30:23.981+0000 7faa0f096700 -1 osd.5 pg_epoch: 330 pg[5.7s0( v 329'780 (0'0,329'780] local-lis/les=325/326 n=17 ec=63/63 lis/c=325/325 les/c/f=326/327/0 sis=325) [5,1,0]p5(0) r=0 lpr=325 crt=329'780 lcod 329'778 mlcod 329'778 active+clean+snaptrim trimq=[a8~1] ps=[aa~1,b4~1,d3~1,e4~1,e8~2,ef~1,f2~1,fc~1]] removing snap head 2022-04-22T17:30:24.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:23 smithi115 bash[46934]: cluster 2022-04-22T17:30:22.958919+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2022-04-22T17:30:24.802 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[42914]: debug 2022-04-22T17:30:24.377+0000 7f8eea62c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2022-04-22T17:30:24.971 INFO:tasks.rados.rados.0.smithi115.stdout:1325: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:30:24.971 INFO:tasks.rados.rados.0.smithi115.stdout:1325: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:30:24.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 793 (ObjNum 304 snap 101 seq_num 304) dirty exists 2022-04-22T17:30:24.971 INFO:tasks.rados.rados.0.smithi115.stdout:1321: done (3 left) 2022-04-22T17:30:24.972 INFO:tasks.rados.rados.0.smithi115.stdout:1322: done (2 left) 2022-04-22T17:30:24.972 INFO:tasks.rados.rados.0.smithi115.stdout:1323: done (1 left) 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:1325: done (0 left) 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:1326: read oid 17 snap 133 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:1326: expect (ObjNum 368 snap 127 seq_num 368) 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:1327: copy_from oid 48 from oid 10 current snap is 137 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 137 2022-04-22T17:30:24.973 INFO:tasks.rados.rados.0.smithi115.stdout:1328: seq_num 398 ranges {1589248=671744,2260992=557056,2818048=532480,3350528=114688} 2022-04-22T17:30:24.991 INFO:tasks.rados.rados.0.smithi115.stdout:1328: writing smithi11541303-20 from 1589248 to 2260992 tid 1 2022-04-22T17:30:24.994 INFO:tasks.rados.rados.0.smithi115.stdout:1328: writing smithi11541303-20 from 2260992 to 2818048 tid 2 2022-04-22T17:30:24.998 INFO:tasks.rados.rados.0.smithi115.stdout:1328: writing smithi11541303-20 from 2818048 to 3350528 tid 3 2022-04-22T17:30:24.999 INFO:tasks.rados.rados.0.smithi115.stdout:1328: writing smithi11541303-20 from 3350528 to 3465216 tid 4 2022-04-22T17:30:24.999 INFO:tasks.rados.rados.0.smithi115.stdout:1329: copy_from oid 11 from oid 36 current snap is 137 2022-04-22T17:30:24.999 INFO:tasks.rados.rados.0.smithi115.stdout:1326: expect (ObjNum 368 snap 127 seq_num 368) 2022-04-22T17:30:25.044 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 1 to smithi11541303-20 2022-04-22T17:30:25.044 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 2 to smithi11541303-20 2022-04-22T17:30:25.044 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 3 to smithi11541303-20 2022-04-22T17:30:25.045 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 4 to smithi11541303-20 2022-04-22T17:30:25.045 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 5 to smithi11541303-20 2022-04-22T17:30:25.045 INFO:tasks.rados.rados.0.smithi115.stdout:1328: finishing write tid 6 to smithi11541303-20 2022-04-22T17:30:25.045 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 599 (ObjNum 398 snap 137 seq_num 398) dirty exists 2022-04-22T17:30:25.045 INFO:tasks.rados.rados.0.smithi115.stdout:1328: left oid 20 (ObjNum 398 snap 137 seq_num 398) 2022-04-22T17:30:25.046 INFO:tasks.rados.rados.0.smithi115.stdout:1329: finishing copy_from to smithi11541303-11 2022-04-22T17:30:25.046 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:25.046 INFO:tasks.rados.rados.0.smithi115.stdout:1329: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:30:25.046 INFO:tasks.rados.rados.0.smithi115.stdout:1327: finishing copy_from to smithi11541303-48 2022-04-22T17:30:25.046 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 635 (ObjNum 380 snap 133 seq_num 380) dirty exists 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1327: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1326: done (3 left) 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1327: done (2 left) 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1328: done (1 left) 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1329: done (0 left) 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1330: read oid 36 snap -1 2022-04-22T17:30:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:1330: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:30:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:1331: snap_remove snap 111 2022-04-22T17:30:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: cluster 2022-04-22T17:30:23.167209+0000 mgr.y (mgr.34107) 126 : cluster [DBG] pgmap v180: 121 pgs: 38 peering, 83 active+clean; 332 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 434 KiB/s wr, 1 op/s; 1.5 MiB/s, 0 objects/s recovering 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: cluster 2022-04-22T17:30:23.962524+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: cluster 2022-04-22T17:30:23.962991+0000 mon.a (mon.0) 366 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/588 objects degraded (2.891%), 4 pgs degraded) 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: cluster 2022-04-22T17:30:23.963049+0000 mon.a (mon.0) 367 : cluster [INF] Cluster is now healthy 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.181772+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.362929+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.370112+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.376060+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:25.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.378569+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.380092+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.381087+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:24 smithi121 bash[36902]: audit 2022-04-22T17:30:24.382572+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: cluster 2022-04-22T17:30:23.167209+0000 mgr.y (mgr.34107) 126 : cluster [DBG] pgmap v180: 121 pgs: 38 peering, 83 active+clean; 332 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 434 KiB/s wr, 1 op/s; 1.5 MiB/s, 0 objects/s recovering 2022-04-22T17:30:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: cluster 2022-04-22T17:30:23.962524+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2022-04-22T17:30:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: cluster 2022-04-22T17:30:23.962991+0000 mon.a (mon.0) 366 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/588 objects degraded (2.891%), 4 pgs degraded) 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: cluster 2022-04-22T17:30:23.963049+0000 mon.a (mon.0) 367 : cluster [INF] Cluster is now healthy 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.181772+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.362929+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.370112+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.376060+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.378569+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.380092+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.381087+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[45495]: audit 2022-04-22T17:30:24.382572+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: cluster 2022-04-22T17:30:23.167209+0000 mgr.y (mgr.34107) 126 : cluster [DBG] pgmap v180: 121 pgs: 38 peering, 83 active+clean; 332 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 434 KiB/s wr, 1 op/s; 1.5 MiB/s, 0 objects/s recovering 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: cluster 2022-04-22T17:30:23.962524+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: cluster 2022-04-22T17:30:23.962991+0000 mon.a (mon.0) 366 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/588 objects degraded (2.891%), 4 pgs degraded) 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: cluster 2022-04-22T17:30:23.963049+0000 mon.a (mon.0) 367 : cluster [INF] Cluster is now healthy 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.181772+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.362929+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.370112+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.376060+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.378569+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.380092+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.381087+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:24 smithi115 bash[46934]: audit 2022-04-22T17:30:24.382572+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:26.006 INFO:tasks.rados.rados.0.smithi115.stdout:1331: done (1 left) 2022-04-22T17:30:26.006 INFO:tasks.rados.rados.0.smithi115.stdout:1332: copy_from oid 11 from oid 43 current snap is 137 2022-04-22T17:30:26.007 INFO:tasks.rados.rados.0.smithi115.stdout:1330: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:30:26.110 INFO:tasks.rados.rados.0.smithi115.stdout:1332: finishing copy_from to smithi11541303-11 2022-04-22T17:30:26.110 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 465 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:30:26.110 INFO:tasks.rados.rados.0.smithi115.stdout:1332: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:30:26.111 INFO:tasks.rados.rados.0.smithi115.stdout:1330: done (1 left) 2022-04-22T17:30:26.114 INFO:tasks.rados.rados.0.smithi115.stdout:1332: done (0 left) 2022-04-22T17:30:26.114 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 137 2022-04-22T17:30:26.114 INFO:tasks.rados.rados.0.smithi115.stdout:1333: seq_num 399 ranges {163840=688128,851968=434176,1286144=745472,2031616=24576} 2022-04-22T17:30:26.126 INFO:tasks.rados.rados.0.smithi115.stdout:1333: writing smithi11541303-25 from 163840 to 851968 tid 1 2022-04-22T17:30:26.129 INFO:tasks.rados.rados.0.smithi115.stdout:1333: writing smithi11541303-25 from 851968 to 1286144 tid 2 2022-04-22T17:30:26.133 INFO:tasks.rados.rados.0.smithi115.stdout:1333: writing smithi11541303-25 from 1286144 to 2031616 tid 3 2022-04-22T17:30:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:1333: writing smithi11541303-25 from 2031616 to 2056192 tid 4 2022-04-22T17:30:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:1334: read oid 13 snap 121 2022-04-22T17:30:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:1334: expect deleted 2022-04-22T17:30:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 137 2022-04-22T17:30:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:1335: seq_num 400 ranges {3366912=704512,4071424=720896,4792320=483328} 2022-04-22T17:30:26.156 INFO:tasks.rados.rados.0.smithi115.stdout:1335: writing smithi11541303-7 from 3366912 to 4071424 tid 1 2022-04-22T17:30:26.160 INFO:tasks.rados.rados.0.smithi115.stdout:1335: writing smithi11541303-7 from 4071424 to 4792320 tid 2 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1335: writing smithi11541303-7 from 4792320 to 5275648 tid 3 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1336: setattr oid 45 current snap is 137 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 1 to smithi11541303-25 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 2 to smithi11541303-25 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 3 to smithi11541303-25 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 4 to smithi11541303-25 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 5 to smithi11541303-25 2022-04-22T17:30:26.164 INFO:tasks.rados.rados.0.smithi115.stdout:1333: finishing write tid 6 to smithi11541303-25 2022-04-22T17:30:26.165 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 613 (ObjNum 399 snap 137 seq_num 399) dirty exists 2022-04-22T17:30:26.165 INFO:tasks.rados.rados.0.smithi115.stdout:1333: left oid 25 (ObjNum 399 snap 137 seq_num 399) 2022-04-22T17:30:26.165 INFO:tasks.rados.rados.0.smithi115.stdout:1335: finishing write tid 1 to smithi11541303-7 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1333: done (3 left) 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1334: done (2 left) 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1337: read oid 40 snap -1 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1337: expect deleted 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1338: read oid 8 snap -1 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1338: expect (ObjNum 374 snap 129 seq_num 374) 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1339: rollback oid 4 current snap is 137 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 130 2022-04-22T17:30:26.166 INFO:tasks.rados.rados.0.smithi115.stdout:1340: snap_create 2022-04-22T17:30:26.168 INFO:tasks.rados.rados.0.smithi115.stdout:1335: finishing write tid 2 to smithi11541303-7 2022-04-22T17:30:26.169 INFO:tasks.rados.rados.0.smithi115.stdout:1335: finishing write tid 3 to smithi11541303-7 2022-04-22T17:30:26.169 INFO:tasks.rados.rados.0.smithi115.stdout:1335: finishing write tid 4 to smithi11541303-7 2022-04-22T17:30:26.172 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 768 (ObjNum 386 snap 134 seq_num 386) dirty exists 2022-04-22T17:30:26.173 INFO:tasks.rados.rados.0.smithi115.stdout:1339: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:30:26.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 582 (ObjNum -1116815872 snap 22075 seq_num 0) dirty dne 2022-04-22T17:30:26.182 INFO:tasks.rados.rados.0.smithi115.stdout:1335: finishing write tid 5 to smithi11541303-7 2022-04-22T17:30:26.182 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 496 (ObjNum 400 snap 137 seq_num 400) dirty exists 2022-04-22T17:30:26.182 INFO:tasks.rados.rados.0.smithi115.stdout:1335: left oid 7 (ObjNum 400 snap 137 seq_num 400) 2022-04-22T17:30:26.185 INFO:tasks.rados.rados.0.smithi115.stdout:1338: expect (ObjNum 374 snap 129 seq_num 374) 2022-04-22T17:30:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:26 smithi121 bash[36902]: cephadm 2022-04-22T17:30:24.352712+0000 mgr.y (mgr.34107) 127 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:30:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:26 smithi121 bash[36902]: audit 2022-04-22T17:30:24.382797+0000 mgr.y (mgr.34107) 128 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:26 smithi121 bash[36902]: cephadm 2022-04-22T17:30:24.383653+0000 mgr.y (mgr.34107) 129 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2022-04-22T17:30:26.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[46934]: cephadm 2022-04-22T17:30:24.352712+0000 mgr.y (mgr.34107) 127 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:30:26.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[46934]: audit 2022-04-22T17:30:24.382797+0000 mgr.y (mgr.34107) 128 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:26.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[46934]: cephadm 2022-04-22T17:30:24.383653+0000 mgr.y (mgr.34107) 129 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2022-04-22T17:30:26.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[45495]: cephadm 2022-04-22T17:30:24.352712+0000 mgr.y (mgr.34107) 127 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:30:26.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[45495]: audit 2022-04-22T17:30:24.382797+0000 mgr.y (mgr.34107) 128 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:26.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:25 smithi115 bash[45495]: cephadm 2022-04-22T17:30:24.383653+0000 mgr.y (mgr.34107) 129 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2022-04-22T17:30:27.014 INFO:tasks.rados.rados.0.smithi115.stdout:1335: done (5 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1336: done (4 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1337: done (3 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1338: done (2 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1339: done (1 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1340: done (0 left) 2022-04-22T17:30:27.015 INFO:tasks.rados.rados.0.smithi115.stdout:1341: delete oid 14 current snap is 138 2022-04-22T17:30:27.023 INFO:tasks.rados.rados.0.smithi115.stdout:1341: done (0 left) 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1342: read oid 24 snap -1 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1342: expect (ObjNum 385 snap 133 seq_num 385) 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1343: read oid 4 snap 133 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1343: expect deleted 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1344: read oid 26 snap 135 2022-04-22T17:30:27.024 INFO:tasks.rados.rados.0.smithi115.stdout:1344: expect deleted 2022-04-22T17:30:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:1345: copy_from oid 2 from oid 27 current snap is 138 2022-04-22T17:30:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:1346: copy_from oid 8 from oid 19 current snap is 138 2022-04-22T17:30:27.025 INFO:tasks.rados.rados.0.smithi115.stdout:1347: delete oid 49 current snap is 138 2022-04-22T17:30:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1343: done (5 left) 2022-04-22T17:30:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1344: done (4 left) 2022-04-22T17:30:27.041 INFO:tasks.rados.rados.0.smithi115.stdout:1347: done (3 left) 2022-04-22T17:30:27.042 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 138 2022-04-22T17:30:27.042 INFO:tasks.rados.rados.0.smithi115.stdout:1348: seq_num 403 ranges {163840=753664,917504=425984,1343488=720896,2064384=57344} 2022-04-22T17:30:27.044 INFO:tasks.rados.rados.0.smithi115.stdout:1348: writing smithi11541303-11 from 163840 to 917504 tid 1 2022-04-22T17:30:27.047 INFO:tasks.rados.rados.0.smithi115.stdout:1348: writing smithi11541303-11 from 917504 to 1343488 tid 2 2022-04-22T17:30:27.058 INFO:tasks.rados.rados.0.smithi115.stdout:1348: writing smithi11541303-11 from 1343488 to 2064384 tid 3 2022-04-22T17:30:27.059 INFO:tasks.rados.rados.0.smithi115.stdout:1348: writing smithi11541303-11 from 2064384 to 2121728 tid 4 2022-04-22T17:30:27.059 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 138 2022-04-22T17:30:27.059 INFO:tasks.rados.rados.0.smithi115.stdout:1349: seq_num 404 ranges {0=770048,770048=712704,1482752=491520} 2022-04-22T17:30:27.059 INFO:tasks.rados.rados.0.smithi115.stdout:1349: writing smithi11541303-32 from 0 to 770048 tid 1 2022-04-22T17:30:27.061 INFO:tasks.rados.rados.0.smithi115.stdout:1349: writing smithi11541303-32 from 770048 to 1482752 tid 2 2022-04-22T17:30:27.064 INFO:tasks.rados.rados.0.smithi115.stdout:1349: writing smithi11541303-32 from 1482752 to 1974272 tid 3 2022-04-22T17:30:27.065 INFO:tasks.rados.rados.0.smithi115.stdout:1350: copy_from oid 14 from oid 12 current snap is 138 2022-04-22T17:30:27.065 INFO:tasks.rados.rados.0.smithi115.stdout:1345: finishing copy_from to smithi11541303-2 2022-04-22T17:30:27.065 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 591 (ObjNum 288 snap 0 seq_num 129) dirty exists 2022-04-22T17:30:27.065 INFO:tasks.rados.rados.0.smithi115.stdout:1345: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:30:27.065 INFO:tasks.rados.rados.0.smithi115.stdout:1342: expect (ObjNum 385 snap 133 seq_num 385) 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 1 to smithi11541303-11 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 2 to smithi11541303-11 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:1346: finishing copy_from to smithi11541303-8 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 487 (ObjNum 391 snap 136 seq_num 391) dirty exists 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:1346: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:30:27.263 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 3 to smithi11541303-11 2022-04-22T17:30:27.264 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 4 to smithi11541303-11 2022-04-22T17:30:27.264 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 5 to smithi11541303-11 2022-04-22T17:30:27.264 INFO:tasks.rados.rados.0.smithi115.stdout:1349: finishing write tid 1 to smithi11541303-32 2022-04-22T17:30:27.264 INFO:tasks.rados.rados.0.smithi115.stdout:1348: finishing write tid 6 to smithi11541303-11 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 644 (ObjNum 403 snap 138 seq_num 403) dirty exists 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:1348: left oid 11 (ObjNum 403 snap 138 seq_num 403) 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:1349: finishing write tid 2 to smithi11541303-32 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:1349: finishing write tid 3 to smithi11541303-32 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:1349: finishing write tid 4 to smithi11541303-32 2022-04-22T17:30:27.265 INFO:tasks.rados.rados.0.smithi115.stdout:1349: finishing write tid 5 to smithi11541303-32 2022-04-22T17:30:27.266 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 548 (ObjNum 404 snap 138 seq_num 404) dirty exists 2022-04-22T17:30:27.266 INFO:tasks.rados.rados.0.smithi115.stdout:1349: left oid 32 (ObjNum 404 snap 138 seq_num 404) 2022-04-22T17:30:27.266 INFO:tasks.rados.rados.0.smithi115.stdout:1350: finishing copy_from to smithi11541303-14 2022-04-22T17:30:27.266 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 468 (ObjNum 361 snap 123 seq_num 361) dirty exists 2022-04-22T17:30:27.266 INFO:tasks.rados.rados.0.smithi115.stdout:1342: done (5 left) 2022-04-22T17:30:27.267 INFO:tasks.rados.rados.0.smithi115.stdout:1345: done (4 left) 2022-04-22T17:30:27.267 INFO:tasks.rados.rados.0.smithi115.stdout:1346: done (3 left) 2022-04-22T17:30:27.267 INFO:tasks.rados.rados.0.smithi115.stdout:1348: done (2 left) 2022-04-22T17:30:27.267 INFO:tasks.rados.rados.0.smithi115.stdout:1349: done (1 left) 2022-04-22T17:30:27.267 INFO:tasks.rados.rados.0.smithi115.stdout:1351: read oid 20 snap -1 2022-04-22T17:30:27.268 INFO:tasks.rados.rados.0.smithi115.stdout:1351: expect (ObjNum 398 snap 137 seq_num 398) 2022-04-22T17:30:27.268 INFO:tasks.rados.rados.0.smithi115.stdout:1352: snap_remove snap 129 2022-04-22T17:30:27.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:27 smithi115 bash[45495]: cluster 2022-04-22T17:30:25.168744+0000 mgr.y (mgr.34107) 130 : cluster [DBG] pgmap v182: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:27.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:27 smithi115 bash[45495]: cluster 2022-04-22T17:30:25.991683+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2022-04-22T17:30:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:27 smithi115 bash[46934]: cluster 2022-04-22T17:30:25.168744+0000 mgr.y (mgr.34107) 130 : cluster [DBG] pgmap v182: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:27 smithi115 bash[46934]: cluster 2022-04-22T17:30:25.991683+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2022-04-22T17:30:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:27 smithi121 bash[36902]: cluster 2022-04-22T17:30:25.168744+0000 mgr.y (mgr.34107) 130 : cluster [DBG] pgmap v182: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:27 smithi121 bash[36902]: cluster 2022-04-22T17:30:25.991683+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2022-04-22T17:30:28.031 INFO:tasks.rados.rados.0.smithi115.stdout:1352: done (2 left) 2022-04-22T17:30:28.031 INFO:tasks.rados.rados.0.smithi115.stdout:1353: snap_remove snap 134 2022-04-22T17:30:28.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:28 smithi115 bash[45495]: cluster 2022-04-22T17:30:26.999204+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2022-04-22T17:30:28.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:28 smithi115 bash[46934]: cluster 2022-04-22T17:30:26.999204+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2022-04-22T17:30:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:28 smithi121 bash[36902]: cluster 2022-04-22T17:30:26.999204+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2022-04-22T17:30:29.039 INFO:tasks.rados.rados.0.smithi115.stdout:1353: done (2 left) 2022-04-22T17:30:29.039 INFO:tasks.rados.rados.0.smithi115.stdout:1354: delete oid 1 current snap is 138 2022-04-22T17:30:29.039 INFO:tasks.rados.rados.0.smithi115.stdout:1350: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:30:29.039 INFO:tasks.rados.rados.0.smithi115.stdout:1351: expect (ObjNum 398 snap 137 seq_num 398) 2022-04-22T17:30:29.164 INFO:tasks.rados.rados.0.smithi115.stdout:1350: done (2 left) 2022-04-22T17:30:29.165 INFO:tasks.rados.rados.0.smithi115.stdout:1351: done (1 left) 2022-04-22T17:30:29.165 INFO:tasks.rados.rados.0.smithi115.stdout:1354: done (0 left) 2022-04-22T17:30:29.165 INFO:tasks.rados.rados.0.smithi115.stdout:1355: snap_remove snap 126 2022-04-22T17:30:29.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:29 smithi115 bash[45495]: cluster 2022-04-22T17:30:27.169558+0000 mgr.y (mgr.34107) 131 : cluster [DBG] pgmap v185: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:29.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:29 smithi115 bash[45495]: cluster 2022-04-22T17:30:28.016110+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2022-04-22T17:30:29.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:29 smithi115 bash[46934]: cluster 2022-04-22T17:30:27.169558+0000 mgr.y (mgr.34107) 131 : cluster [DBG] pgmap v185: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:29.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:29 smithi115 bash[46934]: cluster 2022-04-22T17:30:28.016110+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2022-04-22T17:30:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:29 smithi121 bash[36902]: cluster 2022-04-22T17:30:27.169558+0000 mgr.y (mgr.34107) 131 : cluster [DBG] pgmap v185: 121 pgs: 121 active+clean; 296 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 16 MiB/s, 5 objects/s recovering 2022-04-22T17:30:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:29 smithi121 bash[36902]: cluster 2022-04-22T17:30:28.016110+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2022-04-22T17:30:29.779 INFO:tasks.rados.rados.0.smithi115.stdout:1355: done (0 left) 2022-04-22T17:30:29.779 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 138 2022-04-22T17:30:29.779 INFO:tasks.rados.rados.0.smithi115.stdout:1356: seq_num 406 ranges {1998848=802816,2801664=475136,3276800=499712,3776512=229376} 2022-04-22T17:30:29.801 INFO:tasks.rados.rados.0.smithi115.stdout:1356: writing smithi11541303-16 from 1998848 to 2801664 tid 1 2022-04-22T17:30:29.805 INFO:tasks.rados.rados.0.smithi115.stdout:1356: writing smithi11541303-16 from 2801664 to 3276800 tid 2 2022-04-22T17:30:29.808 INFO:tasks.rados.rados.0.smithi115.stdout:1356: writing smithi11541303-16 from 3276800 to 3776512 tid 3 2022-04-22T17:30:29.810 INFO:tasks.rados.rados.0.smithi115.stdout:1356: writing smithi11541303-16 from 3776512 to 4005888 tid 4 2022-04-22T17:30:29.810 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 138 2022-04-22T17:30:29.810 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 1 to smithi11541303-16 2022-04-22T17:30:29.810 INFO:tasks.rados.rados.0.smithi115.stdout:1357: seq_num 407 ranges {0=425984,425984=761856,1187840=704512,1892352=139264} 2022-04-22T17:30:29.812 INFO:tasks.rados.rados.0.smithi115.stdout:1357: writing smithi11541303-15 from 0 to 425984 tid 1 2022-04-22T17:30:29.818 INFO:tasks.rados.rados.0.smithi115.stdout:1357: writing smithi11541303-15 from 425984 to 1187840 tid 2 2022-04-22T17:30:29.822 INFO:tasks.rados.rados.0.smithi115.stdout:1357: writing smithi11541303-15 from 1187840 to 1892352 tid 3 2022-04-22T17:30:29.823 INFO:tasks.rados.rados.0.smithi115.stdout:1357: writing smithi11541303-15 from 1892352 to 2031616 tid 4 2022-04-22T17:30:29.823 INFO:tasks.rados.rados.0.smithi115.stdout:1358: snap_remove snap 131 2022-04-22T17:30:30.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[45495]: cluster 2022-04-22T17:30:29.023989+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2022-04-22T17:30:30.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[45495]: audit 2022-04-22T17:30:29.184559+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:30.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[45495]: cluster 2022-04-22T17:30:29.763796+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2022-04-22T17:30:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[46934]: cluster 2022-04-22T17:30:29.023989+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2022-04-22T17:30:30.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[46934]: audit 2022-04-22T17:30:29.184559+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:30.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:30 smithi115 bash[46934]: cluster 2022-04-22T17:30:29.763796+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2022-04-22T17:30:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:30 smithi121 bash[36902]: cluster 2022-04-22T17:30:29.023989+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2022-04-22T17:30:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:30 smithi121 bash[36902]: audit 2022-04-22T17:30:29.184559+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:30 smithi121 bash[36902]: cluster 2022-04-22T17:30:29.763796+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2022-04-22T17:30:30.791 INFO:tasks.rados.rados.0.smithi115.stdout:1358: done (2 left) 2022-04-22T17:30:30.791 INFO:tasks.rados.rados.0.smithi115.stdout:1359: read oid 20 snap -1 2022-04-22T17:30:30.791 INFO:tasks.rados.rados.0.smithi115.stdout:1359: expect (ObjNum 398 snap 137 seq_num 398) 2022-04-22T17:30:30.791 INFO:tasks.rados.rados.0.smithi115.stdout:1360: read oid 33 snap 137 2022-04-22T17:30:30.791 INFO:tasks.rados.rados.0.smithi115.stdout:1360: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1361: snap_create 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 2 to smithi11541303-16 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 3 to smithi11541303-16 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 4 to smithi11541303-16 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 5 to smithi11541303-16 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 1 to smithi11541303-15 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: finishing write tid 6 to smithi11541303-16 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 401 (ObjNum 406 snap 138 seq_num 406) dirty exists 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1356: left oid 16 (ObjNum 406 snap 138 seq_num 406) 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 2 to smithi11541303-15 2022-04-22T17:30:30.792 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 3 to smithi11541303-15 2022-04-22T17:30:30.793 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 4 to smithi11541303-15 2022-04-22T17:30:30.793 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 5 to smithi11541303-15 2022-04-22T17:30:30.793 INFO:tasks.rados.rados.0.smithi115.stdout:1357: finishing write tid 6 to smithi11541303-15 2022-04-22T17:30:30.793 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 198 (ObjNum 407 snap 138 seq_num 407) dirty exists 2022-04-22T17:30:30.793 INFO:tasks.rados.rados.0.smithi115.stdout:1357: left oid 15 (ObjNum 407 snap 138 seq_num 407) 2022-04-22T17:30:30.816 INFO:tasks.rados.rados.0.smithi115.stdout:1360: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:30.885 INFO:tasks.rados.rados.0.smithi115.stdout:1359: expect (ObjNum 398 snap 137 seq_num 398) 2022-04-22T17:30:31.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[45495]: cluster 2022-04-22T17:30:29.170955+0000 mgr.y (mgr.34107) 132 : cluster [DBG] pgmap v188: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:31.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[45495]: cluster 2022-04-22T17:30:30.775406+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2022-04-22T17:30:31.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[46934]: cluster 2022-04-22T17:30:29.170955+0000 mgr.y (mgr.34107) 132 : cluster [DBG] pgmap v188: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:31.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[46934]: cluster 2022-04-22T17:30:30.775406+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2022-04-22T17:30:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:31 smithi121 bash[36902]: cluster 2022-04-22T17:30:29.170955+0000 mgr.y (mgr.34107) 132 : cluster [DBG] pgmap v188: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:31 smithi121 bash[36902]: cluster 2022-04-22T17:30:30.775406+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2022-04-22T17:30:31.795 INFO:tasks.rados.rados.0.smithi115.stdout:1356: done (4 left) 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:1357: done (3 left) 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:1359: done (2 left) 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:1360: done (1 left) 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:1361: done (0 left) 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:1362: rollback oid 49 current snap is 139 2022-04-22T17:30:31.828 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 137 2022-04-22T17:30:31.829 INFO:tasks.rados.rados.0.smithi115.stdout:1363: rmattr oid 46 current snap is 139 2022-04-22T17:30:31.829 INFO:tasks.rados.rados.0.smithi115.stdout:1363: done (1 left) 2022-04-22T17:30:31.829 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 139 2022-04-22T17:30:31.829 INFO:tasks.rados.rados.0.smithi115.stdout:1364: seq_num 408 ranges {4005888=442368,4448256=647168,5095424=507904,5603328=450560} 2022-04-22T17:30:31.830 INFO:tasks.rados.rados.0.smithi115.stdout:1364: writing smithi11541303-16 from 4005888 to 4448256 tid 1 2022-04-22T17:30:31.830 INFO:tasks.rados.rados.0.smithi115.stdout:1364: writing smithi11541303-16 from 4448256 to 5095424 tid 2 2022-04-22T17:30:31.831 INFO:tasks.rados.rados.0.smithi115.stdout:1364: writing smithi11541303-16 from 5095424 to 5603328 tid 3 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:1364: writing smithi11541303-16 from 5603328 to 6053888 tid 4 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:1365: snap_create 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:1362: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:1362: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 821 (ObjNum 376 snap 129 seq_num 376) dirty exists 2022-04-22T17:30:31.835 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 1 to smithi11541303-16 2022-04-22T17:30:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 2 to smithi11541303-16 2022-04-22T17:30:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 3 to smithi11541303-16 2022-04-22T17:30:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 4 to smithi11541303-16 2022-04-22T17:30:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 5 to smithi11541303-16 2022-04-22T17:30:31.839 INFO:tasks.rados.rados.0.smithi115.stdout:1364: finishing write tid 6 to smithi11541303-16 2022-04-22T17:30:31.840 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 406 (ObjNum 408 snap 139 seq_num 408) dirty exists 2022-04-22T17:30:31.840 INFO:tasks.rados.rados.0.smithi115.stdout:1364: left oid 16 (ObjNum 408 snap 139 seq_num 408) 2022-04-22T17:30:32.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:31.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:32.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:31.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:32.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:31.621Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:32.803 INFO:tasks.rados.rados.0.smithi115.stdout:1362: done (2 left) 2022-04-22T17:30:32.863 INFO:tasks.rados.rados.0.smithi115.stdout:1364: done (1 left) 2022-04-22T17:30:32.863 INFO:tasks.rados.rados.0.smithi115.stdout:1365: done (0 left) 2022-04-22T17:30:32.863 INFO:tasks.rados.rados.0.smithi115.stdout:1366: delete oid 24 current snap is 140 2022-04-22T17:30:32.864 INFO:tasks.rados.rados.0.smithi115.stdout:1366: done (0 left) 2022-04-22T17:30:32.864 INFO:tasks.rados.rados.0.smithi115.stdout:1367: snap_remove snap 109 2022-04-22T17:30:33.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:32 smithi115 bash[45495]: cluster 2022-04-22T17:30:31.171841+0000 mgr.y (mgr.34107) 133 : cluster [DBG] pgmap v191: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:33.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:32 smithi115 bash[45495]: cluster 2022-04-22T17:30:31.779622+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2022-04-22T17:30:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:32 smithi115 bash[46934]: cluster 2022-04-22T17:30:31.171841+0000 mgr.y (mgr.34107) 133 : cluster [DBG] pgmap v191: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:32 smithi115 bash[46934]: cluster 2022-04-22T17:30:31.779622+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2022-04-22T17:30:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:32 smithi121 bash[36902]: cluster 2022-04-22T17:30:31.171841+0000 mgr.y (mgr.34107) 133 : cluster [DBG] pgmap v191: 121 pgs: 121 active+clean; 305 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2022-04-22T17:30:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:32 smithi121 bash[36902]: cluster 2022-04-22T17:30:31.779622+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2022-04-22T17:30:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:33.516Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:33.516Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:30:33.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[46934]: cluster 2022-04-22T17:30:32.788828+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2022-04-22T17:30:33.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[46934]: audit 2022-04-22T17:30:33.265916+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:33.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[45495]: cluster 2022-04-22T17:30:32.788828+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2022-04-22T17:30:33.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:33 smithi115 bash[45495]: audit 2022-04-22T17:30:33.265916+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:33.812 INFO:tasks.rados.rados.0.smithi115.stdout:1367: done (0 left) 2022-04-22T17:30:33.812 INFO:tasks.rados.rados.0.smithi115.stdout:1368: delete oid 24 current snap is 140 2022-04-22T17:30:33.827 INFO:tasks.rados.rados.0.smithi115.stdout:1368: done (0 left) 2022-04-22T17:30:33.827 INFO:tasks.rados.rados.0.smithi115.stdout:1369: read oid 30 snap -1 2022-04-22T17:30:33.827 INFO:tasks.rados.rados.0.smithi115.stdout:1369: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:30:33.827 INFO:tasks.rados.rados.0.smithi115.stdout:1370: read oid 43 snap -1 2022-04-22T17:30:33.828 INFO:tasks.rados.rados.0.smithi115.stdout:1370: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:30:33.828 INFO:tasks.rados.rados.0.smithi115.stdout:1371: delete oid 29 current snap is 140 2022-04-22T17:30:33.835 INFO:tasks.rados.rados.0.smithi115.stdout:1369: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1369: done (2 left) 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1371: done (1 left) 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1372: read oid 35 snap 133 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1372: expect (ObjNum 369 snap 127 seq_num 369) 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1373: rollback oid 28 current snap is 140 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 139 2022-04-22T17:30:33.843 INFO:tasks.rados.rados.0.smithi115.stdout:1374: snap_create 2022-04-22T17:30:33.845 INFO:tasks.rados.rados.0.smithi115.stdout:1373: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:30:33.845 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 303 (ObjNum 1 snap 22075 seq_num 3179282864) dirty dne 2022-04-22T17:30:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1370: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:30:33.856 INFO:tasks.rados.rados.0.smithi115.stdout:1372: expect (ObjNum 369 snap 127 seq_num 369) 2022-04-22T17:30:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:33 smithi121 bash[36902]: cluster 2022-04-22T17:30:32.788828+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2022-04-22T17:30:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:33 smithi121 bash[36902]: audit 2022-04-22T17:30:33.265916+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:34.785 INFO:tasks.rados.rados.0.smithi115.stdout:1370: done (3 left) 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1372: done (2 left) 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1373: done (1 left) 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1374: done (0 left) 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1375: rollback oid 29 current snap is 141 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 140 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1376: read oid 47 snap -1 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1376: expect (ObjNum 304 snap 101 seq_num 304) 2022-04-22T17:30:34.786 INFO:tasks.rados.rados.0.smithi115.stdout:1377: copy_from oid 19 from oid 8 current snap is 141 2022-04-22T17:30:34.787 INFO:tasks.rados.rados.0.smithi115.stdout:1378: rollback oid 41 current snap is 141 2022-04-22T17:30:34.787 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 138 2022-04-22T17:30:34.787 INFO:tasks.rados.rados.0.smithi115.stdout:1379: snap_remove snap 139 2022-04-22T17:30:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:34 smithi121 bash[36902]: cluster 2022-04-22T17:30:33.172809+0000 mgr.y (mgr.34107) 134 : cluster [DBG] pgmap v194: 121 pgs: 121 active+clean; 309 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 989 KiB/s wr, 2 op/s 2022-04-22T17:30:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:34 smithi121 bash[36902]: cluster 2022-04-22T17:30:33.797602+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2022-04-22T17:30:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:34 smithi121 bash[36902]: audit 2022-04-22T17:30:34.187295+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:34 smithi121 bash[36902]: cluster 2022-04-22T17:30:34.766619+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2022-04-22T17:30:35.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[45495]: cluster 2022-04-22T17:30:33.172809+0000 mgr.y (mgr.34107) 134 : cluster [DBG] pgmap v194: 121 pgs: 121 active+clean; 309 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 989 KiB/s wr, 2 op/s 2022-04-22T17:30:35.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[45495]: cluster 2022-04-22T17:30:33.797602+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2022-04-22T17:30:35.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[45495]: audit 2022-04-22T17:30:34.187295+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:35.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[45495]: cluster 2022-04-22T17:30:34.766619+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2022-04-22T17:30:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[46934]: cluster 2022-04-22T17:30:33.172809+0000 mgr.y (mgr.34107) 134 : cluster [DBG] pgmap v194: 121 pgs: 121 active+clean; 309 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 989 KiB/s wr, 2 op/s 2022-04-22T17:30:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[46934]: cluster 2022-04-22T17:30:33.797602+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2022-04-22T17:30:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[46934]: audit 2022-04-22T17:30:34.187295+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:34 smithi115 bash[46934]: cluster 2022-04-22T17:30:34.766619+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1379: done (4 left) 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1380: delete oid 10 current snap is 141 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1375: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 469 (ObjNum 0 snap 22075 seq_num 3178709696) dirty dne 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1378: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1378: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:30:35.795 INFO:tasks.rados.rados.0.smithi115.stdout:1376: expect (ObjNum 304 snap 101 seq_num 304) 2022-04-22T17:30:35.826 INFO:tasks.rados.rados.0.smithi115.stdout:1378: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:30:35.826 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 642 (ObjNum 369 snap 127 seq_num 369) dirty exists 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1377: finishing copy_from to smithi11541303-19 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 487 (ObjNum 391 snap 136 seq_num 391) dirty exists 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1377: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1375: done (4 left) 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1376: done (3 left) 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1377: done (2 left) 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1378: done (1 left) 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1380: done (0 left) 2022-04-22T17:30:35.827 INFO:tasks.rados.rados.0.smithi115.stdout:1381: snap_remove snap 133 2022-04-22T17:30:36.823 INFO:tasks.rados.rados.0.smithi115.stdout:1381: done (0 left) 2022-04-22T17:30:36.823 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 141 2022-04-22T17:30:36.823 INFO:tasks.rados.rados.0.smithi115.stdout:1382: seq_num 413 ranges {1974272=524288,2498560=516096,3014656=696320,3710976=393216} 2022-04-22T17:30:36.839 INFO:tasks.rados.rados.0.smithi115.stdout:1382: writing smithi11541303-32 from 1974272 to 2498560 tid 1 2022-04-22T17:30:36.843 INFO:tasks.rados.rados.0.smithi115.stdout:1382: writing smithi11541303-32 from 2498560 to 3014656 tid 2 2022-04-22T17:30:36.847 INFO:tasks.rados.rados.0.smithi115.stdout:1382: writing smithi11541303-32 from 3014656 to 3710976 tid 3 2022-04-22T17:30:36.850 INFO:tasks.rados.rados.0.smithi115.stdout:1382: writing smithi11541303-32 from 3710976 to 4104192 tid 4 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1383: read oid 23 snap -1 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1383: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1384: rollback oid 15 current snap is 141 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 140 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 1 to smithi11541303-32 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 2 to smithi11541303-32 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1385: rollback oid 20 current snap is 141 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 101 2022-04-22T17:30:36.851 INFO:tasks.rados.rados.0.smithi115.stdout:1386: copy_from oid 13 from oid 36 current snap is 141 2022-04-22T17:30:36.852 INFO:tasks.rados.rados.0.smithi115.stdout:1387: rmattr oid 10 current snap is 141 2022-04-22T17:30:36.852 INFO:tasks.rados.rados.0.smithi115.stdout:1387: done (5 left) 2022-04-22T17:30:36.852 INFO:tasks.rados.rados.0.smithi115.stdout:1388: setattr oid 33 current snap is 141 2022-04-22T17:30:36.853 INFO:tasks.rados.rados.0.smithi115.stdout:1389: rollback oid 39 current snap is 141 2022-04-22T17:30:36.854 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 137 2022-04-22T17:30:36.854 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 141 2022-04-22T17:30:36.854 INFO:tasks.rados.rados.0.smithi115.stdout:1390: seq_num 414 ranges {10649600=540672,11190272=802816,11993088=458752,12451840=344064} 2022-04-22T17:30:36.917 INFO:tasks.rados.rados.0.smithi115.stdout:1390: writing smithi11541303-42 from 10649600 to 11190272 tid 1 2022-04-22T17:30:36.935 INFO:tasks.rados.rados.0.smithi115.stdout:1390: writing smithi11541303-42 from 11190272 to 11993088 tid 2 2022-04-22T17:30:36.935 INFO:tasks.rados.rados.0.smithi115.stdout:1390: writing smithi11541303-42 from 11993088 to 12451840 tid 3 2022-04-22T17:30:36.935 INFO:tasks.rados.rados.0.smithi115.stdout:1390: writing smithi11541303-42 from 12451840 to 12795904 tid 4 2022-04-22T17:30:36.935 INFO:tasks.rados.rados.0.smithi115.stdout:1391: snap_remove snap 121 2022-04-22T17:30:37.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:36 smithi115 bash[45495]: cluster 2022-04-22T17:30:35.173569+0000 mgr.y (mgr.34107) 135 : cluster [DBG] pgmap v197: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:30:37.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:36 smithi115 bash[45495]: cluster 2022-04-22T17:30:35.779409+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2022-04-22T17:30:37.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:36 smithi115 bash[46934]: cluster 2022-04-22T17:30:35.173569+0000 mgr.y (mgr.34107) 135 : cluster [DBG] pgmap v197: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:30:37.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:36 smithi115 bash[46934]: cluster 2022-04-22T17:30:35.779409+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2022-04-22T17:30:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:36 smithi121 bash[36902]: cluster 2022-04-22T17:30:35.173569+0000 mgr.y (mgr.34107) 135 : cluster [DBG] pgmap v197: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2022-04-22T17:30:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:36 smithi121 bash[36902]: cluster 2022-04-22T17:30:35.779409+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2022-04-22T17:30:37.837 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 3 to smithi11541303-32 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1384: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 4 to smithi11541303-32 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1385: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1385: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1389: finishing rollback tid 0 to smithi11541303-39 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1389: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1385: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 643 (ObjNum 206 snap 64 seq_num 206) dirty exists 2022-04-22T17:30:37.838 INFO:tasks.rados.rados.0.smithi115.stdout:1389: finishing rollback tid 2 to smithi11541303-39 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 638 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1384: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1384: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 208 (ObjNum 407 snap 138 seq_num 407) dirty exists 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 5 to smithi11541303-32 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 554 (ObjNum 346 snap 120 seq_num 346) dirty exists 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1382: finishing write tid 6 to smithi11541303-32 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 553 (ObjNum 413 snap 141 seq_num 413) dirty exists 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1382: left oid 32 (ObjNum 413 snap 141 seq_num 413) 2022-04-22T17:30:37.839 INFO:tasks.rados.rados.0.smithi115.stdout:1383: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:30:38.055 INFO:tasks.rados.rados.0.smithi115.stdout:1386: finishing copy_from to smithi11541303-13 2022-04-22T17:30:38.055 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 370 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:30:38.055 INFO:tasks.rados.rados.0.smithi115.stdout:1386: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 1 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 2 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 3 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 4 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 5 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: finishing write tid 6 to smithi11541303-42 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 835 (ObjNum 414 snap 141 seq_num 414) dirty exists 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1390: left oid 42 (ObjNum 414 snap 141 seq_num 414) 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1382: done (8 left) 2022-04-22T17:30:38.056 INFO:tasks.rados.rados.0.smithi115.stdout:1383: done (7 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1384: done (6 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1385: done (5 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1386: done (4 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1388: done (3 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1389: done (2 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1390: done (1 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1391: done (0 left) 2022-04-22T17:30:38.057 INFO:tasks.rados.rados.0.smithi115.stdout:1392: snap_remove snap 130 2022-04-22T17:30:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:37 smithi121 bash[36902]: cluster 2022-04-22T17:30:36.807390+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2022-04-22T17:30:38.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:37 smithi115 bash[45495]: cluster 2022-04-22T17:30:36.807390+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2022-04-22T17:30:38.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:37 smithi115 bash[46934]: cluster 2022-04-22T17:30:36.807390+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2022-04-22T17:30:38.837 INFO:tasks.rados.rados.0.smithi115.stdout:1392: done (0 left) 2022-04-22T17:30:38.837 INFO:tasks.rados.rados.0.smithi115.stdout:1393: delete oid 13 current snap is 141 2022-04-22T17:30:38.848 INFO:tasks.rados.rados.0.smithi115.stdout:1393: done (0 left) 2022-04-22T17:30:38.848 INFO:tasks.rados.rados.0.smithi115.stdout:1394: setattr oid 18 current snap is 141 2022-04-22T17:30:38.850 INFO:tasks.rados.rados.0.smithi115.stdout:1395: delete oid 22 current snap is 141 2022-04-22T17:30:38.855 INFO:tasks.rados.rados.0.smithi115.stdout:1395: done (1 left) 2022-04-22T17:30:38.855 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 141 2022-04-22T17:30:38.855 INFO:tasks.rados.rados.0.smithi115.stdout:1396: seq_num 417 ranges {2891776=589824,3481600=458752,3940352=679936,4620288=466944} 2022-04-22T17:30:38.876 INFO:tasks.rados.rados.0.smithi115.stdout:1396: writing smithi11541303-6 from 2891776 to 3481600 tid 1 2022-04-22T17:30:38.879 INFO:tasks.rados.rados.0.smithi115.stdout:1396: writing smithi11541303-6 from 3481600 to 3940352 tid 2 2022-04-22T17:30:38.883 INFO:tasks.rados.rados.0.smithi115.stdout:1396: writing smithi11541303-6 from 3940352 to 4620288 tid 3 2022-04-22T17:30:38.886 INFO:tasks.rados.rados.0.smithi115.stdout:1396: writing smithi11541303-6 from 4620288 to 5087232 tid 4 2022-04-22T17:30:38.886 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 141 2022-04-22T17:30:38.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: seq_num 418 ranges {3612672=606208,4218880=737280,4956160=434176,5390336=434176} 2022-04-22T17:30:38.911 INFO:tasks.rados.rados.0.smithi115.stdout:1397: writing smithi11541303-20 from 3612672 to 4218880 tid 1 2022-04-22T17:30:38.916 INFO:tasks.rados.rados.0.smithi115.stdout:1397: writing smithi11541303-20 from 4218880 to 4956160 tid 2 2022-04-22T17:30:38.917 INFO:tasks.rados.rados.0.smithi115.stdout:1397: writing smithi11541303-20 from 4956160 to 5390336 tid 3 2022-04-22T17:30:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1397: writing smithi11541303-20 from 5390336 to 5824512 tid 4 2022-04-22T17:30:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1398: snap_remove snap 137 2022-04-22T17:30:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:38 smithi121 bash[36902]: cluster 2022-04-22T17:30:37.174367+0000 mgr.y (mgr.34107) 136 : cluster [DBG] pgmap v200: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 496 KiB/s wr, 7 op/s 2022-04-22T17:30:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:38 smithi121 bash[36902]: cluster 2022-04-22T17:30:37.815325+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2022-04-22T17:30:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:38 smithi115 bash[45495]: cluster 2022-04-22T17:30:37.174367+0000 mgr.y (mgr.34107) 136 : cluster [DBG] pgmap v200: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 496 KiB/s wr, 7 op/s 2022-04-22T17:30:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:38 smithi115 bash[45495]: cluster 2022-04-22T17:30:37.815325+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2022-04-22T17:30:39.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:38 smithi115 bash[46934]: cluster 2022-04-22T17:30:37.174367+0000 mgr.y (mgr.34107) 136 : cluster [DBG] pgmap v200: 121 pgs: 121 active+clean; 308 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 496 KiB/s wr, 7 op/s 2022-04-22T17:30:39.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:38 smithi115 bash[46934]: cluster 2022-04-22T17:30:37.815325+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2022-04-22T17:30:39.849 INFO:tasks.rados.rados.0.smithi115.stdout:1398: done (3 left) 2022-04-22T17:30:39.849 INFO:tasks.rados.rados.0.smithi115.stdout:1399: snap_remove snap 138 2022-04-22T17:30:40.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[45495]: cluster 2022-04-22T17:30:38.822128+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2022-04-22T17:30:40.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[45495]: audit 2022-04-22T17:30:39.190349+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:40.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[45495]: audit 2022-04-22T17:30:39.396296+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[46934]: cluster 2022-04-22T17:30:38.822128+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2022-04-22T17:30:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[46934]: audit 2022-04-22T17:30:39.190349+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:39 smithi115 bash[46934]: audit 2022-04-22T17:30:39.396296+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:39 smithi121 bash[36902]: cluster 2022-04-22T17:30:38.822128+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2022-04-22T17:30:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:39 smithi121 bash[36902]: audit 2022-04-22T17:30:39.190349+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:39 smithi121 bash[36902]: audit 2022-04-22T17:30:39.396296+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:40.855 INFO:tasks.rados.rados.0.smithi115.stdout:1399: done (3 left) 2022-04-22T17:30:40.885 INFO:tasks.rados.rados.0.smithi115.stdout:1400: snap_create 2022-04-22T17:30:40.885 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 845 (ObjNum 388 snap 134 seq_num 388) dirty exists 2022-04-22T17:30:40.885 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 1 to smithi11541303-6 2022-04-22T17:30:40.885 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 2 to smithi11541303-6 2022-04-22T17:30:40.885 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 3 to smithi11541303-6 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 4 to smithi11541303-6 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 5 to smithi11541303-6 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1396: finishing write tid 6 to smithi11541303-6 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 641 (ObjNum 417 snap 141 seq_num 417) dirty exists 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1396: left oid 6 (ObjNum 417 snap 141 seq_num 417) 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 1 to smithi11541303-20 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 2 to smithi11541303-20 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 3 to smithi11541303-20 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 4 to smithi11541303-20 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 5 to smithi11541303-20 2022-04-22T17:30:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:1397: finishing write tid 6 to smithi11541303-20 2022-04-22T17:30:40.887 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 661 (ObjNum 418 snap 141 seq_num 418) dirty exists 2022-04-22T17:30:40.887 INFO:tasks.rados.rados.0.smithi115.stdout:1397: left oid 20 (ObjNum 418 snap 141 seq_num 418) 2022-04-22T17:30:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:40 smithi121 bash[36902]: cluster 2022-04-22T17:30:39.175924+0000 mgr.y (mgr.34107) 137 : cluster [DBG] pgmap v203: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:40 smithi121 bash[36902]: audit 2022-04-22T17:30:39.396793+0000 mgr.y (mgr.34107) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:40 smithi121 bash[36902]: cephadm 2022-04-22T17:30:39.398993+0000 mgr.y (mgr.34107) 139 : cephadm [INF] Upgrade: osd.1 is safe to restart 2022-04-22T17:30:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:40 smithi121 bash[36902]: cluster 2022-04-22T17:30:39.833120+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2022-04-22T17:30:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[45495]: cluster 2022-04-22T17:30:39.175924+0000 mgr.y (mgr.34107) 137 : cluster [DBG] pgmap v203: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[45495]: audit 2022-04-22T17:30:39.396793+0000 mgr.y (mgr.34107) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[45495]: cephadm 2022-04-22T17:30:39.398993+0000 mgr.y (mgr.34107) 139 : cephadm [INF] Upgrade: osd.1 is safe to restart 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[45495]: cluster 2022-04-22T17:30:39.833120+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[46934]: cluster 2022-04-22T17:30:39.175924+0000 mgr.y (mgr.34107) 137 : cluster [DBG] pgmap v203: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[46934]: audit 2022-04-22T17:30:39.396793+0000 mgr.y (mgr.34107) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[46934]: cephadm 2022-04-22T17:30:39.398993+0000 mgr.y (mgr.34107) 139 : cephadm [INF] Upgrade: osd.1 is safe to restart 2022-04-22T17:30:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:40 smithi115 bash[46934]: cluster 2022-04-22T17:30:39.833120+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2022-04-22T17:30:41.863 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:41.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:41.863 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:41.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:41.863 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:41.621Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:41.866 INFO:tasks.rados.rados.0.smithi115.stdout:1394: done (3 left) 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1396: done (2 left) 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1397: done (1 left) 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1400: done (0 left) 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1401: read oid 50 snap -1 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1401: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:1402: rollback oid 37 current snap is 142 2022-04-22T17:30:41.867 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 140 2022-04-22T17:30:41.868 INFO:tasks.rados.rados.0.smithi115.stdout:1403: delete oid 27 current snap is 142 2022-04-22T17:30:41.873 INFO:tasks.rados.rados.0.smithi115.stdout:1403: done (2 left) 2022-04-22T17:30:41.874 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 142 2022-04-22T17:30:41.874 INFO:tasks.rados.rados.0.smithi115.stdout:1404: seq_num 420 ranges {565248=638976,1204224=557056,1761280=450560,2211840=598016} 2022-04-22T17:30:41.880 INFO:tasks.rados.rados.0.smithi115.stdout:1404: writing smithi11541303-31 from 565248 to 1204224 tid 1 2022-04-22T17:30:41.883 INFO:tasks.rados.rados.0.smithi115.stdout:1404: writing smithi11541303-31 from 1204224 to 1761280 tid 2 2022-04-22T17:30:41.886 INFO:tasks.rados.rados.0.smithi115.stdout:1404: writing smithi11541303-31 from 1761280 to 2211840 tid 3 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1404: writing smithi11541303-31 from 2211840 to 2809856 tid 4 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1405: rollback oid 24 current snap is 142 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 101 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1402: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1402: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1402: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 641 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:30:41.891 INFO:tasks.rados.rados.0.smithi115.stdout:1401: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:30:41.963 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 1 to smithi11541303-31 2022-04-22T17:30:41.963 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 2 to smithi11541303-31 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 3 to smithi11541303-31 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1405: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1405: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 643 (ObjNum 290 snap 97 seq_num 290) dirty exists 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 4 to smithi11541303-31 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1401: done (3 left) 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1402: done (2 left) 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1405: done (1 left) 2022-04-22T17:30:41.964 INFO:tasks.rados.rados.0.smithi115.stdout:1406: copy_from oid 32 from oid 23 current snap is 142 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 5 to smithi11541303-31 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1404: finishing write tid 6 to smithi11541303-31 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 861 (ObjNum 420 snap 142 seq_num 420) dirty exists 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1404: left oid 31 (ObjNum 420 snap 142 seq_num 420) 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1404: done (1 left) 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1407: read oid 25 snap -1 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1407: expect (ObjNum 399 snap 137 seq_num 399) 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1408: read oid 29 snap -1 2022-04-22T17:30:41.965 INFO:tasks.rados.rados.0.smithi115.stdout:1408: expect deleted 2022-04-22T17:30:41.966 INFO:tasks.rados.rados.0.smithi115.stdout:1409: snap_create 2022-04-22T17:30:41.977 INFO:tasks.rados.rados.0.smithi115.stdout:1407: expect (ObjNum 399 snap 137 seq_num 399) 2022-04-22T17:30:42.044 INFO:tasks.rados.rados.0.smithi115.stdout:1406: finishing copy_from to smithi11541303-32 2022-04-22T17:30:42.070 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 536 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:30:42.070 INFO:tasks.rados.rados.0.smithi115.stdout:1406: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:30:42.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:41 smithi115 bash[46934]: cluster 2022-04-22T17:30:40.839933+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2022-04-22T17:30:42.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:41 smithi115 bash[45495]: cluster 2022-04-22T17:30:40.839933+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2022-04-22T17:30:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:41 smithi121 bash[36902]: cluster 2022-04-22T17:30:40.839933+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:1406: done (3 left) 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:1407: done (2 left) 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:1408: done (1 left) 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:1409: done (0 left) 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:1410: rollback oid 45 current snap is 143 2022-04-22T17:30:42.875 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 101 2022-04-22T17:30:42.876 INFO:tasks.rados.rados.0.smithi115.stdout:1411: rmattr oid 3 current snap is 143 2022-04-22T17:30:42.876 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 143 2022-04-22T17:30:42.876 INFO:tasks.rados.rados.0.smithi115.stdout:1412: seq_num 421 ranges {0=655360,655360=442368,1097728=663552,1761280=499712} 2022-04-22T17:30:42.878 INFO:tasks.rados.rados.0.smithi115.stdout:1412: writing smithi11541303-2 from 0 to 655360 tid 1 2022-04-22T17:30:42.881 INFO:tasks.rados.rados.0.smithi115.stdout:1412: writing smithi11541303-2 from 655360 to 1097728 tid 2 2022-04-22T17:30:42.887 INFO:tasks.rados.rados.0.smithi115.stdout:1412: writing smithi11541303-2 from 1097728 to 1761280 tid 3 2022-04-22T17:30:42.890 INFO:tasks.rados.rados.0.smithi115.stdout:1412: writing smithi11541303-2 from 1761280 to 2260992 tid 4 2022-04-22T17:30:42.891 INFO:tasks.rados.rados.0.smithi115.stdout:1413: read oid 39 snap -1 2022-04-22T17:30:42.891 INFO:tasks.rados.rados.0.smithi115.stdout:1413: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:30:42.891 INFO:tasks.rados.rados.0.smithi115.stdout:1414: snap_remove snap 135 2022-04-22T17:30:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[45495]: cluster 2022-04-22T17:30:41.176635+0000 mgr.y (mgr.34107) 140 : cluster [DBG] pgmap v206: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:43.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[45495]: cluster 2022-04-22T17:30:41.851626+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2022-04-22T17:30:43.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[45495]: audit 2022-04-22T17:30:42.377508+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[46934]: cluster 2022-04-22T17:30:41.176635+0000 mgr.y (mgr.34107) 140 : cluster [DBG] pgmap v206: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[46934]: cluster 2022-04-22T17:30:41.851626+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2022-04-22T17:30:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:42 smithi115 bash[46934]: audit 2022-04-22T17:30:42.377508+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:42 smithi121 bash[36902]: cluster 2022-04-22T17:30:41.176635+0000 mgr.y (mgr.34107) 140 : cluster [DBG] pgmap v206: 121 pgs: 121 active+clean; 295 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2022-04-22T17:30:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:42 smithi121 bash[36902]: cluster 2022-04-22T17:30:41.851626+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2022-04-22T17:30:43.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:42 smithi121 bash[36902]: audit 2022-04-22T17:30:42.377508+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:43.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:43.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:43.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:43.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:43.884 INFO:tasks.rados.rados.0.smithi115.stdout:1414: done (4 left) 2022-04-22T17:30:43.884 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 143 2022-04-22T17:30:43.884 INFO:tasks.rados.rados.0.smithi115.stdout:1410: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:30:43.885 INFO:tasks.rados.rados.0.smithi115.stdout:1415: seq_num 422 ranges {2850816=671744,3522560=729088,4251648=417792,4669440=458752} 2022-04-22T17:30:43.909 INFO:tasks.rados.rados.0.smithi115.stdout:1415: writing smithi11541303-36 from 2850816 to 3522560 tid 1 2022-04-22T17:30:43.914 INFO:tasks.rados.rados.0.smithi115.stdout:1415: writing smithi11541303-36 from 3522560 to 4251648 tid 2 2022-04-22T17:30:43.917 INFO:tasks.rados.rados.0.smithi115.stdout:1415: writing smithi11541303-36 from 4251648 to 4669440 tid 3 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1415: writing smithi11541303-36 from 4669440 to 5128192 tid 4 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1416: rollback oid 33 current snap is 143 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 142 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1410: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1410: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 815 (ObjNum 309 snap 101 seq_num 309) dirty exists 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 674 (ObjNum 348 snap 121 seq_num 348) dirty exists 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1410: done (5 left) 2022-04-22T17:30:43.922 INFO:tasks.rados.rados.0.smithi115.stdout:1411: done (4 left) 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1417: setattr oid 42 current snap is 143 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 1 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 2 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 3 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 4 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 5 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: finishing write tid 6 to smithi11541303-2 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 646 (ObjNum 421 snap 143 seq_num 421) dirty exists 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1412: left oid 2 (ObjNum 421 snap 143 seq_num 421) 2022-04-22T17:30:43.923 INFO:tasks.rados.rados.0.smithi115.stdout:1413: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:30:44.032 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 1 to smithi11541303-36 2022-04-22T17:30:44.032 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 2 to smithi11541303-36 2022-04-22T17:30:44.032 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 3 to smithi11541303-36 2022-04-22T17:30:44.032 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 4 to smithi11541303-36 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 5 to smithi11541303-36 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1415: finishing write tid 6 to smithi11541303-36 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 567 (ObjNum 422 snap 143 seq_num 422) dirty exists 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1415: left oid 36 (ObjNum 422 snap 143 seq_num 422) 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1416: finishing rollback tid 0 to smithi11541303-33 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1416: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1416: finishing rollback tid 2 to smithi11541303-33 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 557 (ObjNum 346 snap 120 seq_num 346) dirty exists 2022-04-22T17:30:44.033 INFO:tasks.rados.rados.0.smithi115.stdout:1412: done (4 left) 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:1413: done (3 left) 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:1415: done (2 left) 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:1416: done (1 left) 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:1418: rollback oid 8 current snap is 143 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 136 2022-04-22T17:30:44.034 INFO:tasks.rados.rados.0.smithi115.stdout:1419: snap_remove snap 140 2022-04-22T17:30:44.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[46934]: cephadm 2022-04-22T17:30:42.370236+0000 mgr.y (mgr.34107) 141 : cephadm [INF] Upgrade: Updating osd.1 2022-04-22T17:30:44.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[46934]: cephadm 2022-04-22T17:30:42.380115+0000 mgr.y (mgr.34107) 142 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:30:44.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[46934]: cluster 2022-04-22T17:30:42.858409+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2022-04-22T17:30:44.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[45495]: cephadm 2022-04-22T17:30:42.370236+0000 mgr.y (mgr.34107) 141 : cephadm [INF] Upgrade: Updating osd.1 2022-04-22T17:30:44.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[45495]: cephadm 2022-04-22T17:30:42.380115+0000 mgr.y (mgr.34107) 142 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:30:44.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:43 smithi115 bash[45495]: cluster 2022-04-22T17:30:42.858409+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2022-04-22T17:30:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:43 smithi121 bash[36902]: cephadm 2022-04-22T17:30:42.370236+0000 mgr.y (mgr.34107) 141 : cephadm [INF] Upgrade: Updating osd.1 2022-04-22T17:30:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:43 smithi121 bash[36902]: cephadm 2022-04-22T17:30:42.380115+0000 mgr.y (mgr.34107) 142 : cephadm [INF] Deploying daemon osd.1 on smithi115 2022-04-22T17:30:44.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:43 smithi121 bash[36902]: cluster 2022-04-22T17:30:42.858409+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2022-04-22T17:30:44.782 INFO:tasks.rados.rados.0.smithi115.stdout:1419: done (2 left) 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:1420: rollback oid 34 current snap is 143 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 142 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:1418: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 871 (ObjNum 414 snap 141 seq_num 414) dirty exists 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:1418: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:1418: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 571 (ObjNum 374 snap 129 seq_num 374) dirty exists 2022-04-22T17:30:44.783 INFO:tasks.rados.rados.0.smithi115.stdout:1417: done (2 left) 2022-04-22T17:30:44.784 INFO:tasks.rados.rados.0.smithi115.stdout:1418: done (1 left) 2022-04-22T17:30:44.784 INFO:tasks.rados.rados.0.smithi115.stdout:1421: delete oid 30 current snap is 143 2022-04-22T17:30:44.795 INFO:tasks.rados.rados.0.smithi115.stdout:1421: done (1 left) 2022-04-22T17:30:44.795 INFO:tasks.rados.rados.0.smithi115.stdout:1422: snap_remove snap 101 2022-04-22T17:30:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:44 smithi121 bash[36902]: cluster 2022-04-22T17:30:43.177630+0000 mgr.y (mgr.34107) 143 : cluster [DBG] pgmap v209: 121 pgs: 121 active+clean; 294 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:30:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:44 smithi121 bash[36902]: cluster 2022-04-22T17:30:43.869213+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2022-04-22T17:30:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:44 smithi121 bash[36902]: audit 2022-04-22T17:30:44.193430+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:45.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:44 smithi121 bash[36902]: cluster 2022-04-22T17:30:44.766985+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2022-04-22T17:30:45.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[45495]: cluster 2022-04-22T17:30:43.177630+0000 mgr.y (mgr.34107) 143 : cluster [DBG] pgmap v209: 121 pgs: 121 active+clean; 294 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[45495]: cluster 2022-04-22T17:30:43.869213+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[45495]: audit 2022-04-22T17:30:44.193430+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[45495]: cluster 2022-04-22T17:30:44.766985+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[46934]: cluster 2022-04-22T17:30:43.177630+0000 mgr.y (mgr.34107) 143 : cluster [DBG] pgmap v209: 121 pgs: 121 active+clean; 294 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[46934]: cluster 2022-04-22T17:30:43.869213+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[46934]: audit 2022-04-22T17:30:44.193430+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:44 smithi115 bash[46934]: cluster 2022-04-22T17:30:44.766985+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2022-04-22T17:30:45.657 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:30:45.794 INFO:tasks.rados.rados.0.smithi115.stdout:1422: done (1 left) 2022-04-22T17:30:45.890 INFO:tasks.rados.rados.0.smithi115.stdout:1423: read oid 22 snap 142 2022-04-22T17:30:45.890 INFO:tasks.rados.rados.0.smithi115.stdout:1423: expect deleted 2022-04-22T17:30:45.890 INFO:tasks.rados.rados.0.smithi115.stdout:1424: snap_remove snap 136 2022-04-22T17:30:46.023 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:30:46.023 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (7m) 31s ago 7m 12.9M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:30:46.023 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (7m) 95s ago 7m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:30:46.023 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (8m) 31s ago 8m 13.7M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (8m) 95s ago 8m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3m) 95s ago 12m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3m) 31s ago 15m 444M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2m) 31s ago 15m 51.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (107s) 95s ago 13m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2m) 31s ago 13m 43.6M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 31s ago 7m 9184k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:30:46.024 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (7m) 95s ago 7m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (33s) 31s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 31s ago 12m 203M 2526M 16.2.5 46cf6318c64e 8a8a72481c94 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 31s ago 11m 177M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (10m) 31s ago 10m 116M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (10m) 95s ago 10m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (10m) 95s ago 10m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:30:46.025 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (9m) 95s ago 9m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:30:46.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (9m) 95s ago 9m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:30:46.026 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3m) 95s ago 7m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:30:46.266 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:30:45 smithi121 bash[15740]: debug 2022-04-22T17:30:45.793+0000 7feef140b700 -1 osd.4 pg_epoch: 351 pg[5.es0( v 351'423 (0'0,351'423] local-lis/les=325/326 n=7 ec=63/63 lis/c=325/325 les/c/f=326/326/0 sis=325) [4,5,0]p4(0) r=0 lpr=325 luod=350'421 crt=350'421 lcod 350'419 mlcod 350'419 active+clean+snaptrim trimq=[bf~1] ps=[103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:46.267 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:30:45 smithi121 bash[24219]: debug 2022-04-22T17:30:45.793+0000 7f1df1444700 -1 osd.7 pg_epoch: 351 pg[5.9s0( v 351'829 (0'0,351'829] local-lis/les=63/64 n=13 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [7,6,1]p7(0) r=0 lpr=63 luod=350'827 crt=350'827 lcod 350'825 mlcod 350'825 active+clean+snaptrim trimq=[bf~1] ps=[103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:46.302 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:30:45 smithi115 bash[26675]: debug 2022-04-22T17:30:45.784+0000 7faa078b9700 -1 osd.2 pg_epoch: 351 pg[5.8s0( v 351'405 (0'0,351'405] local-lis/les=325/326 n=6 ec=63/63 lis/c=325/325 les/c/f=326/327/0 sis=325) [2,0,1]p2(0) r=0 lpr=325 luod=349'403 crt=349'403 lcod 349'401 mlcod 349'401 active+clean+snaptrim trimq=[bf~1] ps=[103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:46.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:30:45 smithi115 bash[26675]: debug 2022-04-22T17:30:45.804+0000 7faa028af700 -1 osd.2 pg_epoch: 351 pg[5.as0( v 351'581 (0'0,351'581] local-lis/les=63/64 n=9 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,4,3]p2(0) r=0 lpr=63 luod=351'579 crt=351'579 lcod 351'577 mlcod 351'577 active+clean+snaptrim trimq=[bf~1] ps=[103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:46.431 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:30:46.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 7, 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:30:46.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 9, 2022-04-22T17:30:46.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 6 2022-04-22T17:30:46.434 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:30:46.434 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:30:46.800 INFO:tasks.rados.rados.0.smithi115.stdout:1424: done (2 left) 2022-04-22T17:30:46.800 INFO:tasks.rados.rados.0.smithi115.stdout:1425: snap_create 2022-04-22T17:30:46.800 INFO:tasks.rados.rados.0.smithi115.stdout:1420: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:30:46.800 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 396 (ObjNum 1 snap 4 seq_num 0) dirty dne 2022-04-22T17:30:47.176 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[26675]: debug 2022-04-22T17:30:46.780+0000 7faa078b9700 -1 osd.2 pg_epoch: 352 pg[5.8s0( v 351'407 (0'0,351'407] local-lis/les=325/326 n=4 ec=63/63 lis/c=325/325 les/c/f=326/327/0 sis=325) [2,0,1]p2(0) r=0 lpr=325 crt=351'407 lcod 351'405 mlcod 351'405 active+clean+snaptrim trimq=[104~1] ps=[bf~1,103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:47.177 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[45495]: cluster 2022-04-22T17:30:45.178415+0000 mgr.y (mgr.34107) 144 : cluster [DBG] pgmap v212: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:30:47.177 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[45495]: audit 2022-04-22T17:30:45.628681+0000 mgr.y (mgr.34107) 145 : audit [DBG] from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.177 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[45495]: cluster 2022-04-22T17:30:45.778853+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2022-04-22T17:30:47.177 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[45495]: audit 2022-04-22T17:30:45.986922+0000 mgr.y (mgr.34107) 146 : audit [DBG] from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.178 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[45495]: audit 2022-04-22T17:30:46.415228+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/4153130612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:47.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[46934]: cluster 2022-04-22T17:30:45.178415+0000 mgr.y (mgr.34107) 144 : cluster [DBG] pgmap v212: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:30:47.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[46934]: audit 2022-04-22T17:30:45.628681+0000 mgr.y (mgr.34107) 145 : audit [DBG] from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[46934]: cluster 2022-04-22T17:30:45.778853+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2022-04-22T17:30:47.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[46934]: audit 2022-04-22T17:30:45.986922+0000 mgr.y (mgr.34107) 146 : audit [DBG] from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:46 smithi115 bash[46934]: audit 2022-04-22T17:30:46.415228+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/4153130612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[36902]: cluster 2022-04-22T17:30:45.178415+0000 mgr.y (mgr.34107) 144 : cluster [DBG] pgmap v212: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2022-04-22T17:30:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[36902]: audit 2022-04-22T17:30:45.628681+0000 mgr.y (mgr.34107) 145 : audit [DBG] from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[36902]: cluster 2022-04-22T17:30:45.778853+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2022-04-22T17:30:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[36902]: audit 2022-04-22T17:30:45.986922+0000 mgr.y (mgr.34107) 146 : audit [DBG] from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:30:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[36902]: audit 2022-04-22T17:30:46.415228+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.115:0/4153130612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:30:47.267 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[21417]: debug 2022-04-22T17:30:46.817+0000 7fcbfaaea700 -1 osd.6 pg_epoch: 352 pg[5.2s0( v 352'691 (0'0,352'691] local-lis/les=325/326 n=6 ec=63/63 lis/c=325/325 les/c/f=326/327/0 sis=325) [6,0,5]p6(0) r=0 lpr=325 crt=352'691 lcod 352'689 mlcod 352'689 active+clean+snaptrim trimq=[104~1] ps=[bf~1,103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:47.268 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[24219]: debug 2022-04-22T17:30:46.813+0000 7f1df3c49700 -1 osd.7 pg_epoch: 352 pg[5.4s0( v 351'635 (0'0,351'635] local-lis/les=63/64 n=7 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [7,2,5]p7(0) r=0 lpr=63 crt=351'635 lcod 351'633 mlcod 351'633 active+clean+snaptrim trimq=[104~1] ps=[bf~1,103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:47.268 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:30:46 smithi121 bash[24219]: debug 2022-04-22T17:30:46.817+0000 7f1df3c49700 -1 osd.7 pg_epoch: 352 pg[5.4s0( v 352'639 (0'0,352'639] local-lis/les=63/64 n=4 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [7,2,5]p7(0) r=0 lpr=63 crt=352'639 lcod 352'637 mlcod 352'637 active+clean+snaptrim trimq=[104~1] ps=[bf~1,103~1,109~1,10e~1,111~1]] removing snap head 2022-04-22T17:30:47.729 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:47 smithi115 systemd[1]: Stopping Ceph osd.1 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:30:47.729 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[23816]: debug 2022-04-22T17:30:47.568+0000 7fa488d02700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:30:47.729 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[23816]: debug 2022-04-22T17:30:47.568+0000 7fa488d02700 -1 osd.1 352 *** Got signal Terminated *** 2022-04-22T17:30:47.729 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[23816]: debug 2022-04-22T17:30:47.568+0000 7fa488d02700 -1 osd.1 352 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:30:47.800 INFO:tasks.rados.rados.0.smithi115.stdout:1420: done (2 left) 2022-04-22T17:30:47.800 INFO:tasks.rados.rados.0.smithi115.stdout:1423: done (1 left) 2022-04-22T17:30:47.800 INFO:tasks.rados.rados.0.smithi115.stdout:1425: done (0 left) 2022-04-22T17:30:47.801 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 144 2022-04-22T17:30:47.801 INFO:tasks.rados.rados.0.smithi115.stdout:1426: seq_num 424 ranges {2891776=704512,3596288=491520,4087808=434176,4521984=679936} 2022-04-22T17:30:47.822 INFO:tasks.rados.rados.0.smithi115.stdout:1426: writing smithi11541303-41 from 2891776 to 3596288 tid 1 2022-04-22T17:30:47.825 INFO:tasks.rados.rados.0.smithi115.stdout:1426: writing smithi11541303-41 from 3596288 to 4087808 tid 2 2022-04-22T17:30:47.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: writing smithi11541303-41 from 4087808 to 4521984 tid 3 2022-04-22T17:30:47.831 INFO:tasks.rados.rados.0.smithi115.stdout:1426: writing smithi11541303-41 from 4521984 to 5201920 tid 4 2022-04-22T17:30:47.831 INFO:tasks.rados.rados.0.smithi115.stdout:1427: snap_remove snap 142 2022-04-22T17:30:48.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[45495]: cluster 2022-04-22T17:30:46.784159+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2022-04-22T17:30:48.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.785596+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2022-04-22T17:30:48.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[46934]: cluster 2022-04-22T17:30:46.784159+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2022-04-22T17:30:48.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:47 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.785596+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2022-04-22T17:30:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:47 smithi121 bash[36902]: cluster 2022-04-22T17:30:46.784159+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2022-04-22T17:30:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:47 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.785596+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1427: done (1 left) 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1428: read oid 7 snap -1 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1428: expect (ObjNum 400 snap 137 seq_num 400) 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1429: rollback oid 15 current snap is 144 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 143 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 1 to smithi11541303-41 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 2 to smithi11541303-41 2022-04-22T17:30:48.826 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 3 to smithi11541303-41 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 4 to smithi11541303-41 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 5 to smithi11541303-41 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: finishing write tid 6 to smithi11541303-41 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 648 (ObjNum 424 snap 144 seq_num 424) dirty exists 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: left oid 41 (ObjNum 424 snap 144 seq_num 424) 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1426: done (2 left) 2022-04-22T17:30:48.827 INFO:tasks.rados.rados.0.smithi115.stdout:1430: setattr oid 12 current snap is 144 2022-04-22T17:30:48.829 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 144 2022-04-22T17:30:48.829 INFO:tasks.rados.rados.0.smithi115.stdout:1431: seq_num 425 ranges {0=720896,720896=778240,1499136=638976,2138112=188416} 2022-04-22T17:30:48.835 INFO:tasks.rados.rados.0.smithi115.stdout:1431: writing smithi11541303-29 from 0 to 720896 tid 1 2022-04-22T17:30:48.841 INFO:tasks.rados.rados.0.smithi115.stdout:1431: writing smithi11541303-29 from 720896 to 1499136 tid 2 2022-04-22T17:30:48.846 INFO:tasks.rados.rados.0.smithi115.stdout:1431: writing smithi11541303-29 from 1499136 to 2138112 tid 3 2022-04-22T17:30:48.847 INFO:tasks.rados.rados.0.smithi115.stdout:1431: writing smithi11541303-29 from 2138112 to 2326528 tid 4 2022-04-22T17:30:48.848 INFO:tasks.rados.rados.0.smithi115.stdout:1432: snap_remove snap 141 2022-04-22T17:30:49.053 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[51296]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.1 2022-04-22T17:30:49.088 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[51340]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-1 2022-04-22T17:30:49.088 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[51353]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.1 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.179150+0000 mgr.y (mgr.34107) 147 : cluster [DBG] pgmap v215: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.882946+0000 mon.a (mon.0) 406 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.883196+0000 mon.a (mon.0) 407 : cluster [INF] osd.1 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.883790+0000 mon.a (mon.0) 408 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.884240+0000 mon.a (mon.0) 409 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.884625+0000 mon.a (mon.0) 410 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.884848+0000 mon.a (mon.0) 411 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.885161+0000 mon.a (mon.0) 412 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.885369+0000 mon.a (mon.0) 413 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.885569+0000 mon.a (mon.0) 414 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.885775+0000 mon.a (mon.0) 415 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.885967+0000 mon.a (mon.0) 416 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.886243+0000 mon.a (mon.0) 417 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.886439+0000 mon.a (mon.0) 418 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.886734+0000 mon.a (mon.0) 419 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:47.887053+0000 mon.a (mon.0) 420 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.286963+0000 mon.a (mon.0) 421 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.287669+0000 mon.a (mon.0) 422 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.288094+0000 mon.a (mon.0) 423 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.288210+0000 mon.a (mon.0) 424 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.288535+0000 mon.a (mon.0) 425 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.288835+0000 mon.a (mon.0) 426 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.288956+0000 mon.a (mon.0) 427 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.289209+0000 mon.a (mon.0) 428 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.289323+0000 mon.a (mon.0) 429 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.289571+0000 mon.a (mon.0) 430 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.289750+0000 mon.a (mon.0) 431 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.290029+0000 mon.a (mon.0) 432 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.290296+0000 mon.a (mon.0) 433 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.290549+0000 mon.a (mon.0) 434 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.179150+0000 mgr.y (mgr.34107) 147 : cluster [DBG] pgmap v215: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.882946+0000 mon.a (mon.0) 406 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.883196+0000 mon.a (mon.0) 407 : cluster [INF] osd.1 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.883790+0000 mon.a (mon.0) 408 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.884240+0000 mon.a (mon.0) 409 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.884625+0000 mon.a (mon.0) 410 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.093 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.884848+0000 mon.a (mon.0) 411 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.885161+0000 mon.a (mon.0) 412 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.885369+0000 mon.a (mon.0) 413 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.885569+0000 mon.a (mon.0) 414 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.885775+0000 mon.a (mon.0) 415 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.885967+0000 mon.a (mon.0) 416 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.886243+0000 mon.a (mon.0) 417 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.886439+0000 mon.a (mon.0) 418 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.886734+0000 mon.a (mon.0) 419 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:47.887053+0000 mon.a (mon.0) 420 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.286963+0000 mon.a (mon.0) 421 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.287669+0000 mon.a (mon.0) 422 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.288094+0000 mon.a (mon.0) 423 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.288210+0000 mon.a (mon.0) 424 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.288535+0000 mon.a (mon.0) 425 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.288835+0000 mon.a (mon.0) 426 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.288956+0000 mon.a (mon.0) 427 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.289209+0000 mon.a (mon.0) 428 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.289323+0000 mon.a (mon.0) 429 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.289571+0000 mon.a (mon.0) 430 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.289750+0000 mon.a (mon.0) 431 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.290029+0000 mon.a (mon.0) 432 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.290296+0000 mon.a (mon.0) 433 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:48 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.290549+0000 mon.a (mon.0) 434 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.179150+0000 mgr.y (mgr.34107) 147 : cluster [DBG] pgmap v215: 121 pgs: 121 active+clean; 313 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.882946+0000 mon.a (mon.0) 406 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.883196+0000 mon.a (mon.0) 407 : cluster [INF] osd.1 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.883790+0000 mon.a (mon.0) 408 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.884240+0000 mon.a (mon.0) 409 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.884625+0000 mon.a (mon.0) 410 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.884848+0000 mon.a (mon.0) 411 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.885161+0000 mon.a (mon.0) 412 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.885369+0000 mon.a (mon.0) 413 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.885569+0000 mon.a (mon.0) 414 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.885775+0000 mon.a (mon.0) 415 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.885967+0000 mon.a (mon.0) 416 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.886243+0000 mon.a (mon.0) 417 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.886439+0000 mon.a (mon.0) 418 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.886734+0000 mon.a (mon.0) 419 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:47.887053+0000 mon.a (mon.0) 420 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.286963+0000 mon.a (mon.0) 421 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.287669+0000 mon.a (mon.0) 422 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.288094+0000 mon.a (mon.0) 423 : cluster [DBG] osd.1 reported immediately failed by osd.6 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.288210+0000 mon.a (mon.0) 424 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.288535+0000 mon.a (mon.0) 425 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.288835+0000 mon.a (mon.0) 426 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.288956+0000 mon.a (mon.0) 427 : cluster [DBG] osd.1 reported immediately failed by osd.5 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.289209+0000 mon.a (mon.0) 428 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.289323+0000 mon.a (mon.0) 429 : cluster [DBG] osd.1 reported immediately failed by osd.4 2022-04-22T17:30:49.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.289571+0000 mon.a (mon.0) 430 : cluster [DBG] osd.1 reported immediately failed by osd.3 2022-04-22T17:30:49.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.289750+0000 mon.a (mon.0) 431 : cluster [DBG] osd.1 reported immediately failed by osd.7 2022-04-22T17:30:49.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.290029+0000 mon.a (mon.0) 432 : cluster [DBG] osd.1 reported immediately failed by osd.2 2022-04-22T17:30:49.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.290296+0000 mon.a (mon.0) 433 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:48 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.290549+0000 mon.a (mon.0) 434 : cluster [DBG] osd.1 reported immediately failed by osd.0 2022-04-22T17:30:49.835 INFO:tasks.rados.rados.0.smithi115.stdout:1432: done (4 left) 2022-04-22T17:30:49.835 INFO:tasks.rados.rados.0.smithi115.stdout:1433: rollback oid 34 current snap is 144 2022-04-22T17:30:49.835 INFO:tasks.rados.rados.0.smithi115.stdout:1428: expect (ObjNum 400 snap 137 seq_num 400) 2022-04-22T17:30:50.084 INFO:tasks.rados.rados.0.smithi115.stdout:1429: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:30:50.084 INFO:tasks.rados.rados.0.smithi115.stdout:1429: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:30:50.084 INFO:tasks.rados.rados.0.smithi115.stdout:1429: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:30:50.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 226 (ObjNum 407 snap 138 seq_num 407) dirty exists 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 547 (ObjNum 361 snap 123 seq_num 361) dirty exists 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 1 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 2 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 3 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 4 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 5 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: finishing write tid 6 to smithi11541303-29 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 474 (ObjNum 425 snap 144 seq_num 425) dirty exists 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:1431: left oid 29 (ObjNum 425 snap 144 seq_num 425) 2022-04-22T17:30:50.085 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 143 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1428: done (4 left) 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1429: done (3 left) 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1430: done (2 left) 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1431: done (1 left) 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1434: read oid 6 snap -1 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1434: expect (ObjNum 417 snap 141 seq_num 417) 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 144 2022-04-22T17:30:50.086 INFO:tasks.rados.rados.0.smithi115.stdout:1435: seq_num 426 ranges {565248=737280,1302528=712704,2015232=802816,2818048=90112} 2022-04-22T17:30:50.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.797161+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:50.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[46934]: cluster 2022-04-22T17:30:48.809948+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e354: 8 total, 7 up, 8 in 2022-04-22T17:30:50.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[46934]: audit 2022-04-22T17:30:49.196090+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:50.088 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.797161+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:50.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[45495]: cluster 2022-04-22T17:30:48.809948+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e354: 8 total, 7 up, 8 in 2022-04-22T17:30:50.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:49 smithi115 bash[45495]: audit 2022-04-22T17:30:49.196090+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:50.091 INFO:tasks.rados.rados.0.smithi115.stdout:1435: writing smithi11541303-47 from 565248 to 1302528 tid 1 2022-04-22T17:30:50.095 INFO:tasks.rados.rados.0.smithi115.stdout:1435: writing smithi11541303-47 from 1302528 to 2015232 tid 2 2022-04-22T17:30:50.100 INFO:tasks.rados.rados.0.smithi115.stdout:1435: writing smithi11541303-47 from 2015232 to 2818048 tid 3 2022-04-22T17:30:50.101 INFO:tasks.rados.rados.0.smithi115.stdout:1435: writing smithi11541303-47 from 2818048 to 2908160 tid 4 2022-04-22T17:30:50.101 INFO:tasks.rados.rados.0.smithi115.stdout:1436: copy_from oid 31 from oid 25 current snap is 144 2022-04-22T17:30:50.101 INFO:tasks.rados.rados.0.smithi115.stdout:1433: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:30:50.101 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 0 (ObjNum -1116461984 snap 22075 seq_num 2815700176) dirty dne 2022-04-22T17:30:50.102 INFO:tasks.rados.rados.0.smithi115.stdout:1433: done (3 left) 2022-04-22T17:30:50.102 INFO:tasks.rados.rados.0.smithi115.stdout:1437: snap_create 2022-04-22T17:30:50.102 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 1 to smithi11541303-47 2022-04-22T17:30:50.102 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 2 to smithi11541303-47 2022-04-22T17:30:50.109 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 3 to smithi11541303-47 2022-04-22T17:30:50.109 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 4 to smithi11541303-47 2022-04-22T17:30:50.110 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 5 to smithi11541303-47 2022-04-22T17:30:50.112 INFO:tasks.rados.rados.0.smithi115.stdout:1435: finishing write tid 6 to smithi11541303-47 2022-04-22T17:30:50.112 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 657 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:30:50.112 INFO:tasks.rados.rados.0.smithi115.stdout:1435: left oid 47 (ObjNum 426 snap 144 seq_num 426) 2022-04-22T17:30:50.117 INFO:tasks.rados.rados.0.smithi115.stdout:1434: expect (ObjNum 417 snap 141 seq_num 417) 2022-04-22T17:30:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:49 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.797161+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:30:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:49 smithi121 bash[36902]: cluster 2022-04-22T17:30:48.809948+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e354: 8 total, 7 up, 8 in 2022-04-22T17:30:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:49 smithi121 bash[36902]: audit 2022-04-22T17:30:49.196090+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:50.330 INFO:tasks.rados.rados.0.smithi115.stdout:1436: finishing copy_from to smithi11541303-31 2022-04-22T17:30:50.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 613 (ObjNum 399 snap 137 seq_num 399) dirty exists 2022-04-22T17:30:50.330 INFO:tasks.rados.rados.0.smithi115.stdout:1436: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:30:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1434: done (3 left) 2022-04-22T17:30:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1435: done (2 left) 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1436: done (1 left) 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1437: done (0 left) 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1438: rmattr oid 27 current snap is 145 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1438: done (0 left) 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1439: rollback oid 10 current snap is 145 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 144 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1440: read oid 14 snap -1 2022-04-22T17:30:50.845 INFO:tasks.rados.rados.0.smithi115.stdout:1440: expect (ObjNum 361 snap 123 seq_num 361) 2022-04-22T17:30:50.846 INFO:tasks.rados.rados.0.smithi115.stdout:1441: read oid 42 snap -1 2022-04-22T17:30:50.846 INFO:tasks.rados.rados.0.smithi115.stdout:1441: expect (ObjNum 414 snap 141 seq_num 414) 2022-04-22T17:30:50.846 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 145 2022-04-22T17:30:50.846 INFO:tasks.rados.rados.0.smithi115.stdout:1442: seq_num 427 ranges {2031616=761856,2793472=598016,3391488=606208,3997696=401408} 2022-04-22T17:30:50.861 INFO:tasks.rados.rados.0.smithi115.stdout:1442: writing smithi11541303-15 from 2031616 to 2793472 tid 1 2022-04-22T17:30:50.867 INFO:tasks.rados.rados.0.smithi115.stdout:1442: writing smithi11541303-15 from 2793472 to 3391488 tid 2 2022-04-22T17:30:50.871 INFO:tasks.rados.rados.0.smithi115.stdout:1442: writing smithi11541303-15 from 3391488 to 3997696 tid 3 2022-04-22T17:30:50.873 INFO:tasks.rados.rados.0.smithi115.stdout:1442: writing smithi11541303-15 from 3997696 to 4399104 tid 4 2022-04-22T17:30:50.874 INFO:tasks.rados.rados.0.smithi115.stdout:1443: copy_from oid 36 from oid 37 current snap is 145 2022-04-22T17:30:50.874 INFO:tasks.rados.rados.0.smithi115.stdout:1439: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:30:50.874 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:30:50.874 INFO:tasks.rados.rados.0.smithi115.stdout:1440: expect (ObjNum 361 snap 123 seq_num 361) 2022-04-22T17:30:50.953 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 1 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 2 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 3 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 4 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 5 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:1442: finishing write tid 6 to smithi11541303-15 2022-04-22T17:30:50.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 232 (ObjNum 427 snap 145 seq_num 427) dirty exists 2022-04-22T17:30:50.955 INFO:tasks.rados.rados.0.smithi115.stdout:1442: left oid 15 (ObjNum 427 snap 145 seq_num 427) 2022-04-22T17:30:50.955 INFO:tasks.rados.rados.0.smithi115.stdout:1443: finishing copy_from to smithi11541303-36 2022-04-22T17:30:50.955 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 641 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:30:50.955 INFO:tasks.rados.rados.0.smithi115.stdout:1443: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:30:50.955 INFO:tasks.rados.rados.0.smithi115.stdout:1441: expect (ObjNum 414 snap 141 seq_num 414) 2022-04-22T17:30:51.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[45495]: cluster 2022-04-22T17:30:49.180826+0000 mgr.y (mgr.34107) 148 : cluster [DBG] pgmap v218: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:51.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[45495]: cluster 2022-04-22T17:30:49.809161+0000 mon.a (mon.0) 438 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2022-04-22T17:30:51.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[45495]: cluster 2022-04-22T17:30:49.819108+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e355: 8 total, 7 up, 8 in 2022-04-22T17:30:51.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[46934]: cluster 2022-04-22T17:30:49.180826+0000 mgr.y (mgr.34107) 148 : cluster [DBG] pgmap v218: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:51.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[46934]: cluster 2022-04-22T17:30:49.809161+0000 mon.a (mon.0) 438 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2022-04-22T17:30:51.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:50 smithi115 bash[46934]: cluster 2022-04-22T17:30:49.819108+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e355: 8 total, 7 up, 8 in 2022-04-22T17:30:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:50 smithi121 bash[36902]: cluster 2022-04-22T17:30:49.180826+0000 mgr.y (mgr.34107) 148 : cluster [DBG] pgmap v218: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:50 smithi121 bash[36902]: cluster 2022-04-22T17:30:49.809161+0000 mon.a (mon.0) 438 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2022-04-22T17:30:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:50 smithi121 bash[36902]: cluster 2022-04-22T17:30:49.819108+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e355: 8 total, 7 up, 8 in 2022-04-22T17:30:51.413 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:51 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.1.service: Succeeded. 2022-04-22T17:30:51.413 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:51 smithi115 systemd[1]: Stopped Ceph osd.1 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:30:51.413 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:51 smithi115 systemd[1]: Started Ceph osd.1 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:30:51.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:51.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:51.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:51.621Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:51.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:51.622Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:52.031 INFO:tasks.rados.rados.0.smithi115.stdout:1439: done (4 left) 2022-04-22T17:30:52.031 INFO:tasks.rados.rados.0.smithi115.stdout:1440: done (3 left) 2022-04-22T17:30:52.032 INFO:tasks.rados.rados.0.smithi115.stdout:1441: done (2 left) 2022-04-22T17:30:52.032 INFO:tasks.rados.rados.0.smithi115.stdout:1442: done (1 left) 2022-04-22T17:30:52.032 INFO:tasks.rados.rados.0.smithi115.stdout:1443: done (0 left) 2022-04-22T17:30:52.032 INFO:tasks.rados.rados.0.smithi115.stdout:1444: snap_remove snap 143 2022-04-22T17:30:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:51 smithi121 bash[36902]: cluster 2022-04-22T17:30:50.828585+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e356: 8 total, 7 up, 8 in 2022-04-22T17:30:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:51 smithi121 bash[36902]: audit 2022-04-22T17:30:51.410477+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:52.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:51 smithi121 bash[36902]: audit 2022-04-22T17:30:51.412113+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:52.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:51 smithi121 bash[36902]: audit 2022-04-22T17:30:51.413841+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:52.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:51 smithi121 bash[36902]: audit 2022-04-22T17:30:51.414761+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[45495]: cluster 2022-04-22T17:30:50.828585+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e356: 8 total, 7 up, 8 in 2022-04-22T17:30:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[45495]: audit 2022-04-22T17:30:51.410477+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[45495]: audit 2022-04-22T17:30:51.412113+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[45495]: audit 2022-04-22T17:30:51.413841+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[45495]: audit 2022-04-22T17:30:51.414761+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:52.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[46934]: cluster 2022-04-22T17:30:50.828585+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e356: 8 total, 7 up, 8 in 2022-04-22T17:30:52.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[46934]: audit 2022-04-22T17:30:51.410477+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:52.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[46934]: audit 2022-04-22T17:30:51.412113+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:30:52.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[46934]: audit 2022-04-22T17:30:51.413841+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:30:52.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:51 smithi115 bash[46934]: audit 2022-04-22T17:30:51.414761+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:30:52.860 INFO:tasks.rados.rados.0.smithi115.stdout:1444: done (0 left) 2022-04-22T17:30:52.860 INFO:tasks.rados.rados.0.smithi115.stdout:1445: snap_create 2022-04-22T17:30:53.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:52 smithi115 bash[46934]: cluster 2022-04-22T17:30:51.181792+0000 mgr.y (mgr.34107) 149 : cluster [DBG] pgmap v221: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:53.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:52 smithi115 bash[45495]: cluster 2022-04-22T17:30:51.181792+0000 mgr.y (mgr.34107) 149 : cluster [DBG] pgmap v221: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:52 smithi121 bash[36902]: cluster 2022-04-22T17:30:51.181792+0000 mgr.y (mgr.34107) 149 : cluster [DBG] pgmap v221: 121 pgs: 36 peering, 2 stale+active+clean, 83 active+clean; 141 MiB data, 1020 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 564 KiB/s wr, 2 op/s 2022-04-22T17:30:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:53.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:53.852 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:53.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:30:53.852 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:30:53.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:30:53.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-04-22T17:30:53.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2022-04-22T17:30:53.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2022-04-22T17:30:53.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2022-04-22T17:30:53.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2022-04-22T17:30:53.853 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-04-22T17:30:53.853 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[51538]: --> ceph-volume raw activate successful for osd ID: 1 2022-04-22T17:30:53.861 INFO:tasks.rados.rados.0.smithi115.stdout:1445: done (0 left) 2022-04-22T17:30:53.862 INFO:tasks.rados.rados.0.smithi115.stdout:1446: delete oid 39 current snap is 146 2022-04-22T17:30:53.867 INFO:tasks.rados.rados.0.smithi115.stdout:1446: done (0 left) 2022-04-22T17:30:53.867 INFO:tasks.rados.rados.0.smithi115.stdout:1447: snap_remove snap 145 2022-04-22T17:30:54.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[45495]: cluster 2022-04-22T17:30:52.844844+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e357: 8 total, 7 up, 8 in 2022-04-22T17:30:54.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:53 smithi115 bash[46934]: cluster 2022-04-22T17:30:52.844844+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e357: 8 total, 7 up, 8 in 2022-04-22T17:30:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:53 smithi121 bash[36902]: cluster 2022-04-22T17:30:52.844844+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e357: 8 total, 7 up, 8 in 2022-04-22T17:30:54.790 INFO:tasks.rados.rados.0.smithi115.stdout:1447: done (0 left) 2022-04-22T17:30:54.790 INFO:tasks.rados.rados.0.smithi115.stdout:1448: snap_remove snap 144 2022-04-22T17:30:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:54 smithi121 bash[36902]: cluster 2022-04-22T17:30:53.182946+0000 mgr.y (mgr.34107) 150 : cluster [DBG] pgmap v223: 121 pgs: 1 active+undersized+degraded, 36 peering, 7 active+undersized, 77 active+clean; 141 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 516 KiB/s wr, 2 op/s; 1/228 objects degraded (0.439%) 2022-04-22T17:30:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:54 smithi121 bash[36902]: cluster 2022-04-22T17:30:53.841248+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Degraded data redundancy: 1/228 objects degraded (0.439%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:30:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:54 smithi121 bash[36902]: cluster 2022-04-22T17:30:53.846411+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e358: 8 total, 7 up, 8 in 2022-04-22T17:30:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:54 smithi121 bash[36902]: audit 2022-04-22T17:30:54.198998+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:54 smithi121 bash[36902]: cluster 2022-04-22T17:30:54.774451+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e359: 8 total, 7 up, 8 in 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[45495]: cluster 2022-04-22T17:30:53.182946+0000 mgr.y (mgr.34107) 150 : cluster [DBG] pgmap v223: 121 pgs: 1 active+undersized+degraded, 36 peering, 7 active+undersized, 77 active+clean; 141 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 516 KiB/s wr, 2 op/s; 1/228 objects degraded (0.439%) 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[45495]: cluster 2022-04-22T17:30:53.841248+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Degraded data redundancy: 1/228 objects degraded (0.439%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[45495]: cluster 2022-04-22T17:30:53.846411+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e358: 8 total, 7 up, 8 in 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[45495]: audit 2022-04-22T17:30:54.198998+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[45495]: cluster 2022-04-22T17:30:54.774451+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e359: 8 total, 7 up, 8 in 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[46934]: cluster 2022-04-22T17:30:53.182946+0000 mgr.y (mgr.34107) 150 : cluster [DBG] pgmap v223: 121 pgs: 1 active+undersized+degraded, 36 peering, 7 active+undersized, 77 active+clean; 141 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 516 KiB/s wr, 2 op/s; 1/228 objects degraded (0.439%) 2022-04-22T17:30:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[46934]: cluster 2022-04-22T17:30:53.841248+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Degraded data redundancy: 1/228 objects degraded (0.439%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:30:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[46934]: cluster 2022-04-22T17:30:53.846411+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e358: 8 total, 7 up, 8 in 2022-04-22T17:30:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[46934]: audit 2022-04-22T17:30:54.198998+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:30:55.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:54 smithi115 bash[46934]: cluster 2022-04-22T17:30:54.774451+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e359: 8 total, 7 up, 8 in 2022-04-22T17:30:55.790 INFO:tasks.rados.rados.0.smithi115.stdout:1448: done (0 left) 2022-04-22T17:30:55.791 INFO:tasks.rados.rados.0.smithi115.stdout:1449: rollback oid 19 current snap is 146 2022-04-22T17:30:55.791 INFO:tasks.rados.rados.0.smithi115.stdout:1449: done (0 left) 2022-04-22T17:30:55.791 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 146 2022-04-22T17:30:55.791 INFO:tasks.rados.rados.0.smithi115.stdout:1450: seq_num 429 ranges {376832=794624,1171456=770048,1941504=622592,2564096=212992} 2022-04-22T17:30:55.798 INFO:tasks.rados.rados.0.smithi115.stdout:1450: writing smithi11541303-45 from 376832 to 1171456 tid 1 2022-04-22T17:30:55.803 INFO:tasks.rados.rados.0.smithi115.stdout:1450: writing smithi11541303-45 from 1171456 to 1941504 tid 2 2022-04-22T17:30:55.807 INFO:tasks.rados.rados.0.smithi115.stdout:1450: writing smithi11541303-45 from 1941504 to 2564096 tid 3 2022-04-22T17:30:55.809 INFO:tasks.rados.rados.0.smithi115.stdout:1450: writing smithi11541303-45 from 2564096 to 2777088 tid 4 2022-04-22T17:30:55.809 INFO:tasks.rados.rados.0.smithi115.stdout:1451: snap_create 2022-04-22T17:30:55.809 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 1 to smithi11541303-45 2022-04-22T17:30:55.809 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 2 to smithi11541303-45 2022-04-22T17:30:55.812 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 3 to smithi11541303-45 2022-04-22T17:30:55.813 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 4 to smithi11541303-45 2022-04-22T17:30:55.813 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 5 to smithi11541303-45 2022-04-22T17:30:55.814 INFO:tasks.rados.rados.0.smithi115.stdout:1450: finishing write tid 6 to smithi11541303-45 2022-04-22T17:30:55.814 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 848 (ObjNum 429 snap 146 seq_num 429) dirty exists 2022-04-22T17:30:55.814 INFO:tasks.rados.rados.0.smithi115.stdout:1450: left oid 45 (ObjNum 429 snap 146 seq_num 429) 2022-04-22T17:30:55.946 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:55 smithi115 bash[51714]: debug 2022-04-22T17:30:55.696+0000 7f694209c700 1 -- 172.21.15.115:0/3730811379 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56136cc5a340 con 0x56136ccde800 2022-04-22T17:30:56.276 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:30:55 smithi115 bash[26675]: debug 2022-04-22T17:30:55.924+0000 7faa010ac700 -1 osd.2 pg_epoch: 360 pg[5.ds0( v 359'525 (0'0,359'525] local-lis/les=63/64 n=5 ec=63/63 lis/c=63/63 les/c/f=64/64/0 sis=63) [2,7,5]p2(0) r=0 lpr=63 crt=359'525 lcod 358'523 mlcod 358'523 active+clean+snaptrim trimq=[121~1] ps=[118~2,11e~1,123~1]] removing snap head 2022-04-22T17:30:56.552 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[51714]: debug 2022-04-22T17:30:56.256+0000 7f69496933c0 -1 Falling back to public interface 2022-04-22T17:30:56.799 INFO:tasks.rados.rados.0.smithi115.stdout:1450: done (1 left) 2022-04-22T17:30:56.799 INFO:tasks.rados.rados.0.smithi115.stdout:1451: done (0 left) 2022-04-22T17:30:56.799 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 147 2022-04-22T17:30:56.799 INFO:tasks.rados.rados.0.smithi115.stdout:1452: seq_num 430 ranges {3080192=8192} 2022-04-22T17:30:56.818 INFO:tasks.rados.rados.0.smithi115.stdout:1452: writing smithi11541303-19 from 3080192 to 3088384 tid 1 2022-04-22T17:30:56.818 INFO:tasks.rados.rados.0.smithi115.stdout:1453: copy_from oid 10 from oid 3 current snap is 147 2022-04-22T17:30:56.819 INFO:tasks.rados.rados.0.smithi115.stdout:1454: snap_remove snap 146 2022-04-22T17:30:57.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[45495]: cluster 2022-04-22T17:30:55.183659+0000 mgr.y (mgr.34107) 151 : cluster [DBG] pgmap v226: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 13 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:57.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[45495]: cluster 2022-04-22T17:30:55.774976+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e360: 8 total, 7 up, 8 in 2022-04-22T17:30:57.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[45495]: cluster 2022-04-22T17:30:55.851372+0000 mon.a (mon.0) 451 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2022-04-22T17:30:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[46934]: cluster 2022-04-22T17:30:55.183659+0000 mgr.y (mgr.34107) 151 : cluster [DBG] pgmap v226: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 13 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[46934]: cluster 2022-04-22T17:30:55.774976+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e360: 8 total, 7 up, 8 in 2022-04-22T17:30:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:56 smithi115 bash[46934]: cluster 2022-04-22T17:30:55.851372+0000 mon.a (mon.0) 451 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2022-04-22T17:30:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:56 smithi121 bash[36902]: cluster 2022-04-22T17:30:55.183659+0000 mgr.y (mgr.34107) 151 : cluster [DBG] pgmap v226: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 13 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:56 smithi121 bash[36902]: cluster 2022-04-22T17:30:55.774976+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e360: 8 total, 7 up, 8 in 2022-04-22T17:30:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:56 smithi121 bash[36902]: cluster 2022-04-22T17:30:55.851372+0000 mon.a (mon.0) 451 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2022-04-22T17:30:57.801 INFO:tasks.rados.rados.0.smithi115.stdout:1454: done (2 left) 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1455: delete oid 29 current snap is 147 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1452: finishing write tid 1 to smithi11541303-19 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1452: finishing write tid 2 to smithi11541303-19 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1452: finishing write tid 3 to smithi11541303-19 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 550 (ObjNum 430 snap 147 seq_num 430) dirty exists 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1452: left oid 19 (ObjNum 430 snap 147 seq_num 430) 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1453: finishing copy_from to smithi11541303-10 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 674 (ObjNum 348 snap 121 seq_num 348) dirty exists 2022-04-22T17:30:57.802 INFO:tasks.rados.rados.0.smithi115.stdout:1453: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:30:57.814 INFO:tasks.rados.rados.0.smithi115.stdout:1452: done (2 left) 2022-04-22T17:30:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:1453: done (1 left) 2022-04-22T17:30:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:1455: done (0 left) 2022-04-22T17:30:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:1456: read oid 24 snap -1 2022-04-22T17:30:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:1456: expect (ObjNum 290 snap 97 seq_num 290) 2022-04-22T17:30:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 147 2022-04-22T17:30:57.867 INFO:tasks.rados.rados.0.smithi115.stdout:1457: seq_num 432 ranges {1376256=40960} 2022-04-22T17:30:57.869 INFO:tasks.rados.rados.0.smithi115.stdout:1457: writing smithi11541303-17 from 1376256 to 1417216 tid 1 2022-04-22T17:30:57.869 INFO:tasks.rados.rados.0.smithi115.stdout:1458: read oid 36 snap -1 2022-04-22T17:30:57.869 INFO:tasks.rados.rados.0.smithi115.stdout:1458: expect (ObjNum 390 snap 136 seq_num 390) 2022-04-22T17:30:57.869 INFO:tasks.rados.rados.0.smithi115.stdout:1459: snap_create 2022-04-22T17:30:57.869 INFO:tasks.rados.rados.0.smithi115.stdout:1457: finishing write tid 1 to smithi11541303-17 2022-04-22T17:30:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:1457: finishing write tid 2 to smithi11541303-17 2022-04-22T17:30:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:1457: finishing write tid 3 to smithi11541303-17 2022-04-22T17:30:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 695 (ObjNum 432 snap 147 seq_num 432) dirty exists 2022-04-22T17:30:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:1457: left oid 17 (ObjNum 432 snap 147 seq_num 432) 2022-04-22T17:30:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:1456: expect (ObjNum 290 snap 97 seq_num 290) 2022-04-22T17:30:58.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[45495]: cluster 2022-04-22T17:30:56.782698+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e361: 8 total, 7 up, 8 in 2022-04-22T17:30:58.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[45495]: audit 2022-04-22T17:30:57.050608+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[45495]: audit 2022-04-22T17:30:57.676163+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[46934]: cluster 2022-04-22T17:30:56.782698+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e361: 8 total, 7 up, 8 in 2022-04-22T17:30:58.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[46934]: audit 2022-04-22T17:30:57.050608+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:57 smithi115 bash[46934]: audit 2022-04-22T17:30:57.676163+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.194 INFO:tasks.rados.rados.0.smithi115.stdout:1458: expect (ObjNum 390 snap 136 seq_num 390) 2022-04-22T17:30:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:57 smithi121 bash[36902]: cluster 2022-04-22T17:30:56.782698+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e361: 8 total, 7 up, 8 in 2022-04-22T17:30:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:57 smithi121 bash[36902]: audit 2022-04-22T17:30:57.050608+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:57 smithi121 bash[36902]: audit 2022-04-22T17:30:57.676163+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:1456: done (3 left) 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:1457: done (2 left) 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:1458: done (1 left) 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:1459: done (0 left) 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 148 2022-04-22T17:30:58.821 INFO:tasks.rados.rados.0.smithi115.stdout:1460: seq_num 433 ranges {0=57344} 2022-04-22T17:30:58.822 INFO:tasks.rados.rados.0.smithi115.stdout:1460: writing smithi11541303-44 from 0 to 57344 tid 1 2022-04-22T17:30:58.822 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 148 2022-04-22T17:30:58.822 INFO:tasks.rados.rados.0.smithi115.stdout:1461: seq_num 434 ranges {11501568=73728} 2022-04-22T17:30:58.890 INFO:tasks.rados.rados.0.smithi115.stdout:1461: writing smithi11541303-48 from 11501568 to 11575296 tid 1 2022-04-22T17:30:58.970 INFO:tasks.rados.rados.0.smithi115.stdout:1462: delete oid 25 current snap is 148 2022-04-22T17:30:58.970 INFO:tasks.rados.rados.0.smithi115.stdout:1460: finishing write tid 1 to smithi11541303-44 2022-04-22T17:30:58.970 INFO:tasks.rados.rados.0.smithi115.stdout:1460: finishing write tid 2 to smithi11541303-44 2022-04-22T17:30:58.970 INFO:tasks.rados.rados.0.smithi115.stdout:1460: finishing write tid 3 to smithi11541303-44 2022-04-22T17:30:58.970 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 80 (ObjNum 433 snap 148 seq_num 433) dirty exists 2022-04-22T17:30:58.971 INFO:tasks.rados.rados.0.smithi115.stdout:1460: left oid 44 (ObjNum 433 snap 148 seq_num 433) 2022-04-22T17:30:58.972 INFO:tasks.rados.rados.0.smithi115.stdout:1461: finishing write tid 1 to smithi11541303-48 2022-04-22T17:30:58.972 INFO:tasks.rados.rados.0.smithi115.stdout:1461: finishing write tid 2 to smithi11541303-48 2022-04-22T17:30:58.973 INFO:tasks.rados.rados.0.smithi115.stdout:1460: done (2 left) 2022-04-22T17:30:58.973 INFO:tasks.rados.rados.0.smithi115.stdout:1462: done (1 left) 2022-04-22T17:30:58.973 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 148 2022-04-22T17:30:58.973 INFO:tasks.rados.rados.0.smithi115.stdout:1463: seq_num 436 ranges {0=106496} 2022-04-22T17:30:58.973 INFO:tasks.rados.rados.0.smithi115.stdout:1463: writing smithi11541303-22 from 0 to 106496 tid 1 2022-04-22T17:30:58.974 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 148 2022-04-22T17:30:58.974 INFO:tasks.rados.rados.0.smithi115.stdout:1461: finishing write tid 3 to smithi11541303-48 2022-04-22T17:30:58.974 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 851 (ObjNum 434 snap 148 seq_num 434) dirty exists 2022-04-22T17:30:58.974 INFO:tasks.rados.rados.0.smithi115.stdout:1461: left oid 48 (ObjNum 434 snap 148 seq_num 434) 2022-04-22T17:30:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:1464: seq_num 437 ranges {10149888=122880} 2022-04-22T17:30:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:1464: writing smithi11541303-49 from 10149888 to 10272768 tid 1 2022-04-22T17:30:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:1461: done (2 left) 2022-04-22T17:30:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:1465: read oid 39 snap 147 2022-04-22T17:30:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:1465: expect deleted 2022-04-22T17:30:58.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 148 2022-04-22T17:30:58.976 INFO:tasks.rados.rados.0.smithi115.stdout:1466: seq_num 438 ranges {5824512=139264} 2022-04-22T17:30:58.994 INFO:tasks.rados.rados.0.smithi115.stdout:1466: writing smithi11541303-20 from 5824512 to 5963776 tid 1 2022-04-22T17:30:58.994 INFO:tasks.rados.rados.0.smithi115.stdout:1467: delete oid 40 current snap is 148 2022-04-22T17:30:58.994 INFO:tasks.rados.rados.0.smithi115.stdout:1463: finishing write tid 1 to smithi11541303-22 2022-04-22T17:30:58.994 INFO:tasks.rados.rados.0.smithi115.stdout:1463: finishing write tid 2 to smithi11541303-22 2022-04-22T17:30:58.994 INFO:tasks.rados.rados.0.smithi115.stdout:1463: finishing write tid 3 to smithi11541303-22 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 522 (ObjNum 436 snap 148 seq_num 436) dirty exists 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:1463: left oid 22 (ObjNum 436 snap 148 seq_num 436) 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:1464: finishing write tid 1 to smithi11541303-49 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:1464: finishing write tid 2 to smithi11541303-49 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:1464: finishing write tid 3 to smithi11541303-49 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 896 (ObjNum 437 snap 148 seq_num 437) dirty exists 2022-04-22T17:30:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:1464: left oid 49 (ObjNum 437 snap 148 seq_num 437) 2022-04-22T17:30:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:1463: done (4 left) 2022-04-22T17:30:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:1464: done (3 left) 2022-04-22T17:30:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:1465: done (2 left) 2022-04-22T17:30:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:1467: done (1 left) 2022-04-22T17:30:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:1468: setattr oid 13 current snap is 148 2022-04-22T17:30:58.997 INFO:tasks.rados.rados.0.smithi115.stdout:1466: finishing write tid 1 to smithi11541303-20 2022-04-22T17:30:58.998 INFO:tasks.rados.rados.0.smithi115.stdout:1466: finishing write tid 2 to smithi11541303-20 2022-04-22T17:30:58.999 INFO:tasks.rados.rados.0.smithi115.stdout:1469: snap_remove snap 147 2022-04-22T17:30:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:58 smithi121 bash[36902]: cluster 2022-04-22T17:30:57.184280+0000 mgr.y (mgr.34107) 152 : cluster [DBG] pgmap v229: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.8 MiB/s wr, 14 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:58 smithi121 bash[36902]: cluster 2022-04-22T17:30:57.785235+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e362: 8 total, 7 up, 8 in 2022-04-22T17:30:59.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:58 smithi115 bash[45495]: cluster 2022-04-22T17:30:57.184280+0000 mgr.y (mgr.34107) 152 : cluster [DBG] pgmap v229: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.8 MiB/s wr, 14 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:59.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:58 smithi115 bash[45495]: cluster 2022-04-22T17:30:57.785235+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e362: 8 total, 7 up, 8 in 2022-04-22T17:30:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:58 smithi115 bash[46934]: cluster 2022-04-22T17:30:57.184280+0000 mgr.y (mgr.34107) 152 : cluster [DBG] pgmap v229: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 149 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.8 MiB/s wr, 14 op/s; 35/225 objects degraded (15.556%) 2022-04-22T17:30:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:58 smithi115 bash[46934]: cluster 2022-04-22T17:30:57.785235+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e362: 8 total, 7 up, 8 in 2022-04-22T17:30:59.834 INFO:tasks.rados.rados.0.smithi115.stdout:1469: done (2 left) 2022-04-22T17:30:59.834 INFO:tasks.rados.rados.0.smithi115.stdout:1470: setattr oid 15 current snap is 148 2022-04-22T17:30:59.834 INFO:tasks.rados.rados.0.smithi115.stdout:1466: finishing write tid 3 to smithi11541303-20 2022-04-22T17:30:59.835 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 700 (ObjNum 438 snap 148 seq_num 438) dirty exists 2022-04-22T17:30:59.835 INFO:tasks.rados.rados.0.smithi115.stdout:1466: left oid 20 (ObjNum 438 snap 148 seq_num 438) 2022-04-22T17:30:59.836 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 701 (ObjNum -1116843952 snap 22075 seq_num 3178708000) dirty exists 2022-04-22T17:30:59.839 INFO:tasks.rados.rados.0.smithi115.stdout:1466: done (2 left) 2022-04-22T17:30:59.839 INFO:tasks.rados.rados.0.smithi115.stdout:1468: done (1 left) 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:1471: rollback oid 33 current snap is 148 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:1471: done (1 left) 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:1472: read oid 38 snap -1 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:1472: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 148 2022-04-22T17:30:59.840 INFO:tasks.rados.rados.0.smithi115.stdout:1473: seq_num 440 ranges {4325376=172032} 2022-04-22T17:30:59.867 INFO:tasks.rados.rados.0.smithi115.stdout:1473: writing smithi11541303-8 from 4325376 to 4497408 tid 1 2022-04-22T17:30:59.867 INFO:tasks.rados.rados.0.smithi115.stdout:1474: delete oid 39 current snap is 148 2022-04-22T17:30:59.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 233 (ObjNum 427 snap 145 seq_num 427) dirty exists 2022-04-22T17:30:59.867 INFO:tasks.rados.rados.0.smithi115.stdout:1472: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1473: finishing write tid 1 to smithi11541303-8 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1473: finishing write tid 2 to smithi11541303-8 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1473: finishing write tid 3 to smithi11541303-8 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 643 (ObjNum 440 snap 148 seq_num 440) dirty exists 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1473: left oid 8 (ObjNum 440 snap 148 seq_num 440) 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1470: done (3 left) 2022-04-22T17:31:00.019 INFO:tasks.rados.rados.0.smithi115.stdout:1472: done (2 left) 2022-04-22T17:31:00.020 INFO:tasks.rados.rados.0.smithi115.stdout:1473: done (1 left) 2022-04-22T17:31:00.020 INFO:tasks.rados.rados.0.smithi115.stdout:1474: done (0 left) 2022-04-22T17:31:00.020 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 148 2022-04-22T17:31:00.020 INFO:tasks.rados.rados.0.smithi115.stdout:1475: seq_num 442 ranges {5275648=204800} 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:1475: writing smithi11541303-7 from 5275648 to 5480448 tid 1 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:1476: copy_from oid 15 from oid 13 current snap is 148 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:1477: copy_from oid 48 from oid 47 current snap is 148 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:1478: read oid 29 snap -1 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:1478: expect deleted 2022-04-22T17:31:00.053 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 148 2022-04-22T17:31:00.054 INFO:tasks.rados.rados.0.smithi115.stdout:1479: seq_num 443 ranges {2260992=221184} 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1479: writing smithi11541303-2 from 2260992 to 2482176 tid 1 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1480: read oid 44 snap -1 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1480: expect (ObjNum 433 snap 148 seq_num 433) 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1481: rollback oid 21 current snap is 148 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1481: done (6 left) 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 148 2022-04-22T17:31:00.067 INFO:tasks.rados.rados.0.smithi115.stdout:1482: seq_num 444 ranges {106496=237568} 2022-04-22T17:31:00.069 INFO:tasks.rados.rados.0.smithi115.stdout:1482: writing smithi11541303-22 from 106496 to 344064 tid 1 2022-04-22T17:31:00.069 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 148 2022-04-22T17:31:00.069 INFO:tasks.rados.rados.0.smithi115.stdout:1483: seq_num 445 ranges {1712128=253952} 2022-04-22T17:31:00.082 INFO:tasks.rados.rados.0.smithi115.stdout:1483: writing smithi11541303-18 from 1712128 to 1966080 tid 1 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1484: snap_create 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1475: finishing write tid 1 to smithi11541303-7 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1475: finishing write tid 2 to smithi11541303-7 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1476: finishing copy_from to smithi11541303-15 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 701 (ObjNum -1116282176 snap 22075 seq_num 0) dirty exists 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1476: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1475: finishing write tid 3 to smithi11541303-7 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 645 (ObjNum 442 snap 148 seq_num 442) dirty exists 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1475: left oid 7 (ObjNum 442 snap 148 seq_num 442) 2022-04-22T17:31:00.083 INFO:tasks.rados.rados.0.smithi115.stdout:1480: expect (ObjNum 433 snap 148 seq_num 433) 2022-04-22T17:31:00.087 INFO:tasks.rados.rados.0.smithi115.stdout:1479: finishing write tid 1 to smithi11541303-2 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1479: finishing write tid 2 to smithi11541303-2 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1479: finishing write tid 3 to smithi11541303-2 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 649 (ObjNum 443 snap 148 seq_num 443) dirty exists 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1479: left oid 2 (ObjNum 443 snap 148 seq_num 443) 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1482: finishing write tid 1 to smithi11541303-22 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1477: finishing copy_from to smithi11541303-48 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 657 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1482: finishing write tid 2 to smithi11541303-22 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1477: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:31:00.088 INFO:tasks.rados.rados.0.smithi115.stdout:1482: finishing write tid 3 to smithi11541303-22 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 524 (ObjNum 444 snap 148 seq_num 444) dirty exists 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:1482: left oid 22 (ObjNum 444 snap 148 seq_num 444) 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:1483: finishing write tid 1 to smithi11541303-18 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:1483: finishing write tid 2 to smithi11541303-18 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:1483: finishing write tid 3 to smithi11541303-18 2022-04-22T17:31:00.089 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 898 (ObjNum 445 snap 148 seq_num 445) dirty exists 2022-04-22T17:31:00.090 INFO:tasks.rados.rados.0.smithi115.stdout:1483: left oid 18 (ObjNum 445 snap 148 seq_num 445) 2022-04-22T17:31:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:59 smithi121 bash[36902]: cluster 2022-04-22T17:30:58.805018+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e363: 8 total, 7 up, 8 in 2022-04-22T17:31:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:59 smithi121 bash[36902]: audit 2022-04-22T17:30:59.201773+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:30:59 smithi121 bash[36902]: cluster 2022-04-22T17:30:59.766485+0000 mon.a (mon.0) 458 : cluster [WRN] Health check update: Degraded data redundancy: 35/225 objects degraded (15.556%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:00.267 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:30:59 smithi121 bash[21417]: debug 2022-04-22T17:30:59.889+0000 7fcbfeaf2700 -1 osd.6 pg_epoch: 364 pg[5.2s0( v 363'701 (0'0,363'701] local-lis/les=325/326 n=7 ec=63/63 lis/c=325/325 les/c/f=326/327/0 sis=325) [6,0,5]p6(0) r=0 lpr=325 crt=363'701 lcod 363'700 mlcod 363'700 active+clean+snaptrim trimq=[128~1] ps=[118~2,11e~1,121~1,123~1,126~1]] removing snap head 2022-04-22T17:31:00.303 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[51714]: debug 2022-04-22T17:31:00.028+0000 7f69496933c0 -1 osd.1 352 log_to_monitors true 2022-04-22T17:31:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[45495]: cluster 2022-04-22T17:30:58.805018+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e363: 8 total, 7 up, 8 in 2022-04-22T17:31:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[45495]: audit 2022-04-22T17:30:59.201773+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[45495]: cluster 2022-04-22T17:30:59.766485+0000 mon.a (mon.0) 458 : cluster [WRN] Health check update: Degraded data redundancy: 35/225 objects degraded (15.556%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[46934]: cluster 2022-04-22T17:30:58.805018+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e363: 8 total, 7 up, 8 in 2022-04-22T17:31:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[46934]: audit 2022-04-22T17:30:59.201773+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:30:59 smithi115 bash[46934]: cluster 2022-04-22T17:30:59.766485+0000 mon.a (mon.0) 458 : cluster [WRN] Health check update: Degraded data redundancy: 35/225 objects degraded (15.556%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:00.844 INFO:tasks.rados.rados.0.smithi115.stdout:1475: done (8 left) 2022-04-22T17:31:00.844 INFO:tasks.rados.rados.0.smithi115.stdout:1476: done (7 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1477: done (6 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1478: done (5 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1479: done (4 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1480: done (3 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1482: done (2 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1483: done (1 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1484: done (0 left) 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1485: read oid 26 snap -1 2022-04-22T17:31:00.845 INFO:tasks.rados.rados.0.smithi115.stdout:1485: expect deleted 2022-04-22T17:31:00.846 INFO:tasks.rados.rados.0.smithi115.stdout:1486: rollback oid 25 current snap is 149 2022-04-22T17:31:00.846 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 148 2022-04-22T17:31:00.846 INFO:tasks.rados.rados.0.smithi115.stdout:1487: copy_from oid 11 from oid 42 current snap is 149 2022-04-22T17:31:00.846 INFO:tasks.rados.rados.0.smithi115.stdout:1488: delete oid 14 current snap is 149 2022-04-22T17:31:00.850 INFO:tasks.rados.rados.0.smithi115.stdout:1488: done (3 left) 2022-04-22T17:31:00.850 INFO:tasks.rados.rados.0.smithi115.stdout:1489: rollback oid 37 current snap is 149 2022-04-22T17:31:00.850 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 148 2022-04-22T17:31:00.850 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 149 2022-04-22T17:31:00.850 INFO:tasks.rados.rados.0.smithi115.stdout:1490: seq_num 447 ranges {0=294912} 2022-04-22T17:31:00.852 INFO:tasks.rados.rados.0.smithi115.stdout:1490: writing smithi11541303-30 from 0 to 294912 tid 1 2022-04-22T17:31:00.852 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 149 2022-04-22T17:31:00.852 INFO:tasks.rados.rados.0.smithi115.stdout:1491: seq_num 448 ranges {0=311296} 2022-04-22T17:31:00.854 INFO:tasks.rados.rados.0.smithi115.stdout:1491: writing smithi11541303-9 from 0 to 311296 tid 1 2022-04-22T17:31:00.854 INFO:tasks.rados.rados.0.smithi115.stdout:1492: read oid 27 snap -1 2022-04-22T17:31:00.854 INFO:tasks.rados.rados.0.smithi115.stdout:1492: expect deleted 2022-04-22T17:31:00.854 INFO:tasks.rados.rados.0.smithi115.stdout:1493: delete oid 41 current snap is 149 2022-04-22T17:31:00.855 INFO:tasks.rados.rados.0.smithi115.stdout:1486: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:31:00.855 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:31:00.860 INFO:tasks.rados.rados.0.smithi115.stdout:1489: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1485: done (7 left) 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1486: done (6 left) 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1493: done (5 left) 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1494: copy_from oid 32 from oid 14 current snap is 149 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1495: rmattr oid 19 current snap is 149 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:1495: done (6 left) 2022-04-22T17:31:00.862 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 149 2022-04-22T17:31:00.863 INFO:tasks.rados.rados.0.smithi115.stdout:1496: seq_num 450 ranges {1417216=344064} 2022-04-22T17:31:00.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[45495]: cluster 2022-04-22T17:30:59.185697+0000 mgr.y (mgr.34107) 153 : cluster [DBG] pgmap v232: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 946 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:00.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[45495]: cluster 2022-04-22T17:30:59.818473+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e364: 8 total, 7 up, 8 in 2022-04-22T17:31:00.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[45495]: audit 2022-04-22T17:31:00.039226+0000 mon.c (mon.1) 10 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:00.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[45495]: audit 2022-04-22T17:31:00.039902+0000 mon.a (mon.0) 460 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[46934]: cluster 2022-04-22T17:30:59.185697+0000 mgr.y (mgr.34107) 153 : cluster [DBG] pgmap v232: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 946 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[46934]: cluster 2022-04-22T17:30:59.818473+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e364: 8 total, 7 up, 8 in 2022-04-22T17:31:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[46934]: audit 2022-04-22T17:31:00.039226+0000 mon.c (mon.1) 10 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[46934]: audit 2022-04-22T17:31:00.039902+0000 mon.a (mon.0) 460 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:00.873 INFO:tasks.rados.rados.0.smithi115.stdout:1496: writing smithi11541303-17 from 1417216 to 1761280 tid 1 2022-04-22T17:31:00.873 INFO:tasks.rados.rados.0.smithi115.stdout:1497: rmattr oid 20 current snap is 149 2022-04-22T17:31:00.873 INFO:tasks.rados.rados.0.smithi115.stdout:1497: done (7 left) 2022-04-22T17:31:00.873 INFO:tasks.rados.rados.0.smithi115.stdout:1498: read oid 28 snap -1 2022-04-22T17:31:00.873 INFO:tasks.rados.rados.0.smithi115.stdout:1498: expect deleted 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1499: snap_create 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1489: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1489: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 652 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1490: finishing write tid 1 to smithi11541303-30 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1490: finishing write tid 2 to smithi11541303-30 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1490: finishing write tid 3 to smithi11541303-30 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 854 (ObjNum 447 snap 149 seq_num 447) dirty exists 2022-04-22T17:31:00.874 INFO:tasks.rados.rados.0.smithi115.stdout:1490: left oid 30 (ObjNum 447 snap 149 seq_num 447) 2022-04-22T17:31:00.875 INFO:tasks.rados.rados.0.smithi115.stdout:1494: finishing copy_from to smithi11541303-32 2022-04-22T17:31:00.875 INFO:tasks.rados.rados.0.smithi115.stdout:1494: got expected ENOENT (src dne) 2022-04-22T17:31:00.875 INFO:tasks.rados.rados.0.smithi115.stdout:1494: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:31:00.875 INFO:tasks.rados.rados.0.smithi115.stdout:1491: finishing write tid 1 to smithi11541303-9 2022-04-22T17:31:00.875 INFO:tasks.rados.rados.0.smithi115.stdout:1491: finishing write tid 2 to smithi11541303-9 2022-04-22T17:31:00.879 INFO:tasks.rados.rados.0.smithi115.stdout:1491: finishing write tid 3 to smithi11541303-9 2022-04-22T17:31:00.879 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 900 (ObjNum 448 snap 149 seq_num 448) dirty exists 2022-04-22T17:31:00.879 INFO:tasks.rados.rados.0.smithi115.stdout:1491: left oid 9 (ObjNum 448 snap 149 seq_num 448) 2022-04-22T17:31:00.880 INFO:tasks.rados.rados.0.smithi115.stdout:1496: finishing write tid 1 to smithi11541303-17 2022-04-22T17:31:00.880 INFO:tasks.rados.rados.0.smithi115.stdout:1496: finishing write tid 2 to smithi11541303-17 2022-04-22T17:31:00.882 INFO:tasks.rados.rados.0.smithi115.stdout:1496: finishing write tid 3 to smithi11541303-17 2022-04-22T17:31:00.882 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 708 (ObjNum 450 snap 149 seq_num 450) dirty exists 2022-04-22T17:31:00.883 INFO:tasks.rados.rados.0.smithi115.stdout:1496: left oid 17 (ObjNum 450 snap 149 seq_num 450) 2022-04-22T17:31:00.924 INFO:tasks.rados.rados.0.smithi115.stdout:1487: finishing copy_from to smithi11541303-11 2022-04-22T17:31:00.924 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 871 (ObjNum 414 snap 141 seq_num 414) dirty exists 2022-04-22T17:31:00.925 INFO:tasks.rados.rados.0.smithi115.stdout:1487: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:31:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:00 smithi121 bash[36902]: cluster 2022-04-22T17:30:59.185697+0000 mgr.y (mgr.34107) 153 : cluster [DBG] pgmap v232: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 946 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:00 smithi121 bash[36902]: cluster 2022-04-22T17:30:59.818473+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e364: 8 total, 7 up, 8 in 2022-04-22T17:31:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:00 smithi121 bash[36902]: audit 2022-04-22T17:31:00.039226+0000 mon.c (mon.1) 10 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:00 smithi121 bash[36902]: audit 2022-04-22T17:31:00.039902+0000 mon.a (mon.0) 460 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-04-22T17:31:01.303 INFO:journalctl@ceph.osd.1.smithi115.stdout:Apr 22 17:31:00 smithi115 bash[51714]: debug 2022-04-22T17:31:00.848+0000 7f693aa2e700 -1 osd.1 352 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1487: done (8 left) 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1489: done (7 left) 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1490: done (6 left) 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1491: done (5 left) 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1492: done (4 left) 2022-04-22T17:31:01.859 INFO:tasks.rados.rados.0.smithi115.stdout:1494: done (3 left) 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:1496: done (2 left) 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:1498: done (1 left) 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:1499: done (0 left) 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:1500: rollback oid 45 current snap is 150 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 148 2022-04-22T17:31:01.860 INFO:tasks.rados.rados.0.smithi115.stdout:1501: snap_remove snap 149 2022-04-22T17:31:02.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:01.622Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:02.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:01.622Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:02.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:01.622Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[45495]: audit 2022-04-22T17:31:00.827361+0000 mon.a (mon.0) 461 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[45495]: cluster 2022-04-22T17:31:00.827537+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e365: 8 total, 7 up, 8 in 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[45495]: audit 2022-04-22T17:31:00.828548+0000 mon.c (mon.1) 11 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[45495]: audit 2022-04-22T17:31:00.829030+0000 mon.a (mon.0) 463 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[46934]: audit 2022-04-22T17:31:00.827361+0000 mon.a (mon.0) 461 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:31:02.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[46934]: cluster 2022-04-22T17:31:00.827537+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e365: 8 total, 7 up, 8 in 2022-04-22T17:31:02.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[46934]: audit 2022-04-22T17:31:00.828548+0000 mon.c (mon.1) 11 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:01 smithi115 bash[46934]: audit 2022-04-22T17:31:00.829030+0000 mon.a (mon.0) 463 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:01 smithi121 bash[36902]: audit 2022-04-22T17:31:00.827361+0000 mon.a (mon.0) 461 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-04-22T17:31:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:01 smithi121 bash[36902]: cluster 2022-04-22T17:31:00.827537+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e365: 8 total, 7 up, 8 in 2022-04-22T17:31:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:01 smithi121 bash[36902]: audit 2022-04-22T17:31:00.828548+0000 mon.c (mon.1) 11 : audit [INF] from='osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:01 smithi121 bash[36902]: audit 2022-04-22T17:31:00.829030+0000 mon.a (mon.0) 463 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1501: done (1 left) 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1502: read oid 31 snap 148 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1502: expect (ObjNum 399 snap 137 seq_num 399) 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1503: read oid 7 snap 148 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1503: expect (ObjNum 442 snap 148 seq_num 442) 2022-04-22T17:31:02.861 INFO:tasks.rados.rados.0.smithi115.stdout:1504: read oid 6 snap -1 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1504: expect (ObjNum 417 snap 141 seq_num 417) 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1505: rollback oid 24 current snap is 150 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 148 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1506: rollback oid 1 current snap is 150 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 1 to 148 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1507: read oid 21 snap -1 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1507: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:31:02.862 INFO:tasks.rados.rados.0.smithi115.stdout:1508: snap_create 2022-04-22T17:31:02.865 INFO:tasks.rados.rados.0.smithi115.stdout:1506: finishing rollback tid 1 to smithi11541303-1 2022-04-22T17:31:02.865 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 0 (ObjNum -1124694032 snap 22075 seq_num 1) dirty dne 2022-04-22T17:31:02.906 INFO:tasks.rados.rados.0.smithi115.stdout:1503: expect (ObjNum 442 snap 148 seq_num 442) 2022-04-22T17:31:03.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[46934]: cluster 2022-04-22T17:31:01.186655+0000 mgr.y (mgr.34107) 154 : cluster [DBG] pgmap v235: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 945 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:03.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[46934]: cluster 2022-04-22T17:31:01.823232+0000 mon.a (mon.0) 464 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:03.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[46934]: cluster 2022-04-22T17:31:01.841810+0000 mon.a (mon.0) 465 : cluster [INF] osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477] boot 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[46934]: cluster 2022-04-22T17:31:01.841893+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[46934]: audit 2022-04-22T17:31:01.842267+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[45495]: cluster 2022-04-22T17:31:01.186655+0000 mgr.y (mgr.34107) 154 : cluster [DBG] pgmap v235: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 945 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[45495]: cluster 2022-04-22T17:31:01.823232+0000 mon.a (mon.0) 464 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[45495]: cluster 2022-04-22T17:31:01.841810+0000 mon.a (mon.0) 465 : cluster [INF] osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477] boot 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[45495]: cluster 2022-04-22T17:31:01.841893+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2022-04-22T17:31:03.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:02 smithi115 bash[45495]: audit 2022-04-22T17:31:01.842267+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:31:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:02 smithi121 bash[36902]: cluster 2022-04-22T17:31:01.186655+0000 mgr.y (mgr.34107) 154 : cluster [DBG] pgmap v235: 121 pgs: 11 active+undersized+degraded, 33 active+undersized, 77 active+clean; 143 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 945 KiB/s wr, 14 op/s; 37/237 objects degraded (15.612%) 2022-04-22T17:31:03.351 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:02 smithi121 bash[36902]: cluster 2022-04-22T17:31:01.823232+0000 mon.a (mon.0) 464 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:03.354 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:02 smithi121 bash[36902]: cluster 2022-04-22T17:31:01.841810+0000 mon.a (mon.0) 465 : cluster [INF] osd.1 [v2:172.21.15.115:6810/1073133477,v1:172.21.15.115:6811/1073133477] boot 2022-04-22T17:31:03.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:02 smithi121 bash[36902]: cluster 2022-04-22T17:31:01.841893+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2022-04-22T17:31:03.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:02 smithi121 bash[36902]: audit 2022-04-22T17:31:01.842267+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-04-22T17:31:03.356 INFO:tasks.rados.rados.0.smithi115.stdout:1505: finishing rollback tid 0 to smithi11541303-24 2022-04-22T17:31:03.356 INFO:tasks.rados.rados.0.smithi115.stdout:1505: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:31:03.356 INFO:tasks.rados.rados.0.smithi115.stdout:1507: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:31:03.356 INFO:tasks.rados.rados.0.smithi115.stdout:1502: expect (ObjNum 399 snap 137 seq_num 399) 2022-04-22T17:31:03.396 INFO:tasks.rados.rados.0.smithi115.stdout:1504: expect (ObjNum 417 snap 141 seq_num 417) 2022-04-22T17:31:03.534 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:03.534 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:03.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:03.534 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:03.517Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:03.609 INFO:tasks.rados.rados.0.smithi115.stdout:1505: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:31:03.610 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 646 (ObjNum 290 snap 97 seq_num 290) dirty exists 2022-04-22T17:31:03.610 INFO:tasks.rados.rados.0.smithi115.stdout:1500: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:31:03.610 INFO:tasks.rados.rados.0.smithi115.stdout:1500: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:31:03.610 INFO:tasks.rados.rados.0.smithi115.stdout:1500: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:31:03.610 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 858 (ObjNum 429 snap 146 seq_num 429) dirty exists 2022-04-22T17:31:03.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:31:03 smithi121 bash[24219]: debug 2022-04-22T17:31:03.437+0000 7f1df544c700 -1 osd.7 pg_epoch: 367 pg[5.9s0( v 367'860 (0'0,367'860] local-lis/les=366/367 n=8 ec=63/63 lis/c=366/366 les/c/f=367/367/0 sis=366) [7,6,1]p7(0) r=0 lpr=366 luod=367'858 crt=367'858 lcod 367'857 mlcod 360'848 active+clean+snaptrim trimq=[118~2,11e~1,121~1,123~1,126~1,128~1,12b~1]] removing snap head 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1500: done (7 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1502: done (6 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1503: done (5 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1504: done (4 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1505: done (3 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1506: done (2 left) 2022-04-22T17:31:03.856 INFO:tasks.rados.rados.0.smithi115.stdout:1507: done (1 left) 2022-04-22T17:31:03.857 INFO:tasks.rados.rados.0.smithi115.stdout:1508: done (0 left) 2022-04-22T17:31:03.857 INFO:tasks.rados.rados.0.smithi115.stdout:1509: snap_create 2022-04-22T17:31:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:03 smithi121 bash[36902]: cluster 2022-04-22T17:31:02.842587+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2022-04-22T17:31:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:03 smithi121 bash[36902]: cluster 2022-04-22T17:31:03.840036+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2022-04-22T17:31:04.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[45495]: cluster 2022-04-22T17:31:02.842587+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2022-04-22T17:31:04.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[45495]: cluster 2022-04-22T17:31:03.840036+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2022-04-22T17:31:04.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[46934]: cluster 2022-04-22T17:31:02.842587+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2022-04-22T17:31:04.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:03 smithi115 bash[46934]: cluster 2022-04-22T17:31:03.840036+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2022-04-22T17:31:04.860 INFO:tasks.rados.rados.0.smithi115.stdout:1509: done (0 left) 2022-04-22T17:31:04.860 INFO:tasks.rados.rados.0.smithi115.stdout:1510: copy_from oid 10 from oid 15 current snap is 152 2022-04-22T17:31:04.860 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 152 2022-04-22T17:31:04.860 INFO:tasks.rados.rados.0.smithi115.stdout:1511: seq_num 451 ranges {2891776=360448} 2022-04-22T17:31:04.881 INFO:tasks.rados.rados.0.smithi115.stdout:1511: writing smithi11541303-35 from 2891776 to 3252224 tid 1 2022-04-22T17:31:04.881 INFO:tasks.rados.rados.0.smithi115.stdout:1512: copy_from oid 6 from oid 14 current snap is 152 2022-04-22T17:31:04.881 INFO:tasks.rados.rados.0.smithi115.stdout:1510: finishing copy_from to smithi11541303-10 2022-04-22T17:31:04.881 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 701 (ObjNum 1 snap 32692 seq_num 3178823920) dirty exists 2022-04-22T17:31:04.881 INFO:tasks.rados.rados.0.smithi115.stdout:1510: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:31:04.882 INFO:tasks.rados.rados.0.smithi115.stdout:1510: done (2 left) 2022-04-22T17:31:04.882 INFO:tasks.rados.rados.0.smithi115.stdout:1513: snap_create 2022-04-22T17:31:04.888 INFO:tasks.rados.rados.0.smithi115.stdout:1512: finishing copy_from to smithi11541303-6 2022-04-22T17:31:04.888 INFO:tasks.rados.rados.0.smithi115.stdout:1512: got expected ENOENT (src dne) 2022-04-22T17:31:04.888 INFO:tasks.rados.rados.0.smithi115.stdout:1511: finishing write tid 1 to smithi11541303-35 2022-04-22T17:31:04.888 INFO:tasks.rados.rados.0.smithi115.stdout:1512: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:31:04.888 INFO:tasks.rados.rados.0.smithi115.stdout:1511: finishing write tid 2 to smithi11541303-35 2022-04-22T17:31:04.906 INFO:tasks.rados.rados.0.smithi115.stdout:1511: finishing write tid 3 to smithi11541303-35 2022-04-22T17:31:04.906 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 654 (ObjNum 451 snap 152 seq_num 451) dirty exists 2022-04-22T17:31:04.906 INFO:tasks.rados.rados.0.smithi115.stdout:1511: left oid 35 (ObjNum 451 snap 152 seq_num 451) 2022-04-22T17:31:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:04 smithi121 bash[36902]: cluster 2022-04-22T17:31:03.187235+0000 mgr.y (mgr.34107) 155 : cluster [DBG] pgmap v238: 121 pgs: 10 active+undersized+degraded, 1 active+recovery_wait+degraded, 28 active+undersized, 82 active+clean; 144 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 84 KiB/s wr, 2 op/s; 37/240 objects degraded (15.417%) 2022-04-22T17:31:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:04 smithi121 bash[36902]: audit 2022-04-22T17:31:04.204358+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:04 smithi121 bash[36902]: cluster 2022-04-22T17:31:04.767992+0000 mon.a (mon.0) 471 : cluster [WRN] Health check update: Degraded data redundancy: 37/240 objects degraded (15.417%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:04 smithi121 bash[36902]: cluster 2022-04-22T17:31:04.843253+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[45495]: cluster 2022-04-22T17:31:03.187235+0000 mgr.y (mgr.34107) 155 : cluster [DBG] pgmap v238: 121 pgs: 10 active+undersized+degraded, 1 active+recovery_wait+degraded, 28 active+undersized, 82 active+clean; 144 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 84 KiB/s wr, 2 op/s; 37/240 objects degraded (15.417%) 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[45495]: audit 2022-04-22T17:31:04.204358+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[45495]: cluster 2022-04-22T17:31:04.767992+0000 mon.a (mon.0) 471 : cluster [WRN] Health check update: Degraded data redundancy: 37/240 objects degraded (15.417%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[45495]: cluster 2022-04-22T17:31:04.843253+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[46934]: cluster 2022-04-22T17:31:03.187235+0000 mgr.y (mgr.34107) 155 : cluster [DBG] pgmap v238: 121 pgs: 10 active+undersized+degraded, 1 active+recovery_wait+degraded, 28 active+undersized, 82 active+clean; 144 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 84 KiB/s wr, 2 op/s; 37/240 objects degraded (15.417%) 2022-04-22T17:31:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[46934]: audit 2022-04-22T17:31:04.204358+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[46934]: cluster 2022-04-22T17:31:04.767992+0000 mon.a (mon.0) 471 : cluster [WRN] Health check update: Degraded data redundancy: 37/240 objects degraded (15.417%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:04 smithi115 bash[46934]: cluster 2022-04-22T17:31:04.843253+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2022-04-22T17:31:05.887 INFO:tasks.rados.rados.0.smithi115.stdout:1511: done (2 left) 2022-04-22T17:31:05.887 INFO:tasks.rados.rados.0.smithi115.stdout:1512: done (1 left) 2022-04-22T17:31:05.887 INFO:tasks.rados.rados.0.smithi115.stdout:1513: done (0 left) 2022-04-22T17:31:05.887 INFO:tasks.rados.rados.0.smithi115.stdout:1514: delete oid 36 current snap is 153 2022-04-22T17:31:05.895 INFO:tasks.rados.rados.0.smithi115.stdout:1514: done (0 left) 2022-04-22T17:31:05.895 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 153 2022-04-22T17:31:05.895 INFO:tasks.rados.rados.0.smithi115.stdout:1515: seq_num 453 ranges {0=393216} 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1515: writing smithi11541303-4 from 0 to 393216 tid 1 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1516: rollback oid 39 current snap is 153 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 152 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1517: read oid 48 snap 148 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1517: expect (ObjNum 426 snap 144 seq_num 426) 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1518: snap_create 2022-04-22T17:31:05.899 INFO:tasks.rados.rados.0.smithi115.stdout:1516: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:31:05.900 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 0 (ObjNum -1116748816 snap 22075 seq_num 858862645) dirty dne 2022-04-22T17:31:05.904 INFO:tasks.rados.rados.0.smithi115.stdout:1515: finishing write tid 1 to smithi11541303-4 2022-04-22T17:31:05.905 INFO:tasks.rados.rados.0.smithi115.stdout:1515: finishing write tid 2 to smithi11541303-4 2022-04-22T17:31:05.916 INFO:tasks.rados.rados.0.smithi115.stdout:1515: finishing write tid 3 to smithi11541303-4 2022-04-22T17:31:05.916 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 868 (ObjNum 453 snap 153 seq_num 453) dirty exists 2022-04-22T17:31:05.916 INFO:tasks.rados.rados.0.smithi115.stdout:1515: left oid 4 (ObjNum 453 snap 153 seq_num 453) 2022-04-22T17:31:05.918 INFO:tasks.rados.rados.0.smithi115.stdout:1517: expect (ObjNum 426 snap 144 seq_num 426) 2022-04-22T17:31:06.898 INFO:tasks.rados.rados.0.smithi115.stdout:1515: done (3 left) 2022-04-22T17:31:06.898 INFO:tasks.rados.rados.0.smithi115.stdout:1516: done (2 left) 2022-04-22T17:31:06.899 INFO:tasks.rados.rados.0.smithi115.stdout:1517: done (1 left) 2022-04-22T17:31:06.899 INFO:tasks.rados.rados.0.smithi115.stdout:1518: done (0 left) 2022-04-22T17:31:06.899 INFO:tasks.rados.rados.0.smithi115.stdout:1519: rmattr oid 13 current snap is 154 2022-04-22T17:31:06.899 INFO:tasks.rados.rados.0.smithi115.stdout:1520: snap_remove snap 150 2022-04-22T17:31:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: cluster 2022-04-22T17:31:05.188095+0000 mgr.y (mgr.34107) 156 : cluster [DBG] pgmap v241: 121 pgs: 1 active+recovering+degraded, 4 active+undersized+degraded, 1 active+recovery_wait+degraded, 13 active+undersized, 102 active+clean; 139 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 8/222 objects degraded (3.604%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: cluster 2022-04-22T17:31:05.871007+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.613156+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.619697+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.628280+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.630771+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.632212+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.633452+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:06 smithi121 bash[36902]: audit 2022-04-22T17:31:06.635113+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: cluster 2022-04-22T17:31:05.188095+0000 mgr.y (mgr.34107) 156 : cluster [DBG] pgmap v241: 121 pgs: 1 active+recovering+degraded, 4 active+undersized+degraded, 1 active+recovery_wait+degraded, 13 active+undersized, 102 active+clean; 139 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 8/222 objects degraded (3.604%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: cluster 2022-04-22T17:31:05.871007+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2022-04-22T17:31:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.613156+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.619697+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.628280+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.630771+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.632212+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.633452+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[45495]: audit 2022-04-22T17:31:06.635113+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: cluster 2022-04-22T17:31:05.188095+0000 mgr.y (mgr.34107) 156 : cluster [DBG] pgmap v241: 121 pgs: 1 active+recovering+degraded, 4 active+undersized+degraded, 1 active+recovery_wait+degraded, 13 active+undersized, 102 active+clean; 139 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 8/222 objects degraded (3.604%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: cluster 2022-04-22T17:31:05.871007+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2022-04-22T17:31:07.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.613156+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.619697+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.628280+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.630771+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.632212+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.633452+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:07.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:06 smithi115 bash[46934]: audit 2022-04-22T17:31:06.635113+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:07.908 INFO:tasks.rados.rados.0.smithi115.stdout:1520: done (1 left) 2022-04-22T17:31:07.909 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 154 2022-04-22T17:31:07.909 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 710 (ObjNum 1566667879 snap 1902141803 seq_num 1651007085) dirty exists 2022-04-22T17:31:07.909 INFO:tasks.rados.rados.0.smithi115.stdout:1521: seq_num 454 ranges {0=409600} 2022-04-22T17:31:07.911 INFO:tasks.rados.rados.0.smithi115.stdout:1521: writing smithi11541303-25 from 0 to 409600 tid 1 2022-04-22T17:31:07.911 INFO:tasks.rados.rados.0.smithi115.stdout:1519: done (1 left) 2022-04-22T17:31:07.911 INFO:tasks.rados.rados.0.smithi115.stdout:1522: snap_create 2022-04-22T17:31:07.917 INFO:tasks.rados.rados.0.smithi115.stdout:1521: finishing write tid 1 to smithi11541303-25 2022-04-22T17:31:07.920 INFO:tasks.rados.rados.0.smithi115.stdout:1521: finishing write tid 2 to smithi11541303-25 2022-04-22T17:31:07.922 INFO:tasks.rados.rados.0.smithi115.stdout:1521: finishing write tid 3 to smithi11541303-25 2022-04-22T17:31:07.922 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 712 (ObjNum 454 snap 154 seq_num 454) dirty exists 2022-04-22T17:31:07.922 INFO:tasks.rados.rados.0.smithi115.stdout:1521: left oid 25 (ObjNum 454 snap 154 seq_num 454) 2022-04-22T17:31:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:07 smithi121 bash[36902]: cephadm 2022-04-22T17:31:06.605878+0000 mgr.y (mgr.34107) 157 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:08.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:07 smithi121 bash[36902]: audit 2022-04-22T17:31:06.635351+0000 mgr.y (mgr.34107) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:08.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:07 smithi121 bash[36902]: cephadm 2022-04-22T17:31:06.636738+0000 mgr.y (mgr.34107) 159 : cephadm [INF] Upgrade: osd.2 is safe to restart 2022-04-22T17:31:08.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:07 smithi121 bash[36902]: cluster 2022-04-22T17:31:06.882523+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2022-04-22T17:31:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[46934]: cephadm 2022-04-22T17:31:06.605878+0000 mgr.y (mgr.34107) 157 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[46934]: audit 2022-04-22T17:31:06.635351+0000 mgr.y (mgr.34107) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[46934]: cephadm 2022-04-22T17:31:06.636738+0000 mgr.y (mgr.34107) 159 : cephadm [INF] Upgrade: osd.2 is safe to restart 2022-04-22T17:31:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[46934]: cluster 2022-04-22T17:31:06.882523+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2022-04-22T17:31:08.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[45495]: cephadm 2022-04-22T17:31:06.605878+0000 mgr.y (mgr.34107) 157 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:08.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[45495]: audit 2022-04-22T17:31:06.635351+0000 mgr.y (mgr.34107) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2022-04-22T17:31:08.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[45495]: cephadm 2022-04-22T17:31:06.636738+0000 mgr.y (mgr.34107) 159 : cephadm [INF] Upgrade: osd.2 is safe to restart 2022-04-22T17:31:08.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:07 smithi115 bash[45495]: cluster 2022-04-22T17:31:06.882523+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1521: done (1 left) 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1522: done (0 left) 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1523: read oid 27 snap -1 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1523: expect deleted 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1524: rmattr oid 22 current snap is 155 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1524: done (1 left) 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1525: read oid 21 snap 154 2022-04-22T17:31:08.912 INFO:tasks.rados.rados.0.smithi115.stdout:1525: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:31:08.913 INFO:tasks.rados.rados.0.smithi115.stdout:1526: snap_create 2022-04-22T17:31:08.929 INFO:tasks.rados.rados.0.smithi115.stdout:1525: expect (ObjNum 277 snap 92 seq_num 277) 2022-04-22T17:31:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:08 smithi121 bash[36902]: cluster 2022-04-22T17:31:07.189094+0000 mgr.y (mgr.34107) 160 : cluster [DBG] pgmap v244: 121 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 119 active+clean; 139 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 17 op/s; 2/225 objects degraded (0.889%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:08 smithi121 bash[36902]: cluster 2022-04-22T17:31:07.892226+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2022-04-22T17:31:09.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:08 smithi115 bash[45495]: cluster 2022-04-22T17:31:07.189094+0000 mgr.y (mgr.34107) 160 : cluster [DBG] pgmap v244: 121 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 119 active+clean; 139 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 17 op/s; 2/225 objects degraded (0.889%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:09.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:08 smithi115 bash[45495]: cluster 2022-04-22T17:31:07.892226+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2022-04-22T17:31:09.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:08 smithi115 bash[46934]: cluster 2022-04-22T17:31:07.189094+0000 mgr.y (mgr.34107) 160 : cluster [DBG] pgmap v244: 121 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 119 active+clean; 139 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 17 op/s; 2/225 objects degraded (0.889%); 15 MiB/s, 4 objects/s recovering 2022-04-22T17:31:09.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:08 smithi115 bash[46934]: cluster 2022-04-22T17:31:07.892226+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2022-04-22T17:31:09.917 INFO:tasks.rados.rados.0.smithi115.stdout:1523: done (2 left) 2022-04-22T17:31:09.917 INFO:tasks.rados.rados.0.smithi115.stdout:1525: done (1 left) 2022-04-22T17:31:09.918 INFO:tasks.rados.rados.0.smithi115.stdout:1526: done (0 left) 2022-04-22T17:31:09.918 INFO:tasks.rados.rados.0.smithi115.stdout:1527: setattr oid 18 current snap is 156 2022-04-22T17:31:09.919 INFO:tasks.rados.rados.0.smithi115.stdout:1528: setattr oid 47 current snap is 156 2022-04-22T17:31:09.921 INFO:tasks.rados.rados.0.smithi115.stdout:1529: delete oid 41 current snap is 156 2022-04-22T17:31:09.933 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 702 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:31:09.934 INFO:tasks.rados.rados.0.smithi115.stdout:1528: done (2 left) 2022-04-22T17:31:09.935 INFO:tasks.rados.rados.0.smithi115.stdout:1529: done (1 left) 2022-04-22T17:31:09.935 INFO:tasks.rados.rados.0.smithi115.stdout:1530: snap_create 2022-04-22T17:31:09.936 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 910 (ObjNum 445 snap 148 seq_num 445) dirty exists 2022-04-22T17:31:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: cluster 2022-04-22T17:31:08.895508+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2022-04-22T17:31:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: cephadm 2022-04-22T17:31:09.046384+0000 mgr.y (mgr.34107) 161 : cephadm [INF] Upgrade: Updating osd.2 2022-04-22T17:31:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: audit 2022-04-22T17:31:09.055277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: cephadm 2022-04-22T17:31:09.058394+0000 mgr.y (mgr.34107) 162 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:31:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: audit 2022-04-22T17:31:09.207018+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: cluster 2022-04-22T17:31:09.769602+0000 mon.a (mon.0) 486 : cluster [WRN] Health check update: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:09 smithi121 bash[36902]: cluster 2022-04-22T17:31:09.900234+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2022-04-22T17:31:10.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: cluster 2022-04-22T17:31:08.895508+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2022-04-22T17:31:10.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: cephadm 2022-04-22T17:31:09.046384+0000 mgr.y (mgr.34107) 161 : cephadm [INF] Upgrade: Updating osd.2 2022-04-22T17:31:10.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: audit 2022-04-22T17:31:09.055277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:10.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: cephadm 2022-04-22T17:31:09.058394+0000 mgr.y (mgr.34107) 162 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:31:10.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: audit 2022-04-22T17:31:09.207018+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:10.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: cluster 2022-04-22T17:31:09.769602+0000 mon.a (mon.0) 486 : cluster [WRN] Health check update: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:10.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[45495]: cluster 2022-04-22T17:31:09.900234+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2022-04-22T17:31:10.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: cluster 2022-04-22T17:31:08.895508+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2022-04-22T17:31:10.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: cephadm 2022-04-22T17:31:09.046384+0000 mgr.y (mgr.34107) 161 : cephadm [INF] Upgrade: Updating osd.2 2022-04-22T17:31:10.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: audit 2022-04-22T17:31:09.055277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:10.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: cephadm 2022-04-22T17:31:09.058394+0000 mgr.y (mgr.34107) 162 : cephadm [INF] Deploying daemon osd.2 on smithi115 2022-04-22T17:31:10.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: audit 2022-04-22T17:31:09.207018+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:10.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: cluster 2022-04-22T17:31:09.769602+0000 mon.a (mon.0) 486 : cluster [WRN] Health check update: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:10.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:09 smithi115 bash[46934]: cluster 2022-04-22T17:31:09.900234+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2022-04-22T17:31:10.942 INFO:tasks.rados.rados.0.smithi115.stdout:1527: done (1 left) 2022-04-22T17:31:10.942 INFO:tasks.rados.rados.0.smithi115.stdout:1530: done (0 left) 2022-04-22T17:31:10.943 INFO:tasks.rados.rados.0.smithi115.stdout:1531: copy_from oid 49 from oid 48 current snap is 157 2022-04-22T17:31:10.943 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 157 2022-04-22T17:31:10.943 INFO:tasks.rados.rados.0.smithi115.stdout:1532: seq_num 456 ranges {0=442368} 2022-04-22T17:31:10.946 INFO:tasks.rados.rados.0.smithi115.stdout:1532: writing smithi11541303-15 from 0 to 442368 tid 1 2022-04-22T17:31:10.946 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 157 2022-04-22T17:31:10.946 INFO:tasks.rados.rados.0.smithi115.stdout:1533: seq_num 457 ranges {1761280=458752} 2022-04-22T17:31:10.961 INFO:tasks.rados.rados.0.smithi115.stdout:1533: writing smithi11541303-17 from 1761280 to 2220032 tid 1 2022-04-22T17:31:10.961 INFO:tasks.rados.rados.0.smithi115.stdout:1534: rollback oid 28 current snap is 157 2022-04-22T17:31:10.961 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 153 2022-04-22T17:31:10.961 INFO:tasks.rados.rados.0.smithi115.stdout:1532: finishing write tid 1 to smithi11541303-15 2022-04-22T17:31:10.962 INFO:tasks.rados.rados.0.smithi115.stdout:1532: finishing write tid 2 to smithi11541303-15 2022-04-22T17:31:10.962 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 157 2022-04-22T17:31:10.962 INFO:tasks.rados.rados.0.smithi115.stdout:1535: seq_num 458 ranges {0=475136} 2022-04-22T17:31:10.964 INFO:tasks.rados.rados.0.smithi115.stdout:1535: writing smithi11541303-10 from 0 to 475136 tid 1 2022-04-22T17:31:10.964 INFO:tasks.rados.rados.0.smithi115.stdout:1536: delete oid 50 current snap is 157 2022-04-22T17:31:10.964 INFO:tasks.rados.rados.0.smithi115.stdout:1532: finishing write tid 3 to smithi11541303-15 2022-04-22T17:31:10.964 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 703 (ObjNum 456 snap 157 seq_num 456) dirty exists 2022-04-22T17:31:10.964 INFO:tasks.rados.rados.0.smithi115.stdout:1532: left oid 15 (ObjNum 456 snap 157 seq_num 456) 2022-04-22T17:31:10.967 INFO:tasks.rados.rados.0.smithi115.stdout:1534: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:31:10.967 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 0 (ObjNum -1124694032 snap 22075 seq_num 1) dirty dne 2022-04-22T17:31:10.968 INFO:tasks.rados.rados.0.smithi115.stdout:1531: finishing copy_from to smithi11541303-49 2022-04-22T17:31:10.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 657 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:31:10.969 INFO:tasks.rados.rados.0.smithi115.stdout:1532: done (5 left) 2022-04-22T17:31:10.970 INFO:tasks.rados.rados.0.smithi115.stdout:1534: done (4 left) 2022-04-22T17:31:10.970 INFO:tasks.rados.rados.0.smithi115.stdout:1536: done (3 left) 2022-04-22T17:31:10.970 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 157 2022-04-22T17:31:10.970 INFO:tasks.rados.rados.0.smithi115.stdout:1537: seq_num 460 ranges {344064=507904} 2022-04-22T17:31:10.976 INFO:tasks.rados.rados.0.smithi115.stdout:1537: writing smithi11541303-22 from 344064 to 851968 tid 1 2022-04-22T17:31:10.976 INFO:tasks.rados.rados.0.smithi115.stdout:1538: read oid 25 snap -1 2022-04-22T17:31:10.976 INFO:tasks.rados.rados.0.smithi115.stdout:1538: expect (ObjNum 454 snap 154 seq_num 454) 2022-04-22T17:31:10.976 INFO:tasks.rados.rados.0.smithi115.stdout:1539: snap_create 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1531: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1533: finishing write tid 1 to smithi11541303-17 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1533: finishing write tid 2 to smithi11541303-17 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1533: finishing write tid 3 to smithi11541303-17 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 717 (ObjNum 457 snap 157 seq_num 457) dirty exists 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1533: left oid 17 (ObjNum 457 snap 157 seq_num 457) 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1535: finishing write tid 1 to smithi11541303-10 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1535: finishing write tid 2 to smithi11541303-10 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:1535: finishing write tid 3 to smithi11541303-10 2022-04-22T17:31:10.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 704 (ObjNum 458 snap 157 seq_num 458) dirty exists 2022-04-22T17:31:10.978 INFO:tasks.rados.rados.0.smithi115.stdout:1535: left oid 10 (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:31:10.979 INFO:tasks.rados.rados.0.smithi115.stdout:1537: finishing write tid 1 to smithi11541303-22 2022-04-22T17:31:10.980 INFO:tasks.rados.rados.0.smithi115.stdout:1538: expect (ObjNum 454 snap 154 seq_num 454) 2022-04-22T17:31:10.990 INFO:tasks.rados.rados.0.smithi115.stdout:1537: finishing write tid 2 to smithi11541303-22 2022-04-22T17:31:10.990 INFO:tasks.rados.rados.0.smithi115.stdout:1537: finishing write tid 3 to smithi11541303-22 2022-04-22T17:31:10.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 526 (ObjNum 460 snap 157 seq_num 460) dirty exists 2022-04-22T17:31:10.991 INFO:tasks.rados.rados.0.smithi115.stdout:1537: left oid 22 (ObjNum 460 snap 157 seq_num 460) 2022-04-22T17:31:11.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[45495]: cluster 2022-04-22T17:31:09.190531+0000 mgr.y (mgr.34107) 163 : cluster [DBG] pgmap v247: 121 pgs: 121 active+clean; 137 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:11.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[45495]: cluster 2022-04-22T17:31:10.049342+0000 mon.a (mon.0) 488 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded) 2022-04-22T17:31:11.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[45495]: cluster 2022-04-22T17:31:10.049415+0000 mon.a (mon.0) 489 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:11.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[46934]: cluster 2022-04-22T17:31:09.190531+0000 mgr.y (mgr.34107) 163 : cluster [DBG] pgmap v247: 121 pgs: 121 active+clean; 137 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:11.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[46934]: cluster 2022-04-22T17:31:10.049342+0000 mon.a (mon.0) 488 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded) 2022-04-22T17:31:11.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:10 smithi115 bash[46934]: cluster 2022-04-22T17:31:10.049415+0000 mon.a (mon.0) 489 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:10 smithi121 bash[36902]: cluster 2022-04-22T17:31:09.190531+0000 mgr.y (mgr.34107) 163 : cluster [DBG] pgmap v247: 121 pgs: 121 active+clean; 137 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:10 smithi121 bash[36902]: cluster 2022-04-22T17:31:10.049342+0000 mon.a (mon.0) 488 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/225 objects degraded (0.889%), 2 pgs degraded) 2022-04-22T17:31:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:10 smithi121 bash[36902]: cluster 2022-04-22T17:31:10.049415+0000 mon.a (mon.0) 489 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:11.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:11.622Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:11.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:11.622Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:11.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:11.622Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:31:11.950 INFO:tasks.rados.rados.0.smithi115.stdout:1531: done (5 left) 2022-04-22T17:31:11.950 INFO:tasks.rados.rados.0.smithi115.stdout:1533: done (4 left) 2022-04-22T17:31:11.950 INFO:tasks.rados.rados.0.smithi115.stdout:1535: done (3 left) 2022-04-22T17:31:11.951 INFO:tasks.rados.rados.0.smithi115.stdout:1537: done (2 left) 2022-04-22T17:31:11.951 INFO:tasks.rados.rados.0.smithi115.stdout:1538: done (1 left) 2022-04-22T17:31:11.951 INFO:tasks.rados.rados.0.smithi115.stdout:1539: done (0 left) 2022-04-22T17:31:11.951 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 158 2022-04-22T17:31:11.951 INFO:tasks.rados.rados.0.smithi115.stdout:1540: seq_num 461 ranges {0=524288} 2022-04-22T17:31:11.953 INFO:tasks.rados.rados.0.smithi115.stdout:1540: writing smithi11541303-27 from 0 to 524288 tid 1 2022-04-22T17:31:11.954 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 158 2022-04-22T17:31:11.954 INFO:tasks.rados.rados.0.smithi115.stdout:1541: seq_num 462 ranges {442368=540672} 2022-04-22T17:31:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:1541: writing smithi11541303-15 from 442368 to 983040 tid 1 2022-04-22T17:31:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:1542: read oid 29 snap -1 2022-04-22T17:31:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:1542: expect deleted 2022-04-22T17:31:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 158 2022-04-22T17:31:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:1543: seq_num 463 ranges {2056192=557056} 2022-04-22T17:31:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:1543: writing smithi11541303-31 from 2056192 to 2613248 tid 1 2022-04-22T17:31:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:1544: read oid 35 snap -1 2022-04-22T17:31:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:1544: expect (ObjNum 451 snap 152 seq_num 451) 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1545: delete oid 23 current snap is 158 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1540: finishing write tid 1 to smithi11541303-27 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1540: finishing write tid 2 to smithi11541303-27 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1540: finishing write tid 3 to smithi11541303-27 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 870 (ObjNum 461 snap 158 seq_num 461) dirty exists 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1540: left oid 27 (ObjNum 461 snap 158 seq_num 461) 2022-04-22T17:31:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:1541: finishing write tid 1 to smithi11541303-15 2022-04-22T17:31:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:1541: finishing write tid 2 to smithi11541303-15 2022-04-22T17:31:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:1541: finishing write tid 3 to smithi11541303-15 2022-04-22T17:31:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 705 (ObjNum 462 snap 158 seq_num 462) dirty exists 2022-04-22T17:31:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:1541: left oid 15 (ObjNum 462 snap 158 seq_num 462) 2022-04-22T17:31:11.982 INFO:tasks.rados.rados.0.smithi115.stdout:1540: done (5 left) 2022-04-22T17:31:11.982 INFO:tasks.rados.rados.0.smithi115.stdout:1541: done (4 left) 2022-04-22T17:31:11.982 INFO:tasks.rados.rados.0.smithi115.stdout:1542: done (3 left) 2022-04-22T17:31:11.982 INFO:tasks.rados.rados.0.smithi115.stdout:1545: done (2 left) 2022-04-22T17:31:11.983 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 158 2022-04-22T17:31:11.983 INFO:tasks.rados.rados.0.smithi115.stdout:1546: seq_num 465 ranges {2252800=598016} 2022-04-22T17:31:12.002 INFO:tasks.rados.rados.0.smithi115.stdout:1546: writing smithi11541303-21 from 2252800 to 2850816 tid 1 2022-04-22T17:31:12.002 INFO:tasks.rados.rados.0.smithi115.stdout:1547: snap_remove snap 153 2022-04-22T17:31:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:11 smithi121 bash[36902]: cluster 2022-04-22T17:31:10.926043+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2022-04-22T17:31:12.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:11 smithi115 bash[45495]: cluster 2022-04-22T17:31:10.926043+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2022-04-22T17:31:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:11 smithi115 bash[46934]: cluster 2022-04-22T17:31:10.926043+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2022-04-22T17:31:12.955 INFO:tasks.rados.rados.0.smithi115.stdout:1547: done (3 left) 2022-04-22T17:31:12.955 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 158 2022-04-22T17:31:12.955 INFO:tasks.rados.rados.0.smithi115.stdout:1543: finishing write tid 1 to smithi11541303-31 2022-04-22T17:31:12.955 INFO:tasks.rados.rados.0.smithi115.stdout:1543: finishing write tid 2 to smithi11541303-31 2022-04-22T17:31:12.955 INFO:tasks.rados.rados.0.smithi115.stdout:1544: expect (ObjNum 451 snap 152 seq_num 451) 2022-04-22T17:31:13.127 INFO:tasks.rados.rados.0.smithi115.stdout:1543: finishing write tid 3 to smithi11541303-31 2022-04-22T17:31:13.128 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 915 (ObjNum 463 snap 158 seq_num 463) dirty exists 2022-04-22T17:31:13.128 INFO:tasks.rados.rados.0.smithi115.stdout:1543: left oid 31 (ObjNum 463 snap 158 seq_num 463) 2022-04-22T17:31:13.128 INFO:tasks.rados.rados.0.smithi115.stdout:1548: seq_num 466 ranges {311296=614400} 2022-04-22T17:31:13.137 INFO:tasks.rados.rados.0.smithi115.stdout:1548: writing smithi11541303-9 from 311296 to 925696 tid 1 2022-04-22T17:31:13.137 INFO:tasks.rados.rados.0.smithi115.stdout:1543: done (3 left) 2022-04-22T17:31:13.137 INFO:tasks.rados.rados.0.smithi115.stdout:1544: done (2 left) 2022-04-22T17:31:13.137 INFO:tasks.rados.rados.0.smithi115.stdout:1549: snap_create 2022-04-22T17:31:13.138 INFO:tasks.rados.rados.0.smithi115.stdout:1546: finishing write tid 1 to smithi11541303-21 2022-04-22T17:31:13.138 INFO:tasks.rados.rados.0.smithi115.stdout:1546: finishing write tid 2 to smithi11541303-21 2022-04-22T17:31:13.138 INFO:tasks.rados.rados.0.smithi115.stdout:1546: finishing write tid 3 to smithi11541303-21 2022-04-22T17:31:13.138 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 528 (ObjNum 465 snap 158 seq_num 465) dirty exists 2022-04-22T17:31:13.139 INFO:tasks.rados.rados.0.smithi115.stdout:1546: left oid 21 (ObjNum 465 snap 158 seq_num 465) 2022-04-22T17:31:13.143 INFO:tasks.rados.rados.0.smithi115.stdout:1548: finishing write tid 1 to smithi11541303-9 2022-04-22T17:31:13.144 INFO:tasks.rados.rados.0.smithi115.stdout:1548: finishing write tid 2 to smithi11541303-9 2022-04-22T17:31:13.147 INFO:tasks.rados.rados.0.smithi115.stdout:1548: finishing write tid 3 to smithi11541303-9 2022-04-22T17:31:13.147 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 924 (ObjNum 466 snap 158 seq_num 466) dirty exists 2022-04-22T17:31:13.147 INFO:tasks.rados.rados.0.smithi115.stdout:1548: left oid 9 (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:12 smithi121 bash[36902]: cluster 2022-04-22T17:31:11.191370+0000 mgr.y (mgr.34107) 164 : cluster [DBG] pgmap v250: 121 pgs: 121 active+clean; 137 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:13.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:12 smithi121 bash[36902]: cluster 2022-04-22T17:31:11.932501+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2022-04-22T17:31:13.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:12 smithi115 bash[45495]: cluster 2022-04-22T17:31:11.191370+0000 mgr.y (mgr.34107) 164 : cluster [DBG] pgmap v250: 121 pgs: 121 active+clean; 137 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:13.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:12 smithi115 bash[45495]: cluster 2022-04-22T17:31:11.932501+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2022-04-22T17:31:13.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:12 smithi115 bash[46934]: cluster 2022-04-22T17:31:11.191370+0000 mgr.y (mgr.34107) 164 : cluster [DBG] pgmap v250: 121 pgs: 121 active+clean; 137 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 284 KiB/s wr, 5 op/s; 160 KiB/s, 0 objects/s recovering 2022-04-22T17:31:13.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:12 smithi115 bash[46934]: cluster 2022-04-22T17:31:11.932501+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2022-04-22T17:31:13.785 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:13.517Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:13.785 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:13.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:13.786 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:13.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:13.966 INFO:tasks.rados.rados.0.smithi115.stdout:1546: done (2 left) 2022-04-22T17:31:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1548: done (1 left) 2022-04-22T17:31:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1549: done (0 left) 2022-04-22T17:31:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 159 2022-04-22T17:31:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:1550: seq_num 467 ranges {1032192=630784} 2022-04-22T17:31:13.978 INFO:tasks.rados.rados.0.smithi115.stdout:1550: writing smithi11541303-3 from 1032192 to 1662976 tid 1 2022-04-22T17:31:13.978 INFO:tasks.rados.rados.0.smithi115.stdout:1551: read oid 23 snap -1 2022-04-22T17:31:13.978 INFO:tasks.rados.rados.0.smithi115.stdout:1551: expect deleted 2022-04-22T17:31:13.978 INFO:tasks.rados.rados.0.smithi115.stdout:1552: read oid 30 snap 151 2022-04-22T17:31:13.979 INFO:tasks.rados.rados.0.smithi115.stdout:1552: expect (ObjNum 447 snap 149 seq_num 447) 2022-04-22T17:31:13.979 INFO:tasks.rados.rados.0.smithi115.stdout:1553: rmattr oid 13 current snap is 159 2022-04-22T17:31:13.979 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 159 2022-04-22T17:31:13.979 INFO:tasks.rados.rados.0.smithi115.stdout:1554: seq_num 468 ranges {0=647168} 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:1554: writing smithi11541303-40 from 0 to 647168 tid 1 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:1555: read oid 50 snap -1 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:1555: expect deleted 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:1556: rollback oid 31 current snap is 159 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 151 2022-04-22T17:31:13.984 INFO:tasks.rados.rados.0.smithi115.stdout:1557: delete oid 16 current snap is 159 2022-04-22T17:31:13.985 INFO:tasks.rados.rados.0.smithi115.stdout:1552: expect (ObjNum 447 snap 149 seq_num 447) 2022-04-22T17:31:13.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 723 (ObjNum 1567450988 snap 1819042149 seq_num 1517377876) dirty exists 2022-04-22T17:31:13.999 INFO:tasks.rados.rados.0.smithi115.stdout:1554: finishing write tid 1 to smithi11541303-40 2022-04-22T17:31:13.999 INFO:tasks.rados.rados.0.smithi115.stdout:1554: finishing write tid 2 to smithi11541303-40 2022-04-22T17:31:13.999 INFO:tasks.rados.rados.0.smithi115.stdout:1554: finishing write tid 3 to smithi11541303-40 2022-04-22T17:31:13.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 706 (ObjNum 468 snap 159 seq_num 468) dirty exists 2022-04-22T17:31:14.000 INFO:tasks.rados.rados.0.smithi115.stdout:1554: left oid 40 (ObjNum 468 snap 159 seq_num 468) 2022-04-22T17:31:14.000 INFO:tasks.rados.rados.0.smithi115.stdout:1550: finishing write tid 1 to smithi11541303-3 2022-04-22T17:31:14.000 INFO:tasks.rados.rados.0.smithi115.stdout:1550: finishing write tid 2 to smithi11541303-3 2022-04-22T17:31:14.001 INFO:tasks.rados.rados.0.smithi115.stdout:1551: done (7 left) 2022-04-22T17:31:14.001 INFO:tasks.rados.rados.0.smithi115.stdout:1552: done (6 left) 2022-04-22T17:31:14.001 INFO:tasks.rados.rados.0.smithi115.stdout:1553: done (5 left) 2022-04-22T17:31:14.001 INFO:tasks.rados.rados.0.smithi115.stdout:1554: done (4 left) 2022-04-22T17:31:14.001 INFO:tasks.rados.rados.0.smithi115.stdout:1555: done (3 left) 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1557: done (2 left) 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1558: read oid 45 snap 152 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1558: expect (ObjNum 429 snap 146 seq_num 429) 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1559: read oid 38 snap -1 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1559: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:14.002 INFO:tasks.rados.rados.0.smithi115.stdout:1560: snap_remove snap 156 2022-04-22T17:31:14.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[46934]: cluster 2022-04-22T17:31:12.938862+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2022-04-22T17:31:14.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[46934]: audit 2022-04-22T17:31:13.245960+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:31:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[46934]: audit 2022-04-22T17:31:13.246668+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:31:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[45495]: cluster 2022-04-22T17:31:12.938862+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2022-04-22T17:31:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[45495]: audit 2022-04-22T17:31:13.245960+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:31:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:13 smithi115 bash[45495]: audit 2022-04-22T17:31:13.246668+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:31:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:13 smithi121 bash[36902]: cluster 2022-04-22T17:31:12.938862+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2022-04-22T17:31:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:13 smithi121 bash[36902]: audit 2022-04-22T17:31:13.245960+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:31:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:13 smithi121 bash[36902]: audit 2022-04-22T17:31:13.246668+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:31:14.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:14 smithi115 systemd[1]: Stopping Ceph osd.2 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:31:14.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[26675]: debug 2022-04-22T17:31:14.115+0000 7faa24bec700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:31:14.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[26675]: debug 2022-04-22T17:31:14.115+0000 7faa24bec700 -1 osd.2 378 *** Got signal Terminated *** 2022-04-22T17:31:14.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[26675]: debug 2022-04-22T17:31:14.115+0000 7faa24bec700 -1 osd.2 378 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:31:14.806 INFO:tasks.rados.rados.0.smithi115.stdout:1556: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:1556: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:1556: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 928 (ObjNum 399 snap 137 seq_num 399) dirty exists 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:1550: finishing write tid 3 to smithi11541303-3 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 874 (ObjNum 467 snap 159 seq_num 467) dirty exists 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:1550: left oid 3 (ObjNum 467 snap 159 seq_num 467) 2022-04-22T17:31:14.807 INFO:tasks.rados.rados.0.smithi115.stdout:1559: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:14.914 INFO:tasks.rados.rados.0.smithi115.stdout:1558: expect (ObjNum 429 snap 146 seq_num 429) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1550: done (4 left) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1556: done (3 left) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1558: done (2 left) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1559: done (1 left) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1560: done (0 left) 2022-04-22T17:31:14.994 INFO:tasks.rados.rados.0.smithi115.stdout:1561: snap_create 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:13.192348+0000 mgr.y (mgr.34107) 165 : cluster [DBG] pgmap v253: 121 pgs: 121 active+clean; 138 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 270 KiB/s wr, 2 op/s 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:13.950613+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: audit 2022-04-22T17:31:14.209665+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.391722+0000 mon.a (mon.0) 497 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.391773+0000 mon.a (mon.0) 498 : cluster [INF] osd.2 failed (root=default,host=smithi115) (connection refused reported by osd.3) 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.392011+0000 mon.a (mon.0) 499 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.392200+0000 mon.a (mon.0) 500 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.392534+0000 mon.a (mon.0) 501 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.392763+0000 mon.a (mon.0) 502 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.392980+0000 mon.a (mon.0) 503 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.393795+0000 mon.a (mon.0) 504 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.393868+0000 mon.a (mon.0) 505 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394094+0000 mon.a (mon.0) 506 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394241+0000 mon.a (mon.0) 507 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394564+0000 mon.a (mon.0) 508 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394747+0000 mon.a (mon.0) 509 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394883+0000 mon.a (mon.0) 510 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.394994+0000 mon.a (mon.0) 511 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.395256+0000 mon.a (mon.0) 512 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.446673+0000 mon.a (mon.0) 513 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.446821+0000 mon.a (mon.0) 514 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.456128+0000 mon.a (mon.0) 515 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.649967+0000 mon.a (mon.0) 516 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.773255+0000 mon.a (mon.0) 517 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:15.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:14 smithi121 bash[36902]: cluster 2022-04-22T17:31:14.786970+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e379: 8 total, 7 up, 8 in 2022-04-22T17:31:15.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:15 smithi115 bash[53328]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.2 2022-04-22T17:31:15.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:15 smithi115 bash[53370]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-2 2022-04-22T17:31:15.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:15 smithi115 bash[53383]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.2 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:13.192348+0000 mgr.y (mgr.34107) 165 : cluster [DBG] pgmap v253: 121 pgs: 121 active+clean; 138 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 270 KiB/s wr, 2 op/s 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:13.950613+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: audit 2022-04-22T17:31:14.209665+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.391722+0000 mon.a (mon.0) 497 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.391773+0000 mon.a (mon.0) 498 : cluster [INF] osd.2 failed (root=default,host=smithi115) (connection refused reported by osd.3) 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.392011+0000 mon.a (mon.0) 499 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.392200+0000 mon.a (mon.0) 500 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.392534+0000 mon.a (mon.0) 501 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.392763+0000 mon.a (mon.0) 502 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.392980+0000 mon.a (mon.0) 503 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.393795+0000 mon.a (mon.0) 504 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.393868+0000 mon.a (mon.0) 505 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394094+0000 mon.a (mon.0) 506 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394241+0000 mon.a (mon.0) 507 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394564+0000 mon.a (mon.0) 508 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394747+0000 mon.a (mon.0) 509 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394883+0000 mon.a (mon.0) 510 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.394994+0000 mon.a (mon.0) 511 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.395256+0000 mon.a (mon.0) 512 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.446673+0000 mon.a (mon.0) 513 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.446821+0000 mon.a (mon.0) 514 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.456128+0000 mon.a (mon.0) 515 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.649967+0000 mon.a (mon.0) 516 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.773255+0000 mon.a (mon.0) 517 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[46934]: cluster 2022-04-22T17:31:14.786970+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e379: 8 total, 7 up, 8 in 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:13.192348+0000 mgr.y (mgr.34107) 165 : cluster [DBG] pgmap v253: 121 pgs: 121 active+clean; 138 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 270 KiB/s wr, 2 op/s 2022-04-22T17:31:15.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:13.950613+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: audit 2022-04-22T17:31:14.209665+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.391722+0000 mon.a (mon.0) 497 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.391773+0000 mon.a (mon.0) 498 : cluster [INF] osd.2 failed (root=default,host=smithi115) (connection refused reported by osd.3) 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.392011+0000 mon.a (mon.0) 499 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.392200+0000 mon.a (mon.0) 500 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.392534+0000 mon.a (mon.0) 501 : cluster [DBG] osd.2 reported immediately failed by osd.1 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.392763+0000 mon.a (mon.0) 502 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.392980+0000 mon.a (mon.0) 503 : cluster [DBG] osd.2 reported immediately failed by osd.0 2022-04-22T17:31:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.393795+0000 mon.a (mon.0) 504 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.393868+0000 mon.a (mon.0) 505 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394094+0000 mon.a (mon.0) 506 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394241+0000 mon.a (mon.0) 507 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394564+0000 mon.a (mon.0) 508 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394747+0000 mon.a (mon.0) 509 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394883+0000 mon.a (mon.0) 510 : cluster [DBG] osd.2 reported immediately failed by osd.6 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.394994+0000 mon.a (mon.0) 511 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.395256+0000 mon.a (mon.0) 512 : cluster [DBG] osd.2 reported immediately failed by osd.7 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.446673+0000 mon.a (mon.0) 513 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.446821+0000 mon.a (mon.0) 514 : cluster [DBG] osd.2 reported immediately failed by osd.4 2022-04-22T17:31:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.456128+0000 mon.a (mon.0) 515 : cluster [DBG] osd.2 reported immediately failed by osd.3 2022-04-22T17:31:15.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.649967+0000 mon.a (mon.0) 516 : cluster [DBG] osd.2 reported immediately failed by osd.5 2022-04-22T17:31:15.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.773255+0000 mon.a (mon.0) 517 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:15.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:14 smithi115 bash[45495]: cluster 2022-04-22T17:31:14.786970+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e379: 8 total, 7 up, 8 in 2022-04-22T17:31:15.801 INFO:tasks.rados.rados.0.smithi115.stdout:1561: done (0 left) 2022-04-22T17:31:15.801 INFO:tasks.rados.rados.0.smithi115.stdout:1562: snap_create 2022-04-22T17:31:16.801 INFO:tasks.rados.rados.0.smithi115.stdout:1562: done (0 left) 2022-04-22T17:31:16.801 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 161 2022-04-22T17:31:16.801 INFO:tasks.rados.rados.0.smithi115.stdout:1563: seq_num 470 ranges {647168=679936} 2022-04-22T17:31:16.809 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:31:16.812 INFO:tasks.rados.rados.0.smithi115.stdout:1563: writing smithi11541303-40 from 647168 to 1327104 tid 1 2022-04-22T17:31:16.812 INFO:tasks.rados.rados.0.smithi115.stdout:1564: snap_create 2022-04-22T17:31:16.820 INFO:tasks.rados.rados.0.smithi115.stdout:1563: finishing write tid 1 to smithi11541303-40 2022-04-22T17:31:16.820 INFO:tasks.rados.rados.0.smithi115.stdout:1563: finishing write tid 2 to smithi11541303-40 2022-04-22T17:31:16.822 INFO:tasks.rados.rados.0.smithi115.stdout:1563: finishing write tid 3 to smithi11541303-40 2022-04-22T17:31:16.822 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 708 (ObjNum 470 snap 161 seq_num 470) dirty exists 2022-04-22T17:31:16.823 INFO:tasks.rados.rados.0.smithi115.stdout:1563: left oid 40 (ObjNum 470 snap 161 seq_num 470) 2022-04-22T17:31:17.146 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:31:17.146 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (7m) 20s ago 8m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:31:17.146 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (7m) 2m ago 7m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:31:17.146 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (9m) 20s ago 9m 13.8M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (9m) 2m ago 9m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3m) 2m ago 13m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4m) 20s ago 15m 446M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3m) 20s ago 15m 54.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2m) 2m ago 13m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2m) 20s ago 13m 47.6M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (8m) 20s ago 8m 9195k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (8m) 2m ago 8m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (64s) 20s ago 13m 165M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (22s) 20s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:31:17.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 20s ago 12m 192M 2526M 16.2.5 46cf6318c64e b37f19e557e7 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (11m) 20s ago 11m 130M 2526M 16.2.5 46cf6318c64e b340f4106def 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (11m) 2m ago 11m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (10m) 2m ago 10m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (10m) 2m ago 10m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (9m) 2m ago 9m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:31:17.148 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4m) 2m ago 8m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:31:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:16 smithi121 bash[36902]: cluster 2022-04-22T17:31:15.193144+0000 mgr.y (mgr.34107) 166 : cluster [DBG] pgmap v256: 121 pgs: 13 stale+active+clean, 108 active+clean; 146 MiB data, 925 MiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2022-04-22T17:31:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:16 smithi121 bash[36902]: cluster 2022-04-22T17:31:15.784718+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e380: 8 total, 7 up, 8 in 2022-04-22T17:31:17.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:16 smithi115 bash[45495]: cluster 2022-04-22T17:31:15.193144+0000 mgr.y (mgr.34107) 166 : cluster [DBG] pgmap v256: 121 pgs: 13 stale+active+clean, 108 active+clean; 146 MiB data, 925 MiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2022-04-22T17:31:17.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:16 smithi115 bash[45495]: cluster 2022-04-22T17:31:15.784718+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e380: 8 total, 7 up, 8 in 2022-04-22T17:31:17.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:16 smithi115 bash[46934]: cluster 2022-04-22T17:31:15.193144+0000 mgr.y (mgr.34107) 166 : cluster [DBG] pgmap v256: 121 pgs: 13 stale+active+clean, 108 active+clean; 146 MiB data, 925 MiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2022-04-22T17:31:17.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:16 smithi115 bash[46934]: cluster 2022-04-22T17:31:15.784718+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e380: 8 total, 7 up, 8 in 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:31:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 5, 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:31:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:31:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 7, 2022-04-22T17:31:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 7 2022-04-22T17:31:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:31:17.548 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:31:17.823 INFO:tasks.rados.rados.0.smithi115.stdout:1563: done (1 left) 2022-04-22T17:31:17.823 INFO:tasks.rados.rados.0.smithi115.stdout:1564: done (0 left) 2022-04-22T17:31:17.824 INFO:tasks.rados.rados.0.smithi115.stdout:1565: copy_from oid 50 from oid 10 current snap is 162 2022-04-22T17:31:17.824 INFO:tasks.rados.rados.0.smithi115.stdout:1566: snap_remove snap 155 2022-04-22T17:31:17.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[46934]: audit 2022-04-22T17:31:16.775645+0000 mgr.y (mgr.34107) 167 : audit [DBG] from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:17.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[46934]: cluster 2022-04-22T17:31:16.784761+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e381: 8 total, 7 up, 8 in 2022-04-22T17:31:17.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[46934]: audit 2022-04-22T17:31:17.529520+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/3628246012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:17.966 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:17 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.2.service: Succeeded. 2022-04-22T17:31:17.966 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:17 smithi115 systemd[1]: Stopped Ceph osd.2 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:31:17.966 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[45495]: audit 2022-04-22T17:31:16.775645+0000 mgr.y (mgr.34107) 167 : audit [DBG] from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:17.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[45495]: cluster 2022-04-22T17:31:16.784761+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e381: 8 total, 7 up, 8 in 2022-04-22T17:31:17.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:17 smithi115 bash[45495]: audit 2022-04-22T17:31:17.529520+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/3628246012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:17 smithi121 bash[36902]: audit 2022-04-22T17:31:16.775645+0000 mgr.y (mgr.34107) 167 : audit [DBG] from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:17 smithi121 bash[36902]: cluster 2022-04-22T17:31:16.784761+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e381: 8 total, 7 up, 8 in 2022-04-22T17:31:18.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:17 smithi121 bash[36902]: audit 2022-04-22T17:31:17.529520+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.115:0/3628246012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:18.303 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:17 smithi115 systemd[1]: Started Ceph osd.2 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:31:18.831 INFO:tasks.rados.rados.0.smithi115.stdout:1566: done (1 left) 2022-04-22T17:31:18.831 INFO:tasks.rados.rados.0.smithi115.stdout:1567: copy_from oid 5 from oid 2 current snap is 162 2022-04-22T17:31:18.831 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 162 2022-04-22T17:31:18.831 INFO:tasks.rados.rados.0.smithi115.stdout:1568: seq_num 471 ranges {12795904=696320} 2022-04-22T17:31:18.909 INFO:tasks.rados.rados.0.smithi115.stdout:1568: writing smithi11541303-11 from 12795904 to 13492224 tid 1 2022-04-22T17:31:18.909 INFO:tasks.rados.rados.0.smithi115.stdout:1569: snap_create 2022-04-22T17:31:18.909 INFO:tasks.rados.rados.0.smithi115.stdout:1565: finishing copy_from to smithi11541303-50 2022-04-22T17:31:18.909 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 704 (ObjNum 458 snap 157 seq_num 458) dirty exists 2022-04-22T17:31:18.909 INFO:tasks.rados.rados.0.smithi115.stdout:1565: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:31:18.910 INFO:tasks.rados.rados.0.smithi115.stdout:1567: finishing copy_from to smithi11541303-5 2022-04-22T17:31:18.910 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 649 (ObjNum 443 snap 148 seq_num 443) dirty exists 2022-04-22T17:31:18.910 INFO:tasks.rados.rados.0.smithi115.stdout:1567: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:31:18.914 INFO:tasks.rados.rados.0.smithi115.stdout:1568: finishing write tid 1 to smithi11541303-11 2022-04-22T17:31:18.915 INFO:tasks.rados.rados.0.smithi115.stdout:1568: finishing write tid 2 to smithi11541303-11 2022-04-22T17:31:18.916 INFO:tasks.rados.rados.0.smithi115.stdout:1568: finishing write tid 3 to smithi11541303-11 2022-04-22T17:31:18.917 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 876 (ObjNum 471 snap 162 seq_num 471) dirty exists 2022-04-22T17:31:18.917 INFO:tasks.rados.rados.0.smithi115.stdout:1568: left oid 11 (ObjNum 471 snap 162 seq_num 471) 2022-04-22T17:31:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:17.117077+0000 mgr.y (mgr.34107) 168 : audit [DBG] from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: cluster 2022-04-22T17:31:17.193937+0000 mgr.y (mgr.34107) 169 : cluster [DBG] pgmap v259: 121 pgs: 1 active+undersized+degraded, 12 stale+active+clean, 5 active+undersized, 103 active+clean; 146 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 4/285 objects degraded (1.404%) 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: cluster 2022-04-22T17:31:17.792073+0000 mon.a (mon.0) 521 : cluster [WRN] Health check failed: Degraded data redundancy: 4/285 objects degraded (1.404%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: cluster 2022-04-22T17:31:17.807365+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e382: 8 total, 7 up, 8 in 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:17.988094+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:17.989395+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:17.991257+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:17.992237+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:18 smithi121 bash[36902]: audit 2022-04-22T17:31:18.288425+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:17.117077+0000 mgr.y (mgr.34107) 168 : audit [DBG] from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: cluster 2022-04-22T17:31:17.193937+0000 mgr.y (mgr.34107) 169 : cluster [DBG] pgmap v259: 121 pgs: 1 active+undersized+degraded, 12 stale+active+clean, 5 active+undersized, 103 active+clean; 146 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 4/285 objects degraded (1.404%) 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: cluster 2022-04-22T17:31:17.792073+0000 mon.a (mon.0) 521 : cluster [WRN] Health check failed: Degraded data redundancy: 4/285 objects degraded (1.404%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: cluster 2022-04-22T17:31:17.807365+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e382: 8 total, 7 up, 8 in 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:17.988094+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:17.989395+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:17.991257+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:17.992237+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[45495]: audit 2022-04-22T17:31:18.288425+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:17.117077+0000 mgr.y (mgr.34107) 168 : audit [DBG] from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: cluster 2022-04-22T17:31:17.193937+0000 mgr.y (mgr.34107) 169 : cluster [DBG] pgmap v259: 121 pgs: 1 active+undersized+degraded, 12 stale+active+clean, 5 active+undersized, 103 active+clean; 146 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 4/285 objects degraded (1.404%) 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: cluster 2022-04-22T17:31:17.792073+0000 mon.a (mon.0) 521 : cluster [WRN] Health check failed: Degraded data redundancy: 4/285 objects degraded (1.404%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: cluster 2022-04-22T17:31:17.807365+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e382: 8 total, 7 up, 8 in 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:17.988094+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:17.989395+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:19.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:17.991257+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:19.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:17.992237+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:19.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:18 smithi115 bash[46934]: audit 2022-04-22T17:31:18.288425+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:19.839 INFO:tasks.rados.rados.0.smithi115.stdout:1565: done (3 left) 2022-04-22T17:31:19.839 INFO:tasks.rados.rados.0.smithi115.stdout:1567: done (2 left) 2022-04-22T17:31:19.839 INFO:tasks.rados.rados.0.smithi115.stdout:1568: done (1 left) 2022-04-22T17:31:19.839 INFO:tasks.rados.rados.0.smithi115.stdout:1569: done (0 left) 2022-04-22T17:31:19.840 INFO:tasks.rados.rados.0.smithi115.stdout:1570: read oid 9 snap -1 2022-04-22T17:31:19.840 INFO:tasks.rados.rados.0.smithi115.stdout:1570: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:19.840 INFO:tasks.rados.rados.0.smithi115.stdout:1571: read oid 50 snap -1 2022-04-22T17:31:19.840 INFO:tasks.rados.rados.0.smithi115.stdout:1571: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:31:19.840 INFO:tasks.rados.rados.0.smithi115.stdout:1572: snap_remove snap 160 2022-04-22T17:31:20.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[45495]: cluster 2022-04-22T17:31:18.814485+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e383: 8 total, 7 up, 8 in 2022-04-22T17:31:20.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[45495]: audit 2022-04-22T17:31:19.212429+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-04-22T17:31:20.169 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[53640]: --> ceph-volume raw activate successful for osd ID: 2 2022-04-22T17:31:20.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[46934]: cluster 2022-04-22T17:31:18.814485+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e383: 8 total, 7 up, 8 in 2022-04-22T17:31:20.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:19 smithi115 bash[46934]: audit 2022-04-22T17:31:19.212429+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:19 smithi121 bash[36902]: cluster 2022-04-22T17:31:18.814485+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e383: 8 total, 7 up, 8 in 2022-04-22T17:31:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:19 smithi121 bash[36902]: audit 2022-04-22T17:31:19.212429+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:20.849 INFO:tasks.rados.rados.0.smithi115.stdout:1572: done (2 left) 2022-04-22T17:31:20.849 INFO:tasks.rados.rados.0.smithi115.stdout:1573: copy_from oid 7 from oid 32 current snap is 163 2022-04-22T17:31:20.849 INFO:tasks.rados.rados.0.smithi115.stdout:1571: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:31:20.875 INFO:tasks.rados.rados.0.smithi115.stdout:1570: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:20.924 INFO:tasks.rados.rados.0.smithi115.stdout:1570: done (2 left) 2022-04-22T17:31:20.924 INFO:tasks.rados.rados.0.smithi115.stdout:1571: done (1 left) 2022-04-22T17:31:20.924 INFO:tasks.rados.rados.0.smithi115.stdout:1574: rollback oid 4 current snap is 163 2022-04-22T17:31:20.924 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 161 2022-04-22T17:31:20.925 INFO:tasks.rados.rados.0.smithi115.stdout:1575: read oid 15 snap -1 2022-04-22T17:31:20.925 INFO:tasks.rados.rados.0.smithi115.stdout:1575: expect (ObjNum 462 snap 158 seq_num 462) 2022-04-22T17:31:20.925 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 163 2022-04-22T17:31:20.925 INFO:tasks.rados.rados.0.smithi115.stdout:1576: seq_num 472 ranges {12795904=712704} 2022-04-22T17:31:21.011 INFO:tasks.rados.rados.0.smithi115.stdout:1576: writing smithi11541303-42 from 12795904 to 13508608 tid 1 2022-04-22T17:31:21.011 INFO:tasks.rados.rados.0.smithi115.stdout:1577: setattr oid 9 current snap is 163 2022-04-22T17:31:21.011 INFO:tasks.rados.rados.0.smithi115.stdout:1575: expect (ObjNum 462 snap 158 seq_num 462) 2022-04-22T17:31:21.052 INFO:tasks.rados.rados.0.smithi115.stdout:1574: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:31:21.052 INFO:tasks.rados.rados.0.smithi115.stdout:1574: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:31:21.052 INFO:tasks.rados.rados.0.smithi115.stdout:1574: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:31:21.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 874 (ObjNum 453 snap 153 seq_num 453) dirty exists 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1573: finishing copy_from to smithi11541303-7 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 536 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1573: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1576: finishing write tid 1 to smithi11541303-42 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1576: finishing write tid 2 to smithi11541303-42 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1576: finishing write tid 3 to smithi11541303-42 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 949 (ObjNum 472 snap 163 seq_num 472) dirty exists 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1576: left oid 42 (ObjNum 472 snap 163 seq_num 472) 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1573: done (4 left) 2022-04-22T17:31:21.053 INFO:tasks.rados.rados.0.smithi115.stdout:1574: done (3 left) 2022-04-22T17:31:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:1575: done (2 left) 2022-04-22T17:31:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:1576: done (1 left) 2022-04-22T17:31:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:1578: snap_remove snap 152 2022-04-22T17:31:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:20 smithi121 bash[36902]: cluster 2022-04-22T17:31:19.195372+0000 mgr.y (mgr.34107) 170 : cluster [DBG] pgmap v262: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:21.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:20 smithi121 bash[36902]: cluster 2022-04-22T17:31:19.822741+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e384: 8 total, 7 up, 8 in 2022-04-22T17:31:21.379 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:20 smithi115 bash[45495]: cluster 2022-04-22T17:31:19.195372+0000 mgr.y (mgr.34107) 170 : cluster [DBG] pgmap v262: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:21.380 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:20 smithi115 bash[45495]: cluster 2022-04-22T17:31:19.822741+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e384: 8 total, 7 up, 8 in 2022-04-22T17:31:21.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:20 smithi115 bash[46934]: cluster 2022-04-22T17:31:19.195372+0000 mgr.y (mgr.34107) 170 : cluster [DBG] pgmap v262: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:21.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:20 smithi115 bash[46934]: cluster 2022-04-22T17:31:19.822741+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e384: 8 total, 7 up, 8 in 2022-04-22T17:31:21.858 INFO:tasks.rados.rados.0.smithi115.stdout:1578: done (1 left) 2022-04-22T17:31:21.858 INFO:tasks.rados.rados.0.smithi115.stdout:1579: rmattr oid 23 current snap is 163 2022-04-22T17:31:21.858 INFO:tasks.rados.rados.0.smithi115.stdout:1579: done (1 left) 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1580: read oid 39 snap 158 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1580: expect deleted 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 951 (ObjNum 466 snap 158 seq_num 466) dirty exists 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1581: read oid 21 snap 161 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1581: expect (ObjNum 465 snap 158 seq_num 465) 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1577: done (2 left) 2022-04-22T17:31:21.859 INFO:tasks.rados.rados.0.smithi115.stdout:1582: delete oid 48 current snap is 163 2022-04-22T17:31:21.866 INFO:tasks.rados.rados.0.smithi115.stdout:1580: done (2 left) 2022-04-22T17:31:21.866 INFO:tasks.rados.rados.0.smithi115.stdout:1582: done (1 left) 2022-04-22T17:31:21.866 INFO:tasks.rados.rados.0.smithi115.stdout:1583: read oid 9 snap 158 2022-04-22T17:31:21.866 INFO:tasks.rados.rados.0.smithi115.stdout:1583: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:21.866 INFO:tasks.rados.rados.0.smithi115.stdout:1584: delete oid 27 current snap is 163 2022-04-22T17:31:21.872 INFO:tasks.rados.rados.0.smithi115.stdout:1584: done (2 left) 2022-04-22T17:31:21.872 INFO:tasks.rados.rados.0.smithi115.stdout:1585: delete oid 8 current snap is 163 2022-04-22T17:31:21.880 INFO:tasks.rados.rados.0.smithi115.stdout:1581: expect (ObjNum 465 snap 158 seq_num 465) 2022-04-22T17:31:21.993 INFO:tasks.rados.rados.0.smithi115.stdout:1581: done (2 left) 2022-04-22T17:31:21.994 INFO:tasks.rados.rados.0.smithi115.stdout:1585: done (1 left) 2022-04-22T17:31:21.994 INFO:tasks.rados.rados.0.smithi115.stdout:1586: snap_create 2022-04-22T17:31:21.994 INFO:tasks.rados.rados.0.smithi115.stdout:1583: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:22.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:21 smithi115 bash[45495]: cluster 2022-04-22T17:31:20.832605+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e385: 8 total, 7 up, 8 in 2022-04-22T17:31:22.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:21 smithi115 bash[46934]: cluster 2022-04-22T17:31:20.832605+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e385: 8 total, 7 up, 8 in 2022-04-22T17:31:22.067 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:21.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:22.067 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:21.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:22.068 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:21.623Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:31:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:21 smithi121 bash[36902]: cluster 2022-04-22T17:31:20.832605+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e385: 8 total, 7 up, 8 in 2022-04-22T17:31:22.553 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[53813]: debug 2022-04-22T17:31:22.207+0000 7f5d13710700 1 -- 172.21.15.115:0/2119540759 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557538d4340 con 0x555753958800 2022-04-22T17:31:22.864 INFO:tasks.rados.rados.0.smithi115.stdout:1583: done (1 left) 2022-04-22T17:31:22.864 INFO:tasks.rados.rados.0.smithi115.stdout:1586: done (0 left) 2022-04-22T17:31:22.864 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 164 2022-04-22T17:31:22.864 INFO:tasks.rados.rados.0.smithi115.stdout:1587: seq_num 476 ranges {2908160=778240} 2022-04-22T17:31:22.888 INFO:tasks.rados.rados.0.smithi115.stdout:1587: writing smithi11541303-49 from 2908160 to 3686400 tid 1 2022-04-22T17:31:22.917 INFO:tasks.rados.rados.0.smithi115.stdout:1588: setattr oid 31 current snap is 164 2022-04-22T17:31:22.917 INFO:tasks.rados.rados.0.smithi115.stdout:1589: read oid 35 snap -1 2022-04-22T17:31:22.917 INFO:tasks.rados.rados.0.smithi115.stdout:1589: expect (ObjNum 451 snap 152 seq_num 451) 2022-04-22T17:31:22.917 INFO:tasks.rados.rados.0.smithi115.stdout:1590: snap_remove snap 162 2022-04-22T17:31:23.053 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[53813]: debug 2022-04-22T17:31:22.779+0000 7f5d1ad073c0 -1 Falling back to public interface 2022-04-22T17:31:23.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[45495]: cluster 2022-04-22T17:31:21.196483+0000 mgr.y (mgr.34107) 171 : cluster [DBG] pgmap v265: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:23.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[45495]: cluster 2022-04-22T17:31:21.839641+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e386: 8 total, 7 up, 8 in 2022-04-22T17:31:23.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[46934]: cluster 2022-04-22T17:31:21.196483+0000 mgr.y (mgr.34107) 171 : cluster [DBG] pgmap v265: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:23.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:22 smithi115 bash[46934]: cluster 2022-04-22T17:31:21.839641+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e386: 8 total, 7 up, 8 in 2022-04-22T17:31:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:22 smithi121 bash[36902]: cluster 2022-04-22T17:31:21.196483+0000 mgr.y (mgr.34107) 171 : cluster [DBG] pgmap v265: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s wr, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:22 smithi121 bash[36902]: cluster 2022-04-22T17:31:21.839641+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e386: 8 total, 7 up, 8 in 2022-04-22T17:31:23.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:23.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:23.804 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:23.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:23.804 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:23.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:23.875 INFO:tasks.rados.rados.0.smithi115.stdout:1590: done (3 left) 2022-04-22T17:31:23.876 INFO:tasks.rados.rados.0.smithi115.stdout:1591: setattr oid 18 current snap is 164 2022-04-22T17:31:23.876 INFO:tasks.rados.rados.0.smithi115.stdout:1587: finishing write tid 1 to smithi11541303-49 2022-04-22T17:31:23.876 INFO:tasks.rados.rados.0.smithi115.stdout:1587: finishing write tid 2 to smithi11541303-49 2022-04-22T17:31:23.876 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 968 (ObjNum 399 snap 137 seq_num 399) dirty exists 2022-04-22T17:31:23.876 INFO:tasks.rados.rados.0.smithi115.stdout:1589: expect (ObjNum 451 snap 152 seq_num 451) 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1587: finishing write tid 3 to smithi11541303-49 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 966 (ObjNum 476 snap 164 seq_num 476) dirty exists 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1587: left oid 49 (ObjNum 476 snap 164 seq_num 476) 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1587: done (3 left) 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1588: done (2 left) 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1589: done (1 left) 2022-04-22T17:31:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:1592: snap_create 2022-04-22T17:31:24.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 978 (ObjNum 445 snap 148 seq_num 445) dirty exists 2022-04-22T17:31:24.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[45495]: cluster 2022-04-22T17:31:22.846787+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e387: 8 total, 7 up, 8 in 2022-04-22T17:31:24.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[45495]: audit 2022-04-22T17:31:23.465924+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:24.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[46934]: cluster 2022-04-22T17:31:22.846787+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e387: 8 total, 7 up, 8 in 2022-04-22T17:31:24.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:23 smithi115 bash[46934]: audit 2022-04-22T17:31:23.465924+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:23 smithi121 bash[36902]: cluster 2022-04-22T17:31:22.846787+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e387: 8 total, 7 up, 8 in 2022-04-22T17:31:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:23 smithi121 bash[36902]: audit 2022-04-22T17:31:23.465924+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:24.804 INFO:tasks.rados.rados.0.smithi115.stdout:1591: done (1 left) 2022-04-22T17:31:24.804 INFO:tasks.rados.rados.0.smithi115.stdout:1592: done (0 left) 2022-04-22T17:31:24.804 INFO:tasks.rados.rados.0.smithi115.stdout:1593: snap_create 2022-04-22T17:31:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: cluster 2022-04-22T17:31:23.197527+0000 mgr.y (mgr.34107) 172 : cluster [DBG] pgmap v268: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: cluster 2022-04-22T17:31:23.859070+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e388: 8 total, 7 up, 8 in 2022-04-22T17:31:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: audit 2022-04-22T17:31:24.063115+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: audit 2022-04-22T17:31:24.215456+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: cluster 2022-04-22T17:31:24.776569+0000 mon.a (mon.0) 538 : cluster [WRN] Health check update: Degraded data redundancy: 31/297 objects degraded (10.438%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:25.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:24 smithi121 bash[36902]: cluster 2022-04-22T17:31:24.787541+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e389: 8 total, 7 up, 8 in 2022-04-22T17:31:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: cluster 2022-04-22T17:31:23.197527+0000 mgr.y (mgr.34107) 172 : cluster [DBG] pgmap v268: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: cluster 2022-04-22T17:31:23.859070+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e388: 8 total, 7 up, 8 in 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: audit 2022-04-22T17:31:24.063115+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: audit 2022-04-22T17:31:24.215456+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: cluster 2022-04-22T17:31:24.776569+0000 mon.a (mon.0) 538 : cluster [WRN] Health check update: Degraded data redundancy: 31/297 objects degraded (10.438%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[45495]: cluster 2022-04-22T17:31:24.787541+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e389: 8 total, 7 up, 8 in 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: cluster 2022-04-22T17:31:23.197527+0000 mgr.y (mgr.34107) 172 : cluster [DBG] pgmap v268: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 150 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1 op/s; 31/297 objects degraded (10.438%) 2022-04-22T17:31:25.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: cluster 2022-04-22T17:31:23.859070+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e388: 8 total, 7 up, 8 in 2022-04-22T17:31:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: audit 2022-04-22T17:31:24.063115+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: audit 2022-04-22T17:31:24.215456+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: cluster 2022-04-22T17:31:24.776569+0000 mon.a (mon.0) 538 : cluster [WRN] Health check update: Degraded data redundancy: 31/297 objects degraded (10.438%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:25.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:24 smithi115 bash[46934]: cluster 2022-04-22T17:31:24.787541+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e389: 8 total, 7 up, 8 in 2022-04-22T17:31:25.807 INFO:tasks.rados.rados.0.smithi115.stdout:1593: done (0 left) 2022-04-22T17:31:25.807 INFO:tasks.rados.rados.0.smithi115.stdout:1594: snap_remove snap 165 2022-04-22T17:31:26.553 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[53813]: debug 2022-04-22T17:31:26.246+0000 7f5d1ad073c0 -1 osd.2 378 log_to_monitors true 2022-04-22T17:31:26.811 INFO:tasks.rados.rados.0.smithi115.stdout:1594: done (0 left) 2022-04-22T17:31:26.811 INFO:tasks.rados.rados.0.smithi115.stdout:1595: rollback oid 7 current snap is 166 2022-04-22T17:31:26.812 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 7 to 159 2022-04-22T17:31:26.812 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 166 2022-04-22T17:31:26.812 INFO:tasks.rados.rados.0.smithi115.stdout:1596: seq_num 477 ranges {2056192=794624} 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:1596: writing smithi11541303-31 from 2056192 to 2850816 tid 1 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 166 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:1595: finishing rollback tid 0 to smithi11541303-7 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:1595: finishing rollback tid 1 to smithi11541303-7 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:1595: finishing rollback tid 2 to smithi11541303-7 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 650 (ObjNum 442 snap 148 seq_num 442) dirty exists 2022-04-22T17:31:26.832 INFO:tasks.rados.rados.0.smithi115.stdout:1597: seq_num 478 ranges {475136=409600,884736=401408} 2022-04-22T17:31:26.837 INFO:tasks.rados.rados.0.smithi115.stdout:1597: writing smithi11541303-50 from 475136 to 884736 tid 1 2022-04-22T17:31:26.840 INFO:tasks.rados.rados.0.smithi115.stdout:1597: writing smithi11541303-50 from 884736 to 1286144 tid 2 2022-04-22T17:31:26.841 INFO:tasks.rados.rados.0.smithi115.stdout:1595: done (2 left) 2022-04-22T17:31:26.841 INFO:tasks.rados.rados.0.smithi115.stdout:1598: read oid 2 snap -1 2022-04-22T17:31:26.841 INFO:tasks.rados.rados.0.smithi115.stdout:1598: expect (ObjNum 443 snap 148 seq_num 443) 2022-04-22T17:31:26.841 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 166 2022-04-22T17:31:26.841 INFO:tasks.rados.rados.0.smithi115.stdout:1599: seq_num 479 ranges {0=425984,425984=401408} 2022-04-22T17:31:26.844 INFO:tasks.rados.rados.0.smithi115.stdout:1599: writing smithi11541303-28 from 0 to 425984 tid 1 2022-04-22T17:31:26.847 INFO:tasks.rados.rados.0.smithi115.stdout:1599: writing smithi11541303-28 from 425984 to 827392 tid 2 2022-04-22T17:31:26.847 INFO:tasks.rados.rados.0.smithi115.stdout:1600: snap_remove snap 151 2022-04-22T17:31:27.053 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[53813]: debug 2022-04-22T17:31:26.838+0000 7f5d0e0a6700 -1 osd.2 378 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:31:27.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[45495]: cluster 2022-04-22T17:31:25.198360+0000 mgr.y (mgr.34107) 173 : cluster [DBG] pgmap v271: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 24 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:27.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[45495]: cluster 2022-04-22T17:31:25.789709+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e390: 8 total, 7 up, 8 in 2022-04-22T17:31:27.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[45495]: audit 2022-04-22T17:31:26.253712+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[45495]: audit 2022-04-22T17:31:26.254129+0000 mon.a (mon.0) 541 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[46934]: cluster 2022-04-22T17:31:25.198360+0000 mgr.y (mgr.34107) 173 : cluster [DBG] pgmap v271: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 24 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:27.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[46934]: cluster 2022-04-22T17:31:25.789709+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e390: 8 total, 7 up, 8 in 2022-04-22T17:31:27.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[46934]: audit 2022-04-22T17:31:26.253712+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:26 smithi115 bash[46934]: audit 2022-04-22T17:31:26.254129+0000 mon.a (mon.0) 541 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:26 smithi121 bash[36902]: cluster 2022-04-22T17:31:25.198360+0000 mgr.y (mgr.34107) 173 : cluster [DBG] pgmap v271: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 24 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:26 smithi121 bash[36902]: cluster 2022-04-22T17:31:25.789709+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e390: 8 total, 7 up, 8 in 2022-04-22T17:31:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:26 smithi121 bash[36902]: audit 2022-04-22T17:31:26.253712+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:26 smithi121 bash[36902]: audit 2022-04-22T17:31:26.254129+0000 mon.a (mon.0) 541 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-04-22T17:31:27.829 INFO:tasks.rados.rados.0.smithi115.stdout:1600: done (4 left) 2022-04-22T17:31:27.829 INFO:tasks.rados.rados.0.smithi115.stdout:1601: snap_create 2022-04-22T17:31:27.829 INFO:tasks.rados.rados.0.smithi115.stdout:1596: finishing write tid 1 to smithi11541303-31 2022-04-22T17:31:27.830 INFO:tasks.rados.rados.0.smithi115.stdout:1596: finishing write tid 2 to smithi11541303-31 2022-04-22T17:31:27.830 INFO:tasks.rados.rados.0.smithi115.stdout:1598: expect (ObjNum 443 snap 148 seq_num 443) 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1597: finishing write tid 1 to smithi11541303-50 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1599: finishing write tid 1 to smithi11541303-28 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1597: finishing write tid 2 to smithi11541303-50 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1599: finishing write tid 2 to smithi11541303-28 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1597: finishing write tid 3 to smithi11541303-50 2022-04-22T17:31:27.943 INFO:tasks.rados.rados.0.smithi115.stdout:1599: finishing write tid 3 to smithi11541303-28 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1597: finishing write tid 4 to smithi11541303-50 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 707 (ObjNum 478 snap 166 seq_num 478) dirty exists 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1597: left oid 50 (ObjNum 478 snap 166 seq_num 478) 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1596: finishing write tid 3 to smithi11541303-31 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 981 (ObjNum 477 snap 166 seq_num 477) dirty exists 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1596: left oid 31 (ObjNum 477 snap 166 seq_num 477) 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1599: finishing write tid 4 to smithi11541303-28 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 479 (ObjNum 479 snap 166 seq_num 479) dirty exists 2022-04-22T17:31:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:1599: left oid 28 (ObjNum 479 snap 166 seq_num 479) 2022-04-22T17:31:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:27 smithi121 bash[36902]: audit 2022-04-22T17:31:26.795244+0000 mon.a (mon.0) 542 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:31:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:27 smithi121 bash[36902]: cluster 2022-04-22T17:31:26.795306+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e391: 8 total, 7 up, 8 in 2022-04-22T17:31:28.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:27 smithi121 bash[36902]: audit 2022-04-22T17:31:26.802243+0000 mon.c (mon.1) 14 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:27 smithi121 bash[36902]: audit 2022-04-22T17:31:26.802933+0000 mon.a (mon.0) 544 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[45495]: audit 2022-04-22T17:31:26.795244+0000 mon.a (mon.0) 542 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[45495]: cluster 2022-04-22T17:31:26.795306+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e391: 8 total, 7 up, 8 in 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[45495]: audit 2022-04-22T17:31:26.802243+0000 mon.c (mon.1) 14 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[45495]: audit 2022-04-22T17:31:26.802933+0000 mon.a (mon.0) 544 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[46934]: audit 2022-04-22T17:31:26.795244+0000 mon.a (mon.0) 542 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[46934]: cluster 2022-04-22T17:31:26.795306+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e391: 8 total, 7 up, 8 in 2022-04-22T17:31:28.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[46934]: audit 2022-04-22T17:31:26.802243+0000 mon.c (mon.1) 14 : audit [INF] from='osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:27 smithi115 bash[46934]: audit 2022-04-22T17:31:26.802933+0000 mon.a (mon.0) 544 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:28.839 INFO:tasks.rados.rados.0.smithi115.stdout:1596: done (4 left) 2022-04-22T17:31:28.839 INFO:tasks.rados.rados.0.smithi115.stdout:1597: done (3 left) 2022-04-22T17:31:28.839 INFO:tasks.rados.rados.0.smithi115.stdout:1598: done (2 left) 2022-04-22T17:31:28.839 INFO:tasks.rados.rados.0.smithi115.stdout:1599: done (1 left) 2022-04-22T17:31:28.839 INFO:tasks.rados.rados.0.smithi115.stdout:1601: done (0 left) 2022-04-22T17:31:28.840 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 167 2022-04-22T17:31:28.840 INFO:tasks.rados.rados.0.smithi115.stdout:1602: seq_num 480 ranges {13508608=442368,13950976=401408} 2022-04-22T17:31:28.914 INFO:tasks.rados.rados.0.smithi115.stdout:1602: writing smithi11541303-42 from 13508608 to 13950976 tid 1 2022-04-22T17:31:28.917 INFO:tasks.rados.rados.0.smithi115.stdout:1602: writing smithi11541303-42 from 13950976 to 14352384 tid 2 2022-04-22T17:31:28.917 INFO:tasks.rados.rados.0.smithi115.stdout:1603: copy_from oid 22 from oid 18 current snap is 167 2022-04-22T17:31:28.917 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 167 2022-04-22T17:31:28.917 INFO:tasks.rados.rados.0.smithi115.stdout:1604: seq_num 481 ranges {2850816=458752,3309568=401408} 2022-04-22T17:31:28.937 INFO:tasks.rados.rados.0.smithi115.stdout:1604: writing smithi11541303-21 from 2850816 to 3309568 tid 1 2022-04-22T17:31:28.940 INFO:tasks.rados.rados.0.smithi115.stdout:1604: writing smithi11541303-21 from 3309568 to 3710976 tid 2 2022-04-22T17:31:28.940 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 167 2022-04-22T17:31:28.940 INFO:tasks.rados.rados.0.smithi115.stdout:1605: seq_num 482 ranges {983040=475136,1458176=401408} 2022-04-22T17:31:28.949 INFO:tasks.rados.rados.0.smithi115.stdout:1605: writing smithi11541303-15 from 983040 to 1458176 tid 1 2022-04-22T17:31:28.953 INFO:tasks.rados.rados.0.smithi115.stdout:1605: writing smithi11541303-15 from 1458176 to 1859584 tid 2 2022-04-22T17:31:28.953 INFO:tasks.rados.rados.0.smithi115.stdout:1606: read oid 38 snap -1 2022-04-22T17:31:28.953 INFO:tasks.rados.rados.0.smithi115.stdout:1606: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:28.953 INFO:tasks.rados.rados.0.smithi115.stdout:1607: snap_remove snap 154 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[45495]: cluster 2022-04-22T17:31:27.199249+0000 mgr.y (mgr.34107) 174 : cluster [DBG] pgmap v274: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 22 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[45495]: cluster 2022-04-22T17:31:27.794993+0000 mon.a (mon.0) 545 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[45495]: cluster 2022-04-22T17:31:27.812230+0000 mon.a (mon.0) 546 : cluster [INF] osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769] boot 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[45495]: cluster 2022-04-22T17:31:27.812294+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[45495]: audit 2022-04-22T17:31:27.812795+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:31:29.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[46934]: cluster 2022-04-22T17:31:27.199249+0000 mgr.y (mgr.34107) 174 : cluster [DBG] pgmap v274: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 22 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:29.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[46934]: cluster 2022-04-22T17:31:27.794993+0000 mon.a (mon.0) 545 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:29.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[46934]: cluster 2022-04-22T17:31:27.812230+0000 mon.a (mon.0) 546 : cluster [INF] osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769] boot 2022-04-22T17:31:29.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[46934]: cluster 2022-04-22T17:31:27.812294+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2022-04-22T17:31:29.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:28 smithi115 bash[46934]: audit 2022-04-22T17:31:27.812795+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:31:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:28 smithi121 bash[36902]: cluster 2022-04-22T17:31:27.199249+0000 mgr.y (mgr.34107) 174 : cluster [DBG] pgmap v274: 121 pgs: 10 active+undersized+degraded, 27 active+undersized, 84 active+clean; 155 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 22 op/s; 33/327 objects degraded (10.092%) 2022-04-22T17:31:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:28 smithi121 bash[36902]: cluster 2022-04-22T17:31:27.794993+0000 mon.a (mon.0) 545 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:28 smithi121 bash[36902]: cluster 2022-04-22T17:31:27.812230+0000 mon.a (mon.0) 546 : cluster [INF] osd.2 [v2:172.21.15.115:6818/712263769,v1:172.21.15.115:6819/712263769] boot 2022-04-22T17:31:29.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:28 smithi121 bash[36902]: cluster 2022-04-22T17:31:27.812294+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2022-04-22T17:31:29.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:28 smithi121 bash[36902]: audit 2022-04-22T17:31:27.812795+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-04-22T17:31:29.855 INFO:tasks.rados.rados.0.smithi115.stdout:1607: done (5 left) 2022-04-22T17:31:29.855 INFO:tasks.rados.rados.0.smithi115.stdout:1608: rmattr oid 48 current snap is 167 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1608: done (5 left) 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1609: rollback oid 30 current snap is 167 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 164 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1602: finishing write tid 1 to smithi11541303-42 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1602: finishing write tid 2 to smithi11541303-42 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1602: finishing write tid 3 to smithi11541303-42 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1602: finishing write tid 4 to smithi11541303-42 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 997 (ObjNum 480 snap 167 seq_num 480) dirty exists 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1602: left oid 42 (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:31:29.856 INFO:tasks.rados.rados.0.smithi115.stdout:1603: finishing copy_from to smithi11541303-22 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 978 (ObjNum 445 snap 148 seq_num 445) dirty exists 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1603: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1604: finishing write tid 1 to smithi11541303-21 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1604: finishing write tid 2 to smithi11541303-21 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1604: finishing write tid 3 to smithi11541303-21 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1604: finishing write tid 4 to smithi11541303-21 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 981 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1604: left oid 21 (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1605: finishing write tid 1 to smithi11541303-15 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1605: finishing write tid 2 to smithi11541303-15 2022-04-22T17:31:29.857 INFO:tasks.rados.rados.0.smithi115.stdout:1602: done (5 left) 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1603: done (4 left) 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1604: done (3 left) 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1610: read oid 13 snap -1 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1610: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 167 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1605: finishing write tid 3 to smithi11541303-15 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1605: finishing write tid 4 to smithi11541303-15 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 708 (ObjNum 482 snap 167 seq_num 482) dirty exists 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1605: left oid 15 (ObjNum 482 snap 167 seq_num 482) 2022-04-22T17:31:29.858 INFO:tasks.rados.rados.0.smithi115.stdout:1606: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:29.959 INFO:tasks.rados.rados.0.smithi115.stdout:1610: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:31:29.962 INFO:tasks.rados.rados.0.smithi115.stdout:1609: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:31:29.962 INFO:tasks.rados.rados.0.smithi115.stdout:1609: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:31:29.963 INFO:tasks.rados.rados.0.smithi115.stdout:1611: seq_num 483 ranges {0=491520,491520=401408} 2022-04-22T17:31:29.968 INFO:tasks.rados.rados.0.smithi115.stdout:1611: writing smithi11541303-8 from 0 to 491520 tid 1 2022-04-22T17:31:29.972 INFO:tasks.rados.rados.0.smithi115.stdout:1611: writing smithi11541303-8 from 491520 to 892928 tid 2 2022-04-22T17:31:29.972 INFO:tasks.rados.rados.0.smithi115.stdout:1605: done (4 left) 2022-04-22T17:31:29.972 INFO:tasks.rados.rados.0.smithi115.stdout:1606: done (3 left) 2022-04-22T17:31:29.973 INFO:tasks.rados.rados.0.smithi115.stdout:1610: done (2 left) 2022-04-22T17:31:29.973 INFO:tasks.rados.rados.0.smithi115.stdout:1612: snap_remove snap 166 2022-04-22T17:31:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:29 smithi121 bash[36902]: cluster 2022-04-22T17:31:28.812435+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2022-04-22T17:31:30.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:29 smithi121 bash[36902]: audit 2022-04-22T17:31:29.218385+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:30.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:29 smithi121 bash[36902]: cluster 2022-04-22T17:31:29.779181+0000 mon.a (mon.0) 551 : cluster [WRN] Health check update: Degraded data redundancy: 33/327 objects degraded (10.092%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:30.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[45495]: cluster 2022-04-22T17:31:28.812435+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2022-04-22T17:31:30.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[45495]: audit 2022-04-22T17:31:29.218385+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:30.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[45495]: cluster 2022-04-22T17:31:29.779181+0000 mon.a (mon.0) 551 : cluster [WRN] Health check update: Degraded data redundancy: 33/327 objects degraded (10.092%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:30.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[46934]: cluster 2022-04-22T17:31:28.812435+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2022-04-22T17:31:30.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[46934]: audit 2022-04-22T17:31:29.218385+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:30.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:29 smithi115 bash[46934]: cluster 2022-04-22T17:31:29.779181+0000 mon.a (mon.0) 551 : cluster [WRN] Health check update: Degraded data redundancy: 33/327 objects degraded (10.092%), 10 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:30.865 INFO:tasks.rados.rados.0.smithi115.stdout:1612: done (2 left) 2022-04-22T17:31:30.865 INFO:tasks.rados.rados.0.smithi115.stdout:1613: copy_from oid 13 from oid 38 current snap is 167 2022-04-22T17:31:30.865 INFO:tasks.rados.rados.0.smithi115.stdout:1614: snap_create 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1609: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 890 (ObjNum 447 snap 149 seq_num 447) dirty exists 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1611: finishing write tid 1 to smithi11541303-8 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1611: finishing write tid 2 to smithi11541303-8 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1611: finishing write tid 3 to smithi11541303-8 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1611: finishing write tid 4 to smithi11541303-8 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 653 (ObjNum 483 snap 167 seq_num 483) dirty exists 2022-04-22T17:31:30.866 INFO:tasks.rados.rados.0.smithi115.stdout:1611: left oid 8 (ObjNum 483 snap 167 seq_num 483) 2022-04-22T17:31:30.884 INFO:tasks.rados.rados.0.smithi115.stdout:1613: finishing copy_from to smithi11541303-13 2022-04-22T17:31:30.884 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 474 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:30.884 INFO:tasks.rados.rados.0.smithi115.stdout:1613: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:31:31.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:30 smithi115 bash[45495]: cluster 2022-04-22T17:31:29.200632+0000 mgr.y (mgr.34107) 175 : cluster [DBG] pgmap v277: 121 pgs: 3 peering, 7 active+undersized+degraded, 11 active+undersized, 100 active+clean; 158 MiB data, 887 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:31.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:30 smithi115 bash[45495]: cluster 2022-04-22T17:31:29.838164+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2022-04-22T17:31:31.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:30 smithi115 bash[46934]: cluster 2022-04-22T17:31:29.200632+0000 mgr.y (mgr.34107) 175 : cluster [DBG] pgmap v277: 121 pgs: 3 peering, 7 active+undersized+degraded, 11 active+undersized, 100 active+clean; 158 MiB data, 887 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:31.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:30 smithi115 bash[46934]: cluster 2022-04-22T17:31:29.838164+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2022-04-22T17:31:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:30 smithi121 bash[36902]: cluster 2022-04-22T17:31:29.200632+0000 mgr.y (mgr.34107) 175 : cluster [DBG] pgmap v277: 121 pgs: 3 peering, 7 active+undersized+degraded, 11 active+undersized, 100 active+clean; 158 MiB data, 887 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:30 smithi121 bash[36902]: cluster 2022-04-22T17:31:29.838164+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2022-04-22T17:31:31.878 INFO:tasks.rados.rados.0.smithi115.stdout:1609: done (3 left) 2022-04-22T17:31:31.878 INFO:tasks.rados.rados.0.smithi115.stdout:1611: done (2 left) 2022-04-22T17:31:31.878 INFO:tasks.rados.rados.0.smithi115.stdout:1613: done (1 left) 2022-04-22T17:31:31.878 INFO:tasks.rados.rados.0.smithi115.stdout:1614: done (0 left) 2022-04-22T17:31:31.878 INFO:tasks.rados.rados.0.smithi115.stdout:1615: delete oid 30 current snap is 168 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:1615: done (0 left) 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:1616: rollback oid 41 current snap is 168 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 159 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:1617: read oid 42 snap -1 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:1617: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 168 2022-04-22T17:31:31.883 INFO:tasks.rados.rados.0.smithi115.stdout:1618: seq_num 485 ranges {3252224=532480,3784704=401408} 2022-04-22T17:31:31.909 INFO:tasks.rados.rados.0.smithi115.stdout:1618: writing smithi11541303-35 from 3252224 to 3784704 tid 1 2022-04-22T17:31:31.912 INFO:tasks.rados.rados.0.smithi115.stdout:1618: writing smithi11541303-35 from 3784704 to 4186112 tid 2 2022-04-22T17:31:31.912 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 168 2022-04-22T17:31:31.912 INFO:tasks.rados.rados.0.smithi115.stdout:1616: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:31:31.912 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 675 (ObjNum -1116788576 snap 22075 seq_num 3179280032) dirty dne 2022-04-22T17:31:31.912 INFO:tasks.rados.rados.0.smithi115.stdout:1619: seq_num 486 ranges {0=548864,548864=401408} 2022-04-22T17:31:31.915 INFO:tasks.rados.rados.0.smithi115.stdout:1619: writing smithi11541303-29 from 0 to 548864 tid 1 2022-04-22T17:31:31.918 INFO:tasks.rados.rados.0.smithi115.stdout:1619: writing smithi11541303-29 from 548864 to 950272 tid 2 2022-04-22T17:31:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:1616: done (3 left) 2022-04-22T17:31:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:1620: read oid 41 snap -1 2022-04-22T17:31:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:1620: expect deleted 2022-04-22T17:31:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 168 2022-04-22T17:31:31.919 INFO:tasks.rados.rados.0.smithi115.stdout:1621: seq_num 487 ranges {1286144=565248,1851392=401408} 2022-04-22T17:31:31.930 INFO:tasks.rados.rados.0.smithi115.stdout:1621: writing smithi11541303-50 from 1286144 to 1851392 tid 1 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1621: writing smithi11541303-50 from 1851392 to 2252800 tid 2 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1622: setattr oid 22 current snap is 168 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1618: finishing write tid 1 to smithi11541303-35 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1618: finishing write tid 2 to smithi11541303-35 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1618: finishing write tid 3 to smithi11541303-35 2022-04-22T17:31:31.933 INFO:tasks.rados.rados.0.smithi115.stdout:1619: finishing write tid 1 to smithi11541303-29 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1619: finishing write tid 2 to smithi11541303-29 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1619: finishing write tid 3 to smithi11541303-29 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1618: finishing write tid 4 to smithi11541303-35 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 657 (ObjNum 485 snap 168 seq_num 485) dirty exists 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1618: left oid 35 (ObjNum 485 snap 168 seq_num 485) 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1619: finishing write tid 4 to smithi11541303-29 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 482 (ObjNum 486 snap 168 seq_num 486) dirty exists 2022-04-22T17:31:31.934 INFO:tasks.rados.rados.0.smithi115.stdout:1619: left oid 29 (ObjNum 486 snap 168 seq_num 486) 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1618: done (5 left) 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1619: done (4 left) 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1620: done (3 left) 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1623: rmattr oid 43 current snap is 168 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1623: done (3 left) 2022-04-22T17:31:31.935 INFO:tasks.rados.rados.0.smithi115.stdout:1624: snap_create 2022-04-22T17:31:31.938 INFO:tasks.rados.rados.0.smithi115.stdout:1621: finishing write tid 1 to smithi11541303-50 2022-04-22T17:31:31.939 INFO:tasks.rados.rados.0.smithi115.stdout:1621: finishing write tid 2 to smithi11541303-50 2022-04-22T17:31:31.941 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 982 (ObjNum 445 snap 148 seq_num 445) dirty exists 2022-04-22T17:31:31.941 INFO:tasks.rados.rados.0.smithi115.stdout:1621: finishing write tid 3 to smithi11541303-50 2022-04-22T17:31:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:1621: finishing write tid 4 to smithi11541303-50 2022-04-22T17:31:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 710 (ObjNum 487 snap 168 seq_num 487) dirty exists 2022-04-22T17:31:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:1621: left oid 50 (ObjNum 487 snap 168 seq_num 487) 2022-04-22T17:31:31.968 INFO:tasks.rados.rados.0.smithi115.stdout:1617: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:31:32.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:31.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:32.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:31.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:32.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:31.623Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:31:32.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:31 smithi115 bash[45495]: cluster 2022-04-22T17:31:30.847285+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2022-04-22T17:31:32.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:31 smithi115 bash[46934]: cluster 2022-04-22T17:31:30.847285+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2022-04-22T17:31:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:31 smithi121 bash[36902]: cluster 2022-04-22T17:31:30.847285+0000 mon.a (mon.0) 553 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2022-04-22T17:31:33.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:32 smithi115 bash[45495]: cluster 2022-04-22T17:31:31.201729+0000 mgr.y (mgr.34107) 176 : cluster [DBG] pgmap v280: 121 pgs: 3 peering, 7 active+undersized+degraded, 6 active+undersized, 105 active+clean; 158 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:33.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:32 smithi115 bash[45495]: cluster 2022-04-22T17:31:31.861212+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2022-04-22T17:31:33.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:32 smithi115 bash[46934]: cluster 2022-04-22T17:31:31.201729+0000 mgr.y (mgr.34107) 176 : cluster [DBG] pgmap v280: 121 pgs: 3 peering, 7 active+undersized+degraded, 6 active+undersized, 105 active+clean; 158 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:33.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:32 smithi115 bash[46934]: cluster 2022-04-22T17:31:31.861212+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2022-04-22T17:31:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:32 smithi121 bash[36902]: cluster 2022-04-22T17:31:31.201729+0000 mgr.y (mgr.34107) 176 : cluster [DBG] pgmap v280: 121 pgs: 3 peering, 7 active+undersized+degraded, 6 active+undersized, 105 active+clean; 158 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 812 KiB/s wr, 3 op/s; 22/339 objects degraded (6.490%); 482 KiB/s, 0 objects/s recovering 2022-04-22T17:31:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:32 smithi121 bash[36902]: cluster 2022-04-22T17:31:31.861212+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2022-04-22T17:31:33.480 INFO:tasks.rados.rados.0.smithi115.stdout:1617: done (3 left) 2022-04-22T17:31:33.480 INFO:tasks.rados.rados.0.smithi115.stdout:1621: done (2 left) 2022-04-22T17:31:33.481 INFO:tasks.rados.rados.0.smithi115.stdout:1622: done (1 left) 2022-04-22T17:31:33.481 INFO:tasks.rados.rados.0.smithi115.stdout:1624: done (0 left) 2022-04-22T17:31:33.481 INFO:tasks.rados.rados.0.smithi115.stdout:1625: snap_remove snap 161 2022-04-22T17:31:33.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:33.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:33.804 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:33.518Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:33.804 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:33.518Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:31:34.099 INFO:tasks.rados.rados.0.smithi115.stdout:1625: done (0 left) 2022-04-22T17:31:34.100 INFO:tasks.rados.rados.0.smithi115.stdout:1626: delete oid 49 current snap is 169 2022-04-22T17:31:34.117 INFO:tasks.rados.rados.0.smithi115.stdout:1626: done (0 left) 2022-04-22T17:31:34.117 INFO:tasks.rados.rados.0.smithi115.stdout:1627: delete oid 40 current snap is 169 2022-04-22T17:31:34.122 INFO:tasks.rados.rados.0.smithi115.stdout:1627: done (0 left) 2022-04-22T17:31:34.122 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 169 2022-04-22T17:31:34.123 INFO:tasks.rados.rados.0.smithi115.stdout:1628: seq_num 490 ranges {1859584=614400,2473984=401408} 2022-04-22T17:31:34.139 INFO:tasks.rados.rados.0.smithi115.stdout:1628: writing smithi11541303-15 from 1859584 to 2473984 tid 1 2022-04-22T17:31:34.142 INFO:tasks.rados.rados.0.smithi115.stdout:1628: writing smithi11541303-15 from 2473984 to 2875392 tid 2 2022-04-22T17:31:34.142 INFO:tasks.rados.rados.0.smithi115.stdout:1629: delete oid 41 current snap is 169 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1628: finishing write tid 1 to smithi11541303-15 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1629: done (1 left) 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1630: read oid 43 snap -1 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1630: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1631: read oid 50 snap -1 2022-04-22T17:31:34.146 INFO:tasks.rados.rados.0.smithi115.stdout:1631: expect (ObjNum 487 snap 168 seq_num 487) 2022-04-22T17:31:34.147 INFO:tasks.rados.rados.0.smithi115.stdout:1632: delete oid 28 current snap is 169 2022-04-22T17:31:34.147 INFO:tasks.rados.rados.0.smithi115.stdout:1628: finishing write tid 2 to smithi11541303-15 2022-04-22T17:31:34.147 INFO:tasks.rados.rados.0.smithi115.stdout:1628: finishing write tid 3 to smithi11541303-15 2022-04-22T17:31:34.150 INFO:tasks.rados.rados.0.smithi115.stdout:1632: done (3 left) 2022-04-22T17:31:34.150 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 169 2022-04-22T17:31:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:1630: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:31:34.156 INFO:tasks.rados.rados.0.smithi115.stdout:1628: finishing write tid 4 to smithi11541303-15 2022-04-22T17:31:34.156 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 711 (ObjNum 490 snap 169 seq_num 490) dirty exists 2022-04-22T17:31:34.156 INFO:tasks.rados.rados.0.smithi115.stdout:1628: left oid 15 (ObjNum 490 snap 169 seq_num 490) 2022-04-22T17:31:34.156 INFO:tasks.rados.rados.0.smithi115.stdout:1633: seq_num 493 ranges {5087232=663552,5750784=401408} 2022-04-22T17:31:34.189 INFO:tasks.rados.rados.0.smithi115.stdout:1633: writing smithi11541303-6 from 5087232 to 5750784 tid 1 2022-04-22T17:31:34.191 INFO:tasks.rados.rados.0.smithi115.stdout:1633: writing smithi11541303-6 from 5750784 to 6152192 tid 2 2022-04-22T17:31:34.192 INFO:tasks.rados.rados.0.smithi115.stdout:1628: done (3 left) 2022-04-22T17:31:34.192 INFO:tasks.rados.rados.0.smithi115.stdout:1630: done (2 left) 2022-04-22T17:31:34.192 INFO:tasks.rados.rados.0.smithi115.stdout:1634: snap_remove snap 164 2022-04-22T17:31:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: cluster 2022-04-22T17:31:32.867043+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2022-04-22T17:31:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: cephadm 2022-04-22T17:31:33.059987+0000 mgr.y (mgr.34107) 177 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.068357+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.073310+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.079521+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.081944+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.083202+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.084226+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:33 smithi121 bash[36902]: audit 2022-04-22T17:31:33.085620+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: cluster 2022-04-22T17:31:32.867043+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: cephadm 2022-04-22T17:31:33.059987+0000 mgr.y (mgr.34107) 177 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.068357+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.073310+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.079521+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.081944+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.083202+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.084226+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[46934]: audit 2022-04-22T17:31:33.085620+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: cluster 2022-04-22T17:31:32.867043+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: cephadm 2022-04-22T17:31:33.059987+0000 mgr.y (mgr.34107) 177 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.068357+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.073310+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.079521+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.081944+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.083202+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.084226+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:33 smithi115 bash[45495]: audit 2022-04-22T17:31:33.085620+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:34.805 INFO:tasks.rados.rados.0.smithi115.stdout:1634: done (2 left) 2022-04-22T17:31:34.808 INFO:tasks.rados.rados.0.smithi115.stdout:1635: read oid 49 snap -1 2022-04-22T17:31:34.808 INFO:tasks.rados.rados.0.smithi115.stdout:1635: expect deleted 2022-04-22T17:31:34.808 INFO:tasks.rados.rados.0.smithi115.stdout:1636: delete oid 11 current snap is 169 2022-04-22T17:31:34.808 INFO:tasks.rados.rados.0.smithi115.stdout:1631: expect (ObjNum 487 snap 168 seq_num 487) 2022-04-22T17:31:34.912 INFO:tasks.rados.rados.0.smithi115.stdout:1633: finishing write tid 1 to smithi11541303-6 2022-04-22T17:31:34.912 INFO:tasks.rados.rados.0.smithi115.stdout:1633: finishing write tid 2 to smithi11541303-6 2022-04-22T17:31:34.912 INFO:tasks.rados.rados.0.smithi115.stdout:1633: finishing write tid 3 to smithi11541303-6 2022-04-22T17:31:34.912 INFO:tasks.rados.rados.0.smithi115.stdout:1633: finishing write tid 4 to smithi11541303-6 2022-04-22T17:31:34.912 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 649 (ObjNum 493 snap 169 seq_num 493) dirty exists 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1633: left oid 6 (ObjNum 493 snap 169 seq_num 493) 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1631: done (3 left) 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1633: done (2 left) 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1636: done (1 left) 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1637: read oid 36 snap 157 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1637: expect deleted 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1638: read oid 32 snap 159 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1638: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:31:34.913 INFO:tasks.rados.rados.0.smithi115.stdout:1639: rmattr oid 8 current snap is 169 2022-04-22T17:31:34.914 INFO:tasks.rados.rados.0.smithi115.stdout:1639: done (3 left) 2022-04-22T17:31:34.914 INFO:tasks.rados.rados.0.smithi115.stdout:1640: snap_remove snap 167 2022-04-22T17:31:34.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: audit 2022-04-22T17:31:33.085882+0000 mgr.y (mgr.34107) 178 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:34.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cephadm 2022-04-22T17:31:33.087063+0000 mgr.y (mgr.34107) 179 : cephadm [INF] Upgrade: osd.3 is safe to restart 2022-04-22T17:31:34.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cluster 2022-04-22T17:31:33.202583+0000 mgr.y (mgr.34107) 180 : cluster [DBG] pgmap v283: 121 pgs: 121 active+clean; 168 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 9 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:34.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cluster 2022-04-22T17:31:34.071444+0000 mon.a (mon.0) 563 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/339 objects degraded (6.490%), 7 pgs degraded) 2022-04-22T17:31:34.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cluster 2022-04-22T17:31:34.071567+0000 mon.a (mon.0) 564 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:34.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cluster 2022-04-22T17:31:34.083156+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2022-04-22T17:31:34.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: audit 2022-04-22T17:31:34.220975+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:34.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[46934]: cluster 2022-04-22T17:31:34.788396+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2022-04-22T17:31:34.984 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: audit 2022-04-22T17:31:33.085882+0000 mgr.y (mgr.34107) 178 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:34.984 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cephadm 2022-04-22T17:31:33.087063+0000 mgr.y (mgr.34107) 179 : cephadm [INF] Upgrade: osd.3 is safe to restart 2022-04-22T17:31:34.984 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cluster 2022-04-22T17:31:33.202583+0000 mgr.y (mgr.34107) 180 : cluster [DBG] pgmap v283: 121 pgs: 121 active+clean; 168 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 9 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:34.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cluster 2022-04-22T17:31:34.071444+0000 mon.a (mon.0) 563 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/339 objects degraded (6.490%), 7 pgs degraded) 2022-04-22T17:31:34.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cluster 2022-04-22T17:31:34.071567+0000 mon.a (mon.0) 564 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:34.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cluster 2022-04-22T17:31:34.083156+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2022-04-22T17:31:34.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: audit 2022-04-22T17:31:34.220975+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:34.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:34 smithi115 bash[45495]: cluster 2022-04-22T17:31:34.788396+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2022-04-22T17:31:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: audit 2022-04-22T17:31:33.085882+0000 mgr.y (mgr.34107) 178 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2022-04-22T17:31:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cephadm 2022-04-22T17:31:33.087063+0000 mgr.y (mgr.34107) 179 : cephadm [INF] Upgrade: osd.3 is safe to restart 2022-04-22T17:31:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cluster 2022-04-22T17:31:33.202583+0000 mgr.y (mgr.34107) 180 : cluster [DBG] pgmap v283: 121 pgs: 121 active+clean; 168 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 9 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cluster 2022-04-22T17:31:34.071444+0000 mon.a (mon.0) 563 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/339 objects degraded (6.490%), 7 pgs degraded) 2022-04-22T17:31:35.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cluster 2022-04-22T17:31:34.071567+0000 mon.a (mon.0) 564 : cluster [INF] Cluster is now healthy 2022-04-22T17:31:35.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cluster 2022-04-22T17:31:34.083156+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2022-04-22T17:31:35.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: audit 2022-04-22T17:31:34.220975+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:35.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:34 smithi121 bash[36902]: cluster 2022-04-22T17:31:34.788396+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2022-04-22T17:31:35.899 INFO:tasks.rados.rados.0.smithi115.stdout:1640: done (3 left) 2022-04-22T17:31:35.899 INFO:tasks.rados.rados.0.smithi115.stdout:1641: setattr oid 37 current snap is 169 2022-04-22T17:31:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 169 2022-04-22T17:31:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:1642: seq_num 495 ranges {1089536=696320,1785856=401408} 2022-04-22T17:31:35.923 INFO:tasks.rados.rados.0.smithi115.stdout:1642: writing smithi11541303-46 from 1089536 to 1785856 tid 1 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1642: writing smithi11541303-46 from 1785856 to 2187264 tid 2 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1643: copy_from oid 3 from oid 38 current snap is 169 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1635: done (5 left) 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1644: rollback oid 4 current snap is 169 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 157 2022-04-22T17:31:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1638: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 658 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1644: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1644: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1642: finishing write tid 1 to smithi11541303-46 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1642: finishing write tid 2 to smithi11541303-46 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1642: finishing write tid 3 to smithi11541303-46 2022-04-22T17:31:36.150 INFO:tasks.rados.rados.0.smithi115.stdout:1644: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 914 (ObjNum 453 snap 153 seq_num 453) dirty exists 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1642: finishing write tid 4 to smithi11541303-46 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 880 (ObjNum 495 snap 169 seq_num 495) dirty exists 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1642: left oid 46 (ObjNum 495 snap 169 seq_num 495) 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1643: finishing copy_from to smithi11541303-3 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 474 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1643: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1637: done (5 left) 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1638: done (4 left) 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1641: done (3 left) 2022-04-22T17:31:36.151 INFO:tasks.rados.rados.0.smithi115.stdout:1642: done (2 left) 2022-04-22T17:31:36.152 INFO:tasks.rados.rados.0.smithi115.stdout:1643: done (1 left) 2022-04-22T17:31:36.152 INFO:tasks.rados.rados.0.smithi115.stdout:1644: done (0 left) 2022-04-22T17:31:36.152 INFO:tasks.rados.rados.0.smithi115.stdout:1645: delete oid 7 current snap is 169 2022-04-22T17:31:36.154 INFO:tasks.rados.rados.0.smithi115.stdout:1645: done (0 left) 2022-04-22T17:31:36.154 INFO:tasks.rados.rados.0.smithi115.stdout:1646: copy_from oid 37 from oid 47 current snap is 169 2022-04-22T17:31:36.154 INFO:tasks.rados.rados.0.smithi115.stdout:1647: copy_from oid 17 from oid 36 current snap is 169 2022-04-22T17:31:36.154 INFO:tasks.rados.rados.0.smithi115.stdout:1648: read oid 27 snap -1 2022-04-22T17:31:36.154 INFO:tasks.rados.rados.0.smithi115.stdout:1648: expect deleted 2022-04-22T17:31:36.155 INFO:tasks.rados.rados.0.smithi115.stdout:1649: delete oid 23 current snap is 169 2022-04-22T17:31:36.156 INFO:tasks.rados.rados.0.smithi115.stdout:1647: finishing copy_from to smithi11541303-17 2022-04-22T17:31:36.156 INFO:tasks.rados.rados.0.smithi115.stdout:1647: got expected ENOENT (src dne) 2022-04-22T17:31:36.156 INFO:tasks.rados.rados.0.smithi115.stdout:1647: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:31:36.158 INFO:tasks.rados.rados.0.smithi115.stdout:1647: done (3 left) 2022-04-22T17:31:36.158 INFO:tasks.rados.rados.0.smithi115.stdout:1648: done (2 left) 2022-04-22T17:31:36.158 INFO:tasks.rados.rados.0.smithi115.stdout:1649: done (1 left) 2022-04-22T17:31:36.158 INFO:tasks.rados.rados.0.smithi115.stdout:1650: rmattr oid 17 current snap is 169 2022-04-22T17:31:36.158 INFO:tasks.rados.rados.0.smithi115.stdout:1651: read oid 15 snap -1 2022-04-22T17:31:36.159 INFO:tasks.rados.rados.0.smithi115.stdout:1651: expect (ObjNum 490 snap 169 seq_num 490) 2022-04-22T17:31:36.159 INFO:tasks.rados.rados.0.smithi115.stdout:1652: rollback oid 19 current snap is 169 2022-04-22T17:31:36.159 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 148 2022-04-22T17:31:36.159 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 169 2022-04-22T17:31:36.159 INFO:tasks.rados.rados.0.smithi115.stdout:1653: seq_num 498 ranges {2482176=745472,3227648=401408} 2022-04-22T17:31:36.177 INFO:tasks.rados.rados.0.smithi115.stdout:1653: writing smithi11541303-5 from 2482176 to 3227648 tid 1 2022-04-22T17:31:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:1653: writing smithi11541303-5 from 3227648 to 3629056 tid 2 2022-04-22T17:31:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:1654: snap_create 2022-04-22T17:31:36.179 INFO:tasks.rados.rados.0.smithi115.stdout:1652: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:31:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 751 (ObjNum 457 snap 157 seq_num 457) dirty exists 2022-04-22T17:31:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:1652: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:31:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:1652: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:31:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 558 (ObjNum 430 snap 147 seq_num 430) dirty exists 2022-04-22T17:31:36.180 INFO:tasks.rados.rados.0.smithi115.stdout:1651: expect (ObjNum 490 snap 169 seq_num 490) 2022-04-22T17:31:36.318 INFO:tasks.rados.rados.0.smithi115.stdout:1646: finishing copy_from to smithi11541303-37 2022-04-22T17:31:36.319 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 702 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:31:36.319 INFO:tasks.rados.rados.0.smithi115.stdout:1646: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:31:36.319 INFO:tasks.rados.rados.0.smithi115.stdout:1653: finishing write tid 1 to smithi11541303-5 2022-04-22T17:31:36.319 INFO:tasks.rados.rados.0.smithi115.stdout:1653: finishing write tid 2 to smithi11541303-5 2022-04-22T17:31:36.319 INFO:tasks.rados.rados.0.smithi115.stdout:1653: finishing write tid 3 to smithi11541303-5 2022-04-22T17:31:36.320 INFO:tasks.rados.rados.0.smithi115.stdout:1653: finishing write tid 4 to smithi11541303-5 2022-04-22T17:31:36.320 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1037 (ObjNum 498 snap 169 seq_num 498) dirty exists 2022-04-22T17:31:36.320 INFO:tasks.rados.rados.0.smithi115.stdout:1653: left oid 5 (ObjNum 498 snap 169 seq_num 498) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1646: done (5 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1650: done (4 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1651: done (3 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1652: done (2 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1653: done (1 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1654: done (0 left) 2022-04-22T17:31:36.899 INFO:tasks.rados.rados.0.smithi115.stdout:1655: snap_remove snap 168 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[46934]: cluster 2022-04-22T17:31:35.203355+0000 mgr.y (mgr.34107) 181 : cluster [DBG] pgmap v286: 121 pgs: 121 active+clean; 172 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[46934]: cephadm 2022-04-22T17:31:35.768451+0000 mgr.y (mgr.34107) 182 : cephadm [INF] Upgrade: Updating osd.3 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[46934]: audit 2022-04-22T17:31:35.775359+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[46934]: cephadm 2022-04-22T17:31:35.777842+0000 mgr.y (mgr.34107) 183 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[46934]: cluster 2022-04-22T17:31:35.881089+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2022-04-22T17:31:37.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[45495]: cluster 2022-04-22T17:31:35.203355+0000 mgr.y (mgr.34107) 181 : cluster [DBG] pgmap v286: 121 pgs: 121 active+clean; 172 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:37.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[45495]: cephadm 2022-04-22T17:31:35.768451+0000 mgr.y (mgr.34107) 182 : cephadm [INF] Upgrade: Updating osd.3 2022-04-22T17:31:37.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[45495]: audit 2022-04-22T17:31:35.775359+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:37.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[45495]: cephadm 2022-04-22T17:31:35.777842+0000 mgr.y (mgr.34107) 183 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:31:37.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:36 smithi115 bash[45495]: cluster 2022-04-22T17:31:35.881089+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2022-04-22T17:31:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:36 smithi121 bash[36902]: cluster 2022-04-22T17:31:35.203355+0000 mgr.y (mgr.34107) 181 : cluster [DBG] pgmap v286: 121 pgs: 121 active+clean; 172 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 4.8 MiB/s, 1 objects/s recovering 2022-04-22T17:31:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:36 smithi121 bash[36902]: cephadm 2022-04-22T17:31:35.768451+0000 mgr.y (mgr.34107) 182 : cephadm [INF] Upgrade: Updating osd.3 2022-04-22T17:31:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:36 smithi121 bash[36902]: audit 2022-04-22T17:31:35.775359+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:36 smithi121 bash[36902]: cephadm 2022-04-22T17:31:35.777842+0000 mgr.y (mgr.34107) 183 : cephadm [INF] Deploying daemon osd.3 on smithi115 2022-04-22T17:31:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:36 smithi121 bash[36902]: cluster 2022-04-22T17:31:35.881089+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2022-04-22T17:31:37.908 INFO:tasks.rados.rados.0.smithi115.stdout:1655: done (0 left) 2022-04-22T17:31:37.909 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 170 2022-04-22T17:31:37.909 INFO:tasks.rados.rados.0.smithi115.stdout:1656: seq_num 499 ranges {0=761856,761856=401408} 2022-04-22T17:31:37.916 INFO:tasks.rados.rados.0.smithi115.stdout:1656: writing smithi11541303-49 from 0 to 761856 tid 1 2022-04-22T17:31:37.919 INFO:tasks.rados.rados.0.smithi115.stdout:1656: writing smithi11541303-49 from 761856 to 1163264 tid 2 2022-04-22T17:31:37.919 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 170 2022-04-22T17:31:37.919 INFO:tasks.rados.rados.0.smithi115.stdout:1657: seq_num 500 ranges {409600=778240,1187840=401408} 2022-04-22T17:31:37.927 INFO:tasks.rados.rados.0.smithi115.stdout:1657: writing smithi11541303-25 from 409600 to 1187840 tid 1 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1657: writing smithi11541303-25 from 1187840 to 1589248 tid 2 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1658: copy_from oid 30 from oid 13 current snap is 170 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1656: finishing write tid 1 to smithi11541303-49 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1656: finishing write tid 2 to smithi11541303-49 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1656: finishing write tid 3 to smithi11541303-49 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1659: read oid 1 snap -1 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1659: expect deleted 2022-04-22T17:31:37.930 INFO:tasks.rados.rados.0.smithi115.stdout:1660: setattr oid 17 current snap is 170 2022-04-22T17:31:37.931 INFO:tasks.rados.rados.0.smithi115.stdout:1656: finishing write tid 4 to smithi11541303-49 2022-04-22T17:31:37.931 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1044 (ObjNum 499 snap 170 seq_num 499) dirty exists 2022-04-22T17:31:37.931 INFO:tasks.rados.rados.0.smithi115.stdout:1656: left oid 49 (ObjNum 499 snap 170 seq_num 499) 2022-04-22T17:31:37.932 INFO:tasks.rados.rados.0.smithi115.stdout:1656: done (4 left) 2022-04-22T17:31:37.932 INFO:tasks.rados.rados.0.smithi115.stdout:1659: done (3 left) 2022-04-22T17:31:37.932 INFO:tasks.rados.rados.0.smithi115.stdout:1661: read oid 39 snap -1 2022-04-22T17:31:37.932 INFO:tasks.rados.rados.0.smithi115.stdout:1661: expect deleted 2022-04-22T17:31:37.932 INFO:tasks.rados.rados.0.smithi115.stdout:1662: delete oid 34 current snap is 170 2022-04-22T17:31:37.933 INFO:tasks.rados.rados.0.smithi115.stdout:1661: done (4 left) 2022-04-22T17:31:37.934 INFO:tasks.rados.rados.0.smithi115.stdout:1662: done (3 left) 2022-04-22T17:31:37.934 INFO:tasks.rados.rados.0.smithi115.stdout:1663: snap_remove snap 159 2022-04-22T17:31:38.209 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:37 smithi115 bash[45495]: cluster 2022-04-22T17:31:36.882043+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2022-04-22T17:31:38.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:37 smithi115 bash[46934]: cluster 2022-04-22T17:31:36.882043+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2022-04-22T17:31:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:37 smithi121 bash[36902]: cluster 2022-04-22T17:31:36.882043+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2022-04-22T17:31:38.917 INFO:tasks.rados.rados.0.smithi115.stdout:1663: done (3 left) 2022-04-22T17:31:38.917 INFO:tasks.rados.rados.0.smithi115.stdout:1664: rollback oid 31 current snap is 170 2022-04-22T17:31:38.918 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 158 2022-04-22T17:31:38.918 INFO:tasks.rados.rados.0.smithi115.stdout:1657: finishing write tid 1 to smithi11541303-25 2022-04-22T17:31:38.918 INFO:tasks.rados.rados.0.smithi115.stdout:1657: finishing write tid 2 to smithi11541303-25 2022-04-22T17:31:38.918 INFO:tasks.rados.rados.0.smithi115.stdout:1657: finishing write tid 3 to smithi11541303-25 2022-04-22T17:31:38.919 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 759 (ObjNum 457 snap 157 seq_num 457) dirty exists 2022-04-22T17:31:38.919 INFO:tasks.rados.rados.0.smithi115.stdout:1657: finishing write tid 4 to smithi11541303-25 2022-04-22T17:31:38.919 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 757 (ObjNum 500 snap 170 seq_num 500) dirty exists 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1657: left oid 25 (ObjNum 500 snap 170 seq_num 500) 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1658: finishing copy_from to smithi11541303-30 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 474 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1658: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1657: done (3 left) 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1658: done (2 left) 2022-04-22T17:31:38.920 INFO:tasks.rados.rados.0.smithi115.stdout:1660: done (1 left) 2022-04-22T17:31:38.921 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 170 2022-04-22T17:31:38.921 INFO:tasks.rados.rados.0.smithi115.stdout:1665: seq_num 502 ranges {950272=409600,1359872=417792,1777664=385024} 2022-04-22T17:31:38.928 INFO:tasks.rados.rados.0.smithi115.stdout:1665: writing smithi11541303-29 from 950272 to 1359872 tid 1 2022-04-22T17:31:38.931 INFO:tasks.rados.rados.0.smithi115.stdout:1665: writing smithi11541303-29 from 1359872 to 1777664 tid 2 2022-04-22T17:31:38.934 INFO:tasks.rados.rados.0.smithi115.stdout:1665: writing smithi11541303-29 from 1777664 to 2162688 tid 3 2022-04-22T17:31:38.934 INFO:tasks.rados.rados.0.smithi115.stdout:1666: delete oid 30 current snap is 170 2022-04-22T17:31:38.934 INFO:tasks.rados.rados.0.smithi115.stdout:1664: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:31:38.942 INFO:tasks.rados.rados.0.smithi115.stdout:1664: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:31:38.942 INFO:tasks.rados.rados.0.smithi115.stdout:1666: done (2 left) 2022-04-22T17:31:38.942 INFO:tasks.rados.rados.0.smithi115.stdout:1667: delete oid 24 current snap is 170 2022-04-22T17:31:38.943 INFO:tasks.rados.rados.0.smithi115.stdout:1664: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:31:38.943 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1048 (ObjNum 463 snap 158 seq_num 463) dirty exists 2022-04-22T17:31:38.944 INFO:tasks.rados.rados.0.smithi115.stdout:1665: finishing write tid 1 to smithi11541303-29 2022-04-22T17:31:38.944 INFO:tasks.rados.rados.0.smithi115.stdout:1665: finishing write tid 2 to smithi11541303-29 2022-04-22T17:31:38.945 INFO:tasks.rados.rados.0.smithi115.stdout:1665: finishing write tid 3 to smithi11541303-29 2022-04-22T17:31:38.945 INFO:tasks.rados.rados.0.smithi115.stdout:1665: finishing write tid 4 to smithi11541303-29 2022-04-22T17:31:38.946 INFO:tasks.rados.rados.0.smithi115.stdout:1664: done (2 left) 2022-04-22T17:31:38.946 INFO:tasks.rados.rados.0.smithi115.stdout:1667: done (1 left) 2022-04-22T17:31:38.946 INFO:tasks.rados.rados.0.smithi115.stdout:1668: read oid 28 snap -1 2022-04-22T17:31:38.946 INFO:tasks.rados.rados.0.smithi115.stdout:1668: expect deleted 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:1669: rollback oid 32 current snap is 170 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 158 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:1670: read oid 34 snap -1 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:1670: expect deleted 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 170 2022-04-22T17:31:38.947 INFO:tasks.rados.rados.0.smithi115.stdout:1671: seq_num 505 ranges {5963776=466944,6430720=475136,6905856=327680} 2022-04-22T17:31:38.983 INFO:tasks.rados.rados.0.smithi115.stdout:1671: writing smithi11541303-20 from 5963776 to 6430720 tid 1 2022-04-22T17:31:38.999 INFO:tasks.rados.rados.0.smithi115.stdout:1671: writing smithi11541303-20 from 6430720 to 6905856 tid 2 2022-04-22T17:31:38.999 INFO:tasks.rados.rados.0.smithi115.stdout:1671: writing smithi11541303-20 from 6905856 to 7233536 tid 3 2022-04-22T17:31:38.999 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 170 2022-04-22T17:31:38.999 INFO:tasks.rados.rados.0.smithi115.stdout:1672: seq_num 506 ranges {2777088=483328,3260416=761856,4022272=40960} 2022-04-22T17:31:39.006 INFO:tasks.rados.rados.0.smithi115.stdout:1672: writing smithi11541303-45 from 2777088 to 3260416 tid 1 2022-04-22T17:31:39.010 INFO:tasks.rados.rados.0.smithi115.stdout:1672: writing smithi11541303-45 from 3260416 to 4022272 tid 2 2022-04-22T17:31:39.011 INFO:tasks.rados.rados.0.smithi115.stdout:1672: writing smithi11541303-45 from 4022272 to 4063232 tid 3 2022-04-22T17:31:39.011 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 170 2022-04-22T17:31:39.011 INFO:tasks.rados.rados.0.smithi115.stdout:1673: seq_num 507 ranges {925696=499712,1425408=638976,2064384=163840} 2022-04-22T17:31:39.019 INFO:tasks.rados.rados.0.smithi115.stdout:1673: writing smithi11541303-9 from 925696 to 1425408 tid 1 2022-04-22T17:31:39.023 INFO:tasks.rados.rados.0.smithi115.stdout:1673: writing smithi11541303-9 from 1425408 to 2064384 tid 2 2022-04-22T17:31:39.024 INFO:tasks.rados.rados.0.smithi115.stdout:1673: writing smithi11541303-9 from 2064384 to 2228224 tid 3 2022-04-22T17:31:39.024 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 170 2022-04-22T17:31:39.024 INFO:tasks.rados.rados.0.smithi115.stdout:1674: seq_num 508 ranges {2482176=516096,2998272=524288,3522560=278528} 2022-04-22T17:31:39.041 INFO:tasks.rados.rados.0.smithi115.stdout:1674: writing smithi11541303-2 from 2482176 to 2998272 tid 1 2022-04-22T17:31:39.045 INFO:tasks.rados.rados.0.smithi115.stdout:1674: writing smithi11541303-2 from 2998272 to 3522560 tid 2 2022-04-22T17:31:39.046 INFO:tasks.rados.rados.0.smithi115.stdout:1674: writing smithi11541303-2 from 3522560 to 3801088 tid 3 2022-04-22T17:31:39.047 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 170 2022-04-22T17:31:39.047 INFO:tasks.rados.rados.0.smithi115.stdout:1675: seq_num 509 ranges {4186112=532480,4718592=409600,5128192=393216} 2022-04-22T17:31:39.073 INFO:tasks.rados.rados.0.smithi115.stdout:1675: writing smithi11541303-35 from 4186112 to 4718592 tid 1 2022-04-22T17:31:39.075 INFO:tasks.rados.rados.0.smithi115.stdout:1675: writing smithi11541303-35 from 4718592 to 5128192 tid 2 2022-04-22T17:31:39.079 INFO:tasks.rados.rados.0.smithi115.stdout:1675: writing smithi11541303-35 from 5128192 to 5521408 tid 3 2022-04-22T17:31:39.079 INFO:tasks.rados.rados.0.smithi115.stdout:1676: delete oid 19 current snap is 170 2022-04-22T17:31:39.080 INFO:tasks.rados.rados.0.smithi115.stdout:1665: finishing write tid 5 to smithi11541303-29 2022-04-22T17:31:39.080 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 487 (ObjNum 502 snap 170 seq_num 502) dirty exists 2022-04-22T17:31:39.080 INFO:tasks.rados.rados.0.smithi115.stdout:1665: left oid 29 (ObjNum 502 snap 170 seq_num 502) 2022-04-22T17:31:39.080 INFO:tasks.rados.rados.0.smithi115.stdout:1669: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1669: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1669: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 560 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1671: finishing write tid 1 to smithi11541303-20 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1671: finishing write tid 2 to smithi11541303-20 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1671: finishing write tid 3 to smithi11541303-20 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1671: finishing write tid 4 to smithi11541303-20 2022-04-22T17:31:39.082 INFO:tasks.rados.rados.0.smithi115.stdout:1671: finishing write tid 5 to smithi11541303-20 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 770 (ObjNum 505 snap 170 seq_num 505) dirty exists 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1671: left oid 20 (ObjNum 505 snap 170 seq_num 505) 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1672: finishing write tid 1 to smithi11541303-45 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1672: finishing write tid 2 to smithi11541303-45 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1672: finishing write tid 3 to smithi11541303-45 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1672: finishing write tid 4 to smithi11541303-45 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:1672: finishing write tid 5 to smithi11541303-45 2022-04-22T17:31:39.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 931 (ObjNum 506 snap 170 seq_num 506) dirty exists 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1672: left oid 45 (ObjNum 506 snap 170 seq_num 506) 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: finishing write tid 1 to smithi11541303-9 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: finishing write tid 2 to smithi11541303-9 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: finishing write tid 3 to smithi11541303-9 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: finishing write tid 4 to smithi11541303-9 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: finishing write tid 5 to smithi11541303-9 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1065 (ObjNum 507 snap 170 seq_num 507) dirty exists 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1673: left oid 9 (ObjNum 507 snap 170 seq_num 507) 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1674: finishing write tid 1 to smithi11541303-2 2022-04-22T17:31:39.084 INFO:tasks.rados.rados.0.smithi115.stdout:1674: finishing write tid 2 to smithi11541303-2 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:1674: finishing write tid 3 to smithi11541303-2 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:1674: finishing write tid 4 to smithi11541303-2 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:1674: finishing write tid 5 to smithi11541303-2 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 706 (ObjNum 508 snap 170 seq_num 508) dirty exists 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:1674: left oid 2 (ObjNum 508 snap 170 seq_num 508) 2022-04-22T17:31:39.085 INFO:tasks.rados.rados.0.smithi115.stdout:1675: finishing write tid 1 to smithi11541303-35 2022-04-22T17:31:39.086 INFO:tasks.rados.rados.0.smithi115.stdout:1675: finishing write tid 2 to smithi11541303-35 2022-04-22T17:31:39.086 INFO:tasks.rados.rados.0.smithi115.stdout:1665: done (9 left) 2022-04-22T17:31:39.086 INFO:tasks.rados.rados.0.smithi115.stdout:1668: done (8 left) 2022-04-22T17:31:39.086 INFO:tasks.rados.rados.0.smithi115.stdout:1669: done (7 left) 2022-04-22T17:31:39.086 INFO:tasks.rados.rados.0.smithi115.stdout:1670: done (6 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1671: done (5 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1672: done (4 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1673: done (3 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1674: done (2 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1676: done (1 left) 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1677: rollback oid 49 current snap is 170 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 157 2022-04-22T17:31:39.087 INFO:tasks.rados.rados.0.smithi115.stdout:1678: snap_remove snap 148 2022-04-22T17:31:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:38 smithi121 bash[36902]: cluster 2022-04-22T17:31:37.204001+0000 mgr.y (mgr.34107) 184 : cluster [DBG] pgmap v289: 121 pgs: 121 active+clean; 178 MiB data, 910 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:31:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:38 smithi121 bash[36902]: cluster 2022-04-22T17:31:37.888846+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2022-04-22T17:31:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:38 smithi121 bash[36902]: audit 2022-04-22T17:31:38.306992+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:39.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[45495]: cluster 2022-04-22T17:31:37.204001+0000 mgr.y (mgr.34107) 184 : cluster [DBG] pgmap v289: 121 pgs: 121 active+clean; 178 MiB data, 910 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:31:39.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[45495]: cluster 2022-04-22T17:31:37.888846+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2022-04-22T17:31:39.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[45495]: audit 2022-04-22T17:31:38.306992+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:39.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[46934]: cluster 2022-04-22T17:31:37.204001+0000 mgr.y (mgr.34107) 184 : cluster [DBG] pgmap v289: 121 pgs: 121 active+clean; 178 MiB data, 910 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:31:39.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[46934]: cluster 2022-04-22T17:31:37.888846+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2022-04-22T17:31:39.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:38 smithi115 bash[46934]: audit 2022-04-22T17:31:38.306992+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1678: done (2 left) 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1679: copy_from oid 8 from oid 25 current snap is 170 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1675: finishing write tid 3 to smithi11541303-35 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1675: finishing write tid 4 to smithi11541303-35 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1675: finishing write tid 5 to smithi11541303-35 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 710 (ObjNum 509 snap 170 seq_num 509) dirty exists 2022-04-22T17:31:39.812 INFO:tasks.rados.rados.0.smithi115.stdout:1675: left oid 35 (ObjNum 509 snap 170 seq_num 509) 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1677: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1677: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1677: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1068 (ObjNum 426 snap 144 seq_num 426) dirty exists 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1675: done (2 left) 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1677: done (1 left) 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 170 2022-04-22T17:31:39.813 INFO:tasks.rados.rados.0.smithi115.stdout:1680: seq_num 511 ranges {2908160=565248,3473408=630784,4104192=172032} 2022-04-22T17:31:39.834 INFO:tasks.rados.rados.0.smithi115.stdout:1680: writing smithi11541303-37 from 2908160 to 3473408 tid 1 2022-04-22T17:31:39.839 INFO:tasks.rados.rados.0.smithi115.stdout:1680: writing smithi11541303-37 from 3473408 to 4104192 tid 2 2022-04-22T17:31:39.841 INFO:tasks.rados.rados.0.smithi115.stdout:1680: writing smithi11541303-37 from 4104192 to 4276224 tid 3 2022-04-22T17:31:39.841 INFO:tasks.rados.rados.0.smithi115.stdout:1681: rollback oid 12 current snap is 170 2022-04-22T17:31:39.841 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 157 2022-04-22T17:31:39.841 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 170 2022-04-22T17:31:39.841 INFO:tasks.rados.rados.0.smithi115.stdout:1682: seq_num 512 ranges {2908160=581632,3489792=516096,4005888=286720} 2022-04-22T17:31:39.861 INFO:tasks.rados.rados.0.smithi115.stdout:1682: writing smithi11541303-47 from 2908160 to 3489792 tid 1 2022-04-22T17:31:39.864 INFO:tasks.rados.rados.0.smithi115.stdout:1682: writing smithi11541303-47 from 3489792 to 4005888 tid 2 2022-04-22T17:31:39.866 INFO:tasks.rados.rados.0.smithi115.stdout:1682: writing smithi11541303-47 from 4005888 to 4292608 tid 3 2022-04-22T17:31:39.866 INFO:tasks.rados.rados.0.smithi115.stdout:1683: rollback oid 27 current snap is 170 2022-04-22T17:31:39.866 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 158 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1681: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1680: finishing write tid 1 to smithi11541303-37 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1681: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1681: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 570 (ObjNum 361 snap 123 seq_num 361) dirty exists 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1681: done (4 left) 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1684: setattr oid 7 current snap is 170 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1680: finishing write tid 2 to smithi11541303-37 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:1679: finishing copy_from to smithi11541303-8 2022-04-22T17:31:39.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 757 (ObjNum 500 snap 170 seq_num 500) dirty exists 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1679: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1680: finishing write tid 3 to smithi11541303-37 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1680: finishing write tid 4 to smithi11541303-37 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1680: finishing write tid 5 to smithi11541303-37 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 714 (ObjNum 511 snap 170 seq_num 511) dirty exists 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1680: left oid 37 (ObjNum 511 snap 170 seq_num 511) 2022-04-22T17:31:39.868 INFO:tasks.rados.rados.0.smithi115.stdout:1682: finishing write tid 1 to smithi11541303-47 2022-04-22T17:31:39.869 INFO:tasks.rados.rados.0.smithi115.stdout:1679: done (4 left) 2022-04-22T17:31:39.869 INFO:tasks.rados.rados.0.smithi115.stdout:1680: done (3 left) 2022-04-22T17:31:39.869 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 170 2022-04-22T17:31:39.869 INFO:tasks.rados.rados.0.smithi115.stdout:1685: seq_num 513 ranges {393216=598016,991232=794624,1785856=8192} 2022-04-22T17:31:39.873 INFO:tasks.rados.rados.0.smithi115.stdout:1685: writing smithi11541303-4 from 393216 to 991232 tid 1 2022-04-22T17:31:39.878 INFO:tasks.rados.rados.0.smithi115.stdout:1685: writing smithi11541303-4 from 991232 to 1785856 tid 2 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1685: writing smithi11541303-4 from 1785856 to 1794048 tid 3 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 170 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1683: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1683: finishing rollback tid 2 to smithi11541303-27 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 937 (ObjNum 461 snap 158 seq_num 461) dirty exists 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1682: finishing write tid 2 to smithi11541303-47 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1682: finishing write tid 3 to smithi11541303-47 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 758 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:31:39.879 INFO:tasks.rados.rados.0.smithi115.stdout:1682: finishing write tid 4 to smithi11541303-47 2022-04-22T17:31:39.880 INFO:tasks.rados.rados.0.smithi115.stdout:1682: finishing write tid 5 to smithi11541303-47 2022-04-22T17:31:39.880 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 719 (ObjNum 512 snap 170 seq_num 512) dirty exists 2022-04-22T17:31:39.880 INFO:tasks.rados.rados.0.smithi115.stdout:1682: left oid 47 (ObjNum 512 snap 170 seq_num 512) 2022-04-22T17:31:39.880 INFO:tasks.rados.rados.0.smithi115.stdout:1685: finishing write tid 1 to smithi11541303-4 2022-04-22T17:31:39.880 INFO:tasks.rados.rados.0.smithi115.stdout:1686: seq_num 514 ranges {2072576=614400,2686976=679936,3366912=122880} 2022-04-22T17:31:39.893 INFO:tasks.rados.rados.0.smithi115.stdout:1686: writing smithi11541303-12 from 2072576 to 2686976 tid 1 2022-04-22T17:31:39.897 INFO:tasks.rados.rados.0.smithi115.stdout:1686: writing smithi11541303-12 from 2686976 to 3366912 tid 2 2022-04-22T17:31:39.898 INFO:tasks.rados.rados.0.smithi115.stdout:1686: writing smithi11541303-12 from 3366912 to 3489792 tid 3 2022-04-22T17:31:39.898 INFO:tasks.rados.rados.0.smithi115.stdout:1682: done (4 left) 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1683: done (3 left) 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1684: done (2 left) 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1687: read oid 13 snap -1 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1687: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1688: read oid 50 snap -1 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1688: expect (ObjNum 487 snap 168 seq_num 487) 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1689: copy_from oid 23 from oid 25 current snap is 170 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 170 2022-04-22T17:31:39.899 INFO:tasks.rados.rados.0.smithi115.stdout:1690: seq_num 515 ranges {1966080=630784,2596864=565248,3162112=237568} 2022-04-22T17:31:39.914 INFO:tasks.rados.rados.0.smithi115.stdout:1690: writing smithi11541303-22 from 1966080 to 2596864 tid 1 2022-04-22T17:31:39.918 INFO:tasks.rados.rados.0.smithi115.stdout:1690: writing smithi11541303-22 from 2596864 to 3162112 tid 2 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1690: writing smithi11541303-22 from 3162112 to 3399680 tid 3 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1691: delete oid 47 current snap is 170 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1685: finishing write tid 2 to smithi11541303-4 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1685: finishing write tid 3 to smithi11541303-4 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1685: finishing write tid 4 to smithi11541303-4 2022-04-22T17:31:39.920 INFO:tasks.rados.rados.0.smithi115.stdout:1685: finishing write tid 5 to smithi11541303-4 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 942 (ObjNum 513 snap 170 seq_num 513) dirty exists 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1685: left oid 4 (ObjNum 513 snap 170 seq_num 513) 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: finishing write tid 1 to smithi11541303-12 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: finishing write tid 2 to smithi11541303-12 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: finishing write tid 3 to smithi11541303-12 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: finishing write tid 4 to smithi11541303-12 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: finishing write tid 5 to smithi11541303-12 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 574 (ObjNum 514 snap 170 seq_num 514) dirty exists 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1686: left oid 12 (ObjNum 514 snap 170 seq_num 514) 2022-04-22T17:31:39.921 INFO:tasks.rados.rados.0.smithi115.stdout:1688: expect (ObjNum 487 snap 168 seq_num 487) 2022-04-22T17:31:40.015 INFO:tasks.rados.rados.0.smithi115.stdout:1687: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[46934]: cluster 2022-04-22T17:31:38.899658+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2022-04-22T17:31:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[46934]: audit 2022-04-22T17:31:39.223873+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:40.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[46934]: cluster 2022-04-22T17:31:39.794141+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2022-04-22T17:31:40.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[45495]: cluster 2022-04-22T17:31:38.899658+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2022-04-22T17:31:40.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[45495]: audit 2022-04-22T17:31:39.223873+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:40.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:39 smithi115 bash[45495]: cluster 2022-04-22T17:31:39.794141+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2022-04-22T17:31:40.113 INFO:tasks.rados.rados.0.smithi115.stdout:1689: finishing copy_from to smithi11541303-23 2022-04-22T17:31:40.206 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 757 (ObjNum 500 snap 170 seq_num 500) dirty exists 2022-04-22T17:31:40.206 INFO:tasks.rados.rados.0.smithi115.stdout:1689: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:31:40.206 INFO:tasks.rados.rados.0.smithi115.stdout:1690: finishing write tid 1 to smithi11541303-22 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: finishing write tid 2 to smithi11541303-22 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: finishing write tid 3 to smithi11541303-22 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: finishing write tid 4 to smithi11541303-22 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: finishing write tid 5 to smithi11541303-22 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 986 (ObjNum 515 snap 170 seq_num 515) dirty exists 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: left oid 22 (ObjNum 515 snap 170 seq_num 515) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1685: done (6 left) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1686: done (5 left) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1687: done (4 left) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1688: done (3 left) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1689: done (2 left) 2022-04-22T17:31:40.207 INFO:tasks.rados.rados.0.smithi115.stdout:1690: done (1 left) 2022-04-22T17:31:40.208 INFO:tasks.rados.rados.0.smithi115.stdout:1691: done (0 left) 2022-04-22T17:31:40.208 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 170 2022-04-22T17:31:40.208 INFO:tasks.rados.rados.0.smithi115.stdout:1692: seq_num 517 ranges {2162688=663552,2826240=786432,3612672=16384} 2022-04-22T17:31:40.212 INFO:tasks.rados.rados.0.smithi115.stdout:1692: writing smithi11541303-29 from 2162688 to 2826240 tid 1 2022-04-22T17:31:40.212 INFO:tasks.rados.rados.0.smithi115.stdout:1692: writing smithi11541303-29 from 2826240 to 3612672 tid 2 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1692: writing smithi11541303-29 from 3612672 to 3629056 tid 3 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1693: rollback oid 46 current snap is 170 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 169 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1692: finishing write tid 1 to smithi11541303-29 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 170 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1694: seq_num 518 ranges {4063232=679936,4743168=671744,5414912=131072} 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1694: writing smithi11541303-45 from 4063232 to 4743168 tid 1 2022-04-22T17:31:40.213 INFO:tasks.rados.rados.0.smithi115.stdout:1694: writing smithi11541303-45 from 4743168 to 5414912 tid 2 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1694: writing smithi11541303-45 from 5414912 to 5545984 tid 3 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1695: setattr oid 13 current snap is 170 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1692: finishing write tid 2 to smithi11541303-29 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1692: finishing write tid 3 to smithi11541303-29 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1692: finishing write tid 4 to smithi11541303-29 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1692: finishing write tid 5 to smithi11541303-29 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 491 (ObjNum 517 snap 170 seq_num 517) dirty exists 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1692: left oid 29 (ObjNum 517 snap 170 seq_num 517) 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1693: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:31:40.214 INFO:tasks.rados.rados.0.smithi115.stdout:1693: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1693: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 883 (ObjNum 495 snap 169 seq_num 495) dirty exists 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1692: done (3 left) 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1693: done (2 left) 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1696: setattr oid 17 current snap is 170 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1697: copy_from oid 43 from oid 39 current snap is 170 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1698: read oid 38 snap -1 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1698: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:1699: rollback oid 3 current snap is 170 2022-04-22T17:31:40.215 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 163 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1700: copy_from oid 28 from oid 30 current snap is 170 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1701: rollback oid 37 current snap is 170 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 163 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1702: snap_create 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1694: finishing write tid 1 to smithi11541303-45 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1694: finishing write tid 2 to smithi11541303-45 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1700: finishing copy_from to smithi11541303-28 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1700: got expected ENOENT (src dne) 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1700: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:31:40.216 INFO:tasks.rados.rados.0.smithi115.stdout:1697: finishing copy_from to smithi11541303-43 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1697: got expected ENOENT (src dne) 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1694: finishing write tid 3 to smithi11541303-45 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1699: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1694: finishing write tid 4 to smithi11541303-45 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1699: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1697: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 778 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:1699: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:31:40.217 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 886 (ObjNum 467 snap 159 seq_num 467) dirty exists 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 779 (ObjNum 457 snap 157 seq_num 457) dirty exists 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:1694: finishing write tid 5 to smithi11541303-45 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 946 (ObjNum 518 snap 170 seq_num 518) dirty exists 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:1694: left oid 45 (ObjNum 518 snap 170 seq_num 518) 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:1701: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:31:40.218 INFO:tasks.rados.rados.0.smithi115.stdout:1698: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:31:40.266 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:31:39 smithi121 bash[24219]: debug 2022-04-22T17:31:39.821+0000 7f1df3c49700 -1 osd.7 pg_epoch: 404 pg[5.4s0( v 404'712 (0'0,404'712] local-lis/les=392/393 n=6 ec=63/63 lis/c=392/392 les/c/f=393/393/0 sis=392) [7,2,5]p7(0) r=0 lpr=392 crt=404'712 lcod 404'710 mlcod 404'710 active+clean+snaptrim trimq=[12a~1] ps=[139~1,13b~1,142~1,149~2]] removing snap head 2022-04-22T17:31:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:39 smithi121 bash[36902]: cluster 2022-04-22T17:31:38.899658+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2022-04-22T17:31:40.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:39 smithi121 bash[36902]: audit 2022-04-22T17:31:39.223873+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:40.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:39 smithi121 bash[36902]: cluster 2022-04-22T17:31:39.794141+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2022-04-22T17:31:40.279 INFO:tasks.rados.rados.0.smithi115.stdout:1701: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:31:40.279 INFO:tasks.rados.rados.0.smithi115.stdout:1701: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:31:40.280 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 717 (ObjNum 390 snap 136 seq_num 390) dirty exists 2022-04-22T17:31:40.896 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:40 smithi115 systemd[1]: Stopping Ceph osd.3 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:31:40.932 INFO:tasks.rados.rados.0.smithi115.stdout:1694: done (8 left) 2022-04-22T17:31:40.932 INFO:tasks.rados.rados.0.smithi115.stdout:1695: done (7 left) 2022-04-22T17:31:40.932 INFO:tasks.rados.rados.0.smithi115.stdout:1696: done (6 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1697: done (5 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1698: done (4 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1699: done (3 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1700: done (2 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1701: done (1 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1702: done (0 left) 2022-04-22T17:31:40.933 INFO:tasks.rados.rados.0.smithi115.stdout:1703: snap_create 2022-04-22T17:31:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:40 smithi121 bash[36902]: cluster 2022-04-22T17:31:39.205114+0000 mgr.y (mgr.34107) 185 : cluster [DBG] pgmap v292: 121 pgs: 121 active+clean; 190 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2022-04-22T17:31:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:40 smithi115 bash[45495]: cluster 2022-04-22T17:31:39.205114+0000 mgr.y (mgr.34107) 185 : cluster [DBG] pgmap v292: 121 pgs: 121 active+clean; 190 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2022-04-22T17:31:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:40 smithi115 bash[46934]: cluster 2022-04-22T17:31:39.205114+0000 mgr.y (mgr.34107) 185 : cluster [DBG] pgmap v292: 121 pgs: 121 active+clean; 190 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2022-04-22T17:31:41.296 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:40 smithi115 bash[29268]: debug 2022-04-22T17:31:40.974+0000 7ff81d0d8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:31:41.296 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:40 smithi115 bash[29268]: debug 2022-04-22T17:31:40.974+0000 7ff81d0d8700 -1 osd.3 405 *** Got signal Terminated *** 2022-04-22T17:31:41.297 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:40 smithi115 bash[29268]: debug 2022-04-22T17:31:40.974+0000 7ff81d0d8700 -1 osd.3 405 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:31:41.936 INFO:tasks.rados.rados.0.smithi115.stdout:1703: done (0 left) 2022-04-22T17:31:41.970 INFO:tasks.rados.rados.0.smithi115.stdout:1704: read oid 37 snap -1 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1704: expect (ObjNum 390 snap 136 seq_num 390) 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1705: read oid 17 snap -1 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1705: expect (ObjNum 457 snap 157 seq_num 457) 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1706: read oid 35 snap -1 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1706: expect (ObjNum 509 snap 170 seq_num 509) 2022-04-22T17:31:41.971 INFO:tasks.rados.rados.0.smithi115.stdout:1707: snap_remove snap 169 2022-04-22T17:31:41.972 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:41.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:41.972 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:41.623Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:41.972 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:41.624Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:42.198 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[55391]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.3 2022-04-22T17:31:42.198 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[55434]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-3 2022-04-22T17:31:42.198 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[55447]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.3 2022-04-22T17:31:42.199 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:40.915985+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2022-04-22T17:31:42.199 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.254667+0000 mon.a (mon.0) 577 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.254742+0000 mon.a (mon.0) 578 : cluster [INF] osd.3 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:31:42.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.264971+0000 mon.a (mon.0) 579 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.265321+0000 mon.a (mon.0) 580 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.265927+0000 mon.a (mon.0) 581 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.266300+0000 mon.a (mon.0) 582 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.266654+0000 mon.a (mon.0) 583 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.267280+0000 mon.a (mon.0) 584 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.269720+0000 mon.a (mon.0) 585 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.270123+0000 mon.a (mon.0) 586 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.270463+0000 mon.a (mon.0) 587 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.270681+0000 mon.a (mon.0) 588 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.271002+0000 mon.a (mon.0) 589 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.271143+0000 mon.a (mon.0) 590 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.271565+0000 mon.a (mon.0) 591 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.271905+0000 mon.a (mon.0) 592 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.457628+0000 mon.a (mon.0) 593 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.665379+0000 mon.a (mon.0) 594 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.665754+0000 mon.a (mon.0) 595 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.666437+0000 mon.a (mon.0) 596 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.666647+0000 mon.a (mon.0) 597 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.666862+0000 mon.a (mon.0) 598 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.667452+0000 mon.a (mon.0) 599 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.673527+0000 mon.a (mon.0) 600 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.673959+0000 mon.a (mon.0) 601 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.674151+0000 mon.a (mon.0) 602 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.674313+0000 mon.a (mon.0) 603 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.674543+0000 mon.a (mon.0) 604 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.674748+0000 mon.a (mon.0) 605 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.674914+0000 mon.a (mon.0) 606 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.675130+0000 mon.a (mon.0) 607 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.861767+0000 mon.a (mon.0) 608 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:40.915985+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2022-04-22T17:31:42.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.254667+0000 mon.a (mon.0) 577 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.254742+0000 mon.a (mon.0) 578 : cluster [INF] osd.3 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.264971+0000 mon.a (mon.0) 579 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.265321+0000 mon.a (mon.0) 580 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.265927+0000 mon.a (mon.0) 581 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.266300+0000 mon.a (mon.0) 582 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.266654+0000 mon.a (mon.0) 583 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.267280+0000 mon.a (mon.0) 584 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.269720+0000 mon.a (mon.0) 585 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.270123+0000 mon.a (mon.0) 586 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.270463+0000 mon.a (mon.0) 587 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.270681+0000 mon.a (mon.0) 588 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.271002+0000 mon.a (mon.0) 589 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.271143+0000 mon.a (mon.0) 590 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.271565+0000 mon.a (mon.0) 591 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.271905+0000 mon.a (mon.0) 592 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.457628+0000 mon.a (mon.0) 593 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.665379+0000 mon.a (mon.0) 594 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.665754+0000 mon.a (mon.0) 595 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.666437+0000 mon.a (mon.0) 596 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.666647+0000 mon.a (mon.0) 597 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.666862+0000 mon.a (mon.0) 598 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.667452+0000 mon.a (mon.0) 599 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.673527+0000 mon.a (mon.0) 600 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.673959+0000 mon.a (mon.0) 601 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.674151+0000 mon.a (mon.0) 602 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.674313+0000 mon.a (mon.0) 603 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.674543+0000 mon.a (mon.0) 604 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.674748+0000 mon.a (mon.0) 605 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.674914+0000 mon.a (mon.0) 606 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.675130+0000 mon.a (mon.0) 607 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:41 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.861767+0000 mon.a (mon.0) 608 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:40.915985+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.254667+0000 mon.a (mon.0) 577 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.254742+0000 mon.a (mon.0) 578 : cluster [INF] osd.3 failed (root=default,host=smithi115) (connection refused reported by osd.4) 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.264971+0000 mon.a (mon.0) 579 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.265321+0000 mon.a (mon.0) 580 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.265927+0000 mon.a (mon.0) 581 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.266300+0000 mon.a (mon.0) 582 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.266654+0000 mon.a (mon.0) 583 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.267280+0000 mon.a (mon.0) 584 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.269720+0000 mon.a (mon.0) 585 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.270123+0000 mon.a (mon.0) 586 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.270463+0000 mon.a (mon.0) 587 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.270681+0000 mon.a (mon.0) 588 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.271002+0000 mon.a (mon.0) 589 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.271143+0000 mon.a (mon.0) 590 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.271565+0000 mon.a (mon.0) 591 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.271905+0000 mon.a (mon.0) 592 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.457628+0000 mon.a (mon.0) 593 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.665379+0000 mon.a (mon.0) 594 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.665754+0000 mon.a (mon.0) 595 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.666437+0000 mon.a (mon.0) 596 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.666647+0000 mon.a (mon.0) 597 : cluster [DBG] osd.3 reported immediately failed by osd.0 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.666862+0000 mon.a (mon.0) 598 : cluster [DBG] osd.3 reported immediately failed by osd.2 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.667452+0000 mon.a (mon.0) 599 : cluster [DBG] osd.3 reported immediately failed by osd.1 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.673527+0000 mon.a (mon.0) 600 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.673959+0000 mon.a (mon.0) 601 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.674151+0000 mon.a (mon.0) 602 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.674313+0000 mon.a (mon.0) 603 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.674543+0000 mon.a (mon.0) 604 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.674748+0000 mon.a (mon.0) 605 : cluster [DBG] osd.3 reported immediately failed by osd.5 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.674914+0000 mon.a (mon.0) 606 : cluster [DBG] osd.3 reported immediately failed by osd.6 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.675130+0000 mon.a (mon.0) 607 : cluster [DBG] osd.3 reported immediately failed by osd.7 2022-04-22T17:31:42.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:41 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.861767+0000 mon.a (mon.0) 608 : cluster [DBG] osd.3 reported immediately failed by osd.4 2022-04-22T17:31:42.939 INFO:tasks.rados.rados.0.smithi115.stdout:1707: done (3 left) 2022-04-22T17:31:42.939 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 172 2022-04-22T17:31:42.940 INFO:tasks.rados.rados.0.smithi115.stdout:1708: seq_num 519 ranges {1589248=696320,2285568=548864,2834432=253952} 2022-04-22T17:31:42.960 INFO:tasks.rados.rados.0.smithi115.stdout:1708: writing smithi11541303-25 from 1589248 to 2285568 tid 1 2022-04-22T17:31:42.963 INFO:tasks.rados.rados.0.smithi115.stdout:1708: writing smithi11541303-25 from 2285568 to 2834432 tid 2 2022-04-22T17:31:42.966 INFO:tasks.rados.rados.0.smithi115.stdout:1708: writing smithi11541303-25 from 2834432 to 3088384 tid 3 2022-04-22T17:31:42.966 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 172 2022-04-22T17:31:42.966 INFO:tasks.rados.rados.0.smithi115.stdout:1709: seq_num 520 ranges {3629056=712704,4341760=434176,4775936=368640} 2022-04-22T17:31:42.990 INFO:tasks.rados.rados.0.smithi115.stdout:1709: writing smithi11541303-5 from 3629056 to 4341760 tid 1 2022-04-22T17:31:42.997 INFO:tasks.rados.rados.0.smithi115.stdout:1709: writing smithi11541303-5 from 4341760 to 4775936 tid 2 2022-04-22T17:31:42.998 INFO:tasks.rados.rados.0.smithi115.stdout:1709: writing smithi11541303-5 from 4775936 to 5144576 tid 3 2022-04-22T17:31:42.998 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 172 2022-04-22T17:31:42.998 INFO:tasks.rados.rados.0.smithi115.stdout:1710: seq_num 521 ranges {0=729088,729088=720896,1449984=81920} 2022-04-22T17:31:42.999 INFO:tasks.rados.rados.0.smithi115.stdout:1710: writing smithi11541303-36 from 0 to 729088 tid 1 2022-04-22T17:31:43.004 INFO:tasks.rados.rados.0.smithi115.stdout:1710: writing smithi11541303-36 from 729088 to 1449984 tid 2 2022-04-22T17:31:43.004 INFO:tasks.rados.rados.0.smithi115.stdout:1710: writing smithi11541303-36 from 1449984 to 1531904 tid 3 2022-04-22T17:31:43.005 INFO:tasks.rados.rados.0.smithi115.stdout:1711: snap_create 2022-04-22T17:31:43.005 INFO:tasks.rados.rados.0.smithi115.stdout:1705: expect (ObjNum 457 snap 157 seq_num 457) 2022-04-22T17:31:43.153 INFO:tasks.rados.rados.0.smithi115.stdout:1704: expect (ObjNum 390 snap 136 seq_num 390) 2022-04-22T17:31:43.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.205958+0000 mgr.y (mgr.34107) 186 : cluster [DBG] pgmap v295: 121 pgs: 121 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2022-04-22T17:31:43.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.912350+0000 mon.a (mon.0) 609 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[46934]: cluster 2022-04-22T17:31:41.920411+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[46934]: cluster 2022-04-22T17:31:42.923230+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.205958+0000 mgr.y (mgr.34107) 186 : cluster [DBG] pgmap v295: 121 pgs: 121 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.912350+0000 mon.a (mon.0) 609 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[45495]: cluster 2022-04-22T17:31:41.920411+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2022-04-22T17:31:43.259 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:42 smithi115 bash[45495]: cluster 2022-04-22T17:31:42.923230+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2022-04-22T17:31:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:42 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.205958+0000 mgr.y (mgr.34107) 186 : cluster [DBG] pgmap v295: 121 pgs: 121 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2022-04-22T17:31:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:42 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.912350+0000 mon.a (mon.0) 609 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:31:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:42 smithi121 bash[36902]: cluster 2022-04-22T17:31:41.920411+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2022-04-22T17:31:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:42 smithi121 bash[36902]: cluster 2022-04-22T17:31:42.923230+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2022-04-22T17:31:43.572 INFO:tasks.rados.rados.0.smithi115.stdout:1706: expect (ObjNum 509 snap 170 seq_num 509) 2022-04-22T17:31:43.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:43.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:43.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:43.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:43.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:43.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:43.902 INFO:tasks.rados.rados.0.smithi115.stdout:1708: finishing write tid 1 to smithi11541303-25 2022-04-22T17:31:43.902 INFO:tasks.rados.rados.0.smithi115.stdout:1708: finishing write tid 2 to smithi11541303-25 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1708: finishing write tid 3 to smithi11541303-25 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1708: finishing write tid 4 to smithi11541303-25 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1708: finishing write tid 5 to smithi11541303-25 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 788 (ObjNum 519 snap 172 seq_num 519) dirty exists 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1708: left oid 25 (ObjNum 519 snap 172 seq_num 519) 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1709: finishing write tid 1 to smithi11541303-5 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1709: finishing write tid 2 to smithi11541303-5 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1709: finishing write tid 3 to smithi11541303-5 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1709: finishing write tid 4 to smithi11541303-5 2022-04-22T17:31:43.903 INFO:tasks.rados.rados.0.smithi115.stdout:1709: finishing write tid 5 to smithi11541303-5 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1087 (ObjNum 520 snap 172 seq_num 520) dirty exists 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1709: left oid 5 (ObjNum 520 snap 172 seq_num 520) 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: finishing write tid 1 to smithi11541303-36 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: finishing write tid 2 to smithi11541303-36 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: finishing write tid 3 to smithi11541303-36 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: finishing write tid 4 to smithi11541303-36 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: finishing write tid 5 to smithi11541303-36 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 762 (ObjNum 521 snap 172 seq_num 521) dirty exists 2022-04-22T17:31:43.904 INFO:tasks.rados.rados.0.smithi115.stdout:1710: left oid 36 (ObjNum 521 snap 172 seq_num 521) 2022-04-22T17:31:43.980 INFO:tasks.rados.rados.0.smithi115.stdout:1704: done (6 left) 2022-04-22T17:31:43.980 INFO:tasks.rados.rados.0.smithi115.stdout:1705: done (5 left) 2022-04-22T17:31:43.980 INFO:tasks.rados.rados.0.smithi115.stdout:1706: done (4 left) 2022-04-22T17:31:43.980 INFO:tasks.rados.rados.0.smithi115.stdout:1708: done (3 left) 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1709: done (2 left) 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1710: done (1 left) 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1711: done (0 left) 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1712: rmattr oid 13 current snap is 173 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1713: read oid 26 snap 171 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1713: expect deleted 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 173 2022-04-22T17:31:43.981 INFO:tasks.rados.rados.0.smithi115.stdout:1714: seq_num 522 ranges {6152192=745472,6897664=606208,7503872=196608} 2022-04-22T17:31:44.021 INFO:tasks.rados.rados.0.smithi115.stdout:1714: writing smithi11541303-6 from 6152192 to 6897664 tid 1 2022-04-22T17:31:44.024 INFO:tasks.rados.rados.0.smithi115.stdout:1714: writing smithi11541303-6 from 6897664 to 7503872 tid 2 2022-04-22T17:31:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:1714: writing smithi11541303-6 from 7503872 to 7700480 tid 3 2022-04-22T17:31:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:1715: rollback oid 24 current snap is 173 2022-04-22T17:31:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 171 2022-04-22T17:31:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:1716: read oid 28 snap -1 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:1716: expect deleted 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:1717: read oid 46 snap -1 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:1717: expect (ObjNum 495 snap 169 seq_num 495) 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 173 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 794 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:44.027 INFO:tasks.rados.rados.0.smithi115.stdout:1718: seq_num 523 ranges {2220032=770048,2990080=491520,3481600=311296} 2022-04-22T17:31:44.043 INFO:tasks.rados.rados.0.smithi115.stdout:1718: writing smithi11541303-17 from 2220032 to 2990080 tid 1 2022-04-22T17:31:44.046 INFO:tasks.rados.rados.0.smithi115.stdout:1718: writing smithi11541303-17 from 2990080 to 3481600 tid 2 2022-04-22T17:31:44.049 INFO:tasks.rados.rados.0.smithi115.stdout:1718: writing smithi11541303-17 from 3481600 to 3792896 tid 3 2022-04-22T17:31:44.049 INFO:tasks.rados.rados.0.smithi115.stdout:1712: done (6 left) 2022-04-22T17:31:44.049 INFO:tasks.rados.rados.0.smithi115.stdout:1713: done (5 left) 2022-04-22T17:31:44.049 INFO:tasks.rados.rados.0.smithi115.stdout:1719: copy_from oid 35 from oid 31 current snap is 173 2022-04-22T17:31:44.049 INFO:tasks.rados.rados.0.smithi115.stdout:1714: finishing write tid 1 to smithi11541303-6 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1714: finishing write tid 2 to smithi11541303-6 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1714: finishing write tid 3 to smithi11541303-6 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1714: finishing write tid 4 to smithi11541303-6 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1715: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 650 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1715: done (5 left) 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1716: done (4 left) 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1720: rollback oid 26 current snap is 173 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 157 2022-04-22T17:31:44.050 INFO:tasks.rados.rados.0.smithi115.stdout:1714: finishing write tid 5 to smithi11541303-6 2022-04-22T17:31:44.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 654 (ObjNum 522 snap 173 seq_num 522) dirty exists 2022-04-22T17:31:44.051 INFO:tasks.rados.rados.0.smithi115.stdout:1714: left oid 6 (ObjNum 522 snap 173 seq_num 522) 2022-04-22T17:31:44.051 INFO:tasks.rados.rados.0.smithi115.stdout:1717: expect (ObjNum 495 snap 169 seq_num 495) 2022-04-22T17:31:44.157 INFO:tasks.rados.rados.0.smithi115.stdout:1720: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:31:44.157 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 0 (ObjNum 288 snap 0 seq_num 33) dirty dne 2022-04-22T17:31:44.157 INFO:tasks.rados.rados.0.smithi115.stdout:1718: finishing write tid 1 to smithi11541303-17 2022-04-22T17:31:44.158 INFO:tasks.rados.rados.0.smithi115.stdout:1718: finishing write tid 2 to smithi11541303-17 2022-04-22T17:31:44.158 INFO:tasks.rados.rados.0.smithi115.stdout:1718: finishing write tid 3 to smithi11541303-17 2022-04-22T17:31:44.158 INFO:tasks.rados.rados.0.smithi115.stdout:1718: finishing write tid 4 to smithi11541303-17 2022-04-22T17:31:44.159 INFO:tasks.rados.rados.0.smithi115.stdout:1718: finishing write tid 5 to smithi11541303-17 2022-04-22T17:31:44.159 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 799 (ObjNum 523 snap 173 seq_num 523) dirty exists 2022-04-22T17:31:44.159 INFO:tasks.rados.rados.0.smithi115.stdout:1718: left oid 17 (ObjNum 523 snap 173 seq_num 523) 2022-04-22T17:31:44.159 INFO:tasks.rados.rados.0.smithi115.stdout:1719: finishing copy_from to smithi11541303-35 2022-04-22T17:31:44.160 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1048 (ObjNum 463 snap 158 seq_num 463) dirty exists 2022-04-22T17:31:44.160 INFO:tasks.rados.rados.0.smithi115.stdout:1719: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:31:44.160 INFO:tasks.rados.rados.0.smithi115.stdout:1714: done (4 left) 2022-04-22T17:31:44.163 INFO:tasks.rados.rados.0.smithi115.stdout:1717: done (3 left) 2022-04-22T17:31:44.163 INFO:tasks.rados.rados.0.smithi115.stdout:1718: done (2 left) 2022-04-22T17:31:44.163 INFO:tasks.rados.rados.0.smithi115.stdout:1719: done (1 left) 2022-04-22T17:31:44.163 INFO:tasks.rados.rados.0.smithi115.stdout:1720: done (0 left) 2022-04-22T17:31:44.163 INFO:tasks.rados.rados.0.smithi115.stdout:1721: rollback oid 9 current snap is 173 2022-04-22T17:31:44.164 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 158 2022-04-22T17:31:44.164 INFO:tasks.rados.rados.0.smithi115.stdout:1722: read oid 41 snap -1 2022-04-22T17:31:44.164 INFO:tasks.rados.rados.0.smithi115.stdout:1722: expect deleted 2022-04-22T17:31:44.164 INFO:tasks.rados.rados.0.smithi115.stdout:1723: delete oid 49 current snap is 173 2022-04-22T17:31:44.165 INFO:tasks.rados.rados.0.smithi115.stdout:1721: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:31:44.165 INFO:tasks.rados.rados.0.smithi115.stdout:1721: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:31:44.166 INFO:tasks.rados.rados.0.smithi115.stdout:1721: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:31:44.166 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1091 (ObjNum 466 snap 158 seq_num 466) dirty exists 2022-04-22T17:31:44.166 INFO:tasks.rados.rados.0.smithi115.stdout:1721: done (2 left) 2022-04-22T17:31:44.166 INFO:tasks.rados.rados.0.smithi115.stdout:1722: done (1 left) 2022-04-22T17:31:44.167 INFO:tasks.rados.rados.0.smithi115.stdout:1723: done (0 left) 2022-04-22T17:31:44.167 INFO:tasks.rados.rados.0.smithi115.stdout:1724: snap_remove snap 171 2022-04-22T17:31:44.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:44 smithi115 bash[46934]: cluster 2022-04-22T17:31:43.948761+0000 mon.a (mon.0) 612 : cluster [WRN] Health check failed: Degraded data redundancy: 6/396 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:44.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:44 smithi115 bash[46934]: cluster 2022-04-22T17:31:43.963859+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2022-04-22T17:31:44.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:44 smithi115 bash[45495]: cluster 2022-04-22T17:31:43.948761+0000 mon.a (mon.0) 612 : cluster [WRN] Health check failed: Degraded data redundancy: 6/396 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:44.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:44 smithi115 bash[45495]: cluster 2022-04-22T17:31:43.963859+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2022-04-22T17:31:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:44 smithi121 bash[36902]: cluster 2022-04-22T17:31:43.948761+0000 mon.a (mon.0) 612 : cluster [WRN] Health check failed: Degraded data redundancy: 6/396 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:44 smithi121 bash[36902]: cluster 2022-04-22T17:31:43.963859+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2022-04-22T17:31:44.814 INFO:tasks.rados.rados.0.smithi115.stdout:1724: done (0 left) 2022-04-22T17:31:44.814 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 173 2022-04-22T17:31:44.814 INFO:tasks.rados.rados.0.smithi115.stdout:1725: seq_num 525 ranges {7233536=802816,8036352=712704,8749056=90112} 2022-04-22T17:31:44.846 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:44 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.3.service: Succeeded. 2022-04-22T17:31:44.846 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:44 smithi115 systemd[1]: Stopped Ceph osd.3 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:31:44.864 INFO:tasks.rados.rados.0.smithi115.stdout:1725: writing smithi11541303-20 from 7233536 to 8036352 tid 1 2022-04-22T17:31:44.869 INFO:tasks.rados.rados.0.smithi115.stdout:1725: writing smithi11541303-20 from 8036352 to 8749056 tid 2 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:1725: writing smithi11541303-20 from 8749056 to 8839168 tid 3 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:1726: rollback oid 39 current snap is 173 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 172 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:1727: rmattr oid 9 current snap is 173 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:1727: done (2 left) 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:1728: rollback oid 25 current snap is 173 2022-04-22T17:31:44.871 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 163 2022-04-22T17:31:44.872 INFO:tasks.rados.rados.0.smithi115.stdout:1729: setattr oid 34 current snap is 173 2022-04-22T17:31:44.873 INFO:tasks.rados.rados.0.smithi115.stdout:1730: read oid 42 snap -1 2022-04-22T17:31:44.873 INFO:tasks.rados.rados.0.smithi115.stdout:1730: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:31:44.874 INFO:tasks.rados.rados.0.smithi115.stdout:1731: read oid 33 snap 172 2022-04-22T17:31:44.874 INFO:tasks.rados.rados.0.smithi115.stdout:1731: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:31:44.874 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 173 2022-04-22T17:31:44.874 INFO:tasks.rados.rados.0.smithi115.stdout:1732: seq_num 526 ranges {0=417792,417792=589824,1007616=614400} 2022-04-22T17:31:44.876 INFO:tasks.rados.rados.0.smithi115.stdout:1732: writing smithi11541303-1 from 0 to 417792 tid 1 2022-04-22T17:31:44.881 INFO:tasks.rados.rados.0.smithi115.stdout:1732: writing smithi11541303-1 from 417792 to 1007616 tid 2 2022-04-22T17:31:44.887 INFO:tasks.rados.rados.0.smithi115.stdout:1732: writing smithi11541303-1 from 1007616 to 1622016 tid 3 2022-04-22T17:31:44.887 INFO:tasks.rados.rados.0.smithi115.stdout:1733: snap_create 2022-04-22T17:31:44.887 INFO:tasks.rados.rados.0.smithi115.stdout:1725: finishing write tid 1 to smithi11541303-20 2022-04-22T17:31:44.888 INFO:tasks.rados.rados.0.smithi115.stdout:1725: finishing write tid 2 to smithi11541303-20 2022-04-22T17:31:44.889 INFO:tasks.rados.rados.0.smithi115.stdout:1732: finishing write tid 1 to smithi11541303-1 2022-04-22T17:31:44.889 INFO:tasks.rados.rados.0.smithi115.stdout:1732: finishing write tid 2 to smithi11541303-1 2022-04-22T17:31:44.891 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 711 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty exists 2022-04-22T17:31:44.891 INFO:tasks.rados.rados.0.smithi115.stdout:1731: expect (ObjNum 346 snap 120 seq_num 346) 2022-04-22T17:31:44.976 INFO:tasks.rados.rados.0.smithi115.stdout:1725: finishing write tid 3 to smithi11541303-20 2022-04-22T17:31:44.976 INFO:tasks.rados.rados.0.smithi115.stdout:1725: finishing write tid 4 to smithi11541303-20 2022-04-22T17:31:44.976 INFO:tasks.rados.rados.0.smithi115.stdout:1728: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:31:44.976 INFO:tasks.rados.rados.0.smithi115.stdout:1728: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:31:44.976 INFO:tasks.rados.rados.0.smithi115.stdout:1728: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 808 (ObjNum 454 snap 154 seq_num 454) dirty exists 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1725: finishing write tid 5 to smithi11541303-20 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 804 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1725: left oid 20 (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1732: finishing write tid 3 to smithi11541303-1 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1732: finishing write tid 4 to smithi11541303-1 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1732: finishing write tid 5 to smithi11541303-1 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 766 (ObjNum 526 snap 173 seq_num 526) dirty exists 2022-04-22T17:31:44.977 INFO:tasks.rados.rados.0.smithi115.stdout:1732: left oid 1 (ObjNum 526 snap 173 seq_num 526) 2022-04-22T17:31:44.978 INFO:tasks.rados.rados.0.smithi115.stdout:1726: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:31:44.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 0 (ObjNum 288 snap 0 seq_num 49) dirty dne 2022-04-22T17:31:44.978 INFO:tasks.rados.rados.0.smithi115.stdout:1730: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:31:45.303 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:44 smithi115 systemd[1]: Started Ceph osd.3 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: cluster 2022-04-22T17:31:43.206845+0000 mgr.y (mgr.34107) 187 : cluster [DBG] pgmap v298: 121 pgs: 2 active+undersized+degraded, 14 stale+active+clean, 6 active+undersized, 99 active+clean; 189 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 6/396 objects degraded (1.515%) 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: audit 2022-04-22T17:31:44.226502+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: cluster 2022-04-22T17:31:44.797720+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: audit 2022-04-22T17:31:44.877077+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: audit 2022-04-22T17:31:44.881182+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: audit 2022-04-22T17:31:44.882933+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[46934]: audit 2022-04-22T17:31:44.884474+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: cluster 2022-04-22T17:31:43.206845+0000 mgr.y (mgr.34107) 187 : cluster [DBG] pgmap v298: 121 pgs: 2 active+undersized+degraded, 14 stale+active+clean, 6 active+undersized, 99 active+clean; 189 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 6/396 objects degraded (1.515%) 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: audit 2022-04-22T17:31:44.226502+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: cluster 2022-04-22T17:31:44.797720+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: audit 2022-04-22T17:31:44.877077+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: audit 2022-04-22T17:31:44.881182+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: audit 2022-04-22T17:31:44.882933+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:45.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:45 smithi115 bash[45495]: audit 2022-04-22T17:31:44.884474+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: cluster 2022-04-22T17:31:43.206845+0000 mgr.y (mgr.34107) 187 : cluster [DBG] pgmap v298: 121 pgs: 2 active+undersized+degraded, 14 stale+active+clean, 6 active+undersized, 99 active+clean; 189 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 6/396 objects degraded (1.515%) 2022-04-22T17:31:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: audit 2022-04-22T17:31:44.226502+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: cluster 2022-04-22T17:31:44.797720+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2022-04-22T17:31:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: audit 2022-04-22T17:31:44.877077+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: audit 2022-04-22T17:31:44.881182+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:31:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: audit 2022-04-22T17:31:44.882933+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:31:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:45 smithi121 bash[36902]: audit 2022-04-22T17:31:44.884474+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1725: done (7 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1726: done (6 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1728: done (5 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1729: done (4 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1730: done (3 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1731: done (2 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1732: done (1 left) 2022-04-22T17:31:46.493 INFO:tasks.rados.rados.0.smithi115.stdout:1733: done (0 left) 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1734: copy_from oid 34 from oid 41 current snap is 174 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1735: rollback oid 2 current snap is 174 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 2 to 157 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1736: rollback oid 26 current snap is 174 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 170 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1737: read oid 9 snap -1 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1737: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1738: read oid 16 snap -1 2022-04-22T17:31:46.494 INFO:tasks.rados.rados.0.smithi115.stdout:1738: expect deleted 2022-04-22T17:31:46.495 INFO:tasks.rados.rados.0.smithi115.stdout:1739: read oid 49 snap -1 2022-04-22T17:31:46.495 INFO:tasks.rados.rados.0.smithi115.stdout:1739: expect deleted 2022-04-22T17:31:46.495 INFO:tasks.rados.rados.0.smithi115.stdout:1740: read oid 31 snap -1 2022-04-22T17:31:46.495 INFO:tasks.rados.rados.0.smithi115.stdout:1740: expect (ObjNum 463 snap 158 seq_num 463) 2022-04-22T17:31:46.495 INFO:tasks.rados.rados.0.smithi115.stdout:1741: snap_create 2022-04-22T17:31:46.496 INFO:tasks.rados.rados.0.smithi115.stdout:1734: finishing copy_from to smithi11541303-34 2022-04-22T17:31:46.496 INFO:tasks.rados.rados.0.smithi115.stdout:1734: got expected ENOENT (src dne) 2022-04-22T17:31:46.496 INFO:tasks.rados.rados.0.smithi115.stdout:1734: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:1735: finishing rollback tid 0 to smithi11541303-2 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:1736: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 0 (ObjNum 288 snap 0 seq_num 49) dirty dne 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:1735: finishing rollback tid 1 to smithi11541303-2 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:1735: finishing rollback tid 2 to smithi11541303-2 2022-04-22T17:31:46.497 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1051 (ObjNum 443 snap 148 seq_num 443) dirty exists 2022-04-22T17:31:46.507 INFO:tasks.rados.rados.0.smithi115.stdout:1737: expect (ObjNum 466 snap 158 seq_num 466) 2022-04-22T17:31:46.549 INFO:tasks.rados.rados.0.smithi115.stdout:1740: expect (ObjNum 463 snap 158 seq_num 463) 2022-04-22T17:31:47.030 INFO:tasks.rados.rados.0.smithi115.stdout:1734: done (7 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1735: done (6 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1736: done (5 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1737: done (4 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1738: done (3 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1739: done (2 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1740: done (1 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1741: done (0 left) 2022-04-22T17:31:47.031 INFO:tasks.rados.rados.0.smithi115.stdout:1742: rollback oid 35 current snap is 175 2022-04-22T17:31:47.032 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 163 2022-04-22T17:31:47.032 INFO:tasks.rados.rados.0.smithi115.stdout:1743: snap_remove snap 174 2022-04-22T17:31:47.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[46934]: cluster 2022-04-22T17:31:45.207735+0000 mgr.y (mgr.34107) 188 : cluster [DBG] pgmap v301: 121 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 25 active+undersized, 89 active+clean; 182 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.1 MiB/s wr, 43 op/s; 22/387 objects degraded (5.685%) 2022-04-22T17:31:47.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[46934]: cluster 2022-04-22T17:31:45.806178+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2022-04-22T17:31:47.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[45495]: cluster 2022-04-22T17:31:45.207735+0000 mgr.y (mgr.34107) 188 : cluster [DBG] pgmap v301: 121 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 25 active+undersized, 89 active+clean; 182 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.1 MiB/s wr, 43 op/s; 22/387 objects degraded (5.685%) 2022-04-22T17:31:47.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[45495]: cluster 2022-04-22T17:31:45.806178+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2022-04-22T17:31:47.033 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-04-22T17:31:47.037 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2022-04-22T17:31:47.038 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2022-04-22T17:31:47.038 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2022-04-22T17:31:47.038 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2022-04-22T17:31:47.038 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-04-22T17:31:47.038 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:46 smithi115 bash[55630]: --> ceph-volume raw activate successful for osd ID: 3 2022-04-22T17:31:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:46 smithi121 bash[36902]: cluster 2022-04-22T17:31:45.207735+0000 mgr.y (mgr.34107) 188 : cluster [DBG] pgmap v301: 121 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 25 active+undersized, 89 active+clean; 182 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.1 MiB/s wr, 43 op/s; 22/387 objects degraded (5.685%) 2022-04-22T17:31:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:46 smithi121 bash[36902]: cluster 2022-04-22T17:31:45.806178+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2022-04-22T17:31:47.922 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:31:48.035 INFO:tasks.rados.rados.0.smithi115.stdout:1743: done (1 left) 2022-04-22T17:31:48.036 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 175 2022-04-22T17:31:48.036 INFO:tasks.rados.rados.0.smithi115.stdout:1742: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:31:48.036 INFO:tasks.rados.rados.0.smithi115.stdout:1744: seq_num 527 ranges {1966080=434176,2400256=475136,2875392=565248,3440640=163840} 2022-04-22T17:31:48.050 INFO:tasks.rados.rados.0.smithi115.stdout:1744: writing smithi11541303-18 from 1966080 to 2400256 tid 1 2022-04-22T17:31:48.123 INFO:tasks.rados.rados.0.smithi115.stdout:1744: writing smithi11541303-18 from 2400256 to 2875392 tid 2 2022-04-22T17:31:48.124 INFO:tasks.rados.rados.0.smithi115.stdout:1744: writing smithi11541303-18 from 2875392 to 3440640 tid 3 2022-04-22T17:31:48.124 INFO:tasks.rados.rados.0.smithi115.stdout:1744: writing smithi11541303-18 from 3440640 to 3604480 tid 4 2022-04-22T17:31:48.124 INFO:tasks.rados.rados.0.smithi115.stdout:1745: snap_remove snap 170 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (8m) 24s ago 8m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (8m) 2m ago 8m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (9m) 24s ago 9m 13.9M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (9m) 2m ago 9m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4m) 2m ago 13m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4m) 24s ago 16m 447M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:31:48.283 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3m) 24s ago 16m 55.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2m) 2m ago 14m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3m) 24s ago 14m 48.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (8m) 24s ago 8m 9203k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (8m) 2m ago 8m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (95s) 24s ago 13m 177M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (53s) 24s ago 13m 166M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (27s) 24s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 starting - - - 2526M 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (11m) 2m ago 11m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:31:48.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (11m) 2m ago 11m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:31:48.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (10m) 2m ago 10m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:31:48.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (10m) 2m ago 10m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:31:48.285 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4m) 2m ago 8m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:31:48.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:48 smithi115 bash[45495]: cluster 2022-04-22T17:31:47.013856+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2022-04-22T17:31:48.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:48 smithi115 bash[46934]: cluster 2022-04-22T17:31:47.013856+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2022-04-22T17:31:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:48 smithi121 bash[36902]: cluster 2022-04-22T17:31:47.013856+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2022-04-22T17:31:48.699 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:31:48.699 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4, 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:31:48.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6, 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:31:48.701 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:31:49.033 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:48 smithi115 bash[55827]: debug 2022-04-22T17:31:48.842+0000 7f183b8f4700 1 -- 172.21.15.115:0/3433388833 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55960d800340 con 0x55960d884c00 2022-04-22T17:31:49.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[45495]: cluster 2022-04-22T17:31:47.208714+0000 mgr.y (mgr.34107) 189 : cluster [DBG] pgmap v304: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 186 MiB data, 962 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 32 op/s; 35/387 objects degraded (9.044%) 2022-04-22T17:31:49.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[45495]: audit 2022-04-22T17:31:47.891252+0000 mgr.y (mgr.34107) 190 : audit [DBG] from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:49.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[45495]: cluster 2022-04-22T17:31:48.019697+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2022-04-22T17:31:49.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[45495]: audit 2022-04-22T17:31:48.683346+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.115:0/3081142953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1745: done (2 left) 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1746: rmattr oid 13 current snap is 175 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1742: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1742: finishing rollback tid 2 to smithi11541303-35 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1054 (ObjNum 451 snap 152 seq_num 451) dirty exists 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 1 to smithi11541303-18 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 2 to smithi11541303-18 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 3 to smithi11541303-18 2022-04-22T17:31:49.052 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 4 to smithi11541303-18 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 5 to smithi11541303-18 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1744: finishing write tid 6 to smithi11541303-18 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1105 (ObjNum 527 snap 175 seq_num 527) dirty exists 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1744: left oid 18 (ObjNum 527 snap 175 seq_num 527) 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1742: done (2 left) 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1744: done (1 left) 2022-04-22T17:31:49.053 INFO:tasks.rados.rados.0.smithi115.stdout:1747: snap_create 2022-04-22T17:31:49.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 816 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:31:49.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[46934]: cluster 2022-04-22T17:31:47.208714+0000 mgr.y (mgr.34107) 189 : cluster [DBG] pgmap v304: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 186 MiB data, 962 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 32 op/s; 35/387 objects degraded (9.044%) 2022-04-22T17:31:49.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[46934]: audit 2022-04-22T17:31:47.891252+0000 mgr.y (mgr.34107) 190 : audit [DBG] from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:49.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[46934]: cluster 2022-04-22T17:31:48.019697+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2022-04-22T17:31:49.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[46934]: audit 2022-04-22T17:31:48.683346+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.115:0/3081142953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:49 smithi121 bash[36902]: cluster 2022-04-22T17:31:47.208714+0000 mgr.y (mgr.34107) 189 : cluster [DBG] pgmap v304: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 186 MiB data, 962 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 32 op/s; 35/387 objects degraded (9.044%) 2022-04-22T17:31:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:49 smithi121 bash[36902]: audit 2022-04-22T17:31:47.891252+0000 mgr.y (mgr.34107) 190 : audit [DBG] from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:49.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:49 smithi121 bash[36902]: cluster 2022-04-22T17:31:48.019697+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2022-04-22T17:31:49.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:49 smithi121 bash[36902]: audit 2022-04-22T17:31:48.683346+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.115:0/3081142953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:31:49.803 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:49 smithi115 bash[55827]: debug 2022-04-22T17:31:49.406+0000 7f1842eeb3c0 -1 Falling back to public interface 2022-04-22T17:31:49.812 INFO:tasks.rados.rados.0.smithi115.stdout:1746: done (1 left) 2022-04-22T17:31:49.812 INFO:tasks.rados.rados.0.smithi115.stdout:1747: done (0 left) 2022-04-22T17:31:49.812 INFO:tasks.rados.rados.0.smithi115.stdout:1748: delete oid 42 current snap is 176 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:1748: done (0 left) 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:1749: copy_from oid 11 from oid 21 current snap is 176 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:1750: rollback oid 32 current snap is 176 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 157 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:1751: read oid 39 snap -1 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:1751: expect deleted 2022-04-22T17:31:49.833 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 176 2022-04-22T17:31:49.834 INFO:tasks.rados.rados.0.smithi115.stdout:1752: seq_num 529 ranges {2252800=466944,2719744=647168,3366912=532480,3899392=24576} 2022-04-22T17:31:49.851 INFO:tasks.rados.rados.0.smithi115.stdout:1752: writing smithi11541303-50 from 2252800 to 2719744 tid 1 2022-04-22T17:31:49.855 INFO:tasks.rados.rados.0.smithi115.stdout:1752: writing smithi11541303-50 from 2719744 to 3366912 tid 2 2022-04-22T17:31:49.859 INFO:tasks.rados.rados.0.smithi115.stdout:1752: writing smithi11541303-50 from 3366912 to 3899392 tid 3 2022-04-22T17:31:49.859 INFO:tasks.rados.rados.0.smithi115.stdout:1752: writing smithi11541303-50 from 3899392 to 3923968 tid 4 2022-04-22T17:31:49.859 INFO:tasks.rados.rados.0.smithi115.stdout:1753: snap_remove snap 163 2022-04-22T17:31:50.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: audit 2022-04-22T17:31:48.252598+0000 mgr.y (mgr.34107) 191 : audit [DBG] from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:50.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: cluster 2022-04-22T17:31:49.035234+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: audit 2022-04-22T17:31:49.228740+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: cluster 2022-04-22T17:31:49.788927+0000 mon.a (mon.0) 626 : cluster [WRN] Health check update: Degraded data redundancy: 35/387 objects degraded (9.044%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: cluster 2022-04-22T17:31:49.796193+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[46934]: audit 2022-04-22T17:31:49.967370+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: audit 2022-04-22T17:31:48.252598+0000 mgr.y (mgr.34107) 191 : audit [DBG] from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:50.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: cluster 2022-04-22T17:31:49.035234+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2022-04-22T17:31:50.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: audit 2022-04-22T17:31:49.228740+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:50.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: cluster 2022-04-22T17:31:49.788927+0000 mon.a (mon.0) 626 : cluster [WRN] Health check update: Degraded data redundancy: 35/387 objects degraded (9.044%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:50.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: cluster 2022-04-22T17:31:49.796193+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2022-04-22T17:31:50.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:50 smithi115 bash[45495]: audit 2022-04-22T17:31:49.967370+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: audit 2022-04-22T17:31:48.252598+0000 mgr.y (mgr.34107) 191 : audit [DBG] from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:31:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: cluster 2022-04-22T17:31:49.035234+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2022-04-22T17:31:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: audit 2022-04-22T17:31:49.228740+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: cluster 2022-04-22T17:31:49.788927+0000 mon.a (mon.0) 626 : cluster [WRN] Health check update: Degraded data redundancy: 35/387 objects degraded (9.044%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: cluster 2022-04-22T17:31:49.796193+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2022-04-22T17:31:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[36902]: audit 2022-04-22T17:31:49.967370+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:1753: done (4 left) 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:1754: rollback oid 26 current snap is 176 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 157 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:1750: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:1750: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:31:50.828 INFO:tasks.rados.rados.0.smithi115.stdout:1750: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 563 (ObjNum 151 snap 41 seq_num 151) dirty exists 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 1 to smithi11541303-50 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1749: finishing copy_from to smithi11541303-11 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 981 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1749: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 2 to smithi11541303-50 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1749: done (4 left) 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1750: done (3 left) 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1751: done (2 left) 2022-04-22T17:31:50.829 INFO:tasks.rados.rados.0.smithi115.stdout:1755: delete oid 48 current snap is 176 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 3 to smithi11541303-50 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 4 to smithi11541303-50 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 5 to smithi11541303-50 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:1752: finishing write tid 6 to smithi11541303-50 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 716 (ObjNum 529 snap 176 seq_num 529) dirty exists 2022-04-22T17:31:50.830 INFO:tasks.rados.rados.0.smithi115.stdout:1752: left oid 50 (ObjNum 529 snap 176 seq_num 529) 2022-04-22T17:31:50.838 INFO:tasks.rados.rados.0.smithi115.stdout:1754: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:31:50.839 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 0 (ObjNum -1479266496 snap 32692 seq_num 2815700800) dirty dne 2022-04-22T17:31:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1752: done (2 left) 2022-04-22T17:31:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1754: done (1 left) 2022-04-22T17:31:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1755: done (0 left) 2022-04-22T17:31:50.844 INFO:tasks.rados.rados.0.smithi115.stdout:1756: delete oid 27 current snap is 176 2022-04-22T17:31:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1756: done (0 left) 2022-04-22T17:31:50.851 INFO:tasks.rados.rados.0.smithi115.stdout:1757: snap_remove snap 172 2022-04-22T17:31:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:51 smithi121 bash[36902]: cluster 2022-04-22T17:31:49.209903+0000 mgr.y (mgr.34107) 192 : cluster [DBG] pgmap v307: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:51 smithi121 bash[36902]: audit 2022-04-22T17:31:50.529266+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:51 smithi121 bash[36902]: cluster 2022-04-22T17:31:50.811653+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2022-04-22T17:31:51.267 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:31:50 smithi121 bash[24219]: debug 2022-04-22T17:31:50.824+0000 7f1defc41700 -1 osd.7 pg_epoch: 415 pg[5.4s0( v 415'724 (0'0,415'724] local-lis/les=392/393 n=5 ec=63/63 lis/c=392/392 les/c/f=393/393/0 sis=392) [7,2,5]p7(0) r=0 lpr=392 luod=407'722 crt=407'722 lcod 404'720 mlcod 404'720 active+clean+snaptrim trimq=[140~1] ps=[14e~1,152~2,158~1]] removing snap head 2022-04-22T17:31:51.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[45495]: cluster 2022-04-22T17:31:49.209903+0000 mgr.y (mgr.34107) 192 : cluster [DBG] pgmap v307: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:51.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[45495]: audit 2022-04-22T17:31:50.529266+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:51.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[45495]: cluster 2022-04-22T17:31:50.811653+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2022-04-22T17:31:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[46934]: cluster 2022-04-22T17:31:49.209903+0000 mgr.y (mgr.34107) 192 : cluster [DBG] pgmap v307: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[46934]: audit 2022-04-22T17:31:50.529266+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:31:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[46934]: cluster 2022-04-22T17:31:50.811653+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2022-04-22T17:31:51.830 INFO:tasks.rados.rados.0.smithi115.stdout:1757: done (0 left) 2022-04-22T17:31:51.831 INFO:tasks.rados.rados.0.smithi115.stdout:1758: setattr oid 36 current snap is 176 2022-04-22T17:31:51.832 INFO:tasks.rados.rados.0.smithi115.stdout:1759: copy_from oid 38 from oid 1 current snap is 176 2022-04-22T17:31:51.833 INFO:tasks.rados.rados.0.smithi115.stdout:1760: rollback oid 15 current snap is 176 2022-04-22T17:31:51.833 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 175 2022-04-22T17:31:51.833 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 176 2022-04-22T17:31:51.833 INFO:tasks.rados.rados.0.smithi115.stdout:1761: seq_num 532 ranges {3399680=516096,3915776=696320,4612096=507904} 2022-04-22T17:31:51.857 INFO:tasks.rados.rados.0.smithi115.stdout:1761: writing smithi11541303-22 from 3399680 to 3915776 tid 1 2022-04-22T17:31:51.862 INFO:tasks.rados.rados.0.smithi115.stdout:1761: writing smithi11541303-22 from 3915776 to 4612096 tid 2 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:1761: writing smithi11541303-22 from 4612096 to 5120000 tid 3 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:1762: rollback oid 10 current snap is 176 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 157 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 767 (ObjNum 521 snap 172 seq_num 521) dirty exists 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:1758: done (4 left) 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:1763: snap_create 2022-04-22T17:31:51.866 INFO:tasks.rados.rados.0.smithi115.stdout:1760: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:1760: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:1760: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 714 (ObjNum 490 snap 169 seq_num 490) dirty exists 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:1759: finishing copy_from to smithi11541303-38 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 766 (ObjNum 526 snap 173 seq_num 526) dirty exists 2022-04-22T17:31:51.867 INFO:tasks.rados.rados.0.smithi115.stdout:1759: finishing copy_from racing read to smithi11541303-38 2022-04-22T17:31:51.869 INFO:tasks.rados.rados.0.smithi115.stdout:1761: finishing write tid 1 to smithi11541303-22 2022-04-22T17:31:51.871 INFO:tasks.rados.rados.0.smithi115.stdout:1762: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:31:51.872 INFO:tasks.rados.rados.0.smithi115.stdout:1762: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:31:51.872 INFO:tasks.rados.rados.0.smithi115.stdout:1761: finishing write tid 2 to smithi11541303-22 2022-04-22T17:31:51.873 INFO:tasks.rados.rados.0.smithi115.stdout:1761: finishing write tid 3 to smithi11541303-22 2022-04-22T17:31:51.873 INFO:tasks.rados.rados.0.smithi115.stdout:1761: finishing write tid 4 to smithi11541303-22 2022-04-22T17:31:51.874 INFO:tasks.rados.rados.0.smithi115.stdout:1762: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:31:51.874 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 730 (ObjNum 458 snap 157 seq_num 458) dirty exists 2022-04-22T17:31:51.875 INFO:tasks.rados.rados.0.smithi115.stdout:1761: finishing write tid 5 to smithi11541303-22 2022-04-22T17:31:51.875 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 990 (ObjNum 532 snap 176 seq_num 532) dirty exists 2022-04-22T17:31:51.875 INFO:tasks.rados.rados.0.smithi115.stdout:1761: left oid 22 (ObjNum 532 snap 176 seq_num 532) 2022-04-22T17:31:52.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:51.624Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:52.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:51.624Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:52.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:51.624Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1759: done (4 left) 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1760: done (3 left) 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1761: done (2 left) 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1762: done (1 left) 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1763: done (0 left) 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 177 2022-04-22T17:31:52.839 INFO:tasks.rados.rados.0.smithi115.stdout:1764: seq_num 533 ranges {1589248=532480,2121728=630784,2752512=516096,3268608=57344} 2022-04-22T17:31:52.852 INFO:tasks.rados.rados.0.smithi115.stdout:1764: writing smithi11541303-8 from 1589248 to 2121728 tid 1 2022-04-22T17:31:52.855 INFO:tasks.rados.rados.0.smithi115.stdout:1764: writing smithi11541303-8 from 2121728 to 2752512 tid 2 2022-04-22T17:31:52.859 INFO:tasks.rados.rados.0.smithi115.stdout:1764: writing smithi11541303-8 from 2752512 to 3268608 tid 3 2022-04-22T17:31:52.859 INFO:tasks.rados.rados.0.smithi115.stdout:1764: writing smithi11541303-8 from 3268608 to 3325952 tid 4 2022-04-22T17:31:52.859 INFO:tasks.rados.rados.0.smithi115.stdout:1765: delete oid 12 current snap is 177 2022-04-22T17:31:52.862 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 1 to smithi11541303-8 2022-04-22T17:31:52.863 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 2 to smithi11541303-8 2022-04-22T17:31:52.863 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 3 to smithi11541303-8 2022-04-22T17:31:52.864 INFO:tasks.rados.rados.0.smithi115.stdout:1765: done (1 left) 2022-04-22T17:31:52.864 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 177 2022-04-22T17:31:52.864 INFO:tasks.rados.rados.0.smithi115.stdout:1766: seq_num 535 ranges {3252224=565248,3817472=802816,4620288=401408} 2022-04-22T17:31:52.885 INFO:tasks.rados.rados.0.smithi115.stdout:1766: writing smithi11541303-35 from 3252224 to 3817472 tid 1 2022-04-22T17:31:52.892 INFO:tasks.rados.rados.0.smithi115.stdout:1766: writing smithi11541303-35 from 3817472 to 4620288 tid 2 2022-04-22T17:31:52.894 INFO:tasks.rados.rados.0.smithi115.stdout:1766: writing smithi11541303-35 from 4620288 to 5021696 tid 3 2022-04-22T17:31:52.894 INFO:tasks.rados.rados.0.smithi115.stdout:1767: delete oid 4 current snap is 177 2022-04-22T17:31:52.895 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 4 to smithi11541303-8 2022-04-22T17:31:52.895 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 5 to smithi11541303-8 2022-04-22T17:31:52.895 INFO:tasks.rados.rados.0.smithi115.stdout:1764: finishing write tid 6 to smithi11541303-8 2022-04-22T17:31:52.895 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 772 (ObjNum 533 snap 177 seq_num 533) dirty exists 2022-04-22T17:31:52.895 INFO:tasks.rados.rados.0.smithi115.stdout:1764: left oid 8 (ObjNum 533 snap 177 seq_num 533) 2022-04-22T17:31:52.896 INFO:tasks.rados.rados.0.smithi115.stdout:1766: finishing write tid 1 to smithi11541303-35 2022-04-22T17:31:52.900 INFO:tasks.rados.rados.0.smithi115.stdout:1764: done (2 left) 2022-04-22T17:31:52.901 INFO:tasks.rados.rados.0.smithi115.stdout:1767: done (1 left) 2022-04-22T17:31:52.901 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 177 2022-04-22T17:31:52.901 INFO:tasks.rados.rados.0.smithi115.stdout:1768: seq_num 537 ranges {0=598016,598016=573440,1171456=491520,1662976=139264} 2022-04-22T17:31:52.904 INFO:tasks.rados.rados.0.smithi115.stdout:1768: writing smithi11541303-47 from 0 to 598016 tid 1 2022-04-22T17:31:52.908 INFO:tasks.rados.rados.0.smithi115.stdout:1768: writing smithi11541303-47 from 598016 to 1171456 tid 2 2022-04-22T17:31:52.912 INFO:tasks.rados.rados.0.smithi115.stdout:1768: writing smithi11541303-47 from 1171456 to 1662976 tid 3 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1768: writing smithi11541303-47 from 1662976 to 1802240 tid 4 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1769: rmattr oid 23 current snap is 177 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1769: done (2 left) 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1770: copy_from oid 30 from oid 2 current snap is 177 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1766: finishing write tid 2 to smithi11541303-35 2022-04-22T17:31:52.914 INFO:tasks.rados.rados.0.smithi115.stdout:1771: copy_from oid 7 from oid 46 current snap is 177 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1766: finishing write tid 3 to smithi11541303-35 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1766: finishing write tid 4 to smithi11541303-35 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1766: finishing write tid 5 to smithi11541303-35 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1058 (ObjNum 535 snap 177 seq_num 535) dirty exists 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1766: left oid 35 (ObjNum 535 snap 177 seq_num 535) 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 1 to smithi11541303-47 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 2 to smithi11541303-47 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1766: done (3 left) 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 177 2022-04-22T17:31:52.915 INFO:tasks.rados.rados.0.smithi115.stdout:1772: seq_num 538 ranges {7700480=614400,8314880=450560,8765440=704512,9469952=49152} 2022-04-22T17:31:52.961 INFO:tasks.rados.rados.0.smithi115.stdout:1772: writing smithi11541303-6 from 7700480 to 8314880 tid 1 2022-04-22T17:31:52.964 INFO:tasks.rados.rados.0.smithi115.stdout:1772: writing smithi11541303-6 from 8314880 to 8765440 tid 2 2022-04-22T17:31:52.970 INFO:tasks.rados.rados.0.smithi115.stdout:1772: writing smithi11541303-6 from 8765440 to 9469952 tid 3 2022-04-22T17:31:52.970 INFO:tasks.rados.rados.0.smithi115.stdout:1772: writing smithi11541303-6 from 9469952 to 9519104 tid 4 2022-04-22T17:31:52.970 INFO:tasks.rados.rados.0.smithi115.stdout:1773: delete oid 44 current snap is 177 2022-04-22T17:31:52.970 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 3 to smithi11541303-47 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 4 to smithi11541303-47 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 5 to smithi11541303-47 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1768: finishing write tid 6 to smithi11541303-47 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 735 (ObjNum 537 snap 177 seq_num 537) dirty exists 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1768: left oid 47 (ObjNum 537 snap 177 seq_num 537) 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1771: finishing copy_from to smithi11541303-7 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 883 (ObjNum 495 snap 169 seq_num 495) dirty exists 2022-04-22T17:31:52.971 INFO:tasks.rados.rados.0.smithi115.stdout:1771: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:31:52.972 INFO:tasks.rados.rados.0.smithi115.stdout:1770: finishing copy_from to smithi11541303-30 2022-04-22T17:31:52.972 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1051 (ObjNum 443 snap 148 seq_num 443) dirty exists 2022-04-22T17:31:52.972 INFO:tasks.rados.rados.0.smithi115.stdout:1770: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:31:52.972 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 1 to smithi11541303-6 2022-04-22T17:31:52.972 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 2 to smithi11541303-6 2022-04-22T17:31:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:1768: done (4 left) 2022-04-22T17:31:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:1770: done (3 left) 2022-04-22T17:31:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:1771: done (2 left) 2022-04-22T17:31:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:1773: done (1 left) 2022-04-22T17:31:52.974 INFO:tasks.rados.rados.0.smithi115.stdout:1774: read oid 29 snap -1 2022-04-22T17:31:52.974 INFO:tasks.rados.rados.0.smithi115.stdout:1774: expect (ObjNum 517 snap 170 seq_num 517) 2022-04-22T17:31:52.974 INFO:tasks.rados.rados.0.smithi115.stdout:1775: delete oid 40 current snap is 177 2022-04-22T17:31:52.974 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 3 to smithi11541303-6 2022-04-22T17:31:52.975 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 4 to smithi11541303-6 2022-04-22T17:31:52.975 INFO:tasks.rados.rados.0.smithi115.stdout:1775: done (2 left) 2022-04-22T17:31:52.976 INFO:tasks.rados.rados.0.smithi115.stdout:1776: snap_create 2022-04-22T17:31:52.976 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 5 to smithi11541303-6 2022-04-22T17:31:52.978 INFO:tasks.rados.rados.0.smithi115.stdout:1772: finishing write tid 6 to smithi11541303-6 2022-04-22T17:31:52.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 659 (ObjNum 538 snap 177 seq_num 538) dirty exists 2022-04-22T17:31:52.978 INFO:tasks.rados.rados.0.smithi115.stdout:1772: left oid 6 (ObjNum 538 snap 177 seq_num 538) 2022-04-22T17:31:52.984 INFO:tasks.rados.rados.0.smithi115.stdout:1774: expect (ObjNum 517 snap 170 seq_num 517) 2022-04-22T17:31:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:52 smithi121 bash[36902]: cluster 2022-04-22T17:31:51.210934+0000 mgr.y (mgr.34107) 193 : cluster [DBG] pgmap v310: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 792 KiB/s wr, 12 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:53.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:52 smithi121 bash[36902]: cluster 2022-04-22T17:31:51.814766+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2022-04-22T17:31:53.303 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:52 smithi115 bash[55827]: debug 2022-04-22T17:31:52.922+0000 7f1842eeb3c0 -1 osd.3 405 log_to_monitors true 2022-04-22T17:31:53.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:52 smithi115 bash[45495]: cluster 2022-04-22T17:31:51.210934+0000 mgr.y (mgr.34107) 193 : cluster [DBG] pgmap v310: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 792 KiB/s wr, 12 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:53.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:52 smithi115 bash[45495]: cluster 2022-04-22T17:31:51.814766+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2022-04-22T17:31:53.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:52 smithi115 bash[46934]: cluster 2022-04-22T17:31:51.210934+0000 mgr.y (mgr.34107) 193 : cluster [DBG] pgmap v310: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 189 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 792 KiB/s wr, 12 op/s; 36/396 objects degraded (9.091%) 2022-04-22T17:31:53.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:52 smithi115 bash[46934]: cluster 2022-04-22T17:31:51.814766+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2022-04-22T17:31:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:53.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:53.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:31:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:31:53.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:31:53.841 INFO:tasks.rados.rados.0.smithi115.stdout:1772: done (2 left) 2022-04-22T17:31:53.841 INFO:tasks.rados.rados.0.smithi115.stdout:1774: done (1 left) 2022-04-22T17:31:53.841 INFO:tasks.rados.rados.0.smithi115.stdout:1776: done (0 left) 2022-04-22T17:31:53.841 INFO:tasks.rados.rados.0.smithi115.stdout:1777: delete oid 9 current snap is 178 2022-04-22T17:31:53.858 INFO:tasks.rados.rados.0.smithi115.stdout:1777: done (0 left) 2022-04-22T17:31:53.858 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 178 2022-04-22T17:31:53.858 INFO:tasks.rados.rados.0.smithi115.stdout:1778: seq_num 542 ranges {0=679936,679936=442368,1122304=679936,1802240=81920} 2022-04-22T17:31:53.858 INFO:tasks.rados.rados.0.smithi115.stdout:1778: writing smithi11541303-39 from 0 to 679936 tid 1 2022-04-22T17:31:53.862 INFO:tasks.rados.rados.0.smithi115.stdout:1778: writing smithi11541303-39 from 679936 to 1122304 tid 2 2022-04-22T17:31:53.867 INFO:tasks.rados.rados.0.smithi115.stdout:1778: writing smithi11541303-39 from 1122304 to 1802240 tid 3 2022-04-22T17:31:53.869 INFO:tasks.rados.rados.0.smithi115.stdout:1778: writing smithi11541303-39 from 1802240 to 1884160 tid 4 2022-04-22T17:31:53.869 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 1 to smithi11541303-39 2022-04-22T17:31:53.869 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 178 2022-04-22T17:31:53.869 INFO:tasks.rados.rados.0.smithi115.stdout:1779: seq_num 543 ranges {1802240=704512,2506752=729088,3235840=475136} 2022-04-22T17:31:53.883 INFO:tasks.rados.rados.0.smithi115.stdout:1779: writing smithi11541303-47 from 1802240 to 2506752 tid 1 2022-04-22T17:31:53.888 INFO:tasks.rados.rados.0.smithi115.stdout:1779: writing smithi11541303-47 from 2506752 to 3235840 tid 2 2022-04-22T17:31:53.890 INFO:tasks.rados.rados.0.smithi115.stdout:1779: writing smithi11541303-47 from 3235840 to 3710976 tid 3 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1780: read oid 32 snap -1 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1780: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1781: delete oid 25 current snap is 178 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 2 to smithi11541303-39 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 3 to smithi11541303-39 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 4 to smithi11541303-39 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 5 to smithi11541303-39 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:1778: finishing write tid 6 to smithi11541303-39 2022-04-22T17:31:53.891 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1063 (ObjNum 542 snap 178 seq_num 542) dirty exists 2022-04-22T17:31:53.892 INFO:tasks.rados.rados.0.smithi115.stdout:1778: left oid 39 (ObjNum 542 snap 178 seq_num 542) 2022-04-22T17:31:53.898 INFO:tasks.rados.rados.0.smithi115.stdout:1779: finishing write tid 1 to smithi11541303-47 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1779: finishing write tid 2 to smithi11541303-47 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1778: done (3 left) 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1781: done (2 left) 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1782: rollback oid 10 current snap is 178 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 173 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1783: read oid 19 snap 158 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1783: expect (ObjNum 430 snap 147 seq_num 430) 2022-04-22T17:31:53.901 INFO:tasks.rados.rados.0.smithi115.stdout:1784: read oid 3 snap 157 2022-04-22T17:31:53.902 INFO:tasks.rados.rados.0.smithi115.stdout:1784: expect (ObjNum 348 snap 121 seq_num 348) 2022-04-22T17:31:53.902 INFO:tasks.rados.rados.0.smithi115.stdout:1785: copy_from oid 7 from oid 20 current snap is 178 2022-04-22T17:31:53.902 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 178 2022-04-22T17:31:53.902 INFO:tasks.rados.rados.0.smithi115.stdout:1786: seq_num 545 ranges {1949696=737280,2686976=491520,3178496=450560,3629056=262144} 2022-04-22T17:31:53.922 INFO:tasks.rados.rados.0.smithi115.stdout:1786: writing smithi11541303-33 from 1949696 to 2686976 tid 1 2022-04-22T17:31:53.926 INFO:tasks.rados.rados.0.smithi115.stdout:1786: writing smithi11541303-33 from 2686976 to 3178496 tid 2 2022-04-22T17:31:53.929 INFO:tasks.rados.rados.0.smithi115.stdout:1786: writing smithi11541303-33 from 3178496 to 3629056 tid 3 2022-04-22T17:31:53.931 INFO:tasks.rados.rados.0.smithi115.stdout:1786: writing smithi11541303-33 from 3629056 to 3891200 tid 4 2022-04-22T17:31:53.931 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 178 2022-04-22T17:31:53.932 INFO:tasks.rados.rados.0.smithi115.stdout:1787: seq_num 546 ranges {0=753664,753664=778240,1531904=425984} 2022-04-22T17:31:53.936 INFO:tasks.rados.rados.0.smithi115.stdout:1787: writing smithi11541303-44 from 0 to 753664 tid 1 2022-04-22T17:31:53.941 INFO:tasks.rados.rados.0.smithi115.stdout:1787: writing smithi11541303-44 from 753664 to 1531904 tid 2 2022-04-22T17:31:53.944 INFO:tasks.rados.rados.0.smithi115.stdout:1787: writing smithi11541303-44 from 1531904 to 1957888 tid 3 2022-04-22T17:31:53.944 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 178 2022-04-22T17:31:53.944 INFO:tasks.rados.rados.0.smithi115.stdout:1779: finishing write tid 3 to smithi11541303-47 2022-04-22T17:31:53.944 INFO:tasks.rados.rados.0.smithi115.stdout:1779: finishing write tid 4 to smithi11541303-47 2022-04-22T17:31:53.944 INFO:tasks.rados.rados.0.smithi115.stdout:1779: finishing write tid 5 to smithi11541303-47 2022-04-22T17:31:53.945 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 741 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:31:53.945 INFO:tasks.rados.rados.0.smithi115.stdout:1779: left oid 47 (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:31:53.945 INFO:tasks.rados.rados.0.smithi115.stdout:1788: seq_num 547 ranges {6840320=770048,7610368=663552,8273920=466944,8740864=73728} 2022-04-22T17:31:53.986 INFO:tasks.rados.rados.0.smithi115.stdout:1788: writing smithi11541303-37 from 6840320 to 7610368 tid 1 2022-04-22T17:31:53.990 INFO:tasks.rados.rados.0.smithi115.stdout:1788: writing smithi11541303-37 from 7610368 to 8273920 tid 2 2022-04-22T17:31:53.992 INFO:tasks.rados.rados.0.smithi115.stdout:1788: writing smithi11541303-37 from 8273920 to 8740864 tid 3 2022-04-22T17:31:53.993 INFO:tasks.rados.rados.0.smithi115.stdout:1788: writing smithi11541303-37 from 8740864 to 8814592 tid 4 2022-04-22T17:31:53.994 INFO:tasks.rados.rados.0.smithi115.stdout:1779: done (8 left) 2022-04-22T17:31:53.994 INFO:tasks.rados.rados.0.smithi115.stdout:1789: snap_create 2022-04-22T17:31:53.994 INFO:tasks.rados.rados.0.smithi115.stdout:1782: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:31:53.994 INFO:tasks.rados.rados.0.smithi115.stdout:1784: expect (ObjNum 348 snap 121 seq_num 348) 2022-04-22T17:31:54.026 INFO:tasks.rados.rados.0.smithi115.stdout:1782: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:31:54.026 INFO:tasks.rados.rados.0.smithi115.stdout:1780: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:31:54.241 INFO:tasks.rados.rados.0.smithi115.stdout:1782: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:31:54.241 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 745 (ObjNum 458 snap 157 seq_num 458) dirty exists 2022-04-22T17:31:54.242 INFO:tasks.rados.rados.0.smithi115.stdout:1783: expect (ObjNum 430 snap 147 seq_num 430) 2022-04-22T17:31:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:53 smithi121 bash[36902]: cluster 2022-04-22T17:31:52.822737+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2022-04-22T17:31:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:53 smithi121 bash[36902]: audit 2022-04-22T17:31:52.928400+0000 mon.a (mon.0) 633 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:31:54.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[46934]: cluster 2022-04-22T17:31:52.822737+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2022-04-22T17:31:54.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[46934]: audit 2022-04-22T17:31:52.928400+0000 mon.a (mon.0) 633 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:31:54.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[45495]: cluster 2022-04-22T17:31:52.822737+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2022-04-22T17:31:54.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[45495]: audit 2022-04-22T17:31:52.928400+0000 mon.a (mon.0) 633 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-04-22T17:31:54.304 INFO:journalctl@ceph.osd.3.smithi115.stdout:Apr 22 17:31:53 smithi115 bash[55827]: debug 2022-04-22T17:31:53.858+0000 7f1834a87700 -1 osd.3 405 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:31:54.394 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 1 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 2 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 3 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 4 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 5 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: finishing write tid 6 to smithi11541303-33 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 568 (ObjNum 545 snap 178 seq_num 545) dirty exists 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1786: left oid 33 (ObjNum 545 snap 178 seq_num 545) 2022-04-22T17:31:54.395 INFO:tasks.rados.rados.0.smithi115.stdout:1787: finishing write tid 1 to smithi11541303-44 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1787: finishing write tid 2 to smithi11541303-44 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1787: finishing write tid 3 to smithi11541303-44 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1787: finishing write tid 4 to smithi11541303-44 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1787: finishing write tid 5 to smithi11541303-44 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 88 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1787: left oid 44 (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1785: finishing copy_from to smithi11541303-7 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 804 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1785: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:31:54.396 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 1 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 2 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 3 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 4 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 5 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: finishing write tid 6 to smithi11541303-37 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1068 (ObjNum 547 snap 178 seq_num 547) dirty exists 2022-04-22T17:31:54.397 INFO:tasks.rados.rados.0.smithi115.stdout:1788: left oid 37 (ObjNum 547 snap 178 seq_num 547) 2022-04-22T17:31:54.860 INFO:tasks.rados.rados.0.smithi115.stdout:1780: done (8 left) 2022-04-22T17:31:54.860 INFO:tasks.rados.rados.0.smithi115.stdout:1782: done (7 left) 2022-04-22T17:31:54.860 INFO:tasks.rados.rados.0.smithi115.stdout:1783: done (6 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1784: done (5 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1785: done (4 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1786: done (3 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1787: done (2 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1788: done (1 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1789: done (0 left) 2022-04-22T17:31:54.861 INFO:tasks.rados.rados.0.smithi115.stdout:1790: delete oid 40 current snap is 179 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1790: done (0 left) 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1791: read oid 17 snap 157 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1791: expect (ObjNum 457 snap 157 seq_num 457) 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1792: rollback oid 10 current snap is 179 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 176 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1793: rollback oid 16 current snap is 179 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 175 2022-04-22T17:31:54.875 INFO:tasks.rados.rados.0.smithi115.stdout:1794: snap_create 2022-04-22T17:31:54.889 INFO:tasks.rados.rados.0.smithi115.stdout:1791: expect (ObjNum 457 snap 157 seq_num 457) 2022-04-22T17:31:55.036 INFO:tasks.rados.rados.0.smithi115.stdout:1792: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:31:55.037 INFO:tasks.rados.rados.0.smithi115.stdout:1792: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:31:55.037 INFO:tasks.rados.rados.0.smithi115.stdout:1792: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:31:55.037 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 750 (ObjNum 458 snap 157 seq_num 458) dirty exists 2022-04-22T17:31:55.209 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: cluster 2022-04-22T17:31:53.211940+0000 mgr.y (mgr.34107) 194 : cluster [DBG] pgmap v313: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 193 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 828 KiB/s wr, 8 op/s; 38/405 objects degraded (9.383%) 2022-04-22T17:31:55.209 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: audit 2022-04-22T17:31:53.824779+0000 mon.a (mon.0) 634 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: cluster 2022-04-22T17:31:53.824866+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: audit 2022-04-22T17:31:53.825681+0000 mon.a (mon.0) 636 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: audit 2022-04-22T17:31:54.231532+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[45495]: cluster 2022-04-22T17:31:54.791373+0000 mon.a (mon.0) 638 : cluster [WRN] Health check update: Degraded data redundancy: 38/405 objects degraded (9.383%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: cluster 2022-04-22T17:31:53.211940+0000 mgr.y (mgr.34107) 194 : cluster [DBG] pgmap v313: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 193 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 828 KiB/s wr, 8 op/s; 38/405 objects degraded (9.383%) 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: audit 2022-04-22T17:31:53.824779+0000 mon.a (mon.0) 634 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:31:55.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: cluster 2022-04-22T17:31:53.824866+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2022-04-22T17:31:55.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: audit 2022-04-22T17:31:53.825681+0000 mon.a (mon.0) 636 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:55.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: audit 2022-04-22T17:31:54.231532+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:55.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:54 smithi115 bash[46934]: cluster 2022-04-22T17:31:54.791373+0000 mon.a (mon.0) 638 : cluster [WRN] Health check update: Degraded data redundancy: 38/405 objects degraded (9.383%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: cluster 2022-04-22T17:31:53.211940+0000 mgr.y (mgr.34107) 194 : cluster [DBG] pgmap v313: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 193 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 828 KiB/s wr, 8 op/s; 38/405 objects degraded (9.383%) 2022-04-22T17:31:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: audit 2022-04-22T17:31:53.824779+0000 mon.a (mon.0) 634 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-04-22T17:31:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: cluster 2022-04-22T17:31:53.824866+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2022-04-22T17:31:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: audit 2022-04-22T17:31:53.825681+0000 mon.a (mon.0) 636 : audit [INF] from='osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-04-22T17:31:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: audit 2022-04-22T17:31:54.231532+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:31:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:54 smithi121 bash[36902]: cluster 2022-04-22T17:31:54.791373+0000 mon.a (mon.0) 638 : cluster [WRN] Health check update: Degraded data redundancy: 38/405 objects degraded (9.383%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:31:55.875 INFO:tasks.rados.rados.0.smithi115.stdout:1791: done (3 left) 2022-04-22T17:31:55.885 INFO:tasks.rados.rados.0.smithi115.stdout:1792: done (2 left) 2022-04-22T17:31:55.885 INFO:tasks.rados.rados.0.smithi115.stdout:1794: done (1 left) 2022-04-22T17:31:55.885 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 180 2022-04-22T17:31:55.885 INFO:tasks.rados.rados.0.smithi115.stdout:1795: seq_num 549 ranges {1662976=802816,2465792=483328,2949120=434176,3383296=286720} 2022-04-22T17:31:55.891 INFO:tasks.rados.rados.0.smithi115.stdout:1795: writing smithi11541303-3 from 1662976 to 2465792 tid 1 2022-04-22T17:31:55.895 INFO:tasks.rados.rados.0.smithi115.stdout:1795: writing smithi11541303-3 from 2465792 to 2949120 tid 2 2022-04-22T17:31:55.898 INFO:tasks.rados.rados.0.smithi115.stdout:1795: writing smithi11541303-3 from 2949120 to 3383296 tid 3 2022-04-22T17:31:55.900 INFO:tasks.rados.rados.0.smithi115.stdout:1795: writing smithi11541303-3 from 3383296 to 3670016 tid 4 2022-04-22T17:31:55.900 INFO:tasks.rados.rados.0.smithi115.stdout:1796: snap_create 2022-04-22T17:31:55.912 INFO:tasks.rados.rados.0.smithi115.stdout:1793: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:31:55.912 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 408 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:31:55.914 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 1 to smithi11541303-3 2022-04-22T17:31:55.915 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 2 to smithi11541303-3 2022-04-22T17:31:55.915 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 3 to smithi11541303-3 2022-04-22T17:31:55.915 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 4 to smithi11541303-3 2022-04-22T17:31:55.915 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 5 to smithi11541303-3 2022-04-22T17:31:55.916 INFO:tasks.rados.rados.0.smithi115.stdout:1795: finishing write tid 6 to smithi11541303-3 2022-04-22T17:31:55.916 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 986 (ObjNum 549 snap 180 seq_num 549) dirty exists 2022-04-22T17:31:55.917 INFO:tasks.rados.rados.0.smithi115.stdout:1795: left oid 3 (ObjNum 549 snap 180 seq_num 549) 2022-04-22T17:31:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:55 smithi121 bash[36902]: cluster 2022-04-22T17:31:54.832545+0000 mon.a (mon.0) 639 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:55 smithi121 bash[36902]: cluster 2022-04-22T17:31:54.840170+0000 mon.a (mon.0) 640 : cluster [INF] osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414] boot 2022-04-22T17:31:56.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:55 smithi121 bash[36902]: cluster 2022-04-22T17:31:54.840281+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2022-04-22T17:31:56.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:55 smithi121 bash[36902]: audit 2022-04-22T17:31:54.843527+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:31:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[45495]: cluster 2022-04-22T17:31:54.832545+0000 mon.a (mon.0) 639 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[45495]: cluster 2022-04-22T17:31:54.840170+0000 mon.a (mon.0) 640 : cluster [INF] osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414] boot 2022-04-22T17:31:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[45495]: cluster 2022-04-22T17:31:54.840281+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2022-04-22T17:31:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[45495]: audit 2022-04-22T17:31:54.843527+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:31:56.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[46934]: cluster 2022-04-22T17:31:54.832545+0000 mon.a (mon.0) 639 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:31:56.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[46934]: cluster 2022-04-22T17:31:54.840170+0000 mon.a (mon.0) 640 : cluster [INF] osd.3 [v2:172.21.15.115:6826/194469414,v1:172.21.15.115:6827/194469414] boot 2022-04-22T17:31:56.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[46934]: cluster 2022-04-22T17:31:54.840281+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2022-04-22T17:31:56.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:55 smithi115 bash[46934]: audit 2022-04-22T17:31:54.843527+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-04-22T17:31:56.894 INFO:tasks.rados.rados.0.smithi115.stdout:1793: done (2 left) 2022-04-22T17:31:56.911 INFO:tasks.rados.rados.0.smithi115.stdout:1795: done (1 left) 2022-04-22T17:31:56.911 INFO:tasks.rados.rados.0.smithi115.stdout:1796: done (0 left) 2022-04-22T17:31:56.911 INFO:tasks.rados.rados.0.smithi115.stdout:1797: setattr oid 29 current snap is 181 2022-04-22T17:31:56.912 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 181 2022-04-22T17:31:56.912 INFO:tasks.rados.rados.0.smithi115.stdout:1798: seq_num 550 ranges {2613248=417792,3031040=770048,3801088=638976,4440064=196608} 2022-04-22T17:31:56.916 INFO:tasks.rados.rados.0.smithi115.stdout:1798: writing smithi11541303-31 from 2613248 to 3031040 tid 1 2022-04-22T17:31:56.920 INFO:tasks.rados.rados.0.smithi115.stdout:1798: writing smithi11541303-31 from 3031040 to 3801088 tid 2 2022-04-22T17:31:56.923 INFO:tasks.rados.rados.0.smithi115.stdout:1798: writing smithi11541303-31 from 3801088 to 4440064 tid 3 2022-04-22T17:31:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:1798: writing smithi11541303-31 from 4440064 to 4636672 tid 4 2022-04-22T17:31:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 181 2022-04-22T17:31:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 492 (ObjNum 517 snap 170 seq_num 517) dirty exists 2022-04-22T17:31:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:1799: seq_num 551 ranges {2187264=434176,2621440=647168,3268608=450560,3719168=507904} 2022-04-22T17:31:56.940 INFO:tasks.rados.rados.0.smithi115.stdout:1799: writing smithi11541303-46 from 2187264 to 2621440 tid 1 2022-04-22T17:31:56.944 INFO:tasks.rados.rados.0.smithi115.stdout:1799: writing smithi11541303-46 from 2621440 to 3268608 tid 2 2022-04-22T17:31:56.946 INFO:tasks.rados.rados.0.smithi115.stdout:1799: writing smithi11541303-46 from 3268608 to 3719168 tid 3 2022-04-22T17:31:56.950 INFO:tasks.rados.rados.0.smithi115.stdout:1799: writing smithi11541303-46 from 3719168 to 4227072 tid 4 2022-04-22T17:31:56.950 INFO:tasks.rados.rados.0.smithi115.stdout:1797: done (2 left) 2022-04-22T17:31:56.950 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 181 2022-04-22T17:31:56.950 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 1 to smithi11541303-31 2022-04-22T17:31:56.950 INFO:tasks.rados.rados.0.smithi115.stdout:1800: seq_num 552 ranges {0=450560,450560=532480,983040=655360,1638400=417792} 2022-04-22T17:31:56.952 INFO:tasks.rados.rados.0.smithi115.stdout:1800: writing smithi11541303-25 from 0 to 450560 tid 1 2022-04-22T17:31:56.955 INFO:tasks.rados.rados.0.smithi115.stdout:1800: writing smithi11541303-25 from 450560 to 983040 tid 2 2022-04-22T17:31:56.959 INFO:tasks.rados.rados.0.smithi115.stdout:1800: writing smithi11541303-25 from 983040 to 1638400 tid 3 2022-04-22T17:31:56.964 INFO:tasks.rados.rados.0.smithi115.stdout:1800: writing smithi11541303-25 from 1638400 to 2056192 tid 4 2022-04-22T17:31:56.964 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 181 2022-04-22T17:31:56.965 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 2 to smithi11541303-31 2022-04-22T17:31:56.965 INFO:tasks.rados.rados.0.smithi115.stdout:1801: seq_num 553 ranges {2482176=466944,2949120=417792,3366912=417792,3784704=516096,4300800=253952} 2022-04-22T17:31:56.980 INFO:tasks.rados.rados.0.smithi115.stdout:1801: writing smithi11541303-2 from 2482176 to 2949120 tid 1 2022-04-22T17:31:56.983 INFO:tasks.rados.rados.0.smithi115.stdout:1801: writing smithi11541303-2 from 2949120 to 3366912 tid 2 2022-04-22T17:31:56.985 INFO:tasks.rados.rados.0.smithi115.stdout:1801: writing smithi11541303-2 from 3366912 to 3784704 tid 3 2022-04-22T17:31:56.988 INFO:tasks.rados.rados.0.smithi115.stdout:1801: writing smithi11541303-2 from 3784704 to 4300800 tid 4 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1801: writing smithi11541303-2 from 4300800 to 4554752 tid 5 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1802: rmattr oid 12 current snap is 181 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1802: done (4 left) 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1803: setattr oid 47 current snap is 181 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 3 to smithi11541303-31 2022-04-22T17:31:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 4 to smithi11541303-31 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 5 to smithi11541303-31 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1798: finishing write tid 6 to smithi11541303-31 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1143 (ObjNum 550 snap 181 seq_num 550) dirty exists 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1798: left oid 31 (ObjNum 550 snap 181 seq_num 550) 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 1 to smithi11541303-46 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 2 to smithi11541303-46 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 3 to smithi11541303-46 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 4 to smithi11541303-46 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 5 to smithi11541303-46 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 1 to smithi11541303-25 2022-04-22T17:31:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 2 to smithi11541303-25 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1799: finishing write tid 6 to smithi11541303-46 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 991 (ObjNum 551 snap 181 seq_num 551) dirty exists 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1799: left oid 46 (ObjNum 551 snap 181 seq_num 551) 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 3 to smithi11541303-25 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 4 to smithi11541303-25 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 5 to smithi11541303-25 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1800: finishing write tid 6 to smithi11541303-25 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 847 (ObjNum 552 snap 181 seq_num 552) dirty exists 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1800: left oid 25 (ObjNum 552 snap 181 seq_num 552) 2022-04-22T17:31:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 1 to smithi11541303-2 2022-04-22T17:31:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 2 to smithi11541303-2 2022-04-22T17:31:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:1798: done (4 left) 2022-04-22T17:31:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:1799: done (3 left) 2022-04-22T17:31:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:1800: done (2 left) 2022-04-22T17:31:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:1804: rollback oid 24 current snap is 181 2022-04-22T17:31:56.994 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 173 2022-04-22T17:31:56.994 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 181 2022-04-22T17:31:56.994 INFO:tasks.rados.rados.0.smithi115.stdout:1805: seq_num 554 ranges {0=483328,483328=704512,1187840=622592,1810432=278528} 2022-04-22T17:31:56.994 INFO:tasks.rados.rados.0.smithi115.stdout:1805: writing smithi11541303-40 from 0 to 483328 tid 1 2022-04-22T17:31:56.998 INFO:tasks.rados.rados.0.smithi115.stdout:1805: writing smithi11541303-40 from 483328 to 1187840 tid 2 2022-04-22T17:31:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:1805: writing smithi11541303-40 from 1187840 to 1810432 tid 3 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1805: writing smithi11541303-40 from 1810432 to 2088960 tid 4 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1806: rollback oid 42 current snap is 181 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 180 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1804: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 650 (ObjNum -1479267200 snap 32692 seq_num 3178645088) dirty dne 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1804: done (4 left) 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1807: copy_from oid 37 from oid 21 current snap is 181 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 3 to smithi11541303-2 2022-04-22T17:31:57.006 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 4 to smithi11541303-2 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 5 to smithi11541303-2 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 6 to smithi11541303-2 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1801: finishing write tid 7 to smithi11541303-2 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1074 (ObjNum 553 snap 181 seq_num 553) dirty exists 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1801: left oid 2 (ObjNum 553 snap 181 seq_num 553) 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1801: done (4 left) 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1808: delete oid 28 current snap is 181 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 752 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:31:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 1 to smithi11541303-40 2022-04-22T17:31:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:1803: done (4 left) 2022-04-22T17:31:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:1808: done (3 left) 2022-04-22T17:31:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:1809: rmattr oid 27 current snap is 181 2022-04-22T17:31:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:1809: done (3 left) 2022-04-22T17:31:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:1810: copy_from oid 49 from oid 48 current snap is 181 2022-04-22T17:31:57.009 INFO:tasks.rados.rados.0.smithi115.stdout:1811: setattr oid 24 current snap is 181 2022-04-22T17:31:57.009 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 2 to smithi11541303-40 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1806: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1115 (ObjNum 1 snap 22075 seq_num 3178729968) dirty dne 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 3 to smithi11541303-40 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1810: finishing copy_from to smithi11541303-49 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1810: got expected ENOENT (src dne) 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1806: done (4 left) 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1812: snap_create 2022-04-22T17:31:57.010 INFO:tasks.rados.rados.0.smithi115.stdout:1810: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:31:57.011 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 4 to smithi11541303-40 2022-04-22T17:31:57.011 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 5 to smithi11541303-40 2022-04-22T17:31:57.013 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 660 (ObjNum 0 snap 0 seq_num 3176620048) dirty exists 2022-04-22T17:31:57.013 INFO:tasks.rados.rados.0.smithi115.stdout:1805: finishing write tid 6 to smithi11541303-40 2022-04-22T17:31:57.013 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 757 (ObjNum 554 snap 181 seq_num 554) dirty exists 2022-04-22T17:31:57.014 INFO:tasks.rados.rados.0.smithi115.stdout:1805: left oid 40 (ObjNum 554 snap 181 seq_num 554) 2022-04-22T17:31:57.025 INFO:tasks.rados.rados.0.smithi115.stdout:1807: finishing copy_from to smithi11541303-37 2022-04-22T17:31:57.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 981 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:31:57.025 INFO:tasks.rados.rados.0.smithi115.stdout:1807: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:31:57.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:56 smithi115 bash[46934]: cluster 2022-04-22T17:31:55.212830+0000 mgr.y (mgr.34107) 195 : cluster [DBG] pgmap v316: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 197 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 38/393 objects degraded (9.669%) 2022-04-22T17:31:57.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:56 smithi115 bash[46934]: cluster 2022-04-22T17:31:55.854339+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2022-04-22T17:31:57.161 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:56 smithi115 bash[45495]: cluster 2022-04-22T17:31:55.212830+0000 mgr.y (mgr.34107) 195 : cluster [DBG] pgmap v316: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 197 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 38/393 objects degraded (9.669%) 2022-04-22T17:31:57.162 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:56 smithi115 bash[45495]: cluster 2022-04-22T17:31:55.854339+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2022-04-22T17:31:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:56 smithi121 bash[36902]: cluster 2022-04-22T17:31:55.212830+0000 mgr.y (mgr.34107) 195 : cluster [DBG] pgmap v316: 121 pgs: 8 active+undersized+degraded, 33 active+undersized, 80 active+clean; 197 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 38/393 objects degraded (9.669%) 2022-04-22T17:31:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:56 smithi121 bash[36902]: cluster 2022-04-22T17:31:55.854339+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2022-04-22T17:31:57.905 INFO:tasks.rados.rados.0.smithi115.stdout:1805: done (4 left) 2022-04-22T17:31:57.941 INFO:tasks.rados.rados.0.smithi115.stdout:1807: done (3 left) 2022-04-22T17:31:57.942 INFO:tasks.rados.rados.0.smithi115.stdout:1810: done (2 left) 2022-04-22T17:31:57.942 INFO:tasks.rados.rados.0.smithi115.stdout:1811: done (1 left) 2022-04-22T17:31:57.942 INFO:tasks.rados.rados.0.smithi115.stdout:1812: done (0 left) 2022-04-22T17:31:57.942 INFO:tasks.rados.rados.0.smithi115.stdout:1813: delete oid 24 current snap is 182 2022-04-22T17:31:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:1813: done (0 left) 2022-04-22T17:31:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:1814: rmattr oid 14 current snap is 182 2022-04-22T17:31:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:1814: done (0 left) 2022-04-22T17:31:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:1815: delete oid 15 current snap is 182 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1815: done (0 left) 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 182 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1816: seq_num 558 ranges {0=548864,548864=688128,1236992=606208,1843200=311296} 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1816: writing smithi11541303-4 from 0 to 548864 tid 1 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1816: writing smithi11541303-4 from 548864 to 1236992 tid 2 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1816: writing smithi11541303-4 from 1236992 to 1843200 tid 3 2022-04-22T17:31:57.945 INFO:tasks.rados.rados.0.smithi115.stdout:1816: writing smithi11541303-4 from 1843200 to 2154496 tid 4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1817: snap_create 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 1 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 2 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 3 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 4 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 5 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: finishing write tid 6 to smithi11541303-4 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1056 (ObjNum 558 snap 182 seq_num 558) dirty exists 2022-04-22T17:31:57.946 INFO:tasks.rados.rados.0.smithi115.stdout:1816: left oid 4 (ObjNum 558 snap 182 seq_num 558) 2022-04-22T17:31:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:57 smithi121 bash[36902]: cluster 2022-04-22T17:31:56.878360+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2022-04-22T17:31:58.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:57 smithi115 bash[45495]: cluster 2022-04-22T17:31:56.878360+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2022-04-22T17:31:58.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:57 smithi115 bash[46934]: cluster 2022-04-22T17:31:56.878360+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2022-04-22T17:31:58.910 INFO:tasks.rados.rados.0.smithi115.stdout:1816: done (1 left) 2022-04-22T17:31:58.910 INFO:tasks.rados.rados.0.smithi115.stdout:1817: done (0 left) 2022-04-22T17:31:58.910 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 183 2022-04-22T17:31:58.910 INFO:tasks.rados.rados.0.smithi115.stdout:1818: seq_num 559 ranges {5545984=565248,6111232=573440,6684672=409600,7094272=622592} 2022-04-22T17:31:58.944 INFO:tasks.rados.rados.0.smithi115.stdout:1818: writing smithi11541303-45 from 5545984 to 6111232 tid 1 2022-04-22T17:31:58.947 INFO:tasks.rados.rados.0.smithi115.stdout:1818: writing smithi11541303-45 from 6111232 to 6684672 tid 2 2022-04-22T17:31:58.949 INFO:tasks.rados.rados.0.smithi115.stdout:1818: writing smithi11541303-45 from 6684672 to 7094272 tid 3 2022-04-22T17:31:58.953 INFO:tasks.rados.rados.0.smithi115.stdout:1818: writing smithi11541303-45 from 7094272 to 7716864 tid 4 2022-04-22T17:31:58.953 INFO:tasks.rados.rados.0.smithi115.stdout:1819: snap_remove snap 158 2022-04-22T17:31:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:58 smithi115 bash[45495]: cluster 2022-04-22T17:31:57.213429+0000 mgr.y (mgr.34107) 196 : cluster [DBG] pgmap v319: 121 pgs: 6 peering, 5 active+undersized+degraded, 27 active+undersized, 83 active+clean; 215 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 8.5 MiB/s wr, 22 op/s; 27/414 objects degraded (6.522%); 4.9 MiB/s, 1 objects/s recovering 2022-04-22T17:31:59.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:58 smithi115 bash[45495]: cluster 2022-04-22T17:31:57.889253+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2022-04-22T17:31:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:58 smithi115 bash[46934]: cluster 2022-04-22T17:31:57.213429+0000 mgr.y (mgr.34107) 196 : cluster [DBG] pgmap v319: 121 pgs: 6 peering, 5 active+undersized+degraded, 27 active+undersized, 83 active+clean; 215 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 8.5 MiB/s wr, 22 op/s; 27/414 objects degraded (6.522%); 4.9 MiB/s, 1 objects/s recovering 2022-04-22T17:31:59.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:58 smithi115 bash[46934]: cluster 2022-04-22T17:31:57.889253+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2022-04-22T17:31:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:58 smithi121 bash[36902]: cluster 2022-04-22T17:31:57.213429+0000 mgr.y (mgr.34107) 196 : cluster [DBG] pgmap v319: 121 pgs: 6 peering, 5 active+undersized+degraded, 27 active+undersized, 83 active+clean; 215 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 8.5 MiB/s wr, 22 op/s; 27/414 objects degraded (6.522%); 4.9 MiB/s, 1 objects/s recovering 2022-04-22T17:31:59.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:58 smithi121 bash[36902]: cluster 2022-04-22T17:31:57.889253+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2022-04-22T17:31:59.803 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[42914]: debug 2022-04-22T17:31:59.370+0000 7f8eea62c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:31:59.926 INFO:tasks.rados.rados.0.smithi115.stdout:1819: done (1 left) 2022-04-22T17:31:59.926 INFO:tasks.rados.rados.0.smithi115.stdout:1820: copy_from oid 25 from oid 40 current snap is 183 2022-04-22T17:31:59.927 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 183 2022-04-22T17:31:59.927 INFO:tasks.rados.rados.0.smithi115.stdout:1821: seq_num 560 ranges {3629056=581632,4210688=458752,4669440=614400,5283840=532480} 2022-04-22T17:31:59.946 INFO:tasks.rados.rados.0.smithi115.stdout:1821: writing smithi11541303-29 from 3629056 to 4210688 tid 1 2022-04-22T17:31:59.950 INFO:tasks.rados.rados.0.smithi115.stdout:1821: writing smithi11541303-29 from 4210688 to 4669440 tid 2 2022-04-22T17:31:59.954 INFO:tasks.rados.rados.0.smithi115.stdout:1821: writing smithi11541303-29 from 4669440 to 5283840 tid 3 2022-04-22T17:31:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:1821: writing smithi11541303-29 from 5283840 to 5816320 tid 4 2022-04-22T17:31:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:1822: snap_remove snap 180 2022-04-22T17:32:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: cluster 2022-04-22T17:31:58.893466+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2022-04-22T17:32:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.233987+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.347080+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.353024+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.361935+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.369777+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.372597+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.373629+0000 mon.a (mon.0) 653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: audit 2022-04-22T17:31:59.375115+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:31:59 smithi121 bash[36902]: cluster 2022-04-22T17:31:59.792980+0000 mon.a (mon.0) 655 : cluster [WRN] Health check update: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: cluster 2022-04-22T17:31:58.893466+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2022-04-22T17:32:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.233987+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.347080+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.353024+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.361935+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.369777+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.372597+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.373629+0000 mon.a (mon.0) 653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: audit 2022-04-22T17:31:59.375115+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[45495]: cluster 2022-04-22T17:31:59.792980+0000 mon.a (mon.0) 655 : cluster [WRN] Health check update: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: cluster 2022-04-22T17:31:58.893466+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.233987+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:00.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.347080+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.353024+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.361935+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.369777+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.372597+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.373629+0000 mon.a (mon.0) 653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: audit 2022-04-22T17:31:59.375115+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:00.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:31:59 smithi115 bash[46934]: cluster 2022-04-22T17:31:59.792980+0000 mon.a (mon.0) 655 : cluster [WRN] Health check update: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:00.927 INFO:tasks.rados.rados.0.smithi115.stdout:1822: done (3 left) 2022-04-22T17:32:00.927 INFO:tasks.rados.rados.0.smithi115.stdout:1823: delete oid 17 current snap is 183 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 1 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 2 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 3 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 4 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 5 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: finishing write tid 6 to smithi11541303-45 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1061 (ObjNum 559 snap 183 seq_num 559) dirty exists 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1818: left oid 45 (ObjNum 559 snap 183 seq_num 559) 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:1820: finishing copy_from to smithi11541303-25 2022-04-22T17:32:00.928 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 757 (ObjNum 554 snap 181 seq_num 554) dirty exists 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1820: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 1 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 2 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 3 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 4 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 5 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: finishing write tid 6 to smithi11541303-29 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 497 (ObjNum 560 snap 183 seq_num 560) dirty exists 2022-04-22T17:32:00.929 INFO:tasks.rados.rados.0.smithi115.stdout:1821: left oid 29 (ObjNum 560 snap 183 seq_num 560) 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:1818: done (3 left) 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:1820: done (2 left) 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:1821: done (1 left) 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:1823: done (0 left) 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 183 2022-04-22T17:32:00.935 INFO:tasks.rados.rados.0.smithi115.stdout:1824: seq_num 562 ranges {1589248=622592,2211840=630784,2842624=581632,3424256=393216} 2022-04-22T17:32:00.949 INFO:tasks.rados.rados.0.smithi115.stdout:1824: writing smithi11541303-23 from 1589248 to 2211840 tid 1 2022-04-22T17:32:00.953 INFO:tasks.rados.rados.0.smithi115.stdout:1824: writing smithi11541303-23 from 2211840 to 2842624 tid 2 2022-04-22T17:32:00.956 INFO:tasks.rados.rados.0.smithi115.stdout:1824: writing smithi11541303-23 from 2842624 to 3424256 tid 3 2022-04-22T17:32:00.959 INFO:tasks.rados.rados.0.smithi115.stdout:1824: writing smithi11541303-23 from 3424256 to 3817472 tid 4 2022-04-22T17:32:00.959 INFO:tasks.rados.rados.0.smithi115.stdout:1825: snap_create 2022-04-22T17:32:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 1 to smithi11541303-23 2022-04-22T17:32:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 2 to smithi11541303-23 2022-04-22T17:32:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 3 to smithi11541303-23 2022-04-22T17:32:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 4 to smithi11541303-23 2022-04-22T17:32:00.971 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 5 to smithi11541303-23 2022-04-22T17:32:00.975 INFO:tasks.rados.rados.0.smithi115.stdout:1824: finishing write tid 6 to smithi11541303-23 2022-04-22T17:32:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 996 (ObjNum 562 snap 183 seq_num 562) dirty exists 2022-04-22T17:32:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:1824: left oid 23 (ObjNum 562 snap 183 seq_num 562) 2022-04-22T17:32:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cluster 2022-04-22T17:31:59.214593+0000 mgr.y (mgr.34107) 197 : cluster [DBG] pgmap v322: 121 pgs: 6 peering, 115 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 29 op/s; 8.6 MiB/s, 3 objects/s recovering 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cephadm 2022-04-22T17:31:59.336956+0000 mgr.y (mgr.34107) 198 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: audit 2022-04-22T17:31:59.375336+0000 mgr.y (mgr.34107) 199 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cephadm 2022-04-22T17:31:59.376388+0000 mgr.y (mgr.34107) 200 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cluster 2022-04-22T17:31:59.889844+0000 mon.a (mon.0) 656 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded) 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cluster 2022-04-22T17:31:59.889916+0000 mon.a (mon.0) 657 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:00 smithi121 bash[36902]: cluster 2022-04-22T17:31:59.905154+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2022-04-22T17:32:01.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cluster 2022-04-22T17:31:59.214593+0000 mgr.y (mgr.34107) 197 : cluster [DBG] pgmap v322: 121 pgs: 6 peering, 115 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 29 op/s; 8.6 MiB/s, 3 objects/s recovering 2022-04-22T17:32:01.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cephadm 2022-04-22T17:31:59.336956+0000 mgr.y (mgr.34107) 198 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: audit 2022-04-22T17:31:59.375336+0000 mgr.y (mgr.34107) 199 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cephadm 2022-04-22T17:31:59.376388+0000 mgr.y (mgr.34107) 200 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cluster 2022-04-22T17:31:59.889844+0000 mon.a (mon.0) 656 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded) 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cluster 2022-04-22T17:31:59.889916+0000 mon.a (mon.0) 657 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[45495]: cluster 2022-04-22T17:31:59.905154+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cluster 2022-04-22T17:31:59.214593+0000 mgr.y (mgr.34107) 197 : cluster [DBG] pgmap v322: 121 pgs: 6 peering, 115 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 29 op/s; 8.6 MiB/s, 3 objects/s recovering 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cephadm 2022-04-22T17:31:59.336956+0000 mgr.y (mgr.34107) 198 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:32:01.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: audit 2022-04-22T17:31:59.375336+0000 mgr.y (mgr.34107) 199 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:01.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cephadm 2022-04-22T17:31:59.376388+0000 mgr.y (mgr.34107) 200 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:01.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cluster 2022-04-22T17:31:59.889844+0000 mon.a (mon.0) 656 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/414 objects degraded (6.522%), 5 pgs degraded) 2022-04-22T17:32:01.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cluster 2022-04-22T17:31:59.889916+0000 mon.a (mon.0) 657 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:01.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:00 smithi115 bash[46934]: cluster 2022-04-22T17:31:59.905154+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2022-04-22T17:32:01.927 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:01.624Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:01.927 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:01.624Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:01.928 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:01.624Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:32:01.939 INFO:tasks.rados.rados.0.smithi115.stdout:1824: done (1 left) 2022-04-22T17:32:01.939 INFO:tasks.rados.rados.0.smithi115.stdout:1825: done (0 left) 2022-04-22T17:32:01.939 INFO:tasks.rados.rados.0.smithi115.stdout:1826: read oid 33 snap -1 2022-04-22T17:32:01.939 INFO:tasks.rados.rados.0.smithi115.stdout:1826: expect (ObjNum 545 snap 178 seq_num 545) 2022-04-22T17:32:01.940 INFO:tasks.rados.rados.0.smithi115.stdout:1827: rollback oid 17 current snap is 184 2022-04-22T17:32:01.940 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 17 to 181 2022-04-22T17:32:01.940 INFO:tasks.rados.rados.0.smithi115.stdout:1828: snap_create 2022-04-22T17:32:01.953 INFO:tasks.rados.rados.0.smithi115.stdout:1827: finishing rollback tid 1 to smithi11541303-17 2022-04-22T17:32:01.953 INFO:tasks.rados.rados.0.smithi115.stdout:1827: finishing rollback tid 2 to smithi11541303-17 2022-04-22T17:32:01.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 861 (ObjNum 523 snap 173 seq_num 523) dirty exists 2022-04-22T17:32:01.966 INFO:tasks.rados.rados.0.smithi115.stdout:1826: expect (ObjNum 545 snap 178 seq_num 545) 2022-04-22T17:32:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:01 smithi121 bash[36902]: cluster 2022-04-22T17:32:00.910823+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2022-04-22T17:32:02.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:01 smithi115 bash[45495]: cluster 2022-04-22T17:32:00.910823+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2022-04-22T17:32:02.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:01 smithi115 bash[46934]: cluster 2022-04-22T17:32:00.910823+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2022-04-22T17:32:02.955 INFO:tasks.rados.rados.0.smithi115.stdout:1826: done (2 left) 2022-04-22T17:32:02.955 INFO:tasks.rados.rados.0.smithi115.stdout:1827: done (1 left) 2022-04-22T17:32:02.955 INFO:tasks.rados.rados.0.smithi115.stdout:1828: done (0 left) 2022-04-22T17:32:02.955 INFO:tasks.rados.rados.0.smithi115.stdout:1829: snap_create 2022-04-22T17:32:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:02 smithi121 bash[36902]: cluster 2022-04-22T17:32:01.215529+0000 mgr.y (mgr.34107) 201 : cluster [DBG] pgmap v325: 121 pgs: 121 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 883 KiB/s rd, 3.5 MiB/s wr, 22 op/s; 3.7 MiB/s, 1 objects/s recovering 2022-04-22T17:32:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:02 smithi121 bash[36902]: cluster 2022-04-22T17:32:01.923098+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2022-04-22T17:32:03.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:02 smithi115 bash[45495]: cluster 2022-04-22T17:32:01.215529+0000 mgr.y (mgr.34107) 201 : cluster [DBG] pgmap v325: 121 pgs: 121 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 883 KiB/s rd, 3.5 MiB/s wr, 22 op/s; 3.7 MiB/s, 1 objects/s recovering 2022-04-22T17:32:03.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:02 smithi115 bash[45495]: cluster 2022-04-22T17:32:01.923098+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2022-04-22T17:32:03.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:02 smithi115 bash[46934]: cluster 2022-04-22T17:32:01.215529+0000 mgr.y (mgr.34107) 201 : cluster [DBG] pgmap v325: 121 pgs: 121 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 883 KiB/s rd, 3.5 MiB/s wr, 22 op/s; 3.7 MiB/s, 1 objects/s recovering 2022-04-22T17:32:03.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:02 smithi115 bash[46934]: cluster 2022-04-22T17:32:01.923098+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2022-04-22T17:32:03.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:03.833 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:03.519Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:03.834 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:03.519Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:32:03.961 INFO:tasks.rados.rados.0.smithi115.stdout:1829: done (0 left) 2022-04-22T17:32:03.961 INFO:tasks.rados.rados.0.smithi115.stdout:1830: read oid 37 snap -1 2022-04-22T17:32:03.961 INFO:tasks.rados.rados.0.smithi115.stdout:1830: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:03.962 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 186 2022-04-22T17:32:03.962 INFO:tasks.rados.rados.0.smithi115.stdout:1831: seq_num 563 ranges {3792896=638976,4431872=565248,4997120=794624,5791744=245760} 2022-04-22T17:32:03.990 INFO:tasks.rados.rados.0.smithi115.stdout:1831: writing smithi11541303-17 from 3792896 to 4431872 tid 1 2022-04-22T17:32:03.993 INFO:tasks.rados.rados.0.smithi115.stdout:1831: writing smithi11541303-17 from 4431872 to 4997120 tid 2 2022-04-22T17:32:03.997 INFO:tasks.rados.rados.0.smithi115.stdout:1831: writing smithi11541303-17 from 4997120 to 5791744 tid 3 2022-04-22T17:32:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:1831: writing smithi11541303-17 from 5791744 to 6037504 tid 4 2022-04-22T17:32:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 186 2022-04-22T17:32:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:1832: seq_num 564 ranges {7716864=655360,8372224=442368,8814592=598016,9412608=565248} 2022-04-22T17:32:04.046 INFO:tasks.rados.rados.0.smithi115.stdout:1832: writing smithi11541303-45 from 7716864 to 8372224 tid 1 2022-04-22T17:32:04.049 INFO:tasks.rados.rados.0.smithi115.stdout:1832: writing smithi11541303-45 from 8372224 to 8814592 tid 2 2022-04-22T17:32:04.052 INFO:tasks.rados.rados.0.smithi115.stdout:1832: writing smithi11541303-45 from 8814592 to 9412608 tid 3 2022-04-22T17:32:04.056 INFO:tasks.rados.rados.0.smithi115.stdout:1832: writing smithi11541303-45 from 9412608 to 9977856 tid 4 2022-04-22T17:32:04.056 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 186 2022-04-22T17:32:04.057 INFO:tasks.rados.rados.0.smithi115.stdout:1833: seq_num 565 ranges {0=671744,671744=729088,1400832=753664,2154496=122880} 2022-04-22T17:32:04.059 INFO:tasks.rados.rados.0.smithi115.stdout:1833: writing smithi11541303-12 from 0 to 671744 tid 1 2022-04-22T17:32:04.063 INFO:tasks.rados.rados.0.smithi115.stdout:1833: writing smithi11541303-12 from 671744 to 1400832 tid 2 2022-04-22T17:32:04.067 INFO:tasks.rados.rados.0.smithi115.stdout:1833: writing smithi11541303-12 from 1400832 to 2154496 tid 3 2022-04-22T17:32:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:1833: writing smithi11541303-12 from 2154496 to 2277376 tid 4 2022-04-22T17:32:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:1834: rollback oid 29 current snap is 186 2022-04-22T17:32:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 175 2022-04-22T17:32:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:1830: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:04.255 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 1 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 2 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 3 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 4 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 5 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: finishing write tid 6 to smithi11541303-17 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 867 (ObjNum 563 snap 186 seq_num 563) dirty exists 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1831: left oid 17 (ObjNum 563 snap 186 seq_num 563) 2022-04-22T17:32:04.256 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 1 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 2 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 3 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 4 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1830: done (4 left) 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1831: done (3 left) 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 186 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 5 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: finishing write tid 6 to smithi11541303-45 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1066 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1832: left oid 45 (ObjNum 564 snap 186 seq_num 564) 2022-04-22T17:32:04.257 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 1 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 2 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1834: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1834: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 3 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1834: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 500 (ObjNum 517 snap 170 seq_num 517) dirty exists 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 4 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 5 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:1833: finishing write tid 6 to smithi11541303-12 2022-04-22T17:32:04.258 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 772 (ObjNum 565 snap 186 seq_num 565) dirty exists 2022-04-22T17:32:04.259 INFO:tasks.rados.rados.0.smithi115.stdout:1833: left oid 12 (ObjNum 565 snap 186 seq_num 565) 2022-04-22T17:32:04.259 INFO:tasks.rados.rados.0.smithi115.stdout:1835: seq_num 566 ranges {0=688128,688128=614400,1302528=565248,1867776=425984} 2022-04-22T17:32:04.263 INFO:tasks.rados.rados.0.smithi115.stdout:1835: writing smithi11541303-27 from 0 to 688128 tid 1 2022-04-22T17:32:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:03 smithi121 bash[36902]: cluster 2022-04-22T17:32:02.937883+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2022-04-22T17:32:04.267 INFO:tasks.rados.rados.0.smithi115.stdout:1835: writing smithi11541303-27 from 688128 to 1302528 tid 2 2022-04-22T17:32:04.270 INFO:tasks.rados.rados.0.smithi115.stdout:1835: writing smithi11541303-27 from 1302528 to 1867776 tid 3 2022-04-22T17:32:04.273 INFO:tasks.rados.rados.0.smithi115.stdout:1835: writing smithi11541303-27 from 1867776 to 2293760 tid 4 2022-04-22T17:32:04.273 INFO:tasks.rados.rados.0.smithi115.stdout:1832: done (3 left) 2022-04-22T17:32:04.273 INFO:tasks.rados.rados.0.smithi115.stdout:1833: done (2 left) 2022-04-22T17:32:04.274 INFO:tasks.rados.rados.0.smithi115.stdout:1834: done (1 left) 2022-04-22T17:32:04.274 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 186 2022-04-22T17:32:04.274 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 1 to smithi11541303-27 2022-04-22T17:32:04.274 INFO:tasks.rados.rados.0.smithi115.stdout:1836: seq_num 567 ranges {0=704512,704512=499712,1204224=770048,1974272=335872} 2022-04-22T17:32:04.277 INFO:tasks.rados.rados.0.smithi115.stdout:1836: writing smithi11541303-34 from 0 to 704512 tid 1 2022-04-22T17:32:04.280 INFO:tasks.rados.rados.0.smithi115.stdout:1836: writing smithi11541303-34 from 704512 to 1204224 tid 2 2022-04-22T17:32:04.285 INFO:tasks.rados.rados.0.smithi115.stdout:1836: writing smithi11541303-34 from 1204224 to 1974272 tid 3 2022-04-22T17:32:04.287 INFO:tasks.rados.rados.0.smithi115.stdout:1836: writing smithi11541303-34 from 1974272 to 2310144 tid 4 2022-04-22T17:32:04.287 INFO:tasks.rados.rados.0.smithi115.stdout:1837: delete oid 2 current snap is 186 2022-04-22T17:32:04.287 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 2 to smithi11541303-27 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 3 to smithi11541303-27 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 4 to smithi11541303-27 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 5 to smithi11541303-27 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1835: finishing write tid 6 to smithi11541303-27 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1071 (ObjNum 566 snap 186 seq_num 566) dirty exists 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1835: left oid 27 (ObjNum 566 snap 186 seq_num 566) 2022-04-22T17:32:04.288 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 1 to smithi11541303-34 2022-04-22T17:32:04.289 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 2 to smithi11541303-34 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:1835: done (2 left) 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:1837: done (1 left) 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:1838: read oid 15 snap -1 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:1838: expect deleted 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 186 2022-04-22T17:32:04.291 INFO:tasks.rados.rados.0.smithi115.stdout:1839: seq_num 569 ranges {5120000=737280,5857280=671744,6529024=737280,7266304=196608} 2022-04-22T17:32:04.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:03 smithi115 bash[45495]: cluster 2022-04-22T17:32:02.937883+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2022-04-22T17:32:04.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:03 smithi115 bash[46934]: cluster 2022-04-22T17:32:02.937883+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2022-04-22T17:32:04.325 INFO:tasks.rados.rados.0.smithi115.stdout:1839: writing smithi11541303-22 from 5120000 to 5857280 tid 1 2022-04-22T17:32:04.329 INFO:tasks.rados.rados.0.smithi115.stdout:1839: writing smithi11541303-22 from 5857280 to 6529024 tid 2 2022-04-22T17:32:04.333 INFO:tasks.rados.rados.0.smithi115.stdout:1839: writing smithi11541303-22 from 6529024 to 7266304 tid 3 2022-04-22T17:32:04.341 INFO:tasks.rados.rados.0.smithi115.stdout:1839: writing smithi11541303-22 from 7266304 to 7462912 tid 4 2022-04-22T17:32:04.341 INFO:tasks.rados.rados.0.smithi115.stdout:1840: delete oid 5 current snap is 186 2022-04-22T17:32:04.341 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 3 to smithi11541303-34 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 4 to smithi11541303-34 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 5 to smithi11541303-34 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1836: finishing write tid 6 to smithi11541303-34 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 721 (ObjNum 567 snap 186 seq_num 567) dirty exists 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1836: left oid 34 (ObjNum 567 snap 186 seq_num 567) 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 1 to smithi11541303-22 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 2 to smithi11541303-22 2022-04-22T17:32:04.342 INFO:tasks.rados.rados.0.smithi115.stdout:1836: done (3 left) 2022-04-22T17:32:04.343 INFO:tasks.rados.rados.0.smithi115.stdout:1838: done (2 left) 2022-04-22T17:32:04.343 INFO:tasks.rados.rados.0.smithi115.stdout:1840: done (1 left) 2022-04-22T17:32:04.343 INFO:tasks.rados.rados.0.smithi115.stdout:1841: copy_from oid 26 from oid 50 current snap is 186 2022-04-22T17:32:04.343 INFO:tasks.rados.rados.0.smithi115.stdout:1842: read oid 18 snap -1 2022-04-22T17:32:04.343 INFO:tasks.rados.rados.0.smithi115.stdout:1842: expect (ObjNum 527 snap 175 seq_num 527) 2022-04-22T17:32:04.344 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 186 2022-04-22T17:32:04.344 INFO:tasks.rados.rados.0.smithi115.stdout:1843: seq_num 571 ranges {5021696=770048,5791744=483328,6275072=753664,7028736=368640} 2022-04-22T17:32:04.375 INFO:tasks.rados.rados.0.smithi115.stdout:1843: writing smithi11541303-35 from 5021696 to 5791744 tid 1 2022-04-22T17:32:04.378 INFO:tasks.rados.rados.0.smithi115.stdout:1843: writing smithi11541303-35 from 5791744 to 6275072 tid 2 2022-04-22T17:32:04.383 INFO:tasks.rados.rados.0.smithi115.stdout:1843: writing smithi11541303-35 from 6275072 to 7028736 tid 3 2022-04-22T17:32:04.386 INFO:tasks.rados.rados.0.smithi115.stdout:1843: writing smithi11541303-35 from 7028736 to 7397376 tid 4 2022-04-22T17:32:04.386 INFO:tasks.rados.rados.0.smithi115.stdout:1844: copy_from oid 15 from oid 34 current snap is 186 2022-04-22T17:32:04.386 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 3 to smithi11541303-22 2022-04-22T17:32:04.386 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 4 to smithi11541303-22 2022-04-22T17:32:04.386 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 5 to smithi11541303-22 2022-04-22T17:32:04.387 INFO:tasks.rados.rados.0.smithi115.stdout:1839: finishing write tid 6 to smithi11541303-22 2022-04-22T17:32:04.387 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 995 (ObjNum 569 snap 186 seq_num 569) dirty exists 2022-04-22T17:32:04.387 INFO:tasks.rados.rados.0.smithi115.stdout:1839: left oid 22 (ObjNum 569 snap 186 seq_num 569) 2022-04-22T17:32:04.387 INFO:tasks.rados.rados.0.smithi115.stdout:1839: done (4 left) 2022-04-22T17:32:04.387 INFO:tasks.rados.rados.0.smithi115.stdout:1845: delete oid 23 current snap is 186 2022-04-22T17:32:04.388 INFO:tasks.rados.rados.0.smithi115.stdout:1842: expect (ObjNum 527 snap 175 seq_num 527) 2022-04-22T17:32:04.554 INFO:tasks.rados.rados.0.smithi115.stdout:1841: finishing copy_from to smithi11541303-26 2022-04-22T17:32:04.554 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 716 (ObjNum 529 snap 176 seq_num 529) dirty exists 2022-04-22T17:32:04.554 INFO:tasks.rados.rados.0.smithi115.stdout:1841: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:32:04.554 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 1 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 2 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 3 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 4 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 5 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: finishing write tid 6 to smithi11541303-35 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1080 (ObjNum 571 snap 186 seq_num 571) dirty exists 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: left oid 35 (ObjNum 571 snap 186 seq_num 571) 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1841: done (4 left) 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1842: done (3 left) 2022-04-22T17:32:04.555 INFO:tasks.rados.rados.0.smithi115.stdout:1843: done (2 left) 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1845: done (1 left) 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1846: copy_from oid 26 from oid 37 current snap is 186 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1844: finishing copy_from to smithi11541303-15 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 721 (ObjNum 567 snap 186 seq_num 567) dirty exists 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1844: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1844: done (1 left) 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 186 2022-04-22T17:32:04.556 INFO:tasks.rados.rados.0.smithi115.stdout:1847: seq_num 573 ranges {3891200=802816,4694016=655360,5349376=720896,6070272=229376} 2022-04-22T17:32:04.586 INFO:tasks.rados.rados.0.smithi115.stdout:1847: writing smithi11541303-33 from 3891200 to 4694016 tid 1 2022-04-22T17:32:04.590 INFO:tasks.rados.rados.0.smithi115.stdout:1847: writing smithi11541303-33 from 4694016 to 5349376 tid 2 2022-04-22T17:32:04.594 INFO:tasks.rados.rados.0.smithi115.stdout:1847: writing smithi11541303-33 from 5349376 to 6070272 tid 3 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:1847: writing smithi11541303-33 from 6070272 to 6299648 tid 4 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:1848: delete oid 24 current snap is 186 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:1846: finishing copy_from to smithi11541303-26 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 981 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:1846: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:32:04.596 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 1 to smithi11541303-33 2022-04-22T17:32:04.597 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 2 to smithi11541303-33 2022-04-22T17:32:04.598 INFO:tasks.rados.rados.0.smithi115.stdout:1846: done (2 left) 2022-04-22T17:32:04.598 INFO:tasks.rados.rados.0.smithi115.stdout:1848: done (1 left) 2022-04-22T17:32:04.598 INFO:tasks.rados.rados.0.smithi115.stdout:1849: snap_remove snap 157 2022-04-22T17:32:04.821 INFO:tasks.rados.rados.0.smithi115.stdout:1849: done (1 left) 2022-04-22T17:32:04.821 INFO:tasks.rados.rados.0.smithi115.stdout:1850: read oid 11 snap -1 2022-04-22T17:32:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1850: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:04.822 INFO:tasks.rados.rados.0.smithi115.stdout:1851: snap_remove snap 177 2022-04-22T17:32:05.266 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[18604]: debug 2022-04-22T17:32:04.840+0000 7faa1309e700 -1 osd.5 pg_epoch: 429 pg[5.7s0( v 429'1166 (0'0,429'1166] local-lis/les=366/367 n=14 ec=63/63 lis/c=366/366 les/c/f=367/368/0 sis=366) [5,1,0]p5(0) r=0 lpr=366 luod=429'1164 crt=429'1164 lcod 429'1162 mlcod 429'1162 active+clean+snaptrim trimq=[135~1] ps=[137~1,162~1]] removing snap head 2022-04-22T17:32:05.267 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[24219]: debug 2022-04-22T17:32:04.828+0000 7f1df1444700 -1 osd.7 pg_epoch: 429 pg[5.9s0( v 429'1002 (0'0,429'1002] local-lis/les=366/367 n=11 ec=63/63 lis/c=366/366 les/c/f=367/367/0 sis=366) [7,6,1]p7(0) r=0 lpr=366 crt=429'1002 lcod 429'1000 mlcod 429'1000 active+clean+snaptrim trimq=[135~1] ps=[137~1,162~1]] removing snap head 2022-04-22T17:32:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[36902]: cluster 2022-04-22T17:32:03.216465+0000 mgr.y (mgr.34107) 202 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.9 MiB/s wr, 13 op/s 2022-04-22T17:32:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[36902]: cluster 2022-04-22T17:32:03.944994+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2022-04-22T17:32:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[36902]: audit 2022-04-22T17:32:04.236512+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:04 smithi121 bash[36902]: cluster 2022-04-22T17:32:04.805310+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2022-04-22T17:32:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[45495]: cluster 2022-04-22T17:32:03.216465+0000 mgr.y (mgr.34107) 202 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.9 MiB/s wr, 13 op/s 2022-04-22T17:32:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[45495]: cluster 2022-04-22T17:32:03.944994+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2022-04-22T17:32:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[45495]: audit 2022-04-22T17:32:04.236512+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:05.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[45495]: cluster 2022-04-22T17:32:04.805310+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2022-04-22T17:32:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[46934]: cluster 2022-04-22T17:32:03.216465+0000 mgr.y (mgr.34107) 202 : cluster [DBG] pgmap v328: 121 pgs: 121 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.9 MiB/s wr, 13 op/s 2022-04-22T17:32:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[46934]: cluster 2022-04-22T17:32:03.944994+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2022-04-22T17:32:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[46934]: audit 2022-04-22T17:32:04.236512+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:04 smithi115 bash[46934]: cluster 2022-04-22T17:32:04.805310+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2022-04-22T17:32:05.824 INFO:tasks.rados.rados.0.smithi115.stdout:1851: done (2 left) 2022-04-22T17:32:05.874 INFO:tasks.rados.rados.0.smithi115.stdout:1852: rollback oid 13 current snap is 186 2022-04-22T17:32:05.874 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 173 2022-04-22T17:32:05.875 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 3 to smithi11541303-33 2022-04-22T17:32:05.875 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 4 to smithi11541303-33 2022-04-22T17:32:05.875 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 5 to smithi11541303-33 2022-04-22T17:32:05.875 INFO:tasks.rados.rados.0.smithi115.stdout:1847: finishing write tid 6 to smithi11541303-33 2022-04-22T17:32:05.876 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 986 (ObjNum 573 snap 186 seq_num 573) dirty exists 2022-04-22T17:32:05.876 INFO:tasks.rados.rados.0.smithi115.stdout:1847: left oid 33 (ObjNum 573 snap 186 seq_num 573) 2022-04-22T17:32:05.876 INFO:tasks.rados.rados.0.smithi115.stdout:1850: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1852: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1852: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1852: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 879 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1847: done (2 left) 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1850: done (1 left) 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1852: done (0 left) 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1853: read oid 10 snap -1 2022-04-22T17:32:06.014 INFO:tasks.rados.rados.0.smithi115.stdout:1853: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:06.015 INFO:tasks.rados.rados.0.smithi115.stdout:1854: snap_create 2022-04-22T17:32:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:1853: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:06.968 INFO:tasks.rados.rados.0.smithi115.stdout:1853: done (1 left) 2022-04-22T17:32:06.968 INFO:tasks.rados.rados.0.smithi115.stdout:1854: done (0 left) 2022-04-22T17:32:06.968 INFO:tasks.rados.rados.0.smithi115.stdout:1855: setattr oid 19 current snap is 187 2022-04-22T17:32:06.970 INFO:tasks.rados.rados.0.smithi115.stdout:1856: read oid 44 snap -1 2022-04-22T17:32:06.971 INFO:tasks.rados.rados.0.smithi115.stdout:1856: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:32:06.971 INFO:tasks.rados.rados.0.smithi115.stdout:1857: rollback oid 24 current snap is 187 2022-04-22T17:32:06.971 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 181 2022-04-22T17:32:06.971 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 187 2022-04-22T17:32:06.972 INFO:tasks.rados.rados.0.smithi115.stdout:1858: seq_num 575 ranges {9519104=32768} 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:1858: writing smithi11541303-6 from 9519104 to 9551872 tid 1 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:1859: setattr oid 20 current snap is 187 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:1857: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:1857: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 663 (ObjNum 1885689170 snap 1432442193 seq_num 1666081651) dirty exists 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 773 (ObjNum 1 snap 22075 seq_num 3178446288) dirty exists 2022-04-22T17:32:07.027 INFO:tasks.rados.rados.0.smithi115.stdout:1856: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:32:07.084 INFO:tasks.rados.rados.0.smithi115.stdout:1858: finishing write tid 1 to smithi11541303-6 2022-04-22T17:32:07.084 INFO:tasks.rados.rados.0.smithi115.stdout:1858: finishing write tid 2 to smithi11541303-6 2022-04-22T17:32:07.084 INFO:tasks.rados.rados.0.smithi115.stdout:1858: finishing write tid 3 to smithi11541303-6 2022-04-22T17:32:07.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 665 (ObjNum 575 snap 187 seq_num 575) dirty exists 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1858: left oid 6 (ObjNum 575 snap 187 seq_num 575) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1855: done (4 left) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1856: done (3 left) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1857: done (2 left) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1858: done (1 left) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1860: read oid 42 snap 175 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1860: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:32:07.085 INFO:tasks.rados.rados.0.smithi115.stdout:1861: snap_create 2022-04-22T17:32:07.090 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 885 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:07.154 INFO:tasks.rados.rados.0.smithi115.stdout:1860: expect (ObjNum 480 snap 167 seq_num 480) 2022-04-22T17:32:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:06 smithi121 bash[36902]: cluster 2022-04-22T17:32:05.217279+0000 mgr.y (mgr.34107) 203 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 243 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 15 op/s 2022-04-22T17:32:07.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:06 smithi121 bash[36902]: cluster 2022-04-22T17:32:05.806621+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2022-04-22T17:32:07.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:06 smithi115 bash[46934]: cluster 2022-04-22T17:32:05.217279+0000 mgr.y (mgr.34107) 203 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 243 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 15 op/s 2022-04-22T17:32:07.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:06 smithi115 bash[46934]: cluster 2022-04-22T17:32:05.806621+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2022-04-22T17:32:07.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:06 smithi115 bash[45495]: cluster 2022-04-22T17:32:05.217279+0000 mgr.y (mgr.34107) 203 : cluster [DBG] pgmap v331: 121 pgs: 121 active+clean; 243 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 15 op/s 2022-04-22T17:32:07.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:06 smithi115 bash[45495]: cluster 2022-04-22T17:32:05.806621+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2022-04-22T17:32:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:07 smithi121 bash[36902]: cluster 2022-04-22T17:32:06.951823+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2022-04-22T17:32:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:07 smithi115 bash[46934]: cluster 2022-04-22T17:32:06.951823+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2022-04-22T17:32:08.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:07 smithi115 bash[45495]: cluster 2022-04-22T17:32:06.951823+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:1859: done (2 left) 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:1860: done (1 left) 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:1861: done (0 left) 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:1862: copy_from oid 21 from oid 7 current snap is 188 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 188 2022-04-22T17:32:08.638 INFO:tasks.rados.rados.0.smithi115.stdout:1863: seq_num 576 ranges {1884160=49152} 2022-04-22T17:32:08.654 INFO:tasks.rados.rados.0.smithi115.stdout:1863: writing smithi11541303-39 from 1884160 to 1933312 tid 1 2022-04-22T17:32:08.655 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 188 2022-04-22T17:32:08.655 INFO:tasks.rados.rados.0.smithi115.stdout:1864: seq_num 577 ranges {1622016=65536} 2022-04-22T17:32:08.665 INFO:tasks.rados.rados.0.smithi115.stdout:1864: writing smithi11541303-38 from 1622016 to 1687552 tid 1 2022-04-22T17:32:08.665 INFO:tasks.rados.rados.0.smithi115.stdout:1865: copy_from oid 40 from oid 18 current snap is 188 2022-04-22T17:32:08.665 INFO:tasks.rados.rados.0.smithi115.stdout:1863: finishing write tid 1 to smithi11541303-39 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:1863: finishing write tid 2 to smithi11541303-39 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:1863: finishing write tid 3 to smithi11541303-39 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1082 (ObjNum 576 snap 188 seq_num 576) dirty exists 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:1863: left oid 39 (ObjNum 576 snap 188 seq_num 576) 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:1863: done (3 left) 2022-04-22T17:32:08.666 INFO:tasks.rados.rados.0.smithi115.stdout:1866: delete oid 24 current snap is 188 2022-04-22T17:32:08.671 INFO:tasks.rados.rados.0.smithi115.stdout:1864: finishing write tid 1 to smithi11541303-38 2022-04-22T17:32:08.678 INFO:tasks.rados.rados.0.smithi115.stdout:1864: finishing write tid 2 to smithi11541303-38 2022-04-22T17:32:08.678 INFO:tasks.rados.rados.0.smithi115.stdout:1866: done (3 left) 2022-04-22T17:32:08.678 INFO:tasks.rados.rados.0.smithi115.stdout:1867: read oid 8 snap 181 2022-04-22T17:32:08.678 INFO:tasks.rados.rados.0.smithi115.stdout:1867: expect (ObjNum 533 snap 177 seq_num 533) 2022-04-22T17:32:08.678 INFO:tasks.rados.rados.0.smithi115.stdout:1868: delete oid 42 current snap is 188 2022-04-22T17:32:08.680 INFO:tasks.rados.rados.0.smithi115.stdout:1862: finishing copy_from to smithi11541303-21 2022-04-22T17:32:08.681 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 804 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:08.681 INFO:tasks.rados.rados.0.smithi115.stdout:1862: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:32:08.682 INFO:tasks.rados.rados.0.smithi115.stdout:1864: finishing write tid 3 to smithi11541303-38 2022-04-22T17:32:08.682 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 775 (ObjNum 577 snap 188 seq_num 577) dirty exists 2022-04-22T17:32:08.682 INFO:tasks.rados.rados.0.smithi115.stdout:1864: left oid 38 (ObjNum 577 snap 188 seq_num 577) 2022-04-22T17:32:08.683 INFO:tasks.rados.rados.0.smithi115.stdout:1862: done (4 left) 2022-04-22T17:32:08.683 INFO:tasks.rados.rados.0.smithi115.stdout:1864: done (3 left) 2022-04-22T17:32:08.683 INFO:tasks.rados.rados.0.smithi115.stdout:1868: done (2 left) 2022-04-22T17:32:08.692 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 188 2022-04-22T17:32:08.692 INFO:tasks.rados.rados.0.smithi115.stdout:1869: seq_num 580 ranges {0=114688} 2022-04-22T17:32:08.692 INFO:tasks.rados.rados.0.smithi115.stdout:1869: writing smithi11541303-41 from 0 to 114688 tid 1 2022-04-22T17:32:08.693 INFO:tasks.rados.rados.0.smithi115.stdout:1870: snap_create 2022-04-22T17:32:08.693 INFO:tasks.rados.rados.0.smithi115.stdout:1867: expect (ObjNum 533 snap 177 seq_num 533) 2022-04-22T17:32:08.808 INFO:tasks.rados.rados.0.smithi115.stdout:1865: finishing copy_from to smithi11541303-40 2022-04-22T17:32:08.808 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1105 (ObjNum 527 snap 175 seq_num 527) dirty exists 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:1865: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:1869: finishing write tid 1 to smithi11541303-41 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:1869: finishing write tid 2 to smithi11541303-41 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:1869: finishing write tid 3 to smithi11541303-41 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1107 (ObjNum 580 snap 188 seq_num 580) dirty exists 2022-04-22T17:32:08.809 INFO:tasks.rados.rados.0.smithi115.stdout:1869: left oid 41 (ObjNum 580 snap 188 seq_num 580) 2022-04-22T17:32:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:08 smithi121 bash[36902]: cluster 2022-04-22T17:32:07.217984+0000 mgr.y (mgr.34107) 204 : cluster [DBG] pgmap v334: 121 pgs: 121 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:32:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:08 smithi121 bash[36902]: cluster 2022-04-22T17:32:07.962362+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2022-04-22T17:32:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:08 smithi121 bash[36902]: audit 2022-04-22T17:32:08.327101+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:09.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[46934]: cluster 2022-04-22T17:32:07.217984+0000 mgr.y (mgr.34107) 204 : cluster [DBG] pgmap v334: 121 pgs: 121 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:32:09.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[46934]: cluster 2022-04-22T17:32:07.962362+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2022-04-22T17:32:09.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[46934]: audit 2022-04-22T17:32:08.327101+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:09.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[45495]: cluster 2022-04-22T17:32:07.217984+0000 mgr.y (mgr.34107) 204 : cluster [DBG] pgmap v334: 121 pgs: 121 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:32:09.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[45495]: cluster 2022-04-22T17:32:07.962362+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2022-04-22T17:32:09.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:08 smithi115 bash[45495]: audit 2022-04-22T17:32:08.327101+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1865: done (3 left) 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1867: done (2 left) 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1869: done (1 left) 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1870: done (0 left) 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1871: read oid 14 snap 179 2022-04-22T17:32:09.347 INFO:tasks.rados.rados.0.smithi115.stdout:1871: expect deleted 2022-04-22T17:32:09.348 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 189 2022-04-22T17:32:09.348 INFO:tasks.rados.rados.0.smithi115.stdout:1872: seq_num 581 ranges {2482176=139264} 2022-04-22T17:32:09.363 INFO:tasks.rados.rados.0.smithi115.stdout:1872: writing smithi11541303-30 from 2482176 to 2621440 tid 1 2022-04-22T17:32:09.363 INFO:tasks.rados.rados.0.smithi115.stdout:1873: read oid 12 snap -1 2022-04-22T17:32:09.363 INFO:tasks.rados.rados.0.smithi115.stdout:1873: expect (ObjNum 565 snap 186 seq_num 565) 2022-04-22T17:32:09.363 INFO:tasks.rados.rados.0.smithi115.stdout:1874: snap_create 2022-04-22T17:32:09.372 INFO:tasks.rados.rados.0.smithi115.stdout:1872: finishing write tid 1 to smithi11541303-30 2022-04-22T17:32:09.372 INFO:tasks.rados.rados.0.smithi115.stdout:1872: finishing write tid 2 to smithi11541303-30 2022-04-22T17:32:09.374 INFO:tasks.rados.rados.0.smithi115.stdout:1872: finishing write tid 3 to smithi11541303-30 2022-04-22T17:32:09.375 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1073 (ObjNum 581 snap 189 seq_num 581) dirty exists 2022-04-22T17:32:09.375 INFO:tasks.rados.rados.0.smithi115.stdout:1872: left oid 30 (ObjNum 581 snap 189 seq_num 581) 2022-04-22T17:32:09.375 INFO:tasks.rados.rados.0.smithi115.stdout:1873: expect (ObjNum 565 snap 186 seq_num 565) 2022-04-22T17:32:09.821 INFO:tasks.rados.rados.0.smithi115.stdout:1871: done (3 left) 2022-04-22T17:32:09.821 INFO:tasks.rados.rados.0.smithi115.stdout:1872: done (2 left) 2022-04-22T17:32:09.821 INFO:tasks.rados.rados.0.smithi115.stdout:1873: done (1 left) 2022-04-22T17:32:09.822 INFO:tasks.rados.rados.0.smithi115.stdout:1874: done (0 left) 2022-04-22T17:32:09.822 INFO:tasks.rados.rados.0.smithi115.stdout:1875: delete oid 43 current snap is 190 2022-04-22T17:32:09.829 INFO:tasks.rados.rados.0.smithi115.stdout:1875: done (0 left) 2022-04-22T17:32:09.830 INFO:tasks.rados.rados.0.smithi115.stdout:1876: read oid 30 snap -1 2022-04-22T17:32:09.830 INFO:tasks.rados.rados.0.smithi115.stdout:1876: expect (ObjNum 581 snap 189 seq_num 581) 2022-04-22T17:32:09.830 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 190 2022-04-22T17:32:09.830 INFO:tasks.rados.rados.0.smithi115.stdout:1877: seq_num 583 ranges {1687552=172032} 2022-04-22T17:32:09.844 INFO:tasks.rados.rados.0.smithi115.stdout:1877: writing smithi11541303-38 from 1687552 to 1859584 tid 1 2022-04-22T17:32:09.844 INFO:tasks.rados.rados.0.smithi115.stdout:1878: read oid 28 snap -1 2022-04-22T17:32:09.844 INFO:tasks.rados.rados.0.smithi115.stdout:1878: expect deleted 2022-04-22T17:32:09.844 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 190 2022-04-22T17:32:09.844 INFO:tasks.rados.rados.0.smithi115.stdout:1879: seq_num 584 ranges {0=188416} 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:1879: writing smithi11541303-14 from 0 to 188416 tid 1 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:1880: rollback oid 17 current snap is 190 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 17 to 186 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:1881: read oid 39 snap -1 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:1881: expect (ObjNum 576 snap 188 seq_num 576) 2022-04-22T17:32:09.846 INFO:tasks.rados.rados.0.smithi115.stdout:1882: rollback oid 18 current snap is 190 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 185 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:1883: copy_from oid 19 from oid 45 current snap is 190 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:1884: read oid 35 snap -1 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:1884: expect (ObjNum 571 snap 186 seq_num 571) 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 190 2022-04-22T17:32:09.847 INFO:tasks.rados.rados.0.smithi115.stdout:1885: seq_num 585 ranges {0=204800} 2022-04-22T17:32:09.848 INFO:tasks.rados.rados.0.smithi115.stdout:1885: writing smithi11541303-49 from 0 to 204800 tid 1 2022-04-22T17:32:09.848 INFO:tasks.rados.rados.0.smithi115.stdout:1886: delete oid 46 current snap is 190 2022-04-22T17:32:09.848 INFO:tasks.rados.rados.0.smithi115.stdout:1876: expect (ObjNum 581 snap 189 seq_num 581) 2022-04-22T17:32:10.014 INFO:tasks.rados.rados.0.smithi115.stdout:1880: finishing rollback tid 0 to smithi11541303-17 2022-04-22T17:32:10.015 INFO:tasks.rados.rados.0.smithi115.stdout:1880: finishing rollback tid 1 to smithi11541303-17 2022-04-22T17:32:10.015 INFO:tasks.rados.rados.0.smithi115.stdout:1880: finishing rollback tid 2 to smithi11541303-17 2022-04-22T17:32:10.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 891 (ObjNum 563 snap 186 seq_num 563) dirty exists 2022-04-22T17:32:10.015 INFO:tasks.rados.rados.0.smithi115.stdout:1879: finishing write tid 1 to smithi11541303-14 2022-04-22T17:32:10.015 INFO:tasks.rados.rados.0.smithi115.stdout:1879: finishing write tid 2 to smithi11541303-14 2022-04-22T17:32:10.016 INFO:tasks.rados.rados.0.smithi115.stdout:1881: expect (ObjNum 576 snap 188 seq_num 576) 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1877: finishing write tid 1 to smithi11541303-38 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1877: finishing write tid 2 to smithi11541303-38 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1879: finishing write tid 3 to smithi11541303-14 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 502 (ObjNum 584 snap 190 seq_num 584) dirty exists 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1879: left oid 14 (ObjNum 584 snap 190 seq_num 584) 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1882: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1882: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:32:10.096 INFO:tasks.rados.rados.0.smithi115.stdout:1882: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1183 (ObjNum 527 snap 175 seq_num 527) dirty exists 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1885: finishing write tid 1 to smithi11541303-49 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1885: finishing write tid 2 to smithi11541303-49 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1877: finishing write tid 3 to smithi11541303-38 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 777 (ObjNum 583 snap 190 seq_num 583) dirty exists 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1877: left oid 38 (ObjNum 583 snap 190 seq_num 583) 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1885: finishing write tid 3 to smithi11541303-49 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1185 (ObjNum 585 snap 190 seq_num 585) dirty exists 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1885: left oid 49 (ObjNum 585 snap 190 seq_num 585) 2022-04-22T17:32:10.097 INFO:tasks.rados.rados.0.smithi115.stdout:1884: expect (ObjNum 571 snap 186 seq_num 571) 2022-04-22T17:32:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:09 smithi121 bash[36902]: audit 2022-04-22T17:32:09.239448+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:10.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:09 smithi121 bash[36902]: cluster 2022-04-22T17:32:09.330586+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2022-04-22T17:32:10.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:09 smithi121 bash[36902]: cluster 2022-04-22T17:32:09.805266+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2022-04-22T17:32:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[46934]: audit 2022-04-22T17:32:09.239448+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[46934]: cluster 2022-04-22T17:32:09.330586+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2022-04-22T17:32:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[46934]: cluster 2022-04-22T17:32:09.805266+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2022-04-22T17:32:10.369 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[45495]: audit 2022-04-22T17:32:09.239448+0000 mon.a (mon.0) 669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:10.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[45495]: cluster 2022-04-22T17:32:09.330586+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2022-04-22T17:32:10.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:09 smithi115 bash[45495]: cluster 2022-04-22T17:32:09.805266+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2022-04-22T17:32:10.485 INFO:tasks.rados.rados.0.smithi115.stdout:1883: finishing copy_from to smithi11541303-19 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1066 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:1883: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:1876: done (10 left) 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:1877: done (9 left) 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:1878: done (8 left) 2022-04-22T17:32:10.486 INFO:tasks.rados.rados.0.smithi115.stdout:1879: done (7 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1880: done (6 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1881: done (5 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1882: done (4 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1883: done (3 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1884: done (2 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1885: done (1 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1886: done (0 left) 2022-04-22T17:32:10.487 INFO:tasks.rados.rados.0.smithi115.stdout:1887: delete oid 30 current snap is 190 2022-04-22T17:32:10.490 INFO:tasks.rados.rados.0.smithi115.stdout:1887: done (0 left) 2022-04-22T17:32:10.490 INFO:tasks.rados.rados.0.smithi115.stdout:1888: read oid 32 snap -1 2022-04-22T17:32:10.490 INFO:tasks.rados.rados.0.smithi115.stdout:1888: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:32:10.490 INFO:tasks.rados.rados.0.smithi115.stdout:1889: snap_remove snap 181 2022-04-22T17:32:10.989 INFO:tasks.rados.rados.0.smithi115.stdout:1889: done (1 left) 2022-04-22T17:32:10.989 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 190 2022-04-22T17:32:10.989 INFO:tasks.rados.rados.0.smithi115.stdout:1890: seq_num 588 ranges {0=253952} 2022-04-22T17:32:10.990 INFO:tasks.rados.rados.0.smithi115.stdout:1890: writing smithi11541303-5 from 0 to 253952 tid 1 2022-04-22T17:32:10.991 INFO:tasks.rados.rados.0.smithi115.stdout:1891: setattr oid 44 current snap is 190 2022-04-22T17:32:10.992 INFO:tasks.rados.rados.0.smithi115.stdout:1888: expect (ObjNum 151 snap 41 seq_num 151) 2022-04-22T17:32:11.212 INFO:tasks.rados.rados.0.smithi115.stdout:1890: finishing write tid 1 to smithi11541303-5 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1890: finishing write tid 2 to smithi11541303-5 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1890: finishing write tid 3 to smithi11541303-5 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1191 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1890: left oid 5 (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1888: done (2 left) 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1890: done (1 left) 2022-04-22T17:32:11.213 INFO:tasks.rados.rados.0.smithi115.stdout:1892: delete oid 32 current snap is 190 2022-04-22T17:32:11.217 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 94 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:32:11.218 INFO:tasks.rados.rados.0.smithi115.stdout:1891: done (1 left) 2022-04-22T17:32:11.218 INFO:tasks.rados.rados.0.smithi115.stdout:1892: done (0 left) 2022-04-22T17:32:11.218 INFO:tasks.rados.rados.0.smithi115.stdout:1893: snap_create 2022-04-22T17:32:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:10 smithi121 bash[36902]: cluster 2022-04-22T17:32:09.219267+0000 mgr.y (mgr.34107) 205 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2022-04-22T17:32:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:10 smithi115 bash[46934]: cluster 2022-04-22T17:32:09.219267+0000 mgr.y (mgr.34107) 205 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2022-04-22T17:32:11.343 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:10 smithi115 bash[45495]: cluster 2022-04-22T17:32:09.219267+0000 mgr.y (mgr.34107) 205 : cluster [DBG] pgmap v336: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2022-04-22T17:32:11.998 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:11.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:11.999 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:11.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:11.999 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:11.625Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:12.003 INFO:tasks.rados.rados.0.smithi115.stdout:1893: done (0 left) 2022-04-22T17:32:12.003 INFO:tasks.rados.rados.0.smithi115.stdout:1894: snap_remove snap 175 2022-04-22T17:32:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:11 smithi121 bash[36902]: cluster 2022-04-22T17:32:10.973253+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2022-04-22T17:32:12.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:11 smithi115 bash[45495]: cluster 2022-04-22T17:32:10.973253+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2022-04-22T17:32:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:11 smithi115 bash[46934]: cluster 2022-04-22T17:32:10.973253+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2022-04-22T17:32:13.006 INFO:tasks.rados.rados.0.smithi115.stdout:1894: done (0 left) 2022-04-22T17:32:13.006 INFO:tasks.rados.rados.0.smithi115.stdout:1895: snap_remove snap 184 2022-04-22T17:32:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:13 smithi121 bash[36902]: cluster 2022-04-22T17:32:11.219954+0000 mgr.y (mgr.34107) 206 : cluster [DBG] pgmap v340: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 20 op/s 2022-04-22T17:32:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:13 smithi121 bash[36902]: cluster 2022-04-22T17:32:11.987161+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2022-04-22T17:32:13.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:12 smithi115 bash[45495]: cluster 2022-04-22T17:32:11.219954+0000 mgr.y (mgr.34107) 206 : cluster [DBG] pgmap v340: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 20 op/s 2022-04-22T17:32:13.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:12 smithi115 bash[45495]: cluster 2022-04-22T17:32:11.987161+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2022-04-22T17:32:13.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:12 smithi115 bash[46934]: cluster 2022-04-22T17:32:11.219954+0000 mgr.y (mgr.34107) 206 : cluster [DBG] pgmap v340: 121 pgs: 121 active+clean; 200 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 20 op/s 2022-04-22T17:32:13.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:12 smithi115 bash[46934]: cluster 2022-04-22T17:32:11.987161+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2022-04-22T17:32:13.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:13.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:13.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:13.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:13.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:13.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:14.009 INFO:tasks.rados.rados.0.smithi115.stdout:1895: done (0 left) 2022-04-22T17:32:14.010 INFO:tasks.rados.rados.0.smithi115.stdout:1896: snap_remove snap 183 2022-04-22T17:32:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:14 smithi121 bash[36902]: cluster 2022-04-22T17:32:12.989989+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2022-04-22T17:32:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:14 smithi121 bash[36902]: audit 2022-04-22T17:32:13.258670+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:32:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:14 smithi121 bash[36902]: audit 2022-04-22T17:32:13.259158+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:32:14.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[45495]: cluster 2022-04-22T17:32:12.989989+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2022-04-22T17:32:14.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[45495]: audit 2022-04-22T17:32:13.258670+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:32:14.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[45495]: audit 2022-04-22T17:32:13.259158+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:32:14.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[46934]: cluster 2022-04-22T17:32:12.989989+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2022-04-22T17:32:14.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[46934]: audit 2022-04-22T17:32:13.258670+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:32:14.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:13 smithi115 bash[46934]: audit 2022-04-22T17:32:13.259158+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:32:14.825 INFO:tasks.rados.rados.0.smithi115.stdout:1896: done (0 left) 2022-04-22T17:32:14.825 INFO:tasks.rados.rados.0.smithi115.stdout:1897: delete oid 26 current snap is 191 2022-04-22T17:32:14.840 INFO:tasks.rados.rados.0.smithi115.stdout:1897: done (0 left) 2022-04-22T17:32:14.840 INFO:tasks.rados.rados.0.smithi115.stdout:1898: snap_remove snap 176 2022-04-22T17:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:15 smithi121 bash[36902]: cluster 2022-04-22T17:32:13.220619+0000 mgr.y (mgr.34107) 207 : cluster [DBG] pgmap v343: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2022-04-22T17:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:15 smithi121 bash[36902]: cluster 2022-04-22T17:32:13.993404+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2022-04-22T17:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:15 smithi121 bash[36902]: audit 2022-04-22T17:32:14.242608+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:15 smithi121 bash[36902]: audit 2022-04-22T17:32:14.385411+0000 mon.a (mon.0) 679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:15 smithi121 bash[36902]: cluster 2022-04-22T17:32:14.808768+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2022-04-22T17:32:15.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:14 smithi115 bash[45495]: cluster 2022-04-22T17:32:13.220619+0000 mgr.y (mgr.34107) 207 : cluster [DBG] pgmap v343: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2022-04-22T17:32:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:14 smithi115 bash[45495]: cluster 2022-04-22T17:32:13.993404+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2022-04-22T17:32:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[45495]: audit 2022-04-22T17:32:14.242608+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:15.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[45495]: audit 2022-04-22T17:32:14.385411+0000 mon.a (mon.0) 679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:15.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[45495]: cluster 2022-04-22T17:32:14.808768+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2022-04-22T17:32:15.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[46934]: cluster 2022-04-22T17:32:13.220619+0000 mgr.y (mgr.34107) 207 : cluster [DBG] pgmap v343: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2022-04-22T17:32:15.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[46934]: cluster 2022-04-22T17:32:13.993404+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2022-04-22T17:32:15.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[46934]: audit 2022-04-22T17:32:14.242608+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:15.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[46934]: audit 2022-04-22T17:32:14.385411+0000 mon.a (mon.0) 679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:15.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:15 smithi115 bash[46934]: cluster 2022-04-22T17:32:14.808768+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2022-04-22T17:32:15.837 INFO:tasks.rados.rados.0.smithi115.stdout:1898: done (0 left) 2022-04-22T17:32:15.856 INFO:tasks.rados.rados.0.smithi115.stdout:1899: read oid 49 snap -1 2022-04-22T17:32:15.856 INFO:tasks.rados.rados.0.smithi115.stdout:1899: expect (ObjNum 585 snap 190 seq_num 585) 2022-04-22T17:32:15.856 INFO:tasks.rados.rados.0.smithi115.stdout:1900: copy_from oid 22 from oid 39 current snap is 191 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1901: read oid 50 snap 185 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1901: expect (ObjNum 529 snap 176 seq_num 529) 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1902: read oid 13 snap 178 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1902: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1903: read oid 10 snap -1 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1903: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:15.857 INFO:tasks.rados.rados.0.smithi115.stdout:1904: snap_create 2022-04-22T17:32:15.858 INFO:tasks.rados.rados.0.smithi115.stdout:1899: expect (ObjNum 585 snap 190 seq_num 585) 2022-04-22T17:32:15.862 INFO:tasks.rados.rados.0.smithi115.stdout:1903: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:15.887 INFO:tasks.rados.rados.0.smithi115.stdout:1901: expect (ObjNum 529 snap 176 seq_num 529) 2022-04-22T17:32:16.022 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[45495]: audit 2022-04-22T17:32:14.385726+0000 mgr.y (mgr.34107) 208 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:16.060 INFO:tasks.rados.rados.0.smithi115.stdout:1900: finishing copy_from to smithi11541303-22 2022-04-22T17:32:16.061 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1082 (ObjNum 576 snap 188 seq_num 576) dirty exists 2022-04-22T17:32:16.061 INFO:tasks.rados.rados.0.smithi115.stdout:1900: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:32:16.061 INFO:tasks.rados.rados.0.smithi115.stdout:1902: expect (ObjNum 231 snap 73 seq_num 231) 2022-04-22T17:32:16.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[45495]: cephadm 2022-04-22T17:32:14.388057+0000 mgr.y (mgr.34107) 209 : cephadm [INF] Upgrade: osd.4 is safe to restart 2022-04-22T17:32:16.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[45495]: cluster 2022-04-22T17:32:15.820590+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2022-04-22T17:32:16.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[46934]: audit 2022-04-22T17:32:14.385726+0000 mgr.y (mgr.34107) 208 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:16.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[46934]: cephadm 2022-04-22T17:32:14.388057+0000 mgr.y (mgr.34107) 209 : cephadm [INF] Upgrade: osd.4 is safe to restart 2022-04-22T17:32:16.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:16 smithi115 bash[46934]: cluster 2022-04-22T17:32:15.820590+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2022-04-22T17:32:16.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:16 smithi121 bash[36902]: audit 2022-04-22T17:32:14.385726+0000 mgr.y (mgr.34107) 208 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2022-04-22T17:32:16.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:16 smithi121 bash[36902]: cephadm 2022-04-22T17:32:14.388057+0000 mgr.y (mgr.34107) 209 : cephadm [INF] Upgrade: osd.4 is safe to restart 2022-04-22T17:32:16.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:16 smithi121 bash[36902]: cluster 2022-04-22T17:32:15.820590+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2022-04-22T17:32:16.837 INFO:tasks.rados.rados.0.smithi115.stdout:1899: done (5 left) 2022-04-22T17:32:16.837 INFO:tasks.rados.rados.0.smithi115.stdout:1900: done (4 left) 2022-04-22T17:32:16.837 INFO:tasks.rados.rados.0.smithi115.stdout:1901: done (3 left) 2022-04-22T17:32:16.837 INFO:tasks.rados.rados.0.smithi115.stdout:1902: done (2 left) 2022-04-22T17:32:16.837 INFO:tasks.rados.rados.0.smithi115.stdout:1903: done (1 left) 2022-04-22T17:32:16.838 INFO:tasks.rados.rados.0.smithi115.stdout:1904: done (0 left) 2022-04-22T17:32:16.838 INFO:tasks.rados.rados.0.smithi115.stdout:1905: rollback oid 39 current snap is 192 2022-04-22T17:32:16.838 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 187 2022-04-22T17:32:16.838 INFO:tasks.rados.rados.0.smithi115.stdout:1906: snap_create 2022-04-22T17:32:16.846 INFO:tasks.rados.rados.0.smithi115.stdout:1905: finishing rollback tid 0 to smithi11541303-39 2022-04-22T17:32:16.846 INFO:tasks.rados.rados.0.smithi115.stdout:1905: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:32:16.847 INFO:tasks.rados.rados.0.smithi115.stdout:1905: finishing rollback tid 2 to smithi11541303-39 2022-04-22T17:32:16.847 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1085 (ObjNum 542 snap 178 seq_num 542) dirty exists 2022-04-22T17:32:17.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:17 smithi115 bash[45495]: cluster 2022-04-22T17:32:15.221474+0000 mgr.y (mgr.34107) 210 : cluster [DBG] pgmap v346: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2022-04-22T17:32:17.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:17 smithi115 bash[45495]: cluster 2022-04-22T17:32:16.820496+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2022-04-22T17:32:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:17 smithi115 bash[46934]: cluster 2022-04-22T17:32:15.221474+0000 mgr.y (mgr.34107) 210 : cluster [DBG] pgmap v346: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2022-04-22T17:32:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:17 smithi115 bash[46934]: cluster 2022-04-22T17:32:16.820496+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2022-04-22T17:32:17.328 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:17 smithi121 bash[36902]: cluster 2022-04-22T17:32:15.221474+0000 mgr.y (mgr.34107) 210 : cluster [DBG] pgmap v346: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2022-04-22T17:32:17.329 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:17 smithi121 bash[36902]: cluster 2022-04-22T17:32:16.820496+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2022-04-22T17:32:17.841 INFO:tasks.rados.rados.0.smithi115.stdout:1905: done (1 left) 2022-04-22T17:32:17.841 INFO:tasks.rados.rados.0.smithi115.stdout:1906: done (0 left) 2022-04-22T17:32:17.841 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 193 2022-04-22T17:32:17.841 INFO:tasks.rados.rados.0.smithi115.stdout:1907: seq_num 591 ranges {0=303104} 2022-04-22T17:32:17.843 INFO:tasks.rados.rados.0.smithi115.stdout:1907: writing smithi11541303-28 from 0 to 303104 tid 1 2022-04-22T17:32:17.843 INFO:tasks.rados.rados.0.smithi115.stdout:1908: snap_create 2022-04-22T17:32:17.850 INFO:tasks.rados.rados.0.smithi115.stdout:1907: finishing write tid 1 to smithi11541303-28 2022-04-22T17:32:17.851 INFO:tasks.rados.rados.0.smithi115.stdout:1907: finishing write tid 2 to smithi11541303-28 2022-04-22T17:32:17.852 INFO:tasks.rados.rados.0.smithi115.stdout:1907: finishing write tid 3 to smithi11541303-28 2022-04-22T17:32:17.852 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 504 (ObjNum 591 snap 193 seq_num 591) dirty exists 2022-04-22T17:32:17.853 INFO:tasks.rados.rados.0.smithi115.stdout:1907: left oid 28 (ObjNum 591 snap 193 seq_num 591) 2022-04-22T17:32:18.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:18 smithi121 bash[36902]: cluster 2022-04-22T17:32:17.222415+0000 mgr.y (mgr.34107) 211 : cluster [DBG] pgmap v349: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 146 KiB/s wr, 7 op/s 2022-04-22T17:32:18.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:18 smithi121 bash[36902]: cephadm 2022-04-22T17:32:17.320508+0000 mgr.y (mgr.34107) 212 : cephadm [INF] Upgrade: Updating osd.4 2022-04-22T17:32:18.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:18 smithi121 bash[36902]: audit 2022-04-22T17:32:17.328556+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:18.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:18 smithi121 bash[36902]: cephadm 2022-04-22T17:32:17.332225+0000 mgr.y (mgr.34107) 213 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:32:18.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:18 smithi121 bash[36902]: cluster 2022-04-22T17:32:17.825544+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2022-04-22T17:32:18.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[46934]: cluster 2022-04-22T17:32:17.222415+0000 mgr.y (mgr.34107) 211 : cluster [DBG] pgmap v349: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 146 KiB/s wr, 7 op/s 2022-04-22T17:32:18.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[46934]: cephadm 2022-04-22T17:32:17.320508+0000 mgr.y (mgr.34107) 212 : cephadm [INF] Upgrade: Updating osd.4 2022-04-22T17:32:18.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[46934]: audit 2022-04-22T17:32:17.328556+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:18.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[46934]: cephadm 2022-04-22T17:32:17.332225+0000 mgr.y (mgr.34107) 213 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:32:18.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[46934]: cluster 2022-04-22T17:32:17.825544+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2022-04-22T17:32:18.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[45495]: cluster 2022-04-22T17:32:17.222415+0000 mgr.y (mgr.34107) 211 : cluster [DBG] pgmap v349: 121 pgs: 121 active+clean; 219 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 146 KiB/s wr, 7 op/s 2022-04-22T17:32:18.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[45495]: cephadm 2022-04-22T17:32:17.320508+0000 mgr.y (mgr.34107) 212 : cephadm [INF] Upgrade: Updating osd.4 2022-04-22T17:32:18.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[45495]: audit 2022-04-22T17:32:17.328556+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:18.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[45495]: cephadm 2022-04-22T17:32:17.332225+0000 mgr.y (mgr.34107) 213 : cephadm [INF] Deploying daemon osd.4 on smithi121 2022-04-22T17:32:18.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:18 smithi115 bash[45495]: cluster 2022-04-22T17:32:17.825544+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1907: done (1 left) 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1908: done (0 left) 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 194 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1909: seq_num 592 ranges {1933312=319488} 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1909: writing smithi11541303-22 from 1933312 to 2252800 tid 1 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1910: copy_from oid 50 from oid 15 current snap is 194 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 194 2022-04-22T17:32:18.854 INFO:tasks.rados.rados.0.smithi115.stdout:1911: seq_num 593 ranges {3325952=335872} 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1911: writing smithi11541303-8 from 3325952 to 3661824 tid 1 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1912: copy_from oid 31 from oid 27 current snap is 194 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1909: finishing write tid 1 to smithi11541303-22 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1909: finishing write tid 2 to smithi11541303-22 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1909: finishing write tid 3 to smithi11541303-22 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1084 (ObjNum 592 snap 194 seq_num 592) dirty exists 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1909: left oid 22 (ObjNum 592 snap 194 seq_num 592) 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1909: done (3 left) 2022-04-22T17:32:18.876 INFO:tasks.rados.rados.0.smithi115.stdout:1913: snap_create 2022-04-22T17:32:18.882 INFO:tasks.rados.rados.0.smithi115.stdout:1911: finishing write tid 1 to smithi11541303-8 2022-04-22T17:32:18.882 INFO:tasks.rados.rados.0.smithi115.stdout:1911: finishing write tid 2 to smithi11541303-8 2022-04-22T17:32:18.884 INFO:tasks.rados.rados.0.smithi115.stdout:1911: finishing write tid 3 to smithi11541303-8 2022-04-22T17:32:18.884 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 885 (ObjNum 593 snap 194 seq_num 593) dirty exists 2022-04-22T17:32:18.884 INFO:tasks.rados.rados.0.smithi115.stdout:1911: left oid 8 (ObjNum 593 snap 194 seq_num 593) 2022-04-22T17:32:18.885 INFO:tasks.rados.rados.0.smithi115.stdout:1910: finishing copy_from to smithi11541303-50 2022-04-22T17:32:18.885 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 721 (ObjNum 567 snap 186 seq_num 567) dirty exists 2022-04-22T17:32:18.885 INFO:tasks.rados.rados.0.smithi115.stdout:1910: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:32:18.897 INFO:tasks.rados.rados.0.smithi115.stdout:1912: finishing copy_from to smithi11541303-31 2022-04-22T17:32:18.897 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1071 (ObjNum 566 snap 186 seq_num 566) dirty exists 2022-04-22T17:32:18.898 INFO:tasks.rados.rados.0.smithi115.stdout:1912: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:32:19.085 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (8m) 29s ago 9m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (8m) 3m ago 8m 23.4M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (10m) 29s ago 10m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (10m) 3m ago 10m 17.3M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4m) 3m ago 14m 403M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:32:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5m) 29s ago 16m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4m) 29s ago 16m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3m) 3m ago 14m 32.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3m) 29s ago 14m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (9m) 29s ago 9m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (9m) 3m ago 9m 9224k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2m) 29s ago 14m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (85s) 29s ago 13m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (58s) 29s ago 13m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (31s) 29s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:32:19.439 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (12m) 3m ago 12m 143M 2526M 16.2.5 46cf6318c64e a0623f71ea82 2022-04-22T17:32:19.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (11m) 3m ago 11m 167M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:32:19.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (11m) 3m ago 11m 132M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:32:19.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (10m) 3m ago 10m 144M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:32:19.440 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5m) 3m ago 9m 36.3M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:32:19.825 INFO:tasks.rados.rados.0.smithi115.stdout:1910: done (3 left) 2022-04-22T17:32:19.834 INFO:tasks.rados.rados.0.smithi115.stdout:1911: done (2 left) 2022-04-22T17:32:19.834 INFO:tasks.rados.rados.0.smithi115.stdout:1912: done (1 left) 2022-04-22T17:32:19.834 INFO:tasks.rados.rados.0.smithi115.stdout:1913: done (0 left) 2022-04-22T17:32:19.834 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 195 2022-04-22T17:32:19.835 INFO:tasks.rados.rados.0.smithi115.stdout:1914: seq_num 594 ranges {0=352256} 2022-04-22T17:32:19.836 INFO:tasks.rados.rados.0.smithi115.stdout:1914: writing smithi11541303-43 from 0 to 352256 tid 1 2022-04-22T17:32:19.836 INFO:tasks.rados.rados.0.smithi115.stdout:1915: delete oid 18 current snap is 195 2022-04-22T17:32:19.843 INFO:tasks.rados.rados.0.smithi115.stdout:1914: finishing write tid 1 to smithi11541303-43 2022-04-22T17:32:19.909 INFO:tasks.rados.rados.0.smithi115.stdout:1914: finishing write tid 2 to smithi11541303-43 2022-04-22T17:32:19.910 INFO:tasks.rados.rados.0.smithi115.stdout:1914: finishing write tid 3 to smithi11541303-43 2022-04-22T17:32:19.910 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 937 (ObjNum 594 snap 195 seq_num 594) dirty exists 2022-04-22T17:32:19.910 INFO:tasks.rados.rados.0.smithi115.stdout:1914: left oid 43 (ObjNum 594 snap 195 seq_num 594) 2022-04-22T17:32:19.910 INFO:tasks.rados.rados.0.smithi115.stdout:1914: done (1 left) 2022-04-22T17:32:19.911 INFO:tasks.rados.rados.0.smithi115.stdout:1915: done (0 left) 2022-04-22T17:32:19.911 INFO:tasks.rados.rados.0.smithi115.stdout:1916: read oid 36 snap -1 2022-04-22T17:32:19.911 INFO:tasks.rados.rados.0.smithi115.stdout:1916: expect (ObjNum 521 snap 172 seq_num 521) 2022-04-22T17:32:19.911 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 195 2022-04-22T17:32:19.911 INFO:tasks.rados.rados.0.smithi115.stdout:1917: seq_num 596 ranges {0=385024} 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4, 2022-04-22T17:32:19.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 4 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6, 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 9 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:32:19.913 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:1917: writing smithi11541303-18 from 0 to 385024 tid 1 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 195 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:1918: seq_num 597 ranges {1957888=401408} 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:1918: writing smithi11541303-44 from 1957888 to 2359296 tid 1 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 195 2022-04-22T17:32:19.915 INFO:tasks.rados.rados.0.smithi115.stdout:1919: seq_num 598 ranges {0=417792} 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1919: writing smithi11541303-9 from 0 to 417792 tid 1 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1920: read oid 10 snap -1 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1920: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 195 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1921: seq_num 599 ranges {2277376=434176} 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1921: writing smithi11541303-12 from 2277376 to 2711552 tid 1 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 195 2022-04-22T17:32:19.916 INFO:tasks.rados.rados.0.smithi115.stdout:1916: expect (ObjNum 521 snap 172 seq_num 521) 2022-04-22T17:32:19.936 INFO:tasks.rados.rados.0.smithi115.stdout:1917: finishing write tid 1 to smithi11541303-18 2022-04-22T17:32:19.936 INFO:tasks.rados.rados.0.smithi115.stdout:1917: finishing write tid 2 to smithi11541303-18 2022-04-22T17:32:19.936 INFO:tasks.rados.rados.0.smithi115.stdout:1917: finishing write tid 3 to smithi11541303-18 2022-04-22T17:32:19.936 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1223 (ObjNum 596 snap 195 seq_num 596) dirty exists 2022-04-22T17:32:19.936 INFO:tasks.rados.rados.0.smithi115.stdout:1917: left oid 18 (ObjNum 596 snap 195 seq_num 596) 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:1918: finishing write tid 1 to smithi11541303-44 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:1918: finishing write tid 2 to smithi11541303-44 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:1918: finishing write tid 3 to smithi11541303-44 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 105 (ObjNum 597 snap 195 seq_num 597) dirty exists 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:1918: left oid 44 (ObjNum 597 snap 195 seq_num 597) 2022-04-22T17:32:19.937 INFO:tasks.rados.rados.0.smithi115.stdout:1920: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:19.953 INFO:tasks.rados.rados.0.smithi115.stdout:1919: finishing write tid 1 to smithi11541303-9 2022-04-22T17:32:19.953 INFO:tasks.rados.rados.0.smithi115.stdout:1919: finishing write tid 2 to smithi11541303-9 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1919: finishing write tid 3 to smithi11541303-9 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1225 (ObjNum 598 snap 195 seq_num 598) dirty exists 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1919: left oid 9 (ObjNum 598 snap 195 seq_num 598) 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1921: finishing write tid 1 to smithi11541303-12 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1921: finishing write tid 2 to smithi11541303-12 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1921: finishing write tid 3 to smithi11541303-12 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1068 (ObjNum 599 snap 195 seq_num 599) dirty exists 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1921: left oid 12 (ObjNum 599 snap 195 seq_num 599) 2022-04-22T17:32:19.954 INFO:tasks.rados.rados.0.smithi115.stdout:1922: seq_num 600 ranges {2088960=450560} 2022-04-22T17:32:19.975 INFO:tasks.rados.rados.0.smithi115.stdout:1922: writing smithi11541303-25 from 2088960 to 2539520 tid 1 2022-04-22T17:32:19.975 INFO:tasks.rados.rados.0.smithi115.stdout:1916: done (6 left) 2022-04-22T17:32:19.975 INFO:tasks.rados.rados.0.smithi115.stdout:1917: done (5 left) 2022-04-22T17:32:19.975 INFO:tasks.rados.rados.0.smithi115.stdout:1918: done (4 left) 2022-04-22T17:32:19.975 INFO:tasks.rados.rados.0.smithi115.stdout:1919: done (3 left) 2022-04-22T17:32:19.976 INFO:tasks.rados.rados.0.smithi115.stdout:1920: done (2 left) 2022-04-22T17:32:19.976 INFO:tasks.rados.rados.0.smithi115.stdout:1921: done (1 left) 2022-04-22T17:32:19.976 INFO:tasks.rados.rados.0.smithi115.stdout:1923: copy_from oid 41 from oid 17 current snap is 195 2022-04-22T17:32:19.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 195 2022-04-22T17:32:19.976 INFO:tasks.rados.rados.0.smithi115.stdout:1924: seq_num 601 ranges {0=475136} 2022-04-22T17:32:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:1924: writing smithi11541303-24 from 0 to 475136 tid 1 2022-04-22T17:32:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:1925: read oid 47 snap -1 2022-04-22T17:32:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:1925: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:1926: copy_from oid 36 from oid 7 current snap is 195 2022-04-22T17:32:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:1927: delete oid 29 current snap is 195 2022-04-22T17:32:19.984 INFO:tasks.rados.rados.0.smithi115.stdout:1924: finishing write tid 1 to smithi11541303-24 2022-04-22T17:32:19.985 INFO:tasks.rados.rados.0.smithi115.stdout:1924: finishing write tid 2 to smithi11541303-24 2022-04-22T17:32:19.985 INFO:tasks.rados.rados.0.smithi115.stdout:1927: done (5 left) 2022-04-22T17:32:19.985 INFO:tasks.rados.rados.0.smithi115.stdout:1928: read oid 28 snap -1 2022-04-22T17:32:19.985 INFO:tasks.rados.rados.0.smithi115.stdout:1928: expect (ObjNum 591 snap 193 seq_num 591) 2022-04-22T17:32:19.985 INFO:tasks.rados.rados.0.smithi115.stdout:1929: snap_remove snap 182 2022-04-22T17:32:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:19 smithi121 bash[36902]: cluster 2022-04-22T17:32:18.822523+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2022-04-22T17:32:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:19 smithi121 bash[36902]: audit 2022-04-22T17:32:19.054555+0000 mgr.y (mgr.34107) 214 : audit [DBG] from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:19 smithi121 bash[36902]: audit 2022-04-22T17:32:19.245564+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:20.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:19 smithi121 bash[36902]: cluster 2022-04-22T17:32:19.809659+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2022-04-22T17:32:20.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[45495]: cluster 2022-04-22T17:32:18.822523+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2022-04-22T17:32:20.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[45495]: audit 2022-04-22T17:32:19.054555+0000 mgr.y (mgr.34107) 214 : audit [DBG] from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:20.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[45495]: audit 2022-04-22T17:32:19.245564+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:20.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[45495]: cluster 2022-04-22T17:32:19.809659+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2022-04-22T17:32:20.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[46934]: cluster 2022-04-22T17:32:18.822523+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2022-04-22T17:32:20.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[46934]: audit 2022-04-22T17:32:19.054555+0000 mgr.y (mgr.34107) 214 : audit [DBG] from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:20.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[46934]: audit 2022-04-22T17:32:19.245564+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:20.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:19 smithi115 bash[46934]: cluster 2022-04-22T17:32:19.809659+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2022-04-22T17:32:20.867 INFO:tasks.rados.rados.0.smithi115.stdout:1929: done (6 left) 2022-04-22T17:32:20.886 INFO:tasks.rados.rados.0.smithi115.stdout:1930: copy_from oid 29 from oid 4 current snap is 195 2022-04-22T17:32:20.886 INFO:tasks.rados.rados.0.smithi115.stdout:1924: finishing write tid 3 to smithi11541303-24 2022-04-22T17:32:20.886 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 668 (ObjNum 601 snap 195 seq_num 601) dirty exists 2022-04-22T17:32:20.887 INFO:tasks.rados.rados.0.smithi115.stdout:1924: left oid 24 (ObjNum 601 snap 195 seq_num 601) 2022-04-22T17:32:20.887 INFO:tasks.rados.rados.0.smithi115.stdout:1928: expect (ObjNum 591 snap 193 seq_num 591) 2022-04-22T17:32:20.889 INFO:tasks.rados.rados.0.smithi115.stdout:1922: finishing write tid 1 to smithi11541303-25 2022-04-22T17:32:20.889 INFO:tasks.rados.rados.0.smithi115.stdout:1922: finishing write tid 2 to smithi11541303-25 2022-04-22T17:32:20.889 INFO:tasks.rados.rados.0.smithi115.stdout:1922: finishing write tid 3 to smithi11541303-25 2022-04-22T17:32:20.889 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 940 (ObjNum 600 snap 195 seq_num 600) dirty exists 2022-04-22T17:32:20.889 INFO:tasks.rados.rados.0.smithi115.stdout:1922: left oid 25 (ObjNum 600 snap 195 seq_num 600) 2022-04-22T17:32:20.890 INFO:tasks.rados.rados.0.smithi115.stdout:1925: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:21.011 INFO:tasks.rados.rados.0.smithi115.stdout:1923: finishing copy_from to smithi11541303-41 2022-04-22T17:32:21.011 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 891 (ObjNum 563 snap 186 seq_num 563) dirty exists 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1923: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1926: finishing copy_from to smithi11541303-36 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 804 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1926: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1930: finishing copy_from to smithi11541303-29 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1056 (ObjNum 558 snap 182 seq_num 558) dirty exists 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1930: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1922: done (6 left) 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1923: done (5 left) 2022-04-22T17:32:21.012 INFO:tasks.rados.rados.0.smithi115.stdout:1924: done (4 left) 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1925: done (3 left) 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1926: done (2 left) 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1928: done (1 left) 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1930: done (0 left) 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1931: copy_from oid 49 from oid 8 current snap is 195 2022-04-22T17:32:21.013 INFO:tasks.rados.rados.0.smithi115.stdout:1932: snap_remove snap 191 2022-04-22T17:32:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:20 smithi121 bash[36902]: cluster 2022-04-22T17:32:19.223730+0000 mgr.y (mgr.34107) 215 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 223 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.0 MiB/s wr, 5 op/s 2022-04-22T17:32:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:20 smithi121 bash[36902]: audit 2022-04-22T17:32:19.405371+0000 mgr.y (mgr.34107) 216 : audit [DBG] from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:20 smithi121 bash[36902]: audit 2022-04-22T17:32:19.833292+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/1674906236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[45495]: cluster 2022-04-22T17:32:19.223730+0000 mgr.y (mgr.34107) 215 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 223 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.0 MiB/s wr, 5 op/s 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[45495]: audit 2022-04-22T17:32:19.405371+0000 mgr.y (mgr.34107) 216 : audit [DBG] from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[45495]: audit 2022-04-22T17:32:19.833292+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/1674906236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[46934]: cluster 2022-04-22T17:32:19.223730+0000 mgr.y (mgr.34107) 215 : cluster [DBG] pgmap v352: 121 pgs: 121 active+clean; 223 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.0 MiB/s wr, 5 op/s 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[46934]: audit 2022-04-22T17:32:19.405371+0000 mgr.y (mgr.34107) 216 : audit [DBG] from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:21.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:20 smithi115 bash[46934]: audit 2022-04-22T17:32:19.833292+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.115:0/1674906236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:21.870 INFO:tasks.rados.rados.0.smithi115.stdout:1932: done (1 left) 2022-04-22T17:32:21.870 INFO:tasks.rados.rados.0.smithi115.stdout:1933: setattr oid 40 current snap is 195 2022-04-22T17:32:21.870 INFO:tasks.rados.rados.0.smithi115.stdout:1931: finishing copy_from to smithi11541303-49 2022-04-22T17:32:21.870 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 885 (ObjNum 593 snap 194 seq_num 593) dirty exists 2022-04-22T17:32:21.870 INFO:tasks.rados.rados.0.smithi115.stdout:1931: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:32:21.871 INFO:tasks.rados.rados.0.smithi115.stdout:1931: done (1 left) 2022-04-22T17:32:21.871 INFO:tasks.rados.rados.0.smithi115.stdout:1934: setattr oid 24 current snap is 195 2022-04-22T17:32:21.873 INFO:tasks.rados.rados.0.smithi115.stdout:1935: read oid 23 snap 193 2022-04-22T17:32:21.873 INFO:tasks.rados.rados.0.smithi115.stdout:1935: expect deleted 2022-04-22T17:32:21.873 INFO:tasks.rados.rados.0.smithi115.stdout:1936: delete oid 38 current snap is 195 2022-04-22T17:32:21.879 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 669 (ObjNum 601 snap 195 seq_num 601) dirty exists 2022-04-22T17:32:21.883 INFO:tasks.rados.rados.0.smithi115.stdout:1934: done (3 left) 2022-04-22T17:32:21.883 INFO:tasks.rados.rados.0.smithi115.stdout:1935: done (2 left) 2022-04-22T17:32:21.883 INFO:tasks.rados.rados.0.smithi115.stdout:1936: done (1 left) 2022-04-22T17:32:21.883 INFO:tasks.rados.rados.0.smithi115.stdout:1937: snap_create 2022-04-22T17:32:21.891 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1108 (ObjNum 527 snap 175 seq_num 527) dirty exists 2022-04-22T17:32:22.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:21 smithi115 bash[45495]: cluster 2022-04-22T17:32:20.841706+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2022-04-22T17:32:22.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:21 smithi115 bash[46934]: cluster 2022-04-22T17:32:20.841706+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2022-04-22T17:32:22.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:21.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:22.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:21.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:22.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:21.625Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:21 smithi121 bash[36902]: cluster 2022-04-22T17:32:20.841706+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2022-04-22T17:32:22.583 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:22 smithi121 systemd[1]: Stopping Ceph osd.4 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:32:22.583 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[15740]: debug 2022-04-22T17:32:22.416+0000 7fef0ef3f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:32:22.583 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[15740]: debug 2022-04-22T17:32:22.416+0000 7fef0ef3f700 -1 osd.4 446 *** Got signal Terminated *** 2022-04-22T17:32:22.583 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[15740]: debug 2022-04-22T17:32:22.416+0000 7fef0ef3f700 -1 osd.4 446 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:32:22.870 INFO:tasks.rados.rados.0.smithi115.stdout:1933: done (1 left) 2022-04-22T17:32:22.871 INFO:tasks.rados.rados.0.smithi115.stdout:1937: done (0 left) 2022-04-22T17:32:22.871 INFO:tasks.rados.rados.0.smithi115.stdout:1938: delete oid 8 current snap is 196 2022-04-22T17:32:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:21.224698+0000 mgr.y (mgr.34107) 217 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 224 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:32:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:21.852242+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2022-04-22T17:32:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.720243+0000 mon.a (mon.0) 690 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.720311+0000 mon.a (mon.0) 691 : cluster [INF] osd.4 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.720795+0000 mon.a (mon.0) 692 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.721098+0000 mon.a (mon.0) 693 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.721314+0000 mon.a (mon.0) 694 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.721661+0000 mon.a (mon.0) 695 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.721854+0000 mon.a (mon.0) 696 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.722215+0000 mon.a (mon.0) 697 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.722496+0000 mon.a (mon.0) 698 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.722592+0000 mon.a (mon.0) 699 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.722831+0000 mon.a (mon.0) 700 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.722997+0000 mon.a (mon.0) 701 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.723148+0000 mon.a (mon.0) 702 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.723308+0000 mon.a (mon.0) 703 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:22 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.723500+0000 mon.a (mon.0) 704 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:21.224698+0000 mgr.y (mgr.34107) 217 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 224 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:32:23.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:21.852242+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2022-04-22T17:32:23.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.720243+0000 mon.a (mon.0) 690 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.720311+0000 mon.a (mon.0) 691 : cluster [INF] osd.4 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:32:23.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.720795+0000 mon.a (mon.0) 692 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.721098+0000 mon.a (mon.0) 693 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.721314+0000 mon.a (mon.0) 694 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.721661+0000 mon.a (mon.0) 695 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.721854+0000 mon.a (mon.0) 696 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.722215+0000 mon.a (mon.0) 697 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.722496+0000 mon.a (mon.0) 698 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.722592+0000 mon.a (mon.0) 699 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.722831+0000 mon.a (mon.0) 700 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.722997+0000 mon.a (mon.0) 701 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.723148+0000 mon.a (mon.0) 702 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.723308+0000 mon.a (mon.0) 703 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.723500+0000 mon.a (mon.0) 704 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:21.224698+0000 mgr.y (mgr.34107) 217 : cluster [DBG] pgmap v355: 121 pgs: 121 active+clean; 224 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:21.852242+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.720243+0000 mon.a (mon.0) 690 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.720311+0000 mon.a (mon.0) 691 : cluster [INF] osd.4 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.720795+0000 mon.a (mon.0) 692 : cluster [DBG] osd.4 reported immediately failed by osd.3 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.721098+0000 mon.a (mon.0) 693 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.721314+0000 mon.a (mon.0) 694 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.721661+0000 mon.a (mon.0) 695 : cluster [DBG] osd.4 reported immediately failed by osd.2 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.721854+0000 mon.a (mon.0) 696 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.722215+0000 mon.a (mon.0) 697 : cluster [DBG] osd.4 reported immediately failed by osd.1 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.722496+0000 mon.a (mon.0) 698 : cluster [DBG] osd.4 reported immediately failed by osd.0 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.722592+0000 mon.a (mon.0) 699 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.722831+0000 mon.a (mon.0) 700 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.722997+0000 mon.a (mon.0) 701 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.723148+0000 mon.a (mon.0) 702 : cluster [DBG] osd.4 reported immediately failed by osd.5 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.723308+0000 mon.a (mon.0) 703 : cluster [DBG] osd.4 reported immediately failed by osd.6 2022-04-22T17:32:23.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:22 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.723500+0000 mon.a (mon.0) 704 : cluster [DBG] osd.4 reported immediately failed by osd.7 2022-04-22T17:32:23.766 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:23 smithi121 bash[38907]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.4 2022-04-22T17:32:23.781 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:23 smithi121 bash[38952]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-4 2022-04-22T17:32:23.781 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:23 smithi121 bash[38965]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.4 2022-04-22T17:32:23.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:23.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:23.835 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:23.520Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:23.835 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:23.520Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:32:23.895 INFO:tasks.rados.rados.0.smithi115.stdout:1938: done (0 left) 2022-04-22T17:32:23.895 INFO:tasks.rados.rados.0.smithi115.stdout:1939: rollback oid 13 current snap is 196 2022-04-22T17:32:23.895 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 194 2022-04-22T17:32:23.895 INFO:tasks.rados.rados.0.smithi115.stdout:1940: snap_create 2022-04-22T17:32:23.899 INFO:tasks.rados.rados.0.smithi115.stdout:1939: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:32:23.900 INFO:tasks.rados.rados.0.smithi115.stdout:1939: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:32:23.900 INFO:tasks.rados.rados.0.smithi115.stdout:1939: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:32:23.900 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 956 (ObjNum 231 snap 73 seq_num 231) dirty exists 2022-04-22T17:32:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:23 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.847243+0000 mon.a (mon.0) 705 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:32:24.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:23 smithi121 bash[36902]: cluster 2022-04-22T17:32:22.855390+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e447: 8 total, 7 up, 8 in 2022-04-22T17:32:24.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.847243+0000 mon.a (mon.0) 705 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:32:24.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[45495]: cluster 2022-04-22T17:32:22.855390+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e447: 8 total, 7 up, 8 in 2022-04-22T17:32:24.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.847243+0000 mon.a (mon.0) 705 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:32:24.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:23 smithi115 bash[46934]: cluster 2022-04-22T17:32:22.855390+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e447: 8 total, 7 up, 8 in 2022-04-22T17:32:24.830 INFO:tasks.rados.rados.0.smithi115.stdout:1939: done (1 left) 2022-04-22T17:32:24.830 INFO:tasks.rados.rados.0.smithi115.stdout:1940: done (0 left) 2022-04-22T17:32:24.830 INFO:tasks.rados.rados.0.smithi115.stdout:1941: snap_remove snap 185 2022-04-22T17:32:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:24 smithi121 bash[36902]: cluster 2022-04-22T17:32:23.225708+0000 mgr.y (mgr.34107) 218 : cluster [DBG] pgmap v358: 121 pgs: 8 peering, 7 stale+active+clean, 106 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 8 op/s 2022-04-22T17:32:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:24 smithi121 bash[36902]: cluster 2022-04-22T17:32:23.858141+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e448: 8 total, 7 up, 8 in 2022-04-22T17:32:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:24 smithi121 bash[36902]: cluster 2022-04-22T17:32:23.858852+0000 mon.a (mon.0) 708 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:32:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:24 smithi121 bash[36902]: audit 2022-04-22T17:32:24.248185+0000 mon.a (mon.0) 709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:24 smithi121 bash[36902]: cluster 2022-04-22T17:32:24.814157+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e449: 8 total, 7 up, 8 in 2022-04-22T17:32:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[45495]: cluster 2022-04-22T17:32:23.225708+0000 mgr.y (mgr.34107) 218 : cluster [DBG] pgmap v358: 121 pgs: 8 peering, 7 stale+active+clean, 106 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 8 op/s 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[45495]: cluster 2022-04-22T17:32:23.858141+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e448: 8 total, 7 up, 8 in 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[45495]: cluster 2022-04-22T17:32:23.858852+0000 mon.a (mon.0) 708 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[45495]: audit 2022-04-22T17:32:24.248185+0000 mon.a (mon.0) 709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[45495]: cluster 2022-04-22T17:32:24.814157+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e449: 8 total, 7 up, 8 in 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[46934]: cluster 2022-04-22T17:32:23.225708+0000 mgr.y (mgr.34107) 218 : cluster [DBG] pgmap v358: 121 pgs: 8 peering, 7 stale+active+clean, 106 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 8 op/s 2022-04-22T17:32:25.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[46934]: cluster 2022-04-22T17:32:23.858141+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e448: 8 total, 7 up, 8 in 2022-04-22T17:32:25.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[46934]: cluster 2022-04-22T17:32:23.858852+0000 mon.a (mon.0) 708 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:32:25.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[46934]: audit 2022-04-22T17:32:24.248185+0000 mon.a (mon.0) 709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:25.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:24 smithi115 bash[46934]: cluster 2022-04-22T17:32:24.814157+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e449: 8 total, 7 up, 8 in 2022-04-22T17:32:25.830 INFO:tasks.rados.rados.0.smithi115.stdout:1941: done (0 left) 2022-04-22T17:32:25.830 INFO:tasks.rados.rados.0.smithi115.stdout:1942: copy_from oid 10 from oid 48 current snap is 197 2022-04-22T17:32:25.830 INFO:tasks.rados.rados.0.smithi115.stdout:1943: snap_create 2022-04-22T17:32:25.838 INFO:tasks.rados.rados.0.smithi115.stdout:1942: finishing copy_from to smithi11541303-10 2022-04-22T17:32:25.838 INFO:tasks.rados.rados.0.smithi115.stdout:1942: got expected ENOENT (src dne) 2022-04-22T17:32:25.839 INFO:tasks.rados.rados.0.smithi115.stdout:1942: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:32:26.147 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:25 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.4.service: Succeeded. 2022-04-22T17:32:26.147 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:25 smithi121 systemd[1]: Stopped Ceph osd.4 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:32:26.147 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:26 smithi121 systemd[1]: Started Ceph osd.4 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:1942: done (1 left) 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:1943: done (0 left) 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:1944: read oid 11 snap -1 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:1944: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 198 2022-04-22T17:32:26.835 INFO:tasks.rados.rados.0.smithi115.stdout:1945: seq_num 605 ranges {2310144=540672} 2022-04-22T17:32:26.855 INFO:tasks.rados.rados.0.smithi115.stdout:1945: writing smithi11541303-34 from 2310144 to 2850816 tid 1 2022-04-22T17:32:26.855 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 198 2022-04-22T17:32:26.855 INFO:tasks.rados.rados.0.smithi115.stdout:1946: seq_num 606 ranges {9977856=557056} 2022-04-22T17:32:26.914 INFO:tasks.rados.rados.0.smithi115.stdout:1946: writing smithi11541303-45 from 9977856 to 10534912 tid 1 2022-04-22T17:32:26.914 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 198 2022-04-22T17:32:26.914 INFO:tasks.rados.rados.0.smithi115.stdout:1947: seq_num 607 ranges {2539520=573440} 2022-04-22T17:32:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:1947: writing smithi11541303-25 from 2539520 to 3112960 tid 1 2022-04-22T17:32:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:1948: rollback oid 6 current snap is 198 2022-04-22T17:32:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 195 2022-04-22T17:32:26.932 INFO:tasks.rados.rados.0.smithi115.stdout:1944: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:27.122 INFO:tasks.rados.rados.0.smithi115.stdout:1945: finishing write tid 1 to smithi11541303-34 2022-04-22T17:32:27.122 INFO:tasks.rados.rados.0.smithi115.stdout:1945: finishing write tid 2 to smithi11541303-34 2022-04-22T17:32:27.122 INFO:tasks.rados.rados.0.smithi115.stdout:1945: finishing write tid 3 to smithi11541303-34 2022-04-22T17:32:27.122 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 723 (ObjNum 605 snap 198 seq_num 605) dirty exists 2022-04-22T17:32:27.122 INFO:tasks.rados.rados.0.smithi115.stdout:1945: left oid 34 (ObjNum 605 snap 198 seq_num 605) 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1946: finishing write tid 1 to smithi11541303-45 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1946: finishing write tid 2 to smithi11541303-45 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1946: finishing write tid 3 to smithi11541303-45 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1076 (ObjNum 606 snap 198 seq_num 606) dirty exists 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1946: left oid 45 (ObjNum 606 snap 198 seq_num 606) 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1944: done (4 left) 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1945: done (3 left) 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1946: done (2 left) 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1949: setattr oid 20 current snap is 198 2022-04-22T17:32:27.123 INFO:tasks.rados.rados.0.smithi115.stdout:1948: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:32:27.124 INFO:tasks.rados.rados.0.smithi115.stdout:1948: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:32:27.124 INFO:tasks.rados.rados.0.smithi115.stdout:1948: finishing rollback tid 2 to smithi11541303-6 2022-04-22T17:32:27.124 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 672 (ObjNum 575 snap 187 seq_num 575) dirty exists 2022-04-22T17:32:27.124 INFO:tasks.rados.rados.0.smithi115.stdout:1947: finishing write tid 1 to smithi11541303-25 2022-04-22T17:32:27.125 INFO:tasks.rados.rados.0.smithi115.stdout:1947: finishing write tid 2 to smithi11541303-25 2022-04-22T17:32:27.125 INFO:tasks.rados.rados.0.smithi115.stdout:1947: finishing write tid 3 to smithi11541303-25 2022-04-22T17:32:27.125 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 969 (ObjNum 607 snap 198 seq_num 607) dirty exists 2022-04-22T17:32:27.125 INFO:tasks.rados.rados.0.smithi115.stdout:1947: left oid 25 (ObjNum 607 snap 198 seq_num 607) 2022-04-22T17:32:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:1947: done (2 left) 2022-04-22T17:32:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:1948: done (1 left) 2022-04-22T17:32:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:1950: read oid 26 snap 188 2022-04-22T17:32:27.129 INFO:tasks.rados.rados.0.smithi115.stdout:1950: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:27.129 INFO:tasks.rados.rados.0.smithi115.stdout:1951: read oid 17 snap -1 2022-04-22T17:32:27.129 INFO:tasks.rados.rados.0.smithi115.stdout:1951: expect (ObjNum 563 snap 186 seq_num 563) 2022-04-22T17:32:27.129 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 198 2022-04-22T17:32:27.129 INFO:tasks.rados.rados.0.smithi115.stdout:1952: seq_num 608 ranges {2293760=589824} 2022-04-22T17:32:27.146 INFO:tasks.rados.rados.0.smithi115.stdout:1952: writing smithi11541303-31 from 2293760 to 2883584 tid 1 2022-04-22T17:32:27.146 INFO:tasks.rados.rados.0.smithi115.stdout:1953: rollback oid 7 current snap is 198 2022-04-22T17:32:27.146 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 7 to 192 2022-04-22T17:32:27.147 INFO:tasks.rados.rados.0.smithi115.stdout:1954: copy_from oid 14 from oid 3 current snap is 198 2022-04-22T17:32:27.147 INFO:tasks.rados.rados.0.smithi115.stdout:1955: read oid 32 snap -1 2022-04-22T17:32:27.147 INFO:tasks.rados.rados.0.smithi115.stdout:1955: expect deleted 2022-04-22T17:32:27.147 INFO:tasks.rados.rados.0.smithi115.stdout:1956: delete oid 16 current snap is 198 2022-04-22T17:32:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:1956: done (7 left) 2022-04-22T17:32:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:1957: read oid 24 snap -1 2022-04-22T17:32:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:1957: expect (ObjNum 601 snap 195 seq_num 601) 2022-04-22T17:32:27.149 INFO:tasks.rados.rados.0.smithi115.stdout:1958: snap_remove snap 196 2022-04-22T17:32:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: cluster 2022-04-22T17:32:25.226616+0000 mgr.y (mgr.34107) 219 : cluster [DBG] pgmap v361: 121 pgs: 5 activating+undersized, 3 active+undersized+degraded, 8 peering, 2 stale+active+clean, 19 active+undersized, 84 active+clean; 247 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 6.3 MiB/s wr, 19 op/s; 21/456 objects degraded (4.605%) 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: cluster 2022-04-22T17:32:25.814746+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e450: 8 total, 7 up, 8 in 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: cluster 2022-04-22T17:32:25.870874+0000 mon.a (mon.0) 712 : cluster [WRN] Health check failed: Degraded data redundancy: 21/456 objects degraded (4.605%), 3 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: audit 2022-04-22T17:32:26.175702+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: audit 2022-04-22T17:32:26.177389+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: audit 2022-04-22T17:32:26.179588+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:32:27.286 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:26 smithi121 bash[36902]: audit 2022-04-22T17:32:26.181040+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: cluster 2022-04-22T17:32:25.226616+0000 mgr.y (mgr.34107) 219 : cluster [DBG] pgmap v361: 121 pgs: 5 activating+undersized, 3 active+undersized+degraded, 8 peering, 2 stale+active+clean, 19 active+undersized, 84 active+clean; 247 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 6.3 MiB/s wr, 19 op/s; 21/456 objects degraded (4.605%) 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: cluster 2022-04-22T17:32:25.814746+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e450: 8 total, 7 up, 8 in 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: cluster 2022-04-22T17:32:25.870874+0000 mon.a (mon.0) 712 : cluster [WRN] Health check failed: Degraded data redundancy: 21/456 objects degraded (4.605%), 3 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: audit 2022-04-22T17:32:26.175702+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: audit 2022-04-22T17:32:26.177389+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: audit 2022-04-22T17:32:26.179588+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:32:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[45495]: audit 2022-04-22T17:32:26.181040+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: cluster 2022-04-22T17:32:25.226616+0000 mgr.y (mgr.34107) 219 : cluster [DBG] pgmap v361: 121 pgs: 5 activating+undersized, 3 active+undersized+degraded, 8 peering, 2 stale+active+clean, 19 active+undersized, 84 active+clean; 247 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 6.3 MiB/s wr, 19 op/s; 21/456 objects degraded (4.605%) 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: cluster 2022-04-22T17:32:25.814746+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e450: 8 total, 7 up, 8 in 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: cluster 2022-04-22T17:32:25.870874+0000 mon.a (mon.0) 712 : cluster [WRN] Health check failed: Degraded data redundancy: 21/456 objects degraded (4.605%), 3 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: audit 2022-04-22T17:32:26.175702+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: audit 2022-04-22T17:32:26.177389+0000 mon.a (mon.0) 714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: audit 2022-04-22T17:32:26.179588+0000 mon.a (mon.0) 715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:32:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:26 smithi115 bash[46934]: audit 2022-04-22T17:32:26.181040+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:32:27.858 INFO:tasks.rados.rados.0.smithi115.stdout:1958: done (8 left) 2022-04-22T17:32:27.858 INFO:tasks.rados.rados.0.smithi115.stdout:1959: snap_remove snap 197 2022-04-22T17:32:28.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:27 smithi121 bash[36902]: cluster 2022-04-22T17:32:26.819150+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e451: 8 total, 7 up, 8 in 2022-04-22T17:32:28.115 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-04-22T17:32:28.115 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2022-04-22T17:32:28.116 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2022-04-22T17:32:28.116 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2022-04-22T17:32:28.116 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2022-04-22T17:32:28.116 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-04-22T17:32:28.116 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[39150]: --> ceph-volume raw activate successful for osd ID: 4 2022-04-22T17:32:28.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:27 smithi115 bash[45495]: cluster 2022-04-22T17:32:26.819150+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e451: 8 total, 7 up, 8 in 2022-04-22T17:32:28.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:27 smithi115 bash[46934]: cluster 2022-04-22T17:32:26.819150+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e451: 8 total, 7 up, 8 in 2022-04-22T17:32:28.856 INFO:tasks.rados.rados.0.smithi115.stdout:1959: done (8 left) 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:1960: rollback oid 44 current snap is 198 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 178 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:1953: finishing rollback tid 0 to smithi11541303-7 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:1953: finishing rollback tid 1 to smithi11541303-7 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:1953: finishing rollback tid 2 to smithi11541303-7 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 889 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:28.857 INFO:tasks.rados.rados.0.smithi115.stdout:1957: expect (ObjNum 601 snap 195 seq_num 601) 2022-04-22T17:32:28.883 INFO:tasks.rados.rados.0.smithi115.stdout:1950: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:29.084 INFO:tasks.rados.rados.0.smithi115.stdout:1952: finishing write tid 1 to smithi11541303-31 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:1952: finishing write tid 2 to smithi11541303-31 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:1952: finishing write tid 3 to smithi11541303-31 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1248 (ObjNum 608 snap 198 seq_num 608) dirty exists 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:1952: left oid 31 (ObjNum 608 snap 198 seq_num 608) 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 971 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:29.085 INFO:tasks.rados.rados.0.smithi115.stdout:1951: expect (ObjNum 563 snap 186 seq_num 563) 2022-04-22T17:32:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[36902]: cluster 2022-04-22T17:32:27.227626+0000 mgr.y (mgr.34107) 220 : cluster [DBG] pgmap v364: 121 pgs: 5 activating+undersized, 7 active+undersized+degraded, 8 peering, 27 active+undersized, 74 active+clean; 243 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 38/456 objects degraded (8.333%) 2022-04-22T17:32:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:28 smithi121 bash[36902]: cluster 2022-04-22T17:32:27.842915+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e452: 8 total, 7 up, 8 in 2022-04-22T17:32:29.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:28 smithi115 bash[46934]: cluster 2022-04-22T17:32:27.227626+0000 mgr.y (mgr.34107) 220 : cluster [DBG] pgmap v364: 121 pgs: 5 activating+undersized, 7 active+undersized+degraded, 8 peering, 27 active+undersized, 74 active+clean; 243 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 38/456 objects degraded (8.333%) 2022-04-22T17:32:29.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:28 smithi115 bash[46934]: cluster 2022-04-22T17:32:27.842915+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e452: 8 total, 7 up, 8 in 2022-04-22T17:32:29.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:28 smithi115 bash[45495]: cluster 2022-04-22T17:32:27.227626+0000 mgr.y (mgr.34107) 220 : cluster [DBG] pgmap v364: 121 pgs: 5 activating+undersized, 7 active+undersized+degraded, 8 peering, 27 active+undersized, 74 active+clean; 243 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 17 op/s; 38/456 objects degraded (8.333%) 2022-04-22T17:32:29.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:28 smithi115 bash[45495]: cluster 2022-04-22T17:32:27.842915+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e452: 8 total, 7 up, 8 in 2022-04-22T17:32:29.408 INFO:tasks.rados.rados.0.smithi115.stdout:1954: finishing copy_from to smithi11541303-14 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 986 (ObjNum 549 snap 180 seq_num 549) dirty exists 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1954: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1960: finishing rollback tid 0 to smithi11541303-44 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1960: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1960: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 115 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1949: done (8 left) 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1950: done (7 left) 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1951: done (6 left) 2022-04-22T17:32:29.409 INFO:tasks.rados.rados.0.smithi115.stdout:1952: done (5 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1953: done (4 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1954: done (3 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1955: done (2 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1957: done (1 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1960: done (0 left) 2022-04-22T17:32:29.410 INFO:tasks.rados.rados.0.smithi115.stdout:1961: snap_remove snap 192 2022-04-22T17:32:29.832 INFO:tasks.rados.rados.0.smithi115.stdout:1961: done (0 left) 2022-04-22T17:32:29.843 INFO:tasks.rados.rados.0.smithi115.stdout:1962: snap_create 2022-04-22T17:32:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:29 smithi121 bash[36902]: cluster 2022-04-22T17:32:28.840798+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e453: 8 total, 7 up, 8 in 2022-04-22T17:32:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:29 smithi121 bash[36902]: audit 2022-04-22T17:32:29.250942+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:29 smithi121 bash[36902]: cluster 2022-04-22T17:32:29.816627+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e454: 8 total, 7 up, 8 in 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[45495]: cluster 2022-04-22T17:32:28.840798+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e453: 8 total, 7 up, 8 in 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[45495]: audit 2022-04-22T17:32:29.250942+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[45495]: cluster 2022-04-22T17:32:29.816627+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e454: 8 total, 7 up, 8 in 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[46934]: cluster 2022-04-22T17:32:28.840798+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e453: 8 total, 7 up, 8 in 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[46934]: audit 2022-04-22T17:32:29.250942+0000 mon.a (mon.0) 720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:29 smithi115 bash[46934]: cluster 2022-04-22T17:32:29.816627+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e454: 8 total, 7 up, 8 in 2022-04-22T17:32:30.516 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[39329]: debug 2022-04-22T17:32:30.196+0000 7f255be0e700 1 -- 172.21.15.121:0/281762217 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556ade6c6160 con 0x556ade70ac00 2022-04-22T17:32:30.516 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[39329]: debug 2022-04-22T17:32:30.196+0000 7f255be0e700 1 -- 172.21.15.121:0/281762217 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556ade686340 con 0x556ade70ac00 2022-04-22T17:32:30.834 INFO:tasks.rados.rados.0.smithi115.stdout:1962: done (0 left) 2022-04-22T17:32:30.834 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 199 2022-04-22T17:32:30.835 INFO:tasks.rados.rados.0.smithi115.stdout:1963: seq_num 610 ranges {0=622592} 2022-04-22T17:32:30.839 INFO:tasks.rados.rados.0.smithi115.stdout:1963: writing smithi11541303-48 from 0 to 622592 tid 1 2022-04-22T17:32:30.839 INFO:tasks.rados.rados.0.smithi115.stdout:1964: setattr oid 21 current snap is 199 2022-04-22T17:32:30.841 INFO:tasks.rados.rados.0.smithi115.stdout:1965: snap_create 2022-04-22T17:32:30.848 INFO:tasks.rados.rados.0.smithi115.stdout:1963: finishing write tid 1 to smithi11541303-48 2022-04-22T17:32:30.849 INFO:tasks.rados.rados.0.smithi115.stdout:1963: finishing write tid 2 to smithi11541303-48 2022-04-22T17:32:30.850 INFO:tasks.rados.rados.0.smithi115.stdout:1963: finishing write tid 3 to smithi11541303-48 2022-04-22T17:32:30.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1078 (ObjNum 610 snap 199 seq_num 610) dirty exists 2022-04-22T17:32:30.851 INFO:tasks.rados.rados.0.smithi115.stdout:1963: left oid 48 (ObjNum 610 snap 199 seq_num 610) 2022-04-22T17:32:30.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1085 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[36902]: cluster 2022-04-22T17:32:29.228917+0000 mgr.y (mgr.34107) 221 : cluster [DBG] pgmap v367: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 552 KiB/s wr, 16 op/s; 50/465 objects degraded (10.753%) 2022-04-22T17:32:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[36902]: cluster 2022-04-22T17:32:29.851808+0000 mon.a (mon.0) 722 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:32:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[36902]: cluster 2022-04-22T17:32:30.819024+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e455: 8 total, 7 up, 8 in 2022-04-22T17:32:31.017 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:30 smithi121 bash[39329]: debug 2022-04-22T17:32:30.748+0000 7f25634053c0 -1 Falling back to public interface 2022-04-22T17:32:31.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[45495]: cluster 2022-04-22T17:32:29.228917+0000 mgr.y (mgr.34107) 221 : cluster [DBG] pgmap v367: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 552 KiB/s wr, 16 op/s; 50/465 objects degraded (10.753%) 2022-04-22T17:32:31.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[45495]: cluster 2022-04-22T17:32:29.851808+0000 mon.a (mon.0) 722 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:32:31.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[45495]: cluster 2022-04-22T17:32:30.819024+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e455: 8 total, 7 up, 8 in 2022-04-22T17:32:31.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[46934]: cluster 2022-04-22T17:32:29.228917+0000 mgr.y (mgr.34107) 221 : cluster [DBG] pgmap v367: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 552 KiB/s wr, 16 op/s; 50/465 objects degraded (10.753%) 2022-04-22T17:32:31.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[46934]: cluster 2022-04-22T17:32:29.851808+0000 mon.a (mon.0) 722 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:32:31.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:30 smithi115 bash[46934]: cluster 2022-04-22T17:32:30.819024+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e455: 8 total, 7 up, 8 in 2022-04-22T17:32:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1963: done (2 left) 2022-04-22T17:32:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1964: done (1 left) 2022-04-22T17:32:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1965: done (0 left) 2022-04-22T17:32:31.836 INFO:tasks.rados.rados.0.smithi115.stdout:1966: setattr oid 1 current snap is 200 2022-04-22T17:32:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1967: rmattr oid 4 current snap is 200 2022-04-22T17:32:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1967: done (1 left) 2022-04-22T17:32:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1968: copy_from oid 15 from oid 24 current snap is 200 2022-04-22T17:32:31.838 INFO:tasks.rados.rados.0.smithi115.stdout:1969: snap_remove snap 187 2022-04-22T17:32:32.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:31.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:32.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:31.625Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:32.053 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:31.625Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:32.841 INFO:tasks.rados.rados.0.smithi115.stdout:1969: done (2 left) 2022-04-22T17:32:32.841 INFO:tasks.rados.rados.0.smithi115.stdout:1970: setattr oid 39 current snap is 200 2022-04-22T17:32:32.844 INFO:tasks.rados.rados.0.smithi115.stdout:1971: read oid 43 snap 189 2022-04-22T17:32:32.844 INFO:tasks.rados.rados.0.smithi115.stdout:1971: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:32:32.844 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 200 2022-04-22T17:32:32.844 INFO:tasks.rados.rados.0.smithi115.stdout:1972: seq_num 611 ranges {3112960=638976} 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:1972: writing smithi11541303-25 from 3112960 to 3751936 tid 1 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:1973: copy_from oid 8 from oid 5 current snap is 200 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 890 (ObjNum 526 snap 173 seq_num 526) dirty exists 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:1968: finishing copy_from to smithi11541303-15 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 669 (ObjNum 601 snap 195 seq_num 601) dirty exists 2022-04-22T17:32:32.868 INFO:tasks.rados.rados.0.smithi115.stdout:1968: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:32:32.869 INFO:tasks.rados.rados.0.smithi115.stdout:1971: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:32:32.873 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1086 (ObjNum 542 snap 178 seq_num 542) dirty exists 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1966: done (5 left) 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1968: done (4 left) 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1970: done (3 left) 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1971: done (2 left) 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1974: rollback oid 37 current snap is 200 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 190 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1975: read oid 2 snap -1 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1975: expect deleted 2022-04-22T17:32:32.874 INFO:tasks.rados.rados.0.smithi115.stdout:1976: copy_from oid 36 from oid 44 current snap is 200 2022-04-22T17:32:32.875 INFO:tasks.rados.rados.0.smithi115.stdout:1977: copy_from oid 41 from oid 28 current snap is 200 2022-04-22T17:32:32.875 INFO:tasks.rados.rados.0.smithi115.stdout:1978: snap_remove snap 195 2022-04-22T17:32:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:32 smithi121 bash[36902]: cluster 2022-04-22T17:32:31.229973+0000 mgr.y (mgr.34107) 222 : cluster [DBG] pgmap v370: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 552 KiB/s wr, 12 op/s; 51/468 objects degraded (10.897%) 2022-04-22T17:32:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:32 smithi121 bash[36902]: audit 2022-04-22T17:32:31.548536+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:32 smithi121 bash[36902]: cluster 2022-04-22T17:32:31.819745+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e456: 8 total, 7 up, 8 in 2022-04-22T17:32:33.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:32 smithi121 bash[36902]: cluster 2022-04-22T17:32:31.854316+0000 mon.a (mon.0) 726 : cluster [WRN] Health check update: Degraded data redundancy: 51/468 objects degraded (10.897%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:33.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:32 smithi121 bash[36902]: audit 2022-04-22T17:32:32.153984+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[45495]: cluster 2022-04-22T17:32:31.229973+0000 mgr.y (mgr.34107) 222 : cluster [DBG] pgmap v370: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 552 KiB/s wr, 12 op/s; 51/468 objects degraded (10.897%) 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[45495]: audit 2022-04-22T17:32:31.548536+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[45495]: cluster 2022-04-22T17:32:31.819745+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e456: 8 total, 7 up, 8 in 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[45495]: cluster 2022-04-22T17:32:31.854316+0000 mon.a (mon.0) 726 : cluster [WRN] Health check update: Degraded data redundancy: 51/468 objects degraded (10.897%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[45495]: audit 2022-04-22T17:32:32.153984+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[46934]: cluster 2022-04-22T17:32:31.229973+0000 mgr.y (mgr.34107) 222 : cluster [DBG] pgmap v370: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 245 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 552 KiB/s wr, 12 op/s; 51/468 objects degraded (10.897%) 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[46934]: audit 2022-04-22T17:32:31.548536+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[46934]: cluster 2022-04-22T17:32:31.819745+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e456: 8 total, 7 up, 8 in 2022-04-22T17:32:33.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[46934]: cluster 2022-04-22T17:32:31.854316+0000 mon.a (mon.0) 726 : cluster [WRN] Health check update: Degraded data redundancy: 51/468 objects degraded (10.897%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:33.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:32 smithi115 bash[46934]: audit 2022-04-22T17:32:32.153984+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:33.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:33.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:33.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:33.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:33.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:33.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1978: done (6 left) 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1979: read oid 31 snap -1 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1979: expect (ObjNum 608 snap 198 seq_num 608) 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1980: read oid 26 snap 194 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1980: expect deleted 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1981: snap_create 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1972: finishing write tid 1 to smithi11541303-25 2022-04-22T17:32:33.846 INFO:tasks.rados.rados.0.smithi115.stdout:1972: finishing write tid 2 to smithi11541303-25 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1973: finishing copy_from to smithi11541303-8 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1191 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1973: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1972: finishing write tid 3 to smithi11541303-25 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 998 (ObjNum 611 snap 200 seq_num 611) dirty exists 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1972: left oid 25 (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1974: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1974: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1974: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1089 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:32:33.847 INFO:tasks.rados.rados.0.smithi115.stdout:1977: finishing copy_from to smithi11541303-41 2022-04-22T17:32:33.848 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 504 (ObjNum 591 snap 193 seq_num 591) dirty exists 2022-04-22T17:32:33.848 INFO:tasks.rados.rados.0.smithi115.stdout:1977: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:32:33.848 INFO:tasks.rados.rados.0.smithi115.stdout:1976: finishing copy_from to smithi11541303-36 2022-04-22T17:32:33.848 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 115 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:32:33.848 INFO:tasks.rados.rados.0.smithi115.stdout:1976: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:32:33.866 INFO:tasks.rados.rados.0.smithi115.stdout:1979: expect (ObjNum 608 snap 198 seq_num 608) 2022-04-22T17:32:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:33 smithi121 bash[36902]: cluster 2022-04-22T17:32:32.823844+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e457: 8 total, 7 up, 8 in 2022-04-22T17:32:34.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:33 smithi115 bash[45495]: cluster 2022-04-22T17:32:32.823844+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e457: 8 total, 7 up, 8 in 2022-04-22T17:32:34.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:33 smithi115 bash[46934]: cluster 2022-04-22T17:32:32.823844+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e457: 8 total, 7 up, 8 in 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1972: done (8 left) 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1973: done (7 left) 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1974: done (6 left) 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1975: done (5 left) 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1976: done (4 left) 2022-04-22T17:32:34.850 INFO:tasks.rados.rados.0.smithi115.stdout:1977: done (3 left) 2022-04-22T17:32:34.851 INFO:tasks.rados.rados.0.smithi115.stdout:1979: done (2 left) 2022-04-22T17:32:34.851 INFO:tasks.rados.rados.0.smithi115.stdout:1980: done (1 left) 2022-04-22T17:32:34.851 INFO:tasks.rados.rados.0.smithi115.stdout:1981: done (0 left) 2022-04-22T17:32:34.851 INFO:tasks.rados.rados.0.smithi115.stdout:1982: snap_remove snap 190 2022-04-22T17:32:34.853 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[39329]: debug 2022-04-22T17:32:34.556+0000 7f25634053c0 -1 osd.4 446 log_to_monitors true 2022-04-22T17:32:35.266 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[39329]: debug 2022-04-22T17:32:34.880+0000 7f25557a2700 -1 osd.4 446 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:32:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[36902]: cluster 2022-04-22T17:32:33.231090+0000 mgr.y (mgr.34107) 223 : cluster [DBG] pgmap v373: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 248 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 361 KiB/s rd, 895 KiB/s wr, 12 op/s; 54/483 objects degraded (11.180%) 2022-04-22T17:32:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[36902]: cluster 2022-04-22T17:32:33.829935+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e458: 8 total, 7 up, 8 in 2022-04-22T17:32:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[36902]: audit 2022-04-22T17:32:34.253587+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[36902]: audit 2022-04-22T17:32:34.553305+0000 mon.a (mon.0) 731 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:34 smithi121 bash[36902]: audit 2022-04-22T17:32:34.565024+0000 mon.b (mon.2) 3 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[46934]: cluster 2022-04-22T17:32:33.231090+0000 mgr.y (mgr.34107) 223 : cluster [DBG] pgmap v373: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 248 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 361 KiB/s rd, 895 KiB/s wr, 12 op/s; 54/483 objects degraded (11.180%) 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[46934]: cluster 2022-04-22T17:32:33.829935+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e458: 8 total, 7 up, 8 in 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[46934]: audit 2022-04-22T17:32:34.253587+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[46934]: audit 2022-04-22T17:32:34.553305+0000 mon.a (mon.0) 731 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[46934]: audit 2022-04-22T17:32:34.565024+0000 mon.b (mon.2) 3 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[45495]: cluster 2022-04-22T17:32:33.231090+0000 mgr.y (mgr.34107) 223 : cluster [DBG] pgmap v373: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 248 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 361 KiB/s rd, 895 KiB/s wr, 12 op/s; 54/483 objects degraded (11.180%) 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[45495]: cluster 2022-04-22T17:32:33.829935+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e458: 8 total, 7 up, 8 in 2022-04-22T17:32:35.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[45495]: audit 2022-04-22T17:32:34.253587+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:35.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[45495]: audit 2022-04-22T17:32:34.553305+0000 mon.a (mon.0) 731 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:34 smithi115 bash[45495]: audit 2022-04-22T17:32:34.565024+0000 mon.b (mon.2) 3 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-04-22T17:32:35.855 INFO:tasks.rados.rados.0.smithi115.stdout:1982: done (0 left) 2022-04-22T17:32:35.855 INFO:tasks.rados.rados.0.smithi115.stdout:1983: copy_from oid 3 from oid 28 current snap is 201 2022-04-22T17:32:35.855 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 201 2022-04-22T17:32:35.855 INFO:tasks.rados.rados.0.smithi115.stdout:1984: seq_num 612 ranges {10534912=655360} 2022-04-22T17:32:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1984: writing smithi11541303-45 from 10534912 to 11190272 tid 1 2022-04-22T17:32:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1985: read oid 14 snap -1 2022-04-22T17:32:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1985: expect (ObjNum 549 snap 180 seq_num 549) 2022-04-22T17:32:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 201 2022-04-22T17:32:35.926 INFO:tasks.rados.rados.0.smithi115.stdout:1986: seq_num 613 ranges {0=671744} 2022-04-22T17:32:35.929 INFO:tasks.rados.rados.0.smithi115.stdout:1986: writing smithi11541303-32 from 0 to 671744 tid 1 2022-04-22T17:32:35.930 INFO:tasks.rados.rados.0.smithi115.stdout:1987: read oid 19 snap -1 2022-04-22T17:32:35.930 INFO:tasks.rados.rados.0.smithi115.stdout:1987: expect (ObjNum 564 snap 186 seq_num 564) 2022-04-22T17:32:35.930 INFO:tasks.rados.rados.0.smithi115.stdout:1988: rollback oid 6 current snap is 201 2022-04-22T17:32:35.930 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 193 2022-04-22T17:32:35.931 INFO:tasks.rados.rados.0.smithi115.stdout:1989: setattr oid 2 current snap is 201 2022-04-22T17:32:35.931 INFO:tasks.rados.rados.0.smithi115.stdout:1990: snap_create 2022-04-22T17:32:35.934 INFO:tasks.rados.rados.0.smithi115.stdout:1984: finishing write tid 1 to smithi11541303-45 2022-04-22T17:32:35.936 INFO:tasks.rados.rados.0.smithi115.stdout:1984: finishing write tid 2 to smithi11541303-45 2022-04-22T17:32:35.937 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1090 (ObjNum -1115977408 snap 22075 seq_num 3179495376) dirty exists 2022-04-22T17:32:35.939 INFO:tasks.rados.rados.0.smithi115.stdout:1984: finishing write tid 3 to smithi11541303-45 2022-04-22T17:32:35.939 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1080 (ObjNum 612 snap 201 seq_num 612) dirty exists 2022-04-22T17:32:35.940 INFO:tasks.rados.rados.0.smithi115.stdout:1984: left oid 45 (ObjNum 612 snap 201 seq_num 612) 2022-04-22T17:32:35.941 INFO:tasks.rados.rados.0.smithi115.stdout:1985: expect (ObjNum 549 snap 180 seq_num 549) 2022-04-22T17:32:36.091 INFO:tasks.rados.rados.0.smithi115.stdout:1987: expect (ObjNum 564 snap 186 seq_num 564) 2022-04-22T17:32:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:35 smithi121 bash[36902]: audit 2022-04-22T17:32:34.833691+0000 mon.a (mon.0) 732 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:32:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:35 smithi121 bash[36902]: cluster 2022-04-22T17:32:34.833829+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e459: 8 total, 7 up, 8 in 2022-04-22T17:32:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:35 smithi121 bash[36902]: audit 2022-04-22T17:32:34.843495+0000 mon.a (mon.0) 734 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:35 smithi121 bash[36902]: audit 2022-04-22T17:32:34.847838+0000 mon.b (mon.2) 4 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:35 smithi121 bash[36902]: cluster 2022-04-22T17:32:35.830972+0000 mon.a (mon.0) 735 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:32:36.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[45495]: audit 2022-04-22T17:32:34.833691+0000 mon.a (mon.0) 732 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:32:36.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[45495]: cluster 2022-04-22T17:32:34.833829+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e459: 8 total, 7 up, 8 in 2022-04-22T17:32:36.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[45495]: audit 2022-04-22T17:32:34.843495+0000 mon.a (mon.0) 734 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[45495]: audit 2022-04-22T17:32:34.847838+0000 mon.b (mon.2) 4 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[45495]: cluster 2022-04-22T17:32:35.830972+0000 mon.a (mon.0) 735 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:32:36.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[46934]: audit 2022-04-22T17:32:34.833691+0000 mon.a (mon.0) 732 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-04-22T17:32:36.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[46934]: cluster 2022-04-22T17:32:34.833829+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e459: 8 total, 7 up, 8 in 2022-04-22T17:32:36.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[46934]: audit 2022-04-22T17:32:34.843495+0000 mon.a (mon.0) 734 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[46934]: audit 2022-04-22T17:32:34.847838+0000 mon.b (mon.2) 4 : audit [INF] from='osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:32:36.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:35 smithi115 bash[46934]: cluster 2022-04-22T17:32:35.830972+0000 mon.a (mon.0) 735 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:32:36.857 INFO:tasks.rados.rados.0.smithi115.stdout:1984: done (7 left) 2022-04-22T17:32:36.857 INFO:tasks.rados.rados.0.smithi115.stdout:1985: done (6 left) 2022-04-22T17:32:36.857 INFO:tasks.rados.rados.0.smithi115.stdout:1987: done (5 left) 2022-04-22T17:32:36.857 INFO:tasks.rados.rados.0.smithi115.stdout:1989: done (4 left) 2022-04-22T17:32:36.857 INFO:tasks.rados.rados.0.smithi115.stdout:1990: done (3 left) 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1991: read oid 49 snap -1 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1991: expect (ObjNum 593 snap 194 seq_num 593) 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1992: read oid 21 snap -1 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1992: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1993: copy_from oid 11 from oid 7 current snap is 202 2022-04-22T17:32:36.858 INFO:tasks.rados.rados.0.smithi115.stdout:1994: delete oid 16 current snap is 202 2022-04-22T17:32:36.888 INFO:tasks.rados.rados.0.smithi115.stdout:1994: done (6 left) 2022-04-22T17:32:36.888 INFO:tasks.rados.rados.0.smithi115.stdout:1995: rollback oid 18 current snap is 202 2022-04-22T17:32:36.888 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 199 2022-04-22T17:32:36.888 INFO:tasks.rados.rados.0.smithi115.stdout:1996: copy_from oid 29 from oid 22 current snap is 202 2022-04-22T17:32:36.889 INFO:tasks.rados.rados.0.smithi115.stdout:1997: delete oid 4 current snap is 202 2022-04-22T17:32:36.892 INFO:tasks.rados.rados.0.smithi115.stdout:1997: done (8 left) 2022-04-22T17:32:36.892 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 202 2022-04-22T17:32:36.892 INFO:tasks.rados.rados.0.smithi115.stdout:1998: seq_num 616 ranges {0=720896} 2022-04-22T17:32:36.898 INFO:tasks.rados.rados.0.smithi115.stdout:1998: writing smithi11541303-16 from 0 to 720896 tid 1 2022-04-22T17:32:36.898 INFO:tasks.rados.rados.0.smithi115.stdout:1986: finishing write tid 1 to smithi11541303-32 2022-04-22T17:32:36.900 INFO:tasks.rados.rados.0.smithi115.stdout:1999: copy_from oid 2 from oid 42 current snap is 202 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2000: read oid 19 snap -1 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2000: expect (ObjNum 564 snap 186 seq_num 564) 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2001: setattr oid 14 current snap is 202 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2002: copy_from oid 9 from oid 45 current snap is 202 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2003: read oid 44 snap -1 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2003: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2004: copy_from oid 27 from oid 46 current snap is 202 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout:2005: rmattr oid 47 current snap is 202 2022-04-22T17:32:36.901 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:32:36.902 INFO:tasks.rados.rados.0.smithi115.stdout:1999: finishing copy_from to smithi11541303-2 2022-04-22T17:32:36.902 INFO:tasks.rados.rados.0.smithi115.stdout:1999: got expected ENOENT (src dne) 2022-04-22T17:32:36.902 INFO:tasks.rados.rados.0.smithi115.stdout:1999: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:32:36.903 INFO:tasks.rados.rados.0.smithi115.stdout:1999: done (15 left) 2022-04-22T17:32:36.903 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 202 2022-04-22T17:32:36.903 INFO:tasks.rados.rados.0.smithi115.stdout:2006: seq_num 617 ranges {0=737280} 2022-04-22T17:32:36.907 INFO:tasks.rados.rados.0.smithi115.stdout:2006: writing smithi11541303-23 from 0 to 737280 tid 1 2022-04-22T17:32:36.907 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 16 2022-04-22T17:32:36.908 INFO:tasks.rados.rados.0.smithi115.stdout:1986: finishing write tid 2 to smithi11541303-32 2022-04-22T17:32:36.908 INFO:tasks.rados.rados.0.smithi115.stdout:1991: expect (ObjNum 593 snap 194 seq_num 593) 2022-04-22T17:32:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:36 smithi121 bash[36902]: cluster 2022-04-22T17:32:35.232027+0000 mgr.y (mgr.34107) 224 : cluster [DBG] pgmap v376: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 15 op/s; 55/492 objects degraded (11.179%) 2022-04-22T17:32:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:36 smithi121 bash[36902]: cluster 2022-04-22T17:32:35.837946+0000 mon.a (mon.0) 736 : cluster [INF] osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945] boot 2022-04-22T17:32:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:36 smithi121 bash[36902]: cluster 2022-04-22T17:32:35.838030+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2022-04-22T17:32:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:36 smithi121 bash[36902]: audit 2022-04-22T17:32:35.838385+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:32:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:36 smithi121 bash[36902]: cluster 2022-04-22T17:32:36.841583+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1986: finishing write tid 3 to smithi11541303-32 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 990 (ObjNum 613 snap 201 seq_num 613) dirty exists 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1986: left oid 32 (ObjNum 613 snap 201 seq_num 613) 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1995: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1983: finishing copy_from to smithi11541303-3 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 504 (ObjNum 591 snap 193 seq_num 591) dirty exists 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1998: finishing write tid 1 to smithi11541303-16 2022-04-22T17:32:37.125 INFO:tasks.rados.rados.0.smithi115.stdout:1983: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:2004: finishing copy_from to smithi11541303-27 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:2004: got expected ENOENT (src dne) 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:2004: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:1995: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1109 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:1995: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1274 (ObjNum 596 snap 195 seq_num 596) dirty exists 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:1998: finishing write tid 2 to smithi11541303-16 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:2006: finishing write tid 1 to smithi11541303-23 2022-04-22T17:32:37.126 INFO:tasks.rados.rados.0.smithi115.stdout:2006: finishing write tid 2 to smithi11541303-23 2022-04-22T17:32:37.127 INFO:tasks.rados.rados.0.smithi115.stdout:2006: finishing write tid 3 to smithi11541303-23 2022-04-22T17:32:37.127 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1000 (ObjNum 617 snap 202 seq_num 617) dirty exists 2022-04-22T17:32:37.127 INFO:tasks.rados.rados.0.smithi115.stdout:2006: left oid 23 (ObjNum 617 snap 202 seq_num 617) 2022-04-22T17:32:37.127 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1057 (ObjNum 549 snap 180 seq_num 549) dirty exists 2022-04-22T17:32:37.127 INFO:tasks.rados.rados.0.smithi115.stdout:2003: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:32:37.184 INFO:tasks.rados.rados.0.smithi115.stdout:1998: finishing write tid 3 to smithi11541303-16 2022-04-22T17:32:37.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 725 (ObjNum 616 snap 202 seq_num 616) dirty exists 2022-04-22T17:32:37.184 INFO:tasks.rados.rados.0.smithi115.stdout:1998: left oid 16 (ObjNum 616 snap 202 seq_num 616) 2022-04-22T17:32:37.184 INFO:tasks.rados.rados.0.smithi115.stdout:1996: finishing copy_from to smithi11541303-29 2022-04-22T17:32:37.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1084 (ObjNum 592 snap 194 seq_num 592) dirty exists 2022-04-22T17:32:37.185 INFO:tasks.rados.rados.0.smithi115.stdout:1996: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:32:37.185 INFO:tasks.rados.rados.0.smithi115.stdout:1993: finishing copy_from to smithi11541303-11 2022-04-22T17:32:37.185 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 889 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:37.185 INFO:tasks.rados.rados.0.smithi115.stdout:1993: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:32:37.185 INFO:tasks.rados.rados.0.smithi115.stdout:1992: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:37.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[45495]: cluster 2022-04-22T17:32:35.232027+0000 mgr.y (mgr.34107) 224 : cluster [DBG] pgmap v376: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 15 op/s; 55/492 objects degraded (11.179%) 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[45495]: cluster 2022-04-22T17:32:35.837946+0000 mon.a (mon.0) 736 : cluster [INF] osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945] boot 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[45495]: cluster 2022-04-22T17:32:35.838030+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[45495]: audit 2022-04-22T17:32:35.838385+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[45495]: cluster 2022-04-22T17:32:36.841583+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[46934]: cluster 2022-04-22T17:32:35.232027+0000 mgr.y (mgr.34107) 224 : cluster [DBG] pgmap v376: 121 pgs: 12 active+undersized+degraded, 35 active+undersized, 74 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 15 op/s; 55/492 objects degraded (11.179%) 2022-04-22T17:32:37.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[46934]: cluster 2022-04-22T17:32:35.837946+0000 mon.a (mon.0) 736 : cluster [INF] osd.4 [v2:172.21.15.121:6800/531418945,v1:172.21.15.121:6801/531418945] boot 2022-04-22T17:32:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[46934]: cluster 2022-04-22T17:32:35.838030+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2022-04-22T17:32:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[46934]: audit 2022-04-22T17:32:35.838385+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-04-22T17:32:37.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:36 smithi115 bash[46934]: cluster 2022-04-22T17:32:36.841583+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2022-04-22T17:32:37.880 INFO:tasks.rados.rados.0.smithi115.stdout:2000: expect (ObjNum 564 snap 186 seq_num 564) 2022-04-22T17:32:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:37 smithi121 bash[36902]: cluster 2022-04-22T17:32:37.851083+0000 mon.a (mon.0) 740 : cluster [WRN] Health check update: Degraded data redundancy: 43/498 objects degraded (8.635%), 9 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:38.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:37 smithi115 bash[45495]: cluster 2022-04-22T17:32:37.851083+0000 mon.a (mon.0) 740 : cluster [WRN] Health check update: Degraded data redundancy: 43/498 objects degraded (8.635%), 9 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:38.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:37 smithi115 bash[46934]: cluster 2022-04-22T17:32:37.851083+0000 mon.a (mon.0) 740 : cluster [WRN] Health check update: Degraded data redundancy: 43/498 objects degraded (8.635%), 9 pgs degraded (PG_DEGRADED) 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:2002: finishing copy_from to smithi11541303-9 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1080 (ObjNum 612 snap 201 seq_num 612) dirty exists 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:2002: finishing copy_from racing read to smithi11541303-9 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:1988: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:1988: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:1988: finishing rollback tid 2 to smithi11541303-6 2022-04-22T17:32:38.443 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 675 (ObjNum 575 snap 187 seq_num 575) dirty exists 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1983: done (15 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1986: done (14 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1988: done (13 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1991: done (12 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1992: done (11 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1993: done (10 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1995: done (9 left) 2022-04-22T17:32:38.444 INFO:tasks.rados.rados.0.smithi115.stdout:1996: done (8 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:1998: done (7 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2000: done (6 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2001: done (5 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2002: done (4 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2003: done (3 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2004: done (2 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2005: done (1 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2006: done (0 left) 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:2007: rollback oid 4 current snap is 202 2022-04-22T17:32:38.445 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 188 2022-04-22T17:32:38.446 INFO:tasks.rados.rados.0.smithi115.stdout:2008: copy_from oid 50 from oid 47 current snap is 202 2022-04-22T17:32:38.446 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 202 2022-04-22T17:32:38.446 INFO:tasks.rados.rados.0.smithi115.stdout:2009: seq_num 618 ranges {3670016=753664} 2022-04-22T17:32:38.474 INFO:tasks.rados.rados.0.smithi115.stdout:2009: writing smithi11541303-14 from 3670016 to 4423680 tid 1 2022-04-22T17:32:38.474 INFO:tasks.rados.rados.0.smithi115.stdout:2010: read oid 2 snap -1 2022-04-22T17:32:38.474 INFO:tasks.rados.rados.0.smithi115.stdout:2010: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:32:38.474 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 202 2022-04-22T17:32:38.474 INFO:tasks.rados.rados.0.smithi115.stdout:2007: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:32:38.475 INFO:tasks.rados.rados.0.smithi115.stdout:2011: seq_num 619 ranges {2883584=770048} 2022-04-22T17:32:38.494 INFO:tasks.rados.rados.0.smithi115.stdout:2011: writing smithi11541303-31 from 2883584 to 3653632 tid 1 2022-04-22T17:32:38.494 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 202 2022-04-22T17:32:38.494 INFO:tasks.rados.rados.0.smithi115.stdout:2012: seq_num 620 ranges {671744=794624} 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:2012: writing smithi11541303-32 from 671744 to 1466368 tid 1 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:2013: delete oid 22 current snap is 202 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:2007: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1083 (ObjNum 558 snap 182 seq_num 558) dirty exists 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:2008: finishing copy_from to smithi11541303-50 2022-04-22T17:32:38.503 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1109 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:32:38.504 INFO:tasks.rados.rados.0.smithi115.stdout:2008: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:32:38.504 INFO:tasks.rados.rados.0.smithi115.stdout:2010: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2009: finishing write tid 1 to smithi11541303-14 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2009: finishing write tid 2 to smithi11541303-14 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2007: done (6 left) 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2008: done (5 left) 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2010: done (4 left) 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2013: done (3 left) 2022-04-22T17:32:38.508 INFO:tasks.rados.rados.0.smithi115.stdout:2014: read oid 20 snap -1 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2014: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 202 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2009: finishing write tid 3 to smithi11541303-14 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1086 (ObjNum 618 snap 202 seq_num 618) dirty exists 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2009: left oid 14 (ObjNum 618 snap 202 seq_num 618) 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2011: finishing write tid 1 to smithi11541303-31 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2011: finishing write tid 2 to smithi11541303-31 2022-04-22T17:32:38.509 INFO:tasks.rados.rados.0.smithi115.stdout:2015: seq_num 622 ranges {303104=425984,729088=401408} 2022-04-22T17:32:38.514 INFO:tasks.rados.rados.0.smithi115.stdout:2015: writing smithi11541303-28 from 303104 to 729088 tid 1 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2015: writing smithi11541303-28 from 729088 to 1130496 tid 2 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2009: done (4 left) 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2016: snap_create 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2011: finishing write tid 3 to smithi11541303-31 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1279 (ObjNum 619 snap 202 seq_num 619) dirty exists 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2011: left oid 31 (ObjNum 619 snap 202 seq_num 619) 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2012: finishing write tid 1 to smithi11541303-32 2022-04-22T17:32:38.518 INFO:tasks.rados.rados.0.smithi115.stdout:2012: finishing write tid 2 to smithi11541303-32 2022-04-22T17:32:38.519 INFO:tasks.rados.rados.0.smithi115.stdout:2012: finishing write tid 3 to smithi11541303-32 2022-04-22T17:32:38.519 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 992 (ObjNum 620 snap 202 seq_num 620) dirty exists 2022-04-22T17:32:38.519 INFO:tasks.rados.rados.0.smithi115.stdout:2012: left oid 32 (ObjNum 620 snap 202 seq_num 620) 2022-04-22T17:32:38.519 INFO:tasks.rados.rados.0.smithi115.stdout:2015: finishing write tid 1 to smithi11541303-28 2022-04-22T17:32:38.521 INFO:tasks.rados.rados.0.smithi115.stdout:2015: finishing write tid 2 to smithi11541303-28 2022-04-22T17:32:38.521 INFO:tasks.rados.rados.0.smithi115.stdout:2015: finishing write tid 3 to smithi11541303-28 2022-04-22T17:32:38.523 INFO:tasks.rados.rados.0.smithi115.stdout:2015: finishing write tid 4 to smithi11541303-28 2022-04-22T17:32:38.523 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1089 (ObjNum 622 snap 202 seq_num 622) dirty exists 2022-04-22T17:32:38.523 INFO:tasks.rados.rados.0.smithi115.stdout:2015: left oid 28 (ObjNum 622 snap 202 seq_num 622) 2022-04-22T17:32:38.551 INFO:tasks.rados.rados.0.smithi115.stdout:2014: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:39.252 INFO:tasks.rados.rados.0.smithi115.stdout:2011: done (4 left) 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2012: done (3 left) 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2014: done (2 left) 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2015: done (1 left) 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2016: done (0 left) 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2017: copy_from oid 35 from oid 2 current snap is 203 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2018: copy_from oid 24 from oid 12 current snap is 203 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2019: copy_from oid 44 from oid 36 current snap is 203 2022-04-22T17:32:39.253 INFO:tasks.rados.rados.0.smithi115.stdout:2020: snap_create 2022-04-22T17:32:39.260 INFO:tasks.rados.rados.0.smithi115.stdout:2017: finishing copy_from to smithi11541303-35 2022-04-22T17:32:39.260 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1090 (ObjNum -1115309192 snap 22075 seq_num 3178305328) dirty exists 2022-04-22T17:32:39.261 INFO:tasks.rados.rados.0.smithi115.stdout:2017: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:32:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:38 smithi121 bash[36902]: cluster 2022-04-22T17:32:37.232933+0000 mgr.y (mgr.34107) 225 : cluster [DBG] pgmap v379: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 8 active+undersized+degraded, 30 active+undersized, 75 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 17 op/s; 43/498 objects degraded (8.635%) 2022-04-22T17:32:39.267 INFO:tasks.rados.rados.0.smithi115.stdout:2019: finishing copy_from to smithi11541303-44 2022-04-22T17:32:39.267 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 115 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:32:39.268 INFO:tasks.rados.rados.0.smithi115.stdout:2019: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:32:39.272 INFO:tasks.rados.rados.0.smithi115.stdout:2018: finishing copy_from to smithi11541303-24 2022-04-22T17:32:39.273 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1068 (ObjNum 599 snap 195 seq_num 599) dirty exists 2022-04-22T17:32:39.273 INFO:tasks.rados.rados.0.smithi115.stdout:2018: finishing copy_from racing read to smithi11541303-24 2022-04-22T17:32:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:38 smithi115 bash[45495]: cluster 2022-04-22T17:32:37.232933+0000 mgr.y (mgr.34107) 225 : cluster [DBG] pgmap v379: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 8 active+undersized+degraded, 30 active+undersized, 75 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 17 op/s; 43/498 objects degraded (8.635%) 2022-04-22T17:32:39.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:38 smithi115 bash[46934]: cluster 2022-04-22T17:32:37.232933+0000 mgr.y (mgr.34107) 225 : cluster [DBG] pgmap v379: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 8 active+undersized+degraded, 30 active+undersized, 75 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 17 op/s; 43/498 objects degraded (8.635%) 2022-04-22T17:32:39.940 INFO:tasks.rados.rados.0.smithi115.stdout:2017: done (3 left) 2022-04-22T17:32:39.940 INFO:tasks.rados.rados.0.smithi115.stdout:2018: done (2 left) 2022-04-22T17:32:39.940 INFO:tasks.rados.rados.0.smithi115.stdout:2019: done (1 left) 2022-04-22T17:32:39.941 INFO:tasks.rados.rados.0.smithi115.stdout:2020: done (0 left) 2022-04-22T17:32:39.941 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 204 2022-04-22T17:32:39.941 INFO:tasks.rados.rados.0.smithi115.stdout:2021: seq_num 623 ranges {11190272=442368,11632640=401408} 2022-04-22T17:32:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:2021: writing smithi11541303-45 from 11190272 to 11632640 tid 1 2022-04-22T17:32:40.011 INFO:tasks.rados.rados.0.smithi115.stdout:2021: writing smithi11541303-45 from 11632640 to 12034048 tid 2 2022-04-22T17:32:40.011 INFO:tasks.rados.rados.0.smithi115.stdout:2022: rollback oid 15 current snap is 204 2022-04-22T17:32:40.011 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 15 to 189 2022-04-22T17:32:40.011 INFO:tasks.rados.rados.0.smithi115.stdout:2023: snap_remove snap 186 2022-04-22T17:32:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:39 smithi121 bash[36902]: cluster 2022-04-22T17:32:38.916690+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2022-04-22T17:32:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:39 smithi121 bash[36902]: audit 2022-04-22T17:32:39.256382+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:40.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:39 smithi115 bash[45495]: cluster 2022-04-22T17:32:38.916690+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2022-04-22T17:32:40.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:39 smithi115 bash[45495]: audit 2022-04-22T17:32:39.256382+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:40.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:39 smithi115 bash[46934]: cluster 2022-04-22T17:32:38.916690+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2022-04-22T17:32:40.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:39 smithi115 bash[46934]: audit 2022-04-22T17:32:39.256382+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:40.950 INFO:tasks.rados.rados.0.smithi115.stdout:2023: done (2 left) 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2024: read oid 8 snap 189 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2024: expect (ObjNum 533 snap 177 seq_num 533) 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2025: rollback oid 47 current snap is 204 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 47 to 201 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2021: finishing write tid 1 to smithi11541303-45 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2021: finishing write tid 2 to smithi11541303-45 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2021: finishing write tid 3 to smithi11541303-45 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2022: finishing rollback tid 0 to smithi11541303-15 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2022: finishing rollback tid 1 to smithi11541303-15 2022-04-22T17:32:40.951 INFO:tasks.rados.rados.0.smithi115.stdout:2022: finishing rollback tid 2 to smithi11541303-15 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 724 (ObjNum 567 snap 186 seq_num 567) dirty exists 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:2021: finishing write tid 4 to smithi11541303-45 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1086 (ObjNum 623 snap 204 seq_num 623) dirty exists 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:2021: left oid 45 (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:2021: done (3 left) 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:2022: done (2 left) 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 204 2022-04-22T17:32:40.952 INFO:tasks.rados.rados.0.smithi115.stdout:2026: seq_num 624 ranges {3661824=458752,4120576=401408} 2022-04-22T17:32:40.978 INFO:tasks.rados.rados.0.smithi115.stdout:2026: writing smithi11541303-49 from 3661824 to 4120576 tid 1 2022-04-22T17:32:40.981 INFO:tasks.rados.rados.0.smithi115.stdout:2026: writing smithi11541303-49 from 4120576 to 4521984 tid 2 2022-04-22T17:32:40.981 INFO:tasks.rados.rados.0.smithi115.stdout:2027: copy_from oid 32 from oid 11 current snap is 204 2022-04-22T17:32:40.981 INFO:tasks.rados.rados.0.smithi115.stdout:2025: finishing rollback tid 0 to smithi11541303-47 2022-04-22T17:32:40.981 INFO:tasks.rados.rados.0.smithi115.stdout:2025: finishing rollback tid 1 to smithi11541303-47 2022-04-22T17:32:40.982 INFO:tasks.rados.rados.0.smithi115.stdout:2025: finishing rollback tid 2 to smithi11541303-47 2022-04-22T17:32:40.982 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1112 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:32:40.982 INFO:tasks.rados.rados.0.smithi115.stdout:2025: done (3 left) 2022-04-22T17:32:40.982 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 204 2022-04-22T17:32:40.982 INFO:tasks.rados.rados.0.smithi115.stdout:2028: seq_num 625 ranges {303104=475136,778240=401408} 2022-04-22T17:32:40.986 INFO:tasks.rados.rados.0.smithi115.stdout:2028: writing smithi11541303-3 from 303104 to 778240 tid 1 2022-04-22T17:32:40.988 INFO:tasks.rados.rados.0.smithi115.stdout:2028: writing smithi11541303-3 from 778240 to 1179648 tid 2 2022-04-22T17:32:40.989 INFO:tasks.rados.rados.0.smithi115.stdout:2029: snap_remove snap 203 2022-04-22T17:32:41.052 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[42914]: debug 2022-04-22T17:32:40.674+0000 7f8eea62c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:41.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: cluster 2022-04-22T17:32:39.234085+0000 mgr.y (mgr.34107) 226 : cluster [DBG] pgmap v381: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 3 active+undersized+degraded, 7 active+undersized, 103 active+clean; 267 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 26/519 objects degraded (5.010%); 4.4 MiB/s, 0 objects/s recovering 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: cluster 2022-04-22T17:32:39.924800+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.655041+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.661176+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.669765+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.672734+0000 mon.a (mon.0) 747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.674390+0000 mon.a (mon.0) 748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.675741+0000 mon.a (mon.0) 749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[45495]: audit 2022-04-22T17:32:40.677580+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: cluster 2022-04-22T17:32:39.234085+0000 mgr.y (mgr.34107) 226 : cluster [DBG] pgmap v381: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 3 active+undersized+degraded, 7 active+undersized, 103 active+clean; 267 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 26/519 objects degraded (5.010%); 4.4 MiB/s, 0 objects/s recovering 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: cluster 2022-04-22T17:32:39.924800+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.655041+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.661176+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.669765+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.672734+0000 mon.a (mon.0) 747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.674390+0000 mon.a (mon.0) 748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.675741+0000 mon.a (mon.0) 749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:40 smithi115 bash[46934]: audit 2022-04-22T17:32:40.677580+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: cluster 2022-04-22T17:32:39.234085+0000 mgr.y (mgr.34107) 226 : cluster [DBG] pgmap v381: 121 pgs: 1 activating+degraded, 1 activating, 6 peering, 3 active+undersized+degraded, 7 active+undersized, 103 active+clean; 267 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 26/519 objects degraded (5.010%); 4.4 MiB/s, 0 objects/s recovering 2022-04-22T17:32:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: cluster 2022-04-22T17:32:39.924800+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.655041+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.661176+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.669765+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.672734+0000 mon.a (mon.0) 747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.674390+0000 mon.a (mon.0) 748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.675741+0000 mon.a (mon.0) 749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:40 smithi121 bash[36902]: audit 2022-04-22T17:32:40.677580+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:41.949 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:41.626Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:41.949 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:41.626Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:41.950 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:41.626Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:32:41.961 INFO:tasks.rados.rados.0.smithi115.stdout:2029: done (4 left) 2022-04-22T17:32:41.961 INFO:tasks.rados.rados.0.smithi115.stdout:2030: snap_remove snap 199 2022-04-22T17:32:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:41 smithi121 bash[36902]: cephadm 2022-04-22T17:32:40.641057+0000 mgr.y (mgr.34107) 227 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:32:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:41 smithi121 bash[36902]: audit 2022-04-22T17:32:40.677810+0000 mgr.y (mgr.34107) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:41 smithi121 bash[36902]: cephadm 2022-04-22T17:32:40.678864+0000 mgr.y (mgr.34107) 229 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:41 smithi121 bash[36902]: cluster 2022-04-22T17:32:40.934968+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2022-04-22T17:32:42.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[45495]: cephadm 2022-04-22T17:32:40.641057+0000 mgr.y (mgr.34107) 227 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:32:42.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[45495]: audit 2022-04-22T17:32:40.677810+0000 mgr.y (mgr.34107) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:42.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[45495]: cephadm 2022-04-22T17:32:40.678864+0000 mgr.y (mgr.34107) 229 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:42.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[45495]: cluster 2022-04-22T17:32:40.934968+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2022-04-22T17:32:42.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[46934]: cephadm 2022-04-22T17:32:40.641057+0000 mgr.y (mgr.34107) 227 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:32:42.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[46934]: audit 2022-04-22T17:32:40.677810+0000 mgr.y (mgr.34107) 228 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:42.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[46934]: cephadm 2022-04-22T17:32:40.678864+0000 mgr.y (mgr.34107) 229 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2022-04-22T17:32:42.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:41 smithi115 bash[46934]: cluster 2022-04-22T17:32:40.934968+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2022-04-22T17:32:42.973 INFO:tasks.rados.rados.0.smithi115.stdout:2024: expect (ObjNum 533 snap 177 seq_num 533) 2022-04-22T17:32:43.101 INFO:tasks.rados.rados.0.smithi115.stdout:2024: done (4 left) 2022-04-22T17:32:43.101 INFO:tasks.rados.rados.0.smithi115.stdout:2030: done (3 left) 2022-04-22T17:32:43.101 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 204 2022-04-22T17:32:43.101 INFO:tasks.rados.rados.0.smithi115.stdout:2026: finishing write tid 1 to smithi11541303-49 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2026: finishing write tid 2 to smithi11541303-49 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2026: finishing write tid 3 to smithi11541303-49 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2026: finishing write tid 4 to smithi11541303-49 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1287 (ObjNum 624 snap 204 seq_num 624) dirty exists 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2026: left oid 49 (ObjNum 624 snap 204 seq_num 624) 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2028: finishing write tid 1 to smithi11541303-3 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2028: finishing write tid 2 to smithi11541303-3 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2028: finishing write tid 3 to smithi11541303-3 2022-04-22T17:32:43.102 INFO:tasks.rados.rados.0.smithi115.stdout:2031: seq_num 626 ranges {3604480=491520,4096000=401408} 2022-04-22T17:32:43.125 INFO:tasks.rados.rados.0.smithi115.stdout:2031: writing smithi11541303-40 from 3604480 to 4096000 tid 1 2022-04-22T17:32:43.127 INFO:tasks.rados.rados.0.smithi115.stdout:2031: writing smithi11541303-40 from 4096000 to 4497408 tid 2 2022-04-22T17:32:43.127 INFO:tasks.rados.rados.0.smithi115.stdout:2026: done (3 left) 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:2032: setattr oid 15 current snap is 204 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:2028: finishing write tid 4 to smithi11541303-3 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1003 (ObjNum 625 snap 204 seq_num 625) dirty exists 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:2028: left oid 3 (ObjNum 625 snap 204 seq_num 625) 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:2027: finishing copy_from to smithi11541303-32 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 889 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:43.128 INFO:tasks.rados.rados.0.smithi115.stdout:2027: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:2027: done (3 left) 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:2028: done (2 left) 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:2033: rollback oid 11 current snap is 204 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 193 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:2034: read oid 47 snap -1 2022-04-22T17:32:43.129 INFO:tasks.rados.rados.0.smithi115.stdout:2034: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:43.130 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 204 2022-04-22T17:32:43.130 INFO:tasks.rados.rados.0.smithi115.stdout:2035: seq_num 627 ranges {6037504=507904,6545408=401408} 2022-04-22T17:32:43.166 INFO:tasks.rados.rados.0.smithi115.stdout:2035: writing smithi11541303-17 from 6037504 to 6545408 tid 1 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2035: writing smithi11541303-17 from 6545408 to 6946816 tid 2 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2036: read oid 37 snap -1 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2036: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2037: read oid 39 snap -1 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2037: expect (ObjNum 542 snap 178 seq_num 542) 2022-04-22T17:32:43.199 INFO:tasks.rados.rados.0.smithi115.stdout:2038: read oid 7 snap -1 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2038: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2039: snap_create 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 725 (ObjNum 567 snap 186 seq_num 567) dirty exists 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2033: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2033: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2033: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1006 (ObjNum 481 snap 167 seq_num 481) dirty exists 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2031: finishing write tid 1 to smithi11541303-40 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2031: finishing write tid 2 to smithi11541303-40 2022-04-22T17:32:43.200 INFO:tasks.rados.rados.0.smithi115.stdout:2031: finishing write tid 3 to smithi11541303-40 2022-04-22T17:32:43.201 INFO:tasks.rados.rados.0.smithi115.stdout:2031: finishing write tid 4 to smithi11541303-40 2022-04-22T17:32:43.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1115 (ObjNum 626 snap 204 seq_num 626) dirty exists 2022-04-22T17:32:43.201 INFO:tasks.rados.rados.0.smithi115.stdout:2031: left oid 40 (ObjNum 626 snap 204 seq_num 626) 2022-04-22T17:32:43.201 INFO:tasks.rados.rados.0.smithi115.stdout:2034: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:42 smithi121 bash[36902]: cluster 2022-04-22T17:32:41.234963+0000 mgr.y (mgr.34107) 230 : cluster [DBG] pgmap v384: 121 pgs: 1 activating+degraded, 1 activating, 119 active+clean; 278 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.4 MiB/s wr, 24 op/s; 8/528 objects degraded (1.515%); 8.6 MiB/s, 1 objects/s recovering 2022-04-22T17:32:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:42 smithi121 bash[36902]: cluster 2022-04-22T17:32:41.946529+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2022-04-22T17:32:43.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:42 smithi115 bash[46934]: cluster 2022-04-22T17:32:41.234963+0000 mgr.y (mgr.34107) 230 : cluster [DBG] pgmap v384: 121 pgs: 1 activating+degraded, 1 activating, 119 active+clean; 278 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.4 MiB/s wr, 24 op/s; 8/528 objects degraded (1.515%); 8.6 MiB/s, 1 objects/s recovering 2022-04-22T17:32:43.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:42 smithi115 bash[46934]: cluster 2022-04-22T17:32:41.946529+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2022-04-22T17:32:43.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:42 smithi115 bash[45495]: cluster 2022-04-22T17:32:41.234963+0000 mgr.y (mgr.34107) 230 : cluster [DBG] pgmap v384: 121 pgs: 1 activating+degraded, 1 activating, 119 active+clean; 278 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.4 MiB/s wr, 24 op/s; 8/528 objects degraded (1.515%); 8.6 MiB/s, 1 objects/s recovering 2022-04-22T17:32:43.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:42 smithi115 bash[45495]: cluster 2022-04-22T17:32:41.946529+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2035: finishing write tid 1 to smithi11541303-17 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2035: finishing write tid 2 to smithi11541303-17 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2035: finishing write tid 3 to smithi11541303-17 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2035: finishing write tid 4 to smithi11541303-17 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1026 (ObjNum 627 snap 204 seq_num 627) dirty exists 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2035: left oid 17 (ObjNum 627 snap 204 seq_num 627) 2022-04-22T17:32:43.305 INFO:tasks.rados.rados.0.smithi115.stdout:2036: expect (ObjNum 481 snap 167 seq_num 481) 2022-04-22T17:32:43.486 INFO:tasks.rados.rados.0.smithi115.stdout:2037: expect (ObjNum 542 snap 178 seq_num 542) 2022-04-22T17:32:43.534 INFO:tasks.rados.rados.0.smithi115.stdout:2038: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:43.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:43.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:43.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2031: done (8 left) 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2032: done (7 left) 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2033: done (6 left) 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2034: done (5 left) 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2035: done (4 left) 2022-04-22T17:32:44.191 INFO:tasks.rados.rados.0.smithi115.stdout:2036: done (3 left) 2022-04-22T17:32:44.192 INFO:tasks.rados.rados.0.smithi115.stdout:2037: done (2 left) 2022-04-22T17:32:44.192 INFO:tasks.rados.rados.0.smithi115.stdout:2038: done (1 left) 2022-04-22T17:32:44.192 INFO:tasks.rados.rados.0.smithi115.stdout:2039: done (0 left) 2022-04-22T17:32:44.192 INFO:tasks.rados.rados.0.smithi115.stdout:2040: snap_create 2022-04-22T17:32:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:43 smithi121 bash[36902]: cluster 2022-04-22T17:32:42.957235+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2022-04-22T17:32:44.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:43 smithi115 bash[45495]: cluster 2022-04-22T17:32:42.957235+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2022-04-22T17:32:44.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:43 smithi115 bash[46934]: cluster 2022-04-22T17:32:42.957235+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2022-04-22T17:32:44.857 INFO:tasks.rados.rados.0.smithi115.stdout:2040: done (0 left) 2022-04-22T17:32:44.858 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 206 2022-04-22T17:32:44.858 INFO:tasks.rados.rados.0.smithi115.stdout:2041: seq_num 628 ranges {3710976=524288,4235264=401408} 2022-04-22T17:32:44.873 INFO:tasks.rados.rados.0.smithi115.stdout:2041: writing smithi11541303-11 from 3710976 to 4235264 tid 1 2022-04-22T17:32:44.876 INFO:tasks.rados.rados.0.smithi115.stdout:2041: writing smithi11541303-11 from 4235264 to 4636672 tid 2 2022-04-22T17:32:44.876 INFO:tasks.rados.rados.0.smithi115.stdout:2042: delete oid 48 current snap is 206 2022-04-22T17:32:44.884 INFO:tasks.rados.rados.0.smithi115.stdout:2041: finishing write tid 1 to smithi11541303-11 2022-04-22T17:32:44.884 INFO:tasks.rados.rados.0.smithi115.stdout:2041: finishing write tid 2 to smithi11541303-11 2022-04-22T17:32:44.885 INFO:tasks.rados.rados.0.smithi115.stdout:2041: finishing write tid 3 to smithi11541303-11 2022-04-22T17:32:44.886 INFO:tasks.rados.rados.0.smithi115.stdout:2042: done (1 left) 2022-04-22T17:32:44.886 INFO:tasks.rados.rados.0.smithi115.stdout:2043: snap_create 2022-04-22T17:32:44.890 INFO:tasks.rados.rados.0.smithi115.stdout:2041: finishing write tid 4 to smithi11541303-11 2022-04-22T17:32:44.890 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1009 (ObjNum 628 snap 206 seq_num 628) dirty exists 2022-04-22T17:32:44.890 INFO:tasks.rados.rados.0.smithi115.stdout:2041: left oid 11 (ObjNum 628 snap 206 seq_num 628) 2022-04-22T17:32:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: cluster 2022-04-22T17:32:43.235849+0000 mgr.y (mgr.34107) 231 : cluster [DBG] pgmap v387: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 5.6 MiB/s wr, 16 op/s; 13 MiB/s, 2 objects/s recovering 2022-04-22T17:32:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: cluster 2022-04-22T17:32:43.953087+0000 mon.a (mon.0) 754 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/528 objects degraded (1.515%), 1 pg degraded) 2022-04-22T17:32:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: cluster 2022-04-22T17:32:43.953175+0000 mon.a (mon.0) 755 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:45.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: cluster 2022-04-22T17:32:43.965314+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2022-04-22T17:32:45.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: audit 2022-04-22T17:32:44.258565+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:45.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:44 smithi121 bash[36902]: cluster 2022-04-22T17:32:44.833730+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2022-04-22T17:32:45.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: cluster 2022-04-22T17:32:43.235849+0000 mgr.y (mgr.34107) 231 : cluster [DBG] pgmap v387: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 5.6 MiB/s wr, 16 op/s; 13 MiB/s, 2 objects/s recovering 2022-04-22T17:32:45.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: cluster 2022-04-22T17:32:43.953087+0000 mon.a (mon.0) 754 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/528 objects degraded (1.515%), 1 pg degraded) 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: cluster 2022-04-22T17:32:43.953175+0000 mon.a (mon.0) 755 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: cluster 2022-04-22T17:32:43.965314+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: audit 2022-04-22T17:32:44.258565+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[45495]: cluster 2022-04-22T17:32:44.833730+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: cluster 2022-04-22T17:32:43.235849+0000 mgr.y (mgr.34107) 231 : cluster [DBG] pgmap v387: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 5.6 MiB/s wr, 16 op/s; 13 MiB/s, 2 objects/s recovering 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: cluster 2022-04-22T17:32:43.953087+0000 mon.a (mon.0) 754 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/528 objects degraded (1.515%), 1 pg degraded) 2022-04-22T17:32:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: cluster 2022-04-22T17:32:43.953175+0000 mon.a (mon.0) 755 : cluster [INF] Cluster is now healthy 2022-04-22T17:32:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: cluster 2022-04-22T17:32:43.965314+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2022-04-22T17:32:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: audit 2022-04-22T17:32:44.258565+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:44 smithi115 bash[46934]: cluster 2022-04-22T17:32:44.833730+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2022-04-22T17:32:45.861 INFO:tasks.rados.rados.0.smithi115.stdout:2041: done (1 left) 2022-04-22T17:32:45.862 INFO:tasks.rados.rados.0.smithi115.stdout:2043: done (0 left) 2022-04-22T17:32:45.862 INFO:tasks.rados.rados.0.smithi115.stdout:2044: read oid 1 snap -1 2022-04-22T17:32:45.862 INFO:tasks.rados.rados.0.smithi115.stdout:2044: expect (ObjNum 526 snap 173 seq_num 526) 2022-04-22T17:32:45.862 INFO:tasks.rados.rados.0.smithi115.stdout:2045: setattr oid 42 current snap is 207 2022-04-22T17:32:45.865 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 207 2022-04-22T17:32:45.865 INFO:tasks.rados.rados.0.smithi115.stdout:2046: seq_num 630 ranges {475136=557056,1032192=401408} 2022-04-22T17:32:45.874 INFO:tasks.rados.rados.0.smithi115.stdout:2046: writing smithi11541303-10 from 475136 to 1032192 tid 1 2022-04-22T17:32:45.876 INFO:tasks.rados.rados.0.smithi115.stdout:2046: writing smithi11541303-10 from 1032192 to 1433600 tid 2 2022-04-22T17:32:45.877 INFO:tasks.rados.rados.0.smithi115.stdout:2047: snap_create 2022-04-22T17:32:45.877 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1298 (ObjNum 1835758420 snap 1567774308 seq_num 1785161825) dirty exists 2022-04-22T17:32:45.877 INFO:tasks.rados.rados.0.smithi115.stdout:2044: expect (ObjNum 526 snap 173 seq_num 526) 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:2046: finishing write tid 1 to smithi11541303-10 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:2046: finishing write tid 2 to smithi11541303-10 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:2046: finishing write tid 3 to smithi11541303-10 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:2046: finishing write tid 4 to smithi11541303-10 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1118 (ObjNum 630 snap 207 seq_num 630) dirty exists 2022-04-22T17:32:45.929 INFO:tasks.rados.rados.0.smithi115.stdout:2046: left oid 10 (ObjNum 630 snap 207 seq_num 630) 2022-04-22T17:32:46.873 INFO:tasks.rados.rados.0.smithi115.stdout:2044: done (3 left) 2022-04-22T17:32:46.873 INFO:tasks.rados.rados.0.smithi115.stdout:2045: done (2 left) 2022-04-22T17:32:46.873 INFO:tasks.rados.rados.0.smithi115.stdout:2046: done (1 left) 2022-04-22T17:32:46.873 INFO:tasks.rados.rados.0.smithi115.stdout:2047: done (0 left) 2022-04-22T17:32:46.873 INFO:tasks.rados.rados.0.smithi115.stdout:2048: delete oid 9 current snap is 208 2022-04-22T17:32:46.880 INFO:tasks.rados.rados.0.smithi115.stdout:2048: done (0 left) 2022-04-22T17:32:46.880 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 208 2022-04-22T17:32:46.880 INFO:tasks.rados.rados.0.smithi115.stdout:2049: seq_num 632 ranges {0=589824,589824=401408} 2022-04-22T17:32:46.885 INFO:tasks.rados.rados.0.smithi115.stdout:2049: writing smithi11541303-35 from 0 to 589824 tid 1 2022-04-22T17:32:46.888 INFO:tasks.rados.rados.0.smithi115.stdout:2049: writing smithi11541303-35 from 589824 to 991232 tid 2 2022-04-22T17:32:46.888 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 208 2022-04-22T17:32:46.888 INFO:tasks.rados.rados.0.smithi115.stdout:2050: seq_num 633 ranges {9551872=606208,10158080=401408} 2022-04-22T17:32:46.948 INFO:tasks.rados.rados.0.smithi115.stdout:2050: writing smithi11541303-6 from 9551872 to 10158080 tid 1 2022-04-22T17:32:46.951 INFO:tasks.rados.rados.0.smithi115.stdout:2050: writing smithi11541303-6 from 10158080 to 10559488 tid 2 2022-04-22T17:32:46.951 INFO:tasks.rados.rados.0.smithi115.stdout:2051: delete oid 37 current snap is 208 2022-04-22T17:32:46.951 INFO:tasks.rados.rados.0.smithi115.stdout:2049: finishing write tid 1 to smithi11541303-35 2022-04-22T17:32:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:2049: finishing write tid 2 to smithi11541303-35 2022-04-22T17:32:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:2049: finishing write tid 3 to smithi11541303-35 2022-04-22T17:32:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:2049: finishing write tid 4 to smithi11541303-35 2022-04-22T17:32:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1093 (ObjNum 632 snap 208 seq_num 632) dirty exists 2022-04-22T17:32:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:2049: left oid 35 (ObjNum 632 snap 208 seq_num 632) 2022-04-22T17:32:46.955 INFO:tasks.rados.rados.0.smithi115.stdout:2050: finishing write tid 1 to smithi11541303-6 2022-04-22T17:32:46.956 INFO:tasks.rados.rados.0.smithi115.stdout:2049: done (2 left) 2022-04-22T17:32:46.956 INFO:tasks.rados.rados.0.smithi115.stdout:2051: done (1 left) 2022-04-22T17:32:46.956 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 208 2022-04-22T17:32:46.956 INFO:tasks.rados.rados.0.smithi115.stdout:2052: seq_num 635 ranges {4497408=638976,5136384=401408} 2022-04-22T17:32:46.988 INFO:tasks.rados.rados.0.smithi115.stdout:2052: writing smithi11541303-40 from 4497408 to 5136384 tid 1 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2052: writing smithi11541303-40 from 5136384 to 5537792 tid 2 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2053: copy_from oid 31 from oid 21 current snap is 208 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2050: finishing write tid 2 to smithi11541303-6 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 208 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2050: finishing write tid 3 to smithi11541303-6 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2050: finishing write tid 4 to smithi11541303-6 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1071 (ObjNum 633 snap 208 seq_num 633) dirty exists 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2050: left oid 6 (ObjNum 633 snap 208 seq_num 633) 2022-04-22T17:32:46.991 INFO:tasks.rados.rados.0.smithi115.stdout:2054: seq_num 636 ranges {4423680=655360,5079040=401408} 2022-04-22T17:32:47.021 INFO:tasks.rados.rados.0.smithi115.stdout:2054: writing smithi11541303-14 from 4423680 to 5079040 tid 1 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2054: writing smithi11541303-14 from 5079040 to 5480448 tid 2 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2050: done (3 left) 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2055: copy_from oid 19 from oid 47 current snap is 208 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2052: finishing write tid 1 to smithi11541303-40 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2056: delete oid 12 current snap is 208 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2052: finishing write tid 2 to smithi11541303-40 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2052: finishing write tid 3 to smithi11541303-40 2022-04-22T17:32:47.025 INFO:tasks.rados.rados.0.smithi115.stdout:2052: finishing write tid 4 to smithi11541303-40 2022-04-22T17:32:47.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1121 (ObjNum 635 snap 208 seq_num 635) dirty exists 2022-04-22T17:32:47.026 INFO:tasks.rados.rados.0.smithi115.stdout:2052: left oid 40 (ObjNum 635 snap 208 seq_num 635) 2022-04-22T17:32:47.032 INFO:tasks.rados.rados.0.smithi115.stdout:2054: finishing write tid 1 to smithi11541303-14 2022-04-22T17:32:47.032 INFO:tasks.rados.rados.0.smithi115.stdout:2052: done (4 left) 2022-04-22T17:32:47.033 INFO:tasks.rados.rados.0.smithi115.stdout:2056: done (3 left) 2022-04-22T17:32:47.033 INFO:tasks.rados.rados.0.smithi115.stdout:2057: snap_remove snap 194 2022-04-22T17:32:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:46 smithi121 bash[36902]: cluster 2022-04-22T17:32:45.236704+0000 mgr.y (mgr.34107) 232 : cluster [DBG] pgmap v390: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.6 MiB/s wr, 27 op/s; 8.5 MiB/s, 1 objects/s recovering 2022-04-22T17:32:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:46 smithi121 bash[36902]: cluster 2022-04-22T17:32:45.846150+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2022-04-22T17:32:47.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:46 smithi115 bash[45495]: cluster 2022-04-22T17:32:45.236704+0000 mgr.y (mgr.34107) 232 : cluster [DBG] pgmap v390: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.6 MiB/s wr, 27 op/s; 8.5 MiB/s, 1 objects/s recovering 2022-04-22T17:32:47.389 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:46 smithi115 bash[45495]: cluster 2022-04-22T17:32:45.846150+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2022-04-22T17:32:47.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:46 smithi115 bash[46934]: cluster 2022-04-22T17:32:45.236704+0000 mgr.y (mgr.34107) 232 : cluster [DBG] pgmap v390: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.6 MiB/s wr, 27 op/s; 8.5 MiB/s, 1 objects/s recovering 2022-04-22T17:32:47.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:46 smithi115 bash[46934]: cluster 2022-04-22T17:32:45.846150+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2022-04-22T17:32:47.882 INFO:tasks.rados.rados.0.smithi115.stdout:2057: done (3 left) 2022-04-22T17:32:47.882 INFO:tasks.rados.rados.0.smithi115.stdout:2058: snap_remove snap 200 2022-04-22T17:32:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:47 smithi121 bash[36902]: cluster 2022-04-22T17:32:46.855716+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2022-04-22T17:32:48.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:47 smithi115 bash[45495]: cluster 2022-04-22T17:32:46.855716+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2022-04-22T17:32:48.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:47 smithi115 bash[46934]: cluster 2022-04-22T17:32:46.855716+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2022-04-22T17:32:48.889 INFO:tasks.rados.rados.0.smithi115.stdout:2058: done (3 left) 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2059: copy_from oid 42 from oid 5 current snap is 208 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2054: finishing write tid 2 to smithi11541303-14 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2054: finishing write tid 3 to smithi11541303-14 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2054: finishing write tid 4 to smithi11541303-14 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1092 (ObjNum 636 snap 208 seq_num 636) dirty exists 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2054: left oid 14 (ObjNum 636 snap 208 seq_num 636) 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2053: finishing copy_from to smithi11541303-31 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1085 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:48.890 INFO:tasks.rados.rados.0.smithi115.stdout:2053: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2055: finishing copy_from to smithi11541303-19 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1112 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2055: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2053: done (3 left) 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2054: done (2 left) 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2055: done (1 left) 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2060: read oid 21 snap -1 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2060: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:48.891 INFO:tasks.rados.rados.0.smithi115.stdout:2061: snap_remove snap 193 2022-04-22T17:32:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:48 smithi121 bash[36902]: cluster 2022-04-22T17:32:47.237581+0000 mgr.y (mgr.34107) 233 : cluster [DBG] pgmap v393: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2022-04-22T17:32:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:48 smithi121 bash[36902]: cluster 2022-04-22T17:32:47.866941+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2022-04-22T17:32:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:48 smithi121 bash[36902]: audit 2022-04-22T17:32:48.354978+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[45495]: cluster 2022-04-22T17:32:47.237581+0000 mgr.y (mgr.34107) 233 : cluster [DBG] pgmap v393: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2022-04-22T17:32:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[45495]: cluster 2022-04-22T17:32:47.866941+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2022-04-22T17:32:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[45495]: audit 2022-04-22T17:32:48.354978+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:49.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[46934]: cluster 2022-04-22T17:32:47.237581+0000 mgr.y (mgr.34107) 233 : cluster [DBG] pgmap v393: 121 pgs: 121 active+clean; 305 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2022-04-22T17:32:49.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[46934]: cluster 2022-04-22T17:32:47.866941+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2022-04-22T17:32:49.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:48 smithi115 bash[46934]: audit 2022-04-22T17:32:48.354978+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:49.850 INFO:tasks.rados.rados.0.smithi115.stdout:2061: done (2 left) 2022-04-22T17:32:49.850 INFO:tasks.rados.rados.0.smithi115.stdout:2062: rmattr oid 19 current snap is 208 2022-04-22T17:32:49.851 INFO:tasks.rados.rados.0.smithi115.stdout:2059: finishing copy_from to smithi11541303-42 2022-04-22T17:32:49.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1191 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:32:49.851 INFO:tasks.rados.rados.0.smithi115.stdout:2059: finishing copy_from racing read to smithi11541303-42 2022-04-22T17:32:49.851 INFO:tasks.rados.rados.0.smithi115.stdout:2060: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:49 smithi121 bash[36902]: cluster 2022-04-22T17:32:48.874001+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2022-04-22T17:32:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:49 smithi121 bash[36902]: audit 2022-04-22T17:32:49.261344+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:49 smithi121 bash[36902]: cluster 2022-04-22T17:32:49.834772+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2022-04-22T17:32:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:32:50.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[46934]: cluster 2022-04-22T17:32:48.874001+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2022-04-22T17:32:50.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[46934]: audit 2022-04-22T17:32:49.261344+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:50.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[46934]: cluster 2022-04-22T17:32:49.834772+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2022-04-22T17:32:50.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[45495]: cluster 2022-04-22T17:32:48.874001+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2022-04-22T17:32:50.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[45495]: audit 2022-04-22T17:32:49.261344+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:50.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:49 smithi115 bash[45495]: cluster 2022-04-22T17:32:49.834772+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2022-04-22T17:32:50.561 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1113 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:32:50.561 INFO:tasks.rados.rados.0.smithi115.stdout:2059: done (2 left) 2022-04-22T17:32:50.561 INFO:tasks.rados.rados.0.smithi115.stdout:2060: done (1 left) 2022-04-22T17:32:50.561 INFO:tasks.rados.rados.0.smithi115.stdout:2062: done (0 left) 2022-04-22T17:32:50.561 INFO:tasks.rados.rados.0.smithi115.stdout:2063: snap_create 2022-04-22T17:32:50.617 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (9m) 60s ago 9m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (9m) 19s ago 9m 23.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (10m) 60s ago 10m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (10m) 19s ago 10m 19.1M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5m) 19s ago 14m 404M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5m) 60s ago 17m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4m) 60s ago 17m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:32:50.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3m) 19s ago 15m 48.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4m) 60s ago 15m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (9m) 60s ago 9m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (9m) 19s ago 9m 9415k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2m) 60s ago 14m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (116s) 60s ago 14m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (89s) 60s ago 13m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (62s) 60s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (21s) 19s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (12m) 19s ago 12m 304M 2526M 16.2.5 46cf6318c64e b30c397f18c1 2022-04-22T17:32:50.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (11m) 19s ago 11m 200M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:32:50.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (11m) 19s ago 11m 233M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:32:50.620 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5m) 19s ago 10m 39.8M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:32:50.899 INFO:tasks.rados.rados.0.smithi115.stdout:2063: done (0 left) 2022-04-22T17:32:50.938 INFO:tasks.rados.rados.0.smithi115.stdout:2064: rollback oid 36 current snap is 209 2022-04-22T17:32:50.938 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 178 2022-04-22T17:32:50.938 INFO:tasks.rados.rados.0.smithi115.stdout:2065: delete oid 18 current snap is 209 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2064: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2065: done (1 left) 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2066: read oid 49 snap -1 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2066: expect (ObjNum 624 snap 204 seq_num 624) 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2067: copy_from oid 32 from oid 1 current snap is 209 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2068: rollback oid 43 current snap is 209 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 173 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:2069: rollback oid 41 current snap is 209 2022-04-22T17:32:50.941 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 189 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2070: snap_create 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2064: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2064: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1194 (ObjNum 521 snap 172 seq_num 521) dirty exists 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2069: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2069: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2069: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1124 (ObjNum 580 snap 188 seq_num 580) dirty exists 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2068: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:32:50.942 INFO:tasks.rados.rados.0.smithi115.stdout:2068: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:2068: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1048 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:2067: finishing copy_from to smithi11541303-32 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 890 (ObjNum 526 snap 173 seq_num 526) dirty exists 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:2067: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:32:50.943 INFO:tasks.rados.rados.0.smithi115.stdout:2066: expect (ObjNum 624 snap 204 seq_num 624) 2022-04-22T17:32:51.031 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:32:51.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:32:51.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3, 2022-04-22T17:32:51.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 5 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 5, 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 10 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:32:51.033 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:32:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:50 smithi121 bash[36902]: cluster 2022-04-22T17:32:49.238609+0000 mgr.y (mgr.34107) 234 : cluster [DBG] pgmap v396: 121 pgs: 121 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:32:51.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:50 smithi115 bash[45495]: cluster 2022-04-22T17:32:49.238609+0000 mgr.y (mgr.34107) 234 : cluster [DBG] pgmap v396: 121 pgs: 121 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:32:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:50 smithi115 bash[46934]: cluster 2022-04-22T17:32:49.238609+0000 mgr.y (mgr.34107) 234 : cluster [DBG] pgmap v396: 121 pgs: 121 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 12 op/s 2022-04-22T17:32:51.905 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:51.626Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:51.906 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:51.626Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:51.906 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:51.626Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:51.922 INFO:tasks.rados.rados.0.smithi115.stdout:2064: done (5 left) 2022-04-22T17:32:51.922 INFO:tasks.rados.rados.0.smithi115.stdout:2066: done (4 left) 2022-04-22T17:32:51.922 INFO:tasks.rados.rados.0.smithi115.stdout:2067: done (3 left) 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2068: done (2 left) 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2069: done (1 left) 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2070: done (0 left) 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2071: read oid 42 snap 198 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2071: expect deleted 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 210 2022-04-22T17:32:51.923 INFO:tasks.rados.rados.0.smithi115.stdout:2072: seq_num 639 ranges {0=704512,704512=401408} 2022-04-22T17:32:51.929 INFO:tasks.rados.rados.0.smithi115.stdout:2072: writing smithi11541303-12 from 0 to 704512 tid 1 2022-04-22T17:32:51.933 INFO:tasks.rados.rados.0.smithi115.stdout:2072: writing smithi11541303-12 from 704512 to 1105920 tid 2 2022-04-22T17:32:51.933 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 210 2022-04-22T17:32:51.933 INFO:tasks.rados.rados.0.smithi115.stdout:2073: seq_num 640 ranges {0=729088,729088=401408} 2022-04-22T17:32:51.937 INFO:tasks.rados.rados.0.smithi115.stdout:2073: writing smithi11541303-22 from 0 to 729088 tid 1 2022-04-22T17:32:51.940 INFO:tasks.rados.rados.0.smithi115.stdout:2073: writing smithi11541303-22 from 729088 to 1130496 tid 2 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:2074: rollback oid 35 current snap is 210 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 189 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:2071: done (3 left) 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:2075: rollback oid 21 current snap is 210 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 207 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:2076: read oid 24 snap -1 2022-04-22T17:32:51.942 INFO:tasks.rados.rados.0.smithi115.stdout:2076: expect (ObjNum 599 snap 195 seq_num 599) 2022-04-22T17:32:51.943 INFO:tasks.rados.rados.0.smithi115.stdout:2077: rmattr oid 48 current snap is 210 2022-04-22T17:32:51.943 INFO:tasks.rados.rados.0.smithi115.stdout:2077: done (5 left) 2022-04-22T17:32:51.943 INFO:tasks.rados.rados.0.smithi115.stdout:2078: setattr oid 39 current snap is 210 2022-04-22T17:32:51.943 INFO:tasks.rados.rados.0.smithi115.stdout:2079: delete oid 1 current snap is 210 2022-04-22T17:32:51.945 INFO:tasks.rados.rados.0.smithi115.stdout:2072: finishing write tid 1 to smithi11541303-12 2022-04-22T17:32:51.945 INFO:tasks.rados.rados.0.smithi115.stdout:2072: finishing write tid 2 to smithi11541303-12 2022-04-22T17:32:51.945 INFO:tasks.rados.rados.0.smithi115.stdout:2072: finishing write tid 3 to smithi11541303-12 2022-04-22T17:32:51.949 INFO:tasks.rados.rados.0.smithi115.stdout:2072: finishing write tid 4 to smithi11541303-12 2022-04-22T17:32:51.950 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1116 (ObjNum 639 snap 210 seq_num 639) dirty exists 2022-04-22T17:32:51.950 INFO:tasks.rados.rados.0.smithi115.stdout:2072: left oid 12 (ObjNum 639 snap 210 seq_num 639) 2022-04-22T17:32:51.951 INFO:tasks.rados.rados.0.smithi115.stdout:2074: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:32:51.951 INFO:tasks.rados.rados.0.smithi115.stdout:2074: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:32:51.951 INFO:tasks.rados.rados.0.smithi115.stdout:2074: finishing rollback tid 2 to smithi11541303-35 2022-04-22T17:32:51.952 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1097 (ObjNum 571 snap 186 seq_num 571) dirty exists 2022-04-22T17:32:51.952 INFO:tasks.rados.rados.0.smithi115.stdout:2072: done (6 left) 2022-04-22T17:32:51.952 INFO:tasks.rados.rados.0.smithi115.stdout:2074: done (5 left) 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2079: done (4 left) 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2080: copy_from oid 45 from oid 5 current snap is 210 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2081: delete oid 1 current snap is 210 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2073: finishing write tid 1 to smithi11541303-22 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2073: finishing write tid 2 to smithi11541303-22 2022-04-22T17:32:51.953 INFO:tasks.rados.rados.0.smithi115.stdout:2073: finishing write tid 3 to smithi11541303-22 2022-04-22T17:32:51.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1098 (ObjNum 542 snap 178 seq_num 542) dirty exists 2022-04-22T17:32:51.954 INFO:tasks.rados.rados.0.smithi115.stdout:2078: done (5 left) 2022-04-22T17:32:51.955 INFO:tasks.rados.rados.0.smithi115.stdout:2081: done (4 left) 2022-04-22T17:32:51.955 INFO:tasks.rados.rados.0.smithi115.stdout:2082: read oid 28 snap -1 2022-04-22T17:32:51.955 INFO:tasks.rados.rados.0.smithi115.stdout:2082: expect (ObjNum 622 snap 202 seq_num 622) 2022-04-22T17:32:51.955 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 210 2022-04-22T17:32:51.955 INFO:tasks.rados.rados.0.smithi115.stdout:2083: seq_num 643 ranges {2850816=778240,3629056=401408} 2022-04-22T17:32:51.975 INFO:tasks.rados.rados.0.smithi115.stdout:2083: writing smithi11541303-13 from 2850816 to 3629056 tid 1 2022-04-22T17:32:51.978 INFO:tasks.rados.rados.0.smithi115.stdout:2083: writing smithi11541303-13 from 3629056 to 4030464 tid 2 2022-04-22T17:32:51.978 INFO:tasks.rados.rados.0.smithi115.stdout:2084: read oid 50 snap -1 2022-04-22T17:32:51.978 INFO:tasks.rados.rados.0.smithi115.stdout:2084: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:51.978 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 210 2022-04-22T17:32:51.979 INFO:tasks.rados.rados.0.smithi115.stdout:2085: seq_num 644 ranges {0=794624,794624=401408} 2022-04-22T17:32:51.982 INFO:tasks.rados.rados.0.smithi115.stdout:2085: writing smithi11541303-9 from 0 to 794624 tid 1 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2085: writing smithi11541303-9 from 794624 to 1196032 tid 2 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2086: read oid 14 snap -1 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2086: expect (ObjNum 636 snap 208 seq_num 636) 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2087: delete oid 27 current snap is 210 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2075: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:32:51.986 INFO:tasks.rados.rados.0.smithi115.stdout:2075: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2075: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1092 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2073: finishing write tid 4 to smithi11541303-22 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1089 (ObjNum 640 snap 210 seq_num 640) dirty exists 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2073: left oid 22 (ObjNum 640 snap 210 seq_num 640) 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2080: finishing copy_from to smithi11541303-45 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1191 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2080: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:32:51.987 INFO:tasks.rados.rados.0.smithi115.stdout:2082: expect (ObjNum 622 snap 202 seq_num 622) 2022-04-22T17:32:52.025 INFO:tasks.rados.rados.0.smithi115.stdout:2076: expect (ObjNum 599 snap 195 seq_num 599) 2022-04-22T17:32:52.138 INFO:tasks.rados.rados.0.smithi115.stdout:2083: finishing write tid 1 to smithi11541303-13 2022-04-22T17:32:52.138 INFO:tasks.rados.rados.0.smithi115.stdout:2073: done (9 left) 2022-04-22T17:32:52.138 INFO:tasks.rados.rados.0.smithi115.stdout:2075: done (8 left) 2022-04-22T17:32:52.138 INFO:tasks.rados.rados.0.smithi115.stdout:2076: done (7 left) 2022-04-22T17:32:52.138 INFO:tasks.rados.rados.0.smithi115.stdout:2080: done (6 left) 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2082: done (5 left) 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2087: done (4 left) 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 210 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2085: finishing write tid 1 to smithi11541303-9 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2085: finishing write tid 2 to smithi11541303-9 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2085: finishing write tid 3 to smithi11541303-9 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2083: finishing write tid 2 to smithi11541303-13 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2083: finishing write tid 3 to smithi11541303-13 2022-04-22T17:32:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:2084: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:51 smithi121 bash[36902]: audit 2022-04-22T17:32:50.255719+0000 mgr.y (mgr.34107) 235 : audit [DBG] from='client.44254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:51 smithi121 bash[36902]: audit 2022-04-22T17:32:50.589531+0000 mgr.y (mgr.34107) 236 : audit [DBG] from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:51 smithi121 bash[36902]: cluster 2022-04-22T17:32:50.883245+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2022-04-22T17:32:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:51 smithi121 bash[36902]: audit 2022-04-22T17:32:51.016089+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/1624854832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:52.267 INFO:tasks.rados.rados.0.smithi115.stdout:2085: finishing write tid 4 to smithi11541303-9 2022-04-22T17:32:52.267 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1329 (ObjNum 644 snap 210 seq_num 644) dirty exists 2022-04-22T17:32:52.267 INFO:tasks.rados.rados.0.smithi115.stdout:2085: left oid 9 (ObjNum 644 snap 210 seq_num 644) 2022-04-22T17:32:52.268 INFO:tasks.rados.rados.0.smithi115.stdout:2083: finishing write tid 4 to smithi11541303-13 2022-04-22T17:32:52.268 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1052 (ObjNum 643 snap 210 seq_num 643) dirty exists 2022-04-22T17:32:52.268 INFO:tasks.rados.rados.0.smithi115.stdout:2083: left oid 13 (ObjNum 643 snap 210 seq_num 643) 2022-04-22T17:32:52.268 INFO:tasks.rados.rados.0.smithi115.stdout:2086: expect (ObjNum 636 snap 208 seq_num 636) 2022-04-22T17:32:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[46934]: audit 2022-04-22T17:32:50.255719+0000 mgr.y (mgr.34107) 235 : audit [DBG] from='client.44254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[46934]: audit 2022-04-22T17:32:50.589531+0000 mgr.y (mgr.34107) 236 : audit [DBG] from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[46934]: cluster 2022-04-22T17:32:50.883245+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2022-04-22T17:32:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[46934]: audit 2022-04-22T17:32:51.016089+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/1624854832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[45495]: audit 2022-04-22T17:32:50.255719+0000 mgr.y (mgr.34107) 235 : audit [DBG] from='client.44254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[45495]: audit 2022-04-22T17:32:50.589531+0000 mgr.y (mgr.34107) 236 : audit [DBG] from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:32:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[45495]: cluster 2022-04-22T17:32:50.883245+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2022-04-22T17:32:52.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:51 smithi115 bash[45495]: audit 2022-04-22T17:32:51.016089+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.115:0/1624854832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:32:52.585 INFO:tasks.rados.rados.0.smithi115.stdout:2088: seq_num 646 ranges {1622016=425984,2048000=655360,2703360=147456} 2022-04-22T17:32:52.603 INFO:tasks.rados.rados.0.smithi115.stdout:2088: writing smithi11541303-32 from 1622016 to 2048000 tid 1 2022-04-22T17:32:52.607 INFO:tasks.rados.rados.0.smithi115.stdout:2088: writing smithi11541303-32 from 2048000 to 2703360 tid 2 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2088: writing smithi11541303-32 from 2703360 to 2850816 tid 3 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2083: done (4 left) 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2084: done (3 left) 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2085: done (2 left) 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2086: done (1 left) 2022-04-22T17:32:52.608 INFO:tasks.rados.rados.0.smithi115.stdout:2089: snap_create 2022-04-22T17:32:52.616 INFO:tasks.rados.rados.0.smithi115.stdout:2088: finishing write tid 1 to smithi11541303-32 2022-04-22T17:32:52.616 INFO:tasks.rados.rados.0.smithi115.stdout:2088: finishing write tid 2 to smithi11541303-32 2022-04-22T17:32:52.617 INFO:tasks.rados.rados.0.smithi115.stdout:2088: finishing write tid 3 to smithi11541303-32 2022-04-22T17:32:52.620 INFO:tasks.rados.rados.0.smithi115.stdout:2088: finishing write tid 4 to smithi11541303-32 2022-04-22T17:32:52.622 INFO:tasks.rados.rados.0.smithi115.stdout:2088: finishing write tid 5 to smithi11541303-32 2022-04-22T17:32:52.622 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 996 (ObjNum 646 snap 210 seq_num 646) dirty exists 2022-04-22T17:32:52.622 INFO:tasks.rados.rados.0.smithi115.stdout:2088: left oid 32 (ObjNum 646 snap 210 seq_num 646) 2022-04-22T17:32:52.928 INFO:tasks.rados.rados.0.smithi115.stdout:2088: done (1 left) 2022-04-22T17:32:52.928 INFO:tasks.rados.rados.0.smithi115.stdout:2089: done (0 left) 2022-04-22T17:32:52.928 INFO:tasks.rados.rados.0.smithi115.stdout:2090: snap_remove snap 188 2022-04-22T17:32:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:52 smithi121 bash[36902]: cluster 2022-04-22T17:32:51.239343+0000 mgr.y (mgr.34107) 237 : cluster [DBG] pgmap v399: 121 pgs: 121 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.9 MiB/s wr, 9 op/s 2022-04-22T17:32:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:52 smithi121 bash[36902]: cluster 2022-04-22T17:32:51.907112+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2022-04-22T17:32:53.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:52 smithi115 bash[45495]: cluster 2022-04-22T17:32:51.239343+0000 mgr.y (mgr.34107) 237 : cluster [DBG] pgmap v399: 121 pgs: 121 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.9 MiB/s wr, 9 op/s 2022-04-22T17:32:53.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:52 smithi115 bash[45495]: cluster 2022-04-22T17:32:51.907112+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2022-04-22T17:32:53.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:52 smithi115 bash[46934]: cluster 2022-04-22T17:32:51.239343+0000 mgr.y (mgr.34107) 237 : cluster [DBG] pgmap v399: 121 pgs: 121 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.9 MiB/s wr, 9 op/s 2022-04-22T17:32:53.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:52 smithi115 bash[46934]: cluster 2022-04-22T17:32:51.907112+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2022-04-22T17:32:53.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:53.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:53.521Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:32:53.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:32:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:32:53.521Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:32:53.942 INFO:tasks.rados.rados.0.smithi115.stdout:2090: done (0 left) 2022-04-22T17:32:53.942 INFO:tasks.rados.rados.0.smithi115.stdout:2091: rollback oid 34 current snap is 211 2022-04-22T17:32:53.942 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 34 to 210 2022-04-22T17:32:53.943 INFO:tasks.rados.rados.0.smithi115.stdout:2092: snap_remove snap 201 2022-04-22T17:32:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:53 smithi121 bash[36902]: cluster 2022-04-22T17:32:52.909544+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2022-04-22T17:32:54.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:53 smithi115 bash[45495]: cluster 2022-04-22T17:32:52.909544+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2022-04-22T17:32:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:53 smithi115 bash[46934]: cluster 2022-04-22T17:32:52.909544+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:2092: done (1 left) 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:2093: copy_from oid 27 from oid 37 current snap is 211 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:2091: finishing rollback tid 0 to smithi11541303-34 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:2091: finishing rollback tid 1 to smithi11541303-34 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:2091: finishing rollback tid 2 to smithi11541303-34 2022-04-22T17:32:54.853 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1112 (ObjNum 605 snap 198 seq_num 605) dirty exists 2022-04-22T17:32:54.854 INFO:tasks.rados.rados.0.smithi115.stdout:2091: done (1 left) 2022-04-22T17:32:54.854 INFO:tasks.rados.rados.0.smithi115.stdout:2094: snap_remove snap 206 2022-04-22T17:32:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:54 smithi121 bash[36902]: cluster 2022-04-22T17:32:53.240231+0000 mgr.y (mgr.34107) 238 : cluster [DBG] pgmap v402: 121 pgs: 121 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:32:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:54 smithi121 bash[36902]: cluster 2022-04-22T17:32:53.927585+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2022-04-22T17:32:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:54 smithi121 bash[36902]: audit 2022-04-22T17:32:54.264462+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:54 smithi121 bash[36902]: cluster 2022-04-22T17:32:54.837230+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2022-04-22T17:32:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[45495]: cluster 2022-04-22T17:32:53.240231+0000 mgr.y (mgr.34107) 238 : cluster [DBG] pgmap v402: 121 pgs: 121 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:32:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[45495]: cluster 2022-04-22T17:32:53.927585+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[45495]: audit 2022-04-22T17:32:54.264462+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[45495]: cluster 2022-04-22T17:32:54.837230+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[46934]: cluster 2022-04-22T17:32:53.240231+0000 mgr.y (mgr.34107) 238 : cluster [DBG] pgmap v402: 121 pgs: 121 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[46934]: cluster 2022-04-22T17:32:53.927585+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[46934]: audit 2022-04-22T17:32:54.264462+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:32:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:54 smithi115 bash[46934]: cluster 2022-04-22T17:32:54.837230+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2022-04-22T17:32:55.856 INFO:tasks.rados.rados.0.smithi115.stdout:2094: done (1 left) 2022-04-22T17:32:55.856 INFO:tasks.rados.rados.0.smithi115.stdout:2095: delete oid 11 current snap is 211 2022-04-22T17:32:55.856 INFO:tasks.rados.rados.0.smithi115.stdout:2093: finishing copy_from to smithi11541303-27 2022-04-22T17:32:55.856 INFO:tasks.rados.rados.0.smithi115.stdout:2093: got expected ENOENT (src dne) 2022-04-22T17:32:55.856 INFO:tasks.rados.rados.0.smithi115.stdout:2093: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:32:55.864 INFO:tasks.rados.rados.0.smithi115.stdout:2093: done (1 left) 2022-04-22T17:32:55.864 INFO:tasks.rados.rados.0.smithi115.stdout:2095: done (0 left) 2022-04-22T17:32:55.864 INFO:tasks.rados.rados.0.smithi115.stdout:2096: read oid 47 snap -1 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2096: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2097: read oid 10 snap 179 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2097: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2098: read oid 38 snap 210 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2098: expect deleted 2022-04-22T17:32:55.865 INFO:tasks.rados.rados.0.smithi115.stdout:2099: snap_remove snap 209 2022-04-22T17:32:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:55 smithi121 bash[36902]: audit 2022-04-22T17:32:55.692726+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:56.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:55 smithi121 bash[36902]: cluster 2022-04-22T17:32:55.840869+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2022-04-22T17:32:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:55 smithi115 bash[45495]: audit 2022-04-22T17:32:55.692726+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:56.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:55 smithi115 bash[45495]: cluster 2022-04-22T17:32:55.840869+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2022-04-22T17:32:56.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:55 smithi115 bash[46934]: audit 2022-04-22T17:32:55.692726+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:56.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:55 smithi115 bash[46934]: cluster 2022-04-22T17:32:55.840869+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2022-04-22T17:32:56.858 INFO:tasks.rados.rados.0.smithi115.stdout:2099: done (3 left) 2022-04-22T17:32:56.881 INFO:tasks.rados.rados.0.smithi115.stdout:2100: read oid 22 snap -1 2022-04-22T17:32:56.881 INFO:tasks.rados.rados.0.smithi115.stdout:2100: expect (ObjNum 640 snap 210 seq_num 640) 2022-04-22T17:32:56.881 INFO:tasks.rados.rados.0.smithi115.stdout:2101: snap_create 2022-04-22T17:32:56.881 INFO:tasks.rados.rados.0.smithi115.stdout:2096: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:32:56.995 INFO:tasks.rados.rados.0.smithi115.stdout:2097: expect (ObjNum 458 snap 157 seq_num 458) 2022-04-22T17:32:57.014 INFO:tasks.rados.rados.0.smithi115.stdout:2100: expect (ObjNum 640 snap 210 seq_num 640) 2022-04-22T17:32:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:56 smithi121 bash[36902]: cluster 2022-04-22T17:32:55.241136+0000 mgr.y (mgr.34107) 239 : cluster [DBG] pgmap v405: 121 pgs: 121 active+clean; 329 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2022-04-22T17:32:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:56 smithi121 bash[36902]: audit 2022-04-22T17:32:55.693137+0000 mgr.y (mgr.34107) 240 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:56 smithi121 bash[36902]: cephadm 2022-04-22T17:32:55.695828+0000 mgr.y (mgr.34107) 241 : cephadm [INF] Upgrade: osd.5 is safe to restart 2022-04-22T17:32:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:56 smithi121 bash[36902]: cluster 2022-04-22T17:32:56.842644+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2022-04-22T17:32:57.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[45495]: cluster 2022-04-22T17:32:55.241136+0000 mgr.y (mgr.34107) 239 : cluster [DBG] pgmap v405: 121 pgs: 121 active+clean; 329 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2022-04-22T17:32:57.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[45495]: audit 2022-04-22T17:32:55.693137+0000 mgr.y (mgr.34107) 240 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[45495]: cephadm 2022-04-22T17:32:55.695828+0000 mgr.y (mgr.34107) 241 : cephadm [INF] Upgrade: osd.5 is safe to restart 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[45495]: cluster 2022-04-22T17:32:56.842644+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[46934]: cluster 2022-04-22T17:32:55.241136+0000 mgr.y (mgr.34107) 239 : cluster [DBG] pgmap v405: 121 pgs: 121 active+clean; 329 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[46934]: audit 2022-04-22T17:32:55.693137+0000 mgr.y (mgr.34107) 240 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[46934]: cephadm 2022-04-22T17:32:55.695828+0000 mgr.y (mgr.34107) 241 : cephadm [INF] Upgrade: osd.5 is safe to restart 2022-04-22T17:32:57.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:56 smithi115 bash[46934]: cluster 2022-04-22T17:32:56.842644+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2096: done (4 left) 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2097: done (3 left) 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2098: done (2 left) 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2100: done (1 left) 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2101: done (0 left) 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2102: rollback oid 23 current snap is 212 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 173 2022-04-22T17:32:57.860 INFO:tasks.rados.rados.0.smithi115.stdout:2103: copy_from oid 7 from oid 25 current snap is 212 2022-04-22T17:32:57.861 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 212 2022-04-22T17:32:57.861 INFO:tasks.rados.rados.0.smithi115.stdout:2104: seq_num 648 ranges {253952=458752,712704=475136,1187840=327680} 2022-04-22T17:32:57.866 INFO:tasks.rados.rados.0.smithi115.stdout:2104: writing smithi11541303-8 from 253952 to 712704 tid 1 2022-04-22T17:32:57.870 INFO:tasks.rados.rados.0.smithi115.stdout:2104: writing smithi11541303-8 from 712704 to 1187840 tid 2 2022-04-22T17:32:57.872 INFO:tasks.rados.rados.0.smithi115.stdout:2104: writing smithi11541303-8 from 1187840 to 1515520 tid 3 2022-04-22T17:32:57.872 INFO:tasks.rados.rados.0.smithi115.stdout:2105: snap_remove snap 208 2022-04-22T17:32:58.866 INFO:tasks.rados.rados.0.smithi115.stdout:2105: done (3 left) 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2106: read oid 31 snap -1 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2106: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2107: setattr oid 45 current snap is 212 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2102: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2102: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2102: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1013 (ObjNum 500 snap 170 seq_num 500) dirty exists 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2104: finishing write tid 1 to smithi11541303-8 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2104: finishing write tid 2 to smithi11541303-8 2022-04-22T17:32:58.867 INFO:tasks.rados.rados.0.smithi115.stdout:2104: finishing write tid 3 to smithi11541303-8 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:2104: finishing write tid 4 to smithi11541303-8 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:2103: finishing copy_from to smithi11541303-7 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 998 (ObjNum 611 snap 200 seq_num 611) dirty exists 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:2103: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:2104: finishing write tid 5 to smithi11541303-8 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1199 (ObjNum 648 snap 212 seq_num 648) dirty exists 2022-04-22T17:32:58.868 INFO:tasks.rados.rados.0.smithi115.stdout:2104: left oid 8 (ObjNum 648 snap 212 seq_num 648) 2022-04-22T17:32:58.869 INFO:tasks.rados.rados.0.smithi115.stdout:2102: done (4 left) 2022-04-22T17:32:58.869 INFO:tasks.rados.rados.0.smithi115.stdout:2103: done (3 left) 2022-04-22T17:32:58.869 INFO:tasks.rados.rados.0.smithi115.stdout:2104: done (2 left) 2022-04-22T17:32:58.869 INFO:tasks.rados.rados.0.smithi115.stdout:2108: snap_remove snap 198 2022-04-22T17:32:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:58 smithi121 bash[36902]: cluster 2022-04-22T17:32:57.241985+0000 mgr.y (mgr.34107) 242 : cluster [DBG] pgmap v408: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:32:59.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:58 smithi121 bash[36902]: cluster 2022-04-22T17:32:57.844603+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2022-04-22T17:32:59.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:58 smithi121 bash[36902]: audit 2022-04-22T17:32:58.376198+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[45495]: cluster 2022-04-22T17:32:57.241985+0000 mgr.y (mgr.34107) 242 : cluster [DBG] pgmap v408: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:32:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[45495]: cluster 2022-04-22T17:32:57.844603+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2022-04-22T17:32:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[45495]: audit 2022-04-22T17:32:58.376198+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[46934]: cluster 2022-04-22T17:32:57.241985+0000 mgr.y (mgr.34107) 242 : cluster [DBG] pgmap v408: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2022-04-22T17:32:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[46934]: cluster 2022-04-22T17:32:57.844603+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2022-04-22T17:32:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:58 smithi115 bash[46934]: audit 2022-04-22T17:32:58.376198+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:32:59.858 INFO:tasks.rados.rados.0.smithi115.stdout:2108: done (2 left) 2022-04-22T17:32:59.858 INFO:tasks.rados.rados.0.smithi115.stdout:2109: delete oid 12 current snap is 212 2022-04-22T17:32:59.858 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1193 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:32:59.858 INFO:tasks.rados.rados.0.smithi115.stdout:2106: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:33:00.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:59 smithi121 bash[36902]: cephadm 2022-04-22T17:32:58.365051+0000 mgr.y (mgr.34107) 243 : cephadm [INF] Upgrade: Updating osd.5 2022-04-22T17:33:00.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:59 smithi121 bash[36902]: cephadm 2022-04-22T17:32:58.378881+0000 mgr.y (mgr.34107) 244 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:33:00.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:59 smithi121 bash[36902]: cluster 2022-04-22T17:32:58.851428+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2022-04-22T17:33:00.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:59 smithi121 bash[36902]: audit 2022-04-22T17:32:59.267687+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:00.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:32:59 smithi121 bash[36902]: cluster 2022-04-22T17:32:59.841126+0000 mon.a (mon.0) 779 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2022-04-22T17:33:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[45495]: cephadm 2022-04-22T17:32:58.365051+0000 mgr.y (mgr.34107) 243 : cephadm [INF] Upgrade: Updating osd.5 2022-04-22T17:33:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[45495]: cephadm 2022-04-22T17:32:58.378881+0000 mgr.y (mgr.34107) 244 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:33:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[45495]: cluster 2022-04-22T17:32:58.851428+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2022-04-22T17:33:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[45495]: audit 2022-04-22T17:32:59.267687+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[45495]: cluster 2022-04-22T17:32:59.841126+0000 mon.a (mon.0) 779 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[46934]: cephadm 2022-04-22T17:32:58.365051+0000 mgr.y (mgr.34107) 243 : cephadm [INF] Upgrade: Updating osd.5 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[46934]: cephadm 2022-04-22T17:32:58.378881+0000 mgr.y (mgr.34107) 244 : cephadm [INF] Deploying daemon osd.5 on smithi121 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[46934]: cluster 2022-04-22T17:32:58.851428+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[46934]: audit 2022-04-22T17:32:59.267687+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:32:59 smithi115 bash[46934]: cluster 2022-04-22T17:32:59.841126+0000 mon.a (mon.0) 779 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2022-04-22T17:33:00.570 INFO:tasks.rados.rados.0.smithi115.stdout:2106: done (2 left) 2022-04-22T17:33:00.571 INFO:tasks.rados.rados.0.smithi115.stdout:2107: done (1 left) 2022-04-22T17:33:00.571 INFO:tasks.rados.rados.0.smithi115.stdout:2109: done (0 left) 2022-04-22T17:33:00.571 INFO:tasks.rados.rados.0.smithi115.stdout:2110: delete oid 13 current snap is 212 2022-04-22T17:33:00.575 INFO:tasks.rados.rados.0.smithi115.stdout:2110: done (0 left) 2022-04-22T17:33:00.575 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 212 2022-04-22T17:33:00.575 INFO:tasks.rados.rados.0.smithi115.stdout:2111: seq_num 651 ranges {1884160=507904,2392064=532480,2924544=270336} 2022-04-22T17:33:00.596 INFO:tasks.rados.rados.0.smithi115.stdout:2111: writing smithi11541303-39 from 1884160 to 2392064 tid 1 2022-04-22T17:33:00.599 INFO:tasks.rados.rados.0.smithi115.stdout:2111: writing smithi11541303-39 from 2392064 to 2924544 tid 2 2022-04-22T17:33:00.601 INFO:tasks.rados.rados.0.smithi115.stdout:2111: writing smithi11541303-39 from 2924544 to 3194880 tid 3 2022-04-22T17:33:00.601 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 212 2022-04-22T17:33:00.601 INFO:tasks.rados.rados.0.smithi115.stdout:2112: seq_num 652 ranges {114688=524288,638976=409600,1048576=393216} 2022-04-22T17:33:00.604 INFO:tasks.rados.rados.0.smithi115.stdout:2112: writing smithi11541303-41 from 114688 to 638976 tid 1 2022-04-22T17:33:00.607 INFO:tasks.rados.rados.0.smithi115.stdout:2112: writing smithi11541303-41 from 638976 to 1048576 tid 2 2022-04-22T17:33:00.609 INFO:tasks.rados.rados.0.smithi115.stdout:2112: writing smithi11541303-41 from 1048576 to 1441792 tid 3 2022-04-22T17:33:00.609 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 212 2022-04-22T17:33:00.609 INFO:tasks.rados.rados.0.smithi115.stdout:2113: seq_num 653 ranges {8839168=540672,9379840=696320,10076160=106496} 2022-04-22T17:33:00.661 INFO:tasks.rados.rados.0.smithi115.stdout:2113: writing smithi11541303-20 from 8839168 to 9379840 tid 1 2022-04-22T17:33:00.665 INFO:tasks.rados.rados.0.smithi115.stdout:2113: writing smithi11541303-20 from 9379840 to 10076160 tid 2 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2113: writing smithi11541303-20 from 10076160 to 10182656 tid 3 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2114: setattr oid 19 current snap is 212 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2111: finishing write tid 1 to smithi11541303-39 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2111: finishing write tid 2 to smithi11541303-39 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2111: finishing write tid 3 to smithi11541303-39 2022-04-22T17:33:00.667 INFO:tasks.rados.rados.0.smithi115.stdout:2111: finishing write tid 4 to smithi11541303-39 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2111: finishing write tid 5 to smithi11541303-39 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1102 (ObjNum 651 snap 212 seq_num 651) dirty exists 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2111: left oid 39 (ObjNum 651 snap 212 seq_num 651) 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: finishing write tid 1 to smithi11541303-41 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: finishing write tid 2 to smithi11541303-41 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: finishing write tid 3 to smithi11541303-41 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: finishing write tid 4 to smithi11541303-41 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: finishing write tid 5 to smithi11541303-41 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1128 (ObjNum 652 snap 212 seq_num 652) dirty exists 2022-04-22T17:33:00.668 INFO:tasks.rados.rados.0.smithi115.stdout:2112: left oid 41 (ObjNum 652 snap 212 seq_num 652) 2022-04-22T17:33:00.669 INFO:tasks.rados.rados.0.smithi115.stdout:2111: done (3 left) 2022-04-22T17:33:00.669 INFO:tasks.rados.rados.0.smithi115.stdout:2112: done (2 left) 2022-04-22T17:33:00.669 INFO:tasks.rados.rados.0.smithi115.stdout:2115: delete oid 10 current snap is 212 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2115: done (2 left) 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2116: read oid 3 snap -1 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2116: expect (ObjNum 625 snap 204 seq_num 625) 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2117: read oid 28 snap 204 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2117: expect (ObjNum 622 snap 202 seq_num 622) 2022-04-22T17:33:00.673 INFO:tasks.rados.rados.0.smithi115.stdout:2118: rmattr oid 4 current snap is 212 2022-04-22T17:33:00.674 INFO:tasks.rados.rados.0.smithi115.stdout:2118: done (4 left) 2022-04-22T17:33:00.674 INFO:tasks.rados.rados.0.smithi115.stdout:2119: snap_create 2022-04-22T17:33:00.674 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1118 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:00.675 INFO:tasks.rados.rados.0.smithi115.stdout:2113: finishing write tid 1 to smithi11541303-20 2022-04-22T17:33:00.675 INFO:tasks.rados.rados.0.smithi115.stdout:2113: finishing write tid 2 to smithi11541303-20 2022-04-22T17:33:00.676 INFO:tasks.rados.rados.0.smithi115.stdout:2113: finishing write tid 3 to smithi11541303-20 2022-04-22T17:33:00.676 INFO:tasks.rados.rados.0.smithi115.stdout:2113: finishing write tid 4 to smithi11541303-20 2022-04-22T17:33:00.678 INFO:tasks.rados.rados.0.smithi115.stdout:2113: finishing write tid 5 to smithi11541303-20 2022-04-22T17:33:00.678 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1093 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:33:00.679 INFO:tasks.rados.rados.0.smithi115.stdout:2113: left oid 20 (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:33:00.680 INFO:tasks.rados.rados.0.smithi115.stdout:2117: expect (ObjNum 622 snap 202 seq_num 622) 2022-04-22T17:33:00.716 INFO:tasks.rados.rados.0.smithi115.stdout:2116: expect (ObjNum 625 snap 204 seq_num 625) 2022-04-22T17:33:00.896 INFO:tasks.rados.rados.0.smithi115.stdout:2113: done (4 left) 2022-04-22T17:33:00.896 INFO:tasks.rados.rados.0.smithi115.stdout:2114: done (3 left) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2116: done (2 left) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2117: done (1 left) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2119: done (0 left) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2120: read oid 32 snap 212 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2120: expect (ObjNum 646 snap 210 seq_num 646) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2121: read oid 43 snap -1 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2121: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:33:00.897 INFO:tasks.rados.rados.0.smithi115.stdout:2122: snap_remove snap 202 2022-04-22T17:33:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:00 smithi121 bash[36902]: cluster 2022-04-22T17:32:59.243235+0000 mgr.y (mgr.34107) 245 : cluster [DBG] pgmap v411: 121 pgs: 121 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 270 KiB/s wr, 19 op/s 2022-04-22T17:33:01.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:00 smithi115 bash[45495]: cluster 2022-04-22T17:32:59.243235+0000 mgr.y (mgr.34107) 245 : cluster [DBG] pgmap v411: 121 pgs: 121 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 270 KiB/s wr, 19 op/s 2022-04-22T17:33:01.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:00 smithi115 bash[46934]: cluster 2022-04-22T17:32:59.243235+0000 mgr.y (mgr.34107) 245 : cluster [DBG] pgmap v411: 121 pgs: 121 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 270 KiB/s wr, 19 op/s 2022-04-22T17:33:01.910 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:01.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:01.910 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:01.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:01.910 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:01.627Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:01.920 INFO:tasks.rados.rados.0.smithi115.stdout:2122: done (2 left) 2022-04-22T17:33:01.920 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 213 2022-04-22T17:33:01.920 INFO:tasks.rados.rados.0.smithi115.stdout:2123: seq_num 655 ranges {6299648=573440,6873088=516096,7389184=286720} 2022-04-22T17:33:01.961 INFO:tasks.rados.rados.0.smithi115.stdout:2123: writing smithi11541303-33 from 6299648 to 6873088 tid 1 2022-04-22T17:33:01.964 INFO:tasks.rados.rados.0.smithi115.stdout:2123: writing smithi11541303-33 from 6873088 to 7389184 tid 2 2022-04-22T17:33:01.966 INFO:tasks.rados.rados.0.smithi115.stdout:2123: writing smithi11541303-33 from 7389184 to 7675904 tid 3 2022-04-22T17:33:01.966 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 213 2022-04-22T17:33:01.966 INFO:tasks.rados.rados.0.smithi115.stdout:2124: seq_num 656 ranges {0=589824,589824=802816} 2022-04-22T17:33:01.969 INFO:tasks.rados.rados.0.smithi115.stdout:2124: writing smithi11541303-30 from 0 to 589824 tid 1 2022-04-22T17:33:01.974 INFO:tasks.rados.rados.0.smithi115.stdout:2124: writing smithi11541303-30 from 589824 to 1392640 tid 2 2022-04-22T17:33:01.974 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 213 2022-04-22T17:33:01.974 INFO:tasks.rados.rados.0.smithi115.stdout:2125: seq_num 657 ranges {1130496=606208,1736704=688128,2424832=114688} 2022-04-22T17:33:01.983 INFO:tasks.rados.rados.0.smithi115.stdout:2125: writing smithi11541303-28 from 1130496 to 1736704 tid 1 2022-04-22T17:33:01.987 INFO:tasks.rados.rados.0.smithi115.stdout:2125: writing smithi11541303-28 from 1736704 to 2424832 tid 2 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2125: writing smithi11541303-28 from 2424832 to 2539520 tid 3 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2126: read oid 23 snap -1 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2126: expect (ObjNum 500 snap 170 seq_num 500) 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2127: rollback oid 16 current snap is 213 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 179 2022-04-22T17:33:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2121: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:33:01.998 INFO:tasks.rados.rados.0.smithi115.stdout:2120: expect (ObjNum 646 snap 210 seq_num 646) 2022-04-22T17:33:02.103 INFO:tasks.rados.rados.0.smithi115.stdout:2124: finishing write tid 1 to smithi11541303-30 2022-04-22T17:33:02.103 INFO:tasks.rados.rados.0.smithi115.stdout:2123: finishing write tid 1 to smithi11541303-33 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: finishing write tid 2 to smithi11541303-33 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: finishing write tid 3 to smithi11541303-33 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: finishing write tid 4 to smithi11541303-33 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2124: finishing write tid 2 to smithi11541303-30 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: finishing write tid 5 to smithi11541303-33 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1000 (ObjNum 655 snap 213 seq_num 655) dirty exists 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: left oid 33 (ObjNum 655 snap 213 seq_num 655) 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2120: done (6 left) 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2121: done (5 left) 2022-04-22T17:33:02.104 INFO:tasks.rados.rados.0.smithi115.stdout:2123: done (4 left) 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 213 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2124: finishing write tid 3 to smithi11541303-30 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2124: finishing write tid 4 to smithi11541303-30 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1196 (ObjNum 656 snap 213 seq_num 656) dirty exists 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2124: left oid 30 (ObjNum 656 snap 213 seq_num 656) 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2125: finishing write tid 1 to smithi11541303-28 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2127: finishing rollback tid 0 to smithi11541303-16 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2127: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1114 (ObjNum 0 snap 3 seq_num 6) dirty dne 2022-04-22T17:33:02.105 INFO:tasks.rados.rados.0.smithi115.stdout:2126: expect (ObjNum 500 snap 170 seq_num 500) 2022-04-22T17:33:02.154 INFO:tasks.rados.rados.0.smithi115.stdout:2125: finishing write tid 2 to smithi11541303-28 2022-04-22T17:33:02.154 INFO:tasks.rados.rados.0.smithi115.stdout:2125: finishing write tid 3 to smithi11541303-28 2022-04-22T17:33:02.154 INFO:tasks.rados.rados.0.smithi115.stdout:2125: finishing write tid 4 to smithi11541303-28 2022-04-22T17:33:02.154 INFO:tasks.rados.rados.0.smithi115.stdout:2125: finishing write tid 5 to smithi11541303-28 2022-04-22T17:33:02.154 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1096 (ObjNum 657 snap 213 seq_num 657) dirty exists 2022-04-22T17:33:02.155 INFO:tasks.rados.rados.0.smithi115.stdout:2125: left oid 28 (ObjNum 657 snap 213 seq_num 657) 2022-04-22T17:33:02.155 INFO:tasks.rados.rados.0.smithi115.stdout:2128: seq_num 658 ranges {0=622592,622592=565248,1187840=237568} 2022-04-22T17:33:02.164 INFO:tasks.rados.rados.0.smithi115.stdout:2128: writing smithi11541303-26 from 0 to 622592 tid 1 2022-04-22T17:33:02.168 INFO:tasks.rados.rados.0.smithi115.stdout:2128: writing smithi11541303-26 from 622592 to 1187840 tid 2 2022-04-22T17:33:02.170 INFO:tasks.rados.rados.0.smithi115.stdout:2128: writing smithi11541303-26 from 1187840 to 1425408 tid 3 2022-04-22T17:33:02.170 INFO:tasks.rados.rados.0.smithi115.stdout:2124: done (4 left) 2022-04-22T17:33:02.170 INFO:tasks.rados.rados.0.smithi115.stdout:2125: done (3 left) 2022-04-22T17:33:02.170 INFO:tasks.rados.rados.0.smithi115.stdout:2126: done (2 left) 2022-04-22T17:33:02.170 INFO:tasks.rados.rados.0.smithi115.stdout:2127: done (1 left) 2022-04-22T17:33:02.171 INFO:tasks.rados.rados.0.smithi115.stdout:2129: read oid 21 snap -1 2022-04-22T17:33:02.171 INFO:tasks.rados.rados.0.smithi115.stdout:2129: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:33:02.171 INFO:tasks.rados.rados.0.smithi115.stdout:2130: snap_create 2022-04-22T17:33:02.171 INFO:tasks.rados.rados.0.smithi115.stdout:2128: finishing write tid 1 to smithi11541303-26 2022-04-22T17:33:02.183 INFO:tasks.rados.rados.0.smithi115.stdout:2128: finishing write tid 2 to smithi11541303-26 2022-04-22T17:33:02.183 INFO:tasks.rados.rados.0.smithi115.stdout:2128: finishing write tid 3 to smithi11541303-26 2022-04-22T17:33:02.183 INFO:tasks.rados.rados.0.smithi115.stdout:2128: finishing write tid 4 to smithi11541303-26 2022-04-22T17:33:02.184 INFO:tasks.rados.rados.0.smithi115.stdout:2128: finishing write tid 5 to smithi11541303-26 2022-04-22T17:33:02.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1004 (ObjNum 658 snap 213 seq_num 658) dirty exists 2022-04-22T17:33:02.185 INFO:tasks.rados.rados.0.smithi115.stdout:2128: left oid 26 (ObjNum 658 snap 213 seq_num 658) 2022-04-22T17:33:02.197 INFO:tasks.rados.rados.0.smithi115.stdout:2129: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:33:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:01 smithi121 bash[36902]: cluster 2022-04-22T17:33:00.881168+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2022-04-22T17:33:02.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:01 smithi115 bash[46934]: cluster 2022-04-22T17:33:00.881168+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2022-04-22T17:33:02.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:01 smithi115 bash[45495]: cluster 2022-04-22T17:33:00.881168+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2022-04-22T17:33:02.919 INFO:tasks.rados.rados.0.smithi115.stdout:2128: done (2 left) 2022-04-22T17:33:02.920 INFO:tasks.rados.rados.0.smithi115.stdout:2129: done (1 left) 2022-04-22T17:33:02.920 INFO:tasks.rados.rados.0.smithi115.stdout:2130: done (0 left) 2022-04-22T17:33:02.920 INFO:tasks.rados.rados.0.smithi115.stdout:2131: delete oid 49 current snap is 214 2022-04-22T17:33:02.932 INFO:tasks.rados.rados.0.smithi115.stdout:2131: done (0 left) 2022-04-22T17:33:02.932 INFO:tasks.rados.rados.0.smithi115.stdout:2132: read oid 3 snap 204 2022-04-22T17:33:02.932 INFO:tasks.rados.rados.0.smithi115.stdout:2132: expect (ObjNum 625 snap 204 seq_num 625) 2022-04-22T17:33:02.932 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 214 2022-04-22T17:33:02.932 INFO:tasks.rados.rados.0.smithi115.stdout:2133: seq_num 660 ranges {0=663552,663552=737280,1400832=65536} 2022-04-22T17:33:02.939 INFO:tasks.rados.rados.0.smithi115.stdout:2133: writing smithi11541303-10 from 0 to 663552 tid 1 2022-04-22T17:33:02.943 INFO:tasks.rados.rados.0.smithi115.stdout:2133: writing smithi11541303-10 from 663552 to 1400832 tid 2 2022-04-22T17:33:02.944 INFO:tasks.rados.rados.0.smithi115.stdout:2133: writing smithi11541303-10 from 1400832 to 1466368 tid 3 2022-04-22T17:33:02.944 INFO:tasks.rados.rados.0.smithi115.stdout:2134: snap_create 2022-04-22T17:33:02.944 INFO:tasks.rados.rados.0.smithi115.stdout:2132: expect (ObjNum 625 snap 204 seq_num 625) 2022-04-22T17:33:02.984 INFO:tasks.rados.rados.0.smithi115.stdout:2133: finishing write tid 1 to smithi11541303-10 2022-04-22T17:33:02.984 INFO:tasks.rados.rados.0.smithi115.stdout:2133: finishing write tid 2 to smithi11541303-10 2022-04-22T17:33:02.984 INFO:tasks.rados.rados.0.smithi115.stdout:2133: finishing write tid 3 to smithi11541303-10 2022-04-22T17:33:02.985 INFO:tasks.rados.rados.0.smithi115.stdout:2133: finishing write tid 4 to smithi11541303-10 2022-04-22T17:33:02.985 INFO:tasks.rados.rados.0.smithi115.stdout:2133: finishing write tid 5 to smithi11541303-10 2022-04-22T17:33:02.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1133 (ObjNum 660 snap 214 seq_num 660) dirty exists 2022-04-22T17:33:02.985 INFO:tasks.rados.rados.0.smithi115.stdout:2133: left oid 10 (ObjNum 660 snap 214 seq_num 660) 2022-04-22T17:33:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:02 smithi121 bash[36902]: cluster 2022-04-22T17:33:01.244114+0000 mgr.y (mgr.34107) 246 : cluster [DBG] pgmap v414: 121 pgs: 121 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:33:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:02 smithi121 bash[36902]: cluster 2022-04-22T17:33:01.901882+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2022-04-22T17:33:03.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:02 smithi115 bash[45495]: cluster 2022-04-22T17:33:01.244114+0000 mgr.y (mgr.34107) 246 : cluster [DBG] pgmap v414: 121 pgs: 121 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:33:03.339 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:02 smithi115 bash[45495]: cluster 2022-04-22T17:33:01.901882+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2022-04-22T17:33:03.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:02 smithi115 bash[46934]: cluster 2022-04-22T17:33:01.244114+0000 mgr.y (mgr.34107) 246 : cluster [DBG] pgmap v414: 121 pgs: 121 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:33:03.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:02 smithi115 bash[46934]: cluster 2022-04-22T17:33:01.901882+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2022-04-22T17:33:03.651 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:03 smithi121 systemd[1]: Stopping Ceph osd.5 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:33:03.651 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[18604]: debug 2022-04-22T17:33:03.496+0000 7faa343d9700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:33:03.651 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[18604]: debug 2022-04-22T17:33:03.496+0000 7faa343d9700 -1 osd.5 486 *** Got signal Terminated *** 2022-04-22T17:33:03.652 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[18604]: debug 2022-04-22T17:33:03.496+0000 7faa343d9700 -1 osd.5 486 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:33:03.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:03.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:03.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:03.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:03.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:03.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:33:03.934 INFO:tasks.rados.rados.0.smithi115.stdout:2132: done (2 left) 2022-04-22T17:33:03.934 INFO:tasks.rados.rados.0.smithi115.stdout:2133: done (1 left) 2022-04-22T17:33:03.934 INFO:tasks.rados.rados.0.smithi115.stdout:2134: done (0 left) 2022-04-22T17:33:03.934 INFO:tasks.rados.rados.0.smithi115.stdout:2135: copy_from oid 45 from oid 2 current snap is 215 2022-04-22T17:33:03.934 INFO:tasks.rados.rados.0.smithi115.stdout:2136: read oid 19 snap 213 2022-04-22T17:33:03.935 INFO:tasks.rados.rados.0.smithi115.stdout:2136: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:03.935 INFO:tasks.rados.rados.0.smithi115.stdout:2137: copy_from oid 32 from oid 47 current snap is 215 2022-04-22T17:33:03.935 INFO:tasks.rados.rados.0.smithi115.stdout:2138: snap_create 2022-04-22T17:33:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:02.905051+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2022-04-22T17:33:04.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.804534+0000 mon.a (mon.0) 783 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.804646+0000 mon.a (mon.0) 784 : cluster [INF] osd.5 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:04.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.805245+0000 mon.a (mon.0) 785 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.805568+0000 mon.a (mon.0) 786 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.805897+0000 mon.a (mon.0) 787 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.806127+0000 mon.a (mon.0) 788 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.806490+0000 mon.a (mon.0) 789 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.806774+0000 mon.a (mon.0) 790 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.807219+0000 mon.a (mon.0) 791 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.807515+0000 mon.a (mon.0) 792 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.807790+0000 mon.a (mon.0) 793 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.808109+0000 mon.a (mon.0) 794 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.808312+0000 mon.a (mon.0) 795 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.808500+0000 mon.a (mon.0) 796 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:03 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.808918+0000 mon.a (mon.0) 797 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.337 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:02.905051+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2022-04-22T17:33:04.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.804534+0000 mon.a (mon.0) 783 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.804646+0000 mon.a (mon.0) 784 : cluster [INF] osd.5 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:04.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.805245+0000 mon.a (mon.0) 785 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.805568+0000 mon.a (mon.0) 786 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.339 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.805897+0000 mon.a (mon.0) 787 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.339 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.806127+0000 mon.a (mon.0) 788 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.339 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.806490+0000 mon.a (mon.0) 789 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.339 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.806774+0000 mon.a (mon.0) 790 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.807219+0000 mon.a (mon.0) 791 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.807515+0000 mon.a (mon.0) 792 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.807790+0000 mon.a (mon.0) 793 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.808109+0000 mon.a (mon.0) 794 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.808312+0000 mon.a (mon.0) 795 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.808500+0000 mon.a (mon.0) 796 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.808918+0000 mon.a (mon.0) 797 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:02.905051+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2022-04-22T17:33:04.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.804534+0000 mon.a (mon.0) 783 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.804646+0000 mon.a (mon.0) 784 : cluster [INF] osd.5 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:04.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.805245+0000 mon.a (mon.0) 785 : cluster [DBG] osd.5 reported immediately failed by osd.3 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.805568+0000 mon.a (mon.0) 786 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.805897+0000 mon.a (mon.0) 787 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.806127+0000 mon.a (mon.0) 788 : cluster [DBG] osd.5 reported immediately failed by osd.1 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.806490+0000 mon.a (mon.0) 789 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.806774+0000 mon.a (mon.0) 790 : cluster [DBG] osd.5 reported immediately failed by osd.0 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.807219+0000 mon.a (mon.0) 791 : cluster [DBG] osd.5 reported immediately failed by osd.2 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.807515+0000 mon.a (mon.0) 792 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.807790+0000 mon.a (mon.0) 793 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.808109+0000 mon.a (mon.0) 794 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.808312+0000 mon.a (mon.0) 795 : cluster [DBG] osd.5 reported immediately failed by osd.6 2022-04-22T17:33:04.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.808500+0000 mon.a (mon.0) 796 : cluster [DBG] osd.5 reported immediately failed by osd.7 2022-04-22T17:33:04.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:03 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.808918+0000 mon.a (mon.0) 797 : cluster [DBG] osd.5 reported immediately failed by osd.4 2022-04-22T17:33:04.766 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[41058]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.5 2022-04-22T17:33:04.772 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[41102]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-5 2022-04-22T17:33:04.772 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[41115]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.5 2022-04-22T17:33:04.861 INFO:tasks.rados.rados.0.smithi115.stdout:2138: done (3 left) 2022-04-22T17:33:04.861 INFO:tasks.rados.rados.0.smithi115.stdout:2139: snap_remove snap 214 2022-04-22T17:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.245065+0000 mgr.y (mgr.34107) 247 : cluster [DBG] pgmap v417: 121 pgs: 121 active+clean; 329 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2022-04-22T17:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.903417+0000 mon.a (mon.0) 798 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[36902]: cluster 2022-04-22T17:33:03.919383+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e487: 8 total, 7 up, 8 in 2022-04-22T17:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[36902]: audit 2022-04-22T17:33:04.270859+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:04 smithi121 bash[36902]: cluster 2022-04-22T17:33:04.844583+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e488: 8 total, 7 up, 8 in 2022-04-22T17:33:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.245065+0000 mgr.y (mgr.34107) 247 : cluster [DBG] pgmap v417: 121 pgs: 121 active+clean; 329 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2022-04-22T17:33:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.903417+0000 mon.a (mon.0) 798 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[45495]: cluster 2022-04-22T17:33:03.919383+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e487: 8 total, 7 up, 8 in 2022-04-22T17:33:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[45495]: audit 2022-04-22T17:33:04.270859+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[45495]: cluster 2022-04-22T17:33:04.844583+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e488: 8 total, 7 up, 8 in 2022-04-22T17:33:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.245065+0000 mgr.y (mgr.34107) 247 : cluster [DBG] pgmap v417: 121 pgs: 121 active+clean; 329 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2022-04-22T17:33:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.903417+0000 mon.a (mon.0) 798 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[46934]: cluster 2022-04-22T17:33:03.919383+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e487: 8 total, 7 up, 8 in 2022-04-22T17:33:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[46934]: audit 2022-04-22T17:33:04.270859+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:04 smithi115 bash[46934]: cluster 2022-04-22T17:33:04.844583+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e488: 8 total, 7 up, 8 in 2022-04-22T17:33:05.864 INFO:tasks.rados.rados.0.smithi115.stdout:2139: done (3 left) 2022-04-22T17:33:05.864 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 216 2022-04-22T17:33:05.864 INFO:tasks.rados.rados.0.smithi115.stdout:2135: finishing copy_from to smithi11541303-45 2022-04-22T17:33:05.864 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1090 (ObjNum 0 snap 32692 seq_num 3178903616) dirty exists 2022-04-22T17:33:05.864 INFO:tasks.rados.rados.0.smithi115.stdout:2135: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:33:05.865 INFO:tasks.rados.rados.0.smithi115.stdout:2136: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:06.028 INFO:tasks.rados.rados.0.smithi115.stdout:2137: finishing copy_from to smithi11541303-32 2022-04-22T17:33:06.028 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1112 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:06.029 INFO:tasks.rados.rados.0.smithi115.stdout:2137: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:33:06.029 INFO:tasks.rados.rados.0.smithi115.stdout:2140: seq_num 661 ranges {1425408=679936,2105344=622592,2727936=180224} 2022-04-22T17:33:06.047 INFO:tasks.rados.rados.0.smithi115.stdout:2140: writing smithi11541303-26 from 1425408 to 2105344 tid 1 2022-04-22T17:33:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:2140: writing smithi11541303-26 from 2105344 to 2727936 tid 2 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2140: writing smithi11541303-26 from 2727936 to 2908160 tid 3 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2135: done (3 left) 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2136: done (2 left) 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2137: done (1 left) 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2141: copy_from oid 20 from oid 46 current snap is 216 2022-04-22T17:33:06.053 INFO:tasks.rados.rados.0.smithi115.stdout:2142: rollback oid 4 current snap is 216 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 211 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2143: read oid 31 snap 213 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2143: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2144: read oid 8 snap -1 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2144: expect (ObjNum 648 snap 212 seq_num 648) 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2145: read oid 11 snap -1 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2145: expect deleted 2022-04-22T17:33:06.054 INFO:tasks.rados.rados.0.smithi115.stdout:2146: snap_create 2022-04-22T17:33:06.057 INFO:tasks.rados.rados.0.smithi115.stdout:2141: finishing copy_from to smithi11541303-20 2022-04-22T17:33:06.057 INFO:tasks.rados.rados.0.smithi115.stdout:2141: got expected ENOENT (src dne) 2022-04-22T17:33:06.058 INFO:tasks.rados.rados.0.smithi115.stdout:2141: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:33:06.058 INFO:tasks.rados.rados.0.smithi115.stdout:2140: finishing write tid 1 to smithi11541303-26 2022-04-22T17:33:06.058 INFO:tasks.rados.rados.0.smithi115.stdout:2142: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:33:06.059 INFO:tasks.rados.rados.0.smithi115.stdout:2140: finishing write tid 2 to smithi11541303-26 2022-04-22T17:33:06.061 INFO:tasks.rados.rados.0.smithi115.stdout:2142: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:33:06.061 INFO:tasks.rados.rados.0.smithi115.stdout:2140: finishing write tid 3 to smithi11541303-26 2022-04-22T17:33:06.061 INFO:tasks.rados.rados.0.smithi115.stdout:2142: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:33:06.062 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1199 (ObjNum 558 snap 182 seq_num 558) dirty exists 2022-04-22T17:33:06.062 INFO:tasks.rados.rados.0.smithi115.stdout:2140: finishing write tid 4 to smithi11541303-26 2022-04-22T17:33:06.062 INFO:tasks.rados.rados.0.smithi115.stdout:2140: finishing write tid 5 to smithi11541303-26 2022-04-22T17:33:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1116 (ObjNum 661 snap 216 seq_num 661) dirty exists 2022-04-22T17:33:06.063 INFO:tasks.rados.rados.0.smithi115.stdout:2140: left oid 26 (ObjNum 661 snap 216 seq_num 661) 2022-04-22T17:33:06.066 INFO:tasks.rados.rados.0.smithi115.stdout:2144: expect (ObjNum 648 snap 212 seq_num 648) 2022-04-22T17:33:06.118 INFO:tasks.rados.rados.0.smithi115.stdout:2143: expect (ObjNum 525 snap 173 seq_num 525) 2022-04-22T17:33:06.938 INFO:tasks.rados.rados.0.smithi115.stdout:2140: done (6 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2141: done (5 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2142: done (4 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2143: done (3 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2144: done (2 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2145: done (1 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2146: done (0 left) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2147: read oid 15 snap 179 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2147: expect (ObjNum 490 snap 169 seq_num 490) 2022-04-22T17:33:06.939 INFO:tasks.rados.rados.0.smithi115.stdout:2148: snap_remove snap 215 2022-04-22T17:33:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:06 smithi121 bash[36902]: cluster 2022-04-22T17:33:05.246107+0000 mgr.y (mgr.34107) 248 : cluster [DBG] pgmap v420: 121 pgs: 1 active+undersized+degraded, 16 peering, 11 stale+active+clean, 5 active+undersized, 88 active+clean; 320 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 23 op/s; 10/606 objects degraded (1.650%) 2022-04-22T17:33:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:06 smithi121 bash[36902]: cluster 2022-04-22T17:33:05.849225+0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e489: 8 total, 7 up, 8 in 2022-04-22T17:33:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:06 smithi121 bash[36902]: cluster 2022-04-22T17:33:05.912885+0000 mon.a (mon.0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2022-04-22T17:33:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:06 smithi121 bash[36902]: cluster 2022-04-22T17:33:05.912925+0000 mon.a (mon.0) 804 : cluster [WRN] Health check failed: Degraded data redundancy: 10/606 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:33:07.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:07 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.5.service: Succeeded. 2022-04-22T17:33:07.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:07 smithi121 systemd[1]: Stopped Ceph osd.5 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:33:07.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[45495]: cluster 2022-04-22T17:33:05.246107+0000 mgr.y (mgr.34107) 248 : cluster [DBG] pgmap v420: 121 pgs: 1 active+undersized+degraded, 16 peering, 11 stale+active+clean, 5 active+undersized, 88 active+clean; 320 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 23 op/s; 10/606 objects degraded (1.650%) 2022-04-22T17:33:07.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[45495]: cluster 2022-04-22T17:33:05.849225+0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e489: 8 total, 7 up, 8 in 2022-04-22T17:33:07.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[45495]: cluster 2022-04-22T17:33:05.912885+0000 mon.a (mon.0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2022-04-22T17:33:07.334 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[45495]: cluster 2022-04-22T17:33:05.912925+0000 mon.a (mon.0) 804 : cluster [WRN] Health check failed: Degraded data redundancy: 10/606 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:33:07.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[46934]: cluster 2022-04-22T17:33:05.246107+0000 mgr.y (mgr.34107) 248 : cluster [DBG] pgmap v420: 121 pgs: 1 active+undersized+degraded, 16 peering, 11 stale+active+clean, 5 active+undersized, 88 active+clean; 320 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 23 op/s; 10/606 objects degraded (1.650%) 2022-04-22T17:33:07.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[46934]: cluster 2022-04-22T17:33:05.849225+0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e489: 8 total, 7 up, 8 in 2022-04-22T17:33:07.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[46934]: cluster 2022-04-22T17:33:05.912885+0000 mon.a (mon.0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2022-04-22T17:33:07.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:06 smithi115 bash[46934]: cluster 2022-04-22T17:33:05.912925+0000 mon.a (mon.0) 804 : cluster [WRN] Health check failed: Degraded data redundancy: 10/606 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2022-04-22T17:33:07.766 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:07 smithi121 systemd[1]: Started Ceph osd.5 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:33:07.943 INFO:tasks.rados.rados.0.smithi115.stdout:2148: done (1 left) 2022-04-22T17:33:07.944 INFO:tasks.rados.rados.0.smithi115.stdout:2149: read oid 49 snap -1 2022-04-22T17:33:07.944 INFO:tasks.rados.rados.0.smithi115.stdout:2149: expect deleted 2022-04-22T17:33:07.944 INFO:tasks.rados.rados.0.smithi115.stdout:2150: delete oid 44 current snap is 217 2022-04-22T17:33:07.944 INFO:tasks.rados.rados.0.smithi115.stdout:2147: expect (ObjNum 490 snap 169 seq_num 490) 2022-04-22T17:33:07.946 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[45495]: cluster 2022-04-22T17:33:06.922268+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e490: 8 total, 7 up, 8 in 2022-04-22T17:33:07.946 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[45495]: audit 2022-04-22T17:33:07.293292+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:07.946 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[45495]: audit 2022-04-22T17:33:07.294721+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:07.946 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[45495]: audit 2022-04-22T17:33:07.296458+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:07.947 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[45495]: audit 2022-04-22T17:33:07.297589+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:08.087 INFO:tasks.rados.rados.0.smithi115.stdout:2147: done (2 left) 2022-04-22T17:33:08.087 INFO:tasks.rados.rados.0.smithi115.stdout:2149: done (1 left) 2022-04-22T17:33:08.087 INFO:tasks.rados.rados.0.smithi115.stdout:2150: done (0 left) 2022-04-22T17:33:08.087 INFO:tasks.rados.rados.0.smithi115.stdout:2151: read oid 6 snap 204 2022-04-22T17:33:08.087 INFO:tasks.rados.rados.0.smithi115.stdout:2151: expect (ObjNum 575 snap 187 seq_num 575) 2022-04-22T17:33:08.088 INFO:tasks.rados.rados.0.smithi115.stdout:2152: rollback oid 46 current snap is 217 2022-04-22T17:33:08.088 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 212 2022-04-22T17:33:08.088 INFO:tasks.rados.rados.0.smithi115.stdout:2153: setattr oid 32 current snap is 217 2022-04-22T17:33:08.091 INFO:tasks.rados.rados.0.smithi115.stdout:2152: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:33:08.091 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 998 (ObjNum 1 snap 22075 seq_num 3178473392) dirty dne 2022-04-22T17:33:08.093 INFO:tasks.rados.rados.0.smithi115.stdout:2152: done (2 left) 2022-04-22T17:33:08.093 INFO:tasks.rados.rados.0.smithi115.stdout:2154: read oid 47 snap -1 2022-04-22T17:33:08.093 INFO:tasks.rados.rados.0.smithi115.stdout:2154: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:08.093 INFO:tasks.rados.rados.0.smithi115.stdout:2155: copy_from oid 49 from oid 17 current snap is 217 2022-04-22T17:33:08.093 INFO:tasks.rados.rados.0.smithi115.stdout:2156: snap_create 2022-04-22T17:33:08.100 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1117 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:08.118 INFO:tasks.rados.rados.0.smithi115.stdout:2154: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:08.258 INFO:tasks.rados.rados.0.smithi115.stdout:2151: expect (ObjNum 575 snap 187 seq_num 575) 2022-04-22T17:33:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:07 smithi121 bash[36902]: cluster 2022-04-22T17:33:06.922268+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e490: 8 total, 7 up, 8 in 2022-04-22T17:33:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:07 smithi121 bash[36902]: audit 2022-04-22T17:33:07.293292+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:07 smithi121 bash[36902]: audit 2022-04-22T17:33:07.294721+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:07 smithi121 bash[36902]: audit 2022-04-22T17:33:07.296458+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:07 smithi121 bash[36902]: audit 2022-04-22T17:33:07.297589+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[46934]: cluster 2022-04-22T17:33:06.922268+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e490: 8 total, 7 up, 8 in 2022-04-22T17:33:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[46934]: audit 2022-04-22T17:33:07.293292+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[46934]: audit 2022-04-22T17:33:07.294721+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:08.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[46934]: audit 2022-04-22T17:33:07.296458+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:08.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:07 smithi115 bash[46934]: audit 2022-04-22T17:33:07.297589+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:2155: finishing copy_from to smithi11541303-49 2022-04-22T17:33:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1026 (ObjNum 627 snap 204 seq_num 627) dirty exists 2022-04-22T17:33:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:2155: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2151: done (4 left) 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2153: done (3 left) 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2154: done (2 left) 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2155: done (1 left) 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2156: done (0 left) 2022-04-22T17:33:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2157: snap_create 2022-04-22T17:33:09.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:08 smithi121 bash[36902]: cluster 2022-04-22T17:33:07.247432+0000 mgr.y (mgr.34107) 249 : cluster [DBG] pgmap v423: 121 pgs: 6 active+undersized+degraded, 16 peering, 8 stale+active+clean, 13 active+undersized, 78 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 64/594 objects degraded (10.774%) 2022-04-22T17:33:09.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:08 smithi121 bash[36902]: cluster 2022-04-22T17:33:07.927830+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e491: 8 total, 7 up, 8 in 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-04-22T17:33:09.269 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[41299]: --> ceph-volume raw activate successful for osd ID: 5 2022-04-22T17:33:09.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:08 smithi115 bash[45495]: cluster 2022-04-22T17:33:07.247432+0000 mgr.y (mgr.34107) 249 : cluster [DBG] pgmap v423: 121 pgs: 6 active+undersized+degraded, 16 peering, 8 stale+active+clean, 13 active+undersized, 78 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 64/594 objects degraded (10.774%) 2022-04-22T17:33:09.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:08 smithi115 bash[45495]: cluster 2022-04-22T17:33:07.927830+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e491: 8 total, 7 up, 8 in 2022-04-22T17:33:09.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:08 smithi115 bash[46934]: cluster 2022-04-22T17:33:07.247432+0000 mgr.y (mgr.34107) 249 : cluster [DBG] pgmap v423: 121 pgs: 6 active+undersized+degraded, 16 peering, 8 stale+active+clean, 13 active+undersized, 78 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 64/594 objects degraded (10.774%) 2022-04-22T17:33:09.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:08 smithi115 bash[46934]: cluster 2022-04-22T17:33:07.927830+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e491: 8 total, 7 up, 8 in 2022-04-22T17:33:09.864 INFO:tasks.rados.rados.0.smithi115.stdout:2157: done (0 left) 2022-04-22T17:33:09.864 INFO:tasks.rados.rados.0.smithi115.stdout:2158: snap_create 2022-04-22T17:33:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[36902]: cluster 2022-04-22T17:33:08.946339+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e492: 8 total, 7 up, 8 in 2022-04-22T17:33:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[36902]: audit 2022-04-22T17:33:09.274111+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:09 smithi121 bash[36902]: cluster 2022-04-22T17:33:09.848738+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e493: 8 total, 7 up, 8 in 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[45495]: cluster 2022-04-22T17:33:08.946339+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e492: 8 total, 7 up, 8 in 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[45495]: audit 2022-04-22T17:33:09.274111+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[45495]: cluster 2022-04-22T17:33:09.848738+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e493: 8 total, 7 up, 8 in 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[46934]: cluster 2022-04-22T17:33:08.946339+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e492: 8 total, 7 up, 8 in 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[46934]: audit 2022-04-22T17:33:09.274111+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:10.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:09 smithi115 bash[46934]: cluster 2022-04-22T17:33:09.848738+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e493: 8 total, 7 up, 8 in 2022-04-22T17:33:10.868 INFO:tasks.rados.rados.0.smithi115.stdout:2158: done (0 left) 2022-04-22T17:33:10.868 INFO:tasks.rados.rados.0.smithi115.stdout:2159: delete oid 45 current snap is 220 2022-04-22T17:33:10.877 INFO:tasks.rados.rados.0.smithi115.stdout:2159: done (0 left) 2022-04-22T17:33:10.877 INFO:tasks.rados.rados.0.smithi115.stdout:2160: read oid 40 snap -1 2022-04-22T17:33:10.877 INFO:tasks.rados.rados.0.smithi115.stdout:2160: expect (ObjNum 635 snap 208 seq_num 635) 2022-04-22T17:33:10.877 INFO:tasks.rados.rados.0.smithi115.stdout:2161: snap_create 2022-04-22T17:33:10.907 INFO:tasks.rados.rados.0.smithi115.stdout:2160: expect (ObjNum 635 snap 208 seq_num 635) 2022-04-22T17:33:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:10 smithi121 bash[36902]: cluster 2022-04-22T17:33:09.248682+0000 mgr.y (mgr.34107) 250 : cluster [DBG] pgmap v426: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 344 KiB/s wr, 6 op/s; 104/600 objects degraded (17.333%) 2022-04-22T17:33:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:10 smithi121 bash[36902]: cluster 2022-04-22T17:33:09.942069+0000 mon.a (mon.0) 814 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2022-04-22T17:33:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:10 smithi121 bash[36902]: cluster 2022-04-22T17:33:10.853098+0000 mon.a (mon.0) 815 : cluster [DBG] osdmap e494: 8 total, 7 up, 8 in 2022-04-22T17:33:11.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:11 smithi121 bash[41477]: debug 2022-04-22T17:33:11.167+0000 7f3741936700 1 -- 172.21.15.121:0/3500926001 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cfe0ef0160 con 0x55cfe0f34800 2022-04-22T17:33:11.267 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:11 smithi121 bash[41477]: debug 2022-04-22T17:33:11.171+0000 7f3741936700 1 -- 172.21.15.121:0/3500926001 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cfe0eb0340 con 0x55cfe0f34800 2022-04-22T17:33:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[45495]: cluster 2022-04-22T17:33:09.248682+0000 mgr.y (mgr.34107) 250 : cluster [DBG] pgmap v426: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 344 KiB/s wr, 6 op/s; 104/600 objects degraded (17.333%) 2022-04-22T17:33:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[45495]: cluster 2022-04-22T17:33:09.942069+0000 mon.a (mon.0) 814 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2022-04-22T17:33:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[45495]: cluster 2022-04-22T17:33:10.853098+0000 mon.a (mon.0) 815 : cluster [DBG] osdmap e494: 8 total, 7 up, 8 in 2022-04-22T17:33:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[46934]: cluster 2022-04-22T17:33:09.248682+0000 mgr.y (mgr.34107) 250 : cluster [DBG] pgmap v426: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 321 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 344 KiB/s wr, 6 op/s; 104/600 objects degraded (17.333%) 2022-04-22T17:33:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[46934]: cluster 2022-04-22T17:33:09.942069+0000 mon.a (mon.0) 814 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2022-04-22T17:33:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:10 smithi115 bash[46934]: cluster 2022-04-22T17:33:10.853098+0000 mon.a (mon.0) 815 : cluster [DBG] osdmap e494: 8 total, 7 up, 8 in 2022-04-22T17:33:11.871 INFO:tasks.rados.rados.0.smithi115.stdout:2160: done (1 left) 2022-04-22T17:33:11.871 INFO:tasks.rados.rados.0.smithi115.stdout:2161: done (0 left) 2022-04-22T17:33:11.871 INFO:tasks.rados.rados.0.smithi115.stdout:2162: snap_create 2022-04-22T17:33:12.016 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:11 smithi121 bash[41477]: debug 2022-04-22T17:33:11.727+0000 7f3748f2d3c0 -1 Falling back to public interface 2022-04-22T17:33:12.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:11.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:12.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:11.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:12.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:11.627Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:12.872 INFO:tasks.rados.rados.0.smithi115.stdout:2162: done (0 left) 2022-04-22T17:33:12.872 INFO:tasks.rados.rados.0.smithi115.stdout:2163: setattr oid 22 current snap is 222 2022-04-22T17:33:12.877 INFO:tasks.rados.rados.0.smithi115.stdout:2164: snap_remove snap 221 2022-04-22T17:33:13.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:12 smithi121 bash[36902]: cluster 2022-04-22T17:33:11.249746+0000 mgr.y (mgr.34107) 251 : cluster [DBG] pgmap v429: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 107/609 objects degraded (17.570%) 2022-04-22T17:33:13.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:12 smithi121 bash[36902]: cluster 2022-04-22T17:33:11.856054+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e495: 8 total, 7 up, 8 in 2022-04-22T17:33:13.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:12 smithi121 bash[36902]: cluster 2022-04-22T17:33:11.946065+0000 mon.a (mon.0) 817 : cluster [WRN] Health check update: Degraded data redundancy: 107/609 objects degraded (17.570%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:13.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[45495]: cluster 2022-04-22T17:33:11.249746+0000 mgr.y (mgr.34107) 251 : cluster [DBG] pgmap v429: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 107/609 objects degraded (17.570%) 2022-04-22T17:33:13.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[45495]: cluster 2022-04-22T17:33:11.856054+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e495: 8 total, 7 up, 8 in 2022-04-22T17:33:13.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[45495]: cluster 2022-04-22T17:33:11.946065+0000 mon.a (mon.0) 817 : cluster [WRN] Health check update: Degraded data redundancy: 107/609 objects degraded (17.570%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:13.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[46934]: cluster 2022-04-22T17:33:11.249746+0000 mgr.y (mgr.34107) 251 : cluster [DBG] pgmap v429: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 107/609 objects degraded (17.570%) 2022-04-22T17:33:13.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[46934]: cluster 2022-04-22T17:33:11.856054+0000 mon.a (mon.0) 816 : cluster [DBG] osdmap e495: 8 total, 7 up, 8 in 2022-04-22T17:33:13.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:12 smithi115 bash[46934]: cluster 2022-04-22T17:33:11.946065+0000 mon.a (mon.0) 817 : cluster [WRN] Health check update: Degraded data redundancy: 107/609 objects degraded (17.570%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:13.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:13.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:13.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:13.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:13.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:13.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:13.884 INFO:tasks.rados.rados.0.smithi115.stdout:2164: done (1 left) 2022-04-22T17:33:13.884 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 222 2022-04-22T17:33:13.884 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1093 (ObjNum 640 snap 210 seq_num 640) dirty exists 2022-04-22T17:33:13.884 INFO:tasks.rados.rados.0.smithi115.stdout:2165: seq_num 664 ranges {0=729088,729088=729088,1458176=73728} 2022-04-22T17:33:13.889 INFO:tasks.rados.rados.0.smithi115.stdout:2165: writing smithi11541303-11 from 0 to 729088 tid 1 2022-04-22T17:33:13.894 INFO:tasks.rados.rados.0.smithi115.stdout:2165: writing smithi11541303-11 from 729088 to 1458176 tid 2 2022-04-22T17:33:13.895 INFO:tasks.rados.rados.0.smithi115.stdout:2165: writing smithi11541303-11 from 1458176 to 1531904 tid 3 2022-04-22T17:33:13.895 INFO:tasks.rados.rados.0.smithi115.stdout:2163: done (1 left) 2022-04-22T17:33:13.895 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 222 2022-04-22T17:33:13.895 INFO:tasks.rados.rados.0.smithi115.stdout:2166: seq_num 665 ranges {0=745472,745472=606208,1351680=196608} 2022-04-22T17:33:13.899 INFO:tasks.rados.rados.0.smithi115.stdout:2166: writing smithi11541303-44 from 0 to 745472 tid 1 2022-04-22T17:33:13.903 INFO:tasks.rados.rados.0.smithi115.stdout:2166: writing smithi11541303-44 from 745472 to 1351680 tid 2 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2166: writing smithi11541303-44 from 1351680 to 1548288 tid 3 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2167: setattr oid 23 current snap is 222 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2165: finishing write tid 1 to smithi11541303-11 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2165: finishing write tid 2 to smithi11541303-11 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2165: finishing write tid 3 to smithi11541303-11 2022-04-22T17:33:13.905 INFO:tasks.rados.rados.0.smithi115.stdout:2165: finishing write tid 4 to smithi11541303-11 2022-04-22T17:33:13.906 INFO:tasks.rados.rados.0.smithi115.stdout:2165: finishing write tid 5 to smithi11541303-11 2022-04-22T17:33:13.906 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1017 (ObjNum 664 snap 222 seq_num 664) dirty exists 2022-04-22T17:33:13.906 INFO:tasks.rados.rados.0.smithi115.stdout:2165: left oid 11 (ObjNum 664 snap 222 seq_num 664) 2022-04-22T17:33:13.906 INFO:tasks.rados.rados.0.smithi115.stdout:2165: done (2 left) 2022-04-22T17:33:13.906 INFO:tasks.rados.rados.0.smithi115.stdout:2168: snap_remove snap 210 2022-04-22T17:33:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:13 smithi121 bash[36902]: cluster 2022-04-22T17:33:12.856921+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e496: 8 total, 7 up, 8 in 2022-04-22T17:33:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:13 smithi121 bash[36902]: audit 2022-04-22T17:33:12.871767+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:13 smithi121 bash[36902]: audit 2022-04-22T17:33:13.270728+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:33:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:13 smithi121 bash[36902]: audit 2022-04-22T17:33:13.271171+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:33:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:13 smithi121 bash[36902]: audit 2022-04-22T17:33:13.447012+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[45495]: cluster 2022-04-22T17:33:12.856921+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e496: 8 total, 7 up, 8 in 2022-04-22T17:33:14.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[45495]: audit 2022-04-22T17:33:12.871767+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[45495]: audit 2022-04-22T17:33:13.270728+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:33:14.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[45495]: audit 2022-04-22T17:33:13.271171+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:33:14.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[45495]: audit 2022-04-22T17:33:13.447012+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[46934]: cluster 2022-04-22T17:33:12.856921+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e496: 8 total, 7 up, 8 in 2022-04-22T17:33:14.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[46934]: audit 2022-04-22T17:33:12.871767+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[46934]: audit 2022-04-22T17:33:13.270728+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:33:14.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[46934]: audit 2022-04-22T17:33:13.271171+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:33:14.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:13 smithi115 bash[46934]: audit 2022-04-22T17:33:13.447012+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:2168: done (2 left) 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:2169: read oid 25 snap -1 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:2169: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:2170: delete oid 30 current snap is 222 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:2166: finishing write tid 1 to smithi11541303-44 2022-04-22T17:33:14.903 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1018 (ObjNum 500 snap 170 seq_num 500) dirty exists 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:2166: finishing write tid 2 to smithi11541303-44 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:2166: finishing write tid 3 to smithi11541303-44 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:2166: finishing write tid 4 to smithi11541303-44 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:2166: finishing write tid 5 to smithi11541303-44 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 149 (ObjNum 665 snap 222 seq_num 665) dirty exists 2022-04-22T17:33:14.904 INFO:tasks.rados.rados.0.smithi115.stdout:2166: left oid 44 (ObjNum 665 snap 222 seq_num 665) 2022-04-22T17:33:14.911 INFO:tasks.rados.rados.0.smithi115.stdout:2166: done (3 left) 2022-04-22T17:33:14.911 INFO:tasks.rados.rados.0.smithi115.stdout:2167: done (2 left) 2022-04-22T17:33:14.911 INFO:tasks.rados.rados.0.smithi115.stdout:2170: done (1 left) 2022-04-22T17:33:14.911 INFO:tasks.rados.rados.0.smithi115.stdout:2171: snap_remove snap 173 2022-04-22T17:33:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:14 smithi121 bash[36902]: cluster 2022-04-22T17:33:13.250892+0000 mgr.y (mgr.34107) 252 : cluster [DBG] pgmap v432: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:14 smithi121 bash[36902]: cluster 2022-04-22T17:33:13.868739+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e497: 8 total, 7 up, 8 in 2022-04-22T17:33:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:14 smithi121 bash[36902]: audit 2022-04-22T17:33:14.277261+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:15.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[45495]: cluster 2022-04-22T17:33:13.250892+0000 mgr.y (mgr.34107) 252 : cluster [DBG] pgmap v432: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:15.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[45495]: cluster 2022-04-22T17:33:13.868739+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e497: 8 total, 7 up, 8 in 2022-04-22T17:33:15.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[45495]: audit 2022-04-22T17:33:14.277261+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:15.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[46934]: cluster 2022-04-22T17:33:13.250892+0000 mgr.y (mgr.34107) 252 : cluster [DBG] pgmap v432: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:15.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[46934]: cluster 2022-04-22T17:33:13.868739+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e497: 8 total, 7 up, 8 in 2022-04-22T17:33:15.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:14 smithi115 bash[46934]: audit 2022-04-22T17:33:14.277261+0000 mon.a (mon.0) 824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:15.911 INFO:tasks.rados.rados.0.smithi115.stdout:2171: done (1 left) 2022-04-22T17:33:15.911 INFO:tasks.rados.rados.0.smithi115.stdout:2172: rollback oid 44 current snap is 222 2022-04-22T17:33:15.912 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 212 2022-04-22T17:33:15.912 INFO:tasks.rados.rados.0.smithi115.stdout:2169: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:33:16.149 INFO:tasks.rados.rados.0.smithi115.stdout:2172: finishing rollback tid 0 to smithi11541303-44 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:2172: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:2172: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 152 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:2169: done (1 left) 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:2172: done (0 left) 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 222 2022-04-22T17:33:16.150 INFO:tasks.rados.rados.0.smithi115.stdout:2173: seq_num 667 ranges {2154496=778240,2932736=778240,3710976=24576} 2022-04-22T17:33:16.172 INFO:tasks.rados.rados.0.smithi115.stdout:2173: writing smithi11541303-4 from 2154496 to 2932736 tid 1 2022-04-22T17:33:16.177 INFO:tasks.rados.rados.0.smithi115.stdout:2173: writing smithi11541303-4 from 2932736 to 3710976 tid 2 2022-04-22T17:33:16.177 INFO:tasks.rados.rados.0.smithi115.stdout:2173: writing smithi11541303-4 from 3710976 to 3735552 tid 3 2022-04-22T17:33:16.177 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 222 2022-04-22T17:33:16.177 INFO:tasks.rados.rados.0.smithi115.stdout:2174: seq_num 668 ranges {2539520=794624,3334144=663552,3997696=139264} 2022-04-22T17:33:16.196 INFO:tasks.rados.rados.0.smithi115.stdout:2174: writing smithi11541303-28 from 2539520 to 3334144 tid 1 2022-04-22T17:33:16.200 INFO:tasks.rados.rados.0.smithi115.stdout:2174: writing smithi11541303-28 from 3334144 to 3997696 tid 2 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2174: writing smithi11541303-28 from 3997696 to 4136960 tid 3 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2175: snap_create 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2173: finishing write tid 1 to smithi11541303-4 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2173: finishing write tid 2 to smithi11541303-4 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2173: finishing write tid 3 to smithi11541303-4 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2173: finishing write tid 4 to smithi11541303-4 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:2173: finishing write tid 5 to smithi11541303-4 2022-04-22T17:33:16.201 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1205 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:33:16.202 INFO:tasks.rados.rados.0.smithi115.stdout:2173: left oid 4 (ObjNum 667 snap 222 seq_num 667) 2022-04-22T17:33:16.205 INFO:tasks.rados.rados.0.smithi115.stdout:2174: finishing write tid 1 to smithi11541303-28 2022-04-22T17:33:16.206 INFO:tasks.rados.rados.0.smithi115.stdout:2174: finishing write tid 2 to smithi11541303-28 2022-04-22T17:33:16.206 INFO:tasks.rados.rados.0.smithi115.stdout:2174: finishing write tid 3 to smithi11541303-28 2022-04-22T17:33:16.208 INFO:tasks.rados.rados.0.smithi115.stdout:2174: finishing write tid 4 to smithi11541303-28 2022-04-22T17:33:16.209 INFO:tasks.rados.rados.0.smithi115.stdout:2174: finishing write tid 5 to smithi11541303-28 2022-04-22T17:33:16.209 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1100 (ObjNum 668 snap 222 seq_num 668) dirty exists 2022-04-22T17:33:16.209 INFO:tasks.rados.rados.0.smithi115.stdout:2174: left oid 28 (ObjNum 668 snap 222 seq_num 668) 2022-04-22T17:33:16.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:15 smithi121 bash[36902]: cluster 2022-04-22T17:33:14.887845+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e498: 8 total, 7 up, 8 in 2022-04-22T17:33:16.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:15 smithi115 bash[45495]: cluster 2022-04-22T17:33:14.887845+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e498: 8 total, 7 up, 8 in 2022-04-22T17:33:16.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:15 smithi115 bash[46934]: cluster 2022-04-22T17:33:14.887845+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e498: 8 total, 7 up, 8 in 2022-04-22T17:33:16.516 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[41477]: debug 2022-04-22T17:33:16.227+0000 7f3748f2d3c0 -1 osd.5 486 log_to_monitors true 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2173: done (2 left) 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2174: done (1 left) 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2175: done (0 left) 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2176: rollback oid 16 current snap is 223 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 217 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2177: read oid 42 snap -1 2022-04-22T17:33:16.921 INFO:tasks.rados.rados.0.smithi115.stdout:2177: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:16.922 INFO:tasks.rados.rados.0.smithi115.stdout:2178: snap_remove snap 178 2022-04-22T17:33:17.266 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[41477]: debug 2022-04-22T17:33:16.951+0000 7f373aac9700 -1 osd.5 486 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:33:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[36902]: cluster 2022-04-22T17:33:15.251905+0000 mgr.y (mgr.34107) 253 : cluster [DBG] pgmap v435: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 374 KiB/s wr, 12 op/s; 108/621 objects degraded (17.391%) 2022-04-22T17:33:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[36902]: cluster 2022-04-22T17:33:15.895624+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e499: 8 total, 7 up, 8 in 2022-04-22T17:33:17.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[36902]: audit 2022-04-22T17:33:16.223413+0000 mon.a (mon.0) 827 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:16 smithi121 bash[36902]: audit 2022-04-22T17:33:16.234444+0000 mon.b (mon.2) 5 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[45495]: cluster 2022-04-22T17:33:15.251905+0000 mgr.y (mgr.34107) 253 : cluster [DBG] pgmap v435: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 374 KiB/s wr, 12 op/s; 108/621 objects degraded (17.391%) 2022-04-22T17:33:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[45495]: cluster 2022-04-22T17:33:15.895624+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e499: 8 total, 7 up, 8 in 2022-04-22T17:33:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[45495]: audit 2022-04-22T17:33:16.223413+0000 mon.a (mon.0) 827 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[45495]: audit 2022-04-22T17:33:16.234444+0000 mon.b (mon.2) 5 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[46934]: cluster 2022-04-22T17:33:15.251905+0000 mgr.y (mgr.34107) 253 : cluster [DBG] pgmap v435: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 374 KiB/s wr, 12 op/s; 108/621 objects degraded (17.391%) 2022-04-22T17:33:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[46934]: cluster 2022-04-22T17:33:15.895624+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e499: 8 total, 7 up, 8 in 2022-04-22T17:33:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[46934]: audit 2022-04-22T17:33:16.223413+0000 mon.a (mon.0) 827 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:16 smithi115 bash[46934]: audit 2022-04-22T17:33:16.234444+0000 mon.b (mon.2) 5 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-04-22T17:33:17.936 INFO:tasks.rados.rados.0.smithi115.stdout:2178: done (2 left) 2022-04-22T17:33:17.936 INFO:tasks.rados.rados.0.smithi115.stdout:2179: rmattr oid 2 current snap is 223 2022-04-22T17:33:17.936 INFO:tasks.rados.rados.0.smithi115.stdout:2176: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:33:17.936 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1114 (ObjNum -1116710944 snap 22075 seq_num 3178314016) dirty dne 2022-04-22T17:33:17.936 INFO:tasks.rados.rados.0.smithi115.stdout:2177: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:17.951 INFO:tasks.rados.rados.0.smithi115.stdout:2176: done (2 left) 2022-04-22T17:33:17.951 INFO:tasks.rados.rados.0.smithi115.stdout:2177: done (1 left) 2022-04-22T17:33:17.951 INFO:tasks.rados.rados.0.smithi115.stdout:2180: copy_from oid 5 from oid 28 current snap is 223 2022-04-22T17:33:17.951 INFO:tasks.rados.rados.0.smithi115.stdout:2181: read oid 48 snap -1 2022-04-22T17:33:17.951 INFO:tasks.rados.rados.0.smithi115.stdout:2181: expect deleted 2022-04-22T17:33:17.952 INFO:tasks.rados.rados.0.smithi115.stdout:2182: copy_from oid 14 from oid 3 current snap is 223 2022-04-22T17:33:17.952 INFO:tasks.rados.rados.0.smithi115.stdout:2183: rmattr oid 39 current snap is 223 2022-04-22T17:33:17.952 INFO:tasks.rados.rados.0.smithi115.stdout:2181: done (4 left) 2022-04-22T17:33:17.953 INFO:tasks.rados.rados.0.smithi115.stdout:2184: copy_from oid 6 from oid 43 current snap is 223 2022-04-22T17:33:17.953 INFO:tasks.rados.rados.0.smithi115.stdout:2185: snap_create 2022-04-22T17:33:17.977 INFO:tasks.rados.rados.0.smithi115.stdout:2182: finishing copy_from to smithi11541303-14 2022-04-22T17:33:17.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1003 (ObjNum 625 snap 204 seq_num 625) dirty exists 2022-04-22T17:33:17.978 INFO:tasks.rados.rados.0.smithi115.stdout:2182: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:33:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:17 smithi121 bash[36902]: audit 2022-04-22T17:33:16.906155+0000 mon.a (mon.0) 828 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:33:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:17 smithi121 bash[36902]: cluster 2022-04-22T17:33:16.906234+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e500: 8 total, 7 up, 8 in 2022-04-22T17:33:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:17 smithi121 bash[36902]: audit 2022-04-22T17:33:16.909605+0000 mon.a (mon.0) 830 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:17 smithi121 bash[36902]: audit 2022-04-22T17:33:16.920703+0000 mon.b (mon.2) 6 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[45495]: audit 2022-04-22T17:33:16.906155+0000 mon.a (mon.0) 828 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:33:18.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[45495]: cluster 2022-04-22T17:33:16.906234+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e500: 8 total, 7 up, 8 in 2022-04-22T17:33:18.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[45495]: audit 2022-04-22T17:33:16.909605+0000 mon.a (mon.0) 830 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[45495]: audit 2022-04-22T17:33:16.920703+0000 mon.b (mon.2) 6 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[46934]: audit 2022-04-22T17:33:16.906155+0000 mon.a (mon.0) 828 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-04-22T17:33:18.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[46934]: cluster 2022-04-22T17:33:16.906234+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e500: 8 total, 7 up, 8 in 2022-04-22T17:33:18.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[46934]: audit 2022-04-22T17:33:16.909605+0000 mon.a (mon.0) 830 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:17 smithi115 bash[46934]: audit 2022-04-22T17:33:16.920703+0000 mon.b (mon.2) 6 : audit [INF] from='osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:18.940 INFO:tasks.rados.rados.0.smithi115.stdout:2182: done (5 left) 2022-04-22T17:33:18.940 INFO:tasks.rados.rados.0.smithi115.stdout:2185: done (4 left) 2022-04-22T17:33:18.940 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 224 2022-04-22T17:33:18.940 INFO:tasks.rados.rados.0.smithi115.stdout:2186: seq_num 669 ranges {2310144=409600,2719744=548864,3268608=655360} 2022-04-22T17:33:18.957 INFO:tasks.rados.rados.0.smithi115.stdout:2186: writing smithi11541303-15 from 2310144 to 2719744 tid 1 2022-04-22T17:33:18.961 INFO:tasks.rados.rados.0.smithi115.stdout:2186: writing smithi11541303-15 from 2719744 to 3268608 tid 2 2022-04-22T17:33:18.965 INFO:tasks.rados.rados.0.smithi115.stdout:2186: writing smithi11541303-15 from 3268608 to 3923968 tid 3 2022-04-22T17:33:19.014 INFO:tasks.rados.rados.0.smithi115.stdout:2187: setattr oid 32 current snap is 224 2022-04-22T17:33:19.017 INFO:tasks.rados.rados.0.smithi115.stdout:2186: finishing write tid 1 to smithi11541303-15 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2188: delete oid 3 current snap is 224 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2186: finishing write tid 2 to smithi11541303-15 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2186: finishing write tid 3 to smithi11541303-15 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2186: finishing write tid 4 to smithi11541303-15 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1103 (ObjNum -1124688696 snap 22075 seq_num 3) dirty exists 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2179: done (6 left) 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2188: done (5 left) 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2189: read oid 45 snap -1 2022-04-22T17:33:19.018 INFO:tasks.rados.rados.0.smithi115.stdout:2189: expect deleted 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2190: rollback oid 21 current snap is 224 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 204 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2191: setattr oid 35 current snap is 224 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2186: finishing write tid 5 to smithi11541303-15 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 729 (ObjNum 669 snap 224 seq_num 669) dirty exists 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2186: left oid 15 (ObjNum 669 snap 224 seq_num 669) 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2186: done (7 left) 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 224 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2192: seq_num 671 ranges {1179648=442368,1622016=761856,2383872=442368} 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2192: writing smithi11541303-14 from 1179648 to 1622016 tid 1 2022-04-22T17:33:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:2192: writing smithi11541303-14 from 1622016 to 2383872 tid 2 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2192: writing smithi11541303-14 from 2383872 to 2826240 tid 3 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2193: copy_from oid 17 from oid 25 current snap is 224 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1104 (ObjNum 651 snap 212 seq_num 651) dirty exists 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2184: finishing copy_from to smithi11541303-6 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1048 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2184: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2190: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2190: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:2190: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:33:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1096 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1105 (ObjNum 571 snap 186 seq_num 571) dirty exists 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2192: finishing write tid 1 to smithi11541303-14 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2183: done (8 left) 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2184: done (7 left) 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2189: done (6 left) 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2190: done (5 left) 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2191: done (4 left) 2022-04-22T17:33:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:2194: snap_remove snap 212 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: cluster 2022-04-22T17:33:17.252840+0000 mgr.y (mgr.34107) 254 : cluster [DBG] pgmap v438: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 337 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 10 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: cluster 2022-04-22T17:33:17.904215+0000 mon.a (mon.0) 831 : cluster [WRN] Health check update: Degraded data redundancy: 108/615 objects degraded (17.561%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: cluster 2022-04-22T17:33:17.905599+0000 mon.a (mon.0) 832 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: cluster 2022-04-22T17:33:17.920369+0000 mon.a (mon.0) 833 : cluster [INF] osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057] boot 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: cluster 2022-04-22T17:33:17.920415+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2022-04-22T17:33:19.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:18 smithi121 bash[36902]: audit 2022-04-22T17:33:17.920854+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:33:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: cluster 2022-04-22T17:33:17.252840+0000 mgr.y (mgr.34107) 254 : cluster [DBG] pgmap v438: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 337 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 10 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: cluster 2022-04-22T17:33:17.904215+0000 mon.a (mon.0) 831 : cluster [WRN] Health check update: Degraded data redundancy: 108/615 objects degraded (17.561%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: cluster 2022-04-22T17:33:17.905599+0000 mon.a (mon.0) 832 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:33:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: cluster 2022-04-22T17:33:17.920369+0000 mon.a (mon.0) 833 : cluster [INF] osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057] boot 2022-04-22T17:33:19.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: cluster 2022-04-22T17:33:17.920415+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[45495]: audit 2022-04-22T17:33:17.920854+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: cluster 2022-04-22T17:33:17.252840+0000 mgr.y (mgr.34107) 254 : cluster [DBG] pgmap v438: 121 pgs: 11 active+undersized+degraded, 39 active+undersized, 71 active+clean; 337 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 10 op/s; 108/615 objects degraded (17.561%) 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: cluster 2022-04-22T17:33:17.904215+0000 mon.a (mon.0) 831 : cluster [WRN] Health check update: Degraded data redundancy: 108/615 objects degraded (17.561%), 11 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: cluster 2022-04-22T17:33:17.905599+0000 mon.a (mon.0) 832 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: cluster 2022-04-22T17:33:17.920369+0000 mon.a (mon.0) 833 : cluster [INF] osd.5 [v2:172.21.15.121:6808/3650203057,v1:172.21.15.121:6809/3650203057] boot 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: cluster 2022-04-22T17:33:17.920415+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2022-04-22T17:33:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:18 smithi115 bash[46934]: audit 2022-04-22T17:33:17.920854+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2194: done (4 left) 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2195: snap_create 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2192: finishing write tid 2 to smithi11541303-14 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2192: finishing write tid 3 to smithi11541303-14 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2192: finishing write tid 4 to smithi11541303-14 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:2192: finishing write tid 5 to smithi11541303-14 2022-04-22T17:33:19.950 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1104 (ObjNum 671 snap 224 seq_num 671) dirty exists 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:2192: left oid 14 (ObjNum 671 snap 224 seq_num 671) 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:2180: finishing copy_from to smithi11541303-5 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1100 (ObjNum 668 snap 222 seq_num 668) dirty exists 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:2180: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:2193: finishing copy_from to smithi11541303-17 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 998 (ObjNum 611 snap 200 seq_num 611) dirty exists 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:2193: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:33:19.951 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1118 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:19 smithi121 bash[36902]: cluster 2022-04-22T17:33:18.924110+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2022-04-22T17:33:20.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:19 smithi121 bash[36902]: audit 2022-04-22T17:33:19.280477+0000 mon.a (mon.0) 837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:20.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:19 smithi115 bash[45495]: cluster 2022-04-22T17:33:18.924110+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2022-04-22T17:33:20.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:19 smithi115 bash[45495]: audit 2022-04-22T17:33:19.280477+0000 mon.a (mon.0) 837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:19 smithi115 bash[46934]: cluster 2022-04-22T17:33:18.924110+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2022-04-22T17:33:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:19 smithi115 bash[46934]: audit 2022-04-22T17:33:19.280477+0000 mon.a (mon.0) 837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:20.954 INFO:tasks.rados.rados.0.smithi115.stdout:2180: done (4 left) 2022-04-22T17:33:20.954 INFO:tasks.rados.rados.0.smithi115.stdout:2187: done (3 left) 2022-04-22T17:33:20.954 INFO:tasks.rados.rados.0.smithi115.stdout:2192: done (2 left) 2022-04-22T17:33:20.955 INFO:tasks.rados.rados.0.smithi115.stdout:2193: done (1 left) 2022-04-22T17:33:20.955 INFO:tasks.rados.rados.0.smithi115.stdout:2195: done (0 left) 2022-04-22T17:33:20.955 INFO:tasks.rados.rados.0.smithi115.stdout:2196: rollback oid 29 current snap is 225 2022-04-22T17:33:20.955 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 220 2022-04-22T17:33:20.955 INFO:tasks.rados.rados.0.smithi115.stdout:2197: snap_remove snap 204 2022-04-22T17:33:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:20 smithi121 bash[36902]: cluster 2022-04-22T17:33:19.253914+0000 mgr.y (mgr.34107) 255 : cluster [DBG] pgmap v441: 121 pgs: 7 peering, 6 active+undersized+degraded, 25 active+undersized, 83 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s; 68/597 objects degraded (11.390%) 2022-04-22T17:33:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:20 smithi121 bash[36902]: cluster 2022-04-22T17:33:19.933541+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2022-04-22T17:33:21.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:20 smithi115 bash[46934]: cluster 2022-04-22T17:33:19.253914+0000 mgr.y (mgr.34107) 255 : cluster [DBG] pgmap v441: 121 pgs: 7 peering, 6 active+undersized+degraded, 25 active+undersized, 83 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s; 68/597 objects degraded (11.390%) 2022-04-22T17:33:21.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:20 smithi115 bash[46934]: cluster 2022-04-22T17:33:19.933541+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2022-04-22T17:33:21.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:20 smithi115 bash[45495]: cluster 2022-04-22T17:33:19.253914+0000 mgr.y (mgr.34107) 255 : cluster [DBG] pgmap v441: 121 pgs: 7 peering, 6 active+undersized+degraded, 25 active+undersized, 83 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s; 68/597 objects degraded (11.390%) 2022-04-22T17:33:21.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:20 smithi115 bash[45495]: cluster 2022-04-22T17:33:19.933541+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2022-04-22T17:33:21.405 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (9m) 91s ago 10m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (9m) 8s ago 9m 24.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (11m) 91s ago 11m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (11m) 8s ago 11m 19.4M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5m) 8s ago 15m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6m) 91s ago 17m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:33:21.754 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5m) 91s ago 17m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4m) 8s ago 15m 49.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4m) 91s ago 15m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (10m) 91s ago 10m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (10m) 8s ago 10m 9451k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3m) 91s ago 15m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2m) 91s ago 14m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2m) 91s ago 14m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (93s) 91s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (52s) 8s ago 13m 181M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (11s) 8s ago 12m - 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:33:21.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (12m) 8s ago 12m 219M 2526M 16.2.5 46cf6318c64e 46cd96a030c2 2022-04-22T17:33:21.756 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (11m) 8s ago 11m 252M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:33:21.756 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (6m) 8s ago 10m 40.6M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2196: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2196: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2196: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1107 (ObjNum 592 snap 194 seq_num 592) dirty exists 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2196: done (1 left) 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2197: done (0 left) 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2198: rollback oid 44 current snap is 225 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 216 2022-04-22T17:33:21.957 INFO:tasks.rados.rados.0.smithi115.stdout:2199: delete oid 11 current snap is 225 2022-04-22T17:33:21.970 INFO:tasks.rados.rados.0.smithi115.stdout:2199: done (1 left) 2022-04-22T17:33:21.971 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 225 2022-04-22T17:33:21.971 INFO:tasks.rados.rados.0.smithi115.stdout:2200: seq_num 673 ranges {6946816=475136,7421952=532480,7954432=671744} 2022-04-22T17:33:21.978 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:21.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:22.095 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:21.627Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:22.096 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:21.627Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:22.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[45495]: cluster 2022-04-22T17:33:20.938945+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2022-04-22T17:33:22.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[45495]: cluster 2022-04-22T17:33:21.941382+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2200: writing smithi11541303-49 from 6946816 to 7421952 tid 1 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2200: writing smithi11541303-49 from 7421952 to 7954432 tid 2 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2200: writing smithi11541303-49 from 7954432 to 8626176 tid 3 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2201: rmattr oid 12 current snap is 225 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2201: done (2 left) 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2202: rmattr oid 10 current snap is 225 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2202: done (2 left) 2022-04-22T17:33:22.098 INFO:tasks.rados.rados.0.smithi115.stdout:2203: copy_from oid 23 from oid 36 current snap is 225 2022-04-22T17:33:22.099 INFO:tasks.rados.rados.0.smithi115.stdout:2198: finishing rollback tid 0 to smithi11541303-44 2022-04-22T17:33:22.099 INFO:tasks.rados.rados.0.smithi115.stdout:2204: copy_from oid 11 from oid 50 current snap is 225 2022-04-22T17:33:22.099 INFO:tasks.rados.rados.0.smithi115.stdout:2198: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:33:22.099 INFO:tasks.rados.rados.0.smithi115.stdout:2205: snap_remove snap 189 2022-04-22T17:33:22.182 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2, 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 6 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:22.183 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4, 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 11 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:33:22.184 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:33:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:21 smithi121 bash[36902]: cluster 2022-04-22T17:33:20.938945+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2022-04-22T17:33:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:21 smithi121 bash[36902]: cluster 2022-04-22T17:33:21.941382+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2022-04-22T17:33:22.302 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:33:22 smithi115 bash[42914]: debug 2022-04-22T17:33:22.122+0000 7f8eea62c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2022-04-22T17:33:22.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[46934]: cluster 2022-04-22T17:33:20.938945+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2022-04-22T17:33:22.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:21 smithi115 bash[46934]: cluster 2022-04-22T17:33:21.941382+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2022-04-22T17:33:22.989 INFO:tasks.rados.rados.0.smithi115.stdout:2205: done (4 left) 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:2206: rollback oid 1 current snap is 225 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 1 to 217 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:2207: rollback oid 42 current snap is 225 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 220 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:2208: read oid 47 snap -1 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:2208: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:23.063 INFO:tasks.rados.rados.0.smithi115.stdout:2209: setattr oid 6 current snap is 225 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2210: rollback oid 28 current snap is 225 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2198: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 164 (ObjNum 546 snap 178 seq_num 546) dirty exists 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2200: finishing write tid 1 to smithi11541303-49 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2200: finishing write tid 2 to smithi11541303-49 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2200: finishing write tid 3 to smithi11541303-49 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2200: finishing write tid 4 to smithi11541303-49 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:2200: finishing write tid 5 to smithi11541303-49 2022-04-22T17:33:23.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1403 (ObjNum 673 snap 225 seq_num 673) dirty exists 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2200: left oid 49 (ObjNum 673 snap 225 seq_num 673) 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2203: finishing copy_from to smithi11541303-23 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1194 (ObjNum 521 snap 172 seq_num 521) dirty exists 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2203: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2204: finishing copy_from to smithi11541303-11 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1109 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2204: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 213 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2198: done (8 left) 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2200: done (7 left) 2022-04-22T17:33:23.067 INFO:tasks.rados.rados.0.smithi115.stdout:2203: done (6 left) 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2204: done (5 left) 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 225 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2211: seq_num 674 ranges {7397376=491520,7888896=417792,8306688=483328,8790016=303104} 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2211: writing smithi11541303-35 from 7397376 to 7888896 tid 1 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2211: writing smithi11541303-35 from 7888896 to 8306688 tid 2 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2211: writing smithi11541303-35 from 8306688 to 8790016 tid 3 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2211: writing smithi11541303-35 from 8790016 to 9093120 tid 4 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2212: read oid 34 snap 179 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2212: expect (ObjNum -1116699792 snap 22075 seq_num 3178122832) 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 225 2022-04-22T17:33:23.068 INFO:tasks.rados.rados.0.smithi115.stdout:2213: seq_num 675 ranges {8839168=507904,9347072=704512,10051584=499712} 2022-04-22T17:33:23.100 INFO:tasks.rados.rados.0.smithi115.stdout:2213: writing smithi11541303-21 from 8839168 to 9347072 tid 1 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:2213: writing smithi11541303-21 from 9347072 to 10051584 tid 2 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:2213: writing smithi11541303-21 from 10051584 to 10551296 tid 3 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:2214: rollback oid 45 current snap is 225 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 207 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:2210: finishing rollback tid 0 to smithi11541303-28 2022-04-22T17:33:23.133 INFO:tasks.rados.rados.0.smithi115.stdout:2206: finishing rollback tid 1 to smithi11541303-1 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1195 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1072 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2210: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2210: finishing rollback tid 2 to smithi11541303-28 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1110 (ObjNum 657 snap 213 seq_num 657) dirty exists 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2207: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2207: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2207: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1407 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:33:23.134 INFO:tasks.rados.rados.0.smithi115.stdout:2208: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:23.241 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 1 to smithi11541303-35 2022-04-22T17:33:23.241 INFO:tasks.rados.rados.0.smithi115.stdout:2212: expect (ObjNum -1116699792 snap 22075 seq_num 3178122832) 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 2 to smithi11541303-35 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 3 to smithi11541303-35 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 4 to smithi11541303-35 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 5 to smithi11541303-35 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: finishing write tid 6 to smithi11541303-35 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1110 (ObjNum 674 snap 225 seq_num 674) dirty exists 2022-04-22T17:33:23.243 INFO:tasks.rados.rados.0.smithi115.stdout:2211: left oid 35 (ObjNum 674 snap 225 seq_num 674) 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: finishing write tid 1 to smithi11541303-21 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: finishing write tid 2 to smithi11541303-21 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: finishing write tid 3 to smithi11541303-21 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2214: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: finishing write tid 4 to smithi11541303-21 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: finishing write tid 5 to smithi11541303-21 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1100 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2213: left oid 21 (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:2214: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:33:23.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1207 (ObjNum 623 snap 204 seq_num 623) dirty exists 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2206: done (8 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2207: done (7 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2208: done (6 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2209: done (5 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2210: done (4 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2211: done (3 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2212: done (2 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2213: done (1 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2214: done (0 left) 2022-04-22T17:33:23.245 INFO:tasks.rados.rados.0.smithi115.stdout:2215: delete oid 36 current snap is 225 2022-04-22T17:33:23.248 INFO:tasks.rados.rados.0.smithi115.stdout:2215: done (0 left) 2022-04-22T17:33:23.248 INFO:tasks.rados.rados.0.smithi115.stdout:2216: rollback oid 25 current snap is 225 2022-04-22T17:33:23.248 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 224 2022-04-22T17:33:23.248 INFO:tasks.rados.rados.0.smithi115.stdout:2217: snap_remove snap 216 2022-04-22T17:33:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: cluster 2022-04-22T17:33:21.254749+0000 mgr.y (mgr.34107) 256 : cluster [DBG] pgmap v444: 121 pgs: 7 peering, 4 active+undersized+degraded, 20 active+undersized, 90 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s; 53/597 objects degraded (8.878%); 3.5 MiB/s, 1 objects/s recovering 2022-04-22T17:33:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:21.375682+0000 mgr.y (mgr.34107) 257 : audit [DBG] from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:21.719256+0000 mgr.y (mgr.34107) 258 : audit [DBG] from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: cephadm 2022-04-22T17:33:22.089136+0000 mgr.y (mgr.34107) 259 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.098089+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.105947+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.116083+0000 mon.a (mon.0) 843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.119012+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.121227+0000 mon.a (mon.0) 845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.122649+0000 mon.a (mon.0) 846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.124913+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: audit 2022-04-22T17:33:22.168298+0000 mon.a (mon.0) 848 : audit [DBG] from='client.? 172.21.15.115:0/2126341548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:23 smithi121 bash[36902]: cluster 2022-04-22T17:33:22.973710+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: cluster 2022-04-22T17:33:21.254749+0000 mgr.y (mgr.34107) 256 : cluster [DBG] pgmap v444: 121 pgs: 7 peering, 4 active+undersized+degraded, 20 active+undersized, 90 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s; 53/597 objects degraded (8.878%); 3.5 MiB/s, 1 objects/s recovering 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:21.375682+0000 mgr.y (mgr.34107) 257 : audit [DBG] from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:21.719256+0000 mgr.y (mgr.34107) 258 : audit [DBG] from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: cephadm 2022-04-22T17:33:22.089136+0000 mgr.y (mgr.34107) 259 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.098089+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.105947+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.116083+0000 mon.a (mon.0) 843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.119012+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.121227+0000 mon.a (mon.0) 845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.122649+0000 mon.a (mon.0) 846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.124913+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: audit 2022-04-22T17:33:22.168298+0000 mon.a (mon.0) 848 : audit [DBG] from='client.? 172.21.15.115:0/2126341548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[45495]: cluster 2022-04-22T17:33:22.973710+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: cluster 2022-04-22T17:33:21.254749+0000 mgr.y (mgr.34107) 256 : cluster [DBG] pgmap v444: 121 pgs: 7 peering, 4 active+undersized+degraded, 20 active+undersized, 90 active+clean; 340 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s; 53/597 objects degraded (8.878%); 3.5 MiB/s, 1 objects/s recovering 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:21.375682+0000 mgr.y (mgr.34107) 257 : audit [DBG] from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:21.719256+0000 mgr.y (mgr.34107) 258 : audit [DBG] from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: cephadm 2022-04-22T17:33:22.089136+0000 mgr.y (mgr.34107) 259 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.098089+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.105947+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.116083+0000 mon.a (mon.0) 843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.119012+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.121227+0000 mon.a (mon.0) 845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.122649+0000 mon.a (mon.0) 846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.124913+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: audit 2022-04-22T17:33:22.168298+0000 mon.a (mon.0) 848 : audit [DBG] from='client.? 172.21.15.115:0/2126341548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[46934]: cluster 2022-04-22T17:33:22.973710+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2022-04-22T17:33:23.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:23.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:23.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:23.522Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:23.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:23.522Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2217: done (1 left) 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2218: read oid 35 snap -1 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2218: expect (ObjNum 674 snap 225 seq_num 674) 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2216: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2216: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:33:24.126 INFO:tasks.rados.rados.0.smithi115.stdout:2216: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1143 (ObjNum 611 snap 200 seq_num 611) dirty exists 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2216: done (1 left) 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2219: read oid 29 snap -1 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2219: expect (ObjNum 592 snap 194 seq_num 592) 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2220: read oid 26 snap -1 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2220: expect (ObjNum 661 snap 216 seq_num 661) 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2221: rollback oid 36 current snap is 225 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 211 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2222: rmattr oid 2 current snap is 225 2022-04-22T17:33:24.127 INFO:tasks.rados.rados.0.smithi115.stdout:2223: delete oid 32 current snap is 225 2022-04-22T17:33:24.145 INFO:tasks.rados.rados.0.smithi115.stdout:2221: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:33:24.146 INFO:tasks.rados.rados.0.smithi115.stdout:2221: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:33:24.146 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1202 (ObjNum 521 snap 172 seq_num 521) dirty exists 2022-04-22T17:33:24.149 INFO:tasks.rados.rados.0.smithi115.stdout:2219: expect (ObjNum 592 snap 194 seq_num 592) 2022-04-22T17:33:24.277 INFO:tasks.rados.rados.0.smithi115.stdout:2220: expect (ObjNum 661 snap 216 seq_num 661) 2022-04-22T17:33:24.379 INFO:tasks.rados.rados.0.smithi115.stdout:2218: expect (ObjNum 674 snap 225 seq_num 674) 2022-04-22T17:33:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:24 smithi121 bash[36902]: audit 2022-04-22T17:33:22.125122+0000 mgr.y (mgr.34107) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:24 smithi121 bash[36902]: cephadm 2022-04-22T17:33:22.126281+0000 mgr.y (mgr.34107) 261 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2022-04-22T17:33:24.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:24 smithi115 bash[46934]: audit 2022-04-22T17:33:22.125122+0000 mgr.y (mgr.34107) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:24.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:24 smithi115 bash[46934]: cephadm 2022-04-22T17:33:22.126281+0000 mgr.y (mgr.34107) 261 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2022-04-22T17:33:24.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:24 smithi115 bash[45495]: audit 2022-04-22T17:33:22.125122+0000 mgr.y (mgr.34107) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:24.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:24 smithi115 bash[45495]: cephadm 2022-04-22T17:33:22.126281+0000 mgr.y (mgr.34107) 261 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2022-04-22T17:33:24.937 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1111 (ObjNum -1115774864 snap 22075 seq_num 3178785888) dirty exists 2022-04-22T17:33:24.937 INFO:tasks.rados.rados.0.smithi115.stdout:2218: done (5 left) 2022-04-22T17:33:24.937 INFO:tasks.rados.rados.0.smithi115.stdout:2219: done (4 left) 2022-04-22T17:33:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2220: done (3 left) 2022-04-22T17:33:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2221: done (2 left) 2022-04-22T17:33:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2222: done (1 left) 2022-04-22T17:33:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2223: done (0 left) 2022-04-22T17:33:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2224: delete oid 11 current snap is 225 2022-04-22T17:33:24.940 INFO:tasks.rados.rados.0.smithi115.stdout:2224: done (0 left) 2022-04-22T17:33:24.940 INFO:tasks.rados.rados.0.smithi115.stdout:2225: snap_create 2022-04-22T17:33:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: cluster 2022-04-22T17:33:23.255688+0000 mgr.y (mgr.34107) 262 : cluster [DBG] pgmap v447: 121 pgs: 1 active+clean+snaptrim, 3 peering, 117 active+clean; 320 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.7 MiB/s wr, 17 op/s; 6.1 MiB/s, 1 objects/s recovering 2022-04-22T17:33:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: cluster 2022-04-22T17:33:24.096780+0000 mon.a (mon.0) 850 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/597 objects degraded (8.878%), 4 pgs degraded) 2022-04-22T17:33:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: cluster 2022-04-22T17:33:24.096850+0000 mon.a (mon.0) 851 : cluster [INF] Cluster is now healthy 2022-04-22T17:33:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: cluster 2022-04-22T17:33:24.109626+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2022-04-22T17:33:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: audit 2022-04-22T17:33:24.283602+0000 mon.a (mon.0) 853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:25 smithi121 bash[36902]: cluster 2022-04-22T17:33:24.854838+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: cluster 2022-04-22T17:33:23.255688+0000 mgr.y (mgr.34107) 262 : cluster [DBG] pgmap v447: 121 pgs: 1 active+clean+snaptrim, 3 peering, 117 active+clean; 320 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.7 MiB/s wr, 17 op/s; 6.1 MiB/s, 1 objects/s recovering 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: cluster 2022-04-22T17:33:24.096780+0000 mon.a (mon.0) 850 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/597 objects degraded (8.878%), 4 pgs degraded) 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: cluster 2022-04-22T17:33:24.096850+0000 mon.a (mon.0) 851 : cluster [INF] Cluster is now healthy 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: cluster 2022-04-22T17:33:24.109626+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: audit 2022-04-22T17:33:24.283602+0000 mon.a (mon.0) 853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:25.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[45495]: cluster 2022-04-22T17:33:24.854838+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: cluster 2022-04-22T17:33:23.255688+0000 mgr.y (mgr.34107) 262 : cluster [DBG] pgmap v447: 121 pgs: 1 active+clean+snaptrim, 3 peering, 117 active+clean; 320 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.7 MiB/s wr, 17 op/s; 6.1 MiB/s, 1 objects/s recovering 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: cluster 2022-04-22T17:33:24.096780+0000 mon.a (mon.0) 850 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/597 objects degraded (8.878%), 4 pgs degraded) 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: cluster 2022-04-22T17:33:24.096850+0000 mon.a (mon.0) 851 : cluster [INF] Cluster is now healthy 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: cluster 2022-04-22T17:33:24.109626+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: audit 2022-04-22T17:33:24.283602+0000 mon.a (mon.0) 853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:25.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:25 smithi115 bash[46934]: cluster 2022-04-22T17:33:24.854838+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2022-04-22T17:33:25.880 INFO:tasks.rados.rados.0.smithi115.stdout:2225: done (0 left) 2022-04-22T17:33:25.880 INFO:tasks.rados.rados.0.smithi115.stdout:2226: rmattr oid 43 current snap is 226 2022-04-22T17:33:25.880 INFO:tasks.rados.rados.0.smithi115.stdout:2226: done (0 left) 2022-04-22T17:33:25.880 INFO:tasks.rados.rados.0.smithi115.stdout:2227: read oid 11 snap -1 2022-04-22T17:33:25.880 INFO:tasks.rados.rados.0.smithi115.stdout:2227: expect deleted 2022-04-22T17:33:25.881 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 226 2022-04-22T17:33:25.881 INFO:tasks.rados.rados.0.smithi115.stdout:2228: seq_num 679 ranges {1531904=581632,2113536=688128,2801664=516096} 2022-04-22T17:33:25.900 INFO:tasks.rados.rados.0.smithi115.stdout:2228: writing smithi11541303-23 from 1531904 to 2113536 tid 1 2022-04-22T17:33:25.903 INFO:tasks.rados.rados.0.smithi115.stdout:2228: writing smithi11541303-23 from 2113536 to 2801664 tid 2 2022-04-22T17:33:25.907 INFO:tasks.rados.rados.0.smithi115.stdout:2228: writing smithi11541303-23 from 2801664 to 3317760 tid 3 2022-04-22T17:33:25.907 INFO:tasks.rados.rados.0.smithi115.stdout:2229: read oid 2 snap -1 2022-04-22T17:33:25.907 INFO:tasks.rados.rados.0.smithi115.stdout:2229: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:33:25.907 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 226 2022-04-22T17:33:25.907 INFO:tasks.rados.rados.0.smithi115.stdout:2230: seq_num 680 ranges {5537792=598016,6135808=573440,6709248=434176,7143424=196608} 2022-04-22T17:33:25.942 INFO:tasks.rados.rados.0.smithi115.stdout:2230: writing smithi11541303-40 from 5537792 to 6135808 tid 1 2022-04-22T17:33:25.945 INFO:tasks.rados.rados.0.smithi115.stdout:2230: writing smithi11541303-40 from 6135808 to 6709248 tid 2 2022-04-22T17:33:25.948 INFO:tasks.rados.rados.0.smithi115.stdout:2230: writing smithi11541303-40 from 6709248 to 7143424 tid 3 2022-04-22T17:33:25.949 INFO:tasks.rados.rados.0.smithi115.stdout:2230: writing smithi11541303-40 from 7143424 to 7340032 tid 4 2022-04-22T17:33:25.949 INFO:tasks.rados.rados.0.smithi115.stdout:2231: snap_create 2022-04-22T17:33:25.949 INFO:tasks.rados.rados.0.smithi115.stdout:2229: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:33:25.953 INFO:tasks.rados.rados.0.smithi115.stdout:2228: finishing write tid 1 to smithi11541303-23 2022-04-22T17:33:25.953 INFO:tasks.rados.rados.0.smithi115.stdout:2228: finishing write tid 2 to smithi11541303-23 2022-04-22T17:33:25.954 INFO:tasks.rados.rados.0.smithi115.stdout:2228: finishing write tid 3 to smithi11541303-23 2022-04-22T17:33:25.954 INFO:tasks.rados.rados.0.smithi115.stdout:2228: finishing write tid 4 to smithi11541303-23 2022-04-22T17:33:25.954 INFO:tasks.rados.rados.0.smithi115.stdout:2228: finishing write tid 5 to smithi11541303-23 2022-04-22T17:33:25.954 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1199 (ObjNum 679 snap 226 seq_num 679) dirty exists 2022-04-22T17:33:25.954 INFO:tasks.rados.rados.0.smithi115.stdout:2228: left oid 23 (ObjNum 679 snap 226 seq_num 679) 2022-04-22T17:33:25.955 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 1 to smithi11541303-40 2022-04-22T17:33:25.955 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 2 to smithi11541303-40 2022-04-22T17:33:25.958 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 3 to smithi11541303-40 2022-04-22T17:33:25.959 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 4 to smithi11541303-40 2022-04-22T17:33:25.959 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 5 to smithi11541303-40 2022-04-22T17:33:25.961 INFO:tasks.rados.rados.0.smithi115.stdout:2230: finishing write tid 6 to smithi11541303-40 2022-04-22T17:33:25.962 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1138 (ObjNum 680 snap 226 seq_num 680) dirty exists 2022-04-22T17:33:25.962 INFO:tasks.rados.rados.0.smithi115.stdout:2230: left oid 40 (ObjNum 680 snap 226 seq_num 680) 2022-04-22T17:33:26.887 INFO:tasks.rados.rados.0.smithi115.stdout:2227: done (4 left) 2022-04-22T17:33:26.888 INFO:tasks.rados.rados.0.smithi115.stdout:2228: done (3 left) 2022-04-22T17:33:26.888 INFO:tasks.rados.rados.0.smithi115.stdout:2229: done (2 left) 2022-04-22T17:33:26.888 INFO:tasks.rados.rados.0.smithi115.stdout:2230: done (1 left) 2022-04-22T17:33:26.888 INFO:tasks.rados.rados.0.smithi115.stdout:2231: done (0 left) 2022-04-22T17:33:26.889 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 227 2022-04-22T17:33:26.889 INFO:tasks.rados.rados.0.smithi115.stdout:2232: seq_num 681 ranges {163840=614400,778240=458752,1236992=638976,1875968=106496} 2022-04-22T17:33:26.892 INFO:tasks.rados.rados.0.smithi115.stdout:2232: writing smithi11541303-43 from 163840 to 778240 tid 1 2022-04-22T17:33:26.895 INFO:tasks.rados.rados.0.smithi115.stdout:2232: writing smithi11541303-43 from 778240 to 1236992 tid 2 2022-04-22T17:33:26.900 INFO:tasks.rados.rados.0.smithi115.stdout:2232: writing smithi11541303-43 from 1236992 to 1875968 tid 3 2022-04-22T17:33:26.903 INFO:tasks.rados.rados.0.smithi115.stdout:2232: writing smithi11541303-43 from 1875968 to 1982464 tid 4 2022-04-22T17:33:26.903 INFO:tasks.rados.rados.0.smithi115.stdout:2233: snap_remove snap 218 2022-04-22T17:33:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:26 smithi121 bash[36902]: cluster 2022-04-22T17:33:25.256535+0000 mgr.y (mgr.34107) 263 : cluster [DBG] pgmap v450: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 326 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s; 2.6 MiB/s, 0 objects/s recovering 2022-04-22T17:33:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:26 smithi121 bash[36902]: cluster 2022-04-22T17:33:25.864920+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2022-04-22T17:33:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:26 smithi115 bash[45495]: cluster 2022-04-22T17:33:25.256535+0000 mgr.y (mgr.34107) 263 : cluster [DBG] pgmap v450: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 326 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s; 2.6 MiB/s, 0 objects/s recovering 2022-04-22T17:33:27.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:26 smithi115 bash[45495]: cluster 2022-04-22T17:33:25.864920+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2022-04-22T17:33:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:26 smithi115 bash[46934]: cluster 2022-04-22T17:33:25.256535+0000 mgr.y (mgr.34107) 263 : cluster [DBG] pgmap v450: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 326 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s; 2.6 MiB/s, 0 objects/s recovering 2022-04-22T17:33:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:26 smithi115 bash[46934]: cluster 2022-04-22T17:33:25.864920+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2022-04-22T17:33:27.898 INFO:tasks.rados.rados.0.smithi115.stdout:2233: done (1 left) 2022-04-22T17:33:27.898 INFO:tasks.rados.rados.0.smithi115.stdout:2234: copy_from oid 29 from oid 6 current snap is 227 2022-04-22T17:33:27.898 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 1 to smithi11541303-43 2022-04-22T17:33:27.899 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 2 to smithi11541303-43 2022-04-22T17:33:27.899 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 3 to smithi11541303-43 2022-04-22T17:33:27.899 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 4 to smithi11541303-43 2022-04-22T17:33:27.899 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 5 to smithi11541303-43 2022-04-22T17:33:27.899 INFO:tasks.rados.rados.0.smithi115.stdout:2232: finishing write tid 6 to smithi11541303-43 2022-04-22T17:33:27.900 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1153 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:33:27.900 INFO:tasks.rados.rados.0.smithi115.stdout:2232: left oid 43 (ObjNum 681 snap 227 seq_num 681) 2022-04-22T17:33:27.900 INFO:tasks.rados.rados.0.smithi115.stdout:2232: done (1 left) 2022-04-22T17:33:27.900 INFO:tasks.rados.rados.0.smithi115.stdout:2235: copy_from oid 23 from oid 16 current snap is 227 2022-04-22T17:33:27.900 INFO:tasks.rados.rados.0.smithi115.stdout:2236: delete oid 10 current snap is 227 2022-04-22T17:33:27.908 INFO:tasks.rados.rados.0.smithi115.stdout:2235: finishing copy_from to smithi11541303-23 2022-04-22T17:33:27.909 INFO:tasks.rados.rados.0.smithi115.stdout:2235: got expected ENOENT (src dne) 2022-04-22T17:33:27.912 INFO:tasks.rados.rados.0.smithi115.stdout:2235: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:33:27.913 INFO:tasks.rados.rados.0.smithi115.stdout:2235: done (2 left) 2022-04-22T17:33:27.913 INFO:tasks.rados.rados.0.smithi115.stdout:2236: done (1 left) 2022-04-22T17:33:27.914 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 227 2022-04-22T17:33:27.914 INFO:tasks.rados.rados.0.smithi115.stdout:2237: seq_num 683 ranges {253952=647168,901120=630784,1531904=573440} 2022-04-22T17:33:27.920 INFO:tasks.rados.rados.0.smithi115.stdout:2237: writing smithi11541303-42 from 253952 to 901120 tid 1 2022-04-22T17:33:27.925 INFO:tasks.rados.rados.0.smithi115.stdout:2237: writing smithi11541303-42 from 901120 to 1531904 tid 2 2022-04-22T17:33:27.929 INFO:tasks.rados.rados.0.smithi115.stdout:2237: writing smithi11541303-42 from 1531904 to 2105344 tid 3 2022-04-22T17:33:27.929 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 227 2022-04-22T17:33:27.929 INFO:tasks.rados.rados.0.smithi115.stdout:2234: finishing copy_from to smithi11541303-29 2022-04-22T17:33:27.929 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1072 (ObjNum 296 snap 99 seq_num 296) dirty exists 2022-04-22T17:33:27.930 INFO:tasks.rados.rados.0.smithi115.stdout:2238: seq_num 684 ranges {1515520=663552,2179072=507904,2686976=409600,3096576=286720} 2022-04-22T17:33:27.941 INFO:tasks.rados.rados.0.smithi115.stdout:2238: writing smithi11541303-8 from 1515520 to 2179072 tid 1 2022-04-22T17:33:27.944 INFO:tasks.rados.rados.0.smithi115.stdout:2238: writing smithi11541303-8 from 2179072 to 2686976 tid 2 2022-04-22T17:33:27.947 INFO:tasks.rados.rados.0.smithi115.stdout:2238: writing smithi11541303-8 from 2686976 to 3096576 tid 3 2022-04-22T17:33:27.950 INFO:tasks.rados.rados.0.smithi115.stdout:2238: writing smithi11541303-8 from 3096576 to 3383296 tid 4 2022-04-22T17:33:27.951 INFO:tasks.rados.rados.0.smithi115.stdout:2239: read oid 17 snap -1 2022-04-22T17:33:27.951 INFO:tasks.rados.rados.0.smithi115.stdout:2239: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:33:27.951 INFO:tasks.rados.rados.0.smithi115.stdout:2240: copy_from oid 36 from oid 3 current snap is 227 2022-04-22T17:33:27.951 INFO:tasks.rados.rados.0.smithi115.stdout:2234: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:33:27.952 INFO:tasks.rados.rados.0.smithi115.stdout:2237: finishing write tid 1 to smithi11541303-42 2022-04-22T17:33:27.952 INFO:tasks.rados.rados.0.smithi115.stdout:2234: done (4 left) 2022-04-22T17:33:27.952 INFO:tasks.rados.rados.0.smithi115.stdout:2241: read oid 19 snap -1 2022-04-22T17:33:27.952 INFO:tasks.rados.rados.0.smithi115.stdout:2241: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:27.952 INFO:tasks.rados.rados.0.smithi115.stdout:2242: read oid 23 snap 217 2022-04-22T17:33:27.953 INFO:tasks.rados.rados.0.smithi115.stdout:2242: expect (ObjNum 500 snap 170 seq_num 500) 2022-04-22T17:33:27.953 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 227 2022-04-22T17:33:27.953 INFO:tasks.rados.rados.0.smithi115.stdout:2243: seq_num 685 ranges {3710976=679936,4390912=442368,4833280=622592,5455872=139264} 2022-04-22T17:33:27.976 INFO:tasks.rados.rados.0.smithi115.stdout:2243: writing smithi11541303-47 from 3710976 to 4390912 tid 1 2022-04-22T17:33:27.979 INFO:tasks.rados.rados.0.smithi115.stdout:2243: writing smithi11541303-47 from 4390912 to 4833280 tid 2 2022-04-22T17:33:27.983 INFO:tasks.rados.rados.0.smithi115.stdout:2243: writing smithi11541303-47 from 4833280 to 5455872 tid 3 2022-04-22T17:33:27.984 INFO:tasks.rados.rados.0.smithi115.stdout:2243: writing smithi11541303-47 from 5455872 to 5595136 tid 4 2022-04-22T17:33:27.984 INFO:tasks.rados.rados.0.smithi115.stdout:2244: delete oid 27 current snap is 227 2022-04-22T17:33:27.984 INFO:tasks.rados.rados.0.smithi115.stdout:2237: finishing write tid 2 to smithi11541303-42 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2237: finishing write tid 3 to smithi11541303-42 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2237: finishing write tid 4 to smithi11541303-42 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2237: finishing write tid 5 to smithi11541303-42 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1426 (ObjNum 683 snap 227 seq_num 683) dirty exists 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2237: left oid 42 (ObjNum 683 snap 227 seq_num 683) 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 1 to smithi11541303-8 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 2 to smithi11541303-8 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 3 to smithi11541303-8 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 4 to smithi11541303-8 2022-04-22T17:33:27.985 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 5 to smithi11541303-8 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2238: finishing write tid 6 to smithi11541303-8 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1207 (ObjNum 684 snap 227 seq_num 684) dirty exists 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2238: left oid 8 (ObjNum 684 snap 227 seq_num 684) 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2240: finishing copy_from to smithi11541303-36 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2240: got expected ENOENT (src dne) 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2240: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:33:27.986 INFO:tasks.rados.rados.0.smithi115.stdout:2242: expect (ObjNum 500 snap 170 seq_num 500) 2022-04-22T17:33:28.037 INFO:tasks.rados.rados.0.smithi115.stdout:2241: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:28.166 INFO:tasks.rados.rados.0.smithi115.stdout:2239: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:33:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:27 smithi121 bash[36902]: cluster 2022-04-22T17:33:26.871799+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2022-04-22T17:33:28.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:27 smithi115 bash[46934]: cluster 2022-04-22T17:33:26.871799+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2022-04-22T17:33:28.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:27 smithi115 bash[45495]: cluster 2022-04-22T17:33:26.871799+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 1 to smithi11541303-47 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 2 to smithi11541303-47 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 3 to smithi11541303-47 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2237: done (7 left) 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2238: done (6 left) 2022-04-22T17:33:28.393 INFO:tasks.rados.rados.0.smithi115.stdout:2239: done (5 left) 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2240: done (4 left) 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2241: done (3 left) 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2242: done (2 left) 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2244: done (1 left) 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2245: setattr oid 1 current snap is 227 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 4 to smithi11541303-47 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 5 to smithi11541303-47 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:2243: finishing write tid 6 to smithi11541303-47 2022-04-22T17:33:28.394 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1144 (ObjNum 685 snap 227 seq_num 685) dirty exists 2022-04-22T17:33:28.395 INFO:tasks.rados.rados.0.smithi115.stdout:2243: left oid 47 (ObjNum 685 snap 227 seq_num 685) 2022-04-22T17:33:28.395 INFO:tasks.rados.rados.0.smithi115.stdout:2243: done (1 left) 2022-04-22T17:33:28.395 INFO:tasks.rados.rados.0.smithi115.stdout:2246: snap_remove snap 207 2022-04-22T17:33:28.912 INFO:tasks.rados.rados.0.smithi115.stdout:2246: done (1 left) 2022-04-22T17:33:28.912 INFO:tasks.rados.rados.0.smithi115.stdout:2247: read oid 29 snap -1 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:2247: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:2248: rollback oid 30 current snap is 227 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 220 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:2249: rollback oid 38 current snap is 227 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 222 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:2250: rollback oid 31 current snap is 227 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 211 2022-04-22T17:33:28.913 INFO:tasks.rados.rados.0.smithi115.stdout:2251: snap_create 2022-04-22T17:33:28.914 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1208 (ObjNum -1115644240 snap 22075 seq_num 3178990832) dirty exists 2022-04-22T17:33:28.924 INFO:tasks.rados.rados.0.smithi115.stdout:2248: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:33:28.925 INFO:tasks.rados.rados.0.smithi115.stdout:2248: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:33:28.925 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1220 (ObjNum 656 snap 213 seq_num 656) dirty exists 2022-04-22T17:33:28.925 INFO:tasks.rados.rados.0.smithi115.stdout:2247: expect (ObjNum 296 snap 99 seq_num 296) 2022-04-22T17:33:28.935 INFO:tasks.rados.rados.0.smithi115.stdout:2249: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:33:28.935 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1069 (ObjNum 288 snap 0 seq_num 49) dirty dne 2022-04-22T17:33:28.936 INFO:tasks.rados.rados.0.smithi115.stdout:2250: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:33:28.936 INFO:tasks.rados.rados.0.smithi115.stdout:2250: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:33:28.936 INFO:tasks.rados.rados.0.smithi115.stdout:2250: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:33:28.936 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1430 (ObjNum 525 snap 173 seq_num 525) dirty exists 2022-04-22T17:33:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:28 smithi121 bash[36902]: cluster 2022-04-22T17:33:27.257533+0000 mgr.y (mgr.34107) 264 : cluster [DBG] pgmap v453: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 328 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2022-04-22T17:33:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:28 smithi121 bash[36902]: cluster 2022-04-22T17:33:27.882176+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2022-04-22T17:33:29.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:28 smithi115 bash[45495]: cluster 2022-04-22T17:33:27.257533+0000 mgr.y (mgr.34107) 264 : cluster [DBG] pgmap v453: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 328 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2022-04-22T17:33:29.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:28 smithi115 bash[45495]: cluster 2022-04-22T17:33:27.882176+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2022-04-22T17:33:29.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:28 smithi115 bash[46934]: cluster 2022-04-22T17:33:27.257533+0000 mgr.y (mgr.34107) 264 : cluster [DBG] pgmap v453: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 328 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2022-04-22T17:33:29.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:28 smithi115 bash[46934]: cluster 2022-04-22T17:33:27.882176+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2022-04-22T17:33:29.871 INFO:tasks.rados.rados.0.smithi115.stdout:2245: done (5 left) 2022-04-22T17:33:29.871 INFO:tasks.rados.rados.0.smithi115.stdout:2247: done (4 left) 2022-04-22T17:33:29.871 INFO:tasks.rados.rados.0.smithi115.stdout:2248: done (3 left) 2022-04-22T17:33:29.871 INFO:tasks.rados.rados.0.smithi115.stdout:2249: done (2 left) 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2250: done (1 left) 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2251: done (0 left) 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2252: rollback oid 20 current snap is 228 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 224 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2253: copy_from oid 33 from oid 28 current snap is 228 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2254: read oid 50 snap 217 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2254: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:29.872 INFO:tasks.rados.rados.0.smithi115.stdout:2255: delete oid 5 current snap is 228 2022-04-22T17:33:29.883 INFO:tasks.rados.rados.0.smithi115.stdout:2255: done (3 left) 2022-04-22T17:33:29.883 INFO:tasks.rados.rados.0.smithi115.stdout:2256: copy_from oid 3 from oid 18 current snap is 228 2022-04-22T17:33:29.883 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 228 2022-04-22T17:33:29.884 INFO:tasks.rados.rados.0.smithi115.stdout:2257: seq_num 688 ranges {8626176=729088,9355264=499712,9854976=704512} 2022-04-22T17:33:29.937 INFO:tasks.rados.rados.0.smithi115.stdout:2257: writing smithi11541303-49 from 8626176 to 9355264 tid 1 2022-04-22T17:33:29.940 INFO:tasks.rados.rados.0.smithi115.stdout:2257: writing smithi11541303-49 from 9355264 to 9854976 tid 2 2022-04-22T17:33:29.945 INFO:tasks.rados.rados.0.smithi115.stdout:2257: writing smithi11541303-49 from 9854976 to 10559488 tid 3 2022-04-22T17:33:29.945 INFO:tasks.rados.rados.0.smithi115.stdout:2258: copy_from oid 44 from oid 21 current snap is 228 2022-04-22T17:33:29.945 INFO:tasks.rados.rados.0.smithi115.stdout:2256: finishing copy_from to smithi11541303-3 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2256: got expected ENOENT (src dne) 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2256: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2252: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2252: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2252: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1173 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2252: done (5 left) 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2256: done (4 left) 2022-04-22T17:33:29.946 INFO:tasks.rados.rados.0.smithi115.stdout:2259: rollback oid 19 current snap is 228 2022-04-22T17:33:29.947 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 222 2022-04-22T17:33:29.947 INFO:tasks.rados.rados.0.smithi115.stdout:2254: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:30.081 INFO:tasks.rados.rados.0.smithi115.stdout:2253: finishing copy_from to smithi11541303-33 2022-04-22T17:33:30.081 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1110 (ObjNum 657 snap 213 seq_num 657) dirty exists 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2253: finishing copy_from racing read to smithi11541303-33 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: finishing write tid 1 to smithi11541303-49 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: finishing write tid 2 to smithi11541303-49 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: finishing write tid 3 to smithi11541303-49 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: finishing write tid 4 to smithi11541303-49 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2259: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: finishing write tid 5 to smithi11541303-49 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1449 (ObjNum 688 snap 228 seq_num 688) dirty exists 2022-04-22T17:33:30.082 INFO:tasks.rados.rados.0.smithi115.stdout:2257: left oid 49 (ObjNum 688 snap 228 seq_num 688) 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2259: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2259: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1121 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2253: done (4 left) 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2254: done (3 left) 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2257: done (2 left) 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2259: done (1 left) 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2260: rollback oid 16 current snap is 228 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 225 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:2258: finishing copy_from to smithi11541303-44 2022-04-22T17:33:30.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1100 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2258: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2258: done (1 left) 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2261: rollback oid 8 current snap is 228 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 205 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2262: copy_from oid 48 from oid 46 current snap is 228 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2263: rollback oid 49 current snap is 228 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 223 2022-04-22T17:33:30.084 INFO:tasks.rados.rados.0.smithi115.stdout:2264: delete oid 6 current snap is 228 2022-04-22T17:33:30.085 INFO:tasks.rados.rados.0.smithi115.stdout:2262: finishing copy_from to smithi11541303-48 2022-04-22T17:33:30.085 INFO:tasks.rados.rados.0.smithi115.stdout:2262: got expected ENOENT (src dne) 2022-04-22T17:33:30.085 INFO:tasks.rados.rados.0.smithi115.stdout:2262: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:33:30.085 INFO:tasks.rados.rados.0.smithi115.stdout:2260: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:33:30.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1114 (ObjNum -1115744304 snap 22075 seq_num 3178734944) dirty dne 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2260: done (4 left) 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2262: done (3 left) 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2264: done (2 left) 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2265: read oid 42 snap 220 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2265: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:30.087 INFO:tasks.rados.rados.0.smithi115.stdout:2266: snap_create 2022-04-22T17:33:30.089 INFO:tasks.rados.rados.0.smithi115.stdout:2261: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:33:30.089 INFO:tasks.rados.rados.0.smithi115.stdout:2261: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:33:30.090 INFO:tasks.rados.rados.0.smithi115.stdout:2261: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:33:30.090 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1211 (ObjNum 588 snap 190 seq_num 588) dirty exists 2022-04-22T17:33:30.090 INFO:tasks.rados.rados.0.smithi115.stdout:2263: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:33:30.091 INFO:tasks.rados.rados.0.smithi115.stdout:2263: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:33:30.091 INFO:tasks.rados.rados.0.smithi115.stdout:2263: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:33:30.091 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1452 (ObjNum 627 snap 204 seq_num 627) dirty exists 2022-04-22T17:33:30.096 INFO:tasks.rados.rados.0.smithi115.stdout:2265: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:29 smithi121 bash[36902]: cluster 2022-04-22T17:33:28.894829+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2022-04-22T17:33:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:29 smithi121 bash[36902]: audit 2022-04-22T17:33:29.286362+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:29 smithi121 bash[36902]: cluster 2022-04-22T17:33:29.855944+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2022-04-22T17:33:30.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[45495]: cluster 2022-04-22T17:33:28.894829+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2022-04-22T17:33:30.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[45495]: audit 2022-04-22T17:33:29.286362+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:30.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[45495]: cluster 2022-04-22T17:33:29.855944+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2022-04-22T17:33:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[46934]: cluster 2022-04-22T17:33:28.894829+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2022-04-22T17:33:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[46934]: audit 2022-04-22T17:33:29.286362+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:30.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:29 smithi115 bash[46934]: cluster 2022-04-22T17:33:29.855944+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2022-04-22T17:33:30.922 INFO:tasks.rados.rados.0.smithi115.stdout:2261: done (3 left) 2022-04-22T17:33:30.922 INFO:tasks.rados.rados.0.smithi115.stdout:2263: done (2 left) 2022-04-22T17:33:30.923 INFO:tasks.rados.rados.0.smithi115.stdout:2265: done (1 left) 2022-04-22T17:33:30.923 INFO:tasks.rados.rados.0.smithi115.stdout:2266: done (0 left) 2022-04-22T17:33:30.923 INFO:tasks.rados.rados.0.smithi115.stdout:2267: read oid 21 snap -1 2022-04-22T17:33:30.923 INFO:tasks.rados.rados.0.smithi115.stdout:2267: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:33:30.923 INFO:tasks.rados.rados.0.smithi115.stdout:2268: snap_remove snap 225 2022-04-22T17:33:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:30 smithi121 bash[36902]: cluster 2022-04-22T17:33:29.258499+0000 mgr.y (mgr.34107) 265 : cluster [DBG] pgmap v456: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2022-04-22T17:33:31.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:30 smithi115 bash[45495]: cluster 2022-04-22T17:33:29.258499+0000 mgr.y (mgr.34107) 265 : cluster [DBG] pgmap v456: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2022-04-22T17:33:31.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:30 smithi115 bash[46934]: cluster 2022-04-22T17:33:29.258499+0000 mgr.y (mgr.34107) 265 : cluster [DBG] pgmap v456: 121 pgs: 121 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2022-04-22T17:33:31.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:31 smithi115 bash[45495]: cluster 2022-04-22T17:33:30.906279+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2022-04-22T17:33:31.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:31 smithi115 bash[46934]: cluster 2022-04-22T17:33:30.906279+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2022-04-22T17:33:31.926 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:31.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:31.927 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:31.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:31.927 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:31.628Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:31.930 INFO:tasks.rados.rados.0.smithi115.stdout:2268: done (1 left) 2022-04-22T17:33:31.930 INFO:tasks.rados.rados.0.smithi115.stdout:2269: copy_from oid 41 from oid 32 current snap is 229 2022-04-22T17:33:31.930 INFO:tasks.rados.rados.0.smithi115.stdout:2270: snap_remove snap 211 2022-04-22T17:33:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:31 smithi121 bash[36902]: cluster 2022-04-22T17:33:30.906279+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2270: done (2 left) 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2271: read oid 19 snap -1 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2271: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2272: copy_from oid 26 from oid 9 current snap is 229 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2273: read oid 10 snap -1 2022-04-22T17:33:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2273: expect deleted 2022-04-22T17:33:32.943 INFO:tasks.rados.rados.0.smithi115.stdout:2274: snap_remove snap 179 2022-04-22T17:33:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:32 smithi121 bash[36902]: cluster 2022-04-22T17:33:31.259114+0000 mgr.y (mgr.34107) 266 : cluster [DBG] pgmap v459: 121 pgs: 121 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2022-04-22T17:33:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:32 smithi121 bash[36902]: cluster 2022-04-22T17:33:31.914306+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2022-04-22T17:33:33.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:32 smithi115 bash[45495]: cluster 2022-04-22T17:33:31.259114+0000 mgr.y (mgr.34107) 266 : cluster [DBG] pgmap v459: 121 pgs: 121 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2022-04-22T17:33:33.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:32 smithi115 bash[45495]: cluster 2022-04-22T17:33:31.914306+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2022-04-22T17:33:33.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:32 smithi115 bash[46934]: cluster 2022-04-22T17:33:31.259114+0000 mgr.y (mgr.34107) 266 : cluster [DBG] pgmap v459: 121 pgs: 121 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2022-04-22T17:33:33.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:32 smithi115 bash[46934]: cluster 2022-04-22T17:33:31.914306+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2022-04-22T17:33:33.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:33.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:33.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:33.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:33.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:33.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:33:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2274: done (5 left) 2022-04-22T17:33:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2275: rmattr oid 48 current snap is 229 2022-04-22T17:33:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2275: done (5 left) 2022-04-22T17:33:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 229 2022-04-22T17:33:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2276: seq_num 690 ranges {0=761856,761856=663552,1425408=409600,1835008=131072} 2022-04-22T17:33:33.957 INFO:tasks.rados.rados.0.smithi115.stdout:2276: writing smithi11541303-11 from 0 to 761856 tid 1 2022-04-22T17:33:33.961 INFO:tasks.rados.rados.0.smithi115.stdout:2276: writing smithi11541303-11 from 761856 to 1425408 tid 2 2022-04-22T17:33:33.964 INFO:tasks.rados.rados.0.smithi115.stdout:2276: writing smithi11541303-11 from 1425408 to 1835008 tid 3 2022-04-22T17:33:33.966 INFO:tasks.rados.rados.0.smithi115.stdout:2276: writing smithi11541303-11 from 1835008 to 1966080 tid 4 2022-04-22T17:33:33.966 INFO:tasks.rados.rados.0.smithi115.stdout:2277: snap_create 2022-04-22T17:33:33.966 INFO:tasks.rados.rados.0.smithi115.stdout:2267: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:33:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:33 smithi121 bash[36902]: cluster 2022-04-22T17:33:32.926513+0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2022-04-22T17:33:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:33 smithi121 bash[36902]: audit 2022-04-22T17:33:33.383721+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:34.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[45495]: cluster 2022-04-22T17:33:32.926513+0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2022-04-22T17:33:34.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[45495]: audit 2022-04-22T17:33:33.383721+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:34.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[46934]: cluster 2022-04-22T17:33:32.926513+0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2022-04-22T17:33:34.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:33 smithi115 bash[46934]: audit 2022-04-22T17:33:33.383721+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:34.859 INFO:tasks.rados.rados.0.smithi115.stdout:2269: finishing copy_from to smithi11541303-41 2022-04-22T17:33:34.859 INFO:tasks.rados.rados.0.smithi115.stdout:2269: got expected ENOENT (src dne) 2022-04-22T17:33:34.859 INFO:tasks.rados.rados.0.smithi115.stdout:2269: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:33:34.859 INFO:tasks.rados.rados.0.smithi115.stdout:2271: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:2272: finishing copy_from to smithi11541303-26 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1329 (ObjNum 644 snap 210 seq_num 644) dirty exists 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:2272: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 1 to smithi11541303-11 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 2 to smithi11541303-11 2022-04-22T17:33:35.001 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 3 to smithi11541303-11 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 4 to smithi11541303-11 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 5 to smithi11541303-11 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2276: finishing write tid 6 to smithi11541303-11 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1204 (ObjNum 690 snap 229 seq_num 690) dirty exists 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2276: left oid 11 (ObjNum 690 snap 229 seq_num 690) 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2267: done (6 left) 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2269: done (5 left) 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2271: done (4 left) 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2272: done (3 left) 2022-04-22T17:33:35.002 INFO:tasks.rados.rados.0.smithi115.stdout:2273: done (2 left) 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2276: done (1 left) 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2277: done (0 left) 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2278: read oid 13 snap 223 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2278: expect deleted 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2279: read oid 44 snap 227 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2279: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2280: rmattr oid 35 current snap is 230 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2281: read oid 26 snap -1 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:2281: expect (ObjNum 644 snap 210 seq_num 644) 2022-04-22T17:33:35.003 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 230 2022-04-22T17:33:35.004 INFO:tasks.rados.rados.0.smithi115.stdout:2282: seq_num 691 ranges {2826240=778240,3604480=548864,4153344=614400,4767744=40960} 2022-04-22T17:33:35.026 INFO:tasks.rados.rados.0.smithi115.stdout:2282: writing smithi11541303-14 from 2826240 to 3604480 tid 1 2022-04-22T17:33:35.030 INFO:tasks.rados.rados.0.smithi115.stdout:2282: writing smithi11541303-14 from 3604480 to 4153344 tid 2 2022-04-22T17:33:35.034 INFO:tasks.rados.rados.0.smithi115.stdout:2282: writing smithi11541303-14 from 4153344 to 4767744 tid 3 2022-04-22T17:33:35.034 INFO:tasks.rados.rados.0.smithi115.stdout:2282: writing smithi11541303-14 from 4767744 to 4808704 tid 4 2022-04-22T17:33:35.034 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 230 2022-04-22T17:33:35.035 INFO:tasks.rados.rados.0.smithi115.stdout:2283: seq_num 692 ranges {5595136=794624,6389760=483328,6873088=720896} 2022-04-22T17:33:35.070 INFO:tasks.rados.rados.0.smithi115.stdout:2283: writing smithi11541303-47 from 5595136 to 6389760 tid 1 2022-04-22T17:33:35.073 INFO:tasks.rados.rados.0.smithi115.stdout:2283: writing smithi11541303-47 from 6389760 to 6873088 tid 2 2022-04-22T17:33:35.077 INFO:tasks.rados.rados.0.smithi115.stdout:2283: writing smithi11541303-47 from 6873088 to 7593984 tid 3 2022-04-22T17:33:35.077 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 230 2022-04-22T17:33:35.077 INFO:tasks.rados.rados.0.smithi115.stdout:2284: seq_num 693 ranges {6946816=409600,7356416=770048,8126464=581632,8708096=253952} 2022-04-22T17:33:35.124 INFO:tasks.rados.rados.0.smithi115.stdout:2284: writing smithi11541303-49 from 6946816 to 7356416 tid 1 2022-04-22T17:33:35.128 INFO:tasks.rados.rados.0.smithi115.stdout:2284: writing smithi11541303-49 from 7356416 to 8126464 tid 2 2022-04-22T17:33:35.132 INFO:tasks.rados.rados.0.smithi115.stdout:2284: writing smithi11541303-49 from 8126464 to 8708096 tid 3 2022-04-22T17:33:35.133 INFO:tasks.rados.rados.0.smithi115.stdout:2284: writing smithi11541303-49 from 8708096 to 8962048 tid 4 2022-04-22T17:33:35.134 INFO:tasks.rados.rados.0.smithi115.stdout:2285: delete oid 29 current snap is 230 2022-04-22T17:33:35.134 INFO:tasks.rados.rados.0.smithi115.stdout:2279: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:33:35.187 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1112 (ObjNum 674 snap 225 seq_num 674) dirty exists 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2278: done (7 left) 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2279: done (6 left) 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2280: done (5 left) 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2285: done (4 left) 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2286: setattr oid 23 current snap is 230 2022-04-22T17:33:35.188 INFO:tasks.rados.rados.0.smithi115.stdout:2281: expect (ObjNum 644 snap 210 seq_num 644) 2022-04-22T17:33:35.217 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 1 to smithi11541303-14 2022-04-22T17:33:35.217 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 2 to smithi11541303-14 2022-04-22T17:33:35.217 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 3 to smithi11541303-14 2022-04-22T17:33:35.217 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 4 to smithi11541303-14 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 5 to smithi11541303-14 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2282: finishing write tid 6 to smithi11541303-14 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1115 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2282: left oid 14 (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2283: finishing write tid 1 to smithi11541303-47 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2283: finishing write tid 2 to smithi11541303-47 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2283: finishing write tid 3 to smithi11541303-47 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2283: finishing write tid 4 to smithi11541303-47 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:2283: finishing write tid 5 to smithi11541303-47 2022-04-22T17:33:35.218 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1148 (ObjNum 692 snap 230 seq_num 692) dirty exists 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2283: left oid 47 (ObjNum 692 snap 230 seq_num 692) 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 1 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 2 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 3 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 4 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 5 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: finishing write tid 6 to smithi11541303-49 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1480 (ObjNum 693 snap 230 seq_num 693) dirty exists 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2284: left oid 49 (ObjNum 693 snap 230 seq_num 693) 2022-04-22T17:33:35.219 INFO:tasks.rados.rados.0.smithi115.stdout:2281: done (4 left) 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2282: done (3 left) 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2283: done (2 left) 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2284: done (1 left) 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2287: read oid 39 snap -1 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2287: expect (ObjNum 651 snap 212 seq_num 651) 2022-04-22T17:33:35.220 INFO:tasks.rados.rados.0.smithi115.stdout:2288: delete oid 25 current snap is 230 2022-04-22T17:33:35.223 INFO:tasks.rados.rados.0.smithi115.stdout:2288: done (2 left) 2022-04-22T17:33:35.224 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 230 2022-04-22T17:33:35.224 INFO:tasks.rados.rados.0.smithi115.stdout:2289: seq_num 696 ranges {0=458752,458752=417792,876544=753664,1630208=434176} 2022-04-22T17:33:35.228 INFO:tasks.rados.rados.0.smithi115.stdout:2289: writing smithi11541303-25 from 0 to 458752 tid 1 2022-04-22T17:33:35.231 INFO:tasks.rados.rados.0.smithi115.stdout:2289: writing smithi11541303-25 from 458752 to 876544 tid 2 2022-04-22T17:33:35.236 INFO:tasks.rados.rados.0.smithi115.stdout:2289: writing smithi11541303-25 from 876544 to 1630208 tid 3 2022-04-22T17:33:35.239 INFO:tasks.rados.rados.0.smithi115.stdout:2289: writing smithi11541303-25 from 1630208 to 2064384 tid 4 2022-04-22T17:33:35.239 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 230 2022-04-22T17:33:35.239 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1205 (ObjNum 679 snap 226 seq_num 679) dirty exists 2022-04-22T17:33:35.239 INFO:tasks.rados.rados.0.smithi115.stdout:2290: seq_num 697 ranges {8839168=475136,9314304=704512,10018816=557056,10575872=344064} 2022-04-22T17:33:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:34 smithi121 bash[36902]: cluster 2022-04-22T17:33:33.260024+0000 mgr.y (mgr.34107) 267 : cluster [DBG] pgmap v462: 121 pgs: 121 active+clean; 348 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 21 op/s 2022-04-22T17:33:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:34 smithi121 bash[36902]: cluster 2022-04-22T17:33:33.935165+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2022-04-22T17:33:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:34 smithi121 bash[36902]: audit 2022-04-22T17:33:34.289067+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:34 smithi121 bash[36902]: cluster 2022-04-22T17:33:34.861006+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2022-04-22T17:33:35.291 INFO:tasks.rados.rados.0.smithi115.stdout:2290: writing smithi11541303-31 from 8839168 to 9314304 tid 1 2022-04-22T17:33:35.294 INFO:tasks.rados.rados.0.smithi115.stdout:2290: writing smithi11541303-31 from 9314304 to 10018816 tid 2 2022-04-22T17:33:35.298 INFO:tasks.rados.rados.0.smithi115.stdout:2290: writing smithi11541303-31 from 10018816 to 10575872 tid 3 2022-04-22T17:33:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:2290: writing smithi11541303-31 from 10575872 to 10919936 tid 4 2022-04-22T17:33:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:2286: done (3 left) 2022-04-22T17:33:35.300 INFO:tasks.rados.rados.0.smithi115.stdout:2291: snap_remove snap 219 2022-04-22T17:33:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[45495]: cluster 2022-04-22T17:33:33.260024+0000 mgr.y (mgr.34107) 267 : cluster [DBG] pgmap v462: 121 pgs: 121 active+clean; 348 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 21 op/s 2022-04-22T17:33:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[45495]: cluster 2022-04-22T17:33:33.935165+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2022-04-22T17:33:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[45495]: audit 2022-04-22T17:33:34.289067+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:35.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[45495]: cluster 2022-04-22T17:33:34.861006+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2022-04-22T17:33:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[46934]: cluster 2022-04-22T17:33:33.260024+0000 mgr.y (mgr.34107) 267 : cluster [DBG] pgmap v462: 121 pgs: 121 active+clean; 348 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 21 op/s 2022-04-22T17:33:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[46934]: cluster 2022-04-22T17:33:33.935165+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2022-04-22T17:33:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[46934]: audit 2022-04-22T17:33:34.289067+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:34 smithi115 bash[46934]: cluster 2022-04-22T17:33:34.861006+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2022-04-22T17:33:35.958 INFO:tasks.rados.rados.0.smithi115.stdout:2287: expect (ObjNum 651 snap 212 seq_num 651) 2022-04-22T17:33:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 1 to smithi11541303-25 2022-04-22T17:33:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 2 to smithi11541303-25 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 3 to smithi11541303-25 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2287: done (3 left) 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2291: done (2 left) 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 230 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 4 to smithi11541303-25 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 5 to smithi11541303-25 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2289: finishing write tid 6 to smithi11541303-25 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1202 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:33:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2289: left oid 25 (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 1 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 2 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 3 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 4 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 5 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: finishing write tid 6 to smithi11541303-31 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1486 (ObjNum 697 snap 230 seq_num 697) dirty exists 2022-04-22T17:33:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:2290: left oid 31 (ObjNum 697 snap 230 seq_num 697) 2022-04-22T17:33:36.087 INFO:tasks.rados.rados.0.smithi115.stdout:2292: seq_num 698 ranges {2539520=499712,3039232=589824,3629056=770048,4399104=245760} 2022-04-22T17:33:36.102 INFO:tasks.rados.rados.0.smithi115.stdout:2292: writing smithi11541303-28 from 2539520 to 3039232 tid 1 2022-04-22T17:33:36.106 INFO:tasks.rados.rados.0.smithi115.stdout:2292: writing smithi11541303-28 from 3039232 to 3629056 tid 2 2022-04-22T17:33:36.111 INFO:tasks.rados.rados.0.smithi115.stdout:2292: writing smithi11541303-28 from 3629056 to 4399104 tid 3 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2292: writing smithi11541303-28 from 4399104 to 4644864 tid 4 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2289: done (2 left) 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2290: done (1 left) 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2293: read oid 36 snap 205 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2293: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2294: delete oid 26 current snap is 230 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 1 to smithi11541303-28 2022-04-22T17:33:36.114 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 2 to smithi11541303-28 2022-04-22T17:33:36.116 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 3 to smithi11541303-28 2022-04-22T17:33:36.117 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 4 to smithi11541303-28 2022-04-22T17:33:36.117 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 5 to smithi11541303-28 2022-04-22T17:33:36.118 INFO:tasks.rados.rados.0.smithi115.stdout:2294: done (2 left) 2022-04-22T17:33:36.119 INFO:tasks.rados.rados.0.smithi115.stdout:2295: delete oid 12 current snap is 230 2022-04-22T17:33:36.119 INFO:tasks.rados.rados.0.smithi115.stdout:2292: finishing write tid 6 to smithi11541303-28 2022-04-22T17:33:36.119 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1121 (ObjNum 698 snap 230 seq_num 698) dirty exists 2022-04-22T17:33:36.119 INFO:tasks.rados.rados.0.smithi115.stdout:2292: left oid 28 (ObjNum 698 snap 230 seq_num 698) 2022-04-22T17:33:36.122 INFO:tasks.rados.rados.0.smithi115.stdout:2292: done (2 left) 2022-04-22T17:33:36.123 INFO:tasks.rados.rados.0.smithi115.stdout:2295: done (1 left) 2022-04-22T17:33:36.123 INFO:tasks.rados.rados.0.smithi115.stdout:2296: setattr oid 37 current snap is 230 2022-04-22T17:33:36.123 INFO:tasks.rados.rados.0.smithi115.stdout:2293: expect (ObjNum 546 snap 178 seq_num 546) 2022-04-22T17:33:36.176 INFO:tasks.rados.rados.0.smithi115.stdout:2293: done (1 left) 2022-04-22T17:33:36.176 INFO:tasks.rados.rados.0.smithi115.stdout:2297: snap_remove snap 222 2022-04-22T17:33:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2297: done (1 left) 2022-04-22T17:33:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2298: snap_create 2022-04-22T17:33:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1113 (ObjNum 1683055966 snap 1818516574 seq_num 1768447330) dirty exists 2022-04-22T17:33:37.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:36 smithi121 bash[36902]: cluster 2022-04-22T17:33:35.260803+0000 mgr.y (mgr.34107) 268 : cluster [DBG] pgmap v465: 121 pgs: 121 active+clean; 332 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 20 op/s 2022-04-22T17:33:37.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:36 smithi121 bash[36902]: cluster 2022-04-22T17:33:35.943265+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2022-04-22T17:33:37.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:36 smithi115 bash[45495]: cluster 2022-04-22T17:33:35.260803+0000 mgr.y (mgr.34107) 268 : cluster [DBG] pgmap v465: 121 pgs: 121 active+clean; 332 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 20 op/s 2022-04-22T17:33:37.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:36 smithi115 bash[45495]: cluster 2022-04-22T17:33:35.943265+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2022-04-22T17:33:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:36 smithi115 bash[46934]: cluster 2022-04-22T17:33:35.260803+0000 mgr.y (mgr.34107) 268 : cluster [DBG] pgmap v465: 121 pgs: 121 active+clean; 332 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 20 op/s 2022-04-22T17:33:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:36 smithi115 bash[46934]: cluster 2022-04-22T17:33:35.943265+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2022-04-22T17:33:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2296: done (1 left) 2022-04-22T17:33:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2298: done (0 left) 2022-04-22T17:33:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2299: read oid 42 snap -1 2022-04-22T17:33:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2299: expect (ObjNum 683 snap 227 seq_num 683) 2022-04-22T17:33:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2300: snap_remove snap 230 2022-04-22T17:33:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:37 smithi121 bash[36902]: cluster 2022-04-22T17:33:36.955871+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2022-04-22T17:33:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:37 smithi121 bash[36902]: audit 2022-04-22T17:33:37.140462+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:38.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:37 smithi115 bash[45495]: cluster 2022-04-22T17:33:36.955871+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2022-04-22T17:33:38.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:37 smithi115 bash[45495]: audit 2022-04-22T17:33:37.140462+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:38.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:37 smithi115 bash[46934]: cluster 2022-04-22T17:33:36.955871+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2022-04-22T17:33:38.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:37 smithi115 bash[46934]: audit 2022-04-22T17:33:37.140462+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:38.982 INFO:tasks.rados.rados.0.smithi115.stdout:2300: done (1 left) 2022-04-22T17:33:38.985 INFO:tasks.rados.rados.0.smithi115.stdout:2301: copy_from oid 21 from oid 12 current snap is 231 2022-04-22T17:33:38.985 INFO:tasks.rados.rados.0.smithi115.stdout:2302: copy_from oid 13 from oid 26 current snap is 231 2022-04-22T17:33:38.985 INFO:tasks.rados.rados.0.smithi115.stdout:2299: expect (ObjNum 683 snap 227 seq_num 683) 2022-04-22T17:33:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2301: finishing copy_from to smithi11541303-21 2022-04-22T17:33:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2301: got expected ENOENT (src dne) 2022-04-22T17:33:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2301: finishing copy_from racing read to smithi11541303-21 2022-04-22T17:33:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2299: done (2 left) 2022-04-22T17:33:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2301: done (1 left) 2022-04-22T17:33:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2303: delete oid 37 current snap is 231 2022-04-22T17:33:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:2302: finishing copy_from to smithi11541303-13 2022-04-22T17:33:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:2302: got expected ENOENT (src dne) 2022-04-22T17:33:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:2302: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:33:39.059 INFO:tasks.rados.rados.0.smithi115.stdout:2302: done (1 left) 2022-04-22T17:33:39.059 INFO:tasks.rados.rados.0.smithi115.stdout:2303: done (0 left) 2022-04-22T17:33:39.059 INFO:tasks.rados.rados.0.smithi115.stdout:2304: read oid 34 snap -1 2022-04-22T17:33:39.059 INFO:tasks.rados.rados.0.smithi115.stdout:2304: expect (ObjNum 605 snap 198 seq_num 605) 2022-04-22T17:33:39.059 INFO:tasks.rados.rados.0.smithi115.stdout:2305: snap_remove snap 217 2022-04-22T17:33:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:38 smithi121 bash[36902]: audit 2022-04-22T17:33:37.141025+0000 mgr.y (mgr.34107) 269 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:38 smithi121 bash[36902]: cephadm 2022-04-22T17:33:37.143465+0000 mgr.y (mgr.34107) 270 : cephadm [INF] Upgrade: osd.6 is safe to restart 2022-04-22T17:33:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:38 smithi121 bash[36902]: cluster 2022-04-22T17:33:37.261534+0000 mgr.y (mgr.34107) 271 : cluster [DBG] pgmap v468: 121 pgs: 121 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:33:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:38 smithi121 bash[36902]: cluster 2022-04-22T17:33:37.961773+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2022-04-22T17:33:39.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[45495]: audit 2022-04-22T17:33:37.141025+0000 mgr.y (mgr.34107) 269 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[45495]: cephadm 2022-04-22T17:33:37.143465+0000 mgr.y (mgr.34107) 270 : cephadm [INF] Upgrade: osd.6 is safe to restart 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[45495]: cluster 2022-04-22T17:33:37.261534+0000 mgr.y (mgr.34107) 271 : cluster [DBG] pgmap v468: 121 pgs: 121 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[45495]: cluster 2022-04-22T17:33:37.961773+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[46934]: audit 2022-04-22T17:33:37.141025+0000 mgr.y (mgr.34107) 269 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[46934]: cephadm 2022-04-22T17:33:37.143465+0000 mgr.y (mgr.34107) 270 : cephadm [INF] Upgrade: osd.6 is safe to restart 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[46934]: cluster 2022-04-22T17:33:37.261534+0000 mgr.y (mgr.34107) 271 : cluster [DBG] pgmap v468: 121 pgs: 121 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2022-04-22T17:33:39.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:38 smithi115 bash[46934]: cluster 2022-04-22T17:33:37.961773+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2022-04-22T17:33:39.878 INFO:tasks.rados.rados.0.smithi115.stdout:2305: done (1 left) 2022-04-22T17:33:39.878 INFO:tasks.rados.rados.0.smithi115.stdout:2306: snap_create 2022-04-22T17:33:39.878 INFO:tasks.rados.rados.0.smithi115.stdout:2304: expect (ObjNum 605 snap 198 seq_num 605) 2022-04-22T17:33:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:39 smithi121 bash[36902]: cluster 2022-04-22T17:33:38.966804+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2022-04-22T17:33:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:39 smithi121 bash[36902]: audit 2022-04-22T17:33:39.292239+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:39 smithi121 bash[36902]: cluster 2022-04-22T17:33:39.862883+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2022-04-22T17:33:40.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[45495]: cluster 2022-04-22T17:33:38.966804+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2022-04-22T17:33:40.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[45495]: audit 2022-04-22T17:33:39.292239+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:40.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[45495]: cluster 2022-04-22T17:33:39.862883+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2022-04-22T17:33:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[46934]: cluster 2022-04-22T17:33:38.966804+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2022-04-22T17:33:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[46934]: audit 2022-04-22T17:33:39.292239+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:39 smithi115 bash[46934]: cluster 2022-04-22T17:33:39.862883+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2022-04-22T17:33:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:2304: done (1 left) 2022-04-22T17:33:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:2306: done (0 left) 2022-04-22T17:33:40.886 INFO:tasks.rados.rados.0.smithi115.stdout:2307: snap_remove snap 224 2022-04-22T17:33:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[45495]: cluster 2022-04-22T17:33:39.262575+0000 mgr.y (mgr.34107) 272 : cluster [DBG] pgmap v471: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2022-04-22T17:33:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[45495]: cephadm 2022-04-22T17:33:40.005350+0000 mgr.y (mgr.34107) 273 : cephadm [INF] Upgrade: Updating osd.6 2022-04-22T17:33:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[45495]: audit 2022-04-22T17:33:40.014579+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[45495]: cephadm 2022-04-22T17:33:40.017207+0000 mgr.y (mgr.34107) 274 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:33:41.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[45495]: cluster 2022-04-22T17:33:40.871453+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2022-04-22T17:33:41.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[46934]: cluster 2022-04-22T17:33:39.262575+0000 mgr.y (mgr.34107) 272 : cluster [DBG] pgmap v471: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2022-04-22T17:33:41.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[46934]: cephadm 2022-04-22T17:33:40.005350+0000 mgr.y (mgr.34107) 273 : cephadm [INF] Upgrade: Updating osd.6 2022-04-22T17:33:41.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[46934]: audit 2022-04-22T17:33:40.014579+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:41.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[46934]: cephadm 2022-04-22T17:33:40.017207+0000 mgr.y (mgr.34107) 274 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:33:41.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[46934]: cluster 2022-04-22T17:33:40.871453+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2022-04-22T17:33:41.460 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:41 smithi121 bash[36902]: cluster 2022-04-22T17:33:39.262575+0000 mgr.y (mgr.34107) 272 : cluster [DBG] pgmap v471: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2022-04-22T17:33:41.460 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:41 smithi121 bash[36902]: cephadm 2022-04-22T17:33:40.005350+0000 mgr.y (mgr.34107) 273 : cephadm [INF] Upgrade: Updating osd.6 2022-04-22T17:33:41.461 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:41 smithi121 bash[36902]: audit 2022-04-22T17:33:40.014579+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:41.461 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:41 smithi121 bash[36902]: cephadm 2022-04-22T17:33:40.017207+0000 mgr.y (mgr.34107) 274 : cephadm [INF] Deploying daemon osd.6 on smithi121 2022-04-22T17:33:41.461 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:41 smithi121 bash[36902]: cluster 2022-04-22T17:33:40.871453+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2022-04-22T17:33:41.885 INFO:tasks.rados.rados.0.smithi115.stdout:2307: done (0 left) 2022-04-22T17:33:41.886 INFO:tasks.rados.rados.0.smithi115.stdout:2308: copy_from oid 33 from oid 14 current snap is 232 2022-04-22T17:33:41.886 INFO:tasks.rados.rados.0.smithi115.stdout:2309: rmattr oid 20 current snap is 232 2022-04-22T17:33:41.886 INFO:tasks.rados.rados.0.smithi115.stdout:2310: snap_remove snap 223 2022-04-22T17:33:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:41.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:41.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:42.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:41.628Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:2310: done (2 left) 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:2311: snap_create 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1232 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:2308: finishing copy_from to smithi11541303-33 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1115 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:33:42.886 INFO:tasks.rados.rados.0.smithi115.stdout:2308: finishing copy_from racing read to smithi11541303-33 2022-04-22T17:33:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:42 smithi121 bash[36902]: cluster 2022-04-22T17:33:41.263590+0000 mgr.y (mgr.34107) 275 : cluster [DBG] pgmap v474: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 876 KiB/s rd, 992 KiB/s wr, 9 op/s 2022-04-22T17:33:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:42 smithi121 bash[36902]: cluster 2022-04-22T17:33:41.869159+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2022-04-22T17:33:43.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:42 smithi115 bash[45495]: cluster 2022-04-22T17:33:41.263590+0000 mgr.y (mgr.34107) 275 : cluster [DBG] pgmap v474: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 876 KiB/s rd, 992 KiB/s wr, 9 op/s 2022-04-22T17:33:43.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:42 smithi115 bash[45495]: cluster 2022-04-22T17:33:41.869159+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2022-04-22T17:33:43.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:42 smithi115 bash[46934]: cluster 2022-04-22T17:33:41.263590+0000 mgr.y (mgr.34107) 275 : cluster [DBG] pgmap v474: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 876 KiB/s rd, 992 KiB/s wr, 9 op/s 2022-04-22T17:33:43.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:42 smithi115 bash[46934]: cluster 2022-04-22T17:33:41.869159+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2022-04-22T17:33:43.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:43.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:43.523Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:43.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:43.523Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2308: done (2 left) 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2309: done (1 left) 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2311: done (0 left) 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2312: read oid 47 snap -1 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2312: expect (ObjNum 692 snap 230 seq_num 692) 2022-04-22T17:33:43.887 INFO:tasks.rados.rados.0.smithi115.stdout:2313: rmattr oid 50 current snap is 233 2022-04-22T17:33:43.888 INFO:tasks.rados.rados.0.smithi115.stdout:2314: rollback oid 38 current snap is 233 2022-04-22T17:33:43.888 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 205 2022-04-22T17:33:43.888 INFO:tasks.rados.rados.0.smithi115.stdout:2315: snap_create 2022-04-22T17:33:43.892 INFO:tasks.rados.rados.0.smithi115.stdout:2314: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:33:43.892 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 0 (ObjNum 1 snap 32692 seq_num 3179508048) dirty dne 2022-04-22T17:33:43.897 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1115 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:33:43.927 INFO:tasks.rados.rados.0.smithi115.stdout:2312: expect (ObjNum 692 snap 230 seq_num 692) 2022-04-22T17:33:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:43 smithi121 bash[36902]: cluster 2022-04-22T17:33:42.870979+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2022-04-22T17:33:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:43 smithi121 bash[36902]: cluster 2022-04-22T17:33:43.871992+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2022-04-22T17:33:44.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[46934]: cluster 2022-04-22T17:33:42.870979+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2022-04-22T17:33:44.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[46934]: cluster 2022-04-22T17:33:43.871992+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2022-04-22T17:33:44.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[45495]: cluster 2022-04-22T17:33:42.870979+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2022-04-22T17:33:44.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:43 smithi115 bash[45495]: cluster 2022-04-22T17:33:43.871992+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2312: done (3 left) 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2313: done (2 left) 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2314: done (1 left) 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2315: done (0 left) 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2316: rmattr oid 32 current snap is 234 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2316: done (0 left) 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:2317: rollback oid 3 current snap is 234 2022-04-22T17:33:44.879 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 205 2022-04-22T17:33:44.880 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 234 2022-04-22T17:33:44.880 INFO:tasks.rados.rados.0.smithi115.stdout:2318: seq_num 702 ranges {1196032=565248,1761280=581632,2342912=745472,3088384=278528} 2022-04-22T17:33:44.891 INFO:tasks.rados.rados.0.smithi115.stdout:2318: writing smithi11541303-9 from 1196032 to 1761280 tid 1 2022-04-22T17:33:44.895 INFO:tasks.rados.rados.0.smithi115.stdout:2318: writing smithi11541303-9 from 1761280 to 2342912 tid 2 2022-04-22T17:33:44.900 INFO:tasks.rados.rados.0.smithi115.stdout:2318: writing smithi11541303-9 from 2342912 to 3088384 tid 3 2022-04-22T17:33:44.902 INFO:tasks.rados.rados.0.smithi115.stdout:2318: writing smithi11541303-9 from 3088384 to 3366912 tid 4 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:2319: snap_create 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:2317: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:2317: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1207 (ObjNum 625 snap 204 seq_num 625) dirty exists 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 1 to smithi11541303-9 2022-04-22T17:33:44.903 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 2 to smithi11541303-9 2022-04-22T17:33:44.908 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 3 to smithi11541303-9 2022-04-22T17:33:44.909 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 4 to smithi11541303-9 2022-04-22T17:33:44.909 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 5 to smithi11541303-9 2022-04-22T17:33:44.911 INFO:tasks.rados.rados.0.smithi115.stdout:2318: finishing write tid 6 to smithi11541303-9 2022-04-22T17:33:44.911 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1532 (ObjNum 702 snap 234 seq_num 702) dirty exists 2022-04-22T17:33:44.911 INFO:tasks.rados.rados.0.smithi115.stdout:2318: left oid 9 (ObjNum 702 snap 234 seq_num 702) 2022-04-22T17:33:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:44 smithi121 bash[36902]: cluster 2022-04-22T17:33:43.264444+0000 mgr.y (mgr.34107) 276 : cluster [DBG] pgmap v477: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T17:33:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:44 smithi121 bash[36902]: audit 2022-04-22T17:33:44.295245+0000 mon.a (mon.0) 880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:44 smithi121 bash[36902]: cluster 2022-04-22T17:33:44.863780+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2022-04-22T17:33:45.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:45 smithi121 systemd[1]: Stopping Ceph osd.6 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:33:45.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[21417]: debug 2022-04-22T17:33:45.107+0000 7fcc1c626700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:33:45.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[21417]: debug 2022-04-22T17:33:45.107+0000 7fcc1c626700 -1 osd.6 528 *** Got signal Terminated *** 2022-04-22T17:33:45.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[21417]: debug 2022-04-22T17:33:45.107+0000 7fcc1c626700 -1 osd.6 528 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:33:45.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[45495]: cluster 2022-04-22T17:33:43.264444+0000 mgr.y (mgr.34107) 276 : cluster [DBG] pgmap v477: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T17:33:45.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[45495]: audit 2022-04-22T17:33:44.295245+0000 mon.a (mon.0) 880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:45.410 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[45495]: cluster 2022-04-22T17:33:44.863780+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2022-04-22T17:33:45.411 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[46934]: cluster 2022-04-22T17:33:43.264444+0000 mgr.y (mgr.34107) 276 : cluster [DBG] pgmap v477: 121 pgs: 121 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T17:33:45.411 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[46934]: audit 2022-04-22T17:33:44.295245+0000 mon.a (mon.0) 880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:45.411 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:44 smithi115 bash[46934]: cluster 2022-04-22T17:33:44.863780+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2022-04-22T17:33:45.918 INFO:tasks.rados.rados.0.smithi115.stdout:2317: done (2 left) 2022-04-22T17:33:45.918 INFO:tasks.rados.rados.0.smithi115.stdout:2318: done (1 left) 2022-04-22T17:33:45.918 INFO:tasks.rados.rados.0.smithi115.stdout:2319: done (0 left) 2022-04-22T17:33:45.918 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 235 2022-04-22T17:33:45.919 INFO:tasks.rados.rados.0.smithi115.stdout:2320: seq_num 703 ranges {4808704=581632,5390336=458752,5849088=557056,6406144=417792,6823936=172032} 2022-04-22T17:33:45.949 INFO:tasks.rados.rados.0.smithi115.stdout:2320: writing smithi11541303-33 from 4808704 to 5390336 tid 1 2022-04-22T17:33:45.952 INFO:tasks.rados.rados.0.smithi115.stdout:2320: writing smithi11541303-33 from 5390336 to 5849088 tid 2 2022-04-22T17:33:45.955 INFO:tasks.rados.rados.0.smithi115.stdout:2320: writing smithi11541303-33 from 5849088 to 6406144 tid 3 2022-04-22T17:33:45.958 INFO:tasks.rados.rados.0.smithi115.stdout:2320: writing smithi11541303-33 from 6406144 to 6823936 tid 4 2022-04-22T17:33:45.959 INFO:tasks.rados.rados.0.smithi115.stdout:2320: writing smithi11541303-33 from 6823936 to 6995968 tid 5 2022-04-22T17:33:45.959 INFO:tasks.rados.rados.0.smithi115.stdout:2321: snap_create 2022-04-22T17:33:45.959 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 1 to smithi11541303-33 2022-04-22T17:33:45.959 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 2 to smithi11541303-33 2022-04-22T17:33:45.960 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 3 to smithi11541303-33 2022-04-22T17:33:45.963 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 4 to smithi11541303-33 2022-04-22T17:33:45.964 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 5 to smithi11541303-33 2022-04-22T17:33:45.964 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 6 to smithi11541303-33 2022-04-22T17:33:45.966 INFO:tasks.rados.rados.0.smithi115.stdout:2320: finishing write tid 7 to smithi11541303-33 2022-04-22T17:33:45.966 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1336 (ObjNum 703 snap 235 seq_num 703) dirty exists 2022-04-22T17:33:45.966 INFO:tasks.rados.rados.0.smithi115.stdout:2320: left oid 33 (ObjNum 703 snap 235 seq_num 703) 2022-04-22T17:33:46.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[43146]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.6 2022-04-22T17:33:46.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[43191]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-6 2022-04-22T17:33:46.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[43204]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.6 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.406465+0000 mon.a (mon.0) 882 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.406559+0000 mon.a (mon.0) 883 : cluster [INF] osd.6 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.407257+0000 mon.a (mon.0) 884 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.407882+0000 mon.a (mon.0) 885 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.408386+0000 mon.a (mon.0) 886 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.408679+0000 mon.a (mon.0) 887 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.409096+0000 mon.a (mon.0) 888 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.409382+0000 mon.a (mon.0) 889 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.409772+0000 mon.a (mon.0) 890 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.410044+0000 mon.a (mon.0) 891 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.410424+0000 mon.a (mon.0) 892 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.410801+0000 mon.a (mon.0) 893 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.411111+0000 mon.a (mon.0) 894 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.411467+0000 mon.a (mon.0) 895 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.411915+0000 mon.a (mon.0) 896 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.804984+0000 mon.a (mon.0) 897 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.805376+0000 mon.a (mon.0) 898 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.805751+0000 mon.a (mon.0) 899 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.807199+0000 mon.a (mon.0) 900 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.807599+0000 mon.a (mon.0) 901 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.807802+0000 mon.a (mon.0) 902 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.808148+0000 mon.a (mon.0) 903 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.808588+0000 mon.a (mon.0) 904 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.811723+0000 mon.a (mon.0) 905 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.812172+0000 mon.a (mon.0) 906 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.812489+0000 mon.a (mon.0) 907 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.812900+0000 mon.a (mon.0) 908 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.813073+0000 mon.a (mon.0) 909 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:45 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.813438+0000 mon.a (mon.0) 910 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.406465+0000 mon.a (mon.0) 882 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.406559+0000 mon.a (mon.0) 883 : cluster [INF] osd.6 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:46.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.407257+0000 mon.a (mon.0) 884 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.407882+0000 mon.a (mon.0) 885 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.408386+0000 mon.a (mon.0) 886 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.408679+0000 mon.a (mon.0) 887 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.409096+0000 mon.a (mon.0) 888 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.409382+0000 mon.a (mon.0) 889 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.409772+0000 mon.a (mon.0) 890 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.410044+0000 mon.a (mon.0) 891 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.410424+0000 mon.a (mon.0) 892 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.410801+0000 mon.a (mon.0) 893 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.411111+0000 mon.a (mon.0) 894 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.411467+0000 mon.a (mon.0) 895 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.411915+0000 mon.a (mon.0) 896 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.804984+0000 mon.a (mon.0) 897 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.805376+0000 mon.a (mon.0) 898 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.805751+0000 mon.a (mon.0) 899 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.807199+0000 mon.a (mon.0) 900 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.807599+0000 mon.a (mon.0) 901 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.807802+0000 mon.a (mon.0) 902 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.808148+0000 mon.a (mon.0) 903 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.808588+0000 mon.a (mon.0) 904 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.811723+0000 mon.a (mon.0) 905 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.812172+0000 mon.a (mon.0) 906 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.812489+0000 mon.a (mon.0) 907 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.406465+0000 mon.a (mon.0) 882 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.406559+0000 mon.a (mon.0) 883 : cluster [INF] osd.6 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.407257+0000 mon.a (mon.0) 884 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.407882+0000 mon.a (mon.0) 885 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.408386+0000 mon.a (mon.0) 886 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.408679+0000 mon.a (mon.0) 887 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.409096+0000 mon.a (mon.0) 888 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.409382+0000 mon.a (mon.0) 889 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.409772+0000 mon.a (mon.0) 890 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.410044+0000 mon.a (mon.0) 891 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.410424+0000 mon.a (mon.0) 892 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.410801+0000 mon.a (mon.0) 893 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.411111+0000 mon.a (mon.0) 894 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.411467+0000 mon.a (mon.0) 895 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.411915+0000 mon.a (mon.0) 896 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.804984+0000 mon.a (mon.0) 897 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.805376+0000 mon.a (mon.0) 898 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.805751+0000 mon.a (mon.0) 899 : cluster [DBG] osd.6 reported immediately failed by osd.1 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.807199+0000 mon.a (mon.0) 900 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.807599+0000 mon.a (mon.0) 901 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.807802+0000 mon.a (mon.0) 902 : cluster [DBG] osd.6 reported immediately failed by osd.0 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.808148+0000 mon.a (mon.0) 903 : cluster [DBG] osd.6 reported immediately failed by osd.2 2022-04-22T17:33:46.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.808588+0000 mon.a (mon.0) 904 : cluster [DBG] osd.6 reported immediately failed by osd.3 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.811723+0000 mon.a (mon.0) 905 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.812172+0000 mon.a (mon.0) 906 : cluster [DBG] osd.6 reported immediately failed by osd.4 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.812489+0000 mon.a (mon.0) 907 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.812900+0000 mon.a (mon.0) 908 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.813073+0000 mon.a (mon.0) 909 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.813438+0000 mon.a (mon.0) 910 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.812900+0000 mon.a (mon.0) 908 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.813073+0000 mon.a (mon.0) 909 : cluster [DBG] osd.6 reported immediately failed by osd.7 2022-04-22T17:33:46.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:45 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.813438+0000 mon.a (mon.0) 910 : cluster [DBG] osd.6 reported immediately failed by osd.5 2022-04-22T17:33:46.915 INFO:tasks.rados.rados.0.smithi115.stdout:2320: done (1 left) 2022-04-22T17:33:46.915 INFO:tasks.rados.rados.0.smithi115.stdout:2321: done (0 left) 2022-04-22T17:33:46.915 INFO:tasks.rados.rados.0.smithi115.stdout:2322: delete oid 46 current snap is 236 2022-04-22T17:33:46.929 INFO:tasks.rados.rados.0.smithi115.stdout:2322: done (0 left) 2022-04-22T17:33:46.929 INFO:tasks.rados.rados.0.smithi115.stdout:2323: delete oid 40 current snap is 236 2022-04-22T17:33:46.934 INFO:tasks.rados.rados.0.smithi115.stdout:2323: done (0 left) 2022-04-22T17:33:46.934 INFO:tasks.rados.rados.0.smithi115.stdout:2324: snap_create 2022-04-22T17:33:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.265336+0000 mgr.y (mgr.34107) 277 : cluster [DBG] pgmap v480: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2022-04-22T17:33:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.883402+0000 mon.a (mon.0) 911 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:46 smithi121 bash[36902]: cluster 2022-04-22T17:33:45.904137+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.265336+0000 mgr.y (mgr.34107) 277 : cluster [DBG] pgmap v480: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.883402+0000 mon.a (mon.0) 911 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[45495]: cluster 2022-04-22T17:33:45.904137+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.265336+0000 mgr.y (mgr.34107) 277 : cluster [DBG] pgmap v480: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.883402+0000 mon.a (mon.0) 911 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:33:47.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:46 smithi115 bash[46934]: cluster 2022-04-22T17:33:45.904137+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2022-04-22T17:33:47.939 INFO:tasks.rados.rados.0.smithi115.stdout:2324: done (0 left) 2022-04-22T17:33:47.939 INFO:tasks.rados.rados.0.smithi115.stdout:2325: read oid 50 snap -1 2022-04-22T17:33:47.939 INFO:tasks.rados.rados.0.smithi115.stdout:2325: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:47.939 INFO:tasks.rados.rados.0.smithi115.stdout:2326: snap_create 2022-04-22T17:33:47.959 INFO:tasks.rados.rados.0.smithi115.stdout:2325: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:47 smithi121 bash[36902]: cluster 2022-04-22T17:33:46.900557+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2022-04-22T17:33:48.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:47 smithi115 bash[46934]: cluster 2022-04-22T17:33:46.900557+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2022-04-22T17:33:48.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:47 smithi115 bash[45495]: cluster 2022-04-22T17:33:46.900557+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2022-04-22T17:33:48.915 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:48 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.6.service: Succeeded. 2022-04-22T17:33:48.916 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:48 smithi121 systemd[1]: Stopped Ceph osd.6 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:33:48.946 INFO:tasks.rados.rados.0.smithi115.stdout:2325: done (1 left) 2022-04-22T17:33:48.946 INFO:tasks.rados.rados.0.smithi115.stdout:2326: done (0 left) 2022-04-22T17:33:48.946 INFO:tasks.rados.rados.0.smithi115.stdout:2327: copy_from oid 48 from oid 32 current snap is 238 2022-04-22T17:33:48.946 INFO:tasks.rados.rados.0.smithi115.stdout:2328: rollback oid 19 current snap is 238 2022-04-22T17:33:48.947 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 205 2022-04-22T17:33:48.947 INFO:tasks.rados.rados.0.smithi115.stdout:2329: copy_from oid 36 from oid 37 current snap is 238 2022-04-22T17:33:48.947 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 238 2022-04-22T17:33:48.947 INFO:tasks.rados.rados.0.smithi115.stdout:2330: seq_num 706 ranges {3194880=630784,3825664=516096,4341760=729088,5070848=360448} 2022-04-22T17:33:48.972 INFO:tasks.rados.rados.0.smithi115.stdout:2330: writing smithi11541303-39 from 3194880 to 3825664 tid 1 2022-04-22T17:33:48.976 INFO:tasks.rados.rados.0.smithi115.stdout:2330: writing smithi11541303-39 from 3825664 to 4341760 tid 2 2022-04-22T17:33:48.981 INFO:tasks.rados.rados.0.smithi115.stdout:2330: writing smithi11541303-39 from 4341760 to 5070848 tid 3 2022-04-22T17:33:48.984 INFO:tasks.rados.rados.0.smithi115.stdout:2330: writing smithi11541303-39 from 5070848 to 5431296 tid 4 2022-04-22T17:33:48.984 INFO:tasks.rados.rados.0.smithi115.stdout:2331: snap_create 2022-04-22T17:33:48.984 INFO:tasks.rados.rados.0.smithi115.stdout:2327: finishing copy_from to smithi11541303-48 2022-04-22T17:33:48.984 INFO:tasks.rados.rados.0.smithi115.stdout:2327: got expected ENOENT (src dne) 2022-04-22T17:33:48.984 INFO:tasks.rados.rados.0.smithi115.stdout:2327: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2329: finishing copy_from to smithi11541303-36 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2329: got expected ENOENT (src dne) 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2329: finishing copy_from racing read to smithi11541303-36 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2328: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2328: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2328: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1124 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 1 to smithi11541303-39 2022-04-22T17:33:48.985 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 2 to smithi11541303-39 2022-04-22T17:33:48.988 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 3 to smithi11541303-39 2022-04-22T17:33:48.989 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 4 to smithi11541303-39 2022-04-22T17:33:48.989 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 5 to smithi11541303-39 2022-04-22T17:33:48.990 INFO:tasks.rados.rados.0.smithi115.stdout:2330: finishing write tid 6 to smithi11541303-39 2022-04-22T17:33:48.990 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1119 (ObjNum 706 snap 238 seq_num 706) dirty exists 2022-04-22T17:33:48.991 INFO:tasks.rados.rados.0.smithi115.stdout:2330: left oid 39 (ObjNum 706 snap 238 seq_num 706) 2022-04-22T17:33:49.266 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:48 smithi121 systemd[1]: Started Ceph osd.6 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:33:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:48 smithi121 bash[36902]: cluster 2022-04-22T17:33:47.266220+0000 mgr.y (mgr.34107) 278 : cluster [DBG] pgmap v483: 121 pgs: 2 peering, 10 stale+active+clean, 9 active+undersized, 100 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:33:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:48 smithi121 bash[36902]: cluster 2022-04-22T17:33:47.924560+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2022-04-22T17:33:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:48 smithi115 bash[46934]: cluster 2022-04-22T17:33:47.266220+0000 mgr.y (mgr.34107) 278 : cluster [DBG] pgmap v483: 121 pgs: 2 peering, 10 stale+active+clean, 9 active+undersized, 100 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:33:49.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:48 smithi115 bash[46934]: cluster 2022-04-22T17:33:47.924560+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2022-04-22T17:33:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:48 smithi115 bash[45495]: cluster 2022-04-22T17:33:47.266220+0000 mgr.y (mgr.34107) 278 : cluster [DBG] pgmap v483: 121 pgs: 2 peering, 10 stale+active+clean, 9 active+undersized, 100 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:33:49.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:48 smithi115 bash[45495]: cluster 2022-04-22T17:33:47.924560+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2022-04-22T17:33:49.883 INFO:tasks.rados.rados.0.smithi115.stdout:2327: done (4 left) 2022-04-22T17:33:49.981 INFO:tasks.rados.rados.0.smithi115.stdout:2328: done (3 left) 2022-04-22T17:33:49.981 INFO:tasks.rados.rados.0.smithi115.stdout:2329: done (2 left) 2022-04-22T17:33:49.982 INFO:tasks.rados.rados.0.smithi115.stdout:2330: done (1 left) 2022-04-22T17:33:49.982 INFO:tasks.rados.rados.0.smithi115.stdout:2331: done (0 left) 2022-04-22T17:33:49.982 INFO:tasks.rados.rados.0.smithi115.stdout:2332: read oid 28 snap -1 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2332: expect (ObjNum 698 snap 230 seq_num 698) 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2333: read oid 50 snap -1 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2333: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2334: read oid 48 snap -1 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2334: expect deleted 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2335: read oid 29 snap -1 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2335: expect deleted 2022-04-22T17:33:49.985 INFO:tasks.rados.rados.0.smithi115.stdout:2336: read oid 42 snap -1 2022-04-22T17:33:49.986 INFO:tasks.rados.rados.0.smithi115.stdout:2336: expect (ObjNum 683 snap 227 seq_num 683) 2022-04-22T17:33:49.986 INFO:tasks.rados.rados.0.smithi115.stdout:2337: snap_create 2022-04-22T17:33:49.986 INFO:tasks.rados.rados.0.smithi115.stdout:2333: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:33:50.045 INFO:tasks.rados.rados.0.smithi115.stdout:2332: expect (ObjNum 698 snap 230 seq_num 698) 2022-04-22T17:33:50.231 INFO:tasks.rados.rados.0.smithi115.stdout:2336: expect (ObjNum 683 snap 227 seq_num 683) 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: cluster 2022-04-22T17:33:48.931378+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: audit 2022-04-22T17:33:48.954164+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: audit 2022-04-22T17:33:48.955575+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: audit 2022-04-22T17:33:48.957338+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: audit 2022-04-22T17:33:48.958334+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: audit 2022-04-22T17:33:49.298150+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:50.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:49 smithi121 bash[36902]: cluster 2022-04-22T17:33:49.868581+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2022-04-22T17:33:50.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: cluster 2022-04-22T17:33:48.931378+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2022-04-22T17:33:50.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: audit 2022-04-22T17:33:48.954164+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:50.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: audit 2022-04-22T17:33:48.955575+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: audit 2022-04-22T17:33:48.957338+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: audit 2022-04-22T17:33:48.958334+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: audit 2022-04-22T17:33:49.298150+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[45495]: cluster 2022-04-22T17:33:49.868581+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: cluster 2022-04-22T17:33:48.931378+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: audit 2022-04-22T17:33:48.954164+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:50.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: audit 2022-04-22T17:33:48.955575+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:33:50.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: audit 2022-04-22T17:33:48.957338+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:33:50.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: audit 2022-04-22T17:33:48.958334+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:33:50.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: audit 2022-04-22T17:33:49.298150+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:50.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:49 smithi115 bash[46934]: cluster 2022-04-22T17:33:49.868581+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2332: done (5 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2333: done (4 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2334: done (3 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2335: done (2 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2336: done (1 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2337: done (0 left) 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2338: copy_from oid 22 from oid 19 current snap is 240 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2339: read oid 11 snap -1 2022-04-22T17:33:50.886 INFO:tasks.rados.rados.0.smithi115.stdout:2339: expect (ObjNum 690 snap 229 seq_num 690) 2022-04-22T17:33:50.887 INFO:tasks.rados.rados.0.smithi115.stdout:2340: snap_create 2022-04-22T17:33:50.899 INFO:tasks.rados.rados.0.smithi115.stdout:2339: expect (ObjNum 690 snap 229 seq_num 690) 2022-04-22T17:33:50.965 INFO:tasks.rados.rados.0.smithi115.stdout:2338: finishing copy_from to smithi11541303-22 2022-04-22T17:33:51.000 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1124 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:33:51.000 INFO:tasks.rados.rados.0.smithi115.stdout:2338: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:33:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[36902]: cluster 2022-04-22T17:33:49.267260+0000 mgr.y (mgr.34107) 279 : cluster [DBG] pgmap v486: 121 pgs: 7 active+undersized+degraded, 2 peering, 4 stale+active+clean, 22 active+undersized, 86 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 530 KiB/s wr, 3 op/s; 40/486 objects degraded (8.230%) 2022-04-22T17:33:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[36902]: cluster 2022-04-22T17:33:49.951551+0000 mon.a (mon.0) 922 : cluster [WRN] Health check failed: Degraded data redundancy: 40/486 objects degraded (8.230%), 7 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[36902]: cluster 2022-04-22T17:33:50.869805+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2022-04-22T17:33:51.016 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-04-22T17:33:51.016 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2022-04-22T17:33:51.016 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2022-04-22T17:33:51.016 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2022-04-22T17:33:51.017 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2022-04-22T17:33:51.017 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-04-22T17:33:51.017 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:50 smithi121 bash[43392]: --> ceph-volume raw activate successful for osd ID: 6 2022-04-22T17:33:51.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[45495]: cluster 2022-04-22T17:33:49.267260+0000 mgr.y (mgr.34107) 279 : cluster [DBG] pgmap v486: 121 pgs: 7 active+undersized+degraded, 2 peering, 4 stale+active+clean, 22 active+undersized, 86 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 530 KiB/s wr, 3 op/s; 40/486 objects degraded (8.230%) 2022-04-22T17:33:51.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[45495]: cluster 2022-04-22T17:33:49.951551+0000 mon.a (mon.0) 922 : cluster [WRN] Health check failed: Degraded data redundancy: 40/486 objects degraded (8.230%), 7 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:51.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[45495]: cluster 2022-04-22T17:33:50.869805+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2022-04-22T17:33:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[46934]: cluster 2022-04-22T17:33:49.267260+0000 mgr.y (mgr.34107) 279 : cluster [DBG] pgmap v486: 121 pgs: 7 active+undersized+degraded, 2 peering, 4 stale+active+clean, 22 active+undersized, 86 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 530 KiB/s wr, 3 op/s; 40/486 objects degraded (8.230%) 2022-04-22T17:33:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[46934]: cluster 2022-04-22T17:33:49.951551+0000 mon.a (mon.0) 922 : cluster [WRN] Health check failed: Degraded data redundancy: 40/486 objects degraded (8.230%), 7 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:51.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:50 smithi115 bash[46934]: cluster 2022-04-22T17:33:50.869805+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2022-04-22T17:33:51.887 INFO:tasks.rados.rados.0.smithi115.stdout:2338: done (2 left) 2022-04-22T17:33:51.887 INFO:tasks.rados.rados.0.smithi115.stdout:2339: done (1 left) 2022-04-22T17:33:51.887 INFO:tasks.rados.rados.0.smithi115.stdout:2340: done (0 left) 2022-04-22T17:33:51.888 INFO:tasks.rados.rados.0.smithi115.stdout:2341: rmattr oid 34 current snap is 241 2022-04-22T17:33:51.888 INFO:tasks.rados.rados.0.smithi115.stdout:2342: setattr oid 27 current snap is 241 2022-04-22T17:33:51.891 INFO:tasks.rados.rados.0.smithi115.stdout:2343: read oid 38 snap -1 2022-04-22T17:33:51.891 INFO:tasks.rados.rados.0.smithi115.stdout:2343: expect deleted 2022-04-22T17:33:51.891 INFO:tasks.rados.rados.0.smithi115.stdout:2344: snap_create 2022-04-22T17:33:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1247 (ObjNum -1115664304 snap 22075 seq_num 0) dirty exists 2022-04-22T17:33:51.896 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1116 (ObjNum 605 snap 198 seq_num 605) dirty exists 2022-04-22T17:33:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:51.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:51.628Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:52.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:51.629Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:33:52.558 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:33:52.891 INFO:tasks.rados.rados.0.smithi115.stdout:2341: done (3 left) 2022-04-22T17:33:52.891 INFO:tasks.rados.rados.0.smithi115.stdout:2342: done (2 left) 2022-04-22T17:33:52.891 INFO:tasks.rados.rados.0.smithi115.stdout:2343: done (1 left) 2022-04-22T17:33:52.891 INFO:tasks.rados.rados.0.smithi115.stdout:2344: done (0 left) 2022-04-22T17:33:52.892 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 242 2022-04-22T17:33:52.892 INFO:tasks.rados.rados.0.smithi115.stdout:2345: seq_num 707 ranges {2711552=647168,3358720=802816,4161536=532480,4694016=270336} 2022-04-22T17:33:52.904 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:52 smithi121 bash[36902]: cluster 2022-04-22T17:33:51.268323+0000 mgr.y (mgr.34107) 280 : cluster [DBG] pgmap v489: 121 pgs: 8 active+undersized+degraded, 2 peering, 32 active+undersized, 79 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 534 KiB/s wr, 3 op/s; 41/489 objects degraded (8.384%) 2022-04-22T17:33:52.904 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:52 smithi121 bash[36902]: cluster 2022-04-22T17:33:51.872407+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2022-04-22T17:33:52.904 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:52 smithi121 bash[43565]: debug 2022-04-22T17:33:52.631+0000 7f088a983700 1 -- 172.21.15.121:0/2928603353 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dc0fbe8340 con 0x55dc0fc6cc00 2022-04-22T17:33:52.909 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:33:52.909 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (10m) 2m ago 10m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:33:52.909 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (10m) 40s ago 10m 24.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:33:52.909 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (11m) 2m ago 11m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (11m) 40s ago 11m 19.4M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6m) 40s ago 15m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6m) 2m ago 18m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5m) 2m ago 18m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4m) 40s ago 16m 49.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5m) 2m ago 16m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (10m) 2m ago 10m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (10m) 40s ago 10m 9451k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3m) 2m ago 15m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:33:52.910 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2m) 2m ago 15m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2m) 2m ago 14m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2m) 2m ago 14m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (83s) 40s ago 13m 181M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (42s) 40s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 starting - - - 2526M 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (12m) 40s ago 12m 252M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:33:52.911 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (6m) 40s ago 11m 40.6M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:33:52.911 INFO:tasks.rados.rados.0.smithi115.stdout:2345: writing smithi11541303-24 from 2711552 to 3358720 tid 1 2022-04-22T17:33:52.916 INFO:tasks.rados.rados.0.smithi115.stdout:2345: writing smithi11541303-24 from 3358720 to 4161536 tid 2 2022-04-22T17:33:52.919 INFO:tasks.rados.rados.0.smithi115.stdout:2345: writing smithi11541303-24 from 4161536 to 4694016 tid 3 2022-04-22T17:33:52.922 INFO:tasks.rados.rados.0.smithi115.stdout:2345: writing smithi11541303-24 from 4694016 to 4964352 tid 4 2022-04-22T17:33:52.922 INFO:tasks.rados.rados.0.smithi115.stdout:2346: copy_from oid 30 from oid 12 current snap is 242 2022-04-22T17:33:52.922 INFO:tasks.rados.rados.0.smithi115.stdout:2347: copy_from oid 43 from oid 16 current snap is 242 2022-04-22T17:33:52.922 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 242 2022-04-22T17:33:52.922 INFO:tasks.rados.rados.0.smithi115.stdout:2348: seq_num 708 ranges {5431296=663552,6094848=737280,6832128=696320,7528448=172032} 2022-04-22T17:33:52.956 INFO:tasks.rados.rados.0.smithi115.stdout:2348: writing smithi11541303-39 from 5431296 to 6094848 tid 1 2022-04-22T17:33:52.961 INFO:tasks.rados.rados.0.smithi115.stdout:2348: writing smithi11541303-39 from 6094848 to 6832128 tid 2 2022-04-22T17:33:52.964 INFO:tasks.rados.rados.0.smithi115.stdout:2348: writing smithi11541303-39 from 6832128 to 7528448 tid 3 2022-04-22T17:33:52.966 INFO:tasks.rados.rados.0.smithi115.stdout:2348: writing smithi11541303-39 from 7528448 to 7700480 tid 4 2022-04-22T17:33:52.966 INFO:tasks.rados.rados.0.smithi115.stdout:2349: snap_create 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 1 to smithi11541303-24 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2347: finishing copy_from to smithi11541303-43 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2347: got expected ENOENT (src dne) 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2347: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 2 to smithi11541303-24 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 3 to smithi11541303-24 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 4 to smithi11541303-24 2022-04-22T17:33:52.967 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 5 to smithi11541303-24 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:2345: finishing write tid 6 to smithi11541303-24 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1078 (ObjNum 707 snap 242 seq_num 707) dirty exists 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:2345: left oid 24 (ObjNum 707 snap 242 seq_num 707) 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:2346: finishing copy_from to smithi11541303-30 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:2346: got expected ENOENT (src dne) 2022-04-22T17:33:52.968 INFO:tasks.rados.rados.0.smithi115.stdout:2346: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:33:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 1 to smithi11541303-39 2022-04-22T17:33:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 2 to smithi11541303-39 2022-04-22T17:33:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 3 to smithi11541303-39 2022-04-22T17:33:52.973 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 4 to smithi11541303-39 2022-04-22T17:33:52.974 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 5 to smithi11541303-39 2022-04-22T17:33:52.975 INFO:tasks.rados.rados.0.smithi115.stdout:2348: finishing write tid 6 to smithi11541303-39 2022-04-22T17:33:52.975 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1124 (ObjNum 708 snap 242 seq_num 708) dirty exists 2022-04-22T17:33:52.975 INFO:tasks.rados.rados.0.smithi115.stdout:2348: left oid 39 (ObjNum 708 snap 242 seq_num 708) 2022-04-22T17:33:53.204 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:53 smithi121 bash[43565]: debug 2022-04-22T17:33:53.195+0000 7f0891f7a3c0 -1 Falling back to public interface 2022-04-22T17:33:53.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:52 smithi115 bash[45495]: cluster 2022-04-22T17:33:51.268323+0000 mgr.y (mgr.34107) 280 : cluster [DBG] pgmap v489: 121 pgs: 8 active+undersized+degraded, 2 peering, 32 active+undersized, 79 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 534 KiB/s wr, 3 op/s; 41/489 objects degraded (8.384%) 2022-04-22T17:33:53.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:52 smithi115 bash[45495]: cluster 2022-04-22T17:33:51.872407+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2022-04-22T17:33:53.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:52 smithi115 bash[46934]: cluster 2022-04-22T17:33:51.268323+0000 mgr.y (mgr.34107) 280 : cluster [DBG] pgmap v489: 121 pgs: 8 active+undersized+degraded, 2 peering, 32 active+undersized, 79 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 534 KiB/s wr, 3 op/s; 41/489 objects degraded (8.384%) 2022-04-22T17:33:53.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:52 smithi115 bash[46934]: cluster 2022-04-22T17:33:51.872407+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2022-04-22T17:33:53.348 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 6 2022-04-22T17:33:53.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3, 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 11 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:33:53.350 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:33:53.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:53.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:53.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:53.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:33:53.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:33:53.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:33:53.910 INFO:tasks.rados.rados.0.smithi115.stdout:2345: done (4 left) 2022-04-22T17:33:53.910 INFO:tasks.rados.rados.0.smithi115.stdout:2346: done (3 left) 2022-04-22T17:33:53.910 INFO:tasks.rados.rados.0.smithi115.stdout:2347: done (2 left) 2022-04-22T17:33:53.910 INFO:tasks.rados.rados.0.smithi115.stdout:2348: done (1 left) 2022-04-22T17:33:53.910 INFO:tasks.rados.rados.0.smithi115.stdout:2349: done (0 left) 2022-04-22T17:33:53.911 INFO:tasks.rados.rados.0.smithi115.stdout:2350: read oid 8 snap -1 2022-04-22T17:33:53.911 INFO:tasks.rados.rados.0.smithi115.stdout:2350: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:53.911 INFO:tasks.rados.rados.0.smithi115.stdout:2351: read oid 29 snap 236 2022-04-22T17:33:53.911 INFO:tasks.rados.rados.0.smithi115.stdout:2351: expect deleted 2022-04-22T17:33:53.911 INFO:tasks.rados.rados.0.smithi115.stdout:2352: snap_create 2022-04-22T17:33:53.916 INFO:tasks.rados.rados.0.smithi115.stdout:2350: expect (ObjNum 588 snap 190 seq_num 588) 2022-04-22T17:33:54.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:53 smithi121 bash[36902]: audit 2022-04-22T17:33:52.528993+0000 mgr.y (mgr.34107) 281 : audit [DBG] from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:53 smithi121 bash[36902]: cluster 2022-04-22T17:33:52.876920+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2022-04-22T17:33:54.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:53 smithi121 bash[36902]: audit 2022-04-22T17:33:52.878664+0000 mgr.y (mgr.34107) 282 : audit [DBG] from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:53 smithi121 bash[36902]: audit 2022-04-22T17:33:53.308865+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.115:0/499228864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:54.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[45495]: audit 2022-04-22T17:33:52.528993+0000 mgr.y (mgr.34107) 281 : audit [DBG] from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[45495]: cluster 2022-04-22T17:33:52.876920+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2022-04-22T17:33:54.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[45495]: audit 2022-04-22T17:33:52.878664+0000 mgr.y (mgr.34107) 282 : audit [DBG] from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[45495]: audit 2022-04-22T17:33:53.308865+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.115:0/499228864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:54.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[46934]: audit 2022-04-22T17:33:52.528993+0000 mgr.y (mgr.34107) 281 : audit [DBG] from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[46934]: cluster 2022-04-22T17:33:52.876920+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2022-04-22T17:33:54.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[46934]: audit 2022-04-22T17:33:52.878664+0000 mgr.y (mgr.34107) 282 : audit [DBG] from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:33:54.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:53 smithi115 bash[46934]: audit 2022-04-22T17:33:53.308865+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.115:0/499228864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:33:54.919 INFO:tasks.rados.rados.0.smithi115.stdout:2350: done (2 left) 2022-04-22T17:33:54.920 INFO:tasks.rados.rados.0.smithi115.stdout:2351: done (1 left) 2022-04-22T17:33:54.920 INFO:tasks.rados.rados.0.smithi115.stdout:2352: done (0 left) 2022-04-22T17:33:54.920 INFO:tasks.rados.rados.0.smithi115.stdout:2353: snap_create 2022-04-22T17:33:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:54 smithi121 bash[36902]: cluster 2022-04-22T17:33:53.268981+0000 mgr.y (mgr.34107) 283 : cluster [DBG] pgmap v492: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 8 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:54 smithi121 bash[36902]: cluster 2022-04-22T17:33:53.896058+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2022-04-22T17:33:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:54 smithi121 bash[36902]: audit 2022-04-22T17:33:54.179796+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:54 smithi121 bash[36902]: audit 2022-04-22T17:33:54.301297+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:54 smithi121 bash[36902]: audit 2022-04-22T17:33:54.749321+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[45495]: cluster 2022-04-22T17:33:53.268981+0000 mgr.y (mgr.34107) 283 : cluster [DBG] pgmap v492: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 8 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:55.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[45495]: cluster 2022-04-22T17:33:53.896058+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[45495]: audit 2022-04-22T17:33:54.179796+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[45495]: audit 2022-04-22T17:33:54.301297+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[45495]: audit 2022-04-22T17:33:54.749321+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[46934]: cluster 2022-04-22T17:33:53.268981+0000 mgr.y (mgr.34107) 283 : cluster [DBG] pgmap v492: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.4 MiB/s wr, 8 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[46934]: cluster 2022-04-22T17:33:53.896058+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[46934]: audit 2022-04-22T17:33:54.179796+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[46934]: audit 2022-04-22T17:33:54.301297+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:33:55.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:54 smithi115 bash[46934]: audit 2022-04-22T17:33:54.749321+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:33:55.928 INFO:tasks.rados.rados.0.smithi115.stdout:2353: done (0 left) 2022-04-22T17:33:55.928 INFO:tasks.rados.rados.0.smithi115.stdout:2354: delete oid 49 current snap is 245 2022-04-22T17:33:55.937 INFO:tasks.rados.rados.0.smithi115.stdout:2354: done (0 left) 2022-04-22T17:33:55.937 INFO:tasks.rados.rados.0.smithi115.stdout:2355: delete oid 41 current snap is 245 2022-04-22T17:33:55.945 INFO:tasks.rados.rados.0.smithi115.stdout:2355: done (0 left) 2022-04-22T17:33:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:2356: copy_from oid 45 from oid 18 current snap is 245 2022-04-22T17:33:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 245 2022-04-22T17:33:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:2357: seq_num 711 ranges {0=712704,712704=786432,1499136=516096,2015232=303104} 2022-04-22T17:33:55.951 INFO:tasks.rados.rados.0.smithi115.stdout:2357: writing smithi11541303-6 from 0 to 712704 tid 1 2022-04-22T17:33:55.956 INFO:tasks.rados.rados.0.smithi115.stdout:2357: writing smithi11541303-6 from 712704 to 1499136 tid 2 2022-04-22T17:33:55.959 INFO:tasks.rados.rados.0.smithi115.stdout:2357: writing smithi11541303-6 from 1499136 to 2015232 tid 3 2022-04-22T17:33:55.962 INFO:tasks.rados.rados.0.smithi115.stdout:2357: writing smithi11541303-6 from 2015232 to 2318336 tid 4 2022-04-22T17:33:55.962 INFO:tasks.rados.rados.0.smithi115.stdout:2358: copy_from oid 41 from oid 11 current snap is 245 2022-04-22T17:33:55.962 INFO:tasks.rados.rados.0.smithi115.stdout:2359: setattr oid 37 current snap is 245 2022-04-22T17:33:55.962 INFO:tasks.rados.rados.0.smithi115.stdout:2356: finishing copy_from to smithi11541303-45 2022-04-22T17:33:55.963 INFO:tasks.rados.rados.0.smithi115.stdout:2356: got expected ENOENT (src dne) 2022-04-22T17:33:55.963 INFO:tasks.rados.rados.0.smithi115.stdout:2356: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:33:55.963 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 1 to smithi11541303-6 2022-04-22T17:33:55.963 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 2 to smithi11541303-6 2022-04-22T17:33:55.965 INFO:tasks.rados.rados.0.smithi115.stdout:2356: done (3 left) 2022-04-22T17:33:55.965 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 245 2022-04-22T17:33:55.965 INFO:tasks.rados.rados.0.smithi115.stdout:2360: seq_num 712 ranges {7593984=729088,8323072=671744,8994816=671744,9666560=262144} 2022-04-22T17:33:56.015 INFO:tasks.rados.rados.0.smithi115.stdout:2360: writing smithi11541303-47 from 7593984 to 8323072 tid 1 2022-04-22T17:33:56.020 INFO:tasks.rados.rados.0.smithi115.stdout:2360: writing smithi11541303-47 from 8323072 to 8994816 tid 2 2022-04-22T17:33:56.024 INFO:tasks.rados.rados.0.smithi115.stdout:2360: writing smithi11541303-47 from 8994816 to 9666560 tid 3 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2360: writing smithi11541303-47 from 9666560 to 9928704 tid 4 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2361: rollback oid 31 current snap is 245 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 231 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 3 to smithi11541303-6 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2362: setattr oid 26 current snap is 245 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 4 to smithi11541303-6 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 5 to smithi11541303-6 2022-04-22T17:33:56.027 INFO:tasks.rados.rados.0.smithi115.stdout:2357: finishing write tid 6 to smithi11541303-6 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1083 (ObjNum 711 snap 245 seq_num 711) dirty exists 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:2357: left oid 6 (ObjNum 711 snap 245 seq_num 711) 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1125 (ObjNum 5 snap 1 seq_num 0) dirty exists 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:2358: finishing copy_from to smithi11541303-41 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1204 (ObjNum 690 snap 229 seq_num 690) dirty exists 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:2358: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 1 to smithi11541303-47 2022-04-22T17:33:56.028 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 2 to smithi11541303-47 2022-04-22T17:33:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:2357: done (5 left) 2022-04-22T17:33:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:2358: done (4 left) 2022-04-22T17:33:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:2359: done (3 left) 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2363: copy_from oid 18 from oid 4 current snap is 245 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2364: rmattr oid 6 current snap is 245 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2364: done (4 left) 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2365: copy_from oid 30 from oid 48 current snap is 245 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2366: read oid 43 snap -1 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2366: expect (ObjNum 681 snap 227 seq_num 681) 2022-04-22T17:33:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:2367: delete oid 10 current snap is 245 2022-04-22T17:33:56.031 INFO:tasks.rados.rados.0.smithi115.stdout:2365: finishing copy_from to smithi11541303-30 2022-04-22T17:33:56.032 INFO:tasks.rados.rados.0.smithi115.stdout:2365: got expected ENOENT (src dne) 2022-04-22T17:33:56.032 INFO:tasks.rados.rados.0.smithi115.stdout:2365: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:33:56.032 INFO:tasks.rados.rados.0.smithi115.stdout:2361: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:33:56.034 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 3 to smithi11541303-47 2022-04-22T17:33:56.034 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 4 to smithi11541303-47 2022-04-22T17:33:56.034 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 5 to smithi11541303-47 2022-04-22T17:33:56.034 INFO:tasks.rados.rados.0.smithi115.stdout:2361: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:33:56.034 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1337 (ObjNum -1116219328 snap 22075 seq_num 3179250704) dirty exists 2022-04-22T17:33:56.035 INFO:tasks.rados.rados.0.smithi115.stdout:2361: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:33:56.035 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1538 (ObjNum 697 snap 230 seq_num 697) dirty exists 2022-04-22T17:33:56.036 INFO:tasks.rados.rados.0.smithi115.stdout:2361: done (6 left) 2022-04-22T17:33:56.036 INFO:tasks.rados.rados.0.smithi115.stdout:2362: done (5 left) 2022-04-22T17:33:56.036 INFO:tasks.rados.rados.0.smithi115.stdout:2365: done (4 left) 2022-04-22T17:33:56.036 INFO:tasks.rados.rados.0.smithi115.stdout:2367: done (3 left) 2022-04-22T17:33:56.036 INFO:tasks.rados.rados.0.smithi115.stdout:2368: rmattr oid 26 current snap is 245 2022-04-22T17:33:56.037 INFO:tasks.rados.rados.0.smithi115.stdout:2369: delete oid 36 current snap is 245 2022-04-22T17:33:56.037 INFO:tasks.rados.rados.0.smithi115.stdout:2360: finishing write tid 6 to smithi11541303-47 2022-04-22T17:33:56.037 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1209 (ObjNum 712 snap 245 seq_num 712) dirty exists 2022-04-22T17:33:56.037 INFO:tasks.rados.rados.0.smithi115.stdout:2360: left oid 47 (ObjNum 712 snap 245 seq_num 712) 2022-04-22T17:33:56.039 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1338 (ObjNum -1115665440 snap 22075 seq_num 3179299328) dirty exists 2022-04-22T17:33:56.041 INFO:tasks.rados.rados.0.smithi115.stdout:2360: done (4 left) 2022-04-22T17:33:56.041 INFO:tasks.rados.rados.0.smithi115.stdout:2368: done (3 left) 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2369: done (2 left) 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2370: rollback oid 24 current snap is 245 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 241 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2371: rollback oid 22 current snap is 245 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 22 to 240 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2372: read oid 33 snap 233 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2372: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2373: read oid 38 snap 213 2022-04-22T17:33:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:2373: expect deleted 2022-04-22T17:33:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:2374: rollback oid 21 current snap is 245 2022-04-22T17:33:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 240 2022-04-22T17:33:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:2375: rollback oid 25 current snap is 245 2022-04-22T17:33:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 243 2022-04-22T17:33:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:2376: snap_create 2022-04-22T17:33:56.045 INFO:tasks.rados.rados.0.smithi115.stdout:2370: finishing rollback tid 0 to smithi11541303-24 2022-04-22T17:33:56.045 INFO:tasks.rados.rados.0.smithi115.stdout:2370: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:33:56.045 INFO:tasks.rados.rados.0.smithi115.stdout:2370: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:33:56.045 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1086 (ObjNum 599 snap 195 seq_num 599) dirty exists 2022-04-22T17:33:56.046 INFO:tasks.rados.rados.0.smithi115.stdout:2366: expect (ObjNum 681 snap 227 seq_num 681) 2022-04-22T17:33:56.106 INFO:tasks.rados.rados.0.smithi115.stdout:2375: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:33:56.106 INFO:tasks.rados.rados.0.smithi115.stdout:2371: finishing rollback tid 0 to smithi11541303-22 2022-04-22T17:33:56.107 INFO:tasks.rados.rados.0.smithi115.stdout:2371: finishing rollback tid 1 to smithi11541303-22 2022-04-22T17:33:56.107 INFO:tasks.rados.rados.0.smithi115.stdout:2371: finishing rollback tid 2 to smithi11541303-22 2022-04-22T17:33:56.107 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1127 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:33:56.108 INFO:tasks.rados.rados.0.smithi115.stdout:2374: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:33:56.108 INFO:tasks.rados.rados.0.smithi115.stdout:2374: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:2374: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1130 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:2375: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:2375: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1246 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:33:56.109 INFO:tasks.rados.rados.0.smithi115.stdout:2363: finishing copy_from to smithi11541303-18 2022-04-22T17:33:56.110 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1205 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:33:56.110 INFO:tasks.rados.rados.0.smithi115.stdout:2363: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:33:56.110 INFO:tasks.rados.rados.0.smithi115.stdout:2372: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:33:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:55 smithi121 bash[36902]: cluster 2022-04-22T17:33:54.905175+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2022-04-22T17:33:56.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:55 smithi115 bash[45495]: cluster 2022-04-22T17:33:54.905175+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2022-04-22T17:33:56.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:55 smithi115 bash[46934]: cluster 2022-04-22T17:33:54.905175+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2022-04-22T17:33:56.936 INFO:tasks.rados.rados.0.smithi115.stdout:2363: done (8 left) 2022-04-22T17:33:56.936 INFO:tasks.rados.rados.0.smithi115.stdout:2366: done (7 left) 2022-04-22T17:33:56.936 INFO:tasks.rados.rados.0.smithi115.stdout:2370: done (6 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2371: done (5 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2372: done (4 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2373: done (3 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2374: done (2 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2375: done (1 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2376: done (0 left) 2022-04-22T17:33:56.937 INFO:tasks.rados.rados.0.smithi115.stdout:2377: copy_from oid 12 from oid 43 current snap is 246 2022-04-22T17:33:56.938 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 246 2022-04-22T17:33:56.938 INFO:tasks.rados.rados.0.smithi115.stdout:2378: seq_num 715 ranges {0=778240,778240=770048,1548288=499712,2048000=335872} 2022-04-22T17:33:56.943 INFO:tasks.rados.rados.0.smithi115.stdout:2378: writing smithi11541303-49 from 0 to 778240 tid 1 2022-04-22T17:33:56.947 INFO:tasks.rados.rados.0.smithi115.stdout:2378: writing smithi11541303-49 from 778240 to 1548288 tid 2 2022-04-22T17:33:56.951 INFO:tasks.rados.rados.0.smithi115.stdout:2378: writing smithi11541303-49 from 1548288 to 2048000 tid 3 2022-04-22T17:33:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:2378: writing smithi11541303-49 from 2048000 to 2383872 tid 4 2022-04-22T17:33:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:2379: copy_from oid 6 from oid 21 current snap is 246 2022-04-22T17:33:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 1 to smithi11541303-49 2022-04-22T17:33:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 2 to smithi11541303-49 2022-04-22T17:33:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:2380: snap_remove snap 238 2022-04-22T17:33:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:56 smithi121 bash[36902]: cluster 2022-04-22T17:33:55.269916+0000 mgr.y (mgr.34107) 284 : cluster [DBG] pgmap v495: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:56 smithi121 bash[36902]: cluster 2022-04-22T17:33:55.914252+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2022-04-22T17:33:57.268 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:57 smithi121 bash[43565]: debug 2022-04-22T17:33:57.011+0000 7f0891f7a3c0 -1 osd.6 528 log_to_monitors true 2022-04-22T17:33:57.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:56 smithi115 bash[45495]: cluster 2022-04-22T17:33:55.269916+0000 mgr.y (mgr.34107) 284 : cluster [DBG] pgmap v495: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:57.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:56 smithi115 bash[45495]: cluster 2022-04-22T17:33:55.914252+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2022-04-22T17:33:57.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:56 smithi115 bash[46934]: cluster 2022-04-22T17:33:55.269916+0000 mgr.y (mgr.34107) 284 : cluster [DBG] pgmap v495: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 316 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 42/501 objects degraded (8.383%) 2022-04-22T17:33:57.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:56 smithi115 bash[46934]: cluster 2022-04-22T17:33:55.914252+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2022-04-22T17:33:57.950 INFO:tasks.rados.rados.0.smithi115.stdout:2380: done (3 left) 2022-04-22T17:33:57.951 INFO:tasks.rados.rados.0.smithi115.stdout:2381: snap_remove snap 240 2022-04-22T17:33:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:57 smithi121 bash[36902]: cluster 2022-04-22T17:33:56.921406+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2022-04-22T17:33:58.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:57 smithi121 bash[36902]: audit 2022-04-22T17:33:57.009167+0000 mon.a (mon.0) 933 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:57 smithi121 bash[36902]: audit 2022-04-22T17:33:57.019702+0000 mon.b (mon.2) 7 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.310 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 22 17:33:57 smithi121 bash[43565]: debug 2022-04-22T17:33:57.991+0000 7f0883315700 -1 osd.6 528 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:33:58.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[45495]: cluster 2022-04-22T17:33:56.921406+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2022-04-22T17:33:58.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[45495]: audit 2022-04-22T17:33:57.009167+0000 mon.a (mon.0) 933 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[45495]: audit 2022-04-22T17:33:57.019702+0000 mon.b (mon.2) 7 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[46934]: cluster 2022-04-22T17:33:56.921406+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2022-04-22T17:33:58.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[46934]: audit 2022-04-22T17:33:57.009167+0000 mon.a (mon.0) 933 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:57 smithi115 bash[46934]: audit 2022-04-22T17:33:57.019702+0000 mon.b (mon.2) 7 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-04-22T17:33:58.955 INFO:tasks.rados.rados.0.smithi115.stdout:2381: done (3 left) 2022-04-22T17:33:58.956 INFO:tasks.rados.rados.0.smithi115.stdout:2382: read oid 20 snap 244 2022-04-22T17:33:58.956 INFO:tasks.rados.rados.0.smithi115.stdout:2382: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:33:58.956 INFO:tasks.rados.rados.0.smithi115.stdout:2383: read oid 40 snap -1 2022-04-22T17:33:58.956 INFO:tasks.rados.rados.0.smithi115.stdout:2383: expect deleted 2022-04-22T17:33:58.957 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 246 2022-04-22T17:33:58.957 INFO:tasks.rados.rados.0.smithi115.stdout:2384: seq_num 716 ranges {7700480=794624,8495104=655360,9150464=704512,9854976=245760} 2022-04-22T17:33:59.006 INFO:tasks.rados.rados.0.smithi115.stdout:2384: writing smithi11541303-39 from 7700480 to 8495104 tid 1 2022-04-22T17:33:59.041 INFO:tasks.rados.rados.0.smithi115.stdout:2384: writing smithi11541303-39 from 8495104 to 9150464 tid 2 2022-04-22T17:33:59.041 INFO:tasks.rados.rados.0.smithi115.stdout:2384: writing smithi11541303-39 from 9150464 to 9854976 tid 3 2022-04-22T17:33:59.041 INFO:tasks.rados.rados.0.smithi115.stdout:2384: writing smithi11541303-39 from 9854976 to 10100736 tid 4 2022-04-22T17:33:59.042 INFO:tasks.rados.rados.0.smithi115.stdout:2385: read oid 25 snap -1 2022-04-22T17:33:59.042 INFO:tasks.rados.rados.0.smithi115.stdout:2385: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:33:59.042 INFO:tasks.rados.rados.0.smithi115.stdout:2386: snap_remove snap 235 2022-04-22T17:33:59.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: cluster 2022-04-22T17:33:57.271039+0000 mgr.y (mgr.34107) 285 : cluster [DBG] pgmap v498: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 18 op/s; 42/516 objects degraded (8.140%) 2022-04-22T17:33:59.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: cluster 2022-04-22T17:33:57.918315+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: Degraded data redundancy: 42/516 objects degraded (8.140%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:59.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: audit 2022-04-22T17:33:57.936375+0000 mon.a (mon.0) 935 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:33:59.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: cluster 2022-04-22T17:33:57.936459+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2022-04-22T17:33:59.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: audit 2022-04-22T17:33:57.941262+0000 mon.a (mon.0) 937 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:58 smithi121 bash[36902]: audit 2022-04-22T17:33:57.948498+0000 mon.b (mon.2) 8 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: cluster 2022-04-22T17:33:57.271039+0000 mgr.y (mgr.34107) 285 : cluster [DBG] pgmap v498: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 18 op/s; 42/516 objects degraded (8.140%) 2022-04-22T17:33:59.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: cluster 2022-04-22T17:33:57.918315+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: Degraded data redundancy: 42/516 objects degraded (8.140%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:59.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: audit 2022-04-22T17:33:57.936375+0000 mon.a (mon.0) 935 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: cluster 2022-04-22T17:33:57.936459+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: audit 2022-04-22T17:33:57.941262+0000 mon.a (mon.0) 937 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[45495]: audit 2022-04-22T17:33:57.948498+0000 mon.b (mon.2) 8 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: cluster 2022-04-22T17:33:57.271039+0000 mgr.y (mgr.34107) 285 : cluster [DBG] pgmap v498: 121 pgs: 8 active+undersized+degraded, 34 active+undersized, 79 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 18 op/s; 42/516 objects degraded (8.140%) 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: cluster 2022-04-22T17:33:57.918315+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: Degraded data redundancy: 42/516 objects degraded (8.140%), 8 pgs degraded (PG_DEGRADED) 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: audit 2022-04-22T17:33:57.936375+0000 mon.a (mon.0) 935 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-04-22T17:33:59.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: cluster 2022-04-22T17:33:57.936459+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2022-04-22T17:33:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: audit 2022-04-22T17:33:57.941262+0000 mon.a (mon.0) 937 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:58 smithi115 bash[46934]: audit 2022-04-22T17:33:57.948498+0000 mon.b (mon.2) 8 : audit [INF] from='osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2386: done (7 left) 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2387: delete oid 42 current snap is 246 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 3 to smithi11541303-49 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 4 to smithi11541303-49 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 5 to smithi11541303-49 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2377: finishing copy_from to smithi11541303-12 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1153 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:33:59.957 INFO:tasks.rados.rados.0.smithi115.stdout:2377: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2378: finishing write tid 6 to smithi11541303-49 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1544 (ObjNum 715 snap 246 seq_num 715) dirty exists 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2378: left oid 49 (ObjNum 715 snap 246 seq_num 715) 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2379: finishing copy_from to smithi11541303-6 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1130 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2379: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 1 to smithi11541303-39 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 2 to smithi11541303-39 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 3 to smithi11541303-39 2022-04-22T17:33:59.958 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 4 to smithi11541303-39 2022-04-22T17:33:59.959 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 5 to smithi11541303-39 2022-04-22T17:33:59.959 INFO:tasks.rados.rados.0.smithi115.stdout:2384: finishing write tid 6 to smithi11541303-39 2022-04-22T17:33:59.959 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1130 (ObjNum 716 snap 246 seq_num 716) dirty exists 2022-04-22T17:33:59.959 INFO:tasks.rados.rados.0.smithi115.stdout:2384: left oid 39 (ObjNum 716 snap 246 seq_num 716) 2022-04-22T17:33:59.987 INFO:tasks.rados.rados.0.smithi115.stdout:2377: done (7 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2378: done (6 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2379: done (5 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2383: done (4 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2384: done (3 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2387: done (2 left) 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2388: copy_from oid 35 from oid 42 current snap is 246 2022-04-22T17:33:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2389: delete oid 49 current snap is 246 2022-04-22T17:33:59.998 INFO:tasks.rados.rados.0.smithi115.stdout:2388: finishing copy_from to smithi11541303-35 2022-04-22T17:33:59.998 INFO:tasks.rados.rados.0.smithi115.stdout:2388: got expected ENOENT (src dne) 2022-04-22T17:33:59.998 INFO:tasks.rados.rados.0.smithi115.stdout:2388: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:2388: done (3 left) 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:2389: done (2 left) 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:2390: read oid 31 snap -1 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:2390: expect (ObjNum 697 snap 230 seq_num 697) 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:2391: rollback oid 14 current snap is 246 2022-04-22T17:33:59.999 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 243 2022-04-22T17:34:00.000 INFO:tasks.rados.rados.0.smithi115.stdout:2392: copy_from oid 28 from oid 12 current snap is 246 2022-04-22T17:34:00.000 INFO:tasks.rados.rados.0.smithi115.stdout:2393: delete oid 24 current snap is 246 2022-04-22T17:34:00.004 INFO:tasks.rados.rados.0.smithi115.stdout:2393: done (5 left) 2022-04-22T17:34:00.004 INFO:tasks.rados.rados.0.smithi115.stdout:2394: snap_remove snap 220 2022-04-22T17:34:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:59 smithi121 bash[36902]: cluster 2022-04-22T17:33:58.932152+0000 mon.a (mon.0) 938 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:59 smithi121 bash[36902]: cluster 2022-04-22T17:33:58.936636+0000 mon.a (mon.0) 939 : cluster [INF] osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214] boot 2022-04-22T17:34:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:59 smithi121 bash[36902]: cluster 2022-04-22T17:33:58.936691+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2022-04-22T17:34:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:59 smithi121 bash[36902]: audit 2022-04-22T17:33:58.939745+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:34:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:33:59 smithi121 bash[36902]: audit 2022-04-22T17:33:59.304182+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:00.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[45495]: cluster 2022-04-22T17:33:58.932152+0000 mon.a (mon.0) 938 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:00.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[45495]: cluster 2022-04-22T17:33:58.936636+0000 mon.a (mon.0) 939 : cluster [INF] osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214] boot 2022-04-22T17:34:00.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[45495]: cluster 2022-04-22T17:33:58.936691+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[45495]: audit 2022-04-22T17:33:58.939745+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[45495]: audit 2022-04-22T17:33:59.304182+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[46934]: cluster 2022-04-22T17:33:58.932152+0000 mon.a (mon.0) 938 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[46934]: cluster 2022-04-22T17:33:58.936636+0000 mon.a (mon.0) 939 : cluster [INF] osd.6 [v2:172.21.15.121:6816/75880214,v1:172.21.15.121:6817/75880214] boot 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[46934]: cluster 2022-04-22T17:33:58.936691+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[46934]: audit 2022-04-22T17:33:58.939745+0000 mon.a (mon.0) 941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-04-22T17:34:00.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:33:59 smithi115 bash[46934]: audit 2022-04-22T17:33:59.304182+0000 mon.a (mon.0) 942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2394: done (5 left) 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2395: read oid 9 snap -1 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2395: expect (ObjNum 702 snap 234 seq_num 702) 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2396: snap_create 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2391: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:34:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2391: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2391: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1124 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2392: finishing copy_from to smithi11541303-28 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1153 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2392: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:34:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2382: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:34:01.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:00 smithi121 bash[36902]: cluster 2022-04-22T17:33:59.272015+0000 mgr.y (mgr.34107) 286 : cluster [DBG] pgmap v501: 121 pgs: 7 peering, 5 active+undersized+degraded, 30 active+undersized, 79 active+clean; 331 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 17 op/s; 28/531 objects degraded (5.273%) 2022-04-22T17:34:01.237 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:00 smithi121 bash[36902]: cluster 2022-04-22T17:33:59.942048+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2022-04-22T17:34:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:00 smithi115 bash[45495]: cluster 2022-04-22T17:33:59.272015+0000 mgr.y (mgr.34107) 286 : cluster [DBG] pgmap v501: 121 pgs: 7 peering, 5 active+undersized+degraded, 30 active+undersized, 79 active+clean; 331 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 17 op/s; 28/531 objects degraded (5.273%) 2022-04-22T17:34:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:00 smithi115 bash[45495]: cluster 2022-04-22T17:33:59.942048+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2022-04-22T17:34:01.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:00 smithi115 bash[46934]: cluster 2022-04-22T17:33:59.272015+0000 mgr.y (mgr.34107) 286 : cluster [DBG] pgmap v501: 121 pgs: 7 peering, 5 active+undersized+degraded, 30 active+undersized, 79 active+clean; 331 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 17 op/s; 28/531 objects degraded (5.273%) 2022-04-22T17:34:01.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:00 smithi115 bash[46934]: cluster 2022-04-22T17:33:59.942048+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2022-04-22T17:34:01.856 INFO:tasks.rados.rados.0.smithi115.stdout:2385: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:01.905 INFO:tasks.rados.rados.0.smithi115.stdout:2390: expect (ObjNum 697 snap 230 seq_num 697) 2022-04-22T17:34:01.969 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:01 smithi115 bash[45495]: cluster 2022-04-22T17:34:00.951324+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2022-04-22T17:34:01.970 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:01.629Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:01.970 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:01.629Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:01.970 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:01.629Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:01 smithi121 bash[36902]: cluster 2022-04-22T17:34:00.951324+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2022-04-22T17:34:02.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:01 smithi115 bash[46934]: cluster 2022-04-22T17:34:00.951324+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2022-04-22T17:34:02.774 INFO:tasks.rados.rados.0.smithi115.stdout:2395: expect (ObjNum 702 snap 234 seq_num 702) 2022-04-22T17:34:02.875 INFO:tasks.rados.rados.0.smithi115.stdout:2382: done (6 left) 2022-04-22T17:34:02.914 INFO:tasks.rados.rados.0.smithi115.stdout:2385: done (5 left) 2022-04-22T17:34:02.914 INFO:tasks.rados.rados.0.smithi115.stdout:2390: done (4 left) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2391: done (3 left) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2392: done (2 left) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2395: done (1 left) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2396: done (0 left) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2397: read oid 28 snap 229 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2397: expect (ObjNum 657 snap 213 seq_num 657) 2022-04-22T17:34:02.915 INFO:tasks.rados.rados.0.smithi115.stdout:2398: snap_remove snap 241 2022-04-22T17:34:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:2398: done (1 left) 2022-04-22T17:34:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:2399: rmattr oid 17 current snap is 247 2022-04-22T17:34:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:2399: done (1 left) 2022-04-22T17:34:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:2400: copy_from oid 26 from oid 25 current snap is 247 2022-04-22T17:34:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:2397: expect (ObjNum 657 snap 213 seq_num 657) 2022-04-22T17:34:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:02 smithi121 bash[36902]: cluster 2022-04-22T17:34:01.272755+0000 mgr.y (mgr.34107) 287 : cluster [DBG] pgmap v504: 121 pgs: 11 peering, 4 active+undersized+degraded, 24 active+undersized, 82 active+clean; 343 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 27/540 objects degraded (5.000%) 2022-04-22T17:34:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:02 smithi121 bash[36902]: cluster 2022-04-22T17:34:01.961304+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2022-04-22T17:34:03.085 INFO:tasks.rados.rados.0.smithi115.stdout:2400: finishing copy_from to smithi11541303-26 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1246 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:2400: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:2397: done (1 left) 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:2400: done (0 left) 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:2401: read oid 25 snap -1 2022-04-22T17:34:03.131 INFO:tasks.rados.rados.0.smithi115.stdout:2401: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:03.132 INFO:tasks.rados.rados.0.smithi115.stdout:2402: read oid 42 snap -1 2022-04-22T17:34:03.132 INFO:tasks.rados.rados.0.smithi115.stdout:2402: expect deleted 2022-04-22T17:34:03.132 INFO:tasks.rados.rados.0.smithi115.stdout:2403: setattr oid 16 current snap is 247 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2402: done (2 left) 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2404: read oid 30 snap 228 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2404: expect (ObjNum 656 snap 213 seq_num 656) 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2405: copy_from oid 34 from oid 9 current snap is 247 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 247 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2406: seq_num 720 ranges {1982464=65536} 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2406: writing smithi11541303-12 from 1982464 to 2048000 tid 1 2022-04-22T17:34:03.133 INFO:tasks.rados.rados.0.smithi115.stdout:2407: snap_remove snap 205 2022-04-22T17:34:03.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:02 smithi115 bash[45495]: cluster 2022-04-22T17:34:01.272755+0000 mgr.y (mgr.34107) 287 : cluster [DBG] pgmap v504: 121 pgs: 11 peering, 4 active+undersized+degraded, 24 active+undersized, 82 active+clean; 343 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 27/540 objects degraded (5.000%) 2022-04-22T17:34:03.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:02 smithi115 bash[45495]: cluster 2022-04-22T17:34:01.961304+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2022-04-22T17:34:03.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:02 smithi115 bash[46934]: cluster 2022-04-22T17:34:01.272755+0000 mgr.y (mgr.34107) 287 : cluster [DBG] pgmap v504: 121 pgs: 11 peering, 4 active+undersized+degraded, 24 active+undersized, 82 active+clean; 343 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 27/540 objects degraded (5.000%) 2022-04-22T17:34:03.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:02 smithi115 bash[46934]: cluster 2022-04-22T17:34:01.961304+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2022-04-22T17:34:03.801 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[42914]: debug 2022-04-22T17:34:03.554+0000 7f8eea62c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2022-04-22T17:34:03.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:03.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:03.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:03.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:03.802 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:03.524Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:03.998 INFO:tasks.rados.rados.0.smithi115.stdout:2407: done (5 left) 2022-04-22T17:34:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:2408: copy_from oid 19 from oid 45 current snap is 247 2022-04-22T17:34:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1117 (ObjNum 4 snap 3 seq_num 7) dirty exists 2022-04-22T17:34:03.999 INFO:tasks.rados.rados.0.smithi115.stdout:2401: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:04.067 INFO:tasks.rados.rados.0.smithi115.stdout:2404: expect (ObjNum 656 snap 213 seq_num 656) 2022-04-22T17:34:04.104 INFO:tasks.rados.rados.0.smithi115.stdout:2406: finishing write tid 1 to smithi11541303-12 2022-04-22T17:34:04.104 INFO:tasks.rados.rados.0.smithi115.stdout:2406: finishing write tid 2 to smithi11541303-12 2022-04-22T17:34:04.104 INFO:tasks.rados.rados.0.smithi115.stdout:2406: finishing write tid 3 to smithi11541303-12 2022-04-22T17:34:04.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1155 (ObjNum 720 snap 247 seq_num 720) dirty exists 2022-04-22T17:34:04.104 INFO:tasks.rados.rados.0.smithi115.stdout:2406: left oid 12 (ObjNum 720 snap 247 seq_num 720) 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:2405: finishing copy_from to smithi11541303-34 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1532 (ObjNum 702 snap 234 seq_num 702) dirty exists 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:2405: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:2408: finishing copy_from to smithi11541303-19 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1207 (ObjNum 623 snap 204 seq_num 623) dirty exists 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:2408: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:34:04.105 INFO:tasks.rados.rados.0.smithi115.stdout:2401: done (5 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2403: done (4 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2404: done (3 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2405: done (2 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2406: done (1 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2408: done (0 left) 2022-04-22T17:34:04.106 INFO:tasks.rados.rados.0.smithi115.stdout:2409: rmattr oid 3 current snap is 247 2022-04-22T17:34:04.107 INFO:tasks.rados.rados.0.smithi115.stdout:2409: done (0 left) 2022-04-22T17:34:04.107 INFO:tasks.rados.rados.0.smithi115.stdout:2410: snap_create 2022-04-22T17:34:04.266 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[24219]: debug 2022-04-22T17:34:04.011+0000 7f1df544c700 -1 osd.7 pg_epoch: 547 pg[5.9s0( v 547'1257 (0'0,547'1257] local-lis/les=542/543 n=9 ec=63/63 lis/c=542/542 les/c/f=543/543/0 sis=542) [7,6,1]p7(0) r=0 lpr=542 luod=547'1255 crt=547'1255 lcod 547'1253 mlcod 547'1253 active+clean+snaptrim trimq=[190~1] ps=[1ab~1,1cd~1,1d0~1,1d2~2]] removing snap head 2022-04-22T17:34:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: cluster 2022-04-22T17:34:02.973907+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.531255+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.539981+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.548176+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.551288+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.553116+0000 mon.a (mon.0) 951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.554583+0000 mon.a (mon.0) 952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:03 smithi121 bash[36902]: audit 2022-04-22T17:34:03.556626+0000 mon.a (mon.0) 953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:04.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: cluster 2022-04-22T17:34:02.973907+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2022-04-22T17:34:04.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.531255+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.539981+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.548176+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.551288+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.553116+0000 mon.a (mon.0) 951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.554583+0000 mon.a (mon.0) 952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[45495]: audit 2022-04-22T17:34:03.556626+0000 mon.a (mon.0) 953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: cluster 2022-04-22T17:34:02.973907+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.531255+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.539981+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:04.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.548176+0000 mon.a (mon.0) 949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:04.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.551288+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.553116+0000 mon.a (mon.0) 951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.554583+0000 mon.a (mon.0) 952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:04.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:03 smithi115 bash[46934]: audit 2022-04-22T17:34:03.556626+0000 mon.a (mon.0) 953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:05.007 INFO:tasks.rados.rados.0.smithi115.stdout:2410: done (0 left) 2022-04-22T17:34:05.007 INFO:tasks.rados.rados.0.smithi115.stdout:2411: rollback oid 50 current snap is 248 2022-04-22T17:34:05.007 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 244 2022-04-22T17:34:05.007 INFO:tasks.rados.rados.0.smithi115.stdout:2412: read oid 14 snap -1 2022-04-22T17:34:05.007 INFO:tasks.rados.rados.0.smithi115.stdout:2412: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:34:05.008 INFO:tasks.rados.rados.0.smithi115.stdout:2413: setattr oid 45 current snap is 248 2022-04-22T17:34:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:2414: read oid 1 snap 246 2022-04-22T17:34:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:2414: expect (ObjNum -1116710832 snap 22075 seq_num 3178911216) 2022-04-22T17:34:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:2415: copy_from oid 47 from oid 37 current snap is 248 2022-04-22T17:34:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:2416: copy_from oid 26 from oid 12 current snap is 248 2022-04-22T17:34:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:2417: snap_remove snap 232 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cluster 2022-04-22T17:34:03.273820+0000 mgr.y (mgr.34107) 288 : cluster [DBG] pgmap v507: 121 pgs: 1 active+clean+snaptrim, 11 peering, 6 active+undersized, 103 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.6 MiB/s wr, 18 op/s; 2.3 MiB/s, 0 objects/s recovering 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cephadm 2022-04-22T17:34:03.519997+0000 mgr.y (mgr.34107) 289 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: audit 2022-04-22T17:34:03.556972+0000 mgr.y (mgr.34107) 290 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cephadm 2022-04-22T17:34:03.558461+0000 mgr.y (mgr.34107) 291 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cluster 2022-04-22T17:34:03.966653+0000 mon.a (mon.0) 954 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/540 objects degraded (5.000%), 4 pgs degraded) 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cluster 2022-04-22T17:34:03.966731+0000 mon.a (mon.0) 955 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: cluster 2022-04-22T17:34:03.983507+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2022-04-22T17:34:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:04 smithi121 bash[36902]: audit 2022-04-22T17:34:04.307157+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cluster 2022-04-22T17:34:03.273820+0000 mgr.y (mgr.34107) 288 : cluster [DBG] pgmap v507: 121 pgs: 1 active+clean+snaptrim, 11 peering, 6 active+undersized, 103 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.6 MiB/s wr, 18 op/s; 2.3 MiB/s, 0 objects/s recovering 2022-04-22T17:34:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cephadm 2022-04-22T17:34:03.519997+0000 mgr.y (mgr.34107) 289 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: audit 2022-04-22T17:34:03.556972+0000 mgr.y (mgr.34107) 290 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cephadm 2022-04-22T17:34:03.558461+0000 mgr.y (mgr.34107) 291 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cluster 2022-04-22T17:34:03.966653+0000 mon.a (mon.0) 954 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/540 objects degraded (5.000%), 4 pgs degraded) 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cluster 2022-04-22T17:34:03.966731+0000 mon.a (mon.0) 955 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: cluster 2022-04-22T17:34:03.983507+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[45495]: audit 2022-04-22T17:34:04.307157+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cluster 2022-04-22T17:34:03.273820+0000 mgr.y (mgr.34107) 288 : cluster [DBG] pgmap v507: 121 pgs: 1 active+clean+snaptrim, 11 peering, 6 active+undersized, 103 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.6 MiB/s wr, 18 op/s; 2.3 MiB/s, 0 objects/s recovering 2022-04-22T17:34:05.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cephadm 2022-04-22T17:34:03.519997+0000 mgr.y (mgr.34107) 289 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: audit 2022-04-22T17:34:03.556972+0000 mgr.y (mgr.34107) 290 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cephadm 2022-04-22T17:34:03.558461+0000 mgr.y (mgr.34107) 291 : cephadm [INF] Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cluster 2022-04-22T17:34:03.966653+0000 mon.a (mon.0) 954 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/540 objects degraded (5.000%), 4 pgs degraded) 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cluster 2022-04-22T17:34:03.966731+0000 mon.a (mon.0) 955 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: cluster 2022-04-22T17:34:03.983507+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2022-04-22T17:34:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:04 smithi115 bash[46934]: audit 2022-04-22T17:34:04.307157+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:06.017 INFO:tasks.rados.rados.0.smithi115.stdout:2417: done (6 left) 2022-04-22T17:34:06.017 INFO:tasks.rados.rados.0.smithi115.stdout:2418: copy_from oid 15 from oid 27 current snap is 248 2022-04-22T17:34:06.017 INFO:tasks.rados.rados.0.smithi115.stdout:2414: expect (ObjNum -1116710832 snap 22075 seq_num 3178911216) 2022-04-22T17:34:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1261 (ObjNum 623 snap 204 seq_num 623) dirty exists 2022-04-22T17:34:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:2411: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:34:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:2411: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:34:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:2411: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1535 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:2415: finishing copy_from to smithi11541303-47 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1125 (ObjNum 0 snap 32692 seq_num 3178796784) dirty exists 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:2415: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:2416: finishing copy_from to smithi11541303-26 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1155 (ObjNum 720 snap 247 seq_num 720) dirty exists 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:2416: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:34:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:2412: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2418: finishing copy_from to smithi11541303-15 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1247 (ObjNum 1 snap 22075 seq_num 3178748336) dirty exists 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2418: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2411: done (6 left) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2412: done (5 left) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2413: done (4 left) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2414: done (3 left) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2415: done (2 left) 2022-04-22T17:34:06.228 INFO:tasks.rados.rados.0.smithi115.stdout:2416: done (1 left) 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2418: done (0 left) 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2419: read oid 47 snap -1 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2419: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2420: read oid 20 snap -1 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2420: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2421: copy_from oid 31 from oid 46 current snap is 248 2022-04-22T17:34:06.229 INFO:tasks.rados.rados.0.smithi115.stdout:2422: setattr oid 18 current snap is 248 2022-04-22T17:34:06.230 INFO:tasks.rados.rados.0.smithi115.stdout:2419: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:06.238 INFO:tasks.rados.rados.0.smithi115.stdout:2421: finishing copy_from to smithi11541303-31 2022-04-22T17:34:06.238 INFO:tasks.rados.rados.0.smithi115.stdout:2421: got expected ENOENT (src dne) 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2421: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2419: done (3 left) 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2421: done (2 left) 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2423: read oid 4 snap -1 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2423: expect (ObjNum 667 snap 222 seq_num 667) 2022-04-22T17:34:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:2424: setattr oid 43 current snap is 248 2022-04-22T17:34:06.241 INFO:tasks.rados.rados.0.smithi115.stdout:2425: read oid 17 snap -1 2022-04-22T17:34:06.241 INFO:tasks.rados.rados.0.smithi115.stdout:2425: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:06.241 INFO:tasks.rados.rados.0.smithi115.stdout:2426: setattr oid 5 current snap is 248 2022-04-22T17:34:06.245 INFO:tasks.rados.rados.0.smithi115.stdout:2427: read oid 39 snap -1 2022-04-22T17:34:06.245 INFO:tasks.rados.rados.0.smithi115.stdout:2427: expect (ObjNum 716 snap 246 seq_num 716) 2022-04-22T17:34:06.245 INFO:tasks.rados.rados.0.smithi115.stdout:2428: snap_remove snap 242 2022-04-22T17:34:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:06 smithi121 bash[36902]: cluster 2022-04-22T17:34:04.993339+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2022-04-22T17:34:06.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:05 smithi115 bash[45495]: cluster 2022-04-22T17:34:04.993339+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2022-04-22T17:34:06.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:05 smithi115 bash[46934]: cluster 2022-04-22T17:34:04.993339+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2022-04-22T17:34:07.028 INFO:tasks.rados.rados.0.smithi115.stdout:2428: done (7 left) 2022-04-22T17:34:07.028 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 248 2022-04-22T17:34:07.028 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1595 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:34:07.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1596 (ObjNum -1115696032 snap 22075 seq_num 3179953984) dirty exists 2022-04-22T17:34:07.029 INFO:tasks.rados.rados.0.smithi115.stdout:2423: expect (ObjNum 667 snap 222 seq_num 667) 2022-04-22T17:34:07.169 INFO:tasks.rados.rados.0.smithi115.stdout:2429: seq_num 721 ranges {253952=81920} 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2429: writing smithi11541303-8 from 253952 to 335872 tid 1 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2422: done (7 left) 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2423: done (6 left) 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2426: done (5 left) 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2430: snap_create 2022-04-22T17:34:07.171 INFO:tasks.rados.rados.0.smithi115.stdout:2420: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:34:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:07 smithi121 bash[36902]: cluster 2022-04-22T17:34:05.274643+0000 mgr.y (mgr.34107) 292 : cluster [DBG] pgmap v510: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 317 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.0 MiB/s wr, 9 op/s; 2.3 MiB/s, 1 objects/s recovering 2022-04-22T17:34:07.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:07 smithi121 bash[36902]: cluster 2022-04-22T17:34:06.002952+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2022-04-22T17:34:07.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:07 smithi115 bash[46934]: cluster 2022-04-22T17:34:05.274643+0000 mgr.y (mgr.34107) 292 : cluster [DBG] pgmap v510: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 317 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.0 MiB/s wr, 9 op/s; 2.3 MiB/s, 1 objects/s recovering 2022-04-22T17:34:07.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:07 smithi115 bash[46934]: cluster 2022-04-22T17:34:06.002952+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2022-04-22T17:34:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:07 smithi115 bash[45495]: cluster 2022-04-22T17:34:05.274643+0000 mgr.y (mgr.34107) 292 : cluster [DBG] pgmap v510: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 317 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.0 MiB/s wr, 9 op/s; 2.3 MiB/s, 1 objects/s recovering 2022-04-22T17:34:07.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:07 smithi115 bash[45495]: cluster 2022-04-22T17:34:06.002952+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2420: done (5 left) 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2430: done (4 left) 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2431: read oid 29 snap -1 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2431: expect deleted 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2432: rmattr oid 3 current snap is 249 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:2432: done (5 left) 2022-04-22T17:34:08.041 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 249 2022-04-22T17:34:08.042 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1272 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:34:08.042 INFO:tasks.rados.rados.0.smithi115.stdout:2425: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:08.255 INFO:tasks.rados.rados.0.smithi115.stdout:2427: expect (ObjNum 716 snap 246 seq_num 716) 2022-04-22T17:34:08.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:08 smithi115 bash[46934]: cluster 2022-04-22T17:34:07.013037+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2022-04-22T17:34:08.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:08 smithi115 bash[45495]: cluster 2022-04-22T17:34:07.013037+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2022-04-22T17:34:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:08 smithi121 bash[36902]: cluster 2022-04-22T17:34:07.013037+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:2429: finishing write tid 1 to smithi11541303-8 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:2429: finishing write tid 2 to smithi11541303-8 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:2429: finishing write tid 3 to smithi11541303-8 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1214 (ObjNum 721 snap 248 seq_num 721) dirty exists 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:2429: left oid 8 (ObjNum 721 snap 248 seq_num 721) 2022-04-22T17:34:08.786 INFO:tasks.rados.rados.0.smithi115.stdout:2433: seq_num 722 ranges {0=98304} 2022-04-22T17:34:08.787 INFO:tasks.rados.rados.0.smithi115.stdout:2433: writing smithi11541303-1 from 0 to 98304 tid 1 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2424: done (5 left) 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2425: done (4 left) 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2427: done (3 left) 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2429: done (2 left) 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2431: done (1 left) 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2434: rollback oid 24 current snap is 249 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 244 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:2435: rollback oid 6 current snap is 249 2022-04-22T17:34:08.788 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 228 2022-04-22T17:34:08.789 INFO:tasks.rados.rados.0.smithi115.stdout:2436: snap_create 2022-04-22T17:34:08.791 INFO:tasks.rados.rados.0.smithi115.stdout:2434: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:34:08.792 INFO:tasks.rados.rados.0.smithi115.stdout:2433: finishing write tid 1 to smithi11541303-1 2022-04-22T17:34:08.793 INFO:tasks.rados.rados.0.smithi115.stdout:2433: finishing write tid 2 to smithi11541303-1 2022-04-22T17:34:08.793 INFO:tasks.rados.rados.0.smithi115.stdout:2434: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:34:08.793 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1133 (ObjNum 707 snap 242 seq_num 707) dirty exists 2022-04-22T17:34:08.794 INFO:tasks.rados.rados.0.smithi115.stdout:2435: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:34:08.794 INFO:tasks.rados.rados.0.smithi115.stdout:2435: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:34:08.794 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1135 (ObjNum 1330995052 snap 1449028174 seq_num 1934776934) dirty dne 2022-04-22T17:34:08.795 INFO:tasks.rados.rados.0.smithi115.stdout:2433: finishing write tid 3 to smithi11541303-1 2022-04-22T17:34:08.796 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1216 (ObjNum 722 snap 249 seq_num 722) dirty exists 2022-04-22T17:34:08.796 INFO:tasks.rados.rados.0.smithi115.stdout:2433: left oid 1 (ObjNum 722 snap 249 seq_num 722) 2022-04-22T17:34:09.049 INFO:tasks.rados.rados.0.smithi115.stdout:2433: done (3 left) 2022-04-22T17:34:09.049 INFO:tasks.rados.rados.0.smithi115.stdout:2434: done (2 left) 2022-04-22T17:34:09.049 INFO:tasks.rados.rados.0.smithi115.stdout:2435: done (1 left) 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2436: done (0 left) 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2437: copy_from oid 22 from oid 29 current snap is 250 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2438: read oid 10 snap -1 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2438: expect deleted 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 250 2022-04-22T17:34:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2439: seq_num 723 ranges {3366912=114688} 2022-04-22T17:34:09.072 INFO:tasks.rados.rados.0.smithi115.stdout:2439: writing smithi11541303-34 from 3366912 to 3481600 tid 1 2022-04-22T17:34:09.072 INFO:tasks.rados.rados.0.smithi115.stdout:2440: rollback oid 16 current snap is 250 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 244 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2438: done (3 left) 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2441: read oid 38 snap -1 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2441: expect deleted 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2442: copy_from oid 15 from oid 50 current snap is 250 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2437: finishing copy_from to smithi11541303-22 2022-04-22T17:34:09.073 INFO:tasks.rados.rados.0.smithi115.stdout:2437: got expected ENOENT (src dne) 2022-04-22T17:34:09.074 INFO:tasks.rados.rados.0.smithi115.stdout:2437: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:34:09.074 INFO:tasks.rados.rados.0.smithi115.stdout:2437: done (4 left) 2022-04-22T17:34:09.074 INFO:tasks.rados.rados.0.smithi115.stdout:2443: rollback oid 48 current snap is 250 2022-04-22T17:34:09.074 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 48 to 227 2022-04-22T17:34:09.074 INFO:tasks.rados.rados.0.smithi115.stdout:2444: setattr oid 31 current snap is 250 2022-04-22T17:34:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:2443: finishing rollback tid 1 to smithi11541303-48 2022-04-22T17:34:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 0 (ObjNum -1116207920 snap 22075 seq_num 2815700032) dirty dne 2022-04-22T17:34:09.075 INFO:tasks.rados.rados.0.smithi115.stdout:2441: done (5 left) 2022-04-22T17:34:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2443: done (4 left) 2022-04-22T17:34:09.076 INFO:tasks.rados.rados.0.smithi115.stdout:2445: setattr oid 27 current snap is 250 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2446: read oid 7 snap -1 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2446: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2447: rmattr oid 22 current snap is 250 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2447: done (6 left) 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2448: copy_from oid 18 from oid 49 current snap is 250 2022-04-22T17:34:09.079 INFO:tasks.rados.rados.0.smithi115.stdout:2449: delete oid 13 current snap is 250 2022-04-22T17:34:09.080 INFO:tasks.rados.rados.0.smithi115.stdout:2439: finishing write tid 1 to smithi11541303-34 2022-04-22T17:34:09.082 INFO:tasks.rados.rados.0.smithi115.stdout:2439: finishing write tid 2 to smithi11541303-34 2022-04-22T17:34:09.087 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1267 (ObjNum 1382571874 snap 1465149535 seq_num 1550800480) dirty exists 2022-04-22T17:34:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:2439: finishing write tid 3 to smithi11541303-34 2022-04-22T17:34:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1537 (ObjNum 723 snap 250 seq_num 723) dirty exists 2022-04-22T17:34:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:2439: left oid 34 (ObjNum 723 snap 250 seq_num 723) 2022-04-22T17:34:09.090 INFO:tasks.rados.rados.0.smithi115.stdout:2448: finishing copy_from to smithi11541303-18 2022-04-22T17:34:09.090 INFO:tasks.rados.rados.0.smithi115.stdout:2448: got expected ENOENT (src dne) 2022-04-22T17:34:09.090 INFO:tasks.rados.rados.0.smithi115.stdout:2448: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2439: done (7 left) 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2445: done (6 left) 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2448: done (5 left) 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2449: done (4 left) 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2450: rollback oid 14 current snap is 250 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 237 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2451: read oid 23 snap -1 2022-04-22T17:34:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:2451: expect (ObjNum 679 snap 226 seq_num 679) 2022-04-22T17:34:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 250 2022-04-22T17:34:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:2452: seq_num 725 ranges {12034048=147456} 2022-04-22T17:34:09.160 INFO:tasks.rados.rados.0.smithi115.stdout:2452: writing smithi11541303-45 from 12034048 to 12181504 tid 1 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:2453: snap_create 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:2440: finishing rollback tid 0 to smithi11541303-16 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:2440: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1539 (ObjNum 1432644466 snap 1767466843 seq_num 1616664179) dirty dne 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1604 (ObjNum 697 snap 230 seq_num 697) dirty exists 2022-04-22T17:34:09.161 INFO:tasks.rados.rados.0.smithi115.stdout:2446: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:09.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:09 smithi115 bash[45495]: cluster 2022-04-22T17:34:07.275349+0000 mgr.y (mgr.34107) 293 : cluster [DBG] pgmap v513: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2022-04-22T17:34:09.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:09 smithi115 bash[45495]: cluster 2022-04-22T17:34:08.024270+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2022-04-22T17:34:09.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:09 smithi115 bash[46934]: cluster 2022-04-22T17:34:07.275349+0000 mgr.y (mgr.34107) 293 : cluster [DBG] pgmap v513: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2022-04-22T17:34:09.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:09 smithi115 bash[46934]: cluster 2022-04-22T17:34:08.024270+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2022-04-22T17:34:09.386 INFO:tasks.rados.rados.0.smithi115.stdout:2450: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:34:09.386 INFO:tasks.rados.rados.0.smithi115.stdout:2450: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:34:09.387 INFO:tasks.rados.rados.0.smithi115.stdout:2450: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:34:09.387 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1156 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:34:09.387 INFO:tasks.rados.rados.0.smithi115.stdout:2451: expect (ObjNum 679 snap 226 seq_num 679) 2022-04-22T17:34:09.501 INFO:tasks.rados.rados.0.smithi115.stdout:2442: finishing copy_from to smithi11541303-15 2022-04-22T17:34:09.501 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1535 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:09.501 INFO:tasks.rados.rados.0.smithi115.stdout:2442: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:34:09.501 INFO:tasks.rados.rados.0.smithi115.stdout:2452: finishing write tid 1 to smithi11541303-45 2022-04-22T17:34:09.502 INFO:tasks.rados.rados.0.smithi115.stdout:2452: finishing write tid 2 to smithi11541303-45 2022-04-22T17:34:09.502 INFO:tasks.rados.rados.0.smithi115.stdout:2452: finishing write tid 3 to smithi11541303-45 2022-04-22T17:34:09.502 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1270 (ObjNum 725 snap 250 seq_num 725) dirty exists 2022-04-22T17:34:09.502 INFO:tasks.rados.rados.0.smithi115.stdout:2452: left oid 45 (ObjNum 725 snap 250 seq_num 725) 2022-04-22T17:34:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:09 smithi121 bash[36902]: cluster 2022-04-22T17:34:07.275349+0000 mgr.y (mgr.34107) 293 : cluster [DBG] pgmap v513: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2022-04-22T17:34:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:09 smithi121 bash[36902]: cluster 2022-04-22T17:34:08.024270+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2022-04-22T17:34:09.890 INFO:tasks.rados.rados.0.smithi115.stdout:2440: done (7 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2442: done (6 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2444: done (5 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2446: done (4 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2450: done (3 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2451: done (2 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2452: done (1 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:2453: done (0 left) 2022-04-22T17:34:09.891 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 251 2022-04-22T17:34:09.892 INFO:tasks.rados.rados.0.smithi115.stdout:2454: seq_num 726 ranges {9093120=163840} 2022-04-22T17:34:09.945 INFO:tasks.rados.rados.0.smithi115.stdout:2454: writing smithi11541303-35 from 9093120 to 9256960 tid 1 2022-04-22T17:34:09.945 INFO:tasks.rados.rados.0.smithi115.stdout:2455: snap_create 2022-04-22T17:34:09.956 INFO:tasks.rados.rados.0.smithi115.stdout:2454: finishing write tid 1 to smithi11541303-35 2022-04-22T17:34:09.957 INFO:tasks.rados.rados.0.smithi115.stdout:2454: finishing write tid 2 to smithi11541303-35 2022-04-22T17:34:09.958 INFO:tasks.rados.rados.0.smithi115.stdout:2454: finishing write tid 3 to smithi11541303-35 2022-04-22T17:34:09.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1132 (ObjNum 726 snap 251 seq_num 726) dirty exists 2022-04-22T17:34:09.958 INFO:tasks.rados.rados.0.smithi115.stdout:2454: left oid 35 (ObjNum 726 snap 251 seq_num 726) 2022-04-22T17:34:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:10 smithi121 bash[36902]: cluster 2022-04-22T17:34:09.033999+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2022-04-22T17:34:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:10 smithi121 bash[36902]: audit 2022-04-22T17:34:09.310072+0000 mon.a (mon.0) 963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:10 smithi121 bash[36902]: cluster 2022-04-22T17:34:09.876460+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2022-04-22T17:34:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[45495]: cluster 2022-04-22T17:34:09.033999+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2022-04-22T17:34:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[45495]: audit 2022-04-22T17:34:09.310072+0000 mon.a (mon.0) 963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[45495]: cluster 2022-04-22T17:34:09.876460+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2022-04-22T17:34:10.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[46934]: cluster 2022-04-22T17:34:09.033999+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2022-04-22T17:34:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[46934]: audit 2022-04-22T17:34:09.310072+0000 mon.a (mon.0) 963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:10 smithi115 bash[46934]: cluster 2022-04-22T17:34:09.876460+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2022-04-22T17:34:10.901 INFO:tasks.rados.rados.0.smithi115.stdout:2454: done (1 left) 2022-04-22T17:34:10.901 INFO:tasks.rados.rados.0.smithi115.stdout:2455: done (0 left) 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2456: read oid 2 snap -1 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2456: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2457: rollback oid 9 current snap is 252 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 249 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2458: read oid 25 snap -1 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2458: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:10.902 INFO:tasks.rados.rados.0.smithi115.stdout:2459: delete oid 27 current snap is 252 2022-04-22T17:34:10.904 INFO:tasks.rados.rados.0.smithi115.stdout:2456: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:10.909 INFO:tasks.rados.rados.0.smithi115.stdout:2456: done (3 left) 2022-04-22T17:34:10.910 INFO:tasks.rados.rados.0.smithi115.stdout:2459: done (2 left) 2022-04-22T17:34:10.910 INFO:tasks.rados.rados.0.smithi115.stdout:2460: delete oid 11 current snap is 252 2022-04-22T17:34:10.914 INFO:tasks.rados.rados.0.smithi115.stdout:2457: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:34:10.914 INFO:tasks.rados.rados.0.smithi115.stdout:2460: done (2 left) 2022-04-22T17:34:10.915 INFO:tasks.rados.rados.0.smithi115.stdout:2461: snap_remove snap 243 2022-04-22T17:34:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:11 smithi121 bash[36902]: cluster 2022-04-22T17:34:09.276100+0000 mgr.y (mgr.34107) 294 : cluster [DBG] pgmap v516: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 3.7 MiB/s wr, 43 op/s 2022-04-22T17:34:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:11 smithi121 bash[36902]: cluster 2022-04-22T17:34:10.886074+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2022-04-22T17:34:11.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[45495]: cluster 2022-04-22T17:34:09.276100+0000 mgr.y (mgr.34107) 294 : cluster [DBG] pgmap v516: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 3.7 MiB/s wr, 43 op/s 2022-04-22T17:34:11.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[45495]: cluster 2022-04-22T17:34:10.886074+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2022-04-22T17:34:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[46934]: cluster 2022-04-22T17:34:09.276100+0000 mgr.y (mgr.34107) 294 : cluster [DBG] pgmap v516: 121 pgs: 121 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 3.7 MiB/s wr, 43 op/s 2022-04-22T17:34:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[46934]: cluster 2022-04-22T17:34:10.886074+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2022-04-22T17:34:11.905 INFO:tasks.rados.rados.0.smithi115.stdout:2461: done (2 left) 2022-04-22T17:34:11.905 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 252 2022-04-22T17:34:11.905 INFO:tasks.rados.rados.0.smithi115.stdout:2457: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:34:11.905 INFO:tasks.rados.rados.0.smithi115.stdout:2457: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:34:11.905 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 1608 (ObjNum 702 snap 234 seq_num 702) dirty exists 2022-04-22T17:34:11.906 INFO:tasks.rados.rados.0.smithi115.stdout:2458: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:11.971 INFO:tasks.rados.rados.0.smithi115.stdout:2462: seq_num 729 ranges {0=212992} 2022-04-22T17:34:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:2462: writing smithi11541303-38 from 0 to 212992 tid 1 2022-04-22T17:34:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:2457: done (2 left) 2022-04-22T17:34:11.977 INFO:tasks.rados.rados.0.smithi115.stdout:2458: done (1 left) 2022-04-22T17:34:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 252 2022-04-22T17:34:11.978 INFO:tasks.rados.rados.0.smithi115.stdout:2463: seq_num 730 ranges {4964352=229376} 2022-04-22T17:34:12.010 INFO:tasks.rados.rados.0.smithi115.stdout:2463: writing smithi11541303-24 from 4964352 to 5193728 tid 1 2022-04-22T17:34:12.103 INFO:tasks.rados.rados.0.smithi115.stdout:2464: rollback oid 13 current snap is 252 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 227 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:2462: finishing write tid 1 to smithi11541303-38 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:2462: finishing write tid 2 to smithi11541303-38 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:2462: finishing write tid 3 to smithi11541303-38 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1209 (ObjNum 729 snap 252 seq_num 729) dirty exists 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:2462: left oid 38 (ObjNum 729 snap 252 seq_num 729) 2022-04-22T17:34:12.104 INFO:tasks.rados.rados.0.smithi115.stdout:2462: done (2 left) 2022-04-22T17:34:12.105 INFO:tasks.rados.rados.0.smithi115.stdout:2465: delete oid 45 current snap is 252 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:2464: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 0 (ObjNum -1116272768 snap 22075 seq_num 3179377904) dirty dne 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:2463: finishing write tid 1 to smithi11541303-24 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:2463: finishing write tid 2 to smithi11541303-24 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:2464: done (2 left) 2022-04-22T17:34:12.106 INFO:tasks.rados.rados.0.smithi115.stdout:2465: done (1 left) 2022-04-22T17:34:12.107 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 252 2022-04-22T17:34:12.107 INFO:tasks.rados.rados.0.smithi115.stdout:2466: seq_num 732 ranges {10919936=262144} 2022-04-22T17:34:12.107 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:11.629Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:12.107 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:11.629Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:12.107 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:11.629Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2466: writing smithi11541303-31 from 10919936 to 11182080 tid 1 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2467: delete oid 44 current snap is 252 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2463: finishing write tid 3 to smithi11541303-24 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1137 (ObjNum 730 snap 252 seq_num 730) dirty exists 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2463: left oid 24 (ObjNum 730 snap 252 seq_num 730) 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2463: done (2 left) 2022-04-22T17:34:12.108 INFO:tasks.rados.rados.0.smithi115.stdout:2467: done (1 left) 2022-04-22T17:34:12.109 INFO:tasks.rados.rados.0.smithi115.stdout:2468: snap_remove snap 229 2022-04-22T17:34:13.064 INFO:tasks.rados.rados.0.smithi115.stdout:2468: done (1 left) 2022-04-22T17:34:13.065 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 252 2022-04-22T17:34:13.065 INFO:tasks.rados.rados.0.smithi115.stdout:2469: seq_num 734 ranges {10100736=294912} 2022-04-22T17:34:13.126 INFO:tasks.rados.rados.0.smithi115.stdout:2469: writing smithi11541303-39 from 10100736 to 10395648 tid 1 2022-04-22T17:34:13.126 INFO:tasks.rados.rados.0.smithi115.stdout:2470: snap_remove snap 226 2022-04-22T17:34:13.126 INFO:tasks.rados.rados.0.smithi115.stdout:2466: finishing write tid 1 to smithi11541303-31 2022-04-22T17:34:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:12 smithi121 bash[36902]: cluster 2022-04-22T17:34:11.277037+0000 mgr.y (mgr.34107) 295 : cluster [DBG] pgmap v519: 121 pgs: 121 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2022-04-22T17:34:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:12 smithi121 bash[36902]: cluster 2022-04-22T17:34:11.889175+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2022-04-22T17:34:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:12 smithi115 bash[45495]: cluster 2022-04-22T17:34:11.277037+0000 mgr.y (mgr.34107) 295 : cluster [DBG] pgmap v519: 121 pgs: 121 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2022-04-22T17:34:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:12 smithi115 bash[45495]: cluster 2022-04-22T17:34:11.889175+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2022-04-22T17:34:13.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:12 smithi115 bash[46934]: cluster 2022-04-22T17:34:11.277037+0000 mgr.y (mgr.34107) 295 : cluster [DBG] pgmap v519: 121 pgs: 121 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2022-04-22T17:34:13.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:12 smithi115 bash[46934]: cluster 2022-04-22T17:34:11.889175+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2022-04-22T17:34:13.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:13.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:13.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:13.524Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:13.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:13.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2470: done (2 left) 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2471: copy_from oid 29 from oid 2 current snap is 252 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2466: finishing write tid 2 to smithi11541303-31 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2466: finishing write tid 3 to smithi11541303-31 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1619 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2466: left oid 31 (ObjNum 732 snap 252 seq_num 732) 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2469: finishing write tid 1 to smithi11541303-39 2022-04-22T17:34:14.074 INFO:tasks.rados.rados.0.smithi115.stdout:2469: finishing write tid 2 to smithi11541303-39 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2469: finishing write tid 3 to smithi11541303-39 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1134 (ObjNum 734 snap 252 seq_num 734) dirty exists 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2469: left oid 39 (ObjNum 734 snap 252 seq_num 734) 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2466: done (2 left) 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2469: done (1 left) 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2472: read oid 42 snap 247 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2472: expect deleted 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2473: copy_from oid 10 from oid 27 current snap is 252 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2474: rmattr oid 37 current snap is 252 2022-04-22T17:34:14.075 INFO:tasks.rados.rados.0.smithi115.stdout:2475: rmattr oid 18 current snap is 252 2022-04-22T17:34:14.076 INFO:tasks.rados.rados.0.smithi115.stdout:2476: snap_create 2022-04-22T17:34:14.081 INFO:tasks.rados.rados.0.smithi115.stdout:2473: finishing copy_from to smithi11541303-10 2022-04-22T17:34:14.081 INFO:tasks.rados.rados.0.smithi115.stdout:2473: got expected ENOENT (src dne) 2022-04-22T17:34:14.083 INFO:tasks.rados.rados.0.smithi115.stdout:2473: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:34:14.090 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1135 (ObjNum 1 snap 22075 seq_num 3179257120) dirty exists 2022-04-22T17:34:14.094 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1629 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:34:14.096 INFO:tasks.rados.rados.0.smithi115.stdout:2471: finishing copy_from to smithi11541303-29 2022-04-22T17:34:14.096 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1111 (ObjNum -1116523328 snap 22075 seq_num 3179071728) dirty exists 2022-04-22T17:34:14.096 INFO:tasks.rados.rados.0.smithi115.stdout:2471: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:34:14.516 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:14 smithi121 bash[24219]: debug 2022-04-22T17:34:14.067+0000 7f1df3c49700 -1 osd.7 pg_epoch: 557 pg[5.4s0( v 556'1083 (0'0,556'1083] local-lis/les=501/502 n=12 ec=63/63 lis/c=501/501 les/c/f=502/502/0 sis=501) [7,2,5]p7(0) r=0 lpr=501 crt=556'1083 lcod 556'1081 mlcod 556'1081 active+clean+snaptrim trimq=[1b9~1] ps=[190~1,1c1~1,1ca~1,1d3~3]] removing snap head 2022-04-22T17:34:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:14 smithi121 bash[36902]: cluster 2022-04-22T17:34:13.050758+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2022-04-22T17:34:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:14 smithi121 bash[36902]: audit 2022-04-22T17:34:13.283565+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:34:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:14 smithi121 bash[36902]: audit 2022-04-22T17:34:13.284034+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:34:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:14 smithi121 bash[36902]: audit 2022-04-22T17:34:13.409701+0000 mon.a (mon.0) 970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[45495]: cluster 2022-04-22T17:34:13.050758+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[45495]: audit 2022-04-22T17:34:13.283565+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[45495]: audit 2022-04-22T17:34:13.284034+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[45495]: audit 2022-04-22T17:34:13.409701+0000 mon.a (mon.0) 970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[46934]: cluster 2022-04-22T17:34:13.050758+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2022-04-22T17:34:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[46934]: audit 2022-04-22T17:34:13.283565+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:34:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[46934]: audit 2022-04-22T17:34:13.284034+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:34:14.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:14 smithi115 bash[46934]: audit 2022-04-22T17:34:13.409701+0000 mon.a (mon.0) 970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:14.894 INFO:tasks.rados.rados.0.smithi115.stdout:2471: done (5 left) 2022-04-22T17:34:14.898 INFO:tasks.rados.rados.0.smithi115.stdout:2472: done (4 left) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2473: done (3 left) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2474: done (2 left) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2475: done (1 left) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2476: done (0 left) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2477: read oid 19 snap -1 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2477: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 253 2022-04-22T17:34:14.899 INFO:tasks.rados.rados.0.smithi115.stdout:2478: seq_num 735 ranges {1966080=311296} 2022-04-22T17:34:14.908 INFO:tasks.rados.rados.0.smithi115.stdout:2478: writing smithi11541303-41 from 1966080 to 2277376 tid 1 2022-04-22T17:34:14.908 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 253 2022-04-22T17:34:14.908 INFO:tasks.rados.rados.0.smithi115.stdout:2479: seq_num 736 ranges {3751936=335872} 2022-04-22T17:34:14.932 INFO:tasks.rados.rados.0.smithi115.stdout:2479: writing smithi11541303-7 from 3751936 to 4087808 tid 1 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:2480: snap_create 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:2478: finishing write tid 1 to smithi11541303-41 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:2478: finishing write tid 2 to smithi11541303-41 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:2478: finishing write tid 3 to smithi11541303-41 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1211 (ObjNum 735 snap 253 seq_num 735) dirty exists 2022-04-22T17:34:14.933 INFO:tasks.rados.rados.0.smithi115.stdout:2478: left oid 41 (ObjNum 735 snap 253 seq_num 735) 2022-04-22T17:34:14.948 INFO:tasks.rados.rados.0.smithi115.stdout:2477: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:15 smithi121 bash[36902]: cluster 2022-04-22T17:34:13.277813+0000 mgr.y (mgr.34107) 296 : cluster [DBG] pgmap v522: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 612 KiB/s wr, 15 op/s 2022-04-22T17:34:15.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:15 smithi121 bash[36902]: cluster 2022-04-22T17:34:14.058221+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2022-04-22T17:34:15.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:15 smithi121 bash[36902]: audit 2022-04-22T17:34:14.313168+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:15.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:15 smithi121 bash[36902]: cluster 2022-04-22T17:34:14.880040+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2022-04-22T17:34:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[46934]: cluster 2022-04-22T17:34:13.277813+0000 mgr.y (mgr.34107) 296 : cluster [DBG] pgmap v522: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 612 KiB/s wr, 15 op/s 2022-04-22T17:34:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[46934]: cluster 2022-04-22T17:34:14.058221+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2022-04-22T17:34:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[46934]: audit 2022-04-22T17:34:14.313168+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[46934]: cluster 2022-04-22T17:34:14.880040+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2022-04-22T17:34:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[45495]: cluster 2022-04-22T17:34:13.277813+0000 mgr.y (mgr.34107) 296 : cluster [DBG] pgmap v522: 121 pgs: 121 active+clean; 294 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 612 KiB/s wr, 15 op/s 2022-04-22T17:34:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[45495]: cluster 2022-04-22T17:34:14.058221+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2022-04-22T17:34:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[45495]: audit 2022-04-22T17:34:14.313168+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:15 smithi115 bash[45495]: cluster 2022-04-22T17:34:14.880040+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2022-04-22T17:34:16.127 INFO:tasks.rados.rados.0.smithi115.stdout:2479: finishing write tid 1 to smithi11541303-7 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2479: finishing write tid 2 to smithi11541303-7 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2479: finishing write tid 3 to smithi11541303-7 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1218 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2479: left oid 7 (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2477: done (3 left) 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2478: done (2 left) 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2479: done (1 left) 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2480: done (0 left) 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2481: read oid 29 snap -1 2022-04-22T17:34:16.128 INFO:tasks.rados.rados.0.smithi115.stdout:2481: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:16.129 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 254 2022-04-22T17:34:16.129 INFO:tasks.rados.rados.0.smithi115.stdout:2482: seq_num 737 ranges {0=352256} 2022-04-22T17:34:16.132 INFO:tasks.rados.rados.0.smithi115.stdout:2482: writing smithi11541303-36 from 0 to 352256 tid 1 2022-04-22T17:34:16.132 INFO:tasks.rados.rados.0.smithi115.stdout:2483: snap_remove snap 233 2022-04-22T17:34:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:2483: done (2 left) 2022-04-22T17:34:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 254 2022-04-22T17:34:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:2481: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:17.085 INFO:tasks.rados.rados.0.smithi115.stdout:2482: finishing write tid 1 to smithi11541303-36 2022-04-22T17:34:17.085 INFO:tasks.rados.rados.0.smithi115.stdout:2482: finishing write tid 2 to smithi11541303-36 2022-04-22T17:34:17.085 INFO:tasks.rados.rados.0.smithi115.stdout:2482: finishing write tid 3 to smithi11541303-36 2022-04-22T17:34:17.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1220 (ObjNum 737 snap 254 seq_num 737) dirty exists 2022-04-22T17:34:17.085 INFO:tasks.rados.rados.0.smithi115.stdout:2482: left oid 36 (ObjNum 737 snap 254 seq_num 737) 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:2484: seq_num 738 ranges {0=368640} 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:2484: writing smithi11541303-45 from 0 to 368640 tid 1 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:2481: done (2 left) 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:2482: done (1 left) 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 254 2022-04-22T17:34:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:2485: seq_num 739 ranges {10395648=385024} 2022-04-22T17:34:17.150 INFO:tasks.rados.rados.0.smithi115.stdout:2485: writing smithi11541303-39 from 10395648 to 10780672 tid 1 2022-04-22T17:34:17.150 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 254 2022-04-22T17:34:17.150 INFO:tasks.rados.rados.0.smithi115.stdout:2486: seq_num 740 ranges {5193728=401408} 2022-04-22T17:34:17.181 INFO:tasks.rados.rados.0.smithi115.stdout:2486: writing smithi11541303-24 from 5193728 to 5595136 tid 1 2022-04-22T17:34:17.181 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 254 2022-04-22T17:34:17.182 INFO:tasks.rados.rados.0.smithi115.stdout:2487: seq_num 741 ranges {0=417792} 2022-04-22T17:34:17.184 INFO:tasks.rados.rados.0.smithi115.stdout:2487: writing smithi11541303-48 from 0 to 417792 tid 1 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2488: rollback oid 3 current snap is 254 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 246 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2484: finishing write tid 1 to smithi11541303-45 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2489: snap_create 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2484: finishing write tid 2 to smithi11541303-45 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2484: finishing write tid 3 to smithi11541303-45 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1286 (ObjNum 738 snap 254 seq_num 738) dirty exists 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2484: left oid 45 (ObjNum 738 snap 254 seq_num 738) 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2485: finishing write tid 1 to smithi11541303-39 2022-04-22T17:34:17.185 INFO:tasks.rados.rados.0.smithi115.stdout:2485: finishing write tid 2 to smithi11541303-39 2022-04-22T17:34:17.186 INFO:tasks.rados.rados.0.smithi115.stdout:2485: finishing write tid 3 to smithi11541303-39 2022-04-22T17:34:17.186 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1137 (ObjNum 739 snap 254 seq_num 739) dirty exists 2022-04-22T17:34:17.186 INFO:tasks.rados.rados.0.smithi115.stdout:2485: left oid 39 (ObjNum 739 snap 254 seq_num 739) 2022-04-22T17:34:17.188 INFO:tasks.rados.rados.0.smithi115.stdout:2486: finishing write tid 1 to smithi11541303-24 2022-04-22T17:34:17.188 INFO:tasks.rados.rados.0.smithi115.stdout:2486: finishing write tid 2 to smithi11541303-24 2022-04-22T17:34:17.190 INFO:tasks.rados.rados.0.smithi115.stdout:2486: finishing write tid 3 to smithi11541303-24 2022-04-22T17:34:17.191 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1139 (ObjNum 740 snap 254 seq_num 740) dirty exists 2022-04-22T17:34:17.191 INFO:tasks.rados.rados.0.smithi115.stdout:2486: left oid 24 (ObjNum 740 snap 254 seq_num 740) 2022-04-22T17:34:17.192 INFO:tasks.rados.rados.0.smithi115.stdout:2487: finishing write tid 1 to smithi11541303-48 2022-04-22T17:34:17.193 INFO:tasks.rados.rados.0.smithi115.stdout:2488: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:34:17.193 INFO:tasks.rados.rados.0.smithi115.stdout:2488: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:34:17.193 INFO:tasks.rados.rados.0.smithi115.stdout:2487: finishing write tid 2 to smithi11541303-48 2022-04-22T17:34:17.195 INFO:tasks.rados.rados.0.smithi115.stdout:2488: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:34:17.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1211 (ObjNum 625 snap 204 seq_num 625) dirty exists 2022-04-22T17:34:17.196 INFO:tasks.rados.rados.0.smithi115.stdout:2487: finishing write tid 3 to smithi11541303-48 2022-04-22T17:34:17.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1288 (ObjNum 741 snap 254 seq_num 741) dirty exists 2022-04-22T17:34:17.196 INFO:tasks.rados.rados.0.smithi115.stdout:2487: left oid 48 (ObjNum 741 snap 254 seq_num 741) 2022-04-22T17:34:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:16 smithi121 bash[36902]: cluster 2022-04-22T17:34:15.278624+0000 mgr.y (mgr.34107) 297 : cluster [DBG] pgmap v525: 121 pgs: 121 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1018 KiB/s wr, 14 op/s 2022-04-22T17:34:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:16 smithi121 bash[36902]: cluster 2022-04-22T17:34:15.897509+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2022-04-22T17:34:17.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:16 smithi115 bash[45495]: cluster 2022-04-22T17:34:15.278624+0000 mgr.y (mgr.34107) 297 : cluster [DBG] pgmap v525: 121 pgs: 121 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1018 KiB/s wr, 14 op/s 2022-04-22T17:34:17.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:16 smithi115 bash[45495]: cluster 2022-04-22T17:34:15.897509+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2022-04-22T17:34:17.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:16 smithi115 bash[46934]: cluster 2022-04-22T17:34:15.278624+0000 mgr.y (mgr.34107) 297 : cluster [DBG] pgmap v525: 121 pgs: 121 active+clean; 295 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1018 KiB/s wr, 14 op/s 2022-04-22T17:34:17.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:16 smithi115 bash[46934]: cluster 2022-04-22T17:34:15.897509+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2022-04-22T17:34:18.090 INFO:tasks.rados.rados.0.smithi115.stdout:2484: done (5 left) 2022-04-22T17:34:18.109 INFO:tasks.rados.rados.0.smithi115.stdout:2485: done (4 left) 2022-04-22T17:34:18.109 INFO:tasks.rados.rados.0.smithi115.stdout:2486: done (3 left) 2022-04-22T17:34:18.109 INFO:tasks.rados.rados.0.smithi115.stdout:2487: done (2 left) 2022-04-22T17:34:18.109 INFO:tasks.rados.rados.0.smithi115.stdout:2488: done (1 left) 2022-04-22T17:34:18.110 INFO:tasks.rados.rados.0.smithi115.stdout:2489: done (0 left) 2022-04-22T17:34:18.110 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 255 2022-04-22T17:34:18.110 INFO:tasks.rados.rados.0.smithi115.stdout:2490: seq_num 742 ranges {0=434176} 2022-04-22T17:34:18.111 INFO:tasks.rados.rados.0.smithi115.stdout:2490: writing smithi11541303-10 from 0 to 434176 tid 1 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2491: read oid 21 snap -1 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2491: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2492: copy_from oid 3 from oid 39 current snap is 255 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2493: read oid 13 snap -1 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2493: expect deleted 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2494: rollback oid 8 current snap is 255 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 249 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2495: rmattr oid 9 current snap is 255 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:2495: done (5 left) 2022-04-22T17:34:18.112 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 255 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:2496: seq_num 743 ranges {98304=450560} 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:2496: writing smithi11541303-1 from 98304 to 548864 tid 1 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:2497: read oid 28 snap -1 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:2497: expect (ObjNum 681 snap 227 seq_num 681) 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 255 2022-04-22T17:34:18.113 INFO:tasks.rados.rados.0.smithi115.stdout:2498: seq_num 744 ranges {6995968=466944} 2022-04-22T17:34:18.141 INFO:tasks.rados.rados.0.smithi115.stdout:2498: writing smithi11541303-33 from 6995968 to 7462912 tid 1 2022-04-22T17:34:18.141 INFO:tasks.rados.rados.0.smithi115.stdout:2499: snap_create 2022-04-22T17:34:18.141 INFO:tasks.rados.rados.0.smithi115.stdout:2497: expect (ObjNum 681 snap 227 seq_num 681) 2022-04-22T17:34:18.203 INFO:tasks.rados.rados.0.smithi115.stdout:2494: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2494: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2494: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1223 (ObjNum 721 snap 248 seq_num 721) dirty exists 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2496: finishing write tid 1 to smithi11541303-1 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2496: finishing write tid 2 to smithi11541303-1 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2490: finishing write tid 1 to smithi11541303-10 2022-04-22T17:34:18.204 INFO:tasks.rados.rados.0.smithi115.stdout:2490: finishing write tid 2 to smithi11541303-10 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:2490: finishing write tid 3 to smithi11541303-10 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1213 (ObjNum 742 snap 255 seq_num 742) dirty exists 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:2490: left oid 10 (ObjNum 742 snap 255 seq_num 742) 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:2496: finishing write tid 3 to smithi11541303-1 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1225 (ObjNum 743 snap 255 seq_num 743) dirty exists 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:2496: left oid 1 (ObjNum 743 snap 255 seq_num 743) 2022-04-22T17:34:18.205 INFO:tasks.rados.rados.0.smithi115.stdout:2491: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:18 smithi121 bash[36902]: cluster 2022-04-22T17:34:17.068098+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2022-04-22T17:34:18.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:18 smithi115 bash[46934]: cluster 2022-04-22T17:34:17.068098+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2022-04-22T17:34:18.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:18 smithi115 bash[45495]: cluster 2022-04-22T17:34:17.068098+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2022-04-22T17:34:19.064 INFO:tasks.rados.rados.0.smithi115.stdout:2498: finishing write tid 1 to smithi11541303-33 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:2498: finishing write tid 2 to smithi11541303-33 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:2498: finishing write tid 3 to smithi11541303-33 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1340 (ObjNum 744 snap 255 seq_num 744) dirty exists 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:2498: left oid 33 (ObjNum 744 snap 255 seq_num 744) 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:2492: finishing copy_from to smithi11541303-3 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1137 (ObjNum 739 snap 254 seq_num 739) dirty exists 2022-04-22T17:34:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:2492: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:34:19.092 INFO:tasks.rados.rados.0.smithi115.stdout:2490: done (8 left) 2022-04-22T17:34:19.092 INFO:tasks.rados.rados.0.smithi115.stdout:2491: done (7 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2492: done (6 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2493: done (5 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2494: done (4 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2496: done (3 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2497: done (2 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2498: done (1 left) 2022-04-22T17:34:19.093 INFO:tasks.rados.rados.0.smithi115.stdout:2499: done (0 left) 2022-04-22T17:34:19.094 INFO:tasks.rados.rados.0.smithi115.stdout:2500: copy_from oid 27 from oid 18 current snap is 256 2022-04-22T17:34:19.094 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 256 2022-04-22T17:34:19.094 INFO:tasks.rados.rados.0.smithi115.stdout:2501: seq_num 745 ranges {0=483328} 2022-04-22T17:34:19.095 INFO:tasks.rados.rados.0.smithi115.stdout:2501: writing smithi11541303-13 from 0 to 483328 tid 1 2022-04-22T17:34:19.096 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 256 2022-04-22T17:34:19.096 INFO:tasks.rados.rados.0.smithi115.stdout:2502: seq_num 746 ranges {9256960=499712} 2022-04-22T17:34:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:2502: writing smithi11541303-35 from 9256960 to 9756672 tid 1 2022-04-22T17:34:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:2503: read oid 3 snap -1 2022-04-22T17:34:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:2503: expect (ObjNum 739 snap 254 seq_num 739) 2022-04-22T17:34:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:2504: setattr oid 15 current snap is 256 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:2501: finishing write tid 1 to smithi11541303-13 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:2501: finishing write tid 2 to smithi11541303-13 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:2501: finishing write tid 3 to smithi11541303-13 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1299 (ObjNum 745 snap 256 seq_num 745) dirty exists 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:2501: left oid 13 (ObjNum 745 snap 256 seq_num 745) 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:2500: finishing copy_from to smithi11541303-27 2022-04-22T17:34:19.152 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1629 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:34:19.153 INFO:tasks.rados.rados.0.smithi115.stdout:2501: done (4 left) 2022-04-22T17:34:19.153 INFO:tasks.rados.rados.0.smithi115.stdout:2505: setattr oid 32 current snap is 256 2022-04-22T17:34:19.153 INFO:tasks.rados.rados.0.smithi115.stdout:2506: copy_from oid 28 from oid 36 current snap is 256 2022-04-22T17:34:19.153 INFO:tasks.rados.rados.0.smithi115.stdout:2507: copy_from oid 12 from oid 30 current snap is 256 2022-04-22T17:34:19.154 INFO:tasks.rados.rados.0.smithi115.stdout:2508: snap_create 2022-04-22T17:34:19.154 INFO:tasks.rados.rados.0.smithi115.stdout:2500: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:34:19.156 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1341 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:34:19.160 INFO:tasks.rados.rados.0.smithi115.stdout:2502: finishing write tid 1 to smithi11541303-35 2022-04-22T17:34:19.161 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1536 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:19.161 INFO:tasks.rados.rados.0.smithi115.stdout:2502: finishing write tid 2 to smithi11541303-35 2022-04-22T17:34:19.163 INFO:tasks.rados.rados.0.smithi115.stdout:2502: finishing write tid 3 to smithi11541303-35 2022-04-22T17:34:19.163 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1139 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:34:19.163 INFO:tasks.rados.rados.0.smithi115.stdout:2502: left oid 35 (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:34:19.163 INFO:tasks.rados.rados.0.smithi115.stdout:2506: finishing copy_from to smithi11541303-28 2022-04-22T17:34:19.163 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1220 (ObjNum 737 snap 254 seq_num 737) dirty exists 2022-04-22T17:34:19.164 INFO:tasks.rados.rados.0.smithi115.stdout:2506: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:34:19.169 INFO:tasks.rados.rados.0.smithi115.stdout:2507: finishing copy_from to smithi11541303-12 2022-04-22T17:34:19.169 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1220 (ObjNum 656 snap 213 seq_num 656) dirty exists 2022-04-22T17:34:19.170 INFO:tasks.rados.rados.0.smithi115.stdout:2507: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:34:19.208 INFO:tasks.rados.rados.0.smithi115.stdout:2503: expect (ObjNum 739 snap 254 seq_num 739) 2022-04-22T17:34:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:19 smithi121 bash[36902]: cluster 2022-04-22T17:34:17.279346+0000 mgr.y (mgr.34107) 298 : cluster [DBG] pgmap v528: 121 pgs: 121 active+clean; 296 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2022-04-22T17:34:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:19 smithi121 bash[36902]: cluster 2022-04-22T17:34:18.076083+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2022-04-22T17:34:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:19 smithi121 bash[36902]: audit 2022-04-22T17:34:18.572189+0000 mon.a (mon.0) 977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:19.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[45495]: cluster 2022-04-22T17:34:17.279346+0000 mgr.y (mgr.34107) 298 : cluster [DBG] pgmap v528: 121 pgs: 121 active+clean; 296 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2022-04-22T17:34:19.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[45495]: cluster 2022-04-22T17:34:18.076083+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2022-04-22T17:34:19.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[45495]: audit 2022-04-22T17:34:18.572189+0000 mon.a (mon.0) 977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:19.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[46934]: cluster 2022-04-22T17:34:17.279346+0000 mgr.y (mgr.34107) 298 : cluster [DBG] pgmap v528: 121 pgs: 121 active+clean; 296 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2022-04-22T17:34:19.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[46934]: cluster 2022-04-22T17:34:18.076083+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2022-04-22T17:34:19.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:19 smithi115 bash[46934]: audit 2022-04-22T17:34:18.572189+0000 mon.a (mon.0) 977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:20.147 INFO:tasks.rados.rados.0.smithi115.stdout:2500: done (7 left) 2022-04-22T17:34:20.147 INFO:tasks.rados.rados.0.smithi115.stdout:2502: done (6 left) 2022-04-22T17:34:20.147 INFO:tasks.rados.rados.0.smithi115.stdout:2503: done (5 left) 2022-04-22T17:34:20.147 INFO:tasks.rados.rados.0.smithi115.stdout:2504: done (4 left) 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2505: done (3 left) 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2506: done (2 left) 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2507: done (1 left) 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2508: done (0 left) 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2509: rmattr oid 47 current snap is 257 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 257 2022-04-22T17:34:20.148 INFO:tasks.rados.rados.0.smithi115.stdout:2510: seq_num 747 ranges {0=516096} 2022-04-22T17:34:20.153 INFO:tasks.rados.rados.0.smithi115.stdout:2510: writing smithi11541303-16 from 0 to 516096 tid 1 2022-04-22T17:34:20.153 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 257 2022-04-22T17:34:20.154 INFO:tasks.rados.rados.0.smithi115.stdout:2511: seq_num 748 ranges {368640=532480} 2022-04-22T17:34:20.161 INFO:tasks.rados.rados.0.smithi115.stdout:2511: writing smithi11541303-45 from 368640 to 901120 tid 1 2022-04-22T17:34:20.161 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 257 2022-04-22T17:34:20.161 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1214 (ObjNum 3 snap 1 seq_num 4) dirty exists 2022-04-22T17:34:20.161 INFO:tasks.rados.rados.0.smithi115.stdout:2512: seq_num 749 ranges {0=548864} 2022-04-22T17:34:20.164 INFO:tasks.rados.rados.0.smithi115.stdout:2512: writing smithi11541303-11 from 0 to 548864 tid 1 2022-04-22T17:34:20.164 INFO:tasks.rados.rados.0.smithi115.stdout:2509: done (3 left) 2022-04-22T17:34:20.164 INFO:tasks.rados.rados.0.smithi115.stdout:2513: delete oid 46 current snap is 257 2022-04-22T17:34:20.164 INFO:tasks.rados.rados.0.smithi115.stdout:2510: finishing write tid 1 to smithi11541303-16 2022-04-22T17:34:20.165 INFO:tasks.rados.rados.0.smithi115.stdout:2510: finishing write tid 2 to smithi11541303-16 2022-04-22T17:34:20.165 INFO:tasks.rados.rados.0.smithi115.stdout:2510: finishing write tid 3 to smithi11541303-16 2022-04-22T17:34:20.165 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1541 (ObjNum 747 snap 257 seq_num 747) dirty exists 2022-04-22T17:34:20.165 INFO:tasks.rados.rados.0.smithi115.stdout:2510: left oid 16 (ObjNum 747 snap 257 seq_num 747) 2022-04-22T17:34:20.167 INFO:tasks.rados.rados.0.smithi115.stdout:2511: finishing write tid 1 to smithi11541303-45 2022-04-22T17:34:20.167 INFO:tasks.rados.rados.0.smithi115.stdout:2511: finishing write tid 2 to smithi11541303-45 2022-04-22T17:34:20.170 INFO:tasks.rados.rados.0.smithi115.stdout:2511: finishing write tid 3 to smithi11541303-45 2022-04-22T17:34:20.170 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1631 (ObjNum 748 snap 257 seq_num 748) dirty exists 2022-04-22T17:34:20.171 INFO:tasks.rados.rados.0.smithi115.stdout:2511: left oid 45 (ObjNum 748 snap 257 seq_num 748) 2022-04-22T17:34:20.171 INFO:tasks.rados.rados.0.smithi115.stdout:2512: finishing write tid 1 to smithi11541303-11 2022-04-22T17:34:20.171 INFO:tasks.rados.rados.0.smithi115.stdout:2512: finishing write tid 2 to smithi11541303-11 2022-04-22T17:34:20.172 INFO:tasks.rados.rados.0.smithi115.stdout:2510: done (3 left) 2022-04-22T17:34:20.172 INFO:tasks.rados.rados.0.smithi115.stdout:2511: done (2 left) 2022-04-22T17:34:20.172 INFO:tasks.rados.rados.0.smithi115.stdout:2513: done (1 left) 2022-04-22T17:34:20.172 INFO:tasks.rados.rados.0.smithi115.stdout:2514: snap_create 2022-04-22T17:34:20.173 INFO:tasks.rados.rados.0.smithi115.stdout:2512: finishing write tid 3 to smithi11541303-11 2022-04-22T17:34:20.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1213 (ObjNum 749 snap 257 seq_num 749) dirty exists 2022-04-22T17:34:20.173 INFO:tasks.rados.rados.0.smithi115.stdout:2512: left oid 11 (ObjNum 749 snap 257 seq_num 749) 2022-04-22T17:34:20.392 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:20 smithi121 bash[36902]: audit 2022-04-22T17:34:18.572555+0000 mgr.y (mgr.34107) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:20.393 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:20 smithi121 bash[36902]: cephadm 2022-04-22T17:34:18.574577+0000 mgr.y (mgr.34107) 300 : cephadm [INF] Upgrade: osd.7 is safe to restart 2022-04-22T17:34:20.393 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:20 smithi121 bash[36902]: cluster 2022-04-22T17:34:19.078431+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2022-04-22T17:34:20.393 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:20 smithi121 bash[36902]: audit 2022-04-22T17:34:19.315840+0000 mon.a (mon.0) 979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:20.393 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:20 smithi121 bash[36902]: cluster 2022-04-22T17:34:19.886696+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2022-04-22T17:34:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[45495]: audit 2022-04-22T17:34:18.572555+0000 mgr.y (mgr.34107) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[45495]: cephadm 2022-04-22T17:34:18.574577+0000 mgr.y (mgr.34107) 300 : cephadm [INF] Upgrade: osd.7 is safe to restart 2022-04-22T17:34:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[45495]: cluster 2022-04-22T17:34:19.078431+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2022-04-22T17:34:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[45495]: audit 2022-04-22T17:34:19.315840+0000 mon.a (mon.0) 979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[45495]: cluster 2022-04-22T17:34:19.886696+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2022-04-22T17:34:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[46934]: audit 2022-04-22T17:34:18.572555+0000 mgr.y (mgr.34107) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2022-04-22T17:34:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[46934]: cephadm 2022-04-22T17:34:18.574577+0000 mgr.y (mgr.34107) 300 : cephadm [INF] Upgrade: osd.7 is safe to restart 2022-04-22T17:34:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[46934]: cluster 2022-04-22T17:34:19.078431+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2022-04-22T17:34:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[46934]: audit 2022-04-22T17:34:19.315840+0000 mon.a (mon.0) 979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:20 smithi115 bash[46934]: cluster 2022-04-22T17:34:19.886696+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2022-04-22T17:34:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:2512: done (1 left) 2022-04-22T17:34:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:2514: done (0 left) 2022-04-22T17:34:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 258 2022-04-22T17:34:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:2515: seq_num 751 ranges {417792=581632} 2022-04-22T17:34:21.122 INFO:tasks.rados.rados.0.smithi115.stdout:2515: writing smithi11541303-48 from 417792 to 999424 tid 1 2022-04-22T17:34:21.122 INFO:tasks.rados.rados.0.smithi115.stdout:2516: delete oid 9 current snap is 258 2022-04-22T17:34:21.129 INFO:tasks.rados.rados.0.smithi115.stdout:2516: done (1 left) 2022-04-22T17:34:21.129 INFO:tasks.rados.rados.0.smithi115.stdout:2517: read oid 25 snap -1 2022-04-22T17:34:21.129 INFO:tasks.rados.rados.0.smithi115.stdout:2517: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:21.129 INFO:tasks.rados.rados.0.smithi115.stdout:2518: read oid 18 snap -1 2022-04-22T17:34:21.129 INFO:tasks.rados.rados.0.smithi115.stdout:2518: expect (ObjNum 667 snap 222 seq_num 667) 2022-04-22T17:34:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:2519: rollback oid 50 current snap is 258 2022-04-22T17:34:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 247 2022-04-22T17:34:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 258 2022-04-22T17:34:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:2520: seq_num 753 ranges {548864=614400} 2022-04-22T17:34:21.139 INFO:tasks.rados.rados.0.smithi115.stdout:2520: writing smithi11541303-1 from 548864 to 1163264 tid 1 2022-04-22T17:34:21.139 INFO:tasks.rados.rados.0.smithi115.stdout:2521: read oid 15 snap -1 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2521: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2522: read oid 14 snap -1 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2522: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2523: delete oid 37 current snap is 258 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2515: finishing write tid 1 to smithi11541303-48 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2515: finishing write tid 2 to smithi11541303-48 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2515: finishing write tid 3 to smithi11541303-48 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1633 (ObjNum 751 snap 258 seq_num 751) dirty exists 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2515: left oid 48 (ObjNum 751 snap 258 seq_num 751) 2022-04-22T17:34:21.140 INFO:tasks.rados.rados.0.smithi115.stdout:2519: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:34:21.141 INFO:tasks.rados.rados.0.smithi115.stdout:2519: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:34:21.141 INFO:tasks.rados.rados.0.smithi115.stdout:2519: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:34:21.141 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1544 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:21.141 INFO:tasks.rados.rados.0.smithi115.stdout:2517: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:34:21.201 INFO:tasks.rados.rados.0.smithi115.stdout:2515: done (7 left) 2022-04-22T17:34:21.201 INFO:tasks.rados.rados.0.smithi115.stdout:2517: done (6 left) 2022-04-22T17:34:21.201 INFO:tasks.rados.rados.0.smithi115.stdout:2519: done (5 left) 2022-04-22T17:34:21.201 INFO:tasks.rados.rados.0.smithi115.stdout:2523: done (4 left) 2022-04-22T17:34:21.202 INFO:tasks.rados.rados.0.smithi115.stdout:2524: delete oid 47 current snap is 258 2022-04-22T17:34:21.202 INFO:tasks.rados.rados.0.smithi115.stdout:2518: expect (ObjNum 667 snap 222 seq_num 667) 2022-04-22T17:34:21.335 INFO:tasks.rados.rados.0.smithi115.stdout:2520: finishing write tid 1 to smithi11541303-1 2022-04-22T17:34:21.335 INFO:tasks.rados.rados.0.smithi115.stdout:2520: finishing write tid 2 to smithi11541303-1 2022-04-22T17:34:21.335 INFO:tasks.rados.rados.0.smithi115.stdout:2520: finishing write tid 3 to smithi11541303-1 2022-04-22T17:34:21.336 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1227 (ObjNum 753 snap 258 seq_num 753) dirty exists 2022-04-22T17:34:21.336 INFO:tasks.rados.rados.0.smithi115.stdout:2520: left oid 1 (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:34:21.336 INFO:tasks.rados.rados.0.smithi115.stdout:2521: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:21.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:21 smithi121 bash[36902]: cluster 2022-04-22T17:34:19.280241+0000 mgr.y (mgr.34107) 301 : cluster [DBG] pgmap v531: 121 pgs: 121 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:34:21.472 INFO:tasks.rados.rados.0.smithi115.stdout:2522: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:34:21.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:21 smithi115 bash[45495]: cluster 2022-04-22T17:34:19.280241+0000 mgr.y (mgr.34107) 301 : cluster [DBG] pgmap v531: 121 pgs: 121 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:34:21.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:21 smithi115 bash[46934]: cluster 2022-04-22T17:34:19.280241+0000 mgr.y (mgr.34107) 301 : cluster [DBG] pgmap v531: 121 pgs: 121 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2022-04-22T17:34:21.642 INFO:tasks.rados.rados.0.smithi115.stdout:2518: done (4 left) 2022-04-22T17:34:21.642 INFO:tasks.rados.rados.0.smithi115.stdout:2520: done (3 left) 2022-04-22T17:34:21.642 INFO:tasks.rados.rados.0.smithi115.stdout:2521: done (2 left) 2022-04-22T17:34:21.642 INFO:tasks.rados.rados.0.smithi115.stdout:2522: done (1 left) 2022-04-22T17:34:21.642 INFO:tasks.rados.rados.0.smithi115.stdout:2524: done (0 left) 2022-04-22T17:34:21.643 INFO:tasks.rados.rados.0.smithi115.stdout:2525: read oid 48 snap -1 2022-04-22T17:34:21.751 INFO:tasks.rados.rados.0.smithi115.stdout:2525: expect (ObjNum 751 snap 258 seq_num 751) 2022-04-22T17:34:21.751 INFO:tasks.rados.rados.0.smithi115.stdout:2526: delete oid 31 current snap is 258 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2526: done (1 left) 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2527: read oid 21 snap 245 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2527: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2528: rollback oid 50 current snap is 258 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 244 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 258 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2529: seq_num 757 ranges {901120=688128} 2022-04-22T17:34:21.753 INFO:tasks.rados.rados.0.smithi115.stdout:2529: writing smithi11541303-45 from 901120 to 1589248 tid 1 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2530: snap_create 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2525: expect (ObjNum 751 snap 258 seq_num 751) 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2528: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2528: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2529: finishing write tid 1 to smithi11541303-45 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2528: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1547 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2529: finishing write tid 2 to smithi11541303-45 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:2529: finishing write tid 3 to smithi11541303-45 2022-04-22T17:34:21.754 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1635 (ObjNum 757 snap 258 seq_num 757) dirty exists 2022-04-22T17:34:21.755 INFO:tasks.rados.rados.0.smithi115.stdout:2529: left oid 45 (ObjNum 757 snap 258 seq_num 757) 2022-04-22T17:34:21.755 INFO:tasks.rados.rados.0.smithi115.stdout:2527: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:21.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:21.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:21.630Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:22 smithi121 bash[36902]: cluster 2022-04-22T17:34:21.101472+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2022-04-22T17:34:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:22 smithi121 bash[36902]: audit 2022-04-22T17:34:21.434327+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:22.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:22 smithi115 bash[46934]: cluster 2022-04-22T17:34:21.101472+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2022-04-22T17:34:22.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:22 smithi115 bash[46934]: audit 2022-04-22T17:34:21.434327+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:22.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:22 smithi115 bash[45495]: cluster 2022-04-22T17:34:21.101472+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2022-04-22T17:34:22.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:22 smithi115 bash[45495]: audit 2022-04-22T17:34:21.434327+0000 mon.a (mon.0) 982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:22.560 INFO:tasks.rados.rados.0.smithi115.stdout:2525: done (4 left) 2022-04-22T17:34:22.560 INFO:tasks.rados.rados.0.smithi115.stdout:2527: done (3 left) 2022-04-22T17:34:22.560 INFO:tasks.rados.rados.0.smithi115.stdout:2528: done (2 left) 2022-04-22T17:34:22.561 INFO:tasks.rados.rados.0.smithi115.stdout:2529: done (1 left) 2022-04-22T17:34:22.561 INFO:tasks.rados.rados.0.smithi115.stdout:2530: done (0 left) 2022-04-22T17:34:22.561 INFO:tasks.rados.rados.0.smithi115.stdout:2531: read oid 19 snap -1 2022-04-22T17:34:22.561 INFO:tasks.rados.rados.0.smithi115.stdout:2531: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:22.561 INFO:tasks.rados.rados.0.smithi115.stdout:2532: snap_remove snap 250 2022-04-22T17:34:23.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:23 smithi121 bash[36902]: cluster 2022-04-22T17:34:21.280972+0000 mgr.y (mgr.34107) 302 : cluster [DBG] pgmap v534: 121 pgs: 121 active+clean; 312 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:34:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:23 smithi121 bash[36902]: cephadm 2022-04-22T17:34:21.426057+0000 mgr.y (mgr.34107) 303 : cephadm [INF] Upgrade: Updating osd.7 2022-04-22T17:34:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:23 smithi121 bash[36902]: cephadm 2022-04-22T17:34:21.437037+0000 mgr.y (mgr.34107) 304 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:34:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:23 smithi121 bash[36902]: cluster 2022-04-22T17:34:22.439103+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2022-04-22T17:34:23.458 INFO:tasks.rados.rados.0.smithi115.stdout:2532: done (1 left) 2022-04-22T17:34:23.458 INFO:tasks.rados.rados.0.smithi115.stdout:2533: snap_create 2022-04-22T17:34:23.458 INFO:tasks.rados.rados.0.smithi115.stdout:2531: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[45495]: cluster 2022-04-22T17:34:21.280972+0000 mgr.y (mgr.34107) 302 : cluster [DBG] pgmap v534: 121 pgs: 121 active+clean; 312 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:34:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[45495]: cephadm 2022-04-22T17:34:21.426057+0000 mgr.y (mgr.34107) 303 : cephadm [INF] Upgrade: Updating osd.7 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[45495]: cephadm 2022-04-22T17:34:21.437037+0000 mgr.y (mgr.34107) 304 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[45495]: cluster 2022-04-22T17:34:22.439103+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[46934]: cluster 2022-04-22T17:34:21.280972+0000 mgr.y (mgr.34107) 302 : cluster [DBG] pgmap v534: 121 pgs: 121 active+clean; 312 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 11 op/s 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[46934]: cephadm 2022-04-22T17:34:21.426057+0000 mgr.y (mgr.34107) 303 : cephadm [INF] Upgrade: Updating osd.7 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[46934]: cephadm 2022-04-22T17:34:21.437037+0000 mgr.y (mgr.34107) 304 : cephadm [INF] Deploying daemon osd.7 on smithi121 2022-04-22T17:34:23.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[46934]: cluster 2022-04-22T17:34:22.439103+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2022-04-22T17:34:23.698 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:34:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:23.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:23.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:23.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (10m) 2m ago 11m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (10m) 29s ago 10m 23.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (12m) 2m ago 12m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (12m) 29s ago 12m 19.8M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6m) 29s ago 16m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (7m) 2m ago 18m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:34:24.044 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6m) 2m ago 18m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5m) 29s ago 16m 54.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5m) 2m ago 16m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (11m) 2m ago 11m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (11m) 29s ago 11m 9535k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4m) 2m ago 16m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3m) 2m ago 15m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3m) 2m ago 15m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2m) 2m ago 14m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (115s) 29s ago 14m 223M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:34:24.045 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (74s) 29s ago 13m 308M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:34:24.046 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (32s) 29s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:34:24.046 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (12m) 29s ago 12m 278M 2526M 16.2.5 46cf6318c64e b7eab0f325d4 2022-04-22T17:34:24.046 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (7m) 29s ago 11m 40.6M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:34:24.432 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:34:24.432 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:34:24.432 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:34:24.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:24.432 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 7 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:24.433 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:34:24.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3, 2022-04-22T17:34:24.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 12 2022-04-22T17:34:24.434 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:34:24.434 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:34:24.632 INFO:tasks.rados.rados.0.smithi115.stdout:2531: done (1 left) 2022-04-22T17:34:24.632 INFO:tasks.rados.rados.0.smithi115.stdout:2533: done (0 left) 2022-04-22T17:34:24.632 INFO:tasks.rados.rados.0.smithi115.stdout:2534: copy_from oid 50 from oid 43 current snap is 260 2022-04-22T17:34:24.632 INFO:tasks.rados.rados.0.smithi115.stdout:2535: snap_create 2022-04-22T17:34:24.655 INFO:tasks.rados.rados.0.smithi115.stdout:2534: finishing copy_from to smithi11541303-50 2022-04-22T17:34:24.655 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1272 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:34:24.656 INFO:tasks.rados.rados.0.smithi115.stdout:2534: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:34:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: cluster 2022-04-22T17:34:23.281949+0000 mgr.y (mgr.34107) 305 : cluster [DBG] pgmap v536: 121 pgs: 121 active+clean; 317 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2022-04-22T17:34:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: cluster 2022-04-22T17:34:23.443975+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2022-04-22T17:34:24.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: audit 2022-04-22T17:34:23.670570+0000 mgr.y (mgr.34107) 306 : audit [DBG] from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: audit 2022-04-22T17:34:24.012388+0000 mgr.y (mgr.34107) 307 : audit [DBG] from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: audit 2022-04-22T17:34:24.318258+0000 mon.a (mon.0) 985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:24.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:24 smithi121 bash[36902]: audit 2022-04-22T17:34:24.418614+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.115:0/1487604363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: cluster 2022-04-22T17:34:23.281949+0000 mgr.y (mgr.34107) 305 : cluster [DBG] pgmap v536: 121 pgs: 121 active+clean; 317 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: cluster 2022-04-22T17:34:23.443975+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: audit 2022-04-22T17:34:23.670570+0000 mgr.y (mgr.34107) 306 : audit [DBG] from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: audit 2022-04-22T17:34:24.012388+0000 mgr.y (mgr.34107) 307 : audit [DBG] from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: audit 2022-04-22T17:34:24.318258+0000 mon.a (mon.0) 985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:24.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[45495]: audit 2022-04-22T17:34:24.418614+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.115:0/1487604363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:24.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: cluster 2022-04-22T17:34:23.281949+0000 mgr.y (mgr.34107) 305 : cluster [DBG] pgmap v536: 121 pgs: 121 active+clean; 317 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2022-04-22T17:34:24.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: cluster 2022-04-22T17:34:23.443975+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2022-04-22T17:34:24.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: audit 2022-04-22T17:34:23.670570+0000 mgr.y (mgr.34107) 306 : audit [DBG] from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: audit 2022-04-22T17:34:24.012388+0000 mgr.y (mgr.34107) 307 : audit [DBG] from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:24.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: audit 2022-04-22T17:34:24.318258+0000 mon.a (mon.0) 985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:24.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:24 smithi115 bash[46934]: audit 2022-04-22T17:34:24.418614+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.115:0/1487604363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:24.901 INFO:tasks.rados.rados.0.smithi115.stdout:2534: done (1 left) 2022-04-22T17:34:24.902 INFO:tasks.rados.rados.0.smithi115.stdout:2535: done (0 left) 2022-04-22T17:34:24.902 INFO:tasks.rados.rados.0.smithi115.stdout:2536: rollback oid 44 current snap is 261 2022-04-22T17:34:24.902 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 239 2022-04-22T17:34:24.902 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 261 2022-04-22T17:34:24.902 INFO:tasks.rados.rados.0.smithi115.stdout:2537: seq_num 758 ranges {0=704512} 2022-04-22T17:34:24.906 INFO:tasks.rados.rados.0.smithi115.stdout:2537: writing smithi11541303-42 from 0 to 704512 tid 1 2022-04-22T17:34:24.907 INFO:tasks.rados.rados.0.smithi115.stdout:2538: rollback oid 5 current snap is 261 2022-04-22T17:34:24.907 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 255 2022-04-22T17:34:24.907 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 261 2022-04-22T17:34:24.907 INFO:tasks.rados.rados.0.smithi115.stdout:2539: seq_num 759 ranges {1982464=720896} 2022-04-22T17:34:24.924 INFO:tasks.rados.rados.0.smithi115.stdout:2539: writing smithi11541303-50 from 1982464 to 2703360 tid 1 2022-04-22T17:34:24.924 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 261 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2536: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2536: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1103 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2537: finishing write tid 1 to smithi11541303-42 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2537: finishing write tid 2 to smithi11541303-42 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2538: finishing rollback tid 0 to smithi11541303-5 2022-04-22T17:34:24.925 INFO:tasks.rados.rados.0.smithi115.stdout:2540: seq_num 760 ranges {0=737280} 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2540: writing smithi11541303-32 from 0 to 737280 tid 1 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2536: done (4 left) 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2541: read oid 19 snap -1 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2541: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2542: delete oid 37 current snap is 261 2022-04-22T17:34:24.929 INFO:tasks.rados.rados.0.smithi115.stdout:2538: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2538: finishing rollback tid 2 to smithi11541303-5 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1663 (ObjNum 288 snap 0 seq_num 33) dirty exists 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2537: finishing write tid 3 to smithi11541303-42 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1659 (ObjNum 758 snap 261 seq_num 758) dirty exists 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2537: left oid 42 (ObjNum 758 snap 261 seq_num 758) 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2539: finishing write tid 1 to smithi11541303-50 2022-04-22T17:34:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2539: finishing write tid 2 to smithi11541303-50 2022-04-22T17:34:24.931 INFO:tasks.rados.rados.0.smithi115.stdout:2539: finishing write tid 3 to smithi11541303-50 2022-04-22T17:34:24.931 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1549 (ObjNum 759 snap 261 seq_num 759) dirty exists 2022-04-22T17:34:24.931 INFO:tasks.rados.rados.0.smithi115.stdout:2539: left oid 50 (ObjNum 759 snap 261 seq_num 759) 2022-04-22T17:34:24.936 INFO:tasks.rados.rados.0.smithi115.stdout:2540: finishing write tid 1 to smithi11541303-32 2022-04-22T17:34:24.936 INFO:tasks.rados.rados.0.smithi115.stdout:2540: finishing write tid 2 to smithi11541303-32 2022-04-22T17:34:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2540: finishing write tid 3 to smithi11541303-32 2022-04-22T17:34:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1343 (ObjNum 760 snap 261 seq_num 760) dirty exists 2022-04-22T17:34:24.938 INFO:tasks.rados.rados.0.smithi115.stdout:2540: left oid 32 (ObjNum 760 snap 261 seq_num 760) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2537: done (5 left) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2538: done (4 left) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2539: done (3 left) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2540: done (2 left) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2542: done (1 left) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2543: read oid 21 snap -1 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2543: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:24.945 INFO:tasks.rados.rados.0.smithi115.stdout:2544: copy_from oid 18 from oid 35 current snap is 261 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2545: read oid 6 snap -1 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2545: expect deleted 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2546: read oid 45 snap -1 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2546: expect (ObjNum 757 snap 258 seq_num 757) 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2547: read oid 15 snap -1 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2547: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 261 2022-04-22T17:34:24.946 INFO:tasks.rados.rados.0.smithi115.stdout:2548: seq_num 762 ranges {483328=770048} 2022-04-22T17:34:24.957 INFO:tasks.rados.rados.0.smithi115.stdout:2548: writing smithi11541303-13 from 483328 to 1253376 tid 1 2022-04-22T17:34:24.957 INFO:tasks.rados.rados.0.smithi115.stdout:2549: delete oid 27 current snap is 261 2022-04-22T17:34:24.961 INFO:tasks.rados.rados.0.smithi115.stdout:2546: expect (ObjNum 757 snap 258 seq_num 757) 2022-04-22T17:34:25.034 INFO:tasks.rados.rados.0.smithi115.stdout:2545: done (7 left) 2022-04-22T17:34:25.091 INFO:tasks.rados.rados.0.smithi115.stdout:2546: done (6 left) 2022-04-22T17:34:25.092 INFO:tasks.rados.rados.0.smithi115.stdout:2549: done (5 left) 2022-04-22T17:34:25.092 INFO:tasks.rados.rados.0.smithi115.stdout:2550: rollback oid 33 current snap is 261 2022-04-22T17:34:25.092 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 252 2022-04-22T17:34:25.092 INFO:tasks.rados.rados.0.smithi115.stdout:2547: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:25.165 INFO:tasks.rados.rados.0.smithi115.stdout:2548: finishing write tid 1 to smithi11541303-13 2022-04-22T17:34:25.165 INFO:tasks.rados.rados.0.smithi115.stdout:2548: finishing write tid 2 to smithi11541303-13 2022-04-22T17:34:25.165 INFO:tasks.rados.rados.0.smithi115.stdout:2548: finishing write tid 3 to smithi11541303-13 2022-04-22T17:34:25.165 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1302 (ObjNum 762 snap 261 seq_num 762) dirty exists 2022-04-22T17:34:25.165 INFO:tasks.rados.rados.0.smithi115.stdout:2548: left oid 13 (ObjNum 762 snap 261 seq_num 762) 2022-04-22T17:34:25.166 INFO:tasks.rados.rados.0.smithi115.stdout:2541: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:25 smithi121 bash[36902]: cluster 2022-04-22T17:34:24.442644+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2022-04-22T17:34:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:25 smithi121 bash[36902]: cluster 2022-04-22T17:34:24.887038+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2022-04-22T17:34:25.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:25 smithi115 bash[45495]: cluster 2022-04-22T17:34:24.442644+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2022-04-22T17:34:25.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:25 smithi115 bash[45495]: cluster 2022-04-22T17:34:24.887038+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2022-04-22T17:34:25.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:25 smithi115 bash[46934]: cluster 2022-04-22T17:34:24.442644+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2022-04-22T17:34:25.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:25 smithi115 bash[46934]: cluster 2022-04-22T17:34:24.887038+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2022-04-22T17:34:26.304 INFO:tasks.rados.rados.0.smithi115.stdout:2543: expect (ObjNum 675 snap 225 seq_num 675) 2022-04-22T17:34:26.474 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:26 smithi121 systemd[1]: Stopping Ceph osd.7 for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:34:26.753 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[24219]: debug 2022-04-22T17:34:26.467+0000 7f1e13f82700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:34:26.754 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[24219]: debug 2022-04-22T17:34:26.467+0000 7f1e13f82700 -1 osd.7 568 *** Got signal Terminated *** 2022-04-22T17:34:26.754 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[24219]: debug 2022-04-22T17:34:26.467+0000 7f1e13f82700 -1 osd.7 568 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-22T17:34:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:25.282697+0000 mgr.y (mgr.34107) 308 : cluster [DBG] pgmap v540: 121 pgs: 121 active+clean; 319 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 1.7 MiB/s wr, 26 op/s 2022-04-22T17:34:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.768313+0000 mon.a (mon.0) 988 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.768516+0000 mon.a (mon.0) 989 : cluster [INF] osd.7 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.769086+0000 mon.a (mon.0) 990 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.769963+0000 mon.a (mon.0) 991 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.770190+0000 mon.a (mon.0) 992 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.770497+0000 mon.a (mon.0) 993 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.770677+0000 mon.a (mon.0) 994 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.771003+0000 mon.a (mon.0) 995 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.771244+0000 mon.a (mon.0) 996 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.771486+0000 mon.a (mon.0) 997 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.771751+0000 mon.a (mon.0) 998 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.771910+0000 mon.a (mon.0) 999 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.772217+0000 mon.a (mon.0) 1000 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.772468+0000 mon.a (mon.0) 1001 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:26 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.772711+0000 mon.a (mon.0) 1002 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:2544: finishing copy_from to smithi11541303-18 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1139 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:2544: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:2550: finishing rollback tid 0 to smithi11541303-33 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:2550: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:2550: finishing rollback tid 2 to smithi11541303-33 2022-04-22T17:34:27.144 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1346 (ObjNum 703 snap 235 seq_num 703) dirty exists 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2541: done (5 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2543: done (4 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2544: done (3 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2547: done (2 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2548: done (1 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2550: done (0 left) 2022-04-22T17:34:27.145 INFO:tasks.rados.rados.0.smithi115.stdout:2551: snap_create 2022-04-22T17:34:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:25.282697+0000 mgr.y (mgr.34107) 308 : cluster [DBG] pgmap v540: 121 pgs: 121 active+clean; 319 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 1.7 MiB/s wr, 26 op/s 2022-04-22T17:34:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.768313+0000 mon.a (mon.0) 988 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.768516+0000 mon.a (mon.0) 989 : cluster [INF] osd.7 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.769086+0000 mon.a (mon.0) 990 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.769963+0000 mon.a (mon.0) 991 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.770190+0000 mon.a (mon.0) 992 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.770497+0000 mon.a (mon.0) 993 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.770677+0000 mon.a (mon.0) 994 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.771003+0000 mon.a (mon.0) 995 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.771244+0000 mon.a (mon.0) 996 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.771486+0000 mon.a (mon.0) 997 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.771751+0000 mon.a (mon.0) 998 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.771910+0000 mon.a (mon.0) 999 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.772217+0000 mon.a (mon.0) 1000 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.772468+0000 mon.a (mon.0) 1001 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.772711+0000 mon.a (mon.0) 1002 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:25.282697+0000 mgr.y (mgr.34107) 308 : cluster [DBG] pgmap v540: 121 pgs: 121 active+clean; 319 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 1.7 MiB/s wr, 26 op/s 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.768313+0000 mon.a (mon.0) 988 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.768516+0000 mon.a (mon.0) 989 : cluster [INF] osd.7 failed (root=default,host=smithi121) (connection refused reported by osd.3) 2022-04-22T17:34:27.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.769086+0000 mon.a (mon.0) 990 : cluster [DBG] osd.7 reported immediately failed by osd.3 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.769963+0000 mon.a (mon.0) 991 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.770190+0000 mon.a (mon.0) 992 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.770497+0000 mon.a (mon.0) 993 : cluster [DBG] osd.7 reported immediately failed by osd.2 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.770677+0000 mon.a (mon.0) 994 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.771003+0000 mon.a (mon.0) 995 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.771244+0000 mon.a (mon.0) 996 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.771486+0000 mon.a (mon.0) 997 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.771751+0000 mon.a (mon.0) 998 : cluster [DBG] osd.7 reported immediately failed by osd.4 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.771910+0000 mon.a (mon.0) 999 : cluster [DBG] osd.7 reported immediately failed by osd.0 2022-04-22T17:34:27.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.772217+0000 mon.a (mon.0) 1000 : cluster [DBG] osd.7 reported immediately failed by osd.6 2022-04-22T17:34:27.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.772468+0000 mon.a (mon.0) 1001 : cluster [DBG] osd.7 reported immediately failed by osd.1 2022-04-22T17:34:27.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:26 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.772711+0000 mon.a (mon.0) 1002 : cluster [DBG] osd.7 reported immediately failed by osd.5 2022-04-22T17:34:27.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[45237]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.7 2022-04-22T17:34:27.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[45281]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd-7 2022-04-22T17:34:27.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[45294]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-osd.7 2022-04-22T17:34:27.930 INFO:tasks.rados.rados.0.smithi115.stdout:2551: done (0 left) 2022-04-22T17:34:27.930 INFO:tasks.rados.rados.0.smithi115.stdout:2552: copy_from oid 37 from oid 41 current snap is 262 2022-04-22T17:34:27.930 INFO:tasks.rados.rados.0.smithi115.stdout:2553: snap_create 2022-04-22T17:34:27.955 INFO:tasks.rados.rados.0.smithi115.stdout:2552: finishing copy_from to smithi11541303-37 2022-04-22T17:34:27.955 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1211 (ObjNum 735 snap 253 seq_num 735) dirty exists 2022-04-22T17:34:27.956 INFO:tasks.rados.rados.0.smithi115.stdout:2552: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:34:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.897085+0000 mon.a (mon.0) 1003 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:34:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.897150+0000 mon.a (mon.0) 1004 : cluster [WRN] Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2022-04-22T17:34:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:27 smithi121 bash[36902]: cluster 2022-04-22T17:34:26.907446+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2022-04-22T17:34:28.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.897085+0000 mon.a (mon.0) 1003 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:34:28.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.897150+0000 mon.a (mon.0) 1004 : cluster [WRN] Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2022-04-22T17:34:28.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[45495]: cluster 2022-04-22T17:34:26.907446+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2022-04-22T17:34:28.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.897085+0000 mon.a (mon.0) 1003 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2022-04-22T17:34:28.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.897150+0000 mon.a (mon.0) 1004 : cluster [WRN] Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2022-04-22T17:34:28.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:27 smithi115 bash[46934]: cluster 2022-04-22T17:34:26.907446+0000 mon.a (mon.0) 1005 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2022-04-22T17:34:28.926 INFO:tasks.rados.rados.0.smithi115.stdout:2552: done (1 left) 2022-04-22T17:34:28.927 INFO:tasks.rados.rados.0.smithi115.stdout:2553: done (0 left) 2022-04-22T17:34:28.927 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 263 2022-04-22T17:34:28.927 INFO:tasks.rados.rados.0.smithi115.stdout:2554: seq_num 764 ranges {2703360=802816} 2022-04-22T17:34:28.946 INFO:tasks.rados.rados.0.smithi115.stdout:2554: writing smithi11541303-50 from 2703360 to 3506176 tid 1 2022-04-22T17:34:28.947 INFO:tasks.rados.rados.0.smithi115.stdout:2555: read oid 39 snap 239 2022-04-22T17:34:28.947 INFO:tasks.rados.rados.0.smithi115.stdout:2555: expect (ObjNum 706 snap 238 seq_num 706) 2022-04-22T17:34:28.947 INFO:tasks.rados.rados.0.smithi115.stdout:2556: delete oid 13 current snap is 263 2022-04-22T17:34:28.962 INFO:tasks.rados.rados.0.smithi115.stdout:2554: finishing write tid 1 to smithi11541303-50 2022-04-22T17:34:28.962 INFO:tasks.rados.rados.0.smithi115.stdout:2554: finishing write tid 2 to smithi11541303-50 2022-04-22T17:34:28.964 INFO:tasks.rados.rados.0.smithi115.stdout:2554: finishing write tid 3 to smithi11541303-50 2022-04-22T17:34:28.964 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1551 (ObjNum 764 snap 263 seq_num 764) dirty exists 2022-04-22T17:34:28.965 INFO:tasks.rados.rados.0.smithi115.stdout:2554: left oid 50 (ObjNum 764 snap 263 seq_num 764) 2022-04-22T17:34:28.972 INFO:tasks.rados.rados.0.smithi115.stdout:2554: done (2 left) 2022-04-22T17:34:28.972 INFO:tasks.rados.rados.0.smithi115.stdout:2556: done (1 left) 2022-04-22T17:34:28.972 INFO:tasks.rados.rados.0.smithi115.stdout:2557: snap_create 2022-04-22T17:34:28.978 INFO:tasks.rados.rados.0.smithi115.stdout:2555: expect (ObjNum 706 snap 238 seq_num 706) 2022-04-22T17:34:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:28 smithi121 bash[36902]: cluster 2022-04-22T17:34:27.283655+0000 mgr.y (mgr.34107) 309 : cluster [DBG] pgmap v542: 121 pgs: 10 peering, 16 stale+active+clean, 95 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2022-04-22T17:34:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:28 smithi121 bash[36902]: cluster 2022-04-22T17:34:27.914042+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2022-04-22T17:34:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:28 smithi121 bash[36902]: cluster 2022-04-22T17:34:27.915931+0000 mon.a (mon.0) 1007 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:34:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:28 smithi121 bash[36902]: cluster 2022-04-22T17:34:28.911964+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2022-04-22T17:34:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[45495]: cluster 2022-04-22T17:34:27.283655+0000 mgr.y (mgr.34107) 309 : cluster [DBG] pgmap v542: 121 pgs: 10 peering, 16 stale+active+clean, 95 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[45495]: cluster 2022-04-22T17:34:27.914042+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[45495]: cluster 2022-04-22T17:34:27.915931+0000 mon.a (mon.0) 1007 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[45495]: cluster 2022-04-22T17:34:28.911964+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[46934]: cluster 2022-04-22T17:34:27.283655+0000 mgr.y (mgr.34107) 309 : cluster [DBG] pgmap v542: 121 pgs: 10 peering, 16 stale+active+clean, 95 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[46934]: cluster 2022-04-22T17:34:27.914042+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[46934]: cluster 2022-04-22T17:34:27.915931+0000 mon.a (mon.0) 1007 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2022-04-22T17:34:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:28 smithi115 bash[46934]: cluster 2022-04-22T17:34:28.911964+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2022-04-22T17:34:29.963 INFO:tasks.rados.rados.0.smithi115.stdout:2555: done (1 left) 2022-04-22T17:34:29.963 INFO:tasks.rados.rados.0.smithi115.stdout:2557: done (0 left) 2022-04-22T17:34:29.963 INFO:tasks.rados.rados.0.smithi115.stdout:2558: copy_from oid 8 from oid 3 current snap is 264 2022-04-22T17:34:29.963 INFO:tasks.rados.rados.0.smithi115.stdout:2559: setattr oid 13 current snap is 264 2022-04-22T17:34:29.964 INFO:tasks.rados.rados.0.smithi115.stdout:2560: delete oid 23 current snap is 264 2022-04-22T17:34:29.975 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1305 (ObjNum -1115499536 snap 22075 seq_num 3179756800) dirty exists 2022-04-22T17:34:29.976 INFO:tasks.rados.rados.0.smithi115.stdout:2559: done (2 left) 2022-04-22T17:34:29.976 INFO:tasks.rados.rados.0.smithi115.stdout:2560: done (1 left) 2022-04-22T17:34:29.976 INFO:tasks.rados.rados.0.smithi115.stdout:2561: copy_from oid 37 from oid 48 current snap is 264 2022-04-22T17:34:29.976 INFO:tasks.rados.rados.0.smithi115.stdout:2562: snap_create 2022-04-22T17:34:29.991 INFO:tasks.rados.rados.0.smithi115.stdout:2561: finishing copy_from to smithi11541303-37 2022-04-22T17:34:29.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1633 (ObjNum 751 snap 258 seq_num 751) dirty exists 2022-04-22T17:34:29.993 INFO:tasks.rados.rados.0.smithi115.stdout:2561: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:34:30.020 INFO:tasks.rados.rados.0.smithi115.stdout:2558: finishing copy_from to smithi11541303-8 2022-04-22T17:34:30.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1137 (ObjNum 739 snap 254 seq_num 739) dirty exists 2022-04-22T17:34:30.021 INFO:tasks.rados.rados.0.smithi115.stdout:2558: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:34:30.266 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:30 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@osd.7.service: Succeeded. 2022-04-22T17:34:30.266 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:30 smithi121 systemd[1]: Stopped Ceph osd.7 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:34:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:29 smithi121 bash[36902]: audit 2022-04-22T17:34:29.321301+0000 mon.a (mon.0) 1009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:30.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:29 smithi115 bash[45495]: audit 2022-04-22T17:34:29.321301+0000 mon.a (mon.0) 1009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:30.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:29 smithi115 bash[46934]: audit 2022-04-22T17:34:29.321301+0000 mon.a (mon.0) 1009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:30.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:30 smithi121 systemd[1]: Started Ceph osd.7 for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:34:30.976 INFO:tasks.rados.rados.0.smithi115.stdout:2558: done (2 left) 2022-04-22T17:34:30.977 INFO:tasks.rados.rados.0.smithi115.stdout:2561: done (1 left) 2022-04-22T17:34:30.977 INFO:tasks.rados.rados.0.smithi115.stdout:2562: done (0 left) 2022-04-22T17:34:30.977 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 265 2022-04-22T17:34:30.977 INFO:tasks.rados.rados.0.smithi115.stdout:2563: seq_num 767 ranges {1392640=450560,1843200=401408} 2022-04-22T17:34:30.988 INFO:tasks.rados.rados.0.smithi115.stdout:2563: writing smithi11541303-12 from 1392640 to 1843200 tid 1 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2563: writing smithi11541303-12 from 1843200 to 2244608 tid 2 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2564: read oid 33 snap -1 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2564: expect (ObjNum 703 snap 235 seq_num 703) 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2565: rmattr oid 38 current snap is 265 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2565: done (2 left) 2022-04-22T17:34:30.991 INFO:tasks.rados.rados.0.smithi115.stdout:2566: rollback oid 44 current snap is 265 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 234 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:2567: rollback oid 4 current snap is 265 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 264 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:2568: rollback oid 41 current snap is 265 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 244 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:2569: read oid 29 snap -1 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:2569: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 265 2022-04-22T17:34:30.992 INFO:tasks.rados.rados.0.smithi115.stdout:2570: seq_num 768 ranges {0=466944,466944=401408} 2022-04-22T17:34:30.997 INFO:tasks.rados.rados.0.smithi115.stdout:2570: writing smithi11541303-40 from 0 to 466944 tid 1 2022-04-22T17:34:31.001 INFO:tasks.rados.rados.0.smithi115.stdout:2570: writing smithi11541303-40 from 466944 to 868352 tid 2 2022-04-22T17:34:31.001 INFO:tasks.rados.rados.0.smithi115.stdout:2571: snap_remove snap 248 2022-04-22T17:34:31.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: cluster 2022-04-22T17:34:29.284476+0000 mgr.y (mgr.34107) 310 : cluster [DBG] pgmap v545: 121 pgs: 5 active+undersized+degraded, 10 peering, 8 stale+active+clean, 15 active+undersized, 83 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 3.1 MiB/s wr, 6 op/s; 23/624 objects degraded (3.686%) 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: cluster 2022-04-22T17:34:29.931784+0000 mon.a (mon.0) 1010 : cluster [WRN] Health check failed: Degraded data redundancy: 23/624 objects degraded (3.686%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: cluster 2022-04-22T17:34:29.947889+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: audit 2022-04-22T17:34:30.362755+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: audit 2022-04-22T17:34:30.364591+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: audit 2022-04-22T17:34:30.366775+0000 mon.a (mon.0) 1014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[46934]: audit 2022-04-22T17:34:30.368200+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:31.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: cluster 2022-04-22T17:34:29.284476+0000 mgr.y (mgr.34107) 310 : cluster [DBG] pgmap v545: 121 pgs: 5 active+undersized+degraded, 10 peering, 8 stale+active+clean, 15 active+undersized, 83 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 3.1 MiB/s wr, 6 op/s; 23/624 objects degraded (3.686%) 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: cluster 2022-04-22T17:34:29.931784+0000 mon.a (mon.0) 1010 : cluster [WRN] Health check failed: Degraded data redundancy: 23/624 objects degraded (3.686%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: cluster 2022-04-22T17:34:29.947889+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: audit 2022-04-22T17:34:30.362755+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: audit 2022-04-22T17:34:30.364591+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: audit 2022-04-22T17:34:30.366775+0000 mon.a (mon.0) 1014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:31.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:30 smithi115 bash[45495]: audit 2022-04-22T17:34:30.368200+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: cluster 2022-04-22T17:34:29.284476+0000 mgr.y (mgr.34107) 310 : cluster [DBG] pgmap v545: 121 pgs: 5 active+undersized+degraded, 10 peering, 8 stale+active+clean, 15 active+undersized, 83 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 3.1 MiB/s wr, 6 op/s; 23/624 objects degraded (3.686%) 2022-04-22T17:34:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: cluster 2022-04-22T17:34:29.931784+0000 mon.a (mon.0) 1010 : cluster [WRN] Health check failed: Degraded data redundancy: 23/624 objects degraded (3.686%), 5 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: cluster 2022-04-22T17:34:29.947889+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2022-04-22T17:34:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: audit 2022-04-22T17:34:30.362755+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: audit 2022-04-22T17:34:30.364591+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: audit 2022-04-22T17:34:30.366775+0000 mon.a (mon.0) 1014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:30 smithi121 bash[36902]: audit 2022-04-22T17:34:30.368200+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:31.982 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:31.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:31.982 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:31.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:31.982 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:31.630Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:31.987 INFO:tasks.rados.rados.0.smithi115.stdout:2566: finishing rollback tid 0 to smithi11541303-44 2022-04-22T17:34:31.987 INFO:tasks.rados.rados.0.smithi115.stdout:2569: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:34:31.990 INFO:tasks.rados.rados.0.smithi115.stdout:2566: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:34:31.990 INFO:tasks.rados.rados.0.smithi115.stdout:2566: finishing rollback tid 2 to smithi11541303-44 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1106 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2563: finishing write tid 1 to smithi11541303-12 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2568: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2567: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2563: finishing write tid 2 to smithi11541303-12 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2567: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2568: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:34:31.991 INFO:tasks.rados.rados.0.smithi115.stdout:2563: finishing write tid 3 to smithi11541303-12 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2567: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1639 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2568: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1218 (ObjNum 652 snap 212 seq_num 652) dirty exists 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2570: finishing write tid 1 to smithi11541303-40 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2570: finishing write tid 2 to smithi11541303-40 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2570: finishing write tid 3 to smithi11541303-40 2022-04-22T17:34:31.992 INFO:tasks.rados.rados.0.smithi115.stdout:2563: finishing write tid 4 to smithi11541303-12 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1223 (ObjNum 767 snap 265 seq_num 767) dirty exists 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:2563: left oid 12 (ObjNum 767 snap 265 seq_num 767) 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:2570: finishing write tid 4 to smithi11541303-40 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1221 (ObjNum 768 snap 265 seq_num 768) dirty exists 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:2570: left oid 40 (ObjNum 768 snap 265 seq_num 768) 2022-04-22T17:34:31.993 INFO:tasks.rados.rados.0.smithi115.stdout:2564: expect (ObjNum 703 snap 235 seq_num 703) 2022-04-22T17:34:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:31 smithi121 bash[36902]: cluster 2022-04-22T17:34:30.957812+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2022-04-22T17:34:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:31 smithi115 bash[46934]: cluster 2022-04-22T17:34:30.957812+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2022-04-22T17:34:32.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:31 smithi115 bash[45495]: cluster 2022-04-22T17:34:30.957812+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2022-04-22T17:34:32.337 INFO:tasks.rados.rados.0.smithi115.stdout:2563: done (7 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2564: done (6 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2566: done (5 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2567: done (4 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2568: done (3 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2569: done (2 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2570: done (1 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2571: done (0 left) 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2572: rmattr oid 9 current snap is 265 2022-04-22T17:34:32.338 INFO:tasks.rados.rados.0.smithi115.stdout:2572: done (0 left) 2022-04-22T17:34:32.339 INFO:tasks.rados.rados.0.smithi115.stdout:2573: rollback oid 12 current snap is 265 2022-04-22T17:34:32.339 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 253 2022-04-22T17:34:32.339 INFO:tasks.rados.rados.0.smithi115.stdout:2574: snap_remove snap 227 2022-04-22T17:34:32.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-04-22T17:34:32.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2022-04-22T17:34:32.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2022-04-22T17:34:32.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2022-04-22T17:34:32.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2022-04-22T17:34:32.767 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-04-22T17:34:32.767 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[45478]: --> ceph-volume raw activate successful for osd ID: 7 2022-04-22T17:34:32.991 INFO:tasks.rados.rados.0.smithi115.stdout:2573: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:2573: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:2573: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1226 (ObjNum 720 snap 247 seq_num 720) dirty exists 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:2573: done (1 left) 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:2574: done (0 left) 2022-04-22T17:34:32.992 INFO:tasks.rados.rados.0.smithi115.stdout:2575: delete oid 8 current snap is 265 2022-04-22T17:34:33.013 INFO:tasks.rados.rados.0.smithi115.stdout:2575: done (0 left) 2022-04-22T17:34:33.107 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 265 2022-04-22T17:34:33.108 INFO:tasks.rados.rados.0.smithi115.stdout:2576: seq_num 770 ranges {434176=499712,933888=401408} 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:2576: writing smithi11541303-10 from 434176 to 933888 tid 1 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:2576: writing smithi11541303-10 from 933888 to 1335296 tid 2 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:2577: read oid 6 snap -1 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:2577: expect deleted 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:2578: rollback oid 37 current snap is 265 2022-04-22T17:34:33.109 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 239 2022-04-22T17:34:33.110 INFO:tasks.rados.rados.0.smithi115.stdout:2579: read oid 17 snap -1 2022-04-22T17:34:33.110 INFO:tasks.rados.rados.0.smithi115.stdout:2579: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:33.110 INFO:tasks.rados.rados.0.smithi115.stdout:2580: snap_remove snap 244 2022-04-22T17:34:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[36902]: cluster 2022-04-22T17:34:31.285458+0000 mgr.y (mgr.34107) 311 : cluster [DBG] pgmap v548: 121 pgs: 8 active+undersized+degraded, 10 peering, 3 stale+active+clean, 24 active+undersized, 76 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 3 op/s; 34/630 objects degraded (5.397%) 2022-04-22T17:34:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:32 smithi121 bash[36902]: cluster 2022-04-22T17:34:31.973134+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2022-04-22T17:34:33.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:32 smithi115 bash[46934]: cluster 2022-04-22T17:34:31.285458+0000 mgr.y (mgr.34107) 311 : cluster [DBG] pgmap v548: 121 pgs: 8 active+undersized+degraded, 10 peering, 3 stale+active+clean, 24 active+undersized, 76 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 3 op/s; 34/630 objects degraded (5.397%) 2022-04-22T17:34:33.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:32 smithi115 bash[46934]: cluster 2022-04-22T17:34:31.973134+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2022-04-22T17:34:33.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:32 smithi115 bash[45495]: cluster 2022-04-22T17:34:31.285458+0000 mgr.y (mgr.34107) 311 : cluster [DBG] pgmap v548: 121 pgs: 8 active+undersized+degraded, 10 peering, 3 stale+active+clean, 24 active+undersized, 76 active+clean; 343 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 3 op/s; 34/630 objects degraded (5.397%) 2022-04-22T17:34:33.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:32 smithi115 bash[45495]: cluster 2022-04-22T17:34:31.973134+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2022-04-22T17:34:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:33.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:33.525Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:33.525Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:34.000 INFO:tasks.rados.rados.0.smithi115.stdout:2580: done (4 left) 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2581: read oid 7 snap -1 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2581: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2582: read oid 49 snap -1 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2582: expect deleted 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2583: delete oid 43 current snap is 265 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2578: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:34:34.040 INFO:tasks.rados.rados.0.smithi115.stdout:2578: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1635 (ObjNum 1 snap 22075 seq_num 3179446416) dirty dne 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2576: finishing write tid 1 to smithi11541303-10 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2576: finishing write tid 2 to smithi11541303-10 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2576: finishing write tid 3 to smithi11541303-10 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2576: finishing write tid 4 to smithi11541303-10 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1224 (ObjNum 770 snap 265 seq_num 770) dirty exists 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2576: left oid 10 (ObjNum 770 snap 265 seq_num 770) 2022-04-22T17:34:34.041 INFO:tasks.rados.rados.0.smithi115.stdout:2579: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:34:34.256 INFO:tasks.rados.rados.0.smithi115.stdout:2581: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:34:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:33 smithi121 bash[36902]: cluster 2022-04-22T17:34:32.975778+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2022-04-22T17:34:34.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:33 smithi115 bash[46934]: cluster 2022-04-22T17:34:32.975778+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2022-04-22T17:34:34.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:33 smithi115 bash[45495]: cluster 2022-04-22T17:34:32.975778+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2576: done (6 left) 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2577: done (5 left) 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2578: done (4 left) 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2579: done (3 left) 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2581: done (2 left) 2022-04-22T17:34:34.554 INFO:tasks.rados.rados.0.smithi115.stdout:2582: done (1 left) 2022-04-22T17:34:34.555 INFO:tasks.rados.rados.0.smithi115.stdout:2583: done (0 left) 2022-04-22T17:34:34.555 INFO:tasks.rados.rados.0.smithi115.stdout:2584: snap_create 2022-04-22T17:34:34.766 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[45658]: debug 2022-04-22T17:34:34.458+0000 7f82b0452700 1 -- 172.21.15.121:0/2926158333 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564916652340 con 0x5649166d6800 2022-04-22T17:34:34.904 INFO:tasks.rados.rados.0.smithi115.stdout:2584: done (0 left) 2022-04-22T17:34:34.955 INFO:tasks.rados.rados.0.smithi115.stdout:2585: snap_remove snap 228 2022-04-22T17:34:35.266 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:35 smithi121 bash[45658]: debug 2022-04-22T17:34:35.046+0000 7f82b7a493c0 -1 Falling back to public interface 2022-04-22T17:34:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[36902]: cluster 2022-04-22T17:34:33.286664+0000 mgr.y (mgr.34107) 312 : cluster [DBG] pgmap v551: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 14 op/s; 70/654 objects degraded (10.703%) 2022-04-22T17:34:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[36902]: cluster 2022-04-22T17:34:33.973105+0000 mon.a (mon.0) 1019 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:34:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[36902]: cluster 2022-04-22T17:34:33.984311+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2022-04-22T17:34:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[36902]: audit 2022-04-22T17:34:34.324200+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:34 smithi121 bash[36902]: cluster 2022-04-22T17:34:34.888711+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[45495]: cluster 2022-04-22T17:34:33.286664+0000 mgr.y (mgr.34107) 312 : cluster [DBG] pgmap v551: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 14 op/s; 70/654 objects degraded (10.703%) 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[45495]: cluster 2022-04-22T17:34:33.973105+0000 mon.a (mon.0) 1019 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[45495]: cluster 2022-04-22T17:34:33.984311+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[45495]: audit 2022-04-22T17:34:34.324200+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[45495]: cluster 2022-04-22T17:34:34.888711+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2022-04-22T17:34:35.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[46934]: cluster 2022-04-22T17:34:33.286664+0000 mgr.y (mgr.34107) 312 : cluster [DBG] pgmap v551: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 14 op/s; 70/654 objects degraded (10.703%) 2022-04-22T17:34:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[46934]: cluster 2022-04-22T17:34:33.973105+0000 mon.a (mon.0) 1019 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2022-04-22T17:34:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[46934]: cluster 2022-04-22T17:34:33.984311+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2022-04-22T17:34:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[46934]: audit 2022-04-22T17:34:34.324200+0000 mon.a (mon.0) 1021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:35.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:34 smithi115 bash[46934]: cluster 2022-04-22T17:34:34.888711+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2022-04-22T17:34:35.909 INFO:tasks.rados.rados.0.smithi115.stdout:2585: done (0 left) 2022-04-22T17:34:35.909 INFO:tasks.rados.rados.0.smithi115.stdout:2586: read oid 37 snap 255 2022-04-22T17:34:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:2586: expect (ObjNum -1114778640 snap 22075 seq_num 3178348240) 2022-04-22T17:34:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:2587: copy_from oid 44 from oid 13 current snap is 266 2022-04-22T17:34:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:2588: rollback oid 33 current snap is 266 2022-04-22T17:34:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 260 2022-04-22T17:34:35.910 INFO:tasks.rados.rados.0.smithi115.stdout:2589: delete oid 8 current snap is 266 2022-04-22T17:34:35.913 INFO:tasks.rados.rados.0.smithi115.stdout:2586: expect (ObjNum -1114778640 snap 22075 seq_num 3178348240) 2022-04-22T17:34:35.916 INFO:tasks.rados.rados.0.smithi115.stdout:2588: finishing rollback tid 0 to smithi11541303-33 2022-04-22T17:34:35.917 INFO:tasks.rados.rados.0.smithi115.stdout:2588: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:34:35.924 INFO:tasks.rados.rados.0.smithi115.stdout:2588: finishing rollback tid 2 to smithi11541303-33 2022-04-22T17:34:35.924 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1349 (ObjNum 744 snap 255 seq_num 744) dirty exists 2022-04-22T17:34:35.927 INFO:tasks.rados.rados.0.smithi115.stdout:2586: done (3 left) 2022-04-22T17:34:35.927 INFO:tasks.rados.rados.0.smithi115.stdout:2588: done (2 left) 2022-04-22T17:34:35.927 INFO:tasks.rados.rados.0.smithi115.stdout:2589: done (1 left) 2022-04-22T17:34:35.928 INFO:tasks.rados.rados.0.smithi115.stdout:2590: read oid 15 snap -1 2022-04-22T17:34:35.928 INFO:tasks.rados.rados.0.smithi115.stdout:2590: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:35.928 INFO:tasks.rados.rados.0.smithi115.stdout:2591: delete oid 3 current snap is 266 2022-04-22T17:34:35.932 INFO:tasks.rados.rados.0.smithi115.stdout:2591: done (2 left) 2022-04-22T17:34:35.932 INFO:tasks.rados.rados.0.smithi115.stdout:2592: rmattr oid 21 current snap is 266 2022-04-22T17:34:35.932 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 266 2022-04-22T17:34:35.932 INFO:tasks.rados.rados.0.smithi115.stdout:2593: seq_num 774 ranges {0=565248,565248=401408} 2022-04-22T17:34:35.935 INFO:tasks.rados.rados.0.smithi115.stdout:2593: writing smithi11541303-5 from 0 to 565248 tid 1 2022-04-22T17:34:35.938 INFO:tasks.rados.rados.0.smithi115.stdout:2593: writing smithi11541303-5 from 565248 to 966656 tid 2 2022-04-22T17:34:35.939 INFO:tasks.rados.rados.0.smithi115.stdout:2594: delete oid 46 current snap is 266 2022-04-22T17:34:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2587: finishing copy_from to smithi11541303-44 2022-04-22T17:34:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1305 (ObjNum -1327212024 snap 32692 seq_num 1) dirty exists 2022-04-22T17:34:35.946 INFO:tasks.rados.rados.0.smithi115.stdout:2587: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:34:35.946 INFO:tasks.rados.rados.0.smithi115.stdout:2590: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1131 (ObjNum 675 snap 225 seq_num 675) dirty exists 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2593: finishing write tid 1 to smithi11541303-5 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2593: finishing write tid 2 to smithi11541303-5 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2593: finishing write tid 3 to smithi11541303-5 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2593: finishing write tid 4 to smithi11541303-5 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1703 (ObjNum 774 snap 266 seq_num 774) dirty exists 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2593: left oid 5 (ObjNum 774 snap 266 seq_num 774) 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2587: done (4 left) 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2590: done (3 left) 2022-04-22T17:34:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:2592: done (2 left) 2022-04-22T17:34:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2593: done (1 left) 2022-04-22T17:34:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2594: done (0 left) 2022-04-22T17:34:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2595: rollback oid 50 current snap is 266 2022-04-22T17:34:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 237 2022-04-22T17:34:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:2596: snap_create 2022-04-22T17:34:36.088 INFO:tasks.rados.rados.0.smithi115.stdout:2595: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:34:36.088 INFO:tasks.rados.rados.0.smithi115.stdout:2595: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:34:36.089 INFO:tasks.rados.rados.0.smithi115.stdout:2595: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:34:36.089 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1554 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:36.999 INFO:tasks.rados.rados.0.smithi115.stdout:2595: done (1 left) 2022-04-22T17:34:36.999 INFO:tasks.rados.rados.0.smithi115.stdout:2596: done (0 left) 2022-04-22T17:34:37.000 INFO:tasks.rados.rados.0.smithi115.stdout:2597: delete oid 44 current snap is 267 2022-04-22T17:34:37.005 INFO:tasks.rados.rados.0.smithi115.stdout:2597: done (0 left) 2022-04-22T17:34:37.006 INFO:tasks.rados.rados.0.smithi115.stdout:2598: read oid 11 snap -1 2022-04-22T17:34:37.006 INFO:tasks.rados.rados.0.smithi115.stdout:2598: expect (ObjNum 749 snap 257 seq_num 749) 2022-04-22T17:34:37.006 INFO:tasks.rados.rados.0.smithi115.stdout:2599: copy_from oid 15 from oid 43 current snap is 267 2022-04-22T17:34:37.006 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 267 2022-04-22T17:34:37.006 INFO:tasks.rados.rados.0.smithi115.stdout:2600: seq_num 777 ranges {737280=622592,1359872=401408} 2022-04-22T17:34:37.013 INFO:tasks.rados.rados.0.smithi115.stdout:2600: writing smithi11541303-32 from 737280 to 1359872 tid 1 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2600: writing smithi11541303-32 from 1359872 to 1761280 tid 2 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2601: delete oid 37 current snap is 267 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2599: finishing copy_from to smithi11541303-15 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2599: got expected ENOENT (src dne) 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2599: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:34:37.016 INFO:tasks.rados.rados.0.smithi115.stdout:2598: expect (ObjNum 749 snap 257 seq_num 749) 2022-04-22T17:34:37.030 INFO:tasks.rados.rados.0.smithi115.stdout:2598: done (3 left) 2022-04-22T17:34:37.030 INFO:tasks.rados.rados.0.smithi115.stdout:2599: done (2 left) 2022-04-22T17:34:37.030 INFO:tasks.rados.rados.0.smithi115.stdout:2601: done (1 left) 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2602: setattr oid 29 current snap is 267 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2600: finishing write tid 1 to smithi11541303-32 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2600: finishing write tid 2 to smithi11541303-32 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2600: finishing write tid 3 to smithi11541303-32 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2600: finishing write tid 4 to smithi11541303-32 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1352 (ObjNum 777 snap 267 seq_num 777) dirty exists 2022-04-22T17:34:37.031 INFO:tasks.rados.rados.0.smithi115.stdout:2600: left oid 32 (ObjNum 777 snap 267 seq_num 777) 2022-04-22T17:34:37.032 INFO:tasks.rados.rados.0.smithi115.stdout:2600: done (1 left) 2022-04-22T17:34:37.032 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 267 2022-04-22T17:34:37.032 INFO:tasks.rados.rados.0.smithi115.stdout:2603: seq_num 779 ranges {0=655360,655360=401408} 2022-04-22T17:34:37.035 INFO:tasks.rados.rados.0.smithi115.stdout:2603: writing smithi11541303-3 from 0 to 655360 tid 1 2022-04-22T17:34:37.039 INFO:tasks.rados.rados.0.smithi115.stdout:2603: writing smithi11541303-3 from 655360 to 1056768 tid 2 2022-04-22T17:34:37.039 INFO:tasks.rados.rados.0.smithi115.stdout:2604: setattr oid 37 current snap is 267 2022-04-22T17:34:37.039 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1221 (ObjNum -1118045920 snap 22075 seq_num 3178242496) dirty exists 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2602: done (2 left) 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2605: rollback oid 17 current snap is 267 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 17 to 251 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2606: read oid 34 snap -1 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2606: expect (ObjNum 723 snap 250 seq_num 723) 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2607: read oid 19 snap -1 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2607: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 267 2022-04-22T17:34:37.043 INFO:tasks.rados.rados.0.smithi115.stdout:2608: seq_num 780 ranges {0=671744,671744=401408} 2022-04-22T17:34:37.047 INFO:tasks.rados.rados.0.smithi115.stdout:2608: writing smithi11541303-27 from 0 to 671744 tid 1 2022-04-22T17:34:37.050 INFO:tasks.rados.rados.0.smithi115.stdout:2608: writing smithi11541303-27 from 671744 to 1073152 tid 2 2022-04-22T17:34:37.050 INFO:tasks.rados.rados.0.smithi115.stdout:2609: copy_from oid 38 from oid 2 current snap is 267 2022-04-22T17:34:37.050 INFO:tasks.rados.rados.0.smithi115.stdout:2603: finishing write tid 1 to smithi11541303-3 2022-04-22T17:34:37.050 INFO:tasks.rados.rados.0.smithi115.stdout:2603: finishing write tid 2 to smithi11541303-3 2022-04-22T17:34:37.051 INFO:tasks.rados.rados.0.smithi115.stdout:2603: finishing write tid 3 to smithi11541303-3 2022-04-22T17:34:37.051 INFO:tasks.rados.rados.0.smithi115.stdout:2610: snap_remove snap 253 2022-04-22T17:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:36 smithi121 bash[36902]: cluster 2022-04-22T17:34:35.287804+0000 mgr.y (mgr.34107) 313 : cluster [DBG] pgmap v554: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 4.0 MiB/s wr, 14 op/s; 71/657 objects degraded (10.807%) 2022-04-22T17:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:36 smithi121 bash[36902]: cluster 2022-04-22T17:34:35.889319+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2022-04-22T17:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:36 smithi121 bash[36902]: audit 2022-04-22T17:34:35.909362+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:36 smithi121 bash[36902]: cluster 2022-04-22T17:34:35.976048+0000 mon.a (mon.0) 1025 : cluster [WRN] Health check update: Degraded data redundancy: 71/657 objects degraded (10.807%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:36 smithi121 bash[36902]: audit 2022-04-22T17:34:36.501809+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:37.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[45495]: cluster 2022-04-22T17:34:35.287804+0000 mgr.y (mgr.34107) 313 : cluster [DBG] pgmap v554: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 4.0 MiB/s wr, 14 op/s; 71/657 objects degraded (10.807%) 2022-04-22T17:34:37.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[45495]: cluster 2022-04-22T17:34:35.889319+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2022-04-22T17:34:37.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[45495]: audit 2022-04-22T17:34:35.909362+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:37.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[45495]: cluster 2022-04-22T17:34:35.976048+0000 mon.a (mon.0) 1025 : cluster [WRN] Health check update: Degraded data redundancy: 71/657 objects degraded (10.807%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:37.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[45495]: audit 2022-04-22T17:34:36.501809+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[46934]: cluster 2022-04-22T17:34:35.287804+0000 mgr.y (mgr.34107) 313 : cluster [DBG] pgmap v554: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 371 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 4.0 MiB/s wr, 14 op/s; 71/657 objects degraded (10.807%) 2022-04-22T17:34:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[46934]: cluster 2022-04-22T17:34:35.889319+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2022-04-22T17:34:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[46934]: audit 2022-04-22T17:34:35.909362+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[46934]: cluster 2022-04-22T17:34:35.976048+0000 mon.a (mon.0) 1025 : cluster [WRN] Health check update: Degraded data redundancy: 71/657 objects degraded (10.807%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:37.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:36 smithi115 bash[46934]: audit 2022-04-22T17:34:36.501809+0000 mon.a (mon.0) 1026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:38.008 INFO:tasks.rados.rados.0.smithi115.stdout:2610: done (7 left) 2022-04-22T17:34:38.009 INFO:tasks.rados.rados.0.smithi115.stdout:2611: snap_remove snap 265 2022-04-22T17:34:38.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:38 smithi121 bash[36902]: cluster 2022-04-22T17:34:36.986127+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2022-04-22T17:34:38.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:37 smithi115 bash[45495]: cluster 2022-04-22T17:34:36.986127+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2022-04-22T17:34:38.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:37 smithi115 bash[46934]: cluster 2022-04-22T17:34:36.986127+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2022-04-22T17:34:39.016 INFO:tasks.rados.rados.0.smithi115.stdout:2611: done (7 left) 2022-04-22T17:34:39.016 INFO:tasks.rados.rados.0.smithi115.stdout:2612: rollback oid 14 current snap is 267 2022-04-22T17:34:39.016 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 260 2022-04-22T17:34:39.016 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 267 2022-04-22T17:34:39.017 INFO:tasks.rados.rados.0.smithi115.stdout:2613: seq_num 781 ranges {4087808=688128,4775936=401408} 2022-04-22T17:34:39.046 INFO:tasks.rados.rados.0.smithi115.stdout:2613: writing smithi11541303-7 from 4087808 to 4775936 tid 1 2022-04-22T17:34:39.049 INFO:tasks.rados.rados.0.smithi115.stdout:2613: writing smithi11541303-7 from 4775936 to 5177344 tid 2 2022-04-22T17:34:39.049 INFO:tasks.rados.rados.0.smithi115.stdout:2614: rollback oid 30 current snap is 267 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 264 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2603: finishing write tid 4 to smithi11541303-3 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1218 (ObjNum 779 snap 267 seq_num 779) dirty exists 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2603: left oid 3 (ObjNum 779 snap 267 seq_num 779) 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2605: finishing rollback tid 0 to smithi11541303-17 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2605: finishing rollback tid 1 to smithi11541303-17 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2605: finishing rollback tid 2 to smithi11541303-17 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1327 (ObjNum 611 snap 200 seq_num 611) dirty exists 2022-04-22T17:34:39.050 INFO:tasks.rados.rados.0.smithi115.stdout:2608: finishing write tid 1 to smithi11541303-27 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2608: finishing write tid 2 to smithi11541303-27 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2603: done (9 left) 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2605: done (8 left) 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 267 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2608: finishing write tid 3 to smithi11541303-27 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1636 (ObjNum -1115797760 snap 22075 seq_num 3178681264) dirty exists 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2608: finishing write tid 4 to smithi11541303-27 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1642 (ObjNum 780 snap 267 seq_num 780) dirty exists 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2608: left oid 27 (ObjNum 780 snap 267 seq_num 780) 2022-04-22T17:34:39.051 INFO:tasks.rados.rados.0.smithi115.stdout:2606: expect (ObjNum 723 snap 250 seq_num 723) 2022-04-22T17:34:39.221 INFO:tasks.rados.rados.0.smithi115.stdout:2607: expect (ObjNum 623 snap 204 seq_num 623) 2022-04-22T17:34:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:39 smithi121 bash[36902]: cluster 2022-04-22T17:34:37.288896+0000 mgr.y (mgr.34107) 314 : cluster [DBG] pgmap v557: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 362 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 456 KiB/s wr, 7 op/s; 73/639 objects degraded (11.424%) 2022-04-22T17:34:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:39 smithi121 bash[36902]: cluster 2022-04-22T17:34:37.993347+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2022-04-22T17:34:39.267 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:39 smithi121 bash[45658]: debug 2022-04-22T17:34:39.218+0000 7f82b7a493c0 -1 osd.7 568 log_to_monitors true 2022-04-22T17:34:39.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:38 smithi115 bash[45495]: cluster 2022-04-22T17:34:37.288896+0000 mgr.y (mgr.34107) 314 : cluster [DBG] pgmap v557: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 362 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 456 KiB/s wr, 7 op/s; 73/639 objects degraded (11.424%) 2022-04-22T17:34:39.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:38 smithi115 bash[45495]: cluster 2022-04-22T17:34:37.993347+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2022-04-22T17:34:39.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:38 smithi115 bash[46934]: cluster 2022-04-22T17:34:37.288896+0000 mgr.y (mgr.34107) 314 : cluster [DBG] pgmap v557: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 362 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 456 KiB/s wr, 7 op/s; 73/639 objects degraded (11.424%) 2022-04-22T17:34:39.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:38 smithi115 bash[46934]: cluster 2022-04-22T17:34:37.993347+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[45495]: cluster 2022-04-22T17:34:39.001239+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[45495]: audit 2022-04-22T17:34:39.219989+0000 mon.a (mon.0) 1030 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[45495]: audit 2022-04-22T17:34:39.229783+0000 mon.b (mon.2) 9 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[45495]: audit 2022-04-22T17:34:39.326752+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[46934]: cluster 2022-04-22T17:34:39.001239+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2022-04-22T17:34:40.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[46934]: audit 2022-04-22T17:34:39.219989+0000 mon.a (mon.0) 1030 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[46934]: audit 2022-04-22T17:34:39.229783+0000 mon.b (mon.2) 9 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:40 smithi115 bash[46934]: audit 2022-04-22T17:34:39.326752+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:40.360 INFO:tasks.rados.rados.0.smithi115.stdout:2609: finishing copy_from to smithi11541303-38 2022-04-22T17:34:40.360 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1111 (ObjNum 1785745767 snap 1852665176 seq_num 1650872942) dirty exists 2022-04-22T17:34:40.360 INFO:tasks.rados.rados.0.smithi115.stdout:2609: finishing copy_from racing read to smithi11541303-38 2022-04-22T17:34:40.360 INFO:tasks.rados.rados.0.smithi115.stdout:2612: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:34:40.360 INFO:tasks.rados.rados.0.smithi115.stdout:2612: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:34:40.361 INFO:tasks.rados.rados.0.smithi115.stdout:2612: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:34:40.361 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1224 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:34:40.361 INFO:tasks.rados.rados.0.smithi115.stdout:2613: finishing write tid 1 to smithi11541303-7 2022-04-22T17:34:40.361 INFO:tasks.rados.rados.0.smithi115.stdout:2613: finishing write tid 2 to smithi11541303-7 2022-04-22T17:34:40.361 INFO:tasks.rados.rados.0.smithi115.stdout:2615: seq_num 782 ranges {10551296=704512,11255808=401408} 2022-04-22T17:34:40.428 INFO:tasks.rados.rados.0.smithi115.stdout:2615: writing smithi11541303-21 from 10551296 to 11255808 tid 1 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2615: writing smithi11541303-21 from 11255808 to 11657216 tid 2 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2604: done (8 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2606: done (7 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2607: done (6 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2608: done (5 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2609: done (4 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2612: done (3 left) 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 267 2022-04-22T17:34:40.431 INFO:tasks.rados.rados.0.smithi115.stdout:2613: finishing write tid 3 to smithi11541303-7 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2614: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2614: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2614: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1645 (ObjNum 656 snap 213 seq_num 656) dirty exists 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2613: finishing write tid 4 to smithi11541303-7 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1231 (ObjNum 781 snap 267 seq_num 781) dirty exists 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2613: left oid 7 (ObjNum 781 snap 267 seq_num 781) 2022-04-22T17:34:40.432 INFO:tasks.rados.rados.0.smithi115.stdout:2616: seq_num 783 ranges {1441792=720896,2162688=401408} 2022-04-22T17:34:40.443 INFO:tasks.rados.rados.0.smithi115.stdout:2616: writing smithi11541303-41 from 1441792 to 2162688 tid 1 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2616: writing smithi11541303-41 from 2162688 to 2564096 tid 2 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2613: done (3 left) 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2614: done (2 left) 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2617: read oid 30 snap -1 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2617: expect (ObjNum 656 snap 213 seq_num 656) 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 267 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2615: finishing write tid 1 to smithi11541303-21 2022-04-22T17:34:40.446 INFO:tasks.rados.rados.0.smithi115.stdout:2615: finishing write tid 2 to smithi11541303-21 2022-04-22T17:34:40.447 INFO:tasks.rados.rados.0.smithi115.stdout:2615: finishing write tid 3 to smithi11541303-21 2022-04-22T17:34:40.447 INFO:tasks.rados.rados.0.smithi115.stdout:2615: finishing write tid 4 to smithi11541303-21 2022-04-22T17:34:40.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1134 (ObjNum 782 snap 267 seq_num 782) dirty exists 2022-04-22T17:34:40.447 INFO:tasks.rados.rados.0.smithi115.stdout:2615: left oid 21 (ObjNum 782 snap 267 seq_num 782) 2022-04-22T17:34:40.447 INFO:tasks.rados.rados.0.smithi115.stdout:2618: seq_num 784 ranges {4808704=737280,5545984=401408} 2022-04-22T17:34:40.478 INFO:tasks.rados.rados.0.smithi115.stdout:2618: writing smithi11541303-14 from 4808704 to 5545984 tid 1 2022-04-22T17:34:40.480 INFO:tasks.rados.rados.0.smithi115.stdout:2618: writing smithi11541303-14 from 5545984 to 5947392 tid 2 2022-04-22T17:34:40.480 INFO:tasks.rados.rados.0.smithi115.stdout:2615: done (3 left) 2022-04-22T17:34:40.480 INFO:tasks.rados.rados.0.smithi115.stdout:2619: copy_from oid 16 from oid 48 current snap is 267 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2616: finishing write tid 1 to smithi11541303-41 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2616: finishing write tid 2 to smithi11541303-41 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2616: finishing write tid 3 to smithi11541303-41 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2616: finishing write tid 4 to smithi11541303-41 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1227 (ObjNum 783 snap 267 seq_num 783) dirty exists 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2616: left oid 41 (ObjNum 783 snap 267 seq_num 783) 2022-04-22T17:34:40.481 INFO:tasks.rados.rados.0.smithi115.stdout:2617: expect (ObjNum 656 snap 213 seq_num 656) 2022-04-22T17:34:40.516 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 22 17:34:40 smithi121 bash[45658]: debug 2022-04-22T17:34:40.066+0000 7f82a6de0700 -1 osd.7 568 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-22T17:34:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:40 smithi121 bash[36902]: cluster 2022-04-22T17:34:39.001239+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2022-04-22T17:34:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:40 smithi121 bash[36902]: audit 2022-04-22T17:34:39.219989+0000 mon.a (mon.0) 1030 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:40 smithi121 bash[36902]: audit 2022-04-22T17:34:39.229783+0000 mon.b (mon.2) 9 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-04-22T17:34:40.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:40 smithi121 bash[36902]: audit 2022-04-22T17:34:39.326752+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:40.519 INFO:tasks.rados.rados.0.smithi115.stdout:2618: finishing write tid 1 to smithi11541303-14 2022-04-22T17:34:40.519 INFO:tasks.rados.rados.0.smithi115.stdout:2618: finishing write tid 2 to smithi11541303-14 2022-04-22T17:34:40.519 INFO:tasks.rados.rados.0.smithi115.stdout:2618: finishing write tid 3 to smithi11541303-14 2022-04-22T17:34:40.519 INFO:tasks.rados.rados.0.smithi115.stdout:2618: finishing write tid 4 to smithi11541303-14 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1227 (ObjNum 784 snap 267 seq_num 784) dirty exists 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2618: left oid 14 (ObjNum 784 snap 267 seq_num 784) 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2619: finishing copy_from to smithi11541303-16 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1633 (ObjNum 751 snap 258 seq_num 751) dirty exists 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2619: finishing copy_from racing read to smithi11541303-16 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2616: done (3 left) 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2617: done (2 left) 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2618: done (1 left) 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2619: done (0 left) 2022-04-22T17:34:40.520 INFO:tasks.rados.rados.0.smithi115.stdout:2620: read oid 8 snap -1 2022-04-22T17:34:40.521 INFO:tasks.rados.rados.0.smithi115.stdout:2620: expect deleted 2022-04-22T17:34:40.521 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 267 2022-04-22T17:34:40.521 INFO:tasks.rados.rados.0.smithi115.stdout:2621: seq_num 785 ranges {0=753664,753664=401408} 2022-04-22T17:34:40.523 INFO:tasks.rados.rados.0.smithi115.stdout:2621: writing smithi11541303-2 from 0 to 753664 tid 1 2022-04-22T17:34:40.526 INFO:tasks.rados.rados.0.smithi115.stdout:2621: writing smithi11541303-2 from 753664 to 1155072 tid 2 2022-04-22T17:34:40.526 INFO:tasks.rados.rados.0.smithi115.stdout:2622: snap_remove snap 234 2022-04-22T17:34:41.033 INFO:tasks.rados.rados.0.smithi115.stdout:2620: done (2 left) 2022-04-22T17:34:41.033 INFO:tasks.rados.rados.0.smithi115.stdout:2622: done (1 left) 2022-04-22T17:34:41.033 INFO:tasks.rados.rados.0.smithi115.stdout:2623: snap_remove snap 261 2022-04-22T17:34:41.150 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:41 smithi121 bash[36902]: cluster 2022-04-22T17:34:39.289705+0000 mgr.y (mgr.34107) 315 : cluster [DBG] pgmap v560: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 366 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 718 KiB/s wr, 12 op/s; 73/645 objects degraded (11.318%) 2022-04-22T17:34:41.150 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:41 smithi121 bash[36902]: audit 2022-04-22T17:34:40.014447+0000 mon.a (mon.0) 1032 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:34:41.150 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:41 smithi121 bash[36902]: cluster 2022-04-22T17:34:40.014574+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2022-04-22T17:34:41.150 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:41 smithi121 bash[36902]: audit 2022-04-22T17:34:40.015901+0000 mon.a (mon.0) 1034 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:41.150 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:41 smithi121 bash[36902]: audit 2022-04-22T17:34:40.025883+0000 mon.b (mon.2) 10 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[45495]: cluster 2022-04-22T17:34:39.289705+0000 mgr.y (mgr.34107) 315 : cluster [DBG] pgmap v560: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 366 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 718 KiB/s wr, 12 op/s; 73/645 objects degraded (11.318%) 2022-04-22T17:34:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[45495]: audit 2022-04-22T17:34:40.014447+0000 mon.a (mon.0) 1032 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:34:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[45495]: cluster 2022-04-22T17:34:40.014574+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2022-04-22T17:34:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[45495]: audit 2022-04-22T17:34:40.015901+0000 mon.a (mon.0) 1034 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:41.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[45495]: audit 2022-04-22T17:34:40.025883+0000 mon.b (mon.2) 10 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:41.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[46934]: cluster 2022-04-22T17:34:39.289705+0000 mgr.y (mgr.34107) 315 : cluster [DBG] pgmap v560: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 366 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 718 KiB/s wr, 12 op/s; 73/645 objects degraded (11.318%) 2022-04-22T17:34:41.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[46934]: audit 2022-04-22T17:34:40.014447+0000 mon.a (mon.0) 1032 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-04-22T17:34:41.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[46934]: cluster 2022-04-22T17:34:40.014574+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2022-04-22T17:34:41.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[46934]: audit 2022-04-22T17:34:40.015901+0000 mon.a (mon.0) 1034 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:41.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[46934]: audit 2022-04-22T17:34:40.025883+0000 mon.b (mon.2) 10 : audit [INF] from='osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2022-04-22T17:34:42.033 INFO:tasks.rados.rados.0.smithi115.stdout:2623: done (1 left) 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2624: delete oid 21 current snap is 267 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2621: finishing write tid 1 to smithi11541303-2 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2621: finishing write tid 2 to smithi11541303-2 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2621: finishing write tid 3 to smithi11541303-2 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2621: finishing write tid 4 to smithi11541303-2 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1639 (ObjNum 785 snap 267 seq_num 785) dirty exists 2022-04-22T17:34:42.034 INFO:tasks.rados.rados.0.smithi115.stdout:2621: left oid 2 (ObjNum 785 snap 267 seq_num 785) 2022-04-22T17:34:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:41.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:41.630Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:42.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:41.630Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[45495]: cluster 2022-04-22T17:34:41.008607+0000 mon.a (mon.0) 1035 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[45495]: cluster 2022-04-22T17:34:41.017347+0000 mon.a (mon.0) 1036 : cluster [INF] osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432] boot 2022-04-22T17:34:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[45495]: cluster 2022-04-22T17:34:41.017440+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2022-04-22T17:34:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[45495]: audit 2022-04-22T17:34:41.018169+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:34:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[45495]: cluster 2022-04-22T17:34:42.019596+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:2621: done (1 left) 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:2624: done (0 left) 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:2625: rollback oid 41 current snap is 267 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 41 to 255 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:2626: copy_from oid 2 from oid 3 current snap is 267 2022-04-22T17:34:42.049 INFO:tasks.rados.rados.0.smithi115.stdout:2627: delete oid 12 current snap is 267 2022-04-22T17:34:42.059 INFO:tasks.rados.rados.0.smithi115.stdout:2627: done (2 left) 2022-04-22T17:34:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 267 2022-04-22T17:34:42.060 INFO:tasks.rados.rados.0.smithi115.stdout:2628: seq_num 788 ranges {0=802816,802816=401408} 2022-04-22T17:34:42.065 INFO:tasks.rados.rados.0.smithi115.stdout:2628: writing smithi11541303-46 from 0 to 802816 tid 1 2022-04-22T17:34:42.067 INFO:tasks.rados.rados.0.smithi115.stdout:2628: writing smithi11541303-46 from 802816 to 1204224 tid 2 2022-04-22T17:34:42.068 INFO:tasks.rados.rados.0.smithi115.stdout:2629: copy_from oid 30 from oid 25 current snap is 267 2022-04-22T17:34:42.068 INFO:tasks.rados.rados.0.smithi115.stdout:2630: snap_remove snap 213 2022-04-22T17:34:42.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[46934]: cluster 2022-04-22T17:34:41.008607+0000 mon.a (mon.0) 1035 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[46934]: cluster 2022-04-22T17:34:41.017347+0000 mon.a (mon.0) 1036 : cluster [INF] osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432] boot 2022-04-22T17:34:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[46934]: cluster 2022-04-22T17:34:41.017440+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2022-04-22T17:34:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[46934]: audit 2022-04-22T17:34:41.018169+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:34:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:42 smithi115 bash[46934]: cluster 2022-04-22T17:34:42.019596+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2022-04-22T17:34:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:42 smithi121 bash[36902]: cluster 2022-04-22T17:34:41.008607+0000 mon.a (mon.0) 1035 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2022-04-22T17:34:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:42 smithi121 bash[36902]: cluster 2022-04-22T17:34:41.017347+0000 mon.a (mon.0) 1036 : cluster [INF] osd.7 [v2:172.21.15.121:6824/2375986432,v1:172.21.15.121:6825/2375986432] boot 2022-04-22T17:34:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:42 smithi121 bash[36902]: cluster 2022-04-22T17:34:41.017440+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2022-04-22T17:34:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:42 smithi121 bash[36902]: audit 2022-04-22T17:34:41.018169+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-04-22T17:34:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:42 smithi121 bash[36902]: cluster 2022-04-22T17:34:42.019596+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2022-04-22T17:34:43.057 INFO:tasks.rados.rados.0.smithi115.stdout:2630: done (4 left) 2022-04-22T17:34:43.057 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 267 2022-04-22T17:34:43.057 INFO:tasks.rados.rados.0.smithi115.stdout:2628: finishing write tid 1 to smithi11541303-46 2022-04-22T17:34:43.057 INFO:tasks.rados.rados.0.smithi115.stdout:2631: seq_num 789 ranges {0=417792,417792=663552,1081344=139264} 2022-04-22T17:34:43.059 INFO:tasks.rados.rados.0.smithi115.stdout:2631: writing smithi11541303-31 from 0 to 417792 tid 1 2022-04-22T17:34:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:2631: writing smithi11541303-31 from 417792 to 1081344 tid 2 2022-04-22T17:34:43.066 INFO:tasks.rados.rados.0.smithi115.stdout:2631: writing smithi11541303-31 from 1081344 to 1220608 tid 3 2022-04-22T17:34:43.066 INFO:tasks.rados.rados.0.smithi115.stdout:2632: delete oid 9 current snap is 267 2022-04-22T17:34:43.066 INFO:tasks.rados.rados.0.smithi115.stdout:2628: finishing write tid 2 to smithi11541303-46 2022-04-22T17:34:43.066 INFO:tasks.rados.rados.0.smithi115.stdout:2628: finishing write tid 3 to smithi11541303-46 2022-04-22T17:34:43.067 INFO:tasks.rados.rados.0.smithi115.stdout:2628: finishing write tid 4 to smithi11541303-46 2022-04-22T17:34:43.067 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1221 (ObjNum 788 snap 267 seq_num 788) dirty exists 2022-04-22T17:34:43.067 INFO:tasks.rados.rados.0.smithi115.stdout:2628: left oid 46 (ObjNum 788 snap 267 seq_num 788) 2022-04-22T17:34:43.067 INFO:tasks.rados.rados.0.smithi115.stdout:2626: finishing copy_from to smithi11541303-2 2022-04-22T17:34:43.067 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1218 (ObjNum 779 snap 267 seq_num 779) dirty exists 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:2626: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:2625: finishing rollback tid 0 to smithi11541303-41 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:2629: finishing copy_from to smithi11541303-30 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1246 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:2629: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:34:43.068 INFO:tasks.rados.rados.0.smithi115.stdout:2625: finishing rollback tid 1 to smithi11541303-41 2022-04-22T17:34:43.069 INFO:tasks.rados.rados.0.smithi115.stdout:2625: finishing rollback tid 2 to smithi11541303-41 2022-04-22T17:34:43.069 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1230 (ObjNum 735 snap 253 seq_num 735) dirty exists 2022-04-22T17:34:43.083 INFO:tasks.rados.rados.0.smithi115.stdout:2631: finishing write tid 1 to smithi11541303-31 2022-04-22T17:34:43.083 INFO:tasks.rados.rados.0.smithi115.stdout:2631: finishing write tid 2 to smithi11541303-31 2022-04-22T17:34:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2631: finishing write tid 3 to smithi11541303-31 2022-04-22T17:34:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2631: finishing write tid 4 to smithi11541303-31 2022-04-22T17:34:43.091 INFO:tasks.rados.rados.0.smithi115.stdout:2625: done (5 left) 2022-04-22T17:34:43.092 INFO:tasks.rados.rados.0.smithi115.stdout:2626: done (4 left) 2022-04-22T17:34:43.092 INFO:tasks.rados.rados.0.smithi115.stdout:2628: done (3 left) 2022-04-22T17:34:43.092 INFO:tasks.rados.rados.0.smithi115.stdout:2629: done (2 left) 2022-04-22T17:34:43.092 INFO:tasks.rados.rados.0.smithi115.stdout:2632: done (1 left) 2022-04-22T17:34:43.092 INFO:tasks.rados.rados.0.smithi115.stdout:2633: delete oid 28 current snap is 267 2022-04-22T17:34:43.094 INFO:tasks.rados.rados.0.smithi115.stdout:2631: finishing write tid 5 to smithi11541303-31 2022-04-22T17:34:43.094 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1731 (ObjNum 789 snap 267 seq_num 789) dirty exists 2022-04-22T17:34:43.094 INFO:tasks.rados.rados.0.smithi115.stdout:2631: left oid 31 (ObjNum 789 snap 267 seq_num 789) 2022-04-22T17:34:43.095 INFO:tasks.rados.rados.0.smithi115.stdout:2631: done (1 left) 2022-04-22T17:34:43.095 INFO:tasks.rados.rados.0.smithi115.stdout:2633: done (0 left) 2022-04-22T17:34:43.095 INFO:tasks.rados.rados.0.smithi115.stdout:2634: snap_remove snap 264 2022-04-22T17:34:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:43 smithi121 bash[36902]: cluster 2022-04-22T17:34:41.290570+0000 mgr.y (mgr.34107) 316 : cluster [DBG] pgmap v563: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 770 KiB/s wr, 10 op/s; 74/651 objects degraded (11.367%) 2022-04-22T17:34:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:43 smithi121 bash[36902]: cluster 2022-04-22T17:34:42.021619+0000 mon.a (mon.0) 1040 : cluster [WRN] Health check update: Degraded data redundancy: 74/651 objects degraded (11.367%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:43.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[45495]: cluster 2022-04-22T17:34:41.290570+0000 mgr.y (mgr.34107) 316 : cluster [DBG] pgmap v563: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 770 KiB/s wr, 10 op/s; 74/651 objects degraded (11.367%) 2022-04-22T17:34:43.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[45495]: cluster 2022-04-22T17:34:42.021619+0000 mon.a (mon.0) 1040 : cluster [WRN] Health check update: Degraded data redundancy: 74/651 objects degraded (11.367%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:43.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[46934]: cluster 2022-04-22T17:34:41.290570+0000 mgr.y (mgr.34107) 316 : cluster [DBG] pgmap v563: 121 pgs: 12 active+undersized+degraded, 37 active+undersized, 72 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 770 KiB/s wr, 10 op/s; 74/651 objects degraded (11.367%) 2022-04-22T17:34:43.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[46934]: cluster 2022-04-22T17:34:42.021619+0000 mon.a (mon.0) 1040 : cluster [WRN] Health check update: Degraded data redundancy: 74/651 objects degraded (11.367%), 12 pgs degraded (PG_DEGRADED) 2022-04-22T17:34:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:43.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:43.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:43.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:43.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:43.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:34:44.068 INFO:tasks.rados.rados.0.smithi115.stdout:2634: done (0 left) 2022-04-22T17:34:44.069 INFO:tasks.rados.rados.0.smithi115.stdout:2635: snap_create 2022-04-22T17:34:44.435 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:44 smithi121 bash[36902]: cluster 2022-04-22T17:34:43.043532+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2022-04-22T17:34:44.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:44 smithi115 bash[46934]: cluster 2022-04-22T17:34:43.043532+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2022-04-22T17:34:44.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:44 smithi115 bash[45495]: cluster 2022-04-22T17:34:43.043532+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2022-04-22T17:34:45.076 INFO:tasks.rados.rados.0.smithi115.stdout:2635: done (0 left) 2022-04-22T17:34:45.076 INFO:tasks.rados.rados.0.smithi115.stdout:2636: rmattr oid 49 current snap is 268 2022-04-22T17:34:45.076 INFO:tasks.rados.rados.0.smithi115.stdout:2636: done (0 left) 2022-04-22T17:34:45.077 INFO:tasks.rados.rados.0.smithi115.stdout:2637: snap_remove snap 236 2022-04-22T17:34:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:45 smithi121 bash[36902]: cluster 2022-04-22T17:34:43.291607+0000 mgr.y (mgr.34107) 317 : cluster [DBG] pgmap v566: 121 pgs: 11 peering, 8 active+undersized+degraded, 22 active+undersized, 80 active+clean; 361 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 20 op/s; 51/672 objects degraded (7.589%) 2022-04-22T17:34:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:45 smithi121 bash[36902]: cluster 2022-04-22T17:34:44.055447+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2022-04-22T17:34:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:45 smithi121 bash[36902]: audit 2022-04-22T17:34:44.329785+0000 mon.a (mon.0) 1043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[45495]: cluster 2022-04-22T17:34:43.291607+0000 mgr.y (mgr.34107) 317 : cluster [DBG] pgmap v566: 121 pgs: 11 peering, 8 active+undersized+degraded, 22 active+undersized, 80 active+clean; 361 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 20 op/s; 51/672 objects degraded (7.589%) 2022-04-22T17:34:45.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[45495]: cluster 2022-04-22T17:34:44.055447+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2022-04-22T17:34:45.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[45495]: audit 2022-04-22T17:34:44.329785+0000 mon.a (mon.0) 1043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[46934]: cluster 2022-04-22T17:34:43.291607+0000 mgr.y (mgr.34107) 317 : cluster [DBG] pgmap v566: 121 pgs: 11 peering, 8 active+undersized+degraded, 22 active+undersized, 80 active+clean; 361 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 20 op/s; 51/672 objects degraded (7.589%) 2022-04-22T17:34:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[46934]: cluster 2022-04-22T17:34:44.055447+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2022-04-22T17:34:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:45 smithi115 bash[46934]: audit 2022-04-22T17:34:44.329785+0000 mon.a (mon.0) 1043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:46.078 INFO:tasks.rados.rados.0.smithi115.stdout:2637: done (0 left) 2022-04-22T17:34:46.078 INFO:tasks.rados.rados.0.smithi115.stdout:2638: rollback oid 11 current snap is 268 2022-04-22T17:34:46.079 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 251 2022-04-22T17:34:46.079 INFO:tasks.rados.rados.0.smithi115.stdout:2639: delete oid 36 current snap is 268 2022-04-22T17:34:46.100 INFO:tasks.rados.rados.0.smithi115.stdout:2638: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:34:46.103 INFO:tasks.rados.rados.0.smithi115.stdout:2639: done (1 left) 2022-04-22T17:34:46.103 INFO:tasks.rados.rados.0.smithi115.stdout:2640: copy_from oid 1 from oid 9 current snap is 268 2022-04-22T17:34:46.104 INFO:tasks.rados.rados.0.smithi115.stdout:2641: snap_remove snap 239 2022-04-22T17:34:46.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: cluster 2022-04-22T17:34:45.062268+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2022-04-22T17:34:46.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.223123+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.233083+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.242932+0000 mon.a (mon.0) 1047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:46.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.245912+0000 mon.a (mon.0) 1048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.247329+0000 mon.a (mon.0) 1049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.248508+0000 mon.a (mon.0) 1050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.250210+0000 mon.a (mon.0) 1051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.259686+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.261357+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.268628+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.270131+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.277195+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.278560+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2022-04-22T17:34:46.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.285735+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.287063+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.294121+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.296066+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.303094+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.304644+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.311357+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.312946+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.319854+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.321239+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.326152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2022-04-22T17:34:46.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:45.328846+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "quincy"}]: dispatch 2022-04-22T17:34:46.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: cluster 2022-04-22T17:34:46.055184+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2022-04-22T17:34:46.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: audit 2022-04-22T17:34:46.064235+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "quincy"}]': finished 2022-04-22T17:34:46.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[46934]: cluster 2022-04-22T17:34:46.064384+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2022-04-22T17:34:46.393 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: cluster 2022-04-22T17:34:45.062268+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2022-04-22T17:34:46.393 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.223123+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.233083+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.242932+0000 mon.a (mon.0) 1047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.245912+0000 mon.a (mon.0) 1048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.247329+0000 mon.a (mon.0) 1049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.248508+0000 mon.a (mon.0) 1050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.250210+0000 mon.a (mon.0) 1051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.259686+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.261357+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.268628+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2022-04-22T17:34:46.394 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.270131+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.277195+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.278560+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.285735+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.287063+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.294121+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.296066+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.303094+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.304644+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.311357+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2022-04-22T17:34:46.395 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.312946+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.319854+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.321239+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.326152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:45.328846+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "quincy"}]: dispatch 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: cluster 2022-04-22T17:34:46.055184+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: audit 2022-04-22T17:34:46.064235+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "quincy"}]': finished 2022-04-22T17:34:46.396 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:46 smithi115 bash[45495]: cluster 2022-04-22T17:34:46.064384+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2022-04-22T17:34:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: cluster 2022-04-22T17:34:45.062268+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2022-04-22T17:34:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.223123+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.233083+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.242932+0000 mon.a (mon.0) 1047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.245912+0000 mon.a (mon.0) 1048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.247329+0000 mon.a (mon.0) 1049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.248508+0000 mon.a (mon.0) 1050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.250210+0000 mon.a (mon.0) 1051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.259686+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.261357+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.268628+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.270131+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.277195+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2022-04-22T17:34:46.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.278560+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.285735+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.287063+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.294121+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.296066+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.303094+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2022-04-22T17:34:46.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.304644+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.311357+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.312946+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.319854+0000 mon.a (mon.0) 1066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.321239+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.326152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2022-04-22T17:34:46.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:45.328846+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "quincy"}]: dispatch 2022-04-22T17:34:46.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: cluster 2022-04-22T17:34:46.055184+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2022-04-22T17:34:46.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: audit 2022-04-22T17:34:46.064235+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "quincy"}]': finished 2022-04-22T17:34:46.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:46 smithi121 bash[36902]: cluster 2022-04-22T17:34:46.064384+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2022-04-22T17:34:47.103 INFO:tasks.rados.rados.0.smithi115.stdout:2641: done (2 left) 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2642: setattr oid 41 current snap is 268 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2638: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2640: finishing copy_from to smithi11541303-1 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2640: got expected ENOENT (src dne) 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2638: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1224 (ObjNum 690 snap 229 seq_num 690) dirty exists 2022-04-22T17:34:47.104 INFO:tasks.rados.rados.0.smithi115.stdout:2640: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:34:47.107 INFO:tasks.rados.rados.0.smithi115.stdout:2638: done (2 left) 2022-04-22T17:34:47.107 INFO:tasks.rados.rados.0.smithi115.stdout:2640: done (1 left) 2022-04-22T17:34:47.108 INFO:tasks.rados.rados.0.smithi115.stdout:2643: snap_remove snap 237 2022-04-22T17:34:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cephadm 2022-04-22T17:34:45.209819+0000 mgr.y (mgr.34107) 318 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cephadm 2022-04-22T17:34:45.251068+0000 mgr.y (mgr.34107) 319 : cephadm [INF] Upgrade: Setting container_image for all osd 2022-04-22T17:34:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cluster 2022-04-22T17:34:45.292195+0000 mgr.y (mgr.34107) 320 : cluster [DBG] pgmap v569: 121 pgs: 11 peering, 3 active+undersized, 107 active+clean; 351 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 8.4 MiB/s, 3 objects/s recovering 2022-04-22T17:34:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cephadm 2022-04-22T17:34:45.328353+0000 mgr.y (mgr.34107) 321 : cephadm [INF] Upgrade: Setting require_osd_release to 17 quincy 2022-04-22T17:34:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: audit 2022-04-22T17:34:46.067995+0000 mon.a (mon.0) 1073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:34:47.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cephadm 2022-04-22T17:34:46.070095+0000 mgr.y (mgr.34107) 322 : cephadm [INF] Upgrade: It appears safe to stop mds.a 2022-04-22T17:34:47.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cluster 2022-04-22T17:34:46.290708+0000 mon.a (mon.0) 1074 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/672 objects degraded (7.589%), 8 pgs degraded) 2022-04-22T17:34:47.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:47 smithi121 bash[36902]: cluster 2022-04-22T17:34:46.290748+0000 mon.a (mon.0) 1075 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cephadm 2022-04-22T17:34:45.209819+0000 mgr.y (mgr.34107) 318 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cephadm 2022-04-22T17:34:45.251068+0000 mgr.y (mgr.34107) 319 : cephadm [INF] Upgrade: Setting container_image for all osd 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cluster 2022-04-22T17:34:45.292195+0000 mgr.y (mgr.34107) 320 : cluster [DBG] pgmap v569: 121 pgs: 11 peering, 3 active+undersized, 107 active+clean; 351 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 8.4 MiB/s, 3 objects/s recovering 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cephadm 2022-04-22T17:34:45.328353+0000 mgr.y (mgr.34107) 321 : cephadm [INF] Upgrade: Setting require_osd_release to 17 quincy 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: audit 2022-04-22T17:34:46.067995+0000 mon.a (mon.0) 1073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cephadm 2022-04-22T17:34:46.070095+0000 mgr.y (mgr.34107) 322 : cephadm [INF] Upgrade: It appears safe to stop mds.a 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cluster 2022-04-22T17:34:46.290708+0000 mon.a (mon.0) 1074 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/672 objects degraded (7.589%), 8 pgs degraded) 2022-04-22T17:34:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[45495]: cluster 2022-04-22T17:34:46.290748+0000 mon.a (mon.0) 1075 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cephadm 2022-04-22T17:34:45.209819+0000 mgr.y (mgr.34107) 318 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cephadm 2022-04-22T17:34:45.251068+0000 mgr.y (mgr.34107) 319 : cephadm [INF] Upgrade: Setting container_image for all osd 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cluster 2022-04-22T17:34:45.292195+0000 mgr.y (mgr.34107) 320 : cluster [DBG] pgmap v569: 121 pgs: 11 peering, 3 active+undersized, 107 active+clean; 351 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 8.4 MiB/s, 3 objects/s recovering 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cephadm 2022-04-22T17:34:45.328353+0000 mgr.y (mgr.34107) 321 : cephadm [INF] Upgrade: Setting require_osd_release to 17 quincy 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: audit 2022-04-22T17:34:46.067995+0000 mon.a (mon.0) 1073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["a"]}]: dispatch 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cephadm 2022-04-22T17:34:46.070095+0000 mgr.y (mgr.34107) 322 : cephadm [INF] Upgrade: It appears safe to stop mds.a 2022-04-22T17:34:47.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cluster 2022-04-22T17:34:46.290708+0000 mon.a (mon.0) 1074 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/672 objects degraded (7.589%), 8 pgs degraded) 2022-04-22T17:34:47.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:47 smithi115 bash[46934]: cluster 2022-04-22T17:34:46.290748+0000 mon.a (mon.0) 1075 : cluster [INF] Cluster is now healthy 2022-04-22T17:34:48.121 INFO:tasks.rados.rados.0.smithi115.stdout:2643: done (1 left) 2022-04-22T17:34:48.121 INFO:tasks.rados.rados.0.smithi115.stdout:2644: delete oid 35 current snap is 268 2022-04-22T17:34:48.121 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1231 (ObjNum 735 snap 253 seq_num 735) dirty exists 2022-04-22T17:34:48.136 INFO:tasks.rados.rados.0.smithi115.stdout:2642: done (1 left) 2022-04-22T17:34:48.137 INFO:tasks.rados.rados.0.smithi115.stdout:2644: done (0 left) 2022-04-22T17:34:48.137 INFO:tasks.rados.rados.0.smithi115.stdout:2645: copy_from oid 14 from oid 22 current snap is 268 2022-04-22T17:34:48.137 INFO:tasks.rados.rados.0.smithi115.stdout:2646: delete oid 40 current snap is 268 2022-04-22T17:34:48.143 INFO:tasks.rados.rados.0.smithi115.stdout:2646: done (1 left) 2022-04-22T17:34:48.143 INFO:tasks.rados.rados.0.smithi115.stdout:2647: delete oid 21 current snap is 268 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2647: done (1 left) 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2648: rmattr oid 28 current snap is 268 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2648: done (1 left) 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2649: read oid 7 snap 262 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2649: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2650: copy_from oid 10 from oid 36 current snap is 268 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 268 2022-04-22T17:34:48.154 INFO:tasks.rados.rados.0.smithi115.stdout:2651: seq_num 796 ranges {0=540672,540672=704512,1245184=98304} 2022-04-22T17:34:48.157 INFO:tasks.rados.rados.0.smithi115.stdout:2651: writing smithi11541303-43 from 0 to 540672 tid 1 2022-04-22T17:34:48.162 INFO:tasks.rados.rados.0.smithi115.stdout:2651: writing smithi11541303-43 from 540672 to 1245184 tid 2 2022-04-22T17:34:48.163 INFO:tasks.rados.rados.0.smithi115.stdout:2651: writing smithi11541303-43 from 1245184 to 1343488 tid 3 2022-04-22T17:34:48.163 INFO:tasks.rados.rados.0.smithi115.stdout:2652: read oid 48 snap -1 2022-04-22T17:34:48.163 INFO:tasks.rados.rados.0.smithi115.stdout:2652: expect (ObjNum 751 snap 258 seq_num 751) 2022-04-22T17:34:48.163 INFO:tasks.rados.rados.0.smithi115.stdout:2653: snap_remove snap 252 2022-04-22T17:34:48.376 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:48 smithi115 bash[45495]: cluster 2022-04-22T17:34:47.089436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2022-04-22T17:34:48.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:48 smithi115 bash[46934]: cluster 2022-04-22T17:34:47.089436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2022-04-22T17:34:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:48 smithi121 bash[36902]: cluster 2022-04-22T17:34:47.089436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2022-04-22T17:34:49.135 INFO:tasks.rados.rados.0.smithi115.stdout:2650: finishing copy_from to smithi11541303-10 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2650: got expected ENOENT (src dne) 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2650: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2651: finishing write tid 1 to smithi11541303-43 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2651: finishing write tid 2 to smithi11541303-43 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2651: finishing write tid 3 to smithi11541303-43 2022-04-22T17:34:49.136 INFO:tasks.rados.rados.0.smithi115.stdout:2652: expect (ObjNum 751 snap 258 seq_num 751) 2022-04-22T17:34:49.180 INFO:tasks.rados.rados.0.smithi115.stdout:2651: finishing write tid 4 to smithi11541303-43 2022-04-22T17:34:49.180 INFO:tasks.rados.rados.0.smithi115.stdout:2651: finishing write tid 5 to smithi11541303-43 2022-04-22T17:34:49.180 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1379 (ObjNum 796 snap 268 seq_num 796) dirty exists 2022-04-22T17:34:49.180 INFO:tasks.rados.rados.0.smithi115.stdout:2651: left oid 43 (ObjNum 796 snap 268 seq_num 796) 2022-04-22T17:34:49.180 INFO:tasks.rados.rados.0.smithi115.stdout:2649: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:34:49.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:49 smithi115 bash[46934]: cluster 2022-04-22T17:34:47.293123+0000 mgr.y (mgr.34107) 323 : cluster [DBG] pgmap v572: 121 pgs: 121 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 6 op/s; 9.5 MiB/s, 4 objects/s recovering 2022-04-22T17:34:49.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:49 smithi115 bash[46934]: cluster 2022-04-22T17:34:48.106947+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2022-04-22T17:34:49.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:49 smithi115 bash[45495]: cluster 2022-04-22T17:34:47.293123+0000 mgr.y (mgr.34107) 323 : cluster [DBG] pgmap v572: 121 pgs: 121 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 6 op/s; 9.5 MiB/s, 4 objects/s recovering 2022-04-22T17:34:49.452 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:49 smithi115 bash[45495]: cluster 2022-04-22T17:34:48.106947+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2645: finishing copy_from to smithi11541303-14 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1127 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2645: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2645: done (5 left) 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2649: done (4 left) 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2650: done (3 left) 2022-04-22T17:34:49.487 INFO:tasks.rados.rados.0.smithi115.stdout:2651: done (2 left) 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2652: done (1 left) 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2653: done (0 left) 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2654: copy_from oid 19 from oid 31 current snap is 268 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2655: read oid 40 snap 260 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2655: expect deleted 2022-04-22T17:34:49.488 INFO:tasks.rados.rados.0.smithi115.stdout:2656: delete oid 25 current snap is 268 2022-04-22T17:34:49.493 INFO:tasks.rados.rados.0.smithi115.stdout:2655: done (2 left) 2022-04-22T17:34:49.493 INFO:tasks.rados.rados.0.smithi115.stdout:2656: done (1 left) 2022-04-22T17:34:49.493 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 268 2022-04-22T17:34:49.493 INFO:tasks.rados.rados.0.smithi115.stdout:2657: seq_num 798 ranges {999424=573440,1572864=466944,2039808=335872} 2022-04-22T17:34:49.506 INFO:tasks.rados.rados.0.smithi115.stdout:2657: writing smithi11541303-48 from 999424 to 1572864 tid 1 2022-04-22T17:34:49.509 INFO:tasks.rados.rados.0.smithi115.stdout:2657: writing smithi11541303-48 from 1572864 to 2039808 tid 2 2022-04-22T17:34:49.511 INFO:tasks.rados.rados.0.smithi115.stdout:2657: writing smithi11541303-48 from 2039808 to 2375680 tid 3 2022-04-22T17:34:49.512 INFO:tasks.rados.rados.0.smithi115.stdout:2658: delete oid 27 current snap is 268 2022-04-22T17:34:49.512 INFO:tasks.rados.rados.0.smithi115.stdout:2654: finishing copy_from to smithi11541303-19 2022-04-22T17:34:49.512 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1731 (ObjNum 789 snap 267 seq_num 789) dirty exists 2022-04-22T17:34:49.512 INFO:tasks.rados.rados.0.smithi115.stdout:2654: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:34:49.514 INFO:tasks.rados.rados.0.smithi115.stdout:2657: finishing write tid 1 to smithi11541303-48 2022-04-22T17:34:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:49 smithi121 bash[36902]: cluster 2022-04-22T17:34:47.293123+0000 mgr.y (mgr.34107) 323 : cluster [DBG] pgmap v572: 121 pgs: 121 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 6 op/s; 9.5 MiB/s, 4 objects/s recovering 2022-04-22T17:34:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:49 smithi121 bash[36902]: cluster 2022-04-22T17:34:48.106947+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2022-04-22T17:34:49.518 INFO:tasks.rados.rados.0.smithi115.stdout:2657: finishing write tid 2 to smithi11541303-48 2022-04-22T17:34:49.518 INFO:tasks.rados.rados.0.smithi115.stdout:2657: finishing write tid 3 to smithi11541303-48 2022-04-22T17:34:49.519 INFO:tasks.rados.rados.0.smithi115.stdout:2657: finishing write tid 4 to smithi11541303-48 2022-04-22T17:34:49.522 INFO:tasks.rados.rados.0.smithi115.stdout:2657: finishing write tid 5 to smithi11541303-48 2022-04-22T17:34:49.522 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1649 (ObjNum 798 snap 268 seq_num 798) dirty exists 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:2657: left oid 48 (ObjNum 798 snap 268 seq_num 798) 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:2654: done (2 left) 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:2657: done (1 left) 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:2658: done (0 left) 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 268 2022-04-22T17:34:49.523 INFO:tasks.rados.rados.0.smithi115.stdout:2659: seq_num 800 ranges {999424=606208,1605632=688128,2293760=114688} 2022-04-22T17:34:49.532 INFO:tasks.rados.rados.0.smithi115.stdout:2659: writing smithi11541303-16 from 999424 to 1605632 tid 1 2022-04-22T17:34:49.536 INFO:tasks.rados.rados.0.smithi115.stdout:2659: writing smithi11541303-16 from 1605632 to 2293760 tid 2 2022-04-22T17:34:49.537 INFO:tasks.rados.rados.0.smithi115.stdout:2659: writing smithi11541303-16 from 2293760 to 2408448 tid 3 2022-04-22T17:34:49.537 INFO:tasks.rados.rados.0.smithi115.stdout:2660: delete oid 47 current snap is 268 2022-04-22T17:34:49.537 INFO:tasks.rados.rados.0.smithi115.stdout:2659: finishing write tid 1 to smithi11541303-16 2022-04-22T17:34:49.541 INFO:tasks.rados.rados.0.smithi115.stdout:2660: done (1 left) 2022-04-22T17:34:49.541 INFO:tasks.rados.rados.0.smithi115.stdout:2661: snap_remove snap 267 2022-04-22T17:34:49.904 INFO:tasks.rados.rados.0.smithi115.stdout:2661: done (1 left) 2022-04-22T17:34:49.905 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 268 2022-04-22T17:34:49.905 INFO:tasks.rados.rados.0.smithi115.stdout:2662: seq_num 802 ranges {0=638976,638976=458752,1097728=344064} 2022-04-22T17:34:49.909 INFO:tasks.rados.rados.0.smithi115.stdout:2662: writing smithi11541303-8 from 0 to 638976 tid 1 2022-04-22T17:34:49.912 INFO:tasks.rados.rados.0.smithi115.stdout:2662: writing smithi11541303-8 from 638976 to 1097728 tid 2 2022-04-22T17:34:49.915 INFO:tasks.rados.rados.0.smithi115.stdout:2662: writing smithi11541303-8 from 1097728 to 1441792 tid 3 2022-04-22T17:34:49.915 INFO:tasks.rados.rados.0.smithi115.stdout:2663: read oid 49 snap -1 2022-04-22T17:34:49.915 INFO:tasks.rados.rados.0.smithi115.stdout:2663: expect deleted 2022-04-22T17:34:49.915 INFO:tasks.rados.rados.0.smithi115.stdout:2664: rollback oid 25 current snap is 268 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 266 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: finishing write tid 2 to smithi11541303-16 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: finishing write tid 3 to smithi11541303-16 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: finishing write tid 4 to smithi11541303-16 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: finishing write tid 5 to smithi11541303-16 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1637 (ObjNum 800 snap 268 seq_num 800) dirty exists 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: left oid 16 (ObjNum 800 snap 268 seq_num 800) 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2659: done (3 left) 2022-04-22T17:34:49.916 INFO:tasks.rados.rados.0.smithi115.stdout:2665: read oid 15 snap -1 2022-04-22T17:34:49.917 INFO:tasks.rados.rados.0.smithi115.stdout:2665: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:49.917 INFO:tasks.rados.rados.0.smithi115.stdout:2666: snap_remove snap 246 2022-04-22T17:34:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: cluster 2022-04-22T17:34:49.113928+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2022-04-22T17:34:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: audit 2022-04-22T17:34:49.168651+0000 mon.a (mon.0) 1079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: audit 2022-04-22T17:34:49.170155+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:34:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: audit 2022-04-22T17:34:49.171413+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: audit 2022-04-22T17:34:49.332779+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:50 smithi121 bash[36902]: cluster 2022-04-22T17:34:49.891318+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2022-04-22T17:34:50.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: cluster 2022-04-22T17:34:49.113928+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2022-04-22T17:34:50.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: audit 2022-04-22T17:34:49.168651+0000 mon.a (mon.0) 1079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:50.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: audit 2022-04-22T17:34:49.170155+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: audit 2022-04-22T17:34:49.171413+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: audit 2022-04-22T17:34:49.332779+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[45495]: cluster 2022-04-22T17:34:49.891318+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: cluster 2022-04-22T17:34:49.113928+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: audit 2022-04-22T17:34:49.168651+0000 mon.a (mon.0) 1079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:50.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: audit 2022-04-22T17:34:49.170155+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:34:50.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: audit 2022-04-22T17:34:49.171413+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:50.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: audit 2022-04-22T17:34:49.332779+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:50.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:50 smithi115 bash[46934]: cluster 2022-04-22T17:34:49.891318+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2022-04-22T17:34:50.931 INFO:tasks.rados.rados.0.smithi115.stdout:2664: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2664: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1387 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: finishing write tid 1 to smithi11541303-8 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: finishing write tid 2 to smithi11541303-8 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: finishing write tid 3 to smithi11541303-8 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: finishing write tid 4 to smithi11541303-8 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: finishing write tid 5 to smithi11541303-8 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1236 (ObjNum 802 snap 268 seq_num 802) dirty exists 2022-04-22T17:34:50.932 INFO:tasks.rados.rados.0.smithi115.stdout:2662: left oid 8 (ObjNum 802 snap 268 seq_num 802) 2022-04-22T17:34:50.933 INFO:tasks.rados.rados.0.smithi115.stdout:2665: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:34:51.064 INFO:tasks.rados.rados.0.smithi115.stdout:2662: done (4 left) 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:2663: done (3 left) 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:2664: done (2 left) 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:2665: done (1 left) 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:2666: done (0 left) 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:2667: rmattr oid 5 current snap is 268 2022-04-22T17:34:51.065 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 268 2022-04-22T17:34:51.066 INFO:tasks.rados.rados.0.smithi115.stdout:2668: seq_num 803 ranges {9977856=655360,10633216=745472,11378688=57344} 2022-04-22T17:34:51.135 INFO:tasks.rados.rados.0.smithi115.stdout:2668: writing smithi11541303-22 from 9977856 to 10633216 tid 1 2022-04-22T17:34:51.140 INFO:tasks.rados.rados.0.smithi115.stdout:2668: writing smithi11541303-22 from 10633216 to 11378688 tid 2 2022-04-22T17:34:51.140 INFO:tasks.rados.rados.0.smithi115.stdout:2668: writing smithi11541303-22 from 11378688 to 11436032 tid 3 2022-04-22T17:34:51.140 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 268 2022-04-22T17:34:51.141 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1780 (ObjNum 774 snap 266 seq_num 774) dirty exists 2022-04-22T17:34:51.141 INFO:tasks.rados.rados.0.smithi115.stdout:2669: seq_num 804 ranges {0=671744,671744=622592,1294336=180224} 2022-04-22T17:34:51.145 INFO:tasks.rados.rados.0.smithi115.stdout:2669: writing smithi11541303-28 from 0 to 671744 tid 1 2022-04-22T17:34:51.149 INFO:tasks.rados.rados.0.smithi115.stdout:2669: writing smithi11541303-28 from 671744 to 1294336 tid 2 2022-04-22T17:34:51.150 INFO:tasks.rados.rados.0.smithi115.stdout:2669: writing smithi11541303-28 from 1294336 to 1474560 tid 3 2022-04-22T17:34:51.150 INFO:tasks.rados.rados.0.smithi115.stdout:2667: done (2 left) 2022-04-22T17:34:51.150 INFO:tasks.rados.rados.0.smithi115.stdout:2670: read oid 19 snap -1 2022-04-22T17:34:51.151 INFO:tasks.rados.rados.0.smithi115.stdout:2670: expect (ObjNum 789 snap 267 seq_num 789) 2022-04-22T17:34:51.151 INFO:tasks.rados.rados.0.smithi115.stdout:2671: snap_remove snap 256 2022-04-22T17:34:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:51 smithi121 bash[36902]: cephadm 2022-04-22T17:34:49.161896+0000 mgr.y (mgr.34107) 324 : cephadm [INF] Upgrade: Updating mds.a 2022-04-22T17:34:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:51 smithi121 bash[36902]: cephadm 2022-04-22T17:34:49.172589+0000 mgr.y (mgr.34107) 325 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:34:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:51 smithi121 bash[36902]: cluster 2022-04-22T17:34:49.293719+0000 mgr.y (mgr.34107) 326 : cluster [DBG] pgmap v575: 121 pgs: 121 active+clean; 304 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 556 KiB/s wr, 3 op/s; 1.1 MiB/s, 1 objects/s recovering 2022-04-22T17:34:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:51 smithi121 bash[36902]: cluster 2022-04-22T17:34:50.907615+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2022-04-22T17:34:51.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[45495]: cephadm 2022-04-22T17:34:49.161896+0000 mgr.y (mgr.34107) 324 : cephadm [INF] Upgrade: Updating mds.a 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[45495]: cephadm 2022-04-22T17:34:49.172589+0000 mgr.y (mgr.34107) 325 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[45495]: cluster 2022-04-22T17:34:49.293719+0000 mgr.y (mgr.34107) 326 : cluster [DBG] pgmap v575: 121 pgs: 121 active+clean; 304 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 556 KiB/s wr, 3 op/s; 1.1 MiB/s, 1 objects/s recovering 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[45495]: cluster 2022-04-22T17:34:50.907615+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[46934]: cephadm 2022-04-22T17:34:49.161896+0000 mgr.y (mgr.34107) 324 : cephadm [INF] Upgrade: Updating mds.a 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[46934]: cephadm 2022-04-22T17:34:49.172589+0000 mgr.y (mgr.34107) 325 : cephadm [INF] Deploying daemon mds.a on smithi115 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[46934]: cluster 2022-04-22T17:34:49.293719+0000 mgr.y (mgr.34107) 326 : cluster [DBG] pgmap v575: 121 pgs: 121 active+clean; 304 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 556 KiB/s wr, 3 op/s; 1.1 MiB/s, 1 objects/s recovering 2022-04-22T17:34:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[46934]: cluster 2022-04-22T17:34:50.907615+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2022-04-22T17:34:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:51.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:51.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:51.631Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:52.144 INFO:tasks.rados.rados.0.smithi115.stdout:2671: done (3 left) 2022-04-22T17:34:52.144 INFO:tasks.rados.rados.0.smithi115.stdout:2672: setattr oid 14 current snap is 268 2022-04-22T17:34:52.146 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 268 2022-04-22T17:34:52.146 INFO:tasks.rados.rados.0.smithi115.stdout:2673: seq_num 805 ranges {2375680=688128,3063808=507904,3571712=294912} 2022-04-22T17:34:52.167 INFO:tasks.rados.rados.0.smithi115.stdout:2673: writing smithi11541303-48 from 2375680 to 3063808 tid 1 2022-04-22T17:34:52.170 INFO:tasks.rados.rados.0.smithi115.stdout:2673: writing smithi11541303-48 from 3063808 to 3571712 tid 2 2022-04-22T17:34:52.172 INFO:tasks.rados.rados.0.smithi115.stdout:2673: writing smithi11541303-48 from 3571712 to 3866624 tid 3 2022-04-22T17:34:52.173 INFO:tasks.rados.rados.0.smithi115.stdout:2674: snap_remove snap 247 2022-04-22T17:34:52.180 INFO:tasks.rados.rados.0.smithi115.stdout:2674: done (5 left) 2022-04-22T17:34:52.180 INFO:tasks.rados.rados.0.smithi115.stdout:2675: snap_remove snap 231 2022-04-22T17:34:52.550 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:52 smithi115 systemd[1]: Stopping Ceph mds.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:34:52.551 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:52 smithi115 bash[31291]: debug 2022-04-22T17:34:52.158+0000 7fe9f1ae1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:34:52.551 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:52 smithi115 bash[31291]: debug 2022-04-22T17:34:52.158+0000 7fe9f1ae1700 -1 mds.a *** got signal Terminated *** 2022-04-22T17:34:53.187 INFO:tasks.rados.rados.0.smithi115.stdout:2675: done (5 left) 2022-04-22T17:34:53.187 INFO:tasks.rados.rados.0.smithi115.stdout:2676: delete oid 40 current snap is 268 2022-04-22T17:34:53.188 INFO:tasks.rados.rados.0.smithi115.stdout:2668: finishing write tid 1 to smithi11541303-22 2022-04-22T17:34:53.188 INFO:tasks.rados.rados.0.smithi115.stdout:2668: finishing write tid 2 to smithi11541303-22 2022-04-22T17:34:53.188 INFO:tasks.rados.rados.0.smithi115.stdout:2668: finishing write tid 3 to smithi11541303-22 2022-04-22T17:34:53.188 INFO:tasks.rados.rados.0.smithi115.stdout:2668: finishing write tid 4 to smithi11541303-22 2022-04-22T17:34:53.188 INFO:tasks.rados.rados.0.smithi115.stdout:2668: finishing write tid 5 to smithi11541303-22 2022-04-22T17:34:53.191 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1139 (ObjNum 803 snap 268 seq_num 803) dirty exists 2022-04-22T17:34:53.191 INFO:tasks.rados.rados.0.smithi115.stdout:2668: left oid 22 (ObjNum 803 snap 268 seq_num 803) 2022-04-22T17:34:53.191 INFO:tasks.rados.rados.0.smithi115.stdout:2669: finishing write tid 1 to smithi11541303-28 2022-04-22T17:34:53.192 INFO:tasks.rados.rados.0.smithi115.stdout:2669: finishing write tid 2 to smithi11541303-28 2022-04-22T17:34:53.192 INFO:tasks.rados.rados.0.smithi115.stdout:2669: finishing write tid 3 to smithi11541303-28 2022-04-22T17:34:53.192 INFO:tasks.rados.rados.0.smithi115.stdout:2669: finishing write tid 4 to smithi11541303-28 2022-04-22T17:34:53.192 INFO:tasks.rados.rados.0.smithi115.stdout:2670: expect (ObjNum 789 snap 267 seq_num 789) 2022-04-22T17:34:53.232 INFO:tasks.rados.rados.0.smithi115.stdout:2669: finishing write tid 5 to smithi11541303-28 2022-04-22T17:34:53.232 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1232 (ObjNum 804 snap 268 seq_num 804) dirty exists 2022-04-22T17:34:53.233 INFO:tasks.rados.rados.0.smithi115.stdout:2669: left oid 28 (ObjNum 804 snap 268 seq_num 804) 2022-04-22T17:34:53.233 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1233 (ObjNum 564 snap 186 seq_num 564) dirty exists 2022-04-22T17:34:53.233 INFO:tasks.rados.rados.0.smithi115.stdout:2673: finishing write tid 1 to smithi11541303-48 2022-04-22T17:34:53.233 INFO:tasks.rados.rados.0.smithi115.stdout:2673: finishing write tid 2 to smithi11541303-48 2022-04-22T17:34:53.233 INFO:tasks.rados.rados.0.smithi115.stdout:2673: finishing write tid 3 to smithi11541303-48 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:2673: finishing write tid 4 to smithi11541303-48 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:2673: finishing write tid 5 to smithi11541303-48 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1654 (ObjNum 805 snap 268 seq_num 805) dirty exists 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:2673: left oid 48 (ObjNum 805 snap 268 seq_num 805) 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:2668: done (5 left) 2022-04-22T17:34:53.234 INFO:tasks.rados.rados.0.smithi115.stdout:2669: done (4 left) 2022-04-22T17:34:53.235 INFO:tasks.rados.rados.0.smithi115.stdout:2670: done (3 left) 2022-04-22T17:34:53.235 INFO:tasks.rados.rados.0.smithi115.stdout:2672: done (2 left) 2022-04-22T17:34:53.235 INFO:tasks.rados.rados.0.smithi115.stdout:2673: done (1 left) 2022-04-22T17:34:53.235 INFO:tasks.rados.rados.0.smithi115.stdout:2676: done (0 left) 2022-04-22T17:34:53.235 INFO:tasks.rados.rados.0.smithi115.stdout:2677: delete oid 19 current snap is 268 2022-04-22T17:34:53.236 INFO:tasks.rados.rados.0.smithi115.stdout:2677: done (0 left) 2022-04-22T17:34:53.236 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 268 2022-04-22T17:34:53.237 INFO:tasks.rados.rados.0.smithi115.stdout:2678: seq_num 808 ranges {9756672=737280,10493952=614400,11108352=188416} 2022-04-22T17:34:53.301 INFO:tasks.rados.rados.0.smithi115.stdout:2678: writing smithi11541303-18 from 9756672 to 10493952 tid 1 2022-04-22T17:34:53.304 INFO:tasks.rados.rados.0.smithi115.stdout:2678: writing smithi11541303-18 from 10493952 to 11108352 tid 2 2022-04-22T17:34:53.305 INFO:tasks.rados.rados.0.smithi115.stdout:2678: writing smithi11541303-18 from 11108352 to 11296768 tid 3 2022-04-22T17:34:53.306 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 268 2022-04-22T17:34:53.306 INFO:tasks.rados.rados.0.smithi115.stdout:2679: seq_num 809 ranges {9977856=753664,10731520=499712,11231232=303104} 2022-04-22T17:34:53.368 INFO:tasks.rados.rados.0.smithi115.stdout:2679: writing smithi11541303-14 from 9977856 to 10731520 tid 1 2022-04-22T17:34:53.371 INFO:tasks.rados.rados.0.smithi115.stdout:2679: writing smithi11541303-14 from 10731520 to 11231232 tid 2 2022-04-22T17:34:53.373 INFO:tasks.rados.rados.0.smithi115.stdout:2679: writing smithi11541303-14 from 11231232 to 11534336 tid 3 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2680: delete oid 35 current snap is 268 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2678: finishing write tid 1 to smithi11541303-18 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2678: finishing write tid 2 to smithi11541303-18 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2678: finishing write tid 3 to smithi11541303-18 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2678: finishing write tid 4 to smithi11541303-18 2022-04-22T17:34:53.374 INFO:tasks.rados.rados.0.smithi115.stdout:2678: finishing write tid 5 to smithi11541303-18 2022-04-22T17:34:53.375 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1807 (ObjNum 808 snap 268 seq_num 808) dirty exists 2022-04-22T17:34:53.375 INFO:tasks.rados.rados.0.smithi115.stdout:2678: left oid 18 (ObjNum 808 snap 268 seq_num 808) 2022-04-22T17:34:53.377 INFO:tasks.rados.rados.0.smithi115.stdout:2678: done (2 left) 2022-04-22T17:34:53.377 INFO:tasks.rados.rados.0.smithi115.stdout:2680: done (1 left) 2022-04-22T17:34:53.378 INFO:tasks.rados.rados.0.smithi115.stdout:2681: setattr oid 35 current snap is 268 2022-04-22T17:34:53.379 INFO:tasks.rados.rados.0.smithi115.stdout:2682: copy_from oid 26 from oid 1 current snap is 268 2022-04-22T17:34:53.379 INFO:tasks.rados.rados.0.smithi115.stdout:2683: copy_from oid 17 from oid 29 current snap is 268 2022-04-22T17:34:53.379 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 268 2022-04-22T17:34:53.380 INFO:tasks.rados.rados.0.smithi115.stdout:2684: seq_num 811 ranges {0=786432,786432=663552,1449984=139264} 2022-04-22T17:34:53.384 INFO:tasks.rados.rados.0.smithi115.stdout:2684: writing smithi11541303-27 from 0 to 786432 tid 1 2022-04-22T17:34:53.388 INFO:tasks.rados.rados.0.smithi115.stdout:2684: writing smithi11541303-27 from 786432 to 1449984 tid 2 2022-04-22T17:34:53.390 INFO:tasks.rados.rados.0.smithi115.stdout:2684: writing smithi11541303-27 from 1449984 to 1589248 tid 3 2022-04-22T17:34:53.390 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 268 2022-04-22T17:34:53.390 INFO:tasks.rados.rados.0.smithi115.stdout:2679: finishing write tid 1 to smithi11541303-14 2022-04-22T17:34:53.390 INFO:tasks.rados.rados.0.smithi115.stdout:2685: seq_num 812 ranges {1056768=802816,1859584=548864,2408448=253952} 2022-04-22T17:34:53.400 INFO:tasks.rados.rados.0.smithi115.stdout:2685: writing smithi11541303-2 from 1056768 to 1859584 tid 1 2022-04-22T17:34:53.403 INFO:tasks.rados.rados.0.smithi115.stdout:2685: writing smithi11541303-2 from 1859584 to 2408448 tid 2 2022-04-22T17:34:53.405 INFO:tasks.rados.rados.0.smithi115.stdout:2685: writing smithi11541303-2 from 2408448 to 2662400 tid 3 2022-04-22T17:34:53.406 INFO:tasks.rados.rados.0.smithi115.stdout:2686: read oid 9 snap -1 2022-04-22T17:34:53.406 INFO:tasks.rados.rados.0.smithi115.stdout:2686: expect deleted 2022-04-22T17:34:53.406 INFO:tasks.rados.rados.0.smithi115.stdout:2687: rmattr oid 28 current snap is 268 2022-04-22T17:34:53.406 INFO:tasks.rados.rados.0.smithi115.stdout:2687: done (7 left) 2022-04-22T17:34:53.406 INFO:tasks.rados.rados.0.smithi115.stdout:2688: rollback oid 44 current snap is 268 2022-04-22T17:34:53.407 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 259 2022-04-22T17:34:53.407 INFO:tasks.rados.rados.0.smithi115.stdout:2679: finishing write tid 2 to smithi11541303-14 2022-04-22T17:34:53.407 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 268 2022-04-22T17:34:53.407 INFO:tasks.rados.rados.0.smithi115.stdout:2679: finishing write tid 3 to smithi11541303-14 2022-04-22T17:34:53.407 INFO:tasks.rados.rados.0.smithi115.stdout:2689: seq_num 813 ranges {3710976=417792,4128768=434176,4562944=442368,5005312=327680} 2022-04-22T17:34:53.428 INFO:tasks.rados.rados.0.smithi115.stdout:2689: writing smithi11541303-15 from 3710976 to 4128768 tid 1 2022-04-22T17:34:53.431 INFO:tasks.rados.rados.0.smithi115.stdout:2689: writing smithi11541303-15 from 4128768 to 4562944 tid 2 2022-04-22T17:34:53.433 INFO:tasks.rados.rados.0.smithi115.stdout:2689: writing smithi11541303-15 from 4562944 to 5005312 tid 3 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:2689: writing smithi11541303-15 from 5005312 to 5332992 tid 4 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:2690: read oid 46 snap -1 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:2690: expect (ObjNum 788 snap 267 seq_num 788) 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 268 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:2679: finishing write tid 4 to smithi11541303-14 2022-04-22T17:34:53.436 INFO:tasks.rados.rados.0.smithi115.stdout:2691: seq_num 814 ranges {0=442368,442368=770048,1212416=434176} 2022-04-22T17:34:53.438 INFO:tasks.rados.rados.0.smithi115.stdout:2691: writing smithi11541303-23 from 0 to 442368 tid 1 2022-04-22T17:34:53.443 INFO:tasks.rados.rados.0.smithi115.stdout:2691: writing smithi11541303-23 from 442368 to 1212416 tid 2 2022-04-22T17:34:53.446 INFO:tasks.rados.rados.0.smithi115.stdout:2691: writing smithi11541303-23 from 1212416 to 1646592 tid 3 2022-04-22T17:34:53.446 INFO:tasks.rados.rados.0.smithi115.stdout:2692: copy_from oid 41 from oid 20 current snap is 268 2022-04-22T17:34:53.446 INFO:tasks.rados.rados.0.smithi115.stdout:2679: finishing write tid 5 to smithi11541303-14 2022-04-22T17:34:53.446 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1237 (ObjNum 809 snap 268 seq_num 809) dirty exists 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2679: left oid 14 (ObjNum 809 snap 268 seq_num 809) 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1641 (ObjNum 1 snap 22075 seq_num 3179198704) dirty exists 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2683: finishing copy_from to smithi11541303-17 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1221 (ObjNum 1 snap 22075 seq_num 3179200656) dirty exists 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2682: finishing copy_from to smithi11541303-26 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1227 (ObjNum 753 snap 258 seq_num 753) dirty exists 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2683: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2679: done (11 left) 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2681: done (10 left) 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2683: done (9 left) 2022-04-22T17:34:53.447 INFO:tasks.rados.rados.0.smithi115.stdout:2693: snap_create 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2682: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2684: finishing write tid 1 to smithi11541303-27 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2684: finishing write tid 2 to smithi11541303-27 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2684: finishing write tid 3 to smithi11541303-27 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2684: finishing write tid 4 to smithi11541303-27 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:2684: finishing write tid 5 to smithi11541303-27 2022-04-22T17:34:53.448 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1658 (ObjNum 811 snap 268 seq_num 811) dirty exists 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2684: left oid 27 (ObjNum 811 snap 268 seq_num 811) 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2685: finishing write tid 1 to smithi11541303-2 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2688: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1306 (ObjNum -1116595056 snap 22075 seq_num 3179303936) dirty dne 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2685: finishing write tid 2 to smithi11541303-2 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2685: finishing write tid 3 to smithi11541303-2 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2685: finishing write tid 4 to smithi11541303-2 2022-04-22T17:34:53.449 INFO:tasks.rados.rados.0.smithi115.stdout:2685: finishing write tid 5 to smithi11541303-2 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1645 (ObjNum 812 snap 268 seq_num 812) dirty exists 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2685: left oid 2 (ObjNum 812 snap 268 seq_num 812) 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 1 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 2 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 3 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 4 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 5 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: finishing write tid 6 to smithi11541303-15 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1541 (ObjNum 813 snap 268 seq_num 813) dirty exists 2022-04-22T17:34:53.450 INFO:tasks.rados.rados.0.smithi115.stdout:2689: left oid 15 (ObjNum 813 snap 268 seq_num 813) 2022-04-22T17:34:53.452 INFO:tasks.rados.rados.0.smithi115.stdout:2690: expect (ObjNum 788 snap 267 seq_num 788) 2022-04-22T17:34:53.487 INFO:tasks.rados.rados.0.smithi115.stdout:2691: finishing write tid 1 to smithi11541303-23 2022-04-22T17:34:53.487 INFO:tasks.rados.rados.0.smithi115.stdout:2691: finishing write tid 2 to smithi11541303-23 2022-04-22T17:34:53.487 INFO:tasks.rados.rados.0.smithi115.stdout:2691: finishing write tid 3 to smithi11541303-23 2022-04-22T17:34:53.487 INFO:tasks.rados.rados.0.smithi115.stdout:2691: finishing write tid 4 to smithi11541303-23 2022-04-22T17:34:53.487 INFO:tasks.rados.rados.0.smithi115.stdout:2691: finishing write tid 5 to smithi11541303-23 2022-04-22T17:34:53.488 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1228 (ObjNum 814 snap 268 seq_num 814) dirty exists 2022-04-22T17:34:53.488 INFO:tasks.rados.rados.0.smithi115.stdout:2691: left oid 23 (ObjNum 814 snap 268 seq_num 814) 2022-04-22T17:34:53.495 INFO:tasks.rados.rados.0.smithi115.stdout:2692: finishing copy_from to smithi11541303-41 2022-04-22T17:34:53.495 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1232 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:34:53.496 INFO:tasks.rados.rados.0.smithi115.stdout:2692: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:34:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:53 smithi121 bash[36902]: cluster 2022-04-22T17:34:51.294413+0000 mgr.y (mgr.34107) 327 : cluster [DBG] pgmap v578: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 306 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 732 KiB/s rd, 336 KiB/s wr, 8 op/s 2022-04-22T17:34:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:53 smithi121 bash[36902]: cluster 2022-04-22T17:34:52.129158+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2022-04-22T17:34:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:53 smithi121 bash[36902]: cluster 2022-04-22T17:34:52.161393+0000 mon.a (mon.0) 1086 : cluster [WRN] daemon mds.a compat changed unexpectedly 2022-04-22T17:34:53.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:53 smithi121 bash[36902]: cluster 2022-04-22T17:34:52.167392+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2022-04-22T17:34:53.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[45495]: cluster 2022-04-22T17:34:51.294413+0000 mgr.y (mgr.34107) 327 : cluster [DBG] pgmap v578: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 306 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 732 KiB/s rd, 336 KiB/s wr, 8 op/s 2022-04-22T17:34:53.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[45495]: cluster 2022-04-22T17:34:52.129158+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2022-04-22T17:34:53.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[45495]: cluster 2022-04-22T17:34:52.161393+0000 mon.a (mon.0) 1086 : cluster [WRN] daemon mds.a compat changed unexpectedly 2022-04-22T17:34:53.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[45495]: cluster 2022-04-22T17:34:52.167392+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2022-04-22T17:34:53.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[46934]: cluster 2022-04-22T17:34:51.294413+0000 mgr.y (mgr.34107) 327 : cluster [DBG] pgmap v578: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 306 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 732 KiB/s rd, 336 KiB/s wr, 8 op/s 2022-04-22T17:34:53.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[46934]: cluster 2022-04-22T17:34:52.129158+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2022-04-22T17:34:53.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[46934]: cluster 2022-04-22T17:34:52.161393+0000 mon.a (mon.0) 1086 : cluster [WRN] daemon mds.a compat changed unexpectedly 2022-04-22T17:34:53.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[46934]: cluster 2022-04-22T17:34:52.167392+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2022-04-22T17:34:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:53.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:53.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:34:53.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:34:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:34:53.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2682: done (9 left) 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2684: done (8 left) 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2685: done (7 left) 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2686: done (6 left) 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2688: done (5 left) 2022-04-22T17:34:54.196 INFO:tasks.rados.rados.0.smithi115.stdout:2689: done (4 left) 2022-04-22T17:34:54.197 INFO:tasks.rados.rados.0.smithi115.stdout:2690: done (3 left) 2022-04-22T17:34:54.197 INFO:tasks.rados.rados.0.smithi115.stdout:2691: done (2 left) 2022-04-22T17:34:54.197 INFO:tasks.rados.rados.0.smithi115.stdout:2692: done (1 left) 2022-04-22T17:34:54.197 INFO:tasks.rados.rados.0.smithi115.stdout:2693: done (0 left) 2022-04-22T17:34:54.197 INFO:tasks.rados.rados.0.smithi115.stdout:2694: snap_create 2022-04-22T17:34:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:54 smithi121 bash[36902]: cluster 2022-04-22T17:34:53.127909+0000 mon.a (mon.0) 1088 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 2022-04-22T17:34:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:54 smithi121 bash[36902]: cluster 2022-04-22T17:34:53.172843+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2022-04-22T17:34:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:54 smithi121 bash[36902]: cluster 2022-04-22T17:34:53.295060+0000 mgr.y (mgr.34107) 328 : cluster [DBG] pgmap v582: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.5 MiB/s wr, 23 op/s 2022-04-22T17:34:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:54 smithi121 bash[36902]: audit 2022-04-22T17:34:53.438080+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:54.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[45495]: cluster 2022-04-22T17:34:53.127909+0000 mon.a (mon.0) 1088 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 2022-04-22T17:34:54.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[45495]: cluster 2022-04-22T17:34:53.172843+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2022-04-22T17:34:54.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[45495]: cluster 2022-04-22T17:34:53.295060+0000 mgr.y (mgr.34107) 328 : cluster [DBG] pgmap v582: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.5 MiB/s wr, 23 op/s 2022-04-22T17:34:54.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[45495]: audit 2022-04-22T17:34:53.438080+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:54.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[46934]: cluster 2022-04-22T17:34:53.127909+0000 mon.a (mon.0) 1088 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 2022-04-22T17:34:54.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[46934]: cluster 2022-04-22T17:34:53.172843+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2022-04-22T17:34:54.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[46934]: cluster 2022-04-22T17:34:53.295060+0000 mgr.y (mgr.34107) 328 : cluster [DBG] pgmap v582: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.5 MiB/s wr, 23 op/s 2022-04-22T17:34:54.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:54 smithi115 bash[46934]: audit 2022-04-22T17:34:53.438080+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:54.813 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:34:54.911 INFO:tasks.rados.rados.0.smithi115.stdout:2694: done (0 left) 2022-04-22T17:34:54.911 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 270 2022-04-22T17:34:54.911 INFO:tasks.rados.rados.0.smithi115.stdout:2695: seq_num 815 ranges {704512=458752,1163264=655360,1818624=409600,2228224=139264} 2022-04-22T17:34:54.918 INFO:tasks.rados.rados.0.smithi115.stdout:2695: writing smithi11541303-42 from 704512 to 1163264 tid 1 2022-04-22T17:34:54.922 INFO:tasks.rados.rados.0.smithi115.stdout:2695: writing smithi11541303-42 from 1163264 to 1818624 tid 2 2022-04-22T17:34:54.924 INFO:tasks.rados.rados.0.smithi115.stdout:2695: writing smithi11541303-42 from 1818624 to 2228224 tid 3 2022-04-22T17:34:54.926 INFO:tasks.rados.rados.0.smithi115.stdout:2695: writing smithi11541303-42 from 2228224 to 2367488 tid 4 2022-04-22T17:34:54.926 INFO:tasks.rados.rados.0.smithi115.stdout:2696: read oid 35 snap 263 2022-04-22T17:34:54.926 INFO:tasks.rados.rados.0.smithi115.stdout:2696: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:34:54.926 INFO:tasks.rados.rados.0.smithi115.stdout:2697: delete oid 40 current snap is 270 2022-04-22T17:34:54.932 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 1 to smithi11541303-42 2022-04-22T17:34:54.935 INFO:tasks.rados.rados.0.smithi115.stdout:2697: done (2 left) 2022-04-22T17:34:54.935 INFO:tasks.rados.rados.0.smithi115.stdout:2698: snap_create 2022-04-22T17:34:54.936 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 2 to smithi11541303-42 2022-04-22T17:34:54.936 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 3 to smithi11541303-42 2022-04-22T17:34:54.937 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 4 to smithi11541303-42 2022-04-22T17:34:54.937 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 5 to smithi11541303-42 2022-04-22T17:34:54.939 INFO:tasks.rados.rados.0.smithi115.stdout:2695: finishing write tid 6 to smithi11541303-42 2022-04-22T17:34:54.939 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1813 (ObjNum 815 snap 270 seq_num 815) dirty exists 2022-04-22T17:34:54.939 INFO:tasks.rados.rados.0.smithi115.stdout:2695: left oid 42 (ObjNum 815 snap 270 seq_num 815) 2022-04-22T17:34:54.977 INFO:tasks.rados.rados.0.smithi115.stdout:2696: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (11m) 3m ago 11m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (11m) 19s ago 11m 24.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (12m) 3m ago 12m 14.0M - 16.2.5 46cf6318c64e 87028d9b9205 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (12m) 19s ago 12m 20.1M - 16.2.5 46cf6318c64e 6b38774e4fd8 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (7m) 19s ago 17m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (7m) 3m ago 19m 448M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:34:55.156 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6m) 3m ago 19m 57.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5m) 19s ago 17m 55.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6m) 3m ago 17m 49.7M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (11m) 3m ago 12m 9755k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (11m) 19s ago 11m 9623k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4m) 3m ago 16m 194M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4m) 3m ago 16m 195M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3m) 3m ago 15m 161M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3m) 3m ago 15m - 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2m) 19s ago 14m 231M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:34:55.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (105s) 19s ago 14m 344M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:34:55.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (63s) 19s ago 13m 192M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:34:55.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (21s) 19s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:34:55.158 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (8m) 19s ago 12m 40.7M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: audit 2022-04-22T17:34:54.151978+0000 mon.a (mon.0) 1091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: cluster 2022-04-22T17:34:54.179874+0000 mon.a (mon.0) 1092 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2022-04-22T17:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: audit 2022-04-22T17:34:54.335839+0000 mon.a (mon.0) 1093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: audit 2022-04-22T17:34:54.783807+0000 mgr.y (mgr.34107) 329 : audit [DBG] from='client.34407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: cluster 2022-04-22T17:34:54.898112+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2022-04-22T17:34:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:55 smithi121 bash[36902]: audit 2022-04-22T17:34:55.124904+0000 mgr.y (mgr.34107) 330 : audit [DBG] from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: audit 2022-04-22T17:34:54.151978+0000 mon.a (mon.0) 1091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:34:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: cluster 2022-04-22T17:34:54.179874+0000 mon.a (mon.0) 1092 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2022-04-22T17:34:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: audit 2022-04-22T17:34:54.335839+0000 mon.a (mon.0) 1093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: audit 2022-04-22T17:34:54.783807+0000 mgr.y (mgr.34107) 329 : audit [DBG] from='client.34407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: cluster 2022-04-22T17:34:54.898112+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[45495]: audit 2022-04-22T17:34:55.124904+0000 mgr.y (mgr.34107) 330 : audit [DBG] from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: audit 2022-04-22T17:34:54.151978+0000 mon.a (mon.0) 1091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: cluster 2022-04-22T17:34:54.179874+0000 mon.a (mon.0) 1092 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: audit 2022-04-22T17:34:54.335839+0000 mon.a (mon.0) 1093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: audit 2022-04-22T17:34:54.783807+0000 mgr.y (mgr.34107) 329 : audit [DBG] from='client.34407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: cluster 2022-04-22T17:34:54.898112+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2022-04-22T17:34:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:55 smithi115 bash[46934]: audit 2022-04-22T17:34:55.124904+0000 mgr.y (mgr.34107) 330 : audit [DBG] from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:34:55.569 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:34:55.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 13 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:34:55.571 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:34:55.920 INFO:tasks.rados.rados.0.smithi115.stdout:2695: done (2 left) 2022-04-22T17:34:55.920 INFO:tasks.rados.rados.0.smithi115.stdout:2696: done (1 left) 2022-04-22T17:34:55.920 INFO:tasks.rados.rados.0.smithi115.stdout:2698: done (0 left) 2022-04-22T17:34:55.920 INFO:tasks.rados.rados.0.smithi115.stdout:2699: delete oid 23 current snap is 271 2022-04-22T17:34:55.924 INFO:tasks.rados.rados.0.smithi115.stdout:2699: done (0 left) 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:2700: read oid 33 snap 259 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:2700: expect (ObjNum 744 snap 255 seq_num 744) 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:2701: rmattr oid 11 current snap is 271 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:2701: done (1 left) 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 271 2022-04-22T17:34:55.925 INFO:tasks.rados.rados.0.smithi115.stdout:2702: seq_num 818 ranges {0=507904,507904=704512,1212416=499712} 2022-04-22T17:34:55.929 INFO:tasks.rados.rados.0.smithi115.stdout:2702: writing smithi11541303-12 from 0 to 507904 tid 1 2022-04-22T17:34:55.934 INFO:tasks.rados.rados.0.smithi115.stdout:2702: writing smithi11541303-12 from 507904 to 1212416 tid 2 2022-04-22T17:34:55.939 INFO:tasks.rados.rados.0.smithi115.stdout:2702: writing smithi11541303-12 from 1212416 to 1712128 tid 3 2022-04-22T17:34:55.939 INFO:tasks.rados.rados.0.smithi115.stdout:2703: rollback oid 10 current snap is 271 2022-04-22T17:34:55.939 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 259 2022-04-22T17:34:55.939 INFO:tasks.rados.rados.0.smithi115.stdout:2702: finishing write tid 1 to smithi11541303-12 2022-04-22T17:34:55.939 INFO:tasks.rados.rados.0.smithi115.stdout:2704: setattr oid 37 current snap is 271 2022-04-22T17:34:55.940 INFO:tasks.rados.rados.0.smithi115.stdout:2705: rollback oid 50 current snap is 271 2022-04-22T17:34:55.940 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 259 2022-04-22T17:34:55.940 INFO:tasks.rados.rados.0.smithi115.stdout:2706: snap_create 2022-04-22T17:34:55.941 INFO:tasks.rados.rados.0.smithi115.stdout:2702: finishing write tid 2 to smithi11541303-12 2022-04-22T17:34:55.942 INFO:tasks.rados.rados.0.smithi115.stdout:2702: finishing write tid 3 to smithi11541303-12 2022-04-22T17:34:55.942 INFO:tasks.rados.rados.0.smithi115.stdout:2702: finishing write tid 4 to smithi11541303-12 2022-04-22T17:34:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:2705: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:34:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1646 (ObjNum -1116650880 snap 22075 seq_num 0) dirty exists 2022-04-22T17:34:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:2702: finishing write tid 5 to smithi11541303-12 2022-04-22T17:34:55.947 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1736 (ObjNum 818 snap 271 seq_num 818) dirty exists 2022-04-22T17:34:55.947 INFO:tasks.rados.rados.0.smithi115.stdout:2702: left oid 12 (ObjNum 818 snap 271 seq_num 818) 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:2705: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:2705: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1640 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:2703: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:2703: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:34:55.948 INFO:tasks.rados.rados.0.smithi115.stdout:2703: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:34:55.949 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1235 (ObjNum 742 snap 255 seq_num 742) dirty exists 2022-04-22T17:34:55.958 INFO:tasks.rados.rados.0.smithi115.stdout:2700: expect (ObjNum 744 snap 255 seq_num 744) 2022-04-22T17:34:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:56 smithi121 bash[36902]: cluster 2022-04-22T17:34:55.295849+0000 mgr.y (mgr.34107) 331 : cluster [DBG] pgmap v585: 121 pgs: 121 active+clean; 327 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.6 MiB/s wr, 16 op/s 2022-04-22T17:34:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:56 smithi121 bash[36902]: audit 2022-04-22T17:34:55.556920+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.115:0/1518082165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:56 smithi121 bash[36902]: cluster 2022-04-22T17:34:55.906904+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2022-04-22T17:34:56.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[46934]: cluster 2022-04-22T17:34:55.295849+0000 mgr.y (mgr.34107) 331 : cluster [DBG] pgmap v585: 121 pgs: 121 active+clean; 327 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.6 MiB/s wr, 16 op/s 2022-04-22T17:34:56.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[46934]: audit 2022-04-22T17:34:55.556920+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.115:0/1518082165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:56.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[46934]: cluster 2022-04-22T17:34:55.906904+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2022-04-22T17:34:56.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[45495]: cluster 2022-04-22T17:34:55.295849+0000 mgr.y (mgr.34107) 331 : cluster [DBG] pgmap v585: 121 pgs: 121 active+clean; 327 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.6 MiB/s wr, 16 op/s 2022-04-22T17:34:56.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[45495]: audit 2022-04-22T17:34:55.556920+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.115:0/1518082165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:34:56.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:56 smithi115 bash[45495]: cluster 2022-04-22T17:34:55.906904+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2022-04-22T17:34:56.924 INFO:tasks.rados.rados.0.smithi115.stdout:2700: done (5 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2702: done (4 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2703: done (3 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2704: done (2 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2705: done (1 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2706: done (0 left) 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2707: rollback oid 14 current snap is 272 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 245 2022-04-22T17:34:56.925 INFO:tasks.rados.rados.0.smithi115.stdout:2708: snap_create 2022-04-22T17:34:56.932 INFO:tasks.rados.rados.0.smithi115.stdout:2707: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:34:56.932 INFO:tasks.rados.rados.0.smithi115.stdout:2707: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:34:56.933 INFO:tasks.rados.rados.0.smithi115.stdout:2707: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:34:56.933 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1240 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:34:57.551 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[57747]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds.a 2022-04-22T17:34:57.552 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[57861]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds-a 2022-04-22T17:34:57.552 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[57874]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds.a 2022-04-22T17:34:57.553 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mds.a.service: Succeeded. 2022-04-22T17:34:57.554 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 systemd[1]: Stopped Ceph mds.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:34:57.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[45495]: cluster 2022-04-22T17:34:56.911565+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2022-04-22T17:34:57.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[45495]: audit 2022-04-22T17:34:57.617861+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:57.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[45495]: audit 2022-04-22T17:34:57.619683+0000 mon.a (mon.0) 1098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:57.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[45495]: audit 2022-04-22T17:34:57.621384+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:57.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[45495]: audit 2022-04-22T17:34:57.622171+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:57.927 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:57 smithi115 systemd[1]: Started Ceph mds.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:34:57.929 INFO:tasks.rados.rados.0.smithi115.stdout:2707: done (1 left) 2022-04-22T17:34:57.929 INFO:tasks.rados.rados.0.smithi115.stdout:2708: done (0 left) 2022-04-22T17:34:57.929 INFO:tasks.rados.rados.0.smithi115.stdout:2709: read oid 49 snap -1 2022-04-22T17:34:57.929 INFO:tasks.rados.rados.0.smithi115.stdout:2709: expect deleted 2022-04-22T17:34:57.930 INFO:tasks.rados.rados.0.smithi115.stdout:2710: rmattr oid 6 current snap is 273 2022-04-22T17:34:57.930 INFO:tasks.rados.rados.0.smithi115.stdout:2710: done (1 left) 2022-04-22T17:34:57.930 INFO:tasks.rados.rados.0.smithi115.stdout:2711: rollback oid 28 current snap is 273 2022-04-22T17:34:57.930 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 249 2022-04-22T17:34:57.931 INFO:tasks.rados.rados.0.smithi115.stdout:2712: copy_from oid 34 from oid 35 current snap is 273 2022-04-22T17:34:57.931 INFO:tasks.rados.rados.0.smithi115.stdout:2713: delete oid 27 current snap is 273 2022-04-22T17:34:57.940 INFO:tasks.rados.rados.0.smithi115.stdout:2709: done (3 left) 2022-04-22T17:34:57.941 INFO:tasks.rados.rados.0.smithi115.stdout:2713: done (2 left) 2022-04-22T17:34:57.941 INFO:tasks.rados.rados.0.smithi115.stdout:2714: read oid 19 snap 270 2022-04-22T17:34:57.941 INFO:tasks.rados.rados.0.smithi115.stdout:2714: expect deleted 2022-04-22T17:34:57.941 INFO:tasks.rados.rados.0.smithi115.stdout:2715: snap_create 2022-04-22T17:34:57.943 INFO:tasks.rados.rados.0.smithi115.stdout:2711: finishing rollback tid 0 to smithi11541303-28 2022-04-22T17:34:57.943 INFO:tasks.rados.rados.0.smithi115.stdout:2711: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:34:57.943 INFO:tasks.rados.rados.0.smithi115.stdout:2711: finishing rollback tid 2 to smithi11541303-28 2022-04-22T17:34:57.943 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1243 (ObjNum 681 snap 227 seq_num 681) dirty exists 2022-04-22T17:34:57.943 INFO:tasks.rados.rados.0.smithi115.stdout:2712: finishing copy_from to smithi11541303-34 2022-04-22T17:34:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1641 (ObjNum -1115506512 snap 22075 seq_num 3179424112) dirty exists 2022-04-22T17:34:57.944 INFO:tasks.rados.rados.0.smithi115.stdout:2712: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:34:58.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[46934]: cluster 2022-04-22T17:34:56.911565+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2022-04-22T17:34:58.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[46934]: audit 2022-04-22T17:34:57.617861+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:58.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[46934]: audit 2022-04-22T17:34:57.619683+0000 mon.a (mon.0) 1098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:58.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[46934]: audit 2022-04-22T17:34:57.621384+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:58.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:57 smithi115 bash[46934]: audit 2022-04-22T17:34:57.622171+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:57 smithi121 bash[36902]: cluster 2022-04-22T17:34:56.911565+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2022-04-22T17:34:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:57 smithi121 bash[36902]: audit 2022-04-22T17:34:57.617861+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:34:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:57 smithi121 bash[36902]: audit 2022-04-22T17:34:57.619683+0000 mon.a (mon.0) 1098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:34:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:57 smithi121 bash[36902]: audit 2022-04-22T17:34:57.621384+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:34:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:57 smithi121 bash[36902]: audit 2022-04-22T17:34:57.622171+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:34:58.939 INFO:tasks.rados.rados.0.smithi115.stdout:2711: done (3 left) 2022-04-22T17:34:58.939 INFO:tasks.rados.rados.0.smithi115.stdout:2712: done (2 left) 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2714: done (1 left) 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2715: done (0 left) 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2716: rollback oid 18 current snap is 274 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 266 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2717: copy_from oid 24 from oid 30 current snap is 274 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2718: copy_from oid 10 from oid 22 current snap is 274 2022-04-22T17:34:58.940 INFO:tasks.rados.rados.0.smithi115.stdout:2719: snap_remove snap 260 2022-04-22T17:34:59.176 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:58 smithi115 bash[45495]: cluster 2022-04-22T17:34:57.296429+0000 mgr.y (mgr.34107) 332 : cluster [DBG] pgmap v588: 121 pgs: 121 active+clean; 310 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:34:59.176 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:58 smithi115 bash[45495]: cluster 2022-04-22T17:34:57.916076+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2022-04-22T17:34:59.177 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:58 smithi115 bash[46934]: cluster 2022-04-22T17:34:57.296429+0000 mgr.y (mgr.34107) 332 : cluster [DBG] pgmap v588: 121 pgs: 121 active+clean; 310 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:34:59.177 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:58 smithi115 bash[46934]: cluster 2022-04-22T17:34:57.916076+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2022-04-22T17:34:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:58 smithi121 bash[36902]: cluster 2022-04-22T17:34:57.296429+0000 mgr.y (mgr.34107) 332 : cluster [DBG] pgmap v588: 121 pgs: 121 active+clean; 310 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:34:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:58 smithi121 bash[36902]: cluster 2022-04-22T17:34:57.916076+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2022-04-22T17:34:59.550 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[57947]: starting mds.a at 2022-04-22T17:34:59.919 INFO:tasks.rados.rados.0.smithi115.stdout:2719: done (3 left) 2022-04-22T17:34:59.919 INFO:tasks.rados.rados.0.smithi115.stdout:2720: read oid 16 snap -1 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:2720: expect (ObjNum 800 snap 268 seq_num 800) 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:2721: rollback oid 11 current snap is 274 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 266 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:2722: copy_from oid 35 from oid 42 current snap is 274 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 274 2022-04-22T17:34:59.920 INFO:tasks.rados.rados.0.smithi115.stdout:2723: seq_num 820 ranges {10780672=540672,11321344=475136,11796480=598016,12394496=131072} 2022-04-22T17:34:59.982 INFO:tasks.rados.rados.0.smithi115.stdout:2723: writing smithi11541303-39 from 10780672 to 11321344 tid 1 2022-04-22T17:34:59.985 INFO:tasks.rados.rados.0.smithi115.stdout:2723: writing smithi11541303-39 from 11321344 to 11796480 tid 2 2022-04-22T17:34:59.988 INFO:tasks.rados.rados.0.smithi115.stdout:2723: writing smithi11541303-39 from 11796480 to 12394496 tid 3 2022-04-22T17:34:59.989 INFO:tasks.rados.rados.0.smithi115.stdout:2723: writing smithi11541303-39 from 12394496 to 12525568 tid 4 2022-04-22T17:34:59.989 INFO:tasks.rados.rados.0.smithi115.stdout:2724: snap_remove snap 255 2022-04-22T17:35:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:59 smithi121 bash[36902]: cluster 2022-04-22T17:34:58.925594+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2022-04-22T17:35:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:59 smithi121 bash[36902]: audit 2022-04-22T17:34:59.339015+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:34:59 smithi121 bash[36902]: cluster 2022-04-22T17:34:59.900077+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2022-04-22T17:35:00.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[45495]: cluster 2022-04-22T17:34:58.925594+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2022-04-22T17:35:00.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[45495]: audit 2022-04-22T17:34:59.339015+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:00.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[45495]: cluster 2022-04-22T17:34:59.900077+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2022-04-22T17:35:00.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[46934]: cluster 2022-04-22T17:34:58.925594+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2022-04-22T17:35:00.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[46934]: audit 2022-04-22T17:34:59.339015+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:00.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:34:59 smithi115 bash[46934]: cluster 2022-04-22T17:34:59.900077+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2022-04-22T17:35:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2724: done (7 left) 2022-04-22T17:35:00.967 INFO:tasks.rados.rados.0.smithi115.stdout:2725: delete oid 26 current snap is 274 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2716: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2716: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2716: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1817 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2717: finishing copy_from to smithi11541303-24 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1246 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2717: finishing copy_from racing read to smithi11541303-24 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:2718: finishing copy_from to smithi11541303-10 2022-04-22T17:35:00.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1139 (ObjNum 803 snap 268 seq_num 803) dirty exists 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2718: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2721: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2721: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2721: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1232 (ObjNum 749 snap 257 seq_num 749) dirty exists 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2722: finishing copy_from to smithi11541303-35 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1813 (ObjNum 815 snap 270 seq_num 815) dirty exists 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2722: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:35:00.969 INFO:tasks.rados.rados.0.smithi115.stdout:2720: expect (ObjNum 800 snap 268 seq_num 800) 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 1 to smithi11541303-39 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 2 to smithi11541303-39 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 3 to smithi11541303-39 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 4 to smithi11541303-39 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 5 to smithi11541303-39 2022-04-22T17:35:01.065 INFO:tasks.rados.rados.0.smithi115.stdout:2723: finishing write tid 6 to smithi11541303-39 2022-04-22T17:35:01.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1818 (ObjNum 820 snap 274 seq_num 820) dirty exists 2022-04-22T17:35:01.066 INFO:tasks.rados.rados.0.smithi115.stdout:2723: left oid 39 (ObjNum 820 snap 274 seq_num 820) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2716: done (7 left) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2717: done (6 left) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2718: done (5 left) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2720: done (4 left) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2721: done (3 left) 2022-04-22T17:35:01.070 INFO:tasks.rados.rados.0.smithi115.stdout:2722: done (2 left) 2022-04-22T17:35:01.071 INFO:tasks.rados.rados.0.smithi115.stdout:2723: done (1 left) 2022-04-22T17:35:01.071 INFO:tasks.rados.rados.0.smithi115.stdout:2725: done (0 left) 2022-04-22T17:35:01.071 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 274 2022-04-22T17:35:01.071 INFO:tasks.rados.rados.0.smithi115.stdout:2726: seq_num 822 ranges {1343488=573440,1916928=696320,2613248=507904} 2022-04-22T17:35:01.087 INFO:tasks.rados.rados.0.smithi115.stdout:2726: writing smithi11541303-43 from 1343488 to 1916928 tid 1 2022-04-22T17:35:01.090 INFO:tasks.rados.rados.0.smithi115.stdout:2726: writing smithi11541303-43 from 1916928 to 2613248 tid 2 2022-04-22T17:35:01.093 INFO:tasks.rados.rados.0.smithi115.stdout:2726: writing smithi11541303-43 from 2613248 to 3121152 tid 3 2022-04-22T17:35:01.093 INFO:tasks.rados.rados.0.smithi115.stdout:2727: snap_remove snap 269 2022-04-22T17:35:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:00 smithi121 bash[36902]: cluster 2022-04-22T17:34:59.297097+0000 mgr.y (mgr.34107) 333 : cluster [DBG] pgmap v591: 121 pgs: 121 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.9 MiB/s wr, 24 op/s 2022-04-22T17:35:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:00 smithi121 bash[36902]: cluster 2022-04-22T17:34:59.943553+0000 mon.a (mon.0) 1105 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:boot 2022-04-22T17:35:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:00 smithi121 bash[36902]: cluster 2022-04-22T17:34:59.943615+0000 mon.a (mon.0) 1106 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:35:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:00 smithi121 bash[36902]: audit 2022-04-22T17:34:59.945645+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:35:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[46934]: cluster 2022-04-22T17:34:59.297097+0000 mgr.y (mgr.34107) 333 : cluster [DBG] pgmap v591: 121 pgs: 121 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.9 MiB/s wr, 24 op/s 2022-04-22T17:35:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[46934]: cluster 2022-04-22T17:34:59.943553+0000 mon.a (mon.0) 1105 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:boot 2022-04-22T17:35:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[46934]: cluster 2022-04-22T17:34:59.943615+0000 mon.a (mon.0) 1106 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:35:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[46934]: audit 2022-04-22T17:34:59.945645+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:35:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[45495]: cluster 2022-04-22T17:34:59.297097+0000 mgr.y (mgr.34107) 333 : cluster [DBG] pgmap v591: 121 pgs: 121 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.9 MiB/s wr, 24 op/s 2022-04-22T17:35:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[45495]: cluster 2022-04-22T17:34:59.943553+0000 mon.a (mon.0) 1105 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:boot 2022-04-22T17:35:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[45495]: cluster 2022-04-22T17:34:59.943615+0000 mon.a (mon.0) 1106 : cluster [DBG] fsmap cephfs:1 {0=b=up:active} 1 up:standby 2022-04-22T17:35:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:00 smithi115 bash[45495]: audit 2022-04-22T17:34:59.945645+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-04-22T17:35:01.971 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:01.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:01.979 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:01.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:01.979 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:01.631Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:01.980 INFO:tasks.rados.rados.0.smithi115.stdout:2727: done (1 left) 2022-04-22T17:35:01.980 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 274 2022-04-22T17:35:01.980 INFO:tasks.rados.rados.0.smithi115.stdout:2726: finishing write tid 1 to smithi11541303-43 2022-04-22T17:35:01.980 INFO:tasks.rados.rados.0.smithi115.stdout:2728: seq_num 823 ranges {0=589824,589824=573440,1163264=630784} 2022-04-22T17:35:01.984 INFO:tasks.rados.rados.0.smithi115.stdout:2728: writing smithi11541303-21 from 0 to 589824 tid 1 2022-04-22T17:35:01.989 INFO:tasks.rados.rados.0.smithi115.stdout:2728: writing smithi11541303-21 from 589824 to 1163264 tid 2 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2728: writing smithi11541303-21 from 1163264 to 1794048 tid 3 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2729: rmattr oid 39 current snap is 274 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2726: finishing write tid 2 to smithi11541303-43 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2726: finishing write tid 3 to smithi11541303-43 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2726: finishing write tid 4 to smithi11541303-43 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:2726: finishing write tid 5 to smithi11541303-43 2022-04-22T17:35:01.995 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1437 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:01.996 INFO:tasks.rados.rados.0.smithi115.stdout:2726: left oid 43 (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:35:01.996 INFO:tasks.rados.rados.0.smithi115.stdout:2728: finishing write tid 1 to smithi11541303-21 2022-04-22T17:35:01.996 INFO:tasks.rados.rados.0.smithi115.stdout:2726: done (2 left) 2022-04-22T17:35:01.996 INFO:tasks.rados.rados.0.smithi115.stdout:2730: snap_remove snap 268 2022-04-22T17:35:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:01 smithi121 bash[36902]: cluster 2022-04-22T17:35:00.953356+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2022-04-22T17:35:02.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:01 smithi115 bash[45495]: cluster 2022-04-22T17:35:00.953356+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2022-04-22T17:35:02.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:01 smithi115 bash[46934]: cluster 2022-04-22T17:35:00.953356+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2022-04-22T17:35:02.999 INFO:tasks.rados.rados.0.smithi115.stdout:2730: done (2 left) 2022-04-22T17:35:03.000 INFO:tasks.rados.rados.0.smithi115.stdout:2731: snap_remove snap 266 2022-04-22T17:35:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:02 smithi121 bash[36902]: cluster 2022-04-22T17:35:01.297848+0000 mgr.y (mgr.34107) 334 : cluster [DBG] pgmap v594: 121 pgs: 121 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:35:03.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:02 smithi121 bash[36902]: cluster 2022-04-22T17:35:01.963747+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2022-04-22T17:35:03.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:02 smithi121 bash[36902]: audit 2022-04-22T17:35:02.954238+0000 mon.a (mon.0) 1110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:03.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[46934]: cluster 2022-04-22T17:35:01.297848+0000 mgr.y (mgr.34107) 334 : cluster [DBG] pgmap v594: 121 pgs: 121 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:35:03.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[46934]: cluster 2022-04-22T17:35:01.963747+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2022-04-22T17:35:03.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[46934]: audit 2022-04-22T17:35:02.954238+0000 mon.a (mon.0) 1110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:03.270 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[45495]: cluster 2022-04-22T17:35:01.297848+0000 mgr.y (mgr.34107) 334 : cluster [DBG] pgmap v594: 121 pgs: 121 active+clean; 325 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:35:03.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[45495]: cluster 2022-04-22T17:35:01.963747+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2022-04-22T17:35:03.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:02 smithi115 bash[45495]: audit 2022-04-22T17:35:02.954238+0000 mon.a (mon.0) 1110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:03.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:03.526Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:03.526Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:04.000 INFO:tasks.rados.rados.0.smithi115.stdout:2731: done (2 left) 2022-04-22T17:35:04.000 INFO:tasks.rados.rados.0.smithi115.stdout:2732: rmattr oid 29 current snap is 274 2022-04-22T17:35:04.000 INFO:tasks.rados.rados.0.smithi115.stdout:2733: read oid 27 snap -1 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2733: expect deleted 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2734: read oid 18 snap -1 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2734: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2735: read oid 1 snap -1 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2735: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:35:04.001 INFO:tasks.rados.rados.0.smithi115.stdout:2736: snap_remove snap 258 2022-04-22T17:35:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:03 smithi121 bash[36902]: cluster 2022-04-22T17:35:02.981329+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2022-04-22T17:35:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:03 smithi121 bash[36902]: audit 2022-04-22T17:35:03.526438+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:04.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[45495]: cluster 2022-04-22T17:35:02.981329+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2022-04-22T17:35:04.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[45495]: audit 2022-04-22T17:35:03.526438+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:04.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[46934]: cluster 2022-04-22T17:35:02.981329+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2022-04-22T17:35:04.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:03 smithi115 bash[46934]: audit 2022-04-22T17:35:03.526438+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:04.915 INFO:tasks.rados.rados.0.smithi115.stdout:2736: done (6 left) 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2737: rmattr oid 31 current snap is 274 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2737: done (6 left) 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2738: rmattr oid 24 current snap is 274 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2738: done (6 left) 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2739: read oid 12 snap -1 2022-04-22T17:35:04.922 INFO:tasks.rados.rados.0.smithi115.stdout:2739: expect (ObjNum 818 snap 271 seq_num 818) 2022-04-22T17:35:04.923 INFO:tasks.rados.rados.0.smithi115.stdout:2740: rmattr oid 41 current snap is 274 2022-04-22T17:35:04.923 INFO:tasks.rados.rados.0.smithi115.stdout:2741: snap_remove snap 262 2022-04-22T17:35:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:04 smithi121 bash[36902]: cluster 2022-04-22T17:35:03.298558+0000 mgr.y (mgr.34107) 335 : cluster [DBG] pgmap v597: 121 pgs: 121 active+clean; 339 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 10 op/s 2022-04-22T17:35:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:04 smithi121 bash[36902]: cluster 2022-04-22T17:35:03.983675+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2022-04-22T17:35:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:04 smithi121 bash[36902]: audit 2022-04-22T17:35:04.342128+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:04 smithi121 bash[36902]: cluster 2022-04-22T17:35:04.899844+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2022-04-22T17:35:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[45495]: cluster 2022-04-22T17:35:03.298558+0000 mgr.y (mgr.34107) 335 : cluster [DBG] pgmap v597: 121 pgs: 121 active+clean; 339 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 10 op/s 2022-04-22T17:35:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[45495]: cluster 2022-04-22T17:35:03.983675+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2022-04-22T17:35:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[45495]: audit 2022-04-22T17:35:04.342128+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[45495]: cluster 2022-04-22T17:35:04.899844+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2022-04-22T17:35:05.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[46934]: cluster 2022-04-22T17:35:03.298558+0000 mgr.y (mgr.34107) 335 : cluster [DBG] pgmap v597: 121 pgs: 121 active+clean; 339 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 10 op/s 2022-04-22T17:35:05.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[46934]: cluster 2022-04-22T17:35:03.983675+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2022-04-22T17:35:05.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[46934]: audit 2022-04-22T17:35:04.342128+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:05.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:04 smithi115 bash[46934]: cluster 2022-04-22T17:35:04.899844+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2022-04-22T17:35:05.927 INFO:tasks.rados.rados.0.smithi115.stdout:2741: done (8 left) 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2742: snap_create 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2728: finishing write tid 2 to smithi11541303-21 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2728: finishing write tid 3 to smithi11541303-21 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2728: finishing write tid 4 to smithi11541303-21 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2728: finishing write tid 5 to smithi11541303-21 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1143 (ObjNum 823 snap 274 seq_num 823) dirty exists 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:2728: left oid 21 (ObjNum 823 snap 274 seq_num 823) 2022-04-22T17:35:05.928 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1819 (ObjNum 820 snap 274 seq_num 820) dirty exists 2022-04-22T17:35:05.929 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1244 (ObjNum -1115394288 snap 22075 seq_num 3179102304) dirty exists 2022-04-22T17:35:05.929 INFO:tasks.rados.rados.0.smithi115.stdout:2735: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:35:05.981 INFO:tasks.rados.rados.0.smithi115.stdout:2734: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:35:06.862 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1236 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:35:06.863 INFO:tasks.rados.rados.0.smithi115.stdout:2739: expect (ObjNum 818 snap 271 seq_num 818) 2022-04-22T17:35:06.930 INFO:tasks.rados.rados.0.smithi115.stdout:2728: done (8 left) 2022-04-22T17:35:06.930 INFO:tasks.rados.rados.0.smithi115.stdout:2729: done (7 left) 2022-04-22T17:35:06.930 INFO:tasks.rados.rados.0.smithi115.stdout:2732: done (6 left) 2022-04-22T17:35:06.930 INFO:tasks.rados.rados.0.smithi115.stdout:2733: done (5 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2734: done (4 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2735: done (3 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2739: done (2 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2740: done (1 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2742: done (0 left) 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 275 2022-04-22T17:35:06.931 INFO:tasks.rados.rados.0.smithi115.stdout:2743: seq_num 824 ranges {10182656=606208,10788864=458752,11247616=573440,11821056=172032} 2022-04-22T17:35:06.993 INFO:tasks.rados.rados.0.smithi115.stdout:2743: writing smithi11541303-20 from 10182656 to 10788864 tid 1 2022-04-22T17:35:06.996 INFO:tasks.rados.rados.0.smithi115.stdout:2743: writing smithi11541303-20 from 10788864 to 11247616 tid 2 2022-04-22T17:35:06.999 INFO:tasks.rados.rados.0.smithi115.stdout:2743: writing smithi11541303-20 from 11247616 to 11821056 tid 3 2022-04-22T17:35:07.000 INFO:tasks.rados.rados.0.smithi115.stdout:2743: writing smithi11541303-20 from 11821056 to 11993088 tid 4 2022-04-22T17:35:07.000 INFO:tasks.rados.rados.0.smithi115.stdout:2744: read oid 9 snap 263 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:2744: expect deleted 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:2745: read oid 30 snap -1 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:2745: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:2746: rollback oid 50 current snap is 275 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 273 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 275 2022-04-22T17:35:07.001 INFO:tasks.rados.rados.0.smithi115.stdout:2747: seq_num 825 ranges {2064384=622592,2686976=745472,3432448=458752} 2022-04-22T17:35:07.016 INFO:tasks.rados.rados.0.smithi115.stdout:2747: writing smithi11541303-24 from 2064384 to 2686976 tid 1 2022-04-22T17:35:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:2747: writing smithi11541303-24 from 2686976 to 3432448 tid 2 2022-04-22T17:35:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:2747: writing smithi11541303-24 from 3432448 to 3891200 tid 3 2022-04-22T17:35:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:2748: snap_remove snap 251 2022-04-22T17:35:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:06 smithi121 bash[36902]: cluster 2022-04-22T17:35:05.299519+0000 mgr.y (mgr.34107) 336 : cluster [DBG] pgmap v600: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2022-04-22T17:35:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:06 smithi121 bash[36902]: cluster 2022-04-22T17:35:05.914317+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2022-04-22T17:35:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:06 smithi115 bash[46934]: cluster 2022-04-22T17:35:05.299519+0000 mgr.y (mgr.34107) 336 : cluster [DBG] pgmap v600: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2022-04-22T17:35:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:06 smithi115 bash[46934]: cluster 2022-04-22T17:35:05.914317+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2022-04-22T17:35:07.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:06 smithi115 bash[45495]: cluster 2022-04-22T17:35:05.299519+0000 mgr.y (mgr.34107) 336 : cluster [DBG] pgmap v600: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2022-04-22T17:35:07.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:06 smithi115 bash[45495]: cluster 2022-04-22T17:35:05.914317+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2022-04-22T17:35:07.952 INFO:tasks.rados.rados.0.smithi115.stdout:2748: done (5 left) 2022-04-22T17:35:07.952 INFO:tasks.rados.rados.0.smithi115.stdout:2749: rollback oid 18 current snap is 275 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 254 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2750: delete oid 38 current snap is 275 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2746: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2746: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2746: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1644 (ObjNum 543 snap 178 seq_num 543) dirty exists 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 1 to smithi11541303-20 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 2 to smithi11541303-20 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 3 to smithi11541303-20 2022-04-22T17:35:07.953 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 4 to smithi11541303-20 2022-04-22T17:35:07.954 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 5 to smithi11541303-20 2022-04-22T17:35:07.954 INFO:tasks.rados.rados.0.smithi115.stdout:2745: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:35:08.025 INFO:tasks.rados.rados.0.smithi115.stdout:2743: finishing write tid 6 to smithi11541303-20 2022-04-22T17:35:08.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1467 (ObjNum 824 snap 275 seq_num 824) dirty exists 2022-04-22T17:35:08.025 INFO:tasks.rados.rados.0.smithi115.stdout:2743: left oid 20 (ObjNum 824 snap 275 seq_num 824) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2747: finishing write tid 1 to smithi11541303-24 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2747: finishing write tid 2 to smithi11541303-24 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2743: done (6 left) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2744: done (5 left) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2745: done (4 left) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2746: done (3 left) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2750: done (2 left) 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 275 2022-04-22T17:35:08.026 INFO:tasks.rados.rados.0.smithi115.stdout:2751: seq_num 827 ranges {5177344=655360,5832704=516096,6348800=688128} 2022-04-22T17:35:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:2751: writing smithi11541303-7 from 5177344 to 5832704 tid 1 2022-04-22T17:35:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:2751: writing smithi11541303-7 from 5832704 to 6348800 tid 2 2022-04-22T17:35:08.065 INFO:tasks.rados.rados.0.smithi115.stdout:2751: writing smithi11541303-7 from 6348800 to 7036928 tid 3 2022-04-22T17:35:08.066 INFO:tasks.rados.rados.0.smithi115.stdout:2752: snap_remove snap 245 2022-04-22T17:35:08.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:07 smithi115 bash[46934]: cluster 2022-04-22T17:35:06.914463+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2022-04-22T17:35:08.259 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:07 smithi115 bash[45495]: cluster 2022-04-22T17:35:06.914463+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2022-04-22T17:35:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:07 smithi121 bash[36902]: cluster 2022-04-22T17:35:06.914463+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2022-04-22T17:35:08.961 INFO:tasks.rados.rados.0.smithi115.stdout:2752: done (3 left) 2022-04-22T17:35:08.961 INFO:tasks.rados.rados.0.smithi115.stdout:2753: read oid 27 snap -1 2022-04-22T17:35:08.961 INFO:tasks.rados.rados.0.smithi115.stdout:2753: expect deleted 2022-04-22T17:35:08.961 INFO:tasks.rados.rados.0.smithi115.stdout:2754: rollback oid 29 current snap is 275 2022-04-22T17:35:08.961 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 263 2022-04-22T17:35:08.962 INFO:tasks.rados.rados.0.smithi115.stdout:2755: setattr oid 11 current snap is 275 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2747: finishing write tid 3 to smithi11541303-24 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2747: finishing write tid 4 to smithi11541303-24 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2747: finishing write tid 5 to smithi11541303-24 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1250 (ObjNum 825 snap 275 seq_num 825) dirty exists 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2747: left oid 24 (ObjNum 825 snap 275 seq_num 825) 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2749: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2749: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:2749: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:35:08.964 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1857 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: finishing write tid 1 to smithi11541303-7 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: finishing write tid 2 to smithi11541303-7 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: finishing write tid 3 to smithi11541303-7 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: finishing write tid 4 to smithi11541303-7 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: finishing write tid 5 to smithi11541303-7 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1240 (ObjNum 827 snap 275 seq_num 827) dirty exists 2022-04-22T17:35:08.965 INFO:tasks.rados.rados.0.smithi115.stdout:2751: left oid 7 (ObjNum 827 snap 275 seq_num 827) 2022-04-22T17:35:08.966 INFO:tasks.rados.rados.0.smithi115.stdout:2747: done (5 left) 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2749: done (4 left) 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2751: done (3 left) 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2753: done (2 left) 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2756: copy_from oid 46 from oid 36 current snap is 275 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2757: rmattr oid 45 current snap is 275 2022-04-22T17:35:08.967 INFO:tasks.rados.rados.0.smithi115.stdout:2757: done (3 left) 2022-04-22T17:35:08.968 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 275 2022-04-22T17:35:08.968 INFO:tasks.rados.rados.0.smithi115.stdout:2758: seq_num 828 ranges {12525568=671744,13197312=802816,14000128=401408} 2022-04-22T17:35:09.043 INFO:tasks.rados.rados.0.smithi115.stdout:2758: writing smithi11541303-39 from 12525568 to 13197312 tid 1 2022-04-22T17:35:09.047 INFO:tasks.rados.rados.0.smithi115.stdout:2758: writing smithi11541303-39 from 13197312 to 14000128 tid 2 2022-04-22T17:35:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2758: writing smithi11541303-39 from 14000128 to 14401536 tid 3 2022-04-22T17:35:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2759: delete oid 22 current snap is 275 2022-04-22T17:35:09.050 INFO:tasks.rados.rados.0.smithi115.stdout:2756: finishing copy_from to smithi11541303-46 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:2756: got expected ENOENT (src dne) 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:2754: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:2754: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:2756: finishing copy_from racing read to smithi11541303-46 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1233 (ObjNum 749 snap 257 seq_num 749) dirty exists 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:2754: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:35:09.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1247 (ObjNum 1583766875 snap 1852863574 seq_num 1617390934) dirty exists 2022-04-22T17:35:09.054 INFO:tasks.rados.rados.0.smithi115.stdout:2754: done (4 left) 2022-04-22T17:35:09.054 INFO:tasks.rados.rados.0.smithi115.stdout:2755: done (3 left) 2022-04-22T17:35:09.054 INFO:tasks.rados.rados.0.smithi115.stdout:2756: done (2 left) 2022-04-22T17:35:09.054 INFO:tasks.rados.rados.0.smithi115.stdout:2759: done (1 left) 2022-04-22T17:35:09.054 INFO:tasks.rados.rados.0.smithi115.stdout:2760: snap_remove snap 271 2022-04-22T17:35:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:08 smithi121 bash[36902]: cluster 2022-04-22T17:35:07.300384+0000 mgr.y (mgr.34107) 337 : cluster [DBG] pgmap v603: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 335 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 12 op/s 2022-04-22T17:35:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:08 smithi121 bash[36902]: cluster 2022-04-22T17:35:07.938279+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2022-04-22T17:35:09.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:08 smithi115 bash[45495]: cluster 2022-04-22T17:35:07.300384+0000 mgr.y (mgr.34107) 337 : cluster [DBG] pgmap v603: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 335 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 12 op/s 2022-04-22T17:35:09.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:08 smithi115 bash[45495]: cluster 2022-04-22T17:35:07.938279+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2022-04-22T17:35:09.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:08 smithi115 bash[46934]: cluster 2022-04-22T17:35:07.300384+0000 mgr.y (mgr.34107) 337 : cluster [DBG] pgmap v603: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 335 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 12 op/s 2022-04-22T17:35:09.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:08 smithi115 bash[46934]: cluster 2022-04-22T17:35:07.938279+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2022-04-22T17:35:09.922 INFO:tasks.rados.rados.0.smithi115.stdout:2758: finishing write tid 1 to smithi11541303-39 2022-04-22T17:35:09.922 INFO:tasks.rados.rados.0.smithi115.stdout:2758: finishing write tid 2 to smithi11541303-39 2022-04-22T17:35:09.922 INFO:tasks.rados.rados.0.smithi115.stdout:2758: finishing write tid 3 to smithi11541303-39 2022-04-22T17:35:09.922 INFO:tasks.rados.rados.0.smithi115.stdout:2758: finishing write tid 4 to smithi11541303-39 2022-04-22T17:35:09.922 INFO:tasks.rados.rados.0.smithi115.stdout:2758: finishing write tid 5 to smithi11541303-39 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1823 (ObjNum 828 snap 275 seq_num 828) dirty exists 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2758: left oid 39 (ObjNum 828 snap 275 seq_num 828) 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2758: done (1 left) 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2760: done (0 left) 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2761: read oid 29 snap -1 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2761: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:09.923 INFO:tasks.rados.rados.0.smithi115.stdout:2762: delete oid 47 current snap is 275 2022-04-22T17:35:09.924 INFO:tasks.rados.rados.0.smithi115.stdout:2761: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:09.935 INFO:tasks.rados.rados.0.smithi115.stdout:2761: done (1 left) 2022-04-22T17:35:09.935 INFO:tasks.rados.rados.0.smithi115.stdout:2762: done (0 left) 2022-04-22T17:35:09.936 INFO:tasks.rados.rados.0.smithi115.stdout:2763: setattr oid 41 current snap is 275 2022-04-22T17:35:09.937 INFO:tasks.rados.rados.0.smithi115.stdout:2764: read oid 6 snap 257 2022-04-22T17:35:09.937 INFO:tasks.rados.rados.0.smithi115.stdout:2764: expect deleted 2022-04-22T17:35:09.937 INFO:tasks.rados.rados.0.smithi115.stdout:2765: copy_from oid 39 from oid 14 current snap is 275 2022-04-22T17:35:09.937 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 275 2022-04-22T17:35:09.937 INFO:tasks.rados.rados.0.smithi115.stdout:2766: seq_num 831 ranges {0=720896,720896=499712,1220608=704512} 2022-04-22T17:35:09.941 INFO:tasks.rados.rados.0.smithi115.stdout:2766: writing smithi11541303-13 from 0 to 720896 tid 1 2022-04-22T17:35:09.944 INFO:tasks.rados.rados.0.smithi115.stdout:2766: writing smithi11541303-13 from 720896 to 1220608 tid 2 2022-04-22T17:35:09.948 INFO:tasks.rados.rados.0.smithi115.stdout:2766: writing smithi11541303-13 from 1220608 to 1925120 tid 3 2022-04-22T17:35:09.948 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 275 2022-04-22T17:35:09.949 INFO:tasks.rados.rados.0.smithi115.stdout:2767: seq_num 832 ranges {11436032=737280,12173312=786432,12959744=417792} 2022-04-22T17:35:10.019 INFO:tasks.rados.rados.0.smithi115.stdout:2767: writing smithi11541303-10 from 11436032 to 12173312 tid 1 2022-04-22T17:35:10.025 INFO:tasks.rados.rados.0.smithi115.stdout:2767: writing smithi11541303-10 from 12173312 to 12959744 tid 2 2022-04-22T17:35:10.028 INFO:tasks.rados.rados.0.smithi115.stdout:2767: writing smithi11541303-10 from 12959744 to 13377536 tid 3 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2768: setattr oid 3 current snap is 275 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1253 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: finishing write tid 1 to smithi11541303-13 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: finishing write tid 2 to smithi11541303-13 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: finishing write tid 3 to smithi11541303-13 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: finishing write tid 4 to smithi11541303-13 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: finishing write tid 5 to smithi11541303-13 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1492 (ObjNum 831 snap 275 seq_num 831) dirty exists 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2766: left oid 13 (ObjNum 831 snap 275 seq_num 831) 2022-04-22T17:35:10.029 INFO:tasks.rados.rados.0.smithi115.stdout:2765: finishing copy_from to smithi11541303-39 2022-04-22T17:35:10.030 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1240 (ObjNum 691 snap 230 seq_num 691) dirty exists 2022-04-22T17:35:10.030 INFO:tasks.rados.rados.0.smithi115.stdout:2765: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:35:10.030 INFO:tasks.rados.rados.0.smithi115.stdout:2767: finishing write tid 1 to smithi11541303-10 2022-04-22T17:35:10.030 INFO:tasks.rados.rados.0.smithi115.stdout:2763: done (5 left) 2022-04-22T17:35:10.030 INFO:tasks.rados.rados.0.smithi115.stdout:2764: done (4 left) 2022-04-22T17:35:10.031 INFO:tasks.rados.rados.0.smithi115.stdout:2765: done (3 left) 2022-04-22T17:35:10.031 INFO:tasks.rados.rados.0.smithi115.stdout:2766: done (2 left) 2022-04-22T17:35:10.031 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 275 2022-04-22T17:35:10.031 INFO:tasks.rados.rados.0.smithi115.stdout:2769: seq_num 833 ranges {0=761856,761856=671744,1433600=532480} 2022-04-22T17:35:10.034 INFO:tasks.rados.rados.0.smithi115.stdout:2769: writing smithi11541303-29 from 0 to 761856 tid 1 2022-04-22T17:35:10.038 INFO:tasks.rados.rados.0.smithi115.stdout:2769: writing smithi11541303-29 from 761856 to 1433600 tid 2 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2769: writing smithi11541303-29 from 1433600 to 1966080 tid 3 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2770: read oid 16 snap -1 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2770: expect (ObjNum 800 snap 268 seq_num 800) 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2771: delete oid 6 current snap is 275 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1234 (ObjNum 779 snap 267 seq_num 779) dirty exists 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2767: finishing write tid 2 to smithi11541303-10 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2767: finishing write tid 3 to smithi11541303-10 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2767: finishing write tid 4 to smithi11541303-10 2022-04-22T17:35:10.042 INFO:tasks.rados.rados.0.smithi115.stdout:2767: finishing write tid 5 to smithi11541303-10 2022-04-22T17:35:10.043 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1258 (ObjNum 832 snap 275 seq_num 832) dirty exists 2022-04-22T17:35:10.043 INFO:tasks.rados.rados.0.smithi115.stdout:2767: left oid 10 (ObjNum 832 snap 275 seq_num 832) 2022-04-22T17:35:10.045 INFO:tasks.rados.rados.0.smithi115.stdout:2769: finishing write tid 1 to smithi11541303-29 2022-04-22T17:35:10.045 INFO:tasks.rados.rados.0.smithi115.stdout:2767: done (4 left) 2022-04-22T17:35:10.045 INFO:tasks.rados.rados.0.smithi115.stdout:2768: done (3 left) 2022-04-22T17:35:10.045 INFO:tasks.rados.rados.0.smithi115.stdout:2771: done (2 left) 2022-04-22T17:35:10.045 INFO:tasks.rados.rados.0.smithi115.stdout:2772: read oid 39 snap -1 2022-04-22T17:35:10.046 INFO:tasks.rados.rados.0.smithi115.stdout:2772: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:35:10.046 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 275 2022-04-22T17:35:10.046 INFO:tasks.rados.rados.0.smithi115.stdout:2773: seq_num 835 ranges {1761280=794624,2555904=442368,2998272=712704,3710976=49152} 2022-04-22T17:35:10.060 INFO:tasks.rados.rados.0.smithi115.stdout:2773: writing smithi11541303-32 from 1761280 to 2555904 tid 1 2022-04-22T17:35:10.062 INFO:tasks.rados.rados.0.smithi115.stdout:2773: writing smithi11541303-32 from 2555904 to 2998272 tid 2 2022-04-22T17:35:10.066 INFO:tasks.rados.rados.0.smithi115.stdout:2773: writing smithi11541303-32 from 2998272 to 3710976 tid 3 2022-04-22T17:35:10.067 INFO:tasks.rados.rados.0.smithi115.stdout:2773: writing smithi11541303-32 from 3710976 to 3760128 tid 4 2022-04-22T17:35:10.067 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 275 2022-04-22T17:35:10.067 INFO:tasks.rados.rados.0.smithi115.stdout:2769: finishing write tid 2 to smithi11541303-29 2022-04-22T17:35:10.068 INFO:tasks.rados.rados.0.smithi115.stdout:2774: seq_num 836 ranges {0=409600,409600=720896,1130496=516096,1646592=368640} 2022-04-22T17:35:10.069 INFO:tasks.rados.rados.0.smithi115.stdout:2774: writing smithi11541303-6 from 0 to 409600 tid 1 2022-04-22T17:35:10.074 INFO:tasks.rados.rados.0.smithi115.stdout:2774: writing smithi11541303-6 from 409600 to 1130496 tid 2 2022-04-22T17:35:10.077 INFO:tasks.rados.rados.0.smithi115.stdout:2774: writing smithi11541303-6 from 1130496 to 1646592 tid 3 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:2774: writing smithi11541303-6 from 1646592 to 2015232 tid 4 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:2775: rollback oid 7 current snap is 275 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 7 to 259 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:2769: finishing write tid 3 to smithi11541303-29 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:2769: finishing write tid 4 to smithi11541303-29 2022-04-22T17:35:10.080 INFO:tasks.rados.rados.0.smithi115.stdout:2769: finishing write tid 5 to smithi11541303-29 2022-04-22T17:35:10.081 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1251 (ObjNum 833 snap 275 seq_num 833) dirty exists 2022-04-22T17:35:10.081 INFO:tasks.rados.rados.0.smithi115.stdout:2769: left oid 29 (ObjNum 833 snap 275 seq_num 833) 2022-04-22T17:35:10.081 INFO:tasks.rados.rados.0.smithi115.stdout:2770: expect (ObjNum 800 snap 268 seq_num 800) 2022-04-22T17:35:10.175 INFO:tasks.rados.rados.0.smithi115.stdout:2772: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:35:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:09 smithi121 bash[36902]: cluster 2022-04-22T17:35:08.940509+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2022-04-22T17:35:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:09 smithi121 bash[36902]: audit 2022-04-22T17:35:09.345182+0000 mon.a (mon.0) 1120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:09 smithi121 bash[36902]: cluster 2022-04-22T17:35:09.902440+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2022-04-22T17:35:10.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[45495]: cluster 2022-04-22T17:35:08.940509+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2022-04-22T17:35:10.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[45495]: audit 2022-04-22T17:35:09.345182+0000 mon.a (mon.0) 1120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:10.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[45495]: cluster 2022-04-22T17:35:09.902440+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2022-04-22T17:35:10.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[46934]: cluster 2022-04-22T17:35:08.940509+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2022-04-22T17:35:10.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[46934]: audit 2022-04-22T17:35:09.345182+0000 mon.a (mon.0) 1120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:10.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:09 smithi115 bash[46934]: cluster 2022-04-22T17:35:09.902440+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2022-04-22T17:35:10.377 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 1 to smithi11541303-32 2022-04-22T17:35:10.426 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 1 to smithi11541303-6 2022-04-22T17:35:10.426 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 2 to smithi11541303-32 2022-04-22T17:35:10.426 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 3 to smithi11541303-32 2022-04-22T17:35:10.426 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 4 to smithi11541303-32 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 5 to smithi11541303-32 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2773: finishing write tid 6 to smithi11541303-32 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1358 (ObjNum 835 snap 275 seq_num 835) dirty exists 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2773: left oid 32 (ObjNum 835 snap 275 seq_num 835) 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 2 to smithi11541303-6 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 3 to smithi11541303-6 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2769: done (5 left) 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2770: done (4 left) 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2772: done (3 left) 2022-04-22T17:35:10.427 INFO:tasks.rados.rados.0.smithi115.stdout:2773: done (2 left) 2022-04-22T17:35:10.428 INFO:tasks.rados.rados.0.smithi115.stdout:2776: rmattr oid 49 current snap is 275 2022-04-22T17:35:10.456 INFO:tasks.rados.rados.0.smithi115.stdout:2776: done (2 left) 2022-04-22T17:35:10.456 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 275 2022-04-22T17:35:10.456 INFO:tasks.rados.rados.0.smithi115.stdout:2775: finishing rollback tid 0 to smithi11541303-7 2022-04-22T17:35:10.456 INFO:tasks.rados.rados.0.smithi115.stdout:2775: finishing rollback tid 1 to smithi11541303-7 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2775: finishing rollback tid 2 to smithi11541303-7 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1243 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 4 to smithi11541303-6 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 5 to smithi11541303-6 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2774: finishing write tid 6 to smithi11541303-6 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1255 (ObjNum 836 snap 275 seq_num 836) dirty exists 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2774: left oid 6 (ObjNum 836 snap 275 seq_num 836) 2022-04-22T17:35:10.457 INFO:tasks.rados.rados.0.smithi115.stdout:2777: seq_num 837 ranges {11993088=425984,12419072=655360,13074432=729088,13803520=221184} 2022-04-22T17:35:10.458 INFO:tasks.rados.rados.0.smithi115.stdout:2777: writing smithi11541303-20 from 11993088 to 12419072 tid 1 2022-04-22T17:35:10.458 INFO:tasks.rados.rados.0.smithi115.stdout:2777: writing smithi11541303-20 from 12419072 to 13074432 tid 2 2022-04-22T17:35:10.460 INFO:tasks.rados.rados.0.smithi115.stdout:2777: writing smithi11541303-20 from 13074432 to 13803520 tid 3 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2777: writing smithi11541303-20 from 13803520 to 14024704 tid 4 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2774: done (2 left) 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2775: done (1 left) 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2778: copy_from oid 48 from oid 37 current snap is 275 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2779: read oid 39 snap -1 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2779: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:35:10.462 INFO:tasks.rados.rados.0.smithi115.stdout:2780: setattr oid 38 current snap is 275 2022-04-22T17:35:10.463 INFO:tasks.rados.rados.0.smithi115.stdout:2781: rmattr oid 10 current snap is 275 2022-04-22T17:35:10.463 INFO:tasks.rados.rados.0.smithi115.stdout:2781: done (4 left) 2022-04-22T17:35:10.464 INFO:tasks.rados.rados.0.smithi115.stdout:2782: read oid 23 snap 272 2022-04-22T17:35:10.464 INFO:tasks.rados.rados.0.smithi115.stdout:2782: expect deleted 2022-04-22T17:35:10.464 INFO:tasks.rados.rados.0.smithi115.stdout:2783: snap_create 2022-04-22T17:35:10.468 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 1 to smithi11541303-20 2022-04-22T17:35:10.469 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1738 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:35:10.469 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 2 to smithi11541303-20 2022-04-22T17:35:10.469 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 3 to smithi11541303-20 2022-04-22T17:35:10.470 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 4 to smithi11541303-20 2022-04-22T17:35:10.471 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 5 to smithi11541303-20 2022-04-22T17:35:10.473 INFO:tasks.rados.rados.0.smithi115.stdout:2777: finishing write tid 6 to smithi11541303-20 2022-04-22T17:35:10.473 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1497 (ObjNum 837 snap 275 seq_num 837) dirty exists 2022-04-22T17:35:10.473 INFO:tasks.rados.rados.0.smithi115.stdout:2777: left oid 20 (ObjNum 837 snap 275 seq_num 837) 2022-04-22T17:35:10.473 INFO:tasks.rados.rados.0.smithi115.stdout:2778: finishing copy_from to smithi11541303-48 2022-04-22T17:35:10.473 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1646 (ObjNum 1785490015 snap 1902146159 seq_num 1315728995) dirty exists 2022-04-22T17:35:10.474 INFO:tasks.rados.rados.0.smithi115.stdout:2778: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:35:10.480 INFO:tasks.rados.rados.0.smithi115.stdout:2779: expect (ObjNum 691 snap 230 seq_num 691) 2022-04-22T17:35:10.973 INFO:tasks.rados.rados.0.smithi115.stdout:2777: done (5 left) 2022-04-22T17:35:10.973 INFO:tasks.rados.rados.0.smithi115.stdout:2778: done (4 left) 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:2779: done (3 left) 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:2780: done (2 left) 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:2782: done (1 left) 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:2783: done (0 left) 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 276 2022-04-22T17:35:10.974 INFO:tasks.rados.rados.0.smithi115.stdout:2784: seq_num 838 ranges {5332992=442368,5775360=540672,6316032=532480,6848512=442368,7290880=90112} 2022-04-22T17:35:11.008 INFO:tasks.rados.rados.0.smithi115.stdout:2784: writing smithi11541303-15 from 5332992 to 5775360 tid 1 2022-04-22T17:35:11.011 INFO:tasks.rados.rados.0.smithi115.stdout:2784: writing smithi11541303-15 from 5775360 to 6316032 tid 2 2022-04-22T17:35:11.014 INFO:tasks.rados.rados.0.smithi115.stdout:2784: writing smithi11541303-15 from 6316032 to 6848512 tid 3 2022-04-22T17:35:11.017 INFO:tasks.rados.rados.0.smithi115.stdout:2784: writing smithi11541303-15 from 6848512 to 7290880 tid 4 2022-04-22T17:35:11.018 INFO:tasks.rados.rados.0.smithi115.stdout:2784: writing smithi11541303-15 from 7290880 to 7380992 tid 5 2022-04-22T17:35:11.018 INFO:tasks.rados.rados.0.smithi115.stdout:2785: delete oid 23 current snap is 276 2022-04-22T17:35:11.018 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 1 to smithi11541303-15 2022-04-22T17:35:11.019 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 2 to smithi11541303-15 2022-04-22T17:35:11.020 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 3 to smithi11541303-15 2022-04-22T17:35:11.022 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 4 to smithi11541303-15 2022-04-22T17:35:11.023 INFO:tasks.rados.rados.0.smithi115.stdout:2785: done (1 left) 2022-04-22T17:35:11.023 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 276 2022-04-22T17:35:11.023 INFO:tasks.rados.rados.0.smithi115.stdout:2786: seq_num 840 ranges {1982464=475136,2457600=712704,3170304=499712,3670016=393216} 2022-04-22T17:35:11.038 INFO:tasks.rados.rados.0.smithi115.stdout:2786: writing smithi11541303-28 from 1982464 to 2457600 tid 1 2022-04-22T17:35:11.042 INFO:tasks.rados.rados.0.smithi115.stdout:2786: writing smithi11541303-28 from 2457600 to 3170304 tid 2 2022-04-22T17:35:11.046 INFO:tasks.rados.rados.0.smithi115.stdout:2786: writing smithi11541303-28 from 3170304 to 3670016 tid 3 2022-04-22T17:35:11.049 INFO:tasks.rados.rados.0.smithi115.stdout:2786: writing smithi11541303-28 from 3670016 to 4063232 tid 4 2022-04-22T17:35:11.049 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 276 2022-04-22T17:35:11.049 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 5 to smithi11541303-15 2022-04-22T17:35:11.049 INFO:tasks.rados.rados.0.smithi115.stdout:2787: seq_num 841 ranges {0=491520,491520=598016,1089536=704512,1794048=303104} 2022-04-22T17:35:11.051 INFO:tasks.rados.rados.0.smithi115.stdout:2787: writing smithi11541303-37 from 0 to 491520 tid 1 2022-04-22T17:35:11.055 INFO:tasks.rados.rados.0.smithi115.stdout:2787: writing smithi11541303-37 from 491520 to 1089536 tid 2 2022-04-22T17:35:11.059 INFO:tasks.rados.rados.0.smithi115.stdout:2787: writing smithi11541303-37 from 1089536 to 1794048 tid 3 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2787: writing smithi11541303-37 from 1794048 to 2097152 tid 4 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2788: read oid 42 snap -1 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2788: expect (ObjNum 815 snap 270 seq_num 815) 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2789: delete oid 29 current snap is 276 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 6 to smithi11541303-15 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:2784: finishing write tid 7 to smithi11541303-15 2022-04-22T17:35:11.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1547 (ObjNum 838 snap 276 seq_num 838) dirty exists 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2784: left oid 15 (ObjNum 838 snap 276 seq_num 838) 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 1 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 2 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 3 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 4 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 5 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: finishing write tid 6 to smithi11541303-28 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1256 (ObjNum 840 snap 276 seq_num 840) dirty exists 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2786: left oid 28 (ObjNum 840 snap 276 seq_num 840) 2022-04-22T17:35:11.064 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 1 to smithi11541303-37 2022-04-22T17:35:11.065 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 2 to smithi11541303-37 2022-04-22T17:35:11.065 INFO:tasks.rados.rados.0.smithi115.stdout:2784: done (4 left) 2022-04-22T17:35:11.065 INFO:tasks.rados.rados.0.smithi115.stdout:2786: done (3 left) 2022-04-22T17:35:11.065 INFO:tasks.rados.rados.0.smithi115.stdout:2789: done (2 left) 2022-04-22T17:35:11.066 INFO:tasks.rados.rados.0.smithi115.stdout:2790: copy_from oid 27 from oid 23 current snap is 276 2022-04-22T17:35:11.066 INFO:tasks.rados.rados.0.smithi115.stdout:2791: copy_from oid 14 from oid 6 current snap is 276 2022-04-22T17:35:11.066 INFO:tasks.rados.rados.0.smithi115.stdout:2792: snap_create 2022-04-22T17:35:11.067 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 3 to smithi11541303-37 2022-04-22T17:35:11.068 INFO:tasks.rados.rados.0.smithi115.stdout:2790: finishing copy_from to smithi11541303-27 2022-04-22T17:35:11.068 INFO:tasks.rados.rados.0.smithi115.stdout:2790: got expected ENOENT (src dne) 2022-04-22T17:35:11.068 INFO:tasks.rados.rados.0.smithi115.stdout:2790: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:35:11.069 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 4 to smithi11541303-37 2022-04-22T17:35:11.069 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 5 to smithi11541303-37 2022-04-22T17:35:11.072 INFO:tasks.rados.rados.0.smithi115.stdout:2787: finishing write tid 6 to smithi11541303-37 2022-04-22T17:35:11.072 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1828 (ObjNum 841 snap 276 seq_num 841) dirty exists 2022-04-22T17:35:11.072 INFO:tasks.rados.rados.0.smithi115.stdout:2787: left oid 37 (ObjNum 841 snap 276 seq_num 841) 2022-04-22T17:35:11.077 INFO:tasks.rados.rados.0.smithi115.stdout:2788: expect (ObjNum 815 snap 270 seq_num 815) 2022-04-22T17:35:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:2791: finishing copy_from to smithi11541303-14 2022-04-22T17:35:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1255 (ObjNum 836 snap 275 seq_num 836) dirty exists 2022-04-22T17:35:11.162 INFO:tasks.rados.rados.0.smithi115.stdout:2791: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:35:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:10 smithi121 bash[36902]: cluster 2022-04-22T17:35:09.300956+0000 mgr.y (mgr.34107) 338 : cluster [DBG] pgmap v606: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 442 KiB/s wr, 10 op/s 2022-04-22T17:35:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:10 smithi115 bash[45495]: cluster 2022-04-22T17:35:09.300956+0000 mgr.y (mgr.34107) 338 : cluster [DBG] pgmap v606: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 442 KiB/s wr, 10 op/s 2022-04-22T17:35:11.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:10 smithi115 bash[46934]: cluster 2022-04-22T17:35:09.300956+0000 mgr.y (mgr.34107) 338 : cluster [DBG] pgmap v606: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 442 KiB/s wr, 10 op/s 2022-04-22T17:35:11.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:11.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:11.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:11.631Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:11.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:11.631Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2787: done (4 left) 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2788: done (3 left) 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2790: done (2 left) 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2791: done (1 left) 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2792: done (0 left) 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2793: copy_from oid 4 from oid 16 current snap is 277 2022-04-22T17:35:11.981 INFO:tasks.rados.rados.0.smithi115.stdout:2794: snap_create 2022-04-22T17:35:11.999 INFO:tasks.rados.rados.0.smithi115.stdout:2793: finishing copy_from to smithi11541303-4 2022-04-22T17:35:11.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1637 (ObjNum 800 snap 268 seq_num 800) dirty exists 2022-04-22T17:35:12.000 INFO:tasks.rados.rados.0.smithi115.stdout:2793: finishing copy_from racing read to smithi11541303-4 2022-04-22T17:35:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:11 smithi121 bash[36902]: cluster 2022-04-22T17:35:10.960384+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2022-04-22T17:35:12.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:11 smithi115 bash[45495]: cluster 2022-04-22T17:35:10.960384+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2022-04-22T17:35:12.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:11 smithi115 bash[46934]: cluster 2022-04-22T17:35:10.960384+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2022-04-22T17:35:12.991 INFO:tasks.rados.rados.0.smithi115.stdout:2793: done (1 left) 2022-04-22T17:35:12.991 INFO:tasks.rados.rados.0.smithi115.stdout:2794: done (0 left) 2022-04-22T17:35:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:2795: read oid 49 snap -1 2022-04-22T17:35:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:2795: expect deleted 2022-04-22T17:35:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 278 2022-04-22T17:35:12.992 INFO:tasks.rados.rados.0.smithi115.stdout:2796: seq_num 843 ranges {0=524288,524288=761856,1286144=671744,1957888=172032} 2022-04-22T17:35:12.994 INFO:tasks.rados.rados.0.smithi115.stdout:2796: writing smithi11541303-22 from 0 to 524288 tid 1 2022-04-22T17:35:12.998 INFO:tasks.rados.rados.0.smithi115.stdout:2796: writing smithi11541303-22 from 524288 to 1286144 tid 2 2022-04-22T17:35:13.002 INFO:tasks.rados.rados.0.smithi115.stdout:2796: writing smithi11541303-22 from 1286144 to 1957888 tid 3 2022-04-22T17:35:13.004 INFO:tasks.rados.rados.0.smithi115.stdout:2796: writing smithi11541303-22 from 1957888 to 2129920 tid 4 2022-04-22T17:35:13.004 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 278 2022-04-22T17:35:13.005 INFO:tasks.rados.rados.0.smithi115.stdout:2797: seq_num 844 ranges {0=540672,540672=696320,1236992=475136,1712128=434176} 2022-04-22T17:35:13.006 INFO:tasks.rados.rados.0.smithi115.stdout:2797: writing smithi11541303-26 from 0 to 540672 tid 1 2022-04-22T17:35:13.011 INFO:tasks.rados.rados.0.smithi115.stdout:2797: writing smithi11541303-26 from 540672 to 1236992 tid 2 2022-04-22T17:35:13.013 INFO:tasks.rados.rados.0.smithi115.stdout:2797: writing smithi11541303-26 from 1236992 to 1712128 tid 3 2022-04-22T17:35:13.016 INFO:tasks.rados.rados.0.smithi115.stdout:2797: writing smithi11541303-26 from 1712128 to 2146304 tid 4 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2798: copy_from oid 45 from oid 28 current snap is 278 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 1 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 2 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 3 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 4 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 5 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 1 to smithi11541303-26 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:2796: finishing write tid 6 to smithi11541303-22 2022-04-22T17:35:13.017 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1149 (ObjNum 843 snap 278 seq_num 843) dirty exists 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2796: left oid 22 (ObjNum 843 snap 278 seq_num 843) 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2795: done (3 left) 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2796: done (2 left) 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2799: snap_create 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 2 to smithi11541303-26 2022-04-22T17:35:13.018 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 3 to smithi11541303-26 2022-04-22T17:35:13.020 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 4 to smithi11541303-26 2022-04-22T17:35:13.021 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 5 to smithi11541303-26 2022-04-22T17:35:13.022 INFO:tasks.rados.rados.0.smithi115.stdout:2797: finishing write tid 6 to smithi11541303-26 2022-04-22T17:35:13.023 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1363 (ObjNum 844 snap 278 seq_num 844) dirty exists 2022-04-22T17:35:13.023 INFO:tasks.rados.rados.0.smithi115.stdout:2797: left oid 26 (ObjNum 844 snap 278 seq_num 844) 2022-04-22T17:35:13.040 INFO:tasks.rados.rados.0.smithi115.stdout:2798: finishing copy_from to smithi11541303-45 2022-04-22T17:35:13.040 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1256 (ObjNum 840 snap 276 seq_num 840) dirty exists 2022-04-22T17:35:13.041 INFO:tasks.rados.rados.0.smithi115.stdout:2798: finishing copy_from racing read to smithi11541303-45 2022-04-22T17:35:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: cluster 2022-04-22T17:35:11.301702+0000 mgr.y (mgr.34107) 339 : cluster [DBG] pgmap v609: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2022-04-22T17:35:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: cluster 2022-04-22T17:35:11.967762+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2022-04-22T17:35:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.380220+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.387709+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.397088+0000 mon.a (mon.0) 1126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.399887+0000 mon.a (mon.0) 1127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.401503+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.402747+0000 mon.a (mon.0) 1129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.404658+0000 mon.a (mon.0) 1130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:12 smithi121 bash[36902]: audit 2022-04-22T17:35:12.406774+0000 mon.a (mon.0) 1131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:35:13.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: cluster 2022-04-22T17:35:11.301702+0000 mgr.y (mgr.34107) 339 : cluster [DBG] pgmap v609: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2022-04-22T17:35:13.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: cluster 2022-04-22T17:35:11.967762+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2022-04-22T17:35:13.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.380220+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.387709+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.397088+0000 mon.a (mon.0) 1126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.399887+0000 mon.a (mon.0) 1127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.401503+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.402747+0000 mon.a (mon.0) 1129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.404658+0000 mon.a (mon.0) 1130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[45495]: audit 2022-04-22T17:35:12.406774+0000 mon.a (mon.0) 1131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: cluster 2022-04-22T17:35:11.301702+0000 mgr.y (mgr.34107) 339 : cluster [DBG] pgmap v609: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 327 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: cluster 2022-04-22T17:35:11.967762+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.380220+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.387709+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.397088+0000 mon.a (mon.0) 1126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.399887+0000 mon.a (mon.0) 1127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.401503+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.402747+0000 mon.a (mon.0) 1129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.404658+0000 mon.a (mon.0) 1130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:13.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:12 smithi115 bash[46934]: audit 2022-04-22T17:35:12.406774+0000 mon.a (mon.0) 1131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds ok-to-stop", "ids": ["b"]}]: dispatch 2022-04-22T17:35:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:13.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:13.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:13.527Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:13.998 INFO:tasks.rados.rados.0.smithi115.stdout:2797: done (2 left) 2022-04-22T17:35:13.998 INFO:tasks.rados.rados.0.smithi115.stdout:2798: done (1 left) 2022-04-22T17:35:13.998 INFO:tasks.rados.rados.0.smithi115.stdout:2799: done (0 left) 2022-04-22T17:35:13.998 INFO:tasks.rados.rados.0.smithi115.stdout:2800: snap_remove snap 263 2022-04-22T17:35:14.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:13 smithi121 bash[36902]: cephadm 2022-04-22T17:35:12.368019+0000 mgr.y (mgr.34107) 340 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:35:14.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:13 smithi121 bash[36902]: cephadm 2022-04-22T17:35:12.407636+0000 mgr.y (mgr.34107) 341 : cephadm [INF] Upgrade: It appears safe to stop mds.b 2022-04-22T17:35:14.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:13 smithi121 bash[36902]: cluster 2022-04-22T17:35:12.977139+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2022-04-22T17:35:14.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:13 smithi121 bash[36902]: audit 2022-04-22T17:35:13.296164+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:35:14.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:13 smithi121 bash[36902]: audit 2022-04-22T17:35:13.296657+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:35:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[45495]: cephadm 2022-04-22T17:35:12.368019+0000 mgr.y (mgr.34107) 340 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:35:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[45495]: cephadm 2022-04-22T17:35:12.407636+0000 mgr.y (mgr.34107) 341 : cephadm [INF] Upgrade: It appears safe to stop mds.b 2022-04-22T17:35:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[45495]: cluster 2022-04-22T17:35:12.977139+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2022-04-22T17:35:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[45495]: audit 2022-04-22T17:35:13.296164+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[45495]: audit 2022-04-22T17:35:13.296657+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[46934]: cephadm 2022-04-22T17:35:12.368019+0000 mgr.y (mgr.34107) 340 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[46934]: cephadm 2022-04-22T17:35:12.407636+0000 mgr.y (mgr.34107) 341 : cephadm [INF] Upgrade: It appears safe to stop mds.b 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[46934]: cluster 2022-04-22T17:35:12.977139+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[46934]: audit 2022-04-22T17:35:13.296164+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:35:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:13 smithi115 bash[46934]: audit 2022-04-22T17:35:13.296657+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:35:14.923 INFO:tasks.rados.rados.0.smithi115.stdout:2800: done (0 left) 2022-04-22T17:35:14.923 INFO:tasks.rados.rados.0.smithi115.stdout:2801: snap_remove snap 254 2022-04-22T17:35:15.231 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: cluster 2022-04-22T17:35:13.302300+0000 mgr.y (mgr.34107) 342 : cluster [DBG] pgmap v612: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 330 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:15.231 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: cluster 2022-04-22T17:35:13.985179+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2022-04-22T17:35:15.231 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: audit 2022-04-22T17:35:14.348278+0000 mon.a (mon.0) 1136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:15.231 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: cluster 2022-04-22T17:35:14.905211+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2022-04-22T17:35:15.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: audit 2022-04-22T17:35:14.934095+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:15.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: audit 2022-04-22T17:35:14.937348+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:35:15.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:14 smithi121 bash[36902]: audit 2022-04-22T17:35:14.938921+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: cluster 2022-04-22T17:35:13.302300+0000 mgr.y (mgr.34107) 342 : cluster [DBG] pgmap v612: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 330 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: cluster 2022-04-22T17:35:13.985179+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2022-04-22T17:35:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: audit 2022-04-22T17:35:14.348278+0000 mon.a (mon.0) 1136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: cluster 2022-04-22T17:35:14.905211+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: audit 2022-04-22T17:35:14.934095+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: audit 2022-04-22T17:35:14.937348+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[45495]: audit 2022-04-22T17:35:14.938921+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: cluster 2022-04-22T17:35:13.302300+0000 mgr.y (mgr.34107) 342 : cluster [DBG] pgmap v612: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 330 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: cluster 2022-04-22T17:35:13.985179+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: audit 2022-04-22T17:35:14.348278+0000 mon.a (mon.0) 1136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: cluster 2022-04-22T17:35:14.905211+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2022-04-22T17:35:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: audit 2022-04-22T17:35:14.934095+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:15.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: audit 2022-04-22T17:35:14.937348+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-04-22T17:35:15.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:14 smithi115 bash[46934]: audit 2022-04-22T17:35:14.938921+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:15.921 INFO:tasks.rados.rados.0.smithi115.stdout:2801: done (0 left) 2022-04-22T17:35:15.921 INFO:tasks.rados.rados.0.smithi115.stdout:2802: rollback oid 35 current snap is 279 2022-04-22T17:35:15.921 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 259 2022-04-22T17:35:15.922 INFO:tasks.rados.rados.0.smithi115.stdout:2803: snap_remove snap 276 2022-04-22T17:35:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:16 smithi121 bash[36902]: cephadm 2022-04-22T17:35:14.921247+0000 mgr.y (mgr.34107) 343 : cephadm [INF] Upgrade: Updating mds.b 2022-04-22T17:35:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:16 smithi121 bash[36902]: cephadm 2022-04-22T17:35:14.941088+0000 mgr.y (mgr.34107) 344 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:35:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:16 smithi121 bash[36902]: cluster 2022-04-22T17:35:15.907453+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2022-04-22T17:35:16.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[45495]: cephadm 2022-04-22T17:35:14.921247+0000 mgr.y (mgr.34107) 343 : cephadm [INF] Upgrade: Updating mds.b 2022-04-22T17:35:16.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[45495]: cephadm 2022-04-22T17:35:14.941088+0000 mgr.y (mgr.34107) 344 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:35:16.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[45495]: cluster 2022-04-22T17:35:15.907453+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2022-04-22T17:35:16.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[46934]: cephadm 2022-04-22T17:35:14.921247+0000 mgr.y (mgr.34107) 343 : cephadm [INF] Upgrade: Updating mds.b 2022-04-22T17:35:16.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[46934]: cephadm 2022-04-22T17:35:14.941088+0000 mgr.y (mgr.34107) 344 : cephadm [INF] Deploying daemon mds.b on smithi121 2022-04-22T17:35:16.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:15 smithi115 bash[46934]: cluster 2022-04-22T17:35:15.907453+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2022-04-22T17:35:16.927 INFO:tasks.rados.rados.0.smithi115.stdout:2803: done (1 left) 2022-04-22T17:35:16.934 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 279 2022-04-22T17:35:16.934 INFO:tasks.rados.rados.0.smithi115.stdout:2802: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:35:16.934 INFO:tasks.rados.rados.0.smithi115.stdout:2802: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:35:16.935 INFO:tasks.rados.rados.0.smithi115.stdout:2802: finishing rollback tid 2 to smithi11541303-35 2022-04-22T17:35:16.935 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1831 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:35:16.935 INFO:tasks.rados.rados.0.smithi115.stdout:2804: seq_num 845 ranges {1925120=557056,2482176=581632,3063808=688128,3751936=335872} 2022-04-22T17:35:16.941 INFO:tasks.rados.rados.0.smithi115.stdout:2804: writing smithi11541303-13 from 1925120 to 2482176 tid 1 2022-04-22T17:35:16.945 INFO:tasks.rados.rados.0.smithi115.stdout:2804: writing smithi11541303-13 from 2482176 to 3063808 tid 2 2022-04-22T17:35:16.952 INFO:tasks.rados.rados.0.smithi115.stdout:2804: writing smithi11541303-13 from 3063808 to 3751936 tid 3 2022-04-22T17:35:16.952 INFO:tasks.rados.rados.0.smithi115.stdout:2804: writing smithi11541303-13 from 3751936 to 4087808 tid 4 2022-04-22T17:35:16.952 INFO:tasks.rados.rados.0.smithi115.stdout:2802: done (1 left) 2022-04-22T17:35:16.952 INFO:tasks.rados.rados.0.smithi115.stdout:2805: setattr oid 11 current snap is 279 2022-04-22T17:35:16.953 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 1 to smithi11541303-13 2022-04-22T17:35:16.953 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 2 to smithi11541303-13 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:2806: read oid 49 snap -1 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:2806: expect deleted 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:2807: rollback oid 25 current snap is 279 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 275 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 279 2022-04-22T17:35:16.954 INFO:tasks.rados.rados.0.smithi115.stdout:2808: seq_num 846 ranges {1441792=573440,2015232=466944,2482176=491520,2973696=548864,3522560=98304} 2022-04-22T17:35:16.965 INFO:tasks.rados.rados.0.smithi115.stdout:2808: writing smithi11541303-8 from 1441792 to 2015232 tid 1 2022-04-22T17:35:16.968 INFO:tasks.rados.rados.0.smithi115.stdout:2808: writing smithi11541303-8 from 2015232 to 2482176 tid 2 2022-04-22T17:35:16.971 INFO:tasks.rados.rados.0.smithi115.stdout:2808: writing smithi11541303-8 from 2482176 to 2973696 tid 3 2022-04-22T17:35:16.974 INFO:tasks.rados.rados.0.smithi115.stdout:2808: writing smithi11541303-8 from 2973696 to 3522560 tid 4 2022-04-22T17:35:16.977 INFO:tasks.rados.rados.0.smithi115.stdout:2808: writing smithi11541303-8 from 3522560 to 3620864 tid 5 2022-04-22T17:35:16.977 INFO:tasks.rados.rados.0.smithi115.stdout:2809: read oid 50 snap -1 2022-04-22T17:35:16.977 INFO:tasks.rados.rados.0.smithi115.stdout:2809: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:35:16.977 INFO:tasks.rados.rados.0.smithi115.stdout:2810: rollback oid 27 current snap is 279 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 274 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 3 to smithi11541303-13 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1235 (ObjNum 749 snap 257 seq_num 749) dirty exists 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 4 to smithi11541303-13 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 5 to smithi11541303-13 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:2807: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:35:16.978 INFO:tasks.rados.rados.0.smithi115.stdout:2804: finishing write tid 6 to smithi11541303-13 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1519 (ObjNum 845 snap 279 seq_num 845) dirty exists 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2804: left oid 13 (ObjNum 845 snap 279 seq_num 845) 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2807: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2807: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1523 (ObjNum 696 snap 230 seq_num 696) dirty exists 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 1 to smithi11541303-8 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 2 to smithi11541303-8 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2804: done (6 left) 2022-04-22T17:35:16.979 INFO:tasks.rados.rados.0.smithi115.stdout:2805: done (5 left) 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2806: done (4 left) 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2807: done (3 left) 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2811: read oid 17 snap 257 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2811: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2812: rmattr oid 47 current snap is 279 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2812: done (4 left) 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 279 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 3 to smithi11541303-8 2022-04-22T17:35:16.980 INFO:tasks.rados.rados.0.smithi115.stdout:2813: seq_num 847 ranges {0=589824,589824=753664,1343488=655360,1998848=196608} 2022-04-22T17:35:16.981 INFO:tasks.rados.rados.0.smithi115.stdout:2813: writing smithi11541303-34 from 0 to 589824 tid 1 2022-04-22T17:35:16.985 INFO:tasks.rados.rados.0.smithi115.stdout:2813: writing smithi11541303-34 from 589824 to 1343488 tid 2 2022-04-22T17:35:16.989 INFO:tasks.rados.rados.0.smithi115.stdout:2813: writing smithi11541303-34 from 1343488 to 1998848 tid 3 2022-04-22T17:35:16.991 INFO:tasks.rados.rados.0.smithi115.stdout:2813: writing smithi11541303-34 from 1998848 to 2195456 tid 4 2022-04-22T17:35:16.991 INFO:tasks.rados.rados.0.smithi115.stdout:2814: delete oid 16 current snap is 279 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 4 to smithi11541303-8 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:2810: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1659 (ObjNum -1115625200 snap 22075 seq_num 3178303312) dirty dne 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 5 to smithi11541303-8 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 6 to smithi11541303-8 2022-04-22T17:35:16.992 INFO:tasks.rados.rados.0.smithi115.stdout:2808: finishing write tid 7 to smithi11541303-8 2022-04-22T17:35:16.994 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1249 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:35:16.995 INFO:tasks.rados.rados.0.smithi115.stdout:2808: left oid 8 (ObjNum 846 snap 279 seq_num 846) 2022-04-22T17:35:16.995 INFO:tasks.rados.rados.0.smithi115.stdout:2809: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:35:17.124 INFO:tasks.rados.rados.0.smithi115.stdout:2811: expect (ObjNum 611 snap 200 seq_num 611) 2022-04-22T17:35:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:17 smithi121 bash[36902]: cluster 2022-04-22T17:35:15.303249+0000 mgr.y (mgr.34107) 345 : cluster [DBG] pgmap v615: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 35 op/s 2022-04-22T17:35:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:17 smithi121 bash[36902]: cluster 2022-04-22T17:35:16.911131+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2022-04-22T17:35:17.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:16 smithi115 bash[45495]: cluster 2022-04-22T17:35:15.303249+0000 mgr.y (mgr.34107) 345 : cluster [DBG] pgmap v615: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 35 op/s 2022-04-22T17:35:17.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:16 smithi115 bash[45495]: cluster 2022-04-22T17:35:16.911131+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2022-04-22T17:35:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:16 smithi115 bash[46934]: cluster 2022-04-22T17:35:15.303249+0000 mgr.y (mgr.34107) 345 : cluster [DBG] pgmap v615: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 35 op/s 2022-04-22T17:35:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:16 smithi115 bash[46934]: cluster 2022-04-22T17:35:16.911131+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2022-04-22T17:35:17.351 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 1 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 2 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 3 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 4 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 5 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: finishing write tid 6 to smithi11541303-34 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1649 (ObjNum 847 snap 279 seq_num 847) dirty exists 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2813: left oid 34 (ObjNum 847 snap 279 seq_num 847) 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2808: done (5 left) 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2809: done (4 left) 2022-04-22T17:35:17.352 INFO:tasks.rados.rados.0.smithi115.stdout:2810: done (3 left) 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2811: done (2 left) 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2813: done (1 left) 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2814: done (0 left) 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2815: read oid 29 snap -1 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2815: expect deleted 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 279 2022-04-22T17:35:17.353 INFO:tasks.rados.rados.0.smithi115.stdout:2816: seq_num 849 ranges {2408448=622592,3031040=516096,3547136=663552,4210688=425984} 2022-04-22T17:35:17.373 INFO:tasks.rados.rados.0.smithi115.stdout:2816: writing smithi11541303-4 from 2408448 to 3031040 tid 1 2022-04-22T17:35:17.376 INFO:tasks.rados.rados.0.smithi115.stdout:2816: writing smithi11541303-4 from 3031040 to 3547136 tid 2 2022-04-22T17:35:17.380 INFO:tasks.rados.rados.0.smithi115.stdout:2816: writing smithi11541303-4 from 3547136 to 4210688 tid 3 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:2816: writing smithi11541303-4 from 4210688 to 4636672 tid 4 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:2817: rollback oid 28 current snap is 279 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 259 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 1 to smithi11541303-4 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 2 to smithi11541303-4 2022-04-22T17:35:17.383 INFO:tasks.rados.rados.0.smithi115.stdout:2815: done (2 left) 2022-04-22T17:35:17.384 INFO:tasks.rados.rados.0.smithi115.stdout:2818: snap_remove snap 270 2022-04-22T17:35:17.671 INFO:tasks.rados.rados.0.smithi115.stdout:2818: done (2 left) 2022-04-22T17:35:17.671 INFO:tasks.rados.rados.0.smithi115.stdout:2819: copy_from oid 27 from oid 1 current snap is 279 2022-04-22T17:35:17.671 INFO:tasks.rados.rados.0.smithi115.stdout:2817: finishing rollback tid 0 to smithi11541303-28 2022-04-22T17:35:17.671 INFO:tasks.rados.rados.0.smithi115.stdout:2817: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:35:17.671 INFO:tasks.rados.rados.0.smithi115.stdout:2817: finishing rollback tid 2 to smithi11541303-28 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1260 (ObjNum 737 snap 254 seq_num 737) dirty exists 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 3 to smithi11541303-4 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 4 to smithi11541303-4 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 5 to smithi11541303-4 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: finishing write tid 6 to smithi11541303-4 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1664 (ObjNum 849 snap 279 seq_num 849) dirty exists 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: left oid 4 (ObjNum 849 snap 279 seq_num 849) 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2816: done (2 left) 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2817: done (1 left) 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2820: read oid 34 snap 278 2022-04-22T17:35:17.672 INFO:tasks.rados.rados.0.smithi115.stdout:2820: expect (ObjNum -1117814608 snap 22075 seq_num 3178464144) 2022-04-22T17:35:17.673 INFO:tasks.rados.rados.0.smithi115.stdout:2821: delete oid 3 current snap is 279 2022-04-22T17:35:17.677 INFO:tasks.rados.rados.0.smithi115.stdout:2820: expect (ObjNum -1117814608 snap 22075 seq_num 3178464144) 2022-04-22T17:35:17.681 INFO:tasks.rados.rados.0.smithi115.stdout:2820: done (2 left) 2022-04-22T17:35:17.681 INFO:tasks.rados.rados.0.smithi115.stdout:2821: done (1 left) 2022-04-22T17:35:17.681 INFO:tasks.rados.rados.0.smithi115.stdout:2822: rollback oid 16 current snap is 279 2022-04-22T17:35:17.681 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 272 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2823: rmattr oid 6 current snap is 279 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2823: done (2 left) 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2824: read oid 50 snap -1 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2824: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2825: copy_from oid 25 from oid 48 current snap is 279 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 279 2022-04-22T17:35:17.682 INFO:tasks.rados.rados.0.smithi115.stdout:2826: seq_num 851 ranges {0=655360,655360=688128,1343488=679936,2023424=237568} 2022-04-22T17:35:17.685 INFO:tasks.rados.rados.0.smithi115.stdout:2826: writing smithi11541303-17 from 0 to 655360 tid 1 2022-04-22T17:35:17.690 INFO:tasks.rados.rados.0.smithi115.stdout:2826: writing smithi11541303-17 from 655360 to 1343488 tid 2 2022-04-22T17:35:17.695 INFO:tasks.rados.rados.0.smithi115.stdout:2826: writing smithi11541303-17 from 1343488 to 2023424 tid 3 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2826: writing smithi11541303-17 from 2023424 to 2260992 tid 4 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2827: read oid 38 snap -1 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2827: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2828: snap_create 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2822: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:35:17.697 INFO:tasks.rados.rados.0.smithi115.stdout:2822: finishing rollback tid 2 to smithi11541303-16 2022-04-22T17:35:17.698 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1652 (ObjNum 800 snap 268 seq_num 800) dirty exists 2022-04-22T17:35:17.700 INFO:tasks.rados.rados.0.smithi115.stdout:2819: finishing copy_from to smithi11541303-27 2022-04-22T17:35:17.700 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1227 (ObjNum 753 snap 258 seq_num 753) dirty exists 2022-04-22T17:35:17.700 INFO:tasks.rados.rados.0.smithi115.stdout:2819: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:35:17.700 INFO:tasks.rados.rados.0.smithi115.stdout:2825: finishing copy_from to smithi11541303-25 2022-04-22T17:35:17.700 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1646 (ObjNum 288 snap 0 seq_num 33) dirty exists 2022-04-22T17:35:17.701 INFO:tasks.rados.rados.0.smithi115.stdout:2825: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:35:17.701 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 1 to smithi11541303-17 2022-04-22T17:35:17.701 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 2 to smithi11541303-17 2022-04-22T17:35:17.702 INFO:tasks.rados.rados.0.smithi115.stdout:2827: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:17.705 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 3 to smithi11541303-17 2022-04-22T17:35:17.707 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 4 to smithi11541303-17 2022-04-22T17:35:17.708 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 5 to smithi11541303-17 2022-04-22T17:35:17.712 INFO:tasks.rados.rados.0.smithi115.stdout:2826: finishing write tid 6 to smithi11541303-17 2022-04-22T17:35:17.713 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1651 (ObjNum 851 snap 279 seq_num 851) dirty exists 2022-04-22T17:35:17.713 INFO:tasks.rados.rados.0.smithi115.stdout:2826: left oid 17 (ObjNum 851 snap 279 seq_num 851) 2022-04-22T17:35:17.713 INFO:tasks.rados.rados.0.smithi115.stdout:2824: expect (ObjNum 543 snap 178 seq_num 543) 2022-04-22T17:35:18.015 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:17 smithi121 systemd[1]: Stopping Ceph mds.b for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:35:18.015 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:17 smithi121 bash[25529]: debug 2022-04-22T17:35:17.650+0000 7f1484040700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2022-04-22T17:35:18.015 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:17 smithi121 bash[25529]: debug 2022-04-22T17:35:17.650+0000 7f1484040700 -1 mds.b *** got signal Terminated *** 2022-04-22T17:35:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:18 smithi121 bash[36902]: cluster 2022-04-22T17:35:17.646334+0000 mon.a (mon.0) 1143 : cluster [WRN] daemon mds.b compat changed unexpectedly 2022-04-22T17:35:18.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:18 smithi121 bash[36902]: cluster 2022-04-22T17:35:17.656110+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2022-04-22T17:35:18.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:17 smithi115 bash[45495]: debug 2022-04-22T17:35:17.990+0000 7fe1e9e44700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-04-22T17:35:18.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:18 smithi115 bash[45495]: cluster 2022-04-22T17:35:17.646334+0000 mon.a (mon.0) 1143 : cluster [WRN] daemon mds.b compat changed unexpectedly 2022-04-22T17:35:18.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:18 smithi115 bash[45495]: cluster 2022-04-22T17:35:17.656110+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2022-04-22T17:35:18.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:18 smithi115 bash[46934]: cluster 2022-04-22T17:35:17.646334+0000 mon.a (mon.0) 1143 : cluster [WRN] daemon mds.b compat changed unexpectedly 2022-04-22T17:35:18.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:18 smithi115 bash[46934]: cluster 2022-04-22T17:35:17.656110+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2022-04-22T17:35:18.672 INFO:tasks.rados.rados.0.smithi115.stdout:2819: done (6 left) 2022-04-22T17:35:18.672 INFO:tasks.rados.rados.0.smithi115.stdout:2822: done (5 left) 2022-04-22T17:35:18.672 INFO:tasks.rados.rados.0.smithi115.stdout:2824: done (4 left) 2022-04-22T17:35:18.672 INFO:tasks.rados.rados.0.smithi115.stdout:2825: done (3 left) 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2826: done (2 left) 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2827: done (1 left) 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2828: done (0 left) 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2829: rollback oid 43 current snap is 280 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 278 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2830: copy_from oid 32 from oid 46 current snap is 280 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2831: copy_from oid 33 from oid 40 current snap is 280 2022-04-22T17:35:18.673 INFO:tasks.rados.rados.0.smithi115.stdout:2832: snap_remove snap 274 2022-04-22T17:35:19.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:17.304109+0000 mgr.y (mgr.34107) 346 : cluster [DBG] pgmap v618: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 11 op/s 2022-04-22T17:35:19.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:17.994039+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2022-04-22T17:35:19.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:17.994127+0000 mon.a (mon.0) 1146 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-04-22T17:35:19.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:18.004940+0000 mon.a (mon.0) 1147 : cluster [INF] Standby daemon mds.a assigned to filesystem cephfs as rank 0 2022-04-22T17:35:19.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:18.005047+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2022-04-22T17:35:19.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:18.005295+0000 mon.a (mon.0) 1149 : cluster [DBG] fsmap cephfs:0/1 1 up:standby, 1 failed 2022-04-22T17:35:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:18.009904+0000 mon.a (mon.0) 1150 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:replay} 2022-04-22T17:35:19.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[46934]: cluster 2022-04-22T17:35:18.658536+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2022-04-22T17:35:19.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:17.304109+0000 mgr.y (mgr.34107) 346 : cluster [DBG] pgmap v618: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 11 op/s 2022-04-22T17:35:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:17.994039+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2022-04-22T17:35:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:17.994127+0000 mon.a (mon.0) 1146 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-04-22T17:35:19.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:18.004940+0000 mon.a (mon.0) 1147 : cluster [INF] Standby daemon mds.a assigned to filesystem cephfs as rank 0 2022-04-22T17:35:19.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:18.005047+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2022-04-22T17:35:19.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:18.005295+0000 mon.a (mon.0) 1149 : cluster [DBG] fsmap cephfs:0/1 1 up:standby, 1 failed 2022-04-22T17:35:19.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:18.009904+0000 mon.a (mon.0) 1150 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:replay} 2022-04-22T17:35:19.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:19 smithi115 bash[45495]: cluster 2022-04-22T17:35:18.658536+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2022-04-22T17:35:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:17.304109+0000 mgr.y (mgr.34107) 346 : cluster [DBG] pgmap v618: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 11 op/s 2022-04-22T17:35:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:17.994039+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2022-04-22T17:35:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:17.994127+0000 mon.a (mon.0) 1146 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-04-22T17:35:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:18.004940+0000 mon.a (mon.0) 1147 : cluster [INF] Standby daemon mds.a assigned to filesystem cephfs as rank 0 2022-04-22T17:35:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:18.005047+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2022-04-22T17:35:19.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:18.005295+0000 mon.a (mon.0) 1149 : cluster [DBG] fsmap cephfs:0/1 1 up:standby, 1 failed 2022-04-22T17:35:19.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:18.009904+0000 mon.a (mon.0) 1150 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:replay} 2022-04-22T17:35:19.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:19 smithi121 bash[36902]: cluster 2022-04-22T17:35:18.658536+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2022-04-22T17:35:19.673 INFO:tasks.rados.rados.0.smithi115.stdout:2832: done (3 left) 2022-04-22T17:35:19.674 INFO:tasks.rados.rados.0.smithi115.stdout:2833: delete oid 29 current snap is 280 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2829: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2829: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2829: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1654 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2831: finishing copy_from to smithi11541303-33 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2831: got expected ENOENT (src dne) 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2831: finishing copy_from racing read to smithi11541303-33 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:2830: finishing copy_from to smithi11541303-32 2022-04-22T17:35:19.675 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1221 (ObjNum 788 snap 267 seq_num 788) dirty exists 2022-04-22T17:35:19.676 INFO:tasks.rados.rados.0.smithi115.stdout:2830: finishing copy_from racing read to smithi11541303-32 2022-04-22T17:35:19.686 INFO:tasks.rados.rados.0.smithi115.stdout:2829: done (3 left) 2022-04-22T17:35:19.686 INFO:tasks.rados.rados.0.smithi115.stdout:2830: done (2 left) 2022-04-22T17:35:19.686 INFO:tasks.rados.rados.0.smithi115.stdout:2831: done (1 left) 2022-04-22T17:35:19.687 INFO:tasks.rados.rados.0.smithi115.stdout:2833: done (0 left) 2022-04-22T17:35:19.687 INFO:tasks.rados.rados.0.smithi115.stdout:2834: delete oid 21 current snap is 280 2022-04-22T17:35:19.694 INFO:tasks.rados.rados.0.smithi115.stdout:2834: done (0 left) 2022-04-22T17:35:19.694 INFO:tasks.rados.rados.0.smithi115.stdout:2835: copy_from oid 13 from oid 17 current snap is 280 2022-04-22T17:35:19.694 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 280 2022-04-22T17:35:19.694 INFO:tasks.rados.rados.0.smithi115.stdout:2836: seq_num 854 ranges {2146304=712704,2859008=794624,3653632=450560,4104192=360448} 2022-04-22T17:35:19.710 INFO:tasks.rados.rados.0.smithi115.stdout:2836: writing smithi11541303-26 from 2146304 to 2859008 tid 1 2022-04-22T17:35:19.714 INFO:tasks.rados.rados.0.smithi115.stdout:2836: writing smithi11541303-26 from 2859008 to 3653632 tid 2 2022-04-22T17:35:19.717 INFO:tasks.rados.rados.0.smithi115.stdout:2836: writing smithi11541303-26 from 3653632 to 4104192 tid 3 2022-04-22T17:35:19.719 INFO:tasks.rados.rados.0.smithi115.stdout:2836: writing smithi11541303-26 from 4104192 to 4464640 tid 4 2022-04-22T17:35:19.719 INFO:tasks.rados.rados.0.smithi115.stdout:2837: snap_create 2022-04-22T17:35:19.720 INFO:tasks.rados.rados.0.smithi115.stdout:2835: finishing copy_from to smithi11541303-13 2022-04-22T17:35:19.720 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1651 (ObjNum 851 snap 279 seq_num 851) dirty exists 2022-04-22T17:35:19.720 INFO:tasks.rados.rados.0.smithi115.stdout:2835: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:35:19.720 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 1 to smithi11541303-26 2022-04-22T17:35:19.723 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 2 to smithi11541303-26 2022-04-22T17:35:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 3 to smithi11541303-26 2022-04-22T17:35:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 4 to smithi11541303-26 2022-04-22T17:35:19.724 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 5 to smithi11541303-26 2022-04-22T17:35:19.726 INFO:tasks.rados.rados.0.smithi115.stdout:2836: finishing write tid 6 to smithi11541303-26 2022-04-22T17:35:19.726 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1368 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:19.726 INFO:tasks.rados.rados.0.smithi115.stdout:2836: left oid 26 (ObjNum 854 snap 280 seq_num 854) 2022-04-22T17:35:19.921 INFO:tasks.rados.rados.0.smithi115.stdout:2835: done (2 left) 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2836: done (1 left) 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2837: done (0 left) 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2838: copy_from oid 24 from oid 35 current snap is 281 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2839: read oid 32 snap 257 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2839: expect (ObjNum -1116449600 snap 22075 seq_num 3178572640) 2022-04-22T17:35:19.922 INFO:tasks.rados.rados.0.smithi115.stdout:2840: delete oid 31 current snap is 281 2022-04-22T17:35:19.927 INFO:tasks.rados.rados.0.smithi115.stdout:2839: expect (ObjNum -1116449600 snap 22075 seq_num 3178572640) 2022-04-22T17:35:19.932 INFO:tasks.rados.rados.0.smithi115.stdout:2839: done (2 left) 2022-04-22T17:35:19.932 INFO:tasks.rados.rados.0.smithi115.stdout:2840: done (1 left) 2022-04-22T17:35:19.932 INFO:tasks.rados.rados.0.smithi115.stdout:2841: rollback oid 45 current snap is 281 2022-04-22T17:35:19.932 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 278 2022-04-22T17:35:19.932 INFO:tasks.rados.rados.0.smithi115.stdout:2842: delete oid 39 current snap is 281 2022-04-22T17:35:19.939 INFO:tasks.rados.rados.0.smithi115.stdout:2841: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:35:19.940 INFO:tasks.rados.rados.0.smithi115.stdout:2841: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:35:19.940 INFO:tasks.rados.rados.0.smithi115.stdout:2841: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:35:19.940 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1667 (ObjNum 840 snap 276 seq_num 840) dirty exists 2022-04-22T17:35:19.942 INFO:tasks.rados.rados.0.smithi115.stdout:2841: done (2 left) 2022-04-22T17:35:19.942 INFO:tasks.rados.rados.0.smithi115.stdout:2842: done (1 left) 2022-04-22T17:35:19.942 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 281 2022-04-22T17:35:19.942 INFO:tasks.rados.rados.0.smithi115.stdout:2843: seq_num 857 ranges {2015232=761856,2777088=442368,3219456=630784,3850240=532480} 2022-04-22T17:35:19.962 INFO:tasks.rados.rados.0.smithi115.stdout:2843: writing smithi11541303-6 from 2015232 to 2777088 tid 1 2022-04-22T17:35:19.963 INFO:tasks.rados.rados.0.smithi115.stdout:2843: writing smithi11541303-6 from 2777088 to 3219456 tid 2 2022-04-22T17:35:19.966 INFO:tasks.rados.rados.0.smithi115.stdout:2843: writing smithi11541303-6 from 3219456 to 3850240 tid 3 2022-04-22T17:35:19.970 INFO:tasks.rados.rados.0.smithi115.stdout:2843: writing smithi11541303-6 from 3850240 to 4382720 tid 4 2022-04-22T17:35:19.970 INFO:tasks.rados.rados.0.smithi115.stdout:2844: snap_remove snap 280 2022-04-22T17:35:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: cluster 2022-04-22T17:35:19.013731+0000 mon.a (mon.0) 1152 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:reconnect 2022-04-22T17:35:20.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: cluster 2022-04-22T17:35:19.013816+0000 mon.a (mon.0) 1153 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:reconnect} 2022-04-22T17:35:20.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: audit 2022-04-22T17:35:19.169498+0000 mon.a (mon.0) 1154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:20.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: audit 2022-04-22T17:35:19.351218+0000 mon.a (mon.0) 1155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:20.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: cluster 2022-04-22T17:35:19.660231+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[45495]: cluster 2022-04-22T17:35:19.907737+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: cluster 2022-04-22T17:35:19.013731+0000 mon.a (mon.0) 1152 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:reconnect 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: cluster 2022-04-22T17:35:19.013816+0000 mon.a (mon.0) 1153 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:reconnect} 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: audit 2022-04-22T17:35:19.169498+0000 mon.a (mon.0) 1154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: audit 2022-04-22T17:35:19.351218+0000 mon.a (mon.0) 1155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: cluster 2022-04-22T17:35:19.660231+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2022-04-22T17:35:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:20 smithi115 bash[46934]: cluster 2022-04-22T17:35:19.907737+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: cluster 2022-04-22T17:35:19.013731+0000 mon.a (mon.0) 1152 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:reconnect 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: cluster 2022-04-22T17:35:19.013816+0000 mon.a (mon.0) 1153 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:reconnect} 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: audit 2022-04-22T17:35:19.169498+0000 mon.a (mon.0) 1154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: audit 2022-04-22T17:35:19.351218+0000 mon.a (mon.0) 1155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: cluster 2022-04-22T17:35:19.660231+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2022-04-22T17:35:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:20 smithi121 bash[36902]: cluster 2022-04-22T17:35:19.907737+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2022-04-22T17:35:20.934 INFO:tasks.rados.rados.0.smithi115.stdout:2844: done (2 left) 2022-04-22T17:35:20.935 INFO:tasks.rados.rados.0.smithi115.stdout:2845: snap_remove snap 272 2022-04-22T17:35:21.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[45495]: cluster 2022-04-22T17:35:19.304978+0000 mgr.y (mgr.34107) 347 : cluster [DBG] pgmap v621: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:35:21.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[45495]: cluster 2022-04-22T17:35:20.023371+0000 mon.a (mon.0) 1158 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:rejoin 2022-04-22T17:35:21.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[45495]: cluster 2022-04-22T17:35:20.023465+0000 mon.a (mon.0) 1159 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:rejoin} 2022-04-22T17:35:21.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[45495]: cluster 2022-04-22T17:35:20.023636+0000 mon.a (mon.0) 1160 : cluster [INF] daemon mds.a is now active in filesystem cephfs as rank 0 2022-04-22T17:35:21.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[45495]: cluster 2022-04-22T17:35:20.919404+0000 mon.a (mon.0) 1161 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2022-04-22T17:35:21.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[46934]: cluster 2022-04-22T17:35:19.304978+0000 mgr.y (mgr.34107) 347 : cluster [DBG] pgmap v621: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:35:21.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[46934]: cluster 2022-04-22T17:35:20.023371+0000 mon.a (mon.0) 1158 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:rejoin 2022-04-22T17:35:21.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[46934]: cluster 2022-04-22T17:35:20.023465+0000 mon.a (mon.0) 1159 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:rejoin} 2022-04-22T17:35:21.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[46934]: cluster 2022-04-22T17:35:20.023636+0000 mon.a (mon.0) 1160 : cluster [INF] daemon mds.a is now active in filesystem cephfs as rank 0 2022-04-22T17:35:21.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[46934]: cluster 2022-04-22T17:35:20.919404+0000 mon.a (mon.0) 1161 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2022-04-22T17:35:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:21 smithi121 bash[36902]: cluster 2022-04-22T17:35:19.304978+0000 mgr.y (mgr.34107) 347 : cluster [DBG] pgmap v621: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2022-04-22T17:35:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:21 smithi121 bash[36902]: cluster 2022-04-22T17:35:20.023371+0000 mon.a (mon.0) 1158 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:rejoin 2022-04-22T17:35:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:21 smithi121 bash[36902]: cluster 2022-04-22T17:35:20.023465+0000 mon.a (mon.0) 1159 : cluster [DBG] fsmap cephfs:1/1 {0=a=up:rejoin} 2022-04-22T17:35:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:21 smithi121 bash[36902]: cluster 2022-04-22T17:35:20.023636+0000 mon.a (mon.0) 1160 : cluster [INF] daemon mds.a is now active in filesystem cephfs as rank 0 2022-04-22T17:35:21.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:21 smithi121 bash[36902]: cluster 2022-04-22T17:35:20.919404+0000 mon.a (mon.0) 1161 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2022-04-22T17:35:21.937 INFO:tasks.rados.rados.0.smithi115.stdout:2845: done (2 left) 2022-04-22T17:35:21.937 INFO:tasks.rados.rados.0.smithi115.stdout:2846: delete oid 4 current snap is 281 2022-04-22T17:35:21.937 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 1 to smithi11541303-6 2022-04-22T17:35:21.937 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 2 to smithi11541303-6 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 3 to smithi11541303-6 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 4 to smithi11541303-6 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 5 to smithi11541303-6 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2838: finishing copy_from to smithi11541303-24 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1831 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2838: finishing copy_from racing read to smithi11541303-24 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2843: finishing write tid 6 to smithi11541303-6 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1260 (ObjNum 857 snap 281 seq_num 857) dirty exists 2022-04-22T17:35:21.938 INFO:tasks.rados.rados.0.smithi115.stdout:2843: left oid 6 (ObjNum 857 snap 281 seq_num 857) 2022-04-22T17:35:21.951 INFO:tasks.rados.rados.0.smithi115.stdout:2838: done (2 left) 2022-04-22T17:35:21.951 INFO:tasks.rados.rados.0.smithi115.stdout:2843: done (1 left) 2022-04-22T17:35:21.951 INFO:tasks.rados.rados.0.smithi115.stdout:2846: done (0 left) 2022-04-22T17:35:21.951 INFO:tasks.rados.rados.0.smithi115.stdout:2847: rollback oid 18 current snap is 281 2022-04-22T17:35:21.952 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 259 2022-04-22T17:35:21.952 INFO:tasks.rados.rados.0.smithi115.stdout:2848: rollback oid 20 current snap is 281 2022-04-22T17:35:21.952 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 259 2022-04-22T17:35:21.952 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 281 2022-04-22T17:35:21.952 INFO:tasks.rados.rados.0.smithi115.stdout:2849: seq_num 859 ranges {2260992=794624,3055616=614400,3670016=638976,4308992=352256} 2022-04-22T17:35:21.971 INFO:tasks.rados.rados.0.smithi115.stdout:2849: writing smithi11541303-17 from 2260992 to 3055616 tid 1 2022-04-22T17:35:21.974 INFO:tasks.rados.rados.0.smithi115.stdout:2849: writing smithi11541303-17 from 3055616 to 3670016 tid 2 2022-04-22T17:35:21.978 INFO:tasks.rados.rados.0.smithi115.stdout:2849: writing smithi11541303-17 from 3670016 to 4308992 tid 3 2022-04-22T17:35:21.982 INFO:tasks.rados.rados.0.smithi115.stdout:2849: writing smithi11541303-17 from 4308992 to 4661248 tid 4 2022-04-22T17:35:21.982 INFO:tasks.rados.rados.0.smithi115.stdout:2850: read oid 48 snap -1 2022-04-22T17:35:21.983 INFO:tasks.rados.rados.0.smithi115.stdout:2850: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:21.983 INFO:tasks.rados.rados.0.smithi115.stdout:2848: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:35:21.983 INFO:tasks.rados.rados.0.smithi115.stdout:2851: rollback oid 19 current snap is 281 2022-04-22T17:35:21.983 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 279 2022-04-22T17:35:21.984 INFO:tasks.rados.rados.0.smithi115.stdout:2847: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:35:21.984 INFO:tasks.rados.rados.0.smithi115.stdout:2848: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:35:21.984 INFO:tasks.rados.rados.0.smithi115.stdout:2847: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:35:21.985 INFO:tasks.rados.rados.0.smithi115.stdout:2847: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:35:21.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1905 (ObjNum 667 snap 222 seq_num 667) dirty exists 2022-04-22T17:35:21.985 INFO:tasks.rados.rados.0.smithi115.stdout:2848: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:35:21.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1657 (ObjNum 653 snap 212 seq_num 653) dirty exists 2022-04-22T17:35:21.985 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 1 to smithi11541303-17 2022-04-22T17:35:21.986 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 2 to smithi11541303-17 2022-04-22T17:35:21.986 INFO:tasks.rados.rados.0.smithi115.stdout:2847: done (4 left) 2022-04-22T17:35:21.986 INFO:tasks.rados.rados.0.smithi115.stdout:2848: done (3 left) 2022-04-22T17:35:21.987 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 281 2022-04-22T17:35:21.987 INFO:tasks.rados.rados.0.smithi115.stdout:2852: seq_num 860 ranges {2662400=8192} 2022-04-22T17:35:21.996 INFO:tasks.rados.rados.0.smithi115.stdout:2852: writing smithi11541303-2 from 2662400 to 2670592 tid 1 2022-04-22T17:35:21.996 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 281 2022-04-22T17:35:21.996 INFO:tasks.rados.rados.0.smithi115.stdout:2853: seq_num 861 ranges {1712128=24576} 2022-04-22T17:35:22.006 INFO:tasks.rados.rados.0.smithi115.stdout:2853: writing smithi11541303-12 from 1712128 to 1736704 tid 1 2022-04-22T17:35:22.006 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 281 2022-04-22T17:35:22.006 INFO:tasks.rados.rados.0.smithi115.stdout:2854: seq_num 862 ranges {2097152=40960} 2022-04-22T17:35:22.017 INFO:tasks.rados.rados.0.smithi115.stdout:2854: writing smithi11541303-37 from 2097152 to 2138112 tid 1 2022-04-22T17:35:22.018 INFO:tasks.rados.rados.0.smithi115.stdout:2855: snap_create 2022-04-22T17:35:22.018 INFO:tasks.rados.rados.0.smithi115.stdout:2850: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:2851: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1732 (ObjNum -1116152976 snap 22075 seq_num 3179329920) dirty dne 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 3 to smithi11541303-17 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 4 to smithi11541303-17 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 5 to smithi11541303-17 2022-04-22T17:35:22.025 INFO:tasks.rados.rados.0.smithi115.stdout:2849: finishing write tid 6 to smithi11541303-17 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1662 (ObjNum 859 snap 281 seq_num 859) dirty exists 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2849: left oid 17 (ObjNum 859 snap 281 seq_num 859) 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2852: finishing write tid 1 to smithi11541303-2 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2852: finishing write tid 2 to smithi11541303-2 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2852: finishing write tid 3 to smithi11541303-2 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1834 (ObjNum 860 snap 281 seq_num 860) dirty exists 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2852: left oid 2 (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2853: finishing write tid 1 to smithi11541303-12 2022-04-22T17:35:22.026 INFO:tasks.rados.rados.0.smithi115.stdout:2853: finishing write tid 2 to smithi11541303-12 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2853: finishing write tid 3 to smithi11541303-12 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1740 (ObjNum 861 snap 281 seq_num 861) dirty exists 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2853: left oid 12 (ObjNum 861 snap 281 seq_num 861) 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2854: finishing write tid 1 to smithi11541303-37 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2854: finishing write tid 2 to smithi11541303-37 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2854: finishing write tid 3 to smithi11541303-37 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1836 (ObjNum 862 snap 281 seq_num 862) dirty exists 2022-04-22T17:35:22.027 INFO:tasks.rados.rados.0.smithi115.stdout:2854: left oid 37 (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:35:22.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:21.632Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:22.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:21.632Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:22.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:21.632Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.021728+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2022-04-22T17:35:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.021790+0000 mon.a (mon.0) 1163 : cluster [INF] Cluster is now healthy 2022-04-22T17:35:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.035935+0000 mon.a (mon.0) 1164 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:active 2022-04-22T17:35:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.036023+0000 mon.a (mon.0) 1165 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 2022-04-22T17:35:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.922817+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2022-04-22T17:35:22.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.021728+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2022-04-22T17:35:22.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.021790+0000 mon.a (mon.0) 1163 : cluster [INF] Cluster is now healthy 2022-04-22T17:35:22.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.035935+0000 mon.a (mon.0) 1164 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:active 2022-04-22T17:35:22.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.036023+0000 mon.a (mon.0) 1165 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 2022-04-22T17:35:22.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:22 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.922817+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2022-04-22T17:35:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:22 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.021728+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2022-04-22T17:35:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:22 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.021790+0000 mon.a (mon.0) 1163 : cluster [INF] Cluster is now healthy 2022-04-22T17:35:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:22 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.035935+0000 mon.a (mon.0) 1164 : cluster [DBG] mds.? [v2:172.21.15.115:6834/1664655889,v1:172.21.15.115:6835/1664655889] up:active 2022-04-22T17:35:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:22 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.036023+0000 mon.a (mon.0) 1165 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 2022-04-22T17:35:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:22 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.922817+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2849: done (6 left) 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2850: done (5 left) 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2851: done (4 left) 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2852: done (3 left) 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2853: done (2 left) 2022-04-22T17:35:22.942 INFO:tasks.rados.rados.0.smithi115.stdout:2854: done (1 left) 2022-04-22T17:35:22.943 INFO:tasks.rados.rados.0.smithi115.stdout:2855: done (0 left) 2022-04-22T17:35:22.943 INFO:tasks.rados.rados.0.smithi115.stdout:2856: copy_from oid 48 from oid 43 current snap is 282 2022-04-22T17:35:22.943 INFO:tasks.rados.rados.0.smithi115.stdout:2857: setattr oid 7 current snap is 282 2022-04-22T17:35:22.944 INFO:tasks.rados.rados.0.smithi115.stdout:2858: read oid 36 snap 275 2022-04-22T17:35:22.944 INFO:tasks.rados.rados.0.smithi115.stdout:2858: expect deleted 2022-04-22T17:35:22.945 INFO:tasks.rados.rados.0.smithi115.stdout:2859: read oid 25 snap -1 2022-04-22T17:35:22.945 INFO:tasks.rados.rados.0.smithi115.stdout:2859: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:22.945 INFO:tasks.rados.rados.0.smithi115.stdout:2860: setattr oid 26 current snap is 282 2022-04-22T17:35:22.947 INFO:tasks.rados.rados.0.smithi115.stdout:2861: read oid 39 snap -1 2022-04-22T17:35:22.947 INFO:tasks.rados.rados.0.smithi115.stdout:2861: expect deleted 2022-04-22T17:35:22.947 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 282 2022-04-22T17:35:22.947 INFO:tasks.rados.rados.0.smithi115.stdout:2862: seq_num 863 ranges {4063232=57344} 2022-04-22T17:35:22.971 INFO:tasks.rados.rados.0.smithi115.stdout:2862: writing smithi11541303-45 from 4063232 to 4120576 tid 1 2022-04-22T17:35:22.971 INFO:tasks.rados.rados.0.smithi115.stdout:2863: snap_create 2022-04-22T17:35:22.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1369 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:22.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1250 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:35:22.971 INFO:tasks.rados.rados.0.smithi115.stdout:2859: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:22.978 INFO:tasks.rados.rados.0.smithi115.stdout:2856: finishing copy_from to smithi11541303-48 2022-04-22T17:35:22.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 1654 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:22.978 INFO:tasks.rados.rados.0.smithi115.stdout:2856: finishing copy_from racing read to smithi11541303-48 2022-04-22T17:35:22.979 INFO:tasks.rados.rados.0.smithi115.stdout:2862: finishing write tid 1 to smithi11541303-45 2022-04-22T17:35:22.979 INFO:tasks.rados.rados.0.smithi115.stdout:2862: finishing write tid 2 to smithi11541303-45 2022-04-22T17:35:22.979 INFO:tasks.rados.rados.0.smithi115.stdout:2862: finishing write tid 3 to smithi11541303-45 2022-04-22T17:35:22.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1670 (ObjNum 863 snap 282 seq_num 863) dirty exists 2022-04-22T17:35:22.979 INFO:tasks.rados.rados.0.smithi115.stdout:2862: left oid 45 (ObjNum 863 snap 282 seq_num 863) 2022-04-22T17:35:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:23 smithi121 bash[36902]: cluster 2022-04-22T17:35:21.305720+0000 mgr.y (mgr.34107) 348 : cluster [DBG] pgmap v625: 121 pgs: 121 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 6.7 MiB/s wr, 27 op/s 2022-04-22T17:35:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:23 smithi121 bash[36902]: cluster 2022-04-22T17:35:22.928574+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2022-04-22T17:35:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[45495]: cluster 2022-04-22T17:35:21.305720+0000 mgr.y (mgr.34107) 348 : cluster [DBG] pgmap v625: 121 pgs: 121 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 6.7 MiB/s wr, 27 op/s 2022-04-22T17:35:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[45495]: cluster 2022-04-22T17:35:22.928574+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2022-04-22T17:35:23.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[46934]: cluster 2022-04-22T17:35:21.305720+0000 mgr.y (mgr.34107) 348 : cluster [DBG] pgmap v625: 121 pgs: 121 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 6.7 MiB/s wr, 27 op/s 2022-04-22T17:35:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[46934]: cluster 2022-04-22T17:35:22.928574+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2022-04-22T17:35:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:23.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:23.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:23.527Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:23.942 INFO:tasks.rados.rados.0.smithi115.stdout:2856: done (7 left) 2022-04-22T17:35:23.942 INFO:tasks.rados.rados.0.smithi115.stdout:2857: done (6 left) 2022-04-22T17:35:23.943 INFO:tasks.rados.rados.0.smithi115.stdout:2858: done (5 left) 2022-04-22T17:35:23.943 INFO:tasks.rados.rados.0.smithi115.stdout:2859: done (4 left) 2022-04-22T17:35:23.943 INFO:tasks.rados.rados.0.smithi115.stdout:2860: done (3 left) 2022-04-22T17:35:23.943 INFO:tasks.rados.rados.0.smithi115.stdout:2861: done (2 left) 2022-04-22T17:35:23.943 INFO:tasks.rados.rados.0.smithi115.stdout:2862: done (1 left) 2022-04-22T17:35:23.944 INFO:tasks.rados.rados.0.smithi115.stdout:2863: done (0 left) 2022-04-22T17:35:23.944 INFO:tasks.rados.rados.0.smithi115.stdout:2864: snap_create 2022-04-22T17:35:24.923 INFO:tasks.rados.rados.0.smithi115.stdout:2864: done (0 left) 2022-04-22T17:35:24.923 INFO:tasks.rados.rados.0.smithi115.stdout:2865: delete oid 14 current snap is 284 2022-04-22T17:35:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2865: done (0 left) 2022-04-22T17:35:24.930 INFO:tasks.rados.rados.0.smithi115.stdout:2866: snap_create 2022-04-22T17:35:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:24 smithi121 bash[36902]: cluster 2022-04-22T17:35:23.306873+0000 mgr.y (mgr.34107) 349 : cluster [DBG] pgmap v628: 121 pgs: 121 active+clean; 351 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:24 smithi121 bash[36902]: cluster 2022-04-22T17:35:23.928656+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2022-04-22T17:35:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:24 smithi121 bash[36902]: audit 2022-04-22T17:35:24.353999+0000 mon.a (mon.0) 1169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:24 smithi121 bash[36902]: cluster 2022-04-22T17:35:24.909114+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2022-04-22T17:35:25.300 INFO:journalctl@ceph.mds.a.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[57947]: debug 2022-04-22T17:35:24.950+0000 7f1fc4350700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-22T17:35:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[45495]: cluster 2022-04-22T17:35:23.306873+0000 mgr.y (mgr.34107) 349 : cluster [DBG] pgmap v628: 121 pgs: 121 active+clean; 351 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[45495]: cluster 2022-04-22T17:35:23.928656+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2022-04-22T17:35:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[45495]: audit 2022-04-22T17:35:24.353999+0000 mon.a (mon.0) 1169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[45495]: cluster 2022-04-22T17:35:24.909114+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2022-04-22T17:35:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[46934]: cluster 2022-04-22T17:35:23.306873+0000 mgr.y (mgr.34107) 349 : cluster [DBG] pgmap v628: 121 pgs: 121 active+clean; 351 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 39 op/s 2022-04-22T17:35:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[46934]: cluster 2022-04-22T17:35:23.928656+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2022-04-22T17:35:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[46934]: audit 2022-04-22T17:35:24.353999+0000 mon.a (mon.0) 1169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:24 smithi115 bash[46934]: cluster 2022-04-22T17:35:24.909114+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2022-04-22T17:35:25.763 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 bash[47230]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds.b 2022-04-22T17:35:25.763 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 bash[47272]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds-b 2022-04-22T17:35:25.763 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 bash[47285]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-mds.b 2022-04-22T17:35:25.764 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@mds.b.service: Succeeded. 2022-04-22T17:35:25.764 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 systemd[1]: Stopped Ceph mds.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:25.937 INFO:tasks.rados.rados.0.smithi115.stdout:2866: done (0 left) 2022-04-22T17:35:25.937 INFO:tasks.rados.rados.0.smithi115.stdout:2867: copy_from oid 50 from oid 8 current snap is 285 2022-04-22T17:35:25.937 INFO:tasks.rados.rados.0.smithi115.stdout:2868: copy_from oid 29 from oid 43 current snap is 285 2022-04-22T17:35:25.937 INFO:tasks.rados.rados.0.smithi115.stdout:2869: read oid 40 snap -1 2022-04-22T17:35:25.937 INFO:tasks.rados.rados.0.smithi115.stdout:2869: expect deleted 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:2870: read oid 20 snap -1 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:2870: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 285 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:2871: seq_num 865 ranges {0=90112} 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:2871: writing smithi11541303-31 from 0 to 90112 tid 1 2022-04-22T17:35:25.938 INFO:tasks.rados.rados.0.smithi115.stdout:2872: snap_remove snap 278 2022-04-22T17:35:25.958 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:35:26.016 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:25 smithi121 systemd[1]: Started Ceph mds.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:26.296 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:35:26.296 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (11m) 23s ago 12m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:35:26.296 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (12m) 50s ago 12m 24.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:35:26.296 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (28s) 23s ago 13m 10.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:35:26.296 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 starting - - - - 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (7m) 50s ago 17m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (8m) 23s ago 19m 451M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (7m) 23s ago 19m 78.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6m) 50s ago 17m 55.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6m) 23s ago 18m 63.6M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (12m) 23s ago 12m 9891k - 0.18.1 e5a616e4b9cf b12464c321b5 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (12m) 50s ago 12m 9623k - 0.18.1 e5a616e4b9cf 2bda6bf6744a 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5m) 23s ago 17m 373M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4m) 23s ago 16m 295M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:35:26.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4m) 23s ago 16m 282M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3m) 23s ago 15m 181M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2m) 50s ago 15m 231M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2m) 50s ago 14m 344M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (94s) 50s ago 14m 192M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (52s) 50s ago 13m - 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:35:26.298 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (8m) 50s ago 12m 40.7M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:35:26.696 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:35:26.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 1 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 14 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:35:26.698 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:35:26.948 INFO:tasks.rados.rados.0.smithi115.stdout:2872: done (5 left) 2022-04-22T17:35:26.948 INFO:tasks.rados.rados.0.smithi115.stdout:2873: delete oid 19 current snap is 285 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:2871: finishing write tid 1 to smithi11541303-31 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:2871: finishing write tid 2 to smithi11541303-31 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:2871: finishing write tid 3 to smithi11541303-31 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1907 (ObjNum 865 snap 285 seq_num 865) dirty exists 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:2871: left oid 31 (ObjNum 865 snap 285 seq_num 865) 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:2867: finishing copy_from to smithi11541303-50 2022-04-22T17:35:26.955 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1249 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:35:26.956 INFO:tasks.rados.rados.0.smithi115.stdout:2867: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:35:26.956 INFO:tasks.rados.rados.0.smithi115.stdout:2870: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:35:27.109 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: cluster 2022-04-22T17:35:25.308004+0000 mgr.y (mgr.34107) 350 : cluster [DBG] pgmap v631: 121 pgs: 121 active+clean; 352 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 33 op/s 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:25.793606+0000 mon.a (mon.0) 1171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:25.795216+0000 mon.a (mon.0) 1172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:25.797334+0000 mon.a (mon.0) 1173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:25.798382+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:25.921600+0000 mgr.y (mgr.34107) 351 : audit [DBG] from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: cluster 2022-04-22T17:35:25.923754+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2022-04-22T17:35:27.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:26 smithi121 bash[36902]: audit 2022-04-22T17:35:26.683625+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? 172.21.15.115:0/3380080535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: cluster 2022-04-22T17:35:25.308004+0000 mgr.y (mgr.34107) 350 : cluster [DBG] pgmap v631: 121 pgs: 121 active+clean; 352 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 33 op/s 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:25.793606+0000 mon.a (mon.0) 1171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:25.795216+0000 mon.a (mon.0) 1172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:25.797334+0000 mon.a (mon.0) 1173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:25.798382+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:27.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:25.921600+0000 mgr.y (mgr.34107) 351 : audit [DBG] from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: cluster 2022-04-22T17:35:25.923754+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[45495]: audit 2022-04-22T17:35:26.683625+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? 172.21.15.115:0/3380080535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: cluster 2022-04-22T17:35:25.308004+0000 mgr.y (mgr.34107) 350 : cluster [DBG] pgmap v631: 121 pgs: 121 active+clean; 352 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 33 op/s 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:25.793606+0000 mon.a (mon.0) 1171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:25.795216+0000 mon.a (mon.0) 1172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:25.797334+0000 mon.a (mon.0) 1173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:25.798382+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:27.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:25.921600+0000 mgr.y (mgr.34107) 351 : audit [DBG] from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: cluster 2022-04-22T17:35:25.923754+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2022-04-22T17:35:27.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:26 smithi115 bash[46934]: audit 2022-04-22T17:35:26.683625+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? 172.21.15.115:0/3380080535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:27.516 INFO:journalctl@ceph.mds.b.smithi121.stdout:Apr 22 17:35:27 smithi121 bash[47357]: starting mds.b at 2022-04-22T17:35:27.842 INFO:tasks.rados.rados.0.smithi115.stdout:2868: finishing copy_from to smithi11541303-29 2022-04-22T17:35:27.842 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1654 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:27.842 INFO:tasks.rados.rados.0.smithi115.stdout:2868: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:35:27.842 INFO:tasks.rados.rados.0.smithi115.stdout:2867: done (5 left) 2022-04-22T17:35:27.842 INFO:tasks.rados.rados.0.smithi115.stdout:2868: done (4 left) 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2869: done (3 left) 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2870: done (2 left) 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2871: done (1 left) 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2873: done (0 left) 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2874: rollback oid 31 current snap is 285 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 257 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 285 2022-04-22T17:35:27.843 INFO:tasks.rados.rados.0.smithi115.stdout:2875: seq_num 867 ranges {3121152=122880} 2022-04-22T17:35:27.866 INFO:tasks.rados.rados.0.smithi115.stdout:2875: writing smithi11541303-29 from 3121152 to 3244032 tid 1 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2876: read oid 48 snap 273 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2876: expect (ObjNum 805 snap 268 seq_num 805) 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 285 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2874: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2874: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2874: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1914 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:35:27.867 INFO:tasks.rados.rados.0.smithi115.stdout:2877: seq_num 868 ranges {1736704=139264} 2022-04-22T17:35:27.877 INFO:tasks.rados.rados.0.smithi115.stdout:2877: writing smithi11541303-12 from 1736704 to 1875968 tid 1 2022-04-22T17:35:27.877 INFO:tasks.rados.rados.0.smithi115.stdout:2874: done (3 left) 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:2878: copy_from oid 41 from oid 37 current snap is 285 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:2875: finishing write tid 1 to smithi11541303-29 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:2875: finishing write tid 2 to smithi11541303-29 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:2875: finishing write tid 3 to smithi11541303-29 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1656 (ObjNum 867 snap 285 seq_num 867) dirty exists 2022-04-22T17:35:27.878 INFO:tasks.rados.rados.0.smithi115.stdout:2875: left oid 29 (ObjNum 867 snap 285 seq_num 867) 2022-04-22T17:35:27.879 INFO:tasks.rados.rados.0.smithi115.stdout:2875: done (3 left) 2022-04-22T17:35:27.879 INFO:tasks.rados.rados.0.smithi115.stdout:2879: delete oid 33 current snap is 285 2022-04-22T17:35:27.881 INFO:tasks.rados.rados.0.smithi115.stdout:2879: done (3 left) 2022-04-22T17:35:27.881 INFO:tasks.rados.rados.0.smithi115.stdout:2880: read oid 42 snap -1 2022-04-22T17:35:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:2880: expect (ObjNum 815 snap 270 seq_num 815) 2022-04-22T17:35:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:2881: copy_from oid 10 from oid 47 current snap is 285 2022-04-22T17:35:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 285 2022-04-22T17:35:27.882 INFO:tasks.rados.rados.0.smithi115.stdout:2882: seq_num 870 ranges {1204224=172032} 2022-04-22T17:35:27.892 INFO:tasks.rados.rados.0.smithi115.stdout:2882: writing smithi11541303-46 from 1204224 to 1376256 tid 1 2022-04-22T17:35:27.892 INFO:tasks.rados.rados.0.smithi115.stdout:2883: delete oid 6 current snap is 285 2022-04-22T17:35:27.892 INFO:tasks.rados.rados.0.smithi115.stdout:2877: finishing write tid 1 to smithi11541303-12 2022-04-22T17:35:27.892 INFO:tasks.rados.rados.0.smithi115.stdout:2877: finishing write tid 2 to smithi11541303-12 2022-04-22T17:35:27.893 INFO:tasks.rados.rados.0.smithi115.stdout:2877: finishing write tid 3 to smithi11541303-12 2022-04-22T17:35:27.893 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1742 (ObjNum 868 snap 285 seq_num 868) dirty exists 2022-04-22T17:35:27.893 INFO:tasks.rados.rados.0.smithi115.stdout:2877: left oid 12 (ObjNum 868 snap 285 seq_num 868) 2022-04-22T17:35:27.896 INFO:tasks.rados.rados.0.smithi115.stdout:2877: done (6 left) 2022-04-22T17:35:27.896 INFO:tasks.rados.rados.0.smithi115.stdout:2883: done (5 left) 2022-04-22T17:35:27.896 INFO:tasks.rados.rados.0.smithi115.stdout:2884: rollback oid 3 current snap is 285 2022-04-22T17:35:27.896 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 275 2022-04-22T17:35:27.896 INFO:tasks.rados.rados.0.smithi115.stdout:2885: copy_from oid 11 from oid 29 current snap is 285 2022-04-22T17:35:27.897 INFO:tasks.rados.rados.0.smithi115.stdout:2886: rollback oid 21 current snap is 285 2022-04-22T17:35:27.897 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 275 2022-04-22T17:35:27.897 INFO:tasks.rados.rados.0.smithi115.stdout:2887: delete oid 4 current snap is 285 2022-04-22T17:35:27.903 INFO:tasks.rados.rados.0.smithi115.stdout:2876: expect (ObjNum 805 snap 268 seq_num 805) 2022-04-22T17:35:28.067 INFO:tasks.rados.rados.0.smithi115.stdout:2881: finishing copy_from to smithi11541303-10 2022-04-22T17:35:28.096 INFO:tasks.rados.rados.0.smithi115.stdout:2881: got expected ENOENT (src dne) 2022-04-22T17:35:28.096 INFO:tasks.rados.rados.0.smithi115.stdout:2881: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:35:28.096 INFO:tasks.rados.rados.0.smithi115.stdout:2880: expect (ObjNum 815 snap 270 seq_num 815) 2022-04-22T17:35:28.142 INFO:tasks.rados.rados.0.smithi115.stdout:2882: finishing write tid 1 to smithi11541303-46 2022-04-22T17:35:28.142 INFO:tasks.rados.rados.0.smithi115.stdout:2886: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:35:28.142 INFO:tasks.rados.rados.0.smithi115.stdout:2882: finishing write tid 2 to smithi11541303-46 2022-04-22T17:35:28.142 INFO:tasks.rados.rados.0.smithi115.stdout:2878: finishing copy_from to smithi11541303-41 2022-04-22T17:35:28.142 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1836 (ObjNum 862 snap 281 seq_num 862) dirty exists 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2886: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1152 (ObjNum 823 snap 274 seq_num 823) dirty exists 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2884: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2884: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1259 (ObjNum 779 snap 267 seq_num 779) dirty exists 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2878: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2882: finishing write tid 3 to smithi11541303-46 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1257 (ObjNum 870 snap 285 seq_num 870) dirty exists 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2882: left oid 46 (ObjNum 870 snap 285 seq_num 870) 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:2885: finishing copy_from to smithi11541303-11 2022-04-22T17:35:28.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1656 (ObjNum 867 snap 285 seq_num 867) dirty exists 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2885: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2876: done (8 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2878: done (7 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2880: done (6 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2881: done (5 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2882: done (4 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2884: done (3 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2885: done (2 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2886: done (1 left) 2022-04-22T17:35:28.144 INFO:tasks.rados.rados.0.smithi115.stdout:2887: done (0 left) 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2888: read oid 22 snap -1 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2888: expect (ObjNum 843 snap 278 seq_num 843) 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2889: read oid 9 snap -1 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2889: expect deleted 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2890: read oid 21 snap -1 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2890: expect (ObjNum 823 snap 274 seq_num 823) 2022-04-22T17:35:28.145 INFO:tasks.rados.rados.0.smithi115.stdout:2891: snap_create 2022-04-22T17:35:28.152 INFO:tasks.rados.rados.0.smithi115.stdout:2888: expect (ObjNum 843 snap 278 seq_num 843) 2022-04-22T17:35:28.212 INFO:tasks.rados.rados.0.smithi115.stdout:2890: expect (ObjNum 823 snap 274 seq_num 823) 2022-04-22T17:35:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:27 smithi121 bash[36902]: audit 2022-04-22T17:35:26.264788+0000 mgr.y (mgr.34107) 352 : audit [DBG] from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:27 smithi121 bash[36902]: cluster 2022-04-22T17:35:26.934944+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2022-04-22T17:35:28.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:27 smithi115 bash[45495]: audit 2022-04-22T17:35:26.264788+0000 mgr.y (mgr.34107) 352 : audit [DBG] from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:28.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:27 smithi115 bash[45495]: cluster 2022-04-22T17:35:26.934944+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2022-04-22T17:35:28.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:27 smithi115 bash[46934]: audit 2022-04-22T17:35:26.264788+0000 mgr.y (mgr.34107) 352 : audit [DBG] from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:28.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:27 smithi115 bash[46934]: cluster 2022-04-22T17:35:26.934944+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2022-04-22T17:35:29.131 INFO:tasks.rados.rados.0.smithi115.stdout:2888: done (3 left) 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2889: done (2 left) 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2890: done (1 left) 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2891: done (0 left) 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2892: rollback oid 3 current snap is 286 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 279 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2893: read oid 39 snap -1 2022-04-22T17:35:29.155 INFO:tasks.rados.rados.0.smithi115.stdout:2893: expect deleted 2022-04-22T17:35:29.156 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 286 2022-04-22T17:35:29.156 INFO:tasks.rados.rados.0.smithi115.stdout:2894: seq_num 873 ranges {0=229376} 2022-04-22T17:35:29.156 INFO:tasks.rados.rados.0.smithi115.stdout:2894: writing smithi11541303-19 from 0 to 229376 tid 1 2022-04-22T17:35:29.157 INFO:tasks.rados.rados.0.smithi115.stdout:2895: read oid 26 snap -1 2022-04-22T17:35:29.157 INFO:tasks.rados.rados.0.smithi115.stdout:2895: expect (ObjNum 854 snap 280 seq_num 854) 2022-04-22T17:35:29.157 INFO:tasks.rados.rados.0.smithi115.stdout:2896: snap_remove snap 277 2022-04-22T17:35:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:28 smithi121 bash[36902]: cluster 2022-04-22T17:35:27.308811+0000 mgr.y (mgr.34107) 353 : cluster [DBG] pgmap v634: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 21 op/s 2022-04-22T17:35:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:28 smithi121 bash[36902]: cluster 2022-04-22T17:35:27.942197+0000 mon.a (mon.0) 1178 : cluster [DBG] mds.? [v2:172.21.15.121:6832/437202995,v1:172.21.15.121:6833/437202995] up:boot 2022-04-22T17:35:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:28 smithi121 bash[36902]: cluster 2022-04-22T17:35:27.942260+0000 mon.a (mon.0) 1179 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 1 up:standby 2022-04-22T17:35:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:28 smithi121 bash[36902]: audit 2022-04-22T17:35:27.942368+0000 mon.a (mon.0) 1180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[45495]: cluster 2022-04-22T17:35:27.308811+0000 mgr.y (mgr.34107) 353 : cluster [DBG] pgmap v634: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 21 op/s 2022-04-22T17:35:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[45495]: cluster 2022-04-22T17:35:27.942197+0000 mon.a (mon.0) 1178 : cluster [DBG] mds.? [v2:172.21.15.121:6832/437202995,v1:172.21.15.121:6833/437202995] up:boot 2022-04-22T17:35:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[45495]: cluster 2022-04-22T17:35:27.942260+0000 mon.a (mon.0) 1179 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 1 up:standby 2022-04-22T17:35:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[45495]: audit 2022-04-22T17:35:27.942368+0000 mon.a (mon.0) 1180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:29.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[46934]: cluster 2022-04-22T17:35:27.308811+0000 mgr.y (mgr.34107) 353 : cluster [DBG] pgmap v634: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 21 op/s 2022-04-22T17:35:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[46934]: cluster 2022-04-22T17:35:27.942197+0000 mon.a (mon.0) 1178 : cluster [DBG] mds.? [v2:172.21.15.121:6832/437202995,v1:172.21.15.121:6833/437202995] up:boot 2022-04-22T17:35:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[46934]: cluster 2022-04-22T17:35:27.942260+0000 mon.a (mon.0) 1179 : cluster [DBG] fsmap cephfs:1 {0=a=up:active} 1 up:standby 2022-04-22T17:35:29.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:28 smithi115 bash[46934]: audit 2022-04-22T17:35:27.942368+0000 mon.a (mon.0) 1180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:2896: done (4 left) 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:2897: rollback oid 22 current snap is 286 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 22 to 275 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:2892: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:2892: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:35:29.926 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1658 (ObjNum 0 snap 22075 seq_num 3178806896) dirty dne 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:2894: finishing write tid 1 to smithi11541303-19 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:2894: finishing write tid 2 to smithi11541303-19 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:2894: finishing write tid 3 to smithi11541303-19 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1744 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:2894: left oid 19 (ObjNum 873 snap 286 seq_num 873) 2022-04-22T17:35:29.927 INFO:tasks.rados.rados.0.smithi115.stdout:2895: expect (ObjNum 854 snap 280 seq_num 854) 2022-04-22T17:35:30.101 INFO:tasks.rados.rados.0.smithi115.stdout:2897: finishing rollback tid 0 to smithi11541303-22 2022-04-22T17:35:30.101 INFO:tasks.rados.rados.0.smithi115.stdout:2897: finishing rollback tid 1 to smithi11541303-22 2022-04-22T17:35:30.101 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1154 (ObjNum 1567904620 snap 1801280609 seq_num 1433497956) dirty dne 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2892: done (4 left) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2893: done (3 left) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2894: done (2 left) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2895: done (1 left) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2897: done (0 left) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2898: copy_from oid 40 from oid 31 current snap is 286 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2899: read oid 32 snap 275 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2899: expect (ObjNum 835 snap 275 seq_num 835) 2022-04-22T17:35:30.102 INFO:tasks.rados.rados.0.smithi115.stdout:2900: snap_create 2022-04-22T17:35:30.124 INFO:tasks.rados.rados.0.smithi115.stdout:2899: expect (ObjNum 835 snap 275 seq_num 835) 2022-04-22T17:35:30.274 INFO:tasks.rados.rados.0.smithi115.stdout:2898: finishing copy_from to smithi11541303-40 2022-04-22T17:35:30.275 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1914 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:35:30.275 INFO:tasks.rados.rados.0.smithi115.stdout:2898: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:35:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:30 smithi121 bash[36902]: cluster 2022-04-22T17:35:29.117440+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2022-04-22T17:35:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:30 smithi121 bash[36902]: audit 2022-04-22T17:35:29.357392+0000 mon.a (mon.0) 1182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:30 smithi121 bash[36902]: cluster 2022-04-22T17:35:29.912342+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2022-04-22T17:35:30.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[45495]: cluster 2022-04-22T17:35:29.117440+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2022-04-22T17:35:30.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[45495]: audit 2022-04-22T17:35:29.357392+0000 mon.a (mon.0) 1182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:30.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[45495]: cluster 2022-04-22T17:35:29.912342+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2022-04-22T17:35:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[46934]: cluster 2022-04-22T17:35:29.117440+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2022-04-22T17:35:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[46934]: audit 2022-04-22T17:35:29.357392+0000 mon.a (mon.0) 1182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:30.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:30 smithi115 bash[46934]: cluster 2022-04-22T17:35:29.912342+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2022-04-22T17:35:30.938 INFO:tasks.rados.rados.0.smithi115.stdout:2898: done (2 left) 2022-04-22T17:35:30.939 INFO:tasks.rados.rados.0.smithi115.stdout:2899: done (1 left) 2022-04-22T17:35:30.939 INFO:tasks.rados.rados.0.smithi115.stdout:2900: done (0 left) 2022-04-22T17:35:30.939 INFO:tasks.rados.rados.0.smithi115.stdout:2901: delete oid 11 current snap is 287 2022-04-22T17:35:30.943 INFO:tasks.rados.rados.0.smithi115.stdout:2901: done (0 left) 2022-04-22T17:35:30.943 INFO:tasks.rados.rados.0.smithi115.stdout:2902: copy_from oid 34 from oid 29 current snap is 287 2022-04-22T17:35:30.943 INFO:tasks.rados.rados.0.smithi115.stdout:2903: delete oid 14 current snap is 287 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:2903: done (1 left) 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:2904: setattr oid 36 current snap is 287 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:2905: rollback oid 10 current snap is 287 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 285 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:2906: copy_from oid 46 from oid 49 current snap is 287 2022-04-22T17:35:30.951 INFO:tasks.rados.rados.0.smithi115.stdout:2907: rollback oid 24 current snap is 287 2022-04-22T17:35:30.952 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 286 2022-04-22T17:35:30.952 INFO:tasks.rados.rados.0.smithi115.stdout:2908: read oid 15 snap -1 2022-04-22T17:35:30.952 INFO:tasks.rados.rados.0.smithi115.stdout:2908: expect (ObjNum 838 snap 276 seq_num 838) 2022-04-22T17:35:30.952 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 287 2022-04-22T17:35:30.952 INFO:tasks.rados.rados.0.smithi115.stdout:2909: seq_num 876 ranges {10182656=278528} 2022-04-22T17:35:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:2909: writing smithi11541303-20 from 10182656 to 10461184 tid 1 2022-04-22T17:35:31.011 INFO:tasks.rados.rados.0.smithi115.stdout:2910: read oid 16 snap 259 2022-04-22T17:35:31.011 INFO:tasks.rados.rados.0.smithi115.stdout:2910: expect (ObjNum 747 snap 257 seq_num 747) 2022-04-22T17:35:31.011 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 287 2022-04-22T17:35:31.011 INFO:tasks.rados.rados.0.smithi115.stdout:2911: seq_num 877 ranges {1204224=294912} 2022-04-22T17:35:31.019 INFO:tasks.rados.rados.0.smithi115.stdout:2911: writing smithi11541303-32 from 1204224 to 1499136 tid 1 2022-04-22T17:35:31.019 INFO:tasks.rados.rados.0.smithi115.stdout:2912: snap_remove snap 273 2022-04-22T17:35:31.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:31 smithi121 bash[36902]: cluster 2022-04-22T17:35:29.309567+0000 mgr.y (mgr.34107) 354 : cluster [DBG] pgmap v636: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 359 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2022-04-22T17:35:31.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:31 smithi121 bash[36902]: cluster 2022-04-22T17:35:30.924080+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2022-04-22T17:35:31.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[45495]: cluster 2022-04-22T17:35:29.309567+0000 mgr.y (mgr.34107) 354 : cluster [DBG] pgmap v636: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 359 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2022-04-22T17:35:31.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[45495]: cluster 2022-04-22T17:35:30.924080+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2022-04-22T17:35:31.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[46934]: cluster 2022-04-22T17:35:29.309567+0000 mgr.y (mgr.34107) 354 : cluster [DBG] pgmap v636: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 359 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2022-04-22T17:35:31.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[46934]: cluster 2022-04-22T17:35:30.924080+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2912: done (9 left) 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2913: read oid 13 snap -1 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2913: expect (ObjNum 851 snap 279 seq_num 851) 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2914: read oid 17 snap -1 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2914: expect (ObjNum 859 snap 281 seq_num 859) 2022-04-22T17:35:31.943 INFO:tasks.rados.rados.0.smithi115.stdout:2915: snap_remove snap 283 2022-04-22T17:35:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:31.632Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:31.632Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:31.632Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:32 smithi121 bash[36902]: cluster 2022-04-22T17:35:31.310341+0000 mgr.y (mgr.34107) 355 : cluster [DBG] pgmap v639: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2022-04-22T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:32 smithi121 bash[36902]: audit 2022-04-22T17:35:31.388919+0000 mon.a (mon.0) 1185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:32 smithi121 bash[36902]: cluster 2022-04-22T17:35:31.928731+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2022-04-22T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:32 smithi121 bash[36902]: audit 2022-04-22T17:35:31.999088+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[45495]: cluster 2022-04-22T17:35:31.310341+0000 mgr.y (mgr.34107) 355 : cluster [DBG] pgmap v639: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2022-04-22T17:35:32.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[45495]: audit 2022-04-22T17:35:31.388919+0000 mon.a (mon.0) 1185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[45495]: cluster 2022-04-22T17:35:31.928731+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2022-04-22T17:35:32.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[45495]: audit 2022-04-22T17:35:31.999088+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[46934]: cluster 2022-04-22T17:35:31.310341+0000 mgr.y (mgr.34107) 355 : cluster [DBG] pgmap v639: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2022-04-22T17:35:32.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[46934]: audit 2022-04-22T17:35:31.388919+0000 mon.a (mon.0) 1185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[46934]: cluster 2022-04-22T17:35:31.928731+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2022-04-22T17:35:32.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:32 smithi115 bash[46934]: audit 2022-04-22T17:35:31.999088+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2915: done (11 left) 2022-04-22T17:35:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2916: copy_from oid 6 from oid 31 current snap is 287 2022-04-22T17:35:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2917: read oid 45 snap 284 2022-04-22T17:35:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2917: expect (ObjNum 863 snap 282 seq_num 863) 2022-04-22T17:35:32.942 INFO:tasks.rados.rados.0.smithi115.stdout:2918: copy_from oid 47 from oid 9 current snap is 287 2022-04-22T17:35:32.943 INFO:tasks.rados.rados.0.smithi115.stdout:2919: snap_remove snap 275 2022-04-22T17:35:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:33.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:33.527Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:33.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2919: done (14 left) 2022-04-22T17:35:33.951 INFO:tasks.rados.rados.0.smithi115.stdout:2920: setattr oid 26 current snap is 287 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2921: copy_from oid 28 from oid 50 current snap is 287 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2906: finishing copy_from to smithi11541303-46 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2906: got expected ENOENT (src dne) 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2906: finishing copy_from racing read to smithi11541303-46 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2907: finishing rollback tid 0 to smithi11541303-24 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2907: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1251 (ObjNum -1115993568 snap 22075 seq_num 3178779184) dirty exists 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2905: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:35:33.952 INFO:tasks.rados.rados.0.smithi115.stdout:2907: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1835 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:2905: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:2905: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1917 (ObjNum 832 snap 275 seq_num 832) dirty exists 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:2902: finishing copy_from to smithi11541303-34 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1656 (ObjNum 867 snap 285 seq_num 867) dirty exists 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:2902: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:35:33.953 INFO:tasks.rados.rados.0.smithi115.stdout:2908: expect (ObjNum 838 snap 276 seq_num 838) 2022-04-22T17:35:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:33 smithi121 bash[36902]: cluster 2022-04-22T17:35:32.926978+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2022-04-22T17:35:34.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:33 smithi115 bash[45495]: cluster 2022-04-22T17:35:32.926978+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2022-04-22T17:35:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:33 smithi115 bash[46934]: cluster 2022-04-22T17:35:32.926978+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2902: done (15 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2904: done (14 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2905: done (13 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2906: done (12 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2907: done (11 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2908: done (10 left) 2022-04-22T17:35:34.339 INFO:tasks.rados.rados.0.smithi115.stdout:2922: read oid 42 snap 259 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:2922: expect deleted 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:2923: read oid 48 snap 257 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:2923: expect (ObjNum 741 snap 254 seq_num 741) 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:2924: rollback oid 49 current snap is 287 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 279 2022-04-22T17:35:34.340 INFO:tasks.rados.rados.0.smithi115.stdout:2910: expect (ObjNum 747 snap 257 seq_num 747) 2022-04-22T17:35:34.352 INFO:tasks.rados.rados.0.smithi115.stdout:2909: finishing write tid 1 to smithi11541303-20 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2909: finishing write tid 2 to smithi11541303-20 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2909: finishing write tid 3 to smithi11541303-20 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1664 (ObjNum 876 snap 287 seq_num 876) dirty exists 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2909: left oid 20 (ObjNum 876 snap 287 seq_num 876) 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2911: finishing write tid 1 to smithi11541303-32 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2911: finishing write tid 2 to smithi11541303-32 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2911: finishing write tid 3 to smithi11541303-32 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1372 (ObjNum 877 snap 287 seq_num 877) dirty exists 2022-04-22T17:35:34.353 INFO:tasks.rados.rados.0.smithi115.stdout:2911: left oid 32 (ObjNum 877 snap 287 seq_num 877) 2022-04-22T17:35:34.354 INFO:tasks.rados.rados.0.smithi115.stdout:2909: done (12 left) 2022-04-22T17:35:34.354 INFO:tasks.rados.rados.0.smithi115.stdout:2910: done (11 left) 2022-04-22T17:35:34.354 INFO:tasks.rados.rados.0.smithi115.stdout:2911: done (10 left) 2022-04-22T17:35:34.354 INFO:tasks.rados.rados.0.smithi115.stdout:2925: snap_remove snap 281 2022-04-22T17:35:34.937 INFO:tasks.rados.rados.0.smithi115.stdout:2925: done (10 left) 2022-04-22T17:35:34.937 INFO:tasks.rados.rados.0.smithi115.stdout:2926: snap_create 2022-04-22T17:35:34.937 INFO:tasks.rados.rados.0.smithi115.stdout:2913: expect (ObjNum 851 snap 279 seq_num 851) 2022-04-22T17:35:35.007 INFO:tasks.rados.rados.0.smithi115.stdout:2914: expect (ObjNum 859 snap 281 seq_num 859) 2022-04-22T17:35:35.171 INFO:tasks.rados.rados.0.smithi115.stdout:2918: finishing copy_from to smithi11541303-47 2022-04-22T17:35:35.171 INFO:tasks.rados.rados.0.smithi115.stdout:2918: got expected ENOENT (src dne) 2022-04-22T17:35:35.171 INFO:tasks.rados.rados.0.smithi115.stdout:2918: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:35:35.172 INFO:tasks.rados.rados.0.smithi115.stdout:2917: expect (ObjNum 863 snap 282 seq_num 863) 2022-04-22T17:35:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:34 smithi121 bash[36902]: cluster 2022-04-22T17:35:33.311267+0000 mgr.y (mgr.34107) 356 : cluster [DBG] pgmap v642: 121 pgs: 121 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2022-04-22T17:35:35.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:34 smithi121 bash[36902]: cluster 2022-04-22T17:35:33.936515+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2022-04-22T17:35:35.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:34 smithi121 bash[36902]: audit 2022-04-22T17:35:34.360356+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:35.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:34 smithi121 bash[36902]: cluster 2022-04-22T17:35:34.916464+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2022-04-22T17:35:35.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[46934]: cluster 2022-04-22T17:35:33.311267+0000 mgr.y (mgr.34107) 356 : cluster [DBG] pgmap v642: 121 pgs: 121 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2022-04-22T17:35:35.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[46934]: cluster 2022-04-22T17:35:33.936515+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2022-04-22T17:35:35.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[46934]: audit 2022-04-22T17:35:34.360356+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:35.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[46934]: cluster 2022-04-22T17:35:34.916464+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2022-04-22T17:35:35.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[45495]: cluster 2022-04-22T17:35:33.311267+0000 mgr.y (mgr.34107) 356 : cluster [DBG] pgmap v642: 121 pgs: 121 active+clean; 366 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2022-04-22T17:35:35.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[45495]: cluster 2022-04-22T17:35:33.936515+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2022-04-22T17:35:35.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[45495]: audit 2022-04-22T17:35:34.360356+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:35.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:34 smithi115 bash[45495]: cluster 2022-04-22T17:35:34.916464+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:2916: finishing copy_from to smithi11541303-6 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1914 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:2916: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1373 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:2921: finishing copy_from to smithi11541303-28 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1249 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:2921: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:35:35.381 INFO:tasks.rados.rados.0.smithi115.stdout:2924: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:35:35.382 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:35:35.382 INFO:tasks.rados.rados.0.smithi115.stdout:2923: expect (ObjNum 741 snap 254 seq_num 741) 2022-04-22T17:35:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2913: done (10 left) 2022-04-22T17:35:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2914: done (9 left) 2022-04-22T17:35:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2916: done (8 left) 2022-04-22T17:35:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2917: done (7 left) 2022-04-22T17:35:35.944 INFO:tasks.rados.rados.0.smithi115.stdout:2918: done (6 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2920: done (5 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2921: done (4 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2922: done (3 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2923: done (2 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2924: done (1 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2926: done (0 left) 2022-04-22T17:35:35.945 INFO:tasks.rados.rados.0.smithi115.stdout:2927: read oid 39 snap -1 2022-04-22T17:35:35.946 INFO:tasks.rados.rados.0.smithi115.stdout:2927: expect deleted 2022-04-22T17:35:35.946 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 288 2022-04-22T17:35:35.946 INFO:tasks.rados.rados.0.smithi115.stdout:2928: seq_num 878 ranges {2367488=311296} 2022-04-22T17:35:35.958 INFO:tasks.rados.rados.0.smithi115.stdout:2928: writing smithi11541303-42 from 2367488 to 2678784 tid 1 2022-04-22T17:35:35.958 INFO:tasks.rados.rados.0.smithi115.stdout:2929: read oid 19 snap 285 2022-04-22T17:35:35.959 INFO:tasks.rados.rados.0.smithi115.stdout:2929: expect deleted 2022-04-22T17:35:35.959 INFO:tasks.rados.rados.0.smithi115.stdout:2930: setattr oid 26 current snap is 288 2022-04-22T17:35:35.960 INFO:tasks.rados.rados.0.smithi115.stdout:2927: done (3 left) 2022-04-22T17:35:35.960 INFO:tasks.rados.rados.0.smithi115.stdout:2929: done (2 left) 2022-04-22T17:35:35.961 INFO:tasks.rados.rados.0.smithi115.stdout:2931: copy_from oid 18 from oid 43 current snap is 288 2022-04-22T17:35:35.961 INFO:tasks.rados.rados.0.smithi115.stdout:2932: snap_remove snap 282 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2932: done (3 left) 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2933: rollback oid 10 current snap is 288 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 257 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2934: rollback oid 21 current snap is 288 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 285 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1374 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:36.970 INFO:tasks.rados.rados.0.smithi115.stdout:2928: finishing write tid 1 to smithi11541303-42 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2928: finishing write tid 2 to smithi11541303-42 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2928: finishing write tid 3 to smithi11541303-42 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1929 (ObjNum 878 snap 288 seq_num 878) dirty exists 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2928: left oid 42 (ObjNum 878 snap 288 seq_num 878) 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2931: finishing copy_from to smithi11541303-18 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1654 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2931: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2928: done (4 left) 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2930: done (3 left) 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2931: done (2 left) 2022-04-22T17:35:36.971 INFO:tasks.rados.rados.0.smithi115.stdout:2935: rollback oid 29 current snap is 288 2022-04-22T17:35:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 249 2022-04-22T17:35:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 288 2022-04-22T17:35:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:2936: seq_num 879 ranges {0=327680} 2022-04-22T17:35:36.973 INFO:tasks.rados.rados.0.smithi115.stdout:2936: writing smithi11541303-38 from 0 to 327680 tid 1 2022-04-22T17:35:36.974 INFO:tasks.rados.rados.0.smithi115.stdout:2937: read oid 49 snap -1 2022-04-22T17:35:36.974 INFO:tasks.rados.rados.0.smithi115.stdout:2937: expect deleted 2022-04-22T17:35:36.974 INFO:tasks.rados.rados.0.smithi115.stdout:2938: snap_remove snap 286 2022-04-22T17:35:37.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:36 smithi121 bash[36902]: cluster 2022-04-22T17:35:35.312177+0000 mgr.y (mgr.34107) 357 : cluster [DBG] pgmap v645: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2022-04-22T17:35:37.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:36 smithi121 bash[36902]: cluster 2022-04-22T17:35:35.929740+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2022-04-22T17:35:37.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:36 smithi115 bash[45495]: cluster 2022-04-22T17:35:35.312177+0000 mgr.y (mgr.34107) 357 : cluster [DBG] pgmap v645: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2022-04-22T17:35:37.331 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:36 smithi115 bash[45495]: cluster 2022-04-22T17:35:35.929740+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2022-04-22T17:35:37.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:36 smithi115 bash[46934]: cluster 2022-04-22T17:35:35.312177+0000 mgr.y (mgr.34107) 357 : cluster [DBG] pgmap v645: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2022-04-22T17:35:37.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:36 smithi115 bash[46934]: cluster 2022-04-22T17:35:35.929740+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2933: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2933: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2936: finishing write tid 1 to smithi11541303-38 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2933: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1920 (ObjNum 742 snap 255 seq_num 742) dirty exists 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2936: finishing write tid 2 to smithi11541303-38 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2935: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:35:37.977 INFO:tasks.rados.rados.0.smithi115.stdout:2935: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1658 (ObjNum -1115481952 snap 22075 seq_num 0) dirty dne 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2934: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2936: finishing write tid 3 to smithi11541303-38 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1746 (ObjNum 879 snap 288 seq_num 879) dirty exists 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2936: left oid 38 (ObjNum 879 snap 288 seq_num 879) 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2934: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2934: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1157 (ObjNum 823 snap 274 seq_num 823) dirty exists 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2933: done (5 left) 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2934: done (4 left) 2022-04-22T17:35:37.978 INFO:tasks.rados.rados.0.smithi115.stdout:2935: done (3 left) 2022-04-22T17:35:37.979 INFO:tasks.rados.rados.0.smithi115.stdout:2936: done (2 left) 2022-04-22T17:35:37.979 INFO:tasks.rados.rados.0.smithi115.stdout:2937: done (1 left) 2022-04-22T17:35:37.979 INFO:tasks.rados.rados.0.smithi115.stdout:2938: done (0 left) 2022-04-22T17:35:37.979 INFO:tasks.rados.rados.0.smithi115.stdout:2939: snap_create 2022-04-22T17:35:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:37 smithi121 bash[36902]: cluster 2022-04-22T17:35:36.955160+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2022-04-22T17:35:38.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:37 smithi115 bash[45495]: cluster 2022-04-22T17:35:36.955160+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2022-04-22T17:35:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:37 smithi115 bash[46934]: cluster 2022-04-22T17:35:36.955160+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2022-04-22T17:35:38.979 INFO:tasks.rados.rados.0.smithi115.stdout:2939: done (0 left) 2022-04-22T17:35:38.980 INFO:tasks.rados.rados.0.smithi115.stdout:2940: read oid 27 snap -1 2022-04-22T17:35:38.980 INFO:tasks.rados.rados.0.smithi115.stdout:2940: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:35:38.980 INFO:tasks.rados.rados.0.smithi115.stdout:2941: snap_remove snap 284 2022-04-22T17:35:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:38 smithi121 bash[36902]: cluster 2022-04-22T17:35:37.313140+0000 mgr.y (mgr.34107) 358 : cluster [DBG] pgmap v648: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 376 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.1 MiB/s wr, 14 op/s 2022-04-22T17:35:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:38 smithi121 bash[36902]: cluster 2022-04-22T17:35:37.961891+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2022-04-22T17:35:39.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:38 smithi115 bash[45495]: cluster 2022-04-22T17:35:37.313140+0000 mgr.y (mgr.34107) 358 : cluster [DBG] pgmap v648: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 376 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.1 MiB/s wr, 14 op/s 2022-04-22T17:35:39.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:38 smithi115 bash[45495]: cluster 2022-04-22T17:35:37.961891+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2022-04-22T17:35:39.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:38 smithi115 bash[46934]: cluster 2022-04-22T17:35:37.313140+0000 mgr.y (mgr.34107) 358 : cluster [DBG] pgmap v648: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 376 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.1 MiB/s wr, 14 op/s 2022-04-22T17:35:39.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:38 smithi115 bash[46934]: cluster 2022-04-22T17:35:37.961891+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2022-04-22T17:35:39.934 INFO:tasks.rados.rados.0.smithi115.stdout:2941: done (1 left) 2022-04-22T17:35:39.935 INFO:tasks.rados.rados.0.smithi115.stdout:2942: snap_remove snap 288 2022-04-22T17:35:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:39 smithi121 bash[36902]: cluster 2022-04-22T17:35:38.966582+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2022-04-22T17:35:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:39 smithi121 bash[36902]: audit 2022-04-22T17:35:39.363302+0000 mon.a (mon.0) 1196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:39 smithi121 bash[36902]: cluster 2022-04-22T17:35:39.917241+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2022-04-22T17:35:40.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[45495]: cluster 2022-04-22T17:35:38.966582+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2022-04-22T17:35:40.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[45495]: audit 2022-04-22T17:35:39.363302+0000 mon.a (mon.0) 1196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:40.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[45495]: cluster 2022-04-22T17:35:39.917241+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2022-04-22T17:35:40.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[46934]: cluster 2022-04-22T17:35:38.966582+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2022-04-22T17:35:40.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[46934]: audit 2022-04-22T17:35:39.363302+0000 mon.a (mon.0) 1196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:40.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:39 smithi115 bash[46934]: cluster 2022-04-22T17:35:39.917241+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2022-04-22T17:35:40.934 INFO:tasks.rados.rados.0.smithi115.stdout:2942: done (1 left) 2022-04-22T17:35:40.935 INFO:tasks.rados.rados.0.smithi115.stdout:2943: read oid 30 snap -1 2022-04-22T17:35:40.935 INFO:tasks.rados.rados.0.smithi115.stdout:2943: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:35:40.935 INFO:tasks.rados.rados.0.smithi115.stdout:2944: setattr oid 5 current snap is 289 2022-04-22T17:35:40.937 INFO:tasks.rados.rados.0.smithi115.stdout:2945: snap_remove snap 287 2022-04-22T17:35:41.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: cluster 2022-04-22T17:35:39.314034+0000 mgr.y (mgr.34107) 359 : cluster [DBG] pgmap v651: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 372 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 4.3 MiB/s wr, 3 op/s 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.730979+0000 mon.a (mon.0) 1198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.737392+0000 mon.a (mon.0) 1199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.744648+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.747799+0000 mon.a (mon.0) 1201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.749415+0000 mon.a (mon.0) 1202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.750728+0000 mon.a (mon.0) 1203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.752926+0000 mon.a (mon.0) 1204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.755238+0000 mon.a (mon.0) 1205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.761867+0000 mon.a (mon.0) 1206 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.763323+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]: dispatch 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.768664+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]': finished 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.770021+0000 mon.a (mon.0) 1209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]: dispatch 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.775185+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]': finished 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.776330+0000 mon.a (mon.0) 1211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.783936+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.785330+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.791201+0000 mon.a (mon.0) 1214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.792858+0000 mon.a (mon.0) 1215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: cluster 2022-04-22T17:35:39.314034+0000 mgr.y (mgr.34107) 359 : cluster [DBG] pgmap v651: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 372 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 4.3 MiB/s wr, 3 op/s 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.730979+0000 mon.a (mon.0) 1198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.737392+0000 mon.a (mon.0) 1199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.744648+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.747799+0000 mon.a (mon.0) 1201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.749415+0000 mon.a (mon.0) 1202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.750728+0000 mon.a (mon.0) 1203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.752926+0000 mon.a (mon.0) 1204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.755238+0000 mon.a (mon.0) 1205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.761867+0000 mon.a (mon.0) 1206 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.763323+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]: dispatch 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.768664+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]': finished 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.770021+0000 mon.a (mon.0) 1209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]: dispatch 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.775185+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]': finished 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.776330+0000 mon.a (mon.0) 1211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.783936+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.785330+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.791201+0000 mon.a (mon.0) 1214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.792858+0000 mon.a (mon.0) 1215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.794514+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.801583+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.803085+0000 mon.a (mon.0) 1218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: audit 2022-04-22T17:35:40.809588+0000 mon.a (mon.0) 1219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[46934]: cluster 2022-04-22T17:35:40.921825+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2022-04-22T17:35:41.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.794514+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.801583+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.803085+0000 mon.a (mon.0) 1218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: audit 2022-04-22T17:35:40.809588+0000 mon.a (mon.0) 1219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:40 smithi115 bash[45495]: cluster 2022-04-22T17:35:40.921825+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2022-04-22T17:35:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: cluster 2022-04-22T17:35:39.314034+0000 mgr.y (mgr.34107) 359 : cluster [DBG] pgmap v651: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 372 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 4.3 MiB/s wr, 3 op/s 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.730979+0000 mon.a (mon.0) 1198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.737392+0000 mon.a (mon.0) 1199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.744648+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.747799+0000 mon.a (mon.0) 1201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.749415+0000 mon.a (mon.0) 1202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.750728+0000 mon.a (mon.0) 1203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.752926+0000 mon.a (mon.0) 1204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.755238+0000 mon.a (mon.0) 1205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.761867+0000 mon.a (mon.0) 1206 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.763323+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.768664+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.a"}]': finished 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.770021+0000 mon.a (mon.0) 1209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.775185+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.b"}]': finished 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.776330+0000 mon.a (mon.0) 1211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.783936+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.785330+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.791201+0000 mon.a (mon.0) 1214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.792858+0000 mon.a (mon.0) 1215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.794514+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.801583+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.803085+0000 mon.a (mon.0) 1218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:41.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: audit 2022-04-22T17:35:40.809588+0000 mon.a (mon.0) 1219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:41.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:40 smithi121 bash[36902]: cluster 2022-04-22T17:35:40.921825+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2022-04-22T17:35:41.936 INFO:tasks.rados.rados.0.smithi115.stdout:2945: done (3 left) 2022-04-22T17:35:41.937 INFO:tasks.rados.rados.0.smithi115.stdout:2946: snap_remove snap 249 2022-04-22T17:35:42.001 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:41.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:42.001 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:41.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:42.001 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:41.633Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:41 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.719511+0000 mgr.y (mgr.34107) 360 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:35:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:41 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.756245+0000 mgr.y (mgr.34107) 361 : cephadm [INF] Upgrade: Setting container_image for all mds 2022-04-22T17:35:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:42 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.777257+0000 mgr.y (mgr.34107) 362 : cephadm [INF] Upgrade: Setting container_image for all rgw 2022-04-22T17:35:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:42 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.786291+0000 mgr.y (mgr.34107) 363 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2022-04-22T17:35:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:42 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.795657+0000 mgr.y (mgr.34107) 364 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2022-04-22T17:35:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:42 smithi121 bash[36902]: cephadm 2022-04-22T17:35:40.804301+0000 mgr.y (mgr.34107) 365 : cephadm [INF] Upgrade: Setting container_image for all nfs 2022-04-22T17:35:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:42 smithi121 bash[36902]: cluster 2022-04-22T17:35:41.922730+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.719511+0000 mgr.y (mgr.34107) 360 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.756245+0000 mgr.y (mgr.34107) 361 : cephadm [INF] Upgrade: Setting container_image for all mds 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.777257+0000 mgr.y (mgr.34107) 362 : cephadm [INF] Upgrade: Setting container_image for all rgw 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.786291+0000 mgr.y (mgr.34107) 363 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.795657+0000 mgr.y (mgr.34107) 364 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cephadm 2022-04-22T17:35:40.804301+0000 mgr.y (mgr.34107) 365 : cephadm [INF] Upgrade: Setting container_image for all nfs 2022-04-22T17:35:42.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[45495]: cluster 2022-04-22T17:35:41.922730+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.719511+0000 mgr.y (mgr.34107) 360 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.756245+0000 mgr.y (mgr.34107) 361 : cephadm [INF] Upgrade: Setting container_image for all mds 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.777257+0000 mgr.y (mgr.34107) 362 : cephadm [INF] Upgrade: Setting container_image for all rgw 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.786291+0000 mgr.y (mgr.34107) 363 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.795657+0000 mgr.y (mgr.34107) 364 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cephadm 2022-04-22T17:35:40.804301+0000 mgr.y (mgr.34107) 365 : cephadm [INF] Upgrade: Setting container_image for all nfs 2022-04-22T17:35:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:41 smithi115 bash[46934]: cluster 2022-04-22T17:35:41.922730+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2022-04-22T17:35:42.938 INFO:tasks.rados.rados.0.smithi115.stdout:2946: done (3 left) 2022-04-22T17:35:42.938 INFO:tasks.rados.rados.0.smithi115.stdout:2947: rollback oid 26 current snap is 289 2022-04-22T17:35:42.938 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 285 2022-04-22T17:35:42.939 INFO:tasks.rados.rados.0.smithi115.stdout:2940: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:35:42.995 INFO:tasks.rados.rados.0.smithi115.stdout:2943: expect (ObjNum 696 snap 230 seq_num 696) 2022-04-22T17:35:43.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1945 (ObjNum 774 snap 266 seq_num 774) dirty exists 2022-04-22T17:35:43.062 INFO:tasks.rados.rados.0.smithi115.stdout:2947: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:2947: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:2947: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1377 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:2940: done (3 left) 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:2943: done (2 left) 2022-04-22T17:35:43.063 INFO:tasks.rados.rados.0.smithi115.stdout:2944: done (1 left) 2022-04-22T17:35:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:2947: done (0 left) 2022-04-22T17:35:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:2948: copy_from oid 5 from oid 7 current snap is 289 2022-04-22T17:35:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 289 2022-04-22T17:35:43.064 INFO:tasks.rados.rados.0.smithi115.stdout:2949: seq_num 880 ranges {4661248=344064} 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2949: writing smithi11541303-17 from 4661248 to 5005312 tid 1 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2950: rmattr oid 30 current snap is 289 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2950: done (2 left) 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2951: delete oid 27 current snap is 289 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:2948: finishing copy_from to smithi11541303-5 2022-04-22T17:35:43.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1250 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:35:43.085 INFO:tasks.rados.rados.0.smithi115.stdout:2948: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:35:43.091 INFO:tasks.rados.rados.0.smithi115.stdout:2948: done (2 left) 2022-04-22T17:35:43.091 INFO:tasks.rados.rados.0.smithi115.stdout:2951: done (1 left) 2022-04-22T17:35:43.091 INFO:tasks.rados.rados.0.smithi115.stdout:2952: snap_remove snap 259 2022-04-22T17:35:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:43 smithi121 bash[36902]: cluster 2022-04-22T17:35:41.314859+0000 mgr.y (mgr.34107) 366 : cluster [DBG] pgmap v654: 121 pgs: 3 active+clean+snaptrim, 118 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T17:35:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:43 smithi121 bash[36902]: cluster 2022-04-22T17:35:42.925576+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2022-04-22T17:35:43.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:42 smithi115 bash[45495]: cluster 2022-04-22T17:35:41.314859+0000 mgr.y (mgr.34107) 366 : cluster [DBG] pgmap v654: 121 pgs: 3 active+clean+snaptrim, 118 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T17:35:43.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:42 smithi115 bash[45495]: cluster 2022-04-22T17:35:42.925576+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2022-04-22T17:35:43.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:42 smithi115 bash[46934]: cluster 2022-04-22T17:35:41.314859+0000 mgr.y (mgr.34107) 366 : cluster [DBG] pgmap v654: 121 pgs: 3 active+clean+snaptrim, 118 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T17:35:43.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:42 smithi115 bash[46934]: cluster 2022-04-22T17:35:42.925576+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2022-04-22T17:35:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:43.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:43.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:43.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:44.023 INFO:tasks.rados.rados.0.smithi115.stdout:2952: done (1 left) 2022-04-22T17:35:44.023 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 289 2022-04-22T17:35:44.023 INFO:tasks.rados.rados.0.smithi115.stdout:2953: seq_num 882 ranges {0=376832} 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:2953: writing smithi11541303-3 from 0 to 376832 tid 1 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:2949: finishing write tid 1 to smithi11541303-17 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:2949: finishing write tid 2 to smithi11541303-17 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:2949: finishing write tid 3 to smithi11541303-17 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1666 (ObjNum 880 snap 289 seq_num 880) dirty exists 2022-04-22T17:35:44.025 INFO:tasks.rados.rados.0.smithi115.stdout:2949: left oid 17 (ObjNum 880 snap 289 seq_num 880) 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:2949: done (1 left) 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:2954: rmattr oid 43 current snap is 289 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:2954: done (1 left) 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:2955: rollback oid 37 current snap is 289 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 37 to 257 2022-04-22T17:35:44.026 INFO:tasks.rados.rados.0.smithi115.stdout:2956: snap_remove snap 285 2022-04-22T17:35:44.501 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:44 smithi115 systemd[1]: Stopping Ceph node-exporter.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:35:44.938 INFO:tasks.rados.rados.0.smithi115.stdout:2956: done (2 left) 2022-04-22T17:35:44.938 INFO:tasks.rados.rados.0.smithi115.stdout:2957: rollback oid 44 current snap is 289 2022-04-22T17:35:44.938 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 257 2022-04-22T17:35:44.939 INFO:tasks.rados.rados.0.smithi115.stdout:2953: finishing write tid 1 to smithi11541303-3 2022-04-22T17:35:44.939 INFO:tasks.rados.rados.0.smithi115.stdout:2953: finishing write tid 2 to smithi11541303-3 2022-04-22T17:35:44.939 INFO:tasks.rados.rados.0.smithi115.stdout:2955: finishing rollback tid 0 to smithi11541303-37 2022-04-22T17:35:44.941 INFO:tasks.rados.rados.0.smithi115.stdout:2955: finishing rollback tid 1 to smithi11541303-37 2022-04-22T17:35:44.941 INFO:tasks.rados.rados.0.smithi115.stdout:2953: finishing write tid 3 to smithi11541303-3 2022-04-22T17:35:44.941 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1661 (ObjNum 882 snap 289 seq_num 882) dirty exists 2022-04-22T17:35:44.941 INFO:tasks.rados.rados.0.smithi115.stdout:2953: left oid 3 (ObjNum 882 snap 289 seq_num 882) 2022-04-22T17:35:44.941 INFO:tasks.rados.rados.0.smithi115.stdout:2955: finishing rollback tid 2 to smithi11541303-37 2022-04-22T17:35:44.942 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1839 (ObjNum 0 snap 32692 seq_num 3178934688) dirty exists 2022-04-22T17:35:44.942 INFO:tasks.rados.rados.0.smithi115.stdout:2953: done (2 left) 2022-04-22T17:35:44.942 INFO:tasks.rados.rados.0.smithi115.stdout:2955: done (1 left) 2022-04-22T17:35:44.942 INFO:tasks.rados.rados.0.smithi115.stdout:2958: setattr oid 20 current snap is 289 2022-04-22T17:35:44.943 INFO:tasks.rados.rados.0.smithi115.stdout:2959: delete oid 10 current snap is 289 2022-04-22T17:35:44.947 INFO:tasks.rados.rados.0.smithi115.stdout:2957: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:35:44.947 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 0 (ObjNum -1115581152 snap 22075 seq_num 3178768688) dirty dne 2022-04-22T17:35:44.949 INFO:tasks.rados.rados.0.smithi115.stdout:2957: done (2 left) 2022-04-22T17:35:44.950 INFO:tasks.rados.rados.0.smithi115.stdout:2959: done (1 left) 2022-04-22T17:35:44.950 INFO:tasks.rados.rados.0.smithi115.stdout:2960: copy_from oid 8 from oid 41 current snap is 289 2022-04-22T17:35:44.950 INFO:tasks.rados.rados.0.smithi115.stdout:2961: snap_remove snap 257 2022-04-22T17:35:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: cluster 2022-04-22T17:35:43.315923+0000 mgr.y (mgr.34107) 367 : cluster [DBG] pgmap v657: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 864 KiB/s rd, 80 KiB/s wr, 12 op/s 2022-04-22T17:35:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: cephadm 2022-04-22T17:35:43.655214+0000 mgr.y (mgr.34107) 368 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2022-04-22T17:35:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: cephadm 2022-04-22T17:35:43.655844+0000 mgr.y (mgr.34107) 369 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:35:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: cluster 2022-04-22T17:35:44.006973+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2022-04-22T17:35:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: audit 2022-04-22T17:35:44.367014+0000 mon.a (mon.0) 1224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:45.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:45 smithi121 bash[36902]: cluster 2022-04-22T17:35:44.923731+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2022-04-22T17:35:45.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: cluster 2022-04-22T17:35:43.315923+0000 mgr.y (mgr.34107) 367 : cluster [DBG] pgmap v657: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 864 KiB/s rd, 80 KiB/s wr, 12 op/s 2022-04-22T17:35:45.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: cephadm 2022-04-22T17:35:43.655214+0000 mgr.y (mgr.34107) 368 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2022-04-22T17:35:45.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: cephadm 2022-04-22T17:35:43.655844+0000 mgr.y (mgr.34107) 369 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:35:45.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: cluster 2022-04-22T17:35:44.006973+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2022-04-22T17:35:45.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: audit 2022-04-22T17:35:44.367014+0000 mon.a (mon.0) 1224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:45.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[45495]: cluster 2022-04-22T17:35:44.923731+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2022-04-22T17:35:45.285 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[58915]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter.a 2022-04-22T17:35:45.285 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2022-04-22T17:35:45.285 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[58957]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter-a 2022-04-22T17:35:45.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: cluster 2022-04-22T17:35:43.315923+0000 mgr.y (mgr.34107) 367 : cluster [DBG] pgmap v657: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 370 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 864 KiB/s rd, 80 KiB/s wr, 12 op/s 2022-04-22T17:35:45.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: cephadm 2022-04-22T17:35:43.655214+0000 mgr.y (mgr.34107) 368 : cephadm [INF] Upgrade: Updating node-exporter.a (1/2) 2022-04-22T17:35:45.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: cephadm 2022-04-22T17:35:43.655844+0000 mgr.y (mgr.34107) 369 : cephadm [INF] Deploying daemon node-exporter.a on smithi115 2022-04-22T17:35:45.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: cluster 2022-04-22T17:35:44.006973+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2022-04-22T17:35:45.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: audit 2022-04-22T17:35:44.367014+0000 mon.a (mon.0) 1224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:45.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[46934]: cluster 2022-04-22T17:35:44.923731+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2022-04-22T17:35:45.549 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[58970]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter.a 2022-04-22T17:35:45.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2022-04-22T17:35:45.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 systemd[1]: Stopped Ceph node-exporter.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:45.947 INFO:tasks.rados.rados.0.smithi115.stdout:2961: done (2 left) 2022-04-22T17:35:45.947 INFO:tasks.rados.rados.0.smithi115.stdout:2962: rmattr oid 40 current snap is 289 2022-04-22T17:35:45.954 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 289 2022-04-22T17:35:45.954 INFO:tasks.rados.rados.0.smithi115.stdout:2963: seq_num 884 ranges {5005312=409600} 2022-04-22T17:35:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:2963: writing smithi11541303-17 from 5005312 to 5414912 tid 1 2022-04-22T17:35:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 289 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1667 (ObjNum 876 snap 287 seq_num 876) dirty exists 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:2960: finishing copy_from to smithi11541303-8 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1836 (ObjNum 862 snap 281 seq_num 862) dirty exists 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:2960: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1922 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:35:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:2964: seq_num 885 ranges {0=425984} 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:2964: writing smithi11541303-27 from 0 to 425984 tid 1 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:2958: done (4 left) 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:2960: done (3 left) 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:2962: done (2 left) 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 289 2022-04-22T17:35:45.989 INFO:tasks.rados.rados.0.smithi115.stdout:2965: seq_num 886 ranges {1794048=442368} 2022-04-22T17:35:46.002 INFO:tasks.rados.rados.0.smithi115.stdout:2965: writing smithi11541303-21 from 1794048 to 2236416 tid 1 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2966: copy_from oid 5 from oid 37 current snap is 289 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2963: finishing write tid 1 to smithi11541303-17 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2964: finishing write tid 1 to smithi11541303-27 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2964: finishing write tid 2 to smithi11541303-27 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2963: finishing write tid 2 to smithi11541303-17 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2963: finishing write tid 3 to smithi11541303-17 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1669 (ObjNum 884 snap 289 seq_num 884) dirty exists 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2963: left oid 17 (ObjNum 884 snap 289 seq_num 884) 2022-04-22T17:35:46.003 INFO:tasks.rados.rados.0.smithi115.stdout:2964: finishing write tid 3 to smithi11541303-27 2022-04-22T17:35:46.004 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1673 (ObjNum 885 snap 289 seq_num 885) dirty exists 2022-04-22T17:35:46.004 INFO:tasks.rados.rados.0.smithi115.stdout:2964: left oid 27 (ObjNum 885 snap 289 seq_num 885) 2022-04-22T17:35:46.004 INFO:tasks.rados.rados.0.smithi115.stdout:2963: done (3 left) 2022-04-22T17:35:46.004 INFO:tasks.rados.rados.0.smithi115.stdout:2964: done (2 left) 2022-04-22T17:35:46.004 INFO:tasks.rados.rados.0.smithi115.stdout:2967: snap_create 2022-04-22T17:35:46.005 INFO:tasks.rados.rados.0.smithi115.stdout:2965: finishing write tid 1 to smithi11541303-21 2022-04-22T17:35:46.006 INFO:tasks.rados.rados.0.smithi115.stdout:2966: finishing copy_from to smithi11541303-5 2022-04-22T17:35:46.006 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1839 (ObjNum -1116002592 snap 22075 seq_num 3178925616) dirty exists 2022-04-22T17:35:46.006 INFO:tasks.rados.rados.0.smithi115.stdout:2965: finishing write tid 2 to smithi11541303-21 2022-04-22T17:35:46.006 INFO:tasks.rados.rados.0.smithi115.stdout:2966: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:35:46.007 INFO:tasks.rados.rados.0.smithi115.stdout:2965: finishing write tid 3 to smithi11541303-21 2022-04-22T17:35:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1159 (ObjNum 886 snap 289 seq_num 886) dirty exists 2022-04-22T17:35:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:2965: left oid 21 (ObjNum 886 snap 289 seq_num 886) 2022-04-22T17:35:46.049 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 systemd[1]: Started Ceph node-exporter.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:46.050 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:45 smithi115 bash[59036]: Unable to find image 'quay.io/prometheus/node-exporter:v1.3.1' locally 2022-04-22T17:35:46.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[59036]: v1.3.1: Pulling from prometheus/node-exporter 2022-04-22T17:35:46.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[59036]: aa2a8d90b84c: Pulling fs layer 2022-04-22T17:35:46.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[59036]: b45d31ee2d7f: Pulling fs layer 2022-04-22T17:35:46.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[59036]: b5db1e299295: Pulling fs layer 2022-04-22T17:35:46.952 INFO:tasks.rados.rados.0.smithi115.stdout:2965: done (2 left) 2022-04-22T17:35:46.953 INFO:tasks.rados.rados.0.smithi115.stdout:2966: done (1 left) 2022-04-22T17:35:46.953 INFO:tasks.rados.rados.0.smithi115.stdout:2967: done (0 left) 2022-04-22T17:35:46.953 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 290 2022-04-22T17:35:46.953 INFO:tasks.rados.rados.0.smithi115.stdout:2968: seq_num 887 ranges {2260992=458752} 2022-04-22T17:35:46.972 INFO:tasks.rados.rados.0.smithi115.stdout:2968: writing smithi11541303-13 from 2260992 to 2719744 tid 1 2022-04-22T17:35:46.994 INFO:tasks.rados.rados.0.smithi115.stdout:2969: rollback oid 43 current snap is 290 2022-04-22T17:35:46.994 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 289 2022-04-22T17:35:46.995 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 290 2022-04-22T17:35:46.995 INFO:tasks.rados.rados.0.smithi115.stdout:2970: seq_num 888 ranges {2064384=475136} 2022-04-22T17:35:46.996 INFO:tasks.rados.rados.0.smithi115.stdout:2970: writing smithi11541303-30 from 2064384 to 2539520 tid 1 2022-04-22T17:35:46.996 INFO:tasks.rados.rados.0.smithi115.stdout:2971: snap_create 2022-04-22T17:35:46.997 INFO:tasks.rados.rados.0.smithi115.stdout:2968: finishing write tid 1 to smithi11541303-13 2022-04-22T17:35:46.997 INFO:tasks.rados.rados.0.smithi115.stdout:2968: finishing write tid 2 to smithi11541303-13 2022-04-22T17:35:46.997 INFO:tasks.rados.rados.0.smithi115.stdout:2969: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:35:46.998 INFO:tasks.rados.rados.0.smithi115.stdout:2969: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:35:46.998 INFO:tasks.rados.rados.0.smithi115.stdout:2968: finishing write tid 3 to smithi11541303-13 2022-04-22T17:35:46.998 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1671 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:35:46.999 INFO:tasks.rados.rados.0.smithi115.stdout:2968: left oid 13 (ObjNum 887 snap 290 seq_num 887) 2022-04-22T17:35:46.999 INFO:tasks.rados.rados.0.smithi115.stdout:2969: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:35:46.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1674 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:35:47.000 INFO:tasks.rados.rados.0.smithi115.stdout:2970: finishing write tid 1 to smithi11541303-30 2022-04-22T17:35:47.000 INFO:tasks.rados.rados.0.smithi115.stdout:2970: finishing write tid 2 to smithi11541303-30 2022-04-22T17:35:47.000 INFO:tasks.rados.rados.0.smithi115.stdout:2970: finishing write tid 3 to smithi11541303-30 2022-04-22T17:35:47.000 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1675 (ObjNum 888 snap 290 seq_num 888) dirty exists 2022-04-22T17:35:47.000 INFO:tasks.rados.rados.0.smithi115.stdout:2970: left oid 30 (ObjNum 888 snap 290 seq_num 888) 2022-04-22T17:35:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:46 smithi121 bash[36902]: cluster 2022-04-22T17:35:45.316698+0000 mgr.y (mgr.34107) 370 : cluster [DBG] pgmap v660: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 172 KiB/s wr, 9 op/s 2022-04-22T17:35:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:46 smithi121 bash[36902]: audit 2022-04-22T17:35:45.583829+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:46 smithi121 bash[36902]: cluster 2022-04-22T17:35:45.934261+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2022-04-22T17:35:47.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[46934]: cluster 2022-04-22T17:35:45.316698+0000 mgr.y (mgr.34107) 370 : cluster [DBG] pgmap v660: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 172 KiB/s wr, 9 op/s 2022-04-22T17:35:47.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[46934]: audit 2022-04-22T17:35:45.583829+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:47.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[46934]: cluster 2022-04-22T17:35:45.934261+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2022-04-22T17:35:47.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[45495]: cluster 2022-04-22T17:35:45.316698+0000 mgr.y (mgr.34107) 370 : cluster [DBG] pgmap v660: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 354 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 172 KiB/s wr, 9 op/s 2022-04-22T17:35:47.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[45495]: audit 2022-04-22T17:35:45.583829+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:47.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:46 smithi115 bash[45495]: cluster 2022-04-22T17:35:45.934261+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2022-04-22T17:35:47.960 INFO:tasks.rados.rados.0.smithi115.stdout:2968: done (3 left) 2022-04-22T17:35:47.961 INFO:tasks.rados.rados.0.smithi115.stdout:2969: done (2 left) 2022-04-22T17:35:47.961 INFO:tasks.rados.rados.0.smithi115.stdout:2970: done (1 left) 2022-04-22T17:35:47.961 INFO:tasks.rados.rados.0.smithi115.stdout:2971: done (0 left) 2022-04-22T17:35:47.961 INFO:tasks.rados.rados.0.smithi115.stdout:2972: snap_remove snap 289 2022-04-22T17:35:48.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:47 smithi121 bash[36902]: cluster 2022-04-22T17:35:46.939768+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2022-04-22T17:35:48.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:47 smithi115 bash[45495]: cluster 2022-04-22T17:35:46.939768+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2022-04-22T17:35:48.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:47 smithi115 bash[46934]: cluster 2022-04-22T17:35:46.939768+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2022-04-22T17:35:48.766 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:48 smithi121 systemd[1]: Stopping Ceph node-exporter.b for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:35:48.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[59036]: b45d31ee2d7f: Verifying Checksum 2022-04-22T17:35:48.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[59036]: b45d31ee2d7f: Download complete 2022-04-22T17:35:48.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[59036]: aa2a8d90b84c: Download complete 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2972: done (0 left) 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2973: setattr oid 40 current snap is 291 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2974: read oid 41 snap 279 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2974: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2975: read oid 8 snap -1 2022-04-22T17:35:48.967 INFO:tasks.rados.rados.0.smithi115.stdout:2975: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:35:48.968 INFO:tasks.rados.rados.0.smithi115.stdout:2976: delete oid 35 current snap is 291 2022-04-22T17:35:48.981 INFO:tasks.rados.rados.0.smithi115.stdout:2976: done (3 left) 2022-04-22T17:35:48.981 INFO:tasks.rados.rados.0.smithi115.stdout:2977: snap_create 2022-04-22T17:35:48.988 INFO:tasks.rados.rados.0.smithi115.stdout:2975: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:35:49.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[45495]: cluster 2022-04-22T17:35:47.317602+0000 mgr.y (mgr.34107) 371 : cluster [DBG] pgmap v663: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 5 op/s 2022-04-22T17:35:49.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[45495]: cluster 2022-04-22T17:35:47.946497+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2022-04-22T17:35:49.050 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[59036]: aa2a8d90b84c: Pull complete 2022-04-22T17:35:49.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[46934]: cluster 2022-04-22T17:35:47.317602+0000 mgr.y (mgr.34107) 371 : cluster [DBG] pgmap v663: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 5 op/s 2022-04-22T17:35:49.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:48 smithi115 bash[46934]: cluster 2022-04-22T17:35:47.946497+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2022-04-22T17:35:49.095 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 1923 (ObjNum 732 snap 252 seq_num 732) dirty exists 2022-04-22T17:35:49.095 INFO:tasks.rados.rados.0.smithi115.stdout:2974: expect (ObjNum 653 snap 212 seq_num 653) 2022-04-22T17:35:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:48 smithi121 bash[36902]: cluster 2022-04-22T17:35:47.317602+0000 mgr.y (mgr.34107) 371 : cluster [DBG] pgmap v663: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 5 op/s 2022-04-22T17:35:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:48 smithi121 bash[36902]: cluster 2022-04-22T17:35:47.946497+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2022-04-22T17:35:49.551 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[59036]: b45d31ee2d7f: Pull complete 2022-04-22T17:35:49.975 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[48320]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter.b 2022-04-22T17:35:49.989 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2022-04-22T17:35:49.989 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[48362]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter-b 2022-04-22T17:35:49.989 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[48375]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-node-exporter.b 2022-04-22T17:35:49.989 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2022-04-22T17:35:49.989 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:49 smithi121 systemd[1]: Stopped Ceph node-exporter.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:49.993 INFO:tasks.rados.rados.0.smithi115.stdout:2973: done (3 left) 2022-04-22T17:35:49.993 INFO:tasks.rados.rados.0.smithi115.stdout:2974: done (2 left) 2022-04-22T17:35:49.994 INFO:tasks.rados.rados.0.smithi115.stdout:2975: done (1 left) 2022-04-22T17:35:49.994 INFO:tasks.rados.rados.0.smithi115.stdout:2977: done (0 left) 2022-04-22T17:35:49.994 INFO:tasks.rados.rados.0.smithi115.stdout:2978: setattr oid 28 current snap is 292 2022-04-22T17:35:49.994 INFO:tasks.rados.rados.0.smithi115.stdout:2979: read oid 7 snap 290 2022-04-22T17:35:49.994 INFO:tasks.rados.rados.0.smithi115.stdout:2979: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:35:49.995 INFO:tasks.rados.rados.0.smithi115.stdout:2980: delete oid 20 current snap is 292 2022-04-22T17:35:49.995 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1659 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:35:49.999 INFO:tasks.rados.rados.0.smithi115.stdout:2978: done (2 left) 2022-04-22T17:35:49.999 INFO:tasks.rados.rados.0.smithi115.stdout:2980: done (1 left) 2022-04-22T17:35:50.000 INFO:tasks.rados.rados.0.smithi115.stdout:2981: read oid 32 snap 279 2022-04-22T17:35:50.000 INFO:tasks.rados.rados.0.smithi115.stdout:2981: expect (ObjNum 835 snap 275 seq_num 835) 2022-04-22T17:35:50.005 INFO:tasks.rados.rados.0.smithi115.stdout:2982: read oid 9 snap -1 2022-04-22T17:35:50.006 INFO:tasks.rados.rados.0.smithi115.stdout:2982: expect deleted 2022-04-22T17:35:50.006 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 292 2022-04-22T17:35:50.006 INFO:tasks.rados.rados.0.smithi115.stdout:2983: seq_num 891 ranges {1875968=532480} 2022-04-22T17:35:50.016 INFO:tasks.rados.rados.0.smithi115.stdout:2983: writing smithi11541303-12 from 1875968 to 2408448 tid 1 2022-04-22T17:35:50.016 INFO:tasks.rados.rados.0.smithi115.stdout:2984: read oid 49 snap -1 2022-04-22T17:35:50.016 INFO:tasks.rados.rados.0.smithi115.stdout:2984: expect deleted 2022-04-22T17:35:50.016 INFO:tasks.rados.rados.0.smithi115.stdout:2985: rollback oid 50 current snap is 292 2022-04-22T17:35:50.017 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 290 2022-04-22T17:35:50.017 INFO:tasks.rados.rados.0.smithi115.stdout:2979: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:35:50.262 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[59036]: b5db1e299295: Verifying Checksum 2022-04-22T17:35:50.262 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[59036]: b5db1e299295: Download complete 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[46934]: cephadm 2022-04-22T17:35:48.204562+0000 mgr.y (mgr.34107) 372 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[46934]: cephadm 2022-04-22T17:35:48.205027+0000 mgr.y (mgr.34107) 373 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[46934]: cluster 2022-04-22T17:35:48.953188+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[46934]: audit 2022-04-22T17:35:49.369187+0000 mon.a (mon.0) 1231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[46934]: cluster 2022-04-22T17:35:49.924368+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[45495]: cephadm 2022-04-22T17:35:48.204562+0000 mgr.y (mgr.34107) 372 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2022-04-22T17:35:50.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[45495]: cephadm 2022-04-22T17:35:48.205027+0000 mgr.y (mgr.34107) 373 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:35:50.264 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[45495]: cluster 2022-04-22T17:35:48.953188+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2022-04-22T17:35:50.264 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[45495]: audit 2022-04-22T17:35:49.369187+0000 mon.a (mon.0) 1231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:50.264 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:49 smithi115 bash[45495]: cluster 2022-04-22T17:35:49.924368+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2022-04-22T17:35:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[36902]: cephadm 2022-04-22T17:35:48.204562+0000 mgr.y (mgr.34107) 372 : cephadm [INF] Upgrade: Updating node-exporter.b (2/2) 2022-04-22T17:35:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[36902]: cephadm 2022-04-22T17:35:48.205027+0000 mgr.y (mgr.34107) 373 : cephadm [INF] Deploying daemon node-exporter.b on smithi121 2022-04-22T17:35:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[36902]: cluster 2022-04-22T17:35:48.953188+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2022-04-22T17:35:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[36902]: audit 2022-04-22T17:35:49.369187+0000 mon.a (mon.0) 1231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:49 smithi121 bash[36902]: cluster 2022-04-22T17:35:49.924368+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2022-04-22T17:35:50.266 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 systemd[1]: Started Ceph node-exporter.b for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:35:50.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 bash[48447]: Unable to find image 'quay.io/prometheus/node-exporter:v1.3.1' locally 2022-04-22T17:35:50.321 INFO:tasks.rados.rados.0.smithi115.stdout:2981: expect (ObjNum 835 snap 275 seq_num 835) 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2985: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2985: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2983: finishing write tid 1 to smithi11541303-12 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2983: finishing write tid 2 to smithi11541303-12 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2985: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1659 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:35:50.461 INFO:tasks.rados.rados.0.smithi115.stdout:2983: finishing write tid 3 to smithi11541303-12 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1748 (ObjNum 891 snap 292 seq_num 891) dirty exists 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2983: left oid 12 (ObjNum 891 snap 292 seq_num 891) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2979: done (5 left) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2981: done (4 left) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2982: done (3 left) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2983: done (2 left) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2984: done (1 left) 2022-04-22T17:35:50.462 INFO:tasks.rados.rados.0.smithi115.stdout:2985: done (0 left) 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2986: read oid 5 snap -1 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2986: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2987: read oid 2 snap -1 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2987: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2988: rollback oid 4 current snap is 292 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 290 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:2989: rollback oid 14 current snap is 292 2022-04-22T17:35:50.463 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 279 2022-04-22T17:35:50.464 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 292 2022-04-22T17:35:50.464 INFO:tasks.rados.rados.0.smithi115.stdout:2990: seq_num 892 ranges {11182080=548864} 2022-04-22T17:35:50.514 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:50 smithi115 bash[59036]: b5db1e299295: Pull complete 2022-04-22T17:35:50.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:50 smithi115 bash[59036]: Digest: sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd 2022-04-22T17:35:50.550 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:50 smithi115 bash[59036]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.3.1 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2990: writing smithi11541303-6 from 11182080 to 11730944 tid 1 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2991: rmattr oid 32 current snap is 292 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2991: done (5 left) 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 292 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2992: seq_num 893 ranges {327680=565248} 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2992: writing smithi11541303-38 from 327680 to 892928 tid 1 2022-04-22T17:35:50.552 INFO:tasks.rados.rados.0.smithi115.stdout:2993: setattr oid 29 current snap is 292 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:2986: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:2988: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 1668 (ObjNum 1 snap 32692 seq_num 3179397296) dirty dne 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:2989: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:2989: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1661 (ObjNum 836 snap 275 seq_num 836) dirty exists 2022-04-22T17:35:50.553 INFO:tasks.rados.rados.0.smithi115.stdout:2987: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2990: finishing write tid 1 to smithi11541303-6 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2990: finishing write tid 2 to smithi11541303-6 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2992: finishing write tid 1 to smithi11541303-38 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2992: finishing write tid 2 to smithi11541303-38 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2990: finishing write tid 3 to smithi11541303-6 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1916 (ObjNum 892 snap 292 seq_num 892) dirty exists 2022-04-22T17:35:50.674 INFO:tasks.rados.rados.0.smithi115.stdout:2990: left oid 6 (ObjNum 892 snap 292 seq_num 892) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2992: finishing write tid 3 to smithi11541303-38 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1750 (ObjNum 893 snap 292 seq_num 893) dirty exists 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2992: left oid 38 (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2986: done (6 left) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2987: done (5 left) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2988: done (4 left) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2989: done (3 left) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2990: done (2 left) 2022-04-22T17:35:50.675 INFO:tasks.rados.rados.0.smithi115.stdout:2992: done (1 left) 2022-04-22T17:35:50.676 INFO:tasks.rados.rados.0.smithi115.stdout:2994: snap_remove snap 291 2022-04-22T17:35:51.016 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 bash[48447]: v1.3.1: Pulling from prometheus/node-exporter 2022-04-22T17:35:51.016 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 bash[48447]: aa2a8d90b84c: Pulling fs layer 2022-04-22T17:35:51.017 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 bash[48447]: b45d31ee2d7f: Pulling fs layer 2022-04-22T17:35:51.017 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:50 smithi121 bash[48447]: b5db1e299295: Pulling fs layer 2022-04-22T17:35:51.132 INFO:tasks.rados.rados.0.smithi115.stdout:2994: done (1 left) 2022-04-22T17:35:51.138 INFO:tasks.rados.rados.0.smithi115.stdout:2995: snap_remove snap 279 2022-04-22T17:35:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:51 smithi121 bash[36902]: cluster 2022-04-22T17:35:49.318306+0000 mgr.y (mgr.34107) 374 : cluster [DBG] pgmap v666: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:35:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:51 smithi121 bash[36902]: audit 2022-04-22T17:35:50.109866+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:51 smithi121 bash[36902]: audit 2022-04-22T17:35:50.111274+0000 mon.a (mon.0) 1234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:51 smithi121 bash[36902]: audit 2022-04-22T17:35:50.112873+0000 mon.a (mon.0) 1235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:51 smithi121 bash[36902]: audit 2022-04-22T17:35:50.113806+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[46934]: cluster 2022-04-22T17:35:49.318306+0000 mgr.y (mgr.34107) 374 : cluster [DBG] pgmap v666: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[46934]: audit 2022-04-22T17:35:50.109866+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[46934]: audit 2022-04-22T17:35:50.111274+0000 mon.a (mon.0) 1234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[46934]: audit 2022-04-22T17:35:50.112873+0000 mon.a (mon.0) 1235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[46934]: audit 2022-04-22T17:35:50.113806+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[45495]: cluster 2022-04-22T17:35:49.318306+0000 mgr.y (mgr.34107) 374 : cluster [DBG] pgmap v666: 121 pgs: 2 active+clean+snaptrim, 119 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[45495]: audit 2022-04-22T17:35:50.109866+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[45495]: audit 2022-04-22T17:35:50.111274+0000 mon.a (mon.0) 1234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:35:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[45495]: audit 2022-04-22T17:35:50.112873+0000 mon.a (mon.0) 1235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:35:51.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[45495]: audit 2022-04-22T17:35:50.113806+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:35:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:51.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:51.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:51.633Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:35:52.146 INFO:tasks.rados.rados.0.smithi115.stdout:2995: done (1 left) 2022-04-22T17:35:52.147 INFO:tasks.rados.rados.0.smithi115.stdout:2996: copy_from oid 30 from oid 6 current snap is 292 2022-04-22T17:35:52.147 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1662 (ObjNum -1116199664 snap 22075 seq_num 3180199648) dirty exists 2022-04-22T17:35:52.147 INFO:tasks.rados.rados.0.smithi115.stdout:2993: done (1 left) 2022-04-22T17:35:52.148 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 292 2022-04-22T17:35:52.148 INFO:tasks.rados.rados.0.smithi115.stdout:2997: seq_num 894 ranges {0=581632} 2022-04-22T17:35:52.151 INFO:tasks.rados.rados.0.smithi115.stdout:2997: writing smithi11541303-37 from 0 to 581632 tid 1 2022-04-22T17:35:52.151 INFO:tasks.rados.rados.0.smithi115.stdout:2998: snap_create 2022-04-22T17:35:52.168 INFO:tasks.rados.rados.0.smithi115.stdout:2997: finishing write tid 1 to smithi11541303-37 2022-04-22T17:35:52.169 INFO:tasks.rados.rados.0.smithi115.stdout:2997: finishing write tid 2 to smithi11541303-37 2022-04-22T17:35:52.175 INFO:tasks.rados.rados.0.smithi115.stdout:2997: finishing write tid 3 to smithi11541303-37 2022-04-22T17:35:52.175 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1842 (ObjNum 894 snap 292 seq_num 894) dirty exists 2022-04-22T17:35:52.175 INFO:tasks.rados.rados.0.smithi115.stdout:2997: left oid 37 (ObjNum 894 snap 292 seq_num 894) 2022-04-22T17:35:52.224 INFO:tasks.rados.rados.0.smithi115.stdout:2996: finishing copy_from to smithi11541303-30 2022-04-22T17:35:52.224 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1916 (ObjNum 892 snap 292 seq_num 892) dirty exists 2022-04-22T17:35:52.226 INFO:tasks.rados.rados.0.smithi115.stdout:2996: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:35:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[45495]: cluster 2022-04-22T17:35:51.118694+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2022-04-22T17:35:52.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[46934]: cluster 2022-04-22T17:35:51.118694+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2022-04-22T17:35:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:52 smithi121 bash[36902]: cluster 2022-04-22T17:35:51.118694+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2022-04-22T17:35:52.800 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.485Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2022-04-22T17:35:52.800 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.485Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.485Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.486Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=arp 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=bcache 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=bonding 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=btrfs 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=conntrack 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=cpu 2022-04-22T17:35:52.801 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=cpufreq 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=diskstats 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=dmi 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=edac 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=entropy 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=fibrechannel 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=filefd 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=filesystem 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=hwmon 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=infiniband 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=ipvs 2022-04-22T17:35:52.802 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=loadavg 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=mdadm 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=meminfo 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=netclass 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=netdev 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=netstat 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=nfs 2022-04-22T17:35:52.803 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=nfsd 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=nvme 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=os 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=pressure 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=rapl 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=schedstat 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=sockstat 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=softnet 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=stat 2022-04-22T17:35:52.804 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=tapestats 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=textfile 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=thermal_zone 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=time 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=udp_queues 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=uname 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=vmstat 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=xfs 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:115 level=info collector=zfs 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2022-04-22T17:35:52.805 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Apr 22 17:35:52 smithi115 bash[59036]: ts=2022-04-22T17:35:52.488Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2022-04-22T17:35:53.149 INFO:tasks.rados.rados.0.smithi115.stdout:2996: done (2 left) 2022-04-22T17:35:53.149 INFO:tasks.rados.rados.0.smithi115.stdout:2997: done (1 left) 2022-04-22T17:35:53.149 INFO:tasks.rados.rados.0.smithi115.stdout:2998: done (0 left) 2022-04-22T17:35:53.149 INFO:tasks.rados.rados.0.smithi115.stdout:2999: snap_remove snap 292 2022-04-22T17:35:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[36902]: cluster 2022-04-22T17:35:51.319002+0000 mgr.y (mgr.34107) 375 : cluster [DBG] pgmap v669: 121 pgs: 121 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 758 KiB/s wr, 21 op/s 2022-04-22T17:35:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[36902]: cluster 2022-04-22T17:35:52.133234+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2022-04-22T17:35:53.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:52 smithi121 bash[48447]: b45d31ee2d7f: Verifying Checksum 2022-04-22T17:35:53.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:52 smithi121 bash[48447]: b45d31ee2d7f: Download complete 2022-04-22T17:35:53.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[48447]: aa2a8d90b84c: Verifying Checksum 2022-04-22T17:35:53.267 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[48447]: aa2a8d90b84c: Download complete 2022-04-22T17:35:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[46934]: cluster 2022-04-22T17:35:51.319002+0000 mgr.y (mgr.34107) 375 : cluster [DBG] pgmap v669: 121 pgs: 121 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 758 KiB/s wr, 21 op/s 2022-04-22T17:35:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[46934]: cluster 2022-04-22T17:35:52.133234+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2022-04-22T17:35:53.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[45495]: cluster 2022-04-22T17:35:51.319002+0000 mgr.y (mgr.34107) 375 : cluster [DBG] pgmap v669: 121 pgs: 121 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 758 KiB/s wr, 21 op/s 2022-04-22T17:35:53.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[45495]: cluster 2022-04-22T17:35:52.133234+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2022-04-22T17:35:53.687 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[48447]: aa2a8d90b84c: Pull complete 2022-04-22T17:35:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:53.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:53.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:35:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:35:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:35:53.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:35:54.016 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:53 smithi121 bash[48447]: b45d31ee2d7f: Pull complete 2022-04-22T17:35:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:2999: done (0 left) 2022-04-22T17:35:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3000: copy_from oid 16 from oid 21 current snap is 293 2022-04-22T17:35:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3001: rmattr oid 13 current snap is 293 2022-04-22T17:35:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3002: delete oid 42 current snap is 293 2022-04-22T17:35:54.172 INFO:tasks.rados.rados.0.smithi115.stdout:3002: done (2 left) 2022-04-22T17:35:54.172 INFO:tasks.rados.rados.0.smithi115.stdout:3003: snap_remove snap 290 2022-04-22T17:35:54.455 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:54 smithi121 bash[36902]: cluster 2022-04-22T17:35:53.135270+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2022-04-22T17:35:54.456 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:54 smithi121 bash[48447]: b5db1e299295: Download complete 2022-04-22T17:35:54.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:54 smithi115 bash[45495]: cluster 2022-04-22T17:35:53.135270+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2022-04-22T17:35:54.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:54 smithi115 bash[46934]: cluster 2022-04-22T17:35:53.135270+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2022-04-22T17:35:54.760 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:54 smithi121 bash[48447]: b5db1e299295: Pull complete 2022-04-22T17:35:54.790 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:54 smithi121 bash[48447]: Digest: sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd 2022-04-22T17:35:54.791 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:54 smithi121 bash[48447]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.3.1 2022-04-22T17:35:54.945 INFO:tasks.rados.rados.0.smithi115.stdout:3003: done (2 left) 2022-04-22T17:35:54.945 INFO:tasks.rados.rados.0.smithi115.stdout:3004: rollback oid 49 current snap is 293 2022-04-22T17:35:54.945 INFO:tasks.rados.rados.0.smithi115.stdout:3004: done (2 left) 2022-04-22T17:35:54.946 INFO:tasks.rados.rados.0.smithi115.stdout:3005: snap_create 2022-04-22T17:35:54.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1690 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:35:54.946 INFO:tasks.rados.rados.0.smithi115.stdout:3000: finishing copy_from to smithi11541303-16 2022-04-22T17:35:54.946 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1159 (ObjNum 886 snap 289 seq_num 886) dirty exists 2022-04-22T17:35:54.946 INFO:tasks.rados.rados.0.smithi115.stdout:3000: finishing copy_from racing read to smithi11541303-16 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: cluster 2022-04-22T17:35:53.319562+0000 mgr.y (mgr.34107) 376 : cluster [DBG] pgmap v672: 121 pgs: 121 active+clean; 196 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 622 KiB/s wr, 30 op/s 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: cluster 2022-04-22T17:35:54.145720+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: audit 2022-04-22T17:35:54.322672+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: audit 2022-04-22T17:35:54.371735+0000 mon.a (mon.0) 1242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: audit 2022-04-22T17:35:54.457530+0000 mon.a (mon.0) 1243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: audit 2022-04-22T17:35:54.920739+0000 mon.a (mon.0) 1244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: cluster 2022-04-22T17:35:54.933117+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2022-04-22T17:35:55.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[45495]: audit 2022-04-22T17:35:55.058486+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: cluster 2022-04-22T17:35:53.319562+0000 mgr.y (mgr.34107) 376 : cluster [DBG] pgmap v672: 121 pgs: 121 active+clean; 196 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 622 KiB/s wr, 30 op/s 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: cluster 2022-04-22T17:35:54.145720+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: audit 2022-04-22T17:35:54.322672+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: audit 2022-04-22T17:35:54.371735+0000 mon.a (mon.0) 1242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: audit 2022-04-22T17:35:54.457530+0000 mon.a (mon.0) 1243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: audit 2022-04-22T17:35:54.920739+0000 mon.a (mon.0) 1244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: cluster 2022-04-22T17:35:54.933117+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2022-04-22T17:35:55.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:55 smithi115 bash[46934]: audit 2022-04-22T17:35:55.058486+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: cluster 2022-04-22T17:35:53.319562+0000 mgr.y (mgr.34107) 376 : cluster [DBG] pgmap v672: 121 pgs: 121 active+clean; 196 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 622 KiB/s wr, 30 op/s 2022-04-22T17:35:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: cluster 2022-04-22T17:35:54.145720+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2022-04-22T17:35:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: audit 2022-04-22T17:35:54.322672+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: audit 2022-04-22T17:35:54.371735+0000 mon.a (mon.0) 1242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: audit 2022-04-22T17:35:54.457530+0000 mon.a (mon.0) 1243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: audit 2022-04-22T17:35:54.920739+0000 mon.a (mon.0) 1244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: cluster 2022-04-22T17:35:54.933117+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2022-04-22T17:35:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:55 smithi121 bash[36902]: audit 2022-04-22T17:35:55.058486+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:35:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:3000: done (2 left) 2022-04-22T17:35:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:3001: done (1 left) 2022-04-22T17:35:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:3005: done (0 left) 2022-04-22T17:35:55.946 INFO:tasks.rados.rados.0.smithi115.stdout:3006: setattr oid 38 current snap is 294 2022-04-22T17:35:55.950 INFO:tasks.rados.rados.0.smithi115.stdout:3007: copy_from oid 31 from oid 29 current snap is 294 2022-04-22T17:35:55.950 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 294 2022-04-22T17:35:55.950 INFO:tasks.rados.rados.0.smithi115.stdout:3008: seq_num 896 ranges {11730944=614400} 2022-04-22T17:35:56.025 INFO:tasks.rados.rados.0.smithi115.stdout:3008: writing smithi11541303-30 from 11730944 to 12345344 tid 1 2022-04-22T17:35:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:3009: copy_from oid 22 from oid 7 current snap is 294 2022-04-22T17:35:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1751 (ObjNum 893 snap 292 seq_num 893) dirty exists 2022-04-22T17:35:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:3006: done (3 left) 2022-04-22T17:35:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 294 2022-04-22T17:35:56.029 INFO:tasks.rados.rados.0.smithi115.stdout:3010: seq_num 897 ranges {0=630784} 2022-04-22T17:35:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:3010: writing smithi11541303-39 from 0 to 630784 tid 1 2022-04-22T17:35:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 294 2022-04-22T17:35:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:3007: finishing copy_from to smithi11541303-31 2022-04-22T17:35:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 1662 (ObjNum 0 snap 0 seq_num 0) dirty exists 2022-04-22T17:35:56.030 INFO:tasks.rados.rados.0.smithi115.stdout:3011: seq_num 898 ranges {1499136=647168} 2022-04-22T17:35:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:3011: writing smithi11541303-32 from 1499136 to 2146304 tid 1 2022-04-22T17:35:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:3012: read oid 19 snap -1 2022-04-22T17:35:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:3012: expect (ObjNum 873 snap 286 seq_num 873) 2022-04-22T17:35:56.042 INFO:tasks.rados.rados.0.smithi115.stdout:3013: delete oid 6 current snap is 294 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3007: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3010: finishing write tid 1 to smithi11541303-39 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3010: finishing write tid 2 to smithi11541303-39 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3008: finishing write tid 1 to smithi11541303-30 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3008: finishing write tid 2 to smithi11541303-30 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3008: finishing write tid 3 to smithi11541303-30 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1918 (ObjNum 896 snap 294 seq_num 896) dirty exists 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3008: left oid 30 (ObjNum 896 snap 294 seq_num 896) 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:3010: finishing write tid 3 to smithi11541303-39 2022-04-22T17:35:56.043 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1844 (ObjNum 897 snap 294 seq_num 897) dirty exists 2022-04-22T17:35:56.044 INFO:tasks.rados.rados.0.smithi115.stdout:3010: left oid 39 (ObjNum 897 snap 294 seq_num 897) 2022-04-22T17:35:56.051 INFO:tasks.rados.rados.0.smithi115.stdout:3012: expect (ObjNum 873 snap 286 seq_num 873) 2022-04-22T17:35:56.055 INFO:tasks.rados.rados.0.smithi115.stdout:3007: done (6 left) 2022-04-22T17:35:56.055 INFO:tasks.rados.rados.0.smithi115.stdout:3008: done (5 left) 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3010: done (4 left) 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3012: done (3 left) 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3013: done (2 left) 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3014: read oid 6 snap -1 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3014: expect deleted 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3015: read oid 33 snap -1 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3015: expect deleted 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3016: rollback oid 13 current snap is 294 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 293 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3011: finishing write tid 1 to smithi11541303-32 2022-04-22T17:35:56.056 INFO:tasks.rados.rados.0.smithi115.stdout:3011: finishing write tid 2 to smithi11541303-32 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3009: finishing copy_from to smithi11541303-22 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1250 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3011: finishing write tid 3 to smithi11541303-32 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1379 (ObjNum 898 snap 294 seq_num 898) dirty exists 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3011: left oid 32 (ObjNum 898 snap 294 seq_num 898) 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3009: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3009: done (4 left) 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3011: done (3 left) 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3017: rmattr oid 11 current snap is 294 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3017: done (3 left) 2022-04-22T17:35:56.057 INFO:tasks.rados.rados.0.smithi115.stdout:3018: read oid 29 snap -1 2022-04-22T17:35:56.058 INFO:tasks.rados.rados.0.smithi115.stdout:3018: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:56.058 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 294 2022-04-22T17:35:56.058 INFO:tasks.rados.rados.0.smithi115.stdout:3019: seq_num 900 ranges {581632=679936} 2022-04-22T17:35:56.063 INFO:tasks.rados.rados.0.smithi115.stdout:3019: writing smithi11541303-37 from 581632 to 1261568 tid 1 2022-04-22T17:35:56.063 INFO:tasks.rados.rados.0.smithi115.stdout:3020: rmattr oid 26 current snap is 294 2022-04-22T17:35:56.063 INFO:tasks.rados.rados.0.smithi115.stdout:3018: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:56.070 INFO:tasks.rados.rados.0.smithi115.stdout:3016: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:35:56.070 INFO:tasks.rados.rados.0.smithi115.stdout:3016: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:35:56.070 INFO:tasks.rados.rados.0.smithi115.stdout:3016: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1698 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3014: done (5 left) 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3015: done (4 left) 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3016: done (3 left) 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3018: done (2 left) 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3021: snap_create 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1380 (ObjNum 854 snap 280 seq_num 854) dirty exists 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3019: finishing write tid 1 to smithi11541303-37 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3019: finishing write tid 2 to smithi11541303-37 2022-04-22T17:35:56.071 INFO:tasks.rados.rados.0.smithi115.stdout:3019: finishing write tid 3 to smithi11541303-37 2022-04-22T17:35:56.072 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1846 (ObjNum 900 snap 294 seq_num 900) dirty exists 2022-04-22T17:35:56.072 INFO:tasks.rados.rados.0.smithi115.stdout:3019: left oid 37 (ObjNum 900 snap 294 seq_num 900) 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.540Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.540Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.542Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.542Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2022-04-22T17:35:56.767 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=arp 2022-04-22T17:35:56.768 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=bcache 2022-04-22T17:35:56.768 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=bonding 2022-04-22T17:35:56.768 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=btrfs 2022-04-22T17:35:56.768 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=conntrack 2022-04-22T17:35:56.768 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=cpu 2022-04-22T17:35:56.769 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=cpufreq 2022-04-22T17:35:56.769 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=diskstats 2022-04-22T17:35:56.769 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=dmi 2022-04-22T17:35:56.769 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=edac 2022-04-22T17:35:56.769 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=entropy 2022-04-22T17:35:56.770 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=fibrechannel 2022-04-22T17:35:56.770 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=filefd 2022-04-22T17:35:56.770 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=filesystem 2022-04-22T17:35:56.770 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=hwmon 2022-04-22T17:35:56.770 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=infiniband 2022-04-22T17:35:56.771 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=ipvs 2022-04-22T17:35:56.771 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=loadavg 2022-04-22T17:35:56.771 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=mdadm 2022-04-22T17:35:56.771 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=meminfo 2022-04-22T17:35:56.771 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=netclass 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=netdev 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=netstat 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=nfs 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=nfsd 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=nvme 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=os 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=pressure 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=rapl 2022-04-22T17:35:56.772 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=schedstat 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=sockstat 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=softnet 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=stat 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=tapestats 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=textfile 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=thermal_zone 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=time 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=udp_queues 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=uname 2022-04-22T17:35:56.773 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=vmstat 2022-04-22T17:35:56.774 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=xfs 2022-04-22T17:35:56.774 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:115 level=info collector=zfs 2022-04-22T17:35:56.774 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2022-04-22T17:35:56.774 INFO:journalctl@ceph.node-exporter.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[48447]: ts=2022-04-22T17:35:56.543Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2022-04-22T17:35:56.953 INFO:tasks.rados.rados.0.smithi115.stdout:3019: done (2 left) 2022-04-22T17:35:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:3020: done (1 left) 2022-04-22T17:35:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:3021: done (0 left) 2022-04-22T17:35:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 295 2022-04-22T17:35:56.954 INFO:tasks.rados.rados.0.smithi115.stdout:3022: seq_num 901 ranges {4087808=696320} 2022-04-22T17:35:56.984 INFO:tasks.rados.rados.0.smithi115.stdout:3022: writing smithi11541303-7 from 4087808 to 4784128 tid 1 2022-04-22T17:35:56.985 INFO:tasks.rados.rados.0.smithi115.stdout:3023: rollback oid 44 current snap is 295 2022-04-22T17:35:56.985 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 44 to 293 2022-04-22T17:35:56.985 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 295 2022-04-22T17:35:56.985 INFO:tasks.rados.rados.0.smithi115.stdout:3024: seq_num 902 ranges {0=712704} 2022-04-22T17:35:56.989 INFO:tasks.rados.rados.0.smithi115.stdout:3024: writing smithi11541303-42 from 0 to 712704 tid 1 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3025: rollback oid 9 current snap is 295 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 293 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3023: finishing rollback tid 1 to smithi11541303-44 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 0 (ObjNum -1116234048 snap 22075 seq_num 0) dirty dne 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3023: done (3 left) 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3026: read oid 28 snap -1 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3026: expect (ObjNum 846 snap 279 seq_num 846) 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3027: read oid 2 snap -1 2022-04-22T17:35:56.990 INFO:tasks.rados.rados.0.smithi115.stdout:3027: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:35:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:3028: rollback oid 19 current snap is 295 2022-04-22T17:35:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 294 2022-04-22T17:35:56.991 INFO:tasks.rados.rados.0.smithi115.stdout:3029: setattr oid 10 current snap is 295 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:3030: copy_from oid 13 from oid 6 current snap is 295 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:3031: rollback oid 23 current snap is 295 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 294 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:3032: read oid 36 snap -1 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:3032: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:56.992 INFO:tasks.rados.rados.0.smithi115.stdout:3033: snap_create 2022-04-22T17:35:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:3031: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:35:56.993 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 0 (ObjNum 288 snap 0 seq_num 33) dirty dne 2022-04-22T17:35:56.995 INFO:tasks.rados.rados.0.smithi115.stdout:3032: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:35:57.000 INFO:tasks.rados.rados.0.smithi115.stdout:3028: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:35:57.000 INFO:tasks.rados.rados.0.smithi115.stdout:3022: finishing write tid 1 to smithi11541303-7 2022-04-22T17:35:57.002 INFO:tasks.rados.rados.0.smithi115.stdout:3030: finishing copy_from to smithi11541303-13 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:3030: got expected ENOENT (src dne) 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:3022: finishing write tid 2 to smithi11541303-7 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:3028: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:3028: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1754 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:35:57.003 INFO:tasks.rados.rados.0.smithi115.stdout:3030: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:35:57.004 INFO:tasks.rados.rados.0.smithi115.stdout:3025: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:35:57.004 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 0 (ObjNum -1115643824 snap 22075 seq_num 0) dirty dne 2022-04-22T17:35:57.005 INFO:tasks.rados.rados.0.smithi115.stdout:3022: finishing write tid 3 to smithi11541303-7 2022-04-22T17:35:57.005 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1838 (ObjNum 901 snap 295 seq_num 901) dirty exists 2022-04-22T17:35:57.005 INFO:tasks.rados.rados.0.smithi115.stdout:3022: left oid 7 (ObjNum 901 snap 295 seq_num 901) 2022-04-22T17:35:57.005 INFO:tasks.rados.rados.0.smithi115.stdout:3024: finishing write tid 1 to smithi11541303-42 2022-04-22T17:35:57.005 INFO:tasks.rados.rados.0.smithi115.stdout:3024: finishing write tid 2 to smithi11541303-42 2022-04-22T17:35:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:3024: finishing write tid 3 to smithi11541303-42 2022-04-22T17:35:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1995 (ObjNum 902 snap 295 seq_num 902) dirty exists 2022-04-22T17:35:57.007 INFO:tasks.rados.rados.0.smithi115.stdout:3024: left oid 42 (ObjNum 902 snap 295 seq_num 902) 2022-04-22T17:35:57.008 INFO:tasks.rados.rados.0.smithi115.stdout:3026: expect (ObjNum 846 snap 279 seq_num 846) 2022-04-22T17:35:57.152 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:35:57.185 INFO:tasks.rados.rados.0.smithi115.stdout:3027: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:35:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[36902]: cluster 2022-04-22T17:35:55.320415+0000 mgr.y (mgr.34107) 377 : cluster [DBG] pgmap v675: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2022-04-22T17:35:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:56 smithi121 bash[36902]: cluster 2022-04-22T17:35:55.931999+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2022-04-22T17:35:57.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:56 smithi115 bash[45495]: cluster 2022-04-22T17:35:55.320415+0000 mgr.y (mgr.34107) 377 : cluster [DBG] pgmap v675: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2022-04-22T17:35:57.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:56 smithi115 bash[45495]: cluster 2022-04-22T17:35:55.931999+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2022-04-22T17:35:57.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:56 smithi115 bash[46934]: cluster 2022-04-22T17:35:55.320415+0000 mgr.y (mgr.34107) 377 : cluster [DBG] pgmap v675: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2022-04-22T17:35:57.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:56 smithi115 bash[46934]: cluster 2022-04-22T17:35:55.931999+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2022-04-22T17:35:57.319 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1924 (ObjNum 2 snap 0 seq_num 0) dirty exists 2022-04-22T17:35:57.501 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:35:57.501 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (12m) 3s ago 12m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:35:57.502 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (12m) 3s ago 12m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:35:57.502 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (59s) 3s ago 13m 12.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:35:57.502 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (31s) 3s ago 13m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:35:57.502 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (8m) 3s ago 18m 405M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:35:57.502 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (8m) 3s ago 20m 452M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (7m) 3s ago 20m 82.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6m) 3s ago 18m 61.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (7m) 3s ago 18m 66.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6s) 3s ago 13m - - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running 3s ago 13m - - 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5m) 3s ago 17m 412M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5m) 3s ago 17m 333M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4m) 3s ago 16m 301M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4m) 3s ago 16m 188M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:35:57.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3m) 3s ago 15m 258M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:35:57.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2m) 3s ago 15m 409M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:35:57.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2m) 3s ago 14m 218M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:35:57.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (84s) 3s ago 14m 261M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:35:57.504 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (9m) 3s ago 13m 40.8M - 2.18.1 de242295e225 e79ca2b3558e 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:35:57.915 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:35:57.916 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:35:57.965 INFO:tasks.rados.rados.0.smithi115.stdout:3022: done (10 left) 2022-04-22T17:35:57.965 INFO:tasks.rados.rados.0.smithi115.stdout:3024: done (9 left) 2022-04-22T17:35:57.965 INFO:tasks.rados.rados.0.smithi115.stdout:3025: done (8 left) 2022-04-22T17:35:57.966 INFO:tasks.rados.rados.0.smithi115.stdout:3026: done (7 left) 2022-04-22T17:35:57.966 INFO:tasks.rados.rados.0.smithi115.stdout:3027: done (6 left) 2022-04-22T17:35:57.966 INFO:tasks.rados.rados.0.smithi115.stdout:3028: done (5 left) 2022-04-22T17:35:57.966 INFO:tasks.rados.rados.0.smithi115.stdout:3029: done (4 left) 2022-04-22T17:35:57.966 INFO:tasks.rados.rados.0.smithi115.stdout:3030: done (3 left) 2022-04-22T17:35:57.967 INFO:tasks.rados.rados.0.smithi115.stdout:3031: done (2 left) 2022-04-22T17:35:57.967 INFO:tasks.rados.rados.0.smithi115.stdout:3032: done (1 left) 2022-04-22T17:35:57.967 INFO:tasks.rados.rados.0.smithi115.stdout:3033: done (0 left) 2022-04-22T17:35:57.967 INFO:tasks.rados.rados.0.smithi115.stdout:3034: read oid 8 snap -1 2022-04-22T17:35:57.967 INFO:tasks.rados.rados.0.smithi115.stdout:3034: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:35:57.968 INFO:tasks.rados.rados.0.smithi115.stdout:3035: read oid 43 snap -1 2022-04-22T17:35:57.968 INFO:tasks.rados.rados.0.smithi115.stdout:3035: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:35:57.968 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 296 2022-04-22T17:35:57.968 INFO:tasks.rados.rados.0.smithi115.stdout:3036: seq_num 903 ranges {4464640=729088} 2022-04-22T17:35:57.994 INFO:tasks.rados.rados.0.smithi115.stdout:3036: writing smithi11541303-26 from 4464640 to 5193728 tid 1 2022-04-22T17:35:57.994 INFO:tasks.rados.rados.0.smithi115.stdout:3037: snap_create 2022-04-22T17:35:57.994 INFO:tasks.rados.rados.0.smithi115.stdout:3034: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:35:58.096 INFO:tasks.rados.rados.0.smithi115.stdout:3035: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:35:58.202 INFO:tasks.rados.rados.0.smithi115.stdout:3036: finishing write tid 1 to smithi11541303-26 2022-04-22T17:35:58.202 INFO:tasks.rados.rados.0.smithi115.stdout:3036: finishing write tid 2 to smithi11541303-26 2022-04-22T17:35:58.202 INFO:tasks.rados.rados.0.smithi115.stdout:3036: finishing write tid 3 to smithi11541303-26 2022-04-22T17:35:58.202 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1382 (ObjNum 903 snap 296 seq_num 903) dirty exists 2022-04-22T17:35:58.202 INFO:tasks.rados.rados.0.smithi115.stdout:3036: left oid 26 (ObjNum 903 snap 296 seq_num 903) 2022-04-22T17:35:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:57 smithi121 bash[36902]: cluster 2022-04-22T17:35:56.941131+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2022-04-22T17:35:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:57 smithi121 bash[36902]: audit 2022-04-22T17:35:57.120875+0000 mgr.y (mgr.34107) 378 : audit [DBG] from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:57 smithi121 bash[36902]: audit 2022-04-22T17:35:57.902253+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/1031497556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:58.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[45495]: cluster 2022-04-22T17:35:56.941131+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2022-04-22T17:35:58.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[45495]: audit 2022-04-22T17:35:57.120875+0000 mgr.y (mgr.34107) 378 : audit [DBG] from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:58.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[45495]: audit 2022-04-22T17:35:57.902253+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/1031497556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:58.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[46934]: cluster 2022-04-22T17:35:56.941131+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2022-04-22T17:35:58.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[46934]: audit 2022-04-22T17:35:57.120875+0000 mgr.y (mgr.34107) 378 : audit [DBG] from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:58.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:57 smithi115 bash[46934]: audit 2022-04-22T17:35:57.902253+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.115:0/1031497556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:35:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:3034: done (3 left) 2022-04-22T17:35:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:3035: done (2 left) 2022-04-22T17:35:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:3036: done (1 left) 2022-04-22T17:35:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:3037: done (0 left) 2022-04-22T17:35:58.975 INFO:tasks.rados.rados.0.smithi115.stdout:3038: snap_remove snap 294 2022-04-22T17:35:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:58 smithi121 bash[36902]: cluster 2022-04-22T17:35:57.321013+0000 mgr.y (mgr.34107) 379 : cluster [DBG] pgmap v678: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 135 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:35:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:58 smithi121 bash[36902]: audit 2022-04-22T17:35:57.464918+0000 mgr.y (mgr.34107) 380 : audit [DBG] from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:59.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:58 smithi121 bash[36902]: cluster 2022-04-22T17:35:57.951419+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2022-04-22T17:35:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[45495]: cluster 2022-04-22T17:35:57.321013+0000 mgr.y (mgr.34107) 379 : cluster [DBG] pgmap v678: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 135 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:35:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[45495]: audit 2022-04-22T17:35:57.464918+0000 mgr.y (mgr.34107) 380 : audit [DBG] from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:59.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[45495]: cluster 2022-04-22T17:35:57.951419+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2022-04-22T17:35:59.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[46934]: cluster 2022-04-22T17:35:57.321013+0000 mgr.y (mgr.34107) 379 : cluster [DBG] pgmap v678: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 135 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 11 op/s 2022-04-22T17:35:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[46934]: audit 2022-04-22T17:35:57.464918+0000 mgr.y (mgr.34107) 380 : audit [DBG] from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:35:59.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:58 smithi115 bash[46934]: cluster 2022-04-22T17:35:57.951419+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2022-04-22T17:35:59.942 INFO:tasks.rados.rados.0.smithi115.stdout:3038: done (0 left) 2022-04-22T17:35:59.951 INFO:tasks.rados.rados.0.smithi115.stdout:3039: read oid 1 snap -1 2022-04-22T17:35:59.951 INFO:tasks.rados.rados.0.smithi115.stdout:3039: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:35:59.951 INFO:tasks.rados.rados.0.smithi115.stdout:3040: rollback oid 35 current snap is 297 2022-04-22T17:35:59.952 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 296 2022-04-22T17:35:59.952 INFO:tasks.rados.rados.0.smithi115.stdout:3041: snap_remove snap 293 2022-04-22T17:35:59.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[45495]: cluster 2022-04-22T17:35:58.959704+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2022-04-22T17:35:59.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[45495]: audit 2022-04-22T17:35:59.374423+0000 mon.a (mon.0) 1251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:35:59.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[45495]: cluster 2022-04-22T17:35:59.927822+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2022-04-22T17:36:00.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:59 smithi121 bash[36902]: cluster 2022-04-22T17:35:58.959704+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2022-04-22T17:36:00.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:59 smithi121 bash[36902]: audit 2022-04-22T17:35:59.374423+0000 mon.a (mon.0) 1251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:00.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:35:59 smithi121 bash[36902]: cluster 2022-04-22T17:35:59.927822+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2022-04-22T17:36:00.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[46934]: cluster 2022-04-22T17:35:58.959704+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2022-04-22T17:36:00.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[46934]: audit 2022-04-22T17:35:59.374423+0000 mon.a (mon.0) 1251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:00.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:35:59 smithi115 bash[46934]: cluster 2022-04-22T17:35:59.927822+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2022-04-22T17:36:00.944 INFO:tasks.rados.rados.0.smithi115.stdout:3041: done (2 left) 2022-04-22T17:36:00.945 INFO:tasks.rados.rados.0.smithi115.stdout:3042: rollback oid 33 current snap is 297 2022-04-22T17:36:00.945 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 296 2022-04-22T17:36:00.945 INFO:tasks.rados.rados.0.smithi115.stdout:3043: rmattr oid 22 current snap is 297 2022-04-22T17:36:00.945 INFO:tasks.rados.rados.0.smithi115.stdout:3044: snap_remove snap 295 2022-04-22T17:36:01.240 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:00 smithi121 bash[36902]: cluster 2022-04-22T17:35:59.321683+0000 mgr.y (mgr.34107) 381 : cluster [DBG] pgmap v681: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2022-04-22T17:36:01.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:00 smithi121 bash[36902]: cluster 2022-04-22T17:36:00.931341+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2022-04-22T17:36:01.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:00 smithi115 bash[45495]: cluster 2022-04-22T17:35:59.321683+0000 mgr.y (mgr.34107) 381 : cluster [DBG] pgmap v681: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2022-04-22T17:36:01.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:00 smithi115 bash[45495]: cluster 2022-04-22T17:36:00.931341+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2022-04-22T17:36:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:00 smithi115 bash[46934]: cluster 2022-04-22T17:35:59.321683+0000 mgr.y (mgr.34107) 381 : cluster [DBG] pgmap v681: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 14 op/s 2022-04-22T17:36:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:00 smithi115 bash[46934]: cluster 2022-04-22T17:36:00.931341+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2022-04-22T17:36:01.949 INFO:tasks.rados.rados.0.smithi115.stdout:3044: done (4 left) 2022-04-22T17:36:01.949 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 297 2022-04-22T17:36:01.949 INFO:tasks.rados.rados.0.smithi115.stdout:3040: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:36:01.949 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 0 (ObjNum 0 snap 0 seq_num 2415921360) dirty dne 2022-04-22T17:36:01.949 INFO:tasks.rados.rados.0.smithi115.stdout:3039: expect (ObjNum 753 snap 258 seq_num 753) 2022-04-22T17:36:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3042: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:36:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 0 (ObjNum -1116769952 snap 22075 seq_num 3178936848) dirty dne 2022-04-22T17:36:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1251 (ObjNum 736 snap 253 seq_num 736) dirty exists 2022-04-22T17:36:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3045: seq_num 904 ranges {2408448=745472} 2022-04-22T17:36:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:3045: writing smithi11541303-12 from 2408448 to 3153920 tid 1 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3039: done (4 left) 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3040: done (3 left) 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3042: done (2 left) 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3043: done (1 left) 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3046: read oid 37 snap -1 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3046: expect (ObjNum 900 snap 294 seq_num 900) 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 297 2022-04-22T17:36:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3047: seq_num 905 ranges {0=761856} 2022-04-22T17:36:02.019 INFO:tasks.rados.rados.0.smithi115.stdout:3047: writing smithi11541303-33 from 0 to 761856 tid 1 2022-04-22T17:36:02.019 INFO:tasks.rados.rados.0.smithi115.stdout:3048: rollback oid 27 current snap is 297 2022-04-22T17:36:02.019 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 296 2022-04-22T17:36:02.019 INFO:tasks.rados.rados.0.smithi115.stdout:3049: snap_create 2022-04-22T17:36:02.027 INFO:tasks.rados.rados.0.smithi115.stdout:3046: expect (ObjNum 900 snap 294 seq_num 900) 2022-04-22T17:36:02.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:01.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:01.633Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:01 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:01.633Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3045: finishing write tid 1 to smithi11541303-12 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3045: finishing write tid 2 to smithi11541303-12 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3047: finishing write tid 1 to smithi11541303-33 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3047: finishing write tid 2 to smithi11541303-33 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3048: finishing rollback tid 0 to smithi11541303-27 2022-04-22T17:36:02.077 INFO:tasks.rados.rados.0.smithi115.stdout:3047: finishing write tid 3 to smithi11541303-33 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1384 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:3047: left oid 33 (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:3048: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:3048: finishing rollback tid 2 to smithi11541303-27 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1921 (ObjNum 885 snap 289 seq_num 885) dirty exists 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:3045: finishing write tid 3 to smithi11541303-12 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1756 (ObjNum 904 snap 297 seq_num 904) dirty exists 2022-04-22T17:36:02.078 INFO:tasks.rados.rados.0.smithi115.stdout:3045: left oid 12 (ObjNum 904 snap 297 seq_num 904) 2022-04-22T17:36:02.989 INFO:tasks.rados.rados.0.smithi115.stdout:3045: done (4 left) 2022-04-22T17:36:02.990 INFO:tasks.rados.rados.0.smithi115.stdout:3046: done (3 left) 2022-04-22T17:36:02.990 INFO:tasks.rados.rados.0.smithi115.stdout:3047: done (2 left) 2022-04-22T17:36:02.990 INFO:tasks.rados.rados.0.smithi115.stdout:3048: done (1 left) 2022-04-22T17:36:02.990 INFO:tasks.rados.rados.0.smithi115.stdout:3049: done (0 left) 2022-04-22T17:36:02.990 INFO:tasks.rados.rados.0.smithi115.stdout:3050: snap_remove snap 297 2022-04-22T17:36:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:02 smithi121 bash[36902]: cluster 2022-04-22T17:36:01.322475+0000 mgr.y (mgr.34107) 382 : cluster [DBG] pgmap v684: 121 pgs: 121 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2022-04-22T17:36:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:02 smithi121 bash[36902]: cluster 2022-04-22T17:36:01.936155+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2022-04-22T17:36:03.288 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:02 smithi115 bash[45495]: cluster 2022-04-22T17:36:01.322475+0000 mgr.y (mgr.34107) 382 : cluster [DBG] pgmap v684: 121 pgs: 121 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2022-04-22T17:36:03.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:02 smithi115 bash[45495]: cluster 2022-04-22T17:36:01.936155+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2022-04-22T17:36:03.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:02 smithi115 bash[46934]: cluster 2022-04-22T17:36:01.322475+0000 mgr.y (mgr.34107) 382 : cluster [DBG] pgmap v684: 121 pgs: 121 active+clean; 128 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2022-04-22T17:36:03.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:02 smithi115 bash[46934]: cluster 2022-04-22T17:36:01.936155+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2022-04-22T17:36:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:03.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:03.806 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:03.528Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:03.806 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:03 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:03.528Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:03.995 INFO:tasks.rados.rados.0.smithi115.stdout:3050: done (0 left) 2022-04-22T17:36:03.995 INFO:tasks.rados.rados.0.smithi115.stdout:3051: delete oid 16 current snap is 298 2022-04-22T17:36:04.005 INFO:tasks.rados.rados.0.smithi115.stdout:3051: done (0 left) 2022-04-22T17:36:04.005 INFO:tasks.rados.rados.0.smithi115.stdout:3052: read oid 10 snap -1 2022-04-22T17:36:04.005 INFO:tasks.rados.rados.0.smithi115.stdout:3052: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:04.005 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 298 2022-04-22T17:36:04.005 INFO:tasks.rados.rados.0.smithi115.stdout:3053: seq_num 907 ranges {3121152=794624} 2022-04-22T17:36:04.026 INFO:tasks.rados.rados.0.smithi115.stdout:3053: writing smithi11541303-18 from 3121152 to 3915776 tid 1 2022-04-22T17:36:04.068 INFO:tasks.rados.rados.0.smithi115.stdout:3054: snap_create 2022-04-22T17:36:04.068 INFO:tasks.rados.rados.0.smithi115.stdout:3052: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:3053: finishing write tid 1 to smithi11541303-18 2022-04-22T17:36:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:3053: finishing write tid 2 to smithi11541303-18 2022-04-22T17:36:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:3053: finishing write tid 3 to smithi11541303-18 2022-04-22T17:36:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2000 (ObjNum 907 snap 298 seq_num 907) dirty exists 2022-04-22T17:36:04.069 INFO:tasks.rados.rados.0.smithi115.stdout:3053: left oid 18 (ObjNum 907 snap 298 seq_num 907) 2022-04-22T17:36:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:03 smithi121 bash[36902]: cluster 2022-04-22T17:36:02.975856+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2022-04-22T17:36:04.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:03 smithi115 bash[45495]: cluster 2022-04-22T17:36:02.975856+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2022-04-22T17:36:04.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:03 smithi115 bash[46934]: cluster 2022-04-22T17:36:02.975856+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2022-04-22T17:36:04.944 INFO:tasks.rados.rados.0.smithi115.stdout:3052: done (2 left) 2022-04-22T17:36:04.944 INFO:tasks.rados.rados.0.smithi115.stdout:3053: done (1 left) 2022-04-22T17:36:04.944 INFO:tasks.rados.rados.0.smithi115.stdout:3054: done (0 left) 2022-04-22T17:36:04.944 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 299 2022-04-22T17:36:04.944 INFO:tasks.rados.rados.0.smithi115.stdout:3055: seq_num 908 ranges {0=409600,409600=401408} 2022-04-22T17:36:04.946 INFO:tasks.rados.rados.0.smithi115.stdout:3055: writing smithi11541303-10 from 0 to 409600 tid 1 2022-04-22T17:36:04.950 INFO:tasks.rados.rados.0.smithi115.stdout:3055: writing smithi11541303-10 from 409600 to 811008 tid 2 2022-04-22T17:36:04.950 INFO:tasks.rados.rados.0.smithi115.stdout:3056: read oid 15 snap -1 2022-04-22T17:36:04.950 INFO:tasks.rados.rados.0.smithi115.stdout:3056: expect (ObjNum 838 snap 276 seq_num 838) 2022-04-22T17:36:04.950 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 299 2022-04-22T17:36:04.951 INFO:tasks.rados.rados.0.smithi115.stdout:3057: seq_num 909 ranges {0=425984,425984=401408} 2022-04-22T17:36:04.953 INFO:tasks.rados.rados.0.smithi115.stdout:3057: writing smithi11541303-31 from 0 to 425984 tid 1 2022-04-22T17:36:04.956 INFO:tasks.rados.rados.0.smithi115.stdout:3057: writing smithi11541303-31 from 425984 to 827392 tid 2 2022-04-22T17:36:04.956 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 299 2022-04-22T17:36:04.956 INFO:tasks.rados.rados.0.smithi115.stdout:3058: seq_num 910 ranges {0=442368,442368=401408} 2022-04-22T17:36:04.959 INFO:tasks.rados.rados.0.smithi115.stdout:3058: writing smithi11541303-11 from 0 to 442368 tid 1 2022-04-22T17:36:04.962 INFO:tasks.rados.rados.0.smithi115.stdout:3058: writing smithi11541303-11 from 442368 to 843776 tid 2 2022-04-22T17:36:04.962 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 299 2022-04-22T17:36:04.962 INFO:tasks.rados.rados.0.smithi115.stdout:3059: seq_num 911 ranges {4120576=466944,4587520=401408} 2022-04-22T17:36:04.991 INFO:tasks.rados.rados.0.smithi115.stdout:3059: writing smithi11541303-45 from 4120576 to 4587520 tid 1 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:3059: writing smithi11541303-45 from 4587520 to 4988928 tid 2 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:3060: rollback oid 13 current snap is 299 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 13 to 296 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:3055: finishing write tid 1 to smithi11541303-10 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:3055: finishing write tid 2 to smithi11541303-10 2022-04-22T17:36:05.002 INFO:tasks.rados.rados.0.smithi115.stdout:3055: finishing write tid 3 to smithi11541303-10 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3057: finishing write tid 1 to smithi11541303-31 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3057: finishing write tid 2 to smithi11541303-31 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3058: finishing write tid 1 to smithi11541303-11 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3058: finishing write tid 2 to smithi11541303-11 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3055: finishing write tid 4 to smithi11541303-10 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1927 (ObjNum 908 snap 299 seq_num 908) dirty exists 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3055: left oid 10 (ObjNum 908 snap 299 seq_num 908) 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3057: finishing write tid 3 to smithi11541303-31 2022-04-22T17:36:05.003 INFO:tasks.rados.rados.0.smithi115.stdout:3057: finishing write tid 4 to smithi11541303-31 2022-04-22T17:36:05.004 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2004 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:36:05.004 INFO:tasks.rados.rados.0.smithi115.stdout:3057: left oid 31 (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:36:05.004 INFO:tasks.rados.rados.0.smithi115.stdout:3058: finishing write tid 3 to smithi11541303-11 2022-04-22T17:36:05.004 INFO:tasks.rados.rados.0.smithi115.stdout:3056: expect (ObjNum 838 snap 276 seq_num 838) 2022-04-22T17:36:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: cluster 2022-04-22T17:36:03.323343+0000 mgr.y (mgr.34107) 383 : cluster [DBG] pgmap v687: 121 pgs: 121 active+clean; 118 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 830 KiB/s wr, 17 op/s 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: cluster 2022-04-22T17:36:03.982498+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.213651+0000 mon.a (mon.0) 1257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.378011+0000 mon.a (mon.0) 1258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.744235+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.750452+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.761299+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.764208+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.765965+0000 mon.a (mon.0) 1263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.767301+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.769142+0000 mon.a (mon.0) 1265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.771238+0000 mon.a (mon.0) 1266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.772605+0000 mon.a (mon.0) 1267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.773904+0000 mon.a (mon.0) 1268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.775238+0000 mon.a (mon.0) 1269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.776548+0000 mon.a (mon.0) 1270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.778044+0000 mon.a (mon.0) 1271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: audit 2022-04-22T17:36:04.779574+0000 mon.a (mon.0) 1272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:04 smithi121 bash[36902]: cluster 2022-04-22T17:36:04.929595+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2022-04-22T17:36:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: cluster 2022-04-22T17:36:03.323343+0000 mgr.y (mgr.34107) 383 : cluster [DBG] pgmap v687: 121 pgs: 121 active+clean; 118 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 830 KiB/s wr, 17 op/s 2022-04-22T17:36:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: cluster 2022-04-22T17:36:03.982498+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2022-04-22T17:36:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.213651+0000 mon.a (mon.0) 1257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.378011+0000 mon.a (mon.0) 1258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.744235+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.750452+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.761299+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.764208+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.765965+0000 mon.a (mon.0) 1263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.767301+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.769142+0000 mon.a (mon.0) 1265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.771238+0000 mon.a (mon.0) 1266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.772605+0000 mon.a (mon.0) 1267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.773904+0000 mon.a (mon.0) 1268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.775238+0000 mon.a (mon.0) 1269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.776548+0000 mon.a (mon.0) 1270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.778044+0000 mon.a (mon.0) 1271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: audit 2022-04-22T17:36:04.779574+0000 mon.a (mon.0) 1272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[45495]: cluster 2022-04-22T17:36:04.929595+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2022-04-22T17:36:05.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: cluster 2022-04-22T17:36:03.323343+0000 mgr.y (mgr.34107) 383 : cluster [DBG] pgmap v687: 121 pgs: 121 active+clean; 118 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 830 KiB/s wr, 17 op/s 2022-04-22T17:36:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: cluster 2022-04-22T17:36:03.982498+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2022-04-22T17:36:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.213651+0000 mon.a (mon.0) 1257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.378011+0000 mon.a (mon.0) 1258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.744235+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.750452+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.761299+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.764208+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.765965+0000 mon.a (mon.0) 1263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.767301+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.769142+0000 mon.a (mon.0) 1265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.771238+0000 mon.a (mon.0) 1266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.772605+0000 mon.a (mon.0) 1267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.773904+0000 mon.a (mon.0) 1268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.775238+0000 mon.a (mon.0) 1269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.776548+0000 mon.a (mon.0) 1270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.778044+0000 mon.a (mon.0) 1271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: audit 2022-04-22T17:36:04.779574+0000 mon.a (mon.0) 1272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:05.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:04 smithi115 bash[46934]: cluster 2022-04-22T17:36:04.929595+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2022-04-22T17:36:05.373 INFO:tasks.rados.rados.0.smithi115.stdout:3058: finishing write tid 4 to smithi11541303-11 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1664 (ObjNum 910 snap 299 seq_num 910) dirty exists 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:3058: left oid 11 (ObjNum 910 snap 299 seq_num 910) 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:3059: finishing write tid 1 to smithi11541303-45 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:3059: finishing write tid 2 to smithi11541303-45 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:3059: finishing write tid 3 to smithi11541303-45 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:3059: finishing write tid 4 to smithi11541303-45 2022-04-22T17:36:05.374 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 1924 (ObjNum 911 snap 299 seq_num 911) dirty exists 2022-04-22T17:36:05.375 INFO:tasks.rados.rados.0.smithi115.stdout:3059: left oid 45 (ObjNum 911 snap 299 seq_num 911) 2022-04-22T17:36:05.375 INFO:tasks.rados.rados.0.smithi115.stdout:3060: finishing rollback tid 0 to smithi11541303-13 2022-04-22T17:36:05.375 INFO:tasks.rados.rados.0.smithi115.stdout:3060: finishing rollback tid 1 to smithi11541303-13 2022-04-22T17:36:05.375 INFO:tasks.rados.rados.0.smithi115.stdout:3060: finishing rollback tid 2 to smithi11541303-13 2022-04-22T17:36:05.375 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1704 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3055: done (5 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3056: done (4 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3057: done (3 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3058: done (2 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3059: done (1 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3060: done (0 left) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3061: read oid 27 snap -1 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3061: expect (ObjNum 885 snap 289 seq_num 885) 2022-04-22T17:36:05.376 INFO:tasks.rados.rados.0.smithi115.stdout:3062: read oid 4 snap -1 2022-04-22T17:36:05.377 INFO:tasks.rados.rados.0.smithi115.stdout:3062: expect deleted 2022-04-22T17:36:05.377 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 299 2022-04-22T17:36:05.377 INFO:tasks.rados.rados.0.smithi115.stdout:3063: seq_num 912 ranges {2146304=483328,2629632=401408} 2022-04-22T17:36:05.395 INFO:tasks.rados.rados.0.smithi115.stdout:3063: writing smithi11541303-32 from 2146304 to 2629632 tid 1 2022-04-22T17:36:05.397 INFO:tasks.rados.rados.0.smithi115.stdout:3063: writing smithi11541303-32 from 2629632 to 3031040 tid 2 2022-04-22T17:36:05.397 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 299 2022-04-22T17:36:05.397 INFO:tasks.rados.rados.0.smithi115.stdout:3064: seq_num 913 ranges {3121152=499712,3620864=401408} 2022-04-22T17:36:05.417 INFO:tasks.rados.rados.0.smithi115.stdout:3064: writing smithi11541303-43 from 3121152 to 3620864 tid 1 2022-04-22T17:36:05.420 INFO:tasks.rados.rados.0.smithi115.stdout:3064: writing smithi11541303-43 from 3620864 to 4022272 tid 2 2022-04-22T17:36:05.420 INFO:tasks.rados.rados.0.smithi115.stdout:3065: copy_from oid 18 from oid 36 current snap is 299 2022-04-22T17:36:05.420 INFO:tasks.rados.rados.0.smithi115.stdout:3066: copy_from oid 37 from oid 28 current snap is 299 2022-04-22T17:36:05.420 INFO:tasks.rados.rados.0.smithi115.stdout:3061: expect (ObjNum 885 snap 289 seq_num 885) 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3063: finishing write tid 1 to smithi11541303-32 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3063: finishing write tid 2 to smithi11541303-32 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3063: finishing write tid 3 to smithi11541303-32 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3063: finishing write tid 4 to smithi11541303-32 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1387 (ObjNum 912 snap 299 seq_num 912) dirty exists 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3063: left oid 32 (ObjNum 912 snap 299 seq_num 912) 2022-04-22T17:36:05.440 INFO:tasks.rados.rados.0.smithi115.stdout:3061: done (5 left) 2022-04-22T17:36:05.441 INFO:tasks.rados.rados.0.smithi115.stdout:3062: done (4 left) 2022-04-22T17:36:05.441 INFO:tasks.rados.rados.0.smithi115.stdout:3063: done (3 left) 2022-04-22T17:36:05.441 INFO:tasks.rados.rados.0.smithi115.stdout:3067: snap_remove snap 298 2022-04-22T17:36:06.009 INFO:tasks.rados.rados.0.smithi115.stdout:3067: done (3 left) 2022-04-22T17:36:06.009 INFO:tasks.rados.rados.0.smithi115.stdout:3068: copy_from oid 47 from oid 24 current snap is 299 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3069: rmattr oid 33 current snap is 299 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3069: done (4 left) 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3070: rollback oid 20 current snap is 299 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 296 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3071: copy_from oid 40 from oid 31 current snap is 299 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3072: read oid 48 snap -1 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3072: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:36:06.010 INFO:tasks.rados.rados.0.smithi115.stdout:3073: setattr oid 13 current snap is 299 2022-04-22T17:36:06.011 INFO:tasks.rados.rados.0.smithi115.stdout:3065: finishing copy_from to smithi11541303-18 2022-04-22T17:36:06.011 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1251 (ObjNum 288 snap 0 seq_num 33) dirty exists 2022-04-22T17:36:06.011 INFO:tasks.rados.rados.0.smithi115.stdout:3064: finishing write tid 1 to smithi11541303-43 2022-04-22T17:36:06.011 INFO:tasks.rados.rados.0.smithi115.stdout:3065: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3064: finishing write tid 2 to smithi11541303-43 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3064: finishing write tid 3 to smithi11541303-43 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3064: finishing write tid 4 to smithi11541303-43 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1708 (ObjNum 913 snap 299 seq_num 913) dirty exists 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3064: left oid 43 (ObjNum 913 snap 299 seq_num 913) 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3066: finishing copy_from to smithi11541303-37 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1659 (ObjNum 846 snap 279 seq_num 846) dirty exists 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3066: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3064: done (7 left) 2022-04-22T17:36:06.012 INFO:tasks.rados.rados.0.smithi115.stdout:3065: done (6 left) 2022-04-22T17:36:06.013 INFO:tasks.rados.rados.0.smithi115.stdout:3066: done (5 left) 2022-04-22T17:36:06.013 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 299 2022-04-22T17:36:06.013 INFO:tasks.rados.rados.0.smithi115.stdout:3074: seq_num 914 ranges {3620864=516096,4136960=401408} 2022-04-22T17:36:06.038 INFO:tasks.rados.rados.0.smithi115.stdout:3074: writing smithi11541303-37 from 3620864 to 4136960 tid 1 2022-04-22T17:36:06.041 INFO:tasks.rados.rados.0.smithi115.stdout:3074: writing smithi11541303-37 from 4136960 to 4538368 tid 2 2022-04-22T17:36:06.041 INFO:tasks.rados.rados.0.smithi115.stdout:3075: read oid 46 snap -1 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:3075: expect (ObjNum 870 snap 285 seq_num 870) 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:3076: read oid 33 snap -1 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:3076: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 299 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:3070: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:36:06.042 INFO:tasks.rados.rados.0.smithi115.stdout:3077: seq_num 915 ranges {0=532480,532480=401408} 2022-04-22T17:36:06.046 INFO:tasks.rados.rados.0.smithi115.stdout:3077: writing smithi11541303-16 from 0 to 532480 tid 1 2022-04-22T17:36:06.049 INFO:tasks.rados.rados.0.smithi115.stdout:3077: writing smithi11541303-16 from 532480 to 933888 tid 2 2022-04-22T17:36:06.050 INFO:tasks.rados.rados.0.smithi115.stdout:3070: done (8 left) 2022-04-22T17:36:06.050 INFO:tasks.rados.rados.0.smithi115.stdout:3078: rollback oid 20 current snap is 299 2022-04-22T17:36:06.050 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 296 2022-04-22T17:36:06.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1709 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:36:06.050 INFO:tasks.rados.rados.0.smithi115.stdout:3072: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:36:06.170 INFO:tasks.rados.rados.0.smithi115.stdout:3071: finishing copy_from to smithi11541303-40 2022-04-22T17:36:06.171 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2004 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:36:06.171 INFO:tasks.rados.rados.0.smithi115.stdout:3071: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:36:06.171 INFO:tasks.rados.rados.0.smithi115.stdout:3074: finishing write tid 1 to smithi11541303-37 2022-04-22T17:36:06.171 INFO:tasks.rados.rados.0.smithi115.stdout:3074: finishing write tid 2 to smithi11541303-37 2022-04-22T17:36:06.172 INFO:tasks.rados.rados.0.smithi115.stdout:3074: finishing write tid 3 to smithi11541303-37 2022-04-22T17:36:06.172 INFO:tasks.rados.rados.0.smithi115.stdout:3071: done (8 left) 2022-04-22T17:36:06.172 INFO:tasks.rados.rados.0.smithi115.stdout:3072: done (7 left) 2022-04-22T17:36:06.172 INFO:tasks.rados.rados.0.smithi115.stdout:3073: done (6 left) 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:3079: read oid 35 snap 296 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:3079: expect deleted 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 299 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:3074: finishing write tid 4 to smithi11541303-37 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1849 (ObjNum 914 snap 299 seq_num 914) dirty exists 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:3074: left oid 37 (ObjNum 914 snap 299 seq_num 914) 2022-04-22T17:36:06.173 INFO:tasks.rados.rados.0.smithi115.stdout:3076: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:06.183 INFO:tasks.rados.rados.0.smithi115.stdout:3077: finishing write tid 1 to smithi11541303-16 2022-04-22T17:36:06.183 INFO:tasks.rados.rados.0.smithi115.stdout:3075: expect (ObjNum 870 snap 285 seq_num 870) 2022-04-22T17:36:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:3078: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:36:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:36:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:3077: finishing write tid 2 to smithi11541303-16 2022-04-22T17:36:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:3077: finishing write tid 3 to smithi11541303-16 2022-04-22T17:36:06.239 INFO:tasks.rados.rados.0.smithi115.stdout:3077: finishing write tid 4 to smithi11541303-16 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1663 (ObjNum 915 snap 299 seq_num 915) dirty exists 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:3077: left oid 16 (ObjNum 915 snap 299 seq_num 915) 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:3068: finishing copy_from to smithi11541303-47 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 1835 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:3068: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:36:06.240 INFO:tasks.rados.rados.0.smithi115.stdout:3080: seq_num 916 ranges {376832=548864,925696=401408} 2022-04-22T17:36:06.249 INFO:tasks.rados.rados.0.smithi115.stdout:3080: writing smithi11541303-3 from 376832 to 925696 tid 1 2022-04-22T17:36:06.251 INFO:tasks.rados.rados.0.smithi115.stdout:3080: writing smithi11541303-3 from 925696 to 1327104 tid 2 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3068: done (7 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3074: done (6 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3075: done (5 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3076: done (4 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3077: done (3 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3078: done (2 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3079: done (1 left) 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 299 2022-04-22T17:36:06.252 INFO:tasks.rados.rados.0.smithi115.stdout:3081: seq_num 917 ranges {0=565248,565248=401408} 2022-04-22T17:36:06.255 INFO:tasks.rados.rados.0.smithi115.stdout:3081: writing smithi11541303-29 from 0 to 565248 tid 1 2022-04-22T17:36:06.258 INFO:tasks.rados.rados.0.smithi115.stdout:3081: writing smithi11541303-29 from 565248 to 966656 tid 2 2022-04-22T17:36:06.258 INFO:tasks.rados.rados.0.smithi115.stdout:3082: read oid 35 snap -1 2022-04-22T17:36:06.259 INFO:tasks.rados.rados.0.smithi115.stdout:3082: expect deleted 2022-04-22T17:36:06.259 INFO:tasks.rados.rados.0.smithi115.stdout:3083: copy_from oid 23 from oid 20 current snap is 299 2022-04-22T17:36:06.259 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 299 2022-04-22T17:36:06.259 INFO:tasks.rados.rados.0.smithi115.stdout:3084: seq_num 918 ranges {3620864=581632,4202496=401408} 2022-04-22T17:36:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:05 smithi121 bash[36902]: cephadm 2022-04-22T17:36:04.203407+0000 mgr.y (mgr.34107) 384 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:36:06.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:05 smithi121 bash[36902]: cephadm 2022-04-22T17:36:04.733036+0000 mgr.y (mgr.34107) 385 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:06.297 INFO:tasks.rados.rados.0.smithi115.stdout:3084: writing smithi11541303-28 from 3620864 to 4202496 tid 1 2022-04-22T17:36:06.297 INFO:tasks.rados.rados.0.smithi115.stdout:3084: writing smithi11541303-28 from 4202496 to 4603904 tid 2 2022-04-22T17:36:06.297 INFO:tasks.rados.rados.0.smithi115.stdout:3085: copy_from oid 46 from oid 37 current snap is 299 2022-04-22T17:36:06.297 INFO:tasks.rados.rados.0.smithi115.stdout:3080: finishing write tid 1 to smithi11541303-3 2022-04-22T17:36:06.297 INFO:tasks.rados.rados.0.smithi115.stdout:3086: snap_remove snap 296 2022-04-22T17:36:06.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:05 smithi115 bash[45495]: cephadm 2022-04-22T17:36:04.203407+0000 mgr.y (mgr.34107) 384 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:36:06.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:05 smithi115 bash[45495]: cephadm 2022-04-22T17:36:04.733036+0000 mgr.y (mgr.34107) 385 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:05 smithi115 bash[46934]: cephadm 2022-04-22T17:36:04.203407+0000 mgr.y (mgr.34107) 384 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:36:06.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:05 smithi115 bash[46934]: cephadm 2022-04-22T17:36:04.733036+0000 mgr.y (mgr.34107) 385 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3080: finishing write tid 2 to smithi11541303-3 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3080: finishing write tid 3 to smithi11541303-3 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3083: finishing copy_from to smithi11541303-23 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3083: got expected ENOENT (src dne) 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3083: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:3080: finishing write tid 4 to smithi11541303-3 2022-04-22T17:36:07.021 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1667 (ObjNum 916 snap 299 seq_num 916) dirty exists 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3080: left oid 3 (ObjNum 916 snap 299 seq_num 916) 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3081: finishing write tid 1 to smithi11541303-29 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3081: finishing write tid 2 to smithi11541303-29 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3081: finishing write tid 3 to smithi11541303-29 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3081: finishing write tid 4 to smithi11541303-29 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1665 (ObjNum 917 snap 299 seq_num 917) dirty exists 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3081: left oid 29 (ObjNum 917 snap 299 seq_num 917) 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3084: finishing write tid 1 to smithi11541303-28 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3084: finishing write tid 2 to smithi11541303-28 2022-04-22T17:36:07.022 INFO:tasks.rados.rados.0.smithi115.stdout:3084: finishing write tid 3 to smithi11541303-28 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3084: finishing write tid 4 to smithi11541303-28 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1668 (ObjNum 918 snap 299 seq_num 918) dirty exists 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3084: left oid 28 (ObjNum 918 snap 299 seq_num 918) 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3085: finishing copy_from to smithi11541303-46 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1849 (ObjNum 914 snap 299 seq_num 914) dirty exists 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3085: finishing copy_from racing read to smithi11541303-46 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3080: done (6 left) 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3081: done (5 left) 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3082: done (4 left) 2022-04-22T17:36:07.023 INFO:tasks.rados.rados.0.smithi115.stdout:3083: done (3 left) 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3084: done (2 left) 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3085: done (1 left) 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3086: done (0 left) 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3087: rollback oid 24 current snap is 299 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3087: done (0 left) 2022-04-22T17:36:07.024 INFO:tasks.rados.rados.0.smithi115.stdout:3088: snap_create 2022-04-22T17:36:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:07 smithi121 bash[36902]: cluster 2022-04-22T17:36:05.324405+0000 mgr.y (mgr.34107) 386 : cluster [DBG] pgmap v690: 121 pgs: 121 active+clean; 121 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 940 KiB/s wr, 12 op/s 2022-04-22T17:36:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:07 smithi121 bash[36902]: cluster 2022-04-22T17:36:05.996663+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2022-04-22T17:36:07.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:06 smithi115 bash[45495]: cluster 2022-04-22T17:36:05.324405+0000 mgr.y (mgr.34107) 386 : cluster [DBG] pgmap v690: 121 pgs: 121 active+clean; 121 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 940 KiB/s wr, 12 op/s 2022-04-22T17:36:07.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:06 smithi115 bash[45495]: cluster 2022-04-22T17:36:05.996663+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2022-04-22T17:36:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:06 smithi115 bash[46934]: cluster 2022-04-22T17:36:05.324405+0000 mgr.y (mgr.34107) 386 : cluster [DBG] pgmap v690: 121 pgs: 121 active+clean; 121 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 940 KiB/s wr, 12 op/s 2022-04-22T17:36:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:06 smithi115 bash[46934]: cluster 2022-04-22T17:36:05.996663+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2022-04-22T17:36:08.032 INFO:tasks.rados.rados.0.smithi115.stdout:3088: done (0 left) 2022-04-22T17:36:08.032 INFO:tasks.rados.rados.0.smithi115.stdout:3089: delete oid 6 current snap is 300 2022-04-22T17:36:08.032 INFO:tasks.rados.rados.0.smithi115.stdout:3089: done (0 left) 2022-04-22T17:36:08.032 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 300 2022-04-22T17:36:08.032 INFO:tasks.rados.rados.0.smithi115.stdout:3090: seq_num 920 ranges {3244032=614400,3858432=401408} 2022-04-22T17:36:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3090: writing smithi11541303-34 from 3244032 to 3858432 tid 1 2022-04-22T17:36:08.063 INFO:tasks.rados.rados.0.smithi115.stdout:3090: writing smithi11541303-34 from 3858432 to 4259840 tid 2 2022-04-22T17:36:08.063 INFO:tasks.rados.rados.0.smithi115.stdout:3091: read oid 41 snap 299 2022-04-22T17:36:08.063 INFO:tasks.rados.rados.0.smithi115.stdout:3091: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:36:08.063 INFO:tasks.rados.rados.0.smithi115.stdout:3092: snap_create 2022-04-22T17:36:08.074 INFO:tasks.rados.rados.0.smithi115.stdout:3090: finishing write tid 1 to smithi11541303-34 2022-04-22T17:36:08.075 INFO:tasks.rados.rados.0.smithi115.stdout:3090: finishing write tid 2 to smithi11541303-34 2022-04-22T17:36:08.075 INFO:tasks.rados.rados.0.smithi115.stdout:3090: finishing write tid 3 to smithi11541303-34 2022-04-22T17:36:08.075 INFO:tasks.rados.rados.0.smithi115.stdout:3090: finishing write tid 4 to smithi11541303-34 2022-04-22T17:36:08.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1666 (ObjNum 920 snap 300 seq_num 920) dirty exists 2022-04-22T17:36:08.075 INFO:tasks.rados.rados.0.smithi115.stdout:3090: left oid 34 (ObjNum 920 snap 300 seq_num 920) 2022-04-22T17:36:08.081 INFO:tasks.rados.rados.0.smithi115.stdout:3091: expect (ObjNum 862 snap 281 seq_num 862) 2022-04-22T17:36:08.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:08 smithi121 bash[36902]: cluster 2022-04-22T17:36:07.006892+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2022-04-22T17:36:08.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:08 smithi115 bash[45495]: cluster 2022-04-22T17:36:07.006892+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2022-04-22T17:36:08.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:08 smithi115 bash[46934]: cluster 2022-04-22T17:36:07.006892+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2022-04-22T17:36:09.039 INFO:tasks.rados.rados.0.smithi115.stdout:3090: done (2 left) 2022-04-22T17:36:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:3091: done (1 left) 2022-04-22T17:36:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:3092: done (0 left) 2022-04-22T17:36:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:3093: copy_from oid 24 from oid 15 current snap is 301 2022-04-22T17:36:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 301 2022-04-22T17:36:09.088 INFO:tasks.rados.rados.0.smithi115.stdout:3094: seq_num 921 ranges {2138112=630784,2768896=401408} 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3094: writing smithi11541303-8 from 2138112 to 2768896 tid 1 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3094: writing smithi11541303-8 from 2768896 to 3170304 tid 2 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3095: delete oid 50 current snap is 301 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3094: finishing write tid 1 to smithi11541303-8 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3095: done (2 left) 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3096: read oid 4 snap -1 2022-04-22T17:36:09.092 INFO:tasks.rados.rados.0.smithi115.stdout:3096: expect deleted 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3097: copy_from oid 34 from oid 47 current snap is 301 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 301 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3098: seq_num 923 ranges {1327104=663552,1990656=401408} 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3098: writing smithi11541303-3 from 1327104 to 1990656 tid 1 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3098: writing smithi11541303-3 from 1990656 to 2392064 tid 2 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3099: read oid 42 snap -1 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3099: expect (ObjNum 902 snap 295 seq_num 902) 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 301 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3094: finishing write tid 2 to smithi11541303-8 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3100: seq_num 924 ranges {0=679936,679936=401408} 2022-04-22T17:36:09.093 INFO:tasks.rados.rados.0.smithi115.stdout:3100: writing smithi11541303-25 from 0 to 679936 tid 1 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3100: writing smithi11541303-25 from 679936 to 1081344 tid 2 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3101: read oid 29 snap -1 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3101: expect (ObjNum 917 snap 299 seq_num 917) 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3102: rmattr oid 14 current snap is 301 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3102: done (8 left) 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3103: rollback oid 30 current snap is 301 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 299 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3094: finishing write tid 3 to smithi11541303-8 2022-04-22T17:36:09.094 INFO:tasks.rados.rados.0.smithi115.stdout:3094: finishing write tid 4 to smithi11541303-8 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1841 (ObjNum 921 snap 301 seq_num 921) dirty exists 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3094: left oid 8 (ObjNum 921 snap 301 seq_num 921) 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3094: done (8 left) 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3096: done (7 left) 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3104: copy_from oid 41 from oid 13 current snap is 301 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3093: finishing copy_from to smithi11541303-24 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1547 (ObjNum 838 snap 276 seq_num 838) dirty exists 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3093: finishing copy_from racing read to smithi11541303-24 2022-04-22T17:36:09.095 INFO:tasks.rados.rados.0.smithi115.stdout:3093: done (7 left) 2022-04-22T17:36:09.096 INFO:tasks.rados.rados.0.smithi115.stdout:3105: read oid 9 snap 299 2022-04-22T17:36:09.096 INFO:tasks.rados.rados.0.smithi115.stdout:3105: expect deleted 2022-04-22T17:36:09.096 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 301 2022-04-22T17:36:09.096 INFO:tasks.rados.rados.0.smithi115.stdout:3106: seq_num 925 ranges {4022272=696320,4718592=401408} 2022-04-22T17:36:09.118 INFO:tasks.rados.rados.0.smithi115.stdout:3106: writing smithi11541303-43 from 4022272 to 4718592 tid 1 2022-04-22T17:36:09.121 INFO:tasks.rados.rados.0.smithi115.stdout:3106: writing smithi11541303-43 from 4718592 to 5120000 tid 2 2022-04-22T17:36:09.121 INFO:tasks.rados.rados.0.smithi115.stdout:3107: delete oid 45 current snap is 301 2022-04-22T17:36:09.121 INFO:tasks.rados.rados.0.smithi115.stdout:3098: finishing write tid 1 to smithi11541303-3 2022-04-22T17:36:09.121 INFO:tasks.rados.rados.0.smithi115.stdout:3099: expect (ObjNum 902 snap 295 seq_num 902) 2022-04-22T17:36:09.144 INFO:tasks.rados.rados.0.smithi115.stdout:3101: expect (ObjNum 917 snap 299 seq_num 917) 2022-04-22T17:36:09.173 INFO:tasks.rados.rados.0.smithi115.stdout:3098: finishing write tid 2 to smithi11541303-3 2022-04-22T17:36:09.173 INFO:tasks.rados.rados.0.smithi115.stdout:3103: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:36:09.173 INFO:tasks.rados.rados.0.smithi115.stdout:3098: finishing write tid 3 to smithi11541303-3 2022-04-22T17:36:09.173 INFO:tasks.rados.rados.0.smithi115.stdout:3100: finishing write tid 1 to smithi11541303-25 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3103: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3100: finishing write tid 2 to smithi11541303-25 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3100: finishing write tid 3 to smithi11541303-25 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3099: done (9 left) 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3101: done (8 left) 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3105: done (7 left) 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:3107: done (6 left) 2022-04-22T17:36:09.174 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 301 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:3103: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1927 (ObjNum 896 snap 294 seq_num 896) dirty exists 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:3098: finishing write tid 4 to smithi11541303-3 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1852 (ObjNum 923 snap 301 seq_num 923) dirty exists 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:3098: left oid 3 (ObjNum 923 snap 301 seq_num 923) 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:3100: finishing write tid 4 to smithi11541303-25 2022-04-22T17:36:09.175 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1722 (ObjNum 924 snap 301 seq_num 924) dirty exists 2022-04-22T17:36:09.176 INFO:tasks.rados.rados.0.smithi115.stdout:3100: left oid 25 (ObjNum 924 snap 301 seq_num 924) 2022-04-22T17:36:09.176 INFO:tasks.rados.rados.0.smithi115.stdout:3104: finishing copy_from to smithi11541303-41 2022-04-22T17:36:09.176 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1709 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:36:09.176 INFO:tasks.rados.rados.0.smithi115.stdout:3104: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:36:09.177 INFO:tasks.rados.rados.0.smithi115.stdout:3108: seq_num 927 ranges {0=729088,729088=401408} 2022-04-22T17:36:09.177 INFO:tasks.rados.rados.0.smithi115.stdout:3108: writing smithi11541303-35 from 0 to 729088 tid 1 2022-04-22T17:36:09.179 INFO:tasks.rados.rados.0.smithi115.stdout:3108: writing smithi11541303-35 from 729088 to 1130496 tid 2 2022-04-22T17:36:09.179 INFO:tasks.rados.rados.0.smithi115.stdout:3098: done (6 left) 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3100: done (5 left) 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3103: done (4 left) 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3104: done (3 left) 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3109: snap_create 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3106: finishing write tid 1 to smithi11541303-43 2022-04-22T17:36:09.180 INFO:tasks.rados.rados.0.smithi115.stdout:3106: finishing write tid 2 to smithi11541303-43 2022-04-22T17:36:09.181 INFO:tasks.rados.rados.0.smithi115.stdout:3106: finishing write tid 3 to smithi11541303-43 2022-04-22T17:36:09.181 INFO:tasks.rados.rados.0.smithi115.stdout:3097: finishing copy_from to smithi11541303-34 2022-04-22T17:36:09.182 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1835 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:36:09.182 INFO:tasks.rados.rados.0.smithi115.stdout:3097: finishing copy_from racing read to smithi11541303-34 2022-04-22T17:36:09.182 INFO:tasks.rados.rados.0.smithi115.stdout:3106: finishing write tid 4 to smithi11541303-43 2022-04-22T17:36:09.182 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1726 (ObjNum 925 snap 301 seq_num 925) dirty exists 2022-04-22T17:36:09.183 INFO:tasks.rados.rados.0.smithi115.stdout:3106: left oid 43 (ObjNum 925 snap 301 seq_num 925) 2022-04-22T17:36:09.185 INFO:tasks.rados.rados.0.smithi115.stdout:3108: finishing write tid 1 to smithi11541303-35 2022-04-22T17:36:09.185 INFO:tasks.rados.rados.0.smithi115.stdout:3108: finishing write tid 2 to smithi11541303-35 2022-04-22T17:36:09.185 INFO:tasks.rados.rados.0.smithi115.stdout:3108: finishing write tid 3 to smithi11541303-35 2022-04-22T17:36:09.187 INFO:tasks.rados.rados.0.smithi115.stdout:3108: finishing write tid 4 to smithi11541303-35 2022-04-22T17:36:09.187 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1852 (ObjNum 927 snap 301 seq_num 927) dirty exists 2022-04-22T17:36:09.187 INFO:tasks.rados.rados.0.smithi115.stdout:3108: left oid 35 (ObjNum 927 snap 301 seq_num 927) 2022-04-22T17:36:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[45495]: cluster 2022-04-22T17:36:07.325180+0000 mgr.y (mgr.34107) 387 : cluster [DBG] pgmap v693: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 964 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[45495]: cephadm 2022-04-22T17:36:07.438473+0000 mgr.y (mgr.34107) 388 : cephadm [INF] Upgrade: Updating prometheus.a 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[45495]: cephadm 2022-04-22T17:36:07.462186+0000 mgr.y (mgr.34107) 389 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[45495]: cluster 2022-04-22T17:36:08.016503+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[46934]: cluster 2022-04-22T17:36:07.325180+0000 mgr.y (mgr.34107) 387 : cluster [DBG] pgmap v693: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 964 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[46934]: cephadm 2022-04-22T17:36:07.438473+0000 mgr.y (mgr.34107) 388 : cephadm [INF] Upgrade: Updating prometheus.a 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[46934]: cephadm 2022-04-22T17:36:07.462186+0000 mgr.y (mgr.34107) 389 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:36:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:09 smithi115 bash[46934]: cluster 2022-04-22T17:36:08.016503+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2022-04-22T17:36:09.405 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:09 smithi121 bash[36902]: cluster 2022-04-22T17:36:07.325180+0000 mgr.y (mgr.34107) 387 : cluster [DBG] pgmap v693: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 964 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2022-04-22T17:36:09.406 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:09 smithi121 bash[36902]: cephadm 2022-04-22T17:36:07.438473+0000 mgr.y (mgr.34107) 388 : cephadm [INF] Upgrade: Updating prometheus.a 2022-04-22T17:36:09.406 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:09 smithi121 bash[36902]: cephadm 2022-04-22T17:36:07.462186+0000 mgr.y (mgr.34107) 389 : cephadm [INF] Deploying daemon prometheus.a on smithi121 2022-04-22T17:36:09.406 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:09 smithi121 bash[36902]: cluster 2022-04-22T17:36:08.016503+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2022-04-22T17:36:09.946 INFO:tasks.rados.rados.0.smithi115.stdout:3097: done (3 left) 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3106: done (2 left) 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3108: done (1 left) 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3109: done (0 left) 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3110: copy_from oid 40 from oid 20 current snap is 302 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3111: rmattr oid 46 current snap is 302 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 302 2022-04-22T17:36:09.947 INFO:tasks.rados.rados.0.smithi115.stdout:3112: seq_num 928 ranges {843776=745472,1589248=401408} 2022-04-22T17:36:09.959 INFO:tasks.rados.rados.0.smithi115.stdout:3112: writing smithi11541303-11 from 843776 to 1589248 tid 1 2022-04-22T17:36:09.964 INFO:tasks.rados.rados.0.smithi115.stdout:3112: writing smithi11541303-11 from 1589248 to 1990656 tid 2 2022-04-22T17:36:09.964 INFO:tasks.rados.rados.0.smithi115.stdout:3113: snap_create 2022-04-22T17:36:09.964 INFO:tasks.rados.rados.0.smithi115.stdout:3110: finishing copy_from to smithi11541303-40 2022-04-22T17:36:09.964 INFO:tasks.rados.rados.0.smithi115.stdout:3110: got expected ENOENT (src dne) 2022-04-22T17:36:09.964 INFO:tasks.rados.rados.0.smithi115.stdout:3110: finishing copy_from racing read to smithi11541303-40 2022-04-22T17:36:09.965 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1853 (ObjNum 914 snap 299 seq_num 914) dirty exists 2022-04-22T17:36:09.966 INFO:tasks.rados.rados.0.smithi115.stdout:3112: finishing write tid 1 to smithi11541303-11 2022-04-22T17:36:09.967 INFO:tasks.rados.rados.0.smithi115.stdout:3112: finishing write tid 2 to smithi11541303-11 2022-04-22T17:36:09.967 INFO:tasks.rados.rados.0.smithi115.stdout:3112: finishing write tid 3 to smithi11541303-11 2022-04-22T17:36:09.968 INFO:tasks.rados.rados.0.smithi115.stdout:3112: finishing write tid 4 to smithi11541303-11 2022-04-22T17:36:09.968 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1856 (ObjNum 928 snap 302 seq_num 928) dirty exists 2022-04-22T17:36:09.968 INFO:tasks.rados.rados.0.smithi115.stdout:3112: left oid 11 (ObjNum 928 snap 302 seq_num 928) 2022-04-22T17:36:10.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[45495]: cluster 2022-04-22T17:36:09.021960+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2022-04-22T17:36:10.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[45495]: audit 2022-04-22T17:36:09.380677+0000 mon.a (mon.0) 1278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:10.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[45495]: cluster 2022-04-22T17:36:09.931105+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2022-04-22T17:36:10.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[46934]: cluster 2022-04-22T17:36:09.021960+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2022-04-22T17:36:10.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[46934]: audit 2022-04-22T17:36:09.380677+0000 mon.a (mon.0) 1278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:10.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:10 smithi115 bash[46934]: cluster 2022-04-22T17:36:09.931105+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2022-04-22T17:36:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:10 smithi121 bash[36902]: cluster 2022-04-22T17:36:09.021960+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2022-04-22T17:36:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:10 smithi121 bash[36902]: audit 2022-04-22T17:36:09.380677+0000 mon.a (mon.0) 1278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:10 smithi121 bash[36902]: cluster 2022-04-22T17:36:09.931105+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2022-04-22T17:36:10.959 INFO:tasks.rados.rados.0.smithi115.stdout:3110: done (3 left) 2022-04-22T17:36:10.959 INFO:tasks.rados.rados.0.smithi115.stdout:3111: done (2 left) 2022-04-22T17:36:10.959 INFO:tasks.rados.rados.0.smithi115.stdout:3112: done (1 left) 2022-04-22T17:36:10.959 INFO:tasks.rados.rados.0.smithi115.stdout:3113: done (0 left) 2022-04-22T17:36:10.959 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 303 2022-04-22T17:36:10.960 INFO:tasks.rados.rados.0.smithi115.stdout:3114: seq_num 929 ranges {630784=761856,1392640=401408} 2022-04-22T17:36:10.967 INFO:tasks.rados.rados.0.smithi115.stdout:3114: writing smithi11541303-39 from 630784 to 1392640 tid 1 2022-04-22T17:36:10.970 INFO:tasks.rados.rados.0.smithi115.stdout:3114: writing smithi11541303-39 from 1392640 to 1794048 tid 2 2022-04-22T17:36:10.971 INFO:tasks.rados.rados.0.smithi115.stdout:3115: read oid 17 snap -1 2022-04-22T17:36:10.971 INFO:tasks.rados.rados.0.smithi115.stdout:3115: expect (ObjNum 884 snap 289 seq_num 884) 2022-04-22T17:36:10.971 INFO:tasks.rados.rados.0.smithi115.stdout:3116: read oid 49 snap -1 2022-04-22T17:36:10.971 INFO:tasks.rados.rados.0.smithi115.stdout:3116: expect deleted 2022-04-22T17:36:10.971 INFO:tasks.rados.rados.0.smithi115.stdout:3117: snap_remove snap 301 2022-04-22T17:36:11.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[45495]: cluster 2022-04-22T17:36:09.325935+0000 mgr.y (mgr.34107) 390 : cluster [DBG] pgmap v696: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[45495]: cluster 2022-04-22T17:36:10.944982+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2022-04-22T17:36:11.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[46934]: cluster 2022-04-22T17:36:09.325935+0000 mgr.y (mgr.34107) 390 : cluster [DBG] pgmap v696: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:11.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[46934]: cluster 2022-04-22T17:36:10.944982+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2022-04-22T17:36:11.401 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:11 smithi121 bash[36902]: cluster 2022-04-22T17:36:09.325935+0000 mgr.y (mgr.34107) 390 : cluster [DBG] pgmap v696: 121 pgs: 121 active+clean; 117 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:11.401 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:11 smithi121 bash[36902]: cluster 2022-04-22T17:36:10.944982+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2022-04-22T17:36:11.960 INFO:tasks.rados.rados.0.smithi115.stdout:3117: done (3 left) 2022-04-22T17:36:11.960 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 303 2022-04-22T17:36:11.961 INFO:tasks.rados.rados.0.smithi115.stdout:3118: seq_num 930 ranges {12345344=786432,13131776=401408} 2022-04-22T17:36:12.033 INFO:tasks.rados.rados.0.smithi115.stdout:3118: writing smithi11541303-30 from 12345344 to 13131776 tid 1 2022-04-22T17:36:12.072 INFO:tasks.rados.rados.0.smithi115.stdout:3118: writing smithi11541303-30 from 13131776 to 13533184 tid 2 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3119: read oid 4 snap -1 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3119: expect deleted 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3120: setattr oid 13 current snap is 303 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3114: finishing write tid 1 to smithi11541303-39 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3114: finishing write tid 2 to smithi11541303-39 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3114: finishing write tid 3 to smithi11541303-39 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3114: finishing write tid 4 to smithi11541303-39 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1855 (ObjNum 929 snap 303 seq_num 929) dirty exists 2022-04-22T17:36:12.073 INFO:tasks.rados.rados.0.smithi115.stdout:3114: left oid 39 (ObjNum 929 snap 303 seq_num 929) 2022-04-22T17:36:12.074 INFO:tasks.rados.rados.0.smithi115.stdout:3115: expect (ObjNum 884 snap 289 seq_num 884) 2022-04-22T17:36:12.074 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:11.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:12.074 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:11.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:12.074 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:11 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:11.634Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:12.397 INFO:tasks.rados.rados.0.smithi115.stdout:3118: finishing write tid 1 to smithi11541303-30 2022-04-22T17:36:12.399 INFO:tasks.rados.rados.0.smithi115.stdout:3118: finishing write tid 2 to smithi11541303-30 2022-04-22T17:36:12.399 INFO:tasks.rados.rados.0.smithi115.stdout:3118: finishing write tid 3 to smithi11541303-30 2022-04-22T17:36:12.399 INFO:tasks.rados.rados.0.smithi115.stdout:3118: finishing write tid 4 to smithi11541303-30 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1931 (ObjNum 930 snap 303 seq_num 930) dirty exists 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3118: left oid 30 (ObjNum 930 snap 303 seq_num 930) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3114: done (5 left) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3115: done (4 left) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3116: done (3 left) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3118: done (2 left) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3119: done (1 left) 2022-04-22T17:36:12.400 INFO:tasks.rados.rados.0.smithi115.stdout:3121: setattr oid 44 current snap is 303 2022-04-22T17:36:12.401 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 303 2022-04-22T17:36:12.401 INFO:tasks.rados.rados.0.smithi115.stdout:3122: seq_num 931 ranges {5193728=802816,5996544=401408} 2022-04-22T17:36:12.435 INFO:tasks.rados.rados.0.smithi115.stdout:3122: writing smithi11541303-26 from 5193728 to 5996544 tid 1 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:3122: writing smithi11541303-26 from 5996544 to 6397952 tid 2 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:3123: delete oid 18 current snap is 303 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1728 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1307 (ObjNum 1381985615 snap 1952403280 seq_num 1516523343) dirty exists 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:3120: done (3 left) 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:3121: done (2 left) 2022-04-22T17:36:12.447 INFO:tasks.rados.rados.0.smithi115.stdout:3123: done (1 left) 2022-04-22T17:36:12.448 INFO:tasks.rados.rados.0.smithi115.stdout:3124: rollback oid 28 current snap is 303 2022-04-22T17:36:12.448 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 28 to 300 2022-04-22T17:36:12.448 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 303 2022-04-22T17:36:12.448 INFO:tasks.rados.rados.0.smithi115.stdout:3125: seq_num 933 ranges {425984=434176,860160=548864,1409024=253952} 2022-04-22T17:36:12.448 INFO:tasks.rados.rados.0.smithi115.stdout:3125: writing smithi11541303-27 from 425984 to 860160 tid 1 2022-04-22T17:36:12.453 INFO:tasks.rados.rados.0.smithi115.stdout:3125: writing smithi11541303-27 from 860160 to 1409024 tid 2 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3125: writing smithi11541303-27 from 1409024 to 1662976 tid 3 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3126: read oid 8 snap -1 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3126: expect (ObjNum 921 snap 301 seq_num 921) 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3127: delete oid 36 current snap is 303 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3122: finishing write tid 1 to smithi11541303-26 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3122: finishing write tid 2 to smithi11541303-26 2022-04-22T17:36:12.456 INFO:tasks.rados.rados.0.smithi115.stdout:3122: finishing write tid 3 to smithi11541303-26 2022-04-22T17:36:12.457 INFO:tasks.rados.rados.0.smithi115.stdout:3124: finishing rollback tid 0 to smithi11541303-28 2022-04-22T17:36:12.457 INFO:tasks.rados.rados.0.smithi115.stdout:3124: finishing rollback tid 1 to smithi11541303-28 2022-04-22T17:36:12.457 INFO:tasks.rados.rados.0.smithi115.stdout:3124: finishing rollback tid 2 to smithi11541303-28 2022-04-22T17:36:12.457 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1671 (ObjNum 918 snap 299 seq_num 918) dirty exists 2022-04-22T17:36:12.458 INFO:tasks.rados.rados.0.smithi115.stdout:3122: finishing write tid 4 to smithi11541303-26 2022-04-22T17:36:12.458 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1390 (ObjNum 931 snap 303 seq_num 931) dirty exists 2022-04-22T17:36:12.458 INFO:tasks.rados.rados.0.smithi115.stdout:3122: left oid 26 (ObjNum 931 snap 303 seq_num 931) 2022-04-22T17:36:12.458 INFO:tasks.rados.rados.0.smithi115.stdout:3125: finishing write tid 1 to smithi11541303-27 2022-04-22T17:36:12.460 INFO:tasks.rados.rados.0.smithi115.stdout:3125: finishing write tid 2 to smithi11541303-27 2022-04-22T17:36:12.460 INFO:tasks.rados.rados.0.smithi115.stdout:3125: finishing write tid 3 to smithi11541303-27 2022-04-22T17:36:12.461 INFO:tasks.rados.rados.0.smithi115.stdout:3125: finishing write tid 4 to smithi11541303-27 2022-04-22T17:36:12.463 INFO:tasks.rados.rados.0.smithi115.stdout:3125: finishing write tid 5 to smithi11541303-27 2022-04-22T17:36:12.463 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1935 (ObjNum 933 snap 303 seq_num 933) dirty exists 2022-04-22T17:36:12.463 INFO:tasks.rados.rados.0.smithi115.stdout:3125: left oid 27 (ObjNum 933 snap 303 seq_num 933) 2022-04-22T17:36:12.468 INFO:tasks.rados.rados.0.smithi115.stdout:3126: expect (ObjNum 921 snap 301 seq_num 921) 2022-04-22T17:36:12.630 INFO:tasks.rados.rados.0.smithi115.stdout:3122: done (4 left) 2022-04-22T17:36:12.692 INFO:tasks.rados.rados.0.smithi115.stdout:3124: done (3 left) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3125: done (2 left) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3126: done (1 left) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3127: done (0 left) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3128: read oid 44 snap -1 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3128: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3129: rollback oid 33 current snap is 303 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 33 to 302 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3130: read oid 39 snap -1 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3130: expect (ObjNum 929 snap 303 seq_num 929) 2022-04-22T17:36:12.693 INFO:tasks.rados.rados.0.smithi115.stdout:3131: setattr oid 26 current snap is 303 2022-04-22T17:36:12.695 INFO:tasks.rados.rados.0.smithi115.stdout:3128: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:12.695 INFO:tasks.rados.rados.0.smithi115.stdout:3128: done (3 left) 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3132: rmattr oid 5 current snap is 303 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3133: read oid 22 snap 300 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3133: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3134: copy_from oid 1 from oid 46 current snap is 303 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3135: delete oid 18 current snap is 303 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3129: finishing rollback tid 0 to smithi11541303-33 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3129: finishing rollback tid 1 to smithi11541303-33 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:3129: finishing rollback tid 2 to smithi11541303-33 2022-04-22T17:36:12.696 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 1393 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2018 (ObjNum -1116233216 snap 22075 seq_num 3179756800) dirty exists 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:3129: done (6 left) 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:3132: done (5 left) 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:3135: done (4 left) 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:3136: delete oid 41 current snap is 303 2022-04-22T17:36:12.697 INFO:tasks.rados.rados.0.smithi115.stdout:3130: expect (ObjNum 929 snap 303 seq_num 929) 2022-04-22T17:36:12.711 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1394 (ObjNum 931 snap 303 seq_num 931) dirty exists 2022-04-22T17:36:12.711 INFO:tasks.rados.rados.0.smithi115.stdout:3133: expect (ObjNum 736 snap 253 seq_num 736) 2022-04-22T17:36:12.721 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 systemd[1]: Stopping Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:36:13.009 INFO:tasks.rados.rados.0.smithi115.stdout:3134: finishing copy_from to smithi11541303-1 2022-04-22T17:36:13.009 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1853 (ObjNum 914 snap 299 seq_num 914) dirty exists 2022-04-22T17:36:13.009 INFO:tasks.rados.rados.0.smithi115.stdout:3134: finishing copy_from racing read to smithi11541303-1 2022-04-22T17:36:13.009 INFO:tasks.rados.rados.0.smithi115.stdout:3130: done (4 left) 2022-04-22T17:36:13.010 INFO:tasks.rados.rados.0.smithi115.stdout:3131: done (3 left) 2022-04-22T17:36:13.013 INFO:tasks.rados.rados.0.smithi115.stdout:3133: done (2 left) 2022-04-22T17:36:13.013 INFO:tasks.rados.rados.0.smithi115.stdout:3134: done (1 left) 2022-04-22T17:36:13.013 INFO:tasks.rados.rados.0.smithi115.stdout:3136: done (0 left) 2022-04-22T17:36:13.013 INFO:tasks.rados.rados.0.smithi115.stdout:3137: rollback oid 32 current snap is 303 2022-04-22T17:36:13.014 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 302 2022-04-22T17:36:13.014 INFO:tasks.rados.rados.0.smithi115.stdout:3138: read oid 10 snap -1 2022-04-22T17:36:13.014 INFO:tasks.rados.rados.0.smithi115.stdout:3138: expect (ObjNum 908 snap 299 seq_num 908) 2022-04-22T17:36:13.014 INFO:tasks.rados.rados.0.smithi115.stdout:3139: read oid 40 snap -1 2022-04-22T17:36:13.015 INFO:tasks.rados.rados.0.smithi115.stdout:3139: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:36:13.015 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 303 2022-04-22T17:36:13.015 INFO:tasks.rados.rados.0.smithi115.stdout:3140: seq_num 937 ranges {7380992=499712,7880704=532480,8413184=270336} 2022-04-22T17:36:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[36902]: cluster 2022-04-22T17:36:11.326683+0000 mgr.y (mgr.34107) 391 : cluster [DBG] pgmap v699: 121 pgs: 121 active+clean; 133 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.3 MiB/s wr, 21 op/s 2022-04-22T17:36:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[36902]: cluster 2022-04-22T17:36:11.946461+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2022-04-22T17:36:13.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=warn ts=2022-04-22T17:36:12.718Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2022-04-22T17:36:13.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.718Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2022-04-22T17:36:13.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.718Z caller=main.go:561 msg="Stopping notify discovery manager..." 2022-04-22T17:36:13.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.718Z caller=main.go:583 msg="Stopping scrape manager..." 2022-04-22T17:36:13.017 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.718Z caller=main.go:543 msg="Scrape discovery manager stopped" 2022-04-22T17:36:13.017 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.719Z caller=main.go:557 msg="Notify discovery manager stopped" 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.719Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.719Z caller=main.go:577 msg="Scrape manager stopped" 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.719Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.720Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.720Z caller=main.go:749 msg="Notifier manager stopped" 2022-04-22T17:36:13.018 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:12 smithi121 bash[32438]: level=info ts=2022-04-22T17:36:12.720Z caller=main.go:761 msg="See you next time!" 2022-04-22T17:36:13.059 INFO:tasks.rados.rados.0.smithi115.stdout:3140: writing smithi11541303-24 from 7380992 to 7880704 tid 1 2022-04-22T17:36:13.062 INFO:tasks.rados.rados.0.smithi115.stdout:3140: writing smithi11541303-24 from 7880704 to 8413184 tid 2 2022-04-22T17:36:13.070 INFO:tasks.rados.rados.0.smithi115.stdout:3140: writing smithi11541303-24 from 8413184 to 8683520 tid 3 2022-04-22T17:36:13.070 INFO:tasks.rados.rados.0.smithi115.stdout:3141: copy_from oid 27 from oid 34 current snap is 303 2022-04-22T17:36:13.070 INFO:tasks.rados.rados.0.smithi115.stdout:3137: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:36:13.071 INFO:tasks.rados.rados.0.smithi115.stdout:3137: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:36:13.071 INFO:tasks.rados.rados.0.smithi115.stdout:3137: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:36:13.071 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 1397 (ObjNum 912 snap 299 seq_num 912) dirty exists 2022-04-22T17:36:13.071 INFO:tasks.rados.rados.0.smithi115.stdout:3138: expect (ObjNum 908 snap 299 seq_num 908) 2022-04-22T17:36:13.094 INFO:tasks.rados.rados.0.smithi115.stdout:3139: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:36:13.120 INFO:tasks.rados.rados.0.smithi115.stdout:3140: finishing write tid 1 to smithi11541303-24 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3140: finishing write tid 2 to smithi11541303-24 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3140: finishing write tid 3 to smithi11541303-24 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3140: finishing write tid 4 to smithi11541303-24 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3137: done (4 left) 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3138: done (3 left) 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3139: done (2 left) 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3142: snap_create 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:3140: finishing write tid 5 to smithi11541303-24 2022-04-22T17:36:13.121 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1921 (ObjNum 937 snap 303 seq_num 937) dirty exists 2022-04-22T17:36:13.122 INFO:tasks.rados.rados.0.smithi115.stdout:3140: left oid 24 (ObjNum 937 snap 303 seq_num 937) 2022-04-22T17:36:13.122 INFO:tasks.rados.rados.0.smithi115.stdout:3141: finishing copy_from to smithi11541303-27 2022-04-22T17:36:13.122 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1835 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:36:13.122 INFO:tasks.rados.rados.0.smithi115.stdout:3141: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:36:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:12 smithi115 bash[45495]: cluster 2022-04-22T17:36:11.326683+0000 mgr.y (mgr.34107) 391 : cluster [DBG] pgmap v699: 121 pgs: 121 active+clean; 133 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.3 MiB/s wr, 21 op/s 2022-04-22T17:36:13.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:12 smithi115 bash[45495]: cluster 2022-04-22T17:36:11.946461+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2022-04-22T17:36:13.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:12 smithi115 bash[46934]: cluster 2022-04-22T17:36:11.326683+0000 mgr.y (mgr.34107) 391 : cluster [DBG] pgmap v699: 121 pgs: 121 active+clean; 133 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.3 MiB/s wr, 21 op/s 2022-04-22T17:36:13.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:12 smithi115 bash[46934]: cluster 2022-04-22T17:36:11.946461+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2022-04-22T17:36:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:13.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:13.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:13.529Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:13.935 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 bash[49572]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-prometheus.a 2022-04-22T17:36:13.936 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 bash[49614]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-prometheus-a 2022-04-22T17:36:13.936 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 bash[49627]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-prometheus.a 2022-04-22T17:36:13.936 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@prometheus.a.service: Succeeded. 2022-04-22T17:36:13.936 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 systemd[1]: Stopped Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3140: done (2 left) 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3141: done (1 left) 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3142: done (0 left) 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3143: read oid 38 snap 302 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3143: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 304 2022-04-22T17:36:13.967 INFO:tasks.rados.rados.0.smithi115.stdout:3144: seq_num 938 ranges {9756672=516096,10272768=417792,10690560=385024} 2022-04-22T17:36:14.025 INFO:tasks.rados.rados.0.smithi115.stdout:3144: writing smithi11541303-27 from 9756672 to 10272768 tid 1 2022-04-22T17:36:14.028 INFO:tasks.rados.rados.0.smithi115.stdout:3144: writing smithi11541303-27 from 10272768 to 10690560 tid 2 2022-04-22T17:36:14.030 INFO:tasks.rados.rados.0.smithi115.stdout:3144: writing smithi11541303-27 from 10690560 to 11075584 tid 3 2022-04-22T17:36:14.030 INFO:tasks.rados.rados.0.smithi115.stdout:3145: snap_remove snap 303 2022-04-22T17:36:14.266 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:13 smithi121 systemd[1]: Started Ceph prometheus.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:36:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:13 smithi121 bash[36902]: audit 2022-04-22T17:36:13.304668+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:36:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:13 smithi121 bash[36902]: audit 2022-04-22T17:36:13.310178+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:36:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[45495]: audit 2022-04-22T17:36:13.304668+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:36:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[45495]: audit 2022-04-22T17:36:13.310178+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:36:14.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[46934]: audit 2022-04-22T17:36:13.304668+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:36:14.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:13 smithi115 bash[46934]: audit 2022-04-22T17:36:13.310178+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:36:14.949 INFO:tasks.rados.rados.0.smithi115.stdout:3145: done (2 left) 2022-04-22T17:36:14.949 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 24 current snap is 304 2022-04-22T17:36:14.950 INFO:tasks.rados.rados.0.smithi115.stdout:3146: seq_num 939 ranges {8683520=532480,9216000=704512,9920512=98304} 2022-04-22T17:36:15.005 INFO:tasks.rados.rados.0.smithi115.stdout:3146: writing smithi11541303-24 from 8683520 to 9216000 tid 1 2022-04-22T17:36:15.006 INFO:tasks.rados.rados.0.smithi115.stdout:3146: writing smithi11541303-24 from 9216000 to 9920512 tid 2 2022-04-22T17:36:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:3146: writing smithi11541303-24 from 9920512 to 10018816 tid 3 2022-04-22T17:36:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 304 2022-04-22T17:36:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:3147: seq_num 940 ranges {811008=548864,1359872=589824,1949696=212992} 2022-04-22T17:36:15.015 INFO:tasks.rados.rados.0.smithi115.stdout:3147: writing smithi11541303-10 from 811008 to 1359872 tid 1 2022-04-22T17:36:15.019 INFO:tasks.rados.rados.0.smithi115.stdout:3147: writing smithi11541303-10 from 1359872 to 1949696 tid 2 2022-04-22T17:36:15.021 INFO:tasks.rados.rados.0.smithi115.stdout:3147: writing smithi11541303-10 from 1949696 to 2162688 tid 3 2022-04-22T17:36:15.021 INFO:tasks.rados.rados.0.smithi115.stdout:3148: snap_create 2022-04-22T17:36:15.021 INFO:tasks.rados.rados.0.smithi115.stdout:3143: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:3144: finishing write tid 1 to smithi11541303-27 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:3144: finishing write tid 2 to smithi11541303-27 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:3144: finishing write tid 3 to smithi11541303-27 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:3144: finishing write tid 4 to smithi11541303-27 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:3144: finishing write tid 5 to smithi11541303-27 2022-04-22T17:36:15.060 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 1939 (ObjNum 938 snap 304 seq_num 938) dirty exists 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3144: left oid 27 (ObjNum 938 snap 304 seq_num 938) 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: finishing write tid 1 to smithi11541303-24 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: finishing write tid 2 to smithi11541303-24 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: finishing write tid 3 to smithi11541303-24 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: finishing write tid 4 to smithi11541303-24 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: finishing write tid 5 to smithi11541303-24 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1925 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3146: left oid 24 (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:15.061 INFO:tasks.rados.rados.0.smithi115.stdout:3147: finishing write tid 1 to smithi11541303-10 2022-04-22T17:36:15.062 INFO:tasks.rados.rados.0.smithi115.stdout:3147: finishing write tid 2 to smithi11541303-10 2022-04-22T17:36:15.062 INFO:tasks.rados.rados.0.smithi115.stdout:3147: finishing write tid 3 to smithi11541303-10 2022-04-22T17:36:15.062 INFO:tasks.rados.rados.0.smithi115.stdout:3147: finishing write tid 4 to smithi11541303-10 2022-04-22T17:36:15.063 INFO:tasks.rados.rados.0.smithi115.stdout:3147: finishing write tid 5 to smithi11541303-10 2022-04-22T17:36:15.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2009 (ObjNum 940 snap 304 seq_num 940) dirty exists 2022-04-22T17:36:15.063 INFO:tasks.rados.rados.0.smithi115.stdout:3147: left oid 10 (ObjNum 940 snap 304 seq_num 940) 2022-04-22T17:36:15.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: cluster 2022-04-22T17:36:13.327760+0000 mgr.y (mgr.34107) 392 : cluster [DBG] pgmap v701: 121 pgs: 121 active+clean; 144 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 26 op/s 2022-04-22T17:36:15.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: cluster 2022-04-22T17:36:13.954316+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2022-04-22T17:36:15.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: audit 2022-04-22T17:36:13.965450+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:15.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: audit 2022-04-22T17:36:13.966761+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:15.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: audit 2022-04-22T17:36:13.968046+0000 mon.a (mon.0) 1287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:15.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: audit 2022-04-22T17:36:13.968885+0000 mon.a (mon.0) 1288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:15.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: audit 2022-04-22T17:36:14.382966+0000 mon.a (mon.0) 1289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:15.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:14 smithi121 bash[36902]: cluster 2022-04-22T17:36:14.934710+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2022-04-22T17:36:15.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: cluster 2022-04-22T17:36:13.327760+0000 mgr.y (mgr.34107) 392 : cluster [DBG] pgmap v701: 121 pgs: 121 active+clean; 144 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 26 op/s 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: cluster 2022-04-22T17:36:13.954316+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: audit 2022-04-22T17:36:13.965450+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: audit 2022-04-22T17:36:13.966761+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: audit 2022-04-22T17:36:13.968046+0000 mon.a (mon.0) 1287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: audit 2022-04-22T17:36:13.968885+0000 mon.a (mon.0) 1288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: audit 2022-04-22T17:36:14.382966+0000 mon.a (mon.0) 1289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[45495]: cluster 2022-04-22T17:36:14.934710+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: cluster 2022-04-22T17:36:13.327760+0000 mgr.y (mgr.34107) 392 : cluster [DBG] pgmap v701: 121 pgs: 121 active+clean; 144 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 26 op/s 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: cluster 2022-04-22T17:36:13.954316+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: audit 2022-04-22T17:36:13.965450+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: audit 2022-04-22T17:36:13.966761+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: audit 2022-04-22T17:36:13.968046+0000 mon.a (mon.0) 1287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: audit 2022-04-22T17:36:13.968885+0000 mon.a (mon.0) 1288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: audit 2022-04-22T17:36:14.382966+0000 mon.a (mon.0) 1289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:15.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:14 smithi115 bash[46934]: cluster 2022-04-22T17:36:14.934710+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2022-04-22T17:36:15.766 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:518 level=info host_details="(Linux 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 smithi121 (none))" 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.339Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.343Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.343Z caller=main.go:923 level=info msg="Starting TSDB ..." 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.343Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock 2022-04-22T17:36:15.767 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.346Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.348Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.348Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.944µs 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.348Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.348Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.484Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.541Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.542Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.543Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=49.665µs wal_replay_duration=194.768436ms total_replay_duration=194.833421ms 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.546Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2022-04-22T17:36:15.768 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.546Z caller=main.go:947 level=info msg="TSDB started" 2022-04-22T17:36:15.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.546Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2022-04-22T17:36:15.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.571Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.855406ms db_storage=1.342µs remote_storage=2.02µs web_handler=680ns query_engine=1.222µs scrape=513.53µs scrape_sd=47.172µs notify=67.744µs notify_sd=21.145µs rules=23.638951ms 2022-04-22T17:36:15.769 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 17:36:15 smithi121 bash[49697]: ts=2022-04-22T17:36:15.571Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2022-04-22T17:36:15.975 INFO:tasks.rados.rados.0.smithi115.stdout:3143: done (4 left) 2022-04-22T17:36:16.017 INFO:tasks.rados.rados.0.smithi115.stdout:3144: done (3 left) 2022-04-22T17:36:16.017 INFO:tasks.rados.rados.0.smithi115.stdout:3146: done (2 left) 2022-04-22T17:36:16.017 INFO:tasks.rados.rados.0.smithi115.stdout:3147: done (1 left) 2022-04-22T17:36:16.017 INFO:tasks.rados.rados.0.smithi115.stdout:3148: done (0 left) 2022-04-22T17:36:16.017 INFO:tasks.rados.rados.0.smithi115.stdout:3149: rollback oid 46 current snap is 305 2022-04-22T17:36:16.018 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 300 2022-04-22T17:36:16.018 INFO:tasks.rados.rados.0.smithi115.stdout:3150: read oid 21 snap -1 2022-04-22T17:36:16.018 INFO:tasks.rados.rados.0.smithi115.stdout:3150: expect (ObjNum 886 snap 289 seq_num 886) 2022-04-22T17:36:16.018 INFO:tasks.rados.rados.0.smithi115.stdout:3151: delete oid 26 current snap is 305 2022-04-22T17:36:16.019 INFO:tasks.rados.rados.0.smithi115.stdout:3149: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:36:16.019 INFO:tasks.rados.rados.0.smithi115.stdout:3151: done (2 left) 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3152: read oid 3 snap 304 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3152: expect (ObjNum 923 snap 301 seq_num 923) 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3153: setattr oid 23 current snap is 305 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3149: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3149: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1859 (ObjNum 914 snap 299 seq_num 914) dirty exists 2022-04-22T17:36:16.020 INFO:tasks.rados.rados.0.smithi115.stdout:3150: expect (ObjNum 886 snap 289 seq_num 886) 2022-04-22T17:36:16.076 INFO:tasks.rados.rados.0.smithi115.stdout:3152: expect (ObjNum 923 snap 301 seq_num 923) 2022-04-22T17:36:16.167 INFO:tasks.rados.rados.0.smithi115.stdout:3149: done (3 left) 2022-04-22T17:36:16.167 INFO:tasks.rados.rados.0.smithi115.stdout:3150: done (2 left) 2022-04-22T17:36:16.167 INFO:tasks.rados.rados.0.smithi115.stdout:3152: done (1 left) 2022-04-22T17:36:16.168 INFO:tasks.rados.rados.0.smithi115.stdout:3154: copy_from oid 27 from oid 10 current snap is 305 2022-04-22T17:36:16.168 INFO:tasks.rados.rados.0.smithi115.stdout:3155: delete oid 9 current snap is 305 2022-04-22T17:36:16.172 INFO:tasks.rados.rados.0.smithi115.stdout:3155: done (2 left) 2022-04-22T17:36:16.172 INFO:tasks.rados.rados.0.smithi115.stdout:3156: snap_create 2022-04-22T17:36:16.173 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 1860 (ObjNum 1466389875 snap 1817141101 seq_num 1633775951) dirty exists 2022-04-22T17:36:16.187 INFO:tasks.rados.rados.0.smithi115.stdout:3154: finishing copy_from to smithi11541303-27 2022-04-22T17:36:16.188 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2009 (ObjNum 940 snap 304 seq_num 940) dirty exists 2022-04-22T17:36:16.191 INFO:tasks.rados.rados.0.smithi115.stdout:3154: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:36:16.985 INFO:tasks.rados.rados.0.smithi115.stdout:3153: done (2 left) 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3154: done (1 left) 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3156: done (0 left) 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3157: copy_from oid 14 from oid 19 current snap is 306 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3158: rmattr oid 49 current snap is 306 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3158: done (1 left) 2022-04-22T17:36:17.003 INFO:tasks.rados.rados.0.smithi115.stdout:3159: delete oid 29 current snap is 306 2022-04-22T17:36:17.004 INFO:tasks.rados.rados.0.smithi115.stdout:3159: done (1 left) 2022-04-22T17:36:17.004 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 306 2022-04-22T17:36:17.004 INFO:tasks.rados.rados.0.smithi115.stdout:3160: seq_num 944 ranges {1794048=614400,2408448=573440,2981888=229376} 2022-04-22T17:36:17.006 INFO:tasks.rados.rados.0.smithi115.stdout:3160: writing smithi11541303-39 from 1794048 to 2408448 tid 1 2022-04-22T17:36:17.011 INFO:tasks.rados.rados.0.smithi115.stdout:3160: writing smithi11541303-39 from 2408448 to 2981888 tid 2 2022-04-22T17:36:17.012 INFO:tasks.rados.rados.0.smithi115.stdout:3160: writing smithi11541303-39 from 2981888 to 3211264 tid 3 2022-04-22T17:36:17.012 INFO:tasks.rados.rados.0.smithi115.stdout:3161: delete oid 5 current snap is 306 2022-04-22T17:36:17.012 INFO:tasks.rados.rados.0.smithi115.stdout:3157: finishing copy_from to smithi11541303-14 2022-04-22T17:36:17.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1754 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:17.012 INFO:tasks.rados.rados.0.smithi115.stdout:3157: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:36:17.015 INFO:tasks.rados.rados.0.smithi115.stdout:3160: finishing write tid 1 to smithi11541303-39 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3157: done (2 left) 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3161: done (1 left) 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3162: rmattr oid 47 current snap is 306 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3163: read oid 5 snap 304 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3163: expect (ObjNum -1115276272 snap 22075 seq_num 3178404032) 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3164: rollback oid 35 current snap is 306 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 299 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3165: snap_create 2022-04-22T17:36:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3160: finishing write tid 2 to smithi11541303-39 2022-04-22T17:36:17.018 INFO:tasks.rados.rados.0.smithi115.stdout:3160: finishing write tid 3 to smithi11541303-39 2022-04-22T17:36:17.018 INFO:tasks.rados.rados.0.smithi115.stdout:3163: expect (ObjNum -1115276272 snap 22075 seq_num 3178404032) 2022-04-22T17:36:17.020 INFO:tasks.rados.rados.0.smithi115.stdout:3160: finishing write tid 4 to smithi11541303-39 2022-04-22T17:36:17.020 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2010 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:36:17.024 INFO:tasks.rados.rados.0.smithi115.stdout:3160: finishing write tid 5 to smithi11541303-39 2022-04-22T17:36:17.024 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1859 (ObjNum 944 snap 306 seq_num 944) dirty exists 2022-04-22T17:36:17.025 INFO:tasks.rados.rados.0.smithi115.stdout:3160: left oid 39 (ObjNum 944 snap 306 seq_num 944) 2022-04-22T17:36:17.025 INFO:tasks.rados.rados.0.smithi115.stdout:3164: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:36:17.025 INFO:tasks.rados.rados.0.smithi115.stdout:3164: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:36:17.025 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1861 (ObjNum -1114809056 snap 22075 seq_num 2815700080) dirty dne 2022-04-22T17:36:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:16 smithi121 bash[36902]: cluster 2022-04-22T17:36:15.328669+0000 mgr.y (mgr.34107) 393 : cluster [DBG] pgmap v704: 121 pgs: 121 active+clean; 159 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 7.3 MiB/s wr, 30 op/s 2022-04-22T17:36:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:16 smithi121 bash[36902]: cluster 2022-04-22T17:36:15.961665+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2022-04-22T17:36:17.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:16 smithi115 bash[46934]: cluster 2022-04-22T17:36:15.328669+0000 mgr.y (mgr.34107) 393 : cluster [DBG] pgmap v704: 121 pgs: 121 active+clean; 159 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 7.3 MiB/s wr, 30 op/s 2022-04-22T17:36:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:16 smithi115 bash[46934]: cluster 2022-04-22T17:36:15.961665+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2022-04-22T17:36:17.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:16 smithi115 bash[45495]: cluster 2022-04-22T17:36:15.328669+0000 mgr.y (mgr.34107) 393 : cluster [DBG] pgmap v704: 121 pgs: 121 active+clean; 159 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 7.3 MiB/s wr, 30 op/s 2022-04-22T17:36:17.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:16 smithi115 bash[45495]: cluster 2022-04-22T17:36:15.961665+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3160: done (4 left) 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3162: done (3 left) 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3163: done (2 left) 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3164: done (1 left) 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3165: done (0 left) 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3166: copy_from oid 5 from oid 24 current snap is 307 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3167: copy_from oid 22 from oid 8 current snap is 307 2022-04-22T17:36:17.997 INFO:tasks.rados.rados.0.smithi115.stdout:3168: delete oid 37 current snap is 307 2022-04-22T17:36:18.005 INFO:tasks.rados.rados.0.smithi115.stdout:3168: done (2 left) 2022-04-22T17:36:18.005 INFO:tasks.rados.rados.0.smithi115.stdout:3169: copy_from oid 30 from oid 3 current snap is 307 2022-04-22T17:36:18.005 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 307 2022-04-22T17:36:18.006 INFO:tasks.rados.rados.0.smithi115.stdout:3170: seq_num 947 ranges {892928=663552,1556480=630784,2187264=172032} 2022-04-22T17:36:18.016 INFO:tasks.rados.rados.0.smithi115.stdout:3170: writing smithi11541303-38 from 892928 to 1556480 tid 1 2022-04-22T17:36:18.023 INFO:tasks.rados.rados.0.smithi115.stdout:3170: writing smithi11541303-38 from 1556480 to 2187264 tid 2 2022-04-22T17:36:18.023 INFO:tasks.rados.rados.0.smithi115.stdout:3170: writing smithi11541303-38 from 2187264 to 2359296 tid 3 2022-04-22T17:36:18.023 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 307 2022-04-22T17:36:18.023 INFO:tasks.rados.rados.0.smithi115.stdout:3171: seq_num 948 ranges {0=679936,679936=516096,1196032=286720} 2022-04-22T17:36:18.026 INFO:tasks.rados.rados.0.smithi115.stdout:3171: writing smithi11541303-49 from 0 to 679936 tid 1 2022-04-22T17:36:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:3171: writing smithi11541303-49 from 679936 to 1196032 tid 2 2022-04-22T17:36:18.032 INFO:tasks.rados.rados.0.smithi115.stdout:3171: writing smithi11541303-49 from 1196032 to 1482752 tid 3 2022-04-22T17:36:18.032 INFO:tasks.rados.rados.0.smithi115.stdout:3172: rollback oid 36 current snap is 307 2022-04-22T17:36:18.032 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 300 2022-04-22T17:36:18.032 INFO:tasks.rados.rados.0.smithi115.stdout:3167: finishing copy_from to smithi11541303-22 2022-04-22T17:36:18.032 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1841 (ObjNum 921 snap 301 seq_num 921) dirty exists 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3167: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3169: finishing copy_from to smithi11541303-30 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1852 (ObjNum 923 snap 301 seq_num 923) dirty exists 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3169: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3170: finishing write tid 1 to smithi11541303-38 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3170: finishing write tid 2 to smithi11541303-38 2022-04-22T17:36:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3167: done (5 left) 2022-04-22T17:36:18.034 INFO:tasks.rados.rados.0.smithi115.stdout:3169: done (4 left) 2022-04-22T17:36:18.034 INFO:tasks.rados.rados.0.smithi115.stdout:3173: rmattr oid 33 current snap is 307 2022-04-22T17:36:18.034 INFO:tasks.rados.rados.0.smithi115.stdout:3173: done (4 left) 2022-04-22T17:36:18.034 INFO:tasks.rados.rados.0.smithi115.stdout:3174: setattr oid 44 current snap is 307 2022-04-22T17:36:18.035 INFO:tasks.rados.rados.0.smithi115.stdout:3170: finishing write tid 3 to smithi11541303-38 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:3170: finishing write tid 4 to smithi11541303-38 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:3170: finishing write tid 5 to smithi11541303-38 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1760 (ObjNum 947 snap 307 seq_num 947) dirty exists 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:3170: left oid 38 (ObjNum 947 snap 307 seq_num 947) 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:3170: done (4 left) 2022-04-22T17:36:18.036 INFO:tasks.rados.rados.0.smithi115.stdout:3175: snap_create 2022-04-22T17:36:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:3172: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:36:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:3172: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:36:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1855 (ObjNum 1 snap 32692 seq_num 3179079352) dirty exists 2022-04-22T17:36:18.040 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1308 (ObjNum -1115806128 snap 22075 seq_num 3179161184) dirty exists 2022-04-22T17:36:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3171: finishing write tid 1 to smithi11541303-49 2022-04-22T17:36:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3171: finishing write tid 2 to smithi11541303-49 2022-04-22T17:36:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3171: finishing write tid 3 to smithi11541303-49 2022-04-22T17:36:18.067 INFO:tasks.rados.rados.0.smithi115.stdout:3171: finishing write tid 4 to smithi11541303-49 2022-04-22T17:36:18.068 INFO:tasks.rados.rados.0.smithi115.stdout:3166: finishing copy_from to smithi11541303-5 2022-04-22T17:36:18.068 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 1925 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:18.069 INFO:tasks.rados.rados.0.smithi115.stdout:3166: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:36:18.070 INFO:tasks.rados.rados.0.smithi115.stdout:3171: finishing write tid 5 to smithi11541303-49 2022-04-22T17:36:18.070 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2026 (ObjNum 948 snap 307 seq_num 948) dirty exists 2022-04-22T17:36:18.071 INFO:tasks.rados.rados.0.smithi115.stdout:3171: left oid 49 (ObjNum 948 snap 307 seq_num 948) 2022-04-22T17:36:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:17 smithi121 bash[36902]: cluster 2022-04-22T17:36:16.971403+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2022-04-22T17:36:18.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:17 smithi115 bash[45495]: cluster 2022-04-22T17:36:16.971403+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2022-04-22T17:36:18.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:17 smithi115 bash[46934]: cluster 2022-04-22T17:36:16.971403+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3166: done (4 left) 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3171: done (3 left) 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3172: done (2 left) 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3174: done (1 left) 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3175: done (0 left) 2022-04-22T17:36:19.002 INFO:tasks.rados.rados.0.smithi115.stdout:3176: delete oid 42 current snap is 308 2022-04-22T17:36:19.010 INFO:tasks.rados.rados.0.smithi115.stdout:3176: done (0 left) 2022-04-22T17:36:19.010 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 308 2022-04-22T17:36:19.010 INFO:tasks.rados.rados.0.smithi115.stdout:3177: seq_num 950 ranges {0=720896,720896=679936,1400832=122880} 2022-04-22T17:36:19.014 INFO:tasks.rados.rados.0.smithi115.stdout:3177: writing smithi11541303-26 from 0 to 720896 tid 1 2022-04-22T17:36:19.019 INFO:tasks.rados.rados.0.smithi115.stdout:3177: writing smithi11541303-26 from 720896 to 1400832 tid 2 2022-04-22T17:36:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:3177: writing smithi11541303-26 from 1400832 to 1523712 tid 3 2022-04-22T17:36:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 308 2022-04-22T17:36:19.021 INFO:tasks.rados.rados.0.smithi115.stdout:3178: seq_num 951 ranges {0=737280,737280=614400,1351680=188416} 2022-04-22T17:36:19.025 INFO:tasks.rados.rados.0.smithi115.stdout:3178: writing smithi11541303-42 from 0 to 737280 tid 1 2022-04-22T17:36:19.029 INFO:tasks.rados.rados.0.smithi115.stdout:3178: writing smithi11541303-42 from 737280 to 1351680 tid 2 2022-04-22T17:36:19.030 INFO:tasks.rados.rados.0.smithi115.stdout:3178: writing smithi11541303-42 from 1351680 to 1540096 tid 3 2022-04-22T17:36:19.031 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 308 2022-04-22T17:36:19.031 INFO:tasks.rados.rados.0.smithi115.stdout:3177: finishing write tid 1 to smithi11541303-26 2022-04-22T17:36:19.032 INFO:tasks.rados.rados.0.smithi115.stdout:3177: finishing write tid 2 to smithi11541303-26 2022-04-22T17:36:19.033 INFO:tasks.rados.rados.0.smithi115.stdout:3177: finishing write tid 3 to smithi11541303-26 2022-04-22T17:36:19.033 INFO:tasks.rados.rados.0.smithi115.stdout:3179: seq_num 952 ranges {3170304=753664,3923968=499712,4423680=303104} 2022-04-22T17:36:19.052 INFO:tasks.rados.rados.0.smithi115.stdout:3179: writing smithi11541303-22 from 3170304 to 3923968 tid 1 2022-04-22T17:36:19.055 INFO:tasks.rados.rados.0.smithi115.stdout:3179: writing smithi11541303-22 from 3923968 to 4423680 tid 2 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3179: writing smithi11541303-22 from 4423680 to 4726784 tid 3 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3180: delete oid 44 current snap is 308 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3177: finishing write tid 4 to smithi11541303-26 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3177: finishing write tid 5 to smithi11541303-26 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1402 (ObjNum 950 snap 308 seq_num 950) dirty exists 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3177: left oid 26 (ObjNum 950 snap 308 seq_num 950) 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3178: finishing write tid 1 to smithi11541303-42 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3178: finishing write tid 2 to smithi11541303-42 2022-04-22T17:36:19.057 INFO:tasks.rados.rados.0.smithi115.stdout:3178: finishing write tid 3 to smithi11541303-42 2022-04-22T17:36:19.058 INFO:tasks.rados.rados.0.smithi115.stdout:3178: finishing write tid 4 to smithi11541303-42 2022-04-22T17:36:19.058 INFO:tasks.rados.rados.0.smithi115.stdout:3178: finishing write tid 5 to smithi11541303-42 2022-04-22T17:36:19.058 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2033 (ObjNum 951 snap 308 seq_num 951) dirty exists 2022-04-22T17:36:19.058 INFO:tasks.rados.rados.0.smithi115.stdout:3178: left oid 42 (ObjNum 951 snap 308 seq_num 951) 2022-04-22T17:36:19.064 INFO:tasks.rados.rados.0.smithi115.stdout:3177: done (3 left) 2022-04-22T17:36:19.064 INFO:tasks.rados.rados.0.smithi115.stdout:3178: done (2 left) 2022-04-22T17:36:19.064 INFO:tasks.rados.rados.0.smithi115.stdout:3180: done (1 left) 2022-04-22T17:36:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:3181: read oid 18 snap -1 2022-04-22T17:36:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:3181: expect deleted 2022-04-22T17:36:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 308 2022-04-22T17:36:19.065 INFO:tasks.rados.rados.0.smithi115.stdout:3182: seq_num 954 ranges {4538368=786432,5324800=671744,5996544=131072} 2022-04-22T17:36:19.094 INFO:tasks.rados.rados.0.smithi115.stdout:3182: writing smithi11541303-46 from 4538368 to 5324800 tid 1 2022-04-22T17:36:19.098 INFO:tasks.rados.rados.0.smithi115.stdout:3182: writing smithi11541303-46 from 5324800 to 5996544 tid 2 2022-04-22T17:36:19.100 INFO:tasks.rados.rados.0.smithi115.stdout:3182: writing smithi11541303-46 from 5996544 to 6127616 tid 3 2022-04-22T17:36:19.100 INFO:tasks.rados.rados.0.smithi115.stdout:3183: read oid 29 snap -1 2022-04-22T17:36:19.100 INFO:tasks.rados.rados.0.smithi115.stdout:3183: expect deleted 2022-04-22T17:36:19.101 INFO:tasks.rados.rados.0.smithi115.stdout:3184: read oid 34 snap -1 2022-04-22T17:36:19.101 INFO:tasks.rados.rados.0.smithi115.stdout:3184: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:36:19.101 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 308 2022-04-22T17:36:19.101 INFO:tasks.rados.rados.0.smithi115.stdout:3185: seq_num 955 ranges {1523712=802816,2326528=557056,2883584=245760} 2022-04-22T17:36:19.113 INFO:tasks.rados.rados.0.smithi115.stdout:3185: writing smithi11541303-26 from 1523712 to 2326528 tid 1 2022-04-22T17:36:19.116 INFO:tasks.rados.rados.0.smithi115.stdout:3185: writing smithi11541303-26 from 2326528 to 2883584 tid 2 2022-04-22T17:36:19.121 INFO:tasks.rados.rados.0.smithi115.stdout:3185: writing smithi11541303-26 from 2883584 to 3129344 tid 3 2022-04-22T17:36:19.121 INFO:tasks.rados.rados.0.smithi115.stdout:3186: copy_from oid 30 from oid 9 current snap is 308 2022-04-22T17:36:19.121 INFO:tasks.rados.rados.0.smithi115.stdout:3179: finishing write tid 1 to smithi11541303-22 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: finishing write tid 2 to smithi11541303-22 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: finishing write tid 3 to smithi11541303-22 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: finishing write tid 4 to smithi11541303-22 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: finishing write tid 5 to smithi11541303-22 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1845 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: left oid 22 (ObjNum 952 snap 308 seq_num 952) 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3182: finishing write tid 1 to smithi11541303-46 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3182: finishing write tid 2 to smithi11541303-46 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3182: finishing write tid 3 to smithi11541303-46 2022-04-22T17:36:19.122 INFO:tasks.rados.rados.0.smithi115.stdout:3179: done (6 left) 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3181: done (5 left) 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3183: done (4 left) 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3187: setattr oid 19 current snap is 308 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3182: finishing write tid 4 to smithi11541303-46 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3182: finishing write tid 5 to smithi11541303-46 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 1864 (ObjNum 954 snap 308 seq_num 954) dirty exists 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3182: left oid 46 (ObjNum 954 snap 308 seq_num 954) 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3185: finishing write tid 1 to smithi11541303-26 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3186: finishing copy_from to smithi11541303-30 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3186: got expected ENOENT (src dne) 2022-04-22T17:36:19.123 INFO:tasks.rados.rados.0.smithi115.stdout:3182: done (4 left) 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3188: read oid 38 snap 305 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3188: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3189: read oid 43 snap -1 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3189: expect (ObjNum 925 snap 301 seq_num 925) 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3190: copy_from oid 11 from oid 33 current snap is 308 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3191: rollback oid 50 current snap is 308 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 305 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3186: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3186: done (7 left) 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:3192: rollback oid 29 current snap is 308 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 305 2022-04-22T17:36:19.124 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 308 2022-04-22T17:36:19.125 INFO:tasks.rados.rados.0.smithi115.stdout:3193: seq_num 956 ranges {3153920=417792,3571712=434176,4005888=770048} 2022-04-22T17:36:19.141 INFO:tasks.rados.rados.0.smithi115.stdout:3193: writing smithi11541303-12 from 3153920 to 3571712 tid 1 2022-04-22T17:36:19.144 INFO:tasks.rados.rados.0.smithi115.stdout:3193: writing smithi11541303-12 from 3571712 to 4005888 tid 2 2022-04-22T17:36:19.149 INFO:tasks.rados.rados.0.smithi115.stdout:3193: writing smithi11541303-12 from 4005888 to 4775936 tid 3 2022-04-22T17:36:19.149 INFO:tasks.rados.rados.0.smithi115.stdout:3194: delete oid 16 current snap is 308 2022-04-22T17:36:19.149 INFO:tasks.rados.rados.0.smithi115.stdout:3185: finishing write tid 2 to smithi11541303-26 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:3185: finishing write tid 3 to smithi11541303-26 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:3185: finishing write tid 4 to smithi11541303-26 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:3192: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:3185: finishing write tid 5 to smithi11541303-26 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1406 (ObjNum 955 snap 308 seq_num 955) dirty exists 2022-04-22T17:36:19.150 INFO:tasks.rados.rados.0.smithi115.stdout:3185: left oid 26 (ObjNum 955 snap 308 seq_num 955) 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:3190: finishing copy_from to smithi11541303-11 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 1393 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:3192: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1756 (ObjNum 917 snap 299 seq_num 917) dirty exists 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:3190: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:36:19.151 INFO:tasks.rados.rados.0.smithi115.stdout:3188: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:19.190 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1761 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:19.190 INFO:tasks.rados.rados.0.smithi115.stdout:3184: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:36:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:18 smithi121 bash[36902]: cluster 2022-04-22T17:36:17.329620+0000 mgr.y (mgr.34107) 394 : cluster [DBG] pgmap v707: 121 pgs: 121 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 32 op/s 2022-04-22T17:36:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:18 smithi121 bash[36902]: cluster 2022-04-22T17:36:17.984234+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2022-04-22T17:36:19.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:18 smithi115 bash[45495]: cluster 2022-04-22T17:36:17.329620+0000 mgr.y (mgr.34107) 394 : cluster [DBG] pgmap v707: 121 pgs: 121 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 32 op/s 2022-04-22T17:36:19.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:18 smithi115 bash[45495]: cluster 2022-04-22T17:36:17.984234+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2022-04-22T17:36:19.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:18 smithi115 bash[46934]: cluster 2022-04-22T17:36:17.329620+0000 mgr.y (mgr.34107) 394 : cluster [DBG] pgmap v707: 121 pgs: 121 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 32 op/s 2022-04-22T17:36:19.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:18 smithi115 bash[46934]: cluster 2022-04-22T17:36:17.984234+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2022-04-22T17:36:20.074 INFO:tasks.rados.rados.0.smithi115.stdout:3191: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 1667 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: finishing write tid 1 to smithi11541303-12 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: finishing write tid 2 to smithi11541303-12 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: finishing write tid 3 to smithi11541303-12 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: finishing write tid 4 to smithi11541303-12 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: finishing write tid 5 to smithi11541303-12 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1765 (ObjNum 956 snap 308 seq_num 956) dirty exists 2022-04-22T17:36:20.075 INFO:tasks.rados.rados.0.smithi115.stdout:3193: left oid 12 (ObjNum 956 snap 308 seq_num 956) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3184: done (9 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3185: done (8 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3187: done (7 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3188: done (6 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3190: done (5 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3191: done (4 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3192: done (3 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3193: done (2 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3194: done (1 left) 2022-04-22T17:36:20.076 INFO:tasks.rados.rados.0.smithi115.stdout:3195: read oid 16 snap -1 2022-04-22T17:36:20.077 INFO:tasks.rados.rados.0.smithi115.stdout:3195: expect deleted 2022-04-22T17:36:20.077 INFO:tasks.rados.rados.0.smithi115.stdout:3196: delete oid 30 current snap is 308 2022-04-22T17:36:20.077 INFO:tasks.rados.rados.0.smithi115.stdout:3189: expect (ObjNum 925 snap 301 seq_num 925) 2022-04-22T17:36:20.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:20 smithi121 bash[36902]: cluster 2022-04-22T17:36:18.989941+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2022-04-22T17:36:20.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:20 smithi121 bash[36902]: audit 2022-04-22T17:36:19.385945+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:20.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:20 smithi121 bash[36902]: audit 2022-04-22T17:36:19.400903+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:20 smithi121 bash[36902]: cluster 2022-04-22T17:36:19.935903+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2022-04-22T17:36:20.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:20 smithi121 bash[36902]: audit 2022-04-22T17:36:19.982966+0000 mon.a (mon.0) 1298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[46934]: cluster 2022-04-22T17:36:18.989941+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2022-04-22T17:36:20.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[46934]: audit 2022-04-22T17:36:19.385945+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[46934]: audit 2022-04-22T17:36:19.400903+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[46934]: cluster 2022-04-22T17:36:19.935903+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[46934]: audit 2022-04-22T17:36:19.982966+0000 mon.a (mon.0) 1298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[45495]: cluster 2022-04-22T17:36:18.989941+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[45495]: audit 2022-04-22T17:36:19.385945+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[45495]: audit 2022-04-22T17:36:19.400903+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[45495]: cluster 2022-04-22T17:36:19.935903+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2022-04-22T17:36:20.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:19 smithi115 bash[45495]: audit 2022-04-22T17:36:19.982966+0000 mon.a (mon.0) 1298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:3189: done (2 left) 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:3195: done (1 left) 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:3196: done (0 left) 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:3197: copy_from oid 16 from oid 14 current snap is 308 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 308 2022-04-22T17:36:20.336 INFO:tasks.rados.rados.0.smithi115.stdout:3198: seq_num 959 ranges {4538368=466944,5005312=540672,5545984=557056,6103040=106496} 2022-04-22T17:36:20.372 INFO:tasks.rados.rados.0.smithi115.stdout:3198: writing smithi11541303-1 from 4538368 to 5005312 tid 1 2022-04-22T17:36:20.374 INFO:tasks.rados.rados.0.smithi115.stdout:3198: writing smithi11541303-1 from 5005312 to 5545984 tid 2 2022-04-22T17:36:20.378 INFO:tasks.rados.rados.0.smithi115.stdout:3198: writing smithi11541303-1 from 5545984 to 6103040 tid 3 2022-04-22T17:36:20.379 INFO:tasks.rados.rados.0.smithi115.stdout:3198: writing smithi11541303-1 from 6103040 to 6209536 tid 4 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3199: read oid 27 snap -1 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3199: expect (ObjNum 940 snap 304 seq_num 940) 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3200: read oid 23 snap 302 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3200: expect deleted 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 308 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3197: finishing copy_from to smithi11541303-16 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1754 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:20.380 INFO:tasks.rados.rados.0.smithi115.stdout:3201: seq_num 960 ranges {0=483328,483328=425984,909312=761856,1671168=16384} 2022-04-22T17:36:20.384 INFO:tasks.rados.rados.0.smithi115.stdout:3201: writing smithi11541303-37 from 0 to 483328 tid 1 2022-04-22T17:36:20.385 INFO:tasks.rados.rados.0.smithi115.stdout:3201: writing smithi11541303-37 from 483328 to 909312 tid 2 2022-04-22T17:36:20.391 INFO:tasks.rados.rados.0.smithi115.stdout:3201: writing smithi11541303-37 from 909312 to 1671168 tid 3 2022-04-22T17:36:20.391 INFO:tasks.rados.rados.0.smithi115.stdout:3201: writing smithi11541303-37 from 1671168 to 1687552 tid 4 2022-04-22T17:36:20.391 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 308 2022-04-22T17:36:20.391 INFO:tasks.rados.rados.0.smithi115.stdout:3197: finishing copy_from racing read to smithi11541303-16 2022-04-22T17:36:20.392 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 1 to smithi11541303-1 2022-04-22T17:36:20.392 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 2 to smithi11541303-1 2022-04-22T17:36:20.392 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 3 to smithi11541303-1 2022-04-22T17:36:20.392 INFO:tasks.rados.rados.0.smithi115.stdout:3202: seq_num 961 ranges {0=499712,499712=712704,1212416=491520} 2022-04-22T17:36:20.395 INFO:tasks.rados.rados.0.smithi115.stdout:3202: writing smithi11541303-30 from 0 to 499712 tid 1 2022-04-22T17:36:20.398 INFO:tasks.rados.rados.0.smithi115.stdout:3202: writing smithi11541303-30 from 499712 to 1212416 tid 2 2022-04-22T17:36:20.402 INFO:tasks.rados.rados.0.smithi115.stdout:3202: writing smithi11541303-30 from 1212416 to 1703936 tid 3 2022-04-22T17:36:20.402 INFO:tasks.rados.rados.0.smithi115.stdout:3197: done (5 left) 2022-04-22T17:36:20.402 INFO:tasks.rados.rados.0.smithi115.stdout:3200: done (4 left) 2022-04-22T17:36:20.402 INFO:tasks.rados.rados.0.smithi115.stdout:3203: read oid 29 snap -1 2022-04-22T17:36:20.402 INFO:tasks.rados.rados.0.smithi115.stdout:3203: expect (ObjNum 917 snap 299 seq_num 917) 2022-04-22T17:36:20.403 INFO:tasks.rados.rados.0.smithi115.stdout:3204: snap_remove snap 302 2022-04-22T17:36:21.016 INFO:tasks.rados.rados.0.smithi115.stdout:3204: done (5 left) 2022-04-22T17:36:21.024 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 308 2022-04-22T17:36:21.024 INFO:tasks.rados.rados.0.smithi115.stdout:3205: seq_num 962 ranges {5120000=516096,5636096=589824,6225920=614400} 2022-04-22T17:36:21.049 INFO:tasks.rados.rados.0.smithi115.stdout:3205: writing smithi11541303-43 from 5120000 to 5636096 tid 1 2022-04-22T17:36:21.052 INFO:tasks.rados.rados.0.smithi115.stdout:3205: writing smithi11541303-43 from 5636096 to 6225920 tid 2 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3205: writing smithi11541303-43 from 6225920 to 6840320 tid 3 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3206: read oid 5 snap -1 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3206: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3207: read oid 42 snap 300 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3207: expect (ObjNum 902 snap 295 seq_num 902) 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3208: delete oid 7 current snap is 308 2022-04-22T17:36:21.057 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 4 to smithi11541303-1 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 5 to smithi11541303-1 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 1 to smithi11541303-37 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 2 to smithi11541303-37 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3198: finishing write tid 6 to smithi11541303-1 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1860 (ObjNum 959 snap 308 seq_num 959) dirty exists 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3198: left oid 1 (ObjNum 959 snap 308 seq_num 959) 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 3 to smithi11541303-37 2022-04-22T17:36:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3199: expect (ObjNum 940 snap 304 seq_num 940) 2022-04-22T17:36:21.134 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 4 to smithi11541303-37 2022-04-22T17:36:21.143 INFO:tasks.rados.rados.0.smithi115.stdout:3202: finishing write tid 1 to smithi11541303-30 2022-04-22T17:36:21.143 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 5 to smithi11541303-37 2022-04-22T17:36:21.143 INFO:tasks.rados.rados.0.smithi115.stdout:3201: finishing write tid 6 to smithi11541303-37 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1867 (ObjNum 960 snap 308 seq_num 960) dirty exists 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3201: left oid 37 (ObjNum 960 snap 308 seq_num 960) 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3198: done (8 left) 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3199: done (7 left) 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3201: done (6 left) 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3208: done (5 left) 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 308 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3202: finishing write tid 2 to smithi11541303-30 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3202: finishing write tid 3 to smithi11541303-30 2022-04-22T17:36:21.144 INFO:tasks.rados.rados.0.smithi115.stdout:3203: expect (ObjNum 917 snap 299 seq_num 917) 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3202: finishing write tid 4 to smithi11541303-30 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3202: finishing write tid 5 to smithi11541303-30 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2014 (ObjNum 961 snap 308 seq_num 961) dirty exists 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3202: left oid 30 (ObjNum 961 snap 308 seq_num 961) 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3205: finishing write tid 1 to smithi11541303-43 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3205: finishing write tid 2 to smithi11541303-43 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3205: finishing write tid 3 to smithi11541303-43 2022-04-22T17:36:21.162 INFO:tasks.rados.rados.0.smithi115.stdout:3205: finishing write tid 4 to smithi11541303-43 2022-04-22T17:36:21.163 INFO:tasks.rados.rados.0.smithi115.stdout:3205: finishing write tid 5 to smithi11541303-43 2022-04-22T17:36:21.163 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1735 (ObjNum 962 snap 308 seq_num 962) dirty exists 2022-04-22T17:36:21.163 INFO:tasks.rados.rados.0.smithi115.stdout:3205: left oid 43 (ObjNum 962 snap 308 seq_num 962) 2022-04-22T17:36:21.163 INFO:tasks.rados.rados.0.smithi115.stdout:3206: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:21.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:21 smithi115 bash[45495]: cluster 2022-04-22T17:36:19.330302+0000 mgr.y (mgr.34107) 395 : cluster [DBG] pgmap v710: 121 pgs: 121 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:21.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:21 smithi115 bash[46934]: cluster 2022-04-22T17:36:19.330302+0000 mgr.y (mgr.34107) 395 : cluster [DBG] pgmap v710: 121 pgs: 121 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:21 smithi121 bash[36902]: cluster 2022-04-22T17:36:19.330302+0000 mgr.y (mgr.34107) 395 : cluster [DBG] pgmap v710: 121 pgs: 121 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2022-04-22T17:36:21.845 INFO:tasks.rados.rados.0.smithi115.stdout:3207: expect (ObjNum 902 snap 295 seq_num 902) 2022-04-22T17:36:21.862 INFO:tasks.rados.rados.0.smithi115.stdout:3209: seq_num 964 ranges {3170304=548864,3719168=761856,4481024=442368} 2022-04-22T17:36:21.891 INFO:tasks.rados.rados.0.smithi115.stdout:3209: writing smithi11541303-8 from 3170304 to 3719168 tid 1 2022-04-22T17:36:21.895 INFO:tasks.rados.rados.0.smithi115.stdout:3209: writing smithi11541303-8 from 3719168 to 4481024 tid 2 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3209: writing smithi11541303-8 from 4481024 to 4923392 tid 3 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3202: done (5 left) 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3203: done (4 left) 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3205: done (3 left) 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3206: done (2 left) 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:3207: done (1 left) 2022-04-22T17:36:21.899 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 308 2022-04-22T17:36:21.900 INFO:tasks.rados.rados.0.smithi115.stdout:3210: seq_num 965 ranges {0=565248,565248=647168,1212416=548864,1761280=8192} 2022-04-22T17:36:21.901 INFO:tasks.rados.rados.0.smithi115.stdout:3210: writing smithi11541303-6 from 0 to 565248 tid 1 2022-04-22T17:36:21.905 INFO:tasks.rados.rados.0.smithi115.stdout:3210: writing smithi11541303-6 from 565248 to 1212416 tid 2 2022-04-22T17:36:21.909 INFO:tasks.rados.rados.0.smithi115.stdout:3210: writing smithi11541303-6 from 1212416 to 1761280 tid 3 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3210: writing smithi11541303-6 from 1761280 to 1769472 tid 4 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3211: setattr oid 35 current snap is 308 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3209: finishing write tid 1 to smithi11541303-8 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3209: finishing write tid 2 to smithi11541303-8 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3209: finishing write tid 3 to smithi11541303-8 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3209: finishing write tid 4 to smithi11541303-8 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 1 to smithi11541303-6 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:3209: finishing write tid 5 to smithi11541303-8 2022-04-22T17:36:21.910 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1865 (ObjNum 964 snap 308 seq_num 964) dirty exists 2022-04-22T17:36:21.911 INFO:tasks.rados.rados.0.smithi115.stdout:3209: left oid 8 (ObjNum 964 snap 308 seq_num 964) 2022-04-22T17:36:21.911 INFO:tasks.rados.rados.0.smithi115.stdout:3209: done (2 left) 2022-04-22T17:36:21.911 INFO:tasks.rados.rados.0.smithi115.stdout:3212: rollback oid 14 current snap is 308 2022-04-22T17:36:21.911 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 14 to 304 2022-04-22T17:36:21.912 INFO:tasks.rados.rados.0.smithi115.stdout:3213: setattr oid 22 current snap is 308 2022-04-22T17:36:21.912 INFO:tasks.rados.rados.0.smithi115.stdout:3214: snap_create 2022-04-22T17:36:21.915 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 2 to smithi11541303-6 2022-04-22T17:36:21.915 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 3 to smithi11541303-6 2022-04-22T17:36:21.915 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 4 to smithi11541303-6 2022-04-22T17:36:21.915 INFO:tasks.rados.rados.0.smithi115.stdout:3212: finishing rollback tid 0 to smithi11541303-14 2022-04-22T17:36:21.916 INFO:tasks.rados.rados.0.smithi115.stdout:3212: finishing rollback tid 1 to smithi11541303-14 2022-04-22T17:36:21.916 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 5 to smithi11541303-6 2022-04-22T17:36:21.917 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 1868 (ObjNum -1116761568 snap 22075 seq_num 2) dirty exists 2022-04-22T17:36:21.917 INFO:tasks.rados.rados.0.smithi115.stdout:3212: finishing rollback tid 2 to smithi11541303-14 2022-04-22T17:36:21.917 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1759 (ObjNum 836 snap 275 seq_num 836) dirty exists 2022-04-22T17:36:21.918 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1846 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:36:21.918 INFO:tasks.rados.rados.0.smithi115.stdout:3210: finishing write tid 6 to smithi11541303-6 2022-04-22T17:36:21.918 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1930 (ObjNum 965 snap 308 seq_num 965) dirty exists 2022-04-22T17:36:21.918 INFO:tasks.rados.rados.0.smithi115.stdout:3210: left oid 6 (ObjNum 965 snap 308 seq_num 965) 2022-04-22T17:36:22.034 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:21.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:22.034 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:21.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:22.034 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:21 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:21.634Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3210: done (4 left) 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3211: done (3 left) 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3212: done (2 left) 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3213: done (1 left) 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3214: done (0 left) 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 309 2022-04-22T17:36:22.038 INFO:tasks.rados.rados.0.smithi115.stdout:3215: seq_num 966 ranges {0=581632,581632=581632,1163264=622592} 2022-04-22T17:36:22.042 INFO:tasks.rados.rados.0.smithi115.stdout:3215: writing smithi11541303-45 from 0 to 581632 tid 1 2022-04-22T17:36:22.045 INFO:tasks.rados.rados.0.smithi115.stdout:3215: writing smithi11541303-45 from 581632 to 1163264 tid 2 2022-04-22T17:36:22.049 INFO:tasks.rados.rados.0.smithi115.stdout:3215: writing smithi11541303-45 from 1163264 to 1785856 tid 3 2022-04-22T17:36:22.049 INFO:tasks.rados.rados.0.smithi115.stdout:3216: read oid 41 snap 304 2022-04-22T17:36:22.050 INFO:tasks.rados.rados.0.smithi115.stdout:3216: expect deleted 2022-04-22T17:36:22.050 INFO:tasks.rados.rados.0.smithi115.stdout:3217: delete oid 6 current snap is 309 2022-04-22T17:36:22.054 INFO:tasks.rados.rados.0.smithi115.stdout:3216: done (2 left) 2022-04-22T17:36:22.054 INFO:tasks.rados.rados.0.smithi115.stdout:3217: done (1 left) 2022-04-22T17:36:22.054 INFO:tasks.rados.rados.0.smithi115.stdout:3218: copy_from oid 29 from oid 8 current snap is 309 2022-04-22T17:36:22.054 INFO:tasks.rados.rados.0.smithi115.stdout:3219: setattr oid 15 current snap is 309 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3215: finishing write tid 1 to smithi11541303-45 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3215: finishing write tid 2 to smithi11541303-45 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3220: read oid 38 snap 307 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3220: expect (ObjNum 947 snap 307 seq_num 947) 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3221: copy_from oid 39 from oid 5 current snap is 309 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 309 2022-04-22T17:36:22.057 INFO:tasks.rados.rados.0.smithi115.stdout:3222: seq_num 968 ranges {4603904=614400,5218304=745472,5963776=458752} 2022-04-22T17:36:22.093 INFO:tasks.rados.rados.0.smithi115.stdout:3222: writing smithi11541303-28 from 4603904 to 5218304 tid 1 2022-04-22T17:36:22.099 INFO:tasks.rados.rados.0.smithi115.stdout:3222: writing smithi11541303-28 from 5218304 to 5963776 tid 2 2022-04-22T17:36:22.101 INFO:tasks.rados.rados.0.smithi115.stdout:3222: writing smithi11541303-28 from 5963776 to 6422528 tid 3 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3223: delete oid 33 current snap is 309 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3215: finishing write tid 3 to smithi11541303-45 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3215: finishing write tid 4 to smithi11541303-45 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1548 (ObjNum 838 snap 276 seq_num 838) dirty exists 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3215: finishing write tid 5 to smithi11541303-45 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2018 (ObjNum 966 snap 309 seq_num 966) dirty exists 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3215: left oid 45 (ObjNum 966 snap 309 seq_num 966) 2022-04-22T17:36:22.102 INFO:tasks.rados.rados.0.smithi115.stdout:3220: expect (ObjNum 947 snap 307 seq_num 947) 2022-04-22T17:36:22.194 INFO:tasks.rados.rados.0.smithi115.stdout:3218: finishing copy_from to smithi11541303-29 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1865 (ObjNum 964 snap 308 seq_num 964) dirty exists 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3218: finishing copy_from racing read to smithi11541303-29 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: finishing write tid 1 to smithi11541303-28 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: finishing write tid 2 to smithi11541303-28 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: finishing write tid 3 to smithi11541303-28 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: finishing write tid 4 to smithi11541303-28 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: finishing write tid 5 to smithi11541303-28 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1869 (ObjNum 968 snap 309 seq_num 968) dirty exists 2022-04-22T17:36:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3222: left oid 28 (ObjNum 968 snap 309 seq_num 968) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3221: finishing copy_from to smithi11541303-39 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1925 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3221: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3215: done (6 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3218: done (5 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3219: done (4 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3220: done (3 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3221: done (2 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3222: done (1 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3223: done (0 left) 2022-04-22T17:36:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3224: copy_from oid 26 from oid 49 current snap is 309 2022-04-22T17:36:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:3225: setattr oid 24 current snap is 309 2022-04-22T17:36:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 309 2022-04-22T17:36:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:3226: seq_num 970 ranges {229376=655360,884736=516096,1400832=688128} 2022-04-22T17:36:22.207 INFO:tasks.rados.rados.0.smithi115.stdout:3226: writing smithi11541303-16 from 229376 to 884736 tid 1 2022-04-22T17:36:22.210 INFO:tasks.rados.rados.0.smithi115.stdout:3226: writing smithi11541303-16 from 884736 to 1400832 tid 2 2022-04-22T17:36:22.216 INFO:tasks.rados.rados.0.smithi115.stdout:3226: writing smithi11541303-16 from 1400832 to 2088960 tid 3 2022-04-22T17:36:22.217 INFO:tasks.rados.rados.0.smithi115.stdout:3227: delete oid 41 current snap is 309 2022-04-22T17:36:22.217 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1932 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:22.217 INFO:tasks.rados.rados.0.smithi115.stdout:3224: finishing copy_from to smithi11541303-26 2022-04-22T17:36:22.217 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2026 (ObjNum 948 snap 307 seq_num 948) dirty exists 2022-04-22T17:36:22.218 INFO:tasks.rados.rados.0.smithi115.stdout:3224: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:36:22.218 INFO:tasks.rados.rados.0.smithi115.stdout:3226: finishing write tid 1 to smithi11541303-16 2022-04-22T17:36:22.218 INFO:tasks.rados.rados.0.smithi115.stdout:3226: finishing write tid 2 to smithi11541303-16 2022-04-22T17:36:22.219 INFO:tasks.rados.rados.0.smithi115.stdout:3224: done (3 left) 2022-04-22T17:36:22.220 INFO:tasks.rados.rados.0.smithi115.stdout:3225: done (2 left) 2022-04-22T17:36:22.220 INFO:tasks.rados.rados.0.smithi115.stdout:3227: done (1 left) 2022-04-22T17:36:22.220 INFO:tasks.rados.rados.0.smithi115.stdout:3228: rmattr oid 10 current snap is 309 2022-04-22T17:36:22.220 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 309 2022-04-22T17:36:22.221 INFO:tasks.rados.rados.0.smithi115.stdout:3229: seq_num 972 ranges {2392064=688128,3080192=688128,3768320=516096} 2022-04-22T17:36:22.237 INFO:tasks.rados.rados.0.smithi115.stdout:3229: writing smithi11541303-3 from 2392064 to 3080192 tid 1 2022-04-22T17:36:22.240 INFO:tasks.rados.rados.0.smithi115.stdout:3229: writing smithi11541303-3 from 3080192 to 3768320 tid 2 2022-04-22T17:36:22.243 INFO:tasks.rados.rados.0.smithi115.stdout:3229: writing smithi11541303-3 from 3768320 to 4284416 tid 3 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:3230: snap_create 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:3226: finishing write tid 3 to smithi11541303-16 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:3226: finishing write tid 4 to smithi11541303-16 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:3226: finishing write tid 5 to smithi11541303-16 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1840 (ObjNum 970 snap 309 seq_num 970) dirty exists 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:3226: left oid 16 (ObjNum 970 snap 309 seq_num 970) 2022-04-22T17:36:22.244 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2011 (ObjNum 940 snap 304 seq_num 940) dirty exists 2022-04-22T17:36:22.248 INFO:tasks.rados.rados.0.smithi115.stdout:3229: finishing write tid 1 to smithi11541303-3 2022-04-22T17:36:22.249 INFO:tasks.rados.rados.0.smithi115.stdout:3229: finishing write tid 2 to smithi11541303-3 2022-04-22T17:36:22.250 INFO:tasks.rados.rados.0.smithi115.stdout:3229: finishing write tid 3 to smithi11541303-3 2022-04-22T17:36:22.251 INFO:tasks.rados.rados.0.smithi115.stdout:3229: finishing write tid 4 to smithi11541303-3 2022-04-22T17:36:22.253 INFO:tasks.rados.rados.0.smithi115.stdout:3229: finishing write tid 5 to smithi11541303-3 2022-04-22T17:36:22.253 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1868 (ObjNum 972 snap 309 seq_num 972) dirty exists 2022-04-22T17:36:22.253 INFO:tasks.rados.rados.0.smithi115.stdout:3229: left oid 3 (ObjNum 972 snap 309 seq_num 972) 2022-04-22T17:36:22.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:22 smithi115 bash[45495]: cluster 2022-04-22T17:36:21.002324+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2022-04-22T17:36:22.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:22 smithi115 bash[46934]: cluster 2022-04-22T17:36:21.002324+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2022-04-22T17:36:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:22 smithi121 bash[36902]: cluster 2022-04-22T17:36:21.002324+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2022-04-22T17:36:23.046 INFO:tasks.rados.rados.0.smithi115.stdout:3226: done (3 left) 2022-04-22T17:36:23.046 INFO:tasks.rados.rados.0.smithi115.stdout:3228: done (2 left) 2022-04-22T17:36:23.047 INFO:tasks.rados.rados.0.smithi115.stdout:3229: done (1 left) 2022-04-22T17:36:23.047 INFO:tasks.rados.rados.0.smithi115.stdout:3230: done (0 left) 2022-04-22T17:36:23.047 INFO:tasks.rados.rados.0.smithi115.stdout:3231: delete oid 25 current snap is 310 2022-04-22T17:36:23.054 INFO:tasks.rados.rados.0.smithi115.stdout:3231: done (0 left) 2022-04-22T17:36:23.054 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 310 2022-04-22T17:36:23.054 INFO:tasks.rados.rados.0.smithi115.stdout:3232: seq_num 974 ranges {1785856=720896,2506752=507904,3014656=671744,3686400=24576} 2022-04-22T17:36:23.069 INFO:tasks.rados.rados.0.smithi115.stdout:3232: writing smithi11541303-45 from 1785856 to 2506752 tid 1 2022-04-22T17:36:23.071 INFO:tasks.rados.rados.0.smithi115.stdout:3232: writing smithi11541303-45 from 2506752 to 3014656 tid 2 2022-04-22T17:36:23.075 INFO:tasks.rados.rados.0.smithi115.stdout:3232: writing smithi11541303-45 from 3014656 to 3686400 tid 3 2022-04-22T17:36:23.076 INFO:tasks.rados.rados.0.smithi115.stdout:3232: writing smithi11541303-45 from 3686400 to 3710976 tid 4 2022-04-22T17:36:23.076 INFO:tasks.rados.rados.0.smithi115.stdout:3233: read oid 17 snap -1 2022-04-22T17:36:23.076 INFO:tasks.rados.rados.0.smithi115.stdout:3233: expect (ObjNum 884 snap 289 seq_num 884) 2022-04-22T17:36:23.076 INFO:tasks.rados.rados.0.smithi115.stdout:3234: snap_create 2022-04-22T17:36:23.083 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 1 to smithi11541303-45 2022-04-22T17:36:23.083 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 2 to smithi11541303-45 2022-04-22T17:36:23.086 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 3 to smithi11541303-45 2022-04-22T17:36:23.090 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 4 to smithi11541303-45 2022-04-22T17:36:23.090 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 5 to smithi11541303-45 2022-04-22T17:36:23.092 INFO:tasks.rados.rados.0.smithi115.stdout:3232: finishing write tid 6 to smithi11541303-45 2022-04-22T17:36:23.092 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2023 (ObjNum 974 snap 310 seq_num 974) dirty exists 2022-04-22T17:36:23.092 INFO:tasks.rados.rados.0.smithi115.stdout:3232: left oid 45 (ObjNum 974 snap 310 seq_num 974) 2022-04-22T17:36:23.104 INFO:tasks.rados.rados.0.smithi115.stdout:3233: expect (ObjNum 884 snap 289 seq_num 884) 2022-04-22T17:36:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:23 smithi121 bash[36902]: cluster 2022-04-22T17:36:21.330785+0000 mgr.y (mgr.34107) 396 : cluster [DBG] pgmap v713: 121 pgs: 121 active+clean; 179 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2022-04-22T17:36:23.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:23 smithi121 bash[36902]: cluster 2022-04-22T17:36:22.025127+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2022-04-22T17:36:23.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[45495]: cluster 2022-04-22T17:36:21.330785+0000 mgr.y (mgr.34107) 396 : cluster [DBG] pgmap v713: 121 pgs: 121 active+clean; 179 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2022-04-22T17:36:23.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[45495]: cluster 2022-04-22T17:36:22.025127+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2022-04-22T17:36:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[46934]: cluster 2022-04-22T17:36:21.330785+0000 mgr.y (mgr.34107) 396 : cluster [DBG] pgmap v713: 121 pgs: 121 active+clean; 179 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2022-04-22T17:36:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[46934]: cluster 2022-04-22T17:36:22.025127+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2022-04-22T17:36:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:23.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:23.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:23 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:23.529Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:24.063 INFO:tasks.rados.rados.0.smithi115.stdout:3232: done (2 left) 2022-04-22T17:36:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:3233: done (1 left) 2022-04-22T17:36:24.064 INFO:tasks.rados.rados.0.smithi115.stdout:3234: done (0 left) 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3235: read oid 5 snap -1 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3235: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3236: read oid 35 snap 306 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3236: expect deleted 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3237: read oid 29 snap 310 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3237: expect (ObjNum 964 snap 308 seq_num 964) 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3238: rmattr oid 24 current snap is 311 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3239: copy_from oid 7 from oid 8 current snap is 311 2022-04-22T17:36:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 311 2022-04-22T17:36:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:3240: seq_num 975 ranges {0=737280,737280=786432,1523712=417792} 2022-04-22T17:36:24.067 INFO:tasks.rados.rados.0.smithi115.stdout:3240: writing smithi11541303-25 from 0 to 737280 tid 1 2022-04-22T17:36:24.072 INFO:tasks.rados.rados.0.smithi115.stdout:3240: writing smithi11541303-25 from 737280 to 1523712 tid 2 2022-04-22T17:36:24.076 INFO:tasks.rados.rados.0.smithi115.stdout:3240: writing smithi11541303-25 from 1523712 to 1941504 tid 3 2022-04-22T17:36:24.076 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 311 2022-04-22T17:36:24.076 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1933 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:24.076 INFO:tasks.rados.rados.0.smithi115.stdout:3241: seq_num 976 ranges {0=753664,753664=671744,1425408=532480} 2022-04-22T17:36:24.081 INFO:tasks.rados.rados.0.smithi115.stdout:3241: writing smithi11541303-36 from 0 to 753664 tid 1 2022-04-22T17:36:24.086 INFO:tasks.rados.rados.0.smithi115.stdout:3241: writing smithi11541303-36 from 753664 to 1425408 tid 2 2022-04-22T17:36:24.090 INFO:tasks.rados.rados.0.smithi115.stdout:3241: writing smithi11541303-36 from 1425408 to 1957888 tid 3 2022-04-22T17:36:24.090 INFO:tasks.rados.rados.0.smithi115.stdout:3238: done (6 left) 2022-04-22T17:36:24.090 INFO:tasks.rados.rados.0.smithi115.stdout:3242: rmattr oid 22 current snap is 311 2022-04-22T17:36:24.090 INFO:tasks.rados.rados.0.smithi115.stdout:3240: finishing write tid 1 to smithi11541303-25 2022-04-22T17:36:24.090 INFO:tasks.rados.rados.0.smithi115.stdout:3240: finishing write tid 2 to smithi11541303-25 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3240: finishing write tid 3 to smithi11541303-25 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3240: finishing write tid 4 to smithi11541303-25 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3236: done (6 left) 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3243: snap_create 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3240: finishing write tid 5 to smithi11541303-25 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1741 (ObjNum 975 snap 311 seq_num 975) dirty exists 2022-04-22T17:36:24.091 INFO:tasks.rados.rados.0.smithi115.stdout:3240: left oid 25 (ObjNum 975 snap 311 seq_num 975) 2022-04-22T17:36:24.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1847 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:36:24.101 INFO:tasks.rados.rados.0.smithi115.stdout:3237: expect (ObjNum 964 snap 308 seq_num 964) 2022-04-22T17:36:24.346 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:24 smithi121 bash[36902]: cluster 2022-04-22T17:36:23.034178+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2022-04-22T17:36:24.365 INFO:tasks.rados.rados.0.smithi115.stdout:3241: finishing write tid 1 to smithi11541303-36 2022-04-22T17:36:24.365 INFO:tasks.rados.rados.0.smithi115.stdout:3241: finishing write tid 2 to smithi11541303-36 2022-04-22T17:36:24.365 INFO:tasks.rados.rados.0.smithi115.stdout:3239: finishing copy_from to smithi11541303-7 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1865 (ObjNum 964 snap 308 seq_num 964) dirty exists 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:3239: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:3241: finishing write tid 3 to smithi11541303-36 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:3241: finishing write tid 4 to smithi11541303-36 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:3241: finishing write tid 5 to smithi11541303-36 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1869 (ObjNum 976 snap 311 seq_num 976) dirty exists 2022-04-22T17:36:24.366 INFO:tasks.rados.rados.0.smithi115.stdout:3241: left oid 36 (ObjNum 976 snap 311 seq_num 976) 2022-04-22T17:36:24.367 INFO:tasks.rados.rados.0.smithi115.stdout:3235: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:24.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:24 smithi115 bash[45495]: cluster 2022-04-22T17:36:23.034178+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2022-04-22T17:36:24.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:24 smithi115 bash[46934]: cluster 2022-04-22T17:36:23.034178+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2022-04-22T17:36:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:3235: done (6 left) 2022-04-22T17:36:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:3237: done (5 left) 2022-04-22T17:36:25.047 INFO:tasks.rados.rados.0.smithi115.stdout:3239: done (4 left) 2022-04-22T17:36:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:3240: done (3 left) 2022-04-22T17:36:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:3241: done (2 left) 2022-04-22T17:36:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:3242: done (1 left) 2022-04-22T17:36:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:3243: done (0 left) 2022-04-22T17:36:25.048 INFO:tasks.rados.rados.0.smithi115.stdout:3244: delete oid 7 current snap is 312 2022-04-22T17:36:25.056 INFO:tasks.rados.rados.0.smithi115.stdout:3244: done (0 left) 2022-04-22T17:36:25.056 INFO:tasks.rados.rados.0.smithi115.stdout:3245: snap_remove snap 311 2022-04-22T17:36:25.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:25 smithi121 bash[36902]: cluster 2022-04-22T17:36:23.331218+0000 mgr.y (mgr.34107) 397 : cluster [DBG] pgmap v716: 121 pgs: 121 active+clean; 205 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.7 MiB/s wr, 53 op/s 2022-04-22T17:36:25.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:25 smithi121 bash[36902]: cluster 2022-04-22T17:36:24.049123+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2022-04-22T17:36:25.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:25 smithi121 bash[36902]: audit 2022-04-22T17:36:24.388038+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:25.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:25 smithi121 bash[36902]: cluster 2022-04-22T17:36:24.937946+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2022-04-22T17:36:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[45495]: cluster 2022-04-22T17:36:23.331218+0000 mgr.y (mgr.34107) 397 : cluster [DBG] pgmap v716: 121 pgs: 121 active+clean; 205 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.7 MiB/s wr, 53 op/s 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[45495]: cluster 2022-04-22T17:36:24.049123+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[45495]: audit 2022-04-22T17:36:24.388038+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[45495]: cluster 2022-04-22T17:36:24.937946+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[46934]: cluster 2022-04-22T17:36:23.331218+0000 mgr.y (mgr.34107) 397 : cluster [DBG] pgmap v716: 121 pgs: 121 active+clean; 205 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.7 MiB/s wr, 53 op/s 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[46934]: cluster 2022-04-22T17:36:24.049123+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[46934]: audit 2022-04-22T17:36:24.388038+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:25 smithi115 bash[46934]: cluster 2022-04-22T17:36:24.937946+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2022-04-22T17:36:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3245: done (0 left) 2022-04-22T17:36:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3246: read oid 20 snap -1 2022-04-22T17:36:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3246: expect deleted 2022-04-22T17:36:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3247: read oid 12 snap -1 2022-04-22T17:36:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3247: expect (ObjNum 956 snap 308 seq_num 956) 2022-04-22T17:36:26.069 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 312 2022-04-22T17:36:26.069 INFO:tasks.rados.rados.0.smithi115.stdout:3248: seq_num 978 ranges {4284416=786432,5070848=442368,5513216=647168,6160384=114688} 2022-04-22T17:36:26.101 INFO:tasks.rados.rados.0.smithi115.stdout:3248: writing smithi11541303-3 from 4284416 to 5070848 tid 1 2022-04-22T17:36:26.104 INFO:tasks.rados.rados.0.smithi115.stdout:3248: writing smithi11541303-3 from 5070848 to 5513216 tid 2 2022-04-22T17:36:26.108 INFO:tasks.rados.rados.0.smithi115.stdout:3248: writing smithi11541303-3 from 5513216 to 6160384 tid 3 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3248: writing smithi11541303-3 from 6160384 to 6275072 tid 4 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3249: read oid 24 snap 310 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3249: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3250: rmattr oid 50 current snap is 312 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3250: done (4 left) 2022-04-22T17:36:26.109 INFO:tasks.rados.rados.0.smithi115.stdout:3251: snap_remove snap 306 2022-04-22T17:36:27.085 INFO:tasks.rados.rados.0.smithi115.stdout:3251: done (4 left) 2022-04-22T17:36:27.088 INFO:tasks.rados.rados.0.smithi115.stdout:3247: expect (ObjNum 956 snap 308 seq_num 956) 2022-04-22T17:36:27.401 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 1 to smithi11541303-3 2022-04-22T17:36:27.401 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 2 to smithi11541303-3 2022-04-22T17:36:27.401 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 3 to smithi11541303-3 2022-04-22T17:36:27.403 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 4 to smithi11541303-3 2022-04-22T17:36:27.403 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 5 to smithi11541303-3 2022-04-22T17:36:27.404 INFO:tasks.rados.rados.0.smithi115.stdout:3252: read oid 27 snap -1 2022-04-22T17:36:27.404 INFO:tasks.rados.rados.0.smithi115.stdout:3252: expect (ObjNum 940 snap 304 seq_num 940) 2022-04-22T17:36:27.404 INFO:tasks.rados.rados.0.smithi115.stdout:3246: done (4 left) 2022-04-22T17:36:27.404 INFO:tasks.rados.rados.0.smithi115.stdout:3247: done (3 left) 2022-04-22T17:36:27.404 INFO:tasks.rados.rados.0.smithi115.stdout:3253: snap_remove snap 305 2022-04-22T17:36:27.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:27 smithi121 bash[36902]: cluster 2022-04-22T17:36:25.332185+0000 mgr.y (mgr.34107) 398 : cluster [DBG] pgmap v719: 121 pgs: 121 active+clean; 211 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.4 MiB/s wr, 54 op/s 2022-04-22T17:36:27.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:27 smithi121 bash[36902]: cluster 2022-04-22T17:36:26.053590+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2022-04-22T17:36:27.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:27 smithi115 bash[45495]: cluster 2022-04-22T17:36:25.332185+0000 mgr.y (mgr.34107) 398 : cluster [DBG] pgmap v719: 121 pgs: 121 active+clean; 211 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.4 MiB/s wr, 54 op/s 2022-04-22T17:36:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:27 smithi115 bash[45495]: cluster 2022-04-22T17:36:26.053590+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2022-04-22T17:36:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:27 smithi115 bash[46934]: cluster 2022-04-22T17:36:25.332185+0000 mgr.y (mgr.34107) 398 : cluster [DBG] pgmap v719: 121 pgs: 121 active+clean; 211 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 8.4 MiB/s wr, 54 op/s 2022-04-22T17:36:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:27 smithi115 bash[46934]: cluster 2022-04-22T17:36:26.053590+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3253: done (3 left) 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3254: read oid 37 snap 304 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3254: expect (ObjNum 914 snap 299 seq_num 914) 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3255: read oid 8 snap -1 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3255: expect (ObjNum 964 snap 308 seq_num 964) 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3256: read oid 33 snap -1 2022-04-22T17:36:28.099 INFO:tasks.rados.rados.0.smithi115.stdout:3256: expect deleted 2022-04-22T17:36:28.100 INFO:tasks.rados.rados.0.smithi115.stdout:3257: read oid 11 snap 309 2022-04-22T17:36:28.100 INFO:tasks.rados.rados.0.smithi115.stdout:3257: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:28.100 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 312 2022-04-22T17:36:28.100 INFO:tasks.rados.rados.0.smithi115.stdout:3258: seq_num 979 ranges {4923392=802816,5726208=729088,6455296=458752,6914048=16384} 2022-04-22T17:36:28.131 INFO:tasks.rados.rados.0.smithi115.stdout:3258: writing smithi11541303-29 from 4923392 to 5726208 tid 1 2022-04-22T17:36:28.136 INFO:tasks.rados.rados.0.smithi115.stdout:3258: writing smithi11541303-29 from 5726208 to 6455296 tid 2 2022-04-22T17:36:28.139 INFO:tasks.rados.rados.0.smithi115.stdout:3258: writing smithi11541303-29 from 6455296 to 6914048 tid 3 2022-04-22T17:36:28.139 INFO:tasks.rados.rados.0.smithi115.stdout:3258: writing smithi11541303-29 from 6914048 to 6930432 tid 4 2022-04-22T17:36:28.139 INFO:tasks.rados.rados.0.smithi115.stdout:3259: rollback oid 16 current snap is 312 2022-04-22T17:36:28.140 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 309 2022-04-22T17:36:28.140 INFO:tasks.rados.rados.0.smithi115.stdout:3248: finishing write tid 6 to smithi11541303-3 2022-04-22T17:36:28.140 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1873 (ObjNum 978 snap 312 seq_num 978) dirty exists 2022-04-22T17:36:28.140 INFO:tasks.rados.rados.0.smithi115.stdout:3248: left oid 3 (ObjNum 978 snap 312 seq_num 978) 2022-04-22T17:36:28.140 INFO:tasks.rados.rados.0.smithi115.stdout:3249: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:28.323 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:36:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:28 smithi121 bash[36902]: cluster 2022-04-22T17:36:27.071377+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2022-04-22T17:36:28.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:28 smithi115 bash[46934]: cluster 2022-04-22T17:36:27.071377+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2022-04-22T17:36:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:28 smithi115 bash[45495]: cluster 2022-04-22T17:36:27.071377+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2022-04-22T17:36:28.664 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:36:28.664 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (12m) 34s ago 13m 13.0M - 0.20.0 0881eb8f169f 527525c1fd4e 2022-04-22T17:36:28.664 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (13m) 9s ago 13m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (90s) 34s ago 14m 12.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (62s) 9s ago 14m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (8m) 9s ago 18m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (9m) 34s ago 20m 452M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (8m) 34s ago 20m 82.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:36:28.665 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (7m) 9s ago 18m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (7m) 34s ago 19m 66.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (38s) 34s ago 13m - - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (33s) 9s ago 13m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6m) 34s ago 18m 412M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5m) 34s ago 17m 333M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5m) 34s ago 17m 301M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4m) 34s ago 16m 188M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3m) 9s ago 16m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:36:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3m) 9s ago 15m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:36:28.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2m) 9s ago 15m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:36:28.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (115s) 9s ago 14m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:36:28.667 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (14s) 9s ago 13m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:36:28.868 INFO:tasks.rados.rados.0.smithi115.stdout:3252: expect (ObjNum 940 snap 304 seq_num 940) 2022-04-22T17:36:28.940 INFO:tasks.rados.rados.0.smithi115.stdout:3257: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:28.959 INFO:tasks.rados.rados.0.smithi115.stdout:3255: expect (ObjNum 964 snap 308 seq_num 964) 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:36:29.076 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:36:29.077 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: cluster 2022-04-22T17:36:27.332965+0000 mgr.y (mgr.34107) 399 : cluster [DBG] pgmap v722: 121 pgs: 121 active+clean; 234 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 40 op/s 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: cluster 2022-04-22T17:36:28.081062+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.825896+0000 mon.a (mon.0) 1308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.834661+0000 mon.a (mon.0) 1309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.844700+0000 mon.a (mon.0) 1310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:29.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.847100+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.848969+0000 mon.a (mon.0) 1312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.850044+0000 mon.a (mon.0) 1313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.850858+0000 mon.a (mon.0) 1314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.852404+0000 mon.a (mon.0) 1315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.854535+0000 mon.a (mon.0) 1316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.855621+0000 mon.a (mon.0) 1317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.856761+0000 mon.a (mon.0) 1318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.857945+0000 mon.a (mon.0) 1319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.858995+0000 mon.a (mon.0) 1320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.860048+0000 mon.a (mon.0) 1321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.861172+0000 mon.a (mon.0) 1322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:28.862268+0000 mon.a (mon.0) 1323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[45495]: audit 2022-04-22T17:36:29.063829+0000 mon.a (mon.0) 1324 : audit [DBG] from='client.? 172.21.15.115:0/2213078181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: cluster 2022-04-22T17:36:27.332965+0000 mgr.y (mgr.34107) 399 : cluster [DBG] pgmap v722: 121 pgs: 121 active+clean; 234 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 40 op/s 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: cluster 2022-04-22T17:36:28.081062+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.825896+0000 mon.a (mon.0) 1308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.834661+0000 mon.a (mon.0) 1309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.844700+0000 mon.a (mon.0) 1310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.847100+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.848969+0000 mon.a (mon.0) 1312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.850044+0000 mon.a (mon.0) 1313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.850858+0000 mon.a (mon.0) 1314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.852404+0000 mon.a (mon.0) 1315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.854535+0000 mon.a (mon.0) 1316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.855621+0000 mon.a (mon.0) 1317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.856761+0000 mon.a (mon.0) 1318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.857945+0000 mon.a (mon.0) 1319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.858995+0000 mon.a (mon.0) 1320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.860048+0000 mon.a (mon.0) 1321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.861172+0000 mon.a (mon.0) 1322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:28.862268+0000 mon.a (mon.0) 1323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:29 smithi115 bash[46934]: audit 2022-04-22T17:36:29.063829+0000 mon.a (mon.0) 1324 : audit [DBG] from='client.? 172.21.15.115:0/2213078181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.221 INFO:tasks.rados.rados.0.smithi115.stdout:3259: finishing rollback tid 0 to smithi11541303-16 2022-04-22T17:36:29.221 INFO:tasks.rados.rados.0.smithi115.stdout:3259: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:36:29.221 INFO:tasks.rados.rados.0.smithi115.stdout:3259: finishing rollback tid 2 to smithi11541303-16 2022-04-22T17:36:29.221 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1843 (ObjNum 970 snap 309 seq_num 970) dirty exists 2022-04-22T17:36:29.221 INFO:tasks.rados.rados.0.smithi115.stdout:3254: expect (ObjNum 914 snap 299 seq_num 914) 2022-04-22T17:36:29.426 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 1 to smithi11541303-29 2022-04-22T17:36:29.467 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 2 to smithi11541303-29 2022-04-22T17:36:29.467 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 3 to smithi11541303-29 2022-04-22T17:36:29.468 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 4 to smithi11541303-29 2022-04-22T17:36:29.468 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 5 to smithi11541303-29 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:3258: finishing write tid 6 to smithi11541303-29 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1874 (ObjNum 979 snap 312 seq_num 979) dirty exists 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:3258: left oid 29 (ObjNum 979 snap 312 seq_num 979) 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:3248: done (8 left) 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:3249: done (7 left) 2022-04-22T17:36:29.469 INFO:tasks.rados.rados.0.smithi115.stdout:3252: done (6 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3254: done (5 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3255: done (4 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3256: done (3 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3257: done (2 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3258: done (1 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3259: done (0 left) 2022-04-22T17:36:29.471 INFO:tasks.rados.rados.0.smithi115.stdout:3260: setattr oid 13 current snap is 312 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 312 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:3261: seq_num 980 ranges {3121152=417792,3538944=614400,4153344=663552,4816896=327680} 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:3261: writing smithi11541303-48 from 3121152 to 3538944 tid 1 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:3261: writing smithi11541303-48 from 3538944 to 4153344 tid 2 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:3261: writing smithi11541303-48 from 4153344 to 4816896 tid 3 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:3261: writing smithi11541303-48 from 4816896 to 5144576 tid 4 2022-04-22T17:36:29.473 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 312 2022-04-22T17:36:29.474 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1751 (ObjNum 887 snap 290 seq_num 887) dirty exists 2022-04-22T17:36:29.474 INFO:tasks.rados.rados.0.smithi115.stdout:3262: seq_num 981 ranges {4923392=434176,5357568=540672,5898240=466944,6365184=524288,6889472=73728} 2022-04-22T17:36:29.492 INFO:tasks.rados.rados.0.smithi115.stdout:3262: writing smithi11541303-8 from 4923392 to 5357568 tid 1 2022-04-22T17:36:29.495 INFO:tasks.rados.rados.0.smithi115.stdout:3262: writing smithi11541303-8 from 5357568 to 5898240 tid 2 2022-04-22T17:36:29.498 INFO:tasks.rados.rados.0.smithi115.stdout:3262: writing smithi11541303-8 from 5898240 to 6365184 tid 3 2022-04-22T17:36:29.501 INFO:tasks.rados.rados.0.smithi115.stdout:3262: writing smithi11541303-8 from 6365184 to 6889472 tid 4 2022-04-22T17:36:29.502 INFO:tasks.rados.rados.0.smithi115.stdout:3262: writing smithi11541303-8 from 6889472 to 6963200 tid 5 2022-04-22T17:36:29.502 INFO:tasks.rados.rados.0.smithi115.stdout:3260: done (2 left) 2022-04-22T17:36:29.502 INFO:tasks.rados.rados.0.smithi115.stdout:3263: delete oid 14 current snap is 312 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 1 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 2 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 3 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 4 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 5 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: finishing write tid 6 to smithi11541303-48 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2028 (ObjNum 980 snap 312 seq_num 980) dirty exists 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3261: left oid 48 (ObjNum 980 snap 312 seq_num 980) 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 1 to smithi11541303-8 2022-04-22T17:36:29.503 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 2 to smithi11541303-8 2022-04-22T17:36:29.504 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 3 to smithi11541303-8 2022-04-22T17:36:29.504 INFO:tasks.rados.rados.0.smithi115.stdout:3261: done (2 left) 2022-04-22T17:36:29.504 INFO:tasks.rados.rados.0.smithi115.stdout:3263: done (1 left) 2022-04-22T17:36:29.505 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 312 2022-04-22T17:36:29.505 INFO:tasks.rados.rados.0.smithi115.stdout:3264: seq_num 983 ranges {2162688=466944,2629632=712704,3342336=434176,3776512=458752} 2022-04-22T17:36:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: cluster 2022-04-22T17:36:27.332965+0000 mgr.y (mgr.34107) 399 : cluster [DBG] pgmap v722: 121 pgs: 121 active+clean; 234 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 40 op/s 2022-04-22T17:36:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: cluster 2022-04-22T17:36:28.081062+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2022-04-22T17:36:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.825896+0000 mon.a (mon.0) 1308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.834661+0000 mon.a (mon.0) 1309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.844700+0000 mon.a (mon.0) 1310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.847100+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.848969+0000 mon.a (mon.0) 1312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.850044+0000 mon.a (mon.0) 1313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.850858+0000 mon.a (mon.0) 1314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.852404+0000 mon.a (mon.0) 1315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.854535+0000 mon.a (mon.0) 1316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.855621+0000 mon.a (mon.0) 1317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.856761+0000 mon.a (mon.0) 1318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.857945+0000 mon.a (mon.0) 1319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.858995+0000 mon.a (mon.0) 1320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.860048+0000 mon.a (mon.0) 1321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.861172+0000 mon.a (mon.0) 1322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:28.862268+0000 mon.a (mon.0) 1323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:29 smithi121 bash[36902]: audit 2022-04-22T17:36:29.063829+0000 mon.a (mon.0) 1324 : audit [DBG] from='client.? 172.21.15.115:0/2213078181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:36:29.521 INFO:tasks.rados.rados.0.smithi115.stdout:3264: writing smithi11541303-10 from 2162688 to 2629632 tid 1 2022-04-22T17:36:29.527 INFO:tasks.rados.rados.0.smithi115.stdout:3264: writing smithi11541303-10 from 2629632 to 3342336 tid 2 2022-04-22T17:36:29.530 INFO:tasks.rados.rados.0.smithi115.stdout:3264: writing smithi11541303-10 from 3342336 to 3776512 tid 3 2022-04-22T17:36:29.532 INFO:tasks.rados.rados.0.smithi115.stdout:3264: writing smithi11541303-10 from 3776512 to 4235264 tid 4 2022-04-22T17:36:29.532 INFO:tasks.rados.rados.0.smithi115.stdout:3265: snap_create 2022-04-22T17:36:29.533 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 4 to smithi11541303-8 2022-04-22T17:36:29.533 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 5 to smithi11541303-8 2022-04-22T17:36:29.534 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 6 to smithi11541303-8 2022-04-22T17:36:29.534 INFO:tasks.rados.rados.0.smithi115.stdout:3262: finishing write tid 7 to smithi11541303-8 2022-04-22T17:36:29.534 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1876 (ObjNum 981 snap 312 seq_num 981) dirty exists 2022-04-22T17:36:29.535 INFO:tasks.rados.rados.0.smithi115.stdout:3262: left oid 8 (ObjNum 981 snap 312 seq_num 981) 2022-04-22T17:36:29.535 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 1 to smithi11541303-10 2022-04-22T17:36:29.538 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 2 to smithi11541303-10 2022-04-22T17:36:29.538 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 3 to smithi11541303-10 2022-04-22T17:36:29.539 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 4 to smithi11541303-10 2022-04-22T17:36:29.539 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 5 to smithi11541303-10 2022-04-22T17:36:29.541 INFO:tasks.rados.rados.0.smithi115.stdout:3264: finishing write tid 6 to smithi11541303-10 2022-04-22T17:36:29.541 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2016 (ObjNum 983 snap 312 seq_num 983) dirty exists 2022-04-22T17:36:29.541 INFO:tasks.rados.rados.0.smithi115.stdout:3264: left oid 10 (ObjNum 983 snap 312 seq_num 983) 2022-04-22T17:36:29.953 INFO:tasks.rados.rados.0.smithi115.stdout:3262: done (2 left) 2022-04-22T17:36:29.953 INFO:tasks.rados.rados.0.smithi115.stdout:3264: done (1 left) 2022-04-22T17:36:29.953 INFO:tasks.rados.rados.0.smithi115.stdout:3265: done (0 left) 2022-04-22T17:36:29.953 INFO:tasks.rados.rados.0.smithi115.stdout:3266: read oid 2 snap 309 2022-04-22T17:36:29.953 INFO:tasks.rados.rados.0.smithi115.stdout:3266: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:36:29.954 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 313 2022-04-22T17:36:29.954 INFO:tasks.rados.rados.0.smithi115.stdout:3267: seq_num 984 ranges {4775936=483328,5259264=598016,5857280=647168,6504448=360448} 2022-04-22T17:36:29.982 INFO:tasks.rados.rados.0.smithi115.stdout:3267: writing smithi11541303-12 from 4775936 to 5259264 tid 1 2022-04-22T17:36:30.036 INFO:tasks.rados.rados.0.smithi115.stdout:3267: writing smithi11541303-12 from 5259264 to 5857280 tid 2 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3267: writing smithi11541303-12 from 5857280 to 6504448 tid 3 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3267: writing smithi11541303-12 from 6504448 to 6864896 tid 4 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3268: read oid 18 snap -1 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3268: expect deleted 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3269: read oid 14 snap -1 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3269: expect deleted 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3270: delete oid 50 current snap is 313 2022-04-22T17:36:30.037 INFO:tasks.rados.rados.0.smithi115.stdout:3266: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:36:30.128 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 1 to smithi11541303-12 2022-04-22T17:36:30.128 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 2 to smithi11541303-12 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 3 to smithi11541303-12 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 4 to smithi11541303-12 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3266: done (4 left) 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3268: done (3 left) 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3269: done (2 left) 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3270: done (1 left) 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3271: read oid 28 snap -1 2022-04-22T17:36:30.129 INFO:tasks.rados.rados.0.smithi115.stdout:3271: expect (ObjNum 968 snap 309 seq_num 968) 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3272: read oid 34 snap 299 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3272: expect (ObjNum 867 snap 285 seq_num 867) 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3273: copy_from oid 10 from oid 42 current snap is 313 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 5 to smithi11541303-12 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3267: finishing write tid 6 to smithi11541303-12 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1770 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3267: left oid 12 (ObjNum 984 snap 313 seq_num 984) 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3267: done (3 left) 2022-04-22T17:36:30.130 INFO:tasks.rados.rados.0.smithi115.stdout:3274: read oid 38 snap 299 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3274: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3275: read oid 21 snap 304 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3275: expect (ObjNum 886 snap 289 seq_num 886) 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3276: read oid 46 snap 299 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3276: expect (ObjNum 914 snap 299 seq_num 914) 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3277: copy_from oid 43 from oid 29 current snap is 313 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3278: read oid 16 snap 308 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3278: expect (ObjNum 873 snap 286 seq_num 873) 2022-04-22T17:36:30.131 INFO:tasks.rados.rados.0.smithi115.stdout:3279: snap_remove snap 310 2022-04-22T17:36:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: audit 2022-04-22T17:36:28.296221+0000 mgr.y (mgr.34107) 400 : audit [DBG] from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: audit 2022-04-22T17:36:28.629092+0000 mgr.y (mgr.34107) 401 : audit [DBG] from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: cephadm 2022-04-22T17:36:28.817200+0000 mgr.y (mgr.34107) 402 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:30.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: audit 2022-04-22T17:36:28.845238+0000 mgr.y (mgr.34107) 403 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:30.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: audit 2022-04-22T17:36:29.390116+0000 mon.a (mon.0) 1325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:30.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:30 smithi121 bash[36902]: cluster 2022-04-22T17:36:29.937984+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2022-04-22T17:36:30.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: audit 2022-04-22T17:36:28.296221+0000 mgr.y (mgr.34107) 400 : audit [DBG] from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: audit 2022-04-22T17:36:28.629092+0000 mgr.y (mgr.34107) 401 : audit [DBG] from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: cephadm 2022-04-22T17:36:28.817200+0000 mgr.y (mgr.34107) 402 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:30.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: audit 2022-04-22T17:36:28.845238+0000 mgr.y (mgr.34107) 403 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:30.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: audit 2022-04-22T17:36:29.390116+0000 mon.a (mon.0) 1325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[45495]: cluster 2022-04-22T17:36:29.937984+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: audit 2022-04-22T17:36:28.296221+0000 mgr.y (mgr.34107) 400 : audit [DBG] from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: audit 2022-04-22T17:36:28.629092+0000 mgr.y (mgr.34107) 401 : audit [DBG] from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: cephadm 2022-04-22T17:36:28.817200+0000 mgr.y (mgr.34107) 402 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: audit 2022-04-22T17:36:28.845238+0000 mgr.y (mgr.34107) 403 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: audit 2022-04-22T17:36:29.390116+0000 mon.a (mon.0) 1325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:30 smithi115 bash[46934]: cluster 2022-04-22T17:36:29.937984+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2022-04-22T17:36:31.119 INFO:tasks.rados.rados.0.smithi115.stdout:3279: done (8 left) 2022-04-22T17:36:31.119 INFO:tasks.rados.rados.0.smithi115.stdout:3280: snap_remove snap 312 2022-04-22T17:36:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:31 smithi115 bash[45495]: cluster 2022-04-22T17:36:29.333518+0000 mgr.y (mgr.34107) 404 : cluster [DBG] pgmap v724: 121 pgs: 121 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.2 MiB/s wr, 38 op/s 2022-04-22T17:36:31.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:31 smithi115 bash[46934]: cluster 2022-04-22T17:36:29.333518+0000 mgr.y (mgr.34107) 404 : cluster [DBG] pgmap v724: 121 pgs: 121 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.2 MiB/s wr, 38 op/s 2022-04-22T17:36:31.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:31 smithi121 bash[36902]: cluster 2022-04-22T17:36:29.333518+0000 mgr.y (mgr.34107) 404 : cluster [DBG] pgmap v724: 121 pgs: 121 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 8.2 MiB/s wr, 38 op/s 2022-04-22T17:36:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:31.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:31.634Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:31 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:31.635Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:32.114 INFO:tasks.rados.rados.0.smithi115.stdout:3280: done (8 left) 2022-04-22T17:36:32.114 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 313 2022-04-22T17:36:32.115 INFO:tasks.rados.rados.0.smithi115.stdout:3274: expect (ObjNum 893 snap 292 seq_num 893) 2022-04-22T17:36:32.159 INFO:tasks.rados.rados.0.smithi115.stdout:3275: expect (ObjNum 886 snap 289 seq_num 886) 2022-04-22T17:36:32.248 INFO:tasks.rados.rados.0.smithi115.stdout:3272: expect (ObjNum 867 snap 285 seq_num 867) 2022-04-22T17:36:32.405 INFO:tasks.rados.rados.0.smithi115.stdout:3278: expect (ObjNum 873 snap 286 seq_num 873) 2022-04-22T17:36:32.412 INFO:tasks.rados.rados.0.smithi115.stdout:3273: finishing copy_from to smithi11541303-10 2022-04-22T17:36:32.412 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2033 (ObjNum 951 snap 308 seq_num 951) dirty exists 2022-04-22T17:36:32.412 INFO:tasks.rados.rados.0.smithi115.stdout:3273: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:36:32.412 INFO:tasks.rados.rados.0.smithi115.stdout:3276: expect (ObjNum 914 snap 299 seq_num 914) 2022-04-22T17:36:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:32 smithi121 bash[36902]: cluster 2022-04-22T17:36:31.101843+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2022-04-22T17:36:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:32 smithi121 bash[36902]: cluster 2022-04-22T17:36:32.102039+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2022-04-22T17:36:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:32 smithi115 bash[46934]: cluster 2022-04-22T17:36:31.101843+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2022-04-22T17:36:32.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:32 smithi115 bash[46934]: cluster 2022-04-22T17:36:32.102039+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2022-04-22T17:36:32.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:32 smithi115 bash[45495]: cluster 2022-04-22T17:36:31.101843+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2022-04-22T17:36:32.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:32 smithi115 bash[45495]: cluster 2022-04-22T17:36:32.102039+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2022-04-22T17:36:32.618 INFO:tasks.rados.rados.0.smithi115.stdout:3271: expect (ObjNum 968 snap 309 seq_num 968) 2022-04-22T17:36:32.939 INFO:tasks.rados.rados.0.smithi115.stdout:3277: finishing copy_from to smithi11541303-43 2022-04-22T17:36:32.940 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1874 (ObjNum 979 snap 312 seq_num 979) dirty exists 2022-04-22T17:36:32.940 INFO:tasks.rados.rados.0.smithi115.stdout:3277: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:36:32.940 INFO:tasks.rados.rados.0.smithi115.stdout:3281: seq_num 986 ranges {0=516096,516096=770048,1286144=655360,1941504=180224} 2022-04-22T17:36:32.947 INFO:tasks.rados.rados.0.smithi115.stdout:3281: writing smithi11541303-4 from 0 to 516096 tid 1 2022-04-22T17:36:32.952 INFO:tasks.rados.rados.0.smithi115.stdout:3281: writing smithi11541303-4 from 516096 to 1286144 tid 2 2022-04-22T17:36:32.957 INFO:tasks.rados.rados.0.smithi115.stdout:3281: writing smithi11541303-4 from 1286144 to 1941504 tid 3 2022-04-22T17:36:32.958 INFO:tasks.rados.rados.0.smithi115.stdout:3281: writing smithi11541303-4 from 1941504 to 2121728 tid 4 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3271: done (8 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3272: done (7 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3273: done (6 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3274: done (5 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3275: done (4 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3276: done (3 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3277: done (2 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3278: done (1 left) 2022-04-22T17:36:32.959 INFO:tasks.rados.rados.0.smithi115.stdout:3282: copy_from oid 30 from oid 22 current snap is 313 2022-04-22T17:36:32.960 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 1 to smithi11541303-4 2022-04-22T17:36:32.960 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 313 2022-04-22T17:36:32.960 INFO:tasks.rados.rados.0.smithi115.stdout:3283: seq_num 987 ranges {2719744=532480,3252224=647168,3899392=417792,4317184=409600,4726784=131072} 2022-04-22T17:36:32.977 INFO:tasks.rados.rados.0.smithi115.stdout:3283: writing smithi11541303-13 from 2719744 to 3252224 tid 1 2022-04-22T17:36:32.980 INFO:tasks.rados.rados.0.smithi115.stdout:3283: writing smithi11541303-13 from 3252224 to 3899392 tid 2 2022-04-22T17:36:32.982 INFO:tasks.rados.rados.0.smithi115.stdout:3283: writing smithi11541303-13 from 3899392 to 4317184 tid 3 2022-04-22T17:36:32.985 INFO:tasks.rados.rados.0.smithi115.stdout:3283: writing smithi11541303-13 from 4317184 to 4726784 tid 4 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:3283: writing smithi11541303-13 from 4726784 to 4857856 tid 5 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:3284: rollback oid 48 current snap is 313 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 48 to 299 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 2 to smithi11541303-4 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 313 2022-04-22T17:36:32.986 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 3 to smithi11541303-4 2022-04-22T17:36:32.987 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 4 to smithi11541303-4 2022-04-22T17:36:32.987 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 5 to smithi11541303-4 2022-04-22T17:36:32.987 INFO:tasks.rados.rados.0.smithi115.stdout:3285: seq_num 988 ranges {7380992=557056,7938048=532480,8470528=622592,9093120=450560} 2022-04-22T17:36:33.031 INFO:tasks.rados.rados.0.smithi115.stdout:3285: writing smithi11541303-15 from 7380992 to 7938048 tid 1 2022-04-22T17:36:33.033 INFO:tasks.rados.rados.0.smithi115.stdout:3285: writing smithi11541303-15 from 7938048 to 8470528 tid 2 2022-04-22T17:36:33.037 INFO:tasks.rados.rados.0.smithi115.stdout:3285: writing smithi11541303-15 from 8470528 to 9093120 tid 3 2022-04-22T17:36:33.042 INFO:tasks.rados.rados.0.smithi115.stdout:3285: writing smithi11541303-15 from 9093120 to 9543680 tid 4 2022-04-22T17:36:33.042 INFO:tasks.rados.rados.0.smithi115.stdout:3286: snap_create 2022-04-22T17:36:33.042 INFO:tasks.rados.rados.0.smithi115.stdout:3281: finishing write tid 6 to smithi11541303-4 2022-04-22T17:36:33.042 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2033 (ObjNum 986 snap 313 seq_num 986) dirty exists 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3281: left oid 4 (ObjNum 986 snap 313 seq_num 986) 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 1 to smithi11541303-13 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 2 to smithi11541303-13 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 3 to smithi11541303-13 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 4 to smithi11541303-13 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 5 to smithi11541303-13 2022-04-22T17:36:33.043 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 6 to smithi11541303-13 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3282: finishing copy_from to smithi11541303-30 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1847 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3282: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3284: finishing rollback tid 0 to smithi11541303-48 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3284: finishing rollback tid 1 to smithi11541303-48 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3284: finishing rollback tid 2 to smithi11541303-48 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2036 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:36:33.044 INFO:tasks.rados.rados.0.smithi115.stdout:3283: finishing write tid 7 to smithi11541303-13 2022-04-22T17:36:33.045 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1880 (ObjNum 987 snap 313 seq_num 987) dirty exists 2022-04-22T17:36:33.045 INFO:tasks.rados.rados.0.smithi115.stdout:3283: left oid 13 (ObjNum 987 snap 313 seq_num 987) 2022-04-22T17:36:33.045 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 1 to smithi11541303-15 2022-04-22T17:36:33.045 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 2 to smithi11541303-15 2022-04-22T17:36:33.046 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 3 to smithi11541303-15 2022-04-22T17:36:33.046 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 4 to smithi11541303-15 2022-04-22T17:36:33.046 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 5 to smithi11541303-15 2022-04-22T17:36:33.049 INFO:tasks.rados.rados.0.smithi115.stdout:3285: finishing write tid 6 to smithi11541303-15 2022-04-22T17:36:33.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1553 (ObjNum 988 snap 313 seq_num 988) dirty exists 2022-04-22T17:36:33.050 INFO:tasks.rados.rados.0.smithi115.stdout:3285: left oid 15 (ObjNum 988 snap 313 seq_num 988) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3281: done (5 left) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3282: done (4 left) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3283: done (3 left) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3284: done (2 left) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3285: done (1 left) 2022-04-22T17:36:33.137 INFO:tasks.rados.rados.0.smithi115.stdout:3286: done (0 left) 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:3287: rmattr oid 26 current snap is 314 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:3287: done (0 left) 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:3288: rollback oid 26 current snap is 314 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 307 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 314 2022-04-22T17:36:33.138 INFO:tasks.rados.rados.0.smithi115.stdout:3289: seq_num 989 ranges {0=573440,573440=466944,1040384=434176,1474560=638976,2113536=65536} 2022-04-22T17:36:33.142 INFO:tasks.rados.rados.0.smithi115.stdout:3289: writing smithi11541303-9 from 0 to 573440 tid 1 2022-04-22T17:36:33.146 INFO:tasks.rados.rados.0.smithi115.stdout:3289: writing smithi11541303-9 from 573440 to 1040384 tid 2 2022-04-22T17:36:33.149 INFO:tasks.rados.rados.0.smithi115.stdout:3289: writing smithi11541303-9 from 1040384 to 1474560 tid 3 2022-04-22T17:36:33.153 INFO:tasks.rados.rados.0.smithi115.stdout:3289: writing smithi11541303-9 from 1474560 to 2113536 tid 4 2022-04-22T17:36:33.153 INFO:tasks.rados.rados.0.smithi115.stdout:3289: writing smithi11541303-9 from 2113536 to 2179072 tid 5 2022-04-22T17:36:33.153 INFO:tasks.rados.rados.0.smithi115.stdout:3290: snap_remove snap 309 2022-04-22T17:36:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:33 smithi121 bash[36902]: cluster 2022-04-22T17:36:31.334093+0000 mgr.y (mgr.34107) 405 : cluster [DBG] pgmap v727: 121 pgs: 121 active+clean; 241 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:36:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:33 smithi121 bash[36902]: cephadm 2022-04-22T17:36:31.676957+0000 mgr.y (mgr.34107) 406 : cephadm [INF] Upgrade: Updating alertmanager.a 2022-04-22T17:36:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:33 smithi121 bash[36902]: cephadm 2022-04-22T17:36:31.689833+0000 mgr.y (mgr.34107) 407 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:36:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[46934]: cluster 2022-04-22T17:36:31.334093+0000 mgr.y (mgr.34107) 405 : cluster [DBG] pgmap v727: 121 pgs: 121 active+clean; 241 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:36:33.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[46934]: cephadm 2022-04-22T17:36:31.676957+0000 mgr.y (mgr.34107) 406 : cephadm [INF] Upgrade: Updating alertmanager.a 2022-04-22T17:36:33.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[46934]: cephadm 2022-04-22T17:36:31.689833+0000 mgr.y (mgr.34107) 407 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:36:33.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[45495]: cluster 2022-04-22T17:36:31.334093+0000 mgr.y (mgr.34107) 405 : cluster [DBG] pgmap v727: 121 pgs: 121 active+clean; 241 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:36:33.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[45495]: cephadm 2022-04-22T17:36:31.676957+0000 mgr.y (mgr.34107) 406 : cephadm [INF] Upgrade: Updating alertmanager.a 2022-04-22T17:36:33.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[45495]: cephadm 2022-04-22T17:36:31.689833+0000 mgr.y (mgr.34107) 407 : cephadm [INF] Deploying daemon alertmanager.a on smithi115 2022-04-22T17:36:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:33.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:33.529Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:33 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:33.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:3288: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:36:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:3288: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:36:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2028 (ObjNum -1115498560 snap 22075 seq_num 0) dirty dne 2022-04-22T17:36:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 1 to smithi11541303-9 2022-04-22T17:36:34.151 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 2 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 3 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 4 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 5 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 6 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: finishing write tid 7 to smithi11541303-9 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2051 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: left oid 9 (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3288: done (2 left) 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3289: done (1 left) 2022-04-22T17:36:34.152 INFO:tasks.rados.rados.0.smithi115.stdout:3290: done (0 left) 2022-04-22T17:36:34.153 INFO:tasks.rados.rados.0.smithi115.stdout:3291: setattr oid 27 current snap is 314 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3292: read oid 44 snap -1 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3292: expect deleted 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3293: read oid 50 snap -1 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3293: expect deleted 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3294: copy_from oid 42 from oid 11 current snap is 314 2022-04-22T17:36:34.154 INFO:tasks.rados.rados.0.smithi115.stdout:3295: rmattr oid 19 current snap is 314 2022-04-22T17:36:34.155 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 314 2022-04-22T17:36:34.155 INFO:tasks.rados.rados.0.smithi115.stdout:3296: seq_num 990 ranges {6963200=589824,7553024=753664,8306688=638976,8945664=212992} 2022-04-22T17:36:34.201 INFO:tasks.rados.rados.0.smithi115.stdout:3296: writing smithi11541303-8 from 6963200 to 7553024 tid 1 2022-04-22T17:36:34.202 INFO:tasks.rados.rados.0.smithi115.stdout:3296: writing smithi11541303-8 from 7553024 to 8306688 tid 2 2022-04-22T17:36:34.206 INFO:tasks.rados.rados.0.smithi115.stdout:3296: writing smithi11541303-8 from 8306688 to 8945664 tid 3 2022-04-22T17:36:34.207 INFO:tasks.rados.rados.0.smithi115.stdout:3296: writing smithi11541303-8 from 8945664 to 9158656 tid 4 2022-04-22T17:36:34.207 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 314 2022-04-22T17:36:34.207 INFO:tasks.rados.rados.0.smithi115.stdout:3297: seq_num 991 ranges {6930432=606208,7536640=638976,8175616=794624,8970240=172032} 2022-04-22T17:36:34.249 INFO:tasks.rados.rados.0.smithi115.stdout:3297: writing smithi11541303-29 from 6930432 to 7536640 tid 1 2022-04-22T17:36:34.253 INFO:tasks.rados.rados.0.smithi115.stdout:3297: writing smithi11541303-29 from 7536640 to 8175616 tid 2 2022-04-22T17:36:34.259 INFO:tasks.rados.rados.0.smithi115.stdout:3297: writing smithi11541303-29 from 8175616 to 8970240 tid 3 2022-04-22T17:36:34.259 INFO:tasks.rados.rados.0.smithi115.stdout:3297: writing smithi11541303-29 from 8970240 to 9142272 tid 4 2022-04-22T17:36:34.260 INFO:tasks.rados.rados.0.smithi115.stdout:3298: snap_remove snap 304 2022-04-22T17:36:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:34 smithi121 bash[36902]: cluster 2022-04-22T17:36:33.124323+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2022-04-22T17:36:34.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:34 smithi115 bash[45495]: cluster 2022-04-22T17:36:33.124323+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2022-04-22T17:36:34.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:34 smithi115 bash[46934]: cluster 2022-04-22T17:36:33.124323+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2022-04-22T17:36:34.957 INFO:tasks.rados.rados.0.smithi115.stdout:3298: done (7 left) 2022-04-22T17:36:34.958 INFO:tasks.rados.rados.0.smithi115.stdout:3299: snap_create 2022-04-22T17:36:34.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1771 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:34.958 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2037 (ObjNum 940 snap 304 seq_num 940) dirty exists 2022-04-22T17:36:34.959 INFO:tasks.rados.rados.0.smithi115.stdout:3294: finishing copy_from to smithi11541303-42 2022-04-22T17:36:34.959 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 1393 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:34.959 INFO:tasks.rados.rados.0.smithi115.stdout:3294: finishing copy_from racing read to smithi11541303-42 2022-04-22T17:36:34.960 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 1 to smithi11541303-8 2022-04-22T17:36:34.960 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 2 to smithi11541303-8 2022-04-22T17:36:34.960 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 3 to smithi11541303-8 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 4 to smithi11541303-8 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 5 to smithi11541303-8 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3296: finishing write tid 6 to smithi11541303-8 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1881 (ObjNum 990 snap 314 seq_num 990) dirty exists 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3296: left oid 8 (ObjNum 990 snap 314 seq_num 990) 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 1 to smithi11541303-29 2022-04-22T17:36:34.961 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 2 to smithi11541303-29 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 3 to smithi11541303-29 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 4 to smithi11541303-29 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 5 to smithi11541303-29 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:3297: finishing write tid 6 to smithi11541303-29 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1880 (ObjNum 991 snap 314 seq_num 991) dirty exists 2022-04-22T17:36:34.962 INFO:tasks.rados.rados.0.smithi115.stdout:3297: left oid 29 (ObjNum 991 snap 314 seq_num 991) 2022-04-22T17:36:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:35 smithi121 bash[36902]: cluster 2022-04-22T17:36:33.334785+0000 mgr.y (mgr.34107) 408 : cluster [DBG] pgmap v730: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 5.1 MiB/s wr, 28 op/s 2022-04-22T17:36:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:35 smithi121 bash[36902]: cluster 2022-04-22T17:36:34.137611+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2022-04-22T17:36:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:35 smithi121 bash[36902]: audit 2022-04-22T17:36:34.392690+0000 mon.a (mon.0) 1331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:35 smithi121 bash[36902]: cluster 2022-04-22T17:36:34.942043+0000 mon.a (mon.0) 1332 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2022-04-22T17:36:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[45495]: cluster 2022-04-22T17:36:33.334785+0000 mgr.y (mgr.34107) 408 : cluster [DBG] pgmap v730: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 5.1 MiB/s wr, 28 op/s 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[45495]: cluster 2022-04-22T17:36:34.137611+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[45495]: audit 2022-04-22T17:36:34.392690+0000 mon.a (mon.0) 1331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[45495]: cluster 2022-04-22T17:36:34.942043+0000 mon.a (mon.0) 1332 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[46934]: cluster 2022-04-22T17:36:33.334785+0000 mgr.y (mgr.34107) 408 : cluster [DBG] pgmap v730: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 5.1 MiB/s wr, 28 op/s 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[46934]: cluster 2022-04-22T17:36:34.137611+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[46934]: audit 2022-04-22T17:36:34.392690+0000 mon.a (mon.0) 1331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:35 smithi115 bash[46934]: cluster 2022-04-22T17:36:34.942043+0000 mon.a (mon.0) 1332 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2022-04-22T17:36:35.959 INFO:tasks.rados.rados.0.smithi115.stdout:3291: done (7 left) 2022-04-22T17:36:35.984 INFO:tasks.rados.rados.0.smithi115.stdout:3292: done (6 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3293: done (5 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3294: done (4 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3295: done (3 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3296: done (2 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3297: done (1 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3299: done (0 left) 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 17 current snap is 315 2022-04-22T17:36:35.985 INFO:tasks.rados.rados.0.smithi115.stdout:3300: seq_num 992 ranges {5414912=622592,6037504=524288,6561792=606208,7168000=475136} 2022-04-22T17:36:35.993 INFO:tasks.rados.rados.0.smithi115.stdout:3300: writing smithi11541303-17 from 5414912 to 6037504 tid 1 2022-04-22T17:36:35.996 INFO:tasks.rados.rados.0.smithi115.stdout:3300: writing smithi11541303-17 from 6037504 to 6561792 tid 2 2022-04-22T17:36:36.003 INFO:tasks.rados.rados.0.smithi115.stdout:3300: writing smithi11541303-17 from 6561792 to 7168000 tid 3 2022-04-22T17:36:36.007 INFO:tasks.rados.rados.0.smithi115.stdout:3300: writing smithi11541303-17 from 7168000 to 7643136 tid 4 2022-04-22T17:36:36.007 INFO:tasks.rados.rados.0.smithi115.stdout:3301: read oid 7 snap -1 2022-04-22T17:36:36.007 INFO:tasks.rados.rados.0.smithi115.stdout:3301: expect deleted 2022-04-22T17:36:36.007 INFO:tasks.rados.rados.0.smithi115.stdout:3302: snap_remove snap 314 2022-04-22T17:36:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:3302: done (2 left) 2022-04-22T17:36:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 315 2022-04-22T17:36:36.972 INFO:tasks.rados.rados.0.smithi115.stdout:3303: seq_num 993 ranges {2088960=638976,2727936=409600,3137536=409600,3547136=688128,4235264=98304} 2022-04-22T17:36:36.992 INFO:tasks.rados.rados.0.smithi115.stdout:3303: writing smithi11541303-16 from 2088960 to 2727936 tid 1 2022-04-22T17:36:37.053 INFO:tasks.rados.rados.0.smithi115.stdout:3303: writing smithi11541303-16 from 2727936 to 3137536 tid 2 2022-04-22T17:36:37.054 INFO:tasks.rados.rados.0.smithi115.stdout:3303: writing smithi11541303-16 from 3137536 to 3547136 tid 3 2022-04-22T17:36:37.054 INFO:tasks.rados.rados.0.smithi115.stdout:3303: writing smithi11541303-16 from 3547136 to 4235264 tid 4 2022-04-22T17:36:37.054 INFO:tasks.rados.rados.0.smithi115.stdout:3303: writing smithi11541303-16 from 4235264 to 4333568 tid 5 2022-04-22T17:36:37.054 INFO:tasks.rados.rados.0.smithi115.stdout:3304: rmattr oid 29 current snap is 315 2022-04-22T17:36:37.054 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 1 to smithi11541303-17 2022-04-22T17:36:37.055 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 2 to smithi11541303-17 2022-04-22T17:36:37.055 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 3 to smithi11541303-17 2022-04-22T17:36:37.055 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 4 to smithi11541303-17 2022-04-22T17:36:37.056 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 5 to smithi11541303-17 2022-04-22T17:36:37.056 INFO:tasks.rados.rados.0.smithi115.stdout:3300: finishing write tid 6 to smithi11541303-17 2022-04-22T17:36:37.056 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 1885 (ObjNum 992 snap 315 seq_num 992) dirty exists 2022-04-22T17:36:37.056 INFO:tasks.rados.rados.0.smithi115.stdout:3300: left oid 17 (ObjNum 992 snap 315 seq_num 992) 2022-04-22T17:36:37.062 INFO:tasks.rados.rados.0.smithi115.stdout:3300: done (3 left) 2022-04-22T17:36:37.067 INFO:tasks.rados.rados.0.smithi115.stdout:3301: done (2 left) 2022-04-22T17:36:37.068 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 315 2022-04-22T17:36:37.068 INFO:tasks.rados.rados.0.smithi115.stdout:3305: seq_num 994 ranges {827392=655360,1482752=688128,2170880=622592,2793472=294912} 2022-04-22T17:36:37.068 INFO:tasks.rados.rados.0.smithi115.stdout:3305: writing smithi11541303-31 from 827392 to 1482752 tid 1 2022-04-22T17:36:37.069 INFO:tasks.rados.rados.0.smithi115.stdout:3305: writing smithi11541303-31 from 1482752 to 2170880 tid 2 2022-04-22T17:36:37.069 INFO:tasks.rados.rados.0.smithi115.stdout:3305: writing smithi11541303-31 from 2170880 to 2793472 tid 3 2022-04-22T17:36:37.069 INFO:tasks.rados.rados.0.smithi115.stdout:3305: writing smithi11541303-31 from 2793472 to 3088384 tid 4 2022-04-22T17:36:37.069 INFO:tasks.rados.rados.0.smithi115.stdout:3306: delete oid 36 current snap is 315 2022-04-22T17:36:37.069 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 1 to smithi11541303-16 2022-04-22T17:36:37.070 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 2 to smithi11541303-16 2022-04-22T17:36:37.070 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 3 to smithi11541303-16 2022-04-22T17:36:37.070 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 4 to smithi11541303-16 2022-04-22T17:36:37.070 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 5 to smithi11541303-16 2022-04-22T17:36:37.070 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 6 to smithi11541303-16 2022-04-22T17:36:37.071 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1881 (ObjNum 991 snap 314 seq_num 991) dirty exists 2022-04-22T17:36:37.071 INFO:tasks.rados.rados.0.smithi115.stdout:3303: finishing write tid 7 to smithi11541303-16 2022-04-22T17:36:37.072 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1849 (ObjNum 993 snap 315 seq_num 993) dirty exists 2022-04-22T17:36:37.073 INFO:tasks.rados.rados.0.smithi115.stdout:3303: left oid 16 (ObjNum 993 snap 315 seq_num 993) 2022-04-22T17:36:37.073 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 1 to smithi11541303-31 2022-04-22T17:36:37.073 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 2 to smithi11541303-31 2022-04-22T17:36:37.073 INFO:tasks.rados.rados.0.smithi115.stdout:3303: done (3 left) 2022-04-22T17:36:37.074 INFO:tasks.rados.rados.0.smithi115.stdout:3304: done (2 left) 2022-04-22T17:36:37.074 INFO:tasks.rados.rados.0.smithi115.stdout:3306: done (1 left) 2022-04-22T17:36:37.074 INFO:tasks.rados.rados.0.smithi115.stdout:3307: copy_from oid 35 from oid 7 current snap is 315 2022-04-22T17:36:37.074 INFO:tasks.rados.rados.0.smithi115.stdout:3308: rmattr oid 43 current snap is 315 2022-04-22T17:36:37.075 INFO:tasks.rados.rados.0.smithi115.stdout:3309: snap_remove snap 300 2022-04-22T17:36:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:36 smithi121 bash[36902]: cluster 2022-04-22T17:36:35.335700+0000 mgr.y (mgr.34107) 409 : cluster [DBG] pgmap v733: 121 pgs: 121 active+clean; 250 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:36:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:36 smithi121 bash[36902]: cluster 2022-04-22T17:36:35.944460+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2022-04-22T17:36:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:36 smithi115 bash[45495]: cluster 2022-04-22T17:36:35.335700+0000 mgr.y (mgr.34107) 409 : cluster [DBG] pgmap v733: 121 pgs: 121 active+clean; 250 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:36:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:36 smithi115 bash[45495]: cluster 2022-04-22T17:36:35.944460+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2022-04-22T17:36:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:36 smithi115 bash[46934]: cluster 2022-04-22T17:36:35.335700+0000 mgr.y (mgr.34107) 409 : cluster [DBG] pgmap v733: 121 pgs: 121 active+clean; 250 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2022-04-22T17:36:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:36 smithi115 bash[46934]: cluster 2022-04-22T17:36:35.944460+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2022-04-22T17:36:37.982 INFO:tasks.rados.rados.0.smithi115.stdout:3309: done (3 left) 2022-04-22T17:36:37.982 INFO:tasks.rados.rados.0.smithi115.stdout:3310: setattr oid 34 current snap is 315 2022-04-22T17:36:37.982 INFO:tasks.rados.rados.0.smithi115.stdout:3307: finishing copy_from to smithi11541303-35 2022-04-22T17:36:37.982 INFO:tasks.rados.rados.0.smithi115.stdout:3307: got expected ENOENT (src dne) 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:3307: finishing copy_from racing read to smithi11541303-35 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 3 to smithi11541303-31 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 4 to smithi11541303-31 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 5 to smithi11541303-31 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1886 (ObjNum 979 snap 312 seq_num 979) dirty exists 2022-04-22T17:36:37.983 INFO:tasks.rados.rados.0.smithi115.stdout:3305: finishing write tid 6 to smithi11541303-31 2022-04-22T17:36:37.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2063 (ObjNum 994 snap 315 seq_num 994) dirty exists 2022-04-22T17:36:37.985 INFO:tasks.rados.rados.0.smithi115.stdout:3305: left oid 31 (ObjNum 994 snap 315 seq_num 994) 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3305: done (3 left) 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3307: done (2 left) 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3308: done (1 left) 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3311: copy_from oid 3 from oid 9 current snap is 315 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3312: read oid 44 snap -1 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3312: expect deleted 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3313: read oid 49 snap 307 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3313: expect (ObjNum 948 snap 307 seq_num 948) 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 315 2022-04-22T17:36:37.991 INFO:tasks.rados.rados.0.smithi115.stdout:3314: seq_num 996 ranges {9142272=688128,9830400=507904,10338304=581632,10919936=516096} 2022-04-22T17:36:38.048 INFO:tasks.rados.rados.0.smithi115.stdout:3314: writing smithi11541303-29 from 9142272 to 9830400 tid 1 2022-04-22T17:36:38.107 INFO:tasks.rados.rados.0.smithi115.stdout:3314: writing smithi11541303-29 from 9830400 to 10338304 tid 2 2022-04-22T17:36:38.107 INFO:tasks.rados.rados.0.smithi115.stdout:3314: writing smithi11541303-29 from 10338304 to 10919936 tid 3 2022-04-22T17:36:38.108 INFO:tasks.rados.rados.0.smithi115.stdout:3314: writing smithi11541303-29 from 10919936 to 11436032 tid 4 2022-04-22T17:36:38.108 INFO:tasks.rados.rados.0.smithi115.stdout:3315: snap_remove snap 308 2022-04-22T17:36:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:37 smithi121 bash[36902]: cluster 2022-04-22T17:36:36.956340+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2022-04-22T17:36:38.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:37 smithi115 bash[45495]: cluster 2022-04-22T17:36:36.956340+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2022-04-22T17:36:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:37 smithi115 bash[46934]: cluster 2022-04-22T17:36:36.956340+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2022-04-22T17:36:38.991 INFO:tasks.rados.rados.0.smithi115.stdout:3315: done (5 left) 2022-04-22T17:36:38.991 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 315 2022-04-22T17:36:38.991 INFO:tasks.rados.rados.0.smithi115.stdout:3316: seq_num 997 ranges {0=704512,704512=794624,1499136=794624,2293760=16384} 2022-04-22T17:36:38.996 INFO:tasks.rados.rados.0.smithi115.stdout:3316: writing smithi11541303-33 from 0 to 704512 tid 1 2022-04-22T17:36:39.001 INFO:tasks.rados.rados.0.smithi115.stdout:3316: writing smithi11541303-33 from 704512 to 1499136 tid 2 2022-04-22T17:36:39.011 INFO:tasks.rados.rados.0.smithi115.stdout:3316: writing smithi11541303-33 from 1499136 to 2293760 tid 3 2022-04-22T17:36:39.011 INFO:tasks.rados.rados.0.smithi115.stdout:3316: writing smithi11541303-33 from 2293760 to 2310144 tid 4 2022-04-22T17:36:39.012 INFO:tasks.rados.rados.0.smithi115.stdout:3317: snap_create 2022-04-22T17:36:39.012 INFO:tasks.rados.rados.0.smithi115.stdout:3313: expect (ObjNum 948 snap 307 seq_num 948) 2022-04-22T17:36:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1850 (ObjNum 746 snap 256 seq_num 746) dirty exists 2022-04-22T17:36:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:3311: finishing copy_from to smithi11541303-3 2022-04-22T17:36:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2051 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:36:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:3311: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:36:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 1 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 2 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 3 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 4 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 5 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: finishing write tid 6 to smithi11541303-29 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1886 (ObjNum 996 snap 315 seq_num 996) dirty exists 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3314: left oid 29 (ObjNum 996 snap 315 seq_num 996) 2022-04-22T17:36:39.053 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 1 to smithi11541303-33 2022-04-22T17:36:39.054 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 2 to smithi11541303-33 2022-04-22T17:36:39.054 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 3 to smithi11541303-33 2022-04-22T17:36:39.054 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 4 to smithi11541303-33 2022-04-22T17:36:39.054 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 5 to smithi11541303-33 2022-04-22T17:36:39.055 INFO:tasks.rados.rados.0.smithi115.stdout:3316: finishing write tid 6 to smithi11541303-33 2022-04-22T17:36:39.055 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 2033 (ObjNum 997 snap 315 seq_num 997) dirty exists 2022-04-22T17:36:39.055 INFO:tasks.rados.rados.0.smithi115.stdout:3316: left oid 33 (ObjNum 997 snap 315 seq_num 997) 2022-04-22T17:36:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:38 smithi121 bash[36902]: cluster 2022-04-22T17:36:37.336538+0000 mgr.y (mgr.34107) 410 : cluster [DBG] pgmap v736: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2022-04-22T17:36:39.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:38 smithi121 bash[36902]: cluster 2022-04-22T17:36:37.969456+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2022-04-22T17:36:39.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:38 smithi115 bash[45495]: cluster 2022-04-22T17:36:37.336538+0000 mgr.y (mgr.34107) 410 : cluster [DBG] pgmap v736: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2022-04-22T17:36:39.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:38 smithi115 bash[45495]: cluster 2022-04-22T17:36:37.969456+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2022-04-22T17:36:39.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:38 smithi115 bash[46934]: cluster 2022-04-22T17:36:37.336538+0000 mgr.y (mgr.34107) 410 : cluster [DBG] pgmap v736: 121 pgs: 121 active+clean; 256 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2022-04-22T17:36:39.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:38 smithi115 bash[46934]: cluster 2022-04-22T17:36:37.969456+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2022-04-22T17:36:39.960 INFO:tasks.rados.rados.0.smithi115.stdout:3310: done (6 left) 2022-04-22T17:36:39.960 INFO:tasks.rados.rados.0.smithi115.stdout:3311: done (5 left) 2022-04-22T17:36:39.960 INFO:tasks.rados.rados.0.smithi115.stdout:3312: done (4 left) 2022-04-22T17:36:39.961 INFO:tasks.rados.rados.0.smithi115.stdout:3313: done (3 left) 2022-04-22T17:36:39.961 INFO:tasks.rados.rados.0.smithi115.stdout:3314: done (2 left) 2022-04-22T17:36:39.961 INFO:tasks.rados.rados.0.smithi115.stdout:3316: done (1 left) 2022-04-22T17:36:39.961 INFO:tasks.rados.rados.0.smithi115.stdout:3317: done (0 left) 2022-04-22T17:36:39.961 INFO:tasks.rados.rados.0.smithi115.stdout:3318: read oid 3 snap -1 2022-04-22T17:36:39.962 INFO:tasks.rados.rados.0.smithi115.stdout:3318: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:39.962 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 316 2022-04-22T17:36:39.962 INFO:tasks.rados.rados.0.smithi115.stdout:3319: seq_num 998 ranges {4726784=720896,5447680=679936,6127616=598016,6725632=327680} 2022-04-22T17:36:39.998 INFO:tasks.rados.rados.0.smithi115.stdout:3319: writing smithi11541303-22 from 4726784 to 5447680 tid 1 2022-04-22T17:36:40.003 INFO:tasks.rados.rados.0.smithi115.stdout:3319: writing smithi11541303-22 from 5447680 to 6127616 tid 2 2022-04-22T17:36:40.006 INFO:tasks.rados.rados.0.smithi115.stdout:3319: writing smithi11541303-22 from 6127616 to 6725632 tid 3 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3319: writing smithi11541303-22 from 6725632 to 7053312 tid 4 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3320: read oid 37 snap -1 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3320: expect (ObjNum 960 snap 308 seq_num 960) 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3321: read oid 40 snap -1 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3321: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:36:40.009 INFO:tasks.rados.rados.0.smithi115.stdout:3318: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:40.072 INFO:tasks.rados.rados.0.smithi115.stdout:3322: read oid 42 snap -1 2022-04-22T17:36:40.073 INFO:tasks.rados.rados.0.smithi115.stdout:3322: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:40.073 INFO:tasks.rados.rados.0.smithi115.stdout:3318: done (4 left) 2022-04-22T17:36:40.073 INFO:tasks.rados.rados.0.smithi115.stdout:3323: delete oid 26 current snap is 316 2022-04-22T17:36:40.074 INFO:tasks.rados.rados.0.smithi115.stdout:3320: expect (ObjNum 960 snap 308 seq_num 960) 2022-04-22T17:36:40.125 INFO:tasks.rados.rados.0.smithi115.stdout:3321: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 1 to smithi11541303-22 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 2 to smithi11541303-22 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 3 to smithi11541303-22 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 4 to smithi11541303-22 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 5 to smithi11541303-22 2022-04-22T17:36:40.149 INFO:tasks.rados.rados.0.smithi115.stdout:3319: finishing write tid 6 to smithi11541303-22 2022-04-22T17:36:40.150 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1852 (ObjNum 998 snap 316 seq_num 998) dirty exists 2022-04-22T17:36:40.150 INFO:tasks.rados.rados.0.smithi115.stdout:3319: left oid 22 (ObjNum 998 snap 316 seq_num 998) 2022-04-22T17:36:40.150 INFO:tasks.rados.rados.0.smithi115.stdout:3322: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:40.177 INFO:tasks.rados.rados.0.smithi115.stdout:3319: done (4 left) 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:3320: done (3 left) 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:3321: done (2 left) 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:3322: done (1 left) 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:3323: done (0 left) 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 316 2022-04-22T17:36:40.178 INFO:tasks.rados.rados.0.smithi115.stdout:3324: seq_num 1000 ranges {6930432=753664,7684096=442368,8126464=565248,8691712=458752,9150464=139264} 2022-04-22T17:36:40.221 INFO:tasks.rados.rados.0.smithi115.stdout:3324: writing smithi11541303-43 from 6930432 to 7684096 tid 1 2022-04-22T17:36:40.223 INFO:tasks.rados.rados.0.smithi115.stdout:3324: writing smithi11541303-43 from 7684096 to 8126464 tid 2 2022-04-22T17:36:40.226 INFO:tasks.rados.rados.0.smithi115.stdout:3324: writing smithi11541303-43 from 8126464 to 8691712 tid 3 2022-04-22T17:36:40.229 INFO:tasks.rados.rados.0.smithi115.stdout:3324: writing smithi11541303-43 from 8691712 to 9150464 tid 4 2022-04-22T17:36:40.229 INFO:tasks.rados.rados.0.smithi115.stdout:3324: writing smithi11541303-43 from 9150464 to 9289728 tid 5 2022-04-22T17:36:40.230 INFO:tasks.rados.rados.0.smithi115.stdout:3325: snap_create 2022-04-22T17:36:40.232 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 1 to smithi11541303-43 2022-04-22T17:36:40.233 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 2 to smithi11541303-43 2022-04-22T17:36:40.237 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 3 to smithi11541303-43 2022-04-22T17:36:40.237 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 4 to smithi11541303-43 2022-04-22T17:36:40.239 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 5 to smithi11541303-43 2022-04-22T17:36:40.239 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 6 to smithi11541303-43 2022-04-22T17:36:40.242 INFO:tasks.rados.rados.0.smithi115.stdout:3324: finishing write tid 7 to smithi11541303-43 2022-04-22T17:36:40.242 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1892 (ObjNum 1000 snap 316 seq_num 1000) dirty exists 2022-04-22T17:36:40.242 INFO:tasks.rados.rados.0.smithi115.stdout:3324: left oid 43 (ObjNum 1000 snap 316 seq_num 1000) 2022-04-22T17:36:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:39 smithi121 bash[36902]: cluster 2022-04-22T17:36:38.976099+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2022-04-22T17:36:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:39 smithi121 bash[36902]: audit 2022-04-22T17:36:39.395832+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:39 smithi121 bash[36902]: cluster 2022-04-22T17:36:39.946250+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2022-04-22T17:36:40.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[45495]: cluster 2022-04-22T17:36:38.976099+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2022-04-22T17:36:40.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[45495]: audit 2022-04-22T17:36:39.395832+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:40.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[45495]: cluster 2022-04-22T17:36:39.946250+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2022-04-22T17:36:40.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[46934]: cluster 2022-04-22T17:36:38.976099+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2022-04-22T17:36:40.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[46934]: audit 2022-04-22T17:36:39.395832+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:40.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:39 smithi115 bash[46934]: cluster 2022-04-22T17:36:39.946250+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2022-04-22T17:36:41.010 INFO:tasks.rados.rados.0.smithi115.stdout:3324: done (1 left) 2022-04-22T17:36:41.011 INFO:tasks.rados.rados.0.smithi115.stdout:3325: done (0 left) 2022-04-22T17:36:41.011 INFO:tasks.rados.rados.0.smithi115.stdout:3326: rmattr oid 21 current snap is 317 2022-04-22T17:36:41.011 INFO:tasks.rados.rados.0.smithi115.stdout:3326: done (0 left) 2022-04-22T17:36:41.011 INFO:tasks.rados.rados.0.smithi115.stdout:3327: snap_remove snap 315 2022-04-22T17:36:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:41 smithi121 bash[36902]: cluster 2022-04-22T17:36:39.337250+0000 mgr.y (mgr.34107) 411 : cluster [DBG] pgmap v739: 121 pgs: 121 active+clean; 260 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2022-04-22T17:36:41.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:41 smithi115 bash[45495]: cluster 2022-04-22T17:36:39.337250+0000 mgr.y (mgr.34107) 411 : cluster [DBG] pgmap v739: 121 pgs: 121 active+clean; 260 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2022-04-22T17:36:41.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:41 smithi115 bash[46934]: cluster 2022-04-22T17:36:39.337250+0000 mgr.y (mgr.34107) 411 : cluster [DBG] pgmap v739: 121 pgs: 121 active+clean; 260 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2022-04-22T17:36:42.011 INFO:tasks.rados.rados.0.smithi115.stdout:3327: done (0 left) 2022-04-22T17:36:42.035 INFO:tasks.rados.rados.0.smithi115.stdout:3328: rmattr oid 45 current snap is 317 2022-04-22T17:36:42.036 INFO:tasks.rados.rados.0.smithi115.stdout:3328: done (0 left) 2022-04-22T17:36:42.036 INFO:tasks.rados.rados.0.smithi115.stdout:3329: setattr oid 23 current snap is 317 2022-04-22T17:36:42.037 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:41.635Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:42.037 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:41.635Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:42.037 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:41 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:41.635Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:42.038 INFO:tasks.rados.rados.0.smithi115.stdout:3330: read oid 11 snap 307 2022-04-22T17:36:42.039 INFO:tasks.rados.rados.0.smithi115.stdout:3330: expect (ObjNum 928 snap 302 seq_num 928) 2022-04-22T17:36:42.039 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 37 current snap is 317 2022-04-22T17:36:42.039 INFO:tasks.rados.rados.0.smithi115.stdout:3331: seq_num 1001 ranges {1687552=770048,2457600=729088,3186688=770048,3956736=106496} 2022-04-22T17:36:42.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:42 smithi115 bash[45495]: cluster 2022-04-22T17:36:40.997719+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2022-04-22T17:36:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:42 smithi115 bash[45495]: cluster 2022-04-22T17:36:41.998495+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2022-04-22T17:36:42.041 INFO:tasks.rados.rados.0.smithi115.stdout:3331: writing smithi11541303-37 from 1687552 to 2457600 tid 1 2022-04-22T17:36:42.092 INFO:tasks.rados.rados.0.smithi115.stdout:3331: writing smithi11541303-37 from 2457600 to 3186688 tid 2 2022-04-22T17:36:42.092 INFO:tasks.rados.rados.0.smithi115.stdout:3331: writing smithi11541303-37 from 3186688 to 3956736 tid 3 2022-04-22T17:36:42.093 INFO:tasks.rados.rados.0.smithi115.stdout:3331: writing smithi11541303-37 from 3956736 to 4063232 tid 4 2022-04-22T17:36:42.093 INFO:tasks.rados.rados.0.smithi115.stdout:3332: snap_create 2022-04-22T17:36:42.093 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2052 (ObjNum 1 snap 32692 seq_num 3179513872) dirty exists 2022-04-22T17:36:42.093 INFO:tasks.rados.rados.0.smithi115.stdout:3330: expect (ObjNum 928 snap 302 seq_num 928) 2022-04-22T17:36:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 1 to smithi11541303-37 2022-04-22T17:36:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 2 to smithi11541303-37 2022-04-22T17:36:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 3 to smithi11541303-37 2022-04-22T17:36:42.116 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 4 to smithi11541303-37 2022-04-22T17:36:42.116 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 5 to smithi11541303-37 2022-04-22T17:36:42.116 INFO:tasks.rados.rados.0.smithi115.stdout:3331: finishing write tid 6 to smithi11541303-37 2022-04-22T17:36:42.117 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 37 v 1930 (ObjNum 1001 snap 317 seq_num 1001) dirty exists 2022-04-22T17:36:42.117 INFO:tasks.rados.rados.0.smithi115.stdout:3331: left oid 37 (ObjNum 1001 snap 317 seq_num 1001) 2022-04-22T17:36:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:42 smithi121 bash[36902]: cluster 2022-04-22T17:36:40.997719+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2022-04-22T17:36:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:42 smithi121 bash[36902]: cluster 2022-04-22T17:36:41.998495+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2022-04-22T17:36:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:42 smithi115 bash[46934]: cluster 2022-04-22T17:36:40.997719+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2022-04-22T17:36:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:42 smithi115 bash[46934]: cluster 2022-04-22T17:36:41.998495+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2022-04-22T17:36:43.035 INFO:tasks.rados.rados.0.smithi115.stdout:3329: done (3 left) 2022-04-22T17:36:43.035 INFO:tasks.rados.rados.0.smithi115.stdout:3330: done (2 left) 2022-04-22T17:36:43.036 INFO:tasks.rados.rados.0.smithi115.stdout:3331: done (1 left) 2022-04-22T17:36:43.036 INFO:tasks.rados.rados.0.smithi115.stdout:3332: done (0 left) 2022-04-22T17:36:43.036 INFO:tasks.rados.rados.0.smithi115.stdout:3333: snap_remove snap 316 2022-04-22T17:36:43.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:43 smithi115 bash[45495]: cluster 2022-04-22T17:36:41.337973+0000 mgr.y (mgr.34107) 412 : cluster [DBG] pgmap v742: 121 pgs: 121 active+clean; 256 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 625 KiB/s rd, 2.1 MiB/s wr, 15 op/s 2022-04-22T17:36:43.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:43 smithi115 bash[46934]: cluster 2022-04-22T17:36:41.337973+0000 mgr.y (mgr.34107) 412 : cluster [DBG] pgmap v742: 121 pgs: 121 active+clean; 256 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 625 KiB/s rd, 2.1 MiB/s wr, 15 op/s 2022-04-22T17:36:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:43 smithi121 bash[36902]: cluster 2022-04-22T17:36:41.337973+0000 mgr.y (mgr.34107) 412 : cluster [DBG] pgmap v742: 121 pgs: 121 active+clean; 256 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 625 KiB/s rd, 2.1 MiB/s wr, 15 op/s 2022-04-22T17:36:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:43.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:43.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:43 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:43.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:44.040 INFO:tasks.rados.rados.0.smithi115.stdout:3333: done (0 left) 2022-04-22T17:36:44.041 INFO:tasks.rados.rados.0.smithi115.stdout:3334: snap_create 2022-04-22T17:36:44.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:44 smithi115 bash[45495]: cluster 2022-04-22T17:36:43.023097+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2022-04-22T17:36:44.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:44 smithi115 bash[46934]: cluster 2022-04-22T17:36:43.023097+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2022-04-22T17:36:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:44 smithi121 bash[36902]: cluster 2022-04-22T17:36:43.023097+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2022-04-22T17:36:44.964 INFO:tasks.rados.rados.0.smithi115.stdout:3334: done (0 left) 2022-04-22T17:36:44.964 INFO:tasks.rados.rados.0.smithi115.stdout:3335: read oid 11 snap -1 2022-04-22T17:36:44.964 INFO:tasks.rados.rados.0.smithi115.stdout:3335: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:44.964 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 319 2022-04-22T17:36:44.964 INFO:tasks.rados.rados.0.smithi115.stdout:3336: seq_num 1002 ranges {6422528=786432,7208960=614400,7823360=581632,8404992=409600} 2022-04-22T17:36:45.001 INFO:tasks.rados.rados.0.smithi115.stdout:3336: writing smithi11541303-28 from 6422528 to 7208960 tid 1 2022-04-22T17:36:45.006 INFO:tasks.rados.rados.0.smithi115.stdout:3336: writing smithi11541303-28 from 7208960 to 7823360 tid 2 2022-04-22T17:36:45.008 INFO:tasks.rados.rados.0.smithi115.stdout:3336: writing smithi11541303-28 from 7823360 to 8404992 tid 3 2022-04-22T17:36:45.011 INFO:tasks.rados.rados.0.smithi115.stdout:3336: writing smithi11541303-28 from 8404992 to 8814592 tid 4 2022-04-22T17:36:45.013 INFO:tasks.rados.rados.0.smithi115.stdout:3337: delete oid 46 current snap is 319 2022-04-22T17:36:45.013 INFO:tasks.rados.rados.0.smithi115.stdout:3335: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 1 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 2 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 3 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 4 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 5 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: finishing write tid 6 to smithi11541303-28 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1891 (ObjNum 1002 snap 319 seq_num 1002) dirty exists 2022-04-22T17:36:45.041 INFO:tasks.rados.rados.0.smithi115.stdout:3336: left oid 28 (ObjNum 1002 snap 319 seq_num 1002) 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3335: done (2 left) 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3336: done (1 left) 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3337: done (0 left) 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3338: read oid 9 snap -1 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3338: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3339: rollback oid 49 current snap is 319 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 317 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3340: rmattr oid 38 current snap is 319 2022-04-22T17:36:45.042 INFO:tasks.rados.rados.0.smithi115.stdout:3341: read oid 41 snap -1 2022-04-22T17:36:45.043 INFO:tasks.rados.rados.0.smithi115.stdout:3341: expect deleted 2022-04-22T17:36:45.043 INFO:tasks.rados.rados.0.smithi115.stdout:3342: snap_create 2022-04-22T17:36:45.047 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1772 (ObjNum 947 snap 307 seq_num 947) dirty exists 2022-04-22T17:36:45.052 INFO:tasks.rados.rados.0.smithi115.stdout:3338: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:45.114 INFO:tasks.rados.rados.0.smithi115.stdout:3339: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:36:45.114 INFO:tasks.rados.rados.0.smithi115.stdout:3339: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:36:45.115 INFO:tasks.rados.rados.0.smithi115.stdout:3339: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:36:45.115 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2079 (ObjNum 948 snap 307 seq_num 948) dirty exists 2022-04-22T17:36:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[45495]: cluster 2022-04-22T17:36:43.338860+0000 mgr.y (mgr.34107) 413 : cluster [DBG] pgmap v745: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2022-04-22T17:36:45.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[45495]: cluster 2022-04-22T17:36:44.027321+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2022-04-22T17:36:45.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[45495]: audit 2022-04-22T17:36:44.399079+0000 mon.a (mon.0) 1343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:45.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[45495]: cluster 2022-04-22T17:36:44.947287+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2022-04-22T17:36:45.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[46934]: cluster 2022-04-22T17:36:43.338860+0000 mgr.y (mgr.34107) 413 : cluster [DBG] pgmap v745: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2022-04-22T17:36:45.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[46934]: cluster 2022-04-22T17:36:44.027321+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2022-04-22T17:36:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[46934]: audit 2022-04-22T17:36:44.399079+0000 mon.a (mon.0) 1343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:45.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:45 smithi115 bash[46934]: cluster 2022-04-22T17:36:44.947287+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2022-04-22T17:36:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:45 smithi121 bash[36902]: cluster 2022-04-22T17:36:43.338860+0000 mgr.y (mgr.34107) 413 : cluster [DBG] pgmap v745: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2022-04-22T17:36:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:45 smithi121 bash[36902]: cluster 2022-04-22T17:36:44.027321+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2022-04-22T17:36:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:45 smithi121 bash[36902]: audit 2022-04-22T17:36:44.399079+0000 mon.a (mon.0) 1343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:45 smithi121 bash[36902]: cluster 2022-04-22T17:36:44.947287+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2022-04-22T17:36:45.969 INFO:tasks.rados.rados.0.smithi115.stdout:3338: done (4 left) 2022-04-22T17:36:45.970 INFO:tasks.rados.rados.0.smithi115.stdout:3339: done (3 left) 2022-04-22T17:36:45.970 INFO:tasks.rados.rados.0.smithi115.stdout:3340: done (2 left) 2022-04-22T17:36:45.970 INFO:tasks.rados.rados.0.smithi115.stdout:3341: done (1 left) 2022-04-22T17:36:45.970 INFO:tasks.rados.rados.0.smithi115.stdout:3342: done (0 left) 2022-04-22T17:36:45.970 INFO:tasks.rados.rados.0.smithi115.stdout:3343: delete oid 27 current snap is 320 2022-04-22T17:36:45.975 INFO:tasks.rados.rados.0.smithi115.stdout:3343: done (0 left) 2022-04-22T17:36:45.975 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 320 2022-04-22T17:36:45.975 INFO:tasks.rados.rados.0.smithi115.stdout:3344: seq_num 1005 ranges {0=32768} 2022-04-22T17:36:45.976 INFO:tasks.rados.rados.0.smithi115.stdout:3344: writing smithi11541303-27 from 0 to 32768 tid 1 2022-04-22T17:36:45.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 36 current snap is 320 2022-04-22T17:36:45.976 INFO:tasks.rados.rados.0.smithi115.stdout:3345: seq_num 1006 ranges {0=49152} 2022-04-22T17:36:45.976 INFO:tasks.rados.rados.0.smithi115.stdout:3345: writing smithi11541303-36 from 0 to 49152 tid 1 2022-04-22T17:36:45.976 INFO:tasks.rados.rados.0.smithi115.stdout:3346: delete oid 13 current snap is 320 2022-04-22T17:36:45.980 INFO:tasks.rados.rados.0.smithi115.stdout:3344: finishing write tid 1 to smithi11541303-27 2022-04-22T17:36:45.980 INFO:tasks.rados.rados.0.smithi115.stdout:3345: finishing write tid 1 to smithi11541303-36 2022-04-22T17:36:45.981 INFO:tasks.rados.rados.0.smithi115.stdout:3345: finishing write tid 2 to smithi11541303-36 2022-04-22T17:36:45.983 INFO:tasks.rados.rados.0.smithi115.stdout:3344: finishing write tid 2 to smithi11541303-27 2022-04-22T17:36:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:3345: finishing write tid 3 to smithi11541303-36 2022-04-22T17:36:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1884 (ObjNum 1006 snap 320 seq_num 1006) dirty exists 2022-04-22T17:36:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:3345: left oid 36 (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:36:45.985 INFO:tasks.rados.rados.0.smithi115.stdout:3345: done (2 left) 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3346: done (1 left) 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3347: read oid 6 snap -1 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3347: expect deleted 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3348: read oid 28 snap -1 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3348: expect (ObjNum 1002 snap 319 seq_num 1002) 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 320 2022-04-22T17:36:45.986 INFO:tasks.rados.rados.0.smithi115.stdout:3349: seq_num 1008 ranges {2236416=90112} 2022-04-22T17:36:46.007 INFO:tasks.rados.rados.0.smithi115.stdout:3349: writing smithi11541303-21 from 2236416 to 2326528 tid 1 2022-04-22T17:36:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:3350: read oid 24 snap -1 2022-04-22T17:36:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:3350: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 320 2022-04-22T17:36:46.008 INFO:tasks.rados.rados.0.smithi115.stdout:3351: seq_num 1009 ranges {9756672=106496} 2022-04-22T17:36:46.061 INFO:tasks.rados.rados.0.smithi115.stdout:3351: writing smithi11541303-47 from 9756672 to 9863168 tid 1 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3352: copy_from oid 44 from oid 36 current snap is 320 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3344: finishing write tid 3 to smithi11541303-27 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2040 (ObjNum 1005 snap 320 seq_num 1005) dirty exists 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3344: left oid 27 (ObjNum 1005 snap 320 seq_num 1005) 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3344: done (6 left) 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3347: done (5 left) 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 320 2022-04-22T17:36:46.115 INFO:tasks.rados.rados.0.smithi115.stdout:3349: finishing write tid 1 to smithi11541303-21 2022-04-22T17:36:46.116 INFO:tasks.rados.rados.0.smithi115.stdout:3349: finishing write tid 2 to smithi11541303-21 2022-04-22T17:36:46.116 INFO:tasks.rados.rados.0.smithi115.stdout:3349: finishing write tid 3 to smithi11541303-21 2022-04-22T17:36:46.116 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1854 (ObjNum 1008 snap 320 seq_num 1008) dirty exists 2022-04-22T17:36:46.116 INFO:tasks.rados.rados.0.smithi115.stdout:3349: left oid 21 (ObjNum 1008 snap 320 seq_num 1008) 2022-04-22T17:36:46.116 INFO:tasks.rados.rados.0.smithi115.stdout:3348: expect (ObjNum 1002 snap 319 seq_num 1002) 2022-04-22T17:36:46.577 INFO:tasks.rados.rados.0.smithi115.stdout:3350: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:47.258 INFO:tasks.rados.rados.0.smithi115.stdout:3352: finishing copy_from to smithi11541303-44 2022-04-22T17:36:47.258 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1884 (ObjNum 1006 snap 320 seq_num 1006) dirty exists 2022-04-22T17:36:47.258 INFO:tasks.rados.rados.0.smithi115.stdout:3352: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:36:47.258 INFO:tasks.rados.rados.0.smithi115.stdout:3351: finishing write tid 1 to smithi11541303-47 2022-04-22T17:36:47.258 INFO:tasks.rados.rados.0.smithi115.stdout:3351: finishing write tid 2 to smithi11541303-47 2022-04-22T17:36:47.259 INFO:tasks.rados.rados.0.smithi115.stdout:3351: finishing write tid 3 to smithi11541303-47 2022-04-22T17:36:47.259 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2035 (ObjNum 1009 snap 320 seq_num 1009) dirty exists 2022-04-22T17:36:47.259 INFO:tasks.rados.rados.0.smithi115.stdout:3351: left oid 47 (ObjNum 1009 snap 320 seq_num 1009) 2022-04-22T17:36:47.259 INFO:tasks.rados.rados.0.smithi115.stdout:3353: seq_num 1010 ranges {0=122880} 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3353: writing smithi11541303-20 from 0 to 122880 tid 1 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3348: done (5 left) 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3349: done (4 left) 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3350: done (3 left) 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3351: done (2 left) 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3352: done (1 left) 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 320 2022-04-22T17:36:47.260 INFO:tasks.rados.rados.0.smithi115.stdout:3354: seq_num 1011 ranges {9543680=139264} 2022-04-22T17:36:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:46 smithi121 bash[36902]: cluster 2022-04-22T17:36:45.339824+0000 mgr.y (mgr.34107) 414 : cluster [DBG] pgmap v748: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2022-04-22T17:36:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:46 smithi121 bash[36902]: cluster 2022-04-22T17:36:45.956841+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2022-04-22T17:36:47.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:46 smithi115 bash[46934]: cluster 2022-04-22T17:36:45.339824+0000 mgr.y (mgr.34107) 414 : cluster [DBG] pgmap v748: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2022-04-22T17:36:47.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:46 smithi115 bash[46934]: cluster 2022-04-22T17:36:45.956841+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2022-04-22T17:36:47.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:46 smithi115 bash[45495]: cluster 2022-04-22T17:36:45.339824+0000 mgr.y (mgr.34107) 414 : cluster [DBG] pgmap v748: 121 pgs: 121 active+clean; 261 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2022-04-22T17:36:47.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:46 smithi115 bash[45495]: cluster 2022-04-22T17:36:45.956841+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3354: writing smithi11541303-15 from 9543680 to 9682944 tid 1 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3355: read oid 9 snap 307 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3355: expect deleted 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3356: setattr oid 40 current snap is 320 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3353: finishing write tid 1 to smithi11541303-20 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3353: finishing write tid 2 to smithi11541303-20 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:3353: finishing write tid 3 to smithi11541303-20 2022-04-22T17:36:47.345 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1895 (ObjNum 1010 snap 320 seq_num 1010) dirty exists 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3353: left oid 20 (ObjNum 1010 snap 320 seq_num 1010) 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3353: done (3 left) 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3355: done (2 left) 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3357: delete oid 8 current snap is 320 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3354: finishing write tid 1 to smithi11541303-15 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3354: finishing write tid 2 to smithi11541303-15 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:3354: finishing write tid 3 to smithi11541303-15 2022-04-22T17:36:47.346 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 1555 (ObjNum 1011 snap 320 seq_num 1011) dirty exists 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3354: left oid 15 (ObjNum 1011 snap 320 seq_num 1011) 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3354: done (2 left) 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3357: done (1 left) 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3358: setattr oid 2 current snap is 320 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2036 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3356: done (1 left) 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 320 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3359: seq_num 1013 ranges {1941504=172032} 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3359: writing smithi11541303-25 from 1941504 to 2113536 tid 1 2022-04-22T17:36:47.347 INFO:tasks.rados.rados.0.smithi115.stdout:3360: read oid 38 snap -1 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3360: expect (ObjNum 947 snap 307 seq_num 947) 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3361: copy_from oid 39 from oid 18 current snap is 320 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1931 (ObjNum 860 snap 281 seq_num 860) dirty exists 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3358: done (3 left) 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3362: setattr oid 11 current snap is 320 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3361: finishing copy_from to smithi11541303-39 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3361: got expected ENOENT (src dne) 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3361: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:36:47.348 INFO:tasks.rados.rados.0.smithi115.stdout:3361: done (3 left) 2022-04-22T17:36:47.349 INFO:tasks.rados.rados.0.smithi115.stdout:3363: snap_create 2022-04-22T17:36:47.349 INFO:tasks.rados.rados.0.smithi115.stdout:3359: finishing write tid 1 to smithi11541303-25 2022-04-22T17:36:47.349 INFO:tasks.rados.rados.0.smithi115.stdout:3359: finishing write tid 2 to smithi11541303-25 2022-04-22T17:36:47.349 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2054 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:47.349 INFO:tasks.rados.rados.0.smithi115.stdout:3360: expect (ObjNum 947 snap 307 seq_num 947) 2022-04-22T17:36:47.440 INFO:tasks.rados.rados.0.smithi115.stdout:3359: finishing write tid 3 to smithi11541303-25 2022-04-22T17:36:47.440 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1897 (ObjNum 1013 snap 320 seq_num 1013) dirty exists 2022-04-22T17:36:47.440 INFO:tasks.rados.rados.0.smithi115.stdout:3359: left oid 25 (ObjNum 1013 snap 320 seq_num 1013) 2022-04-22T17:36:47.980 INFO:tasks.rados.rados.0.smithi115.stdout:3359: done (3 left) 2022-04-22T17:36:47.980 INFO:tasks.rados.rados.0.smithi115.stdout:3360: done (2 left) 2022-04-22T17:36:47.980 INFO:tasks.rados.rados.0.smithi115.stdout:3362: done (1 left) 2022-04-22T17:36:47.980 INFO:tasks.rados.rados.0.smithi115.stdout:3363: done (0 left) 2022-04-22T17:36:47.981 INFO:tasks.rados.rados.0.smithi115.stdout:3364: delete oid 34 current snap is 321 2022-04-22T17:36:47.990 INFO:tasks.rados.rados.0.smithi115.stdout:3364: done (0 left) 2022-04-22T17:36:47.991 INFO:tasks.rados.rados.0.smithi115.stdout:3365: rmattr oid 6 current snap is 321 2022-04-22T17:36:47.991 INFO:tasks.rados.rados.0.smithi115.stdout:3365: done (0 left) 2022-04-22T17:36:47.991 INFO:tasks.rados.rados.0.smithi115.stdout:3366: read oid 36 snap -1 2022-04-22T17:36:47.991 INFO:tasks.rados.rados.0.smithi115.stdout:3366: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:36:47.991 INFO:tasks.rados.rados.0.smithi115.stdout:3367: delete oid 50 current snap is 321 2022-04-22T17:36:48.001 INFO:tasks.rados.rados.0.smithi115.stdout:3366: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:36:48.001 INFO:tasks.rados.rados.0.smithi115.stdout:3366: done (1 left) 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:3367: done (0 left) 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:3368: rollback oid 23 current snap is 321 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 307 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:3369: read oid 4 snap 317 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:3369: expect (ObjNum 986 snap 313 seq_num 986) 2022-04-22T17:36:48.002 INFO:tasks.rados.rados.0.smithi115.stdout:3370: snap_remove snap 318 2022-04-22T17:36:48.991 INFO:tasks.rados.rados.0.smithi115.stdout:3370: done (2 left) 2022-04-22T17:36:48.991 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 321 2022-04-22T17:36:48.992 INFO:tasks.rados.rados.0.smithi115.stdout:3371: seq_num 1016 ranges {122880=221184} 2022-04-22T17:36:48.994 INFO:tasks.rados.rados.0.smithi115.stdout:3371: writing smithi11541303-20 from 122880 to 344064 tid 1 2022-04-22T17:36:48.994 INFO:tasks.rados.rados.0.smithi115.stdout:3372: copy_from oid 26 from oid 15 current snap is 321 2022-04-22T17:36:48.994 INFO:tasks.rados.rados.0.smithi115.stdout:3373: read oid 46 snap -1 2022-04-22T17:36:48.994 INFO:tasks.rados.rados.0.smithi115.stdout:3373: expect deleted 2022-04-22T17:36:48.994 INFO:tasks.rados.rados.0.smithi115.stdout:3374: snap_remove snap 313 2022-04-22T17:36:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:48 smithi121 bash[36902]: cluster 2022-04-22T17:36:47.340412+0000 mgr.y (mgr.34107) 415 : cluster [DBG] pgmap v750: 121 pgs: 121 active+clean; 265 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-04-22T17:36:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:48 smithi121 bash[36902]: cluster 2022-04-22T17:36:47.968347+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2022-04-22T17:36:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:48 smithi115 bash[46934]: cluster 2022-04-22T17:36:47.340412+0000 mgr.y (mgr.34107) 415 : cluster [DBG] pgmap v750: 121 pgs: 121 active+clean; 265 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-04-22T17:36:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:48 smithi115 bash[46934]: cluster 2022-04-22T17:36:47.968347+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2022-04-22T17:36:49.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:48 smithi115 bash[45495]: cluster 2022-04-22T17:36:47.340412+0000 mgr.y (mgr.34107) 415 : cluster [DBG] pgmap v750: 121 pgs: 121 active+clean; 265 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-04-22T17:36:49.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:48 smithi115 bash[45495]: cluster 2022-04-22T17:36:47.968347+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2022-04-22T17:36:49.962 INFO:tasks.rados.rados.0.smithi115.stdout:3374: done (5 left) 2022-04-22T17:36:49.962 INFO:tasks.rados.rados.0.smithi115.stdout:3375: snap_remove snap 299 2022-04-22T17:36:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:49 smithi121 bash[36902]: cluster 2022-04-22T17:36:48.978611+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2022-04-22T17:36:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:49 smithi121 bash[36902]: audit 2022-04-22T17:36:49.405761+0000 mon.a (mon.0) 1348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:50.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:49 smithi121 bash[36902]: cluster 2022-04-22T17:36:49.949399+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[45495]: cluster 2022-04-22T17:36:48.978611+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[45495]: audit 2022-04-22T17:36:49.405761+0000 mon.a (mon.0) 1348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[45495]: cluster 2022-04-22T17:36:49.949399+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[46934]: cluster 2022-04-22T17:36:48.978611+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[46934]: audit 2022-04-22T17:36:49.405761+0000 mon.a (mon.0) 1348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:49 smithi115 bash[46934]: cluster 2022-04-22T17:36:49.949399+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2022-04-22T17:36:50.964 INFO:tasks.rados.rados.0.smithi115.stdout:3375: done (5 left) 2022-04-22T17:36:50.987 INFO:tasks.rados.rados.0.smithi115.stdout:3376: copy_from oid 13 from oid 1 current snap is 321 2022-04-22T17:36:50.987 INFO:tasks.rados.rados.0.smithi115.stdout:3368: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:36:50.987 INFO:tasks.rados.rados.0.smithi115.stdout:3368: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:36:50.987 INFO:tasks.rados.rados.0.smithi115.stdout:3368: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:36:50.988 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2057 (ObjNum 0 snap 0 seq_num 2684356816) dirty exists 2022-04-22T17:36:50.988 INFO:tasks.rados.rados.0.smithi115.stdout:3369: expect (ObjNum 986 snap 313 seq_num 986) 2022-04-22T17:36:51.037 INFO:tasks.rados.rados.0.smithi115.stdout:3371: finishing write tid 1 to smithi11541303-20 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3371: finishing write tid 2 to smithi11541303-20 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3371: finishing write tid 3 to smithi11541303-20 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1899 (ObjNum 1016 snap 321 seq_num 1016) dirty exists 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3371: left oid 20 (ObjNum 1016 snap 321 seq_num 1016) 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3372: finishing copy_from to smithi11541303-26 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 1555 (ObjNum 1011 snap 320 seq_num 1011) dirty exists 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3372: finishing copy_from racing read to smithi11541303-26 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3376: finishing copy_from to smithi11541303-13 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1860 (ObjNum 959 snap 308 seq_num 959) dirty exists 2022-04-22T17:36:51.038 INFO:tasks.rados.rados.0.smithi115.stdout:3376: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3368: done (5 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3369: done (4 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3371: done (3 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3372: done (2 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3373: done (1 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3376: done (0 left) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3377: read oid 11 snap -1 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3377: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 321 2022-04-22T17:36:51.039 INFO:tasks.rados.rados.0.smithi115.stdout:3378: seq_num 1017 ranges {2359296=237568} 2022-04-22T17:36:51.056 INFO:tasks.rados.rados.0.smithi115.stdout:3378: writing smithi11541303-38 from 2359296 to 2596864 tid 1 2022-04-22T17:36:51.056 INFO:tasks.rados.rados.0.smithi115.stdout:3379: read oid 25 snap -1 2022-04-22T17:36:51.057 INFO:tasks.rados.rados.0.smithi115.stdout:3379: expect (ObjNum 1013 snap 320 seq_num 1013) 2022-04-22T17:36:51.057 INFO:tasks.rados.rados.0.smithi115.stdout:3380: rmattr oid 1 current snap is 321 2022-04-22T17:36:51.057 INFO:tasks.rados.rados.0.smithi115.stdout:3377: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:36:51.085 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1886 (ObjNum 959 snap 308 seq_num 959) dirty exists 2022-04-22T17:36:51.086 INFO:tasks.rados.rados.0.smithi115.stdout:3378: finishing write tid 1 to smithi11541303-38 2022-04-22T17:36:51.086 INFO:tasks.rados.rados.0.smithi115.stdout:3378: finishing write tid 2 to smithi11541303-38 2022-04-22T17:36:51.086 INFO:tasks.rados.rados.0.smithi115.stdout:3378: finishing write tid 3 to smithi11541303-38 2022-04-22T17:36:51.086 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1774 (ObjNum 1017 snap 321 seq_num 1017) dirty exists 2022-04-22T17:36:51.087 INFO:tasks.rados.rados.0.smithi115.stdout:3378: left oid 38 (ObjNum 1017 snap 321 seq_num 1017) 2022-04-22T17:36:51.087 INFO:tasks.rados.rados.0.smithi115.stdout:3379: expect (ObjNum 1013 snap 320 seq_num 1013) 2022-04-22T17:36:51.174 INFO:tasks.rados.rados.0.smithi115.stdout:3377: done (3 left) 2022-04-22T17:36:51.174 INFO:tasks.rados.rados.0.smithi115.stdout:3378: done (2 left) 2022-04-22T17:36:51.174 INFO:tasks.rados.rados.0.smithi115.stdout:3379: done (1 left) 2022-04-22T17:36:51.175 INFO:tasks.rados.rados.0.smithi115.stdout:3380: done (0 left) 2022-04-22T17:36:51.175 INFO:tasks.rados.rados.0.smithi115.stdout:3381: read oid 39 snap -1 2022-04-22T17:36:51.175 INFO:tasks.rados.rados.0.smithi115.stdout:3381: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:51.176 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 321 2022-04-22T17:36:51.176 INFO:tasks.rados.rados.0.smithi115.stdout:3382: seq_num 1018 ranges {3710976=253952} 2022-04-22T17:36:51.197 INFO:tasks.rados.rados.0.smithi115.stdout:3382: writing smithi11541303-45 from 3710976 to 3964928 tid 1 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:3383: read oid 3 snap -1 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:3383: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:3384: read oid 29 snap -1 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:3384: expect (ObjNum 996 snap 315 seq_num 996) 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 321 2022-04-22T17:36:51.198 INFO:tasks.rados.rados.0.smithi115.stdout:3385: seq_num 1019 ranges {8814592=270336} 2022-04-22T17:36:51.252 INFO:tasks.rados.rados.0.smithi115.stdout:3385: writing smithi11541303-28 from 8814592 to 9084928 tid 1 2022-04-22T17:36:51.253 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 321 2022-04-22T17:36:51.253 INFO:tasks.rados.rados.0.smithi115.stdout:3386: seq_num 1020 ranges {344064=286720} 2022-04-22T17:36:51.256 INFO:tasks.rados.rados.0.smithi115.stdout:3386: writing smithi11541303-20 from 344064 to 630784 tid 1 2022-04-22T17:36:51.256 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 321 2022-04-22T17:36:51.257 INFO:tasks.rados.rados.0.smithi115.stdout:3387: seq_num 1021 ranges {49152=303104} 2022-04-22T17:36:51.259 INFO:tasks.rados.rados.0.smithi115.stdout:3387: writing smithi11541303-44 from 49152 to 352256 tid 1 2022-04-22T17:36:51.259 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 321 2022-04-22T17:36:51.260 INFO:tasks.rados.rados.0.smithi115.stdout:3388: seq_num 1022 ranges {2113536=319488} 2022-04-22T17:36:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:50 smithi121 bash[36902]: cluster 2022-04-22T17:36:49.341086+0000 mgr.y (mgr.34107) 416 : cluster [DBG] pgmap v753: 121 pgs: 121 active+clean; 266 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2022-04-22T17:36:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:50 smithi121 bash[36902]: cluster 2022-04-22T17:36:50.949994+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2022-04-22T17:36:51.273 INFO:tasks.rados.rados.0.smithi115.stdout:3388: writing smithi11541303-25 from 2113536 to 2433024 tid 1 2022-04-22T17:36:51.273 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 321 2022-04-22T17:36:51.273 INFO:tasks.rados.rados.0.smithi115.stdout:3389: seq_num 1023 ranges {6209536=335872} 2022-04-22T17:36:51.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:50 smithi115 bash[46934]: cluster 2022-04-22T17:36:49.341086+0000 mgr.y (mgr.34107) 416 : cluster [DBG] pgmap v753: 121 pgs: 121 active+clean; 266 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2022-04-22T17:36:51.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:50 smithi115 bash[46934]: cluster 2022-04-22T17:36:50.949994+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2022-04-22T17:36:51.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:50 smithi115 bash[45495]: cluster 2022-04-22T17:36:49.341086+0000 mgr.y (mgr.34107) 416 : cluster [DBG] pgmap v753: 121 pgs: 121 active+clean; 266 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2022-04-22T17:36:51.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:50 smithi115 bash[45495]: cluster 2022-04-22T17:36:50.949994+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2022-04-22T17:36:51.310 INFO:tasks.rados.rados.0.smithi115.stdout:3389: writing smithi11541303-13 from 6209536 to 6545408 tid 1 2022-04-22T17:36:51.310 INFO:tasks.rados.rados.0.smithi115.stdout:3390: rmattr oid 8 current snap is 321 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3390: done (9 left) 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3391: rollback oid 21 current snap is 321 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 307 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3382: finishing write tid 1 to smithi11541303-45 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3382: finishing write tid 2 to smithi11541303-45 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3382: finishing write tid 3 to smithi11541303-45 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2042 (ObjNum 1018 snap 321 seq_num 1018) dirty exists 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3382: left oid 45 (ObjNum 1018 snap 321 seq_num 1018) 2022-04-22T17:36:51.311 INFO:tasks.rados.rados.0.smithi115.stdout:3382: done (9 left) 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3392: rmattr oid 27 current snap is 321 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3392: done (9 left) 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3393: rmattr oid 6 current snap is 321 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3393: done (9 left) 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3394: read oid 35 snap -1 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3394: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3395: read oid 37 snap -1 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3395: expect (ObjNum 1001 snap 317 seq_num 1001) 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3396: copy_from oid 7 from oid 24 current snap is 321 2022-04-22T17:36:51.312 INFO:tasks.rados.rados.0.smithi115.stdout:3381: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:36:52.002 INFO:tasks.rados.rados.0.smithi115.stdout:3383: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:36:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:51.635Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:51.635Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:51 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:51.635Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" 2022-04-22T17:36:52.056 INFO:tasks.rados.rados.0.smithi115.stdout:3384: expect (ObjNum 996 snap 315 seq_num 996) 2022-04-22T17:36:52.786 INFO:tasks.rados.rados.0.smithi115.stdout:3385: finishing write tid 1 to smithi11541303-28 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:3381: done (11 left) 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:3383: done (10 left) 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:3384: done (9 left) 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 321 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:3385: finishing write tid 2 to smithi11541303-28 2022-04-22T17:36:52.787 INFO:tasks.rados.rados.0.smithi115.stdout:3385: finishing write tid 3 to smithi11541303-28 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1893 (ObjNum 1019 snap 321 seq_num 1019) dirty exists 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3385: left oid 28 (ObjNum 1019 snap 321 seq_num 1019) 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3387: finishing write tid 1 to smithi11541303-44 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3387: finishing write tid 2 to smithi11541303-44 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3387: finishing write tid 3 to smithi11541303-44 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1886 (ObjNum 1021 snap 321 seq_num 1021) dirty exists 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3387: left oid 44 (ObjNum 1021 snap 321 seq_num 1021) 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3386: finishing write tid 1 to smithi11541303-20 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3386: finishing write tid 2 to smithi11541303-20 2022-04-22T17:36:52.788 INFO:tasks.rados.rados.0.smithi115.stdout:3386: finishing write tid 3 to smithi11541303-20 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1901 (ObjNum 1020 snap 321 seq_num 1020) dirty exists 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3386: left oid 20 (ObjNum 1020 snap 321 seq_num 1020) 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3388: finishing write tid 1 to smithi11541303-25 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3388: finishing write tid 2 to smithi11541303-25 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3388: finishing write tid 3 to smithi11541303-25 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1903 (ObjNum 1022 snap 321 seq_num 1022) dirty exists 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3388: left oid 25 (ObjNum 1022 snap 321 seq_num 1022) 2022-04-22T17:36:52.789 INFO:tasks.rados.rados.0.smithi115.stdout:3397: seq_num 1024 ranges {2596864=352256} 2022-04-22T17:36:52.807 INFO:tasks.rados.rados.0.smithi115.stdout:3397: writing smithi11541303-38 from 2596864 to 2949120 tid 1 2022-04-22T17:36:52.807 INFO:tasks.rados.rados.0.smithi115.stdout:3385: done (9 left) 2022-04-22T17:36:52.808 INFO:tasks.rados.rados.0.smithi115.stdout:3386: done (8 left) 2022-04-22T17:36:52.808 INFO:tasks.rados.rados.0.smithi115.stdout:3387: done (7 left) 2022-04-22T17:36:52.808 INFO:tasks.rados.rados.0.smithi115.stdout:3388: done (6 left) 2022-04-22T17:36:52.808 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 39 current snap is 321 2022-04-22T17:36:52.808 INFO:tasks.rados.rados.0.smithi115.stdout:3398: seq_num 1025 ranges {10018816=368640} 2022-04-22T17:36:52.865 INFO:tasks.rados.rados.0.smithi115.stdout:3398: writing smithi11541303-39 from 10018816 to 10387456 tid 1 2022-04-22T17:36:52.865 INFO:tasks.rados.rados.0.smithi115.stdout:3399: snap_remove snap 317 2022-04-22T17:36:53.021 INFO:tasks.rados.rados.0.smithi115.stdout:3399: done (7 left) 2022-04-22T17:36:53.021 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 321 2022-04-22T17:36:53.021 INFO:tasks.rados.rados.0.smithi115.stdout:3400: seq_num 1026 ranges {6209536=385024} 2022-04-22T17:36:53.061 INFO:tasks.rados.rados.0.smithi115.stdout:3400: writing smithi11541303-1 from 6209536 to 6594560 tid 1 2022-04-22T17:36:53.061 INFO:tasks.rados.rados.0.smithi115.stdout:3401: read oid 10 snap -1 2022-04-22T17:36:53.061 INFO:tasks.rados.rados.0.smithi115.stdout:3401: expect (ObjNum 951 snap 308 seq_num 951) 2022-04-22T17:36:53.061 INFO:tasks.rados.rados.0.smithi115.stdout:3402: snap_create 2022-04-22T17:36:53.061 INFO:tasks.rados.rados.0.smithi115.stdout:3394: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:53.065 INFO:tasks.rados.rados.0.smithi115.stdout:3389: finishing write tid 1 to smithi11541303-13 2022-04-22T17:36:53.065 INFO:tasks.rados.rados.0.smithi115.stdout:3389: finishing write tid 2 to smithi11541303-13 2022-04-22T17:36:53.065 INFO:tasks.rados.rados.0.smithi115.stdout:3391: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:36:53.065 INFO:tasks.rados.rados.0.smithi115.stdout:3391: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:3391: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1857 (ObjNum 886 snap 289 seq_num 886) dirty exists 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:3389: finishing write tid 3 to smithi11541303-13 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1905 (ObjNum 1023 snap 321 seq_num 1023) dirty exists 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:3389: left oid 13 (ObjNum 1023 snap 321 seq_num 1023) 2022-04-22T17:36:53.066 INFO:tasks.rados.rados.0.smithi115.stdout:3395: expect (ObjNum 1001 snap 317 seq_num 1001) 2022-04-22T17:36:53.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:52 smithi115 bash[45495]: cluster 2022-04-22T17:36:51.341586+0000 mgr.y (mgr.34107) 417 : cluster [DBG] pgmap v756: 121 pgs: 121 active+clean; 275 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2022-04-22T17:36:53.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:52 smithi115 bash[46934]: cluster 2022-04-22T17:36:51.341586+0000 mgr.y (mgr.34107) 417 : cluster [DBG] pgmap v756: 121 pgs: 121 active+clean; 275 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2022-04-22T17:36:53.208 INFO:tasks.rados.rados.0.smithi115.stdout:3396: finishing copy_from to smithi11541303-7 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1933 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:3396: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:3397: finishing write tid 1 to smithi11541303-38 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:3397: finishing write tid 2 to smithi11541303-38 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:3397: finishing write tid 3 to smithi11541303-38 2022-04-22T17:36:53.209 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1776 (ObjNum 1024 snap 321 seq_num 1024) dirty exists 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3397: left oid 38 (ObjNum 1024 snap 321 seq_num 1024) 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3398: finishing write tid 1 to smithi11541303-39 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3398: finishing write tid 2 to smithi11541303-39 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3398: finishing write tid 3 to smithi11541303-39 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1933 (ObjNum 1025 snap 321 seq_num 1025) dirty exists 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3398: left oid 39 (ObjNum 1025 snap 321 seq_num 1025) 2022-04-22T17:36:53.210 INFO:tasks.rados.rados.0.smithi115.stdout:3400: finishing write tid 1 to smithi11541303-1 2022-04-22T17:36:53.211 INFO:tasks.rados.rados.0.smithi115.stdout:3400: finishing write tid 2 to smithi11541303-1 2022-04-22T17:36:53.211 INFO:tasks.rados.rados.0.smithi115.stdout:3400: finishing write tid 3 to smithi11541303-1 2022-04-22T17:36:53.211 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1935 (ObjNum 1026 snap 321 seq_num 1026) dirty exists 2022-04-22T17:36:53.211 INFO:tasks.rados.rados.0.smithi115.stdout:3400: left oid 1 (ObjNum 1026 snap 321 seq_num 1026) 2022-04-22T17:36:53.211 INFO:tasks.rados.rados.0.smithi115.stdout:3401: expect (ObjNum 951 snap 308 seq_num 951) 2022-04-22T17:36:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:53 smithi121 bash[36902]: cluster 2022-04-22T17:36:51.341586+0000 mgr.y (mgr.34107) 417 : cluster [DBG] pgmap v756: 121 pgs: 121 active+clean; 275 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2022-04-22T17:36:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:53.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:53.530Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2022-04-22T17:36:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:53 smithi115 bash[34566]: level=error ts=2022-04-22T17:36:53.530Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="Post https://172.21.15.121:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.121 because it doesn't contain any IP SANs; Post https://172.21.15.115:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2022-04-22T17:36:54.023 INFO:tasks.rados.rados.0.smithi115.stdout:3389: done (9 left) 2022-04-22T17:36:54.023 INFO:tasks.rados.rados.0.smithi115.stdout:3391: done (8 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3394: done (7 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3395: done (6 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3396: done (5 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3397: done (4 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3398: done (3 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3400: done (2 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3401: done (1 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3402: done (0 left) 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 47 current snap is 322 2022-04-22T17:36:54.024 INFO:tasks.rados.rados.0.smithi115.stdout:3403: seq_num 1027 ranges {9863168=409600} 2022-04-22T17:36:54.085 INFO:tasks.rados.rados.0.smithi115.stdout:3403: writing smithi11541303-47 from 9863168 to 10272768 tid 1 2022-04-22T17:36:54.085 INFO:tasks.rados.rados.0.smithi115.stdout:3404: read oid 6 snap -1 2022-04-22T17:36:54.086 INFO:tasks.rados.rados.0.smithi115.stdout:3404: expect deleted 2022-04-22T17:36:54.086 INFO:tasks.rados.rados.0.smithi115.stdout:3405: rmattr oid 37 current snap is 322 2022-04-22T17:36:54.086 INFO:tasks.rados.rados.0.smithi115.stdout:3405: done (2 left) 2022-04-22T17:36:54.086 INFO:tasks.rados.rados.0.smithi115.stdout:3406: read oid 37 snap 320 2022-04-22T17:36:54.086 INFO:tasks.rados.rados.0.smithi115.stdout:3406: expect (ObjNum 1001 snap 317 seq_num 1001) 2022-04-22T17:36:54.087 INFO:tasks.rados.rados.0.smithi115.stdout:3407: rollback oid 16 current snap is 322 2022-04-22T17:36:54.087 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 16 to 319 2022-04-22T17:36:54.087 INFO:tasks.rados.rados.0.smithi115.stdout:3408: read oid 2 snap 320 2022-04-22T17:36:54.087 INFO:tasks.rados.rados.0.smithi115.stdout:3408: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:36:54.088 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 322 2022-04-22T17:36:54.088 INFO:tasks.rados.rados.0.smithi115.stdout:3409: seq_num 1028 ranges {3964928=425984} 2022-04-22T17:36:54.110 INFO:tasks.rados.rados.0.smithi115.stdout:3409: writing smithi11541303-45 from 3964928 to 4390912 tid 1 2022-04-22T17:36:54.110 INFO:tasks.rados.rados.0.smithi115.stdout:3410: delete oid 1 current snap is 322 2022-04-22T17:36:54.110 INFO:tasks.rados.rados.0.smithi115.stdout:3403: finishing write tid 1 to smithi11541303-47 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3403: finishing write tid 2 to smithi11541303-47 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3407: finishing rollback tid 0 to smithi11541303-16 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3407: finishing rollback tid 1 to smithi11541303-16 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3407: finishing rollback tid 2 to smithi11541303-16 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 1854 (ObjNum 993 snap 315 seq_num 993) dirty exists 2022-04-22T17:36:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3406: expect (ObjNum 1001 snap 317 seq_num 1001) 2022-04-22T17:36:54.215 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:54 smithi115 bash[45495]: cluster 2022-04-22T17:36:53.008341+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2022-04-22T17:36:54.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:54 smithi115 bash[46934]: cluster 2022-04-22T17:36:53.008341+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2022-04-22T17:36:54.284 INFO:tasks.rados.rados.0.smithi115.stdout:3403: finishing write tid 3 to smithi11541303-47 2022-04-22T17:36:54.284 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2038 (ObjNum 1027 snap 322 seq_num 1027) dirty exists 2022-04-22T17:36:54.285 INFO:tasks.rados.rados.0.smithi115.stdout:3403: left oid 47 (ObjNum 1027 snap 322 seq_num 1027) 2022-04-22T17:36:54.285 INFO:tasks.rados.rados.0.smithi115.stdout:3408: expect (ObjNum 860 snap 281 seq_num 860) 2022-04-22T17:36:54.421 INFO:tasks.rados.rados.0.smithi115.stdout:3409: finishing write tid 1 to smithi11541303-45 2022-04-22T17:36:54.421 INFO:tasks.rados.rados.0.smithi115.stdout:3409: finishing write tid 2 to smithi11541303-45 2022-04-22T17:36:54.421 INFO:tasks.rados.rados.0.smithi115.stdout:3409: finishing write tid 3 to smithi11541303-45 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2044 (ObjNum 1028 snap 322 seq_num 1028) dirty exists 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3409: left oid 45 (ObjNum 1028 snap 322 seq_num 1028) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3403: done (6 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3404: done (5 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3406: done (4 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3407: done (3 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3408: done (2 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3409: done (1 left) 2022-04-22T17:36:54.422 INFO:tasks.rados.rados.0.smithi115.stdout:3410: done (0 left) 2022-04-22T17:36:54.423 INFO:tasks.rados.rados.0.smithi115.stdout:3411: delete oid 2 current snap is 322 2022-04-22T17:36:54.425 INFO:tasks.rados.rados.0.smithi115.stdout:3411: done (0 left) 2022-04-22T17:36:54.425 INFO:tasks.rados.rados.0.smithi115.stdout:3412: snap_create 2022-04-22T17:36:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:54 smithi121 bash[36902]: cluster 2022-04-22T17:36:53.008341+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2022-04-22T17:36:54.525 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:54 smithi115 systemd[1]: Stopping Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c... 2022-04-22T17:36:54.525 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:54 smithi115 bash[34566]: level=info ts=2022-04-22T17:36:54.343Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2022-04-22T17:36:54.963 INFO:tasks.rados.rados.0.smithi115.stdout:3412: done (0 left) 2022-04-22T17:36:54.964 INFO:tasks.rados.rados.0.smithi115.stdout:3413: read oid 49 snap -1 2022-04-22T17:36:54.964 INFO:tasks.rados.rados.0.smithi115.stdout:3413: expect (ObjNum 948 snap 307 seq_num 948) 2022-04-22T17:36:54.964 INFO:tasks.rados.rados.0.smithi115.stdout:3414: snap_remove snap 307 2022-04-22T17:36:55.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[45495]: cluster 2022-04-22T17:36:53.342527+0000 mgr.y (mgr.34107) 418 : cluster [DBG] pgmap v758: 121 pgs: 121 active+clean; 270 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 6.1 MiB/s wr, 34 op/s 2022-04-22T17:36:55.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[45495]: cluster 2022-04-22T17:36:54.010908+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2022-04-22T17:36:55.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[45495]: audit 2022-04-22T17:36:54.408122+0000 mon.a (mon.0) 1353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:55.282 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[45495]: cluster 2022-04-22T17:36:54.947563+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2022-04-22T17:36:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[46934]: cluster 2022-04-22T17:36:53.342527+0000 mgr.y (mgr.34107) 418 : cluster [DBG] pgmap v758: 121 pgs: 121 active+clean; 270 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 6.1 MiB/s wr, 34 op/s 2022-04-22T17:36:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[46934]: cluster 2022-04-22T17:36:54.010908+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2022-04-22T17:36:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[46934]: audit 2022-04-22T17:36:54.408122+0000 mon.a (mon.0) 1353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[46934]: cluster 2022-04-22T17:36:54.947563+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2022-04-22T17:36:55.283 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[60340]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-alertmanager.a 2022-04-22T17:36:55.284 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[60382]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-alertmanager-a 2022-04-22T17:36:55.284 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 bash[60395]: Error response from daemon: No such container: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c-alertmanager.a 2022-04-22T17:36:55.284 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 systemd[1]: ceph-b6061a0a-c25f-11ec-8c39-001a4aab830c@alertmanager.a.service: Succeeded. 2022-04-22T17:36:55.284 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 systemd[1]: Stopped Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:36:55.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:55 smithi121 bash[36902]: cluster 2022-04-22T17:36:53.342527+0000 mgr.y (mgr.34107) 418 : cluster [DBG] pgmap v758: 121 pgs: 121 active+clean; 270 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 MiB/s rd, 6.1 MiB/s wr, 34 op/s 2022-04-22T17:36:55.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:55 smithi121 bash[36902]: cluster 2022-04-22T17:36:54.010908+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2022-04-22T17:36:55.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:55 smithi121 bash[36902]: audit 2022-04-22T17:36:54.408122+0000 mon.a (mon.0) 1353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:36:55.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:55 smithi121 bash[36902]: cluster 2022-04-22T17:36:54.947563+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2022-04-22T17:36:55.539 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:55 smithi115 systemd[1]: Started Ceph alertmanager.a for b6061a0a-c25f-11ec-8c39-001a4aab830c. 2022-04-22T17:36:55.975 INFO:tasks.rados.rados.0.smithi115.stdout:3414: done (1 left) 2022-04-22T17:36:55.975 INFO:tasks.rados.rados.0.smithi115.stdout:3415: snap_remove snap 320 2022-04-22T17:36:56.972 INFO:tasks.rados.rados.0.smithi115.stdout:3415: done (1 left) 2022-04-22T17:36:56.972 INFO:tasks.rados.rados.0.smithi115.stdout:3416: delete oid 14 current snap is 323 2022-04-22T17:36:56.972 INFO:tasks.rados.rados.0.smithi115.stdout:3413: expect (ObjNum 948 snap 307 seq_num 948) 2022-04-22T17:36:57.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: cluster 2022-04-22T17:36:55.343229+0000 mgr.y (mgr.34107) 419 : cluster [DBG] pgmap v761: 121 pgs: 121 active+clean; 241 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2022-04-22T17:36:57.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: audit 2022-04-22T17:36:55.539952+0000 mon.a (mon.0) 1355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:57.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: audit 2022-04-22T17:36:55.541327+0000 mon.a (mon.0) 1356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:57.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: audit 2022-04-22T17:36:55.543094+0000 mon.a (mon.0) 1357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:57.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: audit 2022-04-22T17:36:55.544085+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:57.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[45495]: cluster 2022-04-22T17:36:55.959317+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2022-04-22T17:36:57.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: cluster 2022-04-22T17:36:55.343229+0000 mgr.y (mgr.34107) 419 : cluster [DBG] pgmap v761: 121 pgs: 121 active+clean; 241 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2022-04-22T17:36:57.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: audit 2022-04-22T17:36:55.539952+0000 mon.a (mon.0) 1355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:57.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: audit 2022-04-22T17:36:55.541327+0000 mon.a (mon.0) 1356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:57.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: audit 2022-04-22T17:36:55.543094+0000 mon.a (mon.0) 1357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:57.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: audit 2022-04-22T17:36:55.544085+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:57.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:56 smithi115 bash[46934]: cluster 2022-04-22T17:36:55.959317+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2022-04-22T17:36:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: cluster 2022-04-22T17:36:55.343229+0000 mgr.y (mgr.34107) 419 : cluster [DBG] pgmap v761: 121 pgs: 121 active+clean; 241 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2022-04-22T17:36:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: audit 2022-04-22T17:36:55.539952+0000 mon.a (mon.0) 1355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:36:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: audit 2022-04-22T17:36:55.541327+0000 mon.a (mon.0) 1356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:36:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: audit 2022-04-22T17:36:55.543094+0000 mon.a (mon.0) 1357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:36:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: audit 2022-04-22T17:36:55.544085+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:36:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:56 smithi121 bash[36902]: cluster 2022-04-22T17:36:55.959317+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2022-04-22T17:36:57.019 INFO:tasks.rados.rados.0.smithi115.stdout:3413: done (1 left) 2022-04-22T17:36:57.020 INFO:tasks.rados.rados.0.smithi115.stdout:3416: done (0 left) 2022-04-22T17:36:57.020 INFO:tasks.rados.rados.0.smithi115.stdout:3417: rollback oid 19 current snap is 323 2022-04-22T17:36:57.020 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 322 2022-04-22T17:36:57.020 INFO:tasks.rados.rados.0.smithi115.stdout:3418: copy_from oid 18 from oid 42 current snap is 323 2022-04-22T17:36:57.020 INFO:tasks.rados.rados.0.smithi115.stdout:3419: read oid 30 snap 322 2022-04-22T17:36:57.021 INFO:tasks.rados.rados.0.smithi115.stdout:3419: expect (ObjNum 952 snap 308 seq_num 952) 2022-04-22T17:36:57.021 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 323 2022-04-22T17:36:57.021 INFO:tasks.rados.rados.0.smithi115.stdout:3420: seq_num 1032 ranges {0=491520} 2022-04-22T17:36:57.024 INFO:tasks.rados.rados.0.smithi115.stdout:3420: writing smithi11541303-46 from 0 to 491520 tid 1 2022-04-22T17:36:57.025 INFO:tasks.rados.rados.0.smithi115.stdout:3421: snap_create 2022-04-22T17:36:57.026 INFO:tasks.rados.rados.0.smithi115.stdout:3417: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:36:57.026 INFO:tasks.rados.rados.0.smithi115.stdout:3417: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:36:57.026 INFO:tasks.rados.rados.0.smithi115.stdout:3417: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:36:57.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1779 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:57.030 INFO:tasks.rados.rados.0.smithi115.stdout:3418: finishing copy_from to smithi11541303-18 2022-04-22T17:36:57.030 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1393 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:36:57.031 INFO:tasks.rados.rados.0.smithi115.stdout:3418: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:36:57.035 INFO:tasks.rados.rados.0.smithi115.stdout:3420: finishing write tid 1 to smithi11541303-46 2022-04-22T17:36:57.036 INFO:tasks.rados.rados.0.smithi115.stdout:3420: finishing write tid 2 to smithi11541303-46 2022-04-22T17:36:57.038 INFO:tasks.rados.rados.0.smithi115.stdout:3420: finishing write tid 3 to smithi11541303-46 2022-04-22T17:36:57.038 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 2059 (ObjNum 1032 snap 323 seq_num 1032) dirty exists 2022-04-22T17:36:57.038 INFO:tasks.rados.rados.0.smithi115.stdout:3420: left oid 46 (ObjNum 1032 snap 323 seq_num 1032) 2022-04-22T17:36:57.047 INFO:tasks.rados.rados.0.smithi115.stdout:3419: expect (ObjNum 952 snap 308 seq_num 952) 2022-04-22T17:36:57.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.140Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2022-04-22T17:36:57.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.140Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.145Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.147Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.192Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.192Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.198Z caller=main.go:518 msg=Listening address=:9093 2022-04-22T17:36:57.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:57.198Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2022-04-22T17:36:57.982 INFO:tasks.rados.rados.0.smithi115.stdout:3417: done (4 left) 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:3418: done (3 left) 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:3419: done (2 left) 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:3420: done (1 left) 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:3421: done (0 left) 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 27 current snap is 324 2022-04-22T17:36:57.983 INFO:tasks.rados.rados.0.smithi115.stdout:3422: seq_num 1033 ranges {32768=507904} 2022-04-22T17:36:57.989 INFO:tasks.rados.rados.0.smithi115.stdout:3422: writing smithi11541303-27 from 32768 to 540672 tid 1 2022-04-22T17:36:57.990 INFO:tasks.rados.rados.0.smithi115.stdout:3423: delete oid 49 current snap is 324 2022-04-22T17:36:57.998 INFO:tasks.rados.rados.0.smithi115.stdout:3422: finishing write tid 1 to smithi11541303-27 2022-04-22T17:36:57.999 INFO:tasks.rados.rados.0.smithi115.stdout:3422: finishing write tid 2 to smithi11541303-27 2022-04-22T17:36:57.999 INFO:tasks.rados.rados.0.smithi115.stdout:3423: done (1 left) 2022-04-22T17:36:57.999 INFO:tasks.rados.rados.0.smithi115.stdout:3424: delete oid 16 current snap is 324 2022-04-22T17:36:58.000 INFO:tasks.rados.rados.0.smithi115.stdout:3422: finishing write tid 3 to smithi11541303-27 2022-04-22T17:36:58.000 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2046 (ObjNum 1033 snap 324 seq_num 1033) dirty exists 2022-04-22T17:36:58.000 INFO:tasks.rados.rados.0.smithi115.stdout:3422: left oid 27 (ObjNum 1033 snap 324 seq_num 1033) 2022-04-22T17:36:58.003 INFO:tasks.rados.rados.0.smithi115.stdout:3422: done (1 left) 2022-04-22T17:36:58.003 INFO:tasks.rados.rados.0.smithi115.stdout:3424: done (0 left) 2022-04-22T17:36:58.003 INFO:tasks.rados.rados.0.smithi115.stdout:3425: setattr oid 19 current snap is 324 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:3426: read oid 50 snap -1 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:3426: expect deleted 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:3427: setattr oid 14 current snap is 324 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:3426: done (2 left) 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:3428: copy_from oid 15 from oid 3 current snap is 324 2022-04-22T17:36:58.012 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 324 2022-04-22T17:36:58.013 INFO:tasks.rados.rados.0.smithi115.stdout:3429: seq_num 1036 ranges {761856=557056} 2022-04-22T17:36:58.018 INFO:tasks.rados.rados.0.smithi115.stdout:3429: writing smithi11541303-42 from 761856 to 1318912 tid 1 2022-04-22T17:36:58.018 INFO:tasks.rados.rados.0.smithi115.stdout:3430: delete oid 37 current snap is 324 2022-04-22T17:36:58.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1780 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:36:58.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1894 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty exists 2022-04-22T17:36:58.023 INFO:tasks.rados.rados.0.smithi115.stdout:3425: done (4 left) 2022-04-22T17:36:58.067 INFO:tasks.rados.rados.0.smithi115.stdout:3427: done (3 left) 2022-04-22T17:36:58.067 INFO:tasks.rados.rados.0.smithi115.stdout:3430: done (2 left) 2022-04-22T17:36:58.067 INFO:tasks.rados.rados.0.smithi115.stdout:3431: read oid 23 snap -1 2022-04-22T17:36:58.067 INFO:tasks.rados.rados.0.smithi115.stdout:3431: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:58.067 INFO:tasks.rados.rados.0.smithi115.stdout:3432: read oid 34 snap 319 2022-04-22T17:36:58.068 INFO:tasks.rados.rados.0.smithi115.stdout:3432: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:36:58.068 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 324 2022-04-22T17:36:58.068 INFO:tasks.rados.rados.0.smithi115.stdout:3433: seq_num 1038 ranges {9289728=589824} 2022-04-22T17:36:58.083 INFO:tasks.rados.rados.0.smithi115.stdout:3433: writing smithi11541303-43 from 9289728 to 9879552 tid 1 2022-04-22T17:36:58.084 INFO:tasks.rados.rados.0.smithi115.stdout:3434: delete oid 1 current snap is 324 2022-04-22T17:36:58.084 INFO:tasks.rados.rados.0.smithi115.stdout:3431: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3429: finishing write tid 1 to smithi11541303-42 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3429: finishing write tid 2 to smithi11541303-42 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3428: finishing copy_from to smithi11541303-15 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 2051 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3429: finishing write tid 3 to smithi11541303-42 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2109 (ObjNum 1036 snap 324 seq_num 1036) dirty exists 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3429: left oid 42 (ObjNum 1036 snap 324 seq_num 1036) 2022-04-22T17:36:58.089 INFO:tasks.rados.rados.0.smithi115.stdout:3428: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:36:58.090 INFO:tasks.rados.rados.0.smithi115.stdout:3432: expect (ObjNum 746 snap 256 seq_num 746) 2022-04-22T17:36:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:57 smithi121 bash[36902]: cluster 2022-04-22T17:36:56.957661+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2022-04-22T17:36:58.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[45495]: cluster 2022-04-22T17:36:56.957661+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2022-04-22T17:36:58.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:57 smithi115 bash[46934]: cluster 2022-04-22T17:36:56.957661+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2022-04-22T17:36:58.995 INFO:tasks.rados.rados.0.smithi115.stdout:3433: finishing write tid 1 to smithi11541303-43 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:3433: finishing write tid 2 to smithi11541303-43 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:3433: finishing write tid 3 to smithi11541303-43 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 1907 (ObjNum 1038 snap 324 seq_num 1038) dirty exists 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:3433: left oid 43 (ObjNum 1038 snap 324 seq_num 1038) 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:3428: done (5 left) 2022-04-22T17:36:58.996 INFO:tasks.rados.rados.0.smithi115.stdout:3429: done (4 left) 2022-04-22T17:36:58.997 INFO:tasks.rados.rados.0.smithi115.stdout:3431: done (3 left) 2022-04-22T17:36:58.999 INFO:tasks.rados.rados.0.smithi115.stdout:3432: done (2 left) 2022-04-22T17:36:58.999 INFO:tasks.rados.rados.0.smithi115.stdout:3433: done (1 left) 2022-04-22T17:36:58.999 INFO:tasks.rados.rados.0.smithi115.stdout:3434: done (0 left) 2022-04-22T17:36:59.000 INFO:tasks.rados.rados.0.smithi115.stdout:3435: copy_from oid 47 from oid 48 current snap is 324 2022-04-22T17:36:59.000 INFO:tasks.rados.rados.0.smithi115.stdout:3436: snap_remove snap 321 2022-04-22T17:36:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:58 smithi121 bash[36902]: cluster 2022-04-22T17:36:57.343845+0000 mgr.y (mgr.34107) 420 : cluster [DBG] pgmap v764: 121 pgs: 121 active+clean; 224 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 536 KiB/s wr, 10 op/s 2022-04-22T17:36:59.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:58 smithi121 bash[36902]: cluster 2022-04-22T17:36:57.968424+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2022-04-22T17:36:59.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:36:59 smithi115 bash[60466]: level=info ts=2022-04-22T17:36:59.147Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000735517s 2022-04-22T17:36:59.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:58 smithi115 bash[45495]: cluster 2022-04-22T17:36:57.343845+0000 mgr.y (mgr.34107) 420 : cluster [DBG] pgmap v764: 121 pgs: 121 active+clean; 224 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 536 KiB/s wr, 10 op/s 2022-04-22T17:36:59.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:58 smithi115 bash[45495]: cluster 2022-04-22T17:36:57.968424+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2022-04-22T17:36:59.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:58 smithi115 bash[46934]: cluster 2022-04-22T17:36:57.343845+0000 mgr.y (mgr.34107) 420 : cluster [DBG] pgmap v764: 121 pgs: 121 active+clean; 224 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 536 KiB/s wr, 10 op/s 2022-04-22T17:36:59.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:58 smithi115 bash[46934]: cluster 2022-04-22T17:36:57.968424+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2022-04-22T17:36:59.443 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:36:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:36:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 starting - - - - 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (13m) 40s ago 13m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2m) 65s ago 14m 12.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (93s) 40s ago 14m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (9m) 40s ago 19m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (9m) 65s ago 21m 452M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (8m) 65s ago 21m 82.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (8m) 40s ago 19m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 65s ago 19m 66.3M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (69s) 65s ago 14m - - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (65s) 40s ago 14m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:36:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6m) 65s ago 18m 412M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6m) 65s ago 18m 333M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5m) 65s ago 17m 301M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5m) 65s ago 17m 188M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4m) 40s ago 16m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3m) 40s ago 16m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3m) 40s ago 15m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2m) 40s ago 15m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (45s) 40s ago 14m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:36:59.975 INFO:tasks.rados.rados.0.smithi115.stdout:3436: done (1 left) 2022-04-22T17:36:59.975 INFO:tasks.rados.rados.0.smithi115.stdout:3437: snap_remove snap 319 2022-04-22T17:37:00.220 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:37:00.220 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:37:00.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:37:00.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:00.220 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:37:00.221 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:37:00.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:00.222 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:37:00.222 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:37:00.222 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:37:00.223 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:37:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:59 smithi121 bash[36902]: audit 2022-04-22T17:36:59.410212+0000 mon.a (mon.0) 1362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:00.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:36:59 smithi121 bash[36902]: cluster 2022-04-22T17:36:59.955435+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2022-04-22T17:37:00.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:59 smithi115 bash[45495]: audit 2022-04-22T17:36:59.410212+0000 mon.a (mon.0) 1362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:00.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:36:59 smithi115 bash[45495]: cluster 2022-04-22T17:36:59.955435+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2022-04-22T17:37:00.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:59 smithi115 bash[46934]: audit 2022-04-22T17:36:59.410212+0000 mon.a (mon.0) 1362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:00.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:36:59 smithi115 bash[46934]: cluster 2022-04-22T17:36:59.955435+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2022-04-22T17:37:00.975 INFO:tasks.rados.rados.0.smithi115.stdout:3437: done (1 left) 2022-04-22T17:37:00.975 INFO:tasks.rados.rados.0.smithi115.stdout:3438: rollback oid 46 current snap is 324 2022-04-22T17:37:00.975 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 46 to 323 2022-04-22T17:37:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:3439: copy_from oid 37 from oid 50 current snap is 324 2022-04-22T17:37:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:3440: read oid 28 snap 322 2022-04-22T17:37:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:3440: expect (ObjNum 1019 snap 321 seq_num 1019) 2022-04-22T17:37:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:3441: copy_from oid 5 from oid 15 current snap is 324 2022-04-22T17:37:00.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 324 2022-04-22T17:37:00.977 INFO:tasks.rados.rados.0.smithi115.stdout:3442: seq_num 1040 ranges {0=622592} 2022-04-22T17:37:00.978 INFO:tasks.rados.rados.0.smithi115.stdout:3442: writing smithi11541303-14 from 0 to 622592 tid 1 2022-04-22T17:37:00.978 INFO:tasks.rados.rados.0.smithi115.stdout:3443: snap_create 2022-04-22T17:37:00.978 INFO:tasks.rados.rados.0.smithi115.stdout:3435: finishing copy_from to smithi11541303-47 2022-04-22T17:37:00.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2036 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:37:00.979 INFO:tasks.rados.rados.0.smithi115.stdout:3435: finishing copy_from racing read to smithi11541303-47 2022-04-22T17:37:00.980 INFO:tasks.rados.rados.0.smithi115.stdout:3439: finishing copy_from to smithi11541303-37 2022-04-22T17:37:00.980 INFO:tasks.rados.rados.0.smithi115.stdout:3439: got expected ENOENT (src dne) 2022-04-22T17:37:00.995 INFO:tasks.rados.rados.0.smithi115.stdout:3439: finishing copy_from racing read to smithi11541303-37 2022-04-22T17:37:00.995 INFO:tasks.rados.rados.0.smithi115.stdout:3438: finishing rollback tid 0 to smithi11541303-46 2022-04-22T17:37:00.996 INFO:tasks.rados.rados.0.smithi115.stdout:3438: finishing rollback tid 1 to smithi11541303-46 2022-04-22T17:37:00.996 INFO:tasks.rados.rados.0.smithi115.stdout:3438: finishing rollback tid 2 to smithi11541303-46 2022-04-22T17:37:00.996 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 2062 (ObjNum 1032 snap 323 seq_num 1032) dirty exists 2022-04-22T17:37:01.014 INFO:tasks.rados.rados.0.smithi115.stdout:3441: finishing copy_from to smithi11541303-5 2022-04-22T17:37:01.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2051 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:37:01.017 INFO:tasks.rados.rados.0.smithi115.stdout:3441: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:37:01.019 INFO:tasks.rados.rados.0.smithi115.stdout:3442: finishing write tid 1 to smithi11541303-14 2022-04-22T17:37:01.019 INFO:tasks.rados.rados.0.smithi115.stdout:3442: finishing write tid 2 to smithi11541303-14 2022-04-22T17:37:01.024 INFO:tasks.rados.rados.0.smithi115.stdout:3440: expect (ObjNum 1019 snap 321 seq_num 1019) 2022-04-22T17:37:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: cluster 2022-04-22T17:36:59.344763+0000 mgr.y (mgr.34107) 421 : cluster [DBG] pgmap v766: 121 pgs: 121 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 971 KiB/s wr, 10 op/s 2022-04-22T17:37:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: audit 2022-04-22T17:36:59.415132+0000 mgr.y (mgr.34107) 422 : audit [DBG] from='client.44425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: audit 2022-04-22T17:36:59.755307+0000 mgr.y (mgr.34107) 423 : audit [DBG] from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: audit 2022-04-22T17:37:00.207181+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.115:0/893202841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:01.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: audit 2022-04-22T17:37:00.676422+0000 mon.a (mon.0) 1364 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:01.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:00 smithi121 bash[36902]: cluster 2022-04-22T17:37:00.953991+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2022-04-22T17:37:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: cluster 2022-04-22T17:36:59.344763+0000 mgr.y (mgr.34107) 421 : cluster [DBG] pgmap v766: 121 pgs: 121 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 971 KiB/s wr, 10 op/s 2022-04-22T17:37:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: audit 2022-04-22T17:36:59.415132+0000 mgr.y (mgr.34107) 422 : audit [DBG] from='client.44425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: audit 2022-04-22T17:36:59.755307+0000 mgr.y (mgr.34107) 423 : audit [DBG] from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: audit 2022-04-22T17:37:00.207181+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.115:0/893202841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: audit 2022-04-22T17:37:00.676422+0000 mon.a (mon.0) 1364 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[45495]: cluster 2022-04-22T17:37:00.953991+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: cluster 2022-04-22T17:36:59.344763+0000 mgr.y (mgr.34107) 421 : cluster [DBG] pgmap v766: 121 pgs: 121 active+clean; 219 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 971 KiB/s wr, 10 op/s 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: audit 2022-04-22T17:36:59.415132+0000 mgr.y (mgr.34107) 422 : audit [DBG] from='client.44425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: audit 2022-04-22T17:36:59.755307+0000 mgr.y (mgr.34107) 423 : audit [DBG] from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: audit 2022-04-22T17:37:00.207181+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.115:0/893202841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: audit 2022-04-22T17:37:00.676422+0000 mon.a (mon.0) 1364 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:00 smithi115 bash[46934]: cluster 2022-04-22T17:37:00.953991+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2022-04-22T17:37:01.701 INFO:tasks.rados.rados.0.smithi115.stdout:3442: finishing write tid 3 to smithi11541303-14 2022-04-22T17:37:01.701 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 1896 (ObjNum 1040 snap 324 seq_num 1040) dirty exists 2022-04-22T17:37:01.701 INFO:tasks.rados.rados.0.smithi115.stdout:3442: left oid 14 (ObjNum 1040 snap 324 seq_num 1040) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3435: done (6 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3438: done (5 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3439: done (4 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3440: done (3 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3441: done (2 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3442: done (1 left) 2022-04-22T17:37:01.970 INFO:tasks.rados.rados.0.smithi115.stdout:3443: done (0 left) 2022-04-22T17:37:01.971 INFO:tasks.rados.rados.0.smithi115.stdout:3444: read oid 10 snap -1 2022-04-22T17:37:01.971 INFO:tasks.rados.rados.0.smithi115.stdout:3444: expect (ObjNum 951 snap 308 seq_num 951) 2022-04-22T17:37:01.971 INFO:tasks.rados.rados.0.smithi115.stdout:3445: snap_create 2022-04-22T17:37:01.985 INFO:tasks.rados.rados.0.smithi115.stdout:3444: expect (ObjNum 951 snap 308 seq_num 951) 2022-04-22T17:37:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:02 smithi121 bash[36902]: audit 2022-04-22T17:37:01.296916+0000 mon.a (mon.0) 1366 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:02.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:02 smithi121 bash[36902]: cluster 2022-04-22T17:37:01.345693+0000 mgr.y (mgr.34107) 424 : cluster [DBG] pgmap v769: 121 pgs: 121 active+clean; 210 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2022-04-22T17:37:02.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:02 smithi121 bash[36902]: cluster 2022-04-22T17:37:01.957398+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[45495]: audit 2022-04-22T17:37:01.296916+0000 mon.a (mon.0) 1366 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[45495]: cluster 2022-04-22T17:37:01.345693+0000 mgr.y (mgr.34107) 424 : cluster [DBG] pgmap v769: 121 pgs: 121 active+clean; 210 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[45495]: cluster 2022-04-22T17:37:01.957398+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[46934]: audit 2022-04-22T17:37:01.296916+0000 mon.a (mon.0) 1366 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[46934]: cluster 2022-04-22T17:37:01.345693+0000 mgr.y (mgr.34107) 424 : cluster [DBG] pgmap v769: 121 pgs: 121 active+clean; 210 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2022-04-22T17:37:02.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:02 smithi115 bash[46934]: cluster 2022-04-22T17:37:01.957398+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2022-04-22T17:37:02.971 INFO:tasks.rados.rados.0.smithi115.stdout:3444: done (1 left) 2022-04-22T17:37:02.971 INFO:tasks.rados.rados.0.smithi115.stdout:3445: done (0 left) 2022-04-22T17:37:02.972 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 326 2022-04-22T17:37:02.972 INFO:tasks.rados.rados.0.smithi115.stdout:3446: seq_num 1041 ranges {1318912=638976} 2022-04-22T17:37:02.986 INFO:tasks.rados.rados.0.smithi115.stdout:3446: writing smithi11541303-42 from 1318912 to 1957888 tid 1 2022-04-22T17:37:02.987 INFO:tasks.rados.rados.0.smithi115.stdout:3447: setattr oid 12 current snap is 326 2022-04-22T17:37:02.988 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 326 2022-04-22T17:37:02.988 INFO:tasks.rados.rados.0.smithi115.stdout:3448: seq_num 1042 ranges {2433024=655360} 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3448: writing smithi11541303-25 from 2433024 to 3088384 tid 1 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3449: read oid 23 snap -1 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3449: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3450: read oid 16 snap -1 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3450: expect deleted 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3451: delete oid 8 current snap is 326 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1781 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:37:03.008 INFO:tasks.rados.rados.0.smithi115.stdout:3446: finishing write tid 1 to smithi11541303-42 2022-04-22T17:37:03.009 INFO:tasks.rados.rados.0.smithi115.stdout:3446: finishing write tid 2 to smithi11541303-42 2022-04-22T17:37:03.009 INFO:tasks.rados.rados.0.smithi115.stdout:3446: finishing write tid 3 to smithi11541303-42 2022-04-22T17:37:03.009 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2122 (ObjNum 1041 snap 326 seq_num 1041) dirty exists 2022-04-22T17:37:03.009 INFO:tasks.rados.rados.0.smithi115.stdout:3446: left oid 42 (ObjNum 1041 snap 326 seq_num 1041) 2022-04-22T17:37:03.009 INFO:tasks.rados.rados.0.smithi115.stdout:3449: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:03.012 INFO:tasks.rados.rados.0.smithi115.stdout:3446: done (5 left) 2022-04-22T17:37:03.012 INFO:tasks.rados.rados.0.smithi115.stdout:3447: done (4 left) 2022-04-22T17:37:03.012 INFO:tasks.rados.rados.0.smithi115.stdout:3449: done (3 left) 2022-04-22T17:37:03.013 INFO:tasks.rados.rados.0.smithi115.stdout:3450: done (2 left) 2022-04-22T17:37:03.013 INFO:tasks.rados.rados.0.smithi115.stdout:3451: done (1 left) 2022-04-22T17:37:03.013 INFO:tasks.rados.rados.0.smithi115.stdout:3452: copy_from oid 3 from oid 41 current snap is 326 2022-04-22T17:37:03.013 INFO:tasks.rados.rados.0.smithi115.stdout:3453: snap_create 2022-04-22T17:37:03.017 INFO:tasks.rados.rados.0.smithi115.stdout:3452: finishing copy_from to smithi11541303-3 2022-04-22T17:37:03.018 INFO:tasks.rados.rados.0.smithi115.stdout:3452: got expected ENOENT (src dne) 2022-04-22T17:37:03.018 INFO:tasks.rados.rados.0.smithi115.stdout:3452: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:37:03.019 INFO:tasks.rados.rados.0.smithi115.stdout:3448: finishing write tid 1 to smithi11541303-25 2022-04-22T17:37:03.065 INFO:tasks.rados.rados.0.smithi115.stdout:3448: finishing write tid 2 to smithi11541303-25 2022-04-22T17:37:03.065 INFO:tasks.rados.rados.0.smithi115.stdout:3448: finishing write tid 3 to smithi11541303-25 2022-04-22T17:37:03.065 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1909 (ObjNum 1042 snap 326 seq_num 1042) dirty exists 2022-04-22T17:37:03.065 INFO:tasks.rados.rados.0.smithi115.stdout:3448: left oid 25 (ObjNum 1042 snap 326 seq_num 1042) 2022-04-22T17:37:03.985 INFO:tasks.rados.rados.0.smithi115.stdout:3448: done (2 left) 2022-04-22T17:37:03.989 INFO:tasks.rados.rados.0.smithi115.stdout:3452: done (1 left) 2022-04-22T17:37:03.989 INFO:tasks.rados.rados.0.smithi115.stdout:3453: done (0 left) 2022-04-22T17:37:03.990 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 327 2022-04-22T17:37:03.990 INFO:tasks.rados.rados.0.smithi115.stdout:3454: seq_num 1044 ranges {6545408=688128} 2022-04-22T17:37:04.031 INFO:tasks.rados.rados.0.smithi115.stdout:3454: writing smithi11541303-13 from 6545408 to 7233536 tid 1 2022-04-22T17:37:04.031 INFO:tasks.rados.rados.0.smithi115.stdout:3455: rollback oid 30 current snap is 327 2022-04-22T17:37:04.031 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 324 2022-04-22T17:37:04.031 INFO:tasks.rados.rados.0.smithi115.stdout:3456: snap_create 2022-04-22T17:37:04.044 INFO:tasks.rados.rados.0.smithi115.stdout:3455: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:37:04.045 INFO:tasks.rados.rados.0.smithi115.stdout:3455: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:37:04.045 INFO:tasks.rados.rados.0.smithi115.stdout:3454: finishing write tid 1 to smithi11541303-13 2022-04-22T17:37:04.045 INFO:tasks.rados.rados.0.smithi115.stdout:3454: finishing write tid 2 to smithi11541303-13 2022-04-22T17:37:04.045 INFO:tasks.rados.rados.0.smithi115.stdout:3455: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:37:04.045 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2049 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:37:04.046 INFO:tasks.rados.rados.0.smithi115.stdout:3454: finishing write tid 3 to smithi11541303-13 2022-04-22T17:37:04.046 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1911 (ObjNum 1044 snap 327 seq_num 1044) dirty exists 2022-04-22T17:37:04.046 INFO:tasks.rados.rados.0.smithi115.stdout:3454: left oid 13 (ObjNum 1044 snap 327 seq_num 1044) 2022-04-22T17:37:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:03 smithi121 bash[36902]: cluster 2022-04-22T17:37:02.959412+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2022-04-22T17:37:04.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:03 smithi115 bash[45495]: cluster 2022-04-22T17:37:02.959412+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2022-04-22T17:37:04.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:03 smithi115 bash[46934]: cluster 2022-04-22T17:37:02.959412+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2022-04-22T17:37:04.970 INFO:tasks.rados.rados.0.smithi115.stdout:3454: done (2 left) 2022-04-22T17:37:04.970 INFO:tasks.rados.rados.0.smithi115.stdout:3455: done (1 left) 2022-04-22T17:37:04.970 INFO:tasks.rados.rados.0.smithi115.stdout:3456: done (0 left) 2022-04-22T17:37:04.970 INFO:tasks.rados.rados.0.smithi115.stdout:3457: read oid 41 snap -1 2022-04-22T17:37:04.970 INFO:tasks.rados.rados.0.smithi115.stdout:3457: expect deleted 2022-04-22T17:37:04.971 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 328 2022-04-22T17:37:04.971 INFO:tasks.rados.rados.0.smithi115.stdout:3458: seq_num 1045 ranges {0=704512} 2022-04-22T17:37:04.975 INFO:tasks.rados.rados.0.smithi115.stdout:3458: writing smithi11541303-8 from 0 to 704512 tid 1 2022-04-22T17:37:04.975 INFO:tasks.rados.rados.0.smithi115.stdout:3459: snap_create 2022-04-22T17:37:04.984 INFO:tasks.rados.rados.0.smithi115.stdout:3458: finishing write tid 1 to smithi11541303-8 2022-04-22T17:37:04.985 INFO:tasks.rados.rados.0.smithi115.stdout:3458: finishing write tid 2 to smithi11541303-8 2022-04-22T17:37:04.986 INFO:tasks.rados.rados.0.smithi115.stdout:3458: finishing write tid 3 to smithi11541303-8 2022-04-22T17:37:04.986 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1938 (ObjNum 1045 snap 328 seq_num 1045) dirty exists 2022-04-22T17:37:04.987 INFO:tasks.rados.rados.0.smithi115.stdout:3458: left oid 8 (ObjNum 1045 snap 328 seq_num 1045) 2022-04-22T17:37:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:04 smithi121 bash[36902]: cluster 2022-04-22T17:37:03.346745+0000 mgr.y (mgr.34107) 425 : cluster [DBG] pgmap v772: 121 pgs: 121 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2022-04-22T17:37:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:04 smithi121 bash[36902]: cluster 2022-04-22T17:37:03.972955+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2022-04-22T17:37:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:04 smithi121 bash[36902]: audit 2022-04-22T17:37:04.412675+0000 mon.a (mon.0) 1370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:04 smithi121 bash[36902]: cluster 2022-04-22T17:37:04.957354+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2022-04-22T17:37:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[46934]: cluster 2022-04-22T17:37:03.346745+0000 mgr.y (mgr.34107) 425 : cluster [DBG] pgmap v772: 121 pgs: 121 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2022-04-22T17:37:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[46934]: cluster 2022-04-22T17:37:03.972955+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2022-04-22T17:37:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[46934]: audit 2022-04-22T17:37:04.412675+0000 mon.a (mon.0) 1370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[46934]: cluster 2022-04-22T17:37:04.957354+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2022-04-22T17:37:05.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[45495]: cluster 2022-04-22T17:37:03.346745+0000 mgr.y (mgr.34107) 425 : cluster [DBG] pgmap v772: 121 pgs: 121 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2022-04-22T17:37:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[45495]: cluster 2022-04-22T17:37:03.972955+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2022-04-22T17:37:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[45495]: audit 2022-04-22T17:37:04.412675+0000 mon.a (mon.0) 1370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:04 smithi115 bash[45495]: cluster 2022-04-22T17:37:04.957354+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2022-04-22T17:37:05.979 INFO:tasks.rados.rados.0.smithi115.stdout:3457: done (2 left) 2022-04-22T17:37:05.980 INFO:tasks.rados.rados.0.smithi115.stdout:3458: done (1 left) 2022-04-22T17:37:05.980 INFO:tasks.rados.rados.0.smithi115.stdout:3459: done (0 left) 2022-04-22T17:37:05.980 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 4 current snap is 329 2022-04-22T17:37:05.980 INFO:tasks.rados.rados.0.smithi115.stdout:3460: seq_num 1046 ranges {2121728=729088} 2022-04-22T17:37:05.996 INFO:tasks.rados.rados.0.smithi115.stdout:3460: writing smithi11541303-4 from 2121728 to 2850816 tid 1 2022-04-22T17:37:05.996 INFO:tasks.rados.rados.0.smithi115.stdout:3461: copy_from oid 41 from oid 13 current snap is 329 2022-04-22T17:37:05.996 INFO:tasks.rados.rados.0.smithi115.stdout:3462: setattr oid 47 current snap is 329 2022-04-22T17:37:05.998 INFO:tasks.rados.rados.0.smithi115.stdout:3463: delete oid 37 current snap is 329 2022-04-22T17:37:06.007 INFO:tasks.rados.rados.0.smithi115.stdout:3463: done (3 left) 2022-04-22T17:37:06.007 INFO:tasks.rados.rados.0.smithi115.stdout:3464: delete oid 25 current snap is 329 2022-04-22T17:37:06.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2039 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:37:06.011 INFO:tasks.rados.rados.0.smithi115.stdout:3460: finishing write tid 1 to smithi11541303-4 2022-04-22T17:37:06.013 INFO:tasks.rados.rados.0.smithi115.stdout:3460: finishing write tid 2 to smithi11541303-4 2022-04-22T17:37:06.015 INFO:tasks.rados.rados.0.smithi115.stdout:3460: finishing write tid 3 to smithi11541303-4 2022-04-22T17:37:06.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2051 (ObjNum 1046 snap 329 seq_num 1046) dirty exists 2022-04-22T17:37:06.015 INFO:tasks.rados.rados.0.smithi115.stdout:3460: left oid 4 (ObjNum 1046 snap 329 seq_num 1046) 2022-04-22T17:37:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:3460: done (3 left) 2022-04-22T17:37:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:3462: done (2 left) 2022-04-22T17:37:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:3464: done (1 left) 2022-04-22T17:37:06.020 INFO:tasks.rados.rados.0.smithi115.stdout:3465: snap_remove snap 327 2022-04-22T17:37:06.989 INFO:tasks.rados.rados.0.smithi115.stdout:3465: done (1 left) 2022-04-22T17:37:06.990 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 329 2022-04-22T17:37:06.990 INFO:tasks.rados.rados.0.smithi115.stdout:3466: seq_num 1049 ranges {2310144=778240} 2022-04-22T17:37:07.008 INFO:tasks.rados.rados.0.smithi115.stdout:3466: writing smithi11541303-33 from 2310144 to 3088384 tid 1 2022-04-22T17:37:07.008 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 329 2022-04-22T17:37:07.008 INFO:tasks.rados.rados.0.smithi115.stdout:3461: finishing copy_from to smithi11541303-41 2022-04-22T17:37:07.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 1911 (ObjNum 1044 snap 327 seq_num 1044) dirty exists 2022-04-22T17:37:07.008 INFO:tasks.rados.rados.0.smithi115.stdout:3467: seq_num 1050 ranges {3088384=794624} 2022-04-22T17:37:07.030 INFO:tasks.rados.rados.0.smithi115.stdout:3467: writing smithi11541303-31 from 3088384 to 3883008 tid 1 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3468: read oid 37 snap -1 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3468: expect deleted 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3469: read oid 23 snap -1 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3469: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3470: delete oid 10 current snap is 329 2022-04-22T17:37:07.031 INFO:tasks.rados.rados.0.smithi115.stdout:3461: finishing copy_from racing read to smithi11541303-41 2022-04-22T17:37:07.033 INFO:tasks.rados.rados.0.smithi115.stdout:3469: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:07.040 INFO:tasks.rados.rados.0.smithi115.stdout:3466: finishing write tid 1 to smithi11541303-33 2022-04-22T17:37:07.040 INFO:tasks.rados.rados.0.smithi115.stdout:3466: finishing write tid 2 to smithi11541303-33 2022-04-22T17:37:07.042 INFO:tasks.rados.rados.0.smithi115.stdout:3461: done (5 left) 2022-04-22T17:37:07.042 INFO:tasks.rados.rados.0.smithi115.stdout:3468: done (4 left) 2022-04-22T17:37:07.042 INFO:tasks.rados.rados.0.smithi115.stdout:3469: done (3 left) 2022-04-22T17:37:07.042 INFO:tasks.rados.rados.0.smithi115.stdout:3470: done (2 left) 2022-04-22T17:37:07.042 INFO:tasks.rados.rados.0.smithi115.stdout:3471: snap_remove snap 322 2022-04-22T17:37:07.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:06 smithi115 bash[45495]: cluster 2022-04-22T17:37:05.347689+0000 mgr.y (mgr.34107) 426 : cluster [DBG] pgmap v775: 121 pgs: 121 active+clean; 181 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2022-04-22T17:37:07.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:06 smithi115 bash[45495]: cluster 2022-04-22T17:37:05.966803+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2022-04-22T17:37:07.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:06 smithi115 bash[46934]: cluster 2022-04-22T17:37:05.347689+0000 mgr.y (mgr.34107) 426 : cluster [DBG] pgmap v775: 121 pgs: 121 active+clean; 181 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2022-04-22T17:37:07.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:06 smithi115 bash[46934]: cluster 2022-04-22T17:37:05.966803+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2022-04-22T17:37:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:06 smithi121 bash[36902]: cluster 2022-04-22T17:37:05.347689+0000 mgr.y (mgr.34107) 426 : cluster [DBG] pgmap v775: 121 pgs: 121 active+clean; 181 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2022-04-22T17:37:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:06 smithi121 bash[36902]: cluster 2022-04-22T17:37:05.966803+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2022-04-22T17:37:07.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:07 smithi115 bash[60466]: level=info ts=2022-04-22T17:37:07.151Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004323352s 2022-04-22T17:37:08.002 INFO:tasks.rados.rados.0.smithi115.stdout:3467: finishing write tid 1 to smithi11541303-31 2022-04-22T17:37:08.002 INFO:tasks.rados.rados.0.smithi115.stdout:3467: finishing write tid 2 to smithi11541303-31 2022-04-22T17:37:08.002 INFO:tasks.rados.rados.0.smithi115.stdout:3466: finishing write tid 3 to smithi11541303-33 2022-04-22T17:37:08.002 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 2035 (ObjNum 1049 snap 329 seq_num 1049) dirty exists 2022-04-22T17:37:08.007 INFO:tasks.rados.rados.0.smithi115.stdout:3466: left oid 33 (ObjNum 1049 snap 329 seq_num 1049) 2022-04-22T17:37:08.007 INFO:tasks.rados.rados.0.smithi115.stdout:3467: finishing write tid 3 to smithi11541303-31 2022-04-22T17:37:08.009 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2125 (ObjNum 1050 snap 329 seq_num 1050) dirty exists 2022-04-22T17:37:08.009 INFO:tasks.rados.rados.0.smithi115.stdout:3467: left oid 31 (ObjNum 1050 snap 329 seq_num 1050) 2022-04-22T17:37:08.009 INFO:tasks.rados.rados.0.smithi115.stdout:3466: done (2 left) 2022-04-22T17:37:08.009 INFO:tasks.rados.rados.0.smithi115.stdout:3467: done (1 left) 2022-04-22T17:37:08.009 INFO:tasks.rados.rados.0.smithi115.stdout:3471: done (0 left) 2022-04-22T17:37:08.010 INFO:tasks.rados.rados.0.smithi115.stdout:3472: rollback oid 48 current snap is 329 2022-04-22T17:37:08.010 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 48 to 325 2022-04-22T17:37:08.010 INFO:tasks.rados.rados.0.smithi115.stdout:3473: copy_from oid 44 from oid 42 current snap is 329 2022-04-22T17:37:08.010 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 329 2022-04-22T17:37:08.010 INFO:tasks.rados.rados.0.smithi115.stdout:3474: seq_num 1052 ranges {7233536=425984,7659520=401408} 2022-04-22T17:37:08.049 INFO:tasks.rados.rados.0.smithi115.stdout:3474: writing smithi11541303-41 from 7233536 to 7659520 tid 1 2022-04-22T17:37:08.090 INFO:tasks.rados.rados.0.smithi115.stdout:3474: writing smithi11541303-41 from 7659520 to 8060928 tid 2 2022-04-22T17:37:08.090 INFO:tasks.rados.rados.0.smithi115.stdout:3475: read oid 1 snap -1 2022-04-22T17:37:08.090 INFO:tasks.rados.rados.0.smithi115.stdout:3475: expect deleted 2022-04-22T17:37:08.091 INFO:tasks.rados.rados.0.smithi115.stdout:3476: snap_remove snap 323 2022-04-22T17:37:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:07 smithi121 bash[36902]: cluster 2022-04-22T17:37:06.976155+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2022-04-22T17:37:08.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:07 smithi115 bash[46934]: cluster 2022-04-22T17:37:06.976155+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2022-04-22T17:37:08.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:07 smithi115 bash[45495]: cluster 2022-04-22T17:37:06.976155+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3476: done (4 left) 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3477: snap_create 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3472: finishing rollback tid 0 to smithi11541303-48 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3472: finishing rollback tid 1 to smithi11541303-48 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3472: finishing rollback tid 2 to smithi11541303-48 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2054 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:37:09.007 INFO:tasks.rados.rados.0.smithi115.stdout:3473: finishing copy_from to smithi11541303-44 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2122 (ObjNum 1041 snap 326 seq_num 1041) dirty exists 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3473: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3474: finishing write tid 1 to smithi11541303-41 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3474: finishing write tid 2 to smithi11541303-41 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3474: finishing write tid 3 to smithi11541303-41 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3474: finishing write tid 4 to smithi11541303-41 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 2043 (ObjNum 1052 snap 329 seq_num 1052) dirty exists 2022-04-22T17:37:09.008 INFO:tasks.rados.rados.0.smithi115.stdout:3474: left oid 41 (ObjNum 1052 snap 329 seq_num 1052) 2022-04-22T17:37:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:08 smithi121 bash[36902]: cluster 2022-04-22T17:37:07.348573+0000 mgr.y (mgr.34107) 427 : cluster [DBG] pgmap v778: 121 pgs: 121 active+clean; 182 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 328 KiB/s wr, 9 op/s 2022-04-22T17:37:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:08 smithi121 bash[36902]: cluster 2022-04-22T17:37:07.988265+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2022-04-22T17:37:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:08 smithi115 bash[45495]: cluster 2022-04-22T17:37:07.348573+0000 mgr.y (mgr.34107) 427 : cluster [DBG] pgmap v778: 121 pgs: 121 active+clean; 182 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 328 KiB/s wr, 9 op/s 2022-04-22T17:37:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:08 smithi115 bash[45495]: cluster 2022-04-22T17:37:07.988265+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2022-04-22T17:37:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:08 smithi115 bash[46934]: cluster 2022-04-22T17:37:07.348573+0000 mgr.y (mgr.34107) 427 : cluster [DBG] pgmap v778: 121 pgs: 121 active+clean; 182 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 328 KiB/s wr, 9 op/s 2022-04-22T17:37:09.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:08 smithi115 bash[46934]: cluster 2022-04-22T17:37:07.988265+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2022-04-22T17:37:09.971 INFO:tasks.rados.rados.0.smithi115.stdout:3472: done (4 left) 2022-04-22T17:37:09.972 INFO:tasks.rados.rados.0.smithi115.stdout:3473: done (3 left) 2022-04-22T17:37:09.972 INFO:tasks.rados.rados.0.smithi115.stdout:3474: done (2 left) 2022-04-22T17:37:09.972 INFO:tasks.rados.rados.0.smithi115.stdout:3475: done (1 left) 2022-04-22T17:37:09.972 INFO:tasks.rados.rados.0.smithi115.stdout:3477: done (0 left) 2022-04-22T17:37:09.972 INFO:tasks.rados.rados.0.smithi115.stdout:3478: snap_remove snap 328 2022-04-22T17:37:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:10 smithi121 bash[36902]: cluster 2022-04-22T17:37:08.992894+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2022-04-22T17:37:10.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:10 smithi121 bash[36902]: audit 2022-04-22T17:37:09.414845+0000 mon.a (mon.0) 1376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:10.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:10 smithi121 bash[36902]: cluster 2022-04-22T17:37:09.959217+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2022-04-22T17:37:10.283 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[45495]: cluster 2022-04-22T17:37:08.992894+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2022-04-22T17:37:10.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[45495]: audit 2022-04-22T17:37:09.414845+0000 mon.a (mon.0) 1376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:10.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[45495]: cluster 2022-04-22T17:37:09.959217+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2022-04-22T17:37:10.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[46934]: cluster 2022-04-22T17:37:08.992894+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2022-04-22T17:37:10.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[46934]: audit 2022-04-22T17:37:09.414845+0000 mon.a (mon.0) 1376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:10.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:09 smithi115 bash[46934]: cluster 2022-04-22T17:37:09.959217+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2022-04-22T17:37:10.976 INFO:tasks.rados.rados.0.smithi115.stdout:3478: done (0 left) 2022-04-22T17:37:11.020 INFO:tasks.rados.rados.0.smithi115.stdout:3479: snap_create 2022-04-22T17:37:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: cluster 2022-04-22T17:37:09.349256+0000 mgr.y (mgr.34107) 428 : cluster [DBG] pgmap v781: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 656 KiB/s wr, 8 op/s 2022-04-22T17:37:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.289977+0000 mon.a (mon.0) 1378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.296311+0000 mon.a (mon.0) 1379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.307333+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.310052+0000 mon.a (mon.0) 1381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.318536+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.325161+0000 mon.a (mon.0) 1383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.329374+0000 mon.a (mon.0) 1384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.332023+0000 mon.a (mon.0) 1385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.334118+0000 mon.a (mon.0) 1386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.337033+0000 mon.a (mon.0) 1387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.339993+0000 mon.a (mon.0) 1388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.341754+0000 mon.a (mon.0) 1389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.343277+0000 mon.a (mon.0) 1390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.344851+0000 mon.a (mon.0) 1391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.346584+0000 mon.a (mon.0) 1392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.348010+0000 mon.a (mon.0) 1393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.349442+0000 mon.a (mon.0) 1394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.350916+0000 mon.a (mon.0) 1395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: audit 2022-04-22T17:37:10.352365+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:11 smithi121 bash[36902]: cluster 2022-04-22T17:37:10.959458+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:10 smithi115 bash[45495]: cluster 2022-04-22T17:37:09.349256+0000 mgr.y (mgr.34107) 428 : cluster [DBG] pgmap v781: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 656 KiB/s wr, 8 op/s 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:10 smithi115 bash[45495]: audit 2022-04-22T17:37:10.289977+0000 mon.a (mon.0) 1378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.296311+0000 mon.a (mon.0) 1379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.307333+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.310052+0000 mon.a (mon.0) 1381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.318536+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.325161+0000 mon.a (mon.0) 1383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:11.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.329374+0000 mon.a (mon.0) 1384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.332023+0000 mon.a (mon.0) 1385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.334118+0000 mon.a (mon.0) 1386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.337033+0000 mon.a (mon.0) 1387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.339993+0000 mon.a (mon.0) 1388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.341754+0000 mon.a (mon.0) 1389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.343277+0000 mon.a (mon.0) 1390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.344851+0000 mon.a (mon.0) 1391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.346584+0000 mon.a (mon.0) 1392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.348010+0000 mon.a (mon.0) 1393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.349442+0000 mon.a (mon.0) 1394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.350916+0000 mon.a (mon.0) 1395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: audit 2022-04-22T17:37:10.352365+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[45495]: cluster 2022-04-22T17:37:10.959458+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: cluster 2022-04-22T17:37:09.349256+0000 mgr.y (mgr.34107) 428 : cluster [DBG] pgmap v781: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 656 KiB/s wr, 8 op/s 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.289977+0000 mon.a (mon.0) 1378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.296311+0000 mon.a (mon.0) 1379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.307333+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.310052+0000 mon.a (mon.0) 1381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.318536+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.325161+0000 mon.a (mon.0) 1383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.329374+0000 mon.a (mon.0) 1384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.332023+0000 mon.a (mon.0) 1385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.334118+0000 mon.a (mon.0) 1386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.337033+0000 mon.a (mon.0) 1387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.339993+0000 mon.a (mon.0) 1388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.341754+0000 mon.a (mon.0) 1389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.343277+0000 mon.a (mon.0) 1390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.344851+0000 mon.a (mon.0) 1391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.346584+0000 mon.a (mon.0) 1392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.348010+0000 mon.a (mon.0) 1393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.349442+0000 mon.a (mon.0) 1394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.350916+0000 mon.a (mon.0) 1395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: audit 2022-04-22T17:37:10.352365+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:11.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:11 smithi115 bash[46934]: cluster 2022-04-22T17:37:10.959458+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2022-04-22T17:37:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:3479: done (0 left) 2022-04-22T17:37:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:3480: read oid 23 snap -1 2022-04-22T17:37:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:3480: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 331 2022-04-22T17:37:11.976 INFO:tasks.rados.rados.0.smithi115.stdout:3481: seq_num 1053 ranges {7233536=442368,7675904=401408} 2022-04-22T17:37:12.025 INFO:tasks.rados.rados.0.smithi115.stdout:3481: writing smithi11541303-13 from 7233536 to 7675904 tid 1 2022-04-22T17:37:12.028 INFO:tasks.rados.rados.0.smithi115.stdout:3481: writing smithi11541303-13 from 7675904 to 8077312 tid 2 2022-04-22T17:37:12.028 INFO:tasks.rados.rados.0.smithi115.stdout:3482: rollback oid 36 current snap is 331 2022-04-22T17:37:12.028 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 330 2022-04-22T17:37:12.028 INFO:tasks.rados.rados.0.smithi115.stdout:3480: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:12.031 INFO:tasks.rados.rados.0.smithi115.stdout:3480: done (2 left) 2022-04-22T17:37:12.031 INFO:tasks.rados.rados.0.smithi115.stdout:3483: snap_create 2022-04-22T17:37:12.036 INFO:tasks.rados.rados.0.smithi115.stdout:3482: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:37:12.036 INFO:tasks.rados.rados.0.smithi115.stdout:3482: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:37:12.036 INFO:tasks.rados.rados.0.smithi115.stdout:3482: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:37:12.036 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1941 (ObjNum 1006 snap 320 seq_num 1006) dirty exists 2022-04-22T17:37:12.040 INFO:tasks.rados.rados.0.smithi115.stdout:3481: finishing write tid 1 to smithi11541303-13 2022-04-22T17:37:12.040 INFO:tasks.rados.rados.0.smithi115.stdout:3481: finishing write tid 2 to smithi11541303-13 2022-04-22T17:37:12.040 INFO:tasks.rados.rados.0.smithi115.stdout:3481: finishing write tid 3 to smithi11541303-13 2022-04-22T17:37:12.040 INFO:tasks.rados.rados.0.smithi115.stdout:3481: finishing write tid 4 to smithi11541303-13 2022-04-22T17:37:12.040 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1915 (ObjNum 1053 snap 331 seq_num 1053) dirty exists 2022-04-22T17:37:12.041 INFO:tasks.rados.rados.0.smithi115.stdout:3481: left oid 13 (ObjNum 1053 snap 331 seq_num 1053) 2022-04-22T17:37:12.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[45495]: cephadm 2022-04-22T17:37:10.278561+0000 mgr.y (mgr.34107) 429 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:37:12.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[45495]: audit 2022-04-22T17:37:10.308111+0000 mgr.y (mgr.34107) 430 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:12.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[45495]: audit 2022-04-22T17:37:10.310734+0000 mgr.y (mgr.34107) 431 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:12.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[45495]: cluster 2022-04-22T17:37:11.961849+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2022-04-22T17:37:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[46934]: cephadm 2022-04-22T17:37:10.278561+0000 mgr.y (mgr.34107) 429 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:37:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[46934]: audit 2022-04-22T17:37:10.308111+0000 mgr.y (mgr.34107) 430 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[46934]: audit 2022-04-22T17:37:10.310734+0000 mgr.y (mgr.34107) 431 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:12.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:12 smithi115 bash[46934]: cluster 2022-04-22T17:37:11.961849+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2022-04-22T17:37:12.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:12 smithi121 bash[36902]: cephadm 2022-04-22T17:37:10.278561+0000 mgr.y (mgr.34107) 429 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T17:37:12.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:12 smithi121 bash[36902]: audit 2022-04-22T17:37:10.308111+0000 mgr.y (mgr.34107) 430 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-04-22T17:37:12.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:12 smithi121 bash[36902]: audit 2022-04-22T17:37:10.310734+0000 mgr.y (mgr.34107) 431 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2022-04-22T17:37:12.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:12 smithi121 bash[36902]: cluster 2022-04-22T17:37:11.961849+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2022-04-22T17:37:13.031 INFO:tasks.rados.rados.0.smithi115.stdout:3481: done (2 left) 2022-04-22T17:37:13.097 INFO:tasks.rados.rados.0.smithi115.stdout:3482: done (1 left) 2022-04-22T17:37:13.098 INFO:tasks.rados.rados.0.smithi115.stdout:3483: done (0 left) 2022-04-22T17:37:13.098 INFO:tasks.rados.rados.0.smithi115.stdout:3484: copy_from oid 49 from oid 25 current snap is 332 2022-04-22T17:37:13.098 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 332 2022-04-22T17:37:13.098 INFO:tasks.rados.rados.0.smithi115.stdout:3485: seq_num 1054 ranges {0=458752,458752=401408} 2022-04-22T17:37:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:3485: writing smithi11541303-10 from 0 to 458752 tid 1 2022-04-22T17:37:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:3485: writing smithi11541303-10 from 458752 to 860160 tid 2 2022-04-22T17:37:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:3486: snap_remove snap 331 2022-04-22T17:37:13.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:13 smithi121 bash[36902]: cluster 2022-04-22T17:37:11.350074+0000 mgr.y (mgr.34107) 432 : cluster [DBG] pgmap v784: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.6 MiB/s wr, 11 op/s 2022-04-22T17:37:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:13 smithi115 bash[45495]: cluster 2022-04-22T17:37:11.350074+0000 mgr.y (mgr.34107) 432 : cluster [DBG] pgmap v784: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.6 MiB/s wr, 11 op/s 2022-04-22T17:37:13.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:13 smithi115 bash[46934]: cluster 2022-04-22T17:37:11.350074+0000 mgr.y (mgr.34107) 432 : cluster [DBG] pgmap v784: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 183 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.6 MiB/s wr, 11 op/s 2022-04-22T17:37:14.033 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:13.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3486: done (2 left) 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3487: snap_create 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3484: finishing copy_from to smithi11541303-49 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3484: got expected ENOENT (src dne) 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3484: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3485: finishing write tid 1 to smithi11541303-10 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3485: finishing write tid 2 to smithi11541303-10 2022-04-22T17:37:14.050 INFO:tasks.rados.rados.0.smithi115.stdout:3485: finishing write tid 3 to smithi11541303-10 2022-04-22T17:37:14.051 INFO:tasks.rados.rados.0.smithi115.stdout:3485: finishing write tid 4 to smithi11541303-10 2022-04-22T17:37:14.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2046 (ObjNum 1054 snap 332 seq_num 1054) dirty exists 2022-04-22T17:37:14.051 INFO:tasks.rados.rados.0.smithi115.stdout:3485: left oid 10 (ObjNum 1054 snap 332 seq_num 1054) 2022-04-22T17:37:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[45495]: cluster 2022-04-22T17:37:13.016741+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2022-04-22T17:37:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[45495]: audit 2022-04-22T17:37:13.321883+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:37:14.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[45495]: audit 2022-04-22T17:37:13.322364+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:37:14.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[46934]: cluster 2022-04-22T17:37:13.016741+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2022-04-22T17:37:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[46934]: audit 2022-04-22T17:37:13.321883+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:37:14.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:14 smithi115 bash[46934]: audit 2022-04-22T17:37:13.322364+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:37:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:14 smithi121 bash[36902]: cluster 2022-04-22T17:37:13.016741+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2022-04-22T17:37:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:14 smithi121 bash[36902]: audit 2022-04-22T17:37:13.321883+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:37:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:14 smithi121 bash[36902]: audit 2022-04-22T17:37:13.322364+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:37:14.971 INFO:tasks.rados.rados.0.smithi115.stdout:3484: done (2 left) 2022-04-22T17:37:14.971 INFO:tasks.rados.rados.0.smithi115.stdout:3485: done (1 left) 2022-04-22T17:37:14.972 INFO:tasks.rados.rados.0.smithi115.stdout:3487: done (0 left) 2022-04-22T17:37:14.972 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 333 2022-04-22T17:37:14.972 INFO:tasks.rados.rados.0.smithi115.stdout:3488: seq_num 1055 ranges {4390912=475136,4866048=401408} 2022-04-22T17:37:15.005 INFO:tasks.rados.rados.0.smithi115.stdout:3488: writing smithi11541303-45 from 4390912 to 4866048 tid 1 2022-04-22T17:37:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:3488: writing smithi11541303-45 from 4866048 to 5267456 tid 2 2022-04-22T17:37:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:3489: rollback oid 38 current snap is 333 2022-04-22T17:37:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 332 2022-04-22T17:37:15.007 INFO:tasks.rados.rados.0.smithi115.stdout:3490: rollback oid 1 current snap is 333 2022-04-22T17:37:15.008 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 1 to 329 2022-04-22T17:37:15.008 INFO:tasks.rados.rados.0.smithi115.stdout:3491: setattr oid 22 current snap is 333 2022-04-22T17:37:15.009 INFO:tasks.rados.rados.0.smithi115.stdout:3492: snap_create 2022-04-22T17:37:15.012 INFO:tasks.rados.rados.0.smithi115.stdout:3490: finishing rollback tid 1 to smithi11541303-1 2022-04-22T17:37:15.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 0 (ObjNum -1116689792 snap 22075 seq_num 0) dirty dne 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:3488: finishing write tid 1 to smithi11541303-45 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1858 (ObjNum 998 snap 316 seq_num 998) dirty exists 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:3488: finishing write tid 2 to smithi11541303-45 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:3488: finishing write tid 3 to smithi11541303-45 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:3489: finishing rollback tid 0 to smithi11541303-38 2022-04-22T17:37:15.026 INFO:tasks.rados.rados.0.smithi115.stdout:3489: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:37:15.027 INFO:tasks.rados.rados.0.smithi115.stdout:3489: finishing rollback tid 2 to smithi11541303-38 2022-04-22T17:37:15.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1784 (ObjNum 1024 snap 321 seq_num 1024) dirty exists 2022-04-22T17:37:15.027 INFO:tasks.rados.rados.0.smithi115.stdout:3488: finishing write tid 4 to smithi11541303-45 2022-04-22T17:37:15.027 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2057 (ObjNum 1055 snap 333 seq_num 1055) dirty exists 2022-04-22T17:37:15.027 INFO:tasks.rados.rados.0.smithi115.stdout:3488: left oid 45 (ObjNum 1055 snap 333 seq_num 1055) 2022-04-22T17:37:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: cephadm 2022-04-22T17:37:13.179946+0000 mgr.y (mgr.34107) 433 : cephadm [INF] Upgrade: Updating grafana.a 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: cephadm 2022-04-22T17:37:13.197084+0000 mgr.y (mgr.34107) 434 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: cluster 2022-04-22T17:37:13.350938+0000 mgr.y (mgr.34107) 435 : cluster [DBG] pgmap v787: 121 pgs: 121 active+clean; 167 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: cluster 2022-04-22T17:37:14.033510+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: audit 2022-04-22T17:37:14.417191+0000 mon.a (mon.0) 1403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[46934]: cluster 2022-04-22T17:37:14.958553+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: cephadm 2022-04-22T17:37:13.179946+0000 mgr.y (mgr.34107) 433 : cephadm [INF] Upgrade: Updating grafana.a 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: cephadm 2022-04-22T17:37:13.197084+0000 mgr.y (mgr.34107) 434 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:37:15.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: cluster 2022-04-22T17:37:13.350938+0000 mgr.y (mgr.34107) 435 : cluster [DBG] pgmap v787: 121 pgs: 121 active+clean; 167 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2022-04-22T17:37:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: cluster 2022-04-22T17:37:14.033510+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2022-04-22T17:37:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: audit 2022-04-22T17:37:14.417191+0000 mon.a (mon.0) 1403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:15.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:15 smithi115 bash[45495]: cluster 2022-04-22T17:37:14.958553+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2022-04-22T17:37:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: cephadm 2022-04-22T17:37:13.179946+0000 mgr.y (mgr.34107) 433 : cephadm [INF] Upgrade: Updating grafana.a 2022-04-22T17:37:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: cephadm 2022-04-22T17:37:13.197084+0000 mgr.y (mgr.34107) 434 : cephadm [INF] Deploying daemon grafana.a on smithi121 2022-04-22T17:37:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: cluster 2022-04-22T17:37:13.350938+0000 mgr.y (mgr.34107) 435 : cluster [DBG] pgmap v787: 121 pgs: 121 active+clean; 167 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2022-04-22T17:37:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: cluster 2022-04-22T17:37:14.033510+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2022-04-22T17:37:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: audit 2022-04-22T17:37:14.417191+0000 mon.a (mon.0) 1403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:15.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:15 smithi121 bash[36902]: cluster 2022-04-22T17:37:14.958553+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2022-04-22T17:37:15.984 INFO:tasks.rados.rados.0.smithi115.stdout:3488: done (4 left) 2022-04-22T17:37:15.984 INFO:tasks.rados.rados.0.smithi115.stdout:3489: done (3 left) 2022-04-22T17:37:15.984 INFO:tasks.rados.rados.0.smithi115.stdout:3490: done (2 left) 2022-04-22T17:37:15.985 INFO:tasks.rados.rados.0.smithi115.stdout:3491: done (1 left) 2022-04-22T17:37:15.985 INFO:tasks.rados.rados.0.smithi115.stdout:3492: done (0 left) 2022-04-22T17:37:15.985 INFO:tasks.rados.rados.0.smithi115.stdout:3493: rollback oid 29 current snap is 334 2022-04-22T17:37:15.985 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 326 2022-04-22T17:37:15.985 INFO:tasks.rados.rados.0.smithi115.stdout:3494: snap_create 2022-04-22T17:37:15.993 INFO:tasks.rados.rados.0.smithi115.stdout:3493: finishing rollback tid 0 to smithi11541303-29 2022-04-22T17:37:15.993 INFO:tasks.rados.rados.0.smithi115.stdout:3493: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:37:15.994 INFO:tasks.rados.rados.0.smithi115.stdout:3493: finishing rollback tid 2 to smithi11541303-29 2022-04-22T17:37:15.994 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1899 (ObjNum 996 snap 315 seq_num 996) dirty exists 2022-04-22T17:37:16.989 INFO:tasks.rados.rados.0.smithi115.stdout:3493: done (1 left) 2022-04-22T17:37:16.989 INFO:tasks.rados.rados.0.smithi115.stdout:3494: done (0 left) 2022-04-22T17:37:16.989 INFO:tasks.rados.rados.0.smithi115.stdout:3495: delete oid 19 current snap is 335 2022-04-22T17:37:16.996 INFO:tasks.rados.rados.0.smithi115.stdout:3495: done (0 left) 2022-04-22T17:37:16.996 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 335 2022-04-22T17:37:16.996 INFO:tasks.rados.rados.0.smithi115.stdout:3496: seq_num 1057 ranges {1957888=507904,2465792=401408} 2022-04-22T17:37:17.016 INFO:tasks.rados.rados.0.smithi115.stdout:3496: writing smithi11541303-44 from 1957888 to 2465792 tid 1 2022-04-22T17:37:17.016 INFO:tasks.rados.rados.0.smithi115.stdout:3496: writing smithi11541303-44 from 2465792 to 2867200 tid 2 2022-04-22T17:37:17.017 INFO:tasks.rados.rados.0.smithi115.stdout:3497: snap_create 2022-04-22T17:37:17.020 INFO:tasks.rados.rados.0.smithi115.stdout:3496: finishing write tid 1 to smithi11541303-44 2022-04-22T17:37:17.024 INFO:tasks.rados.rados.0.smithi115.stdout:3496: finishing write tid 2 to smithi11541303-44 2022-04-22T17:37:17.025 INFO:tasks.rados.rados.0.smithi115.stdout:3496: finishing write tid 3 to smithi11541303-44 2022-04-22T17:37:17.026 INFO:tasks.rados.rados.0.smithi115.stdout:3496: finishing write tid 4 to smithi11541303-44 2022-04-22T17:37:17.026 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2125 (ObjNum 1057 snap 335 seq_num 1057) dirty exists 2022-04-22T17:37:17.026 INFO:tasks.rados.rados.0.smithi115.stdout:3496: left oid 44 (ObjNum 1057 snap 335 seq_num 1057) 2022-04-22T17:37:17.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:16 smithi121 bash[36902]: cluster 2022-04-22T17:37:15.351847+0000 mgr.y (mgr.34107) 436 : cluster [DBG] pgmap v790: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 5 op/s 2022-04-22T17:37:17.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:16 smithi121 bash[36902]: cluster 2022-04-22T17:37:15.970837+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2022-04-22T17:37:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:16 smithi115 bash[46934]: cluster 2022-04-22T17:37:15.351847+0000 mgr.y (mgr.34107) 436 : cluster [DBG] pgmap v790: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 5 op/s 2022-04-22T17:37:17.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:16 smithi115 bash[46934]: cluster 2022-04-22T17:37:15.970837+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2022-04-22T17:37:17.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:16 smithi115 bash[45495]: cluster 2022-04-22T17:37:15.351847+0000 mgr.y (mgr.34107) 436 : cluster [DBG] pgmap v790: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 5 op/s 2022-04-22T17:37:17.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:16 smithi115 bash[45495]: cluster 2022-04-22T17:37:15.970837+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2022-04-22T17:37:18.006 INFO:tasks.rados.rados.0.smithi115.stdout:3496: done (1 left) 2022-04-22T17:37:18.006 INFO:tasks.rados.rados.0.smithi115.stdout:3497: done (0 left) 2022-04-22T17:37:18.006 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 336 2022-04-22T17:37:18.006 INFO:tasks.rados.rados.0.smithi115.stdout:3498: seq_num 1058 ranges {0=524288,524288=401408} 2022-04-22T17:37:18.010 INFO:tasks.rados.rados.0.smithi115.stdout:3498: writing smithi11541303-23 from 0 to 524288 tid 1 2022-04-22T17:37:18.010 INFO:tasks.rados.rados.0.smithi115.stdout:3498: writing smithi11541303-23 from 524288 to 925696 tid 2 2022-04-22T17:37:18.010 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 336 2022-04-22T17:37:18.010 INFO:tasks.rados.rados.0.smithi115.stdout:3499: seq_num 1059 ranges {2179072=540672,2719744=401408} 2022-04-22T17:37:18.025 INFO:tasks.rados.rados.0.smithi115.stdout:3499: writing smithi11541303-3 from 2179072 to 2719744 tid 1 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3499: writing smithi11541303-3 from 2719744 to 3121152 tid 2 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3498: finishing write tid 1 to smithi11541303-23 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3500: read oid 34 snap -1 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3500: expect deleted 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3501: rollback oid 19 current snap is 336 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 329 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3498: finishing write tid 2 to smithi11541303-23 2022-04-22T17:37:18.028 INFO:tasks.rados.rados.0.smithi115.stdout:3498: finishing write tid 3 to smithi11541303-23 2022-04-22T17:37:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:3498: finishing write tid 4 to smithi11541303-23 2022-04-22T17:37:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2065 (ObjNum 1058 snap 336 seq_num 1058) dirty exists 2022-04-22T17:37:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:3498: left oid 23 (ObjNum 1058 snap 336 seq_num 1058) 2022-04-22T17:37:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:3498: done (3 left) 2022-04-22T17:37:18.029 INFO:tasks.rados.rados.0.smithi115.stdout:3502: delete oid 5 current snap is 336 2022-04-22T17:37:18.033 INFO:tasks.rados.rados.0.smithi115.stdout:3499: finishing write tid 1 to smithi11541303-3 2022-04-22T17:37:18.034 INFO:tasks.rados.rados.0.smithi115.stdout:3499: finishing write tid 2 to smithi11541303-3 2022-04-22T17:37:18.035 INFO:tasks.rados.rados.0.smithi115.stdout:3499: finishing write tid 3 to smithi11541303-3 2022-04-22T17:37:18.038 INFO:tasks.rados.rados.0.smithi115.stdout:3500: done (3 left) 2022-04-22T17:37:18.038 INFO:tasks.rados.rados.0.smithi115.stdout:3502: done (2 left) 2022-04-22T17:37:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:3503: rollback oid 10 current snap is 336 2022-04-22T17:37:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 334 2022-04-22T17:37:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:3501: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:37:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:3499: finishing write tid 4 to smithi11541303-3 2022-04-22T17:37:18.039 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2068 (ObjNum 1059 snap 336 seq_num 1059) dirty exists 2022-04-22T17:37:18.040 INFO:tasks.rados.rados.0.smithi115.stdout:3499: left oid 3 (ObjNum 1059 snap 336 seq_num 1059) 2022-04-22T17:37:18.040 INFO:tasks.rados.rados.0.smithi115.stdout:3499: done (2 left) 2022-04-22T17:37:18.040 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 336 2022-04-22T17:37:18.041 INFO:tasks.rados.rados.0.smithi115.stdout:3504: seq_num 1061 ranges {9682944=573440,10256384=401408} 2022-04-22T17:37:18.096 INFO:tasks.rados.rados.0.smithi115.stdout:3504: writing smithi11541303-26 from 9682944 to 10256384 tid 1 2022-04-22T17:37:18.099 INFO:tasks.rados.rados.0.smithi115.stdout:3504: writing smithi11541303-26 from 10256384 to 10657792 tid 2 2022-04-22T17:37:18.102 INFO:tasks.rados.rados.0.smithi115.stdout:3505: rollback oid 30 current snap is 336 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 334 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:3501: finishing rollback tid 2 to smithi11541303-19 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1787 (ObjNum 873 snap 286 seq_num 873) dirty exists 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:3501: done (3 left) 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 336 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:3503: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:3503: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:37:18.103 INFO:tasks.rados.rados.0.smithi115.stdout:3503: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:37:18.104 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2049 (ObjNum 1054 snap 332 seq_num 1054) dirty exists 2022-04-22T17:37:18.104 INFO:tasks.rados.rados.0.smithi115.stdout:3506: seq_num 1062 ranges {7053312=589824,7643136=401408} 2022-04-22T17:37:18.144 INFO:tasks.rados.rados.0.smithi115.stdout:3506: writing smithi11541303-22 from 7053312 to 7643136 tid 1 2022-04-22T17:37:18.146 INFO:tasks.rados.rados.0.smithi115.stdout:3506: writing smithi11541303-22 from 7643136 to 8044544 tid 2 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3503: done (3 left) 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3507: snap_create 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3504: finishing write tid 1 to smithi11541303-26 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3505: finishing rollback tid 0 to smithi11541303-30 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3504: finishing write tid 2 to smithi11541303-26 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3505: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:37:18.147 INFO:tasks.rados.rados.0.smithi115.stdout:3504: finishing write tid 3 to smithi11541303-26 2022-04-22T17:37:18.148 INFO:tasks.rados.rados.0.smithi115.stdout:3505: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:37:18.148 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2060 (ObjNum 952 snap 308 seq_num 952) dirty exists 2022-04-22T17:37:18.148 INFO:tasks.rados.rados.0.smithi115.stdout:3504: finishing write tid 4 to smithi11541303-26 2022-04-22T17:37:18.148 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2038 (ObjNum 1061 snap 336 seq_num 1061) dirty exists 2022-04-22T17:37:18.148 INFO:tasks.rados.rados.0.smithi115.stdout:3504: left oid 26 (ObjNum 1061 snap 336 seq_num 1061) 2022-04-22T17:37:18.155 INFO:tasks.rados.rados.0.smithi115.stdout:3506: finishing write tid 1 to smithi11541303-22 2022-04-22T17:37:18.155 INFO:tasks.rados.rados.0.smithi115.stdout:3506: finishing write tid 2 to smithi11541303-22 2022-04-22T17:37:18.156 INFO:tasks.rados.rados.0.smithi115.stdout:3506: finishing write tid 3 to smithi11541303-22 2022-04-22T17:37:18.158 INFO:tasks.rados.rados.0.smithi115.stdout:3506: finishing write tid 4 to smithi11541303-22 2022-04-22T17:37:18.158 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1861 (ObjNum 1062 snap 336 seq_num 1062) dirty exists 2022-04-22T17:37:18.159 INFO:tasks.rados.rados.0.smithi115.stdout:3506: left oid 22 (ObjNum 1062 snap 336 seq_num 1062) 2022-04-22T17:37:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:17 smithi121 bash[36902]: cluster 2022-04-22T17:37:16.976190+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2022-04-22T17:37:18.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:17 smithi115 bash[45495]: cluster 2022-04-22T17:37:16.976190+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2022-04-22T17:37:18.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:17 smithi115 bash[46934]: cluster 2022-04-22T17:37:16.976190+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2022-04-22T17:37:19.016 INFO:tasks.rados.rados.0.smithi115.stdout:3504: done (3 left) 2022-04-22T17:37:19.016 INFO:tasks.rados.rados.0.smithi115.stdout:3505: done (2 left) 2022-04-22T17:37:19.016 INFO:tasks.rados.rados.0.smithi115.stdout:3506: done (1 left) 2022-04-22T17:37:19.016 INFO:tasks.rados.rados.0.smithi115.stdout:3507: done (0 left) 2022-04-22T17:37:19.016 INFO:tasks.rados.rados.0.smithi115.stdout:3508: delete oid 49 current snap is 337 2022-04-22T17:37:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:3508: done (0 left) 2022-04-22T17:37:19.020 INFO:tasks.rados.rados.0.smithi115.stdout:3509: snap_remove snap 329 2022-04-22T17:37:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:18 smithi121 bash[36902]: cluster 2022-04-22T17:37:17.352765+0000 mgr.y (mgr.34107) 437 : cluster [DBG] pgmap v793: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 424 KiB/s wr, 4 op/s 2022-04-22T17:37:19.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:18 smithi121 bash[36902]: cluster 2022-04-22T17:37:17.992792+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2022-04-22T17:37:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:18 smithi115 bash[45495]: cluster 2022-04-22T17:37:17.352765+0000 mgr.y (mgr.34107) 437 : cluster [DBG] pgmap v793: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 424 KiB/s wr, 4 op/s 2022-04-22T17:37:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:18 smithi115 bash[45495]: cluster 2022-04-22T17:37:17.992792+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2022-04-22T17:37:19.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:18 smithi115 bash[46934]: cluster 2022-04-22T17:37:17.352765+0000 mgr.y (mgr.34107) 437 : cluster [DBG] pgmap v793: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 424 KiB/s wr, 4 op/s 2022-04-22T17:37:19.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:18 smithi115 bash[46934]: cluster 2022-04-22T17:37:17.992792+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2022-04-22T17:37:19.977 INFO:tasks.rados.rados.0.smithi115.stdout:3509: done (0 left) 2022-04-22T17:37:19.978 INFO:tasks.rados.rados.0.smithi115.stdout:3510: rollback oid 20 current snap is 337 2022-04-22T17:37:19.979 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 324 2022-04-22T17:37:19.979 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 337 2022-04-22T17:37:19.979 INFO:tasks.rados.rados.0.smithi115.stdout:3511: seq_num 1064 ranges {860160=622592,1482752=401408} 2022-04-22T17:37:19.989 INFO:tasks.rados.rados.0.smithi115.stdout:3511: writing smithi11541303-10 from 860160 to 1482752 tid 1 2022-04-22T17:37:19.990 INFO:tasks.rados.rados.0.smithi115.stdout:3511: writing smithi11541303-10 from 1482752 to 1884160 tid 2 2022-04-22T17:37:19.993 INFO:tasks.rados.rados.0.smithi115.stdout:3512: copy_from oid 24 from oid 49 current snap is 337 2022-04-22T17:37:19.994 INFO:tasks.rados.rados.0.smithi115.stdout:3513: rollback oid 36 current snap is 337 2022-04-22T17:37:19.994 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 36 to 336 2022-04-22T17:37:19.994 INFO:tasks.rados.rados.0.smithi115.stdout:3514: snap_remove snap 326 2022-04-22T17:37:20.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[45495]: cluster 2022-04-22T17:37:19.003360+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2022-04-22T17:37:20.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[45495]: audit 2022-04-22T17:37:19.420196+0000 mon.a (mon.0) 1409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[45495]: cluster 2022-04-22T17:37:19.963319+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2022-04-22T17:37:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[46934]: cluster 2022-04-22T17:37:19.003360+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2022-04-22T17:37:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[46934]: audit 2022-04-22T17:37:19.420196+0000 mon.a (mon.0) 1409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:20.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:20 smithi115 bash[46934]: cluster 2022-04-22T17:37:19.963319+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2022-04-22T17:37:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:20 smithi121 bash[36902]: cluster 2022-04-22T17:37:19.003360+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2022-04-22T17:37:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:20 smithi121 bash[36902]: audit 2022-04-22T17:37:19.420196+0000 mon.a (mon.0) 1409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:20 smithi121 bash[36902]: cluster 2022-04-22T17:37:19.963319+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2022-04-22T17:37:20.991 INFO:tasks.rados.rados.0.smithi115.stdout:3514: done (4 left) 2022-04-22T17:37:20.991 INFO:tasks.rados.rados.0.smithi115.stdout:3515: delete oid 43 current snap is 337 2022-04-22T17:37:20.991 INFO:tasks.rados.rados.0.smithi115.stdout:3510: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:37:20.991 INFO:tasks.rados.rados.0.smithi115.stdout:3510: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:37:20.991 INFO:tasks.rados.rados.0.smithi115.stdout:3510: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1918 (ObjNum 1020 snap 321 seq_num 1020) dirty exists 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3512: finishing copy_from to smithi11541303-24 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3512: got expected ENOENT (src dne) 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3512: finishing copy_from racing read to smithi11541303-24 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3511: finishing write tid 1 to smithi11541303-10 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3511: finishing write tid 2 to smithi11541303-10 2022-04-22T17:37:20.992 INFO:tasks.rados.rados.0.smithi115.stdout:3511: finishing write tid 3 to smithi11541303-10 2022-04-22T17:37:20.993 INFO:tasks.rados.rados.0.smithi115.stdout:3513: finishing rollback tid 0 to smithi11541303-36 2022-04-22T17:37:20.993 INFO:tasks.rados.rados.0.smithi115.stdout:3513: finishing rollback tid 1 to smithi11541303-36 2022-04-22T17:37:20.993 INFO:tasks.rados.rados.0.smithi115.stdout:3513: finishing rollback tid 2 to smithi11541303-36 2022-04-22T17:37:20.993 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 36 v 1944 (ObjNum 1006 snap 320 seq_num 1006) dirty exists 2022-04-22T17:37:20.993 INFO:tasks.rados.rados.0.smithi115.stdout:3511: finishing write tid 4 to smithi11541303-10 2022-04-22T17:37:20.994 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2052 (ObjNum 1064 snap 337 seq_num 1064) dirty exists 2022-04-22T17:37:20.994 INFO:tasks.rados.rados.0.smithi115.stdout:3511: left oid 10 (ObjNum 1064 snap 337 seq_num 1064) 2022-04-22T17:37:21.009 INFO:tasks.rados.rados.0.smithi115.stdout:3510: done (4 left) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3511: done (3 left) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3512: done (2 left) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3513: done (1 left) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3515: done (0 left) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3516: read oid 27 snap -1 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3516: expect (ObjNum 1033 snap 324 seq_num 1033) 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3517: copy_from oid 25 from oid 17 current snap is 337 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 337 2022-04-22T17:37:21.010 INFO:tasks.rados.rados.0.smithi115.stdout:3518: seq_num 1066 ranges {229376=663552,892928=401408} 2022-04-22T17:37:21.016 INFO:tasks.rados.rados.0.smithi115.stdout:3518: writing smithi11541303-19 from 229376 to 892928 tid 1 2022-04-22T17:37:21.019 INFO:tasks.rados.rados.0.smithi115.stdout:3518: writing smithi11541303-19 from 892928 to 1294336 tid 2 2022-04-22T17:37:21.026 INFO:tasks.rados.rados.0.smithi115.stdout:3516: expect (ObjNum 1033 snap 324 seq_num 1033) 2022-04-22T17:37:21.037 INFO:tasks.rados.rados.0.smithi115.stdout:3516: done (2 left) 2022-04-22T17:37:21.037 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 337 2022-04-22T17:37:21.037 INFO:tasks.rados.rados.0.smithi115.stdout:3518: finishing write tid 1 to smithi11541303-19 2022-04-22T17:37:21.037 INFO:tasks.rados.rados.0.smithi115.stdout:3519: seq_num 1067 ranges {2867200=679936,3547136=401408} 2022-04-22T17:37:21.058 INFO:tasks.rados.rados.0.smithi115.stdout:3519: writing smithi11541303-44 from 2867200 to 3547136 tid 1 2022-04-22T17:37:21.063 INFO:tasks.rados.rados.0.smithi115.stdout:3519: writing smithi11541303-44 from 3547136 to 3948544 tid 2 2022-04-22T17:37:21.063 INFO:tasks.rados.rados.0.smithi115.stdout:3520: rollback oid 45 current snap is 337 2022-04-22T17:37:21.063 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 45 to 332 2022-04-22T17:37:21.063 INFO:tasks.rados.rados.0.smithi115.stdout:3518: finishing write tid 2 to smithi11541303-19 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3518: finishing write tid 3 to smithi11541303-19 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3518: finishing write tid 4 to smithi11541303-19 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1790 (ObjNum 1066 snap 337 seq_num 1066) dirty exists 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3518: left oid 19 (ObjNum 1066 snap 337 seq_num 1066) 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3517: finishing copy_from to smithi11541303-25 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1885 (ObjNum 992 snap 315 seq_num 992) dirty exists 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3517: finishing copy_from racing read to smithi11541303-25 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3517: done (3 left) 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3518: done (2 left) 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3521: read oid 15 snap -1 2022-04-22T17:37:21.064 INFO:tasks.rados.rados.0.smithi115.stdout:3521: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:37:21.065 INFO:tasks.rados.rados.0.smithi115.stdout:3522: delete oid 11 current snap is 337 2022-04-22T17:37:21.065 INFO:tasks.rados.rados.0.smithi115.stdout:3519: finishing write tid 1 to smithi11541303-44 2022-04-22T17:37:21.065 INFO:tasks.rados.rados.0.smithi115.stdout:3519: finishing write tid 2 to smithi11541303-44 2022-04-22T17:37:21.065 INFO:tasks.rados.rados.0.smithi115.stdout:3519: finishing write tid 3 to smithi11541303-44 2022-04-22T17:37:21.066 INFO:tasks.rados.rados.0.smithi115.stdout:3519: finishing write tid 4 to smithi11541303-44 2022-04-22T17:37:21.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2128 (ObjNum 1067 snap 337 seq_num 1067) dirty exists 2022-04-22T17:37:21.066 INFO:tasks.rados.rados.0.smithi115.stdout:3519: left oid 44 (ObjNum 1067 snap 337 seq_num 1067) 2022-04-22T17:37:21.067 INFO:tasks.rados.rados.0.smithi115.stdout:3520: finishing rollback tid 0 to smithi11541303-45 2022-04-22T17:37:21.068 INFO:tasks.rados.rados.0.smithi115.stdout:3519: done (3 left) 2022-04-22T17:37:21.068 INFO:tasks.rados.rados.0.smithi115.stdout:3522: done (2 left) 2022-04-22T17:37:21.068 INFO:tasks.rados.rados.0.smithi115.stdout:3523: rollback oid 49 current snap is 337 2022-04-22T17:37:21.068 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 333 2022-04-22T17:37:21.069 INFO:tasks.rados.rados.0.smithi115.stdout:3524: delete oid 29 current snap is 337 2022-04-22T17:37:21.070 INFO:tasks.rados.rados.0.smithi115.stdout:3520: finishing rollback tid 1 to smithi11541303-45 2022-04-22T17:37:21.070 INFO:tasks.rados.rados.0.smithi115.stdout:3523: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:37:21.070 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 0 (ObjNum -1116461968 snap 22075 seq_num 0) dirty dne 2022-04-22T17:37:21.070 INFO:tasks.rados.rados.0.smithi115.stdout:3520: finishing rollback tid 2 to smithi11541303-45 2022-04-22T17:37:21.070 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2063 (ObjNum 1028 snap 322 seq_num 1028) dirty exists 2022-04-22T17:37:21.072 INFO:tasks.rados.rados.0.smithi115.stdout:3521: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:37:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:3520: done (3 left) 2022-04-22T17:37:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:3521: done (2 left) 2022-04-22T17:37:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:3523: done (1 left) 2022-04-22T17:37:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:3524: done (0 left) 2022-04-22T17:37:21.130 INFO:tasks.rados.rados.0.smithi115.stdout:3525: setattr oid 49 current snap is 337 2022-04-22T17:37:21.133 INFO:tasks.rados.rados.0.smithi115.stdout:3526: setattr oid 12 current snap is 337 2022-04-22T17:37:21.136 INFO:tasks.rados.rados.0.smithi115.stdout:3527: read oid 8 snap -1 2022-04-22T17:37:21.136 INFO:tasks.rados.rados.0.smithi115.stdout:3527: expect (ObjNum 1045 snap 328 seq_num 1045) 2022-04-22T17:37:21.136 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 337 2022-04-22T17:37:21.136 INFO:tasks.rados.rados.0.smithi115.stdout:3528: seq_num 1070 ranges {3883008=729088,4612096=401408} 2022-04-22T17:37:21.165 INFO:tasks.rados.rados.0.smithi115.stdout:3528: writing smithi11541303-31 from 3883008 to 4612096 tid 1 2022-04-22T17:37:21.167 INFO:tasks.rados.rados.0.smithi115.stdout:3528: writing smithi11541303-31 from 4612096 to 5013504 tid 2 2022-04-22T17:37:21.167 INFO:tasks.rados.rados.0.smithi115.stdout:3529: read oid 40 snap -1 2022-04-22T17:37:21.167 INFO:tasks.rados.rados.0.smithi115.stdout:3529: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:37:21.168 INFO:tasks.rados.rados.0.smithi115.stdout:3530: snap_remove snap 330 2022-04-22T17:37:21.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:21 smithi121 bash[36902]: cluster 2022-04-22T17:37:19.353527+0000 mgr.y (mgr.34107) 438 : cluster [DBG] pgmap v796: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 6 op/s 2022-04-22T17:37:21.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:21 smithi121 bash[36902]: cluster 2022-04-22T17:37:20.978147+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2022-04-22T17:37:21.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:21 smithi115 bash[45495]: cluster 2022-04-22T17:37:19.353527+0000 mgr.y (mgr.34107) 438 : cluster [DBG] pgmap v796: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 6 op/s 2022-04-22T17:37:21.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:21 smithi115 bash[45495]: cluster 2022-04-22T17:37:20.978147+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2022-04-22T17:37:21.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:21 smithi115 bash[46934]: cluster 2022-04-22T17:37:19.353527+0000 mgr.y (mgr.34107) 438 : cluster [DBG] pgmap v796: 121 pgs: 121 active+clean; 159 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 6 op/s 2022-04-22T17:37:21.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:21 smithi115 bash[46934]: cluster 2022-04-22T17:37:20.978147+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2022-04-22T17:37:22.029 INFO:tasks.rados.rados.0.smithi115.stdout:3530: done (5 left) 2022-04-22T17:37:22.029 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 44 current snap is 337 2022-04-22T17:37:22.029 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2153 (ObjNum -1116461968 snap 22075 seq_num 0) dirty exists 2022-04-22T17:37:22.030 INFO:tasks.rados.rados.0.smithi115.stdout:3531: seq_num 1071 ranges {3948544=745472,4694016=401408} 2022-04-22T17:37:22.058 INFO:tasks.rados.rados.0.smithi115.stdout:3531: writing smithi11541303-44 from 3948544 to 4694016 tid 1 2022-04-22T17:37:22.060 INFO:tasks.rados.rados.0.smithi115.stdout:3531: writing smithi11541303-44 from 4694016 to 5095424 tid 2 2022-04-22T17:37:22.060 INFO:tasks.rados.rados.0.smithi115.stdout:3525: done (5 left) 2022-04-22T17:37:22.061 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 337 2022-04-22T17:37:22.061 INFO:tasks.rados.rados.0.smithi115.stdout:3532: seq_num 1072 ranges {4726784=761856,5488640=401408} 2022-04-22T17:37:22.094 INFO:tasks.rados.rados.0.smithi115.stdout:3532: writing smithi11541303-30 from 4726784 to 5488640 tid 1 2022-04-22T17:37:22.095 INFO:tasks.rados.rados.0.smithi115.stdout:3532: writing smithi11541303-30 from 5488640 to 5890048 tid 2 2022-04-22T17:37:22.095 INFO:tasks.rados.rados.0.smithi115.stdout:3533: read oid 14 snap 332 2022-04-22T17:37:22.095 INFO:tasks.rados.rados.0.smithi115.stdout:3533: expect (ObjNum 1040 snap 324 seq_num 1040) 2022-04-22T17:37:22.095 INFO:tasks.rados.rados.0.smithi115.stdout:3534: copy_from oid 49 from oid 36 current snap is 337 2022-04-22T17:37:22.096 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 1791 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:37:22.096 INFO:tasks.rados.rados.0.smithi115.stdout:3527: expect (ObjNum 1045 snap 328 seq_num 1045) 2022-04-22T17:37:22.121 INFO:tasks.rados.rados.0.smithi115.stdout:3529: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3528: finishing write tid 1 to smithi11541303-31 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3528: finishing write tid 2 to smithi11541303-31 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3528: finishing write tid 3 to smithi11541303-31 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3528: finishing write tid 4 to smithi11541303-31 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2157 (ObjNum 1070 snap 337 seq_num 1070) dirty exists 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3528: left oid 31 (ObjNum 1070 snap 337 seq_num 1070) 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3531: finishing write tid 1 to smithi11541303-44 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3531: finishing write tid 2 to smithi11541303-44 2022-04-22T17:37:22.149 INFO:tasks.rados.rados.0.smithi115.stdout:3531: finishing write tid 3 to smithi11541303-44 2022-04-22T17:37:22.150 INFO:tasks.rados.rados.0.smithi115.stdout:3531: finishing write tid 4 to smithi11541303-44 2022-04-22T17:37:22.150 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2131 (ObjNum 1071 snap 337 seq_num 1071) dirty exists 2022-04-22T17:37:22.150 INFO:tasks.rados.rados.0.smithi115.stdout:3531: left oid 44 (ObjNum 1071 snap 337 seq_num 1071) 2022-04-22T17:37:22.150 INFO:tasks.rados.rados.0.smithi115.stdout:3533: expect (ObjNum 1040 snap 324 seq_num 1040) 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3534: finishing copy_from to smithi11541303-49 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 1944 (ObjNum 1006 snap 320 seq_num 1006) dirty exists 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3532: finishing write tid 1 to smithi11541303-30 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3532: finishing write tid 2 to smithi11541303-30 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3534: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3532: finishing write tid 3 to smithi11541303-30 2022-04-22T17:37:22.168 INFO:tasks.rados.rados.0.smithi115.stdout:3532: finishing write tid 4 to smithi11541303-30 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2066 (ObjNum 1072 snap 337 seq_num 1072) dirty exists 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3532: left oid 30 (ObjNum 1072 snap 337 seq_num 1072) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3526: done (7 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3527: done (6 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3528: done (5 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3529: done (4 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3531: done (3 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3532: done (2 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3533: done (1 left) 2022-04-22T17:37:22.169 INFO:tasks.rados.rados.0.smithi115.stdout:3534: done (0 left) 2022-04-22T17:37:22.170 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 13 current snap is 337 2022-04-22T17:37:22.170 INFO:tasks.rados.rados.0.smithi115.stdout:3535: seq_num 1073 ranges {8077312=778240,8855552=401408} 2022-04-22T17:37:22.222 INFO:tasks.rados.rados.0.smithi115.stdout:3535: writing smithi11541303-13 from 8077312 to 8855552 tid 1 2022-04-22T17:37:22.224 INFO:tasks.rados.rados.0.smithi115.stdout:3535: writing smithi11541303-13 from 8855552 to 9256960 tid 2 2022-04-22T17:37:22.224 INFO:tasks.rados.rados.0.smithi115.stdout:3536: rollback oid 3 current snap is 337 2022-04-22T17:37:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 3 to 332 2022-04-22T17:37:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:3537: rollback oid 23 current snap is 337 2022-04-22T17:37:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 23 to 336 2022-04-22T17:37:22.225 INFO:tasks.rados.rados.0.smithi115.stdout:3538: snap_create 2022-04-22T17:37:22.235 INFO:tasks.rados.rados.0.smithi115.stdout:3535: finishing write tid 1 to smithi11541303-13 2022-04-22T17:37:22.235 INFO:tasks.rados.rados.0.smithi115.stdout:3535: finishing write tid 2 to smithi11541303-13 2022-04-22T17:37:22.235 INFO:tasks.rados.rados.0.smithi115.stdout:3535: finishing write tid 3 to smithi11541303-13 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:3536: finishing rollback tid 0 to smithi11541303-3 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:3536: finishing rollback tid 1 to smithi11541303-3 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:3536: finishing rollback tid 2 to smithi11541303-3 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2072 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:3537: finishing rollback tid 0 to smithi11541303-23 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:3535: finishing write tid 4 to smithi11541303-13 2022-04-22T17:37:22.238 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 13 v 1930 (ObjNum 1073 snap 337 seq_num 1073) dirty exists 2022-04-22T17:37:22.239 INFO:tasks.rados.rados.0.smithi115.stdout:3535: left oid 13 (ObjNum 1073 snap 337 seq_num 1073) 2022-04-22T17:37:22.239 INFO:tasks.rados.rados.0.smithi115.stdout:3537: finishing rollback tid 1 to smithi11541303-23 2022-04-22T17:37:22.240 INFO:tasks.rados.rados.0.smithi115.stdout:3537: finishing rollback tid 2 to smithi11541303-23 2022-04-22T17:37:22.241 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2075 (ObjNum 1058 snap 336 seq_num 1058) dirty exists 2022-04-22T17:37:23.040 INFO:tasks.rados.rados.0.smithi115.stdout:3535: done (3 left) 2022-04-22T17:37:23.040 INFO:tasks.rados.rados.0.smithi115.stdout:3536: done (2 left) 2022-04-22T17:37:23.040 INFO:tasks.rados.rados.0.smithi115.stdout:3537: done (1 left) 2022-04-22T17:37:23.041 INFO:tasks.rados.rados.0.smithi115.stdout:3538: done (0 left) 2022-04-22T17:37:23.041 INFO:tasks.rados.rados.0.smithi115.stdout:3539: read oid 5 snap -1 2022-04-22T17:37:23.041 INFO:tasks.rados.rados.0.smithi115.stdout:3539: expect deleted 2022-04-22T17:37:23.041 INFO:tasks.rados.rados.0.smithi115.stdout:3540: copy_from oid 49 from oid 47 current snap is 338 2022-04-22T17:37:23.041 INFO:tasks.rados.rados.0.smithi115.stdout:3541: delete oid 34 current snap is 338 2022-04-22T17:37:23.048 INFO:tasks.rados.rados.0.smithi115.stdout:3539: done (2 left) 2022-04-22T17:37:23.048 INFO:tasks.rados.rados.0.smithi115.stdout:3541: done (1 left) 2022-04-22T17:37:23.048 INFO:tasks.rados.rados.0.smithi115.stdout:3542: delete oid 23 current snap is 338 2022-04-22T17:37:23.055 INFO:tasks.rados.rados.0.smithi115.stdout:3542: done (1 left) 2022-04-22T17:37:23.056 INFO:tasks.rados.rados.0.smithi115.stdout:3543: snap_create 2022-04-22T17:37:23.072 INFO:tasks.rados.rados.0.smithi115.stdout:3540: finishing copy_from to smithi11541303-49 2022-04-22T17:37:23.072 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2039 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:37:23.072 INFO:tasks.rados.rados.0.smithi115.stdout:3540: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:37:23.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[45495]: cluster 2022-04-22T17:37:21.354384+0000 mgr.y (mgr.34107) 439 : cluster [DBG] pgmap v799: 121 pgs: 121 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 944 KiB/s wr, 10 op/s 2022-04-22T17:37:23.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[45495]: cluster 2022-04-22T17:37:22.017268+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2022-04-22T17:37:23.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[46934]: cluster 2022-04-22T17:37:21.354384+0000 mgr.y (mgr.34107) 439 : cluster [DBG] pgmap v799: 121 pgs: 121 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 944 KiB/s wr, 10 op/s 2022-04-22T17:37:23.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[46934]: cluster 2022-04-22T17:37:22.017268+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2022-04-22T17:37:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:23 smithi121 bash[36902]: cluster 2022-04-22T17:37:21.354384+0000 mgr.y (mgr.34107) 439 : cluster [DBG] pgmap v799: 121 pgs: 121 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 944 KiB/s wr, 10 op/s 2022-04-22T17:37:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:23 smithi121 bash[36902]: cluster 2022-04-22T17:37:22.017268+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2022-04-22T17:37:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:23.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:24.046 INFO:tasks.rados.rados.0.smithi115.stdout:3540: done (1 left) 2022-04-22T17:37:24.047 INFO:tasks.rados.rados.0.smithi115.stdout:3543: done (0 left) 2022-04-22T17:37:24.047 INFO:tasks.rados.rados.0.smithi115.stdout:3544: snap_remove snap 337 2022-04-22T17:37:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:24 smithi121 bash[36902]: cluster 2022-04-22T17:37:23.026018+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2022-04-22T17:37:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:24 smithi121 bash[36902]: audit 2022-04-22T17:37:23.960481+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:24 smithi121 bash[36902]: audit 2022-04-22T17:37:23.961594+0000 mon.a (mon.0) 1415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:24 smithi121 bash[36902]: audit 2022-04-22T17:37:23.963679+0000 mon.a (mon.0) 1416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:24.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:24 smithi121 bash[36902]: audit 2022-04-22T17:37:23.964675+0000 mon.a (mon.0) 1417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:24.278 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 17:37:23 smithi115 bash[42914]: debug 2022-04-22T17:37:23.948+0000 7f8eb1b4f700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[45495]: cluster 2022-04-22T17:37:23.026018+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[45495]: audit 2022-04-22T17:37:23.960481+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[45495]: audit 2022-04-22T17:37:23.961594+0000 mon.a (mon.0) 1415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[45495]: audit 2022-04-22T17:37:23.963679+0000 mon.a (mon.0) 1416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[45495]: audit 2022-04-22T17:37:23.964675+0000 mon.a (mon.0) 1417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:24.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[46934]: cluster 2022-04-22T17:37:23.026018+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2022-04-22T17:37:24.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[46934]: audit 2022-04-22T17:37:23.960481+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:24.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[46934]: audit 2022-04-22T17:37:23.961594+0000 mon.a (mon.0) 1415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:24.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[46934]: audit 2022-04-22T17:37:23.963679+0000 mon.a (mon.0) 1416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:24.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:24 smithi115 bash[46934]: audit 2022-04-22T17:37:23.964675+0000 mon.a (mon.0) 1417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:24.990 INFO:tasks.rados.rados.0.smithi115.stdout:3544: done (0 left) 2022-04-22T17:37:24.992 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 339 2022-04-22T17:37:24.992 INFO:tasks.rados.rados.0.smithi115.stdout:3545: seq_num 1076 ranges {10018816=425984,10444800=548864,10993664=253952} 2022-04-22T17:37:25.049 INFO:tasks.rados.rados.0.smithi115.stdout:3545: writing smithi11541303-7 from 10018816 to 10444800 tid 1 2022-04-22T17:37:25.053 INFO:tasks.rados.rados.0.smithi115.stdout:3545: writing smithi11541303-7 from 10444800 to 10993664 tid 2 2022-04-22T17:37:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3545: writing smithi11541303-7 from 10993664 to 11247616 tid 3 2022-04-22T17:37:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3546: snap_remove snap 325 2022-04-22T17:37:25.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cluster 2022-04-22T17:37:23.355349+0000 mgr.y (mgr.34107) 440 : cluster [DBG] pgmap v802: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 519 KiB/s rd, 4.3 MiB/s wr, 24 op/s 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cephadm 2022-04-22T17:37:23.952620+0000 mgr.y (mgr.34107) 441 : cephadm [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cluster 2022-04-22T17:37:23.961588+0000 mgr.y (mgr.34107) 442 : cluster [DBG] pgmap v803: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cluster 2022-04-22T17:37:24.021431+0000 mon.a (mon.0) 1418 : cluster [WRN] Health check failed: Redeploying daemon grafana.a on host smithi121 failed. (UPGRADE_REDEPLOY_DAEMON) 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cluster 2022-04-22T17:37:24.034469+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.280773+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.294559+0000 mon.a (mon.0) 1421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.423322+0000 mon.a (mon.0) 1422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.654414+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.662692+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.664996+0000 mon.a (mon.0) 1425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.666248+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: audit 2022-04-22T17:37:24.676208+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[46934]: cluster 2022-04-22T17:37:24.974882+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cluster 2022-04-22T17:37:23.355349+0000 mgr.y (mgr.34107) 440 : cluster [DBG] pgmap v802: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 519 KiB/s rd, 4.3 MiB/s wr, 24 op/s 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cephadm 2022-04-22T17:37:23.952620+0000 mgr.y (mgr.34107) 441 : cephadm [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cluster 2022-04-22T17:37:23.961588+0000 mgr.y (mgr.34107) 442 : cluster [DBG] pgmap v803: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2022-04-22T17:37:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cluster 2022-04-22T17:37:24.021431+0000 mon.a (mon.0) 1418 : cluster [WRN] Health check failed: Redeploying daemon grafana.a on host smithi121 failed. (UPGRADE_REDEPLOY_DAEMON) 2022-04-22T17:37:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cluster 2022-04-22T17:37:24.034469+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2022-04-22T17:37:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.280773+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.294559+0000 mon.a (mon.0) 1421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.423322+0000 mon.a (mon.0) 1422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:25.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.654414+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.662692+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.664996+0000 mon.a (mon.0) 1425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.666248+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: audit 2022-04-22T17:37:24.676208+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:25 smithi115 bash[45495]: cluster 2022-04-22T17:37:24.974882+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2022-04-22T17:37:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cluster 2022-04-22T17:37:23.355349+0000 mgr.y (mgr.34107) 440 : cluster [DBG] pgmap v802: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 519 KiB/s rd, 4.3 MiB/s wr, 24 op/s 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cephadm 2022-04-22T17:37:23.952620+0000 mgr.y (mgr.34107) 441 : cephadm [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cluster 2022-04-22T17:37:23.961588+0000 mgr.y (mgr.34107) 442 : cluster [DBG] pgmap v803: 121 pgs: 121 active+clean; 176 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cluster 2022-04-22T17:37:24.021431+0000 mon.a (mon.0) 1418 : cluster [WRN] Health check failed: Redeploying daemon grafana.a on host smithi121 failed. (UPGRADE_REDEPLOY_DAEMON) 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cluster 2022-04-22T17:37:24.034469+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.280773+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.294559+0000 mon.a (mon.0) 1421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.423322+0000 mon.a (mon.0) 1422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.654414+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.662692+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:37:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.664996+0000 mon.a (mon.0) 1425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:37:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.666248+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:37:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: audit 2022-04-22T17:37:24.676208+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:37:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:25 smithi121 bash[36902]: cluster 2022-04-22T17:37:24.974882+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2022-04-22T17:37:26.062 INFO:tasks.rados.rados.0.smithi115.stdout:3546: done (1 left) 2022-04-22T17:37:26.063 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 339 2022-04-22T17:37:26.063 INFO:tasks.rados.rados.0.smithi115.stdout:3545: finishing write tid 1 to smithi11541303-7 2022-04-22T17:37:26.063 INFO:tasks.rados.rados.0.smithi115.stdout:3545: finishing write tid 2 to smithi11541303-7 2022-04-22T17:37:26.063 INFO:tasks.rados.rados.0.smithi115.stdout:3545: finishing write tid 3 to smithi11541303-7 2022-04-22T17:37:26.063 INFO:tasks.rados.rados.0.smithi115.stdout:3547: seq_num 1077 ranges {8060928=442368,8503296=434176,8937472=368640} 2022-04-22T17:37:26.114 INFO:tasks.rados.rados.0.smithi115.stdout:3547: writing smithi11541303-41 from 8060928 to 8503296 tid 1 2022-04-22T17:37:26.115 INFO:tasks.rados.rados.0.smithi115.stdout:3547: writing smithi11541303-41 from 8503296 to 8937472 tid 2 2022-04-22T17:37:26.118 INFO:tasks.rados.rados.0.smithi115.stdout:3547: writing smithi11541303-41 from 8937472 to 9306112 tid 3 2022-04-22T17:37:26.118 INFO:tasks.rados.rados.0.smithi115.stdout:3548: read oid 43 snap 335 2022-04-22T17:37:26.118 INFO:tasks.rados.rados.0.smithi115.stdout:3548: expect (ObjNum 1038 snap 324 seq_num 1038) 2022-04-22T17:37:26.119 INFO:tasks.rados.rados.0.smithi115.stdout:3549: read oid 18 snap 338 2022-04-22T17:37:26.119 INFO:tasks.rados.rados.0.smithi115.stdout:3549: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:37:26.119 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 339 2022-04-22T17:37:26.119 INFO:tasks.rados.rados.0.smithi115.stdout:3545: finishing write tid 4 to smithi11541303-7 2022-04-22T17:37:26.119 INFO:tasks.rados.rados.0.smithi115.stdout:3550: seq_num 1078 ranges {3121152=458752,3579904=720896,4300800=81920} 2022-04-22T17:37:26.138 INFO:tasks.rados.rados.0.smithi115.stdout:3550: writing smithi11541303-49 from 3121152 to 3579904 tid 1 2022-04-22T17:37:26.141 INFO:tasks.rados.rados.0.smithi115.stdout:3550: writing smithi11541303-49 from 3579904 to 4300800 tid 2 2022-04-22T17:37:26.144 INFO:tasks.rados.rados.0.smithi115.stdout:3550: writing smithi11541303-49 from 4300800 to 4382720 tid 3 2022-04-22T17:37:26.145 INFO:tasks.rados.rados.0.smithi115.stdout:3551: rollback oid 42 current snap is 339 2022-04-22T17:37:26.145 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 42 to 335 2022-04-22T17:37:26.145 INFO:tasks.rados.rados.0.smithi115.stdout:3552: setattr oid 39 current snap is 339 2022-04-22T17:37:26.145 INFO:tasks.rados.rados.0.smithi115.stdout:3545: finishing write tid 5 to smithi11541303-7 2022-04-22T17:37:26.147 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1948 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:37:26.147 INFO:tasks.rados.rados.0.smithi115.stdout:3545: left oid 7 (ObjNum 1076 snap 339 seq_num 1076) 2022-04-22T17:37:26.147 INFO:tasks.rados.rados.0.smithi115.stdout:3549: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:37:26.177 INFO:tasks.rados.rados.0.smithi115.stdout:3547: finishing write tid 1 to smithi11541303-41 2022-04-22T17:37:26.177 INFO:tasks.rados.rados.0.smithi115.stdout:3547: finishing write tid 2 to smithi11541303-41 2022-04-22T17:37:26.177 INFO:tasks.rados.rados.0.smithi115.stdout:3547: finishing write tid 3 to smithi11541303-41 2022-04-22T17:37:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:3547: finishing write tid 4 to smithi11541303-41 2022-04-22T17:37:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:3547: finishing write tid 5 to smithi11541303-41 2022-04-22T17:37:26.178 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 2056 (ObjNum 1077 snap 339 seq_num 1077) dirty exists 2022-04-22T17:37:26.179 INFO:tasks.rados.rados.0.smithi115.stdout:3547: left oid 41 (ObjNum 1077 snap 339 seq_num 1077) 2022-04-22T17:37:26.181 INFO:tasks.rados.rados.0.smithi115.stdout:3550: finishing write tid 1 to smithi11541303-49 2022-04-22T17:37:26.181 INFO:tasks.rados.rados.0.smithi115.stdout:3550: finishing write tid 2 to smithi11541303-49 2022-04-22T17:37:26.182 INFO:tasks.rados.rados.0.smithi115.stdout:3550: finishing write tid 3 to smithi11541303-49 2022-04-22T17:37:26.183 INFO:tasks.rados.rados.0.smithi115.stdout:3550: finishing write tid 4 to smithi11541303-49 2022-04-22T17:37:26.183 INFO:tasks.rados.rados.0.smithi115.stdout:3551: finishing rollback tid 0 to smithi11541303-42 2022-04-22T17:37:26.183 INFO:tasks.rados.rados.0.smithi115.stdout:3551: finishing rollback tid 1 to smithi11541303-42 2022-04-22T17:37:26.183 INFO:tasks.rados.rados.0.smithi115.stdout:3550: finishing write tid 5 to smithi11541303-49 2022-04-22T17:37:26.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2177 (ObjNum 1078 snap 339 seq_num 1078) dirty exists 2022-04-22T17:37:26.184 INFO:tasks.rados.rados.0.smithi115.stdout:3550: left oid 49 (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:37:26.184 INFO:tasks.rados.rados.0.smithi115.stdout:3551: finishing rollback tid 2 to smithi11541303-42 2022-04-22T17:37:26.184 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2181 (ObjNum 1041 snap 326 seq_num 1041) dirty exists 2022-04-22T17:37:26.184 INFO:tasks.rados.rados.0.smithi115.stdout:3548: expect (ObjNum 1038 snap 324 seq_num 1038) 2022-04-22T17:37:26.916 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1936 (ObjNum 1025 snap 321 seq_num 1025) dirty exists 2022-04-22T17:37:26.916 INFO:tasks.rados.rados.0.smithi115.stdout:3545: done (6 left) 2022-04-22T17:37:26.916 INFO:tasks.rados.rados.0.smithi115.stdout:3547: done (5 left) 2022-04-22T17:37:26.916 INFO:tasks.rados.rados.0.smithi115.stdout:3548: done (4 left) 2022-04-22T17:37:26.916 INFO:tasks.rados.rados.0.smithi115.stdout:3549: done (3 left) 2022-04-22T17:37:26.917 INFO:tasks.rados.rados.0.smithi115.stdout:3550: done (2 left) 2022-04-22T17:37:26.917 INFO:tasks.rados.rados.0.smithi115.stdout:3551: done (1 left) 2022-04-22T17:37:26.917 INFO:tasks.rados.rados.0.smithi115.stdout:3552: done (0 left) 2022-04-22T17:37:26.917 INFO:tasks.rados.rados.0.smithi115.stdout:3553: snap_create 2022-04-22T17:37:27.073 INFO:tasks.rados.rados.0.smithi115.stdout:3553: done (0 left) 2022-04-22T17:37:27.074 INFO:tasks.rados.rados.0.smithi115.stdout:3554: read oid 20 snap -1 2022-04-22T17:37:27.074 INFO:tasks.rados.rados.0.smithi115.stdout:3554: expect (ObjNum 1020 snap 321 seq_num 1020) 2022-04-22T17:37:27.074 INFO:tasks.rados.rados.0.smithi115.stdout:3555: snap_create 2022-04-22T17:37:27.082 INFO:tasks.rados.rados.0.smithi115.stdout:3554: expect (ObjNum 1020 snap 321 seq_num 1020) 2022-04-22T17:37:27.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:27 smithi121 bash[36902]: cluster 2022-04-22T17:37:25.962656+0000 mgr.y (mgr.34107) 443 : cluster [DBG] pgmap v806: 121 pgs: 121 active+clean; 177 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:27.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:27 smithi121 bash[36902]: cluster 2022-04-22T17:37:26.048361+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2022-04-22T17:37:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:27 smithi115 bash[45495]: cluster 2022-04-22T17:37:25.962656+0000 mgr.y (mgr.34107) 443 : cluster [DBG] pgmap v806: 121 pgs: 121 active+clean; 177 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:27 smithi115 bash[45495]: cluster 2022-04-22T17:37:26.048361+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2022-04-22T17:37:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:27 smithi115 bash[46934]: cluster 2022-04-22T17:37:25.962656+0000 mgr.y (mgr.34107) 443 : cluster [DBG] pgmap v806: 121 pgs: 121 active+clean; 177 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:27 smithi115 bash[46934]: cluster 2022-04-22T17:37:26.048361+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2022-04-22T17:37:28.086 INFO:tasks.rados.rados.0.smithi115.stdout:3554: done (1 left) 2022-04-22T17:37:28.086 INFO:tasks.rados.rados.0.smithi115.stdout:3555: done (0 left) 2022-04-22T17:37:28.087 INFO:tasks.rados.rados.0.smithi115.stdout:3556: snap_create 2022-04-22T17:37:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:28 smithi121 bash[36902]: cluster 2022-04-22T17:37:27.056323+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2022-04-22T17:37:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:28 smithi115 bash[46934]: cluster 2022-04-22T17:37:27.056323+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2022-04-22T17:37:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:28 smithi115 bash[45495]: cluster 2022-04-22T17:37:27.056323+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2022-04-22T17:37:29.089 INFO:tasks.rados.rados.0.smithi115.stdout:3556: done (0 left) 2022-04-22T17:37:29.089 INFO:tasks.rados.rados.0.smithi115.stdout:3557: rmattr oid 8 current snap is 342 2022-04-22T17:37:29.090 INFO:tasks.rados.rados.0.smithi115.stdout:3558: copy_from oid 30 from oid 7 current snap is 342 2022-04-22T17:37:29.090 INFO:tasks.rados.rados.0.smithi115.stdout:3559: setattr oid 47 current snap is 342 2022-04-22T17:37:29.092 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 342 2022-04-22T17:37:29.092 INFO:tasks.rados.rados.0.smithi115.stdout:3560: seq_num 1079 ranges {630784=475136,1105920=606208,1712128=196608} 2022-04-22T17:37:29.100 INFO:tasks.rados.rados.0.smithi115.stdout:3560: writing smithi11541303-20 from 630784 to 1105920 tid 1 2022-04-22T17:37:29.103 INFO:tasks.rados.rados.0.smithi115.stdout:3560: writing smithi11541303-20 from 1105920 to 1712128 tid 2 2022-04-22T17:37:29.104 INFO:tasks.rados.rados.0.smithi115.stdout:3560: writing smithi11541303-20 from 1712128 to 1908736 tid 3 2022-04-22T17:37:29.104 INFO:tasks.rados.rados.0.smithi115.stdout:3561: snap_remove snap 340 2022-04-22T17:37:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:29 smithi121 bash[36902]: cluster 2022-04-22T17:37:27.963463+0000 mgr.y (mgr.34107) 444 : cluster [DBG] pgmap v809: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2022-04-22T17:37:29.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:29 smithi121 bash[36902]: cluster 2022-04-22T17:37:28.074069+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2022-04-22T17:37:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:29 smithi115 bash[45495]: cluster 2022-04-22T17:37:27.963463+0000 mgr.y (mgr.34107) 444 : cluster [DBG] pgmap v809: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2022-04-22T17:37:29.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:29 smithi115 bash[45495]: cluster 2022-04-22T17:37:28.074069+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2022-04-22T17:37:29.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:29 smithi115 bash[46934]: cluster 2022-04-22T17:37:27.963463+0000 mgr.y (mgr.34107) 444 : cluster [DBG] pgmap v809: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2022-04-22T17:37:29.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:29 smithi115 bash[46934]: cluster 2022-04-22T17:37:28.074069+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2022-04-22T17:37:29.995 INFO:tasks.rados.rados.0.smithi115.stdout:3561: done (4 left) 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:3562: read oid 16 snap 336 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:3562: expect deleted 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 342 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 0 (ObjNum 1045 snap 328 seq_num 1045) dirty exists 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 47 v 2057 (ObjNum 822 snap 274 seq_num 822) dirty exists 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:3560: finishing write tid 1 to smithi11541303-20 2022-04-22T17:37:29.996 INFO:tasks.rados.rados.0.smithi115.stdout:3560: finishing write tid 2 to smithi11541303-20 2022-04-22T17:37:29.997 INFO:tasks.rados.rados.0.smithi115.stdout:3560: finishing write tid 3 to smithi11541303-20 2022-04-22T17:37:29.997 INFO:tasks.rados.rados.0.smithi115.stdout:3560: finishing write tid 4 to smithi11541303-20 2022-04-22T17:37:29.997 INFO:tasks.rados.rados.0.smithi115.stdout:3560: finishing write tid 5 to smithi11541303-20 2022-04-22T17:37:29.997 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1943 (ObjNum 1079 snap 342 seq_num 1079) dirty exists 2022-04-22T17:37:29.997 INFO:tasks.rados.rados.0.smithi115.stdout:3560: left oid 20 (ObjNum 1079 snap 342 seq_num 1079) 2022-04-22T17:37:29.998 INFO:tasks.rados.rados.0.smithi115.stdout:3558: finishing copy_from to smithi11541303-30 2022-04-22T17:37:29.998 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 1948 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:37:29.998 INFO:tasks.rados.rados.0.smithi115.stdout:3558: finishing copy_from racing read to smithi11541303-30 2022-04-22T17:37:29.998 INFO:tasks.rados.rados.0.smithi115.stdout:3563: seq_num 1080 ranges {0=491520,491520=540672,1032192=262144} 2022-04-22T17:37:29.999 INFO:tasks.rados.rados.0.smithi115.stdout:3563: writing smithi11541303-11 from 0 to 491520 tid 1 2022-04-22T17:37:30.001 INFO:tasks.rados.rados.0.smithi115.stdout:3563: writing smithi11541303-11 from 491520 to 1032192 tid 2 2022-04-22T17:37:30.003 INFO:tasks.rados.rados.0.smithi115.stdout:3563: writing smithi11541303-11 from 1032192 to 1294336 tid 3 2022-04-22T17:37:30.003 INFO:tasks.rados.rados.0.smithi115.stdout:3557: done (5 left) 2022-04-22T17:37:30.003 INFO:tasks.rados.rados.0.smithi115.stdout:3558: done (4 left) 2022-04-22T17:37:30.004 INFO:tasks.rados.rados.0.smithi115.stdout:3559: done (3 left) 2022-04-22T17:37:30.004 INFO:tasks.rados.rados.0.smithi115.stdout:3560: done (2 left) 2022-04-22T17:37:30.004 INFO:tasks.rados.rados.0.smithi115.stdout:3564: snap_remove snap 335 2022-04-22T17:37:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:30 smithi121 bash[36902]: cluster 2022-04-22T17:37:29.076582+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2022-04-22T17:37:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:30 smithi121 bash[36902]: audit 2022-04-22T17:37:29.426634+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:30 smithi121 bash[36902]: cluster 2022-04-22T17:37:29.982247+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2022-04-22T17:37:30.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[45495]: cluster 2022-04-22T17:37:29.076582+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2022-04-22T17:37:30.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[45495]: audit 2022-04-22T17:37:29.426634+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:30.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[45495]: cluster 2022-04-22T17:37:29.982247+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2022-04-22T17:37:30.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[46934]: cluster 2022-04-22T17:37:29.076582+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2022-04-22T17:37:30.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[46934]: audit 2022-04-22T17:37:29.426634+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:30.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:30 smithi115 bash[46934]: cluster 2022-04-22T17:37:29.982247+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2022-04-22T17:37:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:37:30.944 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:37:30.944 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (35s) 30s ago 14m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:37:30.944 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (14m) 71s ago 14m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:37:30.944 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2m) 30s ago 15m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:37:30.945 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2m) 71s ago 15m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:37:30.945 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (9m) 71s ago 19m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:37:30.945 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (10m) 30s ago 21m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:37:30.945 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (9m) 30s ago 21m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:37:30.945 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (8m) 71s ago 19m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:37:30.946 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 30s ago 20m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:37:30.946 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (100s) 30s ago 14m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:37:30.946 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (96s) 71s ago 14m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:37:30.946 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 30s ago 19m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:37:30.946 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6m) 30s ago 18m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:37:30.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6m) 30s ago 18m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:37:30.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5m) 30s ago 17m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:37:30.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5m) 71s ago 17m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:37:30.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4m) 71s ago 16m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:37:30.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3m) 71s ago 16m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:37:30.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2m) 71s ago 15m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:37:30.948 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (76s) 71s ago 14m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:37:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:3564: done (2 left) 2022-04-22T17:37:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 342 2022-04-22T17:37:31.010 INFO:tasks.rados.rados.0.smithi115.stdout:3565: seq_num 1081 ranges {7643136=507904,8151040=425984,8577024=376832} 2022-04-22T17:37:31.059 INFO:tasks.rados.rados.0.smithi115.stdout:3565: writing smithi11541303-25 from 7643136 to 8151040 tid 1 2022-04-22T17:37:31.061 INFO:tasks.rados.rados.0.smithi115.stdout:3565: writing smithi11541303-25 from 8151040 to 8577024 tid 2 2022-04-22T17:37:31.064 INFO:tasks.rados.rados.0.smithi115.stdout:3565: writing smithi11541303-25 from 8577024 to 8953856 tid 3 2022-04-22T17:37:31.064 INFO:tasks.rados.rados.0.smithi115.stdout:3566: snap_create 2022-04-22T17:37:31.064 INFO:tasks.rados.rados.0.smithi115.stdout:3563: finishing write tid 1 to smithi11541303-11 2022-04-22T17:37:31.064 INFO:tasks.rados.rados.0.smithi115.stdout:3563: finishing write tid 2 to smithi11541303-11 2022-04-22T17:37:31.064 INFO:tasks.rados.rados.0.smithi115.stdout:3563: finishing write tid 3 to smithi11541303-11 2022-04-22T17:37:31.065 INFO:tasks.rados.rados.0.smithi115.stdout:3563: finishing write tid 4 to smithi11541303-11 2022-04-22T17:37:31.065 INFO:tasks.rados.rados.0.smithi115.stdout:3563: finishing write tid 5 to smithi11541303-11 2022-04-22T17:37:31.065 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2080 (ObjNum 1080 snap 342 seq_num 1080) dirty exists 2022-04-22T17:37:31.065 INFO:tasks.rados.rados.0.smithi115.stdout:3563: left oid 11 (ObjNum 1080 snap 342 seq_num 1080) 2022-04-22T17:37:31.073 INFO:tasks.rados.rados.0.smithi115.stdout:3565: finishing write tid 1 to smithi11541303-25 2022-04-22T17:37:31.073 INFO:tasks.rados.rados.0.smithi115.stdout:3565: finishing write tid 2 to smithi11541303-25 2022-04-22T17:37:31.073 INFO:tasks.rados.rados.0.smithi115.stdout:3565: finishing write tid 3 to smithi11541303-25 2022-04-22T17:37:31.073 INFO:tasks.rados.rados.0.smithi115.stdout:3565: finishing write tid 4 to smithi11541303-25 2022-04-22T17:37:31.075 INFO:tasks.rados.rados.0.smithi115.stdout:3565: finishing write tid 5 to smithi11541303-25 2022-04-22T17:37:31.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 1956 (ObjNum 1081 snap 342 seq_num 1081) dirty exists 2022-04-22T17:37:31.075 INFO:tasks.rados.rados.0.smithi115.stdout:3565: left oid 25 (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:37:31.363 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:37:31.382 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:31.383 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:37:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:37:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:37:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:37:31.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:37:31.386 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:37:31.387 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:37:31.387 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:37:31.388 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:37:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:31 smithi121 bash[36902]: cluster 2022-04-22T17:37:29.964401+0000 mgr.y (mgr.34107) 445 : cluster [DBG] pgmap v812: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2022-04-22T17:37:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:31 smithi121 bash[36902]: cluster 2022-04-22T17:37:30.997651+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2022-04-22T17:37:31.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:31 smithi115 bash[45495]: cluster 2022-04-22T17:37:29.964401+0000 mgr.y (mgr.34107) 445 : cluster [DBG] pgmap v812: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2022-04-22T17:37:31.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:31 smithi115 bash[45495]: cluster 2022-04-22T17:37:30.997651+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2022-04-22T17:37:31.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:31 smithi115 bash[46934]: cluster 2022-04-22T17:37:29.964401+0000 mgr.y (mgr.34107) 445 : cluster [DBG] pgmap v812: 121 pgs: 121 active+clean; 184 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2022-04-22T17:37:31.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:31 smithi115 bash[46934]: cluster 2022-04-22T17:37:30.997651+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2022-04-22T17:37:32.012 INFO:tasks.rados.rados.0.smithi115.stdout:3562: done (3 left) 2022-04-22T17:37:32.012 INFO:tasks.rados.rados.0.smithi115.stdout:3563: done (2 left) 2022-04-22T17:37:32.013 INFO:tasks.rados.rados.0.smithi115.stdout:3565: done (1 left) 2022-04-22T17:37:32.013 INFO:tasks.rados.rados.0.smithi115.stdout:3566: done (0 left) 2022-04-22T17:37:32.013 INFO:tasks.rados.rados.0.smithi115.stdout:3567: snap_create 2022-04-22T17:37:32.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:31.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:32 smithi121 bash[36902]: audit 2022-04-22T17:37:30.564395+0000 mgr.y (mgr.34107) 446 : audit [DBG] from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:32 smithi121 bash[36902]: audit 2022-04-22T17:37:30.911786+0000 mgr.y (mgr.34107) 447 : audit [DBG] from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:32 smithi121 bash[36902]: audit 2022-04-22T17:37:31.350796+0000 mon.a (mon.0) 1436 : audit [DBG] from='client.? 172.21.15.115:0/3941246158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:32 smithi121 bash[36902]: cluster 2022-04-22T17:37:31.999411+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2022-04-22T17:37:32.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[45495]: audit 2022-04-22T17:37:30.564395+0000 mgr.y (mgr.34107) 446 : audit [DBG] from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[45495]: audit 2022-04-22T17:37:30.911786+0000 mgr.y (mgr.34107) 447 : audit [DBG] from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[45495]: audit 2022-04-22T17:37:31.350796+0000 mon.a (mon.0) 1436 : audit [DBG] from='client.? 172.21.15.115:0/3941246158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:32.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[45495]: cluster 2022-04-22T17:37:31.999411+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2022-04-22T17:37:32.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[46934]: audit 2022-04-22T17:37:30.564395+0000 mgr.y (mgr.34107) 446 : audit [DBG] from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[46934]: audit 2022-04-22T17:37:30.911786+0000 mgr.y (mgr.34107) 447 : audit [DBG] from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:37:32.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[46934]: audit 2022-04-22T17:37:31.350796+0000 mon.a (mon.0) 1436 : audit [DBG] from='client.? 172.21.15.115:0/3941246158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:37:32.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:32 smithi115 bash[46934]: cluster 2022-04-22T17:37:31.999411+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2022-04-22T17:37:33.014 INFO:tasks.rados.rados.0.smithi115.stdout:3567: done (0 left) 2022-04-22T17:37:33.014 INFO:tasks.rados.rados.0.smithi115.stdout:3568: rollback oid 29 current snap is 344 2022-04-22T17:37:33.014 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 29 to 343 2022-04-22T17:37:33.014 INFO:tasks.rados.rados.0.smithi115.stdout:3569: rmattr oid 15 current snap is 344 2022-04-22T17:37:33.014 INFO:tasks.rados.rados.0.smithi115.stdout:3569: done (1 left) 2022-04-22T17:37:33.015 INFO:tasks.rados.rados.0.smithi115.stdout:3570: rmattr oid 40 current snap is 344 2022-04-22T17:37:33.015 INFO:tasks.rados.rados.0.smithi115.stdout:3571: snap_remove snap 333 2022-04-22T17:37:33.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:33 smithi121 bash[36902]: cluster 2022-04-22T17:37:31.965361+0000 mgr.y (mgr.34107) 448 : cluster [DBG] pgmap v815: 121 pgs: 121 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.0 MiB/s wr, 8 op/s 2022-04-22T17:37:33.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:33 smithi121 bash[36902]: cluster 2022-04-22T17:37:33.001575+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2022-04-22T17:37:33.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:33 smithi115 bash[45495]: cluster 2022-04-22T17:37:31.965361+0000 mgr.y (mgr.34107) 448 : cluster [DBG] pgmap v815: 121 pgs: 121 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.0 MiB/s wr, 8 op/s 2022-04-22T17:37:33.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:33 smithi115 bash[45495]: cluster 2022-04-22T17:37:33.001575+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2022-04-22T17:37:33.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:33 smithi115 bash[46934]: cluster 2022-04-22T17:37:31.965361+0000 mgr.y (mgr.34107) 448 : cluster [DBG] pgmap v815: 121 pgs: 121 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 3.0 MiB/s wr, 8 op/s 2022-04-22T17:37:33.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:33 smithi115 bash[46934]: cluster 2022-04-22T17:37:33.001575+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2022-04-22T17:37:33.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:33.525Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:34.018 INFO:tasks.rados.rados.0.smithi115.stdout:3571: done (2 left) 2022-04-22T17:37:34.018 INFO:tasks.rados.rados.0.smithi115.stdout:3572: delete oid 39 current snap is 344 2022-04-22T17:37:34.018 INFO:tasks.rados.rados.0.smithi115.stdout:3568: finishing rollback tid 1 to smithi11541303-29 2022-04-22T17:37:34.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1900 (ObjNum -1116347600 snap 22075 seq_num 0) dirty dne 2022-04-22T17:37:34.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2058 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:37:34.032 INFO:tasks.rados.rados.0.smithi115.stdout:3568: done (2 left) 2022-04-22T17:37:34.032 INFO:tasks.rados.rados.0.smithi115.stdout:3570: done (1 left) 2022-04-22T17:37:34.032 INFO:tasks.rados.rados.0.smithi115.stdout:3572: done (0 left) 2022-04-22T17:37:34.032 INFO:tasks.rados.rados.0.smithi115.stdout:3573: snap_create 2022-04-22T17:37:34.997 INFO:tasks.rados.rados.0.smithi115.stdout:3573: done (0 left) 2022-04-22T17:37:34.998 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 345 2022-04-22T17:37:34.998 INFO:tasks.rados.rados.0.smithi115.stdout:3574: seq_num 1083 ranges {1294336=540672,1835008=589824,2424832=212992} 2022-04-22T17:37:35.008 INFO:tasks.rados.rados.0.smithi115.stdout:3574: writing smithi11541303-19 from 1294336 to 1835008 tid 1 2022-04-22T17:37:35.011 INFO:tasks.rados.rados.0.smithi115.stdout:3574: writing smithi11541303-19 from 1835008 to 2424832 tid 2 2022-04-22T17:37:35.013 INFO:tasks.rados.rados.0.smithi115.stdout:3574: writing smithi11541303-19 from 2424832 to 2637824 tid 3 2022-04-22T17:37:35.013 INFO:tasks.rados.rados.0.smithi115.stdout:3575: copy_from oid 20 from oid 49 current snap is 345 2022-04-22T17:37:35.013 INFO:tasks.rados.rados.0.smithi115.stdout:3576: snap_remove snap 341 2022-04-22T17:37:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:35 smithi121 bash[36902]: cluster 2022-04-22T17:37:33.966483+0000 mgr.y (mgr.34107) 449 : cluster [DBG] pgmap v818: 121 pgs: 121 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.6 MiB/s wr, 9 op/s 2022-04-22T17:37:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:35 smithi121 bash[36902]: cluster 2022-04-22T17:37:34.004604+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2022-04-22T17:37:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:35 smithi121 bash[36902]: audit 2022-04-22T17:37:34.429820+0000 mon.a (mon.0) 1440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:35 smithi121 bash[36902]: cluster 2022-04-22T17:37:34.983134+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2022-04-22T17:37:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[45495]: cluster 2022-04-22T17:37:33.966483+0000 mgr.y (mgr.34107) 449 : cluster [DBG] pgmap v818: 121 pgs: 121 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.6 MiB/s wr, 9 op/s 2022-04-22T17:37:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[45495]: cluster 2022-04-22T17:37:34.004604+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[45495]: audit 2022-04-22T17:37:34.429820+0000 mon.a (mon.0) 1440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[45495]: cluster 2022-04-22T17:37:34.983134+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[46934]: cluster 2022-04-22T17:37:33.966483+0000 mgr.y (mgr.34107) 449 : cluster [DBG] pgmap v818: 121 pgs: 121 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.6 MiB/s wr, 9 op/s 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[46934]: cluster 2022-04-22T17:37:34.004604+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[46934]: audit 2022-04-22T17:37:34.429820+0000 mon.a (mon.0) 1440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:35 smithi115 bash[46934]: cluster 2022-04-22T17:37:34.983134+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2022-04-22T17:37:36.022 INFO:tasks.rados.rados.0.smithi115.stdout:3576: done (2 left) 2022-04-22T17:37:36.022 INFO:tasks.rados.rados.0.smithi115.stdout:3577: rollback oid 39 current snap is 345 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 39 to 332 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: finishing write tid 1 to smithi11541303-19 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: finishing write tid 2 to smithi11541303-19 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: finishing write tid 3 to smithi11541303-19 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: finishing write tid 4 to smithi11541303-19 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: finishing write tid 5 to smithi11541303-19 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1795 (ObjNum 1083 snap 345 seq_num 1083) dirty exists 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3574: left oid 19 (ObjNum 1083 snap 345 seq_num 1083) 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:3575: finishing copy_from to smithi11541303-20 2022-04-22T17:37:36.023 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 2177 (ObjNum 1078 snap 339 seq_num 1078) dirty exists 2022-04-22T17:37:36.024 INFO:tasks.rados.rados.0.smithi115.stdout:3575: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:37:36.024 INFO:tasks.rados.rados.0.smithi115.stdout:3574: done (2 left) 2022-04-22T17:37:36.024 INFO:tasks.rados.rados.0.smithi115.stdout:3575: done (1 left) 2022-04-22T17:37:36.024 INFO:tasks.rados.rados.0.smithi115.stdout:3578: copy_from oid 27 from oid 46 current snap is 345 2022-04-22T17:37:36.024 INFO:tasks.rados.rados.0.smithi115.stdout:3579: delete oid 4 current snap is 345 2022-04-22T17:37:36.036 INFO:tasks.rados.rados.0.smithi115.stdout:3577: finishing rollback tid 1 to smithi11541303-39 2022-04-22T17:37:36.040 INFO:tasks.rados.rados.0.smithi115.stdout:3579: done (2 left) 2022-04-22T17:37:36.040 INFO:tasks.rados.rados.0.smithi115.stdout:3580: rollback oid 38 current snap is 345 2022-04-22T17:37:36.040 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 343 2022-04-22T17:37:36.040 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 345 2022-04-22T17:37:36.040 INFO:tasks.rados.rados.0.smithi115.stdout:3581: seq_num 1085 ranges {0=581632,581632=761856,1343488=40960} 2022-04-22T17:37:36.047 INFO:tasks.rados.rados.0.smithi115.stdout:3581: writing smithi11541303-23 from 0 to 581632 tid 1 2022-04-22T17:37:36.052 INFO:tasks.rados.rados.0.smithi115.stdout:3581: writing smithi11541303-23 from 581632 to 1343488 tid 2 2022-04-22T17:37:36.053 INFO:tasks.rados.rados.0.smithi115.stdout:3581: writing smithi11541303-23 from 1343488 to 1384448 tid 3 2022-04-22T17:37:36.053 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 345 2022-04-22T17:37:36.053 INFO:tasks.rados.rados.0.smithi115.stdout:3577: finishing rollback tid 2 to smithi11541303-39 2022-04-22T17:37:36.053 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 1939 (ObjNum 1025 snap 321 seq_num 1025) dirty exists 2022-04-22T17:37:36.053 INFO:tasks.rados.rados.0.smithi115.stdout:3582: seq_num 1086 ranges {0=598016,598016=647168,1245184=155648} 2022-04-22T17:37:36.056 INFO:tasks.rados.rados.0.smithi115.stdout:3582: writing smithi11541303-43 from 0 to 598016 tid 1 2022-04-22T17:37:36.060 INFO:tasks.rados.rados.0.smithi115.stdout:3582: writing smithi11541303-43 from 598016 to 1245184 tid 2 2022-04-22T17:37:36.061 INFO:tasks.rados.rados.0.smithi115.stdout:3582: writing smithi11541303-43 from 1245184 to 1400832 tid 3 2022-04-22T17:37:36.061 INFO:tasks.rados.rados.0.smithi115.stdout:3577: done (4 left) 2022-04-22T17:37:36.061 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 345 2022-04-22T17:37:36.061 INFO:tasks.rados.rados.0.smithi115.stdout:3583: seq_num 1087 ranges {0=614400,614400=532480,1146880=270336} 2022-04-22T17:37:36.064 INFO:tasks.rados.rados.0.smithi115.stdout:3583: writing smithi11541303-34 from 0 to 614400 tid 1 2022-04-22T17:37:36.067 INFO:tasks.rados.rados.0.smithi115.stdout:3583: writing smithi11541303-34 from 614400 to 1146880 tid 2 2022-04-22T17:37:36.069 INFO:tasks.rados.rados.0.smithi115.stdout:3583: writing smithi11541303-34 from 1146880 to 1417216 tid 3 2022-04-22T17:37:36.069 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 345 2022-04-22T17:37:36.069 INFO:tasks.rados.rados.0.smithi115.stdout:3584: seq_num 1088 ranges {0=630784,630784=466944,1097728=335872} 2022-04-22T17:37:36.074 INFO:tasks.rados.rados.0.smithi115.stdout:3584: writing smithi11541303-6 from 0 to 630784 tid 1 2022-04-22T17:37:36.077 INFO:tasks.rados.rados.0.smithi115.stdout:3584: writing smithi11541303-6 from 630784 to 1097728 tid 2 2022-04-22T17:37:36.081 INFO:tasks.rados.rados.0.smithi115.stdout:3584: writing smithi11541303-6 from 1097728 to 1433600 tid 3 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3585: delete oid 49 current snap is 345 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3578: finishing copy_from to smithi11541303-27 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2062 (ObjNum 1032 snap 323 seq_num 1032) dirty exists 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3578: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3580: finishing rollback tid 0 to smithi11541303-38 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3580: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:37:36.082 INFO:tasks.rados.rados.0.smithi115.stdout:3580: finishing rollback tid 2 to smithi11541303-38 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1798 (ObjNum 1024 snap 321 seq_num 1024) dirty exists 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: finishing write tid 1 to smithi11541303-23 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: finishing write tid 2 to smithi11541303-23 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: finishing write tid 3 to smithi11541303-23 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: finishing write tid 4 to smithi11541303-23 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: finishing write tid 5 to smithi11541303-23 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2084 (ObjNum 1085 snap 345 seq_num 1085) dirty exists 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3581: left oid 23 (ObjNum 1085 snap 345 seq_num 1085) 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3582: finishing write tid 1 to smithi11541303-43 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3582: finishing write tid 2 to smithi11541303-43 2022-04-22T17:37:36.083 INFO:tasks.rados.rados.0.smithi115.stdout:3582: finishing write tid 3 to smithi11541303-43 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3583: finishing write tid 1 to smithi11541303-34 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3582: finishing write tid 4 to smithi11541303-43 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3582: finishing write tid 5 to smithi11541303-43 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2181 (ObjNum 1086 snap 345 seq_num 1086) dirty exists 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3582: left oid 43 (ObjNum 1086 snap 345 seq_num 1086) 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3583: finishing write tid 2 to smithi11541303-34 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3583: finishing write tid 3 to smithi11541303-34 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3583: finishing write tid 4 to smithi11541303-34 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3584: finishing write tid 1 to smithi11541303-6 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:3583: finishing write tid 5 to smithi11541303-34 2022-04-22T17:37:36.084 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 1859 (ObjNum 1087 snap 345 seq_num 1087) dirty exists 2022-04-22T17:37:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:3583: left oid 34 (ObjNum 1087 snap 345 seq_num 1087) 2022-04-22T17:37:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:3584: finishing write tid 2 to smithi11541303-6 2022-04-22T17:37:36.085 INFO:tasks.rados.rados.0.smithi115.stdout:3584: finishing write tid 3 to smithi11541303-6 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3584: finishing write tid 4 to smithi11541303-6 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3578: done (6 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3580: done (5 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3581: done (4 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3582: done (3 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3583: done (2 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3585: done (1 left) 2022-04-22T17:37:36.086 INFO:tasks.rados.rados.0.smithi115.stdout:3586: delete oid 1 current snap is 345 2022-04-22T17:37:36.087 INFO:tasks.rados.rados.0.smithi115.stdout:3586: done (1 left) 2022-04-22T17:37:36.087 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 345 2022-04-22T17:37:36.088 INFO:tasks.rados.rados.0.smithi115.stdout:3587: seq_num 1091 ranges {0=679936,679936=516096,1196032=286720} 2022-04-22T17:37:36.094 INFO:tasks.rados.rados.0.smithi115.stdout:3587: writing smithi11541303-2 from 0 to 679936 tid 1 2022-04-22T17:37:36.097 INFO:tasks.rados.rados.0.smithi115.stdout:3587: writing smithi11541303-2 from 679936 to 1196032 tid 2 2022-04-22T17:37:36.098 INFO:tasks.rados.rados.0.smithi115.stdout:3587: writing smithi11541303-2 from 1196032 to 1482752 tid 3 2022-04-22T17:37:36.099 INFO:tasks.rados.rados.0.smithi115.stdout:3588: snap_create 2022-04-22T17:37:36.099 INFO:tasks.rados.rados.0.smithi115.stdout:3584: finishing write tid 5 to smithi11541303-6 2022-04-22T17:37:36.099 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1937 (ObjNum 1088 snap 345 seq_num 1088) dirty exists 2022-04-22T17:37:36.099 INFO:tasks.rados.rados.0.smithi115.stdout:3584: left oid 6 (ObjNum 1088 snap 345 seq_num 1088) 2022-04-22T17:37:36.099 INFO:tasks.rados.rados.0.smithi115.stdout:3587: finishing write tid 1 to smithi11541303-2 2022-04-22T17:37:36.101 INFO:tasks.rados.rados.0.smithi115.stdout:3587: finishing write tid 2 to smithi11541303-2 2022-04-22T17:37:36.103 INFO:tasks.rados.rados.0.smithi115.stdout:3587: finishing write tid 3 to smithi11541303-2 2022-04-22T17:37:36.103 INFO:tasks.rados.rados.0.smithi115.stdout:3587: finishing write tid 4 to smithi11541303-2 2022-04-22T17:37:36.106 INFO:tasks.rados.rados.0.smithi115.stdout:3587: finishing write tid 5 to smithi11541303-2 2022-04-22T17:37:36.106 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 1943 (ObjNum 1091 snap 345 seq_num 1091) dirty exists 2022-04-22T17:37:36.106 INFO:tasks.rados.rados.0.smithi115.stdout:3587: left oid 2 (ObjNum 1091 snap 345 seq_num 1091) 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3584: done (2 left) 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3587: done (1 left) 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3588: done (0 left) 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3589: read oid 25 snap -1 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3589: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:37:37.028 INFO:tasks.rados.rados.0.smithi115.stdout:3590: snap_remove snap 336 2022-04-22T17:37:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:37 smithi115 bash[45495]: cluster 2022-04-22T17:37:35.967614+0000 mgr.y (mgr.34107) 450 : cluster [DBG] pgmap v821: 121 pgs: 121 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 948 KiB/s wr, 7 op/s 2022-04-22T17:37:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:37 smithi115 bash[45495]: cluster 2022-04-22T17:37:36.009182+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2022-04-22T17:37:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:37 smithi115 bash[46934]: cluster 2022-04-22T17:37:35.967614+0000 mgr.y (mgr.34107) 450 : cluster [DBG] pgmap v821: 121 pgs: 121 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 948 KiB/s wr, 7 op/s 2022-04-22T17:37:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:37 smithi115 bash[46934]: cluster 2022-04-22T17:37:36.009182+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2022-04-22T17:37:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:37 smithi121 bash[36902]: cluster 2022-04-22T17:37:35.967614+0000 mgr.y (mgr.34107) 450 : cluster [DBG] pgmap v821: 121 pgs: 121 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 948 KiB/s wr, 7 op/s 2022-04-22T17:37:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:37 smithi121 bash[36902]: cluster 2022-04-22T17:37:36.009182+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2022-04-22T17:37:38.043 INFO:tasks.rados.rados.0.smithi115.stdout:3589: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:37:38.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:38 smithi115 bash[45495]: cluster 2022-04-22T17:37:37.015085+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2022-04-22T17:37:38.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:38 smithi115 bash[46934]: cluster 2022-04-22T17:37:37.015085+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2022-04-22T17:37:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:38 smithi121 bash[36902]: cluster 2022-04-22T17:37:37.015085+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2022-04-22T17:37:38.698 INFO:tasks.rados.rados.0.smithi115.stdout:3589: done (1 left) 2022-04-22T17:37:38.708 INFO:tasks.rados.rados.0.smithi115.stdout:3590: done (0 left) 2022-04-22T17:37:38.708 INFO:tasks.rados.rados.0.smithi115.stdout:3591: snap_remove snap 334 2022-04-22T17:37:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:3591: done (0 left) 2022-04-22T17:37:39.052 INFO:tasks.rados.rados.0.smithi115.stdout:3592: snap_create 2022-04-22T17:37:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:39 smithi115 bash[45495]: cluster 2022-04-22T17:37:37.968395+0000 mgr.y (mgr.34107) 451 : cluster [DBG] pgmap v824: 121 pgs: 121 active+clean; 212 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1005 KiB/s wr, 8 op/s 2022-04-22T17:37:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:39 smithi115 bash[45495]: cluster 2022-04-22T17:37:38.031265+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2022-04-22T17:37:39.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:39 smithi115 bash[46934]: cluster 2022-04-22T17:37:37.968395+0000 mgr.y (mgr.34107) 451 : cluster [DBG] pgmap v824: 121 pgs: 121 active+clean; 212 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1005 KiB/s wr, 8 op/s 2022-04-22T17:37:39.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:39 smithi115 bash[46934]: cluster 2022-04-22T17:37:38.031265+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2022-04-22T17:37:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:39 smithi121 bash[36902]: cluster 2022-04-22T17:37:37.968395+0000 mgr.y (mgr.34107) 451 : cluster [DBG] pgmap v824: 121 pgs: 121 active+clean; 212 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1005 KiB/s wr, 8 op/s 2022-04-22T17:37:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:39 smithi121 bash[36902]: cluster 2022-04-22T17:37:38.031265+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2022-04-22T17:37:39.999 INFO:tasks.rados.rados.0.smithi115.stdout:3592: done (0 left) 2022-04-22T17:37:39.999 INFO:tasks.rados.rados.0.smithi115.stdout:3593: snap_remove snap 338 2022-04-22T17:37:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:40 smithi121 bash[36902]: cluster 2022-04-22T17:37:39.040425+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2022-04-22T17:37:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:40 smithi121 bash[36902]: audit 2022-04-22T17:37:39.433001+0000 mon.a (mon.0) 1446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:40 smithi121 bash[36902]: cluster 2022-04-22T17:37:39.984504+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2022-04-22T17:37:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[45495]: cluster 2022-04-22T17:37:39.040425+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2022-04-22T17:37:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[45495]: audit 2022-04-22T17:37:39.433001+0000 mon.a (mon.0) 1446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[45495]: cluster 2022-04-22T17:37:39.984504+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2022-04-22T17:37:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[46934]: cluster 2022-04-22T17:37:39.040425+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2022-04-22T17:37:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[46934]: audit 2022-04-22T17:37:39.433001+0000 mon.a (mon.0) 1446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:40 smithi115 bash[46934]: cluster 2022-04-22T17:37:39.984504+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2022-04-22T17:37:41.001 INFO:tasks.rados.rados.0.smithi115.stdout:3593: done (0 left) 2022-04-22T17:37:41.002 INFO:tasks.rados.rados.0.smithi115.stdout:3594: read oid 32 snap -1 2022-04-22T17:37:41.002 INFO:tasks.rados.rados.0.smithi115.stdout:3594: expect (ObjNum 912 snap 299 seq_num 912) 2022-04-22T17:37:41.002 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 347 2022-04-22T17:37:41.002 INFO:tasks.rados.rados.0.smithi115.stdout:3595: seq_num 1092 ranges {3121152=696320,3817472=802816} 2022-04-22T17:37:41.025 INFO:tasks.rados.rados.0.smithi115.stdout:3595: writing smithi11541303-48 from 3121152 to 3817472 tid 1 2022-04-22T17:37:41.030 INFO:tasks.rados.rados.0.smithi115.stdout:3595: writing smithi11541303-48 from 3817472 to 4620288 tid 2 2022-04-22T17:37:41.030 INFO:tasks.rados.rados.0.smithi115.stdout:3596: rollback oid 43 current snap is 347 2022-04-22T17:37:41.030 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 343 2022-04-22T17:37:41.030 INFO:tasks.rados.rados.0.smithi115.stdout:3594: expect (ObjNum 912 snap 299 seq_num 912) 2022-04-22T17:37:41.197 INFO:tasks.rados.rados.0.smithi115.stdout:3595: finishing write tid 1 to smithi11541303-48 2022-04-22T17:37:41.197 INFO:tasks.rados.rados.0.smithi115.stdout:3596: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:37:41.197 INFO:tasks.rados.rados.0.smithi115.stdout:3596: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:37:41.197 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2183 (ObjNum 0 snap 32692 seq_num 3179398928) dirty dne 2022-04-22T17:37:41.197 INFO:tasks.rados.rados.0.smithi115.stdout:3595: finishing write tid 2 to smithi11541303-48 2022-04-22T17:37:41.198 INFO:tasks.rados.rados.0.smithi115.stdout:3595: finishing write tid 3 to smithi11541303-48 2022-04-22T17:37:41.198 INFO:tasks.rados.rados.0.smithi115.stdout:3595: finishing write tid 4 to smithi11541303-48 2022-04-22T17:37:41.198 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2070 (ObjNum 1092 snap 347 seq_num 1092) dirty exists 2022-04-22T17:37:41.198 INFO:tasks.rados.rados.0.smithi115.stdout:3595: left oid 48 (ObjNum 1092 snap 347 seq_num 1092) 2022-04-22T17:37:41.198 INFO:tasks.rados.rados.0.smithi115.stdout:3594: done (2 left) 2022-04-22T17:37:41.199 INFO:tasks.rados.rados.0.smithi115.stdout:3595: done (1 left) 2022-04-22T17:37:41.199 INFO:tasks.rados.rados.0.smithi115.stdout:3596: done (0 left) 2022-04-22T17:37:41.199 INFO:tasks.rados.rados.0.smithi115.stdout:3597: rollback oid 20 current snap is 347 2022-04-22T17:37:41.199 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 20 to 346 2022-04-22T17:37:41.199 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 347 2022-04-22T17:37:41.200 INFO:tasks.rados.rados.0.smithi115.stdout:3598: seq_num 1093 ranges {2949120=712704,3661824=688128,4349952=114688} 2022-04-22T17:37:41.224 INFO:tasks.rados.rados.0.smithi115.stdout:3598: writing smithi11541303-38 from 2949120 to 3661824 tid 1 2022-04-22T17:37:41.228 INFO:tasks.rados.rados.0.smithi115.stdout:3598: writing smithi11541303-38 from 3661824 to 4349952 tid 2 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3598: writing smithi11541303-38 from 4349952 to 4464640 tid 3 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3599: rollback oid 40 current snap is 347 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 324 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3597: finishing rollback tid 0 to smithi11541303-20 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3597: finishing rollback tid 1 to smithi11541303-20 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3597: finishing rollback tid 2 to smithi11541303-20 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 2186 (ObjNum 1078 snap 339 seq_num 1078) dirty exists 2022-04-22T17:37:41.229 INFO:tasks.rados.rados.0.smithi115.stdout:3597: done (2 left) 2022-04-22T17:37:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3600: setattr oid 4 current snap is 347 2022-04-22T17:37:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 347 2022-04-22T17:37:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3601: seq_num 1094 ranges {1384448=729088,2113536=573440,2686976=229376} 2022-04-22T17:37:41.242 INFO:tasks.rados.rados.0.smithi115.stdout:3601: writing smithi11541303-23 from 1384448 to 2113536 tid 1 2022-04-22T17:37:41.246 INFO:tasks.rados.rados.0.smithi115.stdout:3601: writing smithi11541303-23 from 2113536 to 2686976 tid 2 2022-04-22T17:37:41.248 INFO:tasks.rados.rados.0.smithi115.stdout:3601: writing smithi11541303-23 from 2686976 to 2916352 tid 3 2022-04-22T17:37:41.248 INFO:tasks.rados.rados.0.smithi115.stdout:3602: read oid 45 snap 345 2022-04-22T17:37:41.248 INFO:tasks.rados.rados.0.smithi115.stdout:3602: expect (ObjNum 1028 snap 322 seq_num 1028) 2022-04-22T17:37:41.248 INFO:tasks.rados.rados.0.smithi115.stdout:3603: rollback oid 49 current snap is 347 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 324 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2071 (ObjNum -1116555120 snap 22075 seq_num 3180218144) dirty exists 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3600: done (5 left) 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3604: delete oid 5 current snap is 347 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3598: finishing write tid 1 to smithi11541303-38 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3598: finishing write tid 2 to smithi11541303-38 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3599: finishing rollback tid 0 to smithi11541303-40 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3598: finishing write tid 3 to smithi11541303-38 2022-04-22T17:37:41.249 INFO:tasks.rados.rados.0.smithi115.stdout:3599: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:3598: finishing write tid 4 to smithi11541303-38 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:3599: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2061 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:3598: finishing write tid 5 to smithi11541303-38 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1802 (ObjNum 1093 snap 347 seq_num 1093) dirty exists 2022-04-22T17:37:41.250 INFO:tasks.rados.rados.0.smithi115.stdout:3598: left oid 38 (ObjNum 1093 snap 347 seq_num 1093) 2022-04-22T17:37:41.252 INFO:tasks.rados.rados.0.smithi115.stdout:3603: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:37:41.252 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2195 (ObjNum 0 snap 0 seq_num 0) dirty dne 2022-04-22T17:37:41.253 INFO:tasks.rados.rados.0.smithi115.stdout:3598: done (5 left) 2022-04-22T17:37:41.253 INFO:tasks.rados.rados.0.smithi115.stdout:3599: done (4 left) 2022-04-22T17:37:41.253 INFO:tasks.rados.rados.0.smithi115.stdout:3603: done (3 left) 2022-04-22T17:37:41.254 INFO:tasks.rados.rados.0.smithi115.stdout:3604: done (2 left) 2022-04-22T17:37:41.254 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 347 2022-04-22T17:37:41.254 INFO:tasks.rados.rados.0.smithi115.stdout:3605: seq_num 1096 ranges {11247616=761856,12009472=786432,12795904=16384} 2022-04-22T17:37:41.327 INFO:tasks.rados.rados.0.smithi115.stdout:3605: writing smithi11541303-7 from 11247616 to 12009472 tid 1 2022-04-22T17:37:41.339 INFO:tasks.rados.rados.0.smithi115.stdout:3605: writing smithi11541303-7 from 12009472 to 12795904 tid 2 2022-04-22T17:37:41.339 INFO:tasks.rados.rados.0.smithi115.stdout:3605: writing smithi11541303-7 from 12795904 to 12812288 tid 3 2022-04-22T17:37:41.339 INFO:tasks.rados.rados.0.smithi115.stdout:3606: rmattr oid 32 current snap is 347 2022-04-22T17:37:41.339 INFO:tasks.rados.rados.0.smithi115.stdout:3606: done (3 left) 2022-04-22T17:37:41.339 INFO:tasks.rados.rados.0.smithi115.stdout:3607: snap_remove snap 343 2022-04-22T17:37:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:41 smithi121 bash[36902]: cluster 2022-04-22T17:37:39.969295+0000 mgr.y (mgr.34107) 452 : cluster [DBG] pgmap v827: 121 pgs: 121 active+clean; 218 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:41 smithi121 bash[36902]: cluster 2022-04-22T17:37:40.988978+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2022-04-22T17:37:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:41 smithi115 bash[45495]: cluster 2022-04-22T17:37:39.969295+0000 mgr.y (mgr.34107) 452 : cluster [DBG] pgmap v827: 121 pgs: 121 active+clean; 218 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:41 smithi115 bash[45495]: cluster 2022-04-22T17:37:40.988978+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2022-04-22T17:37:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:41 smithi115 bash[46934]: cluster 2022-04-22T17:37:39.969295+0000 mgr.y (mgr.34107) 452 : cluster [DBG] pgmap v827: 121 pgs: 121 active+clean; 218 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2022-04-22T17:37:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:41 smithi115 bash[46934]: cluster 2022-04-22T17:37:40.988978+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2022-04-22T17:37:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:41.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:42.072 INFO:tasks.rados.rados.0.smithi115.stdout:3607: done (3 left) 2022-04-22T17:37:42.114 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 347 2022-04-22T17:37:42.114 INFO:tasks.rados.rados.0.smithi115.stdout:3601: finishing write tid 1 to smithi11541303-23 2022-04-22T17:37:42.114 INFO:tasks.rados.rados.0.smithi115.stdout:3608: seq_num 1097 ranges {2236416=778240,3014656=671744,3686400=131072} 2022-04-22T17:37:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3608: writing smithi11541303-21 from 2236416 to 3014656 tid 1 2022-04-22T17:37:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3608: writing smithi11541303-21 from 3014656 to 3686400 tid 2 2022-04-22T17:37:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3608: writing smithi11541303-21 from 3686400 to 3817472 tid 3 2022-04-22T17:37:42.115 INFO:tasks.rados.rados.0.smithi115.stdout:3609: snap_remove snap 324 2022-04-22T17:37:43.074 INFO:tasks.rados.rados.0.smithi115.stdout:3609: done (4 left) 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3610: snap_create 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3601: finishing write tid 2 to smithi11541303-23 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3601: finishing write tid 3 to smithi11541303-23 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3601: finishing write tid 4 to smithi11541303-23 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3601: finishing write tid 5 to smithi11541303-23 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2088 (ObjNum 1094 snap 347 seq_num 1094) dirty exists 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3601: left oid 23 (ObjNum 1094 snap 347 seq_num 1094) 2022-04-22T17:37:43.075 INFO:tasks.rados.rados.0.smithi115.stdout:3602: expect (ObjNum 1028 snap 322 seq_num 1028) 2022-04-22T17:37:43.360 INFO:tasks.rados.rados.0.smithi115.stdout:3605: finishing write tid 1 to smithi11541303-7 2022-04-22T17:37:43.360 INFO:tasks.rados.rados.0.smithi115.stdout:3605: finishing write tid 2 to smithi11541303-7 2022-04-22T17:37:43.360 INFO:tasks.rados.rados.0.smithi115.stdout:3605: finishing write tid 3 to smithi11541303-7 2022-04-22T17:37:43.360 INFO:tasks.rados.rados.0.smithi115.stdout:3605: finishing write tid 4 to smithi11541303-7 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3605: finishing write tid 5 to smithi11541303-7 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1952 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3605: left oid 7 (ObjNum 1096 snap 347 seq_num 1096) 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: finishing write tid 1 to smithi11541303-21 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: finishing write tid 2 to smithi11541303-21 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: finishing write tid 3 to smithi11541303-21 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: finishing write tid 4 to smithi11541303-21 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: finishing write tid 5 to smithi11541303-21 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1865 (ObjNum 1097 snap 347 seq_num 1097) dirty exists 2022-04-22T17:37:43.361 INFO:tasks.rados.rados.0.smithi115.stdout:3608: left oid 21 (ObjNum 1097 snap 347 seq_num 1097) 2022-04-22T17:37:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:43 smithi121 bash[36902]: cluster 2022-04-22T17:37:41.970290+0000 mgr.y (mgr.34107) 453 : cluster [DBG] pgmap v830: 121 pgs: 121 active+clean; 222 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 9 op/s 2022-04-22T17:37:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:43 smithi121 bash[36902]: cluster 2022-04-22T17:37:42.059939+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2022-04-22T17:37:43.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:43 smithi115 bash[46934]: cluster 2022-04-22T17:37:41.970290+0000 mgr.y (mgr.34107) 453 : cluster [DBG] pgmap v830: 121 pgs: 121 active+clean; 222 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 9 op/s 2022-04-22T17:37:43.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:43 smithi115 bash[46934]: cluster 2022-04-22T17:37:42.059939+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2022-04-22T17:37:43.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:43 smithi115 bash[45495]: cluster 2022-04-22T17:37:41.970290+0000 mgr.y (mgr.34107) 453 : cluster [DBG] pgmap v830: 121 pgs: 121 active+clean; 222 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 3.2 MiB/s wr, 9 op/s 2022-04-22T17:37:43.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:43 smithi115 bash[45495]: cluster 2022-04-22T17:37:42.059939+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2022-04-22T17:37:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:43.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:44.081 INFO:tasks.rados.rados.0.smithi115.stdout:3601: done (4 left) 2022-04-22T17:37:44.081 INFO:tasks.rados.rados.0.smithi115.stdout:3602: done (3 left) 2022-04-22T17:37:44.081 INFO:tasks.rados.rados.0.smithi115.stdout:3605: done (2 left) 2022-04-22T17:37:44.082 INFO:tasks.rados.rados.0.smithi115.stdout:3608: done (1 left) 2022-04-22T17:37:44.082 INFO:tasks.rados.rados.0.smithi115.stdout:3610: done (0 left) 2022-04-22T17:37:44.082 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 348 2022-04-22T17:37:44.082 INFO:tasks.rados.rados.0.smithi115.stdout:3611: seq_num 1098 ranges {0=794624,794624=557056,1351680=245760} 2022-04-22T17:37:44.087 INFO:tasks.rados.rados.0.smithi115.stdout:3611: writing smithi11541303-1 from 0 to 794624 tid 1 2022-04-22T17:37:44.090 INFO:tasks.rados.rados.0.smithi115.stdout:3611: writing smithi11541303-1 from 794624 to 1351680 tid 2 2022-04-22T17:37:44.092 INFO:tasks.rados.rados.0.smithi115.stdout:3611: writing smithi11541303-1 from 1351680 to 1597440 tid 3 2022-04-22T17:37:44.092 INFO:tasks.rados.rados.0.smithi115.stdout:3612: copy_from oid 2 from oid 33 current snap is 348 2022-04-22T17:37:44.092 INFO:tasks.rados.rados.0.smithi115.stdout:3613: read oid 31 snap -1 2022-04-22T17:37:44.093 INFO:tasks.rados.rados.0.smithi115.stdout:3613: expect (ObjNum 1070 snap 337 seq_num 1070) 2022-04-22T17:37:44.093 INFO:tasks.rados.rados.0.smithi115.stdout:3614: snap_remove snap 342 2022-04-22T17:37:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:44 smithi121 bash[36902]: cluster 2022-04-22T17:37:43.061061+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2022-04-22T17:37:44.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:44 smithi115 bash[45495]: cluster 2022-04-22T17:37:43.061061+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2022-04-22T17:37:44.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:44 smithi115 bash[46934]: cluster 2022-04-22T17:37:43.061061+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2022-04-22T17:37:45.003 INFO:tasks.rados.rados.0.smithi115.stdout:3614: done (3 left) 2022-04-22T17:37:45.003 INFO:tasks.rados.rados.0.smithi115.stdout:3615: copy_from oid 7 from oid 37 current snap is 348 2022-04-22T17:37:45.003 INFO:tasks.rados.rados.0.smithi115.stdout:3616: delete oid 29 current snap is 348 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: finishing write tid 1 to smithi11541303-1 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: finishing write tid 2 to smithi11541303-1 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: finishing write tid 3 to smithi11541303-1 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: finishing write tid 4 to smithi11541303-1 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: finishing write tid 5 to smithi11541303-1 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1956 (ObjNum 1098 snap 348 seq_num 1098) dirty exists 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3611: left oid 1 (ObjNum 1098 snap 348 seq_num 1098) 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:3612: finishing copy_from to smithi11541303-2 2022-04-22T17:37:45.004 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 2035 (ObjNum 1049 snap 329 seq_num 1049) dirty exists 2022-04-22T17:37:45.005 INFO:tasks.rados.rados.0.smithi115.stdout:3612: finishing copy_from racing read to smithi11541303-2 2022-04-22T17:37:45.005 INFO:tasks.rados.rados.0.smithi115.stdout:3613: expect (ObjNum 1070 snap 337 seq_num 1070) 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3615: finishing copy_from to smithi11541303-7 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3615: got expected ENOENT (src dne) 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3615: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3611: done (4 left) 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3612: done (3 left) 2022-04-22T17:37:45.274 INFO:tasks.rados.rados.0.smithi115.stdout:3613: done (2 left) 2022-04-22T17:37:45.275 INFO:tasks.rados.rados.0.smithi115.stdout:3615: done (1 left) 2022-04-22T17:37:45.275 INFO:tasks.rados.rados.0.smithi115.stdout:3616: done (0 left) 2022-04-22T17:37:45.275 INFO:tasks.rados.rados.0.smithi115.stdout:3617: snap_create 2022-04-22T17:37:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:45 smithi121 bash[36902]: cluster 2022-04-22T17:37:43.971274+0000 mgr.y (mgr.34107) 454 : cluster [DBG] pgmap v833: 121 pgs: 121 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 6 op/s 2022-04-22T17:37:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:45 smithi121 bash[36902]: cluster 2022-04-22T17:37:44.068234+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2022-04-22T17:37:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:45 smithi121 bash[36902]: audit 2022-04-22T17:37:44.435997+0000 mon.a (mon.0) 1452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:45 smithi121 bash[36902]: cluster 2022-04-22T17:37:44.988724+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2022-04-22T17:37:45.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[46934]: cluster 2022-04-22T17:37:43.971274+0000 mgr.y (mgr.34107) 454 : cluster [DBG] pgmap v833: 121 pgs: 121 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 6 op/s 2022-04-22T17:37:45.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[46934]: cluster 2022-04-22T17:37:44.068234+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[46934]: audit 2022-04-22T17:37:44.435997+0000 mon.a (mon.0) 1452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[46934]: cluster 2022-04-22T17:37:44.988724+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[45495]: cluster 2022-04-22T17:37:43.971274+0000 mgr.y (mgr.34107) 454 : cluster [DBG] pgmap v833: 121 pgs: 121 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 6 op/s 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[45495]: cluster 2022-04-22T17:37:44.068234+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[45495]: audit 2022-04-22T17:37:44.435997+0000 mon.a (mon.0) 1452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:45 smithi115 bash[45495]: cluster 2022-04-22T17:37:44.988724+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2022-04-22T17:37:46.098 INFO:tasks.rados.rados.0.smithi115.stdout:3617: done (0 left) 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3618: read oid 47 snap 346 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3618: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3619: read oid 2 snap -1 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3619: expect (ObjNum 1049 snap 329 seq_num 1049) 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3620: copy_from oid 50 from oid 10 current snap is 349 2022-04-22T17:37:46.099 INFO:tasks.rados.rados.0.smithi115.stdout:3621: snap_remove snap 348 2022-04-22T17:37:47.101 INFO:tasks.rados.rados.0.smithi115.stdout:3621: done (3 left) 2022-04-22T17:37:47.101 INFO:tasks.rados.rados.0.smithi115.stdout:3622: snap_remove snap 345 2022-04-22T17:37:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:47 smithi121 bash[36902]: cluster 2022-04-22T17:37:45.972386+0000 mgr.y (mgr.34107) 455 : cluster [DBG] pgmap v836: 121 pgs: 121 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2022-04-22T17:37:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:47 smithi121 bash[36902]: cluster 2022-04-22T17:37:46.084796+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2022-04-22T17:37:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:47 smithi115 bash[45495]: cluster 2022-04-22T17:37:45.972386+0000 mgr.y (mgr.34107) 455 : cluster [DBG] pgmap v836: 121 pgs: 121 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2022-04-22T17:37:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:47 smithi115 bash[45495]: cluster 2022-04-22T17:37:46.084796+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2022-04-22T17:37:47.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:47 smithi115 bash[46934]: cluster 2022-04-22T17:37:45.972386+0000 mgr.y (mgr.34107) 455 : cluster [DBG] pgmap v836: 121 pgs: 121 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2022-04-22T17:37:47.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:47 smithi115 bash[46934]: cluster 2022-04-22T17:37:46.084796+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2022-04-22T17:37:48.108 INFO:tasks.rados.rados.0.smithi115.stdout:3622: done (3 left) 2022-04-22T17:37:48.157 INFO:tasks.rados.rados.0.smithi115.stdout:3623: rollback oid 24 current snap is 349 2022-04-22T17:37:48.157 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 24 to 347 2022-04-22T17:37:48.157 INFO:tasks.rados.rados.0.smithi115.stdout:3624: delete oid 35 current snap is 349 2022-04-22T17:37:48.158 INFO:tasks.rados.rados.0.smithi115.stdout:3619: expect (ObjNum 1049 snap 329 seq_num 1049) 2022-04-22T17:37:48.232 INFO:tasks.rados.rados.0.smithi115.stdout:3618: expect (ObjNum 822 snap 274 seq_num 822) 2022-04-22T17:37:48.347 INFO:tasks.rados.rados.0.smithi115.stdout:3620: finishing copy_from to smithi11541303-50 2022-04-22T17:37:48.347 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2052 (ObjNum 1064 snap 337 seq_num 1064) dirty exists 2022-04-22T17:37:48.347 INFO:tasks.rados.rados.0.smithi115.stdout:3620: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:3623: finishing rollback tid 0 to smithi11541303-24 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:3623: finishing rollback tid 1 to smithi11541303-24 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:3623: finishing rollback tid 2 to smithi11541303-24 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 24 v 1940 (ObjNum 939 snap 304 seq_num 939) dirty exists 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:3618: done (4 left) 2022-04-22T17:37:48.348 INFO:tasks.rados.rados.0.smithi115.stdout:3619: done (3 left) 2022-04-22T17:37:48.349 INFO:tasks.rados.rados.0.smithi115.stdout:3620: done (2 left) 2022-04-22T17:37:48.349 INFO:tasks.rados.rados.0.smithi115.stdout:3623: done (1 left) 2022-04-22T17:37:48.349 INFO:tasks.rados.rados.0.smithi115.stdout:3624: done (0 left) 2022-04-22T17:37:48.349 INFO:tasks.rados.rados.0.smithi115.stdout:3625: snap_create 2022-04-22T17:37:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:48 smithi121 bash[36902]: cluster 2022-04-22T17:37:47.089326+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2022-04-22T17:37:48.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:48 smithi115 bash[45495]: cluster 2022-04-22T17:37:47.089326+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2022-04-22T17:37:48.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:48 smithi115 bash[46934]: cluster 2022-04-22T17:37:47.089326+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2022-04-22T17:37:49.134 INFO:tasks.rados.rados.0.smithi115.stdout:3625: done (0 left) 2022-04-22T17:37:49.134 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 350 2022-04-22T17:37:49.135 INFO:tasks.rados.rados.0.smithi115.stdout:3626: seq_num 1101 ranges {5013504=442368,5455872=606208,6062080=598016} 2022-04-22T17:37:49.169 INFO:tasks.rados.rados.0.smithi115.stdout:3626: writing smithi11541303-31 from 5013504 to 5455872 tid 1 2022-04-22T17:37:49.172 INFO:tasks.rados.rados.0.smithi115.stdout:3626: writing smithi11541303-31 from 5455872 to 6062080 tid 2 2022-04-22T17:37:49.175 INFO:tasks.rados.rados.0.smithi115.stdout:3626: writing smithi11541303-31 from 6062080 to 6660096 tid 3 2022-04-22T17:37:49.175 INFO:tasks.rados.rados.0.smithi115.stdout:3627: delete oid 30 current snap is 350 2022-04-22T17:37:49.185 INFO:tasks.rados.rados.0.smithi115.stdout:3627: done (1 left) 2022-04-22T17:37:49.185 INFO:tasks.rados.rados.0.smithi115.stdout:3628: delete oid 22 current snap is 350 2022-04-22T17:37:49.187 INFO:tasks.rados.rados.0.smithi115.stdout:3626: finishing write tid 1 to smithi11541303-31 2022-04-22T17:37:49.187 INFO:tasks.rados.rados.0.smithi115.stdout:3626: finishing write tid 2 to smithi11541303-31 2022-04-22T17:37:49.189 INFO:tasks.rados.rados.0.smithi115.stdout:3626: finishing write tid 3 to smithi11541303-31 2022-04-22T17:37:49.189 INFO:tasks.rados.rados.0.smithi115.stdout:3626: finishing write tid 4 to smithi11541303-31 2022-04-22T17:37:49.195 INFO:tasks.rados.rados.0.smithi115.stdout:3628: done (1 left) 2022-04-22T17:37:49.196 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 350 2022-04-22T17:37:49.196 INFO:tasks.rados.rados.0.smithi115.stdout:3629: seq_num 1104 ranges {1417216=499712,1916928=712704,2629632=491520} 2022-04-22T17:37:49.203 INFO:tasks.rados.rados.0.smithi115.stdout:3629: writing smithi11541303-34 from 1417216 to 1916928 tid 1 2022-04-22T17:37:49.207 INFO:tasks.rados.rados.0.smithi115.stdout:3629: writing smithi11541303-34 from 1916928 to 2629632 tid 2 2022-04-22T17:37:49.211 INFO:tasks.rados.rados.0.smithi115.stdout:3629: writing smithi11541303-34 from 2629632 to 3121152 tid 3 2022-04-22T17:37:49.211 INFO:tasks.rados.rados.0.smithi115.stdout:3630: snap_remove snap 346 2022-04-22T17:37:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:49 smithi121 bash[36902]: cluster 2022-04-22T17:37:47.973224+0000 mgr.y (mgr.34107) 456 : cluster [DBG] pgmap v839: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:37:49.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:49 smithi121 bash[36902]: cluster 2022-04-22T17:37:48.091951+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2022-04-22T17:37:49.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:49 smithi115 bash[45495]: cluster 2022-04-22T17:37:47.973224+0000 mgr.y (mgr.34107) 456 : cluster [DBG] pgmap v839: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:37:49.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:49 smithi115 bash[45495]: cluster 2022-04-22T17:37:48.091951+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2022-04-22T17:37:49.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:49 smithi115 bash[46934]: cluster 2022-04-22T17:37:47.973224+0000 mgr.y (mgr.34107) 456 : cluster [DBG] pgmap v839: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2022-04-22T17:37:49.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:49 smithi115 bash[46934]: cluster 2022-04-22T17:37:48.091951+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3630: done (2 left) 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3631: snap_create 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3626: finishing write tid 5 to smithi11541303-31 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2226 (ObjNum 1101 snap 350 seq_num 1101) dirty exists 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3626: left oid 31 (ObjNum 1101 snap 350 seq_num 1101) 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3629: finishing write tid 1 to smithi11541303-34 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3629: finishing write tid 2 to smithi11541303-34 2022-04-22T17:37:50.011 INFO:tasks.rados.rados.0.smithi115.stdout:3629: finishing write tid 3 to smithi11541303-34 2022-04-22T17:37:50.012 INFO:tasks.rados.rados.0.smithi115.stdout:3629: finishing write tid 4 to smithi11541303-34 2022-04-22T17:37:50.012 INFO:tasks.rados.rados.0.smithi115.stdout:3629: finishing write tid 5 to smithi11541303-34 2022-04-22T17:37:50.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 2056 (ObjNum 1104 snap 350 seq_num 1104) dirty exists 2022-04-22T17:37:50.012 INFO:tasks.rados.rados.0.smithi115.stdout:3629: left oid 34 (ObjNum 1104 snap 350 seq_num 1104) 2022-04-22T17:37:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:50 smithi121 bash[36902]: cluster 2022-04-22T17:37:49.121037+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2022-04-22T17:37:50.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:50 smithi121 bash[36902]: audit 2022-04-22T17:37:49.439286+0000 mon.a (mon.0) 1458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:50.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:50 smithi121 bash[36902]: cluster 2022-04-22T17:37:49.995834+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2022-04-22T17:37:50.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[45495]: cluster 2022-04-22T17:37:49.121037+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2022-04-22T17:37:50.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[45495]: audit 2022-04-22T17:37:49.439286+0000 mon.a (mon.0) 1458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:50.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[45495]: cluster 2022-04-22T17:37:49.995834+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2022-04-22T17:37:50.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[46934]: cluster 2022-04-22T17:37:49.121037+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2022-04-22T17:37:50.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[46934]: audit 2022-04-22T17:37:49.439286+0000 mon.a (mon.0) 1458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:50.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:50 smithi115 bash[46934]: cluster 2022-04-22T17:37:49.995834+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2022-04-22T17:37:51.006 INFO:tasks.rados.rados.0.smithi115.stdout:3626: done (2 left) 2022-04-22T17:37:51.006 INFO:tasks.rados.rados.0.smithi115.stdout:3629: done (1 left) 2022-04-22T17:37:51.006 INFO:tasks.rados.rados.0.smithi115.stdout:3631: done (0 left) 2022-04-22T17:37:51.007 INFO:tasks.rados.rados.0.smithi115.stdout:3632: rollback oid 21 current snap is 351 2022-04-22T17:37:51.007 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 344 2022-04-22T17:37:51.007 INFO:tasks.rados.rados.0.smithi115.stdout:3633: rollback oid 38 current snap is 351 2022-04-22T17:37:51.007 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 339 2022-04-22T17:37:51.007 INFO:tasks.rados.rados.0.smithi115.stdout:3634: snap_create 2022-04-22T17:37:51.013 INFO:tasks.rados.rados.0.smithi115.stdout:3633: finishing rollback tid 0 to smithi11541303-38 2022-04-22T17:37:51.015 INFO:tasks.rados.rados.0.smithi115.stdout:3633: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:37:51.015 INFO:tasks.rados.rados.0.smithi115.stdout:3633: finishing rollback tid 2 to smithi11541303-38 2022-04-22T17:37:51.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 1805 (ObjNum 1024 snap 321 seq_num 1024) dirty exists 2022-04-22T17:37:51.016 INFO:tasks.rados.rados.0.smithi115.stdout:3632: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:37:51.017 INFO:tasks.rados.rados.0.smithi115.stdout:3632: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:37:51.017 INFO:tasks.rados.rados.0.smithi115.stdout:3632: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:37:51.017 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1869 (ObjNum 886 snap 289 seq_num 886) dirty exists 2022-04-22T17:37:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:51 smithi121 bash[36902]: cluster 2022-04-22T17:37:49.974228+0000 mgr.y (mgr.34107) 457 : cluster [DBG] pgmap v842: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:37:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:51 smithi121 bash[36902]: cluster 2022-04-22T17:37:50.994068+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2022-04-22T17:37:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:51 smithi115 bash[45495]: cluster 2022-04-22T17:37:49.974228+0000 mgr.y (mgr.34107) 457 : cluster [DBG] pgmap v842: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:37:51.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:51 smithi115 bash[45495]: cluster 2022-04-22T17:37:50.994068+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2022-04-22T17:37:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:51 smithi115 bash[46934]: cluster 2022-04-22T17:37:49.974228+0000 mgr.y (mgr.34107) 457 : cluster [DBG] pgmap v842: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:37:51.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:51 smithi115 bash[46934]: cluster 2022-04-22T17:37:50.994068+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3632: done (2 left) 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3633: done (1 left) 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3634: done (0 left) 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3635: read oid 9 snap 351 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3635: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:37:52.009 INFO:tasks.rados.rados.0.smithi115.stdout:3636: delete oid 37 current snap is 352 2022-04-22T17:37:52.012 INFO:tasks.rados.rados.0.smithi115.stdout:3636: done (1 left) 2022-04-22T17:37:52.012 INFO:tasks.rados.rados.0.smithi115.stdout:3637: rollback oid 8 current snap is 352 2022-04-22T17:37:52.012 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 344 2022-04-22T17:37:52.012 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 352 2022-04-22T17:37:52.012 INFO:tasks.rados.rados.0.smithi115.stdout:3638: seq_num 1106 ranges {3031040=532480,3563520=483328,4046848=475136,4521984=245760} 2022-04-22T17:37:52.038 INFO:tasks.rados.rados.0.smithi115.stdout:3638: writing smithi11541303-32 from 3031040 to 3563520 tid 1 2022-04-22T17:37:52.038 INFO:tasks.rados.rados.0.smithi115.stdout:3638: writing smithi11541303-32 from 3563520 to 4046848 tid 2 2022-04-22T17:37:52.047 INFO:tasks.rados.rados.0.smithi115.stdout:3638: writing smithi11541303-32 from 4046848 to 4521984 tid 3 2022-04-22T17:37:52.047 INFO:tasks.rados.rados.0.smithi115.stdout:3638: writing smithi11541303-32 from 4521984 to 4767744 tid 4 2022-04-22T17:37:52.047 INFO:tasks.rados.rados.0.smithi115.stdout:3639: rmattr oid 7 current snap is 352 2022-04-22T17:37:52.047 INFO:tasks.rados.rados.0.smithi115.stdout:3637: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:37:52.048 INFO:tasks.rados.rados.0.smithi115.stdout:3640: copy_from oid 43 from oid 41 current snap is 352 2022-04-22T17:37:52.048 INFO:tasks.rados.rados.0.smithi115.stdout:3637: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:37:52.048 INFO:tasks.rados.rados.0.smithi115.stdout:3637: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:37:52.048 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1959 (ObjNum 1045 snap 328 seq_num 1045) dirty exists 2022-04-22T17:37:52.048 INFO:tasks.rados.rados.0.smithi115.stdout:3635: expect (ObjNum 989 snap 314 seq_num 989) 2022-04-22T17:37:52.052 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:51.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:52.109 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 1 to smithi11541303-32 2022-04-22T17:37:52.109 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 2 to smithi11541303-32 2022-04-22T17:37:52.109 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 3 to smithi11541303-32 2022-04-22T17:37:52.109 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 4 to smithi11541303-32 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 1960 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 5 to smithi11541303-32 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:3638: finishing write tid 6 to smithi11541303-32 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2043 (ObjNum 1106 snap 352 seq_num 1106) dirty exists 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:3638: left oid 32 (ObjNum 1106 snap 352 seq_num 1106) 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:3640: finishing copy_from to smithi11541303-43 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2056 (ObjNum 1077 snap 339 seq_num 1077) dirty exists 2022-04-22T17:37:52.110 INFO:tasks.rados.rados.0.smithi115.stdout:3640: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3635: done (4 left) 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3637: done (3 left) 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3638: done (2 left) 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3639: done (1 left) 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3640: done (0 left) 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 2 current snap is 352 2022-04-22T17:37:52.111 INFO:tasks.rados.rados.0.smithi115.stdout:3641: seq_num 1107 ranges {3088384=548864,3637248=770048,4407296=434176} 2022-04-22T17:37:52.139 INFO:tasks.rados.rados.0.smithi115.stdout:3641: writing smithi11541303-2 from 3088384 to 3637248 tid 1 2022-04-22T17:37:52.143 INFO:tasks.rados.rados.0.smithi115.stdout:3641: writing smithi11541303-2 from 3637248 to 4407296 tid 2 2022-04-22T17:37:52.146 INFO:tasks.rados.rados.0.smithi115.stdout:3641: writing smithi11541303-2 from 4407296 to 4841472 tid 3 2022-04-22T17:37:52.146 INFO:tasks.rados.rados.0.smithi115.stdout:3642: delete oid 9 current snap is 352 2022-04-22T17:37:52.147 INFO:tasks.rados.rados.0.smithi115.stdout:3641: finishing write tid 1 to smithi11541303-2 2022-04-22T17:37:52.151 INFO:tasks.rados.rados.0.smithi115.stdout:3641: finishing write tid 2 to smithi11541303-2 2022-04-22T17:37:52.151 INFO:tasks.rados.rados.0.smithi115.stdout:3642: done (1 left) 2022-04-22T17:37:52.152 INFO:tasks.rados.rados.0.smithi115.stdout:3643: rollback oid 10 current snap is 352 2022-04-22T17:37:52.152 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 10 to 351 2022-04-22T17:37:52.152 INFO:tasks.rados.rados.0.smithi115.stdout:3644: setattr oid 21 current snap is 352 2022-04-22T17:37:52.152 INFO:tasks.rados.rados.0.smithi115.stdout:3641: finishing write tid 3 to smithi11541303-2 2022-04-22T17:37:52.153 INFO:tasks.rados.rados.0.smithi115.stdout:3641: finishing write tid 4 to smithi11541303-2 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3645: read oid 4 snap -1 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3645: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3646: rollback oid 40 current snap is 352 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 347 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3647: rollback oid 9 current snap is 352 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 349 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3648: read oid 22 snap 344 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3648: expect (ObjNum 1062 snap 336 seq_num 1062) 2022-04-22T17:37:52.154 INFO:tasks.rados.rados.0.smithi115.stdout:3649: copy_from oid 38 from oid 31 current snap is 352 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3650: read oid 20 snap -1 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3650: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3651: read oid 42 snap 332 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3651: expect (ObjNum 1041 snap 326 seq_num 1041) 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3652: copy_from oid 19 from oid 7 current snap is 352 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3653: copy_from oid 11 from oid 16 current snap is 352 2022-04-22T17:37:52.155 INFO:tasks.rados.rados.0.smithi115.stdout:3654: snap_create 2022-04-22T17:37:52.156 INFO:tasks.rados.rados.0.smithi115.stdout:3645: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:37:52.157 INFO:tasks.rados.rados.0.smithi115.stdout:3641: finishing write tid 5 to smithi11541303-2 2022-04-22T17:37:52.157 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 2040 (ObjNum 1107 snap 352 seq_num 1107) dirty exists 2022-04-22T17:37:52.157 INFO:tasks.rados.rados.0.smithi115.stdout:3641: left oid 2 (ObjNum 1107 snap 352 seq_num 1107) 2022-04-22T17:37:52.158 INFO:tasks.rados.rados.0.smithi115.stdout:3653: finishing copy_from to smithi11541303-11 2022-04-22T17:37:52.158 INFO:tasks.rados.rados.0.smithi115.stdout:3653: got expected ENOENT (src dne) 2022-04-22T17:37:52.161 INFO:tasks.rados.rados.0.smithi115.stdout:3653: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:37:52.162 INFO:tasks.rados.rados.0.smithi115.stdout:3647: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:37:52.162 INFO:tasks.rados.rados.0.smithi115.stdout:3647: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:37:52.162 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2232 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:37:52.163 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1870 (ObjNum 886 snap 289 seq_num 886) dirty exists 2022-04-22T17:37:52.164 INFO:tasks.rados.rados.0.smithi115.stdout:3643: finishing rollback tid 0 to smithi11541303-10 2022-04-22T17:37:52.164 INFO:tasks.rados.rados.0.smithi115.stdout:3643: finishing rollback tid 1 to smithi11541303-10 2022-04-22T17:37:52.164 INFO:tasks.rados.rados.0.smithi115.stdout:3643: finishing rollback tid 2 to smithi11541303-10 2022-04-22T17:37:52.165 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2064 (ObjNum 1064 snap 337 seq_num 1064) dirty exists 2022-04-22T17:37:52.165 INFO:tasks.rados.rados.0.smithi115.stdout:3646: finishing rollback tid 0 to smithi11541303-40 2022-04-22T17:37:52.170 INFO:tasks.rados.rados.0.smithi115.stdout:3646: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:37:52.170 INFO:tasks.rados.rados.0.smithi115.stdout:3646: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:37:52.171 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2067 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:37:52.182 INFO:tasks.rados.rados.0.smithi115.stdout:3651: expect (ObjNum 1041 snap 326 seq_num 1041) 2022-04-22T17:37:52.275 INFO:tasks.rados.rados.0.smithi115.stdout:3650: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:37:52.465 INFO:tasks.rados.rados.0.smithi115.stdout:3648: expect (ObjNum 1062 snap 336 seq_num 1062) 2022-04-22T17:37:52.979 INFO:tasks.rados.rados.0.smithi115.stdout:3649: finishing copy_from to smithi11541303-38 2022-04-22T17:37:52.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2226 (ObjNum 1101 snap 350 seq_num 1101) dirty exists 2022-04-22T17:37:52.979 INFO:tasks.rados.rados.0.smithi115.stdout:3649: finishing copy_from racing read to smithi11541303-38 2022-04-22T17:37:52.979 INFO:tasks.rados.rados.0.smithi115.stdout:3652: finishing copy_from to smithi11541303-19 2022-04-22T17:37:52.979 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 1960 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:37:52.980 INFO:tasks.rados.rados.0.smithi115.stdout:3652: finishing copy_from racing read to smithi11541303-19 2022-04-22T17:37:53.150 INFO:tasks.rados.rados.0.smithi115.stdout:3641: done (12 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3643: done (11 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3644: done (10 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3645: done (9 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3646: done (8 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3647: done (7 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3648: done (6 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3649: done (5 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3650: done (4 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3651: done (3 left) 2022-04-22T17:37:53.151 INFO:tasks.rados.rados.0.smithi115.stdout:3652: done (2 left) 2022-04-22T17:37:53.152 INFO:tasks.rados.rados.0.smithi115.stdout:3653: done (1 left) 2022-04-22T17:37:53.152 INFO:tasks.rados.rados.0.smithi115.stdout:3654: done (0 left) 2022-04-22T17:37:53.152 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 353 2022-04-22T17:37:53.152 INFO:tasks.rados.rados.0.smithi115.stdout:3655: seq_num 1109 ranges {6864896=581632,7446528=532480,7979008=647168,8626176=24576} 2022-04-22T17:37:53.192 INFO:tasks.rados.rados.0.smithi115.stdout:3655: writing smithi11541303-12 from 6864896 to 7446528 tid 1 2022-04-22T17:37:53.195 INFO:tasks.rados.rados.0.smithi115.stdout:3655: writing smithi11541303-12 from 7446528 to 7979008 tid 2 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:3655: writing smithi11541303-12 from 7979008 to 8626176 tid 3 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:3655: writing smithi11541303-12 from 8626176 to 8650752 tid 4 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:3656: copy_from oid 6 from oid 39 current snap is 353 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:3657: copy_from oid 15 from oid 41 current snap is 353 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 353 2022-04-22T17:37:53.200 INFO:tasks.rados.rados.0.smithi115.stdout:3658: seq_num 1110 ranges {2236416=598016,2834432=466944,3301376=450560,3751936=286720} 2022-04-22T17:37:53.216 INFO:tasks.rados.rados.0.smithi115.stdout:3658: writing smithi11541303-21 from 2236416 to 2834432 tid 1 2022-04-22T17:37:53.219 INFO:tasks.rados.rados.0.smithi115.stdout:3658: writing smithi11541303-21 from 2834432 to 3301376 tid 2 2022-04-22T17:37:53.222 INFO:tasks.rados.rados.0.smithi115.stdout:3658: writing smithi11541303-21 from 3301376 to 3751936 tid 3 2022-04-22T17:37:53.225 INFO:tasks.rados.rados.0.smithi115.stdout:3658: writing smithi11541303-21 from 3751936 to 4038656 tid 4 2022-04-22T17:37:53.225 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 353 2022-04-22T17:37:53.226 INFO:tasks.rados.rados.0.smithi115.stdout:3659: seq_num 1111 ranges {4390912=614400,5005312=753664,5758976=450560} 2022-04-22T17:37:53.254 INFO:tasks.rados.rados.0.smithi115.stdout:3659: writing smithi11541303-45 from 4390912 to 5005312 tid 1 2022-04-22T17:37:53.259 INFO:tasks.rados.rados.0.smithi115.stdout:3659: writing smithi11541303-45 from 5005312 to 5758976 tid 2 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3659: writing smithi11541303-45 from 5758976 to 6209536 tid 3 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3660: rollback oid 30 current snap is 353 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 30 to 344 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 1 to smithi11541303-12 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 2 to smithi11541303-12 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 3 to smithi11541303-12 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 4 to smithi11541303-12 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 5 to smithi11541303-12 2022-04-22T17:37:53.263 INFO:tasks.rados.rados.0.smithi115.stdout:3655: finishing write tid 6 to smithi11541303-12 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2231 (ObjNum 1109 snap 353 seq_num 1109) dirty exists 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:3655: left oid 12 (ObjNum 1109 snap 353 seq_num 1109) 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 1 to smithi11541303-21 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:3655: done (5 left) 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 353 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 2 to smithi11541303-21 2022-04-22T17:37:53.264 INFO:tasks.rados.rados.0.smithi115.stdout:3661: seq_num 1112 ranges {9084928=630784,9715712=638976,10354688=466944,10821632=98304} 2022-04-22T17:37:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:53 smithi121 bash[36902]: cluster 2022-04-22T17:37:51.975306+0000 mgr.y (mgr.34107) 458 : cluster [DBG] pgmap v845: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 460 KiB/s wr, 3 op/s 2022-04-22T17:37:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:53 smithi121 bash[36902]: cluster 2022-04-22T17:37:51.993102+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2022-04-22T17:37:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:52 smithi115 bash[45495]: cluster 2022-04-22T17:37:51.975306+0000 mgr.y (mgr.34107) 458 : cluster [DBG] pgmap v845: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 460 KiB/s wr, 3 op/s 2022-04-22T17:37:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:52 smithi115 bash[45495]: cluster 2022-04-22T17:37:51.993102+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2022-04-22T17:37:53.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:52 smithi115 bash[46934]: cluster 2022-04-22T17:37:51.975306+0000 mgr.y (mgr.34107) 458 : cluster [DBG] pgmap v845: 121 pgs: 121 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 460 KiB/s wr, 3 op/s 2022-04-22T17:37:53.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:52 smithi115 bash[46934]: cluster 2022-04-22T17:37:51.993102+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2022-04-22T17:37:53.320 INFO:tasks.rados.rados.0.smithi115.stdout:3661: writing smithi11541303-28 from 9084928 to 9715712 tid 1 2022-04-22T17:37:53.324 INFO:tasks.rados.rados.0.smithi115.stdout:3661: writing smithi11541303-28 from 9715712 to 10354688 tid 2 2022-04-22T17:37:53.327 INFO:tasks.rados.rados.0.smithi115.stdout:3661: writing smithi11541303-28 from 10354688 to 10821632 tid 3 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3661: writing smithi11541303-28 from 10821632 to 10919936 tid 4 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3662: read oid 34 snap 351 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3662: expect (ObjNum 1104 snap 350 seq_num 1104) 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3663: snap_create 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 3 to smithi11541303-21 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 4 to smithi11541303-21 2022-04-22T17:37:53.329 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 5 to smithi11541303-21 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:3658: finishing write tid 6 to smithi11541303-21 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1875 (ObjNum 1110 snap 353 seq_num 1110) dirty exists 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:3658: left oid 21 (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:3656: finishing copy_from to smithi11541303-6 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1939 (ObjNum 1025 snap 321 seq_num 1025) dirty exists 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:3657: finishing copy_from to smithi11541303-15 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 2056 (ObjNum 1077 snap 339 seq_num 1077) dirty exists 2022-04-22T17:37:53.330 INFO:tasks.rados.rados.0.smithi115.stdout:3656: finishing copy_from racing read to smithi11541303-6 2022-04-22T17:37:53.331 INFO:tasks.rados.rados.0.smithi115.stdout:3657: finishing copy_from racing read to smithi11541303-15 2022-04-22T17:37:53.331 INFO:tasks.rados.rados.0.smithi115.stdout:3659: finishing write tid 1 to smithi11541303-45 2022-04-22T17:37:53.331 INFO:tasks.rados.rados.0.smithi115.stdout:3659: finishing write tid 2 to smithi11541303-45 2022-04-22T17:37:53.331 INFO:tasks.rados.rados.0.smithi115.stdout:3659: finishing write tid 3 to smithi11541303-45 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:3659: finishing write tid 4 to smithi11541303-45 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:3659: finishing write tid 5 to smithi11541303-45 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2076 (ObjNum 1111 snap 353 seq_num 1111) dirty exists 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:3659: left oid 45 (ObjNum 1111 snap 353 seq_num 1111) 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:3660: finishing rollback tid 1 to smithi11541303-30 2022-04-22T17:37:53.332 INFO:tasks.rados.rados.0.smithi115.stdout:3660: finishing rollback tid 2 to smithi11541303-30 2022-04-22T17:37:53.333 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2078 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:37:53.333 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 1 to smithi11541303-28 2022-04-22T17:37:53.334 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 2 to smithi11541303-28 2022-04-22T17:37:53.334 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 3 to smithi11541303-28 2022-04-22T17:37:53.334 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 4 to smithi11541303-28 2022-04-22T17:37:53.335 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 5 to smithi11541303-28 2022-04-22T17:37:53.337 INFO:tasks.rados.rados.0.smithi115.stdout:3661: finishing write tid 6 to smithi11541303-28 2022-04-22T17:37:53.337 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1905 (ObjNum 1112 snap 353 seq_num 1112) dirty exists 2022-04-22T17:37:53.338 INFO:tasks.rados.rados.0.smithi115.stdout:3661: left oid 28 (ObjNum 1112 snap 353 seq_num 1112) 2022-04-22T17:37:53.341 INFO:tasks.rados.rados.0.smithi115.stdout:3662: expect (ObjNum 1104 snap 350 seq_num 1104) 2022-04-22T17:37:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:37:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:37:53.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:37:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3656: done (7 left) 2022-04-22T17:37:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3657: done (6 left) 2022-04-22T17:37:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3658: done (5 left) 2022-04-22T17:37:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3659: done (4 left) 2022-04-22T17:37:54.159 INFO:tasks.rados.rados.0.smithi115.stdout:3660: done (3 left) 2022-04-22T17:37:54.160 INFO:tasks.rados.rados.0.smithi115.stdout:3661: done (2 left) 2022-04-22T17:37:54.160 INFO:tasks.rados.rados.0.smithi115.stdout:3662: done (1 left) 2022-04-22T17:37:54.160 INFO:tasks.rados.rados.0.smithi115.stdout:3663: done (0 left) 2022-04-22T17:37:54.160 INFO:tasks.rados.rados.0.smithi115.stdout:3664: setattr oid 35 current snap is 354 2022-04-22T17:37:54.161 INFO:tasks.rados.rados.0.smithi115.stdout:3665: copy_from oid 3 from oid 19 current snap is 354 2022-04-22T17:37:54.161 INFO:tasks.rados.rados.0.smithi115.stdout:3666: read oid 44 snap -1 2022-04-22T17:37:54.161 INFO:tasks.rados.rados.0.smithi115.stdout:3666: expect (ObjNum 1071 snap 337 seq_num 1071) 2022-04-22T17:37:54.161 INFO:tasks.rados.rados.0.smithi115.stdout:3667: snap_remove snap 339 2022-04-22T17:37:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:54 smithi121 bash[36902]: cluster 2022-04-22T17:37:53.137687+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2022-04-22T17:37:54.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:54 smithi115 bash[45495]: cluster 2022-04-22T17:37:53.137687+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2022-04-22T17:37:54.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:54 smithi115 bash[46934]: cluster 2022-04-22T17:37:53.137687+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2022-04-22T17:37:55.011 INFO:tasks.rados.rados.0.smithi115.stdout:3667: done (3 left) 2022-04-22T17:37:55.011 INFO:tasks.rados.rados.0.smithi115.stdout:3668: rollback oid 38 current snap is 354 2022-04-22T17:37:55.011 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 38 to 349 2022-04-22T17:37:55.012 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 2041 (ObjNum -1115317776 snap 22075 seq_num 3176620048) dirty exists 2022-04-22T17:37:55.012 INFO:tasks.rados.rados.0.smithi115.stdout:3666: expect (ObjNum 1071 snap 337 seq_num 1071) 2022-04-22T17:37:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:3665: finishing copy_from to smithi11541303-3 2022-04-22T17:37:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 1960 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:37:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:3665: finishing copy_from racing read to smithi11541303-3 2022-04-22T17:37:55.339 INFO:tasks.rados.rados.0.smithi115.stdout:3668: finishing rollback tid 0 to smithi11541303-38 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3668: finishing rollback tid 1 to smithi11541303-38 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3668: finishing rollback tid 2 to smithi11541303-38 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2234 (ObjNum 1093 snap 347 seq_num 1093) dirty exists 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3664: done (3 left) 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3665: done (2 left) 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3666: done (1 left) 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3668: done (0 left) 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3669: rollback oid 5 current snap is 354 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 5 to 347 2022-04-22T17:37:55.340 INFO:tasks.rados.rados.0.smithi115.stdout:3670: copy_from oid 17 from oid 48 current snap is 354 2022-04-22T17:37:55.341 INFO:tasks.rados.rados.0.smithi115.stdout:3671: setattr oid 32 current snap is 354 2022-04-22T17:37:55.344 INFO:tasks.rados.rados.0.smithi115.stdout:3672: rollback oid 27 current snap is 354 2022-04-22T17:37:55.344 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 27 to 344 2022-04-22T17:37:55.344 INFO:tasks.rados.rados.0.smithi115.stdout:3673: read oid 15 snap -1 2022-04-22T17:37:55.344 INFO:tasks.rados.rados.0.smithi115.stdout:3673: expect (ObjNum 1077 snap 339 seq_num 1077) 2022-04-22T17:37:55.345 INFO:tasks.rados.rados.0.smithi115.stdout:3674: read oid 50 snap 332 2022-04-22T17:37:55.345 INFO:tasks.rados.rados.0.smithi115.stdout:3674: expect deleted 2022-04-22T17:37:55.345 INFO:tasks.rados.rados.0.smithi115.stdout:3675: snap_create 2022-04-22T17:37:55.345 INFO:tasks.rados.rados.0.smithi115.stdout:3669: finishing rollback tid 1 to smithi11541303-5 2022-04-22T17:37:55.346 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2145 (ObjNum 1651206763 snap 1431851627 seq_num 1466389351) dirty dne 2022-04-22T17:37:55.350 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2044 (ObjNum 1106 snap 352 seq_num 1106) dirty exists 2022-04-22T17:37:55.354 INFO:tasks.rados.rados.0.smithi115.stdout:3672: finishing rollback tid 0 to smithi11541303-27 2022-04-22T17:37:55.354 INFO:tasks.rados.rados.0.smithi115.stdout:3672: finishing rollback tid 1 to smithi11541303-27 2022-04-22T17:37:55.354 INFO:tasks.rados.rados.0.smithi115.stdout:3672: finishing rollback tid 2 to smithi11541303-27 2022-04-22T17:37:55.355 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2081 (ObjNum 1033 snap 324 seq_num 1033) dirty exists 2022-04-22T17:37:55.371 INFO:tasks.rados.rados.0.smithi115.stdout:3670: finishing copy_from to smithi11541303-17 2022-04-22T17:37:55.416 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 17 v 2070 (ObjNum 1092 snap 347 seq_num 1092) dirty exists 2022-04-22T17:37:55.416 INFO:tasks.rados.rados.0.smithi115.stdout:3670: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:37:55.416 INFO:tasks.rados.rados.0.smithi115.stdout:3673: expect (ObjNum 1077 snap 339 seq_num 1077) 2022-04-22T17:37:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:55 smithi121 bash[36902]: cluster 2022-04-22T17:37:53.976233+0000 mgr.y (mgr.34107) 459 : cluster [DBG] pgmap v848: 121 pgs: 121 active+clean; 248 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 14 op/s 2022-04-22T17:37:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:55 smithi121 bash[36902]: cluster 2022-04-22T17:37:54.146526+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2022-04-22T17:37:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:55 smithi121 bash[36902]: audit 2022-04-22T17:37:54.442601+0000 mon.a (mon.0) 1464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:55 smithi121 bash[36902]: cluster 2022-04-22T17:37:54.994856+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2022-04-22T17:37:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[45495]: cluster 2022-04-22T17:37:53.976233+0000 mgr.y (mgr.34107) 459 : cluster [DBG] pgmap v848: 121 pgs: 121 active+clean; 248 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 14 op/s 2022-04-22T17:37:55.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[45495]: cluster 2022-04-22T17:37:54.146526+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[45495]: audit 2022-04-22T17:37:54.442601+0000 mon.a (mon.0) 1464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[45495]: cluster 2022-04-22T17:37:54.994856+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[46934]: cluster 2022-04-22T17:37:53.976233+0000 mgr.y (mgr.34107) 459 : cluster [DBG] pgmap v848: 121 pgs: 121 active+clean; 248 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 14 op/s 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[46934]: cluster 2022-04-22T17:37:54.146526+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[46934]: audit 2022-04-22T17:37:54.442601+0000 mon.a (mon.0) 1464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:37:55.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:55 smithi115 bash[46934]: cluster 2022-04-22T17:37:54.994856+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3669: done (6 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3670: done (5 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3671: done (4 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3672: done (3 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3673: done (2 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3674: done (1 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3675: done (0 left) 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3676: copy_from oid 13 from oid 29 current snap is 355 2022-04-22T17:37:56.235 INFO:tasks.rados.rados.0.smithi115.stdout:3677: setattr oid 33 current snap is 355 2022-04-22T17:37:56.239 INFO:tasks.rados.rados.0.smithi115.stdout:3678: setattr oid 30 current snap is 355 2022-04-22T17:37:56.240 INFO:tasks.rados.rados.0.smithi115.stdout:3676: finishing copy_from to smithi11541303-13 2022-04-22T17:37:56.241 INFO:tasks.rados.rados.0.smithi115.stdout:3676: got expected ENOENT (src dne) 2022-04-22T17:37:56.241 INFO:tasks.rados.rados.0.smithi115.stdout:3676: finishing copy_from racing read to smithi11541303-13 2022-04-22T17:37:56.241 INFO:tasks.rados.rados.0.smithi115.stdout:3676: done (2 left) 2022-04-22T17:37:56.242 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 355 2022-04-22T17:37:56.242 INFO:tasks.rados.rados.0.smithi115.stdout:3679: seq_num 1113 ranges {8953856=647168,9601024=524288,10125312=622592,10747904=57344} 2022-04-22T17:37:56.297 INFO:tasks.rados.rados.0.smithi115.stdout:3679: writing smithi11541303-25 from 8953856 to 9601024 tid 1 2022-04-22T17:37:56.300 INFO:tasks.rados.rados.0.smithi115.stdout:3679: writing smithi11541303-25 from 9601024 to 10125312 tid 2 2022-04-22T17:37:56.303 INFO:tasks.rados.rados.0.smithi115.stdout:3679: writing smithi11541303-25 from 10125312 to 10747904 tid 3 2022-04-22T17:37:56.304 INFO:tasks.rados.rados.0.smithi115.stdout:3679: writing smithi11541303-25 from 10747904 to 10805248 tid 4 2022-04-22T17:37:56.304 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 355 2022-04-22T17:37:56.304 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 2045 (ObjNum 1049 snap 329 seq_num 1049) dirty exists 2022-04-22T17:37:56.304 INFO:tasks.rados.rados.0.smithi115.stdout:3680: seq_num 1114 ranges {704512=663552,1368064=802816,2170880=401408} 2022-04-22T17:37:56.312 INFO:tasks.rados.rados.0.smithi115.stdout:3680: writing smithi11541303-8 from 704512 to 1368064 tid 1 2022-04-22T17:37:56.316 INFO:tasks.rados.rados.0.smithi115.stdout:3680: writing smithi11541303-8 from 1368064 to 2170880 tid 2 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:3680: writing smithi11541303-8 from 2170880 to 2572288 tid 3 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2082 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 1 to smithi11541303-25 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 2 to smithi11541303-25 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 3 to smithi11541303-25 2022-04-22T17:37:56.319 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 4 to smithi11541303-25 2022-04-22T17:37:56.320 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 5 to smithi11541303-25 2022-04-22T17:37:56.320 INFO:tasks.rados.rados.0.smithi115.stdout:3677: done (3 left) 2022-04-22T17:37:56.320 INFO:tasks.rados.rados.0.smithi115.stdout:3678: done (2 left) 2022-04-22T17:37:56.320 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 355 2022-04-22T17:37:56.320 INFO:tasks.rados.rados.0.smithi115.stdout:3681: seq_num 1115 ranges {12812288=679936,13492224=688128,14180352=516096} 2022-04-22T17:37:56.396 INFO:tasks.rados.rados.0.smithi115.stdout:3681: writing smithi11541303-19 from 12812288 to 13492224 tid 1 2022-04-22T17:37:56.400 INFO:tasks.rados.rados.0.smithi115.stdout:3681: writing smithi11541303-19 from 13492224 to 14180352 tid 2 2022-04-22T17:37:56.410 INFO:tasks.rados.rados.0.smithi115.stdout:3681: writing smithi11541303-19 from 14180352 to 14696448 tid 3 2022-04-22T17:37:56.410 INFO:tasks.rados.rados.0.smithi115.stdout:3682: read oid 46 snap -1 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3682: expect (ObjNum 1032 snap 323 seq_num 1032) 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 355 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3680: finishing write tid 1 to smithi11541303-8 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3679: finishing write tid 6 to smithi11541303-25 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 2191 (ObjNum 1113 snap 355 seq_num 1113) dirty exists 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3679: left oid 25 (ObjNum 1113 snap 355 seq_num 1113) 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3680: finishing write tid 2 to smithi11541303-8 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3680: finishing write tid 3 to smithi11541303-8 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3680: finishing write tid 4 to smithi11541303-8 2022-04-22T17:37:56.411 INFO:tasks.rados.rados.0.smithi115.stdout:3680: finishing write tid 5 to smithi11541303-8 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 1964 (ObjNum 1114 snap 355 seq_num 1114) dirty exists 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:3680: left oid 8 (ObjNum 1114 snap 355 seq_num 1114) 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:3681: finishing write tid 1 to smithi11541303-19 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:3683: seq_num 1116 ranges {0=696320,696320=573440,1269760=450560,1720320=180224} 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:3683: writing smithi11541303-49 from 0 to 696320 tid 1 2022-04-22T17:37:56.412 INFO:tasks.rados.rados.0.smithi115.stdout:3683: writing smithi11541303-49 from 696320 to 1269760 tid 2 2022-04-22T17:37:56.414 INFO:tasks.rados.rados.0.smithi115.stdout:3683: writing smithi11541303-49 from 1269760 to 1720320 tid 3 2022-04-22T17:37:56.416 INFO:tasks.rados.rados.0.smithi115.stdout:3683: writing smithi11541303-49 from 1720320 to 1900544 tid 4 2022-04-22T17:37:56.416 INFO:tasks.rados.rados.0.smithi115.stdout:3679: done (4 left) 2022-04-22T17:37:56.416 INFO:tasks.rados.rados.0.smithi115.stdout:3680: done (3 left) 2022-04-22T17:37:56.416 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 355 2022-04-22T17:37:56.416 INFO:tasks.rados.rados.0.smithi115.stdout:3684: seq_num 1117 ranges {0=712704,712704=458752,1171456=655360,1826816=90112} 2022-04-22T17:37:56.420 INFO:tasks.rados.rados.0.smithi115.stdout:3684: writing smithi11541303-5 from 0 to 712704 tid 1 2022-04-22T17:37:56.423 INFO:tasks.rados.rados.0.smithi115.stdout:3684: writing smithi11541303-5 from 712704 to 1171456 tid 2 2022-04-22T17:37:56.427 INFO:tasks.rados.rados.0.smithi115.stdout:3684: writing smithi11541303-5 from 1171456 to 1826816 tid 3 2022-04-22T17:37:56.428 INFO:tasks.rados.rados.0.smithi115.stdout:3684: writing smithi11541303-5 from 1826816 to 1916928 tid 4 2022-04-22T17:37:56.428 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 23 current snap is 355 2022-04-22T17:37:56.428 INFO:tasks.rados.rados.0.smithi115.stdout:3681: finishing write tid 2 to smithi11541303-19 2022-04-22T17:37:56.428 INFO:tasks.rados.rados.0.smithi115.stdout:3685: seq_num 1118 ranges {2916352=729088,3645440=794624,4440064=409600} 2022-04-22T17:37:56.448 INFO:tasks.rados.rados.0.smithi115.stdout:3685: writing smithi11541303-23 from 2916352 to 3645440 tid 1 2022-04-22T17:37:56.453 INFO:tasks.rados.rados.0.smithi115.stdout:3685: writing smithi11541303-23 from 3645440 to 4440064 tid 2 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:3685: writing smithi11541303-23 from 4440064 to 4849664 tid 3 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:3686: rmattr oid 38 current snap is 355 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:3681: finishing write tid 3 to smithi11541303-19 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 355 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:3681: finishing write tid 4 to smithi11541303-19 2022-04-22T17:37:56.456 INFO:tasks.rados.rados.0.smithi115.stdout:3682: expect (ObjNum 1032 snap 323 seq_num 1032) 2022-04-22T17:37:56.470 INFO:tasks.rados.rados.0.smithi115.stdout:3681: finishing write tid 5 to smithi11541303-19 2022-04-22T17:37:56.470 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 2238 (ObjNum 1115 snap 355 seq_num 1115) dirty exists 2022-04-22T17:37:56.471 INFO:tasks.rados.rados.0.smithi115.stdout:3681: left oid 19 (ObjNum 1115 snap 355 seq_num 1115) 2022-04-22T17:37:56.471 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 1 to smithi11541303-49 2022-04-22T17:37:56.471 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 2 to smithi11541303-49 2022-04-22T17:37:56.471 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 3 to smithi11541303-49 2022-04-22T17:37:56.471 INFO:tasks.rados.rados.0.smithi115.stdout:3687: seq_num 1119 ranges {0=745472,745472=679936,1425408=524288} 2022-04-22T17:37:56.476 INFO:tasks.rados.rados.0.smithi115.stdout:3687: writing smithi11541303-22 from 0 to 745472 tid 1 2022-04-22T17:37:56.482 INFO:tasks.rados.rados.0.smithi115.stdout:3687: writing smithi11541303-22 from 745472 to 1425408 tid 2 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3687: writing smithi11541303-22 from 1425408 to 1949696 tid 3 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3681: done (6 left) 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3682: done (5 left) 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3688: snap_create 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 4 to smithi11541303-49 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 5 to smithi11541303-49 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 1 to smithi11541303-5 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:3683: finishing write tid 6 to smithi11541303-49 2022-04-22T17:37:56.486 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2244 (ObjNum 1116 snap 355 seq_num 1116) dirty exists 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3683: left oid 49 (ObjNum 1116 snap 355 seq_num 1116) 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 2 to smithi11541303-5 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 3 to smithi11541303-5 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 4 to smithi11541303-5 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 5 to smithi11541303-5 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: finishing write tid 6 to smithi11541303-5 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2249 (ObjNum 1117 snap 355 seq_num 1117) dirty exists 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3684: left oid 5 (ObjNum 1117 snap 355 seq_num 1117) 2022-04-22T17:37:56.487 INFO:tasks.rados.rados.0.smithi115.stdout:3685: finishing write tid 1 to smithi11541303-23 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2239 (ObjNum 1093 snap 347 seq_num 1093) dirty exists 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3685: finishing write tid 2 to smithi11541303-23 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3685: finishing write tid 3 to smithi11541303-23 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3685: finishing write tid 4 to smithi11541303-23 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3685: finishing write tid 5 to smithi11541303-23 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2092 (ObjNum 1118 snap 355 seq_num 1118) dirty exists 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3685: left oid 23 (ObjNum 1118 snap 355 seq_num 1118) 2022-04-22T17:37:56.488 INFO:tasks.rados.rados.0.smithi115.stdout:3687: finishing write tid 1 to smithi11541303-22 2022-04-22T17:37:56.489 INFO:tasks.rados.rados.0.smithi115.stdout:3687: finishing write tid 2 to smithi11541303-22 2022-04-22T17:37:56.490 INFO:tasks.rados.rados.0.smithi115.stdout:3687: finishing write tid 3 to smithi11541303-22 2022-04-22T17:37:56.490 INFO:tasks.rados.rados.0.smithi115.stdout:3687: finishing write tid 4 to smithi11541303-22 2022-04-22T17:37:56.491 INFO:tasks.rados.rados.0.smithi115.stdout:3687: finishing write tid 5 to smithi11541303-22 2022-04-22T17:37:56.491 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1879 (ObjNum 1119 snap 355 seq_num 1119) dirty exists 2022-04-22T17:37:56.491 INFO:tasks.rados.rados.0.smithi115.stdout:3687: left oid 22 (ObjNum 1119 snap 355 seq_num 1119) 2022-04-22T17:37:57.180 INFO:tasks.rados.rados.0.smithi115.stdout:3683: done (5 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3684: done (4 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3685: done (3 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3686: done (2 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3687: done (1 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3688: done (0 left) 2022-04-22T17:37:57.181 INFO:tasks.rados.rados.0.smithi115.stdout:3689: snap_remove snap 351 2022-04-22T17:37:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:57 smithi121 bash[36902]: cluster 2022-04-22T17:37:55.977294+0000 mgr.y (mgr.34107) 460 : cluster [DBG] pgmap v851: 121 pgs: 121 active+clean; 283 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 12 MiB/s wr, 17 op/s 2022-04-22T17:37:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:57 smithi121 bash[36902]: cluster 2022-04-22T17:37:56.163708+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2022-04-22T17:37:57.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:57 smithi115 bash[45495]: cluster 2022-04-22T17:37:55.977294+0000 mgr.y (mgr.34107) 460 : cluster [DBG] pgmap v851: 121 pgs: 121 active+clean; 283 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 12 MiB/s wr, 17 op/s 2022-04-22T17:37:57.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:57 smithi115 bash[45495]: cluster 2022-04-22T17:37:56.163708+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2022-04-22T17:37:57.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:57 smithi115 bash[46934]: cluster 2022-04-22T17:37:55.977294+0000 mgr.y (mgr.34107) 460 : cluster [DBG] pgmap v851: 121 pgs: 121 active+clean; 283 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 12 MiB/s wr, 17 op/s 2022-04-22T17:37:57.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:57 smithi115 bash[46934]: cluster 2022-04-22T17:37:56.163708+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2022-04-22T17:37:58.194 INFO:tasks.rados.rados.0.smithi115.stdout:3689: done (0 left) 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:3690: copy_from oid 49 from oid 2 current snap is 356 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:3691: read oid 18 snap -1 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:3691: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:3692: copy_from oid 12 from oid 4 current snap is 356 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:3693: rollback oid 25 current snap is 356 2022-04-22T17:37:58.195 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 25 to 344 2022-04-22T17:37:58.196 INFO:tasks.rados.rados.0.smithi115.stdout:3694: snap_create 2022-04-22T17:37:58.206 INFO:tasks.rados.rados.0.smithi115.stdout:3692: finishing copy_from to smithi11541303-12 2022-04-22T17:37:58.206 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2071 (ObjNum -1479267008 snap 32692 seq_num 3178419568) dirty exists 2022-04-22T17:37:58.209 INFO:tasks.rados.rados.0.smithi115.stdout:3692: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:37:58.209 INFO:tasks.rados.rados.0.smithi115.stdout:3691: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:37:58.234 INFO:tasks.rados.rados.0.smithi115.stdout:3693: finishing rollback tid 0 to smithi11541303-25 2022-04-22T17:37:58.235 INFO:tasks.rados.rados.0.smithi115.stdout:3693: finishing rollback tid 1 to smithi11541303-25 2022-04-22T17:37:58.235 INFO:tasks.rados.rados.0.smithi115.stdout:3693: finishing rollback tid 2 to smithi11541303-25 2022-04-22T17:37:58.235 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 2194 (ObjNum 1081 snap 342 seq_num 1081) dirty exists 2022-04-22T17:37:58.238 INFO:tasks.rados.rados.0.smithi115.stdout:3690: finishing copy_from to smithi11541303-49 2022-04-22T17:37:58.238 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2040 (ObjNum 1107 snap 352 seq_num 1107) dirty exists 2022-04-22T17:37:58.240 INFO:tasks.rados.rados.0.smithi115.stdout:3690: finishing copy_from racing read to smithi11541303-49 2022-04-22T17:37:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:58 smithi121 bash[36902]: cluster 2022-04-22T17:37:57.168706+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2022-04-22T17:37:58.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:58 smithi115 bash[45495]: cluster 2022-04-22T17:37:57.168706+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2022-04-22T17:37:58.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:58 smithi115 bash[46934]: cluster 2022-04-22T17:37:57.168706+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2022-04-22T17:37:59.194 INFO:tasks.rados.rados.0.smithi115.stdout:3690: done (4 left) 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:3691: done (3 left) 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:3692: done (2 left) 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:3693: done (1 left) 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:3694: done (0 left) 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 357 2022-04-22T17:37:59.195 INFO:tasks.rados.rados.0.smithi115.stdout:3695: seq_num 1120 ranges {4767744=761856,5529600=557056,6086656=425984,6512640=221184} 2022-04-22T17:37:59.227 INFO:tasks.rados.rados.0.smithi115.stdout:3695: writing smithi11541303-32 from 4767744 to 5529600 tid 1 2022-04-22T17:37:59.259 INFO:tasks.rados.rados.0.smithi115.stdout:3695: writing smithi11541303-32 from 5529600 to 6086656 tid 2 2022-04-22T17:37:59.259 INFO:tasks.rados.rados.0.smithi115.stdout:3695: writing smithi11541303-32 from 6086656 to 6512640 tid 3 2022-04-22T17:37:59.260 INFO:tasks.rados.rados.0.smithi115.stdout:3695: writing smithi11541303-32 from 6512640 to 6733824 tid 4 2022-04-22T17:37:59.260 INFO:tasks.rados.rados.0.smithi115.stdout:3696: snap_remove snap 355 2022-04-22T17:37:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:59 smithi121 bash[36902]: cluster 2022-04-22T17:37:57.978118+0000 mgr.y (mgr.34107) 461 : cluster [DBG] pgmap v854: 121 pgs: 121 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 14 MiB/s wr, 31 op/s 2022-04-22T17:37:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:37:59 smithi121 bash[36902]: cluster 2022-04-22T17:37:58.181122+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2022-04-22T17:37:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:59 smithi115 bash[46934]: cluster 2022-04-22T17:37:57.978118+0000 mgr.y (mgr.34107) 461 : cluster [DBG] pgmap v854: 121 pgs: 121 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 14 MiB/s wr, 31 op/s 2022-04-22T17:37:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:37:59 smithi115 bash[46934]: cluster 2022-04-22T17:37:58.181122+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2022-04-22T17:37:59.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:59 smithi115 bash[45495]: cluster 2022-04-22T17:37:57.978118+0000 mgr.y (mgr.34107) 461 : cluster [DBG] pgmap v854: 121 pgs: 121 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 14 MiB/s wr, 31 op/s 2022-04-22T17:37:59.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:37:59 smithi115 bash[45495]: cluster 2022-04-22T17:37:58.181122+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3696: done (1 left) 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3697: copy_from oid 44 from oid 3 current snap is 357 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3698: read oid 39 snap -1 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3698: expect (ObjNum 1025 snap 321 seq_num 1025) 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3699: read oid 10 snap 349 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3699: expect (ObjNum 1064 snap 337 seq_num 1064) 2022-04-22T17:38:00.014 INFO:tasks.rados.rados.0.smithi115.stdout:3700: copy_from oid 18 from oid 14 current snap is 357 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3701: delete oid 13 current snap is 357 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 1 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 2 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 3 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 4 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 5 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: finishing write tid 6 to smithi11541303-32 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2050 (ObjNum 1120 snap 357 seq_num 1120) dirty exists 2022-04-22T17:38:00.015 INFO:tasks.rados.rados.0.smithi115.stdout:3695: left oid 32 (ObjNum 1120 snap 357 seq_num 1120) 2022-04-22T17:38:00.032 INFO:tasks.rados.rados.0.smithi115.stdout:3695: done (5 left) 2022-04-22T17:38:00.032 INFO:tasks.rados.rados.0.smithi115.stdout:3701: done (4 left) 2022-04-22T17:38:00.032 INFO:tasks.rados.rados.0.smithi115.stdout:3702: snap_remove snap 354 2022-04-22T17:38:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:00 smithi121 bash[36902]: cluster 2022-04-22T17:37:59.182215+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2022-04-22T17:38:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:00 smithi121 bash[36902]: audit 2022-04-22T17:37:59.445790+0000 mon.a (mon.0) 1470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:00 smithi121 bash[36902]: cluster 2022-04-22T17:37:59.979060+0000 mgr.y (mgr.34107) 462 : cluster [DBG] pgmap v857: 121 pgs: 121 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 45 op/s 2022-04-22T17:38:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:00 smithi121 bash[36902]: cluster 2022-04-22T17:37:59.999378+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2022-04-22T17:38:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[45495]: cluster 2022-04-22T17:37:59.182215+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2022-04-22T17:38:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[45495]: audit 2022-04-22T17:37:59.445790+0000 mon.a (mon.0) 1470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[45495]: cluster 2022-04-22T17:37:59.979060+0000 mgr.y (mgr.34107) 462 : cluster [DBG] pgmap v857: 121 pgs: 121 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 45 op/s 2022-04-22T17:38:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[45495]: cluster 2022-04-22T17:37:59.999378+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2022-04-22T17:38:00.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[46934]: cluster 2022-04-22T17:37:59.182215+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2022-04-22T17:38:00.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[46934]: audit 2022-04-22T17:37:59.445790+0000 mon.a (mon.0) 1470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:00.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[46934]: cluster 2022-04-22T17:37:59.979060+0000 mgr.y (mgr.34107) 462 : cluster [DBG] pgmap v857: 121 pgs: 121 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 45 op/s 2022-04-22T17:38:00.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:00 smithi115 bash[46934]: cluster 2022-04-22T17:37:59.999378+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2022-04-22T17:38:01.031 INFO:tasks.rados.rados.0.smithi115.stdout:3702: done (4 left) 2022-04-22T17:38:01.031 INFO:tasks.rados.rados.0.smithi115.stdout:3703: setattr oid 30 current snap is 357 2022-04-22T17:38:01.032 INFO:tasks.rados.rados.0.smithi115.stdout:3699: expect (ObjNum 1064 snap 337 seq_num 1064) 2022-04-22T17:38:01.098 INFO:tasks.rados.rados.0.smithi115.stdout:3700: finishing copy_from to smithi11541303-18 2022-04-22T17:38:01.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 1896 (ObjNum 1040 snap 324 seq_num 1040) dirty exists 2022-04-22T17:38:01.098 INFO:tasks.rados.rados.0.smithi115.stdout:3700: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:38:01.098 INFO:tasks.rados.rados.0.smithi115.stdout:3698: expect (ObjNum 1025 snap 321 seq_num 1025) 2022-04-22T17:38:01.766 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:38:01.998 INFO:tasks.rados.rados.0.smithi115.stdout:3697: finishing copy_from to smithi11541303-44 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1960 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3697: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3697: done (4 left) 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3698: done (3 left) 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3699: done (2 left) 2022-04-22T17:38:01.999 INFO:tasks.rados.rados.0.smithi115.stdout:3700: done (1 left) 2022-04-22T17:38:02.000 INFO:tasks.rados.rados.0.smithi115.stdout:3704: delete oid 34 current snap is 357 2022-04-22T17:38:02.006 INFO:tasks.rados.rados.0.smithi115.stdout:3704: done (1 left) 2022-04-22T17:38:02.008 INFO:tasks.rados.rados.0.smithi115.stdout:3705: read oid 49 snap -1 2022-04-22T17:38:02.008 INFO:tasks.rados.rados.0.smithi115.stdout:3705: expect (ObjNum 1107 snap 352 seq_num 1107) 2022-04-22T17:38:02.008 INFO:tasks.rados.rados.0.smithi115.stdout:3706: rollback oid 21 current snap is 357 2022-04-22T17:38:02.009 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 356 2022-04-22T17:38:02.010 INFO:tasks.rados.rados.0.smithi115.stdout:3707: read oid 4 snap -1 2022-04-22T17:38:02.010 INFO:tasks.rados.rados.0.smithi115.stdout:3707: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:38:02.010 INFO:tasks.rados.rados.0.smithi115.stdout:3708: read oid 40 snap -1 2022-04-22T17:38:02.010 INFO:tasks.rados.rados.0.smithi115.stdout:3708: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:02.011 INFO:tasks.rados.rados.0.smithi115.stdout:3709: read oid 36 snap 347 2022-04-22T17:38:02.011 INFO:tasks.rados.rados.0.smithi115.stdout:3709: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:38:02.011 INFO:tasks.rados.rados.0.smithi115.stdout:3710: setattr oid 48 current snap is 357 2022-04-22T17:38:02.011 INFO:tasks.rados.rados.0.smithi115.stdout:3707: expect (ObjNum 0 snap 0 seq_num 0) 2022-04-22T17:38:02.013 INFO:tasks.rados.rados.0.smithi115.stdout:3707: done (6 left) 2022-04-22T17:38:02.013 INFO:tasks.rados.rados.0.smithi115.stdout:3711: read oid 20 snap -1 2022-04-22T17:38:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:3711: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:38:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:3712: rmattr oid 50 current snap is 357 2022-04-22T17:38:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:3713: rmattr oid 28 current snap is 357 2022-04-22T17:38:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:3714: snap_create 2022-04-22T17:38:02.014 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2083 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:38:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 0 (ObjNum 1064 snap 337 seq_num 1064) dirty exists 2022-04-22T17:38:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3706: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:38:02.015 INFO:tasks.rados.rados.0.smithi115.stdout:3709: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:38:02.017 INFO:tasks.rados.rados.0.smithi115.stdout:3706: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:38:02.017 INFO:tasks.rados.rados.0.smithi115.stdout:3706: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:38:02.017 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1882 (ObjNum 1110 snap 353 seq_num 1110) dirty exists 2022-04-22T17:38:02.018 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1906 (ObjNum 1112 snap 353 seq_num 1112) dirty exists 2022-04-22T17:38:02.019 INFO:tasks.rados.rados.0.smithi115.stdout:3708: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:02.026 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:01.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:02.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:02 smithi115 bash[45495]: cluster 2022-04-22T17:38:01.016985+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2022-04-22T17:38:02.051 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2084 (ObjNum 1092 snap 347 seq_num 1092) dirty exists 2022-04-22T17:38:02.051 INFO:tasks.rados.rados.0.smithi115.stdout:3711: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:38:02.126 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:38:02.127 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (66s) 61s ago 14m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:38:02.127 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (14m) 102s ago 14m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:38:02.127 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3m) 61s ago 15m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:38:02.127 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2m) 102s ago 15m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:38:02.127 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (10m) 102s ago 20m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:38:02.128 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (10m) 61s ago 22m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:38:02.128 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (9m) 61s ago 22m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:38:02.128 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (9m) 102s ago 20m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:38:02.128 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 61s ago 20m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:38:02.129 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 61s ago 15m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:38:02.129 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2m) 102s ago 15m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:38:02.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 61s ago 19m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:38:02.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 61s ago 19m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:38:02.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6m) 61s ago 18m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:38:02.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 61s ago 18m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:38:02.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5m) 102s ago 17m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:38:02.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4m) 102s ago 17m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:38:02.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4m) 102s ago 16m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:38:02.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3m) 102s ago 16m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:38:02.132 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (108s) 102s ago 15m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:38:02.250 INFO:tasks.rados.rados.0.smithi115.stdout:3705: expect (ObjNum 1107 snap 352 seq_num 1107) 2022-04-22T17:38:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:02 smithi115 bash[46934]: cluster 2022-04-22T17:38:01.016985+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2022-04-22T17:38:02.468 INFO:tasks.rados.rados.0.smithi115.stdout:3703: done (9 left) 2022-04-22T17:38:02.469 INFO:tasks.rados.rados.0.smithi115.stdout:3705: done (8 left) 2022-04-22T17:38:02.469 INFO:tasks.rados.rados.0.smithi115.stdout:3706: done (7 left) 2022-04-22T17:38:02.469 INFO:tasks.rados.rados.0.smithi115.stdout:3708: done (6 left) 2022-04-22T17:38:02.470 INFO:tasks.rados.rados.0.smithi115.stdout:3709: done (5 left) 2022-04-22T17:38:02.471 INFO:tasks.rados.rados.0.smithi115.stdout:3710: done (4 left) 2022-04-22T17:38:02.472 INFO:tasks.rados.rados.0.smithi115.stdout:3711: done (3 left) 2022-04-22T17:38:02.472 INFO:tasks.rados.rados.0.smithi115.stdout:3712: done (2 left) 2022-04-22T17:38:02.472 INFO:tasks.rados.rados.0.smithi115.stdout:3713: done (1 left) 2022-04-22T17:38:02.472 INFO:tasks.rados.rados.0.smithi115.stdout:3714: done (0 left) 2022-04-22T17:38:02.473 INFO:tasks.rados.rados.0.smithi115.stdout:3715: snap_remove snap 353 2022-04-22T17:38:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:02 smithi121 bash[36902]: cluster 2022-04-22T17:38:01.016985+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2022-04-22T17:38:02.517 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:38:02.517 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:38:02.517 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:38:02.517 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:38:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:38:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:38:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:38:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:38:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:38:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:38:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:38:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:38:02.521 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:38:03.041 INFO:tasks.rados.rados.0.smithi115.stdout:3715: done (0 left) 2022-04-22T17:38:03.041 INFO:tasks.rados.rados.0.smithi115.stdout:3716: snap_remove snap 350 2022-04-22T17:38:03.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[45495]: audit 2022-04-22T17:38:01.736539+0000 mgr.y (mgr.34107) 463 : audit [DBG] from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[45495]: cluster 2022-04-22T17:38:01.979993+0000 mgr.y (mgr.34107) 464 : cluster [DBG] pgmap v860: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 347 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.4 MiB/s wr, 29 op/s 2022-04-22T17:38:03.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[45495]: cluster 2022-04-22T17:38:02.019050+0000 mon.a (mon.0) 1473 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[45495]: audit 2022-04-22T17:38:02.095588+0000 mgr.y (mgr.34107) 465 : audit [DBG] from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[45495]: audit 2022-04-22T17:38:02.503991+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.115:0/3874574051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[46934]: audit 2022-04-22T17:38:01.736539+0000 mgr.y (mgr.34107) 463 : audit [DBG] from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[46934]: cluster 2022-04-22T17:38:01.979993+0000 mgr.y (mgr.34107) 464 : cluster [DBG] pgmap v860: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 347 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.4 MiB/s wr, 29 op/s 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[46934]: cluster 2022-04-22T17:38:02.019050+0000 mon.a (mon.0) 1473 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2022-04-22T17:38:03.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[46934]: audit 2022-04-22T17:38:02.095588+0000 mgr.y (mgr.34107) 465 : audit [DBG] from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[46934]: audit 2022-04-22T17:38:02.503991+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.115:0/3874574051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:03 smithi121 bash[36902]: audit 2022-04-22T17:38:01.736539+0000 mgr.y (mgr.34107) 463 : audit [DBG] from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:03 smithi121 bash[36902]: cluster 2022-04-22T17:38:01.979993+0000 mgr.y (mgr.34107) 464 : cluster [DBG] pgmap v860: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 347 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.4 MiB/s wr, 29 op/s 2022-04-22T17:38:03.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:03 smithi121 bash[36902]: cluster 2022-04-22T17:38:02.019050+0000 mon.a (mon.0) 1473 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2022-04-22T17:38:03.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:03 smithi121 bash[36902]: audit 2022-04-22T17:38:02.095588+0000 mgr.y (mgr.34107) 465 : audit [DBG] from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:03.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:03 smithi121 bash[36902]: audit 2022-04-22T17:38:02.503991+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.115:0/3874574051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:03.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:04.053 INFO:tasks.rados.rados.0.smithi115.stdout:3716: done (0 left) 2022-04-22T17:38:04.053 INFO:tasks.rados.rados.0.smithi115.stdout:3717: snap_create 2022-04-22T17:38:04.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:04 smithi115 bash[45495]: cluster 2022-04-22T17:38:03.028129+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2022-04-22T17:38:04.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:04 smithi115 bash[46934]: cluster 2022-04-22T17:38:03.028129+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2022-04-22T17:38:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:04 smithi121 bash[36902]: cluster 2022-04-22T17:38:03.028129+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2022-04-22T17:38:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:3717: done (0 left) 2022-04-22T17:38:05.011 INFO:tasks.rados.rados.0.smithi115.stdout:3718: read oid 36 snap 349 2022-04-22T17:38:05.012 INFO:tasks.rados.rados.0.smithi115.stdout:3718: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:38:05.012 INFO:tasks.rados.rados.0.smithi115.stdout:3719: setattr oid 19 current snap is 359 2022-04-22T17:38:05.014 INFO:tasks.rados.rados.0.smithi115.stdout:3720: copy_from oid 31 from oid 11 current snap is 359 2022-04-22T17:38:05.014 INFO:tasks.rados.rados.0.smithi115.stdout:3721: snap_remove snap 357 2022-04-22T17:38:05.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[45495]: cluster 2022-04-22T17:38:03.981028+0000 mgr.y (mgr.34107) 466 : cluster [DBG] pgmap v863: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 13 op/s 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[45495]: cluster 2022-04-22T17:38:04.038452+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[45495]: audit 2022-04-22T17:38:04.448964+0000 mon.a (mon.0) 1476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[45495]: cluster 2022-04-22T17:38:04.997676+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[46934]: cluster 2022-04-22T17:38:03.981028+0000 mgr.y (mgr.34107) 466 : cluster [DBG] pgmap v863: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 13 op/s 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[46934]: cluster 2022-04-22T17:38:04.038452+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[46934]: audit 2022-04-22T17:38:04.448964+0000 mon.a (mon.0) 1476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:05.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:05 smithi115 bash[46934]: cluster 2022-04-22T17:38:04.997676+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2022-04-22T17:38:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:05 smithi121 bash[36902]: cluster 2022-04-22T17:38:03.981028+0000 mgr.y (mgr.34107) 466 : cluster [DBG] pgmap v863: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 13 op/s 2022-04-22T17:38:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:05 smithi121 bash[36902]: cluster 2022-04-22T17:38:04.038452+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2022-04-22T17:38:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:05 smithi121 bash[36902]: audit 2022-04-22T17:38:04.448964+0000 mon.a (mon.0) 1476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:05 smithi121 bash[36902]: cluster 2022-04-22T17:38:04.997676+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2022-04-22T17:38:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:3721: done (3 left) 2022-04-22T17:38:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:3722: rmattr oid 15 current snap is 359 2022-04-22T17:38:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:3722: done (3 left) 2022-04-22T17:38:06.021 INFO:tasks.rados.rados.0.smithi115.stdout:3723: rollback oid 12 current snap is 359 2022-04-22T17:38:06.022 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 349 2022-04-22T17:38:06.022 INFO:tasks.rados.rados.0.smithi115.stdout:3724: snap_remove snap 332 2022-04-22T17:38:07.025 INFO:tasks.rados.rados.0.smithi115.stdout:3724: done (4 left) 2022-04-22T17:38:07.025 INFO:tasks.rados.rados.0.smithi115.stdout:3725: snap_remove snap 352 2022-04-22T17:38:07.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:07 smithi115 bash[45495]: cluster 2022-04-22T17:38:05.982095+0000 mgr.y (mgr.34107) 467 : cluster [DBG] pgmap v866: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2022-04-22T17:38:07.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:07 smithi115 bash[45495]: cluster 2022-04-22T17:38:06.009072+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2022-04-22T17:38:07.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:07 smithi115 bash[46934]: cluster 2022-04-22T17:38:05.982095+0000 mgr.y (mgr.34107) 467 : cluster [DBG] pgmap v866: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2022-04-22T17:38:07.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:07 smithi115 bash[46934]: cluster 2022-04-22T17:38:06.009072+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2022-04-22T17:38:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:07 smithi121 bash[36902]: cluster 2022-04-22T17:38:05.982095+0000 mgr.y (mgr.34107) 467 : cluster [DBG] pgmap v866: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2022-04-22T17:38:07.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:07 smithi121 bash[36902]: cluster 2022-04-22T17:38:06.009072+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2022-04-22T17:38:08.033 INFO:tasks.rados.rados.0.smithi115.stdout:3725: done (4 left) 2022-04-22T17:38:08.033 INFO:tasks.rados.rados.0.smithi115.stdout:3726: read oid 17 snap -1 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3726: expect (ObjNum 1092 snap 347 seq_num 1092) 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3727: read oid 2 snap -1 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3727: expect (ObjNum 1107 snap 352 seq_num 1107) 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3728: rollback oid 18 current snap is 359 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 18 to 349 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3729: rmattr oid 8 current snap is 359 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3729: done (7 left) 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 1 current snap is 359 2022-04-22T17:38:08.034 INFO:tasks.rados.rados.0.smithi115.stdout:3730: seq_num 1123 ranges {1597440=409600,2007040=614400,2621440=598016,3219456=393216} 2022-04-22T17:38:08.045 INFO:tasks.rados.rados.0.smithi115.stdout:3730: writing smithi11541303-1 from 1597440 to 2007040 tid 1 2022-04-22T17:38:08.049 INFO:tasks.rados.rados.0.smithi115.stdout:3730: writing smithi11541303-1 from 2007040 to 2621440 tid 2 2022-04-22T17:38:08.054 INFO:tasks.rados.rados.0.smithi115.stdout:3730: writing smithi11541303-1 from 2621440 to 3219456 tid 3 2022-04-22T17:38:08.057 INFO:tasks.rados.rados.0.smithi115.stdout:3730: writing smithi11541303-1 from 3219456 to 3612672 tid 4 2022-04-22T17:38:08.057 INFO:tasks.rados.rados.0.smithi115.stdout:3731: setattr oid 10 current snap is 359 2022-04-22T17:38:08.057 INFO:tasks.rados.rados.0.smithi115.stdout:3718: expect (ObjNum 1006 snap 320 seq_num 1006) 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 2240 (ObjNum 1115 snap 355 seq_num 1115) dirty exists 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:3720: finishing copy_from to smithi11541303-31 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2080 (ObjNum 1080 snap 342 seq_num 1080) dirty exists 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:3720: finishing copy_from racing read to smithi11541303-31 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:3723: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:38:08.058 INFO:tasks.rados.rados.0.smithi115.stdout:3723: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:3723: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2243 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:3728: finishing rollback tid 0 to smithi11541303-18 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:3728: finishing rollback tid 1 to smithi11541303-18 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:3728: finishing rollback tid 2 to smithi11541303-18 2022-04-22T17:38:08.059 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2283 (ObjNum 905 snap 297 seq_num 905) dirty exists 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3718: done (8 left) 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3719: done (7 left) 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3720: done (6 left) 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3723: done (5 left) 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3728: done (4 left) 2022-04-22T17:38:08.060 INFO:tasks.rados.rados.0.smithi115.stdout:3732: read oid 20 snap -1 2022-04-22T17:38:08.061 INFO:tasks.rados.rados.0.smithi115.stdout:3732: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:38:08.061 INFO:tasks.rados.rados.0.smithi115.stdout:3733: delete oid 42 current snap is 359 2022-04-22T17:38:08.066 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 1 to smithi11541303-1 2022-04-22T17:38:08.070 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 2 to smithi11541303-1 2022-04-22T17:38:08.070 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 3 to smithi11541303-1 2022-04-22T17:38:08.071 INFO:tasks.rados.rados.0.smithi115.stdout:3733: done (5 left) 2022-04-22T17:38:08.071 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 6 current snap is 359 2022-04-22T17:38:08.071 INFO:tasks.rados.rados.0.smithi115.stdout:3734: seq_num 1125 ranges {10387456=450560,10838016=786432,11624448=614400,12238848=204800} 2022-04-22T17:38:08.131 INFO:tasks.rados.rados.0.smithi115.stdout:3734: writing smithi11541303-6 from 10387456 to 10838016 tid 1 2022-04-22T17:38:08.134 INFO:tasks.rados.rados.0.smithi115.stdout:3734: writing smithi11541303-6 from 10838016 to 11624448 tid 2 2022-04-22T17:38:08.138 INFO:tasks.rados.rados.0.smithi115.stdout:3734: writing smithi11541303-6 from 11624448 to 12238848 tid 3 2022-04-22T17:38:08.139 INFO:tasks.rados.rados.0.smithi115.stdout:3734: writing smithi11541303-6 from 12238848 to 12443648 tid 4 2022-04-22T17:38:08.140 INFO:tasks.rados.rados.0.smithi115.stdout:3735: snap_remove snap 358 2022-04-22T17:38:08.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:08 smithi115 bash[45495]: cluster 2022-04-22T17:38:07.012662+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2022-04-22T17:38:08.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:08 smithi115 bash[45495]: cluster 2022-04-22T17:38:08.017069+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2022-04-22T17:38:08.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:08 smithi115 bash[46934]: cluster 2022-04-22T17:38:07.012662+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2022-04-22T17:38:08.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:08 smithi115 bash[46934]: cluster 2022-04-22T17:38:08.017069+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2022-04-22T17:38:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:08 smithi121 bash[36902]: cluster 2022-04-22T17:38:07.012662+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2022-04-22T17:38:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:08 smithi121 bash[36902]: cluster 2022-04-22T17:38:08.017069+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2022-04-22T17:38:09.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2068 (ObjNum 1064 snap 337 seq_num 1064) dirty exists 2022-04-22T17:38:09.063 INFO:tasks.rados.rados.0.smithi115.stdout:3726: expect (ObjNum 1092 snap 347 seq_num 1092) 2022-04-22T17:38:09.264 INFO:tasks.rados.rados.0.smithi115.stdout:3727: expect (ObjNum 1107 snap 352 seq_num 1107) 2022-04-22T17:38:09.482 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 4 to smithi11541303-1 2022-04-22T17:38:09.483 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 5 to smithi11541303-1 2022-04-22T17:38:09.483 INFO:tasks.rados.rados.0.smithi115.stdout:3730: finishing write tid 6 to smithi11541303-1 2022-04-22T17:38:09.483 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 1 v 1969 (ObjNum 1123 snap 359 seq_num 1123) dirty exists 2022-04-22T17:38:09.483 INFO:tasks.rados.rados.0.smithi115.stdout:3730: left oid 1 (ObjNum 1123 snap 359 seq_num 1123) 2022-04-22T17:38:09.483 INFO:tasks.rados.rados.0.smithi115.stdout:3732: expect (ObjNum 1078 snap 339 seq_num 1078) 2022-04-22T17:38:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:09 smithi121 bash[36902]: cluster 2022-04-22T17:38:07.982877+0000 mgr.y (mgr.34107) 468 : cluster [DBG] pgmap v869: 121 pgs: 121 active+clean; 355 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 316 KiB/s wr, 21 op/s 2022-04-22T17:38:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:09 smithi115 bash[46934]: cluster 2022-04-22T17:38:07.982877+0000 mgr.y (mgr.34107) 468 : cluster [DBG] pgmap v869: 121 pgs: 121 active+clean; 355 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 316 KiB/s wr, 21 op/s 2022-04-22T17:38:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:09 smithi115 bash[45495]: cluster 2022-04-22T17:38:07.982877+0000 mgr.y (mgr.34107) 468 : cluster [DBG] pgmap v869: 121 pgs: 121 active+clean; 355 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 316 KiB/s wr, 21 op/s 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 1 to smithi11541303-6 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 2 to smithi11541303-6 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 3 to smithi11541303-6 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 4 to smithi11541303-6 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 5 to smithi11541303-6 2022-04-22T17:38:09.673 INFO:tasks.rados.rados.0.smithi115.stdout:3734: finishing write tid 6 to smithi11541303-6 2022-04-22T17:38:09.674 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1945 (ObjNum 1125 snap 359 seq_num 1125) dirty exists 2022-04-22T17:38:09.674 INFO:tasks.rados.rados.0.smithi115.stdout:3734: left oid 6 (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:09.674 INFO:tasks.rados.rados.0.smithi115.stdout:3726: done (6 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3727: done (5 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3730: done (4 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3731: done (3 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3732: done (2 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3734: done (1 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3735: done (0 left) 2022-04-22T17:38:09.675 INFO:tasks.rados.rados.0.smithi115.stdout:3736: snap_create 2022-04-22T17:38:10.012 INFO:tasks.rados.rados.0.smithi115.stdout:3736: done (0 left) 2022-04-22T17:38:10.012 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 28 current snap is 360 2022-04-22T17:38:10.012 INFO:tasks.rados.rados.0.smithi115.stdout:3737: seq_num 1126 ranges {10919936=466944,11386880=720896,12107776=778240,12886016=106496} 2022-04-22T17:38:10.078 INFO:tasks.rados.rados.0.smithi115.stdout:3737: writing smithi11541303-28 from 10919936 to 11386880 tid 1 2022-04-22T17:38:10.082 INFO:tasks.rados.rados.0.smithi115.stdout:3737: writing smithi11541303-28 from 11386880 to 12107776 tid 2 2022-04-22T17:38:10.087 INFO:tasks.rados.rados.0.smithi115.stdout:3737: writing smithi11541303-28 from 12107776 to 12886016 tid 3 2022-04-22T17:38:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:3737: writing smithi11541303-28 from 12886016 to 12992512 tid 4 2022-04-22T17:38:10.088 INFO:tasks.rados.rados.0.smithi115.stdout:3738: snap_remove snap 359 2022-04-22T17:38:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:10 smithi121 bash[36902]: cluster 2022-04-22T17:38:09.048343+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2022-04-22T17:38:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:10 smithi121 bash[36902]: audit 2022-04-22T17:38:09.451807+0000 mon.a (mon.0) 1482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:10 smithi121 bash[36902]: cluster 2022-04-22T17:38:09.999566+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2022-04-22T17:38:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[45495]: cluster 2022-04-22T17:38:09.048343+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2022-04-22T17:38:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[45495]: audit 2022-04-22T17:38:09.451807+0000 mon.a (mon.0) 1482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:10.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[45495]: cluster 2022-04-22T17:38:09.999566+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2022-04-22T17:38:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[46934]: cluster 2022-04-22T17:38:09.048343+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2022-04-22T17:38:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[46934]: audit 2022-04-22T17:38:09.451807+0000 mon.a (mon.0) 1482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:10 smithi115 bash[46934]: cluster 2022-04-22T17:38:09.999566+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2022-04-22T17:38:11.083 INFO:tasks.rados.rados.0.smithi115.stdout:3738: done (1 left) 2022-04-22T17:38:11.084 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 9 current snap is 360 2022-04-22T17:38:11.084 INFO:tasks.rados.rados.0.smithi115.stdout:3739: seq_num 1127 ranges {2179072=483328,2662400=598016,3260416=581632,3842048=425984} 2022-04-22T17:38:11.101 INFO:tasks.rados.rados.0.smithi115.stdout:3739: writing smithi11541303-9 from 2179072 to 2662400 tid 1 2022-04-22T17:38:11.104 INFO:tasks.rados.rados.0.smithi115.stdout:3739: writing smithi11541303-9 from 2662400 to 3260416 tid 2 2022-04-22T17:38:11.107 INFO:tasks.rados.rados.0.smithi115.stdout:3739: writing smithi11541303-9 from 3260416 to 3842048 tid 3 2022-04-22T17:38:11.110 INFO:tasks.rados.rados.0.smithi115.stdout:3739: writing smithi11541303-9 from 3842048 to 4268032 tid 4 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3740: rollback oid 32 current snap is 360 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 347 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 1 to smithi11541303-28 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 2 to smithi11541303-28 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 3 to smithi11541303-28 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 4 to smithi11541303-28 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 5 to smithi11541303-28 2022-04-22T17:38:11.111 INFO:tasks.rados.rados.0.smithi115.stdout:3737: finishing write tid 6 to smithi11541303-28 2022-04-22T17:38:11.112 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 1911 (ObjNum 1126 snap 360 seq_num 1126) dirty exists 2022-04-22T17:38:11.112 INFO:tasks.rados.rados.0.smithi115.stdout:3737: left oid 28 (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:38:11.112 INFO:tasks.rados.rados.0.smithi115.stdout:3737: done (2 left) 2022-04-22T17:38:11.112 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 42 current snap is 360 2022-04-22T17:38:11.112 INFO:tasks.rados.rados.0.smithi115.stdout:3741: seq_num 1128 ranges {0=499712,499712=483328,983040=786432,1769472=335872} 2022-04-22T17:38:11.115 INFO:tasks.rados.rados.0.smithi115.stdout:3741: writing smithi11541303-42 from 0 to 499712 tid 1 2022-04-22T17:38:11.116 INFO:tasks.rados.rados.0.smithi115.stdout:3741: writing smithi11541303-42 from 499712 to 983040 tid 2 2022-04-22T17:38:11.121 INFO:tasks.rados.rados.0.smithi115.stdout:3741: writing smithi11541303-42 from 983040 to 1769472 tid 3 2022-04-22T17:38:11.124 INFO:tasks.rados.rados.0.smithi115.stdout:3741: writing smithi11541303-42 from 1769472 to 2105344 tid 4 2022-04-22T17:38:11.124 INFO:tasks.rados.rados.0.smithi115.stdout:3742: copy_from oid 50 from oid 16 current snap is 360 2022-04-22T17:38:11.124 INFO:tasks.rados.rados.0.smithi115.stdout:3740: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3740: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3740: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2053 (ObjNum 912 snap 299 seq_num 912) dirty exists 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 1 to smithi11541303-9 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 2 to smithi11541303-9 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 3 to smithi11541303-9 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 4 to smithi11541303-9 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 5 to smithi11541303-9 2022-04-22T17:38:11.125 INFO:tasks.rados.rados.0.smithi115.stdout:3740: done (3 left) 2022-04-22T17:38:11.126 INFO:tasks.rados.rados.0.smithi115.stdout:3743: read oid 21 snap -1 2022-04-22T17:38:11.126 INFO:tasks.rados.rados.0.smithi115.stdout:3743: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:38:11.126 INFO:tasks.rados.rados.0.smithi115.stdout:3744: copy_from oid 8 from oid 45 current snap is 360 2022-04-22T17:38:11.126 INFO:tasks.rados.rados.0.smithi115.stdout:3745: delete oid 19 current snap is 360 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:3742: finishing copy_from to smithi11541303-50 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:3742: got expected ENOENT (src dne) 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:3742: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:3739: finishing write tid 6 to smithi11541303-9 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2301 (ObjNum 1127 snap 360 seq_num 1127) dirty exists 2022-04-22T17:38:11.127 INFO:tasks.rados.rados.0.smithi115.stdout:3739: left oid 9 (ObjNum 1127 snap 360 seq_num 1127) 2022-04-22T17:38:11.135 INFO:tasks.rados.rados.0.smithi115.stdout:3739: done (5 left) 2022-04-22T17:38:11.135 INFO:tasks.rados.rados.0.smithi115.stdout:3742: done (4 left) 2022-04-22T17:38:11.135 INFO:tasks.rados.rados.0.smithi115.stdout:3745: done (3 left) 2022-04-22T17:38:11.135 INFO:tasks.rados.rados.0.smithi115.stdout:3746: copy_from oid 11 from oid 49 current snap is 360 2022-04-22T17:38:11.136 INFO:tasks.rados.rados.0.smithi115.stdout:3747: snap_create 2022-04-22T17:38:11.136 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 1 to smithi11541303-42 2022-04-22T17:38:11.138 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 2 to smithi11541303-42 2022-04-22T17:38:11.139 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 3 to smithi11541303-42 2022-04-22T17:38:11.139 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 4 to smithi11541303-42 2022-04-22T17:38:11.139 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 5 to smithi11541303-42 2022-04-22T17:38:11.143 INFO:tasks.rados.rados.0.smithi115.stdout:3741: finishing write tid 6 to smithi11541303-42 2022-04-22T17:38:11.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2306 (ObjNum 1128 snap 360 seq_num 1128) dirty exists 2022-04-22T17:38:11.144 INFO:tasks.rados.rados.0.smithi115.stdout:3741: left oid 42 (ObjNum 1128 snap 360 seq_num 1128) 2022-04-22T17:38:11.144 INFO:tasks.rados.rados.0.smithi115.stdout:3743: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:38:11.327 INFO:tasks.rados.rados.0.smithi115.stdout:3746: finishing copy_from to smithi11541303-11 2022-04-22T17:38:11.370 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2040 (ObjNum 1107 snap 352 seq_num 1107) dirty exists 2022-04-22T17:38:11.370 INFO:tasks.rados.rados.0.smithi115.stdout:3744: finishing copy_from to smithi11541303-8 2022-04-22T17:38:11.370 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 2076 (ObjNum 1111 snap 353 seq_num 1111) dirty exists 2022-04-22T17:38:11.371 INFO:tasks.rados.rados.0.smithi115.stdout:3746: finishing copy_from racing read to smithi11541303-11 2022-04-22T17:38:11.371 INFO:tasks.rados.rados.0.smithi115.stdout:3744: finishing copy_from racing read to smithi11541303-8 2022-04-22T17:38:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:11 smithi121 bash[36902]: cluster 2022-04-22T17:38:09.983835+0000 mgr.y (mgr.34107) 469 : cluster [DBG] pgmap v872: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 344 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 316 KiB/s wr, 9 op/s 2022-04-22T17:38:11.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:11 smithi115 bash[45495]: cluster 2022-04-22T17:38:09.983835+0000 mgr.y (mgr.34107) 469 : cluster [DBG] pgmap v872: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 344 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 316 KiB/s wr, 9 op/s 2022-04-22T17:38:11.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:11 smithi115 bash[46934]: cluster 2022-04-22T17:38:09.983835+0000 mgr.y (mgr.34107) 469 : cluster [DBG] pgmap v872: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 344 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 316 KiB/s wr, 9 op/s 2022-04-22T17:38:12.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:11.630Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:12.179 INFO:tasks.rados.rados.0.smithi115.stdout:3741: done (4 left) 2022-04-22T17:38:12.291 INFO:tasks.rados.rados.0.smithi115.stdout:3743: done (3 left) 2022-04-22T17:38:12.291 INFO:tasks.rados.rados.0.smithi115.stdout:3744: done (2 left) 2022-04-22T17:38:12.291 INFO:tasks.rados.rados.0.smithi115.stdout:3746: done (1 left) 2022-04-22T17:38:12.291 INFO:tasks.rados.rados.0.smithi115.stdout:3747: done (0 left) 2022-04-22T17:38:12.291 INFO:tasks.rados.rados.0.smithi115.stdout:3748: rollback oid 40 current snap is 361 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 40 to 347 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3749: read oid 10 snap -1 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3749: expect (ObjNum 1064 snap 337 seq_num 1064) 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3750: copy_from oid 4 from oid 30 current snap is 361 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 361 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3751: seq_num 1130 ranges {10657792=532480,11190272=655360,11845632=753664,12599296=196608} 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3751: writing smithi11541303-26 from 10657792 to 11190272 tid 1 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3751: writing smithi11541303-26 from 11190272 to 11845632 tid 2 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3751: writing smithi11541303-26 from 11845632 to 12599296 tid 3 2022-04-22T17:38:12.292 INFO:tasks.rados.rados.0.smithi115.stdout:3751: writing smithi11541303-26 from 12599296 to 12795904 tid 4 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:3752: copy_from oid 42 from oid 27 current snap is 361 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:3748: finishing rollback tid 0 to smithi11541303-40 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:3748: finishing rollback tid 1 to smithi11541303-40 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:3748: finishing rollback tid 2 to smithi11541303-40 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2071 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:38:12.293 INFO:tasks.rados.rados.0.smithi115.stdout:3749: expect (ObjNum 1064 snap 337 seq_num 1064) 2022-04-22T17:38:12.294 INFO:tasks.rados.rados.0.smithi115.stdout:3750: finishing copy_from to smithi11541303-4 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2083 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3750: finishing copy_from racing read to smithi11541303-4 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 1 to smithi11541303-26 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 2 to smithi11541303-26 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 3 to smithi11541303-26 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 4 to smithi11541303-26 2022-04-22T17:38:12.295 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 5 to smithi11541303-26 2022-04-22T17:38:12.296 INFO:tasks.rados.rados.0.smithi115.stdout:3751: finishing write tid 6 to smithi11541303-26 2022-04-22T17:38:12.296 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2058 (ObjNum 1130 snap 361 seq_num 1130) dirty exists 2022-04-22T17:38:12.296 INFO:tasks.rados.rados.0.smithi115.stdout:3751: left oid 26 (ObjNum 1130 snap 361 seq_num 1130) 2022-04-22T17:38:12.296 INFO:tasks.rados.rados.0.smithi115.stdout:3752: finishing copy_from to smithi11541303-42 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 42 v 2081 (ObjNum 1033 snap 324 seq_num 1033) dirty exists 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:3752: finishing copy_from racing read to smithi11541303-42 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:3748: done (4 left) 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:3749: done (3 left) 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:3750: done (2 left) 2022-04-22T17:38:12.298 INFO:tasks.rados.rados.0.smithi115.stdout:3751: done (1 left) 2022-04-22T17:38:12.304 INFO:tasks.rados.rados.0.smithi115.stdout:3752: done (0 left) 2022-04-22T17:38:12.304 INFO:tasks.rados.rados.0.smithi115.stdout:3753: snap_remove snap 360 2022-04-22T17:38:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:12 smithi121 bash[36902]: cluster 2022-04-22T17:38:11.070844+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2022-04-22T17:38:12.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:12 smithi115 bash[45495]: cluster 2022-04-22T17:38:11.070844+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2022-04-22T17:38:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:12 smithi115 bash[46934]: cluster 2022-04-22T17:38:11.070844+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2022-04-22T17:38:13.098 INFO:tasks.rados.rados.0.smithi115.stdout:3753: done (0 left) 2022-04-22T17:38:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:3754: rollback oid 9 current snap is 361 2022-04-22T17:38:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 9 to 349 2022-04-22T17:38:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 361 2022-04-22T17:38:13.099 INFO:tasks.rados.rados.0.smithi115.stdout:3755: seq_num 1131 ranges {3031040=548864,3579904=540672,4120576=565248,4685824=425984,5111808=73728} 2022-04-22T17:38:13.122 INFO:tasks.rados.rados.0.smithi115.stdout:3755: writing smithi11541303-32 from 3031040 to 3579904 tid 1 2022-04-22T17:38:13.125 INFO:tasks.rados.rados.0.smithi115.stdout:3755: writing smithi11541303-32 from 3579904 to 4120576 tid 2 2022-04-22T17:38:13.128 INFO:tasks.rados.rados.0.smithi115.stdout:3755: writing smithi11541303-32 from 4120576 to 4685824 tid 3 2022-04-22T17:38:13.130 INFO:tasks.rados.rados.0.smithi115.stdout:3755: writing smithi11541303-32 from 4685824 to 5111808 tid 4 2022-04-22T17:38:13.131 INFO:tasks.rados.rados.0.smithi115.stdout:3755: writing smithi11541303-32 from 5111808 to 5185536 tid 5 2022-04-22T17:38:13.131 INFO:tasks.rados.rados.0.smithi115.stdout:3756: read oid 6 snap 344 2022-04-22T17:38:13.131 INFO:tasks.rados.rados.0.smithi115.stdout:3756: expect deleted 2022-04-22T17:38:13.131 INFO:tasks.rados.rados.0.smithi115.stdout:3757: snap_remove snap 347 2022-04-22T17:38:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:13 smithi121 bash[36902]: cluster 2022-04-22T17:38:11.984826+0000 mgr.y (mgr.34107) 470 : cluster [DBG] pgmap v875: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:38:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:13 smithi121 bash[36902]: cluster 2022-04-22T17:38:12.079351+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2022-04-22T17:38:13.534 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:13 smithi115 bash[45495]: cluster 2022-04-22T17:38:11.984826+0000 mgr.y (mgr.34107) 470 : cluster [DBG] pgmap v875: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:38:13.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:13 smithi115 bash[45495]: cluster 2022-04-22T17:38:12.079351+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2022-04-22T17:38:13.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:13 smithi115 bash[46934]: cluster 2022-04-22T17:38:11.984826+0000 mgr.y (mgr.34107) 470 : cluster [DBG] pgmap v875: 121 pgs: 1 active+clean+snaptrim, 120 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:38:13.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:13 smithi115 bash[46934]: cluster 2022-04-22T17:38:12.079351+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2022-04-22T17:38:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:13.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:14.107 INFO:tasks.rados.rados.0.smithi115.stdout:3757: done (3 left) 2022-04-22T17:38:14.107 INFO:tasks.rados.rados.0.smithi115.stdout:3758: copy_from oid 16 from oid 30 current snap is 361 2022-04-22T17:38:14.107 INFO:tasks.rados.rados.0.smithi115.stdout:3754: finishing rollback tid 0 to smithi11541303-9 2022-04-22T17:38:14.107 INFO:tasks.rados.rados.0.smithi115.stdout:3754: finishing rollback tid 1 to smithi11541303-9 2022-04-22T17:38:14.107 INFO:tasks.rados.rados.0.smithi115.stdout:3754: finishing rollback tid 2 to smithi11541303-9 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2311 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 1 to smithi11541303-32 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 2 to smithi11541303-32 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:3754: done (3 left) 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:3756: done (2 left) 2022-04-22T17:38:14.108 INFO:tasks.rados.rados.0.smithi115.stdout:3759: snap_remove snap 349 2022-04-22T17:38:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:14 smithi121 bash[36902]: cluster 2022-04-22T17:38:13.085656+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2022-04-22T17:38:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:14 smithi121 bash[36902]: audit 2022-04-22T17:38:13.333192+0000 mon.a (mon.0) 1487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:38:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:14 smithi121 bash[36902]: audit 2022-04-22T17:38:13.333594+0000 mon.a (mon.0) 1488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:38:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[46934]: cluster 2022-04-22T17:38:13.085656+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2022-04-22T17:38:14.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[46934]: audit 2022-04-22T17:38:13.333192+0000 mon.a (mon.0) 1487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:38:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[46934]: audit 2022-04-22T17:38:13.333594+0000 mon.a (mon.0) 1488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:38:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[45495]: cluster 2022-04-22T17:38:13.085656+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2022-04-22T17:38:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[45495]: audit 2022-04-22T17:38:13.333192+0000 mon.a (mon.0) 1487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:38:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:14 smithi115 bash[45495]: audit 2022-04-22T17:38:13.333594+0000 mon.a (mon.0) 1488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:38:15.020 INFO:tasks.rados.rados.0.smithi115.stdout:3759: done (2 left) 2022-04-22T17:38:15.020 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 361 2022-04-22T17:38:15.020 INFO:tasks.rados.rados.0.smithi115.stdout:3760: seq_num 1132 ranges {1949696=565248,2514944=425984,2940928=770048,3710976=409600} 2022-04-22T17:38:15.035 INFO:tasks.rados.rados.0.smithi115.stdout:3760: writing smithi11541303-22 from 1949696 to 2514944 tid 1 2022-04-22T17:38:15.038 INFO:tasks.rados.rados.0.smithi115.stdout:3760: writing smithi11541303-22 from 2514944 to 2940928 tid 2 2022-04-22T17:38:15.043 INFO:tasks.rados.rados.0.smithi115.stdout:3760: writing smithi11541303-22 from 2940928 to 3710976 tid 3 2022-04-22T17:38:15.047 INFO:tasks.rados.rados.0.smithi115.stdout:3760: writing smithi11541303-22 from 3710976 to 4120576 tid 4 2022-04-22T17:38:15.048 INFO:tasks.rados.rados.0.smithi115.stdout:3761: snap_create 2022-04-22T17:38:15.048 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 3 to smithi11541303-32 2022-04-22T17:38:15.048 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 4 to smithi11541303-32 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 5 to smithi11541303-32 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 6 to smithi11541303-32 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3755: finishing write tid 7 to smithi11541303-32 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2064 (ObjNum 1131 snap 361 seq_num 1131) dirty exists 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3755: left oid 32 (ObjNum 1131 snap 361 seq_num 1131) 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3758: finishing copy_from to smithi11541303-16 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 2083 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:38:15.049 INFO:tasks.rados.rados.0.smithi115.stdout:3758: finishing copy_from racing read to smithi11541303-16 2022-04-22T17:38:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 1 to smithi11541303-22 2022-04-22T17:38:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 2 to smithi11541303-22 2022-04-22T17:38:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 3 to smithi11541303-22 2022-04-22T17:38:15.054 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 4 to smithi11541303-22 2022-04-22T17:38:15.054 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 5 to smithi11541303-22 2022-04-22T17:38:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3760: finishing write tid 6 to smithi11541303-22 2022-04-22T17:38:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1887 (ObjNum 1132 snap 361 seq_num 1132) dirty exists 2022-04-22T17:38:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3760: left oid 22 (ObjNum 1132 snap 361 seq_num 1132) 2022-04-22T17:38:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:15 smithi121 bash[36902]: cluster 2022-04-22T17:38:13.985849+0000 mgr.y (mgr.34107) 471 : cluster [DBG] pgmap v878: 121 pgs: 121 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 18 op/s 2022-04-22T17:38:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:15 smithi121 bash[36902]: cluster 2022-04-22T17:38:14.094679+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2022-04-22T17:38:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:15 smithi121 bash[36902]: audit 2022-04-22T17:38:14.454786+0000 mon.a (mon.0) 1490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:15 smithi121 bash[36902]: cluster 2022-04-22T17:38:15.004873+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2022-04-22T17:38:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[45495]: cluster 2022-04-22T17:38:13.985849+0000 mgr.y (mgr.34107) 471 : cluster [DBG] pgmap v878: 121 pgs: 121 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 18 op/s 2022-04-22T17:38:15.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[45495]: cluster 2022-04-22T17:38:14.094679+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[45495]: audit 2022-04-22T17:38:14.454786+0000 mon.a (mon.0) 1490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[45495]: cluster 2022-04-22T17:38:15.004873+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[46934]: cluster 2022-04-22T17:38:13.985849+0000 mgr.y (mgr.34107) 471 : cluster [DBG] pgmap v878: 121 pgs: 121 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 18 op/s 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[46934]: cluster 2022-04-22T17:38:14.094679+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[46934]: audit 2022-04-22T17:38:14.454786+0000 mon.a (mon.0) 1490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:15.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:15 smithi115 bash[46934]: cluster 2022-04-22T17:38:15.004873+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2022-04-22T17:38:16.030 INFO:tasks.rados.rados.0.smithi115.stdout:3755: done (3 left) 2022-04-22T17:38:16.030 INFO:tasks.rados.rados.0.smithi115.stdout:3758: done (2 left) 2022-04-22T17:38:16.030 INFO:tasks.rados.rados.0.smithi115.stdout:3760: done (1 left) 2022-04-22T17:38:16.030 INFO:tasks.rados.rados.0.smithi115.stdout:3761: done (0 left) 2022-04-22T17:38:16.030 INFO:tasks.rados.rados.0.smithi115.stdout:3762: snap_remove snap 361 2022-04-22T17:38:17.038 INFO:tasks.rados.rados.0.smithi115.stdout:3762: done (0 left) 2022-04-22T17:38:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 20 current snap is 362 2022-04-22T17:38:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:3763: seq_num 1133 ranges {4382720=581632,4964352=753664,5718016=573440,6291456=278528} 2022-04-22T17:38:17.083 INFO:tasks.rados.rados.0.smithi115.stdout:3763: writing smithi11541303-20 from 4382720 to 4964352 tid 1 2022-04-22T17:38:17.083 INFO:tasks.rados.rados.0.smithi115.stdout:3763: writing smithi11541303-20 from 4964352 to 5718016 tid 2 2022-04-22T17:38:17.084 INFO:tasks.rados.rados.0.smithi115.stdout:3763: writing smithi11541303-20 from 5718016 to 6291456 tid 3 2022-04-22T17:38:17.084 INFO:tasks.rados.rados.0.smithi115.stdout:3763: writing smithi11541303-20 from 6291456 to 6569984 tid 4 2022-04-22T17:38:17.084 INFO:tasks.rados.rados.0.smithi115.stdout:3764: read oid 45 snap -1 2022-04-22T17:38:17.084 INFO:tasks.rados.rados.0.smithi115.stdout:3764: expect (ObjNum 1111 snap 353 seq_num 1111) 2022-04-22T17:38:17.084 INFO:tasks.rados.rados.0.smithi115.stdout:3765: snap_remove snap 356 2022-04-22T17:38:17.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:17 smithi115 bash[45495]: cluster 2022-04-22T17:38:15.987098+0000 mgr.y (mgr.34107) 472 : cluster [DBG] pgmap v881: 121 pgs: 121 active+clean; 342 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 13 op/s 2022-04-22T17:38:17.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:17 smithi115 bash[45495]: cluster 2022-04-22T17:38:16.016774+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2022-04-22T17:38:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:17 smithi115 bash[46934]: cluster 2022-04-22T17:38:15.987098+0000 mgr.y (mgr.34107) 472 : cluster [DBG] pgmap v881: 121 pgs: 121 active+clean; 342 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 13 op/s 2022-04-22T17:38:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:17 smithi115 bash[46934]: cluster 2022-04-22T17:38:16.016774+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2022-04-22T17:38:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:17 smithi121 bash[36902]: cluster 2022-04-22T17:38:15.987098+0000 mgr.y (mgr.34107) 472 : cluster [DBG] pgmap v881: 121 pgs: 121 active+clean; 342 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 13 op/s 2022-04-22T17:38:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:17 smithi121 bash[36902]: cluster 2022-04-22T17:38:16.016774+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2022-04-22T17:38:18.054 INFO:tasks.rados.rados.0.smithi115.stdout:3765: done (2 left) 2022-04-22T17:38:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 22 current snap is 362 2022-04-22T17:38:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 1 to smithi11541303-20 2022-04-22T17:38:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 2 to smithi11541303-20 2022-04-22T17:38:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 3 to smithi11541303-20 2022-04-22T17:38:18.055 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 4 to smithi11541303-20 2022-04-22T17:38:18.056 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 5 to smithi11541303-20 2022-04-22T17:38:18.056 INFO:tasks.rados.rados.0.smithi115.stdout:3763: finishing write tid 6 to smithi11541303-20 2022-04-22T17:38:18.056 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 2200 (ObjNum 1133 snap 362 seq_num 1133) dirty exists 2022-04-22T17:38:18.056 INFO:tasks.rados.rados.0.smithi115.stdout:3763: left oid 20 (ObjNum 1133 snap 362 seq_num 1133) 2022-04-22T17:38:18.056 INFO:tasks.rados.rados.0.smithi115.stdout:3764: expect (ObjNum 1111 snap 353 seq_num 1111) 2022-04-22T17:38:18.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:18 smithi115 bash[45495]: cluster 2022-04-22T17:38:17.024121+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2022-04-22T17:38:18.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:18 smithi115 bash[46934]: cluster 2022-04-22T17:38:17.024121+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2022-04-22T17:38:18.458 INFO:tasks.rados.rados.0.smithi115.stdout:3766: seq_num 1134 ranges {4120576=598016,4718592=638976,5357568=737280,6094848=229376} 2022-04-22T17:38:18.493 INFO:tasks.rados.rados.0.smithi115.stdout:3766: writing smithi11541303-22 from 4120576 to 4718592 tid 1 2022-04-22T17:38:18.495 INFO:tasks.rados.rados.0.smithi115.stdout:3766: writing smithi11541303-22 from 4718592 to 5357568 tid 2 2022-04-22T17:38:18.500 INFO:tasks.rados.rados.0.smithi115.stdout:3766: writing smithi11541303-22 from 5357568 to 6094848 tid 3 2022-04-22T17:38:18.501 INFO:tasks.rados.rados.0.smithi115.stdout:3766: writing smithi11541303-22 from 6094848 to 6324224 tid 4 2022-04-22T17:38:18.501 INFO:tasks.rados.rados.0.smithi115.stdout:3763: done (2 left) 2022-04-22T17:38:18.501 INFO:tasks.rados.rados.0.smithi115.stdout:3764: done (1 left) 2022-04-22T17:38:18.502 INFO:tasks.rados.rados.0.smithi115.stdout:3767: setattr oid 44 current snap is 362 2022-04-22T17:38:18.502 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 1 to smithi11541303-22 2022-04-22T17:38:18.502 INFO:tasks.rados.rados.0.smithi115.stdout:3768: snap_create 2022-04-22T17:38:18.503 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 2 to smithi11541303-22 2022-04-22T17:38:18.506 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 3 to smithi11541303-22 2022-04-22T17:38:18.506 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 2132 (ObjNum 1096 snap 347 seq_num 1096) dirty exists 2022-04-22T17:38:18.507 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 4 to smithi11541303-22 2022-04-22T17:38:18.507 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 5 to smithi11541303-22 2022-04-22T17:38:18.508 INFO:tasks.rados.rados.0.smithi115.stdout:3766: finishing write tid 6 to smithi11541303-22 2022-04-22T17:38:18.508 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1892 (ObjNum 1134 snap 362 seq_num 1134) dirty exists 2022-04-22T17:38:18.509 INFO:tasks.rados.rados.0.smithi115.stdout:3766: left oid 22 (ObjNum 1134 snap 362 seq_num 1134) 2022-04-22T17:38:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:18 smithi121 bash[36902]: cluster 2022-04-22T17:38:17.024121+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2022-04-22T17:38:19.050 INFO:tasks.rados.rados.0.smithi115.stdout:3766: done (2 left) 2022-04-22T17:38:19.061 INFO:tasks.rados.rados.0.smithi115.stdout:3767: done (1 left) 2022-04-22T17:38:19.062 INFO:tasks.rados.rados.0.smithi115.stdout:3768: done (0 left) 2022-04-22T17:38:19.062 INFO:tasks.rados.rados.0.smithi115.stdout:3769: delete oid 11 current snap is 363 2022-04-22T17:38:19.062 INFO:tasks.rados.rados.0.smithi115.stdout:3769: done (0 left) 2022-04-22T17:38:19.062 INFO:tasks.rados.rados.0.smithi115.stdout:3770: snap_remove snap 362 2022-04-22T17:38:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:19 smithi121 bash[36902]: cluster 2022-04-22T17:38:17.988008+0000 mgr.y (mgr.34107) 473 : cluster [DBG] pgmap v884: 121 pgs: 121 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T17:38:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:19 smithi121 bash[36902]: cluster 2022-04-22T17:38:18.040984+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2022-04-22T17:38:19.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:19 smithi115 bash[45495]: cluster 2022-04-22T17:38:17.988008+0000 mgr.y (mgr.34107) 473 : cluster [DBG] pgmap v884: 121 pgs: 121 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T17:38:19.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:19 smithi115 bash[45495]: cluster 2022-04-22T17:38:18.040984+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2022-04-22T17:38:19.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:19 smithi115 bash[46934]: cluster 2022-04-22T17:38:17.988008+0000 mgr.y (mgr.34107) 473 : cluster [DBG] pgmap v884: 121 pgs: 121 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T17:38:19.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:19 smithi115 bash[46934]: cluster 2022-04-22T17:38:18.040984+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2022-04-22T17:38:20.023 INFO:tasks.rados.rados.0.smithi115.stdout:3770: done (0 left) 2022-04-22T17:38:20.023 INFO:tasks.rados.rados.0.smithi115.stdout:3771: read oid 26 snap -1 2022-04-22T17:38:20.023 INFO:tasks.rados.rados.0.smithi115.stdout:3771: expect (ObjNum 1130 snap 361 seq_num 1130) 2022-04-22T17:38:20.023 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 363 2022-04-22T17:38:20.023 INFO:tasks.rados.rados.0.smithi115.stdout:3772: seq_num 1136 ranges {0=630784,630784=409600,1040384=753664,1794048=417792,2211840=24576} 2022-04-22T17:38:20.027 INFO:tasks.rados.rados.0.smithi115.stdout:3772: writing smithi11541303-29 from 0 to 630784 tid 1 2022-04-22T17:38:20.030 INFO:tasks.rados.rados.0.smithi115.stdout:3772: writing smithi11541303-29 from 630784 to 1040384 tid 2 2022-04-22T17:38:20.037 INFO:tasks.rados.rados.0.smithi115.stdout:3772: writing smithi11541303-29 from 1040384 to 1794048 tid 3 2022-04-22T17:38:20.039 INFO:tasks.rados.rados.0.smithi115.stdout:3772: writing smithi11541303-29 from 1794048 to 2211840 tid 4 2022-04-22T17:38:20.039 INFO:tasks.rados.rados.0.smithi115.stdout:3772: writing smithi11541303-29 from 2211840 to 2236416 tid 5 2022-04-22T17:38:20.040 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 363 2022-04-22T17:38:20.040 INFO:tasks.rados.rados.0.smithi115.stdout:3773: seq_num 1137 ranges {1884160=647168,2531328=696320,3227648=557056,3784704=352256} 2022-04-22T17:38:20.054 INFO:tasks.rados.rados.0.smithi115.stdout:3773: writing smithi11541303-50 from 1884160 to 2531328 tid 1 2022-04-22T17:38:20.058 INFO:tasks.rados.rados.0.smithi115.stdout:3773: writing smithi11541303-50 from 2531328 to 3227648 tid 2 2022-04-22T17:38:20.061 INFO:tasks.rados.rados.0.smithi115.stdout:3773: writing smithi11541303-50 from 3227648 to 3784704 tid 3 2022-04-22T17:38:20.064 INFO:tasks.rados.rados.0.smithi115.stdout:3773: writing smithi11541303-50 from 3784704 to 4136960 tid 4 2022-04-22T17:38:20.064 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 363 2022-04-22T17:38:20.064 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 1 to smithi11541303-29 2022-04-22T17:38:20.064 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 2 to smithi11541303-29 2022-04-22T17:38:20.064 INFO:tasks.rados.rados.0.smithi115.stdout:3774: seq_num 1138 ranges {9306112=663552,9969664=581632,10551296=712704,11264000=311296} 2022-04-22T17:38:20.120 INFO:tasks.rados.rados.0.smithi115.stdout:3774: writing smithi11541303-15 from 9306112 to 9969664 tid 1 2022-04-22T17:38:20.123 INFO:tasks.rados.rados.0.smithi115.stdout:3774: writing smithi11541303-15 from 9969664 to 10551296 tid 2 2022-04-22T17:38:20.127 INFO:tasks.rados.rados.0.smithi115.stdout:3774: writing smithi11541303-15 from 10551296 to 11264000 tid 3 2022-04-22T17:38:20.129 INFO:tasks.rados.rados.0.smithi115.stdout:3774: writing smithi11541303-15 from 11264000 to 11575296 tid 4 2022-04-22T17:38:20.129 INFO:tasks.rados.rados.0.smithi115.stdout:3775: snap_remove snap 344 2022-04-22T17:38:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:20 smithi121 bash[36902]: cluster 2022-04-22T17:38:19.037195+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2022-04-22T17:38:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:20 smithi121 bash[36902]: audit 2022-04-22T17:38:19.457857+0000 mon.a (mon.0) 1496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:20 smithi121 bash[36902]: cluster 2022-04-22T17:38:20.005024+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2022-04-22T17:38:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[45495]: cluster 2022-04-22T17:38:19.037195+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2022-04-22T17:38:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[45495]: audit 2022-04-22T17:38:19.457857+0000 mon.a (mon.0) 1496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:20.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[45495]: cluster 2022-04-22T17:38:20.005024+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2022-04-22T17:38:20.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[46934]: cluster 2022-04-22T17:38:19.037195+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2022-04-22T17:38:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[46934]: audit 2022-04-22T17:38:19.457857+0000 mon.a (mon.0) 1496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:20 smithi115 bash[46934]: cluster 2022-04-22T17:38:20.005024+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2022-04-22T17:38:21.076 INFO:tasks.rados.rados.0.smithi115.stdout:3775: done (4 left) 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3776: rollback oid 2 current snap is 363 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 3 to smithi11541303-29 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 4 to smithi11541303-29 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 5 to smithi11541303-29 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 6 to smithi11541303-29 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:3772: finishing write tid 7 to smithi11541303-29 2022-04-22T17:38:21.115 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 1917 (ObjNum 1136 snap 363 seq_num 1136) dirty exists 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3772: left oid 29 (ObjNum 1136 snap 363 seq_num 1136) 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 1 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 2 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 3 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 4 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 5 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: finishing write tid 6 to smithi11541303-50 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2088 (ObjNum 1137 snap 363 seq_num 1137) dirty exists 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3773: left oid 50 (ObjNum 1137 snap 363 seq_num 1137) 2022-04-22T17:38:21.116 INFO:tasks.rados.rados.0.smithi115.stdout:3771: expect (ObjNum 1130 snap 361 seq_num 1130) 2022-04-22T17:38:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:21 smithi121 bash[36902]: cluster 2022-04-22T17:38:19.988986+0000 mgr.y (mgr.34107) 474 : cluster [DBG] pgmap v887: 121 pgs: 121 active+clean; 345 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.8 MiB/s wr, 4 op/s 2022-04-22T17:38:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:21 smithi115 bash[46934]: cluster 2022-04-22T17:38:19.988986+0000 mgr.y (mgr.34107) 474 : cluster [DBG] pgmap v887: 121 pgs: 121 active+clean; 345 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.8 MiB/s wr, 4 op/s 2022-04-22T17:38:21.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:21 smithi115 bash[45495]: cluster 2022-04-22T17:38:19.988986+0000 mgr.y (mgr.34107) 474 : cluster [DBG] pgmap v887: 121 pgs: 121 active+clean; 345 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.8 MiB/s wr, 4 op/s 2022-04-22T17:38:22.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:21.631Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:22.195 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 1 to smithi11541303-15 2022-04-22T17:38:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 2 to smithi11541303-15 2022-04-22T17:38:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 3 to smithi11541303-15 2022-04-22T17:38:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 4 to smithi11541303-15 2022-04-22T17:38:22.196 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 5 to smithi11541303-15 2022-04-22T17:38:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:3774: finishing write tid 6 to smithi11541303-15 2022-04-22T17:38:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 2061 (ObjNum 1138 snap 363 seq_num 1138) dirty exists 2022-04-22T17:38:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:3774: left oid 15 (ObjNum 1138 snap 363 seq_num 1138) 2022-04-22T17:38:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:3771: done (4 left) 2022-04-22T17:38:22.197 INFO:tasks.rados.rados.0.smithi115.stdout:3772: done (3 left) 2022-04-22T17:38:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:3773: done (2 left) 2022-04-22T17:38:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:3774: done (1 left) 2022-04-22T17:38:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:3776: done (0 left) 2022-04-22T17:38:22.198 INFO:tasks.rados.rados.0.smithi115.stdout:3777: read oid 40 snap -1 2022-04-22T17:38:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:3777: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:22.199 INFO:tasks.rados.rados.0.smithi115.stdout:3778: snap_create 2022-04-22T17:38:22.203 INFO:tasks.rados.rados.0.smithi115.stdout:3777: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:22 smithi121 bash[36902]: cluster 2022-04-22T17:38:21.063532+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2022-04-22T17:38:22.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:22 smithi115 bash[45495]: cluster 2022-04-22T17:38:21.063532+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2022-04-22T17:38:22.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:22 smithi115 bash[46934]: cluster 2022-04-22T17:38:21.063532+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2022-04-22T17:38:23.095 INFO:tasks.rados.rados.0.smithi115.stdout:3777: done (1 left) 2022-04-22T17:38:23.095 INFO:tasks.rados.rados.0.smithi115.stdout:3778: done (0 left) 2022-04-22T17:38:23.095 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 11 current snap is 364 2022-04-22T17:38:23.095 INFO:tasks.rados.rados.0.smithi115.stdout:3779: seq_num 1139 ranges {0=679936,679936=458752,1138688=524288,1662976=540672,2203648=81920} 2022-04-22T17:38:23.100 INFO:tasks.rados.rados.0.smithi115.stdout:3779: writing smithi11541303-11 from 0 to 679936 tid 1 2022-04-22T17:38:23.103 INFO:tasks.rados.rados.0.smithi115.stdout:3779: writing smithi11541303-11 from 679936 to 1138688 tid 2 2022-04-22T17:38:23.108 INFO:tasks.rados.rados.0.smithi115.stdout:3779: writing smithi11541303-11 from 1138688 to 1662976 tid 3 2022-04-22T17:38:23.113 INFO:tasks.rados.rados.0.smithi115.stdout:3779: writing smithi11541303-11 from 1662976 to 2203648 tid 4 2022-04-22T17:38:23.114 INFO:tasks.rados.rados.0.smithi115.stdout:3779: writing smithi11541303-11 from 2203648 to 2285568 tid 5 2022-04-22T17:38:23.114 INFO:tasks.rados.rados.0.smithi115.stdout:3780: read oid 3 snap -1 2022-04-22T17:38:23.114 INFO:tasks.rados.rados.0.smithi115.stdout:3780: expect (ObjNum 1096 snap 347 seq_num 1096) 2022-04-22T17:38:23.114 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 5 current snap is 364 2022-04-22T17:38:23.114 INFO:tasks.rados.rados.0.smithi115.stdout:3781: seq_num 1140 ranges {1916928=696320,2613248=745472,3358720=729088,4087808=131072} 2022-04-22T17:38:23.132 INFO:tasks.rados.rados.0.smithi115.stdout:3781: writing smithi11541303-5 from 1916928 to 2613248 tid 1 2022-04-22T17:38:23.136 INFO:tasks.rados.rados.0.smithi115.stdout:3781: writing smithi11541303-5 from 2613248 to 3358720 tid 2 2022-04-22T17:38:23.140 INFO:tasks.rados.rados.0.smithi115.stdout:3781: writing smithi11541303-5 from 3358720 to 4087808 tid 3 2022-04-22T17:38:23.192 INFO:tasks.rados.rados.0.smithi115.stdout:3781: writing smithi11541303-5 from 4087808 to 4218880 tid 4 2022-04-22T17:38:23.192 INFO:tasks.rados.rados.0.smithi115.stdout:3782: read oid 28 snap -1 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3782: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 364 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3783: seq_num 1141 ranges {4841472=712704,5554176=679936,6234112=540672,6774784=385024} 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3783: writing smithi11541303-49 from 4841472 to 5554176 tid 1 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3783: writing smithi11541303-49 from 5554176 to 6234112 tid 2 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3783: writing smithi11541303-49 from 6234112 to 6774784 tid 3 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3783: writing smithi11541303-49 from 6774784 to 7159808 tid 4 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3784: snap_create 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 1 to smithi11541303-11 2022-04-22T17:38:23.193 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 2 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 3 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 4 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 5 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 6 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: finishing write tid 7 to smithi11541303-11 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2099 (ObjNum 1139 snap 364 seq_num 1139) dirty exists 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3779: left oid 11 (ObjNum 1139 snap 364 seq_num 1139) 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 1 to smithi11541303-5 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 2 to smithi11541303-5 2022-04-22T17:38:23.194 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 3 to smithi11541303-5 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 4 to smithi11541303-5 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 5 to smithi11541303-5 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3781: finishing write tid 6 to smithi11541303-5 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2353 (ObjNum 1140 snap 364 seq_num 1140) dirty exists 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3781: left oid 5 (ObjNum 1140 snap 364 seq_num 1140) 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 1 to smithi11541303-49 2022-04-22T17:38:23.195 INFO:tasks.rados.rados.0.smithi115.stdout:3780: expect (ObjNum 1096 snap 347 seq_num 1096) 2022-04-22T17:38:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:23 smithi121 bash[36902]: cluster 2022-04-22T17:38:21.989982+0000 mgr.y (mgr.34107) 475 : cluster [DBG] pgmap v890: 121 pgs: 121 active+clean; 316 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 11 op/s 2022-04-22T17:38:23.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:23 smithi115 bash[45495]: cluster 2022-04-22T17:38:21.989982+0000 mgr.y (mgr.34107) 475 : cluster [DBG] pgmap v890: 121 pgs: 121 active+clean; 316 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 11 op/s 2022-04-22T17:38:23.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:23 smithi115 bash[46934]: cluster 2022-04-22T17:38:21.989982+0000 mgr.y (mgr.34107) 475 : cluster [DBG] pgmap v890: 121 pgs: 121 active+clean; 316 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 11 op/s 2022-04-22T17:38:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:23.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:24.309 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 2 to smithi11541303-49 2022-04-22T17:38:24.309 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 3 to smithi11541303-49 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 4 to smithi11541303-49 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 5 to smithi11541303-49 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:3783: finishing write tid 6 to smithi11541303-49 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2359 (ObjNum 1141 snap 364 seq_num 1141) dirty exists 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:3783: left oid 49 (ObjNum 1141 snap 364 seq_num 1141) 2022-04-22T17:38:24.310 INFO:tasks.rados.rados.0.smithi115.stdout:3782: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:38:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:24 smithi121 bash[36902]: cluster 2022-04-22T17:38:23.083162+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2022-04-22T17:38:24.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:24 smithi115 bash[46934]: cluster 2022-04-22T17:38:23.083162+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2022-04-22T17:38:24.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:24 smithi115 bash[45495]: cluster 2022-04-22T17:38:23.083162+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2022-04-22T17:38:25.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: cluster 2022-04-22T17:38:23.990631+0000 mgr.y (mgr.34107) 476 : cluster [DBG] pgmap v892: 121 pgs: 121 active+clean; 277 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: cluster 2022-04-22T17:38:24.092808+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: audit 2022-04-22T17:38:24.460862+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: audit 2022-04-22T17:38:24.685942+0000 mon.a (mon.0) 1502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: audit 2022-04-22T17:38:24.688746+0000 mon.a (mon.0) 1503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: audit 2022-04-22T17:38:24.690305+0000 mon.a (mon.0) 1504 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:38:25.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: cluster 2022-04-22T17:38:25.005429+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[45495]: audit 2022-04-22T17:38:25.039069+0000 mon.a (mon.0) 1506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: cluster 2022-04-22T17:38:23.990631+0000 mgr.y (mgr.34107) 476 : cluster [DBG] pgmap v892: 121 pgs: 121 active+clean; 277 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: cluster 2022-04-22T17:38:24.092808+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: audit 2022-04-22T17:38:24.460862+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: audit 2022-04-22T17:38:24.685942+0000 mon.a (mon.0) 1502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: audit 2022-04-22T17:38:24.688746+0000 mon.a (mon.0) 1503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: audit 2022-04-22T17:38:24.690305+0000 mon.a (mon.0) 1504 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:38:25.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: cluster 2022-04-22T17:38:25.005429+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2022-04-22T17:38:25.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:25 smithi115 bash[46934]: audit 2022-04-22T17:38:25.039069+0000 mon.a (mon.0) 1506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:38:25.341 INFO:tasks.rados.rados.0.smithi115.stdout:3779: done (5 left) 2022-04-22T17:38:25.342 INFO:tasks.rados.rados.0.smithi115.stdout:3780: done (4 left) 2022-04-22T17:38:25.342 INFO:tasks.rados.rados.0.smithi115.stdout:3781: done (3 left) 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:3782: done (2 left) 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:3783: done (1 left) 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:3784: done (0 left) 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:3785: rollback oid 22 current snap is 365 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 22 to 364 2022-04-22T17:38:25.343 INFO:tasks.rados.rados.0.smithi115.stdout:3786: snap_create 2022-04-22T17:38:25.348 INFO:tasks.rados.rados.0.smithi115.stdout:3785: finishing rollback tid 0 to smithi11541303-22 2022-04-22T17:38:25.349 INFO:tasks.rados.rados.0.smithi115.stdout:3785: finishing rollback tid 1 to smithi11541303-22 2022-04-22T17:38:25.349 INFO:tasks.rados.rados.0.smithi115.stdout:3785: finishing rollback tid 2 to smithi11541303-22 2022-04-22T17:38:25.349 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 22 v 1895 (ObjNum 1134 snap 362 seq_num 1134) dirty exists 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: cluster 2022-04-22T17:38:23.990631+0000 mgr.y (mgr.34107) 476 : cluster [DBG] pgmap v892: 121 pgs: 121 active+clean; 277 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: cluster 2022-04-22T17:38:24.092808+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: audit 2022-04-22T17:38:24.460862+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: audit 2022-04-22T17:38:24.685942+0000 mon.a (mon.0) 1502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: audit 2022-04-22T17:38:24.688746+0000 mon.a (mon.0) 1503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: audit 2022-04-22T17:38:24.690305+0000 mon.a (mon.0) 1504 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: cluster 2022-04-22T17:38:25.005429+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2022-04-22T17:38:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:25 smithi121 bash[36902]: audit 2022-04-22T17:38:25.039069+0000 mon.a (mon.0) 1506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:38:26.116 INFO:tasks.rados.rados.0.smithi115.stdout:3785: done (1 left) 2022-04-22T17:38:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:3786: done (0 left) 2022-04-22T17:38:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:3787: copy_from oid 7 from oid 45 current snap is 366 2022-04-22T17:38:26.134 INFO:tasks.rados.rados.0.smithi115.stdout:3788: snap_remove snap 364 2022-04-22T17:38:27.127 INFO:tasks.rados.rados.0.smithi115.stdout:3788: done (1 left) 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3789: read oid 16 snap 365 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3789: expect (ObjNum 1076 snap 339 seq_num 1076) 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3790: copy_from oid 5 from oid 40 current snap is 366 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3787: finishing copy_from to smithi11541303-7 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 2076 (ObjNum 1111 snap 353 seq_num 1111) dirty exists 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3787: finishing copy_from racing read to smithi11541303-7 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3787: done (2 left) 2022-04-22T17:38:27.128 INFO:tasks.rados.rados.0.smithi115.stdout:3791: snap_create 2022-04-22T17:38:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:3790: finishing copy_from to smithi11541303-5 2022-04-22T17:38:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2071 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:38:27.148 INFO:tasks.rados.rados.0.smithi115.stdout:3790: finishing copy_from racing read to smithi11541303-5 2022-04-22T17:38:27.175 INFO:tasks.rados.rados.0.smithi115.stdout:3789: expect (ObjNum 1076 snap 339 seq_num 1076) 2022-04-22T17:38:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:27 smithi121 bash[36902]: cluster 2022-04-22T17:38:25.991529+0000 mgr.y (mgr.34107) 477 : cluster [DBG] pgmap v895: 121 pgs: 121 active+clean; 257 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2022-04-22T17:38:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:27 smithi121 bash[36902]: cluster 2022-04-22T17:38:26.102376+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2022-04-22T17:38:27.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:27 smithi115 bash[46934]: cluster 2022-04-22T17:38:25.991529+0000 mgr.y (mgr.34107) 477 : cluster [DBG] pgmap v895: 121 pgs: 121 active+clean; 257 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2022-04-22T17:38:27.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:27 smithi115 bash[46934]: cluster 2022-04-22T17:38:26.102376+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2022-04-22T17:38:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:27 smithi115 bash[45495]: cluster 2022-04-22T17:38:25.991529+0000 mgr.y (mgr.34107) 477 : cluster [DBG] pgmap v895: 121 pgs: 121 active+clean; 257 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2022-04-22T17:38:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:27 smithi115 bash[45495]: cluster 2022-04-22T17:38:26.102376+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2022-04-22T17:38:28.137 INFO:tasks.rados.rados.0.smithi115.stdout:3789: done (2 left) 2022-04-22T17:38:28.137 INFO:tasks.rados.rados.0.smithi115.stdout:3790: done (1 left) 2022-04-22T17:38:28.137 INFO:tasks.rados.rados.0.smithi115.stdout:3791: done (0 left) 2022-04-22T17:38:28.137 INFO:tasks.rados.rados.0.smithi115.stdout:3792: copy_from oid 10 from oid 6 current snap is 367 2022-04-22T17:38:28.137 INFO:tasks.rados.rados.0.smithi115.stdout:3793: snap_create 2022-04-22T17:38:28.214 INFO:tasks.rados.rados.0.smithi115.stdout:3792: finishing copy_from to smithi11541303-10 2022-04-22T17:38:28.214 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 1945 (ObjNum 1125 snap 359 seq_num 1125) dirty exists 2022-04-22T17:38:28.216 INFO:tasks.rados.rados.0.smithi115.stdout:3792: finishing copy_from racing read to smithi11541303-10 2022-04-22T17:38:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:28 smithi121 bash[36902]: cluster 2022-04-22T17:38:27.113418+0000 mon.a (mon.0) 1508 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2022-04-22T17:38:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:28 smithi115 bash[45495]: cluster 2022-04-22T17:38:27.113418+0000 mon.a (mon.0) 1508 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2022-04-22T17:38:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:28 smithi115 bash[46934]: cluster 2022-04-22T17:38:27.113418+0000 mon.a (mon.0) 1508 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2022-04-22T17:38:29.142 INFO:tasks.rados.rados.0.smithi115.stdout:3792: done (1 left) 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3793: done (0 left) 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3794: read oid 18 snap -1 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3794: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3795: read oid 24 snap 363 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3795: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 368 2022-04-22T17:38:29.143 INFO:tasks.rados.rados.0.smithi115.stdout:3796: seq_num 1142 ranges {4136960=729088,4866048=565248,5431296=696320,6127616=344064} 2022-04-22T17:38:29.169 INFO:tasks.rados.rados.0.smithi115.stdout:3796: writing smithi11541303-50 from 4136960 to 4866048 tid 1 2022-04-22T17:38:29.173 INFO:tasks.rados.rados.0.smithi115.stdout:3796: writing smithi11541303-50 from 4866048 to 5431296 tid 2 2022-04-22T17:38:29.177 INFO:tasks.rados.rados.0.smithi115.stdout:3796: writing smithi11541303-50 from 5431296 to 6127616 tid 3 2022-04-22T17:38:29.180 INFO:tasks.rados.rados.0.smithi115.stdout:3796: writing smithi11541303-50 from 6127616 to 6471680 tid 4 2022-04-22T17:38:29.180 INFO:tasks.rados.rados.0.smithi115.stdout:3797: snap_remove snap 366 2022-04-22T17:38:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:29 smithi121 bash[36902]: cluster 2022-04-22T17:38:27.992286+0000 mgr.y (mgr.34107) 478 : cluster [DBG] pgmap v898: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.1 MiB/s wr, 9 op/s 2022-04-22T17:38:29.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:29 smithi121 bash[36902]: cluster 2022-04-22T17:38:28.124520+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2022-04-22T17:38:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:29 smithi115 bash[45495]: cluster 2022-04-22T17:38:27.992286+0000 mgr.y (mgr.34107) 478 : cluster [DBG] pgmap v898: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.1 MiB/s wr, 9 op/s 2022-04-22T17:38:29.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:29 smithi115 bash[45495]: cluster 2022-04-22T17:38:28.124520+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2022-04-22T17:38:29.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:29 smithi115 bash[46934]: cluster 2022-04-22T17:38:27.992286+0000 mgr.y (mgr.34107) 478 : cluster [DBG] pgmap v898: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.1 MiB/s wr, 9 op/s 2022-04-22T17:38:29.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:29 smithi115 bash[46934]: cluster 2022-04-22T17:38:28.124520+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2022-04-22T17:38:30.024 INFO:tasks.rados.rados.0.smithi115.stdout:3797: done (3 left) 2022-04-22T17:38:30.025 INFO:tasks.rados.rados.0.smithi115.stdout:3798: snap_create 2022-04-22T17:38:30.025 INFO:tasks.rados.rados.0.smithi115.stdout:3794: expect (ObjNum 905 snap 297 seq_num 905) 2022-04-22T17:38:30.046 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 1 to smithi11541303-50 2022-04-22T17:38:30.046 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 2 to smithi11541303-50 2022-04-22T17:38:30.047 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 3 to smithi11541303-50 2022-04-22T17:38:30.047 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 4 to smithi11541303-50 2022-04-22T17:38:30.047 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 5 to smithi11541303-50 2022-04-22T17:38:30.048 INFO:tasks.rados.rados.0.smithi115.stdout:3795: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:38:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:30 smithi121 bash[36902]: cluster 2022-04-22T17:38:29.128016+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2022-04-22T17:38:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:30 smithi121 bash[36902]: audit 2022-04-22T17:38:29.463740+0000 mon.a (mon.0) 1511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:30 smithi121 bash[36902]: cluster 2022-04-22T17:38:30.012033+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2022-04-22T17:38:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[46934]: cluster 2022-04-22T17:38:29.128016+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2022-04-22T17:38:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[46934]: audit 2022-04-22T17:38:29.463740+0000 mon.a (mon.0) 1511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[46934]: cluster 2022-04-22T17:38:30.012033+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2022-04-22T17:38:30.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[45495]: cluster 2022-04-22T17:38:29.128016+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2022-04-22T17:38:30.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[45495]: audit 2022-04-22T17:38:29.463740+0000 mon.a (mon.0) 1511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:30.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:30 smithi115 bash[45495]: cluster 2022-04-22T17:38:30.012033+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2022-04-22T17:38:30.741 INFO:tasks.rados.rados.0.smithi115.stdout:3796: finishing write tid 6 to smithi11541303-50 2022-04-22T17:38:30.741 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2093 (ObjNum 1142 snap 368 seq_num 1142) dirty exists 2022-04-22T17:38:30.741 INFO:tasks.rados.rados.0.smithi115.stdout:3796: left oid 50 (ObjNum 1142 snap 368 seq_num 1142) 2022-04-22T17:38:31.023 INFO:tasks.rados.rados.0.smithi115.stdout:3794: done (3 left) 2022-04-22T17:38:31.024 INFO:tasks.rados.rados.0.smithi115.stdout:3795: done (2 left) 2022-04-22T17:38:31.024 INFO:tasks.rados.rados.0.smithi115.stdout:3796: done (1 left) 2022-04-22T17:38:31.024 INFO:tasks.rados.rados.0.smithi115.stdout:3798: done (0 left) 2022-04-22T17:38:31.024 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 26 current snap is 369 2022-04-22T17:38:31.024 INFO:tasks.rados.rados.0.smithi115.stdout:3799: seq_num 1143 ranges {12795904=753664,13549568=450560,14000128=507904,14508032=589824,15097856=57344} 2022-04-22T17:38:31.103 INFO:tasks.rados.rados.0.smithi115.stdout:3799: writing smithi11541303-26 from 12795904 to 13549568 tid 1 2022-04-22T17:38:31.105 INFO:tasks.rados.rados.0.smithi115.stdout:3799: writing smithi11541303-26 from 13549568 to 14000128 tid 2 2022-04-22T17:38:31.109 INFO:tasks.rados.rados.0.smithi115.stdout:3799: writing smithi11541303-26 from 14000128 to 14508032 tid 3 2022-04-22T17:38:31.112 INFO:tasks.rados.rados.0.smithi115.stdout:3799: writing smithi11541303-26 from 14508032 to 15097856 tid 4 2022-04-22T17:38:31.113 INFO:tasks.rados.rados.0.smithi115.stdout:3799: writing smithi11541303-26 from 15097856 to 15155200 tid 5 2022-04-22T17:38:31.113 INFO:tasks.rados.rados.0.smithi115.stdout:3800: rollback oid 11 current snap is 369 2022-04-22T17:38:31.114 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 365 2022-04-22T17:38:31.114 INFO:tasks.rados.rados.0.smithi115.stdout:3801: read oid 37 snap -1 2022-04-22T17:38:31.114 INFO:tasks.rados.rados.0.smithi115.stdout:3801: expect deleted 2022-04-22T17:38:31.114 INFO:tasks.rados.rados.0.smithi115.stdout:3802: snap_create 2022-04-22T17:38:31.117 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 1 to smithi11541303-26 2022-04-22T17:38:31.120 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 2 to smithi11541303-26 2022-04-22T17:38:31.120 INFO:tasks.rados.rados.0.smithi115.stdout:3800: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:38:31.121 INFO:tasks.rados.rados.0.smithi115.stdout:3800: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:38:31.123 INFO:tasks.rados.rados.0.smithi115.stdout:3800: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:38:31.123 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2102 (ObjNum 1139 snap 364 seq_num 1139) dirty exists 2022-04-22T17:38:31.123 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 3 to smithi11541303-26 2022-04-22T17:38:31.123 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 4 to smithi11541303-26 2022-04-22T17:38:31.123 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 5 to smithi11541303-26 2022-04-22T17:38:31.124 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 6 to smithi11541303-26 2022-04-22T17:38:31.126 INFO:tasks.rados.rados.0.smithi115.stdout:3799: finishing write tid 7 to smithi11541303-26 2022-04-22T17:38:31.129 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2070 (ObjNum 1143 snap 369 seq_num 1143) dirty exists 2022-04-22T17:38:31.130 INFO:tasks.rados.rados.0.smithi115.stdout:3799: left oid 26 (ObjNum 1143 snap 369 seq_num 1143) 2022-04-22T17:38:31.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:31 smithi121 bash[36902]: cluster 2022-04-22T17:38:29.993183+0000 mgr.y (mgr.34107) 479 : cluster [DBG] pgmap v901: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:38:31.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:31 smithi121 bash[36902]: cluster 2022-04-22T17:38:31.011111+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2022-04-22T17:38:31.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:31 smithi115 bash[45495]: cluster 2022-04-22T17:38:29.993183+0000 mgr.y (mgr.34107) 479 : cluster [DBG] pgmap v901: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:38:31.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:31 smithi115 bash[45495]: cluster 2022-04-22T17:38:31.011111+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2022-04-22T17:38:31.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:31 smithi115 bash[46934]: cluster 2022-04-22T17:38:29.993183+0000 mgr.y (mgr.34107) 479 : cluster [DBG] pgmap v901: 121 pgs: 121 active+clean; 268 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:38:31.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:31 smithi115 bash[46934]: cluster 2022-04-22T17:38:31.011111+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2022-04-22T17:38:32.027 INFO:tasks.rados.rados.0.smithi115.stdout:3799: done (3 left) 2022-04-22T17:38:32.027 INFO:tasks.rados.rados.0.smithi115.stdout:3800: done (2 left) 2022-04-22T17:38:32.028 INFO:tasks.rados.rados.0.smithi115.stdout:3801: done (1 left) 2022-04-22T17:38:32.028 INFO:tasks.rados.rados.0.smithi115.stdout:3802: done (0 left) 2022-04-22T17:38:32.028 INFO:tasks.rados.rados.0.smithi115.stdout:3803: read oid 44 snap -1 2022-04-22T17:38:32.028 INFO:tasks.rados.rados.0.smithi115.stdout:3803: expect (ObjNum 1096 snap 347 seq_num 1096) 2022-04-22T17:38:32.028 INFO:tasks.rados.rados.0.smithi115.stdout:3804: snap_create 2022-04-22T17:38:32.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:31.631Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:32.079 INFO:tasks.rados.rados.0.smithi115.stdout:3803: expect (ObjNum 1096 snap 347 seq_num 1096) 2022-04-22T17:38:32.870 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:38:33.185 INFO:tasks.rados.rados.0.smithi115.stdout:3803: done (1 left) 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:3804: done (0 left) 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:3805: rollback oid 12 current snap is 371 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 12 to 367 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:3806: read oid 10 snap -1 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:3806: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:33.186 INFO:tasks.rados.rados.0.smithi115.stdout:3807: read oid 26 snap -1 2022-04-22T17:38:33.190 INFO:tasks.rados.rados.0.smithi115.stdout:3807: expect (ObjNum 1143 snap 369 seq_num 1143) 2022-04-22T17:38:33.190 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 35 current snap is 371 2022-04-22T17:38:33.191 INFO:tasks.rados.rados.0.smithi115.stdout:3808: seq_num 1144 ranges {0=770048,770048=737280,1507328=712704,2220032=155648} 2022-04-22T17:38:33.197 INFO:tasks.rados.rados.0.smithi115.stdout:3808: writing smithi11541303-35 from 0 to 770048 tid 1 2022-04-22T17:38:33.201 INFO:tasks.rados.rados.0.smithi115.stdout:3808: writing smithi11541303-35 from 770048 to 1507328 tid 2 2022-04-22T17:38:33.207 INFO:tasks.rados.rados.0.smithi115.stdout:3808: writing smithi11541303-35 from 1507328 to 2220032 tid 3 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3808: writing smithi11541303-35 from 2220032 to 2375680 tid 4 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3809: read oid 29 snap 368 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3809: expect (ObjNum 1136 snap 363 seq_num 1136) 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3810: read oid 40 snap -1 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3810: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:33.208 INFO:tasks.rados.rados.0.smithi115.stdout:3811: snap_remove snap 369 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (97s) 92s ago 15m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (15m) 2m ago 15m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3m) 92s ago 16m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3m) 2m ago 16m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (10m) 2m ago 20m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:38:33.213 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (11m) 92s ago 22m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (10m) 92s ago 23m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (9m) 2m ago 21m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 92s ago 21m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 92s ago 15m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2m) 2m ago 15m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (8m) 92s ago 20m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 92s ago 19m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 92s ago 19m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 92s ago 18m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:38:33.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6m) 2m ago 18m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:38:33.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5m) 2m ago 17m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:38:33.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4m) 2m ago 17m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:38:33.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3m) 2m ago 16m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:38:33.217 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 2m ago 15m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:38:33.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:33 smithi115 bash[45495]: cluster 2022-04-22T17:38:31.994049+0000 mgr.y (mgr.34107) 480 : cluster [DBG] pgmap v904: 121 pgs: 121 active+clean; 280 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:33.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:33 smithi115 bash[45495]: cluster 2022-04-22T17:38:32.015158+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2022-04-22T17:38:33.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:33 smithi115 bash[46934]: cluster 2022-04-22T17:38:31.994049+0000 mgr.y (mgr.34107) 480 : cluster [DBG] pgmap v904: 121 pgs: 121 active+clean; 280 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:33.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:33 smithi115 bash[46934]: cluster 2022-04-22T17:38:32.015158+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2022-04-22T17:38:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:33 smithi121 bash[36902]: cluster 2022-04-22T17:38:31.994049+0000 mgr.y (mgr.34107) 480 : cluster [DBG] pgmap v904: 121 pgs: 121 active+clean; 280 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:33.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:33 smithi121 bash[36902]: cluster 2022-04-22T17:38:32.015158+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2022-04-22T17:38:33.642 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:38:33.642 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:33.643 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:38:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:38:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:33.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3811: done (6 left) 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3812: copy_from oid 18 from oid 50 current snap is 371 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3813: delete oid 7 current snap is 371 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3805: finishing rollback tid 0 to smithi11541303-12 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3805: finishing rollback tid 1 to smithi11541303-12 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3805: finishing rollback tid 2 to smithi11541303-12 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2247 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:38:34.054 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 1 to smithi11541303-35 2022-04-22T17:38:34.055 INFO:tasks.rados.rados.0.smithi115.stdout:3809: expect (ObjNum 1136 snap 363 seq_num 1136) 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 2 to smithi11541303-35 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 3 to smithi11541303-35 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 4 to smithi11541303-35 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 5 to smithi11541303-35 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:3808: finishing write tid 6 to smithi11541303-35 2022-04-22T17:38:34.120 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 2046 (ObjNum 1144 snap 371 seq_num 1144) dirty exists 2022-04-22T17:38:34.121 INFO:tasks.rados.rados.0.smithi115.stdout:3808: left oid 35 (ObjNum 1144 snap 371 seq_num 1144) 2022-04-22T17:38:34.121 INFO:tasks.rados.rados.0.smithi115.stdout:3806: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:34.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[45495]: audit 2022-04-22T17:38:32.845109+0000 mgr.y (mgr.34107) 481 : audit [DBG] from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:34.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[45495]: cluster 2022-04-22T17:38:33.015715+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2022-04-22T17:38:34.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[45495]: audit 2022-04-22T17:38:33.630016+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3306173414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[46934]: audit 2022-04-22T17:38:32.845109+0000 mgr.y (mgr.34107) 481 : audit [DBG] from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[46934]: cluster 2022-04-22T17:38:33.015715+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2022-04-22T17:38:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:34 smithi115 bash[46934]: audit 2022-04-22T17:38:33.630016+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3306173414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:34 smithi121 bash[36902]: audit 2022-04-22T17:38:32.845109+0000 mgr.y (mgr.34107) 481 : audit [DBG] from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:34 smithi121 bash[36902]: cluster 2022-04-22T17:38:33.015715+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2022-04-22T17:38:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:34 smithi121 bash[36902]: audit 2022-04-22T17:38:33.630016+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.115:0/3306173414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:38:35.201 INFO:tasks.rados.rados.0.smithi115.stdout:3810: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:38:35.227 INFO:tasks.rados.rados.0.smithi115.stdout:3807: expect (ObjNum 1143 snap 369 seq_num 1143) 2022-04-22T17:38:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[45495]: audit 2022-04-22T17:38:33.183912+0000 mgr.y (mgr.34107) 482 : audit [DBG] from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[45495]: cluster 2022-04-22T17:38:33.995120+0000 mgr.y (mgr.34107) 483 : cluster [DBG] pgmap v907: 121 pgs: 121 active+clean; 282 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 4 op/s 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[45495]: cluster 2022-04-22T17:38:34.038462+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[45495]: audit 2022-04-22T17:38:34.466617+0000 mon.a (mon.0) 1517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[45495]: cluster 2022-04-22T17:38:35.013463+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[46934]: audit 2022-04-22T17:38:33.183912+0000 mgr.y (mgr.34107) 482 : audit [DBG] from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[46934]: cluster 2022-04-22T17:38:33.995120+0000 mgr.y (mgr.34107) 483 : cluster [DBG] pgmap v907: 121 pgs: 121 active+clean; 282 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 4 op/s 2022-04-22T17:38:35.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[46934]: cluster 2022-04-22T17:38:34.038462+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2022-04-22T17:38:35.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[46934]: audit 2022-04-22T17:38:34.466617+0000 mon.a (mon.0) 1517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:35.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:35 smithi115 bash[46934]: cluster 2022-04-22T17:38:35.013463+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2022-04-22T17:38:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:35 smithi121 bash[36902]: audit 2022-04-22T17:38:33.183912+0000 mgr.y (mgr.34107) 482 : audit [DBG] from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:38:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:35 smithi121 bash[36902]: cluster 2022-04-22T17:38:33.995120+0000 mgr.y (mgr.34107) 483 : cluster [DBG] pgmap v907: 121 pgs: 121 active+clean; 282 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 4 op/s 2022-04-22T17:38:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:35 smithi121 bash[36902]: cluster 2022-04-22T17:38:34.038462+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2022-04-22T17:38:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:35 smithi121 bash[36902]: audit 2022-04-22T17:38:34.466617+0000 mon.a (mon.0) 1517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:35 smithi121 bash[36902]: cluster 2022-04-22T17:38:35.013463+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2022-04-22T17:38:36.561 INFO:tasks.rados.rados.0.smithi115.stdout:3812: finishing copy_from to smithi11541303-18 2022-04-22T17:38:36.561 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2093 (ObjNum 1142 snap 368 seq_num 1142) dirty exists 2022-04-22T17:38:36.561 INFO:tasks.rados.rados.0.smithi115.stdout:3812: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:38:36.561 INFO:tasks.rados.rados.0.smithi115.stdout:3805: done (7 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3806: done (6 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3807: done (5 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3808: done (4 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3809: done (3 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3810: done (2 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3812: done (1 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3813: done (0 left) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3814: read oid 21 snap -1 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3814: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:38:36.562 INFO:tasks.rados.rados.0.smithi115.stdout:3815: read oid 25 snap -1 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3815: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3816: copy_from oid 39 from oid 5 current snap is 371 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3817: read oid 34 snap 370 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3817: expect deleted 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3818: read oid 30 snap 363 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3818: expect (ObjNum 1076 snap 339 seq_num 1076) 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3819: rollback oid 2 current snap is 371 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 2 to 365 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3820: read oid 33 snap -1 2022-04-22T17:38:36.563 INFO:tasks.rados.rados.0.smithi115.stdout:3820: expect (ObjNum 1049 snap 329 seq_num 1049) 2022-04-22T17:38:36.564 INFO:tasks.rados.rados.0.smithi115.stdout:3821: read oid 7 snap -1 2022-04-22T17:38:36.564 INFO:tasks.rados.rados.0.smithi115.stdout:3821: expect deleted 2022-04-22T17:38:36.564 INFO:tasks.rados.rados.0.smithi115.stdout:3822: read oid 28 snap -1 2022-04-22T17:38:36.564 INFO:tasks.rados.rados.0.smithi115.stdout:3822: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:38:36.584 INFO:tasks.rados.rados.0.smithi115.stdout:3823: snap_remove snap 367 2022-04-22T17:38:37.058 INFO:tasks.rados.rados.0.smithi115.stdout:3823: done (9 left) 2022-04-22T17:38:37.058 INFO:tasks.rados.rados.0.smithi115.stdout:3824: snap_remove snap 368 2022-04-22T17:38:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:37 smithi115 bash[45495]: cluster 2022-04-22T17:38:35.995855+0000 mgr.y (mgr.34107) 484 : cluster [DBG] pgmap v910: 121 pgs: 121 active+clean; 285 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:38:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:37 smithi115 bash[46934]: cluster 2022-04-22T17:38:35.995855+0000 mgr.y (mgr.34107) 484 : cluster [DBG] pgmap v910: 121 pgs: 121 active+clean; 285 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:38:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:37 smithi121 bash[36902]: cluster 2022-04-22T17:38:35.995855+0000 mgr.y (mgr.34107) 484 : cluster [DBG] pgmap v910: 121 pgs: 121 active+clean; 285 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2022-04-22T17:38:38.068 INFO:tasks.rados.rados.0.smithi115.stdout:3824: done (9 left) 2022-04-22T17:38:38.135 INFO:tasks.rados.rados.0.smithi115.stdout:3825: delete oid 47 current snap is 371 2022-04-22T17:38:38.136 INFO:tasks.rados.rados.0.smithi115.stdout:3819: finishing rollback tid 0 to smithi11541303-2 2022-04-22T17:38:38.136 INFO:tasks.rados.rados.0.smithi115.stdout:3819: finishing rollback tid 1 to smithi11541303-2 2022-04-22T17:38:38.136 INFO:tasks.rados.rados.0.smithi115.stdout:3819: finishing rollback tid 2 to smithi11541303-2 2022-04-22T17:38:38.136 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 2 v 2049 (ObjNum 1107 snap 352 seq_num 1107) dirty exists 2022-04-22T17:38:38.136 INFO:tasks.rados.rados.0.smithi115.stdout:3814: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:38:38.251 INFO:tasks.rados.rados.0.smithi115.stdout:3820: expect (ObjNum 1049 snap 329 seq_num 1049) 2022-04-22T17:38:38.372 INFO:tasks.rados.rados.0.smithi115.stdout:3816: finishing copy_from to smithi11541303-39 2022-04-22T17:38:38.373 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 39 v 2071 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:38:38.373 INFO:tasks.rados.rados.0.smithi115.stdout:3816: finishing copy_from racing read to smithi11541303-39 2022-04-22T17:38:38.373 INFO:tasks.rados.rados.0.smithi115.stdout:3822: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:38:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:38 smithi121 bash[36902]: cluster 2022-04-22T17:38:37.044644+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2022-04-22T17:38:38.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:38 smithi115 bash[46934]: cluster 2022-04-22T17:38:37.044644+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2022-04-22T17:38:38.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:38 smithi115 bash[45495]: cluster 2022-04-22T17:38:37.044644+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2022-04-22T17:38:39.430 INFO:tasks.rados.rados.0.smithi115.stdout:3818: expect (ObjNum 1076 snap 339 seq_num 1076) 2022-04-22T17:38:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:39 smithi121 bash[36902]: cluster 2022-04-22T17:38:37.996640+0000 mgr.y (mgr.34107) 485 : cluster [DBG] pgmap v912: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2022-04-22T17:38:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:39 smithi121 bash[36902]: cluster 2022-04-22T17:38:38.054717+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2022-04-22T17:38:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:39 smithi115 bash[45495]: cluster 2022-04-22T17:38:37.996640+0000 mgr.y (mgr.34107) 485 : cluster [DBG] pgmap v912: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2022-04-22T17:38:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:39 smithi115 bash[45495]: cluster 2022-04-22T17:38:38.054717+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2022-04-22T17:38:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:39 smithi115 bash[46934]: cluster 2022-04-22T17:38:37.996640+0000 mgr.y (mgr.34107) 485 : cluster [DBG] pgmap v912: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2022-04-22T17:38:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:39 smithi115 bash[46934]: cluster 2022-04-22T17:38:38.054717+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2022-04-22T17:38:40.292 INFO:tasks.rados.rados.0.smithi115.stdout:3815: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:38:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:40 smithi121 bash[36902]: audit 2022-04-22T17:38:39.469508+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:40 smithi121 bash[36902]: cluster 2022-04-22T17:38:40.014289+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2022-04-22T17:38:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:40 smithi115 bash[45495]: audit 2022-04-22T17:38:39.469508+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:40 smithi115 bash[45495]: cluster 2022-04-22T17:38:40.014289+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2022-04-22T17:38:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:40 smithi115 bash[46934]: audit 2022-04-22T17:38:39.469508+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:40 smithi115 bash[46934]: cluster 2022-04-22T17:38:40.014289+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2022-04-22T17:38:40.900 INFO:tasks.rados.rados.0.smithi115.stdout:3814: done (9 left) 2022-04-22T17:38:40.900 INFO:tasks.rados.rados.0.smithi115.stdout:3815: done (8 left) 2022-04-22T17:38:40.900 INFO:tasks.rados.rados.0.smithi115.stdout:3816: done (7 left) 2022-04-22T17:38:40.901 INFO:tasks.rados.rados.0.smithi115.stdout:3817: done (6 left) 2022-04-22T17:38:40.901 INFO:tasks.rados.rados.0.smithi115.stdout:3818: done (5 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3819: done (4 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3820: done (3 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3821: done (2 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3822: done (1 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3825: done (0 left) 2022-04-22T17:38:40.902 INFO:tasks.rados.rados.0.smithi115.stdout:3826: rollback oid 26 current snap is 371 2022-04-22T17:38:40.903 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 26 to 363 2022-04-22T17:38:40.903 INFO:tasks.rados.rados.0.smithi115.stdout:3827: rollback oid 50 current snap is 371 2022-04-22T17:38:40.903 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 365 2022-04-22T17:38:40.903 INFO:tasks.rados.rados.0.smithi115.stdout:3828: setattr oid 25 current snap is 371 2022-04-22T17:38:40.907 INFO:tasks.rados.rados.0.smithi115.stdout:3829: rollback oid 11 current snap is 371 2022-04-22T17:38:40.907 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 365 2022-04-22T17:38:40.907 INFO:tasks.rados.rados.0.smithi115.stdout:3830: snap_remove snap 370 2022-04-22T17:38:41.097 INFO:tasks.rados.rados.0.smithi115.stdout:3830: done (4 left) 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3831: copy_from oid 23 from oid 38 current snap is 371 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3827: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3827: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3827: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2096 (ObjNum 1137 snap 363 seq_num 1137) dirty exists 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3826: finishing rollback tid 0 to smithi11541303-26 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3826: finishing rollback tid 1 to smithi11541303-26 2022-04-22T17:38:41.098 INFO:tasks.rados.rados.0.smithi115.stdout:3826: finishing rollback tid 2 to smithi11541303-26 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2073 (ObjNum 1130 snap 361 seq_num 1130) dirty exists 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3829: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 2201 (ObjNum 1081 snap 342 seq_num 1081) dirty exists 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3829: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3829: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2105 (ObjNum 1139 snap 364 seq_num 1139) dirty exists 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3826: done (4 left) 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3827: done (3 left) 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3828: done (2 left) 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3829: done (1 left) 2022-04-22T17:38:41.099 INFO:tasks.rados.rados.0.smithi115.stdout:3832: delete oid 3 current snap is 371 2022-04-22T17:38:41.108 INFO:tasks.rados.rados.0.smithi115.stdout:3832: done (1 left) 2022-04-22T17:38:41.109 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 41 current snap is 371 2022-04-22T17:38:41.109 INFO:tasks.rados.rados.0.smithi115.stdout:3833: seq_num 1148 ranges {9306112=32768} 2022-04-22T17:38:41.163 INFO:tasks.rados.rados.0.smithi115.stdout:3833: writing smithi11541303-41 from 9306112 to 9338880 tid 1 2022-04-22T17:38:41.163 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 371 2022-04-22T17:38:41.163 INFO:tasks.rados.rados.0.smithi115.stdout:3834: seq_num 1149 ranges {11247616=49152} 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3834: writing smithi11541303-30 from 11247616 to 11296768 tid 1 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3835: read oid 3 snap -1 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3835: expect deleted 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3836: read oid 48 snap -1 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3836: expect (ObjNum 1092 snap 347 seq_num 1092) 2022-04-22T17:38:41.230 INFO:tasks.rados.rados.0.smithi115.stdout:3837: rmattr oid 12 current snap is 371 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3831: finishing copy_from to smithi11541303-23 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 23 v 2239 (ObjNum 1093 snap 347 seq_num 1093) dirty exists 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3838: snap_create 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3831: finishing copy_from racing read to smithi11541303-23 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3833: finishing write tid 1 to smithi11541303-41 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3833: finishing write tid 2 to smithi11541303-41 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3833: finishing write tid 3 to smithi11541303-41 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 41 v 2074 (ObjNum 1148 snap 371 seq_num 1148) dirty exists 2022-04-22T17:38:41.231 INFO:tasks.rados.rados.0.smithi115.stdout:3833: left oid 41 (ObjNum 1148 snap 371 seq_num 1148) 2022-04-22T17:38:41.235 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2248 (ObjNum 984 snap 313 seq_num 984) dirty exists 2022-04-22T17:38:41.253 INFO:tasks.rados.rados.0.smithi115.stdout:3834: finishing write tid 1 to smithi11541303-30 2022-04-22T17:38:41.253 INFO:tasks.rados.rados.0.smithi115.stdout:3834: finishing write tid 2 to smithi11541303-30 2022-04-22T17:38:41.261 INFO:tasks.rados.rados.0.smithi115.stdout:3836: expect (ObjNum 1092 snap 347 seq_num 1092) 2022-04-22T17:38:41.473 INFO:tasks.rados.rados.0.smithi115.stdout:3834: finishing write tid 3 to smithi11541303-30 2022-04-22T17:38:41.474 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2086 (ObjNum 1149 snap 371 seq_num 1149) dirty exists 2022-04-22T17:38:41.474 INFO:tasks.rados.rados.0.smithi115.stdout:3834: left oid 30 (ObjNum 1149 snap 371 seq_num 1149) 2022-04-22T17:38:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:41 smithi121 bash[36902]: cluster 2022-04-22T17:38:39.997539+0000 mgr.y (mgr.34107) 486 : cluster [DBG] pgmap v914: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:38:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:41 smithi115 bash[45495]: cluster 2022-04-22T17:38:39.997539+0000 mgr.y (mgr.34107) 486 : cluster [DBG] pgmap v914: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:38:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:41 smithi115 bash[46934]: cluster 2022-04-22T17:38:39.997539+0000 mgr.y (mgr.34107) 486 : cluster [DBG] pgmap v914: 121 pgs: 121 active+clean; 294 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2022-04-22T17:38:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:41.631Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3831: done (6 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3833: done (5 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3834: done (4 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3835: done (3 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3836: done (2 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3837: done (1 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3838: done (0 left) 2022-04-22T17:38:42.101 INFO:tasks.rados.rados.0.smithi115.stdout:3839: snap_create 2022-04-22T17:38:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:42 smithi121 bash[36902]: cluster 2022-04-22T17:38:41.083132+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2022-04-22T17:38:42.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:42 smithi115 bash[45495]: cluster 2022-04-22T17:38:41.083132+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2022-04-22T17:38:42.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:42 smithi115 bash[46934]: cluster 2022-04-22T17:38:41.083132+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2022-04-22T17:38:43.106 INFO:tasks.rados.rados.0.smithi115.stdout:3839: done (0 left) 2022-04-22T17:38:43.135 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 373 2022-04-22T17:38:43.135 INFO:tasks.rados.rados.0.smithi115.stdout:3840: seq_num 1150 ranges {4620288=65536} 2022-04-22T17:38:43.136 INFO:tasks.rados.rados.0.smithi115.stdout:3840: writing smithi11541303-48 from 4620288 to 4685824 tid 1 2022-04-22T17:38:43.136 INFO:tasks.rados.rados.0.smithi115.stdout:3841: snap_create 2022-04-22T17:38:43.143 INFO:tasks.rados.rados.0.smithi115.stdout:3840: finishing write tid 1 to smithi11541303-48 2022-04-22T17:38:43.144 INFO:tasks.rados.rados.0.smithi115.stdout:3840: finishing write tid 2 to smithi11541303-48 2022-04-22T17:38:43.146 INFO:tasks.rados.rados.0.smithi115.stdout:3840: finishing write tid 3 to smithi11541303-48 2022-04-22T17:38:43.146 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2088 (ObjNum 1150 snap 373 seq_num 1150) dirty exists 2022-04-22T17:38:43.147 INFO:tasks.rados.rados.0.smithi115.stdout:3840: left oid 48 (ObjNum 1150 snap 373 seq_num 1150) 2022-04-22T17:38:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:43 smithi121 bash[36902]: cluster 2022-04-22T17:38:41.998593+0000 mgr.y (mgr.34107) 487 : cluster [DBG] pgmap v917: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T17:38:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:43 smithi121 bash[36902]: cluster 2022-04-22T17:38:42.088426+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2022-04-22T17:38:43.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:43 smithi115 bash[45495]: cluster 2022-04-22T17:38:41.998593+0000 mgr.y (mgr.34107) 487 : cluster [DBG] pgmap v917: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T17:38:43.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:43 smithi115 bash[45495]: cluster 2022-04-22T17:38:42.088426+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2022-04-22T17:38:43.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:43 smithi115 bash[46934]: cluster 2022-04-22T17:38:41.998593+0000 mgr.y (mgr.34107) 487 : cluster [DBG] pgmap v917: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T17:38:43.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:43 smithi115 bash[46934]: cluster 2022-04-22T17:38:42.088426+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2022-04-22T17:38:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:43.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:44.117 INFO:tasks.rados.rados.0.smithi115.stdout:3840: done (1 left) 2022-04-22T17:38:44.118 INFO:tasks.rados.rados.0.smithi115.stdout:3841: done (0 left) 2022-04-22T17:38:44.118 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 31 current snap is 374 2022-04-22T17:38:44.118 INFO:tasks.rados.rados.0.smithi115.stdout:3842: seq_num 1151 ranges {1294336=81920} 2022-04-22T17:38:44.128 INFO:tasks.rados.rados.0.smithi115.stdout:3842: writing smithi11541303-31 from 1294336 to 1376256 tid 1 2022-04-22T17:38:44.128 INFO:tasks.rados.rados.0.smithi115.stdout:3843: copy_from oid 50 from oid 15 current snap is 374 2022-04-22T17:38:44.128 INFO:tasks.rados.rados.0.smithi115.stdout:3844: snap_create 2022-04-22T17:38:44.142 INFO:tasks.rados.rados.0.smithi115.stdout:3842: finishing write tid 1 to smithi11541303-31 2022-04-22T17:38:44.143 INFO:tasks.rados.rados.0.smithi115.stdout:3842: finishing write tid 2 to smithi11541303-31 2022-04-22T17:38:44.143 INFO:tasks.rados.rados.0.smithi115.stdout:3842: finishing write tid 3 to smithi11541303-31 2022-04-22T17:38:44.143 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2372 (ObjNum 1151 snap 374 seq_num 1151) dirty exists 2022-04-22T17:38:44.143 INFO:tasks.rados.rados.0.smithi115.stdout:3842: left oid 31 (ObjNum 1151 snap 374 seq_num 1151) 2022-04-22T17:38:44.181 INFO:tasks.rados.rados.0.smithi115.stdout:3843: finishing copy_from to smithi11541303-50 2022-04-22T17:38:44.181 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2061 (ObjNum 1138 snap 363 seq_num 1138) dirty exists 2022-04-22T17:38:44.182 INFO:tasks.rados.rados.0.smithi115.stdout:3843: finishing copy_from racing read to smithi11541303-50 2022-04-22T17:38:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:44 smithi121 bash[36902]: cluster 2022-04-22T17:38:43.090473+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2022-04-22T17:38:44.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:44 smithi115 bash[45495]: cluster 2022-04-22T17:38:43.090473+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2022-04-22T17:38:44.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:44 smithi115 bash[46934]: cluster 2022-04-22T17:38:43.090473+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2022-04-22T17:38:45.035 INFO:tasks.rados.rados.0.smithi115.stdout:3842: done (2 left) 2022-04-22T17:38:45.036 INFO:tasks.rados.rados.0.smithi115.stdout:3843: done (1 left) 2022-04-22T17:38:45.036 INFO:tasks.rados.rados.0.smithi115.stdout:3844: done (0 left) 2022-04-22T17:38:45.037 INFO:tasks.rados.rados.0.smithi115.stdout:3845: rollback oid 31 current snap is 375 2022-04-22T17:38:45.038 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 371 2022-04-22T17:38:45.038 INFO:tasks.rados.rados.0.smithi115.stdout:3846: read oid 29 snap -1 2022-04-22T17:38:45.038 INFO:tasks.rados.rados.0.smithi115.stdout:3846: expect (ObjNum 1136 snap 363 seq_num 1136) 2022-04-22T17:38:45.039 INFO:tasks.rados.rados.0.smithi115.stdout:3847: snap_create 2022-04-22T17:38:45.049 INFO:tasks.rados.rados.0.smithi115.stdout:3846: expect (ObjNum 1136 snap 363 seq_num 1136) 2022-04-22T17:38:45.107 INFO:tasks.rados.rados.0.smithi115.stdout:3845: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:38:45.125 INFO:tasks.rados.rados.0.smithi115.stdout:3845: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:38:45.125 INFO:tasks.rados.rados.0.smithi115.stdout:3845: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:38:45.125 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2376 (ObjNum 1080 snap 342 seq_num 1080) dirty exists 2022-04-22T17:38:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:45 smithi121 bash[36902]: cluster 2022-04-22T17:38:43.999619+0000 mgr.y (mgr.34107) 488 : cluster [DBG] pgmap v920: 121 pgs: 121 active+clean; 286 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T17:38:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:45 smithi121 bash[36902]: cluster 2022-04-22T17:38:44.104569+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2022-04-22T17:38:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:45 smithi121 bash[36902]: audit 2022-04-22T17:38:44.472235+0000 mon.a (mon.0) 1527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:45 smithi121 bash[36902]: cluster 2022-04-22T17:38:45.019948+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2022-04-22T17:38:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[45495]: cluster 2022-04-22T17:38:43.999619+0000 mgr.y (mgr.34107) 488 : cluster [DBG] pgmap v920: 121 pgs: 121 active+clean; 286 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[45495]: cluster 2022-04-22T17:38:44.104569+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[45495]: audit 2022-04-22T17:38:44.472235+0000 mon.a (mon.0) 1527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[45495]: cluster 2022-04-22T17:38:45.019948+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[46934]: cluster 2022-04-22T17:38:43.999619+0000 mgr.y (mgr.34107) 488 : cluster [DBG] pgmap v920: 121 pgs: 121 active+clean; 286 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[46934]: cluster 2022-04-22T17:38:44.104569+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[46934]: audit 2022-04-22T17:38:44.472235+0000 mon.a (mon.0) 1527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:45 smithi115 bash[46934]: cluster 2022-04-22T17:38:45.019948+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2022-04-22T17:38:46.031 INFO:tasks.rados.rados.0.smithi115.stdout:3845: done (2 left) 2022-04-22T17:38:46.068 INFO:tasks.rados.rados.0.smithi115.stdout:3846: done (1 left) 2022-04-22T17:38:46.069 INFO:tasks.rados.rados.0.smithi115.stdout:3847: done (0 left) 2022-04-22T17:38:46.069 INFO:tasks.rados.rados.0.smithi115.stdout:3848: read oid 7 snap -1 2022-04-22T17:38:46.069 INFO:tasks.rados.rados.0.smithi115.stdout:3848: expect deleted 2022-04-22T17:38:46.069 INFO:tasks.rados.rados.0.smithi115.stdout:3849: copy_from oid 43 from oid 33 current snap is 376 2022-04-22T17:38:46.069 INFO:tasks.rados.rados.0.smithi115.stdout:3850: snap_remove snap 373 2022-04-22T17:38:47.039 INFO:tasks.rados.rados.0.smithi115.stdout:3850: done (2 left) 2022-04-22T17:38:47.039 INFO:tasks.rados.rados.0.smithi115.stdout:3851: setattr oid 5 current snap is 376 2022-04-22T17:38:47.041 INFO:tasks.rados.rados.0.smithi115.stdout:3852: snap_remove snap 375 2022-04-22T17:38:47.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:47 smithi115 bash[45495]: cluster 2022-04-22T17:38:46.000320+0000 mgr.y (mgr.34107) 489 : cluster [DBG] pgmap v923: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2022-04-22T17:38:47.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:47 smithi115 bash[45495]: cluster 2022-04-22T17:38:46.018707+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2022-04-22T17:38:47.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:47 smithi115 bash[46934]: cluster 2022-04-22T17:38:46.000320+0000 mgr.y (mgr.34107) 489 : cluster [DBG] pgmap v923: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2022-04-22T17:38:47.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:47 smithi115 bash[46934]: cluster 2022-04-22T17:38:46.018707+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2022-04-22T17:38:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:47 smithi121 bash[36902]: cluster 2022-04-22T17:38:46.000320+0000 mgr.y (mgr.34107) 489 : cluster [DBG] pgmap v923: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2022-04-22T17:38:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:47 smithi121 bash[36902]: cluster 2022-04-22T17:38:46.018707+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2022-04-22T17:38:48.047 INFO:tasks.rados.rados.0.smithi115.stdout:3852: done (3 left) 2022-04-22T17:38:48.047 INFO:tasks.rados.rados.0.smithi115.stdout:3853: read oid 46 snap -1 2022-04-22T17:38:48.047 INFO:tasks.rados.rados.0.smithi115.stdout:3853: expect (ObjNum 1032 snap 323 seq_num 1032) 2022-04-22T17:38:48.047 INFO:tasks.rados.rados.0.smithi115.stdout:3854: copy_from oid 28 from oid 19 current snap is 376 2022-04-22T17:38:48.047 INFO:tasks.rados.rados.0.smithi115.stdout:3849: finishing copy_from to smithi11541303-43 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2045 (ObjNum 1049 snap 329 seq_num 1049) dirty exists 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:3849: finishing copy_from racing read to smithi11541303-43 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 5 v 2378 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:3848: done (4 left) 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:3849: done (3 left) 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:3851: done (2 left) 2022-04-22T17:38:48.048 INFO:tasks.rados.rados.0.smithi115.stdout:3855: snap_remove snap 365 2022-04-22T17:38:48.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:48 smithi115 bash[45495]: cluster 2022-04-22T17:38:47.026031+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2022-04-22T17:38:48.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:48 smithi115 bash[46934]: cluster 2022-04-22T17:38:47.026031+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2022-04-22T17:38:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:48 smithi121 bash[36902]: cluster 2022-04-22T17:38:47.026031+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2022-04-22T17:38:49.063 INFO:tasks.rados.rados.0.smithi115.stdout:3854: finishing copy_from to smithi11541303-28 2022-04-22T17:38:49.063 INFO:tasks.rados.rados.0.smithi115.stdout:3854: got expected ENOENT (src dne) 2022-04-22T17:38:49.063 INFO:tasks.rados.rados.0.smithi115.stdout:3854: finishing copy_from racing read to smithi11541303-28 2022-04-22T17:38:49.063 INFO:tasks.rados.rados.0.smithi115.stdout:3853: expect (ObjNum 1032 snap 323 seq_num 1032) 2022-04-22T17:38:49.071 INFO:tasks.rados.rados.0.smithi115.stdout:3853: done (2 left) 2022-04-22T17:38:49.071 INFO:tasks.rados.rados.0.smithi115.stdout:3854: done (1 left) 2022-04-22T17:38:49.071 INFO:tasks.rados.rados.0.smithi115.stdout:3855: done (0 left) 2022-04-22T17:38:49.072 INFO:tasks.rados.rados.0.smithi115.stdout:3856: delete oid 42 current snap is 376 2022-04-22T17:38:49.074 INFO:tasks.rados.rados.0.smithi115.stdout:3856: done (0 left) 2022-04-22T17:38:49.074 INFO:tasks.rados.rados.0.smithi115.stdout:3857: snap_create 2022-04-22T17:38:49.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:49 smithi115 bash[45495]: cluster 2022-04-22T17:38:48.001036+0000 mgr.y (mgr.34107) 490 : cluster [DBG] pgmap v926: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2022-04-22T17:38:49.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:49 smithi115 bash[45495]: cluster 2022-04-22T17:38:48.034308+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2022-04-22T17:38:49.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:49 smithi115 bash[46934]: cluster 2022-04-22T17:38:48.001036+0000 mgr.y (mgr.34107) 490 : cluster [DBG] pgmap v926: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2022-04-22T17:38:49.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:49 smithi115 bash[46934]: cluster 2022-04-22T17:38:48.034308+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2022-04-22T17:38:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:49 smithi121 bash[36902]: cluster 2022-04-22T17:38:48.001036+0000 mgr.y (mgr.34107) 490 : cluster [DBG] pgmap v926: 121 pgs: 121 active+clean; 288 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2022-04-22T17:38:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:49 smithi121 bash[36902]: cluster 2022-04-22T17:38:48.034308+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2022-04-22T17:38:50.029 INFO:tasks.rados.rados.0.smithi115.stdout:3857: done (0 left) 2022-04-22T17:38:50.059 INFO:tasks.rados.rados.0.smithi115.stdout:3858: copy_from oid 20 from oid 28 current snap is 377 2022-04-22T17:38:50.059 INFO:tasks.rados.rados.0.smithi115.stdout:3859: rollback oid 6 current snap is 377 2022-04-22T17:38:50.059 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 372 2022-04-22T17:38:50.059 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 377 2022-04-22T17:38:50.059 INFO:tasks.rados.rados.0.smithi115.stdout:3860: seq_num 1153 ranges {3088384=114688} 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:3860: writing smithi11541303-33 from 3088384 to 3203072 tid 1 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 19 current snap is 377 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:3859: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:3861: seq_num 1154 ranges {0=131072} 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:3861: writing smithi11541303-19 from 0 to 131072 tid 1 2022-04-22T17:38:50.062 INFO:tasks.rados.rados.0.smithi115.stdout:3862: rmattr oid 48 current snap is 377 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3863: copy_from oid 17 from oid 47 current snap is 377 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3864: setattr oid 35 current snap is 377 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3859: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3859: finishing rollback tid 2 to smithi11541303-6 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1948 (ObjNum 1125 snap 359 seq_num 1125) dirty exists 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3859: done (6 left) 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3865: rollback oid 32 current snap is 377 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 32 to 363 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3866: read oid 45 snap -1 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3866: expect (ObjNum 1111 snap 353 seq_num 1111) 2022-04-22T17:38:50.063 INFO:tasks.rados.rados.0.smithi115.stdout:3867: rollback oid 31 current snap is 377 2022-04-22T17:38:50.064 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 371 2022-04-22T17:38:50.064 INFO:tasks.rados.rados.0.smithi115.stdout:3868: rollback oid 4 current snap is 377 2022-04-22T17:38:50.064 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 4 to 376 2022-04-22T17:38:50.064 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 377 2022-04-22T17:38:50.064 INFO:tasks.rados.rados.0.smithi115.stdout:3869: seq_num 1155 ranges {6471680=147456} 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3869: writing smithi11541303-18 from 6471680 to 6619136 tid 1 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3870: snap_create 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3861: finishing write tid 1 to smithi11541303-19 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3863: finishing copy_from to smithi11541303-17 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3863: got expected ENOENT (src dne) 2022-04-22T17:38:50.096 INFO:tasks.rados.rados.0.smithi115.stdout:3863: finishing copy_from racing read to smithi11541303-17 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3867: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2089 (ObjNum 1150 snap 373 seq_num 1150) dirty exists 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3860: finishing write tid 1 to smithi11541303-33 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3860: finishing write tid 2 to smithi11541303-33 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3865: finishing rollback tid 0 to smithi11541303-32 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3865: finishing rollback tid 1 to smithi11541303-32 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3861: finishing write tid 2 to smithi11541303-19 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 2072 (ObjNum 1144 snap 371 seq_num 1144) dirty exists 2022-04-22T17:38:50.097 INFO:tasks.rados.rados.0.smithi115.stdout:3865: finishing rollback tid 2 to smithi11541303-32 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2078 (ObjNum 1131 snap 361 seq_num 1131) dirty exists 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3860: finishing write tid 3 to smithi11541303-33 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 2075 (ObjNum 1153 snap 377 seq_num 1153) dirty exists 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3860: left oid 33 (ObjNum 1153 snap 377 seq_num 1153) 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3867: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3867: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2394 (ObjNum 1080 snap 342 seq_num 1080) dirty exists 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3861: finishing write tid 3 to smithi11541303-19 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 2250 (ObjNum 1154 snap 377 seq_num 1154) dirty exists 2022-04-22T17:38:50.098 INFO:tasks.rados.rados.0.smithi115.stdout:3861: left oid 19 (ObjNum 1154 snap 377 seq_num 1154) 2022-04-22T17:38:50.103 INFO:tasks.rados.rados.0.smithi115.stdout:3868: finishing rollback tid 0 to smithi11541303-4 2022-04-22T17:38:50.104 INFO:tasks.rados.rados.0.smithi115.stdout:3869: finishing write tid 1 to smithi11541303-18 2022-04-22T17:38:50.104 INFO:tasks.rados.rados.0.smithi115.stdout:3869: finishing write tid 2 to smithi11541303-18 2022-04-22T17:38:50.105 INFO:tasks.rados.rados.0.smithi115.stdout:3858: finishing copy_from to smithi11541303-20 2022-04-22T17:38:50.105 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 1911 (ObjNum 1126 snap 360 seq_num 1126) dirty exists 2022-04-22T17:38:50.107 INFO:tasks.rados.rados.0.smithi115.stdout:3858: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:38:50.107 INFO:tasks.rados.rados.0.smithi115.stdout:3869: finishing write tid 3 to smithi11541303-18 2022-04-22T17:38:50.107 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2397 (ObjNum 1155 snap 377 seq_num 1155) dirty exists 2022-04-22T17:38:50.107 INFO:tasks.rados.rados.0.smithi115.stdout:3869: left oid 18 (ObjNum 1155 snap 377 seq_num 1155) 2022-04-22T17:38:50.114 INFO:tasks.rados.rados.0.smithi115.stdout:3866: expect (ObjNum 1111 snap 353 seq_num 1111) 2022-04-22T17:38:50.512 INFO:tasks.rados.rados.0.smithi115.stdout:3868: finishing rollback tid 1 to smithi11541303-4 2022-04-22T17:38:50.557 INFO:tasks.rados.rados.0.smithi115.stdout:3868: finishing rollback tid 2 to smithi11541303-4 2022-04-22T17:38:50.559 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 4 v 2092 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:50 smithi121 bash[36902]: cluster 2022-04-22T17:38:49.042270+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:50 smithi121 bash[36902]: audit 2022-04-22T17:38:49.475469+0000 mon.a (mon.0) 1533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:50 smithi121 bash[36902]: cluster 2022-04-22T17:38:50.015970+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[45495]: cluster 2022-04-22T17:38:49.042270+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[45495]: audit 2022-04-22T17:38:49.475469+0000 mon.a (mon.0) 1533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:50.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[45495]: cluster 2022-04-22T17:38:50.015970+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2022-04-22T17:38:50.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[46934]: cluster 2022-04-22T17:38:49.042270+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2022-04-22T17:38:50.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[46934]: audit 2022-04-22T17:38:49.475469+0000 mon.a (mon.0) 1533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:50.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:50 smithi115 bash[46934]: cluster 2022-04-22T17:38:50.015970+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2022-04-22T17:38:51.076 INFO:tasks.rados.rados.0.smithi115.stdout:3858: done (11 left) 2022-04-22T17:38:51.076 INFO:tasks.rados.rados.0.smithi115.stdout:3860: done (10 left) 2022-04-22T17:38:51.076 INFO:tasks.rados.rados.0.smithi115.stdout:3861: done (9 left) 2022-04-22T17:38:51.076 INFO:tasks.rados.rados.0.smithi115.stdout:3862: done (8 left) 2022-04-22T17:38:51.076 INFO:tasks.rados.rados.0.smithi115.stdout:3863: done (7 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3864: done (6 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3865: done (5 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3866: done (4 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3867: done (3 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3868: done (2 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3869: done (1 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3870: done (0 left) 2022-04-22T17:38:51.077 INFO:tasks.rados.rados.0.smithi115.stdout:3871: snap_create 2022-04-22T17:38:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:51 smithi121 bash[36902]: cluster 2022-04-22T17:38:50.001989+0000 mgr.y (mgr.34107) 491 : cluster [DBG] pgmap v929: 121 pgs: 121 active+clean; 302 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T17:38:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:51 smithi115 bash[45495]: cluster 2022-04-22T17:38:50.001989+0000 mgr.y (mgr.34107) 491 : cluster [DBG] pgmap v929: 121 pgs: 121 active+clean; 302 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T17:38:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:51 smithi115 bash[46934]: cluster 2022-04-22T17:38:50.001989+0000 mgr.y (mgr.34107) 491 : cluster [DBG] pgmap v929: 121 pgs: 121 active+clean; 302 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T17:38:52.056 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:51.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:52.075 INFO:tasks.rados.rados.0.smithi115.stdout:3871: done (0 left) 2022-04-22T17:38:52.075 INFO:tasks.rados.rados.0.smithi115.stdout:3872: copy_from oid 12 from oid 13 current snap is 379 2022-04-22T17:38:52.075 INFO:tasks.rados.rados.0.smithi115.stdout:3873: snap_create 2022-04-22T17:38:52.082 INFO:tasks.rados.rados.0.smithi115.stdout:3872: finishing copy_from to smithi11541303-12 2022-04-22T17:38:52.083 INFO:tasks.rados.rados.0.smithi115.stdout:3872: got expected ENOENT (src dne) 2022-04-22T17:38:52.083 INFO:tasks.rados.rados.0.smithi115.stdout:3872: finishing copy_from racing read to smithi11541303-12 2022-04-22T17:38:52.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:52 smithi121 bash[36902]: cluster 2022-04-22T17:38:51.059668+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2022-04-22T17:38:52.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:52 smithi121 bash[36902]: cluster 2022-04-22T17:38:52.062071+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2022-04-22T17:38:52.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:52 smithi115 bash[45495]: cluster 2022-04-22T17:38:51.059668+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2022-04-22T17:38:52.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:52 smithi115 bash[45495]: cluster 2022-04-22T17:38:52.062071+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2022-04-22T17:38:52.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:52 smithi115 bash[46934]: cluster 2022-04-22T17:38:51.059668+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2022-04-22T17:38:52.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:52 smithi115 bash[46934]: cluster 2022-04-22T17:38:52.062071+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2022-04-22T17:38:53.078 INFO:tasks.rados.rados.0.smithi115.stdout:3872: done (1 left) 2022-04-22T17:38:53.114 INFO:tasks.rados.rados.0.smithi115.stdout:3873: done (0 left) 2022-04-22T17:38:53.115 INFO:tasks.rados.rados.0.smithi115.stdout:3874: rollback oid 8 current snap is 380 2022-04-22T17:38:53.115 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 379 2022-04-22T17:38:53.115 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 32 current snap is 380 2022-04-22T17:38:53.115 INFO:tasks.rados.rados.0.smithi115.stdout:3875: seq_num 1156 ranges {5185536=163840} 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:3875: writing smithi11541303-32 from 5185536 to 5349376 tid 1 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:3876: snap_create 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:3874: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:3874: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:3874: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:38:53.116 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 2080 (ObjNum 1111 snap 353 seq_num 1111) dirty exists 2022-04-22T17:38:53.117 INFO:tasks.rados.rados.0.smithi115.stdout:3875: finishing write tid 1 to smithi11541303-32 2022-04-22T17:38:53.118 INFO:tasks.rados.rados.0.smithi115.stdout:3875: finishing write tid 2 to smithi11541303-32 2022-04-22T17:38:53.123 INFO:tasks.rados.rados.0.smithi115.stdout:3875: finishing write tid 3 to smithi11541303-32 2022-04-22T17:38:53.123 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 32 v 2080 (ObjNum 1156 snap 380 seq_num 1156) dirty exists 2022-04-22T17:38:53.123 INFO:tasks.rados.rados.0.smithi115.stdout:3875: left oid 32 (ObjNum 1156 snap 380 seq_num 1156) 2022-04-22T17:38:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:53 smithi121 bash[36902]: cluster 2022-04-22T17:38:52.003074+0000 mgr.y (mgr.34107) 492 : cluster [DBG] pgmap v932: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 361 KiB/s rd, 3.6 MiB/s wr, 18 op/s 2022-04-22T17:38:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:53 smithi121 bash[36902]: cluster 2022-04-22T17:38:53.065225+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2022-04-22T17:38:53.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:53 smithi115 bash[46934]: cluster 2022-04-22T17:38:52.003074+0000 mgr.y (mgr.34107) 492 : cluster [DBG] pgmap v932: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 361 KiB/s rd, 3.6 MiB/s wr, 18 op/s 2022-04-22T17:38:53.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:53 smithi115 bash[46934]: cluster 2022-04-22T17:38:53.065225+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2022-04-22T17:38:53.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:53 smithi115 bash[45495]: cluster 2022-04-22T17:38:52.003074+0000 mgr.y (mgr.34107) 492 : cluster [DBG] pgmap v932: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 361 KiB/s rd, 3.6 MiB/s wr, 18 op/s 2022-04-22T17:38:53.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:53 smithi115 bash[45495]: cluster 2022-04-22T17:38:53.065225+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2022-04-22T17:38:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:38:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:38:53.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:38:54.102 INFO:tasks.rados.rados.0.smithi115.stdout:3874: done (2 left) 2022-04-22T17:38:54.102 INFO:tasks.rados.rados.0.smithi115.stdout:3875: done (1 left) 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3876: done (0 left) 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3877: rmattr oid 22 current snap is 381 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3877: done (0 left) 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3878: read oid 10 snap -1 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3878: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:54.103 INFO:tasks.rados.rados.0.smithi115.stdout:3879: rollback oid 19 current snap is 381 2022-04-22T17:38:54.104 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 19 to 374 2022-04-22T17:38:54.104 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 381 2022-04-22T17:38:54.104 INFO:tasks.rados.rados.0.smithi115.stdout:3880: seq_num 1157 ranges {0=180224} 2022-04-22T17:38:54.104 INFO:tasks.rados.rados.0.smithi115.stdout:3880: writing smithi11541303-34 from 0 to 180224 tid 1 2022-04-22T17:38:54.104 INFO:tasks.rados.rados.0.smithi115.stdout:3881: delete oid 12 current snap is 381 2022-04-22T17:38:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3879: finishing rollback tid 0 to smithi11541303-19 2022-04-22T17:38:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:3879: finishing rollback tid 1 to smithi11541303-19 2022-04-22T17:38:54.111 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 19 v 2252 (ObjNum 0 snap 32692 seq_num 3178115104) dirty dne 2022-04-22T17:38:54.112 INFO:tasks.rados.rados.0.smithi115.stdout:3880: finishing write tid 1 to smithi11541303-34 2022-04-22T17:38:54.112 INFO:tasks.rados.rados.0.smithi115.stdout:3880: finishing write tid 2 to smithi11541303-34 2022-04-22T17:38:54.115 INFO:tasks.rados.rados.0.smithi115.stdout:3879: done (3 left) 2022-04-22T17:38:54.115 INFO:tasks.rados.rados.0.smithi115.stdout:3881: done (2 left) 2022-04-22T17:38:54.115 INFO:tasks.rados.rados.0.smithi115.stdout:3882: snap_remove snap 377 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3882: done (2 left) 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3883: read oid 3 snap -1 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3883: expect deleted 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3884: read oid 25 snap -1 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3884: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 16 current snap is 381 2022-04-22T17:38:55.039 INFO:tasks.rados.rados.0.smithi115.stdout:3885: seq_num 1159 ranges {11247616=212992} 2022-04-22T17:38:55.101 INFO:tasks.rados.rados.0.smithi115.stdout:3885: writing smithi11541303-16 from 11247616 to 11460608 tid 1 2022-04-22T17:38:55.101 INFO:tasks.rados.rados.0.smithi115.stdout:3886: delete oid 19 current snap is 381 2022-04-22T17:38:55.101 INFO:tasks.rados.rados.0.smithi115.stdout:3880: finishing write tid 3 to smithi11541303-34 2022-04-22T17:38:55.101 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 2098 (ObjNum 1157 snap 381 seq_num 1157) dirty exists 2022-04-22T17:38:55.101 INFO:tasks.rados.rados.0.smithi115.stdout:3880: left oid 34 (ObjNum 1157 snap 381 seq_num 1157) 2022-04-22T17:38:55.102 INFO:tasks.rados.rados.0.smithi115.stdout:3878: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:55 smithi121 bash[36902]: cluster 2022-04-22T17:38:54.004142+0000 mgr.y (mgr.34107) 493 : cluster [DBG] pgmap v935: 121 pgs: 121 active+clean; 312 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.2 MiB/s wr, 16 op/s 2022-04-22T17:38:55.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:55 smithi121 bash[36902]: cluster 2022-04-22T17:38:54.088802+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2022-04-22T17:38:55.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:55 smithi121 bash[36902]: audit 2022-04-22T17:38:54.478667+0000 mon.a (mon.0) 1539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:55.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:55 smithi121 bash[36902]: cluster 2022-04-22T17:38:55.019313+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2022-04-22T17:38:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[46934]: cluster 2022-04-22T17:38:54.004142+0000 mgr.y (mgr.34107) 493 : cluster [DBG] pgmap v935: 121 pgs: 121 active+clean; 312 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.2 MiB/s wr, 16 op/s 2022-04-22T17:38:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[46934]: cluster 2022-04-22T17:38:54.088802+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[46934]: audit 2022-04-22T17:38:54.478667+0000 mon.a (mon.0) 1539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[46934]: cluster 2022-04-22T17:38:55.019313+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[45495]: cluster 2022-04-22T17:38:54.004142+0000 mgr.y (mgr.34107) 493 : cluster [DBG] pgmap v935: 121 pgs: 121 active+clean; 312 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.2 MiB/s wr, 16 op/s 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[45495]: cluster 2022-04-22T17:38:54.088802+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[45495]: audit 2022-04-22T17:38:54.478667+0000 mon.a (mon.0) 1539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:38:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:55 smithi115 bash[45495]: cluster 2022-04-22T17:38:55.019313+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2022-04-22T17:38:56.178 INFO:tasks.rados.rados.0.smithi115.stdout:3884: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:38:56.797 INFO:tasks.rados.rados.0.smithi115.stdout:3885: finishing write tid 1 to smithi11541303-16 2022-04-22T17:38:56.801 INFO:tasks.rados.rados.0.smithi115.stdout:3885: finishing write tid 2 to smithi11541303-16 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3885: finishing write tid 3 to smithi11541303-16 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 16 v 2100 (ObjNum 1159 snap 381 seq_num 1159) dirty exists 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3885: left oid 16 (ObjNum 1159 snap 381 seq_num 1159) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3878: done (5 left) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3880: done (4 left) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3883: done (3 left) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3884: done (2 left) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3885: done (1 left) 2022-04-22T17:38:56.802 INFO:tasks.rados.rados.0.smithi115.stdout:3886: done (0 left) 2022-04-22T17:38:56.803 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 18 current snap is 381 2022-04-22T17:38:56.803 INFO:tasks.rados.rados.0.smithi115.stdout:3887: seq_num 1161 ranges {6619136=245760} 2022-04-22T17:38:56.843 INFO:tasks.rados.rados.0.smithi115.stdout:3887: writing smithi11541303-18 from 6619136 to 6864896 tid 1 2022-04-22T17:38:56.843 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 34 current snap is 381 2022-04-22T17:38:56.843 INFO:tasks.rados.rados.0.smithi115.stdout:3888: seq_num 1162 ranges {180224=270336} 2022-04-22T17:38:56.845 INFO:tasks.rados.rados.0.smithi115.stdout:3888: writing smithi11541303-34 from 180224 to 450560 tid 1 2022-04-22T17:38:56.845 INFO:tasks.rados.rados.0.smithi115.stdout:3889: delete oid 32 current snap is 381 2022-04-22T17:38:56.851 INFO:tasks.rados.rados.0.smithi115.stdout:3888: finishing write tid 1 to smithi11541303-34 2022-04-22T17:38:56.851 INFO:tasks.rados.rados.0.smithi115.stdout:3888: finishing write tid 2 to smithi11541303-34 2022-04-22T17:38:56.852 INFO:tasks.rados.rados.0.smithi115.stdout:3889: done (2 left) 2022-04-22T17:38:56.852 INFO:tasks.rados.rados.0.smithi115.stdout:3890: read oid 24 snap -1 2022-04-22T17:38:56.852 INFO:tasks.rados.rados.0.smithi115.stdout:3890: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:38:56.852 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 381 2022-04-22T17:38:56.852 INFO:tasks.rados.rados.0.smithi115.stdout:3891: seq_num 1164 ranges {7159808=303104} 2022-04-22T17:38:56.896 INFO:tasks.rados.rados.0.smithi115.stdout:3891: writing smithi11541303-49 from 7159808 to 7462912 tid 1 2022-04-22T17:38:56.896 INFO:tasks.rados.rados.0.smithi115.stdout:3892: rmattr oid 38 current snap is 381 2022-04-22T17:38:56.896 INFO:tasks.rados.rados.0.smithi115.stdout:3887: finishing write tid 1 to smithi11541303-18 2022-04-22T17:38:56.896 INFO:tasks.rados.rados.0.smithi115.stdout:3893: read oid 6 snap -1 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3893: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3894: delete oid 40 current snap is 381 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3888: finishing write tid 3 to smithi11541303-34 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 34 v 2102 (ObjNum 1162 snap 381 seq_num 1162) dirty exists 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3888: left oid 34 (ObjNum 1162 snap 381 seq_num 1162) 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3887: finishing write tid 2 to smithi11541303-18 2022-04-22T17:38:56.897 INFO:tasks.rados.rados.0.smithi115.stdout:3887: finishing write tid 3 to smithi11541303-18 2022-04-22T17:38:56.898 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2400 (ObjNum 1161 snap 381 seq_num 1161) dirty exists 2022-04-22T17:38:56.898 INFO:tasks.rados.rados.0.smithi115.stdout:3887: left oid 18 (ObjNum 1161 snap 381 seq_num 1161) 2022-04-22T17:38:56.898 INFO:tasks.rados.rados.0.smithi115.stdout:3890: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:38:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:57 smithi121 bash[36902]: cluster 2022-04-22T17:38:56.004804+0000 mgr.y (mgr.34107) 494 : cluster [DBG] pgmap v938: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:57 smithi115 bash[46934]: cluster 2022-04-22T17:38:56.004804+0000 mgr.y (mgr.34107) 494 : cluster [DBG] pgmap v938: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:57.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:57 smithi115 bash[45495]: cluster 2022-04-22T17:38:56.004804+0000 mgr.y (mgr.34107) 494 : cluster [DBG] pgmap v938: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2254 (ObjNum 1093 snap 347 seq_num 1093) dirty exists 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:3891: finishing write tid 1 to smithi11541303-49 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:3891: finishing write tid 2 to smithi11541303-49 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:3891: finishing write tid 3 to smithi11541303-49 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2403 (ObjNum 1164 snap 381 seq_num 1164) dirty exists 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:3891: left oid 49 (ObjNum 1164 snap 381 seq_num 1164) 2022-04-22T17:38:57.589 INFO:tasks.rados.rados.0.smithi115.stdout:3893: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:38:58.634 INFO:tasks.rados.rados.0.smithi115.stdout:3887: done (6 left) 2022-04-22T17:38:58.634 INFO:tasks.rados.rados.0.smithi115.stdout:3888: done (5 left) 2022-04-22T17:38:58.634 INFO:tasks.rados.rados.0.smithi115.stdout:3890: done (4 left) 2022-04-22T17:38:58.634 INFO:tasks.rados.rados.0.smithi115.stdout:3891: done (3 left) 2022-04-22T17:38:58.635 INFO:tasks.rados.rados.0.smithi115.stdout:3892: done (2 left) 2022-04-22T17:38:58.635 INFO:tasks.rados.rados.0.smithi115.stdout:3893: done (1 left) 2022-04-22T17:38:58.635 INFO:tasks.rados.rados.0.smithi115.stdout:3894: done (0 left) 2022-04-22T17:38:58.635 INFO:tasks.rados.rados.0.smithi115.stdout:3895: snap_remove snap 372 2022-04-22T17:38:59.119 INFO:tasks.rados.rados.0.smithi115.stdout:3895: done (0 left) 2022-04-22T17:38:59.119 INFO:tasks.rados.rados.0.smithi115.stdout:3896: snap_create 2022-04-22T17:38:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:38:59 smithi121 bash[36902]: cluster 2022-04-22T17:38:58.005746+0000 mgr.y (mgr.34107) 495 : cluster [DBG] pgmap v939: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:38:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:38:59 smithi115 bash[45495]: cluster 2022-04-22T17:38:58.005746+0000 mgr.y (mgr.34107) 495 : cluster [DBG] pgmap v939: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:38:59.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:38:59 smithi115 bash[46934]: cluster 2022-04-22T17:38:58.005746+0000 mgr.y (mgr.34107) 495 : cluster [DBG] pgmap v939: 121 pgs: 121 active+clean; 313 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:39:00.128 INFO:tasks.rados.rados.0.smithi115.stdout:3896: done (0 left) 2022-04-22T17:39:00.128 INFO:tasks.rados.rados.0.smithi115.stdout:3897: copy_from oid 22 from oid 12 current snap is 382 2022-04-22T17:39:00.128 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 45 current snap is 382 2022-04-22T17:39:00.128 INFO:tasks.rados.rados.0.smithi115.stdout:3898: seq_num 1166 ranges {6209536=335872} 2022-04-22T17:39:00.166 INFO:tasks.rados.rados.0.smithi115.stdout:3898: writing smithi11541303-45 from 6209536 to 6545408 tid 1 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3899: rollback oid 21 current snap is 382 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 21 to 371 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3897: finishing copy_from to smithi11541303-22 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3897: got expected ENOENT (src dne) 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3897: finishing copy_from racing read to smithi11541303-22 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3897: done (2 left) 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3900: read oid 26 snap -1 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3900: expect (ObjNum 1130 snap 361 seq_num 1130) 2022-04-22T17:39:00.167 INFO:tasks.rados.rados.0.smithi115.stdout:3901: snap_remove snap 379 2022-04-22T17:39:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:00 smithi121 bash[36902]: cluster 2022-04-22T17:38:59.103504+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2022-04-22T17:39:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:00 smithi121 bash[36902]: audit 2022-04-22T17:38:59.481728+0000 mon.a (mon.0) 1542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:00 smithi115 bash[45495]: cluster 2022-04-22T17:38:59.103504+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2022-04-22T17:39:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:00 smithi115 bash[45495]: audit 2022-04-22T17:38:59.481728+0000 mon.a (mon.0) 1542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:00.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:00 smithi115 bash[46934]: cluster 2022-04-22T17:38:59.103504+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2022-04-22T17:39:00.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:00 smithi115 bash[46934]: audit 2022-04-22T17:38:59.481728+0000 mon.a (mon.0) 1542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:01.133 INFO:tasks.rados.rados.0.smithi115.stdout:3901: done (3 left) 2022-04-22T17:39:01.134 INFO:tasks.rados.rados.0.smithi115.stdout:3902: snap_remove snap 363 2022-04-22T17:39:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:01 smithi121 bash[36902]: cluster 2022-04-22T17:39:00.006587+0000 mgr.y (mgr.34107) 496 : cluster [DBG] pgmap v941: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 243 KiB/s wr, 7 op/s 2022-04-22T17:39:01.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:01 smithi121 bash[36902]: cluster 2022-04-22T17:39:00.112370+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2022-04-22T17:39:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:01 smithi115 bash[45495]: cluster 2022-04-22T17:39:00.006587+0000 mgr.y (mgr.34107) 496 : cluster [DBG] pgmap v941: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 243 KiB/s wr, 7 op/s 2022-04-22T17:39:01.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:01 smithi115 bash[45495]: cluster 2022-04-22T17:39:00.112370+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2022-04-22T17:39:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:01 smithi115 bash[46934]: cluster 2022-04-22T17:39:00.006587+0000 mgr.y (mgr.34107) 496 : cluster [DBG] pgmap v941: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 243 KiB/s wr, 7 op/s 2022-04-22T17:39:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:01 smithi115 bash[46934]: cluster 2022-04-22T17:39:00.112370+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2022-04-22T17:39:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:01.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3898: finishing write tid 1 to smithi11541303-45 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3898: finishing write tid 2 to smithi11541303-45 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3899: finishing rollback tid 0 to smithi11541303-21 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3899: finishing rollback tid 1 to smithi11541303-21 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3898: finishing write tid 3 to smithi11541303-45 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 45 v 2094 (ObjNum 1166 snap 382 seq_num 1166) dirty exists 2022-04-22T17:39:02.148 INFO:tasks.rados.rados.0.smithi115.stdout:3898: left oid 45 (ObjNum 1166 snap 382 seq_num 1166) 2022-04-22T17:39:02.149 INFO:tasks.rados.rados.0.smithi115.stdout:3899: finishing rollback tid 2 to smithi11541303-21 2022-04-22T17:39:02.149 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1898 (ObjNum 1110 snap 353 seq_num 1110) dirty exists 2022-04-22T17:39:02.149 INFO:tasks.rados.rados.0.smithi115.stdout:3900: expect (ObjNum 1130 snap 361 seq_num 1130) 2022-04-22T17:39:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:02 smithi121 bash[36902]: cluster 2022-04-22T17:39:01.121736+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2022-04-22T17:39:02.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:02 smithi115 bash[45495]: cluster 2022-04-22T17:39:01.121736+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2022-04-22T17:39:02.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:02 smithi115 bash[46934]: cluster 2022-04-22T17:39:01.121736+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2022-04-22T17:39:03.240 INFO:tasks.rados.rados.0.smithi115.stdout:3898: done (3 left) 2022-04-22T17:39:03.240 INFO:tasks.rados.rados.0.smithi115.stdout:3899: done (2 left) 2022-04-22T17:39:03.240 INFO:tasks.rados.rados.0.smithi115.stdout:3900: done (1 left) 2022-04-22T17:39:03.240 INFO:tasks.rados.rados.0.smithi115.stdout:3902: done (0 left) 2022-04-22T17:39:03.240 INFO:tasks.rados.rados.0.smithi115.stdout:3903: snap_create 2022-04-22T17:39:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:03 smithi121 bash[36902]: cluster 2022-04-22T17:39:02.007735+0000 mgr.y (mgr.34107) 497 : cluster [DBG] pgmap v944: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 197 KiB/s wr, 6 op/s 2022-04-22T17:39:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:03 smithi121 bash[36902]: cluster 2022-04-22T17:39:02.135326+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2022-04-22T17:39:03.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:03 smithi115 bash[46934]: cluster 2022-04-22T17:39:02.007735+0000 mgr.y (mgr.34107) 497 : cluster [DBG] pgmap v944: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 197 KiB/s wr, 6 op/s 2022-04-22T17:39:03.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:03 smithi115 bash[46934]: cluster 2022-04-22T17:39:02.135326+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2022-04-22T17:39:03.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:03 smithi115 bash[45495]: cluster 2022-04-22T17:39:02.007735+0000 mgr.y (mgr.34107) 497 : cluster [DBG] pgmap v944: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 197 KiB/s wr, 6 op/s 2022-04-22T17:39:03.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:03 smithi115 bash[45495]: cluster 2022-04-22T17:39:02.135326+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2022-04-22T17:39:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:03.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:03.994 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:3903: done (0 left) 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:3904: rollback oid 50 current snap is 383 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 381 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:3905: rollback oid 8 current snap is 383 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 8 to 380 2022-04-22T17:39:04.155 INFO:tasks.rados.rados.0.smithi115.stdout:3906: snap_remove snap 382 2022-04-22T17:39:04.349 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 2m ago 15m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (15m) 2m ago 15m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4m) 2m ago 16m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3m) 2m ago 16m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (11m) 2m ago 21m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:39:04.350 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (12m) 2m ago 23m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:39:04.351 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (11m) 2m ago 23m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:39:04.351 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (10m) 2m ago 21m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:39:04.351 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 2m ago 21m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:39:04.351 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 2m ago 16m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3m) 2m ago 16m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (8m) 2m ago 20m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 2m ago 20m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 2m ago 19m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 2m ago 19m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6m) 2m ago 18m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5m) 2m ago 18m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:39:04.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5m) 2m ago 17m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:39:04.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4m) 2m ago 17m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:39:04.353 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2m) 2m ago 16m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:39:04.748 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:39:04.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:39:04.750 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:39:05.047 INFO:tasks.rados.rados.0.smithi115.stdout:3906: done (2 left) 2022-04-22T17:39:05.082 INFO:tasks.rados.rados.0.smithi115.stdout:3907: snap_remove snap 371 2022-04-22T17:39:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: audit 2022-04-22T17:39:03.967765+0000 mgr.y (mgr.34107) 498 : audit [DBG] from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: cluster 2022-04-22T17:39:04.008508+0000 mgr.y (mgr.34107) 499 : cluster [DBG] pgmap v946: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 0 B/s wr, 2 op/s 2022-04-22T17:39:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: cluster 2022-04-22T17:39:04.141100+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2022-04-22T17:39:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: audit 2022-04-22T17:39:04.484603+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:05.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: audit 2022-04-22T17:39:04.736101+0000 mon.a (mon.0) 1548 : audit [DBG] from='client.? 172.21.15.115:0/1209826466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:05.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:05 smithi121 bash[36902]: cluster 2022-04-22T17:39:05.033077+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: audit 2022-04-22T17:39:03.967765+0000 mgr.y (mgr.34107) 498 : audit [DBG] from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: cluster 2022-04-22T17:39:04.008508+0000 mgr.y (mgr.34107) 499 : cluster [DBG] pgmap v946: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 0 B/s wr, 2 op/s 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: cluster 2022-04-22T17:39:04.141100+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: audit 2022-04-22T17:39:04.484603+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: audit 2022-04-22T17:39:04.736101+0000 mon.a (mon.0) 1548 : audit [DBG] from='client.? 172.21.15.115:0/1209826466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[45495]: cluster 2022-04-22T17:39:05.033077+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: audit 2022-04-22T17:39:03.967765+0000 mgr.y (mgr.34107) 498 : audit [DBG] from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: cluster 2022-04-22T17:39:04.008508+0000 mgr.y (mgr.34107) 499 : cluster [DBG] pgmap v946: 121 pgs: 121 active+clean; 314 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 0 B/s wr, 2 op/s 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: cluster 2022-04-22T17:39:04.141100+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: audit 2022-04-22T17:39:04.484603+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: audit 2022-04-22T17:39:04.736101+0000 mon.a (mon.0) 1548 : audit [DBG] from='client.? 172.21.15.115:0/1209826466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:05 smithi115 bash[46934]: cluster 2022-04-22T17:39:05.033077+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2022-04-22T17:39:06.051 INFO:tasks.rados.rados.0.smithi115.stdout:3907: done (2 left) 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3908: snap_create 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3905: finishing rollback tid 0 to smithi11541303-8 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3905: finishing rollback tid 1 to smithi11541303-8 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3905: finishing rollback tid 2 to smithi11541303-8 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 2083 (ObjNum 1111 snap 353 seq_num 1111) dirty exists 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3904: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3904: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:3904: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:39:06.083 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2105 (ObjNum 1138 snap 363 seq_num 1138) dirty exists 2022-04-22T17:39:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:06 smithi121 bash[36902]: audit 2022-04-22T17:39:04.318616+0000 mgr.y (mgr.34107) 500 : audit [DBG] from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:06 smithi121 bash[36902]: cluster 2022-04-22T17:39:06.035961+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2022-04-22T17:39:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:06 smithi115 bash[45495]: audit 2022-04-22T17:39:04.318616+0000 mgr.y (mgr.34107) 500 : audit [DBG] from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:06 smithi115 bash[45495]: cluster 2022-04-22T17:39:06.035961+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2022-04-22T17:39:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:06 smithi115 bash[46934]: audit 2022-04-22T17:39:04.318616+0000 mgr.y (mgr.34107) 500 : audit [DBG] from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:06 smithi115 bash[46934]: cluster 2022-04-22T17:39:06.035961+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2022-04-22T17:39:07.051 INFO:tasks.rados.rados.0.smithi115.stdout:3904: done (2 left) 2022-04-22T17:39:07.052 INFO:tasks.rados.rados.0.smithi115.stdout:3905: done (1 left) 2022-04-22T17:39:07.052 INFO:tasks.rados.rados.0.smithi115.stdout:3908: done (0 left) 2022-04-22T17:39:07.052 INFO:tasks.rados.rados.0.smithi115.stdout:3909: rmattr oid 26 current snap is 384 2022-04-22T17:39:07.052 INFO:tasks.rados.rados.0.smithi115.stdout:3910: snap_remove snap 374 2022-04-22T17:39:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:07 smithi121 bash[36902]: cluster 2022-04-22T17:39:06.009404+0000 mgr.y (mgr.34107) 501 : cluster [DBG] pgmap v949: 121 pgs: 121 active+clean; 293 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 67 KiB/s wr, 2 op/s 2022-04-22T17:39:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:07 smithi121 bash[36902]: cluster 2022-04-22T17:39:07.038945+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2022-04-22T17:39:07.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:07 smithi115 bash[45495]: cluster 2022-04-22T17:39:06.009404+0000 mgr.y (mgr.34107) 501 : cluster [DBG] pgmap v949: 121 pgs: 121 active+clean; 293 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 67 KiB/s wr, 2 op/s 2022-04-22T17:39:07.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:07 smithi115 bash[45495]: cluster 2022-04-22T17:39:07.038945+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2022-04-22T17:39:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:07 smithi115 bash[46934]: cluster 2022-04-22T17:39:06.009404+0000 mgr.y (mgr.34107) 501 : cluster [DBG] pgmap v949: 121 pgs: 121 active+clean; 293 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 67 KiB/s wr, 2 op/s 2022-04-22T17:39:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:07 smithi115 bash[46934]: cluster 2022-04-22T17:39:07.038945+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2022-04-22T17:39:08.052 INFO:tasks.rados.rados.0.smithi115.stdout:3910: done (1 left) 2022-04-22T17:39:08.052 INFO:tasks.rados.rados.0.smithi115.stdout:3911: snap_create 2022-04-22T17:39:08.052 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 26 v 2082 (ObjNum 1130 snap 361 seq_num 1130) dirty exists 2022-04-22T17:39:09.060 INFO:tasks.rados.rados.0.smithi115.stdout:3909: done (1 left) 2022-04-22T17:39:09.100 INFO:tasks.rados.rados.0.smithi115.stdout:3911: done (0 left) 2022-04-22T17:39:09.100 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 50 current snap is 385 2022-04-22T17:39:09.100 INFO:tasks.rados.rados.0.smithi115.stdout:3912: seq_num 1167 ranges {11575296=352256} 2022-04-22T17:39:09.132 INFO:tasks.rados.rados.0.smithi115.stdout:3912: writing smithi11541303-50 from 11575296 to 11927552 tid 1 2022-04-22T17:39:09.133 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 7 current snap is 385 2022-04-22T17:39:09.133 INFO:tasks.rados.rados.0.smithi115.stdout:3913: seq_num 1168 ranges {0=368640} 2022-04-22T17:39:09.135 INFO:tasks.rados.rados.0.smithi115.stdout:3913: writing smithi11541303-7 from 0 to 368640 tid 1 2022-04-22T17:39:09.135 INFO:tasks.rados.rados.0.smithi115.stdout:3914: rollback oid 43 current snap is 385 2022-04-22T17:39:09.135 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 43 to 384 2022-04-22T17:39:09.135 INFO:tasks.rados.rados.0.smithi115.stdout:3915: delete oid 23 current snap is 385 2022-04-22T17:39:09.147 INFO:tasks.rados.rados.0.smithi115.stdout:3915: done (3 left) 2022-04-22T17:39:09.147 INFO:tasks.rados.rados.0.smithi115.stdout:3916: read oid 1 snap 380 2022-04-22T17:39:09.147 INFO:tasks.rados.rados.0.smithi115.stdout:3916: expect (ObjNum 1123 snap 359 seq_num 1123) 2022-04-22T17:39:09.147 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 49 current snap is 385 2022-04-22T17:39:09.147 INFO:tasks.rados.rados.0.smithi115.stdout:3917: seq_num 1170 ranges {7462912=401408} 2022-04-22T17:39:09.195 INFO:tasks.rados.rados.0.smithi115.stdout:3917: writing smithi11541303-49 from 7462912 to 7864320 tid 1 2022-04-22T17:39:09.206 INFO:tasks.rados.rados.0.smithi115.stdout:3918: delete oid 23 current snap is 385 2022-04-22T17:39:09.206 INFO:tasks.rados.rados.0.smithi115.stdout:3912: finishing write tid 1 to smithi11541303-50 2022-04-22T17:39:09.206 INFO:tasks.rados.rados.0.smithi115.stdout:3914: finishing rollback tid 0 to smithi11541303-43 2022-04-22T17:39:09.206 INFO:tasks.rados.rados.0.smithi115.stdout:3912: finishing write tid 2 to smithi11541303-50 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3914: finishing rollback tid 1 to smithi11541303-43 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3914: finishing rollback tid 2 to smithi11541303-43 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2204 (ObjNum 1049 snap 329 seq_num 1049) dirty exists 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3912: finishing write tid 3 to smithi11541303-50 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2107 (ObjNum 1167 snap 385 seq_num 1167) dirty exists 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3912: left oid 50 (ObjNum 1167 snap 385 seq_num 1167) 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3913: finishing write tid 1 to smithi11541303-7 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3913: finishing write tid 2 to smithi11541303-7 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:3913: finishing write tid 3 to smithi11541303-7 2022-04-22T17:39:09.207 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 7 v 2085 (ObjNum 1168 snap 385 seq_num 1168) dirty exists 2022-04-22T17:39:09.208 INFO:tasks.rados.rados.0.smithi115.stdout:3913: left oid 7 (ObjNum 1168 snap 385 seq_num 1168) 2022-04-22T17:39:09.208 INFO:tasks.rados.rados.0.smithi115.stdout:3916: expect (ObjNum 1123 snap 359 seq_num 1123) 2022-04-22T17:39:09.324 INFO:tasks.rados.rados.0.smithi115.stdout:3917: finishing write tid 1 to smithi11541303-49 2022-04-22T17:39:09.324 INFO:tasks.rados.rados.0.smithi115.stdout:3917: finishing write tid 2 to smithi11541303-49 2022-04-22T17:39:09.324 INFO:tasks.rados.rados.0.smithi115.stdout:3917: finishing write tid 3 to smithi11541303-49 2022-04-22T17:39:09.324 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2450 (ObjNum 1170 snap 385 seq_num 1170) dirty exists 2022-04-22T17:39:09.324 INFO:tasks.rados.rados.0.smithi115.stdout:3917: left oid 49 (ObjNum 1170 snap 385 seq_num 1170) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3912: done (5 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3913: done (4 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3914: done (3 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3916: done (2 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3917: done (1 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3918: done (0 left) 2022-04-22T17:39:09.325 INFO:tasks.rados.rados.0.smithi115.stdout:3919: delete oid 14 current snap is 385 2022-04-22T17:39:09.329 INFO:tasks.rados.rados.0.smithi115.stdout:3919: done (0 left) 2022-04-22T17:39:09.329 INFO:tasks.rados.rados.0.smithi115.stdout:3920: copy_from oid 14 from oid 39 current snap is 385 2022-04-22T17:39:09.329 INFO:tasks.rados.rados.0.smithi115.stdout:3921: read oid 25 snap -1 2022-04-22T17:39:09.330 INFO:tasks.rados.rados.0.smithi115.stdout:3921: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:39:09.330 INFO:tasks.rados.rados.0.smithi115.stdout:3922: setattr oid 3 current snap is 385 2022-04-22T17:39:09.332 INFO:tasks.rados.rados.0.smithi115.stdout:3923: rmattr oid 35 current snap is 385 2022-04-22T17:39:09.332 INFO:tasks.rados.rados.0.smithi115.stdout:3924: snap_remove snap 380 2022-04-22T17:39:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:09 smithi121 bash[36902]: cluster 2022-04-22T17:39:08.010255+0000 mgr.y (mgr.34107) 502 : cluster [DBG] pgmap v952: 121 pgs: 121 active+clean; 268 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.2 MiB/s rd, 82 KiB/s wr, 2 op/s 2022-04-22T17:39:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:09 smithi121 bash[36902]: cluster 2022-04-22T17:39:08.039676+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2022-04-22T17:39:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:09 smithi115 bash[45495]: cluster 2022-04-22T17:39:08.010255+0000 mgr.y (mgr.34107) 502 : cluster [DBG] pgmap v952: 121 pgs: 121 active+clean; 268 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.2 MiB/s rd, 82 KiB/s wr, 2 op/s 2022-04-22T17:39:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:09 smithi115 bash[45495]: cluster 2022-04-22T17:39:08.039676+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2022-04-22T17:39:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:09 smithi115 bash[46934]: cluster 2022-04-22T17:39:08.010255+0000 mgr.y (mgr.34107) 502 : cluster [DBG] pgmap v952: 121 pgs: 121 active+clean; 268 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.2 MiB/s rd, 82 KiB/s wr, 2 op/s 2022-04-22T17:39:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:09 smithi115 bash[46934]: cluster 2022-04-22T17:39:08.039676+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2022-04-22T17:39:10.049 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 2073 (ObjNum 1144 snap 371 seq_num 1144) dirty exists 2022-04-22T17:39:10.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2241 (ObjNum 1953066355 snap 825321832 seq_num 858862645) dirty exists 2022-04-22T17:39:10.050 INFO:tasks.rados.rados.0.smithi115.stdout:3920: finishing copy_from to smithi11541303-14 2022-04-22T17:39:10.050 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 2071 (ObjNum 909 snap 299 seq_num 909) dirty exists 2022-04-22T17:39:10.050 INFO:tasks.rados.rados.0.smithi115.stdout:3920: finishing copy_from racing read to smithi11541303-14 2022-04-22T17:39:10.050 INFO:tasks.rados.rados.0.smithi115.stdout:3921: expect (ObjNum 1081 snap 342 seq_num 1081) 2022-04-22T17:39:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:10 smithi121 bash[36902]: cluster 2022-04-22T17:39:09.046638+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2022-04-22T17:39:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:10 smithi121 bash[36902]: audit 2022-04-22T17:39:09.487467+0000 mon.a (mon.0) 1554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:10 smithi121 bash[36902]: cluster 2022-04-22T17:39:10.036272+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2022-04-22T17:39:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[46934]: cluster 2022-04-22T17:39:09.046638+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2022-04-22T17:39:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[46934]: audit 2022-04-22T17:39:09.487467+0000 mon.a (mon.0) 1554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[46934]: cluster 2022-04-22T17:39:10.036272+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2022-04-22T17:39:10.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[45495]: cluster 2022-04-22T17:39:09.046638+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2022-04-22T17:39:10.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[45495]: audit 2022-04-22T17:39:09.487467+0000 mon.a (mon.0) 1554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:10.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:10 smithi115 bash[45495]: cluster 2022-04-22T17:39:10.036272+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2022-04-22T17:39:10.709 INFO:tasks.rados.rados.0.smithi115.stdout:3920: done (4 left) 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:3921: done (3 left) 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:3922: done (2 left) 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:3923: done (1 left) 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:3924: done (0 left) 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 40 current snap is 385 2022-04-22T17:39:10.710 INFO:tasks.rados.rados.0.smithi115.stdout:3925: seq_num 1173 ranges {0=450560} 2022-04-22T17:39:10.712 INFO:tasks.rados.rados.0.smithi115.stdout:3925: writing smithi11541303-40 from 0 to 450560 tid 1 2022-04-22T17:39:10.712 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 14 current snap is 385 2022-04-22T17:39:10.712 INFO:tasks.rados.rados.0.smithi115.stdout:3926: seq_num 1174 ranges {827392=466944} 2022-04-22T17:39:10.719 INFO:tasks.rados.rados.0.smithi115.stdout:3926: writing smithi11541303-14 from 827392 to 1294336 tid 1 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:3927: read oid 19 snap -1 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:3927: expect deleted 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:3928: rollback oid 6 current snap is 385 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 6 to 381 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 3 current snap is 385 2022-04-22T17:39:10.720 INFO:tasks.rados.rados.0.smithi115.stdout:3929: seq_num 1175 ranges {0=483328} 2022-04-22T17:39:10.722 INFO:tasks.rados.rados.0.smithi115.stdout:3929: writing smithi11541303-3 from 0 to 483328 tid 1 2022-04-22T17:39:10.723 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 43 current snap is 385 2022-04-22T17:39:10.723 INFO:tasks.rados.rados.0.smithi115.stdout:3930: seq_num 1176 ranges {3088384=499712} 2022-04-22T17:39:10.744 INFO:tasks.rados.rados.0.smithi115.stdout:3930: writing smithi11541303-43 from 3088384 to 3588096 tid 1 2022-04-22T17:39:10.744 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 12 current snap is 385 2022-04-22T17:39:10.744 INFO:tasks.rados.rados.0.smithi115.stdout:3931: seq_num 1177 ranges {0=516096} 2022-04-22T17:39:10.746 INFO:tasks.rados.rados.0.smithi115.stdout:3931: writing smithi11541303-12 from 0 to 516096 tid 1 2022-04-22T17:39:10.747 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 25 current snap is 385 2022-04-22T17:39:10.747 INFO:tasks.rados.rados.0.smithi115.stdout:3932: seq_num 1178 ranges {8953856=532480} 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:3932: writing smithi11541303-25 from 8953856 to 9486336 tid 1 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:3933: read oid 32 snap 378 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:3933: expect (ObjNum 1131 snap 361 seq_num 1131) 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 8 current snap is 385 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:3925: finishing write tid 1 to smithi11541303-40 2022-04-22T17:39:10.800 INFO:tasks.rados.rados.0.smithi115.stdout:3934: seq_num 1179 ranges {6209536=548864} 2022-04-22T17:39:10.843 INFO:tasks.rados.rados.0.smithi115.stdout:3934: writing smithi11541303-8 from 6209536 to 6758400 tid 1 2022-04-22T17:39:10.843 INFO:tasks.rados.rados.0.smithi115.stdout:3935: snap_remove snap 384 2022-04-22T17:39:10.843 INFO:tasks.rados.rados.0.smithi115.stdout:3925: finishing write tid 2 to smithi11541303-40 2022-04-22T17:39:10.844 INFO:tasks.rados.rados.0.smithi115.stdout:3925: finishing write tid 3 to smithi11541303-40 2022-04-22T17:39:10.893 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 40 v 2077 (ObjNum 1173 snap 385 seq_num 1173) dirty exists 2022-04-22T17:39:10.893 INFO:tasks.rados.rados.0.smithi115.stdout:3925: left oid 40 (ObjNum 1173 snap 385 seq_num 1173) 2022-04-22T17:39:10.893 INFO:tasks.rados.rados.0.smithi115.stdout:3929: finishing write tid 1 to smithi11541303-3 2022-04-22T17:39:11.095 INFO:tasks.rados.rados.0.smithi115.stdout:3929: finishing write tid 2 to smithi11541303-3 2022-04-22T17:39:11.095 INFO:tasks.rados.rados.0.smithi115.stdout:3928: finishing rollback tid 0 to smithi11541303-6 2022-04-22T17:39:11.095 INFO:tasks.rados.rados.0.smithi115.stdout:3926: finishing write tid 1 to smithi11541303-14 2022-04-22T17:39:11.095 INFO:tasks.rados.rados.0.smithi115.stdout:3926: finishing write tid 2 to smithi11541303-14 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:3928: finishing rollback tid 1 to smithi11541303-6 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:3929: finishing write tid 3 to smithi11541303-3 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 3 v 2243 (ObjNum 1175 snap 385 seq_num 1175) dirty exists 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:3929: left oid 3 (ObjNum 1175 snap 385 seq_num 1175) 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:3928: finishing rollback tid 2 to smithi11541303-6 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 6 v 1951 (ObjNum 1125 snap 359 seq_num 1125) dirty exists 2022-04-22T17:39:11.096 INFO:tasks.rados.rados.0.smithi115.stdout:3926: finishing write tid 3 to smithi11541303-14 2022-04-22T17:39:11.097 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 14 v 2073 (ObjNum 1174 snap 385 seq_num 1174) dirty exists 2022-04-22T17:39:11.098 INFO:tasks.rados.rados.0.smithi115.stdout:3926: left oid 14 (ObjNum 1174 snap 385 seq_num 1174) 2022-04-22T17:39:11.098 INFO:tasks.rados.rados.0.smithi115.stdout:3930: finishing write tid 1 to smithi11541303-43 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3931: finishing write tid 1 to smithi11541303-12 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3930: finishing write tid 2 to smithi11541303-43 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3931: finishing write tid 2 to smithi11541303-12 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3931: finishing write tid 3 to smithi11541303-12 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 12 v 2256 (ObjNum 1177 snap 385 seq_num 1177) dirty exists 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3931: left oid 12 (ObjNum 1177 snap 385 seq_num 1177) 2022-04-22T17:39:11.099 INFO:tasks.rados.rados.0.smithi115.stdout:3930: finishing write tid 3 to smithi11541303-43 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 43 v 2206 (ObjNum 1176 snap 385 seq_num 1176) dirty exists 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3930: left oid 43 (ObjNum 1176 snap 385 seq_num 1176) 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3932: finishing write tid 1 to smithi11541303-25 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3932: finishing write tid 2 to smithi11541303-25 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3932: finishing write tid 3 to smithi11541303-25 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 25 v 2208 (ObjNum 1178 snap 385 seq_num 1178) dirty exists 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3932: left oid 25 (ObjNum 1178 snap 385 seq_num 1178) 2022-04-22T17:39:11.100 INFO:tasks.rados.rados.0.smithi115.stdout:3933: expect (ObjNum 1131 snap 361 seq_num 1131) 2022-04-22T17:39:11.363 INFO:tasks.rados.rados.0.smithi115.stdout:3934: finishing write tid 1 to smithi11541303-8 2022-04-22T17:39:11.363 INFO:tasks.rados.rados.0.smithi115.stdout:3934: finishing write tid 2 to smithi11541303-8 2022-04-22T17:39:11.363 INFO:tasks.rados.rados.0.smithi115.stdout:3934: finishing write tid 3 to smithi11541303-8 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 8 v 2087 (ObjNum 1179 snap 385 seq_num 1179) dirty exists 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3934: left oid 8 (ObjNum 1179 snap 385 seq_num 1179) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3925: done (10 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3926: done (9 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3927: done (8 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3928: done (7 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3929: done (6 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3930: done (5 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3931: done (4 left) 2022-04-22T17:39:11.364 INFO:tasks.rados.rados.0.smithi115.stdout:3932: done (3 left) 2022-04-22T17:39:11.365 INFO:tasks.rados.rados.0.smithi115.stdout:3933: done (2 left) 2022-04-22T17:39:11.365 INFO:tasks.rados.rados.0.smithi115.stdout:3934: done (1 left) 2022-04-22T17:39:11.365 INFO:tasks.rados.rados.0.smithi115.stdout:3935: done (0 left) 2022-04-22T17:39:11.365 INFO:tasks.rados.rados.0.smithi115.stdout:3936: copy_from oid 18 from oid 4 current snap is 385 2022-04-22T17:39:11.365 INFO:tasks.rados.rados.0.smithi115.stdout:3937: setattr oid 49 current snap is 385 2022-04-22T17:39:11.368 INFO:tasks.rados.rados.0.smithi115.stdout:3938: rollback oid 50 current snap is 385 2022-04-22T17:39:11.368 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 50 to 383 2022-04-22T17:39:11.368 INFO:tasks.rados.rados.0.smithi115.stdout:3939: rollback oid 31 current snap is 385 2022-04-22T17:39:11.368 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 31 to 378 2022-04-22T17:39:11.369 INFO:tasks.rados.rados.0.smithi115.stdout:3940: read oid 21 snap -1 2022-04-22T17:39:11.369 INFO:tasks.rados.rados.0.smithi115.stdout:3940: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:39:11.369 INFO:tasks.rados.rados.0.smithi115.stdout:3941: read oid 37 snap -1 2022-04-22T17:39:11.369 INFO:tasks.rados.rados.0.smithi115.stdout:3941: expect deleted 2022-04-22T17:39:11.369 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 15 current snap is 385 2022-04-22T17:39:11.370 INFO:tasks.rados.rados.0.smithi115.stdout:3942: seq_num 1180 ranges {11575296=565248} 2022-04-22T17:39:11.444 INFO:tasks.rados.rados.0.smithi115.stdout:3942: writing smithi11541303-15 from 11575296 to 12140544 tid 1 2022-04-22T17:39:11.444 INFO:tasks.rados.rados.0.smithi115.stdout:3943: copy_from oid 44 from oid 6 current snap is 385 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3941: done (7 left) 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3944: setattr oid 9 current snap is 385 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2457 (ObjNum 1170 snap 385 seq_num 1170) dirty exists 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3938: finishing rollback tid 0 to smithi11541303-50 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3938: finishing rollback tid 1 to smithi11541303-50 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3938: finishing rollback tid 2 to smithi11541303-50 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 50 v 2110 (ObjNum 1138 snap 363 seq_num 1138) dirty exists 2022-04-22T17:39:11.445 INFO:tasks.rados.rados.0.smithi115.stdout:3940: expect (ObjNum 1110 snap 353 seq_num 1110) 2022-04-22T17:39:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:11 smithi121 bash[36902]: cluster 2022-04-22T17:39:10.011183+0000 mgr.y (mgr.34107) 503 : cluster [DBG] pgmap v955: 121 pgs: 121 active+clean; 264 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:39:11.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:11 smithi115 bash[45495]: cluster 2022-04-22T17:39:10.011183+0000 mgr.y (mgr.34107) 503 : cluster [DBG] pgmap v955: 121 pgs: 121 active+clean; 264 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:39:11.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:11 smithi115 bash[46934]: cluster 2022-04-22T17:39:10.011183+0000 mgr.y (mgr.34107) 503 : cluster [DBG] pgmap v955: 121 pgs: 121 active+clean; 264 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 10 op/s 2022-04-22T17:39:11.635 INFO:tasks.rados.rados.0.smithi115.stdout:3939: finishing rollback tid 0 to smithi11541303-31 2022-04-22T17:39:11.635 INFO:tasks.rados.rados.0.smithi115.stdout:3939: finishing rollback tid 1 to smithi11541303-31 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:3939: finishing rollback tid 2 to smithi11541303-31 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 31 v 2461 (ObjNum 1080 snap 342 seq_num 1080) dirty exists 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:3936: finishing copy_from to smithi11541303-18 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 18 v 2092 (ObjNum 1076 snap 339 seq_num 1076) dirty exists 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:3936: finishing copy_from racing read to smithi11541303-18 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:3942: finishing write tid 1 to smithi11541303-15 2022-04-22T17:39:11.636 INFO:tasks.rados.rados.0.smithi115.stdout:3942: finishing write tid 2 to smithi11541303-15 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:3942: finishing write tid 3 to smithi11541303-15 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 15 v 2063 (ObjNum 1180 snap 385 seq_num 1180) dirty exists 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:3942: left oid 15 (ObjNum 1180 snap 385 seq_num 1180) 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:3943: finishing copy_from to smithi11541303-44 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 44 v 1951 (ObjNum 1125 snap 359 seq_num 1125) dirty exists 2022-04-22T17:39:11.637 INFO:tasks.rados.rados.0.smithi115.stdout:3943: finishing copy_from racing read to smithi11541303-44 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3936: done (7 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3937: done (6 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3938: done (5 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3939: done (4 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3940: done (3 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3942: done (2 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3943: done (1 left) 2022-04-22T17:39:11.638 INFO:tasks.rados.rados.0.smithi115.stdout:3945: delete oid 3 current snap is 385 2022-04-22T17:39:11.639 INFO:tasks.rados.rados.0.smithi115.stdout:3945: done (1 left) 2022-04-22T17:39:11.640 INFO:tasks.rados.rados.0.smithi115.stdout:3946: delete oid 16 current snap is 385 2022-04-22T17:39:11.642 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2465 (ObjNum 989 snap 314 seq_num 989) dirty exists 2022-04-22T17:39:11.646 INFO:tasks.rados.rados.0.smithi115.stdout:3944: done (1 left) 2022-04-22T17:39:11.646 INFO:tasks.rados.rados.0.smithi115.stdout:3946: done (0 left) 2022-04-22T17:39:11.646 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 10 current snap is 385 2022-04-22T17:39:11.646 INFO:tasks.rados.rados.0.smithi115.stdout:3947: seq_num 1183 ranges {12443648=622592} 2022-04-22T17:39:11.721 INFO:tasks.rados.rados.0.smithi115.stdout:3947: writing smithi11541303-10 from 12443648 to 13066240 tid 1 2022-04-22T17:39:11.721 INFO:tasks.rados.rados.0.smithi115.stdout:3948: snap_create 2022-04-22T17:39:11.729 INFO:tasks.rados.rados.0.smithi115.stdout:3947: finishing write tid 1 to smithi11541303-10 2022-04-22T17:39:11.730 INFO:tasks.rados.rados.0.smithi115.stdout:3947: finishing write tid 2 to smithi11541303-10 2022-04-22T17:39:11.733 INFO:tasks.rados.rados.0.smithi115.stdout:3947: finishing write tid 3 to smithi11541303-10 2022-04-22T17:39:11.733 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 10 v 2079 (ObjNum 1183 snap 385 seq_num 1183) dirty exists 2022-04-22T17:39:11.733 INFO:tasks.rados.rados.0.smithi115.stdout:3947: left oid 10 (ObjNum 1183 snap 385 seq_num 1183) 2022-04-22T17:39:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:11.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:12.092 INFO:tasks.rados.rados.0.smithi115.stdout:3947: done (1 left) 2022-04-22T17:39:12.093 INFO:tasks.rados.rados.0.smithi115.stdout:3948: done (0 left) 2022-04-22T17:39:12.093 INFO:tasks.rados.rados.0.smithi115.stdout:3949: read oid 1 snap 378 2022-04-22T17:39:12.093 INFO:tasks.rados.rados.0.smithi115.stdout:3949: expect (ObjNum 1123 snap 359 seq_num 1123) 2022-04-22T17:39:12.093 INFO:tasks.rados.rados.0.smithi115.stdout:3950: read oid 44 snap -1 2022-04-22T17:39:12.094 INFO:tasks.rados.rados.0.smithi115.stdout:3950: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:39:12.094 INFO:tasks.rados.rados.0.smithi115.stdout:3951: delete oid 40 current snap is 386 2022-04-22T17:39:12.109 INFO:tasks.rados.rados.0.smithi115.stdout:3951: done (2 left) 2022-04-22T17:39:12.109 INFO:tasks.rados.rados.0.smithi115.stdout:3952: read oid 3 snap -1 2022-04-22T17:39:12.109 INFO:tasks.rados.rados.0.smithi115.stdout:3952: expect deleted 2022-04-22T17:39:12.110 INFO:tasks.rados.rados.0.smithi115.stdout:3953: snap_remove snap 385 2022-04-22T17:39:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:12 smithi121 bash[36902]: cluster 2022-04-22T17:39:11.072659+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2022-04-22T17:39:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:12 smithi115 bash[45495]: cluster 2022-04-22T17:39:11.072659+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2022-04-22T17:39:12.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:12 smithi115 bash[46934]: cluster 2022-04-22T17:39:11.072659+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2022-04-22T17:39:13.110 INFO:tasks.rados.rados.0.smithi115.stdout:3953: done (3 left) 2022-04-22T17:39:13.110 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 38 current snap is 386 2022-04-22T17:39:13.110 INFO:tasks.rados.rados.0.smithi115.stdout:3949: expect (ObjNum 1123 snap 359 seq_num 1123) 2022-04-22T17:39:13.239 INFO:tasks.rados.rados.0.smithi115.stdout:3950: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:39:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:13 smithi121 bash[36902]: cluster 2022-04-22T17:39:12.012138+0000 mgr.y (mgr.34107) 504 : cluster [DBG] pgmap v958: 121 pgs: 121 active+clean; 289 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 30 op/s 2022-04-22T17:39:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:13 smithi121 bash[36902]: cluster 2022-04-22T17:39:12.078668+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2022-04-22T17:39:13.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:13 smithi115 bash[45495]: cluster 2022-04-22T17:39:12.012138+0000 mgr.y (mgr.34107) 504 : cluster [DBG] pgmap v958: 121 pgs: 121 active+clean; 289 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 30 op/s 2022-04-22T17:39:13.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:13 smithi115 bash[45495]: cluster 2022-04-22T17:39:12.078668+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2022-04-22T17:39:13.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:13 smithi115 bash[46934]: cluster 2022-04-22T17:39:12.012138+0000 mgr.y (mgr.34107) 504 : cluster [DBG] pgmap v958: 121 pgs: 121 active+clean; 289 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 30 op/s 2022-04-22T17:39:13.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:13 smithi115 bash[46934]: cluster 2022-04-22T17:39:12.078668+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2022-04-22T17:39:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:13.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:14.308 INFO:tasks.rados.rados.0.smithi115.stdout:3954: seq_num 1185 ranges {4464640=655360} 2022-04-22T17:39:14.344 INFO:tasks.rados.rados.0.smithi115.stdout:3954: writing smithi11541303-38 from 4464640 to 5120000 tid 1 2022-04-22T17:39:14.344 INFO:tasks.rados.rados.0.smithi115.stdout:3949: done (3 left) 2022-04-22T17:39:14.344 INFO:tasks.rados.rados.0.smithi115.stdout:3950: done (2 left) 2022-04-22T17:39:14.345 INFO:tasks.rados.rados.0.smithi115.stdout:3952: done (1 left) 2022-04-22T17:39:14.345 INFO:tasks.rados.rados.0.smithi115.stdout:3955: copy_from oid 9 from oid 15 current snap is 386 2022-04-22T17:39:14.345 INFO:tasks.rados.rados.0.smithi115.stdout:3956: read oid 48 snap -1 2022-04-22T17:39:14.345 INFO:tasks.rados.rados.0.smithi115.stdout:3956: expect (ObjNum 1150 snap 373 seq_num 1150) 2022-04-22T17:39:14.345 INFO:tasks.rados.rados.0.smithi115.stdout:3957: snap_create 2022-04-22T17:39:14.354 INFO:tasks.rados.rados.0.smithi115.stdout:3954: finishing write tid 1 to smithi11541303-38 2022-04-22T17:39:14.354 INFO:tasks.rados.rados.0.smithi115.stdout:3954: finishing write tid 2 to smithi11541303-38 2022-04-22T17:39:14.356 INFO:tasks.rados.rados.0.smithi115.stdout:3954: finishing write tid 3 to smithi11541303-38 2022-04-22T17:39:14.357 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2258 (ObjNum 1185 snap 386 seq_num 1185) dirty exists 2022-04-22T17:39:14.357 INFO:tasks.rados.rados.0.smithi115.stdout:3954: left oid 38 (ObjNum 1185 snap 386 seq_num 1185) 2022-04-22T17:39:14.370 INFO:tasks.rados.rados.0.smithi115.stdout:3956: expect (ObjNum 1150 snap 373 seq_num 1150) 2022-04-22T17:39:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:14 smithi121 bash[36902]: cluster 2022-04-22T17:39:13.092561+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2022-04-22T17:39:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:14 smithi121 bash[36902]: audit 2022-04-22T17:39:13.343645+0000 mon.a (mon.0) 1559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:39:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:14 smithi121 bash[36902]: audit 2022-04-22T17:39:13.347193+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:39:14.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[45495]: cluster 2022-04-22T17:39:13.092561+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2022-04-22T17:39:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[45495]: audit 2022-04-22T17:39:13.343645+0000 mon.a (mon.0) 1559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:39:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[45495]: audit 2022-04-22T17:39:13.347193+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:39:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[46934]: cluster 2022-04-22T17:39:13.092561+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2022-04-22T17:39:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[46934]: audit 2022-04-22T17:39:13.343645+0000 mon.a (mon.0) 1559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:39:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:14 smithi115 bash[46934]: audit 2022-04-22T17:39:13.347193+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:39:14.659 INFO:tasks.rados.rados.0.smithi115.stdout:3955: finishing copy_from to smithi11541303-9 2022-04-22T17:39:14.660 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 9 v 2063 (ObjNum 1180 snap 385 seq_num 1180) dirty exists 2022-04-22T17:39:14.660 INFO:tasks.rados.rados.0.smithi115.stdout:3955: finishing copy_from racing read to smithi11541303-9 2022-04-22T17:39:15.051 INFO:tasks.rados.rados.0.smithi115.stdout:3954: done (3 left) 2022-04-22T17:39:15.051 INFO:tasks.rados.rados.0.smithi115.stdout:3955: done (2 left) 2022-04-22T17:39:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3956: done (1 left) 2022-04-22T17:39:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3957: done (0 left) 2022-04-22T17:39:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3958: read oid 24 snap -1 2022-04-22T17:39:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3958: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:39:15.052 INFO:tasks.rados.rados.0.smithi115.stdout:3959: delete oid 46 current snap is 387 2022-04-22T17:39:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3959: done (1 left) 2022-04-22T17:39:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3960: rmattr oid 22 current snap is 387 2022-04-22T17:39:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3960: done (1 left) 2022-04-22T17:39:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3961: rmattr oid 28 current snap is 387 2022-04-22T17:39:15.056 INFO:tasks.rados.rados.0.smithi115.stdout:3962: delete oid 33 current snap is 387 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:3962: done (2 left) 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:3963: rollback oid 11 current snap is 387 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 11 to 386 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:3964: copy_from oid 27 from oid 2 current snap is 387 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 21 current snap is 387 2022-04-22T17:39:15.067 INFO:tasks.rados.rados.0.smithi115.stdout:3965: seq_num 1188 ranges {4038656=704512} 2022-04-22T17:39:15.095 INFO:tasks.rados.rados.0.smithi115.stdout:3965: writing smithi11541303-21 from 4038656 to 4743168 tid 1 2022-04-22T17:39:15.137 INFO:tasks.rados.rados.0.smithi115.stdout:3966: snap_create 2022-04-22T17:39:15.137 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 28 v 2074 (ObjNum 1126 snap 360 seq_num 1126) dirty exists 2022-04-22T17:39:15.137 INFO:tasks.rados.rados.0.smithi115.stdout:3963: finishing rollback tid 0 to smithi11541303-11 2022-04-22T17:39:15.137 INFO:tasks.rados.rados.0.smithi115.stdout:3963: finishing rollback tid 1 to smithi11541303-11 2022-04-22T17:39:15.138 INFO:tasks.rados.rados.0.smithi115.stdout:3963: finishing rollback tid 2 to smithi11541303-11 2022-04-22T17:39:15.138 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 11 v 2248 (ObjNum 1139 snap 364 seq_num 1139) dirty exists 2022-04-22T17:39:15.139 INFO:tasks.rados.rados.0.smithi115.stdout:3964: finishing copy_from to smithi11541303-27 2022-04-22T17:39:15.139 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 27 v 2049 (ObjNum 1107 snap 352 seq_num 1107) dirty exists 2022-04-22T17:39:15.139 INFO:tasks.rados.rados.0.smithi115.stdout:3964: finishing copy_from racing read to smithi11541303-27 2022-04-22T17:39:15.140 INFO:tasks.rados.rados.0.smithi115.stdout:3958: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:39:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:15 smithi121 bash[36902]: cluster 2022-04-22T17:39:14.013019+0000 mgr.y (mgr.34107) 505 : cluster [DBG] pgmap v961: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2022-04-22T17:39:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:15 smithi121 bash[36902]: audit 2022-04-22T17:39:14.490481+0000 mon.a (mon.0) 1561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:15 smithi121 bash[36902]: cluster 2022-04-22T17:39:15.038418+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2022-04-22T17:39:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[45495]: cluster 2022-04-22T17:39:14.013019+0000 mgr.y (mgr.34107) 505 : cluster [DBG] pgmap v961: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2022-04-22T17:39:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[45495]: audit 2022-04-22T17:39:14.490481+0000 mon.a (mon.0) 1561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[45495]: cluster 2022-04-22T17:39:15.038418+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2022-04-22T17:39:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[46934]: cluster 2022-04-22T17:39:14.013019+0000 mgr.y (mgr.34107) 505 : cluster [DBG] pgmap v961: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2022-04-22T17:39:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[46934]: audit 2022-04-22T17:39:14.490481+0000 mon.a (mon.0) 1561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:15 smithi115 bash[46934]: cluster 2022-04-22T17:39:15.038418+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2022-04-22T17:39:15.800 INFO:tasks.rados.rados.0.smithi115.stdout:3965: finishing write tid 1 to smithi11541303-21 2022-04-22T17:39:15.800 INFO:tasks.rados.rados.0.smithi115.stdout:3965: finishing write tid 2 to smithi11541303-21 2022-04-22T17:39:15.800 INFO:tasks.rados.rados.0.smithi115.stdout:3965: finishing write tid 3 to smithi11541303-21 2022-04-22T17:39:15.800 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1900 (ObjNum 1188 snap 387 seq_num 1188) dirty exists 2022-04-22T17:39:15.801 INFO:tasks.rados.rados.0.smithi115.stdout:3965: left oid 21 (ObjNum 1188 snap 387 seq_num 1188) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3958: done (5 left) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3961: done (4 left) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3963: done (3 left) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3964: done (2 left) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3965: done (1 left) 2022-04-22T17:39:16.062 INFO:tasks.rados.rados.0.smithi115.stdout:3966: done (0 left) 2022-04-22T17:39:16.063 INFO:tasks.rados.rados.0.smithi115.stdout:3967: snap_remove snap 387 2022-04-22T17:39:17.064 INFO:tasks.rados.rados.0.smithi115.stdout:3967: done (0 left) 2022-04-22T17:39:17.065 INFO:tasks.rados.rados.0.smithi115.stdout:3968: delete oid 2 current snap is 388 2022-04-22T17:39:17.076 INFO:tasks.rados.rados.0.smithi115.stdout:3968: done (0 left) 2022-04-22T17:39:17.077 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 33 current snap is 388 2022-04-22T17:39:17.077 INFO:tasks.rados.rados.0.smithi115.stdout:3969: seq_num 1190 ranges {0=737280} 2022-04-22T17:39:17.081 INFO:tasks.rados.rados.0.smithi115.stdout:3969: writing smithi11541303-33 from 0 to 737280 tid 1 2022-04-22T17:39:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:3970: read oid 39 snap -1 2022-04-22T17:39:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:3970: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:39:17.082 INFO:tasks.rados.rados.0.smithi115.stdout:3971: delete oid 7 current snap is 388 2022-04-22T17:39:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:3969: finishing write tid 1 to smithi11541303-33 2022-04-22T17:39:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:3969: finishing write tid 2 to smithi11541303-33 2022-04-22T17:39:17.088 INFO:tasks.rados.rados.0.smithi115.stdout:3970: expect (ObjNum 909 snap 299 seq_num 909) 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3969: finishing write tid 3 to smithi11541303-33 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 33 v 2085 (ObjNum 1190 snap 388 seq_num 1190) dirty exists 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3969: left oid 33 (ObjNum 1190 snap 388 seq_num 1190) 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3969: done (2 left) 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3970: done (1 left) 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3971: done (0 left) 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3972: read oid 15 snap -1 2022-04-22T17:39:17.120 INFO:tasks.rados.rados.0.smithi115.stdout:3972: expect (ObjNum 1180 snap 385 seq_num 1180) 2022-04-22T17:39:17.121 INFO:tasks.rados.rados.0.smithi115.stdout:3973: read oid 6 snap -1 2022-04-22T17:39:17.121 INFO:tasks.rados.rados.0.smithi115.stdout:3973: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:39:17.121 INFO:tasks.rados.rados.0.smithi115.stdout:3974: snap_create 2022-04-22T17:39:17.167 INFO:tasks.rados.rados.0.smithi115.stdout:3972: expect (ObjNum 1180 snap 385 seq_num 1180) 2022-04-22T17:39:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:17 smithi121 bash[36902]: cluster 2022-04-22T17:39:16.014071+0000 mgr.y (mgr.34107) 506 : cluster [DBG] pgmap v963: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 5.9 MiB/s wr, 26 op/s 2022-04-22T17:39:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:17 smithi121 bash[36902]: cluster 2022-04-22T17:39:16.048994+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2022-04-22T17:39:17.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:17 smithi115 bash[45495]: cluster 2022-04-22T17:39:16.014071+0000 mgr.y (mgr.34107) 506 : cluster [DBG] pgmap v963: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 5.9 MiB/s wr, 26 op/s 2022-04-22T17:39:17.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:17 smithi115 bash[45495]: cluster 2022-04-22T17:39:16.048994+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2022-04-22T17:39:17.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:17 smithi115 bash[46934]: cluster 2022-04-22T17:39:16.014071+0000 mgr.y (mgr.34107) 506 : cluster [DBG] pgmap v963: 121 pgs: 121 active+clean; 282 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 5.9 MiB/s wr, 26 op/s 2022-04-22T17:39:17.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:17 smithi115 bash[46934]: cluster 2022-04-22T17:39:16.048994+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2022-04-22T17:39:18.417 INFO:tasks.rados.rados.0.smithi115.stdout:3973: expect (ObjNum 1125 snap 359 seq_num 1125) 2022-04-22T17:39:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:18 smithi121 bash[36902]: cluster 2022-04-22T17:39:17.050549+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2022-04-22T17:39:18.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:18 smithi115 bash[45495]: cluster 2022-04-22T17:39:17.050549+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2022-04-22T17:39:18.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:18 smithi115 bash[46934]: cluster 2022-04-22T17:39:17.050549+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2022-04-22T17:39:19.465 INFO:tasks.rados.rados.0.smithi115.stdout:3972: done (2 left) 2022-04-22T17:39:19.465 INFO:tasks.rados.rados.0.smithi115.stdout:3973: done (1 left) 2022-04-22T17:39:19.465 INFO:tasks.rados.rados.0.smithi115.stdout:3974: done (0 left) 2022-04-22T17:39:19.465 INFO:tasks.rados.rados.0.smithi115.stdout:3975: snap_remove snap 383 2022-04-22T17:39:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:19 smithi121 bash[36902]: cluster 2022-04-22T17:39:18.014877+0000 mgr.y (mgr.34107) 507 : cluster [DBG] pgmap v966: 121 pgs: 121 active+clean; 294 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:39:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:19 smithi121 bash[36902]: cluster 2022-04-22T17:39:18.068122+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2022-04-22T17:39:19.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:19 smithi115 bash[45495]: cluster 2022-04-22T17:39:18.014877+0000 mgr.y (mgr.34107) 507 : cluster [DBG] pgmap v966: 121 pgs: 121 active+clean; 294 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:39:19.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:19 smithi115 bash[45495]: cluster 2022-04-22T17:39:18.068122+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2022-04-22T17:39:19.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:19 smithi115 bash[46934]: cluster 2022-04-22T17:39:18.014877+0000 mgr.y (mgr.34107) 507 : cluster [DBG] pgmap v966: 121 pgs: 121 active+clean; 294 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2022-04-22T17:39:19.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:19 smithi115 bash[46934]: cluster 2022-04-22T17:39:18.068122+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2022-04-22T17:39:20.063 INFO:tasks.rados.rados.0.smithi115.stdout:3975: done (0 left) 2022-04-22T17:39:20.063 INFO:tasks.rados.rados.0.smithi115.stdout:3976: snap_remove snap 376 2022-04-22T17:39:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:20 smithi121 bash[36902]: audit 2022-04-22T17:39:19.493524+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:20 smithi121 bash[36902]: cluster 2022-04-22T17:39:20.041339+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2022-04-22T17:39:20.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:20 smithi115 bash[45495]: audit 2022-04-22T17:39:19.493524+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:20.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:20 smithi115 bash[45495]: cluster 2022-04-22T17:39:20.041339+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2022-04-22T17:39:20.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:20 smithi115 bash[46934]: audit 2022-04-22T17:39:19.493524+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:20.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:20 smithi115 bash[46934]: cluster 2022-04-22T17:39:20.041339+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2022-04-22T17:39:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:3976: done (0 left) 2022-04-22T17:39:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:3977: read oid 24 snap 386 2022-04-22T17:39:21.054 INFO:tasks.rados.rados.0.smithi115.stdout:3977: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:39:21.055 INFO:tasks.rados.rados.0.smithi115.stdout:3978: read oid 28 snap -1 2022-04-22T17:39:21.055 INFO:tasks.rados.rados.0.smithi115.stdout:3978: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:39:21.055 INFO:tasks.rados.rados.0.smithi115.stdout:3979: copy_from oid 20 from oid 15 current snap is 389 2022-04-22T17:39:21.055 INFO:tasks.rados.rados.0.smithi115.stdout:3980: delete oid 26 current snap is 389 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3980: done (3 left) 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3981: rmattr oid 16 current snap is 389 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3981: done (3 left) 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3982: read oid 32 snap -1 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3982: expect deleted 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3983: rmattr oid 21 current snap is 389 2022-04-22T17:39:21.081 INFO:tasks.rados.rados.0.smithi115.stdout:3984: snap_remove snap 388 2022-04-22T17:39:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:21 smithi121 bash[36902]: cluster 2022-04-22T17:39:20.015945+0000 mgr.y (mgr.34107) 508 : cluster [DBG] pgmap v968: 121 pgs: 121 active+clean; 296 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 12 op/s 2022-04-22T17:39:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:21 smithi121 bash[36902]: cluster 2022-04-22T17:39:21.041162+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2022-04-22T17:39:21.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:21 smithi115 bash[45495]: cluster 2022-04-22T17:39:20.015945+0000 mgr.y (mgr.34107) 508 : cluster [DBG] pgmap v968: 121 pgs: 121 active+clean; 296 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 12 op/s 2022-04-22T17:39:21.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:21 smithi115 bash[45495]: cluster 2022-04-22T17:39:21.041162+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2022-04-22T17:39:21.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:21 smithi115 bash[46934]: cluster 2022-04-22T17:39:20.015945+0000 mgr.y (mgr.34107) 508 : cluster [DBG] pgmap v968: 121 pgs: 121 active+clean; 296 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 12 op/s 2022-04-22T17:39:21.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:21 smithi115 bash[46934]: cluster 2022-04-22T17:39:21.041162+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2022-04-22T17:39:22.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:21.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:22.059 INFO:tasks.rados.rados.0.smithi115.stdout:3984: done (5 left) 2022-04-22T17:39:22.059 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 30 current snap is 389 2022-04-22T17:39:22.059 INFO:tasks.rados.rados.0.smithi115.stdout:3985: seq_num 1193 ranges {11296768=786432} 2022-04-22T17:39:22.126 INFO:tasks.rados.rados.0.smithi115.stdout:3985: writing smithi11541303-30 from 11296768 to 12083200 tid 1 2022-04-22T17:39:22.127 INFO:tasks.rados.rados.0.smithi115.stdout:3986: rollback oid 35 current snap is 389 2022-04-22T17:39:22.127 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 35 to 386 2022-04-22T17:39:22.127 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 21 v 1901 (ObjNum 1188 snap 387 seq_num 1188) dirty exists 2022-04-22T17:39:22.127 INFO:tasks.rados.rados.0.smithi115.stdout:3978: expect (ObjNum 1126 snap 360 seq_num 1126) 2022-04-22T17:39:23.186 INFO:tasks.rados.rados.0.smithi115.stdout:3977: expect (ObjNum 939 snap 304 seq_num 939) 2022-04-22T17:39:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:23 smithi121 bash[36902]: cluster 2022-04-22T17:39:22.017010+0000 mgr.y (mgr.34107) 509 : cluster [DBG] pgmap v971: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2022-04-22T17:39:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:23 smithi121 bash[36902]: cluster 2022-04-22T17:39:22.044268+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2022-04-22T17:39:23.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:23 smithi115 bash[46934]: cluster 2022-04-22T17:39:22.017010+0000 mgr.y (mgr.34107) 509 : cluster [DBG] pgmap v971: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2022-04-22T17:39:23.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:23 smithi115 bash[46934]: cluster 2022-04-22T17:39:22.044268+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2022-04-22T17:39:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:23 smithi115 bash[45495]: cluster 2022-04-22T17:39:22.017010+0000 mgr.y (mgr.34107) 509 : cluster [DBG] pgmap v971: 121 pgs: 121 active+clean; 300 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2022-04-22T17:39:23.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:23 smithi115 bash[45495]: cluster 2022-04-22T17:39:22.044268+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2022-04-22T17:39:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:23.525Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:23.851 INFO:tasks.rados.rados.0.smithi115.stdout:3979: finishing copy_from to smithi11541303-20 2022-04-22T17:39:23.851 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 20 v 2063 (ObjNum 1180 snap 385 seq_num 1180) dirty exists 2022-04-22T17:39:23.851 INFO:tasks.rados.rados.0.smithi115.stdout:3979: finishing copy_from racing read to smithi11541303-20 2022-04-22T17:39:23.851 INFO:tasks.rados.rados.0.smithi115.stdout:3986: finishing rollback tid 0 to smithi11541303-35 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3985: finishing write tid 1 to smithi11541303-30 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3986: finishing rollback tid 1 to smithi11541303-35 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3986: finishing rollback tid 2 to smithi11541303-35 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 35 v 2077 (ObjNum 1144 snap 371 seq_num 1144) dirty exists 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3985: finishing write tid 2 to smithi11541303-30 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3985: finishing write tid 3 to smithi11541303-30 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 30 v 2096 (ObjNum 1193 snap 389 seq_num 1193) dirty exists 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3985: left oid 30 (ObjNum 1193 snap 389 seq_num 1193) 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3977: done (6 left) 2022-04-22T17:39:23.852 INFO:tasks.rados.rados.0.smithi115.stdout:3978: done (5 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3979: done (4 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3982: done (3 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3983: done (2 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3985: done (1 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3986: done (0 left) 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 48 current snap is 389 2022-04-22T17:39:23.853 INFO:tasks.rados.rados.0.smithi115.stdout:3987: seq_num 1194 ranges {4685824=802816} 2022-04-22T17:39:23.887 INFO:tasks.rados.rados.0.smithi115.stdout:3987: writing smithi11541303-48 from 4685824 to 5488640 tid 1 2022-04-22T17:39:23.887 INFO:tasks.rados.rados.0.smithi115.stdout:3988: rollback oid 49 current snap is 389 2022-04-22T17:39:23.887 INFO:tasks.rados.rados.0.smithi115.stdout:rollback oid 49 to 378 2022-04-22T17:39:23.887 INFO:tasks.rados.rados.0.smithi115.stdout:3989: snap_remove snap 386 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3989: done (2 left) 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3990: read oid 18 snap 381 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3990: expect (ObjNum 1161 snap 381 seq_num 1161) 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3991: snap_create 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3987: finishing write tid 1 to smithi11541303-48 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3988: finishing rollback tid 0 to smithi11541303-49 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3987: finishing write tid 2 to smithi11541303-48 2022-04-22T17:39:24.065 INFO:tasks.rados.rados.0.smithi115.stdout:3988: finishing rollback tid 1 to smithi11541303-49 2022-04-22T17:39:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:3988: finishing rollback tid 2 to smithi11541303-49 2022-04-22T17:39:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 49 v 2486 (ObjNum 1141 snap 364 seq_num 1141) dirty exists 2022-04-22T17:39:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:3987: finishing write tid 3 to smithi11541303-48 2022-04-22T17:39:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 48 v 2098 (ObjNum 1194 snap 389 seq_num 1194) dirty exists 2022-04-22T17:39:24.066 INFO:tasks.rados.rados.0.smithi115.stdout:3987: left oid 48 (ObjNum 1194 snap 389 seq_num 1194) 2022-04-22T17:39:24.112 INFO:tasks.rados.rados.0.smithi115.stdout:3990: expect (ObjNum 1161 snap 381 seq_num 1161) 2022-04-22T17:39:25.054 INFO:tasks.rados.rados.0.smithi115.stdout:3987: done (3 left) 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3988: done (2 left) 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3990: done (1 left) 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3991: done (0 left) 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3992: read oid 13 snap -1 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3992: expect deleted 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 46 current snap is 390 2022-04-22T17:39:25.055 INFO:tasks.rados.rados.0.smithi115.stdout:3993: seq_num 1195 ranges {0=417792,417792=401408} 2022-04-22T17:39:25.058 INFO:tasks.rados.rados.0.smithi115.stdout:3993: writing smithi11541303-46 from 0 to 417792 tid 1 2022-04-22T17:39:25.060 INFO:tasks.rados.rados.0.smithi115.stdout:3993: writing smithi11541303-46 from 417792 to 819200 tid 2 2022-04-22T17:39:25.060 INFO:tasks.rados.rados.0.smithi115.stdout:3994: snap_create 2022-04-22T17:39:25.066 INFO:tasks.rados.rados.0.smithi115.stdout:3993: finishing write tid 1 to smithi11541303-46 2022-04-22T17:39:25.066 INFO:tasks.rados.rados.0.smithi115.stdout:3993: finishing write tid 2 to smithi11541303-46 2022-04-22T17:39:25.067 INFO:tasks.rados.rados.0.smithi115.stdout:3993: finishing write tid 3 to smithi11541303-46 2022-04-22T17:39:25.069 INFO:tasks.rados.rados.0.smithi115.stdout:3993: finishing write tid 4 to smithi11541303-46 2022-04-22T17:39:25.069 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 46 v 2251 (ObjNum 1195 snap 390 seq_num 1195) dirty exists 2022-04-22T17:39:25.069 INFO:tasks.rados.rados.0.smithi115.stdout:3993: left oid 46 (ObjNum 1195 snap 390 seq_num 1195) 2022-04-22T17:39:25.399 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[45495]: cluster 2022-04-22T17:39:24.018134+0000 mgr.y (mgr.34107) 510 : cluster [DBG] pgmap v973: 121 pgs: 121 active+clean; 306 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2022-04-22T17:39:25.399 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[45495]: cluster 2022-04-22T17:39:24.051835+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2022-04-22T17:39:25.399 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[45495]: audit 2022-04-22T17:39:24.496286+0000 mon.a (mon.0) 1571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[45495]: cluster 2022-04-22T17:39:25.041149+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[45495]: audit 2022-04-22T17:39:25.047616+0000 mon.a (mon.0) 1573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[46934]: cluster 2022-04-22T17:39:24.018134+0000 mgr.y (mgr.34107) 510 : cluster [DBG] pgmap v973: 121 pgs: 121 active+clean; 306 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[46934]: cluster 2022-04-22T17:39:24.051835+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[46934]: audit 2022-04-22T17:39:24.496286+0000 mon.a (mon.0) 1571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[46934]: cluster 2022-04-22T17:39:25.041149+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2022-04-22T17:39:25.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:25 smithi115 bash[46934]: audit 2022-04-22T17:39:25.047616+0000 mon.a (mon.0) 1573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:39:25.413 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:25 smithi121 bash[36902]: cluster 2022-04-22T17:39:24.018134+0000 mgr.y (mgr.34107) 510 : cluster [DBG] pgmap v973: 121 pgs: 121 active+clean; 306 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 3.0 MiB/s wr, 7 op/s 2022-04-22T17:39:25.413 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:25 smithi121 bash[36902]: cluster 2022-04-22T17:39:24.051835+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2022-04-22T17:39:25.413 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:25 smithi121 bash[36902]: audit 2022-04-22T17:39:24.496286+0000 mon.a (mon.0) 1571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:25.413 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:25 smithi121 bash[36902]: cluster 2022-04-22T17:39:25.041149+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2022-04-22T17:39:25.413 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:25 smithi121 bash[36902]: audit 2022-04-22T17:39:25.047616+0000 mon.a (mon.0) 1573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3992: done (2 left) 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3993: done (1 left) 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3994: done (0 left) 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3995: read oid 2 snap -1 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3995: expect deleted 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3996: rmattr oid 12 current snap is 391 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:3996: done (1 left) 2022-04-22T17:39:26.068 INFO:tasks.rados.rados.0.smithi115.stdout:append oid 29 current snap is 391 2022-04-22T17:39:26.069 INFO:tasks.rados.rados.0.smithi115.stdout:3997: seq_num 1196 ranges {2236416=434176,2670592=401408} 2022-04-22T17:39:26.084 INFO:tasks.rados.rados.0.smithi115.stdout:3997: writing smithi11541303-29 from 2236416 to 2670592 tid 1 2022-04-22T17:39:26.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: writing smithi11541303-29 from 2670592 to 3072000 tid 2 2022-04-22T17:39:26.087 INFO:tasks.rados.rados.0.smithi115.stdout:3998: rmattr oid 38 current snap is 391 2022-04-22T17:39:26.088 INFO:tasks.rados.rados.0.smithi115.stdout:3995: done (2 left) 2022-04-22T17:39:26.088 INFO:tasks.rados.rados.0.smithi115.stdout:3999: snap_remove snap 390 2022-04-22T17:39:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:26 smithi121 bash[36902]: audit 2022-04-22T17:39:25.049705+0000 mon.a (mon.0) 1574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:39:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:26 smithi121 bash[36902]: audit 2022-04-22T17:39:25.050767+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:39:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:26 smithi121 bash[36902]: audit 2022-04-22T17:39:25.412321+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[45495]: audit 2022-04-22T17:39:25.049705+0000 mon.a (mon.0) 1574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[45495]: audit 2022-04-22T17:39:25.050767+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[45495]: audit 2022-04-22T17:39:25.412321+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[46934]: audit 2022-04-22T17:39:25.049705+0000 mon.a (mon.0) 1574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[46934]: audit 2022-04-22T17:39:25.050767+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:39:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:26 smithi115 bash[46934]: audit 2022-04-22T17:39:25.412321+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:39:27.086 INFO:tasks.rados.rados.0.smithi115.stdout:3999: done (2 left) 2022-04-22T17:39:27.086 INFO:tasks.rados.rados.0.smithi115.stdout: waiting on 2 2022-04-22T17:39:27.086 INFO:tasks.rados.rados.0.smithi115.stdout:3997: finishing write tid 1 to smithi11541303-29 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: finishing write tid 2 to smithi11541303-29 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 38 v 2259 (ObjNum 1185 snap 386 seq_num 1185) dirty exists 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: finishing write tid 3 to smithi11541303-29 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: finishing write tid 4 to smithi11541303-29 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:update_object_version oid 29 v 2077 (ObjNum 1196 snap 391 seq_num 1196) dirty exists 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: left oid 29 (ObjNum 1196 snap 391 seq_num 1196) 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3997: done (1 left) 2022-04-22T17:39:27.087 INFO:tasks.rados.rados.0.smithi115.stdout:3998: done (0 left) 2022-04-22T17:39:27.098 INFO:tasks.rados.rados.0.smithi115.stderr:0 errors. 2022-04-22T17:39:27.098 INFO:tasks.rados.rados.0.smithi115.stderr: 2022-04-22T17:39:27.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd dump --format=json 2022-04-22T17:39:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:27 smithi121 bash[36902]: cluster 2022-04-22T17:39:26.019192+0000 mgr.y (mgr.34107) 511 : cluster [DBG] pgmap v976: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 7 op/s 2022-04-22T17:39:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:27 smithi121 bash[36902]: cluster 2022-04-22T17:39:26.054004+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2022-04-22T17:39:27.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:27 smithi115 bash[45495]: cluster 2022-04-22T17:39:26.019192+0000 mgr.y (mgr.34107) 511 : cluster [DBG] pgmap v976: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 7 op/s 2022-04-22T17:39:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:27 smithi115 bash[45495]: cluster 2022-04-22T17:39:26.054004+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2022-04-22T17:39:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:27 smithi115 bash[46934]: cluster 2022-04-22T17:39:26.019192+0000 mgr.y (mgr.34107) 511 : cluster [DBG] pgmap v976: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 7 op/s 2022-04-22T17:39:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:27 smithi115 bash[46934]: cluster 2022-04-22T17:39:26.054004+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2022-04-22T17:39:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:28 smithi121 bash[36902]: cluster 2022-04-22T17:39:27.073582+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2022-04-22T17:39:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:28 smithi115 bash[45495]: cluster 2022-04-22T17:39:27.073582+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2022-04-22T17:39:28.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:28 smithi115 bash[46934]: cluster 2022-04-22T17:39:27.073582+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2022-04-22T17:39:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:29 smithi121 bash[36902]: cluster 2022-04-22T17:39:28.020059+0000 mgr.y (mgr.34107) 512 : cluster [DBG] pgmap v979: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 592 KiB/s wr, 6 op/s 2022-04-22T17:39:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:29 smithi115 bash[45495]: cluster 2022-04-22T17:39:28.020059+0000 mgr.y (mgr.34107) 512 : cluster [DBG] pgmap v979: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 592 KiB/s wr, 6 op/s 2022-04-22T17:39:29.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:29 smithi115 bash[46934]: cluster 2022-04-22T17:39:28.020059+0000 mgr.y (mgr.34107) 512 : cluster [DBG] pgmap v979: 121 pgs: 121 active+clean; 307 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 592 KiB/s wr, 6 op/s 2022-04-22T17:39:29.899 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:39:30.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:30 smithi115 bash[45495]: audit 2022-04-22T17:39:29.499273+0000 mon.a (mon.0) 1579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:30.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:30 smithi115 bash[45495]: cluster 2022-04-22T17:39:30.040977+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2022-04-22T17:39:30.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:30 smithi115 bash[46934]: audit 2022-04-22T17:39:29.499273+0000 mon.a (mon.0) 1579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:30.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:30 smithi115 bash[46934]: cluster 2022-04-22T17:39:30.040977+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2022-04-22T17:39:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:30 smithi121 bash[36902]: audit 2022-04-22T17:39:29.499273+0000 mon.a (mon.0) 1579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:30 smithi121 bash[36902]: cluster 2022-04-22T17:39:30.040977+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2022-04-22T17:39:31.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:31 smithi115 bash[46934]: cluster 2022-04-22T17:39:30.021315+0000 mgr.y (mgr.34107) 513 : cluster [DBG] pgmap v980: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 531 KiB/s wr, 5 op/s 2022-04-22T17:39:31.378 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:31 smithi115 bash[45495]: cluster 2022-04-22T17:39:30.021315+0000 mgr.y (mgr.34107) 513 : cluster [DBG] pgmap v980: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 531 KiB/s wr, 5 op/s 2022-04-22T17:39:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:31 smithi121 bash[36902]: cluster 2022-04-22T17:39:30.021315+0000 mgr.y (mgr.34107) 513 : cluster [DBG] pgmap v980: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 531 KiB/s wr, 5 op/s 2022-04-22T17:39:32.141 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:31.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:33 smithi121 bash[36902]: cluster 2022-04-22T17:39:32.022602+0000 mgr.y (mgr.34107) 514 : cluster [DBG] pgmap v982: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 269 KiB/s wr, 3 op/s 2022-04-22T17:39:33.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:33 smithi115 bash[45495]: cluster 2022-04-22T17:39:32.022602+0000 mgr.y (mgr.34107) 514 : cluster [DBG] pgmap v982: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 269 KiB/s wr, 3 op/s 2022-04-22T17:39:33.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:33 smithi115 bash[46934]: cluster 2022-04-22T17:39:32.022602+0000 mgr.y (mgr.34107) 514 : cluster [DBG] pgmap v982: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 269 KiB/s wr, 3 op/s 2022-04-22T17:39:33.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:33.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:34.091 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:39:34.091 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":856,"fsid":"b6061a0a-c25f-11ec-8c39-001a4aab830c","created":"2022-04-22T17:15:33.309888+0000","modified":"2022-04-22T17:39:30.035213+0000","last_up_change":"2022-04-22T17:34:41.007287+0000","last_in_change":"2022-04-22T17:21:23.160018+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-22T17:19:28.627096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{},"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-04-22T17:22:16.858039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-04-22T17:22:17.796936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"rbd","create_time":"2022-04-22T17:23:10.004854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"unique_pool_0","create_time":"2022-04-22T17:25:31.659908+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"855","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":780,"snap_epoch":855,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bc5fef33-43b5-4c7a-bbf5-cc37de953e22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":312,"up_from":325,"up_thru":583,"down_at":313,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":3548478629},{"type":"v1","addr":"172.21.15.115:6803","nonce":3548478629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":3548478629},{"type":"v1","addr":"172.21.15.115:6805","nonce":3548478629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":3548478629},{"type":"v1","addr":"172.21.15.115:6809","nonce":3548478629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":3548478629},{"type":"v1","addr":"172.21.15.115:6807","nonce":3548478629}]},"public_addr":"172.21.15.115:6803/3548478629","cluster_addr":"172.21.15.115:6805/3548478629","heartbeat_back_addr":"172.21.15.115:6809/3548478629","heartbeat_front_addr":"172.21.15.115:6807/3548478629","state":["exists","up"]},{"osd":1,"uuid":"e8cd3e45-8c5d-4446-9e93-5f4e4fdb0094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":352,"up_from":366,"up_thru":583,"down_at":354,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1073133477},{"type":"v1","addr":"172.21.15.115:6811","nonce":1073133477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1073133477},{"type":"v1","addr":"172.21.15.115:6813","nonce":1073133477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":1073133477},{"type":"v1","addr":"172.21.15.115:6817","nonce":1073133477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1073133477},{"type":"v1","addr":"172.21.15.115:6815","nonce":1073133477}]},"public_addr":"172.21.15.115:6811/1073133477","cluster_addr":"172.21.15.115:6813/1073133477","heartbeat_back_addr":"172.21.15.115:6817/1073133477","heartbeat_front_addr":"172.21.15.115:6815/1073133477","state":["exists","up"]},{"osd":2,"uuid":"e3aecb00-afa4-4473-abf6-aba50f8b2b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":378,"up_from":392,"up_thru":583,"down_at":379,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":712263769},{"type":"v1","addr":"172.21.15.115:6819","nonce":712263769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":712263769},{"type":"v1","addr":"172.21.15.115:6821","nonce":712263769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":712263769},{"type":"v1","addr":"172.21.15.115:6825","nonce":712263769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":712263769},{"type":"v1","addr":"172.21.15.115:6823","nonce":712263769}]},"public_addr":"172.21.15.115:6819/712263769","cluster_addr":"172.21.15.115:6821/712263769","heartbeat_back_addr":"172.21.15.115:6825/712263769","heartbeat_front_addr":"172.21.15.115:6823/712263769","state":["exists","up"]},{"osd":3,"uuid":"96c0386b-e1ba-44a5-8daf-f76b1303b81d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":405,"up_from":419,"up_thru":583,"down_at":406,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":194469414},{"type":"v1","addr":"172.21.15.115:6827","nonce":194469414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":194469414},{"type":"v1","addr":"172.21.15.115:6829","nonce":194469414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":194469414},{"type":"v1","addr":"172.21.15.115:6833","nonce":194469414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":194469414},{"type":"v1","addr":"172.21.15.115:6831","nonce":194469414}]},"public_addr":"172.21.15.115:6827/194469414","cluster_addr":"172.21.15.115:6829/194469414","heartbeat_back_addr":"172.21.15.115:6833/194469414","heartbeat_front_addr":"172.21.15.115:6831/194469414","state":["exists","up"]},{"osd":4,"uuid":"d0f6ab0c-14ee-4800-86bc-daa9f68b116c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":446,"up_from":460,"up_thru":583,"down_at":447,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":531418945},{"type":"v1","addr":"172.21.15.121:6801","nonce":531418945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":531418945},{"type":"v1","addr":"172.21.15.121:6803","nonce":531418945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":531418945},{"type":"v1","addr":"172.21.15.121:6807","nonce":531418945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":531418945},{"type":"v1","addr":"172.21.15.121:6805","nonce":531418945}]},"public_addr":"172.21.15.121:6801/531418945","cluster_addr":"172.21.15.121:6803/531418945","heartbeat_back_addr":"172.21.15.121:6807/531418945","heartbeat_front_addr":"172.21.15.121:6805/531418945","state":["exists","up"]},{"osd":5,"uuid":"9b85a231-7194-42e3-ae6c-f61741e030fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":486,"up_from":501,"up_thru":583,"down_at":487,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":3650203057},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650203057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3650203057},{"type":"v1","addr":"172.21.15.121:6811","nonce":3650203057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650203057},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650203057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":3650203057},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650203057}]},"public_addr":"172.21.15.121:6809/3650203057","cluster_addr":"172.21.15.121:6811/3650203057","heartbeat_back_addr":"172.21.15.121:6815/3650203057","heartbeat_front_addr":"172.21.15.121:6813/3650203057","state":["exists","up"]},{"osd":6,"uuid":"d0e12e76-ac56-4f67-85c0-c945cecd1bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":42,"last_clean_end":528,"up_from":542,"up_thru":583,"down_at":529,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":75880214},{"type":"v1","addr":"172.21.15.121:6817","nonce":75880214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":75880214},{"type":"v1","addr":"172.21.15.121:6819","nonce":75880214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":75880214},{"type":"v1","addr":"172.21.15.121:6823","nonce":75880214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":75880214},{"type":"v1","addr":"172.21.15.121:6821","nonce":75880214}]},"public_addr":"172.21.15.121:6817/75880214","cluster_addr":"172.21.15.121:6819/75880214","heartbeat_back_addr":"172.21.15.121:6823/75880214","heartbeat_front_addr":"172.21.15.121:6821/75880214","state":["exists","up"]},{"osd":7,"uuid":"d4a829b3-bc21-4e29-aa3e-b1e4b15938b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":568,"up_from":583,"up_thru":583,"down_at":569,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2375986432},{"type":"v1","addr":"172.21.15.121:6825","nonce":2375986432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2375986432},{"type":"v1","addr":"172.21.15.121:6827","nonce":2375986432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2375986432},{"type":"v1","addr":"172.21.15.121:6831","nonce":2375986432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2375986432},{"type":"v1","addr":"172.21.15.121:6829","nonce":2375986432}]},"public_addr":"172.21.15.121:6825/2375986432","cluster_addr":"172.21.15.121:6827/2375986432","heartbeat_back_addr":"172.21.15.121:6831/2375986432","heartbeat_front_addr":"172.21.15.121:6829/2375986432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:26.300198+0000","dead_epoch":313},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:18:56.327288+0000","dead_epoch":354},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:26.122128+0000","dead_epoch":379},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:19:55.465445+0000","dead_epoch":406},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:20.334604+0000","dead_epoch":447},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:20:47.028408+0000","dead_epoch":487},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:16.138563+0000","dead_epoch":529},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-04-22T17:21:43.159935+0000","dead_epoch":569}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.3b","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/3174306807":"2022-04-23T17:27:18.025998+0000","172.21.15.115:6801/567188723":"2022-04-23T17:26:14.467763+0000","172.21.15.115:0/48028127":"2022-04-23T17:16:03.213585+0000","172.21.15.121:0/3821697752":"2022-04-23T17:27:18.025998+0000","172.21.15.121:0/3855222727":"2022-04-23T17:27:18.025998+0000","172.21.15.115:6800/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.121:6833/2211373279":"2022-04-23T17:35:17.646384+0000","172.21.15.121:6835/2438883244":"2022-04-23T17:27:18.025998+0000","172.21.15.115:0/1162730484":"2022-04-23T17:28:12.869795+0000","172.21.15.121:6834/2438883244":"2022-04-23T17:27:18.025998+0000","172.21.15.115:0/2552821320":"2022-04-23T17:16:36.269748+0000","172.21.15.121:0/3045429302":"2022-04-23T17:27:18.025998+0000","172.21.15.115:0/529432417":"2022-04-23T17:16:36.269748+0000","172.21.15.115:6801/593518270":"2022-04-23T17:16:36.269748+0000","172.21.15.115:0/4035732501":"2022-04-23T17:28:12.869795+0000","172.21.15.121:6832/2211373279":"2022-04-23T17:35:17.646384+0000","172.21.15.115:0/2286932496":"2022-04-23T17:26:14.467763+0000","172.21.15.115:0/2425258310":"2022-04-23T17:28:12.869795+0000","172.21.15.115:6800/3990000267":"2022-04-23T17:28:12.869795+0000","172.21.15.115:0/1876895505":"2022-04-23T17:26:14.467763+0000","172.21.15.115:0/777728622":"2022-04-23T17:16:03.213585+0000","172.21.15.121:0/3882903977":"2022-04-23T17:27:18.025998+0000","172.21.15.115:0/1172148055":"2022-04-23T17:28:12.869795+0000","172.21.15.115:0/1738214082":"2022-04-23T17:28:12.869795+0000","172.21.15.121:0/2624634203":"2022-04-23T17:27:18.025998+0000","172.21.15.115:6801/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6800/2486359415":"2022-04-23T17:16:03.213585+0000","172.21.15.115:6801/3990000267":"2022-04-23T17:28:12.869795+0000","172.21.15.115:6800/567188723":"2022-04-23T17:26:14.467763+0000","172.21.15.115:0/2507339421":"2022-04-23T17:28:12.869795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":770,"length":1},{"begin":779,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":750,"length":1},{"begin":763,"length":1},{"begin":773,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-22T17:39:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:34 smithi121 bash[36902]: audit 2022-04-22T17:39:34.077307+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/1780898001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:39:34.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:34 smithi115 bash[45495]: audit 2022-04-22T17:39:34.077307+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/1780898001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:39:34.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:34 smithi115 bash[46934]: audit 2022-04-22T17:39:34.077307+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.115:0/1780898001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-04-22T17:39:35.000 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph pg dump --format=json 2022-04-22T17:39:35.122 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:39:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[46934]: cluster 2022-04-22T17:39:34.023721+0000 mgr.y (mgr.34107) 515 : cluster [DBG] pgmap v983: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 203 KiB/s wr, 2 op/s 2022-04-22T17:39:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[46934]: audit 2022-04-22T17:39:34.501911+0000 mon.a (mon.0) 1581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[46934]: cluster 2022-04-22T17:39:35.045495+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2022-04-22T17:39:35.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[45495]: cluster 2022-04-22T17:39:34.023721+0000 mgr.y (mgr.34107) 515 : cluster [DBG] pgmap v983: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 203 KiB/s wr, 2 op/s 2022-04-22T17:39:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[45495]: audit 2022-04-22T17:39:34.501911+0000 mon.a (mon.0) 1581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:35.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:35 smithi115 bash[45495]: cluster 2022-04-22T17:39:35.045495+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2022-04-22T17:39:35.466 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:39:35.466 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 2m ago 16m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:39:35.466 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (16m) 3m ago 16m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:39:35.466 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4m) 2m ago 17m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:39:35.466 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4m) 3m ago 17m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (11m) 3m ago 21m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (12m) 2m ago 23m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (11m) 2m ago 24m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (10m) 3m ago 22m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 2m ago 22m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 2m ago 16m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3m) 3m ago 16m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 2m ago 21m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 2m ago 20m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:39:35.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 2m ago 20m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 2m ago 19m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (7m) 3m ago 19m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6m) 3m ago 18m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5m) 3m ago 18m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5m) 3m ago 18m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:39:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3m) 3m ago 16m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:39:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:35 smithi121 bash[36902]: cluster 2022-04-22T17:39:34.023721+0000 mgr.y (mgr.34107) 515 : cluster [DBG] pgmap v983: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 203 KiB/s wr, 2 op/s 2022-04-22T17:39:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:35 smithi121 bash[36902]: audit 2022-04-22T17:39:34.501911+0000 mon.a (mon.0) 1581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:35 smithi121 bash[36902]: cluster 2022-04-22T17:39:35.045495+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2022-04-22T17:39:35.879 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:39:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:39:35.881 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:39:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:36 smithi121 bash[36902]: audit 2022-04-22T17:39:35.092084+0000 mgr.y (mgr.34107) 516 : audit [DBG] from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:36 smithi121 bash[36902]: audit 2022-04-22T17:39:35.866873+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/2479735702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:36.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:36 smithi115 bash[45495]: audit 2022-04-22T17:39:35.092084+0000 mgr.y (mgr.34107) 516 : audit [DBG] from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:36.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:36 smithi115 bash[45495]: audit 2022-04-22T17:39:35.866873+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/2479735702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:36.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:36 smithi115 bash[46934]: audit 2022-04-22T17:39:35.092084+0000 mgr.y (mgr.34107) 516 : audit [DBG] from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:36.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:36 smithi115 bash[46934]: audit 2022-04-22T17:39:35.866873+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.115:0/2479735702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:39:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:37 smithi121 bash[36902]: audit 2022-04-22T17:39:35.433379+0000 mgr.y (mgr.34107) 517 : audit [DBG] from='client.44527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:37 smithi121 bash[36902]: cluster 2022-04-22T17:39:36.024705+0000 mgr.y (mgr.34107) 518 : cluster [DBG] pgmap v985: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 100 KiB/s wr, 0 op/s 2022-04-22T17:39:37.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:37 smithi115 bash[45495]: audit 2022-04-22T17:39:35.433379+0000 mgr.y (mgr.34107) 517 : audit [DBG] from='client.44527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:37.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:37 smithi115 bash[45495]: cluster 2022-04-22T17:39:36.024705+0000 mgr.y (mgr.34107) 518 : cluster [DBG] pgmap v985: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 100 KiB/s wr, 0 op/s 2022-04-22T17:39:37.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:37 smithi115 bash[46934]: audit 2022-04-22T17:39:35.433379+0000 mgr.y (mgr.34107) 517 : audit [DBG] from='client.44527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:39:37.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:37 smithi115 bash[46934]: cluster 2022-04-22T17:39:36.024705+0000 mgr.y (mgr.34107) 518 : cluster [DBG] pgmap v985: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 100 KiB/s wr, 0 op/s 2022-04-22T17:39:37.787 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:39:39.381 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:39 smithi115 bash[45495]: cluster 2022-04-22T17:39:38.025819+0000 mgr.y (mgr.34107) 519 : cluster [DBG] pgmap v986: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:39.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:39 smithi115 bash[46934]: cluster 2022-04-22T17:39:38.025819+0000 mgr.y (mgr.34107) 519 : cluster [DBG] pgmap v986: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:39.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:39 smithi121 bash[36902]: cluster 2022-04-22T17:39:38.025819+0000 mgr.y (mgr.34107) 519 : cluster [DBG] pgmap v986: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:40 smithi121 bash[36902]: audit 2022-04-22T17:39:39.504896+0000 mon.a (mon.0) 1583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:40 smithi115 bash[45495]: audit 2022-04-22T17:39:39.504896+0000 mon.a (mon.0) 1583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:40 smithi115 bash[46934]: audit 2022-04-22T17:39:39.504896+0000 mon.a (mon.0) 1583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:41 smithi121 bash[36902]: cluster 2022-04-22T17:39:40.026977+0000 mgr.y (mgr.34107) 520 : cluster [DBG] pgmap v987: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:41 smithi115 bash[46934]: cluster 2022-04-22T17:39:40.026977+0000 mgr.y (mgr.34107) 520 : cluster [DBG] pgmap v987: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:41.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:41 smithi115 bash[45495]: cluster 2022-04-22T17:39:40.026977+0000 mgr.y (mgr.34107) 520 : cluster [DBG] pgmap v987: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:41.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-04-22T17:39:41.642 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2022-04-22T17:39:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:41.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:42.475 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":987,"stamp":"2022-04-22T17:39:40.026170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322893734,"num_objects":108,"num_object_clones":40,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":7,"num_read":4167,"num_read_kb":5939362,"num_write":8710,"num_write_kb":2048213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":339558400,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21181,"ondisk_log_size":21181,"up":363,"acting":363,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":363,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2287244,"kb_used_data":547852,"kb_used_omap":795,"kb_used_meta":1738532,"kb_avail":747673972,"statfs":{"total":767960285184,"available":765618147328,"internally_reserved":0,"allocated":561000448,"data_stored":777462692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":814384,"internal_metadata":1780257488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.987871"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":805,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745993+0000","last_change":"2022-04-22T17:34:42.070985+0000","last_active":"2022-04-22T17:39:27.745993+0000","last_peered":"2022-04-22T17:39:27.745993+0000","last_clean":"2022-04-22T17:39:27.745993+0000","last_became_active":"2022-04-22T17:34:42.070653+0000","last_became_peered":"2022-04-22T17:34:42.070653+0000","last_unstale":"2022-04-22T17:39:27.745993+0000","last_undegraded":"2022-04-22T17:39:27.745993+0000","last_fullsized":"2022-04-22T17:39:27.745993+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:51:44.180557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045802+0000","last_change":"2022-04-22T17:33:59.973105+0000","last_active":"2022-04-22T17:39:30.045802+0000","last_peered":"2022-04-22T17:39:30.045802+0000","last_clean":"2022-04-22T17:39:30.045802+0000","last_became_active":"2022-04-22T17:33:59.972694+0000","last_became_peered":"2022-04-22T17:33:59.972694+0000","last_unstale":"2022-04-22T17:39:30.045802+0000","last_undegraded":"2022-04-22T17:39:30.045802+0000","last_fullsized":"2022-04-22T17:39:30.045802+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:11:59.029343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.112324+0000","last_change":"2022-04-22T17:31:28.834964+0000","last_active":"2022-04-22T17:39:30.112324+0000","last_peered":"2022-04-22T17:39:30.112324+0000","last_clean":"2022-04-22T17:39:30.112324+0000","last_became_active":"2022-04-22T17:31:28.834595+0000","last_became_peered":"2022-04-22T17:31:28.834595+0000","last_unstale":"2022-04-22T17:39:30.112324+0000","last_undegraded":"2022-04-22T17:39:30.112324+0000","last_fullsized":"2022-04-22T17:39:30.112324+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:18:56.356332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":805,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051933+0000","last_change":"2022-04-22T17:34:42.059331+0000","last_active":"2022-04-22T17:39:30.051933+0000","last_peered":"2022-04-22T17:39:30.051933+0000","last_clean":"2022-04-22T17:39:30.051933+0000","last_became_active":"2022-04-22T17:34:42.058145+0000","last_became_peered":"2022-04-22T17:34:42.058145+0000","last_unstale":"2022-04-22T17:39:30.051933+0000","last_undegraded":"2022-04-22T17:39:30.051933+0000","last_fullsized":"2022-04-22T17:39:30.051933+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:55:20.947760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":815,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745120+0000","last_change":"2022-04-22T17:34:42.056173+0000","last_active":"2022-04-22T17:39:27.745120+0000","last_peered":"2022-04-22T17:39:27.745120+0000","last_clean":"2022-04-22T17:39:27.745120+0000","last_became_active":"2022-04-22T17:34:42.052897+0000","last_became_peered":"2022-04-22T17:34:42.052897+0000","last_unstale":"2022-04-22T17:39:27.745120+0000","last_undegraded":"2022-04-22T17:39:27.745120+0000","last_fullsized":"2022-04-22T17:39:27.745120+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:25:36.319552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":791,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052128+0000","last_change":"2022-04-22T17:33:18.964744+0000","last_active":"2022-04-22T17:39:30.052128+0000","last_peered":"2022-04-22T17:39:30.052128+0000","last_clean":"2022-04-22T17:39:30.052128+0000","last_became_active":"2022-04-22T17:33:18.963007+0000","last_became_peered":"2022-04-22T17:33:18.963007+0000","last_unstale":"2022-04-22T17:39:30.052128+0000","last_undegraded":"2022-04-22T17:39:30.052128+0000","last_fullsized":"2022-04-22T17:39:30.052128+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:41:05.011996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":806,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.744749+0000","last_change":"2022-04-22T17:34:42.065944+0000","last_active":"2022-04-22T17:39:27.744749+0000","last_peered":"2022-04-22T17:39:27.744749+0000","last_clean":"2022-04-22T17:39:27.744749+0000","last_became_active":"2022-04-22T17:34:42.065555+0000","last_became_peered":"2022-04-22T17:34:42.065555+0000","last_unstale":"2022-04-22T17:39:27.744749+0000","last_undegraded":"2022-04-22T17:39:27.744749+0000","last_fullsized":"2022-04-22T17:39:27.744749+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:30:27.177240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":818,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.040807+0000","last_change":"2022-04-22T17:33:59.966436+0000","last_active":"2022-04-22T17:39:30.040807+0000","last_peered":"2022-04-22T17:39:30.040807+0000","last_clean":"2022-04-22T17:39:30.040807+0000","last_became_active":"2022-04-22T17:33:59.965998+0000","last_became_peered":"2022-04-22T17:33:59.965998+0000","last_unstale":"2022-04-22T17:39:30.040807+0000","last_undegraded":"2022-04-22T17:39:30.040807+0000","last_fullsized":"2022-04-22T17:39:30.040807+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:07:53.822281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":806,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.746127+0000","last_change":"2022-04-22T17:34:42.068425+0000","last_active":"2022-04-22T17:39:27.746127+0000","last_peered":"2022-04-22T17:39:27.746127+0000","last_clean":"2022-04-22T17:39:27.746127+0000","last_became_active":"2022-04-22T17:34:42.068157+0000","last_became_peered":"2022-04-22T17:34:42.068157+0000","last_unstale":"2022-04-22T17:39:27.746127+0000","last_undegraded":"2022-04-22T17:39:27.746127+0000","last_fullsized":"2022-04-22T17:39:27.746127+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:32:02.167865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":798,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.102996+0000","last_change":"2022-04-22T17:33:59.987247+0000","last_active":"2022-04-22T17:39:30.102996+0000","last_peered":"2022-04-22T17:39:30.102996+0000","last_clean":"2022-04-22T17:39:30.102996+0000","last_became_active":"2022-04-22T17:33:59.986588+0000","last_became_peered":"2022-04-22T17:33:59.986588+0000","last_unstale":"2022-04-22T17:39:30.102996+0000","last_undegraded":"2022-04-22T17:39:30.102996+0000","last_fullsized":"2022-04-22T17:39:30.102996+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:07:26.092105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":798,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.102978+0000","last_change":"2022-04-22T17:33:59.988658+0000","last_active":"2022-04-22T17:39:30.102978+0000","last_peered":"2022-04-22T17:39:30.102978+0000","last_clean":"2022-04-22T17:39:30.102978+0000","last_became_active":"2022-04-22T17:33:59.988042+0000","last_became_peered":"2022-04-22T17:33:59.988042+0000","last_unstale":"2022-04-22T17:39:30.102978+0000","last_undegraded":"2022-04-22T17:39:30.102978+0000","last_fullsized":"2022-04-22T17:39:30.102978+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:58:13.105609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.120304+0000","last_change":"2022-04-22T17:33:59.977732+0000","last_active":"2022-04-22T17:39:30.120304+0000","last_peered":"2022-04-22T17:39:30.120304+0000","last_clean":"2022-04-22T17:39:30.120304+0000","last_became_active":"2022-04-22T17:33:59.977367+0000","last_became_peered":"2022-04-22T17:33:59.977367+0000","last_unstale":"2022-04-22T17:39:30.120304+0000","last_undegraded":"2022-04-22T17:39:30.120304+0000","last_fullsized":"2022-04-22T17:39:30.120304+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:32:24.797326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":802,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.042427+0000","last_change":"2022-04-22T17:33:59.966056+0000","last_active":"2022-04-22T17:39:30.042427+0000","last_peered":"2022-04-22T17:39:30.042427+0000","last_clean":"2022-04-22T17:39:30.042427+0000","last_became_active":"2022-04-22T17:33:59.965669+0000","last_became_peered":"2022-04-22T17:33:59.965669+0000","last_unstale":"2022-04-22T17:39:30.042427+0000","last_undegraded":"2022-04-22T17:39:30.042427+0000","last_fullsized":"2022-04-22T17:39:30.042427+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:04:46.951806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111976+0000","last_change":"2022-04-22T17:33:18.961338+0000","last_active":"2022-04-22T17:39:30.111976+0000","last_peered":"2022-04-22T17:39:30.111976+0000","last_clean":"2022-04-22T17:39:30.111976+0000","last_became_active":"2022-04-22T17:33:18.959702+0000","last_became_peered":"2022-04-22T17:33:18.959702+0000","last_unstale":"2022-04-22T17:39:30.111976+0000","last_undegraded":"2022-04-22T17:39:30.111976+0000","last_fullsized":"2022-04-22T17:39:30.111976+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:37:51.238181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045155+0000","last_change":"2022-04-22T17:33:18.951402+0000","last_active":"2022-04-22T17:39:30.045155+0000","last_peered":"2022-04-22T17:39:30.045155+0000","last_clean":"2022-04-22T17:39:30.045155+0000","last_became_active":"2022-04-22T17:33:18.951061+0000","last_became_peered":"2022-04-22T17:33:18.951061+0000","last_unstale":"2022-04-22T17:39:30.045155+0000","last_undegraded":"2022-04-22T17:39:30.045155+0000","last_fullsized":"2022-04-22T17:39:30.045155+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:28:11.111464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":793,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051204+0000","last_change":"2022-04-22T17:34:42.064031+0000","last_active":"2022-04-22T17:39:30.051204+0000","last_peered":"2022-04-22T17:39:30.051204+0000","last_clean":"2022-04-22T17:39:30.051204+0000","last_became_active":"2022-04-22T17:34:42.062643+0000","last_became_peered":"2022-04-22T17:34:42.062643+0000","last_unstale":"2022-04-22T17:39:30.051204+0000","last_undegraded":"2022-04-22T17:39:30.051204+0000","last_fullsized":"2022-04-22T17:39:30.051204+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:44:05.817044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":804,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.053717+0000","last_change":"2022-04-22T17:34:42.064082+0000","last_active":"2022-04-22T17:39:30.053717+0000","last_peered":"2022-04-22T17:39:30.053717+0000","last_clean":"2022-04-22T17:39:30.053717+0000","last_became_active":"2022-04-22T17:34:42.063734+0000","last_became_peered":"2022-04-22T17:34:42.063734+0000","last_unstale":"2022-04-22T17:39:30.053717+0000","last_undegraded":"2022-04-22T17:39:30.053717+0000","last_fullsized":"2022-04-22T17:39:30.053717+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:27:11.409543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.114448+0000","last_change":"2022-04-22T17:34:42.043002+0000","last_active":"2022-04-22T17:39:30.114448+0000","last_peered":"2022-04-22T17:39:30.114448+0000","last_clean":"2022-04-22T17:39:30.114448+0000","last_became_active":"2022-04-22T17:34:42.042680+0000","last_became_peered":"2022-04-22T17:34:42.042680+0000","last_unstale":"2022-04-22T17:39:30.114448+0000","last_undegraded":"2022-04-22T17:39:30.114448+0000","last_fullsized":"2022-04-22T17:39:30.114448+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:53:54.093993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":805,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.746097+0000","last_change":"2022-04-22T17:34:42.069326+0000","last_active":"2022-04-22T17:39:27.746097+0000","last_peered":"2022-04-22T17:39:27.746097+0000","last_clean":"2022-04-22T17:39:27.746097+0000","last_became_active":"2022-04-22T17:34:42.068941+0000","last_became_peered":"2022-04-22T17:34:42.068941+0000","last_unstale":"2022-04-22T17:39:27.746097+0000","last_undegraded":"2022-04-22T17:39:27.746097+0000","last_fullsized":"2022-04-22T17:39:27.746097+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:21:22.586499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"621'5","reported_seq":834,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.115203+0000","last_change":"2022-04-22T17:34:42.048443+0000","last_active":"2022-04-22T17:39:30.115203+0000","last_peered":"2022-04-22T17:39:30.115203+0000","last_clean":"2022-04-22T17:39:30.115203+0000","last_became_active":"2022-04-22T17:34:42.048035+0000","last_became_peered":"2022-04-22T17:34:42.048035+0000","last_unstale":"2022-04-22T17:39:30.115203+0000","last_undegraded":"2022-04-22T17:39:30.115203+0000","last_fullsized":"2022-04-22T17:39:30.115203+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:16:13.453884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":827,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.043894+0000","last_change":"2022-04-22T17:33:59.976186+0000","last_active":"2022-04-22T17:39:30.043894+0000","last_peered":"2022-04-22T17:39:30.043894+0000","last_clean":"2022-04-22T17:39:30.043894+0000","last_became_active":"2022-04-22T17:33:59.975819+0000","last_became_peered":"2022-04-22T17:33:59.975819+0000","last_unstale":"2022-04-22T17:39:30.043894+0000","last_undegraded":"2022-04-22T17:39:30.043894+0000","last_fullsized":"2022-04-22T17:39:30.043894+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:17:35.105115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"621'2","reported_seq":806,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.044431+0000","last_change":"2022-04-22T17:33:18.940758+0000","last_active":"2022-04-22T17:39:30.044431+0000","last_peered":"2022-04-22T17:39:30.044431+0000","last_clean":"2022-04-22T17:39:30.044431+0000","last_became_active":"2022-04-22T17:33:18.940155+0000","last_became_peered":"2022-04-22T17:33:18.940155+0000","last_unstale":"2022-04-22T17:39:30.044431+0000","last_undegraded":"2022-04-22T17:39:30.044431+0000","last_fullsized":"2022-04-22T17:39:30.044431+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:35:08.358508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113876+0000","last_change":"2022-04-22T17:33:18.949386+0000","last_active":"2022-04-22T17:39:30.113876+0000","last_peered":"2022-04-22T17:39:30.113876+0000","last_clean":"2022-04-22T17:39:30.113876+0000","last_became_active":"2022-04-22T17:33:18.948640+0000","last_became_peered":"2022-04-22T17:33:18.948640+0000","last_unstale":"2022-04-22T17:39:30.113876+0000","last_undegraded":"2022-04-22T17:39:30.113876+0000","last_fullsized":"2022-04-22T17:39:30.113876+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:29:31.945940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"53'2","reported_seq":810,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745934+0000","last_change":"2022-04-22T17:34:42.075226+0000","last_active":"2022-04-22T17:39:27.745934+0000","last_peered":"2022-04-22T17:39:27.745934+0000","last_clean":"2022-04-22T17:39:27.745934+0000","last_became_active":"2022-04-22T17:34:42.074506+0000","last_became_peered":"2022-04-22T17:34:42.074506+0000","last_unstale":"2022-04-22T17:39:27.745934+0000","last_undegraded":"2022-04-22T17:39:27.745934+0000","last_fullsized":"2022-04-22T17:39:27.745934+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:45:23.999534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":802,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052894+0000","last_change":"2022-04-22T17:33:18.964461+0000","last_active":"2022-04-22T17:39:30.052894+0000","last_peered":"2022-04-22T17:39:30.052894+0000","last_clean":"2022-04-22T17:39:30.052894+0000","last_became_active":"2022-04-22T17:33:18.963012+0000","last_became_peered":"2022-04-22T17:33:18.963012+0000","last_unstale":"2022-04-22T17:39:30.052894+0000","last_undegraded":"2022-04-22T17:39:30.052894+0000","last_fullsized":"2022-04-22T17:39:30.052894+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:19:48.634066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"53'2","reported_seq":811,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745350+0000","last_change":"2022-04-22T17:34:42.073444+0000","last_active":"2022-04-22T17:39:27.745350+0000","last_peered":"2022-04-22T17:39:27.745350+0000","last_clean":"2022-04-22T17:39:27.745350+0000","last_became_active":"2022-04-22T17:34:42.067704+0000","last_became_peered":"2022-04-22T17:34:42.067704+0000","last_unstale":"2022-04-22T17:39:27.745350+0000","last_undegraded":"2022-04-22T17:39:27.745350+0000","last_fullsized":"2022-04-22T17:39:27.745350+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:05:42.724520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":804,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052272+0000","last_change":"2022-04-22T17:33:59.984740+0000","last_active":"2022-04-22T17:39:30.052272+0000","last_peered":"2022-04-22T17:39:30.052272+0000","last_clean":"2022-04-22T17:39:30.052272+0000","last_became_active":"2022-04-22T17:33:59.984247+0000","last_became_peered":"2022-04-22T17:33:59.984247+0000","last_unstale":"2022-04-22T17:39:30.052272+0000","last_undegraded":"2022-04-22T17:39:30.052272+0000","last_fullsized":"2022-04-22T17:39:30.052272+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:49:02.544112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"621'2","reported_seq":821,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041840+0000","last_change":"2022-04-22T17:34:42.039208+0000","last_active":"2022-04-22T17:39:30.041840+0000","last_peered":"2022-04-22T17:39:30.041840+0000","last_clean":"2022-04-22T17:39:30.041840+0000","last_became_active":"2022-04-22T17:34:42.038840+0000","last_became_peered":"2022-04-22T17:34:42.038840+0000","last_unstale":"2022-04-22T17:39:30.041840+0000","last_undegraded":"2022-04-22T17:39:30.041840+0000","last_fullsized":"2022-04-22T17:39:30.041840+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:44:33.651829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":821,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.121503+0000","last_change":"2022-04-22T17:32:36.873361+0000","last_active":"2022-04-22T17:39:30.121503+0000","last_peered":"2022-04-22T17:39:30.121503+0000","last_clean":"2022-04-22T17:39:30.121503+0000","last_became_active":"2022-04-22T17:32:36.872987+0000","last_became_peered":"2022-04-22T17:32:36.872987+0000","last_unstale":"2022-04-22T17:39:30.121503+0000","last_undegraded":"2022-04-22T17:39:30.121503+0000","last_fullsized":"2022-04-22T17:39:30.121503+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:03:52.949883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"621'2","reported_seq":804,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.102085+0000","last_change":"2022-04-22T17:34:42.069359+0000","last_active":"2022-04-22T17:39:30.102085+0000","last_peered":"2022-04-22T17:39:30.102085+0000","last_clean":"2022-04-22T17:39:30.102085+0000","last_became_active":"2022-04-22T17:34:42.068906+0000","last_became_peered":"2022-04-22T17:34:42.068906+0000","last_unstale":"2022-04-22T17:39:30.102085+0000","last_undegraded":"2022-04-22T17:39:30.102085+0000","last_fullsized":"2022-04-22T17:39:30.102085+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:04:03.248887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.112670+0000","last_change":"2022-04-22T17:34:42.039951+0000","last_active":"2022-04-22T17:39:30.112670+0000","last_peered":"2022-04-22T17:39:30.112670+0000","last_clean":"2022-04-22T17:39:30.112670+0000","last_became_active":"2022-04-22T17:34:42.039644+0000","last_became_peered":"2022-04-22T17:34:42.039644+0000","last_unstale":"2022-04-22T17:39:30.112670+0000","last_undegraded":"2022-04-22T17:39:30.112670+0000","last_fullsized":"2022-04-22T17:39:30.112670+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:02:46.565765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":804,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041610+0000","last_change":"2022-04-22T17:33:59.975695+0000","last_active":"2022-04-22T17:39:30.041610+0000","last_peered":"2022-04-22T17:39:30.041610+0000","last_clean":"2022-04-22T17:39:30.041610+0000","last_became_active":"2022-04-22T17:33:59.975202+0000","last_became_peered":"2022-04-22T17:33:59.975202+0000","last_unstale":"2022-04-22T17:39:30.041610+0000","last_undegraded":"2022-04-22T17:39:30.041610+0000","last_fullsized":"2022-04-22T17:39:30.041610+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:51:10.653673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.040943+0000","last_change":"2022-04-22T17:31:55.895438+0000","last_active":"2022-04-22T17:39:30.040943+0000","last_peered":"2022-04-22T17:39:30.040943+0000","last_clean":"2022-04-22T17:39:30.040943+0000","last_became_active":"2022-04-22T17:31:55.891628+0000","last_became_peered":"2022-04-22T17:31:55.891628+0000","last_unstale":"2022-04-22T17:39:30.040943+0000","last_undegraded":"2022-04-22T17:39:30.040943+0000","last_fullsized":"2022-04-22T17:39:30.040943+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:00:14.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":812,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051626+0000","last_change":"2022-04-22T17:33:18.953637+0000","last_active":"2022-04-22T17:39:30.051626+0000","last_peered":"2022-04-22T17:39:30.051626+0000","last_clean":"2022-04-22T17:39:30.051626+0000","last_became_active":"2022-04-22T17:33:18.952861+0000","last_became_peered":"2022-04-22T17:33:18.952861+0000","last_unstale":"2022-04-22T17:39:30.051626+0000","last_undegraded":"2022-04-22T17:39:30.051626+0000","last_fullsized":"2022-04-22T17:39:30.051626+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:40:33.710728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045101+0000","last_change":"2022-04-22T17:32:36.862335+0000","last_active":"2022-04-22T17:39:30.045101+0000","last_peered":"2022-04-22T17:39:30.045101+0000","last_clean":"2022-04-22T17:39:30.045101+0000","last_became_active":"2022-04-22T17:32:36.861954+0000","last_became_peered":"2022-04-22T17:32:36.861954+0000","last_unstale":"2022-04-22T17:39:30.045101+0000","last_undegraded":"2022-04-22T17:39:30.045101+0000","last_fullsized":"2022-04-22T17:39:30.045101+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:33:47.860496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.103546+0000","last_change":"2022-04-22T17:33:59.988130+0000","last_active":"2022-04-22T17:39:30.103546+0000","last_peered":"2022-04-22T17:39:30.103546+0000","last_clean":"2022-04-22T17:39:30.103546+0000","last_became_active":"2022-04-22T17:33:59.987633+0000","last_became_peered":"2022-04-22T17:33:59.987633+0000","last_unstale":"2022-04-22T17:39:30.103546+0000","last_undegraded":"2022-04-22T17:39:30.103546+0000","last_fullsized":"2022-04-22T17:39:30.103546+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:42:39.767932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":793,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052944+0000","last_change":"2022-04-22T17:34:42.058793+0000","last_active":"2022-04-22T17:39:30.052944+0000","last_peered":"2022-04-22T17:39:30.052944+0000","last_clean":"2022-04-22T17:39:30.052944+0000","last_became_active":"2022-04-22T17:34:42.058462+0000","last_became_peered":"2022-04-22T17:34:42.058462+0000","last_unstale":"2022-04-22T17:39:30.052944+0000","last_undegraded":"2022-04-22T17:39:30.052944+0000","last_fullsized":"2022-04-22T17:39:30.052944+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:42:01.882884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":820,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052570+0000","last_change":"2022-04-22T17:33:59.987693+0000","last_active":"2022-04-22T17:39:30.052570+0000","last_peered":"2022-04-22T17:39:30.052570+0000","last_clean":"2022-04-22T17:39:30.052570+0000","last_became_active":"2022-04-22T17:33:59.985309+0000","last_became_peered":"2022-04-22T17:33:59.985309+0000","last_unstale":"2022-04-22T17:39:30.052570+0000","last_undegraded":"2022-04-22T17:39:30.052570+0000","last_fullsized":"2022-04-22T17:39:30.052570+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:48:44.125945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.114344+0000","last_change":"2022-04-22T17:33:18.949554+0000","last_active":"2022-04-22T17:39:30.114344+0000","last_peered":"2022-04-22T17:39:30.114344+0000","last_clean":"2022-04-22T17:39:30.114344+0000","last_became_active":"2022-04-22T17:33:18.949076+0000","last_became_peered":"2022-04-22T17:33:18.949076+0000","last_unstale":"2022-04-22T17:39:30.114344+0000","last_undegraded":"2022-04-22T17:39:30.114344+0000","last_fullsized":"2022-04-22T17:39:30.114344+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:22:37.110089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"53'3","reported_seq":806,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045758+0000","last_change":"2022-04-22T17:33:18.944067+0000","last_active":"2022-04-22T17:39:30.045758+0000","last_peered":"2022-04-22T17:39:30.045758+0000","last_clean":"2022-04-22T17:39:30.045758+0000","last_became_active":"2022-04-22T17:33:18.943691+0000","last_became_peered":"2022-04-22T17:33:18.943691+0000","last_unstale":"2022-04-22T17:39:30.045758+0000","last_undegraded":"2022-04-22T17:39:30.045758+0000","last_fullsized":"2022-04-22T17:39:30.045758+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:57:54.470561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":815,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.120869+0000","last_change":"2022-04-22T17:34:42.060130+0000","last_active":"2022-04-22T17:39:30.120869+0000","last_peered":"2022-04-22T17:39:30.120869+0000","last_clean":"2022-04-22T17:39:30.120869+0000","last_became_active":"2022-04-22T17:34:42.059750+0000","last_became_peered":"2022-04-22T17:34:42.059750+0000","last_unstale":"2022-04-22T17:39:30.120869+0000","last_undegraded":"2022-04-22T17:39:30.120869+0000","last_fullsized":"2022-04-22T17:39:30.120869+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:06:25.879695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"621'23","reported_seq":828,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.102931+0000","last_change":"2022-04-22T17:33:59.987314+0000","last_active":"2022-04-22T17:39:30.102931+0000","last_peered":"2022-04-22T17:39:30.102931+0000","last_clean":"2022-04-22T17:39:30.102931+0000","last_became_active":"2022-04-22T17:33:59.986837+0000","last_became_peered":"2022-04-22T17:33:59.986837+0000","last_unstale":"2022-04-22T17:39:30.102931+0000","last_undegraded":"2022-04-22T17:39:30.102931+0000","last_fullsized":"2022-04-22T17:39:30.102931+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:18:39.336880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4740,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":806,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745527+0000","last_change":"2022-04-22T17:34:42.066543+0000","last_active":"2022-04-22T17:39:27.745527+0000","last_peered":"2022-04-22T17:39:27.745527+0000","last_clean":"2022-04-22T17:39:27.745527+0000","last_became_active":"2022-04-22T17:34:42.066097+0000","last_became_peered":"2022-04-22T17:34:42.066097+0000","last_unstale":"2022-04-22T17:39:27.745527+0000","last_undegraded":"2022-04-22T17:39:27.745527+0000","last_fullsized":"2022-04-22T17:39:27.745527+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:07:23.152349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"53'2","reported_seq":831,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113406+0000","last_change":"2022-04-22T17:32:36.867094+0000","last_active":"2022-04-22T17:39:30.113406+0000","last_peered":"2022-04-22T17:39:30.113406+0000","last_clean":"2022-04-22T17:39:30.113406+0000","last_became_active":"2022-04-22T17:32:36.866711+0000","last_became_peered":"2022-04-22T17:32:36.866711+0000","last_unstale":"2022-04-22T17:39:30.113406+0000","last_undegraded":"2022-04-22T17:39:30.113406+0000","last_fullsized":"2022-04-22T17:39:30.113406+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:32:20.698038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.110111+0000","last_change":"2022-04-22T17:34:42.040958+0000","last_active":"2022-04-22T17:39:30.110111+0000","last_peered":"2022-04-22T17:39:30.110111+0000","last_clean":"2022-04-22T17:39:30.110111+0000","last_became_active":"2022-04-22T17:34:42.040644+0000","last_became_peered":"2022-04-22T17:34:42.040644+0000","last_unstale":"2022-04-22T17:39:30.110111+0000","last_undegraded":"2022-04-22T17:39:30.110111+0000","last_fullsized":"2022-04-22T17:39:30.110111+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:38:48.203107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":806,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051886+0000","last_change":"2022-04-22T17:34:42.058690+0000","last_active":"2022-04-22T17:39:30.051886+0000","last_peered":"2022-04-22T17:39:30.051886+0000","last_clean":"2022-04-22T17:39:30.051886+0000","last_became_active":"2022-04-22T17:34:42.057356+0000","last_became_peered":"2022-04-22T17:34:42.057356+0000","last_unstale":"2022-04-22T17:39:30.051886+0000","last_undegraded":"2022-04-22T17:39:30.051886+0000","last_fullsized":"2022-04-22T17:39:30.051886+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:19:37.918516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":817,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045414+0000","last_change":"2022-04-22T17:32:36.862188+0000","last_active":"2022-04-22T17:39:30.045414+0000","last_peered":"2022-04-22T17:39:30.045414+0000","last_clean":"2022-04-22T17:39:30.045414+0000","last_became_active":"2022-04-22T17:32:36.861784+0000","last_became_peered":"2022-04-22T17:32:36.861784+0000","last_unstale":"2022-04-22T17:39:30.045414+0000","last_undegraded":"2022-04-22T17:39:30.045414+0000","last_fullsized":"2022-04-22T17:39:30.045414+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:29:28.302010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.101605+0000","last_change":"2022-04-22T17:33:59.985180+0000","last_active":"2022-04-22T17:39:30.101605+0000","last_peered":"2022-04-22T17:39:30.101605+0000","last_clean":"2022-04-22T17:39:30.101605+0000","last_became_active":"2022-04-22T17:33:59.983370+0000","last_became_peered":"2022-04-22T17:33:59.983370+0000","last_unstale":"2022-04-22T17:39:30.101605+0000","last_undegraded":"2022-04-22T17:39:30.101605+0000","last_fullsized":"2022-04-22T17:39:30.101605+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:19:18.936144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":798,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.101603+0000","last_change":"2022-04-22T17:33:59.989578+0000","last_active":"2022-04-22T17:39:30.101603+0000","last_peered":"2022-04-22T17:39:30.101603+0000","last_clean":"2022-04-22T17:39:30.101603+0000","last_became_active":"2022-04-22T17:33:59.986742+0000","last_became_peered":"2022-04-22T17:33:59.986742+0000","last_unstale":"2022-04-22T17:39:30.101603+0000","last_undegraded":"2022-04-22T17:39:30.101603+0000","last_fullsized":"2022-04-22T17:39:30.101603+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:30:57.023945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"53'8","reported_seq":838,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113411+0000","last_change":"2022-04-22T17:31:28.837863+0000","last_active":"2022-04-22T17:39:30.113411+0000","last_peered":"2022-04-22T17:39:30.113411+0000","last_clean":"2022-04-22T17:39:30.113411+0000","last_became_active":"2022-04-22T17:31:28.837345+0000","last_became_peered":"2022-04-22T17:31:28.837345+0000","last_unstale":"2022-04-22T17:39:30.113411+0000","last_undegraded":"2022-04-22T17:39:30.113411+0000","last_fullsized":"2022-04-22T17:39:30.113411+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:35:45.321278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":805,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.744847+0000","last_change":"2022-04-22T17:34:42.077183+0000","last_active":"2022-04-22T17:39:27.744847+0000","last_peered":"2022-04-22T17:39:27.744847+0000","last_clean":"2022-04-22T17:39:27.744847+0000","last_became_active":"2022-04-22T17:34:42.074628+0000","last_became_peered":"2022-04-22T17:34:42.074628+0000","last_unstale":"2022-04-22T17:39:27.744847+0000","last_undegraded":"2022-04-22T17:39:27.744847+0000","last_fullsized":"2022-04-22T17:39:27.744847+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:59:24.073905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"855'1267","reported_seq":3797,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.865192+0000","last_change":"2022-04-22T17:39:27.778507+0000","last_active":"2022-04-22T17:39:35.865192+0000","last_peered":"2022-04-22T17:39:35.865192+0000","last_clean":"2022-04-22T17:39:35.865192+0000","last_became_active":"2022-04-22T17:31:28.840526+0000","last_became_peered":"2022-04-22T17:31:28.840526+0000","last_unstale":"2022-04-22T17:39:35.865192+0000","last_undegraded":"2022-04-22T17:39:35.865192+0000","last_fullsized":"2022-04-22T17:39:35.865192+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1267,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:48:20.102757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010561116000000001,"stat_sum":{"num_bytes":17981440,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":278,"num_read_kb":424065,"num_write":505,"num_write_kb":99920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13115392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"53'1","reported_seq":802,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.121576+0000","last_change":"2022-04-22T17:34:42.058203+0000","last_active":"2022-04-22T17:39:30.121576+0000","last_peered":"2022-04-22T17:39:30.121576+0000","last_clean":"2022-04-22T17:39:30.121576+0000","last_became_active":"2022-04-22T17:34:42.057784+0000","last_became_peered":"2022-04-22T17:34:42.057784+0000","last_unstale":"2022-04-22T17:39:30.121576+0000","last_undegraded":"2022-04-22T17:39:30.121576+0000","last_fullsized":"2022-04-22T17:39:30.121576+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:25:07.196543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":806,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.746112+0000","last_change":"2022-04-22T17:34:42.076933+0000","last_active":"2022-04-22T17:39:27.746112+0000","last_peered":"2022-04-22T17:39:27.746112+0000","last_clean":"2022-04-22T17:39:27.746112+0000","last_became_active":"2022-04-22T17:34:42.075509+0000","last_became_peered":"2022-04-22T17:34:42.075509+0000","last_unstale":"2022-04-22T17:39:27.746112+0000","last_undegraded":"2022-04-22T17:39:27.746112+0000","last_fullsized":"2022-04-22T17:39:27.746112+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:55:08.893936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"850'1112","reported_seq":3602,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.044214+0000","last_change":"2022-04-22T17:39:27.774854+0000","last_active":"2022-04-22T17:39:35.044214+0000","last_peered":"2022-04-22T17:39:35.044214+0000","last_clean":"2022-04-22T17:39:35.044214+0000","last_became_active":"2022-04-22T17:33:59.965858+0000","last_became_peered":"2022-04-22T17:33:59.965858+0000","last_unstale":"2022-04-22T17:39:35.044214+0000","last_undegraded":"2022-04-22T17:39:35.044214+0000","last_fullsized":"2022-04-22T17:39:35.044214+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1112,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:35:33.891670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052123299999999996,"stat_sum":{"num_bytes":23486464,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":202,"num_read_kb":294221,"num_write":324,"num_write_kb":109336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":18120704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045513+0000","last_change":"2022-04-22T17:33:18.951340+0000","last_active":"2022-04-22T17:39:30.045513+0000","last_peered":"2022-04-22T17:39:30.045513+0000","last_clean":"2022-04-22T17:39:30.045513+0000","last_became_active":"2022-04-22T17:33:18.950984+0000","last_became_peered":"2022-04-22T17:33:18.950984+0000","last_unstale":"2022-04-22T17:39:30.045513+0000","last_undegraded":"2022-04-22T17:39:30.045513+0000","last_fullsized":"2022-04-22T17:39:30.045513+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:14:11.130442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045828+0000","last_change":"2022-04-22T17:34:42.058154+0000","last_active":"2022-04-22T17:39:30.045828+0000","last_peered":"2022-04-22T17:39:30.045828+0000","last_clean":"2022-04-22T17:39:30.045828+0000","last_became_active":"2022-04-22T17:34:42.057810+0000","last_became_peered":"2022-04-22T17:34:42.057810+0000","last_unstale":"2022-04-22T17:39:30.045828+0000","last_undegraded":"2022-04-22T17:39:30.045828+0000","last_fullsized":"2022-04-22T17:39:30.045828+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:29:41.278362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":803,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052061+0000","last_change":"2022-04-22T17:33:18.958087+0000","last_active":"2022-04-22T17:39:30.052061+0000","last_peered":"2022-04-22T17:39:30.052061+0000","last_clean":"2022-04-22T17:39:30.052061+0000","last_became_active":"2022-04-22T17:33:18.955141+0000","last_became_peered":"2022-04-22T17:33:18.955141+0000","last_unstale":"2022-04-22T17:39:30.052061+0000","last_undegraded":"2022-04-22T17:39:30.052061+0000","last_fullsized":"2022-04-22T17:39:30.052061+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:06:29.500826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"852'959","reported_seq":3446,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.044489+0000","last_change":"2022-04-22T17:39:27.774866+0000","last_active":"2022-04-22T17:39:35.044489+0000","last_peered":"2022-04-22T17:39:35.044489+0000","last_clean":"2022-04-22T17:39:35.044489+0000","last_became_active":"2022-04-22T17:32:36.875373+0000","last_became_peered":"2022-04-22T17:32:36.875373+0000","last_unstale":"2022-04-22T17:39:35.044489+0000","last_undegraded":"2022-04-22T17:39:35.044489+0000","last_fullsized":"2022-04-22T17:39:35.044489+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":959,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:33:11.951779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001399345,"stat_sum":{"num_bytes":11067392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":158,"num_read_kb":351326,"num_write":424,"num_write_kb":93480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":18022400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045619+0000","last_change":"2022-04-22T17:31:55.893236+0000","last_active":"2022-04-22T17:39:30.045619+0000","last_peered":"2022-04-22T17:39:30.045619+0000","last_clean":"2022-04-22T17:39:30.045619+0000","last_became_active":"2022-04-22T17:31:55.892828+0000","last_became_peered":"2022-04-22T17:31:55.892828+0000","last_unstale":"2022-04-22T17:39:30.045619+0000","last_undegraded":"2022-04-22T17:39:30.045619+0000","last_fullsized":"2022-04-22T17:39:30.045619+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:12:09.551718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":794,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051531+0000","last_change":"2022-04-22T17:33:59.988165+0000","last_active":"2022-04-22T17:39:30.051531+0000","last_peered":"2022-04-22T17:39:30.051531+0000","last_clean":"2022-04-22T17:39:30.051531+0000","last_became_active":"2022-04-22T17:33:59.987768+0000","last_became_peered":"2022-04-22T17:33:59.987768+0000","last_unstale":"2022-04-22T17:39:30.051531+0000","last_undegraded":"2022-04-22T17:39:30.051531+0000","last_fullsized":"2022-04-22T17:39:30.051531+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:40:55.253247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.121092+0000","last_change":"2022-04-22T17:33:59.977597+0000","last_active":"2022-04-22T17:39:30.121092+0000","last_peered":"2022-04-22T17:39:30.121092+0000","last_clean":"2022-04-22T17:39:30.121092+0000","last_became_active":"2022-04-22T17:33:59.977268+0000","last_became_peered":"2022-04-22T17:33:59.977268+0000","last_unstale":"2022-04-22T17:39:30.121092+0000","last_undegraded":"2022-04-22T17:39:30.121092+0000","last_fullsized":"2022-04-22T17:39:30.121092+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:47:51.933989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"855'1389","reported_seq":3857,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.867045+0000","last_change":"2022-04-22T17:39:27.777072+0000","last_active":"2022-04-22T17:39:35.867045+0000","last_peered":"2022-04-22T17:39:35.867045+0000","last_clean":"2022-04-22T17:39:35.867045+0000","last_became_active":"2022-04-22T17:33:18.943063+0000","last_became_peered":"2022-04-22T17:33:18.943063+0000","last_unstale":"2022-04-22T17:39:35.867045+0000","last_undegraded":"2022-04-22T17:39:35.867045+0000","last_fullsized":"2022-04-22T17:39:35.867045+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1389,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:20:17.806634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079064540000000003,"stat_sum":{"num_bytes":12632064,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":258,"num_read_kb":509797,"num_write":623,"num_write_kb":136232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10313728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":794,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.046180+0000","last_change":"2022-04-22T17:33:18.951896+0000","last_active":"2022-04-22T17:39:30.046180+0000","last_peered":"2022-04-22T17:39:30.046180+0000","last_clean":"2022-04-22T17:39:30.046180+0000","last_became_active":"2022-04-22T17:33:18.951547+0000","last_became_peered":"2022-04-22T17:33:18.951547+0000","last_unstale":"2022-04-22T17:39:30.046180+0000","last_undegraded":"2022-04-22T17:39:30.046180+0000","last_fullsized":"2022-04-22T17:39:30.046180+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:22:13.935116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.044818+0000","last_change":"2022-04-22T17:33:59.967719+0000","last_active":"2022-04-22T17:39:30.044818+0000","last_peered":"2022-04-22T17:39:30.044818+0000","last_clean":"2022-04-22T17:39:30.044818+0000","last_became_active":"2022-04-22T17:33:59.967289+0000","last_became_peered":"2022-04-22T17:33:59.967289+0000","last_unstale":"2022-04-22T17:39:30.044818+0000","last_undegraded":"2022-04-22T17:39:30.044818+0000","last_fullsized":"2022-04-22T17:39:30.044818+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:38:32.704862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.110691+0000","last_change":"2022-04-22T17:31:55.876229+0000","last_active":"2022-04-22T17:39:30.110691+0000","last_peered":"2022-04-22T17:39:30.110691+0000","last_clean":"2022-04-22T17:39:30.110691+0000","last_became_active":"2022-04-22T17:31:55.875844+0000","last_became_peered":"2022-04-22T17:31:55.875844+0000","last_unstale":"2022-04-22T17:39:30.110691+0000","last_undegraded":"2022-04-22T17:39:30.110691+0000","last_fullsized":"2022-04-22T17:39:30.110691+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:01:17.374715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"850'1673","reported_seq":4087,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.123304+0000","last_change":"2022-04-22T17:39:27.746522+0000","last_active":"2022-04-22T17:39:30.123304+0000","last_peered":"2022-04-22T17:39:30.123304+0000","last_clean":"2022-04-22T17:39:30.123304+0000","last_became_active":"2022-04-22T17:34:42.072269+0000","last_became_peered":"2022-04-22T17:34:42.072269+0000","last_unstale":"2022-04-22T17:39:30.123304+0000","last_undegraded":"2022-04-22T17:39:30.123304+0000","last_fullsized":"2022-04-22T17:39:30.123304+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1673,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:48:35.450756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078939099999999996,"stat_sum":{"num_bytes":23232512,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":296,"num_read_kb":423854,"num_write":736,"num_write_kb":164584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10895360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"302","length":"1"},{"start":"30b","length":"1"}]},{"pgid":"4.5","version":"0'0","reported_seq":796,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.103344+0000","last_change":"2022-04-22T17:34:42.068992+0000","last_active":"2022-04-22T17:39:30.103344+0000","last_peered":"2022-04-22T17:39:30.103344+0000","last_clean":"2022-04-22T17:39:30.103344+0000","last_became_active":"2022-04-22T17:34:42.068588+0000","last_became_peered":"2022-04-22T17:34:42.068588+0000","last_unstale":"2022-04-22T17:39:30.103344+0000","last_undegraded":"2022-04-22T17:39:30.103344+0000","last_fullsized":"2022-04-22T17:39:30.103344+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:24:51.871504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":818,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.044302+0000","last_change":"2022-04-22T17:33:59.975650+0000","last_active":"2022-04-22T17:39:30.044302+0000","last_peered":"2022-04-22T17:39:30.044302+0000","last_clean":"2022-04-22T17:39:30.044302+0000","last_became_active":"2022-04-22T17:33:59.975069+0000","last_became_peered":"2022-04-22T17:33:59.975069+0000","last_unstale":"2022-04-22T17:39:30.044302+0000","last_undegraded":"2022-04-22T17:39:30.044302+0000","last_fullsized":"2022-04-22T17:39:30.044302+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:27:03.602404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":820,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051837+0000","last_change":"2022-04-22T17:34:42.059379+0000","last_active":"2022-04-22T17:39:30.051837+0000","last_peered":"2022-04-22T17:39:30.051837+0000","last_clean":"2022-04-22T17:39:30.051837+0000","last_became_active":"2022-04-22T17:34:42.058974+0000","last_became_peered":"2022-04-22T17:34:42.058974+0000","last_unstale":"2022-04-22T17:39:30.051837+0000","last_undegraded":"2022-04-22T17:39:30.051837+0000","last_fullsized":"2022-04-22T17:39:30.051837+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:50:06.765464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"217'139","reported_seq":1037,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.744967+0000","last_change":"2022-04-22T17:34:42.071801+0000","last_active":"2022-04-22T17:39:27.744967+0000","last_peered":"2022-04-22T17:39:27.744967+0000","last_clean":"2022-04-22T17:39:27.744967+0000","last_became_active":"2022-04-22T17:34:42.071184+0000","last_became_peered":"2022-04-22T17:34:42.071184+0000","last_unstale":"2022-04-22T17:39:27.744967+0000","last_undegraded":"2022-04-22T17:39:27.744967+0000","last_fullsized":"2022-04-22T17:39:27.744967+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:19:29.157177+0000","last_clean_scrub_stamp":"2022-04-22T17:19:29.157177+0000","objects_scrubbed":0,"log_size":139,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:29:45.789104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":294,"num_read_kb":500,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"852'2488","reported_seq":4867,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.324657+0000","last_change":"2022-04-22T17:39:27.667265+0000","last_active":"2022-04-22T17:39:35.324657+0000","last_peered":"2022-04-22T17:39:35.324657+0000","last_clean":"2022-04-22T17:39:35.324657+0000","last_became_active":"2022-04-22T17:33:18.969835+0000","last_became_peered":"2022-04-22T17:33:18.969835+0000","last_unstale":"2022-04-22T17:39:35.324657+0000","last_undegraded":"2022-04-22T17:39:35.324657+0000","last_fullsized":"2022-04-22T17:39:35.324657+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":2488,"ondisk_log_size":2488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:51:18.997100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000711211,"stat_sum":{"num_bytes":42016768,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":429,"num_read_kb":675958,"num_write":972,"num_write_kb":250440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":60727296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113961+0000","last_change":"2022-04-22T17:31:28.843671+0000","last_active":"2022-04-22T17:39:30.113961+0000","last_peered":"2022-04-22T17:39:30.113961+0000","last_clean":"2022-04-22T17:39:30.113961+0000","last_became_active":"2022-04-22T17:31:28.843294+0000","last_became_peered":"2022-04-22T17:31:28.843294+0000","last_unstale":"2022-04-22T17:39:30.113961+0000","last_undegraded":"2022-04-22T17:39:30.113961+0000","last_fullsized":"2022-04-22T17:39:30.113961+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:31:33.751080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111340+0000","last_change":"2022-04-22T17:32:36.858622+0000","last_active":"2022-04-22T17:39:30.111340+0000","last_peered":"2022-04-22T17:39:30.111340+0000","last_clean":"2022-04-22T17:39:30.111340+0000","last_became_active":"2022-04-22T17:32:36.858274+0000","last_became_peered":"2022-04-22T17:32:36.858274+0000","last_unstale":"2022-04-22T17:39:30.111340+0000","last_undegraded":"2022-04-22T17:39:30.111340+0000","last_fullsized":"2022-04-22T17:39:30.111340+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:49:03.062593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":795,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.744667+0000","last_change":"2022-04-22T17:34:42.076391+0000","last_active":"2022-04-22T17:39:27.744667+0000","last_peered":"2022-04-22T17:39:27.744667+0000","last_clean":"2022-04-22T17:39:27.744667+0000","last_became_active":"2022-04-22T17:34:42.075339+0000","last_became_peered":"2022-04-22T17:34:42.075339+0000","last_unstale":"2022-04-22T17:39:27.744667+0000","last_undegraded":"2022-04-22T17:39:27.744667+0000","last_fullsized":"2022-04-22T17:39:27.744667+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:59:22.261405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"852'2165","reported_seq":4555,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.997091+0000","last_change":"2022-04-22T17:39:27.303682+0000","last_active":"2022-04-22T17:39:35.997091+0000","last_peered":"2022-04-22T17:39:35.997091+0000","last_clean":"2022-04-22T17:39:35.997091+0000","last_became_active":"2022-04-22T17:33:59.989237+0000","last_became_peered":"2022-04-22T17:33:59.989237+0000","last_unstale":"2022-04-22T17:39:35.997091+0000","last_undegraded":"2022-04-22T17:39:35.997091+0000","last_fullsized":"2022-04-22T17:39:35.997091+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":2165,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:58:14.231514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056596200000000004,"stat_sum":{"num_bytes":42827776,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":388,"num_read_kb":696978,"num_write":898,"num_write_kb":191688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5718016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111227+0000","last_change":"2022-04-22T17:34:42.044585+0000","last_active":"2022-04-22T17:39:30.111227+0000","last_peered":"2022-04-22T17:39:30.111227+0000","last_clean":"2022-04-22T17:39:30.111227+0000","last_became_active":"2022-04-22T17:34:42.042931+0000","last_became_peered":"2022-04-22T17:34:42.042931+0000","last_unstale":"2022-04-22T17:39:30.111227+0000","last_undegraded":"2022-04-22T17:39:30.111227+0000","last_fullsized":"2022-04-22T17:39:30.111227+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:54:27.602713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":807,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745744+0000","last_change":"2022-04-22T17:34:42.066566+0000","last_active":"2022-04-22T17:39:27.745744+0000","last_peered":"2022-04-22T17:39:27.745744+0000","last_clean":"2022-04-22T17:39:27.745744+0000","last_became_active":"2022-04-22T17:34:42.066093+0000","last_became_peered":"2022-04-22T17:34:42.066093+0000","last_unstale":"2022-04-22T17:39:27.745744+0000","last_undegraded":"2022-04-22T17:39:27.745744+0000","last_fullsized":"2022-04-22T17:39:27.745744+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:59:07.619623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045555+0000","last_change":"2022-04-22T17:33:18.951467+0000","last_active":"2022-04-22T17:39:30.045555+0000","last_peered":"2022-04-22T17:39:30.045555+0000","last_clean":"2022-04-22T17:39:30.045555+0000","last_became_active":"2022-04-22T17:33:18.951140+0000","last_became_peered":"2022-04-22T17:33:18.951140+0000","last_unstale":"2022-04-22T17:39:30.045555+0000","last_undegraded":"2022-04-22T17:39:30.045555+0000","last_fullsized":"2022-04-22T17:39:30.045555+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:36:26.343024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"853'1692","reported_seq":4157,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.055361+0000","last_change":"2022-04-22T17:39:27.087298+0000","last_active":"2022-04-22T17:39:35.055361+0000","last_peered":"2022-04-22T17:39:35.055361+0000","last_clean":"2022-04-22T17:39:35.055361+0000","last_became_active":"2022-04-22T17:34:42.058716+0000","last_became_peered":"2022-04-22T17:34:42.058716+0000","last_unstale":"2022-04-22T17:39:35.055361+0000","last_undegraded":"2022-04-22T17:39:35.055361+0000","last_fullsized":"2022-04-22T17:39:35.055361+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1692,"ondisk_log_size":1692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:32:59.584393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010645450000000001,"stat_sum":{"num_bytes":8060928,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":320,"num_read_kb":299284,"num_write":588,"num_write_kb":169480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":18866176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":798,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041101+0000","last_change":"2022-04-22T17:34:42.037951+0000","last_active":"2022-04-22T17:39:30.041101+0000","last_peered":"2022-04-22T17:39:30.041101+0000","last_clean":"2022-04-22T17:39:30.041101+0000","last_became_active":"2022-04-22T17:34:42.037073+0000","last_became_peered":"2022-04-22T17:34:42.037073+0000","last_unstale":"2022-04-22T17:39:30.041101+0000","last_undegraded":"2022-04-22T17:39:30.041101+0000","last_fullsized":"2022-04-22T17:39:30.041101+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:41:59.693025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"628'2","reported_seq":821,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.046069+0000","last_change":"2022-04-22T17:33:59.972843+0000","last_active":"2022-04-22T17:39:30.046069+0000","last_peered":"2022-04-22T17:39:30.046069+0000","last_clean":"2022-04-22T17:39:30.046069+0000","last_became_active":"2022-04-22T17:33:59.972438+0000","last_became_peered":"2022-04-22T17:33:59.972438+0000","last_unstale":"2022-04-22T17:39:30.046069+0000","last_undegraded":"2022-04-22T17:39:30.046069+0000","last_fullsized":"2022-04-22T17:39:30.046069+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:42:05.602565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":803,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041584+0000","last_change":"2022-04-22T17:34:42.042868+0000","last_active":"2022-04-22T17:39:30.041584+0000","last_peered":"2022-04-22T17:39:30.041584+0000","last_clean":"2022-04-22T17:39:30.041584+0000","last_became_active":"2022-04-22T17:34:42.042533+0000","last_became_peered":"2022-04-22T17:34:42.042533+0000","last_unstale":"2022-04-22T17:39:30.041584+0000","last_undegraded":"2022-04-22T17:39:30.041584+0000","last_fullsized":"2022-04-22T17:39:30.041584+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T17:51:21.317599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"850'533","reported_seq":3111,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.046644+0000","last_change":"2022-04-22T17:39:27.074873+0000","last_active":"2022-04-22T17:39:35.046644+0000","last_peered":"2022-04-22T17:39:35.046644+0000","last_clean":"2022-04-22T17:39:35.046644+0000","last_became_active":"2022-04-22T17:32:36.858288+0000","last_became_peered":"2022-04-22T17:32:36.858288+0000","last_unstale":"2022-04-22T17:39:35.046644+0000","last_undegraded":"2022-04-22T17:39:35.046644+0000","last_fullsized":"2022-04-22T17:39:35.046644+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":533,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T21:17:22.145163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017470400000000001,"stat_sum":{"num_bytes":12140544,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":215953,"num_write":196,"num_write_kb":42984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12599296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":815,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.120388+0000","last_change":"2022-04-22T17:33:59.984870+0000","last_active":"2022-04-22T17:39:30.120388+0000","last_peered":"2022-04-22T17:39:30.120388+0000","last_clean":"2022-04-22T17:39:30.120388+0000","last_became_active":"2022-04-22T17:33:59.984537+0000","last_became_peered":"2022-04-22T17:33:59.984537+0000","last_unstale":"2022-04-22T17:39:30.120388+0000","last_undegraded":"2022-04-22T17:39:30.120388+0000","last_fullsized":"2022-04-22T17:39:30.120388+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:23:10.139429+0000","last_clean_scrub_stamp":"2022-04-22T17:23:10.139429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:32:07.280280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"53'2","reported_seq":820,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.103331+0000","last_change":"2022-04-22T17:34:42.069074+0000","last_active":"2022-04-22T17:39:30.103331+0000","last_peered":"2022-04-22T17:39:30.103331+0000","last_clean":"2022-04-22T17:39:30.103331+0000","last_became_active":"2022-04-22T17:34:42.068594+0000","last_became_peered":"2022-04-22T17:34:42.068594+0000","last_unstale":"2022-04-22T17:39:30.103331+0000","last_undegraded":"2022-04-22T17:39:30.103331+0000","last_fullsized":"2022-04-22T17:39:30.103331+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:13:16.309599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113201+0000","last_change":"2022-04-22T17:32:36.860894+0000","last_active":"2022-04-22T17:39:30.113201+0000","last_peered":"2022-04-22T17:39:30.113201+0000","last_clean":"2022-04-22T17:39:30.113201+0000","last_became_active":"2022-04-22T17:32:36.860598+0000","last_became_peered":"2022-04-22T17:32:36.860598+0000","last_unstale":"2022-04-22T17:39:30.113201+0000","last_undegraded":"2022-04-22T17:39:30.113201+0000","last_fullsized":"2022-04-22T17:39:30.113201+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:51:22.982975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2488,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.324043+0000","last_change":"2022-04-22T17:39:27.666991+0000","last_active":"2022-04-22T17:39:35.324043+0000","last_peered":"2022-04-22T17:39:35.324043+0000","last_clean":"2022-04-22T17:39:35.324043+0000","last_became_active":"2022-04-22T17:33:59.984833+0000","last_became_peered":"2022-04-22T17:33:59.984833+0000","last_unstale":"2022-04-22T17:39:35.324043+0000","last_undegraded":"2022-04-22T17:39:35.324043+0000","last_fullsized":"2022-04-22T17:39:35.324043+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:53:57.165623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072794199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":795,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.744783+0000","last_change":"2022-04-22T17:34:42.077470+0000","last_active":"2022-04-22T17:39:27.744783+0000","last_peered":"2022-04-22T17:39:27.744783+0000","last_clean":"2022-04-22T17:39:27.744783+0000","last_became_active":"2022-04-22T17:34:42.077148+0000","last_became_peered":"2022-04-22T17:34:42.077148+0000","last_unstale":"2022-04-22T17:39:27.744783+0000","last_undegraded":"2022-04-22T17:39:27.744783+0000","last_fullsized":"2022-04-22T17:39:27.744783+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T20:01:31.644216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":823,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.120218+0000","last_change":"2022-04-22T17:34:42.060377+0000","last_active":"2022-04-22T17:39:30.120218+0000","last_peered":"2022-04-22T17:39:30.120218+0000","last_clean":"2022-04-22T17:39:30.120218+0000","last_became_active":"2022-04-22T17:34:42.057774+0000","last_became_peered":"2022-04-22T17:34:42.057774+0000","last_unstale":"2022-04-22T17:39:30.120218+0000","last_undegraded":"2022-04-22T17:39:30.120218+0000","last_fullsized":"2022-04-22T17:39:30.120218+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:10:45.669842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"852'1230","reported_seq":3689,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.055136+0000","last_change":"2022-04-22T17:39:27.088762+0000","last_active":"2022-04-22T17:39:35.055136+0000","last_peered":"2022-04-22T17:39:35.055136+0000","last_clean":"2022-04-22T17:39:35.055136+0000","last_became_active":"2022-04-22T17:33:18.960331+0000","last_became_peered":"2022-04-22T17:33:18.960331+0000","last_unstale":"2022-04-22T17:39:35.055136+0000","last_undegraded":"2022-04-22T17:39:35.055136+0000","last_fullsized":"2022-04-22T17:39:35.055136+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1230,"ondisk_log_size":1230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:44:29.414204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011688060000000001,"stat_sum":{"num_bytes":21921792,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":210,"num_read_kb":404694,"num_write":619,"num_write_kb":146336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045357+0000","last_change":"2022-04-22T17:34:42.056885+0000","last_active":"2022-04-22T17:39:30.045357+0000","last_peered":"2022-04-22T17:39:30.045357+0000","last_clean":"2022-04-22T17:39:30.045357+0000","last_became_active":"2022-04-22T17:34:42.056517+0000","last_became_peered":"2022-04-22T17:34:42.056517+0000","last_unstale":"2022-04-22T17:39:30.045357+0000","last_undegraded":"2022-04-22T17:39:30.045357+0000","last_fullsized":"2022-04-22T17:39:30.045357+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:02:33.568199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":802,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041264+0000","last_change":"2022-04-22T17:34:42.038174+0000","last_active":"2022-04-22T17:39:30.041264+0000","last_peered":"2022-04-22T17:39:30.041264+0000","last_clean":"2022-04-22T17:39:30.041264+0000","last_became_active":"2022-04-22T17:34:42.037609+0000","last_became_peered":"2022-04-22T17:34:42.037609+0000","last_unstale":"2022-04-22T17:39:30.041264+0000","last_undegraded":"2022-04-22T17:39:30.041264+0000","last_fullsized":"2022-04-22T17:39:30.041264+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:42:30.111519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"852'1612","reported_seq":4097,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.866243+0000","last_change":"2022-04-22T17:39:27.769770+0000","last_active":"2022-04-22T17:39:35.866243+0000","last_peered":"2022-04-22T17:39:35.866243+0000","last_clean":"2022-04-22T17:39:35.866243+0000","last_became_active":"2022-04-22T17:34:42.044059+0000","last_became_peered":"2022-04-22T17:34:42.044059+0000","last_unstale":"2022-04-22T17:39:35.866243+0000","last_undegraded":"2022-04-22T17:39:35.866243+0000","last_fullsized":"2022-04-22T17:39:35.866243+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1612,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:52:22.538098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089125,"stat_sum":{"num_bytes":8044544,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":305,"num_read_kb":485239,"num_write":736,"num_write_kb":125272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4431872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045302+0000","last_change":"2022-04-22T17:34:42.047111+0000","last_active":"2022-04-22T17:39:30.045302+0000","last_peered":"2022-04-22T17:39:30.045302+0000","last_clean":"2022-04-22T17:39:30.045302+0000","last_became_active":"2022-04-22T17:34:42.046776+0000","last_became_peered":"2022-04-22T17:34:42.046776+0000","last_unstale":"2022-04-22T17:39:30.045302+0000","last_undegraded":"2022-04-22T17:39:30.045302+0000","last_fullsized":"2022-04-22T17:39:30.045302+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:40:12.628855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":836,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041644+0000","last_change":"2022-04-22T17:32:36.858445+0000","last_active":"2022-04-22T17:39:30.041644+0000","last_peered":"2022-04-22T17:39:30.041644+0000","last_clean":"2022-04-22T17:39:30.041644+0000","last_became_active":"2022-04-22T17:32:36.858120+0000","last_became_peered":"2022-04-22T17:32:36.858120+0000","last_unstale":"2022-04-22T17:39:30.041644+0000","last_undegraded":"2022-04-22T17:39:30.041644+0000","last_fullsized":"2022-04-22T17:39:30.041644+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:37:31.920744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"850'845","reported_seq":3349,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.365624+0000","last_change":"2022-04-22T17:39:27.081567+0000","last_active":"2022-04-22T17:39:35.365624+0000","last_peered":"2022-04-22T17:39:35.365624+0000","last_clean":"2022-04-22T17:39:35.365624+0000","last_became_active":"2022-04-22T17:32:36.862551+0000","last_became_peered":"2022-04-22T17:32:36.862551+0000","last_unstale":"2022-04-22T17:39:35.365624+0000","last_undegraded":"2022-04-22T17:39:35.365624+0000","last_fullsized":"2022-04-22T17:39:35.365624+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":845,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:42:08.563991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605623,"stat_sum":{"num_bytes":22462464,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":133,"num_read_kb":336766,"num_write":368,"num_write_kb":101672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":35250176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"621'3","reported_seq":811,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745363+0000","last_change":"2022-04-22T17:34:42.068216+0000","last_active":"2022-04-22T17:39:27.745363+0000","last_peered":"2022-04-22T17:39:27.745363+0000","last_clean":"2022-04-22T17:39:27.745363+0000","last_became_active":"2022-04-22T17:34:42.067708+0000","last_became_peered":"2022-04-22T17:34:42.067708+0000","last_unstale":"2022-04-22T17:39:27.745363+0000","last_undegraded":"2022-04-22T17:39:27.745363+0000","last_fullsized":"2022-04-22T17:39:27.745363+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:02:49.881782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":821,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111293+0000","last_change":"2022-04-22T17:32:36.862852+0000","last_active":"2022-04-22T17:39:30.111293+0000","last_peered":"2022-04-22T17:39:30.111293+0000","last_clean":"2022-04-22T17:39:30.111293+0000","last_became_active":"2022-04-22T17:32:36.862455+0000","last_became_peered":"2022-04-22T17:32:36.862455+0000","last_unstale":"2022-04-22T17:39:30.111293+0000","last_undegraded":"2022-04-22T17:39:30.111293+0000","last_fullsized":"2022-04-22T17:39:30.111293+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:08:24.652632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"850'333","reported_seq":2865,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.865567+0000","last_change":"2022-04-22T17:39:27.769241+0000","last_active":"2022-04-22T17:39:35.865567+0000","last_peered":"2022-04-22T17:39:35.865567+0000","last_clean":"2022-04-22T17:39:35.865567+0000","last_became_active":"2022-04-22T17:33:18.939077+0000","last_became_peered":"2022-04-22T17:33:18.939077+0000","last_unstale":"2022-04-22T17:39:35.865567+0000","last_undegraded":"2022-04-22T17:39:35.865567+0000","last_fullsized":"2022-04-22T17:39:35.865567+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:15:40.506874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011048690000000001,"stat_sum":{"num_bytes":25255936,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":106398,"num_write":156,"num_write_kb":47968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3915776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":825,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.113335+0000","last_change":"2022-04-22T17:33:18.948304+0000","last_active":"2022-04-22T17:39:30.113335+0000","last_peered":"2022-04-22T17:39:30.113335+0000","last_clean":"2022-04-22T17:39:30.113335+0000","last_became_active":"2022-04-22T17:33:18.947947+0000","last_became_peered":"2022-04-22T17:33:18.947947+0000","last_unstale":"2022-04-22T17:39:30.113335+0000","last_undegraded":"2022-04-22T17:39:30.113335+0000","last_fullsized":"2022-04-22T17:39:30.113335+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:54:21.455506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":805,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.745945+0000","last_change":"2022-04-22T17:34:42.075266+0000","last_active":"2022-04-22T17:39:27.745945+0000","last_peered":"2022-04-22T17:39:27.745945+0000","last_clean":"2022-04-22T17:39:27.745945+0000","last_became_active":"2022-04-22T17:34:42.074621+0000","last_became_peered":"2022-04-22T17:34:42.074621+0000","last_unstale":"2022-04-22T17:39:27.745945+0000","last_undegraded":"2022-04-22T17:39:27.745945+0000","last_fullsized":"2022-04-22T17:39:27.745945+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T04:32:16.264838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"852'1584","reported_seq":4089,"reported_epoch":857,"state":"active+clean","last_fresh":"2022-04-22T17:39:35.865242+0000","last_change":"2022-04-22T17:39:27.770197+0000","last_active":"2022-04-22T17:39:35.865242+0000","last_peered":"2022-04-22T17:39:35.865242+0000","last_clean":"2022-04-22T17:39:35.865242+0000","last_became_active":"2022-04-22T17:32:36.860105+0000","last_became_peered":"2022-04-22T17:32:36.860105+0000","last_unstale":"2022-04-22T17:39:35.865242+0000","last_undegraded":"2022-04-22T17:39:35.865242+0000","last_fullsized":"2022-04-22T17:39:35.865242+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":1584,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:03:22.440083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001406877,"stat_sum":{"num_bytes":10420224,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":310,"num_read_kb":350639,"num_write":478,"num_write_kb":163840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":75677696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"53'1","reported_seq":819,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045624+0000","last_change":"2022-04-22T17:32:36.872902+0000","last_active":"2022-04-22T17:39:30.045624+0000","last_peered":"2022-04-22T17:39:30.045624+0000","last_clean":"2022-04-22T17:39:30.045624+0000","last_became_active":"2022-04-22T17:32:36.872512+0000","last_became_peered":"2022-04-22T17:32:36.872512+0000","last_unstale":"2022-04-22T17:39:30.045624+0000","last_undegraded":"2022-04-22T17:39:30.045624+0000","last_fullsized":"2022-04-22T17:39:30.045624+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T19:41:42.669836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":805,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052627+0000","last_change":"2022-04-22T17:33:59.979980+0000","last_active":"2022-04-22T17:39:30.052627+0000","last_peered":"2022-04-22T17:39:30.052627+0000","last_clean":"2022-04-22T17:39:30.052627+0000","last_became_active":"2022-04-22T17:33:59.979680+0000","last_became_peered":"2022-04-22T17:33:59.979680+0000","last_unstale":"2022-04-22T17:39:30.052627+0000","last_undegraded":"2022-04-22T17:39:30.052627+0000","last_fullsized":"2022-04-22T17:39:30.052627+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:40:25.335273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"852'2098","reported_seq":4477,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.123992+0000","last_change":"2022-04-22T17:39:27.747254+0000","last_active":"2022-04-22T17:39:30.123992+0000","last_peered":"2022-04-22T17:39:30.123992+0000","last_clean":"2022-04-22T17:39:30.123992+0000","last_became_active":"2022-04-22T17:34:42.075848+0000","last_became_peered":"2022-04-22T17:34:42.075848+0000","last_unstale":"2022-04-22T17:39:30.123992+0000","last_undegraded":"2022-04-22T17:39:30.123992+0000","last_fullsized":"2022-04-22T17:39:30.123992+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:25:32.218030+0000","last_clean_scrub_stamp":"2022-04-22T17:25:32.218030+0000","objects_scrubbed":0,"log_size":2098,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:13:54.237153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014592819999999999,"stat_sum":{"num_bytes":40747008,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":347,"num_read_kb":363606,"num_write":826,"num_write_kb":202536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":28811264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"302","length":"1"},{"start":"30b","length":"1"}]},{"pgid":"2.e","version":"53'2","reported_seq":804,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111960+0000","last_change":"2022-04-22T17:34:42.041183+0000","last_active":"2022-04-22T17:39:30.111960+0000","last_peered":"2022-04-22T17:39:30.111960+0000","last_clean":"2022-04-22T17:39:30.111960+0000","last_became_active":"2022-04-22T17:34:42.040791+0000","last_became_peered":"2022-04-22T17:34:42.040791+0000","last_unstale":"2022-04-22T17:39:30.111960+0000","last_undegraded":"2022-04-22T17:39:30.111960+0000","last_fullsized":"2022-04-22T17:39:30.111960+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:17.074592+0000","last_clean_scrub_stamp":"2022-04-22T17:22:17.074592+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T03:07:43.767282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":806,"reported_epoch":855,"state":"active+clean","last_fresh":"2022-04-22T17:39:27.746882+0000","last_change":"2022-04-22T17:34:42.063996+0000","last_active":"2022-04-22T17:39:27.746882+0000","last_peered":"2022-04-22T17:39:27.746882+0000","last_clean":"2022-04-22T17:39:27.746882+0000","last_became_active":"2022-04-22T17:34:42.063393+0000","last_became_peered":"2022-04-22T17:34:42.063393+0000","last_unstale":"2022-04-22T17:39:27.746882+0000","last_undegraded":"2022-04-22T17:39:27.746882+0000","last_fullsized":"2022-04-22T17:39:27.746882+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T23:00:03.793724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":798,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.101648+0000","last_change":"2022-04-22T17:33:59.988646+0000","last_active":"2022-04-22T17:39:30.101648+0000","last_peered":"2022-04-22T17:39:30.101648+0000","last_clean":"2022-04-22T17:39:30.101648+0000","last_became_active":"2022-04-22T17:33:59.986296+0000","last_became_peered":"2022-04-22T17:33:59.986296+0000","last_unstale":"2022-04-22T17:39:30.101648+0000","last_undegraded":"2022-04-22T17:39:30.101648+0000","last_fullsized":"2022-04-22T17:39:30.101648+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:46:08.362996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.110935+0000","last_change":"2022-04-22T17:33:59.976631+0000","last_active":"2022-04-22T17:39:30.110935+0000","last_peered":"2022-04-22T17:39:30.110935+0000","last_clean":"2022-04-22T17:39:30.110935+0000","last_became_active":"2022-04-22T17:33:59.976180+0000","last_became_peered":"2022-04-22T17:33:59.976180+0000","last_unstale":"2022-04-22T17:39:30.110935+0000","last_undegraded":"2022-04-22T17:39:30.110935+0000","last_fullsized":"2022-04-22T17:39:30.110935+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:35:56.256460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":803,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.052234+0000","last_change":"2022-04-22T17:33:18.960704+0000","last_active":"2022-04-22T17:39:30.052234+0000","last_peered":"2022-04-22T17:39:30.052234+0000","last_clean":"2022-04-22T17:39:30.052234+0000","last_became_active":"2022-04-22T17:33:18.959187+0000","last_became_peered":"2022-04-22T17:33:18.959187+0000","last_unstale":"2022-04-22T17:39:30.052234+0000","last_undegraded":"2022-04-22T17:39:30.052234+0000","last_fullsized":"2022-04-22T17:39:30.052234+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:55:05.555015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.112917+0000","last_change":"2022-04-22T17:32:36.859733+0000","last_active":"2022-04-22T17:39:30.112917+0000","last_peered":"2022-04-22T17:39:30.112917+0000","last_clean":"2022-04-22T17:39:30.112917+0000","last_became_active":"2022-04-22T17:32:36.859325+0000","last_became_peered":"2022-04-22T17:32:36.859325+0000","last_unstale":"2022-04-22T17:39:30.112917+0000","last_undegraded":"2022-04-22T17:39:30.112917+0000","last_fullsized":"2022-04-22T17:39:30.112917+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:33:30.028574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":803,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.042726+0000","last_change":"2022-04-22T17:32:36.857619+0000","last_active":"2022-04-22T17:39:30.042726+0000","last_peered":"2022-04-22T17:39:30.042726+0000","last_clean":"2022-04-22T17:39:30.042726+0000","last_became_active":"2022-04-22T17:32:36.857292+0000","last_became_peered":"2022-04-22T17:32:36.857292+0000","last_unstale":"2022-04-22T17:39:30.042726+0000","last_undegraded":"2022-04-22T17:39:30.042726+0000","last_fullsized":"2022-04-22T17:39:30.042726+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T18:59:24.920617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":816,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.045709+0000","last_change":"2022-04-22T17:31:55.893519+0000","last_active":"2022-04-22T17:39:30.045709+0000","last_peered":"2022-04-22T17:39:30.045709+0000","last_clean":"2022-04-22T17:39:30.045709+0000","last_became_active":"2022-04-22T17:31:55.893170+0000","last_became_peered":"2022-04-22T17:31:55.893170+0000","last_unstale":"2022-04-22T17:39:30.045709+0000","last_undegraded":"2022-04-22T17:39:30.045709+0000","last_fullsized":"2022-04-22T17:39:30.045709+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:57:57.471269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":801,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.103337+0000","last_change":"2022-04-22T17:34:42.069080+0000","last_active":"2022-04-22T17:39:30.103337+0000","last_peered":"2022-04-22T17:39:30.103337+0000","last_clean":"2022-04-22T17:39:30.103337+0000","last_became_active":"2022-04-22T17:34:42.068527+0000","last_became_peered":"2022-04-22T17:34:42.068527+0000","last_unstale":"2022-04-22T17:39:30.103337+0000","last_undegraded":"2022-04-22T17:39:30.103337+0000","last_fullsized":"2022-04-22T17:39:30.103337+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:55:15.669402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":827,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041681+0000","last_change":"2022-04-22T17:32:36.863141+0000","last_active":"2022-04-22T17:39:30.041681+0000","last_peered":"2022-04-22T17:39:30.041681+0000","last_clean":"2022-04-22T17:39:30.041681+0000","last_became_active":"2022-04-22T17:32:36.862833+0000","last_became_peered":"2022-04-22T17:32:36.862833+0000","last_unstale":"2022-04-22T17:39:30.041681+0000","last_undegraded":"2022-04-22T17:39:30.041681+0000","last_fullsized":"2022-04-22T17:39:30.041681+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-23T22:43:23.554309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":803,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051414+0000","last_change":"2022-04-22T17:33:18.948262+0000","last_active":"2022-04-22T17:39:30.051414+0000","last_peered":"2022-04-22T17:39:30.051414+0000","last_clean":"2022-04-22T17:39:30.051414+0000","last_became_active":"2022-04-22T17:33:18.947103+0000","last_became_peered":"2022-04-22T17:33:18.947103+0000","last_unstale":"2022-04-22T17:39:30.051414+0000","last_undegraded":"2022-04-22T17:39:30.051414+0000","last_fullsized":"2022-04-22T17:39:30.051414+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:11:08.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":802,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.041416+0000","last_change":"2022-04-22T17:32:36.860489+0000","last_active":"2022-04-22T17:39:30.041416+0000","last_peered":"2022-04-22T17:39:30.041416+0000","last_clean":"2022-04-22T17:39:30.041416+0000","last_became_active":"2022-04-22T17:32:36.860125+0000","last_became_peered":"2022-04-22T17:32:36.860125+0000","last_unstale":"2022-04-22T17:39:30.041416+0000","last_undegraded":"2022-04-22T17:39:30.041416+0000","last_fullsized":"2022-04-22T17:39:30.041416+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T00:56:48.649014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":810,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.051428+0000","last_change":"2022-04-22T17:34:42.059387+0000","last_active":"2022-04-22T17:39:30.051428+0000","last_peered":"2022-04-22T17:39:30.051428+0000","last_clean":"2022-04-22T17:39:30.051428+0000","last_became_active":"2022-04-22T17:34:42.059025+0000","last_became_peered":"2022-04-22T17:34:42.059025+0000","last_unstale":"2022-04-22T17:39:30.051428+0000","last_undegraded":"2022-04-22T17:39:30.051428+0000","last_fullsized":"2022-04-22T17:39:30.051428+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T05:12:07.398319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":826,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.111550+0000","last_change":"2022-04-22T17:34:42.043845+0000","last_active":"2022-04-22T17:39:30.111550+0000","last_peered":"2022-04-22T17:39:30.111550+0000","last_clean":"2022-04-22T17:39:30.111550+0000","last_became_active":"2022-04-22T17:34:42.042446+0000","last_became_peered":"2022-04-22T17:34:42.042446+0000","last_unstale":"2022-04-22T17:39:30.111550+0000","last_undegraded":"2022-04-22T17:39:30.111550+0000","last_fullsized":"2022-04-22T17:39:30.111550+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T01:14:14.199067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2022-04-22T17:39:30.102890+0000","last_change":"2022-04-22T17:33:59.976346+0000","last_active":"2022-04-22T17:39:30.102890+0000","last_peered":"2022-04-22T17:39:30.102890+0000","last_clean":"2022-04-22T17:39:30.102890+0000","last_became_active":"2022-04-22T17:33:59.975940+0000","last_became_peered":"2022-04-22T17:33:59.975940+0000","last_unstale":"2022-04-22T17:39:30.102890+0000","last_undegraded":"2022-04-22T17:39:30.102890+0000","last_fullsized":"2022-04-22T17:39:30.102890+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-22T17:22:18.078190+0000","last_clean_scrub_stamp":"2022-04-22T17:22:18.078190+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-24T02:21:09.397407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":322297856,"num_objects":84,"num_object_clones":40,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":7,"num_read":3794,"num_read_kb":5938778,"num_write":8449,"num_write_kb":2045768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":339558400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483446784,"data_stored":732672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20980,"ondisk_log_size":20980,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":294,"num_read_kb":500,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5510,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":79,"num_read_kb":84,"num_write":62,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76498,"internal_metadata":0},"log_size":62,"ondisk_log_size":62,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":583,"seq":2503965933628,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186456,"kb_used_data":48912,"kb_used_omap":134,"kb_used_meta":137401,"kb_avail":93558696,"statfs":{"total":95995035648,"available":95804104704,"internally_reserved":0,"allocated":50085888,"data_stored":65648110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138134,"internal_metadata":140698730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":0.93100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.131,"5min":1.869,"15min":1.869},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.89600000000000002,"15min":0.94399999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.1379999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.877,"15min":0.88},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.2509999999999999,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96499999999999997,"15min":1.0029999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2729999999999999,"5min":1.786,"15min":1.786},"last":1.1659999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.96799999999999997,"15min":1.0780000000000001},"min":{"1min":0.48199999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.056,"5min":4.0270000000000001,"15min":4.0270000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95199999999999996,"15min":0.97599999999999998},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.202,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.91900000000000004}]},{"osd":3,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.98299999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.999},{"interface":"front","average":{"1min":0.995,"5min":1.024,"15min":1.0529999999999999},"min":{"1min":0.68300000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.3480000000000001,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1100000000000001}]},{"osd":4,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.999,"15min":1.0069999999999999},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2909999999999999,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.254},{"interface":"front","average":{"1min":1.083,"5min":1.0640000000000001,"15min":1.083},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":1.236}]},{"osd":5,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1060000000000001,"15min":1.1379999999999999},"min":{"1min":0.51300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.3400000000000001,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.052},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.115,"15min":1.145},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.788,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0760000000000001}]},{"osd":6,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.155,"15min":1.175},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.9039999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.125,"15min":1.145},"min":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.4179999999999999,"5min":1.762,"15min":1.762},"last":0.96399999999999997}]}]},{"osd":6,"up_from":542,"seq":2327872274501,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195940,"kb_used_data":62108,"kb_used_omap":86,"kb_used_meta":133737,"kb_avail":93549212,"statfs":{"total":95995035648,"available":95794393088,"internally_reserved":0,"allocated":63598592,"data_stored":88031072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88978,"internal_metadata":136946798},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.876,"15min":0.84899999999999998},"min":{"1min":0.64200000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.486,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96099999999999997,"15min":0.90400000000000003},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.403,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.92500000000000004}]},{"osd":1,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96599999999999997,"15min":0.94499999999999995},"min":{"1min":0.56799999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.401,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.153},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.878,"15min":0.83799999999999997},"min":{"1min":0.69099999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1499999999999999,"5min":1.385,"15min":1.385},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.90600000000000003,"15min":0.88300000000000001},"min":{"1min":0.502,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3799999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0389999999999999,"15min":0.98999999999999999},"min":{"1min":0.76300000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.419,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.0740000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0569999999999999,"15min":0.96099999999999997},"min":{"1min":0.80700000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5349999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.028},{"interface":"front","average":{"1min":1.0509999999999999,"5min":0.95499999999999996,"15min":0.90100000000000002},"min":{"1min":0.69599999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.284,"5min":1.595,"15min":1.595},"last":0.82499999999999996}]},{"osd":4,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.94099999999999995,"15min":0.88700000000000001},"min":{"1min":0.55200000000000005,"5min":0.375,"15min":0.375},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.181,"5min":1.097,"15min":1.0249999999999999},"min":{"1min":0.91300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5649999999999999,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.1140000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:39:04 2022","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0149999999999999,"15min":0.97399999999999998},"min":{"1min":0.60399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.385,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":0.755},{"interface":"front","average":{"1min":1.167,"5min":1.0680000000000001,"15min":1.018},"min":{"1min":0.83199999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6000000000000001,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":0.88500000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:38:52 2022","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.1919999999999999,"15min":1.2},"min":{"1min":0.93200000000000005,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.518,"5min":2.895,"15min":2.895},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.1240000000000001,"15min":1.1359999999999999},"min":{"1min":0.996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.7609999999999999,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.96899999999999997}]}]},{"osd":1,"up_from":366,"seq":1571958030440,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":375620,"kb_used_data":86908,"kb_used_omap":85,"kb_used_meta":288618,"kb_avail":93369532,"statfs":{"total":95995035648,"available":95610400768,"internally_reserved":0,"allocated":88993792,"data_stored":135781212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87952,"internal_metadata":295544944},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:39:12 2022","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.66500000000000004,"15min":0.621},"min":{"1min":0.54900000000000004,"5min":0.38500000000000001,"15min":0.252},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":2.6739999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74099999999999999,"15min":0.71399999999999997},"min":{"1min":0.57999999999999996,"5min":0.40200000000000002,"15min":0.26700000000000002},"max":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":2.5649999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Apr 22 17:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.754,"15min":0.73399999999999999},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.24,"5min":1.5840000000000001,"15min":1.8919999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.82199999999999995,"15min":0.79100000000000004},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.256,"5min":1.256,"15min":2.0179999999999998},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Apr 22 17:39:12 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.78500000000000003,"15min":0.76500000000000001},"min":{"1min":0.58099999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.663},"last":1.149},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.97099999999999997,"15min":0.95099999999999996},"min":{"1min":0.80700000000000005,"5min":0.60199999999999998,"15min":0.53800000000000003},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.7130000000000001},"last":1.2130000000000001}]},{"osd":4,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.0229999999999999},"min":{"1min":0.78800000000000003,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6779999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.002,"15min":0.97799999999999998},"min":{"1min":0.76100000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2430000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.99299999999999999}]},{"osd":5,"last update":"Fri Apr 22 17:39:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.052,"15min":1.038},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.65900000000000003},"max":{"1min":1.4390000000000001,"5min":2.452,"15min":2.452},"last":1.234},{"interface":"front","average":{"1min":1.171,"5min":1.137,"15min":1.1220000000000001},"min":{"1min":0.90700000000000003,"5min":0.72799999999999998,"15min":0.621},"max":{"1min":1.716,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.411}]},{"osd":6,"last update":"Fri Apr 22 17:39:12 2022","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.131,"15min":1.169},"min":{"1min":0.97399999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.768,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1120000000000001,"15min":1.079},"min":{"1min":0.88600000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.627,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.387}]},{"osd":7,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.167,"15min":1.1259999999999999},"min":{"1min":0.91600000000000004,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.5329999999999999,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":1.262},{"interface":"front","average":{"1min":1.222,"5min":1.2010000000000001,"15min":1.171},"min":{"1min":0.96999999999999997,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.5940000000000001,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.444}]}]},{"osd":0,"up_from":325,"seq":1395864371313,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":454736,"kb_used_data":110920,"kb_used_omap":97,"kb_used_meta":343710,"kb_avail":93290416,"statfs":{"total":95995035648,"available":95529385984,"internally_reserved":0,"allocated":113582080,"data_stored":153882520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99345,"internal_metadata":351960047},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 22 17:39:17 2022","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.754,"15min":0.71999999999999997},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77500000000000002,"15min":0.72499999999999998},"min":{"1min":0.52400000000000002,"5min":0.34300000000000003,"15min":0.32400000000000001},"max":{"1min":0.95499999999999996,"5min":1.859,"15min":1.859},"last":0.50700000000000001}]},{"osd":2,"last update":"Fri Apr 22 17:38:44 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82799999999999996,"15min":0.86599999999999999},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.929,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.876,"15min":0.84899999999999998},"min":{"1min":0.62,"5min":0.46600000000000003,"15min":0.30599999999999999},"max":{"1min":2.8809999999999998,"5min":3.6190000000000002,"15min":3.6190000000000002},"last":0.67600000000000005}]},{"osd":3,"last update":"Fri Apr 22 17:39:08 2022","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78100000000000003,"15min":0.81100000000000005},"min":{"1min":0.49199999999999999,"5min":0.46600000000000003,"15min":0.41699999999999998},"max":{"1min":0.98899999999999999,"5min":1.8440000000000001,"15min":6.7359999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.91100000000000003},"min":{"1min":0.76600000000000001,"5min":0.497,"15min":0.30099999999999999},"max":{"1min":1.244,"5min":2.0019999999999998,"15min":2.4289999999999998},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.1779999999999999,"15min":1.0860000000000001},"min":{"1min":0.72299999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":9.6069999999999993,"5min":9.6069999999999993,"15min":9.6069999999999993},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1739999999999999,"15min":1.079},"min":{"1min":0.77000000000000002,"5min":0.48899999999999999,"15min":0.46300000000000002},"max":{"1min":3.556,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Apr 22 17:39:34 2022","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1279999999999999,"15min":1.099},"min":{"1min":0.91900000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.8069999999999999,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.04,"5min":1.1000000000000001,"15min":1.304},"min":{"1min":0.65300000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.758,"5min":2.0070000000000001,"15min":7.0789999999999997},"last":0.93300000000000005}]},{"osd":6,"last update":"Fri Apr 22 17:39:08 2022","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.151,"15min":1.1259999999999999},"min":{"1min":0.88200000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5109999999999999,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.163,"15min":1.145},"min":{"1min":0.84399999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6579999999999999,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.89100000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:38:50 2022","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.2430000000000001,"15min":1.1970000000000001},"min":{"1min":0.71699999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":3.4049999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.044},{"interface":"front","average":{"1min":1.393,"5min":1.2490000000000001,"15min":1.2010000000000001},"min":{"1min":1.016,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"last":0.96499999999999997}]}]},{"osd":2,"up_from":392,"seq":1683627180131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300216,"kb_used_data":56880,"kb_used_omap":74,"kb_used_meta":243253,"kb_avail":93444936,"statfs":{"total":95995035648,"available":95687614464,"internally_reserved":0,"allocated":58245120,"data_stored":82418364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76284,"internal_metadata":249091588},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:38:44 2022","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69799999999999995,"15min":0.83299999999999996},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22600000000000001},"max":{"1min":0.94699999999999995,"5min":2.7290000000000001,"15min":5.6379999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.012,"5min":0.80700000000000005,"15min":0.92100000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":5.5700000000000003,"5min":5.5700000000000003,"15min":5.5700000000000003},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Apr 22 17:38:44 2022","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.79300000000000004,"15min":0.89400000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.2330000000000005,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.625},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.85399999999999998,"15min":0.79100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32200000000000001},"max":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Apr 22 17:39:07 2022","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.91400000000000003,"15min":1.0660000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1220000000000001,"5min":8.2899999999999991,"15min":8.2899999999999991},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85999999999999999,"15min":0.86799999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.41299999999999998},"max":{"1min":1.131,"5min":1.252,"15min":2.851},"last":0.92300000000000004}]},{"osd":4,"last update":"Fri Apr 22 17:38:52 2022","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":0.98499999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.46200000000000002},"max":{"1min":1.6899999999999999,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.754},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.1539999999999999,"15min":1.0980000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.438},"max":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"last":1.0860000000000001}]},{"osd":5,"last update":"Fri Apr 22 17:39:33 2022","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0129999999999999,"15min":1.04},"min":{"1min":0.621,"5min":0.53200000000000003,"15min":0.46600000000000003},"max":{"1min":1.3240000000000001,"5min":1.6519999999999999,"15min":2.5569999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.196,"15min":1.091},"min":{"1min":0.59799999999999998,"5min":0.45300000000000001,"15min":0.38600000000000001},"max":{"1min":1.429,"5min":8.4499999999999993,"15min":8.4499999999999993},"last":1.03}]},{"osd":6,"last update":"Fri Apr 22 17:39:07 2022","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.286,"15min":1.137},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":11.077999999999999},"last":1.246},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.2390000000000001,"15min":1.155},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":1.272}]},{"osd":7,"last update":"Fri Apr 22 17:38:44 2022","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.3129999999999999,"15min":1.266},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":8.5340000000000007,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":1.165},{"interface":"front","average":{"1min":1.085,"5min":1.179,"15min":1.2090000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.6120000000000001,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":1.2090000000000001}]}]},{"osd":3,"up_from":419,"seq":1799591297118,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153768,"kb_used_data":35680,"kb_used_omap":65,"kb_used_meta":118014,"kb_avail":93591384,"statfs":{"total":95995035648,"available":95837577216,"internally_reserved":0,"allocated":36536320,"data_stored":56987048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67474,"internal_metadata":120846446},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:39:09 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70699999999999996,"15min":0.747},"min":{"1min":0.55800000000000005,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.546,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.499},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.751,"15min":0.77700000000000002},"min":{"1min":0.56699999999999995,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.97199999999999998,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":0.754}]},{"osd":1,"last update":"Fri Apr 22 17:39:09 2022","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.79400000000000004,"15min":0.81699999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.0979999999999999,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.82699999999999996,"15min":0.82399999999999995},"min":{"1min":0.57299999999999995,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Apr 22 17:39:09 2022","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.94999999999999996,"15min":0.91800000000000004},"min":{"1min":0.432,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.01,"5min":12.394,"15min":12.394},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.90000000000000002,"15min":0.92400000000000004},"min":{"1min":0.54000000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.7769999999999999,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Apr 22 17:38:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.0940000000000001,"15min":1.129},"min":{"1min":0.79200000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":3.6190000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.012},{"interface":"front","average":{"1min":1.7809999999999999,"5min":1.1739999999999999,"15min":1.0760000000000001},"min":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":17.364999999999998,"5min":17.364999999999998,"15min":17.364999999999998},"last":0.93999999999999995}]},{"osd":5,"last update":"Fri Apr 22 17:39:28 2022","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.2,"15min":1.1100000000000001},"min":{"1min":0.86299999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.476,"5min":6.1559999999999997,"15min":6.1559999999999997},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.167,"15min":1.131},"min":{"1min":0.90500000000000003,"5min":0.439,"15min":0.439},"max":{"1min":1.4470000000000001,"5min":4.133,"15min":4.133},"last":0.90900000000000003}]},{"osd":6,"last update":"Fri Apr 22 17:39:09 2022","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1679999999999999,"15min":1.1140000000000001},"min":{"1min":0.71999999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7030000000000001,"5min":3.6909999999999998,"15min":3.6909999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2410000000000001,"15min":1.175},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.615,"5min":3.794,"15min":3.794},"last":0.85699999999999998}]},{"osd":7,"last update":"Fri Apr 22 17:38:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1779999999999999,"15min":1.1140000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":3.1680000000000001,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.2310000000000001,"15min":1.159},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":5.2999999999999998,"5min":5.2999999999999998,"15min":5.2999999999999998},"last":0.88}]}]},{"osd":4,"up_from":460,"seq":1975684956246,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213404,"kb_used_data":51284,"kb_used_omap":97,"kb_used_meta":162014,"kb_avail":93531748,"statfs":{"total":95995035648,"available":95776509952,"internally_reserved":0,"allocated":52514816,"data_stored":69795172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99540,"internal_metadata":165903148},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:38:49 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.52700000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.137,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.65000000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.321,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0409999999999999}]},{"osd":1,"last update":"Fri Apr 22 17:38:49 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.876,"15min":0.876},"min":{"1min":0.52900000000000003,"5min":0.44400000000000001,"15min":0.34899999999999998},"max":{"1min":1.3680000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.89200000000000002,"15min":0.92500000000000004},"min":{"1min":0.52500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2809999999999999,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":0.754}]},{"osd":2,"last update":"Fri Apr 22 17:38:49 2022","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.93400000000000005,"15min":0.95299999999999996},"min":{"1min":0.69699999999999995,"5min":0.46400000000000002,"15min":0.38200000000000001},"max":{"1min":1.2909999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91800000000000004,"15min":1.111},"min":{"1min":0.58799999999999997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2410000000000001,"5min":3.4900000000000002,"15min":5.5590000000000002},"last":1.0700000000000001}]},{"osd":3,"last update":"Fri Apr 22 17:38:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.995,"15min":0.96099999999999997},"min":{"1min":0.67500000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.397,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.872},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95099999999999996,"15min":1.002},"min":{"1min":0.629,"5min":0.52600000000000002,"15min":0.47899999999999998},"max":{"1min":1.22,"5min":3.165,"15min":3.165},"last":1.1459999999999999}]},{"osd":5,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.96699999999999997,"15min":0.90500000000000003},"min":{"1min":0.73999999999999999,"5min":0.40400000000000003,"15min":0.375},"max":{"1min":1.319,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.075,"5min":1.048,"15min":0.93799999999999994},"min":{"1min":0.73499999999999999,"5min":0.53900000000000003,"15min":0.41199999999999998},"max":{"1min":1.419,"5min":3.0609999999999999,"15min":3.0609999999999999},"last":0.99199999999999999}]},{"osd":6,"last update":"Fri Apr 22 17:39:18 2022","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1000000000000001,"15min":1.127},"min":{"1min":0.91600000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.5589999999999999,"5min":4.4169999999999998,"15min":4.4169999999999998},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2370000000000001,"15min":1.23},"min":{"1min":1.1040000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.617,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":1.2410000000000001}]},{"osd":7,"last update":"Fri Apr 22 17:38:49 2022","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1279999999999999,"15min":1.0860000000000001},"min":{"1min":0.97399999999999998,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.494,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.21,"15min":1.1319999999999999},"min":{"1min":0.95099999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.5660000000000001,"5min":3.875,"15min":3.875},"last":1.181}]}]},{"osd":5,"up_from":501,"seq":2151778615373,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407104,"kb_used_data":95160,"kb_used_omap":153,"kb_used_meta":311782,"kb_avail":93338048,"statfs":{"total":95995035648,"available":95578161152,"internally_reserved":0,"allocated":97443840,"data_stored":124919194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":156677,"internal_metadata":319265787},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.85199999999999998,"15min":0.88800000000000001},"min":{"1min":0.625,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.081,"5min":1.421,"15min":1.4750000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.92400000000000004,"15min":0.92000000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2709999999999999,"5min":1.992,"15min":1.992},"last":1.129}]},{"osd":1,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.91900000000000004},"min":{"1min":0.47199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.204,"5min":1.395,"15min":1.46},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.94699999999999995,"15min":0.88500000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.228,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90800000000000003,"15min":0.91200000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1180000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.89300000000000002,"15min":0.879},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1319999999999999,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":0.95999999999999996}]},{"osd":3,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.65700000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.1749999999999998,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.083,"15min":1.0569999999999999},"min":{"1min":0.64800000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.286}]},{"osd":4,"last update":"Fri Apr 22 17:39:24 2022","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88400000000000001,"15min":0.91700000000000004},"min":{"1min":0.55700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.26,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0369999999999999,"15min":0.99299999999999999},"min":{"1min":0.75700000000000001,"5min":0.54800000000000004,"15min":0.52100000000000002},"max":{"1min":1.2869999999999999,"5min":3.718,"15min":3.718},"last":1.26}]},{"osd":6,"last update":"Fri Apr 22 17:39:14 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1200000000000001,"15min":1.1870000000000001},"min":{"1min":0.69699999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.337,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.105,"15min":1.137},"min":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3959999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":1.409}]},{"osd":7,"last update":"Fri Apr 22 17:38:44 2022","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.161,"15min":1.117},"min":{"1min":0.72999999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7,"5min":3.9620000000000002,"15min":3.9620000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1639999999999999,"15min":1.121},"min":{"1min":0.71299999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.4299999999999999,"5min":3.8500000000000001,"15min":3.8500000000000001},"last":1.5009999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13663,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6854,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10694,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15469,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13414,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5623,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6221,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4560,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103510016,"data_stored":147914752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79523840,"data_stored":130404352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48783360,"data_stored":77041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27054080,"data_stored":51605504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43036672,"data_stored":64417792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87965696,"data_stored":119537664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53530624,"data_stored":82059264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40042496,"data_stored":59691008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-22T17:39:42.528 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2022-04-22T17:39:42.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid b6061a0a-c25f-11ec-8c39-001a4aab830c -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-04-22T17:39:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:43 smithi121 bash[36902]: audit 2022-04-22T17:39:41.599956+0000 mgr.y (mgr.34107) 521 : audit [DBG] from='client.34602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:39:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:43 smithi121 bash[36902]: cluster 2022-04-22T17:39:42.028215+0000 mgr.y (mgr.34107) 522 : cluster [DBG] pgmap v988: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:43.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:43 smithi115 bash[45495]: audit 2022-04-22T17:39:41.599956+0000 mgr.y (mgr.34107) 521 : audit [DBG] from='client.34602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:39:43.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:43 smithi115 bash[45495]: cluster 2022-04-22T17:39:42.028215+0000 mgr.y (mgr.34107) 522 : cluster [DBG] pgmap v988: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:43 smithi115 bash[46934]: audit 2022-04-22T17:39:41.599956+0000 mgr.y (mgr.34107) 521 : audit [DBG] from='client.34602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-04-22T17:39:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:43 smithi115 bash[46934]: cluster 2022-04-22T17:39:42.028215+0000 mgr.y (mgr.34107) 522 : cluster [DBG] pgmap v988: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:43.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:43.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:45.323 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/b6061a0a-c25f-11ec-8c39-001a4aab830c/mon.a/config 2022-04-22T17:39:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:45 smithi121 bash[36902]: cluster 2022-04-22T17:39:44.029250+0000 mgr.y (mgr.34107) 523 : cluster [DBG] pgmap v989: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:45 smithi121 bash[36902]: audit 2022-04-22T17:39:44.507809+0000 mon.a (mon.0) 1584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:45 smithi115 bash[45495]: cluster 2022-04-22T17:39:44.029250+0000 mgr.y (mgr.34107) 523 : cluster [DBG] pgmap v989: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:45 smithi115 bash[45495]: audit 2022-04-22T17:39:44.507809+0000 mon.a (mon.0) 1584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:45 smithi115 bash[46934]: cluster 2022-04-22T17:39:44.029250+0000 mgr.y (mgr.34107) 523 : cluster [DBG] pgmap v989: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:45 smithi115 bash[46934]: audit 2022-04-22T17:39:44.507809+0000 mon.a (mon.0) 1584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:47.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:47 smithi115 bash[45495]: cluster 2022-04-22T17:39:46.030318+0000 mgr.y (mgr.34107) 524 : cluster [DBG] pgmap v990: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:47.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:47 smithi115 bash[46934]: cluster 2022-04-22T17:39:46.030318+0000 mgr.y (mgr.34107) 524 : cluster [DBG] pgmap v990: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:47 smithi121 bash[36902]: cluster 2022-04-22T17:39:46.030318+0000 mgr.y (mgr.34107) 524 : cluster [DBG] pgmap v990: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:49.180 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'unique_pool_0' removed 2022-04-22T17:39:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:49 smithi121 bash[36902]: cluster 2022-04-22T17:39:48.031419+0000 mgr.y (mgr.34107) 525 : cluster [DBG] pgmap v991: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:49.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:49 smithi121 bash[36902]: audit 2022-04-22T17:39:48.939101+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2022-04-22T17:39:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:49 smithi115 bash[45495]: cluster 2022-04-22T17:39:48.031419+0000 mgr.y (mgr.34107) 525 : cluster [DBG] pgmap v991: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:49.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:49 smithi115 bash[45495]: audit 2022-04-22T17:39:48.939101+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2022-04-22T17:39:49.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:49 smithi115 bash[46934]: cluster 2022-04-22T17:39:48.031419+0000 mgr.y (mgr.34107) 525 : cluster [DBG] pgmap v991: 121 pgs: 121 active+clean; 308 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:49.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:49 smithi115 bash[46934]: audit 2022-04-22T17:39:48.939101+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2022-04-22T17:39:50.006 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T17:39:50.007 INFO:teuthology.task.print:**** done end ec-rados-default.yaml 2022-04-22T17:39:50.007 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T17:39:50.007 INFO:teuthology.task.print:**** done start rados_api.yaml 2022-04-22T17:39:50.007 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2022-04-22T17:39:50.012 INFO:tasks.workunit:Pulling workunits from ref pacific 2022-04-22T17:39:50.013 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-22T17:39:50.013 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.0 2022-04-22T17:39:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T17:39:50.021 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-04-22T17:39:50.022 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T17:39:50.073 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-04-22T17:39:50.073 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-04-22T17:39:50.078 INFO:tasks.workunit:timeout=3h 2022-04-22T17:39:50.078 INFO:tasks.workunit:cleanup=True 2022-04-22T17:39:50.079 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2022-04-22T17:39:50.085 INFO:tasks.workunit.client.0.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[45495]: audit 2022-04-22T17:39:49.153864+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[45495]: cluster 2022-04-22T17:39:49.153967+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[45495]: audit 2022-04-22T17:39:49.510854+0000 mon.a (mon.0) 1588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[46934]: audit 2022-04-22T17:39:49.153864+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[46934]: cluster 2022-04-22T17:39:49.153967+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2022-04-22T17:39:50.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:50 smithi115 bash[46934]: audit 2022-04-22T17:39:49.510854+0000 mon.a (mon.0) 1588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:50 smithi121 bash[36902]: audit 2022-04-22T17:39:49.153864+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? 172.21.15.115:0/2359653665' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2022-04-22T17:39:50.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:50 smithi121 bash[36902]: cluster 2022-04-22T17:39:49.153967+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2022-04-22T17:39:50.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:50 smithi121 bash[36902]: audit 2022-04-22T17:39:49.510854+0000 mon.a (mon.0) 1588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:51 smithi121 bash[36902]: cluster 2022-04-22T17:39:50.032242+0000 mgr.y (mgr.34107) 526 : cluster [DBG] pgmap v993: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:51 smithi115 bash[45495]: cluster 2022-04-22T17:39:50.032242+0000 mgr.y (mgr.34107) 526 : cluster [DBG] pgmap v993: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:51.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:51 smithi115 bash[46934]: cluster 2022-04-22T17:39:50.032242+0000 mgr.y (mgr.34107) 526 : cluster [DBG] pgmap v993: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:51.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:53 smithi121 bash[36902]: cluster 2022-04-22T17:39:52.033357+0000 mgr.y (mgr.34107) 527 : cluster [DBG] pgmap v994: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:53.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:53 smithi115 bash[46934]: cluster 2022-04-22T17:39:52.033357+0000 mgr.y (mgr.34107) 527 : cluster [DBG] pgmap v994: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:53.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:53 smithi115 bash[45495]: cluster 2022-04-22T17:39:52.033357+0000 mgr.y (mgr.34107) 527 : cluster [DBG] pgmap v994: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:53.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:39:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:39:53.526Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:39:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:55 smithi121 bash[36902]: cluster 2022-04-22T17:39:54.034087+0000 mgr.y (mgr.34107) 528 : cluster [DBG] pgmap v995: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:55 smithi121 bash[36902]: audit 2022-04-22T17:39:54.513937+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:55 smithi115 bash[45495]: cluster 2022-04-22T17:39:54.034087+0000 mgr.y (mgr.34107) 528 : cluster [DBG] pgmap v995: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:55 smithi115 bash[45495]: audit 2022-04-22T17:39:54.513937+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:55 smithi115 bash[46934]: cluster 2022-04-22T17:39:54.034087+0000 mgr.y (mgr.34107) 528 : cluster [DBG] pgmap v995: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:55 smithi115 bash[46934]: audit 2022-04-22T17:39:54.513937+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:39:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:56 smithi121 bash[36902]: cluster 2022-04-22T17:39:56.034874+0000 mgr.y (mgr.34107) 529 : cluster [DBG] pgmap v996: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:56 smithi115 bash[45495]: cluster 2022-04-22T17:39:56.034874+0000 mgr.y (mgr.34107) 529 : cluster [DBG] pgmap v996: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:56.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:56 smithi115 bash[46934]: cluster 2022-04-22T17:39:56.034874+0000 mgr.y (mgr.34107) 529 : cluster [DBG] pgmap v996: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:57.344 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-04-22T17:39:57.405 INFO:tasks.workunit.client.0.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-22T17:39:57.407 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T17:39:57.407 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-22T17:39:58.126 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-22T17:39:58.202 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-22T17:39:58.262 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T17:39:58.264 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T17:39:58.264 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-22T17:39:58.326 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T17:39:58.332 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:39:58.332 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-04-22T17:39:58.338 INFO:tasks.workunit:Running workunits matching cls on client.0... 2022-04-22T17:39:58.339 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2022-04-22T17:39:58.339 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2022-04-22T17:39:58.365 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:39:58.365 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 22 tests from 1 test suite. 2022-04-22T17:39:58.365 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:39:58.366 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 22 tests from TestCls2PCQueue 2022-04-22T17:39:58.366 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2022-04-22T17:39:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:39:59 smithi121 bash[36902]: cluster 2022-04-22T17:39:58.035943+0000 mgr.y (mgr.34107) 530 : cluster [DBG] pgmap v997: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:59.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:39:59 smithi115 bash[45495]: cluster 2022-04-22T17:39:58.035943+0000 mgr.y (mgr.34107) 530 : cluster [DBG] pgmap v997: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:39:59.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:39:59 smithi115 bash[46934]: cluster 2022-04-22T17:39:58.035943+0000 mgr.y (mgr.34107) 530 : cluster [DBG] pgmap v997: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:39:59.107792+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2022-04-22T17:40:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: audit 2022-04-22T17:39:59.113645+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.115:0/748525098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: audit 2022-04-22T17:39:59.117633+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: audit 2022-04-22T17:39:59.516759+0000 mon.a (mon.0) 1592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000167+0000 mon.a (mon.0) 1593 : cluster [WRN] Health detail: HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000228+0000 mon.a (mon.0) 1594 : cluster [WRN] [WRN] UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000257+0000 mon.a (mon.0) 1595 : cluster [WRN] Upgrade daemon: grafana.a: cephadm exited with an error code: 1, stderr: Redeploy daemon grafana.a ... 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000297+0000 mon.a (mon.0) 1596 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000344+0000 mon.a (mon.0) 1597 : cluster [WRN] stat: stderr Unable to find image 'quay.io/ceph/ceph-grafana:8.3.5' locally 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000418+0000 mon.a (mon.0) 1598 : cluster [WRN] stat: stderr docker: Error response from daemon: received unexpected HTTP status: 504 Gateway Time-out. 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000437+0000 mon.a (mon.0) 1599 : cluster [WRN] stat: stderr See 'docker run --help'. 2022-04-22T17:40:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:00 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.000462+0000 mon.a (mon.0) 1600 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:39:59.107792+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2022-04-22T17:40:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: audit 2022-04-22T17:39:59.113645+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.115:0/748525098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: audit 2022-04-22T17:39:59.117633+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: audit 2022-04-22T17:39:59.516759+0000 mon.a (mon.0) 1592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000167+0000 mon.a (mon.0) 1593 : cluster [WRN] Health detail: HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000228+0000 mon.a (mon.0) 1594 : cluster [WRN] [WRN] UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000257+0000 mon.a (mon.0) 1595 : cluster [WRN] Upgrade daemon: grafana.a: cephadm exited with an error code: 1, stderr: Redeploy daemon grafana.a ... 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000297+0000 mon.a (mon.0) 1596 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000344+0000 mon.a (mon.0) 1597 : cluster [WRN] stat: stderr Unable to find image 'quay.io/ceph/ceph-grafana:8.3.5' locally 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000418+0000 mon.a (mon.0) 1598 : cluster [WRN] stat: stderr docker: Error response from daemon: received unexpected HTTP status: 504 Gateway Time-out. 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000437+0000 mon.a (mon.0) 1599 : cluster [WRN] stat: stderr See 'docker run --help'. 2022-04-22T17:40:00.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.000462+0000 mon.a (mon.0) 1600 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:39:59.107792+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: audit 2022-04-22T17:39:59.113645+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.115:0/748525098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: audit 2022-04-22T17:39:59.117633+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: audit 2022-04-22T17:39:59.516759+0000 mon.a (mon.0) 1592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000167+0000 mon.a (mon.0) 1593 : cluster [WRN] Health detail: HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000228+0000 mon.a (mon.0) 1594 : cluster [WRN] [WRN] UPGRADE_REDEPLOY_DAEMON: Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000257+0000 mon.a (mon.0) 1595 : cluster [WRN] Upgrade daemon: grafana.a: cephadm exited with an error code: 1, stderr: Redeploy daemon grafana.a ... 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000297+0000 mon.a (mon.0) 1596 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:00.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000344+0000 mon.a (mon.0) 1597 : cluster [WRN] stat: stderr Unable to find image 'quay.io/ceph/ceph-grafana:8.3.5' locally 2022-04-22T17:40:00.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000418+0000 mon.a (mon.0) 1598 : cluster [WRN] stat: stderr docker: Error response from daemon: received unexpected HTTP status: 504 Gateway Time-out. 2022-04-22T17:40:00.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000437+0000 mon.a (mon.0) 1599 : cluster [WRN] stat: stderr See 'docker run --help'. 2022-04-22T17:40:00.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:00 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.000462+0000 mon.a (mon.0) 1600 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph-grafana:8.3.5 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph-grafana:8.3.5 -c %u %g /var/lib/grafana 2022-04-22T17:40:01.134 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2769 ms) 2022-04-22T17:40:01.135 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2022-04-22T17:40:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:01 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.036776+0000 mgr.y (mgr.34107) 531 : cluster [DBG] pgmap v999: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:01 smithi121 bash[36902]: audit 2022-04-22T17:40:00.115048+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:01 smithi121 bash[36902]: cluster 2022-04-22T17:40:00.115203+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.036776+0000 mgr.y (mgr.34107) 531 : cluster [DBG] pgmap v999: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[45495]: audit 2022-04-22T17:40:00.115048+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[45495]: cluster 2022-04-22T17:40:00.115203+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.036776+0000 mgr.y (mgr.34107) 531 : cluster [DBG] pgmap v999: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[46934]: audit 2022-04-22T17:40:00.115048+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[46934]: cluster 2022-04-22T17:40:00.115203+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2022-04-22T17:40:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:01.633Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:02 smithi121 bash[36902]: cluster 2022-04-22T17:40:01.118959+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2022-04-22T17:40:02.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:02 smithi115 bash[45495]: cluster 2022-04-22T17:40:01.118959+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2022-04-22T17:40:02.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:02 smithi115 bash[46934]: cluster 2022-04-22T17:40:01.118959+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2022-04-22T17:40:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:03 smithi121 bash[36902]: cluster 2022-04-22T17:40:02.037764+0000 mgr.y (mgr.34107) 532 : cluster [DBG] pgmap v1002: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:03 smithi121 bash[36902]: cluster 2022-04-22T17:40:02.141817+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2022-04-22T17:40:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:03 smithi121 bash[36902]: audit 2022-04-22T17:40:02.144380+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[45495]: cluster 2022-04-22T17:40:02.037764+0000 mgr.y (mgr.34107) 532 : cluster [DBG] pgmap v1002: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[45495]: cluster 2022-04-22T17:40:02.141817+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2022-04-22T17:40:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[45495]: audit 2022-04-22T17:40:02.144380+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[46934]: cluster 2022-04-22T17:40:02.037764+0000 mgr.y (mgr.34107) 532 : cluster [DBG] pgmap v1002: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[46934]: cluster 2022-04-22T17:40:02.141817+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2022-04-22T17:40:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[46934]: audit 2022-04-22T17:40:02.144380+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:03.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:03.527Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:04.178 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3043 ms) 2022-04-22T17:40:04.178 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.Reserve 2022-04-22T17:40:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:04 smithi121 bash[36902]: audit 2022-04-22T17:40:03.152999+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:04 smithi121 bash[36902]: cluster 2022-04-22T17:40:03.153272+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2022-04-22T17:40:04.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:04 smithi115 bash[45495]: audit 2022-04-22T17:40:03.152999+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:04.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:04 smithi115 bash[45495]: cluster 2022-04-22T17:40:03.153272+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2022-04-22T17:40:04.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:04 smithi115 bash[46934]: audit 2022-04-22T17:40:03.152999+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? 172.21.15.115:0/727579241' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:04.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:04 smithi115 bash[46934]: cluster 2022-04-22T17:40:03.153272+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2022-04-22T17:40:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:05 smithi121 bash[36902]: cluster 2022-04-22T17:40:04.038741+0000 mgr.y (mgr.34107) 533 : cluster [DBG] pgmap v1005: 137 pgs: 13 creating+peering, 14 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:05 smithi121 bash[36902]: cluster 2022-04-22T17:40:04.163182+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2022-04-22T17:40:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:05 smithi121 bash[36902]: audit 2022-04-22T17:40:04.622219+0000 mon.a (mon.0) 1609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[45495]: cluster 2022-04-22T17:40:04.038741+0000 mgr.y (mgr.34107) 533 : cluster [DBG] pgmap v1005: 137 pgs: 13 creating+peering, 14 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[45495]: cluster 2022-04-22T17:40:04.163182+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2022-04-22T17:40:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[45495]: audit 2022-04-22T17:40:04.622219+0000 mon.a (mon.0) 1609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[46934]: cluster 2022-04-22T17:40:04.038741+0000 mgr.y (mgr.34107) 533 : cluster [DBG] pgmap v1005: 137 pgs: 13 creating+peering, 14 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[46934]: cluster 2022-04-22T17:40:04.163182+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2022-04-22T17:40:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:05 smithi115 bash[46934]: audit 2022-04-22T17:40:04.622219+0000 mon.a (mon.0) 1609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:06.251 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:40:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:06 smithi121 bash[36902]: cluster 2022-04-22T17:40:05.174404+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2022-04-22T17:40:06.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:06 smithi121 bash[36902]: audit 2022-04-22T17:40:05.176706+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.115:0/1144911473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:06 smithi121 bash[36902]: audit 2022-04-22T17:40:05.181317+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[45495]: cluster 2022-04-22T17:40:05.174404+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[45495]: audit 2022-04-22T17:40:05.176706+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.115:0/1144911473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[45495]: audit 2022-04-22T17:40:05.181317+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[46934]: cluster 2022-04-22T17:40:05.174404+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[46934]: audit 2022-04-22T17:40:05.176706+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.115:0/1144911473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:06 smithi115 bash[46934]: audit 2022-04-22T17:40:05.181317+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 3m ago 16m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (16m) 3m ago 16m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5m) 3m ago 17m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4m) 3m ago 17m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (12m) 3m ago 22m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (13m) 3m ago 24m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:40:06.596 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (12m) 3m ago 24m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (11m) 3m ago 22m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (11m) 3m ago 22m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 3m ago 17m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4m) 3m ago 17m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 3m ago 21m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 3m ago 21m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 3m ago 20m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 3m ago 20m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (7m) 3m ago 19m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:40:06.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6m) 3m ago 19m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:40:06.598 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6m) 3m ago 19m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:40:06.598 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5m) 3m ago 18m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:40:06.598 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3m) 3m ago 17m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:40:06.987 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:40:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:40:06.989 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:40:07.205 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.Reserve (3027 ms) 2022-04-22T17:40:07.205 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2022-04-22T17:40:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:07 smithi121 bash[36902]: cluster 2022-04-22T17:40:06.039623+0000 mgr.y (mgr.34107) 534 : cluster [DBG] pgmap v1008: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:07 smithi121 bash[36902]: audit 2022-04-22T17:40:06.183759+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:07 smithi121 bash[36902]: cluster 2022-04-22T17:40:06.183949+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2022-04-22T17:40:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:07 smithi121 bash[36902]: audit 2022-04-22T17:40:06.975127+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? 172.21.15.115:0/394049627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[45495]: cluster 2022-04-22T17:40:06.039623+0000 mgr.y (mgr.34107) 534 : cluster [DBG] pgmap v1008: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[45495]: audit 2022-04-22T17:40:06.183759+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[45495]: cluster 2022-04-22T17:40:06.183949+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[45495]: audit 2022-04-22T17:40:06.975127+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? 172.21.15.115:0/394049627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[46934]: cluster 2022-04-22T17:40:06.039623+0000 mgr.y (mgr.34107) 534 : cluster [DBG] pgmap v1008: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[46934]: audit 2022-04-22T17:40:06.183759+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:07.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[46934]: cluster 2022-04-22T17:40:06.183949+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2022-04-22T17:40:07.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:07 smithi115 bash[46934]: audit 2022-04-22T17:40:06.975127+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? 172.21.15.115:0/394049627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:08 smithi121 bash[36902]: audit 2022-04-22T17:40:06.225167+0000 mgr.y (mgr.34107) 535 : audit [DBG] from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:08 smithi121 bash[36902]: audit 2022-04-22T17:40:06.563351+0000 mgr.y (mgr.34107) 536 : audit [DBG] from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:08 smithi121 bash[36902]: cluster 2022-04-22T17:40:07.189800+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2022-04-22T17:40:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:08 smithi121 bash[36902]: cluster 2022-04-22T17:40:08.040430+0000 mgr.y (mgr.34107) 537 : cluster [DBG] pgmap v1011: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:08.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[45495]: audit 2022-04-22T17:40:06.225167+0000 mgr.y (mgr.34107) 535 : audit [DBG] from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[45495]: audit 2022-04-22T17:40:06.563351+0000 mgr.y (mgr.34107) 536 : audit [DBG] from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[45495]: cluster 2022-04-22T17:40:07.189800+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2022-04-22T17:40:08.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[45495]: cluster 2022-04-22T17:40:08.040430+0000 mgr.y (mgr.34107) 537 : cluster [DBG] pgmap v1011: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:08.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[46934]: audit 2022-04-22T17:40:06.225167+0000 mgr.y (mgr.34107) 535 : audit [DBG] from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[46934]: audit 2022-04-22T17:40:06.563351+0000 mgr.y (mgr.34107) 536 : audit [DBG] from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:08.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[46934]: cluster 2022-04-22T17:40:07.189800+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2022-04-22T17:40:08.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:08 smithi115 bash[46934]: cluster 2022-04-22T17:40:08.040430+0000 mgr.y (mgr.34107) 537 : cluster [DBG] pgmap v1011: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:09 smithi121 bash[36902]: cluster 2022-04-22T17:40:08.217335+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2022-04-22T17:40:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:09 smithi121 bash[36902]: audit 2022-04-22T17:40:08.218840+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.115:0/2595337364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:09 smithi121 bash[36902]: audit 2022-04-22T17:40:08.220085+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[45495]: cluster 2022-04-22T17:40:08.217335+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2022-04-22T17:40:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[45495]: audit 2022-04-22T17:40:08.218840+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.115:0/2595337364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[45495]: audit 2022-04-22T17:40:08.220085+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[46934]: cluster 2022-04-22T17:40:08.217335+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2022-04-22T17:40:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[46934]: audit 2022-04-22T17:40:08.218840+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.115:0/2595337364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:09 smithi115 bash[46934]: audit 2022-04-22T17:40:08.220085+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:10.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3051 ms) 2022-04-22T17:40:10.257 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.Commit 2022-04-22T17:40:10.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:10 smithi121 bash[36902]: audit 2022-04-22T17:40:09.217913+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:10 smithi121 bash[36902]: cluster 2022-04-22T17:40:09.218104+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2022-04-22T17:40:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:10 smithi121 bash[36902]: audit 2022-04-22T17:40:09.524170+0000 mon.a (mon.0) 1620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:10 smithi121 bash[36902]: cluster 2022-04-22T17:40:10.041350+0000 mgr.y (mgr.34107) 538 : cluster [DBG] pgmap v1014: 137 pgs: 7 creating+peering, 23 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[46934]: audit 2022-04-22T17:40:09.217913+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[46934]: cluster 2022-04-22T17:40:09.218104+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2022-04-22T17:40:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[46934]: audit 2022-04-22T17:40:09.524170+0000 mon.a (mon.0) 1620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[46934]: cluster 2022-04-22T17:40:10.041350+0000 mgr.y (mgr.34107) 538 : cluster [DBG] pgmap v1014: 137 pgs: 7 creating+peering, 23 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[45495]: audit 2022-04-22T17:40:09.217913+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[45495]: cluster 2022-04-22T17:40:09.218104+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2022-04-22T17:40:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[45495]: audit 2022-04-22T17:40:09.524170+0000 mon.a (mon.0) 1620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:10 smithi115 bash[45495]: cluster 2022-04-22T17:40:10.041350+0000 mgr.y (mgr.34107) 538 : cluster [DBG] pgmap v1014: 137 pgs: 7 creating+peering, 23 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:11 smithi121 bash[36902]: cluster 2022-04-22T17:40:10.240534+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2022-04-22T17:40:11.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:11 smithi115 bash[45495]: cluster 2022-04-22T17:40:10.240534+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2022-04-22T17:40:11.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:11 smithi115 bash[46934]: cluster 2022-04-22T17:40:10.240534+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2022-04-22T17:40:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:11.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:12.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[45495]: cluster 2022-04-22T17:40:11.251064+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2022-04-22T17:40:12.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[45495]: audit 2022-04-22T17:40:11.252711+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:12.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[45495]: cluster 2022-04-22T17:40:12.042390+0000 mgr.y (mgr.34107) 539 : cluster [DBG] pgmap v1017: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[46934]: cluster 2022-04-22T17:40:11.251064+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2022-04-22T17:40:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[46934]: audit 2022-04-22T17:40:11.252711+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:12.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:12 smithi115 bash[46934]: cluster 2022-04-22T17:40:12.042390+0000 mgr.y (mgr.34107) 539 : cluster [DBG] pgmap v1017: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:12 smithi121 bash[36902]: cluster 2022-04-22T17:40:11.251064+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2022-04-22T17:40:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:12 smithi121 bash[36902]: audit 2022-04-22T17:40:11.252711+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:12 smithi121 bash[36902]: cluster 2022-04-22T17:40:12.042390+0000 mgr.y (mgr.34107) 539 : cluster [DBG] pgmap v1017: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:13 smithi115 bash[46934]: audit 2022-04-22T17:40:12.266300+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:13 smithi115 bash[46934]: cluster 2022-04-22T17:40:12.266604+0000 mon.a (mon.0) 1625 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2022-04-22T17:40:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:13 smithi115 bash[45495]: audit 2022-04-22T17:40:12.266300+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:13 smithi115 bash[45495]: cluster 2022-04-22T17:40:12.266604+0000 mon.a (mon.0) 1625 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2022-04-22T17:40:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:13 smithi121 bash[36902]: audit 2022-04-22T17:40:12.266300+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? 172.21.15.115:0/2542498343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:13 smithi121 bash[36902]: cluster 2022-04-22T17:40:12.266604+0000 mon.a (mon.0) 1625 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2022-04-22T17:40:13.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:13.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:14.299 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.Commit (4043 ms) 2022-04-22T17:40:14.299 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.Abort 2022-04-22T17:40:14.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[46934]: cluster 2022-04-22T17:40:13.266621+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2022-04-22T17:40:14.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[46934]: audit 2022-04-22T17:40:13.357577+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:40:14.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[46934]: audit 2022-04-22T17:40:13.358570+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:40:14.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[46934]: cluster 2022-04-22T17:40:14.043375+0000 mgr.y (mgr.34107) 540 : cluster [DBG] pgmap v1020: 137 pgs: 6 creating+peering, 11 unknown, 120 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:14.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[45495]: cluster 2022-04-22T17:40:13.266621+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2022-04-22T17:40:14.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[45495]: audit 2022-04-22T17:40:13.357577+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:40:14.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[45495]: audit 2022-04-22T17:40:13.358570+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:40:14.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:14 smithi115 bash[45495]: cluster 2022-04-22T17:40:14.043375+0000 mgr.y (mgr.34107) 540 : cluster [DBG] pgmap v1020: 137 pgs: 6 creating+peering, 11 unknown, 120 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:14 smithi121 bash[36902]: cluster 2022-04-22T17:40:13.266621+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2022-04-22T17:40:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:14 smithi121 bash[36902]: audit 2022-04-22T17:40:13.357577+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:40:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:14 smithi121 bash[36902]: audit 2022-04-22T17:40:13.358570+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:40:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:14 smithi121 bash[36902]: cluster 2022-04-22T17:40:14.043375+0000 mgr.y (mgr.34107) 540 : cluster [DBG] pgmap v1020: 137 pgs: 6 creating+peering, 11 unknown, 120 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:15 smithi121 bash[36902]: cluster 2022-04-22T17:40:14.283099+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2022-04-22T17:40:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:15 smithi121 bash[36902]: audit 2022-04-22T17:40:14.755812+0000 mon.a (mon.0) 1630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:15 smithi115 bash[45495]: cluster 2022-04-22T17:40:14.283099+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2022-04-22T17:40:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:15 smithi115 bash[45495]: audit 2022-04-22T17:40:14.755812+0000 mon.a (mon.0) 1630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:15 smithi115 bash[46934]: cluster 2022-04-22T17:40:14.283099+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2022-04-22T17:40:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:15 smithi115 bash[46934]: audit 2022-04-22T17:40:14.755812+0000 mon.a (mon.0) 1630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: cluster 2022-04-22T17:40:15.292755+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: audit 2022-04-22T17:40:15.295669+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.115:0/236315226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: audit 2022-04-22T17:40:15.305908+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: cluster 2022-04-22T17:40:16.044384+0000 mgr.y (mgr.34107) 541 : cluster [DBG] pgmap v1023: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: audit 2022-04-22T17:40:16.293238+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:16 smithi121 bash[36902]: cluster 2022-04-22T17:40:16.293535+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2022-04-22T17:40:16.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: cluster 2022-04-22T17:40:15.292755+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2022-04-22T17:40:16.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: audit 2022-04-22T17:40:15.295669+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.115:0/236315226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: audit 2022-04-22T17:40:15.305908+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: cluster 2022-04-22T17:40:16.044384+0000 mgr.y (mgr.34107) 541 : cluster [DBG] pgmap v1023: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:16.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: audit 2022-04-22T17:40:16.293238+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[45495]: cluster 2022-04-22T17:40:16.293535+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: cluster 2022-04-22T17:40:15.292755+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: audit 2022-04-22T17:40:15.295669+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.115:0/236315226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: audit 2022-04-22T17:40:15.305908+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: cluster 2022-04-22T17:40:16.044384+0000 mgr.y (mgr.34107) 541 : cluster [DBG] pgmap v1023: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: audit 2022-04-22T17:40:16.293238+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:16 smithi115 bash[46934]: cluster 2022-04-22T17:40:16.293535+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2022-04-22T17:40:17.332 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.Abort (3033 ms) 2022-04-22T17:40:17.333 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2022-04-22T17:40:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:18 smithi121 bash[36902]: cluster 2022-04-22T17:40:17.317187+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2022-04-22T17:40:18.804 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:18 smithi121 bash[36902]: cluster 2022-04-22T17:40:18.045225+0000 mgr.y (mgr.34107) 542 : cluster [DBG] pgmap v1026: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:18.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:18 smithi115 bash[45495]: cluster 2022-04-22T17:40:17.317187+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2022-04-22T17:40:18.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:18 smithi115 bash[45495]: cluster 2022-04-22T17:40:18.045225+0000 mgr.y (mgr.34107) 542 : cluster [DBG] pgmap v1026: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:18.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:18 smithi115 bash[46934]: cluster 2022-04-22T17:40:17.317187+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2022-04-22T17:40:18.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:18 smithi115 bash[46934]: cluster 2022-04-22T17:40:18.045225+0000 mgr.y (mgr.34107) 542 : cluster [DBG] pgmap v1026: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:19 smithi121 bash[36902]: cluster 2022-04-22T17:40:18.321076+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2022-04-22T17:40:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:19 smithi121 bash[36902]: audit 2022-04-22T17:40:18.324636+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/3269867954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:19 smithi121 bash[36902]: audit 2022-04-22T17:40:18.326124+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:19.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[46934]: cluster 2022-04-22T17:40:18.321076+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2022-04-22T17:40:19.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[46934]: audit 2022-04-22T17:40:18.324636+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/3269867954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:19.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[46934]: audit 2022-04-22T17:40:18.326124+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:19.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[45495]: cluster 2022-04-22T17:40:18.321076+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2022-04-22T17:40:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[45495]: audit 2022-04-22T17:40:18.324636+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.115:0/3269867954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:19 smithi115 bash[45495]: audit 2022-04-22T17:40:18.326124+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:20.372 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.ReserveError (3040 ms) 2022-04-22T17:40:20.373 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.CommitError 2022-04-22T17:40:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:20 smithi121 bash[36902]: audit 2022-04-22T17:40:19.329065+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:20.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:20 smithi121 bash[36902]: cluster 2022-04-22T17:40:19.329697+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2022-04-22T17:40:20.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:20 smithi121 bash[36902]: audit 2022-04-22T17:40:19.531637+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:20.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:20 smithi121 bash[36902]: cluster 2022-04-22T17:40:20.046165+0000 mgr.y (mgr.34107) 543 : cluster [DBG] pgmap v1029: 137 pgs: 3 creating+peering, 24 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[45495]: audit 2022-04-22T17:40:19.329065+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[45495]: cluster 2022-04-22T17:40:19.329697+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2022-04-22T17:40:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[45495]: audit 2022-04-22T17:40:19.531637+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[45495]: cluster 2022-04-22T17:40:20.046165+0000 mgr.y (mgr.34107) 543 : cluster [DBG] pgmap v1029: 137 pgs: 3 creating+peering, 24 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[46934]: audit 2022-04-22T17:40:19.329065+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[46934]: cluster 2022-04-22T17:40:19.329697+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2022-04-22T17:40:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[46934]: audit 2022-04-22T17:40:19.531637+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:20 smithi115 bash[46934]: cluster 2022-04-22T17:40:20.046165+0000 mgr.y (mgr.34107) 543 : cluster [DBG] pgmap v1029: 137 pgs: 3 creating+peering, 24 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:21.650 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:21 smithi115 bash[45495]: cluster 2022-04-22T17:40:20.351736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2022-04-22T17:40:21.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:21 smithi115 bash[46934]: cluster 2022-04-22T17:40:20.351736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2022-04-22T17:40:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:21 smithi121 bash[36902]: cluster 2022-04-22T17:40:20.351736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2022-04-22T17:40:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:21.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:22 smithi121 bash[36902]: cluster 2022-04-22T17:40:21.360021+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2022-04-22T17:40:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:22 smithi121 bash[36902]: audit 2022-04-22T17:40:21.366732+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.115:0/2305410698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:22 smithi121 bash[36902]: audit 2022-04-22T17:40:21.367654+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:22 smithi121 bash[36902]: cluster 2022-04-22T17:40:22.047207+0000 mgr.y (mgr.34107) 544 : cluster [DBG] pgmap v1032: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:22.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[45495]: cluster 2022-04-22T17:40:21.360021+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2022-04-22T17:40:22.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[45495]: audit 2022-04-22T17:40:21.366732+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.115:0/2305410698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[45495]: audit 2022-04-22T17:40:21.367654+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[45495]: cluster 2022-04-22T17:40:22.047207+0000 mgr.y (mgr.34107) 544 : cluster [DBG] pgmap v1032: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:22.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[46934]: cluster 2022-04-22T17:40:21.360021+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2022-04-22T17:40:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[46934]: audit 2022-04-22T17:40:21.366732+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.115:0/2305410698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[46934]: audit 2022-04-22T17:40:21.367654+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:22 smithi115 bash[46934]: cluster 2022-04-22T17:40:22.047207+0000 mgr.y (mgr.34107) 544 : cluster [DBG] pgmap v1032: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:23.394 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.CommitError (3022 ms) 2022-04-22T17:40:23.405 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.AbortError 2022-04-22T17:40:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:23 smithi121 bash[36902]: audit 2022-04-22T17:40:22.371767+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:23 smithi121 bash[36902]: cluster 2022-04-22T17:40:22.371904+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2022-04-22T17:40:23.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:23 smithi115 bash[46934]: audit 2022-04-22T17:40:22.371767+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:23.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:23 smithi115 bash[46934]: cluster 2022-04-22T17:40:22.371904+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2022-04-22T17:40:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:23.527Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:23.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:23 smithi115 bash[45495]: audit 2022-04-22T17:40:22.371767+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:23.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:23 smithi115 bash[45495]: cluster 2022-04-22T17:40:22.371904+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2022-04-22T17:40:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:24 smithi121 bash[36902]: cluster 2022-04-22T17:40:23.377759+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2022-04-22T17:40:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:24 smithi121 bash[36902]: cluster 2022-04-22T17:40:24.048159+0000 mgr.y (mgr.34107) 545 : cluster [DBG] pgmap v1035: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:24.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:24 smithi115 bash[45495]: cluster 2022-04-22T17:40:23.377759+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2022-04-22T17:40:24.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:24 smithi115 bash[45495]: cluster 2022-04-22T17:40:24.048159+0000 mgr.y (mgr.34107) 545 : cluster [DBG] pgmap v1035: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:24.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:24 smithi115 bash[46934]: cluster 2022-04-22T17:40:23.377759+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2022-04-22T17:40:24.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:24 smithi115 bash[46934]: cluster 2022-04-22T17:40:24.048159+0000 mgr.y (mgr.34107) 545 : cluster [DBG] pgmap v1035: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:25.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[46934]: cluster 2022-04-22T17:40:24.394364+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2022-04-22T17:40:25.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[46934]: audit 2022-04-22T17:40:24.397109+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:25.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[46934]: audit 2022-04-22T17:40:24.534741+0000 mon.a (mon.0) 1649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:25.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[45495]: cluster 2022-04-22T17:40:24.394364+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2022-04-22T17:40:25.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[45495]: audit 2022-04-22T17:40:24.397109+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:25.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:25 smithi115 bash[45495]: audit 2022-04-22T17:40:24.534741+0000 mon.a (mon.0) 1649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:25 smithi121 bash[36902]: cluster 2022-04-22T17:40:24.394364+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2022-04-22T17:40:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:25 smithi121 bash[36902]: audit 2022-04-22T17:40:24.397109+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:25 smithi121 bash[36902]: audit 2022-04-22T17:40:24.534741+0000 mon.a (mon.0) 1649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:26.426 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.AbortError (3032 ms) 2022-04-22T17:40:26.426 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2022-04-22T17:40:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: audit 2022-04-22T17:40:25.398950+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: cluster 2022-04-22T17:40:25.399083+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2022-04-22T17:40:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: audit 2022-04-22T17:40:25.421362+0000 mon.a (mon.0) 1652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:40:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: audit 2022-04-22T17:40:25.423187+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:40:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: audit 2022-04-22T17:40:25.424636+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:40:26.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: audit 2022-04-22T17:40:25.776416+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:40:26.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:26 smithi121 bash[36902]: cluster 2022-04-22T17:40:26.049130+0000 mgr.y (mgr.34107) 546 : cluster [DBG] pgmap v1038: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:26.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: audit 2022-04-22T17:40:25.398950+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:26.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: cluster 2022-04-22T17:40:25.399083+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2022-04-22T17:40:26.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: audit 2022-04-22T17:40:25.421362+0000 mon.a (mon.0) 1652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:40:26.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: audit 2022-04-22T17:40:25.423187+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: audit 2022-04-22T17:40:25.424636+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: audit 2022-04-22T17:40:25.776416+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[45495]: cluster 2022-04-22T17:40:26.049130+0000 mgr.y (mgr.34107) 546 : cluster [DBG] pgmap v1038: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: audit 2022-04-22T17:40:25.398950+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? 172.21.15.115:0/634477669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: cluster 2022-04-22T17:40:25.399083+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: audit 2022-04-22T17:40:25.421362+0000 mon.a (mon.0) 1652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: audit 2022-04-22T17:40:25.423187+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:40:26.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: audit 2022-04-22T17:40:25.424636+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:40:26.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: audit 2022-04-22T17:40:25.776416+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:40:26.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:26 smithi115 bash[46934]: cluster 2022-04-22T17:40:26.049130+0000 mgr.y (mgr.34107) 546 : cluster [DBG] pgmap v1038: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:27 smithi121 bash[36902]: cluster 2022-04-22T17:40:26.411571+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2022-04-22T17:40:27.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:27 smithi115 bash[45495]: cluster 2022-04-22T17:40:26.411571+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2022-04-22T17:40:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:27 smithi115 bash[46934]: cluster 2022-04-22T17:40:26.411571+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2022-04-22T17:40:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:28 smithi121 bash[36902]: cluster 2022-04-22T17:40:27.429314+0000 mon.a (mon.0) 1657 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2022-04-22T17:40:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:28 smithi121 bash[36902]: audit 2022-04-22T17:40:27.437722+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/176426157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:28 smithi121 bash[36902]: audit 2022-04-22T17:40:27.439468+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:28 smithi121 bash[36902]: cluster 2022-04-22T17:40:28.050114+0000 mgr.y (mgr.34107) 547 : cluster [DBG] pgmap v1041: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[45495]: cluster 2022-04-22T17:40:27.429314+0000 mon.a (mon.0) 1657 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2022-04-22T17:40:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[45495]: audit 2022-04-22T17:40:27.437722+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/176426157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[45495]: audit 2022-04-22T17:40:27.439468+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[45495]: cluster 2022-04-22T17:40:28.050114+0000 mgr.y (mgr.34107) 547 : cluster [DBG] pgmap v1041: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:28.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[46934]: cluster 2022-04-22T17:40:27.429314+0000 mon.a (mon.0) 1657 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2022-04-22T17:40:28.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[46934]: audit 2022-04-22T17:40:27.437722+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.115:0/176426157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[46934]: audit 2022-04-22T17:40:27.439468+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:28.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:28 smithi115 bash[46934]: cluster 2022-04-22T17:40:28.050114+0000 mgr.y (mgr.34107) 547 : cluster [DBG] pgmap v1041: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:29.467 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3041 ms) 2022-04-22T17:40:29.467 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2022-04-22T17:40:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:29 smithi121 bash[36902]: audit 2022-04-22T17:40:28.430098+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:29 smithi121 bash[36902]: cluster 2022-04-22T17:40:28.430311+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2022-04-22T17:40:29.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:29 smithi115 bash[46934]: audit 2022-04-22T17:40:28.430098+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:29.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:29 smithi115 bash[46934]: cluster 2022-04-22T17:40:28.430311+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2022-04-22T17:40:29.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:29 smithi115 bash[45495]: audit 2022-04-22T17:40:28.430098+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:29.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:29 smithi115 bash[45495]: cluster 2022-04-22T17:40:28.430311+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2022-04-22T17:40:30.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:30 smithi121 bash[36902]: cluster 2022-04-22T17:40:29.451710+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2022-04-22T17:40:30.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:30 smithi121 bash[36902]: audit 2022-04-22T17:40:29.924797+0000 mon.a (mon.0) 1662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:30.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:30 smithi121 bash[36902]: cluster 2022-04-22T17:40:30.050977+0000 mgr.y (mgr.34107) 548 : cluster [DBG] pgmap v1044: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[45495]: cluster 2022-04-22T17:40:29.451710+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2022-04-22T17:40:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[45495]: audit 2022-04-22T17:40:29.924797+0000 mon.a (mon.0) 1662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[45495]: cluster 2022-04-22T17:40:30.050977+0000 mgr.y (mgr.34107) 548 : cluster [DBG] pgmap v1044: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[46934]: cluster 2022-04-22T17:40:29.451710+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2022-04-22T17:40:30.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[46934]: audit 2022-04-22T17:40:29.924797+0000 mon.a (mon.0) 1662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:30.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:30 smithi115 bash[46934]: cluster 2022-04-22T17:40:30.050977+0000 mgr.y (mgr.34107) 548 : cluster [DBG] pgmap v1044: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:31 smithi121 bash[36902]: cluster 2022-04-22T17:40:30.471165+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2022-04-22T17:40:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:31 smithi121 bash[36902]: audit 2022-04-22T17:40:30.477191+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/1542501679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:31 smithi121 bash[36902]: audit 2022-04-22T17:40:30.477700+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:31.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:31.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:31.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[45495]: cluster 2022-04-22T17:40:30.471165+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2022-04-22T17:40:31.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[45495]: audit 2022-04-22T17:40:30.477191+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/1542501679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:31.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[45495]: audit 2022-04-22T17:40:30.477700+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:31.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[46934]: cluster 2022-04-22T17:40:30.471165+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2022-04-22T17:40:31.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[46934]: audit 2022-04-22T17:40:30.477191+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.115:0/1542501679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:31.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:31 smithi115 bash[46934]: audit 2022-04-22T17:40:30.477700+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:32.513 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3046 ms) 2022-04-22T17:40:32.513 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2022-04-22T17:40:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:32 smithi121 bash[36902]: audit 2022-04-22T17:40:31.466439+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:32 smithi121 bash[36902]: cluster 2022-04-22T17:40:31.466717+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2022-04-22T17:40:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:32 smithi121 bash[36902]: cluster 2022-04-22T17:40:32.052154+0000 mgr.y (mgr.34107) 549 : cluster [DBG] pgmap v1047: 137 pgs: 3 creating+peering, 17 unknown, 117 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:32.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[46934]: audit 2022-04-22T17:40:31.466439+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:32.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[46934]: cluster 2022-04-22T17:40:31.466717+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2022-04-22T17:40:32.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[46934]: cluster 2022-04-22T17:40:32.052154+0000 mgr.y (mgr.34107) 549 : cluster [DBG] pgmap v1047: 137 pgs: 3 creating+peering, 17 unknown, 117 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:32.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[45495]: audit 2022-04-22T17:40:31.466439+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:32.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[45495]: cluster 2022-04-22T17:40:31.466717+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2022-04-22T17:40:32.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:32 smithi115 bash[45495]: cluster 2022-04-22T17:40:32.052154+0000 mgr.y (mgr.34107) 549 : cluster [DBG] pgmap v1047: 137 pgs: 3 creating+peering, 17 unknown, 117 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:33 smithi121 bash[36902]: cluster 2022-04-22T17:40:32.494083+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2022-04-22T17:40:33.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:33 smithi115 bash[45495]: cluster 2022-04-22T17:40:32.494083+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2022-04-22T17:40:33.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:33 smithi115 bash[46934]: cluster 2022-04-22T17:40:32.494083+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2022-04-22T17:40:33.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:33.529Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:34.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[45495]: cluster 2022-04-22T17:40:33.499528+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2022-04-22T17:40:34.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[45495]: audit 2022-04-22T17:40:33.501258+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:34.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[45495]: cluster 2022-04-22T17:40:34.053265+0000 mgr.y (mgr.34107) 550 : cluster [DBG] pgmap v1050: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:34.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[46934]: cluster 2022-04-22T17:40:33.499528+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2022-04-22T17:40:34.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[46934]: audit 2022-04-22T17:40:33.501258+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:34.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:34 smithi115 bash[46934]: cluster 2022-04-22T17:40:34.053265+0000 mgr.y (mgr.34107) 550 : cluster [DBG] pgmap v1050: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:34 smithi121 bash[36902]: cluster 2022-04-22T17:40:33.499528+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2022-04-22T17:40:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:34 smithi121 bash[36902]: audit 2022-04-22T17:40:33.501258+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:34 smithi121 bash[36902]: cluster 2022-04-22T17:40:34.053265+0000 mgr.y (mgr.34107) 550 : cluster [DBG] pgmap v1050: 137 pgs: 6 creating+peering, 26 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2022-04-22T17:40:35.543 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3027 ms) 2022-04-22T17:40:35.552 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2022-04-22T17:40:35.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[45495]: audit 2022-04-22T17:40:34.506736+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:35.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[45495]: cluster 2022-04-22T17:40:34.506973+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2022-04-22T17:40:35.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[45495]: audit 2022-04-22T17:40:34.541876+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:35.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[46934]: audit 2022-04-22T17:40:34.506736+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:35.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[46934]: cluster 2022-04-22T17:40:34.506973+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2022-04-22T17:40:35.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:35 smithi115 bash[46934]: audit 2022-04-22T17:40:34.541876+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:35 smithi121 bash[36902]: audit 2022-04-22T17:40:34.506736+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.115:0/2396011335' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:36.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:35 smithi121 bash[36902]: cluster 2022-04-22T17:40:34.506973+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2022-04-22T17:40:36.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:35 smithi121 bash[36902]: audit 2022-04-22T17:40:34.541876+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:36 smithi121 bash[36902]: cluster 2022-04-22T17:40:35.522296+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2022-04-22T17:40:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:36 smithi121 bash[36902]: cluster 2022-04-22T17:40:36.054273+0000 mgr.y (mgr.34107) 551 : cluster [DBG] pgmap v1053: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:37.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:36 smithi115 bash[45495]: cluster 2022-04-22T17:40:35.522296+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2022-04-22T17:40:37.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:36 smithi115 bash[45495]: cluster 2022-04-22T17:40:36.054273+0000 mgr.y (mgr.34107) 551 : cluster [DBG] pgmap v1053: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:37.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:36 smithi115 bash[46934]: cluster 2022-04-22T17:40:35.522296+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2022-04-22T17:40:37.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:36 smithi115 bash[46934]: cluster 2022-04-22T17:40:36.054273+0000 mgr.y (mgr.34107) 551 : cluster [DBG] pgmap v1053: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:37.356 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:40:37.721 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:40:37.816 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 3m ago 17m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:40:37.816 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (17m) 4m ago 17m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5m) 3m ago 18m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5m) 4m ago 18m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (12m) 4m ago 22m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (13m) 3m ago 24m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (12m) 3m ago 25m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (11m) 4m ago 23m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:40:37.817 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (12m) 3m ago 23m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 3m ago 17m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4m) 4m ago 17m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 3m ago 22m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 3m ago 21m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 3m ago 21m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 3m ago 20m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (8m) 4m ago 20m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (7m) 4m ago 20m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6m) 4m ago 19m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:40:37.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6m) 4m ago 19m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:40:37.819 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4m) 4m ago 17m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:40:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:37 smithi121 bash[36902]: cluster 2022-04-22T17:40:36.546236+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2022-04-22T17:40:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:37 smithi121 bash[36902]: audit 2022-04-22T17:40:36.548515+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:38.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:37 smithi115 bash[46934]: cluster 2022-04-22T17:40:36.546236+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2022-04-22T17:40:38.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:37 smithi115 bash[46934]: audit 2022-04-22T17:40:36.548515+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:38.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:37 smithi115 bash[45495]: cluster 2022-04-22T17:40:36.546236+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2022-04-22T17:40:38.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:37 smithi115 bash[45495]: audit 2022-04-22T17:40:36.548515+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:38.146 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:40:38.146 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:40:38.146 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:40:38.146 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:40:38.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:40:38.148 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:40:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:40:38.565 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3025 ms) 2022-04-22T17:40:38.565 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: audit 2022-04-22T17:40:37.327526+0000 mgr.y (mgr.34107) 552 : audit [DBG] from='client.44644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: audit 2022-04-22T17:40:37.552869+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: cluster 2022-04-22T17:40:37.553048+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: audit 2022-04-22T17:40:37.694341+0000 mgr.y (mgr.34107) 553 : audit [DBG] from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: cluster 2022-04-22T17:40:38.054933+0000 mgr.y (mgr.34107) 554 : cluster [DBG] pgmap v1056: 137 pgs: 8 creating+peering, 22 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: audit 2022-04-22T17:40:38.132869+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.? 172.21.15.115:0/3307480800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:39.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:38 smithi121 bash[36902]: cluster 2022-04-22T17:40:38.549363+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: audit 2022-04-22T17:40:37.327526+0000 mgr.y (mgr.34107) 552 : audit [DBG] from='client.44644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: audit 2022-04-22T17:40:37.552869+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: cluster 2022-04-22T17:40:37.553048+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: audit 2022-04-22T17:40:37.694341+0000 mgr.y (mgr.34107) 553 : audit [DBG] from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: cluster 2022-04-22T17:40:38.054933+0000 mgr.y (mgr.34107) 554 : cluster [DBG] pgmap v1056: 137 pgs: 8 creating+peering, 22 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: audit 2022-04-22T17:40:38.132869+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.? 172.21.15.115:0/3307480800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:39.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[45495]: cluster 2022-04-22T17:40:38.549363+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: audit 2022-04-22T17:40:37.327526+0000 mgr.y (mgr.34107) 552 : audit [DBG] from='client.44644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: audit 2022-04-22T17:40:37.552869+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.115:0/332622979' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: cluster 2022-04-22T17:40:37.553048+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: audit 2022-04-22T17:40:37.694341+0000 mgr.y (mgr.34107) 553 : audit [DBG] from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: cluster 2022-04-22T17:40:38.054933+0000 mgr.y (mgr.34107) 554 : cluster [DBG] pgmap v1056: 137 pgs: 8 creating+peering, 22 unknown, 107 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: audit 2022-04-22T17:40:38.132869+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.? 172.21.15.115:0/3307480800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:40:39.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:38 smithi115 bash[46934]: cluster 2022-04-22T17:40:38.549363+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2022-04-22T17:40:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:39 smithi121 bash[36902]: audit 2022-04-22T17:40:39.544520+0000 mon.a (mon.0) 1680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:40.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:39 smithi115 bash[45495]: audit 2022-04-22T17:40:39.544520+0000 mon.a (mon.0) 1680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:39 smithi115 bash[46934]: audit 2022-04-22T17:40:39.544520+0000 mon.a (mon.0) 1680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:40 smithi121 bash[36902]: cluster 2022-04-22T17:40:39.574692+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2022-04-22T17:40:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:40 smithi121 bash[36902]: audit 2022-04-22T17:40:39.578986+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.115:0/757952957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:40 smithi121 bash[36902]: audit 2022-04-22T17:40:39.588410+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:40 smithi121 bash[36902]: cluster 2022-04-22T17:40:40.055865+0000 mgr.y (mgr.34107) 555 : cluster [DBG] pgmap v1059: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:41.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[45495]: cluster 2022-04-22T17:40:39.574692+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2022-04-22T17:40:41.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[45495]: audit 2022-04-22T17:40:39.578986+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.115:0/757952957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[45495]: audit 2022-04-22T17:40:39.588410+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[45495]: cluster 2022-04-22T17:40:40.055865+0000 mgr.y (mgr.34107) 555 : cluster [DBG] pgmap v1059: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:41.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[46934]: cluster 2022-04-22T17:40:39.574692+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2022-04-22T17:40:41.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[46934]: audit 2022-04-22T17:40:39.578986+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.115:0/757952957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[46934]: audit 2022-04-22T17:40:39.588410+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:41.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:40 smithi115 bash[46934]: cluster 2022-04-22T17:40:40.055865+0000 mgr.y (mgr.34107) 555 : cluster [DBG] pgmap v1059: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:41.601 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3034 ms) 2022-04-22T17:40:41.602 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2022-04-22T17:40:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:41 smithi121 bash[36902]: audit 2022-04-22T17:40:40.581783+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:42.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:41 smithi121 bash[36902]: cluster 2022-04-22T17:40:40.581930+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2022-04-22T17:40:42.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:41 smithi121 bash[36902]: cluster 2022-04-22T17:40:41.583692+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2022-04-22T17:40:42.055 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:41.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:42.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[45495]: audit 2022-04-22T17:40:40.581783+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:42.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[45495]: cluster 2022-04-22T17:40:40.581930+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2022-04-22T17:40:42.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[45495]: cluster 2022-04-22T17:40:41.583692+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2022-04-22T17:40:42.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[46934]: audit 2022-04-22T17:40:40.581783+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:42.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[46934]: cluster 2022-04-22T17:40:40.581930+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2022-04-22T17:40:42.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:41 smithi115 bash[46934]: cluster 2022-04-22T17:40:41.583692+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2022-04-22T17:40:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:42 smithi121 bash[36902]: cluster 2022-04-22T17:40:42.056483+0000 mgr.y (mgr.34107) 556 : cluster [DBG] pgmap v1062: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:43.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:42 smithi115 bash[45495]: cluster 2022-04-22T17:40:42.056483+0000 mgr.y (mgr.34107) 556 : cluster [DBG] pgmap v1062: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:43.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:42 smithi115 bash[46934]: cluster 2022-04-22T17:40:42.056483+0000 mgr.y (mgr.34107) 556 : cluster [DBG] pgmap v1062: 105 pgs: 105 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:43 smithi121 bash[36902]: cluster 2022-04-22T17:40:42.613324+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2022-04-22T17:40:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:43 smithi121 bash[36902]: audit 2022-04-22T17:40:42.616696+0000 mon.c (mon.1) 35 : audit [INF] from='client.? 172.21.15.115:0/2325322124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:43 smithi121 bash[36902]: audit 2022-04-22T17:40:42.621577+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:43.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[45495]: cluster 2022-04-22T17:40:42.613324+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2022-04-22T17:40:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[45495]: audit 2022-04-22T17:40:42.616696+0000 mon.c (mon.1) 35 : audit [INF] from='client.? 172.21.15.115:0/2325322124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[45495]: audit 2022-04-22T17:40:42.621577+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[46934]: cluster 2022-04-22T17:40:42.613324+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2022-04-22T17:40:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[46934]: audit 2022-04-22T17:40:42.616696+0000 mon.c (mon.1) 35 : audit [INF] from='client.? 172.21.15.115:0/2325322124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:43 smithi115 bash[46934]: audit 2022-04-22T17:40:42.621577+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:44.656 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3057 ms) 2022-04-22T17:40:44.656 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2022-04-22T17:40:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:44 smithi121 bash[36902]: audit 2022-04-22T17:40:43.619196+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:44 smithi121 bash[36902]: cluster 2022-04-22T17:40:43.619412+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2022-04-22T17:40:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:44 smithi121 bash[36902]: cluster 2022-04-22T17:40:44.057248+0000 mgr.y (mgr.34107) 557 : cluster [DBG] pgmap v1065: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:44 smithi121 bash[36902]: audit 2022-04-22T17:40:44.547906+0000 mon.a (mon.0) 1690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:45.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[45495]: audit 2022-04-22T17:40:43.619196+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:45.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[45495]: cluster 2022-04-22T17:40:43.619412+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2022-04-22T17:40:45.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[45495]: cluster 2022-04-22T17:40:44.057248+0000 mgr.y (mgr.34107) 557 : cluster [DBG] pgmap v1065: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:45.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[45495]: audit 2022-04-22T17:40:44.547906+0000 mon.a (mon.0) 1690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:45.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[46934]: audit 2022-04-22T17:40:43.619196+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:45.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[46934]: cluster 2022-04-22T17:40:43.619412+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2022-04-22T17:40:45.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[46934]: cluster 2022-04-22T17:40:44.057248+0000 mgr.y (mgr.34107) 557 : cluster [DBG] pgmap v1065: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:45.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:44 smithi115 bash[46934]: audit 2022-04-22T17:40:44.547906+0000 mon.a (mon.0) 1690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:45 smithi121 bash[36902]: cluster 2022-04-22T17:40:44.636363+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2022-04-22T17:40:46.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:45 smithi115 bash[45495]: cluster 2022-04-22T17:40:44.636363+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2022-04-22T17:40:46.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:45 smithi115 bash[46934]: cluster 2022-04-22T17:40:44.636363+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2022-04-22T17:40:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:46 smithi121 bash[36902]: cluster 2022-04-22T17:40:45.651914+0000 mon.a (mon.0) 1692 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2022-04-22T17:40:47.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:46 smithi121 bash[36902]: audit 2022-04-22T17:40:45.655259+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:47.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:46 smithi121 bash[36902]: cluster 2022-04-22T17:40:46.058223+0000 mgr.y (mgr.34107) 558 : cluster [DBG] pgmap v1068: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:47.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[45495]: cluster 2022-04-22T17:40:45.651914+0000 mon.a (mon.0) 1692 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2022-04-22T17:40:47.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[45495]: audit 2022-04-22T17:40:45.655259+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:47.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[45495]: cluster 2022-04-22T17:40:46.058223+0000 mgr.y (mgr.34107) 558 : cluster [DBG] pgmap v1068: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:47.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[46934]: cluster 2022-04-22T17:40:45.651914+0000 mon.a (mon.0) 1692 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2022-04-22T17:40:47.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[46934]: audit 2022-04-22T17:40:45.655259+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:47.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:46 smithi115 bash[46934]: cluster 2022-04-22T17:40:46.058223+0000 mgr.y (mgr.34107) 558 : cluster [DBG] pgmap v1068: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:47.687 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.Cleanup (3026 ms) 2022-04-22T17:40:47.687 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2022-04-22T17:40:48.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:47 smithi121 bash[36902]: audit 2022-04-22T17:40:46.660554+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:48.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:47 smithi121 bash[36902]: cluster 2022-04-22T17:40:46.660835+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2022-04-22T17:40:48.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:47 smithi115 bash[45495]: audit 2022-04-22T17:40:46.660554+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:48.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:47 smithi115 bash[45495]: cluster 2022-04-22T17:40:46.660835+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2022-04-22T17:40:48.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:47 smithi115 bash[46934]: audit 2022-04-22T17:40:46.660554+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.115:0/1213047481' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-16","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:48.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:47 smithi115 bash[46934]: cluster 2022-04-22T17:40:46.660835+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2022-04-22T17:40:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:48 smithi121 bash[36902]: cluster 2022-04-22T17:40:47.667111+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2022-04-22T17:40:49.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:48 smithi121 bash[36902]: cluster 2022-04-22T17:40:48.058979+0000 mgr.y (mgr.34107) 559 : cluster [DBG] pgmap v1071: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:49.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:48 smithi115 bash[45495]: cluster 2022-04-22T17:40:47.667111+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2022-04-22T17:40:49.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:48 smithi115 bash[45495]: cluster 2022-04-22T17:40:48.058979+0000 mgr.y (mgr.34107) 559 : cluster [DBG] pgmap v1071: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:49.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:48 smithi115 bash[46934]: cluster 2022-04-22T17:40:47.667111+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2022-04-22T17:40:49.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:48 smithi115 bash[46934]: cluster 2022-04-22T17:40:48.058979+0000 mgr.y (mgr.34107) 559 : cluster [DBG] pgmap v1071: 105 pgs: 105 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:49 smithi121 bash[36902]: cluster 2022-04-22T17:40:48.687632+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2022-04-22T17:40:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:49 smithi121 bash[36902]: audit 2022-04-22T17:40:48.691157+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:49 smithi121 bash[36902]: audit 2022-04-22T17:40:49.551159+0000 mon.a (mon.0) 1699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:50.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[45495]: cluster 2022-04-22T17:40:48.687632+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2022-04-22T17:40:50.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[45495]: audit 2022-04-22T17:40:48.691157+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:50.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[45495]: audit 2022-04-22T17:40:49.551159+0000 mon.a (mon.0) 1699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:50.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[46934]: cluster 2022-04-22T17:40:48.687632+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2022-04-22T17:40:50.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[46934]: audit 2022-04-22T17:40:48.691157+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:40:50.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:49 smithi115 bash[46934]: audit 2022-04-22T17:40:49.551159+0000 mon.a (mon.0) 1699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:50 smithi121 bash[36902]: audit 2022-04-22T17:40:49.689694+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:50 smithi121 bash[36902]: cluster 2022-04-22T17:40:49.690056+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2022-04-22T17:40:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:50 smithi121 bash[36902]: cluster 2022-04-22T17:40:50.059769+0000 mgr.y (mgr.34107) 560 : cluster [DBG] pgmap v1074: 137 pgs: 3 creating+peering, 23 unknown, 111 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:51.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[46934]: audit 2022-04-22T17:40:49.689694+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:51.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[46934]: cluster 2022-04-22T17:40:49.690056+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2022-04-22T17:40:51.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[46934]: cluster 2022-04-22T17:40:50.059769+0000 mgr.y (mgr.34107) 560 : cluster [DBG] pgmap v1074: 137 pgs: 3 creating+peering, 23 unknown, 111 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:51.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[45495]: audit 2022-04-22T17:40:49.689694+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? 172.21.15.115:0/2102775754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:40:51.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[45495]: cluster 2022-04-22T17:40:49.690056+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2022-04-22T17:40:51.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:50 smithi115 bash[45495]: cluster 2022-04-22T17:40:50.059769+0000 mgr.y (mgr.34107) 560 : cluster [DBG] pgmap v1074: 137 pgs: 3 creating+peering, 23 unknown, 111 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:40:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:51 smithi121 bash[36902]: cluster 2022-04-22T17:40:50.706673+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2022-04-22T17:40:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:51 smithi115 bash[46934]: cluster 2022-04-22T17:40:50.706673+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2022-04-22T17:40:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:51.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:52.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:51 smithi115 bash[45495]: cluster 2022-04-22T17:40:50.706673+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2022-04-22T17:40:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:52 smithi121 bash[36902]: cluster 2022-04-22T17:40:52.060681+0000 mgr.y (mgr.34107) 561 : cluster [DBG] pgmap v1076: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 1.3 MiB/s wr, 1.51k op/s 2022-04-22T17:40:53.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:52 smithi115 bash[45495]: cluster 2022-04-22T17:40:52.060681+0000 mgr.y (mgr.34107) 561 : cluster [DBG] pgmap v1076: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 1.3 MiB/s wr, 1.51k op/s 2022-04-22T17:40:53.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:52 smithi115 bash[46934]: cluster 2022-04-22T17:40:52.060681+0000 mgr.y (mgr.34107) 561 : cluster [DBG] pgmap v1076: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 1.3 MiB/s wr, 1.51k op/s 2022-04-22T17:40:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:40:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:40:53.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:40:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:55 smithi121 bash[36902]: cluster 2022-04-22T17:40:54.061304+0000 mgr.y (mgr.34107) 562 : cluster [DBG] pgmap v1077: 137 pgs: 137 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s rd, 1004 KiB/s wr, 1.10k op/s 2022-04-22T17:40:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:55 smithi121 bash[36902]: audit 2022-04-22T17:40:54.554026+0000 mon.a (mon.0) 1703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:55 smithi115 bash[45495]: cluster 2022-04-22T17:40:54.061304+0000 mgr.y (mgr.34107) 562 : cluster [DBG] pgmap v1077: 137 pgs: 137 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s rd, 1004 KiB/s wr, 1.10k op/s 2022-04-22T17:40:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:55 smithi115 bash[45495]: audit 2022-04-22T17:40:54.554026+0000 mon.a (mon.0) 1703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:55 smithi115 bash[46934]: cluster 2022-04-22T17:40:54.061304+0000 mgr.y (mgr.34107) 562 : cluster [DBG] pgmap v1077: 137 pgs: 137 active+clean; 741 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s rd, 1004 KiB/s wr, 1.10k op/s 2022-04-22T17:40:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:55 smithi115 bash[46934]: audit 2022-04-22T17:40:54.554026+0000 mon.a (mon.0) 1703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:40:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:57 smithi121 bash[36902]: cluster 2022-04-22T17:40:56.062105+0000 mgr.y (mgr.34107) 563 : cluster [DBG] pgmap v1078: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 370 KiB/s rd, 3.8 MiB/s wr, 4.26k op/s 2022-04-22T17:40:57.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:57 smithi115 bash[45495]: cluster 2022-04-22T17:40:56.062105+0000 mgr.y (mgr.34107) 563 : cluster [DBG] pgmap v1078: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 370 KiB/s rd, 3.8 MiB/s wr, 4.26k op/s 2022-04-22T17:40:57.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:57 smithi115 bash[46934]: cluster 2022-04-22T17:40:56.062105+0000 mgr.y (mgr.34107) 563 : cluster [DBG] pgmap v1078: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 370 KiB/s rd, 3.8 MiB/s wr, 4.26k op/s 2022-04-22T17:40:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:40:59 smithi121 bash[36902]: cluster 2022-04-22T17:40:58.063039+0000 mgr.y (mgr.34107) 564 : cluster [DBG] pgmap v1079: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2022-04-22T17:40:59.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:40:59 smithi115 bash[45495]: cluster 2022-04-22T17:40:58.063039+0000 mgr.y (mgr.34107) 564 : cluster [DBG] pgmap v1079: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2022-04-22T17:40:59.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:40:59 smithi115 bash[46934]: cluster 2022-04-22T17:40:58.063039+0000 mgr.y (mgr.34107) 564 : cluster [DBG] pgmap v1079: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 3.3 MiB/s wr, 3.75k op/s 2022-04-22T17:41:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:00 smithi121 bash[36902]: audit 2022-04-22T17:40:59.556441+0000 mon.a (mon.0) 1704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:00.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:00 smithi115 bash[46934]: audit 2022-04-22T17:40:59.556441+0000 mon.a (mon.0) 1704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:00 smithi115 bash[45495]: audit 2022-04-22T17:40:59.556441+0000 mon.a (mon.0) 1704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:01 smithi121 bash[36902]: cluster 2022-04-22T17:41:00.063971+0000 mgr.y (mgr.34107) 565 : cluster [DBG] pgmap v1080: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 273 KiB/s rd, 2.8 MiB/s wr, 3.14k op/s 2022-04-22T17:41:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:01 smithi115 bash[46934]: cluster 2022-04-22T17:41:00.063971+0000 mgr.y (mgr.34107) 565 : cluster [DBG] pgmap v1080: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 273 KiB/s rd, 2.8 MiB/s wr, 3.14k op/s 2022-04-22T17:41:01.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:01 smithi115 bash[45495]: cluster 2022-04-22T17:41:00.063971+0000 mgr.y (mgr.34107) 565 : cluster [DBG] pgmap v1080: 137 pgs: 137 active+clean; 1.2 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 273 KiB/s rd, 2.8 MiB/s wr, 3.14k op/s 2022-04-22T17:41:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:01.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:03 smithi121 bash[36902]: cluster 2022-04-22T17:41:02.064887+0000 mgr.y (mgr.34107) 566 : cluster [DBG] pgmap v1081: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 387 KiB/s rd, 3.9 MiB/s wr, 4.45k op/s 2022-04-22T17:41:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:03 smithi115 bash[45495]: cluster 2022-04-22T17:41:02.064887+0000 mgr.y (mgr.34107) 566 : cluster [DBG] pgmap v1081: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 387 KiB/s rd, 3.9 MiB/s wr, 4.45k op/s 2022-04-22T17:41:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:03 smithi115 bash[46934]: cluster 2022-04-22T17:41:02.064887+0000 mgr.y (mgr.34107) 566 : cluster [DBG] pgmap v1081: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 387 KiB/s rd, 3.9 MiB/s wr, 4.45k op/s 2022-04-22T17:41:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:03.530Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:04 smithi121 bash[36902]: audit 2022-04-22T17:41:03.541912+0000 mon.a (mon.0) 1705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:04 smithi121 bash[36902]: cluster 2022-04-22T17:41:04.065923+0000 mgr.y (mgr.34107) 567 : cluster [DBG] pgmap v1082: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 318 KiB/s rd, 3.2 MiB/s wr, 3.66k op/s 2022-04-22T17:41:05.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:04 smithi115 bash[45495]: audit 2022-04-22T17:41:03.541912+0000 mon.a (mon.0) 1705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:05.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:04 smithi115 bash[45495]: cluster 2022-04-22T17:41:04.065923+0000 mgr.y (mgr.34107) 567 : cluster [DBG] pgmap v1082: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 318 KiB/s rd, 3.2 MiB/s wr, 3.66k op/s 2022-04-22T17:41:05.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:04 smithi115 bash[46934]: audit 2022-04-22T17:41:03.541912+0000 mon.a (mon.0) 1705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:05.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:04 smithi115 bash[46934]: cluster 2022-04-22T17:41:04.065923+0000 mgr.y (mgr.34107) 567 : cluster [DBG] pgmap v1082: 137 pgs: 137 active+clean; 1.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 318 KiB/s rd, 3.2 MiB/s wr, 3.66k op/s 2022-04-22T17:41:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:05 smithi121 bash[36902]: audit 2022-04-22T17:41:04.558656+0000 mon.a (mon.0) 1706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:06.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:05 smithi115 bash[45495]: audit 2022-04-22T17:41:04.558656+0000 mon.a (mon.0) 1706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:06.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:05 smithi115 bash[46934]: audit 2022-04-22T17:41:04.558656+0000 mon.a (mon.0) 1706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:06 smithi121 bash[36902]: cluster 2022-04-22T17:41:06.066956+0000 mgr.y (mgr.34107) 568 : cluster [DBG] pgmap v1083: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 440 KiB/s rd, 4.5 MiB/s wr, 5.06k op/s 2022-04-22T17:41:07.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:06 smithi115 bash[45495]: cluster 2022-04-22T17:41:06.066956+0000 mgr.y (mgr.34107) 568 : cluster [DBG] pgmap v1083: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 440 KiB/s rd, 4.5 MiB/s wr, 5.06k op/s 2022-04-22T17:41:07.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:06 smithi115 bash[46934]: cluster 2022-04-22T17:41:06.066956+0000 mgr.y (mgr.34107) 568 : cluster [DBG] pgmap v1083: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 440 KiB/s rd, 4.5 MiB/s wr, 5.06k op/s 2022-04-22T17:41:08.540 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:41:08.909 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 4m ago 18m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (17m) 4m ago 17m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6m) 4m ago 19m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5m) 4m ago 18m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (13m) 4m ago 23m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (14m) 4m ago 25m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (13m) 4m ago 25m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (12m) 4m ago 23m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:41:08.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (12m) 4m ago 23m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5m) 4m ago 18m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5m) 4m ago 18m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 4m ago 22m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 4m ago 22m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 4m ago 21m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 4m ago 21m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (8m) 4m ago 20m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (7m) 4m ago 20m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:41:08.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (7m) 4m ago 20m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:41:08.912 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6m) 4m ago 19m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:41:08.912 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4m) 4m ago 18m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:41:09.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:41:09.332 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:41:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:09 smithi121 bash[36902]: cluster 2022-04-22T17:41:08.067957+0000 mgr.y (mgr.34107) 569 : cluster [DBG] pgmap v1084: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:09 smithi115 bash[46934]: cluster 2022-04-22T17:41:08.067957+0000 mgr.y (mgr.34107) 569 : cluster [DBG] pgmap v1084: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:09 smithi115 bash[45495]: cluster 2022-04-22T17:41:08.067957+0000 mgr.y (mgr.34107) 569 : cluster [DBG] pgmap v1084: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:10 smithi121 bash[36902]: audit 2022-04-22T17:41:08.512789+0000 mgr.y (mgr.34107) 570 : audit [DBG] from='client.44686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:10 smithi121 bash[36902]: audit 2022-04-22T17:41:08.880353+0000 mgr.y (mgr.34107) 571 : audit [DBG] from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:10 smithi121 bash[36902]: audit 2022-04-22T17:41:09.318211+0000 mon.a (mon.0) 1707 : audit [DBG] from='client.? 172.21.15.115:0/3877777904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:10 smithi121 bash[36902]: audit 2022-04-22T17:41:09.560778+0000 mon.a (mon.0) 1708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[45495]: audit 2022-04-22T17:41:08.512789+0000 mgr.y (mgr.34107) 570 : audit [DBG] from='client.44686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[45495]: audit 2022-04-22T17:41:08.880353+0000 mgr.y (mgr.34107) 571 : audit [DBG] from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[45495]: audit 2022-04-22T17:41:09.318211+0000 mon.a (mon.0) 1707 : audit [DBG] from='client.? 172.21.15.115:0/3877777904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[45495]: audit 2022-04-22T17:41:09.560778+0000 mon.a (mon.0) 1708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[46934]: audit 2022-04-22T17:41:08.512789+0000 mgr.y (mgr.34107) 570 : audit [DBG] from='client.44686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[46934]: audit 2022-04-22T17:41:08.880353+0000 mgr.y (mgr.34107) 571 : audit [DBG] from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[46934]: audit 2022-04-22T17:41:09.318211+0000 mon.a (mon.0) 1707 : audit [DBG] from='client.? 172.21.15.115:0/3877777904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:10 smithi115 bash[46934]: audit 2022-04-22T17:41:09.560778+0000 mon.a (mon.0) 1708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:11 smithi121 bash[36902]: cluster 2022-04-22T17:41:10.068802+0000 mgr.y (mgr.34107) 572 : cluster [DBG] pgmap v1085: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:11.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:11 smithi115 bash[45495]: cluster 2022-04-22T17:41:10.068802+0000 mgr.y (mgr.34107) 572 : cluster [DBG] pgmap v1085: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:11 smithi115 bash[46934]: cluster 2022-04-22T17:41:10.068802+0000 mgr.y (mgr.34107) 572 : cluster [DBG] pgmap v1085: 137 pgs: 137 active+clean; 2.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 260 KiB/s rd, 2.7 MiB/s wr, 2.99k op/s 2022-04-22T17:41:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:11.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:13 smithi121 bash[36902]: cluster 2022-04-22T17:41:12.070026+0000 mgr.y (mgr.34107) 573 : cluster [DBG] pgmap v1086: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 3.8 MiB/s wr, 4.23k op/s 2022-04-22T17:41:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:13 smithi115 bash[45495]: cluster 2022-04-22T17:41:12.070026+0000 mgr.y (mgr.34107) 573 : cluster [DBG] pgmap v1086: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 3.8 MiB/s wr, 4.23k op/s 2022-04-22T17:41:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:13 smithi115 bash[46934]: cluster 2022-04-22T17:41:12.070026+0000 mgr.y (mgr.34107) 573 : cluster [DBG] pgmap v1086: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 3.8 MiB/s wr, 4.23k op/s 2022-04-22T17:41:13.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:13.532Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:14 smithi121 bash[36902]: audit 2022-04-22T17:41:13.209505+0000 mon.a (mon.0) 1709 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:14 smithi121 bash[36902]: audit 2022-04-22T17:41:13.209840+0000 mon.a (mon.0) 1710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:14 smithi121 bash[36902]: audit 2022-04-22T17:41:13.210112+0000 mon.a (mon.0) 1711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:14 smithi121 bash[36902]: audit 2022-04-22T17:41:13.367795+0000 mon.a (mon.0) 1712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:41:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:14 smithi121 bash[36902]: audit 2022-04-22T17:41:13.368233+0000 mon.a (mon.0) 1713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:41:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[45495]: audit 2022-04-22T17:41:13.209505+0000 mon.a (mon.0) 1709 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[45495]: audit 2022-04-22T17:41:13.209840+0000 mon.a (mon.0) 1710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[45495]: audit 2022-04-22T17:41:13.210112+0000 mon.a (mon.0) 1711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[45495]: audit 2022-04-22T17:41:13.367795+0000 mon.a (mon.0) 1712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[45495]: audit 2022-04-22T17:41:13.368233+0000 mon.a (mon.0) 1713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[46934]: audit 2022-04-22T17:41:13.209505+0000 mon.a (mon.0) 1709 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[46934]: audit 2022-04-22T17:41:13.209840+0000 mon.a (mon.0) 1710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[46934]: audit 2022-04-22T17:41:13.210112+0000 mon.a (mon.0) 1711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[46934]: audit 2022-04-22T17:41:13.367795+0000 mon.a (mon.0) 1712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:41:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:14 smithi115 bash[46934]: audit 2022-04-22T17:41:13.368233+0000 mon.a (mon.0) 1713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: cluster 2022-04-22T17:41:14.070778+0000 mgr.y (mgr.34107) 574 : cluster [DBG] pgmap v1087: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 230 KiB/s rd, 2.3 MiB/s wr, 2.64k op/s 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: audit 2022-04-22T17:41:14.150471+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: audit 2022-04-22T17:41:14.150585+0000 mon.a (mon.0) 1715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]': finished 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: audit 2022-04-22T17:41:14.150661+0000 mon.a (mon.0) 1716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: cluster 2022-04-22T17:41:14.150837+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2022-04-22T17:41:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:15 smithi121 bash[36902]: audit 2022-04-22T17:41:14.563349+0000 mon.a (mon.0) 1718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: cluster 2022-04-22T17:41:14.070778+0000 mgr.y (mgr.34107) 574 : cluster [DBG] pgmap v1087: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 230 KiB/s rd, 2.3 MiB/s wr, 2.64k op/s 2022-04-22T17:41:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: audit 2022-04-22T17:41:14.150471+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2022-04-22T17:41:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: audit 2022-04-22T17:41:14.150585+0000 mon.a (mon.0) 1715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]': finished 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: audit 2022-04-22T17:41:14.150661+0000 mon.a (mon.0) 1716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: cluster 2022-04-22T17:41:14.150837+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[45495]: audit 2022-04-22T17:41:14.563349+0000 mon.a (mon.0) 1718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: cluster 2022-04-22T17:41:14.070778+0000 mgr.y (mgr.34107) 574 : cluster [DBG] pgmap v1087: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 230 KiB/s rd, 2.3 MiB/s wr, 2.64k op/s 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: audit 2022-04-22T17:41:14.150471+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: audit 2022-04-22T17:41:14.150585+0000 mon.a (mon.0) 1715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.11", "id": [7, 0]}]': finished 2022-04-22T17:41:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: audit 2022-04-22T17:41:14.150661+0000 mon.a (mon.0) 1716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2022-04-22T17:41:15.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: cluster 2022-04-22T17:41:14.150837+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2022-04-22T17:41:15.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:15 smithi115 bash[46934]: audit 2022-04-22T17:41:14.563349+0000 mon.a (mon.0) 1718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:16 smithi121 bash[36902]: cluster 2022-04-22T17:41:15.158783+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2022-04-22T17:41:16.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:16 smithi115 bash[45495]: cluster 2022-04-22T17:41:15.158783+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2022-04-22T17:41:16.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:16 smithi115 bash[46934]: cluster 2022-04-22T17:41:15.158783+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2022-04-22T17:41:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:17 smithi121 bash[36902]: cluster 2022-04-22T17:41:16.071667+0000 mgr.y (mgr.34107) 575 : cluster [DBG] pgmap v1090: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:17.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:17 smithi115 bash[45495]: cluster 2022-04-22T17:41:16.071667+0000 mgr.y (mgr.34107) 575 : cluster [DBG] pgmap v1090: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:17.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:17 smithi115 bash[46934]: cluster 2022-04-22T17:41:16.071667+0000 mgr.y (mgr.34107) 575 : cluster [DBG] pgmap v1090: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:19.195 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.MultiProducer (31512 ms) 2022-04-22T17:41:19.195 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2022-04-22T17:41:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:19 smithi121 bash[36902]: cluster 2022-04-22T17:41:18.072688+0000 mgr.y (mgr.34107) 576 : cluster [DBG] pgmap v1091: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:19.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:19 smithi115 bash[45495]: cluster 2022-04-22T17:41:18.072688+0000 mgr.y (mgr.34107) 576 : cluster [DBG] pgmap v1091: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:19.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:19 smithi115 bash[46934]: cluster 2022-04-22T17:41:18.072688+0000 mgr.y (mgr.34107) 576 : cluster [DBG] pgmap v1091: 137 pgs: 137 active+clean; 2.3 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 1.8 MiB/s wr, 2.54k op/s 2022-04-22T17:41:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:20 smithi121 bash[36902]: cluster 2022-04-22T17:41:19.179382+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2022-04-22T17:41:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:20 smithi121 bash[36902]: audit 2022-04-22T17:41:19.650472+0000 mon.a (mon.0) 1721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:20 smithi115 bash[45495]: cluster 2022-04-22T17:41:19.179382+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2022-04-22T17:41:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:20 smithi115 bash[45495]: audit 2022-04-22T17:41:19.650472+0000 mon.a (mon.0) 1721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:20.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:20 smithi115 bash[46934]: cluster 2022-04-22T17:41:19.179382+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2022-04-22T17:41:20.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:20 smithi115 bash[46934]: audit 2022-04-22T17:41:19.650472+0000 mon.a (mon.0) 1721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:21 smithi121 bash[36902]: cluster 2022-04-22T17:41:20.073537+0000 mgr.y (mgr.34107) 577 : cluster [DBG] pgmap v1093: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:21 smithi121 bash[36902]: cluster 2022-04-22T17:41:20.191928+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2022-04-22T17:41:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:21 smithi121 bash[36902]: audit 2022-04-22T17:41:20.194800+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.115:0/324337794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:21 smithi121 bash[36902]: audit 2022-04-22T17:41:20.196169+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:21.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[45495]: cluster 2022-04-22T17:41:20.073537+0000 mgr.y (mgr.34107) 577 : cluster [DBG] pgmap v1093: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:21.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[45495]: cluster 2022-04-22T17:41:20.191928+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[45495]: audit 2022-04-22T17:41:20.194800+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.115:0/324337794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[45495]: audit 2022-04-22T17:41:20.196169+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[46934]: cluster 2022-04-22T17:41:20.073537+0000 mgr.y (mgr.34107) 577 : cluster [DBG] pgmap v1093: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[46934]: cluster 2022-04-22T17:41:20.191928+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[46934]: audit 2022-04-22T17:41:20.194800+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.115:0/324337794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:21.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[46934]: audit 2022-04-22T17:41:20.196169+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:22.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:21.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:22 smithi121 bash[36902]: audit 2022-04-22T17:41:21.204905+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:22 smithi121 bash[36902]: cluster 2022-04-22T17:41:21.205086+0000 mon.a (mon.0) 1725 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2022-04-22T17:41:22.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:22 smithi115 bash[45495]: audit 2022-04-22T17:41:21.204905+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:22.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:22 smithi115 bash[45495]: cluster 2022-04-22T17:41:21.205086+0000 mon.a (mon.0) 1725 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2022-04-22T17:41:22.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:22 smithi115 bash[46934]: audit 2022-04-22T17:41:21.204905+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:22.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:22 smithi115 bash[46934]: cluster 2022-04-22T17:41:21.205086+0000 mon.a (mon.0) 1725 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2022-04-22T17:41:23.234 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (4040 ms) 2022-04-22T17:41:23.234 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2022-04-22T17:41:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:23 smithi121 bash[36902]: cluster 2022-04-22T17:41:22.074284+0000 mgr.y (mgr.34107) 578 : cluster [DBG] pgmap v1096: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:23 smithi121 bash[36902]: cluster 2022-04-22T17:41:22.210484+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2022-04-22T17:41:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:23 smithi115 bash[46934]: cluster 2022-04-22T17:41:22.074284+0000 mgr.y (mgr.34107) 578 : cluster [DBG] pgmap v1096: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:23 smithi115 bash[46934]: cluster 2022-04-22T17:41:22.210484+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2022-04-22T17:41:23.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:23 smithi115 bash[45495]: cluster 2022-04-22T17:41:22.074284+0000 mgr.y (mgr.34107) 578 : cluster [DBG] pgmap v1096: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:23.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:23 smithi115 bash[45495]: cluster 2022-04-22T17:41:22.210484+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2022-04-22T17:41:23.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:23.534Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:24 smithi121 bash[36902]: cluster 2022-04-22T17:41:23.218100+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2022-04-22T17:41:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:24 smithi121 bash[36902]: cluster 2022-04-22T17:41:24.075111+0000 mgr.y (mgr.34107) 579 : cluster [DBG] pgmap v1099: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:24.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:24 smithi115 bash[45495]: cluster 2022-04-22T17:41:23.218100+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2022-04-22T17:41:24.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:24 smithi115 bash[45495]: cluster 2022-04-22T17:41:24.075111+0000 mgr.y (mgr.34107) 579 : cluster [DBG] pgmap v1099: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:24.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:24 smithi115 bash[46934]: cluster 2022-04-22T17:41:23.218100+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2022-04-22T17:41:24.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:24 smithi115 bash[46934]: cluster 2022-04-22T17:41:24.075111+0000 mgr.y (mgr.34107) 579 : cluster [DBG] pgmap v1099: 105 pgs: 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:25 smithi121 bash[36902]: cluster 2022-04-22T17:41:24.228280+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2022-04-22T17:41:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:25 smithi121 bash[36902]: audit 2022-04-22T17:41:24.231836+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.115:0/2952557781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:25 smithi121 bash[36902]: audit 2022-04-22T17:41:24.232369+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:25 smithi121 bash[36902]: audit 2022-04-22T17:41:24.570233+0000 mon.a (mon.0) 1730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[46934]: cluster 2022-04-22T17:41:24.228280+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2022-04-22T17:41:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[46934]: audit 2022-04-22T17:41:24.231836+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.115:0/2952557781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[46934]: audit 2022-04-22T17:41:24.232369+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[46934]: audit 2022-04-22T17:41:24.570233+0000 mon.a (mon.0) 1730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[45495]: cluster 2022-04-22T17:41:24.228280+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2022-04-22T17:41:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[45495]: audit 2022-04-22T17:41:24.231836+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.115:0/2952557781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[45495]: audit 2022-04-22T17:41:24.232369+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:41:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:25 smithi115 bash[45495]: audit 2022-04-22T17:41:24.570233+0000 mon.a (mon.0) 1730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: audit 2022-04-22T17:41:25.237267+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: cluster 2022-04-22T17:41:25.237495+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: audit 2022-04-22T17:41:25.783868+0000 mon.a (mon.0) 1733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: audit 2022-04-22T17:41:25.785803+0000 mon.a (mon.0) 1734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: audit 2022-04-22T17:41:25.787079+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: cluster 2022-04-22T17:41:26.075794+0000 mgr.y (mgr.34107) 580 : cluster [DBG] pgmap v1102: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:26 smithi121 bash[36902]: audit 2022-04-22T17:41:26.136793+0000 mon.a (mon.0) 1736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: audit 2022-04-22T17:41:25.237267+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: cluster 2022-04-22T17:41:25.237495+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: audit 2022-04-22T17:41:25.783868+0000 mon.a (mon.0) 1733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: audit 2022-04-22T17:41:25.785803+0000 mon.a (mon.0) 1734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: audit 2022-04-22T17:41:25.787079+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: cluster 2022-04-22T17:41:26.075794+0000 mgr.y (mgr.34107) 580 : cluster [DBG] pgmap v1102: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[45495]: audit 2022-04-22T17:41:26.136793+0000 mon.a (mon.0) 1736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: audit 2022-04-22T17:41:25.237267+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: cluster 2022-04-22T17:41:25.237495+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2022-04-22T17:41:26.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: audit 2022-04-22T17:41:25.783868+0000 mon.a (mon.0) 1733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:41:26.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: audit 2022-04-22T17:41:25.785803+0000 mon.a (mon.0) 1734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:41:26.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: audit 2022-04-22T17:41:25.787079+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:41:26.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: cluster 2022-04-22T17:41:26.075794+0000 mgr.y (mgr.34107) 580 : cluster [DBG] pgmap v1102: 137 pgs: 8 creating+peering, 19 unknown, 110 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:26.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:26 smithi115 bash[46934]: audit 2022-04-22T17:41:26.136793+0000 mon.a (mon.0) 1736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:27 smithi115 bash[45495]: cluster 2022-04-22T17:41:26.257258+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2022-04-22T17:41:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:27 smithi115 bash[46934]: cluster 2022-04-22T17:41:26.257258+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2022-04-22T17:41:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:27 smithi121 bash[36902]: cluster 2022-04-22T17:41:26.257258+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2022-04-22T17:41:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:28 smithi115 bash[45495]: cluster 2022-04-22T17:41:28.076737+0000 mgr.y (mgr.34107) 581 : cluster [DBG] pgmap v1104: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:28.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:28 smithi115 bash[46934]: cluster 2022-04-22T17:41:28.076737+0000 mgr.y (mgr.34107) 581 : cluster [DBG] pgmap v1104: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:28 smithi121 bash[36902]: cluster 2022-04-22T17:41:28.076737+0000 mgr.y (mgr.34107) 581 : cluster [DBG] pgmap v1104: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:41:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:29 smithi121 bash[36902]: audit 2022-04-22T17:41:29.573111+0000 mon.a (mon.0) 1738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:30.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:29 smithi115 bash[45495]: audit 2022-04-22T17:41:29.573111+0000 mon.a (mon.0) 1738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:30.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:29 smithi115 bash[46934]: audit 2022-04-22T17:41:29.573111+0000 mon.a (mon.0) 1738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:31 smithi121 bash[36902]: cluster 2022-04-22T17:41:30.077467+0000 mgr.y (mgr.34107) 582 : cluster [DBG] pgmap v1105: 137 pgs: 6 creating+peering, 131 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 468 KiB/s rd, 2.4 MiB/s wr, 2.93k op/s 2022-04-22T17:41:31.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:31 smithi115 bash[45495]: cluster 2022-04-22T17:41:30.077467+0000 mgr.y (mgr.34107) 582 : cluster [DBG] pgmap v1105: 137 pgs: 6 creating+peering, 131 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 468 KiB/s rd, 2.4 MiB/s wr, 2.93k op/s 2022-04-22T17:41:31.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:31 smithi115 bash[46934]: cluster 2022-04-22T17:41:30.077467+0000 mgr.y (mgr.34107) 582 : cluster [DBG] pgmap v1105: 137 pgs: 6 creating+peering, 131 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 468 KiB/s rd, 2.4 MiB/s wr, 2.93k op/s 2022-04-22T17:41:32.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:31.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:33 smithi121 bash[36902]: cluster 2022-04-22T17:41:32.078308+0000 mgr.y (mgr.34107) 583 : cluster [DBG] pgmap v1106: 137 pgs: 137 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 358 KiB/s rd, 1.8 MiB/s wr, 2.24k op/s 2022-04-22T17:41:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:33 smithi115 bash[46934]: cluster 2022-04-22T17:41:32.078308+0000 mgr.y (mgr.34107) 583 : cluster [DBG] pgmap v1106: 137 pgs: 137 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 358 KiB/s rd, 1.8 MiB/s wr, 2.24k op/s 2022-04-22T17:41:33.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:33.531Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:33 smithi115 bash[45495]: cluster 2022-04-22T17:41:32.078308+0000 mgr.y (mgr.34107) 583 : cluster [DBG] pgmap v1106: 137 pgs: 137 active+clean; 935 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 358 KiB/s rd, 1.8 MiB/s wr, 2.24k op/s 2022-04-22T17:41:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:35 smithi121 bash[36902]: cluster 2022-04-22T17:41:34.079253+0000 mgr.y (mgr.34107) 584 : cluster [DBG] pgmap v1107: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 673 KiB/s rd, 3.4 MiB/s wr, 4.15k op/s 2022-04-22T17:41:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:35 smithi121 bash[36902]: audit 2022-04-22T17:41:34.575718+0000 mon.a (mon.0) 1739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:35 smithi115 bash[46934]: cluster 2022-04-22T17:41:34.079253+0000 mgr.y (mgr.34107) 584 : cluster [DBG] pgmap v1107: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 673 KiB/s rd, 3.4 MiB/s wr, 4.15k op/s 2022-04-22T17:41:35.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:35 smithi115 bash[46934]: audit 2022-04-22T17:41:34.575718+0000 mon.a (mon.0) 1739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:35.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:35 smithi115 bash[45495]: cluster 2022-04-22T17:41:34.079253+0000 mgr.y (mgr.34107) 584 : cluster [DBG] pgmap v1107: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 673 KiB/s rd, 3.4 MiB/s wr, 4.15k op/s 2022-04-22T17:41:35.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:35 smithi115 bash[45495]: audit 2022-04-22T17:41:34.575718+0000 mon.a (mon.0) 1739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:37 smithi121 bash[36902]: cluster 2022-04-22T17:41:36.080183+0000 mgr.y (mgr.34107) 585 : cluster [DBG] pgmap v1108: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 595 KiB/s rd, 3.0 MiB/s wr, 3.67k op/s 2022-04-22T17:41:37.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:37 smithi115 bash[45495]: cluster 2022-04-22T17:41:36.080183+0000 mgr.y (mgr.34107) 585 : cluster [DBG] pgmap v1108: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 595 KiB/s rd, 3.0 MiB/s wr, 3.67k op/s 2022-04-22T17:41:37.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:37 smithi115 bash[46934]: cluster 2022-04-22T17:41:36.080183+0000 mgr.y (mgr.34107) 585 : cluster [DBG] pgmap v1108: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 595 KiB/s rd, 3.0 MiB/s wr, 3.67k op/s 2022-04-22T17:41:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:39 smithi121 bash[36902]: cluster 2022-04-22T17:41:38.080879+0000 mgr.y (mgr.34107) 586 : cluster [DBG] pgmap v1109: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 2.5 MiB/s wr, 3.10k op/s 2022-04-22T17:41:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:39 smithi121 bash[36902]: audit 2022-04-22T17:41:38.573042+0000 mon.a (mon.0) 1740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:39.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:39 smithi115 bash[46934]: cluster 2022-04-22T17:41:38.080879+0000 mgr.y (mgr.34107) 586 : cluster [DBG] pgmap v1109: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 2.5 MiB/s wr, 3.10k op/s 2022-04-22T17:41:39.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:39 smithi115 bash[46934]: audit 2022-04-22T17:41:38.573042+0000 mon.a (mon.0) 1740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:39.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:39 smithi115 bash[45495]: cluster 2022-04-22T17:41:38.080879+0000 mgr.y (mgr.34107) 586 : cluster [DBG] pgmap v1109: 137 pgs: 137 active+clean; 1.3 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 2.5 MiB/s wr, 3.10k op/s 2022-04-22T17:41:39.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:39 smithi115 bash[45495]: audit 2022-04-22T17:41:38.573042+0000 mon.a (mon.0) 1740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:41:39.758 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:41:40.141 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:41:40.141 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 4m ago 18m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:41:40.141 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (18m) 5m ago 18m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:41:40.141 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6m) 4m ago 19m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6m) 5m ago 19m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (13m) 5m ago 23m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (14m) 4m ago 25m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (13m) 4m ago 26m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (12m) 5m ago 24m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (13m) 4m ago 24m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5m) 4m ago 18m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5m) 5m ago 18m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:41:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (11m) 4m ago 23m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 4m ago 22m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 4m ago 22m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 4m ago 21m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (9m) 5m ago 21m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (8m) 5m ago 21m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (7m) 5m ago 20m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (7m) 5m ago 20m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:41:40.143 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5m) 5m ago 18m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:41:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:40 smithi121 bash[36902]: audit 2022-04-22T17:41:39.578261+0000 mon.a (mon.0) 1741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:40.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:40 smithi115 bash[45495]: audit 2022-04-22T17:41:39.578261+0000 mon.a (mon.0) 1741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:40.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:40 smithi115 bash[46934]: audit 2022-04-22T17:41:39.578261+0000 mon.a (mon.0) 1741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:40.595 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:41:40.597 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:41:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:41 smithi121 bash[36902]: audit 2022-04-22T17:41:39.727882+0000 mgr.y (mgr.34107) 587 : audit [DBG] from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:41 smithi121 bash[36902]: cluster 2022-04-22T17:41:40.081585+0000 mgr.y (mgr.34107) 588 : cluster [DBG] pgmap v1110: 137 pgs: 137 active+clean; 1.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 4.46k op/s 2022-04-22T17:41:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:41 smithi121 bash[36902]: audit 2022-04-22T17:41:40.112593+0000 mgr.y (mgr.34107) 589 : audit [DBG] from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:41 smithi121 bash[36902]: audit 2022-04-22T17:41:40.582357+0000 mon.a (mon.0) 1742 : audit [DBG] from='client.? 172.21.15.115:0/3458347188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[46934]: audit 2022-04-22T17:41:39.727882+0000 mgr.y (mgr.34107) 587 : audit [DBG] from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[46934]: cluster 2022-04-22T17:41:40.081585+0000 mgr.y (mgr.34107) 588 : cluster [DBG] pgmap v1110: 137 pgs: 137 active+clean; 1.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 4.46k op/s 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[46934]: audit 2022-04-22T17:41:40.112593+0000 mgr.y (mgr.34107) 589 : audit [DBG] from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[46934]: audit 2022-04-22T17:41:40.582357+0000 mon.a (mon.0) 1742 : audit [DBG] from='client.? 172.21.15.115:0/3458347188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[45495]: audit 2022-04-22T17:41:39.727882+0000 mgr.y (mgr.34107) 587 : audit [DBG] from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[45495]: cluster 2022-04-22T17:41:40.081585+0000 mgr.y (mgr.34107) 588 : cluster [DBG] pgmap v1110: 137 pgs: 137 active+clean; 1.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 4.46k op/s 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[45495]: audit 2022-04-22T17:41:40.112593+0000 mgr.y (mgr.34107) 589 : audit [DBG] from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:41:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[45495]: audit 2022-04-22T17:41:40.582357+0000 mon.a (mon.0) 1742 : audit [DBG] from='client.? 172.21.15.115:0/3458347188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:41:42.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:41.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:43 smithi121 bash[36902]: cluster 2022-04-22T17:41:42.082546+0000 mgr.y (mgr.34107) 590 : cluster [DBG] pgmap v1111: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 3.00k op/s 2022-04-22T17:41:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:43 smithi115 bash[45495]: cluster 2022-04-22T17:41:42.082546+0000 mgr.y (mgr.34107) 590 : cluster [DBG] pgmap v1111: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 3.00k op/s 2022-04-22T17:41:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:43 smithi115 bash[46934]: cluster 2022-04-22T17:41:42.082546+0000 mgr.y (mgr.34107) 590 : cluster [DBG] pgmap v1111: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 3.00k op/s 2022-04-22T17:41:43.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:43.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:45 smithi121 bash[36902]: cluster 2022-04-22T17:41:44.083329+0000 mgr.y (mgr.34107) 591 : cluster [DBG] pgmap v1112: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.3 MiB/s wr, 4.08k op/s 2022-04-22T17:41:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:45 smithi121 bash[36902]: audit 2022-04-22T17:41:44.580265+0000 mon.a (mon.0) 1743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:45 smithi115 bash[45495]: cluster 2022-04-22T17:41:44.083329+0000 mgr.y (mgr.34107) 591 : cluster [DBG] pgmap v1112: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.3 MiB/s wr, 4.08k op/s 2022-04-22T17:41:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:45 smithi115 bash[45495]: audit 2022-04-22T17:41:44.580265+0000 mon.a (mon.0) 1743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:45 smithi115 bash[46934]: cluster 2022-04-22T17:41:44.083329+0000 mgr.y (mgr.34107) 591 : cluster [DBG] pgmap v1112: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.3 MiB/s wr, 4.08k op/s 2022-04-22T17:41:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:45 smithi115 bash[46934]: audit 2022-04-22T17:41:44.580265+0000 mon.a (mon.0) 1743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:47 smithi121 bash[36902]: cluster 2022-04-22T17:41:46.084220+0000 mgr.y (mgr.34107) 592 : cluster [DBG] pgmap v1113: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:47.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:47 smithi115 bash[45495]: cluster 2022-04-22T17:41:46.084220+0000 mgr.y (mgr.34107) 592 : cluster [DBG] pgmap v1113: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:47.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:47 smithi115 bash[46934]: cluster 2022-04-22T17:41:46.084220+0000 mgr.y (mgr.34107) 592 : cluster [DBG] pgmap v1113: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:49 smithi121 bash[36902]: cluster 2022-04-22T17:41:48.085194+0000 mgr.y (mgr.34107) 593 : cluster [DBG] pgmap v1114: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:49.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:49 smithi115 bash[45495]: cluster 2022-04-22T17:41:48.085194+0000 mgr.y (mgr.34107) 593 : cluster [DBG] pgmap v1114: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:49.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:49 smithi115 bash[46934]: cluster 2022-04-22T17:41:48.085194+0000 mgr.y (mgr.34107) 593 : cluster [DBG] pgmap v1114: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 2.48k op/s 2022-04-22T17:41:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:50 smithi121 bash[36902]: audit 2022-04-22T17:41:49.582961+0000 mon.a (mon.0) 1744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:50.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:50 smithi115 bash[46934]: audit 2022-04-22T17:41:49.582961+0000 mon.a (mon.0) 1744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:50.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:50 smithi115 bash[45495]: audit 2022-04-22T17:41:49.582961+0000 mon.a (mon.0) 1744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:51 smithi121 bash[36902]: cluster 2022-04-22T17:41:50.086076+0000 mgr.y (mgr.34107) 594 : cluster [DBG] pgmap v1115: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 616 KiB/s rd, 2.5 MiB/s wr, 3.18k op/s 2022-04-22T17:41:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:51 smithi115 bash[45495]: cluster 2022-04-22T17:41:50.086076+0000 mgr.y (mgr.34107) 594 : cluster [DBG] pgmap v1115: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 616 KiB/s rd, 2.5 MiB/s wr, 3.18k op/s 2022-04-22T17:41:51.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:51 smithi115 bash[46934]: cluster 2022-04-22T17:41:50.086076+0000 mgr.y (mgr.34107) 594 : cluster [DBG] pgmap v1115: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 616 KiB/s rd, 2.5 MiB/s wr, 3.18k op/s 2022-04-22T17:41:52.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:51.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:53 smithi121 bash[36902]: cluster 2022-04-22T17:41:52.087066+0000 mgr.y (mgr.34107) 595 : cluster [DBG] pgmap v1116: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 1.4 MiB/s wr, 1.78k op/s 2022-04-22T17:41:53.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:53 smithi115 bash[45495]: cluster 2022-04-22T17:41:52.087066+0000 mgr.y (mgr.34107) 595 : cluster [DBG] pgmap v1116: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 1.4 MiB/s wr, 1.78k op/s 2022-04-22T17:41:53.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:53 smithi115 bash[46934]: cluster 2022-04-22T17:41:52.087066+0000 mgr.y (mgr.34107) 595 : cluster [DBG] pgmap v1116: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 1.4 MiB/s wr, 1.78k op/s 2022-04-22T17:41:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:41:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:41:53.534Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:41:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:55 smithi121 bash[36902]: cluster 2022-04-22T17:41:54.088009+0000 mgr.y (mgr.34107) 596 : cluster [DBG] pgmap v1117: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 569 KiB/s rd, 1.9 MiB/s wr, 2.48k op/s 2022-04-22T17:41:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:55 smithi121 bash[36902]: audit 2022-04-22T17:41:54.585734+0000 mon.a (mon.0) 1745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:55 smithi115 bash[45495]: cluster 2022-04-22T17:41:54.088009+0000 mgr.y (mgr.34107) 596 : cluster [DBG] pgmap v1117: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 569 KiB/s rd, 1.9 MiB/s wr, 2.48k op/s 2022-04-22T17:41:55.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:55 smithi115 bash[45495]: audit 2022-04-22T17:41:54.585734+0000 mon.a (mon.0) 1745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:55 smithi115 bash[46934]: cluster 2022-04-22T17:41:54.088009+0000 mgr.y (mgr.34107) 596 : cluster [DBG] pgmap v1117: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 569 KiB/s rd, 1.9 MiB/s wr, 2.48k op/s 2022-04-22T17:41:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:55 smithi115 bash[46934]: audit 2022-04-22T17:41:54.585734+0000 mon.a (mon.0) 1745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:41:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:57 smithi121 bash[36902]: cluster 2022-04-22T17:41:56.088853+0000 mgr.y (mgr.34107) 597 : cluster [DBG] pgmap v1118: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:41:57.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:57 smithi115 bash[45495]: cluster 2022-04-22T17:41:56.088853+0000 mgr.y (mgr.34107) 597 : cluster [DBG] pgmap v1118: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:41:57.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:57 smithi115 bash[46934]: cluster 2022-04-22T17:41:56.088853+0000 mgr.y (mgr.34107) 597 : cluster [DBG] pgmap v1118: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:41:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:41:59 smithi121 bash[36902]: cluster 2022-04-22T17:41:58.089801+0000 mgr.y (mgr.34107) 598 : cluster [DBG] pgmap v1119: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:41:59.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:41:59 smithi115 bash[45495]: cluster 2022-04-22T17:41:58.089801+0000 mgr.y (mgr.34107) 598 : cluster [DBG] pgmap v1119: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:41:59.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:41:59 smithi115 bash[46934]: cluster 2022-04-22T17:41:58.089801+0000 mgr.y (mgr.34107) 598 : cluster [DBG] pgmap v1119: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 KiB/s rd, 1.0 MiB/s wr, 1.40k op/s 2022-04-22T17:42:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:00 smithi121 bash[36902]: audit 2022-04-22T17:41:59.588066+0000 mon.a (mon.0) 1746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:00.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:00 smithi115 bash[45495]: audit 2022-04-22T17:41:59.588066+0000 mon.a (mon.0) 1746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:00.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:00 smithi115 bash[46934]: audit 2022-04-22T17:41:59.588066+0000 mon.a (mon.0) 1746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:01 smithi121 bash[36902]: cluster 2022-04-22T17:42:00.090461+0000 mgr.y (mgr.34107) 599 : cluster [DBG] pgmap v1120: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 2.14k op/s 2022-04-22T17:42:01.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:01 smithi115 bash[46934]: cluster 2022-04-22T17:42:00.090461+0000 mgr.y (mgr.34107) 599 : cluster [DBG] pgmap v1120: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 2.14k op/s 2022-04-22T17:42:01.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:01 smithi115 bash[45495]: cluster 2022-04-22T17:42:00.090461+0000 mgr.y (mgr.34107) 599 : cluster [DBG] pgmap v1120: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 2.14k op/s 2022-04-22T17:42:02.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:01.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:03 smithi121 bash[36902]: cluster 2022-04-22T17:42:02.091326+0000 mgr.y (mgr.34107) 600 : cluster [DBG] pgmap v1121: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 1.44k op/s 2022-04-22T17:42:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:03 smithi115 bash[45495]: cluster 2022-04-22T17:42:02.091326+0000 mgr.y (mgr.34107) 600 : cluster [DBG] pgmap v1121: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 1.44k op/s 2022-04-22T17:42:03.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:03 smithi115 bash[46934]: cluster 2022-04-22T17:42:02.091326+0000 mgr.y (mgr.34107) 600 : cluster [DBG] pgmap v1121: 137 pgs: 137 active+clean; 1.6 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 1.44k op/s 2022-04-22T17:42:03.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:03.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:04.233 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (40998 ms) 2022-04-22T17:42:04.233 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2022-04-22T17:42:05.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:05 smithi121 bash[36902]: cluster 2022-04-22T17:42:04.092459+0000 mgr.y (mgr.34107) 601 : cluster [DBG] pgmap v1122: 137 pgs: 137 active+clean; 1.6 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 2.12k op/s 2022-04-22T17:42:05.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:05 smithi121 bash[36902]: cluster 2022-04-22T17:42:04.211350+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2022-04-22T17:42:05.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:05 smithi121 bash[36902]: audit 2022-04-22T17:42:04.676318+0000 mon.a (mon.0) 1748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:05.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[45495]: cluster 2022-04-22T17:42:04.092459+0000 mgr.y (mgr.34107) 601 : cluster [DBG] pgmap v1122: 137 pgs: 137 active+clean; 1.6 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 2.12k op/s 2022-04-22T17:42:05.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[45495]: cluster 2022-04-22T17:42:04.211350+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2022-04-22T17:42:05.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[45495]: audit 2022-04-22T17:42:04.676318+0000 mon.a (mon.0) 1748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[46934]: cluster 2022-04-22T17:42:04.092459+0000 mgr.y (mgr.34107) 601 : cluster [DBG] pgmap v1122: 137 pgs: 137 active+clean; 1.6 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 2.12k op/s 2022-04-22T17:42:05.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[46934]: cluster 2022-04-22T17:42:04.211350+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2022-04-22T17:42:05.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:05 smithi115 bash[46934]: audit 2022-04-22T17:42:04.676318+0000 mon.a (mon.0) 1748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:06 smithi121 bash[36902]: cluster 2022-04-22T17:42:05.223832+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2022-04-22T17:42:06.547 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:06 smithi121 bash[36902]: audit 2022-04-22T17:42:05.225783+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.115:0/1910273618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.547 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:06 smithi121 bash[36902]: audit 2022-04-22T17:42:05.227272+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.547 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:06 smithi121 bash[36902]: cluster 2022-04-22T17:42:06.093580+0000 mgr.y (mgr.34107) 602 : cluster [DBG] pgmap v1125: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[46934]: cluster 2022-04-22T17:42:05.223832+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2022-04-22T17:42:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[46934]: audit 2022-04-22T17:42:05.225783+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.115:0/1910273618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[46934]: audit 2022-04-22T17:42:05.227272+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[46934]: cluster 2022-04-22T17:42:06.093580+0000 mgr.y (mgr.34107) 602 : cluster [DBG] pgmap v1125: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:06.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[45495]: cluster 2022-04-22T17:42:05.223832+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2022-04-22T17:42:06.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[45495]: audit 2022-04-22T17:42:05.225783+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.115:0/1910273618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[45495]: audit 2022-04-22T17:42:05.227272+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:06.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:06 smithi115 bash[45495]: cluster 2022-04-22T17:42:06.093580+0000 mgr.y (mgr.34107) 602 : cluster [DBG] pgmap v1125: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:07 smithi121 bash[36902]: audit 2022-04-22T17:42:06.234346+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:07 smithi121 bash[36902]: cluster 2022-04-22T17:42:06.234619+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2022-04-22T17:42:07.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:07 smithi115 bash[45495]: audit 2022-04-22T17:42:06.234346+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:07.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:07 smithi115 bash[45495]: cluster 2022-04-22T17:42:06.234619+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2022-04-22T17:42:07.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:07 smithi115 bash[46934]: audit 2022-04-22T17:42:06.234346+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:07.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:07 smithi115 bash[46934]: cluster 2022-04-22T17:42:06.234619+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2022-04-22T17:42:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:08 smithi121 bash[36902]: cluster 2022-04-22T17:42:07.240358+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2022-04-22T17:42:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:08 smithi121 bash[36902]: cluster 2022-04-22T17:42:08.094362+0000 mgr.y (mgr.34107) 603 : cluster [DBG] pgmap v1128: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:08.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:08 smithi115 bash[45495]: cluster 2022-04-22T17:42:07.240358+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2022-04-22T17:42:08.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:08 smithi115 bash[45495]: cluster 2022-04-22T17:42:08.094362+0000 mgr.y (mgr.34107) 603 : cluster [DBG] pgmap v1128: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:08.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:08 smithi115 bash[46934]: cluster 2022-04-22T17:42:07.240358+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2022-04-22T17:42:08.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:08 smithi115 bash[46934]: cluster 2022-04-22T17:42:08.094362+0000 mgr.y (mgr.34107) 603 : cluster [DBG] pgmap v1128: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:09.274 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (5041 ms) 2022-04-22T17:42:09.274 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2022-04-22T17:42:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[45495]: cluster 2022-04-22T17:42:09.257912+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2022-04-22T17:42:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[45495]: audit 2022-04-22T17:42:09.733221+0000 mon.a (mon.0) 1755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:10.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[45495]: cluster 2022-04-22T17:42:10.095238+0000 mgr.y (mgr.34107) 604 : cluster [DBG] pgmap v1130: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:10.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[46934]: cluster 2022-04-22T17:42:09.257912+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2022-04-22T17:42:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[46934]: audit 2022-04-22T17:42:09.733221+0000 mon.a (mon.0) 1755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:10 smithi115 bash[46934]: cluster 2022-04-22T17:42:10.095238+0000 mgr.y (mgr.34107) 604 : cluster [DBG] pgmap v1130: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:10 smithi121 bash[36902]: cluster 2022-04-22T17:42:09.257912+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2022-04-22T17:42:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:10 smithi121 bash[36902]: audit 2022-04-22T17:42:09.733221+0000 mon.a (mon.0) 1755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:10 smithi121 bash[36902]: cluster 2022-04-22T17:42:10.095238+0000 mgr.y (mgr.34107) 604 : cluster [DBG] pgmap v1130: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:10.966 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:42:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:42:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5m) 5m ago 19m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (18m) 5m ago 18m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (7m) 5m ago 20m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6m) 5m ago 20m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (14m) 5m ago 24m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (15m) 5m ago 26m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (14m) 5m ago 26m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (13m) 5m ago 24m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (13m) 5m ago 24m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 5m ago 19m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:42:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6m) 5m ago 19m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (11m) 5m ago 23m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 5m ago 23m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 5m ago 22m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (10m) 5m ago 22m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (9m) 5m ago 22m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (9m) 5m ago 21m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (8m) 5m ago 21m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (7m) 5m ago 20m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:42:11.333 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5m) 5m ago 19m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:42:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[46934]: cluster 2022-04-22T17:42:10.269721+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2022-04-22T17:42:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[46934]: audit 2022-04-22T17:42:10.274998+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.115:0/3231816925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[46934]: audit 2022-04-22T17:42:10.276509+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[46934]: audit 2022-04-22T17:42:10.938440+0000 mgr.y (mgr.34107) 605 : audit [DBG] from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:11.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[45495]: cluster 2022-04-22T17:42:10.269721+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2022-04-22T17:42:11.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[45495]: audit 2022-04-22T17:42:10.274998+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.115:0/3231816925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[45495]: audit 2022-04-22T17:42:10.276509+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[45495]: audit 2022-04-22T17:42:10.938440+0000 mgr.y (mgr.34107) 605 : audit [DBG] from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:11.739 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:42:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:11.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:42:11.741 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:42:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:11 smithi121 bash[36902]: cluster 2022-04-22T17:42:10.269721+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2022-04-22T17:42:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:11 smithi121 bash[36902]: audit 2022-04-22T17:42:10.274998+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.115:0/3231816925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:11 smithi121 bash[36902]: audit 2022-04-22T17:42:10.276509+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:11 smithi121 bash[36902]: audit 2022-04-22T17:42:10.938440+0000 mgr.y (mgr.34107) 605 : audit [DBG] from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:12.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:11.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:12.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[45495]: audit 2022-04-22T17:42:11.280589+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:12.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[45495]: cluster 2022-04-22T17:42:11.280755+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2022-04-22T17:42:12.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[45495]: audit 2022-04-22T17:42:11.304285+0000 mgr.y (mgr.34107) 606 : audit [DBG] from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:12.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[45495]: audit 2022-04-22T17:42:11.725640+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.115:0/1299388330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[45495]: cluster 2022-04-22T17:42:12.096326+0000 mgr.y (mgr.34107) 607 : cluster [DBG] pgmap v1133: 137 pgs: 7 creating+peering, 18 unknown, 112 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 381 KiB/s rd, 383 KiB/s wr, 214 op/s 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[46934]: audit 2022-04-22T17:42:11.280589+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[46934]: cluster 2022-04-22T17:42:11.280755+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[46934]: audit 2022-04-22T17:42:11.304285+0000 mgr.y (mgr.34107) 606 : audit [DBG] from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[46934]: audit 2022-04-22T17:42:11.725640+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.115:0/1299388330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:12.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:12 smithi115 bash[46934]: cluster 2022-04-22T17:42:12.096326+0000 mgr.y (mgr.34107) 607 : cluster [DBG] pgmap v1133: 137 pgs: 7 creating+peering, 18 unknown, 112 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 381 KiB/s rd, 383 KiB/s wr, 214 op/s 2022-04-22T17:42:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:12 smithi121 bash[36902]: audit 2022-04-22T17:42:11.280589+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:12 smithi121 bash[36902]: cluster 2022-04-22T17:42:11.280755+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2022-04-22T17:42:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:12 smithi121 bash[36902]: audit 2022-04-22T17:42:11.304285+0000 mgr.y (mgr.34107) 606 : audit [DBG] from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:12 smithi121 bash[36902]: audit 2022-04-22T17:42:11.725640+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.115:0/1299388330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:12 smithi121 bash[36902]: cluster 2022-04-22T17:42:12.096326+0000 mgr.y (mgr.34107) 607 : cluster [DBG] pgmap v1133: 137 pgs: 7 creating+peering, 18 unknown, 112 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 381 KiB/s rd, 383 KiB/s wr, 214 op/s 2022-04-22T17:42:13.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:13 smithi115 bash[46934]: cluster 2022-04-22T17:42:12.287123+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2022-04-22T17:42:13.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:13 smithi115 bash[45495]: cluster 2022-04-22T17:42:12.287123+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2022-04-22T17:42:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:13 smithi121 bash[36902]: cluster 2022-04-22T17:42:12.287123+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2022-04-22T17:42:13.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:13.536Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:14 smithi121 bash[36902]: audit 2022-04-22T17:42:13.378170+0000 mon.a (mon.0) 1761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:42:14.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:14 smithi121 bash[36902]: audit 2022-04-22T17:42:13.378685+0000 mon.a (mon.0) 1762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:42:14.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:14 smithi121 bash[36902]: cluster 2022-04-22T17:42:14.097406+0000 mgr.y (mgr.34107) 608 : cluster [DBG] pgmap v1135: 137 pgs: 7 creating+peering, 9 unknown, 121 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 384 KiB/s wr, 215 op/s 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[46934]: audit 2022-04-22T17:42:13.378170+0000 mon.a (mon.0) 1761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[46934]: audit 2022-04-22T17:42:13.378685+0000 mon.a (mon.0) 1762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[46934]: cluster 2022-04-22T17:42:14.097406+0000 mgr.y (mgr.34107) 608 : cluster [DBG] pgmap v1135: 137 pgs: 7 creating+peering, 9 unknown, 121 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 384 KiB/s wr, 215 op/s 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[45495]: audit 2022-04-22T17:42:13.378170+0000 mon.a (mon.0) 1761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[45495]: audit 2022-04-22T17:42:13.378685+0000 mon.a (mon.0) 1762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:42:14.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:14 smithi115 bash[45495]: cluster 2022-04-22T17:42:14.097406+0000 mgr.y (mgr.34107) 608 : cluster [DBG] pgmap v1135: 137 pgs: 7 creating+peering, 9 unknown, 121 active+clean; 591 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 384 KiB/s wr, 215 op/s 2022-04-22T17:42:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:15 smithi121 bash[36902]: audit 2022-04-22T17:42:14.597570+0000 mon.a (mon.0) 1763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:15.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:15 smithi115 bash[45495]: audit 2022-04-22T17:42:14.597570+0000 mon.a (mon.0) 1763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:15.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:15 smithi115 bash[46934]: audit 2022-04-22T17:42:14.597570+0000 mon.a (mon.0) 1763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:16 smithi121 bash[36902]: cluster 2022-04-22T17:42:16.098588+0000 mgr.y (mgr.34107) 609 : cluster [DBG] pgmap v1136: 137 pgs: 137 active+clean; 591 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 310 KiB/s wr, 173 op/s 2022-04-22T17:42:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:16 smithi115 bash[45495]: cluster 2022-04-22T17:42:16.098588+0000 mgr.y (mgr.34107) 609 : cluster [DBG] pgmap v1136: 137 pgs: 137 active+clean; 591 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 310 KiB/s wr, 173 op/s 2022-04-22T17:42:16.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:16 smithi115 bash[46934]: cluster 2022-04-22T17:42:16.098588+0000 mgr.y (mgr.34107) 609 : cluster [DBG] pgmap v1136: 137 pgs: 137 active+clean; 591 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 310 KiB/s wr, 173 op/s 2022-04-22T17:42:17.330 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (8056 ms) 2022-04-22T17:42:17.330 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2022-04-22T17:42:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:18 smithi121 bash[36902]: cluster 2022-04-22T17:42:17.313825+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2022-04-22T17:42:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:18 smithi121 bash[36902]: cluster 2022-04-22T17:42:18.099552+0000 mgr.y (mgr.34107) 610 : cluster [DBG] pgmap v1138: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:18.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:18 smithi115 bash[45495]: cluster 2022-04-22T17:42:17.313825+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2022-04-22T17:42:18.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:18 smithi115 bash[45495]: cluster 2022-04-22T17:42:18.099552+0000 mgr.y (mgr.34107) 610 : cluster [DBG] pgmap v1138: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:18.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:18 smithi115 bash[46934]: cluster 2022-04-22T17:42:17.313825+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2022-04-22T17:42:18.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:18 smithi115 bash[46934]: cluster 2022-04-22T17:42:18.099552+0000 mgr.y (mgr.34107) 610 : cluster [DBG] pgmap v1138: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:19 smithi121 bash[36902]: cluster 2022-04-22T17:42:18.328612+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2022-04-22T17:42:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:19 smithi121 bash[36902]: audit 2022-04-22T17:42:18.330853+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.115:0/4234234343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:19 smithi121 bash[36902]: audit 2022-04-22T17:42:18.331784+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:19 smithi121 bash[36902]: audit 2022-04-22T17:42:18.602095+0000 mon.a (mon.0) 1767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[45495]: cluster 2022-04-22T17:42:18.328612+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2022-04-22T17:42:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[45495]: audit 2022-04-22T17:42:18.330853+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.115:0/4234234343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[45495]: audit 2022-04-22T17:42:18.331784+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[45495]: audit 2022-04-22T17:42:18.602095+0000 mon.a (mon.0) 1767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:19.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[46934]: cluster 2022-04-22T17:42:18.328612+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2022-04-22T17:42:19.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[46934]: audit 2022-04-22T17:42:18.330853+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.115:0/4234234343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[46934]: audit 2022-04-22T17:42:18.331784+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:19.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:19 smithi115 bash[46934]: audit 2022-04-22T17:42:18.602095+0000 mon.a (mon.0) 1767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:20 smithi121 bash[36902]: audit 2022-04-22T17:42:19.333677+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:20 smithi121 bash[36902]: cluster 2022-04-22T17:42:19.334061+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2022-04-22T17:42:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:20 smithi121 bash[36902]: audit 2022-04-22T17:42:19.600311+0000 mon.a (mon.0) 1770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:20 smithi121 bash[36902]: cluster 2022-04-22T17:42:20.100261+0000 mgr.y (mgr.34107) 611 : cluster [DBG] pgmap v1141: 137 pgs: 4 creating+peering, 25 unknown, 108 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[46934]: audit 2022-04-22T17:42:19.333677+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[46934]: cluster 2022-04-22T17:42:19.334061+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2022-04-22T17:42:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[46934]: audit 2022-04-22T17:42:19.600311+0000 mon.a (mon.0) 1770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:20.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[46934]: cluster 2022-04-22T17:42:20.100261+0000 mgr.y (mgr.34107) 611 : cluster [DBG] pgmap v1141: 137 pgs: 4 creating+peering, 25 unknown, 108 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:20.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[45495]: audit 2022-04-22T17:42:19.333677+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-62331-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:20.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[45495]: cluster 2022-04-22T17:42:19.334061+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2022-04-22T17:42:20.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[45495]: audit 2022-04-22T17:42:19.600311+0000 mon.a (mon.0) 1770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:20.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:20 smithi115 bash[45495]: cluster 2022-04-22T17:42:20.100261+0000 mgr.y (mgr.34107) 611 : cluster [DBG] pgmap v1141: 137 pgs: 4 creating+peering, 25 unknown, 108 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:21.668 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:21 smithi115 bash[46934]: cluster 2022-04-22T17:42:20.341557+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2022-04-22T17:42:21.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:21 smithi115 bash[45495]: cluster 2022-04-22T17:42:20.341557+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2022-04-22T17:42:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:21 smithi121 bash[36902]: cluster 2022-04-22T17:42:20.341557+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2022-04-22T17:42:22.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:21.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:22 smithi121 bash[36902]: cluster 2022-04-22T17:42:22.101103+0000 mgr.y (mgr.34107) 612 : cluster [DBG] pgmap v1143: 137 pgs: 4 creating+peering, 8 unknown, 125 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:22.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:22 smithi115 bash[45495]: cluster 2022-04-22T17:42:22.101103+0000 mgr.y (mgr.34107) 612 : cluster [DBG] pgmap v1143: 137 pgs: 4 creating+peering, 8 unknown, 125 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:22 smithi115 bash[46934]: cluster 2022-04-22T17:42:22.101103+0000 mgr.y (mgr.34107) 612 : cluster [DBG] pgmap v1143: 137 pgs: 4 creating+peering, 8 unknown, 125 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:23.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:23.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (7848 ms) 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 22 tests from TestCls2PCQueue (146812 ms total) 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 22 tests from 1 test suite ran. (146812 ms total) 2022-04-22T17:42:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 22 tests. 2022-04-22T17:42:25.182 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:42:25.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:42:25.196 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2022-04-22T17:42:25.196 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2022-04-22T17:42:25.212 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:42:25.212 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 31 tests from 1 test suite. 2022-04-22T17:42:25.212 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:42:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:25 smithi121 bash[36902]: cluster 2022-04-22T17:42:24.101870+0000 mgr.y (mgr.34107) 613 : cluster [DBG] pgmap v1144: 137 pgs: 137 active+clean; 605 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 802 op/s 2022-04-22T17:42:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:25 smithi121 bash[36902]: audit 2022-04-22T17:42:24.603554+0000 mon.a (mon.0) 1772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:25 smithi115 bash[45495]: cluster 2022-04-22T17:42:24.101870+0000 mgr.y (mgr.34107) 613 : cluster [DBG] pgmap v1144: 137 pgs: 137 active+clean; 605 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 802 op/s 2022-04-22T17:42:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:25 smithi115 bash[45495]: audit 2022-04-22T17:42:24.603554+0000 mon.a (mon.0) 1772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:25.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:25 smithi115 bash[46934]: cluster 2022-04-22T17:42:24.101870+0000 mgr.y (mgr.34107) 613 : cluster [DBG] pgmap v1144: 137 pgs: 137 active+clean; 605 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 802 op/s 2022-04-22T17:42:25.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:25 smithi115 bash[46934]: audit 2022-04-22T17:42:24.603554+0000 mon.a (mon.0) 1772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:26.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[45495]: cluster 2022-04-22T17:42:25.160863+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2022-04-22T17:42:26.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[45495]: audit 2022-04-22T17:42:26.145091+0000 mon.a (mon.0) 1774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[45495]: audit 2022-04-22T17:42:26.148178+0000 mon.a (mon.0) 1775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[45495]: audit 2022-04-22T17:42:26.149898+0000 mon.a (mon.0) 1776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[46934]: cluster 2022-04-22T17:42:25.160863+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[46934]: audit 2022-04-22T17:42:26.145091+0000 mon.a (mon.0) 1774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[46934]: audit 2022-04-22T17:42:26.148178+0000 mon.a (mon.0) 1775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:42:26.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:26 smithi115 bash[46934]: audit 2022-04-22T17:42:26.149898+0000 mon.a (mon.0) 1776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:42:26.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:26 smithi121 bash[36902]: cluster 2022-04-22T17:42:25.160863+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2022-04-22T17:42:26.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:26 smithi121 bash[36902]: audit 2022-04-22T17:42:26.145091+0000 mon.a (mon.0) 1774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:42:26.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:26 smithi121 bash[36902]: audit 2022-04-22T17:42:26.148178+0000 mon.a (mon.0) 1775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:42:26.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:26 smithi121 bash[36902]: audit 2022-04-22T17:42:26.149898+0000 mon.a (mon.0) 1776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:42:27.213 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 31 tests from CmpOmap 2022-04-22T17:42:27.255 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (20 ms) 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (3 ms) 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (3 ms) 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (5 ms) 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_str 2022-04-22T17:42:27.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_str (12 ms) 2022-04-22T17:42:27.257 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2022-04-22T17:42:27.270 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_u64 (14 ms) 2022-04-22T17:42:27.270 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2022-04-22T17:42:27.277 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (7 ms) 2022-04-22T17:42:27.277 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2022-04-22T17:42:27.283 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (6 ms) 2022-04-22T17:42:27.283 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_u64_empty_default 2022-04-22T17:42:27.288 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_u64_empty_default (5 ms) 2022-04-22T17:42:27.288 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2022-04-22T17:42:27.297 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (9 ms) 2022-04-22T17:42:27.297 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2022-04-22T17:42:27.299 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (2 ms) 2022-04-22T17:42:27.299 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2022-04-22T17:42:27.300 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (1 ms) 2022-04-22T17:42:27.301 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2022-04-22T17:42:27.312 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (12 ms) 2022-04-22T17:42:27.312 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2022-04-22T17:42:27.323 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (11 ms) 2022-04-22T17:42:27.323 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2022-04-22T17:42:27.335 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (12 ms) 2022-04-22T17:42:27.335 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2022-04-22T17:42:27.348 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (13 ms) 2022-04-22T17:42:27.348 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2022-04-22T17:42:27.361 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_str (13 ms) 2022-04-22T17:42:27.362 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2022-04-22T17:42:27.375 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (14 ms) 2022-04-22T17:42:27.375 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2022-04-22T17:42:27.379 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (4 ms) 2022-04-22T17:42:27.380 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2022-04-22T17:42:27.384 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (5 ms) 2022-04-22T17:42:27.384 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2022-04-22T17:42:27.386 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (2 ms) 2022-04-22T17:42:27.386 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2022-04-22T17:42:27.387 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (1 ms) 2022-04-22T17:42:27.387 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2022-04-22T17:42:27.399 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (12 ms) 2022-04-22T17:42:27.399 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2022-04-22T17:42:27.410 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (11 ms) 2022-04-22T17:42:27.410 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2022-04-22T17:42:27.425 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (15 ms) 2022-04-22T17:42:27.425 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2022-04-22T17:42:27.439 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (14 ms) 2022-04-22T17:42:27.439 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2022-04-22T17:42:27.443 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (4 ms) 2022-04-22T17:42:27.443 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2022-04-22T17:42:27.447 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (4 ms) 2022-04-22T17:42:27.447 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2022-04-22T17:42:27.450 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (3 ms) 2022-04-22T17:42:27.450 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2022-04-22T17:42:27.451 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (1 ms) 2022-04-22T17:42:27.451 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_empty 2022-04-22T17:42:27.466 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_empty (15 ms) 2022-04-22T17:42:27.466 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 31 tests from CmpOmap (253 ms total) 2022-04-22T17:42:27.466 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:42:27.466 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:42:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:27 smithi121 bash[36902]: cluster 2022-04-22T17:42:26.102686+0000 mgr.y (mgr.34107) 614 : cluster [DBG] pgmap v1146: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:27 smithi121 bash[36902]: cluster 2022-04-22T17:42:26.193056+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2022-04-22T17:42:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:27 smithi121 bash[36902]: audit 2022-04-22T17:42:26.200770+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:27 smithi121 bash[36902]: audit 2022-04-22T17:42:26.498727+0000 mon.a (mon.0) 1779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:27.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[45495]: cluster 2022-04-22T17:42:26.102686+0000 mgr.y (mgr.34107) 614 : cluster [DBG] pgmap v1146: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:27.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[45495]: cluster 2022-04-22T17:42:26.193056+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2022-04-22T17:42:27.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[45495]: audit 2022-04-22T17:42:26.200770+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:27.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[45495]: audit 2022-04-22T17:42:26.498727+0000 mon.a (mon.0) 1779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[46934]: cluster 2022-04-22T17:42:26.102686+0000 mgr.y (mgr.34107) 614 : cluster [DBG] pgmap v1146: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[46934]: cluster 2022-04-22T17:42:26.193056+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2022-04-22T17:42:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[46934]: audit 2022-04-22T17:42:26.200770+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:27 smithi115 bash[46934]: audit 2022-04-22T17:42:26.498727+0000 mon.a (mon.0) 1779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:28.235 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 31 tests from 1 test suite ran. (3024 ms total) 2022-04-22T17:42:28.235 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 31 tests. 2022-04-22T17:42:28.238 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:42:28.238 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:42:28.296 INFO:tasks.workunit:Running workunit cls/test_cls_hello.sh... 2022-04-22T17:42:28.297 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_hello.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_hello.sh 2022-04-22T17:42:28.318 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:42:28.319 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 6 tests from 1 test suite. 2022-04-22T17:42:28.319 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:42:28.319 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 6 tests from ClsHello 2022-04-22T17:42:28.319 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.SayHello 2022-04-22T17:42:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:28 smithi121 bash[36902]: audit 2022-04-22T17:42:27.196318+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:28 smithi121 bash[36902]: cluster 2022-04-22T17:42:27.196408+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2022-04-22T17:42:28.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:28 smithi115 bash[45495]: audit 2022-04-22T17:42:27.196318+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:28.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:28 smithi115 bash[45495]: cluster 2022-04-22T17:42:27.196408+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2022-04-22T17:42:28.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:28 smithi115 bash[46934]: audit 2022-04-22T17:42:27.196318+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.115:0/96954486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63198-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:28.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:28 smithi115 bash[46934]: cluster 2022-04-22T17:42:27.196408+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e935: 8 total, 8 up, 8 in 2022-04-22T17:42:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:29 smithi121 bash[36902]: cluster 2022-04-22T17:42:28.103622+0000 mgr.y (mgr.34107) 615 : cluster [DBG] pgmap v1149: 137 pgs: 7 creating+peering, 19 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 3 op/s 2022-04-22T17:42:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:29 smithi121 bash[36902]: cluster 2022-04-22T17:42:28.213009+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e936: 8 total, 8 up, 8 in 2022-04-22T17:42:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:29 smithi121 bash[36902]: audit 2022-04-22T17:42:28.774910+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[45495]: cluster 2022-04-22T17:42:28.103622+0000 mgr.y (mgr.34107) 615 : cluster [DBG] pgmap v1149: 137 pgs: 7 creating+peering, 19 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 3 op/s 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[45495]: cluster 2022-04-22T17:42:28.213009+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e936: 8 total, 8 up, 8 in 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[45495]: audit 2022-04-22T17:42:28.774910+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[46934]: cluster 2022-04-22T17:42:28.103622+0000 mgr.y (mgr.34107) 615 : cluster [DBG] pgmap v1149: 137 pgs: 7 creating+peering, 19 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 3 op/s 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[46934]: cluster 2022-04-22T17:42:28.213009+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e936: 8 total, 8 up, 8 in 2022-04-22T17:42:29.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:29 smithi115 bash[46934]: audit 2022-04-22T17:42:28.774910+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:30 smithi121 bash[36902]: cluster 2022-04-22T17:42:29.236197+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e937: 8 total, 8 up, 8 in 2022-04-22T17:42:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:30 smithi121 bash[36902]: audit 2022-04-22T17:42:29.237420+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:30 smithi121 bash[36902]: audit 2022-04-22T17:42:29.606863+0000 mon.a (mon.0) 1786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:30 smithi121 bash[36902]: cluster 2022-04-22T17:42:30.104457+0000 mgr.y (mgr.34107) 616 : cluster [DBG] pgmap v1152: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:30.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[45495]: cluster 2022-04-22T17:42:29.236197+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e937: 8 total, 8 up, 8 in 2022-04-22T17:42:30.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[45495]: audit 2022-04-22T17:42:29.237420+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:30.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[45495]: audit 2022-04-22T17:42:29.606863+0000 mon.a (mon.0) 1786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:30.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[45495]: cluster 2022-04-22T17:42:30.104457+0000 mgr.y (mgr.34107) 616 : cluster [DBG] pgmap v1152: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:30.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[46934]: cluster 2022-04-22T17:42:29.236197+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e937: 8 total, 8 up, 8 in 2022-04-22T17:42:30.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[46934]: audit 2022-04-22T17:42:29.237420+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:30.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[46934]: audit 2022-04-22T17:42:29.606863+0000 mon.a (mon.0) 1786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:30.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:30 smithi115 bash[46934]: cluster 2022-04-22T17:42:30.104457+0000 mgr.y (mgr.34107) 616 : cluster [DBG] pgmap v1152: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:31.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.SayHello (2957 ms) 2022-04-22T17:42:31.276 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.RecordHello 2022-04-22T17:42:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:31 smithi121 bash[36902]: audit 2022-04-22T17:42:30.247125+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:31 smithi121 bash[36902]: cluster 2022-04-22T17:42:30.247372+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2022-04-22T17:42:31.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:31 smithi115 bash[46934]: audit 2022-04-22T17:42:30.247125+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:31.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:31 smithi115 bash[46934]: cluster 2022-04-22T17:42:30.247372+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2022-04-22T17:42:31.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:31 smithi115 bash[45495]: audit 2022-04-22T17:42:30.247125+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.115:0/1297278695' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:31.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:31 smithi115 bash[45495]: cluster 2022-04-22T17:42:30.247372+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e938: 8 total, 8 up, 8 in 2022-04-22T17:42:32.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:31.644Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:32.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:32 smithi115 bash[45495]: cluster 2022-04-22T17:42:31.259764+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2022-04-22T17:42:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:32 smithi115 bash[45495]: cluster 2022-04-22T17:42:32.105444+0000 mgr.y (mgr.34107) 617 : cluster [DBG] pgmap v1155: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:32.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:32 smithi115 bash[46934]: cluster 2022-04-22T17:42:31.259764+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2022-04-22T17:42:32.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:32 smithi115 bash[46934]: cluster 2022-04-22T17:42:32.105444+0000 mgr.y (mgr.34107) 617 : cluster [DBG] pgmap v1155: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:32 smithi121 bash[36902]: cluster 2022-04-22T17:42:31.259764+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2022-04-22T17:42:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:32 smithi121 bash[36902]: cluster 2022-04-22T17:42:32.105444+0000 mgr.y (mgr.34107) 617 : cluster [DBG] pgmap v1155: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:33.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:33 smithi115 bash[46934]: cluster 2022-04-22T17:42:32.272893+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2022-04-22T17:42:33.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:33 smithi115 bash[46934]: audit 2022-04-22T17:42:32.284493+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:33.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:33 smithi115 bash[45495]: cluster 2022-04-22T17:42:32.272893+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2022-04-22T17:42:33.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:33 smithi115 bash[45495]: audit 2022-04-22T17:42:32.284493+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:33 smithi121 bash[36902]: cluster 2022-04-22T17:42:32.272893+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2022-04-22T17:42:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:33 smithi121 bash[36902]: audit 2022-04-22T17:42:32.284493+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:34.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:33.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:34.316 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.RecordHello (3041 ms) 2022-04-22T17:42:34.326 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.WriteReturnData 2022-04-22T17:42:34.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[46934]: audit 2022-04-22T17:42:33.275992+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:34.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[46934]: cluster 2022-04-22T17:42:33.276092+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2022-04-22T17:42:34.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[46934]: cluster 2022-04-22T17:42:34.106433+0000 mgr.y (mgr.34107) 618 : cluster [DBG] pgmap v1158: 137 pgs: 10 creating+peering, 12 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:34.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[45495]: audit 2022-04-22T17:42:33.275992+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:34.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[45495]: cluster 2022-04-22T17:42:33.276092+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2022-04-22T17:42:34.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:34 smithi115 bash[45495]: cluster 2022-04-22T17:42:34.106433+0000 mgr.y (mgr.34107) 618 : cluster [DBG] pgmap v1158: 137 pgs: 10 creating+peering, 12 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:34 smithi121 bash[36902]: audit 2022-04-22T17:42:33.275992+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? 172.21.15.115:0/3546563819' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:34.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:34 smithi121 bash[36902]: cluster 2022-04-22T17:42:33.276092+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2022-04-22T17:42:34.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:34 smithi121 bash[36902]: cluster 2022-04-22T17:42:34.106433+0000 mgr.y (mgr.34107) 618 : cluster [DBG] pgmap v1158: 137 pgs: 10 creating+peering, 12 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:35 smithi121 bash[36902]: cluster 2022-04-22T17:42:34.299345+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2022-04-22T17:42:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:35 smithi121 bash[36902]: audit 2022-04-22T17:42:34.762053+0000 mon.a (mon.0) 1795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:35.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:35 smithi115 bash[45495]: cluster 2022-04-22T17:42:34.299345+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2022-04-22T17:42:35.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:35 smithi115 bash[45495]: audit 2022-04-22T17:42:34.762053+0000 mon.a (mon.0) 1795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:35.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:35 smithi115 bash[46934]: cluster 2022-04-22T17:42:34.299345+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2022-04-22T17:42:35.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:35 smithi115 bash[46934]: audit 2022-04-22T17:42:34.762053+0000 mon.a (mon.0) 1795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:36.323 INFO:tasks.workunit.client.0.smithi115.stdout:require_osd_release = quincy 2022-04-22T17:42:36.341 INFO:tasks.workunit.client.0.smithi115.stdout:0 2022-04-22T17:42:36.343 INFO:tasks.workunit.client.0.smithi115.stdout:1 2022-04-22T17:42:36.344 INFO:tasks.workunit.client.0.smithi115.stdout:2 2022-04-22T17:42:36.346 INFO:tasks.workunit.client.0.smithi115.stdout:3 2022-04-22T17:42:36.347 INFO:tasks.workunit.client.0.smithi115.stdout:4 2022-04-22T17:42:36.348 INFO:tasks.workunit.client.0.smithi115.stdout:5 2022-04-22T17:42:36.349 INFO:tasks.workunit.client.0.smithi115.stdout:6 2022-04-22T17:42:36.350 INFO:tasks.workunit.client.0.smithi115.stdout:7 2022-04-22T17:42:36.351 INFO:tasks.workunit.client.0.smithi115.stdout:8 2022-04-22T17:42:36.352 INFO:tasks.workunit.client.0.smithi115.stdout:9 2022-04-22T17:42:36.353 INFO:tasks.workunit.client.0.smithi115.stdout:10 2022-04-22T17:42:36.354 INFO:tasks.workunit.client.0.smithi115.stdout:11 2022-04-22T17:42:36.355 INFO:tasks.workunit.client.0.smithi115.stdout:12 2022-04-22T17:42:36.357 INFO:tasks.workunit.client.0.smithi115.stdout:13 2022-04-22T17:42:36.358 INFO:tasks.workunit.client.0.smithi115.stdout:14 2022-04-22T17:42:36.359 INFO:tasks.workunit.client.0.smithi115.stdout:15 2022-04-22T17:42:36.361 INFO:tasks.workunit.client.0.smithi115.stdout:16 2022-04-22T17:42:36.362 INFO:tasks.workunit.client.0.smithi115.stdout:17 2022-04-22T17:42:36.364 INFO:tasks.workunit.client.0.smithi115.stdout:18 2022-04-22T17:42:36.366 INFO:tasks.workunit.client.0.smithi115.stdout:19 2022-04-22T17:42:36.367 INFO:tasks.workunit.client.0.smithi115.stdout:20 2022-04-22T17:42:36.369 INFO:tasks.workunit.client.0.smithi115.stdout:21 2022-04-22T17:42:36.371 INFO:tasks.workunit.client.0.smithi115.stdout:22 2022-04-22T17:42:36.372 INFO:tasks.workunit.client.0.smithi115.stdout:23 2022-04-22T17:42:36.374 INFO:tasks.workunit.client.0.smithi115.stdout:24 2022-04-22T17:42:36.376 INFO:tasks.workunit.client.0.smithi115.stdout:25 2022-04-22T17:42:36.377 INFO:tasks.workunit.client.0.smithi115.stdout:26 2022-04-22T17:42:36.379 INFO:tasks.workunit.client.0.smithi115.stdout:27 2022-04-22T17:42:36.380 INFO:tasks.workunit.client.0.smithi115.stdout:28 2022-04-22T17:42:36.382 INFO:tasks.workunit.client.0.smithi115.stdout:29 2022-04-22T17:42:36.384 INFO:tasks.workunit.client.0.smithi115.stdout:30 2022-04-22T17:42:36.385 INFO:tasks.workunit.client.0.smithi115.stdout:31 2022-04-22T17:42:36.387 INFO:tasks.workunit.client.0.smithi115.stdout:32 2022-04-22T17:42:36.389 INFO:tasks.workunit.client.0.smithi115.stdout:33 2022-04-22T17:42:36.391 INFO:tasks.workunit.client.0.smithi115.stdout:34 2022-04-22T17:42:36.392 INFO:tasks.workunit.client.0.smithi115.stdout:35 2022-04-22T17:42:36.394 INFO:tasks.workunit.client.0.smithi115.stdout:36 2022-04-22T17:42:36.396 INFO:tasks.workunit.client.0.smithi115.stdout:37 2022-04-22T17:42:36.397 INFO:tasks.workunit.client.0.smithi115.stdout:38 2022-04-22T17:42:36.399 INFO:tasks.workunit.client.0.smithi115.stdout:39 2022-04-22T17:42:36.401 INFO:tasks.workunit.client.0.smithi115.stdout:40 2022-04-22T17:42:36.403 INFO:tasks.workunit.client.0.smithi115.stdout:41 2022-04-22T17:42:36.405 INFO:tasks.workunit.client.0.smithi115.stdout:42 2022-04-22T17:42:36.406 INFO:tasks.workunit.client.0.smithi115.stdout:43 2022-04-22T17:42:36.408 INFO:tasks.workunit.client.0.smithi115.stdout:44 2022-04-22T17:42:36.410 INFO:tasks.workunit.client.0.smithi115.stdout:45 2022-04-22T17:42:36.411 INFO:tasks.workunit.client.0.smithi115.stdout:46 2022-04-22T17:42:36.413 INFO:tasks.workunit.client.0.smithi115.stdout:47 2022-04-22T17:42:36.415 INFO:tasks.workunit.client.0.smithi115.stdout:48 2022-04-22T17:42:36.417 INFO:tasks.workunit.client.0.smithi115.stdout:49 2022-04-22T17:42:36.418 INFO:tasks.workunit.client.0.smithi115.stdout:50 2022-04-22T17:42:36.420 INFO:tasks.workunit.client.0.smithi115.stdout:51 2022-04-22T17:42:36.422 INFO:tasks.workunit.client.0.smithi115.stdout:52 2022-04-22T17:42:36.424 INFO:tasks.workunit.client.0.smithi115.stdout:53 2022-04-22T17:42:36.425 INFO:tasks.workunit.client.0.smithi115.stdout:54 2022-04-22T17:42:36.427 INFO:tasks.workunit.client.0.smithi115.stdout:55 2022-04-22T17:42:36.429 INFO:tasks.workunit.client.0.smithi115.stdout:56 2022-04-22T17:42:36.430 INFO:tasks.workunit.client.0.smithi115.stdout:57 2022-04-22T17:42:36.432 INFO:tasks.workunit.client.0.smithi115.stdout:58 2022-04-22T17:42:36.434 INFO:tasks.workunit.client.0.smithi115.stdout:59 2022-04-22T17:42:36.436 INFO:tasks.workunit.client.0.smithi115.stdout:60 2022-04-22T17:42:36.437 INFO:tasks.workunit.client.0.smithi115.stdout:61 2022-04-22T17:42:36.439 INFO:tasks.workunit.client.0.smithi115.stdout:62 2022-04-22T17:42:36.441 INFO:tasks.workunit.client.0.smithi115.stdout:63 2022-04-22T17:42:36.443 INFO:tasks.workunit.client.0.smithi115.stdout:64 2022-04-22T17:42:36.445 INFO:tasks.workunit.client.0.smithi115.stdout:65 2022-04-22T17:42:36.447 INFO:tasks.workunit.client.0.smithi115.stdout:66 2022-04-22T17:42:36.448 INFO:tasks.workunit.client.0.smithi115.stdout:67 2022-04-22T17:42:36.450 INFO:tasks.workunit.client.0.smithi115.stdout:68 2022-04-22T17:42:36.451 INFO:tasks.workunit.client.0.smithi115.stdout:69 2022-04-22T17:42:36.454 INFO:tasks.workunit.client.0.smithi115.stdout:70 2022-04-22T17:42:36.455 INFO:tasks.workunit.client.0.smithi115.stdout:71 2022-04-22T17:42:36.457 INFO:tasks.workunit.client.0.smithi115.stdout:72 2022-04-22T17:42:36.459 INFO:tasks.workunit.client.0.smithi115.stdout:73 2022-04-22T17:42:36.461 INFO:tasks.workunit.client.0.smithi115.stdout:74 2022-04-22T17:42:36.462 INFO:tasks.workunit.client.0.smithi115.stdout:75 2022-04-22T17:42:36.464 INFO:tasks.workunit.client.0.smithi115.stdout:76 2022-04-22T17:42:36.466 INFO:tasks.workunit.client.0.smithi115.stdout:77 2022-04-22T17:42:36.468 INFO:tasks.workunit.client.0.smithi115.stdout:78 2022-04-22T17:42:36.470 INFO:tasks.workunit.client.0.smithi115.stdout:79 2022-04-22T17:42:36.471 INFO:tasks.workunit.client.0.smithi115.stdout:80 2022-04-22T17:42:36.473 INFO:tasks.workunit.client.0.smithi115.stdout:81 2022-04-22T17:42:36.475 INFO:tasks.workunit.client.0.smithi115.stdout:82 2022-04-22T17:42:36.477 INFO:tasks.workunit.client.0.smithi115.stdout:83 2022-04-22T17:42:36.479 INFO:tasks.workunit.client.0.smithi115.stdout:84 2022-04-22T17:42:36.481 INFO:tasks.workunit.client.0.smithi115.stdout:85 2022-04-22T17:42:36.482 INFO:tasks.workunit.client.0.smithi115.stdout:86 2022-04-22T17:42:36.485 INFO:tasks.workunit.client.0.smithi115.stdout:87 2022-04-22T17:42:36.486 INFO:tasks.workunit.client.0.smithi115.stdout:88 2022-04-22T17:42:36.488 INFO:tasks.workunit.client.0.smithi115.stdout:89 2022-04-22T17:42:36.490 INFO:tasks.workunit.client.0.smithi115.stdout:90 2022-04-22T17:42:36.492 INFO:tasks.workunit.client.0.smithi115.stdout:91 2022-04-22T17:42:36.493 INFO:tasks.workunit.client.0.smithi115.stdout:92 2022-04-22T17:42:36.495 INFO:tasks.workunit.client.0.smithi115.stdout:93 2022-04-22T17:42:36.497 INFO:tasks.workunit.client.0.smithi115.stdout:94 2022-04-22T17:42:36.498 INFO:tasks.workunit.client.0.smithi115.stdout:95 2022-04-22T17:42:36.500 INFO:tasks.workunit.client.0.smithi115.stdout:96 2022-04-22T17:42:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:97 2022-04-22T17:42:36.504 INFO:tasks.workunit.client.0.smithi115.stdout:98 2022-04-22T17:42:36.506 INFO:tasks.workunit.client.0.smithi115.stdout:99 2022-04-22T17:42:36.508 INFO:tasks.workunit.client.0.smithi115.stdout:100 2022-04-22T17:42:36.510 INFO:tasks.workunit.client.0.smithi115.stdout:101 2022-04-22T17:42:36.512 INFO:tasks.workunit.client.0.smithi115.stdout:102 2022-04-22T17:42:36.513 INFO:tasks.workunit.client.0.smithi115.stdout:103 2022-04-22T17:42:36.515 INFO:tasks.workunit.client.0.smithi115.stdout:104 2022-04-22T17:42:36.517 INFO:tasks.workunit.client.0.smithi115.stdout:105 2022-04-22T17:42:36.518 INFO:tasks.workunit.client.0.smithi115.stdout:106 2022-04-22T17:42:36.520 INFO:tasks.workunit.client.0.smithi115.stdout:107 2022-04-22T17:42:36.522 INFO:tasks.workunit.client.0.smithi115.stdout:108 2022-04-22T17:42:36.523 INFO:tasks.workunit.client.0.smithi115.stdout:109 2022-04-22T17:42:36.525 INFO:tasks.workunit.client.0.smithi115.stdout:110 2022-04-22T17:42:36.527 INFO:tasks.workunit.client.0.smithi115.stdout:111 2022-04-22T17:42:36.528 INFO:tasks.workunit.client.0.smithi115.stdout:112 2022-04-22T17:42:36.530 INFO:tasks.workunit.client.0.smithi115.stdout:113 2022-04-22T17:42:36.532 INFO:tasks.workunit.client.0.smithi115.stdout:114 2022-04-22T17:42:36.534 INFO:tasks.workunit.client.0.smithi115.stdout:115 2022-04-22T17:42:36.536 INFO:tasks.workunit.client.0.smithi115.stdout:116 2022-04-22T17:42:36.538 INFO:tasks.workunit.client.0.smithi115.stdout:117 2022-04-22T17:42:36.539 INFO:tasks.workunit.client.0.smithi115.stdout:118 2022-04-22T17:42:36.541 INFO:tasks.workunit.client.0.smithi115.stdout:119 2022-04-22T17:42:36.543 INFO:tasks.workunit.client.0.smithi115.stdout:120 2022-04-22T17:42:36.545 INFO:tasks.workunit.client.0.smithi115.stdout:121 2022-04-22T17:42:36.547 INFO:tasks.workunit.client.0.smithi115.stdout:122 2022-04-22T17:42:36.549 INFO:tasks.workunit.client.0.smithi115.stdout:123 2022-04-22T17:42:36.550 INFO:tasks.workunit.client.0.smithi115.stdout:124 2022-04-22T17:42:36.552 INFO:tasks.workunit.client.0.smithi115.stdout:125 2022-04-22T17:42:36.554 INFO:tasks.workunit.client.0.smithi115.stdout:126 2022-04-22T17:42:36.555 INFO:tasks.workunit.client.0.smithi115.stdout:127 2022-04-22T17:42:36.557 INFO:tasks.workunit.client.0.smithi115.stdout:128 2022-04-22T17:42:36.559 INFO:tasks.workunit.client.0.smithi115.stdout:129 2022-04-22T17:42:36.561 INFO:tasks.workunit.client.0.smithi115.stdout:130 2022-04-22T17:42:36.562 INFO:tasks.workunit.client.0.smithi115.stdout:131 2022-04-22T17:42:36.564 INFO:tasks.workunit.client.0.smithi115.stdout:132 2022-04-22T17:42:36.568 INFO:tasks.workunit.client.0.smithi115.stdout:133 2022-04-22T17:42:36.570 INFO:tasks.workunit.client.0.smithi115.stdout:134 2022-04-22T17:42:36.570 INFO:tasks.workunit.client.0.smithi115.stdout:135 2022-04-22T17:42:36.571 INFO:tasks.workunit.client.0.smithi115.stdout:136 2022-04-22T17:42:36.573 INFO:tasks.workunit.client.0.smithi115.stdout:137 2022-04-22T17:42:36.575 INFO:tasks.workunit.client.0.smithi115.stdout:138 2022-04-22T17:42:36.577 INFO:tasks.workunit.client.0.smithi115.stdout:139 2022-04-22T17:42:36.579 INFO:tasks.workunit.client.0.smithi115.stdout:140 2022-04-22T17:42:36.580 INFO:tasks.workunit.client.0.smithi115.stdout:141 2022-04-22T17:42:36.582 INFO:tasks.workunit.client.0.smithi115.stdout:142 2022-04-22T17:42:36.584 INFO:tasks.workunit.client.0.smithi115.stdout:143 2022-04-22T17:42:36.586 INFO:tasks.workunit.client.0.smithi115.stdout:144 2022-04-22T17:42:36.588 INFO:tasks.workunit.client.0.smithi115.stdout:145 2022-04-22T17:42:36.589 INFO:tasks.workunit.client.0.smithi115.stdout:146 2022-04-22T17:42:36.591 INFO:tasks.workunit.client.0.smithi115.stdout:147 2022-04-22T17:42:36.593 INFO:tasks.workunit.client.0.smithi115.stdout:148 2022-04-22T17:42:36.594 INFO:tasks.workunit.client.0.smithi115.stdout:149 2022-04-22T17:42:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:150 2022-04-22T17:42:36.598 INFO:tasks.workunit.client.0.smithi115.stdout:151 2022-04-22T17:42:36.600 INFO:tasks.workunit.client.0.smithi115.stdout:152 2022-04-22T17:42:36.602 INFO:tasks.workunit.client.0.smithi115.stdout:153 2022-04-22T17:42:36.604 INFO:tasks.workunit.client.0.smithi115.stdout:154 2022-04-22T17:42:36.606 INFO:tasks.workunit.client.0.smithi115.stdout:155 2022-04-22T17:42:36.607 INFO:tasks.workunit.client.0.smithi115.stdout:156 2022-04-22T17:42:36.609 INFO:tasks.workunit.client.0.smithi115.stdout:157 2022-04-22T17:42:36.611 INFO:tasks.workunit.client.0.smithi115.stdout:158 2022-04-22T17:42:36.613 INFO:tasks.workunit.client.0.smithi115.stdout:159 2022-04-22T17:42:36.615 INFO:tasks.workunit.client.0.smithi115.stdout:160 2022-04-22T17:42:36.617 INFO:tasks.workunit.client.0.smithi115.stdout:161 2022-04-22T17:42:36.619 INFO:tasks.workunit.client.0.smithi115.stdout:162 2022-04-22T17:42:36.621 INFO:tasks.workunit.client.0.smithi115.stdout:163 2022-04-22T17:42:36.622 INFO:tasks.workunit.client.0.smithi115.stdout:164 2022-04-22T17:42:36.624 INFO:tasks.workunit.client.0.smithi115.stdout:165 2022-04-22T17:42:36.626 INFO:tasks.workunit.client.0.smithi115.stdout:166 2022-04-22T17:42:36.628 INFO:tasks.workunit.client.0.smithi115.stdout:167 2022-04-22T17:42:36.629 INFO:tasks.workunit.client.0.smithi115.stdout:168 2022-04-22T17:42:36.631 INFO:tasks.workunit.client.0.smithi115.stdout:169 2022-04-22T17:42:36.633 INFO:tasks.workunit.client.0.smithi115.stdout:170 2022-04-22T17:42:36.635 INFO:tasks.workunit.client.0.smithi115.stdout:171 2022-04-22T17:42:36.637 INFO:tasks.workunit.client.0.smithi115.stdout:172 2022-04-22T17:42:36.640 INFO:tasks.workunit.client.0.smithi115.stdout:173 2022-04-22T17:42:36.641 INFO:tasks.workunit.client.0.smithi115.stdout:174 2022-04-22T17:42:36.643 INFO:tasks.workunit.client.0.smithi115.stdout:175 2022-04-22T17:42:36.645 INFO:tasks.workunit.client.0.smithi115.stdout:176 2022-04-22T17:42:36.647 INFO:tasks.workunit.client.0.smithi115.stdout:177 2022-04-22T17:42:36.649 INFO:tasks.workunit.client.0.smithi115.stdout:178 2022-04-22T17:42:36.652 INFO:tasks.workunit.client.0.smithi115.stdout:179 2022-04-22T17:42:36.653 INFO:tasks.workunit.client.0.smithi115.stdout:180 2022-04-22T17:42:36.654 INFO:tasks.workunit.client.0.smithi115.stdout:181 2022-04-22T17:42:36.656 INFO:tasks.workunit.client.0.smithi115.stdout:182 2022-04-22T17:42:36.658 INFO:tasks.workunit.client.0.smithi115.stdout:183 2022-04-22T17:42:36.660 INFO:tasks.workunit.client.0.smithi115.stdout:184 2022-04-22T17:42:36.662 INFO:tasks.workunit.client.0.smithi115.stdout:185 2022-04-22T17:42:36.664 INFO:tasks.workunit.client.0.smithi115.stdout:186 2022-04-22T17:42:36.666 INFO:tasks.workunit.client.0.smithi115.stdout:187 2022-04-22T17:42:36.669 INFO:tasks.workunit.client.0.smithi115.stdout:188 2022-04-22T17:42:36.671 INFO:tasks.workunit.client.0.smithi115.stdout:189 2022-04-22T17:42:36.672 INFO:tasks.workunit.client.0.smithi115.stdout:190 2022-04-22T17:42:36.674 INFO:tasks.workunit.client.0.smithi115.stdout:191 2022-04-22T17:42:36.676 INFO:tasks.workunit.client.0.smithi115.stdout:192 2022-04-22T17:42:36.679 INFO:tasks.workunit.client.0.smithi115.stdout:193 2022-04-22T17:42:36.681 INFO:tasks.workunit.client.0.smithi115.stdout:194 2022-04-22T17:42:36.682 INFO:tasks.workunit.client.0.smithi115.stdout:195 2022-04-22T17:42:36.684 INFO:tasks.workunit.client.0.smithi115.stdout:196 2022-04-22T17:42:36.686 INFO:tasks.workunit.client.0.smithi115.stdout:197 2022-04-22T17:42:36.688 INFO:tasks.workunit.client.0.smithi115.stdout:198 2022-04-22T17:42:36.690 INFO:tasks.workunit.client.0.smithi115.stdout:199 2022-04-22T17:42:36.692 INFO:tasks.workunit.client.0.smithi115.stdout:200 2022-04-22T17:42:36.694 INFO:tasks.workunit.client.0.smithi115.stdout:201 2022-04-22T17:42:36.696 INFO:tasks.workunit.client.0.smithi115.stdout:202 2022-04-22T17:42:36.698 INFO:tasks.workunit.client.0.smithi115.stdout:203 2022-04-22T17:42:36.700 INFO:tasks.workunit.client.0.smithi115.stdout:204 2022-04-22T17:42:36.702 INFO:tasks.workunit.client.0.smithi115.stdout:205 2022-04-22T17:42:36.703 INFO:tasks.workunit.client.0.smithi115.stdout:206 2022-04-22T17:42:36.705 INFO:tasks.workunit.client.0.smithi115.stdout:207 2022-04-22T17:42:36.708 INFO:tasks.workunit.client.0.smithi115.stdout:208 2022-04-22T17:42:36.710 INFO:tasks.workunit.client.0.smithi115.stdout:209 2022-04-22T17:42:36.712 INFO:tasks.workunit.client.0.smithi115.stdout:210 2022-04-22T17:42:36.714 INFO:tasks.workunit.client.0.smithi115.stdout:211 2022-04-22T17:42:36.716 INFO:tasks.workunit.client.0.smithi115.stdout:212 2022-04-22T17:42:36.718 INFO:tasks.workunit.client.0.smithi115.stdout:213 2022-04-22T17:42:36.719 INFO:tasks.workunit.client.0.smithi115.stdout:214 2022-04-22T17:42:36.722 INFO:tasks.workunit.client.0.smithi115.stdout:215 2022-04-22T17:42:36.723 INFO:tasks.workunit.client.0.smithi115.stdout:216 2022-04-22T17:42:36.726 INFO:tasks.workunit.client.0.smithi115.stdout:217 2022-04-22T17:42:36.727 INFO:tasks.workunit.client.0.smithi115.stdout:218 2022-04-22T17:42:36.729 INFO:tasks.workunit.client.0.smithi115.stdout:219 2022-04-22T17:42:36.731 INFO:tasks.workunit.client.0.smithi115.stdout:220 2022-04-22T17:42:36.733 INFO:tasks.workunit.client.0.smithi115.stdout:221 2022-04-22T17:42:36.735 INFO:tasks.workunit.client.0.smithi115.stdout:222 2022-04-22T17:42:36.737 INFO:tasks.workunit.client.0.smithi115.stdout:223 2022-04-22T17:42:36.739 INFO:tasks.workunit.client.0.smithi115.stdout:224 2022-04-22T17:42:36.741 INFO:tasks.workunit.client.0.smithi115.stdout:225 2022-04-22T17:42:36.743 INFO:tasks.workunit.client.0.smithi115.stdout:226 2022-04-22T17:42:36.745 INFO:tasks.workunit.client.0.smithi115.stdout:227 2022-04-22T17:42:36.747 INFO:tasks.workunit.client.0.smithi115.stdout:228 2022-04-22T17:42:36.748 INFO:tasks.workunit.client.0.smithi115.stdout:229 2022-04-22T17:42:36.751 INFO:tasks.workunit.client.0.smithi115.stdout:230 2022-04-22T17:42:36.752 INFO:tasks.workunit.client.0.smithi115.stdout:231 2022-04-22T17:42:36.754 INFO:tasks.workunit.client.0.smithi115.stdout:232 2022-04-22T17:42:36.756 INFO:tasks.workunit.client.0.smithi115.stdout:233 2022-04-22T17:42:36.758 INFO:tasks.workunit.client.0.smithi115.stdout:234 2022-04-22T17:42:36.759 INFO:tasks.workunit.client.0.smithi115.stdout:235 2022-04-22T17:42:36.761 INFO:tasks.workunit.client.0.smithi115.stdout:236 2022-04-22T17:42:36.763 INFO:tasks.workunit.client.0.smithi115.stdout:237 2022-04-22T17:42:36.765 INFO:tasks.workunit.client.0.smithi115.stdout:238 2022-04-22T17:42:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:36 smithi121 bash[36902]: cluster 2022-04-22T17:42:35.300610+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2022-04-22T17:42:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:36 smithi121 bash[36902]: audit 2022-04-22T17:42:35.305201+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:36 smithi121 bash[36902]: cluster 2022-04-22T17:42:36.107422+0000 mgr.y (mgr.34107) 619 : cluster [DBG] pgmap v1161: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:36.767 INFO:tasks.workunit.client.0.smithi115.stdout:239 2022-04-22T17:42:36.768 INFO:tasks.workunit.client.0.smithi115.stdout:240 2022-04-22T17:42:36.770 INFO:tasks.workunit.client.0.smithi115.stdout:241 2022-04-22T17:42:36.772 INFO:tasks.workunit.client.0.smithi115.stdout:242 2022-04-22T17:42:36.774 INFO:tasks.workunit.client.0.smithi115.stdout:243 2022-04-22T17:42:36.776 INFO:tasks.workunit.client.0.smithi115.stdout:244 2022-04-22T17:42:36.779 INFO:tasks.workunit.client.0.smithi115.stdout:245 2022-04-22T17:42:36.780 INFO:tasks.workunit.client.0.smithi115.stdout:246 2022-04-22T17:42:36.782 INFO:tasks.workunit.client.0.smithi115.stdout:247 2022-04-22T17:42:36.784 INFO:tasks.workunit.client.0.smithi115.stdout:248 2022-04-22T17:42:36.786 INFO:tasks.workunit.client.0.smithi115.stdout:249 2022-04-22T17:42:36.795 INFO:tasks.workunit.client.0.smithi115.stdout:250 2022-04-22T17:42:36.795 INFO:tasks.workunit.client.0.smithi115.stdout:251 2022-04-22T17:42:36.795 INFO:tasks.workunit.client.0.smithi115.stdout:252 2022-04-22T17:42:36.795 INFO:tasks.workunit.client.0.smithi115.stdout:253 2022-04-22T17:42:36.796 INFO:tasks.workunit.client.0.smithi115.stdout:254 2022-04-22T17:42:36.798 INFO:tasks.workunit.client.0.smithi115.stdout:255 2022-04-22T17:42:36.800 INFO:tasks.workunit.client.0.smithi115.stdout:256 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[45495]: cluster 2022-04-22T17:42:35.300610+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[45495]: audit 2022-04-22T17:42:35.305201+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[45495]: cluster 2022-04-22T17:42:36.107422+0000 mgr.y (mgr.34107) 619 : cluster [DBG] pgmap v1161: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[46934]: cluster 2022-04-22T17:42:35.300610+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[46934]: audit 2022-04-22T17:42:35.305201+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:36.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:36 smithi115 bash[46934]: cluster 2022-04-22T17:42:36.107422+0000 mgr.y (mgr.34107) 619 : cluster [DBG] pgmap v1161: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:36.802 INFO:tasks.workunit.client.0.smithi115.stdout:257 2022-04-22T17:42:36.804 INFO:tasks.workunit.client.0.smithi115.stdout:258 2022-04-22T17:42:36.806 INFO:tasks.workunit.client.0.smithi115.stdout:259 2022-04-22T17:42:36.808 INFO:tasks.workunit.client.0.smithi115.stdout:260 2022-04-22T17:42:36.810 INFO:tasks.workunit.client.0.smithi115.stdout:261 2022-04-22T17:42:36.812 INFO:tasks.workunit.client.0.smithi115.stdout:262 2022-04-22T17:42:36.814 INFO:tasks.workunit.client.0.smithi115.stdout:263 2022-04-22T17:42:36.815 INFO:tasks.workunit.client.0.smithi115.stdout:264 2022-04-22T17:42:36.817 INFO:tasks.workunit.client.0.smithi115.stdout:265 2022-04-22T17:42:36.819 INFO:tasks.workunit.client.0.smithi115.stdout:266 2022-04-22T17:42:36.822 INFO:tasks.workunit.client.0.smithi115.stdout:267 2022-04-22T17:42:36.823 INFO:tasks.workunit.client.0.smithi115.stdout:268 2022-04-22T17:42:36.825 INFO:tasks.workunit.client.0.smithi115.stdout:269 2022-04-22T17:42:36.827 INFO:tasks.workunit.client.0.smithi115.stdout:270 2022-04-22T17:42:36.829 INFO:tasks.workunit.client.0.smithi115.stdout:271 2022-04-22T17:42:36.831 INFO:tasks.workunit.client.0.smithi115.stdout:272 2022-04-22T17:42:36.834 INFO:tasks.workunit.client.0.smithi115.stdout:273 2022-04-22T17:42:36.835 INFO:tasks.workunit.client.0.smithi115.stdout:274 2022-04-22T17:42:36.838 INFO:tasks.workunit.client.0.smithi115.stdout:275 2022-04-22T17:42:36.840 INFO:tasks.workunit.client.0.smithi115.stdout:276 2022-04-22T17:42:36.841 INFO:tasks.workunit.client.0.smithi115.stdout:277 2022-04-22T17:42:36.843 INFO:tasks.workunit.client.0.smithi115.stdout:278 2022-04-22T17:42:36.846 INFO:tasks.workunit.client.0.smithi115.stdout:279 2022-04-22T17:42:36.848 INFO:tasks.workunit.client.0.smithi115.stdout:280 2022-04-22T17:42:36.849 INFO:tasks.workunit.client.0.smithi115.stdout:281 2022-04-22T17:42:36.851 INFO:tasks.workunit.client.0.smithi115.stdout:282 2022-04-22T17:42:36.853 INFO:tasks.workunit.client.0.smithi115.stdout:283 2022-04-22T17:42:36.855 INFO:tasks.workunit.client.0.smithi115.stdout:284 2022-04-22T17:42:36.857 INFO:tasks.workunit.client.0.smithi115.stdout:285 2022-04-22T17:42:36.858 INFO:tasks.workunit.client.0.smithi115.stdout:286 2022-04-22T17:42:36.860 INFO:tasks.workunit.client.0.smithi115.stdout:287 2022-04-22T17:42:36.862 INFO:tasks.workunit.client.0.smithi115.stdout:288 2022-04-22T17:42:36.865 INFO:tasks.workunit.client.0.smithi115.stdout:289 2022-04-22T17:42:36.867 INFO:tasks.workunit.client.0.smithi115.stdout:290 2022-04-22T17:42:36.869 INFO:tasks.workunit.client.0.smithi115.stdout:291 2022-04-22T17:42:36.871 INFO:tasks.workunit.client.0.smithi115.stdout:292 2022-04-22T17:42:36.873 INFO:tasks.workunit.client.0.smithi115.stdout:293 2022-04-22T17:42:36.875 INFO:tasks.workunit.client.0.smithi115.stdout:294 2022-04-22T17:42:36.876 INFO:tasks.workunit.client.0.smithi115.stdout:295 2022-04-22T17:42:36.878 INFO:tasks.workunit.client.0.smithi115.stdout:296 2022-04-22T17:42:36.880 INFO:tasks.workunit.client.0.smithi115.stdout:297 2022-04-22T17:42:36.882 INFO:tasks.workunit.client.0.smithi115.stdout:298 2022-04-22T17:42:36.884 INFO:tasks.workunit.client.0.smithi115.stdout:299 2022-04-22T17:42:36.886 INFO:tasks.workunit.client.0.smithi115.stdout:300 2022-04-22T17:42:36.888 INFO:tasks.workunit.client.0.smithi115.stdout:301 2022-04-22T17:42:36.890 INFO:tasks.workunit.client.0.smithi115.stdout:302 2022-04-22T17:42:36.892 INFO:tasks.workunit.client.0.smithi115.stdout:303 2022-04-22T17:42:36.894 INFO:tasks.workunit.client.0.smithi115.stdout:304 2022-04-22T17:42:36.896 INFO:tasks.workunit.client.0.smithi115.stdout:305 2022-04-22T17:42:36.898 INFO:tasks.workunit.client.0.smithi115.stdout:306 2022-04-22T17:42:36.900 INFO:tasks.workunit.client.0.smithi115.stdout:307 2022-04-22T17:42:36.902 INFO:tasks.workunit.client.0.smithi115.stdout:308 2022-04-22T17:42:36.904 INFO:tasks.workunit.client.0.smithi115.stdout:309 2022-04-22T17:42:36.906 INFO:tasks.workunit.client.0.smithi115.stdout:310 2022-04-22T17:42:36.908 INFO:tasks.workunit.client.0.smithi115.stdout:311 2022-04-22T17:42:36.910 INFO:tasks.workunit.client.0.smithi115.stdout:312 2022-04-22T17:42:36.911 INFO:tasks.workunit.client.0.smithi115.stdout:313 2022-04-22T17:42:36.913 INFO:tasks.workunit.client.0.smithi115.stdout:314 2022-04-22T17:42:36.915 INFO:tasks.workunit.client.0.smithi115.stdout:315 2022-04-22T17:42:36.917 INFO:tasks.workunit.client.0.smithi115.stdout:316 2022-04-22T17:42:36.918 INFO:tasks.workunit.client.0.smithi115.stdout:317 2022-04-22T17:42:36.920 INFO:tasks.workunit.client.0.smithi115.stdout:318 2022-04-22T17:42:36.922 INFO:tasks.workunit.client.0.smithi115.stdout:319 2022-04-22T17:42:36.924 INFO:tasks.workunit.client.0.smithi115.stdout:320 2022-04-22T17:42:36.926 INFO:tasks.workunit.client.0.smithi115.stdout:321 2022-04-22T17:42:36.928 INFO:tasks.workunit.client.0.smithi115.stdout:322 2022-04-22T17:42:36.930 INFO:tasks.workunit.client.0.smithi115.stdout:323 2022-04-22T17:42:36.932 INFO:tasks.workunit.client.0.smithi115.stdout:324 2022-04-22T17:42:36.934 INFO:tasks.workunit.client.0.smithi115.stdout:325 2022-04-22T17:42:36.937 INFO:tasks.workunit.client.0.smithi115.stdout:326 2022-04-22T17:42:36.939 INFO:tasks.workunit.client.0.smithi115.stdout:327 2022-04-22T17:42:36.941 INFO:tasks.workunit.client.0.smithi115.stdout:328 2022-04-22T17:42:36.943 INFO:tasks.workunit.client.0.smithi115.stdout:329 2022-04-22T17:42:36.945 INFO:tasks.workunit.client.0.smithi115.stdout:330 2022-04-22T17:42:36.947 INFO:tasks.workunit.client.0.smithi115.stdout:331 2022-04-22T17:42:36.949 INFO:tasks.workunit.client.0.smithi115.stdout:332 2022-04-22T17:42:36.951 INFO:tasks.workunit.client.0.smithi115.stdout:333 2022-04-22T17:42:36.953 INFO:tasks.workunit.client.0.smithi115.stdout:334 2022-04-22T17:42:36.955 INFO:tasks.workunit.client.0.smithi115.stdout:335 2022-04-22T17:42:36.957 INFO:tasks.workunit.client.0.smithi115.stdout:336 2022-04-22T17:42:36.960 INFO:tasks.workunit.client.0.smithi115.stdout:337 2022-04-22T17:42:36.961 INFO:tasks.workunit.client.0.smithi115.stdout:338 2022-04-22T17:42:36.962 INFO:tasks.workunit.client.0.smithi115.stdout:339 2022-04-22T17:42:36.964 INFO:tasks.workunit.client.0.smithi115.stdout:340 2022-04-22T17:42:36.967 INFO:tasks.workunit.client.0.smithi115.stdout:341 2022-04-22T17:42:36.968 INFO:tasks.workunit.client.0.smithi115.stdout:342 2022-04-22T17:42:36.970 INFO:tasks.workunit.client.0.smithi115.stdout:343 2022-04-22T17:42:36.972 INFO:tasks.workunit.client.0.smithi115.stdout:344 2022-04-22T17:42:36.974 INFO:tasks.workunit.client.0.smithi115.stdout:345 2022-04-22T17:42:36.976 INFO:tasks.workunit.client.0.smithi115.stdout:346 2022-04-22T17:42:36.978 INFO:tasks.workunit.client.0.smithi115.stdout:347 2022-04-22T17:42:36.980 INFO:tasks.workunit.client.0.smithi115.stdout:348 2022-04-22T17:42:36.982 INFO:tasks.workunit.client.0.smithi115.stdout:349 2022-04-22T17:42:36.984 INFO:tasks.workunit.client.0.smithi115.stdout:350 2022-04-22T17:42:36.986 INFO:tasks.workunit.client.0.smithi115.stdout:351 2022-04-22T17:42:36.988 INFO:tasks.workunit.client.0.smithi115.stdout:352 2022-04-22T17:42:36.990 INFO:tasks.workunit.client.0.smithi115.stdout:353 2022-04-22T17:42:36.992 INFO:tasks.workunit.client.0.smithi115.stdout:354 2022-04-22T17:42:36.994 INFO:tasks.workunit.client.0.smithi115.stdout:355 2022-04-22T17:42:36.996 INFO:tasks.workunit.client.0.smithi115.stdout:356 2022-04-22T17:42:36.998 INFO:tasks.workunit.client.0.smithi115.stdout:357 2022-04-22T17:42:37.000 INFO:tasks.workunit.client.0.smithi115.stdout:358 2022-04-22T17:42:37.002 INFO:tasks.workunit.client.0.smithi115.stdout:359 2022-04-22T17:42:37.004 INFO:tasks.workunit.client.0.smithi115.stdout:360 2022-04-22T17:42:37.005 INFO:tasks.workunit.client.0.smithi115.stdout:361 2022-04-22T17:42:37.007 INFO:tasks.workunit.client.0.smithi115.stdout:362 2022-04-22T17:42:37.009 INFO:tasks.workunit.client.0.smithi115.stdout:363 2022-04-22T17:42:37.011 INFO:tasks.workunit.client.0.smithi115.stdout:364 2022-04-22T17:42:37.013 INFO:tasks.workunit.client.0.smithi115.stdout:365 2022-04-22T17:42:37.015 INFO:tasks.workunit.client.0.smithi115.stdout:366 2022-04-22T17:42:37.018 INFO:tasks.workunit.client.0.smithi115.stdout:367 2022-04-22T17:42:37.019 INFO:tasks.workunit.client.0.smithi115.stdout:368 2022-04-22T17:42:37.021 INFO:tasks.workunit.client.0.smithi115.stdout:369 2022-04-22T17:42:37.023 INFO:tasks.workunit.client.0.smithi115.stdout:370 2022-04-22T17:42:37.026 INFO:tasks.workunit.client.0.smithi115.stdout:371 2022-04-22T17:42:37.028 INFO:tasks.workunit.client.0.smithi115.stdout:372 2022-04-22T17:42:37.030 INFO:tasks.workunit.client.0.smithi115.stdout:373 2022-04-22T17:42:37.032 INFO:tasks.workunit.client.0.smithi115.stdout:374 2022-04-22T17:42:37.034 INFO:tasks.workunit.client.0.smithi115.stdout:375 2022-04-22T17:42:37.036 INFO:tasks.workunit.client.0.smithi115.stdout:376 2022-04-22T17:42:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:377 2022-04-22T17:42:37.040 INFO:tasks.workunit.client.0.smithi115.stdout:378 2022-04-22T17:42:37.042 INFO:tasks.workunit.client.0.smithi115.stdout:379 2022-04-22T17:42:37.044 INFO:tasks.workunit.client.0.smithi115.stdout:380 2022-04-22T17:42:37.046 INFO:tasks.workunit.client.0.smithi115.stdout:381 2022-04-22T17:42:37.048 INFO:tasks.workunit.client.0.smithi115.stdout:382 2022-04-22T17:42:37.050 INFO:tasks.workunit.client.0.smithi115.stdout:383 2022-04-22T17:42:37.052 INFO:tasks.workunit.client.0.smithi115.stdout:384 2022-04-22T17:42:37.054 INFO:tasks.workunit.client.0.smithi115.stdout:385 2022-04-22T17:42:37.056 INFO:tasks.workunit.client.0.smithi115.stdout:386 2022-04-22T17:42:37.058 INFO:tasks.workunit.client.0.smithi115.stdout:387 2022-04-22T17:42:37.060 INFO:tasks.workunit.client.0.smithi115.stdout:388 2022-04-22T17:42:37.062 INFO:tasks.workunit.client.0.smithi115.stdout:389 2022-04-22T17:42:37.063 INFO:tasks.workunit.client.0.smithi115.stdout:390 2022-04-22T17:42:37.065 INFO:tasks.workunit.client.0.smithi115.stdout:391 2022-04-22T17:42:37.067 INFO:tasks.workunit.client.0.smithi115.stdout:392 2022-04-22T17:42:37.069 INFO:tasks.workunit.client.0.smithi115.stdout:393 2022-04-22T17:42:37.071 INFO:tasks.workunit.client.0.smithi115.stdout:394 2022-04-22T17:42:37.073 INFO:tasks.workunit.client.0.smithi115.stdout:395 2022-04-22T17:42:37.075 INFO:tasks.workunit.client.0.smithi115.stdout:396 2022-04-22T17:42:37.077 INFO:tasks.workunit.client.0.smithi115.stdout:397 2022-04-22T17:42:37.079 INFO:tasks.workunit.client.0.smithi115.stdout:398 2022-04-22T17:42:37.081 INFO:tasks.workunit.client.0.smithi115.stdout:399 2022-04-22T17:42:37.082 INFO:tasks.workunit.client.0.smithi115.stdout:400 2022-04-22T17:42:37.085 INFO:tasks.workunit.client.0.smithi115.stdout:401 2022-04-22T17:42:37.087 INFO:tasks.workunit.client.0.smithi115.stdout:402 2022-04-22T17:42:37.089 INFO:tasks.workunit.client.0.smithi115.stdout:403 2022-04-22T17:42:37.091 INFO:tasks.workunit.client.0.smithi115.stdout:404 2022-04-22T17:42:37.093 INFO:tasks.workunit.client.0.smithi115.stdout:405 2022-04-22T17:42:37.097 INFO:tasks.workunit.client.0.smithi115.stdout:406 2022-04-22T17:42:37.097 INFO:tasks.workunit.client.0.smithi115.stdout:407 2022-04-22T17:42:37.099 INFO:tasks.workunit.client.0.smithi115.stdout:408 2022-04-22T17:42:37.101 INFO:tasks.workunit.client.0.smithi115.stdout:409 2022-04-22T17:42:37.103 INFO:tasks.workunit.client.0.smithi115.stdout:410 2022-04-22T17:42:37.105 INFO:tasks.workunit.client.0.smithi115.stdout:411 2022-04-22T17:42:37.107 INFO:tasks.workunit.client.0.smithi115.stdout:412 2022-04-22T17:42:37.109 INFO:tasks.workunit.client.0.smithi115.stdout:413 2022-04-22T17:42:37.110 INFO:tasks.workunit.client.0.smithi115.stdout:414 2022-04-22T17:42:37.112 INFO:tasks.workunit.client.0.smithi115.stdout:415 2022-04-22T17:42:37.114 INFO:tasks.workunit.client.0.smithi115.stdout:416 2022-04-22T17:42:37.116 INFO:tasks.workunit.client.0.smithi115.stdout:417 2022-04-22T17:42:37.118 INFO:tasks.workunit.client.0.smithi115.stdout:418 2022-04-22T17:42:37.120 INFO:tasks.workunit.client.0.smithi115.stdout:419 2022-04-22T17:42:37.123 INFO:tasks.workunit.client.0.smithi115.stdout:420 2022-04-22T17:42:37.125 INFO:tasks.workunit.client.0.smithi115.stdout:421 2022-04-22T17:42:37.127 INFO:tasks.workunit.client.0.smithi115.stdout:422 2022-04-22T17:42:37.129 INFO:tasks.workunit.client.0.smithi115.stdout:423 2022-04-22T17:42:37.131 INFO:tasks.workunit.client.0.smithi115.stdout:424 2022-04-22T17:42:37.133 INFO:tasks.workunit.client.0.smithi115.stdout:425 2022-04-22T17:42:37.135 INFO:tasks.workunit.client.0.smithi115.stdout:426 2022-04-22T17:42:37.137 INFO:tasks.workunit.client.0.smithi115.stdout:427 2022-04-22T17:42:37.138 INFO:tasks.workunit.client.0.smithi115.stdout:428 2022-04-22T17:42:37.140 INFO:tasks.workunit.client.0.smithi115.stdout:429 2022-04-22T17:42:37.143 INFO:tasks.workunit.client.0.smithi115.stdout:430 2022-04-22T17:42:37.145 INFO:tasks.workunit.client.0.smithi115.stdout:431 2022-04-22T17:42:37.147 INFO:tasks.workunit.client.0.smithi115.stdout:432 2022-04-22T17:42:37.149 INFO:tasks.workunit.client.0.smithi115.stdout:433 2022-04-22T17:42:37.151 INFO:tasks.workunit.client.0.smithi115.stdout:434 2022-04-22T17:42:37.153 INFO:tasks.workunit.client.0.smithi115.stdout:435 2022-04-22T17:42:37.155 INFO:tasks.workunit.client.0.smithi115.stdout:436 2022-04-22T17:42:37.157 INFO:tasks.workunit.client.0.smithi115.stdout:437 2022-04-22T17:42:37.159 INFO:tasks.workunit.client.0.smithi115.stdout:438 2022-04-22T17:42:37.161 INFO:tasks.workunit.client.0.smithi115.stdout:439 2022-04-22T17:42:37.163 INFO:tasks.workunit.client.0.smithi115.stdout:440 2022-04-22T17:42:37.164 INFO:tasks.workunit.client.0.smithi115.stdout:441 2022-04-22T17:42:37.166 INFO:tasks.workunit.client.0.smithi115.stdout:442 2022-04-22T17:42:37.168 INFO:tasks.workunit.client.0.smithi115.stdout:443 2022-04-22T17:42:37.170 INFO:tasks.workunit.client.0.smithi115.stdout:444 2022-04-22T17:42:37.172 INFO:tasks.workunit.client.0.smithi115.stdout:445 2022-04-22T17:42:37.174 INFO:tasks.workunit.client.0.smithi115.stdout:446 2022-04-22T17:42:37.176 INFO:tasks.workunit.client.0.smithi115.stdout:447 2022-04-22T17:42:37.178 INFO:tasks.workunit.client.0.smithi115.stdout:448 2022-04-22T17:42:37.180 INFO:tasks.workunit.client.0.smithi115.stdout:449 2022-04-22T17:42:37.182 INFO:tasks.workunit.client.0.smithi115.stdout:450 2022-04-22T17:42:37.184 INFO:tasks.workunit.client.0.smithi115.stdout:451 2022-04-22T17:42:37.187 INFO:tasks.workunit.client.0.smithi115.stdout:452 2022-04-22T17:42:37.189 INFO:tasks.workunit.client.0.smithi115.stdout:453 2022-04-22T17:42:37.191 INFO:tasks.workunit.client.0.smithi115.stdout:454 2022-04-22T17:42:37.192 INFO:tasks.workunit.client.0.smithi115.stdout:455 2022-04-22T17:42:37.195 INFO:tasks.workunit.client.0.smithi115.stdout:456 2022-04-22T17:42:37.197 INFO:tasks.workunit.client.0.smithi115.stdout:457 2022-04-22T17:42:37.199 INFO:tasks.workunit.client.0.smithi115.stdout:458 2022-04-22T17:42:37.200 INFO:tasks.workunit.client.0.smithi115.stdout:459 2022-04-22T17:42:37.202 INFO:tasks.workunit.client.0.smithi115.stdout:460 2022-04-22T17:42:37.204 INFO:tasks.workunit.client.0.smithi115.stdout:461 2022-04-22T17:42:37.206 INFO:tasks.workunit.client.0.smithi115.stdout:462 2022-04-22T17:42:37.208 INFO:tasks.workunit.client.0.smithi115.stdout:463 2022-04-22T17:42:37.210 INFO:tasks.workunit.client.0.smithi115.stdout:464 2022-04-22T17:42:37.212 INFO:tasks.workunit.client.0.smithi115.stdout:465 2022-04-22T17:42:37.214 INFO:tasks.workunit.client.0.smithi115.stdout:466 2022-04-22T17:42:37.216 INFO:tasks.workunit.client.0.smithi115.stdout:467 2022-04-22T17:42:37.218 INFO:tasks.workunit.client.0.smithi115.stdout:468 2022-04-22T17:42:37.220 INFO:tasks.workunit.client.0.smithi115.stdout:469 2022-04-22T17:42:37.222 INFO:tasks.workunit.client.0.smithi115.stdout:470 2022-04-22T17:42:37.224 INFO:tasks.workunit.client.0.smithi115.stdout:471 2022-04-22T17:42:37.226 INFO:tasks.workunit.client.0.smithi115.stdout:472 2022-04-22T17:42:37.229 INFO:tasks.workunit.client.0.smithi115.stdout:473 2022-04-22T17:42:37.231 INFO:tasks.workunit.client.0.smithi115.stdout:474 2022-04-22T17:42:37.233 INFO:tasks.workunit.client.0.smithi115.stdout:475 2022-04-22T17:42:37.235 INFO:tasks.workunit.client.0.smithi115.stdout:476 2022-04-22T17:42:37.237 INFO:tasks.workunit.client.0.smithi115.stdout:477 2022-04-22T17:42:37.241 INFO:tasks.workunit.client.0.smithi115.stdout:478 2022-04-22T17:42:37.243 INFO:tasks.workunit.client.0.smithi115.stdout:479 2022-04-22T17:42:37.244 INFO:tasks.workunit.client.0.smithi115.stdout:480 2022-04-22T17:42:37.246 INFO:tasks.workunit.client.0.smithi115.stdout:481 2022-04-22T17:42:37.248 INFO:tasks.workunit.client.0.smithi115.stdout:482 2022-04-22T17:42:37.250 INFO:tasks.workunit.client.0.smithi115.stdout:483 2022-04-22T17:42:37.253 INFO:tasks.workunit.client.0.smithi115.stdout:484 2022-04-22T17:42:37.255 INFO:tasks.workunit.client.0.smithi115.stdout:485 2022-04-22T17:42:37.256 INFO:tasks.workunit.client.0.smithi115.stdout:486 2022-04-22T17:42:37.258 INFO:tasks.workunit.client.0.smithi115.stdout:487 2022-04-22T17:42:37.260 INFO:tasks.workunit.client.0.smithi115.stdout:488 2022-04-22T17:42:37.262 INFO:tasks.workunit.client.0.smithi115.stdout:489 2022-04-22T17:42:37.264 INFO:tasks.workunit.client.0.smithi115.stdout:490 2022-04-22T17:42:37.266 INFO:tasks.workunit.client.0.smithi115.stdout:491 2022-04-22T17:42:37.268 INFO:tasks.workunit.client.0.smithi115.stdout:492 2022-04-22T17:42:37.270 INFO:tasks.workunit.client.0.smithi115.stdout:493 2022-04-22T17:42:37.272 INFO:tasks.workunit.client.0.smithi115.stdout:494 2022-04-22T17:42:37.274 INFO:tasks.workunit.client.0.smithi115.stdout:495 2022-04-22T17:42:37.276 INFO:tasks.workunit.client.0.smithi115.stdout:496 2022-04-22T17:42:37.278 INFO:tasks.workunit.client.0.smithi115.stdout:497 2022-04-22T17:42:37.280 INFO:tasks.workunit.client.0.smithi115.stdout:498 2022-04-22T17:42:37.282 INFO:tasks.workunit.client.0.smithi115.stdout:499 2022-04-22T17:42:37.284 INFO:tasks.workunit.client.0.smithi115.stdout:500 2022-04-22T17:42:37.286 INFO:tasks.workunit.client.0.smithi115.stdout:501 2022-04-22T17:42:37.288 INFO:tasks.workunit.client.0.smithi115.stdout:502 2022-04-22T17:42:37.290 INFO:tasks.workunit.client.0.smithi115.stdout:503 2022-04-22T17:42:37.292 INFO:tasks.workunit.client.0.smithi115.stdout:504 2022-04-22T17:42:37.294 INFO:tasks.workunit.client.0.smithi115.stdout:505 2022-04-22T17:42:37.296 INFO:tasks.workunit.client.0.smithi115.stdout:506 2022-04-22T17:42:37.298 INFO:tasks.workunit.client.0.smithi115.stdout:507 2022-04-22T17:42:37.300 INFO:tasks.workunit.client.0.smithi115.stdout:508 2022-04-22T17:42:37.302 INFO:tasks.workunit.client.0.smithi115.stdout:509 2022-04-22T17:42:37.304 INFO:tasks.workunit.client.0.smithi115.stdout:510 2022-04-22T17:42:37.306 INFO:tasks.workunit.client.0.smithi115.stdout:511 2022-04-22T17:42:37.308 INFO:tasks.workunit.client.0.smithi115.stdout:512 2022-04-22T17:42:37.310 INFO:tasks.workunit.client.0.smithi115.stdout:513 2022-04-22T17:42:37.312 INFO:tasks.workunit.client.0.smithi115.stdout:514 2022-04-22T17:42:37.314 INFO:tasks.workunit.client.0.smithi115.stdout:515 2022-04-22T17:42:37.316 INFO:tasks.workunit.client.0.smithi115.stdout:516 2022-04-22T17:42:37.318 INFO:tasks.workunit.client.0.smithi115.stdout:517 2022-04-22T17:42:37.320 INFO:tasks.workunit.client.0.smithi115.stdout:518 2022-04-22T17:42:37.322 INFO:tasks.workunit.client.0.smithi115.stdout:519 2022-04-22T17:42:37.324 INFO:tasks.workunit.client.0.smithi115.stdout:520 2022-04-22T17:42:37.330 INFO:tasks.workunit.client.0.smithi115.stdout:521 2022-04-22T17:42:37.333 INFO:tasks.workunit.client.0.smithi115.stdout:522 2022-04-22T17:42:37.335 INFO:tasks.workunit.client.0.smithi115.stdout:523 2022-04-22T17:42:37.337 INFO:tasks.workunit.client.0.smithi115.stdout:524 2022-04-22T17:42:37.339 INFO:tasks.workunit.client.0.smithi115.stdout:525 2022-04-22T17:42:37.341 INFO:tasks.workunit.client.0.smithi115.stdout:526 2022-04-22T17:42:37.342 INFO:tasks.workunit.client.0.smithi115.stdout:527 2022-04-22T17:42:37.344 INFO:tasks.workunit.client.0.smithi115.stdout:528 2022-04-22T17:42:37.346 INFO:tasks.workunit.client.0.smithi115.stdout:529 2022-04-22T17:42:37.348 INFO:tasks.workunit.client.0.smithi115.stdout:530 2022-04-22T17:42:37.349 INFO:tasks.workunit.client.0.smithi115.stdout:531 2022-04-22T17:42:37.351 INFO:tasks.workunit.client.0.smithi115.stdout:532 2022-04-22T17:42:37.353 INFO:tasks.workunit.client.0.smithi115.stdout:533 2022-04-22T17:42:37.355 INFO:tasks.workunit.client.0.smithi115.stdout:534 2022-04-22T17:42:37.358 INFO:tasks.workunit.client.0.smithi115.stdout:535 2022-04-22T17:42:37.360 INFO:tasks.workunit.client.0.smithi115.stdout:536 2022-04-22T17:42:37.362 INFO:tasks.workunit.client.0.smithi115.stdout:537 2022-04-22T17:42:37.364 INFO:tasks.workunit.client.0.smithi115.stdout:538 2022-04-22T17:42:37.368 INFO:tasks.workunit.client.0.smithi115.stdout:539 2022-04-22T17:42:37.369 INFO:tasks.workunit.client.0.smithi115.stdout:540 2022-04-22T17:42:37.372 INFO:tasks.workunit.client.0.smithi115.stdout:541 2022-04-22T17:42:37.374 INFO:tasks.workunit.client.0.smithi115.stdout:542 2022-04-22T17:42:37.376 INFO:tasks.workunit.client.0.smithi115.stdout:543 2022-04-22T17:42:37.378 INFO:tasks.workunit.client.0.smithi115.stdout:544 2022-04-22T17:42:37.379 INFO:tasks.workunit.client.0.smithi115.stdout:545 2022-04-22T17:42:37.381 INFO:tasks.workunit.client.0.smithi115.stdout:546 2022-04-22T17:42:37.383 INFO:tasks.workunit.client.0.smithi115.stdout:547 2022-04-22T17:42:37.385 INFO:tasks.workunit.client.0.smithi115.stdout:548 2022-04-22T17:42:37.388 INFO:tasks.workunit.client.0.smithi115.stdout:549 2022-04-22T17:42:37.390 INFO:tasks.workunit.client.0.smithi115.stdout:550 2022-04-22T17:42:37.392 INFO:tasks.workunit.client.0.smithi115.stdout:551 2022-04-22T17:42:37.394 INFO:tasks.workunit.client.0.smithi115.stdout:552 2022-04-22T17:42:37.397 INFO:tasks.workunit.client.0.smithi115.stdout:553 2022-04-22T17:42:37.399 INFO:tasks.workunit.client.0.smithi115.stdout:554 2022-04-22T17:42:37.401 INFO:tasks.workunit.client.0.smithi115.stdout:555 2022-04-22T17:42:37.402 INFO:tasks.workunit.client.0.smithi115.stdout:556 2022-04-22T17:42:37.404 INFO:tasks.workunit.client.0.smithi115.stdout:557 2022-04-22T17:42:37.406 INFO:tasks.workunit.client.0.smithi115.stdout:558 2022-04-22T17:42:37.408 INFO:tasks.workunit.client.0.smithi115.stdout:559 2022-04-22T17:42:37.410 INFO:tasks.workunit.client.0.smithi115.stdout:560 2022-04-22T17:42:37.515 INFO:tasks.workunit.client.0.smithi115.stdout:561 2022-04-22T17:42:37.515 INFO:tasks.workunit.client.0.smithi115.stdout:562 2022-04-22T17:42:37.515 INFO:tasks.workunit.client.0.smithi115.stdout:563 2022-04-22T17:42:37.515 INFO:tasks.workunit.client.0.smithi115.stdout:564 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:565 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:566 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:567 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:568 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:569 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:570 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:571 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:572 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:573 2022-04-22T17:42:37.516 INFO:tasks.workunit.client.0.smithi115.stdout:574 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:575 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:576 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:577 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:578 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:579 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:580 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:581 2022-04-22T17:42:37.517 INFO:tasks.workunit.client.0.smithi115.stdout:582 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:583 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:584 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:585 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:586 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:587 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:588 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:589 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:590 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:591 2022-04-22T17:42:37.518 INFO:tasks.workunit.client.0.smithi115.stdout:592 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:593 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:594 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:595 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:596 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:597 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:598 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:599 2022-04-22T17:42:37.519 INFO:tasks.workunit.client.0.smithi115.stdout:600 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:601 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:602 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:603 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:604 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:605 2022-04-22T17:42:37.520 INFO:tasks.workunit.client.0.smithi115.stdout:606 2022-04-22T17:42:37.521 INFO:tasks.workunit.client.0.smithi115.stdout:607 2022-04-22T17:42:37.521 INFO:tasks.workunit.client.0.smithi115.stdout:608 2022-04-22T17:42:37.521 INFO:tasks.workunit.client.0.smithi115.stdout:609 2022-04-22T17:42:37.524 INFO:tasks.workunit.client.0.smithi115.stdout:610 2022-04-22T17:42:37.574 INFO:tasks.workunit.client.0.smithi115.stdout:611 2022-04-22T17:42:37.574 INFO:tasks.workunit.client.0.smithi115.stdout:612 2022-04-22T17:42:37.574 INFO:tasks.workunit.client.0.smithi115.stdout:613 2022-04-22T17:42:37.574 INFO:tasks.workunit.client.0.smithi115.stdout:614 2022-04-22T17:42:37.574 INFO:tasks.workunit.client.0.smithi115.stdout:615 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:616 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:617 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:618 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:619 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:620 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:621 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:622 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:623 2022-04-22T17:42:37.575 INFO:tasks.workunit.client.0.smithi115.stdout:624 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:625 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:626 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:627 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:628 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:629 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:630 2022-04-22T17:42:37.576 INFO:tasks.workunit.client.0.smithi115.stdout:631 2022-04-22T17:42:37.577 INFO:tasks.workunit.client.0.smithi115.stdout:632 2022-04-22T17:42:37.577 INFO:tasks.workunit.client.0.smithi115.stdout:633 2022-04-22T17:42:37.579 INFO:tasks.workunit.client.0.smithi115.stdout:634 2022-04-22T17:42:37.581 INFO:tasks.workunit.client.0.smithi115.stdout:635 2022-04-22T17:42:37.584 INFO:tasks.workunit.client.0.smithi115.stdout:636 2022-04-22T17:42:37.586 INFO:tasks.workunit.client.0.smithi115.stdout:637 2022-04-22T17:42:37.589 INFO:tasks.workunit.client.0.smithi115.stdout:638 2022-04-22T17:42:37.591 INFO:tasks.workunit.client.0.smithi115.stdout:639 2022-04-22T17:42:37.593 INFO:tasks.workunit.client.0.smithi115.stdout:640 2022-04-22T17:42:37.596 INFO:tasks.workunit.client.0.smithi115.stdout:641 2022-04-22T17:42:37.598 INFO:tasks.workunit.client.0.smithi115.stdout:642 2022-04-22T17:42:37.601 INFO:tasks.workunit.client.0.smithi115.stdout:643 2022-04-22T17:42:37.603 INFO:tasks.workunit.client.0.smithi115.stdout:644 2022-04-22T17:42:37.606 INFO:tasks.workunit.client.0.smithi115.stdout:645 2022-04-22T17:42:37.608 INFO:tasks.workunit.client.0.smithi115.stdout:646 2022-04-22T17:42:37.611 INFO:tasks.workunit.client.0.smithi115.stdout:647 2022-04-22T17:42:37.613 INFO:tasks.workunit.client.0.smithi115.stdout:648 2022-04-22T17:42:37.615 INFO:tasks.workunit.client.0.smithi115.stdout:649 2022-04-22T17:42:37.617 INFO:tasks.workunit.client.0.smithi115.stdout:650 2022-04-22T17:42:37.619 INFO:tasks.workunit.client.0.smithi115.stdout:651 2022-04-22T17:42:37.621 INFO:tasks.workunit.client.0.smithi115.stdout:652 2022-04-22T17:42:37.623 INFO:tasks.workunit.client.0.smithi115.stdout:653 2022-04-22T17:42:37.625 INFO:tasks.workunit.client.0.smithi115.stdout:654 2022-04-22T17:42:37.627 INFO:tasks.workunit.client.0.smithi115.stdout:655 2022-04-22T17:42:37.629 INFO:tasks.workunit.client.0.smithi115.stdout:656 2022-04-22T17:42:37.632 INFO:tasks.workunit.client.0.smithi115.stdout:657 2022-04-22T17:42:37.634 INFO:tasks.workunit.client.0.smithi115.stdout:658 2022-04-22T17:42:37.636 INFO:tasks.workunit.client.0.smithi115.stdout:659 2022-04-22T17:42:37.638 INFO:tasks.workunit.client.0.smithi115.stdout:660 2022-04-22T17:42:37.641 INFO:tasks.workunit.client.0.smithi115.stdout:661 2022-04-22T17:42:37.643 INFO:tasks.workunit.client.0.smithi115.stdout:662 2022-04-22T17:42:37.646 INFO:tasks.workunit.client.0.smithi115.stdout:663 2022-04-22T17:42:37.648 INFO:tasks.workunit.client.0.smithi115.stdout:664 2022-04-22T17:42:37.650 INFO:tasks.workunit.client.0.smithi115.stdout:665 2022-04-22T17:42:37.653 INFO:tasks.workunit.client.0.smithi115.stdout:666 2022-04-22T17:42:37.655 INFO:tasks.workunit.client.0.smithi115.stdout:667 2022-04-22T17:42:37.657 INFO:tasks.workunit.client.0.smithi115.stdout:668 2022-04-22T17:42:37.659 INFO:tasks.workunit.client.0.smithi115.stdout:669 2022-04-22T17:42:37.661 INFO:tasks.workunit.client.0.smithi115.stdout:670 2022-04-22T17:42:37.664 INFO:tasks.workunit.client.0.smithi115.stdout:671 2022-04-22T17:42:37.666 INFO:tasks.workunit.client.0.smithi115.stdout:672 2022-04-22T17:42:37.669 INFO:tasks.workunit.client.0.smithi115.stdout:673 2022-04-22T17:42:37.671 INFO:tasks.workunit.client.0.smithi115.stdout:674 2022-04-22T17:42:37.673 INFO:tasks.workunit.client.0.smithi115.stdout:675 2022-04-22T17:42:37.676 INFO:tasks.workunit.client.0.smithi115.stdout:676 2022-04-22T17:42:37.678 INFO:tasks.workunit.client.0.smithi115.stdout:677 2022-04-22T17:42:37.680 INFO:tasks.workunit.client.0.smithi115.stdout:678 2022-04-22T17:42:37.683 INFO:tasks.workunit.client.0.smithi115.stdout:679 2022-04-22T17:42:37.685 INFO:tasks.workunit.client.0.smithi115.stdout:680 2022-04-22T17:42:37.687 INFO:tasks.workunit.client.0.smithi115.stdout:681 2022-04-22T17:42:37.689 INFO:tasks.workunit.client.0.smithi115.stdout:682 2022-04-22T17:42:37.692 INFO:tasks.workunit.client.0.smithi115.stdout:683 2022-04-22T17:42:37.694 INFO:tasks.workunit.client.0.smithi115.stdout:684 2022-04-22T17:42:37.696 INFO:tasks.workunit.client.0.smithi115.stdout:685 2022-04-22T17:42:37.698 INFO:tasks.workunit.client.0.smithi115.stdout:686 2022-04-22T17:42:37.701 INFO:tasks.workunit.client.0.smithi115.stdout:687 2022-04-22T17:42:37.703 INFO:tasks.workunit.client.0.smithi115.stdout:688 2022-04-22T17:42:37.705 INFO:tasks.workunit.client.0.smithi115.stdout:689 2022-04-22T17:42:37.707 INFO:tasks.workunit.client.0.smithi115.stdout:690 2022-04-22T17:42:37.710 INFO:tasks.workunit.client.0.smithi115.stdout:691 2022-04-22T17:42:37.713 INFO:tasks.workunit.client.0.smithi115.stdout:692 2022-04-22T17:42:37.715 INFO:tasks.workunit.client.0.smithi115.stdout:693 2022-04-22T17:42:37.717 INFO:tasks.workunit.client.0.smithi115.stdout:694 2022-04-22T17:42:37.719 INFO:tasks.workunit.client.0.smithi115.stdout:695 2022-04-22T17:42:37.722 INFO:tasks.workunit.client.0.smithi115.stdout:696 2022-04-22T17:42:37.727 INFO:tasks.workunit.client.0.smithi115.stdout:697 2022-04-22T17:42:37.731 INFO:tasks.workunit.client.0.smithi115.stdout:698 2022-04-22T17:42:37.734 INFO:tasks.workunit.client.0.smithi115.stdout:699 2022-04-22T17:42:37.736 INFO:tasks.workunit.client.0.smithi115.stdout:700 2022-04-22T17:42:37.738 INFO:tasks.workunit.client.0.smithi115.stdout:701 2022-04-22T17:42:37.740 INFO:tasks.workunit.client.0.smithi115.stdout:702 2022-04-22T17:42:37.742 INFO:tasks.workunit.client.0.smithi115.stdout:703 2022-04-22T17:42:37.744 INFO:tasks.workunit.client.0.smithi115.stdout:704 2022-04-22T17:42:37.746 INFO:tasks.workunit.client.0.smithi115.stdout:705 2022-04-22T17:42:37.749 INFO:tasks.workunit.client.0.smithi115.stdout:706 2022-04-22T17:42:37.751 INFO:tasks.workunit.client.0.smithi115.stdout:707 2022-04-22T17:42:37.754 INFO:tasks.workunit.client.0.smithi115.stdout:708 2022-04-22T17:42:37.756 INFO:tasks.workunit.client.0.smithi115.stdout:709 2022-04-22T17:42:37.759 INFO:tasks.workunit.client.0.smithi115.stdout:710 2022-04-22T17:42:37.761 INFO:tasks.workunit.client.0.smithi115.stdout:711 2022-04-22T17:42:37.763 INFO:tasks.workunit.client.0.smithi115.stdout:712 2022-04-22T17:42:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:37 smithi121 bash[36902]: audit 2022-04-22T17:42:36.304982+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:37 smithi121 bash[36902]: cluster 2022-04-22T17:42:36.305245+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2022-04-22T17:42:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:37 smithi121 bash[36902]: audit 2022-04-22T17:42:36.307679+0000 mon.a (mon.0) 1800 : audit [DBG] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2022-04-22T17:42:37.766 INFO:tasks.workunit.client.0.smithi115.stdout:713 2022-04-22T17:42:37.767 INFO:tasks.workunit.client.0.smithi115.stdout:714 2022-04-22T17:42:37.769 INFO:tasks.workunit.client.0.smithi115.stdout:715 2022-04-22T17:42:37.777 INFO:tasks.workunit.client.0.smithi115.stdout:716 2022-04-22T17:42:37.777 INFO:tasks.workunit.client.0.smithi115.stdout:717 2022-04-22T17:42:37.777 INFO:tasks.workunit.client.0.smithi115.stdout:718 2022-04-22T17:42:37.778 INFO:tasks.workunit.client.0.smithi115.stdout:719 2022-04-22T17:42:37.780 INFO:tasks.workunit.client.0.smithi115.stdout:720 2022-04-22T17:42:37.782 INFO:tasks.workunit.client.0.smithi115.stdout:721 2022-04-22T17:42:37.784 INFO:tasks.workunit.client.0.smithi115.stdout:722 2022-04-22T17:42:37.787 INFO:tasks.workunit.client.0.smithi115.stdout:723 2022-04-22T17:42:37.789 INFO:tasks.workunit.client.0.smithi115.stdout:724 2022-04-22T17:42:37.791 INFO:tasks.workunit.client.0.smithi115.stdout:725 2022-04-22T17:42:37.794 INFO:tasks.workunit.client.0.smithi115.stdout:726 2022-04-22T17:42:37.796 INFO:tasks.workunit.client.0.smithi115.stdout:727 2022-04-22T17:42:37.799 INFO:tasks.workunit.client.0.smithi115.stdout:728 2022-04-22T17:42:37.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[45495]: audit 2022-04-22T17:42:36.304982+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:37.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[45495]: cluster 2022-04-22T17:42:36.305245+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2022-04-22T17:42:37.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[45495]: audit 2022-04-22T17:42:36.307679+0000 mon.a (mon.0) 1800 : audit [DBG] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2022-04-22T17:42:37.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[46934]: audit 2022-04-22T17:42:36.304982+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:37.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[46934]: cluster 2022-04-22T17:42:36.305245+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2022-04-22T17:42:37.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:37 smithi115 bash[46934]: audit 2022-04-22T17:42:36.307679+0000 mon.a (mon.0) 1800 : audit [DBG] from='client.? 172.21.15.115:0/3575527813' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2022-04-22T17:42:37.801 INFO:tasks.workunit.client.0.smithi115.stdout:729 2022-04-22T17:42:37.803 INFO:tasks.workunit.client.0.smithi115.stdout:730 2022-04-22T17:42:37.806 INFO:tasks.workunit.client.0.smithi115.stdout:731 2022-04-22T17:42:37.808 INFO:tasks.workunit.client.0.smithi115.stdout:732 2022-04-22T17:42:37.811 INFO:tasks.workunit.client.0.smithi115.stdout:733 2022-04-22T17:42:37.813 INFO:tasks.workunit.client.0.smithi115.stdout:734 2022-04-22T17:42:37.815 INFO:tasks.workunit.client.0.smithi115.stdout:735 2022-04-22T17:42:37.817 INFO:tasks.workunit.client.0.smithi115.stdout:736 2022-04-22T17:42:37.819 INFO:tasks.workunit.client.0.smithi115.stdout:737 2022-04-22T17:42:37.821 INFO:tasks.workunit.client.0.smithi115.stdout:738 2022-04-22T17:42:37.824 INFO:tasks.workunit.client.0.smithi115.stdout:739 2022-04-22T17:42:37.826 INFO:tasks.workunit.client.0.smithi115.stdout:740 2022-04-22T17:42:37.829 INFO:tasks.workunit.client.0.smithi115.stdout:741 2022-04-22T17:42:37.831 INFO:tasks.workunit.client.0.smithi115.stdout:742 2022-04-22T17:42:37.834 INFO:tasks.workunit.client.0.smithi115.stdout:743 2022-04-22T17:42:37.837 INFO:tasks.workunit.client.0.smithi115.stdout:744 2022-04-22T17:42:37.839 INFO:tasks.workunit.client.0.smithi115.stdout:745 2022-04-22T17:42:37.842 INFO:tasks.workunit.client.0.smithi115.stdout:746 2022-04-22T17:42:37.845 INFO:tasks.workunit.client.0.smithi115.stdout:747 2022-04-22T17:42:37.847 INFO:tasks.workunit.client.0.smithi115.stdout:748 2022-04-22T17:42:37.850 INFO:tasks.workunit.client.0.smithi115.stdout:749 2022-04-22T17:42:37.852 INFO:tasks.workunit.client.0.smithi115.stdout:750 2022-04-22T17:42:37.854 INFO:tasks.workunit.client.0.smithi115.stdout:751 2022-04-22T17:42:37.857 INFO:tasks.workunit.client.0.smithi115.stdout:752 2022-04-22T17:42:37.859 INFO:tasks.workunit.client.0.smithi115.stdout:753 2022-04-22T17:42:37.862 INFO:tasks.workunit.client.0.smithi115.stdout:754 2022-04-22T17:42:37.864 INFO:tasks.workunit.client.0.smithi115.stdout:755 2022-04-22T17:42:37.867 INFO:tasks.workunit.client.0.smithi115.stdout:756 2022-04-22T17:42:37.870 INFO:tasks.workunit.client.0.smithi115.stdout:757 2022-04-22T17:42:37.872 INFO:tasks.workunit.client.0.smithi115.stdout:758 2022-04-22T17:42:37.874 INFO:tasks.workunit.client.0.smithi115.stdout:759 2022-04-22T17:42:37.877 INFO:tasks.workunit.client.0.smithi115.stdout:760 2022-04-22T17:42:37.879 INFO:tasks.workunit.client.0.smithi115.stdout:761 2022-04-22T17:42:37.881 INFO:tasks.workunit.client.0.smithi115.stdout:762 2022-04-22T17:42:37.884 INFO:tasks.workunit.client.0.smithi115.stdout:763 2022-04-22T17:42:37.886 INFO:tasks.workunit.client.0.smithi115.stdout:764 2022-04-22T17:42:37.889 INFO:tasks.workunit.client.0.smithi115.stdout:765 2022-04-22T17:42:37.891 INFO:tasks.workunit.client.0.smithi115.stdout:766 2022-04-22T17:42:37.893 INFO:tasks.workunit.client.0.smithi115.stdout:767 2022-04-22T17:42:37.896 INFO:tasks.workunit.client.0.smithi115.stdout:768 2022-04-22T17:42:37.898 INFO:tasks.workunit.client.0.smithi115.stdout:769 2022-04-22T17:42:37.901 INFO:tasks.workunit.client.0.smithi115.stdout:770 2022-04-22T17:42:37.903 INFO:tasks.workunit.client.0.smithi115.stdout:771 2022-04-22T17:42:37.905 INFO:tasks.workunit.client.0.smithi115.stdout:772 2022-04-22T17:42:37.908 INFO:tasks.workunit.client.0.smithi115.stdout:773 2022-04-22T17:42:37.910 INFO:tasks.workunit.client.0.smithi115.stdout:774 2022-04-22T17:42:37.912 INFO:tasks.workunit.client.0.smithi115.stdout:775 2022-04-22T17:42:37.914 INFO:tasks.workunit.client.0.smithi115.stdout:776 2022-04-22T17:42:37.917 INFO:tasks.workunit.client.0.smithi115.stdout:777 2022-04-22T17:42:37.919 INFO:tasks.workunit.client.0.smithi115.stdout:778 2022-04-22T17:42:37.922 INFO:tasks.workunit.client.0.smithi115.stdout:779 2022-04-22T17:42:37.924 INFO:tasks.workunit.client.0.smithi115.stdout:780 2022-04-22T17:42:37.926 INFO:tasks.workunit.client.0.smithi115.stdout:781 2022-04-22T17:42:37.928 INFO:tasks.workunit.client.0.smithi115.stdout:782 2022-04-22T17:42:37.930 INFO:tasks.workunit.client.0.smithi115.stdout:783 2022-04-22T17:42:37.933 INFO:tasks.workunit.client.0.smithi115.stdout:784 2022-04-22T17:42:37.935 INFO:tasks.workunit.client.0.smithi115.stdout:785 2022-04-22T17:42:37.938 INFO:tasks.workunit.client.0.smithi115.stdout:786 2022-04-22T17:42:37.940 INFO:tasks.workunit.client.0.smithi115.stdout:787 2022-04-22T17:42:37.943 INFO:tasks.workunit.client.0.smithi115.stdout:788 2022-04-22T17:42:37.945 INFO:tasks.workunit.client.0.smithi115.stdout:789 2022-04-22T17:42:37.947 INFO:tasks.workunit.client.0.smithi115.stdout:790 2022-04-22T17:42:37.950 INFO:tasks.workunit.client.0.smithi115.stdout:791 2022-04-22T17:42:37.953 INFO:tasks.workunit.client.0.smithi115.stdout:792 2022-04-22T17:42:37.955 INFO:tasks.workunit.client.0.smithi115.stdout:793 2022-04-22T17:42:37.957 INFO:tasks.workunit.client.0.smithi115.stdout:794 2022-04-22T17:42:37.960 INFO:tasks.workunit.client.0.smithi115.stdout:795 2022-04-22T17:42:37.963 INFO:tasks.workunit.client.0.smithi115.stdout:796 2022-04-22T17:42:37.965 INFO:tasks.workunit.client.0.smithi115.stdout:797 2022-04-22T17:42:37.967 INFO:tasks.workunit.client.0.smithi115.stdout:798 2022-04-22T17:42:37.970 INFO:tasks.workunit.client.0.smithi115.stdout:799 2022-04-22T17:42:37.972 INFO:tasks.workunit.client.0.smithi115.stdout:800 2022-04-22T17:42:37.975 INFO:tasks.workunit.client.0.smithi115.stdout:801 2022-04-22T17:42:37.977 INFO:tasks.workunit.client.0.smithi115.stdout:802 2022-04-22T17:42:37.979 INFO:tasks.workunit.client.0.smithi115.stdout:803 2022-04-22T17:42:37.981 INFO:tasks.workunit.client.0.smithi115.stdout:804 2022-04-22T17:42:37.983 INFO:tasks.workunit.client.0.smithi115.stdout:805 2022-04-22T17:42:37.985 INFO:tasks.workunit.client.0.smithi115.stdout:806 2022-04-22T17:42:37.988 INFO:tasks.workunit.client.0.smithi115.stdout:807 2022-04-22T17:42:37.991 INFO:tasks.workunit.client.0.smithi115.stdout:808 2022-04-22T17:42:37.993 INFO:tasks.workunit.client.0.smithi115.stdout:809 2022-04-22T17:42:37.995 INFO:tasks.workunit.client.0.smithi115.stdout:810 2022-04-22T17:42:37.998 INFO:tasks.workunit.client.0.smithi115.stdout:811 2022-04-22T17:42:38.000 INFO:tasks.workunit.client.0.smithi115.stdout:812 2022-04-22T17:42:38.003 INFO:tasks.workunit.client.0.smithi115.stdout:813 2022-04-22T17:42:38.006 INFO:tasks.workunit.client.0.smithi115.stdout:814 2022-04-22T17:42:38.008 INFO:tasks.workunit.client.0.smithi115.stdout:815 2022-04-22T17:42:38.011 INFO:tasks.workunit.client.0.smithi115.stdout:816 2022-04-22T17:42:38.013 INFO:tasks.workunit.client.0.smithi115.stdout:817 2022-04-22T17:42:38.016 INFO:tasks.workunit.client.0.smithi115.stdout:818 2022-04-22T17:42:38.018 INFO:tasks.workunit.client.0.smithi115.stdout:819 2022-04-22T17:42:38.021 INFO:tasks.workunit.client.0.smithi115.stdout:820 2022-04-22T17:42:38.023 INFO:tasks.workunit.client.0.smithi115.stdout:821 2022-04-22T17:42:38.026 INFO:tasks.workunit.client.0.smithi115.stdout:822 2022-04-22T17:42:38.028 INFO:tasks.workunit.client.0.smithi115.stdout:823 2022-04-22T17:42:38.031 INFO:tasks.workunit.client.0.smithi115.stdout:824 2022-04-22T17:42:38.033 INFO:tasks.workunit.client.0.smithi115.stdout:825 2022-04-22T17:42:38.036 INFO:tasks.workunit.client.0.smithi115.stdout:826 2022-04-22T17:42:38.038 INFO:tasks.workunit.client.0.smithi115.stdout:827 2022-04-22T17:42:38.041 INFO:tasks.workunit.client.0.smithi115.stdout:828 2022-04-22T17:42:38.044 INFO:tasks.workunit.client.0.smithi115.stdout:829 2022-04-22T17:42:38.047 INFO:tasks.workunit.client.0.smithi115.stdout:830 2022-04-22T17:42:38.050 INFO:tasks.workunit.client.0.smithi115.stdout:831 2022-04-22T17:42:38.052 INFO:tasks.workunit.client.0.smithi115.stdout:832 2022-04-22T17:42:38.055 INFO:tasks.workunit.client.0.smithi115.stdout:833 2022-04-22T17:42:38.057 INFO:tasks.workunit.client.0.smithi115.stdout:834 2022-04-22T17:42:38.060 INFO:tasks.workunit.client.0.smithi115.stdout:835 2022-04-22T17:42:38.062 INFO:tasks.workunit.client.0.smithi115.stdout:836 2022-04-22T17:42:38.064 INFO:tasks.workunit.client.0.smithi115.stdout:837 2022-04-22T17:42:38.066 INFO:tasks.workunit.client.0.smithi115.stdout:838 2022-04-22T17:42:38.069 INFO:tasks.workunit.client.0.smithi115.stdout:839 2022-04-22T17:42:38.071 INFO:tasks.workunit.client.0.smithi115.stdout:840 2022-04-22T17:42:38.073 INFO:tasks.workunit.client.0.smithi115.stdout:841 2022-04-22T17:42:38.076 INFO:tasks.workunit.client.0.smithi115.stdout:842 2022-04-22T17:42:38.078 INFO:tasks.workunit.client.0.smithi115.stdout:843 2022-04-22T17:42:38.080 INFO:tasks.workunit.client.0.smithi115.stdout:844 2022-04-22T17:42:38.083 INFO:tasks.workunit.client.0.smithi115.stdout:845 2022-04-22T17:42:38.085 INFO:tasks.workunit.client.0.smithi115.stdout:846 2022-04-22T17:42:38.088 INFO:tasks.workunit.client.0.smithi115.stdout:847 2022-04-22T17:42:38.090 INFO:tasks.workunit.client.0.smithi115.stdout:848 2022-04-22T17:42:38.093 INFO:tasks.workunit.client.0.smithi115.stdout:849 2022-04-22T17:42:38.095 INFO:tasks.workunit.client.0.smithi115.stdout:850 2022-04-22T17:42:38.098 INFO:tasks.workunit.client.0.smithi115.stdout:851 2022-04-22T17:42:38.100 INFO:tasks.workunit.client.0.smithi115.stdout:852 2022-04-22T17:42:38.103 INFO:tasks.workunit.client.0.smithi115.stdout:853 2022-04-22T17:42:38.105 INFO:tasks.workunit.client.0.smithi115.stdout:854 2022-04-22T17:42:38.107 INFO:tasks.workunit.client.0.smithi115.stdout:855 2022-04-22T17:42:38.110 INFO:tasks.workunit.client.0.smithi115.stdout:856 2022-04-22T17:42:38.112 INFO:tasks.workunit.client.0.smithi115.stdout:857 2022-04-22T17:42:38.115 INFO:tasks.workunit.client.0.smithi115.stdout:858 2022-04-22T17:42:38.117 INFO:tasks.workunit.client.0.smithi115.stdout:859 2022-04-22T17:42:38.119 INFO:tasks.workunit.client.0.smithi115.stdout:860 2022-04-22T17:42:38.122 INFO:tasks.workunit.client.0.smithi115.stdout:861 2022-04-22T17:42:38.170 INFO:tasks.workunit.client.0.smithi115.stdout:862 2022-04-22T17:42:38.178 INFO:tasks.workunit.client.0.smithi115.stdout:863 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:864 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:865 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:866 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:867 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:868 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:869 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:870 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:871 2022-04-22T17:42:38.179 INFO:tasks.workunit.client.0.smithi115.stdout:872 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:873 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:874 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:875 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:876 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:877 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:878 2022-04-22T17:42:38.180 INFO:tasks.workunit.client.0.smithi115.stdout:879 2022-04-22T17:42:38.181 INFO:tasks.workunit.client.0.smithi115.stdout:880 2022-04-22T17:42:38.181 INFO:tasks.workunit.client.0.smithi115.stdout:881 2022-04-22T17:42:38.182 INFO:tasks.workunit.client.0.smithi115.stdout:882 2022-04-22T17:42:38.182 INFO:tasks.workunit.client.0.smithi115.stdout:883 2022-04-22T17:42:38.182 INFO:tasks.workunit.client.0.smithi115.stdout:884 2022-04-22T17:42:38.182 INFO:tasks.workunit.client.0.smithi115.stdout:885 2022-04-22T17:42:38.182 INFO:tasks.workunit.client.0.smithi115.stdout:886 2022-04-22T17:42:38.185 INFO:tasks.workunit.client.0.smithi115.stdout:887 2022-04-22T17:42:38.187 INFO:tasks.workunit.client.0.smithi115.stdout:888 2022-04-22T17:42:38.190 INFO:tasks.workunit.client.0.smithi115.stdout:889 2022-04-22T17:42:38.192 INFO:tasks.workunit.client.0.smithi115.stdout:890 2022-04-22T17:42:38.195 INFO:tasks.workunit.client.0.smithi115.stdout:891 2022-04-22T17:42:38.197 INFO:tasks.workunit.client.0.smithi115.stdout:892 2022-04-22T17:42:38.200 INFO:tasks.workunit.client.0.smithi115.stdout:893 2022-04-22T17:42:38.202 INFO:tasks.workunit.client.0.smithi115.stdout:894 2022-04-22T17:42:38.204 INFO:tasks.workunit.client.0.smithi115.stdout:895 2022-04-22T17:42:38.207 INFO:tasks.workunit.client.0.smithi115.stdout:896 2022-04-22T17:42:38.210 INFO:tasks.workunit.client.0.smithi115.stdout:897 2022-04-22T17:42:38.212 INFO:tasks.workunit.client.0.smithi115.stdout:898 2022-04-22T17:42:38.215 INFO:tasks.workunit.client.0.smithi115.stdout:899 2022-04-22T17:42:38.217 INFO:tasks.workunit.client.0.smithi115.stdout:900 2022-04-22T17:42:38.220 INFO:tasks.workunit.client.0.smithi115.stdout:901 2022-04-22T17:42:38.223 INFO:tasks.workunit.client.0.smithi115.stdout:902 2022-04-22T17:42:38.225 INFO:tasks.workunit.client.0.smithi115.stdout:903 2022-04-22T17:42:38.227 INFO:tasks.workunit.client.0.smithi115.stdout:904 2022-04-22T17:42:38.230 INFO:tasks.workunit.client.0.smithi115.stdout:905 2022-04-22T17:42:38.232 INFO:tasks.workunit.client.0.smithi115.stdout:906 2022-04-22T17:42:38.235 INFO:tasks.workunit.client.0.smithi115.stdout:907 2022-04-22T17:42:38.290 INFO:tasks.workunit.client.0.smithi115.stdout:908 2022-04-22T17:42:38.290 INFO:tasks.workunit.client.0.smithi115.stdout:909 2022-04-22T17:42:38.290 INFO:tasks.workunit.client.0.smithi115.stdout:910 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:911 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:912 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:913 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:914 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:915 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:916 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:917 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:918 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:919 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:920 2022-04-22T17:42:38.291 INFO:tasks.workunit.client.0.smithi115.stdout:921 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:922 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:923 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:924 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:925 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:926 2022-04-22T17:42:38.292 INFO:tasks.workunit.client.0.smithi115.stdout:927 2022-04-22T17:42:38.293 INFO:tasks.workunit.client.0.smithi115.stdout:928 2022-04-22T17:42:38.293 INFO:tasks.workunit.client.0.smithi115.stdout:929 2022-04-22T17:42:38.294 INFO:tasks.workunit.client.0.smithi115.stdout:930 2022-04-22T17:42:38.296 INFO:tasks.workunit.client.0.smithi115.stdout:931 2022-04-22T17:42:38.299 INFO:tasks.workunit.client.0.smithi115.stdout:932 2022-04-22T17:42:38.301 INFO:tasks.workunit.client.0.smithi115.stdout:933 2022-04-22T17:42:38.304 INFO:tasks.workunit.client.0.smithi115.stdout:934 2022-04-22T17:42:38.306 INFO:tasks.workunit.client.0.smithi115.stdout:935 2022-04-22T17:42:38.309 INFO:tasks.workunit.client.0.smithi115.stdout:936 2022-04-22T17:42:38.311 INFO:tasks.workunit.client.0.smithi115.stdout:937 2022-04-22T17:42:38.313 INFO:tasks.workunit.client.0.smithi115.stdout:938 2022-04-22T17:42:38.316 INFO:tasks.workunit.client.0.smithi115.stdout:939 2022-04-22T17:42:38.318 INFO:tasks.workunit.client.0.smithi115.stdout:940 2022-04-22T17:42:38.321 INFO:tasks.workunit.client.0.smithi115.stdout:941 2022-04-22T17:42:38.323 INFO:tasks.workunit.client.0.smithi115.stdout:942 2022-04-22T17:42:38.326 INFO:tasks.workunit.client.0.smithi115.stdout:943 2022-04-22T17:42:38.328 INFO:tasks.workunit.client.0.smithi115.stdout:944 2022-04-22T17:42:38.331 INFO:tasks.workunit.client.0.smithi115.stdout:945 2022-04-22T17:42:38.334 INFO:tasks.workunit.client.0.smithi115.stdout:946 2022-04-22T17:42:38.336 INFO:tasks.workunit.client.0.smithi115.stdout:947 2022-04-22T17:42:38.338 INFO:tasks.workunit.client.0.smithi115.stdout:948 2022-04-22T17:42:38.341 INFO:tasks.workunit.client.0.smithi115.stdout:949 2022-04-22T17:42:38.344 INFO:tasks.workunit.client.0.smithi115.stdout:950 2022-04-22T17:42:38.347 INFO:tasks.workunit.client.0.smithi115.stdout:951 2022-04-22T17:42:38.349 INFO:tasks.workunit.client.0.smithi115.stdout:952 2022-04-22T17:42:38.351 INFO:tasks.workunit.client.0.smithi115.stdout:953 2022-04-22T17:42:38.353 INFO:tasks.workunit.client.0.smithi115.stdout:954 2022-04-22T17:42:38.356 INFO:tasks.workunit.client.0.smithi115.stdout:955 2022-04-22T17:42:38.437 INFO:tasks.workunit.client.0.smithi115.stdout:956 2022-04-22T17:42:38.437 INFO:tasks.workunit.client.0.smithi115.stdout:957 2022-04-22T17:42:38.437 INFO:tasks.workunit.client.0.smithi115.stdout:958 2022-04-22T17:42:38.437 INFO:tasks.workunit.client.0.smithi115.stdout:959 2022-04-22T17:42:38.438 INFO:tasks.workunit.client.0.smithi115.stdout:960 2022-04-22T17:42:38.438 INFO:tasks.workunit.client.0.smithi115.stdout:961 2022-04-22T17:42:38.438 INFO:tasks.workunit.client.0.smithi115.stdout:962 2022-04-22T17:42:38.438 INFO:tasks.workunit.client.0.smithi115.stdout:963 2022-04-22T17:42:38.439 INFO:tasks.workunit.client.0.smithi115.stdout:964 2022-04-22T17:42:38.439 INFO:tasks.workunit.client.0.smithi115.stdout:965 2022-04-22T17:42:38.439 INFO:tasks.workunit.client.0.smithi115.stdout:966 2022-04-22T17:42:38.439 INFO:tasks.workunit.client.0.smithi115.stdout:967 2022-04-22T17:42:38.439 INFO:tasks.workunit.client.0.smithi115.stdout:968 2022-04-22T17:42:38.440 INFO:tasks.workunit.client.0.smithi115.stdout:969 2022-04-22T17:42:38.440 INFO:tasks.workunit.client.0.smithi115.stdout:970 2022-04-22T17:42:38.440 INFO:tasks.workunit.client.0.smithi115.stdout:971 2022-04-22T17:42:38.441 INFO:tasks.workunit.client.0.smithi115.stdout:972 2022-04-22T17:42:38.441 INFO:tasks.workunit.client.0.smithi115.stdout:973 2022-04-22T17:42:38.441 INFO:tasks.workunit.client.0.smithi115.stdout:974 2022-04-22T17:42:38.441 INFO:tasks.workunit.client.0.smithi115.stdout:975 2022-04-22T17:42:38.442 INFO:tasks.workunit.client.0.smithi115.stdout:976 2022-04-22T17:42:38.442 INFO:tasks.workunit.client.0.smithi115.stdout:977 2022-04-22T17:42:38.442 INFO:tasks.workunit.client.0.smithi115.stdout:978 2022-04-22T17:42:38.443 INFO:tasks.workunit.client.0.smithi115.stdout:979 2022-04-22T17:42:38.443 INFO:tasks.workunit.client.0.smithi115.stdout:980 2022-04-22T17:42:38.443 INFO:tasks.workunit.client.0.smithi115.stdout:981 2022-04-22T17:42:38.443 INFO:tasks.workunit.client.0.smithi115.stdout:982 2022-04-22T17:42:38.443 INFO:tasks.workunit.client.0.smithi115.stdout:983 2022-04-22T17:42:38.444 INFO:tasks.workunit.client.0.smithi115.stdout:984 2022-04-22T17:42:38.444 INFO:tasks.workunit.client.0.smithi115.stdout:985 2022-04-22T17:42:38.444 INFO:tasks.workunit.client.0.smithi115.stdout:986 2022-04-22T17:42:38.446 INFO:tasks.workunit.client.0.smithi115.stdout:987 2022-04-22T17:42:38.446 INFO:tasks.workunit.client.0.smithi115.stdout:988 2022-04-22T17:42:38.446 INFO:tasks.workunit.client.0.smithi115.stdout:989 2022-04-22T17:42:38.446 INFO:tasks.workunit.client.0.smithi115.stdout:990 2022-04-22T17:42:38.448 INFO:tasks.workunit.client.0.smithi115.stdout:991 2022-04-22T17:42:38.451 INFO:tasks.workunit.client.0.smithi115.stdout:992 2022-04-22T17:42:38.453 INFO:tasks.workunit.client.0.smithi115.stdout:993 2022-04-22T17:42:38.455 INFO:tasks.workunit.client.0.smithi115.stdout:994 2022-04-22T17:42:38.458 INFO:tasks.workunit.client.0.smithi115.stdout:995 2022-04-22T17:42:38.460 INFO:tasks.workunit.client.0.smithi115.stdout:996 2022-04-22T17:42:38.463 INFO:tasks.workunit.client.0.smithi115.stdout:997 2022-04-22T17:42:38.466 INFO:tasks.workunit.client.0.smithi115.stdout:998 2022-04-22T17:42:38.468 INFO:tasks.workunit.client.0.smithi115.stdout:999 2022-04-22T17:42:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:38 smithi121 bash[36902]: cluster 2022-04-22T17:42:37.318248+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2022-04-22T17:42:38.849 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:38 smithi121 bash[36902]: cluster 2022-04-22T17:42:38.108493+0000 mgr.y (mgr.34107) 620 : cluster [DBG] pgmap v1164: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2022-04-22T17:42:38.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:38 smithi115 bash[45495]: cluster 2022-04-22T17:42:37.318248+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2022-04-22T17:42:38.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:38 smithi115 bash[45495]: cluster 2022-04-22T17:42:38.108493+0000 mgr.y (mgr.34107) 620 : cluster [DBG] pgmap v1164: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2022-04-22T17:42:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:38 smithi115 bash[46934]: cluster 2022-04-22T17:42:37.318248+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2022-04-22T17:42:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:38 smithi115 bash[46934]: cluster 2022-04-22T17:42:38.108493+0000 mgr.y (mgr.34107) 620 : cluster [DBG] pgmap v1164: 137 pgs: 5 creating+peering, 19 unknown, 113 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2022-04-22T17:42:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2022-04-22T17:42:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:00000010 69 73 |is| 2022-04-22T17:42:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:00000012 2022-04-22T17:42:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2022-04-22T17:42:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:00000010 69 73 |is| 2022-04-22T17:42:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:00000012 2022-04-22T17:42:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.WriteReturnData (5039 ms) 2022-04-22T17:42:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.Loud 2022-04-22T17:42:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:40 smithi121 bash[36902]: cluster 2022-04-22T17:42:39.338117+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2022-04-22T17:42:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:40 smithi121 bash[36902]: audit 2022-04-22T17:42:39.804836+0000 mon.a (mon.0) 1803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:40 smithi121 bash[36902]: cluster 2022-04-22T17:42:40.109371+0000 mgr.y (mgr.34107) 621 : cluster [DBG] pgmap v1166: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:40.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[45495]: cluster 2022-04-22T17:42:39.338117+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2022-04-22T17:42:40.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[45495]: audit 2022-04-22T17:42:39.804836+0000 mon.a (mon.0) 1803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:40.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[45495]: cluster 2022-04-22T17:42:40.109371+0000 mgr.y (mgr.34107) 621 : cluster [DBG] pgmap v1166: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:40.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[46934]: cluster 2022-04-22T17:42:39.338117+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2022-04-22T17:42:40.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[46934]: audit 2022-04-22T17:42:39.804836+0000 mon.a (mon.0) 1803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:40.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:40 smithi115 bash[46934]: cluster 2022-04-22T17:42:40.109371+0000 mgr.y (mgr.34107) 621 : cluster [DBG] pgmap v1166: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:41 smithi115 bash[45495]: cluster 2022-04-22T17:42:40.345767+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2022-04-22T17:42:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:41 smithi115 bash[45495]: audit 2022-04-22T17:42:40.352657+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:41 smithi115 bash[46934]: cluster 2022-04-22T17:42:40.345767+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2022-04-22T17:42:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:41 smithi115 bash[46934]: audit 2022-04-22T17:42:40.352657+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:41 smithi121 bash[36902]: cluster 2022-04-22T17:42:40.345767+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2022-04-22T17:42:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:41 smithi121 bash[36902]: audit 2022-04-22T17:42:40.352657+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:42.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:41.645Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:42.112 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:42:42.388 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.Loud (3033 ms) 2022-04-22T17:42:42.388 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.BadMethods 2022-04-22T17:42:42.449 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:42:42.449 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5m) 5m ago 19m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (19m) 6m ago 19m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (7m) 5m ago 20m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (7m) 6m ago 20m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (15m) 6m ago 24m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (15m) 5m ago 27m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (14m) 5m ago 27m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (13m) 6m ago 25m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:42:42.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (14m) 5m ago 25m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 5m ago 19m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6m) 6m ago 19m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (12m) 5m ago 24m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 5m ago 23m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 5m ago 23m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (10m) 5m ago 22m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:42:42.500 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (10m) 6m ago 22m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:42:42.502 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (9m) 6m ago 22m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:42:42.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (8m) 6m ago 21m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:42:42.503 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (8m) 6m ago 21m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:42:42.503 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (6m) 6m ago 19m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:42:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:42 smithi121 bash[36902]: audit 2022-04-22T17:42:41.353483+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:42 smithi121 bash[36902]: cluster 2022-04-22T17:42:41.353711+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2022-04-22T17:42:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:42 smithi121 bash[36902]: audit 2022-04-22T17:42:42.084298+0000 mgr.y (mgr.34107) 622 : audit [DBG] from='client.34863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:42 smithi121 bash[36902]: cluster 2022-04-22T17:42:42.110275+0000 mgr.y (mgr.34107) 623 : cluster [DBG] pgmap v1169: 137 pgs: 6 creating+peering, 19 unknown, 112 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:42.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[46934]: audit 2022-04-22T17:42:41.353483+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:42.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[46934]: cluster 2022-04-22T17:42:41.353711+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2022-04-22T17:42:42.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[46934]: audit 2022-04-22T17:42:42.084298+0000 mgr.y (mgr.34107) 622 : audit [DBG] from='client.34863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:42.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[46934]: cluster 2022-04-22T17:42:42.110275+0000 mgr.y (mgr.34107) 623 : cluster [DBG] pgmap v1169: 137 pgs: 6 creating+peering, 19 unknown, 112 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:42.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[45495]: audit 2022-04-22T17:42:41.353483+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.115:0/2415037366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:42.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[45495]: cluster 2022-04-22T17:42:41.353711+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2022-04-22T17:42:42.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[45495]: audit 2022-04-22T17:42:42.084298+0000 mgr.y (mgr.34107) 622 : audit [DBG] from='client.34863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:42.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:42 smithi115 bash[45495]: cluster 2022-04-22T17:42:42.110275+0000 mgr.y (mgr.34107) 623 : cluster [DBG] pgmap v1169: 137 pgs: 6 creating+peering, 19 unknown, 112 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:42.869 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:42:42.869 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:42:42.869 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:42:42.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:42:42.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:42.871 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:42:42.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:42:42.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:42:42.871 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:42:42.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:42:42.872 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:42:42.872 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:42:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:43 smithi121 bash[36902]: cluster 2022-04-22T17:42:42.369011+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2022-04-22T17:42:43.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:43 smithi121 bash[36902]: audit 2022-04-22T17:42:42.421916+0000 mgr.y (mgr.34107) 624 : audit [DBG] from='client.34869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:43.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:43 smithi121 bash[36902]: audit 2022-04-22T17:42:42.855072+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/858418860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:43.801 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:43.538Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:43.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[45495]: cluster 2022-04-22T17:42:42.369011+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2022-04-22T17:42:43.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[45495]: audit 2022-04-22T17:42:42.421916+0000 mgr.y (mgr.34107) 624 : audit [DBG] from='client.34869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:43.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[45495]: audit 2022-04-22T17:42:42.855072+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/858418860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:43.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[46934]: cluster 2022-04-22T17:42:42.369011+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2022-04-22T17:42:43.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[46934]: audit 2022-04-22T17:42:42.421916+0000 mgr.y (mgr.34107) 624 : audit [DBG] from='client.34869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:42:43.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:43 smithi115 bash[46934]: audit 2022-04-22T17:42:42.855072+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.115:0/858418860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:42:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:44 smithi121 bash[36902]: cluster 2022-04-22T17:42:43.381184+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2022-04-22T17:42:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:44 smithi121 bash[36902]: audit 2022-04-22T17:42:43.385256+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:44 smithi121 bash[36902]: audit 2022-04-22T17:42:43.800126+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:44 smithi121 bash[36902]: cluster 2022-04-22T17:42:44.111080+0000 mgr.y (mgr.34107) 625 : cluster [DBG] pgmap v1172: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:44.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[45495]: cluster 2022-04-22T17:42:43.381184+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2022-04-22T17:42:44.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[45495]: audit 2022-04-22T17:42:43.385256+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:44.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[45495]: audit 2022-04-22T17:42:43.800126+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:44.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[45495]: cluster 2022-04-22T17:42:44.111080+0000 mgr.y (mgr.34107) 625 : cluster [DBG] pgmap v1172: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:44.801 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[53813]: debug 2022-04-22T17:42:44.413+0000 7f5cf7dad700 -1 osd.2 pg_epoch: 951 pg[33.2( v 951'1 (0'0,951'1] local-lis/les=950/951 n=1 ec=950/950 lis/c=950/950 les/c/f=951/951/0 sis=950) [2,3,6] r=0 lpr=950 crt=951'1 lcod 0'0 mlcod 0'0 active+clean] method hello.bad_reader tried to read object but is not marked RD 2022-04-22T17:42:44.801 INFO:journalctl@ceph.osd.2.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[53813]: debug 2022-04-22T17:42:44.413+0000 7f5cf7dad700 -1 osd.2 pg_epoch: 951 pg[33.2( v 951'2 (0'0,951'2] local-lis/les=950/951 n=1 ec=950/950 lis/c=950/950 les/c/f=951/951/0 sis=950) [2,3,6] r=0 lpr=950 crt=951'1 lcod 951'1 mlcod 951'1 active+clean] method hello.bad_writer tried to update object but is not marked WR 2022-04-22T17:42:44.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[46934]: cluster 2022-04-22T17:42:43.381184+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2022-04-22T17:42:44.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[46934]: audit 2022-04-22T17:42:43.385256+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:44.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[46934]: audit 2022-04-22T17:42:43.800126+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:42:44.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:44 smithi115 bash[46934]: cluster 2022-04-22T17:42:44.111080+0000 mgr.y (mgr.34107) 625 : cluster [DBG] pgmap v1172: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:45.426 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.BadMethods (3038 ms) 2022-04-22T17:42:45.426 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsHello.Filter 2022-04-22T17:42:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:45 smithi121 bash[36902]: audit 2022-04-22T17:42:44.385356+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:45.951 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:45 smithi121 bash[36902]: cluster 2022-04-22T17:42:44.385793+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2022-04-22T17:42:45.951 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:45 smithi121 bash[36902]: audit 2022-04-22T17:42:44.618911+0000 mon.a (mon.0) 1814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:45.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[46934]: audit 2022-04-22T17:42:44.385356+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:45.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[46934]: cluster 2022-04-22T17:42:44.385793+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2022-04-22T17:42:45.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[46934]: audit 2022-04-22T17:42:44.618911+0000 mon.a (mon.0) 1814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:45.953 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[45495]: audit 2022-04-22T17:42:44.385356+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? 172.21.15.115:0/1172238135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:45.953 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[45495]: cluster 2022-04-22T17:42:44.385793+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2022-04-22T17:42:45.953 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:45 smithi115 bash[45495]: audit 2022-04-22T17:42:44.618911+0000 mon.a (mon.0) 1814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:46 smithi121 bash[36902]: cluster 2022-04-22T17:42:45.402512+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2022-04-22T17:42:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:46 smithi121 bash[36902]: cluster 2022-04-22T17:42:46.112013+0000 mgr.y (mgr.34107) 626 : cluster [DBG] pgmap v1175: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:46.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:46 smithi115 bash[45495]: cluster 2022-04-22T17:42:45.402512+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2022-04-22T17:42:46.877 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:46 smithi115 bash[45495]: cluster 2022-04-22T17:42:46.112013+0000 mgr.y (mgr.34107) 626 : cluster [DBG] pgmap v1175: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:46.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:46 smithi115 bash[46934]: cluster 2022-04-22T17:42:45.402512+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2022-04-22T17:42:46.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:46 smithi115 bash[46934]: cluster 2022-04-22T17:42:46.112013+0000 mgr.y (mgr.34107) 626 : cluster [DBG] pgmap v1175: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:47 smithi121 bash[36902]: cluster 2022-04-22T17:42:46.426643+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2022-04-22T17:42:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:47 smithi121 bash[36902]: audit 2022-04-22T17:42:46.428451+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:47.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:47 smithi115 bash[45495]: cluster 2022-04-22T17:42:46.426643+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2022-04-22T17:42:47.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:47 smithi115 bash[45495]: audit 2022-04-22T17:42:46.428451+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:47.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:47 smithi115 bash[46934]: cluster 2022-04-22T17:42:46.426643+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2022-04-22T17:42:47.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:47 smithi115 bash[46934]: audit 2022-04-22T17:42:46.428451+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:48.464 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsHello.Filter (3039 ms) 2022-04-22T17:42:48.465 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 6 tests from ClsHello (20147 ms total) 2022-04-22T17:42:48.465 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:42:48.465 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:42:48.465 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 6 tests from 1 test suite ran. (20147 ms total) 2022-04-22T17:42:48.465 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 6 tests. 2022-04-22T17:42:48.467 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:42:48.467 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:42:48.480 INFO:tasks.workunit:Running workunit cls/test_cls_journal.sh... 2022-04-22T17:42:48.480 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2022-04-22T17:42:48.547 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:42:48.548 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 27 tests from 1 test suite. 2022-04-22T17:42:48.548 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:42:48.548 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 27 tests from TestClsJournal 2022-04-22T17:42:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:48 smithi121 bash[36902]: audit 2022-04-22T17:42:47.432888+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:48 smithi121 bash[36902]: cluster 2022-04-22T17:42:47.433123+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2022-04-22T17:42:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:48 smithi121 bash[36902]: cluster 2022-04-22T17:42:48.112888+0000 mgr.y (mgr.34107) 627 : cluster [DBG] pgmap v1178: 137 pgs: 5 creating+peering, 25 unknown, 107 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:48.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[46934]: audit 2022-04-22T17:42:47.432888+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:48.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[46934]: cluster 2022-04-22T17:42:47.433123+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2022-04-22T17:42:48.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[46934]: cluster 2022-04-22T17:42:48.112888+0000 mgr.y (mgr.34107) 627 : cluster [DBG] pgmap v1178: 137 pgs: 5 creating+peering, 25 unknown, 107 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:48.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[45495]: audit 2022-04-22T17:42:47.432888+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? 172.21.15.115:0/3832055209' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:48.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[45495]: cluster 2022-04-22T17:42:47.433123+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2022-04-22T17:42:48.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:48 smithi115 bash[45495]: cluster 2022-04-22T17:42:48.112888+0000 mgr.y (mgr.34107) 627 : cluster [DBG] pgmap v1178: 137 pgs: 5 creating+peering, 25 unknown, 107 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:49 smithi121 bash[36902]: cluster 2022-04-22T17:42:48.448536+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2022-04-22T17:42:49.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:49 smithi115 bash[45495]: cluster 2022-04-22T17:42:48.448536+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2022-04-22T17:42:49.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:49 smithi115 bash[46934]: cluster 2022-04-22T17:42:48.448536+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2022-04-22T17:42:50.489 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.Create 2022-04-22T17:42:50.505 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.Create (16 ms) 2022-04-22T17:42:50.505 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.MinimumSet 2022-04-22T17:42:50.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.MinimumSet (10 ms) 2022-04-22T17:42:50.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.MinimumSetStale 2022-04-22T17:42:50.521 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.MinimumSetStale (7 ms) 2022-04-22T17:42:50.521 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.MinimumSetOrderConstraint 2022-04-22T17:42:50.526 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.MinimumSetOrderConstraint (5 ms) 2022-04-22T17:42:50.527 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ActiveSet 2022-04-22T17:42:50.531 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ActiveSet (5 ms) 2022-04-22T17:42:50.531 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ActiveSetStale 2022-04-22T17:42:50.537 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ActiveSetStale (6 ms) 2022-04-22T17:42:50.537 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.CreateDuplicate 2022-04-22T17:42:50.541 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.CreateDuplicate (4 ms) 2022-04-22T17:42:50.541 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.GetClient 2022-04-22T17:42:50.547 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.GetClient (5 ms) 2022-04-22T17:42:50.547 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientRegister 2022-04-22T17:42:50.551 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientRegister (5 ms) 2022-04-22T17:42:50.551 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientRegisterDuplicate 2022-04-22T17:42:50.558 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientRegisterDuplicate (7 ms) 2022-04-22T17:42:50.558 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientUpdateData 2022-04-22T17:42:50.566 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientUpdateData (8 ms) 2022-04-22T17:42:50.567 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientUpdateState 2022-04-22T17:42:50.575 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientUpdateState (9 ms) 2022-04-22T17:42:50.576 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientUnregister 2022-04-22T17:42:50.583 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientUnregister (7 ms) 2022-04-22T17:42:50.583 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientUnregisterDNE 2022-04-22T17:42:50.590 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientUnregisterDNE (8 ms) 2022-04-22T17:42:50.591 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientUnregisterPruneTags 2022-04-22T17:42:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:50 smithi121 bash[36902]: cluster 2022-04-22T17:42:49.461009+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2022-04-22T17:42:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:50 smithi121 bash[36902]: audit 2022-04-22T17:42:49.463211+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.115:0/3484790827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:50 smithi121 bash[36902]: audit 2022-04-22T17:42:49.464674+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:50 smithi121 bash[36902]: audit 2022-04-22T17:42:49.622596+0000 mon.a (mon.0) 1823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:50 smithi121 bash[36902]: cluster 2022-04-22T17:42:50.113758+0000 mgr.y (mgr.34107) 628 : cluster [DBG] pgmap v1181: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[45495]: cluster 2022-04-22T17:42:49.461009+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2022-04-22T17:42:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[45495]: audit 2022-04-22T17:42:49.463211+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.115:0/3484790827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[45495]: audit 2022-04-22T17:42:49.464674+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[45495]: audit 2022-04-22T17:42:49.622596+0000 mon.a (mon.0) 1823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[45495]: cluster 2022-04-22T17:42:50.113758+0000 mgr.y (mgr.34107) 628 : cluster [DBG] pgmap v1181: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[46934]: cluster 2022-04-22T17:42:49.461009+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[46934]: audit 2022-04-22T17:42:49.463211+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.115:0/3484790827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[46934]: audit 2022-04-22T17:42:49.464674+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[46934]: audit 2022-04-22T17:42:49.622596+0000 mon.a (mon.0) 1823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:50.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:50 smithi115 bash[46934]: cluster 2022-04-22T17:42:50.113758+0000 mgr.y (mgr.34107) 628 : cluster [DBG] pgmap v1181: 137 pgs: 3 creating+peering, 29 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:50.802 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientUnregisterPruneTags (210 ms) 2022-04-22T17:42:50.802 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientCommit 2022-04-22T17:42:50.808 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientCommit (8 ms) 2022-04-22T17:42:50.808 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientCommitInvalid 2022-04-22T17:42:50.814 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientCommitInvalid (6 ms) 2022-04-22T17:42:50.814 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientCommitDNE 2022-04-22T17:42:50.816 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientCommitDNE (1 ms) 2022-04-22T17:42:50.816 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.ClientList 2022-04-22T17:42:50.877 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.ClientList (62 ms) 2022-04-22T17:42:50.888 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.GetNextTagTid 2022-04-22T17:42:50.888 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.GetNextTagTid (10 ms) 2022-04-22T17:42:50.888 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.TagCreate 2022-04-22T17:42:50.907 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.TagCreate (19 ms) 2022-04-22T17:42:50.907 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.TagCreatePrunesTags 2022-04-22T17:42:50.921 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.TagCreatePrunesTags (14 ms) 2022-04-22T17:42:50.921 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.TagList 2022-04-22T17:42:51.126 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.TagList (206 ms) 2022-04-22T17:42:51.127 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.GuardAppend 2022-04-22T17:42:51.130 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.GuardAppend (4 ms) 2022-04-22T17:42:51.130 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.GuardAppendDNE 2022-04-22T17:42:51.132 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.GuardAppendDNE (2 ms) 2022-04-22T17:42:51.132 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.GuardAppendOverflow 2022-04-22T17:42:51.136 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.GuardAppendOverflow (4 ms) 2022-04-22T17:42:51.136 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsJournal.Append 2022-04-22T17:42:51.150 INFO:tasks.workunit.client.0.smithi115.stdout:padding is not enabled 2022-04-22T17:42:51.151 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsJournal.Append (14 ms) 2022-04-22T17:42:51.495 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 27 tests from TestClsJournal (662 ms total) 2022-04-22T17:42:51.495 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:42:51.495 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:42:51.495 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 27 tests from 1 test suite ran. (2947 ms total) 2022-04-22T17:42:51.496 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 27 tests. 2022-04-22T17:42:51.499 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:42:51.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:42:51.509 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2022-04-22T17:42:51.509 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2022-04-22T17:42:51.532 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:42:51.579 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 11 tests from 1 test suite. 2022-04-22T17:42:51.579 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:42:51.579 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from ClsLock 2022-04-22T17:42:51.579 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestMultiLocking 2022-04-22T17:42:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:51 smithi121 bash[36902]: audit 2022-04-22T17:42:50.473972+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:51 smithi121 bash[36902]: cluster 2022-04-22T17:42:50.474445+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2022-04-22T17:42:51.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:51.646Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:51.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:51 smithi115 bash[45495]: audit 2022-04-22T17:42:50.473972+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:51.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:51 smithi115 bash[45495]: cluster 2022-04-22T17:42:50.474445+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2022-04-22T17:42:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:51 smithi115 bash[46934]: audit 2022-04-22T17:42:50.473972+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63416-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:51 smithi115 bash[46934]: cluster 2022-04-22T17:42:50.474445+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2022-04-22T17:42:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:52 smithi121 bash[36902]: cluster 2022-04-22T17:42:51.475885+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2022-04-22T17:42:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:52 smithi121 bash[36902]: cluster 2022-04-22T17:42:52.114698+0000 mgr.y (mgr.34107) 629 : cluster [DBG] pgmap v1184: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:52.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:52 smithi115 bash[45495]: cluster 2022-04-22T17:42:51.475885+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2022-04-22T17:42:52.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:52 smithi115 bash[45495]: cluster 2022-04-22T17:42:52.114698+0000 mgr.y (mgr.34107) 629 : cluster [DBG] pgmap v1184: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:52.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:52 smithi115 bash[46934]: cluster 2022-04-22T17:42:51.475885+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2022-04-22T17:42:52.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:52 smithi115 bash[46934]: cluster 2022-04-22T17:42:52.114698+0000 mgr.y (mgr.34107) 629 : cluster [DBG] pgmap v1184: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:53.563 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:53.563 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: exclusive 2022-04-22T17:42:53.563 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:53.563 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:53.563 INFO:tasks.workunit.client.0.smithi115.stdout: client.44836 expiration=2022-04-22T17:44:53.544602+0000 addr=v1:172.21.15.115:0/2679651320 cookie= 2022-04-22T17:42:53.568 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: shared 2022-04-22T17:42:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: client.34905 expiration=0.000000 addr=v1:172.21.15.115:0/1298722199 cookie= 2022-04-22T17:42:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: client.44836 expiration=2022-04-22T17:44:53.554669+0000 addr=v1:172.21.15.115:0/2679651320 cookie= 2022-04-22T17:42:53.571 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:53.571 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: shared 2022-04-22T17:42:53.571 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:53.571 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:53.571 INFO:tasks.workunit.client.0.smithi115.stdout: client.34905 expiration=0.000000 addr=v1:172.21.15.115:0/1298722199 cookie= 2022-04-22T17:42:53.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:42:53.534Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[45495]: cluster 2022-04-22T17:42:52.494917+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[45495]: audit 2022-04-22T17:42:52.499389+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.115:0/2679651320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[45495]: audit 2022-04-22T17:42:52.500586+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[46934]: cluster 2022-04-22T17:42:52.494917+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[46934]: audit 2022-04-22T17:42:52.499389+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.115:0/2679651320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:53.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:53 smithi115 bash[46934]: audit 2022-04-22T17:42:52.500586+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:53 smithi121 bash[36902]: cluster 2022-04-22T17:42:52.494917+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2022-04-22T17:42:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:53 smithi121 bash[36902]: audit 2022-04-22T17:42:52.499389+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.115:0/2679651320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:53 smithi121 bash[36902]: audit 2022-04-22T17:42:52.500586+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:54.539 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestMultiLocking (3007 ms) 2022-04-22T17:42:54.539 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestMeta 2022-04-22T17:42:54.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[45495]: audit 2022-04-22T17:42:53.501475+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:54.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[45495]: cluster 2022-04-22T17:42:53.501733+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2022-04-22T17:42:54.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[45495]: cluster 2022-04-22T17:42:54.115553+0000 mgr.y (mgr.34107) 630 : cluster [DBG] pgmap v1187: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:54.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[46934]: audit 2022-04-22T17:42:53.501475+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:54.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[46934]: cluster 2022-04-22T17:42:53.501733+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2022-04-22T17:42:54.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:54 smithi115 bash[46934]: cluster 2022-04-22T17:42:54.115553+0000 mgr.y (mgr.34107) 630 : cluster [DBG] pgmap v1187: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:54 smithi121 bash[36902]: audit 2022-04-22T17:42:53.501475+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:54 smithi121 bash[36902]: cluster 2022-04-22T17:42:53.501733+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2022-04-22T17:42:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:54 smithi121 bash[36902]: cluster 2022-04-22T17:42:54.115553+0000 mgr.y (mgr.34107) 630 : cluster [DBG] pgmap v1187: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:55.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:55 smithi115 bash[45495]: cluster 2022-04-22T17:42:54.511455+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2022-04-22T17:42:55.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:55 smithi115 bash[45495]: audit 2022-04-22T17:42:54.992572+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:55.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:55 smithi115 bash[46934]: cluster 2022-04-22T17:42:54.511455+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2022-04-22T17:42:55.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:55 smithi115 bash[46934]: audit 2022-04-22T17:42:54.992572+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:55 smithi121 bash[36902]: cluster 2022-04-22T17:42:54.511455+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2022-04-22T17:42:56.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:55 smithi121 bash[36902]: audit 2022-04-22T17:42:54.992572+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:42:56.594 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:56.594 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: shared 2022-04-22T17:42:56.594 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:56.594 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:56.594 INFO:tasks.workunit.client.0.smithi115.stdout: client.34917 expiration=0.000000 addr=v1:172.21.15.115:0/1577396342 cookie= 2022-04-22T17:42:56.597 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:56.597 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: exclusive 2022-04-22T17:42:56.597 INFO:tasks.workunit.client.0.smithi115.stdout: tag: new_tag 2022-04-22T17:42:56.597 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:56.597 INFO:tasks.workunit.client.0.smithi115.stdout: client.34911 expiration=0.000000 addr=v1:172.21.15.115:0/1652867428 cookie= 2022-04-22T17:42:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:56 smithi121 bash[36902]: cluster 2022-04-22T17:42:55.534646+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2022-04-22T17:42:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:56 smithi121 bash[36902]: audit 2022-04-22T17:42:55.537076+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:56 smithi121 bash[36902]: cluster 2022-04-22T17:42:56.116517+0000 mgr.y (mgr.34107) 631 : cluster [DBG] pgmap v1190: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:57.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[45495]: cluster 2022-04-22T17:42:55.534646+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2022-04-22T17:42:57.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[45495]: audit 2022-04-22T17:42:55.537076+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:57.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[45495]: cluster 2022-04-22T17:42:56.116517+0000 mgr.y (mgr.34107) 631 : cluster [DBG] pgmap v1190: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[46934]: cluster 2022-04-22T17:42:55.534646+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2022-04-22T17:42:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[46934]: audit 2022-04-22T17:42:55.537076+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:42:57.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:56 smithi115 bash[46934]: cluster 2022-04-22T17:42:56.116517+0000 mgr.y (mgr.34107) 631 : cluster [DBG] pgmap v1190: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:57.579 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestMeta (3028 ms) 2022-04-22T17:42:57.580 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestCookie 2022-04-22T17:42:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:57 smithi121 bash[36902]: audit 2022-04-22T17:42:56.544484+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:57 smithi121 bash[36902]: cluster 2022-04-22T17:42:56.544686+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2022-04-22T17:42:58.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:57 smithi115 bash[45495]: audit 2022-04-22T17:42:56.544484+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:58.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:57 smithi115 bash[45495]: cluster 2022-04-22T17:42:56.544686+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2022-04-22T17:42:58.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:57 smithi115 bash[46934]: audit 2022-04-22T17:42:56.544484+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.115:0/1652867428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:42:58.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:57 smithi115 bash[46934]: cluster 2022-04-22T17:42:56.544686+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2022-04-22T17:42:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:58 smithi121 bash[36902]: cluster 2022-04-22T17:42:57.551341+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2022-04-22T17:42:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:58 smithi121 bash[36902]: cluster 2022-04-22T17:42:58.117381+0000 mgr.y (mgr.34107) 632 : cluster [DBG] pgmap v1193: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:59.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:58 smithi115 bash[45495]: cluster 2022-04-22T17:42:57.551341+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2022-04-22T17:42:59.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:58 smithi115 bash[45495]: cluster 2022-04-22T17:42:58.117381+0000 mgr.y (mgr.34107) 632 : cluster [DBG] pgmap v1193: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:59.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:58 smithi115 bash[46934]: cluster 2022-04-22T17:42:57.551341+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2022-04-22T17:42:59.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:58 smithi115 bash[46934]: cluster 2022-04-22T17:42:58.117381+0000 mgr.y (mgr.34107) 632 : cluster [DBG] pgmap v1193: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:42:59.620 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:59.640 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: exclusive 2022-04-22T17:42:59.640 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:59.640 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout:lock: mylock 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout: lock_type: shared 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout: tag: 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout: lockers: 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout: client.34923 expiration=0.000000 addr=v1:172.21.15.115:0/531439955 cookie= 2022-04-22T17:42:59.641 INFO:tasks.workunit.client.0.smithi115.stdout: client.34923 expiration=0.000000 addr=v1:172.21.15.115:0/531439955 cookie=new cookie 2022-04-22T17:43:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:59 smithi121 bash[36902]: cluster 2022-04-22T17:42:58.562289+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2022-04-22T17:43:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:42:59 smithi121 bash[36902]: audit 2022-04-22T17:42:58.563502+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:00.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:59 smithi115 bash[45495]: cluster 2022-04-22T17:42:58.562289+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2022-04-22T17:43:00.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:42:59 smithi115 bash[45495]: audit 2022-04-22T17:42:58.563502+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:00.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:59 smithi115 bash[46934]: cluster 2022-04-22T17:42:58.562289+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2022-04-22T17:43:00.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:42:59 smithi115 bash[46934]: audit 2022-04-22T17:42:58.563502+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:00.599 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestCookie (3032 ms) 2022-04-22T17:43:00.599 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestMultipleLocks 2022-04-22T17:43:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:00 smithi121 bash[36902]: audit 2022-04-22T17:42:59.576035+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:00 smithi121 bash[36902]: cluster 2022-04-22T17:42:59.576219+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2022-04-22T17:43:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:00 smithi121 bash[36902]: audit 2022-04-22T17:42:59.630583+0000 mon.a (mon.0) 1842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:00 smithi121 bash[36902]: cluster 2022-04-22T17:43:00.118256+0000 mgr.y (mgr.34107) 633 : cluster [DBG] pgmap v1196: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:01.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[46934]: audit 2022-04-22T17:42:59.576035+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:01.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[46934]: cluster 2022-04-22T17:42:59.576219+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2022-04-22T17:43:01.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[46934]: audit 2022-04-22T17:42:59.630583+0000 mon.a (mon.0) 1842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:01.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[46934]: cluster 2022-04-22T17:43:00.118256+0000 mgr.y (mgr.34107) 633 : cluster [DBG] pgmap v1196: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[45495]: audit 2022-04-22T17:42:59.576035+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? 172.21.15.115:0/531439955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[45495]: cluster 2022-04-22T17:42:59.576219+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2022-04-22T17:43:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[45495]: audit 2022-04-22T17:42:59.630583+0000 mon.a (mon.0) 1842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:00 smithi115 bash[45495]: cluster 2022-04-22T17:43:00.118256+0000 mgr.y (mgr.34107) 633 : cluster [DBG] pgmap v1196: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:01 smithi121 bash[36902]: cluster 2022-04-22T17:43:00.584396+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2022-04-22T17:43:02.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:01 smithi115 bash[45495]: cluster 2022-04-22T17:43:00.584396+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2022-04-22T17:43:02.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:01.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:02.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:01 smithi115 bash[46934]: cluster 2022-04-22T17:43:00.584396+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2022-04-22T17:43:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:02 smithi121 bash[36902]: cluster 2022-04-22T17:43:01.593559+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2022-04-22T17:43:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:02 smithi121 bash[36902]: audit 2022-04-22T17:43:01.595810+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.115:0/2487654235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:02 smithi121 bash[36902]: audit 2022-04-22T17:43:01.597129+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:02 smithi121 bash[36902]: cluster 2022-04-22T17:43:02.119348+0000 mgr.y (mgr.34107) 634 : cluster [DBG] pgmap v1199: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:03.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[45495]: cluster 2022-04-22T17:43:01.593559+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2022-04-22T17:43:03.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[45495]: audit 2022-04-22T17:43:01.595810+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.115:0/2487654235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[45495]: audit 2022-04-22T17:43:01.597129+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[45495]: cluster 2022-04-22T17:43:02.119348+0000 mgr.y (mgr.34107) 634 : cluster [DBG] pgmap v1199: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:03.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[46934]: cluster 2022-04-22T17:43:01.593559+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2022-04-22T17:43:03.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[46934]: audit 2022-04-22T17:43:01.595810+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.115:0/2487654235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[46934]: audit 2022-04-22T17:43:01.597129+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:03.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:02 smithi115 bash[46934]: cluster 2022-04-22T17:43:02.119348+0000 mgr.y (mgr.34107) 634 : cluster [DBG] pgmap v1199: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:03.636 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestMultipleLocks (3034 ms) 2022-04-22T17:43:03.636 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestLockDuration 2022-04-22T17:43:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:03 smithi121 bash[36902]: audit 2022-04-22T17:43:02.603531+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:03 smithi121 bash[36902]: cluster 2022-04-22T17:43:02.603858+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2022-04-22T17:43:04.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:03.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:04.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:03 smithi115 bash[45495]: audit 2022-04-22T17:43:02.603531+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:04.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:03 smithi115 bash[45495]: cluster 2022-04-22T17:43:02.603858+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2022-04-22T17:43:04.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:03 smithi115 bash[46934]: audit 2022-04-22T17:43:02.603531+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:04.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:03 smithi115 bash[46934]: cluster 2022-04-22T17:43:02.603858+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2022-04-22T17:43:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:04 smithi121 bash[36902]: cluster 2022-04-22T17:43:03.617126+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2022-04-22T17:43:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:04 smithi121 bash[36902]: cluster 2022-04-22T17:43:04.120134+0000 mgr.y (mgr.34107) 635 : cluster [DBG] pgmap v1202: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:05.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:04 smithi115 bash[45495]: cluster 2022-04-22T17:43:03.617126+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2022-04-22T17:43:05.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:04 smithi115 bash[45495]: cluster 2022-04-22T17:43:04.120134+0000 mgr.y (mgr.34107) 635 : cluster [DBG] pgmap v1202: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:05.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:04 smithi115 bash[46934]: cluster 2022-04-22T17:43:03.617126+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2022-04-22T17:43:05.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:04 smithi115 bash[46934]: cluster 2022-04-22T17:43:04.120134+0000 mgr.y (mgr.34107) 635 : cluster [DBG] pgmap v1202: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:05 smithi121 bash[36902]: cluster 2022-04-22T17:43:04.638750+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2022-04-22T17:43:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:05 smithi121 bash[36902]: audit 2022-04-22T17:43:04.639632+0000 mon.a (mon.0) 1850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:05 smithi121 bash[36902]: audit 2022-04-22T17:43:04.640580+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:06.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[45495]: cluster 2022-04-22T17:43:04.638750+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2022-04-22T17:43:06.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[45495]: audit 2022-04-22T17:43:04.639632+0000 mon.a (mon.0) 1850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:06.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[45495]: audit 2022-04-22T17:43:04.640580+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:06.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[46934]: cluster 2022-04-22T17:43:04.638750+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2022-04-22T17:43:06.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[46934]: audit 2022-04-22T17:43:04.639632+0000 mon.a (mon.0) 1850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:06.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:05 smithi115 bash[46934]: audit 2022-04-22T17:43:04.640580+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:06 smithi121 bash[36902]: audit 2022-04-22T17:43:05.641479+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:06 smithi121 bash[36902]: cluster 2022-04-22T17:43:05.641746+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2022-04-22T17:43:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:06 smithi121 bash[36902]: cluster 2022-04-22T17:43:06.120973+0000 mgr.y (mgr.34107) 636 : cluster [DBG] pgmap v1205: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:07.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[45495]: audit 2022-04-22T17:43:05.641479+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:07.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[45495]: cluster 2022-04-22T17:43:05.641746+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2022-04-22T17:43:07.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[45495]: cluster 2022-04-22T17:43:06.120973+0000 mgr.y (mgr.34107) 636 : cluster [DBG] pgmap v1205: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:07.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[46934]: audit 2022-04-22T17:43:05.641479+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.115:0/2452916719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:07.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[46934]: cluster 2022-04-22T17:43:05.641746+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2022-04-22T17:43:07.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:06 smithi115 bash[46934]: cluster 2022-04-22T17:43:06.120973+0000 mgr.y (mgr.34107) 636 : cluster [DBG] pgmap v1205: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:08.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:07 smithi121 bash[36902]: cluster 2022-04-22T17:43:06.657283+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2022-04-22T17:43:08.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:07 smithi115 bash[45495]: cluster 2022-04-22T17:43:06.657283+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2022-04-22T17:43:08.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:07 smithi115 bash[46934]: cluster 2022-04-22T17:43:06.657283+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2022-04-22T17:43:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:08 smithi121 bash[36902]: cluster 2022-04-22T17:43:08.122135+0000 mgr.y (mgr.34107) 637 : cluster [DBG] pgmap v1207: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:09.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:08 smithi115 bash[45495]: cluster 2022-04-22T17:43:08.122135+0000 mgr.y (mgr.34107) 637 : cluster [DBG] pgmap v1207: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:08 smithi115 bash[46934]: cluster 2022-04-22T17:43:08.122135+0000 mgr.y (mgr.34107) 637 : cluster [DBG] pgmap v1207: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:09 smithi121 bash[36902]: audit 2022-04-22T17:43:09.639979+0000 mon.a (mon.0) 1855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:10.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:09 smithi115 bash[45495]: audit 2022-04-22T17:43:09.639979+0000 mon.a (mon.0) 1855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:10.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:09 smithi115 bash[46934]: audit 2022-04-22T17:43:09.639979+0000 mon.a (mon.0) 1855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:11 smithi121 bash[36902]: cluster 2022-04-22T17:43:10.123222+0000 mgr.y (mgr.34107) 638 : cluster [DBG] pgmap v1208: 137 pgs: 1 creating+peering, 136 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:11.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:11 smithi115 bash[45495]: cluster 2022-04-22T17:43:10.123222+0000 mgr.y (mgr.34107) 638 : cluster [DBG] pgmap v1208: 137 pgs: 1 creating+peering, 136 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:11.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:11 smithi115 bash[46934]: cluster 2022-04-22T17:43:10.123222+0000 mgr.y (mgr.34107) 638 : cluster [DBG] pgmap v1208: 137 pgs: 1 creating+peering, 136 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:11.696 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestLockDuration (8062 ms) 2022-04-22T17:43:11.696 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestAssertLocked 2022-04-22T17:43:12.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:11.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:12 smithi121 bash[36902]: cluster 2022-04-22T17:43:11.678232+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2022-04-22T17:43:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:12 smithi121 bash[36902]: cluster 2022-04-22T17:43:12.124182+0000 mgr.y (mgr.34107) 639 : cluster [DBG] pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:13.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:12 smithi115 bash[45495]: cluster 2022-04-22T17:43:11.678232+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2022-04-22T17:43:13.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:12 smithi115 bash[45495]: cluster 2022-04-22T17:43:12.124182+0000 mgr.y (mgr.34107) 639 : cluster [DBG] pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:12 smithi115 bash[46934]: cluster 2022-04-22T17:43:11.678232+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2022-04-22T17:43:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:12 smithi115 bash[46934]: cluster 2022-04-22T17:43:12.124182+0000 mgr.y (mgr.34107) 639 : cluster [DBG] pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:13.232 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:43:13.571 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:43:13.571 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 6m ago 20m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (19m) 6m ago 19m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (8m) 6m ago 21m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (7m) 6m ago 21m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (15m) 6m ago 25m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (16m) 6m ago 27m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (15m) 6m ago 27m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (14m) 6m ago 25m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (14m) 6m ago 25m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:43:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 6m ago 20m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (7m) 6m ago 20m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (13m) 6m ago 24m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (12m) 6m ago 24m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 6m ago 23m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (11m) 6m ago 23m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:43:13.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (10m) 6m ago 23m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:43:13.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (10m) 6m ago 22m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:43:13.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (9m) 6m ago 22m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:43:13.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (8m) 6m ago 21m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:43:13.574 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (6m) 6m ago 20m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:43:13.993 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:43:14.025 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:43:14.025 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:43:14.026 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:14.026 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:43:14.026 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:43:14.026 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:43:14.027 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:43:14.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: cluster 2022-04-22T17:43:12.686795+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2022-04-22T17:43:14.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: audit 2022-04-22T17:43:12.688290+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.115:0/367510789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: audit 2022-04-22T17:43:12.689136+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: audit 2022-04-22T17:43:13.201144+0000 mgr.y (mgr.34107) 640 : audit [DBG] from='client.44884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:14.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: audit 2022-04-22T17:43:13.388366+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:43:14.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:13 smithi121 bash[36902]: audit 2022-04-22T17:43:13.388627+0000 mon.a (mon.0) 1860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:43:14.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:13.540Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: cluster 2022-04-22T17:43:12.686795+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2022-04-22T17:43:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: audit 2022-04-22T17:43:12.688290+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.115:0/367510789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: audit 2022-04-22T17:43:12.689136+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: audit 2022-04-22T17:43:13.201144+0000 mgr.y (mgr.34107) 640 : audit [DBG] from='client.44884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: audit 2022-04-22T17:43:13.388366+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[46934]: audit 2022-04-22T17:43:13.388627+0000 mon.a (mon.0) 1860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: cluster 2022-04-22T17:43:12.686795+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: audit 2022-04-22T17:43:12.688290+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.115:0/367510789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: audit 2022-04-22T17:43:12.689136+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: audit 2022-04-22T17:43:13.201144+0000 mgr.y (mgr.34107) 640 : audit [DBG] from='client.44884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: audit 2022-04-22T17:43:13.388366+0000 mon.a (mon.0) 1859 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:43:14.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:13 smithi115 bash[45495]: audit 2022-04-22T17:43:13.388627+0000 mon.a (mon.0) 1860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:43:14.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestAssertLocked (3027 ms) 2022-04-22T17:43:14.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestSetCookie 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: audit 2022-04-22T17:43:13.539766+0000 mgr.y (mgr.34107) 641 : audit [DBG] from='client.44890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: audit 2022-04-22T17:43:13.686852+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: cluster 2022-04-22T17:43:13.687075+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: audit 2022-04-22T17:43:13.980832+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.115:0/2979492825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: cluster 2022-04-22T17:43:14.125261+0000 mgr.y (mgr.34107) 642 : cluster [DBG] pgmap v1213: 137 pgs: 9 creating+peering, 20 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:14 smithi121 bash[36902]: audit 2022-04-22T17:43:14.643027+0000 mon.a (mon.0) 1863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: audit 2022-04-22T17:43:13.539766+0000 mgr.y (mgr.34107) 641 : audit [DBG] from='client.44890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: audit 2022-04-22T17:43:13.686852+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: cluster 2022-04-22T17:43:13.687075+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: audit 2022-04-22T17:43:13.980832+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.115:0/2979492825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: cluster 2022-04-22T17:43:14.125261+0000 mgr.y (mgr.34107) 642 : cluster [DBG] pgmap v1213: 137 pgs: 9 creating+peering, 20 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[46934]: audit 2022-04-22T17:43:14.643027+0000 mon.a (mon.0) 1863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: audit 2022-04-22T17:43:13.539766+0000 mgr.y (mgr.34107) 641 : audit [DBG] from='client.44890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: audit 2022-04-22T17:43:13.686852+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: cluster 2022-04-22T17:43:13.687075+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: audit 2022-04-22T17:43:13.980832+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.115:0/2979492825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: cluster 2022-04-22T17:43:14.125261+0000 mgr.y (mgr.34107) 642 : cluster [DBG] pgmap v1213: 137 pgs: 9 creating+peering, 20 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:14 smithi115 bash[45495]: audit 2022-04-22T17:43:14.643027+0000 mon.a (mon.0) 1863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:15 smithi121 bash[36902]: cluster 2022-04-22T17:43:14.705312+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2022-04-22T17:43:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:15 smithi115 bash[45495]: cluster 2022-04-22T17:43:14.705312+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2022-04-22T17:43:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:15 smithi115 bash[46934]: cluster 2022-04-22T17:43:14.705312+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2022-04-22T17:43:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:16 smithi121 bash[36902]: cluster 2022-04-22T17:43:15.723512+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2022-04-22T17:43:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:16 smithi121 bash[36902]: audit 2022-04-22T17:43:15.728084+0000 mon.c (mon.1) 48 : audit [INF] from='client.? 172.21.15.115:0/26799267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:16 smithi121 bash[36902]: audit 2022-04-22T17:43:15.729190+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:16 smithi121 bash[36902]: cluster 2022-04-22T17:43:16.126284+0000 mgr.y (mgr.34107) 643 : cluster [DBG] pgmap v1216: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:17.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[45495]: cluster 2022-04-22T17:43:15.723512+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2022-04-22T17:43:17.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[45495]: audit 2022-04-22T17:43:15.728084+0000 mon.c (mon.1) 48 : audit [INF] from='client.? 172.21.15.115:0/26799267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[45495]: audit 2022-04-22T17:43:15.729190+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[45495]: cluster 2022-04-22T17:43:16.126284+0000 mgr.y (mgr.34107) 643 : cluster [DBG] pgmap v1216: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:17.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[46934]: cluster 2022-04-22T17:43:15.723512+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2022-04-22T17:43:17.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[46934]: audit 2022-04-22T17:43:15.728084+0000 mon.c (mon.1) 48 : audit [INF] from='client.? 172.21.15.115:0/26799267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[46934]: audit 2022-04-22T17:43:15.729190+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:17.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:16 smithi115 bash[46934]: cluster 2022-04-22T17:43:16.126284+0000 mgr.y (mgr.34107) 643 : cluster [DBG] pgmap v1216: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:17.760 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestSetCookie (3038 ms) 2022-04-22T17:43:17.760 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestRenew 2022-04-22T17:43:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:17 smithi121 bash[36902]: audit 2022-04-22T17:43:16.730788+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:17 smithi121 bash[36902]: cluster 2022-04-22T17:43:16.731149+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2022-04-22T17:43:18.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:17 smithi115 bash[45495]: audit 2022-04-22T17:43:16.730788+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:18.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:17 smithi115 bash[45495]: cluster 2022-04-22T17:43:16.731149+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2022-04-22T17:43:18.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:17 smithi115 bash[46934]: audit 2022-04-22T17:43:16.730788+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:18.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:17 smithi115 bash[46934]: cluster 2022-04-22T17:43:16.731149+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2022-04-22T17:43:19.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:18 smithi115 bash[45495]: cluster 2022-04-22T17:43:17.741823+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2022-04-22T17:43:19.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:18 smithi115 bash[45495]: cluster 2022-04-22T17:43:18.127187+0000 mgr.y (mgr.34107) 644 : cluster [DBG] pgmap v1219: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:18 smithi115 bash[46934]: cluster 2022-04-22T17:43:17.741823+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2022-04-22T17:43:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:18 smithi115 bash[46934]: cluster 2022-04-22T17:43:18.127187+0000 mgr.y (mgr.34107) 644 : cluster [DBG] pgmap v1219: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:18 smithi121 bash[36902]: cluster 2022-04-22T17:43:17.741823+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2022-04-22T17:43:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:18 smithi121 bash[36902]: cluster 2022-04-22T17:43:18.127187+0000 mgr.y (mgr.34107) 644 : cluster [DBG] pgmap v1219: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:20.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[45495]: cluster 2022-04-22T17:43:18.762158+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[45495]: audit 2022-04-22T17:43:18.765450+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.115:0/3439653154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[45495]: audit 2022-04-22T17:43:18.766130+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[45495]: audit 2022-04-22T17:43:19.646294+0000 mon.a (mon.0) 1872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[46934]: cluster 2022-04-22T17:43:18.762158+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[46934]: audit 2022-04-22T17:43:18.765450+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.115:0/3439653154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[46934]: audit 2022-04-22T17:43:18.766130+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:19 smithi115 bash[46934]: audit 2022-04-22T17:43:19.646294+0000 mon.a (mon.0) 1872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:19 smithi121 bash[36902]: cluster 2022-04-22T17:43:18.762158+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2022-04-22T17:43:20.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:19 smithi121 bash[36902]: audit 2022-04-22T17:43:18.765450+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.115:0/3439653154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:19 smithi121 bash[36902]: audit 2022-04-22T17:43:18.766130+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:20.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:19 smithi121 bash[36902]: audit 2022-04-22T17:43:19.646294+0000 mon.a (mon.0) 1872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[45495]: audit 2022-04-22T17:43:19.769229+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:21.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[45495]: cluster 2022-04-22T17:43:19.769412+0000 mon.a (mon.0) 1874 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2022-04-22T17:43:21.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[45495]: cluster 2022-04-22T17:43:20.128061+0000 mgr.y (mgr.34107) 645 : cluster [DBG] pgmap v1222: 137 pgs: 5 creating+peering, 22 unknown, 110 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:21.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[46934]: audit 2022-04-22T17:43:19.769229+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:21.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[46934]: cluster 2022-04-22T17:43:19.769412+0000 mon.a (mon.0) 1874 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2022-04-22T17:43:21.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:20 smithi115 bash[46934]: cluster 2022-04-22T17:43:20.128061+0000 mgr.y (mgr.34107) 645 : cluster [DBG] pgmap v1222: 137 pgs: 5 creating+peering, 22 unknown, 110 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:20 smithi121 bash[36902]: audit 2022-04-22T17:43:19.769229+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:20 smithi121 bash[36902]: cluster 2022-04-22T17:43:19.769412+0000 mon.a (mon.0) 1874 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2022-04-22T17:43:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:20 smithi121 bash[36902]: cluster 2022-04-22T17:43:20.128061+0000 mgr.y (mgr.34107) 645 : cluster [DBG] pgmap v1222: 137 pgs: 5 creating+peering, 22 unknown, 110 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:21 smithi115 bash[45495]: cluster 2022-04-22T17:43:20.779388+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2022-04-22T17:43:22.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:21.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:22.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:21 smithi115 bash[46934]: cluster 2022-04-22T17:43:20.779388+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2022-04-22T17:43:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:21 smithi121 bash[36902]: cluster 2022-04-22T17:43:20.779388+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2022-04-22T17:43:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:22 smithi121 bash[36902]: cluster 2022-04-22T17:43:22.129127+0000 mgr.y (mgr.34107) 646 : cluster [DBG] pgmap v1224: 137 pgs: 5 creating+peering, 4 unknown, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:23.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:22 smithi115 bash[45495]: cluster 2022-04-22T17:43:22.129127+0000 mgr.y (mgr.34107) 646 : cluster [DBG] pgmap v1224: 137 pgs: 5 creating+peering, 4 unknown, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:23.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:22 smithi115 bash[46934]: cluster 2022-04-22T17:43:22.129127+0000 mgr.y (mgr.34107) 646 : cluster [DBG] pgmap v1224: 137 pgs: 5 creating+peering, 4 unknown, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:23.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:25 smithi121 bash[36902]: cluster 2022-04-22T17:43:24.130134+0000 mgr.y (mgr.34107) 647 : cluster [DBG] pgmap v1225: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:25 smithi121 bash[36902]: audit 2022-04-22T17:43:24.649563+0000 mon.a (mon.0) 1876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:25 smithi115 bash[45495]: cluster 2022-04-22T17:43:24.130134+0000 mgr.y (mgr.34107) 647 : cluster [DBG] pgmap v1225: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:25.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:25 smithi115 bash[45495]: audit 2022-04-22T17:43:24.649563+0000 mon.a (mon.0) 1876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:25 smithi115 bash[46934]: cluster 2022-04-22T17:43:24.130134+0000 mgr.y (mgr.34107) 647 : cluster [DBG] pgmap v1225: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:25 smithi115 bash[46934]: audit 2022-04-22T17:43:24.649563+0000 mon.a (mon.0) 1876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:27 smithi121 bash[36902]: cluster 2022-04-22T17:43:26.131386+0000 mgr.y (mgr.34107) 648 : cluster [DBG] pgmap v1226: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 138 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:27 smithi121 bash[36902]: audit 2022-04-22T17:43:26.506888+0000 mon.a (mon.0) 1877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:43:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:27 smithi121 bash[36902]: audit 2022-04-22T17:43:26.510000+0000 mon.a (mon.0) 1878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:43:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:27 smithi121 bash[36902]: audit 2022-04-22T17:43:26.511803+0000 mon.a (mon.0) 1879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:43:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:27 smithi121 bash[36902]: audit 2022-04-22T17:43:26.869240+0000 mon.a (mon.0) 1880 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[45495]: cluster 2022-04-22T17:43:26.131386+0000 mgr.y (mgr.34107) 648 : cluster [DBG] pgmap v1226: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 138 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[45495]: audit 2022-04-22T17:43:26.506888+0000 mon.a (mon.0) 1877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:43:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[45495]: audit 2022-04-22T17:43:26.510000+0000 mon.a (mon.0) 1878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:43:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[45495]: audit 2022-04-22T17:43:26.511803+0000 mon.a (mon.0) 1879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:43:27.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[45495]: audit 2022-04-22T17:43:26.869240+0000 mon.a (mon.0) 1880 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[46934]: cluster 2022-04-22T17:43:26.131386+0000 mgr.y (mgr.34107) 648 : cluster [DBG] pgmap v1226: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 138 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[46934]: audit 2022-04-22T17:43:26.506888+0000 mon.a (mon.0) 1877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:43:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[46934]: audit 2022-04-22T17:43:26.510000+0000 mon.a (mon.0) 1878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:43:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[46934]: audit 2022-04-22T17:43:26.511803+0000 mon.a (mon.0) 1879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:43:27.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:27 smithi115 bash[46934]: audit 2022-04-22T17:43:26.869240+0000 mon.a (mon.0) 1880 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:29 smithi121 bash[36902]: cluster 2022-04-22T17:43:28.132587+0000 mgr.y (mgr.34107) 649 : cluster [DBG] pgmap v1227: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:29 smithi115 bash[45495]: cluster 2022-04-22T17:43:28.132587+0000 mgr.y (mgr.34107) 649 : cluster [DBG] pgmap v1227: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:29.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:29 smithi115 bash[46934]: cluster 2022-04-22T17:43:28.132587+0000 mgr.y (mgr.34107) 649 : cluster [DBG] pgmap v1227: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:30 smithi121 bash[36902]: audit 2022-04-22T17:43:29.652896+0000 mon.a (mon.0) 1881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:30.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:30 smithi115 bash[45495]: audit 2022-04-22T17:43:29.652896+0000 mon.a (mon.0) 1881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:30 smithi115 bash[46934]: audit 2022-04-22T17:43:29.652896+0000 mon.a (mon.0) 1881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:31 smithi121 bash[36902]: cluster 2022-04-22T17:43:30.133781+0000 mgr.y (mgr.34107) 650 : cluster [DBG] pgmap v1228: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:31.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:31 smithi115 bash[45495]: cluster 2022-04-22T17:43:30.133781+0000 mgr.y (mgr.34107) 650 : cluster [DBG] pgmap v1228: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:31.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:31 smithi115 bash[46934]: cluster 2022-04-22T17:43:30.133781+0000 mgr.y (mgr.34107) 650 : cluster [DBG] pgmap v1228: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:32.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:31.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:33 smithi121 bash[36902]: cluster 2022-04-22T17:43:32.134979+0000 mgr.y (mgr.34107) 651 : cluster [DBG] pgmap v1229: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 360 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:33.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:33 smithi115 bash[45495]: cluster 2022-04-22T17:43:32.134979+0000 mgr.y (mgr.34107) 651 : cluster [DBG] pgmap v1229: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 360 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:33 smithi115 bash[46934]: cluster 2022-04-22T17:43:32.134979+0000 mgr.y (mgr.34107) 651 : cluster [DBG] pgmap v1229: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 360 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:34.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:33.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:34 smithi121 bash[36902]: audit 2022-04-22T17:43:33.829161+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:34 smithi121 bash[36902]: cluster 2022-04-22T17:43:34.136155+0000 mgr.y (mgr.34107) 652 : cluster [DBG] pgmap v1230: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:34 smithi121 bash[36902]: audit 2022-04-22T17:43:34.656146+0000 mon.a (mon.0) 1883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[45495]: audit 2022-04-22T17:43:33.829161+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[45495]: cluster 2022-04-22T17:43:34.136155+0000 mgr.y (mgr.34107) 652 : cluster [DBG] pgmap v1230: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[45495]: audit 2022-04-22T17:43:34.656146+0000 mon.a (mon.0) 1883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[46934]: audit 2022-04-22T17:43:33.829161+0000 mon.a (mon.0) 1882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[46934]: cluster 2022-04-22T17:43:34.136155+0000 mgr.y (mgr.34107) 652 : cluster [DBG] pgmap v1230: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:35.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:34 smithi115 bash[46934]: audit 2022-04-22T17:43:34.656146+0000 mon.a (mon.0) 1883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:37 smithi121 bash[36902]: cluster 2022-04-22T17:43:36.137426+0000 mgr.y (mgr.34107) 653 : cluster [DBG] pgmap v1231: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:37.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:37 smithi115 bash[45495]: cluster 2022-04-22T17:43:36.137426+0000 mgr.y (mgr.34107) 653 : cluster [DBG] pgmap v1231: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:37.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:37 smithi115 bash[46934]: cluster 2022-04-22T17:43:36.137426+0000 mgr.y (mgr.34107) 653 : cluster [DBG] pgmap v1231: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:38.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestRenew (20451 ms) 2022-04-22T17:43:38.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2022-04-22T17:43:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:39 smithi121 bash[36902]: cluster 2022-04-22T17:43:38.138727+0000 mgr.y (mgr.34107) 654 : cluster [DBG] pgmap v1232: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:39 smithi121 bash[36902]: cluster 2022-04-22T17:43:38.193571+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2022-04-22T17:43:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:39 smithi115 bash[45495]: cluster 2022-04-22T17:43:38.138727+0000 mgr.y (mgr.34107) 654 : cluster [DBG] pgmap v1232: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:39 smithi115 bash[45495]: cluster 2022-04-22T17:43:38.193571+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2022-04-22T17:43:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:39 smithi115 bash[46934]: cluster 2022-04-22T17:43:38.138727+0000 mgr.y (mgr.34107) 654 : cluster [DBG] pgmap v1232: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:39 smithi115 bash[46934]: cluster 2022-04-22T17:43:38.193571+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2022-04-22T17:43:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:40 smithi121 bash[36902]: cluster 2022-04-22T17:43:39.205291+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2022-04-22T17:43:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:40 smithi121 bash[36902]: audit 2022-04-22T17:43:39.209385+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.115:0/3096969063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:40 smithi121 bash[36902]: audit 2022-04-22T17:43:39.210346+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:40 smithi121 bash[36902]: audit 2022-04-22T17:43:39.659363+0000 mon.a (mon.0) 1887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[45495]: cluster 2022-04-22T17:43:39.205291+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2022-04-22T17:43:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[45495]: audit 2022-04-22T17:43:39.209385+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.115:0/3096969063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[45495]: audit 2022-04-22T17:43:39.210346+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[45495]: audit 2022-04-22T17:43:39.659363+0000 mon.a (mon.0) 1887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[46934]: cluster 2022-04-22T17:43:39.205291+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[46934]: audit 2022-04-22T17:43:39.209385+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.115:0/3096969063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[46934]: audit 2022-04-22T17:43:39.210346+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:40.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:40 smithi115 bash[46934]: audit 2022-04-22T17:43:39.659363+0000 mon.a (mon.0) 1887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:41 smithi121 bash[36902]: cluster 2022-04-22T17:43:40.139776+0000 mgr.y (mgr.34107) 655 : cluster [DBG] pgmap v1235: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:41.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:41 smithi121 bash[36902]: audit 2022-04-22T17:43:40.214143+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:41.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:41 smithi121 bash[36902]: cluster 2022-04-22T17:43:40.214340+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2022-04-22T17:43:41.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[46934]: cluster 2022-04-22T17:43:40.139776+0000 mgr.y (mgr.34107) 655 : cluster [DBG] pgmap v1235: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:41.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[46934]: audit 2022-04-22T17:43:40.214143+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:41.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[46934]: cluster 2022-04-22T17:43:40.214340+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2022-04-22T17:43:41.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[45495]: cluster 2022-04-22T17:43:40.139776+0000 mgr.y (mgr.34107) 655 : cluster [DBG] pgmap v1235: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:41.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[45495]: audit 2022-04-22T17:43:40.214143+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:41.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[45495]: cluster 2022-04-22T17:43:40.214340+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2022-04-22T17:43:42.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:41.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:42 smithi121 bash[36902]: cluster 2022-04-22T17:43:41.218764+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2022-04-22T17:43:42.555 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:42 smithi121 bash[36902]: audit 2022-04-22T17:43:42.047108+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:42.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:42 smithi115 bash[45495]: cluster 2022-04-22T17:43:41.218764+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2022-04-22T17:43:42.557 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:42 smithi115 bash[45495]: audit 2022-04-22T17:43:42.047108+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:42.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:42 smithi115 bash[46934]: cluster 2022-04-22T17:43:41.218764+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2022-04-22T17:43:42.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:42 smithi115 bash[46934]: audit 2022-04-22T17:43:42.047108+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:43 smithi121 bash[36902]: cluster 2022-04-22T17:43:42.140786+0000 mgr.y (mgr.34107) 656 : cluster [DBG] pgmap v1238: 137 pgs: 9 creating+peering, 13 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:43.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:43 smithi115 bash[46934]: cluster 2022-04-22T17:43:42.140786+0000 mgr.y (mgr.34107) 656 : cluster [DBG] pgmap v1238: 137 pgs: 9 creating+peering, 13 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:43.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:43 smithi115 bash[45495]: cluster 2022-04-22T17:43:42.140786+0000 mgr.y (mgr.34107) 656 : cluster [DBG] pgmap v1238: 137 pgs: 9 creating+peering, 13 unknown, 115 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:44.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:43.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:44.363 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 6m ago 20m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (20m) 7m ago 20m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (8m) 6m ago 21m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (8m) 7m ago 21m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (16m) 7m ago 25m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:43:44.707 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (16m) 6m ago 28m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (15m) 6m ago 28m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (14m) 7m ago 26m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (15m) 6m ago 26m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 6m ago 20m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (7m) 7m ago 20m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (13m) 6m ago 25m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (12m) 6m ago 24m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (12m) 6m ago 24m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (11m) 6m ago 23m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (11m) 7m ago 23m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:43:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (10m) 7m ago 23m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:43:44.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (9m) 7m ago 22m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:43:44.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (9m) 7m ago 22m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:43:44.709 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (7m) 7m ago 20m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:45.110 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:43:45.111 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:43:45.264 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7054 ms) 2022-04-22T17:43:45.265 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2022-04-22T17:43:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[45495]: cluster 2022-04-22T17:43:44.141875+0000 mgr.y (mgr.34107) 657 : cluster [DBG] pgmap v1239: 137 pgs: 9 creating+peering, 128 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[45495]: audit 2022-04-22T17:43:44.661730+0000 mon.a (mon.0) 1892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[45495]: audit 2022-04-22T17:43:45.097712+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.115:0/2723193084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:45.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[46934]: cluster 2022-04-22T17:43:44.141875+0000 mgr.y (mgr.34107) 657 : cluster [DBG] pgmap v1239: 137 pgs: 9 creating+peering, 128 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[46934]: audit 2022-04-22T17:43:44.661730+0000 mon.a (mon.0) 1892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:45 smithi115 bash[46934]: audit 2022-04-22T17:43:45.097712+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.115:0/2723193084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:45 smithi121 bash[36902]: cluster 2022-04-22T17:43:44.141875+0000 mgr.y (mgr.34107) 657 : cluster [DBG] pgmap v1239: 137 pgs: 9 creating+peering, 128 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 B/s wr, 1 op/s 2022-04-22T17:43:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:45 smithi121 bash[36902]: audit 2022-04-22T17:43:44.661730+0000 mon.a (mon.0) 1892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:45 smithi121 bash[36902]: audit 2022-04-22T17:43:45.097712+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.115:0/2723193084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:43:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[45495]: audit 2022-04-22T17:43:44.336878+0000 mgr.y (mgr.34107) 658 : audit [DBG] from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[45495]: audit 2022-04-22T17:43:44.676373+0000 mgr.y (mgr.34107) 659 : audit [DBG] from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[45495]: cluster 2022-04-22T17:43:45.241083+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2022-04-22T17:43:46.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[45495]: cluster 2022-04-22T17:43:46.142826+0000 mgr.y (mgr.34107) 660 : cluster [DBG] pgmap v1241: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[46934]: audit 2022-04-22T17:43:44.336878+0000 mgr.y (mgr.34107) 658 : audit [DBG] from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[46934]: audit 2022-04-22T17:43:44.676373+0000 mgr.y (mgr.34107) 659 : audit [DBG] from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[46934]: cluster 2022-04-22T17:43:45.241083+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2022-04-22T17:43:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:46 smithi115 bash[46934]: cluster 2022-04-22T17:43:46.142826+0000 mgr.y (mgr.34107) 660 : cluster [DBG] pgmap v1241: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:46 smithi121 bash[36902]: audit 2022-04-22T17:43:44.336878+0000 mgr.y (mgr.34107) 658 : audit [DBG] from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:46 smithi121 bash[36902]: audit 2022-04-22T17:43:44.676373+0000 mgr.y (mgr.34107) 659 : audit [DBG] from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:43:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:46 smithi121 bash[36902]: cluster 2022-04-22T17:43:45.241083+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2022-04-22T17:43:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:46 smithi121 bash[36902]: cluster 2022-04-22T17:43:46.142826+0000 mgr.y (mgr.34107) 660 : cluster [DBG] pgmap v1241: 105 pgs: 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[45495]: cluster 2022-04-22T17:43:46.270261+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2022-04-22T17:43:47.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[45495]: audit 2022-04-22T17:43:46.274771+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.115:0/3670473325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[45495]: audit 2022-04-22T17:43:46.280736+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[45495]: audit 2022-04-22T17:43:47.050247+0000 mon.a (mon.0) 1896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[46934]: cluster 2022-04-22T17:43:46.270261+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[46934]: audit 2022-04-22T17:43:46.274771+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.115:0/3670473325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[46934]: audit 2022-04-22T17:43:46.280736+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:47 smithi115 bash[46934]: audit 2022-04-22T17:43:47.050247+0000 mon.a (mon.0) 1896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:47 smithi121 bash[36902]: cluster 2022-04-22T17:43:46.270261+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2022-04-22T17:43:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:47 smithi121 bash[36902]: audit 2022-04-22T17:43:46.274771+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.115:0/3670473325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:47 smithi121 bash[36902]: audit 2022-04-22T17:43:46.280736+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:47 smithi121 bash[36902]: audit 2022-04-22T17:43:47.050247+0000 mon.a (mon.0) 1896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:48.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[45495]: audit 2022-04-22T17:43:47.274361+0000 mon.a (mon.0) 1897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:48.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[45495]: cluster 2022-04-22T17:43:47.274609+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2022-04-22T17:43:48.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[45495]: cluster 2022-04-22T17:43:48.143947+0000 mgr.y (mgr.34107) 661 : cluster [DBG] pgmap v1244: 137 pgs: 7 creating+peering, 14 unknown, 116 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:48.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[46934]: audit 2022-04-22T17:43:47.274361+0000 mon.a (mon.0) 1897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:48.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[46934]: cluster 2022-04-22T17:43:47.274609+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2022-04-22T17:43:48.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:48 smithi115 bash[46934]: cluster 2022-04-22T17:43:48.143947+0000 mgr.y (mgr.34107) 661 : cluster [DBG] pgmap v1244: 137 pgs: 7 creating+peering, 14 unknown, 116 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:48 smithi121 bash[36902]: audit 2022-04-22T17:43:47.274361+0000 mon.a (mon.0) 1897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:48 smithi121 bash[36902]: cluster 2022-04-22T17:43:47.274609+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2022-04-22T17:43:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:48 smithi121 bash[36902]: cluster 2022-04-22T17:43:48.143947+0000 mgr.y (mgr.34107) 661 : cluster [DBG] pgmap v1244: 137 pgs: 7 creating+peering, 14 unknown, 116 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:49 smithi121 bash[36902]: cluster 2022-04-22T17:43:48.296305+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2022-04-22T17:43:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:49 smithi115 bash[45495]: cluster 2022-04-22T17:43:48.296305+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2022-04-22T17:43:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:49 smithi115 bash[46934]: cluster 2022-04-22T17:43:48.296305+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2022-04-22T17:43:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:50 smithi121 bash[36902]: audit 2022-04-22T17:43:49.664776+0000 mon.a (mon.0) 1900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:50 smithi121 bash[36902]: cluster 2022-04-22T17:43:50.145054+0000 mgr.y (mgr.34107) 662 : cluster [DBG] pgmap v1246: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:50 smithi115 bash[45495]: audit 2022-04-22T17:43:49.664776+0000 mon.a (mon.0) 1900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:50 smithi115 bash[45495]: cluster 2022-04-22T17:43:50.145054+0000 mgr.y (mgr.34107) 662 : cluster [DBG] pgmap v1246: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:50 smithi115 bash[46934]: audit 2022-04-22T17:43:49.664776+0000 mon.a (mon.0) 1900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:50 smithi115 bash[46934]: cluster 2022-04-22T17:43:50.145054+0000 mgr.y (mgr.34107) 662 : cluster [DBG] pgmap v1246: 137 pgs: 7 creating+peering, 4 unknown, 126 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:52.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:51.653Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:52 smithi121 bash[36902]: audit 2022-04-22T17:43:52.053746+0000 mon.a (mon.0) 1901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:52.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:52 smithi115 bash[45495]: audit 2022-04-22T17:43:52.053746+0000 mon.a (mon.0) 1901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:52.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:52 smithi115 bash[46934]: audit 2022-04-22T17:43:52.053746+0000 mon.a (mon.0) 1901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:53.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (7873 ms) 2022-04-22T17:43:53.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2022-04-22T17:43:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:53 smithi121 bash[36902]: cluster 2022-04-22T17:43:52.146263+0000 mgr.y (mgr.34107) 663 : cluster [DBG] pgmap v1247: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:53.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:53 smithi115 bash[45495]: cluster 2022-04-22T17:43:52.146263+0000 mgr.y (mgr.34107) 663 : cluster [DBG] pgmap v1247: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:53.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:53 smithi115 bash[46934]: cluster 2022-04-22T17:43:52.146263+0000 mgr.y (mgr.34107) 663 : cluster [DBG] pgmap v1247: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:54.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:43:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:43:53.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:43:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:54 smithi121 bash[36902]: cluster 2022-04-22T17:43:53.120324+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2022-04-22T17:43:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:54 smithi121 bash[36902]: audit 2022-04-22T17:43:53.857420+0000 mon.a (mon.0) 1903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:54.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:54 smithi115 bash[45495]: cluster 2022-04-22T17:43:53.120324+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2022-04-22T17:43:54.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:54 smithi115 bash[45495]: audit 2022-04-22T17:43:53.857420+0000 mon.a (mon.0) 1903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:54.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:54 smithi115 bash[46934]: cluster 2022-04-22T17:43:53.120324+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2022-04-22T17:43:54.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:54 smithi115 bash[46934]: audit 2022-04-22T17:43:53.857420+0000 mon.a (mon.0) 1903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:43:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:55 smithi121 bash[36902]: cluster 2022-04-22T17:43:54.131501+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2022-04-22T17:43:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:55 smithi121 bash[36902]: audit 2022-04-22T17:43:54.134899+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.115:0/3430422875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:55 smithi121 bash[36902]: audit 2022-04-22T17:43:54.136431+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:55 smithi121 bash[36902]: cluster 2022-04-22T17:43:54.146881+0000 mgr.y (mgr.34107) 664 : cluster [DBG] pgmap v1250: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:55 smithi121 bash[36902]: audit 2022-04-22T17:43:54.668159+0000 mon.a (mon.0) 1906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[45495]: cluster 2022-04-22T17:43:54.131501+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2022-04-22T17:43:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[45495]: audit 2022-04-22T17:43:54.134899+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.115:0/3430422875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[45495]: audit 2022-04-22T17:43:54.136431+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[45495]: cluster 2022-04-22T17:43:54.146881+0000 mgr.y (mgr.34107) 664 : cluster [DBG] pgmap v1250: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[45495]: audit 2022-04-22T17:43:54.668159+0000 mon.a (mon.0) 1906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[46934]: cluster 2022-04-22T17:43:54.131501+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2022-04-22T17:43:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[46934]: audit 2022-04-22T17:43:54.134899+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.115:0/3430422875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[46934]: audit 2022-04-22T17:43:54.136431+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:43:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[46934]: cluster 2022-04-22T17:43:54.146881+0000 mgr.y (mgr.34107) 664 : cluster [DBG] pgmap v1250: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:55 smithi115 bash[46934]: audit 2022-04-22T17:43:54.668159+0000 mon.a (mon.0) 1906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:43:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:56 smithi121 bash[36902]: audit 2022-04-22T17:43:55.137055+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:56 smithi121 bash[36902]: cluster 2022-04-22T17:43:55.137413+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2022-04-22T17:43:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:56 smithi115 bash[45495]: audit 2022-04-22T17:43:55.137055+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:56 smithi115 bash[45495]: cluster 2022-04-22T17:43:55.137413+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2022-04-22T17:43:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:56 smithi115 bash[46934]: audit 2022-04-22T17:43:55.137055+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63440-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:43:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:56 smithi115 bash[46934]: cluster 2022-04-22T17:43:55.137413+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2022-04-22T17:43:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:57 smithi121 bash[36902]: cluster 2022-04-22T17:43:56.147917+0000 mgr.y (mgr.34107) 665 : cluster [DBG] pgmap v1252: 137 pgs: 4 creating+peering, 23 unknown, 110 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:57 smithi121 bash[36902]: cluster 2022-04-22T17:43:56.155041+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2022-04-22T17:43:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:57 smithi121 bash[36902]: audit 2022-04-22T17:43:57.056825+0000 mon.a (mon.0) 1910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:57.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[45495]: cluster 2022-04-22T17:43:56.147917+0000 mgr.y (mgr.34107) 665 : cluster [DBG] pgmap v1252: 137 pgs: 4 creating+peering, 23 unknown, 110 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:57.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[45495]: cluster 2022-04-22T17:43:56.155041+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2022-04-22T17:43:57.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[45495]: audit 2022-04-22T17:43:57.056825+0000 mon.a (mon.0) 1910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[46934]: cluster 2022-04-22T17:43:56.147917+0000 mgr.y (mgr.34107) 665 : cluster [DBG] pgmap v1252: 137 pgs: 4 creating+peering, 23 unknown, 110 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:43:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[46934]: cluster 2022-04-22T17:43:56.155041+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2022-04-22T17:43:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:57 smithi115 bash[46934]: audit 2022-04-22T17:43:57.056825+0000 mon.a (mon.0) 1910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:43:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:43:59 smithi121 bash[36902]: cluster 2022-04-22T17:43:58.149158+0000 mgr.y (mgr.34107) 666 : cluster [DBG] pgmap v1254: 137 pgs: 4 creating+peering, 3 unknown, 130 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:43:59 smithi115 bash[45495]: cluster 2022-04-22T17:43:58.149158+0000 mgr.y (mgr.34107) 666 : cluster [DBG] pgmap v1254: 137 pgs: 4 creating+peering, 3 unknown, 130 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:43:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:43:59 smithi115 bash[46934]: cluster 2022-04-22T17:43:58.149158+0000 mgr.y (mgr.34107) 666 : cluster [DBG] pgmap v1254: 137 pgs: 4 creating+peering, 3 unknown, 130 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:44:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:00 smithi121 bash[36902]: audit 2022-04-22T17:43:59.671543+0000 mon.a (mon.0) 1911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:00 smithi115 bash[45495]: audit 2022-04-22T17:43:59.671543+0000 mon.a (mon.0) 1911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:00 smithi115 bash[46934]: audit 2022-04-22T17:43:59.671543+0000 mon.a (mon.0) 1911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:01.191 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8054 ms) 2022-04-22T17:44:01.191 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from ClsLock (69660 ms total) 2022-04-22T17:44:01.192 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:44:01.192 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:44:01.192 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 11 tests from 1 test suite ran. (69660 ms total) 2022-04-22T17:44:01.192 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 11 tests. 2022-04-22T17:44:01.194 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:44:01.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:44:01.252 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2022-04-22T17:44:01.252 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2022-04-22T17:44:01.274 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:44:01.274 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 4 tests from 1 test suite. 2022-04-22T17:44:01.274 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:44:01.274 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from cls_log 2022-04-22T17:44:01.274 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_log.test_log_add_same_time 2022-04-22T17:44:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:01 smithi121 bash[36902]: cluster 2022-04-22T17:44:00.150421+0000 mgr.y (mgr.34107) 667 : cluster [DBG] pgmap v1255: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:44:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:01 smithi121 bash[36902]: cluster 2022-04-22T17:44:01.177023+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2022-04-22T17:44:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:01 smithi115 bash[46934]: cluster 2022-04-22T17:44:00.150421+0000 mgr.y (mgr.34107) 667 : cluster [DBG] pgmap v1255: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:44:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:01 smithi115 bash[46934]: cluster 2022-04-22T17:44:01.177023+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2022-04-22T17:44:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:01 smithi115 bash[45495]: cluster 2022-04-22T17:44:00.150421+0000 mgr.y (mgr.34107) 667 : cluster [DBG] pgmap v1255: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2022-04-22T17:44:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:01 smithi115 bash[45495]: cluster 2022-04-22T17:44:01.177023+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2022-04-22T17:44:02.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:01.653Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:02 smithi121 bash[36902]: audit 2022-04-22T17:44:02.059926+0000 mon.a (mon.0) 1913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:02.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:02 smithi115 bash[45495]: audit 2022-04-22T17:44:02.059926+0000 mon.a (mon.0) 1913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:02.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:02 smithi115 bash[46934]: audit 2022-04-22T17:44:02.059926+0000 mon.a (mon.0) 1913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:03 smithi121 bash[36902]: cluster 2022-04-22T17:44:02.151617+0000 mgr.y (mgr.34107) 668 : cluster [DBG] pgmap v1257: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:03 smithi121 bash[36902]: cluster 2022-04-22T17:44:02.208971+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2022-04-22T17:44:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:03 smithi121 bash[36902]: audit 2022-04-22T17:44:02.216307+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[46934]: cluster 2022-04-22T17:44:02.151617+0000 mgr.y (mgr.34107) 668 : cluster [DBG] pgmap v1257: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[46934]: cluster 2022-04-22T17:44:02.208971+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2022-04-22T17:44:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[46934]: audit 2022-04-22T17:44:02.216307+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[45495]: cluster 2022-04-22T17:44:02.151617+0000 mgr.y (mgr.34107) 668 : cluster [DBG] pgmap v1257: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[45495]: cluster 2022-04-22T17:44:02.208971+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2022-04-22T17:44:03.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[45495]: audit 2022-04-22T17:44:02.216307+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:04.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:03.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:04.253 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_log.test_log_add_same_time (2979 ms) 2022-04-22T17:44:04.253 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_log.test_log_add_different_time 2022-04-22T17:44:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:04 smithi121 bash[36902]: audit 2022-04-22T17:44:03.214541+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:04 smithi121 bash[36902]: cluster 2022-04-22T17:44:03.214823+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2022-04-22T17:44:04.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:04 smithi115 bash[45495]: audit 2022-04-22T17:44:03.214541+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:04.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:04 smithi115 bash[45495]: cluster 2022-04-22T17:44:03.214823+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2022-04-22T17:44:04.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:04 smithi115 bash[46934]: audit 2022-04-22T17:44:03.214541+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? 172.21.15.115:0/2562925458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:04.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:04 smithi115 bash[46934]: cluster 2022-04-22T17:44:03.214823+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2022-04-22T17:44:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:05 smithi121 bash[36902]: cluster 2022-04-22T17:44:04.152510+0000 mgr.y (mgr.34107) 669 : cluster [DBG] pgmap v1260: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:05 smithi121 bash[36902]: cluster 2022-04-22T17:44:04.235305+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2022-04-22T17:44:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:05 smithi121 bash[36902]: audit 2022-04-22T17:44:04.700833+0000 mon.a (mon.0) 1919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:05 smithi121 bash[36902]: audit 2022-04-22T17:44:04.943658+0000 mon.a (mon.0) 1920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[45495]: cluster 2022-04-22T17:44:04.152510+0000 mgr.y (mgr.34107) 669 : cluster [DBG] pgmap v1260: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[45495]: cluster 2022-04-22T17:44:04.235305+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[45495]: audit 2022-04-22T17:44:04.700833+0000 mon.a (mon.0) 1919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[45495]: audit 2022-04-22T17:44:04.943658+0000 mon.a (mon.0) 1920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[46934]: cluster 2022-04-22T17:44:04.152510+0000 mgr.y (mgr.34107) 669 : cluster [DBG] pgmap v1260: 137 pgs: 7 creating+peering, 22 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[46934]: cluster 2022-04-22T17:44:04.235305+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[46934]: audit 2022-04-22T17:44:04.700833+0000 mon.a (mon.0) 1919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:05 smithi115 bash[46934]: audit 2022-04-22T17:44:04.943658+0000 mon.a (mon.0) 1920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: cluster 2022-04-22T17:44:05.240433+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2022-04-22T17:44:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: audit 2022-04-22T17:44:05.243302+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.115:0/701343204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: audit 2022-04-22T17:44:05.254086+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: cluster 2022-04-22T17:44:06.153414+0000 mgr.y (mgr.34107) 670 : cluster [DBG] pgmap v1263: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: audit 2022-04-22T17:44:06.244433+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[45495]: cluster 2022-04-22T17:44:06.244635+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: cluster 2022-04-22T17:44:05.240433+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: audit 2022-04-22T17:44:05.243302+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.115:0/701343204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: audit 2022-04-22T17:44:05.254086+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: cluster 2022-04-22T17:44:06.153414+0000 mgr.y (mgr.34107) 670 : cluster [DBG] pgmap v1263: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: audit 2022-04-22T17:44:06.244433+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:06 smithi115 bash[46934]: cluster 2022-04-22T17:44:06.244635+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: cluster 2022-04-22T17:44:05.240433+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: audit 2022-04-22T17:44:05.243302+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.115:0/701343204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: audit 2022-04-22T17:44:05.254086+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: cluster 2022-04-22T17:44:06.153414+0000 mgr.y (mgr.34107) 670 : cluster [DBG] pgmap v1263: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: audit 2022-04-22T17:44:06.244433+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:06 smithi121 bash[36902]: cluster 2022-04-22T17:44:06.244635+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2022-04-22T17:44:07.274 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_log.test_log_add_different_time (3020 ms) 2022-04-22T17:44:07.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_log.trim_by_time 2022-04-22T17:44:07.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:07 smithi115 bash[45495]: audit 2022-04-22T17:44:07.063826+0000 mon.a (mon.0) 1925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:07.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:07 smithi115 bash[45495]: cluster 2022-04-22T17:44:07.259484+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2022-04-22T17:44:07.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:07 smithi115 bash[46934]: audit 2022-04-22T17:44:07.063826+0000 mon.a (mon.0) 1925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:07.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:07 smithi115 bash[46934]: cluster 2022-04-22T17:44:07.259484+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2022-04-22T17:44:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:07 smithi121 bash[36902]: audit 2022-04-22T17:44:07.063826+0000 mon.a (mon.0) 1925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:07 smithi121 bash[36902]: cluster 2022-04-22T17:44:07.259484+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2022-04-22T17:44:08.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:08 smithi115 bash[45495]: cluster 2022-04-22T17:44:08.154413+0000 mgr.y (mgr.34107) 671 : cluster [DBG] pgmap v1266: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:08 smithi115 bash[46934]: cluster 2022-04-22T17:44:08.154413+0000 mgr.y (mgr.34107) 671 : cluster [DBG] pgmap v1266: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:08 smithi121 bash[36902]: cluster 2022-04-22T17:44:08.154413+0000 mgr.y (mgr.34107) 671 : cluster [DBG] pgmap v1266: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:09 smithi121 bash[36902]: cluster 2022-04-22T17:44:08.288235+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2022-04-22T17:44:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:09 smithi121 bash[36902]: audit 2022-04-22T17:44:08.295308+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:09 smithi121 bash[36902]: audit 2022-04-22T17:44:08.882402+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[45495]: cluster 2022-04-22T17:44:08.288235+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2022-04-22T17:44:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[45495]: audit 2022-04-22T17:44:08.295308+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[45495]: audit 2022-04-22T17:44:08.882402+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:09.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[46934]: cluster 2022-04-22T17:44:08.288235+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2022-04-22T17:44:09.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[46934]: audit 2022-04-22T17:44:08.295308+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:09.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:09 smithi115 bash[46934]: audit 2022-04-22T17:44:08.882402+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:10.324 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_log.trim_by_time (3050 ms) 2022-04-22T17:44:10.324 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_log.trim_by_marker 2022-04-22T17:44:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:10 smithi121 bash[36902]: audit 2022-04-22T17:44:09.290280+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:10 smithi121 bash[36902]: cluster 2022-04-22T17:44:09.290534+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2022-04-22T17:44:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:10 smithi121 bash[36902]: audit 2022-04-22T17:44:09.678940+0000 mon.a (mon.0) 1932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:10 smithi121 bash[36902]: audit 2022-04-22T17:44:09.946532+0000 mon.a (mon.0) 1933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:10 smithi121 bash[36902]: cluster 2022-04-22T17:44:10.155532+0000 mgr.y (mgr.34107) 672 : cluster [DBG] pgmap v1269: 137 pgs: 3 creating+peering, 18 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2022-04-22T17:44:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[46934]: audit 2022-04-22T17:44:09.290280+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[46934]: cluster 2022-04-22T17:44:09.290534+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[46934]: audit 2022-04-22T17:44:09.678940+0000 mon.a (mon.0) 1932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[46934]: audit 2022-04-22T17:44:09.946532+0000 mon.a (mon.0) 1933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[46934]: cluster 2022-04-22T17:44:10.155532+0000 mgr.y (mgr.34107) 672 : cluster [DBG] pgmap v1269: 137 pgs: 3 creating+peering, 18 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[45495]: audit 2022-04-22T17:44:09.290280+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? 172.21.15.115:0/3876100146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[45495]: cluster 2022-04-22T17:44:09.290534+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2022-04-22T17:44:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[45495]: audit 2022-04-22T17:44:09.678940+0000 mon.a (mon.0) 1932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:10.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[45495]: audit 2022-04-22T17:44:09.946532+0000 mon.a (mon.0) 1933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:10.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:10 smithi115 bash[45495]: cluster 2022-04-22T17:44:10.155532+0000 mgr.y (mgr.34107) 672 : cluster [DBG] pgmap v1269: 137 pgs: 3 creating+peering, 18 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2022-04-22T17:44:11.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:11 smithi115 bash[45495]: cluster 2022-04-22T17:44:10.309953+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2022-04-22T17:44:11.667 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:11 smithi115 bash[46934]: cluster 2022-04-22T17:44:10.309953+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2022-04-22T17:44:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:11 smithi121 bash[36902]: cluster 2022-04-22T17:44:10.309953+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2022-04-22T17:44:12.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:11.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:12 smithi121 bash[36902]: cluster 2022-04-22T17:44:11.320840+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2022-04-22T17:44:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:12 smithi121 bash[36902]: audit 2022-04-22T17:44:11.327177+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.115:0/2192931169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:12 smithi121 bash[36902]: audit 2022-04-22T17:44:11.328446+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:12 smithi121 bash[36902]: audit 2022-04-22T17:44:12.067028+0000 mon.a (mon.0) 1937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:12 smithi121 bash[36902]: cluster 2022-04-22T17:44:12.156563+0000 mgr.y (mgr.34107) 673 : cluster [DBG] pgmap v1272: 137 pgs: 11 creating+peering, 21 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[45495]: cluster 2022-04-22T17:44:11.320840+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2022-04-22T17:44:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[45495]: audit 2022-04-22T17:44:11.327177+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.115:0/2192931169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[45495]: audit 2022-04-22T17:44:11.328446+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[45495]: audit 2022-04-22T17:44:12.067028+0000 mon.a (mon.0) 1937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:12.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[45495]: cluster 2022-04-22T17:44:12.156563+0000 mgr.y (mgr.34107) 673 : cluster [DBG] pgmap v1272: 137 pgs: 11 creating+peering, 21 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:12.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[46934]: cluster 2022-04-22T17:44:11.320840+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2022-04-22T17:44:12.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[46934]: audit 2022-04-22T17:44:11.327177+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.115:0/2192931169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[46934]: audit 2022-04-22T17:44:11.328446+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:12.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[46934]: audit 2022-04-22T17:44:12.067028+0000 mon.a (mon.0) 1937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:12.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:12 smithi115 bash[46934]: cluster 2022-04-22T17:44:12.156563+0000 mgr.y (mgr.34107) 673 : cluster [DBG] pgmap v1272: 137 pgs: 11 creating+peering, 21 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_log.trim_by_marker (3031 ms) 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from cls_log (12081 ms total) 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 4 tests from 1 test suite ran. (12081 ms total) 2022-04-22T17:44:13.355 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 4 tests. 2022-04-22T17:44:13.357 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:44:13.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:44:13.369 INFO:tasks.workunit:Running workunit cls/test_cls_numops.sh... 2022-04-22T17:44:13.369 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_numops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_numops.sh 2022-04-22T17:44:13.433 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:44:13.434 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 4 tests from 1 test suite. 2022-04-22T17:44:13.434 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:44:13.434 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from ClsNumOps 2022-04-22T17:44:13.434 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsNumOps.Add 2022-04-22T17:44:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:13 smithi121 bash[36902]: audit 2022-04-22T17:44:12.332288+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:13 smithi121 bash[36902]: cluster 2022-04-22T17:44:12.332455+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2022-04-22T17:44:13.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:13.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:13.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:13 smithi115 bash[45495]: audit 2022-04-22T17:44:12.332288+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:13.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:13 smithi115 bash[45495]: cluster 2022-04-22T17:44:12.332455+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2022-04-22T17:44:13.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:13 smithi115 bash[46934]: audit 2022-04-22T17:44:12.332288+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63821-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:13.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:13 smithi115 bash[46934]: cluster 2022-04-22T17:44:12.332455+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2022-04-22T17:44:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:14 smithi121 bash[36902]: cluster 2022-04-22T17:44:13.339720+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2022-04-22T17:44:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:14 smithi121 bash[36902]: audit 2022-04-22T17:44:13.397215+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:44:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:14 smithi121 bash[36902]: audit 2022-04-22T17:44:13.397689+0000 mon.a (mon.0) 1942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:44:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:14 smithi121 bash[36902]: cluster 2022-04-22T17:44:14.157483+0000 mgr.y (mgr.34107) 674 : cluster [DBG] pgmap v1275: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[45495]: cluster 2022-04-22T17:44:13.339720+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[45495]: audit 2022-04-22T17:44:13.397215+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[45495]: audit 2022-04-22T17:44:13.397689+0000 mon.a (mon.0) 1942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[45495]: cluster 2022-04-22T17:44:14.157483+0000 mgr.y (mgr.34107) 674 : cluster [DBG] pgmap v1275: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[46934]: cluster 2022-04-22T17:44:13.339720+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[46934]: audit 2022-04-22T17:44:13.397215+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[46934]: audit 2022-04-22T17:44:13.397689+0000 mon.a (mon.0) 1942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:44:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:14 smithi115 bash[46934]: cluster 2022-04-22T17:44:14.157483+0000 mgr.y (mgr.34107) 674 : cluster [DBG] pgmap v1275: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:15.503 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:44:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:15 smithi121 bash[36902]: cluster 2022-04-22T17:44:14.364834+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2022-04-22T17:44:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:15 smithi121 bash[36902]: audit 2022-04-22T17:44:14.369419+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:15 smithi121 bash[36902]: audit 2022-04-22T17:44:14.682326+0000 mon.a (mon.0) 1945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:15 smithi121 bash[36902]: audit 2022-04-22T17:44:14.950314+0000 mon.a (mon.0) 1946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[45495]: cluster 2022-04-22T17:44:14.364834+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[45495]: audit 2022-04-22T17:44:14.369419+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[45495]: audit 2022-04-22T17:44:14.682326+0000 mon.a (mon.0) 1945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[45495]: audit 2022-04-22T17:44:14.950314+0000 mon.a (mon.0) 1946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[46934]: cluster 2022-04-22T17:44:14.364834+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[46934]: audit 2022-04-22T17:44:14.369419+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[46934]: audit 2022-04-22T17:44:14.682326+0000 mon.a (mon.0) 1945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:15 smithi115 bash[46934]: audit 2022-04-22T17:44:14.950314+0000 mon.a (mon.0) 1946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:15.855 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (7m) 7m ago 21m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (20m) 7m ago 20m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (9m) 7m ago 22m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (8m) 7m ago 22m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (16m) 7m ago 26m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (17m) 7m ago 28m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (16m) 7m ago 28m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:44:15.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (15m) 7m ago 26m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (15m) 7m ago 26m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (8m) 7m ago 21m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (8m) 7m ago 21m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (14m) 7m ago 25m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (13m) 7m ago 25m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (12m) 7m ago 25m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (12m) 7m ago 24m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:44:15.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (11m) 7m ago 24m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:44:15.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (11m) 7m ago 23m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:44:15.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (10m) 7m ago 23m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:44:15.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (9m) 7m ago 22m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:44:15.866 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (8m) 7m ago 21m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:44:16.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:44:16.269 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:44:16.407 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsNumOps.Add (2973 ms) 2022-04-22T17:44:16.452 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsNumOps.Sub 2022-04-22T17:44:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: audit 2022-04-22T17:44:15.366157+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: cluster 2022-04-22T17:44:15.366551+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2022-04-22T17:44:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: audit 2022-04-22T17:44:15.472888+0000 mgr.y (mgr.34107) 675 : audit [DBG] from='client.35037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: audit 2022-04-22T17:44:15.821865+0000 mgr.y (mgr.34107) 676 : audit [DBG] from='client.44986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: cluster 2022-04-22T17:44:16.158461+0000 mgr.y (mgr.34107) 677 : cluster [DBG] pgmap v1278: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:16 smithi121 bash[36902]: audit 2022-04-22T17:44:16.256414+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.115:0/2051192205' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: audit 2022-04-22T17:44:15.366157+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: cluster 2022-04-22T17:44:15.366551+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: audit 2022-04-22T17:44:15.472888+0000 mgr.y (mgr.34107) 675 : audit [DBG] from='client.35037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: audit 2022-04-22T17:44:15.821865+0000 mgr.y (mgr.34107) 676 : audit [DBG] from='client.44986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: cluster 2022-04-22T17:44:16.158461+0000 mgr.y (mgr.34107) 677 : cluster [DBG] pgmap v1278: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[45495]: audit 2022-04-22T17:44:16.256414+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.115:0/2051192205' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: audit 2022-04-22T17:44:15.366157+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? 172.21.15.115:0/4183028466' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: cluster 2022-04-22T17:44:15.366551+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2022-04-22T17:44:16.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: audit 2022-04-22T17:44:15.472888+0000 mgr.y (mgr.34107) 675 : audit [DBG] from='client.35037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: audit 2022-04-22T17:44:15.821865+0000 mgr.y (mgr.34107) 676 : audit [DBG] from='client.44986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:16.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: cluster 2022-04-22T17:44:16.158461+0000 mgr.y (mgr.34107) 677 : cluster [DBG] pgmap v1278: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:16.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:16 smithi115 bash[46934]: audit 2022-04-22T17:44:16.256414+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.115:0/2051192205' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:17 smithi121 bash[36902]: cluster 2022-04-22T17:44:16.386423+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2022-04-22T17:44:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:17 smithi121 bash[36902]: audit 2022-04-22T17:44:17.070163+0000 mon.a (mon.0) 1950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:17 smithi115 bash[45495]: cluster 2022-04-22T17:44:16.386423+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2022-04-22T17:44:17.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:17 smithi115 bash[45495]: audit 2022-04-22T17:44:17.070163+0000 mon.a (mon.0) 1950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:17 smithi115 bash[46934]: cluster 2022-04-22T17:44:16.386423+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2022-04-22T17:44:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:17 smithi115 bash[46934]: audit 2022-04-22T17:44:17.070163+0000 mon.a (mon.0) 1950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:18 smithi121 bash[36902]: cluster 2022-04-22T17:44:17.399412+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2022-04-22T17:44:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:18 smithi121 bash[36902]: audit 2022-04-22T17:44:17.405043+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.115:0/312689784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:18 smithi121 bash[36902]: audit 2022-04-22T17:44:17.405620+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:18 smithi121 bash[36902]: cluster 2022-04-22T17:44:18.159532+0000 mgr.y (mgr.34107) 678 : cluster [DBG] pgmap v1281: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:18.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[45495]: cluster 2022-04-22T17:44:17.399412+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[45495]: audit 2022-04-22T17:44:17.405043+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.115:0/312689784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[45495]: audit 2022-04-22T17:44:17.405620+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[45495]: cluster 2022-04-22T17:44:18.159532+0000 mgr.y (mgr.34107) 678 : cluster [DBG] pgmap v1281: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[46934]: cluster 2022-04-22T17:44:17.399412+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[46934]: audit 2022-04-22T17:44:17.405043+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.115:0/312689784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[46934]: audit 2022-04-22T17:44:17.405620+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:18.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:18 smithi115 bash[46934]: cluster 2022-04-22T17:44:18.159532+0000 mgr.y (mgr.34107) 678 : cluster [DBG] pgmap v1281: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:19.434 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsNumOps.Sub (3027 ms) 2022-04-22T17:44:19.434 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsNumOps.Mul 2022-04-22T17:44:19.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:19 smithi121 bash[36902]: audit 2022-04-22T17:44:18.407485+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:19.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:19 smithi121 bash[36902]: cluster 2022-04-22T17:44:18.407675+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2022-04-22T17:44:19.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:19 smithi115 bash[45495]: audit 2022-04-22T17:44:18.407485+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:19.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:19 smithi115 bash[45495]: cluster 2022-04-22T17:44:18.407675+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2022-04-22T17:44:19.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:19 smithi115 bash[46934]: audit 2022-04-22T17:44:18.407485+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:19.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:19 smithi115 bash[46934]: cluster 2022-04-22T17:44:18.407675+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2022-04-22T17:44:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:20 smithi121 bash[36902]: cluster 2022-04-22T17:44:19.417090+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2022-04-22T17:44:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:20 smithi121 bash[36902]: audit 2022-04-22T17:44:19.873720+0000 mon.a (mon.0) 1956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:20 smithi121 bash[36902]: audit 2022-04-22T17:44:19.954163+0000 mon.a (mon.0) 1957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:20 smithi121 bash[36902]: cluster 2022-04-22T17:44:20.160440+0000 mgr.y (mgr.34107) 679 : cluster [DBG] pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[46934]: cluster 2022-04-22T17:44:19.417090+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[46934]: audit 2022-04-22T17:44:19.873720+0000 mon.a (mon.0) 1956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[46934]: audit 2022-04-22T17:44:19.954163+0000 mon.a (mon.0) 1957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[46934]: cluster 2022-04-22T17:44:20.160440+0000 mgr.y (mgr.34107) 679 : cluster [DBG] pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[45495]: cluster 2022-04-22T17:44:19.417090+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2022-04-22T17:44:20.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[45495]: audit 2022-04-22T17:44:19.873720+0000 mon.a (mon.0) 1956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[45495]: audit 2022-04-22T17:44:19.954163+0000 mon.a (mon.0) 1957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:20 smithi115 bash[45495]: cluster 2022-04-22T17:44:20.160440+0000 mgr.y (mgr.34107) 679 : cluster [DBG] pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:21 smithi121 bash[36902]: cluster 2022-04-22T17:44:20.433550+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2022-04-22T17:44:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:21 smithi121 bash[36902]: audit 2022-04-22T17:44:20.438473+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.115:0/985237886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:21 smithi121 bash[36902]: audit 2022-04-22T17:44:20.440263+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[46934]: cluster 2022-04-22T17:44:20.433550+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2022-04-22T17:44:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[46934]: audit 2022-04-22T17:44:20.438473+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.115:0/985237886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[46934]: audit 2022-04-22T17:44:20.440263+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:21.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:21.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[45495]: cluster 2022-04-22T17:44:20.433550+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2022-04-22T17:44:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[45495]: audit 2022-04-22T17:44:20.438473+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.115:0/985237886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:21 smithi115 bash[45495]: audit 2022-04-22T17:44:20.440263+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:22.462 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsNumOps.Mul (3029 ms) 2022-04-22T17:44:22.463 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsNumOps.Div 2022-04-22T17:44:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:22 smithi121 bash[36902]: audit 2022-04-22T17:44:21.438019+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:22 smithi121 bash[36902]: cluster 2022-04-22T17:44:21.438164+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2022-04-22T17:44:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:22 smithi121 bash[36902]: audit 2022-04-22T17:44:22.073100+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:22 smithi121 bash[36902]: cluster 2022-04-22T17:44:22.161462+0000 mgr.y (mgr.34107) 680 : cluster [DBG] pgmap v1287: 137 pgs: 5 creating+peering, 21 unknown, 111 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T17:44:22.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[45495]: audit 2022-04-22T17:44:21.438019+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[45495]: cluster 2022-04-22T17:44:21.438164+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[45495]: audit 2022-04-22T17:44:22.073100+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[45495]: cluster 2022-04-22T17:44:22.161462+0000 mgr.y (mgr.34107) 680 : cluster [DBG] pgmap v1287: 137 pgs: 5 creating+peering, 21 unknown, 111 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[46934]: audit 2022-04-22T17:44:21.438019+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[46934]: cluster 2022-04-22T17:44:21.438164+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[46934]: audit 2022-04-22T17:44:22.073100+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:22 smithi115 bash[46934]: cluster 2022-04-22T17:44:22.161462+0000 mgr.y (mgr.34107) 680 : cluster [DBG] pgmap v1287: 137 pgs: 5 creating+peering, 21 unknown, 111 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T17:44:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:23 smithi121 bash[36902]: cluster 2022-04-22T17:44:22.446652+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2022-04-22T17:44:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:23 smithi115 bash[45495]: cluster 2022-04-22T17:44:22.446652+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2022-04-22T17:44:23.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:23.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:23.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:23 smithi115 bash[46934]: cluster 2022-04-22T17:44:22.446652+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2022-04-22T17:44:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:24 smithi121 bash[36902]: cluster 2022-04-22T17:44:23.455507+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2022-04-22T17:44:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:24 smithi121 bash[36902]: audit 2022-04-22T17:44:23.458810+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:24 smithi121 bash[36902]: cluster 2022-04-22T17:44:24.162512+0000 mgr.y (mgr.34107) 681 : cluster [DBG] pgmap v1290: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:24.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[45495]: cluster 2022-04-22T17:44:23.455507+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2022-04-22T17:44:24.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[45495]: audit 2022-04-22T17:44:23.458810+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:24.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[45495]: cluster 2022-04-22T17:44:24.162512+0000 mgr.y (mgr.34107) 681 : cluster [DBG] pgmap v1290: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:24.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[46934]: cluster 2022-04-22T17:44:23.455507+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2022-04-22T17:44:24.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[46934]: audit 2022-04-22T17:44:23.458810+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:24.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:24 smithi115 bash[46934]: cluster 2022-04-22T17:44:24.162512+0000 mgr.y (mgr.34107) 681 : cluster [DBG] pgmap v1290: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:25.496 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsNumOps.Div (3034 ms) 2022-04-22T17:44:25.496 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from ClsNumOps (12063 ms total) 2022-04-22T17:44:25.496 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:44:25.497 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:44:25.497 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 4 tests from 1 test suite ran. (12063 ms total) 2022-04-22T17:44:25.497 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 4 tests. 2022-04-22T17:44:25.499 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:44:25.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:44:25.511 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2022-04-22T17:44:25.511 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2022-04-22T17:44:25.527 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:44:25.527 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 69 tests from 1 test suite. 2022-04-22T17:44:25.527 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:44:25.527 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 69 tests from TestClsRbd 2022-04-22T17:44:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:25 smithi121 bash[36902]: audit 2022-04-22T17:44:24.467715+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:25.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:25 smithi121 bash[36902]: cluster 2022-04-22T17:44:24.467949+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2022-04-22T17:44:25.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:25 smithi121 bash[36902]: audit 2022-04-22T17:44:24.689928+0000 mon.a (mon.0) 1968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:25.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:25 smithi121 bash[36902]: audit 2022-04-22T17:44:24.957397+0000 mon.a (mon.0) 1969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:25.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[45495]: audit 2022-04-22T17:44:24.467715+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:25.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[45495]: cluster 2022-04-22T17:44:24.467949+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[45495]: audit 2022-04-22T17:44:24.689928+0000 mon.a (mon.0) 1968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[45495]: audit 2022-04-22T17:44:24.957397+0000 mon.a (mon.0) 1969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[46934]: audit 2022-04-22T17:44:24.467715+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.115:0/3583239634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-63896-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[46934]: cluster 2022-04-22T17:44:24.467949+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[46934]: audit 2022-04-22T17:44:24.689928+0000 mon.a (mon.0) 1968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:25 smithi115 bash[46934]: audit 2022-04-22T17:44:24.957397+0000 mon.a (mon.0) 1969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:26 smithi121 bash[36902]: cluster 2022-04-22T17:44:25.474702+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2022-04-22T17:44:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:26 smithi121 bash[36902]: cluster 2022-04-22T17:44:26.163562+0000 mgr.y (mgr.34107) 682 : cluster [DBG] pgmap v1293: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:26 smithi115 bash[45495]: cluster 2022-04-22T17:44:25.474702+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2022-04-22T17:44:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:26 smithi115 bash[45495]: cluster 2022-04-22T17:44:26.163562+0000 mgr.y (mgr.34107) 682 : cluster [DBG] pgmap v1293: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:26.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:26 smithi115 bash[46934]: cluster 2022-04-22T17:44:25.474702+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2022-04-22T17:44:26.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:26 smithi115 bash[46934]: cluster 2022-04-22T17:44:26.163562+0000 mgr.y (mgr.34107) 682 : cluster [DBG] pgmap v1293: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:27.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_all_features 2022-04-22T17:44:27.534 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_all_features (19 ms) 2022-04-22T17:44:27.534 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.copyup 2022-04-22T17:44:27.743 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.copyup (209 ms) 2022-04-22T17:44:27.744 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.sparse_copyup 2022-04-22T17:44:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: cluster 2022-04-22T17:44:26.486455+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:26.491559+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.115:0/1966819261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:26.492424+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:26.877808+0000 mon.a (mon.0) 1973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:26.880807+0000 mon.a (mon.0) 1974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:26.882605+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:27.076072+0000 mon.a (mon.0) 1976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:27 smithi121 bash[36902]: audit 2022-04-22T17:44:27.236981+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:27.767 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.sparse_copyup (24 ms) 2022-04-22T17:44:27.768 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_and_set_id 2022-04-22T17:44:27.782 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_and_set_id (14 ms) 2022-04-22T17:44:27.782 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.add_remove_child 2022-04-22T17:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: cluster 2022-04-22T17:44:26.486455+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2022-04-22T17:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:26.491559+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.115:0/1966819261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:26.492424+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:26.877808+0000 mon.a (mon.0) 1973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:26.880807+0000 mon.a (mon.0) 1974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:26.882605+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:27.076072+0000 mon.a (mon.0) 1976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[45495]: audit 2022-04-22T17:44:27.236981+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: cluster 2022-04-22T17:44:26.486455+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:26.491559+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.115:0/1966819261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:26.492424+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:26.877808+0000 mon.a (mon.0) 1973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:44:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:26.880807+0000 mon.a (mon.0) 1974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:44:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:26.882605+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:44:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:27.076072+0000 mon.a (mon.0) 1976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:27 smithi115 bash[46934]: audit 2022-04-22T17:44:27.236981+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:27.804 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.add_remove_child (22 ms) 2022-04-22T17:44:27.804 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.directory_methods 2022-04-22T17:44:27.863 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.directory_methods (59 ms) 2022-04-22T17:44:27.863 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.create 2022-04-22T17:44:27.889 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.create (26 ms) 2022-04-22T17:44:27.935 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_features 2022-04-22T17:44:27.937 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_features (3 ms) 2022-04-22T17:44:27.937 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_object_prefix 2022-04-22T17:44:27.937 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_object_prefix (4 ms) 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_access_timestamp (3 ms) 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_modify_timestamp (2 ms) 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_data_pool 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_data_pool (7 ms) 2022-04-22T17:44:27.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.get_size 2022-04-22T17:44:27.939 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.get_size (7 ms) 2022-04-22T17:44:27.939 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.set_size 2022-04-22T17:44:27.939 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.set_size (11 ms) 2022-04-22T17:44:27.939 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.protection_status 2022-04-22T17:44:27.965 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.protection_status (36 ms) 2022-04-22T17:44:27.997 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.snapshot_limits 2022-04-22T17:44:27.997 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.snapshot_limits (16 ms) 2022-04-22T17:44:27.998 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.parents_v1 2022-04-22T17:44:28.070 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.parents_v1 (88 ms) 2022-04-22T17:44:28.070 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.parents_v2 2022-04-22T17:44:28.133 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.parents_v2 (63 ms) 2022-04-22T17:44:28.133 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.snapshots 2022-04-22T17:44:28.180 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.snapshots (48 ms) 2022-04-22T17:44:28.181 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.snapid_race 2022-04-22T17:44:28.189 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.snapid_race (8 ms) 2022-04-22T17:44:28.189 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.stripingv2 2022-04-22T17:44:28.209 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.stripingv2 (20 ms) 2022-04-22T17:44:28.209 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_save 2022-04-22T17:44:28.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_save (4 ms) 2022-04-22T17:44:28.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_resize 2022-04-22T17:44:28.227 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_resize (15 ms) 2022-04-22T17:44:28.227 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_update 2022-04-22T17:44:28.240 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_update (13 ms) 2022-04-22T17:44:28.241 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2022-04-22T17:44:28.241 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_load_enoent (1 ms) 2022-04-22T17:44:28.241 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2022-04-22T17:44:28.252 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_snap_add (11 ms) 2022-04-22T17:44:28.252 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2022-04-22T17:44:28.264 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.object_map_snap_remove (11 ms) 2022-04-22T17:44:28.264 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.flags 2022-04-22T17:44:28.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.flags (11 ms) 2022-04-22T17:44:28.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.metadata 2022-04-22T17:44:28.321 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.metadata (46 ms) 2022-04-22T17:44:28.321 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.set_features 2022-04-22T17:44:28.334 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.set_features (14 ms) 2022-04-22T17:44:28.334 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror 2022-04-22T17:44:28.403 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror (69 ms) 2022-04-22T17:44:28.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror_image 2022-04-22T17:44:28.433 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror_image (30 ms) 2022-04-22T17:44:28.434 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror_image_status 2022-04-22T17:44:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:28 smithi121 bash[36902]: audit 2022-04-22T17:44:27.498074+0000 mon.a (mon.0) 1978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:28 smithi121 bash[36902]: cluster 2022-04-22T17:44:27.498230+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2022-04-22T17:44:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:28 smithi121 bash[36902]: cluster 2022-04-22T17:44:28.164714+0000 mgr.y (mgr.34107) 683 : cluster [DBG] pgmap v1296: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 9.2 KiB/s wr, 59 op/s 2022-04-22T17:44:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[46934]: audit 2022-04-22T17:44:27.498074+0000 mon.a (mon.0) 1978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[46934]: cluster 2022-04-22T17:44:27.498230+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2022-04-22T17:44:28.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[46934]: cluster 2022-04-22T17:44:28.164714+0000 mgr.y (mgr.34107) 683 : cluster [DBG] pgmap v1296: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 9.2 KiB/s wr, 59 op/s 2022-04-22T17:44:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[45495]: audit 2022-04-22T17:44:27.498074+0000 mon.a (mon.0) 1978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64043-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[45495]: cluster 2022-04-22T17:44:27.498230+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2022-04-22T17:44:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:28 smithi115 bash[45495]: cluster 2022-04-22T17:44:28.164714+0000 mgr.y (mgr.34107) 683 : cluster [DBG] pgmap v1296: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 9.2 KiB/s wr, 59 op/s 2022-04-22T17:44:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:29 smithi121 bash[36902]: cluster 2022-04-22T17:44:28.507181+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2022-04-22T17:44:29.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:29 smithi115 bash[45495]: cluster 2022-04-22T17:44:28.507181+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2022-04-22T17:44:29.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:29 smithi115 bash[46934]: cluster 2022-04-22T17:44:28.507181+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2022-04-22T17:44:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[46934]: audit 2022-04-22T17:44:29.692949+0000 mon.a (mon.0) 1981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[46934]: audit 2022-04-22T17:44:29.960845+0000 mon.a (mon.0) 1982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[46934]: cluster 2022-04-22T17:44:30.165554+0000 mgr.y (mgr.34107) 684 : cluster [DBG] pgmap v1298: 137 pgs: 8 creating+peering, 2 unknown, 127 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 113 KiB/s rd, 30 KiB/s wr, 156 op/s 2022-04-22T17:44:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[45495]: audit 2022-04-22T17:44:29.692949+0000 mon.a (mon.0) 1981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[45495]: audit 2022-04-22T17:44:29.960845+0000 mon.a (mon.0) 1982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:30 smithi115 bash[45495]: cluster 2022-04-22T17:44:30.165554+0000 mgr.y (mgr.34107) 684 : cluster [DBG] pgmap v1298: 137 pgs: 8 creating+peering, 2 unknown, 127 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 113 KiB/s rd, 30 KiB/s wr, 156 op/s 2022-04-22T17:44:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:30 smithi121 bash[36902]: audit 2022-04-22T17:44:29.692949+0000 mon.a (mon.0) 1981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:30 smithi121 bash[36902]: audit 2022-04-22T17:44:29.960845+0000 mon.a (mon.0) 1982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:30 smithi121 bash[36902]: cluster 2022-04-22T17:44:30.165554+0000 mgr.y (mgr.34107) 684 : cluster [DBG] pgmap v1298: 137 pgs: 8 creating+peering, 2 unknown, 127 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 113 KiB/s rd, 30 KiB/s wr, 156 op/s 2022-04-22T17:44:32.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:31.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:32 smithi121 bash[36902]: audit 2022-04-22T17:44:32.078494+0000 mon.a (mon.0) 1983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:32 smithi115 bash[45495]: audit 2022-04-22T17:44:32.078494+0000 mon.a (mon.0) 1983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:32.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:32 smithi115 bash[46934]: audit 2022-04-22T17:44:32.078494+0000 mon.a (mon.0) 1983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:33.138 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror_image_status (4704 ms) 2022-04-22T17:44:33.138 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror_image_map 2022-04-22T17:44:33.303 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror_image_map (165 ms) 2022-04-22T17:44:33.303 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror_instances 2022-04-22T17:44:33.319 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror_instances (16 ms) 2022-04-22T17:44:33.320 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2022-04-22T17:44:33.344 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.mirror_snapshot (25 ms) 2022-04-22T17:44:33.345 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_dir_list 2022-04-22T17:44:33.349 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_dir_list (4 ms) 2022-04-22T17:44:33.349 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_dir_add 2022-04-22T17:44:33.353 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_dir_add (5 ms) 2022-04-22T17:44:33.353 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2022-04-22T17:44:33.359 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.dir_add_already_existing (5 ms) 2022-04-22T17:44:33.359 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_dir_rename 2022-04-22T17:44:33.371 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_dir_rename (12 ms) 2022-04-22T17:44:33.371 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_dir_remove 2022-04-22T17:44:33.379 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_dir_remove (7 ms) 2022-04-22T17:44:33.379 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2022-04-22T17:44:33.387 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (8 ms) 2022-04-22T17:44:33.387 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_image_add 2022-04-22T17:44:33.392 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_image_add (5 ms) 2022-04-22T17:44:33.392 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_image_remove 2022-04-22T17:44:33.399 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_image_remove (7 ms) 2022-04-22T17:44:33.399 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_image_list 2022-04-22T17:44:33.404 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_image_list (5 ms) 2022-04-22T17:44:33.405 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_image_clean 2022-04-22T17:44:33.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_image_clean (8 ms) 2022-04-22T17:44:33.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.image_group_add 2022-04-22T17:44:33.417 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.image_group_add (5 ms) 2022-04-22T17:44:33.417 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.image_group_remove 2022-04-22T17:44:33.424 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.image_group_remove (6 ms) 2022-04-22T17:44:33.424 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.image_group_get 2022-04-22T17:44:33.429 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.image_group_get (6 ms) 2022-04-22T17:44:33.429 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2022-04-22T17:44:33.432 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (3 ms) 2022-04-22T17:44:33.432 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2022-04-22T17:44:33.436 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (4 ms) 2022-04-22T17:44:33.436 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2022-04-22T17:44:33.442 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (6 ms) 2022-04-22T17:44:33.442 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2022-04-22T17:44:33.448 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (6 ms) 2022-04-22T17:44:33.448 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_set 2022-04-22T17:44:33.452 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_set (4 ms) 2022-04-22T17:44:33.452 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_list 2022-04-22T17:44:33.459 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_list (6 ms) 2022-04-22T17:44:33.459 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2022-04-22T17:44:33.490 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (32 ms) 2022-04-22T17:44:33.491 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_remove 2022-04-22T17:44:33.497 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_remove (7 ms) 2022-04-22T17:44:33.497 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2022-04-22T17:44:33.501 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (4 ms) 2022-04-22T17:44:33.501 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.trash_methods 2022-04-22T17:44:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:33 smithi121 bash[36902]: cluster 2022-04-22T17:44:32.166661+0000 mgr.y (mgr.34107) 685 : cluster [DBG] pgmap v1299: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 1.14k op/s 2022-04-22T17:44:33.517 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.trash_methods (16 ms) 2022-04-22T17:44:33.517 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.op_features 2022-04-22T17:44:33.530 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.op_features (13 ms) 2022-04-22T17:44:33.530 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.clone_parent 2022-04-22T17:44:33.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:33 smithi115 bash[45495]: cluster 2022-04-22T17:44:32.166661+0000 mgr.y (mgr.34107) 685 : cluster [DBG] pgmap v1299: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 1.14k op/s 2022-04-22T17:44:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:33 smithi115 bash[46934]: cluster 2022-04-22T17:44:32.166661+0000 mgr.y (mgr.34107) 685 : cluster [DBG] pgmap v1299: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 1.14k op/s 2022-04-22T17:44:33.569 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.clone_parent (39 ms) 2022-04-22T17:44:33.569 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2022-04-22T17:44:33.587 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.clone_parent_ns (17 ms) 2022-04-22T17:44:33.587 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.clone_child 2022-04-22T17:44:33.610 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.clone_child (24 ms) 2022-04-22T17:44:33.611 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.namespace_methods 2022-04-22T17:44:33.623 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.namespace_methods (13 ms) 2022-04-22T17:44:33.623 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.migration 2022-04-22T17:44:33.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.migration (19 ms) 2022-04-22T17:44:33.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.migration_v1 2022-04-22T17:44:33.667 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.migration_v1 (24 ms) 2022-04-22T17:44:33.667 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2022-04-22T17:44:34.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:33.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:35.168 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1500 ms) 2022-04-22T17:44:35.168 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestClsRbd.sparsify 2022-04-22T17:44:35.229 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestClsRbd.sparsify (62 ms) 2022-04-22T17:44:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:35 smithi121 bash[36902]: cluster 2022-04-22T17:44:34.151875+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2022-04-22T17:44:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:35 smithi121 bash[36902]: cluster 2022-04-22T17:44:34.167228+0000 mgr.y (mgr.34107) 686 : cluster [DBG] pgmap v1301: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1018 KiB/s wr, 1.05k op/s 2022-04-22T17:44:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:35 smithi121 bash[36902]: audit 2022-04-22T17:44:34.695877+0000 mon.a (mon.0) 1985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:35 smithi121 bash[36902]: audit 2022-04-22T17:44:34.964188+0000 mon.a (mon.0) 1986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[45495]: cluster 2022-04-22T17:44:34.151875+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2022-04-22T17:44:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[45495]: cluster 2022-04-22T17:44:34.167228+0000 mgr.y (mgr.34107) 686 : cluster [DBG] pgmap v1301: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1018 KiB/s wr, 1.05k op/s 2022-04-22T17:44:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[45495]: audit 2022-04-22T17:44:34.695877+0000 mon.a (mon.0) 1985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[45495]: audit 2022-04-22T17:44:34.964188+0000 mon.a (mon.0) 1986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[46934]: cluster 2022-04-22T17:44:34.151875+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2022-04-22T17:44:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[46934]: cluster 2022-04-22T17:44:34.167228+0000 mgr.y (mgr.34107) 686 : cluster [DBG] pgmap v1301: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1018 KiB/s wr, 1.05k op/s 2022-04-22T17:44:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[46934]: audit 2022-04-22T17:44:34.695877+0000 mon.a (mon.0) 1985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:35 smithi115 bash[46934]: audit 2022-04-22T17:44:34.964188+0000 mon.a (mon.0) 1986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:36.244 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 69 tests from TestClsRbd (7714 ms total) 2022-04-22T17:44:36.245 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:44:36.245 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:44:36.245 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 69 tests from 1 test suite ran. (10718 ms total) 2022-04-22T17:44:36.245 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 69 tests. 2022-04-22T17:44:36.247 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:44:36.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:44:36.305 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2022-04-22T17:44:36.306 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2022-04-22T17:44:36.329 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:44:36.364 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 5 tests from 1 test suite. 2022-04-22T17:44:36.364 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:44:36.364 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 5 tests from cls_rgw 2022-04-22T17:44:36.364 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.test_implicit 2022-04-22T17:44:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:36 smithi121 bash[36902]: cluster 2022-04-22T17:44:35.155849+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2022-04-22T17:44:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:36 smithi115 bash[45495]: cluster 2022-04-22T17:44:35.155849+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2022-04-22T17:44:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:36 smithi115 bash[46934]: cluster 2022-04-22T17:44:35.155849+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2022-04-22T17:44:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:37 smithi121 bash[36902]: cluster 2022-04-22T17:44:36.168194+0000 mgr.y (mgr.34107) 687 : cluster [DBG] pgmap v1303: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 890 KiB/s wr, 918 op/s 2022-04-22T17:44:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:37 smithi121 bash[36902]: cluster 2022-04-22T17:44:36.224644+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2022-04-22T17:44:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:37 smithi121 bash[36902]: audit 2022-04-22T17:44:37.081451+0000 mon.a (mon.0) 1989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[45495]: cluster 2022-04-22T17:44:36.168194+0000 mgr.y (mgr.34107) 687 : cluster [DBG] pgmap v1303: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 890 KiB/s wr, 918 op/s 2022-04-22T17:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[45495]: cluster 2022-04-22T17:44:36.224644+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2022-04-22T17:44:37.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[45495]: audit 2022-04-22T17:44:37.081451+0000 mon.a (mon.0) 1989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:37.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[46934]: cluster 2022-04-22T17:44:36.168194+0000 mgr.y (mgr.34107) 687 : cluster [DBG] pgmap v1303: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 890 KiB/s wr, 918 op/s 2022-04-22T17:44:37.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[46934]: cluster 2022-04-22T17:44:36.224644+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2022-04-22T17:44:37.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:37 smithi115 bash[46934]: audit 2022-04-22T17:44:37.081451+0000 mon.a (mon.0) 1989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:38 smithi121 bash[36902]: cluster 2022-04-22T17:44:37.229505+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2022-04-22T17:44:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:38 smithi121 bash[36902]: audit 2022-04-22T17:44:37.230943+0000 mon.a (mon.0) 1991 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:38 smithi115 bash[45495]: cluster 2022-04-22T17:44:37.229505+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2022-04-22T17:44:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:38 smithi115 bash[45495]: audit 2022-04-22T17:44:37.230943+0000 mon.a (mon.0) 1991 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:38.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:38 smithi115 bash[46934]: cluster 2022-04-22T17:44:37.229505+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2022-04-22T17:44:38.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:38 smithi115 bash[46934]: audit 2022-04-22T17:44:37.230943+0000 mon.a (mon.0) 1991 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:39.267 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.test_implicit (2938 ms) 2022-04-22T17:44:39.267 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.test_implicit_idempotent 2022-04-22T17:44:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:39 smithi121 bash[36902]: cluster 2022-04-22T17:44:38.169253+0000 mgr.y (mgr.34107) 688 : cluster [DBG] pgmap v1306: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:39 smithi121 bash[36902]: audit 2022-04-22T17:44:38.243172+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:39 smithi121 bash[36902]: cluster 2022-04-22T17:44:38.243660+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2022-04-22T17:44:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:39 smithi121 bash[36902]: audit 2022-04-22T17:44:38.908511+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[45495]: cluster 2022-04-22T17:44:38.169253+0000 mgr.y (mgr.34107) 688 : cluster [DBG] pgmap v1306: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[45495]: audit 2022-04-22T17:44:38.243172+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[45495]: cluster 2022-04-22T17:44:38.243660+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[45495]: audit 2022-04-22T17:44:38.908511+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[46934]: cluster 2022-04-22T17:44:38.169253+0000 mgr.y (mgr.34107) 688 : cluster [DBG] pgmap v1306: 137 pgs: 9 creating+peering, 23 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[46934]: audit 2022-04-22T17:44:38.243172+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? 172.21.15.115:0/277955289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[46934]: cluster 2022-04-22T17:44:38.243660+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2022-04-22T17:44:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:39 smithi115 bash[46934]: audit 2022-04-22T17:44:38.908511+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:44:40.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:40 smithi121 bash[36902]: cluster 2022-04-22T17:44:39.254299+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2022-04-22T17:44:40.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:40 smithi121 bash[36902]: audit 2022-04-22T17:44:39.737202+0000 mon.a (mon.0) 1996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:40.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:40 smithi121 bash[36902]: audit 2022-04-22T17:44:39.968004+0000 mon.a (mon.0) 1997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:40.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:40 smithi121 bash[36902]: cluster 2022-04-22T17:44:40.170266+0000 mgr.y (mgr.34107) 689 : cluster [DBG] pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[45495]: cluster 2022-04-22T17:44:39.254299+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2022-04-22T17:44:40.651 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[45495]: audit 2022-04-22T17:44:39.737202+0000 mon.a (mon.0) 1996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[45495]: audit 2022-04-22T17:44:39.968004+0000 mon.a (mon.0) 1997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[45495]: cluster 2022-04-22T17:44:40.170266+0000 mgr.y (mgr.34107) 689 : cluster [DBG] pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[46934]: cluster 2022-04-22T17:44:39.254299+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[46934]: audit 2022-04-22T17:44:39.737202+0000 mon.a (mon.0) 1996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[46934]: audit 2022-04-22T17:44:39.968004+0000 mon.a (mon.0) 1997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:40.652 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:40 smithi115 bash[46934]: cluster 2022-04-22T17:44:40.170266+0000 mgr.y (mgr.34107) 689 : cluster [DBG] pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[45495]: cluster 2022-04-22T17:44:40.263815+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2022-04-22T17:44:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[45495]: audit 2022-04-22T17:44:40.269389+0000 mon.c (mon.1) 60 : audit [INF] from='client.? 172.21.15.115:0/147231010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[45495]: audit 2022-04-22T17:44:40.270595+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[46934]: cluster 2022-04-22T17:44:40.263815+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2022-04-22T17:44:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[46934]: audit 2022-04-22T17:44:40.269389+0000 mon.c (mon.1) 60 : audit [INF] from='client.? 172.21.15.115:0/147231010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[46934]: audit 2022-04-22T17:44:40.270595+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:41.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:41 smithi121 bash[36902]: cluster 2022-04-22T17:44:40.263815+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2022-04-22T17:44:41.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:41 smithi121 bash[36902]: audit 2022-04-22T17:44:40.269389+0000 mon.c (mon.1) 60 : audit [INF] from='client.? 172.21.15.115:0/147231010' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:41.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:41 smithi121 bash[36902]: audit 2022-04-22T17:44:40.270595+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:42.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:41.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:42.307 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.test_implicit_idempotent (3039 ms) 2022-04-22T17:44:42.307 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.test_put_snap 2022-04-22T17:44:42.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[46934]: audit 2022-04-22T17:44:41.270862+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:42.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[46934]: cluster 2022-04-22T17:44:41.271235+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2022-04-22T17:44:42.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[46934]: audit 2022-04-22T17:44:42.084341+0000 mon.a (mon.0) 2002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:42.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[46934]: cluster 2022-04-22T17:44:42.171313+0000 mgr.y (mgr.34107) 690 : cluster [DBG] pgmap v1312: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:42.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[45495]: audit 2022-04-22T17:44:41.270862+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:42.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[45495]: cluster 2022-04-22T17:44:41.271235+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2022-04-22T17:44:42.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[45495]: audit 2022-04-22T17:44:42.084341+0000 mon.a (mon.0) 2002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:42.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:42 smithi115 bash[45495]: cluster 2022-04-22T17:44:42.171313+0000 mgr.y (mgr.34107) 690 : cluster [DBG] pgmap v1312: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:42 smithi121 bash[36902]: audit 2022-04-22T17:44:41.270862+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:42 smithi121 bash[36902]: cluster 2022-04-22T17:44:41.271235+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2022-04-22T17:44:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:42 smithi121 bash[36902]: audit 2022-04-22T17:44:42.084341+0000 mon.a (mon.0) 2002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:42 smithi121 bash[36902]: cluster 2022-04-22T17:44:42.171313+0000 mgr.y (mgr.34107) 690 : cluster [DBG] pgmap v1312: 137 pgs: 2 creating+peering, 20 unknown, 115 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:43 smithi115 bash[45495]: cluster 2022-04-22T17:44:42.288960+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2022-04-22T17:44:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:43 smithi115 bash[46934]: cluster 2022-04-22T17:44:42.288960+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2022-04-22T17:44:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:43 smithi121 bash[36902]: cluster 2022-04-22T17:44:42.288960+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2022-04-22T17:44:44.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:43.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:44 smithi121 bash[36902]: cluster 2022-04-22T17:44:43.302695+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2022-04-22T17:44:44.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:44 smithi121 bash[36902]: audit 2022-04-22T17:44:43.307752+0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.115:0/2270034102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:44 smithi121 bash[36902]: audit 2022-04-22T17:44:43.310894+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:44 smithi121 bash[36902]: cluster 2022-04-22T17:44:44.172215+0000 mgr.y (mgr.34107) 691 : cluster [DBG] pgmap v1315: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[45495]: cluster 2022-04-22T17:44:43.302695+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2022-04-22T17:44:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[45495]: audit 2022-04-22T17:44:43.307752+0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.115:0/2270034102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[45495]: audit 2022-04-22T17:44:43.310894+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[45495]: cluster 2022-04-22T17:44:44.172215+0000 mgr.y (mgr.34107) 691 : cluster [DBG] pgmap v1315: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[46934]: cluster 2022-04-22T17:44:43.302695+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2022-04-22T17:44:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[46934]: audit 2022-04-22T17:44:43.307752+0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.115:0/2270034102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[46934]: audit 2022-04-22T17:44:43.310894+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:44 smithi115 bash[46934]: cluster 2022-04-22T17:44:44.172215+0000 mgr.y (mgr.34107) 691 : cluster [DBG] pgmap v1315: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:45 smithi121 bash[36902]: audit 2022-04-22T17:44:44.314336+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:45 smithi121 bash[36902]: cluster 2022-04-22T17:44:44.314538+0000 mon.a (mon.0) 2007 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2022-04-22T17:44:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:45 smithi121 bash[36902]: audit 2022-04-22T17:44:44.702988+0000 mon.a (mon.0) 2008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:45 smithi121 bash[36902]: audit 2022-04-22T17:44:44.971641+0000 mon.a (mon.0) 2009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[45495]: audit 2022-04-22T17:44:44.314336+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[45495]: cluster 2022-04-22T17:44:44.314538+0000 mon.a (mon.0) 2007 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2022-04-22T17:44:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[45495]: audit 2022-04-22T17:44:44.702988+0000 mon.a (mon.0) 2008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[45495]: audit 2022-04-22T17:44:44.971641+0000 mon.a (mon.0) 2009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[46934]: audit 2022-04-22T17:44:44.314336+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[46934]: cluster 2022-04-22T17:44:44.314538+0000 mon.a (mon.0) 2007 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2022-04-22T17:44:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[46934]: audit 2022-04-22T17:44:44.702988+0000 mon.a (mon.0) 2008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:45 smithi115 bash[46934]: audit 2022-04-22T17:44:44.971641+0000 mon.a (mon.0) 2009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:46.652 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:44:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:46 smithi121 bash[36902]: cluster 2022-04-22T17:44:45.319658+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2022-04-22T17:44:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:46 smithi121 bash[36902]: cluster 2022-04-22T17:44:46.173215+0000 mgr.y (mgr.34107) 692 : cluster [DBG] pgmap v1318: 137 pgs: 4 creating+peering, 11 unknown, 122 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:46 smithi115 bash[46934]: cluster 2022-04-22T17:44:45.319658+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2022-04-22T17:44:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:46 smithi115 bash[46934]: cluster 2022-04-22T17:44:46.173215+0000 mgr.y (mgr.34107) 692 : cluster [DBG] pgmap v1318: 137 pgs: 4 creating+peering, 11 unknown, 122 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:46.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:46 smithi115 bash[45495]: cluster 2022-04-22T17:44:45.319658+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2022-04-22T17:44:46.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:46 smithi115 bash[45495]: cluster 2022-04-22T17:44:46.173215+0000 mgr.y (mgr.34107) 692 : cluster [DBG] pgmap v1318: 137 pgs: 4 creating+peering, 11 unknown, 122 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:47.003 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:44:47.018 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (7m) 7m ago 21m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:44:47.018 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (21m) 8m ago 21m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:44:47.018 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (9m) 7m ago 22m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:44:47.018 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (9m) 8m ago 22m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:44:47.019 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (17m) 8m ago 26m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:44:47.019 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (17m) 7m ago 29m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (16m) 7m ago 29m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (15m) 8m ago 27m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (16m) 7m ago 27m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (8m) 7m ago 21m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (8m) 8m ago 21m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (14m) 7m ago 26m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:44:47.020 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (13m) 7m ago 26m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (13m) 7m ago 25m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (12m) 7m ago 25m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (12m) 8m ago 24m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (11m) 8m ago 24m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (10m) 8m ago 23m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:44:47.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (10m) 8m ago 23m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:44:47.022 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (8m) 8m ago 21m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:44:47.413 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:44:47.439 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:47.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:44:47.441 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:44:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:47 smithi121 bash[36902]: audit 2022-04-22T17:44:46.625548+0000 mgr.y (mgr.34107) 693 : audit [DBG] from='client.45034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:47 smithi121 bash[36902]: audit 2022-04-22T17:44:46.974160+0000 mgr.y (mgr.34107) 694 : audit [DBG] from='client.54899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:47 smithi121 bash[36902]: audit 2022-04-22T17:44:47.087428+0000 mon.a (mon.0) 2011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[45495]: audit 2022-04-22T17:44:46.625548+0000 mgr.y (mgr.34107) 693 : audit [DBG] from='client.45034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[45495]: audit 2022-04-22T17:44:46.974160+0000 mgr.y (mgr.34107) 694 : audit [DBG] from='client.54899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[45495]: audit 2022-04-22T17:44:47.087428+0000 mon.a (mon.0) 2011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:47.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[46934]: audit 2022-04-22T17:44:46.625548+0000 mgr.y (mgr.34107) 693 : audit [DBG] from='client.45034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[46934]: audit 2022-04-22T17:44:46.974160+0000 mgr.y (mgr.34107) 694 : audit [DBG] from='client.54899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:44:47.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:47 smithi115 bash[46934]: audit 2022-04-22T17:44:47.087428+0000 mon.a (mon.0) 2011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:48 smithi121 bash[36902]: cluster 2022-04-22T17:44:47.388248+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2022-04-22T17:44:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:48 smithi121 bash[36902]: audit 2022-04-22T17:44:47.402796+0000 mon.c (mon.1) 62 : audit [DBG] from='client.? 172.21.15.115:0/1648533595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:48 smithi121 bash[36902]: cluster 2022-04-22T17:44:48.174256+0000 mgr.y (mgr.34107) 695 : cluster [DBG] pgmap v1320: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[45495]: cluster 2022-04-22T17:44:47.388248+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2022-04-22T17:44:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[45495]: audit 2022-04-22T17:44:47.402796+0000 mon.c (mon.1) 62 : audit [DBG] from='client.? 172.21.15.115:0/1648533595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[45495]: cluster 2022-04-22T17:44:48.174256+0000 mgr.y (mgr.34107) 695 : cluster [DBG] pgmap v1320: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[46934]: cluster 2022-04-22T17:44:47.388248+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2022-04-22T17:44:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[46934]: audit 2022-04-22T17:44:47.402796+0000 mon.c (mon.1) 62 : audit [DBG] from='client.? 172.21.15.115:0/1648533595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:44:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:48 smithi115 bash[46934]: cluster 2022-04-22T17:44:48.174256+0000 mgr.y (mgr.34107) 695 : cluster [DBG] pgmap v1320: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:49 smithi121 bash[36902]: cluster 2022-04-22T17:44:48.398421+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2022-04-22T17:44:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:49 smithi115 bash[45495]: cluster 2022-04-22T17:44:48.398421+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2022-04-22T17:44:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:49 smithi115 bash[46934]: cluster 2022-04-22T17:44:48.398421+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2022-04-22T17:44:50.435 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.test_put_snap (8128 ms) 2022-04-22T17:44:50.435 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.test_explicit 2022-04-22T17:44:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:50 smithi121 bash[36902]: cluster 2022-04-22T17:44:49.406176+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2022-04-22T17:44:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:50 smithi121 bash[36902]: audit 2022-04-22T17:44:49.706313+0000 mon.a (mon.0) 2015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:50 smithi121 bash[36902]: audit 2022-04-22T17:44:49.975320+0000 mon.a (mon.0) 2016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:50 smithi121 bash[36902]: cluster 2022-04-22T17:44:50.175314+0000 mgr.y (mgr.34107) 696 : cluster [DBG] pgmap v1323: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2022-04-22T17:44:50.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[45495]: cluster 2022-04-22T17:44:49.406176+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2022-04-22T17:44:50.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[45495]: audit 2022-04-22T17:44:49.706313+0000 mon.a (mon.0) 2015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:50.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[45495]: audit 2022-04-22T17:44:49.975320+0000 mon.a (mon.0) 2016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:50.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[45495]: cluster 2022-04-22T17:44:50.175314+0000 mgr.y (mgr.34107) 696 : cluster [DBG] pgmap v1323: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2022-04-22T17:44:50.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[46934]: cluster 2022-04-22T17:44:49.406176+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2022-04-22T17:44:50.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[46934]: audit 2022-04-22T17:44:49.706313+0000 mon.a (mon.0) 2015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:50.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[46934]: audit 2022-04-22T17:44:49.975320+0000 mon.a (mon.0) 2016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:50.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:50 smithi115 bash[46934]: cluster 2022-04-22T17:44:50.175314+0000 mgr.y (mgr.34107) 696 : cluster [DBG] pgmap v1323: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2022-04-22T17:44:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:51 smithi121 bash[36902]: cluster 2022-04-22T17:44:50.415063+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2022-04-22T17:44:51.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:51.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:51 smithi115 bash[45495]: cluster 2022-04-22T17:44:50.415063+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2022-04-22T17:44:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:51 smithi115 bash[46934]: cluster 2022-04-22T17:44:50.415063+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2022-04-22T17:44:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:52 smithi121 bash[36902]: cluster 2022-04-22T17:44:51.426979+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2022-04-22T17:44:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:52 smithi121 bash[36902]: audit 2022-04-22T17:44:51.439291+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.115:0/1746828241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:52 smithi121 bash[36902]: audit 2022-04-22T17:44:51.443497+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:52 smithi121 bash[36902]: audit 2022-04-22T17:44:52.090258+0000 mon.a (mon.0) 2020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:52 smithi121 bash[36902]: cluster 2022-04-22T17:44:52.176284+0000 mgr.y (mgr.34107) 697 : cluster [DBG] pgmap v1326: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:52.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[45495]: cluster 2022-04-22T17:44:51.426979+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2022-04-22T17:44:52.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[45495]: audit 2022-04-22T17:44:51.439291+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.115:0/1746828241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[45495]: audit 2022-04-22T17:44:51.443497+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[45495]: audit 2022-04-22T17:44:52.090258+0000 mon.a (mon.0) 2020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:52.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[45495]: cluster 2022-04-22T17:44:52.176284+0000 mgr.y (mgr.34107) 697 : cluster [DBG] pgmap v1326: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[46934]: cluster 2022-04-22T17:44:51.426979+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2022-04-22T17:44:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[46934]: audit 2022-04-22T17:44:51.439291+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.115:0/1746828241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[46934]: audit 2022-04-22T17:44:51.443497+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[46934]: audit 2022-04-22T17:44:52.090258+0000 mon.a (mon.0) 2020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:52 smithi115 bash[46934]: cluster 2022-04-22T17:44:52.176284+0000 mgr.y (mgr.34107) 697 : cluster [DBG] pgmap v1326: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:53.464 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.test_explicit (3029 ms) 2022-04-22T17:44:53.464 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.set 2022-04-22T17:44:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:53 smithi121 bash[36902]: audit 2022-04-22T17:44:52.442714+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:53 smithi121 bash[36902]: cluster 2022-04-22T17:44:52.443178+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2022-04-22T17:44:53.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:44:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:44:53.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:44:53.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:53 smithi115 bash[45495]: audit 2022-04-22T17:44:52.442714+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:53.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:53 smithi115 bash[45495]: cluster 2022-04-22T17:44:52.443178+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2022-04-22T17:44:53.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:53 smithi115 bash[46934]: audit 2022-04-22T17:44:52.442714+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:53.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:53 smithi115 bash[46934]: cluster 2022-04-22T17:44:52.443178+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2022-04-22T17:44:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:54 smithi121 bash[36902]: cluster 2022-04-22T17:44:53.448641+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2022-04-22T17:44:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:54 smithi121 bash[36902]: cluster 2022-04-22T17:44:54.177111+0000 mgr.y (mgr.34107) 698 : cluster [DBG] pgmap v1329: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:54 smithi115 bash[46934]: cluster 2022-04-22T17:44:53.448641+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2022-04-22T17:44:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:54 smithi115 bash[46934]: cluster 2022-04-22T17:44:54.177111+0000 mgr.y (mgr.34107) 698 : cluster [DBG] pgmap v1329: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:54.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:54 smithi115 bash[45495]: cluster 2022-04-22T17:44:53.448641+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2022-04-22T17:44:54.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:54 smithi115 bash[45495]: cluster 2022-04-22T17:44:54.177111+0000 mgr.y (mgr.34107) 698 : cluster [DBG] pgmap v1329: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:55 smithi121 bash[36902]: cluster 2022-04-22T17:44:54.467749+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2022-04-22T17:44:55.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:55 smithi121 bash[36902]: audit 2022-04-22T17:44:54.475232+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:55.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:55 smithi121 bash[36902]: audit 2022-04-22T17:44:54.709448+0000 mon.a (mon.0) 2026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:55.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:55 smithi121 bash[36902]: audit 2022-04-22T17:44:54.978539+0000 mon.a (mon.0) 2027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:55.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[45495]: cluster 2022-04-22T17:44:54.467749+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2022-04-22T17:44:55.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[45495]: audit 2022-04-22T17:44:54.475232+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:55.862 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[45495]: audit 2022-04-22T17:44:54.709448+0000 mon.a (mon.0) 2026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:55.863 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[45495]: audit 2022-04-22T17:44:54.978539+0000 mon.a (mon.0) 2027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[46934]: cluster 2022-04-22T17:44:54.467749+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2022-04-22T17:44:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[46934]: audit 2022-04-22T17:44:54.475232+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[46934]: audit 2022-04-22T17:44:54.709448+0000 mon.a (mon.0) 2026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:44:55.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:55 smithi115 bash[46934]: audit 2022-04-22T17:44:54.978539+0000 mon.a (mon.0) 2027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.set (3040 ms) 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 5 tests from cls_rgw (20175 ms total) 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 5 tests from 1 test suite ran. (20175 ms total) 2022-04-22T17:44:56.505 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 5 tests. 2022-04-22T17:44:56.509 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:44:56.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:44:56.521 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2022-04-22T17:44:56.521 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2022-04-22T17:44:56.590 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:44:56.590 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 14 tests from 1 test suite. 2022-04-22T17:44:56.590 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:44:56.591 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 14 tests from cls_rgw 2022-04-22T17:44:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:56 smithi121 bash[36902]: audit 2022-04-22T17:44:55.473557+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:56.849 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:56 smithi121 bash[36902]: cluster 2022-04-22T17:44:55.473727+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2022-04-22T17:44:56.849 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:56 smithi121 bash[36902]: cluster 2022-04-22T17:44:56.178117+0000 mgr.y (mgr.34107) 699 : cluster [DBG] pgmap v1332: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[45495]: audit 2022-04-22T17:44:55.473557+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[45495]: cluster 2022-04-22T17:44:55.473727+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2022-04-22T17:44:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[45495]: cluster 2022-04-22T17:44:56.178117+0000 mgr.y (mgr.34107) 699 : cluster [DBG] pgmap v1332: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:56.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[46934]: audit 2022-04-22T17:44:55.473557+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? 172.21.15.115:0/1390306158' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64082-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:56.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[46934]: cluster 2022-04-22T17:44:55.473727+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2022-04-22T17:44:56.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:56 smithi115 bash[46934]: cluster 2022-04-22T17:44:56.178117+0000 mgr.y (mgr.34107) 699 : cluster [DBG] pgmap v1332: 137 pgs: 10 creating+peering, 19 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:57 smithi121 bash[36902]: cluster 2022-04-22T17:44:56.490809+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2022-04-22T17:44:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:57 smithi121 bash[36902]: audit 2022-04-22T17:44:57.092528+0000 mon.a (mon.0) 2031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:57 smithi115 bash[45495]: cluster 2022-04-22T17:44:56.490809+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2022-04-22T17:44:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:57 smithi115 bash[45495]: audit 2022-04-22T17:44:57.092528+0000 mon.a (mon.0) 2031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:57 smithi115 bash[46934]: cluster 2022-04-22T17:44:56.490809+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2022-04-22T17:44:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:57 smithi115 bash[46934]: audit 2022-04-22T17:44:57.092528+0000 mon.a (mon.0) 2031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:44:58.516 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_basic 2022-04-22T17:44:58.573 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_basic (57 ms) 2022-04-22T17:44:58.573 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2022-04-22T17:44:58.626 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (53 ms) 2022-04-22T17:44:58.626 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_remove_object 2022-04-22T17:44:58.701 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_remove_object (75 ms) 2022-04-22T17:44:58.701 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_suggest 2022-04-22T17:44:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[46934]: cluster 2022-04-22T17:44:57.498555+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2022-04-22T17:44:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[46934]: audit 2022-04-22T17:44:57.506881+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[46934]: cluster 2022-04-22T17:44:58.179116+0000 mgr.y (mgr.34107) 700 : cluster [DBG] pgmap v1335: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[45495]: cluster 2022-04-22T17:44:57.498555+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2022-04-22T17:44:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[45495]: audit 2022-04-22T17:44:57.506881+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:58 smithi115 bash[45495]: cluster 2022-04-22T17:44:58.179116+0000 mgr.y (mgr.34107) 700 : cluster [DBG] pgmap v1335: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:58 smithi121 bash[36902]: cluster 2022-04-22T17:44:57.498555+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2022-04-22T17:44:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:58 smithi121 bash[36902]: audit 2022-04-22T17:44:57.506881+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:44:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:58 smithi121 bash[36902]: cluster 2022-04-22T17:44:58.179116+0000 mgr.y (mgr.34107) 700 : cluster [DBG] pgmap v1335: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:44:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[45495]: audit 2022-04-22T17:44:58.502695+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[45495]: cluster 2022-04-22T17:44:58.502938+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2022-04-22T17:44:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[45495]: cluster 2022-04-22T17:44:59.518967+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2022-04-22T17:44:59.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[46934]: audit 2022-04-22T17:44:58.502695+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:44:59.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[46934]: cluster 2022-04-22T17:44:58.502938+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2022-04-22T17:44:59.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:44:59 smithi115 bash[46934]: cluster 2022-04-22T17:44:59.518967+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2022-04-22T17:45:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:59 smithi121 bash[36902]: audit 2022-04-22T17:44:58.502695+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? 172.21.15.115:0/1717061597' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:45:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:59 smithi121 bash[36902]: cluster 2022-04-22T17:44:58.502938+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2022-04-22T17:45:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:44:59 smithi121 bash[36902]: cluster 2022-04-22T17:44:59.518967+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2022-04-22T17:45:00.670 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_suggest (1969 ms) 2022-04-22T17:45:00.670 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_list 2022-04-22T17:45:00.705 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_list (35 ms) 2022-04-22T17:45:00.706 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.index_list_delimited 2022-04-22T17:45:00.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[46934]: audit 2022-04-22T17:44:59.712751+0000 mon.a (mon.0) 2037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:00.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[46934]: audit 2022-04-22T17:44:59.982116+0000 mon.a (mon.0) 2038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[46934]: cluster 2022-04-22T17:45:00.180231+0000 mgr.y (mgr.34107) 701 : cluster [DBG] pgmap v1338: 137 pgs: 4 creating+peering, 17 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 126 KiB/s wr, 396 op/s 2022-04-22T17:45:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[45495]: audit 2022-04-22T17:44:59.712751+0000 mon.a (mon.0) 2037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[45495]: audit 2022-04-22T17:44:59.982116+0000 mon.a (mon.0) 2038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:00.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:00 smithi115 bash[45495]: cluster 2022-04-22T17:45:00.180231+0000 mgr.y (mgr.34107) 701 : cluster [DBG] pgmap v1338: 137 pgs: 4 creating+peering, 17 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 126 KiB/s wr, 396 op/s 2022-04-22T17:45:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:00 smithi121 bash[36902]: audit 2022-04-22T17:44:59.712751+0000 mon.a (mon.0) 2037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:00 smithi121 bash[36902]: audit 2022-04-22T17:44:59.982116+0000 mon.a (mon.0) 2038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:00 smithi121 bash[36902]: cluster 2022-04-22T17:45:00.180231+0000 mgr.y (mgr.34107) 701 : cluster [DBG] pgmap v1338: 137 pgs: 4 creating+peering, 17 unknown, 116 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 126 KiB/s wr, 396 op/s 2022-04-22T17:45:02.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:01.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:02 smithi121 bash[36902]: audit 2022-04-22T17:45:02.095069+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:02.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:02 smithi115 bash[45495]: audit 2022-04-22T17:45:02.095069+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:02.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:02 smithi115 bash[46934]: audit 2022-04-22T17:45:02.095069+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:03 smithi121 bash[36902]: cluster 2022-04-22T17:45:02.181499+0000 mgr.y (mgr.34107) 702 : cluster [DBG] pgmap v1339: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 130 KiB/s wr, 398 op/s 2022-04-22T17:45:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:03 smithi115 bash[45495]: cluster 2022-04-22T17:45:02.181499+0000 mgr.y (mgr.34107) 702 : cluster [DBG] pgmap v1339: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 130 KiB/s wr, 398 op/s 2022-04-22T17:45:03.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:03 smithi115 bash[46934]: cluster 2022-04-22T17:45:02.181499+0000 mgr.y (mgr.34107) 702 : cluster [DBG] pgmap v1339: 137 pgs: 4 creating+peering, 133 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 130 KiB/s wr, 398 op/s 2022-04-22T17:45:04.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:03.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:05 smithi121 bash[36902]: cluster 2022-04-22T17:45:04.182651+0000 mgr.y (mgr.34107) 703 : cluster [DBG] pgmap v1340: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 189 KiB/s rd, 111 KiB/s wr, 339 op/s 2022-04-22T17:45:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:05 smithi121 bash[36902]: audit 2022-04-22T17:45:04.715815+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:05 smithi121 bash[36902]: audit 2022-04-22T17:45:04.985622+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[45495]: cluster 2022-04-22T17:45:04.182651+0000 mgr.y (mgr.34107) 703 : cluster [DBG] pgmap v1340: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 189 KiB/s rd, 111 KiB/s wr, 339 op/s 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[45495]: audit 2022-04-22T17:45:04.715815+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[45495]: audit 2022-04-22T17:45:04.985622+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[46934]: cluster 2022-04-22T17:45:04.182651+0000 mgr.y (mgr.34107) 703 : cluster [DBG] pgmap v1340: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 189 KiB/s rd, 111 KiB/s wr, 339 op/s 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[46934]: audit 2022-04-22T17:45:04.715815+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:05 smithi115 bash[46934]: audit 2022-04-22T17:45:04.985622+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:07 smithi121 bash[36902]: cluster 2022-04-22T17:45:06.183546+0000 mgr.y (mgr.34107) 704 : cluster [DBG] pgmap v1341: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 580 KiB/s rd, 365 KiB/s wr, 1.11k op/s 2022-04-22T17:45:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:07 smithi121 bash[36902]: audit 2022-04-22T17:45:07.098007+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:07.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:07 smithi115 bash[45495]: cluster 2022-04-22T17:45:06.183546+0000 mgr.y (mgr.34107) 704 : cluster [DBG] pgmap v1341: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 580 KiB/s rd, 365 KiB/s wr, 1.11k op/s 2022-04-22T17:45:07.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:07 smithi115 bash[45495]: audit 2022-04-22T17:45:07.098007+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:07.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:07 smithi115 bash[46934]: cluster 2022-04-22T17:45:06.183546+0000 mgr.y (mgr.34107) 704 : cluster [DBG] pgmap v1341: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 580 KiB/s rd, 365 KiB/s wr, 1.11k op/s 2022-04-22T17:45:07.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:07 smithi115 bash[46934]: audit 2022-04-22T17:45:07.098007+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:09 smithi121 bash[36902]: cluster 2022-04-22T17:45:08.184337+0000 mgr.y (mgr.34107) 705 : cluster [DBG] pgmap v1342: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 479 KiB/s rd, 302 KiB/s wr, 921 op/s 2022-04-22T17:45:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:09 smithi115 bash[45495]: cluster 2022-04-22T17:45:08.184337+0000 mgr.y (mgr.34107) 705 : cluster [DBG] pgmap v1342: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 479 KiB/s rd, 302 KiB/s wr, 921 op/s 2022-04-22T17:45:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:09 smithi115 bash[46934]: cluster 2022-04-22T17:45:08.184337+0000 mgr.y (mgr.34107) 705 : cluster [DBG] pgmap v1342: 137 pgs: 137 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 479 KiB/s rd, 302 KiB/s wr, 921 op/s 2022-04-22T17:45:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:10 smithi121 bash[36902]: audit 2022-04-22T17:45:09.718616+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:10 smithi121 bash[36902]: audit 2022-04-22T17:45:09.988734+0000 mon.a (mon.0) 2044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:10 smithi115 bash[45495]: audit 2022-04-22T17:45:09.718616+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:10.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:10 smithi115 bash[45495]: audit 2022-04-22T17:45:09.988734+0000 mon.a (mon.0) 2044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:10.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:10 smithi115 bash[46934]: audit 2022-04-22T17:45:09.718616+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:10.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:10 smithi115 bash[46934]: audit 2022-04-22T17:45:09.988734+0000 mon.a (mon.0) 2044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:11 smithi115 bash[45495]: cluster 2022-04-22T17:45:10.185315+0000 mgr.y (mgr.34107) 706 : cluster [DBG] pgmap v1343: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 271 KiB/s wr, 828 op/s 2022-04-22T17:45:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:11 smithi115 bash[46934]: cluster 2022-04-22T17:45:10.185315+0000 mgr.y (mgr.34107) 706 : cluster [DBG] pgmap v1343: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 271 KiB/s wr, 828 op/s 2022-04-22T17:45:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:11 smithi121 bash[36902]: cluster 2022-04-22T17:45:10.185315+0000 mgr.y (mgr.34107) 706 : cluster [DBG] pgmap v1343: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 271 KiB/s wr, 828 op/s 2022-04-22T17:45:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:11.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:12.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:12 smithi115 bash[45495]: audit 2022-04-22T17:45:12.100410+0000 mon.a (mon.0) 2045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:12.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:12 smithi115 bash[45495]: cluster 2022-04-22T17:45:12.186442+0000 mgr.y (mgr.34107) 707 : cluster [DBG] pgmap v1344: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 521 KiB/s rd, 340 KiB/s wr, 1.03k op/s 2022-04-22T17:45:12.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:12 smithi115 bash[46934]: audit 2022-04-22T17:45:12.100410+0000 mon.a (mon.0) 2045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:12.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:12 smithi115 bash[46934]: cluster 2022-04-22T17:45:12.186442+0000 mgr.y (mgr.34107) 707 : cluster [DBG] pgmap v1344: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 521 KiB/s rd, 340 KiB/s wr, 1.03k op/s 2022-04-22T17:45:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:12 smithi121 bash[36902]: audit 2022-04-22T17:45:12.100410+0000 mon.a (mon.0) 2045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:12 smithi121 bash[36902]: cluster 2022-04-22T17:45:12.186442+0000 mgr.y (mgr.34107) 707 : cluster [DBG] pgmap v1344: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 521 KiB/s rd, 340 KiB/s wr, 1.03k op/s 2022-04-22T17:45:13.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:13 smithi121 bash[36902]: audit 2022-04-22T17:45:13.407389+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:45:13.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:13 smithi121 bash[36902]: audit 2022-04-22T17:45:13.407777+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:45:13.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:13.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:13 smithi115 bash[46934]: audit 2022-04-22T17:45:13.407389+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:45:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:13 smithi115 bash[46934]: audit 2022-04-22T17:45:13.407777+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:45:13.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:13 smithi115 bash[45495]: audit 2022-04-22T17:45:13.407389+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:45:13.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:13 smithi115 bash[45495]: audit 2022-04-22T17:45:13.407777+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:45:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:14 smithi121 bash[36902]: audit 2022-04-22T17:45:13.933916+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:14 smithi121 bash[36902]: cluster 2022-04-22T17:45:14.187324+0000 mgr.y (mgr.34107) 708 : cluster [DBG] pgmap v1345: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 490 KiB/s rd, 321 KiB/s wr, 970 op/s 2022-04-22T17:45:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:14 smithi121 bash[36902]: audit 2022-04-22T17:45:14.721220+0000 mon.a (mon.0) 2049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[45495]: audit 2022-04-22T17:45:13.933916+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[45495]: cluster 2022-04-22T17:45:14.187324+0000 mgr.y (mgr.34107) 708 : cluster [DBG] pgmap v1345: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 490 KiB/s rd, 321 KiB/s wr, 970 op/s 2022-04-22T17:45:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[45495]: audit 2022-04-22T17:45:14.721220+0000 mon.a (mon.0) 2049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[46934]: audit 2022-04-22T17:45:13.933916+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[46934]: cluster 2022-04-22T17:45:14.187324+0000 mgr.y (mgr.34107) 708 : cluster [DBG] pgmap v1345: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 490 KiB/s rd, 321 KiB/s wr, 970 op/s 2022-04-22T17:45:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:14 smithi115 bash[46934]: audit 2022-04-22T17:45:14.721220+0000 mon.a (mon.0) 2049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:15 smithi121 bash[36902]: audit 2022-04-22T17:45:14.991644+0000 mon.a (mon.0) 2050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:15 smithi115 bash[45495]: audit 2022-04-22T17:45:14.991644+0000 mon.a (mon.0) 2050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:16.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:15 smithi115 bash[46934]: audit 2022-04-22T17:45:14.991644+0000 mon.a (mon.0) 2050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:16 smithi121 bash[36902]: cluster 2022-04-22T17:45:16.188375+0000 mgr.y (mgr.34107) 709 : cluster [DBG] pgmap v1346: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 437 KiB/s wr, 1.32k op/s 2022-04-22T17:45:17.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:16 smithi115 bash[45495]: cluster 2022-04-22T17:45:16.188375+0000 mgr.y (mgr.34107) 709 : cluster [DBG] pgmap v1346: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 437 KiB/s wr, 1.32k op/s 2022-04-22T17:45:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:16 smithi115 bash[46934]: cluster 2022-04-22T17:45:16.188375+0000 mgr.y (mgr.34107) 709 : cluster [DBG] pgmap v1346: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 664 KiB/s rd, 437 KiB/s wr, 1.32k op/s 2022-04-22T17:45:17.801 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:45:18.170 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (8m) 8m ago 22m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (21m) 8m ago 21m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (10m) 8m ago 23m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (9m) 8m ago 23m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (17m) 8m ago 27m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (18m) 8m ago 29m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (17m) 8m ago 29m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (16m) 8m ago 27m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (16m) 8m ago 27m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:45:18.171 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (9m) 8m ago 22m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (9m) 8m ago 22m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (15m) 8m ago 27m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (14m) 8m ago 26m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (13m) 8m ago 26m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (13m) 8m ago 25m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (12m) 8m ago 25m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:45:18.172 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (12m) 8m ago 24m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:45:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (11m) 8m ago 24m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:45:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (10m) 8m ago 23m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:45:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (9m) 8m ago 22m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:45:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:17 smithi121 bash[36902]: audit 2022-04-22T17:45:17.102818+0000 mon.a (mon.0) 2051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:17 smithi115 bash[46934]: audit 2022-04-22T17:45:17.102818+0000 mon.a (mon.0) 2051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:17 smithi115 bash[45495]: audit 2022-04-22T17:45:17.102818+0000 mon.a (mon.0) 2051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:18.589 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:45:18.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:45:18.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:45:18.625 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:45:18.625 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:45:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:18 smithi121 bash[36902]: audit 2022-04-22T17:45:17.776174+0000 mgr.y (mgr.34107) 710 : audit [DBG] from='client.45070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:18 smithi121 bash[36902]: audit 2022-04-22T17:45:18.139064+0000 mgr.y (mgr.34107) 711 : audit [DBG] from='client.45076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:18 smithi121 bash[36902]: cluster 2022-04-22T17:45:18.189172+0000 mgr.y (mgr.34107) 712 : cluster [DBG] pgmap v1347: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:19.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:18 smithi121 bash[36902]: audit 2022-04-22T17:45:18.577002+0000 mon.a (mon.0) 2052 : audit [DBG] from='client.? 172.21.15.115:0/3289231038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[46934]: audit 2022-04-22T17:45:17.776174+0000 mgr.y (mgr.34107) 710 : audit [DBG] from='client.45070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[46934]: audit 2022-04-22T17:45:18.139064+0000 mgr.y (mgr.34107) 711 : audit [DBG] from='client.45076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[46934]: cluster 2022-04-22T17:45:18.189172+0000 mgr.y (mgr.34107) 712 : cluster [DBG] pgmap v1347: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:19.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[46934]: audit 2022-04-22T17:45:18.577002+0000 mon.a (mon.0) 2052 : audit [DBG] from='client.? 172.21.15.115:0/3289231038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[45495]: audit 2022-04-22T17:45:17.776174+0000 mgr.y (mgr.34107) 710 : audit [DBG] from='client.45070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[45495]: audit 2022-04-22T17:45:18.139064+0000 mgr.y (mgr.34107) 711 : audit [DBG] from='client.45076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[45495]: cluster 2022-04-22T17:45:18.189172+0000 mgr.y (mgr.34107) 712 : cluster [DBG] pgmap v1347: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:19.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:18 smithi115 bash[45495]: audit 2022-04-22T17:45:18.577002+0000 mon.a (mon.0) 2052 : audit [DBG] from='client.? 172.21.15.115:0/3289231038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:19 smithi121 bash[36902]: audit 2022-04-22T17:45:19.724352+0000 mon.a (mon.0) 2053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:20.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:19 smithi115 bash[45495]: audit 2022-04-22T17:45:19.724352+0000 mon.a (mon.0) 2053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:20.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:19 smithi115 bash[46934]: audit 2022-04-22T17:45:19.724352+0000 mon.a (mon.0) 2053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:20 smithi121 bash[36902]: audit 2022-04-22T17:45:19.994608+0000 mon.a (mon.0) 2054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:20 smithi121 bash[36902]: cluster 2022-04-22T17:45:20.190097+0000 mgr.y (mgr.34107) 713 : cluster [DBG] pgmap v1348: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:20 smithi115 bash[46934]: audit 2022-04-22T17:45:19.994608+0000 mon.a (mon.0) 2054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:20 smithi115 bash[46934]: cluster 2022-04-22T17:45:20.190097+0000 mgr.y (mgr.34107) 713 : cluster [DBG] pgmap v1348: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:20 smithi115 bash[45495]: audit 2022-04-22T17:45:19.994608+0000 mon.a (mon.0) 2054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:20 smithi115 bash[45495]: cluster 2022-04-22T17:45:20.190097+0000 mgr.y (mgr.34107) 713 : cluster [DBG] pgmap v1348: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 KiB/s rd, 255 KiB/s wr, 765 op/s 2022-04-22T17:45:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:21.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:22 smithi121 bash[36902]: audit 2022-04-22T17:45:22.105663+0000 mon.a (mon.0) 2055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:22.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:22 smithi115 bash[46934]: audit 2022-04-22T17:45:22.105663+0000 mon.a (mon.0) 2055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:22.561 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:22 smithi115 bash[45495]: audit 2022-04-22T17:45:22.105663+0000 mon.a (mon.0) 2055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:23 smithi121 bash[36902]: cluster 2022-04-22T17:45:22.191079+0000 mgr.y (mgr.34107) 714 : cluster [DBG] pgmap v1349: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 535 KiB/s rd, 356 KiB/s wr, 1.07k op/s 2022-04-22T17:45:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:23 smithi115 bash[45495]: cluster 2022-04-22T17:45:22.191079+0000 mgr.y (mgr.34107) 714 : cluster [DBG] pgmap v1349: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 535 KiB/s rd, 356 KiB/s wr, 1.07k op/s 2022-04-22T17:45:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:23 smithi115 bash[46934]: cluster 2022-04-22T17:45:22.191079+0000 mgr.y (mgr.34107) 714 : cluster [DBG] pgmap v1349: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 535 KiB/s rd, 356 KiB/s wr, 1.07k op/s 2022-04-22T17:45:24.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:23.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:25 smithi121 bash[36902]: cluster 2022-04-22T17:45:24.192058+0000 mgr.y (mgr.34107) 715 : cluster [DBG] pgmap v1350: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 217 KiB/s wr, 652 op/s 2022-04-22T17:45:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:25 smithi121 bash[36902]: audit 2022-04-22T17:45:24.727075+0000 mon.a (mon.0) 2056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:25.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:25 smithi121 bash[36902]: audit 2022-04-22T17:45:24.997868+0000 mon.a (mon.0) 2057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[45495]: cluster 2022-04-22T17:45:24.192058+0000 mgr.y (mgr.34107) 715 : cluster [DBG] pgmap v1350: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 217 KiB/s wr, 652 op/s 2022-04-22T17:45:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[45495]: audit 2022-04-22T17:45:24.727075+0000 mon.a (mon.0) 2056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[45495]: audit 2022-04-22T17:45:24.997868+0000 mon.a (mon.0) 2057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[46934]: cluster 2022-04-22T17:45:24.192058+0000 mgr.y (mgr.34107) 715 : cluster [DBG] pgmap v1350: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 217 KiB/s wr, 652 op/s 2022-04-22T17:45:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[46934]: audit 2022-04-22T17:45:24.727075+0000 mon.a (mon.0) 2056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:25 smithi115 bash[46934]: audit 2022-04-22T17:45:24.997868+0000 mon.a (mon.0) 2057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:27 smithi121 bash[36902]: cluster 2022-04-22T17:45:26.193133+0000 mgr.y (mgr.34107) 716 : cluster [DBG] pgmap v1351: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 473 KiB/s rd, 315 KiB/s wr, 945 op/s 2022-04-22T17:45:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:27 smithi121 bash[36902]: audit 2022-04-22T17:45:27.107926+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:27 smithi121 bash[36902]: audit 2022-04-22T17:45:27.247370+0000 mon.a (mon.0) 2059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:45:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[45495]: cluster 2022-04-22T17:45:26.193133+0000 mgr.y (mgr.34107) 716 : cluster [DBG] pgmap v1351: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 473 KiB/s rd, 315 KiB/s wr, 945 op/s 2022-04-22T17:45:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[45495]: audit 2022-04-22T17:45:27.107926+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[45495]: audit 2022-04-22T17:45:27.247370+0000 mon.a (mon.0) 2059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:45:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[46934]: cluster 2022-04-22T17:45:26.193133+0000 mgr.y (mgr.34107) 716 : cluster [DBG] pgmap v1351: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 473 KiB/s rd, 315 KiB/s wr, 945 op/s 2022-04-22T17:45:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[46934]: audit 2022-04-22T17:45:27.107926+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:27 smithi115 bash[46934]: audit 2022-04-22T17:45:27.247370+0000 mon.a (mon.0) 2059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:45:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:28 smithi121 bash[36902]: audit 2022-04-22T17:45:27.249932+0000 mon.a (mon.0) 2060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:45:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:28 smithi121 bash[36902]: audit 2022-04-22T17:45:27.256578+0000 mon.a (mon.0) 2061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:45:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:28 smithi121 bash[36902]: audit 2022-04-22T17:45:27.618147+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[45495]: audit 2022-04-22T17:45:27.249932+0000 mon.a (mon.0) 2060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[45495]: audit 2022-04-22T17:45:27.256578+0000 mon.a (mon.0) 2061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[45495]: audit 2022-04-22T17:45:27.618147+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[46934]: audit 2022-04-22T17:45:27.249932+0000 mon.a (mon.0) 2060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[46934]: audit 2022-04-22T17:45:27.256578+0000 mon.a (mon.0) 2061 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:45:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:28 smithi115 bash[46934]: audit 2022-04-22T17:45:27.618147+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:45:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:29 smithi121 bash[36902]: cluster 2022-04-22T17:45:28.194201+0000 mgr.y (mgr.34107) 717 : cluster [DBG] pgmap v1352: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:29.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:29 smithi115 bash[45495]: cluster 2022-04-22T17:45:28.194201+0000 mgr.y (mgr.34107) 717 : cluster [DBG] pgmap v1352: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:29.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:29 smithi115 bash[46934]: cluster 2022-04-22T17:45:28.194201+0000 mgr.y (mgr.34107) 717 : cluster [DBG] pgmap v1352: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:30 smithi115 bash[45495]: audit 2022-04-22T17:45:29.729663+0000 mon.a (mon.0) 2063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:30 smithi115 bash[45495]: audit 2022-04-22T17:45:30.000478+0000 mon.a (mon.0) 2064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:30 smithi115 bash[46934]: audit 2022-04-22T17:45:29.729663+0000 mon.a (mon.0) 2063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:30 smithi115 bash[46934]: audit 2022-04-22T17:45:30.000478+0000 mon.a (mon.0) 2064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:30 smithi121 bash[36902]: audit 2022-04-22T17:45:29.729663+0000 mon.a (mon.0) 2063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:30 smithi121 bash[36902]: audit 2022-04-22T17:45:30.000478+0000 mon.a (mon.0) 2064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:31 smithi115 bash[45495]: cluster 2022-04-22T17:45:30.195119+0000 mgr.y (mgr.34107) 718 : cluster [DBG] pgmap v1353: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:31 smithi115 bash[46934]: cluster 2022-04-22T17:45:30.195119+0000 mgr.y (mgr.34107) 718 : cluster [DBG] pgmap v1353: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:31 smithi121 bash[36902]: cluster 2022-04-22T17:45:30.195119+0000 mgr.y (mgr.34107) 718 : cluster [DBG] pgmap v1353: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 199 KiB/s wr, 597 op/s 2022-04-22T17:45:32.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:31.660Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:32 smithi115 bash[45495]: audit 2022-04-22T17:45:32.110467+0000 mon.a (mon.0) 2065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:32 smithi115 bash[45495]: cluster 2022-04-22T17:45:32.196328+0000 mgr.y (mgr.34107) 719 : cluster [DBG] pgmap v1354: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 447 KiB/s rd, 298 KiB/s wr, 894 op/s 2022-04-22T17:45:32.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:32 smithi115 bash[46934]: audit 2022-04-22T17:45:32.110467+0000 mon.a (mon.0) 2065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:32.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:32 smithi115 bash[46934]: cluster 2022-04-22T17:45:32.196328+0000 mgr.y (mgr.34107) 719 : cluster [DBG] pgmap v1354: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 447 KiB/s rd, 298 KiB/s wr, 894 op/s 2022-04-22T17:45:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:32 smithi121 bash[36902]: audit 2022-04-22T17:45:32.110467+0000 mon.a (mon.0) 2065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:32 smithi121 bash[36902]: cluster 2022-04-22T17:45:32.196328+0000 mgr.y (mgr.34107) 719 : cluster [DBG] pgmap v1354: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 447 KiB/s rd, 298 KiB/s wr, 894 op/s 2022-04-22T17:45:34.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:33.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:35 smithi121 bash[36902]: cluster 2022-04-22T17:45:34.197361+0000 mgr.y (mgr.34107) 720 : cluster [DBG] pgmap v1355: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 295 KiB/s rd, 197 KiB/s wr, 590 op/s 2022-04-22T17:45:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:35 smithi121 bash[36902]: audit 2022-04-22T17:45:34.732324+0000 mon.a (mon.0) 2066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:35 smithi121 bash[36902]: audit 2022-04-22T17:45:35.003355+0000 mon.a (mon.0) 2067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[45495]: cluster 2022-04-22T17:45:34.197361+0000 mgr.y (mgr.34107) 720 : cluster [DBG] pgmap v1355: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 295 KiB/s rd, 197 KiB/s wr, 590 op/s 2022-04-22T17:45:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[45495]: audit 2022-04-22T17:45:34.732324+0000 mon.a (mon.0) 2066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[45495]: audit 2022-04-22T17:45:35.003355+0000 mon.a (mon.0) 2067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[46934]: cluster 2022-04-22T17:45:34.197361+0000 mgr.y (mgr.34107) 720 : cluster [DBG] pgmap v1355: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 295 KiB/s rd, 197 KiB/s wr, 590 op/s 2022-04-22T17:45:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[46934]: audit 2022-04-22T17:45:34.732324+0000 mon.a (mon.0) 2066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:35 smithi115 bash[46934]: audit 2022-04-22T17:45:35.003355+0000 mon.a (mon.0) 2067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:37 smithi115 bash[45495]: cluster 2022-04-22T17:45:36.198513+0000 mgr.y (mgr.34107) 721 : cluster [DBG] pgmap v1356: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 292 KiB/s wr, 875 op/s 2022-04-22T17:45:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:37 smithi115 bash[45495]: audit 2022-04-22T17:45:37.113554+0000 mon.a (mon.0) 2068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:37 smithi115 bash[46934]: cluster 2022-04-22T17:45:36.198513+0000 mgr.y (mgr.34107) 721 : cluster [DBG] pgmap v1356: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 292 KiB/s wr, 875 op/s 2022-04-22T17:45:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:37 smithi115 bash[46934]: audit 2022-04-22T17:45:37.113554+0000 mon.a (mon.0) 2068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:37 smithi121 bash[36902]: cluster 2022-04-22T17:45:36.198513+0000 mgr.y (mgr.34107) 721 : cluster [DBG] pgmap v1356: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 292 KiB/s wr, 875 op/s 2022-04-22T17:45:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:37 smithi121 bash[36902]: audit 2022-04-22T17:45:37.113554+0000 mon.a (mon.0) 2068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:38 smithi121 bash[36902]: cluster 2022-04-22T17:45:38.199633+0000 mgr.y (mgr.34107) 722 : cluster [DBG] pgmap v1357: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:38 smithi115 bash[45495]: cluster 2022-04-22T17:45:38.199633+0000 mgr.y (mgr.34107) 722 : cluster [DBG] pgmap v1357: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:38 smithi115 bash[46934]: cluster 2022-04-22T17:45:38.199633+0000 mgr.y (mgr.34107) 722 : cluster [DBG] pgmap v1357: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:40.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:39 smithi115 bash[45495]: audit 2022-04-22T17:45:39.734922+0000 mon.a (mon.0) 2069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:40.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:39 smithi115 bash[46934]: audit 2022-04-22T17:45:39.734922+0000 mon.a (mon.0) 2069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:39 smithi121 bash[36902]: audit 2022-04-22T17:45:39.734922+0000 mon.a (mon.0) 2069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:41.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:40 smithi121 bash[36902]: audit 2022-04-22T17:45:40.006432+0000 mon.a (mon.0) 2070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:41.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:40 smithi121 bash[36902]: cluster 2022-04-22T17:45:40.200810+0000 mgr.y (mgr.34107) 723 : cluster [DBG] pgmap v1358: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:40 smithi115 bash[45495]: audit 2022-04-22T17:45:40.006432+0000 mon.a (mon.0) 2070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:40 smithi115 bash[45495]: cluster 2022-04-22T17:45:40.200810+0000 mgr.y (mgr.34107) 723 : cluster [DBG] pgmap v1358: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:41.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:40 smithi115 bash[46934]: audit 2022-04-22T17:45:40.006432+0000 mon.a (mon.0) 2070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:41.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:40 smithi115 bash[46934]: cluster 2022-04-22T17:45:40.200810+0000 mgr.y (mgr.34107) 723 : cluster [DBG] pgmap v1358: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 291 KiB/s rd, 194 KiB/s wr, 582 op/s 2022-04-22T17:45:42.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:41.661Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:42 smithi121 bash[36902]: audit 2022-04-22T17:45:42.115968+0000 mon.a (mon.0) 2071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:42.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:42 smithi115 bash[45495]: audit 2022-04-22T17:45:42.115968+0000 mon.a (mon.0) 2071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:42.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:42 smithi115 bash[46934]: audit 2022-04-22T17:45:42.115968+0000 mon.a (mon.0) 2071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:43 smithi121 bash[36902]: cluster 2022-04-22T17:45:42.202192+0000 mgr.y (mgr.34107) 724 : cluster [DBG] pgmap v1359: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 437 KiB/s rd, 291 KiB/s wr, 873 op/s 2022-04-22T17:45:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:43 smithi115 bash[45495]: cluster 2022-04-22T17:45:42.202192+0000 mgr.y (mgr.34107) 724 : cluster [DBG] pgmap v1359: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 437 KiB/s rd, 291 KiB/s wr, 873 op/s 2022-04-22T17:45:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:43 smithi115 bash[46934]: cluster 2022-04-22T17:45:42.202192+0000 mgr.y (mgr.34107) 724 : cluster [DBG] pgmap v1359: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 437 KiB/s rd, 291 KiB/s wr, 873 op/s 2022-04-22T17:45:44.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:43.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[45495]: cluster 2022-04-22T17:45:44.203060+0000 mgr.y (mgr.34107) 725 : cluster [DBG] pgmap v1360: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:45:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[45495]: audit 2022-04-22T17:45:44.737583+0000 mon.a (mon.0) 2072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[45495]: audit 2022-04-22T17:45:45.009762+0000 mon.a (mon.0) 2073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[46934]: cluster 2022-04-22T17:45:44.203060+0000 mgr.y (mgr.34107) 725 : cluster [DBG] pgmap v1360: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:45:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[46934]: audit 2022-04-22T17:45:44.737583+0000 mon.a (mon.0) 2072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:45 smithi115 bash[46934]: audit 2022-04-22T17:45:45.009762+0000 mon.a (mon.0) 2073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:45 smithi121 bash[36902]: cluster 2022-04-22T17:45:44.203060+0000 mgr.y (mgr.34107) 725 : cluster [DBG] pgmap v1360: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:45:45.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:45 smithi121 bash[36902]: audit 2022-04-22T17:45:44.737583+0000 mon.a (mon.0) 2072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:45.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:45 smithi121 bash[36902]: audit 2022-04-22T17:45:45.009762+0000 mon.a (mon.0) 2073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:46 smithi121 bash[36902]: cluster 2022-04-22T17:45:46.204248+0000 mgr.y (mgr.34107) 726 : cluster [DBG] pgmap v1361: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:45:46.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:46 smithi115 bash[45495]: cluster 2022-04-22T17:45:46.204248+0000 mgr.y (mgr.34107) 726 : cluster [DBG] pgmap v1361: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:45:46.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:46 smithi115 bash[46934]: cluster 2022-04-22T17:45:46.204248+0000 mgr.y (mgr.34107) 726 : cluster [DBG] pgmap v1361: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:45:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:47 smithi121 bash[36902]: audit 2022-04-22T17:45:47.118742+0000 mon.a (mon.0) 2074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:47 smithi115 bash[45495]: audit 2022-04-22T17:45:47.118742+0000 mon.a (mon.0) 2074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:47 smithi115 bash[46934]: audit 2022-04-22T17:45:47.118742+0000 mon.a (mon.0) 2074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:48 smithi121 bash[36902]: cluster 2022-04-22T17:45:48.205123+0000 mgr.y (mgr.34107) 727 : cluster [DBG] pgmap v1362: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:48.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:48 smithi115 bash[45495]: cluster 2022-04-22T17:45:48.205123+0000 mgr.y (mgr.34107) 727 : cluster [DBG] pgmap v1362: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:48 smithi115 bash[46934]: cluster 2022-04-22T17:45:48.205123+0000 mgr.y (mgr.34107) 727 : cluster [DBG] pgmap v1362: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:49.000 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:45:49.363 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (8m) 8m ago 22m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (22m) 9m ago 22m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (10m) 8m ago 23m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (10m) 9m ago 23m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (18m) 9m ago 27m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:45:49.364 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (18m) 8m ago 30m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (17m) 8m ago 30m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (16m) 9m ago 28m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (17m) 8m ago 28m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (9m) 8m ago 22m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (9m) 9m ago 22m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (15m) 8m ago 27m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (14m) 8m ago 27m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:45:49.365 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (14m) 8m ago 26m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (14m) 8m ago 26m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (13m) 9m ago 25m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (12m) 9m ago 25m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (11m) 9m ago 24m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (11m) 9m ago 24m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:45:49.366 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (9m) 9m ago 23m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:45:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:49 smithi121 bash[36902]: audit 2022-04-22T17:45:48.974130+0000 mgr.y (mgr.34107) 728 : audit [DBG] from='client.45088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:49 smithi115 bash[45495]: audit 2022-04-22T17:45:48.974130+0000 mgr.y (mgr.34107) 728 : audit [DBG] from='client.45088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:49 smithi115 bash[46934]: audit 2022-04-22T17:45:48.974130+0000 mgr.y (mgr.34107) 728 : audit [DBG] from='client.45088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:49.811 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:45:49.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:45:49.813 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:45:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:50 smithi121 bash[36902]: audit 2022-04-22T17:45:49.333345+0000 mgr.y (mgr.34107) 729 : audit [DBG] from='client.45094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:50 smithi121 bash[36902]: audit 2022-04-22T17:45:49.740069+0000 mon.a (mon.0) 2075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:50 smithi121 bash[36902]: audit 2022-04-22T17:45:49.801823+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? 172.21.15.115:0/922893071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:50 smithi121 bash[36902]: audit 2022-04-22T17:45:50.012713+0000 mon.a (mon.0) 2077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:50 smithi121 bash[36902]: cluster 2022-04-22T17:45:50.206210+0000 mgr.y (mgr.34107) 730 : cluster [DBG] pgmap v1363: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:50.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[45495]: audit 2022-04-22T17:45:49.333345+0000 mgr.y (mgr.34107) 729 : audit [DBG] from='client.45094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:50.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[45495]: audit 2022-04-22T17:45:49.740069+0000 mon.a (mon.0) 2075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:50.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[45495]: audit 2022-04-22T17:45:49.801823+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? 172.21.15.115:0/922893071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:50.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[45495]: audit 2022-04-22T17:45:50.012713+0000 mon.a (mon.0) 2077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:50.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[45495]: cluster 2022-04-22T17:45:50.206210+0000 mgr.y (mgr.34107) 730 : cluster [DBG] pgmap v1363: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[46934]: audit 2022-04-22T17:45:49.333345+0000 mgr.y (mgr.34107) 729 : audit [DBG] from='client.45094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:45:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[46934]: audit 2022-04-22T17:45:49.740069+0000 mon.a (mon.0) 2075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[46934]: audit 2022-04-22T17:45:49.801823+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? 172.21.15.115:0/922893071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:45:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[46934]: audit 2022-04-22T17:45:50.012713+0000 mon.a (mon.0) 2077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:50 smithi115 bash[46934]: cluster 2022-04-22T17:45:50.206210+0000 mgr.y (mgr.34107) 730 : cluster [DBG] pgmap v1363: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:45:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:51.661Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:52 smithi121 bash[36902]: audit 2022-04-22T17:45:52.121401+0000 mon.a (mon.0) 2078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:52 smithi115 bash[45495]: audit 2022-04-22T17:45:52.121401+0000 mon.a (mon.0) 2078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:52.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:52 smithi115 bash[46934]: audit 2022-04-22T17:45:52.121401+0000 mon.a (mon.0) 2078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:53 smithi121 bash[36902]: cluster 2022-04-22T17:45:52.207329+0000 mgr.y (mgr.34107) 731 : cluster [DBG] pgmap v1364: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:53 smithi115 bash[46934]: cluster 2022-04-22T17:45:52.207329+0000 mgr.y (mgr.34107) 731 : cluster [DBG] pgmap v1364: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:53 smithi115 bash[45495]: cluster 2022-04-22T17:45:52.207329+0000 mgr.y (mgr.34107) 731 : cluster [DBG] pgmap v1364: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:54.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:45:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:45:53.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:45:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:54 smithi121 bash[36902]: cluster 2022-04-22T17:45:54.208098+0000 mgr.y (mgr.34107) 732 : cluster [DBG] pgmap v1365: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 193 KiB/s wr, 580 op/s 2022-04-22T17:45:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:54 smithi115 bash[45495]: cluster 2022-04-22T17:45:54.208098+0000 mgr.y (mgr.34107) 732 : cluster [DBG] pgmap v1365: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 193 KiB/s wr, 580 op/s 2022-04-22T17:45:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:54 smithi115 bash[46934]: cluster 2022-04-22T17:45:54.208098+0000 mgr.y (mgr.34107) 732 : cluster [DBG] pgmap v1365: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 193 KiB/s wr, 580 op/s 2022-04-22T17:45:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:55 smithi121 bash[36902]: audit 2022-04-22T17:45:54.742494+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:55 smithi121 bash[36902]: audit 2022-04-22T17:45:55.015457+0000 mon.a (mon.0) 2080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:55 smithi115 bash[45495]: audit 2022-04-22T17:45:54.742494+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:55 smithi115 bash[45495]: audit 2022-04-22T17:45:55.015457+0000 mon.a (mon.0) 2080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:55 smithi115 bash[46934]: audit 2022-04-22T17:45:54.742494+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:45:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:55 smithi115 bash[46934]: audit 2022-04-22T17:45:55.015457+0000 mon.a (mon.0) 2080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:45:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:56 smithi121 bash[36902]: cluster 2022-04-22T17:45:56.208973+0000 mgr.y (mgr.34107) 733 : cluster [DBG] pgmap v1366: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:56 smithi115 bash[45495]: cluster 2022-04-22T17:45:56.208973+0000 mgr.y (mgr.34107) 733 : cluster [DBG] pgmap v1366: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:56.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:56 smithi115 bash[46934]: cluster 2022-04-22T17:45:56.208973+0000 mgr.y (mgr.34107) 733 : cluster [DBG] pgmap v1366: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 290 KiB/s wr, 871 op/s 2022-04-22T17:45:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:57 smithi121 bash[36902]: audit 2022-04-22T17:45:57.124196+0000 mon.a (mon.0) 2081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:57 smithi115 bash[45495]: audit 2022-04-22T17:45:57.124196+0000 mon.a (mon.0) 2081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:57 smithi115 bash[46934]: audit 2022-04-22T17:45:57.124196+0000 mon.a (mon.0) 2081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:45:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:58 smithi121 bash[36902]: cluster 2022-04-22T17:45:58.210085+0000 mgr.y (mgr.34107) 734 : cluster [DBG] pgmap v1367: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:45:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:58 smithi115 bash[45495]: cluster 2022-04-22T17:45:58.210085+0000 mgr.y (mgr.34107) 734 : cluster [DBG] pgmap v1367: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:45:58.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:58 smithi115 bash[46934]: cluster 2022-04-22T17:45:58.210085+0000 mgr.y (mgr.34107) 734 : cluster [DBG] pgmap v1367: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:46:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:45:59 smithi121 bash[36902]: audit 2022-04-22T17:45:59.745160+0000 mon.a (mon.0) 2082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:45:59 smithi115 bash[45495]: audit 2022-04-22T17:45:59.745160+0000 mon.a (mon.0) 2082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:45:59 smithi115 bash[46934]: audit 2022-04-22T17:45:59.745160+0000 mon.a (mon.0) 2082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:00 smithi121 bash[36902]: audit 2022-04-22T17:46:00.018408+0000 mon.a (mon.0) 2083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:01.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:00 smithi121 bash[36902]: cluster 2022-04-22T17:46:00.210963+0000 mgr.y (mgr.34107) 735 : cluster [DBG] pgmap v1368: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:46:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:00 smithi115 bash[45495]: audit 2022-04-22T17:46:00.018408+0000 mon.a (mon.0) 2083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:00 smithi115 bash[45495]: cluster 2022-04-22T17:46:00.210963+0000 mgr.y (mgr.34107) 735 : cluster [DBG] pgmap v1368: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:46:01.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:00 smithi115 bash[46934]: audit 2022-04-22T17:46:00.018408+0000 mon.a (mon.0) 2083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:01.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:00 smithi115 bash[46934]: cluster 2022-04-22T17:46:00.210963+0000 mgr.y (mgr.34107) 735 : cluster [DBG] pgmap v1368: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 292 KiB/s rd, 194 KiB/s wr, 583 op/s 2022-04-22T17:46:02.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:01.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:02 smithi121 bash[36902]: audit 2022-04-22T17:46:02.126687+0000 mon.a (mon.0) 2084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:02 smithi115 bash[45495]: audit 2022-04-22T17:46:02.126687+0000 mon.a (mon.0) 2084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:02 smithi115 bash[46934]: audit 2022-04-22T17:46:02.126687+0000 mon.a (mon.0) 2084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:03 smithi121 bash[36902]: cluster 2022-04-22T17:46:02.212058+0000 mgr.y (mgr.34107) 736 : cluster [DBG] pgmap v1369: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 291 KiB/s wr, 871 op/s 2022-04-22T17:46:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:03 smithi115 bash[46934]: cluster 2022-04-22T17:46:02.212058+0000 mgr.y (mgr.34107) 736 : cluster [DBG] pgmap v1369: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 291 KiB/s wr, 871 op/s 2022-04-22T17:46:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:03 smithi115 bash[45495]: cluster 2022-04-22T17:46:02.212058+0000 mgr.y (mgr.34107) 736 : cluster [DBG] pgmap v1369: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 436 KiB/s rd, 291 KiB/s wr, 871 op/s 2022-04-22T17:46:04.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:03.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:04 smithi121 bash[36902]: cluster 2022-04-22T17:46:04.212953+0000 mgr.y (mgr.34107) 737 : cluster [DBG] pgmap v1370: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:46:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:04 smithi115 bash[45495]: cluster 2022-04-22T17:46:04.212953+0000 mgr.y (mgr.34107) 737 : cluster [DBG] pgmap v1370: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:46:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:04 smithi115 bash[46934]: cluster 2022-04-22T17:46:04.212953+0000 mgr.y (mgr.34107) 737 : cluster [DBG] pgmap v1370: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 193 KiB/s wr, 578 op/s 2022-04-22T17:46:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:05 smithi121 bash[36902]: audit 2022-04-22T17:46:04.748173+0000 mon.a (mon.0) 2085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:05.816 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:05 smithi121 bash[36902]: audit 2022-04-22T17:46:05.021393+0000 mon.a (mon.0) 2086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:05.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:05 smithi115 bash[45495]: audit 2022-04-22T17:46:04.748173+0000 mon.a (mon.0) 2085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:05.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:05 smithi115 bash[45495]: audit 2022-04-22T17:46:05.021393+0000 mon.a (mon.0) 2086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:05.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:05 smithi115 bash[46934]: audit 2022-04-22T17:46:04.748173+0000 mon.a (mon.0) 2085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:05.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:05 smithi115 bash[46934]: audit 2022-04-22T17:46:05.021393+0000 mon.a (mon.0) 2086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:06 smithi121 bash[36902]: cluster 2022-04-22T17:46:06.214051+0000 mgr.y (mgr.34107) 738 : cluster [DBG] pgmap v1371: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 289 KiB/s wr, 866 op/s 2022-04-22T17:46:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:06 smithi115 bash[45495]: cluster 2022-04-22T17:46:06.214051+0000 mgr.y (mgr.34107) 738 : cluster [DBG] pgmap v1371: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 289 KiB/s wr, 866 op/s 2022-04-22T17:46:06.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:06 smithi115 bash[46934]: cluster 2022-04-22T17:46:06.214051+0000 mgr.y (mgr.34107) 738 : cluster [DBG] pgmap v1371: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 289 KiB/s wr, 866 op/s 2022-04-22T17:46:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:07 smithi121 bash[36902]: audit 2022-04-22T17:46:07.129283+0000 mon.a (mon.0) 2087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:07 smithi115 bash[45495]: audit 2022-04-22T17:46:07.129283+0000 mon.a (mon.0) 2087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:07 smithi115 bash[46934]: audit 2022-04-22T17:46:07.129283+0000 mon.a (mon.0) 2087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:08 smithi121 bash[36902]: cluster 2022-04-22T17:46:08.215103+0000 mgr.y (mgr.34107) 739 : cluster [DBG] pgmap v1372: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:08.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:08 smithi115 bash[46934]: cluster 2022-04-22T17:46:08.215103+0000 mgr.y (mgr.34107) 739 : cluster [DBG] pgmap v1372: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:08.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:08 smithi115 bash[45495]: cluster 2022-04-22T17:46:08.215103+0000 mgr.y (mgr.34107) 739 : cluster [DBG] pgmap v1372: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:09 smithi121 bash[36902]: audit 2022-04-22T17:46:09.751118+0000 mon.a (mon.0) 2088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:09 smithi115 bash[45495]: audit 2022-04-22T17:46:09.751118+0000 mon.a (mon.0) 2088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:09 smithi115 bash[46934]: audit 2022-04-22T17:46:09.751118+0000 mon.a (mon.0) 2088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:10 smithi121 bash[36902]: audit 2022-04-22T17:46:10.024538+0000 mon.a (mon.0) 2089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:10 smithi121 bash[36902]: cluster 2022-04-22T17:46:10.216106+0000 mgr.y (mgr.34107) 740 : cluster [DBG] pgmap v1373: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:10 smithi115 bash[45495]: audit 2022-04-22T17:46:10.024538+0000 mon.a (mon.0) 2089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:10 smithi115 bash[45495]: cluster 2022-04-22T17:46:10.216106+0000 mgr.y (mgr.34107) 740 : cluster [DBG] pgmap v1373: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:11.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:10 smithi115 bash[46934]: audit 2022-04-22T17:46:10.024538+0000 mon.a (mon.0) 2089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:11.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:10 smithi115 bash[46934]: cluster 2022-04-22T17:46:10.216106+0000 mgr.y (mgr.34107) 740 : cluster [DBG] pgmap v1373: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 575 op/s 2022-04-22T17:46:12.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:11.663Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:12 smithi121 bash[36902]: audit 2022-04-22T17:46:12.132304+0000 mon.a (mon.0) 2090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:12.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:12 smithi115 bash[45495]: audit 2022-04-22T17:46:12.132304+0000 mon.a (mon.0) 2090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:12.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:12 smithi115 bash[46934]: audit 2022-04-22T17:46:12.132304+0000 mon.a (mon.0) 2090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:13 smithi121 bash[36902]: cluster 2022-04-22T17:46:12.216977+0000 mgr.y (mgr.34107) 741 : cluster [DBG] pgmap v1374: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:46:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:13 smithi115 bash[45495]: cluster 2022-04-22T17:46:12.216977+0000 mgr.y (mgr.34107) 741 : cluster [DBG] pgmap v1374: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:46:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:13 smithi115 bash[46934]: cluster 2022-04-22T17:46:12.216977+0000 mgr.y (mgr.34107) 741 : cluster [DBG] pgmap v1374: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 288 KiB/s wr, 864 op/s 2022-04-22T17:46:14.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:13.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:14 smithi121 bash[36902]: audit 2022-04-22T17:46:13.419442+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:46:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:14 smithi121 bash[36902]: audit 2022-04-22T17:46:13.419927+0000 mon.a (mon.0) 2092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:46:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:14 smithi115 bash[46934]: audit 2022-04-22T17:46:13.419442+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:46:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:14 smithi115 bash[46934]: audit 2022-04-22T17:46:13.419927+0000 mon.a (mon.0) 2092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:46:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:14 smithi115 bash[45495]: audit 2022-04-22T17:46:13.419442+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:46:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:14 smithi115 bash[45495]: audit 2022-04-22T17:46:13.419927+0000 mon.a (mon.0) 2092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:46:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:15 smithi121 bash[36902]: cluster 2022-04-22T17:46:14.217982+0000 mgr.y (mgr.34107) 742 : cluster [DBG] pgmap v1375: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:46:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:15 smithi121 bash[36902]: audit 2022-04-22T17:46:14.753999+0000 mon.a (mon.0) 2093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:15 smithi121 bash[36902]: audit 2022-04-22T17:46:15.027350+0000 mon.a (mon.0) 2094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[45495]: cluster 2022-04-22T17:46:14.217982+0000 mgr.y (mgr.34107) 742 : cluster [DBG] pgmap v1375: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[45495]: audit 2022-04-22T17:46:14.753999+0000 mon.a (mon.0) 2093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[45495]: audit 2022-04-22T17:46:15.027350+0000 mon.a (mon.0) 2094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[46934]: cluster 2022-04-22T17:46:14.217982+0000 mgr.y (mgr.34107) 742 : cluster [DBG] pgmap v1375: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 192 KiB/s wr, 576 op/s 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[46934]: audit 2022-04-22T17:46:14.753999+0000 mon.a (mon.0) 2093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:15 smithi115 bash[46934]: audit 2022-04-22T17:46:15.027350+0000 mon.a (mon.0) 2094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:17 smithi121 bash[36902]: cluster 2022-04-22T17:46:16.219054+0000 mgr.y (mgr.34107) 743 : cluster [DBG] pgmap v1376: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 430 KiB/s rd, 287 KiB/s wr, 860 op/s 2022-04-22T17:46:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:17 smithi121 bash[36902]: audit 2022-04-22T17:46:17.134822+0000 mon.a (mon.0) 2095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:17 smithi115 bash[46934]: cluster 2022-04-22T17:46:16.219054+0000 mgr.y (mgr.34107) 743 : cluster [DBG] pgmap v1376: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 430 KiB/s rd, 287 KiB/s wr, 860 op/s 2022-04-22T17:46:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:17 smithi115 bash[46934]: audit 2022-04-22T17:46:17.134822+0000 mon.a (mon.0) 2095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:17.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:17 smithi115 bash[45495]: cluster 2022-04-22T17:46:16.219054+0000 mgr.y (mgr.34107) 743 : cluster [DBG] pgmap v1376: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 430 KiB/s rd, 287 KiB/s wr, 860 op/s 2022-04-22T17:46:17.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:17 smithi115 bash[45495]: audit 2022-04-22T17:46:17.134822+0000 mon.a (mon.0) 2095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:19 smithi121 bash[36902]: cluster 2022-04-22T17:46:18.220183+0000 mgr.y (mgr.34107) 744 : cluster [DBG] pgmap v1377: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:19 smithi115 bash[45495]: cluster 2022-04-22T17:46:18.220183+0000 mgr.y (mgr.34107) 744 : cluster [DBG] pgmap v1377: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:19.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:19 smithi115 bash[46934]: cluster 2022-04-22T17:46:18.220183+0000 mgr.y (mgr.34107) 744 : cluster [DBG] pgmap v1377: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:19.700 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.index_list_delimited (78996 ms) 2022-04-22T17:46:19.701 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.bi_list 2022-04-22T17:46:19.848 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.bi_list (148 ms) 2022-04-22T17:46:19.849 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.gc_set 2022-04-22T17:46:19.896 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.gc_set (48 ms) 2022-04-22T17:46:19.896 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.gc_list 2022-04-22T17:46:19.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.gc_list (41 ms) 2022-04-22T17:46:19.938 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.gc_defer 2022-04-22T17:46:20.642 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:46:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:20 smithi121 bash[36902]: audit 2022-04-22T17:46:19.756548+0000 mon.a (mon.0) 2096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:20 smithi115 bash[45495]: audit 2022-04-22T17:46:19.756548+0000 mon.a (mon.0) 2096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:20 smithi115 bash[46934]: audit 2022-04-22T17:46:19.756548+0000 mon.a (mon.0) 2096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:46:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (9m) 9m ago 23m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:46:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (22m) 10m ago 22m 23.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:46:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (11m) 9m ago 24m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:46:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (10m) 10m ago 24m 10.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (18m) 10m ago 28m 406M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (19m) 9m ago 30m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (18m) 9m ago 30m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (17m) 10m ago 28m 62.8M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (17m) 9m ago 28m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (10m) 9m ago 23m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (10m) 10m ago 23m 7168k - 1dbe0e931976 f4b044642a96 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (16m) 9m ago 28m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (15m) 9m ago 27m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:46:20.995 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (15m) 9m ago 27m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (14m) 9m ago 26m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (13m) 10m ago 26m 255M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (13m) 10m ago 25m 405M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (12m) 10m ago 25m 219M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (11m) 10m ago 24m 271M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:46:20.996 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (10m) 10m ago 23m 32.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:46:21.394 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:46:21.394 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:46:21.394 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:46:21.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:21.396 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:46:21.396 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:46:21.396 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:46:21.396 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:46:21.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: cluster 2022-04-22T17:46:20.221304+0000 mgr.y (mgr.34107) 745 : cluster [DBG] pgmap v1378: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:21.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:20.392792+0000 mon.a (mon.0) 2097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:21.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: cluster 2022-04-22T17:46:20.450237+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2022-04-22T17:46:21.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:20.451751+0000 mon.c (mon.1) 64 : audit [INF] from='client.? 172.21.15.115:0/302399117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:20.454626+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:20.615660+0000 mgr.y (mgr.34107) 746 : audit [DBG] from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:20.963355+0000 mgr.y (mgr.34107) 747 : audit [DBG] from='client.45118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[45495]: audit 2022-04-22T17:46:21.384101+0000 mon.a (mon.0) 2100 : audit [DBG] from='client.? 172.21.15.115:0/3838109262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: cluster 2022-04-22T17:46:20.221304+0000 mgr.y (mgr.34107) 745 : cluster [DBG] pgmap v1378: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:20.392792+0000 mon.a (mon.0) 2097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: cluster 2022-04-22T17:46:20.450237+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:20.451751+0000 mon.c (mon.1) 64 : audit [INF] from='client.? 172.21.15.115:0/302399117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:20.454626+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:20.615660+0000 mgr.y (mgr.34107) 746 : audit [DBG] from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:20.963355+0000 mgr.y (mgr.34107) 747 : audit [DBG] from='client.45118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[46934]: audit 2022-04-22T17:46:21.384101+0000 mon.a (mon.0) 2100 : audit [DBG] from='client.? 172.21.15.115:0/3838109262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: cluster 2022-04-22T17:46:20.221304+0000 mgr.y (mgr.34107) 745 : cluster [DBG] pgmap v1378: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 191 KiB/s wr, 573 op/s 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:20.392792+0000 mon.a (mon.0) 2097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: cluster 2022-04-22T17:46:20.450237+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:20.451751+0000 mon.c (mon.1) 64 : audit [INF] from='client.? 172.21.15.115:0/302399117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:20.454626+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:20.615660+0000 mgr.y (mgr.34107) 746 : audit [DBG] from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:20.963355+0000 mgr.y (mgr.34107) 747 : audit [DBG] from='client.45118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:21 smithi121 bash[36902]: audit 2022-04-22T17:46:21.384101+0000 mon.a (mon.0) 2100 : audit [DBG] from='client.? 172.21.15.115:0/3838109262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:21.664Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:22 smithi121 bash[36902]: audit 2022-04-22T17:46:21.454642+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:22.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:22 smithi121 bash[36902]: cluster 2022-04-22T17:46:21.455209+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2022-04-22T17:46:22.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:22 smithi121 bash[36902]: audit 2022-04-22T17:46:22.137902+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:22.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:22 smithi121 bash[36902]: cluster 2022-04-22T17:46:22.222440+0000 mgr.y (mgr.34107) 748 : cluster [DBG] pgmap v1381: 169 pgs: 3 creating+peering, 24 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 266 KiB/s wr, 793 op/s 2022-04-22T17:46:22.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[45495]: audit 2022-04-22T17:46:21.454642+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:22.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[45495]: cluster 2022-04-22T17:46:21.455209+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2022-04-22T17:46:22.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[45495]: audit 2022-04-22T17:46:22.137902+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:22.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[45495]: cluster 2022-04-22T17:46:22.222440+0000 mgr.y (mgr.34107) 748 : cluster [DBG] pgmap v1381: 169 pgs: 3 creating+peering, 24 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 266 KiB/s wr, 793 op/s 2022-04-22T17:46:22.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[46934]: audit 2022-04-22T17:46:21.454642+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64247-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:22.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[46934]: cluster 2022-04-22T17:46:21.455209+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2022-04-22T17:46:22.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[46934]: audit 2022-04-22T17:46:22.137902+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:22.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:22 smithi115 bash[46934]: cluster 2022-04-22T17:46:22.222440+0000 mgr.y (mgr.34107) 748 : cluster [DBG] pgmap v1381: 169 pgs: 3 creating+peering, 24 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 578 KiB/s rd, 266 KiB/s wr, 793 op/s 2022-04-22T17:46:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:23 smithi121 bash[36902]: cluster 2022-04-22T17:46:22.465685+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2022-04-22T17:46:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:23 smithi115 bash[45495]: cluster 2022-04-22T17:46:22.465685+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2022-04-22T17:46:23.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:23.551Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:23 smithi115 bash[46934]: cluster 2022-04-22T17:46:22.465685+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2022-04-22T17:46:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:24 smithi121 bash[36902]: cluster 2022-04-22T17:46:24.223656+0000 mgr.y (mgr.34107) 749 : cluster [DBG] pgmap v1383: 169 pgs: 3 creating+peering, 13 unknown, 153 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 165 KiB/s wr, 488 op/s 2022-04-22T17:46:24.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:24 smithi115 bash[45495]: cluster 2022-04-22T17:46:24.223656+0000 mgr.y (mgr.34107) 749 : cluster [DBG] pgmap v1383: 169 pgs: 3 creating+peering, 13 unknown, 153 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 165 KiB/s wr, 488 op/s 2022-04-22T17:46:24.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:24 smithi115 bash[46934]: cluster 2022-04-22T17:46:24.223656+0000 mgr.y (mgr.34107) 749 : cluster [DBG] pgmap v1383: 169 pgs: 3 creating+peering, 13 unknown, 153 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 165 KiB/s wr, 488 op/s 2022-04-22T17:46:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:25 smithi121 bash[36902]: audit 2022-04-22T17:46:24.759444+0000 mon.a (mon.0) 2105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:25 smithi121 bash[36902]: audit 2022-04-22T17:46:25.034546+0000 mon.a (mon.0) 2106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:25 smithi115 bash[45495]: audit 2022-04-22T17:46:24.759444+0000 mon.a (mon.0) 2105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:25 smithi115 bash[45495]: audit 2022-04-22T17:46:25.034546+0000 mon.a (mon.0) 2106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:25 smithi115 bash[46934]: audit 2022-04-22T17:46:24.759444+0000 mon.a (mon.0) 2105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:25 smithi115 bash[46934]: audit 2022-04-22T17:46:25.034546+0000 mon.a (mon.0) 2106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:26 smithi121 bash[36902]: cluster 2022-04-22T17:46:26.224900+0000 mgr.y (mgr.34107) 750 : cluster [DBG] pgmap v1384: 169 pgs: 169 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 487 KiB/s rd, 166 KiB/s wr, 490 op/s 2022-04-22T17:46:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:26 smithi115 bash[45495]: cluster 2022-04-22T17:46:26.224900+0000 mgr.y (mgr.34107) 750 : cluster [DBG] pgmap v1384: 169 pgs: 169 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 487 KiB/s rd, 166 KiB/s wr, 490 op/s 2022-04-22T17:46:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:26 smithi115 bash[46934]: cluster 2022-04-22T17:46:26.224900+0000 mgr.y (mgr.34107) 750 : cluster [DBG] pgmap v1384: 169 pgs: 169 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 487 KiB/s rd, 166 KiB/s wr, 490 op/s 2022-04-22T17:46:27.503 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.gc_defer (7565 ms) 2022-04-22T17:46:27.503 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.usage_basic 2022-04-22T17:46:27.614 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.usage_basic (111 ms) 2022-04-22T17:46:27.615 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2022-04-22T17:46:27.616 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.usage_clear_no_obj (2 ms) 2022-04-22T17:46:27.616 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.usage_clear 2022-04-22T17:46:27.735 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.usage_clear (119 ms) 2022-04-22T17:46:27.735 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] cls_rgw.bi_log_trim 2022-04-22T17:46:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:27 smithi121 bash[36902]: audit 2022-04-22T17:46:27.140915+0000 mon.a (mon.0) 2107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:27 smithi115 bash[46934]: audit 2022-04-22T17:46:27.140915+0000 mon.a (mon.0) 2107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:27 smithi115 bash[45495]: audit 2022-04-22T17:46:27.140915+0000 mon.a (mon.0) 2107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:27.805 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] cls_rgw.bi_log_trim (70 ms) 2022-04-22T17:46:28.524 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 14 tests from cls_rgw (89290 ms total) 2022-04-22T17:46:28.524 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:46:28.524 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:46:28.524 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 14 tests from 1 test suite ran. (91931 ms total) 2022-04-22T17:46:28.524 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 14 tests. 2022-04-22T17:46:28.524 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:46:28.525 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:46:28.537 INFO:tasks.workunit:Running workunit cls/test_cls_sdk.sh... 2022-04-22T17:46:28.537 DEBUG:teuthology.orchestra.run.smithi115:workunit test cls/test_cls_sdk.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_sdk.sh 2022-04-22T17:46:28.552 INFO:tasks.workunit.client.0.smithi115.stdout:Running main() from gmock_main.cc 2022-04-22T17:46:28.552 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 2 tests from 1 test suite. 2022-04-22T17:46:28.553 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T17:46:28.553 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from ClsSDK 2022-04-22T17:46:28.553 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsSDK.TestSDKCoverageWrite 2022-04-22T17:46:28.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:28 smithi121 bash[36902]: cluster 2022-04-22T17:46:27.490143+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2022-04-22T17:46:28.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:28 smithi121 bash[36902]: audit 2022-04-22T17:46:27.627136+0000 mon.a (mon.0) 2109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:46:28.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:28 smithi121 bash[36902]: audit 2022-04-22T17:46:27.630293+0000 mon.a (mon.0) 2110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:46:28.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:28 smithi121 bash[36902]: audit 2022-04-22T17:46:27.632050+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:46:28.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:28 smithi121 bash[36902]: cluster 2022-04-22T17:46:28.226020+0000 mgr.y (mgr.34107) 751 : cluster [DBG] pgmap v1386: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[45495]: cluster 2022-04-22T17:46:27.490143+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[45495]: audit 2022-04-22T17:46:27.627136+0000 mon.a (mon.0) 2109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[45495]: audit 2022-04-22T17:46:27.630293+0000 mon.a (mon.0) 2110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[45495]: audit 2022-04-22T17:46:27.632050+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[45495]: cluster 2022-04-22T17:46:28.226020+0000 mgr.y (mgr.34107) 751 : cluster [DBG] pgmap v1386: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[46934]: cluster 2022-04-22T17:46:27.490143+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[46934]: audit 2022-04-22T17:46:27.627136+0000 mon.a (mon.0) 2109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:46:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[46934]: audit 2022-04-22T17:46:27.630293+0000 mon.a (mon.0) 2110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:46:28.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[46934]: audit 2022-04-22T17:46:27.632050+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:46:28.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:28 smithi115 bash[46934]: cluster 2022-04-22T17:46:28.226020+0000 mgr.y (mgr.34107) 751 : cluster [DBG] pgmap v1386: 137 pgs: 137 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:29 smithi115 bash[45495]: cluster 2022-04-22T17:46:28.508209+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2022-04-22T17:46:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:29 smithi115 bash[46934]: cluster 2022-04-22T17:46:28.508209+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2022-04-22T17:46:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:29 smithi121 bash[36902]: cluster 2022-04-22T17:46:28.508209+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2022-04-22T17:46:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[45495]: cluster 2022-04-22T17:46:29.519643+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2022-04-22T17:46:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[45495]: audit 2022-04-22T17:46:29.524408+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[45495]: audit 2022-04-22T17:46:29.762818+0000 mon.a (mon.0) 2115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[45495]: audit 2022-04-22T17:46:30.038411+0000 mon.a (mon.0) 2116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[45495]: cluster 2022-04-22T17:46:30.227090+0000 mgr.y (mgr.34107) 752 : cluster [DBG] pgmap v1389: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[46934]: cluster 2022-04-22T17:46:29.519643+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[46934]: audit 2022-04-22T17:46:29.524408+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[46934]: audit 2022-04-22T17:46:29.762818+0000 mon.a (mon.0) 2115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[46934]: audit 2022-04-22T17:46:30.038411+0000 mon.a (mon.0) 2116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:30 smithi115 bash[46934]: cluster 2022-04-22T17:46:30.227090+0000 mgr.y (mgr.34107) 752 : cluster [DBG] pgmap v1389: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:30 smithi121 bash[36902]: cluster 2022-04-22T17:46:29.519643+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1054: 8 total, 8 up, 8 in 2022-04-22T17:46:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:30 smithi121 bash[36902]: audit 2022-04-22T17:46:29.524408+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:30 smithi121 bash[36902]: audit 2022-04-22T17:46:29.762818+0000 mon.a (mon.0) 2115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:30 smithi121 bash[36902]: audit 2022-04-22T17:46:30.038411+0000 mon.a (mon.0) 2116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:30 smithi121 bash[36902]: cluster 2022-04-22T17:46:30.227090+0000 mgr.y (mgr.34107) 752 : cluster [DBG] pgmap v1389: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:31.554 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsSDK.TestSDKCoverageWrite (3001 ms) 2022-04-22T17:46:31.554 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] ClsSDK.TestSDKCoverageReplay 2022-04-22T17:46:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:31 smithi121 bash[36902]: audit 2022-04-22T17:46:30.532570+0000 mon.a (mon.0) 2117 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:31 smithi121 bash[36902]: cluster 2022-04-22T17:46:30.532754+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e1055: 8 total, 8 up, 8 in 2022-04-22T17:46:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:31 smithi115 bash[46934]: audit 2022-04-22T17:46:30.532570+0000 mon.a (mon.0) 2117 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:31 smithi115 bash[46934]: cluster 2022-04-22T17:46:30.532754+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e1055: 8 total, 8 up, 8 in 2022-04-22T17:46:32.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:31.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:32.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:31 smithi115 bash[45495]: audit 2022-04-22T17:46:30.532570+0000 mon.a (mon.0) 2117 : audit [INF] from='client.? 172.21.15.115:0/393076719' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:31 smithi115 bash[45495]: cluster 2022-04-22T17:46:30.532754+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e1055: 8 total, 8 up, 8 in 2022-04-22T17:46:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:32 smithi121 bash[36902]: cluster 2022-04-22T17:46:31.537716+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e1056: 8 total, 8 up, 8 in 2022-04-22T17:46:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:32 smithi121 bash[36902]: audit 2022-04-22T17:46:32.105715+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:32 smithi121 bash[36902]: audit 2022-04-22T17:46:32.143801+0000 mon.a (mon.0) 2121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:32 smithi121 bash[36902]: cluster 2022-04-22T17:46:32.228123+0000 mgr.y (mgr.34107) 753 : cluster [DBG] pgmap v1392: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:32 smithi121 bash[36902]: audit 2022-04-22T17:46:32.440115+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[45495]: cluster 2022-04-22T17:46:31.537716+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e1056: 8 total, 8 up, 8 in 2022-04-22T17:46:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[45495]: audit 2022-04-22T17:46:32.105715+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[45495]: audit 2022-04-22T17:46:32.143801+0000 mon.a (mon.0) 2121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[45495]: cluster 2022-04-22T17:46:32.228123+0000 mgr.y (mgr.34107) 753 : cluster [DBG] pgmap v1392: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[45495]: audit 2022-04-22T17:46:32.440115+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[46934]: cluster 2022-04-22T17:46:31.537716+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e1056: 8 total, 8 up, 8 in 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[46934]: audit 2022-04-22T17:46:32.105715+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[46934]: audit 2022-04-22T17:46:32.143801+0000 mon.a (mon.0) 2121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[46934]: cluster 2022-04-22T17:46:32.228123+0000 mgr.y (mgr.34107) 753 : cluster [DBG] pgmap v1392: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:32 smithi115 bash[46934]: audit 2022-04-22T17:46:32.440115+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:33 smithi121 bash[36902]: cluster 2022-04-22T17:46:32.558768+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2022-04-22T17:46:34.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:33 smithi121 bash[36902]: audit 2022-04-22T17:46:32.561610+0000 mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.115:0/1027086611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:33 smithi121 bash[36902]: audit 2022-04-22T17:46:32.562131+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:33.551Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[45495]: cluster 2022-04-22T17:46:32.558768+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2022-04-22T17:46:34.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[45495]: audit 2022-04-22T17:46:32.561610+0000 mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.115:0/1027086611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[45495]: audit 2022-04-22T17:46:32.562131+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[46934]: cluster 2022-04-22T17:46:32.558768+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1057: 8 total, 8 up, 8 in 2022-04-22T17:46:34.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[46934]: audit 2022-04-22T17:46:32.561610+0000 mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.115:0/1027086611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:33 smithi115 bash[46934]: audit 2022-04-22T17:46:32.562131+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T17:46:34.599 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] ClsSDK.TestSDKCoverageReplay (3046 ms) 2022-04-22T17:46:34.600 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from ClsSDK (6047 ms total) 2022-04-22T17:46:34.600 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T17:46:34.600 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T17:46:34.600 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 2 tests from 1 test suite ran. (6047 ms total) 2022-04-22T17:46:34.600 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 2 tests. 2022-04-22T17:46:34.602 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T17:46:34.602 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T17:46:34.660 INFO:tasks.workunit:Stopping ['cls'] on client.0... 2022-04-22T17:46:34.661 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-04-22T17:46:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[45495]: audit 2022-04-22T17:46:33.564056+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[45495]: cluster 2022-04-22T17:46:33.564421+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2022-04-22T17:46:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[45495]: audit 2022-04-22T17:46:33.978256+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:34.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[45495]: cluster 2022-04-22T17:46:34.229025+0000 mgr.y (mgr.34107) 754 : cluster [DBG] pgmap v1395: 137 pgs: 6 creating+peering, 23 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[46934]: audit 2022-04-22T17:46:33.564056+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[46934]: cluster 2022-04-22T17:46:33.564421+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2022-04-22T17:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[46934]: audit 2022-04-22T17:46:33.978256+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:34 smithi115 bash[46934]: cluster 2022-04-22T17:46:34.229025+0000 mgr.y (mgr.34107) 754 : cluster [DBG] pgmap v1395: 137 pgs: 6 creating+peering, 23 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:34.886 DEBUG:teuthology.parallel:result is None 2022-04-22T17:46:34.887 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T17:46:34.900 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T17:46:34.901 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T17:46:34.967 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T17:46:34.967 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T17:46:34.967 INFO:teuthology.task.print:**** done end rados_api.yaml 2022-04-22T17:46:34.967 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T17:46:34.967 INFO:teuthology.task.print:**** done start rados_loadgenbig.yaml 2022-04-22T17:46:34.968 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2022-04-22T17:46:34.969 INFO:tasks.workunit:Pulling workunits from ref pacific 2022-04-22T17:46:34.970 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-22T17:46:34.971 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.0 2022-04-22T17:46:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T17:46:34.978 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-04-22T17:46:34.978 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T17:46:34.985 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-04-22T17:46:34.985 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-04-22T17:46:35.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:34 smithi121 bash[36902]: audit 2022-04-22T17:46:33.564056+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi115-64534-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T17:46:35.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:34 smithi121 bash[36902]: cluster 2022-04-22T17:46:33.564421+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e1058: 8 total, 8 up, 8 in 2022-04-22T17:46:35.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:34 smithi121 bash[36902]: audit 2022-04-22T17:46:33.978256+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:46:35.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:34 smithi121 bash[36902]: cluster 2022-04-22T17:46:34.229025+0000 mgr.y (mgr.34107) 754 : cluster [DBG] pgmap v1395: 137 pgs: 6 creating+peering, 23 unknown, 108 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:35.035 INFO:tasks.workunit:timeout=3h 2022-04-22T17:46:35.035 INFO:tasks.workunit:cleanup=True 2022-04-22T17:46:35.035 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2022-04-22T17:46:35.044 INFO:tasks.workunit.client.0.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-04-22T17:46:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:35 smithi121 bash[36902]: cluster 2022-04-22T17:46:34.579308+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2022-04-22T17:46:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:35 smithi121 bash[36902]: audit 2022-04-22T17:46:34.765528+0000 mon.a (mon.0) 2129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:35 smithi121 bash[36902]: audit 2022-04-22T17:46:35.041975+0000 mon.a (mon.0) 2130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[45495]: cluster 2022-04-22T17:46:34.579308+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2022-04-22T17:46:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[45495]: audit 2022-04-22T17:46:34.765528+0000 mon.a (mon.0) 2129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[45495]: audit 2022-04-22T17:46:35.041975+0000 mon.a (mon.0) 2130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[46934]: cluster 2022-04-22T17:46:34.579308+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e1059: 8 total, 8 up, 8 in 2022-04-22T17:46:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[46934]: audit 2022-04-22T17:46:34.765528+0000 mon.a (mon.0) 2129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:35 smithi115 bash[46934]: audit 2022-04-22T17:46:35.041975+0000 mon.a (mon.0) 2130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:36 smithi121 bash[36902]: cluster 2022-04-22T17:46:36.230052+0000 mgr.y (mgr.34107) 755 : cluster [DBG] pgmap v1397: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:36 smithi115 bash[45495]: cluster 2022-04-22T17:46:36.230052+0000 mgr.y (mgr.34107) 755 : cluster [DBG] pgmap v1397: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:36 smithi115 bash[46934]: cluster 2022-04-22T17:46:36.230052+0000 mgr.y (mgr.34107) 755 : cluster [DBG] pgmap v1397: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:37 smithi121 bash[36902]: audit 2022-04-22T17:46:37.146769+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:37 smithi115 bash[45495]: audit 2022-04-22T17:46:37.146769+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:37 smithi115 bash[46934]: audit 2022-04-22T17:46:37.146769+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:38 smithi121 bash[36902]: cluster 2022-04-22T17:46:38.231193+0000 mgr.y (mgr.34107) 756 : cluster [DBG] pgmap v1398: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:38 smithi115 bash[46934]: cluster 2022-04-22T17:46:38.231193+0000 mgr.y (mgr.34107) 756 : cluster [DBG] pgmap v1398: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:38 smithi115 bash[45495]: cluster 2022-04-22T17:46:38.231193+0000 mgr.y (mgr.34107) 756 : cluster [DBG] pgmap v1398: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:39 smithi121 bash[36902]: audit 2022-04-22T17:46:39.767642+0000 mon.a (mon.0) 2132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:39 smithi115 bash[45495]: audit 2022-04-22T17:46:39.767642+0000 mon.a (mon.0) 2132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:39 smithi115 bash[46934]: audit 2022-04-22T17:46:39.767642+0000 mon.a (mon.0) 2132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:40 smithi121 bash[36902]: audit 2022-04-22T17:46:40.045260+0000 mon.a (mon.0) 2133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:40 smithi121 bash[36902]: cluster 2022-04-22T17:46:40.232260+0000 mgr.y (mgr.34107) 757 : cluster [DBG] pgmap v1399: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:40 smithi115 bash[45495]: audit 2022-04-22T17:46:40.045260+0000 mon.a (mon.0) 2133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:40 smithi115 bash[45495]: cluster 2022-04-22T17:46:40.232260+0000 mgr.y (mgr.34107) 757 : cluster [DBG] pgmap v1399: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:41.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:40 smithi115 bash[46934]: audit 2022-04-22T17:46:40.045260+0000 mon.a (mon.0) 2133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:40 smithi115 bash[46934]: cluster 2022-04-22T17:46:40.232260+0000 mgr.y (mgr.34107) 757 : cluster [DBG] pgmap v1399: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:42.033 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-04-22T17:46:42.040 INFO:tasks.workunit.client.0.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-22T17:46:42.043 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T17:46:42.043 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-22T17:46:42.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:41.666Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:42.110 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-22T17:46:42.174 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-22T17:46:42.230 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T17:46:42.233 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T17:46:42.233 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-22T17:46:42.291 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T17:46:42.296 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T17:46:42.296 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-04-22T17:46:42.347 INFO:tasks.workunit:Running workunits matching rados/load-gen-big.sh on client.0... 2022-04-22T17:46:42.348 INFO:tasks.workunit:Running workunit rados/load-gen-big.sh... 2022-04-22T17:46:42.348 DEBUG:teuthology.orchestra.run.smithi115:workunit test rados/load-gen-big.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-big.sh 2022-04-22T17:46:42.408 INFO:tasks.workunit.client.0.smithi115.stdout:run length 1200 seconds 2022-04-22T17:46:42.408 INFO:tasks.workunit.client.0.smithi115.stdout:preparing 10240 objects 2022-04-22T17:46:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:42 smithi121 bash[36902]: audit 2022-04-22T17:46:42.149719+0000 mon.a (mon.0) 2134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:42.960 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:42 smithi115 bash[45495]: audit 2022-04-22T17:46:42.149719+0000 mon.a (mon.0) 2134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:42.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:42 smithi115 bash[46934]: audit 2022-04-22T17:46:42.149719+0000 mon.a (mon.0) 2134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:43 smithi121 bash[36902]: cluster 2022-04-22T17:46:42.233307+0000 mgr.y (mgr.34107) 758 : cluster [DBG] pgmap v1400: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:43 smithi115 bash[45495]: cluster 2022-04-22T17:46:42.233307+0000 mgr.y (mgr.34107) 758 : cluster [DBG] pgmap v1400: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:43 smithi115 bash[46934]: cluster 2022-04-22T17:46:42.233307+0000 mgr.y (mgr.34107) 758 : cluster [DBG] pgmap v1400: 105 pgs: 105 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2022-04-22T17:46:44.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:43.549Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:44.413 INFO:tasks.workunit.client.0.smithi115.stdout:load-gen will run 1200 seconds 2022-04-22T17:46:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:44 smithi121 bash[36902]: cluster 2022-04-22T17:46:44.233980+0000 mgr.y (mgr.34107) 759 : cluster [DBG] pgmap v1401: 105 pgs: 105 active+clean; 32 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s wr, 260 op/s 2022-04-22T17:46:44.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:44 smithi115 bash[45495]: cluster 2022-04-22T17:46:44.233980+0000 mgr.y (mgr.34107) 759 : cluster [DBG] pgmap v1401: 105 pgs: 105 active+clean; 32 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s wr, 260 op/s 2022-04-22T17:46:44.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:44 smithi115 bash[46934]: cluster 2022-04-22T17:46:44.233980+0000 mgr.y (mgr.34107) 759 : cluster [DBG] pgmap v1401: 105 pgs: 105 active+clean; 32 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s wr, 260 op/s 2022-04-22T17:46:45.413 INFO:tasks.workunit.client.0.smithi115.stdout: 1: throughput=0MB/sec pending data=0 2022-04-22T17:46:45.414 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uSby4P7QxQW-GYh off=2446380 len=1077272 2022-04-22T17:46:45.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 0 completed, throughput=1.02MB/sec 2022-04-22T17:46:45.422 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0GN6jiuAXLvDIGG off=5092494 len=774685 2022-04-22T17:46:45.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 1 completed, throughput=1.74MB/sec 2022-04-22T17:46:45.426 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lXW3_JeQu5tJvIU off=19483754 len=1728311 2022-04-22T17:46:45.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 2 completed, throughput=3.34MB/sec 2022-04-22T17:46:45.437 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ImT07d1DpdhHDdK off=34268 len=684188 2022-04-22T17:46:45.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 3 completed, throughput=3.96MB/sec 2022-04-22T17:46:45.439 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hR6D8PLyRw2u5lv off=13633936 len=1665265 2022-04-22T17:46:45.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 4 completed, throughput=5.49MB/sec 2022-04-22T17:46:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:45 smithi121 bash[36902]: audit 2022-04-22T17:46:44.770711+0000 mon.a (mon.0) 2135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:45.808 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:45 smithi121 bash[36902]: audit 2022-04-22T17:46:45.048450+0000 mon.a (mon.0) 2136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:45.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:45 smithi115 bash[45495]: audit 2022-04-22T17:46:44.770711+0000 mon.a (mon.0) 2135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:45.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:45 smithi115 bash[45495]: audit 2022-04-22T17:46:45.048450+0000 mon.a (mon.0) 2136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:45.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:45 smithi115 bash[46934]: audit 2022-04-22T17:46:44.770711+0000 mon.a (mon.0) 2135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:45.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:45 smithi115 bash[46934]: audit 2022-04-22T17:46:45.048450+0000 mon.a (mon.0) 2136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:46.444 INFO:tasks.workunit.client.0.smithi115.stdout: 2: throughput=2.79MB/sec pending data=0 2022-04-22T17:46:46.444 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eGqer3AWQHAJukk off=1061924 len=726218 2022-04-22T17:46:46.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 5 completed, throughput=3.12MB/sec 2022-04-22T17:46:46.446 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yFTuH2IaytscisX off=1267580 len=1519335 2022-04-22T17:46:46.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 6 completed, throughput=3.82MB/sec 2022-04-22T17:46:46.456 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9TIo9zJLipQLbNO off=21761974 len=2006584 2022-04-22T17:46:46.460 INFO:tasks.workunit.client.0.smithi115.stdout:op 7 completed, throughput=4.74MB/sec 2022-04-22T17:46:46.460 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M0jEPh8qIQIChdp off=10904237 len=316947 2022-04-22T17:46:46.465 INFO:tasks.workunit.client.0.smithi115.stdout:op 8 completed, throughput=4.88MB/sec 2022-04-22T17:46:46.465 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UN4Y8CXj3_ekD1b off=11198475 len=1387649 2022-04-22T17:46:46.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 9 completed, throughput=5.52MB/sec 2022-04-22T17:46:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:46 smithi121 bash[36902]: cluster 2022-04-22T17:46:46.235113+0000 mgr.y (mgr.34107) 760 : cluster [DBG] pgmap v1402: 105 pgs: 105 active+clean; 95 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 749 KiB/s wr, 658 op/s 2022-04-22T17:46:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:46 smithi115 bash[45495]: cluster 2022-04-22T17:46:46.235113+0000 mgr.y (mgr.34107) 760 : cluster [DBG] pgmap v1402: 105 pgs: 105 active+clean; 95 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 749 KiB/s wr, 658 op/s 2022-04-22T17:46:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:46 smithi115 bash[46934]: cluster 2022-04-22T17:46:46.235113+0000 mgr.y (mgr.34107) 760 : cluster [DBG] pgmap v1402: 105 pgs: 105 active+clean; 95 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 749 KiB/s wr, 658 op/s 2022-04-22T17:46:47.467 INFO:tasks.workunit.client.0.smithi115.stdout: 3: throughput=3.71MB/sec pending data=0 2022-04-22T17:46:47.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6kZDl7SpnXXtkcL off=1071933 len=56557 2022-04-22T17:46:47.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 10 completed, throughput=3.73MB/sec 2022-04-22T17:46:47.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zAhJ6cvFLDBfkVI off=2927835 len=1900541 2022-04-22T17:46:47.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 11 completed, throughput=4.31MB/sec 2022-04-22T17:46:47.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vd9Mk39RmzPGAA4 off=10753793 len=285839 2022-04-22T17:46:47.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 12 completed, throughput=4.39MB/sec 2022-04-22T17:46:47.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SLTeycdSFHh6w8A off=6098319 len=1357908 2022-04-22T17:46:47.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 13 completed, throughput=4.81MB/sec 2022-04-22T17:46:47.518 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5zkWy1OfwFQZHw7 off=1117260 len=1155715 2022-04-22T17:46:47.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 14 completed, throughput=5.16MB/sec 2022-04-22T17:46:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:47 smithi121 bash[36902]: audit 2022-04-22T17:46:47.152520+0000 mon.a (mon.0) 2137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:47 smithi115 bash[45495]: audit 2022-04-22T17:46:47.152520+0000 mon.a (mon.0) 2137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:47 smithi115 bash[46934]: audit 2022-04-22T17:46:47.152520+0000 mon.a (mon.0) 2137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:48.492 INFO:tasks.workunit.client.0.smithi115.stdout: 4: throughput=3.89MB/sec pending data=0 2022-04-22T17:46:48.492 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SUI2UnWOIr0UXfe off=1789635 len=1009413 2022-04-22T17:46:48.495 INFO:tasks.workunit.client.0.smithi115.stdout:op 15 completed, throughput=4.12MB/sec 2022-04-22T17:46:48.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dQIgn7JkMbu2akC off=12557369 len=1834837 2022-04-22T17:46:48.499 INFO:tasks.workunit.client.0.smithi115.stdout:op 16 completed, throughput=4.55MB/sec 2022-04-22T17:46:48.499 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RcQA_pTGuNLP2Lr off=1834916 len=1169377 2022-04-22T17:46:48.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 17 completed, throughput=4.81MB/sec 2022-04-22T17:46:48.508 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5xEAGi0JoxvvWtV off=0 len=1419490 2022-04-22T17:46:48.514 INFO:tasks.workunit.client.0.smithi115.stdout:op 18 completed, throughput=5.13MB/sec 2022-04-22T17:46:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:48 smithi121 bash[36902]: cluster 2022-04-22T17:46:48.236326+0000 mgr.y (mgr.34107) 761 : cluster [DBG] pgmap v1403: 105 pgs: 105 active+clean; 124 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 842 op/s 2022-04-22T17:46:48.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:48 smithi115 bash[45495]: cluster 2022-04-22T17:46:48.236326+0000 mgr.y (mgr.34107) 761 : cluster [DBG] pgmap v1403: 105 pgs: 105 active+clean; 124 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 842 op/s 2022-04-22T17:46:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:48 smithi115 bash[46934]: cluster 2022-04-22T17:46:48.236326+0000 mgr.y (mgr.34107) 761 : cluster [DBG] pgmap v1403: 105 pgs: 105 active+clean; 124 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 842 op/s 2022-04-22T17:46:49.515 INFO:tasks.workunit.client.0.smithi115.stdout: 5: throughput=4.13MB/sec pending data=0 2022-04-22T17:46:49.515 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cEdmvwQaXjbQaEo off=6396180 len=927802 2022-04-22T17:46:49.522 INFO:tasks.workunit.client.0.smithi115.stdout:op 19 completed, throughput=4.29MB/sec 2022-04-22T17:46:49.523 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RaB7LY0VPQ8i_6l off=3940209 len=490808 2022-04-22T17:46:49.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 20 completed, throughput=4.38MB/sec 2022-04-22T17:46:49.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wcW1ZXtwJ8e-d8p off=872133 len=1998489 2022-04-22T17:46:49.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 21 completed, throughput=4.75MB/sec 2022-04-22T17:46:49.533 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q8v5qSi_dV9qi74 off=674654 len=399596 2022-04-22T17:46:49.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 22 completed, throughput=4.82MB/sec 2022-04-22T17:46:49.535 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KaUT3YwWPPMcvEf off=13947718 len=591450 2022-04-22T17:46:49.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 23 completed, throughput=4.93MB/sec 2022-04-22T17:46:49.542 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DZWnB58RRlXr9Gp off=1199007 len=959240 2022-04-22T17:46:49.543 INFO:tasks.workunit.client.0.smithi115.stdout:op 24 completed, throughput=5.1MB/sec 2022-04-22T17:46:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:49 smithi121 bash[36902]: audit 2022-04-22T17:46:49.773686+0000 mon.a (mon.0) 2138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:50.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:49 smithi115 bash[45495]: audit 2022-04-22T17:46:49.773686+0000 mon.a (mon.0) 2138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:49 smithi115 bash[46934]: audit 2022-04-22T17:46:49.773686+0000 mon.a (mon.0) 2138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:50.545 INFO:tasks.workunit.client.0.smithi115.stdout: 6: throughput=4.27MB/sec pending data=0 2022-04-22T17:46:50.545 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5mOzWO0hV-A6U_i off=4148336 len=1630706 2022-04-22T17:46:50.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 25 completed, throughput=4.52MB/sec 2022-04-22T17:46:50.553 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iVPLmslBti0oKV5 off=3048683 len=127946 2022-04-22T17:46:50.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 26 completed, throughput=4.54MB/sec 2022-04-22T17:46:50.553 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s6hb-rD1NbAag_p off=809183 len=1865490 2022-04-22T17:46:50.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 27 completed, throughput=4.82MB/sec 2022-04-22T17:46:50.564 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DS6YXlIctUJ9WcK off=10306395 len=2096301 2022-04-22T17:46:50.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 28 completed, throughput=5.13MB/sec 2022-04-22T17:46:51.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:50 smithi121 bash[36902]: audit 2022-04-22T17:46:50.051774+0000 mon.a (mon.0) 2139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:51.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:50 smithi121 bash[36902]: cluster 2022-04-22T17:46:50.237411+0000 mgr.y (mgr.34107) 762 : cluster [DBG] pgmap v1404: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 854 op/s 2022-04-22T17:46:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:50 smithi115 bash[46934]: audit 2022-04-22T17:46:50.051774+0000 mon.a (mon.0) 2139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:50 smithi115 bash[46934]: cluster 2022-04-22T17:46:50.237411+0000 mgr.y (mgr.34107) 762 : cluster [DBG] pgmap v1404: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 854 op/s 2022-04-22T17:46:51.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:50 smithi115 bash[45495]: audit 2022-04-22T17:46:50.051774+0000 mon.a (mon.0) 2139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:51.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:50 smithi115 bash[45495]: cluster 2022-04-22T17:46:50.237411+0000 mgr.y (mgr.34107) 762 : cluster [DBG] pgmap v1404: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 854 op/s 2022-04-22T17:46:51.575 INFO:tasks.workunit.client.0.smithi115.stdout: 7: throughput=4.42MB/sec pending data=0 2022-04-22T17:46:51.576 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XbFE4AWftqofaWC off=0 len=1126982 2022-04-22T17:46:51.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 29 completed, throughput=4.56MB/sec 2022-04-22T17:46:51.577 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dXVqZxxrIshdl12 off=5580025 len=2094374 2022-04-22T17:46:51.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 30 completed, throughput=4.84MB/sec 2022-04-22T17:46:51.585 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fBwk0lsUxn3ieew off=16173428 len=879815 2022-04-22T17:46:51.591 INFO:tasks.workunit.client.0.smithi115.stdout:op 31 completed, throughput=4.95MB/sec 2022-04-22T17:46:51.591 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FcmbKGkgIzeWSlF off=1218829 len=857845 2022-04-22T17:46:51.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 32 completed, throughput=5.06MB/sec 2022-04-22T17:46:51.778 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:46:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:51.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:52.119 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (9m) 9m ago 23m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (23m) 20s ago 23m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (11m) 9m ago 24m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (11m) 20s ago 24m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (19m) 20s ago 28m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (19m) 9m ago 31m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (18m) 9m ago 31m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (17m) 20s ago 29m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:46:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (18m) 9m ago 29m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (11m) 9m ago 23m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (10m) 20s ago 23m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (16m) 9m ago 28m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (15m) 9m ago 28m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (15m) 9m ago 27m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (15m) 9m ago 27m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (14m) 20s ago 26m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (13m) 20s ago 26m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (13m) 20s ago 25m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:46:52.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (12m) 20s ago 25m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:46:52.122 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (10m) 20s ago 24m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:46:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:52 smithi121 bash[36902]: audit 2022-04-22T17:46:52.155518+0000 mon.a (mon.0) 2140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:52.523 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:46:52.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:46:52.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:46:52.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:52.524 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:46:52.524 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:46:52.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:46:52.525 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:46:52.526 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:46:52.526 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:46:52.526 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:46:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:52 smithi115 bash[45495]: audit 2022-04-22T17:46:52.155518+0000 mon.a (mon.0) 2140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:52.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:52 smithi115 bash[46934]: audit 2022-04-22T17:46:52.155518+0000 mon.a (mon.0) 2140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:52.596 INFO:tasks.workunit.client.0.smithi115.stdout: 8: throughput=4.44MB/sec pending data=0 2022-04-22T17:46:52.596 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ByKAfZbx5XxOOSt off=5148371 len=1936750 2022-04-22T17:46:52.605 INFO:tasks.workunit.client.0.smithi115.stdout:op 33 completed, throughput=4.66MB/sec 2022-04-22T17:46:52.658 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KMlR61x2XMW0zsL off=574678 len=1152360 2022-04-22T17:46:52.659 INFO:tasks.workunit.client.0.smithi115.stdout:op 34 completed, throughput=4.8MB/sec 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iwokGmdM1W0F2VF off=4408330 len=407966 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 35 completed, throughput=4.84MB/sec 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pzigmsy33BTE8pH off=1098138 len=738597 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 36 completed, throughput=4.93MB/sec 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-udqFfI962Y5Lcbr off=1271514 len=406064 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 37 completed, throughput=4.97MB/sec 2022-04-22T17:46:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0qOv0aozYCKzqBq off=5722784 len=1817520 2022-04-22T17:46:52.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 38 completed, throughput=5.18MB/sec 2022-04-22T17:46:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:53 smithi121 bash[36902]: audit 2022-04-22T17:46:51.753918+0000 mgr.y (mgr.34107) 763 : audit [DBG] from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:53 smithi121 bash[36902]: audit 2022-04-22T17:46:52.089236+0000 mgr.y (mgr.34107) 764 : audit [DBG] from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:53 smithi121 bash[36902]: cluster 2022-04-22T17:46:52.238477+0000 mgr.y (mgr.34107) 765 : cluster [DBG] pgmap v1405: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 854 op/s 2022-04-22T17:46:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:53 smithi121 bash[36902]: audit 2022-04-22T17:46:52.513440+0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.115:0/4103055277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[45495]: audit 2022-04-22T17:46:51.753918+0000 mgr.y (mgr.34107) 763 : audit [DBG] from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[45495]: audit 2022-04-22T17:46:52.089236+0000 mgr.y (mgr.34107) 764 : audit [DBG] from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[45495]: cluster 2022-04-22T17:46:52.238477+0000 mgr.y (mgr.34107) 765 : cluster [DBG] pgmap v1405: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 854 op/s 2022-04-22T17:46:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[45495]: audit 2022-04-22T17:46:52.513440+0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.115:0/4103055277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[46934]: audit 2022-04-22T17:46:51.753918+0000 mgr.y (mgr.34107) 763 : audit [DBG] from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[46934]: audit 2022-04-22T17:46:52.089236+0000 mgr.y (mgr.34107) 764 : audit [DBG] from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:46:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[46934]: cluster 2022-04-22T17:46:52.238477+0000 mgr.y (mgr.34107) 765 : cluster [DBG] pgmap v1405: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 854 op/s 2022-04-22T17:46:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[46934]: audit 2022-04-22T17:46:52.513440+0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.115:0/4103055277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:46:53.619 INFO:tasks.workunit.client.0.smithi115.stdout: 9: throughput=4.62MB/sec pending data=0 2022-04-22T17:46:53.626 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ywW3HThEp8DRtD8 off=8422049 len=1489363 2022-04-22T17:46:53.627 INFO:tasks.workunit.client.0.smithi115.stdout:op 39 completed, throughput=4.77MB/sec 2022-04-22T17:46:53.627 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ILlLAmu9ogIUgqS off=4633287 len=502081 2022-04-22T17:46:53.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 40 completed, throughput=4.82MB/sec 2022-04-22T17:46:53.629 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Yi7wHq9OtcYu803 off=9914808 len=1432043 2022-04-22T17:46:53.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 41 completed, throughput=4.97MB/sec 2022-04-22T17:46:53.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNIsKmv_6cDWE3e off=1918624 len=1476270 2022-04-22T17:46:53.636 INFO:tasks.workunit.client.0.smithi115.stdout:op 42 completed, throughput=5.12MB/sec 2022-04-22T17:46:54.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:46:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:46:53.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:46:54.637 INFO:tasks.workunit.client.0.smithi115.stdout: 10: throughput=4.62MB/sec pending data=0 2022-04-22T17:46:54.637 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-j73E0KVAARoFEsp off=5099827 len=128834 2022-04-22T17:46:54.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 43 completed, throughput=4.63MB/sec 2022-04-22T17:46:54.641 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k5xan6TXqIkEyw6 off=7118764 len=1413585 2022-04-22T17:46:54.649 INFO:tasks.workunit.client.0.smithi115.stdout:op 44 completed, throughput=4.75MB/sec 2022-04-22T17:46:54.649 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Dtsvn-UK7tXJSS7 off=7391442 len=1255085 2022-04-22T17:46:54.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 45 completed, throughput=4.87MB/sec 2022-04-22T17:46:54.657 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ObGjrVAtI0JMCpB off=9391582 len=1073052 2022-04-22T17:46:54.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 46 completed, throughput=4.96MB/sec 2022-04-22T17:46:54.664 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BgmNxRh5suNf7DH off=924811 len=326229 2022-04-22T17:46:54.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 47 completed, throughput=4.99MB/sec 2022-04-22T17:46:54.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--1Cd-565flUDtNp off=0 len=1347428 2022-04-22T17:46:54.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 48 completed, throughput=5.12MB/sec 2022-04-22T17:46:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:54 smithi121 bash[36902]: cluster 2022-04-22T17:46:54.239626+0000 mgr.y (mgr.34107) 766 : cluster [DBG] pgmap v1406: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 855 op/s 2022-04-22T17:46:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:54 smithi115 bash[45495]: cluster 2022-04-22T17:46:54.239626+0000 mgr.y (mgr.34107) 766 : cluster [DBG] pgmap v1406: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 855 op/s 2022-04-22T17:46:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:54 smithi115 bash[46934]: cluster 2022-04-22T17:46:54.239626+0000 mgr.y (mgr.34107) 766 : cluster [DBG] pgmap v1406: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 855 op/s 2022-04-22T17:46:55.667 INFO:tasks.workunit.client.0.smithi115.stdout: 11: throughput=4.66MB/sec pending data=0 2022-04-22T17:46:55.667 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-90GPp1WAyFrUTrs off=791249 len=841799 2022-04-22T17:46:55.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 49 completed, throughput=4.73MB/sec 2022-04-22T17:46:55.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tSmCWmhIW834quo off=4470492 len=1302513 2022-04-22T17:46:55.678 INFO:tasks.workunit.client.0.smithi115.stdout:op 50 completed, throughput=4.84MB/sec 2022-04-22T17:46:55.678 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wWHutWH3BVmryhH off=4886256 len=2038707 2022-04-22T17:46:55.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 51 completed, throughput=5.01MB/sec 2022-04-22T17:46:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:55 smithi121 bash[36902]: audit 2022-04-22T17:46:54.776833+0000 mon.a (mon.0) 2141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:55 smithi121 bash[36902]: audit 2022-04-22T17:46:55.055121+0000 mon.a (mon.0) 2142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:55 smithi115 bash[45495]: audit 2022-04-22T17:46:54.776833+0000 mon.a (mon.0) 2141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:55 smithi115 bash[45495]: audit 2022-04-22T17:46:55.055121+0000 mon.a (mon.0) 2142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:55 smithi115 bash[46934]: audit 2022-04-22T17:46:54.776833+0000 mon.a (mon.0) 2141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:46:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:55 smithi115 bash[46934]: audit 2022-04-22T17:46:55.055121+0000 mon.a (mon.0) 2142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:46:56.684 INFO:tasks.workunit.client.0.smithi115.stdout: 12: throughput=4.6MB/sec pending data=0 2022-04-22T17:46:56.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hC68UJ-8N4kRiXA off=10678489 len=1503806 2022-04-22T17:46:56.688 INFO:tasks.workunit.client.0.smithi115.stdout:op 52 completed, throughput=4.72MB/sec 2022-04-22T17:46:56.688 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tc85rPSAF_up8CW off=11878934 len=1889764 2022-04-22T17:46:56.697 INFO:tasks.workunit.client.0.smithi115.stdout:op 53 completed, throughput=4.86MB/sec 2022-04-22T17:46:56.697 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RFWRF5hWwkpSHaN off=1037382 len=1252810 2022-04-22T17:46:56.702 INFO:tasks.workunit.client.0.smithi115.stdout:op 54 completed, throughput=4.96MB/sec 2022-04-22T17:46:56.703 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RayionCBkCrHCr3 off=3375198 len=1012379 2022-04-22T17:46:56.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 55 completed, throughput=5.03MB/sec 2022-04-22T17:46:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:56 smithi121 bash[36902]: cluster 2022-04-22T17:46:56.240625+0000 mgr.y (mgr.34107) 767 : cluster [DBG] pgmap v1407: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 639 op/s 2022-04-22T17:46:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:56 smithi115 bash[45495]: cluster 2022-04-22T17:46:56.240625+0000 mgr.y (mgr.34107) 767 : cluster [DBG] pgmap v1407: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 639 op/s 2022-04-22T17:46:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:56 smithi115 bash[46934]: cluster 2022-04-22T17:46:56.240625+0000 mgr.y (mgr.34107) 767 : cluster [DBG] pgmap v1407: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 639 op/s 2022-04-22T17:46:57.709 INFO:tasks.workunit.client.0.smithi115.stdout: 13: throughput=4.65MB/sec pending data=0 2022-04-22T17:46:57.710 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QB7oPtuRSmDG5Lq off=6981947 len=1318608 2022-04-22T17:46:57.713 INFO:tasks.workunit.client.0.smithi115.stdout:op 56 completed, throughput=4.75MB/sec 2022-04-22T17:46:57.713 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-N9AFZACJKaJN18X off=224467 len=1750797 2022-04-22T17:46:57.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 57 completed, throughput=4.87MB/sec 2022-04-22T17:46:57.725 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7DhgypeNIfZdZXs off=10224247 len=1821483 2022-04-22T17:46:57.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 58 completed, throughput=5MB/sec 2022-04-22T17:46:57.729 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-htyxUrxlkgu9Ioi off=16158788 len=1135616 2022-04-22T17:46:57.735 INFO:tasks.workunit.client.0.smithi115.stdout:op 59 completed, throughput=5.07MB/sec 2022-04-22T17:46:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:57 smithi121 bash[36902]: audit 2022-04-22T17:46:57.158642+0000 mon.a (mon.0) 2143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:57 smithi115 bash[46934]: audit 2022-04-22T17:46:57.158642+0000 mon.a (mon.0) 2143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:57 smithi115 bash[45495]: audit 2022-04-22T17:46:57.158642+0000 mon.a (mon.0) 2143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:46:58.736 INFO:tasks.workunit.client.0.smithi115.stdout: 14: throughput=4.72MB/sec pending data=0 2022-04-22T17:46:58.736 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mIJmBV1SRF93wYT off=4065186 len=1044156 2022-04-22T17:46:58.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 60 completed, throughput=4.79MB/sec 2022-04-22T17:46:58.744 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-enb-ZJYSrjMcOdh off=3266673 len=650892 2022-04-22T17:46:58.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 61 completed, throughput=4.83MB/sec 2022-04-22T17:46:58.750 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jzbmk99AHUUejnR off=10645419 len=1884928 2022-04-22T17:46:58.760 INFO:tasks.workunit.client.0.smithi115.stdout:op 62 completed, throughput=4.95MB/sec 2022-04-22T17:46:58.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tY_67IZuuYsk-Lq off=4513085 len=1455402 2022-04-22T17:46:58.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 63 completed, throughput=5.05MB/sec 2022-04-22T17:46:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:58 smithi121 bash[36902]: cluster 2022-04-22T17:46:58.241794+0000 mgr.y (mgr.34107) 768 : cluster [DBG] pgmap v1408: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 217 op/s 2022-04-22T17:46:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:58 smithi115 bash[45495]: cluster 2022-04-22T17:46:58.241794+0000 mgr.y (mgr.34107) 768 : cluster [DBG] pgmap v1408: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 217 op/s 2022-04-22T17:46:58.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:58 smithi115 bash[46934]: cluster 2022-04-22T17:46:58.241794+0000 mgr.y (mgr.34107) 768 : cluster [DBG] pgmap v1408: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 217 op/s 2022-04-22T17:46:59.763 INFO:tasks.workunit.client.0.smithi115.stdout: 15: throughput=4.72MB/sec pending data=0 2022-04-22T17:46:59.763 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6T1LuLavixPbEFz off=9818929 len=1251394 2022-04-22T17:46:59.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 64 completed, throughput=4.79MB/sec 2022-04-22T17:46:59.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QWRGaMeiffAV1nm off=2361960 len=1815321 2022-04-22T17:46:59.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 65 completed, throughput=4.9MB/sec 2022-04-22T17:46:59.776 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TCntVHkkrgABV73 off=0 len=1063505 2022-04-22T17:46:59.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 66 completed, throughput=4.97MB/sec 2022-04-22T17:46:59.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pb0Qu-REcTDs8dP off=11668618 len=892895 2022-04-22T17:46:59.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 67 completed, throughput=5.02MB/sec 2022-04-22T17:47:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:46:59 smithi121 bash[36902]: audit 2022-04-22T17:46:59.779958+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:00.337 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:46:59 smithi115 bash[45495]: audit 2022-04-22T17:46:59.779958+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:00.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:46:59 smithi115 bash[46934]: audit 2022-04-22T17:46:59.779958+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:00.784 INFO:tasks.workunit.client.0.smithi115.stdout: 16: throughput=4.72MB/sec pending data=0 2022-04-22T17:47:00.785 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-saYavJWh89is8SU off=9539560 len=1920485 2022-04-22T17:47:00.789 INFO:tasks.workunit.client.0.smithi115.stdout:op 68 completed, throughput=4.83MB/sec 2022-04-22T17:47:00.789 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NVFzdNMZ9-VH3t8 off=6497931 len=1634035 2022-04-22T17:47:00.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 69 completed, throughput=4.92MB/sec 2022-04-22T17:47:00.798 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gTzkd9e7dnjKMGS off=9606631 len=1872925 2022-04-22T17:47:00.802 INFO:tasks.workunit.client.0.smithi115.stdout:op 70 completed, throughput=5.03MB/sec 2022-04-22T17:47:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:00 smithi121 bash[36902]: audit 2022-04-22T17:47:00.058827+0000 mon.a (mon.0) 2145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:00 smithi121 bash[36902]: cluster 2022-04-22T17:47:00.242905+0000 mgr.y (mgr.34107) 769 : cluster [DBG] pgmap v1409: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:47:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:00 smithi115 bash[45495]: audit 2022-04-22T17:47:00.058827+0000 mon.a (mon.0) 2145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:00 smithi115 bash[45495]: cluster 2022-04-22T17:47:00.242905+0000 mgr.y (mgr.34107) 769 : cluster [DBG] pgmap v1409: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:47:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:00 smithi115 bash[46934]: audit 2022-04-22T17:47:00.058827+0000 mon.a (mon.0) 2145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:01.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:00 smithi115 bash[46934]: cluster 2022-04-22T17:47:00.242905+0000 mgr.y (mgr.34107) 769 : cluster [DBG] pgmap v1409: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2022-04-22T17:47:01.802 INFO:tasks.workunit.client.0.smithi115.stdout: 17: throughput=4.74MB/sec pending data=0 2022-04-22T17:47:01.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VlmvEXwskPl_fp2 off=7687615 len=1825988 2022-04-22T17:47:01.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 71 completed, throughput=4.83MB/sec 2022-04-22T17:47:01.812 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gtvtC0JWY89JfXI off=9004671 len=376297 2022-04-22T17:47:01.813 INFO:tasks.workunit.client.0.smithi115.stdout:op 72 completed, throughput=4.86MB/sec 2022-04-22T17:47:01.813 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cfqtukUyJ2ZYNI9 off=92891 len=464954 2022-04-22T17:47:01.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 73 completed, throughput=4.88MB/sec 2022-04-22T17:47:01.815 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xga1v9esqrIlkAY off=1911519 len=1772654 2022-04-22T17:47:01.822 INFO:tasks.workunit.client.0.smithi115.stdout:op 74 completed, throughput=4.98MB/sec 2022-04-22T17:47:01.822 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sGI6W-5hThThDVH off=16716162 len=1939828 2022-04-22T17:47:01.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 75 completed, throughput=5.08MB/sec 2022-04-22T17:47:02.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:01.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:02 smithi121 bash[36902]: audit 2022-04-22T17:47:02.161886+0000 mon.a (mon.0) 2146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:02 smithi115 bash[45495]: audit 2022-04-22T17:47:02.161886+0000 mon.a (mon.0) 2146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:02.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:02 smithi115 bash[46934]: audit 2022-04-22T17:47:02.161886+0000 mon.a (mon.0) 2146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:02.826 INFO:tasks.workunit.client.0.smithi115.stdout: 18: throughput=4.8MB/sec pending data=0 2022-04-22T17:47:02.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-07irIg9N9tfC_no off=3291766 len=1866233 2022-04-22T17:47:02.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 76 completed, throughput=4.9MB/sec 2022-04-22T17:47:02.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e4pLtAgw0UAevad off=5092580 len=1822076 2022-04-22T17:47:02.843 INFO:tasks.workunit.client.0.smithi115.stdout:op 77 completed, throughput=4.99MB/sec 2022-04-22T17:47:02.844 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bZqq-Y4OLGoLK-u off=4607367 len=1793177 2022-04-22T17:47:02.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 78 completed, throughput=5.08MB/sec 2022-04-22T17:47:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:03 smithi121 bash[36902]: cluster 2022-04-22T17:47:02.244084+0000 mgr.y (mgr.34107) 770 : cluster [DBG] pgmap v1410: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:03 smithi115 bash[45495]: cluster 2022-04-22T17:47:02.244084+0000 mgr.y (mgr.34107) 770 : cluster [DBG] pgmap v1410: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:03 smithi115 bash[46934]: cluster 2022-04-22T17:47:02.244084+0000 mgr.y (mgr.34107) 770 : cluster [DBG] pgmap v1410: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:03.853 INFO:tasks.workunit.client.0.smithi115.stdout: 19: throughput=4.82MB/sec pending data=0 2022-04-22T17:47:03.853 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SLUiox92TaN9ufu off=418675 len=1718492 2022-04-22T17:47:03.857 INFO:tasks.workunit.client.0.smithi115.stdout:op 79 completed, throughput=4.9MB/sec 2022-04-22T17:47:03.857 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SLMr5hVU6xDgOnU off=1435869 len=2035866 2022-04-22T17:47:03.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 80 completed, throughput=5MB/sec 2022-04-22T17:47:03.868 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-94ew_Xj9KYTy8Ge off=18601453 len=1424213 2022-04-22T17:47:03.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 81 completed, throughput=5.07MB/sec 2022-04-22T17:47:04.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:03.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:04 smithi121 bash[36902]: cluster 2022-04-22T17:47:04.245184+0000 mgr.y (mgr.34107) 771 : cluster [DBG] pgmap v1411: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:04 smithi115 bash[45495]: cluster 2022-04-22T17:47:04.245184+0000 mgr.y (mgr.34107) 771 : cluster [DBG] pgmap v1411: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:04.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:04 smithi115 bash[46934]: cluster 2022-04-22T17:47:04.245184+0000 mgr.y (mgr.34107) 771 : cluster [DBG] pgmap v1411: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:04.875 INFO:tasks.workunit.client.0.smithi115.stdout: 20: throughput=4.82MB/sec pending data=0 2022-04-22T17:47:04.875 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VmNYVhujmMM-hpR off=5982394 len=1877567 2022-04-22T17:47:04.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 82 completed, throughput=4.91MB/sec 2022-04-22T17:47:04.879 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gV_WnbKUJsN1BIP off=714199 len=329981 2022-04-22T17:47:04.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 83 completed, throughput=4.92MB/sec 2022-04-22T17:47:04.885 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0n-8dinhcvsaWgx off=4902306 len=739761 2022-04-22T17:47:04.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 84 completed, throughput=4.95MB/sec 2022-04-22T17:47:04.891 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pBnKUFjiY9wadlX off=2028014 len=1487522 2022-04-22T17:47:04.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 85 completed, throughput=5.02MB/sec 2022-04-22T17:47:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:05 smithi121 bash[36902]: audit 2022-04-22T17:47:04.782319+0000 mon.a (mon.0) 2147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:05 smithi121 bash[36902]: audit 2022-04-22T17:47:05.062469+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:05 smithi115 bash[45495]: audit 2022-04-22T17:47:04.782319+0000 mon.a (mon.0) 2147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:05 smithi115 bash[45495]: audit 2022-04-22T17:47:05.062469+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:05 smithi115 bash[46934]: audit 2022-04-22T17:47:04.782319+0000 mon.a (mon.0) 2147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:05 smithi115 bash[46934]: audit 2022-04-22T17:47:05.062469+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:05.897 INFO:tasks.workunit.client.0.smithi115.stdout: 21: throughput=4.79MB/sec pending data=0 2022-04-22T17:47:05.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZYhA_GAjJVkooeP off=3117582 len=1408949 2022-04-22T17:47:05.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 86 completed, throughput=4.85MB/sec 2022-04-22T17:47:05.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8YRdEre8XGhz9qt off=5999620 len=1610126 2022-04-22T17:47:05.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 87 completed, throughput=4.92MB/sec 2022-04-22T17:47:05.908 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CaDV96acgwmriWZ off=4634218 len=1187572 2022-04-22T17:47:05.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 88 completed, throughput=4.97MB/sec 2022-04-22T17:47:05.913 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dJt09tdV5RbiA-V off=4995005 len=695476 2022-04-22T17:47:05.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 89 completed, throughput=5MB/sec 2022-04-22T17:47:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:06 smithi121 bash[36902]: cluster 2022-04-22T17:47:06.246317+0000 mgr.y (mgr.34107) 772 : cluster [DBG] pgmap v1412: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:06 smithi115 bash[45495]: cluster 2022-04-22T17:47:06.246317+0000 mgr.y (mgr.34107) 772 : cluster [DBG] pgmap v1412: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:06.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:06 smithi115 bash[46934]: cluster 2022-04-22T17:47:06.246317+0000 mgr.y (mgr.34107) 772 : cluster [DBG] pgmap v1412: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:47:06.915 INFO:tasks.workunit.client.0.smithi115.stdout: 22: throughput=4.78MB/sec pending data=0 2022-04-22T17:47:06.915 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HIzIR1cI2n71p9f off=8191928 len=1527537 2022-04-22T17:47:06.920 INFO:tasks.workunit.client.0.smithi115.stdout:op 90 completed, throughput=4.84MB/sec 2022-04-22T17:47:06.920 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-54p_VC2lRVHjYJC off=3423884 len=175264 2022-04-22T17:47:06.921 INFO:tasks.workunit.client.0.smithi115.stdout:op 91 completed, throughput=4.85MB/sec 2022-04-22T17:47:06.922 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3T_qLeydR3pRyM off=9023384 len=104899 2022-04-22T17:47:06.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 92 completed, throughput=4.85MB/sec 2022-04-22T17:47:06.926 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ouViYNMeMi56Jhg off=14765844 len=1769231 2022-04-22T17:47:06.930 INFO:tasks.workunit.client.0.smithi115.stdout:op 93 completed, throughput=4.93MB/sec 2022-04-22T17:47:06.930 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gMvUBheVsopGBzS off=2662840 len=1548256 2022-04-22T17:47:06.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 94 completed, throughput=4.99MB/sec 2022-04-22T17:47:06.939 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-J-oC44j1WmloYue off=7005598 len=1056728 2022-04-22T17:47:06.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 95 completed, throughput=5.03MB/sec 2022-04-22T17:47:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:07 smithi121 bash[36902]: audit 2022-04-22T17:47:07.164951+0000 mon.a (mon.0) 2149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:07 smithi115 bash[45495]: audit 2022-04-22T17:47:07.164951+0000 mon.a (mon.0) 2149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:07 smithi115 bash[46934]: audit 2022-04-22T17:47:07.164951+0000 mon.a (mon.0) 2149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:07.945 INFO:tasks.workunit.client.0.smithi115.stdout: 23: throughput=4.82MB/sec pending data=0 2022-04-22T17:47:07.945 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lsEIb2MYQSfQjJd off=10809323 len=1994298 2022-04-22T17:47:07.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 96 completed, throughput=4.9MB/sec 2022-04-22T17:47:07.949 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P-vCy2VDyqutFc2 off=3883444 len=570380 2022-04-22T17:47:07.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 97 completed, throughput=4.92MB/sec 2022-04-22T17:47:07.952 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S-SBDx6gJ9QyEFn off=6764907 len=1282725 2022-04-22T17:47:07.960 INFO:tasks.workunit.client.0.smithi115.stdout:op 98 completed, throughput=4.97MB/sec 2022-04-22T17:47:07.960 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xt4MFhnf3RLeoZp off=16349886 len=1504315 2022-04-22T17:47:07.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 99 completed, throughput=5.03MB/sec 2022-04-22T17:47:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:08 smithi121 bash[36902]: cluster 2022-04-22T17:47:08.247516+0000 mgr.y (mgr.34107) 773 : cluster [DBG] pgmap v1413: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2022-04-22T17:47:08.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:08 smithi115 bash[45495]: cluster 2022-04-22T17:47:08.247516+0000 mgr.y (mgr.34107) 773 : cluster [DBG] pgmap v1413: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2022-04-22T17:47:08.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:08 smithi115 bash[46934]: cluster 2022-04-22T17:47:08.247516+0000 mgr.y (mgr.34107) 773 : cluster [DBG] pgmap v1413: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 3 op/s 2022-04-22T17:47:08.963 INFO:tasks.workunit.client.0.smithi115.stdout: 24: throughput=4.83MB/sec pending data=0 2022-04-22T17:47:09.007 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u9JgHqwQO0ggkHT off=2347107 len=1474317 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 100 completed, throughput=4.88MB/sec 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O4eoUEvyPwLGNYj off=7132179 len=96790 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 101 completed, throughput=4.89MB/sec 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-85zKmbHwq2RqNjH off=223173 len=956578 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 102 completed, throughput=4.92MB/sec 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k1l06KjL6u6OVj0 off=3002617 len=1720114 2022-04-22T17:47:09.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 103 completed, throughput=4.99MB/sec 2022-04-22T17:47:09.010 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IRae_k9rEKeNp0X off=2894886 len=832412 2022-04-22T17:47:09.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 104 completed, throughput=5.02MB/sec 2022-04-22T17:47:09.981 INFO:tasks.workunit.client.0.smithi115.stdout: 25: throughput=4.83MB/sec pending data=0 2022-04-22T17:47:09.981 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z6V1xvt506bq3rg off=3322114 len=355436 2022-04-22T17:47:09.983 INFO:tasks.workunit.client.0.smithi115.stdout:op 105 completed, throughput=4.84MB/sec 2022-04-22T17:47:09.983 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-isGszFHaVFQATGF off=274140 len=2064112 2022-04-22T17:47:09.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 106 completed, throughput=4.91MB/sec 2022-04-22T17:47:09.992 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9qonbz2l5en_6b- off=806853 len=1099981 2022-04-22T17:47:09.999 INFO:tasks.workunit.client.0.smithi115.stdout:op 107 completed, throughput=4.95MB/sec 2022-04-22T17:47:09.999 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--uvHgtbHwX8lbpU off=603088 len=714850 2022-04-22T17:47:10.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 108 completed, throughput=4.98MB/sec 2022-04-22T17:47:10.004 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2it9giWeCfIxyGj off=9559938 len=431212 2022-04-22T17:47:10.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 109 completed, throughput=4.99MB/sec 2022-04-22T17:47:10.006 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D_ub7eWALUWCdDD off=4588380 len=1622421 2022-04-22T17:47:10.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 110 completed, throughput=5.05MB/sec 2022-04-22T17:47:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:09 smithi121 bash[36902]: audit 2022-04-22T17:47:09.785443+0000 mon.a (mon.0) 2150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:09 smithi115 bash[45495]: audit 2022-04-22T17:47:09.785443+0000 mon.a (mon.0) 2150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:09 smithi115 bash[46934]: audit 2022-04-22T17:47:09.785443+0000 mon.a (mon.0) 2150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:11.009 INFO:tasks.workunit.client.0.smithi115.stdout: 26: throughput=4.86MB/sec pending data=0 2022-04-22T17:47:11.009 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-649yD2pvttl3pGb off=7022941 len=1902815 2022-04-22T17:47:11.012 INFO:tasks.workunit.client.0.smithi115.stdout:op 111 completed, throughput=4.93MB/sec 2022-04-22T17:47:11.012 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jks8yf73aH9El5x off=10068194 len=1463898 2022-04-22T17:47:11.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 112 completed, throughput=4.98MB/sec 2022-04-22T17:47:11.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dozoLCP7mHPEXf5 off=11205687 len=1310017 2022-04-22T17:47:11.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 113 completed, throughput=5.03MB/sec 2022-04-22T17:47:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:10 smithi121 bash[36902]: audit 2022-04-22T17:47:10.066267+0000 mon.a (mon.0) 2151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:10 smithi121 bash[36902]: cluster 2022-04-22T17:47:10.248528+0000 mgr.y (mgr.34107) 774 : cluster [DBG] pgmap v1414: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:47:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:10 smithi115 bash[45495]: audit 2022-04-22T17:47:10.066267+0000 mon.a (mon.0) 2151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:10 smithi115 bash[45495]: cluster 2022-04-22T17:47:10.248528+0000 mgr.y (mgr.34107) 774 : cluster [DBG] pgmap v1414: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:47:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:10 smithi115 bash[46934]: audit 2022-04-22T17:47:10.066267+0000 mon.a (mon.0) 2151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:10 smithi115 bash[46934]: cluster 2022-04-22T17:47:10.248528+0000 mgr.y (mgr.34107) 774 : cluster [DBG] pgmap v1414: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:47:12.022 INFO:tasks.workunit.client.0.smithi115.stdout: 27: throughput=4.85MB/sec pending data=0 2022-04-22T17:47:12.022 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0BfD79nH_w2_Kw8 off=7019031 len=785058 2022-04-22T17:47:12.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 114 completed, throughput=4.87MB/sec 2022-04-22T17:47:12.026 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m8viPClFoNA2mGj off=175546 len=1827499 2022-04-22T17:47:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 115 completed, throughput=4.94MB/sec 2022-04-22T17:47:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GCmIujKYDwUo6f- off=1747499 len=1055728 2022-04-22T17:47:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:op 116 completed, throughput=4.97MB/sec 2022-04-22T17:47:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hq3JsqWAUWYdLck off=18099777 len=576342 2022-04-22T17:47:12.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 117 completed, throughput=4.99MB/sec 2022-04-22T17:47:12.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RqNwG73w1q2oErc off=8057900 len=2071780 2022-04-22T17:47:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:11.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:12.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 118 completed, throughput=5.06MB/sec 2022-04-22T17:47:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:12 smithi121 bash[36902]: audit 2022-04-22T17:47:12.167740+0000 mon.a (mon.0) 2152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:12.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:12 smithi115 bash[45495]: audit 2022-04-22T17:47:12.167740+0000 mon.a (mon.0) 2152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:12.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:12 smithi115 bash[46934]: audit 2022-04-22T17:47:12.167740+0000 mon.a (mon.0) 2152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:13.047 INFO:tasks.workunit.client.0.smithi115.stdout: 28: throughput=4.88MB/sec pending data=0 2022-04-22T17:47:13.047 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1_nHtOsRzO1lUJa off=18657856 len=1352845 2022-04-22T17:47:13.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 119 completed, throughput=4.93MB/sec 2022-04-22T17:47:13.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e7QgRfeafg99IsY off=4989860 len=1302903 2022-04-22T17:47:13.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 120 completed, throughput=4.97MB/sec 2022-04-22T17:47:13.058 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DjjMfF2kGqwcThD off=3537344 len=1101905 2022-04-22T17:47:13.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 121 completed, throughput=5.01MB/sec 2022-04-22T17:47:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:13 smithi121 bash[36902]: cluster 2022-04-22T17:47:12.249740+0000 mgr.y (mgr.34107) 775 : cluster [DBG] pgmap v1415: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:47:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:13 smithi115 bash[45495]: cluster 2022-04-22T17:47:12.249740+0000 mgr.y (mgr.34107) 775 : cluster [DBG] pgmap v1415: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:47:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:13 smithi115 bash[46934]: cluster 2022-04-22T17:47:12.249740+0000 mgr.y (mgr.34107) 775 : cluster [DBG] pgmap v1415: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:47:14.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:13.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:14.062 INFO:tasks.workunit.client.0.smithi115.stdout: 29: throughput=4.84MB/sec pending data=0 2022-04-22T17:47:14.062 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1oW3W7G_0Tak65u off=3078463 len=1524930 2022-04-22T17:47:14.067 INFO:tasks.workunit.client.0.smithi115.stdout:op 122 completed, throughput=4.89MB/sec 2022-04-22T17:47:14.067 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eQCcYoCZW0GAm4G off=4620177 len=222908 2022-04-22T17:47:14.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 123 completed, throughput=4.89MB/sec 2022-04-22T17:47:14.072 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-POEakxRd8n-DE8y off=420013 len=1846003 2022-04-22T17:47:14.076 INFO:tasks.workunit.client.0.smithi115.stdout:op 124 completed, throughput=4.95MB/sec 2022-04-22T17:47:14.076 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P6HwK3FDystxr5P off=20149658 len=1991730 2022-04-22T17:47:14.085 INFO:tasks.workunit.client.0.smithi115.stdout:op 125 completed, throughput=5.01MB/sec 2022-04-22T17:47:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:14 smithi121 bash[36902]: audit 2022-04-22T17:47:13.430263+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:47:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:14 smithi121 bash[36902]: audit 2022-04-22T17:47:13.431005+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:47:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:14 smithi115 bash[45495]: audit 2022-04-22T17:47:13.430263+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:47:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:14 smithi115 bash[45495]: audit 2022-04-22T17:47:13.431005+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:47:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:14 smithi115 bash[46934]: audit 2022-04-22T17:47:13.430263+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:47:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:14 smithi115 bash[46934]: audit 2022-04-22T17:47:13.431005+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:47:15.085 INFO:tasks.workunit.client.0.smithi115.stdout: 30: throughput=4.85MB/sec pending data=0 2022-04-22T17:47:15.085 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gy0Nikg7mjfDOMC off=9607726 len=326136 2022-04-22T17:47:15.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 126 completed, throughput=4.86MB/sec 2022-04-22T17:47:15.087 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0t_y-pH36Xm-7-Y off=8147434 len=981966 2022-04-22T17:47:15.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 127 completed, throughput=4.89MB/sec 2022-04-22T17:47:15.094 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jNC6mGYF1r19bfM off=5591803 len=1309518 2022-04-22T17:47:15.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 128 completed, throughput=4.93MB/sec 2022-04-22T17:47:15.097 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o62bCzYOlpZbS5j off=15663890 len=1226209 2022-04-22T17:47:15.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 129 completed, throughput=4.97MB/sec 2022-04-22T17:47:15.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e72BAGXRyjaCGbr off=7126404 len=461256 2022-04-22T17:47:15.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 130 completed, throughput=4.98MB/sec 2022-04-22T17:47:15.107 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GCmIujKYDwUo6f- off=2200391 len=650852 2022-04-22T17:47:15.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 131 completed, throughput=5MB/sec 2022-04-22T17:47:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:15 smithi121 bash[36902]: cluster 2022-04-22T17:47:14.250793+0000 mgr.y (mgr.34107) 776 : cluster [DBG] pgmap v1416: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:47:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:15 smithi121 bash[36902]: audit 2022-04-22T17:47:14.788437+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:15 smithi121 bash[36902]: audit 2022-04-22T17:47:15.069938+0000 mon.a (mon.0) 2156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[45495]: cluster 2022-04-22T17:47:14.250793+0000 mgr.y (mgr.34107) 776 : cluster [DBG] pgmap v1416: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:47:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[45495]: audit 2022-04-22T17:47:14.788437+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[45495]: audit 2022-04-22T17:47:15.069938+0000 mon.a (mon.0) 2156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[46934]: cluster 2022-04-22T17:47:14.250793+0000 mgr.y (mgr.34107) 776 : cluster [DBG] pgmap v1416: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:47:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[46934]: audit 2022-04-22T17:47:14.788437+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:15 smithi115 bash[46934]: audit 2022-04-22T17:47:15.069938+0000 mon.a (mon.0) 2156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:16.110 INFO:tasks.workunit.client.0.smithi115.stdout: 31: throughput=4.84MB/sec pending data=0 2022-04-22T17:47:16.124 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Y5lO-Fp68gE_SfG off=6269395 len=1477198 2022-04-22T17:47:16.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 132 completed, throughput=4.89MB/sec 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dSHHT74OI4gTr2E off=13094863 len=764411 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 133 completed, throughput=4.91MB/sec 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VW398o4qh29Hea3 off=9965601 len=957896 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 134 completed, throughput=4.94MB/sec 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D3g24lUQgrbUTVK off=99420 len=1967159 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 135 completed, throughput=5MB/sec 2022-04-22T17:47:16.127 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KfoKSzSiZFa5cGW off=5894992 len=1145039 2022-04-22T17:47:16.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 136 completed, throughput=5.03MB/sec 2022-04-22T17:47:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:16 smithi121 bash[36902]: cluster 2022-04-22T17:47:16.252023+0000 mgr.y (mgr.34107) 777 : cluster [DBG] pgmap v1417: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:47:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:16 smithi115 bash[45495]: cluster 2022-04-22T17:47:16.252023+0000 mgr.y (mgr.34107) 777 : cluster [DBG] pgmap v1417: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:47:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:16 smithi115 bash[46934]: cluster 2022-04-22T17:47:16.252023+0000 mgr.y (mgr.34107) 777 : cluster [DBG] pgmap v1417: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:47:17.129 INFO:tasks.workunit.client.0.smithi115.stdout: 32: throughput=4.88MB/sec pending data=0 2022-04-22T17:47:17.130 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kWCF3mDjjtdwrl8 off=3486443 len=419823 2022-04-22T17:47:17.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 137 completed, throughput=4.89MB/sec 2022-04-22T17:47:17.135 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aeL2rXDr_0UnaKJ off=824752 len=1569368 2022-04-22T17:47:17.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 138 completed, throughput=4.93MB/sec 2022-04-22T17:47:17.145 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lpBDMI1FtQ3Ol7e off=1417610 len=594408 2022-04-22T17:47:17.146 INFO:tasks.workunit.client.0.smithi115.stdout:op 139 completed, throughput=4.95MB/sec 2022-04-22T17:47:17.146 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oMOTov2Al_X-Id0 off=5319236 len=1874442 2022-04-22T17:47:17.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 140 completed, throughput=5MB/sec 2022-04-22T17:47:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:17 smithi121 bash[36902]: audit 2022-04-22T17:47:17.170656+0000 mon.a (mon.0) 2157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:17.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:17 smithi115 bash[45495]: audit 2022-04-22T17:47:17.170656+0000 mon.a (mon.0) 2157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:17 smithi115 bash[46934]: audit 2022-04-22T17:47:17.170656+0000 mon.a (mon.0) 2157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:18.154 INFO:tasks.workunit.client.0.smithi115.stdout: 33: throughput=4.85MB/sec pending data=0 2022-04-22T17:47:18.154 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5tmC7RF1_K6IWGh off=11674068 len=630727 2022-04-22T17:47:18.154 INFO:tasks.workunit.client.0.smithi115.stdout:op 141 completed, throughput=4.87MB/sec 2022-04-22T17:47:18.154 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l1GJzZG2uAOreKx off=3907016 len=681960 2022-04-22T17:47:18.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 142 completed, throughput=4.89MB/sec 2022-04-22T17:47:18.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uRtxTGK56bfWsE4 off=8939297 len=931939 2022-04-22T17:47:18.163 INFO:tasks.workunit.client.0.smithi115.stdout:op 143 completed, throughput=4.92MB/sec 2022-04-22T17:47:18.163 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FbLPszm-OkqI3Xu off=17726370 len=25706 2022-04-22T17:47:18.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 144 completed, throughput=4.92MB/sec 2022-04-22T17:47:18.166 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iaeRmY6GYNS2RZi off=18318281 len=49410 2022-04-22T17:47:18.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 145 completed, throughput=4.92MB/sec 2022-04-22T17:47:18.167 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wZuy21DtSmjglJn off=9193600 len=185182 2022-04-22T17:47:18.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 146 completed, throughput=4.92MB/sec 2022-04-22T17:47:18.168 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qo7ojJOgAoufVC9 off=1383085 len=2084956 2022-04-22T17:47:18.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 147 completed, throughput=4.98MB/sec 2022-04-22T17:47:18.178 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RCZ7LjhIq48mCIh off=12770730 len=491722 2022-04-22T17:47:18.179 INFO:tasks.workunit.client.0.smithi115.stdout:op 148 completed, throughput=4.99MB/sec 2022-04-22T17:47:18.179 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JIvRhPLRRBB1dPY off=5900612 len=1740725 2022-04-22T17:47:18.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 149 completed, throughput=5.04MB/sec 2022-04-22T17:47:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:18 smithi121 bash[36902]: cluster 2022-04-22T17:47:18.253319+0000 mgr.y (mgr.34107) 778 : cluster [DBG] pgmap v1418: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:47:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:18 smithi115 bash[46934]: cluster 2022-04-22T17:47:18.253319+0000 mgr.y (mgr.34107) 778 : cluster [DBG] pgmap v1418: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:47:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:18 smithi115 bash[45495]: cluster 2022-04-22T17:47:18.253319+0000 mgr.y (mgr.34107) 778 : cluster [DBG] pgmap v1418: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:47:19.183 INFO:tasks.workunit.client.0.smithi115.stdout: 34: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:19.183 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HccvK-yImOODJ_x off=819274 len=26997 2022-04-22T17:47:19.187 INFO:tasks.workunit.client.0.smithi115.stdout:op 150 completed, throughput=4.9MB/sec 2022-04-22T17:47:19.187 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rYb0IGDDDLPcDN4 off=10930442 len=522042 2022-04-22T17:47:19.189 INFO:tasks.workunit.client.0.smithi115.stdout:op 151 completed, throughput=4.91MB/sec 2022-04-22T17:47:19.189 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-umhgUqib8NNxF47 off=595820 len=1550163 2022-04-22T17:47:19.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 152 completed, throughput=4.95MB/sec 2022-04-22T17:47:19.192 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zX1ivAfKutKkXdM off=521099 len=817474 2022-04-22T17:47:19.194 INFO:tasks.workunit.client.0.smithi115.stdout:op 153 completed, throughput=4.98MB/sec 2022-04-22T17:47:19.194 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2e2V0uDcsoWkDfr off=3686599 len=352343 2022-04-22T17:47:19.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 154 completed, throughput=4.98MB/sec 2022-04-22T17:47:19.196 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hcfJTUlBMNGjWoF off=4141612 len=1864495 2022-04-22T17:47:19.199 INFO:tasks.workunit.client.0.smithi115.stdout:op 155 completed, throughput=5.04MB/sec 2022-04-22T17:47:20.200 INFO:tasks.workunit.client.0.smithi115.stdout: 35: throughput=4.89MB/sec pending data=0 2022-04-22T17:47:20.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kxpLQuAXx9P3VVp off=15352056 len=304956 2022-04-22T17:47:20.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 156 completed, throughput=4.9MB/sec 2022-04-22T17:47:20.202 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PqZmWVKg8UCHfSS off=10932569 len=205852 2022-04-22T17:47:20.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 157 completed, throughput=4.91MB/sec 2022-04-22T17:47:20.203 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y406drPPMui5ghb off=7958473 len=788842 2022-04-22T17:47:20.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 158 completed, throughput=4.93MB/sec 2022-04-22T17:47:20.209 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dsTPOHu6yWb10vv off=2253822 len=1459297 2022-04-22T17:47:20.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 159 completed, throughput=4.97MB/sec 2022-04-22T17:47:20.212 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-glxhDdm9t2i19jC off=6805698 len=1893005 2022-04-22T17:47:20.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 160 completed, throughput=5.02MB/sec 2022-04-22T17:47:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:19 smithi121 bash[36902]: audit 2022-04-22T17:47:19.791655+0000 mon.a (mon.0) 2158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:19 smithi115 bash[45495]: audit 2022-04-22T17:47:19.791655+0000 mon.a (mon.0) 2158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:19 smithi115 bash[46934]: audit 2022-04-22T17:47:19.791655+0000 mon.a (mon.0) 2158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:21.215 INFO:tasks.workunit.client.0.smithi115.stdout: 36: throughput=4.88MB/sec pending data=0 2022-04-22T17:47:21.255 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2_zNQxR2AFjt0nL off=653952 len=902597 2022-04-22T17:47:21.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 161 completed, throughput=4.9MB/sec 2022-04-22T17:47:21.256 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ap4r5d4NYZuhySd off=3420011 len=1982936 2022-04-22T17:47:21.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 162 completed, throughput=4.95MB/sec 2022-04-22T17:47:21.257 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-feSMCEIBleIROqF off=7892193 len=589793 2022-04-22T17:47:21.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 163 completed, throughput=4.97MB/sec 2022-04-22T17:47:21.265 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yLcIgS9smiQuKow off=1097365 len=1265057 2022-04-22T17:47:21.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 164 completed, throughput=5MB/sec 2022-04-22T17:47:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:20 smithi121 bash[36902]: audit 2022-04-22T17:47:20.073446+0000 mon.a (mon.0) 2159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:20 smithi121 bash[36902]: cluster 2022-04-22T17:47:20.254519+0000 mgr.y (mgr.34107) 779 : cluster [DBG] pgmap v1419: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:47:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:20 smithi115 bash[45495]: audit 2022-04-22T17:47:20.073446+0000 mon.a (mon.0) 2159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:20 smithi115 bash[45495]: cluster 2022-04-22T17:47:20.254519+0000 mgr.y (mgr.34107) 779 : cluster [DBG] pgmap v1419: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:47:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:20 smithi115 bash[46934]: audit 2022-04-22T17:47:20.073446+0000 mon.a (mon.0) 2159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:20 smithi115 bash[46934]: cluster 2022-04-22T17:47:20.254519+0000 mgr.y (mgr.34107) 779 : cluster [DBG] pgmap v1419: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:47:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:21.669Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:22.237 INFO:tasks.workunit.client.0.smithi115.stdout: 37: throughput=4.87MB/sec pending data=0 2022-04-22T17:47:22.237 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Aj9kXHmKu5XcX9W off=1361418 len=2076473 2022-04-22T17:47:22.244 INFO:tasks.workunit.client.0.smithi115.stdout:op 165 completed, throughput=4.92MB/sec 2022-04-22T17:47:22.244 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tT9L9BPujh4kt5g off=666949 len=1053414 2022-04-22T17:47:22.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 166 completed, throughput=4.95MB/sec 2022-04-22T17:47:22.250 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i4R2UtLCw33nTRO off=5228805 len=1524358 2022-04-22T17:47:22.252 INFO:tasks.workunit.client.0.smithi115.stdout:op 167 completed, throughput=4.98MB/sec 2022-04-22T17:47:22.252 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x0wuNqV8yesD88F off=10441127 len=2072540 2022-04-22T17:47:22.262 INFO:tasks.workunit.client.0.smithi115.stdout:op 168 completed, throughput=5.03MB/sec 2022-04-22T17:47:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:22 smithi121 bash[36902]: audit 2022-04-22T17:47:22.173726+0000 mon.a (mon.0) 2160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:22 smithi115 bash[45495]: audit 2022-04-22T17:47:22.173726+0000 mon.a (mon.0) 2160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:22.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:22 smithi115 bash[46934]: audit 2022-04-22T17:47:22.173726+0000 mon.a (mon.0) 2160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:22.901 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (10m) 10m ago 24m 12.7M - ba2b418f427c 9255ba08f72f 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (23m) 51s ago 23m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (12m) 10m ago 25m 13.0M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (11m) 51s ago 25m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:47:23.242 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (19m) 51s ago 29m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:47:23.243 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (20m) 10m ago 31m 455M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:47:23.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (19m) 10m ago 31m 87.5M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:47:23.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (18m) 51s ago 29m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:47:23.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (18m) 10m ago 29m 70.0M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (11m) 10m ago 24m 8756k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (11m) 51s ago 24m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (17m) 10m ago 29m 462M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (16m) 10m ago 28m 370M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (16m) 10m ago 28m 357M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:47:23.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (15m) 10m ago 27m 208M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:47:23.245 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (14m) 51s ago 27m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:47:23.245 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (14m) 51s ago 26m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:47:23.245 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (13m) 51s ago 26m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:47:23.245 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (12m) 51s ago 25m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:47:23.245 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (11m) 51s ago 24m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:47:23.263 INFO:tasks.workunit.client.0.smithi115.stdout: 38: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:23.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y3e25AAk7A4WJ3a off=5925152 len=1899822 2022-04-22T17:47:23.268 INFO:tasks.workunit.client.0.smithi115.stdout:op 169 completed, throughput=4.95MB/sec 2022-04-22T17:47:23.269 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xJf4mHbtpZEgp2V off=3467100 len=1595625 2022-04-22T17:47:23.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 170 completed, throughput=4.99MB/sec 2022-04-22T17:47:23.276 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4tpWtsekZvG9lm3 off=426567 len=1887305 2022-04-22T17:47:23.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 171 completed, throughput=5.03MB/sec 2022-04-22T17:47:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:23 smithi121 bash[36902]: cluster 2022-04-22T17:47:22.255659+0000 mgr.y (mgr.34107) 780 : cluster [DBG] pgmap v1420: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:23 smithi115 bash[45495]: cluster 2022-04-22T17:47:22.255659+0000 mgr.y (mgr.34107) 780 : cluster [DBG] pgmap v1420: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:23 smithi115 bash[46934]: cluster 2022-04-22T17:47:22.255659+0000 mgr.y (mgr.34107) 780 : cluster [DBG] pgmap v1420: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:23.648 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:47:23.648 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:47:23.648 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:47:23.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:23.648 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:47:23.649 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:47:23.650 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:47:24.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:23.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:24.278 INFO:tasks.workunit.client.0.smithi115.stdout: 39: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:24.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HkWpdPM51EIcplo off=8872037 len=1168392 2022-04-22T17:47:24.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 172 completed, throughput=4.94MB/sec 2022-04-22T17:47:24.282 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_XykTrp3p3F-ksV off=563360 len=1140651 2022-04-22T17:47:24.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 173 completed, throughput=4.96MB/sec 2022-04-22T17:47:24.286 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gvNEkaOfz7_VvM8 off=0 len=1279752 2022-04-22T17:47:24.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 174 completed, throughput=4.99MB/sec 2022-04-22T17:47:24.290 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BmU0qBJip3ZJ_pT off=5220328 len=1227036 2022-04-22T17:47:24.297 INFO:tasks.workunit.client.0.smithi115.stdout:op 175 completed, throughput=5.02MB/sec 2022-04-22T17:47:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:24 smithi121 bash[36902]: audit 2022-04-22T17:47:22.873609+0000 mgr.y (mgr.34107) 781 : audit [DBG] from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:24 smithi121 bash[36902]: audit 2022-04-22T17:47:23.211871+0000 mgr.y (mgr.34107) 782 : audit [DBG] from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:24 smithi121 bash[36902]: audit 2022-04-22T17:47:23.638034+0000 mon.a (mon.0) 2161 : audit [DBG] from='client.? 172.21.15.115:0/3829658089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:24.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[45495]: audit 2022-04-22T17:47:22.873609+0000 mgr.y (mgr.34107) 781 : audit [DBG] from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[45495]: audit 2022-04-22T17:47:23.211871+0000 mgr.y (mgr.34107) 782 : audit [DBG] from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[45495]: audit 2022-04-22T17:47:23.638034+0000 mon.a (mon.0) 2161 : audit [DBG] from='client.? 172.21.15.115:0/3829658089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[46934]: audit 2022-04-22T17:47:22.873609+0000 mgr.y (mgr.34107) 781 : audit [DBG] from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[46934]: audit 2022-04-22T17:47:23.211871+0000 mgr.y (mgr.34107) 782 : audit [DBG] from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:24.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:24 smithi115 bash[46934]: audit 2022-04-22T17:47:23.638034+0000 mon.a (mon.0) 2161 : audit [DBG] from='client.? 172.21.15.115:0/3829658089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:25.297 INFO:tasks.workunit.client.0.smithi115.stdout: 40: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:25.298 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2BDZgHEhTj77n6l off=2677783 len=1300754 2022-04-22T17:47:25.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 176 completed, throughput=4.93MB/sec 2022-04-22T17:47:25.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-daRTtPoHNWlkybt off=12651606 len=774372 2022-04-22T17:47:25.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 177 completed, throughput=4.95MB/sec 2022-04-22T17:47:25.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NYZwSKwqddK1W8w off=1084357 len=1857126 2022-04-22T17:47:25.317 INFO:tasks.workunit.client.0.smithi115.stdout:op 178 completed, throughput=4.99MB/sec 2022-04-22T17:47:25.317 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-a31Jnx374ZDntVt off=4803568 len=2058038 2022-04-22T17:47:25.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 179 completed, throughput=5.03MB/sec 2022-04-22T17:47:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:25 smithi121 bash[36902]: cluster 2022-04-22T17:47:24.256627+0000 mgr.y (mgr.34107) 783 : cluster [DBG] pgmap v1421: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:47:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:25 smithi121 bash[36902]: audit 2022-04-22T17:47:24.794357+0000 mon.a (mon.0) 2162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:25 smithi121 bash[36902]: audit 2022-04-22T17:47:25.077242+0000 mon.a (mon.0) 2163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[45495]: cluster 2022-04-22T17:47:24.256627+0000 mgr.y (mgr.34107) 783 : cluster [DBG] pgmap v1421: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:47:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[45495]: audit 2022-04-22T17:47:24.794357+0000 mon.a (mon.0) 2162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[45495]: audit 2022-04-22T17:47:25.077242+0000 mon.a (mon.0) 2163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[46934]: cluster 2022-04-22T17:47:24.256627+0000 mgr.y (mgr.34107) 783 : cluster [DBG] pgmap v1421: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:47:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[46934]: audit 2022-04-22T17:47:24.794357+0000 mon.a (mon.0) 2162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:25 smithi115 bash[46934]: audit 2022-04-22T17:47:25.077242+0000 mon.a (mon.0) 2163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:26.327 INFO:tasks.workunit.client.0.smithi115.stdout: 41: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:26.327 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ElYT_8r8yHHllxI off=9821754 len=1226068 2022-04-22T17:47:26.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 180 completed, throughput=4.94MB/sec 2022-04-22T17:47:26.330 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DV4-eBMFxx2EA2A off=19356107 len=1180553 2022-04-22T17:47:26.334 INFO:tasks.workunit.client.0.smithi115.stdout:op 181 completed, throughput=4.97MB/sec 2022-04-22T17:47:26.334 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5nkqozM_aObsH7i off=16550031 len=686514 2022-04-22T17:47:26.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 182 completed, throughput=4.98MB/sec 2022-04-22T17:47:26.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KwS2sZX3dURkiAH off=14785105 len=874178 2022-04-22T17:47:26.341 INFO:tasks.workunit.client.0.smithi115.stdout:op 183 completed, throughput=5MB/sec 2022-04-22T17:47:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:26 smithi121 bash[36902]: cluster 2022-04-22T17:47:26.257680+0000 mgr.y (mgr.34107) 784 : cluster [DBG] pgmap v1422: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:26 smithi115 bash[45495]: cluster 2022-04-22T17:47:26.257680+0000 mgr.y (mgr.34107) 784 : cluster [DBG] pgmap v1422: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:26 smithi115 bash[46934]: cluster 2022-04-22T17:47:26.257680+0000 mgr.y (mgr.34107) 784 : cluster [DBG] pgmap v1422: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:27.342 INFO:tasks.workunit.client.0.smithi115.stdout: 42: throughput=4.89MB/sec pending data=0 2022-04-22T17:47:27.342 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2BnRteaW8e5Klgr off=21296922 len=2039766 2022-04-22T17:47:27.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 184 completed, throughput=4.93MB/sec 2022-04-22T17:47:27.351 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QUqYg57E13DOThO off=7170737 len=321890 2022-04-22T17:47:27.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 185 completed, throughput=4.94MB/sec 2022-04-22T17:47:27.355 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MZrvyNNJi9WiatH off=6898446 len=902602 2022-04-22T17:47:27.359 INFO:tasks.workunit.client.0.smithi115.stdout:op 186 completed, throughput=4.96MB/sec 2022-04-22T17:47:27.359 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--odu_Uez50jhmo0 off=4699511 len=1562694 2022-04-22T17:47:27.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 187 completed, throughput=4.99MB/sec 2022-04-22T17:47:27.364 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PN6x2OAWs-8kchp off=5947369 len=458664 2022-04-22T17:47:27.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 188 completed, throughput=5MB/sec 2022-04-22T17:47:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:27 smithi121 bash[36902]: audit 2022-04-22T17:47:27.177003+0000 mon.a (mon.0) 2164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:27 smithi115 bash[45495]: audit 2022-04-22T17:47:27.177003+0000 mon.a (mon.0) 2164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:27 smithi115 bash[46934]: audit 2022-04-22T17:47:27.177003+0000 mon.a (mon.0) 2164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:28.367 INFO:tasks.workunit.client.0.smithi115.stdout: 43: throughput=4.89MB/sec pending data=0 2022-04-22T17:47:28.367 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fmlzeOByHO4Uf-o off=2927809 len=1324694 2022-04-22T17:47:28.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 189 completed, throughput=4.91MB/sec 2022-04-22T17:47:28.375 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t-7Rnm2rIGAQUTH off=6446533 len=1413922 2022-04-22T17:47:28.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 190 completed, throughput=4.94MB/sec 2022-04-22T17:47:28.384 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FzqJ2LPTlATqtBe off=2462311 len=526104 2022-04-22T17:47:28.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 191 completed, throughput=4.96MB/sec 2022-04-22T17:47:28.388 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--snRZV7AD1jh0Mh off=1921988 len=648825 2022-04-22T17:47:28.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 192 completed, throughput=4.97MB/sec 2022-04-22T17:47:28.392 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LZaHVF-7jtFz6Sj off=387969 len=1306969 2022-04-22T17:47:28.394 INFO:tasks.workunit.client.0.smithi115.stdout:op 193 completed, throughput=5MB/sec 2022-04-22T17:47:28.394 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8F4Rb1pQ3KkYabk off=16874926 len=894769 2022-04-22T17:47:28.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 194 completed, throughput=5.02MB/sec 2022-04-22T17:47:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:28 smithi121 bash[36902]: cluster 2022-04-22T17:47:28.258826+0000 mgr.y (mgr.34107) 785 : cluster [DBG] pgmap v1423: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:47:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:28 smithi115 bash[45495]: cluster 2022-04-22T17:47:28.258826+0000 mgr.y (mgr.34107) 785 : cluster [DBG] pgmap v1423: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:47:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:28 smithi115 bash[46934]: cluster 2022-04-22T17:47:28.258826+0000 mgr.y (mgr.34107) 785 : cluster [DBG] pgmap v1423: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:47:29.399 INFO:tasks.workunit.client.0.smithi115.stdout: 44: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:29.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Of3MY2R2uy1INEP off=968831 len=1009509 2022-04-22T17:47:29.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 195 completed, throughput=4.93MB/sec 2022-04-22T17:47:29.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MfMVrPa9lAwUUxe off=0 len=1783729 2022-04-22T17:47:29.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 196 completed, throughput=4.96MB/sec 2022-04-22T17:47:29.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yYmEnyk5G1NQB5X off=2292534 len=1391443 2022-04-22T17:47:29.409 INFO:tasks.workunit.client.0.smithi115.stdout:op 197 completed, throughput=4.99MB/sec 2022-04-22T17:47:29.409 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IID3_f4gfnONfwz off=87389 len=897186 2022-04-22T17:47:29.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 198 completed, throughput=5.01MB/sec 2022-04-22T17:47:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:29 smithi121 bash[36902]: audit 2022-04-22T17:47:29.797349+0000 mon.a (mon.0) 2165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:29 smithi115 bash[45495]: audit 2022-04-22T17:47:29.797349+0000 mon.a (mon.0) 2165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:29 smithi115 bash[46934]: audit 2022-04-22T17:47:29.797349+0000 mon.a (mon.0) 2165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:30.411 INFO:tasks.workunit.client.0.smithi115.stdout: 45: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:30.432 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--7xuEHPK9JunldX off=786948 len=633716 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 199 completed, throughput=4.91MB/sec 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h7KUCnL9IntRkIU off=1718967 len=1993897 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 200 completed, throughput=4.96MB/sec 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-I40rQLNKv4PsZJt off=5948784 len=1750839 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 201 completed, throughput=4.99MB/sec 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DzJbiyXwqeCzhwE off=11066873 len=1828114 2022-04-22T17:47:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 202 completed, throughput=5.03MB/sec 2022-04-22T17:47:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:30 smithi121 bash[36902]: audit 2022-04-22T17:47:30.080417+0000 mon.a (mon.0) 2166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:30 smithi121 bash[36902]: cluster 2022-04-22T17:47:30.259980+0000 mgr.y (mgr.34107) 786 : cluster [DBG] pgmap v1424: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:30 smithi115 bash[46934]: audit 2022-04-22T17:47:30.080417+0000 mon.a (mon.0) 2166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:30 smithi115 bash[46934]: cluster 2022-04-22T17:47:30.259980+0000 mgr.y (mgr.34107) 786 : cluster [DBG] pgmap v1424: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:30 smithi115 bash[45495]: audit 2022-04-22T17:47:30.080417+0000 mon.a (mon.0) 2166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:30 smithi115 bash[45495]: cluster 2022-04-22T17:47:30.259980+0000 mgr.y (mgr.34107) 786 : cluster [DBG] pgmap v1424: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:47:31.433 INFO:tasks.workunit.client.0.smithi115.stdout: 46: throughput=4.92MB/sec pending data=0 2022-04-22T17:47:31.439 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q6P5wiSDVxJ8Pu7 off=17897762 len=1335629 2022-04-22T17:47:31.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 203 completed, throughput=4.95MB/sec 2022-04-22T17:47:31.440 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aMtlVL11f_kkyr9 off=832756 len=1105951 2022-04-22T17:47:31.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 204 completed, throughput=4.97MB/sec 2022-04-22T17:47:31.443 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KRJ6TA9zV9TeSL5 off=6784081 len=1040216 2022-04-22T17:47:31.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 205 completed, throughput=4.99MB/sec 2022-04-22T17:47:31.449 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RzpeEusu0MXINig off=1864459 len=629921 2022-04-22T17:47:31.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 206 completed, throughput=5MB/sec 2022-04-22T17:47:32.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:31.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:32.452 INFO:tasks.workunit.client.0.smithi115.stdout: 47: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:32.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xRfpauW9yW4fzhg off=10714212 len=919252 2022-04-22T17:47:32.454 INFO:tasks.workunit.client.0.smithi115.stdout:op 207 completed, throughput=4.92MB/sec 2022-04-22T17:47:32.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XmHBqc4DQd4zEMl off=21482682 len=338289 2022-04-22T17:47:32.460 INFO:tasks.workunit.client.0.smithi115.stdout:op 208 completed, throughput=4.92MB/sec 2022-04-22T17:47:32.460 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WcQjkD52GO8omsX off=619748 len=1315321 2022-04-22T17:47:32.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 209 completed, throughput=4.95MB/sec 2022-04-22T17:47:32.463 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ATQEf_fOblfjrPe off=7850523 len=1088140 2022-04-22T17:47:32.471 INFO:tasks.workunit.client.0.smithi115.stdout:op 210 completed, throughput=4.97MB/sec 2022-04-22T17:47:32.471 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-glT0fnaqZH3mfGA off=6027176 len=2008124 2022-04-22T17:47:32.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 211 completed, throughput=5.01MB/sec 2022-04-22T17:47:32.498 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:32 smithi115 bash[45495]: audit 2022-04-22T17:47:32.180048+0000 mon.a (mon.0) 2167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:32.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:32 smithi115 bash[46934]: audit 2022-04-22T17:47:32.180048+0000 mon.a (mon.0) 2167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:32.499 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:32 smithi121 bash[36902]: audit 2022-04-22T17:47:32.180048+0000 mon.a (mon.0) 2167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:33.480 INFO:tasks.workunit.client.0.smithi115.stdout: 48: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:33.506 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NavT7I6ybwFQw9y off=8539088 len=1017553 2022-04-22T17:47:33.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 212 completed, throughput=4.93MB/sec 2022-04-22T17:47:33.508 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SUI2UnWOIr0UXfe off=668656 len=519498 2022-04-22T17:47:33.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 213 completed, throughput=4.94MB/sec 2022-04-22T17:47:33.508 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-j9ltRD5POGrhl95 off=8477601 len=640992 2022-04-22T17:47:33.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 214 completed, throughput=4.95MB/sec 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6g-_f-sEphPh68y off=14905263 len=501606 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 215 completed, throughput=4.96MB/sec 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EIWlsyrgdzptb4b off=1173107 len=811603 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 216 completed, throughput=4.97MB/sec 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AqQT9f8Kl_vZbiM off=12475768 len=638468 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 217 completed, throughput=4.98MB/sec 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CLJDUwFMskZzdgn off=19284625 len=1457906 2022-04-22T17:47:33.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 218 completed, throughput=5.01MB/sec 2022-04-22T17:47:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: cluster 2022-04-22T17:47:32.261057+0000 mgr.y (mgr.34107) 787 : cluster [DBG] pgmap v1425: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: audit 2022-04-22T17:47:32.450248+0000 mon.a (mon.0) 2168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:47:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: audit 2022-04-22T17:47:32.452896+0000 mon.a (mon.0) 2169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:47:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: audit 2022-04-22T17:47:32.454345+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:47:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: audit 2022-04-22T17:47:32.780756+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:33 smithi121 bash[36902]: audit 2022-04-22T17:47:32.786962+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: cluster 2022-04-22T17:47:32.261057+0000 mgr.y (mgr.34107) 787 : cluster [DBG] pgmap v1425: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: audit 2022-04-22T17:47:32.450248+0000 mon.a (mon.0) 2168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:47:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: audit 2022-04-22T17:47:32.452896+0000 mon.a (mon.0) 2169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: audit 2022-04-22T17:47:32.454345+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: audit 2022-04-22T17:47:32.780756+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[45495]: audit 2022-04-22T17:47:32.786962+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: cluster 2022-04-22T17:47:32.261057+0000 mgr.y (mgr.34107) 787 : cluster [DBG] pgmap v1425: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: audit 2022-04-22T17:47:32.450248+0000 mon.a (mon.0) 2168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: audit 2022-04-22T17:47:32.452896+0000 mon.a (mon.0) 2169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: audit 2022-04-22T17:47:32.454345+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: audit 2022-04-22T17:47:32.780756+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:33.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[46934]: audit 2022-04-22T17:47:32.786962+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:34.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:33.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:34.508 INFO:tasks.workunit.client.0.smithi115.stdout: 49: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:34.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z4jRuPWHhzsWpg6 off=14075357 len=1504578 2022-04-22T17:47:34.513 INFO:tasks.workunit.client.0.smithi115.stdout:op 219 completed, throughput=4.94MB/sec 2022-04-22T17:47:34.513 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j18FMk0F6UtXer9 off=1611521 len=53980 2022-04-22T17:47:34.514 INFO:tasks.workunit.client.0.smithi115.stdout:op 220 completed, throughput=4.94MB/sec 2022-04-22T17:47:34.514 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NrS_7GAn5P_e9D7 off=426771 len=676578 2022-04-22T17:47:34.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 221 completed, throughput=4.95MB/sec 2022-04-22T17:47:34.517 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hJBps7J5HeZz7AB off=0 len=1073037 2022-04-22T17:47:34.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 222 completed, throughput=4.97MB/sec 2022-04-22T17:47:34.523 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EvYEB8xOEqsjAOs off=2991782 len=1916556 2022-04-22T17:47:34.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 223 completed, throughput=5.01MB/sec 2022-04-22T17:47:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:34 smithi121 bash[36902]: cluster 2022-04-22T17:47:34.262152+0000 mgr.y (mgr.34107) 788 : cluster [DBG] pgmap v1426: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:34.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:34 smithi115 bash[45495]: cluster 2022-04-22T17:47:34.262152+0000 mgr.y (mgr.34107) 788 : cluster [DBG] pgmap v1426: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:34 smithi115 bash[46934]: cluster 2022-04-22T17:47:34.262152+0000 mgr.y (mgr.34107) 788 : cluster [DBG] pgmap v1426: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:47:35.527 INFO:tasks.workunit.client.0.smithi115.stdout: 50: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:35.527 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNIsKmv_6cDWE3e off=1542959 len=1851935 2022-04-22T17:47:35.530 INFO:tasks.workunit.client.0.smithi115.stdout:op 224 completed, throughput=4.95MB/sec 2022-04-22T17:47:35.531 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QwSEpMVII0e5tb4 off=7164571 len=1614256 2022-04-22T17:47:35.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 225 completed, throughput=4.97MB/sec 2022-04-22T17:47:35.540 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HtFb5zXv0r1EpLi off=1936719 len=1406190 2022-04-22T17:47:35.546 INFO:tasks.workunit.client.0.smithi115.stdout:op 226 completed, throughput=5MB/sec 2022-04-22T17:47:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:35 smithi121 bash[36902]: audit 2022-04-22T17:47:34.800471+0000 mon.a (mon.0) 2173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:35 smithi121 bash[36902]: audit 2022-04-22T17:47:35.084309+0000 mon.a (mon.0) 2174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:35 smithi115 bash[45495]: audit 2022-04-22T17:47:34.800471+0000 mon.a (mon.0) 2173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:35 smithi115 bash[45495]: audit 2022-04-22T17:47:35.084309+0000 mon.a (mon.0) 2174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:35 smithi115 bash[46934]: audit 2022-04-22T17:47:34.800471+0000 mon.a (mon.0) 2173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:35 smithi115 bash[46934]: audit 2022-04-22T17:47:35.084309+0000 mon.a (mon.0) 2174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:36.546 INFO:tasks.workunit.client.0.smithi115.stdout: 51: throughput=4.9MB/sec pending data=0 2022-04-22T17:47:36.594 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VAa9mh80MvSqTI5 off=2053048 len=1673829 2022-04-22T17:47:36.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 227 completed, throughput=4.93MB/sec 2022-04-22T17:47:36.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1lecoH3C3o2l9IL off=163964 len=17872 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 228 completed, throughput=4.94MB/sec 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ImT07d1DpdhHDdK off=195649 len=291770 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 229 completed, throughput=4.94MB/sec 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n4TwLDwDRtbO4ks off=1739035 len=1245615 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 230 completed, throughput=4.96MB/sec 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kfUr9fOL1QA8pKY off=11483685 len=647722 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 231 completed, throughput=4.97MB/sec 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yViJfyIUKn69Tkv off=841956 len=500897 2022-04-22T17:47:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 232 completed, throughput=4.98MB/sec 2022-04-22T17:47:36.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--LpvD7280U7HK08 off=3836375 len=1417545 2022-04-22T17:47:36.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 233 completed, throughput=5.01MB/sec 2022-04-22T17:47:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:36 smithi121 bash[36902]: cluster 2022-04-22T17:47:36.262785+0000 mgr.y (mgr.34107) 789 : cluster [DBG] pgmap v1427: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:47:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:36 smithi115 bash[45495]: cluster 2022-04-22T17:47:36.262785+0000 mgr.y (mgr.34107) 789 : cluster [DBG] pgmap v1427: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:47:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:36 smithi115 bash[46934]: cluster 2022-04-22T17:47:36.262785+0000 mgr.y (mgr.34107) 789 : cluster [DBG] pgmap v1427: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:47:37.568 INFO:tasks.workunit.client.0.smithi115.stdout: 52: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:37.568 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uoujpEnHkxw6wmU off=19563161 len=91341 2022-04-22T17:47:37.572 INFO:tasks.workunit.client.0.smithi115.stdout:op 234 completed, throughput=4.92MB/sec 2022-04-22T17:47:37.572 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XNuemQHN97X7tMK off=2145714 len=787264 2022-04-22T17:47:37.578 INFO:tasks.workunit.client.0.smithi115.stdout:op 235 completed, throughput=4.93MB/sec 2022-04-22T17:47:37.578 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-keU1ejKpq9Lzatc off=357885 len=865122 2022-04-22T17:47:37.580 INFO:tasks.workunit.client.0.smithi115.stdout:op 236 completed, throughput=4.94MB/sec 2022-04-22T17:47:37.580 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3kdOIF3sj8BXdJL off=1153093 len=1072197 2022-04-22T17:47:37.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 237 completed, throughput=4.96MB/sec 2022-04-22T17:47:37.584 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s-j9L9ezMSB2qDN off=5189982 len=530672 2022-04-22T17:47:37.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 238 completed, throughput=4.97MB/sec 2022-04-22T17:47:37.590 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LlB_9eS7XoVs3TX off=3004417 len=619195 2022-04-22T17:47:37.591 INFO:tasks.workunit.client.0.smithi115.stdout:op 239 completed, throughput=4.98MB/sec 2022-04-22T17:47:37.592 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oscVaF5azmwqG7r off=9622577 len=1909734 2022-04-22T17:47:37.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 240 completed, throughput=5.02MB/sec 2022-04-22T17:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:37 smithi121 bash[36902]: audit 2022-04-22T17:47:36.893568+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:37 smithi121 bash[36902]: audit 2022-04-22T17:47:37.182728+0000 mon.a (mon.0) 2176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:37 smithi121 bash[36902]: audit 2022-04-22T17:47:37.237876+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[45495]: audit 2022-04-22T17:47:36.893568+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[45495]: audit 2022-04-22T17:47:37.182728+0000 mon.a (mon.0) 2176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[45495]: audit 2022-04-22T17:47:37.237876+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[46934]: audit 2022-04-22T17:47:36.893568+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[46934]: audit 2022-04-22T17:47:37.182728+0000 mon.a (mon.0) 2176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:37 smithi115 bash[46934]: audit 2022-04-22T17:47:37.237876+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:47:38.600 INFO:tasks.workunit.client.0.smithi115.stdout: 53: throughput=4.92MB/sec pending data=0 2022-04-22T17:47:38.600 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C_QNVAIRR0Gsk-P off=15178176 len=604287 2022-04-22T17:47:38.606 INFO:tasks.workunit.client.0.smithi115.stdout:op 241 completed, throughput=4.93MB/sec 2022-04-22T17:47:38.606 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pmxTYLulUxaSsbn off=2498602 len=1621663 2022-04-22T17:47:38.611 INFO:tasks.workunit.client.0.smithi115.stdout:op 242 completed, throughput=4.96MB/sec 2022-04-22T17:47:38.611 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kxC1SFwE-AvTdkr off=2480621 len=1316827 2022-04-22T17:47:38.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 243 completed, throughput=4.99MB/sec 2022-04-22T17:47:38.614 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c8beyoi6uE_SGQF off=668103 len=589859 2022-04-22T17:47:38.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 244 completed, throughput=5MB/sec 2022-04-22T17:47:38.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OIlfExEQx43IqxI off=295782 len=1244818 2022-04-22T17:47:38.617 INFO:tasks.workunit.client.0.smithi115.stdout:op 245 completed, throughput=5.02MB/sec 2022-04-22T17:47:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:38 smithi121 bash[36902]: cluster 2022-04-22T17:47:38.264012+0000 mgr.y (mgr.34107) 790 : cluster [DBG] pgmap v1428: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:38 smithi115 bash[45495]: cluster 2022-04-22T17:47:38.264012+0000 mgr.y (mgr.34107) 790 : cluster [DBG] pgmap v1428: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:39.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:38 smithi115 bash[46934]: cluster 2022-04-22T17:47:38.264012+0000 mgr.y (mgr.34107) 790 : cluster [DBG] pgmap v1428: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:47:39.617 INFO:tasks.workunit.client.0.smithi115.stdout: 54: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:39.618 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y-rpWtYCp5SjM7R off=17558200 len=958552 2022-04-22T17:47:39.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 246 completed, throughput=4.94MB/sec 2022-04-22T17:47:39.620 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k2q2MNpo8QGLs4- off=1268344 len=1323979 2022-04-22T17:47:39.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 247 completed, throughput=4.97MB/sec 2022-04-22T17:47:39.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eZdRdOaQqR8IdB8 off=7285458 len=983954 2022-04-22T17:47:39.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 248 completed, throughput=4.98MB/sec 2022-04-22T17:47:39.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_bNq5Fi6SCA2i5I off=1696413 len=625556 2022-04-22T17:47:39.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 249 completed, throughput=4.99MB/sec 2022-04-22T17:47:39.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Kh_0WY5jhVj8Put off=8665594 len=2088684 2022-04-22T17:47:39.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 250 completed, throughput=5.03MB/sec 2022-04-22T17:47:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:39 smithi121 bash[36902]: audit 2022-04-22T17:47:39.803726+0000 mon.a (mon.0) 2178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:40.322 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:39 smithi115 bash[45495]: audit 2022-04-22T17:47:39.803726+0000 mon.a (mon.0) 2178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:40.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:39 smithi115 bash[46934]: audit 2022-04-22T17:47:39.803726+0000 mon.a (mon.0) 2178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:40.636 INFO:tasks.workunit.client.0.smithi115.stdout: 55: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:40.636 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BQtakDf3nh4zO9y off=1567152 len=15896 2022-04-22T17:47:40.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 251 completed, throughput=4.94MB/sec 2022-04-22T17:47:40.641 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L4nhAohLgsg4-Ew off=20211919 len=767887 2022-04-22T17:47:40.646 INFO:tasks.workunit.client.0.smithi115.stdout:op 252 completed, throughput=4.95MB/sec 2022-04-22T17:47:40.646 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1wZSn1Dp47fE70n off=4802822 len=1371739 2022-04-22T17:47:40.649 INFO:tasks.workunit.client.0.smithi115.stdout:op 253 completed, throughput=4.97MB/sec 2022-04-22T17:47:40.649 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OZxLMkqskDV0mf4 off=1480780 len=1379868 2022-04-22T17:47:40.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 254 completed, throughput=5MB/sec 2022-04-22T17:47:40.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tjz_GR9p8cwTG6x off=4816721 len=633226 2022-04-22T17:47:40.659 INFO:tasks.workunit.client.0.smithi115.stdout:op 255 completed, throughput=5.01MB/sec 2022-04-22T17:47:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:40 smithi121 bash[36902]: audit 2022-04-22T17:47:40.087799+0000 mon.a (mon.0) 2179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:41.284 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:40 smithi121 bash[36902]: cluster 2022-04-22T17:47:40.265076+0000 mgr.y (mgr.34107) 791 : cluster [DBG] pgmap v1429: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:40 smithi115 bash[45495]: audit 2022-04-22T17:47:40.087799+0000 mon.a (mon.0) 2179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:41.391 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:40 smithi115 bash[45495]: cluster 2022-04-22T17:47:40.265076+0000 mgr.y (mgr.34107) 791 : cluster [DBG] pgmap v1429: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:41.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:40 smithi115 bash[46934]: audit 2022-04-22T17:47:40.087799+0000 mon.a (mon.0) 2179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:41.391 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:40 smithi115 bash[46934]: cluster 2022-04-22T17:47:40.265076+0000 mgr.y (mgr.34107) 791 : cluster [DBG] pgmap v1429: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:41.660 INFO:tasks.workunit.client.0.smithi115.stdout: 56: throughput=4.92MB/sec pending data=0 2022-04-22T17:47:41.660 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XDxjO-NhgpohT3j off=1638225 len=1211882 2022-04-22T17:47:41.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 256 completed, throughput=4.94MB/sec 2022-04-22T17:47:41.668 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9SFiE91R2pZSJE7 off=18780544 len=67652 2022-04-22T17:47:41.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 257 completed, throughput=4.94MB/sec 2022-04-22T17:47:41.669 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fZXLG3I4Mn07Nw_ off=855687 len=1156992 2022-04-22T17:47:41.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 258 completed, throughput=4.96MB/sec 2022-04-22T17:47:41.676 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fm0NGu6KreAXci_ off=1655648 len=686335 2022-04-22T17:47:41.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 259 completed, throughput=4.97MB/sec 2022-04-22T17:47:41.681 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NYZwSKwqddK1W8w off=1128477 len=1887323 2022-04-22T17:47:41.688 INFO:tasks.workunit.client.0.smithi115.stdout:op 260 completed, throughput=5MB/sec 2022-04-22T17:47:42.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:41.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:42 smithi121 bash[36902]: audit 2022-04-22T17:47:42.185167+0000 mon.a (mon.0) 2180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:42.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:42 smithi115 bash[45495]: audit 2022-04-22T17:47:42.185167+0000 mon.a (mon.0) 2180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:42.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:42 smithi115 bash[46934]: audit 2022-04-22T17:47:42.185167+0000 mon.a (mon.0) 2180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:42.688 INFO:tasks.workunit.client.0.smithi115.stdout: 57: throughput=4.91MB/sec pending data=0 2022-04-22T17:47:42.688 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wlFTxncsH5WXyZb off=6926973 len=1135171 2022-04-22T17:47:42.691 INFO:tasks.workunit.client.0.smithi115.stdout:op 261 completed, throughput=4.93MB/sec 2022-04-22T17:47:42.691 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GIqFzaJzoDeAuo5 off=136698 len=481683 2022-04-22T17:47:42.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 262 completed, throughput=4.94MB/sec 2022-04-22T17:47:42.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yINBxO2so3fGJzE off=20497748 len=994757 2022-04-22T17:47:42.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 263 completed, throughput=4.96MB/sec 2022-04-22T17:47:42.701 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sxOWUFqM0p8N-iL off=8987045 len=26160 2022-04-22T17:47:42.704 INFO:tasks.workunit.client.0.smithi115.stdout:op 264 completed, throughput=4.96MB/sec 2022-04-22T17:47:42.704 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WfWaWZbWoCxiKFJ off=372752 len=55465 2022-04-22T17:47:42.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 265 completed, throughput=4.96MB/sec 2022-04-22T17:47:42.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o3--PlNIHjs9o9S off=2352958 len=423238 2022-04-22T17:47:42.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 266 completed, throughput=4.96MB/sec 2022-04-22T17:47:42.712 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lEcrA6-a_DZOnEv off=6344420 len=163847 2022-04-22T17:47:42.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 267 completed, throughput=4.97MB/sec 2022-04-22T17:47:42.712 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--6VQMMNk9QJyHMv off=7637619 len=1573414 2022-04-22T17:47:42.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 268 completed, throughput=4.99MB/sec 2022-04-22T17:47:42.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v34OVV3wWHYEVEl off=1742643 len=1945843 2022-04-22T17:47:42.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 269 completed, throughput=5.02MB/sec 2022-04-22T17:47:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:43 smithi121 bash[36902]: cluster 2022-04-22T17:47:42.266194+0000 mgr.y (mgr.34107) 792 : cluster [DBG] pgmap v1430: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:43 smithi115 bash[45495]: cluster 2022-04-22T17:47:42.266194+0000 mgr.y (mgr.34107) 792 : cluster [DBG] pgmap v1430: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:43 smithi115 bash[46934]: cluster 2022-04-22T17:47:42.266194+0000 mgr.y (mgr.34107) 792 : cluster [DBG] pgmap v1430: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:43.718 INFO:tasks.workunit.client.0.smithi115.stdout: 58: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:43.719 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9RNf1d0DfNXSxeL off=21727906 len=1404646 2022-04-22T17:47:43.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 270 completed, throughput=4.96MB/sec 2022-04-22T17:47:43.727 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gh2I6eU9TJcMNRY off=889550 len=1905667 2022-04-22T17:47:43.734 INFO:tasks.workunit.client.0.smithi115.stdout:op 271 completed, throughput=4.99MB/sec 2022-04-22T17:47:43.734 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fkDq2txhSGoUZ0S off=14956416 len=904151 2022-04-22T17:47:43.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 272 completed, throughput=5MB/sec 2022-04-22T17:47:44.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:43.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:44.738 INFO:tasks.workunit.client.0.smithi115.stdout: 59: throughput=4.92MB/sec pending data=0 2022-04-22T17:47:44.842 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TP_49qxVqWnCA3A off=14983610 len=1905343 2022-04-22T17:47:44.843 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:44 smithi121 bash[36902]: cluster 2022-04-22T17:47:44.267329+0000 mgr.y (mgr.34107) 793 : cluster [DBG] pgmap v1431: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:47:44.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 273 completed, throughput=4.95MB/sec 2022-04-22T17:47:44.894 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oYA7oWGp0dGIzXZ off=3532815 len=601774 2022-04-22T17:47:44.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 274 completed, throughput=4.96MB/sec 2022-04-22T17:47:44.895 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nMkRQz3GgQsb3_g off=8238249 len=890175 2022-04-22T17:47:44.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 275 completed, throughput=4.97MB/sec 2022-04-22T17:47:44.896 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-A03cfywiaC0BsIw off=7485976 len=148320 2022-04-22T17:47:44.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 276 completed, throughput=4.98MB/sec 2022-04-22T17:47:44.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fq1uYrfJb8wL7o3 off=653982 len=102973 2022-04-22T17:47:44.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 277 completed, throughput=4.98MB/sec 2022-04-22T17:47:44.897 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-399DD61eHpMR9at off=13681619 len=837730 2022-04-22T17:47:44.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 278 completed, throughput=4.99MB/sec 2022-04-22T17:47:44.897 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-g8DCV40n5fIXYki off=9579057 len=1883761 2022-04-22T17:47:44.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 279 completed, throughput=5.02MB/sec 2022-04-22T17:47:44.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:44 smithi115 bash[45495]: cluster 2022-04-22T17:47:44.267329+0000 mgr.y (mgr.34107) 793 : cluster [DBG] pgmap v1431: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:47:44.898 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:44 smithi115 bash[46934]: cluster 2022-04-22T17:47:44.267329+0000 mgr.y (mgr.34107) 793 : cluster [DBG] pgmap v1431: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:47:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:45 smithi121 bash[36902]: audit 2022-04-22T17:47:44.807046+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:45 smithi121 bash[36902]: audit 2022-04-22T17:47:45.091037+0000 mon.a (mon.0) 2182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:45.773 INFO:tasks.workunit.client.0.smithi115.stdout: 60: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:45.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LeGKVY4pl6bzLAV off=225890 len=653367 2022-04-22T17:47:45.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 280 completed, throughput=4.95MB/sec 2022-04-22T17:47:45.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VX9gz5j1SnTQmS3 off=10620785 len=875913 2022-04-22T17:47:45.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 281 completed, throughput=4.96MB/sec 2022-04-22T17:47:45.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tIy_mnGkiEsdU1E off=9366004 len=1436201 2022-04-22T17:47:45.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 282 completed, throughput=4.98MB/sec 2022-04-22T17:47:45.801 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-azr8txY8Snxn-PC off=2621929 len=2043063 2022-04-22T17:47:45.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:45 smithi115 bash[45495]: audit 2022-04-22T17:47:44.807046+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:45.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:45 smithi115 bash[45495]: audit 2022-04-22T17:47:45.091037+0000 mon.a (mon.0) 2182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:45.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:45 smithi115 bash[46934]: audit 2022-04-22T17:47:44.807046+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:45.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:45 smithi115 bash[46934]: audit 2022-04-22T17:47:45.091037+0000 mon.a (mon.0) 2182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:45.803 INFO:tasks.workunit.client.0.smithi115.stdout:op 283 completed, throughput=5.01MB/sec 2022-04-22T17:47:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:46 smithi121 bash[36902]: cluster 2022-04-22T17:47:46.268409+0000 mgr.y (mgr.34107) 794 : cluster [DBG] pgmap v1432: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:46 smithi115 bash[45495]: cluster 2022-04-22T17:47:46.268409+0000 mgr.y (mgr.34107) 794 : cluster [DBG] pgmap v1432: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:46 smithi115 bash[46934]: cluster 2022-04-22T17:47:46.268409+0000 mgr.y (mgr.34107) 794 : cluster [DBG] pgmap v1432: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:46.798 INFO:tasks.workunit.client.0.smithi115.stdout: 61: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:46.798 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--7xuEHPK9JunldX off=0 len=1489733 2022-04-22T17:47:46.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 284 completed, throughput=4.96MB/sec 2022-04-22T17:47:46.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wi2-i5YXEv_VtFD off=15790203 len=455214 2022-04-22T17:47:46.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 285 completed, throughput=4.96MB/sec 2022-04-22T17:47:46.807 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7vDComtwVdEip5s off=206003 len=1331163 2022-04-22T17:47:46.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 286 completed, throughput=4.98MB/sec 2022-04-22T17:47:46.815 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qYrpi4pEsgzsyJv off=2547149 len=300467 2022-04-22T17:47:46.816 INFO:tasks.workunit.client.0.smithi115.stdout:op 287 completed, throughput=4.99MB/sec 2022-04-22T17:47:46.816 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kGSBEn7IkD_1SBa off=1973576 len=1209412 2022-04-22T17:47:46.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 288 completed, throughput=5MB/sec 2022-04-22T17:47:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:47 smithi121 bash[36902]: audit 2022-04-22T17:47:47.188231+0000 mon.a (mon.0) 2183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:47 smithi115 bash[45495]: audit 2022-04-22T17:47:47.188231+0000 mon.a (mon.0) 2183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:47 smithi115 bash[46934]: audit 2022-04-22T17:47:47.188231+0000 mon.a (mon.0) 2183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:47.823 INFO:tasks.workunit.client.0.smithi115.stdout: 62: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:47.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ni64edyOdmhQEzF off=324182 len=809991 2022-04-22T17:47:47.827 INFO:tasks.workunit.client.0.smithi115.stdout:op 289 completed, throughput=4.94MB/sec 2022-04-22T17:47:47.827 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GbAGj8pkItOIYVg off=1571740 len=1450838 2022-04-22T17:47:47.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 290 completed, throughput=4.96MB/sec 2022-04-22T17:47:47.836 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gNLy3GFkhkylnOy off=10264333 len=301726 2022-04-22T17:47:47.840 INFO:tasks.workunit.client.0.smithi115.stdout:op 291 completed, throughput=4.96MB/sec 2022-04-22T17:47:47.840 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IjD_W_vLQGMdC84 off=14499576 len=2055828 2022-04-22T17:47:47.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 292 completed, throughput=4.99MB/sec 2022-04-22T17:47:47.844 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KRENgDB76f4xs05 off=7330607 len=703856 2022-04-22T17:47:47.849 INFO:tasks.workunit.client.0.smithi115.stdout:op 293 completed, throughput=5MB/sec 2022-04-22T17:47:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:48 smithi121 bash[36902]: cluster 2022-04-22T17:47:48.269596+0000 mgr.y (mgr.34107) 795 : cluster [DBG] pgmap v1433: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:48.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:48 smithi115 bash[45495]: cluster 2022-04-22T17:47:48.269596+0000 mgr.y (mgr.34107) 795 : cluster [DBG] pgmap v1433: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:48 smithi115 bash[46934]: cluster 2022-04-22T17:47:48.269596+0000 mgr.y (mgr.34107) 795 : cluster [DBG] pgmap v1433: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:48.850 INFO:tasks.workunit.client.0.smithi115.stdout: 63: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:48.872 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K5nGH5KdJay0v1s off=21502798 len=880072 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 294 completed, throughput=4.94MB/sec 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LPwpBi4TiVWPIo1 off=2398737 len=2086642 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 295 completed, throughput=4.97MB/sec 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yS32Q36D3DO7wfk off=3629610 len=900092 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 296 completed, throughput=4.98MB/sec 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bJg-omKB7yRDL-j off=9510234 len=207857 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 297 completed, throughput=4.98MB/sec 2022-04-22T17:47:48.874 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yyfv_ZIImnOW7mG off=1177234 len=1295759 2022-04-22T17:47:48.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 298 completed, throughput=5MB/sec 2022-04-22T17:47:49.875 INFO:tasks.workunit.client.0.smithi115.stdout: 64: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:49.902 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4DXHdl6NkajqXD2 off=13852444 len=1777315 2022-04-22T17:47:49.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 299 completed, throughput=4.95MB/sec 2022-04-22T17:47:49.903 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9TIo9zJLipQLbNO off=12160002 len=438311 2022-04-22T17:47:49.904 INFO:tasks.workunit.client.0.smithi115.stdout:op 300 completed, throughput=4.96MB/sec 2022-04-22T17:47:49.907 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_I2Cz8Y82ctHqWD off=2181821 len=1069177 2022-04-22T17:47:49.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 301 completed, throughput=4.97MB/sec 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s_X-VobiyB_YzM2 off=3555531 len=1320719 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 302 completed, throughput=4.99MB/sec 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xvvBxWLgYbG8Wug off=1074128 len=468328 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 303 completed, throughput=5MB/sec 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tr1usPO7gV67YO5 off=3047358 len=739983 2022-04-22T17:47:49.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 304 completed, throughput=5.01MB/sec 2022-04-22T17:47:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:49 smithi121 bash[36902]: audit 2022-04-22T17:47:49.810222+0000 mon.a (mon.0) 2184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:49 smithi115 bash[45495]: audit 2022-04-22T17:47:49.810222+0000 mon.a (mon.0) 2184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:49 smithi115 bash[46934]: audit 2022-04-22T17:47:49.810222+0000 mon.a (mon.0) 2184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:50.901 INFO:tasks.workunit.client.0.smithi115.stdout: 65: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:50.901 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-36z51IIBbqmgo57 off=2839627 len=1650151 2022-04-22T17:47:50.904 INFO:tasks.workunit.client.0.smithi115.stdout:op 305 completed, throughput=4.96MB/sec 2022-04-22T17:47:50.904 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4YU_6cz7g5khjTF off=7791263 len=1432803 2022-04-22T17:47:50.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 306 completed, throughput=4.98MB/sec 2022-04-22T17:47:50.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pgAWS03_TaXFCfj off=5667827 len=625714 2022-04-22T17:47:50.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 307 completed, throughput=4.99MB/sec 2022-04-22T17:47:50.910 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LGNHX86Ogs0u-hU off=16768892 len=1057259 2022-04-22T17:47:50.912 INFO:tasks.workunit.client.0.smithi115.stdout:op 308 completed, throughput=5MB/sec 2022-04-22T17:47:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:50 smithi121 bash[36902]: audit 2022-04-22T17:47:50.094734+0000 mon.a (mon.0) 2185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:50 smithi121 bash[36902]: cluster 2022-04-22T17:47:50.270649+0000 mgr.y (mgr.34107) 796 : cluster [DBG] pgmap v1434: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:51.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:50 smithi115 bash[45495]: audit 2022-04-22T17:47:50.094734+0000 mon.a (mon.0) 2185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:51.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:50 smithi115 bash[45495]: cluster 2022-04-22T17:47:50.270649+0000 mgr.y (mgr.34107) 796 : cluster [DBG] pgmap v1434: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:50 smithi115 bash[46934]: audit 2022-04-22T17:47:50.094734+0000 mon.a (mon.0) 2185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:50 smithi115 bash[46934]: cluster 2022-04-22T17:47:50.270649+0000 mgr.y (mgr.34107) 796 : cluster [DBG] pgmap v1434: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:47:51.913 INFO:tasks.workunit.client.0.smithi115.stdout: 66: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:51.937 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9YZim91vnwd45l2 off=1103237 len=812628 2022-04-22T17:47:51.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 309 completed, throughput=4.94MB/sec 2022-04-22T17:47:51.939 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xzT4qT9frgUc92L off=1981385 len=1869848 2022-04-22T17:47:51.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 310 completed, throughput=4.96MB/sec 2022-04-22T17:47:51.939 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-g5yUWXmfBT9yeTZ off=3050786 len=1582621 2022-04-22T17:47:51.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 311 completed, throughput=4.99MB/sec 2022-04-22T17:47:51.940 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rVb4viTvlMCPV2T off=0 len=1850719 2022-04-22T17:47:51.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 312 completed, throughput=5.01MB/sec 2022-04-22T17:47:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:51.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:52 smithi121 bash[36902]: audit 2022-04-22T17:47:52.191169+0000 mon.a (mon.0) 2186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:52 smithi115 bash[45495]: audit 2022-04-22T17:47:52.191169+0000 mon.a (mon.0) 2186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:52 smithi115 bash[46934]: audit 2022-04-22T17:47:52.191169+0000 mon.a (mon.0) 2186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:52.939 INFO:tasks.workunit.client.0.smithi115.stdout: 67: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:52.940 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-djWc5i2-Uu0yWaa off=7511263 len=83072 2022-04-22T17:47:52.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 313 completed, throughput=4.94MB/sec 2022-04-22T17:47:52.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qtfawYv7PHCZvDG off=3846572 len=5424 2022-04-22T17:47:52.947 INFO:tasks.workunit.client.0.smithi115.stdout:op 314 completed, throughput=4.94MB/sec 2022-04-22T17:47:52.947 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D6D7D1CE9tzzxNP off=3091923 len=1734631 2022-04-22T17:47:52.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 315 completed, throughput=4.96MB/sec 2022-04-22T17:47:52.954 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OQy6LVc-giuz0v- off=1837145 len=1784259 2022-04-22T17:47:52.960 INFO:tasks.workunit.client.0.smithi115.stdout:op 316 completed, throughput=4.99MB/sec 2022-04-22T17:47:52.960 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Sfv62wqYdMlKyfh off=10838101 len=555384 2022-04-22T17:47:52.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 317 completed, throughput=5MB/sec 2022-04-22T17:47:52.961 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1pHs7_3BoLMX9yG off=1794719 len=386782 2022-04-22T17:47:52.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 318 completed, throughput=5MB/sec 2022-04-22T17:47:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:53 smithi121 bash[36902]: cluster 2022-04-22T17:47:52.271808+0000 mgr.y (mgr.34107) 797 : cluster [DBG] pgmap v1435: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:47:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:53 smithi115 bash[45495]: cluster 2022-04-22T17:47:52.271808+0000 mgr.y (mgr.34107) 797 : cluster [DBG] pgmap v1435: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:47:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:53 smithi115 bash[46934]: cluster 2022-04-22T17:47:52.271808+0000 mgr.y (mgr.34107) 797 : cluster [DBG] pgmap v1435: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:47:53.962 INFO:tasks.workunit.client.0.smithi115.stdout: 68: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z2TzBW97i8rDNo_ off=848320 len=541194 2022-04-22T17:47:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:op 319 completed, throughput=4.94MB/sec 2022-04-22T17:47:54.014 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0eoz4XatIudiiEA off=12561841 len=1355601 2022-04-22T17:47:54.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 320 completed, throughput=4.95MB/sec 2022-04-22T17:47:54.016 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nQZ4GSuqbja_54A off=17128308 len=1105033 2022-04-22T17:47:54.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 321 completed, throughput=4.97MB/sec 2022-04-22T17:47:54.016 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UJ8_A8pIPKcmJNs off=7234973 len=811062 2022-04-22T17:47:54.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 322 completed, throughput=4.98MB/sec 2022-04-22T17:47:54.017 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pvK1dXOGTXs29Zn off=13176771 len=1600153 2022-04-22T17:47:54.017 INFO:tasks.workunit.client.0.smithi115.stdout:op 323 completed, throughput=5MB/sec 2022-04-22T17:47:54.017 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:47:54.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:47:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:47:53.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:47:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:47:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (10m) 17s ago 24m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:47:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (24m) 82s ago 24m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:47:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (12m) 17s ago 25m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:47:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (12m) 82s ago 25m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (20m) 82s ago 30m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (20m) 17s ago 32m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (19m) 17s ago 32m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (18m) 82s ago 30m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (19m) 17s ago 30m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (12m) 17s ago 24m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (11m) 82s ago 24m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (17m) 17s ago 29m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (16m) 17s ago 29m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:47:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (16m) 17s ago 28m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (16m) 17s ago 28m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (15m) 82s ago 27m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (14m) 82s ago 27m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (14m) 82s ago 26m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (13m) 82s ago 26m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:47:54.361 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (11m) 82s ago 25m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:47:54.762 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:47:54.762 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:47:54.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:47:54.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:47:54.763 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:47:54.764 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:47:54.764 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:47:54.764 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:47:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:54 smithi121 bash[36902]: audit 2022-04-22T17:47:53.991955+0000 mgr.y (mgr.34107) 798 : audit [DBG] from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:54 smithi121 bash[36902]: cluster 2022-04-22T17:47:54.272890+0000 mgr.y (mgr.34107) 799 : cluster [DBG] pgmap v1436: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:47:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:54 smithi115 bash[45495]: audit 2022-04-22T17:47:53.991955+0000 mgr.y (mgr.34107) 798 : audit [DBG] from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:54 smithi115 bash[45495]: cluster 2022-04-22T17:47:54.272890+0000 mgr.y (mgr.34107) 799 : cluster [DBG] pgmap v1436: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:47:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:54 smithi115 bash[46934]: audit 2022-04-22T17:47:53.991955+0000 mgr.y (mgr.34107) 798 : audit [DBG] from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:54 smithi115 bash[46934]: cluster 2022-04-22T17:47:54.272890+0000 mgr.y (mgr.34107) 799 : cluster [DBG] pgmap v1436: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:47:54.983 INFO:tasks.workunit.client.0.smithi115.stdout: 69: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:54.984 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sPewKmoIskh1pK1 off=0 len=1245702 2022-04-22T17:47:54.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 324 completed, throughput=4.95MB/sec 2022-04-22T17:47:54.992 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7845OtfdR4sWscw off=2982181 len=2002900 2022-04-22T17:47:54.996 INFO:tasks.workunit.client.0.smithi115.stdout:op 325 completed, throughput=4.97MB/sec 2022-04-22T17:47:54.996 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Lp5be_HokhmMg7X off=6703740 len=508216 2022-04-22T17:47:54.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 326 completed, throughput=4.98MB/sec 2022-04-22T17:47:54.998 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mdKR0vbUmd98MhN off=3608537 len=265750 2022-04-22T17:47:55.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 327 completed, throughput=4.98MB/sec 2022-04-22T17:47:55.002 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_52sXt5v5cG-FKY off=10295885 len=1153173 2022-04-22T17:47:55.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 328 completed, throughput=5MB/sec 2022-04-22T17:47:55.004 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7b3il-wHN5BWooF off=8140229 len=1134705 2022-04-22T17:47:55.007 INFO:tasks.workunit.client.0.smithi115.stdout:op 329 completed, throughput=5.01MB/sec 2022-04-22T17:47:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:55 smithi121 bash[36902]: audit 2022-04-22T17:47:54.334986+0000 mgr.y (mgr.34107) 800 : audit [DBG] from='client.45193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:55 smithi121 bash[36902]: audit 2022-04-22T17:47:54.751999+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.115:0/3404831707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:55 smithi121 bash[36902]: audit 2022-04-22T17:47:54.813326+0000 mon.a (mon.0) 2187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:55 smithi121 bash[36902]: audit 2022-04-22T17:47:55.098299+0000 mon.a (mon.0) 2188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[45495]: audit 2022-04-22T17:47:54.334986+0000 mgr.y (mgr.34107) 800 : audit [DBG] from='client.45193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[45495]: audit 2022-04-22T17:47:54.751999+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.115:0/3404831707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[45495]: audit 2022-04-22T17:47:54.813326+0000 mon.a (mon.0) 2187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[45495]: audit 2022-04-22T17:47:55.098299+0000 mon.a (mon.0) 2188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[46934]: audit 2022-04-22T17:47:54.334986+0000 mgr.y (mgr.34107) 800 : audit [DBG] from='client.45193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:47:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[46934]: audit 2022-04-22T17:47:54.751999+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.115:0/3404831707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:47:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[46934]: audit 2022-04-22T17:47:54.813326+0000 mon.a (mon.0) 2187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:47:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:55 smithi115 bash[46934]: audit 2022-04-22T17:47:55.098299+0000 mon.a (mon.0) 2188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:47:56.007 INFO:tasks.workunit.client.0.smithi115.stdout: 70: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:56.007 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6kfIqwZ84BQYcEe off=3421107 len=367515 2022-04-22T17:47:56.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 330 completed, throughput=4.95MB/sec 2022-04-22T17:47:56.008 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2y1wPjolwHj2dDa off=1908331 len=1890330 2022-04-22T17:47:56.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 331 completed, throughput=4.97MB/sec 2022-04-22T17:47:56.011 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uoXdQaDfqfKf_0R off=15169575 len=1507965 2022-04-22T17:47:56.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 332 completed, throughput=4.99MB/sec 2022-04-22T17:47:56.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gymiAs-Wc_L42s5 off=13452515 len=1019290 2022-04-22T17:47:56.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 333 completed, throughput=5.01MB/sec 2022-04-22T17:47:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:56 smithi121 bash[36902]: cluster 2022-04-22T17:47:56.273986+0000 mgr.y (mgr.34107) 801 : cluster [DBG] pgmap v1437: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:47:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:56 smithi115 bash[45495]: cluster 2022-04-22T17:47:56.273986+0000 mgr.y (mgr.34107) 801 : cluster [DBG] pgmap v1437: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:47:56.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:56 smithi115 bash[46934]: cluster 2022-04-22T17:47:56.273986+0000 mgr.y (mgr.34107) 801 : cluster [DBG] pgmap v1437: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:47:57.021 INFO:tasks.workunit.client.0.smithi115.stdout: 71: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:57.021 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-in8i_EW9Aeowd0X off=7413297 len=1757709 2022-04-22T17:47:57.025 INFO:tasks.workunit.client.0.smithi115.stdout:op 334 completed, throughput=4.96MB/sec 2022-04-22T17:47:57.025 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W9TORoLL59IHRhW off=1036283 len=1622527 2022-04-22T17:47:57.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 335 completed, throughput=4.98MB/sec 2022-04-22T17:47:57.032 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-83vJUWpPAz2LwVw off=3474011 len=213021 2022-04-22T17:47:57.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 336 completed, throughput=4.98MB/sec 2022-04-22T17:47:57.037 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9XwdfHx9qym3ntx off=0 len=1487194 2022-04-22T17:47:57.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 337 completed, throughput=5MB/sec 2022-04-22T17:47:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:57 smithi121 bash[36902]: audit 2022-04-22T17:47:57.194106+0000 mon.a (mon.0) 2189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:57 smithi115 bash[45495]: audit 2022-04-22T17:47:57.194106+0000 mon.a (mon.0) 2189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:57.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:57 smithi115 bash[46934]: audit 2022-04-22T17:47:57.194106+0000 mon.a (mon.0) 2189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:47:58.043 INFO:tasks.workunit.client.0.smithi115.stdout: 72: throughput=4.94MB/sec pending data=0 2022-04-22T17:47:58.043 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Nfou2akAqoWIN8c off=1144534 len=1445219 2022-04-22T17:47:58.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 338 completed, throughput=4.95MB/sec 2022-04-22T17:47:58.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AsDJZJgKwkU9yzL off=11080504 len=1169299 2022-04-22T17:47:58.059 INFO:tasks.workunit.client.0.smithi115.stdout:op 339 completed, throughput=4.97MB/sec 2022-04-22T17:47:58.059 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sUyzUIuOIiq7EE- off=5110069 len=756098 2022-04-22T17:47:58.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 340 completed, throughput=4.98MB/sec 2022-04-22T17:47:58.061 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-csB-wRYEuJeKszS off=15640335 len=297242 2022-04-22T17:47:58.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 341 completed, throughput=4.98MB/sec 2022-04-22T17:47:58.062 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AR_YZGCDlFePr_9 off=15463844 len=561861 2022-04-22T17:47:58.064 INFO:tasks.workunit.client.0.smithi115.stdout:op 342 completed, throughput=4.99MB/sec 2022-04-22T17:47:58.064 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uWVuDnqTCMcpQAM off=0 len=1079550 2022-04-22T17:47:58.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 343 completed, throughput=5MB/sec 2022-04-22T17:47:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:58 smithi121 bash[36902]: cluster 2022-04-22T17:47:58.275217+0000 mgr.y (mgr.34107) 802 : cluster [DBG] pgmap v1438: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:58 smithi115 bash[45495]: cluster 2022-04-22T17:47:58.275217+0000 mgr.y (mgr.34107) 802 : cluster [DBG] pgmap v1438: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:58.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:58 smithi115 bash[46934]: cluster 2022-04-22T17:47:58.275217+0000 mgr.y (mgr.34107) 802 : cluster [DBG] pgmap v1438: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:47:59.071 INFO:tasks.workunit.client.0.smithi115.stdout: 73: throughput=4.93MB/sec pending data=0 2022-04-22T17:47:59.071 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wc3I65IxOfSQoXe off=649673 len=502388 2022-04-22T17:47:59.076 INFO:tasks.workunit.client.0.smithi115.stdout:op 344 completed, throughput=4.94MB/sec 2022-04-22T17:47:59.076 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nDRTmHfR4mkaWqg off=162373 len=909295 2022-04-22T17:47:59.083 INFO:tasks.workunit.client.0.smithi115.stdout:op 345 completed, throughput=4.95MB/sec 2022-04-22T17:47:59.083 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-prAeu1aI7zatPqN off=13576839 len=1870542 2022-04-22T17:47:59.086 INFO:tasks.workunit.client.0.smithi115.stdout:op 346 completed, throughput=4.98MB/sec 2022-04-22T17:47:59.086 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vUgGsjEEFo9V6-1 off=8228631 len=827127 2022-04-22T17:47:59.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 347 completed, throughput=4.99MB/sec 2022-04-22T17:47:59.089 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zzk6Z_7k28n2akX off=5221240 len=2034600 2022-04-22T17:47:59.093 INFO:tasks.workunit.client.0.smithi115.stdout:op 348 completed, throughput=5.01MB/sec 2022-04-22T17:48:00.093 INFO:tasks.workunit.client.0.smithi115.stdout: 74: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:00.093 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q95isC1XBs3iYeo off=2985134 len=1136898 2022-04-22T17:48:00.100 INFO:tasks.workunit.client.0.smithi115.stdout:op 349 completed, throughput=4.96MB/sec 2022-04-22T17:48:00.101 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U5a1KbKTUKE3ecs off=5200689 len=1817828 2022-04-22T17:48:00.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 350 completed, throughput=4.98MB/sec 2022-04-22T17:48:00.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tyLAuKfBbt7MfNI off=1690509 len=694314 2022-04-22T17:48:00.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 351 completed, throughput=4.99MB/sec 2022-04-22T17:48:00.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c3hJmpXCAzI-oLo off=1612119 len=1033826 2022-04-22T17:48:00.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 352 completed, throughput=5MB/sec 2022-04-22T17:48:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:47:59 smithi121 bash[36902]: audit 2022-04-22T17:47:59.816556+0000 mon.a (mon.0) 2190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:47:59 smithi115 bash[45495]: audit 2022-04-22T17:47:59.816556+0000 mon.a (mon.0) 2190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:47:59 smithi115 bash[46934]: audit 2022-04-22T17:47:59.816556+0000 mon.a (mon.0) 2190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:01.118 INFO:tasks.workunit.client.0.smithi115.stdout: 75: throughput=4.94MB/sec pending data=0 2022-04-22T17:48:01.118 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BlP5YLHZZ9VJ5hG off=0 len=1554161 2022-04-22T17:48:01.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 353 completed, throughput=4.96MB/sec 2022-04-22T17:48:01.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xkKYgrg4MJVy8jq off=10702511 len=1932453 2022-04-22T17:48:01.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 354 completed, throughput=4.98MB/sec 2022-04-22T17:48:01.137 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-liQO_pRBiTXChYr off=3131382 len=1005573 2022-04-22T17:48:01.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 355 completed, throughput=4.99MB/sec 2022-04-22T17:48:01.140 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_Wo-IAFp6h1lnZQ off=10600940 len=632140 2022-04-22T17:48:01.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 356 completed, throughput=5MB/sec 2022-04-22T17:48:01.145 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bUxK8sF8QfjDAdX off=1991751 len=1781165 2022-04-22T17:48:01.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 357 completed, throughput=5.02MB/sec 2022-04-22T17:48:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:00 smithi121 bash[36902]: audit 2022-04-22T17:48:00.101792+0000 mon.a (mon.0) 2191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:00 smithi121 bash[36902]: cluster 2022-04-22T17:48:00.276444+0000 mgr.y (mgr.34107) 803 : cluster [DBG] pgmap v1439: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:48:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:00 smithi115 bash[45495]: audit 2022-04-22T17:48:00.101792+0000 mon.a (mon.0) 2191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:00 smithi115 bash[45495]: cluster 2022-04-22T17:48:00.276444+0000 mgr.y (mgr.34107) 803 : cluster [DBG] pgmap v1439: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:48:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:00 smithi115 bash[46934]: audit 2022-04-22T17:48:00.101792+0000 mon.a (mon.0) 2191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:00 smithi115 bash[46934]: cluster 2022-04-22T17:48:00.276444+0000 mgr.y (mgr.34107) 803 : cluster [DBG] pgmap v1439: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 3 op/s 2022-04-22T17:48:02.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:01.671Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:02.149 INFO:tasks.workunit.client.0.smithi115.stdout: 76: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:02.149 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h8RjFRMvHdfJjbq off=5164087 len=1073548 2022-04-22T17:48:02.156 INFO:tasks.workunit.client.0.smithi115.stdout:op 358 completed, throughput=4.97MB/sec 2022-04-22T17:48:02.156 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nuNcIS2bbq0QNqp off=2340917 len=1358671 2022-04-22T17:48:02.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 359 completed, throughput=4.99MB/sec 2022-04-22T17:48:02.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wudk5-0sx4gqRzu off=13689813 len=278895 2022-04-22T17:48:02.161 INFO:tasks.workunit.client.0.smithi115.stdout:op 360 completed, throughput=4.99MB/sec 2022-04-22T17:48:02.162 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R4L4SSfdeiSyH50 off=6960795 len=847112 2022-04-22T17:48:02.164 INFO:tasks.workunit.client.0.smithi115.stdout:op 361 completed, throughput=5MB/sec 2022-04-22T17:48:02.164 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SlMZmcLztdr0DiJ off=1364675 len=5633 2022-04-22T17:48:02.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 362 completed, throughput=5MB/sec 2022-04-22T17:48:02.167 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E_0t2oiv9VJEqbb off=16536297 len=1289950 2022-04-22T17:48:02.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 363 completed, throughput=5.02MB/sec 2022-04-22T17:48:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:02 smithi121 bash[36902]: audit 2022-04-22T17:48:02.197111+0000 mon.a (mon.0) 2192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:02 smithi115 bash[45495]: audit 2022-04-22T17:48:02.197111+0000 mon.a (mon.0) 2192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:02 smithi115 bash[46934]: audit 2022-04-22T17:48:02.197111+0000 mon.a (mon.0) 2192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:03.173 INFO:tasks.workunit.client.0.smithi115.stdout: 77: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:03.174 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1pm12xYIMlOKoFt off=4819516 len=1850414 2022-04-22T17:48:03.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 364 completed, throughput=4.97MB/sec 2022-04-22T17:48:03.184 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EQiOpHFpVtXAXEH off=1862805 len=1527310 2022-04-22T17:48:03.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 365 completed, throughput=4.99MB/sec 2022-04-22T17:48:03.192 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p58gNVdNEVE4cMk off=2960965 len=1573073 2022-04-22T17:48:03.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 366 completed, throughput=5.01MB/sec 2022-04-22T17:48:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:03 smithi115 bash[46934]: cluster 2022-04-22T17:48:02.277503+0000 mgr.y (mgr.34107) 804 : cluster [DBG] pgmap v1440: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:48:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:03 smithi115 bash[45495]: cluster 2022-04-22T17:48:02.277503+0000 mgr.y (mgr.34107) 804 : cluster [DBG] pgmap v1440: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:48:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:03 smithi121 bash[36902]: cluster 2022-04-22T17:48:02.277503+0000 mgr.y (mgr.34107) 804 : cluster [DBG] pgmap v1440: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:48:04.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:03.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:04.195 INFO:tasks.workunit.client.0.smithi115.stdout: 78: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:04.196 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9GodEky_yVqKRH_ off=3926385 len=1818953 2022-04-22T17:48:04.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 367 completed, throughput=4.97MB/sec 2022-04-22T17:48:04.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i2PwZT1_5ZNp6IV off=13232713 len=371350 2022-04-22T17:48:04.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 368 completed, throughput=4.97MB/sec 2022-04-22T17:48:04.202 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ge7owQwp3fLbdxD off=2911012 len=1755865 2022-04-22T17:48:04.211 INFO:tasks.workunit.client.0.smithi115.stdout:op 369 completed, throughput=4.99MB/sec 2022-04-22T17:48:04.211 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jVGO42RWyfTHEkH off=7276074 len=173458 2022-04-22T17:48:04.211 INFO:tasks.workunit.client.0.smithi115.stdout:op 370 completed, throughput=5MB/sec 2022-04-22T17:48:04.212 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2Sq5_IvtVTfMQaY off=17883396 len=31559 2022-04-22T17:48:04.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 371 completed, throughput=5MB/sec 2022-04-22T17:48:04.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X4XAjIuHEuTr73J off=18013305 len=352619 2022-04-22T17:48:04.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 372 completed, throughput=5MB/sec 2022-04-22T17:48:04.216 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9B-RB3fnr0IsnP3 off=7437268 len=555213 2022-04-22T17:48:04.221 INFO:tasks.workunit.client.0.smithi115.stdout:op 373 completed, throughput=5.01MB/sec 2022-04-22T17:48:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:04 smithi121 bash[36902]: cluster 2022-04-22T17:48:04.278547+0000 mgr.y (mgr.34107) 805 : cluster [DBG] pgmap v1441: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:04 smithi115 bash[45495]: cluster 2022-04-22T17:48:04.278547+0000 mgr.y (mgr.34107) 805 : cluster [DBG] pgmap v1441: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:04.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:04 smithi115 bash[46934]: cluster 2022-04-22T17:48:04.278547+0000 mgr.y (mgr.34107) 805 : cluster [DBG] pgmap v1441: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:05.221 INFO:tasks.workunit.client.0.smithi115.stdout: 79: throughput=4.94MB/sec pending data=0 2022-04-22T17:48:05.222 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oHz4KtQScLxL5wF off=5201072 len=1569393 2022-04-22T17:48:05.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 374 completed, throughput=4.96MB/sec 2022-04-22T17:48:05.225 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E4DLINX9_-xhb-0 off=8061332 len=1944947 2022-04-22T17:48:05.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 375 completed, throughput=4.99MB/sec 2022-04-22T17:48:05.230 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LgHS3hBxLJxmU8x off=5801598 len=1771713 2022-04-22T17:48:05.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 376 completed, throughput=5.01MB/sec 2022-04-22T17:48:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:05 smithi121 bash[36902]: audit 2022-04-22T17:48:04.819836+0000 mon.a (mon.0) 2193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:05.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:05 smithi121 bash[36902]: audit 2022-04-22T17:48:05.106380+0000 mon.a (mon.0) 2194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:05.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:05 smithi115 bash[45495]: audit 2022-04-22T17:48:04.819836+0000 mon.a (mon.0) 2193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:05.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:05 smithi115 bash[45495]: audit 2022-04-22T17:48:05.106380+0000 mon.a (mon.0) 2194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:05 smithi115 bash[46934]: audit 2022-04-22T17:48:04.819836+0000 mon.a (mon.0) 2193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:05 smithi115 bash[46934]: audit 2022-04-22T17:48:05.106380+0000 mon.a (mon.0) 2194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:06.236 INFO:tasks.workunit.client.0.smithi115.stdout: 80: throughput=4.94MB/sec pending data=0 2022-04-22T17:48:06.236 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HXjLW6tuNZCn3Zu off=2803589 len=247282 2022-04-22T17:48:06.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 377 completed, throughput=4.95MB/sec 2022-04-22T17:48:06.241 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-njnuvQcWRZ_ODLe off=17709846 len=827539 2022-04-22T17:48:06.244 INFO:tasks.workunit.client.0.smithi115.stdout:op 378 completed, throughput=4.96MB/sec 2022-04-22T17:48:06.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3wffABXMzRMmccB off=5074925 len=40386 2022-04-22T17:48:06.245 INFO:tasks.workunit.client.0.smithi115.stdout:op 379 completed, throughput=4.96MB/sec 2022-04-22T17:48:06.245 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f3T80l3GuoPnJVV off=14699578 len=1318759 2022-04-22T17:48:06.253 INFO:tasks.workunit.client.0.smithi115.stdout:op 380 completed, throughput=4.97MB/sec 2022-04-22T17:48:06.253 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y3e25AAk7A4WJ3a off=10707922 len=789081 2022-04-22T17:48:06.258 INFO:tasks.workunit.client.0.smithi115.stdout:op 381 completed, throughput=4.98MB/sec 2022-04-22T17:48:06.258 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YCxDD6gDxDyiTiN off=17637093 len=1130480 2022-04-22T17:48:06.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 382 completed, throughput=4.99MB/sec 2022-04-22T17:48:06.264 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VS4ypcYupqsBhdp off=2897865 len=1960064 2022-04-22T17:48:06.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 383 completed, throughput=5.02MB/sec 2022-04-22T17:48:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:06 smithi121 bash[36902]: cluster 2022-04-22T17:48:06.279499+0000 mgr.y (mgr.34107) 806 : cluster [DBG] pgmap v1442: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:06 smithi115 bash[45495]: cluster 2022-04-22T17:48:06.279499+0000 mgr.y (mgr.34107) 806 : cluster [DBG] pgmap v1442: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:06.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:06 smithi115 bash[46934]: cluster 2022-04-22T17:48:06.279499+0000 mgr.y (mgr.34107) 806 : cluster [DBG] pgmap v1442: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:07.267 INFO:tasks.workunit.client.0.smithi115.stdout: 81: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:07.268 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MV-cdOemrKr4mkS off=4172421 len=1559167 2022-04-22T17:48:07.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 384 completed, throughput=4.97MB/sec 2022-04-22T17:48:07.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dZB3Q7PKwEStXbI off=6140638 len=1993843 2022-04-22T17:48:07.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 385 completed, throughput=5MB/sec 2022-04-22T17:48:07.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--eo307AV7cgLsjx off=8724218 len=304303 2022-04-22T17:48:07.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 386 completed, throughput=5MB/sec 2022-04-22T17:48:07.278 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CxmSjzgWRxcUQou off=2311553 len=773952 2022-04-22T17:48:07.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 387 completed, throughput=5.01MB/sec 2022-04-22T17:48:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:07 smithi121 bash[36902]: audit 2022-04-22T17:48:07.199816+0000 mon.a (mon.0) 2195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:07 smithi115 bash[45495]: audit 2022-04-22T17:48:07.199816+0000 mon.a (mon.0) 2195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:07.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:07 smithi115 bash[46934]: audit 2022-04-22T17:48:07.199816+0000 mon.a (mon.0) 2195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:08.284 INFO:tasks.workunit.client.0.smithi115.stdout: 82: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:08.335 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-N3_HfuHZZ2cOOF_ off=1273868 len=1605570 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 388 completed, throughput=4.97MB/sec 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-su9Ecnq-hBeawCD off=7335330 len=1115088 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 389 completed, throughput=4.98MB/sec 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mPyN6vumRS_sQ7T off=16200896 len=1742394 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 390 completed, throughput=5MB/sec 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aUWb7yQekXVYvS6 off=15030931 len=1617607 2022-04-22T17:48:08.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 391 completed, throughput=5.02MB/sec 2022-04-22T17:48:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:08 smithi121 bash[36902]: cluster 2022-04-22T17:48:08.281122+0000 mgr.y (mgr.34107) 807 : cluster [DBG] pgmap v1443: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:48:08.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:08 smithi115 bash[45495]: cluster 2022-04-22T17:48:08.281122+0000 mgr.y (mgr.34107) 807 : cluster [DBG] pgmap v1443: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:48:08.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:08 smithi115 bash[46934]: cluster 2022-04-22T17:48:08.281122+0000 mgr.y (mgr.34107) 807 : cluster [DBG] pgmap v1443: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:48:09.311 INFO:tasks.workunit.client.0.smithi115.stdout: 83: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:09.360 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vk6UFr6F0erndgd off=1807467 len=418296 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 392 completed, throughput=4.96MB/sec 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eEQ4F3Bjgje2sWr off=9867475 len=333917 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 393 completed, throughput=4.96MB/sec 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vGrUMfKu9haUV5b off=3318521 len=698447 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 394 completed, throughput=4.97MB/sec 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0fnjgsaRDmb_U35 off=9930098 len=1310808 2022-04-22T17:48:09.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 395 completed, throughput=4.99MB/sec 2022-04-22T17:48:09.363 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lKyrFaHu85tZx0X off=4376672 len=1507501 2022-04-22T17:48:09.363 INFO:tasks.workunit.client.0.smithi115.stdout:op 396 completed, throughput=5MB/sec 2022-04-22T17:48:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:09 smithi121 bash[36902]: audit 2022-04-22T17:48:09.823001+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:09 smithi115 bash[45495]: audit 2022-04-22T17:48:09.823001+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:09 smithi115 bash[46934]: audit 2022-04-22T17:48:09.823001+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:10.335 INFO:tasks.workunit.client.0.smithi115.stdout: 84: throughput=4.94MB/sec pending data=0 2022-04-22T17:48:10.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W-IQ4UJfu2eKIgf off=2255826 len=750317 2022-04-22T17:48:10.342 INFO:tasks.workunit.client.0.smithi115.stdout:op 397 completed, throughput=4.95MB/sec 2022-04-22T17:48:10.342 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GyQOZwtw3X6Jr7N off=13508010 len=239186 2022-04-22T17:48:10.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 398 completed, throughput=4.96MB/sec 2022-04-22T17:48:10.346 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FuL9hQlJ1CGzQkX off=411356 len=176718 2022-04-22T17:48:10.350 INFO:tasks.workunit.client.0.smithi115.stdout:op 399 completed, throughput=4.96MB/sec 2022-04-22T17:48:10.350 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yEKv1VrqOvXOmUK off=3392563 len=362709 2022-04-22T17:48:10.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 400 completed, throughput=4.96MB/sec 2022-04-22T17:48:10.354 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nCaIqgqkJDS5Y_Y off=2878492 len=1307766 2022-04-22T17:48:10.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 401 completed, throughput=4.97MB/sec 2022-04-22T17:48:10.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OeqvKMPMC8n6MQ1 off=1648175 len=961717 2022-04-22T17:48:10.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 402 completed, throughput=4.99MB/sec 2022-04-22T17:48:10.360 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nWRNzzdD6gNtjo4 off=3605102 len=1404956 2022-04-22T17:48:10.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 403 completed, throughput=5MB/sec 2022-04-22T17:48:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:10 smithi121 bash[36902]: audit 2022-04-22T17:48:10.109816+0000 mon.a (mon.0) 2197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:10 smithi121 bash[36902]: cluster 2022-04-22T17:48:10.282247+0000 mgr.y (mgr.34107) 808 : cluster [DBG] pgmap v1444: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:10 smithi115 bash[45495]: audit 2022-04-22T17:48:10.109816+0000 mon.a (mon.0) 2197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:10 smithi115 bash[45495]: cluster 2022-04-22T17:48:10.282247+0000 mgr.y (mgr.34107) 808 : cluster [DBG] pgmap v1444: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:10 smithi115 bash[46934]: audit 2022-04-22T17:48:10.109816+0000 mon.a (mon.0) 2197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:10 smithi115 bash[46934]: cluster 2022-04-22T17:48:10.282247+0000 mgr.y (mgr.34107) 808 : cluster [DBG] pgmap v1444: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:11.363 INFO:tasks.workunit.client.0.smithi115.stdout: 85: throughput=4.94MB/sec pending data=0 2022-04-22T17:48:11.363 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e7AJkf0sw-6i9u7 off=3106300 len=2044252 2022-04-22T17:48:11.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 404 completed, throughput=4.97MB/sec 2022-04-22T17:48:11.367 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qd92sE5lqDq3j5P off=514891 len=556670 2022-04-22T17:48:11.373 INFO:tasks.workunit.client.0.smithi115.stdout:op 405 completed, throughput=4.97MB/sec 2022-04-22T17:48:11.373 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1Xm7FTClDKhWf0h off=465313 len=1799958 2022-04-22T17:48:11.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 406 completed, throughput=4.99MB/sec 2022-04-22T17:48:11.375 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1ZyiajjjXRmujH- off=2070750 len=1883817 2022-04-22T17:48:11.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 407 completed, throughput=5.01MB/sec 2022-04-22T17:48:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:11.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:12.385 INFO:tasks.workunit.client.0.smithi115.stdout: 86: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:12.385 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3ou25ExVP692H6U off=1069726 len=1881319 2022-04-22T17:48:12.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 408 completed, throughput=4.97MB/sec 2022-04-22T17:48:12.398 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eSO4Ijd_agAneK8 off=3701823 len=1254490 2022-04-22T17:48:12.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 409 completed, throughput=4.99MB/sec 2022-04-22T17:48:12.399 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E_K2XoJSEvNJIlX off=566138 len=621071 2022-04-22T17:48:12.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 410 completed, throughput=4.99MB/sec 2022-04-22T17:48:12.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yJLbo9y-Trm1TXe off=18154 len=1531806 2022-04-22T17:48:12.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 411 completed, throughput=5.01MB/sec 2022-04-22T17:48:12.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:12 smithi115 bash[45495]: audit 2022-04-22T17:48:12.202675+0000 mon.a (mon.0) 2198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:12.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:12 smithi115 bash[46934]: audit 2022-04-22T17:48:12.202675+0000 mon.a (mon.0) 2198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:12 smithi121 bash[36902]: audit 2022-04-22T17:48:12.202675+0000 mon.a (mon.0) 2198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:13.406 INFO:tasks.workunit.client.0.smithi115.stdout: 87: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:13.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6j0SIur0suEtU91 off=2486504 len=396289 2022-04-22T17:48:13.408 INFO:tasks.workunit.client.0.smithi115.stdout:op 412 completed, throughput=4.96MB/sec 2022-04-22T17:48:13.408 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1IO3Uohmsr5bRPY off=3864277 len=1044569 2022-04-22T17:48:13.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 413 completed, throughput=4.97MB/sec 2022-04-22T17:48:13.411 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bZqq-Y4OLGoLK-u off=4950942 len=1800317 2022-04-22T17:48:13.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 414 completed, throughput=4.99MB/sec 2022-04-22T17:48:13.415 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PCuxtMPaJeAKXtJ off=46693 len=1202065 2022-04-22T17:48:13.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 415 completed, throughput=5MB/sec 2022-04-22T17:48:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:13 smithi115 bash[45495]: cluster 2022-04-22T17:48:12.283442+0000 mgr.y (mgr.34107) 809 : cluster [DBG] pgmap v1445: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:13 smithi115 bash[46934]: cluster 2022-04-22T17:48:12.283442+0000 mgr.y (mgr.34107) 809 : cluster [DBG] pgmap v1445: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:13 smithi121 bash[36902]: cluster 2022-04-22T17:48:12.283442+0000 mgr.y (mgr.34107) 809 : cluster [DBG] pgmap v1445: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:14.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:13.557Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:14.419 INFO:tasks.workunit.client.0.smithi115.stdout: 88: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:14.419 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-11lNbuBAQ91974l off=2548556 len=974574 2022-04-22T17:48:14.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 416 completed, throughput=4.96MB/sec 2022-04-22T17:48:14.427 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sIUFfhy6r6mXBqM off=1109158 len=312566 2022-04-22T17:48:14.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 417 completed, throughput=4.96MB/sec 2022-04-22T17:48:14.428 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BDYF8G9K0ukvozo off=7825027 len=1772672 2022-04-22T17:48:14.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 418 completed, throughput=4.98MB/sec 2022-04-22T17:48:14.437 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d6SKoO4iggJvRRb off=321464 len=1601691 2022-04-22T17:48:14.442 INFO:tasks.workunit.client.0.smithi115.stdout:op 419 completed, throughput=4.99MB/sec 2022-04-22T17:48:14.443 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NS0_MW0gSPq7Ldf off=14693256 len=912981 2022-04-22T17:48:14.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 420 completed, throughput=5MB/sec 2022-04-22T17:48:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:14 smithi115 bash[45495]: audit 2022-04-22T17:48:13.443264+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:48:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:14 smithi115 bash[45495]: audit 2022-04-22T17:48:13.443754+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:48:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:14 smithi115 bash[46934]: audit 2022-04-22T17:48:13.443264+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:48:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:14 smithi115 bash[46934]: audit 2022-04-22T17:48:13.443754+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:48:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:14 smithi121 bash[36902]: audit 2022-04-22T17:48:13.443264+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:48:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:14 smithi121 bash[36902]: audit 2022-04-22T17:48:13.443754+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:48:15.448 INFO:tasks.workunit.client.0.smithi115.stdout: 89: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:15.448 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7K6jRa05hwtZEKF off=332100 len=1403049 2022-04-22T17:48:15.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 421 completed, throughput=4.96MB/sec 2022-04-22T17:48:15.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KPgfdDJnR9Ws8l9 off=944041 len=1759283 2022-04-22T17:48:15.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 422 completed, throughput=4.98MB/sec 2022-04-22T17:48:15.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rWAOpu01bEtl5Lf off=5384975 len=2024895 2022-04-22T17:48:15.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 423 completed, throughput=5MB/sec 2022-04-22T17:48:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[45495]: cluster 2022-04-22T17:48:14.284488+0000 mgr.y (mgr.34107) 810 : cluster [DBG] pgmap v1446: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:48:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[45495]: audit 2022-04-22T17:48:14.826275+0000 mon.a (mon.0) 2201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[45495]: audit 2022-04-22T17:48:15.113451+0000 mon.a (mon.0) 2202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[46934]: cluster 2022-04-22T17:48:14.284488+0000 mgr.y (mgr.34107) 810 : cluster [DBG] pgmap v1446: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:48:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[46934]: audit 2022-04-22T17:48:14.826275+0000 mon.a (mon.0) 2201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:15 smithi115 bash[46934]: audit 2022-04-22T17:48:15.113451+0000 mon.a (mon.0) 2202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:15 smithi121 bash[36902]: cluster 2022-04-22T17:48:14.284488+0000 mgr.y (mgr.34107) 810 : cluster [DBG] pgmap v1446: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:48:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:15 smithi121 bash[36902]: audit 2022-04-22T17:48:14.826275+0000 mon.a (mon.0) 2201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:15 smithi121 bash[36902]: audit 2022-04-22T17:48:15.113451+0000 mon.a (mon.0) 2202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:16.467 INFO:tasks.workunit.client.0.smithi115.stdout: 90: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:16.468 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2XR-PTuASXbZdYl off=12742101 len=59126 2022-04-22T17:48:16.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 424 completed, throughput=4.95MB/sec 2022-04-22T17:48:16.472 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z5kOHgU3ZSxyYmI off=834235 len=1556525 2022-04-22T17:48:16.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 425 completed, throughput=4.96MB/sec 2022-04-22T17:48:16.480 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bzpKeD1JBwD76qR off=5241053 len=749148 2022-04-22T17:48:16.485 INFO:tasks.workunit.client.0.smithi115.stdout:op 426 completed, throughput=4.97MB/sec 2022-04-22T17:48:16.485 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OFtEoEViz2Linxg off=20340176 len=1080979 2022-04-22T17:48:16.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 427 completed, throughput=4.98MB/sec 2022-04-22T17:48:16.488 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8rMfHjl4FVF1pt6 off=16021856 len=1436720 2022-04-22T17:48:16.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 428 completed, throughput=5MB/sec 2022-04-22T17:48:16.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mjcvV-1_7PIvgyG off=3723737 len=771186 2022-04-22T17:48:16.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 429 completed, throughput=5MB/sec 2022-04-22T17:48:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:16 smithi121 bash[36902]: cluster 2022-04-22T17:48:16.285619+0000 mgr.y (mgr.34107) 811 : cluster [DBG] pgmap v1447: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:48:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:16 smithi115 bash[45495]: cluster 2022-04-22T17:48:16.285619+0000 mgr.y (mgr.34107) 811 : cluster [DBG] pgmap v1447: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:48:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:16 smithi115 bash[46934]: cluster 2022-04-22T17:48:16.285619+0000 mgr.y (mgr.34107) 811 : cluster [DBG] pgmap v1447: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:48:17.496 INFO:tasks.workunit.client.0.smithi115.stdout: 91: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:17.547 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gm5oeMsP3dqyh9f off=15883806 len=1945287 2022-04-22T17:48:17.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 430 completed, throughput=4.97MB/sec 2022-04-22T17:48:17.550 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MMHadlm033yZvNa off=13040168 len=1638580 2022-04-22T17:48:17.551 INFO:tasks.workunit.client.0.smithi115.stdout:op 431 completed, throughput=4.99MB/sec 2022-04-22T17:48:17.551 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VX9gz5j1SnTQmS3 off=373723 len=1009860 2022-04-22T17:48:17.551 INFO:tasks.workunit.client.0.smithi115.stdout:op 432 completed, throughput=5MB/sec 2022-04-22T17:48:17.552 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lwFXeNnISKpEOfB off=636073 len=1254639 2022-04-22T17:48:17.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 433 completed, throughput=5.01MB/sec 2022-04-22T17:48:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:17 smithi121 bash[36902]: audit 2022-04-22T17:48:17.205795+0000 mon.a (mon.0) 2203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:17.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:17 smithi115 bash[45495]: audit 2022-04-22T17:48:17.205795+0000 mon.a (mon.0) 2203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:17 smithi115 bash[46934]: audit 2022-04-22T17:48:17.205795+0000 mon.a (mon.0) 2203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:18.520 INFO:tasks.workunit.client.0.smithi115.stdout: 92: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:18.520 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iju7LVK0tegqFiN off=3577386 len=1493543 2022-04-22T17:48:18.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 434 completed, throughput=4.97MB/sec 2022-04-22T17:48:18.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t88i-OnJ9D1hhvj off=8729889 len=848960 2022-04-22T17:48:18.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 435 completed, throughput=4.98MB/sec 2022-04-22T17:48:18.535 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PZSLGaEuggSCYPb off=17505790 len=979761 2022-04-22T17:48:18.542 INFO:tasks.workunit.client.0.smithi115.stdout:op 436 completed, throughput=4.99MB/sec 2022-04-22T17:48:18.542 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mOxJP1GowStTDSr off=5229938 len=1547249 2022-04-22T17:48:18.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 437 completed, throughput=5MB/sec 2022-04-22T17:48:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:18 smithi121 bash[36902]: cluster 2022-04-22T17:48:18.286744+0000 mgr.y (mgr.34107) 812 : cluster [DBG] pgmap v1448: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:18 smithi115 bash[45495]: cluster 2022-04-22T17:48:18.286744+0000 mgr.y (mgr.34107) 812 : cluster [DBG] pgmap v1448: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:18 smithi115 bash[46934]: cluster 2022-04-22T17:48:18.286744+0000 mgr.y (mgr.34107) 812 : cluster [DBG] pgmap v1448: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:19.548 INFO:tasks.workunit.client.0.smithi115.stdout: 93: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:19.548 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AK4GDKiDr_b_6pi off=210864 len=1491636 2022-04-22T17:48:19.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 438 completed, throughput=4.97MB/sec 2022-04-22T17:48:19.558 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NVbdo9QNJrDDur7 off=5012326 len=1148073 2022-04-22T17:48:19.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 439 completed, throughput=4.98MB/sec 2022-04-22T17:48:19.566 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-euH9GuQilTMh1WZ off=10794492 len=555987 2022-04-22T17:48:19.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 440 completed, throughput=4.98MB/sec 2022-04-22T17:48:19.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wN8yPp8g-3Y9krn off=3761538 len=1373982 2022-04-22T17:48:19.574 INFO:tasks.workunit.client.0.smithi115.stdout:op 441 completed, throughput=5MB/sec 2022-04-22T17:48:19.574 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5wOylFuAO6pLLfH off=8148779 len=478611 2022-04-22T17:48:19.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 442 completed, throughput=5MB/sec 2022-04-22T17:48:19.575 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7Aj5aRolhs6xtpY off=17764176 len=49040 2022-04-22T17:48:19.578 INFO:tasks.workunit.client.0.smithi115.stdout:op 443 completed, throughput=5MB/sec 2022-04-22T17:48:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:19 smithi121 bash[36902]: audit 2022-04-22T17:48:19.829353+0000 mon.a (mon.0) 2204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:19 smithi115 bash[46934]: audit 2022-04-22T17:48:19.829353+0000 mon.a (mon.0) 2204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:19 smithi115 bash[45495]: audit 2022-04-22T17:48:19.829353+0000 mon.a (mon.0) 2204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:20.579 INFO:tasks.workunit.client.0.smithi115.stdout: 94: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:20.579 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-96uzkQYfip5H5FN off=1695195 len=1932868 2022-04-22T17:48:20.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 444 completed, throughput=4.97MB/sec 2022-04-22T17:48:20.590 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zGjp4CHVMa4tTX_ off=2233478 len=1880735 2022-04-22T17:48:20.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 445 completed, throughput=4.99MB/sec 2022-04-22T17:48:20.594 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7pSW2S4rtIRHdXe off=1368771 len=971718 2022-04-22T17:48:20.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 446 completed, throughput=4.99MB/sec 2022-04-22T17:48:20.596 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IeUvR5lC_TD4t9f off=8562978 len=635613 2022-04-22T17:48:20.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 447 completed, throughput=5MB/sec 2022-04-22T17:48:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:20 smithi121 bash[36902]: audit 2022-04-22T17:48:20.116997+0000 mon.a (mon.0) 2205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:21.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:20 smithi121 bash[36902]: cluster 2022-04-22T17:48:20.287869+0000 mgr.y (mgr.34107) 813 : cluster [DBG] pgmap v1449: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:20 smithi115 bash[45495]: audit 2022-04-22T17:48:20.116997+0000 mon.a (mon.0) 2205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:20 smithi115 bash[45495]: cluster 2022-04-22T17:48:20.287869+0000 mgr.y (mgr.34107) 813 : cluster [DBG] pgmap v1449: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:20 smithi115 bash[46934]: audit 2022-04-22T17:48:20.116997+0000 mon.a (mon.0) 2205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:20 smithi115 bash[46934]: cluster 2022-04-22T17:48:20.287869+0000 mgr.y (mgr.34107) 813 : cluster [DBG] pgmap v1449: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:21.598 INFO:tasks.workunit.client.0.smithi115.stdout: 95: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:21.598 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z3ZMWKCCGGP1WxO off=10856640 len=179076 2022-04-22T17:48:21.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 448 completed, throughput=4.95MB/sec 2022-04-22T17:48:21.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C6j6Kgal_9KNq5D off=2575129 len=675094 2022-04-22T17:48:21.605 INFO:tasks.workunit.client.0.smithi115.stdout:op 449 completed, throughput=4.96MB/sec 2022-04-22T17:48:21.605 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KB-UOYoLyhGk7uo off=3183667 len=235011 2022-04-22T17:48:21.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 450 completed, throughput=4.96MB/sec 2022-04-22T17:48:21.607 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zhb2tPjFmz-UlCu off=4312761 len=1145897 2022-04-22T17:48:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 451 completed, throughput=4.97MB/sec 2022-04-22T17:48:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jPooiDi2Y3lBi9l off=773475 len=685372 2022-04-22T17:48:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 452 completed, throughput=4.98MB/sec 2022-04-22T17:48:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2cc3ip0FnxHVCsm off=197780 len=827558 2022-04-22T17:48:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:op 453 completed, throughput=4.99MB/sec 2022-04-22T17:48:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uOYn-QZaXha6o0s off=0 len=1387047 2022-04-22T17:48:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 454 completed, throughput=5MB/sec 2022-04-22T17:48:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-opz1fUjLO08qsjd off=21054 len=2072596 2022-04-22T17:48:21.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 455 completed, throughput=5.02MB/sec 2022-04-22T17:48:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:21.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:22 smithi115 bash[45495]: audit 2022-04-22T17:48:22.209109+0000 mon.a (mon.0) 2206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:22.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:22 smithi115 bash[46934]: audit 2022-04-22T17:48:22.209109+0000 mon.a (mon.0) 2206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:22.627 INFO:tasks.workunit.client.0.smithi115.stdout: 96: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:22.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LCBhTp7VgyrCqdZ off=47677 len=1631630 2022-04-22T17:48:22.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 456 completed, throughput=4.98MB/sec 2022-04-22T17:48:22.708 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uiZWgae_XzIFULG off=6004423 len=1302244 2022-04-22T17:48:22.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 457 completed, throughput=5MB/sec 2022-04-22T17:48:22.708 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OAStGVwvGvhJDZb off=11037489 len=1692604 2022-04-22T17:48:22.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 458 completed, throughput=5.01MB/sec 2022-04-22T17:48:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:22 smithi121 bash[36902]: audit 2022-04-22T17:48:22.209109+0000 mon.a (mon.0) 2206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:23.649 INFO:tasks.workunit.client.0.smithi115.stdout: 97: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:23.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZBWOvHe4Era4YrA off=10838059 len=313092 2022-04-22T17:48:23.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 459 completed, throughput=4.96MB/sec 2022-04-22T17:48:23.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cqU58uOhr8bJzqQ off=15717185 len=726195 2022-04-22T17:48:23.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 460 completed, throughput=4.97MB/sec 2022-04-22T17:48:23.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mAOfooDeIvz7_Jx off=3535123 len=1582017 2022-04-22T17:48:23.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 461 completed, throughput=4.99MB/sec 2022-04-22T17:48:23.694 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xr1p84_N6PTr_4P off=709354 len=685328 2022-04-22T17:48:23.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 462 completed, throughput=4.99MB/sec 2022-04-22T17:48:23.694 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GCU9GBBEmrOPU_l off=16886715 len=656574 2022-04-22T17:48:23.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 463 completed, throughput=5MB/sec 2022-04-22T17:48:23.695 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LoN98Sn6mxTnb0E off=10835883 len=694417 2022-04-22T17:48:23.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 464 completed, throughput=5MB/sec 2022-04-22T17:48:24.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:23.558Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:24.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:24 smithi115 bash[45495]: cluster 2022-04-22T17:48:22.289120+0000 mgr.y (mgr.34107) 814 : cluster [DBG] pgmap v1450: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:24 smithi115 bash[46934]: cluster 2022-04-22T17:48:22.289120+0000 mgr.y (mgr.34107) 814 : cluster [DBG] pgmap v1450: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:24.680 INFO:tasks.workunit.client.0.smithi115.stdout: 98: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:24.680 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eYQQi9g_x57z9fG off=11959702 len=1860190 2022-04-22T17:48:24.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 465 completed, throughput=4.97MB/sec 2022-04-22T17:48:24.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nSb_ioDzDHMUlOf off=9685133 len=448167 2022-04-22T17:48:24.691 INFO:tasks.workunit.client.0.smithi115.stdout:op 466 completed, throughput=4.98MB/sec 2022-04-22T17:48:24.691 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x4H5tahkaHrSaaw off=4860532 len=1524981 2022-04-22T17:48:24.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 467 completed, throughput=4.99MB/sec 2022-04-22T17:48:24.695 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wdPQuCeKn5Ht80X off=14222405 len=37172 2022-04-22T17:48:24.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 468 completed, throughput=4.99MB/sec 2022-04-22T17:48:24.698 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CMRfV5G3jH9UuRu off=1722958 len=1652630 2022-04-22T17:48:24.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 469 completed, throughput=5.01MB/sec 2022-04-22T17:48:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:24 smithi121 bash[36902]: cluster 2022-04-22T17:48:22.289120+0000 mgr.y (mgr.34107) 814 : cluster [DBG] pgmap v1450: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:25.134 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:48:25.475 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:48:25.510 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (11m) 48s ago 25m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:48:25.510 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (25m) 113s ago 25m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (13m) 48s ago 26m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (12m) 113s ago 26m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (20m) 113s ago 30m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (21m) 48s ago 32m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (20m) 48s ago 32m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (19m) 113s ago 30m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (19m) 48s ago 31m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:48:25.511 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (12m) 48s ago 25m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (12m) 113s ago 25m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (18m) 48s ago 30m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (17m) 48s ago 29m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (17m) 48s ago 29m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (16m) 48s ago 28m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (15m) 113s ago 28m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (15m) 113s ago 27m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (14m) 113s ago 27m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:48:25.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (13m) 113s ago 26m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:48:25.513 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (12m) 113s ago 25m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:48:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:25 smithi115 bash[45495]: audit 2022-04-22T17:48:24.832455+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:25.569 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:25 smithi115 bash[45495]: audit 2022-04-22T17:48:25.120383+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:25.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:25 smithi115 bash[46934]: audit 2022-04-22T17:48:24.832455+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:25.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:25 smithi115 bash[46934]: audit 2022-04-22T17:48:25.120383+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:25.707 INFO:tasks.workunit.client.0.smithi115.stdout: 99: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:25.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2FJuFZ9tQ-fzbjV off=1077637 len=1668221 2022-04-22T17:48:25.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 470 completed, throughput=4.97MB/sec 2022-04-22T17:48:25.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lr8tWD1uEI6cdt4 off=9049575 len=1560972 2022-04-22T17:48:25.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 471 completed, throughput=4.99MB/sec 2022-04-22T17:48:25.718 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xyeg2n940Omaygi off=7914330 len=342906 2022-04-22T17:48:25.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 472 completed, throughput=4.99MB/sec 2022-04-22T17:48:25.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vCJJOdGEs2d-Agv off=4136691 len=2006135 2022-04-22T17:48:25.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 473 completed, throughput=5.01MB/sec 2022-04-22T17:48:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:25 smithi121 bash[36902]: audit 2022-04-22T17:48:24.832455+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:25 smithi121 bash[36902]: audit 2022-04-22T17:48:25.120383+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:25.879 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:48:25.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:48:25.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:25.881 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:48:25.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:48:25.881 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:48:25.881 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:48:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[45495]: cluster 2022-04-22T17:48:24.290101+0000 mgr.y (mgr.34107) 815 : cluster [DBG] pgmap v1451: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[45495]: audit 2022-04-22T17:48:25.109409+0000 mgr.y (mgr.34107) 816 : audit [DBG] from='client.45205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[45495]: audit 2022-04-22T17:48:25.869920+0000 mon.a (mon.0) 2209 : audit [DBG] from='client.? 172.21.15.115:0/741471138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[46934]: cluster 2022-04-22T17:48:24.290101+0000 mgr.y (mgr.34107) 815 : cluster [DBG] pgmap v1451: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[46934]: audit 2022-04-22T17:48:25.109409+0000 mgr.y (mgr.34107) 816 : audit [DBG] from='client.45205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:26 smithi115 bash[46934]: audit 2022-04-22T17:48:25.869920+0000 mon.a (mon.0) 2209 : audit [DBG] from='client.? 172.21.15.115:0/741471138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:26.725 INFO:tasks.workunit.client.0.smithi115.stdout: 100: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:26.725 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xs4eJAfy9m25Lzp off=3432297 len=424935 2022-04-22T17:48:26.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 474 completed, throughput=4.96MB/sec 2022-04-22T17:48:26.727 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UtkvwWKnU2rXxWJ off=6759997 len=266741 2022-04-22T17:48:26.732 INFO:tasks.workunit.client.0.smithi115.stdout:op 475 completed, throughput=4.96MB/sec 2022-04-22T17:48:26.732 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pssn3-UzwjLMGuM off=187044 len=327577 2022-04-22T17:48:26.734 INFO:tasks.workunit.client.0.smithi115.stdout:op 476 completed, throughput=4.97MB/sec 2022-04-22T17:48:26.734 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pRPYXa0qDj9BXhr off=6306775 len=1486631 2022-04-22T17:48:26.736 INFO:tasks.workunit.client.0.smithi115.stdout:op 477 completed, throughput=4.98MB/sec 2022-04-22T17:48:26.737 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1e3s-8RcgPmsl0h off=3198737 len=440769 2022-04-22T17:48:26.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 478 completed, throughput=4.99MB/sec 2022-04-22T17:48:26.741 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6JnD9JEggaJy5qV off=768263 len=1729834 2022-04-22T17:48:26.745 INFO:tasks.workunit.client.0.smithi115.stdout:op 479 completed, throughput=5MB/sec 2022-04-22T17:48:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:26 smithi121 bash[36902]: cluster 2022-04-22T17:48:24.290101+0000 mgr.y (mgr.34107) 815 : cluster [DBG] pgmap v1451: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:26 smithi121 bash[36902]: audit 2022-04-22T17:48:25.109409+0000 mgr.y (mgr.34107) 816 : audit [DBG] from='client.45205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:26 smithi121 bash[36902]: audit 2022-04-22T17:48:25.869920+0000 mon.a (mon.0) 2209 : audit [DBG] from='client.? 172.21.15.115:0/741471138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:27 smithi115 bash[45495]: audit 2022-04-22T17:48:25.451360+0000 mgr.y (mgr.34107) 817 : audit [DBG] from='client.45211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:27 smithi115 bash[45495]: audit 2022-04-22T17:48:27.212210+0000 mon.a (mon.0) 2210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:27 smithi115 bash[46934]: audit 2022-04-22T17:48:25.451360+0000 mgr.y (mgr.34107) 817 : audit [DBG] from='client.45211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:27 smithi115 bash[46934]: audit 2022-04-22T17:48:27.212210+0000 mon.a (mon.0) 2210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:27.746 INFO:tasks.workunit.client.0.smithi115.stdout: 101: throughput=4.95MB/sec pending data=0 2022-04-22T17:48:27.746 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QSFdOFgDoWZosQ5 off=4364311 len=743165 2022-04-22T17:48:27.752 INFO:tasks.workunit.client.0.smithi115.stdout:op 480 completed, throughput=4.96MB/sec 2022-04-22T17:48:27.752 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oL56IjzJ03NeOb7 off=13541719 len=597704 2022-04-22T17:48:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 481 completed, throughput=4.96MB/sec 2022-04-22T17:48:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tgjA0zD71YWUhMZ off=21165845 len=1414763 2022-04-22T17:48:27.757 INFO:tasks.workunit.client.0.smithi115.stdout:op 482 completed, throughput=4.98MB/sec 2022-04-22T17:48:27.757 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-osWdm89cHu6qirC off=178732 len=1952065 2022-04-22T17:48:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:27 smithi121 bash[36902]: audit 2022-04-22T17:48:25.451360+0000 mgr.y (mgr.34107) 817 : audit [DBG] from='client.45211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:27 smithi121 bash[36902]: audit 2022-04-22T17:48:27.212210+0000 mon.a (mon.0) 2210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:27.766 INFO:tasks.workunit.client.0.smithi115.stdout:op 483 completed, throughput=5MB/sec 2022-04-22T17:48:27.767 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2QQEijz4RtrSBYE off=8546650 len=1279938 2022-04-22T17:48:27.771 INFO:tasks.workunit.client.0.smithi115.stdout:op 484 completed, throughput=5.01MB/sec 2022-04-22T17:48:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:28 smithi115 bash[45495]: cluster 2022-04-22T17:48:26.291138+0000 mgr.y (mgr.34107) 818 : cluster [DBG] pgmap v1452: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:28 smithi115 bash[46934]: cluster 2022-04-22T17:48:26.291138+0000 mgr.y (mgr.34107) 818 : cluster [DBG] pgmap v1452: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:28 smithi121 bash[36902]: cluster 2022-04-22T17:48:26.291138+0000 mgr.y (mgr.34107) 818 : cluster [DBG] pgmap v1452: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:48:28.772 INFO:tasks.workunit.client.0.smithi115.stdout: 102: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:28.772 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxyrX4kOPDeX2rJ off=17674803 len=2083814 2022-04-22T17:48:28.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 485 completed, throughput=4.98MB/sec 2022-04-22T17:48:28.776 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TsmKtVAYuEZrvG7 off=680449 len=948479 2022-04-22T17:48:28.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 486 completed, throughput=4.99MB/sec 2022-04-22T17:48:28.784 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qjE-1tojuKFt0gm off=6426623 len=3196 2022-04-22T17:48:28.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 487 completed, throughput=4.99MB/sec 2022-04-22T17:48:28.785 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MD-ZYQoY0fw1anF off=2419775 len=383372 2022-04-22T17:48:28.789 INFO:tasks.workunit.client.0.smithi115.stdout:op 488 completed, throughput=4.99MB/sec 2022-04-22T17:48:28.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-foFo2Zbir6aOu1s off=18314315 len=535665 2022-04-22T17:48:28.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 489 completed, throughput=4.99MB/sec 2022-04-22T17:48:28.792 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7q6N0n7kB-0Pcsk off=10272568 len=1509317 2022-04-22T17:48:28.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 490 completed, throughput=5.01MB/sec 2022-04-22T17:48:29.799 INFO:tasks.workunit.client.0.smithi115.stdout: 103: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:29.799 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZrWVfIDFMrAce6F off=2123648 len=664885 2022-04-22T17:48:29.805 INFO:tasks.workunit.client.0.smithi115.stdout:op 491 completed, throughput=4.97MB/sec 2022-04-22T17:48:29.805 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5XGJj8rFu5kl5oe off=8267854 len=203109 2022-04-22T17:48:29.810 INFO:tasks.workunit.client.0.smithi115.stdout:op 492 completed, throughput=4.97MB/sec 2022-04-22T17:48:29.810 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tRUkFaBdxttRwyx off=15944519 len=677057 2022-04-22T17:48:29.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 493 completed, throughput=4.97MB/sec 2022-04-22T17:48:29.812 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JqMp7jY2fWHeGmu off=8528261 len=307724 2022-04-22T17:48:29.813 INFO:tasks.workunit.client.0.smithi115.stdout:op 494 completed, throughput=4.98MB/sec 2022-04-22T17:48:29.813 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YOr832LiL3XwIlM off=3796258 len=1185647 2022-04-22T17:48:29.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 495 completed, throughput=4.99MB/sec 2022-04-22T17:48:29.815 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IOu4OXdxD9-fRbS off=2137741 len=564166 2022-04-22T17:48:29.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 496 completed, throughput=4.99MB/sec 2022-04-22T17:48:29.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ot6HKA8bg4uTYGl off=7388847 len=824285 2022-04-22T17:48:29.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 497 completed, throughput=5MB/sec 2022-04-22T17:48:29.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZZYOtYi4TF2rnnP off=7877162 len=918581 2022-04-22T17:48:29.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 498 completed, throughput=5.01MB/sec 2022-04-22T17:48:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:30 smithi121 bash[36902]: cluster 2022-04-22T17:48:28.291976+0000 mgr.y (mgr.34107) 819 : cluster [DBG] pgmap v1453: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:30 smithi121 bash[36902]: audit 2022-04-22T17:48:29.835205+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:30 smithi121 bash[36902]: audit 2022-04-22T17:48:30.123807+0000 mon.a (mon.0) 2212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[45495]: cluster 2022-04-22T17:48:28.291976+0000 mgr.y (mgr.34107) 819 : cluster [DBG] pgmap v1453: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[45495]: audit 2022-04-22T17:48:29.835205+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[45495]: audit 2022-04-22T17:48:30.123807+0000 mon.a (mon.0) 2212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[46934]: cluster 2022-04-22T17:48:28.291976+0000 mgr.y (mgr.34107) 819 : cluster [DBG] pgmap v1453: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:48:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[46934]: audit 2022-04-22T17:48:29.835205+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:30 smithi115 bash[46934]: audit 2022-04-22T17:48:30.123807+0000 mon.a (mon.0) 2212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:30.831 INFO:tasks.workunit.client.0.smithi115.stdout: 104: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:30.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pgvLjSXVM7SRQ1Y off=10940431 len=1063971 2022-04-22T17:48:30.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 499 completed, throughput=4.97MB/sec 2022-04-22T17:48:30.839 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bpMhpukMQckJQIU off=3386018 len=287788 2022-04-22T17:48:30.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 500 completed, throughput=4.97MB/sec 2022-04-22T17:48:30.844 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J9beZZQf_gIAwc2 off=5006286 len=1821961 2022-04-22T17:48:30.848 INFO:tasks.workunit.client.0.smithi115.stdout:op 501 completed, throughput=4.99MB/sec 2022-04-22T17:48:30.848 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sbwRkNG4xT3uT2w off=11533890 len=628350 2022-04-22T17:48:30.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 502 completed, throughput=4.99MB/sec 2022-04-22T17:48:30.850 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L7HEf13dvKZ5fuo off=14339518 len=460692 2022-04-22T17:48:30.854 INFO:tasks.workunit.client.0.smithi115.stdout:op 503 completed, throughput=5MB/sec 2022-04-22T17:48:30.854 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sln4wc5_goK_orK off=789001 len=1482899 2022-04-22T17:48:30.857 INFO:tasks.workunit.client.0.smithi115.stdout:op 504 completed, throughput=5.01MB/sec 2022-04-22T17:48:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:31 smithi121 bash[36902]: cluster 2022-04-22T17:48:30.293242+0000 mgr.y (mgr.34107) 820 : cluster [DBG] pgmap v1454: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:31 smithi115 bash[45495]: cluster 2022-04-22T17:48:30.293242+0000 mgr.y (mgr.34107) 820 : cluster [DBG] pgmap v1454: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:31.850 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:31.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:31.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:31 smithi115 bash[46934]: cluster 2022-04-22T17:48:30.293242+0000 mgr.y (mgr.34107) 820 : cluster [DBG] pgmap v1454: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:48:31.857 INFO:tasks.workunit.client.0.smithi115.stdout: 105: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:31.857 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DXgCuFHNwCKswzD off=7467515 len=996833 2022-04-22T17:48:31.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 505 completed, throughput=4.97MB/sec 2022-04-22T17:48:31.860 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8LwXqlQn3S-OJr4 off=12692291 len=188265 2022-04-22T17:48:31.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 506 completed, throughput=4.97MB/sec 2022-04-22T17:48:31.862 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WV1he_UU9Y-4n8q off=8197842 len=710969 2022-04-22T17:48:31.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 507 completed, throughput=4.98MB/sec 2022-04-22T17:48:31.869 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OCk_laY76Wazfax off=3610964 len=967990 2022-04-22T17:48:31.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 508 completed, throughput=4.99MB/sec 2022-04-22T17:48:31.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XM1c56wuPEekyM6 off=20687339 len=1805174 2022-04-22T17:48:31.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 509 completed, throughput=5MB/sec 2022-04-22T17:48:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:32 smithi121 bash[36902]: audit 2022-04-22T17:48:32.215349+0000 mon.a (mon.0) 2213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:32.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:32 smithi115 bash[45495]: audit 2022-04-22T17:48:32.215349+0000 mon.a (mon.0) 2213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:32.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:32 smithi115 bash[46934]: audit 2022-04-22T17:48:32.215349+0000 mon.a (mon.0) 2213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:32.886 INFO:tasks.workunit.client.0.smithi115.stdout: 106: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:32.886 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2h6CGXpSDB5MzSi off=228982 len=248214 2022-04-22T17:48:32.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 510 completed, throughput=4.96MB/sec 2022-04-22T17:48:32.891 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iJV57TmKzCLiBpy off=11141408 len=1101751 2022-04-22T17:48:32.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 511 completed, throughput=4.97MB/sec 2022-04-22T17:48:32.896 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HccvK-yImOODJ_x off=335950 len=976050 2022-04-22T17:48:32.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 512 completed, throughput=4.98MB/sec 2022-04-22T17:48:32.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-baZv4TAvdFHg0a2 off=10289585 len=941407 2022-04-22T17:48:32.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 513 completed, throughput=4.99MB/sec 2022-04-22T17:48:32.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-87_fL0Mn7tnhhGW off=12550604 len=1458045 2022-04-22T17:48:32.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 514 completed, throughput=5MB/sec 2022-04-22T17:48:32.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-747yudOZCOnmPko off=5149390 len=627069 2022-04-22T17:48:32.915 INFO:tasks.workunit.client.0.smithi115.stdout:op 515 completed, throughput=5MB/sec 2022-04-22T17:48:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:33 smithi121 bash[36902]: cluster 2022-04-22T17:48:32.294373+0000 mgr.y (mgr.34107) 821 : cluster [DBG] pgmap v1455: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:33 smithi115 bash[45495]: cluster 2022-04-22T17:48:32.294373+0000 mgr.y (mgr.34107) 821 : cluster [DBG] pgmap v1455: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:33.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:33.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:33 smithi115 bash[46934]: cluster 2022-04-22T17:48:32.294373+0000 mgr.y (mgr.34107) 821 : cluster [DBG] pgmap v1455: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:33.915 INFO:tasks.workunit.client.0.smithi115.stdout: 107: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:33.916 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x0wuNqV8yesD88F off=12887448 len=81465 2022-04-22T17:48:33.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 516 completed, throughput=4.96MB/sec 2022-04-22T17:48:33.917 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CWSlyVBvW7vM_cY off=5648985 len=378665 2022-04-22T17:48:33.918 INFO:tasks.workunit.client.0.smithi115.stdout:op 517 completed, throughput=4.96MB/sec 2022-04-22T17:48:33.918 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SAr2ubq7EKZQY9X off=8369312 len=112249 2022-04-22T17:48:33.919 INFO:tasks.workunit.client.0.smithi115.stdout:op 518 completed, throughput=4.96MB/sec 2022-04-22T17:48:33.919 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7wLMDJRmZB-jrRu off=4446354 len=899613 2022-04-22T17:48:33.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 519 completed, throughput=4.97MB/sec 2022-04-22T17:48:33.926 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IDawrWA93AvMJY1 off=1964246 len=410207 2022-04-22T17:48:33.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 520 completed, throughput=4.97MB/sec 2022-04-22T17:48:33.928 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i4R2UtLCw33nTRO off=13651869 len=1584939 2022-04-22T17:48:33.936 INFO:tasks.workunit.client.0.smithi115.stdout:op 521 completed, throughput=4.99MB/sec 2022-04-22T17:48:33.936 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M8gz1R9D_nDMEYj off=5542076 len=906257 2022-04-22T17:48:33.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 522 completed, throughput=4.99MB/sec 2022-04-22T17:48:33.941 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_pqk-rWFjlCw7FJ off=3524188 len=396236 2022-04-22T17:48:33.945 INFO:tasks.workunit.client.0.smithi115.stdout:op 523 completed, throughput=5MB/sec 2022-04-22T17:48:33.945 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IRae_k9rEKeNp0X off=251305 len=1790855 2022-04-22T17:48:33.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 524 completed, throughput=5.01MB/sec 2022-04-22T17:48:34.954 INFO:tasks.workunit.client.0.smithi115.stdout: 108: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:34.955 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kOGO2iSWo3uLGY5 off=4208875 len=235539 2022-04-22T17:48:34.956 INFO:tasks.workunit.client.0.smithi115.stdout:op 525 completed, throughput=4.97MB/sec 2022-04-22T17:48:34.956 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-324RNyWA6i0lfwB off=2318848 len=1044097 2022-04-22T17:48:34.959 INFO:tasks.workunit.client.0.smithi115.stdout:op 526 completed, throughput=4.98MB/sec 2022-04-22T17:48:34.959 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oX4HmwKeKS0hNCl off=1612982 len=1564261 2022-04-22T17:48:34.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 527 completed, throughput=4.99MB/sec 2022-04-22T17:48:34.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gtMZ00KhP6z2_RP off=3053770 len=228904 2022-04-22T17:48:34.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 528 completed, throughput=4.99MB/sec 2022-04-22T17:48:34.968 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3852XkMWz1j-9a5 off=7947622 len=627250 2022-04-22T17:48:34.970 INFO:tasks.workunit.client.0.smithi115.stdout:op 529 completed, throughput=5MB/sec 2022-04-22T17:48:34.970 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2ZUOXvfOVvHGSlW off=3083170 len=1445361 2022-04-22T17:48:34.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 530 completed, throughput=5.01MB/sec 2022-04-22T17:48:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:35 smithi121 bash[36902]: cluster 2022-04-22T17:48:34.295393+0000 mgr.y (mgr.34107) 822 : cluster [DBG] pgmap v1456: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:48:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:35 smithi121 bash[36902]: audit 2022-04-22T17:48:34.838194+0000 mon.a (mon.0) 2214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:35 smithi121 bash[36902]: audit 2022-04-22T17:48:35.127434+0000 mon.a (mon.0) 2215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[45495]: cluster 2022-04-22T17:48:34.295393+0000 mgr.y (mgr.34107) 822 : cluster [DBG] pgmap v1456: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:48:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[45495]: audit 2022-04-22T17:48:34.838194+0000 mon.a (mon.0) 2214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[45495]: audit 2022-04-22T17:48:35.127434+0000 mon.a (mon.0) 2215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[46934]: cluster 2022-04-22T17:48:34.295393+0000 mgr.y (mgr.34107) 822 : cluster [DBG] pgmap v1456: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:48:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[46934]: audit 2022-04-22T17:48:34.838194+0000 mon.a (mon.0) 2214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:35 smithi115 bash[46934]: audit 2022-04-22T17:48:35.127434+0000 mon.a (mon.0) 2215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:35.976 INFO:tasks.workunit.client.0.smithi115.stdout: 109: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:35.976 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BNvg1aBiFBF2XCX off=3025340 len=397989 2022-04-22T17:48:35.978 INFO:tasks.workunit.client.0.smithi115.stdout:op 531 completed, throughput=4.97MB/sec 2022-04-22T17:48:35.978 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f0lckTRu25ayMoG off=11192461 len=1343448 2022-04-22T17:48:35.981 INFO:tasks.workunit.client.0.smithi115.stdout:op 532 completed, throughput=4.98MB/sec 2022-04-22T17:48:35.981 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L5cv2EPW77efcoD off=1614235 len=1161479 2022-04-22T17:48:35.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 533 completed, throughput=4.99MB/sec 2022-04-22T17:48:35.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yriph92pu6D4t4x off=1875469 len=440548 2022-04-22T17:48:35.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 534 completed, throughput=4.99MB/sec 2022-04-22T17:48:35.991 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Cqelm9mj9fbJ0_l off=5759124 len=626193 2022-04-22T17:48:35.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 535 completed, throughput=5MB/sec 2022-04-22T17:48:35.993 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aaop2pm4BWZ-kGq off=12866602 len=842886 2022-04-22T17:48:36.005 INFO:tasks.workunit.client.0.smithi115.stdout:op 536 completed, throughput=5.01MB/sec 2022-04-22T17:48:37.006 INFO:tasks.workunit.client.0.smithi115.stdout: 110: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:37.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f9FI8dWFWrpZxq4 off=21507062 len=236718 2022-04-22T17:48:37.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 537 completed, throughput=4.96MB/sec 2022-04-22T17:48:37.037 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-amJmU7FXDgwN9Mg off=226672 len=357528 2022-04-22T17:48:37.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 538 completed, throughput=4.97MB/sec 2022-04-22T17:48:37.037 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cE_ayvrKz9ou8lW off=9354633 len=1808613 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:op 539 completed, throughput=4.98MB/sec 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qOcHpvt8yPPH9sF off=5732799 len=1231875 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:op 540 completed, throughput=4.99MB/sec 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VEcAk9YWTJy-FhX off=559938 len=458869 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:op 541 completed, throughput=4.99MB/sec 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0gXVZEO740TeTKP off=4066634 len=1927012 2022-04-22T17:48:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:op 542 completed, throughput=5.01MB/sec 2022-04-22T17:48:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[45495]: cluster 2022-04-22T17:48:36.296528+0000 mgr.y (mgr.34107) 823 : cluster [DBG] pgmap v1457: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[45495]: audit 2022-04-22T17:48:37.218452+0000 mon.a (mon.0) 2216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[45495]: audit 2022-04-22T17:48:37.248268+0000 mon.a (mon.0) 2217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[45495]: audit 2022-04-22T17:48:37.251141+0000 mon.a (mon.0) 2218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[45495]: audit 2022-04-22T17:48:37.252795+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[46934]: cluster 2022-04-22T17:48:36.296528+0000 mgr.y (mgr.34107) 823 : cluster [DBG] pgmap v1457: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[46934]: audit 2022-04-22T17:48:37.218452+0000 mon.a (mon.0) 2216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[46934]: audit 2022-04-22T17:48:37.248268+0000 mon.a (mon.0) 2217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[46934]: audit 2022-04-22T17:48:37.251141+0000 mon.a (mon.0) 2218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:48:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:37 smithi115 bash[46934]: audit 2022-04-22T17:48:37.252795+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:48:37.595 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:37 smithi121 bash[36902]: cluster 2022-04-22T17:48:36.296528+0000 mgr.y (mgr.34107) 823 : cluster [DBG] pgmap v1457: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:37.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:37 smithi121 bash[36902]: audit 2022-04-22T17:48:37.218452+0000 mon.a (mon.0) 2216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:37.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:37 smithi121 bash[36902]: audit 2022-04-22T17:48:37.248268+0000 mon.a (mon.0) 2217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:48:37.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:37 smithi121 bash[36902]: audit 2022-04-22T17:48:37.251141+0000 mon.a (mon.0) 2218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:48:37.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:37 smithi121 bash[36902]: audit 2022-04-22T17:48:37.252795+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:48:38.037 INFO:tasks.workunit.client.0.smithi115.stdout: 111: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:38.037 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qFi8hXaIePyh1xw off=10297199 len=1322663 2022-04-22T17:48:38.044 INFO:tasks.workunit.client.0.smithi115.stdout:op 543 completed, throughput=4.98MB/sec 2022-04-22T17:48:38.044 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NavT7I6ybwFQw9y off=9491858 len=569358 2022-04-22T17:48:38.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 544 completed, throughput=4.98MB/sec 2022-04-22T17:48:38.050 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-opz1fUjLO08qsjd off=1984699 len=505393 2022-04-22T17:48:38.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 545 completed, throughput=4.99MB/sec 2022-04-22T17:48:38.054 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5toWEyXWLNKWcmE off=11912419 len=1045132 2022-04-22T17:48:38.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 546 completed, throughput=4.99MB/sec 2022-04-22T17:48:38.060 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YOZqDIeM-3lGBQ3 off=633820 len=7648 2022-04-22T17:48:38.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 547 completed, throughput=4.99MB/sec 2022-04-22T17:48:38.063 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1J0dlc05M5BMyAr off=6991283 len=169917 2022-04-22T17:48:38.064 INFO:tasks.workunit.client.0.smithi115.stdout:op 548 completed, throughput=5MB/sec 2022-04-22T17:48:38.064 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uH12_CVbk7PlXid off=9442514 len=944463 2022-04-22T17:48:38.066 INFO:tasks.workunit.client.0.smithi115.stdout:op 549 completed, throughput=5MB/sec 2022-04-22T17:48:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:38 smithi121 bash[36902]: audit 2022-04-22T17:48:37.609690+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:48:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:38 smithi115 bash[45495]: audit 2022-04-22T17:48:37.609690+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:48:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:38 smithi115 bash[46934]: audit 2022-04-22T17:48:37.609690+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:48:39.067 INFO:tasks.workunit.client.0.smithi115.stdout: 112: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:39.067 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GrX2jGDbNwEBfMz off=1992178 len=1463824 2022-04-22T17:48:39.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 550 completed, throughput=4.97MB/sec 2022-04-22T17:48:39.070 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9fRCXWLLkPIhArE off=13585914 len=1067082 2022-04-22T17:48:39.077 INFO:tasks.workunit.client.0.smithi115.stdout:op 551 completed, throughput=4.98MB/sec 2022-04-22T17:48:39.077 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nu3GWkMrF1uwfPl off=16558655 len=1317433 2022-04-22T17:48:39.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 552 completed, throughput=4.99MB/sec 2022-04-22T17:48:39.079 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XAPqzkS-bEmid9V off=12672081 len=2074039 2022-04-22T17:48:39.086 INFO:tasks.workunit.client.0.smithi115.stdout:op 553 completed, throughput=5.01MB/sec 2022-04-22T17:48:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:39 smithi121 bash[36902]: cluster 2022-04-22T17:48:38.297777+0000 mgr.y (mgr.34107) 824 : cluster [DBG] pgmap v1458: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:48:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:39 smithi115 bash[45495]: cluster 2022-04-22T17:48:38.297777+0000 mgr.y (mgr.34107) 824 : cluster [DBG] pgmap v1458: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:48:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:39 smithi115 bash[46934]: cluster 2022-04-22T17:48:38.297777+0000 mgr.y (mgr.34107) 824 : cluster [DBG] pgmap v1458: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:48:40.086 INFO:tasks.workunit.client.0.smithi115.stdout: 113: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:40.086 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R_5B04YdEOKwBsJ off=2753497 len=1756300 2022-04-22T17:48:40.090 INFO:tasks.workunit.client.0.smithi115.stdout:op 554 completed, throughput=4.98MB/sec 2022-04-22T17:48:40.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sfuCbEv5zVFwi2W off=2427557 len=794854 2022-04-22T17:48:40.093 INFO:tasks.workunit.client.0.smithi115.stdout:op 555 completed, throughput=4.99MB/sec 2022-04-22T17:48:40.093 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Qvd5KYIzsCW5057 off=1707934 len=2018703 2022-04-22T17:48:40.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 556 completed, throughput=5MB/sec 2022-04-22T17:48:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:40 smithi121 bash[36902]: audit 2022-04-22T17:48:39.841196+0000 mon.a (mon.0) 2221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:40 smithi121 bash[36902]: audit 2022-04-22T17:48:40.131122+0000 mon.a (mon.0) 2222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:41.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:40 smithi115 bash[45495]: audit 2022-04-22T17:48:39.841196+0000 mon.a (mon.0) 2221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:41.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:40 smithi115 bash[45495]: audit 2022-04-22T17:48:40.131122+0000 mon.a (mon.0) 2222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:40 smithi115 bash[46934]: audit 2022-04-22T17:48:39.841196+0000 mon.a (mon.0) 2221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:40 smithi115 bash[46934]: audit 2022-04-22T17:48:40.131122+0000 mon.a (mon.0) 2222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:41.097 INFO:tasks.workunit.client.0.smithi115.stdout: 114: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:41.097 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DKsWmDZMf7q02U2 off=4405172 len=33536 2022-04-22T17:48:41.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 557 completed, throughput=4.96MB/sec 2022-04-22T17:48:41.101 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-61b7BQBH61J1M7d off=3293395 len=264084 2022-04-22T17:48:41.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 558 completed, throughput=4.96MB/sec 2022-04-22T17:48:41.103 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P7Bq5vVSZddu2vZ off=3193460 len=1417085 2022-04-22T17:48:41.106 INFO:tasks.workunit.client.0.smithi115.stdout:op 559 completed, throughput=4.97MB/sec 2022-04-22T17:48:41.106 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lKw2aVzZOYW3udH off=11958214 len=2086557 2022-04-22T17:48:41.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 560 completed, throughput=4.99MB/sec 2022-04-22T17:48:41.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zm4981Au-xk3tMC off=9328099 len=466120 2022-04-22T17:48:41.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 561 completed, throughput=4.99MB/sec 2022-04-22T17:48:41.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-znIgJcA1ESir9U3 off=8397440 len=1875070 2022-04-22T17:48:41.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 562 completed, throughput=5.01MB/sec 2022-04-22T17:48:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:41 smithi121 bash[36902]: cluster 2022-04-22T17:48:40.298832+0000 mgr.y (mgr.34107) 825 : cluster [DBG] pgmap v1459: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:41 smithi115 bash[45495]: cluster 2022-04-22T17:48:40.298832+0000 mgr.y (mgr.34107) 825 : cluster [DBG] pgmap v1459: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:42.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:41.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:41 smithi115 bash[46934]: cluster 2022-04-22T17:48:40.298832+0000 mgr.y (mgr.34107) 825 : cluster [DBG] pgmap v1459: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:42.118 INFO:tasks.workunit.client.0.smithi115.stdout: 115: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:42.184 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qS_nFVYD_ANQ1Hd off=7622047 len=1679607 2022-04-22T17:48:42.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 563 completed, throughput=4.98MB/sec 2022-04-22T17:48:42.185 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6rPYHUiVU8yKKM0 off=466306 len=1534557 2022-04-22T17:48:42.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 564 completed, throughput=4.99MB/sec 2022-04-22T17:48:42.185 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0f0cwoV3L1oewAw off=5211 len=2080956 2022-04-22T17:48:42.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 565 completed, throughput=5.01MB/sec 2022-04-22T17:48:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:42 smithi121 bash[36902]: audit 2022-04-22T17:48:42.221477+0000 mon.a (mon.0) 2223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:42 smithi115 bash[45495]: audit 2022-04-22T17:48:42.221477+0000 mon.a (mon.0) 2223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:42 smithi115 bash[46934]: audit 2022-04-22T17:48:42.221477+0000 mon.a (mon.0) 2223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:43.143 INFO:tasks.workunit.client.0.smithi115.stdout: 116: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:43.143 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-33sWqbpluG5W46t off=389503 len=1213885 2022-04-22T17:48:43.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 566 completed, throughput=4.98MB/sec 2022-04-22T17:48:43.151 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lpp40ID9gEceRiS off=4607762 len=992594 2022-04-22T17:48:43.157 INFO:tasks.workunit.client.0.smithi115.stdout:op 567 completed, throughput=4.98MB/sec 2022-04-22T17:48:43.157 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TsmKtVAYuEZrvG7 off=1911294 len=1088876 2022-04-22T17:48:43.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 568 completed, throughput=4.99MB/sec 2022-04-22T17:48:43.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J8a2MjiEy0BiAzL off=6490107 len=613360 2022-04-22T17:48:43.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 569 completed, throughput=5MB/sec 2022-04-22T17:48:43.162 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ouViYNMeMi56Jhg off=11173761 len=1854431 2022-04-22T17:48:43.164 INFO:tasks.workunit.client.0.smithi115.stdout:op 570 completed, throughput=5.01MB/sec 2022-04-22T17:48:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:43 smithi121 bash[36902]: cluster 2022-04-22T17:48:42.299949+0000 mgr.y (mgr.34107) 826 : cluster [DBG] pgmap v1460: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:43 smithi115 bash[45495]: cluster 2022-04-22T17:48:42.299949+0000 mgr.y (mgr.34107) 826 : cluster [DBG] pgmap v1460: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:44.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:43.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:43 smithi115 bash[46934]: cluster 2022-04-22T17:48:42.299949+0000 mgr.y (mgr.34107) 826 : cluster [DBG] pgmap v1460: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:44.164 INFO:tasks.workunit.client.0.smithi115.stdout: 117: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:44.165 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0QUxvMouWLhhDUa off=2265750 len=698342 2022-04-22T17:48:44.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 571 completed, throughput=4.97MB/sec 2022-04-22T17:48:44.171 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w_aTGMtSyNPkDKh off=9063760 len=1534129 2022-04-22T17:48:44.180 INFO:tasks.workunit.client.0.smithi115.stdout:op 572 completed, throughput=4.99MB/sec 2022-04-22T17:48:44.180 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WyR3yF5rc5FgAsB off=6787556 len=1031940 2022-04-22T17:48:44.186 INFO:tasks.workunit.client.0.smithi115.stdout:op 573 completed, throughput=4.99MB/sec 2022-04-22T17:48:44.186 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3xVgjDFpCjVk72q off=11683945 len=437112 2022-04-22T17:48:44.191 INFO:tasks.workunit.client.0.smithi115.stdout:op 574 completed, throughput=5MB/sec 2022-04-22T17:48:44.191 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9tkSsFi4sX557Xf off=3325222 len=1638598 2022-04-22T17:48:44.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 575 completed, throughput=5.01MB/sec 2022-04-22T17:48:45.194 INFO:tasks.workunit.client.0.smithi115.stdout: 118: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:45.194 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yDIM7aq3MxunO8I off=18481667 len=355765 2022-04-22T17:48:45.199 INFO:tasks.workunit.client.0.smithi115.stdout:op 576 completed, throughput=4.97MB/sec 2022-04-22T17:48:45.199 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DfTxScvZMNRdd84 off=1109431 len=326465 2022-04-22T17:48:45.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 577 completed, throughput=4.97MB/sec 2022-04-22T17:48:45.203 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-goBdV4lmsf0k2_H off=8920749 len=921998 2022-04-22T17:48:45.206 INFO:tasks.workunit.client.0.smithi115.stdout:op 578 completed, throughput=4.98MB/sec 2022-04-22T17:48:45.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2yMvHCiiFbarEae off=1847876 len=1012794 2022-04-22T17:48:45.213 INFO:tasks.workunit.client.0.smithi115.stdout:op 579 completed, throughput=4.99MB/sec 2022-04-22T17:48:45.213 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NNOqUwp6aMCqjip off=5038068 len=417479 2022-04-22T17:48:45.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 580 completed, throughput=4.99MB/sec 2022-04-22T17:48:45.214 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kCNX_edAq6NqDSM off=5270134 len=901454 2022-04-22T17:48:45.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 581 completed, throughput=5MB/sec 2022-04-22T17:48:45.216 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ydb7O1-4fUrCitO off=9844760 len=1692239 2022-04-22T17:48:45.222 INFO:tasks.workunit.client.0.smithi115.stdout:op 582 completed, throughput=5.01MB/sec 2022-04-22T17:48:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:45 smithi121 bash[36902]: cluster 2022-04-22T17:48:44.301021+0000 mgr.y (mgr.34107) 827 : cluster [DBG] pgmap v1461: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:45 smithi121 bash[36902]: audit 2022-04-22T17:48:44.844494+0000 mon.a (mon.0) 2224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:45 smithi121 bash[36902]: audit 2022-04-22T17:48:45.134989+0000 mon.a (mon.0) 2225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[45495]: cluster 2022-04-22T17:48:44.301021+0000 mgr.y (mgr.34107) 827 : cluster [DBG] pgmap v1461: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[45495]: audit 2022-04-22T17:48:44.844494+0000 mon.a (mon.0) 2224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[45495]: audit 2022-04-22T17:48:45.134989+0000 mon.a (mon.0) 2225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[46934]: cluster 2022-04-22T17:48:44.301021+0000 mgr.y (mgr.34107) 827 : cluster [DBG] pgmap v1461: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:48:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[46934]: audit 2022-04-22T17:48:44.844494+0000 mon.a (mon.0) 2224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:45 smithi115 bash[46934]: audit 2022-04-22T17:48:45.134989+0000 mon.a (mon.0) 2225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:46.223 INFO:tasks.workunit.client.0.smithi115.stdout: 119: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:46.223 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gDmOXfclhs4zUBc off=713754 len=569022 2022-04-22T17:48:46.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 583 completed, throughput=4.98MB/sec 2022-04-22T17:48:46.228 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CHeGXbY0B3cKd_H off=12453568 len=76015 2022-04-22T17:48:46.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 584 completed, throughput=4.98MB/sec 2022-04-22T17:48:46.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HopMno6aYyVijlF off=17878994 len=828520 2022-04-22T17:48:46.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 585 completed, throughput=4.98MB/sec 2022-04-22T17:48:46.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GfcQGr5PgTMUtaS off=4150610 len=1813661 2022-04-22T17:48:46.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 586 completed, throughput=5MB/sec 2022-04-22T17:48:46.246 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IzBJ4aHXIr7-fkZ off=943028 len=1295730 2022-04-22T17:48:46.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 587 completed, throughput=5.01MB/sec 2022-04-22T17:48:47.250 INFO:tasks.workunit.client.0.smithi115.stdout: 120: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:47.251 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eTcyslDYv5c_uNC off=11902745 len=501131 2022-04-22T17:48:47.252 INFO:tasks.workunit.client.0.smithi115.stdout:op 588 completed, throughput=4.97MB/sec 2022-04-22T17:48:47.306 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5Ev7sSlHu3QM5eq off=6063569 len=681535 2022-04-22T17:48:47.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 589 completed, throughput=4.97MB/sec 2022-04-22T17:48:47.307 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6IjTA-LEP1feucg off=6262637 len=1728620 2022-04-22T17:48:47.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 590 completed, throughput=4.99MB/sec 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b30dTATPC-lcFAO off=2280702 len=933664 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 591 completed, throughput=4.99MB/sec 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WvjgwJOPmzf3Jnr off=1844659 len=120102 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 592 completed, throughput=5MB/sec 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F7kUn7EbYVEK9TS off=362373 len=1307706 2022-04-22T17:48:47.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 593 completed, throughput=5.01MB/sec 2022-04-22T17:48:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:47 smithi121 bash[36902]: cluster 2022-04-22T17:48:46.302148+0000 mgr.y (mgr.34107) 828 : cluster [DBG] pgmap v1462: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:47 smithi121 bash[36902]: audit 2022-04-22T17:48:47.224597+0000 mon.a (mon.0) 2226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:47 smithi115 bash[45495]: cluster 2022-04-22T17:48:46.302148+0000 mgr.y (mgr.34107) 828 : cluster [DBG] pgmap v1462: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:47 smithi115 bash[45495]: audit 2022-04-22T17:48:47.224597+0000 mon.a (mon.0) 2226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:47 smithi115 bash[46934]: cluster 2022-04-22T17:48:46.302148+0000 mgr.y (mgr.34107) 828 : cluster [DBG] pgmap v1462: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:48:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:47 smithi115 bash[46934]: audit 2022-04-22T17:48:47.224597+0000 mon.a (mon.0) 2226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:48.273 INFO:tasks.workunit.client.0.smithi115.stdout: 121: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:48.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--378x6yp2SROx-z off=15129058 len=809713 2022-04-22T17:48:48.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 594 completed, throughput=4.97MB/sec 2022-04-22T17:48:48.279 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wRb6sKcPx50bAsq off=17274965 len=883058 2022-04-22T17:48:48.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 595 completed, throughput=4.98MB/sec 2022-04-22T17:48:48.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fweg51lmxt_64re off=6126445 len=490450 2022-04-22T17:48:48.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 596 completed, throughput=4.98MB/sec 2022-04-22T17:48:48.283 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_KZvhIvEsqPW64v off=322010 len=285037 2022-04-22T17:48:48.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 597 completed, throughput=4.98MB/sec 2022-04-22T17:48:48.284 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KFsFkW07vgtFp3N off=880300 len=1060400 2022-04-22T17:48:48.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 598 completed, throughput=4.99MB/sec 2022-04-22T17:48:48.292 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ARpNZfCmnHa-8YK off=2307425 len=606478 2022-04-22T17:48:48.294 INFO:tasks.workunit.client.0.smithi115.stdout:op 599 completed, throughput=5MB/sec 2022-04-22T17:48:48.294 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-I8uWvGIbY1EWb5B off=7460037 len=1291039 2022-04-22T17:48:48.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 600 completed, throughput=5.01MB/sec 2022-04-22T17:48:49.300 INFO:tasks.workunit.client.0.smithi115.stdout: 122: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:49.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LZZezJdalF_b3Kl off=11861102 len=1497964 2022-04-22T17:48:49.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 601 completed, throughput=4.98MB/sec 2022-04-22T17:48:49.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f4a2RucVlHN542N off=1993568 len=578518 2022-04-22T17:48:49.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 602 completed, throughput=4.98MB/sec 2022-04-22T17:48:49.314 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zd9BpZ_qK4xL7I0 off=245286 len=812985 2022-04-22T17:48:49.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 603 completed, throughput=4.99MB/sec 2022-04-22T17:48:49.319 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kHBW6egrViRjnq_ off=6271367 len=668050 2022-04-22T17:48:49.322 INFO:tasks.workunit.client.0.smithi115.stdout:op 604 completed, throughput=4.99MB/sec 2022-04-22T17:48:49.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XZIcDJ5-gkkkfNB off=970187 len=134442 2022-04-22T17:48:49.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 605 completed, throughput=4.99MB/sec 2022-04-22T17:48:49.326 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6haRao4Fe7Z0dUC off=5572094 len=1658799 2022-04-22T17:48:49.332 INFO:tasks.workunit.client.0.smithi115.stdout:op 606 completed, throughput=5.01MB/sec 2022-04-22T17:48:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:49 smithi121 bash[36902]: cluster 2022-04-22T17:48:48.303161+0000 mgr.y (mgr.34107) 829 : cluster [DBG] pgmap v1463: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:49 smithi115 bash[45495]: cluster 2022-04-22T17:48:48.303161+0000 mgr.y (mgr.34107) 829 : cluster [DBG] pgmap v1463: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:49 smithi115 bash[46934]: cluster 2022-04-22T17:48:48.303161+0000 mgr.y (mgr.34107) 829 : cluster [DBG] pgmap v1463: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:50.333 INFO:tasks.workunit.client.0.smithi115.stdout: 123: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:50.333 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6cvCYluFcfLw9VT off=9930637 len=582447 2022-04-22T17:48:50.339 INFO:tasks.workunit.client.0.smithi115.stdout:op 607 completed, throughput=4.97MB/sec 2022-04-22T17:48:50.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0qiDikgRlMv-dNJ off=6975515 len=1679601 2022-04-22T17:48:50.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 608 completed, throughput=4.98MB/sec 2022-04-22T17:48:50.348 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b-qUleVJnK-MMFc off=123401 len=37221 2022-04-22T17:48:50.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 609 completed, throughput=4.98MB/sec 2022-04-22T17:48:50.351 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OLqGe6DM4mtXEGm off=9040380 len=209201 2022-04-22T17:48:50.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 610 completed, throughput=4.98MB/sec 2022-04-22T17:48:50.352 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oEtYU3F4hkl154V off=1269483 len=322487 2022-04-22T17:48:50.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 611 completed, throughput=4.99MB/sec 2022-04-22T17:48:50.354 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-apbojOs2Mdjb_Ex off=8840913 len=1974220 2022-04-22T17:48:50.363 INFO:tasks.workunit.client.0.smithi115.stdout:op 612 completed, throughput=5MB/sec 2022-04-22T17:48:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:50 smithi121 bash[36902]: audit 2022-04-22T17:48:49.847461+0000 mon.a (mon.0) 2227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:50 smithi121 bash[36902]: audit 2022-04-22T17:48:50.138631+0000 mon.a (mon.0) 2228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:50 smithi115 bash[45495]: audit 2022-04-22T17:48:49.847461+0000 mon.a (mon.0) 2227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:50 smithi115 bash[45495]: audit 2022-04-22T17:48:50.138631+0000 mon.a (mon.0) 2228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:50 smithi115 bash[46934]: audit 2022-04-22T17:48:49.847461+0000 mon.a (mon.0) 2227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:50 smithi115 bash[46934]: audit 2022-04-22T17:48:50.138631+0000 mon.a (mon.0) 2228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:51.363 INFO:tasks.workunit.client.0.smithi115.stdout: 124: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:51.364 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--UAWluf41jFn1XE off=829468 len=715152 2022-04-22T17:48:51.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 613 completed, throughput=4.97MB/sec 2022-04-22T17:48:51.366 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vuNmcXNtGkWbQOm off=5545180 len=1465961 2022-04-22T17:48:51.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 614 completed, throughput=4.98MB/sec 2022-04-22T17:48:51.371 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IUBE1YisSKXmrs3 off=935054 len=1936324 2022-04-22T17:48:51.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 615 completed, throughput=4.99MB/sec 2022-04-22T17:48:51.376 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jNC6mGYF1r19bfM off=8594725 len=1797692 2022-04-22T17:48:51.380 INFO:tasks.workunit.client.0.smithi115.stdout:op 616 completed, throughput=5MB/sec 2022-04-22T17:48:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:51 smithi121 bash[36902]: cluster 2022-04-22T17:48:50.304177+0000 mgr.y (mgr.34107) 830 : cluster [DBG] pgmap v1464: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:48:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:51 smithi115 bash[45495]: cluster 2022-04-22T17:48:50.304177+0000 mgr.y (mgr.34107) 830 : cluster [DBG] pgmap v1464: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:48:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:51.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:51 smithi115 bash[46934]: cluster 2022-04-22T17:48:50.304177+0000 mgr.y (mgr.34107) 830 : cluster [DBG] pgmap v1464: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:48:52.380 INFO:tasks.workunit.client.0.smithi115.stdout: 125: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:52.380 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u52UCv-4KQceM1e off=4550554 len=1427414 2022-04-22T17:48:52.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 617 completed, throughput=4.98MB/sec 2022-04-22T17:48:52.390 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fn7ZQKYtSN6rFqH off=8956158 len=1117662 2022-04-22T17:48:52.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 618 completed, throughput=4.98MB/sec 2022-04-22T17:48:52.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E1ilTPExHTs6fLL off=1964800 len=1494515 2022-04-22T17:48:52.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 619 completed, throughput=5MB/sec 2022-04-22T17:48:52.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-INTAYZtD4D0q9tJ off=7615175 len=1469083 2022-04-22T17:48:52.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 620 completed, throughput=5.01MB/sec 2022-04-22T17:48:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:52 smithi121 bash[36902]: audit 2022-04-22T17:48:52.227789+0000 mon.a (mon.0) 2229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:53.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:52 smithi115 bash[45495]: audit 2022-04-22T17:48:52.227789+0000 mon.a (mon.0) 2229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:52 smithi115 bash[46934]: audit 2022-04-22T17:48:52.227789+0000 mon.a (mon.0) 2229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:53.399 INFO:tasks.workunit.client.0.smithi115.stdout: 126: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:53.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zvsmThru_5yJHrn off=9220267 len=1362809 2022-04-22T17:48:53.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 621 completed, throughput=4.98MB/sec 2022-04-22T17:48:53.403 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rMAcKdpCO79W_3D off=194475 len=2094184 2022-04-22T17:48:53.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 622 completed, throughput=4.99MB/sec 2022-04-22T17:48:53.415 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jks8yf73aH9El5x off=6738888 len=14439 2022-04-22T17:48:53.415 INFO:tasks.workunit.client.0.smithi115.stdout:op 623 completed, throughput=4.99MB/sec 2022-04-22T17:48:53.415 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-T13KczuhdsifnGI off=2447910 len=1437362 2022-04-22T17:48:53.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 624 completed, throughput=5MB/sec 2022-04-22T17:48:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:53 smithi121 bash[36902]: cluster 2022-04-22T17:48:52.305240+0000 mgr.y (mgr.34107) 831 : cluster [DBG] pgmap v1465: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:54.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:53 smithi115 bash[45495]: cluster 2022-04-22T17:48:52.305240+0000 mgr.y (mgr.34107) 831 : cluster [DBG] pgmap v1465: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:54.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:48:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:48:53.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:48:54.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:53 smithi115 bash[46934]: cluster 2022-04-22T17:48:52.305240+0000 mgr.y (mgr.34107) 831 : cluster [DBG] pgmap v1465: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:48:54.422 INFO:tasks.workunit.client.0.smithi115.stdout: 127: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:54.422 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1xdCqO3uWR9HLJa off=2895133 len=1612945 2022-04-22T17:48:54.425 INFO:tasks.workunit.client.0.smithi115.stdout:op 625 completed, throughput=4.98MB/sec 2022-04-22T17:48:54.425 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cXOUFdcYJDc4Jnp off=465725 len=893193 2022-04-22T17:48:54.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 626 completed, throughput=4.98MB/sec 2022-04-22T17:48:54.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ovZeuTyxoGX6hFA off=733443 len=839709 2022-04-22T17:48:54.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 627 completed, throughput=4.99MB/sec 2022-04-22T17:48:54.439 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iju7LVK0tegqFiN off=1443786 len=223022 2022-04-22T17:48:54.442 INFO:tasks.workunit.client.0.smithi115.stdout:op 628 completed, throughput=4.99MB/sec 2022-04-22T17:48:54.442 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DezqeCqhq4f0Fqp off=670810 len=892831 2022-04-22T17:48:54.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 629 completed, throughput=5MB/sec 2022-04-22T17:48:54.445 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_zrj55HR67OF2sb off=1200224 len=1702273 2022-04-22T17:48:54.448 INFO:tasks.workunit.client.0.smithi115.stdout:op 630 completed, throughput=5.01MB/sec 2022-04-22T17:48:55.448 INFO:tasks.workunit.client.0.smithi115.stdout: 128: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:55.487 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c6inKKcd5PFtSXu off=13156687 len=1135206 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 631 completed, throughput=4.98MB/sec 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-du5iBjUCIYc7uly off=2037275 len=1486032 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 632 completed, throughput=4.99MB/sec 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3NZyMmWuXozwmp5 off=4280231 len=787789 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 633 completed, throughput=4.99MB/sec 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aokFpmZa2Cq2Zle off=11529619 len=696858 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 634 completed, throughput=5MB/sec 2022-04-22T17:48:55.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-88hpdVeHGvGNUQE off=13805979 len=1176400 2022-04-22T17:48:55.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 635 completed, throughput=5.01MB/sec 2022-04-22T17:48:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:55 smithi121 bash[36902]: cluster 2022-04-22T17:48:54.306323+0000 mgr.y (mgr.34107) 832 : cluster [DBG] pgmap v1466: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:55 smithi121 bash[36902]: audit 2022-04-22T17:48:54.850558+0000 mon.a (mon.0) 2230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:55 smithi121 bash[36902]: audit 2022-04-22T17:48:55.142281+0000 mon.a (mon.0) 2231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[46934]: cluster 2022-04-22T17:48:54.306323+0000 mgr.y (mgr.34107) 832 : cluster [DBG] pgmap v1466: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[46934]: audit 2022-04-22T17:48:54.850558+0000 mon.a (mon.0) 2230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[46934]: audit 2022-04-22T17:48:55.142281+0000 mon.a (mon.0) 2231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[45495]: cluster 2022-04-22T17:48:54.306323+0000 mgr.y (mgr.34107) 832 : cluster [DBG] pgmap v1466: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:48:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[45495]: audit 2022-04-22T17:48:54.850558+0000 mon.a (mon.0) 2230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:48:56.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:55 smithi115 bash[45495]: audit 2022-04-22T17:48:55.142281+0000 mon.a (mon.0) 2231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:48:56.245 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:48:56.473 INFO:tasks.workunit.client.0.smithi115.stdout: 129: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:56.473 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PEorqRxqnajRzyE off=3442472 len=9296 2022-04-22T17:48:56.474 INFO:tasks.workunit.client.0.smithi115.stdout:op 636 completed, throughput=4.97MB/sec 2022-04-22T17:48:56.474 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EkBTn5IZAt7No0K off=14282617 len=219115 2022-04-22T17:48:56.479 INFO:tasks.workunit.client.0.smithi115.stdout:op 637 completed, throughput=4.97MB/sec 2022-04-22T17:48:56.479 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fjECVnxhhuH5TDr off=14704920 len=1863462 2022-04-22T17:48:56.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 638 completed, throughput=4.98MB/sec 2022-04-22T17:48:56.486 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-g5yUWXmfBT9yeTZ off=10541939 len=1039036 2022-04-22T17:48:56.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 639 completed, throughput=4.99MB/sec 2022-04-22T17:48:56.490 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ymIKxznq7tJLhY2 off=1000111 len=540088 2022-04-22T17:48:56.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 640 completed, throughput=5MB/sec 2022-04-22T17:48:56.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NrCk8huwszcO7T4 off=8068155 len=884887 2022-04-22T17:48:56.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 641 completed, throughput=5MB/sec 2022-04-22T17:48:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:48:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (12m) 79s ago 25m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:48:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (25m) 2m ago 25m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (13m) 79s ago 26m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (13m) 2m ago 26m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (21m) 2m ago 31m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (21m) 79s ago 33m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (20m) 79s ago 33m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (19m) 2m ago 31m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:48:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (20m) 79s ago 31m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (13m) 79s ago 26m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (13m) 2m ago 26m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (18m) 79s ago 30m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (18m) 79s ago 30m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (17m) 79s ago 29m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (17m) 79s ago 29m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (16m) 2m ago 28m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (15m) 2m ago 28m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (15m) 2m ago 27m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (14m) 2m ago 27m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:48:56.599 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (12m) 2m ago 26m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:48:57.002 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:48:57.002 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:48:57.002 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:48:57.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:57.002 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:48:57.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:48:57.004 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:48:57.004 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:48:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:56 smithi121 bash[36902]: audit 2022-04-22T17:48:56.218139+0000 mgr.y (mgr.34107) 833 : audit [DBG] from='client.45223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:57.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:56 smithi115 bash[45495]: audit 2022-04-22T17:48:56.218139+0000 mgr.y (mgr.34107) 833 : audit [DBG] from='client.45223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:57.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:56 smithi115 bash[46934]: audit 2022-04-22T17:48:56.218139+0000 mgr.y (mgr.34107) 833 : audit [DBG] from='client.45223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:57.497 INFO:tasks.workunit.client.0.smithi115.stdout: 130: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:57.497 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5Q44dKcodTgKvDX off=1166904 len=547962 2022-04-22T17:48:57.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 642 completed, throughput=4.97MB/sec 2022-04-22T17:48:57.499 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OsYyHqY1mWISmHq off=2783497 len=64352 2022-04-22T17:48:57.500 INFO:tasks.workunit.client.0.smithi115.stdout:op 643 completed, throughput=4.97MB/sec 2022-04-22T17:48:57.500 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mawT0cWGUuOao4k off=19366894 len=278529 2022-04-22T17:48:57.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 644 completed, throughput=4.97MB/sec 2022-04-22T17:48:57.501 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3W-mQ3vxoFHQowR off=4139277 len=133339 2022-04-22T17:48:57.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 645 completed, throughput=4.97MB/sec 2022-04-22T17:48:57.505 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ao2gMjQzyZ7SF71 off=289281 len=1471639 2022-04-22T17:48:57.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 646 completed, throughput=4.98MB/sec 2022-04-22T17:48:57.508 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lu3tHsiHSuirBZI off=431583 len=162245 2022-04-22T17:48:57.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 647 completed, throughput=4.98MB/sec 2022-04-22T17:48:57.509 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GX7_aUWVpAhEuQF off=7262876 len=607484 2022-04-22T17:48:57.513 INFO:tasks.workunit.client.0.smithi115.stdout:op 648 completed, throughput=4.99MB/sec 2022-04-22T17:48:57.514 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XjX1sAiZbTiVHyO off=14347528 len=1774335 2022-04-22T17:48:57.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 649 completed, throughput=5MB/sec 2022-04-22T17:48:57.517 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wB_e58gXkSk91G7 off=11500298 len=1558324 2022-04-22T17:48:57.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 650 completed, throughput=5.01MB/sec 2022-04-22T17:48:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:57 smithi121 bash[36902]: cluster 2022-04-22T17:48:56.307421+0000 mgr.y (mgr.34107) 834 : cluster [DBG] pgmap v1467: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:48:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:57 smithi121 bash[36902]: audit 2022-04-22T17:48:56.568452+0000 mgr.y (mgr.34107) 835 : audit [DBG] from='client.45229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:57 smithi121 bash[36902]: audit 2022-04-22T17:48:56.992216+0000 mon.a (mon.0) 2232 : audit [DBG] from='client.? 172.21.15.115:0/2139773858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:57 smithi121 bash[36902]: audit 2022-04-22T17:48:57.231030+0000 mon.a (mon.0) 2233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[45495]: cluster 2022-04-22T17:48:56.307421+0000 mgr.y (mgr.34107) 834 : cluster [DBG] pgmap v1467: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[45495]: audit 2022-04-22T17:48:56.568452+0000 mgr.y (mgr.34107) 835 : audit [DBG] from='client.45229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[45495]: audit 2022-04-22T17:48:56.992216+0000 mon.a (mon.0) 2232 : audit [DBG] from='client.? 172.21.15.115:0/2139773858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[45495]: audit 2022-04-22T17:48:57.231030+0000 mon.a (mon.0) 2233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[46934]: cluster 2022-04-22T17:48:56.307421+0000 mgr.y (mgr.34107) 834 : cluster [DBG] pgmap v1467: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:48:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[46934]: audit 2022-04-22T17:48:56.568452+0000 mgr.y (mgr.34107) 835 : audit [DBG] from='client.45229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:48:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[46934]: audit 2022-04-22T17:48:56.992216+0000 mon.a (mon.0) 2232 : audit [DBG] from='client.? 172.21.15.115:0/2139773858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:48:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:57 smithi115 bash[46934]: audit 2022-04-22T17:48:57.231030+0000 mon.a (mon.0) 2233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:48:58.524 INFO:tasks.workunit.client.0.smithi115.stdout: 131: throughput=4.97MB/sec pending data=0 2022-04-22T17:48:58.564 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uBtWn21kQelVYgM off=1278006 len=1848898 2022-04-22T17:48:58.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 651 completed, throughput=4.99MB/sec 2022-04-22T17:48:58.566 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-85ZaG6IRXU6s8Bv off=1370829 len=97998 2022-04-22T17:48:58.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 652 completed, throughput=4.99MB/sec 2022-04-22T17:48:58.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8eQouDslznUq3Ss off=18988434 len=2067475 2022-04-22T17:48:58.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 653 completed, throughput=5MB/sec 2022-04-22T17:48:59.547 INFO:tasks.workunit.client.0.smithi115.stdout: 132: throughput=4.96MB/sec pending data=0 2022-04-22T17:48:59.593 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NQYeN8gwgbXlUf6 off=10885921 len=76695 2022-04-22T17:48:59.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 654 completed, throughput=4.96MB/sec 2022-04-22T17:48:59.594 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3qAkhQSQXuonSl_ off=9834596 len=1104384 2022-04-22T17:48:59.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 655 completed, throughput=4.97MB/sec 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dZ4MgEKZQW7a5jF off=588156 len=1144376 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 656 completed, throughput=4.98MB/sec 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eMWugvkQKCVkHen off=18842814 len=1822270 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 657 completed, throughput=4.99MB/sec 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0LmTBCrAS046tAW off=3388492 len=1803873 2022-04-22T17:48:59.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 658 completed, throughput=5.01MB/sec 2022-04-22T17:49:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:48:59 smithi121 bash[36902]: cluster 2022-04-22T17:48:58.308750+0000 mgr.y (mgr.34107) 836 : cluster [DBG] pgmap v1468: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:00.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:48:59 smithi115 bash[45495]: cluster 2022-04-22T17:48:58.308750+0000 mgr.y (mgr.34107) 836 : cluster [DBG] pgmap v1468: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:00.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:48:59 smithi115 bash[46934]: cluster 2022-04-22T17:48:58.308750+0000 mgr.y (mgr.34107) 836 : cluster [DBG] pgmap v1468: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:00.563 INFO:tasks.workunit.client.0.smithi115.stdout: 133: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:00.563 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rDlHqgW60GfaOWd off=2019066 len=881250 2022-04-22T17:49:00.571 INFO:tasks.workunit.client.0.smithi115.stdout:op 659 completed, throughput=4.97MB/sec 2022-04-22T17:49:00.571 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bDvd1mrhUcwHFdG off=1673920 len=508134 2022-04-22T17:49:00.576 INFO:tasks.workunit.client.0.smithi115.stdout:op 660 completed, throughput=4.98MB/sec 2022-04-22T17:49:00.576 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XR3v0aMa3FVWBsC off=11074370 len=923882 2022-04-22T17:49:00.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 661 completed, throughput=4.98MB/sec 2022-04-22T17:49:00.582 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VzC0tFWGmch3rpP off=5407028 len=1183090 2022-04-22T17:49:00.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 662 completed, throughput=4.99MB/sec 2022-04-22T17:49:00.584 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GiGX5-pyh6o1hyL off=1232613 len=148146 2022-04-22T17:49:00.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 663 completed, throughput=4.99MB/sec 2022-04-22T17:49:00.585 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y5lO-Fp68gE_SfG off=618754 len=275166 2022-04-22T17:49:00.589 INFO:tasks.workunit.client.0.smithi115.stdout:op 664 completed, throughput=4.99MB/sec 2022-04-22T17:49:00.589 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zKyxtzV6Q6nl4jH off=4627561 len=1869247 2022-04-22T17:49:00.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 665 completed, throughput=5.01MB/sec 2022-04-22T17:49:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:00 smithi121 bash[36902]: audit 2022-04-22T17:48:59.853333+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:00 smithi121 bash[36902]: audit 2022-04-22T17:49:00.145941+0000 mon.a (mon.0) 2235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:01.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:00 smithi115 bash[45495]: audit 2022-04-22T17:48:59.853333+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:01.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:00 smithi115 bash[45495]: audit 2022-04-22T17:49:00.145941+0000 mon.a (mon.0) 2235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:00 smithi115 bash[46934]: audit 2022-04-22T17:48:59.853333+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:00 smithi115 bash[46934]: audit 2022-04-22T17:49:00.145941+0000 mon.a (mon.0) 2235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:01.591 INFO:tasks.workunit.client.0.smithi115.stdout: 134: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:01.591 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NfDGsUKQB3WSZPH off=2479642 len=513478 2022-04-22T17:49:01.593 INFO:tasks.workunit.client.0.smithi115.stdout:op 666 completed, throughput=4.97MB/sec 2022-04-22T17:49:01.593 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VTgFodxgiQJCfi1 off=8537032 len=618860 2022-04-22T17:49:01.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 667 completed, throughput=4.98MB/sec 2022-04-22T17:49:01.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pPTHH0YCpo_s7fL off=414098 len=41851 2022-04-22T17:49:01.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 668 completed, throughput=4.98MB/sec 2022-04-22T17:49:01.599 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YCQgRzREZ3AC4Ok off=15818944 len=1327028 2022-04-22T17:49:01.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 669 completed, throughput=4.99MB/sec 2022-04-22T17:49:01.607 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NF9cOMrWBZQ8Rpp off=2293573 len=318853 2022-04-22T17:49:01.608 INFO:tasks.workunit.client.0.smithi115.stdout:op 670 completed, throughput=4.99MB/sec 2022-04-22T17:49:01.608 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ucjvFHhCx71Vc2B off=7625975 len=1342933 2022-04-22T17:49:01.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 671 completed, throughput=5MB/sec 2022-04-22T17:49:01.611 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6CJgLAdW6UoPbsj off=3390665 len=613877 2022-04-22T17:49:01.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 672 completed, throughput=5MB/sec 2022-04-22T17:49:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:01 smithi121 bash[36902]: cluster 2022-04-22T17:49:00.309952+0000 mgr.y (mgr.34107) 837 : cluster [DBG] pgmap v1469: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:01 smithi115 bash[45495]: cluster 2022-04-22T17:49:00.309952+0000 mgr.y (mgr.34107) 837 : cluster [DBG] pgmap v1469: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:02.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:01.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:02.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:01 smithi115 bash[46934]: cluster 2022-04-22T17:49:00.309952+0000 mgr.y (mgr.34107) 837 : cluster [DBG] pgmap v1469: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:02.616 INFO:tasks.workunit.client.0.smithi115.stdout: 135: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:02.616 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AC2DXija5-O0ced off=21934287 len=1070242 2022-04-22T17:49:02.623 INFO:tasks.workunit.client.0.smithi115.stdout:op 673 completed, throughput=4.97MB/sec 2022-04-22T17:49:02.623 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IayzeckD4tceOAQ off=2818336 len=178023 2022-04-22T17:49:02.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 674 completed, throughput=4.98MB/sec 2022-04-22T17:49:02.624 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9YFfKezV31jD9DA off=3016822 len=656187 2022-04-22T17:49:02.630 INFO:tasks.workunit.client.0.smithi115.stdout:op 675 completed, throughput=4.98MB/sec 2022-04-22T17:49:02.630 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QxS3fHiPtv7CSfV off=9218792 len=1668735 2022-04-22T17:49:02.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 676 completed, throughput=4.99MB/sec 2022-04-22T17:49:02.632 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kujpBgHAmCjrD_- off=3959383 len=1106981 2022-04-22T17:49:02.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 677 completed, throughput=5MB/sec 2022-04-22T17:49:02.638 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2QQEijz4RtrSBYE off=6515008 len=601503 2022-04-22T17:49:02.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 678 completed, throughput=5MB/sec 2022-04-22T17:49:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:02 smithi121 bash[36902]: audit 2022-04-22T17:49:02.234146+0000 mon.a (mon.0) 2236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:02 smithi115 bash[45495]: audit 2022-04-22T17:49:02.234146+0000 mon.a (mon.0) 2236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:02 smithi115 bash[46934]: audit 2022-04-22T17:49:02.234146+0000 mon.a (mon.0) 2236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:03.642 INFO:tasks.workunit.client.0.smithi115.stdout: 136: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:03.642 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eZRIYyxrhMAuneG off=3169844 len=125580 2022-04-22T17:49:03.646 INFO:tasks.workunit.client.0.smithi115.stdout:op 679 completed, throughput=4.97MB/sec 2022-04-22T17:49:03.646 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HXN6l85Kotv-K3n off=9714724 len=1542108 2022-04-22T17:49:03.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 680 completed, throughput=4.98MB/sec 2022-04-22T17:49:03.655 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-btZkz8klGj9dLlX off=597523 len=102640 2022-04-22T17:49:03.656 INFO:tasks.workunit.client.0.smithi115.stdout:op 681 completed, throughput=4.98MB/sec 2022-04-22T17:49:03.656 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u1c-uCfaRUEgKz5 off=281360 len=467697 2022-04-22T17:49:03.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 682 completed, throughput=4.98MB/sec 2022-04-22T17:49:03.657 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oyiVmZom_hz1p_j off=2380672 len=991262 2022-04-22T17:49:03.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 683 completed, throughput=4.99MB/sec 2022-04-22T17:49:03.658 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kpkHx-vaeL77DBS off=1153014 len=862515 2022-04-22T17:49:03.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 684 completed, throughput=4.99MB/sec 2022-04-22T17:49:03.663 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2IT5jfe61aHhuAb off=8841273 len=2062814 2022-04-22T17:49:03.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 685 completed, throughput=5.01MB/sec 2022-04-22T17:49:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:03 smithi121 bash[36902]: cluster 2022-04-22T17:49:02.311173+0000 mgr.y (mgr.34107) 838 : cluster [DBG] pgmap v1470: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:03 smithi115 bash[45495]: cluster 2022-04-22T17:49:02.311173+0000 mgr.y (mgr.34107) 838 : cluster [DBG] pgmap v1470: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:03 smithi115 bash[46934]: cluster 2022-04-22T17:49:02.311173+0000 mgr.y (mgr.34107) 838 : cluster [DBG] pgmap v1470: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:04.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:03.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:04.673 INFO:tasks.workunit.client.0.smithi115.stdout: 137: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:04.673 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cQo8jJUNqvayRJ6 off=15379236 len=1263234 2022-04-22T17:49:04.682 INFO:tasks.workunit.client.0.smithi115.stdout:op 686 completed, throughput=4.98MB/sec 2022-04-22T17:49:04.682 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jf2vGj-JJ5xGNQY off=7780180 len=435457 2022-04-22T17:49:04.686 INFO:tasks.workunit.client.0.smithi115.stdout:op 687 completed, throughput=4.98MB/sec 2022-04-22T17:49:04.686 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QapHrxrhlWSJtwT off=5157526 len=1665973 2022-04-22T17:49:04.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 688 completed, throughput=4.99MB/sec 2022-04-22T17:49:04.689 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_UJZ-3uCB8RfSG- off=5432938 len=94526 2022-04-22T17:49:04.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 689 completed, throughput=5MB/sec 2022-04-22T17:49:04.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vI-dneRDU5cNYND off=1537744 len=1683047 2022-04-22T17:49:04.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 690 completed, throughput=5.01MB/sec 2022-04-22T17:49:05.700 INFO:tasks.workunit.client.0.smithi115.stdout: 138: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:05.701 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CgPsUck0pou5Vv5 off=7664324 len=2069540 2022-04-22T17:49:05.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 691 completed, throughput=4.98MB/sec 2022-04-22T17:49:05.710 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yZmNRblb3QjKcmd off=4268490 len=507483 2022-04-22T17:49:05.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 692 completed, throughput=4.99MB/sec 2022-04-22T17:49:05.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UI6gaoJXBED115S off=2125809 len=1408802 2022-04-22T17:49:05.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 693 completed, throughput=5MB/sec 2022-04-22T17:49:05.716 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E_K2XoJSEvNJIlX off=2050326 len=598795 2022-04-22T17:49:05.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 694 completed, throughput=5MB/sec 2022-04-22T17:49:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:05 smithi121 bash[36902]: cluster 2022-04-22T17:49:04.312258+0000 mgr.y (mgr.34107) 839 : cluster [DBG] pgmap v1471: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:05 smithi121 bash[36902]: audit 2022-04-22T17:49:04.856492+0000 mon.a (mon.0) 2237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:05 smithi121 bash[36902]: audit 2022-04-22T17:49:05.149517+0000 mon.a (mon.0) 2238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[45495]: cluster 2022-04-22T17:49:04.312258+0000 mgr.y (mgr.34107) 839 : cluster [DBG] pgmap v1471: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[45495]: audit 2022-04-22T17:49:04.856492+0000 mon.a (mon.0) 2237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[45495]: audit 2022-04-22T17:49:05.149517+0000 mon.a (mon.0) 2238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[46934]: cluster 2022-04-22T17:49:04.312258+0000 mgr.y (mgr.34107) 839 : cluster [DBG] pgmap v1471: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:49:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[46934]: audit 2022-04-22T17:49:04.856492+0000 mon.a (mon.0) 2237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:05 smithi115 bash[46934]: audit 2022-04-22T17:49:05.149517+0000 mon.a (mon.0) 2238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:06.718 INFO:tasks.workunit.client.0.smithi115.stdout: 139: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:06.718 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pLN9u3jD4oFevlH off=20722090 len=70269 2022-04-22T17:49:06.722 INFO:tasks.workunit.client.0.smithi115.stdout:op 695 completed, throughput=4.97MB/sec 2022-04-22T17:49:06.722 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EWlokytYI5Fs2qD off=15826622 len=258801 2022-04-22T17:49:06.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 696 completed, throughput=4.97MB/sec 2022-04-22T17:49:06.727 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wzObXnQdes3Nio4 off=2111506 len=1705579 2022-04-22T17:49:06.730 INFO:tasks.workunit.client.0.smithi115.stdout:op 697 completed, throughput=4.98MB/sec 2022-04-22T17:49:06.730 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mjcvV-1_7PIvgyG off=1177295 len=1843387 2022-04-22T17:49:06.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 698 completed, throughput=4.99MB/sec 2022-04-22T17:49:06.734 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P9SHfd46JR1OlCd off=2445517 len=1006918 2022-04-22T17:49:06.739 INFO:tasks.workunit.client.0.smithi115.stdout:op 699 completed, throughput=5MB/sec 2022-04-22T17:49:06.739 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O3eIZQRRCUfyF8A off=1787281 len=1045102 2022-04-22T17:49:06.740 INFO:tasks.workunit.client.0.smithi115.stdout:op 700 completed, throughput=5.01MB/sec 2022-04-22T17:49:07.740 INFO:tasks.workunit.client.0.smithi115.stdout: 140: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:07.789 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-coG0sXFm681RQlm off=11902588 len=259499 2022-04-22T17:49:07.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 701 completed, throughput=4.97MB/sec 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4ygmsQqeBiOl7qu off=3397226 len=215845 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 702 completed, throughput=4.97MB/sec 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xd8Bkki0I65osCH off=2089120 len=840841 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 703 completed, throughput=4.98MB/sec 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o7mcd7Pi6BWOvaI off=1267657 len=216227 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 704 completed, throughput=4.98MB/sec 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MaCDmCL22HkrtBY off=1751511 len=86344 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 705 completed, throughput=4.98MB/sec 2022-04-22T17:49:07.791 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-omyqNwMhYsokJ15 off=3971885 len=1556191 2022-04-22T17:49:07.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 706 completed, throughput=4.99MB/sec 2022-04-22T17:49:07.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2EwTRazBl2VMvXl off=2442365 len=961236 2022-04-22T17:49:07.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 707 completed, throughput=5MB/sec 2022-04-22T17:49:07.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-78i5sIWvEDYFXNX off=3352739 len=1278601 2022-04-22T17:49:07.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 708 completed, throughput=5.01MB/sec 2022-04-22T17:49:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:07 smithi121 bash[36902]: cluster 2022-04-22T17:49:06.313407+0000 mgr.y (mgr.34107) 840 : cluster [DBG] pgmap v1472: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T17:49:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:07 smithi121 bash[36902]: audit 2022-04-22T17:49:07.237271+0000 mon.a (mon.0) 2239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:07 smithi115 bash[45495]: cluster 2022-04-22T17:49:06.313407+0000 mgr.y (mgr.34107) 840 : cluster [DBG] pgmap v1472: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T17:49:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:07 smithi115 bash[45495]: audit 2022-04-22T17:49:07.237271+0000 mon.a (mon.0) 2239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:07 smithi115 bash[46934]: cluster 2022-04-22T17:49:06.313407+0000 mgr.y (mgr.34107) 840 : cluster [DBG] pgmap v1472: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T17:49:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:07 smithi115 bash[46934]: audit 2022-04-22T17:49:07.237271+0000 mon.a (mon.0) 2239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:08.762 INFO:tasks.workunit.client.0.smithi115.stdout: 141: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:08.763 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hiKWKcEYxW3y2Da off=1825779 len=770765 2022-04-22T17:49:08.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 709 completed, throughput=4.98MB/sec 2022-04-22T17:49:08.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QiiVXsJsircHDk1 off=3219291 len=1481543 2022-04-22T17:49:08.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 710 completed, throughput=4.99MB/sec 2022-04-22T17:49:08.777 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xfyAsF4S3yWSBxd off=901165 len=169605 2022-04-22T17:49:08.780 INFO:tasks.workunit.client.0.smithi115.stdout:op 711 completed, throughput=4.99MB/sec 2022-04-22T17:49:08.781 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--ja6K88ZXPMZqC- off=2073407 len=540953 2022-04-22T17:49:08.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 712 completed, throughput=4.99MB/sec 2022-04-22T17:49:08.785 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h1Z_tl59OTT9BR2 off=9560583 len=607279 2022-04-22T17:49:08.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 713 completed, throughput=4.99MB/sec 2022-04-22T17:49:08.787 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sikLUlziwpF_bb5 off=16363820 len=585973 2022-04-22T17:49:08.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 714 completed, throughput=5MB/sec 2022-04-22T17:49:08.788 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8l3M-V4sl_67ko_ off=485232 len=232247 2022-04-22T17:49:08.789 INFO:tasks.workunit.client.0.smithi115.stdout:op 715 completed, throughput=5MB/sec 2022-04-22T17:49:08.789 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oXnh1L1d8FamZxd off=16912659 len=1989714 2022-04-22T17:49:08.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 716 completed, throughput=5.01MB/sec 2022-04-22T17:49:09.793 INFO:tasks.workunit.client.0.smithi115.stdout: 142: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:09.794 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YDLQlJjWY_87u9w off=9145515 len=1136350 2022-04-22T17:49:09.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 717 completed, throughput=4.98MB/sec 2022-04-22T17:49:09.800 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ko-xQjzUMV8BuRQ off=11557202 len=660323 2022-04-22T17:49:09.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 718 completed, throughput=4.99MB/sec 2022-04-22T17:49:09.807 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TeWnjpHfXwoF6Zo off=6996473 len=470163 2022-04-22T17:49:09.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 719 completed, throughput=4.99MB/sec 2022-04-22T17:49:09.811 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xIL1Gw1GMgMTvl4 off=584198 len=276424 2022-04-22T17:49:09.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 720 completed, throughput=4.99MB/sec 2022-04-22T17:49:09.815 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5yYU99aNIsPOVBr off=688951 len=1570565 2022-04-22T17:49:09.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 721 completed, throughput=5MB/sec 2022-04-22T17:49:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:09 smithi121 bash[36902]: cluster 2022-04-22T17:49:08.314523+0000 mgr.y (mgr.34107) 841 : cluster [DBG] pgmap v1473: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:09 smithi115 bash[45495]: cluster 2022-04-22T17:49:08.314523+0000 mgr.y (mgr.34107) 841 : cluster [DBG] pgmap v1473: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:09 smithi115 bash[46934]: cluster 2022-04-22T17:49:08.314523+0000 mgr.y (mgr.34107) 841 : cluster [DBG] pgmap v1473: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: 143: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dif4pZNFWPvb8Ka off=8399035 len=2061557 2022-04-22T17:49:10.828 INFO:tasks.workunit.client.0.smithi115.stdout:op 722 completed, throughput=4.98MB/sec 2022-04-22T17:49:10.828 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GLO0mPQY4jYks_E off=17649023 len=1506960 2022-04-22T17:49:10.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 723 completed, throughput=4.99MB/sec 2022-04-22T17:49:10.836 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7fa0gYOajZxrFdP off=2162243 len=1886417 2022-04-22T17:49:10.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 724 completed, throughput=5MB/sec 2022-04-22T17:49:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:10 smithi121 bash[36902]: audit 2022-04-22T17:49:09.859616+0000 mon.a (mon.0) 2240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:10 smithi121 bash[36902]: audit 2022-04-22T17:49:10.153116+0000 mon.a (mon.0) 2241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:10 smithi115 bash[45495]: audit 2022-04-22T17:49:09.859616+0000 mon.a (mon.0) 2240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:11.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:10 smithi115 bash[45495]: audit 2022-04-22T17:49:10.153116+0000 mon.a (mon.0) 2241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:11.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:10 smithi115 bash[46934]: audit 2022-04-22T17:49:09.859616+0000 mon.a (mon.0) 2240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:11.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:10 smithi115 bash[46934]: audit 2022-04-22T17:49:10.153116+0000 mon.a (mon.0) 2241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:11.845 INFO:tasks.workunit.client.0.smithi115.stdout: 144: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:11.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-a1Gc_vQ1P7VWkzv off=9834350 len=520816 2022-04-22T17:49:11.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 725 completed, throughput=4.97MB/sec 2022-04-22T17:49:11.851 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c_kNPKxsPJj1YEz off=6135251 len=766360 2022-04-22T17:49:11.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 726 completed, throughput=4.98MB/sec 2022-04-22T17:49:11.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bht78-NgtpZYuh4 off=6173283 len=1636355 2022-04-22T17:49:11.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 727 completed, throughput=4.99MB/sec 2022-04-22T17:49:11.862 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-28esCuGv5lCME6v off=1147386 len=385931 2022-04-22T17:49:11.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 728 completed, throughput=4.99MB/sec 2022-04-22T17:49:11.867 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6vEWA1kKgt63kLo off=7474552 len=140195 2022-04-22T17:49:11.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 729 completed, throughput=4.99MB/sec 2022-04-22T17:49:11.871 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qTbeuwU8e9hUHvG off=22938673 len=2014323 2022-04-22T17:49:11.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 730 completed, throughput=5MB/sec 2022-04-22T17:49:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:11 smithi121 bash[36902]: cluster 2022-04-22T17:49:10.315590+0000 mgr.y (mgr.34107) 842 : cluster [DBG] pgmap v1474: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:49:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:11 smithi115 bash[45495]: cluster 2022-04-22T17:49:10.315590+0000 mgr.y (mgr.34107) 842 : cluster [DBG] pgmap v1474: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:49:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:11.676Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:11 smithi115 bash[46934]: cluster 2022-04-22T17:49:10.315590+0000 mgr.y (mgr.34107) 842 : cluster [DBG] pgmap v1474: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:49:12.876 INFO:tasks.workunit.client.0.smithi115.stdout: 145: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:12.877 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IAH8NsMVPnz2FLz off=17470295 len=240945 2022-04-22T17:49:12.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 731 completed, throughput=4.97MB/sec 2022-04-22T17:49:12.882 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OVjq-2tOSYMhdfu off=9337499 len=829984 2022-04-22T17:49:12.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 732 completed, throughput=4.98MB/sec 2022-04-22T17:49:12.885 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ahKedXHeESh82-m off=420520 len=1606802 2022-04-22T17:49:12.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 733 completed, throughput=4.99MB/sec 2022-04-22T17:49:12.889 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mdux6u-popa74z4 off=10713593 len=578352 2022-04-22T17:49:12.894 INFO:tasks.workunit.client.0.smithi115.stdout:op 734 completed, throughput=4.99MB/sec 2022-04-22T17:49:12.894 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QlUAVEslM3iK3NR off=1715087 len=1307230 2022-04-22T17:49:12.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 735 completed, throughput=5MB/sec 2022-04-22T17:49:12.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-45fczfvBTqXLIw- off=3173279 len=1710444 2022-04-22T17:49:12.915 INFO:tasks.workunit.client.0.smithi115.stdout:op 736 completed, throughput=5.01MB/sec 2022-04-22T17:49:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:12 smithi115 bash[46934]: audit 2022-04-22T17:49:12.240290+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:13.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:12 smithi115 bash[45495]: audit 2022-04-22T17:49:12.240290+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:12 smithi121 bash[36902]: audit 2022-04-22T17:49:12.240290+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:13.910 INFO:tasks.workunit.client.0.smithi115.stdout: 146: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:13.951 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SQocHM2k6GyEFrp off=5080755 len=1292294 2022-04-22T17:49:13.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 737 completed, throughput=4.98MB/sec 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-efXYzI7_LiTzTxn off=17565545 len=626500 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 738 completed, throughput=4.99MB/sec 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--fbPKcXKNzUXsgV off=23991208 len=907162 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 739 completed, throughput=4.99MB/sec 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bWRrFe6_KV50UIr off=18648726 len=557267 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 740 completed, throughput=5MB/sec 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KQPRfSsHY1hplvT off=4475664 len=1044509 2022-04-22T17:49:13.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 741 completed, throughput=5MB/sec 2022-04-22T17:49:14.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:13.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[45495]: cluster 2022-04-22T17:49:12.316631+0000 mgr.y (mgr.34107) 843 : cluster [DBG] pgmap v1475: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:49:14.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[45495]: audit 2022-04-22T17:49:13.451502+0000 mon.a (mon.0) 2243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:49:14.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[45495]: audit 2022-04-22T17:49:13.453619+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:49:14.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[46934]: cluster 2022-04-22T17:49:12.316631+0000 mgr.y (mgr.34107) 843 : cluster [DBG] pgmap v1475: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:49:14.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[46934]: audit 2022-04-22T17:49:13.451502+0000 mon.a (mon.0) 2243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:49:14.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:13 smithi115 bash[46934]: audit 2022-04-22T17:49:13.453619+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:49:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:13 smithi121 bash[36902]: cluster 2022-04-22T17:49:12.316631+0000 mgr.y (mgr.34107) 843 : cluster [DBG] pgmap v1475: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:49:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:13 smithi121 bash[36902]: audit 2022-04-22T17:49:13.451502+0000 mon.a (mon.0) 2243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:49:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:13 smithi121 bash[36902]: audit 2022-04-22T17:49:13.453619+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:49:14.928 INFO:tasks.workunit.client.0.smithi115.stdout: 147: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:14.952 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D_KV6bYlhnyvW_y off=6388342 len=381538 2022-04-22T17:49:14.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 742 completed, throughput=4.97MB/sec 2022-04-22T17:49:14.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yrTXysYjrgl5fgl off=3679188 len=454877 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 743 completed, throughput=4.98MB/sec 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lPvRjTqkW8Hkh7B off=2986127 len=1168929 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 744 completed, throughput=4.98MB/sec 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cQD-NeuxkJc01GG off=8036936 len=1805146 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 745 completed, throughput=4.99MB/sec 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S-SBDx6gJ9QyEFn off=8150084 len=63411 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 746 completed, throughput=4.99MB/sec 2022-04-22T17:49:14.954 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wIBwaL7NURU6JQE off=3615444 len=503818 2022-04-22T17:49:14.955 INFO:tasks.workunit.client.0.smithi115.stdout:op 747 completed, throughput=5MB/sec 2022-04-22T17:49:14.955 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lE2AfpfyLNLUICd off=7062961 len=402609 2022-04-22T17:49:14.955 INFO:tasks.workunit.client.0.smithi115.stdout:op 748 completed, throughput=5MB/sec 2022-04-22T17:49:15.955 INFO:tasks.workunit.client.0.smithi115.stdout: 148: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:15.955 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xd8Bkki0I65osCH off=2444497 len=831991 2022-04-22T17:49:15.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 749 completed, throughput=4.97MB/sec 2022-04-22T17:49:15.962 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sLHmgrTFIRz1L3f off=8720205 len=439962 2022-04-22T17:49:15.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 750 completed, throughput=4.97MB/sec 2022-04-22T17:49:15.964 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NRQRGAwwt6jZ0eW off=2513415 len=1342329 2022-04-22T17:49:15.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 751 completed, throughput=4.98MB/sec 2022-04-22T17:49:15.972 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5UndnuF9wwFjeW_ off=1367338 len=1650243 2022-04-22T17:49:15.978 INFO:tasks.workunit.client.0.smithi115.stdout:op 752 completed, throughput=4.99MB/sec 2022-04-22T17:49:15.978 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ARQsoYtWIJBoT_u off=16863104 len=1004718 2022-04-22T17:49:15.983 INFO:tasks.workunit.client.0.smithi115.stdout:op 753 completed, throughput=5MB/sec 2022-04-22T17:49:15.983 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-83vJUWpPAz2LwVw off=2225550 len=42276 2022-04-22T17:49:15.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 754 completed, throughput=5MB/sec 2022-04-22T17:49:15.984 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Oub3-kwFT-ekQPj off=2623933 len=1186310 2022-04-22T17:49:15.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 755 completed, throughput=5.01MB/sec 2022-04-22T17:49:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[45495]: cluster 2022-04-22T17:49:14.317721+0000 mgr.y (mgr.34107) 844 : cluster [DBG] pgmap v1476: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2022-04-22T17:49:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[45495]: audit 2022-04-22T17:49:14.862537+0000 mon.a (mon.0) 2245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[45495]: audit 2022-04-22T17:49:15.156793+0000 mon.a (mon.0) 2246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[46934]: cluster 2022-04-22T17:49:14.317721+0000 mgr.y (mgr.34107) 844 : cluster [DBG] pgmap v1476: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2022-04-22T17:49:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[46934]: audit 2022-04-22T17:49:14.862537+0000 mon.a (mon.0) 2245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:15 smithi115 bash[46934]: audit 2022-04-22T17:49:15.156793+0000 mon.a (mon.0) 2246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:15 smithi121 bash[36902]: cluster 2022-04-22T17:49:14.317721+0000 mgr.y (mgr.34107) 844 : cluster [DBG] pgmap v1476: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 6 op/s 2022-04-22T17:49:16.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:15 smithi121 bash[36902]: audit 2022-04-22T17:49:14.862537+0000 mon.a (mon.0) 2245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:16.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:15 smithi121 bash[36902]: audit 2022-04-22T17:49:15.156793+0000 mon.a (mon.0) 2246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:16.989 INFO:tasks.workunit.client.0.smithi115.stdout: 149: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:16.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TcXSLFbXIOhfSde off=8400173 len=2031352 2022-04-22T17:49:16.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 756 completed, throughput=4.99MB/sec 2022-04-22T17:49:16.993 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PnbPoOXnsQEHHvd off=2358958 len=52575 2022-04-22T17:49:16.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 757 completed, throughput=4.99MB/sec 2022-04-22T17:49:16.994 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dwlCWiUjKeP2hdl off=4941499 len=825215 2022-04-22T17:49:16.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 758 completed, throughput=4.99MB/sec 2022-04-22T17:49:16.998 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R7i4ehMbX6Dziqf off=10129907 len=73702 2022-04-22T17:49:16.999 INFO:tasks.workunit.client.0.smithi115.stdout:op 759 completed, throughput=4.99MB/sec 2022-04-22T17:49:16.999 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GoMWWgYrBCrflaP off=275010 len=1245193 2022-04-22T17:49:17.007 INFO:tasks.workunit.client.0.smithi115.stdout:op 760 completed, throughput=5MB/sec 2022-04-22T17:49:17.007 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lqqPNkSjihVEEXO off=8511815 len=190118 2022-04-22T17:49:17.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 761 completed, throughput=5MB/sec 2022-04-22T17:49:18.009 INFO:tasks.workunit.client.0.smithi115.stdout: 150: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:18.009 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-axZhN6gjx9cSbkz off=3972837 len=1973554 2022-04-22T17:49:18.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 762 completed, throughput=4.98MB/sec 2022-04-22T17:49:18.013 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-20fJFiN0SyMYa7Z off=1371987 len=198315 2022-04-22T17:49:18.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 763 completed, throughput=4.98MB/sec 2022-04-22T17:49:18.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZAcNFwjEMr3FS2i off=5424578 len=1984780 2022-04-22T17:49:18.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 764 completed, throughput=4.99MB/sec 2022-04-22T17:49:18.022 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8yTT3OTpgMTjKqV off=9227306 len=1356169 2022-04-22T17:49:18.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 765 completed, throughput=5MB/sec 2022-04-22T17:49:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:17 smithi115 bash[45495]: cluster 2022-04-22T17:49:16.318800+0000 mgr.y (mgr.34107) 845 : cluster [DBG] pgmap v1477: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:49:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:17 smithi115 bash[45495]: audit 2022-04-22T17:49:17.243271+0000 mon.a (mon.0) 2247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:17 smithi115 bash[46934]: cluster 2022-04-22T17:49:16.318800+0000 mgr.y (mgr.34107) 845 : cluster [DBG] pgmap v1477: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:49:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:17 smithi115 bash[46934]: audit 2022-04-22T17:49:17.243271+0000 mon.a (mon.0) 2247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:17 smithi121 bash[36902]: cluster 2022-04-22T17:49:16.318800+0000 mgr.y (mgr.34107) 845 : cluster [DBG] pgmap v1477: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:49:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:17 smithi121 bash[36902]: audit 2022-04-22T17:49:17.243271+0000 mon.a (mon.0) 2247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:19.030 INFO:tasks.workunit.client.0.smithi115.stdout: 151: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:19.030 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z6mjugT14A58Zwf off=2844147 len=2030858 2022-04-22T17:49:19.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 766 completed, throughput=4.98MB/sec 2022-04-22T17:49:19.038 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jE9RyhIkfkPHUKO off=2207355 len=2044056 2022-04-22T17:49:19.044 INFO:tasks.workunit.client.0.smithi115.stdout:op 767 completed, throughput=4.99MB/sec 2022-04-22T17:49:19.044 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n4TwLDwDRtbO4ks off=2985480 len=1313808 2022-04-22T17:49:19.046 INFO:tasks.workunit.client.0.smithi115.stdout:op 768 completed, throughput=5MB/sec 2022-04-22T17:49:20.046 INFO:tasks.workunit.client.0.smithi115.stdout: 152: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:20.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5LeCvEb8mUd-dq5 off=3203459 len=471765 2022-04-22T17:49:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:19 smithi115 bash[45495]: cluster 2022-04-22T17:49:18.319917+0000 mgr.y (mgr.34107) 846 : cluster [DBG] pgmap v1478: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:19 smithi115 bash[46934]: cluster 2022-04-22T17:49:18.319917+0000 mgr.y (mgr.34107) 846 : cluster [DBG] pgmap v1478: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:20.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 769 completed, throughput=4.97MB/sec 2022-04-22T17:49:20.051 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-00FM0j3OFF-GLTg off=4340520 len=954362 2022-04-22T17:49:20.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 770 completed, throughput=4.98MB/sec 2022-04-22T17:49:20.053 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vUWdZkLGMm6ACHE off=2151438 len=570677 2022-04-22T17:49:20.055 INFO:tasks.workunit.client.0.smithi115.stdout:op 771 completed, throughput=4.98MB/sec 2022-04-22T17:49:20.055 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SmnmChpWRyC2OzU off=4508584 len=921692 2022-04-22T17:49:20.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 772 completed, throughput=4.99MB/sec 2022-04-22T17:49:20.058 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H_eR7LZQmkbDsra off=11235982 len=975249 2022-04-22T17:49:20.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 773 completed, throughput=4.99MB/sec 2022-04-22T17:49:20.060 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HPAIafeW3BrxP_9 off=10482492 len=1086437 2022-04-22T17:49:20.066 INFO:tasks.workunit.client.0.smithi115.stdout:op 774 completed, throughput=5MB/sec 2022-04-22T17:49:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:19 smithi121 bash[36902]: cluster 2022-04-22T17:49:18.319917+0000 mgr.y (mgr.34107) 846 : cluster [DBG] pgmap v1478: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:20 smithi115 bash[45495]: audit 2022-04-22T17:49:19.865961+0000 mon.a (mon.0) 2248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:20 smithi115 bash[45495]: audit 2022-04-22T17:49:20.160486+0000 mon.a (mon.0) 2249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:20 smithi115 bash[46934]: audit 2022-04-22T17:49:19.865961+0000 mon.a (mon.0) 2248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:21.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:20 smithi115 bash[46934]: audit 2022-04-22T17:49:20.160486+0000 mon.a (mon.0) 2249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:21.066 INFO:tasks.workunit.client.0.smithi115.stdout: 153: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:21.066 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PM1anALiH_wEZwY off=6741706 len=994919 2022-04-22T17:49:21.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 775 completed, throughput=4.97MB/sec 2022-04-22T17:49:21.069 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZxaHFNfR7ovMzkR off=2965066 len=317912 2022-04-22T17:49:21.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 776 completed, throughput=4.98MB/sec 2022-04-22T17:49:21.075 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rYb0IGDDDLPcDN4 off=17534846 len=76000 2022-04-22T17:49:21.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 777 completed, throughput=4.98MB/sec 2022-04-22T17:49:21.076 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RgyC4zNq-2-64pa off=3236828 len=538405 2022-04-22T17:49:21.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 778 completed, throughput=4.98MB/sec 2022-04-22T17:49:21.081 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uSyBRtkcHErLUBa off=8353121 len=1742055 2022-04-22T17:49:21.083 INFO:tasks.workunit.client.0.smithi115.stdout:op 779 completed, throughput=4.99MB/sec 2022-04-22T17:49:21.083 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hLhbFW4E0kCahs9 off=252676 len=1790474 2022-04-22T17:49:21.086 INFO:tasks.workunit.client.0.smithi115.stdout:op 780 completed, throughput=5MB/sec 2022-04-22T17:49:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:20 smithi121 bash[36902]: audit 2022-04-22T17:49:19.865961+0000 mon.a (mon.0) 2248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:20 smithi121 bash[36902]: audit 2022-04-22T17:49:20.160486+0000 mon.a (mon.0) 2249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:21 smithi115 bash[45495]: cluster 2022-04-22T17:49:20.321110+0000 mgr.y (mgr.34107) 847 : cluster [DBG] pgmap v1479: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:21.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:21 smithi115 bash[46934]: cluster 2022-04-22T17:49:20.321110+0000 mgr.y (mgr.34107) 847 : cluster [DBG] pgmap v1479: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:22.086 INFO:tasks.workunit.client.0.smithi115.stdout: 154: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:22.086 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ksdUaa8OrgQukGx off=0 len=1392086 2022-04-22T17:49:22.090 INFO:tasks.workunit.client.0.smithi115.stdout:op 781 completed, throughput=4.98MB/sec 2022-04-22T17:49:22.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2tL482yoX5kK-AZ off=19414 len=1146246 2022-04-22T17:49:22.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 782 completed, throughput=4.98MB/sec 2022-04-22T17:49:22.095 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rQTLTQhVEDFdnpZ off=4248125 len=1398258 2022-04-22T17:49:22.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 783 completed, throughput=4.99MB/sec 2022-04-22T17:49:22.101 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I_PMnweZgvVoIJe off=2270964 len=372512 2022-04-22T17:49:22.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 784 completed, throughput=4.99MB/sec 2022-04-22T17:49:22.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8cNz5J46L_NrwaX off=8285 len=573133 2022-04-22T17:49:22.104 INFO:tasks.workunit.client.0.smithi115.stdout:op 785 completed, throughput=5MB/sec 2022-04-22T17:49:22.104 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z9MmWg0iZlzgBbu off=6904362 len=664659 2022-04-22T17:49:22.108 INFO:tasks.workunit.client.0.smithi115.stdout:op 786 completed, throughput=5MB/sec 2022-04-22T17:49:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:21 smithi121 bash[36902]: cluster 2022-04-22T17:49:20.321110+0000 mgr.y (mgr.34107) 847 : cluster [DBG] pgmap v1479: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:23.109 INFO:tasks.workunit.client.0.smithi115.stdout: 155: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:23.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FIWXbCi_Yke6TNu off=6274945 len=1890869 2022-04-22T17:49:23.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 787 completed, throughput=4.98MB/sec 2022-04-22T17:49:23.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZP-FvEwBah8GM1w off=1524510 len=580387 2022-04-22T17:49:23.123 INFO:tasks.workunit.client.0.smithi115.stdout:op 788 completed, throughput=4.98MB/sec 2022-04-22T17:49:23.123 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nUwLdfCKGYbVPlc off=10615890 len=241546 2022-04-22T17:49:23.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 789 completed, throughput=4.99MB/sec 2022-04-22T17:49:23.124 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-piBhWCPc5EImMKq off=6003064 len=1772548 2022-04-22T17:49:23.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 790 completed, throughput=5MB/sec 2022-04-22T17:49:23.130 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kMdVKdOGpNWXhVR off=6746473 len=825730 2022-04-22T17:49:23.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 791 completed, throughput=5MB/sec 2022-04-22T17:49:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:22 smithi121 bash[36902]: audit 2022-04-22T17:49:22.246337+0000 mon.a (mon.0) 2250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:22 smithi115 bash[45495]: audit 2022-04-22T17:49:22.246337+0000 mon.a (mon.0) 2250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:22 smithi115 bash[46934]: audit 2022-04-22T17:49:22.246337+0000 mon.a (mon.0) 2250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:23 smithi115 bash[45495]: cluster 2022-04-22T17:49:22.322240+0000 mgr.y (mgr.34107) 848 : cluster [DBG] pgmap v1480: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:24.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:23.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:24.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:23 smithi115 bash[46934]: cluster 2022-04-22T17:49:22.322240+0000 mgr.y (mgr.34107) 848 : cluster [DBG] pgmap v1480: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:24.136 INFO:tasks.workunit.client.0.smithi115.stdout: 156: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:24.136 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lZ_lR_tFgurvMAP off=0 len=1067568 2022-04-22T17:49:24.139 INFO:tasks.workunit.client.0.smithi115.stdout:op 792 completed, throughput=4.98MB/sec 2022-04-22T17:49:24.139 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JQmCqeel74pmRq_ off=53757 len=1553643 2022-04-22T17:49:24.142 INFO:tasks.workunit.client.0.smithi115.stdout:op 793 completed, throughput=4.99MB/sec 2022-04-22T17:49:24.142 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fi2RA7wNw2baxQW off=397603 len=1147157 2022-04-22T17:49:24.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 794 completed, throughput=4.99MB/sec 2022-04-22T17:49:24.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HgbZKgeZz1q8DB6 off=148216 len=1952242 2022-04-22T17:49:24.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 795 completed, throughput=5MB/sec 2022-04-22T17:49:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:23 smithi121 bash[36902]: cluster 2022-04-22T17:49:22.322240+0000 mgr.y (mgr.34107) 848 : cluster [DBG] pgmap v1480: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:25.154 INFO:tasks.workunit.client.0.smithi115.stdout: 157: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:25.154 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-akf3GCLRPjWEhSc off=7023828 len=1446255 2022-04-22T17:49:25.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 796 completed, throughput=4.98MB/sec 2022-04-22T17:49:25.163 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xzT4qT9frgUc92L off=1585753 len=287279 2022-04-22T17:49:25.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 797 completed, throughput=4.98MB/sec 2022-04-22T17:49:25.167 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Reedoz_qsNWfFEg off=1231503 len=153260 2022-04-22T17:49:25.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 798 completed, throughput=4.98MB/sec 2022-04-22T17:49:25.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BpAsCN0pOJ1e4ZR off=6841470 len=580042 2022-04-22T17:49:25.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 799 completed, throughput=4.99MB/sec 2022-04-22T17:49:25.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IkEJ6Cl_li2lzIC off=3310603 len=975476 2022-04-22T17:49:25.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 800 completed, throughput=4.99MB/sec 2022-04-22T17:49:25.175 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1JOuq7h4czcZBPP off=15537146 len=1841459 2022-04-22T17:49:25.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 801 completed, throughput=5MB/sec 2022-04-22T17:49:26.185 INFO:tasks.workunit.client.0.smithi115.stdout: 158: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:26.275 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vrwZwI-Cqm5V6b4 off=1122165 len=1777143 2022-04-22T17:49:26.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:25 smithi121 bash[36902]: cluster 2022-04-22T17:49:24.323323+0000 mgr.y (mgr.34107) 849 : cluster [DBG] pgmap v1481: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:26.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:25 smithi121 bash[36902]: audit 2022-04-22T17:49:24.869140+0000 mon.a (mon.0) 2251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:26.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:25 smithi121 bash[36902]: audit 2022-04-22T17:49:25.164049+0000 mon.a (mon.0) 2252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:26.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 802 completed, throughput=4.98MB/sec 2022-04-22T17:49:26.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cquPiHud6vmo8g6 off=4047622 len=1835327 2022-04-22T17:49:26.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 803 completed, throughput=4.99MB/sec 2022-04-22T17:49:26.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s-m18Sg7BAs8Xil off=10302599 len=1419067 2022-04-22T17:49:26.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 804 completed, throughput=5MB/sec 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[45495]: cluster 2022-04-22T17:49:24.323323+0000 mgr.y (mgr.34107) 849 : cluster [DBG] pgmap v1481: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[45495]: audit 2022-04-22T17:49:24.869140+0000 mon.a (mon.0) 2251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[45495]: audit 2022-04-22T17:49:25.164049+0000 mon.a (mon.0) 2252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[46934]: cluster 2022-04-22T17:49:24.323323+0000 mgr.y (mgr.34107) 849 : cluster [DBG] pgmap v1481: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[46934]: audit 2022-04-22T17:49:24.869140+0000 mon.a (mon.0) 2251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:25 smithi115 bash[46934]: audit 2022-04-22T17:49:25.164049+0000 mon.a (mon.0) 2252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:27.205 INFO:tasks.workunit.client.0.smithi115.stdout: 159: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:27.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IID3_f4gfnONfwz off=1609727 len=1280708 2022-04-22T17:49:27.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 805 completed, throughput=4.98MB/sec 2022-04-22T17:49:27.214 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0i-qcWi6PnltWds off=212333 len=1557014 2022-04-22T17:49:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 806 completed, throughput=4.99MB/sec 2022-04-22T17:49:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AnT-fsLEzvd_KVZ off=15297554 len=1697733 2022-04-22T17:49:27.223 INFO:tasks.workunit.client.0.smithi115.stdout:op 807 completed, throughput=5MB/sec 2022-04-22T17:49:27.223 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BIWhCc2wNA5_xt8 off=12040249 len=1965612 2022-04-22T17:49:27.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 808 completed, throughput=5.01MB/sec 2022-04-22T17:49:27.371 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (12m) 110s ago 26m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (26m) 2m ago 26m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (14m) 110s ago 27m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (14m) 2m ago 27m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (21m) 2m ago 31m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:49:27.719 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (22m) 110s ago 33m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (21m) 110s ago 33m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (20m) 2m ago 31m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (20m) 110s ago 32m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (13m) 110s ago 26m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (13m) 2m ago 26m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (19m) 110s ago 31m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (18m) 110s ago 30m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (18m) 110s ago 30m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (17m) 110s ago 29m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:49:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (16m) 2m ago 29m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:49:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (16m) 2m ago 28m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:49:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (15m) 2m ago 28m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:49:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (14m) 2m ago 27m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:49:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (13m) 2m ago 26m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:49:28.127 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:49:28.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:49:28.129 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:49:28.231 INFO:tasks.workunit.client.0.smithi115.stdout: 160: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:28.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ey7wYd4cYbrzkCn off=867531 len=817097 2022-04-22T17:49:28.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 809 completed, throughput=4.98MB/sec 2022-04-22T17:49:28.238 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zHYZxcznXg8un3q off=5087822 len=979490 2022-04-22T17:49:28.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 810 completed, throughput=4.99MB/sec 2022-04-22T17:49:28.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jd87JdK2cgzH0V6 off=624710 len=400955 2022-04-22T17:49:28.243 INFO:tasks.workunit.client.0.smithi115.stdout:op 811 completed, throughput=4.99MB/sec 2022-04-22T17:49:28.243 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mizzW6852QzwglD off=1702585 len=1339677 2022-04-22T17:49:28.249 INFO:tasks.workunit.client.0.smithi115.stdout:op 812 completed, throughput=5MB/sec 2022-04-22T17:49:28.249 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xzIfZPD8gYU2c9y off=8675679 len=934978 2022-04-22T17:49:28.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 813 completed, throughput=5MB/sec 2022-04-22T17:49:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:27 smithi121 bash[36902]: cluster 2022-04-22T17:49:26.324396+0000 mgr.y (mgr.34107) 850 : cluster [DBG] pgmap v1482: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:27 smithi121 bash[36902]: audit 2022-04-22T17:49:27.249495+0000 mon.a (mon.0) 2253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:27 smithi115 bash[45495]: cluster 2022-04-22T17:49:26.324396+0000 mgr.y (mgr.34107) 850 : cluster [DBG] pgmap v1482: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:27 smithi115 bash[45495]: audit 2022-04-22T17:49:27.249495+0000 mon.a (mon.0) 2253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:27 smithi115 bash[46934]: cluster 2022-04-22T17:49:26.324396+0000 mgr.y (mgr.34107) 850 : cluster [DBG] pgmap v1482: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:27 smithi115 bash[46934]: audit 2022-04-22T17:49:27.249495+0000 mon.a (mon.0) 2253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:29.256 INFO:tasks.workunit.client.0.smithi115.stdout: 161: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:29.256 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F1L5IK6RbVmtpt4 off=16246396 len=904888 2022-04-22T17:49:29.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 814 completed, throughput=4.98MB/sec 2022-04-22T17:49:29.263 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r1f_eyRlbNSG3wX off=1229815 len=1297043 2022-04-22T17:49:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:28 smithi121 bash[36902]: audit 2022-04-22T17:49:27.345161+0000 mgr.y (mgr.34107) 851 : audit [DBG] from='client.45241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:28 smithi121 bash[36902]: audit 2022-04-22T17:49:27.690629+0000 mgr.y (mgr.34107) 852 : audit [DBG] from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:28 smithi121 bash[36902]: audit 2022-04-22T17:49:28.117726+0000 mon.c (mon.1) 68 : audit [DBG] from='client.? 172.21.15.115:0/4260729241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:49:29.269 INFO:tasks.workunit.client.0.smithi115.stdout:op 815 completed, throughput=4.98MB/sec 2022-04-22T17:49:29.269 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sn6varEbT310TX0 off=953289 len=303210 2022-04-22T17:49:29.270 INFO:tasks.workunit.client.0.smithi115.stdout:op 816 completed, throughput=4.99MB/sec 2022-04-22T17:49:29.270 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MvObY5T39yxGec8 off=6201114 len=1894236 2022-04-22T17:49:29.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 817 completed, throughput=5MB/sec 2022-04-22T17:49:29.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x-LeIKUr7KnYokc off=2587518 len=388926 2022-04-22T17:49:29.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 818 completed, throughput=5MB/sec 2022-04-22T17:49:29.282 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M8hrtx9-gevLG6h off=6077168 len=1515141 2022-04-22T17:49:29.289 INFO:tasks.workunit.client.0.smithi115.stdout:op 819 completed, throughput=5.01MB/sec 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[46934]: audit 2022-04-22T17:49:27.345161+0000 mgr.y (mgr.34107) 851 : audit [DBG] from='client.45241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[46934]: audit 2022-04-22T17:49:27.690629+0000 mgr.y (mgr.34107) 852 : audit [DBG] from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[46934]: audit 2022-04-22T17:49:28.117726+0000 mon.c (mon.1) 68 : audit [DBG] from='client.? 172.21.15.115:0/4260729241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[45495]: audit 2022-04-22T17:49:27.345161+0000 mgr.y (mgr.34107) 851 : audit [DBG] from='client.45241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[45495]: audit 2022-04-22T17:49:27.690629+0000 mgr.y (mgr.34107) 852 : audit [DBG] from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:49:29.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:28 smithi115 bash[45495]: audit 2022-04-22T17:49:28.117726+0000 mon.c (mon.1) 68 : audit [DBG] from='client.? 172.21.15.115:0/4260729241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:49:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:29 smithi121 bash[36902]: cluster 2022-04-22T17:49:28.325561+0000 mgr.y (mgr.34107) 853 : cluster [DBG] pgmap v1483: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:49:30.292 INFO:tasks.workunit.client.0.smithi115.stdout: 162: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:30.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PSk0HQt8SssLmHz off=19230784 len=1082323 2022-04-22T17:49:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:29 smithi115 bash[45495]: cluster 2022-04-22T17:49:28.325561+0000 mgr.y (mgr.34107) 853 : cluster [DBG] pgmap v1483: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:49:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:29 smithi115 bash[46934]: cluster 2022-04-22T17:49:28.325561+0000 mgr.y (mgr.34107) 853 : cluster [DBG] pgmap v1483: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:49:30.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 820 completed, throughput=4.98MB/sec 2022-04-22T17:49:30.299 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ktSCm8nrAtgBf8P off=5521353 len=827241 2022-04-22T17:49:30.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 821 completed, throughput=4.99MB/sec 2022-04-22T17:49:30.300 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-09kS1On3ZfhRpo7 off=1795303 len=687705 2022-04-22T17:49:30.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 822 completed, throughput=4.99MB/sec 2022-04-22T17:49:30.302 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7P76hooiEeFtwep off=306251 len=1112115 2022-04-22T17:49:30.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 823 completed, throughput=5MB/sec 2022-04-22T17:49:30.308 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XxnY9wIkmKSu8k5 off=5815348 len=30759 2022-04-22T17:49:30.309 INFO:tasks.workunit.client.0.smithi115.stdout:op 824 completed, throughput=5MB/sec 2022-04-22T17:49:30.309 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WSxzzhpm6og9nyR off=2530302 len=4078 2022-04-22T17:49:30.310 INFO:tasks.workunit.client.0.smithi115.stdout:op 825 completed, throughput=5MB/sec 2022-04-22T17:49:30.310 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BE-3QbCCqAPDOz9 off=15039978 len=1235081 2022-04-22T17:49:30.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 826 completed, throughput=5.01MB/sec 2022-04-22T17:49:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:30 smithi121 bash[36902]: audit 2022-04-22T17:49:29.872163+0000 mon.a (mon.0) 2254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:31.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:30 smithi121 bash[36902]: audit 2022-04-22T17:49:30.167132+0000 mon.a (mon.0) 2255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:31.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:30 smithi115 bash[45495]: audit 2022-04-22T17:49:29.872163+0000 mon.a (mon.0) 2254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:31.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:30 smithi115 bash[45495]: audit 2022-04-22T17:49:30.167132+0000 mon.a (mon.0) 2255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:30 smithi115 bash[46934]: audit 2022-04-22T17:49:29.872163+0000 mon.a (mon.0) 2254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:30 smithi115 bash[46934]: audit 2022-04-22T17:49:30.167132+0000 mon.a (mon.0) 2255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout: 163: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Myq3wlWrXim1Tb7 off=466702 len=187843 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 827 completed, throughput=4.98MB/sec 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hTvl0KTNhf0288C off=1766296 len=1608946 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 828 completed, throughput=4.99MB/sec 2022-04-22T17:49:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r2Xoyu2kC0JP-yh off=3192337 len=1502503 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 829 completed, throughput=4.99MB/sec 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RgFHs34oFxcTYGf off=2655811 len=77075 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 830 completed, throughput=4.99MB/sec 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pmxTYLulUxaSsbn off=9374252 len=252513 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 831 completed, throughput=5MB/sec 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P2DbCUd9qGKawIK off=732553 len=1087851 2022-04-22T17:49:31.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 832 completed, throughput=5MB/sec 2022-04-22T17:49:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:31 smithi115 bash[45495]: cluster 2022-04-22T17:49:30.326628+0000 mgr.y (mgr.34107) 854 : cluster [DBG] pgmap v1484: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:32.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:31.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:31 smithi115 bash[46934]: cluster 2022-04-22T17:49:30.326628+0000 mgr.y (mgr.34107) 854 : cluster [DBG] pgmap v1484: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:31 smithi121 bash[36902]: cluster 2022-04-22T17:49:30.326628+0000 mgr.y (mgr.34107) 854 : cluster [DBG] pgmap v1484: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:49:32.336 INFO:tasks.workunit.client.0.smithi115.stdout: 164: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:32.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yo7MBT1SCxv-5Ya off=2146385 len=992682 2022-04-22T17:49:32.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 833 completed, throughput=4.98MB/sec 2022-04-22T17:49:32.344 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qNNkLTB6RoLNyoP off=3249627 len=128980 2022-04-22T17:49:32.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 834 completed, throughput=4.98MB/sec 2022-04-22T17:49:32.348 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H7BavmFMNnK29JJ off=531331 len=603988 2022-04-22T17:49:32.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 835 completed, throughput=4.98MB/sec 2022-04-22T17:49:32.353 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F1i-tGaHP5opjeG off=3175328 len=464979 2022-04-22T17:49:32.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 836 completed, throughput=4.98MB/sec 2022-04-22T17:49:32.354 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Aw_SQPPWegvFW_j off=1396262 len=422816 2022-04-22T17:49:32.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 837 completed, throughput=4.99MB/sec 2022-04-22T17:49:32.355 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-piBhWCPc5EImMKq off=7385807 len=1991794 2022-04-22T17:49:32.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 838 completed, throughput=5MB/sec 2022-04-22T17:49:32.359 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z2uUMbo2OO6MZWk off=2230746 len=206562 2022-04-22T17:49:32.359 INFO:tasks.workunit.client.0.smithi115.stdout:op 839 completed, throughput=5MB/sec 2022-04-22T17:49:32.359 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z9eT8J1vhOe7v21 off=4416838 len=1449674 2022-04-22T17:49:32.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 840 completed, throughput=5.01MB/sec 2022-04-22T17:49:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:32 smithi121 bash[36902]: audit 2022-04-22T17:49:32.252441+0000 mon.a (mon.0) 2256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:33.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:32 smithi115 bash[45495]: audit 2022-04-22T17:49:32.252441+0000 mon.a (mon.0) 2256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:32 smithi115 bash[46934]: audit 2022-04-22T17:49:32.252441+0000 mon.a (mon.0) 2256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:33.367 INFO:tasks.workunit.client.0.smithi115.stdout: 165: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:33.378 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d8RMOuCSUjzMMP9 off=14280892 len=1627239 2022-04-22T17:49:33.378 INFO:tasks.workunit.client.0.smithi115.stdout:op 841 completed, throughput=4.99MB/sec 2022-04-22T17:49:33.379 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZkHLwQpv_TI4NW9 off=524747 len=1594841 2022-04-22T17:49:33.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 842 completed, throughput=4.99MB/sec 2022-04-22T17:49:33.379 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FgR-Axf2LI5ECcp off=9028643 len=1485933 2022-04-22T17:49:33.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 843 completed, throughput=5MB/sec 2022-04-22T17:49:33.847 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:33.562Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:33 smithi121 bash[36902]: cluster 2022-04-22T17:49:32.327697+0000 mgr.y (mgr.34107) 855 : cluster [DBG] pgmap v1485: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:33 smithi115 bash[45495]: cluster 2022-04-22T17:49:32.327697+0000 mgr.y (mgr.34107) 855 : cluster [DBG] pgmap v1485: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:33 smithi115 bash[46934]: cluster 2022-04-22T17:49:32.327697+0000 mgr.y (mgr.34107) 855 : cluster [DBG] pgmap v1485: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:34.385 INFO:tasks.workunit.client.0.smithi115.stdout: 166: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:34.385 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gi51R55HLBJf_iA off=1140654 len=1650357 2022-04-22T17:49:34.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 844 completed, throughput=4.98MB/sec 2022-04-22T17:49:34.390 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BSF3ZElfmqlnNsV off=14471852 len=118267 2022-04-22T17:49:34.394 INFO:tasks.workunit.client.0.smithi115.stdout:op 845 completed, throughput=4.98MB/sec 2022-04-22T17:49:34.395 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UnATYe1yZ6ovhqZ off=686315 len=1929735 2022-04-22T17:49:34.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 846 completed, throughput=4.99MB/sec 2022-04-22T17:49:34.399 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e8fyy_4e5za1-bQ off=2642539 len=131230 2022-04-22T17:49:34.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 847 completed, throughput=4.99MB/sec 2022-04-22T17:49:34.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z6jYhqsITMkUQ-J off=8399575 len=861043 2022-04-22T17:49:34.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 848 completed, throughput=5MB/sec 2022-04-22T17:49:34.402 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C2lymRtN5QDMeGz off=8978380 len=1810257 2022-04-22T17:49:34.412 INFO:tasks.workunit.client.0.smithi115.stdout:op 849 completed, throughput=5.01MB/sec 2022-04-22T17:49:35.413 INFO:tasks.workunit.client.0.smithi115.stdout: 167: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:35.413 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WkhAkA1v_XiZGhO off=5185147 len=1482988 2022-04-22T17:49:35.417 INFO:tasks.workunit.client.0.smithi115.stdout:op 850 completed, throughput=4.99MB/sec 2022-04-22T17:49:35.418 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lvMJT6hKtxfD7_K off=15511661 len=138490 2022-04-22T17:49:35.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 851 completed, throughput=4.99MB/sec 2022-04-22T17:49:35.422 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NwDY2xutO9UeSdv off=8351396 len=1207679 2022-04-22T17:49:35.425 INFO:tasks.workunit.client.0.smithi115.stdout:op 852 completed, throughput=5MB/sec 2022-04-22T17:49:35.425 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ath60MKsNPCDBOd off=1577661 len=2027310 2022-04-22T17:49:35.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 853 completed, throughput=5.01MB/sec 2022-04-22T17:49:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:35 smithi121 bash[36902]: cluster 2022-04-22T17:49:34.328664+0000 mgr.y (mgr.34107) 856 : cluster [DBG] pgmap v1486: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:35 smithi121 bash[36902]: audit 2022-04-22T17:49:34.874993+0000 mon.a (mon.0) 2257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:35 smithi121 bash[36902]: audit 2022-04-22T17:49:35.170624+0000 mon.a (mon.0) 2258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[45495]: cluster 2022-04-22T17:49:34.328664+0000 mgr.y (mgr.34107) 856 : cluster [DBG] pgmap v1486: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[45495]: audit 2022-04-22T17:49:34.874993+0000 mon.a (mon.0) 2257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[45495]: audit 2022-04-22T17:49:35.170624+0000 mon.a (mon.0) 2258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[46934]: cluster 2022-04-22T17:49:34.328664+0000 mgr.y (mgr.34107) 856 : cluster [DBG] pgmap v1486: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[46934]: audit 2022-04-22T17:49:34.874993+0000 mon.a (mon.0) 2257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:35 smithi115 bash[46934]: audit 2022-04-22T17:49:35.170624+0000 mon.a (mon.0) 2258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:36.430 INFO:tasks.workunit.client.0.smithi115.stdout: 168: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:36.430 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KMhgIqzHqRSEfCx off=12272133 len=645449 2022-04-22T17:49:36.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 854 completed, throughput=4.98MB/sec 2022-04-22T17:49:36.436 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0nn0ZnKFa4dE_pA off=3713955 len=93100 2022-04-22T17:49:36.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 855 completed, throughput=4.98MB/sec 2022-04-22T17:49:36.437 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wmSV4B8IHzGhgRs off=7676469 len=410968 2022-04-22T17:49:36.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 856 completed, throughput=4.98MB/sec 2022-04-22T17:49:36.439 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GiGX5-pyh6o1hyL off=3564036 len=2096828 2022-04-22T17:49:36.444 INFO:tasks.workunit.client.0.smithi115.stdout:op 857 completed, throughput=4.99MB/sec 2022-04-22T17:49:36.444 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R1FefXEjwHP9Wd_ off=1796698 len=944880 2022-04-22T17:49:36.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 858 completed, throughput=5MB/sec 2022-04-22T17:49:36.446 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9NkFfAaRVg2m111 off=10061347 len=495717 2022-04-22T17:49:36.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 859 completed, throughput=5MB/sec 2022-04-22T17:49:37.451 INFO:tasks.workunit.client.0.smithi115.stdout: 169: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:37.452 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JumbOEurRnfauKr off=1536195 len=832896 2022-04-22T17:49:37.457 INFO:tasks.workunit.client.0.smithi115.stdout:op 860 completed, throughput=4.98MB/sec 2022-04-22T17:49:37.457 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v2ANI4XXWKDOxxb off=771671 len=644906 2022-04-22T17:49:37.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 861 completed, throughput=4.98MB/sec 2022-04-22T17:49:37.463 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lxWx8QtQM2Hn5bh off=11597991 len=13457 2022-04-22T17:49:37.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 862 completed, throughput=4.98MB/sec 2022-04-22T17:49:37.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bx_vKBO_RItmKtK off=0 len=1493161 2022-04-22T17:49:37.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 863 completed, throughput=4.99MB/sec 2022-04-22T17:49:37.467 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JokCgcjduh3vOh- off=3585805 len=1124149 2022-04-22T17:49:37.474 INFO:tasks.workunit.client.0.smithi115.stdout:op 864 completed, throughput=5MB/sec 2022-04-22T17:49:37.474 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PnbPoOXnsQEHHvd off=4626042 len=176623 2022-04-22T17:49:37.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 865 completed, throughput=5MB/sec 2022-04-22T17:49:37.475 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rxwhPUV0HGEe6iC off=3296951 len=663747 2022-04-22T17:49:37.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 866 completed, throughput=5MB/sec 2022-04-22T17:49:37.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:37 smithi121 bash[36902]: cluster 2022-04-22T17:49:36.329695+0000 mgr.y (mgr.34107) 857 : cluster [DBG] pgmap v1487: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:37.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:37 smithi121 bash[36902]: audit 2022-04-22T17:49:37.255576+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:37.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:37 smithi121 bash[36902]: audit 2022-04-22T17:49:37.618790+0000 mon.a (mon.0) 2260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:49:37.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:37 smithi121 bash[36902]: audit 2022-04-22T17:49:37.621780+0000 mon.a (mon.0) 2261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:49:37.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:37 smithi121 bash[36902]: audit 2022-04-22T17:49:37.623496+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:49:37.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[46934]: cluster 2022-04-22T17:49:36.329695+0000 mgr.y (mgr.34107) 857 : cluster [DBG] pgmap v1487: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:37.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[46934]: audit 2022-04-22T17:49:37.255576+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:37.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[46934]: audit 2022-04-22T17:49:37.618790+0000 mon.a (mon.0) 2260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:49:37.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[46934]: audit 2022-04-22T17:49:37.621780+0000 mon.a (mon.0) 2261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:49:37.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[46934]: audit 2022-04-22T17:49:37.623496+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:49:37.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[45495]: cluster 2022-04-22T17:49:36.329695+0000 mgr.y (mgr.34107) 857 : cluster [DBG] pgmap v1487: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:49:37.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[45495]: audit 2022-04-22T17:49:37.255576+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:37.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[45495]: audit 2022-04-22T17:49:37.618790+0000 mon.a (mon.0) 2260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:49:37.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[45495]: audit 2022-04-22T17:49:37.621780+0000 mon.a (mon.0) 2261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:49:37.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:37 smithi115 bash[45495]: audit 2022-04-22T17:49:37.623496+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:49:38.477 INFO:tasks.workunit.client.0.smithi115.stdout: 170: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:38.477 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PTj0iCgr6dB8XsY off=2697492 len=1881233 2022-04-22T17:49:38.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 867 completed, throughput=4.98MB/sec 2022-04-22T17:49:38.486 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CPzPnOQO5ZWM8LQ off=13679482 len=2023770 2022-04-22T17:49:38.497 INFO:tasks.workunit.client.0.smithi115.stdout:op 868 completed, throughput=4.99MB/sec 2022-04-22T17:49:38.497 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--WeXCHgdjDXAdim off=5868187 len=855537 2022-04-22T17:49:38.499 INFO:tasks.workunit.client.0.smithi115.stdout:op 869 completed, throughput=5MB/sec 2022-04-22T17:49:38.499 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4SdsnVAbMbrl5tj off=12338731 len=1542399 2022-04-22T17:49:38.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 870 completed, throughput=5.01MB/sec 2022-04-22T17:49:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:39 smithi121 bash[36902]: audit 2022-04-22T17:49:37.990565+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:49:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:38 smithi115 bash[45495]: audit 2022-04-22T17:49:37.990565+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:49:39.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:38 smithi115 bash[46934]: audit 2022-04-22T17:49:37.990565+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:49:39.506 INFO:tasks.workunit.client.0.smithi115.stdout: 171: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:39.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P8rGDQzjh9gAwHY off=11949100 len=1480639 2022-04-22T17:49:39.515 INFO:tasks.workunit.client.0.smithi115.stdout:op 871 completed, throughput=4.98MB/sec 2022-04-22T17:49:39.515 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JoEYM5hLDXfaSQC off=561455 len=1164714 2022-04-22T17:49:39.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 872 completed, throughput=4.99MB/sec 2022-04-22T17:49:39.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8ONw0nGV-kfe4My off=12267323 len=1410874 2022-04-22T17:49:39.519 INFO:tasks.workunit.client.0.smithi115.stdout:op 873 completed, throughput=5MB/sec 2022-04-22T17:49:39.519 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MrVqtpEX1prcgAx off=6181986 len=1213033 2022-04-22T17:49:39.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 874 completed, throughput=5MB/sec 2022-04-22T17:49:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:40 smithi121 bash[36902]: cluster 2022-04-22T17:49:38.330780+0000 mgr.y (mgr.34107) 858 : cluster [DBG] pgmap v1488: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:40 smithi121 bash[36902]: audit 2022-04-22T17:49:39.878282+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:39 smithi115 bash[45495]: cluster 2022-04-22T17:49:38.330780+0000 mgr.y (mgr.34107) 858 : cluster [DBG] pgmap v1488: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:39 smithi115 bash[45495]: audit 2022-04-22T17:49:39.878282+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:39 smithi115 bash[46934]: cluster 2022-04-22T17:49:38.330780+0000 mgr.y (mgr.34107) 858 : cluster [DBG] pgmap v1488: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:49:40.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:39 smithi115 bash[46934]: audit 2022-04-22T17:49:39.878282+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:40.525 INFO:tasks.workunit.client.0.smithi115.stdout: 172: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:40.552 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FON-NM9yf9KI8SJ off=5412392 len=2084798 2022-04-22T17:49:40.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 875 completed, throughput=4.99MB/sec 2022-04-22T17:49:40.554 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mgTYBU6ouQiFOML off=10032599 len=601280 2022-04-22T17:49:40.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 876 completed, throughput=4.99MB/sec 2022-04-22T17:49:40.554 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gBQ7s8SbuSMaUHM off=3462750 len=707720 2022-04-22T17:49:40.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 877 completed, throughput=4.99MB/sec 2022-04-22T17:49:40.555 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sbwRkNG4xT3uT2w off=14794068 len=1482602 2022-04-22T17:49:40.555 INFO:tasks.workunit.client.0.smithi115.stdout:op 878 completed, throughput=5MB/sec 2022-04-22T17:49:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:41 smithi121 bash[36902]: audit 2022-04-22T17:49:40.174426+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:41 smithi115 bash[46934]: audit 2022-04-22T17:49:40.174426+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:41.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:41 smithi115 bash[45495]: audit 2022-04-22T17:49:40.174426+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:41.546 INFO:tasks.workunit.client.0.smithi115.stdout: 173: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:41.546 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6cF6fybE3LyFGIe off=245212 len=327325 2022-04-22T17:49:41.547 INFO:tasks.workunit.client.0.smithi115.stdout:op 879 completed, throughput=4.98MB/sec 2022-04-22T17:49:41.547 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-STxURAEyhsp9P6e off=169647 len=98852 2022-04-22T17:49:41.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 880 completed, throughput=4.98MB/sec 2022-04-22T17:49:41.549 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vlgsC0OfG6RWG-P off=12572320 len=1528187 2022-04-22T17:49:41.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 881 completed, throughput=4.98MB/sec 2022-04-22T17:49:41.556 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-55UOXdVgnxGUbgU off=4588314 len=842562 2022-04-22T17:49:41.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 882 completed, throughput=4.99MB/sec 2022-04-22T17:49:41.558 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y1QWtXHE_h2FQ6i off=357357 len=485551 2022-04-22T17:49:41.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 883 completed, throughput=4.99MB/sec 2022-04-22T17:49:41.559 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G-U9dsdNyYaUQ9d off=923399 len=1907829 2022-04-22T17:49:41.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 884 completed, throughput=5MB/sec 2022-04-22T17:49:42.017 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:41.680Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:42.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:42 smithi115 bash[45495]: cluster 2022-04-22T17:49:40.331979+0000 mgr.y (mgr.34107) 859 : cluster [DBG] pgmap v1489: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:42 smithi115 bash[46934]: cluster 2022-04-22T17:49:40.331979+0000 mgr.y (mgr.34107) 859 : cluster [DBG] pgmap v1489: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:42 smithi121 bash[36902]: cluster 2022-04-22T17:49:40.331979+0000 mgr.y (mgr.34107) 859 : cluster [DBG] pgmap v1489: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:49:42.565 INFO:tasks.workunit.client.0.smithi115.stdout: 174: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:42.566 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8nbt4tH94THPh8f off=8681480 len=795450 2022-04-22T17:49:42.569 INFO:tasks.workunit.client.0.smithi115.stdout:op 885 completed, throughput=4.98MB/sec 2022-04-22T17:49:42.569 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9_1OoCH4Jt9OEv7 off=5539958 len=19544 2022-04-22T17:49:42.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 886 completed, throughput=4.98MB/sec 2022-04-22T17:49:42.573 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_pCWCi1imtQjeZA off=8688044 len=974963 2022-04-22T17:49:42.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 887 completed, throughput=4.98MB/sec 2022-04-22T17:49:42.579 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UikSjVrHLqkb-2v off=6058412 len=206142 2022-04-22T17:49:42.580 INFO:tasks.workunit.client.0.smithi115.stdout:op 888 completed, throughput=4.98MB/sec 2022-04-22T17:49:42.580 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7hiiVOZauLSAjVW off=1908111 len=75829 2022-04-22T17:49:42.581 INFO:tasks.workunit.client.0.smithi115.stdout:op 889 completed, throughput=4.98MB/sec 2022-04-22T17:49:42.581 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lwG2Um2XPqydWDk off=19669694 len=903642 2022-04-22T17:49:42.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 890 completed, throughput=4.99MB/sec 2022-04-22T17:49:42.583 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_9n7D6RvabRIoTS off=804496 len=1562294 2022-04-22T17:49:42.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 891 completed, throughput=5MB/sec 2022-04-22T17:49:42.587 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-J9aiJDAivhEVQgW off=11026661 len=1696762 2022-04-22T17:49:42.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 892 completed, throughput=5.01MB/sec 2022-04-22T17:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:43 smithi115 bash[45495]: audit 2022-04-22T17:49:42.258591+0000 mon.a (mon.0) 2266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:43 smithi115 bash[46934]: audit 2022-04-22T17:49:42.258591+0000 mon.a (mon.0) 2266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:43 smithi121 bash[36902]: audit 2022-04-22T17:49:42.258591+0000 mon.a (mon.0) 2266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:43.594 INFO:tasks.workunit.client.0.smithi115.stdout: 175: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:43.594 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r_mVyMtZQBqf7Ni off=8836197 len=1384242 2022-04-22T17:49:43.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 893 completed, throughput=4.98MB/sec 2022-04-22T17:49:43.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RYTedAa_V763Vx6 off=7613819 len=1710287 2022-04-22T17:49:43.606 INFO:tasks.workunit.client.0.smithi115.stdout:op 894 completed, throughput=4.99MB/sec 2022-04-22T17:49:43.606 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mXOk-BScOwIQ84y off=16739204 len=1499226 2022-04-22T17:49:43.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 895 completed, throughput=5MB/sec 2022-04-22T17:49:44.026 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:43.563Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:44.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:44 smithi115 bash[45495]: cluster 2022-04-22T17:49:42.333064+0000 mgr.y (mgr.34107) 860 : cluster [DBG] pgmap v1490: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:44 smithi115 bash[46934]: cluster 2022-04-22T17:49:42.333064+0000 mgr.y (mgr.34107) 860 : cluster [DBG] pgmap v1490: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:44 smithi121 bash[36902]: cluster 2022-04-22T17:49:42.333064+0000 mgr.y (mgr.34107) 860 : cluster [DBG] pgmap v1490: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:49:44.612 INFO:tasks.workunit.client.0.smithi115.stdout: 176: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:44.612 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f2Ut3c3qiZBPvlD off=5885255 len=989985 2022-04-22T17:49:44.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 896 completed, throughput=4.98MB/sec 2022-04-22T17:49:44.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6kZDl7SpnXXtkcL off=10662005 len=1970096 2022-04-22T17:49:44.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 897 completed, throughput=4.99MB/sec 2022-04-22T17:49:44.622 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YrE_KMiCBiHA9lB off=3016556 len=1759854 2022-04-22T17:49:44.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 898 completed, throughput=5MB/sec 2022-04-22T17:49:44.625 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hQ7ct3iLxkwR1HZ off=3651533 len=1777140 2022-04-22T17:49:44.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 899 completed, throughput=5.01MB/sec 2022-04-22T17:49:45.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:45 smithi115 bash[45495]: audit 2022-04-22T17:49:44.881488+0000 mon.a (mon.0) 2267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:45 smithi115 bash[46934]: audit 2022-04-22T17:49:44.881488+0000 mon.a (mon.0) 2267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:45 smithi121 bash[36902]: audit 2022-04-22T17:49:44.881488+0000 mon.a (mon.0) 2267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:45.629 INFO:tasks.workunit.client.0.smithi115.stdout: 177: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:45.662 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cA8zzZOjosv7rjS off=9452310 len=1338348 2022-04-22T17:49:45.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 900 completed, throughput=4.99MB/sec 2022-04-22T17:49:45.664 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fAH5XI8U4td69wL off=19745134 len=845001 2022-04-22T17:49:45.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 901 completed, throughput=4.99MB/sec 2022-04-22T17:49:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uDggGJPM6F_fHmf off=595302 len=137844 2022-04-22T17:49:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 902 completed, throughput=4.99MB/sec 2022-04-22T17:49:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JYbhgoGhfYf1FQO off=1576112 len=1690125 2022-04-22T17:49:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 903 completed, throughput=5MB/sec 2022-04-22T17:49:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:46 smithi115 bash[45495]: cluster 2022-04-22T17:49:44.334131+0000 mgr.y (mgr.34107) 861 : cluster [DBG] pgmap v1491: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:46 smithi115 bash[45495]: audit 2022-04-22T17:49:45.178152+0000 mon.a (mon.0) 2268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:46 smithi115 bash[46934]: cluster 2022-04-22T17:49:44.334131+0000 mgr.y (mgr.34107) 861 : cluster [DBG] pgmap v1491: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:46 smithi115 bash[46934]: audit 2022-04-22T17:49:45.178152+0000 mon.a (mon.0) 2268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:46 smithi121 bash[36902]: cluster 2022-04-22T17:49:44.334131+0000 mgr.y (mgr.34107) 861 : cluster [DBG] pgmap v1491: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:49:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:46 smithi121 bash[36902]: audit 2022-04-22T17:49:45.178152+0000 mon.a (mon.0) 2268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:46.648 INFO:tasks.workunit.client.0.smithi115.stdout: 178: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:46.649 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WoeULpnl2CX_FAI off=4924405 len=1620673 2022-04-22T17:49:46.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 904 completed, throughput=4.98MB/sec 2022-04-22T17:49:46.658 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v6ViSnllXSP08fO off=2039663 len=1698647 2022-04-22T17:49:46.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 905 completed, throughput=4.99MB/sec 2022-04-22T17:49:46.661 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w9BEEOu_Unj8lV6 off=3476893 len=240652 2022-04-22T17:49:46.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 906 completed, throughput=4.99MB/sec 2022-04-22T17:49:46.662 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_ulC7lT-MsmpJb7 off=6699061 len=1279310 2022-04-22T17:49:46.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 907 completed, throughput=5MB/sec 2022-04-22T17:49:46.669 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jqcpqmk6AHT8NB8 off=14479460 len=869601 2022-04-22T17:49:46.672 INFO:tasks.workunit.client.0.smithi115.stdout:op 908 completed, throughput=5MB/sec 2022-04-22T17:49:47.672 INFO:tasks.workunit.client.0.smithi115.stdout: 179: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:47.672 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i7bE2F0RNdP-G2K off=573387 len=805609 2022-04-22T17:49:47.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 909 completed, throughput=4.98MB/sec 2022-04-22T17:49:47.676 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-a_xdTHoRPtbSTmF off=3676808 len=627963 2022-04-22T17:49:47.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 910 completed, throughput=4.98MB/sec 2022-04-22T17:49:47.681 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YUq_DkvSNF615jJ off=1822584 len=1651376 2022-04-22T17:49:47.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 911 completed, throughput=4.99MB/sec 2022-04-22T17:49:47.692 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_CdLqo4sKW-CJfQ off=12233532 len=888012 2022-04-22T17:49:47.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 912 completed, throughput=5MB/sec 2022-04-22T17:49:47.695 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LIK4fd25oIE5uSB off=714434 len=1039563 2022-04-22T17:49:47.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 913 completed, throughput=5MB/sec 2022-04-22T17:49:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:48 smithi115 bash[45495]: cluster 2022-04-22T17:49:46.335216+0000 mgr.y (mgr.34107) 862 : cluster [DBG] pgmap v1492: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:48 smithi115 bash[45495]: audit 2022-04-22T17:49:47.261592+0000 mon.a (mon.0) 2269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:48.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:48 smithi115 bash[46934]: cluster 2022-04-22T17:49:46.335216+0000 mgr.y (mgr.34107) 862 : cluster [DBG] pgmap v1492: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:48.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:48 smithi115 bash[46934]: audit 2022-04-22T17:49:47.261592+0000 mon.a (mon.0) 2269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:48 smithi121 bash[36902]: cluster 2022-04-22T17:49:46.335216+0000 mgr.y (mgr.34107) 862 : cluster [DBG] pgmap v1492: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:48 smithi121 bash[36902]: audit 2022-04-22T17:49:47.261592+0000 mon.a (mon.0) 2269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:48.701 INFO:tasks.workunit.client.0.smithi115.stdout: 180: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:48.701 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wzxS3_gXvJhTOtu off=4306503 len=526114 2022-04-22T17:49:48.703 INFO:tasks.workunit.client.0.smithi115.stdout:op 914 completed, throughput=4.98MB/sec 2022-04-22T17:49:48.703 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-btZkz8klGj9dLlX off=166507 len=598725 2022-04-22T17:49:48.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 915 completed, throughput=4.98MB/sec 2022-04-22T17:49:48.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pfa5-UKV1qJizYR off=653538 len=236488 2022-04-22T17:49:48.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 916 completed, throughput=4.98MB/sec 2022-04-22T17:49:48.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hBqcnuCxHjWQzW0 off=520454 len=433140 2022-04-22T17:49:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 917 completed, throughput=4.98MB/sec 2022-04-22T17:49:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fk5E4lPAe2Elwso off=732309 len=710653 2022-04-22T17:49:48.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 918 completed, throughput=4.99MB/sec 2022-04-22T17:49:48.711 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ne_optWWl15Qpiz off=6074864 len=993547 2022-04-22T17:49:48.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 919 completed, throughput=4.99MB/sec 2022-04-22T17:49:48.715 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zHh6mZaAfpW75sv off=2563588 len=243133 2022-04-22T17:49:48.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 920 completed, throughput=4.99MB/sec 2022-04-22T17:49:48.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNIsKmv_6cDWE3e off=215974 len=1262292 2022-04-22T17:49:48.722 INFO:tasks.workunit.client.0.smithi115.stdout:op 921 completed, throughput=5MB/sec 2022-04-22T17:49:48.723 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_jKphHSenPZT8Rm off=8391108 len=786346 2022-04-22T17:49:48.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 922 completed, throughput=5MB/sec 2022-04-22T17:49:49.728 INFO:tasks.workunit.client.0.smithi115.stdout: 181: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:49.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hoslt3Bgpd9pfzq off=19105917 len=1440338 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 923 completed, throughput=4.98MB/sec 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cFbI3GKBnNBGk8I off=9554517 len=635969 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 924 completed, throughput=4.99MB/sec 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Son1K9U-b9NwVEz off=760105 len=1545917 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 925 completed, throughput=4.99MB/sec 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9F4QcUcEIqrNtgE off=4002986 len=819795 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 926 completed, throughput=5MB/sec 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JpGWWvqhY9WvBiq off=12724759 len=693140 2022-04-22T17:49:49.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 927 completed, throughput=5MB/sec 2022-04-22T17:49:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:50 smithi115 bash[45495]: cluster 2022-04-22T17:49:48.336356+0000 mgr.y (mgr.34107) 863 : cluster [DBG] pgmap v1493: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:49:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:50 smithi115 bash[45495]: audit 2022-04-22T17:49:49.884621+0000 mon.a (mon.0) 2270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:50 smithi115 bash[46934]: cluster 2022-04-22T17:49:48.336356+0000 mgr.y (mgr.34107) 863 : cluster [DBG] pgmap v1493: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:49:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:50 smithi115 bash[46934]: audit 2022-04-22T17:49:49.884621+0000 mon.a (mon.0) 2270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:50 smithi121 bash[36902]: cluster 2022-04-22T17:49:48.336356+0000 mgr.y (mgr.34107) 863 : cluster [DBG] pgmap v1493: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:49:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:50 smithi121 bash[36902]: audit 2022-04-22T17:49:49.884621+0000 mon.a (mon.0) 2270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:50.753 INFO:tasks.workunit.client.0.smithi115.stdout: 182: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:50.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yfPivhkb_DodkBu off=7929908 len=37300 2022-04-22T17:49:50.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 928 completed, throughput=4.97MB/sec 2022-04-22T17:49:50.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mrZIw5rhT7YZiQq off=907610 len=44896 2022-04-22T17:49:50.755 INFO:tasks.workunit.client.0.smithi115.stdout:op 929 completed, throughput=4.97MB/sec 2022-04-22T17:49:50.755 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rDmYxB_3SIWhhqi off=717341 len=551806 2022-04-22T17:49:50.761 INFO:tasks.workunit.client.0.smithi115.stdout:op 930 completed, throughput=4.98MB/sec 2022-04-22T17:49:50.761 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-USE4Um1tZjf8oqk off=1889993 len=149148 2022-04-22T17:49:50.765 INFO:tasks.workunit.client.0.smithi115.stdout:op 931 completed, throughput=4.98MB/sec 2022-04-22T17:49:50.765 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XjX1sAiZbTiVHyO off=2995942 len=112094 2022-04-22T17:49:50.766 INFO:tasks.workunit.client.0.smithi115.stdout:op 932 completed, throughput=4.98MB/sec 2022-04-22T17:49:50.766 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3PxjekE18uvwlaX off=8000386 len=582064 2022-04-22T17:49:50.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 933 completed, throughput=4.98MB/sec 2022-04-22T17:49:50.767 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sEtAcanMle-v4jc off=3214551 len=1294882 2022-04-22T17:49:50.769 INFO:tasks.workunit.client.0.smithi115.stdout:op 934 completed, throughput=4.99MB/sec 2022-04-22T17:49:50.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oL56IjzJ03NeOb7 off=9554657 len=1890874 2022-04-22T17:49:50.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 935 completed, throughput=5MB/sec 2022-04-22T17:49:50.777 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UiFbVcXr6craN0V off=0 len=1617681 2022-04-22T17:49:50.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 936 completed, throughput=5.01MB/sec 2022-04-22T17:49:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:51 smithi121 bash[36902]: audit 2022-04-22T17:49:50.181897+0000 mon.a (mon.0) 2271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:51 smithi115 bash[45495]: audit 2022-04-22T17:49:50.181897+0000 mon.a (mon.0) 2271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:51.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:51 smithi115 bash[46934]: audit 2022-04-22T17:49:50.181897+0000 mon.a (mon.0) 2271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:51.785 INFO:tasks.workunit.client.0.smithi115.stdout: 183: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:51.786 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nSa67ed8OAUQlYR off=10352240 len=32163 2022-04-22T17:49:51.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 937 completed, throughput=4.98MB/sec 2022-04-22T17:49:51.786 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pVKCD4Ub2T_49Y3 off=7488421 len=1130317 2022-04-22T17:49:51.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 938 completed, throughput=4.98MB/sec 2022-04-22T17:49:51.793 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SAk_P87Ex1qur2e off=8807184 len=382650 2022-04-22T17:49:51.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 939 completed, throughput=4.99MB/sec 2022-04-22T17:49:51.794 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-acbdzDAsPxty3-2 off=0 len=1433367 2022-04-22T17:49:51.802 INFO:tasks.workunit.client.0.smithi115.stdout:op 940 completed, throughput=4.99MB/sec 2022-04-22T17:49:51.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KajZbr6AMIPrJSa off=5594936 len=1351951 2022-04-22T17:49:51.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 941 completed, throughput=5MB/sec 2022-04-22T17:49:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:51.679Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:52 smithi121 bash[36902]: cluster 2022-04-22T17:49:50.337472+0000 mgr.y (mgr.34107) 864 : cluster [DBG] pgmap v1494: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:49:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:52 smithi115 bash[45495]: cluster 2022-04-22T17:49:50.337472+0000 mgr.y (mgr.34107) 864 : cluster [DBG] pgmap v1494: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:49:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:52 smithi115 bash[46934]: cluster 2022-04-22T17:49:50.337472+0000 mgr.y (mgr.34107) 864 : cluster [DBG] pgmap v1494: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:49:52.808 INFO:tasks.workunit.client.0.smithi115.stdout: 184: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:52.809 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qgGArOtjMAPL2Zj off=6720609 len=746865 2022-04-22T17:49:52.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 942 completed, throughput=4.98MB/sec 2022-04-22T17:49:52.811 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8NCzwdv_21UJwtG off=8082358 len=1393228 2022-04-22T17:49:52.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 943 completed, throughput=4.98MB/sec 2022-04-22T17:49:52.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1oW3W7G_0Tak65u off=2746231 len=1425694 2022-04-22T17:49:52.827 INFO:tasks.workunit.client.0.smithi115.stdout:op 944 completed, throughput=4.99MB/sec 2022-04-22T17:49:52.828 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I5nTiGu9yHj1S1O off=11160462 len=553227 2022-04-22T17:49:52.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 945 completed, throughput=4.99MB/sec 2022-04-22T17:49:52.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S9IdYWcX_uSOBJl off=8230681 len=1245365 2022-04-22T17:49:52.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 946 completed, throughput=5MB/sec 2022-04-22T17:49:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:53 smithi121 bash[36902]: audit 2022-04-22T17:49:52.264728+0000 mon.a (mon.0) 2272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:53.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:53 smithi115 bash[45495]: audit 2022-04-22T17:49:52.264728+0000 mon.a (mon.0) 2272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:53.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:53 smithi115 bash[46934]: audit 2022-04-22T17:49:52.264728+0000 mon.a (mon.0) 2272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:53.835 INFO:tasks.workunit.client.0.smithi115.stdout: 185: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IZT1oZe1lD_ndxq off=4564358 len=1821203 2022-04-22T17:49:53.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 947 completed, throughput=4.98MB/sec 2022-04-22T17:49:53.844 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e-b4yDh4LCnpYZJ off=7529970 len=328392 2022-04-22T17:49:53.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 948 completed, throughput=4.98MB/sec 2022-04-22T17:49:53.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q8iv6dkYHO3MZxe off=1091640 len=1643016 2022-04-22T17:49:53.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 949 completed, throughput=4.99MB/sec 2022-04-22T17:49:53.855 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vfOtVVeGK2kltNx off=795261 len=2058371 2022-04-22T17:49:53.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 950 completed, throughput=5MB/sec 2022-04-22T17:49:54.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:49:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:49:53.563Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:49:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:54 smithi121 bash[36902]: cluster 2022-04-22T17:49:52.338548+0000 mgr.y (mgr.34107) 865 : cluster [DBG] pgmap v1495: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:49:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:54 smithi115 bash[45495]: cluster 2022-04-22T17:49:52.338548+0000 mgr.y (mgr.34107) 865 : cluster [DBG] pgmap v1495: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:49:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:54 smithi115 bash[46934]: cluster 2022-04-22T17:49:52.338548+0000 mgr.y (mgr.34107) 865 : cluster [DBG] pgmap v1495: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:49:54.863 INFO:tasks.workunit.client.0.smithi115.stdout: 186: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:54.863 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ogPbh9wsWELjJbz off=15097705 len=1967995 2022-04-22T17:49:54.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 951 completed, throughput=4.99MB/sec 2022-04-22T17:49:54.869 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QdFzK3zATnOXSBo off=2564949 len=1752179 2022-04-22T17:49:54.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 952 completed, throughput=4.99MB/sec 2022-04-22T17:49:54.879 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oL56IjzJ03NeOb7 off=6516643 len=1059699 2022-04-22T17:49:54.881 INFO:tasks.workunit.client.0.smithi115.stdout:op 953 completed, throughput=5MB/sec 2022-04-22T17:49:54.881 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fHUnY-gE-blAxxO off=5032491 len=14282 2022-04-22T17:49:54.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 954 completed, throughput=5MB/sec 2022-04-22T17:49:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:55 smithi121 bash[36902]: audit 2022-04-22T17:49:54.888099+0000 mon.a (mon.0) 2273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:55 smithi115 bash[45495]: audit 2022-04-22T17:49:54.888099+0000 mon.a (mon.0) 2273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:55.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:55 smithi115 bash[46934]: audit 2022-04-22T17:49:54.888099+0000 mon.a (mon.0) 2273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:49:55.882 INFO:tasks.workunit.client.0.smithi115.stdout: 187: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:55.883 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_6FGeo5eESbdQq5 off=3407486 len=1606520 2022-04-22T17:49:55.886 INFO:tasks.workunit.client.0.smithi115.stdout:op 955 completed, throughput=4.98MB/sec 2022-04-22T17:49:55.886 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6nypMT4O533_6nW off=114902 len=1306283 2022-04-22T17:49:55.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 956 completed, throughput=4.99MB/sec 2022-04-22T17:49:55.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IYcIPqPtyUbr6xb off=13644038 len=1062213 2022-04-22T17:49:55.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 957 completed, throughput=4.99MB/sec 2022-04-22T17:49:55.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QTc4qXCBZPin0vK off=7887425 len=1777292 2022-04-22T17:49:55.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 958 completed, throughput=5MB/sec 2022-04-22T17:49:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:56 smithi121 bash[36902]: cluster 2022-04-22T17:49:54.339632+0000 mgr.y (mgr.34107) 866 : cluster [DBG] pgmap v1496: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:56 smithi121 bash[36902]: audit 2022-04-22T17:49:55.185508+0000 mon.a (mon.0) 2274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:56.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:56 smithi115 bash[45495]: cluster 2022-04-22T17:49:54.339632+0000 mgr.y (mgr.34107) 866 : cluster [DBG] pgmap v1496: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:56.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:56 smithi115 bash[45495]: audit 2022-04-22T17:49:55.185508+0000 mon.a (mon.0) 2274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:56.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:56 smithi115 bash[46934]: cluster 2022-04-22T17:49:54.339632+0000 mgr.y (mgr.34107) 866 : cluster [DBG] pgmap v1496: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:49:56.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:56 smithi115 bash[46934]: audit 2022-04-22T17:49:55.185508+0000 mon.a (mon.0) 2274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:49:56.900 INFO:tasks.workunit.client.0.smithi115.stdout: 188: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:56.900 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RDbxR-jS0UP5AOK off=1692093 len=1902448 2022-04-22T17:49:56.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 959 completed, throughput=4.99MB/sec 2022-04-22T17:49:56.910 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7ngwppI69i0fvlX off=7935610 len=1536589 2022-04-22T17:49:56.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 960 completed, throughput=4.99MB/sec 2022-04-22T17:49:56.917 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-45o5RDROwipymMo off=7768286 len=1640582 2022-04-22T17:49:56.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 961 completed, throughput=5MB/sec 2022-04-22T17:49:57.926 INFO:tasks.workunit.client.0.smithi115.stdout: 189: throughput=4.97MB/sec pending data=0 2022-04-22T17:49:57.938 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0vHr9ouza5E_7c8 off=742045 len=482570 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 962 completed, throughput=4.98MB/sec 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nOMnXm5fdwUYbsl off=865504 len=2029191 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 963 completed, throughput=4.99MB/sec 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gIZPSK9vUW4-cJZ off=1883646 len=87435 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 964 completed, throughput=4.99MB/sec 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CPzPnOQO5ZWM8LQ off=18304554 len=1180676 2022-04-22T17:49:57.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 965 completed, throughput=4.99MB/sec 2022-04-22T17:49:57.941 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_BgXPuhRnOOoT5U off=277676 len=45075 2022-04-22T17:49:57.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 966 completed, throughput=4.99MB/sec 2022-04-22T17:49:57.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z6Ql9VIus6sFMMU off=245865 len=118820 2022-04-22T17:49:57.947 INFO:tasks.workunit.client.0.smithi115.stdout:op 967 completed, throughput=4.99MB/sec 2022-04-22T17:49:57.947 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7kWWZmsS0twajnC off=630008 len=931678 2022-04-22T17:49:57.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 968 completed, throughput=5MB/sec 2022-04-22T17:49:57.949 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cHQvrlM8FUV2Ksc off=382297 len=828834 2022-04-22T17:49:57.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 969 completed, throughput=5MB/sec 2022-04-22T17:49:58.499 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:49:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:58 smithi121 bash[36902]: cluster 2022-04-22T17:49:56.340687+0000 mgr.y (mgr.34107) 867 : cluster [DBG] pgmap v1497: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:49:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:49:58 smithi121 bash[36902]: audit 2022-04-22T17:49:57.267726+0000 mon.a (mon.0) 2275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:58.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:58 smithi115 bash[46934]: cluster 2022-04-22T17:49:56.340687+0000 mgr.y (mgr.34107) 867 : cluster [DBG] pgmap v1497: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:49:58.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:49:58 smithi115 bash[46934]: audit 2022-04-22T17:49:57.267726+0000 mon.a (mon.0) 2275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:58 smithi115 bash[45495]: cluster 2022-04-22T17:49:56.340687+0000 mgr.y (mgr.34107) 867 : cluster [DBG] pgmap v1497: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:49:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:49:58 smithi115 bash[45495]: audit 2022-04-22T17:49:57.267726+0000 mon.a (mon.0) 2275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (13m) 2m ago 26m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (26m) 3m ago 26m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (15m) 2m ago 27m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (14m) 3m ago 27m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:49:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (22m) 3m ago 32m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (22m) 2m ago 34m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (21m) 2m ago 34m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (21m) 3m ago 32m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (21m) 2m ago 32m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (14m) 2m ago 27m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (14m) 3m ago 27m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (19m) 2m ago 31m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (19m) 2m ago 31m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (18m) 2m ago 30m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:49:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (18m) 2m ago 30m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:49:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (17m) 3m ago 29m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:49:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (16m) 3m ago 29m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:49:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (16m) 3m ago 28m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:49:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (15m) 3m ago 28m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:49:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (13m) 3m ago 27m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:49:58.952 INFO:tasks.workunit.client.0.smithi115.stdout: 190: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:58.952 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CsyRIj1j0hnIMZg off=819245 len=222553 2022-04-22T17:49:58.956 INFO:tasks.workunit.client.0.smithi115.stdout:op 970 completed, throughput=4.98MB/sec 2022-04-22T17:49:58.956 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oK_Mfkw0uFcaAr7 off=8751051 len=1690001 2022-04-22T17:49:58.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 971 completed, throughput=4.99MB/sec 2022-04-22T17:49:58.958 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5TOkNf5YA2yBeoM off=3483612 len=1763639 2022-04-22T17:49:58.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 972 completed, throughput=4.99MB/sec 2022-04-22T17:49:58.967 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-be0YS_a85ExH-nZ off=1248613 len=566538 2022-04-22T17:49:58.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 973 completed, throughput=5MB/sec 2022-04-22T17:49:58.971 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gDHSe1-VjbyKVwY off=3347836 len=1509709 2022-04-22T17:49:58.973 INFO:tasks.workunit.client.0.smithi115.stdout:op 974 completed, throughput=5MB/sec 2022-04-22T17:49:59.247 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:49:59.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:49:59.249 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:49:59.974 INFO:tasks.workunit.client.0.smithi115.stdout: 191: throughput=4.98MB/sec pending data=0 2022-04-22T17:49:59.974 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5XGJj8rFu5kl5oe off=6045821 len=2013377 2022-04-22T17:49:59.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 975 completed, throughput=4.99MB/sec 2022-04-22T17:49:59.984 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-quZV13EIW3x5fz9 off=16433736 len=176797 2022-04-22T17:49:59.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 976 completed, throughput=4.99MB/sec 2022-04-22T17:49:59.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oij225SMrYyl7mL off=11193984 len=1520135 2022-04-22T17:49:59.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 977 completed, throughput=5MB/sec 2022-04-22T17:49:59.993 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F8tRe9Rfqsqx7hB off=5625125 len=256012 2022-04-22T17:49:59.997 INFO:tasks.workunit.client.0.smithi115.stdout:op 978 completed, throughput=5MB/sec 2022-04-22T17:49:59.997 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-647cuitENb13W8J off=1214468 len=522047 2022-04-22T17:50:00.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 979 completed, throughput=5MB/sec 2022-04-22T17:50:00.000 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kL2gRiJY1y8uPK5 off=17000071 len=326531 2022-04-22T17:50:00.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 980 completed, throughput=5MB/sec 2022-04-22T17:50:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: cluster 2022-04-22T17:49:58.341785+0000 mgr.y (mgr.34107) 868 : cluster [DBG] pgmap v1498: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: audit 2022-04-22T17:49:58.474696+0000 mgr.y (mgr.34107) 869 : audit [DBG] from='client.45259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: audit 2022-04-22T17:49:58.811903+0000 mgr.y (mgr.34107) 870 : audit [DBG] from='client.45265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: audit 2022-04-22T17:49:59.237493+0000 mon.a (mon.0) 2276 : audit [DBG] from='client.? 172.21.15.115:0/2707826140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:00.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: audit 2022-04-22T17:49:59.891164+0000 mon.a (mon.0) 2277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:00.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:00 smithi121 bash[36902]: cluster 2022-04-22T17:50:00.000133+0000 mon.a (mon.0) 2278 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:50:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: cluster 2022-04-22T17:49:58.341785+0000 mgr.y (mgr.34107) 868 : cluster [DBG] pgmap v1498: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: audit 2022-04-22T17:49:58.474696+0000 mgr.y (mgr.34107) 869 : audit [DBG] from='client.45259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: audit 2022-04-22T17:49:58.811903+0000 mgr.y (mgr.34107) 870 : audit [DBG] from='client.45265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: audit 2022-04-22T17:49:59.237493+0000 mon.a (mon.0) 2276 : audit [DBG] from='client.? 172.21.15.115:0/2707826140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: audit 2022-04-22T17:49:59.891164+0000 mon.a (mon.0) 2277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[45495]: cluster 2022-04-22T17:50:00.000133+0000 mon.a (mon.0) 2278 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: cluster 2022-04-22T17:49:58.341785+0000 mgr.y (mgr.34107) 868 : cluster [DBG] pgmap v1498: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: audit 2022-04-22T17:49:58.474696+0000 mgr.y (mgr.34107) 869 : audit [DBG] from='client.45259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: audit 2022-04-22T17:49:58.811903+0000 mgr.y (mgr.34107) 870 : audit [DBG] from='client.45265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: audit 2022-04-22T17:49:59.237493+0000 mon.a (mon.0) 2276 : audit [DBG] from='client.? 172.21.15.115:0/2707826140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: audit 2022-04-22T17:49:59.891164+0000 mon.a (mon.0) 2277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:00 smithi115 bash[46934]: cluster 2022-04-22T17:50:00.000133+0000 mon.a (mon.0) 2278 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T17:50:01.001 INFO:tasks.workunit.client.0.smithi115.stdout: 192: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:01.001 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cvAteHF2bSXHh_i off=7409751 len=601745 2022-04-22T17:50:01.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 981 completed, throughput=4.98MB/sec 2022-04-22T17:50:01.004 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MHzxzyYNcSm9gIe off=8638259 len=533898 2022-04-22T17:50:01.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 982 completed, throughput=4.98MB/sec 2022-04-22T17:50:01.006 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cUkBxbWT2hqVzna off=8329703 len=1512785 2022-04-22T17:50:01.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 983 completed, throughput=4.99MB/sec 2022-04-22T17:50:01.014 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kH8AGyD8EKu0jzW off=186005 len=2053638 2022-04-22T17:50:01.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 984 completed, throughput=5MB/sec 2022-04-22T17:50:01.023 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lE8fJtWJcVvBhp5 off=1560117 len=643698 2022-04-22T17:50:01.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 985 completed, throughput=5MB/sec 2022-04-22T17:50:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:01 smithi121 bash[36902]: audit 2022-04-22T17:50:00.189301+0000 mon.a (mon.0) 2279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:01 smithi115 bash[45495]: audit 2022-04-22T17:50:00.189301+0000 mon.a (mon.0) 2279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:01.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:01 smithi115 bash[46934]: audit 2022-04-22T17:50:00.189301+0000 mon.a (mon.0) 2279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:02.030 INFO:tasks.workunit.client.0.smithi115.stdout: 193: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:02.030 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0XD5-Tv-O1IH6CU off=10302382 len=684735 2022-04-22T17:50:02.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 986 completed, throughput=4.98MB/sec 2022-04-22T17:50:02.032 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hcJ8NZm134hc6Hf off=166127 len=20117 2022-04-22T17:50:02.035 INFO:tasks.workunit.client.0.smithi115.stdout:op 987 completed, throughput=4.98MB/sec 2022-04-22T17:50:02.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C3GHd6iDozm9y9- off=2503560 len=872417 2022-04-22T17:50:02.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 988 completed, throughput=4.98MB/sec 2022-04-22T17:50:02.041 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i1pNrEE5fnRk-pW off=9342540 len=529393 2022-04-22T17:50:02.044 INFO:tasks.workunit.client.0.smithi115.stdout:op 989 completed, throughput=4.99MB/sec 2022-04-22T17:50:02.044 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zTRUHOF9YmJijNC off=1315942 len=1903407 2022-04-22T17:50:02.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:01.680Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:02.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 990 completed, throughput=4.99MB/sec 2022-04-22T17:50:02.054 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8l6X-5kKQhyBjaL off=1171901 len=1979916 2022-04-22T17:50:02.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 991 completed, throughput=5MB/sec 2022-04-22T17:50:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:02 smithi121 bash[36902]: cluster 2022-04-22T17:50:00.342839+0000 mgr.y (mgr.34107) 871 : cluster [DBG] pgmap v1499: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:02 smithi115 bash[45495]: cluster 2022-04-22T17:50:00.342839+0000 mgr.y (mgr.34107) 871 : cluster [DBG] pgmap v1499: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:02 smithi115 bash[46934]: cluster 2022-04-22T17:50:00.342839+0000 mgr.y (mgr.34107) 871 : cluster [DBG] pgmap v1499: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:03.063 INFO:tasks.workunit.client.0.smithi115.stdout: 194: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:03.064 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vka1Fongd7LLuTP off=16337408 len=1865310 2022-04-22T17:50:03.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 992 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z3YDeigJYLr5wty off=2052570 len=11682 2022-04-22T17:50:03.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 993 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.069 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tLYPa7KaDkwQ-vo off=683418 len=286770 2022-04-22T17:50:03.073 INFO:tasks.workunit.client.0.smithi115.stdout:op 994 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.073 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z7h-_jvTDX3yzng off=865786 len=98201 2022-04-22T17:50:03.077 INFO:tasks.workunit.client.0.smithi115.stdout:op 995 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.077 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-upVZEaHEbErMVU_ off=4780777 len=749274 2022-04-22T17:50:03.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 996 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.080 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tLYPa7KaDkwQ-vo off=10455964 len=358578 2022-04-22T17:50:03.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 997 completed, throughput=4.99MB/sec 2022-04-22T17:50:03.081 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-axZhN6gjx9cSbkz off=11391237 len=1894755 2022-04-22T17:50:03.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 998 completed, throughput=5MB/sec 2022-04-22T17:50:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:03 smithi121 bash[36902]: audit 2022-04-22T17:50:02.270803+0000 mon.a (mon.0) 2280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:03 smithi115 bash[45495]: audit 2022-04-22T17:50:02.270803+0000 mon.a (mon.0) 2280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:03 smithi115 bash[46934]: audit 2022-04-22T17:50:02.270803+0000 mon.a (mon.0) 2280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:04.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:03.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:04.084 INFO:tasks.workunit.client.0.smithi115.stdout: 195: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:04.085 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dwlCWiUjKeP2hdl off=19049369 len=585226 2022-04-22T17:50:04.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 999 completed, throughput=4.98MB/sec 2022-04-22T17:50:04.088 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y5OagErxq6hLn-k off=5416351 len=1351347 2022-04-22T17:50:04.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 1000 completed, throughput=4.99MB/sec 2022-04-22T17:50:04.091 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RaB7LY0VPQ8i_6l off=7228621 len=381776 2022-04-22T17:50:04.096 INFO:tasks.workunit.client.0.smithi115.stdout:op 1001 completed, throughput=4.99MB/sec 2022-04-22T17:50:04.096 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sZO1tRy5c1koHXR off=3883189 len=1636189 2022-04-22T17:50:04.104 INFO:tasks.workunit.client.0.smithi115.stdout:op 1002 completed, throughput=5MB/sec 2022-04-22T17:50:04.104 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yDAjPdwcS9NTdPi off=1852180 len=1309710 2022-04-22T17:50:04.112 INFO:tasks.workunit.client.0.smithi115.stdout:op 1003 completed, throughput=5MB/sec 2022-04-22T17:50:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:04 smithi121 bash[36902]: cluster 2022-04-22T17:50:02.343930+0000 mgr.y (mgr.34107) 872 : cluster [DBG] pgmap v1500: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:50:04.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:04 smithi115 bash[45495]: cluster 2022-04-22T17:50:02.343930+0000 mgr.y (mgr.34107) 872 : cluster [DBG] pgmap v1500: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:50:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:04 smithi115 bash[46934]: cluster 2022-04-22T17:50:02.343930+0000 mgr.y (mgr.34107) 872 : cluster [DBG] pgmap v1500: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:50:05.112 INFO:tasks.workunit.client.0.smithi115.stdout: 196: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:05.112 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uOqbCRmDFUz7FDt off=1582684 len=375875 2022-04-22T17:50:05.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 1004 completed, throughput=4.98MB/sec 2022-04-22T17:50:05.118 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jLD82t7mUDnteao off=265349 len=25256 2022-04-22T17:50:05.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 1005 completed, throughput=4.98MB/sec 2022-04-22T17:50:05.119 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l-NpUtQ_UOLs7Jy off=2110750 len=877140 2022-04-22T17:50:05.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 1006 completed, throughput=4.98MB/sec 2022-04-22T17:50:05.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-20_by16VudK6ydq off=14824102 len=2043372 2022-04-22T17:50:05.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 1007 completed, throughput=4.99MB/sec 2022-04-22T17:50:05.126 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1ulfs7psKWHcyVa off=2749172 len=851040 2022-04-22T17:50:05.131 INFO:tasks.workunit.client.0.smithi115.stdout:op 1008 completed, throughput=5MB/sec 2022-04-22T17:50:05.131 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MgXB93wCQRmUXEf off=16815290 len=1099657 2022-04-22T17:50:05.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 1009 completed, throughput=5MB/sec 2022-04-22T17:50:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:05 smithi121 bash[36902]: audit 2022-04-22T17:50:04.894245+0000 mon.a (mon.0) 2281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:05 smithi115 bash[45495]: audit 2022-04-22T17:50:04.894245+0000 mon.a (mon.0) 2281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:05 smithi115 bash[46934]: audit 2022-04-22T17:50:04.894245+0000 mon.a (mon.0) 2281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:06.137 INFO:tasks.workunit.client.0.smithi115.stdout: 197: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:06.137 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fRlYRNR_FUVyZ6K off=11680312 len=498080 2022-04-22T17:50:06.141 INFO:tasks.workunit.client.0.smithi115.stdout:op 1010 completed, throughput=4.98MB/sec 2022-04-22T17:50:06.141 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ReQWfAS3zHa8Cnh off=3931408 len=1928034 2022-04-22T17:50:06.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 1011 completed, throughput=4.99MB/sec 2022-04-22T17:50:06.144 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cqU58uOhr8bJzqQ off=22433367 len=800350 2022-04-22T17:50:06.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 1012 completed, throughput=4.99MB/sec 2022-04-22T17:50:06.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gcy071Yl-2_Jrcs off=12204914 len=1670891 2022-04-22T17:50:06.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 1013 completed, throughput=5MB/sec 2022-04-22T17:50:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:06 smithi121 bash[36902]: cluster 2022-04-22T17:50:04.344956+0000 mgr.y (mgr.34107) 873 : cluster [DBG] pgmap v1501: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:50:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:06 smithi121 bash[36902]: audit 2022-04-22T17:50:05.192901+0000 mon.a (mon.0) 2282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:06 smithi115 bash[45495]: cluster 2022-04-22T17:50:04.344956+0000 mgr.y (mgr.34107) 873 : cluster [DBG] pgmap v1501: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:50:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:06 smithi115 bash[45495]: audit 2022-04-22T17:50:05.192901+0000 mon.a (mon.0) 2282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:06.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:06 smithi115 bash[46934]: cluster 2022-04-22T17:50:04.344956+0000 mgr.y (mgr.34107) 873 : cluster [DBG] pgmap v1501: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:50:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:06 smithi115 bash[46934]: audit 2022-04-22T17:50:05.192901+0000 mon.a (mon.0) 2282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:07.155 INFO:tasks.workunit.client.0.smithi115.stdout: 198: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:07.155 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k-YUP-tDX-UgYGa off=2120675 len=813594 2022-04-22T17:50:07.163 INFO:tasks.workunit.client.0.smithi115.stdout:op 1014 completed, throughput=4.98MB/sec 2022-04-22T17:50:07.163 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5JnGcR3OW_vBj0v off=1808800 len=1288727 2022-04-22T17:50:07.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 1015 completed, throughput=4.99MB/sec 2022-04-22T17:50:07.165 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jZRE5BTHzKRfUex off=9015926 len=1224823 2022-04-22T17:50:07.170 INFO:tasks.workunit.client.0.smithi115.stdout:op 1016 completed, throughput=4.99MB/sec 2022-04-22T17:50:07.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R248bkMUkCp0NPx off=14174501 len=185892 2022-04-22T17:50:07.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 1017 completed, throughput=4.99MB/sec 2022-04-22T17:50:07.172 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XjPzQOc5y_506Sk off=1444741 len=410442 2022-04-22T17:50:07.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 1018 completed, throughput=4.99MB/sec 2022-04-22T17:50:07.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AHfqFceM02izpWH off=4971740 len=445145 2022-04-22T17:50:07.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 1019 completed, throughput=5MB/sec 2022-04-22T17:50:07.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-shAEnew5Z2K9YU9 off=1179847 len=1636513 2022-04-22T17:50:07.179 INFO:tasks.workunit.client.0.smithi115.stdout:op 1020 completed, throughput=5MB/sec 2022-04-22T17:50:08.180 INFO:tasks.workunit.client.0.smithi115.stdout: 199: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:08.180 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NKt0djQp9Xwu_ME off=7913720 len=102939 2022-04-22T17:50:08.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 1021 completed, throughput=4.98MB/sec 2022-04-22T17:50:08.181 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t7v57Efewl1ckCE off=9563612 len=1923413 2022-04-22T17:50:08.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 1022 completed, throughput=4.99MB/sec 2022-04-22T17:50:08.192 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nRj3HzwWNf5UR73 off=1422860 len=640156 2022-04-22T17:50:08.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 1023 completed, throughput=4.99MB/sec 2022-04-22T17:50:08.196 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q087anOTdsbjGU5 off=12531563 len=560234 2022-04-22T17:50:08.198 INFO:tasks.workunit.client.0.smithi115.stdout:op 1024 completed, throughput=4.99MB/sec 2022-04-22T17:50:08.198 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HkWpdPM51EIcplo off=6680888 len=464500 2022-04-22T17:50:08.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 1025 completed, throughput=5MB/sec 2022-04-22T17:50:08.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eXPFylS0PQOd0hF off=21813881 len=604318 2022-04-22T17:50:08.206 INFO:tasks.workunit.client.0.smithi115.stdout:op 1026 completed, throughput=5MB/sec 2022-04-22T17:50:08.206 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--WeXCHgdjDXAdim off=22863247 len=733716 2022-04-22T17:50:08.207 INFO:tasks.workunit.client.0.smithi115.stdout:op 1027 completed, throughput=5MB/sec 2022-04-22T17:50:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:08 smithi121 bash[36902]: cluster 2022-04-22T17:50:06.346217+0000 mgr.y (mgr.34107) 874 : cluster [DBG] pgmap v1502: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:50:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:08 smithi121 bash[36902]: audit 2022-04-22T17:50:07.274159+0000 mon.a (mon.0) 2283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:08 smithi115 bash[45495]: cluster 2022-04-22T17:50:06.346217+0000 mgr.y (mgr.34107) 874 : cluster [DBG] pgmap v1502: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:50:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:08 smithi115 bash[45495]: audit 2022-04-22T17:50:07.274159+0000 mon.a (mon.0) 2283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:08 smithi115 bash[46934]: cluster 2022-04-22T17:50:06.346217+0000 mgr.y (mgr.34107) 874 : cluster [DBG] pgmap v1502: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:50:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:08 smithi115 bash[46934]: audit 2022-04-22T17:50:07.274159+0000 mon.a (mon.0) 2283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:09.208 INFO:tasks.workunit.client.0.smithi115.stdout: 200: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:09.208 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GRX5l1yIEx3T5zu off=14137376 len=1688759 2022-04-22T17:50:09.218 INFO:tasks.workunit.client.0.smithi115.stdout:op 1028 completed, throughput=4.98MB/sec 2022-04-22T17:50:09.218 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mjcvV-1_7PIvgyG off=2505493 len=1484285 2022-04-22T17:50:09.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 1029 completed, throughput=4.99MB/sec 2022-04-22T17:50:09.219 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4XOWYw9vtX8UIJk off=10157795 len=137885 2022-04-22T17:50:09.221 INFO:tasks.workunit.client.0.smithi115.stdout:op 1030 completed, throughput=4.99MB/sec 2022-04-22T17:50:09.221 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m49J_v5II4vjw5l off=5942813 len=361298 2022-04-22T17:50:09.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 1031 completed, throughput=4.99MB/sec 2022-04-22T17:50:09.225 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-snvAbxQodiN6KOR off=422905 len=907923 2022-04-22T17:50:09.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 1032 completed, throughput=5MB/sec 2022-04-22T17:50:09.229 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ENvAndcc3o5U358 off=2109059 len=53579 2022-04-22T17:50:09.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 1033 completed, throughput=5MB/sec 2022-04-22T17:50:09.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wpVHQrVYEhFjqRB off=4161118 len=472507 2022-04-22T17:50:09.237 INFO:tasks.workunit.client.0.smithi115.stdout:op 1034 completed, throughput=5MB/sec 2022-04-22T17:50:10.237 INFO:tasks.workunit.client.0.smithi115.stdout: 201: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:10.237 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UrDVS1yiK4REmAB off=108364 len=1714857 2022-04-22T17:50:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 1035 completed, throughput=4.98MB/sec 2022-04-22T17:50:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q0gnGwEfu7gN52E off=8483160 len=820497 2022-04-22T17:50:10.245 INFO:tasks.workunit.client.0.smithi115.stdout:op 1036 completed, throughput=4.99MB/sec 2022-04-22T17:50:10.245 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FO_hd2-qN5bJ3N- off=4250560 len=1117785 2022-04-22T17:50:10.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 1037 completed, throughput=4.99MB/sec 2022-04-22T17:50:10.247 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sln4wc5_goK_orK off=12419024 len=2011434 2022-04-22T17:50:10.252 INFO:tasks.workunit.client.0.smithi115.stdout:op 1038 completed, throughput=5MB/sec 2022-04-22T17:50:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:10 smithi121 bash[36902]: cluster 2022-04-22T17:50:08.347419+0000 mgr.y (mgr.34107) 875 : cluster [DBG] pgmap v1503: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:10 smithi121 bash[36902]: audit 2022-04-22T17:50:09.897291+0000 mon.a (mon.0) 2284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:10 smithi115 bash[45495]: cluster 2022-04-22T17:50:08.347419+0000 mgr.y (mgr.34107) 875 : cluster [DBG] pgmap v1503: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:10 smithi115 bash[45495]: audit 2022-04-22T17:50:09.897291+0000 mon.a (mon.0) 2284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:10.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:10 smithi115 bash[46934]: cluster 2022-04-22T17:50:08.347419+0000 mgr.y (mgr.34107) 875 : cluster [DBG] pgmap v1503: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:10 smithi115 bash[46934]: audit 2022-04-22T17:50:09.897291+0000 mon.a (mon.0) 2284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:11.252 INFO:tasks.workunit.client.0.smithi115.stdout: 202: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:11.252 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9KxcorJzcI3N3nf off=3030674 len=1354564 2022-04-22T17:50:11.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 1039 completed, throughput=4.98MB/sec 2022-04-22T17:50:11.256 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cADEwBH7gd2p2c0 off=205375 len=1662271 2022-04-22T17:50:11.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 1040 completed, throughput=4.99MB/sec 2022-04-22T17:50:11.265 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZwD7yPifKskn1lE off=13677378 len=178451 2022-04-22T17:50:11.269 INFO:tasks.workunit.client.0.smithi115.stdout:op 1041 completed, throughput=4.99MB/sec 2022-04-22T17:50:11.269 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PEorqRxqnajRzyE off=4105362 len=1707165 2022-04-22T17:50:11.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 1042 completed, throughput=5MB/sec 2022-04-22T17:50:11.274 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h8RjFRMvHdfJjbq off=3437752 len=1560540 2022-04-22T17:50:11.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 1043 completed, throughput=5.01MB/sec 2022-04-22T17:50:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:11 smithi121 bash[36902]: audit 2022-04-22T17:50:10.196733+0000 mon.a (mon.0) 2285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:11 smithi115 bash[45495]: audit 2022-04-22T17:50:10.196733+0000 mon.a (mon.0) 2285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:11.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:11 smithi115 bash[46934]: audit 2022-04-22T17:50:10.196733+0000 mon.a (mon.0) 2285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:11.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:12.277 INFO:tasks.workunit.client.0.smithi115.stdout: 203: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:12.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LwUpwCprCkDC02N off=331202 len=806493 2022-04-22T17:50:12.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 1044 completed, throughput=4.99MB/sec 2022-04-22T17:50:12.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sBYzRCSt9BS1hrV off=2949143 len=1125426 2022-04-22T17:50:12.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 1045 completed, throughput=4.99MB/sec 2022-04-22T17:50:12.283 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b9nhRTsLduw1NvN off=12292172 len=2048328 2022-04-22T17:50:12.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 1046 completed, throughput=5MB/sec 2022-04-22T17:50:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:12 smithi121 bash[36902]: cluster 2022-04-22T17:50:10.348492+0000 mgr.y (mgr.34107) 876 : cluster [DBG] pgmap v1504: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:12.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:12 smithi115 bash[45495]: cluster 2022-04-22T17:50:10.348492+0000 mgr.y (mgr.34107) 876 : cluster [DBG] pgmap v1504: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:12.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:12 smithi115 bash[46934]: cluster 2022-04-22T17:50:10.348492+0000 mgr.y (mgr.34107) 876 : cluster [DBG] pgmap v1504: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:13.287 INFO:tasks.workunit.client.0.smithi115.stdout: 204: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:13.287 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gADZd0o-hFE3i3Q off=5401185 len=1417802 2022-04-22T17:50:13.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 1047 completed, throughput=4.98MB/sec 2022-04-22T17:50:13.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zH5FJJyhiab4R90 off=8884334 len=1764636 2022-04-22T17:50:13.301 INFO:tasks.workunit.client.0.smithi115.stdout:op 1048 completed, throughput=4.99MB/sec 2022-04-22T17:50:13.301 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C3oVxwMGvZwGpPK off=10272971 len=1048899 2022-04-22T17:50:13.303 INFO:tasks.workunit.client.0.smithi115.stdout:op 1049 completed, throughput=5MB/sec 2022-04-22T17:50:13.303 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4wHK1yION1NjFdb off=1013319 len=1772503 2022-04-22T17:50:13.312 INFO:tasks.workunit.client.0.smithi115.stdout:op 1050 completed, throughput=5MB/sec 2022-04-22T17:50:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:13 smithi121 bash[36902]: audit 2022-04-22T17:50:12.277138+0000 mon.a (mon.0) 2286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:13 smithi115 bash[45495]: audit 2022-04-22T17:50:12.277138+0000 mon.a (mon.0) 2286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:13.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:13 smithi115 bash[46934]: audit 2022-04-22T17:50:12.277138+0000 mon.a (mon.0) 2286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:14.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:13.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:14.313 INFO:tasks.workunit.client.0.smithi115.stdout: 205: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:14.313 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iIceFc7EcWN3DZ5 off=3138482 len=89057 2022-04-22T17:50:14.318 INFO:tasks.workunit.client.0.smithi115.stdout:op 1051 completed, throughput=4.98MB/sec 2022-04-22T17:50:14.318 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gpptwXUcwGjo-22 off=6747181 len=1489313 2022-04-22T17:50:14.322 INFO:tasks.workunit.client.0.smithi115.stdout:op 1052 completed, throughput=4.99MB/sec 2022-04-22T17:50:14.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-48cOehitoRjqFlm off=1242729 len=262205 2022-04-22T17:50:14.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 1053 completed, throughput=4.99MB/sec 2022-04-22T17:50:14.326 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S7IT-76es-Qs0kk off=5316265 len=662194 2022-04-22T17:50:14.331 INFO:tasks.workunit.client.0.smithi115.stdout:op 1054 completed, throughput=4.99MB/sec 2022-04-22T17:50:14.331 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PAJz3farZBNCZuM off=17356634 len=1443428 2022-04-22T17:50:14.337 INFO:tasks.workunit.client.0.smithi115.stdout:op 1055 completed, throughput=5MB/sec 2022-04-22T17:50:14.337 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ep5BJbio4SnCBOI off=9535713 len=1001434 2022-04-22T17:50:14.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 1056 completed, throughput=5MB/sec 2022-04-22T17:50:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:14 smithi121 bash[36902]: cluster 2022-04-22T17:50:12.349584+0000 mgr.y (mgr.34107) 877 : cluster [DBG] pgmap v1505: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:14.559 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:14 smithi121 bash[36902]: audit 2022-04-22T17:50:13.468794+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:50:14.559 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:14 smithi121 bash[36902]: audit 2022-04-22T17:50:13.471979+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:50:14.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[45495]: cluster 2022-04-22T17:50:12.349584+0000 mgr.y (mgr.34107) 877 : cluster [DBG] pgmap v1505: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:14.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[45495]: audit 2022-04-22T17:50:13.468794+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:50:14.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[45495]: audit 2022-04-22T17:50:13.471979+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:50:14.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[46934]: cluster 2022-04-22T17:50:12.349584+0000 mgr.y (mgr.34107) 877 : cluster [DBG] pgmap v1505: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:50:14.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[46934]: audit 2022-04-22T17:50:13.468794+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:50:14.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:14 smithi115 bash[46934]: audit 2022-04-22T17:50:13.471979+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:50:15.338 INFO:tasks.workunit.client.0.smithi115.stdout: 206: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:15.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VcwXh4nQP2Co3_M off=1414011 len=241372 2022-04-22T17:50:15.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 1057 completed, throughput=4.98MB/sec 2022-04-22T17:50:15.343 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4oQ5M-3sQ3peCes off=7005446 len=175570 2022-04-22T17:50:15.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 1058 completed, throughput=4.98MB/sec 2022-04-22T17:50:15.347 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gWWEc8C-OH1T4s- off=1563849 len=21748 2022-04-22T17:50:15.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 1059 completed, throughput=4.98MB/sec 2022-04-22T17:50:15.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oMOTov2Al_X-Id0 off=4367518 len=431335 2022-04-22T17:50:15.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 1060 completed, throughput=4.98MB/sec 2022-04-22T17:50:15.349 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QWRGaMeiffAV1nm off=14241932 len=1137490 2022-04-22T17:50:15.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 1061 completed, throughput=4.99MB/sec 2022-04-22T17:50:15.353 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MT0mDSv44fYpA0S off=9063001 len=1274356 2022-04-22T17:50:15.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 1062 completed, throughput=4.99MB/sec 2022-04-22T17:50:15.361 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AliNTuGXEXbmRR3 off=4541041 len=1221508 2022-04-22T17:50:15.363 INFO:tasks.workunit.client.0.smithi115.stdout:op 1063 completed, throughput=5MB/sec 2022-04-22T17:50:15.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AiY0Q8k-f0-u7Nd off=10103061 len=1350699 2022-04-22T17:50:15.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 1064 completed, throughput=5MB/sec 2022-04-22T17:50:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:15 smithi121 bash[36902]: audit 2022-04-22T17:50:14.900764+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:15 smithi115 bash[45495]: audit 2022-04-22T17:50:14.900764+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:15 smithi115 bash[46934]: audit 2022-04-22T17:50:14.900764+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:16.371 INFO:tasks.workunit.client.0.smithi115.stdout: 207: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:16.372 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OAStGVwvGvhJDZb off=15513666 len=885326 2022-04-22T17:50:16.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 1065 completed, throughput=4.98MB/sec 2022-04-22T17:50:16.375 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E89jelBWcDApS_6 off=8028291 len=2081888 2022-04-22T17:50:16.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 1066 completed, throughput=4.99MB/sec 2022-04-22T17:50:16.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AYDuYcIdxQcz2hO off=17968411 len=319933 2022-04-22T17:50:16.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 1067 completed, throughput=4.99MB/sec 2022-04-22T17:50:16.388 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2KzW1CKhTuTHHCf off=8893170 len=1517952 2022-04-22T17:50:16.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 1068 completed, throughput=5MB/sec 2022-04-22T17:50:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:16 smithi121 bash[36902]: cluster 2022-04-22T17:50:14.350645+0000 mgr.y (mgr.34107) 878 : cluster [DBG] pgmap v1506: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:16 smithi121 bash[36902]: audit 2022-04-22T17:50:15.200427+0000 mon.a (mon.0) 2290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:16 smithi115 bash[45495]: cluster 2022-04-22T17:50:14.350645+0000 mgr.y (mgr.34107) 878 : cluster [DBG] pgmap v1506: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:16 smithi115 bash[45495]: audit 2022-04-22T17:50:15.200427+0000 mon.a (mon.0) 2290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:16 smithi115 bash[46934]: cluster 2022-04-22T17:50:14.350645+0000 mgr.y (mgr.34107) 878 : cluster [DBG] pgmap v1506: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:16 smithi115 bash[46934]: audit 2022-04-22T17:50:15.200427+0000 mon.a (mon.0) 2290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:17.391 INFO:tasks.workunit.client.0.smithi115.stdout: 208: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:17.391 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1k88fCZm42KF3nm off=11343356 len=1770697 2022-04-22T17:50:17.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 1069 completed, throughput=4.99MB/sec 2022-04-22T17:50:17.400 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WpSseHmmA88ndMB off=2713089 len=556637 2022-04-22T17:50:17.405 INFO:tasks.workunit.client.0.smithi115.stdout:op 1070 completed, throughput=4.99MB/sec 2022-04-22T17:50:17.405 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GRDEziye4Bw71rR off=17728 len=1284745 2022-04-22T17:50:17.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 1071 completed, throughput=4.99MB/sec 2022-04-22T17:50:17.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_SCKj75V7KR1lVR off=2568721 len=2010416 2022-04-22T17:50:17.412 INFO:tasks.workunit.client.0.smithi115.stdout:op 1072 completed, throughput=5MB/sec 2022-04-22T17:50:18.413 INFO:tasks.workunit.client.0.smithi115.stdout: 209: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:18.413 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kE3UQlmAdl6f2R0 off=2746384 len=971421 2022-04-22T17:50:18.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 1073 completed, throughput=4.98MB/sec 2022-04-22T17:50:18.420 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KZEk-aHXd9Hzi_7 off=1886749 len=837195 2022-04-22T17:50:18.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 1074 completed, throughput=4.99MB/sec 2022-04-22T17:50:18.421 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FzedSlfP86jO9nc off=11586591 len=860119 2022-04-22T17:50:18.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 1075 completed, throughput=4.99MB/sec 2022-04-22T17:50:18.428 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lE2AfpfyLNLUICd off=4701437 len=1776154 2022-04-22T17:50:18.431 INFO:tasks.workunit.client.0.smithi115.stdout:op 1076 completed, throughput=5MB/sec 2022-04-22T17:50:18.431 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TqRFi7hi0ZL6oyj off=574672 len=1165501 2022-04-22T17:50:18.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 1077 completed, throughput=5MB/sec 2022-04-22T17:50:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:18 smithi121 bash[36902]: cluster 2022-04-22T17:50:16.351828+0000 mgr.y (mgr.34107) 879 : cluster [DBG] pgmap v1507: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:18 smithi121 bash[36902]: audit 2022-04-22T17:50:17.280025+0000 mon.a (mon.0) 2291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:18 smithi115 bash[45495]: cluster 2022-04-22T17:50:16.351828+0000 mgr.y (mgr.34107) 879 : cluster [DBG] pgmap v1507: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:18 smithi115 bash[45495]: audit 2022-04-22T17:50:17.280025+0000 mon.a (mon.0) 2291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:18 smithi115 bash[46934]: cluster 2022-04-22T17:50:16.351828+0000 mgr.y (mgr.34107) 879 : cluster [DBG] pgmap v1507: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:18 smithi115 bash[46934]: audit 2022-04-22T17:50:17.280025+0000 mon.a (mon.0) 2291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:19.439 INFO:tasks.workunit.client.0.smithi115.stdout: 210: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:19.439 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xS5rkZxAvhwQ518 off=1927281 len=1696195 2022-04-22T17:50:19.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 1078 completed, throughput=4.99MB/sec 2022-04-22T17:50:19.447 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BnXvaXZjVwGzrJX off=4217157 len=1690361 2022-04-22T17:50:19.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 1079 completed, throughput=4.99MB/sec 2022-04-22T17:50:19.456 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V4HvW9zmzT1Z33a off=4580756 len=1246239 2022-04-22T17:50:19.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 1080 completed, throughput=5MB/sec 2022-04-22T17:50:20.459 INFO:tasks.workunit.client.0.smithi115.stdout: 211: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:20.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VDk4iXYjGUhG_UF off=1193277 len=441126 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 1081 completed, throughput=4.98MB/sec 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-974D91-RknZFpRu off=1988694 len=718396 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 1082 completed, throughput=4.98MB/sec 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UsHLXjdGYQ6HV96 off=6324096 len=1861965 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 1083 completed, throughput=4.99MB/sec 2022-04-22T17:50:20.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oX4HmwKeKS0hNCl off=9170682 len=1278952 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 1084 completed, throughput=5MB/sec 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oj5RnE2DGimEKJV off=1813821 len=34104 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 1085 completed, throughput=5MB/sec 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vLTEUm3FmQTUAxl off=17712252 len=188059 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 1086 completed, throughput=5MB/sec 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2_zNQxR2AFjt0nL off=411852 len=38987 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 1087 completed, throughput=5MB/sec 2022-04-22T17:50:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-poXrtg2oZgKNY2X off=8335650 len=1258360 2022-04-22T17:50:20.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 1088 completed, throughput=5MB/sec 2022-04-22T17:50:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:20 smithi121 bash[36902]: cluster 2022-04-22T17:50:18.352959+0000 mgr.y (mgr.34107) 880 : cluster [DBG] pgmap v1508: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:50:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:20 smithi121 bash[36902]: audit 2022-04-22T17:50:19.903858+0000 mon.a (mon.0) 2292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:20 smithi115 bash[45495]: cluster 2022-04-22T17:50:18.352959+0000 mgr.y (mgr.34107) 880 : cluster [DBG] pgmap v1508: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:50:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:20 smithi115 bash[45495]: audit 2022-04-22T17:50:19.903858+0000 mon.a (mon.0) 2292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:20 smithi115 bash[46934]: cluster 2022-04-22T17:50:18.352959+0000 mgr.y (mgr.34107) 880 : cluster [DBG] pgmap v1508: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:50:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:20 smithi115 bash[46934]: audit 2022-04-22T17:50:19.903858+0000 mon.a (mon.0) 2292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:21.485 INFO:tasks.workunit.client.0.smithi115.stdout: 212: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:21.485 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M5HBhsTGGbgJZj0 off=7010336 len=1010643 2022-04-22T17:50:21.493 INFO:tasks.workunit.client.0.smithi115.stdout:op 1089 completed, throughput=4.98MB/sec 2022-04-22T17:50:21.493 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s6hb-rD1NbAag_p off=2009087 len=1331228 2022-04-22T17:50:21.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 1090 completed, throughput=4.99MB/sec 2022-04-22T17:50:21.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xr3D2_BbyW1X1WD off=7849852 len=1873280 2022-04-22T17:50:21.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 1091 completed, throughput=5MB/sec 2022-04-22T17:50:21.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pkcLcE6v7mkJIua off=3420245 len=164004 2022-04-22T17:50:21.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 1092 completed, throughput=5MB/sec 2022-04-22T17:50:21.503 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rdXPUftV70Um_5d off=3578537 len=515257 2022-04-22T17:50:21.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 1093 completed, throughput=5MB/sec 2022-04-22T17:50:21.509 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wx-_TCzSQgLq3Qa off=0 len=1648734 2022-04-22T17:50:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:21 smithi121 bash[36902]: audit 2022-04-22T17:50:20.204039+0000 mon.a (mon.0) 2293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:21.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 1094 completed, throughput=5.01MB/sec 2022-04-22T17:50:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:21 smithi115 bash[45495]: audit 2022-04-22T17:50:20.204039+0000 mon.a (mon.0) 2293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:21.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:21 smithi115 bash[46934]: audit 2022-04-22T17:50:20.204039+0000 mon.a (mon.0) 2293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:22.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:21.682Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:22 smithi121 bash[36902]: cluster 2022-04-22T17:50:20.354074+0000 mgr.y (mgr.34107) 881 : cluster [DBG] pgmap v1509: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:22.517 INFO:tasks.workunit.client.0.smithi115.stdout: 213: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:22.518 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6p6N8E-27vgN2X2 off=13320531 len=1726161 2022-04-22T17:50:22.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 1095 completed, throughput=4.99MB/sec 2022-04-22T17:50:22.523 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L7EUwaLIJ6Jwv7Y off=2268360 len=723647 2022-04-22T17:50:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:op 1096 completed, throughput=4.99MB/sec 2022-04-22T17:50:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rP61sbEc3ZJZuQt off=14350186 len=743279 2022-04-22T17:50:22.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 1097 completed, throughput=5MB/sec 2022-04-22T17:50:22.532 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i_FPQvlIg6VHkV7 off=4779360 len=182739 2022-04-22T17:50:22.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 1098 completed, throughput=5MB/sec 2022-04-22T17:50:22.532 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JfKxnWupF-rWiBl off=820356 len=700799 2022-04-22T17:50:22.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 1099 completed, throughput=5MB/sec 2022-04-22T17:50:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:22 smithi115 bash[45495]: cluster 2022-04-22T17:50:20.354074+0000 mgr.y (mgr.34107) 881 : cluster [DBG] pgmap v1509: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:22.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:22 smithi115 bash[46934]: cluster 2022-04-22T17:50:20.354074+0000 mgr.y (mgr.34107) 881 : cluster [DBG] pgmap v1509: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:50:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:23 smithi121 bash[36902]: audit 2022-04-22T17:50:22.283225+0000 mon.a (mon.0) 2294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:23.535 INFO:tasks.workunit.client.0.smithi115.stdout: 214: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:23.535 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mRSk9ApQEh6Oevt off=488702 len=1742443 2022-04-22T17:50:23.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 1100 completed, throughput=4.99MB/sec 2022-04-22T17:50:23.539 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FkHD2gvVWV7nAa- off=10895993 len=598809 2022-04-22T17:50:23.545 INFO:tasks.workunit.client.0.smithi115.stdout:op 1101 completed, throughput=4.99MB/sec 2022-04-22T17:50:23.545 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yPS8HYrxoN6r3PB off=2677637 len=1526843 2022-04-22T17:50:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:23 smithi115 bash[45495]: audit 2022-04-22T17:50:22.283225+0000 mon.a (mon.0) 2294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:23 smithi115 bash[46934]: audit 2022-04-22T17:50:22.283225+0000 mon.a (mon.0) 2294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:23.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 1102 completed, throughput=5MB/sec 2022-04-22T17:50:23.549 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1_NB6seKG5JzH27 off=1793260 len=930758 2022-04-22T17:50:23.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 1103 completed, throughput=5MB/sec 2022-04-22T17:50:23.552 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GIfUcj0uG1cOHui off=1622346 len=116427 2022-04-22T17:50:23.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 1104 completed, throughput=5MB/sec 2022-04-22T17:50:23.556 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j3wlVAkssjlYrAX off=3110032 len=715405 2022-04-22T17:50:23.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 1105 completed, throughput=5MB/sec 2022-04-22T17:50:24.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:23.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:24 smithi121 bash[36902]: cluster 2022-04-22T17:50:22.355149+0000 mgr.y (mgr.34107) 882 : cluster [DBG] pgmap v1510: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:24 smithi115 bash[45495]: cluster 2022-04-22T17:50:22.355149+0000 mgr.y (mgr.34107) 882 : cluster [DBG] pgmap v1510: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:24 smithi115 bash[46934]: cluster 2022-04-22T17:50:22.355149+0000 mgr.y (mgr.34107) 882 : cluster [DBG] pgmap v1510: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:24.558 INFO:tasks.workunit.client.0.smithi115.stdout: 215: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:24.558 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nGwiej9tpXQH7Io off=16181303 len=1833005 2022-04-22T17:50:24.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 1106 completed, throughput=4.99MB/sec 2022-04-22T17:50:24.568 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-78g7G0LlWAzupx8 off=0 len=1397750 2022-04-22T17:50:24.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 1107 completed, throughput=4.99MB/sec 2022-04-22T17:50:24.570 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UVTBNj5MmQns9yB off=4479694 len=647579 2022-04-22T17:50:24.576 INFO:tasks.workunit.client.0.smithi115.stdout:op 1108 completed, throughput=5MB/sec 2022-04-22T17:50:24.576 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yW1EhuRuSODG0Gb off=2886905 len=1411790 2022-04-22T17:50:24.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 1109 completed, throughput=5MB/sec 2022-04-22T17:50:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:25 smithi121 bash[36902]: audit 2022-04-22T17:50:24.906990+0000 mon.a (mon.0) 2295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:25 smithi115 bash[45495]: audit 2022-04-22T17:50:24.906990+0000 mon.a (mon.0) 2295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:25 smithi115 bash[46934]: audit 2022-04-22T17:50:24.906990+0000 mon.a (mon.0) 2295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:25.583 INFO:tasks.workunit.client.0.smithi115.stdout: 216: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:25.583 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c1fZsq2jE9X_xag off=4268096 len=1834205 2022-04-22T17:50:25.591 INFO:tasks.workunit.client.0.smithi115.stdout:op 1110 completed, throughput=4.99MB/sec 2022-04-22T17:50:25.592 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--j2DUX8iVBcmfhG off=931069 len=1039937 2022-04-22T17:50:25.593 INFO:tasks.workunit.client.0.smithi115.stdout:op 1111 completed, throughput=4.99MB/sec 2022-04-22T17:50:25.593 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nIGm3hU5JJ-SrKD off=865775 len=100626 2022-04-22T17:50:25.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 1112 completed, throughput=4.99MB/sec 2022-04-22T17:50:25.594 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X6GmMJz7nf8pceW off=10730861 len=1517809 2022-04-22T17:50:25.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 1113 completed, throughput=5MB/sec 2022-04-22T17:50:25.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tj1GKeJOuGq_Cth off=7077416 len=694092 2022-04-22T17:50:25.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 1114 completed, throughput=5MB/sec 2022-04-22T17:50:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:26 smithi121 bash[36902]: cluster 2022-04-22T17:50:24.356258+0000 mgr.y (mgr.34107) 883 : cluster [DBG] pgmap v1511: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:26.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:26 smithi121 bash[36902]: audit 2022-04-22T17:50:25.207256+0000 mon.a (mon.0) 2296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:26 smithi115 bash[45495]: cluster 2022-04-22T17:50:24.356258+0000 mgr.y (mgr.34107) 883 : cluster [DBG] pgmap v1511: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:26 smithi115 bash[45495]: audit 2022-04-22T17:50:25.207256+0000 mon.a (mon.0) 2296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:26 smithi115 bash[46934]: cluster 2022-04-22T17:50:24.356258+0000 mgr.y (mgr.34107) 883 : cluster [DBG] pgmap v1511: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:26 smithi115 bash[46934]: audit 2022-04-22T17:50:25.207256+0000 mon.a (mon.0) 2296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:26.599 INFO:tasks.workunit.client.0.smithi115.stdout: 217: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:26.599 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K7H9BafMR4e64Ly off=6524545 len=1961597 2022-04-22T17:50:26.608 INFO:tasks.workunit.client.0.smithi115.stdout:op 1115 completed, throughput=4.99MB/sec 2022-04-22T17:50:26.608 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8sE5ok_whAKPzEx off=6220132 len=340023 2022-04-22T17:50:26.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 1116 completed, throughput=4.99MB/sec 2022-04-22T17:50:26.612 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Kgdqd6vY7569B_t off=521082 len=1511223 2022-04-22T17:50:26.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 1117 completed, throughput=5MB/sec 2022-04-22T17:50:26.614 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g4m7A598YhQHdRT off=2723641 len=469543 2022-04-22T17:50:26.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 1118 completed, throughput=5MB/sec 2022-04-22T17:50:26.615 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nr82ytHvehcOucl off=1852397 len=75780 2022-04-22T17:50:26.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 1119 completed, throughput=5MB/sec 2022-04-22T17:50:26.618 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yIAQYH6s3vMCfwk off=7206369 len=17609 2022-04-22T17:50:26.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 1120 completed, throughput=5MB/sec 2022-04-22T17:50:26.618 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PXvk0uwjNi0BjLJ off=3591413 len=38606 2022-04-22T17:50:26.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 1121 completed, throughput=5MB/sec 2022-04-22T17:50:26.622 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5Ok8tkYYKSMQuLm off=8437939 len=416863 2022-04-22T17:50:26.623 INFO:tasks.workunit.client.0.smithi115.stdout:op 1122 completed, throughput=5MB/sec 2022-04-22T17:50:26.623 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jtoVuQrTAQpGJwX off=1258426 len=1974192 2022-04-22T17:50:26.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 1123 completed, throughput=5.01MB/sec 2022-04-22T17:50:27.627 INFO:tasks.workunit.client.0.smithi115.stdout: 218: throughput=4.99MB/sec pending data=0 2022-04-22T17:50:27.627 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nIW5Dd2PHPPa7q_ off=19366199 len=503360 2022-04-22T17:50:27.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 1124 completed, throughput=4.99MB/sec 2022-04-22T17:50:27.629 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sntg8gGMoAW1Z0- off=11716769 len=573442 2022-04-22T17:50:27.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 1125 completed, throughput=4.99MB/sec 2022-04-22T17:50:27.635 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TB4mZa6xEs2N1hp off=788456 len=272157 2022-04-22T17:50:27.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 1126 completed, throughput=4.99MB/sec 2022-04-22T17:50:27.640 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4lkIZDH3NPk_hDV off=1889067 len=1321315 2022-04-22T17:50:27.643 INFO:tasks.workunit.client.0.smithi115.stdout:op 1127 completed, throughput=5MB/sec 2022-04-22T17:50:27.644 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5C5Fs__U5Was-rr off=13016617 len=916130 2022-04-22T17:50:27.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 1128 completed, throughput=5MB/sec 2022-04-22T17:50:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:28 smithi121 bash[36902]: cluster 2022-04-22T17:50:26.357309+0000 mgr.y (mgr.34107) 884 : cluster [DBG] pgmap v1512: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:28 smithi121 bash[36902]: audit 2022-04-22T17:50:27.286244+0000 mon.a (mon.0) 2297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:28 smithi115 bash[45495]: cluster 2022-04-22T17:50:26.357309+0000 mgr.y (mgr.34107) 884 : cluster [DBG] pgmap v1512: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:28 smithi115 bash[45495]: audit 2022-04-22T17:50:27.286244+0000 mon.a (mon.0) 2297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:28 smithi115 bash[46934]: cluster 2022-04-22T17:50:26.357309+0000 mgr.y (mgr.34107) 884 : cluster [DBG] pgmap v1512: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:28 smithi115 bash[46934]: audit 2022-04-22T17:50:27.286244+0000 mon.a (mon.0) 2297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:28.646 INFO:tasks.workunit.client.0.smithi115.stdout: 219: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:28.646 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9tkSsFi4sX557Xf off=1424658 len=1999262 2022-04-22T17:50:28.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 1129 completed, throughput=4.99MB/sec 2022-04-22T17:50:28.658 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r3mj4tZBzRFfIqP off=1641345 len=1947606 2022-04-22T17:50:28.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 1130 completed, throughput=4.99MB/sec 2022-04-22T17:50:28.666 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v7JeefhjAeUxR7P off=11465942 len=577198 2022-04-22T17:50:28.670 INFO:tasks.workunit.client.0.smithi115.stdout:op 1131 completed, throughput=5MB/sec 2022-04-22T17:50:28.670 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ud1BcdArkDxWz5R off=2394936 len=287288 2022-04-22T17:50:28.670 INFO:tasks.workunit.client.0.smithi115.stdout:op 1132 completed, throughput=5MB/sec 2022-04-22T17:50:28.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-szav7WJWNucTZ_0 off=918186 len=1151865 2022-04-22T17:50:28.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 1133 completed, throughput=5MB/sec 2022-04-22T17:50:29.629 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:50:29.677 INFO:tasks.workunit.client.0.smithi115.stdout: 220: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:29.677 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VImp0Bjn6LTuJmw off=14292349 len=1259251 2022-04-22T17:50:29.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 1134 completed, throughput=4.99MB/sec 2022-04-22T17:50:29.681 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qBvkSUGs7DPVmNA off=845136 len=1717563 2022-04-22T17:50:29.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 1135 completed, throughput=4.99MB/sec 2022-04-22T17:50:29.684 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9J0rPCuyhpKk3WW off=7851573 len=1029632 2022-04-22T17:50:29.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 1136 completed, throughput=5MB/sec 2022-04-22T17:50:29.692 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i7bE2F0RNdP-G2K off=1879331 len=14793 2022-04-22T17:50:29.696 INFO:tasks.workunit.client.0.smithi115.stdout:op 1137 completed, throughput=5MB/sec 2022-04-22T17:50:29.696 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uaPyFnOpoBQTgDn off=12378381 len=1066198 2022-04-22T17:50:29.702 INFO:tasks.workunit.client.0.smithi115.stdout:op 1138 completed, throughput=5MB/sec 2022-04-22T17:50:29.974 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (13m) 2m ago 27m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (27m) 3m ago 27m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (15m) 2m ago 28m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (15m) 3m ago 28m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (22m) 3m ago 32m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (23m) 2m ago 34m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:50:29.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (22m) 2m ago 34m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (21m) 3m ago 32m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (21m) 2m ago 33m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (14m) 2m ago 27m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (14m) 3m ago 27m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (20m) 2m ago 32m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (19m) 2m ago 31m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (19m) 2m ago 31m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:50:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (18m) 2m ago 30m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:50:29.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (18m) 3m ago 30m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:50:29.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (17m) 3m ago 29m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:50:29.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (16m) 3m ago 29m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:50:29.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (15m) 3m ago 28m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:50:29.977 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (14m) 3m ago 27m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:50:30.388 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:50:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:50:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:50:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:50:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:50:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:50:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:50:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:50:30.390 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:50:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:30 smithi121 bash[36902]: cluster 2022-04-22T17:50:28.358361+0000 mgr.y (mgr.34107) 885 : cluster [DBG] pgmap v1513: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:30 smithi121 bash[36902]: audit 2022-04-22T17:50:29.909856+0000 mon.a (mon.0) 2298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:30 smithi115 bash[45495]: cluster 2022-04-22T17:50:28.358361+0000 mgr.y (mgr.34107) 885 : cluster [DBG] pgmap v1513: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:30 smithi115 bash[45495]: audit 2022-04-22T17:50:29.909856+0000 mon.a (mon.0) 2298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:30.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:30 smithi115 bash[46934]: cluster 2022-04-22T17:50:28.358361+0000 mgr.y (mgr.34107) 885 : cluster [DBG] pgmap v1513: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:30.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:30 smithi115 bash[46934]: audit 2022-04-22T17:50:29.909856+0000 mon.a (mon.0) 2298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:30.702 INFO:tasks.workunit.client.0.smithi115.stdout: 221: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:30.702 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tr1usPO7gV67YO5 off=15399987 len=1946181 2022-04-22T17:50:30.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 1139 completed, throughput=4.99MB/sec 2022-04-22T17:50:30.714 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JUfsa8xWeyCwYOU off=1135336 len=1784951 2022-04-22T17:50:30.718 INFO:tasks.workunit.client.0.smithi115.stdout:op 1140 completed, throughput=4.99MB/sec 2022-04-22T17:50:30.718 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vYElvWBIDJef5Zx off=1539365 len=369264 2022-04-22T17:50:30.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 1141 completed, throughput=5MB/sec 2022-04-22T17:50:30.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RxHD_MOCUpZl28q off=2255008 len=1974782 2022-04-22T17:50:30.724 INFO:tasks.workunit.client.0.smithi115.stdout:op 1142 completed, throughput=5MB/sec 2022-04-22T17:50:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:31 smithi121 bash[36902]: audit 2022-04-22T17:50:29.601014+0000 mgr.y (mgr.34107) 886 : audit [DBG] from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:31 smithi121 bash[36902]: audit 2022-04-22T17:50:29.946078+0000 mgr.y (mgr.34107) 887 : audit [DBG] from='client.45283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:31 smithi121 bash[36902]: audit 2022-04-22T17:50:30.213195+0000 mon.a (mon.0) 2299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:31.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:31 smithi121 bash[36902]: audit 2022-04-22T17:50:30.378897+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.115:0/1702748379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[45495]: audit 2022-04-22T17:50:29.601014+0000 mgr.y (mgr.34107) 886 : audit [DBG] from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[45495]: audit 2022-04-22T17:50:29.946078+0000 mgr.y (mgr.34107) 887 : audit [DBG] from='client.45283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[45495]: audit 2022-04-22T17:50:30.213195+0000 mon.a (mon.0) 2299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[45495]: audit 2022-04-22T17:50:30.378897+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.115:0/1702748379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:31.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[46934]: audit 2022-04-22T17:50:29.601014+0000 mgr.y (mgr.34107) 886 : audit [DBG] from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[46934]: audit 2022-04-22T17:50:29.946078+0000 mgr.y (mgr.34107) 887 : audit [DBG] from='client.45283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:50:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[46934]: audit 2022-04-22T17:50:30.213195+0000 mon.a (mon.0) 2299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[46934]: audit 2022-04-22T17:50:30.378897+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.115:0/1702748379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:50:31.724 INFO:tasks.workunit.client.0.smithi115.stdout: 222: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:31.724 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e6IQ2uHNMeYXelU off=6886107 len=571214 2022-04-22T17:50:31.726 INFO:tasks.workunit.client.0.smithi115.stdout:op 1143 completed, throughput=4.98MB/sec 2022-04-22T17:50:31.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gtMZ00KhP6z2_RP off=3160308 len=905454 2022-04-22T17:50:31.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 1144 completed, throughput=4.99MB/sec 2022-04-22T17:50:31.728 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--b6azmVJcraCLry off=2815493 len=1460449 2022-04-22T17:50:31.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 1145 completed, throughput=4.99MB/sec 2022-04-22T17:50:31.738 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9e_o3_1jZwbCUnY off=9199852 len=1147955 2022-04-22T17:50:31.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 1146 completed, throughput=5MB/sec 2022-04-22T17:50:31.741 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nyQyK49gUqcki6c off=731975 len=1355125 2022-04-22T17:50:31.745 INFO:tasks.workunit.client.0.smithi115.stdout:op 1147 completed, throughput=5MB/sec 2022-04-22T17:50:32.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:31.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:32 smithi121 bash[36902]: cluster 2022-04-22T17:50:30.359354+0000 mgr.y (mgr.34107) 888 : cluster [DBG] pgmap v1514: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:32 smithi115 bash[45495]: cluster 2022-04-22T17:50:30.359354+0000 mgr.y (mgr.34107) 888 : cluster [DBG] pgmap v1514: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:32.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:32 smithi115 bash[46934]: cluster 2022-04-22T17:50:30.359354+0000 mgr.y (mgr.34107) 888 : cluster [DBG] pgmap v1514: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:32.746 INFO:tasks.workunit.client.0.smithi115.stdout: 223: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:32.746 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9aVAqx7uOIiEh2d off=583468 len=1358883 2022-04-22T17:50:32.749 INFO:tasks.workunit.client.0.smithi115.stdout:op 1148 completed, throughput=4.99MB/sec 2022-04-22T17:50:32.749 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Kn14fLaCmZtFJyM off=3575972 len=1867072 2022-04-22T17:50:32.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 1149 completed, throughput=5MB/sec 2022-04-22T17:50:32.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9-tPfBc8lWtg7Cq off=695189 len=1944470 2022-04-22T17:50:32.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 1150 completed, throughput=5MB/sec 2022-04-22T17:50:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:33 smithi121 bash[36902]: audit 2022-04-22T17:50:32.289317+0000 mon.a (mon.0) 2300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:33 smithi115 bash[45495]: audit 2022-04-22T17:50:32.289317+0000 mon.a (mon.0) 2300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:33 smithi115 bash[46934]: audit 2022-04-22T17:50:32.289317+0000 mon.a (mon.0) 2300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:33.765 INFO:tasks.workunit.client.0.smithi115.stdout: 224: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:33.765 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1M9JrjPl8CvTzMr off=1394961 len=642271 2022-04-22T17:50:33.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 1151 completed, throughput=4.98MB/sec 2022-04-22T17:50:33.767 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GaKW2Y0-p3h8dPd off=8599390 len=645836 2022-04-22T17:50:33.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 1152 completed, throughput=4.99MB/sec 2022-04-22T17:50:33.774 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yfPivhkb_DodkBu off=3336926 len=727522 2022-04-22T17:50:33.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 1153 completed, throughput=4.99MB/sec 2022-04-22T17:50:33.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vd9Mk39RmzPGAA4 off=8085516 len=448407 2022-04-22T17:50:33.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 1154 completed, throughput=4.99MB/sec 2022-04-22T17:50:33.784 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VYkC1kpi-Uep-bx off=2071872 len=2045170 2022-04-22T17:50:33.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 1155 completed, throughput=5MB/sec 2022-04-22T17:50:34.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:33.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:34 smithi121 bash[36902]: cluster 2022-04-22T17:50:32.360429+0000 mgr.y (mgr.34107) 889 : cluster [DBG] pgmap v1515: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:34 smithi115 bash[45495]: cluster 2022-04-22T17:50:32.360429+0000 mgr.y (mgr.34107) 889 : cluster [DBG] pgmap v1515: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:34 smithi115 bash[46934]: cluster 2022-04-22T17:50:32.360429+0000 mgr.y (mgr.34107) 889 : cluster [DBG] pgmap v1515: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:50:34.787 INFO:tasks.workunit.client.0.smithi115.stdout: 225: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:34.787 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lf9SJTqocEVGgVc off=11919699 len=1513411 2022-04-22T17:50:34.795 INFO:tasks.workunit.client.0.smithi115.stdout:op 1156 completed, throughput=4.98MB/sec 2022-04-22T17:50:34.796 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6rPYHUiVU8yKKM0 off=2669888 len=268954 2022-04-22T17:50:34.799 INFO:tasks.workunit.client.0.smithi115.stdout:op 1157 completed, throughput=4.99MB/sec 2022-04-22T17:50:34.800 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5ColNKhcPyxfQXq off=9407418 len=796539 2022-04-22T17:50:34.802 INFO:tasks.workunit.client.0.smithi115.stdout:op 1158 completed, throughput=4.99MB/sec 2022-04-22T17:50:34.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fA394hiZTTuRDb4 off=14728388 len=401801 2022-04-22T17:50:34.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 1159 completed, throughput=4.99MB/sec 2022-04-22T17:50:34.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8Y47XDyT8Cd36ww off=9556027 len=915973 2022-04-22T17:50:34.809 INFO:tasks.workunit.client.0.smithi115.stdout:op 1160 completed, throughput=4.99MB/sec 2022-04-22T17:50:34.809 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CRi0TrBs2E9Edp7 off=18130684 len=842768 2022-04-22T17:50:34.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 1161 completed, throughput=5MB/sec 2022-04-22T17:50:34.815 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qX9qP6SB8v8s94V off=63392 len=497648 2022-04-22T17:50:34.818 INFO:tasks.workunit.client.0.smithi115.stdout:op 1162 completed, throughput=5MB/sec 2022-04-22T17:50:34.818 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TREZRqhyL_3k50W off=4175364 len=766935 2022-04-22T17:50:34.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 1163 completed, throughput=5MB/sec 2022-04-22T17:50:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:35 smithi121 bash[36902]: audit 2022-04-22T17:50:34.912947+0000 mon.a (mon.0) 2301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:35 smithi121 bash[36902]: audit 2022-04-22T17:50:35.215807+0000 mon.a (mon.0) 2302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:35 smithi115 bash[45495]: audit 2022-04-22T17:50:34.912947+0000 mon.a (mon.0) 2301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:35.581 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:35 smithi115 bash[45495]: audit 2022-04-22T17:50:35.215807+0000 mon.a (mon.0) 2302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:35.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:35 smithi115 bash[46934]: audit 2022-04-22T17:50:34.912947+0000 mon.a (mon.0) 2301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:35.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:35 smithi115 bash[46934]: audit 2022-04-22T17:50:35.215807+0000 mon.a (mon.0) 2302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:35.823 INFO:tasks.workunit.client.0.smithi115.stdout: 226: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:35.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jK6iJUkQ9MtzL0R off=5829778 len=21462 2022-04-22T17:50:35.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 1164 completed, throughput=4.98MB/sec 2022-04-22T17:50:35.872 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DCXBa_Gv8m5CZIU off=130634 len=118196 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1165 completed, throughput=4.98MB/sec 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yv8YMWIdHtcq7Ca off=14908319 len=1300665 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1166 completed, throughput=4.99MB/sec 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wJDU4N7H84UV6-z off=2625069 len=180483 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1167 completed, throughput=4.99MB/sec 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bfQEaLTL4Juy_DG off=9331423 len=1680380 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1168 completed, throughput=4.99MB/sec 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4auUxcIPz-kXyvJ off=3630386 len=1505717 2022-04-22T17:50:35.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1169 completed, throughput=5MB/sec 2022-04-22T17:50:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:36 smithi121 bash[36902]: cluster 2022-04-22T17:50:34.361536+0000 mgr.y (mgr.34107) 890 : cluster [DBG] pgmap v1516: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:36.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:36 smithi115 bash[45495]: cluster 2022-04-22T17:50:34.361536+0000 mgr.y (mgr.34107) 890 : cluster [DBG] pgmap v1516: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:36 smithi115 bash[46934]: cluster 2022-04-22T17:50:34.361536+0000 mgr.y (mgr.34107) 890 : cluster [DBG] pgmap v1516: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:50:36.852 INFO:tasks.workunit.client.0.smithi115.stdout: 227: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:36.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zQhSG-ixiVge55O off=15707934 len=1913217 2022-04-22T17:50:36.864 INFO:tasks.workunit.client.0.smithi115.stdout:op 1170 completed, throughput=4.99MB/sec 2022-04-22T17:50:36.864 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PXvk0uwjNi0BjLJ off=17827928 len=297618 2022-04-22T17:50:36.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 1171 completed, throughput=4.99MB/sec 2022-04-22T17:50:36.868 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aTayA51d08Zb5lS off=792684 len=140038 2022-04-22T17:50:36.872 INFO:tasks.workunit.client.0.smithi115.stdout:op 1172 completed, throughput=4.99MB/sec 2022-04-22T17:50:36.873 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r4OEXMW_A5QS3DQ off=6894279 len=1379112 2022-04-22T17:50:36.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 1173 completed, throughput=4.99MB/sec 2022-04-22T17:50:36.877 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fA394hiZTTuRDb4 off=12827079 len=1340744 2022-04-22T17:50:36.884 INFO:tasks.workunit.client.0.smithi115.stdout:op 1174 completed, throughput=5MB/sec 2022-04-22T17:50:36.884 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5TE-ybKH6GXtWHS off=2324345 len=948165 2022-04-22T17:50:36.886 INFO:tasks.workunit.client.0.smithi115.stdout:op 1175 completed, throughput=5MB/sec 2022-04-22T17:50:37.886 INFO:tasks.workunit.client.0.smithi115.stdout: 228: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:37.886 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CxTYJ5Tl7wmjavm off=11162636 len=1911243 2022-04-22T17:50:37.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 1176 completed, throughput=4.99MB/sec 2022-04-22T17:50:37.896 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PgImgFvoax6JljN off=9066644 len=570116 2022-04-22T17:50:37.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 1177 completed, throughput=4.99MB/sec 2022-04-22T17:50:37.898 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eZVaFN6wp5G7gbz off=9675389 len=1695586 2022-04-22T17:50:37.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 1178 completed, throughput=5MB/sec 2022-04-22T17:50:37.902 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LGrp5TK95Yqf4GM off=52225 len=273416 2022-04-22T17:50:37.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 1179 completed, throughput=5MB/sec 2022-04-22T17:50:37.903 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3UsGWBEOXM5M1xS off=1128759 len=1521831 2022-04-22T17:50:37.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 1180 completed, throughput=5.01MB/sec 2022-04-22T17:50:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[46934]: cluster 2022-04-22T17:50:36.362715+0000 mgr.y (mgr.34107) 891 : cluster [DBG] pgmap v1517: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[46934]: audit 2022-04-22T17:50:37.292212+0000 mon.a (mon.0) 2303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[46934]: audit 2022-04-22T17:50:37.999907+0000 mon.a (mon.0) 2304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[46934]: audit 2022-04-22T17:50:38.002867+0000 mon.a (mon.0) 2305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[46934]: audit 2022-04-22T17:50:38.004621+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[45495]: cluster 2022-04-22T17:50:36.362715+0000 mgr.y (mgr.34107) 891 : cluster [DBG] pgmap v1517: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[45495]: audit 2022-04-22T17:50:37.292212+0000 mon.a (mon.0) 2303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[45495]: audit 2022-04-22T17:50:37.999907+0000 mon.a (mon.0) 2304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[45495]: audit 2022-04-22T17:50:38.002867+0000 mon.a (mon.0) 2305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:50:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:38 smithi115 bash[45495]: audit 2022-04-22T17:50:38.004621+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:50:38.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:38 smithi121 bash[36902]: cluster 2022-04-22T17:50:36.362715+0000 mgr.y (mgr.34107) 891 : cluster [DBG] pgmap v1517: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:38.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:38 smithi121 bash[36902]: audit 2022-04-22T17:50:37.292212+0000 mon.a (mon.0) 2303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:38.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:38 smithi121 bash[36902]: audit 2022-04-22T17:50:37.999907+0000 mon.a (mon.0) 2304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:50:38.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:38 smithi121 bash[36902]: audit 2022-04-22T17:50:38.002867+0000 mon.a (mon.0) 2305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:50:38.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:38 smithi121 bash[36902]: audit 2022-04-22T17:50:38.004621+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:50:38.911 INFO:tasks.workunit.client.0.smithi115.stdout: 229: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:38.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WkUwY-z0NXxMGmI off=9768214 len=1458386 2022-04-22T17:50:38.915 INFO:tasks.workunit.client.0.smithi115.stdout:op 1181 completed, throughput=4.99MB/sec 2022-04-22T17:50:38.915 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hk-T1DKU7qK1Td_ off=5970490 len=491719 2022-04-22T17:50:38.921 INFO:tasks.workunit.client.0.smithi115.stdout:op 1182 completed, throughput=4.99MB/sec 2022-04-22T17:50:38.921 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RhsT3ML_P4xfd4u off=3461507 len=326780 2022-04-22T17:50:38.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 1183 completed, throughput=4.99MB/sec 2022-04-22T17:50:38.922 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FApNEfylyfcUbgJ off=4755139 len=149968 2022-04-22T17:50:38.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 1184 completed, throughput=4.99MB/sec 2022-04-22T17:50:38.923 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7vDComtwVdEip5s off=17088808 len=703382 2022-04-22T17:50:38.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 1185 completed, throughput=5MB/sec 2022-04-22T17:50:38.928 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fSXQTwdsaI4XJ_u off=17901774 len=127893 2022-04-22T17:50:38.931 INFO:tasks.workunit.client.0.smithi115.stdout:op 1186 completed, throughput=5MB/sec 2022-04-22T17:50:38.931 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TFMNCde_yGW4Li- off=13342987 len=899264 2022-04-22T17:50:38.937 INFO:tasks.workunit.client.0.smithi115.stdout:op 1187 completed, throughput=5MB/sec 2022-04-22T17:50:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:39 smithi121 bash[36902]: cluster 2022-04-22T17:50:38.363392+0000 mgr.y (mgr.34107) 892 : cluster [DBG] pgmap v1518: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:39.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:39 smithi121 bash[36902]: audit 2022-04-22T17:50:38.370477+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:50:39.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:39 smithi115 bash[46934]: cluster 2022-04-22T17:50:38.363392+0000 mgr.y (mgr.34107) 892 : cluster [DBG] pgmap v1518: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:39.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:39 smithi115 bash[46934]: audit 2022-04-22T17:50:38.370477+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:50:39.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:39 smithi115 bash[45495]: cluster 2022-04-22T17:50:38.363392+0000 mgr.y (mgr.34107) 892 : cluster [DBG] pgmap v1518: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:39.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:39 smithi115 bash[45495]: audit 2022-04-22T17:50:38.370477+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:50:39.938 INFO:tasks.workunit.client.0.smithi115.stdout: 230: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:39.938 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d1ZSIeA4gDBBpHP off=9836640 len=1798395 2022-04-22T17:50:39.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 1188 completed, throughput=4.99MB/sec 2022-04-22T17:50:39.941 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u1Bif7su0gz4DvL off=683714 len=270050 2022-04-22T17:50:39.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 1189 completed, throughput=4.99MB/sec 2022-04-22T17:50:39.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0zLnWEZUYrILlcK off=12583099 len=602769 2022-04-22T17:50:39.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 1190 completed, throughput=4.99MB/sec 2022-04-22T17:50:39.949 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HCr92qYNqDCbokL off=11430808 len=1385797 2022-04-22T17:50:39.956 INFO:tasks.workunit.client.0.smithi115.stdout:op 1191 completed, throughput=4.99MB/sec 2022-04-22T17:50:39.957 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q7dH5sXPdPI1L7V off=4773311 len=609016 2022-04-22T17:50:39.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 1192 completed, throughput=5MB/sec 2022-04-22T17:50:39.958 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UHE_PawgrC46tVp off=3749673 len=941069 2022-04-22T17:50:39.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 1193 completed, throughput=5MB/sec 2022-04-22T17:50:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:40 smithi121 bash[36902]: audit 2022-04-22T17:50:39.916231+0000 mon.a (mon.0) 2308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:40 smithi121 bash[36902]: audit 2022-04-22T17:50:40.219773+0000 mon.a (mon.0) 2309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:40 smithi115 bash[45495]: audit 2022-04-22T17:50:39.916231+0000 mon.a (mon.0) 2308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:40 smithi115 bash[45495]: audit 2022-04-22T17:50:40.219773+0000 mon.a (mon.0) 2309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:40 smithi115 bash[46934]: audit 2022-04-22T17:50:39.916231+0000 mon.a (mon.0) 2308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:40 smithi115 bash[46934]: audit 2022-04-22T17:50:40.219773+0000 mon.a (mon.0) 2309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:40.962 INFO:tasks.workunit.client.0.smithi115.stdout: 231: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:40.962 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L4nhAohLgsg4-Ew off=19818369 len=156595 2022-04-22T17:50:40.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 1194 completed, throughput=4.98MB/sec 2022-04-22T17:50:40.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ctip9h-Yaj2Jx9k off=1534088 len=1131061 2022-04-22T17:50:40.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 1195 completed, throughput=4.98MB/sec 2022-04-22T17:50:40.970 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4e6_1rBtZQD7rGe off=412699 len=1430879 2022-04-22T17:50:40.978 INFO:tasks.workunit.client.0.smithi115.stdout:op 1196 completed, throughput=4.99MB/sec 2022-04-22T17:50:40.978 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fwJP8kikyfciGIF off=5623554 len=1601729 2022-04-22T17:50:40.986 INFO:tasks.workunit.client.0.smithi115.stdout:op 1197 completed, throughput=5MB/sec 2022-04-22T17:50:40.986 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VrVILcNaXTsQgrm off=1307488 len=126797 2022-04-22T17:50:40.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 1198 completed, throughput=5MB/sec 2022-04-22T17:50:40.987 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GyWcpMH_bUft9mL off=8564936 len=1300319 2022-04-22T17:50:40.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 1199 completed, throughput=5MB/sec 2022-04-22T17:50:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:41 smithi121 bash[36902]: cluster 2022-04-22T17:50:40.364514+0000 mgr.y (mgr.34107) 893 : cluster [DBG] pgmap v1519: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:50:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:41 smithi115 bash[45495]: cluster 2022-04-22T17:50:40.364514+0000 mgr.y (mgr.34107) 893 : cluster [DBG] pgmap v1519: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:50:41.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:41.684Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:41.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:41 smithi115 bash[46934]: cluster 2022-04-22T17:50:40.364514+0000 mgr.y (mgr.34107) 893 : cluster [DBG] pgmap v1519: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:50:41.993 INFO:tasks.workunit.client.0.smithi115.stdout: 232: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:41.993 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gtfaNemvxCEQa80 off=3315799 len=407115 2022-04-22T17:50:41.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 1200 completed, throughput=4.98MB/sec 2022-04-22T17:50:41.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GrX2jGDbNwEBfMz off=3422328 len=698449 2022-04-22T17:50:42.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 1201 completed, throughput=4.99MB/sec 2022-04-22T17:50:42.001 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6LjWGiraljHeDxx off=1478172 len=1063276 2022-04-22T17:50:42.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 1202 completed, throughput=4.99MB/sec 2022-04-22T17:50:42.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aeHVHDFEGm_9h6v off=3329387 len=1422297 2022-04-22T17:50:42.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 1203 completed, throughput=5MB/sec 2022-04-22T17:50:42.015 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lc7tXiYn_j8a2RY off=4616259 len=866038 2022-04-22T17:50:42.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 1204 completed, throughput=5MB/sec 2022-04-22T17:50:42.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EyXVij-eZ9M3vTe off=105741 len=1676444 2022-04-22T17:50:42.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 1205 completed, throughput=5.01MB/sec 2022-04-22T17:50:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:42 smithi121 bash[36902]: audit 2022-04-22T17:50:42.295145+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:42 smithi115 bash[45495]: audit 2022-04-22T17:50:42.295145+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:42 smithi115 bash[46934]: audit 2022-04-22T17:50:42.295145+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:43.024 INFO:tasks.workunit.client.0.smithi115.stdout: 233: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:43.025 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X7gwsy6X6Wf_L2o off=5116319 len=362923 2022-04-22T17:50:43.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 1206 completed, throughput=4.99MB/sec 2022-04-22T17:50:43.030 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1_NB6seKG5JzH27 off=7437150 len=1065677 2022-04-22T17:50:43.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 1207 completed, throughput=4.99MB/sec 2022-04-22T17:50:43.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AiCDysDTD12xqtq off=8673637 len=280981 2022-04-22T17:50:43.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 1208 completed, throughput=4.99MB/sec 2022-04-22T17:50:43.041 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jSWQVM-DanRDdIa off=5121138 len=876452 2022-04-22T17:50:43.044 INFO:tasks.workunit.client.0.smithi115.stdout:op 1209 completed, throughput=4.99MB/sec 2022-04-22T17:50:43.044 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5vEo_TF8CE2CylA off=6227663 len=994094 2022-04-22T17:50:43.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 1210 completed, throughput=5MB/sec 2022-04-22T17:50:43.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4yVioOAOMUz5PT4 off=4433785 len=162812 2022-04-22T17:50:43.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 1211 completed, throughput=5MB/sec 2022-04-22T17:50:43.049 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m8U5R-K2CTlBsKw off=7810605 len=1275248 2022-04-22T17:50:43.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 1212 completed, throughput=5MB/sec 2022-04-22T17:50:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:43 smithi121 bash[36902]: cluster 2022-04-22T17:50:42.365652+0000 mgr.y (mgr.34107) 894 : cluster [DBG] pgmap v1520: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:43 smithi115 bash[45495]: cluster 2022-04-22T17:50:42.365652+0000 mgr.y (mgr.34107) 894 : cluster [DBG] pgmap v1520: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:43.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:43.565Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:43.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:43 smithi115 bash[46934]: cluster 2022-04-22T17:50:42.365652+0000 mgr.y (mgr.34107) 894 : cluster [DBG] pgmap v1520: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:44.051 INFO:tasks.workunit.client.0.smithi115.stdout: 234: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:44.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OPM8jbqWMa4QFoq off=4719734 len=1851644 2022-04-22T17:50:44.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 1213 completed, throughput=4.99MB/sec 2022-04-22T17:50:44.062 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-stHwrQkqiBWsPLn off=646389 len=1894793 2022-04-22T17:50:44.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 1214 completed, throughput=5MB/sec 2022-04-22T17:50:44.068 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2i5hSiU-GP9lX4g off=5874080 len=950280 2022-04-22T17:50:44.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 1215 completed, throughput=5MB/sec 2022-04-22T17:50:45.075 INFO:tasks.workunit.client.0.smithi115.stdout: 235: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:45.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bOrz8e08_XwjqpH off=1639971 len=438882 2022-04-22T17:50:45.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 1216 completed, throughput=4.98MB/sec 2022-04-22T17:50:45.081 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8hR5UprNnfFxaK1 off=20164826 len=1014723 2022-04-22T17:50:45.085 INFO:tasks.workunit.client.0.smithi115.stdout:op 1217 completed, throughput=4.99MB/sec 2022-04-22T17:50:45.085 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2pTwzUIAvuu0dKo off=12639949 len=693741 2022-04-22T17:50:45.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 1218 completed, throughput=4.99MB/sec 2022-04-22T17:50:45.091 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SuKV1rHdndVRPVw off=496964 len=1343657 2022-04-22T17:50:45.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 1219 completed, throughput=4.99MB/sec 2022-04-22T17:50:45.094 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zaaYh9L1Tt3oO7C off=10307145 len=1080591 2022-04-22T17:50:45.095 INFO:tasks.workunit.client.0.smithi115.stdout:op 1220 completed, throughput=5MB/sec 2022-04-22T17:50:45.095 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NQYeN8gwgbXlUf6 off=13289070 len=226317 2022-04-22T17:50:45.096 INFO:tasks.workunit.client.0.smithi115.stdout:op 1221 completed, throughput=5MB/sec 2022-04-22T17:50:45.096 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AxdxxjV-Kcz8c5Y off=11986498 len=488107 2022-04-22T17:50:45.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 1222 completed, throughput=5MB/sec 2022-04-22T17:50:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:45 smithi121 bash[36902]: cluster 2022-04-22T17:50:44.366762+0000 mgr.y (mgr.34107) 895 : cluster [DBG] pgmap v1521: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:45 smithi121 bash[36902]: audit 2022-04-22T17:50:44.919097+0000 mon.a (mon.0) 2311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:45.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:45 smithi121 bash[36902]: audit 2022-04-22T17:50:45.223600+0000 mon.a (mon.0) 2312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[46934]: cluster 2022-04-22T17:50:44.366762+0000 mgr.y (mgr.34107) 895 : cluster [DBG] pgmap v1521: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[46934]: audit 2022-04-22T17:50:44.919097+0000 mon.a (mon.0) 2311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[46934]: audit 2022-04-22T17:50:45.223600+0000 mon.a (mon.0) 2312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[45495]: cluster 2022-04-22T17:50:44.366762+0000 mgr.y (mgr.34107) 895 : cluster [DBG] pgmap v1521: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:50:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[45495]: audit 2022-04-22T17:50:44.919097+0000 mon.a (mon.0) 2311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:45 smithi115 bash[45495]: audit 2022-04-22T17:50:45.223600+0000 mon.a (mon.0) 2312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:46.098 INFO:tasks.workunit.client.0.smithi115.stdout: 236: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:46.098 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vJvTtLZ7pUktTMS off=9215866 len=24572 2022-04-22T17:50:46.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 1223 completed, throughput=4.98MB/sec 2022-04-22T17:50:46.099 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QVjqu052CXYdlnI off=19690373 len=1990241 2022-04-22T17:50:46.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 1224 completed, throughput=4.99MB/sec 2022-04-22T17:50:46.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VOa04JxdQAohbvL off=8074362 len=121507 2022-04-22T17:50:46.108 INFO:tasks.workunit.client.0.smithi115.stdout:op 1225 completed, throughput=4.99MB/sec 2022-04-22T17:50:46.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JMgH76KnCv97hsa off=22198114 len=466314 2022-04-22T17:50:46.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 1226 completed, throughput=4.99MB/sec 2022-04-22T17:50:46.114 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l1GJzZG2uAOreKx off=13341805 len=1886747 2022-04-22T17:50:46.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 1227 completed, throughput=5MB/sec 2022-04-22T17:50:46.118 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Y57kuWxHlxCKXPe off=301655 len=1448457 2022-04-22T17:50:46.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 1228 completed, throughput=5MB/sec 2022-04-22T17:50:47.120 INFO:tasks.workunit.client.0.smithi115.stdout: 237: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:47.128 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1NtiIpYK1PxG8RX off=74841 len=179582 2022-04-22T17:50:47.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 1229 completed, throughput=4.98MB/sec 2022-04-22T17:50:47.129 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ikRokOytdT5lMId off=8717779 len=427455 2022-04-22T17:50:47.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 1230 completed, throughput=4.98MB/sec 2022-04-22T17:50:47.130 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iOGeuaW6ov0_8-E off=17843536 len=288758 2022-04-22T17:50:47.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 1231 completed, throughput=4.99MB/sec 2022-04-22T17:50:47.130 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pssn3-UzwjLMGuM off=2420136 len=282389 2022-04-22T17:50:47.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 1232 completed, throughput=4.99MB/sec 2022-04-22T17:50:47.130 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EIWlsyrgdzptb4b off=1944061 len=563929 2022-04-22T17:50:47.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 1233 completed, throughput=4.99MB/sec 2022-04-22T17:50:47.135 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7DhgypeNIfZdZXs off=9891347 len=1775937 2022-04-22T17:50:47.139 INFO:tasks.workunit.client.0.smithi115.stdout:op 1234 completed, throughput=5MB/sec 2022-04-22T17:50:47.139 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q-TUFJBxOv6-e5o off=946833 len=235762 2022-04-22T17:50:47.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 1235 completed, throughput=5MB/sec 2022-04-22T17:50:47.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-324RNyWA6i0lfwB off=575320 len=944486 2022-04-22T17:50:47.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 1236 completed, throughput=5MB/sec 2022-04-22T17:50:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:47 smithi121 bash[36902]: cluster 2022-04-22T17:50:46.367872+0000 mgr.y (mgr.34107) 896 : cluster [DBG] pgmap v1522: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:47 smithi121 bash[36902]: audit 2022-04-22T17:50:47.298441+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:47 smithi115 bash[45495]: cluster 2022-04-22T17:50:46.367872+0000 mgr.y (mgr.34107) 896 : cluster [DBG] pgmap v1522: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:47 smithi115 bash[45495]: audit 2022-04-22T17:50:47.298441+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:47 smithi115 bash[46934]: cluster 2022-04-22T17:50:46.367872+0000 mgr.y (mgr.34107) 896 : cluster [DBG] pgmap v1522: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:50:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:47 smithi115 bash[46934]: audit 2022-04-22T17:50:47.298441+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:48.145 INFO:tasks.workunit.client.0.smithi115.stdout: 238: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:48.145 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7cpElHBdNgmlIyP off=955163 len=800135 2022-04-22T17:50:48.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 1237 completed, throughput=4.98MB/sec 2022-04-22T17:50:48.148 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NP54KIDCJP1ookQ off=11570257 len=963144 2022-04-22T17:50:48.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 1238 completed, throughput=4.99MB/sec 2022-04-22T17:50:48.155 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QNh9dsc_yDC7vAX off=9495223 len=357377 2022-04-22T17:50:48.158 INFO:tasks.workunit.client.0.smithi115.stdout:op 1239 completed, throughput=4.99MB/sec 2022-04-22T17:50:48.158 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d7-4WET9Fdww8n_ off=6522353 len=1014162 2022-04-22T17:50:48.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 1240 completed, throughput=4.99MB/sec 2022-04-22T17:50:48.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h6EShFNbLm30hsv off=5770875 len=980203 2022-04-22T17:50:48.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 1241 completed, throughput=5MB/sec 2022-04-22T17:50:48.171 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s4xQqK5NcwPFo1h off=863378 len=532147 2022-04-22T17:50:48.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 1242 completed, throughput=5MB/sec 2022-04-22T17:50:48.218 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XlvKuI-zBZk3RKl off=5670761 len=1790422 2022-04-22T17:50:48.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 1243 completed, throughput=5MB/sec 2022-04-22T17:50:49.183 INFO:tasks.workunit.client.0.smithi115.stdout: 239: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:49.184 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MRtYDnupx-GSXch off=5244627 len=802897 2022-04-22T17:50:49.186 INFO:tasks.workunit.client.0.smithi115.stdout:op 1244 completed, throughput=4.99MB/sec 2022-04-22T17:50:49.186 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-glzw8TYze68mfng off=7957768 len=1709438 2022-04-22T17:50:49.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 1245 completed, throughput=4.99MB/sec 2022-04-22T17:50:49.196 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MptRw1537102Ks5 off=16805597 len=257788 2022-04-22T17:50:49.197 INFO:tasks.workunit.client.0.smithi115.stdout:op 1246 completed, throughput=4.99MB/sec 2022-04-22T17:50:49.197 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cXKbF-a_C8d_xee off=17607697 len=97823 2022-04-22T17:50:49.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 1247 completed, throughput=4.99MB/sec 2022-04-22T17:50:49.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k1pVCkrP9YL5LBv off=1662809 len=196491 2022-04-22T17:50:49.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 1248 completed, throughput=5MB/sec 2022-04-22T17:50:49.201 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3xeD2CU3aIliifr off=10421926 len=1697384 2022-04-22T17:50:49.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 1249 completed, throughput=5MB/sec 2022-04-22T17:50:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:49 smithi115 bash[45495]: cluster 2022-04-22T17:50:48.369079+0000 mgr.y (mgr.34107) 897 : cluster [DBG] pgmap v1523: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:49 smithi115 bash[46934]: cluster 2022-04-22T17:50:48.369079+0000 mgr.y (mgr.34107) 897 : cluster [DBG] pgmap v1523: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:49 smithi121 bash[36902]: cluster 2022-04-22T17:50:48.369079+0000 mgr.y (mgr.34107) 897 : cluster [DBG] pgmap v1523: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:50:50.210 INFO:tasks.workunit.client.0.smithi115.stdout: 240: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:50.210 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6Q6T1znfVVUaidT off=675906 len=1915552 2022-04-22T17:50:50.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 1250 completed, throughput=4.99MB/sec 2022-04-22T17:50:50.216 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1OZh_215uE-MB6r off=6143425 len=1843791 2022-04-22T17:50:50.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 1251 completed, throughput=5MB/sec 2022-04-22T17:50:50.225 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ww1zwhjcEAIvwan off=3060052 len=2071762 2022-04-22T17:50:50.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 1252 completed, throughput=5MB/sec 2022-04-22T17:50:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:50 smithi115 bash[45495]: audit 2022-04-22T17:50:49.921774+0000 mon.a (mon.0) 2314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:50 smithi115 bash[45495]: audit 2022-04-22T17:50:50.227213+0000 mon.a (mon.0) 2315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:50 smithi115 bash[46934]: audit 2022-04-22T17:50:49.921774+0000 mon.a (mon.0) 2314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:50 smithi115 bash[46934]: audit 2022-04-22T17:50:50.227213+0000 mon.a (mon.0) 2315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:50 smithi121 bash[36902]: audit 2022-04-22T17:50:49.921774+0000 mon.a (mon.0) 2314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:50 smithi121 bash[36902]: audit 2022-04-22T17:50:50.227213+0000 mon.a (mon.0) 2315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:51.233 INFO:tasks.workunit.client.0.smithi115.stdout: 241: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:51.233 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cWV6sX6pOpyxGXj off=1814707 len=1362020 2022-04-22T17:50:51.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 1253 completed, throughput=4.99MB/sec 2022-04-22T17:50:51.242 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Od9x9Ez-0qsFgJR off=9014239 len=1388410 2022-04-22T17:50:51.249 INFO:tasks.workunit.client.0.smithi115.stdout:op 1254 completed, throughput=4.99MB/sec 2022-04-22T17:50:51.249 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qqaIdkIf0OFnhQI off=2677096 len=909430 2022-04-22T17:50:51.254 INFO:tasks.workunit.client.0.smithi115.stdout:op 1255 completed, throughput=5MB/sec 2022-04-22T17:50:51.254 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ul9BKcHC7lFECw2 off=11279983 len=996488 2022-04-22T17:50:51.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 1256 completed, throughput=5MB/sec 2022-04-22T17:50:51.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:51.685Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:51 smithi115 bash[45495]: cluster 2022-04-22T17:50:50.370173+0000 mgr.y (mgr.34107) 898 : cluster [DBG] pgmap v1524: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:51 smithi115 bash[46934]: cluster 2022-04-22T17:50:50.370173+0000 mgr.y (mgr.34107) 898 : cluster [DBG] pgmap v1524: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:51 smithi121 bash[36902]: cluster 2022-04-22T17:50:50.370173+0000 mgr.y (mgr.34107) 898 : cluster [DBG] pgmap v1524: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:50:52.259 INFO:tasks.workunit.client.0.smithi115.stdout: 242: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:52.259 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qf1tEsjKpJJAlOT off=3963088 len=1283047 2022-04-22T17:50:52.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 1257 completed, throughput=4.99MB/sec 2022-04-22T17:50:52.267 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SqneHdWlvXgwTm6 off=1837752 len=778369 2022-04-22T17:50:52.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 1258 completed, throughput=4.99MB/sec 2022-04-22T17:50:52.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-akf3GCLRPjWEhSc off=15906499 len=291116 2022-04-22T17:50:52.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 1259 completed, throughput=4.99MB/sec 2022-04-22T17:50:52.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UPG32xPZqmY32Ux off=1319125 len=1635379 2022-04-22T17:50:52.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 1260 completed, throughput=5MB/sec 2022-04-22T17:50:52.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E0sI18AzsDKaC3w off=1833923 len=1281353 2022-04-22T17:50:52.288 INFO:tasks.workunit.client.0.smithi115.stdout:op 1261 completed, throughput=5MB/sec 2022-04-22T17:50:52.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:52 smithi115 bash[45495]: audit 2022-04-22T17:50:52.301822+0000 mon.a (mon.0) 2316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:52.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:52 smithi115 bash[46934]: audit 2022-04-22T17:50:52.301822+0000 mon.a (mon.0) 2316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:52 smithi121 bash[36902]: audit 2022-04-22T17:50:52.301822+0000 mon.a (mon.0) 2316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:53.288 INFO:tasks.workunit.client.0.smithi115.stdout: 243: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:53.289 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hABzeTPqVKdqpRK off=4233467 len=1471488 2022-04-22T17:50:53.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 1262 completed, throughput=4.99MB/sec 2022-04-22T17:50:53.292 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Pzigmsy33BTE8pH off=5201271 len=865899 2022-04-22T17:50:53.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 1263 completed, throughput=4.99MB/sec 2022-04-22T17:50:53.298 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SIQrTikK6WQ481O off=1268940 len=142804 2022-04-22T17:50:53.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 1264 completed, throughput=4.99MB/sec 2022-04-22T17:50:53.299 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nyQyK49gUqcki6c off=11910526 len=1572700 2022-04-22T17:50:53.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 1265 completed, throughput=5MB/sec 2022-04-22T17:50:53.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PERcGBbKlXxZOft off=5655411 len=1556589 2022-04-22T17:50:53.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 1266 completed, throughput=5MB/sec 2022-04-22T17:50:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:53 smithi115 bash[45495]: cluster 2022-04-22T17:50:52.371165+0000 mgr.y (mgr.34107) 899 : cluster [DBG] pgmap v1525: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:53.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:50:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:50:53.567Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:50:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:53 smithi115 bash[46934]: cluster 2022-04-22T17:50:52.371165+0000 mgr.y (mgr.34107) 899 : cluster [DBG] pgmap v1525: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:53 smithi121 bash[36902]: cluster 2022-04-22T17:50:52.371165+0000 mgr.y (mgr.34107) 899 : cluster [DBG] pgmap v1525: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:50:54.315 INFO:tasks.workunit.client.0.smithi115.stdout: 244: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:54.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-A9dLKrNeYYNSPuO off=6270741 len=1895368 2022-04-22T17:50:54.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 1267 completed, throughput=4.99MB/sec 2022-04-22T17:50:54.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ePW8NZMNReFsLWc off=1517940 len=563409 2022-04-22T17:50:54.327 INFO:tasks.workunit.client.0.smithi115.stdout:op 1268 completed, throughput=4.99MB/sec 2022-04-22T17:50:54.328 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e4KYvq_iaMzpLBI off=11075927 len=612876 2022-04-22T17:50:54.333 INFO:tasks.workunit.client.0.smithi115.stdout:op 1269 completed, throughput=4.99MB/sec 2022-04-22T17:50:54.334 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s9NOZhCGoWo1lR4 off=935329 len=139716 2022-04-22T17:50:54.337 INFO:tasks.workunit.client.0.smithi115.stdout:op 1270 completed, throughput=4.99MB/sec 2022-04-22T17:50:54.337 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gq7llfAnN930VTV off=25378 len=1789099 2022-04-22T17:50:54.339 INFO:tasks.workunit.client.0.smithi115.stdout:op 1271 completed, throughput=5MB/sec 2022-04-22T17:50:54.340 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sq_gH0AQzqRb6wu off=3797785 len=451871 2022-04-22T17:50:54.340 INFO:tasks.workunit.client.0.smithi115.stdout:op 1272 completed, throughput=5MB/sec 2022-04-22T17:50:55.341 INFO:tasks.workunit.client.0.smithi115.stdout: 245: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:55.341 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u0OCUxdk4OpMi_4 off=10204111 len=1354210 2022-04-22T17:50:55.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 1273 completed, throughput=4.99MB/sec 2022-04-22T17:50:55.344 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UWPmjDBvoQfMWgl off=3926500 len=1784198 2022-04-22T17:50:55.350 INFO:tasks.workunit.client.0.smithi115.stdout:op 1274 completed, throughput=4.99MB/sec 2022-04-22T17:50:55.350 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dJpDfr7x1-msNu3 off=5482729 len=2047209 2022-04-22T17:50:55.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 1275 completed, throughput=5MB/sec 2022-04-22T17:50:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[45495]: cluster 2022-04-22T17:50:54.372021+0000 mgr.y (mgr.34107) 900 : cluster [DBG] pgmap v1526: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[45495]: audit 2022-04-22T17:50:54.925070+0000 mon.a (mon.0) 2317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[45495]: audit 2022-04-22T17:50:55.230372+0000 mon.a (mon.0) 2318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[46934]: cluster 2022-04-22T17:50:54.372021+0000 mgr.y (mgr.34107) 900 : cluster [DBG] pgmap v1526: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[46934]: audit 2022-04-22T17:50:54.925070+0000 mon.a (mon.0) 2317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:55 smithi115 bash[46934]: audit 2022-04-22T17:50:55.230372+0000 mon.a (mon.0) 2318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:55 smithi121 bash[36902]: cluster 2022-04-22T17:50:54.372021+0000 mgr.y (mgr.34107) 900 : cluster [DBG] pgmap v1526: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:55 smithi121 bash[36902]: audit 2022-04-22T17:50:54.925070+0000 mon.a (mon.0) 2317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:50:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:55 smithi121 bash[36902]: audit 2022-04-22T17:50:55.230372+0000 mon.a (mon.0) 2318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:50:56.360 INFO:tasks.workunit.client.0.smithi115.stdout: 246: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:56.360 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AhJRDpWTwRXRBpE off=1296172 len=895891 2022-04-22T17:50:56.363 INFO:tasks.workunit.client.0.smithi115.stdout:op 1276 completed, throughput=4.98MB/sec 2022-04-22T17:50:56.363 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gnB2vpfJqzVpBxa off=5507109 len=554340 2022-04-22T17:50:56.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 1277 completed, throughput=4.99MB/sec 2022-04-22T17:50:56.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hinLVIR6pLWfvFF off=1457259 len=465459 2022-04-22T17:50:56.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 1278 completed, throughput=4.99MB/sec 2022-04-22T17:50:56.367 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J8a2MjiEy0BiAzL off=114043 len=1867960 2022-04-22T17:50:56.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 1279 completed, throughput=5MB/sec 2022-04-22T17:50:56.370 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x-ZUMrZttENgl5c off=10213242 len=165931 2022-04-22T17:50:56.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 1280 completed, throughput=5MB/sec 2022-04-22T17:50:56.372 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--NvtWF0aBsvLrjq off=3606224 len=148817 2022-04-22T17:50:56.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 1281 completed, throughput=5MB/sec 2022-04-22T17:50:56.372 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Eb-hhgZ5dPcZcEX off=5016648 len=1869866 2022-04-22T17:50:56.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 1282 completed, throughput=5MB/sec 2022-04-22T17:50:57.375 INFO:tasks.workunit.client.0.smithi115.stdout: 247: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:57.376 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O4eoUEvyPwLGNYj off=6046019 len=1006291 2022-04-22T17:50:57.382 INFO:tasks.workunit.client.0.smithi115.stdout:op 1283 completed, throughput=4.99MB/sec 2022-04-22T17:50:57.383 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_HTyvQ5wGbBiEK0 off=2956444 len=878356 2022-04-22T17:50:57.385 INFO:tasks.workunit.client.0.smithi115.stdout:op 1284 completed, throughput=4.99MB/sec 2022-04-22T17:50:57.385 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tIu-ps63-27rzK1 off=7645342 len=1955150 2022-04-22T17:50:57.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 1285 completed, throughput=5MB/sec 2022-04-22T17:50:57.395 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yyPB8xT8KNOGIyS off=1155556 len=1561289 2022-04-22T17:50:57.401 INFO:tasks.workunit.client.0.smithi115.stdout:op 1286 completed, throughput=5MB/sec 2022-04-22T17:50:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:57 smithi115 bash[45495]: cluster 2022-04-22T17:50:56.373056+0000 mgr.y (mgr.34107) 901 : cluster [DBG] pgmap v1527: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:57 smithi115 bash[45495]: audit 2022-04-22T17:50:57.304947+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:57 smithi115 bash[46934]: cluster 2022-04-22T17:50:56.373056+0000 mgr.y (mgr.34107) 901 : cluster [DBG] pgmap v1527: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:57 smithi115 bash[46934]: audit 2022-04-22T17:50:57.304947+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:57 smithi121 bash[36902]: cluster 2022-04-22T17:50:56.373056+0000 mgr.y (mgr.34107) 901 : cluster [DBG] pgmap v1527: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:50:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:57 smithi121 bash[36902]: audit 2022-04-22T17:50:57.304947+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:50:58.401 INFO:tasks.workunit.client.0.smithi115.stdout: 248: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:58.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-niAmAXv1VmGjJE9 off=4390234 len=1677833 2022-04-22T17:50:58.405 INFO:tasks.workunit.client.0.smithi115.stdout:op 1287 completed, throughput=4.99MB/sec 2022-04-22T17:50:58.405 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pTMfSRHB3tt3meQ off=412819 len=1466190 2022-04-22T17:50:58.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 1288 completed, throughput=5MB/sec 2022-04-22T17:50:58.410 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cm7cgy-ZhqotEnE off=11260720 len=1862930 2022-04-22T17:50:58.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 1289 completed, throughput=5MB/sec 2022-04-22T17:50:59.414 INFO:tasks.workunit.client.0.smithi115.stdout: 249: throughput=4.98MB/sec pending data=0 2022-04-22T17:50:59.414 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E8OazZK1u9-FRaS off=7133037 len=1315764 2022-04-22T17:50:59.424 INFO:tasks.workunit.client.0.smithi115.stdout:op 1290 completed, throughput=4.99MB/sec 2022-04-22T17:50:59.424 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lD2A0nedBGAKlzw off=1781364 len=224387 2022-04-22T17:50:59.425 INFO:tasks.workunit.client.0.smithi115.stdout:op 1291 completed, throughput=4.99MB/sec 2022-04-22T17:50:59.425 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-quiwKQQFzGNnzon off=7835521 len=98269 2022-04-22T17:50:59.429 INFO:tasks.workunit.client.0.smithi115.stdout:op 1292 completed, throughput=4.99MB/sec 2022-04-22T17:50:59.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PHvxTvq3EFeDEaC off=30107 len=364004 2022-04-22T17:50:59.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 1293 completed, throughput=4.99MB/sec 2022-04-22T17:50:59.434 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KQPRfSsHY1hplvT off=1425843 len=1707023 2022-04-22T17:50:59.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 1294 completed, throughput=5MB/sec 2022-04-22T17:50:59.436 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aoXA3eDWPCuWl-E off=7924575 len=695142 2022-04-22T17:50:59.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 1295 completed, throughput=5MB/sec 2022-04-22T17:50:59.442 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JeBfrtvUVNvhk9H off=219143 len=34234 2022-04-22T17:50:59.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 1296 completed, throughput=5MB/sec 2022-04-22T17:50:59.443 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y3oCHiztp8pZiO7 off=970858 len=993179 2022-04-22T17:50:59.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 1297 completed, throughput=5MB/sec 2022-04-22T17:51:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:50:59 smithi121 bash[36902]: cluster 2022-04-22T17:50:58.374218+0000 mgr.y (mgr.34107) 902 : cluster [DBG] pgmap v1528: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:50:59 smithi115 bash[45495]: cluster 2022-04-22T17:50:58.374218+0000 mgr.y (mgr.34107) 902 : cluster [DBG] pgmap v1528: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:50:59 smithi115 bash[46934]: cluster 2022-04-22T17:50:58.374218+0000 mgr.y (mgr.34107) 902 : cluster [DBG] pgmap v1528: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:00.448 INFO:tasks.workunit.client.0.smithi115.stdout: 250: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:00.448 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5cQ5paoDSKILraH off=5915874 len=1689568 2022-04-22T17:51:00.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 1298 completed, throughput=4.99MB/sec 2022-04-22T17:51:00.457 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-43rdElk7Lg_pn4V off=13297542 len=1785925 2022-04-22T17:51:00.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 1299 completed, throughput=5MB/sec 2022-04-22T17:51:00.466 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PMB0KJrMev6EKCH off=5038855 len=1608043 2022-04-22T17:51:00.473 INFO:tasks.workunit.client.0.smithi115.stdout:op 1300 completed, throughput=5MB/sec 2022-04-22T17:51:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:51:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:00 smithi121 bash[36902]: audit 2022-04-22T17:50:59.928083+0000 mon.a (mon.0) 2320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:00 smithi121 bash[36902]: audit 2022-04-22T17:51:00.234269+0000 mon.a (mon.0) 2321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:00 smithi115 bash[45495]: audit 2022-04-22T17:50:59.928083+0000 mon.a (mon.0) 2320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:00 smithi115 bash[45495]: audit 2022-04-22T17:51:00.234269+0000 mon.a (mon.0) 2321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:00 smithi115 bash[46934]: audit 2022-04-22T17:50:59.928083+0000 mon.a (mon.0) 2320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:00 smithi115 bash[46934]: audit 2022-04-22T17:51:00.234269+0000 mon.a (mon.0) 2321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:01.115 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:51:01.115 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (14m) 3m ago 27m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:51:01.116 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (27m) 4m ago 27m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:51:01.116 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (16m) 3m ago 28m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:51:01.116 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (15m) 4m ago 28m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:51:01.116 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (23m) 4m ago 33m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:51:01.116 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (23m) 3m ago 35m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:51:01.117 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (22m) 3m ago 35m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:51:01.117 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (22m) 4m ago 33m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:51:01.117 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (22m) 3m ago 33m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:51:01.117 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (15m) 3m ago 28m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:51:01.117 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (15m) 4m ago 28m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (20m) 3m ago 32m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (20m) 3m ago 32m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (19m) 3m ago 31m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (19m) 3m ago 31m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (18m) 4m ago 30m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (17m) 4m ago 30m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (17m) 4m ago 29m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (16m) 4m ago 29m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:51:01.118 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (14m) 4m ago 28m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:51:01.474 INFO:tasks.workunit.client.0.smithi115.stdout: 251: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:01.474 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_BJpSxlsm2OeI6- off=5193340 len=12621 2022-04-22T17:51:01.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 1301 completed, throughput=4.98MB/sec 2022-04-22T17:51:01.478 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XYukezWnMRgLgwJ off=7234738 len=653920 2022-04-22T17:51:01.482 INFO:tasks.workunit.client.0.smithi115.stdout:op 1302 completed, throughput=4.98MB/sec 2022-04-22T17:51:01.482 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4qZyo86VSDLWjFL off=5386190 len=1395828 2022-04-22T17:51:01.485 INFO:tasks.workunit.client.0.smithi115.stdout:op 1303 completed, throughput=4.99MB/sec 2022-04-22T17:51:01.531 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GcDvahIrvWEpZKE off=868538 len=764809 2022-04-22T17:51:01.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 1304 completed, throughput=4.99MB/sec 2022-04-22T17:51:01.532 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4XHLqsPzIpKjsym off=2056286 len=700799 2022-04-22T17:51:01.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 1305 completed, throughput=4.99MB/sec 2022-04-22T17:51:01.532 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v6lT0aksENn61EN off=2980686 len=874293 2022-04-22T17:51:01.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 1306 completed, throughput=5MB/sec 2022-04-22T17:51:01.533 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1nxkDrShBrhWjlE off=5375277 len=1646629 2022-04-22T17:51:01.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 1307 completed, throughput=5MB/sec 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:51:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:51:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:51:01.535 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:51:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:01 smithi121 bash[36902]: cluster 2022-04-22T17:51:00.375332+0000 mgr.y (mgr.34107) 903 : cluster [DBG] pgmap v1529: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:02.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:01 smithi121 bash[36902]: audit 2022-04-22T17:51:00.744316+0000 mgr.y (mgr.34107) 904 : audit [DBG] from='client.45295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:01 smithi121 bash[36902]: audit 2022-04-22T17:51:01.085003+0000 mgr.y (mgr.34107) 905 : audit [DBG] from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.031 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:01 smithi121 bash[36902]: audit 2022-04-22T17:51:01.503938+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.115:0/1017683425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:02.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:01.684Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[45495]: cluster 2022-04-22T17:51:00.375332+0000 mgr.y (mgr.34107) 903 : cluster [DBG] pgmap v1529: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[45495]: audit 2022-04-22T17:51:00.744316+0000 mgr.y (mgr.34107) 904 : audit [DBG] from='client.45295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[45495]: audit 2022-04-22T17:51:01.085003+0000 mgr.y (mgr.34107) 905 : audit [DBG] from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[45495]: audit 2022-04-22T17:51:01.503938+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.115:0/1017683425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[46934]: cluster 2022-04-22T17:51:00.375332+0000 mgr.y (mgr.34107) 903 : cluster [DBG] pgmap v1529: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[46934]: audit 2022-04-22T17:51:00.744316+0000 mgr.y (mgr.34107) 904 : audit [DBG] from='client.45295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[46934]: audit 2022-04-22T17:51:01.085003+0000 mgr.y (mgr.34107) 905 : audit [DBG] from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:01 smithi115 bash[46934]: audit 2022-04-22T17:51:01.503938+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.115:0/1017683425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:02.503 INFO:tasks.workunit.client.0.smithi115.stdout: 252: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:02.547 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DyEAfETmSqhlxqd off=0 len=1168022 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 1308 completed, throughput=4.99MB/sec 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-I40rQLNKv4PsZJt off=8402518 len=609635 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 1309 completed, throughput=4.99MB/sec 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vLUkyEZJwRipRlc off=6826699 len=1748860 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 1310 completed, throughput=5MB/sec 2022-04-22T17:51:02.548 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gg_z4ua-0C7snO- off=2211546 len=1882074 2022-04-22T17:51:02.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 1311 completed, throughput=5MB/sec 2022-04-22T17:51:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:02 smithi121 bash[36902]: audit 2022-04-22T17:51:02.308137+0000 mon.a (mon.0) 2322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:02 smithi115 bash[45495]: audit 2022-04-22T17:51:02.308137+0000 mon.a (mon.0) 2322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:02 smithi115 bash[46934]: audit 2022-04-22T17:51:02.308137+0000 mon.a (mon.0) 2322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:03.521 INFO:tasks.workunit.client.0.smithi115.stdout: 253: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:03.521 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KPF-PZ2rXtLgIG_ off=11280018 len=926967 2022-04-22T17:51:03.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 1312 completed, throughput=4.99MB/sec 2022-04-22T17:51:03.524 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-atsoT4shY4572BF off=9878624 len=523429 2022-04-22T17:51:03.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 1313 completed, throughput=4.99MB/sec 2022-04-22T17:51:03.526 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iIU5RWSGtXChB6D off=18581404 len=1962892 2022-04-22T17:51:03.534 INFO:tasks.workunit.client.0.smithi115.stdout:op 1314 completed, throughput=5MB/sec 2022-04-22T17:51:03.535 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Cr4e88d5v3Kk0YC off=0 len=1534687 2022-04-22T17:51:03.542 INFO:tasks.workunit.client.0.smithi115.stdout:op 1315 completed, throughput=5MB/sec 2022-04-22T17:51:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:03 smithi121 bash[36902]: cluster 2022-04-22T17:51:02.376386+0000 mgr.y (mgr.34107) 906 : cluster [DBG] pgmap v1530: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:03 smithi115 bash[45495]: cluster 2022-04-22T17:51:02.376386+0000 mgr.y (mgr.34107) 906 : cluster [DBG] pgmap v1530: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:03 smithi115 bash[46934]: cluster 2022-04-22T17:51:02.376386+0000 mgr.y (mgr.34107) 906 : cluster [DBG] pgmap v1530: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:04.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:03.568Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:04.542 INFO:tasks.workunit.client.0.smithi115.stdout: 254: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:04.542 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eC4IgOKJa3F5xs5 off=8177172 len=444281 2022-04-22T17:51:04.547 INFO:tasks.workunit.client.0.smithi115.stdout:op 1316 completed, throughput=4.98MB/sec 2022-04-22T17:51:04.548 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xxx0shzLqdQJaCS off=11583134 len=1419570 2022-04-22T17:51:04.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 1317 completed, throughput=4.99MB/sec 2022-04-22T17:51:04.556 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hHZ9ceMqxDXQFBH off=3443118 len=601830 2022-04-22T17:51:04.561 INFO:tasks.workunit.client.0.smithi115.stdout:op 1318 completed, throughput=4.99MB/sec 2022-04-22T17:51:04.561 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PaCB--o2CNpgkc3 off=6316411 len=510368 2022-04-22T17:51:04.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 1319 completed, throughput=4.99MB/sec 2022-04-22T17:51:04.565 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-amJmU7FXDgwN9Mg off=270649 len=1848758 2022-04-22T17:51:04.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 1320 completed, throughput=5MB/sec 2022-04-22T17:51:05.568 INFO:tasks.workunit.client.0.smithi115.stdout: 255: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:05.601 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CSpc8yV7hG0-teQ off=902074 len=1456074 2022-04-22T17:51:05.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 1321 completed, throughput=4.99MB/sec 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-So3sgzaZX2js7IM off=3148001 len=4172 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 1322 completed, throughput=4.99MB/sec 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-V4vJsOIQSZL7eHo off=13226954 len=988017 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 1323 completed, throughput=4.99MB/sec 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jVBMx9iBSKKTxNr off=642206 len=203548 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 1324 completed, throughput=4.99MB/sec 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1EhAnYB5JrWji-T off=2794191 len=631192 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 1325 completed, throughput=4.99MB/sec 2022-04-22T17:51:05.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z5s-oJKtemJBj9g off=280454 len=688693 2022-04-22T17:51:05.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 1326 completed, throughput=5MB/sec 2022-04-22T17:51:05.604 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GZa1UE_x2qRwj_- off=1956220 len=1662482 2022-04-22T17:51:05.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 1327 completed, throughput=5MB/sec 2022-04-22T17:51:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:05 smithi121 bash[36902]: cluster 2022-04-22T17:51:04.377476+0000 mgr.y (mgr.34107) 907 : cluster [DBG] pgmap v1531: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:05 smithi121 bash[36902]: audit 2022-04-22T17:51:04.930998+0000 mon.a (mon.0) 2323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:05 smithi121 bash[36902]: audit 2022-04-22T17:51:05.238190+0000 mon.a (mon.0) 2324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[45495]: cluster 2022-04-22T17:51:04.377476+0000 mgr.y (mgr.34107) 907 : cluster [DBG] pgmap v1531: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[45495]: audit 2022-04-22T17:51:04.930998+0000 mon.a (mon.0) 2323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[45495]: audit 2022-04-22T17:51:05.238190+0000 mon.a (mon.0) 2324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[46934]: cluster 2022-04-22T17:51:04.377476+0000 mgr.y (mgr.34107) 907 : cluster [DBG] pgmap v1531: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[46934]: audit 2022-04-22T17:51:04.930998+0000 mon.a (mon.0) 2323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:05 smithi115 bash[46934]: audit 2022-04-22T17:51:05.238190+0000 mon.a (mon.0) 2324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:06.595 INFO:tasks.workunit.client.0.smithi115.stdout: 256: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:06.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0MlBwyKncL5ZmjS off=5633247 len=1093016 2022-04-22T17:51:06.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 1328 completed, throughput=4.99MB/sec 2022-04-22T17:51:06.602 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8SVBNLdOjkvnd29 off=15676074 len=1857314 2022-04-22T17:51:06.611 INFO:tasks.workunit.client.0.smithi115.stdout:op 1329 completed, throughput=4.99MB/sec 2022-04-22T17:51:06.611 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jVle4PuKIInCyVP off=1099362 len=307198 2022-04-22T17:51:06.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 1330 completed, throughput=4.99MB/sec 2022-04-22T17:51:06.614 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tsgfnic3Gnf0ICW off=6404141 len=711209 2022-04-22T17:51:06.616 INFO:tasks.workunit.client.0.smithi115.stdout:op 1331 completed, throughput=5MB/sec 2022-04-22T17:51:06.616 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ByQOfGILQIJRj6H off=3212977 len=134559 2022-04-22T17:51:06.619 INFO:tasks.workunit.client.0.smithi115.stdout:op 1332 completed, throughput=5MB/sec 2022-04-22T17:51:06.619 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PWHi1BEEmNaYzSD off=11997195 len=1237437 2022-04-22T17:51:06.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 1333 completed, throughput=5MB/sec 2022-04-22T17:51:07.626 INFO:tasks.workunit.client.0.smithi115.stdout: 257: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:07.627 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W3YSQDztqjm4e7l off=1779177 len=1816076 2022-04-22T17:51:07.630 INFO:tasks.workunit.client.0.smithi115.stdout:op 1334 completed, throughput=4.99MB/sec 2022-04-22T17:51:07.630 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2X_GlX237WPJ9Ky off=1176534 len=1910021 2022-04-22T17:51:07.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 1335 completed, throughput=5MB/sec 2022-04-22T17:51:07.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--kQNgnFR1GkV5TU off=71497 len=2035537 2022-04-22T17:51:07.642 INFO:tasks.workunit.client.0.smithi115.stdout:op 1336 completed, throughput=5MB/sec 2022-04-22T17:51:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:07 smithi121 bash[36902]: cluster 2022-04-22T17:51:06.378517+0000 mgr.y (mgr.34107) 908 : cluster [DBG] pgmap v1532: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:07 smithi121 bash[36902]: audit 2022-04-22T17:51:07.311137+0000 mon.a (mon.0) 2325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:07 smithi115 bash[45495]: cluster 2022-04-22T17:51:06.378517+0000 mgr.y (mgr.34107) 908 : cluster [DBG] pgmap v1532: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:07 smithi115 bash[45495]: audit 2022-04-22T17:51:07.311137+0000 mon.a (mon.0) 2325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:07 smithi115 bash[46934]: cluster 2022-04-22T17:51:06.378517+0000 mgr.y (mgr.34107) 908 : cluster [DBG] pgmap v1532: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:07 smithi115 bash[46934]: audit 2022-04-22T17:51:07.311137+0000 mon.a (mon.0) 2325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:08.642 INFO:tasks.workunit.client.0.smithi115.stdout: 258: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:08.643 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eUOtjyygTTMrzKd off=5519907 len=1606839 2022-04-22T17:51:08.651 INFO:tasks.workunit.client.0.smithi115.stdout:op 1337 completed, throughput=4.99MB/sec 2022-04-22T17:51:08.651 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sGFBAkpnRo8P4W_ off=14829879 len=1958260 2022-04-22T17:51:08.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 1338 completed, throughput=5MB/sec 2022-04-22T17:51:08.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h5FEqfPVDYpwdoA off=4979635 len=1801528 2022-04-22T17:51:08.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 1339 completed, throughput=5MB/sec 2022-04-22T17:51:09.664 INFO:tasks.workunit.client.0.smithi115.stdout: 259: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:09.664 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BZaHEbQU3V-NH2H off=881699 len=1247075 2022-04-22T17:51:09.671 INFO:tasks.workunit.client.0.smithi115.stdout:op 1340 completed, throughput=4.99MB/sec 2022-04-22T17:51:09.671 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Aj0xNjb7YTpL2kK off=13938446 len=1250648 2022-04-22T17:51:09.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 1341 completed, throughput=4.99MB/sec 2022-04-22T17:51:09.673 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7YWUDtoqXv9KAMM off=9380906 len=17677 2022-04-22T17:51:09.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 1342 completed, throughput=4.99MB/sec 2022-04-22T17:51:09.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HPlDuNPENPxjISj off=19902487 len=1661682 2022-04-22T17:51:09.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 1343 completed, throughput=5MB/sec 2022-04-22T17:51:09.679 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nWdf39EH9iYe9Lf off=3783038 len=227114 2022-04-22T17:51:09.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 1344 completed, throughput=5MB/sec 2022-04-22T17:51:09.683 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eZdRdOaQqR8IdB8 off=7255533 len=314642 2022-04-22T17:51:09.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 1345 completed, throughput=5MB/sec 2022-04-22T17:51:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:09 smithi121 bash[36902]: cluster 2022-04-22T17:51:08.379643+0000 mgr.y (mgr.34107) 909 : cluster [DBG] pgmap v1533: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:10.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:09 smithi115 bash[45495]: cluster 2022-04-22T17:51:08.379643+0000 mgr.y (mgr.34107) 909 : cluster [DBG] pgmap v1533: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:10.264 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:09 smithi115 bash[46934]: cluster 2022-04-22T17:51:08.379643+0000 mgr.y (mgr.34107) 909 : cluster [DBG] pgmap v1533: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:51:10.684 INFO:tasks.workunit.client.0.smithi115.stdout: 260: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:10.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MFJPRjTjplgD3_y off=1484796 len=1785389 2022-04-22T17:51:10.690 INFO:tasks.workunit.client.0.smithi115.stdout:op 1346 completed, throughput=4.99MB/sec 2022-04-22T17:51:10.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2ox6Gya7kvzgvRR off=9556440 len=1483718 2022-04-22T17:51:10.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 1347 completed, throughput=4.99MB/sec 2022-04-22T17:51:10.718 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bCAPCq1ewIn7AQc off=7007433 len=14470 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 1348 completed, throughput=4.99MB/sec 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HL_t7udGVDXZpy0 off=11095857 len=703963 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 1349 completed, throughput=5MB/sec 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L_PB7Ep4zddV3zl off=4806414 len=899591 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 1350 completed, throughput=5MB/sec 2022-04-22T17:51:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UwnhtpsGmrFN6V7 off=14819949 len=210649 2022-04-22T17:51:10.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 1351 completed, throughput=5MB/sec 2022-04-22T17:51:10.720 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PXvk0uwjNi0BjLJ off=16725550 len=1572855 2022-04-22T17:51:10.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 1352 completed, throughput=5MB/sec 2022-04-22T17:51:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:10 smithi121 bash[36902]: audit 2022-04-22T17:51:09.934186+0000 mon.a (mon.0) 2326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:10 smithi121 bash[36902]: audit 2022-04-22T17:51:10.241878+0000 mon.a (mon.0) 2327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:11.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:10 smithi115 bash[45495]: audit 2022-04-22T17:51:09.934186+0000 mon.a (mon.0) 2326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:10 smithi115 bash[45495]: audit 2022-04-22T17:51:10.241878+0000 mon.a (mon.0) 2327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:10 smithi115 bash[46934]: audit 2022-04-22T17:51:09.934186+0000 mon.a (mon.0) 2326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:10 smithi115 bash[46934]: audit 2022-04-22T17:51:10.241878+0000 mon.a (mon.0) 2327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:11.715 INFO:tasks.workunit.client.0.smithi115.stdout: 261: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:11.716 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RcG8gWYRnNY6WGd off=4351703 len=867385 2022-04-22T17:51:11.722 INFO:tasks.workunit.client.0.smithi115.stdout:op 1353 completed, throughput=4.99MB/sec 2022-04-22T17:51:11.722 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VYkC1kpi-Uep-bx off=1450567 len=1660595 2022-04-22T17:51:11.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 1354 completed, throughput=4.99MB/sec 2022-04-22T17:51:11.725 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qez-q8hpWfjpS45 off=3527767 len=948984 2022-04-22T17:51:11.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 1355 completed, throughput=5MB/sec 2022-04-22T17:51:11.733 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Twd8xSe-6GVE1Eb off=6101722 len=1120220 2022-04-22T17:51:11.739 INFO:tasks.workunit.client.0.smithi115.stdout:op 1356 completed, throughput=5MB/sec 2022-04-22T17:51:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:11 smithi121 bash[36902]: cluster 2022-04-22T17:51:10.380602+0000 mgr.y (mgr.34107) 910 : cluster [DBG] pgmap v1534: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:11 smithi115 bash[45495]: cluster 2022-04-22T17:51:10.380602+0000 mgr.y (mgr.34107) 910 : cluster [DBG] pgmap v1534: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:11 smithi115 bash[46934]: cluster 2022-04-22T17:51:10.380602+0000 mgr.y (mgr.34107) 910 : cluster [DBG] pgmap v1534: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:12.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:11.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:12.740 INFO:tasks.workunit.client.0.smithi115.stdout: 262: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:12.740 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ECuUBPDqqo6d6sp off=4218201 len=64581 2022-04-22T17:51:12.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 1357 completed, throughput=4.98MB/sec 2022-04-22T17:51:12.741 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xo9xBZhwAX_CUHw off=7284746 len=1028392 2022-04-22T17:51:12.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 1358 completed, throughput=4.99MB/sec 2022-04-22T17:51:12.744 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sh_T3k-XCQBff7d off=10555785 len=677175 2022-04-22T17:51:12.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 1359 completed, throughput=4.99MB/sec 2022-04-22T17:51:12.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eBdaO6zkuzdxQba off=7808923 len=1814984 2022-04-22T17:51:12.756 INFO:tasks.workunit.client.0.smithi115.stdout:op 1360 completed, throughput=5MB/sec 2022-04-22T17:51:12.756 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y0EZefMh-cWkPS_ off=9057604 len=2005647 2022-04-22T17:51:12.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 1361 completed, throughput=5MB/sec 2022-04-22T17:51:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:12 smithi121 bash[36902]: audit 2022-04-22T17:51:12.314349+0000 mon.a (mon.0) 2328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:12 smithi115 bash[45495]: audit 2022-04-22T17:51:12.314349+0000 mon.a (mon.0) 2328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:13.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:12 smithi115 bash[46934]: audit 2022-04-22T17:51:12.314349+0000 mon.a (mon.0) 2328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:13.765 INFO:tasks.workunit.client.0.smithi115.stdout: 263: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:13.765 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X_OkHJSU9G5rvjo off=642690 len=1739816 2022-04-22T17:51:13.769 INFO:tasks.workunit.client.0.smithi115.stdout:op 1362 completed, throughput=4.99MB/sec 2022-04-22T17:51:13.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DAo6R-0zs2bLmk4 off=4068962 len=1375627 2022-04-22T17:51:13.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 1363 completed, throughput=5MB/sec 2022-04-22T17:51:13.776 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-re0uXCC28ag_AxO off=23267233 len=1443327 2022-04-22T17:51:13.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 1364 completed, throughput=5MB/sec 2022-04-22T17:51:14.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:13 smithi121 bash[36902]: cluster 2022-04-22T17:51:12.381693+0000 mgr.y (mgr.34107) 911 : cluster [DBG] pgmap v1535: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:51:14.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:13 smithi121 bash[36902]: audit 2022-04-22T17:51:13.478446+0000 mon.a (mon.0) 2329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:51:14.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:13 smithi121 bash[36902]: audit 2022-04-22T17:51:13.483597+0000 mon.a (mon.0) 2330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:51:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:13.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[45495]: cluster 2022-04-22T17:51:12.381693+0000 mgr.y (mgr.34107) 911 : cluster [DBG] pgmap v1535: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[45495]: audit 2022-04-22T17:51:13.478446+0000 mon.a (mon.0) 2329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[45495]: audit 2022-04-22T17:51:13.483597+0000 mon.a (mon.0) 2330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[46934]: cluster 2022-04-22T17:51:12.381693+0000 mgr.y (mgr.34107) 911 : cluster [DBG] pgmap v1535: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[46934]: audit 2022-04-22T17:51:13.478446+0000 mon.a (mon.0) 2329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:51:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:13 smithi115 bash[46934]: audit 2022-04-22T17:51:13.483597+0000 mon.a (mon.0) 2330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:51:14.779 INFO:tasks.workunit.client.0.smithi115.stdout: 264: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:14.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TeqrnYnDZZ0So2a off=10157096 len=1863278 2022-04-22T17:51:14.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 1365 completed, throughput=4.99MB/sec 2022-04-22T17:51:14.788 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--LpvD7280U7HK08 off=8865836 len=458920 2022-04-22T17:51:14.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 1366 completed, throughput=4.99MB/sec 2022-04-22T17:51:14.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nmexH9M-bK1aymB off=7469952 len=1066427 2022-04-22T17:51:14.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 1367 completed, throughput=4.99MB/sec 2022-04-22T17:51:14.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZPPlOZhxLHXQgqZ off=5874253 len=270911 2022-04-22T17:51:14.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 1368 completed, throughput=4.99MB/sec 2022-04-22T17:51:14.794 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sxf7eM0qgaZ4ytZ off=3825778 len=2067134 2022-04-22T17:51:14.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 1369 completed, throughput=5MB/sec 2022-04-22T17:51:15.801 INFO:tasks.workunit.client.0.smithi115.stdout: 265: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:15.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6qKqliUz9sAM2G8 off=5043678 len=273865 2022-04-22T17:51:15.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 1370 completed, throughput=4.98MB/sec 2022-04-22T17:51:15.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4lkIZDH3NPk_hDV off=1622122 len=5525 2022-04-22T17:51:15.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 1371 completed, throughput=4.98MB/sec 2022-04-22T17:51:15.807 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o0_Azi-BWPCBWuF off=14950617 len=742285 2022-04-22T17:51:15.813 INFO:tasks.workunit.client.0.smithi115.stdout:op 1372 completed, throughput=4.99MB/sec 2022-04-22T17:51:15.813 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UFDDBaVRuHhb0gz off=4335667 len=1128307 2022-04-22T17:51:15.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 1373 completed, throughput=4.99MB/sec 2022-04-22T17:51:15.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f39DVuwffRY-1V4 off=816588 len=1813949 2022-04-22T17:51:15.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 1374 completed, throughput=5MB/sec 2022-04-22T17:51:15.829 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dB8xFyISlTgjduW off=609618 len=26860 2022-04-22T17:51:15.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 1375 completed, throughput=5MB/sec 2022-04-22T17:51:15.830 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fCrHhwxwzZymI8E off=7849535 len=2043233 2022-04-22T17:51:15.832 INFO:tasks.workunit.client.0.smithi115.stdout:op 1376 completed, throughput=5MB/sec 2022-04-22T17:51:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:15 smithi121 bash[36902]: cluster 2022-04-22T17:51:14.382816+0000 mgr.y (mgr.34107) 912 : cluster [DBG] pgmap v1536: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:15 smithi121 bash[36902]: audit 2022-04-22T17:51:14.937301+0000 mon.a (mon.0) 2331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:15 smithi121 bash[36902]: audit 2022-04-22T17:51:15.245717+0000 mon.a (mon.0) 2332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[46934]: cluster 2022-04-22T17:51:14.382816+0000 mgr.y (mgr.34107) 912 : cluster [DBG] pgmap v1536: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[46934]: audit 2022-04-22T17:51:14.937301+0000 mon.a (mon.0) 2331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[46934]: audit 2022-04-22T17:51:15.245717+0000 mon.a (mon.0) 2332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[45495]: cluster 2022-04-22T17:51:14.382816+0000 mgr.y (mgr.34107) 912 : cluster [DBG] pgmap v1536: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[45495]: audit 2022-04-22T17:51:14.937301+0000 mon.a (mon.0) 2331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:15 smithi115 bash[45495]: audit 2022-04-22T17:51:15.245717+0000 mon.a (mon.0) 2332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:16.832 INFO:tasks.workunit.client.0.smithi115.stdout: 266: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:16.833 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j02YfOnznTTHxgd off=953493 len=357022 2022-04-22T17:51:16.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 1377 completed, throughput=4.99MB/sec 2022-04-22T17:51:16.835 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cnrO-EW8qL1shel off=12275019 len=1396368 2022-04-22T17:51:16.843 INFO:tasks.workunit.client.0.smithi115.stdout:op 1378 completed, throughput=4.99MB/sec 2022-04-22T17:51:16.843 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vppelyxfv7M6L14 off=4590602 len=327605 2022-04-22T17:51:16.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 1379 completed, throughput=4.99MB/sec 2022-04-22T17:51:16.844 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZURFM3M1yMeCH5k off=1339422 len=1851009 2022-04-22T17:51:16.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 1380 completed, throughput=5MB/sec 2022-04-22T17:51:16.850 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vYdp2u87-yCLxJZ off=6113238 len=1132969 2022-04-22T17:51:16.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 1381 completed, throughput=5MB/sec 2022-04-22T17:51:17.856 INFO:tasks.workunit.client.0.smithi115.stdout: 267: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:17.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4qHvaojok1q-clT off=13156262 len=1471514 2022-04-22T17:51:17.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 1382 completed, throughput=4.99MB/sec 2022-04-22T17:51:17.866 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QPLdDFVmZlHdILp off=13074367 len=1171656 2022-04-22T17:51:17.867 INFO:tasks.workunit.client.0.smithi115.stdout:op 1383 completed, throughput=4.99MB/sec 2022-04-22T17:51:17.867 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yCCNMJdXs4bahoD off=16796912 len=714753 2022-04-22T17:51:17.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 1384 completed, throughput=5MB/sec 2022-04-22T17:51:17.874 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nJrtN9NmSzAaoPw off=23980498 len=676571 2022-04-22T17:51:17.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 1385 completed, throughput=5MB/sec 2022-04-22T17:51:17.879 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GrzOsOqql6KWR98 off=7809752 len=1796762 2022-04-22T17:51:17.888 INFO:tasks.workunit.client.0.smithi115.stdout:op 1386 completed, throughput=5MB/sec 2022-04-22T17:51:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:17 smithi121 bash[36902]: cluster 2022-04-22T17:51:16.383987+0000 mgr.y (mgr.34107) 913 : cluster [DBG] pgmap v1537: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:17 smithi121 bash[36902]: audit 2022-04-22T17:51:17.317349+0000 mon.a (mon.0) 2333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:17 smithi115 bash[45495]: cluster 2022-04-22T17:51:16.383987+0000 mgr.y (mgr.34107) 913 : cluster [DBG] pgmap v1537: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:17 smithi115 bash[45495]: audit 2022-04-22T17:51:17.317349+0000 mon.a (mon.0) 2333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:17 smithi115 bash[46934]: cluster 2022-04-22T17:51:16.383987+0000 mgr.y (mgr.34107) 913 : cluster [DBG] pgmap v1537: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:51:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:17 smithi115 bash[46934]: audit 2022-04-22T17:51:17.317349+0000 mon.a (mon.0) 2333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:18.888 INFO:tasks.workunit.client.0.smithi115.stdout: 268: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:18.889 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ghMP66D2YIhCLcu off=20016020 len=748182 2022-04-22T17:51:18.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 1387 completed, throughput=4.99MB/sec 2022-04-22T17:51:18.891 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XR4Dt4TsXBTN585 off=367312 len=1460955 2022-04-22T17:51:18.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 1388 completed, throughput=4.99MB/sec 2022-04-22T17:51:18.900 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3a4upcjn9ZP9X2u off=6922140 len=949131 2022-04-22T17:51:18.906 INFO:tasks.workunit.client.0.smithi115.stdout:op 1389 completed, throughput=5MB/sec 2022-04-22T17:51:18.906 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fT-py88R2O7_xuG off=465840 len=1759656 2022-04-22T17:51:18.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 1390 completed, throughput=5MB/sec 2022-04-22T17:51:19.920 INFO:tasks.workunit.client.0.smithi115.stdout: 269: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:19.920 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_VJxbnF_BfklUI- off=3530525 len=1502756 2022-04-22T17:51:19.925 INFO:tasks.workunit.client.0.smithi115.stdout:op 1391 completed, throughput=4.99MB/sec 2022-04-22T17:51:19.925 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GrzOsOqql6KWR98 off=3893723 len=882734 2022-04-22T17:51:19.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 1392 completed, throughput=4.99MB/sec 2022-04-22T17:51:19.926 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cQTyEs1Z9_d76BF off=1392179 len=172127 2022-04-22T17:51:19.927 INFO:tasks.workunit.client.0.smithi115.stdout:op 1393 completed, throughput=4.99MB/sec 2022-04-22T17:51:19.927 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wEwukQrs1IKaEpQ off=767286 len=753073 2022-04-22T17:51:19.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 1394 completed, throughput=5MB/sec 2022-04-22T17:51:19.929 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ADGI7VB885D_bNB off=691727 len=1726840 2022-04-22T17:51:19.935 INFO:tasks.workunit.client.0.smithi115.stdout:op 1395 completed, throughput=5MB/sec 2022-04-22T17:51:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:19 smithi121 bash[36902]: cluster 2022-04-22T17:51:18.385136+0000 mgr.y (mgr.34107) 914 : cluster [DBG] pgmap v1538: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:51:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:19 smithi115 bash[45495]: cluster 2022-04-22T17:51:18.385136+0000 mgr.y (mgr.34107) 914 : cluster [DBG] pgmap v1538: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:51:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:19 smithi115 bash[46934]: cluster 2022-04-22T17:51:18.385136+0000 mgr.y (mgr.34107) 914 : cluster [DBG] pgmap v1538: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:51:20.943 INFO:tasks.workunit.client.0.smithi115.stdout: 270: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:20.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r1f_eyRlbNSG3wX off=847285 len=421569 2022-04-22T17:51:20.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 1396 completed, throughput=4.98MB/sec 2022-04-22T17:51:20.944 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lsuwOxmoGyORZ3w off=3080413 len=713322 2022-04-22T17:51:20.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 1397 completed, throughput=4.99MB/sec 2022-04-22T17:51:20.944 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5GFsInV9spRnHnx off=1097329 len=488542 2022-04-22T17:51:20.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 1398 completed, throughput=4.99MB/sec 2022-04-22T17:51:20.949 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z_7wnqo0S_wX6sV off=2802008 len=1035768 2022-04-22T17:51:20.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 1399 completed, throughput=4.99MB/sec 2022-04-22T17:51:20.955 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1GH2CTT8mIaPEU3 off=1697645 len=1288341 2022-04-22T17:51:20.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 1400 completed, throughput=5MB/sec 2022-04-22T17:51:20.958 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v88M_biwG9Bh5XO off=2848019 len=1707531 2022-04-22T17:51:20.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 1401 completed, throughput=5MB/sec 2022-04-22T17:51:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:20 smithi121 bash[36902]: audit 2022-04-22T17:51:19.940135+0000 mon.a (mon.0) 2334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:20 smithi121 bash[36902]: audit 2022-04-22T17:51:20.249489+0000 mon.a (mon.0) 2335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:21.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:20 smithi115 bash[45495]: audit 2022-04-22T17:51:19.940135+0000 mon.a (mon.0) 2334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:21.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:20 smithi115 bash[45495]: audit 2022-04-22T17:51:20.249489+0000 mon.a (mon.0) 2335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:20 smithi115 bash[46934]: audit 2022-04-22T17:51:19.940135+0000 mon.a (mon.0) 2334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:20 smithi115 bash[46934]: audit 2022-04-22T17:51:20.249489+0000 mon.a (mon.0) 2335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:21.966 INFO:tasks.workunit.client.0.smithi115.stdout: 271: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:21.967 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-buxz0rJmmyFsRD6 off=1734608 len=1787863 2022-04-22T17:51:21.977 INFO:tasks.workunit.client.0.smithi115.stdout:op 1402 completed, throughput=4.99MB/sec 2022-04-22T17:51:21.977 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BwsuyX3xCVGQVuH off=10576344 len=1934054 2022-04-22T17:51:21.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 1403 completed, throughput=5MB/sec 2022-04-22T17:51:21.987 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iLOg-WHF88-8jmy off=19307808 len=603174 2022-04-22T17:51:21.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 1404 completed, throughput=5MB/sec 2022-04-22T17:51:21.988 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MKj4hBHie0VLjuY off=1543392 len=154048 2022-04-22T17:51:21.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 1405 completed, throughput=5MB/sec 2022-04-22T17:51:21.993 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yP1naFBBlwmukf1 off=2481054 len=1987652 2022-04-22T17:51:22.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 1406 completed, throughput=5.01MB/sec 2022-04-22T17:51:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:21 smithi121 bash[36902]: cluster 2022-04-22T17:51:20.386239+0000 mgr.y (mgr.34107) 915 : cluster [DBG] pgmap v1539: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:21 smithi115 bash[46934]: cluster 2022-04-22T17:51:20.386239+0000 mgr.y (mgr.34107) 915 : cluster [DBG] pgmap v1539: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:22.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:21 smithi115 bash[45495]: cluster 2022-04-22T17:51:20.386239+0000 mgr.y (mgr.34107) 915 : cluster [DBG] pgmap v1539: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:22.082 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:21.687Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:23.000 INFO:tasks.workunit.client.0.smithi115.stdout: 272: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:23.000 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-82ZKnKShpSvbrrK off=22715439 len=1233451 2022-04-22T17:51:23.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 1407 completed, throughput=4.99MB/sec 2022-04-22T17:51:23.008 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bZqq-Y4OLGoLK-u off=165264 len=1551774 2022-04-22T17:51:23.011 INFO:tasks.workunit.client.0.smithi115.stdout:op 1408 completed, throughput=5MB/sec 2022-04-22T17:51:23.012 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jWKH74jkqocen37 off=10464789 len=1843214 2022-04-22T17:51:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:22 smithi121 bash[36902]: audit 2022-04-22T17:51:22.320358+0000 mon.a (mon.0) 2336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:23.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 1409 completed, throughput=5MB/sec 2022-04-22T17:51:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:22 smithi115 bash[45495]: audit 2022-04-22T17:51:22.320358+0000 mon.a (mon.0) 2336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:22 smithi115 bash[46934]: audit 2022-04-22T17:51:22.320358+0000 mon.a (mon.0) 2336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:23 smithi121 bash[36902]: cluster 2022-04-22T17:51:22.387356+0000 mgr.y (mgr.34107) 916 : cluster [DBG] pgmap v1540: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:24.017 INFO:tasks.workunit.client.0.smithi115.stdout: 273: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:24.017 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xfOqplxVGsdlPMS off=4546170 len=1843899 2022-04-22T17:51:24.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 1410 completed, throughput=4.99MB/sec 2022-04-22T17:51:24.021 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5PMq6WWn-bgl_J7 off=942719 len=1763300 2022-04-22T17:51:24.033 INFO:tasks.workunit.client.0.smithi115.stdout:op 1411 completed, throughput=5MB/sec 2022-04-22T17:51:24.033 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xMu4uZJd6oFEmEh off=1384858 len=1321526 2022-04-22T17:51:24.039 INFO:tasks.workunit.client.0.smithi115.stdout:op 1412 completed, throughput=5MB/sec 2022-04-22T17:51:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:23 smithi115 bash[46934]: cluster 2022-04-22T17:51:22.387356+0000 mgr.y (mgr.34107) 916 : cluster [DBG] pgmap v1540: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:24.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:23.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:23 smithi115 bash[45495]: cluster 2022-04-22T17:51:22.387356+0000 mgr.y (mgr.34107) 916 : cluster [DBG] pgmap v1540: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:25.040 INFO:tasks.workunit.client.0.smithi115.stdout: 274: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:25.040 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X9q2EP5NzrvDYb- off=3340193 len=1622430 2022-04-22T17:51:25.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 1413 completed, throughput=4.99MB/sec 2022-04-22T17:51:25.045 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iAa0D4Li2KTgE0y off=2568061 len=1950603 2022-04-22T17:51:25.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 1414 completed, throughput=5MB/sec 2022-04-22T17:51:25.054 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C3lTI0JJYQBi7Mt off=4974386 len=400064 2022-04-22T17:51:25.059 INFO:tasks.workunit.client.0.smithi115.stdout:op 1415 completed, throughput=5MB/sec 2022-04-22T17:51:25.059 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_npQ2Wm3ODQDv6H off=1645884 len=1076276 2022-04-22T17:51:25.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 1416 completed, throughput=5MB/sec 2022-04-22T17:51:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:25 smithi121 bash[36902]: cluster 2022-04-22T17:51:24.388406+0000 mgr.y (mgr.34107) 917 : cluster [DBG] pgmap v1541: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:25 smithi121 bash[36902]: audit 2022-04-22T17:51:24.942822+0000 mon.a (mon.0) 2337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:25 smithi121 bash[36902]: audit 2022-04-22T17:51:25.253328+0000 mon.a (mon.0) 2338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:26.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[45495]: cluster 2022-04-22T17:51:24.388406+0000 mgr.y (mgr.34107) 917 : cluster [DBG] pgmap v1541: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[45495]: audit 2022-04-22T17:51:24.942822+0000 mon.a (mon.0) 2337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[45495]: audit 2022-04-22T17:51:25.253328+0000 mon.a (mon.0) 2338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[46934]: cluster 2022-04-22T17:51:24.388406+0000 mgr.y (mgr.34107) 917 : cluster [DBG] pgmap v1541: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[46934]: audit 2022-04-22T17:51:24.942822+0000 mon.a (mon.0) 2337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:25 smithi115 bash[46934]: audit 2022-04-22T17:51:25.253328+0000 mon.a (mon.0) 2338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:26.062 INFO:tasks.workunit.client.0.smithi115.stdout: 275: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:26.062 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xTCnxl2xJLJjUVn off=2933417 len=1928775 2022-04-22T17:51:26.073 INFO:tasks.workunit.client.0.smithi115.stdout:op 1417 completed, throughput=4.99MB/sec 2022-04-22T17:51:26.073 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LEPA3s4SqmWFgRP off=10140780 len=248196 2022-04-22T17:51:26.077 INFO:tasks.workunit.client.0.smithi115.stdout:op 1418 completed, throughput=4.99MB/sec 2022-04-22T17:51:26.077 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z3tlPCxsMInrQuw off=5860125 len=1670237 2022-04-22T17:51:26.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 1419 completed, throughput=5MB/sec 2022-04-22T17:51:26.081 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8EeiE1B0lgvLhlk off=535345 len=1489387 2022-04-22T17:51:26.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 1420 completed, throughput=5MB/sec 2022-04-22T17:51:27.089 INFO:tasks.workunit.client.0.smithi115.stdout: 276: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:27.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rJPYxxLzaw2b_qX off=188412 len=1881569 2022-04-22T17:51:27.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 1421 completed, throughput=4.99MB/sec 2022-04-22T17:51:27.092 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cQD-NeuxkJc01GG off=10070990 len=738932 2022-04-22T17:51:27.095 INFO:tasks.workunit.client.0.smithi115.stdout:op 1422 completed, throughput=4.99MB/sec 2022-04-22T17:51:27.095 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wzqxcmrBxwNDdMG off=3688943 len=431914 2022-04-22T17:51:27.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 1423 completed, throughput=4.99MB/sec 2022-04-22T17:51:27.097 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0LMES1BIjm6NtXY off=1851577 len=432187 2022-04-22T17:51:27.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 1424 completed, throughput=4.99MB/sec 2022-04-22T17:51:27.099 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NicbSF6--RNrmYL off=5052401 len=1651085 2022-04-22T17:51:27.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 1425 completed, throughput=5MB/sec 2022-04-22T17:51:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:27 smithi121 bash[36902]: cluster 2022-04-22T17:51:26.389544+0000 mgr.y (mgr.34107) 918 : cluster [DBG] pgmap v1542: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:27 smithi121 bash[36902]: audit 2022-04-22T17:51:27.323433+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:27 smithi115 bash[45495]: cluster 2022-04-22T17:51:26.389544+0000 mgr.y (mgr.34107) 918 : cluster [DBG] pgmap v1542: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:27 smithi115 bash[45495]: audit 2022-04-22T17:51:27.323433+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:27 smithi115 bash[46934]: cluster 2022-04-22T17:51:26.389544+0000 mgr.y (mgr.34107) 918 : cluster [DBG] pgmap v1542: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:27 smithi115 bash[46934]: audit 2022-04-22T17:51:27.323433+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:28.101 INFO:tasks.workunit.client.0.smithi115.stdout: 277: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:28.101 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NizmnJErrX7DNwE off=1547806 len=1325058 2022-04-22T17:51:28.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 1426 completed, throughput=4.99MB/sec 2022-04-22T17:51:28.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0626YnA-CRL3f9e off=6719635 len=710552 2022-04-22T17:51:28.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 1427 completed, throughput=4.99MB/sec 2022-04-22T17:51:28.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HWtd4EmxasYKhfb off=6810819 len=156865 2022-04-22T17:51:28.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 1428 completed, throughput=4.99MB/sec 2022-04-22T17:51:28.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0vj8pSGhCTxoYmX off=2466360 len=1129316 2022-04-22T17:51:28.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 1429 completed, throughput=4.99MB/sec 2022-04-22T17:51:28.118 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ctvolD4bRxu7eiM off=1881773 len=1418775 2022-04-22T17:51:28.120 INFO:tasks.workunit.client.0.smithi115.stdout:op 1430 completed, throughput=5MB/sec 2022-04-22T17:51:28.120 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XxmXN6yWwOR1oKf off=1438707 len=658264 2022-04-22T17:51:28.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 1431 completed, throughput=5MB/sec 2022-04-22T17:51:29.122 INFO:tasks.workunit.client.0.smithi115.stdout: 278: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:29.122 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jd-kr2R7DY1XHA8 off=684525 len=1199904 2022-04-22T17:51:29.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 1432 completed, throughput=4.99MB/sec 2022-04-22T17:51:29.125 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KK-YmV_2LOUIYFq off=28474 len=321581 2022-04-22T17:51:29.131 INFO:tasks.workunit.client.0.smithi115.stdout:op 1433 completed, throughput=4.99MB/sec 2022-04-22T17:51:29.131 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--b6azmVJcraCLry off=410552 len=1013978 2022-04-22T17:51:29.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 1434 completed, throughput=4.99MB/sec 2022-04-22T17:51:29.137 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M8hrtx9-gevLG6h off=9316583 len=1009194 2022-04-22T17:51:29.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 1435 completed, throughput=4.99MB/sec 2022-04-22T17:51:29.144 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P-0_tDhc5Y3DOCW off=575658 len=346042 2022-04-22T17:51:29.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 1436 completed, throughput=5MB/sec 2022-04-22T17:51:29.145 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XjR_tJsG2OUpxD5 off=679100 len=1392068 2022-04-22T17:51:29.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 1437 completed, throughput=5MB/sec 2022-04-22T17:51:29.151 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hbn1mTmaKHtpJOY off=2245081 len=115334 2022-04-22T17:51:29.154 INFO:tasks.workunit.client.0.smithi115.stdout:op 1438 completed, throughput=5MB/sec 2022-04-22T17:51:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:29 smithi121 bash[36902]: cluster 2022-04-22T17:51:28.390767+0000 mgr.y (mgr.34107) 919 : cluster [DBG] pgmap v1543: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:30.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:29 smithi115 bash[45495]: cluster 2022-04-22T17:51:28.390767+0000 mgr.y (mgr.34107) 919 : cluster [DBG] pgmap v1543: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:29 smithi115 bash[46934]: cluster 2022-04-22T17:51:28.390767+0000 mgr.y (mgr.34107) 919 : cluster [DBG] pgmap v1543: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:30.154 INFO:tasks.workunit.client.0.smithi115.stdout: 279: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:30.155 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d1D9hi9enexxMGd off=13870426 len=1906589 2022-04-22T17:51:30.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 1439 completed, throughput=4.99MB/sec 2022-04-22T17:51:30.165 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LsVehFepJFCwy_t off=6120228 len=328422 2022-04-22T17:51:30.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 1440 completed, throughput=4.99MB/sec 2022-04-22T17:51:30.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yp3rNYtJZMvL2w0 off=2045231 len=1634262 2022-04-22T17:51:30.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 1441 completed, throughput=5MB/sec 2022-04-22T17:51:30.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yCCNMJdXs4bahoD off=15105754 len=1063120 2022-04-22T17:51:30.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 1442 completed, throughput=5MB/sec 2022-04-22T17:51:30.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qbiHp82xJOTwmhS off=8690306 len=644305 2022-04-22T17:51:30.176 INFO:tasks.workunit.client.0.smithi115.stdout:op 1443 completed, throughput=5MB/sec 2022-04-22T17:51:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:30 smithi121 bash[36902]: audit 2022-04-22T17:51:29.945569+0000 mon.a (mon.0) 2340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:30 smithi121 bash[36902]: audit 2022-04-22T17:51:30.257288+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:30 smithi115 bash[45495]: audit 2022-04-22T17:51:29.945569+0000 mon.a (mon.0) 2340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:30 smithi115 bash[45495]: audit 2022-04-22T17:51:30.257288+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:30 smithi115 bash[46934]: audit 2022-04-22T17:51:29.945569+0000 mon.a (mon.0) 2340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:30 smithi115 bash[46934]: audit 2022-04-22T17:51:30.257288+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:31.177 INFO:tasks.workunit.client.0.smithi115.stdout: 280: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:31.177 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lnt1Fi3pUKb-BT_ off=5444975 len=1935623 2022-04-22T17:51:31.186 INFO:tasks.workunit.client.0.smithi115.stdout:op 1444 completed, throughput=4.99MB/sec 2022-04-22T17:51:31.186 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZwD7yPifKskn1lE off=9776023 len=1250389 2022-04-22T17:51:31.188 INFO:tasks.workunit.client.0.smithi115.stdout:op 1445 completed, throughput=4.99MB/sec 2022-04-22T17:51:31.188 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b71_u-7ZWURwFMn off=12794187 len=1766462 2022-04-22T17:51:31.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 1446 completed, throughput=5MB/sec 2022-04-22T17:51:31.197 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e8ZAnmBk2ZVmTPm off=598634 len=1242329 2022-04-22T17:51:31.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 1447 completed, throughput=5MB/sec 2022-04-22T17:51:31.881 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:51:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:31 smithi121 bash[36902]: cluster 2022-04-22T17:51:30.391816+0000 mgr.y (mgr.34107) 920 : cluster [DBG] pgmap v1544: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:51:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:31.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:31 smithi115 bash[45495]: cluster 2022-04-22T17:51:30.391816+0000 mgr.y (mgr.34107) 920 : cluster [DBG] pgmap v1544: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:51:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:31 smithi115 bash[46934]: cluster 2022-04-22T17:51:30.391816+0000 mgr.y (mgr.34107) 920 : cluster [DBG] pgmap v1544: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 3 op/s 2022-04-22T17:51:32.202 INFO:tasks.workunit.client.0.smithi115.stdout: 281: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:32.202 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KTG9hrbBB-OEYQB off=2622167 len=1533690 2022-04-22T17:51:32.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 1448 completed, throughput=4.99MB/sec 2022-04-22T17:51:32.212 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KmydLV_dpgY1PdT off=2517083 len=635267 2022-04-22T17:51:32.213 INFO:tasks.workunit.client.0.smithi115.stdout:op 1449 completed, throughput=4.99MB/sec 2022-04-22T17:51:32.213 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MNGBOr8MYvDP0zd off=6402530 len=2067186 2022-04-22T17:51:32.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 1450 completed, throughput=5MB/sec 2022-04-22T17:51:32.216 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rQTLTQhVEDFdnpZ off=417554 len=1752548 2022-04-22T17:51:32.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 1451 completed, throughput=5.01MB/sec 2022-04-22T17:51:32.226 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (14m) 3m ago 28m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (28m) 5m ago 28m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (16m) 3m ago 29m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (16m) 5m ago 29m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (23m) 5m ago 33m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:51:32.227 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (24m) 3m ago 35m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (23m) 3m ago 36m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (22m) 5m ago 34m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (22m) 3m ago 34m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (15m) 3m ago 28m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (15m) 5m ago 28m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (21m) 3m ago 33m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (20m) 3m ago 32m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (20m) 3m ago 32m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (19m) 3m ago 31m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:51:32.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (19m) 5m ago 31m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:51:32.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (18m) 5m ago 30m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:51:32.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (17m) 5m ago 30m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:51:32.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (16m) 5m ago 29m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:51:32.229 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (15m) 5m ago 28m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:51:32.630 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:51:32.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:51:32.632 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:51:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:32 smithi121 bash[36902]: audit 2022-04-22T17:51:31.857970+0000 mgr.y (mgr.34107) 921 : audit [DBG] from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:32 smithi121 bash[36902]: audit 2022-04-22T17:51:32.195756+0000 mgr.y (mgr.34107) 922 : audit [DBG] from='client.45319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:32 smithi121 bash[36902]: audit 2022-04-22T17:51:32.326572+0000 mon.a (mon.0) 2342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:32 smithi121 bash[36902]: audit 2022-04-22T17:51:32.622258+0000 mon.a (mon.0) 2343 : audit [DBG] from='client.? 172.21.15.115:0/4210264050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[45495]: audit 2022-04-22T17:51:31.857970+0000 mgr.y (mgr.34107) 921 : audit [DBG] from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[45495]: audit 2022-04-22T17:51:32.195756+0000 mgr.y (mgr.34107) 922 : audit [DBG] from='client.45319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[45495]: audit 2022-04-22T17:51:32.326572+0000 mon.a (mon.0) 2342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[45495]: audit 2022-04-22T17:51:32.622258+0000 mon.a (mon.0) 2343 : audit [DBG] from='client.? 172.21.15.115:0/4210264050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[46934]: audit 2022-04-22T17:51:31.857970+0000 mgr.y (mgr.34107) 921 : audit [DBG] from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[46934]: audit 2022-04-22T17:51:32.195756+0000 mgr.y (mgr.34107) 922 : audit [DBG] from='client.45319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:51:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[46934]: audit 2022-04-22T17:51:32.326572+0000 mon.a (mon.0) 2342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:32 smithi115 bash[46934]: audit 2022-04-22T17:51:32.622258+0000 mon.a (mon.0) 2343 : audit [DBG] from='client.? 172.21.15.115:0/4210264050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:51:33.219 INFO:tasks.workunit.client.0.smithi115.stdout: 282: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:33.219 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y41Q6PvoCS-MnqP off=10692060 len=1592661 2022-04-22T17:51:33.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 1452 completed, throughput=4.99MB/sec 2022-04-22T17:51:33.229 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XvSgwQIptlInd0q off=18607551 len=639989 2022-04-22T17:51:33.235 INFO:tasks.workunit.client.0.smithi115.stdout:op 1453 completed, throughput=5MB/sec 2022-04-22T17:51:33.235 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MQ9pm4-L0briooK off=4123570 len=985796 2022-04-22T17:51:33.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 1454 completed, throughput=5MB/sec 2022-04-22T17:51:33.241 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wys9mNWqEeOGVLb off=8790880 len=214944 2022-04-22T17:51:33.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 1455 completed, throughput=5MB/sec 2022-04-22T17:51:33.242 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-88hpdVeHGvGNUQE off=16100014 len=1183652 2022-04-22T17:51:33.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 1456 completed, throughput=5MB/sec 2022-04-22T17:51:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:33 smithi121 bash[36902]: cluster 2022-04-22T17:51:32.392871+0000 mgr.y (mgr.34107) 923 : cluster [DBG] pgmap v1545: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:34.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:33 smithi115 bash[45495]: cluster 2022-04-22T17:51:32.392871+0000 mgr.y (mgr.34107) 923 : cluster [DBG] pgmap v1545: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:33 smithi115 bash[46934]: cluster 2022-04-22T17:51:32.392871+0000 mgr.y (mgr.34107) 923 : cluster [DBG] pgmap v1545: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:34.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:33.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:34.248 INFO:tasks.workunit.client.0.smithi115.stdout: 283: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:34.248 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-68l4V00WaFTbWzP off=3432483 len=1584981 2022-04-22T17:51:34.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 1457 completed, throughput=4.99MB/sec 2022-04-22T17:51:34.256 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jks8yf73aH9El5x off=6260380 len=195364 2022-04-22T17:51:34.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 1458 completed, throughput=4.99MB/sec 2022-04-22T17:51:34.257 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eIZ3tgOE-29V-wp off=5664483 len=231567 2022-04-22T17:51:34.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 1459 completed, throughput=4.99MB/sec 2022-04-22T17:51:34.258 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5BwzXnANn0PsWBg off=12420440 len=1916232 2022-04-22T17:51:34.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 1460 completed, throughput=5MB/sec 2022-04-22T17:51:34.267 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oSPU_VBHzSoTz2x off=10272880 len=465041 2022-04-22T17:51:34.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 1461 completed, throughput=5MB/sec 2022-04-22T17:51:35.268 INFO:tasks.workunit.client.0.smithi115.stdout: 284: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:35.268 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hX90gIxF3gLngTl off=11369719 len=1488587 2022-04-22T17:51:35.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 1462 completed, throughput=4.99MB/sec 2022-04-22T17:51:35.271 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--FGVEyGxHocUyYd off=3853002 len=1297775 2022-04-22T17:51:35.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 1463 completed, throughput=4.99MB/sec 2022-04-22T17:51:35.279 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MptRw1537102Ks5 off=4210063 len=925406 2022-04-22T17:51:35.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 1464 completed, throughput=4.99MB/sec 2022-04-22T17:51:35.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z0VdOqWZ0aDHiis off=3905642 len=1859743 2022-04-22T17:51:35.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 1465 completed, throughput=5MB/sec 2022-04-22T17:51:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:35 smithi121 bash[36902]: cluster 2022-04-22T17:51:34.394029+0000 mgr.y (mgr.34107) 924 : cluster [DBG] pgmap v1546: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:35 smithi121 bash[36902]: audit 2022-04-22T17:51:34.948712+0000 mon.a (mon.0) 2344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:35 smithi121 bash[36902]: audit 2022-04-22T17:51:35.260935+0000 mon.a (mon.0) 2345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[45495]: cluster 2022-04-22T17:51:34.394029+0000 mgr.y (mgr.34107) 924 : cluster [DBG] pgmap v1546: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[45495]: audit 2022-04-22T17:51:34.948712+0000 mon.a (mon.0) 2344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[45495]: audit 2022-04-22T17:51:35.260935+0000 mon.a (mon.0) 2345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[46934]: cluster 2022-04-22T17:51:34.394029+0000 mgr.y (mgr.34107) 924 : cluster [DBG] pgmap v1546: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:51:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[46934]: audit 2022-04-22T17:51:34.948712+0000 mon.a (mon.0) 2344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:35 smithi115 bash[46934]: audit 2022-04-22T17:51:35.260935+0000 mon.a (mon.0) 2345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:36.284 INFO:tasks.workunit.client.0.smithi115.stdout: 285: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:36.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LlDKpymNlTTB0AU off=1532826 len=323587 2022-04-22T17:51:36.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 1466 completed, throughput=4.98MB/sec 2022-04-22T17:51:36.292 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nu1r79MBZZr7Zbq off=638079 len=326724 2022-04-22T17:51:36.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 1467 completed, throughput=4.99MB/sec 2022-04-22T17:51:36.292 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U7dMattX9mSfDt7 off=495386 len=1228413 2022-04-22T17:51:36.294 INFO:tasks.workunit.client.0.smithi115.stdout:op 1468 completed, throughput=4.99MB/sec 2022-04-22T17:51:36.294 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vaGmBVZyJYiRrCP off=5942989 len=1129705 2022-04-22T17:51:36.295 INFO:tasks.workunit.client.0.smithi115.stdout:op 1469 completed, throughput=4.99MB/sec 2022-04-22T17:51:36.295 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k4pjiZt6mk78ExB off=5667094 len=1409982 2022-04-22T17:51:36.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 1470 completed, throughput=5MB/sec 2022-04-22T17:51:36.298 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D_KV6bYlhnyvW_y off=10443076 len=1760142 2022-04-22T17:51:36.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 1471 completed, throughput=5MB/sec 2022-04-22T17:51:37.307 INFO:tasks.workunit.client.0.smithi115.stdout: 286: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:37.363 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2BnRteaW8e5Klgr off=10973704 len=419463 2022-04-22T17:51:37.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 1472 completed, throughput=4.99MB/sec 2022-04-22T17:51:37.364 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--ZdTNJ-UdYMxC69 off=7401419 len=1342256 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 1473 completed, throughput=4.99MB/sec 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZVC2jX_SMK9cZRD off=70560 len=60551 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 1474 completed, throughput=4.99MB/sec 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jdc9C1ATUa0rqVi off=11685276 len=1638612 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 1475 completed, throughput=5MB/sec 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NuLVUura5igivwP off=935539 len=255356 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 1476 completed, throughput=5MB/sec 2022-04-22T17:51:37.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VU7sFOgufkzoOsC off=21700241 len=1786101 2022-04-22T17:51:37.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 1477 completed, throughput=5MB/sec 2022-04-22T17:51:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:37 smithi121 bash[36902]: cluster 2022-04-22T17:51:36.395093+0000 mgr.y (mgr.34107) 925 : cluster [DBG] pgmap v1547: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:51:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:37 smithi121 bash[36902]: audit 2022-04-22T17:51:37.329085+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:37 smithi115 bash[45495]: cluster 2022-04-22T17:51:36.395093+0000 mgr.y (mgr.34107) 925 : cluster [DBG] pgmap v1547: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:51:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:37 smithi115 bash[45495]: audit 2022-04-22T17:51:37.329085+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:37 smithi115 bash[46934]: cluster 2022-04-22T17:51:36.395093+0000 mgr.y (mgr.34107) 925 : cluster [DBG] pgmap v1547: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:51:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:37 smithi115 bash[46934]: audit 2022-04-22T17:51:37.329085+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:38.334 INFO:tasks.workunit.client.0.smithi115.stdout: 287: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:38.335 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_lexw-UtDmM0VvK off=3066266 len=756822 2022-04-22T17:51:38.340 INFO:tasks.workunit.client.0.smithi115.stdout:op 1478 completed, throughput=4.99MB/sec 2022-04-22T17:51:38.340 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MIbMAQmVfMCu6vg off=17138543 len=587581 2022-04-22T17:51:38.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 1479 completed, throughput=4.99MB/sec 2022-04-22T17:51:38.343 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8iCQuur8pYFS-Ph off=13933053 len=747024 2022-04-22T17:51:38.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 1480 completed, throughput=4.99MB/sec 2022-04-22T17:51:38.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CKr29TAmcLFbCBu off=2034967 len=1408289 2022-04-22T17:51:38.350 INFO:tasks.workunit.client.0.smithi115.stdout:op 1481 completed, throughput=5MB/sec 2022-04-22T17:51:38.350 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y6jKDmHq6GSqQy3 off=16924547 len=1357702 2022-04-22T17:51:38.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 1482 completed, throughput=5MB/sec 2022-04-22T17:51:38.696 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[46934]: audit 2022-04-22T17:51:38.378826+0000 mon.a (mon.0) 2347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:51:38.696 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[46934]: audit 2022-04-22T17:51:38.381659+0000 mon.a (mon.0) 2348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:51:38.696 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[46934]: audit 2022-04-22T17:51:38.383241+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:51:38.696 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[45495]: audit 2022-04-22T17:51:38.378826+0000 mon.a (mon.0) 2347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:51:38.696 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[45495]: audit 2022-04-22T17:51:38.381659+0000 mon.a (mon.0) 2348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:51:38.697 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:38 smithi115 bash[45495]: audit 2022-04-22T17:51:38.383241+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:51:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:38 smithi121 bash[36902]: audit 2022-04-22T17:51:38.378826+0000 mon.a (mon.0) 2347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:51:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:38 smithi121 bash[36902]: audit 2022-04-22T17:51:38.381659+0000 mon.a (mon.0) 2348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:51:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:38 smithi121 bash[36902]: audit 2022-04-22T17:51:38.383241+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:51:39.352 INFO:tasks.workunit.client.0.smithi115.stdout: 288: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NSPCeivnfThE2_O off=1856817 len=105956 2022-04-22T17:51:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:op 1483 completed, throughput=4.99MB/sec 2022-04-22T17:51:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nsv-Jh-NzNe0a3l off=8216228 len=81253 2022-04-22T17:51:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 1484 completed, throughput=4.99MB/sec 2022-04-22T17:51:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oS3yXgEdBkSJBWt off=16394107 len=1518576 2022-04-22T17:51:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 1485 completed, throughput=4.99MB/sec 2022-04-22T17:51:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zBej6olSO1O_ksA off=3961600 len=455796 2022-04-22T17:51:39.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 1486 completed, throughput=4.99MB/sec 2022-04-22T17:51:39.367 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8l_SQbewcmPfurX off=1993778 len=395253 2022-04-22T17:51:39.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 1487 completed, throughput=4.99MB/sec 2022-04-22T17:51:39.372 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yOW0GxSW7BTNo9z off=1381059 len=1812033 2022-04-22T17:51:39.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 1488 completed, throughput=5MB/sec 2022-04-22T17:51:39.379 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uJTg1rkTjuI3gnN off=220328 len=481547 2022-04-22T17:51:39.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 1489 completed, throughput=5MB/sec 2022-04-22T17:51:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:39 smithi121 bash[36902]: cluster 2022-04-22T17:51:38.396150+0000 mgr.y (mgr.34107) 926 : cluster [DBG] pgmap v1548: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:39 smithi121 bash[36902]: audit 2022-04-22T17:51:38.727275+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:51:40.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:39 smithi115 bash[45495]: cluster 2022-04-22T17:51:38.396150+0000 mgr.y (mgr.34107) 926 : cluster [DBG] pgmap v1548: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:40.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:39 smithi115 bash[45495]: audit 2022-04-22T17:51:38.727275+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:51:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:39 smithi115 bash[46934]: cluster 2022-04-22T17:51:38.396150+0000 mgr.y (mgr.34107) 926 : cluster [DBG] pgmap v1548: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:39 smithi115 bash[46934]: audit 2022-04-22T17:51:38.727275+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:51:40.384 INFO:tasks.workunit.client.0.smithi115.stdout: 289: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:40.385 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PWsv1AdRgNlvJyZ off=16475640 len=1948792 2022-04-22T17:51:40.389 INFO:tasks.workunit.client.0.smithi115.stdout:op 1490 completed, throughput=4.99MB/sec 2022-04-22T17:51:40.389 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ifJw31EBbBuyPlM off=13791115 len=1945059 2022-04-22T17:51:40.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 1491 completed, throughput=5MB/sec 2022-04-22T17:51:40.391 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yc3jqk-ULZN3LWp off=4217001 len=216810 2022-04-22T17:51:40.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 1492 completed, throughput=5MB/sec 2022-04-22T17:51:40.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-T8c1IoRq1O_rBg2 off=1559324 len=1425922 2022-04-22T17:51:40.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 1493 completed, throughput=5MB/sec 2022-04-22T17:51:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:40 smithi121 bash[36902]: audit 2022-04-22T17:51:39.951619+0000 mon.a (mon.0) 2351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:40 smithi121 bash[36902]: audit 2022-04-22T17:51:40.264525+0000 mon.a (mon.0) 2352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:40 smithi115 bash[45495]: audit 2022-04-22T17:51:39.951619+0000 mon.a (mon.0) 2351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:40 smithi115 bash[45495]: audit 2022-04-22T17:51:40.264525+0000 mon.a (mon.0) 2352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:40 smithi115 bash[46934]: audit 2022-04-22T17:51:39.951619+0000 mon.a (mon.0) 2351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:40 smithi115 bash[46934]: audit 2022-04-22T17:51:40.264525+0000 mon.a (mon.0) 2352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:41.399 INFO:tasks.workunit.client.0.smithi115.stdout: 290: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:41.399 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tzY9l3C2MIUVQKs off=2638797 len=1153702 2022-04-22T17:51:41.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 1494 completed, throughput=4.99MB/sec 2022-04-22T17:51:41.403 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IdJ9Dyccz2znpaA off=219410 len=1015723 2022-04-22T17:51:41.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 1495 completed, throughput=4.99MB/sec 2022-04-22T17:51:41.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WtdsI2k6F3VimrQ off=800139 len=721285 2022-04-22T17:51:41.408 INFO:tasks.workunit.client.0.smithi115.stdout:op 1496 completed, throughput=4.99MB/sec 2022-04-22T17:51:41.408 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--l5bkyxqaOjntV0 off=8318074 len=1928532 2022-04-22T17:51:41.416 INFO:tasks.workunit.client.0.smithi115.stdout:op 1497 completed, throughput=5MB/sec 2022-04-22T17:51:41.416 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HL_t7udGVDXZpy0 off=14684390 len=1143702 2022-04-22T17:51:41.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 1498 completed, throughput=5MB/sec 2022-04-22T17:51:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:41 smithi121 bash[36902]: cluster 2022-04-22T17:51:40.396972+0000 mgr.y (mgr.34107) 927 : cluster [DBG] pgmap v1549: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:41.687Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:41 smithi115 bash[45495]: cluster 2022-04-22T17:51:40.396972+0000 mgr.y (mgr.34107) 927 : cluster [DBG] pgmap v1549: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:41 smithi115 bash[46934]: cluster 2022-04-22T17:51:40.396972+0000 mgr.y (mgr.34107) 927 : cluster [DBG] pgmap v1549: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:42.424 INFO:tasks.workunit.client.0.smithi115.stdout: 291: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:42.424 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8gEsX5bQ2iZ83io off=8621997 len=689508 2022-04-22T17:51:42.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 1499 completed, throughput=4.99MB/sec 2022-04-22T17:51:42.426 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SYZ_dh_wUbPkbgK off=1748983 len=1698170 2022-04-22T17:51:42.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 1500 completed, throughput=4.99MB/sec 2022-04-22T17:51:42.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2HisR9MLBVUiSj2 off=2961377 len=2041680 2022-04-22T17:51:42.448 INFO:tasks.workunit.client.0.smithi115.stdout:op 1501 completed, throughput=5MB/sec 2022-04-22T17:51:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:42 smithi121 bash[36902]: audit 2022-04-22T17:51:42.331710+0000 mon.a (mon.0) 2353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:42 smithi115 bash[45495]: audit 2022-04-22T17:51:42.331710+0000 mon.a (mon.0) 2353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:42 smithi115 bash[46934]: audit 2022-04-22T17:51:42.331710+0000 mon.a (mon.0) 2353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:43.448 INFO:tasks.workunit.client.0.smithi115.stdout: 292: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:43.448 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XyABFOG0a4y74Wj off=7988061 len=1778174 2022-04-22T17:51:43.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 1502 completed, throughput=4.99MB/sec 2022-04-22T17:51:43.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8fTcUwgoKCykOQl off=860404 len=159512 2022-04-22T17:51:43.453 INFO:tasks.workunit.client.0.smithi115.stdout:op 1503 completed, throughput=4.99MB/sec 2022-04-22T17:51:43.453 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BdRyVT4WlID6uz1 off=11386630 len=2004544 2022-04-22T17:51:43.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 1504 completed, throughput=5MB/sec 2022-04-22T17:51:43.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--kmTyByT4o_gAUX off=5780889 len=2073939 2022-04-22T17:51:43.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 1505 completed, throughput=5MB/sec 2022-04-22T17:51:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:43 smithi121 bash[36902]: cluster 2022-04-22T17:51:42.398052+0000 mgr.y (mgr.34107) 928 : cluster [DBG] pgmap v1550: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:43 smithi115 bash[45495]: cluster 2022-04-22T17:51:42.398052+0000 mgr.y (mgr.34107) 928 : cluster [DBG] pgmap v1550: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:44.096 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:43.572Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:44.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:43 smithi115 bash[46934]: cluster 2022-04-22T17:51:42.398052+0000 mgr.y (mgr.34107) 928 : cluster [DBG] pgmap v1550: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:44.467 INFO:tasks.workunit.client.0.smithi115.stdout: 293: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:44.467 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dTmi1BlS5oB3D08 off=242175 len=1408030 2022-04-22T17:51:44.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 1506 completed, throughput=4.99MB/sec 2022-04-22T17:51:44.475 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qD095AvrT-nqsQu off=5382103 len=299654 2022-04-22T17:51:44.479 INFO:tasks.workunit.client.0.smithi115.stdout:op 1507 completed, throughput=4.99MB/sec 2022-04-22T17:51:44.479 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fANowjc_RldpHHl off=3539007 len=2073507 2022-04-22T17:51:44.484 INFO:tasks.workunit.client.0.smithi115.stdout:op 1508 completed, throughput=5MB/sec 2022-04-22T17:51:44.484 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-67Rt9e7EJVoZC46 off=18490407 len=736324 2022-04-22T17:51:44.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 1509 completed, throughput=5MB/sec 2022-04-22T17:51:44.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-g_shbAEXsMbendU off=11014917 len=842628 2022-04-22T17:51:44.494 INFO:tasks.workunit.client.0.smithi115.stdout:op 1510 completed, throughput=5MB/sec 2022-04-22T17:51:45.495 INFO:tasks.workunit.client.0.smithi115.stdout: 294: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:45.495 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tazBmvgmv8O_diK off=4626559 len=591892 2022-04-22T17:51:45.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 1511 completed, throughput=4.99MB/sec 2022-04-22T17:51:45.501 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8pezobbdq0yOvvo off=707770 len=901095 2022-04-22T17:51:45.504 INFO:tasks.workunit.client.0.smithi115.stdout:op 1512 completed, throughput=4.99MB/sec 2022-04-22T17:51:45.504 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IqBej0I2rdFI1pF off=820606 len=322713 2022-04-22T17:51:45.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 1513 completed, throughput=4.99MB/sec 2022-04-22T17:51:45.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lHxX0c5JPqxem8C off=593296 len=8274 2022-04-22T17:51:45.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 1514 completed, throughput=4.99MB/sec 2022-04-22T17:51:45.511 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AK4GDKiDr_b_6pi off=1665190 len=1507045 2022-04-22T17:51:45.515 INFO:tasks.workunit.client.0.smithi115.stdout:op 1515 completed, throughput=5MB/sec 2022-04-22T17:51:45.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u3ie0UMDBQPYyfN off=960386 len=176382 2022-04-22T17:51:45.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 1516 completed, throughput=5MB/sec 2022-04-22T17:51:45.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wBgeXvFfgKqtpEz off=5145133 len=1812156 2022-04-22T17:51:45.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 1517 completed, throughput=5MB/sec 2022-04-22T17:51:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:45 smithi121 bash[36902]: cluster 2022-04-22T17:51:44.399102+0000 mgr.y (mgr.34107) 929 : cluster [DBG] pgmap v1551: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:45 smithi121 bash[36902]: audit 2022-04-22T17:51:44.954349+0000 mon.a (mon.0) 2354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:45 smithi121 bash[36902]: audit 2022-04-22T17:51:45.268299+0000 mon.a (mon.0) 2355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[45495]: cluster 2022-04-22T17:51:44.399102+0000 mgr.y (mgr.34107) 929 : cluster [DBG] pgmap v1551: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[45495]: audit 2022-04-22T17:51:44.954349+0000 mon.a (mon.0) 2354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[45495]: audit 2022-04-22T17:51:45.268299+0000 mon.a (mon.0) 2355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:46.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[46934]: cluster 2022-04-22T17:51:44.399102+0000 mgr.y (mgr.34107) 929 : cluster [DBG] pgmap v1551: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:51:46.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[46934]: audit 2022-04-22T17:51:44.954349+0000 mon.a (mon.0) 2354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:46.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:45 smithi115 bash[46934]: audit 2022-04-22T17:51:45.268299+0000 mon.a (mon.0) 2355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:46.524 INFO:tasks.workunit.client.0.smithi115.stdout: 295: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:46.524 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dJpDfr7x1-msNu3 off=155952 len=1817800 2022-04-22T17:51:46.534 INFO:tasks.workunit.client.0.smithi115.stdout:op 1518 completed, throughput=4.99MB/sec 2022-04-22T17:51:46.534 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jKMQXOmsjEKDQqr off=1006379 len=435668 2022-04-22T17:51:46.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 1519 completed, throughput=4.99MB/sec 2022-04-22T17:51:46.539 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3iEM6ndlP5XxkYU off=4928778 len=1002708 2022-04-22T17:51:46.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 1520 completed, throughput=5MB/sec 2022-04-22T17:51:46.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7f9m4tnXVXTuyrE off=3379455 len=1582247 2022-04-22T17:51:46.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 1521 completed, throughput=5MB/sec 2022-04-22T17:51:47.549 INFO:tasks.workunit.client.0.smithi115.stdout: 296: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:47.549 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-me-DA9qL6q-neUw off=421656 len=12632 2022-04-22T17:51:47.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 1522 completed, throughput=4.98MB/sec 2022-04-22T17:51:47.550 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HXjLW6tuNZCn3Zu off=2933951 len=656559 2022-04-22T17:51:47.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 1523 completed, throughput=4.99MB/sec 2022-04-22T17:51:47.557 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Clza1DY1LnWyt6t off=19638459 len=1003348 2022-04-22T17:51:47.563 INFO:tasks.workunit.client.0.smithi115.stdout:op 1524 completed, throughput=4.99MB/sec 2022-04-22T17:51:47.564 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7pSW2S4rtIRHdXe off=2030098 len=2001592 2022-04-22T17:51:47.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 1525 completed, throughput=4.99MB/sec 2022-04-22T17:51:47.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W4svtHZqOX_TuzF off=11686150 len=1217824 2022-04-22T17:51:47.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 1526 completed, throughput=5MB/sec 2022-04-22T17:51:47.573 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TU1Oy3Nn6kHAPsE off=12846705 len=329891 2022-04-22T17:51:47.574 INFO:tasks.workunit.client.0.smithi115.stdout:op 1527 completed, throughput=5MB/sec 2022-04-22T17:51:47.574 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iWxxq53Dt1cy-aR off=1119995 len=2032798 2022-04-22T17:51:47.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 1528 completed, throughput=5.01MB/sec 2022-04-22T17:51:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:47 smithi115 bash[45495]: cluster 2022-04-22T17:51:46.400230+0000 mgr.y (mgr.34107) 930 : cluster [DBG] pgmap v1552: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:47 smithi115 bash[45495]: audit 2022-04-22T17:51:47.334645+0000 mon.a (mon.0) 2356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:47 smithi115 bash[46934]: cluster 2022-04-22T17:51:46.400230+0000 mgr.y (mgr.34107) 930 : cluster [DBG] pgmap v1552: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:47 smithi115 bash[46934]: audit 2022-04-22T17:51:47.334645+0000 mon.a (mon.0) 2356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:47 smithi121 bash[36902]: cluster 2022-04-22T17:51:46.400230+0000 mgr.y (mgr.34107) 930 : cluster [DBG] pgmap v1552: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:51:48.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:47 smithi121 bash[36902]: audit 2022-04-22T17:51:47.334645+0000 mon.a (mon.0) 2356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:48.584 INFO:tasks.workunit.client.0.smithi115.stdout: 297: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:48.584 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FO_hd2-qN5bJ3N- off=3371409 len=1996936 2022-04-22T17:51:48.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 1529 completed, throughput=5MB/sec 2022-04-22T17:51:48.591 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-elTleUblWJvFEcS off=305259 len=216911 2022-04-22T17:51:48.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 1530 completed, throughput=5MB/sec 2022-04-22T17:51:48.592 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-07irIg9N9tfC_no off=8759302 len=1003673 2022-04-22T17:51:48.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 1531 completed, throughput=5MB/sec 2022-04-22T17:51:48.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--68hlUk4t4FDScy off=412884 len=1301139 2022-04-22T17:51:48.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 1532 completed, throughput=5MB/sec 2022-04-22T17:51:49.602 INFO:tasks.workunit.client.0.smithi115.stdout: 298: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:49.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ajoI1MPD70vKfah off=3169427 len=1112960 2022-04-22T17:51:49.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 1533 completed, throughput=4.99MB/sec 2022-04-22T17:51:49.610 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n27ixGdRKKb1Exm off=5564420 len=68735 2022-04-22T17:51:49.611 INFO:tasks.workunit.client.0.smithi115.stdout:op 1534 completed, throughput=4.99MB/sec 2022-04-22T17:51:49.612 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zgGcgE5PRtAN1-N off=11077663 len=1735282 2022-04-22T17:51:49.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 1535 completed, throughput=5MB/sec 2022-04-22T17:51:49.615 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xy8GIe6OuXBi2R3 off=16393800 len=2042870 2022-04-22T17:51:49.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 1536 completed, throughput=5MB/sec 2022-04-22T17:51:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:49 smithi115 bash[45495]: cluster 2022-04-22T17:51:48.401368+0000 mgr.y (mgr.34107) 931 : cluster [DBG] pgmap v1553: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:49 smithi115 bash[46934]: cluster 2022-04-22T17:51:48.401368+0000 mgr.y (mgr.34107) 931 : cluster [DBG] pgmap v1553: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:49 smithi121 bash[36902]: cluster 2022-04-22T17:51:48.401368+0000 mgr.y (mgr.34107) 931 : cluster [DBG] pgmap v1553: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:51:50.626 INFO:tasks.workunit.client.0.smithi115.stdout: 299: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:50.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VO_mJnHI1UFijXv off=1913606 len=1192855 2022-04-22T17:51:50.633 INFO:tasks.workunit.client.0.smithi115.stdout:op 1537 completed, throughput=4.99MB/sec 2022-04-22T17:51:50.633 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-quiwKQQFzGNnzon off=20037796 len=1319044 2022-04-22T17:51:50.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 1538 completed, throughput=4.99MB/sec 2022-04-22T17:51:50.636 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HJxt7YwLTeNXW1A off=12727570 len=140361 2022-04-22T17:51:50.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 1539 completed, throughput=4.99MB/sec 2022-04-22T17:51:50.640 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-et33dvWWLGZk8pP off=4571697 len=1640774 2022-04-22T17:51:50.643 INFO:tasks.workunit.client.0.smithi115.stdout:op 1540 completed, throughput=5MB/sec 2022-04-22T17:51:50.643 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q_TizQVokhXowS9 off=13508081 len=1964641 2022-04-22T17:51:50.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 1541 completed, throughput=5MB/sec 2022-04-22T17:51:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:50 smithi115 bash[45495]: audit 2022-04-22T17:51:49.957443+0000 mon.a (mon.0) 2357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:50 smithi115 bash[45495]: audit 2022-04-22T17:51:50.271974+0000 mon.a (mon.0) 2358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:50 smithi115 bash[46934]: audit 2022-04-22T17:51:49.957443+0000 mon.a (mon.0) 2357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:50 smithi115 bash[46934]: audit 2022-04-22T17:51:50.271974+0000 mon.a (mon.0) 2358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:50 smithi121 bash[36902]: audit 2022-04-22T17:51:49.957443+0000 mon.a (mon.0) 2357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:50 smithi121 bash[36902]: audit 2022-04-22T17:51:50.271974+0000 mon.a (mon.0) 2358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:51.652 INFO:tasks.workunit.client.0.smithi115.stdout: 300: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:51.653 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ymknom4QrWUD8oW off=673580 len=2086585 2022-04-22T17:51:51.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 1542 completed, throughput=4.99MB/sec 2022-04-22T17:51:51.664 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-13t-9JzRonvtbcP off=1719825 len=1698648 2022-04-22T17:51:51.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 1543 completed, throughput=5MB/sec 2022-04-22T17:51:51.668 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VoIM6GhRM_eG8mg off=17596842 len=1519450 2022-04-22T17:51:51.670 INFO:tasks.workunit.client.0.smithi115.stdout:op 1544 completed, throughput=5MB/sec 2022-04-22T17:51:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:51 smithi115 bash[45495]: cluster 2022-04-22T17:51:50.402435+0000 mgr.y (mgr.34107) 932 : cluster [DBG] pgmap v1554: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:52.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:51.688Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:51 smithi115 bash[46934]: cluster 2022-04-22T17:51:50.402435+0000 mgr.y (mgr.34107) 932 : cluster [DBG] pgmap v1554: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:51 smithi121 bash[36902]: cluster 2022-04-22T17:51:50.402435+0000 mgr.y (mgr.34107) 932 : cluster [DBG] pgmap v1554: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:51:52.670 INFO:tasks.workunit.client.0.smithi115.stdout: 301: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:52.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BIZx-xIHWoymjEH off=831191 len=1186855 2022-04-22T17:51:52.678 INFO:tasks.workunit.client.0.smithi115.stdout:op 1545 completed, throughput=4.99MB/sec 2022-04-22T17:51:52.678 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1k-2nAuhHrBHAwn off=12733436 len=187020 2022-04-22T17:51:52.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 1546 completed, throughput=4.99MB/sec 2022-04-22T17:51:52.683 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pb0Qu-REcTDs8dP off=7375228 len=565572 2022-04-22T17:51:52.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 1547 completed, throughput=4.99MB/sec 2022-04-22T17:51:52.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PkdAX51HMPB3aHf off=1786546 len=1406176 2022-04-22T17:51:52.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 1548 completed, throughput=5MB/sec 2022-04-22T17:51:52.687 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FUaF_ft8AGVoHZ5 off=1059953 len=1743329 2022-04-22T17:51:52.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 1549 completed, throughput=5MB/sec 2022-04-22T17:51:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:52 smithi115 bash[45495]: audit 2022-04-22T17:51:52.337929+0000 mon.a (mon.0) 2359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:52 smithi115 bash[46934]: audit 2022-04-22T17:51:52.337929+0000 mon.a (mon.0) 2359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:52 smithi121 bash[36902]: audit 2022-04-22T17:51:52.337929+0000 mon.a (mon.0) 2359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:53.690 INFO:tasks.workunit.client.0.smithi115.stdout: 302: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:53.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nzgZCgwCs4KhjdJ off=5254212 len=33584 2022-04-22T17:51:53.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 1550 completed, throughput=4.99MB/sec 2022-04-22T17:51:53.694 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NcX5NtoLGCRNs0F off=9107079 len=772341 2022-04-22T17:51:53.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 1551 completed, throughput=4.99MB/sec 2022-04-22T17:51:53.700 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H7EA1lRYuJeCByJ off=1887643 len=1722243 2022-04-22T17:51:53.703 INFO:tasks.workunit.client.0.smithi115.stdout:op 1552 completed, throughput=5MB/sec 2022-04-22T17:51:53.703 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HDMp0DHYtxk4GNb off=3516726 len=1747575 2022-04-22T17:51:53.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 1553 completed, throughput=5MB/sec 2022-04-22T17:51:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:53 smithi115 bash[45495]: cluster 2022-04-22T17:51:52.403500+0000 mgr.y (mgr.34107) 933 : cluster [DBG] pgmap v1555: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:53 smithi115 bash[46934]: cluster 2022-04-22T17:51:52.403500+0000 mgr.y (mgr.34107) 933 : cluster [DBG] pgmap v1555: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:54.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:51:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:51:53.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:51:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:53 smithi121 bash[36902]: cluster 2022-04-22T17:51:52.403500+0000 mgr.y (mgr.34107) 933 : cluster [DBG] pgmap v1555: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:51:54.712 INFO:tasks.workunit.client.0.smithi115.stdout: 303: throughput=4.98MB/sec pending data=0 2022-04-22T17:51:54.713 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1nbRwAFOYdrbkII off=1957717 len=176728 2022-04-22T17:51:54.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 1554 completed, throughput=4.98MB/sec 2022-04-22T17:51:54.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VtgRW-eqFlgcpBQ off=4804462 len=877805 2022-04-22T17:51:54.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 1555 completed, throughput=4.99MB/sec 2022-04-22T17:51:54.720 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2ZUOXvfOVvHGSlW off=3331061 len=107364 2022-04-22T17:51:54.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 1556 completed, throughput=4.99MB/sec 2022-04-22T17:51:54.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7NHOLeGAX49oiMH off=19985619 len=572120 2022-04-22T17:51:54.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 1557 completed, throughput=4.99MB/sec 2022-04-22T17:51:54.725 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ABrtg2syagO0Jpo off=12581700 len=287866 2022-04-22T17:51:54.726 INFO:tasks.workunit.client.0.smithi115.stdout:op 1558 completed, throughput=4.99MB/sec 2022-04-22T17:51:54.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-byGoytaInQ2488k off=2311742 len=406827 2022-04-22T17:51:54.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 1559 completed, throughput=4.99MB/sec 2022-04-22T17:51:54.728 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FAF3Z2A4Tvp-ap1 off=10324444 len=1850802 2022-04-22T17:51:54.736 INFO:tasks.workunit.client.0.smithi115.stdout:op 1560 completed, throughput=5MB/sec 2022-04-22T17:51:54.736 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dSHHT74OI4gTr2E off=7412744 len=581684 2022-04-22T17:51:54.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 1561 completed, throughput=5MB/sec 2022-04-22T17:51:54.738 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AFDkCojob2xzpmJ off=5529808 len=333670 2022-04-22T17:51:54.742 INFO:tasks.workunit.client.0.smithi115.stdout:op 1562 completed, throughput=5MB/sec 2022-04-22T17:51:54.742 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sbwRkNG4xT3uT2w off=1547858 len=694426 2022-04-22T17:51:54.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 1563 completed, throughput=5MB/sec 2022-04-22T17:51:55.744 INFO:tasks.workunit.client.0.smithi115.stdout: 304: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:55.744 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3di7_IG9dwZ8PNt off=2850547 len=857313 2022-04-22T17:51:55.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 1564 completed, throughput=4.99MB/sec 2022-04-22T17:51:55.747 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bufwqu0-PkljMbc off=12882870 len=616761 2022-04-22T17:51:55.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 1565 completed, throughput=4.99MB/sec 2022-04-22T17:51:55.748 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dG1u-P0jVVgXW2x off=3715301 len=1839159 2022-04-22T17:51:55.760 INFO:tasks.workunit.client.0.smithi115.stdout:op 1566 completed, throughput=5MB/sec 2022-04-22T17:51:55.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nOUclJcQPMaTnP0 off=2886039 len=1903047 2022-04-22T17:51:55.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 1567 completed, throughput=5MB/sec 2022-04-22T17:51:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:55 smithi121 bash[36902]: cluster 2022-04-22T17:51:54.404566+0000 mgr.y (mgr.34107) 934 : cluster [DBG] pgmap v1556: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:51:56.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:55 smithi121 bash[36902]: audit 2022-04-22T17:51:54.960615+0000 mon.a (mon.0) 2360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:56.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:55 smithi121 bash[36902]: audit 2022-04-22T17:51:55.275762+0000 mon.a (mon.0) 2361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:56.385 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[45495]: cluster 2022-04-22T17:51:54.404566+0000 mgr.y (mgr.34107) 934 : cluster [DBG] pgmap v1556: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:51:56.385 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[45495]: audit 2022-04-22T17:51:54.960615+0000 mon.a (mon.0) 2360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:56.385 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[45495]: audit 2022-04-22T17:51:55.275762+0000 mon.a (mon.0) 2361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:56.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[46934]: cluster 2022-04-22T17:51:54.404566+0000 mgr.y (mgr.34107) 934 : cluster [DBG] pgmap v1556: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:51:56.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[46934]: audit 2022-04-22T17:51:54.960615+0000 mon.a (mon.0) 2360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:51:56.386 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:55 smithi115 bash[46934]: audit 2022-04-22T17:51:55.275762+0000 mon.a (mon.0) 2361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:51:56.762 INFO:tasks.workunit.client.0.smithi115.stdout: 305: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:56.777 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qNxmxwLHTzfDq_N off=8392756 len=1201592 2022-04-22T17:51:56.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 1568 completed, throughput=4.99MB/sec 2022-04-22T17:51:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lpBDMI1FtQ3Ol7e off=418677 len=863137 2022-04-22T17:51:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 1569 completed, throughput=4.99MB/sec 2022-04-22T17:51:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hHZ9ceMqxDXQFBH off=192951 len=335561 2022-04-22T17:51:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 1570 completed, throughput=4.99MB/sec 2022-04-22T17:51:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x2pa1xJ48W_pEJq off=3845487 len=336530 2022-04-22T17:51:56.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 1571 completed, throughput=4.99MB/sec 2022-04-22T17:51:56.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jX9bjCj9KJX3fD- off=7676991 len=1375658 2022-04-22T17:51:56.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 1572 completed, throughput=5MB/sec 2022-04-22T17:51:56.806 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z3Ljmxsx21RMFNn off=4513214 len=1978664 2022-04-22T17:51:56.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 1573 completed, throughput=5MB/sec 2022-04-22T17:51:57.794 INFO:tasks.workunit.client.0.smithi115.stdout: 306: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:57.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZZYOtYi4TF2rnnP off=5541070 len=1982924 2022-04-22T17:51:57.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 1574 completed, throughput=4.99MB/sec 2022-04-22T17:51:57.800 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7oySEZJex0HoTxF off=8511087 len=1250237 2022-04-22T17:51:57.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 1575 completed, throughput=5MB/sec 2022-04-22T17:51:57.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Yr2ZnxsXRPhUZeD off=4765085 len=1534265 2022-04-22T17:51:57.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 1576 completed, throughput=5MB/sec 2022-04-22T17:51:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:57 smithi121 bash[36902]: cluster 2022-04-22T17:51:56.405578+0000 mgr.y (mgr.34107) 935 : cluster [DBG] pgmap v1557: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:51:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:57 smithi121 bash[36902]: audit 2022-04-22T17:51:57.340941+0000 mon.a (mon.0) 2362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:57 smithi115 bash[45495]: cluster 2022-04-22T17:51:56.405578+0000 mgr.y (mgr.34107) 935 : cluster [DBG] pgmap v1557: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:51:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:57 smithi115 bash[45495]: audit 2022-04-22T17:51:57.340941+0000 mon.a (mon.0) 2362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:57 smithi115 bash[46934]: cluster 2022-04-22T17:51:56.405578+0000 mgr.y (mgr.34107) 935 : cluster [DBG] pgmap v1557: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:51:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:57 smithi115 bash[46934]: audit 2022-04-22T17:51:57.340941+0000 mon.a (mon.0) 2362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:51:58.812 INFO:tasks.workunit.client.0.smithi115.stdout: 307: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:58.812 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LGbOAAK8VoukYF4 off=21210513 len=1139612 2022-04-22T17:51:58.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 1577 completed, throughput=4.99MB/sec 2022-04-22T17:51:58.815 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uGikJREh2FVSX1o off=1244715 len=1160728 2022-04-22T17:51:58.817 INFO:tasks.workunit.client.0.smithi115.stdout:op 1578 completed, throughput=4.99MB/sec 2022-04-22T17:51:58.817 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QCKpOhXAzswjrw5 off=142244 len=1042801 2022-04-22T17:51:58.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 1579 completed, throughput=5MB/sec 2022-04-22T17:51:58.825 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D_KV6bYlhnyvW_y off=6536305 len=1697329 2022-04-22T17:51:58.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 1580 completed, throughput=5MB/sec 2022-04-22T17:51:59.830 INFO:tasks.workunit.client.0.smithi115.stdout: 308: throughput=4.99MB/sec pending data=0 2022-04-22T17:51:59.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IM-f7dCgI5zco8F off=3817638 len=628722 2022-04-22T17:51:59.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 1581 completed, throughput=4.99MB/sec 2022-04-22T17:51:59.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ioWfBr7tgVFNeCI off=0 len=1509541 2022-04-22T17:51:59.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 1582 completed, throughput=4.99MB/sec 2022-04-22T17:51:59.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-du5iBjUCIYc7uly off=9817682 len=1938803 2022-04-22T17:51:59.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 1583 completed, throughput=5MB/sec 2022-04-22T17:51:59.855 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vT1beBFoO7pi-77 off=7250978 len=613749 2022-04-22T17:51:59.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 1584 completed, throughput=5MB/sec 2022-04-22T17:51:59.859 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zTCNhzJK1R4lNqM off=18136450 len=845025 2022-04-22T17:51:59.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 1585 completed, throughput=5MB/sec 2022-04-22T17:52:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:51:59 smithi121 bash[36902]: cluster 2022-04-22T17:51:58.406734+0000 mgr.y (mgr.34107) 936 : cluster [DBG] pgmap v1558: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:51:59 smithi115 bash[45495]: cluster 2022-04-22T17:51:58.406734+0000 mgr.y (mgr.34107) 936 : cluster [DBG] pgmap v1558: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:51:59 smithi115 bash[46934]: cluster 2022-04-22T17:51:58.406734+0000 mgr.y (mgr.34107) 936 : cluster [DBG] pgmap v1558: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:00.861 INFO:tasks.workunit.client.0.smithi115.stdout: 309: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:00.861 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zAjjidvJ9XiXKly off=244303 len=868030 2022-04-22T17:52:00.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 1586 completed, throughput=4.99MB/sec 2022-04-22T17:52:00.868 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RPHkZeATC7rN4Ia off=169448 len=1392555 2022-04-22T17:52:00.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 1587 completed, throughput=4.99MB/sec 2022-04-22T17:52:00.874 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bVjPXrqpTTUCFRw off=2264277 len=301757 2022-04-22T17:52:00.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 1588 completed, throughput=4.99MB/sec 2022-04-22T17:52:00.875 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qp9bViEvdN1FyH8 off=10298602 len=1708051 2022-04-22T17:52:00.884 INFO:tasks.workunit.client.0.smithi115.stdout:op 1589 completed, throughput=5MB/sec 2022-04-22T17:52:00.884 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5jpSqac9nkoSFqv off=907995 len=1863466 2022-04-22T17:52:00.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 1590 completed, throughput=5MB/sec 2022-04-22T17:52:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:00 smithi121 bash[36902]: audit 2022-04-22T17:51:59.963767+0000 mon.a (mon.0) 2363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:00 smithi121 bash[36902]: audit 2022-04-22T17:52:00.279429+0000 mon.a (mon.0) 2364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:00 smithi115 bash[45495]: audit 2022-04-22T17:51:59.963767+0000 mon.a (mon.0) 2363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:00 smithi115 bash[45495]: audit 2022-04-22T17:52:00.279429+0000 mon.a (mon.0) 2364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:00 smithi115 bash[46934]: audit 2022-04-22T17:51:59.963767+0000 mon.a (mon.0) 2363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:00 smithi115 bash[46934]: audit 2022-04-22T17:52:00.279429+0000 mon.a (mon.0) 2364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:01.886 INFO:tasks.workunit.client.0.smithi115.stdout: 310: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:01.886 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VC_Ok2ionKCx09U off=215934 len=1101322 2022-04-22T17:52:01.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 1591 completed, throughput=4.99MB/sec 2022-04-22T17:52:01.889 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CL7wRiu9ucxlZCq off=2368208 len=1682478 2022-04-22T17:52:01.892 INFO:tasks.workunit.client.0.smithi115.stdout:op 1592 completed, throughput=5MB/sec 2022-04-22T17:52:01.892 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fkDq2txhSGoUZ0S off=7367858 len=1903449 2022-04-22T17:52:01.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 1593 completed, throughput=5MB/sec 2022-04-22T17:52:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:01 smithi115 bash[45495]: cluster 2022-04-22T17:52:00.407831+0000 mgr.y (mgr.34107) 937 : cluster [DBG] pgmap v1559: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:01.689Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:01 smithi115 bash[46934]: cluster 2022-04-22T17:52:00.407831+0000 mgr.y (mgr.34107) 937 : cluster [DBG] pgmap v1559: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:01 smithi121 bash[36902]: cluster 2022-04-22T17:52:00.407831+0000 mgr.y (mgr.34107) 937 : cluster [DBG] pgmap v1559: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:02.900 INFO:tasks.workunit.client.0.smithi115.stdout: 311: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:02.900 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rjQCuCRQeaiaVhr off=399144 len=1343845 2022-04-22T17:52:02.905 INFO:tasks.workunit.client.0.smithi115.stdout:op 1594 completed, throughput=4.99MB/sec 2022-04-22T17:52:02.905 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XRQpf2UkiLpeY8U off=924687 len=1739919 2022-04-22T17:52:02.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 1595 completed, throughput=5MB/sec 2022-04-22T17:52:02.914 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KPDSWMUbOMxPQtV off=2870780 len=2062407 2022-04-22T17:52:02.919 INFO:tasks.workunit.client.0.smithi115.stdout:op 1596 completed, throughput=5MB/sec 2022-04-22T17:52:03.010 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:52:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:02 smithi121 bash[36902]: audit 2022-04-22T17:52:02.344024+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:02 smithi115 bash[45495]: audit 2022-04-22T17:52:02.344024+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:02 smithi115 bash[46934]: audit 2022-04-22T17:52:02.344024+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:03.358 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (15m) 4m ago 28m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (28m) 5m ago 28m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (17m) 4m ago 29m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (16m) 5m ago 29m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (24m) 5m ago 34m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (25m) 4m ago 36m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (23m) 4m ago 36m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:52:03.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (23m) 5m ago 34m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (23m) 4m ago 34m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (16m) 4m ago 29m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (16m) 5m ago 29m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (21m) 4m ago 33m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (21m) 4m ago 33m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (20m) 4m ago 32m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (20m) 4m ago 32m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (19m) 5m ago 31m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (18m) 5m ago 31m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (18m) 5m ago 30m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:52:03.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (17m) 5m ago 30m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:52:03.361 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (15m) 5m ago 29m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:52:03.767 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:52:03.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:52:03.769 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:52:03.831 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:03.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:03.919 INFO:tasks.workunit.client.0.smithi115.stdout: 312: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:03.919 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TyRYdzGE8DFK2U4 off=11993812 len=1173509 2022-04-22T17:52:03.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 1597 completed, throughput=4.99MB/sec 2022-04-22T17:52:03.922 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KTulhFZGnPZ5iXZ off=10720416 len=1740124 2022-04-22T17:52:03.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 1598 completed, throughput=4.99MB/sec 2022-04-22T17:52:03.927 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7FHeA0pL4Z42-If off=13462525 len=298470 2022-04-22T17:52:03.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 1599 completed, throughput=4.99MB/sec 2022-04-22T17:52:03.928 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JrXQBtHoPq0ElqX off=14185414 len=245926 2022-04-22T17:52:03.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 1600 completed, throughput=5MB/sec 2022-04-22T17:52:03.929 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hs3dNN3NfT1G562 off=574950 len=195573 2022-04-22T17:52:03.930 INFO:tasks.workunit.client.0.smithi115.stdout:op 1601 completed, throughput=5MB/sec 2022-04-22T17:52:03.930 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IQZUvIY6p3_AyZZ off=12057274 len=1796211 2022-04-22T17:52:03.936 INFO:tasks.workunit.client.0.smithi115.stdout:op 1602 completed, throughput=5MB/sec 2022-04-22T17:52:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:03 smithi121 bash[36902]: cluster 2022-04-22T17:52:02.408977+0000 mgr.y (mgr.34107) 938 : cluster [DBG] pgmap v1560: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:03 smithi121 bash[36902]: audit 2022-04-22T17:52:02.986821+0000 mgr.y (mgr.34107) 939 : audit [DBG] from='client.45331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:03 smithi121 bash[36902]: audit 2022-04-22T17:52:03.329787+0000 mgr.y (mgr.34107) 940 : audit [DBG] from='client.35400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:03 smithi121 bash[36902]: audit 2022-04-22T17:52:03.759649+0000 mon.a (mon.0) 2366 : audit [DBG] from='client.? 172.21.15.115:0/1352172834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:04.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[45495]: cluster 2022-04-22T17:52:02.408977+0000 mgr.y (mgr.34107) 938 : cluster [DBG] pgmap v1560: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:04.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[45495]: audit 2022-04-22T17:52:02.986821+0000 mgr.y (mgr.34107) 939 : audit [DBG] from='client.45331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[45495]: audit 2022-04-22T17:52:03.329787+0000 mgr.y (mgr.34107) 940 : audit [DBG] from='client.35400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[45495]: audit 2022-04-22T17:52:03.759649+0000 mon.a (mon.0) 2366 : audit [DBG] from='client.? 172.21.15.115:0/1352172834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[46934]: cluster 2022-04-22T17:52:02.408977+0000 mgr.y (mgr.34107) 938 : cluster [DBG] pgmap v1560: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[46934]: audit 2022-04-22T17:52:02.986821+0000 mgr.y (mgr.34107) 939 : audit [DBG] from='client.45331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[46934]: audit 2022-04-22T17:52:03.329787+0000 mgr.y (mgr.34107) 940 : audit [DBG] from='client.35400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:03 smithi115 bash[46934]: audit 2022-04-22T17:52:03.759649+0000 mon.a (mon.0) 2366 : audit [DBG] from='client.? 172.21.15.115:0/1352172834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:04.937 INFO:tasks.workunit.client.0.smithi115.stdout: 313: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:04.937 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WIwkeb-lM8A_ZMm off=11231329 len=1050285 2022-04-22T17:52:04.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 1603 completed, throughput=4.99MB/sec 2022-04-22T17:52:04.939 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S_PWETkXKOvwFc2 off=8730834 len=913453 2022-04-22T17:52:04.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 1604 completed, throughput=4.99MB/sec 2022-04-22T17:52:04.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8ONw0nGV-kfe4My off=4600391 len=569553 2022-04-22T17:52:04.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 1605 completed, throughput=4.99MB/sec 2022-04-22T17:52:04.949 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ppHHCp-CtPM2fUx off=3973320 len=1532646 2022-04-22T17:52:04.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 1606 completed, throughput=5MB/sec 2022-04-22T17:52:04.996 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rGUW2Qq9BTb7DK1 off=3756564 len=1424606 2022-04-22T17:52:04.996 INFO:tasks.workunit.client.0.smithi115.stdout:op 1607 completed, throughput=5MB/sec 2022-04-22T17:52:05.959 INFO:tasks.workunit.client.0.smithi115.stdout: 314: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:05.959 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GdIAdz5BxkF-f-R off=15680344 len=1613140 2022-04-22T17:52:05.964 INFO:tasks.workunit.client.0.smithi115.stdout:op 1608 completed, throughput=4.99MB/sec 2022-04-22T17:52:05.964 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ax8qjpnBHOrPtVm off=8489083 len=1387141 2022-04-22T17:52:05.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 1609 completed, throughput=4.99MB/sec 2022-04-22T17:52:05.972 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NygtxiSpVG23g7q off=5308968 len=479848 2022-04-22T17:52:05.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 1610 completed, throughput=5MB/sec 2022-04-22T17:52:05.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C7j6zo8Yepdc05C off=0 len=1071158 2022-04-22T17:52:05.981 INFO:tasks.workunit.client.0.smithi115.stdout:op 1611 completed, throughput=5MB/sec 2022-04-22T17:52:05.981 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bx_vKBO_RItmKtK off=0 len=1493161 2022-04-22T17:52:05.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 1612 completed, throughput=5MB/sec 2022-04-22T17:52:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:05 smithi121 bash[36902]: cluster 2022-04-22T17:52:04.410074+0000 mgr.y (mgr.34107) 941 : cluster [DBG] pgmap v1561: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:05 smithi121 bash[36902]: audit 2022-04-22T17:52:04.966853+0000 mon.a (mon.0) 2367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:05 smithi121 bash[36902]: audit 2022-04-22T17:52:05.283186+0000 mon.a (mon.0) 2368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[45495]: cluster 2022-04-22T17:52:04.410074+0000 mgr.y (mgr.34107) 941 : cluster [DBG] pgmap v1561: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[45495]: audit 2022-04-22T17:52:04.966853+0000 mon.a (mon.0) 2367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[45495]: audit 2022-04-22T17:52:05.283186+0000 mon.a (mon.0) 2368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[46934]: cluster 2022-04-22T17:52:04.410074+0000 mgr.y (mgr.34107) 941 : cluster [DBG] pgmap v1561: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[46934]: audit 2022-04-22T17:52:04.966853+0000 mon.a (mon.0) 2367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:05 smithi115 bash[46934]: audit 2022-04-22T17:52:05.283186+0000 mon.a (mon.0) 2368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:06.989 INFO:tasks.workunit.client.0.smithi115.stdout: 315: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:06.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wTDJfR2eb547gMU off=3079301 len=119650 2022-04-22T17:52:06.990 INFO:tasks.workunit.client.0.smithi115.stdout:op 1613 completed, throughput=4.99MB/sec 2022-04-22T17:52:06.990 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gH_o-chiYpyF4G2 off=702139 len=228900 2022-04-22T17:52:06.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 1614 completed, throughput=4.99MB/sec 2022-04-22T17:52:06.991 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4SBbe7XRwYoJdcz off=1241088 len=345656 2022-04-22T17:52:06.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 1615 completed, throughput=4.99MB/sec 2022-04-22T17:52:06.994 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AsnNbodRcVr7V2k off=146508 len=423525 2022-04-22T17:52:06.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 1616 completed, throughput=4.99MB/sec 2022-04-22T17:52:06.995 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HHUgDkQkhpfecG2 off=2760155 len=1068912 2022-04-22T17:52:06.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 1617 completed, throughput=4.99MB/sec 2022-04-22T17:52:06.998 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jUAfj92bydPfvg4 off=10185657 len=172674 2022-04-22T17:52:06.999 INFO:tasks.workunit.client.0.smithi115.stdout:op 1618 completed, throughput=5MB/sec 2022-04-22T17:52:07.000 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-37QS9DM3IHS3rHP off=4005014 len=1795041 2022-04-22T17:52:07.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 1619 completed, throughput=5MB/sec 2022-04-22T17:52:08.009 INFO:tasks.workunit.client.0.smithi115.stdout: 316: throughput=4.98MB/sec pending data=0 2022-04-22T17:52:08.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8DjEZIhBsDvifhC off=8698944 len=1520441 2022-04-22T17:52:08.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 1620 completed, throughput=4.99MB/sec 2022-04-22T17:52:08.018 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e5fzgsuQNEPSOga off=16065027 len=2035741 2022-04-22T17:52:08.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 1621 completed, throughput=5MB/sec 2022-04-22T17:52:08.024 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OBO9cHxLmSoPfQD off=8343960 len=1394001 2022-04-22T17:52:08.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 1622 completed, throughput=5MB/sec 2022-04-22T17:52:08.027 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nuvnm4DD53yZfqh off=691961 len=800749 2022-04-22T17:52:08.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 1623 completed, throughput=5MB/sec 2022-04-22T17:52:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:07 smithi121 bash[36902]: cluster 2022-04-22T17:52:06.411116+0000 mgr.y (mgr.34107) 942 : cluster [DBG] pgmap v1562: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:08.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:07 smithi121 bash[36902]: audit 2022-04-22T17:52:07.346547+0000 mon.a (mon.0) 2369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:07 smithi115 bash[45495]: cluster 2022-04-22T17:52:06.411116+0000 mgr.y (mgr.34107) 942 : cluster [DBG] pgmap v1562: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:07 smithi115 bash[45495]: audit 2022-04-22T17:52:07.346547+0000 mon.a (mon.0) 2369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:07 smithi115 bash[46934]: cluster 2022-04-22T17:52:06.411116+0000 mgr.y (mgr.34107) 942 : cluster [DBG] pgmap v1562: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:07 smithi115 bash[46934]: audit 2022-04-22T17:52:07.346547+0000 mon.a (mon.0) 2369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:09.030 INFO:tasks.workunit.client.0.smithi115.stdout: 317: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:09.030 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s8b-sErq77Wd5Az off=390719 len=1990756 2022-04-22T17:52:09.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 1624 completed, throughput=4.99MB/sec 2022-04-22T17:52:09.035 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QMU890GZpOSoswY off=9858055 len=572459 2022-04-22T17:52:09.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 1625 completed, throughput=4.99MB/sec 2022-04-22T17:52:09.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FyCOFBcQqjZNRiS off=1606364 len=1838296 2022-04-22T17:52:09.046 INFO:tasks.workunit.client.0.smithi115.stdout:op 1626 completed, throughput=5MB/sec 2022-04-22T17:52:09.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GqKy3iHwSSVXHKJ off=2503682 len=492914 2022-04-22T17:52:09.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 1627 completed, throughput=5MB/sec 2022-04-22T17:52:10.051 INFO:tasks.workunit.client.0.smithi115.stdout: 318: throughput=4.98MB/sec pending data=0 2022-04-22T17:52:10.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BrmU-TJy4JKB7vy off=4967156 len=110624 2022-04-22T17:52:10.056 INFO:tasks.workunit.client.0.smithi115.stdout:op 1628 completed, throughput=4.98MB/sec 2022-04-22T17:52:10.056 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lpUu2OirUAip-oJ off=20464348 len=1861207 2022-04-22T17:52:10.059 INFO:tasks.workunit.client.0.smithi115.stdout:op 1629 completed, throughput=4.99MB/sec 2022-04-22T17:52:10.060 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ey2RYMlK-u3KIIy off=11117197 len=242326 2022-04-22T17:52:10.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 1630 completed, throughput=4.99MB/sec 2022-04-22T17:52:10.061 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nTrhqXEtVqS0u_M off=9507429 len=1211593 2022-04-22T17:52:10.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 1631 completed, throughput=4.99MB/sec 2022-04-22T17:52:10.063 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k30BJ-zA0BTMik5 off=4226333 len=1118451 2022-04-22T17:52:10.065 INFO:tasks.workunit.client.0.smithi115.stdout:op 1632 completed, throughput=5MB/sec 2022-04-22T17:52:10.065 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-USE4Um1tZjf8oqk off=13359822 len=1743767 2022-04-22T17:52:10.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 1633 completed, throughput=5MB/sec 2022-04-22T17:52:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:09 smithi121 bash[36902]: cluster 2022-04-22T17:52:08.412384+0000 mgr.y (mgr.34107) 943 : cluster [DBG] pgmap v1563: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:09 smithi115 bash[45495]: cluster 2022-04-22T17:52:08.412384+0000 mgr.y (mgr.34107) 943 : cluster [DBG] pgmap v1563: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:09 smithi115 bash[46934]: cluster 2022-04-22T17:52:08.412384+0000 mgr.y (mgr.34107) 943 : cluster [DBG] pgmap v1563: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:11.075 INFO:tasks.workunit.client.0.smithi115.stdout: 319: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:11.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xh1zPBJ8liWiK52 off=5673791 len=1086219 2022-04-22T17:52:11.082 INFO:tasks.workunit.client.0.smithi115.stdout:op 1634 completed, throughput=4.99MB/sec 2022-04-22T17:52:11.082 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MFgzS0Vazltt1JB off=12863973 len=1762725 2022-04-22T17:52:11.088 INFO:tasks.workunit.client.0.smithi115.stdout:op 1635 completed, throughput=5MB/sec 2022-04-22T17:52:11.088 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mzilGm7Xux2v3xs off=1065268 len=388627 2022-04-22T17:52:11.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 1636 completed, throughput=5MB/sec 2022-04-22T17:52:11.089 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zg6rpz8IAWykSAS off=5194758 len=1844541 2022-04-22T17:52:11.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 1637 completed, throughput=5MB/sec 2022-04-22T17:52:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:10 smithi121 bash[36902]: audit 2022-04-22T17:52:09.969894+0000 mon.a (mon.0) 2370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:10 smithi121 bash[36902]: audit 2022-04-22T17:52:10.286913+0000 mon.a (mon.0) 2371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:10 smithi115 bash[45495]: audit 2022-04-22T17:52:09.969894+0000 mon.a (mon.0) 2370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:10 smithi115 bash[45495]: audit 2022-04-22T17:52:10.286913+0000 mon.a (mon.0) 2371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:10 smithi115 bash[46934]: audit 2022-04-22T17:52:09.969894+0000 mon.a (mon.0) 2370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:10 smithi115 bash[46934]: audit 2022-04-22T17:52:10.286913+0000 mon.a (mon.0) 2371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:11 smithi115 bash[45495]: cluster 2022-04-22T17:52:10.413490+0000 mgr.y (mgr.34107) 944 : cluster [DBG] pgmap v1564: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:11 smithi115 bash[46934]: cluster 2022-04-22T17:52:10.413490+0000 mgr.y (mgr.34107) 944 : cluster [DBG] pgmap v1564: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:12.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:11.690Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:12.092 INFO:tasks.workunit.client.0.smithi115.stdout: 320: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:12.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-INTAYZtD4D0q9tJ off=21510665 len=1785724 2022-04-22T17:52:12.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 1638 completed, throughput=4.99MB/sec 2022-04-22T17:52:12.116 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-j-of-qBMFhGdcDF off=3047392 len=1668037 2022-04-22T17:52:12.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 1639 completed, throughput=5MB/sec 2022-04-22T17:52:12.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SyY7IK-wtzhv4cC off=354957 len=2065372 2022-04-22T17:52:12.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 1640 completed, throughput=5MB/sec 2022-04-22T17:52:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:11 smithi121 bash[36902]: cluster 2022-04-22T17:52:10.413490+0000 mgr.y (mgr.34107) 944 : cluster [DBG] pgmap v1564: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:52:13.110 INFO:tasks.workunit.client.0.smithi115.stdout: 321: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:13.110 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VDaKEQvvZzou7US off=1566713 len=2076899 2022-04-22T17:52:13.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 1641 completed, throughput=4.99MB/sec 2022-04-22T17:52:13.119 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wMwOtFvRT0MUwH7 off=3066383 len=1047119 2022-04-22T17:52:13.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 1642 completed, throughput=5MB/sec 2022-04-22T17:52:13.121 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SfTey3EA1ABNYKq off=362002 len=1905328 2022-04-22T17:52:13.131 INFO:tasks.workunit.client.0.smithi115.stdout:op 1643 completed, throughput=5MB/sec 2022-04-22T17:52:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:12 smithi121 bash[36902]: audit 2022-04-22T17:52:12.349329+0000 mon.a (mon.0) 2372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:12 smithi115 bash[45495]: audit 2022-04-22T17:52:12.349329+0000 mon.a (mon.0) 2372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:12 smithi115 bash[46934]: audit 2022-04-22T17:52:12.349329+0000 mon.a (mon.0) 2372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:13.865 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:13.571Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:14.131 INFO:tasks.workunit.client.0.smithi115.stdout: 322: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:14.131 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XfP7W4MsCd4DvT5 off=3126600 len=1420945 2022-04-22T17:52:14.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 1644 completed, throughput=4.99MB/sec 2022-04-22T17:52:14.140 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-p82UE3kENMifVM- off=8814108 len=1803544 2022-04-22T17:52:14.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 1645 completed, throughput=5MB/sec 2022-04-22T17:52:14.149 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LN_Hxeft4zy6DJc off=2043249 len=732952 2022-04-22T17:52:14.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 1646 completed, throughput=5MB/sec 2022-04-22T17:52:14.151 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GUP9gfHvAF91dLq off=5915188 len=606049 2022-04-22T17:52:14.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 1647 completed, throughput=5MB/sec 2022-04-22T17:52:14.153 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Zq0jyxTLUt6fBF4 off=10984338 len=1438411 2022-04-22T17:52:14.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 1648 completed, throughput=5MB/sec 2022-04-22T17:52:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:13 smithi121 bash[36902]: cluster 2022-04-22T17:52:12.414633+0000 mgr.y (mgr.34107) 945 : cluster [DBG] pgmap v1565: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:52:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:13 smithi121 bash[36902]: audit 2022-04-22T17:52:13.495256+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:52:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:13 smithi121 bash[36902]: audit 2022-04-22T17:52:13.495512+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:52:14.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[45495]: cluster 2022-04-22T17:52:12.414633+0000 mgr.y (mgr.34107) 945 : cluster [DBG] pgmap v1565: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:52:14.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[45495]: audit 2022-04-22T17:52:13.495256+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:52:14.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[45495]: audit 2022-04-22T17:52:13.495512+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:52:14.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[46934]: cluster 2022-04-22T17:52:12.414633+0000 mgr.y (mgr.34107) 945 : cluster [DBG] pgmap v1565: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:52:14.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[46934]: audit 2022-04-22T17:52:13.495256+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:52:14.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:13 smithi115 bash[46934]: audit 2022-04-22T17:52:13.495512+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:52:15.160 INFO:tasks.workunit.client.0.smithi115.stdout: 323: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:15.161 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kmF7Rl1Wj7VNhH0 off=4266492 len=1337255 2022-04-22T17:52:15.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 1649 completed, throughput=4.99MB/sec 2022-04-22T17:52:15.169 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BrmU-TJy4JKB7vy off=5629437 len=1739114 2022-04-22T17:52:15.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 1650 completed, throughput=5MB/sec 2022-04-22T17:52:15.178 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bzcgg1hqzcNg9cG off=5175679 len=1090564 2022-04-22T17:52:15.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 1651 completed, throughput=5MB/sec 2022-04-22T17:52:15.184 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IpFCH4JAPmmLmtg off=1962082 len=473203 2022-04-22T17:52:15.189 INFO:tasks.workunit.client.0.smithi115.stdout:op 1652 completed, throughput=5MB/sec 2022-04-22T17:52:16.189 INFO:tasks.workunit.client.0.smithi115.stdout: 324: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:16.189 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yKxR_eUBTe9dWlU off=4689313 len=269293 2022-04-22T17:52:16.194 INFO:tasks.workunit.client.0.smithi115.stdout:op 1653 completed, throughput=4.99MB/sec 2022-04-22T17:52:16.194 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DHAgxHYhXAGeiVk off=1979972 len=637656 2022-04-22T17:52:16.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 1654 completed, throughput=4.99MB/sec 2022-04-22T17:52:16.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nBWnZ2WU986oAKo off=10535286 len=780964 2022-04-22T17:52:16.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 1655 completed, throughput=4.99MB/sec 2022-04-22T17:52:16.202 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nklrs-nTDETLRat off=6028008 len=735385 2022-04-22T17:52:16.205 INFO:tasks.workunit.client.0.smithi115.stdout:op 1656 completed, throughput=4.99MB/sec 2022-04-22T17:52:16.205 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bq9I4uYlhRj22IB off=498688 len=1847004 2022-04-22T17:52:16.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 1657 completed, throughput=5MB/sec 2022-04-22T17:52:16.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pOYQnWp8dMDiUvl off=3209657 len=1067220 2022-04-22T17:52:16.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 1658 completed, throughput=5MB/sec 2022-04-22T17:52:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:15 smithi121 bash[36902]: cluster 2022-04-22T17:52:14.415717+0000 mgr.y (mgr.34107) 946 : cluster [DBG] pgmap v1566: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:15 smithi121 bash[36902]: audit 2022-04-22T17:52:14.973329+0000 mon.a (mon.0) 2375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:15 smithi121 bash[36902]: audit 2022-04-22T17:52:15.290840+0000 mon.a (mon.0) 2376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[45495]: cluster 2022-04-22T17:52:14.415717+0000 mgr.y (mgr.34107) 946 : cluster [DBG] pgmap v1566: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[45495]: audit 2022-04-22T17:52:14.973329+0000 mon.a (mon.0) 2375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[45495]: audit 2022-04-22T17:52:15.290840+0000 mon.a (mon.0) 2376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[46934]: cluster 2022-04-22T17:52:14.415717+0000 mgr.y (mgr.34107) 946 : cluster [DBG] pgmap v1566: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[46934]: audit 2022-04-22T17:52:14.973329+0000 mon.a (mon.0) 2375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:15 smithi115 bash[46934]: audit 2022-04-22T17:52:15.290840+0000 mon.a (mon.0) 2376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:17.217 INFO:tasks.workunit.client.0.smithi115.stdout: 325: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:17.217 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QcJkHeuQm2h4KxF off=14808766 len=1353948 2022-04-22T17:52:17.224 INFO:tasks.workunit.client.0.smithi115.stdout:op 1659 completed, throughput=4.99MB/sec 2022-04-22T17:52:17.224 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YjtXWHXlApdjrF4 off=9425711 len=1683363 2022-04-22T17:52:17.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 1660 completed, throughput=4.99MB/sec 2022-04-22T17:52:17.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-blEJVxB84V6nJlH off=7954959 len=1713060 2022-04-22T17:52:17.235 INFO:tasks.workunit.client.0.smithi115.stdout:op 1661 completed, throughput=5MB/sec 2022-04-22T17:52:17.236 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3b0_OsVKJcYHZkm off=6528832 len=2086747 2022-04-22T17:52:17.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 1662 completed, throughput=5.01MB/sec 2022-04-22T17:52:18.238 INFO:tasks.workunit.client.0.smithi115.stdout: 326: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:18.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O84muAUTvsCt0zn off=5544335 len=1534859 2022-04-22T17:52:18.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 1663 completed, throughput=4.99MB/sec 2022-04-22T17:52:18.247 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TvfnRTLlQVlJrhI off=3809087 len=561051 2022-04-22T17:52:18.248 INFO:tasks.workunit.client.0.smithi115.stdout:op 1664 completed, throughput=5MB/sec 2022-04-22T17:52:18.249 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0fZVzIEEu35jem9 off=194583 len=1433822 2022-04-22T17:52:18.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 1665 completed, throughput=5MB/sec 2022-04-22T17:52:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:17 smithi121 bash[36902]: cluster 2022-04-22T17:52:16.416780+0000 mgr.y (mgr.34107) 947 : cluster [DBG] pgmap v1567: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:17 smithi121 bash[36902]: audit 2022-04-22T17:52:17.352540+0000 mon.a (mon.0) 2377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:17 smithi115 bash[45495]: cluster 2022-04-22T17:52:16.416780+0000 mgr.y (mgr.34107) 947 : cluster [DBG] pgmap v1567: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:17 smithi115 bash[45495]: audit 2022-04-22T17:52:17.352540+0000 mon.a (mon.0) 2377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:17 smithi115 bash[46934]: cluster 2022-04-22T17:52:16.416780+0000 mgr.y (mgr.34107) 947 : cluster [DBG] pgmap v1567: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:52:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:17 smithi115 bash[46934]: audit 2022-04-22T17:52:17.352540+0000 mon.a (mon.0) 2377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:19.257 INFO:tasks.workunit.client.0.smithi115.stdout: 327: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:19.257 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-awC3gguKb_dar4Z off=8198411 len=673755 2022-04-22T17:52:19.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 1666 completed, throughput=4.99MB/sec 2022-04-22T17:52:19.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_wRvAVUnVJeTnQA off=1438146 len=170108 2022-04-22T17:52:19.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 1667 completed, throughput=4.99MB/sec 2022-04-22T17:52:19.264 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K5nGH5KdJay0v1s off=21705893 len=269781 2022-04-22T17:52:19.266 INFO:tasks.workunit.client.0.smithi115.stdout:op 1668 completed, throughput=4.99MB/sec 2022-04-22T17:52:19.266 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vd9Mk39RmzPGAA4 off=1497976 len=794874 2022-04-22T17:52:19.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 1669 completed, throughput=4.99MB/sec 2022-04-22T17:52:19.272 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JhV5Bu52jMxYdwX off=848460 len=1867728 2022-04-22T17:52:19.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 1670 completed, throughput=5MB/sec 2022-04-22T17:52:19.275 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mPlxDORmHkvXQSK off=8490356 len=523907 2022-04-22T17:52:19.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 1671 completed, throughput=5MB/sec 2022-04-22T17:52:19.276 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-twoOGwU5F8BXNvf off=3937768 len=85081 2022-04-22T17:52:19.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 1672 completed, throughput=5MB/sec 2022-04-22T17:52:19.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ba1uMYnLI96VMiI off=4975725 len=1891455 2022-04-22T17:52:19.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 1673 completed, throughput=5MB/sec 2022-04-22T17:52:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:19 smithi121 bash[36902]: cluster 2022-04-22T17:52:18.417981+0000 mgr.y (mgr.34107) 948 : cluster [DBG] pgmap v1568: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:20.281 INFO:tasks.workunit.client.0.smithi115.stdout: 328: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:20.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zBqQJBlZGL4t2q- off=8003428 len=980658 2022-04-22T17:52:20.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 1674 completed, throughput=4.99MB/sec 2022-04-22T17:52:20.285 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u5mnYfZtKrcfPA8 off=4333024 len=1869738 2022-04-22T17:52:20.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:19 smithi115 bash[45495]: cluster 2022-04-22T17:52:18.417981+0000 mgr.y (mgr.34107) 948 : cluster [DBG] pgmap v1568: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:19 smithi115 bash[46934]: cluster 2022-04-22T17:52:18.417981+0000 mgr.y (mgr.34107) 948 : cluster [DBG] pgmap v1568: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:20.296 INFO:tasks.workunit.client.0.smithi115.stdout:op 1675 completed, throughput=5MB/sec 2022-04-22T17:52:20.296 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-acs7aQFWgXG9r6z off=840874 len=516873 2022-04-22T17:52:20.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 1676 completed, throughput=5MB/sec 2022-04-22T17:52:20.298 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ECAPOZZZm3wRH9M off=1935651 len=1994771 2022-04-22T17:52:20.301 INFO:tasks.workunit.client.0.smithi115.stdout:op 1677 completed, throughput=5MB/sec 2022-04-22T17:52:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:20 smithi121 bash[36902]: audit 2022-04-22T17:52:19.976142+0000 mon.a (mon.0) 2378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:20 smithi121 bash[36902]: audit 2022-04-22T17:52:20.294475+0000 mon.a (mon.0) 2379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:21.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:20 smithi115 bash[45495]: audit 2022-04-22T17:52:19.976142+0000 mon.a (mon.0) 2378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:21.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:20 smithi115 bash[45495]: audit 2022-04-22T17:52:20.294475+0000 mon.a (mon.0) 2379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:20 smithi115 bash[46934]: audit 2022-04-22T17:52:19.976142+0000 mon.a (mon.0) 2378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:20 smithi115 bash[46934]: audit 2022-04-22T17:52:20.294475+0000 mon.a (mon.0) 2379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:21.301 INFO:tasks.workunit.client.0.smithi115.stdout: 329: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:21.301 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wFQGkZ9jPvqGp7U off=8965626 len=1269822 2022-04-22T17:52:21.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 1678 completed, throughput=4.99MB/sec 2022-04-22T17:52:21.305 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kK6Mi3I_gexH5qb off=586790 len=1792101 2022-04-22T17:52:21.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 1679 completed, throughput=5MB/sec 2022-04-22T17:52:21.316 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_ulC7lT-MsmpJb7 off=1192428 len=1004751 2022-04-22T17:52:21.318 INFO:tasks.workunit.client.0.smithi115.stdout:op 1680 completed, throughput=5MB/sec 2022-04-22T17:52:21.318 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0OlwTi4lEGtWqTE off=1244057 len=596242 2022-04-22T17:52:21.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 1681 completed, throughput=5MB/sec 2022-04-22T17:52:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:21.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:21 smithi115 bash[45495]: cluster 2022-04-22T17:52:20.419083+0000 mgr.y (mgr.34107) 949 : cluster [DBG] pgmap v1569: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:21 smithi115 bash[46934]: cluster 2022-04-22T17:52:20.419083+0000 mgr.y (mgr.34107) 949 : cluster [DBG] pgmap v1569: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:21 smithi121 bash[36902]: cluster 2022-04-22T17:52:20.419083+0000 mgr.y (mgr.34107) 949 : cluster [DBG] pgmap v1569: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:22.320 INFO:tasks.workunit.client.0.smithi115.stdout: 330: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:22.320 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yO7yfoQwWVKfZeU off=4895686 len=238797 2022-04-22T17:52:22.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 1682 completed, throughput=4.99MB/sec 2022-04-22T17:52:22.321 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eGqer3AWQHAJukk off=0 len=1788142 2022-04-22T17:52:22.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 1683 completed, throughput=4.99MB/sec 2022-04-22T17:52:22.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z7pYUdoZZlhij6g off=10725141 len=1678855 2022-04-22T17:52:22.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 1684 completed, throughput=5MB/sec 2022-04-22T17:52:22.330 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6g-_f-sEphPh68y off=3348379 len=1505874 2022-04-22T17:52:22.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 1685 completed, throughput=5MB/sec 2022-04-22T17:52:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:22 smithi121 bash[36902]: audit 2022-04-22T17:52:22.356229+0000 mon.a (mon.0) 2380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:22 smithi115 bash[45495]: audit 2022-04-22T17:52:22.356229+0000 mon.a (mon.0) 2380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:23.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:22 smithi115 bash[46934]: audit 2022-04-22T17:52:22.356229+0000 mon.a (mon.0) 2380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:23.338 INFO:tasks.workunit.client.0.smithi115.stdout: 331: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:23.338 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8MvIbtx1L5tenXK off=11401605 len=1711187 2022-04-22T17:52:23.342 INFO:tasks.workunit.client.0.smithi115.stdout:op 1686 completed, throughput=4.99MB/sec 2022-04-22T17:52:23.342 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7wLMDJRmZB-jrRu off=6174894 len=867443 2022-04-22T17:52:23.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 1687 completed, throughput=4.99MB/sec 2022-04-22T17:52:23.349 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yYj1Hw-CvejWFYO off=2347429 len=2038047 2022-04-22T17:52:23.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 1688 completed, throughput=5MB/sec 2022-04-22T17:52:23.351 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hTcPpi28FDoVEWk off=3940229 len=702132 2022-04-22T17:52:23.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 1689 completed, throughput=5MB/sec 2022-04-22T17:52:23.903 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:23.572Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:23 smithi121 bash[36902]: cluster 2022-04-22T17:52:22.420208+0000 mgr.y (mgr.34107) 950 : cluster [DBG] pgmap v1570: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:52:24.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:23 smithi115 bash[45495]: cluster 2022-04-22T17:52:22.420208+0000 mgr.y (mgr.34107) 950 : cluster [DBG] pgmap v1570: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:52:24.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:23 smithi115 bash[46934]: cluster 2022-04-22T17:52:22.420208+0000 mgr.y (mgr.34107) 950 : cluster [DBG] pgmap v1570: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:52:24.353 INFO:tasks.workunit.client.0.smithi115.stdout: 332: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:24.353 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VBQB3QjmDpscbyk off=14949159 len=1552016 2022-04-22T17:52:24.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 1690 completed, throughput=4.99MB/sec 2022-04-22T17:52:24.361 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z0_OkiLpKD24jBx off=3353080 len=1543812 2022-04-22T17:52:24.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 1691 completed, throughput=4.99MB/sec 2022-04-22T17:52:24.367 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HC-B9Jr-JbyDQub off=19193119 len=1219360 2022-04-22T17:52:24.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 1692 completed, throughput=5MB/sec 2022-04-22T17:52:24.369 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nxcNgFLr041EAR7 off=1744448 len=1311157 2022-04-22T17:52:24.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 1693 completed, throughput=5MB/sec 2022-04-22T17:52:25.376 INFO:tasks.workunit.client.0.smithi115.stdout: 333: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:25.376 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZxaHFNfR7ovMzkR off=336972 len=291766 2022-04-22T17:52:25.378 INFO:tasks.workunit.client.0.smithi115.stdout:op 1694 completed, throughput=4.99MB/sec 2022-04-22T17:52:25.379 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n_Vl39whB1kvUlh off=499939 len=708901 2022-04-22T17:52:25.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 1695 completed, throughput=4.99MB/sec 2022-04-22T17:52:25.381 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9sX1-l65DwKqOx3 off=12725314 len=288478 2022-04-22T17:52:25.386 INFO:tasks.workunit.client.0.smithi115.stdout:op 1696 completed, throughput=4.99MB/sec 2022-04-22T17:52:25.386 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6g-_f-sEphPh68y off=17572819 len=215756 2022-04-22T17:52:25.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 1697 completed, throughput=4.99MB/sec 2022-04-22T17:52:25.390 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Pag1NMmQ3kxGNJk off=18121538 len=1494840 2022-04-22T17:52:25.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 1698 completed, throughput=4.99MB/sec 2022-04-22T17:52:25.399 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7qnFLT0I2WX5kel off=1755861 len=1422375 2022-04-22T17:52:25.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 1699 completed, throughput=5MB/sec 2022-04-22T17:52:25.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ctvolD4bRxu7eiM off=11148626 len=1356731 2022-04-22T17:52:25.408 INFO:tasks.workunit.client.0.smithi115.stdout:op 1700 completed, throughput=5MB/sec 2022-04-22T17:52:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:25 smithi121 bash[36902]: cluster 2022-04-22T17:52:24.421347+0000 mgr.y (mgr.34107) 951 : cluster [DBG] pgmap v1571: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:25 smithi121 bash[36902]: audit 2022-04-22T17:52:24.979249+0000 mon.a (mon.0) 2381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:25 smithi121 bash[36902]: audit 2022-04-22T17:52:25.297726+0000 mon.a (mon.0) 2382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[45495]: cluster 2022-04-22T17:52:24.421347+0000 mgr.y (mgr.34107) 951 : cluster [DBG] pgmap v1571: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[45495]: audit 2022-04-22T17:52:24.979249+0000 mon.a (mon.0) 2381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[45495]: audit 2022-04-22T17:52:25.297726+0000 mon.a (mon.0) 2382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[46934]: cluster 2022-04-22T17:52:24.421347+0000 mgr.y (mgr.34107) 951 : cluster [DBG] pgmap v1571: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[46934]: audit 2022-04-22T17:52:24.979249+0000 mon.a (mon.0) 2381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:25 smithi115 bash[46934]: audit 2022-04-22T17:52:25.297726+0000 mon.a (mon.0) 2382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:26.408 INFO:tasks.workunit.client.0.smithi115.stdout: 334: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:26.409 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bykhzw9gZLVUTDc off=4697558 len=1514046 2022-04-22T17:52:26.418 INFO:tasks.workunit.client.0.smithi115.stdout:op 1701 completed, throughput=4.99MB/sec 2022-04-22T17:52:26.419 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x-R5pyLQ_VmpJ3O off=9696441 len=1198329 2022-04-22T17:52:26.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 1702 completed, throughput=4.99MB/sec 2022-04-22T17:52:26.424 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9ej7wHm8qBkWyhE off=4205429 len=82842 2022-04-22T17:52:26.427 INFO:tasks.workunit.client.0.smithi115.stdout:op 1703 completed, throughput=5MB/sec 2022-04-22T17:52:26.474 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Rk9-1-GxCZe1k8e off=6709482 len=648192 2022-04-22T17:52:26.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 1704 completed, throughput=5MB/sec 2022-04-22T17:52:26.475 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qk04FlenkA7yZo2 off=6903628 len=1718936 2022-04-22T17:52:26.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 1705 completed, throughput=5MB/sec 2022-04-22T17:52:27.437 INFO:tasks.workunit.client.0.smithi115.stdout: 335: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:27.476 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--QadwjN_F2JYlLM off=12103251 len=1963480 2022-04-22T17:52:27.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 1706 completed, throughput=4.99MB/sec 2022-04-22T17:52:27.477 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Iuu1Lvz__Ia7EzN off=6610729 len=1409378 2022-04-22T17:52:27.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 1707 completed, throughput=5MB/sec 2022-04-22T17:52:27.478 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QP-zIeHIPtEC-4Q off=3389802 len=815873 2022-04-22T17:52:27.478 INFO:tasks.workunit.client.0.smithi115.stdout:op 1708 completed, throughput=5MB/sec 2022-04-22T17:52:27.478 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xZhHVAj6uJcGhqP off=3805308 len=1390581 2022-04-22T17:52:27.478 INFO:tasks.workunit.client.0.smithi115.stdout:op 1709 completed, throughput=5MB/sec 2022-04-22T17:52:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:27 smithi121 bash[36902]: cluster 2022-04-22T17:52:26.422318+0000 mgr.y (mgr.34107) 952 : cluster [DBG] pgmap v1572: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:52:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:27 smithi121 bash[36902]: audit 2022-04-22T17:52:27.359239+0000 mon.a (mon.0) 2383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:27 smithi115 bash[45495]: cluster 2022-04-22T17:52:26.422318+0000 mgr.y (mgr.34107) 952 : cluster [DBG] pgmap v1572: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:52:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:27 smithi115 bash[45495]: audit 2022-04-22T17:52:27.359239+0000 mon.a (mon.0) 2383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:27 smithi115 bash[46934]: cluster 2022-04-22T17:52:26.422318+0000 mgr.y (mgr.34107) 952 : cluster [DBG] pgmap v1572: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:52:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:27 smithi115 bash[46934]: audit 2022-04-22T17:52:27.359239+0000 mon.a (mon.0) 2383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:28.454 INFO:tasks.workunit.client.0.smithi115.stdout: 336: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:28.454 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IPeP7rHHb4fOhji off=3332433 len=522497 2022-04-22T17:52:28.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 1710 completed, throughput=4.99MB/sec 2022-04-22T17:52:28.456 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-veCisbHosL2y91C off=3919829 len=1094537 2022-04-22T17:52:28.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 1711 completed, throughput=4.99MB/sec 2022-04-22T17:52:28.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rbIF2_uzRye16fH off=10389106 len=1329268 2022-04-22T17:52:28.468 INFO:tasks.workunit.client.0.smithi115.stdout:op 1712 completed, throughput=5MB/sec 2022-04-22T17:52:28.468 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xQVriQgdme5h3Cq off=8511796 len=1410510 2022-04-22T17:52:28.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 1713 completed, throughput=5MB/sec 2022-04-22T17:52:28.478 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e1YS3e90qg4ltGm off=2184595 len=1411066 2022-04-22T17:52:28.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 1714 completed, throughput=5MB/sec 2022-04-22T17:52:29.481 INFO:tasks.workunit.client.0.smithi115.stdout: 337: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:29.481 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--UAWluf41jFn1XE off=4451219 len=518227 2022-04-22T17:52:29.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 1715 completed, throughput=4.99MB/sec 2022-04-22T17:52:29.486 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WzIRtLyF6GzMMGW off=2042816 len=1169603 2022-04-22T17:52:29.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 1716 completed, throughput=4.99MB/sec 2022-04-22T17:52:29.493 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0D7X78jMm_LvU58 off=4270648 len=1520507 2022-04-22T17:52:29.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 1717 completed, throughput=5MB/sec 2022-04-22T17:52:29.498 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NB72_vsGjqi9nQB off=982848 len=1960803 2022-04-22T17:52:29.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 1718 completed, throughput=5MB/sec 2022-04-22T17:52:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:29 smithi121 bash[36902]: cluster 2022-04-22T17:52:28.423425+0000 mgr.y (mgr.34107) 953 : cluster [DBG] pgmap v1573: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:52:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:29 smithi115 bash[45495]: cluster 2022-04-22T17:52:28.423425+0000 mgr.y (mgr.34107) 953 : cluster [DBG] pgmap v1573: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:52:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:29 smithi115 bash[46934]: cluster 2022-04-22T17:52:28.423425+0000 mgr.y (mgr.34107) 953 : cluster [DBG] pgmap v1573: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:52:30.506 INFO:tasks.workunit.client.0.smithi115.stdout: 338: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:30.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1u59W5gDbgimLyl off=201695 len=483046 2022-04-22T17:52:30.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 1719 completed, throughput=4.99MB/sec 2022-04-22T17:52:30.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MymTWO4wFDO7M-- off=171260 len=1124538 2022-04-22T17:52:30.519 INFO:tasks.workunit.client.0.smithi115.stdout:op 1720 completed, throughput=4.99MB/sec 2022-04-22T17:52:30.519 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zjDYVB_nh2axWK4 off=866202 len=843659 2022-04-22T17:52:30.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 1721 completed, throughput=4.99MB/sec 2022-04-22T17:52:30.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ESFBcnY78gItsvz off=1170480 len=1132944 2022-04-22T17:52:30.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 1722 completed, throughput=5MB/sec 2022-04-22T17:52:30.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MykzqqpNaxtS8bN off=556579 len=1549418 2022-04-22T17:52:30.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 1723 completed, throughput=5MB/sec 2022-04-22T17:52:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:30 smithi121 bash[36902]: audit 2022-04-22T17:52:29.982439+0000 mon.a (mon.0) 2384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:30 smithi121 bash[36902]: audit 2022-04-22T17:52:30.301560+0000 mon.a (mon.0) 2385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:31.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:30 smithi115 bash[45495]: audit 2022-04-22T17:52:29.982439+0000 mon.a (mon.0) 2384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:30 smithi115 bash[45495]: audit 2022-04-22T17:52:30.301560+0000 mon.a (mon.0) 2385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:30 smithi115 bash[46934]: audit 2022-04-22T17:52:29.982439+0000 mon.a (mon.0) 2384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:30 smithi115 bash[46934]: audit 2022-04-22T17:52:30.301560+0000 mon.a (mon.0) 2385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:31.535 INFO:tasks.workunit.client.0.smithi115.stdout: 339: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:31.536 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TF1-JputJW40afU off=575079 len=1260487 2022-04-22T17:52:31.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 1724 completed, throughput=4.99MB/sec 2022-04-22T17:52:31.539 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xo8KJxiSMbK9w05 off=8247110 len=72007 2022-04-22T17:52:31.543 INFO:tasks.workunit.client.0.smithi115.stdout:op 1725 completed, throughput=4.99MB/sec 2022-04-22T17:52:31.543 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hE4XJSrNQPmGZYe off=387077 len=1605428 2022-04-22T17:52:31.547 INFO:tasks.workunit.client.0.smithi115.stdout:op 1726 completed, throughput=5MB/sec 2022-04-22T17:52:31.547 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kjz1WhkPX3nk6Kl off=737504 len=1667091 2022-04-22T17:52:31.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 1727 completed, throughput=5MB/sec 2022-04-22T17:52:31.554 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9pjvV2Tz4P5_XoZ off=3354415 len=1275597 2022-04-22T17:52:31.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 1728 completed, throughput=5MB/sec 2022-04-22T17:52:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:31 smithi115 bash[45495]: cluster 2022-04-22T17:52:30.424625+0000 mgr.y (mgr.34107) 954 : cluster [DBG] pgmap v1574: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:31.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:31 smithi115 bash[46934]: cluster 2022-04-22T17:52:30.424625+0000 mgr.y (mgr.34107) 954 : cluster [DBG] pgmap v1574: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:31 smithi121 bash[36902]: cluster 2022-04-22T17:52:30.424625+0000 mgr.y (mgr.34107) 954 : cluster [DBG] pgmap v1574: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:52:32.559 INFO:tasks.workunit.client.0.smithi115.stdout: 340: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:32.559 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yg3BskhSHWKpgFo off=6216900 len=345701 2022-04-22T17:52:32.561 INFO:tasks.workunit.client.0.smithi115.stdout:op 1729 completed, throughput=4.99MB/sec 2022-04-22T17:52:32.561 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_zrg4W5VreT5n5e off=18253508 len=164717 2022-04-22T17:52:32.562 INFO:tasks.workunit.client.0.smithi115.stdout:op 1730 completed, throughput=4.99MB/sec 2022-04-22T17:52:32.562 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VCcYDWVgAFJRko1 off=4346791 len=1454742 2022-04-22T17:52:32.571 INFO:tasks.workunit.client.0.smithi115.stdout:op 1731 completed, throughput=4.99MB/sec 2022-04-22T17:52:32.571 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pgvLjSXVM7SRQ1Y off=5087551 len=1124568 2022-04-22T17:52:32.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 1732 completed, throughput=5MB/sec 2022-04-22T17:52:32.577 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UHE_PawgrC46tVp off=3072841 len=238800 2022-04-22T17:52:32.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 1733 completed, throughput=5MB/sec 2022-04-22T17:52:32.579 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-33rXAnGh4rJS6NG off=2617251 len=1742962 2022-04-22T17:52:32.581 INFO:tasks.workunit.client.0.smithi115.stdout:op 1734 completed, throughput=5MB/sec 2022-04-22T17:52:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:32 smithi121 bash[36902]: audit 2022-04-22T17:52:32.362243+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:32 smithi115 bash[45495]: audit 2022-04-22T17:52:32.362243+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:32 smithi115 bash[46934]: audit 2022-04-22T17:52:32.362243+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:33.582 INFO:tasks.workunit.client.0.smithi115.stdout: 341: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:33.583 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5YOiYl0FEh2pEy5 off=1379249 len=469088 2022-04-22T17:52:33.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 1735 completed, throughput=4.99MB/sec 2022-04-22T17:52:33.584 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lpBDMI1FtQ3Ol7e off=836523 len=216428 2022-04-22T17:52:33.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 1736 completed, throughput=4.99MB/sec 2022-04-22T17:52:33.586 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xzT4qT9frgUc92L off=8963723 len=136984 2022-04-22T17:52:33.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 1737 completed, throughput=4.99MB/sec 2022-04-22T17:52:33.590 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G0hZwuJyHitf9kC off=507569 len=1097832 2022-04-22T17:52:33.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 1738 completed, throughput=4.99MB/sec 2022-04-22T17:52:33.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dybSnCKeVd6luAL off=6291241 len=2012002 2022-04-22T17:52:33.601 INFO:tasks.workunit.client.0.smithi115.stdout:op 1739 completed, throughput=5MB/sec 2022-04-22T17:52:33.601 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8IUkHnrBg1Cj1W8 off=2544131 len=661586 2022-04-22T17:52:33.606 INFO:tasks.workunit.client.0.smithi115.stdout:op 1740 completed, throughput=5MB/sec 2022-04-22T17:52:33.935 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:33.573Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:33.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:33 smithi115 bash[46934]: cluster 2022-04-22T17:52:32.425712+0000 mgr.y (mgr.34107) 955 : cluster [DBG] pgmap v1575: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:33.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:33 smithi115 bash[45495]: cluster 2022-04-22T17:52:32.425712+0000 mgr.y (mgr.34107) 955 : cluster [DBG] pgmap v1575: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:52:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:33 smithi121 bash[36902]: cluster 2022-04-22T17:52:32.425712+0000 mgr.y (mgr.34107) 955 : cluster [DBG] pgmap v1575: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:34.491 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:52:34.491 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (15m) 4m ago 29m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:52:34.491 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (29m) 6m ago 29m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:52:34.491 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (17m) 4m ago 30m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:52:34.491 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (17m) 6m ago 30m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (24m) 6m ago 34m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (25m) 4m ago 36m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (24m) 4m ago 37m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (23m) 6m ago 35m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (23m) 4m ago 35m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (16m) 4m ago 29m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (16m) 6m ago 29m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (22m) 4m ago 34m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (21m) 4m ago 33m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:52:34.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (21m) 4m ago 33m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (20m) 4m ago 32m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (20m) 6m ago 32m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (19m) 6m ago 31m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (18m) 6m ago 31m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (18m) 6m ago 31m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:52:34.493 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (16m) 6m ago 29m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:52:34.607 INFO:tasks.workunit.client.0.smithi115.stdout: 342: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:34.635 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mVnd2eyPfx9cfni off=448564 len=342184 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 1741 completed, throughput=4.99MB/sec 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-umQjgTccz5yaj8x off=7680007 len=1946870 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 1742 completed, throughput=4.99MB/sec 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rlbZ7rDTmVzUkYi off=7106780 len=1100870 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 1743 completed, throughput=5MB/sec 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vQwAFV9YJTn1bkc off=7325886 len=777106 2022-04-22T17:52:34.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 1744 completed, throughput=5MB/sec 2022-04-22T17:52:34.638 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bzcgg1hqzcNg9cG off=122899 len=689179 2022-04-22T17:52:34.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 1745 completed, throughput=5MB/sec 2022-04-22T17:52:34.638 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0HMyWR3M2IBT7XS off=12854579 len=1407348 2022-04-22T17:52:34.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 1746 completed, throughput=5MB/sec 2022-04-22T17:52:34.892 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:52:34.892 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:52:34.892 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:52:34.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:34.893 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:52:34.893 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:52:34.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:34.893 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:52:34.893 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:52:34.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:34.894 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:52:34.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:52:34.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:52:34.895 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:52:34.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:52:34.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:52:34.895 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:52:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:34 smithi121 bash[36902]: audit 2022-04-22T17:52:34.124522+0000 mgr.y (mgr.34107) 956 : audit [DBG] from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:34 smithi121 bash[36902]: audit 2022-04-22T17:52:34.883893+0000 mon.a (mon.0) 2387 : audit [DBG] from='client.? 172.21.15.115:0/1999091201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:35.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:34 smithi115 bash[45495]: audit 2022-04-22T17:52:34.124522+0000 mgr.y (mgr.34107) 956 : audit [DBG] from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:35.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:34 smithi115 bash[45495]: audit 2022-04-22T17:52:34.883893+0000 mon.a (mon.0) 2387 : audit [DBG] from='client.? 172.21.15.115:0/1999091201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:34 smithi115 bash[46934]: audit 2022-04-22T17:52:34.124522+0000 mgr.y (mgr.34107) 956 : audit [DBG] from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:34 smithi115 bash[46934]: audit 2022-04-22T17:52:34.883893+0000 mon.a (mon.0) 2387 : audit [DBG] from='client.? 172.21.15.115:0/1999091201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:52:35.627 INFO:tasks.workunit.client.0.smithi115.stdout: 343: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:35.628 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VWBptdrCblKWb6m off=9528204 len=1414460 2022-04-22T17:52:35.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 1747 completed, throughput=4.99MB/sec 2022-04-22T17:52:35.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1cp3Zg3I3u1RpPq off=439836 len=2058748 2022-04-22T17:52:35.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 1748 completed, throughput=5MB/sec 2022-04-22T17:52:35.642 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xHKRecAqCJqIsdI off=21694551 len=386834 2022-04-22T17:52:35.643 INFO:tasks.workunit.client.0.smithi115.stdout:op 1749 completed, throughput=5MB/sec 2022-04-22T17:52:35.644 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kSlw2ABjQMTmehM off=4902382 len=349318 2022-04-22T17:52:35.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 1750 completed, throughput=5MB/sec 2022-04-22T17:52:35.644 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jRiJXFjWdh_0VLH off=5860418 len=317748 2022-04-22T17:52:35.647 INFO:tasks.workunit.client.0.smithi115.stdout:op 1751 completed, throughput=5MB/sec 2022-04-22T17:52:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:35 smithi121 bash[36902]: cluster 2022-04-22T17:52:34.426749+0000 mgr.y (mgr.34107) 957 : cluster [DBG] pgmap v1576: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:35 smithi121 bash[36902]: audit 2022-04-22T17:52:34.461732+0000 mgr.y (mgr.34107) 958 : audit [DBG] from='client.45355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:35 smithi121 bash[36902]: audit 2022-04-22T17:52:34.985373+0000 mon.a (mon.0) 2388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:35 smithi121 bash[36902]: audit 2022-04-22T17:52:35.305022+0000 mon.a (mon.0) 2389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[45495]: cluster 2022-04-22T17:52:34.426749+0000 mgr.y (mgr.34107) 957 : cluster [DBG] pgmap v1576: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[45495]: audit 2022-04-22T17:52:34.461732+0000 mgr.y (mgr.34107) 958 : audit [DBG] from='client.45355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[45495]: audit 2022-04-22T17:52:34.985373+0000 mon.a (mon.0) 2388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[45495]: audit 2022-04-22T17:52:35.305022+0000 mon.a (mon.0) 2389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[46934]: cluster 2022-04-22T17:52:34.426749+0000 mgr.y (mgr.34107) 957 : cluster [DBG] pgmap v1576: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[46934]: audit 2022-04-22T17:52:34.461732+0000 mgr.y (mgr.34107) 958 : audit [DBG] from='client.45355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:52:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[46934]: audit 2022-04-22T17:52:34.985373+0000 mon.a (mon.0) 2388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:35 smithi115 bash[46934]: audit 2022-04-22T17:52:35.305022+0000 mon.a (mon.0) 2389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:36.648 INFO:tasks.workunit.client.0.smithi115.stdout: 344: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:36.648 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZUpFC5aHtmY9uYK off=6219171 len=1531847 2022-04-22T17:52:36.651 INFO:tasks.workunit.client.0.smithi115.stdout:op 1752 completed, throughput=4.99MB/sec 2022-04-22T17:52:36.651 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CeF31z5JqqW9kUS off=11190407 len=1602250 2022-04-22T17:52:36.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 1753 completed, throughput=4.99MB/sec 2022-04-22T17:52:36.660 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wDc5XExGe_CReIr off=10830512 len=1581771 2022-04-22T17:52:36.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 1754 completed, throughput=5MB/sec 2022-04-22T17:52:36.666 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ze5CV3iOXfLzUUX off=8299172 len=797826 2022-04-22T17:52:36.671 INFO:tasks.workunit.client.0.smithi115.stdout:op 1755 completed, throughput=5MB/sec 2022-04-22T17:52:37.671 INFO:tasks.workunit.client.0.smithi115.stdout: 345: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:37.671 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dmMZ-Nm51ATuEB1 off=729380 len=1239720 2022-04-22T17:52:37.675 INFO:tasks.workunit.client.0.smithi115.stdout:op 1756 completed, throughput=4.99MB/sec 2022-04-22T17:52:37.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9QDF9cKxHq106Qs off=5954499 len=1216817 2022-04-22T17:52:37.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 1757 completed, throughput=4.99MB/sec 2022-04-22T17:52:37.679 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0j9uuTw0PdDhhMj off=43727 len=1355744 2022-04-22T17:52:37.688 INFO:tasks.workunit.client.0.smithi115.stdout:op 1758 completed, throughput=5MB/sec 2022-04-22T17:52:37.688 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G9D2K_BQMZuNXKV off=2508397 len=1440068 2022-04-22T17:52:37.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 1759 completed, throughput=5MB/sec 2022-04-22T17:52:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:37 smithi121 bash[36902]: cluster 2022-04-22T17:52:36.427873+0000 mgr.y (mgr.34107) 959 : cluster [DBG] pgmap v1577: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:52:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:37 smithi121 bash[36902]: audit 2022-04-22T17:52:37.365300+0000 mon.a (mon.0) 2390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:37 smithi115 bash[45495]: cluster 2022-04-22T17:52:36.427873+0000 mgr.y (mgr.34107) 959 : cluster [DBG] pgmap v1577: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:52:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:37 smithi115 bash[45495]: audit 2022-04-22T17:52:37.365300+0000 mon.a (mon.0) 2390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:37 smithi115 bash[46934]: cluster 2022-04-22T17:52:36.427873+0000 mgr.y (mgr.34107) 959 : cluster [DBG] pgmap v1577: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:52:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:37 smithi115 bash[46934]: audit 2022-04-22T17:52:37.365300+0000 mon.a (mon.0) 2390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:38.690 INFO:tasks.workunit.client.0.smithi115.stdout: 346: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:38.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gfyskg_ktcqohYu off=16820314 len=565110 2022-04-22T17:52:38.696 INFO:tasks.workunit.client.0.smithi115.stdout:op 1760 completed, throughput=4.99MB/sec 2022-04-22T17:52:38.696 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E6n1EwoMFRnZThq off=0 len=1464670 2022-04-22T17:52:38.704 INFO:tasks.workunit.client.0.smithi115.stdout:op 1761 completed, throughput=4.99MB/sec 2022-04-22T17:52:38.704 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bYRRhDlO9izebOa off=7459644 len=493710 2022-04-22T17:52:38.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 1762 completed, throughput=4.99MB/sec 2022-04-22T17:52:38.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tfFgw4IucC7uVq7 off=8207535 len=1921972 2022-04-22T17:52:38.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 1763 completed, throughput=5MB/sec 2022-04-22T17:52:38.710 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n48yRpeUohZd23y off=17010995 len=120526 2022-04-22T17:52:38.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 1764 completed, throughput=5MB/sec 2022-04-22T17:52:38.711 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-glT0fnaqZH3mfGA off=17885674 len=1421127 2022-04-22T17:52:38.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 1765 completed, throughput=5MB/sec 2022-04-22T17:52:39.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[45495]: audit 2022-04-22T17:52:38.735654+0000 mon.a (mon.0) 2391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:52:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[45495]: audit 2022-04-22T17:52:38.738668+0000 mon.a (mon.0) 2392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:52:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[45495]: audit 2022-04-22T17:52:38.740461+0000 mon.a (mon.0) 2393 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:52:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[46934]: audit 2022-04-22T17:52:38.735654+0000 mon.a (mon.0) 2391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:52:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[46934]: audit 2022-04-22T17:52:38.738668+0000 mon.a (mon.0) 2392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:52:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:38 smithi115 bash[46934]: audit 2022-04-22T17:52:38.740461+0000 mon.a (mon.0) 2393 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:52:39.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:38 smithi121 bash[36902]: audit 2022-04-22T17:52:38.735654+0000 mon.a (mon.0) 2391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:52:39.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:38 smithi121 bash[36902]: audit 2022-04-22T17:52:38.738668+0000 mon.a (mon.0) 2392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:52:39.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:38 smithi121 bash[36902]: audit 2022-04-22T17:52:38.740461+0000 mon.a (mon.0) 2393 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:52:39.720 INFO:tasks.workunit.client.0.smithi115.stdout: 347: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:39.720 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rxDWexu3oXjJN9A off=7822901 len=1253527 2022-04-22T17:52:39.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 1766 completed, throughput=4.99MB/sec 2022-04-22T17:52:39.727 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ptmVNHWjHilZzPn off=544906 len=1868684 2022-04-22T17:52:39.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 1767 completed, throughput=5MB/sec 2022-04-22T17:52:39.737 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Fm8hegFuFsT-6LE off=4210221 len=1386239 2022-04-22T17:52:39.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 1768 completed, throughput=5MB/sec 2022-04-22T17:52:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:40 smithi121 bash[36902]: cluster 2022-04-22T17:52:38.429141+0000 mgr.y (mgr.34107) 960 : cluster [DBG] pgmap v1578: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:40 smithi121 bash[36902]: audit 2022-04-22T17:52:39.119706+0000 mon.a (mon.0) 2394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:52:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:40 smithi121 bash[36902]: audit 2022-04-22T17:52:39.988526+0000 mon.a (mon.0) 2395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[46934]: cluster 2022-04-22T17:52:38.429141+0000 mgr.y (mgr.34107) 960 : cluster [DBG] pgmap v1578: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[46934]: audit 2022-04-22T17:52:39.119706+0000 mon.a (mon.0) 2394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:52:40.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[46934]: audit 2022-04-22T17:52:39.988526+0000 mon.a (mon.0) 2395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:40.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[45495]: cluster 2022-04-22T17:52:38.429141+0000 mgr.y (mgr.34107) 960 : cluster [DBG] pgmap v1578: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:40.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[45495]: audit 2022-04-22T17:52:39.119706+0000 mon.a (mon.0) 2394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:52:40.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:40 smithi115 bash[45495]: audit 2022-04-22T17:52:39.988526+0000 mon.a (mon.0) 2395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:40.744 INFO:tasks.workunit.client.0.smithi115.stdout: 348: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:40.745 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V2X9ioPeHlpuORU off=1133875 len=529766 2022-04-22T17:52:40.746 INFO:tasks.workunit.client.0.smithi115.stdout:op 1769 completed, throughput=4.99MB/sec 2022-04-22T17:52:40.746 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mibaMdEi6sX-Az7 off=3464469 len=877892 2022-04-22T17:52:40.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 1770 completed, throughput=4.99MB/sec 2022-04-22T17:52:40.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uLCVdgXyCSoR1ip off=7837487 len=2079899 2022-04-22T17:52:40.758 INFO:tasks.workunit.client.0.smithi115.stdout:op 1771 completed, throughput=5MB/sec 2022-04-22T17:52:40.758 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u4CkN0yGWv3SIog off=4511030 len=1875243 2022-04-22T17:52:40.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 1772 completed, throughput=5MB/sec 2022-04-22T17:52:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:41 smithi121 bash[36902]: audit 2022-04-22T17:52:40.308626+0000 mon.a (mon.0) 2396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:41 smithi115 bash[45495]: audit 2022-04-22T17:52:40.308626+0000 mon.a (mon.0) 2396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:41 smithi115 bash[46934]: audit 2022-04-22T17:52:40.308626+0000 mon.a (mon.0) 2396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:41.762 INFO:tasks.workunit.client.0.smithi115.stdout: 349: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:41.817 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rbcOAuwhrz7Qpyx off=4639382 len=731012 2022-04-22T17:52:41.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 1773 completed, throughput=4.99MB/sec 2022-04-22T17:52:41.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gEV8iC52tiULi5H off=2015731 len=713027 2022-04-22T17:52:41.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 1774 completed, throughput=4.99MB/sec 2022-04-22T17:52:41.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R_XyFx_42XIa7M9 off=1859364 len=469579 2022-04-22T17:52:41.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 1775 completed, throughput=4.99MB/sec 2022-04-22T17:52:41.824 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NOrdGbyU6pn_3ap off=10603300 len=1926194 2022-04-22T17:52:41.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 1776 completed, throughput=5MB/sec 2022-04-22T17:52:41.824 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pT-A5ZJJGwLYLmQ off=10015728 len=1498695 2022-04-22T17:52:41.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 1777 completed, throughput=5MB/sec 2022-04-22T17:52:41.825 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PRyT2FmOsFfj9_0 off=17276802 len=247937 2022-04-22T17:52:41.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 1778 completed, throughput=5MB/sec 2022-04-22T17:52:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:41.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:42 smithi121 bash[36902]: cluster 2022-04-22T17:52:40.430256+0000 mgr.y (mgr.34107) 961 : cluster [DBG] pgmap v1579: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:42 smithi115 bash[45495]: cluster 2022-04-22T17:52:40.430256+0000 mgr.y (mgr.34107) 961 : cluster [DBG] pgmap v1579: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:42 smithi115 bash[46934]: cluster 2022-04-22T17:52:40.430256+0000 mgr.y (mgr.34107) 961 : cluster [DBG] pgmap v1579: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:52:42.792 INFO:tasks.workunit.client.0.smithi115.stdout: 350: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:42.793 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xTCnxl2xJLJjUVn off=5476350 len=2017567 2022-04-22T17:52:42.804 INFO:tasks.workunit.client.0.smithi115.stdout:op 1779 completed, throughput=4.99MB/sec 2022-04-22T17:52:42.804 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TPwzV-eSHqn3ZR2 off=10235960 len=1378519 2022-04-22T17:52:42.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 1780 completed, throughput=5MB/sec 2022-04-22T17:52:42.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-45fczfvBTqXLIw- off=3082217 len=1110756 2022-04-22T17:52:42.810 INFO:tasks.workunit.client.0.smithi115.stdout:op 1781 completed, throughput=5MB/sec 2022-04-22T17:52:42.810 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-T5e4pahgB1n2EnL off=5664755 len=352083 2022-04-22T17:52:42.813 INFO:tasks.workunit.client.0.smithi115.stdout:op 1782 completed, throughput=5MB/sec 2022-04-22T17:52:42.814 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ucjvFHhCx71Vc2B off=7683412 len=1228137 2022-04-22T17:52:42.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 1783 completed, throughput=5MB/sec 2022-04-22T17:52:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:43 smithi121 bash[36902]: audit 2022-04-22T17:52:42.368493+0000 mon.a (mon.0) 2397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:43 smithi115 bash[46934]: audit 2022-04-22T17:52:42.368493+0000 mon.a (mon.0) 2397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:43 smithi115 bash[45495]: audit 2022-04-22T17:52:42.368493+0000 mon.a (mon.0) 2397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:43.819 INFO:tasks.workunit.client.0.smithi115.stdout: 351: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:43.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2sDWXNPCEiq3Gml off=4180244 len=811360 2022-04-22T17:52:43.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 1784 completed, throughput=4.99MB/sec 2022-04-22T17:52:43.825 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wEwukQrs1IKaEpQ off=2015733 len=526241 2022-04-22T17:52:43.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 1785 completed, throughput=4.99MB/sec 2022-04-22T17:52:43.830 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sfzAQCtRHTPTG9i off=412460 len=520987 2022-04-22T17:52:43.831 INFO:tasks.workunit.client.0.smithi115.stdout:op 1786 completed, throughput=4.99MB/sec 2022-04-22T17:52:43.831 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jqG5G1400wXGPo1 off=3782471 len=134960 2022-04-22T17:52:43.832 INFO:tasks.workunit.client.0.smithi115.stdout:op 1787 completed, throughput=4.99MB/sec 2022-04-22T17:52:43.832 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rJQxCFBFeSrivyZ off=2266181 len=700918 2022-04-22T17:52:43.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 1788 completed, throughput=5MB/sec 2022-04-22T17:52:43.835 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b9cRqH2uYrxTgRa off=7035766 len=1507262 2022-04-22T17:52:43.843 INFO:tasks.workunit.client.0.smithi115.stdout:op 1789 completed, throughput=5MB/sec 2022-04-22T17:52:43.843 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X4XAjIuHEuTr73J off=11116679 len=572817 2022-04-22T17:52:43.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 1790 completed, throughput=5MB/sec 2022-04-22T17:52:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:43.573Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:44 smithi121 bash[36902]: cluster 2022-04-22T17:52:42.431325+0000 mgr.y (mgr.34107) 962 : cluster [DBG] pgmap v1580: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:44 smithi115 bash[45495]: cluster 2022-04-22T17:52:42.431325+0000 mgr.y (mgr.34107) 962 : cluster [DBG] pgmap v1580: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:44 smithi115 bash[46934]: cluster 2022-04-22T17:52:42.431325+0000 mgr.y (mgr.34107) 962 : cluster [DBG] pgmap v1580: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:44.844 INFO:tasks.workunit.client.0.smithi115.stdout: 352: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:44.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iju7LVK0tegqFiN off=8401443 len=1824529 2022-04-22T17:52:44.848 INFO:tasks.workunit.client.0.smithi115.stdout:op 1791 completed, throughput=4.99MB/sec 2022-04-22T17:52:44.849 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SRLhPM1JyRiM9Q9 off=582615 len=396205 2022-04-22T17:52:44.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 1792 completed, throughput=4.99MB/sec 2022-04-22T17:52:44.851 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UgUug79VNc9LVbZ off=19684773 len=332412 2022-04-22T17:52:44.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 1793 completed, throughput=4.99MB/sec 2022-04-22T17:52:44.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R8yccBOCaLFQHFD off=13475751 len=2065601 2022-04-22T17:52:44.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 1794 completed, throughput=5MB/sec 2022-04-22T17:52:44.877 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dTYpitwax-3gUVT off=2227023 len=230155 2022-04-22T17:52:44.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 1795 completed, throughput=5MB/sec 2022-04-22T17:52:44.877 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2-LPzkSuMkAeGe7 off=1617919 len=1194547 2022-04-22T17:52:44.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 1796 completed, throughput=5MB/sec 2022-04-22T17:52:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:45 smithi121 bash[36902]: audit 2022-04-22T17:52:44.991739+0000 mon.a (mon.0) 2398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:45 smithi115 bash[45495]: audit 2022-04-22T17:52:44.991739+0000 mon.a (mon.0) 2398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:45 smithi115 bash[46934]: audit 2022-04-22T17:52:44.991739+0000 mon.a (mon.0) 2398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:45.872 INFO:tasks.workunit.client.0.smithi115.stdout: 353: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:45.872 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zKyxtzV6Q6nl4jH off=9197009 len=139731 2022-04-22T17:52:45.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 1797 completed, throughput=4.99MB/sec 2022-04-22T17:52:45.876 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gh2I6eU9TJcMNRY off=1708231 len=1464223 2022-04-22T17:52:45.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 1798 completed, throughput=4.99MB/sec 2022-04-22T17:52:45.879 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6W8iRnnPxZcVv8F off=7246855 len=1793672 2022-04-22T17:52:45.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 1799 completed, throughput=5MB/sec 2022-04-22T17:52:45.883 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QpRlT-pOKxMfmZr off=10728689 len=1704448 2022-04-22T17:52:45.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 1800 completed, throughput=5MB/sec 2022-04-22T17:52:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:46 smithi121 bash[36902]: cluster 2022-04-22T17:52:44.432446+0000 mgr.y (mgr.34107) 963 : cluster [DBG] pgmap v1581: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:46 smithi121 bash[36902]: audit 2022-04-22T17:52:45.312235+0000 mon.a (mon.0) 2399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:46 smithi115 bash[45495]: cluster 2022-04-22T17:52:44.432446+0000 mgr.y (mgr.34107) 963 : cluster [DBG] pgmap v1581: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:46 smithi115 bash[45495]: audit 2022-04-22T17:52:45.312235+0000 mon.a (mon.0) 2399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:46 smithi115 bash[46934]: cluster 2022-04-22T17:52:44.432446+0000 mgr.y (mgr.34107) 963 : cluster [DBG] pgmap v1581: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:46 smithi115 bash[46934]: audit 2022-04-22T17:52:45.312235+0000 mon.a (mon.0) 2399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:46.885 INFO:tasks.workunit.client.0.smithi115.stdout: 354: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:46.885 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n4mIvkIbr-Bwvjj off=5769307 len=1484428 2022-04-22T17:52:46.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 1801 completed, throughput=4.99MB/sec 2022-04-22T17:52:46.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_us2XhuNOBWgCN- off=2508439 len=881606 2022-04-22T17:52:46.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 1802 completed, throughput=4.99MB/sec 2022-04-22T17:52:46.898 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HyysU425gq_82ZW off=220396 len=1955680 2022-04-22T17:52:46.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 1803 completed, throughput=5MB/sec 2022-04-22T17:52:46.901 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WaIBJbiQJS2gQ_8 off=1741307 len=550111 2022-04-22T17:52:46.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 1804 completed, throughput=5MB/sec 2022-04-22T17:52:47.903 INFO:tasks.workunit.client.0.smithi115.stdout: 355: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:47.904 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XwFIQPG9A6iyUda off=2011519 len=408810 2022-04-22T17:52:47.906 INFO:tasks.workunit.client.0.smithi115.stdout:op 1805 completed, throughput=4.99MB/sec 2022-04-22T17:52:47.906 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fwtoY_4cpglD0Hx off=4626297 len=1166373 2022-04-22T17:52:47.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 1806 completed, throughput=4.99MB/sec 2022-04-22T17:52:47.913 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O4gqMIflG9hSBZl off=16807291 len=1545377 2022-04-22T17:52:47.918 INFO:tasks.workunit.client.0.smithi115.stdout:op 1807 completed, throughput=5MB/sec 2022-04-22T17:52:47.918 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BAEwqcw2aUIxkyI off=4515124 len=336112 2022-04-22T17:52:47.919 INFO:tasks.workunit.client.0.smithi115.stdout:op 1808 completed, throughput=5MB/sec 2022-04-22T17:52:47.919 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-euF5juItcRPBMEa off=7470867 len=2009901 2022-04-22T17:52:47.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 1809 completed, throughput=5MB/sec 2022-04-22T17:52:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:48 smithi121 bash[36902]: cluster 2022-04-22T17:52:46.433559+0000 mgr.y (mgr.34107) 964 : cluster [DBG] pgmap v1582: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:48 smithi121 bash[36902]: audit 2022-04-22T17:52:47.371576+0000 mon.a (mon.0) 2400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:48 smithi115 bash[46934]: cluster 2022-04-22T17:52:46.433559+0000 mgr.y (mgr.34107) 964 : cluster [DBG] pgmap v1582: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:48 smithi115 bash[46934]: audit 2022-04-22T17:52:47.371576+0000 mon.a (mon.0) 2400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:48 smithi115 bash[45495]: cluster 2022-04-22T17:52:46.433559+0000 mgr.y (mgr.34107) 964 : cluster [DBG] pgmap v1582: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:52:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:48 smithi115 bash[45495]: audit 2022-04-22T17:52:47.371576+0000 mon.a (mon.0) 2400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:48.923 INFO:tasks.workunit.client.0.smithi115.stdout: 356: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:48.923 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t3kQ3swgMQ1bFT_ off=128948 len=411795 2022-04-22T17:52:48.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 1810 completed, throughput=4.99MB/sec 2022-04-22T17:52:48.929 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u9JgHqwQO0ggkHT off=7194728 len=1180159 2022-04-22T17:52:48.932 INFO:tasks.workunit.client.0.smithi115.stdout:op 1811 completed, throughput=4.99MB/sec 2022-04-22T17:52:48.932 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vppelyxfv7M6L14 off=5945996 len=125724 2022-04-22T17:52:48.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 1812 completed, throughput=4.99MB/sec 2022-04-22T17:52:48.933 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zRUDfZznJYqoTk1 off=18490735 len=1773372 2022-04-22T17:52:48.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 1813 completed, throughput=5MB/sec 2022-04-22T17:52:48.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hO4iKH50tRaUcHF off=23261892 len=407791 2022-04-22T17:52:48.947 INFO:tasks.workunit.client.0.smithi115.stdout:op 1814 completed, throughput=5MB/sec 2022-04-22T17:52:48.947 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Zxp5hEtV1oil9PQ off=3291581 len=1736264 2022-04-22T17:52:48.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 1815 completed, throughput=5MB/sec 2022-04-22T17:52:49.955 INFO:tasks.workunit.client.0.smithi115.stdout: 357: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:49.955 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ey2RYMlK-u3KIIy off=3656 len=883457 2022-04-22T17:52:49.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 1816 completed, throughput=4.99MB/sec 2022-04-22T17:52:49.963 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2ox6Gya7kvzgvRR off=10412203 len=552253 2022-04-22T17:52:49.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 1817 completed, throughput=4.99MB/sec 2022-04-22T17:52:49.965 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w9VMPDX3buAPzBA off=3694193 len=985370 2022-04-22T17:52:49.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 1818 completed, throughput=4.99MB/sec 2022-04-22T17:52:50.021 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1NtiIpYK1PxG8RX off=9189703 len=135631 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 1819 completed, throughput=4.99MB/sec 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D1sAqWM7PSKp8PO off=1026626 len=387845 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 1820 completed, throughput=5MB/sec 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GrX2jGDbNwEBfMz off=3754029 len=508041 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 1821 completed, throughput=5MB/sec 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2RG3ws-Dmf4R39X off=14818180 len=1833515 2022-04-22T17:52:50.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 1822 completed, throughput=5MB/sec 2022-04-22T17:52:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:50 smithi121 bash[36902]: cluster 2022-04-22T17:52:48.434696+0000 mgr.y (mgr.34107) 965 : cluster [DBG] pgmap v1583: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:50 smithi121 bash[36902]: audit 2022-04-22T17:52:49.994313+0000 mon.a (mon.0) 2401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:50 smithi115 bash[45495]: cluster 2022-04-22T17:52:48.434696+0000 mgr.y (mgr.34107) 965 : cluster [DBG] pgmap v1583: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:50 smithi115 bash[45495]: audit 2022-04-22T17:52:49.994313+0000 mon.a (mon.0) 2401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:50 smithi115 bash[46934]: cluster 2022-04-22T17:52:48.434696+0000 mgr.y (mgr.34107) 965 : cluster [DBG] pgmap v1583: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:52:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:50 smithi115 bash[46934]: audit 2022-04-22T17:52:49.994313+0000 mon.a (mon.0) 2401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:50.989 INFO:tasks.workunit.client.0.smithi115.stdout: 358: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:50.989 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5ymva5FzL2FTvIs off=65622 len=336803 2022-04-22T17:52:50.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 1823 completed, throughput=4.99MB/sec 2022-04-22T17:52:50.994 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S57--1-aFxxYpRT off=8306311 len=805262 2022-04-22T17:52:51.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 1824 completed, throughput=4.99MB/sec 2022-04-22T17:52:51.000 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VeOxZ0GrndU5yAW off=2444171 len=1668046 2022-04-22T17:52:51.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 1825 completed, throughput=4.99MB/sec 2022-04-22T17:52:51.003 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SzgQa3W2Cj2qQSF off=1957693 len=1925742 2022-04-22T17:52:51.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 1826 completed, throughput=5MB/sec 2022-04-22T17:52:51.006 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L74OogsvLurszB1 off=5764189 len=264362 2022-04-22T17:52:51.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 1827 completed, throughput=5MB/sec 2022-04-22T17:52:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:51 smithi121 bash[36902]: audit 2022-04-22T17:52:50.316030+0000 mon.a (mon.0) 2402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:51 smithi115 bash[45495]: audit 2022-04-22T17:52:50.316030+0000 mon.a (mon.0) 2402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:51 smithi115 bash[46934]: audit 2022-04-22T17:52:50.316030+0000 mon.a (mon.0) 2402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:52.010 INFO:tasks.workunit.client.0.smithi115.stdout: 359: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:52.010 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0xUetVde6OI_gQK off=1495790 len=263074 2022-04-22T17:52:52.015 INFO:tasks.workunit.client.0.smithi115.stdout:op 1828 completed, throughput=4.99MB/sec 2022-04-22T17:52:52.015 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-haFED3XXnJy7KDm off=741555 len=1469569 2022-04-22T17:52:52.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 1829 completed, throughput=4.99MB/sec 2022-04-22T17:52:52.018 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yQb-wIhidKnEgN7 off=11907652 len=490723 2022-04-22T17:52:52.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 1830 completed, throughput=4.99MB/sec 2022-04-22T17:52:52.020 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BaUyDPPBH8w4XRj off=571711 len=376487 2022-04-22T17:52:52.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 1831 completed, throughput=4.99MB/sec 2022-04-22T17:52:52.024 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UuqvSezWrCYOczM off=10284317 len=715338 2022-04-22T17:52:52.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 1832 completed, throughput=5MB/sec 2022-04-22T17:52:52.026 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QbchQq-a-9E-2KB off=1506074 len=366180 2022-04-22T17:52:52.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 1833 completed, throughput=5MB/sec 2022-04-22T17:52:52.029 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1OZh_215uE-MB6r off=16740951 len=1011462 2022-04-22T17:52:52.031 INFO:tasks.workunit.client.0.smithi115.stdout:op 1834 completed, throughput=5MB/sec 2022-04-22T17:52:52.031 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YhEfqLOszCE5u7M off=4924566 len=765395 2022-04-22T17:52:52.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 1835 completed, throughput=5MB/sec 2022-04-22T17:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:51.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:52 smithi121 bash[36902]: cluster 2022-04-22T17:52:50.435837+0000 mgr.y (mgr.34107) 966 : cluster [DBG] pgmap v1584: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:52:52.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:52 smithi115 bash[45495]: cluster 2022-04-22T17:52:50.435837+0000 mgr.y (mgr.34107) 966 : cluster [DBG] pgmap v1584: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:52:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:52 smithi115 bash[46934]: cluster 2022-04-22T17:52:50.435837+0000 mgr.y (mgr.34107) 966 : cluster [DBG] pgmap v1584: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:52:53.038 INFO:tasks.workunit.client.0.smithi115.stdout: 360: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:53.038 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cUk-dHL_gOQvLhD off=1791847 len=689441 2022-04-22T17:52:53.040 INFO:tasks.workunit.client.0.smithi115.stdout:op 1836 completed, throughput=4.99MB/sec 2022-04-22T17:52:53.040 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K3gGy_cAAs-5U_2 off=2695984 len=941026 2022-04-22T17:52:53.046 INFO:tasks.workunit.client.0.smithi115.stdout:op 1837 completed, throughput=4.99MB/sec 2022-04-22T17:52:53.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i2ISD3n4-Xnc360 off=69730 len=666344 2022-04-22T17:52:53.052 INFO:tasks.workunit.client.0.smithi115.stdout:op 1838 completed, throughput=4.99MB/sec 2022-04-22T17:52:53.052 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k5EbZaLh3Iyn1ue off=21236712 len=450343 2022-04-22T17:52:53.056 INFO:tasks.workunit.client.0.smithi115.stdout:op 1839 completed, throughput=4.99MB/sec 2022-04-22T17:52:53.056 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ncY7SgnFOYSn2jl off=11842384 len=433736 2022-04-22T17:52:53.057 INFO:tasks.workunit.client.0.smithi115.stdout:op 1840 completed, throughput=4.99MB/sec 2022-04-22T17:52:53.058 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yZri12OUlBzQHzg off=607747 len=1257066 2022-04-22T17:52:53.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 1841 completed, throughput=5MB/sec 2022-04-22T17:52:53.062 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V3Cm8dCQtlIB_s- off=4145497 len=1463334 2022-04-22T17:52:53.066 INFO:tasks.workunit.client.0.smithi115.stdout:op 1842 completed, throughput=5MB/sec 2022-04-22T17:52:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:53 smithi121 bash[36902]: audit 2022-04-22T17:52:52.374509+0000 mon.a (mon.0) 2403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:53 smithi115 bash[45495]: audit 2022-04-22T17:52:52.374509+0000 mon.a (mon.0) 2403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:53 smithi115 bash[46934]: audit 2022-04-22T17:52:52.374509+0000 mon.a (mon.0) 2403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:52:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:52:53.574Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:52:54.067 INFO:tasks.workunit.client.0.smithi115.stdout: 361: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:54.067 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vVWGOCHcx8hvXQF off=3591727 len=630955 2022-04-22T17:52:54.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 1843 completed, throughput=4.99MB/sec 2022-04-22T17:52:54.069 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jbtqEdyjCVOVLSN off=8088399 len=1722031 2022-04-22T17:52:54.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 1844 completed, throughput=4.99MB/sec 2022-04-22T17:52:54.079 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ScpTW2te005jc89 off=3738002 len=1067282 2022-04-22T17:52:54.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 1845 completed, throughput=5MB/sec 2022-04-22T17:52:54.081 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vc7Mq2e-0_zvxBd off=6349600 len=2011033 2022-04-22T17:52:54.090 INFO:tasks.workunit.client.0.smithi115.stdout:op 1846 completed, throughput=5MB/sec 2022-04-22T17:52:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:54 smithi121 bash[36902]: cluster 2022-04-22T17:52:52.436884+0000 mgr.y (mgr.34107) 967 : cluster [DBG] pgmap v1585: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:54 smithi115 bash[45495]: cluster 2022-04-22T17:52:52.436884+0000 mgr.y (mgr.34107) 967 : cluster [DBG] pgmap v1585: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:54 smithi115 bash[46934]: cluster 2022-04-22T17:52:52.436884+0000 mgr.y (mgr.34107) 967 : cluster [DBG] pgmap v1585: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:52:55.091 INFO:tasks.workunit.client.0.smithi115.stdout: 362: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:55.091 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K9jgUYa5uK43I5s off=14044910 len=1246163 2022-04-22T17:52:55.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 1847 completed, throughput=4.99MB/sec 2022-04-22T17:52:55.094 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aOcVuDgGWOe80er off=3076053 len=1128417 2022-04-22T17:52:55.096 INFO:tasks.workunit.client.0.smithi115.stdout:op 1848 completed, throughput=4.99MB/sec 2022-04-22T17:52:55.096 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aJsjEmU1YpumghH off=9192406 len=751636 2022-04-22T17:52:55.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 1849 completed, throughput=5MB/sec 2022-04-22T17:52:55.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7mNoSv3EPVwgH25 off=7862664 len=1553564 2022-04-22T17:52:55.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 1850 completed, throughput=5MB/sec 2022-04-22T17:52:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:55 smithi121 bash[36902]: audit 2022-04-22T17:52:54.997569+0000 mon.a (mon.0) 2404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:55 smithi115 bash[45495]: audit 2022-04-22T17:52:54.997569+0000 mon.a (mon.0) 2404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:55 smithi115 bash[46934]: audit 2022-04-22T17:52:54.997569+0000 mon.a (mon.0) 2404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:52:56.107 INFO:tasks.workunit.client.0.smithi115.stdout: 363: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:56.108 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MaRNRBAKt69RXo5 off=6865281 len=1171878 2022-04-22T17:52:56.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 1851 completed, throughput=4.99MB/sec 2022-04-22T17:52:56.116 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bi6pGft7OYjFc6H off=4872939 len=718912 2022-04-22T17:52:56.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 1852 completed, throughput=4.99MB/sec 2022-04-22T17:52:56.121 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E5O63YB-1orVOlJ off=2454117 len=1458494 2022-04-22T17:52:56.128 INFO:tasks.workunit.client.0.smithi115.stdout:op 1853 completed, throughput=5MB/sec 2022-04-22T17:52:56.128 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O2Jk56mKe6ZTkal off=2094257 len=1698288 2022-04-22T17:52:56.132 INFO:tasks.workunit.client.0.smithi115.stdout:op 1854 completed, throughput=5MB/sec 2022-04-22T17:52:56.132 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RzROhuP07zy10Zu off=1654241 len=1293318 2022-04-22T17:52:56.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 1855 completed, throughput=5MB/sec 2022-04-22T17:52:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:56 smithi121 bash[36902]: cluster 2022-04-22T17:52:54.437869+0000 mgr.y (mgr.34107) 968 : cluster [DBG] pgmap v1586: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:56.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:56 smithi121 bash[36902]: audit 2022-04-22T17:52:55.319813+0000 mon.a (mon.0) 2405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:56.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:56 smithi115 bash[45495]: cluster 2022-04-22T17:52:54.437869+0000 mgr.y (mgr.34107) 968 : cluster [DBG] pgmap v1586: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:56.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:56 smithi115 bash[45495]: audit 2022-04-22T17:52:55.319813+0000 mon.a (mon.0) 2405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:56 smithi115 bash[46934]: cluster 2022-04-22T17:52:54.437869+0000 mgr.y (mgr.34107) 968 : cluster [DBG] pgmap v1586: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:52:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:56 smithi115 bash[46934]: audit 2022-04-22T17:52:55.319813+0000 mon.a (mon.0) 2405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:52:57.136 INFO:tasks.workunit.client.0.smithi115.stdout: 364: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:57.136 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qoey9PfS6qj_FiP off=8635734 len=1259090 2022-04-22T17:52:57.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 1856 completed, throughput=4.99MB/sec 2022-04-22T17:52:57.144 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2HCIxJZtHp9MRUB off=2401581 len=123947 2022-04-22T17:52:57.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 1857 completed, throughput=4.99MB/sec 2022-04-22T17:52:57.145 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qId_rdRnQFiR6vn off=7932684 len=1134183 2022-04-22T17:52:57.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 1858 completed, throughput=5MB/sec 2022-04-22T17:52:57.148 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BC_RTZZKSnNt4Tq off=2238091 len=182514 2022-04-22T17:52:57.149 INFO:tasks.workunit.client.0.smithi115.stdout:op 1859 completed, throughput=5MB/sec 2022-04-22T17:52:57.149 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yZri12OUlBzQHzg off=1009583 len=1039842 2022-04-22T17:52:57.157 INFO:tasks.workunit.client.0.smithi115.stdout:op 1860 completed, throughput=5MB/sec 2022-04-22T17:52:57.157 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TvqI60lUDpWIGk4 off=4132474 len=535107 2022-04-22T17:52:57.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 1861 completed, throughput=5MB/sec 2022-04-22T17:52:57.162 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XalcoIWTOYMa8SN off=2043338 len=153655 2022-04-22T17:52:57.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 1862 completed, throughput=5MB/sec 2022-04-22T17:52:58.164 INFO:tasks.workunit.client.0.smithi115.stdout: 365: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:58.165 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_miA9Mu-4VxNoL5 off=12943638 len=124998 2022-04-22T17:52:58.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 1863 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CY2Hz7plqfQzcXg off=7760062 len=160579 2022-04-22T17:52:58.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 1864 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-agzOkizUuDtP3QX off=2732096 len=202687 2022-04-22T17:52:58.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 1865 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.171 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wlFTxncsH5WXyZb off=14909047 len=615136 2022-04-22T17:52:58.176 INFO:tasks.workunit.client.0.smithi115.stdout:op 1866 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.177 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BEqNTq1TEhoYj04 off=16087574 len=630740 2022-04-22T17:52:58.180 INFO:tasks.workunit.client.0.smithi115.stdout:op 1867 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.180 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nCkyWKiX4BByBC4 off=7278086 len=810921 2022-04-22T17:52:58.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 1868 completed, throughput=4.99MB/sec 2022-04-22T17:52:58.183 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gkiEQptNFsd-J79 off=11301191 len=1345567 2022-04-22T17:52:58.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 1869 completed, throughput=5MB/sec 2022-04-22T17:52:58.184 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-voOgd3P_3sPneuT off=190748 len=208906 2022-04-22T17:52:58.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 1870 completed, throughput=5MB/sec 2022-04-22T17:52:58.190 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_AWko7BG7HXjegi off=86073 len=24359 2022-04-22T17:52:58.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 1871 completed, throughput=5MB/sec 2022-04-22T17:52:58.190 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5w-inXCjzeg3a6f off=2936605 len=856625 2022-04-22T17:52:58.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 1872 completed, throughput=5MB/sec 2022-04-22T17:52:58.204 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O7V3czVDnL60ILa off=1757875 len=1494142 2022-04-22T17:52:58.205 INFO:tasks.workunit.client.0.smithi115.stdout:op 1873 completed, throughput=5MB/sec 2022-04-22T17:52:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:58 smithi121 bash[36902]: cluster 2022-04-22T17:52:56.438906+0000 mgr.y (mgr.34107) 969 : cluster [DBG] pgmap v1587: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:52:58 smithi121 bash[36902]: audit 2022-04-22T17:52:57.377564+0000 mon.a (mon.0) 2406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:58 smithi115 bash[45495]: cluster 2022-04-22T17:52:56.438906+0000 mgr.y (mgr.34107) 969 : cluster [DBG] pgmap v1587: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:52:58 smithi115 bash[45495]: audit 2022-04-22T17:52:57.377564+0000 mon.a (mon.0) 2406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:58 smithi115 bash[46934]: cluster 2022-04-22T17:52:56.438906+0000 mgr.y (mgr.34107) 969 : cluster [DBG] pgmap v1587: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:52:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:52:58 smithi115 bash[46934]: audit 2022-04-22T17:52:57.377564+0000 mon.a (mon.0) 2406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:52:59.198 INFO:tasks.workunit.client.0.smithi115.stdout: 366: throughput=4.99MB/sec pending data=0 2022-04-22T17:52:59.199 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zH5FJJyhiab4R90 off=9026701 len=1622269 2022-04-22T17:52:59.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 1874 completed, throughput=4.99MB/sec 2022-04-22T17:52:59.202 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dVdxeKX4bxOqSKb off=8560281 len=581198 2022-04-22T17:52:59.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 1875 completed, throughput=5MB/sec 2022-04-22T17:52:59.204 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1vkbsrkTUunFwdu off=843998 len=1317371 2022-04-22T17:52:59.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 1876 completed, throughput=5MB/sec 2022-04-22T17:52:59.214 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-N4yJlA6wZzdzJ1Z off=3816236 len=418303 2022-04-22T17:52:59.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 1877 completed, throughput=5MB/sec 2022-04-22T17:52:59.217 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pbjK0PPMXdMJGm3 off=4464124 len=705764 2022-04-22T17:52:59.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 1878 completed, throughput=5MB/sec 2022-04-22T17:53:00.220 INFO:tasks.workunit.client.0.smithi115.stdout: 367: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:00.221 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q8v5qSi_dV9qi74 off=161433 len=1666200 2022-04-22T17:53:00.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 1879 completed, throughput=4.99MB/sec 2022-04-22T17:53:00.225 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Bc0ZXaecBmrRoxj off=16615957 len=824778 2022-04-22T17:53:00.226 INFO:tasks.workunit.client.0.smithi115.stdout:op 1880 completed, throughput=4.99MB/sec 2022-04-22T17:53:00.226 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x7eiBnfCyLII4K4 off=3846093 len=590797 2022-04-22T17:53:00.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 1881 completed, throughput=5MB/sec 2022-04-22T17:53:00.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-J_GZa4_-4yT-BHM off=2793767 len=1054874 2022-04-22T17:53:00.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 1882 completed, throughput=5MB/sec 2022-04-22T17:53:00.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9_YvaiOl1zlCZ4L off=220272 len=248460 2022-04-22T17:53:00.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 1883 completed, throughput=5MB/sec 2022-04-22T17:53:00.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yatnlLbNLdlXeuX off=4219069 len=862674 2022-04-22T17:53:00.244 INFO:tasks.workunit.client.0.smithi115.stdout:op 1884 completed, throughput=5MB/sec 2022-04-22T17:53:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:00 smithi121 bash[36902]: cluster 2022-04-22T17:52:58.440043+0000 mgr.y (mgr.34107) 970 : cluster [DBG] pgmap v1588: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:00 smithi121 bash[36902]: audit 2022-04-22T17:53:00.001372+0000 mon.a (mon.0) 2407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:00 smithi115 bash[45495]: cluster 2022-04-22T17:52:58.440043+0000 mgr.y (mgr.34107) 970 : cluster [DBG] pgmap v1588: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:00 smithi115 bash[45495]: audit 2022-04-22T17:53:00.001372+0000 mon.a (mon.0) 2407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:00 smithi115 bash[46934]: cluster 2022-04-22T17:52:58.440043+0000 mgr.y (mgr.34107) 970 : cluster [DBG] pgmap v1588: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:00 smithi115 bash[46934]: audit 2022-04-22T17:53:00.001372+0000 mon.a (mon.0) 2407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:01.244 INFO:tasks.workunit.client.0.smithi115.stdout: 368: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:01.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3x4_HXlbGLPhfvB off=5925425 len=281717 2022-04-22T17:53:01.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 1885 completed, throughput=4.99MB/sec 2022-04-22T17:53:01.246 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yPS8HYrxoN6r3PB off=9116847 len=1162308 2022-04-22T17:53:01.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 1886 completed, throughput=4.99MB/sec 2022-04-22T17:53:01.250 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2LyUViBJVvVAJi2 off=11409346 len=1433356 2022-04-22T17:53:01.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 1887 completed, throughput=4.99MB/sec 2022-04-22T17:53:01.259 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R3yIj53tWlyrcNm off=1451406 len=706805 2022-04-22T17:53:01.261 INFO:tasks.workunit.client.0.smithi115.stdout:op 1888 completed, throughput=5MB/sec 2022-04-22T17:53:01.261 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNW8Ly8HLz5qVqa off=1992793 len=735831 2022-04-22T17:53:01.262 INFO:tasks.workunit.client.0.smithi115.stdout:op 1889 completed, throughput=5MB/sec 2022-04-22T17:53:01.263 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7oySEZJex0HoTxF off=687687 len=1867090 2022-04-22T17:53:01.270 INFO:tasks.workunit.client.0.smithi115.stdout:op 1890 completed, throughput=5MB/sec 2022-04-22T17:53:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:01 smithi121 bash[36902]: audit 2022-04-22T17:53:00.323585+0000 mon.a (mon.0) 2408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:01 smithi115 bash[45495]: audit 2022-04-22T17:53:00.323585+0000 mon.a (mon.0) 2408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:01.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:01 smithi115 bash[46934]: audit 2022-04-22T17:53:00.323585+0000 mon.a (mon.0) 2408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:01.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:02.270 INFO:tasks.workunit.client.0.smithi115.stdout: 369: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:02.271 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ggQQSxmpyovnwYB off=10239335 len=1515566 2022-04-22T17:53:02.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 1891 completed, throughput=4.99MB/sec 2022-04-22T17:53:02.274 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BVEiKSC3lTVOXcu off=3637998 len=1132944 2022-04-22T17:53:02.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 1892 completed, throughput=5MB/sec 2022-04-22T17:53:02.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lznRfHVcmTcXqIU off=6867787 len=1518508 2022-04-22T17:53:02.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 1893 completed, throughput=5MB/sec 2022-04-22T17:53:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:02 smithi121 bash[36902]: cluster 2022-04-22T17:53:00.441077+0000 mgr.y (mgr.34107) 971 : cluster [DBG] pgmap v1589: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:02 smithi115 bash[46934]: cluster 2022-04-22T17:53:00.441077+0000 mgr.y (mgr.34107) 971 : cluster [DBG] pgmap v1589: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:02.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:02 smithi115 bash[45495]: cluster 2022-04-22T17:53:00.441077+0000 mgr.y (mgr.34107) 971 : cluster [DBG] pgmap v1589: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:03.282 INFO:tasks.workunit.client.0.smithi115.stdout: 370: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:03.282 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gf4wNcHG8x5A8Xd off=586574 len=638683 2022-04-22T17:53:03.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 1894 completed, throughput=4.99MB/sec 2022-04-22T17:53:03.287 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Zx7eYNNtKBfsEsb off=4838904 len=387402 2022-04-22T17:53:03.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 1895 completed, throughput=4.99MB/sec 2022-04-22T17:53:03.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xMu4uZJd6oFEmEh off=182372 len=1656842 2022-04-22T17:53:03.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 1896 completed, throughput=4.99MB/sec 2022-04-22T17:53:03.300 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3bzKAFpYDbSifZZ off=8407734 len=999337 2022-04-22T17:53:03.303 INFO:tasks.workunit.client.0.smithi115.stdout:op 1897 completed, throughput=5MB/sec 2022-04-22T17:53:03.303 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ks_TZG1kqzlWiGL off=15606478 len=1458264 2022-04-22T17:53:03.306 INFO:tasks.workunit.client.0.smithi115.stdout:op 1898 completed, throughput=5MB/sec 2022-04-22T17:53:03.307 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fzdfPeFTsQyhzDt off=2293261 len=1568575 2022-04-22T17:53:03.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 1899 completed, throughput=5MB/sec 2022-04-22T17:53:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:03 smithi121 bash[36902]: audit 2022-04-22T17:53:02.380560+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:03 smithi115 bash[45495]: audit 2022-04-22T17:53:02.380560+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:03 smithi115 bash[46934]: audit 2022-04-22T17:53:02.380560+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:03.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:04.314 INFO:tasks.workunit.client.0.smithi115.stdout: 371: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:04.314 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8fTcUwgoKCykOQl off=995304 len=1814827 2022-04-22T17:53:04.318 INFO:tasks.workunit.client.0.smithi115.stdout:op 1900 completed, throughput=4.99MB/sec 2022-04-22T17:53:04.318 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gr-mT4X7JuIlt1K off=2048822 len=1244836 2022-04-22T17:53:04.320 INFO:tasks.workunit.client.0.smithi115.stdout:op 1901 completed, throughput=5MB/sec 2022-04-22T17:53:04.320 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aTnVSw_3tUgZc47 off=3469097 len=1237368 2022-04-22T17:53:04.323 INFO:tasks.workunit.client.0.smithi115.stdout:op 1902 completed, throughput=5MB/sec 2022-04-22T17:53:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:04 smithi121 bash[36902]: cluster 2022-04-22T17:53:02.442175+0000 mgr.y (mgr.34107) 972 : cluster [DBG] pgmap v1590: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:04 smithi115 bash[45495]: cluster 2022-04-22T17:53:02.442175+0000 mgr.y (mgr.34107) 972 : cluster [DBG] pgmap v1590: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:04 smithi115 bash[46934]: cluster 2022-04-22T17:53:02.442175+0000 mgr.y (mgr.34107) 972 : cluster [DBG] pgmap v1590: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:05.257 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:53:05.323 INFO:tasks.workunit.client.0.smithi115.stdout: 372: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:05.324 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qBCxLIhfddXtYNd off=24726229 len=312716 2022-04-22T17:53:05.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 1903 completed, throughput=4.99MB/sec 2022-04-22T17:53:05.326 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HZ-KZ4rzgOHEeYZ off=10618726 len=65463 2022-04-22T17:53:05.327 INFO:tasks.workunit.client.0.smithi115.stdout:op 1904 completed, throughput=4.99MB/sec 2022-04-22T17:53:05.327 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CIwAAwPoX_53u0B off=6713771 len=1742478 2022-04-22T17:53:05.335 INFO:tasks.workunit.client.0.smithi115.stdout:op 1905 completed, throughput=4.99MB/sec 2022-04-22T17:53:05.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AM9G_9jX30j-pBI off=7199771 len=537527 2022-04-22T17:53:05.339 INFO:tasks.workunit.client.0.smithi115.stdout:op 1906 completed, throughput=4.99MB/sec 2022-04-22T17:53:05.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RBzLW7qYnmHsDR8 off=737288 len=447295 2022-04-22T17:53:05.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 1907 completed, throughput=5MB/sec 2022-04-22T17:53:05.344 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mIJmBV1SRF93wYT off=3764466 len=1344876 2022-04-22T17:53:05.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 1908 completed, throughput=5MB/sec 2022-04-22T17:53:05.352 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DLSF-6HmG7NK7Ln off=20944612 len=1408643 2022-04-22T17:53:05.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 1909 completed, throughput=5MB/sec 2022-04-22T17:53:05.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:05 smithi121 bash[36902]: audit 2022-04-22T17:53:05.004331+0000 mon.a (mon.0) 2410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:05 smithi115 bash[45495]: audit 2022-04-22T17:53:05.004331+0000 mon.a (mon.0) 2410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:05 smithi115 bash[46934]: audit 2022-04-22T17:53:05.004331+0000 mon.a (mon.0) 2410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:05.610 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (16m) 5m ago 29m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (29m) 6m ago 29m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (18m) 5m ago 30m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (17m) 6m ago 30m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (25m) 6m ago 35m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (26m) 5m ago 37m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (25m) 5m ago 37m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (24m) 6m ago 35m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:53:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (24m) 5m ago 35m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (17m) 5m ago 30m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (17m) 6m ago 30m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (22m) 5m ago 34m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (22m) 5m ago 34m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (21m) 5m ago 33m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (21m) 5m ago 33m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (20m) 6m ago 32m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (19m) 6m ago 32m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (19m) 6m ago 31m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:53:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (18m) 6m ago 31m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:53:05.613 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (16m) 6m ago 30m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:53:06.024 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:53:06.063 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:53:06.063 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:53:06.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:06.063 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:53:06.063 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:53:06.064 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:53:06.065 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:53:06.358 INFO:tasks.workunit.client.0.smithi115.stdout: 373: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:06.359 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6xLPhxqg5mJarpU off=0 len=1795368 2022-04-22T17:53:06.363 INFO:tasks.workunit.client.0.smithi115.stdout:op 1910 completed, throughput=4.99MB/sec 2022-04-22T17:53:06.363 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PGhiRhv1uE75XXy off=2762909 len=1344403 2022-04-22T17:53:06.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 1911 completed, throughput=5MB/sec 2022-04-22T17:53:06.371 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7YWUDtoqXv9KAMM off=15425564 len=829898 2022-04-22T17:53:06.377 INFO:tasks.workunit.client.0.smithi115.stdout:op 1912 completed, throughput=5MB/sec 2022-04-22T17:53:06.377 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NNv3P9Udrp0iu3v off=3145646 len=91512 2022-04-22T17:53:06.378 INFO:tasks.workunit.client.0.smithi115.stdout:op 1913 completed, throughput=5MB/sec 2022-04-22T17:53:06.378 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kJjvurWsV0aiVKA off=2392680 len=64116 2022-04-22T17:53:06.382 INFO:tasks.workunit.client.0.smithi115.stdout:op 1914 completed, throughput=5MB/sec 2022-04-22T17:53:06.382 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1mNq64LdJv4Yp24 off=19963172 len=767520 2022-04-22T17:53:06.383 INFO:tasks.workunit.client.0.smithi115.stdout:op 1915 completed, throughput=5MB/sec 2022-04-22T17:53:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:06 smithi121 bash[36902]: cluster 2022-04-22T17:53:04.443421+0000 mgr.y (mgr.34107) 973 : cluster [DBG] pgmap v1591: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:06 smithi121 bash[36902]: audit 2022-04-22T17:53:05.233879+0000 mgr.y (mgr.34107) 974 : audit [DBG] from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:06 smithi121 bash[36902]: audit 2022-04-22T17:53:05.327193+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:06 smithi121 bash[36902]: audit 2022-04-22T17:53:06.016614+0000 mon.a (mon.0) 2412 : audit [DBG] from='client.? 172.21.15.115:0/2207822299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[45495]: cluster 2022-04-22T17:53:04.443421+0000 mgr.y (mgr.34107) 973 : cluster [DBG] pgmap v1591: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[45495]: audit 2022-04-22T17:53:05.233879+0000 mgr.y (mgr.34107) 974 : audit [DBG] from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[45495]: audit 2022-04-22T17:53:05.327193+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[45495]: audit 2022-04-22T17:53:06.016614+0000 mon.a (mon.0) 2412 : audit [DBG] from='client.? 172.21.15.115:0/2207822299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[46934]: cluster 2022-04-22T17:53:04.443421+0000 mgr.y (mgr.34107) 973 : cluster [DBG] pgmap v1591: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[46934]: audit 2022-04-22T17:53:05.233879+0000 mgr.y (mgr.34107) 974 : audit [DBG] from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[46934]: audit 2022-04-22T17:53:05.327193+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:06 smithi115 bash[46934]: audit 2022-04-22T17:53:06.016614+0000 mon.a (mon.0) 2412 : audit [DBG] from='client.? 172.21.15.115:0/2207822299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:07.383 INFO:tasks.workunit.client.0.smithi115.stdout: 374: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:07.383 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NiYWxltIfIuqE9_ off=13321648 len=1215317 2022-04-22T17:53:07.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 1916 completed, throughput=4.99MB/sec 2022-04-22T17:53:07.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kebur2Y_TALBPbz off=1031171 len=2061520 2022-04-22T17:53:07.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 1917 completed, throughput=5MB/sec 2022-04-22T17:53:07.395 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qy4DyZvr_z3jr5u off=984329 len=292642 2022-04-22T17:53:07.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 1918 completed, throughput=5MB/sec 2022-04-22T17:53:07.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oKb2ch_zaC-JN-Y off=3670174 len=2084623 2022-04-22T17:53:07.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 1919 completed, throughput=5MB/sec 2022-04-22T17:53:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:07 smithi121 bash[36902]: audit 2022-04-22T17:53:05.581321+0000 mgr.y (mgr.34107) 975 : audit [DBG] from='client.45373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:07 smithi115 bash[45495]: audit 2022-04-22T17:53:05.581321+0000 mgr.y (mgr.34107) 975 : audit [DBG] from='client.45373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:07 smithi115 bash[46934]: audit 2022-04-22T17:53:05.581321+0000 mgr.y (mgr.34107) 975 : audit [DBG] from='client.45373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:08.404 INFO:tasks.workunit.client.0.smithi115.stdout: 375: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:08.404 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CxmSjzgWRxcUQou off=2524448 len=1617834 2022-04-22T17:53:08.407 INFO:tasks.workunit.client.0.smithi115.stdout:op 1920 completed, throughput=4.99MB/sec 2022-04-22T17:53:08.408 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rMU7Q-Ndm_SXn4v off=16287995 len=1520186 2022-04-22T17:53:08.418 INFO:tasks.workunit.client.0.smithi115.stdout:op 1921 completed, throughput=5MB/sec 2022-04-22T17:53:08.418 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p-nyKzfrfjl9SLd off=0 len=1230113 2022-04-22T17:53:08.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 1922 completed, throughput=5MB/sec 2022-04-22T17:53:08.423 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AanayK8vGmjiTEa off=1241916 len=2052985 2022-04-22T17:53:08.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 1923 completed, throughput=5MB/sec 2022-04-22T17:53:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:08 smithi121 bash[36902]: cluster 2022-04-22T17:53:06.444523+0000 mgr.y (mgr.34107) 976 : cluster [DBG] pgmap v1592: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:08 smithi121 bash[36902]: audit 2022-04-22T17:53:07.383827+0000 mon.a (mon.0) 2413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:08 smithi115 bash[45495]: cluster 2022-04-22T17:53:06.444523+0000 mgr.y (mgr.34107) 976 : cluster [DBG] pgmap v1592: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:08 smithi115 bash[45495]: audit 2022-04-22T17:53:07.383827+0000 mon.a (mon.0) 2413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:08 smithi115 bash[46934]: cluster 2022-04-22T17:53:06.444523+0000 mgr.y (mgr.34107) 976 : cluster [DBG] pgmap v1592: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:08 smithi115 bash[46934]: audit 2022-04-22T17:53:07.383827+0000 mon.a (mon.0) 2413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:09.432 INFO:tasks.workunit.client.0.smithi115.stdout: 376: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:09.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MlJcjk1NCD1_dxi off=3955090 len=307580 2022-04-22T17:53:09.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 1924 completed, throughput=4.99MB/sec 2022-04-22T17:53:09.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QTc4qXCBZPin0vK off=10795415 len=1689091 2022-04-22T17:53:09.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 1925 completed, throughput=5MB/sec 2022-04-22T17:53:09.446 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5oxPCTC7vEGS2-e off=15065819 len=1148817 2022-04-22T17:53:09.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 1926 completed, throughput=5MB/sec 2022-04-22T17:53:09.451 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8BdWXfHkgKsbO5e off=456510 len=952432 2022-04-22T17:53:09.453 INFO:tasks.workunit.client.0.smithi115.stdout:op 1927 completed, throughput=5MB/sec 2022-04-22T17:53:10.453 INFO:tasks.workunit.client.0.smithi115.stdout: 377: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:10.453 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2HCIxJZtHp9MRUB off=10266277 len=501957 2022-04-22T17:53:10.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 1928 completed, throughput=4.99MB/sec 2022-04-22T17:53:10.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SKMSarVhikeZbxO off=2064825 len=1474436 2022-04-22T17:53:10.465 INFO:tasks.workunit.client.0.smithi115.stdout:op 1929 completed, throughput=4.99MB/sec 2022-04-22T17:53:10.466 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q9l-kUT4w9WTYIJ off=8380198 len=2018619 2022-04-22T17:53:10.473 INFO:tasks.workunit.client.0.smithi115.stdout:op 1930 completed, throughput=5MB/sec 2022-04-22T17:53:10.474 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DJtt49rkjGxNO2T off=6587522 len=962009 2022-04-22T17:53:10.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 1931 completed, throughput=5MB/sec 2022-04-22T17:53:10.476 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-efsAChN2e4an6J5 off=1982077 len=1041495 2022-04-22T17:53:10.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 1932 completed, throughput=5MB/sec 2022-04-22T17:53:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:10 smithi121 bash[36902]: cluster 2022-04-22T17:53:08.445821+0000 mgr.y (mgr.34107) 977 : cluster [DBG] pgmap v1593: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:10 smithi121 bash[36902]: audit 2022-04-22T17:53:10.007607+0000 mon.a (mon.0) 2414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:10.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:10 smithi115 bash[45495]: cluster 2022-04-22T17:53:08.445821+0000 mgr.y (mgr.34107) 977 : cluster [DBG] pgmap v1593: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:10.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:10 smithi115 bash[45495]: audit 2022-04-22T17:53:10.007607+0000 mon.a (mon.0) 2414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:10 smithi115 bash[46934]: cluster 2022-04-22T17:53:08.445821+0000 mgr.y (mgr.34107) 977 : cluster [DBG] pgmap v1593: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:10 smithi115 bash[46934]: audit 2022-04-22T17:53:10.007607+0000 mon.a (mon.0) 2414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:11.482 INFO:tasks.workunit.client.0.smithi115.stdout: 378: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:11.482 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gNW8Ly8HLz5qVqa off=767266 len=404218 2022-04-22T17:53:11.487 INFO:tasks.workunit.client.0.smithi115.stdout:op 1933 completed, throughput=4.99MB/sec 2022-04-22T17:53:11.487 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I5NEt4Vw6V813v5 off=1195619 len=271076 2022-04-22T17:53:11.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 1934 completed, throughput=4.99MB/sec 2022-04-22T17:53:11.488 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FQ-vPNQzt4L16Ry off=17450168 len=2008563 2022-04-22T17:53:11.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 1935 completed, throughput=5MB/sec 2022-04-22T17:53:11.492 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ch8CnLXKBzVlRYw off=2415083 len=1366391 2022-04-22T17:53:11.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 1936 completed, throughput=5MB/sec 2022-04-22T17:53:11.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GlzQxG9SOkTStHK off=2121651 len=1241885 2022-04-22T17:53:11.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 1937 completed, throughput=5MB/sec 2022-04-22T17:53:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:11 smithi115 bash[45495]: audit 2022-04-22T17:53:10.330375+0000 mon.a (mon.0) 2415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:11 smithi115 bash[46934]: audit 2022-04-22T17:53:10.330375+0000 mon.a (mon.0) 2415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:11 smithi121 bash[36902]: audit 2022-04-22T17:53:10.330375+0000 mon.a (mon.0) 2415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:12.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:11.693Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:12.499 INFO:tasks.workunit.client.0.smithi115.stdout: 379: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:12.499 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s6d2R5CQSfJfv8f off=14135171 len=1789572 2022-04-22T17:53:12.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 1938 completed, throughput=4.99MB/sec 2022-04-22T17:53:12.503 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VCGDjuR113NMFmG off=328283 len=1071587 2022-04-22T17:53:12.504 INFO:tasks.workunit.client.0.smithi115.stdout:op 1939 completed, throughput=5MB/sec 2022-04-22T17:53:12.504 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IxwN_kzLDmbEFZR off=11075922 len=890998 2022-04-22T17:53:12.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 1940 completed, throughput=5MB/sec 2022-04-22T17:53:12.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Nuvnm4DD53yZfqh off=1678209 len=592648 2022-04-22T17:53:12.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 1941 completed, throughput=5MB/sec 2022-04-22T17:53:12.512 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-38euddPBT2-XPPz off=17108217 len=2083444 2022-04-22T17:53:12.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 1942 completed, throughput=5MB/sec 2022-04-22T17:53:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:12 smithi115 bash[45495]: cluster 2022-04-22T17:53:10.446942+0000 mgr.y (mgr.34107) 978 : cluster [DBG] pgmap v1594: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:53:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:12 smithi115 bash[46934]: cluster 2022-04-22T17:53:10.446942+0000 mgr.y (mgr.34107) 978 : cluster [DBG] pgmap v1594: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:53:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:12 smithi121 bash[36902]: cluster 2022-04-22T17:53:10.446942+0000 mgr.y (mgr.34107) 978 : cluster [DBG] pgmap v1594: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:53:13.520 INFO:tasks.workunit.client.0.smithi115.stdout: 380: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:13.539 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x7Gzl4OdF_IiHYp off=14550466 len=993715 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 1943 completed, throughput=4.99MB/sec 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-snwadpro7ut7Yy6 off=18876431 len=140393 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 1944 completed, throughput=4.99MB/sec 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nxcNgFLr041EAR7 off=1520492 len=1535113 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 1945 completed, throughput=5MB/sec 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sop39zVwV7cFvv7 off=6125492 len=1568836 2022-04-22T17:53:13.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 1946 completed, throughput=5MB/sec 2022-04-22T17:53:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:13 smithi115 bash[45495]: audit 2022-04-22T17:53:12.387111+0000 mon.a (mon.0) 2416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:13 smithi115 bash[46934]: audit 2022-04-22T17:53:12.387111+0000 mon.a (mon.0) 2416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:13 smithi121 bash[36902]: audit 2022-04-22T17:53:12.387111+0000 mon.a (mon.0) 2416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:13.576Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:14.539 INFO:tasks.workunit.client.0.smithi115.stdout: 381: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:14.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9_1OoCH4Jt9OEv7 off=12349212 len=707763 2022-04-22T17:53:14.542 INFO:tasks.workunit.client.0.smithi115.stdout:op 1947 completed, throughput=4.99MB/sec 2022-04-22T17:53:14.543 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8FyVocw0ko3c6kZ off=1250051 len=1383501 2022-04-22T17:53:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[46934]: cluster 2022-04-22T17:53:12.448046+0000 mgr.y (mgr.34107) 979 : cluster [DBG] pgmap v1595: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[46934]: audit 2022-04-22T17:53:13.506283+0000 mon.a (mon.0) 2417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:53:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[46934]: audit 2022-04-22T17:53:13.506843+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:53:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[45495]: cluster 2022-04-22T17:53:12.448046+0000 mgr.y (mgr.34107) 979 : cluster [DBG] pgmap v1595: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[45495]: audit 2022-04-22T17:53:13.506283+0000 mon.a (mon.0) 2417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:53:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:14 smithi115 bash[45495]: audit 2022-04-22T17:53:13.506843+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:53:14.546 INFO:tasks.workunit.client.0.smithi115.stdout:op 1948 completed, throughput=4.99MB/sec 2022-04-22T17:53:14.546 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cUk-dHL_gOQvLhD off=13146205 len=1963409 2022-04-22T17:53:14.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 1949 completed, throughput=5MB/sec 2022-04-22T17:53:14.556 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--efUPiiaJ7AzVKB off=12440544 len=647846 2022-04-22T17:53:14.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 1950 completed, throughput=5MB/sec 2022-04-22T17:53:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:14 smithi121 bash[36902]: cluster 2022-04-22T17:53:12.448046+0000 mgr.y (mgr.34107) 979 : cluster [DBG] pgmap v1595: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:53:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:14 smithi121 bash[36902]: audit 2022-04-22T17:53:13.506283+0000 mon.a (mon.0) 2417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:53:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:14 smithi121 bash[36902]: audit 2022-04-22T17:53:13.506843+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:53:15.558 INFO:tasks.workunit.client.0.smithi115.stdout: 382: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:15.558 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3zVIMtUs8qeo4UN off=12879853 len=1311689 2022-04-22T17:53:15.566 INFO:tasks.workunit.client.0.smithi115.stdout:op 1951 completed, throughput=4.99MB/sec 2022-04-22T17:53:15.566 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZPpZC_YNukYhzGB off=0 len=1890018 2022-04-22T17:53:15.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 1952 completed, throughput=5MB/sec 2022-04-22T17:53:15.570 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ro-SMuhCXC5eMDT off=2312585 len=316385 2022-04-22T17:53:15.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 1953 completed, throughput=5MB/sec 2022-04-22T17:53:15.575 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0admHP8ncxTx4-_ off=7362805 len=929599 2022-04-22T17:53:15.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 1954 completed, throughput=5MB/sec 2022-04-22T17:53:15.577 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iP9IllxiyEYlJAb off=3353176 len=682930 2022-04-22T17:53:15.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 1955 completed, throughput=5MB/sec 2022-04-22T17:53:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:15 smithi121 bash[36902]: audit 2022-04-22T17:53:15.010757+0000 mon.a (mon.0) 2419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:15 smithi115 bash[45495]: audit 2022-04-22T17:53:15.010757+0000 mon.a (mon.0) 2419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:15 smithi115 bash[46934]: audit 2022-04-22T17:53:15.010757+0000 mon.a (mon.0) 2419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:16.579 INFO:tasks.workunit.client.0.smithi115.stdout: 383: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:16.579 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BUq66W4qzc5fLv8 off=2219071 len=1583228 2022-04-22T17:53:16.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 1956 completed, throughput=4.99MB/sec 2022-04-22T17:53:16.583 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3852XkMWz1j-9a5 off=7339608 len=335636 2022-04-22T17:53:16.588 INFO:tasks.workunit.client.0.smithi115.stdout:op 1957 completed, throughput=4.99MB/sec 2022-04-22T17:53:16.589 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zye_xDwokYeAUah off=397466 len=591231 2022-04-22T17:53:16.593 INFO:tasks.workunit.client.0.smithi115.stdout:op 1958 completed, throughput=4.99MB/sec 2022-04-22T17:53:16.593 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ys16LL95TsqGTI_ off=2493500 len=1824935 2022-04-22T17:53:16.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 1959 completed, throughput=5MB/sec 2022-04-22T17:53:16.602 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l3l6ZQku8c_8gJZ off=6415335 len=1708068 2022-04-22T17:53:16.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 1960 completed, throughput=5MB/sec 2022-04-22T17:53:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:16 smithi121 bash[36902]: cluster 2022-04-22T17:53:14.449154+0000 mgr.y (mgr.34107) 980 : cluster [DBG] pgmap v1596: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:16 smithi121 bash[36902]: audit 2022-04-22T17:53:15.334047+0000 mon.a (mon.0) 2420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:16 smithi115 bash[45495]: cluster 2022-04-22T17:53:14.449154+0000 mgr.y (mgr.34107) 980 : cluster [DBG] pgmap v1596: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:16 smithi115 bash[45495]: audit 2022-04-22T17:53:15.334047+0000 mon.a (mon.0) 2420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:16 smithi115 bash[46934]: cluster 2022-04-22T17:53:14.449154+0000 mgr.y (mgr.34107) 980 : cluster [DBG] pgmap v1596: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:16 smithi115 bash[46934]: audit 2022-04-22T17:53:15.334047+0000 mon.a (mon.0) 2420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:17.605 INFO:tasks.workunit.client.0.smithi115.stdout: 384: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:17.605 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s4KLzr4KpamRVrb off=3311131 len=27224 2022-04-22T17:53:17.608 INFO:tasks.workunit.client.0.smithi115.stdout:op 1961 completed, throughput=4.99MB/sec 2022-04-22T17:53:17.609 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--1LDt-RMmtmbIFn off=3102606 len=361079 2022-04-22T17:53:17.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 1962 completed, throughput=4.99MB/sec 2022-04-22T17:53:17.610 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Lpp40ID9gEceRiS off=4011670 len=1538364 2022-04-22T17:53:17.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 1963 completed, throughput=4.99MB/sec 2022-04-22T17:53:17.612 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bIFmXlymQUY2YX1 off=431172 len=1734699 2022-04-22T17:53:17.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 1964 completed, throughput=5MB/sec 2022-04-22T17:53:17.661 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pb_Bg1NNuA7OVUH off=7225912 len=1084011 2022-04-22T17:53:17.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 1965 completed, throughput=5MB/sec 2022-04-22T17:53:18.624 INFO:tasks.workunit.client.0.smithi115.stdout: 385: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:18.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kKCIbNFfOG5jljO off=15609603 len=437847 2022-04-22T17:53:18.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 1966 completed, throughput=4.99MB/sec 2022-04-22T17:53:18.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mWTXPWavOYQjREs off=18807276 len=1304561 2022-04-22T17:53:18.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 1967 completed, throughput=4.99MB/sec 2022-04-22T17:53:18.634 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1UAgbUWsUIkCOtu off=0 len=1069067 2022-04-22T17:53:18.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 1968 completed, throughput=4.99MB/sec 2022-04-22T17:53:18.641 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x3mNM-XLRaP7gD- off=2476308 len=1995250 2022-04-22T17:53:18.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 1969 completed, throughput=5MB/sec 2022-04-22T17:53:18.644 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u9JgHqwQO0ggkHT off=5798978 len=1118249 2022-04-22T17:53:18.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 1970 completed, throughput=5MB/sec 2022-04-22T17:53:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:18 smithi121 bash[36902]: cluster 2022-04-22T17:53:16.450161+0000 mgr.y (mgr.34107) 981 : cluster [DBG] pgmap v1597: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:53:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:18 smithi121 bash[36902]: audit 2022-04-22T17:53:17.390244+0000 mon.a (mon.0) 2421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:18 smithi115 bash[45495]: cluster 2022-04-22T17:53:16.450161+0000 mgr.y (mgr.34107) 981 : cluster [DBG] pgmap v1597: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:53:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:18 smithi115 bash[45495]: audit 2022-04-22T17:53:17.390244+0000 mon.a (mon.0) 2421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:18 smithi115 bash[46934]: cluster 2022-04-22T17:53:16.450161+0000 mgr.y (mgr.34107) 981 : cluster [DBG] pgmap v1597: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:53:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:18 smithi115 bash[46934]: audit 2022-04-22T17:53:17.390244+0000 mon.a (mon.0) 2421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:19.646 INFO:tasks.workunit.client.0.smithi115.stdout: 386: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:19.646 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--i7n5wbFZICej0o off=5365673 len=968738 2022-04-22T17:53:19.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 1971 completed, throughput=4.99MB/sec 2022-04-22T17:53:19.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_q0i1oywZdnXNyM off=372318 len=1131554 2022-04-22T17:53:19.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 1972 completed, throughput=4.99MB/sec 2022-04-22T17:53:19.704 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Mgc24l4mkukDQGi off=2307683 len=1663387 2022-04-22T17:53:19.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 1973 completed, throughput=5MB/sec 2022-04-22T17:53:19.706 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KbdMvk8ljBjGJ61 off=6868885 len=529402 2022-04-22T17:53:19.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 1974 completed, throughput=5MB/sec 2022-04-22T17:53:19.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dwlCWiUjKeP2hdl off=10291412 len=1392864 2022-04-22T17:53:19.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 1975 completed, throughput=5MB/sec 2022-04-22T17:53:20.674 INFO:tasks.workunit.client.0.smithi115.stdout: 387: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:20.674 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4PxARb-SPN5lvpV off=609965 len=1094344 2022-04-22T17:53:20.677 INFO:tasks.workunit.client.0.smithi115.stdout:op 1976 completed, throughput=4.99MB/sec 2022-04-22T17:53:20.677 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6dcfI0CVJZVNv8M off=9678561 len=671690 2022-04-22T17:53:20.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 1977 completed, throughput=4.99MB/sec 2022-04-22T17:53:20.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TJ-yUP-k-fzOfW8 off=14076109 len=192900 2022-04-22T17:53:20.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 1978 completed, throughput=4.99MB/sec 2022-04-22T17:53:20.685 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U5zR-pXkU95pjV9 off=13706875 len=1829263 2022-04-22T17:53:20.688 INFO:tasks.workunit.client.0.smithi115.stdout:op 1979 completed, throughput=5MB/sec 2022-04-22T17:53:20.688 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8sE5ok_whAKPzEx off=1236345 len=863138 2022-04-22T17:53:20.690 INFO:tasks.workunit.client.0.smithi115.stdout:op 1980 completed, throughput=5MB/sec 2022-04-22T17:53:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:20 smithi121 bash[36902]: cluster 2022-04-22T17:53:18.451561+0000 mgr.y (mgr.34107) 982 : cluster [DBG] pgmap v1598: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:20 smithi121 bash[36902]: audit 2022-04-22T17:53:20.013809+0000 mon.a (mon.0) 2422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:20 smithi115 bash[45495]: cluster 2022-04-22T17:53:18.451561+0000 mgr.y (mgr.34107) 982 : cluster [DBG] pgmap v1598: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:20 smithi115 bash[45495]: audit 2022-04-22T17:53:20.013809+0000 mon.a (mon.0) 2422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:20 smithi115 bash[46934]: cluster 2022-04-22T17:53:18.451561+0000 mgr.y (mgr.34107) 982 : cluster [DBG] pgmap v1598: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:53:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:20 smithi115 bash[46934]: audit 2022-04-22T17:53:20.013809+0000 mon.a (mon.0) 2422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:21.691 INFO:tasks.workunit.client.0.smithi115.stdout: 388: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:21.692 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-scRlbZWwddVYfdm off=4780978 len=726085 2022-04-22T17:53:21.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 1981 completed, throughput=4.99MB/sec 2022-04-22T17:53:21.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-inYU3zBAZ8UVaE6 off=12217629 len=1045651 2022-04-22T17:53:21.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 1982 completed, throughput=4.99MB/sec 2022-04-22T17:53:21.701 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UQNlVo9EXn8or8S off=6047508 len=1046091 2022-04-22T17:53:21.703 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:21 smithi115 bash[45495]: audit 2022-04-22T17:53:20.337437+0000 mon.a (mon.0) 2423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:21.703 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:21 smithi115 bash[46934]: audit 2022-04-22T17:53:20.337437+0000 mon.a (mon.0) 2423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:21.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 1983 completed, throughput=4.99MB/sec 2022-04-22T17:53:21.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8vzxa3u_3gmmm93 off=20402264 len=1366960 2022-04-22T17:53:21.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 1984 completed, throughput=5MB/sec 2022-04-22T17:53:21.715 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n5Acg3e-TzGrJK0 off=106049 len=1188668 2022-04-22T17:53:21.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 1985 completed, throughput=5MB/sec 2022-04-22T17:53:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:21 smithi121 bash[36902]: audit 2022-04-22T17:53:20.337437+0000 mon.a (mon.0) 2423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:21.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:22.721 INFO:tasks.workunit.client.0.smithi115.stdout: 389: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:22.721 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tgjA0zD71YWUhMZ off=3741476 len=1073765 2022-04-22T17:53:22.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 1986 completed, throughput=4.99MB/sec 2022-04-22T17:53:22.729 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q-ylzNfxBQW1sNI off=17045615 len=332803 2022-04-22T17:53:22.730 INFO:tasks.workunit.client.0.smithi115.stdout:op 1987 completed, throughput=4.99MB/sec 2022-04-22T17:53:22.730 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1HOQ3b2COiEkO2A off=5105854 len=1149647 2022-04-22T17:53:22.735 INFO:tasks.workunit.client.0.smithi115.stdout:op 1988 completed, throughput=4.99MB/sec 2022-04-22T17:53:22.736 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3Ur8eIT6aedu_Qa off=4525190 len=1798536 2022-04-22T17:53:22.745 INFO:tasks.workunit.client.0.smithi115.stdout:op 1989 completed, throughput=5MB/sec 2022-04-22T17:53:22.745 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jF3V7V9yEYivXup off=3875436 len=1368740 2022-04-22T17:53:22.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 1990 completed, throughput=5MB/sec 2022-04-22T17:53:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:22 smithi121 bash[36902]: cluster 2022-04-22T17:53:20.452679+0000 mgr.y (mgr.34107) 983 : cluster [DBG] pgmap v1599: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:22 smithi115 bash[45495]: cluster 2022-04-22T17:53:20.452679+0000 mgr.y (mgr.34107) 983 : cluster [DBG] pgmap v1599: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:22 smithi115 bash[46934]: cluster 2022-04-22T17:53:20.452679+0000 mgr.y (mgr.34107) 983 : cluster [DBG] pgmap v1599: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:23.584 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:23 smithi115 bash[45495]: audit 2022-04-22T17:53:22.393428+0000 mon.a (mon.0) 2424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:23.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:23 smithi115 bash[46934]: audit 2022-04-22T17:53:22.393428+0000 mon.a (mon.0) 2424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:23.747 INFO:tasks.workunit.client.0.smithi115.stdout: 390: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:23.748 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aAYwvPERC4bACYC off=181795 len=656238 2022-04-22T17:53:23.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 1991 completed, throughput=4.99MB/sec 2022-04-22T17:53:23.750 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5HApOhturdQfU47 off=13957561 len=460302 2022-04-22T17:53:23.751 INFO:tasks.workunit.client.0.smithi115.stdout:op 1992 completed, throughput=4.99MB/sec 2022-04-22T17:53:23.751 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y1UZGqNWDk1mt2w off=16613787 len=949563 2022-04-22T17:53:23.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 1993 completed, throughput=4.99MB/sec 2022-04-22T17:53:23.759 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0gfscLMhowsHmhj off=15447870 len=2089397 2022-04-22T17:53:23.765 INFO:tasks.workunit.client.0.smithi115.stdout:op 1994 completed, throughput=5MB/sec 2022-04-22T17:53:23.765 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qMYAiaX5zkiYpl4 off=2058055 len=1296366 2022-04-22T17:53:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:23 smithi121 bash[36902]: audit 2022-04-22T17:53:22.393428+0000 mon.a (mon.0) 2424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:23.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 1995 completed, throughput=5MB/sec 2022-04-22T17:53:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:23.576Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:24 smithi121 bash[36902]: cluster 2022-04-22T17:53:22.454074+0000 mgr.y (mgr.34107) 984 : cluster [DBG] pgmap v1600: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:24.774 INFO:tasks.workunit.client.0.smithi115.stdout: 391: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:24.774 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R9KBKG7eSFw-jtO off=1734557 len=1293865 2022-04-22T17:53:24.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 1996 completed, throughput=4.99MB/sec 2022-04-22T17:53:24.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9ej7wHm8qBkWyhE off=4417965 len=782105 2022-04-22T17:53:24.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 1997 completed, throughput=4.99MB/sec 2022-04-22T17:53:24.784 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-37aGrdco7sKYasi off=838341 len=680452 2022-04-22T17:53:24.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 1998 completed, throughput=5MB/sec 2022-04-22T17:53:24.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ArgpYbwvkl-vMwm off=3618976 len=1182220 2022-04-22T17:53:24.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 1999 completed, throughput=5MB/sec 2022-04-22T17:53:24.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Rxhl0L3YHDJLddr off=5153324 len=805025 2022-04-22T17:53:24.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 2000 completed, throughput=5MB/sec 2022-04-22T17:53:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:24 smithi115 bash[45495]: cluster 2022-04-22T17:53:22.454074+0000 mgr.y (mgr.34107) 984 : cluster [DBG] pgmap v1600: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:24 smithi115 bash[46934]: cluster 2022-04-22T17:53:22.454074+0000 mgr.y (mgr.34107) 984 : cluster [DBG] pgmap v1600: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:53:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:25 smithi121 bash[36902]: audit 2022-04-22T17:53:25.016611+0000 mon.a (mon.0) 2425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:25.794 INFO:tasks.workunit.client.0.smithi115.stdout: 392: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:25.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mawT0cWGUuOao4k off=16024912 len=671934 2022-04-22T17:53:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:25 smithi115 bash[45495]: audit 2022-04-22T17:53:25.016611+0000 mon.a (mon.0) 2425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:25 smithi115 bash[46934]: audit 2022-04-22T17:53:25.016611+0000 mon.a (mon.0) 2425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:25.797 INFO:tasks.workunit.client.0.smithi115.stdout:op 2001 completed, throughput=4.99MB/sec 2022-04-22T17:53:25.797 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mV-yOV5HSvRtyhs off=4981416 len=2016400 2022-04-22T17:53:25.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 2002 completed, throughput=4.99MB/sec 2022-04-22T17:53:25.801 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fxmpohmne6I2bdb off=10334192 len=1588139 2022-04-22T17:53:25.803 INFO:tasks.workunit.client.0.smithi115.stdout:op 2003 completed, throughput=5MB/sec 2022-04-22T17:53:25.803 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RVui680bSD2toTr off=1766119 len=1018682 2022-04-22T17:53:25.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 2004 completed, throughput=5MB/sec 2022-04-22T17:53:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:26 smithi121 bash[36902]: cluster 2022-04-22T17:53:24.455198+0000 mgr.y (mgr.34107) 985 : cluster [DBG] pgmap v1601: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:26 smithi121 bash[36902]: audit 2022-04-22T17:53:25.340757+0000 mon.a (mon.0) 2426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:26 smithi115 bash[45495]: cluster 2022-04-22T17:53:24.455198+0000 mgr.y (mgr.34107) 985 : cluster [DBG] pgmap v1601: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:26 smithi115 bash[45495]: audit 2022-04-22T17:53:25.340757+0000 mon.a (mon.0) 2426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:26.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:26 smithi115 bash[46934]: cluster 2022-04-22T17:53:24.455198+0000 mgr.y (mgr.34107) 985 : cluster [DBG] pgmap v1601: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:26.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:26 smithi115 bash[46934]: audit 2022-04-22T17:53:25.340757+0000 mon.a (mon.0) 2426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:26.806 INFO:tasks.workunit.client.0.smithi115.stdout: 393: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:26.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OjAEvkaXw-p7Nmj off=17761791 len=145245 2022-04-22T17:53:26.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 2005 completed, throughput=4.99MB/sec 2022-04-22T17:53:26.808 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rZL1faZGAwW6yCI off=19019188 len=1189904 2022-04-22T17:53:26.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 2006 completed, throughput=4.99MB/sec 2022-04-22T17:53:26.812 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1mNq64LdJv4Yp24 off=11220225 len=1084785 2022-04-22T17:53:26.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 2007 completed, throughput=4.99MB/sec 2022-04-22T17:53:26.820 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yFTuH2IaytscisX off=1827647 len=2034673 2022-04-22T17:53:26.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 2008 completed, throughput=5MB/sec 2022-04-22T17:53:26.824 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zxnu5d54eu9nb3e off=2453552 len=1641093 2022-04-22T17:53:26.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 2009 completed, throughput=5MB/sec 2022-04-22T17:53:27.826 INFO:tasks.workunit.client.0.smithi115.stdout: 394: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:27.826 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X7lUQz0OQtEVxaO off=4543053 len=1467341 2022-04-22T17:53:27.831 INFO:tasks.workunit.client.0.smithi115.stdout:op 2010 completed, throughput=4.99MB/sec 2022-04-22T17:53:27.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Gy0Nikg7mjfDOMC off=3589549 len=653514 2022-04-22T17:53:27.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 2011 completed, throughput=4.99MB/sec 2022-04-22T17:53:27.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WZBQqYLU6kZFpRb off=18810448 len=1625984 2022-04-22T17:53:27.847 INFO:tasks.workunit.client.0.smithi115.stdout:op 2012 completed, throughput=5MB/sec 2022-04-22T17:53:27.847 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AcK1iAUMajroII7 off=9363720 len=274464 2022-04-22T17:53:27.847 INFO:tasks.workunit.client.0.smithi115.stdout:op 2013 completed, throughput=5MB/sec 2022-04-22T17:53:27.848 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5YXAY-eOULNSNBT off=437717 len=176846 2022-04-22T17:53:27.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 2014 completed, throughput=5MB/sec 2022-04-22T17:53:27.851 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zwL9Xi8MlnSarCE off=2388575 len=409589 2022-04-22T17:53:27.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 2015 completed, throughput=5MB/sec 2022-04-22T17:53:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:28 smithi121 bash[36902]: cluster 2022-04-22T17:53:26.456280+0000 mgr.y (mgr.34107) 986 : cluster [DBG] pgmap v1602: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:28 smithi121 bash[36902]: audit 2022-04-22T17:53:27.396454+0000 mon.a (mon.0) 2427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:28 smithi115 bash[45495]: cluster 2022-04-22T17:53:26.456280+0000 mgr.y (mgr.34107) 986 : cluster [DBG] pgmap v1602: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:28 smithi115 bash[45495]: audit 2022-04-22T17:53:27.396454+0000 mon.a (mon.0) 2427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:28.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:28 smithi115 bash[46934]: cluster 2022-04-22T17:53:26.456280+0000 mgr.y (mgr.34107) 986 : cluster [DBG] pgmap v1602: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:53:28.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:28 smithi115 bash[46934]: audit 2022-04-22T17:53:27.396454+0000 mon.a (mon.0) 2427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:28.853 INFO:tasks.workunit.client.0.smithi115.stdout: 395: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:28.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oEHJEX4rBGZdQnj off=1202648 len=105241 2022-04-22T17:53:28.858 INFO:tasks.workunit.client.0.smithi115.stdout:op 2016 completed, throughput=4.99MB/sec 2022-04-22T17:53:28.858 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CYn8vuDrXaXhILB off=2925419 len=140677 2022-04-22T17:53:28.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 2017 completed, throughput=4.99MB/sec 2022-04-22T17:53:28.862 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BO9zXhz7GvKZxYm off=4228374 len=842737 2022-04-22T17:53:28.864 INFO:tasks.workunit.client.0.smithi115.stdout:op 2018 completed, throughput=4.99MB/sec 2022-04-22T17:53:28.864 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DzHIGWSZYgwvdlW off=3164337 len=1966735 2022-04-22T17:53:28.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 2019 completed, throughput=4.99MB/sec 2022-04-22T17:53:28.874 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hnhAqd1Cf0XTe8x off=9147546 len=879031 2022-04-22T17:53:28.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 2020 completed, throughput=5MB/sec 2022-04-22T17:53:28.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gNLy3GFkhkylnOy off=10600345 len=176327 2022-04-22T17:53:28.880 INFO:tasks.workunit.client.0.smithi115.stdout:op 2021 completed, throughput=5MB/sec 2022-04-22T17:53:28.880 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2Cxnu_KtdL5fqHH off=18709788 len=1901755 2022-04-22T17:53:28.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 2022 completed, throughput=5MB/sec 2022-04-22T17:53:29.890 INFO:tasks.workunit.client.0.smithi115.stdout: 396: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:29.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F0nM6rkFVXffyR7 off=1017841 len=525854 2022-04-22T17:53:29.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 2023 completed, throughput=4.99MB/sec 2022-04-22T17:53:29.896 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7P76hooiEeFtwep off=0 len=1418366 2022-04-22T17:53:29.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 2024 completed, throughput=4.99MB/sec 2022-04-22T17:53:29.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CFE7XXkJE9jaB73 off=6500725 len=385762 2022-04-22T17:53:29.905 INFO:tasks.workunit.client.0.smithi115.stdout:op 2025 completed, throughput=4.99MB/sec 2022-04-22T17:53:29.905 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XUBhHzCDf_Aa9cy off=3616756 len=569936 2022-04-22T17:53:29.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 2026 completed, throughput=5MB/sec 2022-04-22T17:53:29.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ColL9xIR8f0Emo4 off=2980135 len=1397063 2022-04-22T17:53:29.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 2027 completed, throughput=5MB/sec 2022-04-22T17:53:29.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6HWoPBk6spOtoHk off=2040254 len=1821592 2022-04-22T17:53:29.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 2028 completed, throughput=5MB/sec 2022-04-22T17:53:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:30 smithi121 bash[36902]: cluster 2022-04-22T17:53:28.457459+0000 mgr.y (mgr.34107) 987 : cluster [DBG] pgmap v1603: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:30 smithi121 bash[36902]: audit 2022-04-22T17:53:30.019804+0000 mon.a (mon.0) 2428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:30 smithi115 bash[45495]: cluster 2022-04-22T17:53:28.457459+0000 mgr.y (mgr.34107) 987 : cluster [DBG] pgmap v1603: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:30 smithi115 bash[45495]: audit 2022-04-22T17:53:30.019804+0000 mon.a (mon.0) 2428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:30 smithi115 bash[46934]: cluster 2022-04-22T17:53:28.457459+0000 mgr.y (mgr.34107) 987 : cluster [DBG] pgmap v1603: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:30 smithi115 bash[46934]: audit 2022-04-22T17:53:30.019804+0000 mon.a (mon.0) 2428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:30.912 INFO:tasks.workunit.client.0.smithi115.stdout: 397: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:30.912 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LR5A4Y7aNY8fD_p off=14215484 len=236932 2022-04-22T17:53:30.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 2029 completed, throughput=4.99MB/sec 2022-04-22T17:53:30.917 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-T33Agmws6BP4vHx off=6074077 len=2051180 2022-04-22T17:53:30.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 2030 completed, throughput=5MB/sec 2022-04-22T17:53:30.923 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sibXI1YSVTG1r2M off=1131740 len=67669 2022-04-22T17:53:30.924 INFO:tasks.workunit.client.0.smithi115.stdout:op 2031 completed, throughput=5MB/sec 2022-04-22T17:53:30.924 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1DbcZB95w6Q6Pq9 off=934559 len=1071034 2022-04-22T17:53:30.930 INFO:tasks.workunit.client.0.smithi115.stdout:op 2032 completed, throughput=5MB/sec 2022-04-22T17:53:30.931 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cEr8bWQ1B-pY8hg off=11032602 len=1446518 2022-04-22T17:53:30.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 2033 completed, throughput=5MB/sec 2022-04-22T17:53:31.703 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:31 smithi115 bash[46934]: audit 2022-04-22T17:53:30.348821+0000 mon.a (mon.0) 2429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:31.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:31 smithi115 bash[45495]: audit 2022-04-22T17:53:30.348821+0000 mon.a (mon.0) 2429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:31 smithi121 bash[36902]: audit 2022-04-22T17:53:30.348821+0000 mon.a (mon.0) 2429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:31.934 INFO:tasks.workunit.client.0.smithi115.stdout: 398: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:31.950 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RkhtEpWRla8tp8Y off=8438517 len=1906086 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 2034 completed, throughput=4.99MB/sec 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JbYgnBPuD6pm7Mw off=3266654 len=1215566 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 2035 completed, throughput=5MB/sec 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5JnGcR3OW_vBj0v off=1296932 len=782563 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 2036 completed, throughput=5MB/sec 2022-04-22T17:53:31.952 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-foSKMGFRnH5mdaM off=2580056 len=1206296 2022-04-22T17:53:31.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 2037 completed, throughput=5MB/sec 2022-04-22T17:53:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:31.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:32 smithi121 bash[36902]: cluster 2022-04-22T17:53:30.458506+0000 mgr.y (mgr.34107) 988 : cluster [DBG] pgmap v1604: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:32 smithi115 bash[45495]: cluster 2022-04-22T17:53:30.458506+0000 mgr.y (mgr.34107) 988 : cluster [DBG] pgmap v1604: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:32.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:32 smithi115 bash[46934]: cluster 2022-04-22T17:53:30.458506+0000 mgr.y (mgr.34107) 988 : cluster [DBG] pgmap v1604: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:32.953 INFO:tasks.workunit.client.0.smithi115.stdout: 399: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:32.964 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fq7TuuohEStDEJ1 off=9885702 len=559480 2022-04-22T17:53:32.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 2038 completed, throughput=4.99MB/sec 2022-04-22T17:53:32.965 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WYhloVARWRlUAXT off=21535965 len=116193 2022-04-22T17:53:32.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 2039 completed, throughput=4.99MB/sec 2022-04-22T17:53:32.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FAgZcdh3B58kVcR off=16943225 len=1334640 2022-04-22T17:53:32.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 2040 completed, throughput=4.99MB/sec 2022-04-22T17:53:32.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8Rbxh-BNbyhiYKY off=11621020 len=1951157 2022-04-22T17:53:32.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 2041 completed, throughput=5MB/sec 2022-04-22T17:53:32.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-poXrtg2oZgKNY2X off=12447817 len=227224 2022-04-22T17:53:32.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 2042 completed, throughput=5MB/sec 2022-04-22T17:53:32.968 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hfRDY95huKZtSAY off=8065666 len=1914676 2022-04-22T17:53:32.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 2043 completed, throughput=5MB/sec 2022-04-22T17:53:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:33 smithi121 bash[36902]: audit 2022-04-22T17:53:32.399548+0000 mon.a (mon.0) 2430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:33 smithi121 bash[36902]: cluster 2022-04-22T17:53:32.459626+0000 mgr.y (mgr.34107) 989 : cluster [DBG] pgmap v1605: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:53:33.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:33.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:33.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:33 smithi115 bash[45495]: audit 2022-04-22T17:53:32.399548+0000 mon.a (mon.0) 2430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:33 smithi115 bash[45495]: cluster 2022-04-22T17:53:32.459626+0000 mgr.y (mgr.34107) 989 : cluster [DBG] pgmap v1605: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:53:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:33 smithi115 bash[46934]: audit 2022-04-22T17:53:32.399548+0000 mon.a (mon.0) 2430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:33 smithi115 bash[46934]: cluster 2022-04-22T17:53:32.459626+0000 mgr.y (mgr.34107) 989 : cluster [DBG] pgmap v1605: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:53:33.970 INFO:tasks.workunit.client.0.smithi115.stdout: 400: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:33.970 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3OGM0Zh1Vyzj7EV off=8290852 len=1085021 2022-04-22T17:53:33.977 INFO:tasks.workunit.client.0.smithi115.stdout:op 2044 completed, throughput=4.99MB/sec 2022-04-22T17:53:33.977 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tT9L9BPujh4kt5g off=6712544 len=684396 2022-04-22T17:53:33.983 INFO:tasks.workunit.client.0.smithi115.stdout:op 2045 completed, throughput=5MB/sec 2022-04-22T17:53:33.984 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kSlw2ABjQMTmehM off=8442886 len=1655278 2022-04-22T17:53:33.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 2046 completed, throughput=5MB/sec 2022-04-22T17:53:33.987 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CH9VHsU2mnKLuvZ off=4860820 len=224048 2022-04-22T17:53:33.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 2047 completed, throughput=5MB/sec 2022-04-22T17:53:33.991 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XZIcDJ5-gkkkfNB off=1851960 len=518412 2022-04-22T17:53:33.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 2048 completed, throughput=5MB/sec 2022-04-22T17:53:34.997 INFO:tasks.workunit.client.0.smithi115.stdout: 401: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:34.997 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PKFv0QiiVKXAftO off=62787 len=166730 2022-04-22T17:53:34.997 INFO:tasks.workunit.client.0.smithi115.stdout:op 2049 completed, throughput=4.99MB/sec 2022-04-22T17:53:34.997 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tvc7kg6CyqdmCRx off=3138658 len=2093351 2022-04-22T17:53:34.999 INFO:tasks.workunit.client.0.smithi115.stdout:op 2050 completed, throughput=4.99MB/sec 2022-04-22T17:53:35.000 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7q6N0n7kB-0Pcsk off=11413346 len=983636 2022-04-22T17:53:35.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 2051 completed, throughput=5MB/sec 2022-04-22T17:53:35.003 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IdlURgJsPdt9OF7 off=13805417 len=243663 2022-04-22T17:53:35.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 2052 completed, throughput=5MB/sec 2022-04-22T17:53:35.004 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GFUmyrUziUcx3-D off=10822791 len=694633 2022-04-22T17:53:35.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 2053 completed, throughput=5MB/sec 2022-04-22T17:53:35.006 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LoN98Sn6mxTnb0E off=11261792 len=209525 2022-04-22T17:53:35.011 INFO:tasks.workunit.client.0.smithi115.stdout:op 2054 completed, throughput=5MB/sec 2022-04-22T17:53:35.011 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hylOPOz8pIAInG9 off=603539 len=722496 2022-04-22T17:53:35.012 INFO:tasks.workunit.client.0.smithi115.stdout:op 2055 completed, throughput=5MB/sec 2022-04-22T17:53:35.012 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Up1f7gmJN8msU5G off=4630173 len=1593804 2022-04-22T17:53:35.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 2056 completed, throughput=5MB/sec 2022-04-22T17:53:35.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[45495]: cluster 2022-04-22T17:53:34.460598+0000 mgr.y (mgr.34107) 990 : cluster [DBG] pgmap v1606: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:35.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[45495]: audit 2022-04-22T17:53:35.022605+0000 mon.a (mon.0) 2431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:35.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[45495]: audit 2022-04-22T17:53:35.348698+0000 mon.a (mon.0) 2432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:35.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[46934]: cluster 2022-04-22T17:53:34.460598+0000 mgr.y (mgr.34107) 990 : cluster [DBG] pgmap v1606: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:35.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[46934]: audit 2022-04-22T17:53:35.022605+0000 mon.a (mon.0) 2431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:35.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:35 smithi115 bash[46934]: audit 2022-04-22T17:53:35.348698+0000 mon.a (mon.0) 2432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:35 smithi121 bash[36902]: cluster 2022-04-22T17:53:34.460598+0000 mgr.y (mgr.34107) 990 : cluster [DBG] pgmap v1606: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:35 smithi121 bash[36902]: audit 2022-04-22T17:53:35.022605+0000 mon.a (mon.0) 2431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:35 smithi121 bash[36902]: audit 2022-04-22T17:53:35.348698+0000 mon.a (mon.0) 2432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:36.019 INFO:tasks.workunit.client.0.smithi115.stdout: 402: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:36.019 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y8DzZYf9dVB_ugK off=1575315 len=1086214 2022-04-22T17:53:36.028 INFO:tasks.workunit.client.0.smithi115.stdout:op 2057 completed, throughput=4.99MB/sec 2022-04-22T17:53:36.028 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h6QzLri-JiyJSc5 off=3529898 len=527373 2022-04-22T17:53:36.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 2058 completed, throughput=4.99MB/sec 2022-04-22T17:53:36.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QPzzZpxAzgCnTM7 off=5539372 len=2096926 2022-04-22T17:53:36.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 2059 completed, throughput=5MB/sec 2022-04-22T17:53:36.045 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qs2iAKZ5bVH42rl off=1260387 len=1021362 2022-04-22T17:53:36.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 2060 completed, throughput=5MB/sec 2022-04-22T17:53:36.387 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (16m) 5m ago 30m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (30m) 7m ago 30m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (18m) 5m ago 31m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (18m) 7m ago 31m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (25m) 7m ago 35m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (26m) 5m ago 37m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:53:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (25m) 5m ago 38m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (24m) 7m ago 36m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (25m) 5m ago 36m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (17m) 5m ago 30m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (17m) 7m ago 30m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (23m) 5m ago 35m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (22m) 5m ago 34m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (22m) 5m ago 34m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (21m) 5m ago 33m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (21m) 7m ago 33m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:53:36.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (20m) 7m ago 32m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:53:36.740 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (19m) 7m ago 32m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:53:36.740 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (19m) 7m ago 32m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:53:36.740 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (17m) 7m ago 30m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:53:37.050 INFO:tasks.workunit.client.0.smithi115.stdout: 403: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:37.050 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-44Ll0kXZF3Pe1R1 off=4588543 len=1835700 2022-04-22T17:53:37.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 2061 completed, throughput=4.99MB/sec 2022-04-22T17:53:37.060 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RTnhKzfeEPtd6vW off=12053385 len=1079443 2022-04-22T17:53:37.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 2062 completed, throughput=5MB/sec 2022-04-22T17:53:37.062 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p9MnCcdNP9tXuvd off=5508182 len=1919709 2022-04-22T17:53:37.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 2063 completed, throughput=5MB/sec 2022-04-22T17:53:37.137 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:53:37.137 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:53:37.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:53:37.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:53:37.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:53:37.139 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:53:37.139 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:53:37.139 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:53:37.139 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:53:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:37 smithi121 bash[36902]: audit 2022-04-22T17:53:36.364917+0000 mgr.y (mgr.34107) 991 : audit [DBG] from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:37 smithi121 bash[36902]: cluster 2022-04-22T17:53:36.461562+0000 mgr.y (mgr.34107) 992 : cluster [DBG] pgmap v1607: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:37 smithi121 bash[36902]: audit 2022-04-22T17:53:36.709141+0000 mgr.y (mgr.34107) 993 : audit [DBG] from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:37 smithi121 bash[36902]: audit 2022-04-22T17:53:37.129591+0000 mon.a (mon.0) 2433 : audit [DBG] from='client.? 172.21.15.115:0/935161981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:37 smithi121 bash[36902]: audit 2022-04-22T17:53:37.402581+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[45495]: audit 2022-04-22T17:53:36.364917+0000 mgr.y (mgr.34107) 991 : audit [DBG] from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[45495]: cluster 2022-04-22T17:53:36.461562+0000 mgr.y (mgr.34107) 992 : cluster [DBG] pgmap v1607: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[45495]: audit 2022-04-22T17:53:36.709141+0000 mgr.y (mgr.34107) 993 : audit [DBG] from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[45495]: audit 2022-04-22T17:53:37.129591+0000 mon.a (mon.0) 2433 : audit [DBG] from='client.? 172.21.15.115:0/935161981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[45495]: audit 2022-04-22T17:53:37.402581+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[46934]: audit 2022-04-22T17:53:36.364917+0000 mgr.y (mgr.34107) 991 : audit [DBG] from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[46934]: cluster 2022-04-22T17:53:36.461562+0000 mgr.y (mgr.34107) 992 : cluster [DBG] pgmap v1607: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[46934]: audit 2022-04-22T17:53:36.709141+0000 mgr.y (mgr.34107) 993 : audit [DBG] from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:53:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[46934]: audit 2022-04-22T17:53:37.129591+0000 mon.a (mon.0) 2433 : audit [DBG] from='client.? 172.21.15.115:0/935161981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:53:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:37 smithi115 bash[46934]: audit 2022-04-22T17:53:37.402581+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:38.068 INFO:tasks.workunit.client.0.smithi115.stdout: 404: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:38.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sl2_EiZ7mqsfe6i off=16978595 len=1956281 2022-04-22T17:53:38.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 2064 completed, throughput=4.99MB/sec 2022-04-22T17:53:38.073 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BdRyVT4WlID6uz1 off=2827753 len=658376 2022-04-22T17:53:38.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 2065 completed, throughput=4.99MB/sec 2022-04-22T17:53:38.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-erieZ1K_vPwrZGh off=462747 len=1018488 2022-04-22T17:53:38.082 INFO:tasks.workunit.client.0.smithi115.stdout:op 2066 completed, throughput=5MB/sec 2022-04-22T17:53:38.082 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1Pzdy0EEftXHTXY off=3223170 len=291057 2022-04-22T17:53:38.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 2067 completed, throughput=5MB/sec 2022-04-22T17:53:38.084 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EzmYWxSaXxDfB63 off=16785041 len=56893 2022-04-22T17:53:38.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 2068 completed, throughput=5MB/sec 2022-04-22T17:53:38.097 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R4L4SSfdeiSyH50 off=8166683 len=1381063 2022-04-22T17:53:38.098 INFO:tasks.workunit.client.0.smithi115.stdout:op 2069 completed, throughput=5MB/sec 2022-04-22T17:53:39.087 INFO:tasks.workunit.client.0.smithi115.stdout: 405: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:39.087 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QDw1rgN6RW_xiaK off=2174487 len=2068743 2022-04-22T17:53:39.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 2070 completed, throughput=4.99MB/sec 2022-04-22T17:53:39.091 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aub66Jmmr0La95e off=1858850 len=1018735 2022-04-22T17:53:39.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 2071 completed, throughput=5MB/sec 2022-04-22T17:53:39.094 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NpOPek_GVuMUPcV off=1647288 len=1776258 2022-04-22T17:53:39.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 2072 completed, throughput=5MB/sec 2022-04-22T17:53:39.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eMWugvkQKCVkHen off=9256442 len=1174491 2022-04-22T17:53:39.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 2073 completed, throughput=5MB/sec 2022-04-22T17:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:39 smithi121 bash[36902]: cluster 2022-04-22T17:53:38.462809+0000 mgr.y (mgr.34107) 994 : cluster [DBG] pgmap v1608: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:39 smithi121 bash[36902]: audit 2022-04-22T17:53:39.128763+0000 mon.a (mon.0) 2435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:39 smithi121 bash[36902]: audit 2022-04-22T17:53:39.131154+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:39 smithi121 bash[36902]: audit 2022-04-22T17:53:39.132703+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:39 smithi121 bash[36902]: audit 2022-04-22T17:53:39.492744+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[46934]: cluster 2022-04-22T17:53:38.462809+0000 mgr.y (mgr.34107) 994 : cluster [DBG] pgmap v1608: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[46934]: audit 2022-04-22T17:53:39.128763+0000 mon.a (mon.0) 2435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[46934]: audit 2022-04-22T17:53:39.131154+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[46934]: audit 2022-04-22T17:53:39.132703+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[46934]: audit 2022-04-22T17:53:39.492744+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[45495]: cluster 2022-04-22T17:53:38.462809+0000 mgr.y (mgr.34107) 994 : cluster [DBG] pgmap v1608: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[45495]: audit 2022-04-22T17:53:39.128763+0000 mon.a (mon.0) 2435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[45495]: audit 2022-04-22T17:53:39.131154+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[45495]: audit 2022-04-22T17:53:39.132703+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:39 smithi115 bash[45495]: audit 2022-04-22T17:53:39.492744+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:53:40.104 INFO:tasks.workunit.client.0.smithi115.stdout: 406: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:40.104 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5cOMW_FLIBjk_Ex off=10508048 len=846873 2022-04-22T17:53:40.111 INFO:tasks.workunit.client.0.smithi115.stdout:op 2074 completed, throughput=4.99MB/sec 2022-04-22T17:53:40.111 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_pQ6KQm1n6eCqps off=1012023 len=773066 2022-04-22T17:53:40.113 INFO:tasks.workunit.client.0.smithi115.stdout:op 2075 completed, throughput=4.99MB/sec 2022-04-22T17:53:40.113 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u_H0kmg1LIw9nQx off=146129 len=981783 2022-04-22T17:53:40.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 2076 completed, throughput=5MB/sec 2022-04-22T17:53:40.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nYQFKN3fbwrUARx off=10779491 len=2009179 2022-04-22T17:53:40.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 2077 completed, throughput=5MB/sec 2022-04-22T17:53:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:40 smithi121 bash[36902]: audit 2022-04-22T17:53:40.025130+0000 mon.a (mon.0) 2439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:40 smithi121 bash[36902]: audit 2022-04-22T17:53:40.352301+0000 mon.a (mon.0) 2440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:40 smithi115 bash[45495]: audit 2022-04-22T17:53:40.025130+0000 mon.a (mon.0) 2439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:40 smithi115 bash[45495]: audit 2022-04-22T17:53:40.352301+0000 mon.a (mon.0) 2440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:40 smithi115 bash[46934]: audit 2022-04-22T17:53:40.025130+0000 mon.a (mon.0) 2439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:40 smithi115 bash[46934]: audit 2022-04-22T17:53:40.352301+0000 mon.a (mon.0) 2440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:41.130 INFO:tasks.workunit.client.0.smithi115.stdout: 407: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:41.130 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oW4VO5uN48wC5Tm off=946871 len=169034 2022-04-22T17:53:41.132 INFO:tasks.workunit.client.0.smithi115.stdout:op 2078 completed, throughput=4.99MB/sec 2022-04-22T17:53:41.132 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C2WGJmXo6vMYu5z off=196046 len=1434609 2022-04-22T17:53:41.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 2079 completed, throughput=4.99MB/sec 2022-04-22T17:53:41.136 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-domoWVd5c6yCAIX off=9400243 len=776818 2022-04-22T17:53:41.138 INFO:tasks.workunit.client.0.smithi115.stdout:op 2080 completed, throughput=4.99MB/sec 2022-04-22T17:53:41.139 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wFi4YbX4vild_8f off=16454960 len=137122 2022-04-22T17:53:41.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2081 completed, throughput=4.99MB/sec 2022-04-22T17:53:41.143 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H3cbK8quEnHEp51 off=5180970 len=54990 2022-04-22T17:53:41.147 INFO:tasks.workunit.client.0.smithi115.stdout:op 2082 completed, throughput=4.99MB/sec 2022-04-22T17:53:41.147 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JNOgluCbc7KCFZT off=11594142 len=652062 2022-04-22T17:53:41.149 INFO:tasks.workunit.client.0.smithi115.stdout:op 2083 completed, throughput=5MB/sec 2022-04-22T17:53:41.149 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--2LawLv3RuNRP9F off=19385272 len=40304 2022-04-22T17:53:41.152 INFO:tasks.workunit.client.0.smithi115.stdout:op 2084 completed, throughput=5MB/sec 2022-04-22T17:53:41.152 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fMzN0rStHVL-mCa off=7812358 len=1172521 2022-04-22T17:53:41.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 2085 completed, throughput=5MB/sec 2022-04-22T17:53:41.155 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PbFETZ3-nUxTevK off=707448 len=128306 2022-04-22T17:53:41.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 2086 completed, throughput=5MB/sec 2022-04-22T17:53:41.155 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WyR3yF5rc5FgAsB off=6033690 len=93081 2022-04-22T17:53:41.161 INFO:tasks.workunit.client.0.smithi115.stdout:op 2087 completed, throughput=5MB/sec 2022-04-22T17:53:41.161 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hl8gCN4Me6XKx57 off=6809910 len=1234575 2022-04-22T17:53:41.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 2088 completed, throughput=5MB/sec 2022-04-22T17:53:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:41 smithi121 bash[36902]: cluster 2022-04-22T17:53:40.463961+0000 mgr.y (mgr.34107) 995 : cluster [DBG] pgmap v1609: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:41 smithi115 bash[45495]: cluster 2022-04-22T17:53:40.463961+0000 mgr.y (mgr.34107) 995 : cluster [DBG] pgmap v1609: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:41 smithi115 bash[46934]: cluster 2022-04-22T17:53:40.463961+0000 mgr.y (mgr.34107) 995 : cluster [DBG] pgmap v1609: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:53:42.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:41.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:42.162 INFO:tasks.workunit.client.0.smithi115.stdout: 408: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:42.163 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jrS4WaEvT0Jz17Y off=144614 len=1947572 2022-04-22T17:53:42.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 2089 completed, throughput=4.99MB/sec 2022-04-22T17:53:42.173 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sIUFfhy6r6mXBqM off=702181 len=743954 2022-04-22T17:53:42.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 2090 completed, throughput=5MB/sec 2022-04-22T17:53:42.178 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mG4p1JSUd4vMf3b off=1160563 len=625687 2022-04-22T17:53:42.180 INFO:tasks.workunit.client.0.smithi115.stdout:op 2091 completed, throughput=5MB/sec 2022-04-22T17:53:42.180 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LhePq1QDk2A-48h off=874195 len=2037420 2022-04-22T17:53:42.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 2092 completed, throughput=5MB/sec 2022-04-22T17:53:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:42 smithi121 bash[36902]: audit 2022-04-22T17:53:42.405670+0000 mon.a (mon.0) 2441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:42 smithi115 bash[45495]: audit 2022-04-22T17:53:42.405670+0000 mon.a (mon.0) 2441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:42 smithi115 bash[46934]: audit 2022-04-22T17:53:42.405670+0000 mon.a (mon.0) 2441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:43.184 INFO:tasks.workunit.client.0.smithi115.stdout: 409: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:43.184 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZPPlOZhxLHXQgqZ off=4714341 len=1598798 2022-04-22T17:53:43.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 2093 completed, throughput=4.99MB/sec 2022-04-22T17:53:43.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZI1LKuK_p72SSuo off=8468741 len=1505831 2022-04-22T17:53:43.201 INFO:tasks.workunit.client.0.smithi115.stdout:op 2094 completed, throughput=5MB/sec 2022-04-22T17:53:43.201 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0L0LyXroY_nwRMA off=11136021 len=392888 2022-04-22T17:53:43.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 2095 completed, throughput=5MB/sec 2022-04-22T17:53:43.203 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3sZ1DrcvU3LO6lL off=1577193 len=1023048 2022-04-22T17:53:43.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 2096 completed, throughput=5MB/sec 2022-04-22T17:53:43.209 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l7DWIlCzMDwsaFA off=7627052 len=531558 2022-04-22T17:53:43.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 2097 completed, throughput=5MB/sec 2022-04-22T17:53:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:43 smithi121 bash[36902]: cluster 2022-04-22T17:53:42.465054+0000 mgr.y (mgr.34107) 996 : cluster [DBG] pgmap v1610: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:53:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:43 smithi115 bash[45495]: cluster 2022-04-22T17:53:42.465054+0000 mgr.y (mgr.34107) 996 : cluster [DBG] pgmap v1610: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:53:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:43 smithi115 bash[46934]: cluster 2022-04-22T17:53:42.465054+0000 mgr.y (mgr.34107) 996 : cluster [DBG] pgmap v1610: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:53:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:43.579Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:44.212 INFO:tasks.workunit.client.0.smithi115.stdout: 410: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:44.213 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iYa1uhkhWRGN3mA off=5627436 len=813447 2022-04-22T17:53:44.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 2098 completed, throughput=4.99MB/sec 2022-04-22T17:53:44.216 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U_HM809QO8WU0hN off=2767790 len=1822356 2022-04-22T17:53:44.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 2099 completed, throughput=5MB/sec 2022-04-22T17:53:44.220 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zBqQJBlZGL4t2q- off=7853823 len=1130263 2022-04-22T17:53:44.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 2100 completed, throughput=5MB/sec 2022-04-22T17:53:44.228 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-avfnk4bMoAFg0Xb off=1085248 len=136167 2022-04-22T17:53:44.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 2101 completed, throughput=5MB/sec 2022-04-22T17:53:44.233 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2oheWKyjBIT1QFX off=10356988 len=69632 2022-04-22T17:53:44.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 2102 completed, throughput=5MB/sec 2022-04-22T17:53:44.233 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6g-_f-sEphPh68y off=8168864 len=420082 2022-04-22T17:53:44.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 2103 completed, throughput=5MB/sec 2022-04-22T17:53:44.238 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J87iD_gINaB14TH off=10357313 len=277105 2022-04-22T17:53:44.239 INFO:tasks.workunit.client.0.smithi115.stdout:op 2104 completed, throughput=5MB/sec 2022-04-22T17:53:44.239 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JFzFt2qEWhgLh1R off=21208056 len=501634 2022-04-22T17:53:44.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 2105 completed, throughput=5MB/sec 2022-04-22T17:53:45.241 INFO:tasks.workunit.client.0.smithi115.stdout: 411: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1MV-o5q2ZlugDdT off=751743 len=608824 2022-04-22T17:53:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 2106 completed, throughput=4.99MB/sec 2022-04-22T17:53:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IbHl912EivRqox1 off=8782344 len=248404 2022-04-22T17:53:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:op 2107 completed, throughput=4.99MB/sec 2022-04-22T17:53:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KAa6EFSu6D68Dww off=495462 len=782038 2022-04-22T17:53:45.255 INFO:tasks.workunit.client.0.smithi115.stdout:op 2108 completed, throughput=4.99MB/sec 2022-04-22T17:53:45.255 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FKKbrzZAdcD3yM- off=9354418 len=863340 2022-04-22T17:53:45.261 INFO:tasks.workunit.client.0.smithi115.stdout:op 2109 completed, throughput=4.99MB/sec 2022-04-22T17:53:45.262 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0or5X63V90dLg8N off=4088811 len=1174369 2022-04-22T17:53:45.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 2110 completed, throughput=5MB/sec 2022-04-22T17:53:45.265 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4V6VORNkMvaqnwn off=4362396 len=649835 2022-04-22T17:53:45.266 INFO:tasks.workunit.client.0.smithi115.stdout:op 2111 completed, throughput=5MB/sec 2022-04-22T17:53:45.266 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JD7Oc96Q_Lt2pF5 off=6787598 len=1189724 2022-04-22T17:53:45.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 2112 completed, throughput=5MB/sec 2022-04-22T17:53:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:45 smithi121 bash[36902]: cluster 2022-04-22T17:53:44.466115+0000 mgr.y (mgr.34107) 997 : cluster [DBG] pgmap v1611: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:45 smithi121 bash[36902]: audit 2022-04-22T17:53:45.028243+0000 mon.a (mon.0) 2442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:45 smithi121 bash[36902]: audit 2022-04-22T17:53:45.356057+0000 mon.a (mon.0) 2443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[46934]: cluster 2022-04-22T17:53:44.466115+0000 mgr.y (mgr.34107) 997 : cluster [DBG] pgmap v1611: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[46934]: audit 2022-04-22T17:53:45.028243+0000 mon.a (mon.0) 2442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[46934]: audit 2022-04-22T17:53:45.356057+0000 mon.a (mon.0) 2443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[45495]: cluster 2022-04-22T17:53:44.466115+0000 mgr.y (mgr.34107) 997 : cluster [DBG] pgmap v1611: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[45495]: audit 2022-04-22T17:53:45.028243+0000 mon.a (mon.0) 2442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:45 smithi115 bash[45495]: audit 2022-04-22T17:53:45.356057+0000 mon.a (mon.0) 2443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:46.272 INFO:tasks.workunit.client.0.smithi115.stdout: 412: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:46.272 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PNXUBBIlB7607Ex off=7621297 len=1525871 2022-04-22T17:53:46.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 2113 completed, throughput=4.99MB/sec 2022-04-22T17:53:46.276 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CjbhyH3WW5mmnkB off=1025916 len=1211656 2022-04-22T17:53:46.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 2114 completed, throughput=5MB/sec 2022-04-22T17:53:46.282 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2iZ4vilYPjaYf7h off=5400049 len=519547 2022-04-22T17:53:46.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 2115 completed, throughput=5MB/sec 2022-04-22T17:53:46.285 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NaKCX7wLhP8Hnxj off=14246986 len=2061311 2022-04-22T17:53:46.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 2116 completed, throughput=5MB/sec 2022-04-22T17:53:47.294 INFO:tasks.workunit.client.0.smithi115.stdout: 413: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:47.294 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SMkKRmkUAVW8b5n off=8921877 len=1273570 2022-04-22T17:53:47.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 2117 completed, throughput=4.99MB/sec 2022-04-22T17:53:47.299 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7wLMDJRmZB-jrRu off=4079966 len=381777 2022-04-22T17:53:47.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 2118 completed, throughput=4.99MB/sec 2022-04-22T17:53:47.301 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k0d7_WzAvAPAtTD off=4432944 len=1378851 2022-04-22T17:53:47.304 INFO:tasks.workunit.client.0.smithi115.stdout:op 2119 completed, throughput=5MB/sec 2022-04-22T17:53:47.305 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d7UPLG6MG7QaQBm off=13201878 len=1455039 2022-04-22T17:53:47.312 INFO:tasks.workunit.client.0.smithi115.stdout:op 2120 completed, throughput=5MB/sec 2022-04-22T17:53:47.312 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IQ-PqNJII8th_v5 off=8223429 len=861167 2022-04-22T17:53:47.318 INFO:tasks.workunit.client.0.smithi115.stdout:op 2121 completed, throughput=5MB/sec 2022-04-22T17:53:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:47 smithi121 bash[36902]: cluster 2022-04-22T17:53:46.467207+0000 mgr.y (mgr.34107) 998 : cluster [DBG] pgmap v1612: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:47 smithi121 bash[36902]: audit 2022-04-22T17:53:47.408989+0000 mon.a (mon.0) 2444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:47 smithi115 bash[45495]: cluster 2022-04-22T17:53:46.467207+0000 mgr.y (mgr.34107) 998 : cluster [DBG] pgmap v1612: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:47 smithi115 bash[45495]: audit 2022-04-22T17:53:47.408989+0000 mon.a (mon.0) 2444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:47 smithi115 bash[46934]: cluster 2022-04-22T17:53:46.467207+0000 mgr.y (mgr.34107) 998 : cluster [DBG] pgmap v1612: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:53:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:47 smithi115 bash[46934]: audit 2022-04-22T17:53:47.408989+0000 mon.a (mon.0) 2444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:48.319 INFO:tasks.workunit.client.0.smithi115.stdout: 414: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:48.319 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gq0f0EsJoxRAIGY off=943739 len=411302 2022-04-22T17:53:48.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 2122 completed, throughput=4.99MB/sec 2022-04-22T17:53:48.321 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qd92sE5lqDq3j5P off=3300880 len=1272691 2022-04-22T17:53:48.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 2123 completed, throughput=4.99MB/sec 2022-04-22T17:53:48.330 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jtS6_x9tY2BiORs off=2613964 len=1738579 2022-04-22T17:53:48.335 INFO:tasks.workunit.client.0.smithi115.stdout:op 2124 completed, throughput=5MB/sec 2022-04-22T17:53:48.356 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J3IQ2q63XLtqvIz off=402015 len=427774 2022-04-22T17:53:48.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 2125 completed, throughput=5MB/sec 2022-04-22T17:53:48.356 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LeGKVY4pl6bzLAV off=5011735 len=1594480 2022-04-22T17:53:48.357 INFO:tasks.workunit.client.0.smithi115.stdout:op 2126 completed, throughput=5MB/sec 2022-04-22T17:53:49.346 INFO:tasks.workunit.client.0.smithi115.stdout: 415: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:49.346 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qs2iAKZ5bVH42rl off=2322757 len=1027747 2022-04-22T17:53:49.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 2127 completed, throughput=4.99MB/sec 2022-04-22T17:53:49.353 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bBdBXKA3ZN4JHKs off=6736418 len=255402 2022-04-22T17:53:49.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 2128 completed, throughput=4.99MB/sec 2022-04-22T17:53:49.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-snvAbxQodiN6KOR off=1549135 len=1595976 2022-04-22T17:53:49.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 2129 completed, throughput=5MB/sec 2022-04-22T17:53:49.361 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X3VhnF1Ui7OpH_T off=1348266 len=337449 2022-04-22T17:53:49.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 2130 completed, throughput=5MB/sec 2022-04-22T17:53:49.367 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_5nMD2vm83953cp off=20126638 len=20841 2022-04-22T17:53:49.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 2131 completed, throughput=5MB/sec 2022-04-22T17:53:49.369 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-apbojOs2Mdjb_Ex off=4571145 len=1501523 2022-04-22T17:53:49.373 INFO:tasks.workunit.client.0.smithi115.stdout:op 2132 completed, throughput=5MB/sec 2022-04-22T17:53:49.373 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CYn8vuDrXaXhILB off=1984251 len=1351768 2022-04-22T17:53:49.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 2133 completed, throughput=5MB/sec 2022-04-22T17:53:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:49 smithi121 bash[36902]: cluster 2022-04-22T17:53:48.468345+0000 mgr.y (mgr.34107) 999 : cluster [DBG] pgmap v1613: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:49 smithi115 bash[45495]: cluster 2022-04-22T17:53:48.468345+0000 mgr.y (mgr.34107) 999 : cluster [DBG] pgmap v1613: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:49 smithi115 bash[46934]: cluster 2022-04-22T17:53:48.468345+0000 mgr.y (mgr.34107) 999 : cluster [DBG] pgmap v1613: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:53:50.380 INFO:tasks.workunit.client.0.smithi115.stdout: 416: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:50.380 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4OR_AA_gysDUUO- off=2673367 len=2018586 2022-04-22T17:53:50.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 2134 completed, throughput=4.99MB/sec 2022-04-22T17:53:50.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WzIRtLyF6GzMMGW off=1860969 len=254475 2022-04-22T17:53:50.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 2135 completed, throughput=5MB/sec 2022-04-22T17:53:50.392 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TWKuKGoGMtAsEfV off=3666371 len=1583234 2022-04-22T17:53:50.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 2136 completed, throughput=5MB/sec 2022-04-22T17:53:50.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FjOVj3RGpkpGdK4 off=2515857 len=177377 2022-04-22T17:53:50.401 INFO:tasks.workunit.client.0.smithi115.stdout:op 2137 completed, throughput=5MB/sec 2022-04-22T17:53:50.401 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TaFVKhat646V0LG off=10139703 len=1943003 2022-04-22T17:53:50.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 2138 completed, throughput=5MB/sec 2022-04-22T17:53:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:50 smithi121 bash[36902]: audit 2022-04-22T17:53:50.031346+0000 mon.a (mon.0) 2445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:50 smithi121 bash[36902]: audit 2022-04-22T17:53:50.359733+0000 mon.a (mon.0) 2446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:50 smithi115 bash[45495]: audit 2022-04-22T17:53:50.031346+0000 mon.a (mon.0) 2445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:50 smithi115 bash[45495]: audit 2022-04-22T17:53:50.359733+0000 mon.a (mon.0) 2446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:50 smithi115 bash[46934]: audit 2022-04-22T17:53:50.031346+0000 mon.a (mon.0) 2445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:50 smithi115 bash[46934]: audit 2022-04-22T17:53:50.359733+0000 mon.a (mon.0) 2446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:51.410 INFO:tasks.workunit.client.0.smithi115.stdout: 417: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:51.411 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I2TEPqm-LVR0fBq off=6082582 len=1952790 2022-04-22T17:53:51.416 INFO:tasks.workunit.client.0.smithi115.stdout:op 2139 completed, throughput=5MB/sec 2022-04-22T17:53:51.416 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2i5hSiU-GP9lX4g off=3815571 len=113867 2022-04-22T17:53:51.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 2140 completed, throughput=5MB/sec 2022-04-22T17:53:51.421 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wy-4BZ5ACqUDPTS off=6069 len=1382197 2022-04-22T17:53:51.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 2141 completed, throughput=5MB/sec 2022-04-22T17:53:51.428 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FZZBEGciSLEe-oN off=3201471 len=584433 2022-04-22T17:53:51.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 2142 completed, throughput=5MB/sec 2022-04-22T17:53:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:51 smithi121 bash[36902]: cluster 2022-04-22T17:53:50.469407+0000 mgr.y (mgr.34107) 1000 : cluster [DBG] pgmap v1614: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:51 smithi115 bash[45495]: cluster 2022-04-22T17:53:50.469407+0000 mgr.y (mgr.34107) 1000 : cluster [DBG] pgmap v1614: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:52.075 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:51.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:52.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:51 smithi115 bash[46934]: cluster 2022-04-22T17:53:50.469407+0000 mgr.y (mgr.34107) 1000 : cluster [DBG] pgmap v1614: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:53:52.430 INFO:tasks.workunit.client.0.smithi115.stdout: 418: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:52.431 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5SWSdyBo6vvtOb- off=327700 len=570283 2022-04-22T17:53:52.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 2143 completed, throughput=4.99MB/sec 2022-04-22T17:53:52.432 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3UwtYkZINTjmPxv off=6806148 len=1338706 2022-04-22T17:53:52.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 2144 completed, throughput=4.99MB/sec 2022-04-22T17:53:52.441 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iaeRmY6GYNS2RZi off=21727532 len=1814922 2022-04-22T17:53:52.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 2145 completed, throughput=5MB/sec 2022-04-22T17:53:52.451 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YHOo9Y8OsHvLf3N off=4599601 len=1521016 2022-04-22T17:53:52.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 2146 completed, throughput=5MB/sec 2022-04-22T17:53:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:52 smithi121 bash[36902]: audit 2022-04-22T17:53:52.412045+0000 mon.a (mon.0) 2447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:52 smithi115 bash[45495]: audit 2022-04-22T17:53:52.412045+0000 mon.a (mon.0) 2447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:52 smithi115 bash[46934]: audit 2022-04-22T17:53:52.412045+0000 mon.a (mon.0) 2447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:53.456 INFO:tasks.workunit.client.0.smithi115.stdout: 419: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:53.457 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NrEYhsiB3jaUQg1 off=255503 len=820711 2022-04-22T17:53:53.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 2147 completed, throughput=4.99MB/sec 2022-04-22T17:53:53.463 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pDJEM5-nC3edgLI off=13704943 len=1706073 2022-04-22T17:53:53.471 INFO:tasks.workunit.client.0.smithi115.stdout:op 2148 completed, throughput=4.99MB/sec 2022-04-22T17:53:53.471 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9FcwVZm5xKQ55AS off=882167 len=1709884 2022-04-22T17:53:53.479 INFO:tasks.workunit.client.0.smithi115.stdout:op 2149 completed, throughput=5MB/sec 2022-04-22T17:53:53.479 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WeFanu6VFgiwflJ off=1381111 len=1794866 2022-04-22T17:53:53.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 2150 completed, throughput=5MB/sec 2022-04-22T17:53:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:53 smithi121 bash[36902]: cluster 2022-04-22T17:53:52.470540+0000 mgr.y (mgr.34107) 1001 : cluster [DBG] pgmap v1615: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:53 smithi115 bash[45495]: cluster 2022-04-22T17:53:52.470540+0000 mgr.y (mgr.34107) 1001 : cluster [DBG] pgmap v1615: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:53:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:53:53.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:53:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:53 smithi115 bash[46934]: cluster 2022-04-22T17:53:52.470540+0000 mgr.y (mgr.34107) 1001 : cluster [DBG] pgmap v1615: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:53:54.481 INFO:tasks.workunit.client.0.smithi115.stdout: 420: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:54.490 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gEV8iC52tiULi5H off=2176873 len=668233 2022-04-22T17:53:54.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 2151 completed, throughput=4.99MB/sec 2022-04-22T17:53:54.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--IHcVtqTXMqq5A5 off=2677661 len=1196106 2022-04-22T17:53:54.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 2152 completed, throughput=4.99MB/sec 2022-04-22T17:53:54.493 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8md80FRkpXT4Ubx off=6322265 len=1799886 2022-04-22T17:53:54.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 2153 completed, throughput=5MB/sec 2022-04-22T17:53:54.501 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_Uxp-Lzcg3nyDmy off=2258983 len=1982591 2022-04-22T17:53:54.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 2154 completed, throughput=5MB/sec 2022-04-22T17:53:55.506 INFO:tasks.workunit.client.0.smithi115.stdout: 421: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:55.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xMu4uZJd6oFEmEh off=1149717 len=674880 2022-04-22T17:53:55.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 2155 completed, throughput=4.99MB/sec 2022-04-22T17:53:55.512 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-04fXZ5FGGjr8lwc off=4781927 len=164083 2022-04-22T17:53:55.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 2156 completed, throughput=4.99MB/sec 2022-04-22T17:53:55.516 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8zrNhJiECyswboL off=12354046 len=1572835 2022-04-22T17:53:55.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 2157 completed, throughput=5MB/sec 2022-04-22T17:53:55.524 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-skCbsXRYx-s_tC7 off=6403009 len=1579633 2022-04-22T17:53:55.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 2158 completed, throughput=5MB/sec 2022-04-22T17:53:55.526 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LszvnRgY4ERe09F off=15227755 len=713087 2022-04-22T17:53:55.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 2159 completed, throughput=5MB/sec 2022-04-22T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:55 smithi121 bash[36902]: cluster 2022-04-22T17:53:54.471644+0000 mgr.y (mgr.34107) 1002 : cluster [DBG] pgmap v1616: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:55 smithi121 bash[36902]: audit 2022-04-22T17:53:55.034398+0000 mon.a (mon.0) 2448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:55 smithi121 bash[36902]: audit 2022-04-22T17:53:55.363717+0000 mon.a (mon.0) 2449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[45495]: cluster 2022-04-22T17:53:54.471644+0000 mgr.y (mgr.34107) 1002 : cluster [DBG] pgmap v1616: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:53:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[45495]: audit 2022-04-22T17:53:55.034398+0000 mon.a (mon.0) 2448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[45495]: audit 2022-04-22T17:53:55.363717+0000 mon.a (mon.0) 2449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[46934]: cluster 2022-04-22T17:53:54.471644+0000 mgr.y (mgr.34107) 1002 : cluster [DBG] pgmap v1616: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:53:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[46934]: audit 2022-04-22T17:53:55.034398+0000 mon.a (mon.0) 2448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:53:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:55 smithi115 bash[46934]: audit 2022-04-22T17:53:55.363717+0000 mon.a (mon.0) 2449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:53:56.528 INFO:tasks.workunit.client.0.smithi115.stdout: 422: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:56.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G4v7t1gykD5VTEu off=11401602 len=508988 2022-04-22T17:53:56.534 INFO:tasks.workunit.client.0.smithi115.stdout:op 2160 completed, throughput=4.99MB/sec 2022-04-22T17:53:56.534 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TBP5LldiZEr6Za8 off=2469458 len=245991 2022-04-22T17:53:56.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 2161 completed, throughput=4.99MB/sec 2022-04-22T17:53:56.535 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TgjvUJjcHncW7dO off=1516541 len=1551039 2022-04-22T17:53:56.543 INFO:tasks.workunit.client.0.smithi115.stdout:op 2162 completed, throughput=4.99MB/sec 2022-04-22T17:53:56.543 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KEagjuqrofU42AS off=8978038 len=2083542 2022-04-22T17:53:56.555 INFO:tasks.workunit.client.0.smithi115.stdout:op 2163 completed, throughput=5MB/sec 2022-04-22T17:53:56.555 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RTh2c9_TUNt7V-- off=13156937 len=693400 2022-04-22T17:53:56.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 2164 completed, throughput=5MB/sec 2022-04-22T17:53:57.557 INFO:tasks.workunit.client.0.smithi115.stdout: 423: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:57.557 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wu-Pn_B34s-EP_r off=4321581 len=867938 2022-04-22T17:53:57.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 2165 completed, throughput=4.99MB/sec 2022-04-22T17:53:57.559 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bx7860KhfTThHvK off=21784794 len=335537 2022-04-22T17:53:57.561 INFO:tasks.workunit.client.0.smithi115.stdout:op 2166 completed, throughput=4.99MB/sec 2022-04-22T17:53:57.561 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VjkZD4lfqDDeXVz off=5444567 len=1991027 2022-04-22T17:53:57.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 2167 completed, throughput=5MB/sec 2022-04-22T17:53:57.565 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tghqUFVf1upfses off=8509275 len=215745 2022-04-22T17:53:57.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 2168 completed, throughput=5MB/sec 2022-04-22T17:53:57.570 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--yFZmU7UiE5W0E8 off=10638664 len=1269953 2022-04-22T17:53:57.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 2169 completed, throughput=5MB/sec 2022-04-22T17:53:57.573 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Y6x6A-dKAk-Y4Xs off=712348 len=117132 2022-04-22T17:53:57.574 INFO:tasks.workunit.client.0.smithi115.stdout:op 2170 completed, throughput=5MB/sec 2022-04-22T17:53:57.574 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nVCkaCAz5uXGFer off=3102789 len=1198414 2022-04-22T17:53:57.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 2171 completed, throughput=5MB/sec 2022-04-22T17:53:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:57 smithi121 bash[36902]: cluster 2022-04-22T17:53:56.472724+0000 mgr.y (mgr.34107) 1003 : cluster [DBG] pgmap v1617: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:53:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:57 smithi121 bash[36902]: audit 2022-04-22T17:53:57.415080+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:57 smithi115 bash[45495]: cluster 2022-04-22T17:53:56.472724+0000 mgr.y (mgr.34107) 1003 : cluster [DBG] pgmap v1617: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:53:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:57 smithi115 bash[45495]: audit 2022-04-22T17:53:57.415080+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:57 smithi115 bash[46934]: cluster 2022-04-22T17:53:56.472724+0000 mgr.y (mgr.34107) 1003 : cluster [DBG] pgmap v1617: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:53:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:57 smithi115 bash[46934]: audit 2022-04-22T17:53:57.415080+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:53:58.582 INFO:tasks.workunit.client.0.smithi115.stdout: 424: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:58.582 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8KptWNSmoBJEdJj off=13754888 len=428007 2022-04-22T17:53:58.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 2172 completed, throughput=4.99MB/sec 2022-04-22T17:53:58.584 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h-K066ZuYOLkXpi off=16292749 len=1637090 2022-04-22T17:53:58.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 2173 completed, throughput=4.99MB/sec 2022-04-22T17:53:58.594 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YiE_IuQ9cKqrX3T off=3291240 len=875151 2022-04-22T17:53:58.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 2174 completed, throughput=5MB/sec 2022-04-22T17:53:58.596 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-blEJVxB84V6nJlH off=3107036 len=1942107 2022-04-22T17:53:58.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 2175 completed, throughput=5MB/sec 2022-04-22T17:53:59.607 INFO:tasks.workunit.client.0.smithi115.stdout: 425: throughput=4.99MB/sec pending data=0 2022-04-22T17:53:59.608 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ARPXfDIOLzbScqK off=6239223 len=1035176 2022-04-22T17:53:59.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 2176 completed, throughput=4.99MB/sec 2022-04-22T17:53:59.610 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x7_c9Dpa30w9uA3 off=4996852 len=1202839 2022-04-22T17:53:59.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 2177 completed, throughput=4.99MB/sec 2022-04-22T17:53:59.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NS0_MW0gSPq7Ldf off=6005678 len=951617 2022-04-22T17:53:59.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 2178 completed, throughput=5MB/sec 2022-04-22T17:53:59.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZuTpftm6syanRYH off=5647384 len=1684229 2022-04-22T17:53:59.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 2179 completed, throughput=5MB/sec 2022-04-22T17:53:59.621 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UdnlSVKsRMus0Oy off=7974236 len=1892954 2022-04-22T17:53:59.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 2180 completed, throughput=5MB/sec 2022-04-22T17:54:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:53:59 smithi121 bash[36902]: cluster 2022-04-22T17:53:58.474044+0000 mgr.y (mgr.34107) 1004 : cluster [DBG] pgmap v1618: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T17:54:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:53:59 smithi115 bash[45495]: cluster 2022-04-22T17:53:58.474044+0000 mgr.y (mgr.34107) 1004 : cluster [DBG] pgmap v1618: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T17:54:00.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:53:59 smithi115 bash[46934]: cluster 2022-04-22T17:53:58.474044+0000 mgr.y (mgr.34107) 1004 : cluster [DBG] pgmap v1618: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T17:54:00.624 INFO:tasks.workunit.client.0.smithi115.stdout: 426: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:00.625 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--Hax6ZX1mNTTcRQ off=2249246 len=121572 2022-04-22T17:54:00.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 2181 completed, throughput=4.99MB/sec 2022-04-22T17:54:00.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2iEdmg2X6Qt50gA off=1054864 len=1916469 2022-04-22T17:54:00.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 2182 completed, throughput=5MB/sec 2022-04-22T17:54:00.636 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E6r8ng4QmnMiYzQ off=5850373 len=1057369 2022-04-22T17:54:00.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 2183 completed, throughput=5MB/sec 2022-04-22T17:54:00.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tIy_mnGkiEsdU1E off=176521 len=490906 2022-04-22T17:54:00.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 2184 completed, throughput=5MB/sec 2022-04-22T17:54:00.638 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aAEC0LQ3yoUBiDD off=3777297 len=1631452 2022-04-22T17:54:00.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 2185 completed, throughput=5MB/sec 2022-04-22T17:54:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:00 smithi121 bash[36902]: audit 2022-04-22T17:54:00.037390+0000 mon.a (mon.0) 2451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:01.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:00 smithi121 bash[36902]: audit 2022-04-22T17:54:00.367346+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:00 smithi115 bash[46934]: audit 2022-04-22T17:54:00.037390+0000 mon.a (mon.0) 2451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:00 smithi115 bash[46934]: audit 2022-04-22T17:54:00.367346+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:00 smithi115 bash[45495]: audit 2022-04-22T17:54:00.037390+0000 mon.a (mon.0) 2451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:00 smithi115 bash[45495]: audit 2022-04-22T17:54:00.367346+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:01.640 INFO:tasks.workunit.client.0.smithi115.stdout: 427: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:01.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jcx4rJ8_tUKB4IC off=14131665 len=1047377 2022-04-22T17:54:01.646 INFO:tasks.workunit.client.0.smithi115.stdout:op 2186 completed, throughput=4.99MB/sec 2022-04-22T17:54:01.646 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GwZIuj-O80ZLqZP off=15458930 len=1256024 2022-04-22T17:54:01.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 2187 completed, throughput=5MB/sec 2022-04-22T17:54:01.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lnCKY8zW7PlgzzJ off=3475190 len=958654 2022-04-22T17:54:01.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 2188 completed, throughput=5MB/sec 2022-04-22T17:54:01.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3l3bfo5yEuhCyen off=19399597 len=956429 2022-04-22T17:54:01.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 2189 completed, throughput=5MB/sec 2022-04-22T17:54:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:01 smithi121 bash[36902]: cluster 2022-04-22T17:54:00.475163+0000 mgr.y (mgr.34107) 1005 : cluster [DBG] pgmap v1619: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:54:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:01 smithi115 bash[46934]: cluster 2022-04-22T17:54:00.475163+0000 mgr.y (mgr.34107) 1005 : cluster [DBG] pgmap v1619: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:54:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:01.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:01 smithi115 bash[45495]: cluster 2022-04-22T17:54:00.475163+0000 mgr.y (mgr.34107) 1005 : cluster [DBG] pgmap v1619: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:54:02.660 INFO:tasks.workunit.client.0.smithi115.stdout: 428: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:02.661 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VuDnkMOxygHR4rq off=1052832 len=924426 2022-04-22T17:54:02.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 2190 completed, throughput=4.99MB/sec 2022-04-22T17:54:02.663 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-evlMmqA0GRyMmaP off=12137757 len=669569 2022-04-22T17:54:02.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 2191 completed, throughput=4.99MB/sec 2022-04-22T17:54:02.666 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-j_CiyHVOqbq-_DQ off=9885898 len=2042828 2022-04-22T17:54:02.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 2192 completed, throughput=5MB/sec 2022-04-22T17:54:02.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kpkHx-vaeL77DBS off=1250972 len=2009481 2022-04-22T17:54:02.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 2193 completed, throughput=5MB/sec 2022-04-22T17:54:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:02 smithi121 bash[36902]: audit 2022-04-22T17:54:02.418261+0000 mon.a (mon.0) 2453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:02 smithi115 bash[46934]: audit 2022-04-22T17:54:02.418261+0000 mon.a (mon.0) 2453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:02 smithi115 bash[45495]: audit 2022-04-22T17:54:02.418261+0000 mon.a (mon.0) 2453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:03.680 INFO:tasks.workunit.client.0.smithi115.stdout: 429: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:03.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SKMSarVhikeZbxO off=5763714 len=47643 2022-04-22T17:54:03.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 2194 completed, throughput=4.99MB/sec 2022-04-22T17:54:03.709 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_CPOj0zdQ2f8urR off=6689902 len=1624937 2022-04-22T17:54:03.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 2195 completed, throughput=4.99MB/sec 2022-04-22T17:54:03.709 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mYCKU2xbzH39Usv off=3476320 len=2017094 2022-04-22T17:54:03.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 2196 completed, throughput=5MB/sec 2022-04-22T17:54:03.710 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bzUAeKaEvoSqaYi off=9834007 len=1288186 2022-04-22T17:54:03.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 2197 completed, throughput=5MB/sec 2022-04-22T17:54:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:03 smithi121 bash[36902]: cluster 2022-04-22T17:54:02.476085+0000 mgr.y (mgr.34107) 1006 : cluster [DBG] pgmap v1620: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:03 smithi115 bash[45495]: cluster 2022-04-22T17:54:02.476085+0000 mgr.y (mgr.34107) 1006 : cluster [DBG] pgmap v1620: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:03.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:03 smithi115 bash[46934]: cluster 2022-04-22T17:54:02.476085+0000 mgr.y (mgr.34107) 1006 : cluster [DBG] pgmap v1620: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:04.704 INFO:tasks.workunit.client.0.smithi115.stdout: 430: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:04.704 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hj7M5Uz9dbRcn80 off=15782957 len=1986087 2022-04-22T17:54:04.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 2198 completed, throughput=4.99MB/sec 2022-04-22T17:54:04.715 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-I-9MeSJwezWvkpg off=565673 len=1067270 2022-04-22T17:54:04.721 INFO:tasks.workunit.client.0.smithi115.stdout:op 2199 completed, throughput=5MB/sec 2022-04-22T17:54:04.721 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YFYgw9tw3jbAY9m off=1225498 len=1996628 2022-04-22T17:54:04.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 2200 completed, throughput=5MB/sec 2022-04-22T17:54:05.725 INFO:tasks.workunit.client.0.smithi115.stdout: 431: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:05.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1QwIgr1V6qYkq5R off=493227 len=656991 2022-04-22T17:54:05.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 2201 completed, throughput=4.99MB/sec 2022-04-22T17:54:05.728 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--xpCvBQpxNaiP6R off=1809696 len=2012840 2022-04-22T17:54:05.739 INFO:tasks.workunit.client.0.smithi115.stdout:op 2202 completed, throughput=4.99MB/sec 2022-04-22T17:54:05.739 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n0Mb5l8hp7UcnuK off=2604990 len=1489739 2022-04-22T17:54:05.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 2203 completed, throughput=5MB/sec 2022-04-22T17:54:05.743 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tc6oTB4iYPkiJKz off=128256 len=671460 2022-04-22T17:54:05.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 2204 completed, throughput=5MB/sec 2022-04-22T17:54:05.748 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yA-sceFTwQHFaYd off=3781505 len=834687 2022-04-22T17:54:05.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 2205 completed, throughput=5MB/sec 2022-04-22T17:54:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:05 smithi121 bash[36902]: cluster 2022-04-22T17:54:04.477146+0000 mgr.y (mgr.34107) 1007 : cluster [DBG] pgmap v1621: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:54:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:05 smithi121 bash[36902]: audit 2022-04-22T17:54:05.040091+0000 mon.a (mon.0) 2454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:05 smithi121 bash[36902]: audit 2022-04-22T17:54:05.371108+0000 mon.a (mon.0) 2455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[45495]: cluster 2022-04-22T17:54:04.477146+0000 mgr.y (mgr.34107) 1007 : cluster [DBG] pgmap v1621: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:54:06.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[45495]: audit 2022-04-22T17:54:05.040091+0000 mon.a (mon.0) 2454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:06.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[45495]: audit 2022-04-22T17:54:05.371108+0000 mon.a (mon.0) 2455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[46934]: cluster 2022-04-22T17:54:04.477146+0000 mgr.y (mgr.34107) 1007 : cluster [DBG] pgmap v1621: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:54:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[46934]: audit 2022-04-22T17:54:05.040091+0000 mon.a (mon.0) 2454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:05 smithi115 bash[46934]: audit 2022-04-22T17:54:05.371108+0000 mon.a (mon.0) 2455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:06.751 INFO:tasks.workunit.client.0.smithi115.stdout: 432: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:06.751 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7b4eAYY7rcGliA6 off=13139578 len=1207475 2022-04-22T17:54:06.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 2206 completed, throughput=4.99MB/sec 2022-04-22T17:54:06.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tSylgU6h6_y6YUe off=6733506 len=869601 2022-04-22T17:54:06.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 2207 completed, throughput=4.99MB/sec 2022-04-22T17:54:06.762 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--BeTlyAYcm-gKXP off=9293104 len=636097 2022-04-22T17:54:06.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 2208 completed, throughput=5MB/sec 2022-04-22T17:54:06.764 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4qZyo86VSDLWjFL off=3477893 len=1705393 2022-04-22T17:54:06.772 INFO:tasks.workunit.client.0.smithi115.stdout:op 2209 completed, throughput=5MB/sec 2022-04-22T17:54:06.772 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZsPNXGYqkhzDDyF off=19717922 len=326695 2022-04-22T17:54:06.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 2210 completed, throughput=5MB/sec 2022-04-22T17:54:06.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LdJMQ0ZxaYgZHhY off=7191083 len=594208 2022-04-22T17:54:06.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 2211 completed, throughput=5MB/sec 2022-04-22T17:54:07.511 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:54:07.775 INFO:tasks.workunit.client.0.smithi115.stdout: 433: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:07.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sNtYYCD7eRr3H9v off=16685 len=397320 2022-04-22T17:54:07.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 2212 completed, throughput=4.99MB/sec 2022-04-22T17:54:07.777 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4oQ5M-3sQ3peCes off=275427 len=218017 2022-04-22T17:54:07.780 INFO:tasks.workunit.client.0.smithi115.stdout:op 2213 completed, throughput=4.99MB/sec 2022-04-22T17:54:07.780 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w2ohr8CVxvJNs8_ off=5608264 len=1002901 2022-04-22T17:54:07.787 INFO:tasks.workunit.client.0.smithi115.stdout:op 2214 completed, throughput=4.99MB/sec 2022-04-22T17:54:07.787 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tUEYqOo9A8LgUsd off=2081985 len=265330 2022-04-22T17:54:07.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 2215 completed, throughput=4.99MB/sec 2022-04-22T17:54:07.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yS32Q36D3DO7wfk off=10351937 len=1014212 2022-04-22T17:54:07.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 2216 completed, throughput=5MB/sec 2022-04-22T17:54:07.794 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dRdpfje5ODFJv-h off=535683 len=901179 2022-04-22T17:54:07.795 INFO:tasks.workunit.client.0.smithi115.stdout:op 2217 completed, throughput=5MB/sec 2022-04-22T17:54:07.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dG1u-P0jVVgXW2x off=3749426 len=1605949 2022-04-22T17:54:07.797 INFO:tasks.workunit.client.0.smithi115.stdout:op 2218 completed, throughput=5MB/sec 2022-04-22T17:54:07.863 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:54:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (17m) 6m ago 30m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:54:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (30m) 7m ago 30m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:54:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (19m) 6m ago 32m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:54:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (18m) 7m ago 31m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:54:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (26m) 7m ago 36m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:54:07.880 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (27m) 6m ago 38m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:54:07.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (26m) 6m ago 38m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:54:07.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (25m) 7m ago 36m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:54:07.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (25m) 6m ago 36m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:54:07.880 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (18m) 6m ago 31m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:54:07.881 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (18m) 7m ago 31m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:54:07.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (23m) 6m ago 35m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:54:07.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (23m) 6m ago 35m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:54:07.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (22m) 6m ago 34m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:54:07.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (22m) 6m ago 34m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:54:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (21m) 7m ago 33m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:54:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (20m) 7m ago 33m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:54:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (20m) 7m ago 33m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:54:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (19m) 7m ago 32m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:54:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (17m) 7m ago 31m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:54:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:07 smithi121 bash[36902]: cluster 2022-04-22T17:54:06.478159+0000 mgr.y (mgr.34107) 1008 : cluster [DBG] pgmap v1622: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:07 smithi121 bash[36902]: audit 2022-04-22T17:54:07.421171+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:07 smithi115 bash[45495]: cluster 2022-04-22T17:54:06.478159+0000 mgr.y (mgr.34107) 1008 : cluster [DBG] pgmap v1622: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:07 smithi115 bash[45495]: audit 2022-04-22T17:54:07.421171+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:07 smithi115 bash[46934]: cluster 2022-04-22T17:54:06.478159+0000 mgr.y (mgr.34107) 1008 : cluster [DBG] pgmap v1622: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:54:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:07 smithi115 bash[46934]: audit 2022-04-22T17:54:07.421171+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:08.259 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:54:08.259 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:54:08.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:54:08.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:08.261 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:54:08.261 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:54:08.261 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:54:08.261 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:54:08.798 INFO:tasks.workunit.client.0.smithi115.stdout: 434: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:08.798 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M0jEPh8qIQIChdp off=11234773 len=1152472 2022-04-22T17:54:08.805 INFO:tasks.workunit.client.0.smithi115.stdout:op 2219 completed, throughput=4.99MB/sec 2022-04-22T17:54:08.805 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_HTyvQ5wGbBiEK0 off=847134 len=1490384 2022-04-22T17:54:08.809 INFO:tasks.workunit.client.0.smithi115.stdout:op 2220 completed, throughput=5MB/sec 2022-04-22T17:54:08.809 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VYyp3j_38vwez3d off=11755125 len=607763 2022-04-22T17:54:08.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 2221 completed, throughput=5MB/sec 2022-04-22T17:54:08.815 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G53o5FvP-KEdtzz off=266128 len=59051 2022-04-22T17:54:08.816 INFO:tasks.workunit.client.0.smithi115.stdout:op 2222 completed, throughput=5MB/sec 2022-04-22T17:54:08.816 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U0eD-Nm2UzM3IhE off=9376558 len=1398474 2022-04-22T17:54:08.822 INFO:tasks.workunit.client.0.smithi115.stdout:op 2223 completed, throughput=5MB/sec 2022-04-22T17:54:08.822 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-a3OXnltKfPIxDO9 off=4368906 len=395348 2022-04-22T17:54:08.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 2224 completed, throughput=5MB/sec 2022-04-22T17:54:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:08 smithi121 bash[36902]: audit 2022-04-22T17:54:07.486868+0000 mgr.y (mgr.34107) 1009 : audit [DBG] from='client.45403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:08 smithi121 bash[36902]: audit 2022-04-22T17:54:07.836377+0000 mgr.y (mgr.34107) 1010 : audit [DBG] from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:08 smithi121 bash[36902]: audit 2022-04-22T17:54:08.252192+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.115:0/3921974249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[45495]: audit 2022-04-22T17:54:07.486868+0000 mgr.y (mgr.34107) 1009 : audit [DBG] from='client.45403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[45495]: audit 2022-04-22T17:54:07.836377+0000 mgr.y (mgr.34107) 1010 : audit [DBG] from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[45495]: audit 2022-04-22T17:54:08.252192+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.115:0/3921974249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[46934]: audit 2022-04-22T17:54:07.486868+0000 mgr.y (mgr.34107) 1009 : audit [DBG] from='client.45403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[46934]: audit 2022-04-22T17:54:07.836377+0000 mgr.y (mgr.34107) 1010 : audit [DBG] from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:08 smithi115 bash[46934]: audit 2022-04-22T17:54:08.252192+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.115:0/3921974249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:09.824 INFO:tasks.workunit.client.0.smithi115.stdout: 435: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:09.824 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-44Ll0kXZF3Pe1R1 off=7502276 len=1096425 2022-04-22T17:54:09.832 INFO:tasks.workunit.client.0.smithi115.stdout:op 2225 completed, throughput=4.99MB/sec 2022-04-22T17:54:09.832 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kw5i2xp7SCGdwyE off=2520167 len=722648 2022-04-22T17:54:09.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 2226 completed, throughput=4.99MB/sec 2022-04-22T17:54:09.834 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QeqR4B5EgTfM_tQ off=9336131 len=643855 2022-04-22T17:54:09.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 2227 completed, throughput=4.99MB/sec 2022-04-22T17:54:09.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3Jg3Cen1HjadWJF off=1851396 len=1376862 2022-04-22T17:54:09.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 2228 completed, throughput=5MB/sec 2022-04-22T17:54:09.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QXt-B5_TJP_gRb- off=16354174 len=1327767 2022-04-22T17:54:09.847 INFO:tasks.workunit.client.0.smithi115.stdout:op 2229 completed, throughput=5MB/sec 2022-04-22T17:54:09.847 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VuDnkMOxygHR4rq off=1717711 len=1356809 2022-04-22T17:54:09.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 2230 completed, throughput=5MB/sec 2022-04-22T17:54:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:09 smithi121 bash[36902]: cluster 2022-04-22T17:54:08.479427+0000 mgr.y (mgr.34107) 1011 : cluster [DBG] pgmap v1623: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:09 smithi115 bash[45495]: cluster 2022-04-22T17:54:08.479427+0000 mgr.y (mgr.34107) 1011 : cluster [DBG] pgmap v1623: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:09 smithi115 bash[46934]: cluster 2022-04-22T17:54:08.479427+0000 mgr.y (mgr.34107) 1011 : cluster [DBG] pgmap v1623: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:10.854 INFO:tasks.workunit.client.0.smithi115.stdout: 436: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:10.854 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rQTfjFrAwSXuqGe off=4909455 len=1693668 2022-04-22T17:54:10.864 INFO:tasks.workunit.client.0.smithi115.stdout:op 2231 completed, throughput=4.99MB/sec 2022-04-22T17:54:10.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxaufqIvu4KwqKP off=7690322 len=206097 2022-04-22T17:54:10.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 2232 completed, throughput=5MB/sec 2022-04-22T17:54:10.909 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PHvxTvq3EFeDEaC off=7160891 len=1770923 2022-04-22T17:54:10.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 2233 completed, throughput=5MB/sec 2022-04-22T17:54:10.909 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Y_8ga8AGLK4AyvC off=392952 len=1158349 2022-04-22T17:54:10.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 2234 completed, throughput=5MB/sec 2022-04-22T17:54:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:10 smithi121 bash[36902]: audit 2022-04-22T17:54:10.043166+0000 mon.a (mon.0) 2457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:10 smithi121 bash[36902]: audit 2022-04-22T17:54:10.374902+0000 mon.a (mon.0) 2458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:10 smithi115 bash[45495]: audit 2022-04-22T17:54:10.043166+0000 mon.a (mon.0) 2457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:10 smithi115 bash[45495]: audit 2022-04-22T17:54:10.374902+0000 mon.a (mon.0) 2458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:10 smithi115 bash[46934]: audit 2022-04-22T17:54:10.043166+0000 mon.a (mon.0) 2457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:10 smithi115 bash[46934]: audit 2022-04-22T17:54:10.374902+0000 mon.a (mon.0) 2458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:11.873 INFO:tasks.workunit.client.0.smithi115.stdout: 437: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:11.873 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fo3vFRgXd_myeuO off=5729612 len=966275 2022-04-22T17:54:11.881 INFO:tasks.workunit.client.0.smithi115.stdout:op 2235 completed, throughput=4.99MB/sec 2022-04-22T17:54:11.881 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zjWpwVksqk6GmYz off=14129005 len=1393536 2022-04-22T17:54:11.888 INFO:tasks.workunit.client.0.smithi115.stdout:op 2236 completed, throughput=5MB/sec 2022-04-22T17:54:11.889 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6Z4mn6qsCFyLIV6 off=18813064 len=392450 2022-04-22T17:54:11.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 2237 completed, throughput=5MB/sec 2022-04-22T17:54:11.890 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sh_x1C232-V8WAG off=14563322 len=870453 2022-04-22T17:54:11.892 INFO:tasks.workunit.client.0.smithi115.stdout:op 2238 completed, throughput=5MB/sec 2022-04-22T17:54:11.892 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HQ8JyFAVW1uiG97 off=16974827 len=1818401 2022-04-22T17:54:11.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 2239 completed, throughput=5MB/sec 2022-04-22T17:54:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:11 smithi121 bash[36902]: cluster 2022-04-22T17:54:10.480517+0000 mgr.y (mgr.34107) 1012 : cluster [DBG] pgmap v1624: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:12.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:11.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:11 smithi115 bash[45495]: cluster 2022-04-22T17:54:10.480517+0000 mgr.y (mgr.34107) 1012 : cluster [DBG] pgmap v1624: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:11 smithi115 bash[46934]: cluster 2022-04-22T17:54:10.480517+0000 mgr.y (mgr.34107) 1012 : cluster [DBG] pgmap v1624: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:12.896 INFO:tasks.workunit.client.0.smithi115.stdout: 438: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:12.896 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C7FM6-2yYE7TXhk off=11225497 len=1769994 2022-04-22T17:54:12.905 INFO:tasks.workunit.client.0.smithi115.stdout:op 2240 completed, throughput=4.99MB/sec 2022-04-22T17:54:12.906 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i0pabcZTNClNheb off=1165132 len=841929 2022-04-22T17:54:12.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 2241 completed, throughput=5MB/sec 2022-04-22T17:54:12.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6WPGwCa3LMADo9N off=10726599 len=423090 2022-04-22T17:54:12.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 2242 completed, throughput=5MB/sec 2022-04-22T17:54:12.916 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XxnY9wIkmKSu8k5 off=13054572 len=1662534 2022-04-22T17:54:12.919 INFO:tasks.workunit.client.0.smithi115.stdout:op 2243 completed, throughput=5MB/sec 2022-04-22T17:54:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:12 smithi121 bash[36902]: audit 2022-04-22T17:54:12.423758+0000 mon.a (mon.0) 2459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:12 smithi115 bash[45495]: audit 2022-04-22T17:54:12.423758+0000 mon.a (mon.0) 2459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:12 smithi115 bash[46934]: audit 2022-04-22T17:54:12.423758+0000 mon.a (mon.0) 2459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:13.920 INFO:tasks.workunit.client.0.smithi115.stdout: 439: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:13.920 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D2t9_JjFY3u_Y6M off=239307 len=1928740 2022-04-22T17:54:13.924 INFO:tasks.workunit.client.0.smithi115.stdout:op 2244 completed, throughput=4.99MB/sec 2022-04-22T17:54:13.924 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Aotqf3IVYuFeGiq off=6799251 len=1716815 2022-04-22T17:54:13.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 2245 completed, throughput=5MB/sec 2022-04-22T17:54:13.929 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dgftm1ihiWSrF0m off=22101132 len=1883550 2022-04-22T17:54:13.932 INFO:tasks.workunit.client.0.smithi115.stdout:op 2246 completed, throughput=5MB/sec 2022-04-22T17:54:14.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:13 smithi121 bash[36902]: cluster 2022-04-22T17:54:12.481572+0000 mgr.y (mgr.34107) 1013 : cluster [DBG] pgmap v1625: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:14.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:13 smithi121 bash[36902]: audit 2022-04-22T17:54:13.517478+0000 mon.a (mon.0) 2460 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:54:14.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:13 smithi121 bash[36902]: audit 2022-04-22T17:54:13.517683+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:54:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:13.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[46934]: cluster 2022-04-22T17:54:12.481572+0000 mgr.y (mgr.34107) 1013 : cluster [DBG] pgmap v1625: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[46934]: audit 2022-04-22T17:54:13.517478+0000 mon.a (mon.0) 2460 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[46934]: audit 2022-04-22T17:54:13.517683+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[45495]: cluster 2022-04-22T17:54:12.481572+0000 mgr.y (mgr.34107) 1013 : cluster [DBG] pgmap v1625: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[45495]: audit 2022-04-22T17:54:13.517478+0000 mon.a (mon.0) 2460 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:54:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:13 smithi115 bash[45495]: audit 2022-04-22T17:54:13.517683+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:54:14.932 INFO:tasks.workunit.client.0.smithi115.stdout: 440: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:14.932 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bYRRhDlO9izebOa off=13210042 len=2092054 2022-04-22T17:54:14.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 2247 completed, throughput=4.99MB/sec 2022-04-22T17:54:14.941 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mV-yOV5HSvRtyhs off=3202003 len=669263 2022-04-22T17:54:14.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 2248 completed, throughput=5MB/sec 2022-04-22T17:54:14.943 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bEBYXZPP01JWzI2 off=17197039 len=1721214 2022-04-22T17:54:14.946 INFO:tasks.workunit.client.0.smithi115.stdout:op 2249 completed, throughput=5MB/sec 2022-04-22T17:54:14.946 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TC0YPTTXrzHsPZh off=19843041 len=1834260 2022-04-22T17:54:14.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 2250 completed, throughput=5MB/sec 2022-04-22T17:54:15.953 INFO:tasks.workunit.client.0.smithi115.stdout: 441: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:15.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NJ9AncWTUAI63fZ off=2711739 len=1943106 2022-04-22T17:54:15.957 INFO:tasks.workunit.client.0.smithi115.stdout:op 2251 completed, throughput=5MB/sec 2022-04-22T17:54:15.957 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m2qo3b9a_-A9zi- off=1317678 len=1340906 2022-04-22T17:54:15.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 2252 completed, throughput=5MB/sec 2022-04-22T17:54:15.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0TirOxnmb3k7jFW off=1486733 len=1092490 2022-04-22T17:54:15.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 2253 completed, throughput=5MB/sec 2022-04-22T17:54:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:15 smithi121 bash[36902]: cluster 2022-04-22T17:54:14.482695+0000 mgr.y (mgr.34107) 1014 : cluster [DBG] pgmap v1626: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:15 smithi121 bash[36902]: audit 2022-04-22T17:54:15.046036+0000 mon.a (mon.0) 2462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:15 smithi121 bash[36902]: audit 2022-04-22T17:54:15.378903+0000 mon.a (mon.0) 2463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[45495]: cluster 2022-04-22T17:54:14.482695+0000 mgr.y (mgr.34107) 1014 : cluster [DBG] pgmap v1626: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:16.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[45495]: audit 2022-04-22T17:54:15.046036+0000 mon.a (mon.0) 2462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:16.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[45495]: audit 2022-04-22T17:54:15.378903+0000 mon.a (mon.0) 2463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[46934]: cluster 2022-04-22T17:54:14.482695+0000 mgr.y (mgr.34107) 1014 : cluster [DBG] pgmap v1626: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[46934]: audit 2022-04-22T17:54:15.046036+0000 mon.a (mon.0) 2462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:15 smithi115 bash[46934]: audit 2022-04-22T17:54:15.378903+0000 mon.a (mon.0) 2463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:16.968 INFO:tasks.workunit.client.0.smithi115.stdout: 442: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:16.969 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k-Cbg2xmk-KPsdX off=6467276 len=273333 2022-04-22T17:54:16.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 2254 completed, throughput=4.99MB/sec 2022-04-22T17:54:16.971 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gQu8Vlfe2yMNVF- off=4177214 len=1684688 2022-04-22T17:54:16.975 INFO:tasks.workunit.client.0.smithi115.stdout:op 2255 completed, throughput=4.99MB/sec 2022-04-22T17:54:17.019 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xDTTYxbFEqxZiug off=2540551 len=724192 2022-04-22T17:54:17.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 2256 completed, throughput=5MB/sec 2022-04-22T17:54:17.020 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K9Tb2gmdQLA-0z8 off=6635272 len=494683 2022-04-22T17:54:17.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 2257 completed, throughput=5MB/sec 2022-04-22T17:54:17.021 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LGrp5TK95Yqf4GM off=5875865 len=1184296 2022-04-22T17:54:17.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 2258 completed, throughput=5MB/sec 2022-04-22T17:54:17.021 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o3yKIL9axX45Wsb off=9887070 len=924455 2022-04-22T17:54:17.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 2259 completed, throughput=5MB/sec 2022-04-22T17:54:17.992 INFO:tasks.workunit.client.0.smithi115.stdout: 443: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:17.992 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2vJXJ7WF4f8jxy0 off=11381334 len=1821134 2022-04-22T17:54:17.996 INFO:tasks.workunit.client.0.smithi115.stdout:op 2260 completed, throughput=4.99MB/sec 2022-04-22T17:54:17.996 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cvAteHF2bSXHh_i off=19234762 len=803449 2022-04-22T17:54:18.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 2261 completed, throughput=4.99MB/sec 2022-04-22T17:54:18.004 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M8hrtx9-gevLG6h off=3866816 len=341704 2022-04-22T17:54:18.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 2262 completed, throughput=5MB/sec 2022-04-22T17:54:18.009 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8u8TqIDHmYAI9T_ off=2872142 len=1793527 2022-04-22T17:54:18.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 2263 completed, throughput=5MB/sec 2022-04-22T17:54:18.013 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uZnjxY5vjeK0-CJ off=5263056 len=1629949 2022-04-22T17:54:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:17 smithi121 bash[36902]: cluster 2022-04-22T17:54:16.483806+0000 mgr.y (mgr.34107) 1015 : cluster [DBG] pgmap v1627: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:17 smithi121 bash[36902]: audit 2022-04-22T17:54:17.426741+0000 mon.a (mon.0) 2464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:18.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 2264 completed, throughput=5MB/sec 2022-04-22T17:54:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:17 smithi115 bash[45495]: cluster 2022-04-22T17:54:16.483806+0000 mgr.y (mgr.34107) 1015 : cluster [DBG] pgmap v1627: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:17 smithi115 bash[45495]: audit 2022-04-22T17:54:17.426741+0000 mon.a (mon.0) 2464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:17 smithi115 bash[46934]: cluster 2022-04-22T17:54:16.483806+0000 mgr.y (mgr.34107) 1015 : cluster [DBG] pgmap v1627: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:54:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:17 smithi115 bash[46934]: audit 2022-04-22T17:54:17.426741+0000 mon.a (mon.0) 2464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:19.022 INFO:tasks.workunit.client.0.smithi115.stdout: 444: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:19.067 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zWwlDtK64XaL0k4 off=1109139 len=611254 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 2265 completed, throughput=4.99MB/sec 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cJUtGkcYG3lUTKf off=976763 len=789017 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 2266 completed, throughput=4.99MB/sec 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h_KAdm4AYsC4LOA off=1592382 len=571283 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 2267 completed, throughput=5MB/sec 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O6KQ9SAjxUTtzf7 off=557795 len=306760 2022-04-22T17:54:19.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 2268 completed, throughput=5MB/sec 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jy5GKceiVHu9xgB off=1151782 len=274546 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 2269 completed, throughput=5MB/sec 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FLf9MURUonlD4wt off=19729229 len=577223 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 2270 completed, throughput=5MB/sec 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EC1l7U7RWuiZEzP off=9097251 len=1898830 2022-04-22T17:54:19.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 2271 completed, throughput=5MB/sec 2022-04-22T17:54:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:19 smithi121 bash[36902]: cluster 2022-04-22T17:54:18.484999+0000 mgr.y (mgr.34107) 1016 : cluster [DBG] pgmap v1628: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:54:20.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:19 smithi115 bash[45495]: cluster 2022-04-22T17:54:18.484999+0000 mgr.y (mgr.34107) 1016 : cluster [DBG] pgmap v1628: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:54:20.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:19 smithi115 bash[46934]: cluster 2022-04-22T17:54:18.484999+0000 mgr.y (mgr.34107) 1016 : cluster [DBG] pgmap v1628: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:54:20.045 INFO:tasks.workunit.client.0.smithi115.stdout: 445: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:20.045 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ec2O6GHqO8IzOr6 off=0 len=1192632 2022-04-22T17:54:20.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 2272 completed, throughput=4.99MB/sec 2022-04-22T17:54:20.048 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v6Pi5aMs0HCLFtR off=19063094 len=771267 2022-04-22T17:54:20.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 2273 completed, throughput=4.99MB/sec 2022-04-22T17:54:20.050 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ynZFsPE7aPFlI1w off=7428895 len=2084661 2022-04-22T17:54:20.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 2274 completed, throughput=5MB/sec 2022-04-22T17:54:20.055 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S2JB-h4PBLT7P_7 off=2036315 len=153860 2022-04-22T17:54:20.059 INFO:tasks.workunit.client.0.smithi115.stdout:op 2275 completed, throughput=5MB/sec 2022-04-22T17:54:20.059 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t392oZ4zQw9IZI0 off=3525403 len=837952 2022-04-22T17:54:20.065 INFO:tasks.workunit.client.0.smithi115.stdout:op 2276 completed, throughput=5MB/sec 2022-04-22T17:54:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:20 smithi121 bash[36902]: audit 2022-04-22T17:54:20.048780+0000 mon.a (mon.0) 2465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:20 smithi121 bash[36902]: audit 2022-04-22T17:54:20.382292+0000 mon.a (mon.0) 2466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:20 smithi115 bash[45495]: audit 2022-04-22T17:54:20.048780+0000 mon.a (mon.0) 2465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:20 smithi115 bash[45495]: audit 2022-04-22T17:54:20.382292+0000 mon.a (mon.0) 2466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:20 smithi115 bash[46934]: audit 2022-04-22T17:54:20.048780+0000 mon.a (mon.0) 2465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:20 smithi115 bash[46934]: audit 2022-04-22T17:54:20.382292+0000 mon.a (mon.0) 2466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:21.065 INFO:tasks.workunit.client.0.smithi115.stdout: 446: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:21.065 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XW7NouyPvvotpWn off=1960942 len=1663710 2022-04-22T17:54:21.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 2277 completed, throughput=4.99MB/sec 2022-04-22T17:54:21.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dtpv6SJIQYx4grT off=21950825 len=622534 2022-04-22T17:54:21.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 2278 completed, throughput=4.99MB/sec 2022-04-22T17:54:21.081 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2rKHyDJPIyNE9HL off=14635860 len=1603132 2022-04-22T17:54:21.088 INFO:tasks.workunit.client.0.smithi115.stdout:op 2279 completed, throughput=5MB/sec 2022-04-22T17:54:21.088 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dXk5Aj5yiNoafRp off=3668768 len=1080840 2022-04-22T17:54:21.093 INFO:tasks.workunit.client.0.smithi115.stdout:op 2280 completed, throughput=5MB/sec 2022-04-22T17:54:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:21 smithi121 bash[36902]: cluster 2022-04-22T17:54:20.486086+0000 mgr.y (mgr.34107) 1017 : cluster [DBG] pgmap v1629: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:21 smithi115 bash[45495]: cluster 2022-04-22T17:54:20.486086+0000 mgr.y (mgr.34107) 1017 : cluster [DBG] pgmap v1629: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:21 smithi115 bash[46934]: cluster 2022-04-22T17:54:20.486086+0000 mgr.y (mgr.34107) 1017 : cluster [DBG] pgmap v1629: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:21.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:22.093 INFO:tasks.workunit.client.0.smithi115.stdout: 447: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:22.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yY6A_PKsEfKlbc7 off=13797725 len=1109273 2022-04-22T17:54:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 2281 completed, throughput=4.99MB/sec 2022-04-22T17:54:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SIZir2WNhh9hU5i off=5180609 len=269341 2022-04-22T17:54:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 2282 completed, throughput=4.99MB/sec 2022-04-22T17:54:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v2QQYo7Eol-2yOC off=3652554 len=484066 2022-04-22T17:54:22.105 INFO:tasks.workunit.client.0.smithi115.stdout:op 2283 completed, throughput=4.99MB/sec 2022-04-22T17:54:22.105 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tDjD3Z9fS8LupL3 off=2864698 len=1120566 2022-04-22T17:54:22.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 2284 completed, throughput=5MB/sec 2022-04-22T17:54:22.108 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7zlgNqSyPK7wpMy off=3489093 len=579520 2022-04-22T17:54:22.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 2285 completed, throughput=5MB/sec 2022-04-22T17:54:22.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SDSptynCXNaIK_n off=7481318 len=851315 2022-04-22T17:54:22.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 2286 completed, throughput=5MB/sec 2022-04-22T17:54:22.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lsuwOxmoGyORZ3w off=4476711 len=1628486 2022-04-22T17:54:22.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 2287 completed, throughput=5MB/sec 2022-04-22T17:54:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:22 smithi121 bash[36902]: audit 2022-04-22T17:54:22.429987+0000 mon.a (mon.0) 2467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:22 smithi115 bash[45495]: audit 2022-04-22T17:54:22.429987+0000 mon.a (mon.0) 2467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:22 smithi115 bash[46934]: audit 2022-04-22T17:54:22.429987+0000 mon.a (mon.0) 2467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:23.119 INFO:tasks.workunit.client.0.smithi115.stdout: 448: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:23.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KbdMvk8ljBjGJ61 off=8393664 len=1519784 2022-04-22T17:54:23.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 2288 completed, throughput=4.99MB/sec 2022-04-22T17:54:23.128 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JIvRhPLRRBB1dPY off=13473062 len=1565713 2022-04-22T17:54:23.133 INFO:tasks.workunit.client.0.smithi115.stdout:op 2289 completed, throughput=5MB/sec 2022-04-22T17:54:23.133 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--YpglVihh8OFOrS off=5568680 len=242893 2022-04-22T17:54:23.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 2290 completed, throughput=5MB/sec 2022-04-22T17:54:23.136 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UpPP8_88ue4oUP0 off=20724529 len=1911247 2022-04-22T17:54:23.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 2291 completed, throughput=5MB/sec 2022-04-22T17:54:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:23 smithi121 bash[36902]: cluster 2022-04-22T17:54:22.487327+0000 mgr.y (mgr.34107) 1018 : cluster [DBG] pgmap v1630: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:54:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:23 smithi115 bash[45495]: cluster 2022-04-22T17:54:22.487327+0000 mgr.y (mgr.34107) 1018 : cluster [DBG] pgmap v1630: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:54:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:23 smithi115 bash[46934]: cluster 2022-04-22T17:54:22.487327+0000 mgr.y (mgr.34107) 1018 : cluster [DBG] pgmap v1630: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:54:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:23.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:24.140 INFO:tasks.workunit.client.0.smithi115.stdout: 449: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:24.141 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xhj4ecMX_6e4tMs off=2283079 len=326505 2022-04-22T17:54:24.141 INFO:tasks.workunit.client.0.smithi115.stdout:op 2292 completed, throughput=4.99MB/sec 2022-04-22T17:54:24.142 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eZVaFN6wp5G7gbz off=1890877 len=1937599 2022-04-22T17:54:24.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 2293 completed, throughput=5MB/sec 2022-04-22T17:54:24.151 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b567kkdK4tqSbNE off=7040299 len=1080169 2022-04-22T17:54:24.158 INFO:tasks.workunit.client.0.smithi115.stdout:op 2294 completed, throughput=5MB/sec 2022-04-22T17:54:24.158 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I-h_4gxGMrXRXYx off=4672386 len=443562 2022-04-22T17:54:24.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 2295 completed, throughput=5MB/sec 2022-04-22T17:54:24.160 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sd4DIIq8EeD_ybr off=14738495 len=721992 2022-04-22T17:54:24.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 2296 completed, throughput=5MB/sec 2022-04-22T17:54:24.166 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-45ZLB_OZ_SAT8mf off=1941658 len=1178361 2022-04-22T17:54:24.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 2297 completed, throughput=5MB/sec 2022-04-22T17:54:25.168 INFO:tasks.workunit.client.0.smithi115.stdout: 450: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:25.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2y8etj0rcsmNfGO off=2431482 len=1155740 2022-04-22T17:54:25.172 INFO:tasks.workunit.client.0.smithi115.stdout:op 2298 completed, throughput=4.99MB/sec 2022-04-22T17:54:25.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3nqTJP6D0xVuKEN off=11220424 len=1090959 2022-04-22T17:54:25.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 2299 completed, throughput=5MB/sec 2022-04-22T17:54:25.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1vkbsrkTUunFwdu off=567339 len=1594030 2022-04-22T17:54:25.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 2300 completed, throughput=5MB/sec 2022-04-22T17:54:25.179 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5LeCvEb8mUd-dq5 off=4084597 len=1098942 2022-04-22T17:54:25.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 2301 completed, throughput=5MB/sec 2022-04-22T17:54:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:25 smithi121 bash[36902]: cluster 2022-04-22T17:54:24.488493+0000 mgr.y (mgr.34107) 1019 : cluster [DBG] pgmap v1631: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:54:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:25 smithi121 bash[36902]: audit 2022-04-22T17:54:25.051492+0000 mon.a (mon.0) 2468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:25 smithi121 bash[36902]: audit 2022-04-22T17:54:25.385574+0000 mon.a (mon.0) 2469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[45495]: cluster 2022-04-22T17:54:24.488493+0000 mgr.y (mgr.34107) 1019 : cluster [DBG] pgmap v1631: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:54:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[45495]: audit 2022-04-22T17:54:25.051492+0000 mon.a (mon.0) 2468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[45495]: audit 2022-04-22T17:54:25.385574+0000 mon.a (mon.0) 2469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:26.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[46934]: cluster 2022-04-22T17:54:24.488493+0000 mgr.y (mgr.34107) 1019 : cluster [DBG] pgmap v1631: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:54:26.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[46934]: audit 2022-04-22T17:54:25.051492+0000 mon.a (mon.0) 2468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:26.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:25 smithi115 bash[46934]: audit 2022-04-22T17:54:25.385574+0000 mon.a (mon.0) 2469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:26.184 INFO:tasks.workunit.client.0.smithi115.stdout: 451: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:26.184 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6v9yxH50XBH_6YA off=1558998 len=1167592 2022-04-22T17:54:26.188 INFO:tasks.workunit.client.0.smithi115.stdout:op 2302 completed, throughput=4.99MB/sec 2022-04-22T17:54:26.188 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s6hb-rD1NbAag_p off=2675940 len=1145842 2022-04-22T17:54:26.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 2303 completed, throughput=5MB/sec 2022-04-22T17:54:26.196 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ezaK_4RvbCip6nD off=6737126 len=602518 2022-04-22T17:54:26.201 INFO:tasks.workunit.client.0.smithi115.stdout:op 2304 completed, throughput=5MB/sec 2022-04-22T17:54:26.201 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_cRoTxrzRwpywhK off=7930996 len=891917 2022-04-22T17:54:26.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 2305 completed, throughput=5MB/sec 2022-04-22T17:54:26.203 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lUtvi8blUcqOzZp off=6417168 len=1032028 2022-04-22T17:54:26.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 2306 completed, throughput=5MB/sec 2022-04-22T17:54:27.210 INFO:tasks.workunit.client.0.smithi115.stdout: 452: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:27.211 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d5tt-DrE9KwByQO off=3694627 len=427052 2022-04-22T17:54:27.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 2307 completed, throughput=4.99MB/sec 2022-04-22T17:54:27.212 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1DbcZB95w6Q6Pq9 off=1675073 len=669026 2022-04-22T17:54:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 2308 completed, throughput=4.99MB/sec 2022-04-22T17:54:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dL1kkBdzaZZiIDI off=11578380 len=1866535 2022-04-22T17:54:27.224 INFO:tasks.workunit.client.0.smithi115.stdout:op 2309 completed, throughput=5MB/sec 2022-04-22T17:54:27.225 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fTuheQKDgH_5VtL off=4807663 len=1345716 2022-04-22T17:54:27.226 INFO:tasks.workunit.client.0.smithi115.stdout:op 2310 completed, throughput=5MB/sec 2022-04-22T17:54:27.227 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RHae6JaV4Yzz8FG off=1481915 len=1427266 2022-04-22T17:54:27.230 INFO:tasks.workunit.client.0.smithi115.stdout:op 2311 completed, throughput=5MB/sec 2022-04-22T17:54:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:27 smithi121 bash[36902]: cluster 2022-04-22T17:54:26.489583+0000 mgr.y (mgr.34107) 1020 : cluster [DBG] pgmap v1632: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:27 smithi121 bash[36902]: audit 2022-04-22T17:54:27.432998+0000 mon.a (mon.0) 2470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:27 smithi115 bash[45495]: cluster 2022-04-22T17:54:26.489583+0000 mgr.y (mgr.34107) 1020 : cluster [DBG] pgmap v1632: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:27 smithi115 bash[45495]: audit 2022-04-22T17:54:27.432998+0000 mon.a (mon.0) 2470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:28.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:27 smithi115 bash[46934]: cluster 2022-04-22T17:54:26.489583+0000 mgr.y (mgr.34107) 1020 : cluster [DBG] pgmap v1632: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:27 smithi115 bash[46934]: audit 2022-04-22T17:54:27.432998+0000 mon.a (mon.0) 2470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:28.229 INFO:tasks.workunit.client.0.smithi115.stdout: 453: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:28.230 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wXZ69R9kW7OU2Pf off=11130276 len=1429299 2022-04-22T17:54:28.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 2312 completed, throughput=4.99MB/sec 2022-04-22T17:54:28.233 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eSlBS2iZ32YLYq6 off=3169888 len=1104462 2022-04-22T17:54:28.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 2313 completed, throughput=5MB/sec 2022-04-22T17:54:28.241 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mI_RcCdRbtDJeYH off=16192359 len=803697 2022-04-22T17:54:28.248 INFO:tasks.workunit.client.0.smithi115.stdout:op 2314 completed, throughput=5MB/sec 2022-04-22T17:54:28.248 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s-g0Q3dwGnTY67t off=19058674 len=983524 2022-04-22T17:54:28.249 INFO:tasks.workunit.client.0.smithi115.stdout:op 2315 completed, throughput=5MB/sec 2022-04-22T17:54:28.249 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4qZyo86VSDLWjFL off=233055 len=1639117 2022-04-22T17:54:28.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 2316 completed, throughput=5MB/sec 2022-04-22T17:54:29.258 INFO:tasks.workunit.client.0.smithi115.stdout: 454: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:29.258 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-azr8txY8Snxn-PC off=6393739 len=1765122 2022-04-22T17:54:29.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 2317 completed, throughput=4.99MB/sec 2022-04-22T17:54:29.264 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Uyedr7_DRfr1l7S off=2334192 len=596543 2022-04-22T17:54:29.269 INFO:tasks.workunit.client.0.smithi115.stdout:op 2318 completed, throughput=5MB/sec 2022-04-22T17:54:29.269 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fq7TuuohEStDEJ1 off=6815173 len=1575778 2022-04-22T17:54:29.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2319 completed, throughput=5MB/sec 2022-04-22T17:54:29.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MWzrsarfcpYOIN7 off=9341820 len=5759 2022-04-22T17:54:29.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 2320 completed, throughput=5MB/sec 2022-04-22T17:54:29.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RBR0RDoz5R9z5GX off=1368104 len=1404789 2022-04-22T17:54:29.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 2321 completed, throughput=5MB/sec 2022-04-22T17:54:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:29 smithi121 bash[36902]: cluster 2022-04-22T17:54:28.490811+0000 mgr.y (mgr.34107) 1021 : cluster [DBG] pgmap v1633: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:29 smithi115 bash[45495]: cluster 2022-04-22T17:54:28.490811+0000 mgr.y (mgr.34107) 1021 : cluster [DBG] pgmap v1633: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:30.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:29 smithi115 bash[46934]: cluster 2022-04-22T17:54:28.490811+0000 mgr.y (mgr.34107) 1021 : cluster [DBG] pgmap v1633: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:30.282 INFO:tasks.workunit.client.0.smithi115.stdout: 455: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:30.283 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C3lTI0JJYQBi7Mt off=2519389 len=638825 2022-04-22T17:54:30.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 2322 completed, throughput=4.99MB/sec 2022-04-22T17:54:30.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dnoGZ1j-ddz7tTo off=12501218 len=1422311 2022-04-22T17:54:30.289 INFO:tasks.workunit.client.0.smithi115.stdout:op 2323 completed, throughput=5MB/sec 2022-04-22T17:54:30.289 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0Qikv01bKVGdunn off=6275889 len=1351672 2022-04-22T17:54:30.297 INFO:tasks.workunit.client.0.smithi115.stdout:op 2324 completed, throughput=5MB/sec 2022-04-22T17:54:30.297 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6Er6LrY7iFk4qyH off=10331691 len=359114 2022-04-22T17:54:30.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 2325 completed, throughput=5MB/sec 2022-04-22T17:54:30.298 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7r9Fx84YZWF5AwW off=5656814 len=13676 2022-04-22T17:54:30.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 2326 completed, throughput=5MB/sec 2022-04-22T17:54:30.302 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vpp28L1W0e1vgAJ off=12921828 len=1842695 2022-04-22T17:54:30.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 2327 completed, throughput=5MB/sec 2022-04-22T17:54:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:30 smithi121 bash[36902]: audit 2022-04-22T17:54:30.054618+0000 mon.a (mon.0) 2471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:30 smithi121 bash[36902]: audit 2022-04-22T17:54:30.388973+0000 mon.a (mon.0) 2472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:30 smithi115 bash[45495]: audit 2022-04-22T17:54:30.054618+0000 mon.a (mon.0) 2471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:30 smithi115 bash[45495]: audit 2022-04-22T17:54:30.388973+0000 mon.a (mon.0) 2472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:30 smithi115 bash[46934]: audit 2022-04-22T17:54:30.054618+0000 mon.a (mon.0) 2471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:30 smithi115 bash[46934]: audit 2022-04-22T17:54:30.388973+0000 mon.a (mon.0) 2472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:31.311 INFO:tasks.workunit.client.0.smithi115.stdout: 456: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:31.311 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-34sO2o9pqtQruF6 off=1900069 len=182364 2022-04-22T17:54:31.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 2328 completed, throughput=4.99MB/sec 2022-04-22T17:54:31.316 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_DSzTCgrmsUxgl5 off=13319523 len=1876939 2022-04-22T17:54:31.324 INFO:tasks.workunit.client.0.smithi115.stdout:op 2329 completed, throughput=5MB/sec 2022-04-22T17:54:31.324 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-arMkNSWKxoCpwwB off=15799365 len=229956 2022-04-22T17:54:31.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 2330 completed, throughput=5MB/sec 2022-04-22T17:54:31.325 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zXSaMXzGmk39PCd off=6064079 len=418025 2022-04-22T17:54:31.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 2331 completed, throughput=5MB/sec 2022-04-22T17:54:31.330 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TeSphhlBRQXNjS_ off=283815 len=968816 2022-04-22T17:54:31.332 INFO:tasks.workunit.client.0.smithi115.stdout:op 2332 completed, throughput=5MB/sec 2022-04-22T17:54:31.332 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E0sI18AzsDKaC3w off=1380699 len=571566 2022-04-22T17:54:31.334 INFO:tasks.workunit.client.0.smithi115.stdout:op 2333 completed, throughput=5MB/sec 2022-04-22T17:54:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:31 smithi121 bash[36902]: cluster 2022-04-22T17:54:30.491895+0000 mgr.y (mgr.34107) 1022 : cluster [DBG] pgmap v1634: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:31 smithi115 bash[45495]: cluster 2022-04-22T17:54:30.491895+0000 mgr.y (mgr.34107) 1022 : cluster [DBG] pgmap v1634: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:31 smithi115 bash[46934]: cluster 2022-04-22T17:54:30.491895+0000 mgr.y (mgr.34107) 1022 : cluster [DBG] pgmap v1634: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:54:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:31.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:32.335 INFO:tasks.workunit.client.0.smithi115.stdout: 457: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:32.335 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IypgJ-5YRBC7CUQ off=12190350 len=945195 2022-04-22T17:54:32.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 2334 completed, throughput=4.99MB/sec 2022-04-22T17:54:32.338 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H_eR7LZQmkbDsra off=206859 len=2054130 2022-04-22T17:54:32.350 INFO:tasks.workunit.client.0.smithi115.stdout:op 2335 completed, throughput=5MB/sec 2022-04-22T17:54:32.350 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rdWaXp5O6Hp84fx off=8334383 len=1901845 2022-04-22T17:54:32.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 2336 completed, throughput=5MB/sec 2022-04-22T17:54:32.353 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3hvMkfZaqj3Iwud off=7397861 len=1394730 2022-04-22T17:54:32.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 2337 completed, throughput=5MB/sec 2022-04-22T17:54:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:32 smithi121 bash[36902]: audit 2022-04-22T17:54:32.436531+0000 mon.a (mon.0) 2473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:32 smithi115 bash[45495]: audit 2022-04-22T17:54:32.436531+0000 mon.a (mon.0) 2473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:32 smithi115 bash[46934]: audit 2022-04-22T17:54:32.436531+0000 mon.a (mon.0) 2473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:33.361 INFO:tasks.workunit.client.0.smithi115.stdout: 458: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:33.361 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nER1j1abN65Y8-E off=0 len=1631982 2022-04-22T17:54:33.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 2338 completed, throughput=4.99MB/sec 2022-04-22T17:54:33.372 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VBQB3QjmDpscbyk off=14843427 len=1465286 2022-04-22T17:54:33.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 2339 completed, throughput=5MB/sec 2022-04-22T17:54:33.375 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cfJYLXsmfPdfPpi off=7828149 len=1808354 2022-04-22T17:54:33.385 INFO:tasks.workunit.client.0.smithi115.stdout:op 2340 completed, throughput=5MB/sec 2022-04-22T17:54:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:33 smithi121 bash[36902]: cluster 2022-04-22T17:54:32.492968+0000 mgr.y (mgr.34107) 1023 : cluster [DBG] pgmap v1635: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:33 smithi115 bash[45495]: cluster 2022-04-22T17:54:32.492968+0000 mgr.y (mgr.34107) 1023 : cluster [DBG] pgmap v1635: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:33.581Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:34.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:33 smithi115 bash[46934]: cluster 2022-04-22T17:54:32.492968+0000 mgr.y (mgr.34107) 1023 : cluster [DBG] pgmap v1635: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:54:34.386 INFO:tasks.workunit.client.0.smithi115.stdout: 459: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:34.386 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j8M5M3-YBAMPChK off=12322388 len=1927053 2022-04-22T17:54:34.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 2341 completed, throughput=4.99MB/sec 2022-04-22T17:54:34.390 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ef6fakPGEDfG4aV off=9062421 len=1550819 2022-04-22T17:54:34.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 2342 completed, throughput=5MB/sec 2022-04-22T17:54:34.395 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Mkr22QP_ec_RKH8 off=210151 len=1166082 2022-04-22T17:54:34.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 2343 completed, throughput=5MB/sec 2022-04-22T17:54:34.399 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xpFoClnJdGTZzQB off=1767292 len=1630652 2022-04-22T17:54:34.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 2344 completed, throughput=5MB/sec 2022-04-22T17:54:35.401 INFO:tasks.workunit.client.0.smithi115.stdout: 460: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:35.401 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mV8fDu8wWD7JpI0 off=2555123 len=574218 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 2345 completed, throughput=4.99MB/sec 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sEYsvz63Hzekr_t off=110148 len=939737 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 2346 completed, throughput=5MB/sec 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dQ6qxzh4V6GUYKZ off=3400698 len=665922 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 2347 completed, throughput=5MB/sec 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l556MWsoPSGQTkW off=13919754 len=1979257 2022-04-22T17:54:35.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 2348 completed, throughput=5MB/sec 2022-04-22T17:54:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:35 smithi121 bash[36902]: cluster 2022-04-22T17:54:34.493946+0000 mgr.y (mgr.34107) 1024 : cluster [DBG] pgmap v1636: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:54:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:35 smithi121 bash[36902]: audit 2022-04-22T17:54:35.057743+0000 mon.a (mon.0) 2474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:35 smithi121 bash[36902]: audit 2022-04-22T17:54:35.392674+0000 mon.a (mon.0) 2475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[45495]: cluster 2022-04-22T17:54:34.493946+0000 mgr.y (mgr.34107) 1024 : cluster [DBG] pgmap v1636: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:54:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[45495]: audit 2022-04-22T17:54:35.057743+0000 mon.a (mon.0) 2474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[45495]: audit 2022-04-22T17:54:35.392674+0000 mon.a (mon.0) 2475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[46934]: cluster 2022-04-22T17:54:34.493946+0000 mgr.y (mgr.34107) 1024 : cluster [DBG] pgmap v1636: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:54:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[46934]: audit 2022-04-22T17:54:35.057743+0000 mon.a (mon.0) 2474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:35 smithi115 bash[46934]: audit 2022-04-22T17:54:35.392674+0000 mon.a (mon.0) 2475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:36.413 INFO:tasks.workunit.client.0.smithi115.stdout: 461: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:36.413 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lv83yiTNgQSKojn off=3838586 len=312002 2022-04-22T17:54:36.418 INFO:tasks.workunit.client.0.smithi115.stdout:op 2349 completed, throughput=4.99MB/sec 2022-04-22T17:54:36.418 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tpplac2MsJEnSCU off=18650254 len=1758397 2022-04-22T17:54:36.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 2350 completed, throughput=4.99MB/sec 2022-04-22T17:54:36.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bx7860KhfTThHvK off=4233990 len=128194 2022-04-22T17:54:36.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 2351 completed, throughput=4.99MB/sec 2022-04-22T17:54:36.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rC4aSWfrOOOZgB2 off=691060 len=487922 2022-04-22T17:54:36.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 2352 completed, throughput=5MB/sec 2022-04-22T17:54:36.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3J2etMdF0xvtvQ off=7146949 len=1702294 2022-04-22T17:54:36.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 2353 completed, throughput=5MB/sec 2022-04-22T17:54:36.447 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QaRhYLjCHmwXsBm off=7028531 len=178624 2022-04-22T17:54:36.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 2354 completed, throughput=5MB/sec 2022-04-22T17:54:36.450 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I7jlQXzeUwtoCFP off=16319337 len=1374541 2022-04-22T17:54:36.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 2355 completed, throughput=5MB/sec 2022-04-22T17:54:37.452 INFO:tasks.workunit.client.0.smithi115.stdout: 462: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:37.452 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_mtXwFThZwKntdp off=3119720 len=74908 2022-04-22T17:54:37.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 2356 completed, throughput=4.99MB/sec 2022-04-22T17:54:37.456 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sZxl1tuqzkL7vRv off=19998309 len=620046 2022-04-22T17:54:37.460 INFO:tasks.workunit.client.0.smithi115.stdout:op 2357 completed, throughput=4.99MB/sec 2022-04-22T17:54:37.460 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MK3NOpz48f1WwRH off=5878715 len=2044571 2022-04-22T17:54:37.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 2358 completed, throughput=5MB/sec 2022-04-22T17:54:37.464 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SByA9WkIeTxd3Fs off=2991300 len=927520 2022-04-22T17:54:37.471 INFO:tasks.workunit.client.0.smithi115.stdout:op 2359 completed, throughput=5MB/sec 2022-04-22T17:54:37.471 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-67Rt9e7EJVoZC46 off=14575326 len=1416219 2022-04-22T17:54:37.473 INFO:tasks.workunit.client.0.smithi115.stdout:op 2360 completed, throughput=5MB/sec 2022-04-22T17:54:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:37 smithi115 bash[45495]: cluster 2022-04-22T17:54:36.495067+0000 mgr.y (mgr.34107) 1025 : cluster [DBG] pgmap v1637: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:37 smithi115 bash[45495]: audit 2022-04-22T17:54:37.439684+0000 mon.a (mon.0) 2476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:37 smithi115 bash[46934]: cluster 2022-04-22T17:54:36.495067+0000 mgr.y (mgr.34107) 1025 : cluster [DBG] pgmap v1637: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:37 smithi115 bash[46934]: audit 2022-04-22T17:54:37.439684+0000 mon.a (mon.0) 2476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:37 smithi121 bash[36902]: cluster 2022-04-22T17:54:36.495067+0000 mgr.y (mgr.34107) 1025 : cluster [DBG] pgmap v1637: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:38.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:37 smithi121 bash[36902]: audit 2022-04-22T17:54:37.439684+0000 mon.a (mon.0) 2476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:38.473 INFO:tasks.workunit.client.0.smithi115.stdout: 463: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:38.474 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-huzosWtQTHZYD6q off=21579157 len=1576721 2022-04-22T17:54:38.483 INFO:tasks.workunit.client.0.smithi115.stdout:op 2361 completed, throughput=4.99MB/sec 2022-04-22T17:54:38.483 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZXOCv6TWabm2Lx6 off=1149732 len=957457 2022-04-22T17:54:38.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 2362 completed, throughput=5MB/sec 2022-04-22T17:54:38.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lj6mPlaaSeps3XM off=5890844 len=709200 2022-04-22T17:54:38.494 INFO:tasks.workunit.client.0.smithi115.stdout:op 2363 completed, throughput=5MB/sec 2022-04-22T17:54:38.494 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4SLlvZ56b6fZRyt off=12136902 len=1774123 2022-04-22T17:54:38.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 2364 completed, throughput=5MB/sec 2022-04-22T17:54:38.630 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:54:38.976 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:54:38.976 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (17m) 7m ago 31m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (31m) 8m ago 31m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (19m) 7m ago 32m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (19m) 8m ago 32m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (26m) 8m ago 36m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (27m) 7m ago 38m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (26m) 7m ago 39m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (25m) 8m ago 37m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (26m) 7m ago 37m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (18m) 7m ago 31m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:54:38.977 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (18m) 8m ago 31m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (24m) 7m ago 36m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (23m) 7m ago 35m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (23m) 7m ago 35m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (22m) 7m ago 34m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (22m) 8m ago 34m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (21m) 8m ago 34m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (20m) 8m ago 33m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (20m) 8m ago 33m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:54:38.978 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (18m) 8m ago 31m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:54:39.378 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:54:39.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:54:39.380 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:54:39.497 INFO:tasks.workunit.client.0.smithi115.stdout: 464: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:39.497 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-urTyIUpLEyTuX8w off=15614738 len=1977513 2022-04-22T17:54:39.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2365 completed, throughput=4.99MB/sec 2022-04-22T17:54:39.503 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OidBsMkocOiAyFP off=284873 len=704407 2022-04-22T17:54:39.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 2366 completed, throughput=5MB/sec 2022-04-22T17:54:39.508 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TwylL_AQMq3ru-Y off=7226592 len=925816 2022-04-22T17:54:39.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 2367 completed, throughput=5MB/sec 2022-04-22T17:54:39.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yYyE_nGWRJrvtcj off=19125038 len=1503475 2022-04-22T17:54:39.514 INFO:tasks.workunit.client.0.smithi115.stdout:op 2368 completed, throughput=5MB/sec 2022-04-22T17:54:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: cluster 2022-04-22T17:54:38.495796+0000 mgr.y (mgr.34107) 1026 : cluster [DBG] pgmap v1638: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:38.606681+0000 mgr.y (mgr.34107) 1027 : audit [DBG] from='client.35484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:38.954933+0000 mgr.y (mgr.34107) 1028 : audit [DBG] from='client.45427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:39.371230+0000 mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.115:0/2271038627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:39.500510+0000 mon.a (mon.0) 2477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:39.503229+0000 mon.a (mon.0) 2478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[46934]: audit 2022-04-22T17:54:39.504533+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: cluster 2022-04-22T17:54:38.495796+0000 mgr.y (mgr.34107) 1026 : cluster [DBG] pgmap v1638: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:38.606681+0000 mgr.y (mgr.34107) 1027 : audit [DBG] from='client.35484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:38.954933+0000 mgr.y (mgr.34107) 1028 : audit [DBG] from='client.45427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:39.371230+0000 mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.115:0/2271038627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:39.500510+0000 mon.a (mon.0) 2477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:39.503229+0000 mon.a (mon.0) 2478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:39 smithi115 bash[45495]: audit 2022-04-22T17:54:39.504533+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: cluster 2022-04-22T17:54:38.495796+0000 mgr.y (mgr.34107) 1026 : cluster [DBG] pgmap v1638: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:38.606681+0000 mgr.y (mgr.34107) 1027 : audit [DBG] from='client.35484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:38.954933+0000 mgr.y (mgr.34107) 1028 : audit [DBG] from='client.45427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:39.371230+0000 mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.115:0/2271038627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:39.500510+0000 mon.a (mon.0) 2477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:39.503229+0000 mon.a (mon.0) 2478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:54:39.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:39 smithi121 bash[36902]: audit 2022-04-22T17:54:39.504533+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:54:40.515 INFO:tasks.workunit.client.0.smithi115.stdout: 465: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:40.515 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QX4PN6JUSmy9Vem off=7561367 len=601528 2022-04-22T17:54:40.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 2369 completed, throughput=4.99MB/sec 2022-04-22T17:54:40.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0fnNTzhKgeuXR6l off=3571260 len=1146177 2022-04-22T17:54:40.521 INFO:tasks.workunit.client.0.smithi115.stdout:op 2370 completed, throughput=4.99MB/sec 2022-04-22T17:54:40.521 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b-BOR6sX4l9_stb off=2092766 len=2062693 2022-04-22T17:54:40.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 2371 completed, throughput=5MB/sec 2022-04-22T17:54:40.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HEGg22NmxfUkavf off=2745750 len=1032087 2022-04-22T17:54:40.529 INFO:tasks.workunit.client.0.smithi115.stdout:op 2372 completed, throughput=5MB/sec 2022-04-22T17:54:40.529 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g5y6WqlhO3_l9jb off=5155308 len=1204109 2022-04-22T17:54:40.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 2373 completed, throughput=5MB/sec 2022-04-22T17:54:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:40 smithi121 bash[36902]: audit 2022-04-22T17:54:39.847624+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:54:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:40 smithi121 bash[36902]: audit 2022-04-22T17:54:40.060700+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:40 smithi121 bash[36902]: audit 2022-04-22T17:54:40.396476+0000 mon.a (mon.0) 2482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[45495]: audit 2022-04-22T17:54:39.847624+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[45495]: audit 2022-04-22T17:54:40.060700+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[45495]: audit 2022-04-22T17:54:40.396476+0000 mon.a (mon.0) 2482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[46934]: audit 2022-04-22T17:54:39.847624+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[46934]: audit 2022-04-22T17:54:40.060700+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:40 smithi115 bash[46934]: audit 2022-04-22T17:54:40.396476+0000 mon.a (mon.0) 2482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:41.533 INFO:tasks.workunit.client.0.smithi115.stdout: 466: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:41.533 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-riCEw6JiniojbS5 off=2180330 len=1696901 2022-04-22T17:54:41.542 INFO:tasks.workunit.client.0.smithi115.stdout:op 2374 completed, throughput=4.99MB/sec 2022-04-22T17:54:41.543 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zhb2tPjFmz-UlCu off=3319360 len=1736716 2022-04-22T17:54:41.546 INFO:tasks.workunit.client.0.smithi115.stdout:op 2375 completed, throughput=5MB/sec 2022-04-22T17:54:41.546 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-44qrnukD1U8LZsY off=5497014 len=1294534 2022-04-22T17:54:41.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 2376 completed, throughput=5MB/sec 2022-04-22T17:54:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:41 smithi115 bash[45495]: cluster 2022-04-22T17:54:40.497035+0000 mgr.y (mgr.34107) 1029 : cluster [DBG] pgmap v1639: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:54:42.077 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:41.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:42.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:41 smithi115 bash[46934]: cluster 2022-04-22T17:54:40.497035+0000 mgr.y (mgr.34107) 1029 : cluster [DBG] pgmap v1639: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:54:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:41 smithi121 bash[36902]: cluster 2022-04-22T17:54:40.497035+0000 mgr.y (mgr.34107) 1029 : cluster [DBG] pgmap v1639: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:54:42.553 INFO:tasks.workunit.client.0.smithi115.stdout: 467: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:42.554 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mBpd-fPcq00zN0f off=3862905 len=1608503 2022-04-22T17:54:42.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 2377 completed, throughput=4.99MB/sec 2022-04-22T17:54:42.557 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vka1Fongd7LLuTP off=12415059 len=2084148 2022-04-22T17:54:42.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 2378 completed, throughput=5MB/sec 2022-04-22T17:54:42.565 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-upVZEaHEbErMVU_ off=5543996 len=257784 2022-04-22T17:54:42.569 INFO:tasks.workunit.client.0.smithi115.stdout:op 2379 completed, throughput=5MB/sec 2022-04-22T17:54:42.570 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yQPutB1auJAgaNG off=15920091 len=956425 2022-04-22T17:54:42.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 2380 completed, throughput=5MB/sec 2022-04-22T17:54:42.576 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fr5pvonA3-wlGxE off=610758 len=781434 2022-04-22T17:54:42.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 2381 completed, throughput=5MB/sec 2022-04-22T17:54:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:42 smithi121 bash[36902]: audit 2022-04-22T17:54:42.443303+0000 mon.a (mon.0) 2483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:43.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:42 smithi115 bash[45495]: audit 2022-04-22T17:54:42.443303+0000 mon.a (mon.0) 2483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:43.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:42 smithi115 bash[46934]: audit 2022-04-22T17:54:42.443303+0000 mon.a (mon.0) 2483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:43.578 INFO:tasks.workunit.client.0.smithi115.stdout: 468: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:43.578 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CsSGxbSHYuqqn9A off=13997669 len=45651 2022-04-22T17:54:43.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 2382 completed, throughput=4.99MB/sec 2022-04-22T17:54:43.579 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vtswGt91b5I4ZUQ off=1067285 len=159522 2022-04-22T17:54:43.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 2383 completed, throughput=4.99MB/sec 2022-04-22T17:54:43.584 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K2M1CKlhRQrh7yx off=330493 len=1183926 2022-04-22T17:54:43.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 2384 completed, throughput=4.99MB/sec 2022-04-22T17:54:43.592 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8iTkWumDSjhhZMs off=3202066 len=655694 2022-04-22T17:54:43.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 2385 completed, throughput=4.99MB/sec 2022-04-22T17:54:43.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s05YESDyOXZbHgG off=1146817 len=1098441 2022-04-22T17:54:43.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 2386 completed, throughput=5MB/sec 2022-04-22T17:54:43.600 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4UWrz-9L3xBpiz_ off=12908461 len=51036 2022-04-22T17:54:43.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 2387 completed, throughput=5MB/sec 2022-04-22T17:54:43.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1HHh_-W2lkhfd82 off=10727102 len=1363416 2022-04-22T17:54:43.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 2388 completed, throughput=5MB/sec 2022-04-22T17:54:43.610 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IM-f7dCgI5zco8F off=5803919 len=1759141 2022-04-22T17:54:43.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 2389 completed, throughput=5MB/sec 2022-04-22T17:54:43.872 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:43.582Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:43 smithi121 bash[36902]: cluster 2022-04-22T17:54:42.498279+0000 mgr.y (mgr.34107) 1030 : cluster [DBG] pgmap v1640: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:43 smithi115 bash[45495]: cluster 2022-04-22T17:54:42.498279+0000 mgr.y (mgr.34107) 1030 : cluster [DBG] pgmap v1640: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:43 smithi115 bash[46934]: cluster 2022-04-22T17:54:42.498279+0000 mgr.y (mgr.34107) 1030 : cluster [DBG] pgmap v1640: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:54:44.614 INFO:tasks.workunit.client.0.smithi115.stdout: 469: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:44.614 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9aAyv3gQ27rYor0 off=1308452 len=525619 2022-04-22T17:54:44.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 2390 completed, throughput=4.99MB/sec 2022-04-22T17:54:44.620 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-niKjv5xmqAbWxH6 off=15568837 len=221989 2022-04-22T17:54:44.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 2391 completed, throughput=4.99MB/sec 2022-04-22T17:54:44.625 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kaAYETlRzEf-tD9 off=10612302 len=773725 2022-04-22T17:54:44.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 2392 completed, throughput=5MB/sec 2022-04-22T17:54:44.629 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i00So9jpCKV_P0j off=2213412 len=995033 2022-04-22T17:54:44.636 INFO:tasks.workunit.client.0.smithi115.stdout:op 2393 completed, throughput=5MB/sec 2022-04-22T17:54:44.636 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oo34G1CaZBWvgBn off=11300624 len=173032 2022-04-22T17:54:44.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 2394 completed, throughput=5MB/sec 2022-04-22T17:54:44.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ffANINQrpU64XzE off=7010878 len=804451 2022-04-22T17:54:44.646 INFO:tasks.workunit.client.0.smithi115.stdout:op 2395 completed, throughput=5MB/sec 2022-04-22T17:54:44.646 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FgR-Axf2LI5ECcp off=6003379 len=2089347 2022-04-22T17:54:44.653 INFO:tasks.workunit.client.0.smithi115.stdout:op 2396 completed, throughput=5MB/sec 2022-04-22T17:54:45.654 INFO:tasks.workunit.client.0.smithi115.stdout: 470: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:45.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sqE87txVdAoyZOA off=10160224 len=363734 2022-04-22T17:54:45.659 INFO:tasks.workunit.client.0.smithi115.stdout:op 2397 completed, throughput=4.99MB/sec 2022-04-22T17:54:45.659 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-grO_tSMjbntVGc0 off=2474314 len=2033808 2022-04-22T17:54:45.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 2398 completed, throughput=5MB/sec 2022-04-22T17:54:45.671 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Mm2ZYWgaq_nffCM off=15258424 len=1247774 2022-04-22T17:54:45.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 2399 completed, throughput=5MB/sec 2022-04-22T17:54:45.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AxxC0KzLaCC3-bE off=17503869 len=898168 2022-04-22T17:54:45.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 2400 completed, throughput=5MB/sec 2022-04-22T17:54:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:45 smithi121 bash[36902]: cluster 2022-04-22T17:54:44.499401+0000 mgr.y (mgr.34107) 1031 : cluster [DBG] pgmap v1641: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:45 smithi121 bash[36902]: audit 2022-04-22T17:54:45.063814+0000 mon.a (mon.0) 2484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:45 smithi121 bash[36902]: audit 2022-04-22T17:54:45.400320+0000 mon.a (mon.0) 2485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:46.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[46934]: cluster 2022-04-22T17:54:44.499401+0000 mgr.y (mgr.34107) 1031 : cluster [DBG] pgmap v1641: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:46.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[46934]: audit 2022-04-22T17:54:45.063814+0000 mon.a (mon.0) 2484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:46.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[46934]: audit 2022-04-22T17:54:45.400320+0000 mon.a (mon.0) 2485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[45495]: cluster 2022-04-22T17:54:44.499401+0000 mgr.y (mgr.34107) 1031 : cluster [DBG] pgmap v1641: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[45495]: audit 2022-04-22T17:54:45.063814+0000 mon.a (mon.0) 2484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:45 smithi115 bash[45495]: audit 2022-04-22T17:54:45.400320+0000 mon.a (mon.0) 2485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:46.679 INFO:tasks.workunit.client.0.smithi115.stdout: 471: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:46.679 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9DJrE2TSNimGZJ0 off=9157210 len=900785 2022-04-22T17:54:46.682 INFO:tasks.workunit.client.0.smithi115.stdout:op 2401 completed, throughput=4.99MB/sec 2022-04-22T17:54:46.682 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p5KrKIfNS9dbmG0 off=4253996 len=127212 2022-04-22T17:54:46.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 2402 completed, throughput=4.99MB/sec 2022-04-22T17:54:46.684 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ardAWGkT3Y2mBx- off=230581 len=45458 2022-04-22T17:54:46.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 2403 completed, throughput=4.99MB/sec 2022-04-22T17:54:46.687 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4Acbj1NYyEma1qk off=279069 len=1009856 2022-04-22T17:54:46.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 2404 completed, throughput=4.99MB/sec 2022-04-22T17:54:46.695 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3QQDniFM6xQ9jwS off=5254245 len=1619926 2022-04-22T17:54:46.702 INFO:tasks.workunit.client.0.smithi115.stdout:op 2405 completed, throughput=5MB/sec 2022-04-22T17:54:46.702 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-psTp0oXYk6g2UUm off=3772621 len=429407 2022-04-22T17:54:46.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 2406 completed, throughput=5MB/sec 2022-04-22T17:54:46.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o4aApSyDyDiz3Sv off=2404961 len=532267 2022-04-22T17:54:46.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 2407 completed, throughput=5MB/sec 2022-04-22T17:54:46.708 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b09eR3VMeOwaHD0 off=9390792 len=2061559 2022-04-22T17:54:46.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 2408 completed, throughput=5MB/sec 2022-04-22T17:54:47.711 INFO:tasks.workunit.client.0.smithi115.stdout: 472: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:47.711 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uTWlNe9RznOr9va off=7329861 len=1221654 2022-04-22T17:54:47.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 2409 completed, throughput=5MB/sec 2022-04-22T17:54:47.714 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E8OazZK1u9-FRaS off=4796839 len=1131067 2022-04-22T17:54:47.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 2410 completed, throughput=5MB/sec 2022-04-22T17:54:47.723 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AB6yzm6avwsLVmn off=1697750 len=1839776 2022-04-22T17:54:47.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 2411 completed, throughput=5MB/sec 2022-04-22T17:54:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:47 smithi121 bash[36902]: cluster 2022-04-22T17:54:46.500442+0000 mgr.y (mgr.34107) 1032 : cluster [DBG] pgmap v1642: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:48.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:47 smithi121 bash[36902]: audit 2022-04-22T17:54:47.446360+0000 mon.a (mon.0) 2486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:48.373 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:47 smithi115 bash[45495]: cluster 2022-04-22T17:54:46.500442+0000 mgr.y (mgr.34107) 1032 : cluster [DBG] pgmap v1642: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:48.374 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:47 smithi115 bash[45495]: audit 2022-04-22T17:54:47.446360+0000 mon.a (mon.0) 2486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:47 smithi115 bash[46934]: cluster 2022-04-22T17:54:46.500442+0000 mgr.y (mgr.34107) 1032 : cluster [DBG] pgmap v1642: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:54:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:47 smithi115 bash[46934]: audit 2022-04-22T17:54:47.446360+0000 mon.a (mon.0) 2486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:48.733 INFO:tasks.workunit.client.0.smithi115.stdout: 473: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:48.733 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GM7PWbOdggjwUEc off=358647 len=1938615 2022-04-22T17:54:48.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 2412 completed, throughput=4.99MB/sec 2022-04-22T17:54:48.738 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4a9dg8SDzU-NfGd off=10809032 len=672202 2022-04-22T17:54:48.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 2413 completed, throughput=5MB/sec 2022-04-22T17:54:48.743 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jQ4ho4y4uwTl-DJ off=9686472 len=775847 2022-04-22T17:54:48.749 INFO:tasks.workunit.client.0.smithi115.stdout:op 2414 completed, throughput=5MB/sec 2022-04-22T17:54:48.750 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XE9cqwduJmsYj5r off=12541633 len=758748 2022-04-22T17:54:48.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 2415 completed, throughput=5MB/sec 2022-04-22T17:54:48.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qDRXE1Z-i2AC74s off=1135363 len=210355 2022-04-22T17:54:48.755 INFO:tasks.workunit.client.0.smithi115.stdout:op 2416 completed, throughput=5MB/sec 2022-04-22T17:54:48.755 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0jjK4yfZPFirLoZ off=11790358 len=488263 2022-04-22T17:54:48.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 2417 completed, throughput=5MB/sec 2022-04-22T17:54:49.759 INFO:tasks.workunit.client.0.smithi115.stdout: 474: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:49.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--efUPiiaJ7AzVKB off=3545171 len=885231 2022-04-22T17:54:49.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 2418 completed, throughput=4.99MB/sec 2022-04-22T17:54:49.762 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4vTa1i6E_b4P4Hg off=3336090 len=1614369 2022-04-22T17:54:49.765 INFO:tasks.workunit.client.0.smithi115.stdout:op 2419 completed, throughput=5MB/sec 2022-04-22T17:54:49.766 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s2rtqyz3yyMTBn0 off=434885 len=1757822 2022-04-22T17:54:49.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 2420 completed, throughput=5MB/sec 2022-04-22T17:54:49.774 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZtBuk3kCJrtZo6j off=203573 len=980411 2022-04-22T17:54:49.780 INFO:tasks.workunit.client.0.smithi115.stdout:op 2421 completed, throughput=5MB/sec 2022-04-22T17:54:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:49 smithi121 bash[36902]: cluster 2022-04-22T17:54:48.501708+0000 mgr.y (mgr.34107) 1033 : cluster [DBG] pgmap v1643: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:49 smithi115 bash[45495]: cluster 2022-04-22T17:54:48.501708+0000 mgr.y (mgr.34107) 1033 : cluster [DBG] pgmap v1643: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:49 smithi115 bash[46934]: cluster 2022-04-22T17:54:48.501708+0000 mgr.y (mgr.34107) 1033 : cluster [DBG] pgmap v1643: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:50.780 INFO:tasks.workunit.client.0.smithi115.stdout: 475: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:50.780 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-azRNJsauIbjVEUs off=12601554 len=1062328 2022-04-22T17:54:50.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 2422 completed, throughput=4.99MB/sec 2022-04-22T17:54:50.783 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--SG1H2kGeiIKo1M off=9392077 len=2075930 2022-04-22T17:54:50.789 INFO:tasks.workunit.client.0.smithi115.stdout:op 2423 completed, throughput=5MB/sec 2022-04-22T17:54:50.805 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fQ2f9DpmWvFM0ns off=18451878 len=1225950 2022-04-22T17:54:50.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 2424 completed, throughput=5MB/sec 2022-04-22T17:54:50.806 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NzSdUI99zJMyX53 off=5471596 len=572353 2022-04-22T17:54:50.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 2425 completed, throughput=5MB/sec 2022-04-22T17:54:50.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2EwTRazBl2VMvXl off=5687587 len=1537650 2022-04-22T17:54:50.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 2426 completed, throughput=5MB/sec 2022-04-22T17:54:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:50 smithi121 bash[36902]: audit 2022-04-22T17:54:50.066913+0000 mon.a (mon.0) 2487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:50 smithi121 bash[36902]: audit 2022-04-22T17:54:50.404093+0000 mon.a (mon.0) 2488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:50 smithi115 bash[45495]: audit 2022-04-22T17:54:50.066913+0000 mon.a (mon.0) 2487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:50 smithi115 bash[45495]: audit 2022-04-22T17:54:50.404093+0000 mon.a (mon.0) 2488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:50 smithi115 bash[46934]: audit 2022-04-22T17:54:50.066913+0000 mon.a (mon.0) 2487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:50 smithi115 bash[46934]: audit 2022-04-22T17:54:50.404093+0000 mon.a (mon.0) 2488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:51.803 INFO:tasks.workunit.client.0.smithi115.stdout: 476: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:51.850 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vHJWeHyHQhR26tr off=5119068 len=1245854 2022-04-22T17:54:51.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 2427 completed, throughput=4.99MB/sec 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2VMZ1OM7V-0_D1f off=1397763 len=1758133 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 2428 completed, throughput=5MB/sec 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NIHbakJVYa3PU1n off=17808193 len=929617 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 2429 completed, throughput=5MB/sec 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2tOcyIHcewUVMfg off=6100851 len=851112 2022-04-22T17:54:51.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 2430 completed, throughput=5MB/sec 2022-04-22T17:54:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:51 smithi115 bash[45495]: cluster 2022-04-22T17:54:50.502779+0000 mgr.y (mgr.34107) 1034 : cluster [DBG] pgmap v1644: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:52.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:51.700Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:51 smithi115 bash[46934]: cluster 2022-04-22T17:54:50.502779+0000 mgr.y (mgr.34107) 1034 : cluster [DBG] pgmap v1644: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:51 smithi121 bash[36902]: cluster 2022-04-22T17:54:50.502779+0000 mgr.y (mgr.34107) 1034 : cluster [DBG] pgmap v1644: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:54:52.828 INFO:tasks.workunit.client.0.smithi115.stdout: 477: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:52.828 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zG2FeluhV-Vmjiy off=6552333 len=1074726 2022-04-22T17:54:52.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 2431 completed, throughput=4.99MB/sec 2022-04-22T17:54:52.836 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TsmKtVAYuEZrvG7 off=4206557 len=1137705 2022-04-22T17:54:52.840 INFO:tasks.workunit.client.0.smithi115.stdout:op 2432 completed, throughput=5MB/sec 2022-04-22T17:54:52.840 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kCk6Vz8wI5mp2V- off=18520693 len=576726 2022-04-22T17:54:52.841 INFO:tasks.workunit.client.0.smithi115.stdout:op 2433 completed, throughput=5MB/sec 2022-04-22T17:54:52.842 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f4a2RucVlHN542N off=3630018 len=1658992 2022-04-22T17:54:52.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 2434 completed, throughput=5MB/sec 2022-04-22T17:54:52.845 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kg3b04RvGBKqe71 off=7319698 len=953995 2022-04-22T17:54:52.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 2435 completed, throughput=5MB/sec 2022-04-22T17:54:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:52 smithi121 bash[36902]: audit 2022-04-22T17:54:52.449621+0000 mon.a (mon.0) 2489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:52 smithi115 bash[45495]: audit 2022-04-22T17:54:52.449621+0000 mon.a (mon.0) 2489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:52 smithi115 bash[46934]: audit 2022-04-22T17:54:52.449621+0000 mon.a (mon.0) 2489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:53.851 INFO:tasks.workunit.client.0.smithi115.stdout: 478: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:53.851 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--xByjbv9jf8kC-K off=10687072 len=840848 2022-04-22T17:54:53.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 2436 completed, throughput=4.99MB/sec 2022-04-22T17:54:53.853 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dB0Ps1-lB0KMrDN off=1575171 len=1075506 2022-04-22T17:54:53.856 INFO:tasks.workunit.client.0.smithi115.stdout:op 2437 completed, throughput=4.99MB/sec 2022-04-22T17:54:53.856 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b71_u-7ZWURwFMn off=11313286 len=1163774 2022-04-22T17:54:53.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 2438 completed, throughput=5MB/sec 2022-04-22T17:54:53.859 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0D1zIx58MYIfVye off=3821212 len=795725 2022-04-22T17:54:53.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 2439 completed, throughput=5MB/sec 2022-04-22T17:54:53.863 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yrU8j_Z6IRl1Tq- off=3184462 len=2028571 2022-04-22T17:54:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:op 2440 completed, throughput=5MB/sec 2022-04-22T17:54:53.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:54:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:54:53.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:54:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:53 smithi121 bash[36902]: cluster 2022-04-22T17:54:52.503835+0000 mgr.y (mgr.34107) 1035 : cluster [DBG] pgmap v1645: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:53 smithi115 bash[45495]: cluster 2022-04-22T17:54:52.503835+0000 mgr.y (mgr.34107) 1035 : cluster [DBG] pgmap v1645: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:54.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:53 smithi115 bash[46934]: cluster 2022-04-22T17:54:52.503835+0000 mgr.y (mgr.34107) 1035 : cluster [DBG] pgmap v1645: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:54.873 INFO:tasks.workunit.client.0.smithi115.stdout: 479: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:54.873 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pOYQnWp8dMDiUvl off=2109597 len=535633 2022-04-22T17:54:54.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 2441 completed, throughput=4.99MB/sec 2022-04-22T17:54:54.875 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QGgEY4eNPWhmTGB off=17396637 len=797605 2022-04-22T17:54:54.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 2442 completed, throughput=4.99MB/sec 2022-04-22T17:54:54.877 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3NdzyYbrOMHNhoe off=11118002 len=1574328 2022-04-22T17:54:54.881 INFO:tasks.workunit.client.0.smithi115.stdout:op 2443 completed, throughput=5MB/sec 2022-04-22T17:54:54.881 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mYCKU2xbzH39Usv off=15921837 len=1072107 2022-04-22T17:54:54.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 2444 completed, throughput=5MB/sec 2022-04-22T17:54:54.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-STgt44fImdmp0wn off=795532 len=1696149 2022-04-22T17:54:54.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 2445 completed, throughput=5MB/sec 2022-04-22T17:54:55.898 INFO:tasks.workunit.client.0.smithi115.stdout: 480: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:55.899 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XX78p2JfrhqCMvD off=2357980 len=411329 2022-04-22T17:54:55.904 INFO:tasks.workunit.client.0.smithi115.stdout:op 2446 completed, throughput=4.99MB/sec 2022-04-22T17:54:55.904 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wkvJncldcKSOu-L off=111862 len=1288239 2022-04-22T17:54:55.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 2447 completed, throughput=5MB/sec 2022-04-22T17:54:55.912 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V27DPsmMvtsPINM off=3684349 len=621869 2022-04-22T17:54:55.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 2448 completed, throughput=5MB/sec 2022-04-22T17:54:55.913 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kBcdf4xbSqevDp2 off=1059829 len=2083967 2022-04-22T17:54:55.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 2449 completed, throughput=5MB/sec 2022-04-22T17:54:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:55 smithi121 bash[36902]: cluster 2022-04-22T17:54:54.504852+0000 mgr.y (mgr.34107) 1036 : cluster [DBG] pgmap v1646: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:55 smithi121 bash[36902]: audit 2022-04-22T17:54:55.069751+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:55 smithi121 bash[36902]: audit 2022-04-22T17:54:55.407824+0000 mon.a (mon.0) 2491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[45495]: cluster 2022-04-22T17:54:54.504852+0000 mgr.y (mgr.34107) 1036 : cluster [DBG] pgmap v1646: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[45495]: audit 2022-04-22T17:54:55.069751+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:56.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[45495]: audit 2022-04-22T17:54:55.407824+0000 mon.a (mon.0) 2491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[46934]: cluster 2022-04-22T17:54:54.504852+0000 mgr.y (mgr.34107) 1036 : cluster [DBG] pgmap v1646: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:54:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[46934]: audit 2022-04-22T17:54:55.069751+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:54:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:55 smithi115 bash[46934]: audit 2022-04-22T17:54:55.407824+0000 mon.a (mon.0) 2491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:54:56.916 INFO:tasks.workunit.client.0.smithi115.stdout: 481: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:57.008 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HHy_iNoOnnUIZcD off=10487924 len=384150 2022-04-22T17:54:57.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 2450 completed, throughput=4.99MB/sec 2022-04-22T17:54:57.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4S19KeZs43OC6xR off=379925 len=69657 2022-04-22T17:54:57.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 2451 completed, throughput=4.99MB/sec 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iiFjhIIiTPvn6d5 off=8669546 len=2087560 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 2452 completed, throughput=5MB/sec 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BPdac8QoKCSRUYP off=16648328 len=755391 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 2453 completed, throughput=5MB/sec 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fSpJIOCnGwsSvcP off=7171181 len=443464 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 2454 completed, throughput=5MB/sec 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dyt0-S24U-TEnx_ off=2214830 len=1500656 2022-04-22T17:54:57.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 2455 completed, throughput=5MB/sec 2022-04-22T17:54:57.943 INFO:tasks.workunit.client.0.smithi115.stdout: 482: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:57.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OjdrqQVPLlPpl7q off=20557617 len=2092038 2022-04-22T17:54:57.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 2456 completed, throughput=4.99MB/sec 2022-04-22T17:54:57.954 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fN65cjYdh9xRSOz off=1032955 len=215776 2022-04-22T17:54:57.955 INFO:tasks.workunit.client.0.smithi115.stdout:op 2457 completed, throughput=5MB/sec 2022-04-22T17:54:57.955 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UPiBCmBDPR4_BRy off=1383203 len=1474367 2022-04-22T17:54:57.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 2458 completed, throughput=5MB/sec 2022-04-22T17:54:57.962 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IayzeckD4tceOAQ off=2202363 len=253999 2022-04-22T17:54:57.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 2459 completed, throughput=5MB/sec 2022-04-22T17:54:57.963 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nkHY4Jke5kSt110 off=994591 len=1032714 2022-04-22T17:54:57.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 2460 completed, throughput=5MB/sec 2022-04-22T17:54:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:57 smithi121 bash[36902]: cluster 2022-04-22T17:54:56.505958+0000 mgr.y (mgr.34107) 1037 : cluster [DBG] pgmap v1647: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:54:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:57 smithi121 bash[36902]: audit 2022-04-22T17:54:57.452721+0000 mon.a (mon.0) 2492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:57 smithi115 bash[45495]: cluster 2022-04-22T17:54:56.505958+0000 mgr.y (mgr.34107) 1037 : cluster [DBG] pgmap v1647: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:54:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:57 smithi115 bash[45495]: audit 2022-04-22T17:54:57.452721+0000 mon.a (mon.0) 2492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:58.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:57 smithi115 bash[46934]: cluster 2022-04-22T17:54:56.505958+0000 mgr.y (mgr.34107) 1037 : cluster [DBG] pgmap v1647: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:54:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:57 smithi115 bash[46934]: audit 2022-04-22T17:54:57.452721+0000 mon.a (mon.0) 2492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:54:58.966 INFO:tasks.workunit.client.0.smithi115.stdout: 483: throughput=4.99MB/sec pending data=0 2022-04-22T17:54:58.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MGQU5r1T1R-b7rz off=616403 len=542696 2022-04-22T17:54:58.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 2461 completed, throughput=4.99MB/sec 2022-04-22T17:54:58.968 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C5PEnvaYGqZzltb off=1662512 len=139329 2022-04-22T17:54:58.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 2462 completed, throughput=4.99MB/sec 2022-04-22T17:54:58.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uaPyFnOpoBQTgDn off=47013 len=78262 2022-04-22T17:54:58.973 INFO:tasks.workunit.client.0.smithi115.stdout:op 2463 completed, throughput=4.99MB/sec 2022-04-22T17:54:58.973 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6WjR-6pQqUO_Pdx off=6340418 len=1263444 2022-04-22T17:54:58.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 2464 completed, throughput=4.99MB/sec 2022-04-22T17:54:58.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CSCWq84PCZe_4jz off=6458470 len=1346100 2022-04-22T17:54:58.983 INFO:tasks.workunit.client.0.smithi115.stdout:op 2465 completed, throughput=5MB/sec 2022-04-22T17:54:58.983 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KOMw5xOPL6x6I-z off=2070217 len=1488905 2022-04-22T17:54:58.986 INFO:tasks.workunit.client.0.smithi115.stdout:op 2466 completed, throughput=5MB/sec 2022-04-22T17:54:58.986 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eFt6KiRhFCwyLEw off=8938834 len=247302 2022-04-22T17:54:58.990 INFO:tasks.workunit.client.0.smithi115.stdout:op 2467 completed, throughput=5MB/sec 2022-04-22T17:54:58.991 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mEt9yEDn8EDQPsK off=9848231 len=2006139 2022-04-22T17:54:59.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 2468 completed, throughput=5MB/sec 2022-04-22T17:55:00.002 INFO:tasks.workunit.client.0.smithi115.stdout: 484: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:00.002 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QXrqTp9ZQ9H6Aqy off=3148601 len=2001272 2022-04-22T17:55:00.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 2469 completed, throughput=5MB/sec 2022-04-22T17:55:00.014 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bK2nH26uqi_Wxah off=1209634 len=1810070 2022-04-22T17:55:00.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 2470 completed, throughput=5MB/sec 2022-04-22T17:55:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:54:59 smithi121 bash[36902]: cluster 2022-04-22T17:54:58.507100+0000 mgr.y (mgr.34107) 1038 : cluster [DBG] pgmap v1648: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:55:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:54:59 smithi115 bash[45495]: cluster 2022-04-22T17:54:58.507100+0000 mgr.y (mgr.34107) 1038 : cluster [DBG] pgmap v1648: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:55:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:54:59 smithi115 bash[46934]: cluster 2022-04-22T17:54:58.507100+0000 mgr.y (mgr.34107) 1038 : cluster [DBG] pgmap v1648: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:55:01.021 INFO:tasks.workunit.client.0.smithi115.stdout: 485: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:01.022 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_PffJwPL_UbxcVm off=2373420 len=2048483 2022-04-22T17:55:01.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 2471 completed, throughput=4.99MB/sec 2022-04-22T17:55:01.026 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PmE4B6fnNyvxO-E off=2631338 len=1815076 2022-04-22T17:55:01.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 2472 completed, throughput=5MB/sec 2022-04-22T17:55:01.034 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9xW-9tWAldRTtz- off=5378685 len=1852549 2022-04-22T17:55:01.040 INFO:tasks.workunit.client.0.smithi115.stdout:op 2473 completed, throughput=5MB/sec 2022-04-22T17:55:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:00 smithi121 bash[36902]: audit 2022-04-22T17:55:00.072766+0000 mon.a (mon.0) 2493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:00 smithi121 bash[36902]: audit 2022-04-22T17:55:00.411499+0000 mon.a (mon.0) 2494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:00 smithi115 bash[45495]: audit 2022-04-22T17:55:00.072766+0000 mon.a (mon.0) 2493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:00 smithi115 bash[45495]: audit 2022-04-22T17:55:00.411499+0000 mon.a (mon.0) 2494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:01.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:00 smithi115 bash[46934]: audit 2022-04-22T17:55:00.072766+0000 mon.a (mon.0) 2493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:00 smithi115 bash[46934]: audit 2022-04-22T17:55:00.411499+0000 mon.a (mon.0) 2494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:02.041 INFO:tasks.workunit.client.0.smithi115.stdout: 486: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:02.041 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nh6g4uKSFIL411k off=13198526 len=1434950 2022-04-22T17:55:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:01 smithi115 bash[45495]: cluster 2022-04-22T17:55:00.508182+0000 mgr.y (mgr.34107) 1039 : cluster [DBG] pgmap v1649: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:01 smithi115 bash[46934]: cluster 2022-04-22T17:55:00.508182+0000 mgr.y (mgr.34107) 1039 : cluster [DBG] pgmap v1649: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:01.701Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:02.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 2474 completed, throughput=4.99MB/sec 2022-04-22T17:55:02.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n6MW1XjNfB1PMEo off=7269787 len=2031880 2022-04-22T17:55:02.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 2475 completed, throughput=5MB/sec 2022-04-22T17:55:02.060 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZwS2rgvGhSlaSFG off=7062639 len=393313 2022-04-22T17:55:02.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 2476 completed, throughput=5MB/sec 2022-04-22T17:55:02.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MmG37Kru7ItYP-o off=23643329 len=1790214 2022-04-22T17:55:02.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 2477 completed, throughput=5MB/sec 2022-04-22T17:55:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:01 smithi121 bash[36902]: cluster 2022-04-22T17:55:00.508182+0000 mgr.y (mgr.34107) 1039 : cluster [DBG] pgmap v1649: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:03.069 INFO:tasks.workunit.client.0.smithi115.stdout: 487: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:03.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c4Ylq-U8iFsres8 off=1265791 len=981416 2022-04-22T17:55:03.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 2478 completed, throughput=4.99MB/sec 2022-04-22T17:55:03.072 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qFoEDNpPJ0rA3Eo off=16622780 len=528085 2022-04-22T17:55:03.077 INFO:tasks.workunit.client.0.smithi115.stdout:op 2479 completed, throughput=4.99MB/sec 2022-04-22T17:55:03.077 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5tUi4RS8l04vagN off=396604 len=313482 2022-04-22T17:55:03.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 2480 completed, throughput=5MB/sec 2022-04-22T17:55:03.079 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J_GrUSAH3G8O4OQ off=1325792 len=1287499 2022-04-22T17:55:03.082 INFO:tasks.workunit.client.0.smithi115.stdout:op 2481 completed, throughput=5MB/sec 2022-04-22T17:55:03.082 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7NV64l237wKi09g off=8906612 len=1390132 2022-04-22T17:55:03.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 2482 completed, throughput=5MB/sec 2022-04-22T17:55:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:02 smithi121 bash[36902]: audit 2022-04-22T17:55:02.455880+0000 mon.a (mon.0) 2495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:02 smithi115 bash[46934]: audit 2022-04-22T17:55:02.455880+0000 mon.a (mon.0) 2495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:02 smithi115 bash[45495]: audit 2022-04-22T17:55:02.455880+0000 mon.a (mon.0) 2495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:03.936 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:03.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:04.090 INFO:tasks.workunit.client.0.smithi115.stdout: 488: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:04.090 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KwS2sZX3dURkiAH off=9987112 len=840711 2022-04-22T17:55:04.096 INFO:tasks.workunit.client.0.smithi115.stdout:op 2483 completed, throughput=4.99MB/sec 2022-04-22T17:55:04.096 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z-_e0AZ7Lkon9DF off=5909151 len=1384622 2022-04-22T17:55:04.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 2484 completed, throughput=4.99MB/sec 2022-04-22T17:55:04.099 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qoey9PfS6qj_FiP off=13930702 len=142853 2022-04-22T17:55:04.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 2485 completed, throughput=4.99MB/sec 2022-04-22T17:55:04.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QxQfhUeEYCrmrsX off=14333805 len=193050 2022-04-22T17:55:04.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 2486 completed, throughput=4.99MB/sec 2022-04-22T17:55:04.107 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wHoi4_H5-eX0xlM off=6449545 len=569495 2022-04-22T17:55:04.112 INFO:tasks.workunit.client.0.smithi115.stdout:op 2487 completed, throughput=5MB/sec 2022-04-22T17:55:04.112 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-usuprKbudjEnVZX off=5118729 len=2023345 2022-04-22T17:55:04.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 2488 completed, throughput=5MB/sec 2022-04-22T17:55:04.118 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g5yUWXmfBT9yeTZ off=9419752 len=623638 2022-04-22T17:55:04.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 2489 completed, throughput=5MB/sec 2022-04-22T17:55:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:03 smithi121 bash[36902]: cluster 2022-04-22T17:55:02.509200+0000 mgr.y (mgr.34107) 1040 : cluster [DBG] pgmap v1650: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:55:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:03 smithi115 bash[45495]: cluster 2022-04-22T17:55:02.509200+0000 mgr.y (mgr.34107) 1040 : cluster [DBG] pgmap v1650: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:55:04.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:03 smithi115 bash[46934]: cluster 2022-04-22T17:55:02.509200+0000 mgr.y (mgr.34107) 1040 : cluster [DBG] pgmap v1650: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:55:05.119 INFO:tasks.workunit.client.0.smithi115.stdout: 489: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:05.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c5TSoZk7rzFoCzs off=5374193 len=946911 2022-04-22T17:55:05.128 INFO:tasks.workunit.client.0.smithi115.stdout:op 2490 completed, throughput=4.99MB/sec 2022-04-22T17:55:05.128 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y5OagErxq6hLn-k off=2265212 len=229034 2022-04-22T17:55:05.132 INFO:tasks.workunit.client.0.smithi115.stdout:op 2491 completed, throughput=4.99MB/sec 2022-04-22T17:55:05.133 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P4c8IoLMHfVZoSx off=4227856 len=753599 2022-04-22T17:55:05.134 INFO:tasks.workunit.client.0.smithi115.stdout:op 2492 completed, throughput=4.99MB/sec 2022-04-22T17:55:05.134 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-udqFfI962Y5Lcbr off=11041965 len=1910346 2022-04-22T17:55:05.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 2493 completed, throughput=5MB/sec 2022-04-22T17:55:05.137 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5UndnuF9wwFjeW_ off=3503616 len=2055807 2022-04-22T17:55:05.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 2494 completed, throughput=5MB/sec 2022-04-22T17:55:06.146 INFO:tasks.workunit.client.0.smithi115.stdout: 490: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:06.146 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IZKeKRrS3g6WGW8 off=254584 len=892861 2022-04-22T17:55:06.149 INFO:tasks.workunit.client.0.smithi115.stdout:op 2495 completed, throughput=4.99MB/sec 2022-04-22T17:55:06.149 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wp4Xru87SLtlBU_ off=1196573 len=364925 2022-04-22T17:55:06.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 2496 completed, throughput=4.99MB/sec 2022-04-22T17:55:06.154 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8GykYDbY4Zwmsuz off=18237644 len=1372686 2022-04-22T17:55:06.161 INFO:tasks.workunit.client.0.smithi115.stdout:op 2497 completed, throughput=5MB/sec 2022-04-22T17:55:06.161 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4qZyo86VSDLWjFL off=6681334 len=1761936 2022-04-22T17:55:06.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 2498 completed, throughput=5MB/sec 2022-04-22T17:55:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:05 smithi121 bash[36902]: cluster 2022-04-22T17:55:04.510273+0000 mgr.y (mgr.34107) 1041 : cluster [DBG] pgmap v1651: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:05 smithi121 bash[36902]: audit 2022-04-22T17:55:05.075805+0000 mon.a (mon.0) 2496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:05 smithi121 bash[36902]: audit 2022-04-22T17:55:05.415305+0000 mon.a (mon.0) 2497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[45495]: cluster 2022-04-22T17:55:04.510273+0000 mgr.y (mgr.34107) 1041 : cluster [DBG] pgmap v1651: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[45495]: audit 2022-04-22T17:55:05.075805+0000 mon.a (mon.0) 2496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[45495]: audit 2022-04-22T17:55:05.415305+0000 mon.a (mon.0) 2497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[46934]: cluster 2022-04-22T17:55:04.510273+0000 mgr.y (mgr.34107) 1041 : cluster [DBG] pgmap v1651: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[46934]: audit 2022-04-22T17:55:05.075805+0000 mon.a (mon.0) 2496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:05 smithi115 bash[46934]: audit 2022-04-22T17:55:05.415305+0000 mon.a (mon.0) 2497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:07.169 INFO:tasks.workunit.client.0.smithi115.stdout: 491: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:07.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ykmr5h8Bk-1zsat off=6044114 len=1130183 2022-04-22T17:55:07.172 INFO:tasks.workunit.client.0.smithi115.stdout:op 2499 completed, throughput=4.99MB/sec 2022-04-22T17:55:07.172 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7W3Q2R_V7TrdSLy off=15278802 len=963554 2022-04-22T17:55:07.180 INFO:tasks.workunit.client.0.smithi115.stdout:op 2500 completed, throughput=4.99MB/sec 2022-04-22T17:55:07.180 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_PgKOrwj4-2PBdY off=1256423 len=1814193 2022-04-22T17:55:07.189 INFO:tasks.workunit.client.0.smithi115.stdout:op 2501 completed, throughput=5MB/sec 2022-04-22T17:55:07.189 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DYeVbXz2KlMGOf6 off=7122116 len=447924 2022-04-22T17:55:07.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 2502 completed, throughput=5MB/sec 2022-04-22T17:55:07.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6MJpXXS32-ldhIt off=1841635 len=1504502 2022-04-22T17:55:07.199 INFO:tasks.workunit.client.0.smithi115.stdout:op 2503 completed, throughput=5MB/sec 2022-04-22T17:55:08.199 INFO:tasks.workunit.client.0.smithi115.stdout: 492: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:08.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Anp8idaf0RMbwpX off=10328361 len=359571 2022-04-22T17:55:08.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 2504 completed, throughput=4.99MB/sec 2022-04-22T17:55:08.205 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fhmI4SI8ZJq_dk2 off=11029880 len=1261224 2022-04-22T17:55:08.208 INFO:tasks.workunit.client.0.smithi115.stdout:op 2505 completed, throughput=4.99MB/sec 2022-04-22T17:55:08.208 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IypgJ-5YRBC7CUQ off=8483857 len=1216933 2022-04-22T17:55:08.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 2506 completed, throughput=5MB/sec 2022-04-22T17:55:08.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J2nrH2jI12iVwC2 off=381977 len=431263 2022-04-22T17:55:08.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 2507 completed, throughput=5MB/sec 2022-04-22T17:55:08.216 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bNfbEQgsr6f-D3S off=6935853 len=1255150 2022-04-22T17:55:08.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 2508 completed, throughput=5MB/sec 2022-04-22T17:55:08.219 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tj1GKeJOuGq_Cth off=7903737 len=402990 2022-04-22T17:55:08.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 2509 completed, throughput=5MB/sec 2022-04-22T17:55:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:07 smithi121 bash[36902]: cluster 2022-04-22T17:55:06.511250+0000 mgr.y (mgr.34107) 1042 : cluster [DBG] pgmap v1652: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:55:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:07 smithi121 bash[36902]: audit 2022-04-22T17:55:07.459017+0000 mon.a (mon.0) 2498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:07 smithi115 bash[45495]: cluster 2022-04-22T17:55:06.511250+0000 mgr.y (mgr.34107) 1042 : cluster [DBG] pgmap v1652: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:55:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:07 smithi115 bash[45495]: audit 2022-04-22T17:55:07.459017+0000 mon.a (mon.0) 2498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:08.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:07 smithi115 bash[46934]: cluster 2022-04-22T17:55:06.511250+0000 mgr.y (mgr.34107) 1042 : cluster [DBG] pgmap v1652: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T17:55:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:07 smithi115 bash[46934]: audit 2022-04-22T17:55:07.459017+0000 mon.a (mon.0) 2498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:09.220 INFO:tasks.workunit.client.0.smithi115.stdout: 493: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:09.220 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YHtS87Z3qtNOmYt off=9443895 len=1626634 2022-04-22T17:55:09.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 2510 completed, throughput=4.99MB/sec 2022-04-22T17:55:09.231 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W4-96k5o6XluRqH off=2442403 len=337813 2022-04-22T17:55:09.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 2511 completed, throughput=4.99MB/sec 2022-04-22T17:55:09.237 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tubVaOTTkdfSVcr off=14354306 len=1508332 2022-04-22T17:55:09.244 INFO:tasks.workunit.client.0.smithi115.stdout:op 2512 completed, throughput=5MB/sec 2022-04-22T17:55:09.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cjGSd8PGzLxlCiH off=2000541 len=1506635 2022-04-22T17:55:09.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 2513 completed, throughput=5MB/sec 2022-04-22T17:55:09.246 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7aRjrM6jQerHrMJ off=12420674 len=1058835 2022-04-22T17:55:09.251 INFO:tasks.workunit.client.0.smithi115.stdout:op 2514 completed, throughput=5MB/sec 2022-04-22T17:55:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (18m) 7m ago 32m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (31m) 8m ago 31m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (20m) 7m ago 33m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (19m) 8m ago 32m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:55:10.099 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (27m) 8m ago 37m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (28m) 7m ago 39m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (27m) 7m ago 39m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (26m) 8m ago 37m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (26m) 7m ago 37m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (19m) 7m ago 32m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (19m) 8m ago 32m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:55:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (24m) 7m ago 36m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (24m) 7m ago 36m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (23m) 7m ago 35m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (23m) 7m ago 35m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (22m) 8m ago 34m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (22m) 8m ago 34m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (21m) 8m ago 34m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:55:10.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (20m) 8m ago 33m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:55:10.102 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (18m) 8m ago 32m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:55:10.251 INFO:tasks.workunit.client.0.smithi115.stdout: 494: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:10.251 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CsKbEkZ-JiAQ03C off=7173394 len=822978 2022-04-22T17:55:10.254 INFO:tasks.workunit.client.0.smithi115.stdout:op 2515 completed, throughput=4.99MB/sec 2022-04-22T17:55:10.254 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tBmziP1clUJwVc9 off=4897894 len=1790581 2022-04-22T17:55:10.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 2516 completed, throughput=5MB/sec 2022-04-22T17:55:10.257 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_sC8PkKfAUJPhQD off=2777325 len=697261 2022-04-22T17:55:10.262 INFO:tasks.workunit.client.0.smithi115.stdout:op 2517 completed, throughput=5MB/sec 2022-04-22T17:55:10.262 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AFOClqR9pSXL1_N off=10007741 len=972508 2022-04-22T17:55:10.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 2518 completed, throughput=5MB/sec 2022-04-22T17:55:10.264 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zkhj7ekddNtmInh off=7338111 len=1590367 2022-04-22T17:55:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:09 smithi121 bash[36902]: cluster 2022-04-22T17:55:08.512429+0000 mgr.y (mgr.34107) 1043 : cluster [DBG] pgmap v1653: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:10.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 2519 completed, throughput=5MB/sec 2022-04-22T17:55:10.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:09 smithi115 bash[45495]: cluster 2022-04-22T17:55:08.512429+0000 mgr.y (mgr.34107) 1043 : cluster [DBG] pgmap v1653: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:09 smithi115 bash[46934]: cluster 2022-04-22T17:55:08.512429+0000 mgr.y (mgr.34107) 1043 : cluster [DBG] pgmap v1653: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:10.505 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:55:10.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:55:10.507 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:55:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:10 smithi121 bash[36902]: audit 2022-04-22T17:55:09.726399+0000 mgr.y (mgr.34107) 1044 : audit [DBG] from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:10 smithi121 bash[36902]: audit 2022-04-22T17:55:10.071065+0000 mgr.y (mgr.34107) 1045 : audit [DBG] from='client.45445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:10 smithi121 bash[36902]: audit 2022-04-22T17:55:10.078808+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:10 smithi121 bash[36902]: audit 2022-04-22T17:55:10.418975+0000 mon.a (mon.0) 2500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:11.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:10 smithi121 bash[36902]: audit 2022-04-22T17:55:10.498389+0000 mon.c (mon.1) 73 : audit [DBG] from='client.? 172.21.15.115:0/2294487319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:11.267 INFO:tasks.workunit.client.0.smithi115.stdout: 495: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:11.268 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dEStZso_0WYKqp2 off=7546819 len=1971316 2022-04-22T17:55:11.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 2520 completed, throughput=5MB/sec 2022-04-22T17:55:11.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DJtt49rkjGxNO2T off=9007950 len=725204 2022-04-22T17:55:11.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 2521 completed, throughput=5MB/sec 2022-04-22T17:55:11.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iYN_athp-jpkmH6 off=2735877 len=970345 2022-04-22T17:55:11.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 2522 completed, throughput=5MB/sec 2022-04-22T17:55:11.282 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0wu5D82tH6GizcY off=291369 len=1183926 2022-04-22T17:55:11.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 2523 completed, throughput=5MB/sec 2022-04-22T17:55:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[45495]: audit 2022-04-22T17:55:09.726399+0000 mgr.y (mgr.34107) 1044 : audit [DBG] from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[45495]: audit 2022-04-22T17:55:10.071065+0000 mgr.y (mgr.34107) 1045 : audit [DBG] from='client.45445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[45495]: audit 2022-04-22T17:55:10.078808+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[45495]: audit 2022-04-22T17:55:10.418975+0000 mon.a (mon.0) 2500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[45495]: audit 2022-04-22T17:55:10.498389+0000 mon.c (mon.1) 73 : audit [DBG] from='client.? 172.21.15.115:0/2294487319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[46934]: audit 2022-04-22T17:55:09.726399+0000 mgr.y (mgr.34107) 1044 : audit [DBG] from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[46934]: audit 2022-04-22T17:55:10.071065+0000 mgr.y (mgr.34107) 1045 : audit [DBG] from='client.45445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[46934]: audit 2022-04-22T17:55:10.078808+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[46934]: audit 2022-04-22T17:55:10.418975+0000 mon.a (mon.0) 2500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:10 smithi115 bash[46934]: audit 2022-04-22T17:55:10.498389+0000 mon.c (mon.1) 73 : audit [DBG] from='client.? 172.21.15.115:0/2294487319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:11.959 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:11.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:12.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:11 smithi121 bash[36902]: cluster 2022-04-22T17:55:10.513155+0000 mgr.y (mgr.34107) 1046 : cluster [DBG] pgmap v1654: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2022-04-22T17:55:12.285 INFO:tasks.workunit.client.0.smithi115.stdout: 496: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:12.286 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TREZRqhyL_3k50W off=1653590 len=54687 2022-04-22T17:55:12.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 2524 completed, throughput=4.99MB/sec 2022-04-22T17:55:12.290 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sx2-5YMLuNchnOt off=2370090 len=1770400 2022-04-22T17:55:12.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:11 smithi115 bash[45495]: cluster 2022-04-22T17:55:10.513155+0000 mgr.y (mgr.34107) 1046 : cluster [DBG] pgmap v1654: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2022-04-22T17:55:12.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:11 smithi115 bash[46934]: cluster 2022-04-22T17:55:10.513155+0000 mgr.y (mgr.34107) 1046 : cluster [DBG] pgmap v1654: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 3 op/s 2022-04-22T17:55:12.295 INFO:tasks.workunit.client.0.smithi115.stdout:op 2525 completed, throughput=4.99MB/sec 2022-04-22T17:55:12.296 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WoBX-cZSg9VLWw_ off=11195559 len=1527759 2022-04-22T17:55:12.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 2526 completed, throughput=5MB/sec 2022-04-22T17:55:12.298 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GlzQxG9SOkTStHK off=2412673 len=1707459 2022-04-22T17:55:12.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 2527 completed, throughput=5MB/sec 2022-04-22T17:55:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:12 smithi121 bash[36902]: audit 2022-04-22T17:55:12.462320+0000 mon.a (mon.0) 2501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:12 smithi115 bash[45495]: audit 2022-04-22T17:55:12.462320+0000 mon.a (mon.0) 2501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:13.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:12 smithi115 bash[46934]: audit 2022-04-22T17:55:12.462320+0000 mon.a (mon.0) 2501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:13.307 INFO:tasks.workunit.client.0.smithi115.stdout: 497: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:13.307 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U_2E_d4kmBPUMVZ off=17770848 len=689524 2022-04-22T17:55:13.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 2528 completed, throughput=4.99MB/sec 2022-04-22T17:55:13.313 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qKlrwldnas6A3yk off=2106317 len=907690 2022-04-22T17:55:13.320 INFO:tasks.workunit.client.0.smithi115.stdout:op 2529 completed, throughput=4.99MB/sec 2022-04-22T17:55:13.320 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P43hhc5EC5b7J6q off=1845930 len=1232851 2022-04-22T17:55:13.327 INFO:tasks.workunit.client.0.smithi115.stdout:op 2530 completed, throughput=5MB/sec 2022-04-22T17:55:13.328 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n6MW1XjNfB1PMEo off=7984212 len=222351 2022-04-22T17:55:13.331 INFO:tasks.workunit.client.0.smithi115.stdout:op 2531 completed, throughput=5MB/sec 2022-04-22T17:55:13.331 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-14zaqRWWL8mC0Tt off=8387402 len=840150 2022-04-22T17:55:13.337 INFO:tasks.workunit.client.0.smithi115.stdout:op 2532 completed, throughput=5MB/sec 2022-04-22T17:55:13.337 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DEbfS7fbJIxyauq off=3269991 len=1769695 2022-04-22T17:55:13.340 INFO:tasks.workunit.client.0.smithi115.stdout:op 2533 completed, throughput=5MB/sec 2022-04-22T17:55:13.973 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:13.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:13 smithi121 bash[36902]: cluster 2022-04-22T17:55:12.514169+0000 mgr.y (mgr.34107) 1047 : cluster [DBG] pgmap v1655: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:55:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:13 smithi121 bash[36902]: audit 2022-04-22T17:55:13.529570+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:55:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:13 smithi121 bash[36902]: audit 2022-04-22T17:55:13.530022+0000 mon.a (mon.0) 2503 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:55:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[45495]: cluster 2022-04-22T17:55:12.514169+0000 mgr.y (mgr.34107) 1047 : cluster [DBG] pgmap v1655: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:55:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[45495]: audit 2022-04-22T17:55:13.529570+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:55:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[45495]: audit 2022-04-22T17:55:13.530022+0000 mon.a (mon.0) 2503 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:55:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[46934]: cluster 2022-04-22T17:55:12.514169+0000 mgr.y (mgr.34107) 1047 : cluster [DBG] pgmap v1655: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 3 op/s 2022-04-22T17:55:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[46934]: audit 2022-04-22T17:55:13.529570+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:55:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:13 smithi115 bash[46934]: audit 2022-04-22T17:55:13.530022+0000 mon.a (mon.0) 2503 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:55:14.340 INFO:tasks.workunit.client.0.smithi115.stdout: 498: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:14.341 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oqMX6OXXMLFLIox off=17813169 len=639004 2022-04-22T17:55:14.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 2534 completed, throughput=4.99MB/sec 2022-04-22T17:55:14.347 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q0l7cvYJOpnSAUh off=14379197 len=993248 2022-04-22T17:55:14.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 2535 completed, throughput=4.99MB/sec 2022-04-22T17:55:14.354 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sNtYYCD7eRr3H9v off=871532 len=364267 2022-04-22T17:55:14.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 2536 completed, throughput=5MB/sec 2022-04-22T17:55:14.358 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zM5w9XzPbSxnGI2 off=11063552 len=1766886 2022-04-22T17:55:14.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 2537 completed, throughput=5MB/sec 2022-04-22T17:55:14.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iZCqbEJ35T5w0eB off=3595822 len=1332942 2022-04-22T17:55:14.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 2538 completed, throughput=5MB/sec 2022-04-22T17:55:15.368 INFO:tasks.workunit.client.0.smithi115.stdout: 499: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:15.368 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fG8KlaLCSMqNkFF off=1768466 len=1315215 2022-04-22T17:55:15.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 2539 completed, throughput=4.99MB/sec 2022-04-22T17:55:15.372 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ns_VmOeDncd1vi5 off=12561455 len=38803 2022-04-22T17:55:15.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 2540 completed, throughput=4.99MB/sec 2022-04-22T17:55:15.376 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SOCZN1FOrmLMM1N off=20748902 len=372347 2022-04-22T17:55:15.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 2541 completed, throughput=4.99MB/sec 2022-04-22T17:55:15.381 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-liXaO8h3t4TyBw_ off=1065140 len=636606 2022-04-22T17:55:15.383 INFO:tasks.workunit.client.0.smithi115.stdout:op 2542 completed, throughput=5MB/sec 2022-04-22T17:55:15.383 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yejJRJh6_ZEzJjz off=2827251 len=505443 2022-04-22T17:55:15.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 2543 completed, throughput=5MB/sec 2022-04-22T17:55:15.388 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m40Z9VHjIuz93uC off=10177300 len=1518300 2022-04-22T17:55:15.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 2544 completed, throughput=5MB/sec 2022-04-22T17:55:15.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-10kK5lzw7ts8V2L off=2236136 len=58707 2022-04-22T17:55:15.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 2545 completed, throughput=5MB/sec 2022-04-22T17:55:15.392 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e8fyy_4e5za1-bQ off=4033470 len=1242169 2022-04-22T17:55:15.393 INFO:tasks.workunit.client.0.smithi115.stdout:op 2546 completed, throughput=5MB/sec 2022-04-22T17:55:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:15 smithi121 bash[36902]: cluster 2022-04-22T17:55:14.515246+0000 mgr.y (mgr.34107) 1048 : cluster [DBG] pgmap v1656: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:16.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:15 smithi121 bash[36902]: audit 2022-04-22T17:55:15.081455+0000 mon.a (mon.0) 2504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:16.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:15 smithi121 bash[36902]: audit 2022-04-22T17:55:15.422514+0000 mon.a (mon.0) 2505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:16.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[45495]: cluster 2022-04-22T17:55:14.515246+0000 mgr.y (mgr.34107) 1048 : cluster [DBG] pgmap v1656: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:16.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[45495]: audit 2022-04-22T17:55:15.081455+0000 mon.a (mon.0) 2504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:16.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[45495]: audit 2022-04-22T17:55:15.422514+0000 mon.a (mon.0) 2505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:16.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[46934]: cluster 2022-04-22T17:55:14.515246+0000 mgr.y (mgr.34107) 1048 : cluster [DBG] pgmap v1656: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:16.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[46934]: audit 2022-04-22T17:55:15.081455+0000 mon.a (mon.0) 2504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:16.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:15 smithi115 bash[46934]: audit 2022-04-22T17:55:15.422514+0000 mon.a (mon.0) 2505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:16.394 INFO:tasks.workunit.client.0.smithi115.stdout: 500: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:16.394 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ao5t0d0sJSDSqx1 off=5348061 len=53651 2022-04-22T17:55:16.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 2547 completed, throughput=4.99MB/sec 2022-04-22T17:55:16.398 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vo4Ri_aeWKcW8iz off=42471 len=2055335 2022-04-22T17:55:16.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 2548 completed, throughput=5MB/sec 2022-04-22T17:55:16.403 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QGgEY4eNPWhmTGB off=7843746 len=1032621 2022-04-22T17:55:16.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 2549 completed, throughput=5MB/sec 2022-04-22T17:55:16.410 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FES923qzJ-zCHmr off=475357 len=1495261 2022-04-22T17:55:16.416 INFO:tasks.workunit.client.0.smithi115.stdout:op 2550 completed, throughput=5MB/sec 2022-04-22T17:55:16.416 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxgFc6DjtV_zvHh off=2498407 len=201576 2022-04-22T17:55:16.416 INFO:tasks.workunit.client.0.smithi115.stdout:op 2551 completed, throughput=5MB/sec 2022-04-22T17:55:17.417 INFO:tasks.workunit.client.0.smithi115.stdout: 501: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:17.417 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vvnTFphsxDDyAlW off=11644593 len=706209 2022-04-22T17:55:17.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 2552 completed, throughput=4.99MB/sec 2022-04-22T17:55:17.419 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hQ7ct3iLxkwR1HZ off=2819460 len=323841 2022-04-22T17:55:17.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 2553 completed, throughput=4.99MB/sec 2022-04-22T17:55:17.421 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QUG1JHK4oaNDp7i off=19484071 len=65773 2022-04-22T17:55:17.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 2554 completed, throughput=4.99MB/sec 2022-04-22T17:55:17.422 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G53o5FvP-KEdtzz off=4307159 len=1766507 2022-04-22T17:55:17.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 2555 completed, throughput=5MB/sec 2022-04-22T17:55:17.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4PxARb-SPN5lvpV off=4814808 len=365589 2022-04-22T17:55:17.438 INFO:tasks.workunit.client.0.smithi115.stdout:op 2556 completed, throughput=5MB/sec 2022-04-22T17:55:17.438 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2qidpMTOhiv0QTl off=6173044 len=1371438 2022-04-22T17:55:17.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 2557 completed, throughput=5MB/sec 2022-04-22T17:55:17.441 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4g_FNb62Y3JIPTt off=4887497 len=1913868 2022-04-22T17:55:17.444 INFO:tasks.workunit.client.0.smithi115.stdout:op 2558 completed, throughput=5MB/sec 2022-04-22T17:55:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:17 smithi121 bash[36902]: cluster 2022-04-22T17:55:16.516318+0000 mgr.y (mgr.34107) 1049 : cluster [DBG] pgmap v1657: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:55:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:17 smithi121 bash[36902]: audit 2022-04-22T17:55:17.465572+0000 mon.a (mon.0) 2506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:17 smithi115 bash[45495]: cluster 2022-04-22T17:55:16.516318+0000 mgr.y (mgr.34107) 1049 : cluster [DBG] pgmap v1657: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:55:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:17 smithi115 bash[45495]: audit 2022-04-22T17:55:17.465572+0000 mon.a (mon.0) 2506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:17 smithi115 bash[46934]: cluster 2022-04-22T17:55:16.516318+0000 mgr.y (mgr.34107) 1049 : cluster [DBG] pgmap v1657: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:55:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:17 smithi115 bash[46934]: audit 2022-04-22T17:55:17.465572+0000 mon.a (mon.0) 2506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:18.445 INFO:tasks.workunit.client.0.smithi115.stdout: 502: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:18.445 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mDZCZeosXJuFnFv off=7113001 len=513297 2022-04-22T17:55:18.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 2559 completed, throughput=4.99MB/sec 2022-04-22T17:55:18.451 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DMkUeThWu2ohP6w off=10035566 len=1966127 2022-04-22T17:55:18.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 2560 completed, throughput=5MB/sec 2022-04-22T17:55:18.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zAhJ6cvFLDBfkVI off=9623551 len=243391 2022-04-22T17:55:18.459 INFO:tasks.workunit.client.0.smithi115.stdout:op 2561 completed, throughput=5MB/sec 2022-04-22T17:55:18.459 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-niKjv5xmqAbWxH6 off=7646643 len=1175242 2022-04-22T17:55:18.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 2562 completed, throughput=5MB/sec 2022-04-22T17:55:18.466 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wm_eJs9zDG7yUjH off=1099355 len=473009 2022-04-22T17:55:18.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 2563 completed, throughput=5MB/sec 2022-04-22T17:55:19.471 INFO:tasks.workunit.client.0.smithi115.stdout: 503: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:19.471 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HXjLW6tuNZCn3Zu off=3659491 len=146982 2022-04-22T17:55:19.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 2564 completed, throughput=4.99MB/sec 2022-04-22T17:55:19.472 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mrZIw5rhT7YZiQq off=123022 len=1371566 2022-04-22T17:55:19.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 2565 completed, throughput=4.99MB/sec 2022-04-22T17:55:19.481 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CsyRIj1j0hnIMZg off=371333 len=1759346 2022-04-22T17:55:19.484 INFO:tasks.workunit.client.0.smithi115.stdout:op 2566 completed, throughput=5MB/sec 2022-04-22T17:55:19.485 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dkfDekaUPwOl4m2 off=3736273 len=1257459 2022-04-22T17:55:19.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 2567 completed, throughput=5MB/sec 2022-04-22T17:55:19.486 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i5CVXZmUshHn7E3 off=8165819 len=2060194 2022-04-22T17:55:19.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 2568 completed, throughput=5MB/sec 2022-04-22T17:55:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:19 smithi121 bash[36902]: cluster 2022-04-22T17:55:18.517465+0000 mgr.y (mgr.34107) 1050 : cluster [DBG] pgmap v1658: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:55:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:19 smithi115 bash[45495]: cluster 2022-04-22T17:55:18.517465+0000 mgr.y (mgr.34107) 1050 : cluster [DBG] pgmap v1658: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:55:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:19 smithi115 bash[46934]: cluster 2022-04-22T17:55:18.517465+0000 mgr.y (mgr.34107) 1050 : cluster [DBG] pgmap v1658: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:55:20.490 INFO:tasks.workunit.client.0.smithi115.stdout: 504: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:20.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DBlG8Y5-7a57wVU off=19609851 len=591952 2022-04-22T17:55:20.497 INFO:tasks.workunit.client.0.smithi115.stdout:op 2569 completed, throughput=4.99MB/sec 2022-04-22T17:55:20.497 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5b_rkMoo3afdtb3 off=1159740 len=18378 2022-04-22T17:55:20.497 INFO:tasks.workunit.client.0.smithi115.stdout:op 2570 completed, throughput=4.99MB/sec 2022-04-22T17:55:20.498 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eh0qcIVaID6Y86I off=5003489 len=1198310 2022-04-22T17:55:20.500 INFO:tasks.workunit.client.0.smithi115.stdout:op 2571 completed, throughput=5MB/sec 2022-04-22T17:55:20.501 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DsQ-4Iw8TDo6NM8 off=3802612 len=210752 2022-04-22T17:55:20.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2572 completed, throughput=5MB/sec 2022-04-22T17:55:20.502 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sNNQFXs5w1ADmBi off=1625868 len=190452 2022-04-22T17:55:20.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 2573 completed, throughput=5MB/sec 2022-04-22T17:55:20.506 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yRVTvZOnzaWeGE9 off=572091 len=1498246 2022-04-22T17:55:20.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 2574 completed, throughput=5MB/sec 2022-04-22T17:55:20.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-__fc9c4yO9Y2npU off=1633382 len=703773 2022-04-22T17:55:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 2575 completed, throughput=5MB/sec 2022-04-22T17:55:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:20 smithi121 bash[36902]: audit 2022-04-22T17:55:20.084646+0000 mon.a (mon.0) 2507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:20 smithi121 bash[36902]: audit 2022-04-22T17:55:20.426081+0000 mon.a (mon.0) 2508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:20 smithi115 bash[45495]: audit 2022-04-22T17:55:20.084646+0000 mon.a (mon.0) 2507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:20 smithi115 bash[45495]: audit 2022-04-22T17:55:20.426081+0000 mon.a (mon.0) 2508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:20 smithi115 bash[46934]: audit 2022-04-22T17:55:20.084646+0000 mon.a (mon.0) 2507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:20 smithi115 bash[46934]: audit 2022-04-22T17:55:20.426081+0000 mon.a (mon.0) 2508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:21.512 INFO:tasks.workunit.client.0.smithi115.stdout: 505: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:21.512 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LObvUbqjMYaTsRP off=7238999 len=2325 2022-04-22T17:55:21.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 2576 completed, throughput=4.99MB/sec 2022-04-22T17:55:21.516 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4SrsiyJJhB3PN7w off=9727691 len=1659422 2022-04-22T17:55:21.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 2577 completed, throughput=4.99MB/sec 2022-04-22T17:55:21.525 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sEYsvz63Hzekr_t off=116134 len=1391685 2022-04-22T17:55:21.530 INFO:tasks.workunit.client.0.smithi115.stdout:op 2578 completed, throughput=5MB/sec 2022-04-22T17:55:21.530 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P4cBFEuOkS12Eip off=3093714 len=1812803 2022-04-22T17:55:21.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 2579 completed, throughput=5MB/sec 2022-04-22T17:55:21.997 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:21.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:21 smithi121 bash[36902]: cluster 2022-04-22T17:55:20.518595+0000 mgr.y (mgr.34107) 1051 : cluster [DBG] pgmap v1659: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:55:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:21 smithi115 bash[45495]: cluster 2022-04-22T17:55:20.518595+0000 mgr.y (mgr.34107) 1051 : cluster [DBG] pgmap v1659: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:55:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:21 smithi115 bash[46934]: cluster 2022-04-22T17:55:20.518595+0000 mgr.y (mgr.34107) 1051 : cluster [DBG] pgmap v1659: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:55:22.540 INFO:tasks.workunit.client.0.smithi115.stdout: 506: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:22.540 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ou_wAJekKpyPpXI off=773389 len=1655235 2022-04-22T17:55:22.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 2580 completed, throughput=4.99MB/sec 2022-04-22T17:55:22.551 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d1D9hi9enexxMGd off=14356842 len=1144445 2022-04-22T17:55:22.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 2581 completed, throughput=5MB/sec 2022-04-22T17:55:22.557 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C1n1w6vXKPmn1QU off=6864119 len=793060 2022-04-22T17:55:22.562 INFO:tasks.workunit.client.0.smithi115.stdout:op 2582 completed, throughput=5MB/sec 2022-04-22T17:55:22.562 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ILlLAmu9ogIUgqS off=1212648 len=818591 2022-04-22T17:55:22.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 2583 completed, throughput=5MB/sec 2022-04-22T17:55:22.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ODWxF1AHLzjZFNJ off=21832607 len=1187528 2022-04-22T17:55:22.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 2584 completed, throughput=5MB/sec 2022-04-22T17:55:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:23 smithi121 bash[36902]: audit 2022-04-22T17:55:22.468548+0000 mon.a (mon.0) 2509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:23 smithi115 bash[45495]: audit 2022-04-22T17:55:22.468548+0000 mon.a (mon.0) 2509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:23 smithi115 bash[46934]: audit 2022-04-22T17:55:22.468548+0000 mon.a (mon.0) 2509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:23.573 INFO:tasks.workunit.client.0.smithi115.stdout: 507: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:23.573 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-714ChW-y5mIUN8q off=3387688 len=536515 2022-04-22T17:55:23.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 2585 completed, throughput=4.99MB/sec 2022-04-22T17:55:23.581 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-izxT2EmwSVMk5A5 off=17306083 len=832920 2022-04-22T17:55:23.586 INFO:tasks.workunit.client.0.smithi115.stdout:op 2586 completed, throughput=4.99MB/sec 2022-04-22T17:55:23.586 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bJnP-cmHBsElvUJ off=26549 len=249083 2022-04-22T17:55:23.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 2587 completed, throughput=4.99MB/sec 2022-04-22T17:55:23.587 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nNAwnWTlWsQcnt- off=75598 len=834672 2022-04-22T17:55:23.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 2588 completed, throughput=5MB/sec 2022-04-22T17:55:23.592 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pl_EMOVjbqpERho off=1039175 len=2084543 2022-04-22T17:55:23.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 2589 completed, throughput=5MB/sec 2022-04-22T17:55:23.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c3hJmpXCAzI-oLo off=334181 len=602209 2022-04-22T17:55:23.600 INFO:tasks.workunit.client.0.smithi115.stdout:op 2590 completed, throughput=5MB/sec 2022-04-22T17:55:24.011 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:23.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:24 smithi121 bash[36902]: cluster 2022-04-22T17:55:22.519708+0000 mgr.y (mgr.34107) 1052 : cluster [DBG] pgmap v1660: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:55:24.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:24 smithi115 bash[45495]: cluster 2022-04-22T17:55:22.519708+0000 mgr.y (mgr.34107) 1052 : cluster [DBG] pgmap v1660: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:55:24.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:24 smithi115 bash[46934]: cluster 2022-04-22T17:55:22.519708+0000 mgr.y (mgr.34107) 1052 : cluster [DBG] pgmap v1660: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:55:24.601 INFO:tasks.workunit.client.0.smithi115.stdout: 508: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:24.601 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d0rDGgeRHTIHbko off=2287727 len=1996061 2022-04-22T17:55:24.611 INFO:tasks.workunit.client.0.smithi115.stdout:op 2591 completed, throughput=4.99MB/sec 2022-04-22T17:55:24.611 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yubC2s4FRO0WFvg off=14417348 len=426863 2022-04-22T17:55:24.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 2592 completed, throughput=4.99MB/sec 2022-04-22T17:55:24.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r_FG8FbZLPJSCEN off=11255489 len=1088331 2022-04-22T17:55:24.616 INFO:tasks.workunit.client.0.smithi115.stdout:op 2593 completed, throughput=5MB/sec 2022-04-22T17:55:24.616 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oGRZ9Mgc1bj2qVx off=1780116 len=1795847 2022-04-22T17:55:24.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 2594 completed, throughput=5MB/sec 2022-04-22T17:55:24.620 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aWf6nKDrWtlHao6 off=8409794 len=1375878 2022-04-22T17:55:24.623 INFO:tasks.workunit.client.0.smithi115.stdout:op 2595 completed, throughput=5MB/sec 2022-04-22T17:55:25.623 INFO:tasks.workunit.client.0.smithi115.stdout: 509: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:25.623 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-toUsrM-Q3mLyWxa off=207201 len=300498 2022-04-22T17:55:25.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 2596 completed, throughput=4.99MB/sec 2022-04-22T17:55:25.628 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sfuCbEv5zVFwi2W off=1537331 len=1255356 2022-04-22T17:55:25.631 INFO:tasks.workunit.client.0.smithi115.stdout:op 2597 completed, throughput=5MB/sec 2022-04-22T17:55:25.631 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FcYoq91XYK5dR2y off=7335229 len=1704927 2022-04-22T17:55:25.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 2598 completed, throughput=5MB/sec 2022-04-22T17:55:25.634 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k4pjiZt6mk78ExB off=1984240 len=73924 2022-04-22T17:55:25.639 INFO:tasks.workunit.client.0.smithi115.stdout:op 2599 completed, throughput=5MB/sec 2022-04-22T17:55:25.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7BAlZdcxZWsrSp3 off=3827177 len=1091863 2022-04-22T17:55:25.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 2600 completed, throughput=5MB/sec 2022-04-22T17:55:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[45495]: cluster 2022-04-22T17:55:24.520923+0000 mgr.y (mgr.34107) 1053 : cluster [DBG] pgmap v1661: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[45495]: audit 2022-04-22T17:55:25.087355+0000 mon.a (mon.0) 2510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[45495]: audit 2022-04-22T17:55:25.429622+0000 mon.a (mon.0) 2511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[46934]: cluster 2022-04-22T17:55:24.520923+0000 mgr.y (mgr.34107) 1053 : cluster [DBG] pgmap v1661: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[46934]: audit 2022-04-22T17:55:25.087355+0000 mon.a (mon.0) 2510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:26 smithi115 bash[46934]: audit 2022-04-22T17:55:25.429622+0000 mon.a (mon.0) 2511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:26 smithi121 bash[36902]: cluster 2022-04-22T17:55:24.520923+0000 mgr.y (mgr.34107) 1053 : cluster [DBG] pgmap v1661: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:26 smithi121 bash[36902]: audit 2022-04-22T17:55:25.087355+0000 mon.a (mon.0) 2510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:26 smithi121 bash[36902]: audit 2022-04-22T17:55:25.429622+0000 mon.a (mon.0) 2511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:26.640 INFO:tasks.workunit.client.0.smithi115.stdout: 510: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:26.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6KCbruqpKeffBvT off=1644227 len=239050 2022-04-22T17:55:26.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 2601 completed, throughput=4.99MB/sec 2022-04-22T17:55:26.645 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1MV-o5q2ZlugDdT off=14529417 len=836624 2022-04-22T17:55:26.653 INFO:tasks.workunit.client.0.smithi115.stdout:op 2602 completed, throughput=4.99MB/sec 2022-04-22T17:55:26.653 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MsjJGh44DznuW_8 off=4833196 len=265731 2022-04-22T17:55:26.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 2603 completed, throughput=4.99MB/sec 2022-04-22T17:55:26.658 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5FFbewbN4yybCod off=5593534 len=1233247 2022-04-22T17:55:26.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 2604 completed, throughput=5MB/sec 2022-04-22T17:55:26.660 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K9Tb2gmdQLA-0z8 off=5012544 len=932273 2022-04-22T17:55:26.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 2605 completed, throughput=5MB/sec 2022-04-22T17:55:26.667 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yFlANPO9vdy82B6 off=5819448 len=1409355 2022-04-22T17:55:26.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 2606 completed, throughput=5MB/sec 2022-04-22T17:55:26.669 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qNm-FWzLtxQK6Yf off=5586075 len=340348 2022-04-22T17:55:26.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 2607 completed, throughput=5MB/sec 2022-04-22T17:55:27.671 INFO:tasks.workunit.client.0.smithi115.stdout: 511: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:27.671 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qps0YKHbMEa2e7I off=1768784 len=493832 2022-04-22T17:55:27.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 2608 completed, throughput=4.99MB/sec 2022-04-22T17:55:27.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-54p_VC2lRVHjYJC off=330757 len=1880816 2022-04-22T17:55:27.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 2609 completed, throughput=5MB/sec 2022-04-22T17:55:27.682 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QpRlT-pOKxMfmZr off=6089736 len=1586500 2022-04-22T17:55:27.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 2610 completed, throughput=5MB/sec 2022-04-22T17:55:27.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XIaDTisQhasqbWw off=3999320 len=1070991 2022-04-22T17:55:27.686 INFO:tasks.workunit.client.0.smithi115.stdout:op 2611 completed, throughput=5MB/sec 2022-04-22T17:55:27.686 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Yt5XrVkh-UdBhyS off=0 len=1468438 2022-04-22T17:55:27.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 2612 completed, throughput=5MB/sec 2022-04-22T17:55:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:28 smithi115 bash[45495]: cluster 2022-04-22T17:55:26.522029+0000 mgr.y (mgr.34107) 1054 : cluster [DBG] pgmap v1662: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:28.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:28 smithi115 bash[45495]: audit 2022-04-22T17:55:27.471584+0000 mon.a (mon.0) 2512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:28.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:28 smithi115 bash[46934]: cluster 2022-04-22T17:55:26.522029+0000 mgr.y (mgr.34107) 1054 : cluster [DBG] pgmap v1662: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:28.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:28 smithi115 bash[46934]: audit 2022-04-22T17:55:27.471584+0000 mon.a (mon.0) 2512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:28 smithi121 bash[36902]: cluster 2022-04-22T17:55:26.522029+0000 mgr.y (mgr.34107) 1054 : cluster [DBG] pgmap v1662: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:55:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:28 smithi121 bash[36902]: audit 2022-04-22T17:55:27.471584+0000 mon.a (mon.0) 2512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:28.690 INFO:tasks.workunit.client.0.smithi115.stdout: 512: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:28.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YCQgRzREZ3AC4Ok off=16949010 len=1154970 2022-04-22T17:55:28.697 INFO:tasks.workunit.client.0.smithi115.stdout:op 2613 completed, throughput=5MB/sec 2022-04-22T17:55:28.697 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PWLgp9F2O_iu1A5 off=1685547 len=321055 2022-04-22T17:55:28.702 INFO:tasks.workunit.client.0.smithi115.stdout:op 2614 completed, throughput=5MB/sec 2022-04-22T17:55:28.702 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8JtWKC2iywNrYc4 off=21397621 len=1208249 2022-04-22T17:55:28.704 INFO:tasks.workunit.client.0.smithi115.stdout:op 2615 completed, throughput=5MB/sec 2022-04-22T17:55:28.704 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fJgEp-nOXGxQTEr off=1022364 len=593485 2022-04-22T17:55:28.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 2616 completed, throughput=5MB/sec 2022-04-22T17:55:28.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gbB6WK0hnce_-H8 off=4829380 len=1785384 2022-04-22T17:55:28.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 2617 completed, throughput=5MB/sec 2022-04-22T17:55:29.709 INFO:tasks.workunit.client.0.smithi115.stdout: 513: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:29.710 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S2OE8PLH-r3MGYC off=495197 len=1215127 2022-04-22T17:55:29.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 2618 completed, throughput=4.99MB/sec 2022-04-22T17:55:29.714 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7HH32ixM9l3qYTt off=4207954 len=1982608 2022-04-22T17:55:29.718 INFO:tasks.workunit.client.0.smithi115.stdout:op 2619 completed, throughput=5MB/sec 2022-04-22T17:55:29.718 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XlEXLbPaHYGCCRG off=8306048 len=1833869 2022-04-22T17:55:29.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 2620 completed, throughput=5MB/sec 2022-04-22T17:55:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:30 smithi115 bash[45495]: cluster 2022-04-22T17:55:28.523191+0000 mgr.y (mgr.34107) 1055 : cluster [DBG] pgmap v1663: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:30 smithi115 bash[46934]: cluster 2022-04-22T17:55:28.523191+0000 mgr.y (mgr.34107) 1055 : cluster [DBG] pgmap v1663: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:30 smithi121 bash[36902]: cluster 2022-04-22T17:55:28.523191+0000 mgr.y (mgr.34107) 1055 : cluster [DBG] pgmap v1663: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:30.721 INFO:tasks.workunit.client.0.smithi115.stdout: 514: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:30.721 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rssk6a7HQU-Fuwt off=18881168 len=488882 2022-04-22T17:55:30.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 2621 completed, throughput=4.99MB/sec 2022-04-22T17:55:30.723 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_I2Cz8Y82ctHqWD off=4730737 len=706136 2022-04-22T17:55:30.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 2622 completed, throughput=4.99MB/sec 2022-04-22T17:55:30.729 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VpTdcfYrtWR38Md off=2922780 len=736611 2022-04-22T17:55:30.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 2623 completed, throughput=5MB/sec 2022-04-22T17:55:30.731 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TdT40Jc4c3VWgEA off=12743730 len=1816376 2022-04-22T17:55:30.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 2624 completed, throughput=5MB/sec 2022-04-22T17:55:30.737 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z1aNe0LZoAuIyoY off=8015504 len=483102 2022-04-22T17:55:30.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 2625 completed, throughput=5MB/sec 2022-04-22T17:55:30.739 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UCOUGDRfZxqUUrn off=1599024 len=1328812 2022-04-22T17:55:30.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 2626 completed, throughput=5MB/sec 2022-04-22T17:55:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:31 smithi115 bash[45495]: audit 2022-04-22T17:55:30.090529+0000 mon.a (mon.0) 2513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:31 smithi115 bash[45495]: audit 2022-04-22T17:55:30.433200+0000 mon.a (mon.0) 2514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:31 smithi115 bash[46934]: audit 2022-04-22T17:55:30.090529+0000 mon.a (mon.0) 2513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:31 smithi115 bash[46934]: audit 2022-04-22T17:55:30.433200+0000 mon.a (mon.0) 2514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:31 smithi121 bash[36902]: audit 2022-04-22T17:55:30.090529+0000 mon.a (mon.0) 2513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:31 smithi121 bash[36902]: audit 2022-04-22T17:55:30.433200+0000 mon.a (mon.0) 2514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:31.742 INFO:tasks.workunit.client.0.smithi115.stdout: 515: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:31.742 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DSZHcsTmbYevTaL off=10297981 len=652098 2022-04-22T17:55:31.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 2627 completed, throughput=4.99MB/sec 2022-04-22T17:55:31.748 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_LWtSRBTzQi3o_9 off=2450647 len=1956459 2022-04-22T17:55:31.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 2628 completed, throughput=5MB/sec 2022-04-22T17:55:31.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fgu4nHp1e9w9iN2 off=2352882 len=30128 2022-04-22T17:55:31.756 INFO:tasks.workunit.client.0.smithi115.stdout:op 2629 completed, throughput=5MB/sec 2022-04-22T17:55:31.756 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-seol_UVlz9ouGjq off=8751406 len=397881 2022-04-22T17:55:31.761 INFO:tasks.workunit.client.0.smithi115.stdout:op 2630 completed, throughput=5MB/sec 2022-04-22T17:55:31.761 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UAoUJPzatWvIrU9 off=2171155 len=1172580 2022-04-22T17:55:31.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 2631 completed, throughput=5MB/sec 2022-04-22T17:55:31.763 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2ExNYUTF3CqSpFW off=3222248 len=1051810 2022-04-22T17:55:31.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 2632 completed, throughput=5MB/sec 2022-04-22T17:55:32.037 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:31.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:32 smithi115 bash[45495]: cluster 2022-04-22T17:55:30.524264+0000 mgr.y (mgr.34107) 1056 : cluster [DBG] pgmap v1664: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:55:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:32 smithi115 bash[46934]: cluster 2022-04-22T17:55:30.524264+0000 mgr.y (mgr.34107) 1056 : cluster [DBG] pgmap v1664: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:55:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:32 smithi121 bash[36902]: cluster 2022-04-22T17:55:30.524264+0000 mgr.y (mgr.34107) 1056 : cluster [DBG] pgmap v1664: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:55:32.767 INFO:tasks.workunit.client.0.smithi115.stdout: 516: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:32.767 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vwnNacqC3qgBTe- off=2686003 len=76157 2022-04-22T17:55:32.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 2633 completed, throughput=4.99MB/sec 2022-04-22T17:55:32.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wahmmgHqHL93MoH off=2377543 len=2029744 2022-04-22T17:55:32.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 2634 completed, throughput=5MB/sec 2022-04-22T17:55:32.779 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rMCykMGK8g41lRx off=11586428 len=1404857 2022-04-22T17:55:32.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 2635 completed, throughput=5MB/sec 2022-04-22T17:55:32.783 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-INKdguGpPNw6nZP off=3030339 len=1151634 2022-04-22T17:55:32.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 2636 completed, throughput=5MB/sec 2022-04-22T17:55:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:33 smithi121 bash[36902]: audit 2022-04-22T17:55:32.474758+0000 mon.a (mon.0) 2515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:33.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:33 smithi115 bash[45495]: audit 2022-04-22T17:55:32.474758+0000 mon.a (mon.0) 2515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:33 smithi115 bash[46934]: audit 2022-04-22T17:55:32.474758+0000 mon.a (mon.0) 2515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:33.789 INFO:tasks.workunit.client.0.smithi115.stdout: 517: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:33.789 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UxJtEgW9612HTm1 off=5031031 len=1738647 2022-04-22T17:55:33.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 2637 completed, throughput=4.99MB/sec 2022-04-22T17:55:33.800 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8E7PFiuvmpcBM3f off=2700932 len=1526626 2022-04-22T17:55:33.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 2638 completed, throughput=5MB/sec 2022-04-22T17:55:33.808 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-toYW0UwX6N4Q1gz off=3578311 len=1231001 2022-04-22T17:55:33.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 2639 completed, throughput=5MB/sec 2022-04-22T17:55:33.815 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zBej6olSO1O_ksA off=1923572 len=1018631 2022-04-22T17:55:33.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 2640 completed, throughput=5MB/sec 2022-04-22T17:55:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:33.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:34 smithi121 bash[36902]: cluster 2022-04-22T17:55:32.525355+0000 mgr.y (mgr.34107) 1057 : cluster [DBG] pgmap v1665: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:34 smithi115 bash[45495]: cluster 2022-04-22T17:55:32.525355+0000 mgr.y (mgr.34107) 1057 : cluster [DBG] pgmap v1665: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:34 smithi115 bash[46934]: cluster 2022-04-22T17:55:32.525355+0000 mgr.y (mgr.34107) 1057 : cluster [DBG] pgmap v1665: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:34.820 INFO:tasks.workunit.client.0.smithi115.stdout: 518: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:34.820 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Dwe0T43yFpiBZiu off=89980 len=232369 2022-04-22T17:55:34.822 INFO:tasks.workunit.client.0.smithi115.stdout:op 2641 completed, throughput=4.99MB/sec 2022-04-22T17:55:34.822 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Sm0uOwhvYSc2smZ off=17551869 len=75616 2022-04-22T17:55:34.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 2642 completed, throughput=4.99MB/sec 2022-04-22T17:55:34.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n9JBFXQ-SREfNqo off=4327266 len=1317928 2022-04-22T17:55:34.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 2643 completed, throughput=4.99MB/sec 2022-04-22T17:55:34.826 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tLozese3vhaCE47 off=1086818 len=1581645 2022-04-22T17:55:34.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 2644 completed, throughput=5MB/sec 2022-04-22T17:55:34.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rLBSrRILagFb2E7 off=16042829 len=1660791 2022-04-22T17:55:34.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 2645 completed, throughput=5MB/sec 2022-04-22T17:55:34.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NmTdFnVOYcWZCQw off=750160 len=294636 2022-04-22T17:55:34.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 2646 completed, throughput=5MB/sec 2022-04-22T17:55:35.840 INFO:tasks.workunit.client.0.smithi115.stdout: 519: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:35.840 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5BaYB4SZhL3TEHi off=3298758 len=1291218 2022-04-22T17:55:35.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 2647 completed, throughput=4.99MB/sec 2022-04-22T17:55:35.850 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NP54KIDCJP1ookQ off=2481528 len=2085715 2022-04-22T17:55:35.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 2648 completed, throughput=5MB/sec 2022-04-22T17:55:35.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kqJ1guNZMeVGePY off=8830829 len=112909 2022-04-22T17:55:35.857 INFO:tasks.workunit.client.0.smithi115.stdout:op 2649 completed, throughput=5MB/sec 2022-04-22T17:55:35.857 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-spGtGXEhGJhji-a off=2913851 len=1469957 2022-04-22T17:55:35.858 INFO:tasks.workunit.client.0.smithi115.stdout:op 2650 completed, throughput=5MB/sec 2022-04-22T17:55:35.859 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TDjFyxts4QU_mIz off=7151415 len=581778 2022-04-22T17:55:35.861 INFO:tasks.workunit.client.0.smithi115.stdout:op 2651 completed, throughput=5MB/sec 2022-04-22T17:55:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:36 smithi121 bash[36902]: cluster 2022-04-22T17:55:34.526517+0000 mgr.y (mgr.34107) 1058 : cluster [DBG] pgmap v1666: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:36 smithi121 bash[36902]: audit 2022-04-22T17:55:35.093418+0000 mon.a (mon.0) 2516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:36 smithi121 bash[36902]: audit 2022-04-22T17:55:35.436795+0000 mon.a (mon.0) 2517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[45495]: cluster 2022-04-22T17:55:34.526517+0000 mgr.y (mgr.34107) 1058 : cluster [DBG] pgmap v1666: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[45495]: audit 2022-04-22T17:55:35.093418+0000 mon.a (mon.0) 2516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[45495]: audit 2022-04-22T17:55:35.436795+0000 mon.a (mon.0) 2517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[46934]: cluster 2022-04-22T17:55:34.526517+0000 mgr.y (mgr.34107) 1058 : cluster [DBG] pgmap v1666: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[46934]: audit 2022-04-22T17:55:35.093418+0000 mon.a (mon.0) 2516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:36 smithi115 bash[46934]: audit 2022-04-22T17:55:35.436795+0000 mon.a (mon.0) 2517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:36.861 INFO:tasks.workunit.client.0.smithi115.stdout: 520: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:36.862 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jakYJzCQ368LWmd off=16008303 len=600044 2022-04-22T17:55:36.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 2652 completed, throughput=4.99MB/sec 2022-04-22T17:55:36.869 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7zL1OG_gj-oHUao off=8501423 len=508302 2022-04-22T17:55:36.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 2653 completed, throughput=4.99MB/sec 2022-04-22T17:55:36.871 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tsq4hgm3AC1qf32 off=18314154 len=523089 2022-04-22T17:55:36.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 2654 completed, throughput=4.99MB/sec 2022-04-22T17:55:36.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NKt0djQp9Xwu_ME off=7512189 len=1209969 2022-04-22T17:55:36.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 2655 completed, throughput=5MB/sec 2022-04-22T17:55:36.883 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bn3PQX6LgOuphqD off=16217034 len=3741 2022-04-22T17:55:36.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 2656 completed, throughput=5MB/sec 2022-04-22T17:55:36.887 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zi5AkrFEfsmefUE off=5332361 len=1025992 2022-04-22T17:55:36.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 2657 completed, throughput=5MB/sec 2022-04-22T17:55:36.890 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EsbkyQfvp7nk8Rm off=6113520 len=456770 2022-04-22T17:55:36.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 2658 completed, throughput=5MB/sec 2022-04-22T17:55:36.891 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NsVSKOkhQqYvRPA off=14068606 len=1898233 2022-04-22T17:55:36.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 2659 completed, throughput=5MB/sec 2022-04-22T17:55:37.894 INFO:tasks.workunit.client.0.smithi115.stdout: 521: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:37.894 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CaeMWiwx1wz-k9O off=3442206 len=1850037 2022-04-22T17:55:37.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 2660 completed, throughput=5MB/sec 2022-04-22T17:55:37.900 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8FW6MSjbtqITwz5 off=4151774 len=1288268 2022-04-22T17:55:37.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 2661 completed, throughput=5MB/sec 2022-04-22T17:55:37.903 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VTiXm3V_bzZaBEi off=18314750 len=272387 2022-04-22T17:55:37.904 INFO:tasks.workunit.client.0.smithi115.stdout:op 2662 completed, throughput=5MB/sec 2022-04-22T17:55:37.904 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dnoGZ1j-ddz7tTo off=13967848 len=1336471 2022-04-22T17:55:37.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 2663 completed, throughput=5MB/sec 2022-04-22T17:55:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:38 smithi121 bash[36902]: cluster 2022-04-22T17:55:36.527657+0000 mgr.y (mgr.34107) 1059 : cluster [DBG] pgmap v1667: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:55:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:38 smithi121 bash[36902]: audit 2022-04-22T17:55:37.478190+0000 mon.a (mon.0) 2518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:38 smithi115 bash[45495]: cluster 2022-04-22T17:55:36.527657+0000 mgr.y (mgr.34107) 1059 : cluster [DBG] pgmap v1667: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:55:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:38 smithi115 bash[45495]: audit 2022-04-22T17:55:37.478190+0000 mon.a (mon.0) 2518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:38 smithi115 bash[46934]: cluster 2022-04-22T17:55:36.527657+0000 mgr.y (mgr.34107) 1059 : cluster [DBG] pgmap v1667: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T17:55:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:38 smithi115 bash[46934]: audit 2022-04-22T17:55:37.478190+0000 mon.a (mon.0) 2518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:38.911 INFO:tasks.workunit.client.0.smithi115.stdout: 522: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:38.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8sE5ok_whAKPzEx off=1623519 len=570165 2022-04-22T17:55:38.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 2664 completed, throughput=4.99MB/sec 2022-04-22T17:55:38.917 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3V0ekjCf2pGj9z8 off=6925422 len=945892 2022-04-22T17:55:38.924 INFO:tasks.workunit.client.0.smithi115.stdout:op 2665 completed, throughput=4.99MB/sec 2022-04-22T17:55:38.924 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dVtZa-xZtDfnxjh off=5096076 len=121134 2022-04-22T17:55:38.925 INFO:tasks.workunit.client.0.smithi115.stdout:op 2666 completed, throughput=4.99MB/sec 2022-04-22T17:55:38.925 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RQlcC80H0jKszkS off=1594249 len=1609327 2022-04-22T17:55:38.931 INFO:tasks.workunit.client.0.smithi115.stdout:op 2667 completed, throughput=5MB/sec 2022-04-22T17:55:38.931 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gDmOXfclhs4zUBc off=6529808 len=1332345 2022-04-22T17:55:38.936 INFO:tasks.workunit.client.0.smithi115.stdout:op 2668 completed, throughput=5MB/sec 2022-04-22T17:55:38.936 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UdnlSVKsRMus0Oy off=3385299 len=927271 2022-04-22T17:55:38.942 INFO:tasks.workunit.client.0.smithi115.stdout:op 2669 completed, throughput=5MB/sec 2022-04-22T17:55:39.943 INFO:tasks.workunit.client.0.smithi115.stdout: 523: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:39.943 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ipUric7nTuQwJVf off=4215604 len=908099 2022-04-22T17:55:39.946 INFO:tasks.workunit.client.0.smithi115.stdout:op 2670 completed, throughput=4.99MB/sec 2022-04-22T17:55:39.946 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uU_7fseQi7l2qC5 off=10185098 len=1195739 2022-04-22T17:55:39.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 2671 completed, throughput=5MB/sec 2022-04-22T17:55:39.949 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-afSaqTTS2sMIrq5 off=1283926 len=1598570 2022-04-22T17:55:39.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 2672 completed, throughput=5MB/sec 2022-04-22T17:55:39.958 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uXsKJjYJ1udajo1 off=4170658 len=543524 2022-04-22T17:55:39.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 2673 completed, throughput=5MB/sec 2022-04-22T17:55:39.964 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ImT07d1DpdhHDdK off=1204786 len=17948 2022-04-22T17:55:39.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 2674 completed, throughput=5MB/sec 2022-04-22T17:55:39.968 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LDmUW2wB8wIngEh off=2332876 len=1481343 2022-04-22T17:55:39.974 INFO:tasks.workunit.client.0.smithi115.stdout:op 2675 completed, throughput=5MB/sec 2022-04-22T17:55:40.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[45495]: cluster 2022-04-22T17:55:38.528806+0000 mgr.y (mgr.34107) 1060 : cluster [DBG] pgmap v1668: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:40.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[45495]: audit 2022-04-22T17:55:39.855103+0000 mon.a (mon.0) 2519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:55:40.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[45495]: audit 2022-04-22T17:55:39.858184+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:55:40.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[45495]: audit 2022-04-22T17:55:39.860002+0000 mon.a (mon.0) 2521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:55:40.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[46934]: cluster 2022-04-22T17:55:38.528806+0000 mgr.y (mgr.34107) 1060 : cluster [DBG] pgmap v1668: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:40.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[46934]: audit 2022-04-22T17:55:39.855103+0000 mon.a (mon.0) 2519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:55:40.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[46934]: audit 2022-04-22T17:55:39.858184+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:55:40.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:40 smithi115 bash[46934]: audit 2022-04-22T17:55:39.860002+0000 mon.a (mon.0) 2521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:55:40.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:40 smithi121 bash[36902]: cluster 2022-04-22T17:55:38.528806+0000 mgr.y (mgr.34107) 1060 : cluster [DBG] pgmap v1668: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:55:40.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:40 smithi121 bash[36902]: audit 2022-04-22T17:55:39.855103+0000 mon.a (mon.0) 2519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:55:40.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:40 smithi121 bash[36902]: audit 2022-04-22T17:55:39.858184+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:55:40.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:40 smithi121 bash[36902]: audit 2022-04-22T17:55:39.860002+0000 mon.a (mon.0) 2521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:55:40.886 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:55:40.974 INFO:tasks.workunit.client.0.smithi115.stdout: 524: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:40.974 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vJjzvXJ_joCAp77 off=14754335 len=726835 2022-04-22T17:55:40.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 2676 completed, throughput=4.99MB/sec 2022-04-22T17:55:40.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9S_53mcONJnOUHK off=3740848 len=375696 2022-04-22T17:55:40.981 INFO:tasks.workunit.client.0.smithi115.stdout:op 2677 completed, throughput=4.99MB/sec 2022-04-22T17:55:40.981 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KboI8PmK3rnqt1d off=1781074 len=1386250 2022-04-22T17:55:40.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 2678 completed, throughput=5MB/sec 2022-04-22T17:55:40.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--GDUq4mCDI3QnzK off=4000300 len=1175364 2022-04-22T17:55:40.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 2679 completed, throughput=5MB/sec 2022-04-22T17:55:40.992 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CFE7XXkJE9jaB73 off=10577309 len=115895 2022-04-22T17:55:40.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 2680 completed, throughput=5MB/sec 2022-04-22T17:55:40.993 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C7687DAjxQtmAeD off=5144463 len=1638033 2022-04-22T17:55:41.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 2681 completed, throughput=5MB/sec 2022-04-22T17:55:41.230 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:55:41.236 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (18m) 8m ago 32m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (32m) 9m ago 32m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (20m) 8m ago 33m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (20m) 9m ago 33m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (28m) 9m ago 37m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (28m) 8m ago 39m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (27m) 8m ago 40m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (26m) 9m ago 38m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:55:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (27m) 8m ago 38m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (19m) 8m ago 32m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (19m) 9m ago 32m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (25m) 8m ago 37m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (24m) 8m ago 36m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (24m) 8m ago 36m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (23m) 8m ago 35m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (23m) 9m ago 35m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (22m) 9m ago 35m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:55:41.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (21m) 9m ago 34m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:55:41.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (21m) 9m ago 34m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:55:41.239 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (19m) 9m ago 32m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:55:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:41 smithi121 bash[36902]: audit 2022-04-22T17:55:40.096181+0000 mon.a (mon.0) 2522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:41 smithi121 bash[36902]: audit 2022-04-22T17:55:40.222075+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:55:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:41 smithi121 bash[36902]: audit 2022-04-22T17:55:40.440251+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[45495]: audit 2022-04-22T17:55:40.096181+0000 mon.a (mon.0) 2522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:41.579 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[45495]: audit 2022-04-22T17:55:40.222075+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:55:41.579 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[45495]: audit 2022-04-22T17:55:40.440251+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:41.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[46934]: audit 2022-04-22T17:55:40.096181+0000 mon.a (mon.0) 2522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:41.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[46934]: audit 2022-04-22T17:55:40.222075+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:55:41.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[46934]: audit 2022-04-22T17:55:40.440251+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:41.635 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:55:41.635 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:55:41.635 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:55:41.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:41.635 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:55:41.636 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:55:41.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:55:41.637 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:55:41.637 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:55:42.001 INFO:tasks.workunit.client.0.smithi115.stdout: 525: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:42.164 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FaQPcpk1IOrReOx off=9561711 len=868912 2022-04-22T17:55:42.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 2682 completed, throughput=4.99MB/sec 2022-04-22T17:55:42.165 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-076-d14ZXrouQZ3 off=5674134 len=150053 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 2683 completed, throughput=4.99MB/sec 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c0Vt6Ks6N-0yrAs off=15631602 len=1772212 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 2684 completed, throughput=5MB/sec 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qX9qP6SB8v8s94V off=561238 len=228802 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 2685 completed, throughput=5MB/sec 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8hlip0xD4gwCeID off=6674069 len=1338726 2022-04-22T17:55:42.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 2686 completed, throughput=5MB/sec 2022-04-22T17:55:42.167 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:41.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:42 smithi121 bash[36902]: cluster 2022-04-22T17:55:40.529600+0000 mgr.y (mgr.34107) 1061 : cluster [DBG] pgmap v1669: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:55:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:42 smithi121 bash[36902]: audit 2022-04-22T17:55:40.863732+0000 mgr.y (mgr.34107) 1062 : audit [DBG] from='client.45457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:42 smithi121 bash[36902]: audit 2022-04-22T17:55:41.206087+0000 mgr.y (mgr.34107) 1063 : audit [DBG] from='client.45463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:42 smithi121 bash[36902]: audit 2022-04-22T17:55:41.627943+0000 mon.a (mon.0) 2525 : audit [DBG] from='client.? 172.21.15.115:0/3423116292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[45495]: cluster 2022-04-22T17:55:40.529600+0000 mgr.y (mgr.34107) 1061 : cluster [DBG] pgmap v1669: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:55:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[45495]: audit 2022-04-22T17:55:40.863732+0000 mgr.y (mgr.34107) 1062 : audit [DBG] from='client.45457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[45495]: audit 2022-04-22T17:55:41.206087+0000 mgr.y (mgr.34107) 1063 : audit [DBG] from='client.45463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[45495]: audit 2022-04-22T17:55:41.627943+0000 mon.a (mon.0) 2525 : audit [DBG] from='client.? 172.21.15.115:0/3423116292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[46934]: cluster 2022-04-22T17:55:40.529600+0000 mgr.y (mgr.34107) 1061 : cluster [DBG] pgmap v1669: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T17:55:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[46934]: audit 2022-04-22T17:55:40.863732+0000 mgr.y (mgr.34107) 1062 : audit [DBG] from='client.45457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[46934]: audit 2022-04-22T17:55:41.206087+0000 mgr.y (mgr.34107) 1063 : audit [DBG] from='client.45463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:55:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:42 smithi115 bash[46934]: audit 2022-04-22T17:55:41.627943+0000 mon.a (mon.0) 2525 : audit [DBG] from='client.? 172.21.15.115:0/3423116292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:55:43.027 INFO:tasks.workunit.client.0.smithi115.stdout: 526: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:43.028 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZqSnrN_Hb2jf0bi off=9506222 len=283339 2022-04-22T17:55:43.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 2687 completed, throughput=4.99MB/sec 2022-04-22T17:55:43.032 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QJvJU3KhDGX_0Av off=12278016 len=1769977 2022-04-22T17:55:43.035 INFO:tasks.workunit.client.0.smithi115.stdout:op 2688 completed, throughput=4.99MB/sec 2022-04-22T17:55:43.035 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4H4LqKnl2QEGkOC off=7165038 len=405367 2022-04-22T17:55:43.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 2689 completed, throughput=5MB/sec 2022-04-22T17:55:43.037 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s-U_uEBGpmHjLCi off=17094718 len=1396283 2022-04-22T17:55:43.040 INFO:tasks.workunit.client.0.smithi115.stdout:op 2690 completed, throughput=5MB/sec 2022-04-22T17:55:43.040 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rvmiNop-Q5dO0Dd off=9432844 len=1232289 2022-04-22T17:55:43.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 2691 completed, throughput=5MB/sec 2022-04-22T17:55:43.047 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Twx-NKHJM9O4Fjj off=1089731 len=1360037 2022-04-22T17:55:43.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 2692 completed, throughput=5MB/sec 2022-04-22T17:55:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:43 smithi121 bash[36902]: audit 2022-04-22T17:55:42.481082+0000 mon.a (mon.0) 2526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:43 smithi115 bash[45495]: audit 2022-04-22T17:55:42.481082+0000 mon.a (mon.0) 2526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:43 smithi115 bash[46934]: audit 2022-04-22T17:55:42.481082+0000 mon.a (mon.0) 2526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:43.587Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:44.053 INFO:tasks.workunit.client.0.smithi115.stdout: 527: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:44.053 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qQ3bDh1_zLVuuZY off=3255597 len=1955084 2022-04-22T17:55:44.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 2693 completed, throughput=5MB/sec 2022-04-22T17:55:44.063 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Us-ZTrBufwP4XSn off=15634270 len=1095077 2022-04-22T17:55:44.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 2694 completed, throughput=5MB/sec 2022-04-22T17:55:44.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ItYgBxFQnCprMd2 off=11049151 len=1195018 2022-04-22T17:55:44.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 2695 completed, throughput=5MB/sec 2022-04-22T17:55:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:44 smithi121 bash[36902]: cluster 2022-04-22T17:55:42.530630+0000 mgr.y (mgr.34107) 1064 : cluster [DBG] pgmap v1670: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:44 smithi115 bash[45495]: cluster 2022-04-22T17:55:42.530630+0000 mgr.y (mgr.34107) 1064 : cluster [DBG] pgmap v1670: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:44 smithi115 bash[46934]: cluster 2022-04-22T17:55:42.530630+0000 mgr.y (mgr.34107) 1064 : cluster [DBG] pgmap v1670: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:55:45.071 INFO:tasks.workunit.client.0.smithi115.stdout: 528: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:45.071 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LFZkehQxvL161iI off=2711773 len=1642819 2022-04-22T17:55:45.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 2696 completed, throughput=4.99MB/sec 2022-04-22T17:55:45.074 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W57OjY_4wZ3RKc1 off=851723 len=1363277 2022-04-22T17:55:45.083 INFO:tasks.workunit.client.0.smithi115.stdout:op 2697 completed, throughput=5MB/sec 2022-04-22T17:55:45.083 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-93DKAL3eos1yCyX off=14430883 len=1482470 2022-04-22T17:55:45.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 2698 completed, throughput=5MB/sec 2022-04-22T17:55:45.089 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VNKUc3qWncxM-2_ off=5108848 len=1586743 2022-04-22T17:55:45.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 2699 completed, throughput=5MB/sec 2022-04-22T17:55:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:45 smithi121 bash[36902]: audit 2022-04-22T17:55:45.098669+0000 mon.a (mon.0) 2527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:45 smithi115 bash[45495]: audit 2022-04-22T17:55:45.098669+0000 mon.a (mon.0) 2527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:45 smithi115 bash[46934]: audit 2022-04-22T17:55:45.098669+0000 mon.a (mon.0) 2527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:46.092 INFO:tasks.workunit.client.0.smithi115.stdout: 529: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:46.092 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6uaMYblRAKisOZa off=11676731 len=113665 2022-04-22T17:55:46.093 INFO:tasks.workunit.client.0.smithi115.stdout:op 2700 completed, throughput=4.99MB/sec 2022-04-22T17:55:46.093 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gSdtfLh0Wueu9mE off=1430057 len=1280780 2022-04-22T17:55:46.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 2701 completed, throughput=4.99MB/sec 2022-04-22T17:55:46.101 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4YU_6cz7g5khjTF off=12073485 len=1576975 2022-04-22T17:55:46.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 2702 completed, throughput=5MB/sec 2022-04-22T17:55:46.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Als2W7klskjjw1w off=12655373 len=263773 2022-04-22T17:55:46.120 INFO:tasks.workunit.client.0.smithi115.stdout:op 2703 completed, throughput=5MB/sec 2022-04-22T17:55:46.120 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TeqrnYnDZZ0So2a off=4686361 len=930651 2022-04-22T17:55:46.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 2704 completed, throughput=5MB/sec 2022-04-22T17:55:46.122 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bH3H8Z2i_V1Dnho off=723731 len=1231220 2022-04-22T17:55:46.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 2705 completed, throughput=5MB/sec 2022-04-22T17:55:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:46 smithi121 bash[36902]: cluster 2022-04-22T17:55:44.531733+0000 mgr.y (mgr.34107) 1065 : cluster [DBG] pgmap v1671: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:55:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:46 smithi121 bash[36902]: audit 2022-04-22T17:55:45.444142+0000 mon.a (mon.0) 2528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:46 smithi115 bash[45495]: cluster 2022-04-22T17:55:44.531733+0000 mgr.y (mgr.34107) 1065 : cluster [DBG] pgmap v1671: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:55:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:46 smithi115 bash[45495]: audit 2022-04-22T17:55:45.444142+0000 mon.a (mon.0) 2528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:46 smithi115 bash[46934]: cluster 2022-04-22T17:55:44.531733+0000 mgr.y (mgr.34107) 1065 : cluster [DBG] pgmap v1671: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:55:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:46 smithi115 bash[46934]: audit 2022-04-22T17:55:45.444142+0000 mon.a (mon.0) 2528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:47.132 INFO:tasks.workunit.client.0.smithi115.stdout: 530: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:47.164 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6kZDl7SpnXXtkcL off=14922460 len=561623 2022-04-22T17:55:47.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 2706 completed, throughput=4.99MB/sec 2022-04-22T17:55:47.165 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-A-7CxMEnLoLLiJr off=10754688 len=1995340 2022-04-22T17:55:47.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 2707 completed, throughput=5MB/sec 2022-04-22T17:55:47.167 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L5n5tx8NorXfFCa off=1548039 len=1479826 2022-04-22T17:55:47.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 2708 completed, throughput=5MB/sec 2022-04-22T17:55:47.168 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sxOWUFqM0p8N-iL off=16343286 len=373215 2022-04-22T17:55:47.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 2709 completed, throughput=5MB/sec 2022-04-22T17:55:47.168 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sXLEkBsohsKrBVR off=473139 len=817224 2022-04-22T17:55:47.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 2710 completed, throughput=5MB/sec 2022-04-22T17:55:48.147 INFO:tasks.workunit.client.0.smithi115.stdout: 531: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:48.147 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3nc38Zp7ss8LVkc off=15079471 len=1938433 2022-04-22T17:55:48.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 2711 completed, throughput=5MB/sec 2022-04-22T17:55:48.155 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NyO7bObdkdZBfPI off=1807809 len=1762388 2022-04-22T17:55:48.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 2712 completed, throughput=5MB/sec 2022-04-22T17:55:48.162 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-797OV8ZYfy4EaoU off=3602575 len=427795 2022-04-22T17:55:48.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 2713 completed, throughput=5MB/sec 2022-04-22T17:55:48.167 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gtfaNemvxCEQa80 off=5216777 len=1402533 2022-04-22T17:55:48.174 INFO:tasks.workunit.client.0.smithi115.stdout:op 2714 completed, throughput=5MB/sec 2022-04-22T17:55:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:48 smithi121 bash[36902]: cluster 2022-04-22T17:55:46.532813+0000 mgr.y (mgr.34107) 1066 : cluster [DBG] pgmap v1672: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:55:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:48 smithi121 bash[36902]: audit 2022-04-22T17:55:47.484042+0000 mon.a (mon.0) 2529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:48 smithi115 bash[45495]: cluster 2022-04-22T17:55:46.532813+0000 mgr.y (mgr.34107) 1066 : cluster [DBG] pgmap v1672: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:55:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:48 smithi115 bash[45495]: audit 2022-04-22T17:55:47.484042+0000 mon.a (mon.0) 2529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:48 smithi115 bash[46934]: cluster 2022-04-22T17:55:46.532813+0000 mgr.y (mgr.34107) 1066 : cluster [DBG] pgmap v1672: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:55:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:48 smithi115 bash[46934]: audit 2022-04-22T17:55:47.484042+0000 mon.a (mon.0) 2529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:49.174 INFO:tasks.workunit.client.0.smithi115.stdout: 532: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:49.174 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CtmeNbfEBobSxCk off=7868135 len=1821769 2022-04-22T17:55:49.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 2715 completed, throughput=5MB/sec 2022-04-22T17:55:49.179 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bi6pGft7OYjFc6H off=13882398 len=451084 2022-04-22T17:55:49.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 2716 completed, throughput=5MB/sec 2022-04-22T17:55:49.185 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c5bxhLlt-yCIutr off=11004756 len=2096489 2022-04-22T17:55:49.191 INFO:tasks.workunit.client.0.smithi115.stdout:op 2717 completed, throughput=5MB/sec 2022-04-22T17:55:49.192 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b5hhqH8Lsp_75PY off=5945372 len=107073 2022-04-22T17:55:49.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 2718 completed, throughput=5MB/sec 2022-04-22T17:55:49.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KT9r5eGRJoasKD7 off=9706244 len=1938155 2022-04-22T17:55:49.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 2719 completed, throughput=5MB/sec 2022-04-22T17:55:50.203 INFO:tasks.workunit.client.0.smithi115.stdout: 533: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:50.203 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3APRrHj7ZIqNr22 off=17518324 len=1645702 2022-04-22T17:55:50.207 INFO:tasks.workunit.client.0.smithi115.stdout:op 2720 completed, throughput=5MB/sec 2022-04-22T17:55:50.207 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G_bot9vTOA0vI7t off=4936806 len=1096751 2022-04-22T17:55:50.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 2721 completed, throughput=5MB/sec 2022-04-22T17:55:50.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v1hg0yeHQ4Dy1Xa off=19242551 len=1655705 2022-04-22T17:55:50.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 2722 completed, throughput=5MB/sec 2022-04-22T17:55:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:50 smithi121 bash[36902]: cluster 2022-04-22T17:55:48.533939+0000 mgr.y (mgr.34107) 1067 : cluster [DBG] pgmap v1673: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:50 smithi121 bash[36902]: audit 2022-04-22T17:55:50.101199+0000 mon.a (mon.0) 2530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:50 smithi115 bash[45495]: cluster 2022-04-22T17:55:48.533939+0000 mgr.y (mgr.34107) 1067 : cluster [DBG] pgmap v1673: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:50 smithi115 bash[45495]: audit 2022-04-22T17:55:50.101199+0000 mon.a (mon.0) 2530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:50 smithi115 bash[46934]: cluster 2022-04-22T17:55:48.533939+0000 mgr.y (mgr.34107) 1067 : cluster [DBG] pgmap v1673: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:55:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:50 smithi115 bash[46934]: audit 2022-04-22T17:55:50.101199+0000 mon.a (mon.0) 2530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:51.221 INFO:tasks.workunit.client.0.smithi115.stdout: 534: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:51.221 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hpu8injfkzapaYu off=10567904 len=1697696 2022-04-22T17:55:51.230 INFO:tasks.workunit.client.0.smithi115.stdout:op 2723 completed, throughput=5MB/sec 2022-04-22T17:55:51.230 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NrCk8huwszcO7T4 off=13521719 len=1652314 2022-04-22T17:55:51.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 2724 completed, throughput=5MB/sec 2022-04-22T17:55:51.239 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SDG-an_4tg2ubyq off=1935666 len=1801471 2022-04-22T17:55:51.245 INFO:tasks.workunit.client.0.smithi115.stdout:op 2725 completed, throughput=5MB/sec 2022-04-22T17:55:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:51 smithi115 bash[45495]: audit 2022-04-22T17:55:50.447933+0000 mon.a (mon.0) 2531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:51 smithi115 bash[46934]: audit 2022-04-22T17:55:50.447933+0000 mon.a (mon.0) 2531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:51 smithi121 bash[36902]: audit 2022-04-22T17:55:50.447933+0000 mon.a (mon.0) 2531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:51.704Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:52.248 INFO:tasks.workunit.client.0.smithi115.stdout: 535: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:52.248 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7ihR0W7slQ2tQAu off=14501426 len=1872197 2022-04-22T17:55:52.254 INFO:tasks.workunit.client.0.smithi115.stdout:op 2726 completed, throughput=5MB/sec 2022-04-22T17:55:52.255 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yTOU8Rxkqc8sI3a off=6037314 len=1215302 2022-04-22T17:55:52.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 2727 completed, throughput=5MB/sec 2022-04-22T17:55:52.256 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V-Bx8dP1H4yoQVd off=7028763 len=2087363 2022-04-22T17:55:52.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 2728 completed, throughput=5MB/sec 2022-04-22T17:55:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:52 smithi115 bash[45495]: cluster 2022-04-22T17:55:50.534963+0000 mgr.y (mgr.34107) 1068 : cluster [DBG] pgmap v1674: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:55:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:52 smithi115 bash[46934]: cluster 2022-04-22T17:55:50.534963+0000 mgr.y (mgr.34107) 1068 : cluster [DBG] pgmap v1674: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:55:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:52 smithi121 bash[36902]: cluster 2022-04-22T17:55:50.534963+0000 mgr.y (mgr.34107) 1068 : cluster [DBG] pgmap v1674: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:55:53.260 INFO:tasks.workunit.client.0.smithi115.stdout: 536: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:53.260 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ktSCm8nrAtgBf8P off=2853301 len=195958 2022-04-22T17:55:53.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 2729 completed, throughput=4.99MB/sec 2022-04-22T17:55:53.265 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--rckMRKZcSHujVu off=12837868 len=634732 2022-04-22T17:55:53.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 2730 completed, throughput=4.99MB/sec 2022-04-22T17:55:53.268 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_NYBcm9IYUBcf8E off=13532664 len=359870 2022-04-22T17:55:53.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 2731 completed, throughput=4.99MB/sec 2022-04-22T17:55:53.272 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9_YvaiOl1zlCZ4L off=21070794 len=756429 2022-04-22T17:55:53.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 2732 completed, throughput=4.99MB/sec 2022-04-22T17:55:53.274 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lq0C4n37m0Glwsr off=92086 len=51560 2022-04-22T17:55:53.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 2733 completed, throughput=5MB/sec 2022-04-22T17:55:53.274 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ygu32vU6W_Z3L0- off=5150507 len=1423541 2022-04-22T17:55:53.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2734 completed, throughput=5MB/sec 2022-04-22T17:55:53.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3NZyMmWuXozwmp5 off=10808586 len=211243 2022-04-22T17:55:53.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 2735 completed, throughput=5MB/sec 2022-04-22T17:55:53.281 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GLEX1uEbEH8snEA off=4730904 len=179862 2022-04-22T17:55:53.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 2736 completed, throughput=5MB/sec 2022-04-22T17:55:53.286 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IByPujevD56B-_4 off=0 len=1136070 2022-04-22T17:55:53.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 2737 completed, throughput=5MB/sec 2022-04-22T17:55:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:53 smithi115 bash[45495]: audit 2022-04-22T17:55:52.486995+0000 mon.a (mon.0) 2532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:53 smithi115 bash[46934]: audit 2022-04-22T17:55:52.486995+0000 mon.a (mon.0) 2532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:53 smithi121 bash[36902]: audit 2022-04-22T17:55:52.486995+0000 mon.a (mon.0) 2532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:55:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:55:53.587Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:55:54.291 INFO:tasks.workunit.client.0.smithi115.stdout: 537: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:54.291 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HLrIvr0QdNO-tUf off=5401239 len=1300001 2022-04-22T17:55:54.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 2738 completed, throughput=4.99MB/sec 2022-04-22T17:55:54.294 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mZiCVeKoKEWd8u_ off=516462 len=860502 2022-04-22T17:55:54.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 2739 completed, throughput=4.99MB/sec 2022-04-22T17:55:54.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0Gu2AxoQnGPYl3p off=15690607 len=616893 2022-04-22T17:55:54.306 INFO:tasks.workunit.client.0.smithi115.stdout:op 2740 completed, throughput=5MB/sec 2022-04-22T17:55:54.306 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X7A7FHHIQ2g2lHI off=9349323 len=992350 2022-04-22T17:55:54.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 2741 completed, throughput=5MB/sec 2022-04-22T17:55:54.308 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1NtiIpYK1PxG8RX off=8022057 len=1958632 2022-04-22T17:55:54.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 2742 completed, throughput=5MB/sec 2022-04-22T17:55:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:54 smithi115 bash[46934]: cluster 2022-04-22T17:55:52.536074+0000 mgr.y (mgr.34107) 1069 : cluster [DBG] pgmap v1675: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:54 smithi115 bash[45495]: cluster 2022-04-22T17:55:52.536074+0000 mgr.y (mgr.34107) 1069 : cluster [DBG] pgmap v1675: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:54 smithi121 bash[36902]: cluster 2022-04-22T17:55:52.536074+0000 mgr.y (mgr.34107) 1069 : cluster [DBG] pgmap v1675: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:55:55.312 INFO:tasks.workunit.client.0.smithi115.stdout: 538: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:55.312 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ndBd1mouKrJzBif off=10607228 len=1346368 2022-04-22T17:55:55.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 2743 completed, throughput=4.99MB/sec 2022-04-22T17:55:55.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BPdac8QoKCSRUYP off=17405684 len=1455526 2022-04-22T17:55:55.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 2744 completed, throughput=5MB/sec 2022-04-22T17:55:55.329 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nxLzE5s09M9atbB off=5438789 len=409533 2022-04-22T17:55:55.334 INFO:tasks.workunit.client.0.smithi115.stdout:op 2745 completed, throughput=5MB/sec 2022-04-22T17:55:55.334 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wRCktLxqpmzmbVJ off=132807 len=1515590 2022-04-22T17:55:55.342 INFO:tasks.workunit.client.0.smithi115.stdout:op 2746 completed, throughput=5MB/sec 2022-04-22T17:55:55.342 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yDAjPdwcS9NTdPi off=4150268 len=1155870 2022-04-22T17:55:55.345 INFO:tasks.workunit.client.0.smithi115.stdout:op 2747 completed, throughput=5MB/sec 2022-04-22T17:55:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:55 smithi115 bash[45495]: audit 2022-04-22T17:55:55.103947+0000 mon.a (mon.0) 2533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:55.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:55 smithi115 bash[46934]: audit 2022-04-22T17:55:55.103947+0000 mon.a (mon.0) 2533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:55 smithi121 bash[36902]: audit 2022-04-22T17:55:55.103947+0000 mon.a (mon.0) 2533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:55:56.345 INFO:tasks.workunit.client.0.smithi115.stdout: 539: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:56.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jLD82t7mUDnteao off=159062 len=1157450 2022-04-22T17:55:56.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 2748 completed, throughput=4.99MB/sec 2022-04-22T17:55:56.355 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XdoFHVbC3ZcciPs off=7505049 len=2027591 2022-04-22T17:55:56.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 2749 completed, throughput=5MB/sec 2022-04-22T17:55:56.366 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BSKuoXUAklq8ffs off=1749391 len=1635912 2022-04-22T17:55:56.368 INFO:tasks.workunit.client.0.smithi115.stdout:op 2750 completed, throughput=5MB/sec 2022-04-22T17:55:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:56 smithi121 bash[36902]: cluster 2022-04-22T17:55:54.537291+0000 mgr.y (mgr.34107) 1070 : cluster [DBG] pgmap v1676: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:56 smithi121 bash[36902]: audit 2022-04-22T17:55:55.451735+0000 mon.a (mon.0) 2534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:56 smithi115 bash[45495]: cluster 2022-04-22T17:55:54.537291+0000 mgr.y (mgr.34107) 1070 : cluster [DBG] pgmap v1676: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:56 smithi115 bash[45495]: audit 2022-04-22T17:55:55.451735+0000 mon.a (mon.0) 2534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:56.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:56 smithi115 bash[46934]: cluster 2022-04-22T17:55:54.537291+0000 mgr.y (mgr.34107) 1070 : cluster [DBG] pgmap v1676: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:55:56.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:56 smithi115 bash[46934]: audit 2022-04-22T17:55:55.451735+0000 mon.a (mon.0) 2534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:55:57.368 INFO:tasks.workunit.client.0.smithi115.stdout: 540: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:57.369 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-crItOaghNvgqVxC off=6151113 len=610646 2022-04-22T17:55:57.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 2751 completed, throughput=4.99MB/sec 2022-04-22T17:55:57.371 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tubVaOTTkdfSVcr off=1338259 len=649632 2022-04-22T17:55:57.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 2752 completed, throughput=4.99MB/sec 2022-04-22T17:55:57.373 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WcV6fpOtj5HVelh off=5905330 len=843250 2022-04-22T17:55:57.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 2753 completed, throughput=5MB/sec 2022-04-22T17:55:57.380 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UFrs_j8IUD8Csj3 off=5250510 len=158401 2022-04-22T17:55:57.383 INFO:tasks.workunit.client.0.smithi115.stdout:op 2754 completed, throughput=5MB/sec 2022-04-22T17:55:57.384 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jF0wnEeTBDkeENR off=9921298 len=1986943 2022-04-22T17:55:57.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 2755 completed, throughput=5MB/sec 2022-04-22T17:55:57.388 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3J2etMdF0xvtvQ off=2724038 len=1384833 2022-04-22T17:55:57.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 2756 completed, throughput=5MB/sec 2022-04-22T17:55:58.396 INFO:tasks.workunit.client.0.smithi115.stdout: 541: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:58.396 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5RFzis7dVr0orwC off=1658231 len=631806 2022-04-22T17:55:58.401 INFO:tasks.workunit.client.0.smithi115.stdout:op 2757 completed, throughput=4.99MB/sec 2022-04-22T17:55:58.401 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GJ0sAifooTfae-Y off=17879901 len=1477130 2022-04-22T17:55:58.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 2758 completed, throughput=5MB/sec 2022-04-22T17:55:58.411 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vuyy_BXZdIln92K off=9289878 len=1056952 2022-04-22T17:55:58.417 INFO:tasks.workunit.client.0.smithi115.stdout:op 2759 completed, throughput=5MB/sec 2022-04-22T17:55:58.417 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pJTY3Um-Jg7jN78 off=1173810 len=1938344 2022-04-22T17:55:58.425 INFO:tasks.workunit.client.0.smithi115.stdout:op 2760 completed, throughput=5MB/sec 2022-04-22T17:55:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:58 smithi121 bash[36902]: cluster 2022-04-22T17:55:56.538431+0000 mgr.y (mgr.34107) 1071 : cluster [DBG] pgmap v1677: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:55:58 smithi121 bash[36902]: audit 2022-04-22T17:55:57.490207+0000 mon.a (mon.0) 2535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:58 smithi115 bash[46934]: cluster 2022-04-22T17:55:56.538431+0000 mgr.y (mgr.34107) 1071 : cluster [DBG] pgmap v1677: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:55:58 smithi115 bash[46934]: audit 2022-04-22T17:55:57.490207+0000 mon.a (mon.0) 2535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:58 smithi115 bash[45495]: cluster 2022-04-22T17:55:56.538431+0000 mgr.y (mgr.34107) 1071 : cluster [DBG] pgmap v1677: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:55:58.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:55:58 smithi115 bash[45495]: audit 2022-04-22T17:55:57.490207+0000 mon.a (mon.0) 2535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:55:59.425 INFO:tasks.workunit.client.0.smithi115.stdout: 542: throughput=4.99MB/sec pending data=0 2022-04-22T17:55:59.425 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8__SBZoqQGd6EkC off=6157705 len=2015802 2022-04-22T17:55:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:op 2761 completed, throughput=4.99MB/sec 2022-04-22T17:55:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Afw6lOFbM5nc04x off=0 len=1120578 2022-04-22T17:55:59.434 INFO:tasks.workunit.client.0.smithi115.stdout:op 2762 completed, throughput=5MB/sec 2022-04-22T17:55:59.435 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s24yUwXjFppl6Rx off=2431737 len=871882 2022-04-22T17:55:59.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 2763 completed, throughput=5MB/sec 2022-04-22T17:55:59.442 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VYTc_9K81mAV5vj off=5486945 len=809575 2022-04-22T17:55:59.444 INFO:tasks.workunit.client.0.smithi115.stdout:op 2764 completed, throughput=5MB/sec 2022-04-22T17:55:59.444 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9t5aE6RGnppprYD off=7603849 len=2016142 2022-04-22T17:55:59.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 2765 completed, throughput=5MB/sec 2022-04-22T17:56:00.448 INFO:tasks.workunit.client.0.smithi115.stdout: 543: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:00.448 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7mNoSv3EPVwgH25 off=20418634 len=234981 2022-04-22T17:56:00.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 2766 completed, throughput=4.99MB/sec 2022-04-22T17:56:00.450 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0eoz4XatIudiiEA off=1268391 len=364265 2022-04-22T17:56:00.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 2767 completed, throughput=5MB/sec 2022-04-22T17:56:00.451 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xU_Le0QIVHFAywi off=4460610 len=391694 2022-04-22T17:56:00.457 INFO:tasks.workunit.client.0.smithi115.stdout:op 2768 completed, throughput=5MB/sec 2022-04-22T17:56:00.457 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6vpMXsk8-Rqm-2Q off=19938087 len=634999 2022-04-22T17:56:00.462 INFO:tasks.workunit.client.0.smithi115.stdout:op 2769 completed, throughput=5MB/sec 2022-04-22T17:56:00.462 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XU1UksJUO5lGFaO off=182737 len=466074 2022-04-22T17:56:00.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 2770 completed, throughput=5MB/sec 2022-04-22T17:56:00.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zuJXifZqnbNBYpI off=1693286 len=1922671 2022-04-22T17:56:00.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 2771 completed, throughput=5MB/sec 2022-04-22T17:56:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:00 smithi121 bash[36902]: cluster 2022-04-22T17:55:58.539640+0000 mgr.y (mgr.34107) 1072 : cluster [DBG] pgmap v1678: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:00 smithi121 bash[36902]: audit 2022-04-22T17:56:00.106839+0000 mon.a (mon.0) 2536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:00 smithi115 bash[45495]: cluster 2022-04-22T17:55:58.539640+0000 mgr.y (mgr.34107) 1072 : cluster [DBG] pgmap v1678: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:00 smithi115 bash[45495]: audit 2022-04-22T17:56:00.106839+0000 mon.a (mon.0) 2536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:00 smithi115 bash[46934]: cluster 2022-04-22T17:55:58.539640+0000 mgr.y (mgr.34107) 1072 : cluster [DBG] pgmap v1678: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:00 smithi115 bash[46934]: audit 2022-04-22T17:56:00.106839+0000 mon.a (mon.0) 2536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:01.468 INFO:tasks.workunit.client.0.smithi115.stdout: 544: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:01.468 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HiM3QClEKggtN74 off=1468365 len=1373845 2022-04-22T17:56:01.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 2772 completed, throughput=4.99MB/sec 2022-04-22T17:56:01.476 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-npe_fDlORFGtBlC off=781395 len=1867340 2022-04-22T17:56:01.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 2773 completed, throughput=5MB/sec 2022-04-22T17:56:01.480 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--eo307AV7cgLsjx off=9414544 len=1399614 2022-04-22T17:56:01.487 INFO:tasks.workunit.client.0.smithi115.stdout:op 2774 completed, throughput=5MB/sec 2022-04-22T17:56:01.487 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G6m8eu3Jo82IGZi off=2270404 len=121364 2022-04-22T17:56:01.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 2775 completed, throughput=5MB/sec 2022-04-22T17:56:01.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--oABiszB0U2AuDk off=20427922 len=1954425 2022-04-22T17:56:01.500 INFO:tasks.workunit.client.0.smithi115.stdout:op 2776 completed, throughput=5MB/sec 2022-04-22T17:56:01.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:01 smithi115 bash[46934]: audit 2022-04-22T17:56:00.455056+0000 mon.a (mon.0) 2537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:01.713 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:01 smithi115 bash[45495]: audit 2022-04-22T17:56:00.455056+0000 mon.a (mon.0) 2537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:01 smithi121 bash[36902]: audit 2022-04-22T17:56:00.455056+0000 mon.a (mon.0) 2537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:01.705Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:02.501 INFO:tasks.workunit.client.0.smithi115.stdout: 545: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:02.501 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jaEXW6w_ZGf6Vac off=3878500 len=363065 2022-04-22T17:56:02.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 2777 completed, throughput=4.99MB/sec 2022-04-22T17:56:02.505 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f8s8iQH8uQw1t7J off=3404070 len=658160 2022-04-22T17:56:02.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 2778 completed, throughput=5MB/sec 2022-04-22T17:56:02.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CKRrzBXqTLsg_F1 off=587489 len=587946 2022-04-22T17:56:02.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 2779 completed, throughput=5MB/sec 2022-04-22T17:56:02.516 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UfMFRq2J9JNic0I off=3731940 len=630677 2022-04-22T17:56:02.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 2780 completed, throughput=5MB/sec 2022-04-22T17:56:02.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H7G-Frslsc9rRzg off=448557 len=965327 2022-04-22T17:56:02.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 2781 completed, throughput=5MB/sec 2022-04-22T17:56:02.519 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W8JXxYZTDHj7PLq off=12317937 len=1826880 2022-04-22T17:56:02.522 INFO:tasks.workunit.client.0.smithi115.stdout:op 2782 completed, throughput=5MB/sec 2022-04-22T17:56:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:02 smithi121 bash[36902]: cluster 2022-04-22T17:56:00.540680+0000 mgr.y (mgr.34107) 1073 : cluster [DBG] pgmap v1679: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:02 smithi115 bash[45495]: cluster 2022-04-22T17:56:00.540680+0000 mgr.y (mgr.34107) 1073 : cluster [DBG] pgmap v1679: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:02 smithi115 bash[46934]: cluster 2022-04-22T17:56:00.540680+0000 mgr.y (mgr.34107) 1073 : cluster [DBG] pgmap v1679: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:03.522 INFO:tasks.workunit.client.0.smithi115.stdout: 546: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:03.522 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L3Tt7WfuU2AtMls off=424529 len=266824 2022-04-22T17:56:03.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 2783 completed, throughput=4.99MB/sec 2022-04-22T17:56:03.524 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b_Jj5Et_GvlNuIH off=15259664 len=1283088 2022-04-22T17:56:03.531 INFO:tasks.workunit.client.0.smithi115.stdout:op 2784 completed, throughput=5MB/sec 2022-04-22T17:56:03.531 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Id5xe6BBHhA_WO7 off=3842973 len=1674255 2022-04-22T17:56:03.536 INFO:tasks.workunit.client.0.smithi115.stdout:op 2785 completed, throughput=5MB/sec 2022-04-22T17:56:03.536 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GqKy3iHwSSVXHKJ off=3573452 len=295645 2022-04-22T17:56:03.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 2786 completed, throughput=5MB/sec 2022-04-22T17:56:03.541 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RQlcC80H0jKszkS off=4093047 len=1516918 2022-04-22T17:56:03.544 INFO:tasks.workunit.client.0.smithi115.stdout:op 2787 completed, throughput=5MB/sec 2022-04-22T17:56:03.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:03 smithi115 bash[45495]: audit 2022-04-22T17:56:02.493288+0000 mon.a (mon.0) 2538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:03.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:03 smithi115 bash[46934]: audit 2022-04-22T17:56:02.493288+0000 mon.a (mon.0) 2538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:03 smithi121 bash[36902]: audit 2022-04-22T17:56:02.493288+0000 mon.a (mon.0) 2538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:03.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:04.544 INFO:tasks.workunit.client.0.smithi115.stdout: 547: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:04.545 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EdOUUAQdaDV9CBk off=17271785 len=889173 2022-04-22T17:56:04.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 2788 completed, throughput=4.99MB/sec 2022-04-22T17:56:04.552 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Emf07EPJlZjsNpy off=1655217 len=1558300 2022-04-22T17:56:04.560 INFO:tasks.workunit.client.0.smithi115.stdout:op 2789 completed, throughput=5MB/sec 2022-04-22T17:56:04.560 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aIJ1WoBKbrGykDo off=3857540 len=69805 2022-04-22T17:56:04.561 INFO:tasks.workunit.client.0.smithi115.stdout:op 2790 completed, throughput=5MB/sec 2022-04-22T17:56:04.561 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3a2cyd7Gx0u1Eee off=15375652 len=1613532 2022-04-22T17:56:04.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 2791 completed, throughput=5MB/sec 2022-04-22T17:56:04.568 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bGewWV4HphuR77z off=411428 len=1576294 2022-04-22T17:56:04.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 2792 completed, throughput=5MB/sec 2022-04-22T17:56:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:04 smithi121 bash[36902]: cluster 2022-04-22T17:56:02.541774+0000 mgr.y (mgr.34107) 1074 : cluster [DBG] pgmap v1680: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:04 smithi115 bash[45495]: cluster 2022-04-22T17:56:02.541774+0000 mgr.y (mgr.34107) 1074 : cluster [DBG] pgmap v1680: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:04 smithi115 bash[46934]: cluster 2022-04-22T17:56:02.541774+0000 mgr.y (mgr.34107) 1074 : cluster [DBG] pgmap v1680: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:05.570 INFO:tasks.workunit.client.0.smithi115.stdout: 548: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:05.593 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xhj4ecMX_6e4tMs off=1761552 len=1303757 2022-04-22T17:56:05.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 2793 completed, throughput=5MB/sec 2022-04-22T17:56:05.594 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-udqFfI962Y5Lcbr off=2528887 len=876614 2022-04-22T17:56:05.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 2794 completed, throughput=5MB/sec 2022-04-22T17:56:05.594 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GMIjMdnm39odEJI off=246335 len=1053535 2022-04-22T17:56:05.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 2795 completed, throughput=5MB/sec 2022-04-22T17:56:05.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FbBhfP5NIuWoBo3 off=160696 len=1508229 2022-04-22T17:56:05.600 INFO:tasks.workunit.client.0.smithi115.stdout:op 2796 completed, throughput=5MB/sec 2022-04-22T17:56:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:05 smithi121 bash[36902]: audit 2022-04-22T17:56:05.109730+0000 mon.a (mon.0) 2539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:05 smithi115 bash[45495]: audit 2022-04-22T17:56:05.109730+0000 mon.a (mon.0) 2539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:05 smithi115 bash[46934]: audit 2022-04-22T17:56:05.109730+0000 mon.a (mon.0) 2539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:06.600 INFO:tasks.workunit.client.0.smithi115.stdout: 549: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:06.600 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lCoGsayvidh_Lna off=2648476 len=523763 2022-04-22T17:56:06.606 INFO:tasks.workunit.client.0.smithi115.stdout:op 2797 completed, throughput=4.99MB/sec 2022-04-22T17:56:06.606 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-weHzN504xjZnSG7 off=4345901 len=197744 2022-04-22T17:56:06.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 2798 completed, throughput=4.99MB/sec 2022-04-22T17:56:06.607 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hR6D8PLyRw2u5lv off=3680802 len=364753 2022-04-22T17:56:06.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 2799 completed, throughput=4.99MB/sec 2022-04-22T17:56:06.612 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JyoTqqtbNgKX7CB off=18515317 len=1158703 2022-04-22T17:56:06.619 INFO:tasks.workunit.client.0.smithi115.stdout:op 2800 completed, throughput=5MB/sec 2022-04-22T17:56:06.619 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5i0MeedPgBHLaPB off=2410206 len=1589841 2022-04-22T17:56:06.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 2801 completed, throughput=5MB/sec 2022-04-22T17:56:06.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HP6ty9U3q1i0gqH off=314658 len=790139 2022-04-22T17:56:06.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 2802 completed, throughput=5MB/sec 2022-04-22T17:56:06.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u0QDgdlV3kysWCQ off=767467 len=306473 2022-04-22T17:56:06.633 INFO:tasks.workunit.client.0.smithi115.stdout:op 2803 completed, throughput=5MB/sec 2022-04-22T17:56:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:06 smithi121 bash[36902]: cluster 2022-04-22T17:56:04.542851+0000 mgr.y (mgr.34107) 1075 : cluster [DBG] pgmap v1681: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:06 smithi121 bash[36902]: audit 2022-04-22T17:56:05.458579+0000 mon.a (mon.0) 2540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:06 smithi115 bash[45495]: cluster 2022-04-22T17:56:04.542851+0000 mgr.y (mgr.34107) 1075 : cluster [DBG] pgmap v1681: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:06 smithi115 bash[45495]: audit 2022-04-22T17:56:05.458579+0000 mon.a (mon.0) 2540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:06 smithi115 bash[46934]: cluster 2022-04-22T17:56:04.542851+0000 mgr.y (mgr.34107) 1075 : cluster [DBG] pgmap v1681: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:56:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:06 smithi115 bash[46934]: audit 2022-04-22T17:56:05.458579+0000 mon.a (mon.0) 2540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:07.634 INFO:tasks.workunit.client.0.smithi115.stdout: 550: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:07.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JokCgcjduh3vOh- off=3246290 len=1250267 2022-04-22T17:56:07.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 2804 completed, throughput=4.99MB/sec 2022-04-22T17:56:07.782 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2jn3bdntZtc-AA9 off=13209467 len=1819641 2022-04-22T17:56:07.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 2805 completed, throughput=5MB/sec 2022-04-22T17:56:07.782 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9oQPE2alrulK4sl off=524362 len=63086 2022-04-22T17:56:07.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 2806 completed, throughput=5MB/sec 2022-04-22T17:56:07.783 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cpayPV3F_s72L2g off=334506 len=72900 2022-04-22T17:56:07.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 2807 completed, throughput=5MB/sec 2022-04-22T17:56:07.783 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7qnFLT0I2WX5kel off=11980480 len=161221 2022-04-22T17:56:07.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 2808 completed, throughput=5MB/sec 2022-04-22T17:56:07.784 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6wt47AEUciZWwLB off=2317690 len=776323 2022-04-22T17:56:07.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 2809 completed, throughput=5MB/sec 2022-04-22T17:56:07.784 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tVp0Wngwe_nnoiq off=15437637 len=1429964 2022-04-22T17:56:07.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 2810 completed, throughput=5MB/sec 2022-04-22T17:56:08.670 INFO:tasks.workunit.client.0.smithi115.stdout: 551: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:08.670 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EXzOCvj0_yYLq3T off=17251083 len=513973 2022-04-22T17:56:08.671 INFO:tasks.workunit.client.0.smithi115.stdout:op 2811 completed, throughput=4.99MB/sec 2022-04-22T17:56:08.672 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CfTSoPcPga3RXrX off=6887003 len=512087 2022-04-22T17:56:08.674 INFO:tasks.workunit.client.0.smithi115.stdout:op 2812 completed, throughput=4.99MB/sec 2022-04-22T17:56:08.674 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6uqo9OXW1VA0abW off=1396261 len=1819306 2022-04-22T17:56:08.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 2813 completed, throughput=5MB/sec 2022-04-22T17:56:08.679 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lKyrFaHu85tZx0X off=4236661 len=1730604 2022-04-22T17:56:08.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 2814 completed, throughput=5MB/sec 2022-04-22T17:56:08.686 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dKQhd0ezrWIwnKM off=3406046 len=551210 2022-04-22T17:56:08.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 2815 completed, throughput=5MB/sec 2022-04-22T17:56:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:08 smithi121 bash[36902]: cluster 2022-04-22T17:56:06.544022+0000 mgr.y (mgr.34107) 1076 : cluster [DBG] pgmap v1682: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:56:08.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:08 smithi121 bash[36902]: audit 2022-04-22T17:56:07.496555+0000 mon.a (mon.0) 2541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:08 smithi115 bash[45495]: cluster 2022-04-22T17:56:06.544022+0000 mgr.y (mgr.34107) 1076 : cluster [DBG] pgmap v1682: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:56:08.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:08 smithi115 bash[45495]: audit 2022-04-22T17:56:07.496555+0000 mon.a (mon.0) 2541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:08.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:08 smithi115 bash[46934]: cluster 2022-04-22T17:56:06.544022+0000 mgr.y (mgr.34107) 1076 : cluster [DBG] pgmap v1682: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:56:08.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:08 smithi115 bash[46934]: audit 2022-04-22T17:56:07.496555+0000 mon.a (mon.0) 2541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:09.687 INFO:tasks.workunit.client.0.smithi115.stdout: 552: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:09.687 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2fWWizAjM2L3s21 off=4446979 len=1656954 2022-04-22T17:56:09.696 INFO:tasks.workunit.client.0.smithi115.stdout:op 2816 completed, throughput=4.99MB/sec 2022-04-22T17:56:09.697 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ath60MKsNPCDBOd off=15702988 len=830924 2022-04-22T17:56:09.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 2817 completed, throughput=5MB/sec 2022-04-22T17:56:09.699 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XG4Qeybswap8WAl off=989699 len=544482 2022-04-22T17:56:09.704 INFO:tasks.workunit.client.0.smithi115.stdout:op 2818 completed, throughput=5MB/sec 2022-04-22T17:56:09.704 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UNKcUc2a3P20NiW off=6204852 len=1673351 2022-04-22T17:56:09.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 2819 completed, throughput=5MB/sec 2022-04-22T17:56:09.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iGH8zX_GDDiOD67 off=5099613 len=1533851 2022-04-22T17:56:09.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 2820 completed, throughput=5MB/sec 2022-04-22T17:56:10.711 INFO:tasks.workunit.client.0.smithi115.stdout: 553: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:10.942 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MIqVjYQwOJ1eTzd off=5997430 len=1437516 2022-04-22T17:56:10.943 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:10 smithi121 bash[36902]: cluster 2022-04-22T17:56:08.545158+0000 mgr.y (mgr.34107) 1077 : cluster [DBG] pgmap v1683: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:56:10.943 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:10 smithi121 bash[36902]: audit 2022-04-22T17:56:10.112351+0000 mon.a (mon.0) 2542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:10.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 2821 completed, throughput=5MB/sec 2022-04-22T17:56:10.948 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jFH53TGkJ9t0Aoc off=2169245 len=67614 2022-04-22T17:56:10.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 2822 completed, throughput=5MB/sec 2022-04-22T17:56:10.948 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8Be-H-6iCbJPZAr off=20342209 len=1992971 2022-04-22T17:56:10.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 2823 completed, throughput=5MB/sec 2022-04-22T17:56:10.949 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZuTpftm6syanRYH off=1332629 len=1745942 2022-04-22T17:56:10.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 2824 completed, throughput=5MB/sec 2022-04-22T17:56:10.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:10 smithi115 bash[45495]: cluster 2022-04-22T17:56:08.545158+0000 mgr.y (mgr.34107) 1077 : cluster [DBG] pgmap v1683: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:56:10.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:10 smithi115 bash[45495]: audit 2022-04-22T17:56:10.112351+0000 mon.a (mon.0) 2542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:10.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:10 smithi115 bash[46934]: cluster 2022-04-22T17:56:08.545158+0000 mgr.y (mgr.34107) 1077 : cluster [DBG] pgmap v1683: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:56:10.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:10 smithi115 bash[46934]: audit 2022-04-22T17:56:10.112351+0000 mon.a (mon.0) 2542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:11.713 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:11 smithi115 bash[45495]: audit 2022-04-22T17:56:10.462072+0000 mon.a (mon.0) 2543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:11.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:11 smithi115 bash[46934]: audit 2022-04-22T17:56:10.462072+0000 mon.a (mon.0) 2543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:11.734 INFO:tasks.workunit.client.0.smithi115.stdout: 554: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:11.734 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1tg2Wux9o9Rozo5 off=12851770 len=225325 2022-04-22T17:56:11.735 INFO:tasks.workunit.client.0.smithi115.stdout:op 2825 completed, throughput=4.99MB/sec 2022-04-22T17:56:11.735 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MZcV6fFKcWYHrfo off=6299622 len=445552 2022-04-22T17:56:11.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 2826 completed, throughput=4.99MB/sec 2022-04-22T17:56:11.737 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F-UPHUevdaAPNA2 off=6177414 len=720466 2022-04-22T17:56:11.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 2827 completed, throughput=5MB/sec 2022-04-22T17:56:11.743 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZYiyHgrdOzDfVxn off=1192454 len=54267 2022-04-22T17:56:11.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 2828 completed, throughput=5MB/sec 2022-04-22T17:56:11.744 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WlBP7oiPypaJO2e off=1136639 len=2058032 2022-04-22T17:56:11.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 2829 completed, throughput=5MB/sec 2022-04-22T17:56:11.750 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ix_EdvPaVXUtuU7 off=4957144 len=766395 2022-04-22T17:56:11.751 INFO:tasks.workunit.client.0.smithi115.stdout:op 2830 completed, throughput=5MB/sec 2022-04-22T17:56:11.751 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5Hm4Uy8o6pX1ayk off=2999227 len=1377850 2022-04-22T17:56:11.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 2831 completed, throughput=5MB/sec 2022-04-22T17:56:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:11 smithi121 bash[36902]: audit 2022-04-22T17:56:10.462072+0000 mon.a (mon.0) 2543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:12.007 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:56:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:11.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:12.353 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:56:12.353 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (19m) 8m ago 33m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:56:12.353 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (32m) 9m ago 32m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:56:12.353 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (21m) 8m ago 34m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:56:12.353 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (20m) 9m ago 34m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (28m) 9m ago 38m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (29m) 8m ago 40m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (28m) 8m ago 40m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (27m) 9m ago 38m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (27m) 8m ago 38m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:56:12.354 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (20m) 8m ago 33m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (20m) 9m ago 33m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (25m) 8m ago 37m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (25m) 8m ago 37m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (24m) 8m ago 36m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (24m) 8m ago 36m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (23m) 9m ago 36m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:56:12.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (23m) 9m ago 35m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:56:12.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (22m) 9m ago 35m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:56:12.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (21m) 9m ago 34m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:56:12.356 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (19m) 9m ago 33m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:56:12.759 INFO:tasks.workunit.client.0.smithi115.stdout: 555: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:12.759 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8ZFG5HF0L2b6Mz2 off=4330466 len=885365 2022-04-22T17:56:12.760 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:56:12.760 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:56:12.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:56:12.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:56:12.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:56:12.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:12.762 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:56:12.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:56:12.762 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:56:12.763 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:56:12.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 2832 completed, throughput=4.99MB/sec 2022-04-22T17:56:12.763 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dx9fnukj4xpSlG7 off=3599738 len=373580 2022-04-22T17:56:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:12 smithi121 bash[36902]: cluster 2022-04-22T17:56:10.546172+0000 mgr.y (mgr.34107) 1078 : cluster [DBG] pgmap v1684: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:12.766 INFO:tasks.workunit.client.0.smithi115.stdout:op 2833 completed, throughput=5MB/sec 2022-04-22T17:56:12.766 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0jjK4yfZPFirLoZ off=2855360 len=675851 2022-04-22T17:56:12.771 INFO:tasks.workunit.client.0.smithi115.stdout:op 2834 completed, throughput=5MB/sec 2022-04-22T17:56:12.771 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5Yfl3uBwi2dvq89 off=11874038 len=315918 2022-04-22T17:56:12.772 INFO:tasks.workunit.client.0.smithi115.stdout:op 2835 completed, throughput=5MB/sec 2022-04-22T17:56:12.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sXs-I9udNbbmVif off=17880181 len=1486357 2022-04-22T17:56:12.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 2836 completed, throughput=5MB/sec 2022-04-22T17:56:12.832 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JpHkLhE5AX2OBFG off=1053886 len=12513 2022-04-22T17:56:12.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 2837 completed, throughput=5MB/sec 2022-04-22T17:56:12.834 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PVBtbfegix4VZ_o off=9376527 len=552715 2022-04-22T17:56:12.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 2838 completed, throughput=5MB/sec 2022-04-22T17:56:12.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:12 smithi115 bash[45495]: cluster 2022-04-22T17:56:10.546172+0000 mgr.y (mgr.34107) 1078 : cluster [DBG] pgmap v1684: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:12.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:12 smithi115 bash[46934]: cluster 2022-04-22T17:56:10.546172+0000 mgr.y (mgr.34107) 1078 : cluster [DBG] pgmap v1684: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:56:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:13 smithi121 bash[36902]: audit 2022-04-22T17:56:11.984980+0000 mgr.y (mgr.34107) 1079 : audit [DBG] from='client.45475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:13 smithi121 bash[36902]: audit 2022-04-22T17:56:12.327044+0000 mgr.y (mgr.34107) 1080 : audit [DBG] from='client.45481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:13 smithi121 bash[36902]: audit 2022-04-22T17:56:12.499449+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:13 smithi121 bash[36902]: audit 2022-04-22T17:56:12.753149+0000 mon.c (mon.1) 74 : audit [DBG] from='client.? 172.21.15.115:0/1244530539' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:13.782 INFO:tasks.workunit.client.0.smithi115.stdout: 556: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:13.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0Z06TjByUp8Y3tt off=5649091 len=1087049 2022-04-22T17:56:13.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 2839 completed, throughput=4.99MB/sec 2022-04-22T17:56:13.785 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y_YEbmPKyZRrozE off=14585313 len=1151696 2022-04-22T17:56:13.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 2840 completed, throughput=5MB/sec 2022-04-22T17:56:13.793 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-weyi1_BjaG5bG2A off=11418043 len=1164769 2022-04-22T17:56:13.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:13.588Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[45495]: audit 2022-04-22T17:56:11.984980+0000 mgr.y (mgr.34107) 1079 : audit [DBG] from='client.45475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[45495]: audit 2022-04-22T17:56:12.327044+0000 mgr.y (mgr.34107) 1080 : audit [DBG] from='client.45481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[45495]: audit 2022-04-22T17:56:12.499449+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[45495]: audit 2022-04-22T17:56:12.753149+0000 mon.c (mon.1) 74 : audit [DBG] from='client.? 172.21.15.115:0/1244530539' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[46934]: audit 2022-04-22T17:56:11.984980+0000 mgr.y (mgr.34107) 1079 : audit [DBG] from='client.45475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[46934]: audit 2022-04-22T17:56:12.327044+0000 mgr.y (mgr.34107) 1080 : audit [DBG] from='client.45481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[46934]: audit 2022-04-22T17:56:12.499449+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:13 smithi115 bash[46934]: audit 2022-04-22T17:56:12.753149+0000 mon.c (mon.1) 74 : audit [DBG] from='client.? 172.21.15.115:0/1244530539' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:13.797 INFO:tasks.workunit.client.0.smithi115.stdout:op 2841 completed, throughput=5MB/sec 2022-04-22T17:56:13.797 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sVCPM_IQU6tbJvt off=1429399 len=926572 2022-04-22T17:56:13.799 INFO:tasks.workunit.client.0.smithi115.stdout:op 2842 completed, throughput=5MB/sec 2022-04-22T17:56:13.799 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uoujpEnHkxw6wmU off=333634 len=66409 2022-04-22T17:56:13.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 2843 completed, throughput=5MB/sec 2022-04-22T17:56:13.800 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1a11ITKX_wFC0id off=7777445 len=213832 2022-04-22T17:56:13.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 2844 completed, throughput=5MB/sec 2022-04-22T17:56:13.801 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4PWtnWYZQJYe6h5 off=1341048 len=1177094 2022-04-22T17:56:13.809 INFO:tasks.workunit.client.0.smithi115.stdout:op 2845 completed, throughput=5MB/sec 2022-04-22T17:56:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:14 smithi121 bash[36902]: cluster 2022-04-22T17:56:12.547101+0000 mgr.y (mgr.34107) 1081 : cluster [DBG] pgmap v1685: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:14.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:14 smithi121 bash[36902]: audit 2022-04-22T17:56:13.541136+0000 mon.a (mon.0) 2545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:56:14.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:14 smithi121 bash[36902]: audit 2022-04-22T17:56:13.542830+0000 mon.a (mon.0) 2546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:56:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[45495]: cluster 2022-04-22T17:56:12.547101+0000 mgr.y (mgr.34107) 1081 : cluster [DBG] pgmap v1685: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[45495]: audit 2022-04-22T17:56:13.541136+0000 mon.a (mon.0) 2545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:56:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[45495]: audit 2022-04-22T17:56:13.542830+0000 mon.a (mon.0) 2546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:56:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[46934]: cluster 2022-04-22T17:56:12.547101+0000 mgr.y (mgr.34107) 1081 : cluster [DBG] pgmap v1685: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[46934]: audit 2022-04-22T17:56:13.541136+0000 mon.a (mon.0) 2545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:56:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:14 smithi115 bash[46934]: audit 2022-04-22T17:56:13.542830+0000 mon.a (mon.0) 2546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:56:14.809 INFO:tasks.workunit.client.0.smithi115.stdout: 557: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:14.809 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XxnY9wIkmKSu8k5 off=1696572 len=576513 2022-04-22T17:56:14.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 2846 completed, throughput=4.99MB/sec 2022-04-22T17:56:14.811 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YkDoKwtByGzCLUJ off=239991 len=1629759 2022-04-22T17:56:14.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 2847 completed, throughput=5MB/sec 2022-04-22T17:56:14.815 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-p6auvB9BAgJhjdA off=4929004 len=281144 2022-04-22T17:56:14.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 2848 completed, throughput=5MB/sec 2022-04-22T17:56:14.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tgro8mH_3GBGKga off=8242439 len=1222902 2022-04-22T17:56:14.827 INFO:tasks.workunit.client.0.smithi115.stdout:op 2849 completed, throughput=5MB/sec 2022-04-22T17:56:14.827 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tc--6XmMytKz0lC off=8000491 len=807663 2022-04-22T17:56:14.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 2850 completed, throughput=5MB/sec 2022-04-22T17:56:14.829 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4g7oAJisrKj90_7 off=661257 len=224755 2022-04-22T17:56:14.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 2851 completed, throughput=5MB/sec 2022-04-22T17:56:14.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kWaqUsU7qlSlj4y off=1055873 len=1362997 2022-04-22T17:56:14.838 INFO:tasks.workunit.client.0.smithi115.stdout:op 2852 completed, throughput=5MB/sec 2022-04-22T17:56:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:15 smithi121 bash[36902]: audit 2022-04-22T17:56:15.114743+0000 mon.a (mon.0) 2547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:15.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:15 smithi115 bash[45495]: audit 2022-04-22T17:56:15.114743+0000 mon.a (mon.0) 2547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:15.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:15 smithi115 bash[46934]: audit 2022-04-22T17:56:15.114743+0000 mon.a (mon.0) 2547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:15.994 INFO:tasks.workunit.client.0.smithi115.stdout: 558: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qEdK8OucdEvBApj off=9788572 len=61236 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 2853 completed, throughput=4.99MB/sec 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LWrN-_JHEEOefgE off=10184201 len=1964739 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 2854 completed, throughput=5MB/sec 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EzmYWxSaXxDfB63 off=20509999 len=441731 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 2855 completed, throughput=5MB/sec 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QTc4qXCBZPin0vK off=17883764 len=947824 2022-04-22T17:56:15.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 2856 completed, throughput=5MB/sec 2022-04-22T17:56:15.996 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j4C0R90iekfcKyh off=8918405 len=942211 2022-04-22T17:56:15.996 INFO:tasks.workunit.client.0.smithi115.stdout:op 2857 completed, throughput=5MB/sec 2022-04-22T17:56:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:16 smithi121 bash[36902]: cluster 2022-04-22T17:56:14.548190+0000 mgr.y (mgr.34107) 1082 : cluster [DBG] pgmap v1686: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:16 smithi121 bash[36902]: audit 2022-04-22T17:56:15.465762+0000 mon.a (mon.0) 2548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:16 smithi115 bash[45495]: cluster 2022-04-22T17:56:14.548190+0000 mgr.y (mgr.34107) 1082 : cluster [DBG] pgmap v1686: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:16 smithi115 bash[45495]: audit 2022-04-22T17:56:15.465762+0000 mon.a (mon.0) 2548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:16 smithi115 bash[46934]: cluster 2022-04-22T17:56:14.548190+0000 mgr.y (mgr.34107) 1082 : cluster [DBG] pgmap v1686: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:56:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:16 smithi115 bash[46934]: audit 2022-04-22T17:56:15.465762+0000 mon.a (mon.0) 2548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:16.862 INFO:tasks.workunit.client.0.smithi115.stdout: 559: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:16.863 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RpQGsJRVzYjgGkK off=18079317 len=273135 2022-04-22T17:56:16.867 INFO:tasks.workunit.client.0.smithi115.stdout:op 2858 completed, throughput=4.99MB/sec 2022-04-22T17:56:16.867 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-voOgd3P_3sPneuT off=6982720 len=601031 2022-04-22T17:56:16.873 INFO:tasks.workunit.client.0.smithi115.stdout:op 2859 completed, throughput=4.99MB/sec 2022-04-22T17:56:16.873 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PPqsIkjAja7lPBx off=4084358 len=741730 2022-04-22T17:56:16.878 INFO:tasks.workunit.client.0.smithi115.stdout:op 2860 completed, throughput=4.99MB/sec 2022-04-22T17:56:16.878 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rAiIgIkxdIPtACA off=10682122 len=1004331 2022-04-22T17:56:16.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 2861 completed, throughput=5MB/sec 2022-04-22T17:56:16.883 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tTV6nS5--H26BgD off=2415985 len=640777 2022-04-22T17:56:16.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 2862 completed, throughput=5MB/sec 2022-04-22T17:56:16.883 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yViJfyIUKn69Tkv off=8861982 len=426756 2022-04-22T17:56:16.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 2863 completed, throughput=5MB/sec 2022-04-22T17:56:16.887 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pj7ATz4iQBL8B9Q off=723993 len=911323 2022-04-22T17:56:16.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 2864 completed, throughput=5MB/sec 2022-04-22T17:56:16.893 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2UiAfYzgnY0kJFX off=3380889 len=841697 2022-04-22T17:56:16.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 2865 completed, throughput=5MB/sec 2022-04-22T17:56:17.899 INFO:tasks.workunit.client.0.smithi115.stdout: 560: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:17.899 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6Qd5C_IkBEZPRmD off=4690693 len=1761811 2022-04-22T17:56:17.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 2866 completed, throughput=4.99MB/sec 2022-04-22T17:56:17.903 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IZKeKRrS3g6WGW8 off=0 len=1147445 2022-04-22T17:56:17.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 2867 completed, throughput=5MB/sec 2022-04-22T17:56:17.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0AiDoas_OwsdE7X off=1790037 len=340853 2022-04-22T17:56:17.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 2868 completed, throughput=5MB/sec 2022-04-22T17:56:17.909 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-66s5pNGKlNKYeqR off=13132978 len=837394 2022-04-22T17:56:17.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 2869 completed, throughput=5MB/sec 2022-04-22T17:56:17.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8odDUnlDGSo9MJB off=574627 len=1493137 2022-04-22T17:56:17.920 INFO:tasks.workunit.client.0.smithi115.stdout:op 2870 completed, throughput=5MB/sec 2022-04-22T17:56:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:18 smithi121 bash[36902]: cluster 2022-04-22T17:56:16.549253+0000 mgr.y (mgr.34107) 1083 : cluster [DBG] pgmap v1687: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:56:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:18 smithi121 bash[36902]: audit 2022-04-22T17:56:17.502223+0000 mon.a (mon.0) 2549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:18 smithi115 bash[45495]: cluster 2022-04-22T17:56:16.549253+0000 mgr.y (mgr.34107) 1083 : cluster [DBG] pgmap v1687: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:56:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:18 smithi115 bash[45495]: audit 2022-04-22T17:56:17.502223+0000 mon.a (mon.0) 2549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:18 smithi115 bash[46934]: cluster 2022-04-22T17:56:16.549253+0000 mgr.y (mgr.34107) 1083 : cluster [DBG] pgmap v1687: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 6 op/s 2022-04-22T17:56:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:18 smithi115 bash[46934]: audit 2022-04-22T17:56:17.502223+0000 mon.a (mon.0) 2549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:18.920 INFO:tasks.workunit.client.0.smithi115.stdout: 561: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:18.921 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wSJOyqdZo9YwB6G off=1671753 len=1712994 2022-04-22T17:56:18.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 2871 completed, throughput=4.99MB/sec 2022-04-22T17:56:18.926 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bYXEmY-5ztjlOsZ off=20448435 len=1651445 2022-04-22T17:56:18.931 INFO:tasks.workunit.client.0.smithi115.stdout:op 2872 completed, throughput=5MB/sec 2022-04-22T17:56:18.931 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W-fPME67Za2lEVt off=1472759 len=1673847 2022-04-22T17:56:18.935 INFO:tasks.workunit.client.0.smithi115.stdout:op 2873 completed, throughput=5MB/sec 2022-04-22T17:56:19.935 INFO:tasks.workunit.client.0.smithi115.stdout: 562: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:19.935 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gnX8Y_XRTSKbaCy off=1606316 len=422287 2022-04-22T17:56:19.937 INFO:tasks.workunit.client.0.smithi115.stdout:op 2874 completed, throughput=4.99MB/sec 2022-04-22T17:56:19.937 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XTWH5VtWM6QhjzP off=5512426 len=882754 2022-04-22T17:56:19.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 2875 completed, throughput=4.99MB/sec 2022-04-22T17:56:19.941 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cejYx3qVupbU2mQ off=6603084 len=1508215 2022-04-22T17:56:19.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 2876 completed, throughput=5MB/sec 2022-04-22T17:56:19.944 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u9lvP2aGgpbwIc1 off=6692652 len=1349798 2022-04-22T17:56:19.946 INFO:tasks.workunit.client.0.smithi115.stdout:op 2877 completed, throughput=5MB/sec 2022-04-22T17:56:19.946 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p5RlD3M1EVJDL0t off=11040301 len=1159455 2022-04-22T17:56:19.950 INFO:tasks.workunit.client.0.smithi115.stdout:op 2878 completed, throughput=5MB/sec 2022-04-22T17:56:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:20 smithi121 bash[36902]: cluster 2022-04-22T17:56:18.550485+0000 mgr.y (mgr.34107) 1084 : cluster [DBG] pgmap v1688: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2022-04-22T17:56:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:20 smithi121 bash[36902]: audit 2022-04-22T17:56:20.117894+0000 mon.a (mon.0) 2550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:20 smithi115 bash[45495]: cluster 2022-04-22T17:56:18.550485+0000 mgr.y (mgr.34107) 1084 : cluster [DBG] pgmap v1688: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2022-04-22T17:56:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:20 smithi115 bash[45495]: audit 2022-04-22T17:56:20.117894+0000 mon.a (mon.0) 2550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:20 smithi115 bash[46934]: cluster 2022-04-22T17:56:18.550485+0000 mgr.y (mgr.34107) 1084 : cluster [DBG] pgmap v1688: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 6 op/s 2022-04-22T17:56:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:20 smithi115 bash[46934]: audit 2022-04-22T17:56:20.117894+0000 mon.a (mon.0) 2550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:20.951 INFO:tasks.workunit.client.0.smithi115.stdout: 563: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:20.951 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hbNAqFTbVN-dA7l off=1293965 len=61540 2022-04-22T17:56:20.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 2879 completed, throughput=4.99MB/sec 2022-04-22T17:56:20.952 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LShK68GEy7-F9SV off=1327079 len=1947586 2022-04-22T17:56:20.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 2880 completed, throughput=4.99MB/sec 2022-04-22T17:56:20.963 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Rx7m9jr8O674Ocd off=3941746 len=2076891 2022-04-22T17:56:20.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 2881 completed, throughput=5MB/sec 2022-04-22T17:56:20.971 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kTzIX-KnWmSGghr off=8339314 len=907582 2022-04-22T17:56:20.977 INFO:tasks.workunit.client.0.smithi115.stdout:op 2882 completed, throughput=5MB/sec 2022-04-22T17:56:20.978 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9aZAVhiDTFG2Hm- off=18672977 len=1709844 2022-04-22T17:56:20.985 INFO:tasks.workunit.client.0.smithi115.stdout:op 2883 completed, throughput=5MB/sec 2022-04-22T17:56:21.714 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:21 smithi115 bash[45495]: audit 2022-04-22T17:56:20.469476+0000 mon.a (mon.0) 2551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:21.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:21 smithi115 bash[46934]: audit 2022-04-22T17:56:20.469476+0000 mon.a (mon.0) 2551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:21 smithi121 bash[36902]: audit 2022-04-22T17:56:20.469476+0000 mon.a (mon.0) 2551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:21.986 INFO:tasks.workunit.client.0.smithi115.stdout: 564: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:21.986 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xP58TiO6_4FZJr9 off=4253954 len=676474 2022-04-22T17:56:21.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 2884 completed, throughput=4.99MB/sec 2022-04-22T17:56:21.988 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8__SBZoqQGd6EkC off=1595380 len=1758054 2022-04-22T17:56:21.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 2885 completed, throughput=5MB/sec 2022-04-22T17:56:21.993 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uNJVsf7EZjAdAOT off=12197651 len=78344 2022-04-22T17:56:21.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 2886 completed, throughput=5MB/sec 2022-04-22T17:56:21.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o9GvIUKP4eNODlt off=4239656 len=11168 2022-04-22T17:56:21.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 2887 completed, throughput=5MB/sec 2022-04-22T17:56:21.998 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0xoNXa10YbaFcP0 off=4571354 len=2036484 2022-04-22T17:56:22.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 2888 completed, throughput=5MB/sec 2022-04-22T17:56:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:21.707Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:22 smithi121 bash[36902]: cluster 2022-04-22T17:56:20.551662+0000 mgr.y (mgr.34107) 1085 : cluster [DBG] pgmap v1689: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:56:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:22 smithi115 bash[45495]: cluster 2022-04-22T17:56:20.551662+0000 mgr.y (mgr.34107) 1085 : cluster [DBG] pgmap v1689: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:56:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:22 smithi115 bash[46934]: cluster 2022-04-22T17:56:20.551662+0000 mgr.y (mgr.34107) 1085 : cluster [DBG] pgmap v1689: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:56:23.009 INFO:tasks.workunit.client.0.smithi115.stdout: 565: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:23.010 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MO1ZduKTNWfdkbR off=138915 len=967190 2022-04-22T17:56:23.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 2889 completed, throughput=4.99MB/sec 2022-04-22T17:56:23.018 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1fEPV7JmqO8ml1c off=2660592 len=1000111 2022-04-22T17:56:23.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 2890 completed, throughput=5MB/sec 2022-04-22T17:56:23.020 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Od9x9Ez-0qsFgJR off=9896659 len=1440209 2022-04-22T17:56:23.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 2891 completed, throughput=5MB/sec 2022-04-22T17:56:23.022 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L5cv2EPW77efcoD off=2413008 len=1776396 2022-04-22T17:56:23.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 2892 completed, throughput=5MB/sec 2022-04-22T17:56:23.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:23 smithi121 bash[36902]: audit 2022-04-22T17:56:22.505185+0000 mon.a (mon.0) 2552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:23.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:23 smithi121 bash[36902]: cluster 2022-04-22T17:56:22.552799+0000 mgr.y (mgr.34107) 1086 : cluster [DBG] pgmap v1690: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:23.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:23.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:23 smithi115 bash[45495]: audit 2022-04-22T17:56:22.505185+0000 mon.a (mon.0) 2552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:23 smithi115 bash[45495]: cluster 2022-04-22T17:56:22.552799+0000 mgr.y (mgr.34107) 1086 : cluster [DBG] pgmap v1690: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:23 smithi115 bash[46934]: audit 2022-04-22T17:56:22.505185+0000 mon.a (mon.0) 2552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:23 smithi115 bash[46934]: cluster 2022-04-22T17:56:22.552799+0000 mgr.y (mgr.34107) 1086 : cluster [DBG] pgmap v1690: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:24.029 INFO:tasks.workunit.client.0.smithi115.stdout: 566: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:24.029 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DcF3ZdznmCg8p7W off=2380879 len=701747 2022-04-22T17:56:24.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 2893 completed, throughput=4.99MB/sec 2022-04-22T17:56:24.034 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NeF3EkmKfHFMZBo off=5997004 len=290323 2022-04-22T17:56:24.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 2894 completed, throughput=4.99MB/sec 2022-04-22T17:56:24.036 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l_XiWkGVFo8gVNB off=168234 len=1087607 2022-04-22T17:56:24.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 2895 completed, throughput=5MB/sec 2022-04-22T17:56:24.043 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6g1w1GRYB6FhBc7 off=6072870 len=1519492 2022-04-22T17:56:24.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 2896 completed, throughput=5MB/sec 2022-04-22T17:56:24.050 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jz9dvki314R2nHP off=203801 len=592479 2022-04-22T17:56:24.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 2897 completed, throughput=5MB/sec 2022-04-22T17:56:24.051 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-t-pnwskyEGUvxf5 off=9448248 len=1058992 2022-04-22T17:56:24.052 INFO:tasks.workunit.client.0.smithi115.stdout:op 2898 completed, throughput=5MB/sec 2022-04-22T17:56:25.053 INFO:tasks.workunit.client.0.smithi115.stdout: 567: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:25.053 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aVIGfGfMw1YWIxr off=490518 len=6938 2022-04-22T17:56:25.057 INFO:tasks.workunit.client.0.smithi115.stdout:op 2899 completed, throughput=4.99MB/sec 2022-04-22T17:56:25.057 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2AhPDaEsJZlfUKY off=8367772 len=42390 2022-04-22T17:56:25.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 2900 completed, throughput=4.99MB/sec 2022-04-22T17:56:25.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c1fZsq2jE9X_xag off=1599489 len=1515857 2022-04-22T17:56:25.067 INFO:tasks.workunit.client.0.smithi115.stdout:op 2901 completed, throughput=4.99MB/sec 2022-04-22T17:56:25.067 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d8bkto4NlcCZ9IH off=544287 len=1976657 2022-04-22T17:56:25.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 2902 completed, throughput=5MB/sec 2022-04-22T17:56:25.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H8iTvJOFUmHwdLD off=12378084 len=870097 2022-04-22T17:56:25.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 2903 completed, throughput=5MB/sec 2022-04-22T17:56:25.079 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rGFUo_HvR7HLG2i off=5064245 len=670868 2022-04-22T17:56:25.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 2904 completed, throughput=5MB/sec 2022-04-22T17:56:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:25 smithi121 bash[36902]: cluster 2022-04-22T17:56:24.553905+0000 mgr.y (mgr.34107) 1087 : cluster [DBG] pgmap v1691: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:56:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:25 smithi121 bash[36902]: audit 2022-04-22T17:56:25.120848+0000 mon.a (mon.0) 2553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:25 smithi121 bash[36902]: audit 2022-04-22T17:56:25.473045+0000 mon.a (mon.0) 2554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[46934]: cluster 2022-04-22T17:56:24.553905+0000 mgr.y (mgr.34107) 1087 : cluster [DBG] pgmap v1691: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:56:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[46934]: audit 2022-04-22T17:56:25.120848+0000 mon.a (mon.0) 2553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[46934]: audit 2022-04-22T17:56:25.473045+0000 mon.a (mon.0) 2554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[45495]: cluster 2022-04-22T17:56:24.553905+0000 mgr.y (mgr.34107) 1087 : cluster [DBG] pgmap v1691: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 6 op/s 2022-04-22T17:56:26.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[45495]: audit 2022-04-22T17:56:25.120848+0000 mon.a (mon.0) 2553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:26.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:25 smithi115 bash[45495]: audit 2022-04-22T17:56:25.473045+0000 mon.a (mon.0) 2554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:26.080 INFO:tasks.workunit.client.0.smithi115.stdout: 568: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:26.141 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aOcVuDgGWOe80er off=1526002 len=1504739 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2905 completed, throughput=4.99MB/sec 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HuS9CMDWUnsT276 off=956354 len=791468 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2906 completed, throughput=5MB/sec 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rdaDhcJiI9Uymab off=9427132 len=1906578 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2907 completed, throughput=5MB/sec 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YgNHBT3vUefxa1I off=10245652 len=2091032 2022-04-22T17:56:26.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2908 completed, throughput=5MB/sec 2022-04-22T17:56:27.102 INFO:tasks.workunit.client.0.smithi115.stdout: 569: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:27.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iYbz-Eeve823crw off=2170335 len=1437871 2022-04-22T17:56:27.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 2909 completed, throughput=5MB/sec 2022-04-22T17:56:27.107 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iHFVU0YNCqtBaXY off=7456768 len=793713 2022-04-22T17:56:27.113 INFO:tasks.workunit.client.0.smithi115.stdout:op 2910 completed, throughput=5MB/sec 2022-04-22T17:56:27.113 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1u9Jch6-DBoD8wr off=4656172 len=927300 2022-04-22T17:56:27.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 2911 completed, throughput=5MB/sec 2022-04-22T17:56:27.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fxhh7_VUuLnpTDe off=682819 len=344212 2022-04-22T17:56:27.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 2912 completed, throughput=5MB/sec 2022-04-22T17:56:27.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y6jKDmHq6GSqQy3 off=22733761 len=1502107 2022-04-22T17:56:27.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 2913 completed, throughput=5MB/sec 2022-04-22T17:56:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:27 smithi121 bash[36902]: cluster 2022-04-22T17:56:26.555129+0000 mgr.y (mgr.34107) 1088 : cluster [DBG] pgmap v1692: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:27 smithi121 bash[36902]: audit 2022-04-22T17:56:27.508273+0000 mon.a (mon.0) 2555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:27 smithi115 bash[45495]: cluster 2022-04-22T17:56:26.555129+0000 mgr.y (mgr.34107) 1088 : cluster [DBG] pgmap v1692: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:27 smithi115 bash[45495]: audit 2022-04-22T17:56:27.508273+0000 mon.a (mon.0) 2555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:27 smithi115 bash[46934]: cluster 2022-04-22T17:56:26.555129+0000 mgr.y (mgr.34107) 1088 : cluster [DBG] pgmap v1692: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:56:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:27 smithi115 bash[46934]: audit 2022-04-22T17:56:27.508273+0000 mon.a (mon.0) 2555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:28.119 INFO:tasks.workunit.client.0.smithi115.stdout: 570: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:28.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S67s1Trij6YiLc3 off=4190384 len=143555 2022-04-22T17:56:28.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 2914 completed, throughput=4.99MB/sec 2022-04-22T17:56:28.124 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1iV5CVjcUCjWCsP off=15021059 len=1570473 2022-04-22T17:56:28.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 2915 completed, throughput=5MB/sec 2022-04-22T17:56:28.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BnpUnq8-WQcyzxR off=16193435 len=1304554 2022-04-22T17:56:28.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 2916 completed, throughput=5MB/sec 2022-04-22T17:56:28.135 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tid7O3DvvSmd6FC off=11876122 len=460052 2022-04-22T17:56:28.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 2917 completed, throughput=5MB/sec 2022-04-22T17:56:28.140 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RMnUExFot35_zK3 off=8810999 len=1666523 2022-04-22T17:56:28.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 2918 completed, throughput=5MB/sec 2022-04-22T17:56:29.143 INFO:tasks.workunit.client.0.smithi115.stdout: 571: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:29.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2XF5eAsSe31HFBT off=7906328 len=331867 2022-04-22T17:56:29.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 2919 completed, throughput=4.99MB/sec 2022-04-22T17:56:29.145 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--tKvvRFfPQCwZrA off=6012736 len=1480879 2022-04-22T17:56:29.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 2920 completed, throughput=5MB/sec 2022-04-22T17:56:29.150 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kgn8mFGMg-1ECAC off=10853364 len=1225440 2022-04-22T17:56:29.156 INFO:tasks.workunit.client.0.smithi115.stdout:op 2921 completed, throughput=5MB/sec 2022-04-22T17:56:29.156 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9cBc3nEBGGWDWPo off=2542805 len=1580963 2022-04-22T17:56:29.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 2922 completed, throughput=5MB/sec 2022-04-22T17:56:29.162 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5YOiYl0FEh2pEy5 off=3722815 len=1367277 2022-04-22T17:56:29.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 2923 completed, throughput=5MB/sec 2022-04-22T17:56:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:29 smithi121 bash[36902]: cluster 2022-04-22T17:56:28.556497+0000 mgr.y (mgr.34107) 1089 : cluster [DBG] pgmap v1693: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:29 smithi115 bash[45495]: cluster 2022-04-22T17:56:28.556497+0000 mgr.y (mgr.34107) 1089 : cluster [DBG] pgmap v1693: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:29 smithi115 bash[46934]: cluster 2022-04-22T17:56:28.556497+0000 mgr.y (mgr.34107) 1089 : cluster [DBG] pgmap v1693: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:30.167 INFO:tasks.workunit.client.0.smithi115.stdout: 572: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:30.167 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WoBX-cZSg9VLWw_ off=12307270 len=540019 2022-04-22T17:56:30.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 2924 completed, throughput=4.99MB/sec 2022-04-22T17:56:30.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F5eVB_juCpnvjhW off=13917881 len=1849336 2022-04-22T17:56:30.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 2925 completed, throughput=5MB/sec 2022-04-22T17:56:30.178 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aeL2rXDr_0UnaKJ off=24447609 len=568512 2022-04-22T17:56:30.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 2926 completed, throughput=5MB/sec 2022-04-22T17:56:30.183 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6vrT1UvjkRFjnxA off=619843 len=1967740 2022-04-22T17:56:30.187 INFO:tasks.workunit.client.0.smithi115.stdout:op 2927 completed, throughput=5MB/sec 2022-04-22T17:56:30.925 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:30 smithi121 bash[36902]: audit 2022-04-22T17:56:30.123947+0000 mon.a (mon.0) 2556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:30.925 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:30 smithi121 bash[36902]: audit 2022-04-22T17:56:30.476906+0000 mon.a (mon.0) 2557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:30 smithi115 bash[46934]: audit 2022-04-22T17:56:30.123947+0000 mon.a (mon.0) 2556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:30 smithi115 bash[46934]: audit 2022-04-22T17:56:30.476906+0000 mon.a (mon.0) 2557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:30 smithi115 bash[45495]: audit 2022-04-22T17:56:30.123947+0000 mon.a (mon.0) 2556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:30 smithi115 bash[45495]: audit 2022-04-22T17:56:30.476906+0000 mon.a (mon.0) 2557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:31.187 INFO:tasks.workunit.client.0.smithi115.stdout: 573: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:31.188 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OdnHUho3hsSEHjd off=8764554 len=1715334 2022-04-22T17:56:31.197 INFO:tasks.workunit.client.0.smithi115.stdout:op 2928 completed, throughput=5MB/sec 2022-04-22T17:56:31.197 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6V_Q_MDX8LUJMh9 off=10473647 len=1677894 2022-04-22T17:56:31.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 2929 completed, throughput=5MB/sec 2022-04-22T17:56:31.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nh68xdFBWa-_7cI off=1654302 len=161783 2022-04-22T17:56:31.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 2930 completed, throughput=5MB/sec 2022-04-22T17:56:31.203 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MNGBOr8MYvDP0zd off=8408758 len=5371 2022-04-22T17:56:31.203 INFO:tasks.workunit.client.0.smithi115.stdout:op 2931 completed, throughput=5MB/sec 2022-04-22T17:56:31.203 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_01oWeF6ANME9cA off=9474263 len=1852922 2022-04-22T17:56:31.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 2932 completed, throughput=5MB/sec 2022-04-22T17:56:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:31 smithi121 bash[36902]: cluster 2022-04-22T17:56:30.557575+0000 mgr.y (mgr.34107) 1090 : cluster [DBG] pgmap v1694: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:31.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:31 smithi115 bash[45495]: cluster 2022-04-22T17:56:30.557575+0000 mgr.y (mgr.34107) 1090 : cluster [DBG] pgmap v1694: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:31 smithi115 bash[46934]: cluster 2022-04-22T17:56:30.557575+0000 mgr.y (mgr.34107) 1090 : cluster [DBG] pgmap v1694: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:32.211 INFO:tasks.workunit.client.0.smithi115.stdout: 574: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:32.211 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--Y5SeAo183B3umZ off=1967988 len=596853 2022-04-22T17:56:32.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 2933 completed, throughput=4.99MB/sec 2022-04-22T17:56:32.212 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PuhWwI2NYMnW2Bj off=5332157 len=783413 2022-04-22T17:56:32.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 2934 completed, throughput=4.99MB/sec 2022-04-22T17:56:32.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rs4GB4WPqKnErrF off=12436131 len=953550 2022-04-22T17:56:32.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 2935 completed, throughput=5MB/sec 2022-04-22T17:56:32.219 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R3yIj53tWlyrcNm off=121903 len=1991867 2022-04-22T17:56:32.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 2936 completed, throughput=5MB/sec 2022-04-22T17:56:32.228 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_DX_ozFGP_vKKVD off=1909650 len=758591 2022-04-22T17:56:32.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 2937 completed, throughput=5MB/sec 2022-04-22T17:56:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:32 smithi121 bash[36902]: audit 2022-04-22T17:56:32.511051+0000 mon.a (mon.0) 2558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:33.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:32 smithi115 bash[45495]: audit 2022-04-22T17:56:32.511051+0000 mon.a (mon.0) 2558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:33.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:32 smithi115 bash[46934]: audit 2022-04-22T17:56:32.511051+0000 mon.a (mon.0) 2558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:33.276 INFO:tasks.workunit.client.0.smithi115.stdout: 575: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:33.276 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0wu5D82tH6GizcY off=14260580 len=843879 2022-04-22T17:56:33.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 2938 completed, throughput=4.99MB/sec 2022-04-22T17:56:33.276 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RnMnyPa7b9rt3cD off=11830677 len=323909 2022-04-22T17:56:33.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 2939 completed, throughput=4.99MB/sec 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iWaBjqKDju5xrDo off=3381656 len=380122 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2940 completed, throughput=4.99MB/sec 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-omwk59gi3aV5zci off=5161834 len=1653502 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2941 completed, throughput=5MB/sec 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kvd9BXNEk5kdHOs off=1651867 len=1973924 2022-04-22T17:56:33.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2942 completed, throughput=5MB/sec 2022-04-22T17:56:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:33 smithi121 bash[36902]: cluster 2022-04-22T17:56:32.558467+0000 mgr.y (mgr.34107) 1091 : cluster [DBG] pgmap v1695: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:56:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:33 smithi115 bash[45495]: cluster 2022-04-22T17:56:32.558467+0000 mgr.y (mgr.34107) 1091 : cluster [DBG] pgmap v1695: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:56:34.099 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:33.591Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:34.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:33 smithi115 bash[46934]: cluster 2022-04-22T17:56:32.558467+0000 mgr.y (mgr.34107) 1091 : cluster [DBG] pgmap v1695: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:56:34.252 INFO:tasks.workunit.client.0.smithi115.stdout: 576: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:34.253 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8J-BLkUwoSMof2v off=4134158 len=1791748 2022-04-22T17:56:34.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 2943 completed, throughput=4.99MB/sec 2022-04-22T17:56:34.257 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h1WYOP_0dcf7sJ8 off=4867284 len=1147526 2022-04-22T17:56:34.260 INFO:tasks.workunit.client.0.smithi115.stdout:op 2944 completed, throughput=5MB/sec 2022-04-22T17:56:34.260 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xa_eOVRICLzObaO off=5824884 len=1915106 2022-04-22T17:56:34.270 INFO:tasks.workunit.client.0.smithi115.stdout:op 2945 completed, throughput=5MB/sec 2022-04-22T17:56:34.270 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kiAKWnelJrftUeK off=6106049 len=1268572 2022-04-22T17:56:34.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 2946 completed, throughput=5MB/sec 2022-04-22T17:56:35.278 INFO:tasks.workunit.client.0.smithi115.stdout: 577: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:35.278 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2it9giWeCfIxyGj off=9641851 len=827883 2022-04-22T17:56:35.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 2947 completed, throughput=4.99MB/sec 2022-04-22T17:56:35.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JstggGY7LbFkd8E off=309836 len=1451700 2022-04-22T17:56:35.288 INFO:tasks.workunit.client.0.smithi115.stdout:op 2948 completed, throughput=5MB/sec 2022-04-22T17:56:35.288 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PRCQqNS63BWiGk6 off=17742797 len=299618 2022-04-22T17:56:35.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 2949 completed, throughput=5MB/sec 2022-04-22T17:56:35.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KQ2U5mw4KMegd6U off=6097838 len=982570 2022-04-22T17:56:35.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 2950 completed, throughput=5MB/sec 2022-04-22T17:56:35.298 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z74B4cs99ANekW1 off=9410850 len=2019514 2022-04-22T17:56:35.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 2951 completed, throughput=5MB/sec 2022-04-22T17:56:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:35 smithi121 bash[36902]: cluster 2022-04-22T17:56:34.559415+0000 mgr.y (mgr.34107) 1092 : cluster [DBG] pgmap v1696: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:35 smithi121 bash[36902]: audit 2022-04-22T17:56:35.127040+0000 mon.a (mon.0) 2559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:35 smithi121 bash[36902]: audit 2022-04-22T17:56:35.480655+0000 mon.a (mon.0) 2560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[45495]: cluster 2022-04-22T17:56:34.559415+0000 mgr.y (mgr.34107) 1092 : cluster [DBG] pgmap v1696: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[45495]: audit 2022-04-22T17:56:35.127040+0000 mon.a (mon.0) 2559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[45495]: audit 2022-04-22T17:56:35.480655+0000 mon.a (mon.0) 2560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[46934]: cluster 2022-04-22T17:56:34.559415+0000 mgr.y (mgr.34107) 1092 : cluster [DBG] pgmap v1696: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[46934]: audit 2022-04-22T17:56:35.127040+0000 mon.a (mon.0) 2559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:35 smithi115 bash[46934]: audit 2022-04-22T17:56:35.480655+0000 mon.a (mon.0) 2560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:36.302 INFO:tasks.workunit.client.0.smithi115.stdout: 578: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:36.499 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xid-mR_tCKiPzZa off=6912052 len=367789 2022-04-22T17:56:36.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 2952 completed, throughput=4.99MB/sec 2022-04-22T17:56:36.501 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wIVu-Oi6scECsSG off=12057982 len=68848 2022-04-22T17:56:36.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 2953 completed, throughput=4.99MB/sec 2022-04-22T17:56:36.501 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z-_e0AZ7Lkon9DF off=12718325 len=788249 2022-04-22T17:56:36.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 2954 completed, throughput=5MB/sec 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vfDDSNeFBwzFmwI off=12089012 len=500741 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2955 completed, throughput=5MB/sec 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G6hK6e7AiZt_zid off=4363770 len=1251602 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2956 completed, throughput=5MB/sec 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-APUcMqS2n_n6W3_ off=563809 len=169605 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2957 completed, throughput=5MB/sec 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0gfscLMhowsHmhj off=4713708 len=1095106 2022-04-22T17:56:36.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 2958 completed, throughput=5MB/sec 2022-04-22T17:56:37.321 INFO:tasks.workunit.client.0.smithi115.stdout: 579: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:37.321 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vf3Q0EdpuIai1UR off=16981295 len=796721 2022-04-22T17:56:37.323 INFO:tasks.workunit.client.0.smithi115.stdout:op 2959 completed, throughput=4.99MB/sec 2022-04-22T17:56:37.324 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SQxcI2DqEa7OPBK off=13837439 len=544539 2022-04-22T17:56:37.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 2960 completed, throughput=4.99MB/sec 2022-04-22T17:56:37.330 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FW83wx59PsXte2M off=14051448 len=541911 2022-04-22T17:56:37.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 2961 completed, throughput=4.99MB/sec 2022-04-22T17:56:37.336 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nkq8iVjQI9aZEJf off=5520461 len=1124448 2022-04-22T17:56:37.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 2962 completed, throughput=5MB/sec 2022-04-22T17:56:37.343 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TroYF3FeBJcEVZG off=1448573 len=243887 2022-04-22T17:56:37.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 2963 completed, throughput=5MB/sec 2022-04-22T17:56:37.347 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qbvSVV2FjPHlLFh off=17000161 len=1513948 2022-04-22T17:56:37.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 2964 completed, throughput=5MB/sec 2022-04-22T17:56:37.354 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q-KdzSY_z_gJbdN off=11273020 len=2057699 2022-04-22T17:56:37.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 2965 completed, throughput=5MB/sec 2022-04-22T17:56:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:37 smithi121 bash[36902]: cluster 2022-04-22T17:56:36.560438+0000 mgr.y (mgr.34107) 1093 : cluster [DBG] pgmap v1697: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:37 smithi121 bash[36902]: audit 2022-04-22T17:56:37.514119+0000 mon.a (mon.0) 2561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:38.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:37 smithi115 bash[46934]: cluster 2022-04-22T17:56:36.560438+0000 mgr.y (mgr.34107) 1093 : cluster [DBG] pgmap v1697: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:38.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:37 smithi115 bash[46934]: audit 2022-04-22T17:56:37.514119+0000 mon.a (mon.0) 2561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:37 smithi115 bash[45495]: cluster 2022-04-22T17:56:36.560438+0000 mgr.y (mgr.34107) 1093 : cluster [DBG] pgmap v1697: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:56:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:37 smithi115 bash[45495]: audit 2022-04-22T17:56:37.514119+0000 mon.a (mon.0) 2561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:38.361 INFO:tasks.workunit.client.0.smithi115.stdout: 580: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:38.362 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JML4qtL--WLT0b0 off=3341943 len=478047 2022-04-22T17:56:38.368 INFO:tasks.workunit.client.0.smithi115.stdout:op 2966 completed, throughput=4.99MB/sec 2022-04-22T17:56:38.368 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KKARAVN1Bm3BPua off=4876153 len=572972 2022-04-22T17:56:38.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 2967 completed, throughput=5MB/sec 2022-04-22T17:56:38.369 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_Nd6htbRMmvlrpm off=4085086 len=34864 2022-04-22T17:56:38.373 INFO:tasks.workunit.client.0.smithi115.stdout:op 2968 completed, throughput=5MB/sec 2022-04-22T17:56:38.373 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r3UXybCr3XnEDpJ off=3609613 len=436984 2022-04-22T17:56:38.374 INFO:tasks.workunit.client.0.smithi115.stdout:op 2969 completed, throughput=5MB/sec 2022-04-22T17:56:38.374 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-riZfkdftA8LdyWE off=1863400 len=85768 2022-04-22T17:56:38.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 2970 completed, throughput=5MB/sec 2022-04-22T17:56:38.375 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VMi8S9CQbgdK57I off=7385677 len=1116864 2022-04-22T17:56:38.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 2971 completed, throughput=5MB/sec 2022-04-22T17:56:38.379 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3nc38Zp7ss8LVkc off=3670001 len=40353 2022-04-22T17:56:38.383 INFO:tasks.workunit.client.0.smithi115.stdout:op 2972 completed, throughput=5MB/sec 2022-04-22T17:56:38.383 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3DGEyPgkQQiKJZl off=18531667 len=88962 2022-04-22T17:56:38.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 2973 completed, throughput=5MB/sec 2022-04-22T17:56:38.384 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DtbNNb1oeXWXFUJ off=15330533 len=272122 2022-04-22T17:56:38.385 INFO:tasks.workunit.client.0.smithi115.stdout:op 2974 completed, throughput=5MB/sec 2022-04-22T17:56:38.385 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6L__ThOtr-xJ1hr off=9368132 len=802437 2022-04-22T17:56:38.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 2975 completed, throughput=5MB/sec 2022-04-22T17:56:39.388 INFO:tasks.workunit.client.0.smithi115.stdout: 581: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:39.388 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NfotoFm5NXv_rIU off=1450589 len=20590 2022-04-22T17:56:39.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 2976 completed, throughput=4.99MB/sec 2022-04-22T17:56:39.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2j5ybboYJw3Iipw off=14750879 len=708611 2022-04-22T17:56:39.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 2977 completed, throughput=4.99MB/sec 2022-04-22T17:56:39.399 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oiYINHoN4uPhxgR off=18102220 len=922747 2022-04-22T17:56:39.401 INFO:tasks.workunit.client.0.smithi115.stdout:op 2978 completed, throughput=4.99MB/sec 2022-04-22T17:56:39.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2Sc8RN-rXId9M9Q off=9895743 len=404915 2022-04-22T17:56:39.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 2979 completed, throughput=4.99MB/sec 2022-04-22T17:56:39.403 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JsVZTs3oix875St off=24053215 len=1306324 2022-04-22T17:56:39.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 2980 completed, throughput=5MB/sec 2022-04-22T17:56:39.410 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iju7LVK0tegqFiN off=1257154 len=389349 2022-04-22T17:56:39.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 2981 completed, throughput=5MB/sec 2022-04-22T17:56:39.411 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bjve2E-cFkxxuSQ off=2926389 len=1587848 2022-04-22T17:56:39.418 INFO:tasks.workunit.client.0.smithi115.stdout:op 2982 completed, throughput=5MB/sec 2022-04-22T17:56:39.418 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U_PGC5xBgwO83iu off=15866953 len=1134867 2022-04-22T17:56:39.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 2983 completed, throughput=5MB/sec 2022-04-22T17:56:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:39 smithi121 bash[36902]: cluster 2022-04-22T17:56:38.561634+0000 mgr.y (mgr.34107) 1094 : cluster [DBG] pgmap v1698: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:56:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:39 smithi115 bash[46934]: cluster 2022-04-22T17:56:38.561634+0000 mgr.y (mgr.34107) 1094 : cluster [DBG] pgmap v1698: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:56:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:39 smithi115 bash[45495]: cluster 2022-04-22T17:56:38.561634+0000 mgr.y (mgr.34107) 1094 : cluster [DBG] pgmap v1698: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:56:40.423 INFO:tasks.workunit.client.0.smithi115.stdout: 582: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:40.423 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0aZ5NaLfY_ct3aa off=4529214 len=1005154 2022-04-22T17:56:40.431 INFO:tasks.workunit.client.0.smithi115.stdout:op 2984 completed, throughput=4.99MB/sec 2022-04-22T17:56:40.431 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jtoVuQrTAQpGJwX off=1480276 len=1901499 2022-04-22T17:56:40.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 2985 completed, throughput=5MB/sec 2022-04-22T17:56:40.433 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VBnlkb5l-eT10Y0 off=1466903 len=1287497 2022-04-22T17:56:40.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 2986 completed, throughput=5MB/sec 2022-04-22T17:56:40.440 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2ZHehJLWjo3eih2 off=9555589 len=1404373 2022-04-22T17:56:40.442 INFO:tasks.workunit.client.0.smithi115.stdout:op 2987 completed, throughput=5MB/sec 2022-04-22T17:56:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[45495]: audit 2022-04-22T17:56:40.130217+0000 mon.a (mon.0) 2562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:41.223 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[45495]: audit 2022-04-22T17:56:40.232886+0000 mon.a (mon.0) 2563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:56:41.223 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[45495]: audit 2022-04-22T17:56:40.235881+0000 mon.a (mon.0) 2564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:56:41.223 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[45495]: audit 2022-04-22T17:56:40.237553+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:56:41.223 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[45495]: audit 2022-04-22T17:56:40.484121+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:41.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[46934]: audit 2022-04-22T17:56:40.130217+0000 mon.a (mon.0) 2562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:41.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[46934]: audit 2022-04-22T17:56:40.232886+0000 mon.a (mon.0) 2563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:56:41.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[46934]: audit 2022-04-22T17:56:40.235881+0000 mon.a (mon.0) 2564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:56:41.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[46934]: audit 2022-04-22T17:56:40.237553+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:40 smithi115 bash[46934]: audit 2022-04-22T17:56:40.484121+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:40 smithi121 bash[36902]: audit 2022-04-22T17:56:40.130217+0000 mon.a (mon.0) 2562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:40 smithi121 bash[36902]: audit 2022-04-22T17:56:40.232886+0000 mon.a (mon.0) 2563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:40 smithi121 bash[36902]: audit 2022-04-22T17:56:40.235881+0000 mon.a (mon.0) 2564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:40 smithi121 bash[36902]: audit 2022-04-22T17:56:40.237553+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:56:41.228 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:40 smithi121 bash[36902]: audit 2022-04-22T17:56:40.484121+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:41.443 INFO:tasks.workunit.client.0.smithi115.stdout: 583: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:41.443 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-07irIg9N9tfC_no off=16025610 len=1644705 2022-04-22T17:56:41.448 INFO:tasks.workunit.client.0.smithi115.stdout:op 2988 completed, throughput=5MB/sec 2022-04-22T17:56:41.448 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9TH-RqKZhOwCU9v off=1701596 len=229492 2022-04-22T17:56:41.453 INFO:tasks.workunit.client.0.smithi115.stdout:op 2989 completed, throughput=5MB/sec 2022-04-22T17:56:41.453 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ulZQuEsmleg4IPd off=2730687 len=820959 2022-04-22T17:56:41.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 2990 completed, throughput=5MB/sec 2022-04-22T17:56:41.459 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-78i5sIWvEDYFXNX off=2407391 len=587547 2022-04-22T17:56:41.460 INFO:tasks.workunit.client.0.smithi115.stdout:op 2991 completed, throughput=5MB/sec 2022-04-22T17:56:41.460 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TcXSLFbXIOhfSde off=13289191 len=980502 2022-04-22T17:56:41.465 INFO:tasks.workunit.client.0.smithi115.stdout:op 2992 completed, throughput=5MB/sec 2022-04-22T17:56:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:41 smithi121 bash[36902]: cluster 2022-04-22T17:56:40.562409+0000 mgr.y (mgr.34107) 1095 : cluster [DBG] pgmap v1699: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:42.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:41 smithi115 bash[45495]: cluster 2022-04-22T17:56:40.562409+0000 mgr.y (mgr.34107) 1095 : cluster [DBG] pgmap v1699: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:42.056 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:41.709Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:42.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:41 smithi115 bash[46934]: cluster 2022-04-22T17:56:40.562409+0000 mgr.y (mgr.34107) 1095 : cluster [DBG] pgmap v1699: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:56:42.466 INFO:tasks.workunit.client.0.smithi115.stdout: 584: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:42.466 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sfuCbEv5zVFwi2W off=1148895 len=1826452 2022-04-22T17:56:42.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 2993 completed, throughput=4.99MB/sec 2022-04-22T17:56:42.470 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aoePTjMZSKzicY1 off=641663 len=379107 2022-04-22T17:56:42.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 2994 completed, throughput=5MB/sec 2022-04-22T17:56:42.481 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tyLAuKfBbt7MfNI off=3600392 len=868279 2022-04-22T17:56:42.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 2995 completed, throughput=5MB/sec 2022-04-22T17:56:42.481 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-84TTfdcb_lV8HKh off=13830253 len=51779 2022-04-22T17:56:42.482 INFO:tasks.workunit.client.0.smithi115.stdout:op 2996 completed, throughput=5MB/sec 2022-04-22T17:56:42.482 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lbqa00ElfW7v5EC off=19300035 len=2068956 2022-04-22T17:56:42.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 2997 completed, throughput=5MB/sec 2022-04-22T17:56:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:42 smithi121 bash[36902]: audit 2022-04-22T17:56:42.517511+0000 mon.a (mon.0) 2567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:42 smithi115 bash[46934]: audit 2022-04-22T17:56:42.517511+0000 mon.a (mon.0) 2567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:42 smithi115 bash[45495]: audit 2022-04-22T17:56:42.517511+0000 mon.a (mon.0) 2567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:43.140 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:56:43.485 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:56:43.485 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (19m) 9m ago 33m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:56:43.485 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (33m) 10m ago 33m 24.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (21m) 9m ago 34m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (21m) 10m ago 34m 13.1M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (29m) 10m ago 38m 410M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (29m) 9m ago 41m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (28m) 9m ago 41m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (27m) 10m ago 39m 83.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (28m) 9m ago 39m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (20m) 9m ago 33m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:56:43.486 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (20m) 10m ago 33m 12.1M - 1dbe0e931976 f4b044642a96 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (26m) 9m ago 38m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (25m) 9m ago 37m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (25m) 9m ago 37m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (24m) 9m ago 36m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (24m) 10m ago 36m 469M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (23m) 10m ago 36m 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (22m) 10m ago 35m 364M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (22m) 10m ago 35m 465M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:56:43.487 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (20m) 10m ago 33m 50.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:56:43.491 INFO:tasks.workunit.client.0.smithi115.stdout: 585: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:43.491 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eUOtjyygTTMrzKd off=15498458 len=813695 2022-04-22T17:56:43.495 INFO:tasks.workunit.client.0.smithi115.stdout:op 2998 completed, throughput=4.99MB/sec 2022-04-22T17:56:43.495 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-T2jO-8Ye4zfPeZi off=11172816 len=464655 2022-04-22T17:56:43.499 INFO:tasks.workunit.client.0.smithi115.stdout:op 2999 completed, throughput=4.99MB/sec 2022-04-22T17:56:43.499 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0OlwTi4lEGtWqTE off=4411732 len=1677422 2022-04-22T17:56:43.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 3000 completed, throughput=5MB/sec 2022-04-22T17:56:43.503 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XePb_5k0F2gHj9i off=107129 len=188948 2022-04-22T17:56:43.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 3001 completed, throughput=5MB/sec 2022-04-22T17:56:43.503 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3V0ekjCf2pGj9z8 off=2260011 len=426283 2022-04-22T17:56:43.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 3002 completed, throughput=5MB/sec 2022-04-22T17:56:43.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Gbf5g32NwdslCwl off=12547809 len=2056855 2022-04-22T17:56:43.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 3003 completed, throughput=5MB/sec 2022-04-22T17:56:43.894 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:56:44.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:56:44.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:56:44.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:44.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:56:44.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:56:44.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:56:44.095 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:56:44.095 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:56:44.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:43 smithi121 bash[36902]: cluster 2022-04-22T17:56:42.563464+0000 mgr.y (mgr.34107) 1096 : cluster [DBG] pgmap v1700: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:44.096 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:43 smithi121 bash[36902]: audit 2022-04-22T17:56:43.118396+0000 mgr.y (mgr.34107) 1097 : audit [DBG] from='client.45493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:44.096 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:43.591Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:44.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:43 smithi115 bash[45495]: cluster 2022-04-22T17:56:42.563464+0000 mgr.y (mgr.34107) 1096 : cluster [DBG] pgmap v1700: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:44.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:43 smithi115 bash[45495]: audit 2022-04-22T17:56:43.118396+0000 mgr.y (mgr.34107) 1097 : audit [DBG] from='client.45493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:44.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:43 smithi115 bash[46934]: cluster 2022-04-22T17:56:42.563464+0000 mgr.y (mgr.34107) 1096 : cluster [DBG] pgmap v1700: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:44.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:43 smithi115 bash[46934]: audit 2022-04-22T17:56:43.118396+0000 mgr.y (mgr.34107) 1097 : audit [DBG] from='client.45493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:44.517 INFO:tasks.workunit.client.0.smithi115.stdout: 586: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:44.518 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jobxJ7US87n5lOr off=1179373 len=716039 2022-04-22T17:56:44.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 3004 completed, throughput=4.99MB/sec 2022-04-22T17:56:44.523 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lSSUvn_asq64Hc_ off=7603109 len=1823585 2022-04-22T17:56:44.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 3005 completed, throughput=5MB/sec 2022-04-22T17:56:44.526 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aA0GCnLM1paGBPW off=97499 len=2024187 2022-04-22T17:56:44.536 INFO:tasks.workunit.client.0.smithi115.stdout:op 3006 completed, throughput=5MB/sec 2022-04-22T17:56:44.536 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bNfbEQgsr6f-D3S off=5419740 len=204368 2022-04-22T17:56:44.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 3007 completed, throughput=5MB/sec 2022-04-22T17:56:44.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eSBvup61I2ve1AC off=8903336 len=1889228 2022-04-22T17:56:44.544 INFO:tasks.workunit.client.0.smithi115.stdout:op 3008 completed, throughput=5MB/sec 2022-04-22T17:56:45.008 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:44 smithi121 bash[36902]: audit 2022-04-22T17:56:43.459741+0000 mgr.y (mgr.34107) 1098 : audit [DBG] from='client.45499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:45.009 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:44 smithi121 bash[36902]: audit 2022-04-22T17:56:43.886804+0000 mon.a (mon.0) 2568 : audit [DBG] from='client.? 172.21.15.115:0/2147034870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:44 smithi115 bash[45495]: audit 2022-04-22T17:56:43.459741+0000 mgr.y (mgr.34107) 1098 : audit [DBG] from='client.45499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:44 smithi115 bash[45495]: audit 2022-04-22T17:56:43.886804+0000 mon.a (mon.0) 2568 : audit [DBG] from='client.? 172.21.15.115:0/2147034870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:44 smithi115 bash[46934]: audit 2022-04-22T17:56:43.459741+0000 mgr.y (mgr.34107) 1098 : audit [DBG] from='client.45499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:56:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:44 smithi115 bash[46934]: audit 2022-04-22T17:56:43.886804+0000 mon.a (mon.0) 2568 : audit [DBG] from='client.? 172.21.15.115:0/2147034870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:56:45.544 INFO:tasks.workunit.client.0.smithi115.stdout: 587: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:45.544 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nMqn68msHPf2eSh off=9472778 len=1133388 2022-04-22T17:56:45.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 3009 completed, throughput=5MB/sec 2022-04-22T17:56:45.552 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0tW_y4bf5-tLoIc off=3444060 len=1587986 2022-04-22T17:56:45.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 3010 completed, throughput=5MB/sec 2022-04-22T17:56:45.554 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H95Ze78fRizAO_v off=1983205 len=64057 2022-04-22T17:56:45.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 3011 completed, throughput=5MB/sec 2022-04-22T17:56:45.558 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kWaqUsU7qlSlj4y off=2019960 len=841687 2022-04-22T17:56:45.560 INFO:tasks.workunit.client.0.smithi115.stdout:op 3012 completed, throughput=5MB/sec 2022-04-22T17:56:45.560 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pI0nXNH8pVamxb5 off=21661750 len=16733 2022-04-22T17:56:45.561 INFO:tasks.workunit.client.0.smithi115.stdout:op 3013 completed, throughput=5MB/sec 2022-04-22T17:56:45.561 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nP8azHjgLEBZiYg off=2454542 len=806890 2022-04-22T17:56:45.566 INFO:tasks.workunit.client.0.smithi115.stdout:op 3014 completed, throughput=5MB/sec 2022-04-22T17:56:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:45 smithi121 bash[36902]: cluster 2022-04-22T17:56:44.564567+0000 mgr.y (mgr.34107) 1099 : cluster [DBG] pgmap v1701: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:56:46.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:45 smithi121 bash[36902]: audit 2022-04-22T17:56:44.699572+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:45 smithi121 bash[36902]: audit 2022-04-22T17:56:45.013808+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:45 smithi121 bash[36902]: audit 2022-04-22T17:56:45.133145+0000 mon.a (mon.0) 2571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:46.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:45 smithi121 bash[36902]: audit 2022-04-22T17:56:45.487014+0000 mon.a (mon.0) 2572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[45495]: cluster 2022-04-22T17:56:44.564567+0000 mgr.y (mgr.34107) 1099 : cluster [DBG] pgmap v1701: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:56:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[45495]: audit 2022-04-22T17:56:44.699572+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[45495]: audit 2022-04-22T17:56:45.013808+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[45495]: audit 2022-04-22T17:56:45.133145+0000 mon.a (mon.0) 2571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[45495]: audit 2022-04-22T17:56:45.487014+0000 mon.a (mon.0) 2572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[46934]: cluster 2022-04-22T17:56:44.564567+0000 mgr.y (mgr.34107) 1099 : cluster [DBG] pgmap v1701: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 6 op/s 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[46934]: audit 2022-04-22T17:56:44.699572+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[46934]: audit 2022-04-22T17:56:45.013808+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[46934]: audit 2022-04-22T17:56:45.133145+0000 mon.a (mon.0) 2571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:45 smithi115 bash[46934]: audit 2022-04-22T17:56:45.487014+0000 mon.a (mon.0) 2572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:46.566 INFO:tasks.workunit.client.0.smithi115.stdout: 588: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:46.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0MlBwyKncL5ZmjS off=5750397 len=323258 2022-04-22T17:56:46.572 INFO:tasks.workunit.client.0.smithi115.stdout:op 3015 completed, throughput=4.99MB/sec 2022-04-22T17:56:46.572 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-igXJeUIeXeEMiAC off=15924499 len=1331626 2022-04-22T17:56:46.576 INFO:tasks.workunit.client.0.smithi115.stdout:op 3016 completed, throughput=5MB/sec 2022-04-22T17:56:46.576 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-90GPp1WAyFrUTrs off=3148637 len=1728779 2022-04-22T17:56:46.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 3017 completed, throughput=5MB/sec 2022-04-22T17:56:46.585 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2YGBJFTbS382jvD off=1377876 len=1877714 2022-04-22T17:56:46.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 3018 completed, throughput=5MB/sec 2022-04-22T17:56:47.600 INFO:tasks.workunit.client.0.smithi115.stdout: 589: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:47.601 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OZxLMkqskDV0mf4 off=630221 len=1738956 2022-04-22T17:56:47.601 INFO:tasks.workunit.client.0.smithi115.stdout:op 3019 completed, throughput=5MB/sec 2022-04-22T17:56:47.601 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-647cuitENb13W8J off=156793 len=1919607 2022-04-22T17:56:47.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 3020 completed, throughput=5MB/sec 2022-04-22T17:56:47.607 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-McdltjB_s6L_cKv off=13567155 len=1017065 2022-04-22T17:56:47.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 3021 completed, throughput=5MB/sec 2022-04-22T17:56:47.612 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1k88fCZm42KF3nm off=19603596 len=2006832 2022-04-22T17:56:47.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 3022 completed, throughput=5MB/sec 2022-04-22T17:56:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:47 smithi121 bash[36902]: cluster 2022-04-22T17:56:46.565647+0000 mgr.y (mgr.34107) 1100 : cluster [DBG] pgmap v1702: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:48.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:47 smithi121 bash[36902]: audit 2022-04-22T17:56:47.520401+0000 mon.a (mon.0) 2573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:47 smithi115 bash[45495]: cluster 2022-04-22T17:56:46.565647+0000 mgr.y (mgr.34107) 1100 : cluster [DBG] pgmap v1702: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:47 smithi115 bash[45495]: audit 2022-04-22T17:56:47.520401+0000 mon.a (mon.0) 2573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:48.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:47 smithi115 bash[46934]: cluster 2022-04-22T17:56:46.565647+0000 mgr.y (mgr.34107) 1100 : cluster [DBG] pgmap v1702: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:48.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:47 smithi115 bash[46934]: audit 2022-04-22T17:56:47.520401+0000 mon.a (mon.0) 2573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:48.622 INFO:tasks.workunit.client.0.smithi115.stdout: 590: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:48.622 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KrEnZIxum4kHsux off=7789781 len=393572 2022-04-22T17:56:48.627 INFO:tasks.workunit.client.0.smithi115.stdout:op 3023 completed, throughput=5MB/sec 2022-04-22T17:56:48.627 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o0_Azi-BWPCBWuF off=10401086 len=1363159 2022-04-22T17:56:48.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 3024 completed, throughput=5MB/sec 2022-04-22T17:56:48.635 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7SO84XL-6amKBgX off=1006779 len=119696 2022-04-22T17:56:48.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 3025 completed, throughput=5MB/sec 2022-04-22T17:56:48.635 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O75-A23xtNcQvFx off=13418144 len=606912 2022-04-22T17:56:48.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 3026 completed, throughput=5MB/sec 2022-04-22T17:56:48.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Kn1-Iu6iYv2VjO0 off=13328323 len=1614640 2022-04-22T17:56:48.649 INFO:tasks.workunit.client.0.smithi115.stdout:op 3027 completed, throughput=5MB/sec 2022-04-22T17:56:49.650 INFO:tasks.workunit.client.0.smithi115.stdout: 591: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:49.650 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-N3xUuX6nhgXTRT2 off=18388138 len=447093 2022-04-22T17:56:49.655 INFO:tasks.workunit.client.0.smithi115.stdout:op 3028 completed, throughput=4.99MB/sec 2022-04-22T17:56:49.655 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ibUmqU5Zp2W7oK8 off=3520240 len=1739972 2022-04-22T17:56:49.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 3029 completed, throughput=5MB/sec 2022-04-22T17:56:49.664 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-38X9TYHiVvoo_r9 off=335039 len=979316 2022-04-22T17:56:49.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 3030 completed, throughput=5MB/sec 2022-04-22T17:56:49.775 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JbYgnBPuD6pm7Mw off=4354753 len=1216107 2022-04-22T17:56:49.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 3031 completed, throughput=5MB/sec 2022-04-22T17:56:49.777 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uevm3AM0PQFL4YZ off=7950348 len=2001677 2022-04-22T17:56:49.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 3032 completed, throughput=5MB/sec 2022-04-22T17:56:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:49 smithi121 bash[36902]: cluster 2022-04-22T17:56:48.566806+0000 mgr.y (mgr.34107) 1101 : cluster [DBG] pgmap v1703: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:49 smithi115 bash[45495]: cluster 2022-04-22T17:56:48.566806+0000 mgr.y (mgr.34107) 1101 : cluster [DBG] pgmap v1703: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:49 smithi115 bash[46934]: cluster 2022-04-22T17:56:48.566806+0000 mgr.y (mgr.34107) 1101 : cluster [DBG] pgmap v1703: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:56:50.681 INFO:tasks.workunit.client.0.smithi115.stdout: 592: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:50.681 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zBej6olSO1O_ksA off=864060 len=2080350 2022-04-22T17:56:50.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 3033 completed, throughput=5MB/sec 2022-04-22T17:56:50.692 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bYk9JEG5oJ6HMhY off=9923944 len=1444177 2022-04-22T17:56:50.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 3034 completed, throughput=5MB/sec 2022-04-22T17:56:50.695 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Qab-rWzj4nvQ5U9 off=1643877 len=426424 2022-04-22T17:56:50.696 INFO:tasks.workunit.client.0.smithi115.stdout:op 3035 completed, throughput=5MB/sec 2022-04-22T17:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:50 smithi121 bash[36902]: audit 2022-04-22T17:56:50.135997+0000 mon.a (mon.0) 2574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:50 smithi121 bash[36902]: audit 2022-04-22T17:56:50.490840+0000 mon.a (mon.0) 2575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:50 smithi115 bash[45495]: audit 2022-04-22T17:56:50.135997+0000 mon.a (mon.0) 2574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:50 smithi115 bash[45495]: audit 2022-04-22T17:56:50.490840+0000 mon.a (mon.0) 2575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:50 smithi115 bash[46934]: audit 2022-04-22T17:56:50.135997+0000 mon.a (mon.0) 2574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:50 smithi115 bash[46934]: audit 2022-04-22T17:56:50.490840+0000 mon.a (mon.0) 2575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:51.697 INFO:tasks.workunit.client.0.smithi115.stdout: 593: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:51.818 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hkKVWsPYFXkJ6a9 off=10527897 len=867053 2022-04-22T17:56:51.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3036 completed, throughput=4.99MB/sec 2022-04-22T17:56:51.819 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7_YMpNc6plIKWym off=1292456 len=1790740 2022-04-22T17:56:51.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3037 completed, throughput=5MB/sec 2022-04-22T17:56:51.819 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GWERbtTo7cSQpjq off=13013831 len=1753906 2022-04-22T17:56:51.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3038 completed, throughput=5MB/sec 2022-04-22T17:56:51.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3SLN6EXrhqqQ4e off=12551684 len=1203635 2022-04-22T17:56:51.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 3039 completed, throughput=5MB/sec 2022-04-22T17:56:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:51 smithi121 bash[36902]: cluster 2022-04-22T17:56:50.567922+0000 mgr.y (mgr.34107) 1102 : cluster [DBG] pgmap v1704: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:56:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:51 smithi115 bash[45495]: cluster 2022-04-22T17:56:50.567922+0000 mgr.y (mgr.34107) 1102 : cluster [DBG] pgmap v1704: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:56:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:51 smithi115 bash[46934]: cluster 2022-04-22T17:56:50.567922+0000 mgr.y (mgr.34107) 1102 : cluster [DBG] pgmap v1704: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:56:52.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:51.707Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:52.716 INFO:tasks.workunit.client.0.smithi115.stdout: 594: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:52.716 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8yTT3OTpgMTjKqV off=7683413 len=827224 2022-04-22T17:56:52.724 INFO:tasks.workunit.client.0.smithi115.stdout:op 3040 completed, throughput=4.99MB/sec 2022-04-22T17:56:52.724 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5kYgo_mSLZtV0tL off=659089 len=1003757 2022-04-22T17:56:52.726 INFO:tasks.workunit.client.0.smithi115.stdout:op 3041 completed, throughput=5MB/sec 2022-04-22T17:56:52.727 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HDBuEcXpRtL3SPZ off=1222495 len=606939 2022-04-22T17:56:52.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 3042 completed, throughput=5MB/sec 2022-04-22T17:56:52.729 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V2KbiceQ-93l595 off=9399833 len=222850 2022-04-22T17:56:52.730 INFO:tasks.workunit.client.0.smithi115.stdout:op 3043 completed, throughput=5MB/sec 2022-04-22T17:56:52.731 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MEtthNLGYHbgFgN off=2362804 len=196617 2022-04-22T17:56:52.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 3044 completed, throughput=5MB/sec 2022-04-22T17:56:52.732 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wc4rAPiOJS0_WJ_ off=6898698 len=282228 2022-04-22T17:56:52.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 3045 completed, throughput=5MB/sec 2022-04-22T17:56:52.733 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6OK1pkS_YGGwHQk off=20254372 len=1786228 2022-04-22T17:56:52.736 INFO:tasks.workunit.client.0.smithi115.stdout:op 3046 completed, throughput=5MB/sec 2022-04-22T17:56:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:52 smithi121 bash[36902]: audit 2022-04-22T17:56:52.523287+0000 mon.a (mon.0) 2576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:52 smithi115 bash[45495]: audit 2022-04-22T17:56:52.523287+0000 mon.a (mon.0) 2576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:52 smithi115 bash[46934]: audit 2022-04-22T17:56:52.523287+0000 mon.a (mon.0) 2576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:53.736 INFO:tasks.workunit.client.0.smithi115.stdout: 595: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:53.736 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wCNPiSI5rRO8sSL off=1792302 len=59190 2022-04-22T17:56:53.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 3047 completed, throughput=4.99MB/sec 2022-04-22T17:56:53.738 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9eyzRjnUJb3-t4A off=9398300 len=822939 2022-04-22T17:56:53.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 3048 completed, throughput=4.99MB/sec 2022-04-22T17:56:53.743 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ip7-bUwBHhOt9e9 off=922616 len=995992 2022-04-22T17:56:53.749 INFO:tasks.workunit.client.0.smithi115.stdout:op 3049 completed, throughput=4.99MB/sec 2022-04-22T17:56:53.749 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lHgnby7daNiSxsO off=6363329 len=319287 2022-04-22T17:56:53.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 3050 completed, throughput=5MB/sec 2022-04-22T17:56:53.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O7Tc3C9SZjz2-Zy off=2187527 len=1389329 2022-04-22T17:56:53.756 INFO:tasks.workunit.client.0.smithi115.stdout:op 3051 completed, throughput=5MB/sec 2022-04-22T17:56:53.756 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uVwCkBM9mGrkkqi off=14542241 len=864321 2022-04-22T17:56:53.758 INFO:tasks.workunit.client.0.smithi115.stdout:op 3052 completed, throughput=5MB/sec 2022-04-22T17:56:53.758 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K3MPFWn_mwRmgrD off=458874 len=1048696 2022-04-22T17:56:53.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 3053 completed, throughput=5MB/sec 2022-04-22T17:56:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:53 smithi121 bash[36902]: cluster 2022-04-22T17:56:52.569091+0000 mgr.y (mgr.34107) 1103 : cluster [DBG] pgmap v1705: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T17:56:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:53 smithi115 bash[45495]: cluster 2022-04-22T17:56:52.569091+0000 mgr.y (mgr.34107) 1103 : cluster [DBG] pgmap v1705: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T17:56:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:56:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:56:53.592Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:56:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:53 smithi115 bash[46934]: cluster 2022-04-22T17:56:52.569091+0000 mgr.y (mgr.34107) 1103 : cluster [DBG] pgmap v1705: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T17:56:54.764 INFO:tasks.workunit.client.0.smithi115.stdout: 596: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:54.764 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yBXg82sm5BydmPX off=4804401 len=1751068 2022-04-22T17:56:54.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 3054 completed, throughput=4.99MB/sec 2022-04-22T17:56:54.768 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_RFAlIUaXVFuPXL off=9707502 len=1255266 2022-04-22T17:56:54.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 3055 completed, throughput=5MB/sec 2022-04-22T17:56:54.777 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V36tm4kPkPtBYVZ off=6460838 len=457294 2022-04-22T17:56:54.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 3056 completed, throughput=5MB/sec 2022-04-22T17:56:54.778 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5BpdrnjYu3O_SXt off=6674210 len=1988287 2022-04-22T17:56:54.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 3057 completed, throughput=5MB/sec 2022-04-22T17:56:55.782 INFO:tasks.workunit.client.0.smithi115.stdout: 597: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:55.782 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0D7X78jMm_LvU58 off=1182835 len=317851 2022-04-22T17:56:55.787 INFO:tasks.workunit.client.0.smithi115.stdout:op 3058 completed, throughput=4.99MB/sec 2022-04-22T17:56:55.787 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Als2W7klskjjw1w off=2170809 len=454359 2022-04-22T17:56:55.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 3059 completed, throughput=4.99MB/sec 2022-04-22T17:56:55.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LGrp5TK95Yqf4GM off=6011250 len=430243 2022-04-22T17:56:55.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 3060 completed, throughput=4.99MB/sec 2022-04-22T17:56:55.794 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hz24KZbKf1MNBe4 off=1292518 len=663998 2022-04-22T17:56:55.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 3061 completed, throughput=4.99MB/sec 2022-04-22T17:56:55.796 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3xeD2CU3aIliifr off=7170974 len=781337 2022-04-22T17:56:55.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 3062 completed, throughput=5MB/sec 2022-04-22T17:56:55.802 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X39r-WtGViOqDlG off=11155846 len=227230 2022-04-22T17:56:55.805 INFO:tasks.workunit.client.0.smithi115.stdout:op 3063 completed, throughput=5MB/sec 2022-04-22T17:56:55.806 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5dIZfyZWy6aERjW off=1016482 len=2072376 2022-04-22T17:56:55.813 INFO:tasks.workunit.client.0.smithi115.stdout:op 3064 completed, throughput=5MB/sec 2022-04-22T17:56:55.813 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4DXHdl6NkajqXD2 off=19302187 len=314565 2022-04-22T17:56:55.818 INFO:tasks.workunit.client.0.smithi115.stdout:op 3065 completed, throughput=5MB/sec 2022-04-22T17:56:56.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:55 smithi121 bash[36902]: cluster 2022-04-22T17:56:54.570193+0000 mgr.y (mgr.34107) 1104 : cluster [DBG] pgmap v1706: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:56:56.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:55 smithi121 bash[36902]: audit 2022-04-22T17:56:55.139272+0000 mon.a (mon.0) 2577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:55 smithi121 bash[36902]: audit 2022-04-22T17:56:55.494489+0000 mon.a (mon.0) 2578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[45495]: cluster 2022-04-22T17:56:54.570193+0000 mgr.y (mgr.34107) 1104 : cluster [DBG] pgmap v1706: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:56:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[45495]: audit 2022-04-22T17:56:55.139272+0000 mon.a (mon.0) 2577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[45495]: audit 2022-04-22T17:56:55.494489+0000 mon.a (mon.0) 2578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[46934]: cluster 2022-04-22T17:56:54.570193+0000 mgr.y (mgr.34107) 1104 : cluster [DBG] pgmap v1706: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T17:56:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[46934]: audit 2022-04-22T17:56:55.139272+0000 mon.a (mon.0) 2577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:56:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:55 smithi115 bash[46934]: audit 2022-04-22T17:56:55.494489+0000 mon.a (mon.0) 2578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:56:56.818 INFO:tasks.workunit.client.0.smithi115.stdout: 598: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:56.818 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_8A4emTRFbuu6j3 off=1558405 len=964769 2022-04-22T17:56:56.827 INFO:tasks.workunit.client.0.smithi115.stdout:op 3066 completed, throughput=4.99MB/sec 2022-04-22T17:56:56.827 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f6MSNE1tUpAPzQ9 off=7099231 len=1960499 2022-04-22T17:56:56.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 3067 completed, throughput=5MB/sec 2022-04-22T17:56:56.837 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xRsTmY0sqtnA10I off=3365902 len=350554 2022-04-22T17:56:56.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 3068 completed, throughput=5MB/sec 2022-04-22T17:56:56.839 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-T8c1IoRq1O_rBg2 off=5442 len=1507973 2022-04-22T17:56:56.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 3069 completed, throughput=5MB/sec 2022-04-22T17:56:56.846 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3fRJbePT7DNXTZJ off=5770521 len=1227509 2022-04-22T17:56:56.848 INFO:tasks.workunit.client.0.smithi115.stdout:op 3070 completed, throughput=5MB/sec 2022-04-22T17:56:57.848 INFO:tasks.workunit.client.0.smithi115.stdout: 599: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:57.849 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ddhGptccC0sLpUS off=497212 len=1147990 2022-04-22T17:56:57.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 3071 completed, throughput=4.99MB/sec 2022-04-22T17:56:57.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MtQI739HSYhy97D off=5369713 len=848365 2022-04-22T17:56:57.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 3072 completed, throughput=5MB/sec 2022-04-22T17:56:57.860 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0jmx1xn5p__a4E0 off=3152993 len=472101 2022-04-22T17:56:57.861 INFO:tasks.workunit.client.0.smithi115.stdout:op 3073 completed, throughput=5MB/sec 2022-04-22T17:56:57.861 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-weyi1_BjaG5bG2A off=19221287 len=256548 2022-04-22T17:56:57.865 INFO:tasks.workunit.client.0.smithi115.stdout:op 3074 completed, throughput=5MB/sec 2022-04-22T17:56:57.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3W-mQ3vxoFHQowR off=5144654 len=73333 2022-04-22T17:56:57.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 3075 completed, throughput=5MB/sec 2022-04-22T17:56:57.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H95Ze78fRizAO_v off=3635819 len=1897319 2022-04-22T17:56:57.871 INFO:tasks.workunit.client.0.smithi115.stdout:op 3076 completed, throughput=5MB/sec 2022-04-22T17:56:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:57 smithi121 bash[36902]: cluster 2022-04-22T17:56:56.571232+0000 mgr.y (mgr.34107) 1105 : cluster [DBG] pgmap v1707: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:57 smithi121 bash[36902]: audit 2022-04-22T17:56:57.526581+0000 mon.a (mon.0) 2579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:57 smithi115 bash[45495]: cluster 2022-04-22T17:56:56.571232+0000 mgr.y (mgr.34107) 1105 : cluster [DBG] pgmap v1707: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:57 smithi115 bash[45495]: audit 2022-04-22T17:56:57.526581+0000 mon.a (mon.0) 2579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:57 smithi115 bash[46934]: cluster 2022-04-22T17:56:56.571232+0000 mgr.y (mgr.34107) 1105 : cluster [DBG] pgmap v1707: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:56:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:57 smithi115 bash[46934]: audit 2022-04-22T17:56:57.526581+0000 mon.a (mon.0) 2579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:56:58.871 INFO:tasks.workunit.client.0.smithi115.stdout: 600: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:58.871 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jHZu9aM9ejNvfyd off=1712267 len=1194757 2022-04-22T17:56:58.880 INFO:tasks.workunit.client.0.smithi115.stdout:op 3077 completed, throughput=4.99MB/sec 2022-04-22T17:56:58.880 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rwiWUu-JzLuUlcy off=22093836 len=587970 2022-04-22T17:56:58.881 INFO:tasks.workunit.client.0.smithi115.stdout:op 3078 completed, throughput=4.99MB/sec 2022-04-22T17:56:58.882 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I-h_4gxGMrXRXYx off=4186180 len=259880 2022-04-22T17:56:58.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 3079 completed, throughput=4.99MB/sec 2022-04-22T17:56:58.882 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zy97rkAhIcvuQp- off=7059620 len=868943 2022-04-22T17:56:58.884 INFO:tasks.workunit.client.0.smithi115.stdout:op 3080 completed, throughput=5MB/sec 2022-04-22T17:56:58.884 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cphu7-Tvwe47rue off=1484650 len=619649 2022-04-22T17:56:58.888 INFO:tasks.workunit.client.0.smithi115.stdout:op 3081 completed, throughput=5MB/sec 2022-04-22T17:56:58.888 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WXbirh8KqdoIo0h off=14918073 len=321033 2022-04-22T17:56:58.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 3082 completed, throughput=5MB/sec 2022-04-22T17:56:58.889 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x4H5tahkaHrSaaw off=6987972 len=959875 2022-04-22T17:56:58.894 INFO:tasks.workunit.client.0.smithi115.stdout:op 3083 completed, throughput=5MB/sec 2022-04-22T17:56:58.895 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ppZYv0R_OtxqKEW off=57197 len=230617 2022-04-22T17:56:58.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 3084 completed, throughput=5MB/sec 2022-04-22T17:56:58.896 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iWbHuPvxnuq0Rj7 off=1102975 len=739132 2022-04-22T17:56:58.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 3085 completed, throughput=5MB/sec 2022-04-22T17:56:59.899 INFO:tasks.workunit.client.0.smithi115.stdout: 601: throughput=4.99MB/sec pending data=0 2022-04-22T17:56:59.899 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DU1YUeIHoiU_ZEN off=4202518 len=1245230 2022-04-22T17:56:59.903 INFO:tasks.workunit.client.0.smithi115.stdout:op 3086 completed, throughput=4.99MB/sec 2022-04-22T17:56:59.903 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0qCqKlaT59m0j87 off=351898 len=1764307 2022-04-22T17:56:59.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 3087 completed, throughput=5MB/sec 2022-04-22T17:56:59.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y7nGNFqCkKJiMwp off=7671745 len=1167745 2022-04-22T17:56:59.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 3088 completed, throughput=5MB/sec 2022-04-22T17:56:59.910 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IKH0emS0GTa-2Rl off=7345662 len=1494557 2022-04-22T17:56:59.920 INFO:tasks.workunit.client.0.smithi115.stdout:op 3089 completed, throughput=5MB/sec 2022-04-22T17:57:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:56:59 smithi121 bash[36902]: cluster 2022-04-22T17:56:58.572382+0000 mgr.y (mgr.34107) 1106 : cluster [DBG] pgmap v1708: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:57:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:56:59 smithi115 bash[45495]: cluster 2022-04-22T17:56:58.572382+0000 mgr.y (mgr.34107) 1106 : cluster [DBG] pgmap v1708: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:57:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:56:59 smithi115 bash[46934]: cluster 2022-04-22T17:56:58.572382+0000 mgr.y (mgr.34107) 1106 : cluster [DBG] pgmap v1708: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T17:57:00.920 INFO:tasks.workunit.client.0.smithi115.stdout: 602: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:00.920 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OTkRcBnwhpvnyDe off=10775279 len=1741876 2022-04-22T17:57:00.924 INFO:tasks.workunit.client.0.smithi115.stdout:op 3090 completed, throughput=5MB/sec 2022-04-22T17:57:00.974 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C6LdihLmPdadJW8 off=23677732 len=580868 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 3091 completed, throughput=5MB/sec 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jPDo2Wz56PYIEfx off=13532020 len=1128359 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 3092 completed, throughput=5MB/sec 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fUhh7F9Ngbob0Pf off=3530272 len=124948 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 3093 completed, throughput=5MB/sec 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qNm-FWzLtxQK6Yf off=11541962 len=1905259 2022-04-22T17:57:00.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 3094 completed, throughput=5MB/sec 2022-04-22T17:57:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:00 smithi121 bash[36902]: audit 2022-04-22T17:57:00.142428+0000 mon.a (mon.0) 2580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:00 smithi121 bash[36902]: audit 2022-04-22T17:57:00.498097+0000 mon.a (mon.0) 2581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:00 smithi115 bash[45495]: audit 2022-04-22T17:57:00.142428+0000 mon.a (mon.0) 2580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:00 smithi115 bash[45495]: audit 2022-04-22T17:57:00.498097+0000 mon.a (mon.0) 2581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:00 smithi115 bash[46934]: audit 2022-04-22T17:57:00.142428+0000 mon.a (mon.0) 2580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:00 smithi115 bash[46934]: audit 2022-04-22T17:57:00.498097+0000 mon.a (mon.0) 2581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:01.948 INFO:tasks.workunit.client.0.smithi115.stdout: 603: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:01.948 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QMQ4OQk5G8Amh_X off=10333481 len=820566 2022-04-22T17:57:01.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 3095 completed, throughput=4.99MB/sec 2022-04-22T17:57:01.951 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jdlKH0DSUypszAI off=5686737 len=1957338 2022-04-22T17:57:01.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 3096 completed, throughput=5MB/sec 2022-04-22T17:57:01.959 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-46cYd7OkIM9IX53 off=19306898 len=581099 2022-04-22T17:57:01.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 3097 completed, throughput=5MB/sec 2022-04-22T17:57:01.965 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CaDV96acgwmriWZ off=13180968 len=1485223 2022-04-22T17:57:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 3098 completed, throughput=5MB/sec 2022-04-22T17:57:02.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:01 smithi121 bash[36902]: cluster 2022-04-22T17:57:00.573444+0000 mgr.y (mgr.34107) 1107 : cluster [DBG] pgmap v1709: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:57:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:01 smithi115 bash[45495]: cluster 2022-04-22T17:57:00.573444+0000 mgr.y (mgr.34107) 1107 : cluster [DBG] pgmap v1709: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:57:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:01.709Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:01 smithi115 bash[46934]: cluster 2022-04-22T17:57:00.573444+0000 mgr.y (mgr.34107) 1107 : cluster [DBG] pgmap v1709: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:57:02.972 INFO:tasks.workunit.client.0.smithi115.stdout: 604: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:02.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Evq4EvbeFvrmu63 off=2740923 len=137320 2022-04-22T17:57:02.973 INFO:tasks.workunit.client.0.smithi115.stdout:op 3099 completed, throughput=4.99MB/sec 2022-04-22T17:57:02.973 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YpXF5r3lpF031_k off=1465362 len=1612094 2022-04-22T17:57:02.977 INFO:tasks.workunit.client.0.smithi115.stdout:op 3100 completed, throughput=5MB/sec 2022-04-22T17:57:02.977 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L22L5y8CQfY68q7 off=3906455 len=1342438 2022-04-22T17:57:02.986 INFO:tasks.workunit.client.0.smithi115.stdout:op 3101 completed, throughput=5MB/sec 2022-04-22T17:57:02.986 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WoeULpnl2CX_FAI off=2113341 len=721630 2022-04-22T17:57:02.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 3102 completed, throughput=5MB/sec 2022-04-22T17:57:02.991 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KRtBnb56ycSYu2P off=2462014 len=1996031 2022-04-22T17:57:02.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 3103 completed, throughput=5MB/sec 2022-04-22T17:57:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:02 smithi115 bash[45495]: audit 2022-04-22T17:57:02.529928+0000 mon.a (mon.0) 2582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:02 smithi115 bash[46934]: audit 2022-04-22T17:57:02.529928+0000 mon.a (mon.0) 2582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:02 smithi121 bash[36902]: audit 2022-04-22T17:57:02.529928+0000 mon.a (mon.0) 2582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:03.997 INFO:tasks.workunit.client.0.smithi115.stdout: 605: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:03.997 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8l_SQbewcmPfurX off=2564675 len=2044357 2022-04-22T17:57:04.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 3104 completed, throughput=5MB/sec 2022-04-22T17:57:04.003 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1nbRwAFOYdrbkII off=10605654 len=1367812 2022-04-22T17:57:04.011 INFO:tasks.workunit.client.0.smithi115.stdout:op 3105 completed, throughput=5MB/sec 2022-04-22T17:57:04.012 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-18OqTmjsswprg8P off=7090222 len=1124064 2022-04-22T17:57:04.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 3106 completed, throughput=5MB/sec 2022-04-22T17:57:04.014 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lu83tDS8N94e05K off=7916307 len=361564 2022-04-22T17:57:04.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 3107 completed, throughput=5MB/sec 2022-04-22T17:57:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:03 smithi115 bash[45495]: cluster 2022-04-22T17:57:02.574551+0000 mgr.y (mgr.34107) 1108 : cluster [DBG] pgmap v1710: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:57:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:03.591Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:03 smithi115 bash[46934]: cluster 2022-04-22T17:57:02.574551+0000 mgr.y (mgr.34107) 1108 : cluster [DBG] pgmap v1710: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:57:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:03 smithi121 bash[36902]: cluster 2022-04-22T17:57:02.574551+0000 mgr.y (mgr.34107) 1108 : cluster [DBG] pgmap v1710: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:57:05.019 INFO:tasks.workunit.client.0.smithi115.stdout: 606: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:05.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xlMt0PBSByK1DJH off=3566098 len=1387982 2022-04-22T17:57:05.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 3108 completed, throughput=4.99MB/sec 2022-04-22T17:57:05.024 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4H4LqKnl2QEGkOC off=12242152 len=13981 2022-04-22T17:57:05.025 INFO:tasks.workunit.client.0.smithi115.stdout:op 3109 completed, throughput=4.99MB/sec 2022-04-22T17:57:05.025 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ApaQmd6gCMJb1rS off=13790623 len=625489 2022-04-22T17:57:05.031 INFO:tasks.workunit.client.0.smithi115.stdout:op 3110 completed, throughput=5MB/sec 2022-04-22T17:57:05.031 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cEbmFgE7nmR1eG7 off=9099837 len=179322 2022-04-22T17:57:05.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 3111 completed, throughput=5MB/sec 2022-04-22T17:57:05.034 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JjATAbpeBpBs5PO off=94833 len=941355 2022-04-22T17:57:05.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 3112 completed, throughput=5MB/sec 2022-04-22T17:57:05.041 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KinB2J_dy2p_Lpf off=11029017 len=1476487 2022-04-22T17:57:05.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 3113 completed, throughput=5MB/sec 2022-04-22T17:57:05.049 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zAQSEyMqfLHPdAX off=2118343 len=317984 2022-04-22T17:57:05.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 3114 completed, throughput=5MB/sec 2022-04-22T17:57:05.053 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LakZHrT-LxYm27V off=4461437 len=1833242 2022-04-22T17:57:05.055 INFO:tasks.workunit.client.0.smithi115.stdout:op 3115 completed, throughput=5MB/sec 2022-04-22T17:57:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[45495]: cluster 2022-04-22T17:57:04.575628+0000 mgr.y (mgr.34107) 1109 : cluster [DBG] pgmap v1711: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:57:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[45495]: audit 2022-04-22T17:57:05.145474+0000 mon.a (mon.0) 2583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[45495]: audit 2022-04-22T17:57:05.501688+0000 mon.a (mon.0) 2584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[46934]: cluster 2022-04-22T17:57:04.575628+0000 mgr.y (mgr.34107) 1109 : cluster [DBG] pgmap v1711: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:57:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[46934]: audit 2022-04-22T17:57:05.145474+0000 mon.a (mon.0) 2583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:05 smithi115 bash[46934]: audit 2022-04-22T17:57:05.501688+0000 mon.a (mon.0) 2584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:06.055 INFO:tasks.workunit.client.0.smithi115.stdout: 607: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:06.055 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OAStGVwvGvhJDZb off=1493599 len=1809200 2022-04-22T17:57:06.067 INFO:tasks.workunit.client.0.smithi115.stdout:op 3116 completed, throughput=5MB/sec 2022-04-22T17:57:06.067 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MPz6cdLg6_BfGko off=3124038 len=410988 2022-04-22T17:57:06.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 3117 completed, throughput=5MB/sec 2022-04-22T17:57:06.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wjIOh9s-_Zc3s7z off=6169351 len=1693674 2022-04-22T17:57:06.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 3118 completed, throughput=5MB/sec 2022-04-22T17:57:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:05 smithi121 bash[36902]: cluster 2022-04-22T17:57:04.575628+0000 mgr.y (mgr.34107) 1109 : cluster [DBG] pgmap v1711: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:57:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:05 smithi121 bash[36902]: audit 2022-04-22T17:57:05.145474+0000 mon.a (mon.0) 2583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:05 smithi121 bash[36902]: audit 2022-04-22T17:57:05.501688+0000 mon.a (mon.0) 2584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:07.072 INFO:tasks.workunit.client.0.smithi115.stdout: 608: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:07.297 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s-U_uEBGpmHjLCi off=7366964 len=1818969 2022-04-22T17:57:07.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 3119 completed, throughput=4.99MB/sec 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2WWYy-ElNCSmK0x off=5790564 len=1976076 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 3120 completed, throughput=5MB/sec 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w6gwuYmdF_n_KKK off=11648589 len=1192977 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 3121 completed, throughput=5MB/sec 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6rPYHUiVU8yKKM0 off=6019589 len=1641682 2022-04-22T17:57:07.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 3122 completed, throughput=5MB/sec 2022-04-22T17:57:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:07 smithi115 bash[45495]: cluster 2022-04-22T17:57:06.576784+0000 mgr.y (mgr.34107) 1110 : cluster [DBG] pgmap v1712: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:57:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:07 smithi115 bash[45495]: audit 2022-04-22T17:57:07.532984+0000 mon.a (mon.0) 2585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:07 smithi115 bash[46934]: cluster 2022-04-22T17:57:06.576784+0000 mgr.y (mgr.34107) 1110 : cluster [DBG] pgmap v1712: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:57:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:07 smithi115 bash[46934]: audit 2022-04-22T17:57:07.532984+0000 mon.a (mon.0) 2585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:08.103 INFO:tasks.workunit.client.0.smithi115.stdout: 609: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:08.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c3z5_fgD4dimHGE off=1455617 len=1325949 2022-04-22T17:57:08.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 3123 completed, throughput=5MB/sec 2022-04-22T17:57:08.107 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kQhSp9eEA_WAd5O off=7532894 len=180626 2022-04-22T17:57:08.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 3124 completed, throughput=5MB/sec 2022-04-22T17:57:08.109 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KJJNU3apXYhLzu9 off=2367961 len=250166 2022-04-22T17:57:08.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 3125 completed, throughput=5MB/sec 2022-04-22T17:57:08.115 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xtoIqo3vc1YE50z off=2626315 len=1848636 2022-04-22T17:57:08.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 3126 completed, throughput=5MB/sec 2022-04-22T17:57:08.117 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XYukezWnMRgLgwJ off=14695055 len=1381693 2022-04-22T17:57:08.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 3127 completed, throughput=5MB/sec 2022-04-22T17:57:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:07 smithi121 bash[36902]: cluster 2022-04-22T17:57:06.576784+0000 mgr.y (mgr.34107) 1110 : cluster [DBG] pgmap v1712: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:57:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:07 smithi121 bash[36902]: audit 2022-04-22T17:57:07.532984+0000 mon.a (mon.0) 2585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:09.124 INFO:tasks.workunit.client.0.smithi115.stdout: 610: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:09.125 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pwfqN6caO-ruwFC off=2234654 len=204801 2022-04-22T17:57:09.126 INFO:tasks.workunit.client.0.smithi115.stdout:op 3128 completed, throughput=4.99MB/sec 2022-04-22T17:57:09.126 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nXLVSJ15JKHho-B off=5592526 len=97771 2022-04-22T17:57:09.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 3129 completed, throughput=4.99MB/sec 2022-04-22T17:57:09.130 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cQo8jJUNqvayRJ6 off=2643308 len=353536 2022-04-22T17:57:09.131 INFO:tasks.workunit.client.0.smithi115.stdout:op 3130 completed, throughput=4.99MB/sec 2022-04-22T17:57:09.131 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q6_7Cw4gfKxMlfL off=9337044 len=1786217 2022-04-22T17:57:09.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 3131 completed, throughput=5MB/sec 2022-04-22T17:57:09.135 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tQqinVX_ky_Wztr off=6373557 len=1113100 2022-04-22T17:57:09.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 3132 completed, throughput=5MB/sec 2022-04-22T17:57:09.145 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sn6varEbT310TX0 off=0 len=1256499 2022-04-22T17:57:09.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 3133 completed, throughput=5MB/sec 2022-04-22T17:57:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:09 smithi115 bash[45495]: cluster 2022-04-22T17:57:08.578117+0000 mgr.y (mgr.34107) 1111 : cluster [DBG] pgmap v1713: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:57:10.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:09 smithi115 bash[46934]: cluster 2022-04-22T17:57:08.578117+0000 mgr.y (mgr.34107) 1111 : cluster [DBG] pgmap v1713: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:57:10.149 INFO:tasks.workunit.client.0.smithi115.stdout: 611: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:10.149 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_f-CsnnkhmdBDnr off=93944 len=247838 2022-04-22T17:57:10.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 3134 completed, throughput=4.99MB/sec 2022-04-22T17:57:10.151 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-acXMxkbuyEaRIJm off=1755743 len=549692 2022-04-22T17:57:10.157 INFO:tasks.workunit.client.0.smithi115.stdout:op 3135 completed, throughput=4.99MB/sec 2022-04-22T17:57:10.157 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Yh9IGpbwstfRWrJ off=1262199 len=1518719 2022-04-22T17:57:10.159 INFO:tasks.workunit.client.0.smithi115.stdout:op 3136 completed, throughput=5MB/sec 2022-04-22T17:57:10.159 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lIXrf5JR5GIbZ59 off=16731421 len=824681 2022-04-22T17:57:10.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 3137 completed, throughput=5MB/sec 2022-04-22T17:57:10.165 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MlJcjk1NCD1_dxi off=3081905 len=492230 2022-04-22T17:57:10.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 3138 completed, throughput=5MB/sec 2022-04-22T17:57:10.168 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fOAg1h8TKvJPHgQ off=17850621 len=353614 2022-04-22T17:57:10.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 3139 completed, throughput=5MB/sec 2022-04-22T17:57:10.169 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UVL95XlJn-O5l1l off=9609845 len=1872130 2022-04-22T17:57:10.179 INFO:tasks.workunit.client.0.smithi115.stdout:op 3140 completed, throughput=5MB/sec 2022-04-22T17:57:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:09 smithi121 bash[36902]: cluster 2022-04-22T17:57:08.578117+0000 mgr.y (mgr.34107) 1111 : cluster [DBG] pgmap v1713: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:57:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:10 smithi115 bash[45495]: audit 2022-04-22T17:57:10.148887+0000 mon.a (mon.0) 2586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:10 smithi115 bash[45495]: audit 2022-04-22T17:57:10.505394+0000 mon.a (mon.0) 2587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:10 smithi115 bash[46934]: audit 2022-04-22T17:57:10.148887+0000 mon.a (mon.0) 2586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:10 smithi115 bash[46934]: audit 2022-04-22T17:57:10.505394+0000 mon.a (mon.0) 2587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:11.179 INFO:tasks.workunit.client.0.smithi115.stdout: 612: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:11.180 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-imHsoF2_XiLKVVi off=5932902 len=1843669 2022-04-22T17:57:11.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 3141 completed, throughput=5MB/sec 2022-04-22T17:57:11.190 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RTK2L-6_wB2rQus off=4089459 len=1886747 2022-04-22T17:57:11.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 3142 completed, throughput=5MB/sec 2022-04-22T17:57:11.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZYhA_GAjJVkooeP off=3176914 len=2065409 2022-04-22T17:57:11.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 3143 completed, throughput=5MB/sec 2022-04-22T17:57:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:10 smithi121 bash[36902]: audit 2022-04-22T17:57:10.148887+0000 mon.a (mon.0) 2586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:10 smithi121 bash[36902]: audit 2022-04-22T17:57:10.505394+0000 mon.a (mon.0) 2587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:11.711Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:11 smithi115 bash[45495]: cluster 2022-04-22T17:57:10.579281+0000 mgr.y (mgr.34107) 1112 : cluster [DBG] pgmap v1714: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:11 smithi115 bash[46934]: cluster 2022-04-22T17:57:10.579281+0000 mgr.y (mgr.34107) 1112 : cluster [DBG] pgmap v1714: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:12.210 INFO:tasks.workunit.client.0.smithi115.stdout: 613: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:12.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zYECI-41S9SZZvj off=1785195 len=121571 2022-04-22T17:57:12.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 3144 completed, throughput=4.99MB/sec 2022-04-22T17:57:12.215 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tbyXV3bcO_0a7XK off=1199005 len=2025143 2022-04-22T17:57:12.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 3145 completed, throughput=5MB/sec 2022-04-22T17:57:12.225 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xDT6-xoU9gXDRAR off=7424870 len=1439575 2022-04-22T17:57:12.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 3146 completed, throughput=5MB/sec 2022-04-22T17:57:12.228 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I6idwQ-ABNolfbY off=1235394 len=521981 2022-04-22T17:57:12.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 3147 completed, throughput=5MB/sec 2022-04-22T17:57:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:11 smithi121 bash[36902]: cluster 2022-04-22T17:57:10.579281+0000 mgr.y (mgr.34107) 1112 : cluster [DBG] pgmap v1714: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:13.230 INFO:tasks.workunit.client.0.smithi115.stdout: 614: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:13.230 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vdYRGpmrbphxDL6 off=5547360 len=874916 2022-04-22T17:57:13.237 INFO:tasks.workunit.client.0.smithi115.stdout:op 3148 completed, throughput=4.99MB/sec 2022-04-22T17:57:13.237 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gH_o-chiYpyF4G2 off=705640 len=123463 2022-04-22T17:57:13.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 3149 completed, throughput=4.99MB/sec 2022-04-22T17:57:13.239 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TC0YPTTXrzHsPZh off=7880402 len=1401083 2022-04-22T17:57:13.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 3150 completed, throughput=5MB/sec 2022-04-22T17:57:13.246 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1G7fR8nfNAQEKjO off=11040907 len=842722 2022-04-22T17:57:13.248 INFO:tasks.workunit.client.0.smithi115.stdout:op 3151 completed, throughput=5MB/sec 2022-04-22T17:57:13.248 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-V4vJsOIQSZL7eHo off=2650737 len=814353 2022-04-22T17:57:13.254 INFO:tasks.workunit.client.0.smithi115.stdout:op 3152 completed, throughput=5MB/sec 2022-04-22T17:57:13.254 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HLcHgD6hY2Hbzag off=2467790 len=1297496 2022-04-22T17:57:13.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 3153 completed, throughput=5MB/sec 2022-04-22T17:57:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:12 smithi121 bash[36902]: audit 2022-04-22T17:57:12.536042+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:12 smithi115 bash[45495]: audit 2022-04-22T17:57:12.536042+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:12 smithi115 bash[46934]: audit 2022-04-22T17:57:12.536042+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:13.593Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[46934]: cluster 2022-04-22T17:57:12.580382+0000 mgr.y (mgr.34107) 1113 : cluster [DBG] pgmap v1715: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[46934]: audit 2022-04-22T17:57:13.554086+0000 mon.a (mon.0) 2589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:57:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[46934]: audit 2022-04-22T17:57:13.554704+0000 mon.a (mon.0) 2590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:57:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[45495]: cluster 2022-04-22T17:57:12.580382+0000 mgr.y (mgr.34107) 1113 : cluster [DBG] pgmap v1715: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[45495]: audit 2022-04-22T17:57:13.554086+0000 mon.a (mon.0) 2589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:57:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:13 smithi115 bash[45495]: audit 2022-04-22T17:57:13.554704+0000 mon.a (mon.0) 2590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:57:14.256 INFO:tasks.workunit.client.0.smithi115.stdout: 615: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JYbhgoGhfYf1FQO off=1324185 len=1942052 2022-04-22T17:57:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 3154 completed, throughput=5MB/sec 2022-04-22T17:57:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cUk-dHL_gOQvLhD off=13119630 len=1989984 2022-04-22T17:57:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 3155 completed, throughput=5MB/sec 2022-04-22T17:57:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rJWVGzIyq3HZn50 off=1728074 len=1239099 2022-04-22T17:57:14.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 3156 completed, throughput=5MB/sec 2022-04-22T17:57:14.265 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GyQOZwtw3X6Jr7N off=13801471 len=1398130 2022-04-22T17:57:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:13 smithi121 bash[36902]: cluster 2022-04-22T17:57:12.580382+0000 mgr.y (mgr.34107) 1113 : cluster [DBG] pgmap v1715: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:57:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:13 smithi121 bash[36902]: audit 2022-04-22T17:57:13.554086+0000 mon.a (mon.0) 2589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:57:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:13 smithi121 bash[36902]: audit 2022-04-22T17:57:13.554704+0000 mon.a (mon.0) 2590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:57:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:57:14.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3157 completed, throughput=5MB/sec 2022-04-22T17:57:14.617 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:57:14.617 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (20m) 9m ago 34m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (33m) 29s ago 33m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (22m) 9m ago 35m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (21m) 29s ago 35m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (29m) 29s ago 39m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (30m) 9m ago 41m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (29m) 9m ago 41m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (28m) 29s ago 39m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (28m) 9m ago 39m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:57:14.618 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (21m) 9m ago 34m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (21m) 29s ago 34m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (27m) 9m ago 38m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (26m) 9m ago 38m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (25m) 9m ago 37m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (25m) 9m ago 37m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (24m) 29s ago 37m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (24m) 29s ago 36m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (23m) 29s ago 36m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (22m) 29s ago 35m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:57:14.619 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (21m) 29s ago 34m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:57:15.018 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:57:15.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:57:15.219 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:57:15.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:57:15.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:15.221 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:57:15.221 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:57:15.221 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:57:15.221 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:57:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:14 smithi121 bash[36902]: audit 2022-04-22T17:57:14.247092+0000 mgr.y (mgr.34107) 1114 : audit [DBG] from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:15.272 INFO:tasks.workunit.client.0.smithi115.stdout: 616: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:15.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AIu2jYJkrOvtLnN off=1067767 len=653613 2022-04-22T17:57:15.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 3158 completed, throughput=5MB/sec 2022-04-22T17:57:15.279 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-euF5juItcRPBMEa off=2959877 len=2037311 2022-04-22T17:57:15.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 3159 completed, throughput=5MB/sec 2022-04-22T17:57:15.283 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BkbgOjUJ5elnuOd off=10234989 len=1924442 2022-04-22T17:57:15.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 3160 completed, throughput=5MB/sec 2022-04-22T17:57:15.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:14 smithi115 bash[45495]: audit 2022-04-22T17:57:14.247092+0000 mgr.y (mgr.34107) 1114 : audit [DBG] from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:15.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:14 smithi115 bash[46934]: audit 2022-04-22T17:57:14.247092+0000 mgr.y (mgr.34107) 1114 : audit [DBG] from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:15 smithi121 bash[36902]: cluster 2022-04-22T17:57:14.581399+0000 mgr.y (mgr.34107) 1115 : cluster [DBG] pgmap v1716: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:57:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:15 smithi121 bash[36902]: audit 2022-04-22T17:57:14.592508+0000 mgr.y (mgr.34107) 1116 : audit [DBG] from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:15 smithi121 bash[36902]: audit 2022-04-22T17:57:15.011351+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.115:0/4052449990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:15 smithi121 bash[36902]: audit 2022-04-22T17:57:15.152293+0000 mon.a (mon.0) 2591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:15 smithi121 bash[36902]: audit 2022-04-22T17:57:15.509200+0000 mon.a (mon.0) 2592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:16.287 INFO:tasks.workunit.client.0.smithi115.stdout: 617: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:16.287 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_2hWngAxt8s7RMA off=12440144 len=2071444 2022-04-22T17:57:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[45495]: cluster 2022-04-22T17:57:14.581399+0000 mgr.y (mgr.34107) 1115 : cluster [DBG] pgmap v1716: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:57:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[45495]: audit 2022-04-22T17:57:14.592508+0000 mgr.y (mgr.34107) 1116 : audit [DBG] from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[45495]: audit 2022-04-22T17:57:15.011351+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.115:0/4052449990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[45495]: audit 2022-04-22T17:57:15.152293+0000 mon.a (mon.0) 2591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[45495]: audit 2022-04-22T17:57:15.509200+0000 mon.a (mon.0) 2592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[46934]: cluster 2022-04-22T17:57:14.581399+0000 mgr.y (mgr.34107) 1115 : cluster [DBG] pgmap v1716: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:57:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[46934]: audit 2022-04-22T17:57:14.592508+0000 mgr.y (mgr.34107) 1116 : audit [DBG] from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[46934]: audit 2022-04-22T17:57:15.011351+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.115:0/4052449990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[46934]: audit 2022-04-22T17:57:15.152293+0000 mon.a (mon.0) 2591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:15 smithi115 bash[46934]: audit 2022-04-22T17:57:15.509200+0000 mon.a (mon.0) 2592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:16.298 INFO:tasks.workunit.client.0.smithi115.stdout:op 3161 completed, throughput=5MB/sec 2022-04-22T17:57:16.299 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DGee1mCXzzeoZAY off=1355711 len=1619165 2022-04-22T17:57:16.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 3162 completed, throughput=5MB/sec 2022-04-22T17:57:16.307 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kky0xU4CLJcxPQh off=20105683 len=1394776 2022-04-22T17:57:16.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 3163 completed, throughput=5MB/sec 2022-04-22T17:57:17.313 INFO:tasks.workunit.client.0.smithi115.stdout: 618: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:17.314 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9-tPfBc8lWtg7Cq off=3272136 len=830530 2022-04-22T17:57:17.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 3164 completed, throughput=4.99MB/sec 2022-04-22T17:57:17.316 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kxC1SFwE-AvTdkr off=11427536 len=1943268 2022-04-22T17:57:17.322 INFO:tasks.workunit.client.0.smithi115.stdout:op 3165 completed, throughput=5MB/sec 2022-04-22T17:57:17.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jPooiDi2Y3lBi9l off=1230406 len=294002 2022-04-22T17:57:17.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 3166 completed, throughput=5MB/sec 2022-04-22T17:57:17.326 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aJHsAMRBXgPYXFM off=7339453 len=1174057 2022-04-22T17:57:17.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 3167 completed, throughput=5MB/sec 2022-04-22T17:57:17.329 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tgYwgm1Ly55kC0A off=1049343 len=2047385 2022-04-22T17:57:17.339 INFO:tasks.workunit.client.0.smithi115.stdout:op 3168 completed, throughput=5MB/sec 2022-04-22T17:57:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:17 smithi121 bash[36902]: cluster 2022-04-22T17:57:16.582495+0000 mgr.y (mgr.34107) 1117 : cluster [DBG] pgmap v1717: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:18.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:17 smithi121 bash[36902]: audit 2022-04-22T17:57:17.539241+0000 mon.a (mon.0) 2593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:17 smithi115 bash[45495]: cluster 2022-04-22T17:57:16.582495+0000 mgr.y (mgr.34107) 1117 : cluster [DBG] pgmap v1717: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:17 smithi115 bash[45495]: audit 2022-04-22T17:57:17.539241+0000 mon.a (mon.0) 2593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:17 smithi115 bash[46934]: cluster 2022-04-22T17:57:16.582495+0000 mgr.y (mgr.34107) 1117 : cluster [DBG] pgmap v1717: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:17 smithi115 bash[46934]: audit 2022-04-22T17:57:17.539241+0000 mon.a (mon.0) 2593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:18.339 INFO:tasks.workunit.client.0.smithi115.stdout: 619: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:18.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KQ0-5ofgOtzXQGC off=10394647 len=1299422 2022-04-22T17:57:18.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 3169 completed, throughput=5MB/sec 2022-04-22T17:57:18.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7HW0OhjvlULLzUl off=2196760 len=463384 2022-04-22T17:57:18.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 3170 completed, throughput=5MB/sec 2022-04-22T17:57:18.349 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hl_UCyJDSrByIe3 off=3415842 len=1400682 2022-04-22T17:57:18.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 3171 completed, throughput=5MB/sec 2022-04-22T17:57:18.352 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i2PwZT1_5ZNp6IV off=778289 len=2008217 2022-04-22T17:57:18.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 3172 completed, throughput=5MB/sec 2022-04-22T17:57:19.360 INFO:tasks.workunit.client.0.smithi115.stdout: 620: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:19.361 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nwuut-kVJhHrjnt off=2075158 len=1351886 2022-04-22T17:57:19.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 3173 completed, throughput=5MB/sec 2022-04-22T17:57:19.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eOue1WnSCu6dKQz off=10761919 len=324310 2022-04-22T17:57:19.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 3174 completed, throughput=5MB/sec 2022-04-22T17:57:19.369 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MRUHJ0l7zv0Oz1c off=4983342 len=1022809 2022-04-22T17:57:19.377 INFO:tasks.workunit.client.0.smithi115.stdout:op 3175 completed, throughput=5MB/sec 2022-04-22T17:57:19.378 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WWhL4YoEZ_MJCS6 off=3866224 len=1435633 2022-04-22T17:57:19.380 INFO:tasks.workunit.client.0.smithi115.stdout:op 3176 completed, throughput=5MB/sec 2022-04-22T17:57:19.380 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0LmTBCrAS046tAW off=4385066 len=1788166 2022-04-22T17:57:19.383 INFO:tasks.workunit.client.0.smithi115.stdout:op 3177 completed, throughput=5MB/sec 2022-04-22T17:57:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:19 smithi121 bash[36902]: cluster 2022-04-22T17:57:18.583722+0000 mgr.y (mgr.34107) 1118 : cluster [DBG] pgmap v1718: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:19 smithi115 bash[45495]: cluster 2022-04-22T17:57:18.583722+0000 mgr.y (mgr.34107) 1118 : cluster [DBG] pgmap v1718: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:20.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:19 smithi115 bash[46934]: cluster 2022-04-22T17:57:18.583722+0000 mgr.y (mgr.34107) 1118 : cluster [DBG] pgmap v1718: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:20.383 INFO:tasks.workunit.client.0.smithi115.stdout: 621: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:20.383 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fANowjc_RldpHHl off=8943861 len=1178529 2022-04-22T17:57:20.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 3178 completed, throughput=5MB/sec 2022-04-22T17:57:20.391 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UjqUZ23bXG22erq off=2091812 len=326394 2022-04-22T17:57:20.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 3179 completed, throughput=5MB/sec 2022-04-22T17:57:20.398 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gpptwXUcwGjo-22 off=13312239 len=433492 2022-04-22T17:57:20.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 3180 completed, throughput=5MB/sec 2022-04-22T17:57:20.400 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4zPa1_WDl_F05Zf off=3392537 len=574999 2022-04-22T17:57:20.401 INFO:tasks.workunit.client.0.smithi115.stdout:op 3181 completed, throughput=5MB/sec 2022-04-22T17:57:20.401 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-54p_VC2lRVHjYJC off=7112173 len=1125256 2022-04-22T17:57:20.409 INFO:tasks.workunit.client.0.smithi115.stdout:op 3182 completed, throughput=5MB/sec 2022-04-22T17:57:20.409 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R2Bn4zg2Jrxs1Fv off=18043578 len=732945 2022-04-22T17:57:20.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 3183 completed, throughput=5MB/sec 2022-04-22T17:57:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:20 smithi121 bash[36902]: audit 2022-04-22T17:57:20.155254+0000 mon.a (mon.0) 2594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:20 smithi121 bash[36902]: audit 2022-04-22T17:57:20.512872+0000 mon.a (mon.0) 2595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:20 smithi115 bash[45495]: audit 2022-04-22T17:57:20.155254+0000 mon.a (mon.0) 2594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:20 smithi115 bash[45495]: audit 2022-04-22T17:57:20.512872+0000 mon.a (mon.0) 2595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:20 smithi115 bash[46934]: audit 2022-04-22T17:57:20.155254+0000 mon.a (mon.0) 2594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:20 smithi115 bash[46934]: audit 2022-04-22T17:57:20.512872+0000 mon.a (mon.0) 2595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:21.414 INFO:tasks.workunit.client.0.smithi115.stdout: 622: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:21.414 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DZCXjJk8fxnTt_v off=1739915 len=1635657 2022-04-22T17:57:21.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 3184 completed, throughput=5MB/sec 2022-04-22T17:57:21.423 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5-nxwyhJRixzSLG off=10398899 len=162174 2022-04-22T17:57:21.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 3185 completed, throughput=5MB/sec 2022-04-22T17:57:21.427 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-byrdmQ_HL_uoFUz off=12217133 len=1803521 2022-04-22T17:57:21.435 INFO:tasks.workunit.client.0.smithi115.stdout:op 3186 completed, throughput=5MB/sec 2022-04-22T17:57:21.435 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L-oO8-VUsvMC1d- off=2598082 len=1266140 2022-04-22T17:57:21.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 3187 completed, throughput=5MB/sec 2022-04-22T17:57:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:21 smithi115 bash[45495]: cluster 2022-04-22T17:57:20.584785+0000 mgr.y (mgr.34107) 1119 : cluster [DBG] pgmap v1719: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:22.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:21 smithi115 bash[46934]: cluster 2022-04-22T17:57:20.584785+0000 mgr.y (mgr.34107) 1119 : cluster [DBG] pgmap v1719: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:21.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:21 smithi121 bash[36902]: cluster 2022-04-22T17:57:20.584785+0000 mgr.y (mgr.34107) 1119 : cluster [DBG] pgmap v1719: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout: 623: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xLTeBCkW0-r_UdH off=2590652 len=271582 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 3188 completed, throughput=4.99MB/sec 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Fb37arCd2injSYC off=10068541 len=353585 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 3189 completed, throughput=4.99MB/sec 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gjleHvpy-n715SP off=4329174 len=189220 2022-04-22T17:57:22.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 3190 completed, throughput=4.99MB/sec 2022-04-22T17:57:22.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mawT0cWGUuOao4k off=9296551 len=1458745 2022-04-22T17:57:22.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 3191 completed, throughput=5MB/sec 2022-04-22T17:57:22.507 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ChiHPMGC8Id3ZLO off=1854894 len=1588218 2022-04-22T17:57:22.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 3192 completed, throughput=5MB/sec 2022-04-22T17:57:22.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6QJ-zMF8TEOpxAN off=7297980 len=1821139 2022-04-22T17:57:22.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 3193 completed, throughput=5MB/sec 2022-04-22T17:57:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:22 smithi121 bash[36902]: audit 2022-04-22T17:57:22.542320+0000 mon.a (mon.0) 2596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:22 smithi115 bash[45495]: audit 2022-04-22T17:57:22.542320+0000 mon.a (mon.0) 2596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:22 smithi115 bash[46934]: audit 2022-04-22T17:57:22.542320+0000 mon.a (mon.0) 2596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:23.470 INFO:tasks.workunit.client.0.smithi115.stdout: 624: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:23.470 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0fZVzIEEu35jem9 off=61011 len=634404 2022-04-22T17:57:23.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 3194 completed, throughput=4.99MB/sec 2022-04-22T17:57:23.472 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gXfa2F77TUCd_cR off=7526131 len=1317917 2022-04-22T17:57:23.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 3195 completed, throughput=5MB/sec 2022-04-22T17:57:23.476 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pP5XVKjI6h9uTlI off=10622024 len=21043 2022-04-22T17:57:23.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 3196 completed, throughput=5MB/sec 2022-04-22T17:57:23.481 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jqgp_yqiUjv4U6S off=8965145 len=137705 2022-04-22T17:57:23.482 INFO:tasks.workunit.client.0.smithi115.stdout:op 3197 completed, throughput=5MB/sec 2022-04-22T17:57:23.482 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oW4VO5uN48wC5Tm off=1510703 len=309071 2022-04-22T17:57:23.483 INFO:tasks.workunit.client.0.smithi115.stdout:op 3198 completed, throughput=5MB/sec 2022-04-22T17:57:23.483 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8vBETcy70IS-ClC off=359152 len=532979 2022-04-22T17:57:23.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 3199 completed, throughput=5MB/sec 2022-04-22T17:57:23.489 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jz9dvki314R2nHP off=874571 len=267073 2022-04-22T17:57:23.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 3200 completed, throughput=5MB/sec 2022-04-22T17:57:23.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R7i4ehMbX6Dziqf off=15265203 len=1807153 2022-04-22T17:57:23.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 3201 completed, throughput=5MB/sec 2022-04-22T17:57:23.498 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Bf5TEgx6ZruEUmz off=1802221 len=572032 2022-04-22T17:57:23.499 INFO:tasks.workunit.client.0.smithi115.stdout:op 3202 completed, throughput=5MB/sec 2022-04-22T17:57:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:23 smithi115 bash[45495]: cluster 2022-04-22T17:57:22.585921+0000 mgr.y (mgr.34107) 1120 : cluster [DBG] pgmap v1720: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:23.593Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:23 smithi115 bash[46934]: cluster 2022-04-22T17:57:22.585921+0000 mgr.y (mgr.34107) 1120 : cluster [DBG] pgmap v1720: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:23 smithi121 bash[36902]: cluster 2022-04-22T17:57:22.585921+0000 mgr.y (mgr.34107) 1120 : cluster [DBG] pgmap v1720: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:24.500 INFO:tasks.workunit.client.0.smithi115.stdout: 625: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:24.500 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QDw1rgN6RW_xiaK off=1737533 len=631791 2022-04-22T17:57:24.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 3203 completed, throughput=4.99MB/sec 2022-04-22T17:57:24.506 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3GOATesQLl0sKH7 off=8903797 len=1080302 2022-04-22T17:57:24.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 3204 completed, throughput=5MB/sec 2022-04-22T17:57:24.510 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l7DWIlCzMDwsaFA off=1621951 len=1815324 2022-04-22T17:57:24.519 INFO:tasks.workunit.client.0.smithi115.stdout:op 3205 completed, throughput=5MB/sec 2022-04-22T17:57:24.519 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pKFm64IOQkGFMU5 off=860320 len=650375 2022-04-22T17:57:24.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 3206 completed, throughput=5MB/sec 2022-04-22T17:57:24.524 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-a-FkD2XG2otikXs off=9868047 len=1502915 2022-04-22T17:57:24.527 INFO:tasks.workunit.client.0.smithi115.stdout:op 3207 completed, throughput=5MB/sec 2022-04-22T17:57:25.527 INFO:tasks.workunit.client.0.smithi115.stdout: 626: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:25.527 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-By366x7Byd_QFAU off=449886 len=680448 2022-04-22T17:57:25.529 INFO:tasks.workunit.client.0.smithi115.stdout:op 3208 completed, throughput=4.99MB/sec 2022-04-22T17:57:25.530 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--8dyLjTkoa3RpKy off=3653919 len=1436873 2022-04-22T17:57:25.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 3209 completed, throughput=5MB/sec 2022-04-22T17:57:25.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ejKrX8RuyKwW6tf off=1580342 len=243650 2022-04-22T17:57:25.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 3210 completed, throughput=5MB/sec 2022-04-22T17:57:25.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yC3y3u4cxeV0-g0 off=15954478 len=2075801 2022-04-22T17:57:25.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 3211 completed, throughput=5MB/sec 2022-04-22T17:57:25.549 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eh0qcIVaID6Y86I off=14734585 len=1966771 2022-04-22T17:57:25.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 3212 completed, throughput=5MB/sec 2022-04-22T17:57:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:25 smithi121 bash[36902]: cluster 2022-04-22T17:57:24.587145+0000 mgr.y (mgr.34107) 1121 : cluster [DBG] pgmap v1721: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:25 smithi121 bash[36902]: audit 2022-04-22T17:57:25.158390+0000 mon.a (mon.0) 2597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:25 smithi121 bash[36902]: audit 2022-04-22T17:57:25.516459+0000 mon.a (mon.0) 2598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[45495]: cluster 2022-04-22T17:57:24.587145+0000 mgr.y (mgr.34107) 1121 : cluster [DBG] pgmap v1721: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[45495]: audit 2022-04-22T17:57:25.158390+0000 mon.a (mon.0) 2597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[45495]: audit 2022-04-22T17:57:25.516459+0000 mon.a (mon.0) 2598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[46934]: cluster 2022-04-22T17:57:24.587145+0000 mgr.y (mgr.34107) 1121 : cluster [DBG] pgmap v1721: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[46934]: audit 2022-04-22T17:57:25.158390+0000 mon.a (mon.0) 2597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:25 smithi115 bash[46934]: audit 2022-04-22T17:57:25.516459+0000 mon.a (mon.0) 2598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:26.558 INFO:tasks.workunit.client.0.smithi115.stdout: 627: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:26.558 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fvrSAir445JNphh off=902296 len=550607 2022-04-22T17:57:26.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 3213 completed, throughput=5MB/sec 2022-04-22T17:57:26.564 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aoNNftndCEhN4ky off=2198907 len=1569793 2022-04-22T17:57:26.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 3214 completed, throughput=5MB/sec 2022-04-22T17:57:26.567 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ir5jUmAy_skq3g9 off=16380262 len=1073583 2022-04-22T17:57:26.571 INFO:tasks.workunit.client.0.smithi115.stdout:op 3215 completed, throughput=5MB/sec 2022-04-22T17:57:26.571 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SuKV1rHdndVRPVw off=1478452 len=1152904 2022-04-22T17:57:26.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 3216 completed, throughput=5MB/sec 2022-04-22T17:57:27.577 INFO:tasks.workunit.client.0.smithi115.stdout: 628: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:27.577 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aJE03q4Mk-7qDJf off=704280 len=1456331 2022-04-22T17:57:27.586 INFO:tasks.workunit.client.0.smithi115.stdout:op 3217 completed, throughput=5MB/sec 2022-04-22T17:57:27.586 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6IjTA-LEP1feucg off=11015486 len=681758 2022-04-22T17:57:27.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 3218 completed, throughput=5MB/sec 2022-04-22T17:57:27.587 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TSxpLW81c0Y8a2R off=7366844 len=1580194 2022-04-22T17:57:27.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 3219 completed, throughput=5MB/sec 2022-04-22T17:57:27.596 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DEbfS7fbJIxyauq off=1679738 len=1915906 2022-04-22T17:57:27.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 3220 completed, throughput=5MB/sec 2022-04-22T17:57:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:27 smithi121 bash[36902]: cluster 2022-04-22T17:57:26.588394+0000 mgr.y (mgr.34107) 1122 : cluster [DBG] pgmap v1722: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:27 smithi121 bash[36902]: audit 2022-04-22T17:57:27.545500+0000 mon.a (mon.0) 2599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:27 smithi115 bash[45495]: cluster 2022-04-22T17:57:26.588394+0000 mgr.y (mgr.34107) 1122 : cluster [DBG] pgmap v1722: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:27 smithi115 bash[45495]: audit 2022-04-22T17:57:27.545500+0000 mon.a (mon.0) 2599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:28.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:27 smithi115 bash[46934]: cluster 2022-04-22T17:57:26.588394+0000 mgr.y (mgr.34107) 1122 : cluster [DBG] pgmap v1722: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:27 smithi115 bash[46934]: audit 2022-04-22T17:57:27.545500+0000 mon.a (mon.0) 2599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:28.599 INFO:tasks.workunit.client.0.smithi115.stdout: 629: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:28.599 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JAfpZ98sfcoPhNt off=2150581 len=1944114 2022-04-22T17:57:28.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 3221 completed, throughput=5MB/sec 2022-04-22T17:57:28.610 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DCx0u33LEaGJJkv off=1548623 len=1900728 2022-04-22T17:57:28.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 3222 completed, throughput=5MB/sec 2022-04-22T17:57:28.620 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BFubNVrcEvvsXvv off=17111336 len=1002083 2022-04-22T17:57:28.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 3223 completed, throughput=5MB/sec 2022-04-22T17:57:29.623 INFO:tasks.workunit.client.0.smithi115.stdout: 630: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:29.623 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FBti2jpdUSQKIi- off=3021304 len=1407144 2022-04-22T17:57:29.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 3224 completed, throughput=5MB/sec 2022-04-22T17:57:29.626 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LBMABUF59t2sl9b off=346600 len=346538 2022-04-22T17:57:29.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 3225 completed, throughput=5MB/sec 2022-04-22T17:57:29.628 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LTwG7dqN1xqrgCA off=4198523 len=2096529 2022-04-22T17:57:29.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 3226 completed, throughput=5MB/sec 2022-04-22T17:57:29.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7mNoSv3EPVwgH25 off=15681007 len=534670 2022-04-22T17:57:29.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 3227 completed, throughput=5MB/sec 2022-04-22T17:57:29.645 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YVI7_WuZW_z6dFX off=21580615 len=1473159 2022-04-22T17:57:29.648 INFO:tasks.workunit.client.0.smithi115.stdout:op 3228 completed, throughput=5MB/sec 2022-04-22T17:57:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:29 smithi121 bash[36902]: cluster 2022-04-22T17:57:28.589486+0000 mgr.y (mgr.34107) 1123 : cluster [DBG] pgmap v1723: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:57:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:29 smithi115 bash[45495]: cluster 2022-04-22T17:57:28.589486+0000 mgr.y (mgr.34107) 1123 : cluster [DBG] pgmap v1723: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:57:30.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:29 smithi115 bash[46934]: cluster 2022-04-22T17:57:28.589486+0000 mgr.y (mgr.34107) 1123 : cluster [DBG] pgmap v1723: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:57:30.649 INFO:tasks.workunit.client.0.smithi115.stdout: 631: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:30.649 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ic_eUxlXca8BbF0 off=15247820 len=1510339 2022-04-22T17:57:30.656 INFO:tasks.workunit.client.0.smithi115.stdout:op 3229 completed, throughput=5MB/sec 2022-04-22T17:57:30.656 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RYQyDICPzKnQdYq off=10621894 len=2021397 2022-04-22T17:57:30.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 3230 completed, throughput=5MB/sec 2022-04-22T17:57:30.668 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9wg_ObtUc5s0FZK off=21826439 len=2031288 2022-04-22T17:57:30.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 3231 completed, throughput=5MB/sec 2022-04-22T17:57:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:30 smithi121 bash[36902]: audit 2022-04-22T17:57:30.161338+0000 mon.a (mon.0) 2600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:30 smithi121 bash[36902]: audit 2022-04-22T17:57:30.520203+0000 mon.a (mon.0) 2601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:30 smithi115 bash[45495]: audit 2022-04-22T17:57:30.161338+0000 mon.a (mon.0) 2600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:30 smithi115 bash[45495]: audit 2022-04-22T17:57:30.520203+0000 mon.a (mon.0) 2601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:30 smithi115 bash[46934]: audit 2022-04-22T17:57:30.161338+0000 mon.a (mon.0) 2600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:30 smithi115 bash[46934]: audit 2022-04-22T17:57:30.520203+0000 mon.a (mon.0) 2601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:31.678 INFO:tasks.workunit.client.0.smithi115.stdout: 632: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:31.678 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FLf9MURUonlD4wt off=6101281 len=750348 2022-04-22T17:57:31.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 3232 completed, throughput=5MB/sec 2022-04-22T17:57:31.684 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RCeLr43fdj-fzme off=18583597 len=1120410 2022-04-22T17:57:31.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 3233 completed, throughput=5MB/sec 2022-04-22T17:57:31.687 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NExPNACwdtb_JRo off=19900112 len=286997 2022-04-22T17:57:31.691 INFO:tasks.workunit.client.0.smithi115.stdout:op 3234 completed, throughput=5MB/sec 2022-04-22T17:57:31.691 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rMAcKdpCO79W_3D off=12272013 len=818392 2022-04-22T17:57:31.693 INFO:tasks.workunit.client.0.smithi115.stdout:op 3235 completed, throughput=5MB/sec 2022-04-22T17:57:31.693 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BGAsDUpFBGpKr3N off=2845645 len=582478 2022-04-22T17:57:31.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 3236 completed, throughput=5MB/sec 2022-04-22T17:57:31.698 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dDdioAw1V10Lw8P off=19184529 len=1500354 2022-04-22T17:57:31.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 3237 completed, throughput=5MB/sec 2022-04-22T17:57:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:31 smithi115 bash[45495]: cluster 2022-04-22T17:57:30.590659+0000 mgr.y (mgr.34107) 1124 : cluster [DBG] pgmap v1724: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:31.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:31 smithi115 bash[46934]: cluster 2022-04-22T17:57:30.590659+0000 mgr.y (mgr.34107) 1124 : cluster [DBG] pgmap v1724: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:31 smithi121 bash[36902]: cluster 2022-04-22T17:57:30.590659+0000 mgr.y (mgr.34107) 1124 : cluster [DBG] pgmap v1724: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:32.702 INFO:tasks.workunit.client.0.smithi115.stdout: 633: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:32.702 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kLZuHdelThNCk8T off=5186343 len=1610461 2022-04-22T17:57:32.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 3238 completed, throughput=5MB/sec 2022-04-22T17:57:32.711 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8OyWIWBcXBd2-ro off=791594 len=1582374 2022-04-22T17:57:32.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 3239 completed, throughput=5MB/sec 2022-04-22T17:57:32.716 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2YGBJFTbS382jvD off=4447709 len=1365297 2022-04-22T17:57:32.722 INFO:tasks.workunit.client.0.smithi115.stdout:op 3240 completed, throughput=5MB/sec 2022-04-22T17:57:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:32 smithi121 bash[36902]: audit 2022-04-22T17:57:32.548660+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:32 smithi115 bash[45495]: audit 2022-04-22T17:57:32.548660+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:32 smithi115 bash[46934]: audit 2022-04-22T17:57:32.548660+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:33.722 INFO:tasks.workunit.client.0.smithi115.stdout: 634: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:33.722 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JreobDqMLWCg30k off=3116509 len=1753317 2022-04-22T17:57:33.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 3241 completed, throughput=5MB/sec 2022-04-22T17:57:33.728 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DV4-eBMFxx2EA2A off=16640230 len=590453 2022-04-22T17:57:33.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 3242 completed, throughput=5MB/sec 2022-04-22T17:57:33.731 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U5a1KbKTUKE3ecs off=9178885 len=1566298 2022-04-22T17:57:33.739 INFO:tasks.workunit.client.0.smithi115.stdout:op 3243 completed, throughput=5MB/sec 2022-04-22T17:57:33.739 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dTY0vbiUtUHkUSc off=14787060 len=1411722 2022-04-22T17:57:33.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 3244 completed, throughput=5MB/sec 2022-04-22T17:57:33.876 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:33.594Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:33 smithi121 bash[36902]: cluster 2022-04-22T17:57:32.591837+0000 mgr.y (mgr.34107) 1125 : cluster [DBG] pgmap v1725: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:34.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:33 smithi115 bash[45495]: cluster 2022-04-22T17:57:32.591837+0000 mgr.y (mgr.34107) 1125 : cluster [DBG] pgmap v1725: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:34.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:33 smithi115 bash[46934]: cluster 2022-04-22T17:57:32.591837+0000 mgr.y (mgr.34107) 1125 : cluster [DBG] pgmap v1725: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T17:57:34.747 INFO:tasks.workunit.client.0.smithi115.stdout: 635: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:34.817 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0UOAWhwYA6_g63F off=6967631 len=577865 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3245 completed, throughput=4.99MB/sec 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_BJpSxlsm2OeI6- off=3934987 len=670424 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3246 completed, throughput=4.99MB/sec 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H0q6QIIXCpx17QF off=7621963 len=1168438 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3247 completed, throughput=5MB/sec 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h6EShFNbLm30hsv off=14714311 len=1063122 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3248 completed, throughput=5MB/sec 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-68yKZyXCo56KM8b off=6477942 len=1664784 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3249 completed, throughput=5MB/sec 2022-04-22T17:57:34.819 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iim-8Ex6M59Royx off=11163242 len=1382405 2022-04-22T17:57:34.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 3250 completed, throughput=5MB/sec 2022-04-22T17:57:35.775 INFO:tasks.workunit.client.0.smithi115.stdout: 636: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:35.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gh2I6eU9TJcMNRY off=143895 len=880341 2022-04-22T17:57:35.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 3251 completed, throughput=5MB/sec 2022-04-22T17:57:35.777 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WyR3yF5rc5FgAsB off=3678264 len=968490 2022-04-22T17:57:35.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 3252 completed, throughput=5MB/sec 2022-04-22T17:57:35.785 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oih5URRzMlYDhnu off=856020 len=549592 2022-04-22T17:57:35.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 3253 completed, throughput=5MB/sec 2022-04-22T17:57:35.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-efIwUvMF3y8TX7j off=18421144 len=1849219 2022-04-22T17:57:35.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 3254 completed, throughput=5MB/sec 2022-04-22T17:57:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:35 smithi121 bash[36902]: cluster 2022-04-22T17:57:34.592910+0000 mgr.y (mgr.34107) 1126 : cluster [DBG] pgmap v1726: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:35 smithi121 bash[36902]: audit 2022-04-22T17:57:35.164461+0000 mon.a (mon.0) 2603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:35 smithi121 bash[36902]: audit 2022-04-22T17:57:35.523919+0000 mon.a (mon.0) 2604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[45495]: cluster 2022-04-22T17:57:34.592910+0000 mgr.y (mgr.34107) 1126 : cluster [DBG] pgmap v1726: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[45495]: audit 2022-04-22T17:57:35.164461+0000 mon.a (mon.0) 2603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[45495]: audit 2022-04-22T17:57:35.523919+0000 mon.a (mon.0) 2604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[46934]: cluster 2022-04-22T17:57:34.592910+0000 mgr.y (mgr.34107) 1126 : cluster [DBG] pgmap v1726: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T17:57:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[46934]: audit 2022-04-22T17:57:35.164461+0000 mon.a (mon.0) 2603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:35 smithi115 bash[46934]: audit 2022-04-22T17:57:35.523919+0000 mon.a (mon.0) 2604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:36.795 INFO:tasks.workunit.client.0.smithi115.stdout: 637: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:36.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GLwV0TvP6QAixts off=20201819 len=1515077 2022-04-22T17:57:36.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 3255 completed, throughput=4.99MB/sec 2022-04-22T17:57:36.798 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xU2-L6XvYbV4WBh off=6738729 len=1513934 2022-04-22T17:57:36.802 INFO:tasks.workunit.client.0.smithi115.stdout:op 3256 completed, throughput=5MB/sec 2022-04-22T17:57:36.802 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gzbMMZIB-IspzsV off=3389655 len=7238 2022-04-22T17:57:36.803 INFO:tasks.workunit.client.0.smithi115.stdout:op 3257 completed, throughput=5MB/sec 2022-04-22T17:57:36.803 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5h4-W1thMB7WRAi off=1136226 len=1928621 2022-04-22T17:57:36.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 3258 completed, throughput=5MB/sec 2022-04-22T17:57:36.814 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gmrtX0x7FgZOj2F off=11726897 len=964130 2022-04-22T17:57:36.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 3259 completed, throughput=5MB/sec 2022-04-22T17:57:37.821 INFO:tasks.workunit.client.0.smithi115.stdout: 638: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:37.821 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--snRZV7AD1jh0Mh off=156071 len=797366 2022-04-22T17:57:37.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 3260 completed, throughput=4.99MB/sec 2022-04-22T17:57:37.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ImjxyHyjlahjXM6 off=6822646 len=1884258 2022-04-22T17:57:37.833 INFO:tasks.workunit.client.0.smithi115.stdout:op 3261 completed, throughput=5MB/sec 2022-04-22T17:57:37.833 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qQRnSGgEpYMR80t off=1578293 len=669975 2022-04-22T17:57:37.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 3262 completed, throughput=5MB/sec 2022-04-22T17:57:37.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fAUrvKrausNL_Z_ off=11312133 len=1641199 2022-04-22T17:57:37.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 3263 completed, throughput=5MB/sec 2022-04-22T17:57:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:37 smithi121 bash[36902]: cluster 2022-04-22T17:57:36.593944+0000 mgr.y (mgr.34107) 1127 : cluster [DBG] pgmap v1727: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:37 smithi121 bash[36902]: audit 2022-04-22T17:57:37.551709+0000 mon.a (mon.0) 2605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:37 smithi115 bash[46934]: cluster 2022-04-22T17:57:36.593944+0000 mgr.y (mgr.34107) 1127 : cluster [DBG] pgmap v1727: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:37 smithi115 bash[46934]: audit 2022-04-22T17:57:37.551709+0000 mon.a (mon.0) 2605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:37 smithi115 bash[45495]: cluster 2022-04-22T17:57:36.593944+0000 mgr.y (mgr.34107) 1127 : cluster [DBG] pgmap v1727: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:37 smithi115 bash[45495]: audit 2022-04-22T17:57:37.551709+0000 mon.a (mon.0) 2605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:38.845 INFO:tasks.workunit.client.0.smithi115.stdout: 639: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:38.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X3VhnF1Ui7OpH_T off=7227705 len=267920 2022-04-22T17:57:38.847 INFO:tasks.workunit.client.0.smithi115.stdout:op 3264 completed, throughput=4.99MB/sec 2022-04-22T17:57:38.847 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CGEjIguXieWLV8K off=9745904 len=998551 2022-04-22T17:57:38.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 3265 completed, throughput=4.99MB/sec 2022-04-22T17:57:38.850 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-baZv4TAvdFHg0a2 off=11345880 len=2005024 2022-04-22T17:57:38.856 INFO:tasks.workunit.client.0.smithi115.stdout:op 3266 completed, throughput=5MB/sec 2022-04-22T17:57:38.856 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s2r7XiRSBGXn770 off=624898 len=1280387 2022-04-22T17:57:38.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 3267 completed, throughput=5MB/sec 2022-04-22T17:57:38.863 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-B75VXZS8jx8AS6h off=1499605 len=1560276 2022-04-22T17:57:38.864 INFO:tasks.workunit.client.0.smithi115.stdout:op 3268 completed, throughput=5MB/sec 2022-04-22T17:57:39.865 INFO:tasks.workunit.client.0.smithi115.stdout: 640: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:39.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kcTfBKgnj7fjPPc off=1096070 len=180711 2022-04-22T17:57:39.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 3269 completed, throughput=4.99MB/sec 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JZlH7-AY--JLi-H off=1242159 len=1413685 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:op 3270 completed, throughput=5MB/sec 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kcTfBKgnj7fjPPc off=6302182 len=832836 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:op 3271 completed, throughput=5MB/sec 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-J2ZeJ_mEf0Bi72S off=848512 len=1220489 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:op 3272 completed, throughput=5MB/sec 2022-04-22T17:57:39.892 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xbNPI-Zm7e0KMQ7 off=11543322 len=423465 2022-04-22T17:57:39.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 3273 completed, throughput=5MB/sec 2022-04-22T17:57:39.893 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9qsJYT8sASaB7Pf off=8135327 len=1042507 2022-04-22T17:57:39.894 INFO:tasks.workunit.client.0.smithi115.stdout:op 3274 completed, throughput=5MB/sec 2022-04-22T17:57:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:39 smithi121 bash[36902]: cluster 2022-04-22T17:57:38.595130+0000 mgr.y (mgr.34107) 1128 : cluster [DBG] pgmap v1728: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:40.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:39 smithi115 bash[45495]: cluster 2022-04-22T17:57:38.595130+0000 mgr.y (mgr.34107) 1128 : cluster [DBG] pgmap v1728: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:39 smithi115 bash[46934]: cluster 2022-04-22T17:57:38.595130+0000 mgr.y (mgr.34107) 1128 : cluster [DBG] pgmap v1728: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T17:57:40.895 INFO:tasks.workunit.client.0.smithi115.stdout: 641: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:40.895 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZCmSZAV9URguqlV off=1046006 len=648549 2022-04-22T17:57:40.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 3275 completed, throughput=4.99MB/sec 2022-04-22T17:57:40.901 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3APRrHj7ZIqNr22 off=11550805 len=104529 2022-04-22T17:57:40.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 3276 completed, throughput=4.99MB/sec 2022-04-22T17:57:40.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Izo5anohx3ILpJQ off=1491861 len=444902 2022-04-22T17:57:40.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 3277 completed, throughput=5MB/sec 2022-04-22T17:57:41.021 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MDaswWjLcLyZhCd off=4355249 len=1124681 2022-04-22T17:57:41.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 3278 completed, throughput=5MB/sec 2022-04-22T17:57:41.022 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rssk6a7HQU-Fuwt off=4501144 len=2085597 2022-04-22T17:57:41.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 3279 completed, throughput=5MB/sec 2022-04-22T17:57:41.027 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nRj3HzwWNf5UR73 off=9291942 len=1115410 2022-04-22T17:57:41.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 3280 completed, throughput=5MB/sec 2022-04-22T17:57:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:40 smithi121 bash[36902]: audit 2022-04-22T17:57:40.167758+0000 mon.a (mon.0) 2606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:40 smithi121 bash[36902]: audit 2022-04-22T17:57:40.527526+0000 mon.a (mon.0) 2607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:40 smithi115 bash[46934]: audit 2022-04-22T17:57:40.167758+0000 mon.a (mon.0) 2606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:40 smithi115 bash[46934]: audit 2022-04-22T17:57:40.527526+0000 mon.a (mon.0) 2607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:40 smithi115 bash[45495]: audit 2022-04-22T17:57:40.167758+0000 mon.a (mon.0) 2606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:40 smithi115 bash[45495]: audit 2022-04-22T17:57:40.527526+0000 mon.a (mon.0) 2607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:41.920 INFO:tasks.workunit.client.0.smithi115.stdout: 642: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:41.921 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w5DOdTpNHpmgvJn off=15857247 len=720356 2022-04-22T17:57:41.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 3281 completed, throughput=4.99MB/sec 2022-04-22T17:57:41.923 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XE2URpxFZSdiZKJ off=3186102 len=1400649 2022-04-22T17:57:41.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 3282 completed, throughput=5MB/sec 2022-04-22T17:57:41.926 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JPCm6tlWGB391su off=595972 len=1375506 2022-04-22T17:57:41.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 3283 completed, throughput=5MB/sec 2022-04-22T17:57:41.933 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k8lZd31wyipUFDD off=10234215 len=1932903 2022-04-22T17:57:41.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 3284 completed, throughput=5MB/sec 2022-04-22T17:57:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:41 smithi115 bash[45495]: cluster 2022-04-22T17:57:40.596218+0000 mgr.y (mgr.34107) 1129 : cluster [DBG] pgmap v1729: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2022-04-22T17:57:42.264 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:41 smithi115 bash[46934]: cluster 2022-04-22T17:57:40.596218+0000 mgr.y (mgr.34107) 1129 : cluster [DBG] pgmap v1729: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2022-04-22T17:57:42.265 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:41.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:42.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:41 smithi121 bash[36902]: cluster 2022-04-22T17:57:40.596218+0000 mgr.y (mgr.34107) 1129 : cluster [DBG] pgmap v1729: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 3 op/s 2022-04-22T17:57:42.945 INFO:tasks.workunit.client.0.smithi115.stdout: 643: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:42.946 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2HDRBOQ10BMZeLT off=1129040 len=2022816 2022-04-22T17:57:42.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 3285 completed, throughput=5MB/sec 2022-04-22T17:57:42.951 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VI-2YCZmxF646t- off=2804516 len=1254820 2022-04-22T17:57:42.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 3286 completed, throughput=5MB/sec 2022-04-22T17:57:42.965 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LK-SXmK_qSearHE off=15689452 len=480140 2022-04-22T17:57:42.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 3287 completed, throughput=5MB/sec 2022-04-22T17:57:42.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m4bIolLoXpOGwmt off=5243713 len=351324 2022-04-22T17:57:42.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 3288 completed, throughput=5MB/sec 2022-04-22T17:57:42.966 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jNARHXI8TlIFA3o off=10302269 len=509687 2022-04-22T17:57:42.975 INFO:tasks.workunit.client.0.smithi115.stdout:op 3289 completed, throughput=5MB/sec 2022-04-22T17:57:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:42 smithi121 bash[36902]: audit 2022-04-22T17:57:42.554671+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:43.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:42 smithi115 bash[45495]: audit 2022-04-22T17:57:42.554671+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:42 smithi115 bash[46934]: audit 2022-04-22T17:57:42.554671+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:43.909 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:43.594Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:43.969 INFO:tasks.workunit.client.0.smithi115.stdout: 644: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:43.970 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eCCaix8kUkP75CV off=638813 len=383883 2022-04-22T17:57:43.974 INFO:tasks.workunit.client.0.smithi115.stdout:op 3290 completed, throughput=4.99MB/sec 2022-04-22T17:57:43.974 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-suyRyI3sD22UBUg off=8941523 len=167796 2022-04-22T17:57:43.975 INFO:tasks.workunit.client.0.smithi115.stdout:op 3291 completed, throughput=4.99MB/sec 2022-04-22T17:57:43.975 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nh1X6i2YLUEhpIc off=4724445 len=792909 2022-04-22T17:57:43.977 INFO:tasks.workunit.client.0.smithi115.stdout:op 3292 completed, throughput=4.99MB/sec 2022-04-22T17:57:43.977 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HiM3QClEKggtN74 off=6659503 len=1133426 2022-04-22T17:57:43.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 3293 completed, throughput=5MB/sec 2022-04-22T17:57:43.984 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V2U6RIc035xVU1x off=16926283 len=1920831 2022-04-22T17:57:43.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 3294 completed, throughput=5MB/sec 2022-04-22T17:57:43.989 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ktOYT_W85ZrRtsb off=3518569 len=699683 2022-04-22T17:57:43.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 3295 completed, throughput=5MB/sec 2022-04-22T17:57:43.991 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tQ_9eIrbHZ0R8cV off=182550 len=162489 2022-04-22T17:57:43.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 3296 completed, throughput=5MB/sec 2022-04-22T17:57:44.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:43 smithi121 bash[36902]: cluster 2022-04-22T17:57:42.597295+0000 mgr.y (mgr.34107) 1130 : cluster [DBG] pgmap v1730: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:44.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:43 smithi115 bash[45495]: cluster 2022-04-22T17:57:42.597295+0000 mgr.y (mgr.34107) 1130 : cluster [DBG] pgmap v1730: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:43 smithi115 bash[46934]: cluster 2022-04-22T17:57:42.597295+0000 mgr.y (mgr.34107) 1130 : cluster [DBG] pgmap v1730: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:57:44.995 INFO:tasks.workunit.client.0.smithi115.stdout: 645: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:45.048 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VBHouGh0EA_iprD off=0 len=1404554 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 3297 completed, throughput=4.99MB/sec 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nr6ZegyFUDS8f4A off=3737993 len=1758607 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 3298 completed, throughput=5MB/sec 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4NTahulr5IZAdQZ off=1318916 len=1386685 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 3299 completed, throughput=5MB/sec 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bdt8cRh4ABpvnyz off=3716962 len=1513672 2022-04-22T17:57:45.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 3300 completed, throughput=5MB/sec 2022-04-22T17:57:45.402 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:57:45.762 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:57:46.017 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (20m) 10m ago 34m 15.1M - ba2b418f427c 9255ba08f72f 2022-04-22T17:57:46.115 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (34m) 61s ago 34m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:57:46.115 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (22m) 10m ago 35m 17.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:57:46.115 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (22m) 61s ago 35m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:57:46.115 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (30m) 61s ago 39m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:57:46.115 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (30m) 10m ago 42m 462M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (29m) 10m ago 42m 126M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (28m) 61s ago 40m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (29m) 10m ago 40m 89.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (21m) 10m ago 34m 9780k - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (21m) 61s ago 34m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:57:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (27m) 10m ago 39m 595M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (26m) 10m ago 39m 674M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (26m) 10m ago 38m 435M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (25m) 10m ago 37m 385M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (25m) 61s ago 37m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (24m) 61s ago 37m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:57:46.117 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (23m) 61s ago 36m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:57:46.118 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (23m) 61s ago 36m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:57:46.118 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (21m) 61s ago 34m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: cluster 2022-04-22T17:57:44.598399+0000 mgr.y (mgr.34107) 1131 : cluster [DBG] pgmap v1731: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.023301+0000 mon.a (mon.0) 2609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.025842+0000 mon.a (mon.0) 2610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.027261+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.170615+0000 mon.a (mon.0) 2612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.335613+0000 mon.a (mon.0) 2613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.354327+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.379893+0000 mgr.y (mgr.34107) 1132 : audit [DBG] from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:46.119 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:45 smithi121 bash[36902]: audit 2022-04-22T17:57:45.530797+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout: 646: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hX5T_kWp2rvtZX9 off=7528220 len=1864953 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 3301 completed, throughput=5MB/sec 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P72qzBRENz3oE9I off=2950992 len=1196554 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 3302 completed, throughput=5MB/sec 2022-04-22T17:57:46.121 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FApNEfylyfcUbgJ off=2767819 len=514910 2022-04-22T17:57:46.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 3303 completed, throughput=5MB/sec 2022-04-22T17:57:46.122 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oif3KooCEvGg5tv off=7257389 len=1432666 2022-04-22T17:57:46.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 3304 completed, throughput=5MB/sec 2022-04-22T17:57:46.183 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:57:46.183 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:57:46.183 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:57:46.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:46.184 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:57:46.185 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:57:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: cluster 2022-04-22T17:57:44.598399+0000 mgr.y (mgr.34107) 1131 : cluster [DBG] pgmap v1731: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.023301+0000 mon.a (mon.0) 2609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.025842+0000 mon.a (mon.0) 2610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.027261+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.170615+0000 mon.a (mon.0) 2612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.335613+0000 mon.a (mon.0) 2613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.354327+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.379893+0000 mgr.y (mgr.34107) 1132 : audit [DBG] from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[45495]: audit 2022-04-22T17:57:45.530797+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: cluster 2022-04-22T17:57:44.598399+0000 mgr.y (mgr.34107) 1131 : cluster [DBG] pgmap v1731: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.023301+0000 mon.a (mon.0) 2609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.025842+0000 mon.a (mon.0) 2610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.027261+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.170615+0000 mon.a (mon.0) 2612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.335613+0000 mon.a (mon.0) 2613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.354327+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.379893+0000 mgr.y (mgr.34107) 1132 : audit [DBG] from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:45 smithi115 bash[46934]: audit 2022-04-22T17:57:45.530797+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:47.039 INFO:tasks.workunit.client.0.smithi115.stdout: 647: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:47.142 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DI2Nb6Y-Fv1-j2Z off=3625742 len=2005362 2022-04-22T17:57:47.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 3305 completed, throughput=5MB/sec 2022-04-22T17:57:47.145 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ImjxyHyjlahjXM6 off=8509079 len=1842227 2022-04-22T17:57:47.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 3306 completed, throughput=5MB/sec 2022-04-22T17:57:47.154 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-onI0fiZHa9RDQe6 off=555224 len=1330325 2022-04-22T17:57:47.154 INFO:tasks.workunit.client.0.smithi115.stdout:op 3307 completed, throughput=5MB/sec 2022-04-22T17:57:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:46 smithi121 bash[36902]: audit 2022-04-22T17:57:45.733669+0000 mgr.y (mgr.34107) 1133 : audit [DBG] from='client.45535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:46 smithi121 bash[36902]: audit 2022-04-22T17:57:46.176478+0000 mon.a (mon.0) 2616 : audit [DBG] from='client.? 172.21.15.115:0/2806858510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:46 smithi115 bash[45495]: audit 2022-04-22T17:57:45.733669+0000 mgr.y (mgr.34107) 1133 : audit [DBG] from='client.45535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:46 smithi115 bash[45495]: audit 2022-04-22T17:57:46.176478+0000 mon.a (mon.0) 2616 : audit [DBG] from='client.? 172.21.15.115:0/2806858510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:47.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:46 smithi115 bash[46934]: audit 2022-04-22T17:57:45.733669+0000 mgr.y (mgr.34107) 1133 : audit [DBG] from='client.45535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:57:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:46 smithi115 bash[46934]: audit 2022-04-22T17:57:46.176478+0000 mon.a (mon.0) 2616 : audit [DBG] from='client.? 172.21.15.115:0/2806858510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:57:48.061 INFO:tasks.workunit.client.0.smithi115.stdout: 648: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:48.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F8tRe9Rfqsqx7hB off=4642543 len=765097 2022-04-22T17:57:48.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 3308 completed, throughput=4.99MB/sec 2022-04-22T17:57:48.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wkrqn3iMhpbdygr off=463163 len=189444 2022-04-22T17:57:48.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 3309 completed, throughput=4.99MB/sec 2022-04-22T17:57:48.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b5hhqH8Lsp_75PY off=332714 len=1871334 2022-04-22T17:57:48.073 INFO:tasks.workunit.client.0.smithi115.stdout:op 3310 completed, throughput=5MB/sec 2022-04-22T17:57:48.116 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxaufqIvu4KwqKP off=9736920 len=711092 2022-04-22T17:57:48.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 3311 completed, throughput=5MB/sec 2022-04-22T17:57:48.118 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-83vJUWpPAz2LwVw off=1261239 len=618720 2022-04-22T17:57:48.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 3312 completed, throughput=5MB/sec 2022-04-22T17:57:48.118 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jizMQfanm7ddtWy off=3733627 len=1853508 2022-04-22T17:57:48.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 3313 completed, throughput=5MB/sec 2022-04-22T17:57:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:47 smithi121 bash[36902]: cluster 2022-04-22T17:57:46.599511+0000 mgr.y (mgr.34107) 1134 : cluster [DBG] pgmap v1732: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:47 smithi121 bash[36902]: audit 2022-04-22T17:57:47.557705+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:47 smithi115 bash[46934]: cluster 2022-04-22T17:57:46.599511+0000 mgr.y (mgr.34107) 1134 : cluster [DBG] pgmap v1732: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:47 smithi115 bash[46934]: audit 2022-04-22T17:57:47.557705+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:47 smithi115 bash[45495]: cluster 2022-04-22T17:57:46.599511+0000 mgr.y (mgr.34107) 1134 : cluster [DBG] pgmap v1732: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:57:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:47 smithi115 bash[45495]: audit 2022-04-22T17:57:47.557705+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:49.089 INFO:tasks.workunit.client.0.smithi115.stdout: 649: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:49.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WsAm2gB5W66BOTK off=7673325 len=1127625 2022-04-22T17:57:49.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 3314 completed, throughput=5MB/sec 2022-04-22T17:57:49.092 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GXAAj5y656S8ZLf off=2204860 len=752858 2022-04-22T17:57:49.093 INFO:tasks.workunit.client.0.smithi115.stdout:op 3315 completed, throughput=5MB/sec 2022-04-22T17:57:49.094 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wdPQuCeKn5Ht80X off=12257018 len=1742757 2022-04-22T17:57:49.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 3316 completed, throughput=5MB/sec 2022-04-22T17:57:49.097 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1j2TEXyrZXfEldV off=7182021 len=1174515 2022-04-22T17:57:49.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 3317 completed, throughput=5MB/sec 2022-04-22T17:57:50.104 INFO:tasks.workunit.client.0.smithi115.stdout: 650: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:50.104 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pb_Bg1NNuA7OVUH off=7363016 len=1722004 2022-04-22T17:57:50.113 INFO:tasks.workunit.client.0.smithi115.stdout:op 3318 completed, throughput=5MB/sec 2022-04-22T17:57:50.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2Odb2X5i8iwDIJc off=12136505 len=847390 2022-04-22T17:57:50.118 INFO:tasks.workunit.client.0.smithi115.stdout:op 3319 completed, throughput=5MB/sec 2022-04-22T17:57:50.118 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Mu_VI4Rs3OSRDKL off=1044001 len=1159001 2022-04-22T17:57:50.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 3320 completed, throughput=5MB/sec 2022-04-22T17:57:50.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7Rn1BthrmXFr8RQ off=145128 len=2083954 2022-04-22T17:57:50.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 3321 completed, throughput=5MB/sec 2022-04-22T17:57:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:49 smithi121 bash[36902]: cluster 2022-04-22T17:57:48.600729+0000 mgr.y (mgr.34107) 1135 : cluster [DBG] pgmap v1733: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:57:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:49 smithi121 bash[36902]: audit 2022-04-22T17:57:49.504027+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:49 smithi121 bash[36902]: audit 2022-04-22T17:57:49.843650+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[45495]: cluster 2022-04-22T17:57:48.600729+0000 mgr.y (mgr.34107) 1135 : cluster [DBG] pgmap v1733: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:57:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[45495]: audit 2022-04-22T17:57:49.504027+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:50.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[45495]: audit 2022-04-22T17:57:49.843650+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[46934]: cluster 2022-04-22T17:57:48.600729+0000 mgr.y (mgr.34107) 1135 : cluster [DBG] pgmap v1733: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:57:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[46934]: audit 2022-04-22T17:57:49.504027+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:49 smithi115 bash[46934]: audit 2022-04-22T17:57:49.843650+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:57:51.127 INFO:tasks.workunit.client.0.smithi115.stdout: 651: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:51.180 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cu8galFBIqEM4iZ off=7785516 len=417581 2022-04-22T17:57:51.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 3322 completed, throughput=4.99MB/sec 2022-04-22T17:57:51.181 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-trhEaHPzQiq4KC- off=1621009 len=1322329 2022-04-22T17:57:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 3323 completed, throughput=5MB/sec 2022-04-22T17:57:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Mz5u6Hiq5tZH6LG off=6208011 len=1709454 2022-04-22T17:57:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 3324 completed, throughput=5MB/sec 2022-04-22T17:57:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f9Y7Xoya_GHK62t off=5822508 len=1783768 2022-04-22T17:57:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 3325 completed, throughput=5MB/sec 2022-04-22T17:57:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:50 smithi121 bash[36902]: audit 2022-04-22T17:57:50.173542+0000 mon.a (mon.0) 2620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:50 smithi121 bash[36902]: audit 2022-04-22T17:57:50.533888+0000 mon.a (mon.0) 2621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:50 smithi115 bash[45495]: audit 2022-04-22T17:57:50.173542+0000 mon.a (mon.0) 2620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:50 smithi115 bash[45495]: audit 2022-04-22T17:57:50.533888+0000 mon.a (mon.0) 2621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:50 smithi115 bash[46934]: audit 2022-04-22T17:57:50.173542+0000 mon.a (mon.0) 2620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:50 smithi115 bash[46934]: audit 2022-04-22T17:57:50.533888+0000 mon.a (mon.0) 2621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:51 smithi115 bash[45495]: cluster 2022-04-22T17:57:50.601861+0000 mgr.y (mgr.34107) 1136 : cluster [DBG] pgmap v1734: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:57:52.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:51.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:51 smithi115 bash[46934]: cluster 2022-04-22T17:57:50.601861+0000 mgr.y (mgr.34107) 1136 : cluster [DBG] pgmap v1734: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:57:52.144 INFO:tasks.workunit.client.0.smithi115.stdout: 652: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:52.144 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-prZpycUjnhD40H3 off=0 len=1258854 2022-04-22T17:57:52.156 INFO:tasks.workunit.client.0.smithi115.stdout:op 3326 completed, throughput=5MB/sec 2022-04-22T17:57:52.156 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W-nespie0FbnKCL off=11913449 len=1145488 2022-04-22T17:57:52.156 INFO:tasks.workunit.client.0.smithi115.stdout:op 3327 completed, throughput=5MB/sec 2022-04-22T17:57:52.156 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KJ4ezhFViz2u9pv off=816852 len=655762 2022-04-22T17:57:52.159 INFO:tasks.workunit.client.0.smithi115.stdout:op 3328 completed, throughput=5MB/sec 2022-04-22T17:57:52.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SNhhXR96TOF24_I off=9356946 len=1468726 2022-04-22T17:57:52.161 INFO:tasks.workunit.client.0.smithi115.stdout:op 3329 completed, throughput=5MB/sec 2022-04-22T17:57:52.161 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MF9F1phXyllPEbL off=3777168 len=1739389 2022-04-22T17:57:52.164 INFO:tasks.workunit.client.0.smithi115.stdout:op 3330 completed, throughput=5MB/sec 2022-04-22T17:57:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:51 smithi121 bash[36902]: cluster 2022-04-22T17:57:50.601861+0000 mgr.y (mgr.34107) 1136 : cluster [DBG] pgmap v1734: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:57:53.165 INFO:tasks.workunit.client.0.smithi115.stdout: 653: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:53.182 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JLSXlq1JfNokdw1 off=6088505 len=1304414 2022-04-22T17:57:53.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 3331 completed, throughput=5MB/sec 2022-04-22T17:57:53.184 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-76YaYAfjH_Krxif off=2829132 len=607033 2022-04-22T17:57:53.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 3332 completed, throughput=5MB/sec 2022-04-22T17:57:53.184 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cjGSd8PGzLxlCiH off=3403022 len=381532 2022-04-22T17:57:53.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 3333 completed, throughput=5MB/sec 2022-04-22T17:57:53.185 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d2zgLD4OzxgeK05 off=3234104 len=234378 2022-04-22T17:57:53.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 3334 completed, throughput=5MB/sec 2022-04-22T17:57:53.189 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eGqer3AWQHAJukk off=1001055 len=89686 2022-04-22T17:57:53.189 INFO:tasks.workunit.client.0.smithi115.stdout:op 3335 completed, throughput=5MB/sec 2022-04-22T17:57:53.189 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uSyBRtkcHErLUBa off=3828593 len=1982803 2022-04-22T17:57:53.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 3336 completed, throughput=5MB/sec 2022-04-22T17:57:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:52 smithi121 bash[36902]: audit 2022-04-22T17:57:52.560834+0000 mon.a (mon.0) 2622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:52 smithi115 bash[45495]: audit 2022-04-22T17:57:52.560834+0000 mon.a (mon.0) 2622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:52 smithi115 bash[46934]: audit 2022-04-22T17:57:52.560834+0000 mon.a (mon.0) 2622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:53.948 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:57:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:57:53.597Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:57:54.183 INFO:tasks.workunit.client.0.smithi115.stdout: 654: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:54.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TzD1YPs4Y-K09YQ off=3542006 len=1014188 2022-04-22T17:57:54.235 INFO:tasks.workunit.client.0.smithi115.stdout:op 3337 completed, throughput=5MB/sec 2022-04-22T17:57:54.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SAPWBau2rdkwckQ off=1104326 len=1369607 2022-04-22T17:57:54.315 INFO:tasks.workunit.client.0.smithi115.stdout:op 3338 completed, throughput=5MB/sec 2022-04-22T17:57:54.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kjxHE3v5n7h6k_0 off=596256 len=306749 2022-04-22T17:57:54.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 3339 completed, throughput=5MB/sec 2022-04-22T17:57:54.316 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KQPRfSsHY1hplvT off=4058983 len=180160 2022-04-22T17:57:54.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 3340 completed, throughput=5MB/sec 2022-04-22T17:57:54.316 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-foFo2Zbir6aOu1s off=19302970 len=1807537 2022-04-22T17:57:54.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 3341 completed, throughput=5MB/sec 2022-04-22T17:57:54.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:53 smithi121 bash[36902]: cluster 2022-04-22T17:57:52.602941+0000 mgr.y (mgr.34107) 1137 : cluster [DBG] pgmap v1735: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:57:54.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:53 smithi115 bash[45495]: cluster 2022-04-22T17:57:52.602941+0000 mgr.y (mgr.34107) 1137 : cluster [DBG] pgmap v1735: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:57:54.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:53 smithi115 bash[46934]: cluster 2022-04-22T17:57:52.602941+0000 mgr.y (mgr.34107) 1137 : cluster [DBG] pgmap v1735: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:57:55.206 INFO:tasks.workunit.client.0.smithi115.stdout: 655: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:55.269 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HkGmOdx77M5xnB5 off=6088424 len=1780988 2022-04-22T17:57:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3342 completed, throughput=5MB/sec 2022-04-22T17:57:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-twoOGwU5F8BXNvf off=3467578 len=409994 2022-04-22T17:57:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3343 completed, throughput=5MB/sec 2022-04-22T17:57:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tPvC2RHJ5g0PK5a off=8125271 len=1178910 2022-04-22T17:57:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3344 completed, throughput=5MB/sec 2022-04-22T17:57:55.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-M1yL7tkJbfAvIEn off=1100908 len=1718121 2022-04-22T17:57:55.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 3345 completed, throughput=5MB/sec 2022-04-22T17:57:55.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k7G1_v3d3ZidipX off=12606718 len=1429319 2022-04-22T17:57:55.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 3346 completed, throughput=5MB/sec 2022-04-22T17:57:56.224 INFO:tasks.workunit.client.0.smithi115.stdout: 656: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:56.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2ADGV9Spc5fKCSL off=3994680 len=670399 2022-04-22T17:57:56.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:55 smithi121 bash[36902]: cluster 2022-04-22T17:57:54.604038+0000 mgr.y (mgr.34107) 1138 : cluster [DBG] pgmap v1736: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:57:56.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:55 smithi121 bash[36902]: audit 2022-04-22T17:57:55.176571+0000 mon.a (mon.0) 2623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:56.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:55 smithi121 bash[36902]: audit 2022-04-22T17:57:55.537514+0000 mon.a (mon.0) 2624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 3347 completed, throughput=5MB/sec 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nUk5SOmqZNH1huZ off=3308050 len=1055982 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 3348 completed, throughput=5MB/sec 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GoMWWgYrBCrflaP off=5976763 len=682125 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 3349 completed, throughput=5MB/sec 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3ihvNkXN4-1Q-44 off=14423338 len=2050613 2022-04-22T17:57:56.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 3350 completed, throughput=5MB/sec 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[45495]: cluster 2022-04-22T17:57:54.604038+0000 mgr.y (mgr.34107) 1138 : cluster [DBG] pgmap v1736: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[45495]: audit 2022-04-22T17:57:55.176571+0000 mon.a (mon.0) 2623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[45495]: audit 2022-04-22T17:57:55.537514+0000 mon.a (mon.0) 2624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[46934]: cluster 2022-04-22T17:57:54.604038+0000 mgr.y (mgr.34107) 1138 : cluster [DBG] pgmap v1736: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 3 op/s 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[46934]: audit 2022-04-22T17:57:55.176571+0000 mon.a (mon.0) 2623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:57:56.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:55 smithi115 bash[46934]: audit 2022-04-22T17:57:55.537514+0000 mon.a (mon.0) 2624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:57:57.252 INFO:tasks.workunit.client.0.smithi115.stdout: 657: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:57.253 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DBlG8Y5-7a57wVU off=16377220 len=2094143 2022-04-22T17:57:57.262 INFO:tasks.workunit.client.0.smithi115.stdout:op 3351 completed, throughput=5MB/sec 2022-04-22T17:57:57.267 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jzbmk99AHUUejnR off=1329893 len=2091700 2022-04-22T17:57:57.270 INFO:tasks.workunit.client.0.smithi115.stdout:op 3352 completed, throughput=5MB/sec 2022-04-22T17:57:57.270 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EQPe9mkt_FbxT0m off=298592 len=1039446 2022-04-22T17:57:57.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 3353 completed, throughput=5MB/sec 2022-04-22T17:57:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:57 smithi121 bash[36902]: cluster 2022-04-22T17:57:56.605162+0000 mgr.y (mgr.34107) 1139 : cluster [DBG] pgmap v1737: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:57:58.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:57 smithi121 bash[36902]: audit 2022-04-22T17:57:57.564054+0000 mon.a (mon.0) 2625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:58.304 INFO:tasks.workunit.client.0.smithi115.stdout: 658: throughput=4.99MB/sec pending data=0 2022-04-22T17:57:58.304 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MKocNpXwKse4Mfw off=7960205 len=781885 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3354 completed, throughput=4.99MB/sec 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wm_eJs9zDG7yUjH off=1090329 len=769744 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3355 completed, throughput=5MB/sec 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ybqfMkiSMUEXdbe off=2445385 len=1469570 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3356 completed, throughput=5MB/sec 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-abVl2x1iQrbDUBq off=4517931 len=1916948 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3357 completed, throughput=5MB/sec 2022-04-22T17:57:58.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UrDVS1yiK4REmAB off=1929291 len=1942477 2022-04-22T17:57:58.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:57 smithi115 bash[45495]: cluster 2022-04-22T17:57:56.605162+0000 mgr.y (mgr.34107) 1139 : cluster [DBG] pgmap v1737: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:57:58.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:57 smithi115 bash[45495]: audit 2022-04-22T17:57:57.564054+0000 mon.a (mon.0) 2625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:58.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:57 smithi115 bash[46934]: cluster 2022-04-22T17:57:56.605162+0000 mgr.y (mgr.34107) 1139 : cluster [DBG] pgmap v1737: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:57:58.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:57 smithi115 bash[46934]: audit 2022-04-22T17:57:57.564054+0000 mon.a (mon.0) 2625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:57:58.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 3358 completed, throughput=5MB/sec 2022-04-22T17:57:59.297 INFO:tasks.workunit.client.0.smithi115.stdout: 659: throughput=5MB/sec pending data=0 2022-04-22T17:57:59.297 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cVEIAxzRPJ4xOKt off=3298382 len=1130503 2022-04-22T17:57:59.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3359 completed, throughput=5MB/sec 2022-04-22T17:57:59.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NnKyArUowPSOTBh off=12437189 len=578593 2022-04-22T17:57:59.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3360 completed, throughput=5MB/sec 2022-04-22T17:57:59.306 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pBy-T3r8--px2BK off=9218698 len=1785468 2022-04-22T17:57:59.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 3361 completed, throughput=5MB/sec 2022-04-22T17:58:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:57:59 smithi121 bash[36902]: cluster 2022-04-22T17:57:58.606397+0000 mgr.y (mgr.34107) 1140 : cluster [DBG] pgmap v1738: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:57:59 smithi115 bash[45495]: cluster 2022-04-22T17:57:58.606397+0000 mgr.y (mgr.34107) 1140 : cluster [DBG] pgmap v1738: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:00.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:57:59 smithi115 bash[46934]: cluster 2022-04-22T17:57:58.606397+0000 mgr.y (mgr.34107) 1140 : cluster [DBG] pgmap v1738: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:00.317 INFO:tasks.workunit.client.0.smithi115.stdout: 660: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:00.317 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hudvZh-Wd5_QznA off=10204429 len=787089 2022-04-22T17:58:00.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 3362 completed, throughput=4.99MB/sec 2022-04-22T17:58:00.319 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EOF_onL9Bks85kB off=16408234 len=110545 2022-04-22T17:58:00.323 INFO:tasks.workunit.client.0.smithi115.stdout:op 3363 completed, throughput=4.99MB/sec 2022-04-22T17:58:00.323 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--TaXn3Jihr3ZJ4S off=2716229 len=934131 2022-04-22T17:58:00.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 3364 completed, throughput=5MB/sec 2022-04-22T17:58:00.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HjrxkQVilYwaJkY off=1254360 len=854165 2022-04-22T17:58:00.327 INFO:tasks.workunit.client.0.smithi115.stdout:op 3365 completed, throughput=5MB/sec 2022-04-22T17:58:00.327 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0eoz4XatIudiiEA off=12240843 len=75246 2022-04-22T17:58:00.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 3366 completed, throughput=5MB/sec 2022-04-22T17:58:00.331 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uXKM1FbsUCaOkEH off=6106945 len=1443870 2022-04-22T17:58:00.333 INFO:tasks.workunit.client.0.smithi115.stdout:op 3367 completed, throughput=5MB/sec 2022-04-22T17:58:00.333 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z6mjugT14A58Zwf off=4439476 len=1200369 2022-04-22T17:58:00.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 3368 completed, throughput=5MB/sec 2022-04-22T17:58:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:00 smithi121 bash[36902]: audit 2022-04-22T17:58:00.180022+0000 mon.a (mon.0) 2626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:00 smithi121 bash[36902]: audit 2022-04-22T17:58:00.541039+0000 mon.a (mon.0) 2627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:00 smithi115 bash[45495]: audit 2022-04-22T17:58:00.180022+0000 mon.a (mon.0) 2626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:00 smithi115 bash[45495]: audit 2022-04-22T17:58:00.541039+0000 mon.a (mon.0) 2627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:00 smithi115 bash[46934]: audit 2022-04-22T17:58:00.180022+0000 mon.a (mon.0) 2626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:00 smithi115 bash[46934]: audit 2022-04-22T17:58:00.541039+0000 mon.a (mon.0) 2627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:01.338 INFO:tasks.workunit.client.0.smithi115.stdout: 661: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:01.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Dpj58hcMp0Fh4f5 off=0 len=1459533 2022-04-22T17:58:01.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 3369 completed, throughput=4.99MB/sec 2022-04-22T17:58:01.347 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ffGdeKDP2yX5UYC off=6215458 len=1357479 2022-04-22T17:58:01.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 3370 completed, throughput=5MB/sec 2022-04-22T17:58:01.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fpIOAwbI_6ZSwNC off=16731651 len=303198 2022-04-22T17:58:01.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 3371 completed, throughput=5MB/sec 2022-04-22T17:58:01.350 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AQHcarVFQiWzTk7 off=0 len=1257230 2022-04-22T17:58:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 3372 completed, throughput=5MB/sec 2022-04-22T17:58:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kw4mY_OuCBXW4r4 off=5056814 len=701944 2022-04-22T17:58:01.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 3373 completed, throughput=5MB/sec 2022-04-22T17:58:01.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PdljHhQ3kQW7x5B off=12480748 len=1097031 2022-04-22T17:58:01.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 3374 completed, throughput=5MB/sec 2022-04-22T17:58:01.973 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:01.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:01 smithi121 bash[36902]: cluster 2022-04-22T17:58:00.607561+0000 mgr.y (mgr.34107) 1141 : cluster [DBG] pgmap v1739: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:02.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:01 smithi115 bash[45495]: cluster 2022-04-22T17:58:00.607561+0000 mgr.y (mgr.34107) 1141 : cluster [DBG] pgmap v1739: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:01 smithi115 bash[46934]: cluster 2022-04-22T17:58:00.607561+0000 mgr.y (mgr.34107) 1141 : cluster [DBG] pgmap v1739: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:02.360 INFO:tasks.workunit.client.0.smithi115.stdout: 662: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:02.361 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MXXr7VQFeASjss8 off=3193978 len=1676464 2022-04-22T17:58:02.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 3375 completed, throughput=5MB/sec 2022-04-22T17:58:02.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nKYhgydzamvJgvq off=8599258 len=137125 2022-04-22T17:58:02.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 3376 completed, throughput=5MB/sec 2022-04-22T17:58:02.366 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wej68jDXSxE0gUC off=2633467 len=1516770 2022-04-22T17:58:02.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 3377 completed, throughput=5MB/sec 2022-04-22T17:58:02.369 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9AZM8wPmEb7JSQ_ off=21132229 len=273782 2022-04-22T17:58:02.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 3378 completed, throughput=5MB/sec 2022-04-22T17:58:02.371 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aE3d7NrKctXOVjk off=5051390 len=693586 2022-04-22T17:58:02.377 INFO:tasks.workunit.client.0.smithi115.stdout:op 3379 completed, throughput=5MB/sec 2022-04-22T17:58:02.377 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iu8vOCHma0l2eVG off=3137770 len=787914 2022-04-22T17:58:02.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 3380 completed, throughput=5MB/sec 2022-04-22T17:58:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:02 smithi121 bash[36902]: audit 2022-04-22T17:58:02.567124+0000 mon.a (mon.0) 2628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:03.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:02 smithi115 bash[45495]: audit 2022-04-22T17:58:02.567124+0000 mon.a (mon.0) 2628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:02 smithi115 bash[46934]: audit 2022-04-22T17:58:02.567124+0000 mon.a (mon.0) 2628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:03.380 INFO:tasks.workunit.client.0.smithi115.stdout: 663: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:03.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jvxl4nHjmY4MjLo off=11492197 len=1750748 2022-04-22T17:58:03.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 3381 completed, throughput=5MB/sec 2022-04-22T17:58:03.387 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mrZIw5rhT7YZiQq off=768384 len=726204 2022-04-22T17:58:03.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 3382 completed, throughput=5MB/sec 2022-04-22T17:58:03.391 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8SkV3D6RMABiwnf off=434723 len=904180 2022-04-22T17:58:03.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 3383 completed, throughput=5MB/sec 2022-04-22T17:58:03.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X9M6FEXeSCd7xhx off=2329512 len=1394450 2022-04-22T17:58:03.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 3384 completed, throughput=5MB/sec 2022-04-22T17:58:03.988 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:03.598Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:03 smithi121 bash[36902]: cluster 2022-04-22T17:58:02.608625+0000 mgr.y (mgr.34107) 1142 : cluster [DBG] pgmap v1740: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:04.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:03 smithi115 bash[45495]: cluster 2022-04-22T17:58:02.608625+0000 mgr.y (mgr.34107) 1142 : cluster [DBG] pgmap v1740: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:04.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:03 smithi115 bash[46934]: cluster 2022-04-22T17:58:02.608625+0000 mgr.y (mgr.34107) 1142 : cluster [DBG] pgmap v1740: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:04.405 INFO:tasks.workunit.client.0.smithi115.stdout: 664: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:04.406 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-esKOpGnA3tivN8d off=832876 len=891222 2022-04-22T17:58:04.409 INFO:tasks.workunit.client.0.smithi115.stdout:op 3385 completed, throughput=4.99MB/sec 2022-04-22T17:58:04.409 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gdijSzFZWKgE5hD off=825487 len=1443853 2022-04-22T17:58:04.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 3386 completed, throughput=5MB/sec 2022-04-22T17:58:04.419 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pE9y-j_OlOTG7Qg off=2737325 len=965210 2022-04-22T17:58:04.425 INFO:tasks.workunit.client.0.smithi115.stdout:op 3387 completed, throughput=5MB/sec 2022-04-22T17:58:04.426 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DMkUeThWu2ohP6w off=6274161 len=606631 2022-04-22T17:58:04.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 3388 completed, throughput=5MB/sec 2022-04-22T17:58:04.427 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nqni2cSykGvCioE off=7631712 len=12128 2022-04-22T17:58:04.427 INFO:tasks.workunit.client.0.smithi115.stdout:op 3389 completed, throughput=5MB/sec 2022-04-22T17:58:04.427 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2kTyBKYZSBFzKQP off=7052422 len=920867 2022-04-22T17:58:04.429 INFO:tasks.workunit.client.0.smithi115.stdout:op 3390 completed, throughput=5MB/sec 2022-04-22T17:58:04.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tgYwgm1Ly55kC0A off=2329317 len=767411 2022-04-22T17:58:04.434 INFO:tasks.workunit.client.0.smithi115.stdout:op 3391 completed, throughput=5MB/sec 2022-04-22T17:58:05.435 INFO:tasks.workunit.client.0.smithi115.stdout: 665: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:05.435 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GSfWNiFkBXEF7dS off=847681 len=1607683 2022-04-22T17:58:05.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 3392 completed, throughput=5MB/sec 2022-04-22T17:58:05.440 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-21BS6zkKGCF_SFC off=8642301 len=609746 2022-04-22T17:58:05.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 3393 completed, throughput=5MB/sec 2022-04-22T17:58:05.446 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UAPJmpvTFI43B1C off=691963 len=1751920 2022-04-22T17:58:05.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 3394 completed, throughput=5MB/sec 2022-04-22T17:58:05.449 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3NZyMmWuXozwmp5 off=10279073 len=781976 2022-04-22T17:58:05.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 3395 completed, throughput=5MB/sec 2022-04-22T17:58:05.452 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JB5yHP8wDBi528q off=1233797 len=1912 2022-04-22T17:58:05.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 3396 completed, throughput=5MB/sec 2022-04-22T17:58:05.455 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jX9bjCj9KJX3fD- off=4388685 len=1739979 2022-04-22T17:58:05.457 INFO:tasks.workunit.client.0.smithi115.stdout:op 3397 completed, throughput=5MB/sec 2022-04-22T17:58:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:05 smithi121 bash[36902]: cluster 2022-04-22T17:58:04.609684+0000 mgr.y (mgr.34107) 1143 : cluster [DBG] pgmap v1741: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:58:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:05 smithi121 bash[36902]: audit 2022-04-22T17:58:05.183225+0000 mon.a (mon.0) 2629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:05 smithi121 bash[36902]: audit 2022-04-22T17:58:05.544577+0000 mon.a (mon.0) 2630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[45495]: cluster 2022-04-22T17:58:04.609684+0000 mgr.y (mgr.34107) 1143 : cluster [DBG] pgmap v1741: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:58:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[45495]: audit 2022-04-22T17:58:05.183225+0000 mon.a (mon.0) 2629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[45495]: audit 2022-04-22T17:58:05.544577+0000 mon.a (mon.0) 2630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[46934]: cluster 2022-04-22T17:58:04.609684+0000 mgr.y (mgr.34107) 1143 : cluster [DBG] pgmap v1741: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T17:58:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[46934]: audit 2022-04-22T17:58:05.183225+0000 mon.a (mon.0) 2629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:05 smithi115 bash[46934]: audit 2022-04-22T17:58:05.544577+0000 mon.a (mon.0) 2630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:06.459 INFO:tasks.workunit.client.0.smithi115.stdout: 666: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:06.459 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xtoIqo3vc1YE50z off=8534717 len=1631310 2022-04-22T17:58:06.461 INFO:tasks.workunit.client.0.smithi115.stdout:op 3398 completed, throughput=5MB/sec 2022-04-22T17:58:06.461 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0jF9dpRCvdCebri off=10910817 len=603669 2022-04-22T17:58:06.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 3399 completed, throughput=5MB/sec 2022-04-22T17:58:06.463 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-87_fL0Mn7tnhhGW off=12318310 len=1119769 2022-04-22T17:58:06.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 3400 completed, throughput=5MB/sec 2022-04-22T17:58:06.466 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iU25Bb1AjySU8Oy off=4433979 len=491725 2022-04-22T17:58:06.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 3401 completed, throughput=5MB/sec 2022-04-22T17:58:06.472 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xcvj9y4vn4PqWap off=624184 len=1005761 2022-04-22T17:58:06.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 3402 completed, throughput=5MB/sec 2022-04-22T17:58:07.479 INFO:tasks.workunit.client.0.smithi115.stdout: 667: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:07.479 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qxP8YhQFG297l5F off=1547390 len=1929268 2022-04-22T17:58:07.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 3403 completed, throughput=5MB/sec 2022-04-22T17:58:07.489 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XUAL3D4reYmE3nW off=3935072 len=912199 2022-04-22T17:58:07.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 3404 completed, throughput=5MB/sec 2022-04-22T17:58:07.492 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_JPFktm4OpKSdaK off=740838 len=1003382 2022-04-22T17:58:07.495 INFO:tasks.workunit.client.0.smithi115.stdout:op 3405 completed, throughput=5MB/sec 2022-04-22T17:58:07.495 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tEOGKRhl5OzY_yR off=6946324 len=2051202 2022-04-22T17:58:07.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 3406 completed, throughput=5MB/sec 2022-04-22T17:58:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:07 smithi121 bash[36902]: cluster 2022-04-22T17:58:06.610703+0000 mgr.y (mgr.34107) 1144 : cluster [DBG] pgmap v1742: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:07 smithi121 bash[36902]: audit 2022-04-22T17:58:07.570351+0000 mon.a (mon.0) 2631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:07 smithi115 bash[45495]: cluster 2022-04-22T17:58:06.610703+0000 mgr.y (mgr.34107) 1144 : cluster [DBG] pgmap v1742: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:07 smithi115 bash[45495]: audit 2022-04-22T17:58:07.570351+0000 mon.a (mon.0) 2631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:07 smithi115 bash[46934]: cluster 2022-04-22T17:58:06.610703+0000 mgr.y (mgr.34107) 1144 : cluster [DBG] pgmap v1742: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:07 smithi115 bash[46934]: audit 2022-04-22T17:58:07.570351+0000 mon.a (mon.0) 2631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:08.502 INFO:tasks.workunit.client.0.smithi115.stdout: 668: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:08.502 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tcNETZWp8rGM4M2 off=10689929 len=1037747 2022-04-22T17:58:08.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 3407 completed, throughput=5MB/sec 2022-04-22T17:58:08.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fJgEp-nOXGxQTEr off=1563446 len=1430881 2022-04-22T17:58:08.513 INFO:tasks.workunit.client.0.smithi115.stdout:op 3408 completed, throughput=5MB/sec 2022-04-22T17:58:08.513 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E8VIoSkEWfFiAEa off=2123590 len=1333349 2022-04-22T17:58:08.515 INFO:tasks.workunit.client.0.smithi115.stdout:op 3409 completed, throughput=5MB/sec 2022-04-22T17:58:08.515 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AC2DXija5-O0ced off=2447824 len=591488 2022-04-22T17:58:08.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 3410 completed, throughput=5MB/sec 2022-04-22T17:58:09.518 INFO:tasks.workunit.client.0.smithi115.stdout: 669: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:09.519 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l4YGTn_aoar-aDY off=7372105 len=1672143 2022-04-22T17:58:09.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 3411 completed, throughput=5MB/sec 2022-04-22T17:58:09.520 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YDSKG0vY01Xw4SM off=5351724 len=1921060 2022-04-22T17:58:09.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 3412 completed, throughput=5MB/sec 2022-04-22T17:58:09.525 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hLhbFW4E0kCahs9 off=16751747 len=770775 2022-04-22T17:58:09.531 INFO:tasks.workunit.client.0.smithi115.stdout:op 3413 completed, throughput=5MB/sec 2022-04-22T17:58:09.532 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sQu5VpLwEuliB23 off=1735758 len=1617881 2022-04-22T17:58:09.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 3414 completed, throughput=5MB/sec 2022-04-22T17:58:10.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:09 smithi121 bash[36902]: cluster 2022-04-22T17:58:08.611938+0000 mgr.y (mgr.34107) 1145 : cluster [DBG] pgmap v1743: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:10.386 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:09 smithi115 bash[45495]: cluster 2022-04-22T17:58:08.611938+0000 mgr.y (mgr.34107) 1145 : cluster [DBG] pgmap v1743: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:10.386 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:09 smithi115 bash[46934]: cluster 2022-04-22T17:58:08.611938+0000 mgr.y (mgr.34107) 1145 : cluster [DBG] pgmap v1743: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:10.537 INFO:tasks.workunit.client.0.smithi115.stdout: 670: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:10.538 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2aBWEj3GubUOyS1 off=21093513 len=1338885 2022-04-22T17:58:10.547 INFO:tasks.workunit.client.0.smithi115.stdout:op 3415 completed, throughput=5MB/sec 2022-04-22T17:58:10.547 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hR-md27aYiR5KC4 off=5129260 len=456869 2022-04-22T17:58:10.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 3416 completed, throughput=5MB/sec 2022-04-22T17:58:10.550 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UBJ9P4cM_zQKHyc off=5795206 len=943917 2022-04-22T17:58:10.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 3417 completed, throughput=5MB/sec 2022-04-22T17:58:10.558 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qqllc4mbL87ltg8 off=8051581 len=1169310 2022-04-22T17:58:10.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 3418 completed, throughput=5MB/sec 2022-04-22T17:58:10.559 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CjbhyH3WW5mmnkB off=1116906 len=1653874 2022-04-22T17:58:10.566 INFO:tasks.workunit.client.0.smithi115.stdout:op 3419 completed, throughput=5MB/sec 2022-04-22T17:58:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:11 smithi121 bash[36902]: audit 2022-04-22T17:58:10.186602+0000 mon.a (mon.0) 2632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:11 smithi121 bash[36902]: audit 2022-04-22T17:58:10.547974+0000 mon.a (mon.0) 2633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:11 smithi115 bash[45495]: audit 2022-04-22T17:58:10.186602+0000 mon.a (mon.0) 2632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:11 smithi115 bash[45495]: audit 2022-04-22T17:58:10.547974+0000 mon.a (mon.0) 2633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:11 smithi115 bash[46934]: audit 2022-04-22T17:58:10.186602+0000 mon.a (mon.0) 2632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:11 smithi115 bash[46934]: audit 2022-04-22T17:58:10.547974+0000 mon.a (mon.0) 2633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:11.566 INFO:tasks.workunit.client.0.smithi115.stdout: 671: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:11.567 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qtETjtOdkoMZRR5 off=8371556 len=1248679 2022-04-22T17:58:11.574 INFO:tasks.workunit.client.0.smithi115.stdout:op 3420 completed, throughput=5MB/sec 2022-04-22T17:58:11.575 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hj7M5Uz9dbRcn80 off=8628953 len=557784 2022-04-22T17:58:11.580 INFO:tasks.workunit.client.0.smithi115.stdout:op 3421 completed, throughput=5MB/sec 2022-04-22T17:58:11.580 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PIHl_8mwukAzELM off=2548103 len=1932319 2022-04-22T17:58:11.589 INFO:tasks.workunit.client.0.smithi115.stdout:op 3422 completed, throughput=5MB/sec 2022-04-22T17:58:11.589 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FgR-Axf2LI5ECcp off=12443623 len=1785918 2022-04-22T17:58:11.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 3423 completed, throughput=5MB/sec 2022-04-22T17:58:12.011 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:11.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:12.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:12 smithi121 bash[36902]: cluster 2022-04-22T17:58:10.612938+0000 mgr.y (mgr.34107) 1146 : cluster [DBG] pgmap v1744: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T17:58:12.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:12 smithi115 bash[45495]: cluster 2022-04-22T17:58:10.612938+0000 mgr.y (mgr.34107) 1146 : cluster [DBG] pgmap v1744: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T17:58:12.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:12 smithi115 bash[46934]: cluster 2022-04-22T17:58:10.612938+0000 mgr.y (mgr.34107) 1146 : cluster [DBG] pgmap v1744: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T17:58:12.595 INFO:tasks.workunit.client.0.smithi115.stdout: 672: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:12.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Gy0Nikg7mjfDOMC off=16538891 len=1899021 2022-04-22T17:58:12.605 INFO:tasks.workunit.client.0.smithi115.stdout:op 3424 completed, throughput=5MB/sec 2022-04-22T17:58:12.606 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tpplac2MsJEnSCU off=9815296 len=1713668 2022-04-22T17:58:12.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 3425 completed, throughput=5MB/sec 2022-04-22T17:58:12.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nr6ZegyFUDS8f4A off=3416506 len=1129592 2022-04-22T17:58:12.617 INFO:tasks.workunit.client.0.smithi115.stdout:op 3426 completed, throughput=5MB/sec 2022-04-22T17:58:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:13 smithi115 bash[45495]: audit 2022-04-22T17:58:12.573399+0000 mon.a (mon.0) 2634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:13 smithi115 bash[46934]: audit 2022-04-22T17:58:12.573399+0000 mon.a (mon.0) 2634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:13 smithi121 bash[36902]: audit 2022-04-22T17:58:12.573399+0000 mon.a (mon.0) 2634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:13.617 INFO:tasks.workunit.client.0.smithi115.stdout: 673: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:13.618 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-i1gT8OYhl4CfkJw off=1044341 len=1865661 2022-04-22T17:58:13.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 3427 completed, throughput=5MB/sec 2022-04-22T17:58:13.629 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tvc7kg6CyqdmCRx off=4673844 len=768224 2022-04-22T17:58:13.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 3428 completed, throughput=5MB/sec 2022-04-22T17:58:13.635 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pbLfW_Ap9ki-8O5 off=9047727 len=2041272 2022-04-22T17:58:13.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 3429 completed, throughput=5MB/sec 2022-04-22T17:58:14.022 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:13.597Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[45495]: cluster 2022-04-22T17:58:12.614042+0000 mgr.y (mgr.34107) 1147 : cluster [DBG] pgmap v1745: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[45495]: audit 2022-04-22T17:58:13.564297+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:58:14.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[45495]: audit 2022-04-22T17:58:13.564698+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:58:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[46934]: cluster 2022-04-22T17:58:12.614042+0000 mgr.y (mgr.34107) 1147 : cluster [DBG] pgmap v1745: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[46934]: audit 2022-04-22T17:58:13.564297+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:58:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:14 smithi115 bash[46934]: audit 2022-04-22T17:58:13.564698+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:58:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:14 smithi121 bash[36902]: cluster 2022-04-22T17:58:12.614042+0000 mgr.y (mgr.34107) 1147 : cluster [DBG] pgmap v1745: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:14 smithi121 bash[36902]: audit 2022-04-22T17:58:13.564297+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:58:14.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:14 smithi121 bash[36902]: audit 2022-04-22T17:58:13.564698+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:58:14.639 INFO:tasks.workunit.client.0.smithi115.stdout: 674: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:14.639 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VnD0Y_jAEkHTD0F off=2116534 len=926031 2022-04-22T17:58:14.647 INFO:tasks.workunit.client.0.smithi115.stdout:op 3430 completed, throughput=4.99MB/sec 2022-04-22T17:58:14.647 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ArgpYbwvkl-vMwm off=5284143 len=1075929 2022-04-22T17:58:14.653 INFO:tasks.workunit.client.0.smithi115.stdout:op 3431 completed, throughput=5MB/sec 2022-04-22T17:58:14.653 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gii9QehJ6hmD51J off=10170476 len=384275 2022-04-22T17:58:14.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 3432 completed, throughput=5MB/sec 2022-04-22T17:58:14.654 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2E3ytdisD_SLiZ4 off=5816888 len=412885 2022-04-22T17:58:14.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 3433 completed, throughput=5MB/sec 2022-04-22T17:58:14.658 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p1wNMF-9R2TvVqi off=2038753 len=1321023 2022-04-22T17:58:14.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 3434 completed, throughput=5MB/sec 2022-04-22T17:58:14.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JZieRc33YcxRZ1n off=11655946 len=826011 2022-04-22T17:58:14.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 3435 completed, throughput=5MB/sec 2022-04-22T17:58:14.662 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g_OLUELbMEZ5kn- off=3974513 len=1775396 2022-04-22T17:58:14.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 3436 completed, throughput=5MB/sec 2022-04-22T17:58:15.665 INFO:tasks.workunit.client.0.smithi115.stdout: 675: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:15.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AR_YZGCDlFePr_9 off=9991943 len=613365 2022-04-22T17:58:15.667 INFO:tasks.workunit.client.0.smithi115.stdout:op 3437 completed, throughput=5MB/sec 2022-04-22T17:58:15.667 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZP0huJyEGbMjxRN off=434873 len=1165084 2022-04-22T17:58:15.670 INFO:tasks.workunit.client.0.smithi115.stdout:op 3438 completed, throughput=5MB/sec 2022-04-22T17:58:15.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sX80m9HTq9wOaEs off=2055307 len=1828043 2022-04-22T17:58:15.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 3439 completed, throughput=5MB/sec 2022-04-22T17:58:15.681 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8acmrBIzvozvXWw off=9217602 len=5960 2022-04-22T17:58:15.681 INFO:tasks.workunit.client.0.smithi115.stdout:op 3440 completed, throughput=5MB/sec 2022-04-22T17:58:15.682 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QMoQg1nvULzpb9J off=14300081 len=2010563 2022-04-22T17:58:15.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 3441 completed, throughput=5MB/sec 2022-04-22T17:58:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[45495]: cluster 2022-04-22T17:58:14.615160+0000 mgr.y (mgr.34107) 1148 : cluster [DBG] pgmap v1746: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:58:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[45495]: audit 2022-04-22T17:58:15.189726+0000 mon.a (mon.0) 2637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[45495]: audit 2022-04-22T17:58:15.551417+0000 mon.a (mon.0) 2638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[46934]: cluster 2022-04-22T17:58:14.615160+0000 mgr.y (mgr.34107) 1148 : cluster [DBG] pgmap v1746: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:58:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[46934]: audit 2022-04-22T17:58:15.189726+0000 mon.a (mon.0) 2637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:16 smithi115 bash[46934]: audit 2022-04-22T17:58:15.551417+0000 mon.a (mon.0) 2638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:16 smithi121 bash[36902]: cluster 2022-04-22T17:58:14.615160+0000 mgr.y (mgr.34107) 1148 : cluster [DBG] pgmap v1746: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T17:58:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:16 smithi121 bash[36902]: audit 2022-04-22T17:58:15.189726+0000 mon.a (mon.0) 2637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:16 smithi121 bash[36902]: audit 2022-04-22T17:58:15.551417+0000 mon.a (mon.0) 2638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:16.562 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:58:16.685 INFO:tasks.workunit.client.0.smithi115.stdout: 676: throughput=5MB/sec pending data=0 2022-04-22T17:58:16.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kL-tSdgxHcC3VYp off=14291890 len=975999 2022-04-22T17:58:16.691 INFO:tasks.workunit.client.0.smithi115.stdout:op 3442 completed, throughput=5MB/sec 2022-04-22T17:58:16.691 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-J0dPxwIAKsmNpg6 off=22485993 len=507249 2022-04-22T17:58:16.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 3443 completed, throughput=5MB/sec 2022-04-22T17:58:16.695 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rs66Cl5kS0UVhhy off=8615131 len=1988304 2022-04-22T17:58:16.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 3444 completed, throughput=5MB/sec 2022-04-22T17:58:16.698 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KSex5l-TallVYe- off=3485351 len=931769 2022-04-22T17:58:16.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 3445 completed, throughput=5MB/sec 2022-04-22T17:58:16.908 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:58:16.981 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (21m) 27s ago 35m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T17:58:17.005 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (34m) 92s ago 34m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:58:17.005 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (23m) 27s ago 36m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:58:17.005 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (22m) 92s ago 36m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:58:17.005 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (30m) 92s ago 40m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:58:17.005 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (31m) 27s ago 42m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (30m) 27s ago 42m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (29m) 92s ago 40m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (29m) 27s ago 40m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (22m) 27s ago 35m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (22m) 92s ago 35m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:58:17.006 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (28m) 27s ago 40m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:58:17.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (27m) 27s ago 39m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:58:17.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (26m) 27s ago 39m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:58:17.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (26m) 27s ago 38m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:58:17.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (25m) 92s ago 38m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:58:17.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (25m) 92s ago 37m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:58:17.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (24m) 92s ago 37m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:58:17.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (23m) 92s ago 36m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:58:17.008 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (22m) 92s ago 35m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:58:17.313 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:58:17.313 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:58:17.314 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:58:17.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:17.314 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:58:17.314 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:58:17.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:17.315 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:58:17.315 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:58:17.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:17.316 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:58:17.316 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:58:17.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:17.316 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:58:17.316 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:58:17.317 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:58:17.317 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:58:17.700 INFO:tasks.workunit.client.0.smithi115.stdout: 677: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:17.700 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cKed-h-D7vIEI4r off=2487301 len=1155322 2022-04-22T17:58:17.703 INFO:tasks.workunit.client.0.smithi115.stdout:op 3446 completed, throughput=5MB/sec 2022-04-22T17:58:17.703 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bdWD6X2cDNToNZh off=1837684 len=1470082 2022-04-22T17:58:17.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 3447 completed, throughput=5MB/sec 2022-04-22T17:58:17.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2cc3ip0FnxHVCsm off=10410587 len=689753 2022-04-22T17:58:17.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 3448 completed, throughput=5MB/sec 2022-04-22T17:58:17.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BwKJoKbYZJG5KFQ off=5774142 len=1437265 2022-04-22T17:58:17.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 3449 completed, throughput=5MB/sec 2022-04-22T17:58:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[45495]: audit 2022-04-22T17:58:16.538420+0000 mgr.y (mgr.34107) 1149 : audit [DBG] from='client.45547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[45495]: cluster 2022-04-22T17:58:16.616240+0000 mgr.y (mgr.34107) 1150 : cluster [DBG] pgmap v1747: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[45495]: audit 2022-04-22T17:58:16.881798+0000 mgr.y (mgr.34107) 1151 : audit [DBG] from='client.45553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[45495]: audit 2022-04-22T17:58:17.306339+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? 172.21.15.115:0/1264939122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[45495]: audit 2022-04-22T17:58:17.576544+0000 mon.a (mon.0) 2640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[46934]: audit 2022-04-22T17:58:16.538420+0000 mgr.y (mgr.34107) 1149 : audit [DBG] from='client.45547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[46934]: cluster 2022-04-22T17:58:16.616240+0000 mgr.y (mgr.34107) 1150 : cluster [DBG] pgmap v1747: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[46934]: audit 2022-04-22T17:58:16.881798+0000 mgr.y (mgr.34107) 1151 : audit [DBG] from='client.45553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[46934]: audit 2022-04-22T17:58:17.306339+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? 172.21.15.115:0/1264939122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:18 smithi115 bash[46934]: audit 2022-04-22T17:58:17.576544+0000 mon.a (mon.0) 2640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:18 smithi121 bash[36902]: audit 2022-04-22T17:58:16.538420+0000 mgr.y (mgr.34107) 1149 : audit [DBG] from='client.45547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:18 smithi121 bash[36902]: cluster 2022-04-22T17:58:16.616240+0000 mgr.y (mgr.34107) 1150 : cluster [DBG] pgmap v1747: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:18.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:18 smithi121 bash[36902]: audit 2022-04-22T17:58:16.881798+0000 mgr.y (mgr.34107) 1151 : audit [DBG] from='client.45553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:18.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:18 smithi121 bash[36902]: audit 2022-04-22T17:58:17.306339+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? 172.21.15.115:0/1264939122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:18.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:18 smithi121 bash[36902]: audit 2022-04-22T17:58:17.576544+0000 mon.a (mon.0) 2640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:18.720 INFO:tasks.workunit.client.0.smithi115.stdout: 678: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:18.720 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EYlEvhnM9aZMSou off=3178188 len=1291148 2022-04-22T17:58:18.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 3450 completed, throughput=4.99MB/sec 2022-04-22T17:58:18.728 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h_OD9_b4qxs0lGj off=8407518 len=2082421 2022-04-22T17:58:18.732 INFO:tasks.workunit.client.0.smithi115.stdout:op 3451 completed, throughput=5MB/sec 2022-04-22T17:58:18.732 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z_wDBSqW0i1XMGs off=10637603 len=1364679 2022-04-22T17:58:18.734 INFO:tasks.workunit.client.0.smithi115.stdout:op 3452 completed, throughput=5MB/sec 2022-04-22T17:58:18.735 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HDMp0DHYtxk4GNb off=8468385 len=1247501 2022-04-22T17:58:18.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 3453 completed, throughput=5MB/sec 2022-04-22T17:58:19.741 INFO:tasks.workunit.client.0.smithi115.stdout: 679: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:19.741 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-niAmAXv1VmGjJE9 off=13378058 len=1535285 2022-04-22T17:58:19.749 INFO:tasks.workunit.client.0.smithi115.stdout:op 3454 completed, throughput=5MB/sec 2022-04-22T17:58:19.750 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k7fXIUx2k-d-E8k off=1082074 len=1291168 2022-04-22T17:58:19.757 INFO:tasks.workunit.client.0.smithi115.stdout:op 3455 completed, throughput=5MB/sec 2022-04-22T17:58:19.757 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s44PSll15TQ_bTV off=17057317 len=417396 2022-04-22T17:58:19.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 3456 completed, throughput=5MB/sec 2022-04-22T17:58:19.763 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tnswOSOjS4hG5El off=6325333 len=1454111 2022-04-22T17:58:19.769 INFO:tasks.workunit.client.0.smithi115.stdout:op 3457 completed, throughput=5MB/sec 2022-04-22T17:58:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:20 smithi115 bash[45495]: cluster 2022-04-22T17:58:18.617357+0000 mgr.y (mgr.34107) 1152 : cluster [DBG] pgmap v1748: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:20.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:20 smithi115 bash[46934]: cluster 2022-04-22T17:58:18.617357+0000 mgr.y (mgr.34107) 1152 : cluster [DBG] pgmap v1748: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:20 smithi121 bash[36902]: cluster 2022-04-22T17:58:18.617357+0000 mgr.y (mgr.34107) 1152 : cluster [DBG] pgmap v1748: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:20.769 INFO:tasks.workunit.client.0.smithi115.stdout: 680: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:20.769 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-asg8vLORVfZE-1X off=5513713 len=366153 2022-04-22T17:58:20.770 INFO:tasks.workunit.client.0.smithi115.stdout:op 3458 completed, throughput=4.99MB/sec 2022-04-22T17:58:20.771 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SqneHdWlvXgwTm6 off=1900312 len=379942 2022-04-22T17:58:20.772 INFO:tasks.workunit.client.0.smithi115.stdout:op 3459 completed, throughput=4.99MB/sec 2022-04-22T17:58:20.772 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WmdXtuARZQY6qnn off=10485234 len=449056 2022-04-22T17:58:20.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 3460 completed, throughput=4.99MB/sec 2022-04-22T17:58:20.779 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bxVAW_nnnS2-pMw off=8474322 len=1802623 2022-04-22T17:58:20.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 3461 completed, throughput=5MB/sec 2022-04-22T17:58:20.783 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CyV5mKEO0uwPxo6 off=7103203 len=1944521 2022-04-22T17:58:20.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 3462 completed, throughput=5MB/sec 2022-04-22T17:58:20.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6pM7Y2Za8ffeVxC off=489366 len=1154192 2022-04-22T17:58:20.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 3463 completed, throughput=5MB/sec 2022-04-22T17:58:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:21 smithi121 bash[36902]: audit 2022-04-22T17:58:20.192744+0000 mon.a (mon.0) 2641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:21 smithi121 bash[36902]: audit 2022-04-22T17:58:20.555182+0000 mon.a (mon.0) 2642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:21 smithi115 bash[45495]: audit 2022-04-22T17:58:20.192744+0000 mon.a (mon.0) 2641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:21 smithi115 bash[45495]: audit 2022-04-22T17:58:20.555182+0000 mon.a (mon.0) 2642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:21 smithi115 bash[46934]: audit 2022-04-22T17:58:20.192744+0000 mon.a (mon.0) 2641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:21 smithi115 bash[46934]: audit 2022-04-22T17:58:20.555182+0000 mon.a (mon.0) 2642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:21.792 INFO:tasks.workunit.client.0.smithi115.stdout: 681: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:21.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7nvfjyMRZWvPwtQ off=12638469 len=1973133 2022-04-22T17:58:21.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 3464 completed, throughput=5MB/sec 2022-04-22T17:58:21.798 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tkER4U1zJ5SfvZ- off=761153 len=769292 2022-04-22T17:58:21.805 INFO:tasks.workunit.client.0.smithi115.stdout:op 3465 completed, throughput=5MB/sec 2022-04-22T17:58:21.805 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7RcRlCXUg-YFMW7 off=6392684 len=766581 2022-04-22T17:58:21.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 3466 completed, throughput=5MB/sec 2022-04-22T17:58:21.807 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ve6Keml1E6H-xV3 off=3645743 len=1459236 2022-04-22T17:58:21.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 3467 completed, throughput=5MB/sec 2022-04-22T17:58:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:21.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:22 smithi121 bash[36902]: cluster 2022-04-22T17:58:20.618443+0000 mgr.y (mgr.34107) 1153 : cluster [DBG] pgmap v1749: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:22.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:22 smithi115 bash[45495]: cluster 2022-04-22T17:58:20.618443+0000 mgr.y (mgr.34107) 1153 : cluster [DBG] pgmap v1749: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:22 smithi115 bash[46934]: cluster 2022-04-22T17:58:20.618443+0000 mgr.y (mgr.34107) 1153 : cluster [DBG] pgmap v1749: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:58:22.816 INFO:tasks.workunit.client.0.smithi115.stdout: 682: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:22.816 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rXlcgpgwMiF08gN off=18237686 len=1263632 2022-04-22T17:58:22.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 3468 completed, throughput=5MB/sec 2022-04-22T17:58:22.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4TR_JIQa8TGOE7M off=6598097 len=838246 2022-04-22T17:58:22.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 3469 completed, throughput=5MB/sec 2022-04-22T17:58:22.825 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Rx7m9jr8O674Ocd off=572632 len=1423631 2022-04-22T17:58:22.833 INFO:tasks.workunit.client.0.smithi115.stdout:op 3470 completed, throughput=5MB/sec 2022-04-22T17:58:22.833 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lnjDO2CWK22bYOx off=5622734 len=67483 2022-04-22T17:58:22.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 3471 completed, throughput=5MB/sec 2022-04-22T17:58:22.836 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PTWVVRqrevZcqdi off=4378594 len=1842834 2022-04-22T17:58:22.838 INFO:tasks.workunit.client.0.smithi115.stdout:op 3472 completed, throughput=5MB/sec 2022-04-22T17:58:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:23 smithi121 bash[36902]: audit 2022-04-22T17:58:22.579832+0000 mon.a (mon.0) 2643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:23 smithi115 bash[45495]: audit 2022-04-22T17:58:22.579832+0000 mon.a (mon.0) 2643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:23 smithi115 bash[46934]: audit 2022-04-22T17:58:22.579832+0000 mon.a (mon.0) 2643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:23.838 INFO:tasks.workunit.client.0.smithi115.stdout: 683: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:23.838 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Clza1DY1LnWyt6t off=13588085 len=12575 2022-04-22T17:58:23.838 INFO:tasks.workunit.client.0.smithi115.stdout:op 3473 completed, throughput=4.99MB/sec 2022-04-22T17:58:23.839 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-joh29bhy8ySjjD3 off=6388825 len=402488 2022-04-22T17:58:23.840 INFO:tasks.workunit.client.0.smithi115.stdout:op 3474 completed, throughput=4.99MB/sec 2022-04-22T17:58:23.840 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sbgBTt-DYfYE_1V off=9311677 len=238032 2022-04-22T17:58:23.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 3475 completed, throughput=4.99MB/sec 2022-04-22T17:58:23.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JjgTC_RYQt54a2D off=4560663 len=2095091 2022-04-22T17:58:23.849 INFO:tasks.workunit.client.0.smithi115.stdout:op 3476 completed, throughput=5MB/sec 2022-04-22T17:58:23.849 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RHoQ1374wmz8kXh off=4066804 len=558875 2022-04-22T17:58:23.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 3477 completed, throughput=5MB/sec 2022-04-22T17:58:23.855 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-agzOkizUuDtP3QX off=9381948 len=2046717 2022-04-22T17:58:23.861 INFO:tasks.workunit.client.0.smithi115.stdout:op 3478 completed, throughput=5MB/sec 2022-04-22T17:58:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:23.599Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:24 smithi121 bash[36902]: cluster 2022-04-22T17:58:22.619511+0000 mgr.y (mgr.34107) 1154 : cluster [DBG] pgmap v1750: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:58:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:24 smithi115 bash[45495]: cluster 2022-04-22T17:58:22.619511+0000 mgr.y (mgr.34107) 1154 : cluster [DBG] pgmap v1750: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:58:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:24 smithi115 bash[46934]: cluster 2022-04-22T17:58:22.619511+0000 mgr.y (mgr.34107) 1154 : cluster [DBG] pgmap v1750: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:58:24.861 INFO:tasks.workunit.client.0.smithi115.stdout: 684: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:24.862 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xAn4O1ITqcIZVN8 off=12558415 len=2006942 2022-04-22T17:58:24.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 3479 completed, throughput=5MB/sec 2022-04-22T17:58:24.867 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9pjvV2Tz4P5_XoZ off=4064650 len=401792 2022-04-22T17:58:24.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 3480 completed, throughput=5MB/sec 2022-04-22T17:58:24.869 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BJYgAQ9GxLOCd7V off=3424731 len=501449 2022-04-22T17:58:24.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 3481 completed, throughput=5MB/sec 2022-04-22T17:58:24.875 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0HMyWR3M2IBT7XS off=16408743 len=368711 2022-04-22T17:58:24.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 3482 completed, throughput=5MB/sec 2022-04-22T17:58:24.879 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yttMuEQVSjfqrT5 off=9248165 len=1323127 2022-04-22T17:58:24.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 3483 completed, throughput=5MB/sec 2022-04-22T17:58:24.888 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hQ7ct3iLxkwR1HZ off=16800396 len=893030 2022-04-22T17:58:24.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 3484 completed, throughput=5MB/sec 2022-04-22T17:58:25.893 INFO:tasks.workunit.client.0.smithi115.stdout: 685: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:25.894 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ffGdeKDP2yX5UYC off=1532782 len=1816950 2022-04-22T17:58:25.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 3485 completed, throughput=5MB/sec 2022-04-22T17:58:25.902 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xbNPI-Zm7e0KMQ7 off=3405176 len=340565 2022-04-22T17:58:25.904 INFO:tasks.workunit.client.0.smithi115.stdout:op 3486 completed, throughput=5MB/sec 2022-04-22T17:58:25.904 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XLYkHo0SLEJ404h off=12603700 len=464654 2022-04-22T17:58:25.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 3487 completed, throughput=5MB/sec 2022-04-22T17:58:25.909 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xhj4ecMX_6e4tMs off=3590076 len=929968 2022-04-22T17:58:25.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 3488 completed, throughput=5MB/sec 2022-04-22T17:58:25.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aQ1A_hxTJjMeKtP off=4473362 len=1905706 2022-04-22T17:58:25.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 3489 completed, throughput=5MB/sec 2022-04-22T17:58:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:26 smithi121 bash[36902]: cluster 2022-04-22T17:58:24.620576+0000 mgr.y (mgr.34107) 1155 : cluster [DBG] pgmap v1751: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:58:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:26 smithi121 bash[36902]: audit 2022-04-22T17:58:25.195986+0000 mon.a (mon.0) 2644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:26 smithi121 bash[36902]: audit 2022-04-22T17:58:25.558898+0000 mon.a (mon.0) 2645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[45495]: cluster 2022-04-22T17:58:24.620576+0000 mgr.y (mgr.34107) 1155 : cluster [DBG] pgmap v1751: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:58:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[45495]: audit 2022-04-22T17:58:25.195986+0000 mon.a (mon.0) 2644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[45495]: audit 2022-04-22T17:58:25.558898+0000 mon.a (mon.0) 2645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[46934]: cluster 2022-04-22T17:58:24.620576+0000 mgr.y (mgr.34107) 1155 : cluster [DBG] pgmap v1751: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:58:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[46934]: audit 2022-04-22T17:58:25.195986+0000 mon.a (mon.0) 2644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:26 smithi115 bash[46934]: audit 2022-04-22T17:58:25.558898+0000 mon.a (mon.0) 2645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:26.913 INFO:tasks.workunit.client.0.smithi115.stdout: 686: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:27.044 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yp1wco0IDGrVHrh off=1836343 len=978017 2022-04-22T17:58:27.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 3490 completed, throughput=5MB/sec 2022-04-22T17:58:27.045 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bOrz8e08_XwjqpH off=5966916 len=795279 2022-04-22T17:58:27.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 3491 completed, throughput=5MB/sec 2022-04-22T17:58:27.045 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NLFZgKO1bzixqED off=1008397 len=1517132 2022-04-22T17:58:27.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 3492 completed, throughput=5MB/sec 2022-04-22T17:58:27.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ulG0XybULD1D2On off=14364472 len=1747546 2022-04-22T17:58:27.046 INFO:tasks.workunit.client.0.smithi115.stdout:op 3493 completed, throughput=5MB/sec 2022-04-22T17:58:27.939 INFO:tasks.workunit.client.0.smithi115.stdout: 687: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:27.939 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Epd8nxPs0AMjFNO off=6473087 len=602013 2022-04-22T17:58:27.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 3494 completed, throughput=4.99MB/sec 2022-04-22T17:58:27.941 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TdT40Jc4c3VWgEA off=8751588 len=173519 2022-04-22T17:58:27.947 INFO:tasks.workunit.client.0.smithi115.stdout:op 3495 completed, throughput=4.99MB/sec 2022-04-22T17:58:27.947 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NnmFpNQdMWOyGBk off=1248749 len=2019455 2022-04-22T17:58:27.950 INFO:tasks.workunit.client.0.smithi115.stdout:op 3496 completed, throughput=5MB/sec 2022-04-22T17:58:27.950 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I2TEPqm-LVR0fBq off=376823 len=1297103 2022-04-22T17:58:27.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 3497 completed, throughput=5MB/sec 2022-04-22T17:58:27.954 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_8A4emTRFbuu6j3 off=372746 len=1123924 2022-04-22T17:58:27.960 INFO:tasks.workunit.client.0.smithi115.stdout:op 3498 completed, throughput=5MB/sec 2022-04-22T17:58:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:28 smithi121 bash[36902]: cluster 2022-04-22T17:58:26.621658+0000 mgr.y (mgr.34107) 1156 : cluster [DBG] pgmap v1752: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:58:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:28 smithi121 bash[36902]: audit 2022-04-22T17:58:27.582876+0000 mon.a (mon.0) 2646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:28 smithi115 bash[45495]: cluster 2022-04-22T17:58:26.621658+0000 mgr.y (mgr.34107) 1156 : cluster [DBG] pgmap v1752: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:58:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:28 smithi115 bash[45495]: audit 2022-04-22T17:58:27.582876+0000 mon.a (mon.0) 2646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:28 smithi115 bash[46934]: cluster 2022-04-22T17:58:26.621658+0000 mgr.y (mgr.34107) 1156 : cluster [DBG] pgmap v1752: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:58:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:28 smithi115 bash[46934]: audit 2022-04-22T17:58:27.582876+0000 mon.a (mon.0) 2646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:28.961 INFO:tasks.workunit.client.0.smithi115.stdout: 688: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:28.961 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HajAa8DQxGfSkle off=17240853 len=937356 2022-04-22T17:58:28.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 3499 completed, throughput=4.99MB/sec 2022-04-22T17:58:28.969 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nklrs-nTDETLRat off=8203348 len=92416 2022-04-22T17:58:28.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 3500 completed, throughput=4.99MB/sec 2022-04-22T17:58:28.970 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sFx_Y4KHSWB4Vt6 off=2517421 len=1101399 2022-04-22T17:58:28.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 3501 completed, throughput=5MB/sec 2022-04-22T17:58:28.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mawT0cWGUuOao4k off=11295009 len=1030018 2022-04-22T17:58:28.975 INFO:tasks.workunit.client.0.smithi115.stdout:op 3502 completed, throughput=5MB/sec 2022-04-22T17:58:28.975 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jC5a5CxSUz9LXOW off=4540657 len=722077 2022-04-22T17:58:28.981 INFO:tasks.workunit.client.0.smithi115.stdout:op 3503 completed, throughput=5MB/sec 2022-04-22T17:58:28.981 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FN_4l94uZJW8Hu2 off=4106890 len=2027883 2022-04-22T17:58:28.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 3504 completed, throughput=5MB/sec 2022-04-22T17:58:29.985 INFO:tasks.workunit.client.0.smithi115.stdout: 689: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:29.985 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8ED4J_Vl72cNB-r off=401031 len=1377203 2022-04-22T17:58:29.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 3505 completed, throughput=5MB/sec 2022-04-22T17:58:29.994 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dsBkG-IYLpgM0h1 off=10209236 len=1122753 2022-04-22T17:58:29.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 3506 completed, throughput=5MB/sec 2022-04-22T17:58:29.995 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O_qhOpevPQBf0PU off=5117107 len=1801234 2022-04-22T17:58:30.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 3507 completed, throughput=5MB/sec 2022-04-22T17:58:30.001 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uz37N7Qn6wUqwJu off=2282932 len=713694 2022-04-22T17:58:30.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 3508 completed, throughput=5MB/sec 2022-04-22T17:58:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:30 smithi121 bash[36902]: cluster 2022-04-22T17:58:28.622847+0000 mgr.y (mgr.34107) 1157 : cluster [DBG] pgmap v1753: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:30 smithi115 bash[45495]: cluster 2022-04-22T17:58:28.622847+0000 mgr.y (mgr.34107) 1157 : cluster [DBG] pgmap v1753: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:30 smithi115 bash[46934]: cluster 2022-04-22T17:58:28.622847+0000 mgr.y (mgr.34107) 1157 : cluster [DBG] pgmap v1753: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:31.003 INFO:tasks.workunit.client.0.smithi115.stdout: 690: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:31.018 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZxMUJ-F-670JGBa off=7313793 len=1736811 2022-04-22T17:58:31.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 3509 completed, throughput=5MB/sec 2022-04-22T17:58:31.020 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7IgaGZybw3Im7tg off=926839 len=1508894 2022-04-22T17:58:31.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 3510 completed, throughput=5MB/sec 2022-04-22T17:58:31.021 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2axK6zP3WWboHa0 off=16503892 len=2045391 2022-04-22T17:58:31.031 INFO:tasks.workunit.client.0.smithi115.stdout:op 3511 completed, throughput=5MB/sec 2022-04-22T17:58:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:31 smithi121 bash[36902]: audit 2022-04-22T17:58:30.199123+0000 mon.a (mon.0) 2647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:31 smithi121 bash[36902]: audit 2022-04-22T17:58:30.562628+0000 mon.a (mon.0) 2648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:31 smithi115 bash[45495]: audit 2022-04-22T17:58:30.199123+0000 mon.a (mon.0) 2647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:31 smithi115 bash[45495]: audit 2022-04-22T17:58:30.562628+0000 mon.a (mon.0) 2648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:31 smithi115 bash[46934]: audit 2022-04-22T17:58:30.199123+0000 mon.a (mon.0) 2647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:31 smithi115 bash[46934]: audit 2022-04-22T17:58:30.562628+0000 mon.a (mon.0) 2648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:32.031 INFO:tasks.workunit.client.0.smithi115.stdout: 691: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:32.031 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EUT38xz6YzQtH_b off=2269296 len=2011142 2022-04-22T17:58:32.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 3512 completed, throughput=5MB/sec 2022-04-22T17:58:32.037 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KHU_W2dbzb76dct off=8620770 len=2005035 2022-04-22T17:58:32.039 INFO:tasks.workunit.client.0.smithi115.stdout:op 3513 completed, throughput=5MB/sec 2022-04-22T17:58:32.039 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ezaK_4RvbCip6nD off=8061858 len=1770456 2022-04-22T17:58:32.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 3514 completed, throughput=5MB/sec 2022-04-22T17:58:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:31.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:32.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:32 smithi121 bash[36902]: cluster 2022-04-22T17:58:30.623979+0000 mgr.y (mgr.34107) 1158 : cluster [DBG] pgmap v1754: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:32 smithi115 bash[45495]: cluster 2022-04-22T17:58:30.623979+0000 mgr.y (mgr.34107) 1158 : cluster [DBG] pgmap v1754: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:32.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:32 smithi115 bash[46934]: cluster 2022-04-22T17:58:30.623979+0000 mgr.y (mgr.34107) 1158 : cluster [DBG] pgmap v1754: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T17:58:33.044 INFO:tasks.workunit.client.0.smithi115.stdout: 692: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:33.044 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P7Bq5vVSZddu2vZ off=3817392 len=1277472 2022-04-22T17:58:33.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 3515 completed, throughput=5MB/sec 2022-04-22T17:58:33.047 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-82p3xfPShfgRU8q off=8337952 len=1143058 2022-04-22T17:58:33.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 3516 completed, throughput=5MB/sec 2022-04-22T17:58:33.051 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NVbdo9QNJrDDur7 off=2898920 len=1817598 2022-04-22T17:58:33.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 3517 completed, throughput=5MB/sec 2022-04-22T17:58:33.061 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KEagjuqrofU42AS off=3713080 len=1338062 2022-04-22T17:58:33.065 INFO:tasks.workunit.client.0.smithi115.stdout:op 3518 completed, throughput=5MB/sec 2022-04-22T17:58:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:33 smithi121 bash[36902]: audit 2022-04-22T17:58:32.585952+0000 mon.a (mon.0) 2649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:33.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:33 smithi115 bash[45495]: audit 2022-04-22T17:58:32.585952+0000 mon.a (mon.0) 2649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:33 smithi115 bash[46934]: audit 2022-04-22T17:58:32.585952+0000 mon.a (mon.0) 2649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:33.599Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:34.065 INFO:tasks.workunit.client.0.smithi115.stdout: 693: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:34.065 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-B5TiCykhhJhH8Ol off=4016437 len=1624223 2022-04-22T17:58:34.071 INFO:tasks.workunit.client.0.smithi115.stdout:op 3519 completed, throughput=5MB/sec 2022-04-22T17:58:34.071 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_zrj55HR67OF2sb off=2800661 len=65954 2022-04-22T17:58:34.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 3520 completed, throughput=5MB/sec 2022-04-22T17:58:34.072 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d2zjIqjwezi2Uf8 off=5067351 len=1633278 2022-04-22T17:58:34.082 INFO:tasks.workunit.client.0.smithi115.stdout:op 3521 completed, throughput=5MB/sec 2022-04-22T17:58:34.082 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-692M7916Pf7Kug2 off=34132 len=1204015 2022-04-22T17:58:34.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 3522 completed, throughput=5MB/sec 2022-04-22T17:58:34.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:34 smithi121 bash[36902]: cluster 2022-04-22T17:58:32.625049+0000 mgr.y (mgr.34107) 1159 : cluster [DBG] pgmap v1755: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:34 smithi115 bash[45495]: cluster 2022-04-22T17:58:32.625049+0000 mgr.y (mgr.34107) 1159 : cluster [DBG] pgmap v1755: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:34.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:34 smithi115 bash[46934]: cluster 2022-04-22T17:58:32.625049+0000 mgr.y (mgr.34107) 1159 : cluster [DBG] pgmap v1755: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:35.084 INFO:tasks.workunit.client.0.smithi115.stdout: 694: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:35.085 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8xEqq51AP5bea5w off=3295323 len=953157 2022-04-22T17:58:35.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 3523 completed, throughput=4.99MB/sec 2022-04-22T17:58:35.092 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-azRNJsauIbjVEUs off=1974520 len=887709 2022-04-22T17:58:35.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 3524 completed, throughput=5MB/sec 2022-04-22T17:58:35.094 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PKFv0QiiVKXAftO off=3576619 len=1843904 2022-04-22T17:58:35.104 INFO:tasks.workunit.client.0.smithi115.stdout:op 3525 completed, throughput=5MB/sec 2022-04-22T17:58:35.104 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jX-sXy8Qwih9u7q off=12128484 len=1153678 2022-04-22T17:58:35.110 INFO:tasks.workunit.client.0.smithi115.stdout:op 3526 completed, throughput=5MB/sec 2022-04-22T17:58:35.110 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AHfqFceM02izpWH off=16396163 len=1326662 2022-04-22T17:58:35.113 INFO:tasks.workunit.client.0.smithi115.stdout:op 3527 completed, throughput=5MB/sec 2022-04-22T17:58:36.114 INFO:tasks.workunit.client.0.smithi115.stdout: 695: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:36.168 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V21518NZQ2LvWkd off=17760580 len=786246 2022-04-22T17:58:36.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 3528 completed, throughput=5MB/sec 2022-04-22T17:58:36.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NK4CZ-JbZI_CrdP off=14404313 len=2014916 2022-04-22T17:58:36.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 3529 completed, throughput=5MB/sec 2022-04-22T17:58:36.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4a07wsnn3cptZ8A off=2291157 len=983134 2022-04-22T17:58:36.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 3530 completed, throughput=5MB/sec 2022-04-22T17:58:36.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-30e84UBFXaVQBRr off=20857694 len=1313066 2022-04-22T17:58:36.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 3531 completed, throughput=5MB/sec 2022-04-22T17:58:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:36 smithi121 bash[36902]: cluster 2022-04-22T17:58:34.626178+0000 mgr.y (mgr.34107) 1160 : cluster [DBG] pgmap v1756: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:36 smithi121 bash[36902]: audit 2022-04-22T17:58:35.202059+0000 mon.a (mon.0) 2650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:36 smithi121 bash[36902]: audit 2022-04-22T17:58:35.566255+0000 mon.a (mon.0) 2651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[45495]: cluster 2022-04-22T17:58:34.626178+0000 mgr.y (mgr.34107) 1160 : cluster [DBG] pgmap v1756: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[45495]: audit 2022-04-22T17:58:35.202059+0000 mon.a (mon.0) 2650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[45495]: audit 2022-04-22T17:58:35.566255+0000 mon.a (mon.0) 2651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[46934]: cluster 2022-04-22T17:58:34.626178+0000 mgr.y (mgr.34107) 1160 : cluster [DBG] pgmap v1756: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:58:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[46934]: audit 2022-04-22T17:58:35.202059+0000 mon.a (mon.0) 2650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:36 smithi115 bash[46934]: audit 2022-04-22T17:58:35.566255+0000 mon.a (mon.0) 2651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:37.125 INFO:tasks.workunit.client.0.smithi115.stdout: 696: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:37.125 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H2m-8mK0oa3D8dq off=10989831 len=353249 2022-04-22T17:58:37.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 3532 completed, throughput=4.99MB/sec 2022-04-22T17:58:37.130 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aeL2rXDr_0UnaKJ off=10489549 len=1826085 2022-04-22T17:58:37.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 3533 completed, throughput=5MB/sec 2022-04-22T17:58:37.136 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2_MffhJWXPZypdO off=4242407 len=620329 2022-04-22T17:58:37.141 INFO:tasks.workunit.client.0.smithi115.stdout:op 3534 completed, throughput=5MB/sec 2022-04-22T17:58:37.141 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7hiiVOZauLSAjVW off=1849537 len=1612474 2022-04-22T17:58:37.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 3535 completed, throughput=5MB/sec 2022-04-22T17:58:37.145 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rL1helCeD6a1VQZ off=7912283 len=901215 2022-04-22T17:58:37.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 3536 completed, throughput=5MB/sec 2022-04-22T17:58:38.152 INFO:tasks.workunit.client.0.smithi115.stdout: 697: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:38.152 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_ynZPpPNJicHBWx off=1065114 len=1620470 2022-04-22T17:58:38.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 3537 completed, throughput=5MB/sec 2022-04-22T17:58:38.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bNfbEQgsr6f-D3S off=2997297 len=718295 2022-04-22T17:58:38.162 INFO:tasks.workunit.client.0.smithi115.stdout:op 3538 completed, throughput=5MB/sec 2022-04-22T17:58:38.162 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pscfYfOReN4DKkE off=9373293 len=1905455 2022-04-22T17:58:38.164 INFO:tasks.workunit.client.0.smithi115.stdout:op 3539 completed, throughput=5MB/sec 2022-04-22T17:58:38.164 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XV_jLrtfXvkHVCf off=685578 len=1404045 2022-04-22T17:58:38.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 3540 completed, throughput=5MB/sec 2022-04-22T17:58:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:38 smithi121 bash[36902]: cluster 2022-04-22T17:58:36.627433+0000 mgr.y (mgr.34107) 1161 : cluster [DBG] pgmap v1757: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:38 smithi121 bash[36902]: audit 2022-04-22T17:58:37.589303+0000 mon.a (mon.0) 2652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:38 smithi115 bash[45495]: cluster 2022-04-22T17:58:36.627433+0000 mgr.y (mgr.34107) 1161 : cluster [DBG] pgmap v1757: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:38 smithi115 bash[45495]: audit 2022-04-22T17:58:37.589303+0000 mon.a (mon.0) 2652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:38 smithi115 bash[46934]: cluster 2022-04-22T17:58:36.627433+0000 mgr.y (mgr.34107) 1161 : cluster [DBG] pgmap v1757: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T17:58:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:38 smithi115 bash[46934]: audit 2022-04-22T17:58:37.589303+0000 mon.a (mon.0) 2652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:39.173 INFO:tasks.workunit.client.0.smithi115.stdout: 698: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:39.174 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aHP0wp4vUIP3c9h off=5983906 len=242380 2022-04-22T17:58:39.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 3541 completed, throughput=4.99MB/sec 2022-04-22T17:58:39.175 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0TKzf3VwfXb8B1O off=378754 len=197185 2022-04-22T17:58:39.180 INFO:tasks.workunit.client.0.smithi115.stdout:op 3542 completed, throughput=4.99MB/sec 2022-04-22T17:58:39.180 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MOnGIvUq724ZhJh off=2583682 len=355847 2022-04-22T17:58:39.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 3543 completed, throughput=5MB/sec 2022-04-22T17:58:39.181 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BI5A-axKdS1kz51 off=2754449 len=486686 2022-04-22T17:58:39.187 INFO:tasks.workunit.client.0.smithi115.stdout:op 3544 completed, throughput=5MB/sec 2022-04-22T17:58:39.187 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g2mHghgnOnxQTmC off=83253 len=984536 2022-04-22T17:58:39.189 INFO:tasks.workunit.client.0.smithi115.stdout:op 3545 completed, throughput=5MB/sec 2022-04-22T17:58:39.189 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Uew1uEwmKwprhf5 off=5155479 len=584782 2022-04-22T17:58:39.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 3546 completed, throughput=5MB/sec 2022-04-22T17:58:39.194 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CLq7cyCTI-YAP0G off=3126605 len=749864 2022-04-22T17:58:39.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 3547 completed, throughput=5MB/sec 2022-04-22T17:58:39.196 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cfqtukUyJ2ZYNI9 off=1398958 len=144324 2022-04-22T17:58:39.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 3548 completed, throughput=5MB/sec 2022-04-22T17:58:39.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UsHLXjdGYQ6HV96 off=6846067 len=497621 2022-04-22T17:58:39.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 3549 completed, throughput=5MB/sec 2022-04-22T17:58:39.204 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_zp0dHD1r9nKYDc off=892464 len=1823880 2022-04-22T17:58:39.213 INFO:tasks.workunit.client.0.smithi115.stdout:op 3550 completed, throughput=5MB/sec 2022-04-22T17:58:40.214 INFO:tasks.workunit.client.0.smithi115.stdout: 699: throughput=5MB/sec pending data=0 2022-04-22T17:58:40.214 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mYCKU2xbzH39Usv off=14397806 len=2059095 2022-04-22T17:58:40.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 3551 completed, throughput=5MB/sec 2022-04-22T17:58:40.220 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-StPQ5Kkhjcsubpt off=9255152 len=2025043 2022-04-22T17:58:40.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 3552 completed, throughput=5MB/sec 2022-04-22T17:58:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:40 smithi121 bash[36902]: cluster 2022-04-22T17:58:38.628671+0000 mgr.y (mgr.34107) 1162 : cluster [DBG] pgmap v1758: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:58:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:40 smithi115 bash[45495]: cluster 2022-04-22T17:58:38.628671+0000 mgr.y (mgr.34107) 1162 : cluster [DBG] pgmap v1758: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:58:40.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:40 smithi115 bash[46934]: cluster 2022-04-22T17:58:38.628671+0000 mgr.y (mgr.34107) 1162 : cluster [DBG] pgmap v1758: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 3 op/s 2022-04-22T17:58:41.232 INFO:tasks.workunit.client.0.smithi115.stdout: 700: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:41.232 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dR1auHRk9gxazgJ off=13624117 len=558980 2022-04-22T17:58:41.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 3553 completed, throughput=4.99MB/sec 2022-04-22T17:58:41.233 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eFt6KiRhFCwyLEw off=7180474 len=1990689 2022-04-22T17:58:41.237 INFO:tasks.workunit.client.0.smithi115.stdout:op 3554 completed, throughput=5MB/sec 2022-04-22T17:58:41.237 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JstggGY7LbFkd8E off=18182578 len=887809 2022-04-22T17:58:41.243 INFO:tasks.workunit.client.0.smithi115.stdout:op 3555 completed, throughput=5MB/sec 2022-04-22T17:58:41.243 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E_r-fotB7LEI0gY off=2669026 len=1383882 2022-04-22T17:58:41.251 INFO:tasks.workunit.client.0.smithi115.stdout:op 3556 completed, throughput=5MB/sec 2022-04-22T17:58:41.252 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E_gdSoedkS5WaWl off=4020586 len=630970 2022-04-22T17:58:41.253 INFO:tasks.workunit.client.0.smithi115.stdout:op 3557 completed, throughput=5MB/sec 2022-04-22T17:58:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:41 smithi121 bash[36902]: audit 2022-04-22T17:58:40.204987+0000 mon.a (mon.0) 2653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:41 smithi121 bash[36902]: audit 2022-04-22T17:58:40.569933+0000 mon.a (mon.0) 2654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:41 smithi115 bash[46934]: audit 2022-04-22T17:58:40.204987+0000 mon.a (mon.0) 2653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:41 smithi115 bash[46934]: audit 2022-04-22T17:58:40.569933+0000 mon.a (mon.0) 2654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:41 smithi115 bash[45495]: audit 2022-04-22T17:58:40.204987+0000 mon.a (mon.0) 2653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:41 smithi115 bash[45495]: audit 2022-04-22T17:58:40.569933+0000 mon.a (mon.0) 2654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:41.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:42.253 INFO:tasks.workunit.client.0.smithi115.stdout: 701: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:42.254 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-me-DA9qL6q-neUw off=2987607 len=529986 2022-04-22T17:58:42.260 INFO:tasks.workunit.client.0.smithi115.stdout:op 3558 completed, throughput=4.99MB/sec 2022-04-22T17:58:42.260 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O0wgsjKSgf-HP6X off=10125141 len=1965845 2022-04-22T17:58:42.268 INFO:tasks.workunit.client.0.smithi115.stdout:op 3559 completed, throughput=5MB/sec 2022-04-22T17:58:42.268 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xeqllK_gqiOaKZq off=19804963 len=671309 2022-04-22T17:58:42.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 3560 completed, throughput=5MB/sec 2022-04-22T17:58:42.271 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q4b6EeJOVmROPjY off=2186440 len=1569251 2022-04-22T17:58:42.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3561 completed, throughput=5MB/sec 2022-04-22T17:58:42.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QccbYVa5jg7nEFD off=21416086 len=1364598 2022-04-22T17:58:42.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 3562 completed, throughput=5MB/sec 2022-04-22T17:58:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:42 smithi121 bash[36902]: cluster 2022-04-22T17:58:40.629795+0000 mgr.y (mgr.34107) 1163 : cluster [DBG] pgmap v1759: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2022-04-22T17:58:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:42 smithi115 bash[45495]: cluster 2022-04-22T17:58:40.629795+0000 mgr.y (mgr.34107) 1163 : cluster [DBG] pgmap v1759: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2022-04-22T17:58:42.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:42 smithi115 bash[46934]: cluster 2022-04-22T17:58:40.629795+0000 mgr.y (mgr.34107) 1163 : cluster [DBG] pgmap v1759: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 3 op/s 2022-04-22T17:58:43.279 INFO:tasks.workunit.client.0.smithi115.stdout: 702: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:43.279 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QIXGwpNCPiRPK63 off=8931397 len=352572 2022-04-22T17:58:43.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 3563 completed, throughput=5MB/sec 2022-04-22T17:58:43.284 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1cp3Zg3I3u1RpPq off=13620453 len=582946 2022-04-22T17:58:43.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 3564 completed, throughput=5MB/sec 2022-04-22T17:58:43.286 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LvS3F_IZZRQ4fSx off=256010 len=322465 2022-04-22T17:58:43.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 3565 completed, throughput=5MB/sec 2022-04-22T17:58:43.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8GZ7b64ocG4jaUT off=14959020 len=609440 2022-04-22T17:58:43.296 INFO:tasks.workunit.client.0.smithi115.stdout:op 3566 completed, throughput=5MB/sec 2022-04-22T17:58:43.296 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IypgJ-5YRBC7CUQ off=1375448 len=13887 2022-04-22T17:58:43.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 3567 completed, throughput=5MB/sec 2022-04-22T17:58:43.300 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zo61HSfCTa26zVm off=3453693 len=907384 2022-04-22T17:58:43.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 3568 completed, throughput=5MB/sec 2022-04-22T17:58:43.303 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CY1F0aYlFCuGzNk off=6487708 len=1112638 2022-04-22T17:58:43.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 3569 completed, throughput=5MB/sec 2022-04-22T17:58:43.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SKMSarVhikeZbxO off=3452873 len=197596 2022-04-22T17:58:43.306 INFO:tasks.workunit.client.0.smithi115.stdout:op 3570 completed, throughput=5MB/sec 2022-04-22T17:58:43.306 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nqIjC4xF7rZ37pz off=7944661 len=1820507 2022-04-22T17:58:43.315 INFO:tasks.workunit.client.0.smithi115.stdout:op 3571 completed, throughput=5MB/sec 2022-04-22T17:58:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:43 smithi121 bash[36902]: audit 2022-04-22T17:58:42.592282+0000 mon.a (mon.0) 2655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:43 smithi115 bash[45495]: audit 2022-04-22T17:58:42.592282+0000 mon.a (mon.0) 2655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:43 smithi115 bash[46934]: audit 2022-04-22T17:58:42.592282+0000 mon.a (mon.0) 2655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:43.599Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:44.315 INFO:tasks.workunit.client.0.smithi115.stdout: 703: throughput=5MB/sec pending data=0 2022-04-22T17:58:44.342 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Jz9dvki314R2nHP off=160501 len=574884 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 3572 completed, throughput=5MB/sec 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KEzY3knMwDlAD09 off=548885 len=793747 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 3573 completed, throughput=5MB/sec 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0LJ67NSBVBR4trX off=6497906 len=1672607 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 3574 completed, throughput=5MB/sec 2022-04-22T17:58:44.343 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aOVYa3Uwt6ZDINn off=2137441 len=472619 2022-04-22T17:58:44.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 3575 completed, throughput=5MB/sec 2022-04-22T17:58:44.344 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XLWyYxcxYrOTh-4 off=13988903 len=525708 2022-04-22T17:58:44.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 3576 completed, throughput=5MB/sec 2022-04-22T17:58:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:44 smithi121 bash[36902]: cluster 2022-04-22T17:58:42.630920+0000 mgr.y (mgr.34107) 1164 : cluster [DBG] pgmap v1760: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:58:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:44 smithi115 bash[45495]: cluster 2022-04-22T17:58:42.630920+0000 mgr.y (mgr.34107) 1164 : cluster [DBG] pgmap v1760: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:58:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:44 smithi115 bash[46934]: cluster 2022-04-22T17:58:42.630920+0000 mgr.y (mgr.34107) 1164 : cluster [DBG] pgmap v1760: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 3 op/s 2022-04-22T17:58:45.334 INFO:tasks.workunit.client.0.smithi115.stdout: 704: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:45.334 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cLNLapQnYrcVlIa off=17112180 len=833784 2022-04-22T17:58:45.341 INFO:tasks.workunit.client.0.smithi115.stdout:op 3577 completed, throughput=4.99MB/sec 2022-04-22T17:58:45.341 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hVcaWRIMRmXPwnk off=1440091 len=679452 2022-04-22T17:58:45.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 3578 completed, throughput=5MB/sec 2022-04-22T17:58:45.344 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q7Zag8OtzGtqixF off=14138081 len=772977 2022-04-22T17:58:45.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 3579 completed, throughput=5MB/sec 2022-04-22T17:58:45.346 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Dtsvn-UK7tXJSS7 off=4843127 len=771555 2022-04-22T17:58:45.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 3580 completed, throughput=5MB/sec 2022-04-22T17:58:45.352 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pT-A5ZJJGwLYLmQ off=14204830 len=396061 2022-04-22T17:58:45.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 3581 completed, throughput=5MB/sec 2022-04-22T17:58:45.357 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TP_49qxVqWnCA3A off=5157360 len=174990 2022-04-22T17:58:45.357 INFO:tasks.workunit.client.0.smithi115.stdout:op 3582 completed, throughput=5MB/sec 2022-04-22T17:58:45.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tc85rPSAF_up8CW off=13275458 len=1193636 2022-04-22T17:58:45.360 INFO:tasks.workunit.client.0.smithi115.stdout:op 3583 completed, throughput=5MB/sec 2022-04-22T17:58:45.360 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8SWACI9S-qsz_SF off=6666044 len=714177 2022-04-22T17:58:45.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 3584 completed, throughput=5MB/sec 2022-04-22T17:58:46.365 INFO:tasks.workunit.client.0.smithi115.stdout: 705: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:46.366 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Pb1rjbzpF7tuPQr off=548659 len=1015607 2022-04-22T17:58:46.368 INFO:tasks.workunit.client.0.smithi115.stdout:op 3585 completed, throughput=5MB/sec 2022-04-22T17:58:46.368 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rpki8FccZH_X6cc off=1554085 len=1878077 2022-04-22T17:58:46.378 INFO:tasks.workunit.client.0.smithi115.stdout:op 3586 completed, throughput=5MB/sec 2022-04-22T17:58:46.378 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AFOClqR9pSXL1_N off=7875190 len=788567 2022-04-22T17:58:46.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 3587 completed, throughput=5MB/sec 2022-04-22T17:58:46.384 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aulMWmp1NC1h-_I off=9591805 len=75322 2022-04-22T17:58:46.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 3588 completed, throughput=5MB/sec 2022-04-22T17:58:46.387 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zH5FJJyhiab4R90 off=2350770 len=136685 2022-04-22T17:58:46.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 3589 completed, throughput=5MB/sec 2022-04-22T17:58:46.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tThk-jzvmhCl9nm off=3751741 len=1504015 2022-04-22T17:58:46.393 INFO:tasks.workunit.client.0.smithi115.stdout:op 3590 completed, throughput=5MB/sec 2022-04-22T17:58:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:46 smithi121 bash[36902]: cluster 2022-04-22T17:58:44.632095+0000 mgr.y (mgr.34107) 1165 : cluster [DBG] pgmap v1761: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:46 smithi121 bash[36902]: audit 2022-04-22T17:58:45.208169+0000 mon.a (mon.0) 2656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:46 smithi121 bash[36902]: audit 2022-04-22T17:58:45.573619+0000 mon.a (mon.0) 2657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[46934]: cluster 2022-04-22T17:58:44.632095+0000 mgr.y (mgr.34107) 1165 : cluster [DBG] pgmap v1761: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[46934]: audit 2022-04-22T17:58:45.208169+0000 mon.a (mon.0) 2656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[46934]: audit 2022-04-22T17:58:45.573619+0000 mon.a (mon.0) 2657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[45495]: cluster 2022-04-22T17:58:44.632095+0000 mgr.y (mgr.34107) 1165 : cluster [DBG] pgmap v1761: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[45495]: audit 2022-04-22T17:58:45.208169+0000 mon.a (mon.0) 2656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:46 smithi115 bash[45495]: audit 2022-04-22T17:58:45.573619+0000 mon.a (mon.0) 2657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:47.393 INFO:tasks.workunit.client.0.smithi115.stdout: 706: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:47.393 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KSex5l-TallVYe- off=1216940 len=1869714 2022-04-22T17:58:47.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 3591 completed, throughput=5MB/sec 2022-04-22T17:58:47.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dzm4Qe_36pg5U1s off=2897887 len=1676094 2022-04-22T17:58:47.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 3592 completed, throughput=5MB/sec 2022-04-22T17:58:47.407 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YFYgw9tw3jbAY9m off=5268588 len=70747 2022-04-22T17:58:47.407 INFO:tasks.workunit.client.0.smithi115.stdout:op 3593 completed, throughput=5MB/sec 2022-04-22T17:58:47.407 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OLCig4LiTh52bX_ off=1587840 len=1550635 2022-04-22T17:58:47.412 INFO:tasks.workunit.client.0.smithi115.stdout:op 3594 completed, throughput=5MB/sec 2022-04-22T17:58:47.692 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:58:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:58:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (21m) 58s ago 35m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (35m) 2m ago 35m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (23m) 58s ago 36m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (23m) 2m ago 36m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (31m) 2m ago 40m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (31m) 58s ago 43m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (30m) 58s ago 43m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (29m) 2m ago 41m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (30m) 58s ago 41m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (22m) 58s ago 35m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:58:48.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (22m) 2m ago 35m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (28m) 58s ago 40m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (27m) 58s ago 40m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (27m) 58s ago 39m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (27m) 58s ago 39m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (26m) 2m ago 38m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (25m) 2m ago 38m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (24m) 2m ago 37m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (24m) 2m ago 37m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:58:48.036 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (22m) 2m ago 35m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:58:48.412 INFO:tasks.workunit.client.0.smithi115.stdout: 707: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:48.412 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hviuPHM9r876jJx off=15507187 len=876173 2022-04-22T17:58:48.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 3595 completed, throughput=4.99MB/sec 2022-04-22T17:58:48.419 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7l9dwmdrbtbvKP5 off=2248642 len=1748499 2022-04-22T17:58:48.420 INFO:tasks.workunit.client.0.smithi115.stdout:op 3596 completed, throughput=5MB/sec 2022-04-22T17:58:48.420 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u3Lhyv26dw3mauN off=5881481 len=1602224 2022-04-22T17:58:48.429 INFO:tasks.workunit.client.0.smithi115.stdout:op 3597 completed, throughput=5MB/sec 2022-04-22T17:58:48.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZrSaVDbTKtxi5Kb off=10340702 len=1812261 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:58:48.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:58:48.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:48.434 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:58:48.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:58:48.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:48.434 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:58:48.435 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:58:48.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 3598 completed, throughput=5MB/sec 2022-04-22T17:58:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:48 smithi121 bash[36902]: cluster 2022-04-22T17:58:46.633188+0000 mgr.y (mgr.34107) 1166 : cluster [DBG] pgmap v1762: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:48.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:48 smithi121 bash[36902]: audit 2022-04-22T17:58:47.595169+0000 mon.a (mon.0) 2658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:48 smithi115 bash[45495]: cluster 2022-04-22T17:58:46.633188+0000 mgr.y (mgr.34107) 1166 : cluster [DBG] pgmap v1762: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:48 smithi115 bash[45495]: audit 2022-04-22T17:58:47.595169+0000 mon.a (mon.0) 2658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:48 smithi115 bash[46934]: cluster 2022-04-22T17:58:46.633188+0000 mgr.y (mgr.34107) 1166 : cluster [DBG] pgmap v1762: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:58:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:48 smithi115 bash[46934]: audit 2022-04-22T17:58:47.595169+0000 mon.a (mon.0) 2658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:49.437 INFO:tasks.workunit.client.0.smithi115.stdout: 708: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:49.438 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4nj0-s4cDGkp2nb off=2652435 len=686338 2022-04-22T17:58:49.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 3599 completed, throughput=5MB/sec 2022-04-22T17:58:49.443 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S0xrzkL2-4pSv9e off=17685995 len=771641 2022-04-22T17:58:49.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 3600 completed, throughput=5MB/sec 2022-04-22T17:58:49.449 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BP-hg4c53zOAexP off=7658372 len=1884354 2022-04-22T17:58:49.452 INFO:tasks.workunit.client.0.smithi115.stdout:op 3601 completed, throughput=5MB/sec 2022-04-22T17:58:49.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jU1-hZveYFg0PSm off=3715074 len=638041 2022-04-22T17:58:49.454 INFO:tasks.workunit.client.0.smithi115.stdout:op 3602 completed, throughput=5MB/sec 2022-04-22T17:58:49.454 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W4s9nX5kvKUwvUp off=5463606 len=1904054 2022-04-22T17:58:49.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 3603 completed, throughput=5MB/sec 2022-04-22T17:58:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:49 smithi121 bash[36902]: audit 2022-04-22T17:58:47.670459+0000 mgr.y (mgr.34107) 1167 : audit [DBG] from='client.45565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:49 smithi121 bash[36902]: audit 2022-04-22T17:58:48.013300+0000 mgr.y (mgr.34107) 1168 : audit [DBG] from='client.45571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:49 smithi121 bash[36902]: audit 2022-04-22T17:58:48.424741+0000 mon.a (mon.0) 2659 : audit [DBG] from='client.? 172.21.15.115:0/2448463991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[45495]: audit 2022-04-22T17:58:47.670459+0000 mgr.y (mgr.34107) 1167 : audit [DBG] from='client.45565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[45495]: audit 2022-04-22T17:58:48.013300+0000 mgr.y (mgr.34107) 1168 : audit [DBG] from='client.45571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[45495]: audit 2022-04-22T17:58:48.424741+0000 mon.a (mon.0) 2659 : audit [DBG] from='client.? 172.21.15.115:0/2448463991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[46934]: audit 2022-04-22T17:58:47.670459+0000 mgr.y (mgr.34107) 1167 : audit [DBG] from='client.45565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[46934]: audit 2022-04-22T17:58:48.013300+0000 mgr.y (mgr.34107) 1168 : audit [DBG] from='client.45571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:58:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:49 smithi115 bash[46934]: audit 2022-04-22T17:58:48.424741+0000 mon.a (mon.0) 2659 : audit [DBG] from='client.? 172.21.15.115:0/2448463991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:58:50.457 INFO:tasks.workunit.client.0.smithi115.stdout: 709: throughput=5MB/sec pending data=0 2022-04-22T17:58:50.458 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4HEtUOHDtPJ-0Oq off=1226335 len=793672 2022-04-22T17:58:50.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 3604 completed, throughput=5MB/sec 2022-04-22T17:58:50.464 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Uk-gaZmpOd9N-2v off=0 len=1761975 2022-04-22T17:58:50.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 3605 completed, throughput=5MB/sec 2022-04-22T17:58:50.467 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QsYOU7yS9zUA7M0 off=4882886 len=1440680 2022-04-22T17:58:50.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 3606 completed, throughput=5MB/sec 2022-04-22T17:58:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:50 smithi121 bash[36902]: cluster 2022-04-22T17:58:48.634359+0000 mgr.y (mgr.34107) 1169 : cluster [DBG] pgmap v1763: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:58:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:50 smithi121 bash[36902]: audit 2022-04-22T17:58:49.852405+0000 mon.a (mon.0) 2660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:58:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:50 smithi121 bash[36902]: audit 2022-04-22T17:58:49.855538+0000 mon.a (mon.0) 2661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:58:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:50 smithi121 bash[36902]: audit 2022-04-22T17:58:49.857314+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:58:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[45495]: cluster 2022-04-22T17:58:48.634359+0000 mgr.y (mgr.34107) 1169 : cluster [DBG] pgmap v1763: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[45495]: audit 2022-04-22T17:58:49.852405+0000 mon.a (mon.0) 2660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[45495]: audit 2022-04-22T17:58:49.855538+0000 mon.a (mon.0) 2661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[45495]: audit 2022-04-22T17:58:49.857314+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[46934]: cluster 2022-04-22T17:58:48.634359+0000 mgr.y (mgr.34107) 1169 : cluster [DBG] pgmap v1763: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[46934]: audit 2022-04-22T17:58:49.852405+0000 mon.a (mon.0) 2660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[46934]: audit 2022-04-22T17:58:49.855538+0000 mon.a (mon.0) 2661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:58:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:50 smithi115 bash[46934]: audit 2022-04-22T17:58:49.857314+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:58:51.476 INFO:tasks.workunit.client.0.smithi115.stdout: 710: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:51.477 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bq2re5PHlzqXv5_ off=7501390 len=612233 2022-04-22T17:58:51.483 INFO:tasks.workunit.client.0.smithi115.stdout:op 3607 completed, throughput=4.99MB/sec 2022-04-22T17:58:51.483 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NwDY2xutO9UeSdv off=4681595 len=1894563 2022-04-22T17:58:51.487 INFO:tasks.workunit.client.0.smithi115.stdout:op 3608 completed, throughput=5MB/sec 2022-04-22T17:58:51.487 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LRWP0W5_3GoffFs off=6413658 len=1140449 2022-04-22T17:58:51.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 3609 completed, throughput=5MB/sec 2022-04-22T17:58:51.491 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E89jelBWcDApS_6 off=15732783 len=1262531 2022-04-22T17:58:51.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 3610 completed, throughput=5MB/sec 2022-04-22T17:58:51.492 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bx7iX9sGVlYf_Ip off=2557308 len=1844019 2022-04-22T17:58:51.508 INFO:tasks.workunit.client.0.smithi115.stdout:op 3611 completed, throughput=5MB/sec 2022-04-22T17:58:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:51 smithi121 bash[36902]: audit 2022-04-22T17:58:50.211755+0000 mon.a (mon.0) 2663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:58:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:51 smithi121 bash[36902]: audit 2022-04-22T17:58:50.212103+0000 mon.a (mon.0) 2664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:51 smithi121 bash[36902]: audit 2022-04-22T17:58:50.577421+0000 mon.a (mon.0) 2665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[45495]: audit 2022-04-22T17:58:50.211755+0000 mon.a (mon.0) 2663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:58:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[45495]: audit 2022-04-22T17:58:50.212103+0000 mon.a (mon.0) 2664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[45495]: audit 2022-04-22T17:58:50.577421+0000 mon.a (mon.0) 2665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[46934]: audit 2022-04-22T17:58:50.211755+0000 mon.a (mon.0) 2663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:58:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[46934]: audit 2022-04-22T17:58:50.212103+0000 mon.a (mon.0) 2664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[46934]: audit 2022-04-22T17:58:50.577421+0000 mon.a (mon.0) 2665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:51.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:52.502 INFO:tasks.workunit.client.0.smithi115.stdout: 711: throughput=5MB/sec pending data=0 2022-04-22T17:58:52.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S02a5MPJZYTrV4F off=3316797 len=1191057 2022-04-22T17:58:52.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 3612 completed, throughput=5MB/sec 2022-04-22T17:58:52.509 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bCAPCq1ewIn7AQc off=5977971 len=468275 2022-04-22T17:58:52.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 3613 completed, throughput=5MB/sec 2022-04-22T17:58:52.510 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kDy-7cF-6oyz_1B off=7593353 len=1713421 2022-04-22T17:58:52.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 3614 completed, throughput=5MB/sec 2022-04-22T17:58:52.512 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-11GpQkjdIaocgUI off=3100894 len=619254 2022-04-22T17:58:52.515 INFO:tasks.workunit.client.0.smithi115.stdout:op 3615 completed, throughput=5MB/sec 2022-04-22T17:58:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:52 smithi121 bash[36902]: cluster 2022-04-22T17:58:50.635448+0000 mgr.y (mgr.34107) 1170 : cluster [DBG] pgmap v1764: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:52 smithi115 bash[45495]: cluster 2022-04-22T17:58:50.635448+0000 mgr.y (mgr.34107) 1170 : cluster [DBG] pgmap v1764: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:52 smithi115 bash[46934]: cluster 2022-04-22T17:58:50.635448+0000 mgr.y (mgr.34107) 1170 : cluster [DBG] pgmap v1764: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T17:58:53.528 INFO:tasks.workunit.client.0.smithi115.stdout: 712: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:53.529 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YLkc_0HGuBzpOwZ off=2575903 len=967416 2022-04-22T17:58:53.529 INFO:tasks.workunit.client.0.smithi115.stdout:op 3616 completed, throughput=4.99MB/sec 2022-04-22T17:58:53.529 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CwApThLBdQ-JHeK off=19854260 len=753540 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 3617 completed, throughput=5MB/sec 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EIqT1Wia_h72Ca_ off=4969897 len=1013573 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 3618 completed, throughput=5MB/sec 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LwibS33C9cmUExw off=1414932 len=965087 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:op 3619 completed, throughput=5MB/sec 2022-04-22T17:58:53.582 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0FxvEiSSRu9detf off=6343061 len=878137 2022-04-22T17:58:53.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 3620 completed, throughput=5MB/sec 2022-04-22T17:58:53.583 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XLYkHo0SLEJ404h off=16246160 len=203390 2022-04-22T17:58:53.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:53 smithi121 bash[36902]: audit 2022-04-22T17:58:52.598304+0000 mon.a (mon.0) 2666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:53.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 3621 completed, throughput=5MB/sec 2022-04-22T17:58:53.584 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pKFm64IOQkGFMU5 off=8023647 len=1792138 2022-04-22T17:58:53.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 3622 completed, throughput=5MB/sec 2022-04-22T17:58:53.584 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:53 smithi115 bash[45495]: audit 2022-04-22T17:58:52.598304+0000 mon.a (mon.0) 2666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:53.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:53 smithi115 bash[46934]: audit 2022-04-22T17:58:52.598304+0000 mon.a (mon.0) 2666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:58:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:58:53.599Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:58:54.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:54 smithi121 bash[36902]: cluster 2022-04-22T17:58:52.636516+0000 mgr.y (mgr.34107) 1171 : cluster [DBG] pgmap v1765: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:54.540 INFO:tasks.workunit.client.0.smithi115.stdout: 713: throughput=5MB/sec pending data=0 2022-04-22T17:58:54.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vppelyxfv7M6L14 off=1094977 len=1821425 2022-04-22T17:58:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:54 smithi115 bash[45495]: cluster 2022-04-22T17:58:52.636516+0000 mgr.y (mgr.34107) 1171 : cluster [DBG] pgmap v1765: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:54 smithi115 bash[46934]: cluster 2022-04-22T17:58:52.636516+0000 mgr.y (mgr.34107) 1171 : cluster [DBG] pgmap v1765: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:54.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 3623 completed, throughput=5MB/sec 2022-04-22T17:58:54.551 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lE8fJtWJcVvBhp5 off=1510785 len=335493 2022-04-22T17:58:54.555 INFO:tasks.workunit.client.0.smithi115.stdout:op 3624 completed, throughput=5MB/sec 2022-04-22T17:58:54.555 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vCDnola3sfrdyjo off=4529888 len=48911 2022-04-22T17:58:54.557 INFO:tasks.workunit.client.0.smithi115.stdout:op 3625 completed, throughput=5MB/sec 2022-04-22T17:58:54.557 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tf6yEE6DxVdPb5u off=5998535 len=381018 2022-04-22T17:58:54.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 3626 completed, throughput=5MB/sec 2022-04-22T17:58:54.559 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v0fff7O6V4LgBeX off=5727017 len=1150774 2022-04-22T17:58:54.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 3627 completed, throughput=5MB/sec 2022-04-22T17:58:54.565 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--_Iot56NRMAzjvu off=11788343 len=858364 2022-04-22T17:58:54.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 3628 completed, throughput=5MB/sec 2022-04-22T17:58:55.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:55 smithi121 bash[36902]: audit 2022-04-22T17:58:55.214364+0000 mon.a (mon.0) 2667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:55.650 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:55 smithi115 bash[45495]: audit 2022-04-22T17:58:55.214364+0000 mon.a (mon.0) 2667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:55.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:55 smithi115 bash[46934]: audit 2022-04-22T17:58:55.214364+0000 mon.a (mon.0) 2667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:58:55.651 INFO:tasks.workunit.client.0.smithi115.stdout: 714: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:55.651 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-69hlnCFlloKs_Hm off=17490424 len=746559 2022-04-22T17:58:55.651 INFO:tasks.workunit.client.0.smithi115.stdout:op 3629 completed, throughput=4.99MB/sec 2022-04-22T17:58:55.651 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-A6ht4waoR9hgigC off=4381372 len=1304906 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 3630 completed, throughput=5MB/sec 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PAlr6ex6YAt0LBu off=6575311 len=531683 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 3631 completed, throughput=5MB/sec 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MXYkYe4LId65Jt_ off=2210691 len=1390948 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 3632 completed, throughput=5MB/sec 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1jo0Q0XkxOeAhL5 off=618719 len=943121 2022-04-22T17:58:55.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 3633 completed, throughput=5MB/sec 2022-04-22T17:58:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:56 smithi121 bash[36902]: cluster 2022-04-22T17:58:54.637567+0000 mgr.y (mgr.34107) 1172 : cluster [DBG] pgmap v1766: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:56 smithi121 bash[36902]: audit 2022-04-22T17:58:55.580957+0000 mon.a (mon.0) 2668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:56 smithi115 bash[45495]: cluster 2022-04-22T17:58:54.637567+0000 mgr.y (mgr.34107) 1172 : cluster [DBG] pgmap v1766: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:56 smithi115 bash[45495]: audit 2022-04-22T17:58:55.580957+0000 mon.a (mon.0) 2668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:56 smithi115 bash[46934]: cluster 2022-04-22T17:58:54.637567+0000 mgr.y (mgr.34107) 1172 : cluster [DBG] pgmap v1766: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:56 smithi115 bash[46934]: audit 2022-04-22T17:58:55.580957+0000 mon.a (mon.0) 2668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:58:56.595 INFO:tasks.workunit.client.0.smithi115.stdout: 715: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:56.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5UVEltpyRtCvW0w off=10674439 len=315899 2022-04-22T17:58:56.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 3634 completed, throughput=4.99MB/sec 2022-04-22T17:58:56.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-31WGP-8tWwgrFw6 off=8888377 len=296716 2022-04-22T17:58:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 3635 completed, throughput=4.99MB/sec 2022-04-22T17:58:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aAYwvPERC4bACYC off=4545610 len=468141 2022-04-22T17:58:56.600 INFO:tasks.workunit.client.0.smithi115.stdout:op 3636 completed, throughput=4.99MB/sec 2022-04-22T17:58:56.600 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-66YnpUEHK1QWx93 off=12116267 len=2045088 2022-04-22T17:58:56.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 3637 completed, throughput=5MB/sec 2022-04-22T17:58:56.604 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8md80FRkpXT4Ubx off=10295379 len=955544 2022-04-22T17:58:56.611 INFO:tasks.workunit.client.0.smithi115.stdout:op 3638 completed, throughput=5MB/sec 2022-04-22T17:58:56.611 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AaTdFjV501BpKcW off=9621010 len=143874 2022-04-22T17:58:56.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 3639 completed, throughput=5MB/sec 2022-04-22T17:58:56.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Sa8R9RQ1tD9SrX1 off=2023425 len=34531 2022-04-22T17:58:56.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 3640 completed, throughput=5MB/sec 2022-04-22T17:58:56.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-USqg3buLXlPgiaP off=9182496 len=1390642 2022-04-22T17:58:56.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 3641 completed, throughput=5MB/sec 2022-04-22T17:58:57.615 INFO:tasks.workunit.client.0.smithi115.stdout: 716: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:57.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7HH32ixM9l3qYTt off=2567660 len=1221533 2022-04-22T17:58:57.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 3642 completed, throughput=5MB/sec 2022-04-22T17:58:57.619 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5IKb2lHefPUHWbT off=5552279 len=959779 2022-04-22T17:58:57.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 3643 completed, throughput=5MB/sec 2022-04-22T17:58:57.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-30e84UBFXaVQBRr off=2024631 len=542698 2022-04-22T17:58:57.627 INFO:tasks.workunit.client.0.smithi115.stdout:op 3644 completed, throughput=5MB/sec 2022-04-22T17:58:57.627 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wahmmgHqHL93MoH off=3414049 len=993238 2022-04-22T17:58:57.633 INFO:tasks.workunit.client.0.smithi115.stdout:op 3645 completed, throughput=5MB/sec 2022-04-22T17:58:57.633 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8zrNhJiECyswboL off=2814851 len=323599 2022-04-22T17:58:57.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 3646 completed, throughput=5MB/sec 2022-04-22T17:58:57.634 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2FYHihOdXFZdp-R off=13581950 len=86989 2022-04-22T17:58:57.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 3647 completed, throughput=5MB/sec 2022-04-22T17:58:57.638 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sx2-5YMLuNchnOt off=13406640 len=989118 2022-04-22T17:58:57.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 3648 completed, throughput=5MB/sec 2022-04-22T17:58:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:58 smithi121 bash[36902]: cluster 2022-04-22T17:58:56.638571+0000 mgr.y (mgr.34107) 1173 : cluster [DBG] pgmap v1767: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:58:58 smithi121 bash[36902]: audit 2022-04-22T17:58:57.601460+0000 mon.a (mon.0) 2669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:58 smithi115 bash[45495]: cluster 2022-04-22T17:58:56.638571+0000 mgr.y (mgr.34107) 1173 : cluster [DBG] pgmap v1767: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:58:58 smithi115 bash[45495]: audit 2022-04-22T17:58:57.601460+0000 mon.a (mon.0) 2669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:58 smithi115 bash[46934]: cluster 2022-04-22T17:58:56.638571+0000 mgr.y (mgr.34107) 1173 : cluster [DBG] pgmap v1767: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:58:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:58:58 smithi115 bash[46934]: audit 2022-04-22T17:58:57.601460+0000 mon.a (mon.0) 2669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:58:58.645 INFO:tasks.workunit.client.0.smithi115.stdout: 717: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:58.645 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ffANINQrpU64XzE off=6885364 len=1559127 2022-04-22T17:58:58.655 INFO:tasks.workunit.client.0.smithi115.stdout:op 3649 completed, throughput=5MB/sec 2022-04-22T17:58:58.655 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WAiojNFz9KKmWnI off=10445963 len=766778 2022-04-22T17:58:58.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 3650 completed, throughput=5MB/sec 2022-04-22T17:58:58.661 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-64486FEqJKhGyGJ off=12235504 len=1011917 2022-04-22T17:58:58.667 INFO:tasks.workunit.client.0.smithi115.stdout:op 3651 completed, throughput=5MB/sec 2022-04-22T17:58:58.667 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BZaHEbQU3V-NH2H off=3167289 len=1722267 2022-04-22T17:58:58.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 3652 completed, throughput=5MB/sec 2022-04-22T17:58:58.673 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3b0_OsVKJcYHZkm off=845867 len=573389 2022-04-22T17:58:58.675 INFO:tasks.workunit.client.0.smithi115.stdout:op 3653 completed, throughput=5MB/sec 2022-04-22T17:58:59.675 INFO:tasks.workunit.client.0.smithi115.stdout: 718: throughput=4.99MB/sec pending data=0 2022-04-22T17:58:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lpp40ID9gEceRiS off=4983121 len=2092897 2022-04-22T17:58:59.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 3654 completed, throughput=5MB/sec 2022-04-22T17:58:59.685 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sOgOiwTCuZjcOHa off=10728982 len=1889644 2022-04-22T17:58:59.686 INFO:tasks.workunit.client.0.smithi115.stdout:op 3655 completed, throughput=5MB/sec 2022-04-22T17:58:59.686 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VU7sFOgufkzoOsC off=19276090 len=472414 2022-04-22T17:58:59.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 3656 completed, throughput=5MB/sec 2022-04-22T17:58:59.687 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hQ1oh9sSZ6Kzq0O off=8407241 len=427747 2022-04-22T17:58:59.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 3657 completed, throughput=5MB/sec 2022-04-22T17:59:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:00 smithi121 bash[36902]: cluster 2022-04-22T17:58:58.640131+0000 mgr.y (mgr.34107) 1174 : cluster [DBG] pgmap v1768: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:00.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:00 smithi121 bash[36902]: audit 2022-04-22T17:59:00.217492+0000 mon.a (mon.0) 2670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:00 smithi115 bash[45495]: cluster 2022-04-22T17:58:58.640131+0000 mgr.y (mgr.34107) 1174 : cluster [DBG] pgmap v1768: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:00 smithi115 bash[45495]: audit 2022-04-22T17:59:00.217492+0000 mon.a (mon.0) 2670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:00.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:00 smithi115 bash[46934]: cluster 2022-04-22T17:58:58.640131+0000 mgr.y (mgr.34107) 1174 : cluster [DBG] pgmap v1768: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:00.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:00 smithi115 bash[46934]: audit 2022-04-22T17:59:00.217492+0000 mon.a (mon.0) 2670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:00.689 INFO:tasks.workunit.client.0.smithi115.stdout: 719: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:00.721 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4yVioOAOMUz5PT4 off=6658645 len=157609 2022-04-22T17:59:00.722 INFO:tasks.workunit.client.0.smithi115.stdout:op 3658 completed, throughput=4.99MB/sec 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DmmCST6AovbDvuL off=8514492 len=1511776 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 3659 completed, throughput=5MB/sec 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FUEiaOPvRwB7dlh off=10808588 len=1144987 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 3660 completed, throughput=5MB/sec 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rce5dN3UcsvyFc7 off=1459226 len=1022347 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 3661 completed, throughput=5MB/sec 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MzvYnyW0mOeZ6aV off=674521 len=1465436 2022-04-22T17:59:00.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 3662 completed, throughput=5MB/sec 2022-04-22T17:59:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:01 smithi121 bash[36902]: audit 2022-04-22T17:59:00.584254+0000 mon.a (mon.0) 2671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:01 smithi115 bash[45495]: audit 2022-04-22T17:59:00.584254+0000 mon.a (mon.0) 2671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:01 smithi115 bash[46934]: audit 2022-04-22T17:59:00.584254+0000 mon.a (mon.0) 2671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:01.705 INFO:tasks.workunit.client.0.smithi115.stdout: 720: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:01.705 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ssDNk1wwyth5_NY off=5253571 len=1816598 2022-04-22T17:59:01.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 3663 completed, throughput=5MB/sec 2022-04-22T17:59:01.711 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--1Cd-565flUDtNp off=825335 len=522093 2022-04-22T17:59:01.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 3664 completed, throughput=5MB/sec 2022-04-22T17:59:01.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6kfIqwZ84BQYcEe off=7654327 len=962604 2022-04-22T17:59:01.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 3665 completed, throughput=5MB/sec 2022-04-22T17:59:01.720 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-p5RlD3M1EVJDL0t off=1090330 len=929262 2022-04-22T17:59:01.726 INFO:tasks.workunit.client.0.smithi115.stdout:op 3666 completed, throughput=5MB/sec 2022-04-22T17:59:01.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-__fc9c4yO9Y2npU off=213836 len=2031805 2022-04-22T17:59:01.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 3667 completed, throughput=5MB/sec 2022-04-22T17:59:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:01 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:01.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:02 smithi121 bash[36902]: cluster 2022-04-22T17:59:00.641228+0000 mgr.y (mgr.34107) 1175 : cluster [DBG] pgmap v1769: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:02 smithi115 bash[45495]: cluster 2022-04-22T17:59:00.641228+0000 mgr.y (mgr.34107) 1175 : cluster [DBG] pgmap v1769: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:02 smithi115 bash[46934]: cluster 2022-04-22T17:59:00.641228+0000 mgr.y (mgr.34107) 1175 : cluster [DBG] pgmap v1769: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:02.730 INFO:tasks.workunit.client.0.smithi115.stdout: 721: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:02.731 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jDkZZTx5WeuO_l7 off=1021683 len=1670569 2022-04-22T17:59:02.734 INFO:tasks.workunit.client.0.smithi115.stdout:op 3668 completed, throughput=5MB/sec 2022-04-22T17:59:02.735 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ouViYNMeMi56Jhg off=10443062 len=1470857 2022-04-22T17:59:02.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 3669 completed, throughput=5MB/sec 2022-04-22T17:59:02.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DezqeCqhq4f0Fqp off=2032334 len=1179978 2022-04-22T17:59:02.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 3670 completed, throughput=5MB/sec 2022-04-22T17:59:02.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rP61sbEc3ZJZuQt off=20680664 len=1204867 2022-04-22T17:59:02.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 3671 completed, throughput=5MB/sec 2022-04-22T17:59:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:03 smithi115 bash[45495]: audit 2022-04-22T17:59:02.604304+0000 mon.a (mon.0) 2672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:03 smithi115 bash[46934]: audit 2022-04-22T17:59:02.604304+0000 mon.a (mon.0) 2672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:03.750 INFO:tasks.workunit.client.0.smithi115.stdout: 722: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:03.750 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RPv7jbhq2URS-LZ off=321166 len=1744681 2022-04-22T17:59:03.756 INFO:tasks.workunit.client.0.smithi115.stdout:op 3672 completed, throughput=5MB/sec 2022-04-22T17:59:03.756 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-M77BwV9RTw6YjV4 off=13368554 len=938109 2022-04-22T17:59:03.758 INFO:tasks.workunit.client.0.smithi115.stdout:op 3673 completed, throughput=5MB/sec 2022-04-22T17:59:03.759 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qyG5UFsx-6Xp8c- off=7569756 len=281273 2022-04-22T17:59:03.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 3674 completed, throughput=5MB/sec 2022-04-22T17:59:03.764 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dKCJVjdQeQGYNnT off=3080606 len=523053 2022-04-22T17:59:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:03 smithi121 bash[36902]: audit 2022-04-22T17:59:02.604304+0000 mon.a (mon.0) 2672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:03.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 3675 completed, throughput=5MB/sec 2022-04-22T17:59:03.768 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gRDuVLrMZkXUOPu off=3912787 len=525758 2022-04-22T17:59:03.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 3676 completed, throughput=5MB/sec 2022-04-22T17:59:03.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-27YxWYBdmklF7wZ off=15394010 len=1915079 2022-04-22T17:59:03.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 3677 completed, throughput=5MB/sec 2022-04-22T17:59:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:03 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:03.600Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:04 smithi115 bash[45495]: cluster 2022-04-22T17:59:02.642298+0000 mgr.y (mgr.34107) 1176 : cluster [DBG] pgmap v1770: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:04.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:04 smithi115 bash[46934]: cluster 2022-04-22T17:59:02.642298+0000 mgr.y (mgr.34107) 1176 : cluster [DBG] pgmap v1770: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:04 smithi121 bash[36902]: cluster 2022-04-22T17:59:02.642298+0000 mgr.y (mgr.34107) 1176 : cluster [DBG] pgmap v1770: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:04.776 INFO:tasks.workunit.client.0.smithi115.stdout: 723: throughput=5MB/sec pending data=0 2022-04-22T17:59:04.776 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X-vSwh6tUjG9-sv off=5634619 len=1509124 2022-04-22T17:59:04.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 3678 completed, throughput=5MB/sec 2022-04-22T17:59:04.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-79Yb9sPw9y3l1a1 off=5455022 len=1888093 2022-04-22T17:59:04.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 3679 completed, throughput=5MB/sec 2022-04-22T17:59:04.790 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5FVeH_RGyqx1Rh7 off=8040188 len=924330 2022-04-22T17:59:04.797 INFO:tasks.workunit.client.0.smithi115.stdout:op 3680 completed, throughput=5MB/sec 2022-04-22T17:59:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:05 smithi115 bash[45495]: audit 2022-04-22T17:59:05.220727+0000 mon.a (mon.0) 2673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:05 smithi115 bash[46934]: audit 2022-04-22T17:59:05.220727+0000 mon.a (mon.0) 2673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:05 smithi121 bash[36902]: audit 2022-04-22T17:59:05.220727+0000 mon.a (mon.0) 2673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:05.797 INFO:tasks.workunit.client.0.smithi115.stdout: 724: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:05.797 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qJLe0TsHMry8XLe off=17890760 len=1205861 2022-04-22T17:59:05.804 INFO:tasks.workunit.client.0.smithi115.stdout:op 3681 completed, throughput=5MB/sec 2022-04-22T17:59:05.805 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-okjYNPIxMbFqC7X off=6645509 len=1012538 2022-04-22T17:59:05.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 3682 completed, throughput=5MB/sec 2022-04-22T17:59:05.811 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xrDBd0cU4qktFOC off=11984243 len=262156 2022-04-22T17:59:05.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 3683 completed, throughput=5MB/sec 2022-04-22T17:59:05.813 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S6jZMo69O99oeeK off=271490 len=1367975 2022-04-22T17:59:05.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 3684 completed, throughput=5MB/sec 2022-04-22T17:59:05.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xau_aOzXwDs__aE off=5618123 len=81083 2022-04-22T17:59:05.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 3685 completed, throughput=5MB/sec 2022-04-22T17:59:05.824 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hkrlr0X4BGQgLiT off=9802824 len=770348 2022-04-22T17:59:05.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 3686 completed, throughput=5MB/sec 2022-04-22T17:59:05.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--1LDt-RMmtmbIFn off=295472 len=1755915 2022-04-22T17:59:05.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 3687 completed, throughput=5MB/sec 2022-04-22T17:59:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:06 smithi115 bash[45495]: cluster 2022-04-22T17:59:04.643515+0000 mgr.y (mgr.34107) 1177 : cluster [DBG] pgmap v1771: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:59:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:06 smithi115 bash[45495]: audit 2022-04-22T17:59:05.587884+0000 mon.a (mon.0) 2674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:06 smithi115 bash[46934]: cluster 2022-04-22T17:59:04.643515+0000 mgr.y (mgr.34107) 1177 : cluster [DBG] pgmap v1771: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:59:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:06 smithi115 bash[46934]: audit 2022-04-22T17:59:05.587884+0000 mon.a (mon.0) 2674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:06 smithi121 bash[36902]: cluster 2022-04-22T17:59:04.643515+0000 mgr.y (mgr.34107) 1177 : cluster [DBG] pgmap v1771: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T17:59:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:06 smithi121 bash[36902]: audit 2022-04-22T17:59:05.587884+0000 mon.a (mon.0) 2674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:06.836 INFO:tasks.workunit.client.0.smithi115.stdout: 725: throughput=5MB/sec pending data=0 2022-04-22T17:59:06.836 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AvggRMH62-AAn7d off=2951208 len=1307491 2022-04-22T17:59:06.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 3688 completed, throughput=5MB/sec 2022-04-22T17:59:06.839 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-irSH4OVXtKnEC2C off=2488482 len=1539233 2022-04-22T17:59:06.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 3689 completed, throughput=5MB/sec 2022-04-22T17:59:06.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5Q44dKcodTgKvDX off=3169118 len=1068995 2022-04-22T17:59:06.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 3690 completed, throughput=5MB/sec 2022-04-22T17:59:07.853 INFO:tasks.workunit.client.0.smithi115.stdout: 726: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:07.854 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UtkvwWKnU2rXxWJ off=5097440 len=1523868 2022-04-22T17:59:07.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 3691 completed, throughput=5MB/sec 2022-04-22T17:59:07.863 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-V2QgS5wOoqHivwt off=20710829 len=833023 2022-04-22T17:59:07.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 3692 completed, throughput=5MB/sec 2022-04-22T17:59:07.868 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rDmYxB_3SIWhhqi off=7872608 len=122669 2022-04-22T17:59:07.872 INFO:tasks.workunit.client.0.smithi115.stdout:op 3693 completed, throughput=5MB/sec 2022-04-22T17:59:07.872 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-usuprKbudjEnVZX off=3374103 len=710787 2022-04-22T17:59:07.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 3694 completed, throughput=5MB/sec 2022-04-22T17:59:07.877 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F7M6HBGmH4p3riI off=4589966 len=355726 2022-04-22T17:59:07.878 INFO:tasks.workunit.client.0.smithi115.stdout:op 3695 completed, throughput=5MB/sec 2022-04-22T17:59:07.878 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2it9giWeCfIxyGj off=10193483 len=1798539 2022-04-22T17:59:07.881 INFO:tasks.workunit.client.0.smithi115.stdout:op 3696 completed, throughput=5MB/sec 2022-04-22T17:59:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:08 smithi115 bash[45495]: cluster 2022-04-22T17:59:06.644633+0000 mgr.y (mgr.34107) 1178 : cluster [DBG] pgmap v1772: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:08 smithi115 bash[45495]: audit 2022-04-22T17:59:07.607289+0000 mon.a (mon.0) 2675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:08 smithi115 bash[46934]: cluster 2022-04-22T17:59:06.644633+0000 mgr.y (mgr.34107) 1178 : cluster [DBG] pgmap v1772: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:08 smithi115 bash[46934]: audit 2022-04-22T17:59:07.607289+0000 mon.a (mon.0) 2675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:08 smithi121 bash[36902]: cluster 2022-04-22T17:59:06.644633+0000 mgr.y (mgr.34107) 1178 : cluster [DBG] pgmap v1772: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T17:59:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:08 smithi121 bash[36902]: audit 2022-04-22T17:59:07.607289+0000 mon.a (mon.0) 2675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:08.882 INFO:tasks.workunit.client.0.smithi115.stdout: 727: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:08.882 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ns_VmOeDncd1vi5 off=14637899 len=1563746 2022-04-22T17:59:08.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 3697 completed, throughput=5MB/sec 2022-04-22T17:59:08.887 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yyQmL9Z5B47sIH5 off=3884519 len=657144 2022-04-22T17:59:08.889 INFO:tasks.workunit.client.0.smithi115.stdout:op 3698 completed, throughput=5MB/sec 2022-04-22T17:59:08.890 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gwFMoDMOqwoucCQ off=10318744 len=2082390 2022-04-22T17:59:08.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 3699 completed, throughput=5MB/sec 2022-04-22T17:59:08.895 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5wOylFuAO6pLLfH off=3887648 len=621076 2022-04-22T17:59:08.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 3700 completed, throughput=5MB/sec 2022-04-22T17:59:08.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VQL8TtlLWcQKml2 off=1636269 len=929048 2022-04-22T17:59:08.899 INFO:tasks.workunit.client.0.smithi115.stdout:op 3701 completed, throughput=5MB/sec 2022-04-22T17:59:09.900 INFO:tasks.workunit.client.0.smithi115.stdout: 728: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:09.900 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KaCjpuVpWtqkxvh off=5387834 len=449768 2022-04-22T17:59:09.905 INFO:tasks.workunit.client.0.smithi115.stdout:op 3702 completed, throughput=4.99MB/sec 2022-04-22T17:59:09.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oiQtfQnFpin9aZJ off=7504674 len=1417693 2022-04-22T17:59:09.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 3703 completed, throughput=5MB/sec 2022-04-22T17:59:09.910 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YUrUDy6PROzhSiA off=3679460 len=83896 2022-04-22T17:59:09.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 3704 completed, throughput=5MB/sec 2022-04-22T17:59:09.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AdMT1wQ10InV_8d off=1190145 len=170587 2022-04-22T17:59:09.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 3705 completed, throughput=5MB/sec 2022-04-22T17:59:09.913 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mgTYBU6ouQiFOML off=12085735 len=1707081 2022-04-22T17:59:09.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 3706 completed, throughput=5MB/sec 2022-04-22T17:59:09.916 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7NHOLeGAX49oiMH off=10729708 len=411729 2022-04-22T17:59:09.921 INFO:tasks.workunit.client.0.smithi115.stdout:op 3707 completed, throughput=5MB/sec 2022-04-22T17:59:09.921 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-umQjgTccz5yaj8x off=4130021 len=427918 2022-04-22T17:59:09.925 INFO:tasks.workunit.client.0.smithi115.stdout:op 3708 completed, throughput=5MB/sec 2022-04-22T17:59:09.926 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AIu2jYJkrOvtLnN off=0 len=1721380 2022-04-22T17:59:09.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 3709 completed, throughput=5MB/sec 2022-04-22T17:59:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:10 smithi115 bash[45495]: cluster 2022-04-22T17:59:08.645793+0000 mgr.y (mgr.34107) 1179 : cluster [DBG] pgmap v1773: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:10 smithi115 bash[45495]: audit 2022-04-22T17:59:10.223948+0000 mon.a (mon.0) 2676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:10 smithi115 bash[46934]: cluster 2022-04-22T17:59:08.645793+0000 mgr.y (mgr.34107) 1179 : cluster [DBG] pgmap v1773: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:10 smithi115 bash[46934]: audit 2022-04-22T17:59:10.223948+0000 mon.a (mon.0) 2676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:10 smithi121 bash[36902]: cluster 2022-04-22T17:59:08.645793+0000 mgr.y (mgr.34107) 1179 : cluster [DBG] pgmap v1773: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:10 smithi121 bash[36902]: audit 2022-04-22T17:59:10.223948+0000 mon.a (mon.0) 2676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:10.929 INFO:tasks.workunit.client.0.smithi115.stdout: 729: throughput=5MB/sec pending data=0 2022-04-22T17:59:10.930 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-a44ytHwwx3zfIcJ off=787071 len=1309331 2022-04-22T17:59:10.932 INFO:tasks.workunit.client.0.smithi115.stdout:op 3710 completed, throughput=5MB/sec 2022-04-22T17:59:10.932 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bzcgg1hqzcNg9cG off=5273439 len=1207622 2022-04-22T17:59:10.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 3711 completed, throughput=5MB/sec 2022-04-22T17:59:10.941 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-un06369G2ym1mre off=886405 len=1932689 2022-04-22T17:59:10.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 3712 completed, throughput=5MB/sec 2022-04-22T17:59:11.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:11 smithi115 bash[45495]: audit 2022-04-22T17:59:10.591345+0000 mon.a (mon.0) 2677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:11.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:11 smithi115 bash[46934]: audit 2022-04-22T17:59:10.591345+0000 mon.a (mon.0) 2677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:11 smithi121 bash[36902]: audit 2022-04-22T17:59:10.591345+0000 mon.a (mon.0) 2677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:11.944 INFO:tasks.workunit.client.0.smithi115.stdout: 730: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:11.944 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-69hlnCFlloKs_Hm off=3865458 len=1825309 2022-04-22T17:59:11.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 3713 completed, throughput=5MB/sec 2022-04-22T17:59:11.948 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PtaWpwSRkIibf4x off=4928643 len=1570571 2022-04-22T17:59:11.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 3714 completed, throughput=5MB/sec 2022-04-22T17:59:11.958 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n48yRpeUohZd23y off=11846104 len=20552 2022-04-22T17:59:11.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 3715 completed, throughput=5MB/sec 2022-04-22T17:59:11.961 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s_hNmNzUtYGH7HZ off=6903994 len=1804869 2022-04-22T17:59:11.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 3716 completed, throughput=5MB/sec 2022-04-22T17:59:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:11 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:11.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:12 smithi121 bash[36902]: cluster 2022-04-22T17:59:10.646979+0000 mgr.y (mgr.34107) 1180 : cluster [DBG] pgmap v1774: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:12 smithi115 bash[45495]: cluster 2022-04-22T17:59:10.646979+0000 mgr.y (mgr.34107) 1180 : cluster [DBG] pgmap v1774: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:12 smithi115 bash[46934]: cluster 2022-04-22T17:59:10.646979+0000 mgr.y (mgr.34107) 1180 : cluster [DBG] pgmap v1774: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:12.968 INFO:tasks.workunit.client.0.smithi115.stdout: 731: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:12.969 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4RFETodz6kOKLU0 off=1660154 len=1558149 2022-04-22T17:59:12.979 INFO:tasks.workunit.client.0.smithi115.stdout:op 3717 completed, throughput=5MB/sec 2022-04-22T17:59:12.979 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MepTApR4hsxFsA8 off=7735183 len=873052 2022-04-22T17:59:12.985 INFO:tasks.workunit.client.0.smithi115.stdout:op 3718 completed, throughput=5MB/sec 2022-04-22T17:59:12.985 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-68yKZyXCo56KM8b off=1183781 len=965453 2022-04-22T17:59:12.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 3719 completed, throughput=5MB/sec 2022-04-22T17:59:12.989 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DzHIGWSZYgwvdlW off=1710118 len=1637784 2022-04-22T17:59:12.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 3720 completed, throughput=5MB/sec 2022-04-22T17:59:13.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:13 smithi115 bash[45495]: audit 2022-04-22T17:59:12.610624+0000 mon.a (mon.0) 2678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:13 smithi115 bash[46934]: audit 2022-04-22T17:59:12.610624+0000 mon.a (mon.0) 2678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:13 smithi121 bash[36902]: audit 2022-04-22T17:59:12.610624+0000 mon.a (mon.0) 2678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:13.998 INFO:tasks.workunit.client.0.smithi115.stdout: 732: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:13.999 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-domoWVd5c6yCAIX off=3851445 len=1464195 2022-04-22T17:59:14.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 3721 completed, throughput=5MB/sec 2022-04-22T17:59:14.002 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dmPdMH5wCg1XxiC off=7216149 len=1311797 2022-04-22T17:59:14.010 INFO:tasks.workunit.client.0.smithi115.stdout:op 3722 completed, throughput=5MB/sec 2022-04-22T17:59:14.010 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-68_48ad7YIVlAoT off=11474253 len=1466665 2022-04-22T17:59:14.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 3723 completed, throughput=5MB/sec 2022-04-22T17:59:14.014 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UeieQm_GN6hlph9 off=18976274 len=156848 2022-04-22T17:59:14.015 INFO:tasks.workunit.client.0.smithi115.stdout:op 3724 completed, throughput=5MB/sec 2022-04-22T17:59:14.015 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w5vjaUD5g5qijk8 off=5651968 len=1102421 2022-04-22T17:59:14.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 3725 completed, throughput=5MB/sec 2022-04-22T17:59:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:13 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:13.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:14 smithi121 bash[36902]: cluster 2022-04-22T17:59:12.648116+0000 mgr.y (mgr.34107) 1181 : cluster [DBG] pgmap v1775: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:14.857 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:14 smithi121 bash[36902]: audit 2022-04-22T17:59:13.576519+0000 mon.a (mon.0) 2679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:59:14.858 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:14 smithi121 bash[36902]: audit 2022-04-22T17:59:13.577042+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:59:14.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[45495]: cluster 2022-04-22T17:59:12.648116+0000 mgr.y (mgr.34107) 1181 : cluster [DBG] pgmap v1775: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:14.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[45495]: audit 2022-04-22T17:59:13.576519+0000 mon.a (mon.0) 2679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:59:14.869 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[45495]: audit 2022-04-22T17:59:13.577042+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:59:14.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[46934]: cluster 2022-04-22T17:59:12.648116+0000 mgr.y (mgr.34107) 1181 : cluster [DBG] pgmap v1775: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:14.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[46934]: audit 2022-04-22T17:59:13.576519+0000 mon.a (mon.0) 2679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T17:59:14.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:14 smithi115 bash[46934]: audit 2022-04-22T17:59:13.577042+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T17:59:15.017 INFO:tasks.workunit.client.0.smithi115.stdout: 733: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:15.017 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5TgdE52E_Xx5ebi off=16247114 len=644090 2022-04-22T17:59:15.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 3726 completed, throughput=4.99MB/sec 2022-04-22T17:59:15.024 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--fbrfj7u2f1JBnL off=2964815 len=1704265 2022-04-22T17:59:15.033 INFO:tasks.workunit.client.0.smithi115.stdout:op 3727 completed, throughput=5MB/sec 2022-04-22T17:59:15.033 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LfVR1Mhz0RL3C2U off=20242160 len=1223211 2022-04-22T17:59:15.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 3728 completed, throughput=5MB/sec 2022-04-22T17:59:15.034 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kwKSXl6DnIVVEfs off=9705240 len=1103994 2022-04-22T17:59:15.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 3729 completed, throughput=5MB/sec 2022-04-22T17:59:15.043 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5DofXO7rKm4MB53 off=1942291 len=2012495 2022-04-22T17:59:15.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 3730 completed, throughput=5MB/sec 2022-04-22T17:59:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:15 smithi121 bash[36902]: audit 2022-04-22T17:59:15.226993+0000 mon.a (mon.0) 2681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:15 smithi115 bash[45495]: audit 2022-04-22T17:59:15.226993+0000 mon.a (mon.0) 2681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:15 smithi115 bash[46934]: audit 2022-04-22T17:59:15.226993+0000 mon.a (mon.0) 2681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:16.049 INFO:tasks.workunit.client.0.smithi115.stdout: 734: throughput=5MB/sec pending data=0 2022-04-22T17:59:16.049 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tQ_9eIrbHZ0R8cV off=6078417 len=509196 2022-04-22T17:59:16.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 3731 completed, throughput=5MB/sec 2022-04-22T17:59:16.051 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CV42rN3IgsheFrG off=11715189 len=795058 2022-04-22T17:59:16.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 3732 completed, throughput=5MB/sec 2022-04-22T17:59:16.054 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FtAYxW7h7beLZdS off=11470594 len=74468 2022-04-22T17:59:16.059 INFO:tasks.workunit.client.0.smithi115.stdout:op 3733 completed, throughput=5MB/sec 2022-04-22T17:59:16.059 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xMu4uZJd6oFEmEh off=801409 len=1904975 2022-04-22T17:59:16.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 3734 completed, throughput=5MB/sec 2022-04-22T17:59:16.063 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NTazfEoRs5V3ori off=7005175 len=1785969 2022-04-22T17:59:16.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 3735 completed, throughput=5MB/sec 2022-04-22T17:59:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:16 smithi121 bash[36902]: cluster 2022-04-22T17:59:14.649162+0000 mgr.y (mgr.34107) 1182 : cluster [DBG] pgmap v1776: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:16 smithi121 bash[36902]: audit 2022-04-22T17:59:15.595110+0000 mon.a (mon.0) 2682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:16 smithi115 bash[45495]: cluster 2022-04-22T17:59:14.649162+0000 mgr.y (mgr.34107) 1182 : cluster [DBG] pgmap v1776: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:16 smithi115 bash[45495]: audit 2022-04-22T17:59:15.595110+0000 mon.a (mon.0) 2682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:16 smithi115 bash[46934]: cluster 2022-04-22T17:59:14.649162+0000 mgr.y (mgr.34107) 1182 : cluster [DBG] pgmap v1776: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T17:59:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:16 smithi115 bash[46934]: audit 2022-04-22T17:59:15.595110+0000 mon.a (mon.0) 2682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:17.074 INFO:tasks.workunit.client.0.smithi115.stdout: 735: throughput=5MB/sec pending data=0 2022-04-22T17:59:17.074 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k2_onOJQ01RgI7M off=6614339 len=625827 2022-04-22T17:59:17.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 3736 completed, throughput=5MB/sec 2022-04-22T17:59:17.081 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TsF3RjbZ8NyXbnB off=8102192 len=591518 2022-04-22T17:59:17.085 INFO:tasks.workunit.client.0.smithi115.stdout:op 3737 completed, throughput=5MB/sec 2022-04-22T17:59:17.085 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dj_CzVmi-DemM2p off=11783611 len=632847 2022-04-22T17:59:17.090 INFO:tasks.workunit.client.0.smithi115.stdout:op 3738 completed, throughput=5MB/sec 2022-04-22T17:59:17.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XyABFOG0a4y74Wj off=14259747 len=307329 2022-04-22T17:59:17.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 3739 completed, throughput=5MB/sec 2022-04-22T17:59:17.092 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E4eBrVVHHkkvBtq off=10678094 len=1973575 2022-04-22T17:59:17.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 3740 completed, throughput=5MB/sec 2022-04-22T17:59:18.101 INFO:tasks.workunit.client.0.smithi115.stdout: 736: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:18.101 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3a4upcjn9ZP9X2u off=12046891 len=476875 2022-04-22T17:59:18.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 3741 completed, throughput=4.99MB/sec 2022-04-22T17:59:18.107 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NcGD1ohtnv_pRTI off=1857772 len=1668438 2022-04-22T17:59:18.112 INFO:tasks.workunit.client.0.smithi115.stdout:op 3742 completed, throughput=5MB/sec 2022-04-22T17:59:18.112 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_Tb2BCqc_hoGPMt off=8881105 len=127397 2022-04-22T17:59:18.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 3743 completed, throughput=5MB/sec 2022-04-22T17:59:18.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wc3I65IxOfSQoXe off=16070817 len=2076570 2022-04-22T17:59:18.123 INFO:tasks.workunit.client.0.smithi115.stdout:op 3744 completed, throughput=5MB/sec 2022-04-22T17:59:18.123 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jORfHHqHNPCm8OM off=9131678 len=1826830 2022-04-22T17:59:18.133 INFO:tasks.workunit.client.0.smithi115.stdout:op 3745 completed, throughput=5MB/sec 2022-04-22T17:59:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:18 smithi121 bash[36902]: cluster 2022-04-22T17:59:16.650256+0000 mgr.y (mgr.34107) 1183 : cluster [DBG] pgmap v1777: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:18.934 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:18 smithi121 bash[36902]: audit 2022-04-22T17:59:17.613715+0000 mon.a (mon.0) 2683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:18.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:18 smithi115 bash[46934]: cluster 2022-04-22T17:59:16.650256+0000 mgr.y (mgr.34107) 1183 : cluster [DBG] pgmap v1777: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:18.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:18 smithi115 bash[46934]: audit 2022-04-22T17:59:17.613715+0000 mon.a (mon.0) 2683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:18.936 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:18 smithi115 bash[45495]: cluster 2022-04-22T17:59:16.650256+0000 mgr.y (mgr.34107) 1183 : cluster [DBG] pgmap v1777: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:18.942 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:18 smithi115 bash[45495]: audit 2022-04-22T17:59:17.613715+0000 mon.a (mon.0) 2683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:18.943 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:59:19.133 INFO:tasks.workunit.client.0.smithi115.stdout: 737: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:19.134 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tJq_cjtdvdWEe2K off=14375644 len=986845 2022-04-22T17:59:19.141 INFO:tasks.workunit.client.0.smithi115.stdout:op 3746 completed, throughput=5MB/sec 2022-04-22T17:59:19.141 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k2q2MNpo8QGLs4- off=2203587 len=724737 2022-04-22T17:59:19.146 INFO:tasks.workunit.client.0.smithi115.stdout:op 3747 completed, throughput=5MB/sec 2022-04-22T17:59:19.146 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H2m-8mK0oa3D8dq off=2787917 len=288039 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (22m) 89s ago 36m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (35m) 2m ago 35m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (24m) 89s ago 37m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (23m) 2m ago 37m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (31m) 2m ago 41m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (32m) 89s ago 43m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:59:19.149 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (31m) 89s ago 43m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (30m) 2m ago 41m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (30m) 89s ago 41m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (23m) 89s ago 36m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (23m) 2m ago 36m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (29m) 89s ago 41m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (28m) 89s ago 40m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:59:19.150 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (27m) 89s ago 40m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (27m) 89s ago 39m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (26m) 2m ago 39m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (26m) 2m ago 38m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (25m) 2m ago 38m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (24m) 2m ago 37m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:59:19.151 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (23m) 2m ago 36m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:59:19.152 INFO:tasks.workunit.client.0.smithi115.stdout:op 3748 completed, throughput=5MB/sec 2022-04-22T17:59:19.152 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nXn10ySlReV7PFx off=8038174 len=391142 2022-04-22T17:59:19.152 INFO:tasks.workunit.client.0.smithi115.stdout:op 3749 completed, throughput=5MB/sec 2022-04-22T17:59:19.152 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5tlNu8JiUcLhS5I off=7920120 len=946184 2022-04-22T17:59:19.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 3750 completed, throughput=5MB/sec 2022-04-22T17:59:19.153 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--kmTyByT4o_gAUX off=1048870 len=720513 2022-04-22T17:59:19.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 3751 completed, throughput=5MB/sec 2022-04-22T17:59:19.155 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LlDKpymNlTTB0AU off=12715267 len=1999067 2022-04-22T17:59:19.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 3752 completed, throughput=5MB/sec 2022-04-22T17:59:19.555 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:59:19.555 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:59:19.555 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:59:19.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:19.556 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:59:19.557 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:59:19.557 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:59:19.557 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:59:20.160 INFO:tasks.workunit.client.0.smithi115.stdout: 738: throughput=5MB/sec pending data=0 2022-04-22T17:59:20.160 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KWpwSpTIU9QH85P off=382019 len=211653 2022-04-22T17:59:20.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 3753 completed, throughput=5MB/sec 2022-04-22T17:59:20.165 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dhfCtyKPIP8Pexq off=18482655 len=1093568 2022-04-22T17:59:20.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 3754 completed, throughput=5MB/sec 2022-04-22T17:59:20.171 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kg7c_frRB6SBwfP off=10606642 len=300201 2022-04-22T17:59:20.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 3755 completed, throughput=5MB/sec 2022-04-22T17:59:20.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5SWSdyBo6vvtOb- off=14234 len=1979202 2022-04-22T17:59:20.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 3756 completed, throughput=5MB/sec 2022-04-22T17:59:20.177 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GSdaL5DyG3rSQqo off=1188805 len=1347516 2022-04-22T17:59:20.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 3757 completed, throughput=5MB/sec 2022-04-22T17:59:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:20 smithi121 bash[36902]: cluster 2022-04-22T17:59:18.651373+0000 mgr.y (mgr.34107) 1184 : cluster [DBG] pgmap v1778: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:20 smithi121 bash[36902]: audit 2022-04-22T17:59:18.785179+0000 mgr.y (mgr.34107) 1185 : audit [DBG] from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:20 smithi121 bash[36902]: audit 2022-04-22T17:59:19.126138+0000 mgr.y (mgr.34107) 1186 : audit [DBG] from='client.45589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:20 smithi121 bash[36902]: audit 2022-04-22T17:59:19.548075+0000 mon.a (mon.0) 2684 : audit [DBG] from='client.? 172.21.15.115:0/2705899164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:20 smithi121 bash[36902]: audit 2022-04-22T17:59:20.230502+0000 mon.a (mon.0) 2685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[45495]: cluster 2022-04-22T17:59:18.651373+0000 mgr.y (mgr.34107) 1184 : cluster [DBG] pgmap v1778: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[45495]: audit 2022-04-22T17:59:18.785179+0000 mgr.y (mgr.34107) 1185 : audit [DBG] from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[45495]: audit 2022-04-22T17:59:19.126138+0000 mgr.y (mgr.34107) 1186 : audit [DBG] from='client.45589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[45495]: audit 2022-04-22T17:59:19.548075+0000 mon.a (mon.0) 2684 : audit [DBG] from='client.? 172.21.15.115:0/2705899164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[45495]: audit 2022-04-22T17:59:20.230502+0000 mon.a (mon.0) 2685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[46934]: cluster 2022-04-22T17:59:18.651373+0000 mgr.y (mgr.34107) 1184 : cluster [DBG] pgmap v1778: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[46934]: audit 2022-04-22T17:59:18.785179+0000 mgr.y (mgr.34107) 1185 : audit [DBG] from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[46934]: audit 2022-04-22T17:59:19.126138+0000 mgr.y (mgr.34107) 1186 : audit [DBG] from='client.45589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[46934]: audit 2022-04-22T17:59:19.548075+0000 mon.a (mon.0) 2684 : audit [DBG] from='client.? 172.21.15.115:0/2705899164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:20 smithi115 bash[46934]: audit 2022-04-22T17:59:20.230502+0000 mon.a (mon.0) 2685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:21.184 INFO:tasks.workunit.client.0.smithi115.stdout: 739: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:21.184 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mhAwsvdyWCLqypE off=7665874 len=1276185 2022-04-22T17:59:21.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 3758 completed, throughput=5MB/sec 2022-04-22T17:59:21.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wZy_gRMT7-U5N6u off=9004719 len=1439451 2022-04-22T17:59:21.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 3759 completed, throughput=5MB/sec 2022-04-22T17:59:21.201 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xau_aOzXwDs__aE off=3109812 len=1584130 2022-04-22T17:59:21.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 3760 completed, throughput=5MB/sec 2022-04-22T17:59:21.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:21 smithi115 bash[45495]: audit 2022-04-22T17:59:20.598935+0000 mon.a (mon.0) 2686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:21.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:21 smithi115 bash[46934]: audit 2022-04-22T17:59:20.598935+0000 mon.a (mon.0) 2686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:21 smithi121 bash[36902]: audit 2022-04-22T17:59:20.598935+0000 mon.a (mon.0) 2686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:21 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:21.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:22.206 INFO:tasks.workunit.client.0.smithi115.stdout: 740: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:22.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pNWPPI9ZYPv8nAa off=20278004 len=1660185 2022-04-22T17:59:22.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 3761 completed, throughput=5MB/sec 2022-04-22T17:59:22.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fb8KNx6M4PnpiQa off=19246046 len=724112 2022-04-22T17:59:22.216 INFO:tasks.workunit.client.0.smithi115.stdout:op 3762 completed, throughput=5MB/sec 2022-04-22T17:59:22.217 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VV5HEkCPwDWnskI off=20778214 len=358251 2022-04-22T17:59:22.221 INFO:tasks.workunit.client.0.smithi115.stdout:op 3763 completed, throughput=5MB/sec 2022-04-22T17:59:22.222 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L98tW_LfqUpXrud off=11963278 len=292989 2022-04-22T17:59:22.223 INFO:tasks.workunit.client.0.smithi115.stdout:op 3764 completed, throughput=5MB/sec 2022-04-22T17:59:22.223 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xQN8Nm3LgHbCp31 off=17478132 len=1614638 2022-04-22T17:59:22.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 3765 completed, throughput=5MB/sec 2022-04-22T17:59:22.229 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Gr5_Ip04CyYhglD off=2448594 len=49705 2022-04-22T17:59:22.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 3766 completed, throughput=5MB/sec 2022-04-22T17:59:22.233 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lc7tXiYn_j8a2RY off=9400070 len=982508 2022-04-22T17:59:22.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 3767 completed, throughput=5MB/sec 2022-04-22T17:59:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:22 smithi121 bash[36902]: cluster 2022-04-22T17:59:20.652388+0000 mgr.y (mgr.34107) 1187 : cluster [DBG] pgmap v1779: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:59:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:22 smithi115 bash[45495]: cluster 2022-04-22T17:59:20.652388+0000 mgr.y (mgr.34107) 1187 : cluster [DBG] pgmap v1779: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:59:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:22 smithi115 bash[46934]: cluster 2022-04-22T17:59:20.652388+0000 mgr.y (mgr.34107) 1187 : cluster [DBG] pgmap v1779: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T17:59:23.238 INFO:tasks.workunit.client.0.smithi115.stdout: 741: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:23.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dH8PQ7sZQIPYnxL off=2887306 len=99576 2022-04-22T17:59:23.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 3768 completed, throughput=4.99MB/sec 2022-04-22T17:59:23.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rU66JSnmjL0CCJO off=8618391 len=1511392 2022-04-22T17:59:23.245 INFO:tasks.workunit.client.0.smithi115.stdout:op 3769 completed, throughput=5MB/sec 2022-04-22T17:59:23.245 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yi7wHq9OtcYu803 off=5999135 len=216178 2022-04-22T17:59:23.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 3770 completed, throughput=5MB/sec 2022-04-22T17:59:23.251 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BJYgAQ9GxLOCd7V off=4295907 len=1761104 2022-04-22T17:59:23.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 3771 completed, throughput=5MB/sec 2022-04-22T17:59:23.256 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4gxyhHxJUh5S8rt off=2384732 len=970950 2022-04-22T17:59:23.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 3772 completed, throughput=5MB/sec 2022-04-22T17:59:23.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k0cUR6j26MSrpmq off=3479954 len=1241881 2022-04-22T17:59:23.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 3773 completed, throughput=5MB/sec 2022-04-22T17:59:23.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:23 smithi115 bash[45495]: audit 2022-04-22T17:59:22.616884+0000 mon.a (mon.0) 2687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:23 smithi115 bash[46934]: audit 2022-04-22T17:59:22.616884+0000 mon.a (mon.0) 2687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:23 smithi121 bash[36902]: audit 2022-04-22T17:59:22.616884+0000 mon.a (mon.0) 2687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:23 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:23.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:24.268 INFO:tasks.workunit.client.0.smithi115.stdout: 742: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:24.269 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DHzU4rcv6ZsDkVd off=531861 len=1097330 2022-04-22T17:59:24.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 3774 completed, throughput=5MB/sec 2022-04-22T17:59:24.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2RVt23MFIrpLm23 off=303343 len=1191537 2022-04-22T17:59:24.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 3775 completed, throughput=5MB/sec 2022-04-22T17:59:24.280 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ep5BJbio4SnCBOI off=17475099 len=2010070 2022-04-22T17:59:24.289 INFO:tasks.workunit.client.0.smithi115.stdout:op 3776 completed, throughput=5MB/sec 2022-04-22T17:59:24.289 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yFTuH2IaytscisX off=19825624 len=2080186 2022-04-22T17:59:24.296 INFO:tasks.workunit.client.0.smithi115.stdout:op 3777 completed, throughput=5MB/sec 2022-04-22T17:59:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:24 smithi121 bash[36902]: cluster 2022-04-22T17:59:22.653430+0000 mgr.y (mgr.34107) 1188 : cluster [DBG] pgmap v1780: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:59:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:24 smithi115 bash[45495]: cluster 2022-04-22T17:59:22.653430+0000 mgr.y (mgr.34107) 1188 : cluster [DBG] pgmap v1780: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:59:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:24 smithi115 bash[46934]: cluster 2022-04-22T17:59:22.653430+0000 mgr.y (mgr.34107) 1188 : cluster [DBG] pgmap v1780: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T17:59:25.296 INFO:tasks.workunit.client.0.smithi115.stdout: 743: throughput=5MB/sec pending data=0 2022-04-22T17:59:25.296 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Cr4e88d5v3Kk0YC off=33507 len=306359 2022-04-22T17:59:25.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 3778 completed, throughput=5MB/sec 2022-04-22T17:59:25.303 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m8uzBFjmXMJVGZo off=6805908 len=814987 2022-04-22T17:59:25.308 INFO:tasks.workunit.client.0.smithi115.stdout:op 3779 completed, throughput=5MB/sec 2022-04-22T17:59:25.308 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gBCB2IGU4v0f7we off=963403 len=1213254 2022-04-22T17:59:25.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 3780 completed, throughput=5MB/sec 2022-04-22T17:59:25.311 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c_jS1BIaxQgPdCe off=1605159 len=1829915 2022-04-22T17:59:25.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 3781 completed, throughput=5MB/sec 2022-04-22T17:59:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:25 smithi121 bash[36902]: audit 2022-04-22T17:59:25.233600+0000 mon.a (mon.0) 2688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:25 smithi115 bash[45495]: audit 2022-04-22T17:59:25.233600+0000 mon.a (mon.0) 2688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:25 smithi115 bash[46934]: audit 2022-04-22T17:59:25.233600+0000 mon.a (mon.0) 2688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:26.314 INFO:tasks.workunit.client.0.smithi115.stdout: 744: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:26.314 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wZuy21DtSmjglJn off=8571689 len=1718578 2022-04-22T17:59:26.323 INFO:tasks.workunit.client.0.smithi115.stdout:op 3782 completed, throughput=5MB/sec 2022-04-22T17:59:26.365 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gDHSe1-VjbyKVwY off=1210367 len=1059301 2022-04-22T17:59:26.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 3783 completed, throughput=5MB/sec 2022-04-22T17:59:26.366 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qnettb0fR5ck_00 off=11402239 len=1597031 2022-04-22T17:59:26.366 INFO:tasks.workunit.client.0.smithi115.stdout:op 3784 completed, throughput=5MB/sec 2022-04-22T17:59:26.367 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fupUit61hwvItIe off=13050273 len=1745531 2022-04-22T17:59:26.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 3785 completed, throughput=5MB/sec 2022-04-22T17:59:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:26 smithi121 bash[36902]: cluster 2022-04-22T17:59:24.654497+0000 mgr.y (mgr.34107) 1189 : cluster [DBG] pgmap v1781: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:26.808 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:26 smithi121 bash[36902]: audit 2022-04-22T17:59:25.602702+0000 mon.a (mon.0) 2689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:26.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:26 smithi115 bash[45495]: cluster 2022-04-22T17:59:24.654497+0000 mgr.y (mgr.34107) 1189 : cluster [DBG] pgmap v1781: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:26.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:26 smithi115 bash[45495]: audit 2022-04-22T17:59:25.602702+0000 mon.a (mon.0) 2689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:26.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:26 smithi115 bash[46934]: cluster 2022-04-22T17:59:24.654497+0000 mgr.y (mgr.34107) 1189 : cluster [DBG] pgmap v1781: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:26.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:26 smithi115 bash[46934]: audit 2022-04-22T17:59:25.602702+0000 mon.a (mon.0) 2689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:27.345 INFO:tasks.workunit.client.0.smithi115.stdout: 745: throughput=5MB/sec pending data=0 2022-04-22T17:59:27.346 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yFro023mrv9ogtD off=5182580 len=783976 2022-04-22T17:59:27.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 3786 completed, throughput=5MB/sec 2022-04-22T17:59:27.349 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R_S3fz4omWEDajC off=2151336 len=924169 2022-04-22T17:59:27.351 INFO:tasks.workunit.client.0.smithi115.stdout:op 3787 completed, throughput=5MB/sec 2022-04-22T17:59:27.352 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BUyCWypdRwxExu2 off=1794766 len=1255817 2022-04-22T17:59:27.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 3788 completed, throughput=5MB/sec 2022-04-22T17:59:27.356 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mKjRI0JYM8IDwd0 off=14342841 len=1350341 2022-04-22T17:59:27.359 INFO:tasks.workunit.client.0.smithi115.stdout:op 3789 completed, throughput=5MB/sec 2022-04-22T17:59:28.360 INFO:tasks.workunit.client.0.smithi115.stdout: 746: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:28.360 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PzSbZwrmFbkPuoY off=1470030 len=907194 2022-04-22T17:59:28.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 3790 completed, throughput=4.99MB/sec 2022-04-22T17:59:28.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fWxYzc11ziPJqHX off=364420 len=40958 2022-04-22T17:59:28.368 INFO:tasks.workunit.client.0.smithi115.stdout:op 3791 completed, throughput=4.99MB/sec 2022-04-22T17:59:28.368 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wc3I65IxOfSQoXe off=2698506 len=1790489 2022-04-22T17:59:28.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 3792 completed, throughput=5MB/sec 2022-04-22T17:59:28.376 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_kpRsJ-snfDLy8Y off=17462532 len=1548017 2022-04-22T17:59:28.382 INFO:tasks.workunit.client.0.smithi115.stdout:op 3793 completed, throughput=5MB/sec 2022-04-22T17:59:28.383 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxyrX4kOPDeX2rJ off=17468652 len=700891 2022-04-22T17:59:28.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 3794 completed, throughput=5MB/sec 2022-04-22T17:59:28.384 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BO0HmWhztPjEcaG off=7093712 len=1299393 2022-04-22T17:59:28.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 3795 completed, throughput=5MB/sec 2022-04-22T17:59:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:28 smithi121 bash[36902]: cluster 2022-04-22T17:59:26.655648+0000 mgr.y (mgr.34107) 1190 : cluster [DBG] pgmap v1782: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:28 smithi121 bash[36902]: audit 2022-04-22T17:59:27.620109+0000 mon.a (mon.0) 2690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:28 smithi115 bash[45495]: cluster 2022-04-22T17:59:26.655648+0000 mgr.y (mgr.34107) 1190 : cluster [DBG] pgmap v1782: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:28 smithi115 bash[45495]: audit 2022-04-22T17:59:27.620109+0000 mon.a (mon.0) 2690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:28.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:28 smithi115 bash[46934]: cluster 2022-04-22T17:59:26.655648+0000 mgr.y (mgr.34107) 1190 : cluster [DBG] pgmap v1782: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:28.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:28 smithi115 bash[46934]: audit 2022-04-22T17:59:27.620109+0000 mon.a (mon.0) 2690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:29.388 INFO:tasks.workunit.client.0.smithi115.stdout: 747: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:29.388 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cdMSSo7XsLRuypK off=388705 len=7515 2022-04-22T17:59:29.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 3796 completed, throughput=4.99MB/sec 2022-04-22T17:59:29.392 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PAJz3farZBNCZuM off=1012582 len=851394 2022-04-22T17:59:29.393 INFO:tasks.workunit.client.0.smithi115.stdout:op 3797 completed, throughput=5MB/sec 2022-04-22T17:59:29.393 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n84EXtj4Aj1B6O5 off=844861 len=1598605 2022-04-22T17:59:29.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 3798 completed, throughput=5MB/sec 2022-04-22T17:59:29.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u8Rxj67m8-Y92n7 off=5725778 len=995680 2022-04-22T17:59:29.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 3799 completed, throughput=5MB/sec 2022-04-22T17:59:29.398 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5JnGcR3OW_vBj0v off=1307114 len=833723 2022-04-22T17:59:29.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 3800 completed, throughput=5MB/sec 2022-04-22T17:59:30.400 INFO:tasks.workunit.client.0.smithi115.stdout: 748: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:30.400 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8chql7ldhAQlRZp off=11702078 len=441840 2022-04-22T17:59:30.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 3801 completed, throughput=4.99MB/sec 2022-04-22T17:59:30.406 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jewA2vasJBfsm3j off=53487 len=1328918 2022-04-22T17:59:30.413 INFO:tasks.workunit.client.0.smithi115.stdout:op 3802 completed, throughput=5MB/sec 2022-04-22T17:59:30.413 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-thPWrtEtDJ0Z-LS off=12898164 len=365721 2022-04-22T17:59:30.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 3803 completed, throughput=5MB/sec 2022-04-22T17:59:30.414 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S9IdYWcX_uSOBJl off=17814188 len=1277242 2022-04-22T17:59:30.418 INFO:tasks.workunit.client.0.smithi115.stdout:op 3804 completed, throughput=5MB/sec 2022-04-22T17:59:30.418 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WPb4RaW1o6VvkFp off=11084088 len=491083 2022-04-22T17:59:30.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 3805 completed, throughput=5MB/sec 2022-04-22T17:59:30.423 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AxzvM5QH6_D1wKU off=996548 len=1280409 2022-04-22T17:59:30.426 INFO:tasks.workunit.client.0.smithi115.stdout:op 3806 completed, throughput=5MB/sec 2022-04-22T17:59:30.426 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-neEIjVdIUapqa_L off=331346 len=1222606 2022-04-22T17:59:30.433 INFO:tasks.workunit.client.0.smithi115.stdout:op 3807 completed, throughput=5MB/sec 2022-04-22T17:59:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:30 smithi121 bash[36902]: cluster 2022-04-22T17:59:28.656802+0000 mgr.y (mgr.34107) 1191 : cluster [DBG] pgmap v1783: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:30 smithi121 bash[36902]: audit 2022-04-22T17:59:30.236895+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:30 smithi115 bash[45495]: cluster 2022-04-22T17:59:28.656802+0000 mgr.y (mgr.34107) 1191 : cluster [DBG] pgmap v1783: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:30 smithi115 bash[45495]: audit 2022-04-22T17:59:30.236895+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:30.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:30 smithi115 bash[46934]: cluster 2022-04-22T17:59:28.656802+0000 mgr.y (mgr.34107) 1191 : cluster [DBG] pgmap v1783: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T17:59:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:30 smithi115 bash[46934]: audit 2022-04-22T17:59:30.236895+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:31.433 INFO:tasks.workunit.client.0.smithi115.stdout: 749: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:31.433 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7juRKDNE_pzlUqH off=4719250 len=1223394 2022-04-22T17:59:31.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 3808 completed, throughput=5MB/sec 2022-04-22T17:59:31.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3ifBDUe12Ae334k off=7988311 len=546856 2022-04-22T17:59:31.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 3809 completed, throughput=5MB/sec 2022-04-22T17:59:31.443 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-StPQ5Kkhjcsubpt off=1537898 len=1697654 2022-04-22T17:59:31.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 3810 completed, throughput=5MB/sec 2022-04-22T17:59:31.451 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D-zki4oy5tiI7SH off=5930709 len=1866952 2022-04-22T17:59:31.459 INFO:tasks.workunit.client.0.smithi115.stdout:op 3811 completed, throughput=5MB/sec 2022-04-22T17:59:31.725 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:31 smithi115 bash[45495]: audit 2022-04-22T17:59:30.606305+0000 mon.a (mon.0) 2692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:31.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:31 smithi115 bash[46934]: audit 2022-04-22T17:59:30.606305+0000 mon.a (mon.0) 2692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:31 smithi121 bash[36902]: audit 2022-04-22T17:59:30.606305+0000 mon.a (mon.0) 2692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:31 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:31.718Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:32.459 INFO:tasks.workunit.client.0.smithi115.stdout: 750: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:32.469 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CTxSUCi6mXSgvet off=10073760 len=595936 2022-04-22T17:59:32.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 3812 completed, throughput=5MB/sec 2022-04-22T17:59:32.470 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-M60upIo2C7J1wuq off=5482210 len=466433 2022-04-22T17:59:32.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 3813 completed, throughput=5MB/sec 2022-04-22T17:59:32.471 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Usf00tHdaCXvTCt off=262769 len=991681 2022-04-22T17:59:32.471 INFO:tasks.workunit.client.0.smithi115.stdout:op 3814 completed, throughput=5MB/sec 2022-04-22T17:59:32.471 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ax04zlw-bEAxZYR off=8935467 len=571942 2022-04-22T17:59:32.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 3815 completed, throughput=5MB/sec 2022-04-22T17:59:32.475 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2AIidpfGEmN3FMQ off=3324580 len=1005901 2022-04-22T17:59:32.483 INFO:tasks.workunit.client.0.smithi115.stdout:op 3816 completed, throughput=5MB/sec 2022-04-22T17:59:32.483 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4usLI9y80KF3prx off=7511100 len=1074568 2022-04-22T17:59:32.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 3817 completed, throughput=5MB/sec 2022-04-22T17:59:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:32 smithi121 bash[36902]: cluster 2022-04-22T17:59:30.657866+0000 mgr.y (mgr.34107) 1192 : cluster [DBG] pgmap v1784: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:32 smithi115 bash[45495]: cluster 2022-04-22T17:59:30.657866+0000 mgr.y (mgr.34107) 1192 : cluster [DBG] pgmap v1784: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:32 smithi115 bash[46934]: cluster 2022-04-22T17:59:30.657866+0000 mgr.y (mgr.34107) 1192 : cluster [DBG] pgmap v1784: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:33.489 INFO:tasks.workunit.client.0.smithi115.stdout: 751: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:33.489 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CUUPmYPSWJ0FEJo off=7043425 len=1601974 2022-04-22T17:59:33.493 INFO:tasks.workunit.client.0.smithi115.stdout:op 3818 completed, throughput=5MB/sec 2022-04-22T17:59:33.493 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JMywhazi6199ezM off=2142210 len=794881 2022-04-22T17:59:33.495 INFO:tasks.workunit.client.0.smithi115.stdout:op 3819 completed, throughput=5MB/sec 2022-04-22T17:59:33.495 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_zrj55HR67OF2sb off=4849171 len=139482 2022-04-22T17:59:33.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 3820 completed, throughput=5MB/sec 2022-04-22T17:59:33.497 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8CfwrUNTW6WtE7g off=2643049 len=1770535 2022-04-22T17:59:33.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 3821 completed, throughput=5MB/sec 2022-04-22T17:59:33.507 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5Zz8J7nXufZHjpO off=14629825 len=1177928 2022-04-22T17:59:33.510 INFO:tasks.workunit.client.0.smithi115.stdout:op 3822 completed, throughput=5MB/sec 2022-04-22T17:59:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:33 smithi121 bash[36902]: audit 2022-04-22T17:59:32.623326+0000 mon.a (mon.0) 2693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:33 smithi115 bash[45495]: audit 2022-04-22T17:59:32.623326+0000 mon.a (mon.0) 2693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:33.875 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:33 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:33.602Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:33.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:33 smithi115 bash[46934]: audit 2022-04-22T17:59:32.623326+0000 mon.a (mon.0) 2693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:34.510 INFO:tasks.workunit.client.0.smithi115.stdout: 752: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:34.511 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Var-ShhYlGxDexr off=12466823 len=239957 2022-04-22T17:59:34.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 3823 completed, throughput=4.99MB/sec 2022-04-22T17:59:34.512 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WfWaWZbWoCxiKFJ off=0 len=1221760 2022-04-22T17:59:34.516 INFO:tasks.workunit.client.0.smithi115.stdout:op 3824 completed, throughput=5MB/sec 2022-04-22T17:59:34.516 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mvjy7IcbvHV1NPH off=6135799 len=1366969 2022-04-22T17:59:34.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 3825 completed, throughput=5MB/sec 2022-04-22T17:59:34.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7w-SQYwIpgJeJ3e off=9739880 len=1383884 2022-04-22T17:59:34.527 INFO:tasks.workunit.client.0.smithi115.stdout:op 3826 completed, throughput=5MB/sec 2022-04-22T17:59:34.527 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jks8yf73aH9El5x off=6423462 len=1668375 2022-04-22T17:59:34.534 INFO:tasks.workunit.client.0.smithi115.stdout:op 3827 completed, throughput=5MB/sec 2022-04-22T17:59:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:34 smithi121 bash[36902]: cluster 2022-04-22T17:59:32.658890+0000 mgr.y (mgr.34107) 1193 : cluster [DBG] pgmap v1785: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:59:34.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:34 smithi115 bash[45495]: cluster 2022-04-22T17:59:32.658890+0000 mgr.y (mgr.34107) 1193 : cluster [DBG] pgmap v1785: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:59:34.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:34 smithi115 bash[46934]: cluster 2022-04-22T17:59:32.658890+0000 mgr.y (mgr.34107) 1193 : cluster [DBG] pgmap v1785: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T17:59:35.534 INFO:tasks.workunit.client.0.smithi115.stdout: 753: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:35.535 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jneLISIesmR-AV9 off=3782029 len=1196808 2022-04-22T17:59:35.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 3828 completed, throughput=5MB/sec 2022-04-22T17:59:35.538 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bK2nH26uqi_Wxah off=5333825 len=648661 2022-04-22T17:59:35.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 3829 completed, throughput=5MB/sec 2022-04-22T17:59:35.541 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7kWE9FXnSx9xYb- off=353403 len=215026 2022-04-22T17:59:35.542 INFO:tasks.workunit.client.0.smithi115.stdout:op 3830 completed, throughput=5MB/sec 2022-04-22T17:59:35.542 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RTK2L-6_wB2rQus off=5289432 len=1912991 2022-04-22T17:59:35.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 3831 completed, throughput=5MB/sec 2022-04-22T17:59:35.554 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-njVOThprWSvpdAc off=3780479 len=1844815 2022-04-22T17:59:35.563 INFO:tasks.workunit.client.0.smithi115.stdout:op 3832 completed, throughput=5MB/sec 2022-04-22T17:59:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:35 smithi121 bash[36902]: audit 2022-04-22T17:59:35.240080+0000 mon.a (mon.0) 2694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:35 smithi115 bash[45495]: audit 2022-04-22T17:59:35.240080+0000 mon.a (mon.0) 2694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:35 smithi115 bash[46934]: audit 2022-04-22T17:59:35.240080+0000 mon.a (mon.0) 2694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:36.563 INFO:tasks.workunit.client.0.smithi115.stdout: 754: throughput=5MB/sec pending data=0 2022-04-22T17:59:36.592 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9AZM8wPmEb7JSQ_ off=14977349 len=530417 2022-04-22T17:59:36.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 3833 completed, throughput=5MB/sec 2022-04-22T17:59:36.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u7E4uPYjVgMhVdl off=19339339 len=1424494 2022-04-22T17:59:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 3834 completed, throughput=5MB/sec 2022-04-22T17:59:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2aEpHMObZMV-qZ4 off=2889429 len=1791614 2022-04-22T17:59:36.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 3835 completed, throughput=5MB/sec 2022-04-22T17:59:36.597 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-seAXV-Eyj6g-jgP off=3885601 len=1696540 2022-04-22T17:59:36.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 3836 completed, throughput=5MB/sec 2022-04-22T17:59:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:36 smithi121 bash[36902]: cluster 2022-04-22T17:59:34.660035+0000 mgr.y (mgr.34107) 1194 : cluster [DBG] pgmap v1786: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T17:59:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:36 smithi121 bash[36902]: audit 2022-04-22T17:59:35.610143+0000 mon.a (mon.0) 2695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:36 smithi115 bash[45495]: cluster 2022-04-22T17:59:34.660035+0000 mgr.y (mgr.34107) 1194 : cluster [DBG] pgmap v1786: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T17:59:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:36 smithi115 bash[45495]: audit 2022-04-22T17:59:35.610143+0000 mon.a (mon.0) 2695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:36 smithi115 bash[46934]: cluster 2022-04-22T17:59:34.660035+0000 mgr.y (mgr.34107) 1194 : cluster [DBG] pgmap v1786: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T17:59:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:36 smithi115 bash[46934]: audit 2022-04-22T17:59:35.610143+0000 mon.a (mon.0) 2695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:37.590 INFO:tasks.workunit.client.0.smithi115.stdout: 755: throughput=5MB/sec pending data=0 2022-04-22T17:59:37.590 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7ZgkMTxYnqoPNDX off=1924551 len=918658 2022-04-22T17:59:37.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 3837 completed, throughput=5MB/sec 2022-04-22T17:59:37.597 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y44pM1OKWXl-y5A off=5315724 len=507182 2022-04-22T17:59:37.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 3838 completed, throughput=5MB/sec 2022-04-22T17:59:37.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-48cOehitoRjqFlm off=2022911 len=692953 2022-04-22T17:59:37.607 INFO:tasks.workunit.client.0.smithi115.stdout:op 3839 completed, throughput=5MB/sec 2022-04-22T17:59:37.607 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2iFanGRd75_Lc6c off=550183 len=2078092 2022-04-22T17:59:37.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 3840 completed, throughput=5MB/sec 2022-04-22T17:59:38.613 INFO:tasks.workunit.client.0.smithi115.stdout: 756: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:38.613 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YotmyXFh_3DOtxT off=3815727 len=846184 2022-04-22T17:59:38.616 INFO:tasks.workunit.client.0.smithi115.stdout:op 3841 completed, throughput=5MB/sec 2022-04-22T17:59:38.616 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7K6jRa05hwtZEKF off=1181926 len=832350 2022-04-22T17:59:38.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 3842 completed, throughput=5MB/sec 2022-04-22T17:59:38.618 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BzSSSUXqthTu4kD off=668019 len=2074669 2022-04-22T17:59:38.626 INFO:tasks.workunit.client.0.smithi115.stdout:op 3843 completed, throughput=5MB/sec 2022-04-22T17:59:38.626 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pJLM2OpJ1Hy9_zZ off=10694764 len=315030 2022-04-22T17:59:38.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 3844 completed, throughput=5MB/sec 2022-04-22T17:59:38.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xK8R_Acr9aR4h0H off=6102207 len=1115987 2022-04-22T17:59:38.636 INFO:tasks.workunit.client.0.smithi115.stdout:op 3845 completed, throughput=5MB/sec 2022-04-22T17:59:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:38 smithi121 bash[36902]: cluster 2022-04-22T17:59:36.661112+0000 mgr.y (mgr.34107) 1195 : cluster [DBG] pgmap v1787: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:59:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:38 smithi121 bash[36902]: audit 2022-04-22T17:59:37.626294+0000 mon.a (mon.0) 2696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:38 smithi115 bash[45495]: cluster 2022-04-22T17:59:36.661112+0000 mgr.y (mgr.34107) 1195 : cluster [DBG] pgmap v1787: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:59:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:38 smithi115 bash[45495]: audit 2022-04-22T17:59:37.626294+0000 mon.a (mon.0) 2696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:38 smithi115 bash[46934]: cluster 2022-04-22T17:59:36.661112+0000 mgr.y (mgr.34107) 1195 : cluster [DBG] pgmap v1787: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T17:59:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:38 smithi115 bash[46934]: audit 2022-04-22T17:59:37.626294+0000 mon.a (mon.0) 2696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:39.636 INFO:tasks.workunit.client.0.smithi115.stdout: 757: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:39.636 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p-nyKzfrfjl9SLd off=524508 len=434058 2022-04-22T17:59:39.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 3846 completed, throughput=4.99MB/sec 2022-04-22T17:59:39.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MT0mDSv44fYpA0S off=10172984 len=1997336 2022-04-22T17:59:39.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 3847 completed, throughput=5MB/sec 2022-04-22T17:59:39.644 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-a0Bz-rSNLpPAhAz off=3291484 len=936381 2022-04-22T17:59:39.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 3848 completed, throughput=5MB/sec 2022-04-22T17:59:39.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BabCK5KawJa4lOV off=17126879 len=2037266 2022-04-22T17:59:39.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 3849 completed, throughput=5MB/sec 2022-04-22T17:59:40.657 INFO:tasks.workunit.client.0.smithi115.stdout: 758: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:40.658 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0gZFfyewKEz18SH off=14702502 len=1289093 2022-04-22T17:59:40.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 3850 completed, throughput=5MB/sec 2022-04-22T17:59:40.661 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u1Zz5AYutYXiDsg off=8361226 len=947789 2022-04-22T17:59:40.671 INFO:tasks.workunit.client.0.smithi115.stdout:op 3851 completed, throughput=5MB/sec 2022-04-22T17:59:40.671 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kkiwbaqp9G5tLcm off=6420655 len=2080888 2022-04-22T17:59:40.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 3852 completed, throughput=5MB/sec 2022-04-22T17:59:40.673 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SjM-Oyw0c4QcRiK off=15531099 len=1109762 2022-04-22T17:59:40.675 INFO:tasks.workunit.client.0.smithi115.stdout:op 3853 completed, throughput=5MB/sec 2022-04-22T17:59:40.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:40 smithi121 bash[36902]: cluster 2022-04-22T17:59:38.661982+0000 mgr.y (mgr.34107) 1196 : cluster [DBG] pgmap v1788: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:40.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:40 smithi121 bash[36902]: audit 2022-04-22T17:59:40.242998+0000 mon.a (mon.0) 2697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:40 smithi115 bash[45495]: cluster 2022-04-22T17:59:38.661982+0000 mgr.y (mgr.34107) 1196 : cluster [DBG] pgmap v1788: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:40 smithi115 bash[45495]: audit 2022-04-22T17:59:40.242998+0000 mon.a (mon.0) 2697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:40 smithi115 bash[46934]: cluster 2022-04-22T17:59:38.661982+0000 mgr.y (mgr.34107) 1196 : cluster [DBG] pgmap v1788: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T17:59:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:40 smithi115 bash[46934]: audit 2022-04-22T17:59:40.242998+0000 mon.a (mon.0) 2697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:41.675 INFO:tasks.workunit.client.0.smithi115.stdout: 759: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:41.698 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tj3DhxnLmgz8uDo off=3753587 len=419439 2022-04-22T17:59:41.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 3854 completed, throughput=4.99MB/sec 2022-04-22T17:59:41.699 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3Jg3Cen1HjadWJF off=4162756 len=120257 2022-04-22T17:59:41.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 3855 completed, throughput=4.99MB/sec 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gBQ7s8SbuSMaUHM off=737773 len=1486652 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 3856 completed, throughput=5MB/sec 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uffE3aMDrlwwl7h off=134927 len=1624271 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 3857 completed, throughput=5MB/sec 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3BF4wHc_1u93tZV off=2881009 len=429725 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 3858 completed, throughput=5MB/sec 2022-04-22T17:59:41.700 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SQxcI2DqEa7OPBK off=12086285 len=1115 2022-04-22T17:59:41.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 3859 completed, throughput=5MB/sec 2022-04-22T17:59:41.702 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3r_TitiYwz7Gsra off=665740 len=285561 2022-04-22T17:59:41.703 INFO:tasks.workunit.client.0.smithi115.stdout:op 3860 completed, throughput=5MB/sec 2022-04-22T17:59:41.703 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1mNq64LdJv4Yp24 off=2752323 len=946110 2022-04-22T17:59:41.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 3861 completed, throughput=5MB/sec 2022-04-22T17:59:41.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:41 smithi115 bash[45495]: audit 2022-04-22T17:59:40.613692+0000 mon.a (mon.0) 2698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:41.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:41 smithi115 bash[46934]: audit 2022-04-22T17:59:40.613692+0000 mon.a (mon.0) 2698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:41 smithi121 bash[36902]: audit 2022-04-22T17:59:40.613692+0000 mon.a (mon.0) 2698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:41 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:41.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:42.708 INFO:tasks.workunit.client.0.smithi115.stdout: 760: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:42.709 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_eGpkKF9ZcE50ZW off=2065936 len=1299854 2022-04-22T17:59:42.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 3862 completed, throughput=5MB/sec 2022-04-22T17:59:42.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kqJ1guNZMeVGePY off=8219716 len=853963 2022-04-22T17:59:42.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 3863 completed, throughput=5MB/sec 2022-04-22T17:59:42.721 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IYcIPqPtyUbr6xb off=6147432 len=1831807 2022-04-22T17:59:42.724 INFO:tasks.workunit.client.0.smithi115.stdout:op 3864 completed, throughput=5MB/sec 2022-04-22T17:59:42.724 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-j8T5PompVaEWLn9 off=2965877 len=1615801 2022-04-22T17:59:42.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 3865 completed, throughput=5MB/sec 2022-04-22T17:59:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:42 smithi121 bash[36902]: cluster 2022-04-22T17:59:40.663017+0000 mgr.y (mgr.34107) 1197 : cluster [DBG] pgmap v1789: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:42 smithi115 bash[45495]: cluster 2022-04-22T17:59:40.663017+0000 mgr.y (mgr.34107) 1197 : cluster [DBG] pgmap v1789: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:42 smithi115 bash[46934]: cluster 2022-04-22T17:59:40.663017+0000 mgr.y (mgr.34107) 1197 : cluster [DBG] pgmap v1789: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:43.730 INFO:tasks.workunit.client.0.smithi115.stdout: 761: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:43.730 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HFQriNuAdX1Hqt- off=1833400 len=915053 2022-04-22T17:59:43.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 3866 completed, throughput=5MB/sec 2022-04-22T17:59:43.737 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E3F3-eMti455MeU off=3102291 len=1817846 2022-04-22T17:59:43.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 3867 completed, throughput=5MB/sec 2022-04-22T17:59:43.741 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zaaYh9L1Tt3oO7C off=8451738 len=641227 2022-04-22T17:59:43.746 INFO:tasks.workunit.client.0.smithi115.stdout:op 3868 completed, throughput=5MB/sec 2022-04-22T17:59:43.746 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7YUkqpGTL-Ujcoq off=1764237 len=1530845 2022-04-22T17:59:43.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 3869 completed, throughput=5MB/sec 2022-04-22T17:59:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:43 smithi121 bash[36902]: audit 2022-04-22T17:59:42.628805+0000 mon.a (mon.0) 2699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:43 smithi115 bash[46934]: audit 2022-04-22T17:59:42.628805+0000 mon.a (mon.0) 2699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:43.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:43 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:43.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:43.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:43 smithi115 bash[45495]: audit 2022-04-22T17:59:42.628805+0000 mon.a (mon.0) 2699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:44.751 INFO:tasks.workunit.client.0.smithi115.stdout: 762: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:44.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cpayPV3F_s72L2g off=3027505 len=1860012 2022-04-22T17:59:44.755 INFO:tasks.workunit.client.0.smithi115.stdout:op 3870 completed, throughput=5MB/sec 2022-04-22T17:59:44.755 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vc7Mq2e-0_zvxBd off=6569449 len=1752245 2022-04-22T17:59:44.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 3871 completed, throughput=5MB/sec 2022-04-22T17:59:44.759 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eNl6PzmCmgXh1__ off=10006620 len=1512907 2022-04-22T17:59:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:44 smithi121 bash[36902]: cluster 2022-04-22T17:59:42.664129+0000 mgr.y (mgr.34107) 1198 : cluster [DBG] pgmap v1790: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:44.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 3872 completed, throughput=5MB/sec 2022-04-22T17:59:44.768 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iP-Gqkl7nodsQ5A off=9349554 len=1701426 2022-04-22T17:59:44.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 3873 completed, throughput=5MB/sec 2022-04-22T17:59:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:44 smithi115 bash[45495]: cluster 2022-04-22T17:59:42.664129+0000 mgr.y (mgr.34107) 1198 : cluster [DBG] pgmap v1790: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:44 smithi115 bash[46934]: cluster 2022-04-22T17:59:42.664129+0000 mgr.y (mgr.34107) 1198 : cluster [DBG] pgmap v1790: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T17:59:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:45 smithi121 bash[36902]: cluster 2022-04-22T17:59:44.665334+0000 mgr.y (mgr.34107) 1199 : cluster [DBG] pgmap v1791: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:59:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:45 smithi121 bash[36902]: audit 2022-04-22T17:59:45.246216+0000 mon.a (mon.0) 2700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:45.778 INFO:tasks.workunit.client.0.smithi115.stdout: 763: throughput=5MB/sec pending data=0 2022-04-22T17:59:45.778 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BjbfR2Sn6GzGmSP off=19930619 len=970907 2022-04-22T17:59:45.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 3874 completed, throughput=5MB/sec 2022-04-22T17:59:45.785 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cZ2ahR11haB6Pef off=14475968 len=1045871 2022-04-22T17:59:45.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 3875 completed, throughput=5MB/sec 2022-04-22T17:59:45.788 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FW83wx59PsXte2M off=2347333 len=1033582 2022-04-22T17:59:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:45 smithi115 bash[45495]: cluster 2022-04-22T17:59:44.665334+0000 mgr.y (mgr.34107) 1199 : cluster [DBG] pgmap v1791: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:59:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:45 smithi115 bash[45495]: audit 2022-04-22T17:59:45.246216+0000 mon.a (mon.0) 2700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:45 smithi115 bash[46934]: cluster 2022-04-22T17:59:44.665334+0000 mgr.y (mgr.34107) 1199 : cluster [DBG] pgmap v1791: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T17:59:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:45 smithi115 bash[46934]: audit 2022-04-22T17:59:45.246216+0000 mon.a (mon.0) 2700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:45.795 INFO:tasks.workunit.client.0.smithi115.stdout:op 3876 completed, throughput=5MB/sec 2022-04-22T17:59:45.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_PgKOrwj4-2PBdY off=5990944 len=233921 2022-04-22T17:59:45.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 3877 completed, throughput=5MB/sec 2022-04-22T17:59:45.797 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KjtKwuNVoN8Wjc0 off=41961 len=250458 2022-04-22T17:59:45.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 3878 completed, throughput=5MB/sec 2022-04-22T17:59:45.798 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qhomqL0rxl4bOBX off=11024523 len=251270 2022-04-22T17:59:45.799 INFO:tasks.workunit.client.0.smithi115.stdout:op 3879 completed, throughput=5MB/sec 2022-04-22T17:59:45.799 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D3g24lUQgrbUTVK off=3708554 len=1954833 2022-04-22T17:59:45.804 INFO:tasks.workunit.client.0.smithi115.stdout:op 3880 completed, throughput=5MB/sec 2022-04-22T17:59:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:46 smithi121 bash[36902]: audit 2022-04-22T17:59:45.617440+0000 mon.a (mon.0) 2701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:46.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:46 smithi115 bash[45495]: audit 2022-04-22T17:59:45.617440+0000 mon.a (mon.0) 2701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:46.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:46 smithi115 bash[46934]: audit 2022-04-22T17:59:45.617440+0000 mon.a (mon.0) 2701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:46.803 INFO:tasks.workunit.client.0.smithi115.stdout: 764: throughput=5MB/sec pending data=0 2022-04-22T17:59:46.803 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sop39zVwV7cFvv7 off=7063583 len=918137 2022-04-22T17:59:46.810 INFO:tasks.workunit.client.0.smithi115.stdout:op 3881 completed, throughput=5MB/sec 2022-04-22T17:59:46.810 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8EwPtfPq-rIe6yw off=1592070 len=101893 2022-04-22T17:59:46.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 3882 completed, throughput=5MB/sec 2022-04-22T17:59:46.814 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DKmlgVwvtU14FHD off=492206 len=1454078 2022-04-22T17:59:46.821 INFO:tasks.workunit.client.0.smithi115.stdout:op 3883 completed, throughput=5MB/sec 2022-04-22T17:59:46.821 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cejYx3qVupbU2mQ off=5865057 len=524809 2022-04-22T17:59:46.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 3884 completed, throughput=5MB/sec 2022-04-22T17:59:46.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FGqgR8iY9kgkHEc off=2782101 len=758827 2022-04-22T17:59:46.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 3885 completed, throughput=5MB/sec 2022-04-22T17:59:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:47 smithi121 bash[36902]: cluster 2022-04-22T17:59:46.666476+0000 mgr.y (mgr.34107) 1200 : cluster [DBG] pgmap v1792: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:47 smithi115 bash[46934]: cluster 2022-04-22T17:59:46.666476+0000 mgr.y (mgr.34107) 1200 : cluster [DBG] pgmap v1792: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:47 smithi115 bash[45495]: cluster 2022-04-22T17:59:46.666476+0000 mgr.y (mgr.34107) 1200 : cluster [DBG] pgmap v1792: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:47.829 INFO:tasks.workunit.client.0.smithi115.stdout: 765: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:47.912 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vaGmBVZyJYiRrCP off=10457073 len=1739418 2022-04-22T17:59:47.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 3886 completed, throughput=5MB/sec 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Fm6fo2Dp_aiNfYf off=2417252 len=1907710 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 3887 completed, throughput=5MB/sec 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-as3LLCx0svAiQfy off=1689113 len=717614 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 3888 completed, throughput=5MB/sec 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YFYgw9tw3jbAY9m off=4873520 len=1888890 2022-04-22T17:59:47.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 3889 completed, throughput=5MB/sec 2022-04-22T17:59:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:48 smithi121 bash[36902]: audit 2022-04-22T17:59:47.632185+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:48 smithi115 bash[46934]: audit 2022-04-22T17:59:47.632185+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:48 smithi115 bash[45495]: audit 2022-04-22T17:59:47.632185+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:48.855 INFO:tasks.workunit.client.0.smithi115.stdout: 766: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:48.855 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VO_mJnHI1UFijXv off=2252693 len=1043034 2022-04-22T17:59:48.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 3890 completed, throughput=5MB/sec 2022-04-22T17:59:48.862 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tThk-jzvmhCl9nm off=4065137 len=1673217 2022-04-22T17:59:48.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 3891 completed, throughput=5MB/sec 2022-04-22T17:59:48.869 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vs5VW-KYY5Ctv3v off=678999 len=396361 2022-04-22T17:59:48.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 3892 completed, throughput=5MB/sec 2022-04-22T17:59:48.870 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UROrgumn9T19skg off=1478569 len=1396434 2022-04-22T17:59:48.871 INFO:tasks.workunit.client.0.smithi115.stdout:op 3893 completed, throughput=5MB/sec 2022-04-22T17:59:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:49 smithi121 bash[36902]: cluster 2022-04-22T17:59:48.667694+0000 mgr.y (mgr.34107) 1201 : cluster [DBG] pgmap v1793: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:49.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:49 smithi115 bash[46934]: cluster 2022-04-22T17:59:48.667694+0000 mgr.y (mgr.34107) 1201 : cluster [DBG] pgmap v1793: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:49.938 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:49 smithi115 bash[45495]: cluster 2022-04-22T17:59:48.667694+0000 mgr.y (mgr.34107) 1201 : cluster [DBG] pgmap v1793: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout: 767: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zcxMf1f8t8tOarb off=1218751 len=955227 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 3894 completed, throughput=5MB/sec 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_66BO2ZMNR1JOwt off=6561861 len=1296880 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 3895 completed, throughput=5MB/sec 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2bJKg9e6pA5VFFs off=10609076 len=588676 2022-04-22T17:59:49.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 3896 completed, throughput=5MB/sec 2022-04-22T17:59:49.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m8viPClFoNA2mGj off=0 len=2003045 2022-04-22T17:59:49.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 3897 completed, throughput=5MB/sec 2022-04-22T17:59:49.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VJlOGkOWo8mgdO6 off=21219082 len=632018 2022-04-22T17:59:49.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 3898 completed, throughput=5MB/sec 2022-04-22T17:59:49.940 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T17:59:50.281 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (22m) 2m ago 36m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (36m) 3m ago 36m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (24m) 2m ago 37m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (24m) 3m ago 37m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (32m) 3m ago 41m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (32m) 2m ago 44m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (31m) 2m ago 44m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (30m) 3m ago 42m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T17:59:50.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (31m) 2m ago 42m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (23m) 2m ago 36m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (23m) 3m ago 36m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (29m) 2m ago 41m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (28m) 2m ago 41m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (28m) 2m ago 40m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (28m) 2m ago 40m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (27m) 3m ago 39m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (26m) 3m ago 39m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (25m) 3m ago 38m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (25m) 3m ago 38m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T17:59:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (23m) 3m ago 37m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T17:59:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:49.911786+0000 mgr.y (mgr.34107) 1202 : audit [DBG] from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:50.221585+0000 mon.a (mon.0) 2703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:59:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:50.224131+0000 mon.a (mon.0) 2704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:59:50.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:50.225616+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:59:50.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:50.248798+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:50.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:50 smithi121 bash[36902]: audit 2022-04-22T17:59:50.256313+0000 mgr.y (mgr.34107) 1203 : audit [DBG] from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:49.911786+0000 mgr.y (mgr.34107) 1202 : audit [DBG] from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:50.221585+0000 mon.a (mon.0) 2703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:59:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:50.224131+0000 mon.a (mon.0) 2704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:50.225616+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:50.248798+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[45495]: audit 2022-04-22T17:59:50.256313+0000 mgr.y (mgr.34107) 1203 : audit [DBG] from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:49.911786+0000 mgr.y (mgr.34107) 1202 : audit [DBG] from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:50.221585+0000 mon.a (mon.0) 2703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:50.224131+0000 mon.a (mon.0) 2704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T17:59:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:50.225616+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T17:59:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:50.248798+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:50 smithi115 bash[46934]: audit 2022-04-22T17:59:50.256313+0000 mgr.y (mgr.34107) 1203 : audit [DBG] from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T17:59:50.686 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T17:59:50.686 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T17:59:50.686 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T17:59:50.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T17:59:50.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T17:59:50.688 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T17:59:50.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T17:59:50.688 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T17:59:50.688 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T17:59:50.891 INFO:tasks.workunit.client.0.smithi115.stdout: 768: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:50.891 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Neerp3PzAxJ9FgF off=2790104 len=1031935 2022-04-22T17:59:50.895 INFO:tasks.workunit.client.0.smithi115.stdout:op 3899 completed, throughput=5MB/sec 2022-04-22T17:59:50.895 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-apbojOs2Mdjb_Ex off=1243499 len=415798 2022-04-22T17:59:50.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 3900 completed, throughput=5MB/sec 2022-04-22T17:59:50.898 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CwXP7StnPoGVPR0 off=24089009 len=192557 2022-04-22T17:59:50.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 3901 completed, throughput=5MB/sec 2022-04-22T17:59:50.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z4TsDK2SOtgwTCw off=1055769 len=1059671 2022-04-22T17:59:50.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 3902 completed, throughput=5MB/sec 2022-04-22T17:59:50.908 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OAStGVwvGvhJDZb off=4442553 len=1601591 2022-04-22T17:59:50.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 3903 completed, throughput=5MB/sec 2022-04-22T17:59:50.917 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PMpU_AvqPtYKPpE off=9658989 len=468060 2022-04-22T17:59:50.921 INFO:tasks.workunit.client.0.smithi115.stdout:op 3904 completed, throughput=5MB/sec 2022-04-22T17:59:50.921 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vKGQ2Agt2mfTQdH off=2092127 len=812373 2022-04-22T17:59:50.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 3905 completed, throughput=5MB/sec 2022-04-22T17:59:51.924 INFO:tasks.workunit.client.0.smithi115.stdout: 769: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:51.924 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LUuorHv3czQv4Ev off=1448061 len=2069237 2022-04-22T17:59:51.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 3906 completed, throughput=5MB/sec 2022-04-22T17:59:51.928 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2QB5AL3-99Xg-ft off=204804 len=1203151 2022-04-22T17:59:51.932 INFO:tasks.workunit.client.0.smithi115.stdout:op 3907 completed, throughput=5MB/sec 2022-04-22T17:59:51.932 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WddAyD8UdWCaje6 off=8115837 len=1654160 2022-04-22T17:59:51.935 INFO:tasks.workunit.client.0.smithi115.stdout:op 3908 completed, throughput=5MB/sec 2022-04-22T17:59:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:51 smithi121 bash[36902]: audit 2022-04-22T17:59:50.594557+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:59:52.050 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:51 smithi121 bash[36902]: audit 2022-04-22T17:59:50.620817+0000 mon.a (mon.0) 2708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:52.050 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:51 smithi121 bash[36902]: cluster 2022-04-22T17:59:50.668661+0000 mgr.y (mgr.34107) 1204 : cluster [DBG] pgmap v1794: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:52.050 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:51 smithi121 bash[36902]: audit 2022-04-22T17:59:50.679439+0000 mon.a (mon.0) 2709 : audit [DBG] from='client.? 172.21.15.115:0/4175971058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:52.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:51.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:52.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[45495]: audit 2022-04-22T17:59:50.594557+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:59:52.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[45495]: audit 2022-04-22T17:59:50.620817+0000 mon.a (mon.0) 2708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:52.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[45495]: cluster 2022-04-22T17:59:50.668661+0000 mgr.y (mgr.34107) 1204 : cluster [DBG] pgmap v1794: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:52.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[45495]: audit 2022-04-22T17:59:50.679439+0000 mon.a (mon.0) 2709 : audit [DBG] from='client.? 172.21.15.115:0/4175971058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:52.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[46934]: audit 2022-04-22T17:59:50.594557+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T17:59:52.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[46934]: audit 2022-04-22T17:59:50.620817+0000 mon.a (mon.0) 2708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:52.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[46934]: cluster 2022-04-22T17:59:50.668661+0000 mgr.y (mgr.34107) 1204 : cluster [DBG] pgmap v1794: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T17:59:52.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:51 smithi115 bash[46934]: audit 2022-04-22T17:59:50.679439+0000 mon.a (mon.0) 2709 : audit [DBG] from='client.? 172.21.15.115:0/4175971058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T17:59:52.935 INFO:tasks.workunit.client.0.smithi115.stdout: 770: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:52.936 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UQNw3XUn_hG-gq3 off=10432092 len=1295656 2022-04-22T17:59:52.938 INFO:tasks.workunit.client.0.smithi115.stdout:op 3909 completed, throughput=5MB/sec 2022-04-22T17:59:52.938 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vQwAFV9YJTn1bkc off=9224604 len=1376564 2022-04-22T17:59:52.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 3910 completed, throughput=5MB/sec 2022-04-22T17:59:52.941 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LWGkL8yA-W_3quk off=18962386 len=667340 2022-04-22T17:59:52.945 INFO:tasks.workunit.client.0.smithi115.stdout:op 3911 completed, throughput=5MB/sec 2022-04-22T17:59:52.946 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iJzfXtmg3C25ZvL off=9814696 len=1833001 2022-04-22T17:59:52.955 INFO:tasks.workunit.client.0.smithi115.stdout:op 3912 completed, throughput=5MB/sec 2022-04-22T17:59:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:52 smithi121 bash[36902]: audit 2022-04-22T17:59:52.635327+0000 mon.a (mon.0) 2710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:52 smithi115 bash[45495]: audit 2022-04-22T17:59:52.635327+0000 mon.a (mon.0) 2710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:52 smithi115 bash[46934]: audit 2022-04-22T17:59:52.635327+0000 mon.a (mon.0) 2710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:53.956 INFO:tasks.workunit.client.0.smithi115.stdout: 771: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:54.001 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g6CnlZSoiiz0MLT off=21687126 len=181685 2022-04-22T17:59:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 3913 completed, throughput=4.99MB/sec 2022-04-22T17:59:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q5H_8U-OyPs8kpi off=12690482 len=1259110 2022-04-22T17:59:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 3914 completed, throughput=5MB/sec 2022-04-22T17:59:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0T7gx0A8LJg5Dxq off=16260091 len=1235825 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 3915 completed, throughput=5MB/sec 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gF_brOVcnMPoi-P off=1008530 len=918152 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 3916 completed, throughput=5MB/sec 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RrNjf7FAXPTEbdk off=10046156 len=440824 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 3917 completed, throughput=5MB/sec 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W60fKIjmNwbkkf3 off=4443779 len=1876468 2022-04-22T17:59:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 3918 completed, throughput=5MB/sec 2022-04-22T17:59:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:53 smithi121 bash[36902]: cluster 2022-04-22T17:59:52.669757+0000 mgr.y (mgr.34107) 1205 : cluster [DBG] pgmap v1795: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:53 smithi115 bash[45495]: cluster 2022-04-22T17:59:52.669757+0000 mgr.y (mgr.34107) 1205 : cluster [DBG] pgmap v1795: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:54.059 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 17:59:53 smithi115 bash[60466]: level=error ts=2022-04-22T17:59:53.602Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T17:59:54.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:53 smithi115 bash[46934]: cluster 2022-04-22T17:59:52.669757+0000 mgr.y (mgr.34107) 1205 : cluster [DBG] pgmap v1795: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T17:59:54.987 INFO:tasks.workunit.client.0.smithi115.stdout: 772: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:54.987 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0hX_aYFwAW5k98V off=2304721 len=1037446 2022-04-22T17:59:54.990 INFO:tasks.workunit.client.0.smithi115.stdout:op 3919 completed, throughput=5MB/sec 2022-04-22T17:59:54.990 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YxgFc6DjtV_zvHh off=6735071 len=956167 2022-04-22T17:59:54.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 3920 completed, throughput=5MB/sec 2022-04-22T17:59:54.992 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zY2rmRAPxW07Bjg off=818238 len=651138 2022-04-22T17:59:54.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 3921 completed, throughput=5MB/sec 2022-04-22T17:59:54.994 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ydUXAHUiWo4K46m off=3093753 len=1691725 2022-04-22T17:59:55.004 INFO:tasks.workunit.client.0.smithi115.stdout:op 3922 completed, throughput=5MB/sec 2022-04-22T17:59:55.005 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vdBIMxsSVuUIT5w off=3636460 len=1390728 2022-04-22T17:59:55.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 3923 completed, throughput=5MB/sec 2022-04-22T17:59:56.006 INFO:tasks.workunit.client.0.smithi115.stdout: 773: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:56.007 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DYg1whRHl6th2kK off=6766735 len=1961402 2022-04-22T17:59:56.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 3924 completed, throughput=5MB/sec 2022-04-22T17:59:56.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Fy2x2_05Q-KczA7 off=825004 len=192942 2022-04-22T17:59:56.017 INFO:tasks.workunit.client.0.smithi115.stdout:op 3925 completed, throughput=5MB/sec 2022-04-22T17:59:56.017 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CMbY7UJTYCT7Uuh off=7844841 len=370659 2022-04-22T17:59:56.017 INFO:tasks.workunit.client.0.smithi115.stdout:op 3926 completed, throughput=5MB/sec 2022-04-22T17:59:56.017 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sBYzRCSt9BS1hrV off=2426196 len=1012997 2022-04-22T17:59:56.017 INFO:tasks.workunit.client.0.smithi115.stdout:op 3927 completed, throughput=5MB/sec 2022-04-22T17:59:56.018 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DUtAhYf1SFCSrJX off=3257771 len=1092916 2022-04-22T17:59:56.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:55 smithi121 bash[36902]: cluster 2022-04-22T17:59:54.670831+0000 mgr.y (mgr.34107) 1206 : cluster [DBG] pgmap v1796: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:55 smithi121 bash[36902]: audit 2022-04-22T17:59:55.251348+0000 mon.a (mon.0) 2711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:55 smithi121 bash[36902]: audit 2022-04-22T17:59:55.624185+0000 mon.a (mon.0) 2712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:56.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 3928 completed, throughput=5MB/sec 2022-04-22T17:59:56.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[46934]: cluster 2022-04-22T17:59:54.670831+0000 mgr.y (mgr.34107) 1206 : cluster [DBG] pgmap v1796: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[46934]: audit 2022-04-22T17:59:55.251348+0000 mon.a (mon.0) 2711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[46934]: audit 2022-04-22T17:59:55.624185+0000 mon.a (mon.0) 2712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[45495]: cluster 2022-04-22T17:59:54.670831+0000 mgr.y (mgr.34107) 1206 : cluster [DBG] pgmap v1796: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[45495]: audit 2022-04-22T17:59:55.251348+0000 mon.a (mon.0) 2711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T17:59:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:55 smithi115 bash[45495]: audit 2022-04-22T17:59:55.624185+0000 mon.a (mon.0) 2712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T17:59:57.020 INFO:tasks.workunit.client.0.smithi115.stdout: 774: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:57.020 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yZSJ1ggHReDq9Uq off=11658854 len=1039518 2022-04-22T17:59:57.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 3929 completed, throughput=5MB/sec 2022-04-22T17:59:57.024 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e3fB-NyoiCCk6EJ off=245492 len=1528453 2022-04-22T17:59:57.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 3930 completed, throughput=5MB/sec 2022-04-22T17:59:57.027 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_JVPFc4wjn0Y7dY off=940330 len=266073 2022-04-22T17:59:57.028 INFO:tasks.workunit.client.0.smithi115.stdout:op 3931 completed, throughput=5MB/sec 2022-04-22T17:59:57.028 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8__SBZoqQGd6EkC off=4960024 len=138979 2022-04-22T17:59:57.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 3932 completed, throughput=5MB/sec 2022-04-22T17:59:57.032 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hoslt3Bgpd9pfzq off=11973275 len=1572182 2022-04-22T17:59:57.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 3933 completed, throughput=5MB/sec 2022-04-22T17:59:57.041 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IgCUUBhcLHksVfY off=552902 len=609637 2022-04-22T17:59:57.046 INFO:tasks.workunit.client.0.smithi115.stdout:op 3934 completed, throughput=5MB/sec 2022-04-22T17:59:57.046 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yIAQYH6s3vMCfwk off=7286405 len=555413 2022-04-22T17:59:57.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 3935 completed, throughput=5MB/sec 2022-04-22T17:59:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:57 smithi121 bash[36902]: cluster 2022-04-22T17:59:56.671963+0000 mgr.y (mgr.34107) 1207 : cluster [DBG] pgmap v1797: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:57 smithi121 bash[36902]: audit 2022-04-22T17:59:57.638418+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:57 smithi115 bash[45495]: cluster 2022-04-22T17:59:56.671963+0000 mgr.y (mgr.34107) 1207 : cluster [DBG] pgmap v1797: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:57 smithi115 bash[45495]: audit 2022-04-22T17:59:57.638418+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:57 smithi115 bash[46934]: cluster 2022-04-22T17:59:56.671963+0000 mgr.y (mgr.34107) 1207 : cluster [DBG] pgmap v1797: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T17:59:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:57 smithi115 bash[46934]: audit 2022-04-22T17:59:57.638418+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T17:59:58.051 INFO:tasks.workunit.client.0.smithi115.stdout: 775: throughput=4.99MB/sec pending data=0 2022-04-22T17:59:58.051 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FFuGQUNKwZCDzpL off=20639643 len=987136 2022-04-22T17:59:58.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 3936 completed, throughput=5MB/sec 2022-04-22T17:59:58.054 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--1TDNXWfIscotp- off=3721135 len=930650 2022-04-22T17:59:58.056 INFO:tasks.workunit.client.0.smithi115.stdout:op 3937 completed, throughput=5MB/sec 2022-04-22T17:59:58.056 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9kXXVg5RrsTar8a off=1575187 len=247533 2022-04-22T17:59:58.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 3938 completed, throughput=5MB/sec 2022-04-22T17:59:58.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mb2lIXppUsAKHP6 off=1923095 len=1178895 2022-04-22T17:59:58.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 3939 completed, throughput=5MB/sec 2022-04-22T17:59:58.068 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MJyNR2ykvqtYB5i off=5732061 len=989390 2022-04-22T17:59:58.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 3940 completed, throughput=5MB/sec 2022-04-22T17:59:58.075 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xTCnxl2xJLJjUVn off=482954 len=1598820 2022-04-22T17:59:58.077 INFO:tasks.workunit.client.0.smithi115.stdout:op 3941 completed, throughput=5MB/sec 2022-04-22T17:59:59.077 INFO:tasks.workunit.client.0.smithi115.stdout: 776: throughput=5MB/sec pending data=0 2022-04-22T17:59:59.077 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XuxbuDlr4z8MVR8 off=4944961 len=349807 2022-04-22T17:59:59.079 INFO:tasks.workunit.client.0.smithi115.stdout:op 3942 completed, throughput=5MB/sec 2022-04-22T17:59:59.079 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4N4nh6JM4VvzdbW off=1360913 len=1343116 2022-04-22T17:59:59.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 3943 completed, throughput=5MB/sec 2022-04-22T17:59:59.088 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mqJdoXF-ZVErYGv off=10395943 len=1570790 2022-04-22T17:59:59.096 INFO:tasks.workunit.client.0.smithi115.stdout:op 3944 completed, throughput=5MB/sec 2022-04-22T17:59:59.096 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mG5yIfPqjc51jqX off=20644508 len=1156339 2022-04-22T17:59:59.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 3945 completed, throughput=5MB/sec 2022-04-22T18:00:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 17:59:59 smithi121 bash[36902]: cluster 2022-04-22T17:59:58.673173+0000 mgr.y (mgr.34107) 1208 : cluster [DBG] pgmap v1798: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:00:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 17:59:59 smithi115 bash[45495]: cluster 2022-04-22T17:59:58.673173+0000 mgr.y (mgr.34107) 1208 : cluster [DBG] pgmap v1798: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:00:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 17:59:59 smithi115 bash[46934]: cluster 2022-04-22T17:59:58.673173+0000 mgr.y (mgr.34107) 1208 : cluster [DBG] pgmap v1798: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:00:00.099 INFO:tasks.workunit.client.0.smithi115.stdout: 777: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:00.099 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Aj9kXHmKu5XcX9W off=10342264 len=140971 2022-04-22T18:00:00.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 3946 completed, throughput=4.99MB/sec 2022-04-22T18:00:00.101 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Bf5TEgx6ZruEUmz off=6775165 len=2086779 2022-04-22T18:00:00.106 INFO:tasks.workunit.client.0.smithi115.stdout:op 3947 completed, throughput=5MB/sec 2022-04-22T18:00:00.106 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aMdFFkw67lW8b54 off=0 len=1689038 2022-04-22T18:00:00.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 3948 completed, throughput=5MB/sec 2022-04-22T18:00:00.109 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LeHe9SCkeYggvxm off=10317567 len=212078 2022-04-22T18:00:00.110 INFO:tasks.workunit.client.0.smithi115.stdout:op 3949 completed, throughput=5MB/sec 2022-04-22T18:00:00.110 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Uew1uEwmKwprhf5 off=11355602 len=1733173 2022-04-22T18:00:00.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 3950 completed, throughput=5MB/sec 2022-04-22T18:00:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:00 smithi121 bash[36902]: cluster 2022-04-22T18:00:00.000121+0000 mon.a (mon.0) 2714 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:00:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:00 smithi121 bash[36902]: audit 2022-04-22T18:00:00.254493+0000 mon.a (mon.0) 2715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:00 smithi121 bash[36902]: audit 2022-04-22T18:00:00.627785+0000 mon.a (mon.0) 2716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[45495]: cluster 2022-04-22T18:00:00.000121+0000 mon.a (mon.0) 2714 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:00:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[45495]: audit 2022-04-22T18:00:00.254493+0000 mon.a (mon.0) 2715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[45495]: audit 2022-04-22T18:00:00.627785+0000 mon.a (mon.0) 2716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[46934]: cluster 2022-04-22T18:00:00.000121+0000 mon.a (mon.0) 2714 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:00:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[46934]: audit 2022-04-22T18:00:00.254493+0000 mon.a (mon.0) 2715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:00 smithi115 bash[46934]: audit 2022-04-22T18:00:00.627785+0000 mon.a (mon.0) 2716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:01.119 INFO:tasks.workunit.client.0.smithi115.stdout: 778: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:01.119 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vjc3_zsXWXtAw4m off=6741004 len=32779 2022-04-22T18:00:01.123 INFO:tasks.workunit.client.0.smithi115.stdout:op 3951 completed, throughput=4.99MB/sec 2022-04-22T18:00:01.123 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oWRxu0x_CZAa-Qs off=327549 len=167232 2022-04-22T18:00:01.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 3952 completed, throughput=4.99MB/sec 2022-04-22T18:00:01.124 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x_RKbLnJOR097Ya off=9423108 len=462676 2022-04-22T18:00:01.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 3953 completed, throughput=5MB/sec 2022-04-22T18:00:01.129 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uUIa70yvYYZGjIu off=7081661 len=1782158 2022-04-22T18:00:01.133 INFO:tasks.workunit.client.0.smithi115.stdout:op 3954 completed, throughput=5MB/sec 2022-04-22T18:00:01.225 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-J_GZa4_-4yT-BHM off=2505386 len=1343255 2022-04-22T18:00:01.227 INFO:tasks.workunit.client.0.smithi115.stdout:op 3955 completed, throughput=5MB/sec 2022-04-22T18:00:01.227 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vJjzvXJ_joCAp77 off=933167 len=1196657 2022-04-22T18:00:01.227 INFO:tasks.workunit.client.0.smithi115.stdout:op 3956 completed, throughput=5MB/sec 2022-04-22T18:00:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:01 smithi121 bash[36902]: cluster 2022-04-22T18:00:00.674212+0000 mgr.y (mgr.34107) 1209 : cluster [DBG] pgmap v1799: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:01.720Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:01 smithi115 bash[45495]: cluster 2022-04-22T18:00:00.674212+0000 mgr.y (mgr.34107) 1209 : cluster [DBG] pgmap v1799: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:01 smithi115 bash[46934]: cluster 2022-04-22T18:00:00.674212+0000 mgr.y (mgr.34107) 1209 : cluster [DBG] pgmap v1799: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:02.142 INFO:tasks.workunit.client.0.smithi115.stdout: 779: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:02.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o7szCxo93qB6dA0 off=7763823 len=1907449 2022-04-22T18:00:02.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 3957 completed, throughput=5MB/sec 2022-04-22T18:00:02.148 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aU3R5QrscvZvjQS off=7814654 len=545879 2022-04-22T18:00:02.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 3958 completed, throughput=5MB/sec 2022-04-22T18:00:02.153 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_NdflassBHTNUyX off=2139337 len=1842680 2022-04-22T18:00:02.157 INFO:tasks.workunit.client.0.smithi115.stdout:op 3959 completed, throughput=5MB/sec 2022-04-22T18:00:02.157 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S0_esif8naq6lkq off=1466880 len=999574 2022-04-22T18:00:02.163 INFO:tasks.workunit.client.0.smithi115.stdout:op 3960 completed, throughput=5MB/sec 2022-04-22T18:00:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:02 smithi121 bash[36902]: audit 2022-04-22T18:00:02.641512+0000 mon.a (mon.0) 2717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:02 smithi115 bash[45495]: audit 2022-04-22T18:00:02.641512+0000 mon.a (mon.0) 2717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:02 smithi115 bash[46934]: audit 2022-04-22T18:00:02.641512+0000 mon.a (mon.0) 2717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:03.163 INFO:tasks.workunit.client.0.smithi115.stdout: 780: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:03.163 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y7nGNFqCkKJiMwp off=4573833 len=1578124 2022-04-22T18:00:03.167 INFO:tasks.workunit.client.0.smithi115.stdout:op 3961 completed, throughput=5MB/sec 2022-04-22T18:00:03.167 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K5ZXEvzJYB2bAKq off=87236 len=469073 2022-04-22T18:00:03.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 3962 completed, throughput=5MB/sec 2022-04-22T18:00:03.173 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-seAXV-Eyj6g-jgP off=2900128 len=1130289 2022-04-22T18:00:03.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 3963 completed, throughput=5MB/sec 2022-04-22T18:00:03.181 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vAdWQ7VVPsltWl5 off=125861 len=2030797 2022-04-22T18:00:03.194 INFO:tasks.workunit.client.0.smithi115.stdout:op 3964 completed, throughput=5MB/sec 2022-04-22T18:00:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:03 smithi115 bash[45495]: cluster 2022-04-22T18:00:02.675368+0000 mgr.y (mgr.34107) 1210 : cluster [DBG] pgmap v1800: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:00:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:03.603Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:03 smithi115 bash[46934]: cluster 2022-04-22T18:00:02.675368+0000 mgr.y (mgr.34107) 1210 : cluster [DBG] pgmap v1800: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:00:04.190 INFO:tasks.workunit.client.0.smithi115.stdout: 781: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:04.190 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-36mC9_qVIIP4GId off=7627726 len=1081799 2022-04-22T18:00:04.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 3965 completed, throughput=5MB/sec 2022-04-22T18:00:04.195 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1pm12xYIMlOKoFt off=6507805 len=54494 2022-04-22T18:00:04.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 3966 completed, throughput=5MB/sec 2022-04-22T18:00:04.195 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-frLGWLOMBiNeW83 off=1601599 len=707077 2022-04-22T18:00:04.197 INFO:tasks.workunit.client.0.smithi115.stdout:op 3967 completed, throughput=5MB/sec 2022-04-22T18:00:04.198 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V27DPsmMvtsPINM off=3987293 len=288092 2022-04-22T18:00:04.199 INFO:tasks.workunit.client.0.smithi115.stdout:op 3968 completed, throughput=5MB/sec 2022-04-22T18:00:04.199 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0Up0tLLDshfhliw off=9556530 len=1340442 2022-04-22T18:00:04.208 INFO:tasks.workunit.client.0.smithi115.stdout:op 3969 completed, throughput=5MB/sec 2022-04-22T18:00:04.209 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RYQO_tiSZ9bopNs off=13470801 len=24050 2022-04-22T18:00:04.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 3970 completed, throughput=5MB/sec 2022-04-22T18:00:04.212 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BIWhCc2wNA5_xt8 off=18763958 len=1094580 2022-04-22T18:00:04.218 INFO:tasks.workunit.client.0.smithi115.stdout:op 3971 completed, throughput=5MB/sec 2022-04-22T18:00:04.219 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yTOU8Rxkqc8sI3a off=4833517 len=1051830 2022-04-22T18:00:04.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 3972 completed, throughput=5MB/sec 2022-04-22T18:00:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:03 smithi121 bash[36902]: cluster 2022-04-22T18:00:02.675368+0000 mgr.y (mgr.34107) 1210 : cluster [DBG] pgmap v1800: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:00:05.221 INFO:tasks.workunit.client.0.smithi115.stdout: 782: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:05.279 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4XOWYw9vtX8UIJk off=5354512 len=766345 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 3973 completed, throughput=4.99MB/sec 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EtKdEu780bvithd off=3769918 len=1177708 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 3974 completed, throughput=5MB/sec 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kBcdf4xbSqevDp2 off=6279433 len=1997847 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 3975 completed, throughput=5MB/sec 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xJf4mHbtpZEgp2V off=4320080 len=1580167 2022-04-22T18:00:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 3976 completed, throughput=5MB/sec 2022-04-22T18:00:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[45495]: cluster 2022-04-22T18:00:04.676319+0000 mgr.y (mgr.34107) 1211 : cluster [DBG] pgmap v1801: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[45495]: audit 2022-04-22T18:00:05.257631+0000 mon.a (mon.0) 2718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[45495]: audit 2022-04-22T18:00:05.631277+0000 mon.a (mon.0) 2719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[46934]: cluster 2022-04-22T18:00:04.676319+0000 mgr.y (mgr.34107) 1211 : cluster [DBG] pgmap v1801: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[46934]: audit 2022-04-22T18:00:05.257631+0000 mon.a (mon.0) 2718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:05 smithi115 bash[46934]: audit 2022-04-22T18:00:05.631277+0000 mon.a (mon.0) 2719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:06.246 INFO:tasks.workunit.client.0.smithi115.stdout: 783: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:06.246 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s9Ey-JWyUsvG3xi off=18076336 len=1604266 2022-04-22T18:00:06.254 INFO:tasks.workunit.client.0.smithi115.stdout:op 3977 completed, throughput=5MB/sec 2022-04-22T18:00:06.254 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZYHDW5z1IfsUZwP off=3144381 len=1929915 2022-04-22T18:00:06.261 INFO:tasks.workunit.client.0.smithi115.stdout:op 3978 completed, throughput=5MB/sec 2022-04-22T18:00:06.262 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VI-2YCZmxF646t- off=4027705 len=1760308 2022-04-22T18:00:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:05 smithi121 bash[36902]: cluster 2022-04-22T18:00:04.676319+0000 mgr.y (mgr.34107) 1211 : cluster [DBG] pgmap v1801: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:05 smithi121 bash[36902]: audit 2022-04-22T18:00:05.257631+0000 mon.a (mon.0) 2718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:05 smithi121 bash[36902]: audit 2022-04-22T18:00:05.631277+0000 mon.a (mon.0) 2719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:06.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 3979 completed, throughput=5MB/sec 2022-04-22T18:00:07.267 INFO:tasks.workunit.client.0.smithi115.stdout: 784: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:07.268 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vSlbOT9_Oa-NcnP off=3191224 len=1389473 2022-04-22T18:00:07.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 3980 completed, throughput=5MB/sec 2022-04-22T18:00:07.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cKed-h-D7vIEI4r off=1801545 len=1841078 2022-04-22T18:00:07.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 3981 completed, throughput=5MB/sec 2022-04-22T18:00:07.280 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2McmB_vk4E6U8K2 off=7574667 len=522057 2022-04-22T18:00:07.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 3982 completed, throughput=5MB/sec 2022-04-22T18:00:07.282 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d1D9hi9enexxMGd off=575919 len=25234 2022-04-22T18:00:07.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 3983 completed, throughput=5MB/sec 2022-04-22T18:00:07.283 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kvFFWee7gcX5Wp2 off=16217278 len=521073 2022-04-22T18:00:07.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 3984 completed, throughput=5MB/sec 2022-04-22T18:00:07.287 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K1YtudE30MbvO5x off=3202713 len=1904389 2022-04-22T18:00:07.289 INFO:tasks.workunit.client.0.smithi115.stdout:op 3985 completed, throughput=5MB/sec 2022-04-22T18:00:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:07 smithi115 bash[46934]: cluster 2022-04-22T18:00:06.677422+0000 mgr.y (mgr.34107) 1212 : cluster [DBG] pgmap v1802: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:00:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:07 smithi115 bash[46934]: audit 2022-04-22T18:00:07.644530+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:07 smithi115 bash[45495]: cluster 2022-04-22T18:00:06.677422+0000 mgr.y (mgr.34107) 1212 : cluster [DBG] pgmap v1802: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:00:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:07 smithi115 bash[45495]: audit 2022-04-22T18:00:07.644530+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:07 smithi121 bash[36902]: cluster 2022-04-22T18:00:06.677422+0000 mgr.y (mgr.34107) 1212 : cluster [DBG] pgmap v1802: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:00:08.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:07 smithi121 bash[36902]: audit 2022-04-22T18:00:07.644530+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:08.289 INFO:tasks.workunit.client.0.smithi115.stdout: 785: throughput=5MB/sec pending data=0 2022-04-22T18:00:08.290 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jSWQVM-DanRDdIa off=3953968 len=2043622 2022-04-22T18:00:08.296 INFO:tasks.workunit.client.0.smithi115.stdout:op 3986 completed, throughput=5MB/sec 2022-04-22T18:00:08.296 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DVz3BZv05Yu8UHe off=952926 len=733126 2022-04-22T18:00:08.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 3987 completed, throughput=5MB/sec 2022-04-22T18:00:08.302 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4DXHdl6NkajqXD2 off=23672845 len=973044 2022-04-22T18:00:08.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 3988 completed, throughput=5MB/sec 2022-04-22T18:00:08.308 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3lcpoDD4ThPTCR7 off=12945905 len=588481 2022-04-22T18:00:08.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 3989 completed, throughput=5MB/sec 2022-04-22T18:00:09.314 INFO:tasks.workunit.client.0.smithi115.stdout: 786: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:09.314 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2YGBJFTbS382jvD off=2056340 len=134444 2022-04-22T18:00:09.318 INFO:tasks.workunit.client.0.smithi115.stdout:op 3990 completed, throughput=4.99MB/sec 2022-04-22T18:00:09.318 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-d31YUS_g_e-xFSb off=1857370 len=1727224 2022-04-22T18:00:09.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 3991 completed, throughput=5MB/sec 2022-04-22T18:00:09.321 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ktQmAAWmVDCYGnp off=18798542 len=472166 2022-04-22T18:00:09.327 INFO:tasks.workunit.client.0.smithi115.stdout:op 3992 completed, throughput=5MB/sec 2022-04-22T18:00:09.327 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CsSJpAzEJQnpgN9 off=2422426 len=1947839 2022-04-22T18:00:09.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 3993 completed, throughput=5MB/sec 2022-04-22T18:00:09.338 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TZZmCoCDJgq_6QS off=627525 len=1276751 2022-04-22T18:00:09.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 3994 completed, throughput=5MB/sec 2022-04-22T18:00:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:09 smithi115 bash[45495]: cluster 2022-04-22T18:00:08.678661+0000 mgr.y (mgr.34107) 1213 : cluster [DBG] pgmap v1803: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:09 smithi115 bash[46934]: cluster 2022-04-22T18:00:08.678661+0000 mgr.y (mgr.34107) 1213 : cluster [DBG] pgmap v1803: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:09 smithi121 bash[36902]: cluster 2022-04-22T18:00:08.678661+0000 mgr.y (mgr.34107) 1213 : cluster [DBG] pgmap v1803: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:10.346 INFO:tasks.workunit.client.0.smithi115.stdout: 787: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:10.346 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZHaHzqrqv5BxpcR off=1637447 len=1264090 2022-04-22T18:00:10.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 3995 completed, throughput=5MB/sec 2022-04-22T18:00:10.355 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nRj3HzwWNf5UR73 off=5654706 len=1791662 2022-04-22T18:00:10.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 3996 completed, throughput=5MB/sec 2022-04-22T18:00:10.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qs8qmP95EoPMyFK off=6722144 len=879144 2022-04-22T18:00:10.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 3997 completed, throughput=5MB/sec 2022-04-22T18:00:10.369 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xyddTXNeEjFYoDU off=341174 len=80519 2022-04-22T18:00:10.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 3998 completed, throughput=5MB/sec 2022-04-22T18:00:10.370 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0ceMy_5pdifFDXm off=18082607 len=188917 2022-04-22T18:00:10.374 INFO:tasks.workunit.client.0.smithi115.stdout:op 3999 completed, throughput=5MB/sec 2022-04-22T18:00:10.374 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KJId2bR-yWaDnSk off=9733818 len=1297304 2022-04-22T18:00:10.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 4000 completed, throughput=5MB/sec 2022-04-22T18:00:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:10 smithi115 bash[45495]: audit 2022-04-22T18:00:10.260466+0000 mon.a (mon.0) 2721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:10 smithi115 bash[45495]: audit 2022-04-22T18:00:10.634743+0000 mon.a (mon.0) 2722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:10 smithi115 bash[46934]: audit 2022-04-22T18:00:10.260466+0000 mon.a (mon.0) 2721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:10 smithi115 bash[46934]: audit 2022-04-22T18:00:10.634743+0000 mon.a (mon.0) 2722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:10 smithi121 bash[36902]: audit 2022-04-22T18:00:10.260466+0000 mon.a (mon.0) 2721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:10 smithi121 bash[36902]: audit 2022-04-22T18:00:10.634743+0000 mon.a (mon.0) 2722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:11.377 INFO:tasks.workunit.client.0.smithi115.stdout: 788: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:11.377 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lvwqRBijpnUW9y0 off=3048051 len=464883 2022-04-22T18:00:11.380 INFO:tasks.workunit.client.0.smithi115.stdout:op 4001 completed, throughput=4.99MB/sec 2022-04-22T18:00:11.380 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zPaFTnCu1y7vdbi off=3691961 len=768748 2022-04-22T18:00:11.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 4002 completed, throughput=5MB/sec 2022-04-22T18:00:11.381 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dKTpVDwpgDo8677 off=3631420 len=491677 2022-04-22T18:00:11.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 4003 completed, throughput=5MB/sec 2022-04-22T18:00:11.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OuP8RvJvwlckzlx off=86907 len=1584963 2022-04-22T18:00:11.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 4004 completed, throughput=5MB/sec 2022-04-22T18:00:11.392 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ySPP--s9MM805C2 off=2888091 len=455736 2022-04-22T18:00:11.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 4005 completed, throughput=5MB/sec 2022-04-22T18:00:11.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UCOUGDRfZxqUUrn off=3376645 len=1424416 2022-04-22T18:00:11.404 INFO:tasks.workunit.client.0.smithi115.stdout:op 4006 completed, throughput=5MB/sec 2022-04-22T18:00:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:11 smithi115 bash[45495]: cluster 2022-04-22T18:00:10.679895+0000 mgr.y (mgr.34107) 1214 : cluster [DBG] pgmap v1804: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:11.721Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:11 smithi115 bash[46934]: cluster 2022-04-22T18:00:10.679895+0000 mgr.y (mgr.34107) 1214 : cluster [DBG] pgmap v1804: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:11 smithi121 bash[36902]: cluster 2022-04-22T18:00:10.679895+0000 mgr.y (mgr.34107) 1214 : cluster [DBG] pgmap v1804: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:12.404 INFO:tasks.workunit.client.0.smithi115.stdout: 789: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:12.405 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cQw5LUHQWoCcKiY off=7513225 len=1988789 2022-04-22T18:00:12.415 INFO:tasks.workunit.client.0.smithi115.stdout:op 4007 completed, throughput=5MB/sec 2022-04-22T18:00:12.415 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KJ4ezhFViz2u9pv off=1417634 len=1204057 2022-04-22T18:00:12.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 4008 completed, throughput=5MB/sec 2022-04-22T18:00:12.419 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNZFov-WtuANzsx off=18957 len=676701 2022-04-22T18:00:12.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 4009 completed, throughput=5MB/sec 2022-04-22T18:00:12.421 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s44PSll15TQ_bTV off=18223165 len=849637 2022-04-22T18:00:12.427 INFO:tasks.workunit.client.0.smithi115.stdout:op 4010 completed, throughput=5MB/sec 2022-04-22T18:00:12.427 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HPAIafeW3BrxP_9 off=8036232 len=1783304 2022-04-22T18:00:12.429 INFO:tasks.workunit.client.0.smithi115.stdout:op 4011 completed, throughput=5MB/sec 2022-04-22T18:00:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:12 smithi121 bash[36902]: audit 2022-04-22T18:00:12.647503+0000 mon.a (mon.0) 2723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:12 smithi115 bash[45495]: audit 2022-04-22T18:00:12.647503+0000 mon.a (mon.0) 2723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:12 smithi115 bash[46934]: audit 2022-04-22T18:00:12.647503+0000 mon.a (mon.0) 2723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:13.430 INFO:tasks.workunit.client.0.smithi115.stdout: 790: throughput=5MB/sec pending data=0 2022-04-22T18:00:13.430 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jUQ61Yc6ltcHW11 off=16355623 len=1480128 2022-04-22T18:00:13.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 4012 completed, throughput=5MB/sec 2022-04-22T18:00:13.439 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z_g6cVtlRFrYWyo off=10347660 len=1506576 2022-04-22T18:00:13.442 INFO:tasks.workunit.client.0.smithi115.stdout:op 4013 completed, throughput=5MB/sec 2022-04-22T18:00:13.442 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QccbYVa5jg7nEFD off=16822467 len=341636 2022-04-22T18:00:13.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 4014 completed, throughput=5MB/sec 2022-04-22T18:00:13.446 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-27gJUW3QGYs6Zc9 off=8468744 len=1923677 2022-04-22T18:00:13.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 4015 completed, throughput=5MB/sec 2022-04-22T18:00:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:13.603Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[46934]: cluster 2022-04-22T18:00:12.680960+0000 mgr.y (mgr.34107) 1215 : cluster [DBG] pgmap v1805: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[46934]: audit 2022-04-22T18:00:13.589362+0000 mon.a (mon.0) 2724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:00:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[46934]: audit 2022-04-22T18:00:13.589982+0000 mon.a (mon.0) 2725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:00:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[45495]: cluster 2022-04-22T18:00:12.680960+0000 mgr.y (mgr.34107) 1215 : cluster [DBG] pgmap v1805: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[45495]: audit 2022-04-22T18:00:13.589362+0000 mon.a (mon.0) 2724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:00:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:13 smithi115 bash[45495]: audit 2022-04-22T18:00:13.589982+0000 mon.a (mon.0) 2725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:00:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:13 smithi121 bash[36902]: cluster 2022-04-22T18:00:12.680960+0000 mgr.y (mgr.34107) 1215 : cluster [DBG] pgmap v1805: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:13 smithi121 bash[36902]: audit 2022-04-22T18:00:13.589362+0000 mon.a (mon.0) 2724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:00:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:13 smithi121 bash[36902]: audit 2022-04-22T18:00:13.589982+0000 mon.a (mon.0) 2725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:00:14.456 INFO:tasks.workunit.client.0.smithi115.stdout: 791: throughput=5MB/sec pending data=0 2022-04-22T18:00:14.456 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1wZSn1Dp47fE70n off=23590273 len=944096 2022-04-22T18:00:14.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 4016 completed, throughput=5MB/sec 2022-04-22T18:00:14.464 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eEQ4F3Bjgje2sWr off=4026823 len=854802 2022-04-22T18:00:14.469 INFO:tasks.workunit.client.0.smithi115.stdout:op 4017 completed, throughput=5MB/sec 2022-04-22T18:00:14.470 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mDZCZeosXJuFnFv off=2725918 len=490926 2022-04-22T18:00:14.474 INFO:tasks.workunit.client.0.smithi115.stdout:op 4018 completed, throughput=5MB/sec 2022-04-22T18:00:14.519 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QwSEpMVII0e5tb4 off=2729334 len=1626924 2022-04-22T18:00:14.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 4019 completed, throughput=5MB/sec 2022-04-22T18:00:14.520 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2mqz6sOJkJrCN3S off=3868359 len=492950 2022-04-22T18:00:14.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 4020 completed, throughput=5MB/sec 2022-04-22T18:00:15.484 INFO:tasks.workunit.client.0.smithi115.stdout: 792: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:15.484 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4eLd3yHQDNFTGru off=4664861 len=2002009 2022-04-22T18:00:15.495 INFO:tasks.workunit.client.0.smithi115.stdout:op 4021 completed, throughput=5MB/sec 2022-04-22T18:00:15.495 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-esU6hs_GiLJQRsk off=1832578 len=941491 2022-04-22T18:00:15.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 4022 completed, throughput=5MB/sec 2022-04-22T18:00:15.498 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3B7R5USRXW80II8 off=12341757 len=76972 2022-04-22T18:00:15.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 4023 completed, throughput=5MB/sec 2022-04-22T18:00:15.501 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C8azQkbS0qiTqUh off=9875169 len=689002 2022-04-22T18:00:15.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 4024 completed, throughput=5MB/sec 2022-04-22T18:00:15.505 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d_UruB4aHeA6pSn off=13006476 len=1744407 2022-04-22T18:00:15.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 4025 completed, throughput=5MB/sec 2022-04-22T18:00:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:15 smithi121 bash[36902]: cluster 2022-04-22T18:00:14.682069+0000 mgr.y (mgr.34107) 1216 : cluster [DBG] pgmap v1806: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:16.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:15 smithi121 bash[36902]: audit 2022-04-22T18:00:15.263653+0000 mon.a (mon.0) 2726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:16.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:15 smithi121 bash[36902]: audit 2022-04-22T18:00:15.638625+0000 mon.a (mon.0) 2727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:16.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[45495]: cluster 2022-04-22T18:00:14.682069+0000 mgr.y (mgr.34107) 1216 : cluster [DBG] pgmap v1806: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:16.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[45495]: audit 2022-04-22T18:00:15.263653+0000 mon.a (mon.0) 2726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:16.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[45495]: audit 2022-04-22T18:00:15.638625+0000 mon.a (mon.0) 2727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:16.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[46934]: cluster 2022-04-22T18:00:14.682069+0000 mgr.y (mgr.34107) 1216 : cluster [DBG] pgmap v1806: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:16.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[46934]: audit 2022-04-22T18:00:15.263653+0000 mon.a (mon.0) 2726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:16.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:15 smithi115 bash[46934]: audit 2022-04-22T18:00:15.638625+0000 mon.a (mon.0) 2727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:16.512 INFO:tasks.workunit.client.0.smithi115.stdout: 793: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:16.512 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EjeZG-yfuy_VyNj off=12788012 len=1837467 2022-04-22T18:00:16.522 INFO:tasks.workunit.client.0.smithi115.stdout:op 4026 completed, throughput=5MB/sec 2022-04-22T18:00:16.522 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bbsjXj8jTWEacnN off=4402742 len=1459636 2022-04-22T18:00:16.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 4027 completed, throughput=5MB/sec 2022-04-22T18:00:16.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QxanqXnyiGTC0YM off=9427130 len=1191735 2022-04-22T18:00:16.536 INFO:tasks.workunit.client.0.smithi115.stdout:op 4028 completed, throughput=5MB/sec 2022-04-22T18:00:16.536 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TJ8KnqBG7_ha8HV off=126307 len=2073970 2022-04-22T18:00:16.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 4029 completed, throughput=5MB/sec 2022-04-22T18:00:17.541 INFO:tasks.workunit.client.0.smithi115.stdout: 794: throughput=5MB/sec pending data=0 2022-04-22T18:00:17.542 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zkDh7fgFQNByHNg off=6223531 len=1819729 2022-04-22T18:00:17.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 4030 completed, throughput=5MB/sec 2022-04-22T18:00:17.553 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0fZVzIEEu35jem9 off=175413 len=29237 2022-04-22T18:00:17.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 4031 completed, throughput=5MB/sec 2022-04-22T18:00:17.554 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PZSLGaEuggSCYPb off=13373966 len=2029249 2022-04-22T18:00:17.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 4032 completed, throughput=5MB/sec 2022-04-22T18:00:17.565 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TxQzSzmH-d7jU1s off=7129106 len=1409820 2022-04-22T18:00:17.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 4033 completed, throughput=5MB/sec 2022-04-22T18:00:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:17 smithi121 bash[36902]: cluster 2022-04-22T18:00:16.683205+0000 mgr.y (mgr.34107) 1217 : cluster [DBG] pgmap v1807: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:00:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:17 smithi121 bash[36902]: audit 2022-04-22T18:00:17.650929+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:17 smithi115 bash[45495]: cluster 2022-04-22T18:00:16.683205+0000 mgr.y (mgr.34107) 1217 : cluster [DBG] pgmap v1807: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:00:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:17 smithi115 bash[45495]: audit 2022-04-22T18:00:17.650929+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:17 smithi115 bash[46934]: cluster 2022-04-22T18:00:16.683205+0000 mgr.y (mgr.34107) 1217 : cluster [DBG] pgmap v1807: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:00:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:17 smithi115 bash[46934]: audit 2022-04-22T18:00:17.650929+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:18.573 INFO:tasks.workunit.client.0.smithi115.stdout: 795: throughput=5MB/sec pending data=0 2022-04-22T18:00:18.573 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JPCm6tlWGB391su off=834011 len=1137467 2022-04-22T18:00:18.576 INFO:tasks.workunit.client.0.smithi115.stdout:op 4034 completed, throughput=5MB/sec 2022-04-22T18:00:18.576 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TR5-MV4ZXLu9gzv off=19574066 len=416150 2022-04-22T18:00:18.578 INFO:tasks.workunit.client.0.smithi115.stdout:op 4035 completed, throughput=5MB/sec 2022-04-22T18:00:18.578 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ABTrIZ0m2se5Lcr off=4950926 len=270072 2022-04-22T18:00:18.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 4036 completed, throughput=5MB/sec 2022-04-22T18:00:18.584 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jIei-J-5Dk-r_E7 off=2632615 len=226346 2022-04-22T18:00:18.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 4037 completed, throughput=5MB/sec 2022-04-22T18:00:18.588 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_SzbqF9hjM8nfCa off=2059514 len=1152241 2022-04-22T18:00:18.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 4038 completed, throughput=5MB/sec 2022-04-22T18:00:18.590 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kM3t0Sbgy5V-TMm off=485983 len=772668 2022-04-22T18:00:18.591 INFO:tasks.workunit.client.0.smithi115.stdout:op 4039 completed, throughput=5MB/sec 2022-04-22T18:00:18.592 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bZt96RyxVG4EBHk off=10003279 len=596901 2022-04-22T18:00:18.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 4040 completed, throughput=5MB/sec 2022-04-22T18:00:19.594 INFO:tasks.workunit.client.0.smithi115.stdout: 796: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:19.595 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TQCJY7VGHAcr1JU off=10195293 len=1980503 2022-04-22T18:00:19.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 4041 completed, throughput=5MB/sec 2022-04-22T18:00:19.599 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h7KUCnL9IntRkIU off=1010605 len=1518850 2022-04-22T18:00:19.608 INFO:tasks.workunit.client.0.smithi115.stdout:op 4042 completed, throughput=5MB/sec 2022-04-22T18:00:19.608 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1vZ2zzNr7MCQjyo off=2208412 len=463342 2022-04-22T18:00:19.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 4043 completed, throughput=5MB/sec 2022-04-22T18:00:19.610 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iu8vOCHma0l2eVG off=3956480 len=1542459 2022-04-22T18:00:19.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 4044 completed, throughput=5MB/sec 2022-04-22T18:00:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:19 smithi121 bash[36902]: cluster 2022-04-22T18:00:18.684433+0000 mgr.y (mgr.34107) 1218 : cluster [DBG] pgmap v1808: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T18:00:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:19 smithi115 bash[45495]: cluster 2022-04-22T18:00:18.684433+0000 mgr.y (mgr.34107) 1218 : cluster [DBG] pgmap v1808: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T18:00:20.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:19 smithi115 bash[46934]: cluster 2022-04-22T18:00:18.684433+0000 mgr.y (mgr.34107) 1218 : cluster [DBG] pgmap v1808: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 4 op/s 2022-04-22T18:00:20.617 INFO:tasks.workunit.client.0.smithi115.stdout: 797: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:20.618 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q-ylzNfxBQW1sNI off=10137851 len=1975588 2022-04-22T18:00:20.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 4045 completed, throughput=5MB/sec 2022-04-22T18:00:20.628 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LGi3dtUshl5Dxn_ off=13541047 len=463443 2022-04-22T18:00:20.632 INFO:tasks.workunit.client.0.smithi115.stdout:op 4046 completed, throughput=5MB/sec 2022-04-22T18:00:20.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gNg-00EboldfYGf off=956963 len=1331417 2022-04-22T18:00:20.633 INFO:tasks.workunit.client.0.smithi115.stdout:op 4047 completed, throughput=5MB/sec 2022-04-22T18:00:20.634 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--_pRI4KfQg9KKTw off=4899021 len=2042959 2022-04-22T18:00:20.636 INFO:tasks.workunit.client.0.smithi115.stdout:op 4048 completed, throughput=5MB/sec 2022-04-22T18:00:21.062 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:00:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:20 smithi121 bash[36902]: audit 2022-04-22T18:00:20.266591+0000 mon.a (mon.0) 2729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:20 smithi121 bash[36902]: audit 2022-04-22T18:00:20.642524+0000 mon.a (mon.0) 2730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:20 smithi115 bash[45495]: audit 2022-04-22T18:00:20.266591+0000 mon.a (mon.0) 2729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:20 smithi115 bash[45495]: audit 2022-04-22T18:00:20.642524+0000 mon.a (mon.0) 2730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:20 smithi115 bash[46934]: audit 2022-04-22T18:00:20.266591+0000 mon.a (mon.0) 2729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:20 smithi115 bash[46934]: audit 2022-04-22T18:00:20.642524+0000 mon.a (mon.0) 2730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:21.399 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:00:21.399 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (23m) 2m ago 37m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (36m) 3m ago 36m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (25m) 2m ago 38m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (24m) 3m ago 38m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (32m) 3m ago 42m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (33m) 2m ago 44m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (32m) 2m ago 44m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (31m) 3m ago 42m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (31m) 2m ago 42m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:00:21.400 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (24m) 2m ago 37m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (24m) 3m ago 37m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (30m) 2m ago 42m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (29m) 2m ago 41m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (29m) 2m ago 41m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (28m) 2m ago 40m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (27m) 3m ago 40m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (27m) 3m ago 39m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (26m) 3m ago 39m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (25m) 3m ago 38m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:00:21.401 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (24m) 3m ago 37m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:00:21.637 INFO:tasks.workunit.client.0.smithi115.stdout: 798: throughput=5MB/sec pending data=0 2022-04-22T18:00:21.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hZJhnoeAFPfeCJ8 off=2848579 len=1373154 2022-04-22T18:00:21.640 INFO:tasks.workunit.client.0.smithi115.stdout:op 4049 completed, throughput=5MB/sec 2022-04-22T18:00:21.640 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oL56IjzJ03NeOb7 off=12266163 len=819299 2022-04-22T18:00:21.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 4050 completed, throughput=5MB/sec 2022-04-22T18:00:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jKqMCvgMuneqgL7 off=379421 len=622556 2022-04-22T18:00:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:op 4051 completed, throughput=5MB/sec 2022-04-22T18:00:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O7V3czVDnL60ILa off=9249072 len=779992 2022-04-22T18:00:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:op 4052 completed, throughput=5MB/sec 2022-04-22T18:00:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SOUzdAiFx2gmdn4 off=1210378 len=1128878 2022-04-22T18:00:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:op 4053 completed, throughput=5MB/sec 2022-04-22T18:00:21.809 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:00:21.809 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:00:21.809 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:00:21.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:00:21.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:00:21.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:21.811 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:00:21.811 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:00:21.811 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:00:21.811 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:00:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:21.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[45495]: cluster 2022-04-22T18:00:20.685888+0000 mgr.y (mgr.34107) 1219 : cluster [DBG] pgmap v1809: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:00:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[45495]: audit 2022-04-22T18:00:21.037487+0000 mgr.y (mgr.34107) 1220 : audit [DBG] from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[45495]: audit 2022-04-22T18:00:21.372369+0000 mgr.y (mgr.34107) 1221 : audit [DBG] from='client.45625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[45495]: audit 2022-04-22T18:00:21.801958+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.115:0/2036602227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[46934]: cluster 2022-04-22T18:00:20.685888+0000 mgr.y (mgr.34107) 1219 : cluster [DBG] pgmap v1809: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:00:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[46934]: audit 2022-04-22T18:00:21.037487+0000 mgr.y (mgr.34107) 1220 : audit [DBG] from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[46934]: audit 2022-04-22T18:00:21.372369+0000 mgr.y (mgr.34107) 1221 : audit [DBG] from='client.45625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:21 smithi115 bash[46934]: audit 2022-04-22T18:00:21.801958+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.115:0/2036602227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:21 smithi121 bash[36902]: cluster 2022-04-22T18:00:20.685888+0000 mgr.y (mgr.34107) 1219 : cluster [DBG] pgmap v1809: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:00:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:21 smithi121 bash[36902]: audit 2022-04-22T18:00:21.037487+0000 mgr.y (mgr.34107) 1220 : audit [DBG] from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:21 smithi121 bash[36902]: audit 2022-04-22T18:00:21.372369+0000 mgr.y (mgr.34107) 1221 : audit [DBG] from='client.45625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:21 smithi121 bash[36902]: audit 2022-04-22T18:00:21.801958+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.115:0/2036602227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:22.650 INFO:tasks.workunit.client.0.smithi115.stdout: 799: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:22.650 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TyRYdzGE8DFK2U4 off=4813838 len=862396 2022-04-22T18:00:22.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 4054 completed, throughput=5MB/sec 2022-04-22T18:00:22.657 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6GWvjY6UwtaNF59 off=344367 len=1038083 2022-04-22T18:00:22.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 4055 completed, throughput=5MB/sec 2022-04-22T18:00:22.661 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WG2ChdIWQ08BRkL off=871201 len=1716754 2022-04-22T18:00:22.670 INFO:tasks.workunit.client.0.smithi115.stdout:op 4056 completed, throughput=5MB/sec 2022-04-22T18:00:22.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LoN98Sn6mxTnb0E off=8868377 len=218733 2022-04-22T18:00:22.675 INFO:tasks.workunit.client.0.smithi115.stdout:op 4057 completed, throughput=5MB/sec 2022-04-22T18:00:22.675 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KEagjuqrofU42AS off=10675257 len=475913 2022-04-22T18:00:22.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 4058 completed, throughput=5MB/sec 2022-04-22T18:00:22.679 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mfVeiBSJQah0fho off=4002267 len=1772657 2022-04-22T18:00:22.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 4059 completed, throughput=5MB/sec 2022-04-22T18:00:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:22 smithi121 bash[36902]: audit 2022-04-22T18:00:22.654113+0000 mon.a (mon.0) 2731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:22 smithi115 bash[45495]: audit 2022-04-22T18:00:22.654113+0000 mon.a (mon.0) 2731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:22 smithi115 bash[46934]: audit 2022-04-22T18:00:22.654113+0000 mon.a (mon.0) 2731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:23.683 INFO:tasks.workunit.client.0.smithi115.stdout: 800: throughput=5MB/sec pending data=0 2022-04-22T18:00:23.715 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_npQ2Wm3ODQDv6H off=1886830 len=1446000 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 4060 completed, throughput=5MB/sec 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3kT3wKhIJdOLF8t off=10795410 len=1606997 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 4061 completed, throughput=5MB/sec 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jT4SveDUpJwXrfr off=6605975 len=439403 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 4062 completed, throughput=5MB/sec 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mzilGm7Xux2v3xs off=2558566 len=391431 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 4063 completed, throughput=5MB/sec 2022-04-22T18:00:23.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XpfBROmG46ZcJqa off=12128382 len=526320 2022-04-22T18:00:23.718 INFO:tasks.workunit.client.0.smithi115.stdout:op 4064 completed, throughput=5MB/sec 2022-04-22T18:00:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:23 smithi115 bash[46934]: cluster 2022-04-22T18:00:22.687016+0000 mgr.y (mgr.34107) 1222 : cluster [DBG] pgmap v1810: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:00:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:23.606Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:23 smithi115 bash[45495]: cluster 2022-04-22T18:00:22.687016+0000 mgr.y (mgr.34107) 1222 : cluster [DBG] pgmap v1810: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:00:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:23 smithi121 bash[36902]: cluster 2022-04-22T18:00:22.687016+0000 mgr.y (mgr.34107) 1222 : cluster [DBG] pgmap v1810: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:00:24.705 INFO:tasks.workunit.client.0.smithi115.stdout: 801: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:24.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2UiAfYzgnY0kJFX off=3108974 len=1574643 2022-04-22T18:00:24.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 4065 completed, throughput=5MB/sec 2022-04-22T18:00:24.711 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2ANWlCQmCQMhgb6 off=145472 len=1731461 2022-04-22T18:00:24.715 INFO:tasks.workunit.client.0.smithi115.stdout:op 4066 completed, throughput=5MB/sec 2022-04-22T18:00:24.716 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mvd4d87POp_C4FN off=412048 len=1093963 2022-04-22T18:00:24.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 4067 completed, throughput=5MB/sec 2022-04-22T18:00:24.723 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sn827KsmooJU16p off=15689270 len=1710979 2022-04-22T18:00:24.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 4068 completed, throughput=5MB/sec 2022-04-22T18:00:25.731 INFO:tasks.workunit.client.0.smithi115.stdout: 802: throughput=5MB/sec pending data=0 2022-04-22T18:00:25.731 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R_5B04YdEOKwBsJ off=15034429 len=998128 2022-04-22T18:00:25.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 4069 completed, throughput=5MB/sec 2022-04-22T18:00:25.734 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qc9TdMIA5Lp7mNA off=220743 len=1283373 2022-04-22T18:00:25.741 INFO:tasks.workunit.client.0.smithi115.stdout:op 4070 completed, throughput=5MB/sec 2022-04-22T18:00:25.741 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l4YGTn_aoar-aDY off=2084358 len=704674 2022-04-22T18:00:25.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 4071 completed, throughput=5MB/sec 2022-04-22T18:00:25.747 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-98seqttHf0qJsxf off=89381 len=1994860 2022-04-22T18:00:25.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 4072 completed, throughput=5MB/sec 2022-04-22T18:00:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:25 smithi121 bash[36902]: cluster 2022-04-22T18:00:24.688115+0000 mgr.y (mgr.34107) 1223 : cluster [DBG] pgmap v1811: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:00:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:25 smithi121 bash[36902]: audit 2022-04-22T18:00:25.269865+0000 mon.a (mon.0) 2732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:25 smithi121 bash[36902]: audit 2022-04-22T18:00:25.645827+0000 mon.a (mon.0) 2733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[45495]: cluster 2022-04-22T18:00:24.688115+0000 mgr.y (mgr.34107) 1223 : cluster [DBG] pgmap v1811: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:00:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[45495]: audit 2022-04-22T18:00:25.269865+0000 mon.a (mon.0) 2732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[45495]: audit 2022-04-22T18:00:25.645827+0000 mon.a (mon.0) 2733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[46934]: cluster 2022-04-22T18:00:24.688115+0000 mgr.y (mgr.34107) 1223 : cluster [DBG] pgmap v1811: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:00:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[46934]: audit 2022-04-22T18:00:25.269865+0000 mon.a (mon.0) 2732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:25 smithi115 bash[46934]: audit 2022-04-22T18:00:25.645827+0000 mon.a (mon.0) 2733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:26.751 INFO:tasks.workunit.client.0.smithi115.stdout: 803: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:26.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-puU0e7XGTFJ4g-U off=5882185 len=505187 2022-04-22T18:00:26.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 4073 completed, throughput=5MB/sec 2022-04-22T18:00:26.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NwDY2xutO9UeSdv off=20883828 len=1088714 2022-04-22T18:00:26.755 INFO:tasks.workunit.client.0.smithi115.stdout:op 4074 completed, throughput=5MB/sec 2022-04-22T18:00:26.756 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3wffABXMzRMmccB off=1368389 len=1547668 2022-04-22T18:00:26.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 4075 completed, throughput=5MB/sec 2022-04-22T18:00:26.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TWK-6f_UZGJiRfn off=9942606 len=1063440 2022-04-22T18:00:26.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 4076 completed, throughput=5MB/sec 2022-04-22T18:00:26.762 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3J2etMdF0xvtvQ off=15622991 len=299334 2022-04-22T18:00:26.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 4077 completed, throughput=5MB/sec 2022-04-22T18:00:26.768 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-quZV13EIW3x5fz9 off=4893472 len=997155 2022-04-22T18:00:26.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 4078 completed, throughput=5MB/sec 2022-04-22T18:00:27.774 INFO:tasks.workunit.client.0.smithi115.stdout: 804: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:27.775 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TCntVHkkrgABV73 off=21394 len=1042111 2022-04-22T18:00:27.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 4079 completed, throughput=5MB/sec 2022-04-22T18:00:27.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bb8aQb3SSvMQ9pU off=4123219 len=1499532 2022-04-22T18:00:27.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 4080 completed, throughput=5MB/sec 2022-04-22T18:00:27.867 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HIzIR1cI2n71p9f off=5921085 len=1172016 2022-04-22T18:00:27.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 4081 completed, throughput=5MB/sec 2022-04-22T18:00:27.868 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-niAmAXv1VmGjJE9 off=6828738 len=1141306 2022-04-22T18:00:27.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 4082 completed, throughput=5MB/sec 2022-04-22T18:00:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:27 smithi121 bash[36902]: cluster 2022-04-22T18:00:26.689178+0000 mgr.y (mgr.34107) 1224 : cluster [DBG] pgmap v1812: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:00:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:27 smithi121 bash[36902]: audit 2022-04-22T18:00:27.657197+0000 mon.a (mon.0) 2734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:27 smithi115 bash[45495]: cluster 2022-04-22T18:00:26.689178+0000 mgr.y (mgr.34107) 1224 : cluster [DBG] pgmap v1812: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:00:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:27 smithi115 bash[45495]: audit 2022-04-22T18:00:27.657197+0000 mon.a (mon.0) 2734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:27 smithi115 bash[46934]: cluster 2022-04-22T18:00:26.689178+0000 mgr.y (mgr.34107) 1224 : cluster [DBG] pgmap v1812: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:00:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:27 smithi115 bash[46934]: audit 2022-04-22T18:00:27.657197+0000 mon.a (mon.0) 2734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:28.791 INFO:tasks.workunit.client.0.smithi115.stdout: 805: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:28.906 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D1ksDmY8kjoZuhn off=6496113 len=148709 2022-04-22T18:00:28.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 4083 completed, throughput=4.99MB/sec 2022-04-22T18:00:28.908 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VlmvEXwskPl_fp2 off=6289644 len=771970 2022-04-22T18:00:28.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 4084 completed, throughput=5MB/sec 2022-04-22T18:00:28.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Kl6013KqR4XN8Sf off=11016867 len=421960 2022-04-22T18:00:28.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 4085 completed, throughput=5MB/sec 2022-04-22T18:00:28.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z68lJqZg7WFIw2V off=5710639 len=1897461 2022-04-22T18:00:28.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 4086 completed, throughput=5MB/sec 2022-04-22T18:00:28.912 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mON9NN2cXLZmDQP off=8864649 len=1778048 2022-04-22T18:00:28.912 INFO:tasks.workunit.client.0.smithi115.stdout:op 4087 completed, throughput=5MB/sec 2022-04-22T18:00:28.912 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y05DCWR_8vdJBdy off=1283908 len=1728236 2022-04-22T18:00:28.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 4088 completed, throughput=5MB/sec 2022-04-22T18:00:29.820 INFO:tasks.workunit.client.0.smithi115.stdout: 806: throughput=5MB/sec pending data=0 2022-04-22T18:00:29.820 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O_qhOpevPQBf0PU off=76672 len=442490 2022-04-22T18:00:29.822 INFO:tasks.workunit.client.0.smithi115.stdout:op 4089 completed, throughput=5MB/sec 2022-04-22T18:00:29.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YryeYz7gg0MDWP- off=4069366 len=11660 2022-04-22T18:00:29.827 INFO:tasks.workunit.client.0.smithi115.stdout:op 4090 completed, throughput=5MB/sec 2022-04-22T18:00:29.827 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6SDNdYrCzDpFLjR off=7690835 len=1455797 2022-04-22T18:00:29.831 INFO:tasks.workunit.client.0.smithi115.stdout:op 4091 completed, throughput=5MB/sec 2022-04-22T18:00:29.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JRqy0jheAyfiySi off=8131887 len=40796 2022-04-22T18:00:29.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 4092 completed, throughput=5MB/sec 2022-04-22T18:00:29.835 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7r9Fx84YZWF5AwW off=4312611 len=660516 2022-04-22T18:00:29.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 4093 completed, throughput=5MB/sec 2022-04-22T18:00:29.837 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8yEWcuClg6fMgXw off=15118420 len=493862 2022-04-22T18:00:29.841 INFO:tasks.workunit.client.0.smithi115.stdout:op 4094 completed, throughput=5MB/sec 2022-04-22T18:00:29.841 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qQ3bDh1_zLVuuZY off=7652041 len=332005 2022-04-22T18:00:29.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 4095 completed, throughput=5MB/sec 2022-04-22T18:00:29.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QaRhYLjCHmwXsBm off=9908804 len=501817 2022-04-22T18:00:29.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 4096 completed, throughput=5MB/sec 2022-04-22T18:00:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:29 smithi121 bash[36902]: cluster 2022-04-22T18:00:28.690383+0000 mgr.y (mgr.34107) 1225 : cluster [DBG] pgmap v1813: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:29 smithi115 bash[45495]: cluster 2022-04-22T18:00:28.690383+0000 mgr.y (mgr.34107) 1225 : cluster [DBG] pgmap v1813: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:29 smithi115 bash[46934]: cluster 2022-04-22T18:00:28.690383+0000 mgr.y (mgr.34107) 1225 : cluster [DBG] pgmap v1813: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:30.846 INFO:tasks.workunit.client.0.smithi115.stdout: 807: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:30.847 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--_pRI4KfQg9KKTw off=1009239 len=224140 2022-04-22T18:00:30.852 INFO:tasks.workunit.client.0.smithi115.stdout:op 4097 completed, throughput=4.99MB/sec 2022-04-22T18:00:30.852 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pBHfNkv5vwnOOLv off=13568107 len=674409 2022-04-22T18:00:30.854 INFO:tasks.workunit.client.0.smithi115.stdout:op 4098 completed, throughput=5MB/sec 2022-04-22T18:00:30.854 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6KCbruqpKeffBvT off=904815 len=1885597 2022-04-22T18:00:30.857 INFO:tasks.workunit.client.0.smithi115.stdout:op 4099 completed, throughput=5MB/sec 2022-04-22T18:00:30.858 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PTPF8Aj7-7pAHQe off=8597285 len=1692816 2022-04-22T18:00:30.864 INFO:tasks.workunit.client.0.smithi115.stdout:op 4100 completed, throughput=5MB/sec 2022-04-22T18:00:30.864 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tIy_mnGkiEsdU1E off=4199876 len=1516628 2022-04-22T18:00:30.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 4101 completed, throughput=5MB/sec 2022-04-22T18:00:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:30 smithi121 bash[36902]: audit 2022-04-22T18:00:30.273254+0000 mon.a (mon.0) 2735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:30 smithi121 bash[36902]: audit 2022-04-22T18:00:30.649397+0000 mon.a (mon.0) 2736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:30 smithi115 bash[45495]: audit 2022-04-22T18:00:30.273254+0000 mon.a (mon.0) 2735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:30 smithi115 bash[45495]: audit 2022-04-22T18:00:30.649397+0000 mon.a (mon.0) 2736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:30 smithi115 bash[46934]: audit 2022-04-22T18:00:30.273254+0000 mon.a (mon.0) 2735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:30 smithi115 bash[46934]: audit 2022-04-22T18:00:30.649397+0000 mon.a (mon.0) 2736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:31.866 INFO:tasks.workunit.client.0.smithi115.stdout: 808: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:31.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iP9IllxiyEYlJAb off=601469 len=556681 2022-04-22T18:00:31.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 4102 completed, throughput=5MB/sec 2022-04-22T18:00:31.868 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_PgKOrwj4-2PBdY off=10656956 len=1487619 2022-04-22T18:00:31.873 INFO:tasks.workunit.client.0.smithi115.stdout:op 4103 completed, throughput=5MB/sec 2022-04-22T18:00:31.873 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-07irIg9N9tfC_no off=12396210 len=25355 2022-04-22T18:00:31.877 INFO:tasks.workunit.client.0.smithi115.stdout:op 4104 completed, throughput=5MB/sec 2022-04-22T18:00:31.877 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5UVEltpyRtCvW0w off=10576816 len=233669 2022-04-22T18:00:31.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 4105 completed, throughput=5MB/sec 2022-04-22T18:00:31.882 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xl0Cy18ZvzrHyYk off=14533507 len=2050230 2022-04-22T18:00:31.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 4106 completed, throughput=5MB/sec 2022-04-22T18:00:31.892 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sn6varEbT310TX0 off=0 len=1256499 2022-04-22T18:00:31.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 4107 completed, throughput=5MB/sec 2022-04-22T18:00:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:31 smithi115 bash[45495]: cluster 2022-04-22T18:00:30.691091+0000 mgr.y (mgr.34107) 1226 : cluster [DBG] pgmap v1814: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:31.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:31 smithi115 bash[46934]: cluster 2022-04-22T18:00:30.691091+0000 mgr.y (mgr.34107) 1226 : cluster [DBG] pgmap v1814: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:31 smithi121 bash[36902]: cluster 2022-04-22T18:00:30.691091+0000 mgr.y (mgr.34107) 1226 : cluster [DBG] pgmap v1814: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:32.899 INFO:tasks.workunit.client.0.smithi115.stdout: 809: throughput=5MB/sec pending data=0 2022-04-22T18:00:32.899 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MWzrsarfcpYOIN7 off=3359213 len=1759070 2022-04-22T18:00:32.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 4108 completed, throughput=5MB/sec 2022-04-22T18:00:32.903 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xVXJ2e4zT1QlIcD off=1947782 len=2091332 2022-04-22T18:00:32.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 4109 completed, throughput=5MB/sec 2022-04-22T18:00:32.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BcitRqMYZ378LzW off=1344029 len=972811 2022-04-22T18:00:32.918 INFO:tasks.workunit.client.0.smithi115.stdout:op 4110 completed, throughput=5MB/sec 2022-04-22T18:00:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:32 smithi121 bash[36902]: audit 2022-04-22T18:00:32.660296+0000 mon.a (mon.0) 2737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:32 smithi115 bash[45495]: audit 2022-04-22T18:00:32.660296+0000 mon.a (mon.0) 2737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:32 smithi115 bash[46934]: audit 2022-04-22T18:00:32.660296+0000 mon.a (mon.0) 2737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:33.919 INFO:tasks.workunit.client.0.smithi115.stdout: 810: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:33.919 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AxzvM5QH6_D1wKU off=2405874 len=1510671 2022-04-22T18:00:33.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 4111 completed, throughput=5MB/sec 2022-04-22T18:00:33.922 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q__V_r63hWAJDnw off=10229290 len=1061855 2022-04-22T18:00:33.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 4112 completed, throughput=5MB/sec 2022-04-22T18:00:33.928 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w6gwuYmdF_n_KKK off=12853310 len=1453272 2022-04-22T18:00:33.931 INFO:tasks.workunit.client.0.smithi115.stdout:op 4113 completed, throughput=5MB/sec 2022-04-22T18:00:33.931 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-36mC9_qVIIP4GId off=592473 len=1968037 2022-04-22T18:00:33.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 4114 completed, throughput=5MB/sec 2022-04-22T18:00:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:33 smithi115 bash[45495]: cluster 2022-04-22T18:00:32.692086+0000 mgr.y (mgr.34107) 1227 : cluster [DBG] pgmap v1815: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:00:34.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:33 smithi115 bash[46934]: cluster 2022-04-22T18:00:32.692086+0000 mgr.y (mgr.34107) 1227 : cluster [DBG] pgmap v1815: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:00:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:33.606Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:33 smithi121 bash[36902]: cluster 2022-04-22T18:00:32.692086+0000 mgr.y (mgr.34107) 1227 : cluster [DBG] pgmap v1815: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:00:34.940 INFO:tasks.workunit.client.0.smithi115.stdout: 811: throughput=5MB/sec pending data=0 2022-04-22T18:00:34.940 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Uyedr7_DRfr1l7S off=1962441 len=1245628 2022-04-22T18:00:34.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 4115 completed, throughput=5MB/sec 2022-04-22T18:00:34.948 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CnHpXNrlyEV43Pe off=9381552 len=766621 2022-04-22T18:00:34.950 INFO:tasks.workunit.client.0.smithi115.stdout:op 4116 completed, throughput=5MB/sec 2022-04-22T18:00:34.950 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NrL_4ytSC3izy1v off=5754111 len=1839979 2022-04-22T18:00:34.956 INFO:tasks.workunit.client.0.smithi115.stdout:op 4117 completed, throughput=5MB/sec 2022-04-22T18:00:34.956 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h8RjFRMvHdfJjbq off=8073305 len=1660036 2022-04-22T18:00:34.964 INFO:tasks.workunit.client.0.smithi115.stdout:op 4118 completed, throughput=5MB/sec 2022-04-22T18:00:35.964 INFO:tasks.workunit.client.0.smithi115.stdout: 812: throughput=5MB/sec pending data=0 2022-04-22T18:00:35.964 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gi51R55HLBJf_iA off=6413532 len=41019 2022-04-22T18:00:35.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 4119 completed, throughput=5MB/sec 2022-04-22T18:00:35.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qFoEDNpPJ0rA3Eo off=5140632 len=904831 2022-04-22T18:00:35.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 4120 completed, throughput=5MB/sec 2022-04-22T18:00:35.969 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ap_hMrt1Hb21oeb off=1452223 len=605337 2022-04-22T18:00:35.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 4121 completed, throughput=5MB/sec 2022-04-22T18:00:35.971 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GoJQWO9cERgKfQs off=166425 len=385094 2022-04-22T18:00:35.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 4122 completed, throughput=5MB/sec 2022-04-22T18:00:35.972 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M5HBhsTGGbgJZj0 off=3948028 len=1345158 2022-04-22T18:00:35.980 INFO:tasks.workunit.client.0.smithi115.stdout:op 4123 completed, throughput=5MB/sec 2022-04-22T18:00:35.980 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RYQO_tiSZ9bopNs off=3296701 len=2021328 2022-04-22T18:00:35.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 4124 completed, throughput=5MB/sec 2022-04-22T18:00:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:35 smithi121 bash[36902]: cluster 2022-04-22T18:00:34.693171+0000 mgr.y (mgr.34107) 1228 : cluster [DBG] pgmap v1816: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:35 smithi121 bash[36902]: audit 2022-04-22T18:00:35.276421+0000 mon.a (mon.0) 2738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:36.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:35 smithi121 bash[36902]: audit 2022-04-22T18:00:35.653132+0000 mon.a (mon.0) 2739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[45495]: cluster 2022-04-22T18:00:34.693171+0000 mgr.y (mgr.34107) 1228 : cluster [DBG] pgmap v1816: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[45495]: audit 2022-04-22T18:00:35.276421+0000 mon.a (mon.0) 2738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[45495]: audit 2022-04-22T18:00:35.653132+0000 mon.a (mon.0) 2739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[46934]: cluster 2022-04-22T18:00:34.693171+0000 mgr.y (mgr.34107) 1228 : cluster [DBG] pgmap v1816: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:00:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[46934]: audit 2022-04-22T18:00:35.276421+0000 mon.a (mon.0) 2738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:35 smithi115 bash[46934]: audit 2022-04-22T18:00:35.653132+0000 mon.a (mon.0) 2739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:36.990 INFO:tasks.workunit.client.0.smithi115.stdout: 813: throughput=5MB/sec pending data=0 2022-04-22T18:00:36.990 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F5eVB_juCpnvjhW off=11396516 len=1328120 2022-04-22T18:00:36.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 4125 completed, throughput=5MB/sec 2022-04-22T18:00:36.994 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lNULtJe0dI2NG72 off=6132750 len=2061337 2022-04-22T18:00:37.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 4126 completed, throughput=5MB/sec 2022-04-22T18:00:37.006 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v9tpQvQyQ3g3nvu off=854568 len=1974030 2022-04-22T18:00:37.009 INFO:tasks.workunit.client.0.smithi115.stdout:op 4127 completed, throughput=5MB/sec 2022-04-22T18:00:38.009 INFO:tasks.workunit.client.0.smithi115.stdout: 814: throughput=5MB/sec pending data=0 2022-04-22T18:00:38.010 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L3WmZ2NNjxCbThK off=3173998 len=1295144 2022-04-22T18:00:38.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 4128 completed, throughput=5MB/sec 2022-04-22T18:00:38.019 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UNKcUc2a3P20NiW off=15468466 len=1154468 2022-04-22T18:00:38.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 4129 completed, throughput=5MB/sec 2022-04-22T18:00:38.026 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nzjnfuVm46S_e21 off=0 len=1381369 2022-04-22T18:00:38.029 INFO:tasks.workunit.client.0.smithi115.stdout:op 4130 completed, throughput=5MB/sec 2022-04-22T18:00:38.029 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wO6w1AIcfuiVU0q off=10653411 len=857498 2022-04-22T18:00:38.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 4131 completed, throughput=5MB/sec 2022-04-22T18:00:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:37 smithi121 bash[36902]: cluster 2022-04-22T18:00:36.694272+0000 mgr.y (mgr.34107) 1229 : cluster [DBG] pgmap v1817: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:00:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:37 smithi121 bash[36902]: audit 2022-04-22T18:00:37.663307+0000 mon.a (mon.0) 2740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:37 smithi115 bash[45495]: cluster 2022-04-22T18:00:36.694272+0000 mgr.y (mgr.34107) 1229 : cluster [DBG] pgmap v1817: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:00:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:37 smithi115 bash[45495]: audit 2022-04-22T18:00:37.663307+0000 mon.a (mon.0) 2740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:37 smithi115 bash[46934]: cluster 2022-04-22T18:00:36.694272+0000 mgr.y (mgr.34107) 1229 : cluster [DBG] pgmap v1817: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:00:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:37 smithi115 bash[46934]: audit 2022-04-22T18:00:37.663307+0000 mon.a (mon.0) 2740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:39.032 INFO:tasks.workunit.client.0.smithi115.stdout: 815: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:39.032 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-95UZgat2DEJIE0B off=6379437 len=1874411 2022-04-22T18:00:39.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 4132 completed, throughput=5MB/sec 2022-04-22T18:00:39.041 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ALgQIynmkgMmAOQ off=4559020 len=490093 2022-04-22T18:00:39.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 4133 completed, throughput=5MB/sec 2022-04-22T18:00:39.045 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-paPSYF5Wb214fwr off=12545009 len=903733 2022-04-22T18:00:39.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 4134 completed, throughput=5MB/sec 2022-04-22T18:00:39.047 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4TWFAt1uEpAcqhD off=393728 len=812367 2022-04-22T18:00:39.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 4135 completed, throughput=5MB/sec 2022-04-22T18:00:39.053 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gq7llfAnN930VTV off=1508248 len=1696152 2022-04-22T18:00:39.055 INFO:tasks.workunit.client.0.smithi115.stdout:op 4136 completed, throughput=5MB/sec 2022-04-22T18:00:40.056 INFO:tasks.workunit.client.0.smithi115.stdout: 816: throughput=5MB/sec pending data=0 2022-04-22T18:00:40.056 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--kQNgnFR1GkV5TU off=4110273 len=324227 2022-04-22T18:00:40.057 INFO:tasks.workunit.client.0.smithi115.stdout:op 4137 completed, throughput=5MB/sec 2022-04-22T18:00:40.057 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2MHsdtLg-ASCJMJ off=183460 len=1657152 2022-04-22T18:00:40.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 4138 completed, throughput=5MB/sec 2022-04-22T18:00:40.062 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vydIUr0an9CRtpP off=21153274 len=197167 2022-04-22T18:00:40.065 INFO:tasks.workunit.client.0.smithi115.stdout:op 4139 completed, throughput=5MB/sec 2022-04-22T18:00:40.065 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ypm815WwgxNnlEn off=3225398 len=539436 2022-04-22T18:00:40.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 4140 completed, throughput=5MB/sec 2022-04-22T18:00:40.069 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UukRAR4IvuC0Ura off=4162265 len=428073 2022-04-22T18:00:40.073 INFO:tasks.workunit.client.0.smithi115.stdout:op 4141 completed, throughput=5MB/sec 2022-04-22T18:00:40.073 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SAk_P87Ex1qur2e off=7892288 len=883381 2022-04-22T18:00:40.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 4142 completed, throughput=5MB/sec 2022-04-22T18:00:40.075 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pfj_gkM5QDeFhMD off=2255480 len=875750 2022-04-22T18:00:40.078 INFO:tasks.workunit.client.0.smithi115.stdout:op 4143 completed, throughput=5MB/sec 2022-04-22T18:00:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:39 smithi121 bash[36902]: cluster 2022-04-22T18:00:38.695496+0000 mgr.y (mgr.34107) 1230 : cluster [DBG] pgmap v1818: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:40.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:39 smithi115 bash[45495]: cluster 2022-04-22T18:00:38.695496+0000 mgr.y (mgr.34107) 1230 : cluster [DBG] pgmap v1818: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:39 smithi115 bash[46934]: cluster 2022-04-22T18:00:38.695496+0000 mgr.y (mgr.34107) 1230 : cluster [DBG] pgmap v1818: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:00:41.079 INFO:tasks.workunit.client.0.smithi115.stdout: 817: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:41.118 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZlwpRA6IWVaXTk1 off=1551589 len=1298996 2022-04-22T18:00:41.120 INFO:tasks.workunit.client.0.smithi115.stdout:op 4144 completed, throughput=5MB/sec 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8WKCZeAohfApkCa off=1187324 len=937243 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 4145 completed, throughput=5MB/sec 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gt2qmwofFHV_eq5 off=2576374 len=188224 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 4146 completed, throughput=5MB/sec 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6pWT4liqA4F5j5T off=14563832 len=717085 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 4147 completed, throughput=5MB/sec 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8zlDDg8CbdHABeS off=387830 len=258568 2022-04-22T18:00:41.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 4148 completed, throughput=5MB/sec 2022-04-22T18:00:41.122 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kKlUqZeqvF2nDR6 off=5817709 len=1892082 2022-04-22T18:00:41.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 4149 completed, throughput=5MB/sec 2022-04-22T18:00:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:40 smithi121 bash[36902]: audit 2022-04-22T18:00:40.279442+0000 mon.a (mon.0) 2741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:40 smithi121 bash[36902]: audit 2022-04-22T18:00:40.656807+0000 mon.a (mon.0) 2742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:40 smithi115 bash[45495]: audit 2022-04-22T18:00:40.279442+0000 mon.a (mon.0) 2741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:40 smithi115 bash[45495]: audit 2022-04-22T18:00:40.656807+0000 mon.a (mon.0) 2742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:40 smithi115 bash[46934]: audit 2022-04-22T18:00:40.279442+0000 mon.a (mon.0) 2741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:40 smithi115 bash[46934]: audit 2022-04-22T18:00:40.656807+0000 mon.a (mon.0) 2742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:41 smithi115 bash[45495]: cluster 2022-04-22T18:00:40.696331+0000 mgr.y (mgr.34107) 1231 : cluster [DBG] pgmap v1819: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:41.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:41 smithi115 bash[46934]: cluster 2022-04-22T18:00:40.696331+0000 mgr.y (mgr.34107) 1231 : cluster [DBG] pgmap v1819: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:42.102 INFO:tasks.workunit.client.0.smithi115.stdout: 818: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:42.102 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oZCzI_-NRIkpWSc off=1156111 len=1644897 2022-04-22T18:00:42.112 INFO:tasks.workunit.client.0.smithi115.stdout:op 4150 completed, throughput=5MB/sec 2022-04-22T18:00:42.112 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5VPAVqLKoaJdS0d off=762316 len=1455462 2022-04-22T18:00:42.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 4151 completed, throughput=5MB/sec 2022-04-22T18:00:42.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gjleHvpy-n715SP off=15719098 len=934853 2022-04-22T18:00:42.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 4152 completed, throughput=5MB/sec 2022-04-22T18:00:42.122 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZqKwBNDjpc4ZTgD off=5672861 len=1069938 2022-04-22T18:00:42.131 INFO:tasks.workunit.client.0.smithi115.stdout:op 4153 completed, throughput=5MB/sec 2022-04-22T18:00:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:41 smithi121 bash[36902]: cluster 2022-04-22T18:00:40.696331+0000 mgr.y (mgr.34107) 1231 : cluster [DBG] pgmap v1819: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:43.131 INFO:tasks.workunit.client.0.smithi115.stdout: 819: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:43.132 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nWdf39EH9iYe9Lf off=2398992 len=1623362 2022-04-22T18:00:43.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 4154 completed, throughput=5MB/sec 2022-04-22T18:00:43.135 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mfqGbuGDtb58juB off=2696529 len=1799202 2022-04-22T18:00:43.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 4155 completed, throughput=5MB/sec 2022-04-22T18:00:43.140 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v6ee5PRoruxYZRs off=3152682 len=372506 2022-04-22T18:00:43.142 INFO:tasks.workunit.client.0.smithi115.stdout:op 4156 completed, throughput=5MB/sec 2022-04-22T18:00:43.142 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L1RKbsX0982foSd off=1838267 len=342572 2022-04-22T18:00:43.147 INFO:tasks.workunit.client.0.smithi115.stdout:op 4157 completed, throughput=5MB/sec 2022-04-22T18:00:43.147 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_5nMD2vm83953cp off=413782 len=510871 2022-04-22T18:00:43.152 INFO:tasks.workunit.client.0.smithi115.stdout:op 4158 completed, throughput=5MB/sec 2022-04-22T18:00:43.152 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pp2UZTI9eGapz6E off=5151309 len=30665 2022-04-22T18:00:43.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 4159 completed, throughput=5MB/sec 2022-04-22T18:00:43.153 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qMg_mw5M7GMU1ku off=2958518 len=1505894 2022-04-22T18:00:43.158 INFO:tasks.workunit.client.0.smithi115.stdout:op 4160 completed, throughput=5MB/sec 2022-04-22T18:00:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:42 smithi121 bash[36902]: audit 2022-04-22T18:00:42.666455+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:43.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:42 smithi115 bash[45495]: audit 2022-04-22T18:00:42.666455+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:43.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:42 smithi115 bash[46934]: audit 2022-04-22T18:00:42.666455+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:43.888 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:43.609Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:44.158 INFO:tasks.workunit.client.0.smithi115.stdout: 820: throughput=5MB/sec pending data=0 2022-04-22T18:00:44.158 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Pb2C_pCmtB7u1rq off=5650696 len=553460 2022-04-22T18:00:44.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 4161 completed, throughput=5MB/sec 2022-04-22T18:00:44.165 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eu0wRl6sdKmdckd off=1055016 len=2007717 2022-04-22T18:00:44.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 4162 completed, throughput=5MB/sec 2022-04-22T18:00:44.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F6J154hZJ-KPq7C off=10233352 len=453401 2022-04-22T18:00:44.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 4163 completed, throughput=5MB/sec 2022-04-22T18:00:44.178 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O7Tc3C9SZjz2-Zy off=4310308 len=510159 2022-04-22T18:00:44.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 4164 completed, throughput=5MB/sec 2022-04-22T18:00:44.183 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KT9r5eGRJoasKD7 off=2885527 len=6909 2022-04-22T18:00:44.183 INFO:tasks.workunit.client.0.smithi115.stdout:op 4165 completed, throughput=5MB/sec 2022-04-22T18:00:44.183 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qBCxLIhfddXtYNd off=23812026 len=1526185 2022-04-22T18:00:44.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 4166 completed, throughput=5MB/sec 2022-04-22T18:00:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:43 smithi121 bash[36902]: cluster 2022-04-22T18:00:42.697401+0000 mgr.y (mgr.34107) 1232 : cluster [DBG] pgmap v1820: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:44.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:43 smithi115 bash[45495]: cluster 2022-04-22T18:00:42.697401+0000 mgr.y (mgr.34107) 1232 : cluster [DBG] pgmap v1820: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:44.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:43 smithi115 bash[46934]: cluster 2022-04-22T18:00:42.697401+0000 mgr.y (mgr.34107) 1232 : cluster [DBG] pgmap v1820: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:45.191 INFO:tasks.workunit.client.0.smithi115.stdout: 821: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:45.191 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lGT-AunrnyHmhvb off=1337141 len=233578 2022-04-22T18:00:45.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 4167 completed, throughput=5MB/sec 2022-04-22T18:00:45.193 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nwuut-kVJhHrjnt off=511438 len=528590 2022-04-22T18:00:45.194 INFO:tasks.workunit.client.0.smithi115.stdout:op 4168 completed, throughput=5MB/sec 2022-04-22T18:00:45.194 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5I6x5jIvARzAfUL off=10506409 len=36170 2022-04-22T18:00:45.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 4169 completed, throughput=5MB/sec 2022-04-22T18:00:45.195 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FMQNSgGeqSmKpEI off=3978691 len=630614 2022-04-22T18:00:45.201 INFO:tasks.workunit.client.0.smithi115.stdout:op 4170 completed, throughput=5MB/sec 2022-04-22T18:00:45.201 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tgro8mH_3GBGKga off=1610868 len=1325057 2022-04-22T18:00:45.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 4171 completed, throughput=5MB/sec 2022-04-22T18:00:45.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XfP7W4MsCd4DvT5 off=2170709 len=688724 2022-04-22T18:00:45.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 4172 completed, throughput=5MB/sec 2022-04-22T18:00:45.215 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bIMUDQFkak9sXH4 off=12938018 len=100667 2022-04-22T18:00:45.218 INFO:tasks.workunit.client.0.smithi115.stdout:op 4173 completed, throughput=5MB/sec 2022-04-22T18:00:45.218 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kzj3nWdEsf02VWz off=4517231 len=1212661 2022-04-22T18:00:45.223 INFO:tasks.workunit.client.0.smithi115.stdout:op 4174 completed, throughput=5MB/sec 2022-04-22T18:00:45.223 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-htZLrvEAtvuoca7 off=10295706 len=551479 2022-04-22T18:00:45.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 4175 completed, throughput=5MB/sec 2022-04-22T18:00:46.225 INFO:tasks.workunit.client.0.smithi115.stdout: 822: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:46.226 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eFt6KiRhFCwyLEw off=1044656 len=775035 2022-04-22T18:00:46.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 4176 completed, throughput=5MB/sec 2022-04-22T18:00:46.231 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6SwnAQC-yEfOgea off=13024241 len=1579911 2022-04-22T18:00:46.234 INFO:tasks.workunit.client.0.smithi115.stdout:op 4177 completed, throughput=5MB/sec 2022-04-22T18:00:46.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IdlURgJsPdt9OF7 off=1456702 len=1325305 2022-04-22T18:00:46.237 INFO:tasks.workunit.client.0.smithi115.stdout:op 4178 completed, throughput=5MB/sec 2022-04-22T18:00:46.237 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WyylfWmfhQTrJ0p off=6464113 len=2006374 2022-04-22T18:00:46.243 INFO:tasks.workunit.client.0.smithi115.stdout:op 4179 completed, throughput=5MB/sec 2022-04-22T18:00:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:45 smithi121 bash[36902]: cluster 2022-04-22T18:00:44.698537+0000 mgr.y (mgr.34107) 1233 : cluster [DBG] pgmap v1821: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:00:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:45 smithi121 bash[36902]: audit 2022-04-22T18:00:45.282724+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:45 smithi121 bash[36902]: audit 2022-04-22T18:00:45.660342+0000 mon.a (mon.0) 2745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[45495]: cluster 2022-04-22T18:00:44.698537+0000 mgr.y (mgr.34107) 1233 : cluster [DBG] pgmap v1821: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:00:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[45495]: audit 2022-04-22T18:00:45.282724+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[45495]: audit 2022-04-22T18:00:45.660342+0000 mon.a (mon.0) 2745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[46934]: cluster 2022-04-22T18:00:44.698537+0000 mgr.y (mgr.34107) 1233 : cluster [DBG] pgmap v1821: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:00:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[46934]: audit 2022-04-22T18:00:45.282724+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:45 smithi115 bash[46934]: audit 2022-04-22T18:00:45.660342+0000 mon.a (mon.0) 2745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:47.244 INFO:tasks.workunit.client.0.smithi115.stdout: 823: throughput=5MB/sec pending data=0 2022-04-22T18:00:47.244 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7f9m4tnXVXTuyrE off=8923746 len=1410728 2022-04-22T18:00:47.253 INFO:tasks.workunit.client.0.smithi115.stdout:op 4180 completed, throughput=5MB/sec 2022-04-22T18:00:47.254 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZrUXQulBOGAw3jn off=17028539 len=1796458 2022-04-22T18:00:47.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 4181 completed, throughput=5MB/sec 2022-04-22T18:00:47.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BmU0qBJip3ZJ_pT off=5735473 len=849962 2022-04-22T18:00:47.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 4182 completed, throughput=5MB/sec 2022-04-22T18:00:47.265 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wwS98vp5gyiQg-0 off=699005 len=1731693 2022-04-22T18:00:47.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 4183 completed, throughput=5MB/sec 2022-04-22T18:00:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:47 smithi121 bash[36902]: cluster 2022-04-22T18:00:46.699743+0000 mgr.y (mgr.34107) 1234 : cluster [DBG] pgmap v1822: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:00:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:47 smithi121 bash[36902]: audit 2022-04-22T18:00:47.669605+0000 mon.a (mon.0) 2746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:48.272 INFO:tasks.workunit.client.0.smithi115.stdout: 824: throughput=5MB/sec pending data=0 2022-04-22T18:00:48.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sXs-I9udNbbmVif off=3202021 len=447997 2022-04-22T18:00:48.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 4184 completed, throughput=5MB/sec 2022-04-22T18:00:48.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YHcbDRaxJ-8br4J off=8825750 len=603319 2022-04-22T18:00:48.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 4185 completed, throughput=5MB/sec 2022-04-22T18:00:48.280 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_wRvAVUnVJeTnQA off=8873001 len=2087504 2022-04-22T18:00:48.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 4186 completed, throughput=5MB/sec 2022-04-22T18:00:48.291 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-l783K4Vh_T4Fm8l off=8655515 len=549155 2022-04-22T18:00:48.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 4187 completed, throughput=5MB/sec 2022-04-22T18:00:48.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h_KAdm4AYsC4LOA off=2932104 len=1519992 2022-04-22T18:00:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:47 smithi115 bash[45495]: cluster 2022-04-22T18:00:46.699743+0000 mgr.y (mgr.34107) 1234 : cluster [DBG] pgmap v1822: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:00:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:47 smithi115 bash[45495]: audit 2022-04-22T18:00:47.669605+0000 mon.a (mon.0) 2746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:47 smithi115 bash[46934]: cluster 2022-04-22T18:00:46.699743+0000 mgr.y (mgr.34107) 1234 : cluster [DBG] pgmap v1822: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:00:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:47 smithi115 bash[46934]: audit 2022-04-22T18:00:47.669605+0000 mon.a (mon.0) 2746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:48.301 INFO:tasks.workunit.client.0.smithi115.stdout:op 4188 completed, throughput=5MB/sec 2022-04-22T18:00:49.302 INFO:tasks.workunit.client.0.smithi115.stdout: 825: throughput=5MB/sec pending data=0 2022-04-22T18:00:49.302 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6BIWDqSwYP2rDnR off=7159123 len=331625 2022-04-22T18:00:49.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 4189 completed, throughput=5MB/sec 2022-04-22T18:00:49.307 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v7FT1VmYl5b66nk off=4533619 len=1723364 2022-04-22T18:00:49.317 INFO:tasks.workunit.client.0.smithi115.stdout:op 4190 completed, throughput=5MB/sec 2022-04-22T18:00:49.317 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X-9LN6Zz5xnpV6y off=1286740 len=1539682 2022-04-22T18:00:49.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 4191 completed, throughput=5MB/sec 2022-04-22T18:00:49.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0zLnWEZUYrILlcK off=9311642 len=1430533 2022-04-22T18:00:49.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 4192 completed, throughput=5MB/sec 2022-04-22T18:00:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:49 smithi121 bash[36902]: cluster 2022-04-22T18:00:48.700911+0000 mgr.y (mgr.34107) 1235 : cluster [DBG] pgmap v1823: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:49 smithi115 bash[45495]: cluster 2022-04-22T18:00:48.700911+0000 mgr.y (mgr.34107) 1235 : cluster [DBG] pgmap v1823: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:49 smithi115 bash[46934]: cluster 2022-04-22T18:00:48.700911+0000 mgr.y (mgr.34107) 1235 : cluster [DBG] pgmap v1823: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:00:50.327 INFO:tasks.workunit.client.0.smithi115.stdout: 826: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:50.327 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NDvX1g8kVLethTB off=2349762 len=928321 2022-04-22T18:00:50.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 4193 completed, throughput=5MB/sec 2022-04-22T18:00:50.330 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kQhSp9eEA_WAd5O off=8608098 len=1061943 2022-04-22T18:00:50.334 INFO:tasks.workunit.client.0.smithi115.stdout:op 4194 completed, throughput=5MB/sec 2022-04-22T18:00:50.334 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uixzRd5LFVMglXe off=18448299 len=1234947 2022-04-22T18:00:50.342 INFO:tasks.workunit.client.0.smithi115.stdout:op 4195 completed, throughput=5MB/sec 2022-04-22T18:00:50.342 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eBdaO6zkuzdxQba off=7159102 len=799657 2022-04-22T18:00:50.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 4196 completed, throughput=5MB/sec 2022-04-22T18:00:50.348 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ApaQmd6gCMJb1rS off=8150625 len=365046 2022-04-22T18:00:50.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 4197 completed, throughput=5MB/sec 2022-04-22T18:00:50.352 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aLf06EELRq2tsgR off=7460073 len=283083 2022-04-22T18:00:50.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 4198 completed, throughput=5MB/sec 2022-04-22T18:00:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:50 smithi121 bash[36902]: audit 2022-04-22T18:00:50.285813+0000 mon.a (mon.0) 2747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:50 smithi121 bash[36902]: audit 2022-04-22T18:00:50.602565+0000 mon.a (mon.0) 2748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:00:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:50 smithi121 bash[36902]: audit 2022-04-22T18:00:50.605750+0000 mon.a (mon.0) 2749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:00:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:50 smithi121 bash[36902]: audit 2022-04-22T18:00:50.607745+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:00:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:50 smithi121 bash[36902]: audit 2022-04-22T18:00:50.663836+0000 mon.a (mon.0) 2751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[45495]: audit 2022-04-22T18:00:50.285813+0000 mon.a (mon.0) 2747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[45495]: audit 2022-04-22T18:00:50.602565+0000 mon.a (mon.0) 2748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:00:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[45495]: audit 2022-04-22T18:00:50.605750+0000 mon.a (mon.0) 2749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:00:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[45495]: audit 2022-04-22T18:00:50.607745+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[45495]: audit 2022-04-22T18:00:50.663836+0000 mon.a (mon.0) 2751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[46934]: audit 2022-04-22T18:00:50.285813+0000 mon.a (mon.0) 2747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[46934]: audit 2022-04-22T18:00:50.602565+0000 mon.a (mon.0) 2748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[46934]: audit 2022-04-22T18:00:50.605750+0000 mon.a (mon.0) 2749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[46934]: audit 2022-04-22T18:00:50.607745+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:00:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:50 smithi115 bash[46934]: audit 2022-04-22T18:00:50.663836+0000 mon.a (mon.0) 2751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:51.354 INFO:tasks.workunit.client.0.smithi115.stdout: 827: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:51.354 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CMwb8TZlUC2FVQo off=4198583 len=783297 2022-04-22T18:00:51.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 4199 completed, throughput=4.99MB/sec 2022-04-22T18:00:51.361 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GDbMmytA2kWS_Fg off=1141112 len=1983763 2022-04-22T18:00:51.373 INFO:tasks.workunit.client.0.smithi115.stdout:op 4200 completed, throughput=5MB/sec 2022-04-22T18:00:51.373 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fBwk0lsUxn3ieew off=609053 len=740714 2022-04-22T18:00:51.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 4201 completed, throughput=5MB/sec 2022-04-22T18:00:51.379 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5ZDnUp06iIQ0Loi off=1458688 len=433513 2022-04-22T18:00:51.380 INFO:tasks.workunit.client.0.smithi115.stdout:op 4202 completed, throughput=5MB/sec 2022-04-22T18:00:51.380 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JPj0xURKgzyiYo2 off=18931003 len=1859073 2022-04-22T18:00:51.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 4203 completed, throughput=5MB/sec 2022-04-22T18:00:51.985 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:51.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:52.191 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:00:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:51 smithi121 bash[36902]: cluster 2022-04-22T18:00:50.701855+0000 mgr.y (mgr.34107) 1236 : cluster [DBG] pgmap v1824: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:51 smithi121 bash[36902]: audit 2022-04-22T18:00:50.980374+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:00:52.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:51 smithi115 bash[45495]: cluster 2022-04-22T18:00:50.701855+0000 mgr.y (mgr.34107) 1236 : cluster [DBG] pgmap v1824: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:52.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:51 smithi115 bash[45495]: audit 2022-04-22T18:00:50.980374+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:00:52.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:51 smithi115 bash[46934]: cluster 2022-04-22T18:00:50.701855+0000 mgr.y (mgr.34107) 1236 : cluster [DBG] pgmap v1824: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:00:52.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:51 smithi115 bash[46934]: audit 2022-04-22T18:00:50.980374+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:00:52.386 INFO:tasks.workunit.client.0.smithi115.stdout: 828: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:52.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WksA1YkUzET3nCf off=253858 len=247841 2022-04-22T18:00:52.387 INFO:tasks.workunit.client.0.smithi115.stdout:op 4204 completed, throughput=4.99MB/sec 2022-04-22T18:00:52.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_8A4emTRFbuu6j3 off=202281 len=426778 2022-04-22T18:00:52.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 4205 completed, throughput=5MB/sec 2022-04-22T18:00:52.388 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kYfUUWd5w9KHHMG off=7994261 len=332518 2022-04-22T18:00:52.393 INFO:tasks.workunit.client.0.smithi115.stdout:op 4206 completed, throughput=5MB/sec 2022-04-22T18:00:52.393 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xU2-L6XvYbV4WBh off=3142713 len=1948736 2022-04-22T18:00:52.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 4207 completed, throughput=5MB/sec 2022-04-22T18:00:52.397 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jCQwA6dGWvbiGzK off=996476 len=570660 2022-04-22T18:00:52.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 4208 completed, throughput=5MB/sec 2022-04-22T18:00:52.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IYcIPqPtyUbr6xb off=13689455 len=883032 2022-04-22T18:00:52.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 4209 completed, throughput=5MB/sec 2022-04-22T18:00:52.403 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CKr29TAmcLFbCBu off=16283343 len=1316064 2022-04-22T18:00:52.404 INFO:tasks.workunit.client.0.smithi115.stdout:op 4210 completed, throughput=5MB/sec 2022-04-22T18:00:52.535 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:00:52.536 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (23m) 3m ago 37m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:00:52.536 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (37m) 4m ago 37m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:00:52.536 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (25m) 3m ago 38m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:00:52.536 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (25m) 4m ago 38m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:00:52.536 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (33m) 4m ago 42m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:00:52.537 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (33m) 3m ago 45m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:00:52.537 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (32m) 3m ago 45m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:00:52.537 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (31m) 4m ago 43m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:00:52.537 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (32m) 3m ago 43m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:00:52.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (25m) 3m ago 37m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:00:52.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (24m) 4m ago 37m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:00:52.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (30m) 3m ago 42m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:00:52.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (29m) 3m ago 42m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:00:52.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (29m) 3m ago 41m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (29m) 3m ago 41m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (28m) 4m ago 40m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (27m) 4m ago 40m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (27m) 4m ago 39m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (26m) 4m ago 39m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:00:52.539 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (24m) 4m ago 38m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:00:52.941 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:00:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:00:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:00:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:00:52.943 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:00:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:52 smithi121 bash[36902]: audit 2022-04-22T18:00:52.166418+0000 mgr.y (mgr.34107) 1237 : audit [DBG] from='client.45637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:52 smithi121 bash[36902]: audit 2022-04-22T18:00:52.672297+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:52 smithi121 bash[36902]: audit 2022-04-22T18:00:52.934324+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.115:0/70983892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[45495]: audit 2022-04-22T18:00:52.166418+0000 mgr.y (mgr.34107) 1237 : audit [DBG] from='client.45637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[45495]: audit 2022-04-22T18:00:52.672297+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[45495]: audit 2022-04-22T18:00:52.934324+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.115:0/70983892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[46934]: audit 2022-04-22T18:00:52.166418+0000 mgr.y (mgr.34107) 1237 : audit [DBG] from='client.45637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[46934]: audit 2022-04-22T18:00:52.672297+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:52 smithi115 bash[46934]: audit 2022-04-22T18:00:52.934324+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.115:0/70983892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:00:53.405 INFO:tasks.workunit.client.0.smithi115.stdout: 829: throughput=5MB/sec pending data=0 2022-04-22T18:00:53.405 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jSR4CQ5MYn9B7aW off=6459632 len=759919 2022-04-22T18:00:53.407 INFO:tasks.workunit.client.0.smithi115.stdout:op 4211 completed, throughput=5MB/sec 2022-04-22T18:00:53.407 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z7pYUdoZZlhij6g off=16483850 len=1907014 2022-04-22T18:00:53.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 4212 completed, throughput=5MB/sec 2022-04-22T18:00:53.419 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LvlN5Z4va-_LTzJ off=9222829 len=1940305 2022-04-22T18:00:53.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 4213 completed, throughput=5MB/sec 2022-04-22T18:00:53.996 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:00:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:00:53.607Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:00:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:53 smithi121 bash[36902]: audit 2022-04-22T18:00:52.509311+0000 mgr.y (mgr.34107) 1238 : audit [DBG] from='client.35706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:53 smithi121 bash[36902]: cluster 2022-04-22T18:00:52.702859+0000 mgr.y (mgr.34107) 1239 : cluster [DBG] pgmap v1825: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:53 smithi115 bash[45495]: audit 2022-04-22T18:00:52.509311+0000 mgr.y (mgr.34107) 1238 : audit [DBG] from='client.35706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:53 smithi115 bash[45495]: cluster 2022-04-22T18:00:52.702859+0000 mgr.y (mgr.34107) 1239 : cluster [DBG] pgmap v1825: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:54.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:53 smithi115 bash[46934]: audit 2022-04-22T18:00:52.509311+0000 mgr.y (mgr.34107) 1238 : audit [DBG] from='client.35706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:00:54.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:53 smithi115 bash[46934]: cluster 2022-04-22T18:00:52.702859+0000 mgr.y (mgr.34107) 1239 : cluster [DBG] pgmap v1825: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:00:54.428 INFO:tasks.workunit.client.0.smithi115.stdout: 830: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:54.428 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-95_8WFvaf-kHI9E off=9007178 len=1607223 2022-04-22T18:00:54.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 4214 completed, throughput=5MB/sec 2022-04-22T18:00:54.432 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2MwMhwVjPJBoE9t off=13241273 len=527677 2022-04-22T18:00:54.438 INFO:tasks.workunit.client.0.smithi115.stdout:op 4215 completed, throughput=5MB/sec 2022-04-22T18:00:54.438 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_x_WfOCqUhZaKIJ off=462330 len=274790 2022-04-22T18:00:54.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 4216 completed, throughput=5MB/sec 2022-04-22T18:00:54.440 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wy-4BZ5ACqUDPTS off=6976099 len=1363062 2022-04-22T18:00:54.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 4217 completed, throughput=5MB/sec 2022-04-22T18:00:54.446 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nn9axeq_9NTjP11 off=1944652 len=1685805 2022-04-22T18:00:54.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 4218 completed, throughput=5MB/sec 2022-04-22T18:00:55.448 INFO:tasks.workunit.client.0.smithi115.stdout: 831: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:55.448 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K00KiZMHVVgSsiT off=17116257 len=991240 2022-04-22T18:00:55.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 4219 completed, throughput=5MB/sec 2022-04-22T18:00:55.456 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z68lJqZg7WFIw2V off=9991409 len=1672651 2022-04-22T18:00:55.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 4220 completed, throughput=5MB/sec 2022-04-22T18:00:55.458 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fa4XX55skY1cx50 off=8069386 len=1121952 2022-04-22T18:00:55.461 INFO:tasks.workunit.client.0.smithi115.stdout:op 4221 completed, throughput=5MB/sec 2022-04-22T18:00:55.461 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RxdnxVFYc3QT7U1 off=18867069 len=1051127 2022-04-22T18:00:55.465 INFO:tasks.workunit.client.0.smithi115.stdout:op 4222 completed, throughput=5MB/sec 2022-04-22T18:00:55.465 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--TaXn3Jihr3ZJ4S off=11203921 len=1003802 2022-04-22T18:00:55.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 4223 completed, throughput=5MB/sec 2022-04-22T18:00:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:56 smithi121 bash[36902]: cluster 2022-04-22T18:00:54.703966+0000 mgr.y (mgr.34107) 1240 : cluster [DBG] pgmap v1826: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:56 smithi121 bash[36902]: audit 2022-04-22T18:00:55.288806+0000 mon.a (mon.0) 2754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:56 smithi121 bash[36902]: audit 2022-04-22T18:00:55.667255+0000 mon.a (mon.0) 2755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[45495]: cluster 2022-04-22T18:00:54.703966+0000 mgr.y (mgr.34107) 1240 : cluster [DBG] pgmap v1826: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:56.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[45495]: audit 2022-04-22T18:00:55.288806+0000 mon.a (mon.0) 2754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:56.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[45495]: audit 2022-04-22T18:00:55.667255+0000 mon.a (mon.0) 2755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[46934]: cluster 2022-04-22T18:00:54.703966+0000 mgr.y (mgr.34107) 1240 : cluster [DBG] pgmap v1826: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:00:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[46934]: audit 2022-04-22T18:00:55.288806+0000 mon.a (mon.0) 2754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:00:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:55 smithi115 bash[46934]: audit 2022-04-22T18:00:55.667255+0000 mon.a (mon.0) 2755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:00:56.471 INFO:tasks.workunit.client.0.smithi115.stdout: 832: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:56.471 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-esKOpGnA3tivN8d off=2901834 len=571361 2022-04-22T18:00:56.473 INFO:tasks.workunit.client.0.smithi115.stdout:op 4224 completed, throughput=5MB/sec 2022-04-22T18:00:56.474 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H7sHrNFsnMRblqZ off=2525300 len=815343 2022-04-22T18:00:56.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 4225 completed, throughput=5MB/sec 2022-04-22T18:00:56.480 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ktQmAAWmVDCYGnp off=9098871 len=150059 2022-04-22T18:00:56.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 4226 completed, throughput=5MB/sec 2022-04-22T18:00:56.481 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k5-6CWxTaDaMvGI off=784146 len=779654 2022-04-22T18:00:56.484 INFO:tasks.workunit.client.0.smithi115.stdout:op 4227 completed, throughput=5MB/sec 2022-04-22T18:00:56.484 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xaYioGVseyzMQto off=8221824 len=874172 2022-04-22T18:00:56.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 4228 completed, throughput=5MB/sec 2022-04-22T18:00:56.486 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C_QNVAIRR0Gsk-P off=20356641 len=1903895 2022-04-22T18:00:56.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 4229 completed, throughput=5MB/sec 2022-04-22T18:00:57.490 INFO:tasks.workunit.client.0.smithi115.stdout: 833: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:57.490 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dEStZso_0WYKqp2 off=10866506 len=1148545 2022-04-22T18:00:57.494 INFO:tasks.workunit.client.0.smithi115.stdout:op 4230 completed, throughput=5MB/sec 2022-04-22T18:00:57.494 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uKwyygAUf4jDTL- off=11646153 len=538884 2022-04-22T18:00:57.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 4231 completed, throughput=5MB/sec 2022-04-22T18:00:57.497 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IJV4R7ZEz-q41Ta off=1788462 len=27616 2022-04-22T18:00:57.501 INFO:tasks.workunit.client.0.smithi115.stdout:op 4232 completed, throughput=5MB/sec 2022-04-22T18:00:57.502 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nXLVSJ15JKHho-B off=15133554 len=215387 2022-04-22T18:00:57.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 4233 completed, throughput=5MB/sec 2022-04-22T18:00:57.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dx9fnukj4xpSlG7 off=803635 len=2052278 2022-04-22T18:00:57.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 4234 completed, throughput=5MB/sec 2022-04-22T18:00:57.518 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MDvFjAr85r21yjM off=15883752 len=1446571 2022-04-22T18:00:57.519 INFO:tasks.workunit.client.0.smithi115.stdout:op 4235 completed, throughput=5MB/sec 2022-04-22T18:00:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:58 smithi121 bash[36902]: cluster 2022-04-22T18:00:56.705089+0000 mgr.y (mgr.34107) 1241 : cluster [DBG] pgmap v1827: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:00:58.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:00:58 smithi121 bash[36902]: audit 2022-04-22T18:00:57.674793+0000 mon.a (mon.0) 2756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:58.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:57 smithi115 bash[45495]: cluster 2022-04-22T18:00:56.705089+0000 mgr.y (mgr.34107) 1241 : cluster [DBG] pgmap v1827: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:00:58.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:00:57 smithi115 bash[45495]: audit 2022-04-22T18:00:57.674793+0000 mon.a (mon.0) 2756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:58.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:58 smithi115 bash[46934]: cluster 2022-04-22T18:00:56.705089+0000 mgr.y (mgr.34107) 1241 : cluster [DBG] pgmap v1827: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:00:58.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:00:58 smithi115 bash[46934]: audit 2022-04-22T18:00:57.674793+0000 mon.a (mon.0) 2756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:00:58.519 INFO:tasks.workunit.client.0.smithi115.stdout: 834: throughput=4.99MB/sec pending data=0 2022-04-22T18:00:58.520 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EtKdEu780bvithd off=3017494 len=479069 2022-04-22T18:00:58.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 4236 completed, throughput=5MB/sec 2022-04-22T18:00:58.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oSPU_VBHzSoTz2x off=10278761 len=382563 2022-04-22T18:00:58.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 4237 completed, throughput=5MB/sec 2022-04-22T18:00:58.526 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jKExQminUVJ1hl_ off=6480009 len=1725267 2022-04-22T18:00:58.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 4238 completed, throughput=5MB/sec 2022-04-22T18:00:58.535 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vpp28L1W0e1vgAJ off=2699218 len=1874961 2022-04-22T18:00:58.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 4239 completed, throughput=5MB/sec 2022-04-22T18:00:58.541 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0_PZYpOMbNhyVRH off=4116661 len=1626494 2022-04-22T18:00:58.543 INFO:tasks.workunit.client.0.smithi115.stdout:op 4240 completed, throughput=5MB/sec 2022-04-22T18:00:59.543 INFO:tasks.workunit.client.0.smithi115.stdout: 835: throughput=5MB/sec pending data=0 2022-04-22T18:00:59.544 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XZIcDJ5-gkkkfNB off=371653 len=275548 2022-04-22T18:00:59.549 INFO:tasks.workunit.client.0.smithi115.stdout:op 4241 completed, throughput=5MB/sec 2022-04-22T18:00:59.549 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NhFegIKcEVXba1L off=1310604 len=1515593 2022-04-22T18:00:59.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 4242 completed, throughput=5MB/sec 2022-04-22T18:00:59.552 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LZaHVF-7jtFz6Sj off=2333815 len=942189 2022-04-22T18:00:59.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 4243 completed, throughput=5MB/sec 2022-04-22T18:00:59.558 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PXvk0uwjNi0BjLJ off=5362392 len=1415175 2022-04-22T18:00:59.560 INFO:tasks.workunit.client.0.smithi115.stdout:op 4244 completed, throughput=5MB/sec 2022-04-22T18:00:59.560 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KauVNQYYDOeod4H off=16000209 len=249040 2022-04-22T18:00:59.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 4245 completed, throughput=5MB/sec 2022-04-22T18:01:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:00 smithi121 bash[36902]: cluster 2022-04-22T18:00:58.706314+0000 mgr.y (mgr.34107) 1242 : cluster [DBG] pgmap v1828: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:01:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:00 smithi115 bash[45495]: cluster 2022-04-22T18:00:58.706314+0000 mgr.y (mgr.34107) 1242 : cluster [DBG] pgmap v1828: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:01:00.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:00 smithi115 bash[46934]: cluster 2022-04-22T18:00:58.706314+0000 mgr.y (mgr.34107) 1242 : cluster [DBG] pgmap v1828: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:01:00.564 INFO:tasks.workunit.client.0.smithi115.stdout: 836: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:00.564 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eYErTS_U9-zT7Gx off=3589365 len=1659398 2022-04-22T18:01:00.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 4246 completed, throughput=5MB/sec 2022-04-22T18:01:00.568 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qevYcYPHAbTH1Be off=227936 len=772655 2022-04-22T18:01:00.571 INFO:tasks.workunit.client.0.smithi115.stdout:op 4247 completed, throughput=5MB/sec 2022-04-22T18:01:00.572 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2sYT5Ktm6ygzFAA off=10264623 len=1681978 2022-04-22T18:01:00.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 4248 completed, throughput=5MB/sec 2022-04-22T18:01:00.575 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ehoRz1uUgAy3bMU off=7883544 len=509687 2022-04-22T18:01:00.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 4249 completed, throughput=5MB/sec 2022-04-22T18:01:00.577 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VDk4iXYjGUhG_UF off=5644902 len=765583 2022-04-22T18:01:00.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 4250 completed, throughput=5MB/sec 2022-04-22T18:01:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:01 smithi121 bash[36902]: audit 2022-04-22T18:01:00.291777+0000 mon.a (mon.0) 2757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:01 smithi121 bash[36902]: audit 2022-04-22T18:01:00.670763+0000 mon.a (mon.0) 2758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:01 smithi115 bash[45495]: audit 2022-04-22T18:01:00.291777+0000 mon.a (mon.0) 2757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:01 smithi115 bash[45495]: audit 2022-04-22T18:01:00.670763+0000 mon.a (mon.0) 2758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:01 smithi115 bash[46934]: audit 2022-04-22T18:01:00.291777+0000 mon.a (mon.0) 2757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:01 smithi115 bash[46934]: audit 2022-04-22T18:01:00.670763+0000 mon.a (mon.0) 2758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:01.583 INFO:tasks.workunit.client.0.smithi115.stdout: 837: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:01.583 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1akhOsD5e9jXEU_ off=2715140 len=1713543 2022-04-22T18:01:01.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 4251 completed, throughput=5MB/sec 2022-04-22T18:01:01.587 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0eoz4XatIudiiEA off=6603833 len=1005671 2022-04-22T18:01:01.593 INFO:tasks.workunit.client.0.smithi115.stdout:op 4252 completed, throughput=5MB/sec 2022-04-22T18:01:01.593 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yYaLN2GTDrUdu1p off=9376053 len=414492 2022-04-22T18:01:01.595 INFO:tasks.workunit.client.0.smithi115.stdout:op 4253 completed, throughput=5MB/sec 2022-04-22T18:01:01.595 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PFljSBuLwwWoKlg off=12759908 len=1105272 2022-04-22T18:01:01.603 INFO:tasks.workunit.client.0.smithi115.stdout:op 4254 completed, throughput=5MB/sec 2022-04-22T18:01:01.620 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BhB_XDAIhIOWsU6 off=802878 len=1084232 2022-04-22T18:01:01.621 INFO:tasks.workunit.client.0.smithi115.stdout:op 4255 completed, throughput=5MB/sec 2022-04-22T18:01:02.013 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:01.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:02 smithi121 bash[36902]: cluster 2022-04-22T18:01:00.707451+0000 mgr.y (mgr.34107) 1243 : cluster [DBG] pgmap v1829: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:02 smithi115 bash[45495]: cluster 2022-04-22T18:01:00.707451+0000 mgr.y (mgr.34107) 1243 : cluster [DBG] pgmap v1829: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:02 smithi115 bash[46934]: cluster 2022-04-22T18:01:00.707451+0000 mgr.y (mgr.34107) 1243 : cluster [DBG] pgmap v1829: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:02.610 INFO:tasks.workunit.client.0.smithi115.stdout: 838: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:02.648 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IWi2ekJiFjbIEKT off=5337919 len=1601346 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4256 completed, throughput=5MB/sec 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q5zDcjmowtFSPtd off=1891569 len=1236263 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4257 completed, throughput=5MB/sec 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jqG5G1400wXGPo1 off=8522795 len=516594 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4258 completed, throughput=5MB/sec 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MkR3VqrpbRiGauY off=10127998 len=1061692 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4259 completed, throughput=5MB/sec 2022-04-22T18:01:02.650 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JTr6FLDZFh7dyQ4 off=4674370 len=961356 2022-04-22T18:01:02.651 INFO:tasks.workunit.client.0.smithi115.stdout:op 4260 completed, throughput=5MB/sec 2022-04-22T18:01:03.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:03 smithi115 bash[45495]: audit 2022-04-22T18:01:02.677834+0000 mon.a (mon.0) 2759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:03 smithi115 bash[46934]: audit 2022-04-22T18:01:02.677834+0000 mon.a (mon.0) 2759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:03 smithi121 bash[36902]: audit 2022-04-22T18:01:02.677834+0000 mon.a (mon.0) 2759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:03.630 INFO:tasks.workunit.client.0.smithi115.stdout: 839: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:03.728 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NqxdRIi_2MuNbtu off=6625743 len=1708208 2022-04-22T18:01:03.730 INFO:tasks.workunit.client.0.smithi115.stdout:op 4261 completed, throughput=5MB/sec 2022-04-22T18:01:03.730 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iaNLk-4q6vNkhl- off=11551744 len=431454 2022-04-22T18:01:03.730 INFO:tasks.workunit.client.0.smithi115.stdout:op 4262 completed, throughput=5MB/sec 2022-04-22T18:01:03.732 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UmTIG3ZUOC0_rKx off=17113359 len=17234 2022-04-22T18:01:03.732 INFO:tasks.workunit.client.0.smithi115.stdout:op 4263 completed, throughput=5MB/sec 2022-04-22T18:01:03.732 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k5EbZaLh3Iyn1ue off=5299714 len=1076144 2022-04-22T18:01:03.732 INFO:tasks.workunit.client.0.smithi115.stdout:op 4264 completed, throughput=5MB/sec 2022-04-22T18:01:03.732 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X4XAjIuHEuTr73J off=6036356 len=603610 2022-04-22T18:01:03.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 4265 completed, throughput=5MB/sec 2022-04-22T18:01:03.733 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7JQwQ9zD0F6v3KQ off=12533880 len=153333 2022-04-22T18:01:03.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 4266 completed, throughput=5MB/sec 2022-04-22T18:01:03.733 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZvT2durAF-o1DNd off=751282 len=1697267 2022-04-22T18:01:03.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 4267 completed, throughput=5MB/sec 2022-04-22T18:01:04.023 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:03.608Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:04 smithi115 bash[45495]: cluster 2022-04-22T18:01:02.708568+0000 mgr.y (mgr.34107) 1244 : cluster [DBG] pgmap v1830: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:01:04.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:04 smithi115 bash[46934]: cluster 2022-04-22T18:01:02.708568+0000 mgr.y (mgr.34107) 1244 : cluster [DBG] pgmap v1830: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:01:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:04 smithi121 bash[36902]: cluster 2022-04-22T18:01:02.708568+0000 mgr.y (mgr.34107) 1244 : cluster [DBG] pgmap v1830: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:01:04.661 INFO:tasks.workunit.client.0.smithi115.stdout: 840: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:04.661 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xFH57wv8MRsk8_i off=1939496 len=239761 2022-04-22T18:01:04.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 4268 completed, throughput=4.99MB/sec 2022-04-22T18:01:04.662 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WWix2rXDBNkSYJP off=17110643 len=610163 2022-04-22T18:01:04.667 INFO:tasks.workunit.client.0.smithi115.stdout:op 4269 completed, throughput=5MB/sec 2022-04-22T18:01:04.668 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PgImgFvoax6JljN off=7569685 len=1841342 2022-04-22T18:01:04.671 INFO:tasks.workunit.client.0.smithi115.stdout:op 4270 completed, throughput=5MB/sec 2022-04-22T18:01:04.671 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0BK0uU1PWXkUPRB off=7413213 len=1460856 2022-04-22T18:01:04.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 4271 completed, throughput=5MB/sec 2022-04-22T18:01:04.674 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FfqQoQvH3YQawgR off=7800498 len=2087924 2022-04-22T18:01:04.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 4272 completed, throughput=5MB/sec 2022-04-22T18:01:05.677 INFO:tasks.workunit.client.0.smithi115.stdout: 841: throughput=5MB/sec pending data=0 2022-04-22T18:01:05.710 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5jajVjMMZexewKE off=11106106 len=1370809 2022-04-22T18:01:05.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 4273 completed, throughput=5MB/sec 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-x3mNM-XLRaP7gD- off=15164918 len=1169180 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 4274 completed, throughput=5MB/sec 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EH5xd_6ACTXs1k2 off=2760164 len=751123 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 4275 completed, throughput=5MB/sec 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-19riV0UOqIu2rl1 off=8643910 len=993682 2022-04-22T18:01:05.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 4276 completed, throughput=5MB/sec 2022-04-22T18:01:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[46934]: cluster 2022-04-22T18:01:04.709745+0000 mgr.y (mgr.34107) 1245 : cluster [DBG] pgmap v1831: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[46934]: audit 2022-04-22T18:01:05.294874+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[46934]: audit 2022-04-22T18:01:05.674558+0000 mon.a (mon.0) 2761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[45495]: cluster 2022-04-22T18:01:04.709745+0000 mgr.y (mgr.34107) 1245 : cluster [DBG] pgmap v1831: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[45495]: audit 2022-04-22T18:01:05.294874+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:06 smithi115 bash[45495]: audit 2022-04-22T18:01:05.674558+0000 mon.a (mon.0) 2761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:06 smithi121 bash[36902]: cluster 2022-04-22T18:01:04.709745+0000 mgr.y (mgr.34107) 1245 : cluster [DBG] pgmap v1831: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:06 smithi121 bash[36902]: audit 2022-04-22T18:01:05.294874+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:06 smithi121 bash[36902]: audit 2022-04-22T18:01:05.674558+0000 mon.a (mon.0) 2761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:06.695 INFO:tasks.workunit.client.0.smithi115.stdout: 842: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:06.696 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_0xufSpA2fFQtUJ off=4274321 len=112611 2022-04-22T18:01:06.697 INFO:tasks.workunit.client.0.smithi115.stdout:op 4277 completed, throughput=4.99MB/sec 2022-04-22T18:01:06.697 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_HWgRhjZFd_Fptp off=872695 len=1771389 2022-04-22T18:01:06.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 4278 completed, throughput=5MB/sec 2022-04-22T18:01:06.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MLrYmbRPNgpWTmW off=3144259 len=384501 2022-04-22T18:01:06.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 4279 completed, throughput=5MB/sec 2022-04-22T18:01:06.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NDvX1g8kVLethTB off=721109 len=177356 2022-04-22T18:01:06.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 4280 completed, throughput=5MB/sec 2022-04-22T18:01:06.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dxPdsaSNiX5NvI6 off=2479826 len=575025 2022-04-22T18:01:06.713 INFO:tasks.workunit.client.0.smithi115.stdout:op 4281 completed, throughput=5MB/sec 2022-04-22T18:01:06.713 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lznRfHVcmTcXqIU off=15574849 len=128169 2022-04-22T18:01:06.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 4282 completed, throughput=5MB/sec 2022-04-22T18:01:06.717 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VD4M-htBTco9K5D off=5385638 len=823887 2022-04-22T18:01:06.721 INFO:tasks.workunit.client.0.smithi115.stdout:op 4283 completed, throughput=5MB/sec 2022-04-22T18:01:06.722 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FzqJ2LPTlATqtBe off=3119225 len=849658 2022-04-22T18:01:06.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 4284 completed, throughput=5MB/sec 2022-04-22T18:01:06.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Phrbl__vowrhkkf off=21739256 len=1010217 2022-04-22T18:01:06.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 4285 completed, throughput=5MB/sec 2022-04-22T18:01:07.729 INFO:tasks.workunit.client.0.smithi115.stdout: 843: throughput=5MB/sec pending data=0 2022-04-22T18:01:07.730 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GqKy3iHwSSVXHKJ off=4517476 len=1790122 2022-04-22T18:01:07.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 4286 completed, throughput=5MB/sec 2022-04-22T18:01:07.739 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ouRhbSAxnUIIo3R off=13929109 len=652966 2022-04-22T18:01:07.743 INFO:tasks.workunit.client.0.smithi115.stdout:op 4287 completed, throughput=5MB/sec 2022-04-22T18:01:07.743 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0TKzf3VwfXb8B1O off=1698314 len=165524 2022-04-22T18:01:07.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 4288 completed, throughput=5MB/sec 2022-04-22T18:01:07.744 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5UndnuF9wwFjeW_ off=1975675 len=968213 2022-04-22T18:01:07.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 4289 completed, throughput=5MB/sec 2022-04-22T18:01:07.750 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IYcIPqPtyUbr6xb off=4128209 len=1823061 2022-04-22T18:01:07.758 INFO:tasks.workunit.client.0.smithi115.stdout:op 4290 completed, throughput=5MB/sec 2022-04-22T18:01:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:08 smithi115 bash[45495]: cluster 2022-04-22T18:01:06.710603+0000 mgr.y (mgr.34107) 1246 : cluster [DBG] pgmap v1832: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:08 smithi115 bash[45495]: audit 2022-04-22T18:01:07.680823+0000 mon.a (mon.0) 2762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:08 smithi115 bash[46934]: cluster 2022-04-22T18:01:06.710603+0000 mgr.y (mgr.34107) 1246 : cluster [DBG] pgmap v1832: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:08 smithi115 bash[46934]: audit 2022-04-22T18:01:07.680823+0000 mon.a (mon.0) 2762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:08 smithi121 bash[36902]: cluster 2022-04-22T18:01:06.710603+0000 mgr.y (mgr.34107) 1246 : cluster [DBG] pgmap v1832: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:08 smithi121 bash[36902]: audit 2022-04-22T18:01:07.680823+0000 mon.a (mon.0) 2762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:08.759 INFO:tasks.workunit.client.0.smithi115.stdout: 844: throughput=5MB/sec pending data=0 2022-04-22T18:01:08.759 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zePv8NsPsu0NE-A off=4146185 len=1126822 2022-04-22T18:01:08.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 4291 completed, throughput=5MB/sec 2022-04-22T18:01:08.763 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nFbplF1cEg1ErPf off=8155754 len=1324738 2022-04-22T18:01:08.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 4292 completed, throughput=5MB/sec 2022-04-22T18:01:08.767 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--kQNgnFR1GkV5TU off=10928502 len=56871 2022-04-22T18:01:08.771 INFO:tasks.workunit.client.0.smithi115.stdout:op 4293 completed, throughput=5MB/sec 2022-04-22T18:01:08.771 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Sqk4QZfeSsFFQJu off=11719631 len=1704748 2022-04-22T18:01:08.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 4294 completed, throughput=5MB/sec 2022-04-22T18:01:08.775 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4hx9pih5Nt3agDL off=1480807 len=1624297 2022-04-22T18:01:08.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 4295 completed, throughput=5MB/sec 2022-04-22T18:01:09.783 INFO:tasks.workunit.client.0.smithi115.stdout: 845: throughput=5MB/sec pending data=0 2022-04-22T18:01:09.784 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gtvtC0JWY89JfXI off=3205471 len=1253822 2022-04-22T18:01:09.792 INFO:tasks.workunit.client.0.smithi115.stdout:op 4296 completed, throughput=5MB/sec 2022-04-22T18:01:09.792 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rLBSrRILagFb2E7 off=15416857 len=1304545 2022-04-22T18:01:09.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 4297 completed, throughput=5MB/sec 2022-04-22T18:01:09.798 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RuvFMyZ3ucXfWqC off=882354 len=896753 2022-04-22T18:01:09.803 INFO:tasks.workunit.client.0.smithi115.stdout:op 4298 completed, throughput=5MB/sec 2022-04-22T18:01:09.803 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ddhGptccC0sLpUS off=7787417 len=1021757 2022-04-22T18:01:09.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 4299 completed, throughput=5MB/sec 2022-04-22T18:01:10.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:10 smithi115 bash[45495]: cluster 2022-04-22T18:01:08.711874+0000 mgr.y (mgr.34107) 1247 : cluster [DBG] pgmap v1833: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:10 smithi115 bash[46934]: cluster 2022-04-22T18:01:08.711874+0000 mgr.y (mgr.34107) 1247 : cluster [DBG] pgmap v1833: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:10 smithi121 bash[36902]: cluster 2022-04-22T18:01:08.711874+0000 mgr.y (mgr.34107) 1247 : cluster [DBG] pgmap v1833: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:01:10.811 INFO:tasks.workunit.client.0.smithi115.stdout: 846: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:10.811 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PAqq1xPcPqN8Eyw off=11632834 len=1870049 2022-04-22T18:01:10.822 INFO:tasks.workunit.client.0.smithi115.stdout:op 4300 completed, throughput=5MB/sec 2022-04-22T18:01:10.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pgAWS03_TaXFCfj off=8435215 len=1586407 2022-04-22T18:01:10.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 4301 completed, throughput=5MB/sec 2022-04-22T18:01:10.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PTWVVRqrevZcqdi off=6376270 len=352017 2022-04-22T18:01:10.832 INFO:tasks.workunit.client.0.smithi115.stdout:op 4302 completed, throughput=5MB/sec 2022-04-22T18:01:10.832 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WZK4JxsDH98q3Jt off=5216861 len=471974 2022-04-22T18:01:10.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 4303 completed, throughput=5MB/sec 2022-04-22T18:01:10.835 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yMaHlzuVunJ_sov off=1056240 len=42117 2022-04-22T18:01:10.835 INFO:tasks.workunit.client.0.smithi115.stdout:op 4304 completed, throughput=5MB/sec 2022-04-22T18:01:10.836 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VKJn4UvVC1wYVSs off=1455922 len=1468320 2022-04-22T18:01:10.843 INFO:tasks.workunit.client.0.smithi115.stdout:op 4305 completed, throughput=5MB/sec 2022-04-22T18:01:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:11 smithi115 bash[45495]: audit 2022-04-22T18:01:10.297893+0000 mon.a (mon.0) 2763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:11 smithi115 bash[45495]: audit 2022-04-22T18:01:10.678529+0000 mon.a (mon.0) 2764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:11 smithi115 bash[46934]: audit 2022-04-22T18:01:10.297893+0000 mon.a (mon.0) 2763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:11 smithi115 bash[46934]: audit 2022-04-22T18:01:10.678529+0000 mon.a (mon.0) 2764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:11 smithi121 bash[36902]: audit 2022-04-22T18:01:10.297893+0000 mon.a (mon.0) 2763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:11.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:11 smithi121 bash[36902]: audit 2022-04-22T18:01:10.678529+0000 mon.a (mon.0) 2764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:11.843 INFO:tasks.workunit.client.0.smithi115.stdout: 847: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:11.843 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e1YS3e90qg4ltGm off=288533 len=1739194 2022-04-22T18:01:11.849 INFO:tasks.workunit.client.0.smithi115.stdout:op 4306 completed, throughput=5MB/sec 2022-04-22T18:01:11.849 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yubC2s4FRO0WFvg off=9054209 len=1253984 2022-04-22T18:01:11.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 4307 completed, throughput=5MB/sec 2022-04-22T18:01:11.851 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RovNg6lMaudc77y off=1047539 len=1659096 2022-04-22T18:01:11.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 4308 completed, throughput=5MB/sec 2022-04-22T18:01:12.039 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:11.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:12.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:12 smithi115 bash[45495]: cluster 2022-04-22T18:01:10.713037+0000 mgr.y (mgr.34107) 1248 : cluster [DBG] pgmap v1834: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:12.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:12 smithi115 bash[46934]: cluster 2022-04-22T18:01:10.713037+0000 mgr.y (mgr.34107) 1248 : cluster [DBG] pgmap v1834: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:12 smithi121 bash[36902]: cluster 2022-04-22T18:01:10.713037+0000 mgr.y (mgr.34107) 1248 : cluster [DBG] pgmap v1834: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:12.856 INFO:tasks.workunit.client.0.smithi115.stdout: 848: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:12.856 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C5qfDP2f3C2Z9jA off=15496154 len=669835 2022-04-22T18:01:12.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 4309 completed, throughput=4.99MB/sec 2022-04-22T18:01:12.862 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PsGasVIq-WhTBdV off=700185 len=678894 2022-04-22T18:01:12.865 INFO:tasks.workunit.client.0.smithi115.stdout:op 4310 completed, throughput=5MB/sec 2022-04-22T18:01:12.865 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-USSEUEg0_uhqBHD off=7966763 len=95106 2022-04-22T18:01:12.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 4311 completed, throughput=5MB/sec 2022-04-22T18:01:12.870 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ITidod14b8VX2S_ off=11577249 len=811173 2022-04-22T18:01:12.872 INFO:tasks.workunit.client.0.smithi115.stdout:op 4312 completed, throughput=5MB/sec 2022-04-22T18:01:12.872 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YxgFc6DjtV_zvHh off=4481310 len=1147672 2022-04-22T18:01:12.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 4313 completed, throughput=5MB/sec 2022-04-22T18:01:12.880 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Evq4EvbeFvrmu63 off=316363 len=1901779 2022-04-22T18:01:12.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 4314 completed, throughput=5MB/sec 2022-04-22T18:01:12.883 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G5QCMX1KDGKiDpY off=1602320 len=1209645 2022-04-22T18:01:12.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 4315 completed, throughput=5MB/sec 2022-04-22T18:01:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:13 smithi121 bash[36902]: audit 2022-04-22T18:01:12.684082+0000 mon.a (mon.0) 2765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:13 smithi115 bash[45495]: audit 2022-04-22T18:01:12.684082+0000 mon.a (mon.0) 2765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:13.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:13 smithi115 bash[46934]: audit 2022-04-22T18:01:12.684082+0000 mon.a (mon.0) 2765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:13.890 INFO:tasks.workunit.client.0.smithi115.stdout: 849: throughput=5MB/sec pending data=0 2022-04-22T18:01:13.891 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gnX8Y_XRTSKbaCy off=2505090 len=633293 2022-04-22T18:01:13.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 4316 completed, throughput=5MB/sec 2022-04-22T18:01:13.893 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wx-_TCzSQgLq3Qa off=229939 len=1418795 2022-04-22T18:01:13.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 4317 completed, throughput=5MB/sec 2022-04-22T18:01:13.896 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UIULrcaSqetdR3q off=3623909 len=1580763 2022-04-22T18:01:13.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 4318 completed, throughput=5MB/sec 2022-04-22T18:01:13.902 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TR5-MV4ZXLu9gzv off=17353857 len=1120138 2022-04-22T18:01:13.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 4319 completed, throughput=5MB/sec 2022-04-22T18:01:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:13.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:14 smithi121 bash[36902]: cluster 2022-04-22T18:01:12.714174+0000 mgr.y (mgr.34107) 1249 : cluster [DBG] pgmap v1835: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:14 smithi121 bash[36902]: audit 2022-04-22T18:01:13.602211+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:01:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:14 smithi121 bash[36902]: audit 2022-04-22T18:01:13.602633+0000 mon.a (mon.0) 2767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:01:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[46934]: cluster 2022-04-22T18:01:12.714174+0000 mgr.y (mgr.34107) 1249 : cluster [DBG] pgmap v1835: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[46934]: audit 2022-04-22T18:01:13.602211+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:01:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[46934]: audit 2022-04-22T18:01:13.602633+0000 mon.a (mon.0) 2767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:01:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[45495]: cluster 2022-04-22T18:01:12.714174+0000 mgr.y (mgr.34107) 1249 : cluster [DBG] pgmap v1835: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[45495]: audit 2022-04-22T18:01:13.602211+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:01:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:14 smithi115 bash[45495]: audit 2022-04-22T18:01:13.602633+0000 mon.a (mon.0) 2767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:01:14.910 INFO:tasks.workunit.client.0.smithi115.stdout: 850: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:14.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_Q-_G58Scca4sB7 off=0 len=1472910 2022-04-22T18:01:14.915 INFO:tasks.workunit.client.0.smithi115.stdout:op 4320 completed, throughput=5MB/sec 2022-04-22T18:01:14.915 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-elASsVcxlVcYSTi off=1251874 len=480316 2022-04-22T18:01:14.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 4321 completed, throughput=5MB/sec 2022-04-22T18:01:14.917 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0Up0tLLDshfhliw off=8701184 len=524761 2022-04-22T18:01:14.919 INFO:tasks.workunit.client.0.smithi115.stdout:op 4322 completed, throughput=5MB/sec 2022-04-22T18:01:14.919 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eh15g_JQqj3agC0 off=13791326 len=163725 2022-04-22T18:01:14.923 INFO:tasks.workunit.client.0.smithi115.stdout:op 4323 completed, throughput=5MB/sec 2022-04-22T18:01:14.923 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HvE4p6qTNAZE9Fn off=1587928 len=1108252 2022-04-22T18:01:14.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 4324 completed, throughput=5MB/sec 2022-04-22T18:01:14.929 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3NRmW4qPuSnlib- off=4551096 len=35119 2022-04-22T18:01:14.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 4325 completed, throughput=5MB/sec 2022-04-22T18:01:14.929 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hSGwKiVr3r26kyk off=18945402 len=533686 2022-04-22T18:01:14.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 4326 completed, throughput=5MB/sec 2022-04-22T18:01:14.933 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P72qzBRENz3oE9I off=8437422 len=1596147 2022-04-22T18:01:14.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 4327 completed, throughput=5MB/sec 2022-04-22T18:01:15.940 INFO:tasks.workunit.client.0.smithi115.stdout: 851: throughput=5MB/sec pending data=0 2022-04-22T18:01:15.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zkhj7ekddNtmInh off=3915994 len=2090211 2022-04-22T18:01:15.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 4328 completed, throughput=5MB/sec 2022-04-22T18:01:15.944 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ec186yaXSq5wTkD off=11819855 len=1747861 2022-04-22T18:01:15.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 4329 completed, throughput=5MB/sec 2022-04-22T18:01:15.953 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_PVG8APsJA4kB0G off=2883683 len=1339131 2022-04-22T18:01:15.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 4330 completed, throughput=5MB/sec 2022-04-22T18:01:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:16 smithi121 bash[36902]: cluster 2022-04-22T18:01:14.715310+0000 mgr.y (mgr.34107) 1250 : cluster [DBG] pgmap v1836: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:16 smithi121 bash[36902]: audit 2022-04-22T18:01:15.300840+0000 mon.a (mon.0) 2768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:16 smithi121 bash[36902]: audit 2022-04-22T18:01:15.681859+0000 mon.a (mon.0) 2769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[45495]: cluster 2022-04-22T18:01:14.715310+0000 mgr.y (mgr.34107) 1250 : cluster [DBG] pgmap v1836: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[45495]: audit 2022-04-22T18:01:15.300840+0000 mon.a (mon.0) 2768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[45495]: audit 2022-04-22T18:01:15.681859+0000 mon.a (mon.0) 2769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[46934]: cluster 2022-04-22T18:01:14.715310+0000 mgr.y (mgr.34107) 1250 : cluster [DBG] pgmap v1836: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[46934]: audit 2022-04-22T18:01:15.300840+0000 mon.a (mon.0) 2768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:16 smithi115 bash[46934]: audit 2022-04-22T18:01:15.681859+0000 mon.a (mon.0) 2769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:16.962 INFO:tasks.workunit.client.0.smithi115.stdout: 852: throughput=5MB/sec pending data=0 2022-04-22T18:01:16.962 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NrS_7GAn5P_e9D7 off=2742391 len=1428888 2022-04-22T18:01:16.964 INFO:tasks.workunit.client.0.smithi115.stdout:op 4331 completed, throughput=5MB/sec 2022-04-22T18:01:16.964 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I8TC_333dK7ykaS off=8538608 len=1568423 2022-04-22T18:01:16.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 4332 completed, throughput=5MB/sec 2022-04-22T18:01:16.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v_GQ3FtVfMcRBbZ off=750165 len=301870 2022-04-22T18:01:16.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 4333 completed, throughput=5MB/sec 2022-04-22T18:01:16.968 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LZouTgHeFH3q88k off=4115475 len=22389 2022-04-22T18:01:16.973 INFO:tasks.workunit.client.0.smithi115.stdout:op 4334 completed, throughput=5MB/sec 2022-04-22T18:01:16.973 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HP6ty9U3q1i0gqH off=1204420 len=1201470 2022-04-22T18:01:16.979 INFO:tasks.workunit.client.0.smithi115.stdout:op 4335 completed, throughput=5MB/sec 2022-04-22T18:01:17.979 INFO:tasks.workunit.client.0.smithi115.stdout: 853: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:17.980 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zHYZxcznXg8un3q off=1620190 len=1169474 2022-04-22T18:01:17.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 4336 completed, throughput=5MB/sec 2022-04-22T18:01:17.984 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3GOATesQLl0sKH7 off=8129940 len=1475844 2022-04-22T18:01:17.992 INFO:tasks.workunit.client.0.smithi115.stdout:op 4337 completed, throughput=5MB/sec 2022-04-22T18:01:17.992 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lh72V4RE2oQ8IBh off=7083818 len=1583439 2022-04-22T18:01:18.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 4338 completed, throughput=5MB/sec 2022-04-22T18:01:18.001 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ikRokOytdT5lMId off=11798531 len=1959882 2022-04-22T18:01:18.005 INFO:tasks.workunit.client.0.smithi115.stdout:op 4339 completed, throughput=5MB/sec 2022-04-22T18:01:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:18 smithi121 bash[36902]: cluster 2022-04-22T18:01:16.716554+0000 mgr.y (mgr.34107) 1251 : cluster [DBG] pgmap v1837: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:18 smithi121 bash[36902]: audit 2022-04-22T18:01:17.687009+0000 mon.a (mon.0) 2770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:18 smithi115 bash[45495]: cluster 2022-04-22T18:01:16.716554+0000 mgr.y (mgr.34107) 1251 : cluster [DBG] pgmap v1837: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:18 smithi115 bash[45495]: audit 2022-04-22T18:01:17.687009+0000 mon.a (mon.0) 2770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:18 smithi115 bash[46934]: cluster 2022-04-22T18:01:16.716554+0000 mgr.y (mgr.34107) 1251 : cluster [DBG] pgmap v1837: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:01:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:18 smithi115 bash[46934]: audit 2022-04-22T18:01:17.687009+0000 mon.a (mon.0) 2770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:19.005 INFO:tasks.workunit.client.0.smithi115.stdout: 854: throughput=5MB/sec pending data=0 2022-04-22T18:01:19.006 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-udkp045X_f6xdy6 off=2952791 len=451628 2022-04-22T18:01:19.011 INFO:tasks.workunit.client.0.smithi115.stdout:op 4340 completed, throughput=5MB/sec 2022-04-22T18:01:19.011 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FAgZcdh3B58kVcR off=18041690 len=1223097 2022-04-22T18:01:19.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 4341 completed, throughput=5MB/sec 2022-04-22T18:01:19.019 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JzVmkslOmHRigce off=2964212 len=512206 2022-04-22T18:01:19.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 4342 completed, throughput=5MB/sec 2022-04-22T18:01:19.020 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_dLDVGbcQ7bGOWm off=10027519 len=1279210 2022-04-22T18:01:19.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 4343 completed, throughput=5MB/sec 2022-04-22T18:01:19.022 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dwlCWiUjKeP2hdl off=9378348 len=1410328 2022-04-22T18:01:19.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 4344 completed, throughput=5MB/sec 2022-04-22T18:01:20.027 INFO:tasks.workunit.client.0.smithi115.stdout: 855: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:20.028 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XSEqs5OteXjy-cd off=1435031 len=809845 2022-04-22T18:01:20.030 INFO:tasks.workunit.client.0.smithi115.stdout:op 4345 completed, throughput=5MB/sec 2022-04-22T18:01:20.030 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qY6T8wfZrNofGvp off=8367696 len=192231 2022-04-22T18:01:20.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 4346 completed, throughput=5MB/sec 2022-04-22T18:01:20.035 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o3kvphUnteOfiQw off=989543 len=501450 2022-04-22T18:01:20.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 4347 completed, throughput=5MB/sec 2022-04-22T18:01:20.036 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DBoY9VTKoccv-xe off=929428 len=2045154 2022-04-22T18:01:20.039 INFO:tasks.workunit.client.0.smithi115.stdout:op 4348 completed, throughput=5MB/sec 2022-04-22T18:01:20.039 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lnjDO2CWK22bYOx off=7716213 len=1293215 2022-04-22T18:01:20.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 4349 completed, throughput=5MB/sec 2022-04-22T18:01:20.048 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4usLI9y80KF3prx off=5469954 len=1400469 2022-04-22T18:01:20.051 INFO:tasks.workunit.client.0.smithi115.stdout:op 4350 completed, throughput=5MB/sec 2022-04-22T18:01:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:20 smithi121 bash[36902]: cluster 2022-04-22T18:01:18.717814+0000 mgr.y (mgr.34107) 1252 : cluster [DBG] pgmap v1838: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:20 smithi115 bash[45495]: cluster 2022-04-22T18:01:18.717814+0000 mgr.y (mgr.34107) 1252 : cluster [DBG] pgmap v1838: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:20 smithi115 bash[46934]: cluster 2022-04-22T18:01:18.717814+0000 mgr.y (mgr.34107) 1252 : cluster [DBG] pgmap v1838: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:21.051 INFO:tasks.workunit.client.0.smithi115.stdout: 856: throughput=5MB/sec pending data=0 2022-04-22T18:01:21.052 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s6d2R5CQSfJfv8f off=16569149 len=255687 2022-04-22T18:01:21.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 4351 completed, throughput=5MB/sec 2022-04-22T18:01:21.053 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fT-py88R2O7_xuG off=10263295 len=1570348 2022-04-22T18:01:21.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 4352 completed, throughput=5MB/sec 2022-04-22T18:01:21.063 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BHvGfq1PQ9WHNqY off=14598966 len=1634213 2022-04-22T18:01:21.072 INFO:tasks.workunit.client.0.smithi115.stdout:op 4353 completed, throughput=5MB/sec 2022-04-22T18:01:21.072 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AkwxlxKVRNn2Qyg off=8494371 len=872783 2022-04-22T18:01:21.078 INFO:tasks.workunit.client.0.smithi115.stdout:op 4354 completed, throughput=5MB/sec 2022-04-22T18:01:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:21 smithi121 bash[36902]: audit 2022-04-22T18:01:20.303826+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:21 smithi121 bash[36902]: audit 2022-04-22T18:01:20.685031+0000 mon.a (mon.0) 2772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:21 smithi115 bash[45495]: audit 2022-04-22T18:01:20.303826+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:21.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:21 smithi115 bash[45495]: audit 2022-04-22T18:01:20.685031+0000 mon.a (mon.0) 2772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:21.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:21 smithi115 bash[46934]: audit 2022-04-22T18:01:20.303826+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:21.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:21 smithi115 bash[46934]: audit 2022-04-22T18:01:20.685031+0000 mon.a (mon.0) 2772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:21.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:22.079 INFO:tasks.workunit.client.0.smithi115.stdout: 857: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:22.079 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FLa2mA4jso4trb_ off=4522243 len=1526254 2022-04-22T18:01:22.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 4355 completed, throughput=5MB/sec 2022-04-22T18:01:22.087 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jSWQVM-DanRDdIa off=1717363 len=675487 2022-04-22T18:01:22.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 4356 completed, throughput=5MB/sec 2022-04-22T18:01:22.091 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-60e3XqXPwhTHfsa off=11195266 len=1132865 2022-04-22T18:01:22.094 INFO:tasks.workunit.client.0.smithi115.stdout:op 4357 completed, throughput=5MB/sec 2022-04-22T18:01:22.094 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ADtc7JecWo-P5AT off=7515690 len=1073961 2022-04-22T18:01:22.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 4358 completed, throughput=5MB/sec 2022-04-22T18:01:22.097 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wmqqBa5a78ZDWxg off=8926134 len=426889 2022-04-22T18:01:22.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 4359 completed, throughput=5MB/sec 2022-04-22T18:01:22.099 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ab9EfdJ9FdW24en off=13741741 len=209979 2022-04-22T18:01:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 4360 completed, throughput=5MB/sec 2022-04-22T18:01:22.103 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DGUhNAqsEWGJdmO off=1920827 len=1343542 2022-04-22T18:01:22.106 INFO:tasks.workunit.client.0.smithi115.stdout:op 4361 completed, throughput=5MB/sec 2022-04-22T18:01:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:22 smithi121 bash[36902]: cluster 2022-04-22T18:01:20.718995+0000 mgr.y (mgr.34107) 1253 : cluster [DBG] pgmap v1839: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:22.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:22 smithi115 bash[45495]: cluster 2022-04-22T18:01:20.718995+0000 mgr.y (mgr.34107) 1253 : cluster [DBG] pgmap v1839: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:22 smithi115 bash[46934]: cluster 2022-04-22T18:01:20.718995+0000 mgr.y (mgr.34107) 1253 : cluster [DBG] pgmap v1839: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:23.106 INFO:tasks.workunit.client.0.smithi115.stdout: 858: throughput=5MB/sec pending data=0 2022-04-22T18:01:23.167 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jk0SLLaMAuqUJQk off=2011790 len=561665 2022-04-22T18:01:23.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 4362 completed, throughput=5MB/sec 2022-04-22T18:01:23.168 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o4aApSyDyDiz3Sv off=4481863 len=417277 2022-04-22T18:01:23.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 4363 completed, throughput=5MB/sec 2022-04-22T18:01:23.168 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CRi0TrBs2E9Edp7 off=10397988 len=871918 2022-04-22T18:01:23.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 4364 completed, throughput=5MB/sec 2022-04-22T18:01:23.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r_bvP3GiSgD7I8p off=1934989 len=905783 2022-04-22T18:01:23.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 4365 completed, throughput=5MB/sec 2022-04-22T18:01:23.169 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C_u6JnnXoYD4Vc- off=2740983 len=1669585 2022-04-22T18:01:23.169 INFO:tasks.workunit.client.0.smithi115.stdout:op 4366 completed, throughput=5MB/sec 2022-04-22T18:01:23.322 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:01:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:23 smithi121 bash[36902]: audit 2022-04-22T18:01:22.690355+0000 mon.a (mon.0) 2773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:23.563 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:23 smithi115 bash[45495]: audit 2022-04-22T18:01:22.690355+0000 mon.a (mon.0) 2773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:23.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:23 smithi115 bash[46934]: audit 2022-04-22T18:01:22.690355+0000 mon.a (mon.0) 2773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:01:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (24m) 3m ago 38m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:01:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (37m) 4m ago 37m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (26m) 3m ago 39m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (25m) 4m ago 39m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (33m) 4m ago 43m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (34m) 3m ago 45m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (33m) 3m ago 45m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (32m) 4m ago 43m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (32m) 3m ago 43m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (25m) 3m ago 38m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (25m) 4m ago 38m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (31m) 3m ago 43m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:01:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (30m) 3m ago 42m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (30m) 3m ago 42m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (29m) 3m ago 41m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (28m) 4m ago 41m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (28m) 4m ago 40m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (27m) 4m ago 40m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (26m) 4m ago 39m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:01:23.670 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (25m) 4m ago 38m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:01:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:23.607Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:01:24.075 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:01:24.076 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:01:24.120 INFO:tasks.workunit.client.0.smithi115.stdout: 859: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:24.120 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6aXYCws0hXectQw off=1060051 len=1687555 2022-04-22T18:01:24.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 4367 completed, throughput=5MB/sec 2022-04-22T18:01:24.131 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iSIDOl5pH1-oQns off=2733265 len=37049 2022-04-22T18:01:24.134 INFO:tasks.workunit.client.0.smithi115.stdout:op 4368 completed, throughput=5MB/sec 2022-04-22T18:01:24.134 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9L1C2chrlEqGZf_ off=4967953 len=157907 2022-04-22T18:01:24.138 INFO:tasks.workunit.client.0.smithi115.stdout:op 4369 completed, throughput=5MB/sec 2022-04-22T18:01:24.138 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MOi3sR7j_gvdXbD off=1276608 len=1842262 2022-04-22T18:01:24.142 INFO:tasks.workunit.client.0.smithi115.stdout:op 4370 completed, throughput=5MB/sec 2022-04-22T18:01:24.143 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KGck2PwQHqb7sTU off=37411 len=1292109 2022-04-22T18:01:24.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 4371 completed, throughput=5MB/sec 2022-04-22T18:01:24.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VoIM6GhRM_eG8mg off=4054656 len=1750070 2022-04-22T18:01:24.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 4372 completed, throughput=5MB/sec 2022-04-22T18:01:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:24 smithi121 bash[36902]: cluster 2022-04-22T18:01:22.720014+0000 mgr.y (mgr.34107) 1254 : cluster [DBG] pgmap v1840: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:01:24.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:24 smithi121 bash[36902]: audit 2022-04-22T18:01:23.296964+0000 mgr.y (mgr.34107) 1255 : audit [DBG] from='client.45655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:24.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:24 smithi121 bash[36902]: audit 2022-04-22T18:01:24.067728+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.115:0/1809395722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[45495]: cluster 2022-04-22T18:01:22.720014+0000 mgr.y (mgr.34107) 1254 : cluster [DBG] pgmap v1840: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[45495]: audit 2022-04-22T18:01:23.296964+0000 mgr.y (mgr.34107) 1255 : audit [DBG] from='client.45655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[45495]: audit 2022-04-22T18:01:24.067728+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.115:0/1809395722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[46934]: cluster 2022-04-22T18:01:22.720014+0000 mgr.y (mgr.34107) 1254 : cluster [DBG] pgmap v1840: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[46934]: audit 2022-04-22T18:01:23.296964+0000 mgr.y (mgr.34107) 1255 : audit [DBG] from='client.45655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:24.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:24 smithi115 bash[46934]: audit 2022-04-22T18:01:24.067728+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.115:0/1809395722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:25.152 INFO:tasks.workunit.client.0.smithi115.stdout: 860: throughput=5MB/sec pending data=0 2022-04-22T18:01:25.152 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3uXMaOqOrswKcnn off=1397826 len=1003255 2022-04-22T18:01:25.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 4373 completed, throughput=5MB/sec 2022-04-22T18:01:25.160 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZXF3G51CGkGj9r1 off=10382645 len=1851736 2022-04-22T18:01:25.163 INFO:tasks.workunit.client.0.smithi115.stdout:op 4374 completed, throughput=5MB/sec 2022-04-22T18:01:25.163 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WL-l_yoyRMJay85 off=2465855 len=343438 2022-04-22T18:01:25.165 INFO:tasks.workunit.client.0.smithi115.stdout:op 4375 completed, throughput=5MB/sec 2022-04-22T18:01:25.165 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3shOtHftCorSkhl off=5492922 len=2077255 2022-04-22T18:01:25.176 INFO:tasks.workunit.client.0.smithi115.stdout:op 4376 completed, throughput=5MB/sec 2022-04-22T18:01:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:25 smithi121 bash[36902]: audit 2022-04-22T18:01:23.641431+0000 mgr.y (mgr.34107) 1256 : audit [DBG] from='client.45661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:25 smithi115 bash[45495]: audit 2022-04-22T18:01:23.641431+0000 mgr.y (mgr.34107) 1256 : audit [DBG] from='client.45661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:25 smithi115 bash[46934]: audit 2022-04-22T18:01:23.641431+0000 mgr.y (mgr.34107) 1256 : audit [DBG] from='client.45661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:26.176 INFO:tasks.workunit.client.0.smithi115.stdout: 861: throughput=5MB/sec pending data=0 2022-04-22T18:01:26.176 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BJYgAQ9GxLOCd7V off=193188 len=1943454 2022-04-22T18:01:26.188 INFO:tasks.workunit.client.0.smithi115.stdout:op 4377 completed, throughput=5MB/sec 2022-04-22T18:01:26.188 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WCgIrbpBQd2tQj1 off=14488831 len=1063241 2022-04-22T18:01:26.191 INFO:tasks.workunit.client.0.smithi115.stdout:op 4378 completed, throughput=5MB/sec 2022-04-22T18:01:26.191 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Qn3AtiqUET6upmA off=1375916 len=1206374 2022-04-22T18:01:26.198 INFO:tasks.workunit.client.0.smithi115.stdout:op 4379 completed, throughput=5MB/sec 2022-04-22T18:01:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:26 smithi121 bash[36902]: cluster 2022-04-22T18:01:24.721062+0000 mgr.y (mgr.34107) 1257 : cluster [DBG] pgmap v1841: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:26 smithi121 bash[36902]: audit 2022-04-22T18:01:25.306823+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:26 smithi121 bash[36902]: audit 2022-04-22T18:01:25.688434+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[45495]: cluster 2022-04-22T18:01:24.721062+0000 mgr.y (mgr.34107) 1257 : cluster [DBG] pgmap v1841: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[45495]: audit 2022-04-22T18:01:25.306823+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[45495]: audit 2022-04-22T18:01:25.688434+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[46934]: cluster 2022-04-22T18:01:24.721062+0000 mgr.y (mgr.34107) 1257 : cluster [DBG] pgmap v1841: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[46934]: audit 2022-04-22T18:01:25.306823+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:26 smithi115 bash[46934]: audit 2022-04-22T18:01:25.688434+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:27.198 INFO:tasks.workunit.client.0.smithi115.stdout: 862: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MVEwvwsuSg9_rEn off=606166 len=1149749 2022-04-22T18:01:27.206 INFO:tasks.workunit.client.0.smithi115.stdout:op 4380 completed, throughput=5MB/sec 2022-04-22T18:01:27.207 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pjoviRigYjm4TVR off=10669237 len=79327 2022-04-22T18:01:27.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 4381 completed, throughput=5MB/sec 2022-04-22T18:01:27.210 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Kn1-Iu6iYv2VjO0 off=14846950 len=477421 2022-04-22T18:01:27.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 4382 completed, throughput=5MB/sec 2022-04-22T18:01:27.212 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x0Aa_c_UYp6nLz2 off=214777 len=1900153 2022-04-22T18:01:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 4383 completed, throughput=5MB/sec 2022-04-22T18:01:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Wh4RYqthJ8wKaSL off=6226023 len=1885210 2022-04-22T18:01:27.224 INFO:tasks.workunit.client.0.smithi115.stdout:op 4384 completed, throughput=5MB/sec 2022-04-22T18:01:28.225 INFO:tasks.workunit.client.0.smithi115.stdout: 863: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:28.225 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RV8yaAtAByiplCo off=11823638 len=506202 2022-04-22T18:01:28.226 INFO:tasks.workunit.client.0.smithi115.stdout:op 4385 completed, throughput=5MB/sec 2022-04-22T18:01:28.227 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GAtGXrEbRzCGBxq off=7063751 len=404780 2022-04-22T18:01:28.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 4386 completed, throughput=5MB/sec 2022-04-22T18:01:28.228 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K9jgUYa5uK43I5s off=4274004 len=995289 2022-04-22T18:01:28.235 INFO:tasks.workunit.client.0.smithi115.stdout:op 4387 completed, throughput=5MB/sec 2022-04-22T18:01:28.235 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mdKR0vbUmd98MhN off=2327928 len=400236 2022-04-22T18:01:28.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 4388 completed, throughput=5MB/sec 2022-04-22T18:01:28.236 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oEtYU3F4hkl154V off=3609241 len=656083 2022-04-22T18:01:28.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 4389 completed, throughput=5MB/sec 2022-04-22T18:01:28.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IJTP0X7AkJtWNYv off=21081320 len=913272 2022-04-22T18:01:28.243 INFO:tasks.workunit.client.0.smithi115.stdout:op 4390 completed, throughput=5MB/sec 2022-04-22T18:01:28.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fn7ZQKYtSN6rFqH off=8065369 len=84850 2022-04-22T18:01:28.244 INFO:tasks.workunit.client.0.smithi115.stdout:op 4391 completed, throughput=5MB/sec 2022-04-22T18:01:28.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2p00sEUHZDT7UbP off=1029079 len=146139 2022-04-22T18:01:28.245 INFO:tasks.workunit.client.0.smithi115.stdout:op 4392 completed, throughput=5MB/sec 2022-04-22T18:01:28.245 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CnmHWSRjum_ZHy_ off=3543980 len=72441 2022-04-22T18:01:28.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 4393 completed, throughput=5MB/sec 2022-04-22T18:01:28.247 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qJoeMLCed58KPD6 off=11284311 len=544499 2022-04-22T18:01:28.248 INFO:tasks.workunit.client.0.smithi115.stdout:op 4394 completed, throughput=5MB/sec 2022-04-22T18:01:28.248 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-emHsuInzh9UUt9D off=3719957 len=47592 2022-04-22T18:01:28.251 INFO:tasks.workunit.client.0.smithi115.stdout:op 4395 completed, throughput=5MB/sec 2022-04-22T18:01:28.251 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Mu0A0EQ6RMGJ5Wa off=6895228 len=1509679 2022-04-22T18:01:28.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 4396 completed, throughput=5MB/sec 2022-04-22T18:01:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:28 smithi121 bash[36902]: cluster 2022-04-22T18:01:26.722157+0000 mgr.y (mgr.34107) 1258 : cluster [DBG] pgmap v1842: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:28.565 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:28 smithi121 bash[36902]: audit 2022-04-22T18:01:27.693513+0000 mon.a (mon.0) 2776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:28.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:28 smithi115 bash[45495]: cluster 2022-04-22T18:01:26.722157+0000 mgr.y (mgr.34107) 1258 : cluster [DBG] pgmap v1842: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:28.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:28 smithi115 bash[45495]: audit 2022-04-22T18:01:27.693513+0000 mon.a (mon.0) 2776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:28.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:28 smithi115 bash[46934]: cluster 2022-04-22T18:01:26.722157+0000 mgr.y (mgr.34107) 1258 : cluster [DBG] pgmap v1842: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:28.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:28 smithi115 bash[46934]: audit 2022-04-22T18:01:27.693513+0000 mon.a (mon.0) 2776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:29.259 INFO:tasks.workunit.client.0.smithi115.stdout: 864: throughput=5MB/sec pending data=0 2022-04-22T18:01:29.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mdlV1Lsp8VPCVYX off=7883114 len=146248 2022-04-22T18:01:29.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 4397 completed, throughput=5MB/sec 2022-04-22T18:01:29.290 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-epl4JVjJ5cczwZj off=7483118 len=166526 2022-04-22T18:01:29.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 4398 completed, throughput=5MB/sec 2022-04-22T18:01:29.290 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IwSLZwXaqChCjjV off=12193273 len=1499857 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 4399 completed, throughput=5MB/sec 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cl1b0M22xCrIDmw off=2308414 len=238271 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 4400 completed, throughput=5MB/sec 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c8EX3c-6E05WkiT off=14306594 len=1755232 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 4401 completed, throughput=5MB/sec 2022-04-22T18:01:29.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s-g0Q3dwGnTY67t off=11769007 len=945295 2022-04-22T18:01:29.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 4402 completed, throughput=5MB/sec 2022-04-22T18:01:30.283 INFO:tasks.workunit.client.0.smithi115.stdout: 865: throughput=5MB/sec pending data=0 2022-04-22T18:01:30.283 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IEdGaYjKGGSRPIF off=270919 len=492667 2022-04-22T18:01:30.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 4403 completed, throughput=5MB/sec 2022-04-22T18:01:30.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hudvZh-Wd5_QznA off=6765218 len=1048595 2022-04-22T18:01:30.288 INFO:tasks.workunit.client.0.smithi115.stdout:op 4404 completed, throughput=5MB/sec 2022-04-22T18:01:30.288 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VtRgXzzz7xAw8-p off=9616979 len=1890229 2022-04-22T18:01:30.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 4405 completed, throughput=5MB/sec 2022-04-22T18:01:30.292 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Nklrs-nTDETLRat off=1701926 len=869743 2022-04-22T18:01:30.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 4406 completed, throughput=5MB/sec 2022-04-22T18:01:30.299 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NS0_MW0gSPq7Ldf off=11206883 len=237661 2022-04-22T18:01:30.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 4407 completed, throughput=5MB/sec 2022-04-22T18:01:30.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fqeDIQDhaqMvm0O off=1346156 len=480811 2022-04-22T18:01:30.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 4408 completed, throughput=5MB/sec 2022-04-22T18:01:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:30 smithi121 bash[36902]: cluster 2022-04-22T18:01:28.723377+0000 mgr.y (mgr.34107) 1259 : cluster [DBG] pgmap v1843: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:30 smithi115 bash[45495]: cluster 2022-04-22T18:01:28.723377+0000 mgr.y (mgr.34107) 1259 : cluster [DBG] pgmap v1843: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:30.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:30 smithi115 bash[46934]: cluster 2022-04-22T18:01:28.723377+0000 mgr.y (mgr.34107) 1259 : cluster [DBG] pgmap v1843: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:01:31.305 INFO:tasks.workunit.client.0.smithi115.stdout: 866: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:31.306 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-20_by16VudK6ydq off=11596955 len=1946023 2022-04-22T18:01:31.316 INFO:tasks.workunit.client.0.smithi115.stdout:op 4409 completed, throughput=5MB/sec 2022-04-22T18:01:31.316 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ApCGG5zAELVXsmu off=2736056 len=1743883 2022-04-22T18:01:31.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 4410 completed, throughput=5MB/sec 2022-04-22T18:01:31.327 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RzROhuP07zy10Zu off=1549335 len=748762 2022-04-22T18:01:31.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 4411 completed, throughput=5MB/sec 2022-04-22T18:01:31.329 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KauVNQYYDOeod4H off=20653995 len=1504086 2022-04-22T18:01:31.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 4412 completed, throughput=5MB/sec 2022-04-22T18:01:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:31 smithi121 bash[36902]: audit 2022-04-22T18:01:30.309726+0000 mon.a (mon.0) 2777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:31 smithi121 bash[36902]: audit 2022-04-22T18:01:30.692090+0000 mon.a (mon.0) 2778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:31 smithi115 bash[45495]: audit 2022-04-22T18:01:30.309726+0000 mon.a (mon.0) 2777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:31 smithi115 bash[45495]: audit 2022-04-22T18:01:30.692090+0000 mon.a (mon.0) 2778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:31.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:31 smithi115 bash[46934]: audit 2022-04-22T18:01:30.309726+0000 mon.a (mon.0) 2777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:31 smithi115 bash[46934]: audit 2022-04-22T18:01:30.692090+0000 mon.a (mon.0) 2778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:31.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:32.336 INFO:tasks.workunit.client.0.smithi115.stdout: 867: throughput=5MB/sec pending data=0 2022-04-22T18:01:32.337 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Wh4RYqthJ8wKaSL off=10261151 len=215195 2022-04-22T18:01:32.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 4413 completed, throughput=5MB/sec 2022-04-22T18:01:32.339 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wAmklPAQyQlwHyy off=7712604 len=1484231 2022-04-22T18:01:32.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 4414 completed, throughput=5MB/sec 2022-04-22T18:01:32.347 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tc4uggHepRI_yAS off=18289095 len=971016 2022-04-22T18:01:32.352 INFO:tasks.workunit.client.0.smithi115.stdout:op 4415 completed, throughput=5MB/sec 2022-04-22T18:01:32.353 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jMhnDCrP-H4OF5t off=7730052 len=1219681 2022-04-22T18:01:32.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 4416 completed, throughput=5MB/sec 2022-04-22T18:01:32.356 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uez_sxZnEDfPH4M off=257366 len=864376 2022-04-22T18:01:32.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 4417 completed, throughput=5MB/sec 2022-04-22T18:01:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:32 smithi121 bash[36902]: cluster 2022-04-22T18:01:30.724435+0000 mgr.y (mgr.34107) 1260 : cluster [DBG] pgmap v1844: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:32 smithi115 bash[45495]: cluster 2022-04-22T18:01:30.724435+0000 mgr.y (mgr.34107) 1260 : cluster [DBG] pgmap v1844: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:32.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:32 smithi115 bash[46934]: cluster 2022-04-22T18:01:30.724435+0000 mgr.y (mgr.34107) 1260 : cluster [DBG] pgmap v1844: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:33.363 INFO:tasks.workunit.client.0.smithi115.stdout: 868: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:33.363 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WCgIrbpBQd2tQj1 off=6039839 len=1116586 2022-04-22T18:01:33.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 4418 completed, throughput=5MB/sec 2022-04-22T18:01:33.371 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DsQ-4Iw8TDo6NM8 off=2200429 len=1678853 2022-04-22T18:01:33.380 INFO:tasks.workunit.client.0.smithi115.stdout:op 4419 completed, throughput=5MB/sec 2022-04-22T18:01:33.380 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pmu5n2IBlD_W-Lp off=974877 len=1094502 2022-04-22T18:01:33.386 INFO:tasks.workunit.client.0.smithi115.stdout:op 4420 completed, throughput=5MB/sec 2022-04-22T18:01:33.387 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-POEakxRd8n-DE8y off=5406448 len=1551960 2022-04-22T18:01:33.388 INFO:tasks.workunit.client.0.smithi115.stdout:op 4421 completed, throughput=5MB/sec 2022-04-22T18:01:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:33 smithi121 bash[36902]: audit 2022-04-22T18:01:32.696606+0000 mon.a (mon.0) 2779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:33.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:33 smithi115 bash[45495]: audit 2022-04-22T18:01:32.696606+0000 mon.a (mon.0) 2779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:33 smithi115 bash[46934]: audit 2022-04-22T18:01:32.696606+0000 mon.a (mon.0) 2779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:33.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:34.389 INFO:tasks.workunit.client.0.smithi115.stdout: 869: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:34.389 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6WjR-6pQqUO_Pdx off=7428758 len=1453114 2022-04-22T18:01:34.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 4422 completed, throughput=5MB/sec 2022-04-22T18:01:34.392 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JPCm6tlWGB391su off=21967 len=629285 2022-04-22T18:01:34.394 INFO:tasks.workunit.client.0.smithi115.stdout:op 4423 completed, throughput=5MB/sec 2022-04-22T18:01:34.394 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y41Q6PvoCS-MnqP off=7518633 len=1382508 2022-04-22T18:01:34.403 INFO:tasks.workunit.client.0.smithi115.stdout:op 4424 completed, throughput=5MB/sec 2022-04-22T18:01:34.403 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PZPvOP2W2eys99S off=754222 len=1580126 2022-04-22T18:01:34.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 4425 completed, throughput=5MB/sec 2022-04-22T18:01:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:34 smithi121 bash[36902]: cluster 2022-04-22T18:01:32.725644+0000 mgr.y (mgr.34107) 1261 : cluster [DBG] pgmap v1845: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:34 smithi115 bash[45495]: cluster 2022-04-22T18:01:32.725644+0000 mgr.y (mgr.34107) 1261 : cluster [DBG] pgmap v1845: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:34.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:34 smithi115 bash[46934]: cluster 2022-04-22T18:01:32.725644+0000 mgr.y (mgr.34107) 1261 : cluster [DBG] pgmap v1845: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:35.412 INFO:tasks.workunit.client.0.smithi115.stdout: 870: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:35.412 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hl7ZuZM29zwNWQ6 off=6826653 len=1591495 2022-04-22T18:01:35.415 INFO:tasks.workunit.client.0.smithi115.stdout:op 4426 completed, throughput=5MB/sec 2022-04-22T18:01:35.416 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3b0_OsVKJcYHZkm off=11857105 len=885421 2022-04-22T18:01:35.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 4427 completed, throughput=5MB/sec 2022-04-22T18:01:35.423 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rN90qWCPkc4nQ5Q off=4929829 len=351683 2022-04-22T18:01:35.424 INFO:tasks.workunit.client.0.smithi115.stdout:op 4428 completed, throughput=5MB/sec 2022-04-22T18:01:35.424 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DGUT6Vqz2kkIjUt off=9788626 len=556502 2022-04-22T18:01:35.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 4429 completed, throughput=5MB/sec 2022-04-22T18:01:35.430 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tQqinVX_ky_Wztr off=10736518 len=1619611 2022-04-22T18:01:35.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 4430 completed, throughput=5MB/sec 2022-04-22T18:01:35.437 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9wg_ObtUc5s0FZK off=13731857 len=1372910 2022-04-22T18:01:35.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 4431 completed, throughput=5MB/sec 2022-04-22T18:01:36.439 INFO:tasks.workunit.client.0.smithi115.stdout: 871: throughput=5MB/sec pending data=0 2022-04-22T18:01:36.440 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-92d0TWFzoibM8IP off=2125765 len=976433 2022-04-22T18:01:36.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 4432 completed, throughput=5MB/sec 2022-04-22T18:01:36.447 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E8UzkeUdNtuI6aZ off=11572298 len=1924191 2022-04-22T18:01:36.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 4433 completed, throughput=5MB/sec 2022-04-22T18:01:36.459 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ApCGG5zAELVXsmu off=275576 len=1308460 2022-04-22T18:01:36.461 INFO:tasks.workunit.client.0.smithi115.stdout:op 4434 completed, throughput=5MB/sec 2022-04-22T18:01:36.461 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b71_u-7ZWURwFMn off=16587621 len=1205854 2022-04-22T18:01:36.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 4435 completed, throughput=5MB/sec 2022-04-22T18:01:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:36 smithi121 bash[36902]: cluster 2022-04-22T18:01:34.726812+0000 mgr.y (mgr.34107) 1262 : cluster [DBG] pgmap v1846: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:01:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:36 smithi121 bash[36902]: audit 2022-04-22T18:01:35.312414+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:36 smithi121 bash[36902]: audit 2022-04-22T18:01:35.695806+0000 mon.a (mon.0) 2781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[45495]: cluster 2022-04-22T18:01:34.726812+0000 mgr.y (mgr.34107) 1262 : cluster [DBG] pgmap v1846: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:01:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[45495]: audit 2022-04-22T18:01:35.312414+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[45495]: audit 2022-04-22T18:01:35.695806+0000 mon.a (mon.0) 2781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[46934]: cluster 2022-04-22T18:01:34.726812+0000 mgr.y (mgr.34107) 1262 : cluster [DBG] pgmap v1846: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:01:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[46934]: audit 2022-04-22T18:01:35.312414+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:36 smithi115 bash[46934]: audit 2022-04-22T18:01:35.695806+0000 mon.a (mon.0) 2781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:37.464 INFO:tasks.workunit.client.0.smithi115.stdout: 872: throughput=5MB/sec pending data=0 2022-04-22T18:01:37.464 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FHayTO3SRmnqzwC off=5419203 len=1390105 2022-04-22T18:01:37.474 INFO:tasks.workunit.client.0.smithi115.stdout:op 4436 completed, throughput=5MB/sec 2022-04-22T18:01:37.474 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--WeXCHgdjDXAdim off=5969686 len=886982 2022-04-22T18:01:37.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 4437 completed, throughput=5MB/sec 2022-04-22T18:01:37.476 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SI4o_2nHCPsz_m8 off=9356655 len=1647787 2022-04-22T18:01:37.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 4438 completed, throughput=5MB/sec 2022-04-22T18:01:37.480 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BVEiKSC3lTVOXcu off=2650223 len=2050176 2022-04-22T18:01:37.491 INFO:tasks.workunit.client.0.smithi115.stdout:op 4439 completed, throughput=5MB/sec 2022-04-22T18:01:38.491 INFO:tasks.workunit.client.0.smithi115.stdout: 873: throughput=5MB/sec pending data=0 2022-04-22T18:01:38.492 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MOi3sR7j_gvdXbD off=15848770 len=1300550 2022-04-22T18:01:38.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 4440 completed, throughput=5MB/sec 2022-04-22T18:01:38.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YSEtk_CqKV3xz8l off=1812769 len=765940 2022-04-22T18:01:38.499 INFO:tasks.workunit.client.0.smithi115.stdout:op 4441 completed, throughput=5MB/sec 2022-04-22T18:01:38.499 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CsSJpAzEJQnpgN9 off=2189920 len=1923006 2022-04-22T18:01:38.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 4442 completed, throughput=5MB/sec 2022-04-22T18:01:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:38 smithi121 bash[36902]: cluster 2022-04-22T18:01:36.728060+0000 mgr.y (mgr.34107) 1263 : cluster [DBG] pgmap v1847: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:38 smithi121 bash[36902]: audit 2022-04-22T18:01:37.700053+0000 mon.a (mon.0) 2782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:38 smithi115 bash[45495]: cluster 2022-04-22T18:01:36.728060+0000 mgr.y (mgr.34107) 1263 : cluster [DBG] pgmap v1847: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:38 smithi115 bash[45495]: audit 2022-04-22T18:01:37.700053+0000 mon.a (mon.0) 2782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:38 smithi115 bash[46934]: cluster 2022-04-22T18:01:36.728060+0000 mgr.y (mgr.34107) 1263 : cluster [DBG] pgmap v1847: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:38 smithi115 bash[46934]: audit 2022-04-22T18:01:37.700053+0000 mon.a (mon.0) 2782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:39.503 INFO:tasks.workunit.client.0.smithi115.stdout: 874: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:39.596 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TDcUsRx-VKfOJVp off=3079331 len=1115626 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4443 completed, throughput=5MB/sec 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zK7P4higG-P0wb3 off=3536861 len=1734980 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4444 completed, throughput=5MB/sec 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mZiCVeKoKEWd8u_ off=3232670 len=1266182 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4445 completed, throughput=5MB/sec 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ArgpYbwvkl-vMwm off=8133057 len=2034637 2022-04-22T18:01:39.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4446 completed, throughput=5MB/sec 2022-04-22T18:01:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:40 smithi121 bash[36902]: cluster 2022-04-22T18:01:38.729271+0000 mgr.y (mgr.34107) 1264 : cluster [DBG] pgmap v1848: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:40.527 INFO:tasks.workunit.client.0.smithi115.stdout: 875: throughput=5MB/sec pending data=0 2022-04-22T18:01:40.528 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NM3m6bsCNrT8WxR off=189119 len=801354 2022-04-22T18:01:40.530 INFO:tasks.workunit.client.0.smithi115.stdout:op 4447 completed, throughput=5MB/sec 2022-04-22T18:01:40.530 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BcitRqMYZ378LzW off=7556183 len=1760721 2022-04-22T18:01:40.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 4448 completed, throughput=5MB/sec 2022-04-22T18:01:40.536 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SILlhfcS0jM_NKG off=628786 len=443707 2022-04-22T18:01:40.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 4449 completed, throughput=5MB/sec 2022-04-22T18:01:40.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZRy_Rm7--2XyVxN off=4056819 len=273247 2022-04-22T18:01:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:40 smithi115 bash[45495]: cluster 2022-04-22T18:01:38.729271+0000 mgr.y (mgr.34107) 1264 : cluster [DBG] pgmap v1848: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:40 smithi115 bash[46934]: cluster 2022-04-22T18:01:38.729271+0000 mgr.y (mgr.34107) 1264 : cluster [DBG] pgmap v1848: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:01:40.545 INFO:tasks.workunit.client.0.smithi115.stdout:op 4450 completed, throughput=5MB/sec 2022-04-22T18:01:40.546 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-V3NaMAEVY6cTL5S off=1231734 len=386745 2022-04-22T18:01:40.551 INFO:tasks.workunit.client.0.smithi115.stdout:op 4451 completed, throughput=5MB/sec 2022-04-22T18:01:40.551 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eN4VAxW1DjD3RLj off=8127504 len=974370 2022-04-22T18:01:40.556 INFO:tasks.workunit.client.0.smithi115.stdout:op 4452 completed, throughput=5MB/sec 2022-04-22T18:01:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:41 smithi121 bash[36902]: audit 2022-04-22T18:01:40.315298+0000 mon.a (mon.0) 2783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:41 smithi121 bash[36902]: audit 2022-04-22T18:01:40.699700+0000 mon.a (mon.0) 2784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:41 smithi115 bash[45495]: audit 2022-04-22T18:01:40.315298+0000 mon.a (mon.0) 2783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:41 smithi115 bash[45495]: audit 2022-04-22T18:01:40.699700+0000 mon.a (mon.0) 2784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:41 smithi115 bash[46934]: audit 2022-04-22T18:01:40.315298+0000 mon.a (mon.0) 2783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:41 smithi115 bash[46934]: audit 2022-04-22T18:01:40.699700+0000 mon.a (mon.0) 2784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:41.557 INFO:tasks.workunit.client.0.smithi115.stdout: 876: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:41.557 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FAiPCG96E-pTuK_ off=538746 len=17125 2022-04-22T18:01:41.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 4453 completed, throughput=4.99MB/sec 2022-04-22T18:01:41.558 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2KVAizfHo3XRMJH off=4297343 len=1273248 2022-04-22T18:01:41.567 INFO:tasks.workunit.client.0.smithi115.stdout:op 4454 completed, throughput=5MB/sec 2022-04-22T18:01:41.567 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UjqUZ23bXG22erq off=1504181 len=1943003 2022-04-22T18:01:41.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 4455 completed, throughput=5MB/sec 2022-04-22T18:01:41.570 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W8JXxYZTDHj7PLq off=3125590 len=1777395 2022-04-22T18:01:41.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 4456 completed, throughput=5MB/sec 2022-04-22T18:01:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:41.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:42 smithi121 bash[36902]: cluster 2022-04-22T18:01:40.730421+0000 mgr.y (mgr.34107) 1265 : cluster [DBG] pgmap v1849: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:01:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:42 smithi115 bash[45495]: cluster 2022-04-22T18:01:40.730421+0000 mgr.y (mgr.34107) 1265 : cluster [DBG] pgmap v1849: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:01:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:42 smithi115 bash[46934]: cluster 2022-04-22T18:01:40.730421+0000 mgr.y (mgr.34107) 1265 : cluster [DBG] pgmap v1849: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:01:42.573 INFO:tasks.workunit.client.0.smithi115.stdout: 877: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:42.574 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CI5pCAlHnoU3PgB off=119733 len=1749047 2022-04-22T18:01:42.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 4457 completed, throughput=5MB/sec 2022-04-22T18:01:42.578 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-63Flar-Ph7ZUd3f off=10924474 len=1118119 2022-04-22T18:01:42.581 INFO:tasks.workunit.client.0.smithi115.stdout:op 4458 completed, throughput=5MB/sec 2022-04-22T18:01:42.581 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-V_Yv86fyHyTl9lp off=5292971 len=918192 2022-04-22T18:01:42.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 4459 completed, throughput=5MB/sec 2022-04-22T18:01:42.590 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D7-c2twKnz_62on off=3124290 len=1311940 2022-04-22T18:01:42.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4460 completed, throughput=5MB/sec 2022-04-22T18:01:42.598 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UzLBwOX0d02ETmw off=1033343 len=6522 2022-04-22T18:01:42.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4461 completed, throughput=5MB/sec 2022-04-22T18:01:42.598 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-t_ZZ3Rqdxf_ZBBA off=2279293 len=484056 2022-04-22T18:01:42.599 INFO:tasks.workunit.client.0.smithi115.stdout:op 4462 completed, throughput=5MB/sec 2022-04-22T18:01:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:43 smithi121 bash[36902]: audit 2022-04-22T18:01:42.703373+0000 mon.a (mon.0) 2785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:43 smithi115 bash[45495]: audit 2022-04-22T18:01:42.703373+0000 mon.a (mon.0) 2785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:43 smithi115 bash[46934]: audit 2022-04-22T18:01:42.703373+0000 mon.a (mon.0) 2785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:43.599 INFO:tasks.workunit.client.0.smithi115.stdout: 878: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:43.600 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OcS9HKhQxZpvvtC off=4672540 len=1059050 2022-04-22T18:01:43.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 4463 completed, throughput=5MB/sec 2022-04-22T18:01:43.603 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u9lvP2aGgpbwIc1 off=2351153 len=1546670 2022-04-22T18:01:43.612 INFO:tasks.workunit.client.0.smithi115.stdout:op 4464 completed, throughput=5MB/sec 2022-04-22T18:01:43.613 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uGikJREh2FVSX1o off=4679294 len=763594 2022-04-22T18:01:43.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 4465 completed, throughput=5MB/sec 2022-04-22T18:01:43.618 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F2GIVE14KDxcNZ7 off=2568088 len=1949084 2022-04-22T18:01:43.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 4466 completed, throughput=5MB/sec 2022-04-22T18:01:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:43.610Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:44 smithi121 bash[36902]: cluster 2022-04-22T18:01:42.731570+0000 mgr.y (mgr.34107) 1266 : cluster [DBG] pgmap v1850: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:44 smithi115 bash[45495]: cluster 2022-04-22T18:01:42.731570+0000 mgr.y (mgr.34107) 1266 : cluster [DBG] pgmap v1850: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:44 smithi115 bash[46934]: cluster 2022-04-22T18:01:42.731570+0000 mgr.y (mgr.34107) 1266 : cluster [DBG] pgmap v1850: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:44.621 INFO:tasks.workunit.client.0.smithi115.stdout: 879: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:44.703 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5Yfl3uBwi2dvq89 off=4120364 len=614836 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 4467 completed, throughput=5MB/sec 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KPDSWMUbOMxPQtV off=3139046 len=414088 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 4468 completed, throughput=5MB/sec 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n0Mb5l8hp7UcnuK off=795366 len=1284394 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 4469 completed, throughput=5MB/sec 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SBAqJ2uc7GSTLmK off=4083750 len=1822991 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 4470 completed, throughput=5MB/sec 2022-04-22T18:01:44.705 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kfgGPLXm7pcWSOs off=1306935 len=1511368 2022-04-22T18:01:44.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 4471 completed, throughput=5MB/sec 2022-04-22T18:01:45.645 INFO:tasks.workunit.client.0.smithi115.stdout: 880: throughput=5MB/sec pending data=0 2022-04-22T18:01:45.646 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4bzSAQDnoWYLQme off=3909031 len=772431 2022-04-22T18:01:45.647 INFO:tasks.workunit.client.0.smithi115.stdout:op 4472 completed, throughput=5MB/sec 2022-04-22T18:01:45.647 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NFM9c5uru2GNqsp off=11267435 len=1394006 2022-04-22T18:01:45.656 INFO:tasks.workunit.client.0.smithi115.stdout:op 4473 completed, throughput=5MB/sec 2022-04-22T18:01:45.656 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ohjd_tuWAV7qOad off=8599513 len=285817 2022-04-22T18:01:45.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 4474 completed, throughput=5MB/sec 2022-04-22T18:01:45.662 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VSHPBMLNCOlrKC_ off=10125333 len=879184 2022-04-22T18:01:45.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 4475 completed, throughput=5MB/sec 2022-04-22T18:01:45.663 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8ht3XsadMZbB_KF off=3517579 len=1232762 2022-04-22T18:01:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 4476 completed, throughput=5MB/sec 2022-04-22T18:01:45.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NbmM3Lnj73buC2H off=271706 len=1539177 2022-04-22T18:01:45.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 4477 completed, throughput=5MB/sec 2022-04-22T18:01:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:46 smithi121 bash[36902]: cluster 2022-04-22T18:01:44.732709+0000 mgr.y (mgr.34107) 1267 : cluster [DBG] pgmap v1851: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:46 smithi121 bash[36902]: audit 2022-04-22T18:01:45.318341+0000 mon.a (mon.0) 2786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:46 smithi121 bash[36902]: audit 2022-04-22T18:01:45.704973+0000 mon.a (mon.0) 2787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[45495]: cluster 2022-04-22T18:01:44.732709+0000 mgr.y (mgr.34107) 1267 : cluster [DBG] pgmap v1851: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[45495]: audit 2022-04-22T18:01:45.318341+0000 mon.a (mon.0) 2786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[45495]: audit 2022-04-22T18:01:45.704973+0000 mon.a (mon.0) 2787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[46934]: cluster 2022-04-22T18:01:44.732709+0000 mgr.y (mgr.34107) 1267 : cluster [DBG] pgmap v1851: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:01:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[46934]: audit 2022-04-22T18:01:45.318341+0000 mon.a (mon.0) 2786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:46 smithi115 bash[46934]: audit 2022-04-22T18:01:45.704973+0000 mon.a (mon.0) 2787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:46.670 INFO:tasks.workunit.client.0.smithi115.stdout: 881: throughput=5MB/sec pending data=0 2022-04-22T18:01:46.670 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-toUsrM-Q3mLyWxa off=2268919 len=520795 2022-04-22T18:01:46.675 INFO:tasks.workunit.client.0.smithi115.stdout:op 4478 completed, throughput=5MB/sec 2022-04-22T18:01:46.675 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YYJOTwOghm1tIl9 off=7029406 len=839828 2022-04-22T18:01:46.680 INFO:tasks.workunit.client.0.smithi115.stdout:op 4479 completed, throughput=5MB/sec 2022-04-22T18:01:46.681 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1yGjXxzSAguYeAC off=10799323 len=690659 2022-04-22T18:01:46.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 4480 completed, throughput=5MB/sec 2022-04-22T18:01:46.683 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fo9fKXYhJBKMe_F off=2786517 len=528745 2022-04-22T18:01:46.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 4481 completed, throughput=5MB/sec 2022-04-22T18:01:46.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aeVON_7nIZBshoY off=7120278 len=1442157 2022-04-22T18:01:46.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 4482 completed, throughput=5MB/sec 2022-04-22T18:01:46.692 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SSGta_o-U28XVqR off=1094641 len=1280808 2022-04-22T18:01:46.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 4483 completed, throughput=5MB/sec 2022-04-22T18:01:47.699 INFO:tasks.workunit.client.0.smithi115.stdout: 882: throughput=5MB/sec pending data=0 2022-04-22T18:01:47.699 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zuMV5zvgxfgZtye off=631171 len=1365115 2022-04-22T18:01:47.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 4484 completed, throughput=5MB/sec 2022-04-22T18:01:47.708 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-piAtUSvKmbD1vr8 off=3822239 len=1405255 2022-04-22T18:01:47.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 4485 completed, throughput=5MB/sec 2022-04-22T18:01:47.711 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jqcpqmk6AHT8NB8 off=5568449 len=486831 2022-04-22T18:01:47.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 4486 completed, throughput=5MB/sec 2022-04-22T18:01:47.716 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hjex8kzMzJLA4cJ off=13966946 len=1508692 2022-04-22T18:01:47.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 4487 completed, throughput=5MB/sec 2022-04-22T18:01:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:48 smithi121 bash[36902]: cluster 2022-04-22T18:01:46.733983+0000 mgr.y (mgr.34107) 1268 : cluster [DBG] pgmap v1852: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:48 smithi121 bash[36902]: audit 2022-04-22T18:01:47.706410+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:48 smithi115 bash[45495]: cluster 2022-04-22T18:01:46.733983+0000 mgr.y (mgr.34107) 1268 : cluster [DBG] pgmap v1852: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:48 smithi115 bash[45495]: audit 2022-04-22T18:01:47.706410+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:48 smithi115 bash[46934]: cluster 2022-04-22T18:01:46.733983+0000 mgr.y (mgr.34107) 1268 : cluster [DBG] pgmap v1852: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:01:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:48 smithi115 bash[46934]: audit 2022-04-22T18:01:47.706410+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:48.720 INFO:tasks.workunit.client.0.smithi115.stdout: 883: throughput=5MB/sec pending data=0 2022-04-22T18:01:48.749 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JYbhgoGhfYf1FQO off=1448082 len=897487 2022-04-22T18:01:48.752 INFO:tasks.workunit.client.0.smithi115.stdout:op 4488 completed, throughput=5MB/sec 2022-04-22T18:01:48.752 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8ONw0nGV-kfe4My off=52054 len=1771256 2022-04-22T18:01:48.752 INFO:tasks.workunit.client.0.smithi115.stdout:op 4489 completed, throughput=5MB/sec 2022-04-22T18:01:48.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IsJThhOh33MuzGS off=12348016 len=1150605 2022-04-22T18:01:48.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 4490 completed, throughput=5MB/sec 2022-04-22T18:01:48.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0nn0ZnKFa4dE_pA off=3395924 len=640243 2022-04-22T18:01:48.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 4491 completed, throughput=5MB/sec 2022-04-22T18:01:48.754 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eoutIezlVg1Cgpk off=5286112 len=1662658 2022-04-22T18:01:48.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 4492 completed, throughput=5MB/sec 2022-04-22T18:01:49.748 INFO:tasks.workunit.client.0.smithi115.stdout: 884: throughput=5MB/sec pending data=0 2022-04-22T18:01:49.780 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l57NvAspuBLCc3C off=6722337 len=1603259 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 4493 completed, throughput=5MB/sec 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EGBeEiwLW6ymqbt off=2995098 len=936380 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 4494 completed, throughput=5MB/sec 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NOrdGbyU6pn_3ap off=12294332 len=927980 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 4495 completed, throughput=5MB/sec 2022-04-22T18:01:49.782 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ebaJH-ugcUpv1c8 off=1001155 len=304117 2022-04-22T18:01:49.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 4496 completed, throughput=5MB/sec 2022-04-22T18:01:49.783 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5ItaSrRKpGHGBAA off=811816 len=857899 2022-04-22T18:01:49.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 4497 completed, throughput=5MB/sec 2022-04-22T18:01:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:50 smithi121 bash[36902]: cluster 2022-04-22T18:01:48.734875+0000 mgr.y (mgr.34107) 1269 : cluster [DBG] pgmap v1853: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:01:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:50 smithi115 bash[45495]: cluster 2022-04-22T18:01:48.734875+0000 mgr.y (mgr.34107) 1269 : cluster [DBG] pgmap v1853: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:01:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:50 smithi115 bash[46934]: cluster 2022-04-22T18:01:48.734875+0000 mgr.y (mgr.34107) 1269 : cluster [DBG] pgmap v1853: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:01:50.769 INFO:tasks.workunit.client.0.smithi115.stdout: 885: throughput=5MB/sec pending data=0 2022-04-22T18:01:50.769 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZURFM3M1yMeCH5k off=6532873 len=517577 2022-04-22T18:01:50.771 INFO:tasks.workunit.client.0.smithi115.stdout:op 4498 completed, throughput=5MB/sec 2022-04-22T18:01:50.771 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Vff0q2xl96hwpEV off=4208722 len=973792 2022-04-22T18:01:50.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 4499 completed, throughput=5MB/sec 2022-04-22T18:01:50.777 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rN90qWCPkc4nQ5Q off=9365094 len=635583 2022-04-22T18:01:50.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 4500 completed, throughput=5MB/sec 2022-04-22T18:01:50.779 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-haAbNWfk1-VIlc0 off=20863900 len=570510 2022-04-22T18:01:50.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 4501 completed, throughput=5MB/sec 2022-04-22T18:01:50.781 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5-u2ohDiQHsSLH_ off=6085314 len=1725032 2022-04-22T18:01:50.783 INFO:tasks.workunit.client.0.smithi115.stdout:op 4502 completed, throughput=5MB/sec 2022-04-22T18:01:50.784 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kvPaNEhfZqVcMFk off=865532 len=1944222 2022-04-22T18:01:50.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 4503 completed, throughput=5MB/sec 2022-04-22T18:01:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[45495]: audit 2022-04-22T18:01:50.321437+0000 mon.a (mon.0) 2789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[45495]: audit 2022-04-22T18:01:50.707221+0000 mon.a (mon.0) 2790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[45495]: audit 2022-04-22T18:01:50.987694+0000 mon.a (mon.0) 2791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[45495]: audit 2022-04-22T18:01:50.990717+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[45495]: audit 2022-04-22T18:01:50.992577+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[46934]: audit 2022-04-22T18:01:50.321437+0000 mon.a (mon.0) 2789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[46934]: audit 2022-04-22T18:01:50.707221+0000 mon.a (mon.0) 2790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[46934]: audit 2022-04-22T18:01:50.987694+0000 mon.a (mon.0) 2791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:01:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[46934]: audit 2022-04-22T18:01:50.990717+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:01:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[46934]: audit 2022-04-22T18:01:50.992577+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:01:51.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:51 smithi121 bash[36902]: audit 2022-04-22T18:01:50.321437+0000 mon.a (mon.0) 2789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:51.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:51 smithi121 bash[36902]: audit 2022-04-22T18:01:50.707221+0000 mon.a (mon.0) 2790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:51.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:51 smithi121 bash[36902]: audit 2022-04-22T18:01:50.987694+0000 mon.a (mon.0) 2791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:01:51.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:51 smithi121 bash[36902]: audit 2022-04-22T18:01:50.990717+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:01:51.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:51 smithi121 bash[36902]: audit 2022-04-22T18:01:50.992577+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:01:51.787 INFO:tasks.workunit.client.0.smithi115.stdout: 886: throughput=5MB/sec pending data=0 2022-04-22T18:01:51.787 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IRRiL41Hfn2oNFq off=5868923 len=1187501 2022-04-22T18:01:51.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 4504 completed, throughput=5MB/sec 2022-04-22T18:01:51.792 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yC3y3u4cxeV0-g0 off=17503640 len=1545625 2022-04-22T18:01:51.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 4505 completed, throughput=5MB/sec 2022-04-22T18:01:51.796 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z9eT8J1vhOe7v21 off=456694 len=1028709 2022-04-22T18:01:51.799 INFO:tasks.workunit.client.0.smithi115.stdout:op 4506 completed, throughput=5MB/sec 2022-04-22T18:01:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:51.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:52 smithi121 bash[36902]: cluster 2022-04-22T18:01:50.735968+0000 mgr.y (mgr.34107) 1270 : cluster [DBG] pgmap v1854: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:52 smithi121 bash[36902]: audit 2022-04-22T18:01:51.347249+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:01:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:52 smithi115 bash[45495]: cluster 2022-04-22T18:01:50.735968+0000 mgr.y (mgr.34107) 1270 : cluster [DBG] pgmap v1854: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:52 smithi115 bash[45495]: audit 2022-04-22T18:01:51.347249+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:01:52.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:52 smithi115 bash[46934]: cluster 2022-04-22T18:01:50.735968+0000 mgr.y (mgr.34107) 1270 : cluster [DBG] pgmap v1854: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:52.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:52 smithi115 bash[46934]: audit 2022-04-22T18:01:51.347249+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:01:52.799 INFO:tasks.workunit.client.0.smithi115.stdout: 887: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:52.800 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-udeXMwsfr9dG6Ju off=567964 len=1321308 2022-04-22T18:01:52.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 4507 completed, throughput=5MB/sec 2022-04-22T18:01:52.809 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mDZCZeosXJuFnFv off=10567316 len=1899864 2022-04-22T18:01:52.818 INFO:tasks.workunit.client.0.smithi115.stdout:op 4508 completed, throughput=5MB/sec 2022-04-22T18:01:52.818 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2vIa8PaHmIG1zcj off=97962 len=93365 2022-04-22T18:01:52.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 4509 completed, throughput=5MB/sec 2022-04-22T18:01:52.820 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XtoPi8nMJxY9cZD off=6005911 len=1924767 2022-04-22T18:01:52.821 INFO:tasks.workunit.client.0.smithi115.stdout:op 4510 completed, throughput=5MB/sec 2022-04-22T18:01:52.821 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DdQTpuHYuiuizgC off=2274992 len=1701914 2022-04-22T18:01:52.825 INFO:tasks.workunit.client.0.smithi115.stdout:op 4511 completed, throughput=5MB/sec 2022-04-22T18:01:53.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:53 smithi115 bash[45495]: audit 2022-04-22T18:01:52.709387+0000 mon.a (mon.0) 2795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:53.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:53 smithi115 bash[46934]: audit 2022-04-22T18:01:52.709387+0000 mon.a (mon.0) 2795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:53 smithi121 bash[36902]: audit 2022-04-22T18:01:52.709387+0000 mon.a (mon.0) 2795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:53.825 INFO:tasks.workunit.client.0.smithi115.stdout: 888: throughput=5MB/sec pending data=0 2022-04-22T18:01:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7W3Q2R_V7TrdSLy off=11405697 len=1497078 2022-04-22T18:01:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 4512 completed, throughput=5MB/sec 2022-04-22T18:01:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kMdVKdOGpNWXhVR off=10834380 len=2090638 2022-04-22T18:01:53.840 INFO:tasks.workunit.client.0.smithi115.stdout:op 4513 completed, throughput=5MB/sec 2022-04-22T18:01:53.948 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cGUSpfv_MVoIXRy off=9141971 len=886162 2022-04-22T18:01:53.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 4514 completed, throughput=5MB/sec 2022-04-22T18:01:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:01:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:01:53.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:01:54.449 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:01:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:54 smithi121 bash[36902]: cluster 2022-04-22T18:01:52.737105+0000 mgr.y (mgr.34107) 1271 : cluster [DBG] pgmap v1855: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:54.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:54 smithi115 bash[45495]: cluster 2022-04-22T18:01:52.737105+0000 mgr.y (mgr.34107) 1271 : cluster [DBG] pgmap v1855: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:54.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:54 smithi115 bash[46934]: cluster 2022-04-22T18:01:52.737105+0000 mgr.y (mgr.34107) 1271 : cluster [DBG] pgmap v1855: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:01:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (24m) 4m ago 38m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:01:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (38m) 5m ago 38m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:01:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (26m) 4m ago 39m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:01:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (26m) 5m ago 39m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (34m) 5m ago 44m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (34m) 4m ago 46m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (33m) 4m ago 46m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (32m) 5m ago 44m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (33m) 4m ago 44m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (26m) 4m ago 39m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:01:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (26m) 5m ago 38m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (31m) 4m ago 43m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (31m) 4m ago 43m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (30m) 4m ago 42m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (30m) 4m ago 42m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (29m) 5m ago 41m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (28m) 5m ago 41m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (28m) 5m ago 40m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:01:54.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (27m) 5m ago 40m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:01:54.802 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (25m) 5m ago 39m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:01:54.846 INFO:tasks.workunit.client.0.smithi115.stdout: 889: throughput=5MB/sec pending data=0 2022-04-22T18:01:54.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-he4ucz1H0qBTOhI off=3994164 len=161361 2022-04-22T18:01:54.850 INFO:tasks.workunit.client.0.smithi115.stdout:op 4515 completed, throughput=5MB/sec 2022-04-22T18:01:54.850 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pRa98cZuCyZ6kIN off=16126094 len=1160429 2022-04-22T18:01:54.856 INFO:tasks.workunit.client.0.smithi115.stdout:op 4516 completed, throughput=5MB/sec 2022-04-22T18:01:54.856 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v1hg0yeHQ4Dy1Xa off=15411316 len=839876 2022-04-22T18:01:54.862 INFO:tasks.workunit.client.0.smithi115.stdout:op 4517 completed, throughput=5MB/sec 2022-04-22T18:01:54.862 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FJoR1GZ4x3cm2hj off=14085401 len=170334 2022-04-22T18:01:54.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 4518 completed, throughput=5MB/sec 2022-04-22T18:01:54.863 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9Hwo903-QEPXXwa off=2561056 len=1427607 2022-04-22T18:01:54.865 INFO:tasks.workunit.client.0.smithi115.stdout:op 4519 completed, throughput=5MB/sec 2022-04-22T18:01:54.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MvoLrzpsRw9DKXV off=8895112 len=1878030 2022-04-22T18:01:54.868 INFO:tasks.workunit.client.0.smithi115.stdout:op 4520 completed, throughput=5MB/sec 2022-04-22T18:01:55.212 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:01:55.212 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:01:55.212 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:01:55.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:01:55.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:01:55.214 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:01:55.214 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:01:55.214 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:01:55.214 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:01:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:55 smithi121 bash[36902]: audit 2022-04-22T18:01:54.425140+0000 mgr.y (mgr.34107) 1272 : audit [DBG] from='client.45673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:55.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:55 smithi121 bash[36902]: audit 2022-04-22T18:01:55.204910+0000 mon.c (mon.1) 79 : audit [DBG] from='client.? 172.21.15.115:0/1283869699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:55.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:55 smithi121 bash[36902]: audit 2022-04-22T18:01:55.324437+0000 mon.a (mon.0) 2796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[45495]: audit 2022-04-22T18:01:54.425140+0000 mgr.y (mgr.34107) 1272 : audit [DBG] from='client.45673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[45495]: audit 2022-04-22T18:01:55.204910+0000 mon.c (mon.1) 79 : audit [DBG] from='client.? 172.21.15.115:0/1283869699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[45495]: audit 2022-04-22T18:01:55.324437+0000 mon.a (mon.0) 2796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[46934]: audit 2022-04-22T18:01:54.425140+0000 mgr.y (mgr.34107) 1272 : audit [DBG] from='client.45673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[46934]: audit 2022-04-22T18:01:55.204910+0000 mon.c (mon.1) 79 : audit [DBG] from='client.? 172.21.15.115:0/1283869699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:01:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:55 smithi115 bash[46934]: audit 2022-04-22T18:01:55.324437+0000 mon.a (mon.0) 2796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:01:55.868 INFO:tasks.workunit.client.0.smithi115.stdout: 890: throughput=5MB/sec pending data=0 2022-04-22T18:01:55.868 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y1SnUL_1IDlBDfQ off=840012 len=1554930 2022-04-22T18:01:55.878 INFO:tasks.workunit.client.0.smithi115.stdout:op 4521 completed, throughput=5MB/sec 2022-04-22T18:01:55.878 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KbdMvk8ljBjGJ61 off=9616934 len=45413 2022-04-22T18:01:55.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 4522 completed, throughput=5MB/sec 2022-04-22T18:01:55.879 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u8sVBBro6jwc10f off=12747737 len=888925 2022-04-22T18:01:55.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 4523 completed, throughput=5MB/sec 2022-04-22T18:01:55.886 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DU7ayA5AUBdjyut off=941256 len=900535 2022-04-22T18:01:55.888 INFO:tasks.workunit.client.0.smithi115.stdout:op 4524 completed, throughput=5MB/sec 2022-04-22T18:01:55.888 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zye_xDwokYeAUah off=3998219 len=1707419 2022-04-22T18:01:55.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 4525 completed, throughput=5MB/sec 2022-04-22T18:01:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:56 smithi121 bash[36902]: cluster 2022-04-22T18:01:54.738142+0000 mgr.y (mgr.34107) 1273 : cluster [DBG] pgmap v1856: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:56.849 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:56 smithi121 bash[36902]: audit 2022-04-22T18:01:54.770370+0000 mgr.y (mgr.34107) 1274 : audit [DBG] from='client.35739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:56.849 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:56 smithi121 bash[36902]: audit 2022-04-22T18:01:55.713383+0000 mon.a (mon.0) 2797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[45495]: cluster 2022-04-22T18:01:54.738142+0000 mgr.y (mgr.34107) 1273 : cluster [DBG] pgmap v1856: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[45495]: audit 2022-04-22T18:01:54.770370+0000 mgr.y (mgr.34107) 1274 : audit [DBG] from='client.35739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:56.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[45495]: audit 2022-04-22T18:01:55.713383+0000 mon.a (mon.0) 2797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:56.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[46934]: cluster 2022-04-22T18:01:54.738142+0000 mgr.y (mgr.34107) 1273 : cluster [DBG] pgmap v1856: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:01:56.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[46934]: audit 2022-04-22T18:01:54.770370+0000 mgr.y (mgr.34107) 1274 : audit [DBG] from='client.35739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:01:56.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:56 smithi115 bash[46934]: audit 2022-04-22T18:01:55.713383+0000 mon.a (mon.0) 2797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:01:56.891 INFO:tasks.workunit.client.0.smithi115.stdout: 891: throughput=5MB/sec pending data=0 2022-04-22T18:01:56.891 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_01oWeF6ANME9cA off=14123310 len=501102 2022-04-22T18:01:56.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 4526 completed, throughput=5MB/sec 2022-04-22T18:01:56.893 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JhkIrjc1bMvYYV0 off=6719185 len=96146 2022-04-22T18:01:56.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 4527 completed, throughput=5MB/sec 2022-04-22T18:01:56.898 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9QDF9cKxHq106Qs off=822473 len=1347020 2022-04-22T18:01:56.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 4528 completed, throughput=5MB/sec 2022-04-22T18:01:56.901 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-54PUsIWKMweTFh5 off=721784 len=749080 2022-04-22T18:01:56.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 4529 completed, throughput=5MB/sec 2022-04-22T18:01:56.907 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cquPiHud6vmo8g6 off=1899054 len=597118 2022-04-22T18:01:56.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 4530 completed, throughput=5MB/sec 2022-04-22T18:01:56.911 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w8DqVm1K3SPm5fp off=6608975 len=530003 2022-04-22T18:01:56.916 INFO:tasks.workunit.client.0.smithi115.stdout:op 4531 completed, throughput=5MB/sec 2022-04-22T18:01:56.916 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6HsWGSO5DN-rxHV off=2723440 len=2032473 2022-04-22T18:01:56.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 4532 completed, throughput=5MB/sec 2022-04-22T18:01:57.927 INFO:tasks.workunit.client.0.smithi115.stdout: 892: throughput=5MB/sec pending data=0 2022-04-22T18:01:57.927 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f2JbwgGMmsfd9uM off=8338981 len=1231173 2022-04-22T18:01:57.930 INFO:tasks.workunit.client.0.smithi115.stdout:op 4533 completed, throughput=5MB/sec 2022-04-22T18:01:57.930 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vCJJOdGEs2d-Agv off=3006306 len=1762679 2022-04-22T18:01:57.936 INFO:tasks.workunit.client.0.smithi115.stdout:op 4534 completed, throughput=5MB/sec 2022-04-22T18:01:57.936 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zDCNIVr9utw2yog off=4436210 len=1397107 2022-04-22T18:01:57.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 4535 completed, throughput=5MB/sec 2022-04-22T18:01:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:58 smithi121 bash[36902]: cluster 2022-04-22T18:01:56.739331+0000 mgr.y (mgr.34107) 1275 : cluster [DBG] pgmap v1857: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:01:58 smithi121 bash[36902]: audit 2022-04-22T18:01:57.712532+0000 mon.a (mon.0) 2798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:58 smithi115 bash[45495]: cluster 2022-04-22T18:01:56.739331+0000 mgr.y (mgr.34107) 1275 : cluster [DBG] pgmap v1857: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:58.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:01:58 smithi115 bash[45495]: audit 2022-04-22T18:01:57.712532+0000 mon.a (mon.0) 2798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:58 smithi115 bash[46934]: cluster 2022-04-22T18:01:56.739331+0000 mgr.y (mgr.34107) 1275 : cluster [DBG] pgmap v1857: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:01:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:01:58 smithi115 bash[46934]: audit 2022-04-22T18:01:57.712532+0000 mon.a (mon.0) 2798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:01:58.939 INFO:tasks.workunit.client.0.smithi115.stdout: 893: throughput=4.99MB/sec pending data=0 2022-04-22T18:01:58.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CuXchtFQDZM8hLg off=1832896 len=259217 2022-04-22T18:01:58.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 4536 completed, throughput=5MB/sec 2022-04-22T18:01:58.941 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pjoviRigYjm4TVR off=5142958 len=937893 2022-04-22T18:01:58.945 INFO:tasks.workunit.client.0.smithi115.stdout:op 4537 completed, throughput=5MB/sec 2022-04-22T18:01:59.045 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VQAZG6h8DxQz2Fq off=19562858 len=1153574 2022-04-22T18:01:59.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 4538 completed, throughput=5MB/sec 2022-04-22T18:01:59.047 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dKRjR1TELYFNTJ1 off=6471244 len=1343679 2022-04-22T18:01:59.047 INFO:tasks.workunit.client.0.smithi115.stdout:op 4539 completed, throughput=5MB/sec 2022-04-22T18:01:59.048 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G9oqIuOVje3YnS9 off=738621 len=1324062 2022-04-22T18:01:59.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 4540 completed, throughput=5MB/sec 2022-04-22T18:01:59.048 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lmGozmf0hImZNjy off=8176917 len=1753027 2022-04-22T18:01:59.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 4541 completed, throughput=5MB/sec 2022-04-22T18:01:59.964 INFO:tasks.workunit.client.0.smithi115.stdout: 894: throughput=5MB/sec pending data=0 2022-04-22T18:01:59.964 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8__SBZoqQGd6EkC off=7157326 len=670569 2022-04-22T18:01:59.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 4542 completed, throughput=5MB/sec 2022-04-22T18:01:59.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4HeQKTVk3wFInC3 off=2733939 len=1566015 2022-04-22T18:01:59.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 4543 completed, throughput=5MB/sec 2022-04-22T18:01:59.971 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vhZAd91tT7SO4Go off=5837784 len=1264639 2022-04-22T18:01:59.975 INFO:tasks.workunit.client.0.smithi115.stdout:op 4544 completed, throughput=5MB/sec 2022-04-22T18:01:59.975 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZQZVIftfQDGeMk- off=7918472 len=1332327 2022-04-22T18:01:59.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 4545 completed, throughput=5MB/sec 2022-04-22T18:02:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:00 smithi121 bash[36902]: cluster 2022-04-22T18:01:58.740457+0000 mgr.y (mgr.34107) 1276 : cluster [DBG] pgmap v1858: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:02:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:00 smithi121 bash[36902]: audit 2022-04-22T18:02:00.327462+0000 mon.a (mon.0) 2799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:00 smithi115 bash[45495]: cluster 2022-04-22T18:01:58.740457+0000 mgr.y (mgr.34107) 1276 : cluster [DBG] pgmap v1858: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:02:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:00 smithi115 bash[45495]: audit 2022-04-22T18:02:00.327462+0000 mon.a (mon.0) 2799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:00 smithi115 bash[46934]: cluster 2022-04-22T18:01:58.740457+0000 mgr.y (mgr.34107) 1276 : cluster [DBG] pgmap v1858: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:02:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:00 smithi115 bash[46934]: audit 2022-04-22T18:02:00.327462+0000 mon.a (mon.0) 2799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:00.984 INFO:tasks.workunit.client.0.smithi115.stdout: 895: throughput=5MB/sec pending data=0 2022-04-22T18:02:00.984 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OmT_gDyhe5J3959 off=0 len=1394569 2022-04-22T18:02:00.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 4546 completed, throughput=5MB/sec 2022-04-22T18:02:00.987 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ef6fakPGEDfG4aV off=9727929 len=1141050 2022-04-22T18:02:00.995 INFO:tasks.workunit.client.0.smithi115.stdout:op 4547 completed, throughput=5MB/sec 2022-04-22T18:02:00.995 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xo7gZYiqETtix0B off=5510540 len=1025889 2022-04-22T18:02:01.002 INFO:tasks.workunit.client.0.smithi115.stdout:op 4548 completed, throughput=5MB/sec 2022-04-22T18:02:01.003 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-63ABZoiXwGoveuY off=1328244 len=972004 2022-04-22T18:02:01.020 INFO:tasks.workunit.client.0.smithi115.stdout:op 4549 completed, throughput=5MB/sec 2022-04-22T18:02:02.009 INFO:tasks.workunit.client.0.smithi115.stdout: 896: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:02.009 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VJlOGkOWo8mgdO6 off=13132772 len=824133 2022-04-22T18:02:02.015 INFO:tasks.workunit.client.0.smithi115.stdout:op 4550 completed, throughput=5MB/sec 2022-04-22T18:02:02.015 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_fIzvUnkxjSJNPp off=12170748 len=1116899 2022-04-22T18:02:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:01 smithi121 bash[36902]: audit 2022-04-22T18:02:00.715804+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:01 smithi121 bash[36902]: cluster 2022-04-22T18:02:00.741529+0000 mgr.y (mgr.34107) 1277 : cluster [DBG] pgmap v1859: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:02:02.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 4551 completed, throughput=5MB/sec 2022-04-22T18:02:02.023 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UQo0lAhSAw8-Fux off=4171324 len=760633 2022-04-22T18:02:02.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 4552 completed, throughput=5MB/sec 2022-04-22T18:02:02.025 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZRdgCvZ3NcWGJId off=8896152 len=1134968 2022-04-22T18:02:02.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 4553 completed, throughput=5MB/sec 2022-04-22T18:02:02.027 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FbLPszm-OkqI3Xu off=7101573 len=647666 2022-04-22T18:02:02.031 INFO:tasks.workunit.client.0.smithi115.stdout:op 4554 completed, throughput=5MB/sec 2022-04-22T18:02:02.031 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZwS2rgvGhSlaSFG off=6996818 len=916516 2022-04-22T18:02:02.036 INFO:tasks.workunit.client.0.smithi115.stdout:op 4555 completed, throughput=5MB/sec 2022-04-22T18:02:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:01 smithi115 bash[46934]: audit 2022-04-22T18:02:00.715804+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:01 smithi115 bash[46934]: cluster 2022-04-22T18:02:00.741529+0000 mgr.y (mgr.34107) 1277 : cluster [DBG] pgmap v1859: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:02:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:01.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:01 smithi115 bash[45495]: audit 2022-04-22T18:02:00.715804+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:01 smithi115 bash[45495]: cluster 2022-04-22T18:02:00.741529+0000 mgr.y (mgr.34107) 1277 : cluster [DBG] pgmap v1859: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:02:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:02 smithi121 bash[36902]: audit 2022-04-22T18:02:02.715565+0000 mon.a (mon.0) 2801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:03.037 INFO:tasks.workunit.client.0.smithi115.stdout: 897: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:03.037 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-33sWqbpluG5W46t off=2085482 len=1272220 2022-04-22T18:02:03.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 4556 completed, throughput=5MB/sec 2022-04-22T18:02:03.041 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YnxwyR7z5qGyJKx off=11461325 len=1383788 2022-04-22T18:02:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:02 smithi115 bash[46934]: audit 2022-04-22T18:02:02.715565+0000 mon.a (mon.0) 2801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:02 smithi115 bash[45495]: audit 2022-04-22T18:02:02.715565+0000 mon.a (mon.0) 2801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:03.049 INFO:tasks.workunit.client.0.smithi115.stdout:op 4557 completed, throughput=5MB/sec 2022-04-22T18:02:03.049 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1ILGE2J9IM2frJI off=13498669 len=2095013 2022-04-22T18:02:03.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 4558 completed, throughput=5MB/sec 2022-04-22T18:02:03.060 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sL45o9AQ8XUsMHQ off=10461408 len=1275371 2022-04-22T18:02:03.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 4559 completed, throughput=5MB/sec 2022-04-22T18:02:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:03 smithi121 bash[36902]: cluster 2022-04-22T18:02:02.742660+0000 mgr.y (mgr.34107) 1278 : cluster [DBG] pgmap v1860: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:03 smithi115 bash[45495]: cluster 2022-04-22T18:02:02.742660+0000 mgr.y (mgr.34107) 1278 : cluster [DBG] pgmap v1860: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:03.613Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:03 smithi115 bash[46934]: cluster 2022-04-22T18:02:02.742660+0000 mgr.y (mgr.34107) 1278 : cluster [DBG] pgmap v1860: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:04.056 INFO:tasks.workunit.client.0.smithi115.stdout: 898: throughput=5MB/sec pending data=0 2022-04-22T18:02:04.056 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aFR8_K_xxXJ9_RJ off=1842621 len=794655 2022-04-22T18:02:04.064 INFO:tasks.workunit.client.0.smithi115.stdout:op 4560 completed, throughput=5MB/sec 2022-04-22T18:02:04.065 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NOrdGbyU6pn_3ap off=3531777 len=1347912 2022-04-22T18:02:04.071 INFO:tasks.workunit.client.0.smithi115.stdout:op 4561 completed, throughput=5MB/sec 2022-04-22T18:02:04.071 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vqoP4SXvBdciyj9 off=831752 len=1523702 2022-04-22T18:02:04.078 INFO:tasks.workunit.client.0.smithi115.stdout:op 4562 completed, throughput=5MB/sec 2022-04-22T18:02:04.078 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AhRK-mj57igE2UJ off=16215717 len=981380 2022-04-22T18:02:04.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 4563 completed, throughput=5MB/sec 2022-04-22T18:02:05.084 INFO:tasks.workunit.client.0.smithi115.stdout: 899: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:05.085 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ncY7SgnFOYSn2jl off=12672079 len=1174429 2022-04-22T18:02:05.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 4564 completed, throughput=5MB/sec 2022-04-22T18:02:05.091 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XyABFOG0a4y74Wj off=5348767 len=107123 2022-04-22T18:02:05.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 4565 completed, throughput=5MB/sec 2022-04-22T18:02:05.091 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KAa6EFSu6D68Dww off=1353683 len=516553 2022-04-22T18:02:05.095 INFO:tasks.workunit.client.0.smithi115.stdout:op 4566 completed, throughput=5MB/sec 2022-04-22T18:02:05.095 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TWGXDU-Kj2aVi_c off=2377097 len=356166 2022-04-22T18:02:05.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 4567 completed, throughput=5MB/sec 2022-04-22T18:02:05.099 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0JCYC7HMsjZGe5o off=3326354 len=1426654 2022-04-22T18:02:05.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 4568 completed, throughput=5MB/sec 2022-04-22T18:02:05.102 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o9GvIUKP4eNODlt off=12317158 len=798094 2022-04-22T18:02:05.104 INFO:tasks.workunit.client.0.smithi115.stdout:op 4569 completed, throughput=5MB/sec 2022-04-22T18:02:05.104 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0Q712pm5HEfreUN off=13820846 len=1040752 2022-04-22T18:02:05.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 4570 completed, throughput=5MB/sec 2022-04-22T18:02:06.107 INFO:tasks.workunit.client.0.smithi115.stdout: 900: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:06.108 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zKDI3g5rLwcA2hC off=14887396 len=686544 2022-04-22T18:02:06.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 4571 completed, throughput=5MB/sec 2022-04-22T18:02:06.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AzI6YOfdD1aIH4n off=11177493 len=231546 2022-04-22T18:02:06.119 INFO:tasks.workunit.client.0.smithi115.stdout:op 4572 completed, throughput=5MB/sec 2022-04-22T18:02:06.119 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dozoLCP7mHPEXf5 off=336515 len=1979638 2022-04-22T18:02:06.122 INFO:tasks.workunit.client.0.smithi115.stdout:op 4573 completed, throughput=5MB/sec 2022-04-22T18:02:06.122 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XaTfyQXl8l2IUFn off=12852404 len=444799 2022-04-22T18:02:06.126 INFO:tasks.workunit.client.0.smithi115.stdout:op 4574 completed, throughput=5MB/sec 2022-04-22T18:02:06.126 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k-AXa0lkITl73Gu off=5480098 len=1690692 2022-04-22T18:02:06.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 4575 completed, throughput=5MB/sec 2022-04-22T18:02:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:05 smithi121 bash[36902]: cluster 2022-04-22T18:02:04.743758+0000 mgr.y (mgr.34107) 1279 : cluster [DBG] pgmap v1861: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:05 smithi121 bash[36902]: audit 2022-04-22T18:02:05.330621+0000 mon.a (mon.0) 2802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:05 smithi121 bash[36902]: audit 2022-04-22T18:02:05.718908+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[45495]: cluster 2022-04-22T18:02:04.743758+0000 mgr.y (mgr.34107) 1279 : cluster [DBG] pgmap v1861: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[45495]: audit 2022-04-22T18:02:05.330621+0000 mon.a (mon.0) 2802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[45495]: audit 2022-04-22T18:02:05.718908+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[46934]: cluster 2022-04-22T18:02:04.743758+0000 mgr.y (mgr.34107) 1279 : cluster [DBG] pgmap v1861: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[46934]: audit 2022-04-22T18:02:05.330621+0000 mon.a (mon.0) 2802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:05 smithi115 bash[46934]: audit 2022-04-22T18:02:05.718908+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:07.129 INFO:tasks.workunit.client.0.smithi115.stdout: 901: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:07.129 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aEaOwWec8GaH-h4 off=1600599 len=477314 2022-04-22T18:02:07.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 4576 completed, throughput=5MB/sec 2022-04-22T18:02:07.136 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zx7JCjcKJJTFiuk off=16733620 len=1735168 2022-04-22T18:02:07.142 INFO:tasks.workunit.client.0.smithi115.stdout:op 4577 completed, throughput=5MB/sec 2022-04-22T18:02:07.142 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pZY24E3L-3dDoHo off=7880047 len=1217790 2022-04-22T18:02:07.149 INFO:tasks.workunit.client.0.smithi115.stdout:op 4578 completed, throughput=5MB/sec 2022-04-22T18:02:07.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LMdw9z7xSLIE1Bs off=810786 len=1178360 2022-04-22T18:02:07.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 4579 completed, throughput=5MB/sec 2022-04-22T18:02:07.152 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KQeH8V2SO1g4mHd off=9209178 len=1180767 2022-04-22T18:02:07.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 4580 completed, throughput=5MB/sec 2022-04-22T18:02:08.154 INFO:tasks.workunit.client.0.smithi115.stdout: 902: throughput=5MB/sec pending data=0 2022-04-22T18:02:08.166 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m8egl2EZzSAtivm off=2975803 len=173361 2022-04-22T18:02:08.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 4581 completed, throughput=5MB/sec 2022-04-22T18:02:08.168 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kf89CPI2onNuyOf off=21722164 len=242087 2022-04-22T18:02:08.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 4582 completed, throughput=5MB/sec 2022-04-22T18:02:08.168 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VI-2YCZmxF646t- off=4896097 len=1675091 2022-04-22T18:02:08.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 4583 completed, throughput=5MB/sec 2022-04-22T18:02:08.171 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PNXUBBIlB7607Ex off=8294531 len=1122093 2022-04-22T18:02:08.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 4584 completed, throughput=5MB/sec 2022-04-22T18:02:08.177 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-faDMLmbPnVgVPuP off=16173447 len=2036240 2022-04-22T18:02:08.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 4585 completed, throughput=5MB/sec 2022-04-22T18:02:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:07 smithi121 bash[36902]: cluster 2022-04-22T18:02:06.744784+0000 mgr.y (mgr.34107) 1280 : cluster [DBG] pgmap v1862: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:02:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:07 smithi121 bash[36902]: audit 2022-04-22T18:02:07.718524+0000 mon.a (mon.0) 2804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:07 smithi115 bash[45495]: cluster 2022-04-22T18:02:06.744784+0000 mgr.y (mgr.34107) 1280 : cluster [DBG] pgmap v1862: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:02:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:07 smithi115 bash[45495]: audit 2022-04-22T18:02:07.718524+0000 mon.a (mon.0) 2804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:07 smithi115 bash[46934]: cluster 2022-04-22T18:02:06.744784+0000 mgr.y (mgr.34107) 1280 : cluster [DBG] pgmap v1862: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:02:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:07 smithi115 bash[46934]: audit 2022-04-22T18:02:07.718524+0000 mon.a (mon.0) 2804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:09.181 INFO:tasks.workunit.client.0.smithi115.stdout: 903: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:09.182 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tp4i5_s5gIzvOm1 off=2007282 len=1838907 2022-04-22T18:02:09.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 4586 completed, throughput=5MB/sec 2022-04-22T18:02:09.190 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-THCg6KsFinYnn39 off=929824 len=848152 2022-04-22T18:02:09.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 4587 completed, throughput=5MB/sec 2022-04-22T18:02:09.192 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n3SLN6EXrhqqQ4e off=846058 len=1386821 2022-04-22T18:02:09.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 4588 completed, throughput=5MB/sec 2022-04-22T18:02:09.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ti0ih2bSHQM-OId off=2012403 len=1278399 2022-04-22T18:02:09.207 INFO:tasks.workunit.client.0.smithi115.stdout:op 4589 completed, throughput=5MB/sec 2022-04-22T18:02:10.207 INFO:tasks.workunit.client.0.smithi115.stdout: 904: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:10.208 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yO7yfoQwWVKfZeU off=1844319 len=221908 2022-04-22T18:02:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 4590 completed, throughput=5MB/sec 2022-04-22T18:02:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MMoBrTP55D8U7-L off=18936079 len=1454666 2022-04-22T18:02:10.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 4591 completed, throughput=5MB/sec 2022-04-22T18:02:10.220 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ja9VQWadgi7InV2 off=3151035 len=1687241 2022-04-22T18:02:10.225 INFO:tasks.workunit.client.0.smithi115.stdout:op 4592 completed, throughput=5MB/sec 2022-04-22T18:02:10.225 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WINKbb4qdkaXZlK off=2461668 len=1373038 2022-04-22T18:02:10.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 4593 completed, throughput=5MB/sec 2022-04-22T18:02:10.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EUT38xz6YzQtH_b off=3120131 len=426228 2022-04-22T18:02:10.237 INFO:tasks.workunit.client.0.smithi115.stdout:op 4594 completed, throughput=5MB/sec 2022-04-22T18:02:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:09 smithi121 bash[36902]: cluster 2022-04-22T18:02:08.745878+0000 mgr.y (mgr.34107) 1281 : cluster [DBG] pgmap v1863: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:10.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:09 smithi115 bash[45495]: cluster 2022-04-22T18:02:08.745878+0000 mgr.y (mgr.34107) 1281 : cluster [DBG] pgmap v1863: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:09 smithi115 bash[46934]: cluster 2022-04-22T18:02:08.745878+0000 mgr.y (mgr.34107) 1281 : cluster [DBG] pgmap v1863: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:11.237 INFO:tasks.workunit.client.0.smithi115.stdout: 905: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:11.238 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MJyNR2ykvqtYB5i off=5000609 len=1720842 2022-04-22T18:02:11.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 4595 completed, throughput=5MB/sec 2022-04-22T18:02:11.241 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZQZzHeNHrFAHlLX off=10544068 len=558023 2022-04-22T18:02:11.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 4596 completed, throughput=5MB/sec 2022-04-22T18:02:11.247 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gVgwXbQlXVnzkUq off=5543761 len=569302 2022-04-22T18:02:11.253 INFO:tasks.workunit.client.0.smithi115.stdout:op 4597 completed, throughput=5MB/sec 2022-04-22T18:02:11.253 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h_ErsIRhOUNT8jW off=516073 len=1417525 2022-04-22T18:02:11.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 4598 completed, throughput=5MB/sec 2022-04-22T18:02:11.256 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vTkHBLIhJbZbIuR off=9343174 len=1575089 2022-04-22T18:02:11.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 4599 completed, throughput=5MB/sec 2022-04-22T18:02:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:10 smithi121 bash[36902]: audit 2022-04-22T18:02:10.333947+0000 mon.a (mon.0) 2805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:10 smithi121 bash[36902]: audit 2022-04-22T18:02:10.721968+0000 mon.a (mon.0) 2806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:10 smithi115 bash[45495]: audit 2022-04-22T18:02:10.333947+0000 mon.a (mon.0) 2805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:10 smithi115 bash[45495]: audit 2022-04-22T18:02:10.721968+0000 mon.a (mon.0) 2806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:10 smithi115 bash[46934]: audit 2022-04-22T18:02:10.333947+0000 mon.a (mon.0) 2805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:10 smithi115 bash[46934]: audit 2022-04-22T18:02:10.721968+0000 mon.a (mon.0) 2806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:11 smithi115 bash[45495]: cluster 2022-04-22T18:02:10.746942+0000 mgr.y (mgr.34107) 1282 : cluster [DBG] pgmap v1864: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T18:02:12.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:11.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:11 smithi115 bash[46934]: cluster 2022-04-22T18:02:10.746942+0000 mgr.y (mgr.34107) 1282 : cluster [DBG] pgmap v1864: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T18:02:12.264 INFO:tasks.workunit.client.0.smithi115.stdout: 906: throughput=5MB/sec pending data=0 2022-04-22T18:02:12.265 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9t5aE6RGnppprYD off=12634005 len=793610 2022-04-22T18:02:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:11 smithi121 bash[36902]: cluster 2022-04-22T18:02:10.746942+0000 mgr.y (mgr.34107) 1282 : cluster [DBG] pgmap v1864: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2022-04-22T18:02:12.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 4600 completed, throughput=5MB/sec 2022-04-22T18:02:12.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y8gZFw9xoMfCIih off=6352514 len=445844 2022-04-22T18:02:12.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 4601 completed, throughput=5MB/sec 2022-04-22T18:02:12.276 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_735UOCp0UceSIp off=35406 len=1091712 2022-04-22T18:02:12.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 4602 completed, throughput=5MB/sec 2022-04-22T18:02:12.279 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_PZ1vuwQPH6j8w0 off=1932379 len=1401187 2022-04-22T18:02:12.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 4603 completed, throughput=5MB/sec 2022-04-22T18:02:12.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5REHiT3hekPh8k- off=354078 len=1072001 2022-04-22T18:02:12.282 INFO:tasks.workunit.client.0.smithi115.stdout:op 4604 completed, throughput=5MB/sec 2022-04-22T18:02:12.282 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W2JbZgHd8-t9_Bp off=17416606 len=450570 2022-04-22T18:02:12.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 4605 completed, throughput=5MB/sec 2022-04-22T18:02:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:12 smithi121 bash[36902]: audit 2022-04-22T18:02:12.721534+0000 mon.a (mon.0) 2807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:13.286 INFO:tasks.workunit.client.0.smithi115.stdout: 907: throughput=5MB/sec pending data=0 2022-04-22T18:02:13.286 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z74B4cs99ANekW1 off=7195954 len=960626 2022-04-22T18:02:13.293 INFO:tasks.workunit.client.0.smithi115.stdout:op 4606 completed, throughput=5MB/sec 2022-04-22T18:02:13.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fwJP8kikyfciGIF off=1639812 len=899318 2022-04-22T18:02:13.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:12 smithi115 bash[46934]: audit 2022-04-22T18:02:12.721534+0000 mon.a (mon.0) 2807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:12 smithi115 bash[45495]: audit 2022-04-22T18:02:12.721534+0000 mon.a (mon.0) 2807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:13.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 4607 completed, throughput=5MB/sec 2022-04-22T18:02:13.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XztH1VNAad-zw_k off=2406236 len=1018259 2022-04-22T18:02:13.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 4608 completed, throughput=5MB/sec 2022-04-22T18:02:13.305 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vXxZP1XIed_8lTW off=1865648 len=1491900 2022-04-22T18:02:13.309 INFO:tasks.workunit.client.0.smithi115.stdout:op 4609 completed, throughput=5MB/sec 2022-04-22T18:02:13.309 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uaCnf0YCUAV8U7E off=5439797 len=557856 2022-04-22T18:02:13.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 4610 completed, throughput=5MB/sec 2022-04-22T18:02:13.311 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X-ieh0EgB9Z1kbk off=5698876 len=112730 2022-04-22T18:02:13.312 INFO:tasks.workunit.client.0.smithi115.stdout:op 4611 completed, throughput=5MB/sec 2022-04-22T18:02:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:13.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[45495]: cluster 2022-04-22T18:02:12.748079+0000 mgr.y (mgr.34107) 1283 : cluster [DBG] pgmap v1865: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[45495]: audit 2022-04-22T18:02:13.620155+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:02:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[45495]: audit 2022-04-22T18:02:13.620846+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:02:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[46934]: cluster 2022-04-22T18:02:12.748079+0000 mgr.y (mgr.34107) 1283 : cluster [DBG] pgmap v1865: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[46934]: audit 2022-04-22T18:02:13.620155+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:02:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:13 smithi115 bash[46934]: audit 2022-04-22T18:02:13.620846+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:02:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:13 smithi121 bash[36902]: cluster 2022-04-22T18:02:12.748079+0000 mgr.y (mgr.34107) 1283 : cluster [DBG] pgmap v1865: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 4 op/s 2022-04-22T18:02:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:13 smithi121 bash[36902]: audit 2022-04-22T18:02:13.620155+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:02:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:13 smithi121 bash[36902]: audit 2022-04-22T18:02:13.620846+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:02:14.313 INFO:tasks.workunit.client.0.smithi115.stdout: 908: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:14.313 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oRG52Vd11NwY-0T off=7843440 len=716874 2022-04-22T18:02:14.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 4612 completed, throughput=5MB/sec 2022-04-22T18:02:14.319 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VDk4iXYjGUhG_UF off=11257666 len=1757114 2022-04-22T18:02:14.323 INFO:tasks.workunit.client.0.smithi115.stdout:op 4613 completed, throughput=5MB/sec 2022-04-22T18:02:14.323 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lXW3_JeQu5tJvIU off=497755 len=1716855 2022-04-22T18:02:14.328 INFO:tasks.workunit.client.0.smithi115.stdout:op 4614 completed, throughput=5MB/sec 2022-04-22T18:02:14.328 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_p09GWF-cN9uJrC off=4504437 len=953605 2022-04-22T18:02:14.332 INFO:tasks.workunit.client.0.smithi115.stdout:op 4615 completed, throughput=5MB/sec 2022-04-22T18:02:14.332 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wEwukQrs1IKaEpQ off=1548477 len=1353018 2022-04-22T18:02:14.341 INFO:tasks.workunit.client.0.smithi115.stdout:op 4616 completed, throughput=5MB/sec 2022-04-22T18:02:15.341 INFO:tasks.workunit.client.0.smithi115.stdout: 909: throughput=5MB/sec pending data=0 2022-04-22T18:02:15.341 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QAqyz0IkyfwiCEn off=7685982 len=697253 2022-04-22T18:02:15.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 4617 completed, throughput=5MB/sec 2022-04-22T18:02:15.344 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rYmQA9O0R_ijksg off=3578032 len=549884 2022-04-22T18:02:15.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 4618 completed, throughput=5MB/sec 2022-04-22T18:02:15.347 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VwM5DETpUImbN3P off=14162046 len=1156797 2022-04-22T18:02:15.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 4619 completed, throughput=5MB/sec 2022-04-22T18:02:15.350 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KHVvdJfXr0ksDMe off=205139 len=1104450 2022-04-22T18:02:15.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 4620 completed, throughput=5MB/sec 2022-04-22T18:02:15.355 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vw8IQc_n_caTo_J off=2366070 len=616839 2022-04-22T18:02:15.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 4621 completed, throughput=5MB/sec 2022-04-22T18:02:15.362 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AQ48qAQRdNQiO_b off=13787954 len=1750658 2022-04-22T18:02:15.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 4622 completed, throughput=5MB/sec 2022-04-22T18:02:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:15 smithi121 bash[36902]: cluster 2022-04-22T18:02:14.749140+0000 mgr.y (mgr.34107) 1284 : cluster [DBG] pgmap v1866: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:15 smithi121 bash[36902]: audit 2022-04-22T18:02:15.337004+0000 mon.a (mon.0) 2810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:15 smithi121 bash[36902]: audit 2022-04-22T18:02:15.725296+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[46934]: cluster 2022-04-22T18:02:14.749140+0000 mgr.y (mgr.34107) 1284 : cluster [DBG] pgmap v1866: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:16.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[46934]: audit 2022-04-22T18:02:15.337004+0000 mon.a (mon.0) 2810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:16.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[46934]: audit 2022-04-22T18:02:15.725296+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:16.399 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[45495]: cluster 2022-04-22T18:02:14.749140+0000 mgr.y (mgr.34107) 1284 : cluster [DBG] pgmap v1866: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:16.400 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[45495]: audit 2022-04-22T18:02:15.337004+0000 mon.a (mon.0) 2810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:16.400 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:15 smithi115 bash[45495]: audit 2022-04-22T18:02:15.725296+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:16.401 INFO:tasks.workunit.client.0.smithi115.stdout: 910: throughput=5MB/sec pending data=0 2022-04-22T18:02:16.401 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YsRckZPbC-Iw8WZ off=714319 len=680585 2022-04-22T18:02:16.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 4623 completed, throughput=5MB/sec 2022-04-22T18:02:16.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zZAQhNd13DVIxWH off=18622226 len=1474626 2022-04-22T18:02:16.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 4624 completed, throughput=5MB/sec 2022-04-22T18:02:16.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4PxARb-SPN5lvpV off=4177395 len=1861297 2022-04-22T18:02:16.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 4625 completed, throughput=5MB/sec 2022-04-22T18:02:17.386 INFO:tasks.workunit.client.0.smithi115.stdout: 911: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:17.452 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OjISfcHvAct4tuS off=8016452 len=418447 2022-04-22T18:02:17.454 INFO:tasks.workunit.client.0.smithi115.stdout:op 4626 completed, throughput=5MB/sec 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wAmklPAQyQlwHyy off=5858835 len=1079552 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 4627 completed, throughput=5MB/sec 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w8DqVm1K3SPm5fp off=18291628 len=599335 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 4628 completed, throughput=5MB/sec 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rjroJNRygZ0t4w3 off=6771033 len=230149 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 4629 completed, throughput=5MB/sec 2022-04-22T18:02:17.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8-SJRNmaIHKzhot off=2709143 len=409834 2022-04-22T18:02:17.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 4630 completed, throughput=5MB/sec 2022-04-22T18:02:17.456 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RR34tvWJzf81V07 off=21863802 len=1893653 2022-04-22T18:02:17.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 4631 completed, throughput=5MB/sec 2022-04-22T18:02:17.456 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KXrCLTehtT0GD-p off=2133453 len=1742605 2022-04-22T18:02:17.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 4632 completed, throughput=5MB/sec 2022-04-22T18:02:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:17 smithi121 bash[36902]: cluster 2022-04-22T18:02:16.750145+0000 mgr.y (mgr.34107) 1285 : cluster [DBG] pgmap v1867: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:17 smithi121 bash[36902]: audit 2022-04-22T18:02:17.724511+0000 mon.a (mon.0) 2812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:17 smithi115 bash[45495]: cluster 2022-04-22T18:02:16.750145+0000 mgr.y (mgr.34107) 1285 : cluster [DBG] pgmap v1867: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:17 smithi115 bash[45495]: audit 2022-04-22T18:02:17.724511+0000 mon.a (mon.0) 2812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:17 smithi115 bash[46934]: cluster 2022-04-22T18:02:16.750145+0000 mgr.y (mgr.34107) 1285 : cluster [DBG] pgmap v1867: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:17 smithi115 bash[46934]: audit 2022-04-22T18:02:17.724511+0000 mon.a (mon.0) 2812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:18.416 INFO:tasks.workunit.client.0.smithi115.stdout: 912: throughput=5MB/sec pending data=0 2022-04-22T18:02:18.416 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zKnDfdlyouJJnV0 off=3138613 len=312362 2022-04-22T18:02:18.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 4633 completed, throughput=5MB/sec 2022-04-22T18:02:18.422 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-acbdzDAsPxty3-2 off=648892 len=784475 2022-04-22T18:02:18.423 INFO:tasks.workunit.client.0.smithi115.stdout:op 4634 completed, throughput=5MB/sec 2022-04-22T18:02:18.424 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JIXloF--2zuW3P1 off=8027943 len=514908 2022-04-22T18:02:18.429 INFO:tasks.workunit.client.0.smithi115.stdout:op 4635 completed, throughput=5MB/sec 2022-04-22T18:02:18.429 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DBlG8Y5-7a57wVU off=2795950 len=1752803 2022-04-22T18:02:18.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 4636 completed, throughput=5MB/sec 2022-04-22T18:02:18.432 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E5O63YB-1orVOlJ off=1800620 len=1612580 2022-04-22T18:02:18.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 4637 completed, throughput=5MB/sec 2022-04-22T18:02:19.438 INFO:tasks.workunit.client.0.smithi115.stdout: 913: throughput=5MB/sec pending data=0 2022-04-22T18:02:19.438 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GqKy3iHwSSVXHKJ off=18248849 len=898905 2022-04-22T18:02:19.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 4638 completed, throughput=5MB/sec 2022-04-22T18:02:19.441 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jsAxcQT5nm7OYLh off=6802965 len=1785431 2022-04-22T18:02:19.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 4639 completed, throughput=5MB/sec 2022-04-22T18:02:19.449 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ccLMT9gHchtTijn off=0 len=1137975 2022-04-22T18:02:19.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 4640 completed, throughput=5MB/sec 2022-04-22T18:02:19.451 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GZOwgDh6N7WlWza off=10046462 len=1697687 2022-04-22T18:02:19.457 INFO:tasks.workunit.client.0.smithi115.stdout:op 4641 completed, throughput=5MB/sec 2022-04-22T18:02:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:19 smithi121 bash[36902]: cluster 2022-04-22T18:02:18.751521+0000 mgr.y (mgr.34107) 1286 : cluster [DBG] pgmap v1868: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:19 smithi115 bash[45495]: cluster 2022-04-22T18:02:18.751521+0000 mgr.y (mgr.34107) 1286 : cluster [DBG] pgmap v1868: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:19 smithi115 bash[46934]: cluster 2022-04-22T18:02:18.751521+0000 mgr.y (mgr.34107) 1286 : cluster [DBG] pgmap v1868: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:02:20.458 INFO:tasks.workunit.client.0.smithi115.stdout: 914: throughput=5MB/sec pending data=0 2022-04-22T18:02:20.458 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r4OEXMW_A5QS3DQ off=19859528 len=1885652 2022-04-22T18:02:20.469 INFO:tasks.workunit.client.0.smithi115.stdout:op 4642 completed, throughput=5MB/sec 2022-04-22T18:02:20.469 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b26_zRsE4-P6ESx off=7633995 len=1891207 2022-04-22T18:02:20.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 4643 completed, throughput=5MB/sec 2022-04-22T18:02:20.480 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DyEAfETmSqhlxqd off=0 len=1168022 2022-04-22T18:02:20.482 INFO:tasks.workunit.client.0.smithi115.stdout:op 4644 completed, throughput=5MB/sec 2022-04-22T18:02:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:20 smithi121 bash[36902]: audit 2022-04-22T18:02:20.340226+0000 mon.a (mon.0) 2813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:20 smithi121 bash[36902]: audit 2022-04-22T18:02:20.728408+0000 mon.a (mon.0) 2814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:20 smithi115 bash[45495]: audit 2022-04-22T18:02:20.340226+0000 mon.a (mon.0) 2813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:20 smithi115 bash[45495]: audit 2022-04-22T18:02:20.728408+0000 mon.a (mon.0) 2814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:20 smithi115 bash[46934]: audit 2022-04-22T18:02:20.340226+0000 mon.a (mon.0) 2813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:20 smithi115 bash[46934]: audit 2022-04-22T18:02:20.728408+0000 mon.a (mon.0) 2814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:21.482 INFO:tasks.workunit.client.0.smithi115.stdout: 915: throughput=5MB/sec pending data=0 2022-04-22T18:02:21.483 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7Wd8-ddXiHxpLrN off=4288887 len=343647 2022-04-22T18:02:21.488 INFO:tasks.workunit.client.0.smithi115.stdout:op 4645 completed, throughput=5MB/sec 2022-04-22T18:02:21.488 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Srw2JxOenLn10K4 off=7350781 len=595893 2022-04-22T18:02:21.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 4646 completed, throughput=5MB/sec 2022-04-22T18:02:21.490 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uDr5F-uKUiHhG0B off=13630659 len=546648 2022-04-22T18:02:21.492 INFO:tasks.workunit.client.0.smithi115.stdout:op 4647 completed, throughput=5MB/sec 2022-04-22T18:02:21.534 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ja9VQWadgi7InV2 off=403935 len=1117677 2022-04-22T18:02:21.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 4648 completed, throughput=5MB/sec 2022-04-22T18:02:21.535 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1cp3Zg3I3u1RpPq off=4307009 len=624567 2022-04-22T18:02:21.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 4649 completed, throughput=5MB/sec 2022-04-22T18:02:21.536 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v2OMCYRZFO5gB1k off=5687645 len=1089897 2022-04-22T18:02:21.536 INFO:tasks.workunit.client.0.smithi115.stdout:op 4650 completed, throughput=5MB/sec 2022-04-22T18:02:21.536 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-shFyy7RQBucH80Y off=8931058 len=1522432 2022-04-22T18:02:21.536 INFO:tasks.workunit.client.0.smithi115.stdout:op 4651 completed, throughput=5MB/sec 2022-04-22T18:02:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:21 smithi115 bash[45495]: cluster 2022-04-22T18:02:20.752553+0000 mgr.y (mgr.34107) 1287 : cluster [DBG] pgmap v1869: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:02:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:21.729Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:21 smithi115 bash[46934]: cluster 2022-04-22T18:02:20.752553+0000 mgr.y (mgr.34107) 1287 : cluster [DBG] pgmap v1869: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:02:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:21 smithi121 bash[36902]: cluster 2022-04-22T18:02:20.752553+0000 mgr.y (mgr.34107) 1287 : cluster [DBG] pgmap v1869: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:02:22.503 INFO:tasks.workunit.client.0.smithi115.stdout: 916: throughput=5MB/sec pending data=0 2022-04-22T18:02:22.536 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2lxnMkGd_vO-WXh off=2248504 len=566039 2022-04-22T18:02:22.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 4652 completed, throughput=5MB/sec 2022-04-22T18:02:22.537 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NjhFfdIACyu18sy off=18592943 len=498628 2022-04-22T18:02:22.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 4653 completed, throughput=5MB/sec 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--kQNgnFR1GkV5TU off=7075415 len=1489802 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:op 4654 completed, throughput=5MB/sec 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BfTSy7_gyEBaCS2 off=12607975 len=535941 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:op 4655 completed, throughput=5MB/sec 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eofMQdzGg03cKhP off=2649960 len=688447 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:op 4656 completed, throughput=5MB/sec 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YDLQlJjWY_87u9w off=1053343 len=547938 2022-04-22T18:02:22.538 INFO:tasks.workunit.client.0.smithi115.stdout:op 4657 completed, throughput=5MB/sec 2022-04-22T18:02:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:22 smithi121 bash[36902]: audit 2022-04-22T18:02:22.727586+0000 mon.a (mon.0) 2815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:22 smithi115 bash[46934]: audit 2022-04-22T18:02:22.727586+0000 mon.a (mon.0) 2815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:22 smithi115 bash[45495]: audit 2022-04-22T18:02:22.727586+0000 mon.a (mon.0) 2815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:23.524 INFO:tasks.workunit.client.0.smithi115.stdout: 917: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:23.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lnBJhOi8k_UJRAP off=2149130 len=1840526 2022-04-22T18:02:23.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 4658 completed, throughput=5MB/sec 2022-04-22T18:02:23.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-U_0xEo04wpVr3tC off=9008570 len=393646 2022-04-22T18:02:23.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 4659 completed, throughput=5MB/sec 2022-04-22T18:02:23.533 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QqL1mz2N2uZS0F- off=1783788 len=2011839 2022-04-22T18:02:23.545 INFO:tasks.workunit.client.0.smithi115.stdout:op 4660 completed, throughput=5MB/sec 2022-04-22T18:02:23.545 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0jmx1xn5p__a4E0 off=2713744 len=1706529 2022-04-22T18:02:23.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 4661 completed, throughput=5MB/sec 2022-04-22T18:02:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:23 smithi115 bash[46934]: cluster 2022-04-22T18:02:22.753593+0000 mgr.y (mgr.34107) 1288 : cluster [DBG] pgmap v1870: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:23.614Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:23 smithi115 bash[45495]: cluster 2022-04-22T18:02:22.753593+0000 mgr.y (mgr.34107) 1288 : cluster [DBG] pgmap v1870: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:23 smithi121 bash[36902]: cluster 2022-04-22T18:02:22.753593+0000 mgr.y (mgr.34107) 1288 : cluster [DBG] pgmap v1870: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:24.551 INFO:tasks.workunit.client.0.smithi115.stdout: 918: throughput=5MB/sec pending data=0 2022-04-22T18:02:24.552 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g4m7A598YhQHdRT off=2205161 len=1145585 2022-04-22T18:02:24.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 4662 completed, throughput=5MB/sec 2022-04-22T18:02:24.554 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xhj4ecMX_6e4tMs off=2879219 len=1924453 2022-04-22T18:02:24.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 4663 completed, throughput=5MB/sec 2022-04-22T18:02:24.560 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-B32M3RWzj1bsLFE off=7053597 len=1695324 2022-04-22T18:02:24.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 4664 completed, throughput=5MB/sec 2022-04-22T18:02:25.568 INFO:tasks.workunit.client.0.smithi115.stdout: 919: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:25.569 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3FPU0xkx7rxWfMo off=468860 len=1499214 2022-04-22T18:02:25.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 4665 completed, throughput=5MB/sec 2022-04-22T18:02:25.576 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5sHvl14ja0S2P7Q off=1568284 len=839919 2022-04-22T18:02:25.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 4666 completed, throughput=5MB/sec 2022-04-22T18:02:25.578 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e5fzgsuQNEPSOga off=21312724 len=704131 2022-04-22T18:02:25.580 INFO:tasks.workunit.client.0.smithi115.stdout:op 4667 completed, throughput=5MB/sec 2022-04-22T18:02:25.581 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z74B4cs99ANekW1 off=3660108 len=1522476 2022-04-22T18:02:25.583 INFO:tasks.workunit.client.0.smithi115.stdout:op 4668 completed, throughput=5MB/sec 2022-04-22T18:02:25.583 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P_IyAFyAFhmtuBN off=13709319 len=807666 2022-04-22T18:02:25.584 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:02:25.584 INFO:tasks.workunit.client.0.smithi115.stdout:op 4669 completed, throughput=5MB/sec 2022-04-22T18:02:25.930 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:02:25.931 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (25m) 4m ago 39m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:02:25.931 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (39m) 5m ago 39m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:02:25.931 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (27m) 4m ago 40m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:02:25.931 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (27m) 5m ago 40m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:02:25.932 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (34m) 5m ago 44m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:02:25.932 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (35m) 4m ago 46m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:02:25.932 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (34m) 4m ago 46m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:02:25.932 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (33m) 5m ago 44m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:02:25.933 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (33m) 4m ago 45m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:02:25.933 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (26m) 4m ago 39m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:02:25.933 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (26m) 5m ago 39m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:02:25.933 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (32m) 4m ago 44m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:02:25.934 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (31m) 4m ago 43m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:02:25.934 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (31m) 4m ago 43m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:02:25.934 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (30m) 4m ago 42m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:02:25.934 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (29m) 5m ago 42m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:02:25.935 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (29m) 5m ago 41m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:02:25.935 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (28m) 5m ago 41m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:02:25.935 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (27m) 5m ago 40m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:02:25.935 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (26m) 5m ago 39m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:02:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:25 smithi121 bash[36902]: cluster 2022-04-22T18:02:24.754696+0000 mgr.y (mgr.34107) 1289 : cluster [DBG] pgmap v1871: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:25 smithi121 bash[36902]: audit 2022-04-22T18:02:25.343030+0000 mon.a (mon.0) 2816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:25 smithi121 bash[36902]: audit 2022-04-22T18:02:25.731715+0000 mon.a (mon.0) 2817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[45495]: cluster 2022-04-22T18:02:24.754696+0000 mgr.y (mgr.34107) 1289 : cluster [DBG] pgmap v1871: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[45495]: audit 2022-04-22T18:02:25.343030+0000 mon.a (mon.0) 2816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[45495]: audit 2022-04-22T18:02:25.731715+0000 mon.a (mon.0) 2817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[46934]: cluster 2022-04-22T18:02:24.754696+0000 mgr.y (mgr.34107) 1289 : cluster [DBG] pgmap v1871: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:02:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[46934]: audit 2022-04-22T18:02:25.343030+0000 mon.a (mon.0) 2816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:25 smithi115 bash[46934]: audit 2022-04-22T18:02:25.731715+0000 mon.a (mon.0) 2817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:02:26.346 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:26.347 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:02:26.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:02:26.348 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:02:26.348 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:02:26.584 INFO:tasks.workunit.client.0.smithi115.stdout: 920: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:26.585 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Twd8xSe-6GVE1Eb off=3305686 len=1886957 2022-04-22T18:02:26.591 INFO:tasks.workunit.client.0.smithi115.stdout:op 4670 completed, throughput=5MB/sec 2022-04-22T18:02:26.591 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zRjG8QDK74N2a-B off=438438 len=529091 2022-04-22T18:02:26.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 4671 completed, throughput=5MB/sec 2022-04-22T18:02:26.593 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_g6kyfVmpN0q5C8 off=8340235 len=1723804 2022-04-22T18:02:26.598 INFO:tasks.workunit.client.0.smithi115.stdout:op 4672 completed, throughput=5MB/sec 2022-04-22T18:02:26.598 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rCwmWqPpfAhm-t0 off=3524412 len=1822521 2022-04-22T18:02:26.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 4673 completed, throughput=5MB/sec 2022-04-22T18:02:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:26 smithi121 bash[36902]: audit 2022-04-22T18:02:25.559970+0000 mgr.y (mgr.34107) 1290 : audit [DBG] from='client.45691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:26 smithi121 bash[36902]: audit 2022-04-22T18:02:25.903056+0000 mgr.y (mgr.34107) 1291 : audit [DBG] from='client.35757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:26 smithi121 bash[36902]: audit 2022-04-22T18:02:26.338234+0000 mon.a (mon.0) 2818 : audit [DBG] from='client.? 172.21.15.115:0/3773802904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[45495]: audit 2022-04-22T18:02:25.559970+0000 mgr.y (mgr.34107) 1290 : audit [DBG] from='client.45691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[45495]: audit 2022-04-22T18:02:25.903056+0000 mgr.y (mgr.34107) 1291 : audit [DBG] from='client.35757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[45495]: audit 2022-04-22T18:02:26.338234+0000 mon.a (mon.0) 2818 : audit [DBG] from='client.? 172.21.15.115:0/3773802904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[46934]: audit 2022-04-22T18:02:25.559970+0000 mgr.y (mgr.34107) 1290 : audit [DBG] from='client.45691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[46934]: audit 2022-04-22T18:02:25.903056+0000 mgr.y (mgr.34107) 1291 : audit [DBG] from='client.35757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:26 smithi115 bash[46934]: audit 2022-04-22T18:02:26.338234+0000 mon.a (mon.0) 2818 : audit [DBG] from='client.? 172.21.15.115:0/3773802904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:27.602 INFO:tasks.workunit.client.0.smithi115.stdout: 921: throughput=5MB/sec pending data=0 2022-04-22T18:02:27.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u1-SJaWsTQFbEDW off=1627011 len=1277016 2022-04-22T18:02:27.606 INFO:tasks.workunit.client.0.smithi115.stdout:op 4674 completed, throughput=5MB/sec 2022-04-22T18:02:27.606 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xfOqplxVGsdlPMS off=1106799 len=1706376 2022-04-22T18:02:27.617 INFO:tasks.workunit.client.0.smithi115.stdout:op 4675 completed, throughput=5MB/sec 2022-04-22T18:02:27.617 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RCeLr43fdj-fzme off=10138259 len=1081918 2022-04-22T18:02:27.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 4676 completed, throughput=5MB/sec 2022-04-22T18:02:27.619 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_8BeQTBW7Igv02d off=11486822 len=138213 2022-04-22T18:02:27.619 INFO:tasks.workunit.client.0.smithi115.stdout:op 4677 completed, throughput=5MB/sec 2022-04-22T18:02:27.620 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pQg-l25MEI3ERFy off=1980370 len=377133 2022-04-22T18:02:27.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 4678 completed, throughput=5MB/sec 2022-04-22T18:02:28.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:27 smithi121 bash[36902]: cluster 2022-04-22T18:02:26.755806+0000 mgr.y (mgr.34107) 1292 : cluster [DBG] pgmap v1872: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:02:28.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:27 smithi121 bash[36902]: audit 2022-04-22T18:02:27.730825+0000 mon.a (mon.0) 2819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:27 smithi115 bash[45495]: cluster 2022-04-22T18:02:26.755806+0000 mgr.y (mgr.34107) 1292 : cluster [DBG] pgmap v1872: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:02:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:27 smithi115 bash[45495]: audit 2022-04-22T18:02:27.730825+0000 mon.a (mon.0) 2819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:27 smithi115 bash[46934]: cluster 2022-04-22T18:02:26.755806+0000 mgr.y (mgr.34107) 1292 : cluster [DBG] pgmap v1872: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:02:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:27 smithi115 bash[46934]: audit 2022-04-22T18:02:27.730825+0000 mon.a (mon.0) 2819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:28.621 INFO:tasks.workunit.client.0.smithi115.stdout: 922: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:28.621 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UQVaj6hUJaB54JA off=3487808 len=1256570 2022-04-22T18:02:28.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 4679 completed, throughput=5MB/sec 2022-04-22T18:02:28.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4YU_6cz7g5khjTF off=6969914 len=1342955 2022-04-22T18:02:28.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 4680 completed, throughput=5MB/sec 2022-04-22T18:02:28.629 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gEV8iC52tiULi5H off=12723229 len=416771 2022-04-22T18:02:28.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 4681 completed, throughput=5MB/sec 2022-04-22T18:02:28.635 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-44Ll0kXZF3Pe1R1 off=8158007 len=1625875 2022-04-22T18:02:28.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 4682 completed, throughput=5MB/sec 2022-04-22T18:02:28.645 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4Vw1G80ZfIQnAsd off=7942466 len=1296786 2022-04-22T18:02:28.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4683 completed, throughput=5MB/sec 2022-04-22T18:02:29.651 INFO:tasks.workunit.client.0.smithi115.stdout: 923: throughput=5MB/sec pending data=0 2022-04-22T18:02:29.651 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gh9Y7ZxkamVUbQ9 off=1363691 len=1657036 2022-04-22T18:02:29.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 4684 completed, throughput=5MB/sec 2022-04-22T18:02:29.655 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gR2lPk3aAszjEwz off=3811448 len=1828387 2022-04-22T18:02:29.658 INFO:tasks.workunit.client.0.smithi115.stdout:op 4685 completed, throughput=5MB/sec 2022-04-22T18:02:29.658 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cyTHlcT43k2DZnL off=17667829 len=713476 2022-04-22T18:02:29.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 4686 completed, throughput=5MB/sec 2022-04-22T18:02:29.666 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BjWmJQrsY-gWM8k off=4106245 len=723424 2022-04-22T18:02:29.667 INFO:tasks.workunit.client.0.smithi115.stdout:op 4687 completed, throughput=5MB/sec 2022-04-22T18:02:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:29 smithi121 bash[36902]: cluster 2022-04-22T18:02:28.756993+0000 mgr.y (mgr.34107) 1293 : cluster [DBG] pgmap v1873: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:29 smithi115 bash[45495]: cluster 2022-04-22T18:02:28.756993+0000 mgr.y (mgr.34107) 1293 : cluster [DBG] pgmap v1873: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:30.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:29 smithi115 bash[46934]: cluster 2022-04-22T18:02:28.756993+0000 mgr.y (mgr.34107) 1293 : cluster [DBG] pgmap v1873: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:30.667 INFO:tasks.workunit.client.0.smithi115.stdout: 924: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:30.668 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yh9IGpbwstfRWrJ off=1430659 len=1554532 2022-04-22T18:02:30.677 INFO:tasks.workunit.client.0.smithi115.stdout:op 4688 completed, throughput=5MB/sec 2022-04-22T18:02:30.678 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f742rbVJDYUdMkz off=18437800 len=1869298 2022-04-22T18:02:30.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 4689 completed, throughput=5MB/sec 2022-04-22T18:02:30.687 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z0NyUQigqS8hnGC off=1337420 len=963581 2022-04-22T18:02:30.689 INFO:tasks.workunit.client.0.smithi115.stdout:op 4690 completed, throughput=5MB/sec 2022-04-22T18:02:30.689 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HYfPpoIcbv1O3Cd off=4290656 len=702549 2022-04-22T18:02:30.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 4691 completed, throughput=5MB/sec 2022-04-22T18:02:30.694 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R_S3fz4omWEDajC off=1737985 len=714410 2022-04-22T18:02:30.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 4692 completed, throughput=5MB/sec 2022-04-22T18:02:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:30 smithi121 bash[36902]: audit 2022-04-22T18:02:30.346058+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:30 smithi121 bash[36902]: audit 2022-04-22T18:02:30.734775+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:30 smithi115 bash[45495]: audit 2022-04-22T18:02:30.346058+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:30 smithi115 bash[45495]: audit 2022-04-22T18:02:30.734775+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:30 smithi115 bash[46934]: audit 2022-04-22T18:02:30.346058+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:30 smithi115 bash[46934]: audit 2022-04-22T18:02:30.734775+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:31.700 INFO:tasks.workunit.client.0.smithi115.stdout: 925: throughput=5MB/sec pending data=0 2022-04-22T18:02:31.701 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O4eoUEvyPwLGNYj off=12703536 len=1748031 2022-04-22T18:02:31.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 4693 completed, throughput=5MB/sec 2022-04-22T18:02:31.706 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2N6hIKhc3VkjVWv off=4119063 len=494563 2022-04-22T18:02:31.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 4694 completed, throughput=5MB/sec 2022-04-22T18:02:31.709 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2cAeHy2G_mRvJYh off=4581563 len=1458487 2022-04-22T18:02:31.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 4695 completed, throughput=5MB/sec 2022-04-22T18:02:31.717 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1E83sc0FpKFxFDQ off=1325666 len=1531764 2022-04-22T18:02:31.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 4696 completed, throughput=5MB/sec 2022-04-22T18:02:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:31 smithi115 bash[45495]: cluster 2022-04-22T18:02:30.758096+0000 mgr.y (mgr.34107) 1294 : cluster [DBG] pgmap v1874: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T18:02:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:31.729Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:31 smithi115 bash[46934]: cluster 2022-04-22T18:02:30.758096+0000 mgr.y (mgr.34107) 1294 : cluster [DBG] pgmap v1874: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T18:02:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:31 smithi121 bash[36902]: cluster 2022-04-22T18:02:30.758096+0000 mgr.y (mgr.34107) 1294 : cluster [DBG] pgmap v1874: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2022-04-22T18:02:32.723 INFO:tasks.workunit.client.0.smithi115.stdout: 926: throughput=5MB/sec pending data=0 2022-04-22T18:02:32.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2036Z4wpK-0kAzh off=3893215 len=803248 2022-04-22T18:02:32.726 INFO:tasks.workunit.client.0.smithi115.stdout:op 4697 completed, throughput=5MB/sec 2022-04-22T18:02:32.726 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ucNIkvsKflRNN3P off=2502274 len=1521046 2022-04-22T18:02:32.729 INFO:tasks.workunit.client.0.smithi115.stdout:op 4698 completed, throughput=5MB/sec 2022-04-22T18:02:32.729 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mYCKU2xbzH39Usv off=11959613 len=1808647 2022-04-22T18:02:32.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 4699 completed, throughput=5MB/sec 2022-04-22T18:02:32.737 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PdljHhQ3kQW7x5B off=15905434 len=986374 2022-04-22T18:02:32.737 INFO:tasks.workunit.client.0.smithi115.stdout:op 4700 completed, throughput=5MB/sec 2022-04-22T18:02:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:32 smithi121 bash[36902]: audit 2022-04-22T18:02:32.734159+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:33.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:32 smithi115 bash[45495]: audit 2022-04-22T18:02:32.734159+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:33.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:32 smithi115 bash[46934]: audit 2022-04-22T18:02:32.734159+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:33.738 INFO:tasks.workunit.client.0.smithi115.stdout: 927: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:33.738 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hMF1e2fhjMBLq7l off=693005 len=1851126 2022-04-22T18:02:33.742 INFO:tasks.workunit.client.0.smithi115.stdout:op 4701 completed, throughput=5MB/sec 2022-04-22T18:02:33.742 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3hvMkfZaqj3Iwud off=820829 len=522229 2022-04-22T18:02:33.749 INFO:tasks.workunit.client.0.smithi115.stdout:op 4702 completed, throughput=5MB/sec 2022-04-22T18:02:33.749 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NmTdFnVOYcWZCQw off=4582072 len=744256 2022-04-22T18:02:33.751 INFO:tasks.workunit.client.0.smithi115.stdout:op 4703 completed, throughput=5MB/sec 2022-04-22T18:02:33.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c4Ylq-U8iFsres8 off=881539 len=506257 2022-04-22T18:02:33.752 INFO:tasks.workunit.client.0.smithi115.stdout:op 4704 completed, throughput=5MB/sec 2022-04-22T18:02:33.752 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lsp8RfpP4JoQQlD off=2742701 len=1342781 2022-04-22T18:02:33.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 4705 completed, throughput=5MB/sec 2022-04-22T18:02:33.759 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xe8PC-AK-Tdcj7W off=15951348 len=1087438 2022-04-22T18:02:33.760 INFO:tasks.workunit.client.0.smithi115.stdout:op 4706 completed, throughput=5MB/sec 2022-04-22T18:02:33.896 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:33.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:33 smithi121 bash[36902]: cluster 2022-04-22T18:02:32.759184+0000 mgr.y (mgr.34107) 1295 : cluster [DBG] pgmap v1875: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:34.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:33 smithi115 bash[45495]: cluster 2022-04-22T18:02:32.759184+0000 mgr.y (mgr.34107) 1295 : cluster [DBG] pgmap v1875: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:33 smithi115 bash[46934]: cluster 2022-04-22T18:02:32.759184+0000 mgr.y (mgr.34107) 1295 : cluster [DBG] pgmap v1875: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:34.761 INFO:tasks.workunit.client.0.smithi115.stdout: 928: throughput=5MB/sec pending data=0 2022-04-22T18:02:34.761 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qXWHBuwkNnKDpjJ off=3983746 len=350555 2022-04-22T18:02:34.766 INFO:tasks.workunit.client.0.smithi115.stdout:op 4707 completed, throughput=5MB/sec 2022-04-22T18:02:34.766 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zmg_RaPaYGTIOBx off=9068108 len=1862685 2022-04-22T18:02:34.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 4708 completed, throughput=5MB/sec 2022-04-22T18:02:34.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gHSnnrNLeYlwUe7 off=3149350 len=381518 2022-04-22T18:02:34.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 4709 completed, throughput=5MB/sec 2022-04-22T18:02:34.775 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-198NG9uYXOLTtCC off=15673884 len=270027 2022-04-22T18:02:34.782 INFO:tasks.workunit.client.0.smithi115.stdout:op 4710 completed, throughput=5MB/sec 2022-04-22T18:02:34.782 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F5i9XcgCPs8ANBm off=4271125 len=1859622 2022-04-22T18:02:34.789 INFO:tasks.workunit.client.0.smithi115.stdout:op 4711 completed, throughput=5MB/sec 2022-04-22T18:02:35.789 INFO:tasks.workunit.client.0.smithi115.stdout: 929: throughput=5MB/sec pending data=0 2022-04-22T18:02:35.789 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6WjR-6pQqUO_Pdx off=18113179 len=163341 2022-04-22T18:02:35.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 4712 completed, throughput=5MB/sec 2022-04-22T18:02:35.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rDmYxB_3SIWhhqi off=6692051 len=807437 2022-04-22T18:02:35.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 4713 completed, throughput=5MB/sec 2022-04-22T18:02:35.793 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PR9b05joYgMotBC off=9319523 len=396504 2022-04-22T18:02:35.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 4714 completed, throughput=5MB/sec 2022-04-22T18:02:35.794 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kujpBgHAmCjrD_- off=5322253 len=1750355 2022-04-22T18:02:35.798 INFO:tasks.workunit.client.0.smithi115.stdout:op 4715 completed, throughput=5MB/sec 2022-04-22T18:02:35.798 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kGfYIGsr-SKYmUy off=7052265 len=1044349 2022-04-22T18:02:35.806 INFO:tasks.workunit.client.0.smithi115.stdout:op 4716 completed, throughput=5MB/sec 2022-04-22T18:02:35.806 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GlrN0Q7UBIiRmg_ off=11512077 len=1131113 2022-04-22T18:02:35.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 4717 completed, throughput=5MB/sec 2022-04-22T18:02:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:35 smithi121 bash[36902]: cluster 2022-04-22T18:02:34.760369+0000 mgr.y (mgr.34107) 1296 : cluster [DBG] pgmap v1876: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:35 smithi121 bash[36902]: audit 2022-04-22T18:02:35.349235+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:35 smithi121 bash[36902]: audit 2022-04-22T18:02:35.738025+0000 mon.a (mon.0) 2824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[45495]: cluster 2022-04-22T18:02:34.760369+0000 mgr.y (mgr.34107) 1296 : cluster [DBG] pgmap v1876: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[45495]: audit 2022-04-22T18:02:35.349235+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[45495]: audit 2022-04-22T18:02:35.738025+0000 mon.a (mon.0) 2824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[46934]: cluster 2022-04-22T18:02:34.760369+0000 mgr.y (mgr.34107) 1296 : cluster [DBG] pgmap v1876: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:02:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[46934]: audit 2022-04-22T18:02:35.349235+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:35 smithi115 bash[46934]: audit 2022-04-22T18:02:35.738025+0000 mon.a (mon.0) 2824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:36.814 INFO:tasks.workunit.client.0.smithi115.stdout: 930: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:36.814 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SpKZvGJMv1RDT9Y off=1866181 len=1428382 2022-04-22T18:02:36.817 INFO:tasks.workunit.client.0.smithi115.stdout:op 4718 completed, throughput=5MB/sec 2022-04-22T18:02:36.817 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3b0_OsVKJcYHZkm off=12121139 len=427611 2022-04-22T18:02:36.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 4719 completed, throughput=5MB/sec 2022-04-22T18:02:36.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4qZyo86VSDLWjFL off=17972190 len=1449431 2022-04-22T18:02:36.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 4720 completed, throughput=5MB/sec 2022-04-22T18:02:36.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WQU_U4AJELwciJz off=8157652 len=1364644 2022-04-22T18:02:36.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 4721 completed, throughput=5MB/sec 2022-04-22T18:02:36.835 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ySQZIXSKP9oj9q4 off=8429765 len=1375504 2022-04-22T18:02:36.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 4722 completed, throughput=5MB/sec 2022-04-22T18:02:37.840 INFO:tasks.workunit.client.0.smithi115.stdout: 931: throughput=5MB/sec pending data=0 2022-04-22T18:02:37.858 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0EhRkFyWw3-jwB5 off=2146159 len=597446 2022-04-22T18:02:37.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 4723 completed, throughput=5MB/sec 2022-04-22T18:02:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TN58XTuexQ6tH_8 off=5387618 len=1110052 2022-04-22T18:02:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 4724 completed, throughput=5MB/sec 2022-04-22T18:02:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XR3v0aMa3FVWBsC off=333561 len=1645108 2022-04-22T18:02:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 4725 completed, throughput=5MB/sec 2022-04-22T18:02:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nVn5e5FKBRLlVaV off=6758886 len=322672 2022-04-22T18:02:37.861 INFO:tasks.workunit.client.0.smithi115.stdout:op 4726 completed, throughput=5MB/sec 2022-04-22T18:02:37.861 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Sa77TRN6CMZ13rP off=9138403 len=2041879 2022-04-22T18:02:37.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 4727 completed, throughput=5MB/sec 2022-04-22T18:02:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:37 smithi121 bash[36902]: cluster 2022-04-22T18:02:36.761495+0000 mgr.y (mgr.34107) 1297 : cluster [DBG] pgmap v1877: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2022-04-22T18:02:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:37 smithi121 bash[36902]: audit 2022-04-22T18:02:37.737061+0000 mon.a (mon.0) 2825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:37 smithi115 bash[45495]: cluster 2022-04-22T18:02:36.761495+0000 mgr.y (mgr.34107) 1297 : cluster [DBG] pgmap v1877: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2022-04-22T18:02:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:37 smithi115 bash[45495]: audit 2022-04-22T18:02:37.737061+0000 mon.a (mon.0) 2825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:37 smithi115 bash[46934]: cluster 2022-04-22T18:02:36.761495+0000 mgr.y (mgr.34107) 1297 : cluster [DBG] pgmap v1877: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2022-04-22T18:02:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:37 smithi115 bash[46934]: audit 2022-04-22T18:02:37.737061+0000 mon.a (mon.0) 2825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:38.869 INFO:tasks.workunit.client.0.smithi115.stdout: 932: throughput=5MB/sec pending data=0 2022-04-22T18:02:38.870 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bb8aQb3SSvMQ9pU off=9685415 len=996733 2022-04-22T18:02:38.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 4728 completed, throughput=5MB/sec 2022-04-22T18:02:38.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dswEDwZ2UppwSZ9 off=16649254 len=1609165 2022-04-22T18:02:38.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 4729 completed, throughput=5MB/sec 2022-04-22T18:02:38.887 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AtsCC4B-P6OLb5I off=1752563 len=2051267 2022-04-22T18:02:38.894 INFO:tasks.workunit.client.0.smithi115.stdout:op 4730 completed, throughput=5MB/sec 2022-04-22T18:02:39.894 INFO:tasks.workunit.client.0.smithi115.stdout: 933: throughput=5MB/sec pending data=0 2022-04-22T18:02:39.894 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PAqq1xPcPqN8Eyw off=7761503 len=584051 2022-04-22T18:02:39.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 4731 completed, throughput=5MB/sec 2022-04-22T18:02:39.901 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FuL9hQlJ1CGzQkX off=5201084 len=792455 2022-04-22T18:02:39.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 4732 completed, throughput=5MB/sec 2022-04-22T18:02:39.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xN-i0B2OY1EyhB_ off=17492383 len=1962650 2022-04-22T18:02:39.911 INFO:tasks.workunit.client.0.smithi115.stdout:op 4733 completed, throughput=5MB/sec 2022-04-22T18:02:39.911 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_jO2NjothJ2lO1x off=5997839 len=898118 2022-04-22T18:02:39.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 4734 completed, throughput=5MB/sec 2022-04-22T18:02:39.914 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VMSBnbL2alsGOCc off=1462853 len=259085 2022-04-22T18:02:39.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 4735 completed, throughput=5MB/sec 2022-04-22T18:02:39.918 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fch4jz5eQSSF5aK off=3843774 len=1759676 2022-04-22T18:02:39.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 4736 completed, throughput=5MB/sec 2022-04-22T18:02:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:39 smithi121 bash[36902]: cluster 2022-04-22T18:02:38.762656+0000 mgr.y (mgr.34107) 1298 : cluster [DBG] pgmap v1878: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:40.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:39 smithi115 bash[45495]: cluster 2022-04-22T18:02:38.762656+0000 mgr.y (mgr.34107) 1298 : cluster [DBG] pgmap v1878: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:40.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:39 smithi115 bash[46934]: cluster 2022-04-22T18:02:38.762656+0000 mgr.y (mgr.34107) 1298 : cluster [DBG] pgmap v1878: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 4 op/s 2022-04-22T18:02:40.922 INFO:tasks.workunit.client.0.smithi115.stdout: 934: throughput=5MB/sec pending data=0 2022-04-22T18:02:40.922 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XaKmmCcXZcAAinB off=2129114 len=2015588 2022-04-22T18:02:40.931 INFO:tasks.workunit.client.0.smithi115.stdout:op 4737 completed, throughput=5MB/sec 2022-04-22T18:02:40.931 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EyXAgeC6vCIGaWN off=14979457 len=453694 2022-04-22T18:02:40.936 INFO:tasks.workunit.client.0.smithi115.stdout:op 4738 completed, throughput=5MB/sec 2022-04-22T18:02:40.936 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2i5hSiU-GP9lX4g off=2094366 len=155523 2022-04-22T18:02:40.940 INFO:tasks.workunit.client.0.smithi115.stdout:op 4739 completed, throughput=5MB/sec 2022-04-22T18:02:40.940 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LR5A4Y7aNY8fD_p off=16602130 len=1724554 2022-04-22T18:02:40.948 INFO:tasks.workunit.client.0.smithi115.stdout:op 4740 completed, throughput=5MB/sec 2022-04-22T18:02:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:40 smithi121 bash[36902]: audit 2022-04-22T18:02:40.352429+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:41.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:40 smithi121 bash[36902]: audit 2022-04-22T18:02:40.741531+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:41.320 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:40 smithi115 bash[45495]: audit 2022-04-22T18:02:40.352429+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:41.320 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:40 smithi115 bash[45495]: audit 2022-04-22T18:02:40.741531+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:40 smithi115 bash[46934]: audit 2022-04-22T18:02:40.352429+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:40 smithi115 bash[46934]: audit 2022-04-22T18:02:40.741531+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:41.948 INFO:tasks.workunit.client.0.smithi115.stdout: 935: throughput=5MB/sec pending data=0 2022-04-22T18:02:41.948 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WEpNw2Z9aHv7qXl off=14161366 len=698262 2022-04-22T18:02:41.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 4741 completed, throughput=5MB/sec 2022-04-22T18:02:41.951 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nrTE3hItePy6lsi off=16121463 len=549414 2022-04-22T18:02:41.957 INFO:tasks.workunit.client.0.smithi115.stdout:op 4742 completed, throughput=5MB/sec 2022-04-22T18:02:41.957 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TWGXDU-Kj2aVi_c off=12914887 len=32467 2022-04-22T18:02:41.960 INFO:tasks.workunit.client.0.smithi115.stdout:op 4743 completed, throughput=5MB/sec 2022-04-22T18:02:41.960 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-46E-1lys9PTJQd5 off=15778431 len=932866 2022-04-22T18:02:41.963 INFO:tasks.workunit.client.0.smithi115.stdout:op 4744 completed, throughput=5MB/sec 2022-04-22T18:02:41.963 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CU2RsR_JKL-j9PR off=20530706 len=622529 2022-04-22T18:02:41.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 4745 completed, throughput=5MB/sec 2022-04-22T18:02:41.969 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fWITEOEuzQyUwQo off=12766334 len=414634 2022-04-22T18:02:41.974 INFO:tasks.workunit.client.0.smithi115.stdout:op 4746 completed, throughput=5MB/sec 2022-04-22T18:02:41.975 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZRy_Rm7--2XyVxN off=482994 len=1927738 2022-04-22T18:02:41.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 4747 completed, throughput=5MB/sec 2022-04-22T18:02:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:41 smithi115 bash[45495]: cluster 2022-04-22T18:02:40.763682+0000 mgr.y (mgr.34107) 1299 : cluster [DBG] pgmap v1879: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:02:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:41 smithi115 bash[46934]: cluster 2022-04-22T18:02:40.763682+0000 mgr.y (mgr.34107) 1299 : cluster [DBG] pgmap v1879: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:02:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:41.732Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:41 smithi121 bash[36902]: cluster 2022-04-22T18:02:40.763682+0000 mgr.y (mgr.34107) 1299 : cluster [DBG] pgmap v1879: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:02:42.976 INFO:tasks.workunit.client.0.smithi115.stdout: 936: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:42.976 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1UW47XkJhMYzlG1 off=20190627 len=947958 2022-04-22T18:02:42.984 INFO:tasks.workunit.client.0.smithi115.stdout:op 4748 completed, throughput=5MB/sec 2022-04-22T18:02:42.984 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nVCkaCAz5uXGFer off=21743154 len=1762762 2022-04-22T18:02:42.991 INFO:tasks.workunit.client.0.smithi115.stdout:op 4749 completed, throughput=5MB/sec 2022-04-22T18:02:42.992 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-POegqrNM2Yawcdo off=15148448 len=1830354 2022-04-22T18:02:43.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 4750 completed, throughput=5MB/sec 2022-04-22T18:02:43.001 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iCUQN--2aj-pEXO off=3680715 len=1485294 2022-04-22T18:02:43.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 4751 completed, throughput=5MB/sec 2022-04-22T18:02:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:42 smithi121 bash[36902]: audit 2022-04-22T18:02:42.740326+0000 mon.a (mon.0) 2828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:43.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:42 smithi115 bash[46934]: audit 2022-04-22T18:02:42.740326+0000 mon.a (mon.0) 2828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:42 smithi115 bash[45495]: audit 2022-04-22T18:02:42.740326+0000 mon.a (mon.0) 2828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:43.928 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:43.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:44.004 INFO:tasks.workunit.client.0.smithi115.stdout: 937: throughput=5MB/sec pending data=0 2022-04-22T18:02:44.004 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--5EAtxjlHZjQdIB off=888843 len=691567 2022-04-22T18:02:44.006 INFO:tasks.workunit.client.0.smithi115.stdout:op 4752 completed, throughput=5MB/sec 2022-04-22T18:02:44.006 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zNDzfmdlf5pIBNc off=19483063 len=1523562 2022-04-22T18:02:44.011 INFO:tasks.workunit.client.0.smithi115.stdout:op 4753 completed, throughput=5MB/sec 2022-04-22T18:02:44.012 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_8A4emTRFbuu6j3 off=1828269 len=694905 2022-04-22T18:02:44.018 INFO:tasks.workunit.client.0.smithi115.stdout:op 4754 completed, throughput=5MB/sec 2022-04-22T18:02:44.018 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iYN_athp-jpkmH6 off=4114896 len=368934 2022-04-22T18:02:44.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 4755 completed, throughput=5MB/sec 2022-04-22T18:02:44.022 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rT5XmL0CxXt7f_i off=15144299 len=368370 2022-04-22T18:02:44.025 INFO:tasks.workunit.client.0.smithi115.stdout:op 4756 completed, throughput=5MB/sec 2022-04-22T18:02:44.025 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jlmgpwqh6cPd2Fg off=8115070 len=2069547 2022-04-22T18:02:44.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 4757 completed, throughput=5MB/sec 2022-04-22T18:02:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:43 smithi121 bash[36902]: cluster 2022-04-22T18:02:42.764670+0000 mgr.y (mgr.34107) 1300 : cluster [DBG] pgmap v1880: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:44.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:43 smithi115 bash[45495]: cluster 2022-04-22T18:02:42.764670+0000 mgr.y (mgr.34107) 1300 : cluster [DBG] pgmap v1880: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:44.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:43 smithi115 bash[46934]: cluster 2022-04-22T18:02:42.764670+0000 mgr.y (mgr.34107) 1300 : cluster [DBG] pgmap v1880: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:45.035 INFO:tasks.workunit.client.0.smithi115.stdout: 938: throughput=5MB/sec pending data=0 2022-04-22T18:02:45.035 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jLoJGLd-5D2Ayur off=8123336 len=1147455 2022-04-22T18:02:45.038 INFO:tasks.workunit.client.0.smithi115.stdout:op 4758 completed, throughput=5MB/sec 2022-04-22T18:02:45.038 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dyP9OKCgIvm2yY8 off=2627536 len=509634 2022-04-22T18:02:45.040 INFO:tasks.workunit.client.0.smithi115.stdout:op 4759 completed, throughput=5MB/sec 2022-04-22T18:02:45.040 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_g6kyfVmpN0q5C8 off=7555021 len=1527813 2022-04-22T18:02:45.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 4760 completed, throughput=5MB/sec 2022-04-22T18:02:45.050 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s6DtDyd4xJt0tbV off=12995729 len=698481 2022-04-22T18:02:45.055 INFO:tasks.workunit.client.0.smithi115.stdout:op 4761 completed, throughput=5MB/sec 2022-04-22T18:02:45.055 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dgftm1ihiWSrF0m off=4047620 len=864886 2022-04-22T18:02:45.057 INFO:tasks.workunit.client.0.smithi115.stdout:op 4762 completed, throughput=5MB/sec 2022-04-22T18:02:46.058 INFO:tasks.workunit.client.0.smithi115.stdout: 939: throughput=5MB/sec pending data=0 2022-04-22T18:02:46.058 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u1c-uCfaRUEgKz5 off=9862949 len=1258233 2022-04-22T18:02:46.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 4763 completed, throughput=5MB/sec 2022-04-22T18:02:46.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FbBhfP5NIuWoBo3 off=464471 len=433399 2022-04-22T18:02:46.067 INFO:tasks.workunit.client.0.smithi115.stdout:op 4764 completed, throughput=5MB/sec 2022-04-22T18:02:46.067 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Cb64kDZE7pvbfP6 off=6609195 len=1515541 2022-04-22T18:02:46.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 4765 completed, throughput=5MB/sec 2022-04-22T18:02:46.076 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-88hpdVeHGvGNUQE off=15050116 len=900773 2022-04-22T18:02:46.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 4766 completed, throughput=5MB/sec 2022-04-22T18:02:46.081 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y05DCWR_8vdJBdy off=2220393 len=1074666 2022-04-22T18:02:46.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 4767 completed, throughput=5MB/sec 2022-04-22T18:02:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:45 smithi121 bash[36902]: cluster 2022-04-22T18:02:44.765715+0000 mgr.y (mgr.34107) 1301 : cluster [DBG] pgmap v1881: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:02:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:45 smithi121 bash[36902]: audit 2022-04-22T18:02:45.355577+0000 mon.a (mon.0) 2829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:45 smithi121 bash[36902]: audit 2022-04-22T18:02:45.745405+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[45495]: cluster 2022-04-22T18:02:44.765715+0000 mgr.y (mgr.34107) 1301 : cluster [DBG] pgmap v1881: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:02:46.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[45495]: audit 2022-04-22T18:02:45.355577+0000 mon.a (mon.0) 2829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:46.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[45495]: audit 2022-04-22T18:02:45.745405+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:46.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[46934]: cluster 2022-04-22T18:02:44.765715+0000 mgr.y (mgr.34107) 1301 : cluster [DBG] pgmap v1881: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:02:46.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[46934]: audit 2022-04-22T18:02:45.355577+0000 mon.a (mon.0) 2829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:46.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:45 smithi115 bash[46934]: audit 2022-04-22T18:02:45.745405+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:47.084 INFO:tasks.workunit.client.0.smithi115.stdout: 940: throughput=5MB/sec pending data=0 2022-04-22T18:02:47.085 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wMnR7ToTmO9FTHU off=665302 len=879969 2022-04-22T18:02:47.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 4768 completed, throughput=5MB/sec 2022-04-22T18:02:47.092 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MpIrKWRSqqM99Bk off=14755164 len=886423 2022-04-22T18:02:47.098 INFO:tasks.workunit.client.0.smithi115.stdout:op 4769 completed, throughput=5MB/sec 2022-04-22T18:02:47.099 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z4jUak-zUuOK5XV off=0 len=1109277 2022-04-22T18:02:47.101 INFO:tasks.workunit.client.0.smithi115.stdout:op 4770 completed, throughput=5MB/sec 2022-04-22T18:02:47.101 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O1OyWj7tQU3yGsw off=2088233 len=1784633 2022-04-22T18:02:47.105 INFO:tasks.workunit.client.0.smithi115.stdout:op 4771 completed, throughput=5MB/sec 2022-04-22T18:02:47.106 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C0GpbDSJHnjlGxe off=368601 len=49190 2022-04-22T18:02:47.106 INFO:tasks.workunit.client.0.smithi115.stdout:op 4772 completed, throughput=5MB/sec 2022-04-22T18:02:47.106 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9J0rPCuyhpKk3WW off=6673967 len=1824338 2022-04-22T18:02:47.115 INFO:tasks.workunit.client.0.smithi115.stdout:op 4773 completed, throughput=5MB/sec 2022-04-22T18:02:48.116 INFO:tasks.workunit.client.0.smithi115.stdout: 941: throughput=5MB/sec pending data=0 2022-04-22T18:02:48.117 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cImQzfhiuexLv3H off=2336583 len=134055 2022-04-22T18:02:48.120 INFO:tasks.workunit.client.0.smithi115.stdout:op 4774 completed, throughput=5MB/sec 2022-04-22T18:02:48.120 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S6Be3G0ZnF-LzxM off=5366979 len=757234 2022-04-22T18:02:48.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 4775 completed, throughput=5MB/sec 2022-04-22T18:02:48.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7b4eAYY7rcGliA6 off=14673506 len=920870 2022-04-22T18:02:48.134 INFO:tasks.workunit.client.0.smithi115.stdout:op 4776 completed, throughput=5MB/sec 2022-04-22T18:02:48.134 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-npo-94aWDPfKVwF off=10893574 len=488969 2022-04-22T18:02:48.138 INFO:tasks.workunit.client.0.smithi115.stdout:op 4777 completed, throughput=5MB/sec 2022-04-22T18:02:48.138 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7rTx1tiEhbLzmze off=14712256 len=900676 2022-04-22T18:02:48.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 4778 completed, throughput=5MB/sec 2022-04-22T18:02:48.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_HmkMBl1NrLXQ4R off=1395753 len=970198 2022-04-22T18:02:48.146 INFO:tasks.workunit.client.0.smithi115.stdout:op 4779 completed, throughput=5MB/sec 2022-04-22T18:02:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:47 smithi121 bash[36902]: cluster 2022-04-22T18:02:46.766802+0000 mgr.y (mgr.34107) 1302 : cluster [DBG] pgmap v1882: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:47 smithi121 bash[36902]: audit 2022-04-22T18:02:47.743384+0000 mon.a (mon.0) 2831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:47 smithi115 bash[45495]: cluster 2022-04-22T18:02:46.766802+0000 mgr.y (mgr.34107) 1302 : cluster [DBG] pgmap v1882: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:47 smithi115 bash[45495]: audit 2022-04-22T18:02:47.743384+0000 mon.a (mon.0) 2831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:47 smithi115 bash[46934]: cluster 2022-04-22T18:02:46.766802+0000 mgr.y (mgr.34107) 1302 : cluster [DBG] pgmap v1882: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:47 smithi115 bash[46934]: audit 2022-04-22T18:02:47.743384+0000 mon.a (mon.0) 2831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:49.147 INFO:tasks.workunit.client.0.smithi115.stdout: 942: throughput=4.99MB/sec pending data=0 2022-04-22T18:02:49.147 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q86VIhyew8CEJFz off=603928 len=1195292 2022-04-22T18:02:49.154 INFO:tasks.workunit.client.0.smithi115.stdout:op 4780 completed, throughput=5MB/sec 2022-04-22T18:02:49.154 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-98fe_HkTzKla8JX off=15184104 len=998581 2022-04-22T18:02:49.158 INFO:tasks.workunit.client.0.smithi115.stdout:op 4781 completed, throughput=5MB/sec 2022-04-22T18:02:49.158 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yMrjEc7qfL0xpfU off=2038866 len=1619473 2022-04-22T18:02:49.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 4782 completed, throughput=5MB/sec 2022-04-22T18:02:49.161 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CcgY54l-zpr9bOG off=7902877 len=1769851 2022-04-22T18:02:49.170 INFO:tasks.workunit.client.0.smithi115.stdout:op 4783 completed, throughput=5MB/sec 2022-04-22T18:02:50.172 INFO:tasks.workunit.client.0.smithi115.stdout: 943: throughput=5MB/sec pending data=0 2022-04-22T18:02:50.172 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zyu0f7WrRI5sGV- off=2076661 len=887613 2022-04-22T18:02:50.173 INFO:tasks.workunit.client.0.smithi115.stdout:op 4784 completed, throughput=5MB/sec 2022-04-22T18:02:50.173 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uP1cNCo2ngMfpn6 off=2909450 len=1608696 2022-04-22T18:02:50.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 4785 completed, throughput=5MB/sec 2022-04-22T18:02:50.178 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vmAIM8fMDwNJ0XO off=6134819 len=1266685 2022-04-22T18:02:50.181 INFO:tasks.workunit.client.0.smithi115.stdout:op 4786 completed, throughput=5MB/sec 2022-04-22T18:02:50.181 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X6hBaYXNya8fK7Y off=900114 len=1655772 2022-04-22T18:02:50.184 INFO:tasks.workunit.client.0.smithi115.stdout:op 4787 completed, throughput=5MB/sec 2022-04-22T18:02:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:49 smithi121 bash[36902]: cluster 2022-04-22T18:02:48.767994+0000 mgr.y (mgr.34107) 1303 : cluster [DBG] pgmap v1883: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:49 smithi115 bash[45495]: cluster 2022-04-22T18:02:48.767994+0000 mgr.y (mgr.34107) 1303 : cluster [DBG] pgmap v1883: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:49 smithi115 bash[46934]: cluster 2022-04-22T18:02:48.767994+0000 mgr.y (mgr.34107) 1303 : cluster [DBG] pgmap v1883: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:02:51.185 INFO:tasks.workunit.client.0.smithi115.stdout: 944: throughput=5MB/sec pending data=0 2022-04-22T18:02:51.185 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PaCB--o2CNpgkc3 off=4758970 len=1212834 2022-04-22T18:02:51.188 INFO:tasks.workunit.client.0.smithi115.stdout:op 4788 completed, throughput=5MB/sec 2022-04-22T18:02:51.188 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9YZim91vnwd45l2 off=852812 len=1087006 2022-04-22T18:02:51.196 INFO:tasks.workunit.client.0.smithi115.stdout:op 4789 completed, throughput=5MB/sec 2022-04-22T18:02:51.196 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mI7Dox0DsoKdkFm off=4616806 len=338734 2022-04-22T18:02:51.198 INFO:tasks.workunit.client.0.smithi115.stdout:op 4790 completed, throughput=5MB/sec 2022-04-22T18:02:51.198 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VFaNa-ea9OChYwS off=3795567 len=1159031 2022-04-22T18:02:51.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 4791 completed, throughput=5MB/sec 2022-04-22T18:02:51.204 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7KOQ4XEQ8ZTdudu off=13863137 len=468421 2022-04-22T18:02:51.207 INFO:tasks.workunit.client.0.smithi115.stdout:op 4792 completed, throughput=5MB/sec 2022-04-22T18:02:51.207 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Lpp40ID9gEceRiS off=1766047 len=1350284 2022-04-22T18:02:51.213 INFO:tasks.workunit.client.0.smithi115.stdout:op 4793 completed, throughput=5MB/sec 2022-04-22T18:02:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:50 smithi121 bash[36902]: audit 2022-04-22T18:02:50.358966+0000 mon.a (mon.0) 2832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:51.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:50 smithi121 bash[36902]: audit 2022-04-22T18:02:50.748832+0000 mon.a (mon.0) 2833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:51.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:50 smithi115 bash[45495]: audit 2022-04-22T18:02:50.358966+0000 mon.a (mon.0) 2832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:51.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:50 smithi115 bash[45495]: audit 2022-04-22T18:02:50.748832+0000 mon.a (mon.0) 2833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:51.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:50 smithi115 bash[46934]: audit 2022-04-22T18:02:50.358966+0000 mon.a (mon.0) 2832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:51.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:50 smithi115 bash[46934]: audit 2022-04-22T18:02:50.748832+0000 mon.a (mon.0) 2833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:51 smithi121 bash[36902]: cluster 2022-04-22T18:02:50.769117+0000 mgr.y (mgr.34107) 1304 : cluster [DBG] pgmap v1884: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:02:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:51 smithi121 bash[36902]: audit 2022-04-22T18:02:51.355219+0000 mon.a (mon.0) 2834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:02:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:51 smithi121 bash[36902]: audit 2022-04-22T18:02:51.358313+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:02:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:51 smithi121 bash[36902]: audit 2022-04-22T18:02:51.360076+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:02:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:51 smithi121 bash[36902]: audit 2022-04-22T18:02:51.723229+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:02:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:51.736Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[45495]: cluster 2022-04-22T18:02:50.769117+0000 mgr.y (mgr.34107) 1304 : cluster [DBG] pgmap v1884: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:02:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[45495]: audit 2022-04-22T18:02:51.355219+0000 mon.a (mon.0) 2834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[45495]: audit 2022-04-22T18:02:51.358313+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[45495]: audit 2022-04-22T18:02:51.360076+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[45495]: audit 2022-04-22T18:02:51.723229+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[46934]: cluster 2022-04-22T18:02:50.769117+0000 mgr.y (mgr.34107) 1304 : cluster [DBG] pgmap v1884: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[46934]: audit 2022-04-22T18:02:51.355219+0000 mon.a (mon.0) 2834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[46934]: audit 2022-04-22T18:02:51.358313+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[46934]: audit 2022-04-22T18:02:51.360076+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:02:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:51 smithi115 bash[46934]: audit 2022-04-22T18:02:51.723229+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:02:52.213 INFO:tasks.workunit.client.0.smithi115.stdout: 945: throughput=5MB/sec pending data=0 2022-04-22T18:02:52.213 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l-NpUtQ_UOLs7Jy off=271896 len=242476 2022-04-22T18:02:52.218 INFO:tasks.workunit.client.0.smithi115.stdout:op 4794 completed, throughput=5MB/sec 2022-04-22T18:02:52.218 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AR_YZGCDlFePr_9 off=3551377 len=417428 2022-04-22T18:02:52.221 INFO:tasks.workunit.client.0.smithi115.stdout:op 4795 completed, throughput=5MB/sec 2022-04-22T18:02:52.221 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DEQUrtc0S6bMUpY off=108752 len=234985 2022-04-22T18:02:52.221 INFO:tasks.workunit.client.0.smithi115.stdout:op 4796 completed, throughput=5MB/sec 2022-04-22T18:02:52.221 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AXu5eEqkYS_S5sG off=4950582 len=1450228 2022-04-22T18:02:52.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 4797 completed, throughput=5MB/sec 2022-04-22T18:02:52.229 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ak1suP4tgQwpsZb off=7626977 len=7070 2022-04-22T18:02:52.230 INFO:tasks.workunit.client.0.smithi115.stdout:op 4798 completed, throughput=5MB/sec 2022-04-22T18:02:52.230 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zye_xDwokYeAUah off=3322152 len=14915 2022-04-22T18:02:52.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 4799 completed, throughput=5MB/sec 2022-04-22T18:02:52.231 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_rYaorhzTqZY5DL off=2134592 len=172837 2022-04-22T18:02:52.232 INFO:tasks.workunit.client.0.smithi115.stdout:op 4800 completed, throughput=5MB/sec 2022-04-22T18:02:52.232 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5PX_RYGahs0jjSm off=1371655 len=879620 2022-04-22T18:02:52.234 INFO:tasks.workunit.client.0.smithi115.stdout:op 4801 completed, throughput=5MB/sec 2022-04-22T18:02:52.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b09eR3VMeOwaHD0 off=7264300 len=549353 2022-04-22T18:02:52.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 4802 completed, throughput=5MB/sec 2022-04-22T18:02:52.236 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-25AwAiXSUNGw8AJ off=6988704 len=1833130 2022-04-22T18:02:52.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 4803 completed, throughput=5MB/sec 2022-04-22T18:02:53.246 INFO:tasks.workunit.client.0.smithi115.stdout: 946: throughput=5MB/sec pending data=0 2022-04-22T18:02:53.247 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tXG4tys6zg4QTWE off=4718974 len=2063477 2022-04-22T18:02:53.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 4804 completed, throughput=5MB/sec 2022-04-22T18:02:53.257 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8x1GYeKog6liMqf off=3600927 len=1365757 2022-04-22T18:02:53.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 4805 completed, throughput=5MB/sec 2022-04-22T18:02:53.259 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ABTrIZ0m2se5Lcr off=14552863 len=36286 2022-04-22T18:02:53.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 4806 completed, throughput=5MB/sec 2022-04-22T18:02:53.263 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mHIZk1LtmZGMOu5 off=5974747 len=216373 2022-04-22T18:02:53.264 INFO:tasks.workunit.client.0.smithi115.stdout:op 4807 completed, throughput=5MB/sec 2022-04-22T18:02:53.264 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vuyy_BXZdIln92K off=17572611 len=1912107 2022-04-22T18:02:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:52 smithi121 bash[36902]: audit 2022-04-22T18:02:52.746578+0000 mon.a (mon.0) 2838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:53.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 4808 completed, throughput=5MB/sec 2022-04-22T18:02:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:52 smithi115 bash[45495]: audit 2022-04-22T18:02:52.746578+0000 mon.a (mon.0) 2838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:52 smithi115 bash[46934]: audit 2022-04-22T18:02:52.746578+0000 mon.a (mon.0) 2838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:53.956 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:02:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:02:53.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:02:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:53 smithi121 bash[36902]: cluster 2022-04-22T18:02:52.770197+0000 mgr.y (mgr.34107) 1305 : cluster [DBG] pgmap v1885: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:02:54.266 INFO:tasks.workunit.client.0.smithi115.stdout: 947: throughput=5MB/sec pending data=0 2022-04-22T18:02:54.266 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HkWpdPM51EIcplo off=4842998 len=341580 2022-04-22T18:02:54.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 4809 completed, throughput=5MB/sec 2022-04-22T18:02:54.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ygq1GauAhFVmhBr off=1212325 len=1389460 2022-04-22T18:02:54.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 4810 completed, throughput=5MB/sec 2022-04-22T18:02:54.278 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-acs7aQFWgXG9r6z off=3895595 len=1758971 2022-04-22T18:02:54.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 4811 completed, throughput=5MB/sec 2022-04-22T18:02:54.281 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3a2cyd7Gx0u1Eee off=5582817 len=1344180 2022-04-22T18:02:54.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 4812 completed, throughput=5MB/sec 2022-04-22T18:02:54.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:53 smithi115 bash[46934]: cluster 2022-04-22T18:02:52.770197+0000 mgr.y (mgr.34107) 1305 : cluster [DBG] pgmap v1885: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:02:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:53 smithi115 bash[45495]: cluster 2022-04-22T18:02:52.770197+0000 mgr.y (mgr.34107) 1305 : cluster [DBG] pgmap v1885: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:02:55.285 INFO:tasks.workunit.client.0.smithi115.stdout: 948: throughput=5MB/sec pending data=0 2022-04-22T18:02:55.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hJBps7J5HeZz7AB off=137874 len=503897 2022-04-22T18:02:55.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 4813 completed, throughput=5MB/sec 2022-04-22T18:02:55.287 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yC3y3u4cxeV0-g0 off=14149113 len=1989569 2022-04-22T18:02:55.299 INFO:tasks.workunit.client.0.smithi115.stdout:op 4814 completed, throughput=5MB/sec 2022-04-22T18:02:55.299 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4TWFAt1uEpAcqhD off=375764 len=1070676 2022-04-22T18:02:55.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 4815 completed, throughput=5MB/sec 2022-04-22T18:02:55.307 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5anzWKX-WKwj5Gm off=1978485 len=1047676 2022-04-22T18:02:55.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 4816 completed, throughput=5MB/sec 2022-04-22T18:02:55.313 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_PZ1vuwQPH6j8w0 off=4145223 len=1932396 2022-04-22T18:02:55.322 INFO:tasks.workunit.client.0.smithi115.stdout:op 4817 completed, throughput=5MB/sec 2022-04-22T18:02:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:55 smithi121 bash[36902]: cluster 2022-04-22T18:02:54.771271+0000 mgr.y (mgr.34107) 1306 : cluster [DBG] pgmap v1886: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:55 smithi121 bash[36902]: audit 2022-04-22T18:02:55.362248+0000 mon.a (mon.0) 2839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:55 smithi121 bash[36902]: audit 2022-04-22T18:02:55.752755+0000 mon.a (mon.0) 2840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[45495]: cluster 2022-04-22T18:02:54.771271+0000 mgr.y (mgr.34107) 1306 : cluster [DBG] pgmap v1886: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[45495]: audit 2022-04-22T18:02:55.362248+0000 mon.a (mon.0) 2839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[45495]: audit 2022-04-22T18:02:55.752755+0000 mon.a (mon.0) 2840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[46934]: cluster 2022-04-22T18:02:54.771271+0000 mgr.y (mgr.34107) 1306 : cluster [DBG] pgmap v1886: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:02:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[46934]: audit 2022-04-22T18:02:55.362248+0000 mon.a (mon.0) 2839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:02:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:55 smithi115 bash[46934]: audit 2022-04-22T18:02:55.752755+0000 mon.a (mon.0) 2840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:02:56.322 INFO:tasks.workunit.client.0.smithi115.stdout: 949: throughput=5MB/sec pending data=0 2022-04-22T18:02:56.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-epl4JVjJ5cczwZj off=13021747 len=1096409 2022-04-22T18:02:56.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 4818 completed, throughput=5MB/sec 2022-04-22T18:02:56.331 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-N4yJlA6wZzdzJ1Z off=2096816 len=1348518 2022-04-22T18:02:56.337 INFO:tasks.workunit.client.0.smithi115.stdout:op 4819 completed, throughput=5MB/sec 2022-04-22T18:02:56.337 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xeqllK_gqiOaKZq off=9265645 len=1366272 2022-04-22T18:02:56.345 INFO:tasks.workunit.client.0.smithi115.stdout:op 4820 completed, throughput=5MB/sec 2022-04-22T18:02:56.715 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (26m) 5m ago 39m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (39m) 6m ago 39m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (27m) 5m ago 40m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (27m) 6m ago 40m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:02:57.069 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (35m) 6m ago 45m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (35m) 5m ago 47m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (34m) 5m ago 47m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (33m) 6m ago 45m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (34m) 5m ago 45m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (27m) 5m ago 40m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (27m) 6m ago 40m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (32m) 5m ago 44m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (32m) 5m ago 44m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:02:57.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (31m) 5m ago 43m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (31m) 5m ago 43m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (30m) 6m ago 42m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (29m) 6m ago 42m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (29m) 6m ago 41m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (28m) 6m ago 41m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:02:57.071 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (26m) 6m ago 40m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:02:57.345 INFO:tasks.workunit.client.0.smithi115.stdout: 950: throughput=5MB/sec pending data=0 2022-04-22T18:02:57.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nXLVSJ15JKHho-B off=4175953 len=374643 2022-04-22T18:02:57.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 4821 completed, throughput=5MB/sec 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wN8yPp8g-3Y9krn off=16098146 len=878931 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 4822 completed, throughput=5MB/sec 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-12MXqQEYtTGUtF9 off=2413941 len=1893319 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 4823 completed, throughput=5MB/sec 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eGqer3AWQHAJukk off=1080072 len=178659 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 4824 completed, throughput=5MB/sec 2022-04-22T18:02:57.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iywXQjeG6DWO8ye off=1190082 len=723295 2022-04-22T18:02:57.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 4825 completed, throughput=5MB/sec 2022-04-22T18:02:57.397 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ADGI7VB885D_bNB off=3400136 len=1445109 2022-04-22T18:02:57.397 INFO:tasks.workunit.client.0.smithi115.stdout:op 4826 completed, throughput=5MB/sec 2022-04-22T18:02:57.488 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:02:57.488 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:02:57.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:02:57.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:57.489 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:02:57.489 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:02:57.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:57.490 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:02:57.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:02:57.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:57.490 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:02:57.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:02:57.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:02:57.491 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:02:57.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:02:57.491 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:02:57.491 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:02:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:57 smithi121 bash[36902]: audit 2022-04-22T18:02:56.691445+0000 mgr.y (mgr.34107) 1307 : audit [DBG] from='client.45709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:57 smithi121 bash[36902]: cluster 2022-04-22T18:02:56.772015+0000 mgr.y (mgr.34107) 1308 : cluster [DBG] pgmap v1887: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:57 smithi121 bash[36902]: audit 2022-04-22T18:02:57.036454+0000 mgr.y (mgr.34107) 1309 : audit [DBG] from='client.35775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:57 smithi121 bash[36902]: audit 2022-04-22T18:02:57.485186+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.115:0/1015663047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:57 smithi121 bash[36902]: audit 2022-04-22T18:02:57.749893+0000 mon.a (mon.0) 2841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[45495]: audit 2022-04-22T18:02:56.691445+0000 mgr.y (mgr.34107) 1307 : audit [DBG] from='client.45709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[45495]: cluster 2022-04-22T18:02:56.772015+0000 mgr.y (mgr.34107) 1308 : cluster [DBG] pgmap v1887: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[45495]: audit 2022-04-22T18:02:57.036454+0000 mgr.y (mgr.34107) 1309 : audit [DBG] from='client.35775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[45495]: audit 2022-04-22T18:02:57.485186+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.115:0/1015663047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[45495]: audit 2022-04-22T18:02:57.749893+0000 mon.a (mon.0) 2841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[46934]: audit 2022-04-22T18:02:56.691445+0000 mgr.y (mgr.34107) 1307 : audit [DBG] from='client.45709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[46934]: cluster 2022-04-22T18:02:56.772015+0000 mgr.y (mgr.34107) 1308 : cluster [DBG] pgmap v1887: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[46934]: audit 2022-04-22T18:02:57.036454+0000 mgr.y (mgr.34107) 1309 : audit [DBG] from='client.35775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[46934]: audit 2022-04-22T18:02:57.485186+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.115:0/1015663047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:02:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:57 smithi115 bash[46934]: audit 2022-04-22T18:02:57.749893+0000 mon.a (mon.0) 2841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:02:58.370 INFO:tasks.workunit.client.0.smithi115.stdout: 951: throughput=5MB/sec pending data=0 2022-04-22T18:02:58.371 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SFxAnU4SYL37tCw off=15595892 len=527202 2022-04-22T18:02:58.373 INFO:tasks.workunit.client.0.smithi115.stdout:op 4827 completed, throughput=5MB/sec 2022-04-22T18:02:58.374 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g6ui7-s42xonLMT off=2338960 len=455779 2022-04-22T18:02:58.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 4828 completed, throughput=5MB/sec 2022-04-22T18:02:58.375 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KK-YmV_2LOUIYFq off=1054398 len=1494827 2022-04-22T18:02:58.385 INFO:tasks.workunit.client.0.smithi115.stdout:op 4829 completed, throughput=5MB/sec 2022-04-22T18:02:58.386 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XGTuSXqP8p3L6yh off=9002613 len=450290 2022-04-22T18:02:58.386 INFO:tasks.workunit.client.0.smithi115.stdout:op 4830 completed, throughput=5MB/sec 2022-04-22T18:02:58.386 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kv1rffMWB_KdROY off=6447029 len=980581 2022-04-22T18:02:58.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 4831 completed, throughput=5MB/sec 2022-04-22T18:02:58.395 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YryeYz7gg0MDWP- off=9524649 len=754007 2022-04-22T18:02:58.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 4832 completed, throughput=5MB/sec 2022-04-22T18:02:58.395 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6Z0d6R048bvIwZG off=3976135 len=468837 2022-04-22T18:02:58.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 4833 completed, throughput=5MB/sec 2022-04-22T18:02:59.399 INFO:tasks.workunit.client.0.smithi115.stdout: 952: throughput=5MB/sec pending data=0 2022-04-22T18:02:59.428 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w8lGscMm0cfuTpR off=13642200 len=1922234 2022-04-22T18:02:59.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 4834 completed, throughput=5MB/sec 2022-04-22T18:02:59.430 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jHZu9aM9ejNvfyd off=2809301 len=97723 2022-04-22T18:02:59.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 4835 completed, throughput=5MB/sec 2022-04-22T18:02:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F7_-EnapBU-nvNT off=11613468 len=1365213 2022-04-22T18:02:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:op 4836 completed, throughput=5MB/sec 2022-04-22T18:02:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R3yIj53tWlyrcNm off=3161124 len=1324248 2022-04-22T18:02:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:op 4837 completed, throughput=5MB/sec 2022-04-22T18:02:59.431 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZVC2jX_SMK9cZRD off=0 len=1062588 2022-04-22T18:02:59.432 INFO:tasks.workunit.client.0.smithi115.stdout:op 4838 completed, throughput=5MB/sec 2022-04-22T18:03:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:02:59 smithi121 bash[36902]: cluster 2022-04-22T18:02:58.773148+0000 mgr.y (mgr.34107) 1310 : cluster [DBG] pgmap v1888: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:00.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:02:59 smithi115 bash[45495]: cluster 2022-04-22T18:02:58.773148+0000 mgr.y (mgr.34107) 1310 : cluster [DBG] pgmap v1888: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:00.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:02:59 smithi115 bash[46934]: cluster 2022-04-22T18:02:58.773148+0000 mgr.y (mgr.34107) 1310 : cluster [DBG] pgmap v1888: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:00.421 INFO:tasks.workunit.client.0.smithi115.stdout: 953: throughput=5MB/sec pending data=0 2022-04-22T18:03:00.421 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DzUa1iCu0ouDgp2 off=1716319 len=716784 2022-04-22T18:03:00.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 4839 completed, throughput=5MB/sec 2022-04-22T18:03:00.428 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v5u3ktGYSjbHqRi off=7106142 len=1694569 2022-04-22T18:03:00.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 4840 completed, throughput=5MB/sec 2022-04-22T18:03:00.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gxSCvCj3uPRLvM3 off=334780 len=1067261 2022-04-22T18:03:00.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 4841 completed, throughput=5MB/sec 2022-04-22T18:03:00.443 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_nSrZpcHUPUGufj off=2071795 len=270437 2022-04-22T18:03:00.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 4842 completed, throughput=5MB/sec 2022-04-22T18:03:00.445 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BCNdqT9fXt4E_f1 off=5245967 len=538553 2022-04-22T18:03:00.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 4843 completed, throughput=5MB/sec 2022-04-22T18:03:00.446 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-spZEapFFzSrYMcF off=9145376 len=1393626 2022-04-22T18:03:00.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 4844 completed, throughput=5MB/sec 2022-04-22T18:03:01.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:00 smithi121 bash[36902]: audit 2022-04-22T18:03:00.365230+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:01.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:00 smithi121 bash[36902]: audit 2022-04-22T18:03:00.756307+0000 mon.a (mon.0) 2843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:00 smithi115 bash[45495]: audit 2022-04-22T18:03:00.365230+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:00 smithi115 bash[45495]: audit 2022-04-22T18:03:00.756307+0000 mon.a (mon.0) 2843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:01.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:00 smithi115 bash[46934]: audit 2022-04-22T18:03:00.365230+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:00 smithi115 bash[46934]: audit 2022-04-22T18:03:00.756307+0000 mon.a (mon.0) 2843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:01.449 INFO:tasks.workunit.client.0.smithi115.stdout: 954: throughput=5MB/sec pending data=0 2022-04-22T18:03:01.449 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SQxcI2DqEa7OPBK off=10407685 len=19320 2022-04-22T18:03:01.453 INFO:tasks.workunit.client.0.smithi115.stdout:op 4845 completed, throughput=5MB/sec 2022-04-22T18:03:01.453 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vpp28L1W0e1vgAJ off=2226143 len=1482280 2022-04-22T18:03:01.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 4846 completed, throughput=5MB/sec 2022-04-22T18:03:01.458 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4XOWYw9vtX8UIJk off=2146760 len=107357 2022-04-22T18:03:01.462 INFO:tasks.workunit.client.0.smithi115.stdout:op 4847 completed, throughput=5MB/sec 2022-04-22T18:03:01.462 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9FcwVZm5xKQ55AS off=282971 len=472318 2022-04-22T18:03:01.463 INFO:tasks.workunit.client.0.smithi115.stdout:op 4848 completed, throughput=5MB/sec 2022-04-22T18:03:01.463 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_itlZJN-mGWn91J off=1866743 len=1974713 2022-04-22T18:03:01.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 4849 completed, throughput=5MB/sec 2022-04-22T18:03:01.467 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XqjgFv9ZB1aDKwu off=13052366 len=901785 2022-04-22T18:03:01.469 INFO:tasks.workunit.client.0.smithi115.stdout:op 4850 completed, throughput=5MB/sec 2022-04-22T18:03:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:01 smithi115 bash[45495]: cluster 2022-04-22T18:03:00.774226+0000 mgr.y (mgr.34107) 1311 : cluster [DBG] pgmap v1889: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:02.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:01.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:01 smithi115 bash[46934]: cluster 2022-04-22T18:03:00.774226+0000 mgr.y (mgr.34107) 1311 : cluster [DBG] pgmap v1889: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:01 smithi121 bash[36902]: cluster 2022-04-22T18:03:00.774226+0000 mgr.y (mgr.34107) 1311 : cluster [DBG] pgmap v1889: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:02.470 INFO:tasks.workunit.client.0.smithi115.stdout: 955: throughput=5MB/sec pending data=0 2022-04-22T18:03:02.470 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pfESrFlV1k3hcY0 off=4281631 len=1861127 2022-04-22T18:03:02.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 4851 completed, throughput=5MB/sec 2022-04-22T18:03:02.480 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ic5o_NrO_k3rDZz off=9030571 len=616133 2022-04-22T18:03:02.484 INFO:tasks.workunit.client.0.smithi115.stdout:op 4852 completed, throughput=5MB/sec 2022-04-22T18:03:02.485 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PpOYkPddeOJJhC5 off=6742777 len=1092254 2022-04-22T18:03:02.487 INFO:tasks.workunit.client.0.smithi115.stdout:op 4853 completed, throughput=5MB/sec 2022-04-22T18:03:02.487 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MgPAoWaUEJFsHyN off=8526379 len=1623610 2022-04-22T18:03:02.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 4854 completed, throughput=5MB/sec 2022-04-22T18:03:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:02 smithi121 bash[36902]: audit 2022-04-22T18:03:02.753139+0000 mon.a (mon.0) 2844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:03.423 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:02 smithi115 bash[45495]: audit 2022-04-22T18:03:02.753139+0000 mon.a (mon.0) 2844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:03.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:02 smithi115 bash[46934]: audit 2022-04-22T18:03:02.753139+0000 mon.a (mon.0) 2844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:03.496 INFO:tasks.workunit.client.0.smithi115.stdout: 956: throughput=5MB/sec pending data=0 2022-04-22T18:03:03.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5XMSnc_wWDAHCUp off=2240425 len=1670503 2022-04-22T18:03:03.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 4855 completed, throughput=5MB/sec 2022-04-22T18:03:03.502 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-viepZNsVyX_VDL9 off=5689980 len=1967485 2022-04-22T18:03:03.511 INFO:tasks.workunit.client.0.smithi115.stdout:op 4856 completed, throughput=5MB/sec 2022-04-22T18:03:03.511 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H5-2SooeQQ_MYE5 off=702517 len=629786 2022-04-22T18:03:03.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 4857 completed, throughput=5MB/sec 2022-04-22T18:03:03.518 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DLSF-6HmG7NK7Ln off=16564805 len=546398 2022-04-22T18:03:03.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 4858 completed, throughput=5MB/sec 2022-04-22T18:03:03.521 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jobxJ7US87n5lOr off=2712959 len=1202719 2022-04-22T18:03:03.526 INFO:tasks.workunit.client.0.smithi115.stdout:op 4859 completed, throughput=5MB/sec 2022-04-22T18:03:03.988 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:03.616Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:03 smithi121 bash[36902]: cluster 2022-04-22T18:03:02.775257+0000 mgr.y (mgr.34107) 1312 : cluster [DBG] pgmap v1890: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:03 smithi115 bash[45495]: cluster 2022-04-22T18:03:02.775257+0000 mgr.y (mgr.34107) 1312 : cluster [DBG] pgmap v1890: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:04.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:03 smithi115 bash[46934]: cluster 2022-04-22T18:03:02.775257+0000 mgr.y (mgr.34107) 1312 : cluster [DBG] pgmap v1890: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:04.526 INFO:tasks.workunit.client.0.smithi115.stdout: 957: throughput=5MB/sec pending data=0 2022-04-22T18:03:04.526 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2y8etj0rcsmNfGO off=17334501 len=1014344 2022-04-22T18:03:04.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 4860 completed, throughput=5MB/sec 2022-04-22T18:03:04.534 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BSKuoXUAklq8ffs off=2962549 len=969374 2022-04-22T18:03:04.535 INFO:tasks.workunit.client.0.smithi115.stdout:op 4861 completed, throughput=5MB/sec 2022-04-22T18:03:04.535 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bHic8CkIrMj-g-q off=15194515 len=1849464 2022-04-22T18:03:04.546 INFO:tasks.workunit.client.0.smithi115.stdout:op 4862 completed, throughput=5MB/sec 2022-04-22T18:03:04.546 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CuXchtFQDZM8hLg off=6172082 len=1724678 2022-04-22T18:03:04.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 4863 completed, throughput=5MB/sec 2022-04-22T18:03:05.553 INFO:tasks.workunit.client.0.smithi115.stdout: 958: throughput=5MB/sec pending data=0 2022-04-22T18:03:05.553 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MCeWQJnK75BMNFQ off=397979 len=1750191 2022-04-22T18:03:05.566 INFO:tasks.workunit.client.0.smithi115.stdout:op 4864 completed, throughput=5MB/sec 2022-04-22T18:03:05.643 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lajK2bTyP6GBA3n off=6286708 len=984184 2022-04-22T18:03:05.643 INFO:tasks.workunit.client.0.smithi115.stdout:op 4865 completed, throughput=5MB/sec 2022-04-22T18:03:05.643 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QyHzNltjYEtyZov off=14192025 len=581834 2022-04-22T18:03:05.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 4866 completed, throughput=5MB/sec 2022-04-22T18:03:05.644 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qoELc027peyPrW3 off=6813850 len=220237 2022-04-22T18:03:05.644 INFO:tasks.workunit.client.0.smithi115.stdout:op 4867 completed, throughput=5MB/sec 2022-04-22T18:03:05.644 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C1n1w6vXKPmn1QU off=1159135 len=111589 2022-04-22T18:03:05.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 4868 completed, throughput=5MB/sec 2022-04-22T18:03:05.645 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aLtD-LpF252-YpC off=21769651 len=1876190 2022-04-22T18:03:05.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 4869 completed, throughput=5MB/sec 2022-04-22T18:03:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:05 smithi121 bash[36902]: cluster 2022-04-22T18:03:04.776390+0000 mgr.y (mgr.34107) 1313 : cluster [DBG] pgmap v1891: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:03:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:05 smithi121 bash[36902]: audit 2022-04-22T18:03:05.368650+0000 mon.a (mon.0) 2845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:05 smithi121 bash[36902]: audit 2022-04-22T18:03:05.759420+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[45495]: cluster 2022-04-22T18:03:04.776390+0000 mgr.y (mgr.34107) 1313 : cluster [DBG] pgmap v1891: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[45495]: audit 2022-04-22T18:03:05.368650+0000 mon.a (mon.0) 2845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[45495]: audit 2022-04-22T18:03:05.759420+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[46934]: cluster 2022-04-22T18:03:04.776390+0000 mgr.y (mgr.34107) 1313 : cluster [DBG] pgmap v1891: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 5 op/s 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[46934]: audit 2022-04-22T18:03:05.368650+0000 mon.a (mon.0) 2845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:05 smithi115 bash[46934]: audit 2022-04-22T18:03:05.759420+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:06.583 INFO:tasks.workunit.client.0.smithi115.stdout: 959: throughput=5MB/sec pending data=0 2022-04-22T18:03:06.583 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hVx8LCnue-d2EpO off=10748642 len=376412 2022-04-22T18:03:06.589 INFO:tasks.workunit.client.0.smithi115.stdout:op 4870 completed, throughput=5MB/sec 2022-04-22T18:03:06.589 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cxwl6v4MI_kPWNH off=1372843 len=1205149 2022-04-22T18:03:06.596 INFO:tasks.workunit.client.0.smithi115.stdout:op 4871 completed, throughput=5MB/sec 2022-04-22T18:03:06.597 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VMSBnbL2alsGOCc off=3289349 len=576173 2022-04-22T18:03:06.601 INFO:tasks.workunit.client.0.smithi115.stdout:op 4872 completed, throughput=5MB/sec 2022-04-22T18:03:06.602 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jLEns0sD17IiguB off=1566046 len=212049 2022-04-22T18:03:06.602 INFO:tasks.workunit.client.0.smithi115.stdout:op 4873 completed, throughput=5MB/sec 2022-04-22T18:03:06.603 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qOvXZkG1s_VOBqL off=19356924 len=1150725 2022-04-22T18:03:06.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 4874 completed, throughput=5MB/sec 2022-04-22T18:03:06.604 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cWvPdzMipi23H4w off=6968475 len=1469456 2022-04-22T18:03:06.610 INFO:tasks.workunit.client.0.smithi115.stdout:op 4875 completed, throughput=5MB/sec 2022-04-22T18:03:07.611 INFO:tasks.workunit.client.0.smithi115.stdout: 960: throughput=5MB/sec pending data=0 2022-04-22T18:03:07.611 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DU7ayA5AUBdjyut off=538372 len=500923 2022-04-22T18:03:07.616 INFO:tasks.workunit.client.0.smithi115.stdout:op 4876 completed, throughput=5MB/sec 2022-04-22T18:03:07.616 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1vkbsrkTUunFwdu off=486882 len=1674487 2022-04-22T18:03:07.619 INFO:tasks.workunit.client.0.smithi115.stdout:op 4877 completed, throughput=5MB/sec 2022-04-22T18:03:07.619 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zaaYh9L1Tt3oO7C off=3273932 len=312505 2022-04-22T18:03:07.621 INFO:tasks.workunit.client.0.smithi115.stdout:op 4878 completed, throughput=5MB/sec 2022-04-22T18:03:07.621 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-APUcMqS2n_n6W3_ off=10610946 len=980907 2022-04-22T18:03:07.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 4879 completed, throughput=5MB/sec 2022-04-22T18:03:07.628 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-C4_-KZxFYKm73Qb off=1798289 len=1968191 2022-04-22T18:03:07.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 4880 completed, throughput=5MB/sec 2022-04-22T18:03:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:07 smithi121 bash[36902]: cluster 2022-04-22T18:03:06.777553+0000 mgr.y (mgr.34107) 1314 : cluster [DBG] pgmap v1892: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:03:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:07 smithi121 bash[36902]: audit 2022-04-22T18:03:07.756836+0000 mon.a (mon.0) 2847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:07 smithi115 bash[45495]: cluster 2022-04-22T18:03:06.777553+0000 mgr.y (mgr.34107) 1314 : cluster [DBG] pgmap v1892: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:03:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:07 smithi115 bash[45495]: audit 2022-04-22T18:03:07.756836+0000 mon.a (mon.0) 2847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:08.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:07 smithi115 bash[46934]: cluster 2022-04-22T18:03:06.777553+0000 mgr.y (mgr.34107) 1314 : cluster [DBG] pgmap v1892: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:03:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:07 smithi115 bash[46934]: audit 2022-04-22T18:03:07.756836+0000 mon.a (mon.0) 2847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:08.637 INFO:tasks.workunit.client.0.smithi115.stdout: 961: throughput=5MB/sec pending data=0 2022-04-22T18:03:08.637 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LudhiXB7OoKeVli off=2408806 len=548580 2022-04-22T18:03:08.639 INFO:tasks.workunit.client.0.smithi115.stdout:op 4881 completed, throughput=5MB/sec 2022-04-22T18:03:08.639 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JygtfPyZtrWPoM6 off=6358862 len=1974565 2022-04-22T18:03:08.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 4882 completed, throughput=5MB/sec 2022-04-22T18:03:08.650 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-19riV0UOqIu2rl1 off=16193053 len=843489 2022-04-22T18:03:08.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 4883 completed, throughput=5MB/sec 2022-04-22T18:03:08.657 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eDzwqDHrmJcBDse off=10842814 len=906980 2022-04-22T18:03:08.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 4884 completed, throughput=5MB/sec 2022-04-22T18:03:08.663 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wN8yPp8g-3Y9krn off=3872933 len=1811082 2022-04-22T18:03:08.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 4885 completed, throughput=5MB/sec 2022-04-22T18:03:09.665 INFO:tasks.workunit.client.0.smithi115.stdout: 962: throughput=5MB/sec pending data=0 2022-04-22T18:03:09.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QjjEUmx7hNXmLoY off=12932574 len=176775 2022-04-22T18:03:09.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 4886 completed, throughput=5MB/sec 2022-04-22T18:03:09.668 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KEagjuqrofU42AS off=1263459 len=1313456 2022-04-22T18:03:09.677 INFO:tasks.workunit.client.0.smithi115.stdout:op 4887 completed, throughput=5MB/sec 2022-04-22T18:03:09.677 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BxSoNGplR16e17s off=6235419 len=890507 2022-04-22T18:03:09.684 INFO:tasks.workunit.client.0.smithi115.stdout:op 4888 completed, throughput=5MB/sec 2022-04-22T18:03:09.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6Q6T1znfVVUaidT off=1530719 len=1060739 2022-04-22T18:03:09.688 INFO:tasks.workunit.client.0.smithi115.stdout:op 4889 completed, throughput=5MB/sec 2022-04-22T18:03:09.689 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wCNPiSI5rRO8sSL off=8834549 len=203261 2022-04-22T18:03:09.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 4890 completed, throughput=5MB/sec 2022-04-22T18:03:09.692 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-R1FefXEjwHP9Wd_ off=662264 len=611202 2022-04-22T18:03:09.697 INFO:tasks.workunit.client.0.smithi115.stdout:op 4891 completed, throughput=5MB/sec 2022-04-22T18:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:09 smithi121 bash[36902]: cluster 2022-04-22T18:03:08.778778+0000 mgr.y (mgr.34107) 1315 : cluster [DBG] pgmap v1893: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:03:10.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:09 smithi115 bash[45495]: cluster 2022-04-22T18:03:08.778778+0000 mgr.y (mgr.34107) 1315 : cluster [DBG] pgmap v1893: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:03:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:09 smithi115 bash[46934]: cluster 2022-04-22T18:03:08.778778+0000 mgr.y (mgr.34107) 1315 : cluster [DBG] pgmap v1893: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 4 op/s 2022-04-22T18:03:10.697 INFO:tasks.workunit.client.0.smithi115.stdout: 963: throughput=5MB/sec pending data=0 2022-04-22T18:03:10.697 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D0Hwzex_v__s7rL off=1740971 len=164500 2022-04-22T18:03:10.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 4892 completed, throughput=5MB/sec 2022-04-22T18:03:10.699 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x2hzNxQ0DcDZh-X off=6229679 len=631714 2022-04-22T18:03:10.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 4893 completed, throughput=5MB/sec 2022-04-22T18:03:10.702 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-37QS9DM3IHS3rHP off=2295624 len=1640033 2022-04-22T18:03:10.711 INFO:tasks.workunit.client.0.smithi115.stdout:op 4894 completed, throughput=5MB/sec 2022-04-22T18:03:10.711 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-k-BKpyq45SSpA-3 off=1066059 len=1131285 2022-04-22T18:03:10.718 INFO:tasks.workunit.client.0.smithi115.stdout:op 4895 completed, throughput=5MB/sec 2022-04-22T18:03:10.719 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9EELddS1xX5BOWM off=12878643 len=912519 2022-04-22T18:03:10.725 INFO:tasks.workunit.client.0.smithi115.stdout:op 4896 completed, throughput=5MB/sec 2022-04-22T18:03:10.725 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MulrJ9NpFGwv7yn off=6737017 len=2019876 2022-04-22T18:03:10.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 4897 completed, throughput=5MB/sec 2022-04-22T18:03:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:11 smithi121 bash[36902]: audit 2022-04-22T18:03:10.371835+0000 mon.a (mon.0) 2848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:11 smithi121 bash[36902]: audit 2022-04-22T18:03:10.762988+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:10 smithi115 bash[45495]: audit 2022-04-22T18:03:10.371835+0000 mon.a (mon.0) 2848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:10 smithi115 bash[45495]: audit 2022-04-22T18:03:10.762988+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:11.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:10 smithi115 bash[46934]: audit 2022-04-22T18:03:10.371835+0000 mon.a (mon.0) 2848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:11.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:10 smithi115 bash[46934]: audit 2022-04-22T18:03:10.762988+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:11.728 INFO:tasks.workunit.client.0.smithi115.stdout: 964: throughput=5MB/sec pending data=0 2022-04-22T18:03:11.728 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LgdC-Qq7zkMUMLf off=1595373 len=1713993 2022-04-22T18:03:11.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 4898 completed, throughput=5MB/sec 2022-04-22T18:03:11.733 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D-sDkFWWV2f-73g off=1924168 len=1357147 2022-04-22T18:03:11.736 INFO:tasks.workunit.client.0.smithi115.stdout:op 4899 completed, throughput=5MB/sec 2022-04-22T18:03:11.736 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HuS9CMDWUnsT276 off=973503 len=744398 2022-04-22T18:03:11.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 4900 completed, throughput=5MB/sec 2022-04-22T18:03:12.007 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:11.734Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:12 smithi121 bash[36902]: cluster 2022-04-22T18:03:10.779884+0000 mgr.y (mgr.34107) 1316 : cluster [DBG] pgmap v1894: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:12 smithi115 bash[45495]: cluster 2022-04-22T18:03:10.779884+0000 mgr.y (mgr.34107) 1316 : cluster [DBG] pgmap v1894: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:12 smithi115 bash[46934]: cluster 2022-04-22T18:03:10.779884+0000 mgr.y (mgr.34107) 1316 : cluster [DBG] pgmap v1894: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:12.739 INFO:tasks.workunit.client.0.smithi115.stdout: 965: throughput=4.99MB/sec pending data=0 2022-04-22T18:03:12.739 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8XsXecDf6cBcs0n off=10469141 len=1393483 2022-04-22T18:03:12.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 4901 completed, throughput=5MB/sec 2022-04-22T18:03:12.747 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Sgq9ZIzuYRdL63x off=6126577 len=74086 2022-04-22T18:03:12.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 4902 completed, throughput=5MB/sec 2022-04-22T18:03:12.748 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4qNtxWevzqZ4ugp off=2694265 len=1136340 2022-04-22T18:03:12.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 4903 completed, throughput=5MB/sec 2022-04-22T18:03:12.750 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cJx37Asppu7GGx5 off=427639 len=1576611 2022-04-22T18:03:12.760 INFO:tasks.workunit.client.0.smithi115.stdout:op 4904 completed, throughput=5MB/sec 2022-04-22T18:03:12.760 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UPOe8ui1Gh4kjqZ off=5021700 len=1377791 2022-04-22T18:03:12.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 4905 completed, throughput=5MB/sec 2022-04-22T18:03:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:13 smithi121 bash[36902]: audit 2022-04-22T18:03:12.759855+0000 mon.a (mon.0) 2850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:13 smithi115 bash[45495]: audit 2022-04-22T18:03:12.759855+0000 mon.a (mon.0) 2850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:13.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:13 smithi115 bash[46934]: audit 2022-04-22T18:03:12.759855+0000 mon.a (mon.0) 2850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:13.767 INFO:tasks.workunit.client.0.smithi115.stdout: 966: throughput=5MB/sec pending data=0 2022-04-22T18:03:13.768 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O4gqMIflG9hSBZl off=19434735 len=158104 2022-04-22T18:03:13.769 INFO:tasks.workunit.client.0.smithi115.stdout:op 4906 completed, throughput=5MB/sec 2022-04-22T18:03:13.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PA6Nu5a-d6gEG8E off=1931553 len=302581 2022-04-22T18:03:13.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 4907 completed, throughput=5MB/sec 2022-04-22T18:03:13.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U5zR-pXkU95pjV9 off=7274999 len=1985808 2022-04-22T18:03:13.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 4908 completed, throughput=5MB/sec 2022-04-22T18:03:13.779 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OP673DXlFOIxNl0 off=9664184 len=448188 2022-04-22T18:03:13.779 INFO:tasks.workunit.client.0.smithi115.stdout:op 4909 completed, throughput=5MB/sec 2022-04-22T18:03:13.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Pnnx_Z2kOxmGnAG off=6396706 len=450900 2022-04-22T18:03:13.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 4910 completed, throughput=5MB/sec 2022-04-22T18:03:13.784 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r5l5MkDQ-hTRxw3 off=10318387 len=380099 2022-04-22T18:03:13.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 4911 completed, throughput=5MB/sec 2022-04-22T18:03:13.786 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-h9geqWfCLg1C9o2 off=10160438 len=1451003 2022-04-22T18:03:13.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 4912 completed, throughput=5MB/sec 2022-04-22T18:03:13.793 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XO7i2jNZiKl-vvp off=2882685 len=143563 2022-04-22T18:03:13.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 4913 completed, throughput=5MB/sec 2022-04-22T18:03:14.017 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:13.625Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:14.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[46934]: cluster 2022-04-22T18:03:12.780934+0000 mgr.y (mgr.34107) 1317 : cluster [DBG] pgmap v1895: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:14.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[46934]: audit 2022-04-22T18:03:13.627740+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:03:14.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[46934]: audit 2022-04-22T18:03:13.628219+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:03:14.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[45495]: cluster 2022-04-22T18:03:12.780934+0000 mgr.y (mgr.34107) 1317 : cluster [DBG] pgmap v1895: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:14.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[45495]: audit 2022-04-22T18:03:13.627740+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:03:14.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:14 smithi115 bash[45495]: audit 2022-04-22T18:03:13.628219+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:03:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:14 smithi121 bash[36902]: cluster 2022-04-22T18:03:12.780934+0000 mgr.y (mgr.34107) 1317 : cluster [DBG] pgmap v1895: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:14 smithi121 bash[36902]: audit 2022-04-22T18:03:13.627740+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:03:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:14 smithi121 bash[36902]: audit 2022-04-22T18:03:13.628219+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:03:14.795 INFO:tasks.workunit.client.0.smithi115.stdout: 967: throughput=5MB/sec pending data=0 2022-04-22T18:03:14.795 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tzX4f_b5zYdNfeN off=265344 len=1234601 2022-04-22T18:03:14.803 INFO:tasks.workunit.client.0.smithi115.stdout:op 4914 completed, throughput=5MB/sec 2022-04-22T18:03:14.803 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yViJfyIUKn69Tkv off=3389398 len=1677091 2022-04-22T18:03:14.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 4915 completed, throughput=5MB/sec 2022-04-22T18:03:14.812 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fm8hegFuFsT-6LE off=3625481 len=1765185 2022-04-22T18:03:14.815 INFO:tasks.workunit.client.0.smithi115.stdout:op 4916 completed, throughput=5MB/sec 2022-04-22T18:03:14.815 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4H-Q2wSm2pXDjGs off=8887531 len=1641210 2022-04-22T18:03:14.818 INFO:tasks.workunit.client.0.smithi115.stdout:op 4917 completed, throughput=5MB/sec 2022-04-22T18:03:15.819 INFO:tasks.workunit.client.0.smithi115.stdout: 968: throughput=5MB/sec pending data=0 2022-04-22T18:03:15.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Zky5KqomSLRHRO2 off=1925648 len=29422 2022-04-22T18:03:15.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 4918 completed, throughput=5MB/sec 2022-04-22T18:03:15.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R_XyFx_42XIa7M9 off=13286771 len=670426 2022-04-22T18:03:15.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 4919 completed, throughput=5MB/sec 2022-04-22T18:03:15.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PEorqRxqnajRzyE off=1252305 len=820333 2022-04-22T18:03:15.832 INFO:tasks.workunit.client.0.smithi115.stdout:op 4920 completed, throughput=5MB/sec 2022-04-22T18:03:15.832 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6sjN0CvVCOQl8xy off=2567316 len=555225 2022-04-22T18:03:15.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 4921 completed, throughput=5MB/sec 2022-04-22T18:03:15.834 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-649yD2pvttl3pGb off=17994993 len=1344622 2022-04-22T18:03:15.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 4922 completed, throughput=5MB/sec 2022-04-22T18:03:15.844 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2tOcyIHcewUVMfg off=1493935 len=1713507 2022-04-22T18:03:15.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 4923 completed, throughput=5MB/sec 2022-04-22T18:03:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[45495]: cluster 2022-04-22T18:03:14.782014+0000 mgr.y (mgr.34107) 1318 : cluster [DBG] pgmap v1896: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T18:03:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[45495]: audit 2022-04-22T18:03:15.374485+0000 mon.a (mon.0) 2853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[45495]: audit 2022-04-22T18:03:15.766619+0000 mon.a (mon.0) 2854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[46934]: cluster 2022-04-22T18:03:14.782014+0000 mgr.y (mgr.34107) 1318 : cluster [DBG] pgmap v1896: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T18:03:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[46934]: audit 2022-04-22T18:03:15.374485+0000 mon.a (mon.0) 2853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:16 smithi115 bash[46934]: audit 2022-04-22T18:03:15.766619+0000 mon.a (mon.0) 2854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:16.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:16 smithi121 bash[36902]: cluster 2022-04-22T18:03:14.782014+0000 mgr.y (mgr.34107) 1318 : cluster [DBG] pgmap v1896: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 6 op/s 2022-04-22T18:03:16.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:16 smithi121 bash[36902]: audit 2022-04-22T18:03:15.374485+0000 mon.a (mon.0) 2853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:16.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:16 smithi121 bash[36902]: audit 2022-04-22T18:03:15.766619+0000 mon.a (mon.0) 2854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:16.851 INFO:tasks.workunit.client.0.smithi115.stdout: 969: throughput=5MB/sec pending data=0 2022-04-22T18:03:16.851 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cNuNyp3sv3XVtOc off=2094855 len=908483 2022-04-22T18:03:16.858 INFO:tasks.workunit.client.0.smithi115.stdout:op 4924 completed, throughput=5MB/sec 2022-04-22T18:03:16.858 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QlgxOjwBPwVYK4N off=7183255 len=154956 2022-04-22T18:03:16.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 4925 completed, throughput=5MB/sec 2022-04-22T18:03:16.859 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3uXMaOqOrswKcnn off=14333829 len=1577092 2022-04-22T18:03:16.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 4926 completed, throughput=5MB/sec 2022-04-22T18:03:16.863 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pND-RqN7aRw9iMB off=1704886 len=467465 2022-04-22T18:03:16.867 INFO:tasks.workunit.client.0.smithi115.stdout:op 4927 completed, throughput=5MB/sec 2022-04-22T18:03:16.867 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1iWimdU9YZ5oTAv off=5445992 len=2065782 2022-04-22T18:03:16.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 4928 completed, throughput=5MB/sec 2022-04-22T18:03:17.871 INFO:tasks.workunit.client.0.smithi115.stdout: 970: throughput=5MB/sec pending data=0 2022-04-22T18:03:17.871 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HXN6l85Kotv-K3n off=11520830 len=557370 2022-04-22T18:03:17.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 4929 completed, throughput=5MB/sec 2022-04-22T18:03:17.877 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-elscQdNZqiVQ2pn off=13185711 len=1848064 2022-04-22T18:03:17.886 INFO:tasks.workunit.client.0.smithi115.stdout:op 4930 completed, throughput=5MB/sec 2022-04-22T18:03:17.886 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S6jZMo69O99oeeK off=8510672 len=453440 2022-04-22T18:03:17.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 4931 completed, throughput=5MB/sec 2022-04-22T18:03:17.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EnnBkhPTinJOZ-c off=1947251 len=2055144 2022-04-22T18:03:17.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 4932 completed, throughput=5MB/sec 2022-04-22T18:03:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:18 smithi115 bash[45495]: cluster 2022-04-22T18:03:16.783115+0000 mgr.y (mgr.34107) 1319 : cluster [DBG] pgmap v1897: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:18 smithi115 bash[45495]: audit 2022-04-22T18:03:17.762360+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:18 smithi115 bash[46934]: cluster 2022-04-22T18:03:16.783115+0000 mgr.y (mgr.34107) 1319 : cluster [DBG] pgmap v1897: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:18 smithi115 bash[46934]: audit 2022-04-22T18:03:17.762360+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:18 smithi121 bash[36902]: cluster 2022-04-22T18:03:16.783115+0000 mgr.y (mgr.34107) 1319 : cluster [DBG] pgmap v1897: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:18 smithi121 bash[36902]: audit 2022-04-22T18:03:17.762360+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:18.900 INFO:tasks.workunit.client.0.smithi115.stdout: 971: throughput=5MB/sec pending data=0 2022-04-22T18:03:18.900 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PDU7BH7GAFuBxyZ off=22743497 len=486402 2022-04-22T18:03:18.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 4933 completed, throughput=5MB/sec 2022-04-22T18:03:18.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WskpufXLlEdaeTy off=7295787 len=1075904 2022-04-22T18:03:18.910 INFO:tasks.workunit.client.0.smithi115.stdout:op 4934 completed, throughput=5MB/sec 2022-04-22T18:03:18.910 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OEirtS_JU7H45_D off=10572515 len=858842 2022-04-22T18:03:18.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 4935 completed, throughput=5MB/sec 2022-04-22T18:03:18.917 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DblPxLRCF86h0g7 off=3718048 len=2055318 2022-04-22T18:03:18.927 INFO:tasks.workunit.client.0.smithi115.stdout:op 4936 completed, throughput=5MB/sec 2022-04-22T18:03:18.927 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7EZrU-C9ld-DdFz off=1915962 len=1695540 2022-04-22T18:03:18.934 INFO:tasks.workunit.client.0.smithi115.stdout:op 4937 completed, throughput=5MB/sec 2022-04-22T18:03:19.934 INFO:tasks.workunit.client.0.smithi115.stdout: 972: throughput=5MB/sec pending data=0 2022-04-22T18:03:19.934 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sQu5VpLwEuliB23 off=1766828 len=1586811 2022-04-22T18:03:19.937 INFO:tasks.workunit.client.0.smithi115.stdout:op 4938 completed, throughput=5MB/sec 2022-04-22T18:03:19.938 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SwO077XUBSMvteC off=4360812 len=230791 2022-04-22T18:03:19.943 INFO:tasks.workunit.client.0.smithi115.stdout:op 4939 completed, throughput=5MB/sec 2022-04-22T18:03:19.943 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Fj2k_nYKthHY3F8 off=2414217 len=649605 2022-04-22T18:03:19.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 4940 completed, throughput=5MB/sec 2022-04-22T18:03:19.944 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-McdltjB_s6L_cKv off=1633095 len=907426 2022-04-22T18:03:19.950 INFO:tasks.workunit.client.0.smithi115.stdout:op 4941 completed, throughput=5MB/sec 2022-04-22T18:03:19.950 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WPb4RaW1o6VvkFp off=7918267 len=150547 2022-04-22T18:03:19.954 INFO:tasks.workunit.client.0.smithi115.stdout:op 4942 completed, throughput=5MB/sec 2022-04-22T18:03:19.954 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LZZezJdalF_b3Kl off=3281550 len=1022737 2022-04-22T18:03:19.958 INFO:tasks.workunit.client.0.smithi115.stdout:op 4943 completed, throughput=5MB/sec 2022-04-22T18:03:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:20 smithi115 bash[45495]: cluster 2022-04-22T18:03:18.784308+0000 mgr.y (mgr.34107) 1320 : cluster [DBG] pgmap v1898: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:20 smithi115 bash[46934]: cluster 2022-04-22T18:03:18.784308+0000 mgr.y (mgr.34107) 1320 : cluster [DBG] pgmap v1898: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:20 smithi121 bash[36902]: cluster 2022-04-22T18:03:18.784308+0000 mgr.y (mgr.34107) 1320 : cluster [DBG] pgmap v1898: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:20.959 INFO:tasks.workunit.client.0.smithi115.stdout: 973: throughput=5MB/sec pending data=0 2022-04-22T18:03:20.959 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ajoI1MPD70vKfah off=5502086 len=1680853 2022-04-22T18:03:20.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 4944 completed, throughput=5MB/sec 2022-04-22T18:03:20.971 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8nbt4tH94THPh8f off=8000787 len=1634079 2022-04-22T18:03:20.974 INFO:tasks.workunit.client.0.smithi115.stdout:op 4945 completed, throughput=5MB/sec 2022-04-22T18:03:20.974 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SPuTWPzb-WSQRlm off=971719 len=1013111 2022-04-22T18:03:20.978 INFO:tasks.workunit.client.0.smithi115.stdout:op 4946 completed, throughput=5MB/sec 2022-04-22T18:03:20.978 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hs3dNN3NfT1G562 off=2830227 len=786625 2022-04-22T18:03:20.980 INFO:tasks.workunit.client.0.smithi115.stdout:op 4947 completed, throughput=5MB/sec 2022-04-22T18:03:20.980 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6rPYHUiVU8yKKM0 off=9828235 len=707761 2022-04-22T18:03:20.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 4948 completed, throughput=5MB/sec 2022-04-22T18:03:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:21 smithi115 bash[45495]: audit 2022-04-22T18:03:20.377608+0000 mon.a (mon.0) 2856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:21 smithi115 bash[45495]: audit 2022-04-22T18:03:20.770212+0000 mon.a (mon.0) 2857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:21 smithi115 bash[46934]: audit 2022-04-22T18:03:20.377608+0000 mon.a (mon.0) 2856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:21 smithi115 bash[46934]: audit 2022-04-22T18:03:20.770212+0000 mon.a (mon.0) 2857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:21 smithi121 bash[36902]: audit 2022-04-22T18:03:20.377608+0000 mon.a (mon.0) 2856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:21 smithi121 bash[36902]: audit 2022-04-22T18:03:20.770212+0000 mon.a (mon.0) 2857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:21.987 INFO:tasks.workunit.client.0.smithi115.stdout: 974: throughput=5MB/sec pending data=0 2022-04-22T18:03:21.987 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7VsxNsOl1pJU443 off=1787769 len=1349907 2022-04-22T18:03:21.996 INFO:tasks.workunit.client.0.smithi115.stdout:op 4949 completed, throughput=5MB/sec 2022-04-22T18:03:21.996 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L5n5tx8NorXfFCa off=1515219 len=925559 2022-04-22T18:03:22.003 INFO:tasks.workunit.client.0.smithi115.stdout:op 4950 completed, throughput=5MB/sec 2022-04-22T18:03:22.003 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x-ZUMrZttENgl5c off=8293677 len=755246 2022-04-22T18:03:22.005 INFO:tasks.workunit.client.0.smithi115.stdout:op 4951 completed, throughput=5MB/sec 2022-04-22T18:03:22.005 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hLU-n8js6EerDFL off=1189583 len=1699521 2022-04-22T18:03:22.007 INFO:tasks.workunit.client.0.smithi115.stdout:op 4952 completed, throughput=5MB/sec 2022-04-22T18:03:22.008 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9RNf1d0DfNXSxeL off=21737944 len=606937 2022-04-22T18:03:22.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 4953 completed, throughput=5MB/sec 2022-04-22T18:03:22.035 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:21.734Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:22 smithi115 bash[45495]: cluster 2022-04-22T18:03:20.785439+0000 mgr.y (mgr.34107) 1321 : cluster [DBG] pgmap v1899: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:22 smithi115 bash[46934]: cluster 2022-04-22T18:03:20.785439+0000 mgr.y (mgr.34107) 1321 : cluster [DBG] pgmap v1899: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:22 smithi121 bash[36902]: cluster 2022-04-22T18:03:20.785439+0000 mgr.y (mgr.34107) 1321 : cluster [DBG] pgmap v1899: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:23.013 INFO:tasks.workunit.client.0.smithi115.stdout: 975: throughput=5MB/sec pending data=0 2022-04-22T18:03:23.013 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YsRckZPbC-Iw8WZ off=1734224 len=745370 2022-04-22T18:03:23.016 INFO:tasks.workunit.client.0.smithi115.stdout:op 4954 completed, throughput=5MB/sec 2022-04-22T18:03:23.016 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NuLVUura5igivwP off=2831219 len=560394 2022-04-22T18:03:23.022 INFO:tasks.workunit.client.0.smithi115.stdout:op 4955 completed, throughput=5MB/sec 2022-04-22T18:03:23.022 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VNCaCx4E185gWV- off=5115256 len=693710 2022-04-22T18:03:23.028 INFO:tasks.workunit.client.0.smithi115.stdout:op 4956 completed, throughput=5MB/sec 2022-04-22T18:03:23.028 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1U1yhgZZe-uWyPl off=5167429 len=177384 2022-04-22T18:03:23.030 INFO:tasks.workunit.client.0.smithi115.stdout:op 4957 completed, throughput=5MB/sec 2022-04-22T18:03:23.030 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y9m0uU1J6CV1UGr off=5330685 len=1617193 2022-04-22T18:03:23.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 4958 completed, throughput=5MB/sec 2022-04-22T18:03:23.037 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pNCDarN_mHtk4Qz off=9957381 len=1399874 2022-04-22T18:03:23.044 INFO:tasks.workunit.client.0.smithi115.stdout:op 4959 completed, throughput=5MB/sec 2022-04-22T18:03:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:23 smithi121 bash[36902]: audit 2022-04-22T18:03:22.765663+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:23 smithi115 bash[45495]: audit 2022-04-22T18:03:22.765663+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:23 smithi115 bash[46934]: audit 2022-04-22T18:03:22.765663+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:23.616Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:24.044 INFO:tasks.workunit.client.0.smithi115.stdout: 976: throughput=5MB/sec pending data=0 2022-04-22T18:03:24.044 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8JtWKC2iywNrYc4 off=17794101 len=1359709 2022-04-22T18:03:24.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 4960 completed, throughput=5MB/sec 2022-04-22T18:03:24.054 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5LMG60scOl4zD34 off=3282287 len=1382155 2022-04-22T18:03:24.056 INFO:tasks.workunit.client.0.smithi115.stdout:op 4961 completed, throughput=5MB/sec 2022-04-22T18:03:24.056 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kJ8SIEu_J-NPQPL off=1130170 len=835863 2022-04-22T18:03:24.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 4962 completed, throughput=5MB/sec 2022-04-22T18:03:24.058 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VNCaCx4E185gWV- off=1624125 len=1525954 2022-04-22T18:03:24.060 INFO:tasks.workunit.client.0.smithi115.stdout:op 4963 completed, throughput=5MB/sec 2022-04-22T18:03:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:24 smithi121 bash[36902]: cluster 2022-04-22T18:03:22.786582+0000 mgr.y (mgr.34107) 1322 : cluster [DBG] pgmap v1900: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:24 smithi115 bash[45495]: cluster 2022-04-22T18:03:22.786582+0000 mgr.y (mgr.34107) 1322 : cluster [DBG] pgmap v1900: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:24 smithi115 bash[46934]: cluster 2022-04-22T18:03:22.786582+0000 mgr.y (mgr.34107) 1322 : cluster [DBG] pgmap v1900: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 5 op/s 2022-04-22T18:03:25.060 INFO:tasks.workunit.client.0.smithi115.stdout: 977: throughput=5MB/sec pending data=0 2022-04-22T18:03:25.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QxQfhUeEYCrmrsX off=13095035 len=654097 2022-04-22T18:03:25.066 INFO:tasks.workunit.client.0.smithi115.stdout:op 4964 completed, throughput=5MB/sec 2022-04-22T18:03:25.066 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U6xZNdog8-H8eZE off=8374126 len=912323 2022-04-22T18:03:25.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 4965 completed, throughput=5MB/sec 2022-04-22T18:03:25.068 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_KZvhIvEsqPW64v off=686396 len=1386601 2022-04-22T18:03:25.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 4966 completed, throughput=5MB/sec 2022-04-22T18:03:25.075 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zt8__T-Rfzq9io2 off=7056793 len=374923 2022-04-22T18:03:25.076 INFO:tasks.workunit.client.0.smithi115.stdout:op 4967 completed, throughput=5MB/sec 2022-04-22T18:03:25.076 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bH3H8Z2i_V1Dnho off=2386455 len=560924 2022-04-22T18:03:25.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 4968 completed, throughput=5MB/sec 2022-04-22T18:03:25.081 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qY6T8wfZrNofGvp off=16896329 len=1792687 2022-04-22T18:03:25.091 INFO:tasks.workunit.client.0.smithi115.stdout:op 4969 completed, throughput=5MB/sec 2022-04-22T18:03:26.091 INFO:tasks.workunit.client.0.smithi115.stdout: 978: throughput=5MB/sec pending data=0 2022-04-22T18:03:26.092 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cpayPV3F_s72L2g off=4413866 len=1898336 2022-04-22T18:03:26.100 INFO:tasks.workunit.client.0.smithi115.stdout:op 4970 completed, throughput=5MB/sec 2022-04-22T18:03:26.100 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s-U_uEBGpmHjLCi off=17312859 len=1776361 2022-04-22T18:03:26.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 4971 completed, throughput=5MB/sec 2022-04-22T18:03:26.109 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UE5IgKhlYTHDJA6 off=1195598 len=1842386 2022-04-22T18:03:26.111 INFO:tasks.workunit.client.0.smithi115.stdout:op 4972 completed, throughput=5MB/sec 2022-04-22T18:03:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:26 smithi121 bash[36902]: cluster 2022-04-22T18:03:24.787697+0000 mgr.y (mgr.34107) 1323 : cluster [DBG] pgmap v1901: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:26 smithi121 bash[36902]: audit 2022-04-22T18:03:25.380710+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:26 smithi121 bash[36902]: audit 2022-04-22T18:03:25.774061+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[45495]: cluster 2022-04-22T18:03:24.787697+0000 mgr.y (mgr.34107) 1323 : cluster [DBG] pgmap v1901: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[45495]: audit 2022-04-22T18:03:25.380710+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[45495]: audit 2022-04-22T18:03:25.774061+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[46934]: cluster 2022-04-22T18:03:24.787697+0000 mgr.y (mgr.34107) 1323 : cluster [DBG] pgmap v1901: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:03:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[46934]: audit 2022-04-22T18:03:25.380710+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:26 smithi115 bash[46934]: audit 2022-04-22T18:03:25.774061+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:27.112 INFO:tasks.workunit.client.0.smithi115.stdout: 979: throughput=5MB/sec pending data=0 2022-04-22T18:03:27.112 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LZZezJdalF_b3Kl off=1450984 len=241983 2022-04-22T18:03:27.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 4973 completed, throughput=5MB/sec 2022-04-22T18:03:27.116 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BsrqW_ghPmkJfED off=8383656 len=1251531 2022-04-22T18:03:27.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 4974 completed, throughput=5MB/sec 2022-04-22T18:03:27.124 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lY2unwYOw8aQTvs off=1252410 len=688832 2022-04-22T18:03:27.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 4975 completed, throughput=5MB/sec 2022-04-22T18:03:27.125 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wzxS3_gXvJhTOtu off=12165808 len=613500 2022-04-22T18:03:27.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 4976 completed, throughput=5MB/sec 2022-04-22T18:03:27.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UikSjVrHLqkb-2v off=5446655 len=985747 2022-04-22T18:03:27.133 INFO:tasks.workunit.client.0.smithi115.stdout:op 4977 completed, throughput=5MB/sec 2022-04-22T18:03:27.133 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HpictmTJsVJS3eZ off=11876986 len=1986355 2022-04-22T18:03:27.141 INFO:tasks.workunit.client.0.smithi115.stdout:op 4978 completed, throughput=5MB/sec 2022-04-22T18:03:27.864 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:03:28.141 INFO:tasks.workunit.client.0.smithi115.stdout: 980: throughput=5MB/sec pending data=0 2022-04-22T18:03:28.141 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XNuemQHN97X7tMK off=18235434 len=693887 2022-04-22T18:03:28.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 4979 completed, throughput=5MB/sec 2022-04-22T18:03:28.143 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FBT3Cgy7shjwv7I off=10759240 len=1610476 2022-04-22T18:03:28.153 INFO:tasks.workunit.client.0.smithi115.stdout:op 4980 completed, throughput=5MB/sec 2022-04-22T18:03:28.153 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z3A-1jVt376Fyft off=2706100 len=1538625 2022-04-22T18:03:28.160 INFO:tasks.workunit.client.0.smithi115.stdout:op 4981 completed, throughput=5MB/sec 2022-04-22T18:03:28.160 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1NqAOsfdLoKvoA3 off=1209075 len=1733000 2022-04-22T18:03:28.166 INFO:tasks.workunit.client.0.smithi115.stdout:op 4982 completed, throughput=5MB/sec 2022-04-22T18:03:28.222 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:03:28.222 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (26m) 5m ago 40m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:03:28.223 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (40m) 6m ago 40m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:03:28.223 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (28m) 5m ago 41m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:03:28.223 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (28m) 6m ago 41m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:03:28.223 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (35m) 6m ago 45m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:03:28.224 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (36m) 5m ago 47m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:03:28.224 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (35m) 5m ago 47m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:03:28.224 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (34m) 6m ago 45m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:03:28.224 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (34m) 5m ago 46m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:03:28.225 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (27m) 5m ago 40m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:03:28.225 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (27m) 6m ago 40m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:03:28.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (33m) 5m ago 45m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:03:28.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (32m) 5m ago 44m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:03:28.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (32m) 5m ago 44m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:03:28.226 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (31m) 5m ago 43m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:03:28.226 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (30m) 6m ago 43m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:03:28.226 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (30m) 6m ago 42m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:03:28.226 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (29m) 6m ago 42m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:03:28.227 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (28m) 6m ago 41m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:03:28.227 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (27m) 6m ago 40m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:03:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:28 smithi121 bash[36902]: cluster 2022-04-22T18:03:26.788744+0000 mgr.y (mgr.34107) 1324 : cluster [DBG] pgmap v1902: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:03:28.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:28 smithi121 bash[36902]: audit 2022-04-22T18:03:27.768756+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:28 smithi115 bash[45495]: cluster 2022-04-22T18:03:26.788744+0000 mgr.y (mgr.34107) 1324 : cluster [DBG] pgmap v1902: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:03:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:28 smithi115 bash[45495]: audit 2022-04-22T18:03:27.768756+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:28 smithi115 bash[46934]: cluster 2022-04-22T18:03:26.788744+0000 mgr.y (mgr.34107) 1324 : cluster [DBG] pgmap v1902: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 4 op/s 2022-04-22T18:03:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:28 smithi115 bash[46934]: audit 2022-04-22T18:03:27.768756+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:28.634 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:03:28.634 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:28.635 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:03:28.636 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:03:29.167 INFO:tasks.workunit.client.0.smithi115.stdout: 981: throughput=5MB/sec pending data=0 2022-04-22T18:03:29.167 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0wu5D82tH6GizcY off=15453357 len=1880434 2022-04-22T18:03:29.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 4983 completed, throughput=5MB/sec 2022-04-22T18:03:29.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zf2s6YYzvFU5ZTa off=5201549 len=562745 2022-04-22T18:03:29.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 4984 completed, throughput=5MB/sec 2022-04-22T18:03:29.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TD7Wfxsmrlln3Mb off=2194625 len=1524226 2022-04-22T18:03:29.178 INFO:tasks.workunit.client.0.smithi115.stdout:op 4985 completed, throughput=5MB/sec 2022-04-22T18:03:29.178 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VAa9mh80MvSqTI5 off=2851661 len=1339558 2022-04-22T18:03:29.188 INFO:tasks.workunit.client.0.smithi115.stdout:op 4986 completed, throughput=5MB/sec 2022-04-22T18:03:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:29 smithi121 bash[36902]: audit 2022-04-22T18:03:27.841428+0000 mgr.y (mgr.34107) 1325 : audit [DBG] from='client.35787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:29 smithi121 bash[36902]: audit 2022-04-22T18:03:28.196725+0000 mgr.y (mgr.34107) 1326 : audit [DBG] from='client.35793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:29 smithi121 bash[36902]: audit 2022-04-22T18:03:28.627205+0000 mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.115:0/2985388572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[45495]: audit 2022-04-22T18:03:27.841428+0000 mgr.y (mgr.34107) 1325 : audit [DBG] from='client.35787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[45495]: audit 2022-04-22T18:03:28.196725+0000 mgr.y (mgr.34107) 1326 : audit [DBG] from='client.35793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[45495]: audit 2022-04-22T18:03:28.627205+0000 mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.115:0/2985388572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[46934]: audit 2022-04-22T18:03:27.841428+0000 mgr.y (mgr.34107) 1325 : audit [DBG] from='client.35787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[46934]: audit 2022-04-22T18:03:28.196725+0000 mgr.y (mgr.34107) 1326 : audit [DBG] from='client.35793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:03:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:29 smithi115 bash[46934]: audit 2022-04-22T18:03:28.627205+0000 mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.115:0/2985388572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:03:30.188 INFO:tasks.workunit.client.0.smithi115.stdout: 982: throughput=5MB/sec pending data=0 2022-04-22T18:03:30.188 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CFj3oHh7DwRVBOZ off=9356208 len=1969626 2022-04-22T18:03:30.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 4987 completed, throughput=5MB/sec 2022-04-22T18:03:30.193 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xGJfxULOT8s4Up6 off=5498793 len=688261 2022-04-22T18:03:30.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 4988 completed, throughput=5MB/sec 2022-04-22T18:03:30.195 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BfyBcuk3wbK9u6v off=16482330 len=186406 2022-04-22T18:03:30.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 4989 completed, throughput=5MB/sec 2022-04-22T18:03:30.200 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bD-1CaJhiFo1V8u off=8720608 len=945298 2022-04-22T18:03:30.206 INFO:tasks.workunit.client.0.smithi115.stdout:op 4990 completed, throughput=5MB/sec 2022-04-22T18:03:30.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wKNvDTyPtGkJLl5 off=2193775 len=324859 2022-04-22T18:03:30.210 INFO:tasks.workunit.client.0.smithi115.stdout:op 4991 completed, throughput=5MB/sec 2022-04-22T18:03:30.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OVjq-2tOSYMhdfu off=2667088 len=1935980 2022-04-22T18:03:30.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 4992 completed, throughput=5MB/sec 2022-04-22T18:03:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:30 smithi121 bash[36902]: cluster 2022-04-22T18:03:28.790052+0000 mgr.y (mgr.34107) 1327 : cluster [DBG] pgmap v1903: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:03:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:30 smithi115 bash[45495]: cluster 2022-04-22T18:03:28.790052+0000 mgr.y (mgr.34107) 1327 : cluster [DBG] pgmap v1903: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:03:30.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:30 smithi115 bash[46934]: cluster 2022-04-22T18:03:28.790052+0000 mgr.y (mgr.34107) 1327 : cluster [DBG] pgmap v1903: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:03:31.221 INFO:tasks.workunit.client.0.smithi115.stdout: 983: throughput=5MB/sec pending data=0 2022-04-22T18:03:31.221 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AsnNbodRcVr7V2k off=7215305 len=600784 2022-04-22T18:03:31.227 INFO:tasks.workunit.client.0.smithi115.stdout:op 4993 completed, throughput=5MB/sec 2022-04-22T18:03:31.227 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eCD0dq3NjWemc1C off=2437199 len=1392692 2022-04-22T18:03:31.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 4994 completed, throughput=5MB/sec 2022-04-22T18:03:31.232 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0xwycwqVUvzT4Bz off=13421395 len=468050 2022-04-22T18:03:31.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 4995 completed, throughput=5MB/sec 2022-04-22T18:03:31.236 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IpkFM7vomgcpHsk off=3907504 len=454766 2022-04-22T18:03:31.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 4996 completed, throughput=5MB/sec 2022-04-22T18:03:31.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YvAkdISj581y23b off=10948343 len=1905478 2022-04-22T18:03:31.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 4997 completed, throughput=5MB/sec 2022-04-22T18:03:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:31 smithi121 bash[36902]: audit 2022-04-22T18:03:30.383967+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:31 smithi121 bash[36902]: audit 2022-04-22T18:03:30.777811+0000 mon.a (mon.0) 2863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:31 smithi115 bash[45495]: audit 2022-04-22T18:03:30.383967+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:31 smithi115 bash[45495]: audit 2022-04-22T18:03:30.777811+0000 mon.a (mon.0) 2863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:31 smithi115 bash[46934]: audit 2022-04-22T18:03:30.383967+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:31 smithi115 bash[46934]: audit 2022-04-22T18:03:30.777811+0000 mon.a (mon.0) 2863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:31.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:32.248 INFO:tasks.workunit.client.0.smithi115.stdout: 984: throughput=5MB/sec pending data=0 2022-04-22T18:03:32.248 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wIBwaL7NURU6JQE off=5738302 len=534051 2022-04-22T18:03:32.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 4998 completed, throughput=5MB/sec 2022-04-22T18:03:32.250 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b0laMTcuv6tLPbq off=17466894 len=1151819 2022-04-22T18:03:32.252 INFO:tasks.workunit.client.0.smithi115.stdout:op 4999 completed, throughput=5MB/sec 2022-04-22T18:03:32.253 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BwxY_ENf9Fy_8oS off=1013445 len=1385596 2022-04-22T18:03:32.257 INFO:tasks.workunit.client.0.smithi115.stdout:op 5000 completed, throughput=5MB/sec 2022-04-22T18:03:32.258 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dSsd-NpH7WaGvvP off=9038073 len=1778296 2022-04-22T18:03:32.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 5001 completed, throughput=5MB/sec 2022-04-22T18:03:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:32 smithi121 bash[36902]: cluster 2022-04-22T18:03:30.791150+0000 mgr.y (mgr.34107) 1328 : cluster [DBG] pgmap v1904: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:32 smithi115 bash[45495]: cluster 2022-04-22T18:03:30.791150+0000 mgr.y (mgr.34107) 1328 : cluster [DBG] pgmap v1904: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:32 smithi115 bash[46934]: cluster 2022-04-22T18:03:30.791150+0000 mgr.y (mgr.34107) 1328 : cluster [DBG] pgmap v1904: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:33.267 INFO:tasks.workunit.client.0.smithi115.stdout: 985: throughput=5MB/sec pending data=0 2022-04-22T18:03:33.268 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YX8bXWw1qrXo3oq off=10844423 len=1881644 2022-04-22T18:03:33.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 5002 completed, throughput=5MB/sec 2022-04-22T18:03:33.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ES8fo-6FOKIVn2_ off=611265 len=1669463 2022-04-22T18:03:33.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 5003 completed, throughput=5MB/sec 2022-04-22T18:03:33.284 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oDMadhSlHbuwxSk off=3156918 len=248511 2022-04-22T18:03:33.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 5004 completed, throughput=5MB/sec 2022-04-22T18:03:33.285 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-eRBudEMfHuf_r5J off=3031694 len=610217 2022-04-22T18:03:33.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 5005 completed, throughput=5MB/sec 2022-04-22T18:03:33.286 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XOcw8wAzwALISva off=2977451 len=8579 2022-04-22T18:03:33.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 5006 completed, throughput=5MB/sec 2022-04-22T18:03:33.287 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PIHl_8mwukAzELM off=3468252 len=605797 2022-04-22T18:03:33.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 5007 completed, throughput=5MB/sec 2022-04-22T18:03:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:33 smithi121 bash[36902]: audit 2022-04-22T18:03:32.771699+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:33.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:33 smithi115 bash[45495]: audit 2022-04-22T18:03:32.771699+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:33 smithi115 bash[46934]: audit 2022-04-22T18:03:32.771699+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:33.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:34.292 INFO:tasks.workunit.client.0.smithi115.stdout: 986: throughput=5MB/sec pending data=0 2022-04-22T18:03:34.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OVcouF4a_BbqXNX off=62486 len=148263 2022-04-22T18:03:34.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 5008 completed, throughput=5MB/sec 2022-04-22T18:03:34.347 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DgpmjJBaki2MOoo off=2137279 len=1029089 2022-04-22T18:03:34.347 INFO:tasks.workunit.client.0.smithi115.stdout:op 5009 completed, throughput=5MB/sec 2022-04-22T18:03:34.347 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yrTXysYjrgl5fgl off=4393864 len=769450 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 5010 completed, throughput=5MB/sec 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PZPvOP2W2eys99S off=1240251 len=1204312 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 5011 completed, throughput=5MB/sec 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r2H03fEXk97tzl1 off=15728476 len=536189 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 5012 completed, throughput=5MB/sec 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-a3YORezUWlpzubb off=9631490 len=654438 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 5013 completed, throughput=5MB/sec 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ABrtg2syagO0Jpo off=645393 len=1026209 2022-04-22T18:03:34.348 INFO:tasks.workunit.client.0.smithi115.stdout:op 5014 completed, throughput=5MB/sec 2022-04-22T18:03:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:34 smithi121 bash[36902]: cluster 2022-04-22T18:03:32.792389+0000 mgr.y (mgr.34107) 1329 : cluster [DBG] pgmap v1905: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:34 smithi115 bash[45495]: cluster 2022-04-22T18:03:32.792389+0000 mgr.y (mgr.34107) 1329 : cluster [DBG] pgmap v1905: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:34 smithi115 bash[46934]: cluster 2022-04-22T18:03:32.792389+0000 mgr.y (mgr.34107) 1329 : cluster [DBG] pgmap v1905: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:03:35.314 INFO:tasks.workunit.client.0.smithi115.stdout: 987: throughput=5MB/sec pending data=0 2022-04-22T18:03:35.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eSBvup61I2ve1AC off=1391175 len=1527073 2022-04-22T18:03:35.324 INFO:tasks.workunit.client.0.smithi115.stdout:op 5015 completed, throughput=5MB/sec 2022-04-22T18:03:35.324 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kLR26ULldgM-XI4 off=14610063 len=65741 2022-04-22T18:03:35.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 5016 completed, throughput=5MB/sec 2022-04-22T18:03:35.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p9xulOFWNVBPJeP off=1401494 len=531551 2022-04-22T18:03:35.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 5017 completed, throughput=5MB/sec 2022-04-22T18:03:35.326 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gdPZaBtaVuN5wsz off=4487735 len=1707394 2022-04-22T18:03:35.333 INFO:tasks.workunit.client.0.smithi115.stdout:op 5018 completed, throughput=5MB/sec 2022-04-22T18:03:35.334 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jtNVRNw1snXP8e7 off=2204595 len=1966477 2022-04-22T18:03:35.336 INFO:tasks.workunit.client.0.smithi115.stdout:op 5019 completed, throughput=5MB/sec 2022-04-22T18:03:36.336 INFO:tasks.workunit.client.0.smithi115.stdout: 988: throughput=5MB/sec pending data=0 2022-04-22T18:03:36.337 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RSuJHLLJYu29cqO off=5935330 len=474728 2022-04-22T18:03:36.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 5020 completed, throughput=5MB/sec 2022-04-22T18:03:36.338 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mdRHfU5Fv28kY8q off=17101531 len=52769 2022-04-22T18:03:36.339 INFO:tasks.workunit.client.0.smithi115.stdout:op 5021 completed, throughput=5MB/sec 2022-04-22T18:03:36.340 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--1LDt-RMmtmbIFn off=2142937 len=681364 2022-04-22T18:03:36.342 INFO:tasks.workunit.client.0.smithi115.stdout:op 5022 completed, throughput=5MB/sec 2022-04-22T18:03:36.342 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PtaWpwSRkIibf4x off=2435186 len=321745 2022-04-22T18:03:36.343 INFO:tasks.workunit.client.0.smithi115.stdout:op 5023 completed, throughput=5MB/sec 2022-04-22T18:03:36.343 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wVCzmAdpA4hZuWw off=9090317 len=1294431 2022-04-22T18:03:36.346 INFO:tasks.workunit.client.0.smithi115.stdout:op 5024 completed, throughput=5MB/sec 2022-04-22T18:03:36.346 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F7_-EnapBU-nvNT off=10956544 len=2044012 2022-04-22T18:03:36.354 INFO:tasks.workunit.client.0.smithi115.stdout:op 5025 completed, throughput=5MB/sec 2022-04-22T18:03:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:36 smithi121 bash[36902]: cluster 2022-04-22T18:03:34.793563+0000 mgr.y (mgr.34107) 1330 : cluster [DBG] pgmap v1906: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:03:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:36 smithi121 bash[36902]: audit 2022-04-22T18:03:35.387145+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:36 smithi121 bash[36902]: audit 2022-04-22T18:03:35.781342+0000 mon.a (mon.0) 2866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[45495]: cluster 2022-04-22T18:03:34.793563+0000 mgr.y (mgr.34107) 1330 : cluster [DBG] pgmap v1906: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:03:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[45495]: audit 2022-04-22T18:03:35.387145+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[45495]: audit 2022-04-22T18:03:35.781342+0000 mon.a (mon.0) 2866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[46934]: cluster 2022-04-22T18:03:34.793563+0000 mgr.y (mgr.34107) 1330 : cluster [DBG] pgmap v1906: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 4 op/s 2022-04-22T18:03:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[46934]: audit 2022-04-22T18:03:35.387145+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:36 smithi115 bash[46934]: audit 2022-04-22T18:03:35.781342+0000 mon.a (mon.0) 2866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:37.354 INFO:tasks.workunit.client.0.smithi115.stdout: 989: throughput=5MB/sec pending data=0 2022-04-22T18:03:37.355 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RBzLW7qYnmHsDR8 off=21120 len=37461 2022-04-22T18:03:37.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 5026 completed, throughput=5MB/sec 2022-04-22T18:03:37.356 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-deqIycCoDUMlhVv off=5503605 len=929812 2022-04-22T18:03:37.359 INFO:tasks.workunit.client.0.smithi115.stdout:op 5027 completed, throughput=5MB/sec 2022-04-22T18:03:37.359 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-P2zRpGxXe93aDr3 off=7678468 len=1343099 2022-04-22T18:03:37.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 5028 completed, throughput=5MB/sec 2022-04-22T18:03:37.367 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nu5gIXiFJYPLA_a off=2246048 len=1599832 2022-04-22T18:03:37.371 INFO:tasks.workunit.client.0.smithi115.stdout:op 5029 completed, throughput=5MB/sec 2022-04-22T18:03:37.371 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UrhOFZ0ccgQ2p8G off=5771017 len=85859 2022-04-22T18:03:37.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 5030 completed, throughput=5MB/sec 2022-04-22T18:03:37.376 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4H-Q2wSm2pXDjGs off=613143 len=1616787 2022-04-22T18:03:37.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 5031 completed, throughput=5MB/sec 2022-04-22T18:03:38.384 INFO:tasks.workunit.client.0.smithi115.stdout: 990: throughput=5MB/sec pending data=0 2022-04-22T18:03:38.384 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JnhVef53_PeLP4a off=4112472 len=537669 2022-04-22T18:03:38.390 INFO:tasks.workunit.client.0.smithi115.stdout:op 5032 completed, throughput=5MB/sec 2022-04-22T18:03:38.390 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v8vUeG0GEfT_hTI off=5726250 len=1104428 2022-04-22T18:03:38.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 5033 completed, throughput=5MB/sec 2022-04-22T18:03:38.397 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-L3oG744s-Q-vxZD off=15724577 len=1918328 2022-04-22T18:03:38.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 5034 completed, throughput=5MB/sec 2022-04-22T18:03:38.400 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rFmkReN8VaCe-qL off=206623 len=1246690 2022-04-22T18:03:38.408 INFO:tasks.workunit.client.0.smithi115.stdout:op 5035 completed, throughput=5MB/sec 2022-04-22T18:03:38.408 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sOcGaTymXWN5Jtj off=14791725 len=885073 2022-04-22T18:03:38.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 5036 completed, throughput=5MB/sec 2022-04-22T18:03:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:38 smithi121 bash[36902]: cluster 2022-04-22T18:03:36.794634+0000 mgr.y (mgr.34107) 1331 : cluster [DBG] pgmap v1907: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:03:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:38 smithi121 bash[36902]: audit 2022-04-22T18:03:37.774747+0000 mon.a (mon.0) 2867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:38 smithi115 bash[45495]: cluster 2022-04-22T18:03:36.794634+0000 mgr.y (mgr.34107) 1331 : cluster [DBG] pgmap v1907: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:03:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:38 smithi115 bash[45495]: audit 2022-04-22T18:03:37.774747+0000 mon.a (mon.0) 2867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:38 smithi115 bash[46934]: cluster 2022-04-22T18:03:36.794634+0000 mgr.y (mgr.34107) 1331 : cluster [DBG] pgmap v1907: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:03:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:38 smithi115 bash[46934]: audit 2022-04-22T18:03:37.774747+0000 mon.a (mon.0) 2867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:39.410 INFO:tasks.workunit.client.0.smithi115.stdout: 991: throughput=5MB/sec pending data=0 2022-04-22T18:03:39.411 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y6GZpBMvf23ngeZ off=3046696 len=1511482 2022-04-22T18:03:39.420 INFO:tasks.workunit.client.0.smithi115.stdout:op 5037 completed, throughput=5MB/sec 2022-04-22T18:03:39.421 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mhga63YvxaOh64v off=2635828 len=893713 2022-04-22T18:03:39.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 5038 completed, throughput=5MB/sec 2022-04-22T18:03:39.421 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-q5sGSqRxkzigVbD off=385598 len=1140503 2022-04-22T18:03:39.424 INFO:tasks.workunit.client.0.smithi115.stdout:op 5039 completed, throughput=5MB/sec 2022-04-22T18:03:39.424 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RhOYVbC1FljC9Z6 off=7922889 len=809138 2022-04-22T18:03:39.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 5040 completed, throughput=5MB/sec 2022-04-22T18:03:39.430 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q-ylzNfxBQW1sNI off=2251211 len=1319199 2022-04-22T18:03:39.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 5041 completed, throughput=5MB/sec 2022-04-22T18:03:40.438 INFO:tasks.workunit.client.0.smithi115.stdout: 992: throughput=5MB/sec pending data=0 2022-04-22T18:03:40.438 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kBUWFrH6LjVxb5V off=1299473 len=1101849 2022-04-22T18:03:40.441 INFO:tasks.workunit.client.0.smithi115.stdout:op 5042 completed, throughput=5MB/sec 2022-04-22T18:03:40.441 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kM93EH_laRLSwac off=4910956 len=992674 2022-04-22T18:03:40.443 INFO:tasks.workunit.client.0.smithi115.stdout:op 5043 completed, throughput=5MB/sec 2022-04-22T18:03:40.444 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XYedXN4LYuJ1xvW off=3588112 len=152815 2022-04-22T18:03:40.444 INFO:tasks.workunit.client.0.smithi115.stdout:op 5044 completed, throughput=5MB/sec 2022-04-22T18:03:40.444 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zB7wQotMyHjA3SB off=769261 len=583745 2022-04-22T18:03:40.451 INFO:tasks.workunit.client.0.smithi115.stdout:op 5045 completed, throughput=5MB/sec 2022-04-22T18:03:40.451 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xh1zPBJ8liWiK52 off=2737757 len=1795513 2022-04-22T18:03:40.454 INFO:tasks.workunit.client.0.smithi115.stdout:op 5046 completed, throughput=5MB/sec 2022-04-22T18:03:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:40 smithi121 bash[36902]: cluster 2022-04-22T18:03:38.795860+0000 mgr.y (mgr.34107) 1332 : cluster [DBG] pgmap v1908: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:40 smithi115 bash[45495]: cluster 2022-04-22T18:03:38.795860+0000 mgr.y (mgr.34107) 1332 : cluster [DBG] pgmap v1908: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:40 smithi115 bash[46934]: cluster 2022-04-22T18:03:38.795860+0000 mgr.y (mgr.34107) 1332 : cluster [DBG] pgmap v1908: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:03:41.455 INFO:tasks.workunit.client.0.smithi115.stdout: 993: throughput=5MB/sec pending data=0 2022-04-22T18:03:41.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lvMJT6hKtxfD7_K off=19984921 len=227482 2022-04-22T18:03:41.459 INFO:tasks.workunit.client.0.smithi115.stdout:op 5047 completed, throughput=5MB/sec 2022-04-22T18:03:41.460 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BMOOeBl759cfR8s off=8003333 len=502954 2022-04-22T18:03:41.461 INFO:tasks.workunit.client.0.smithi115.stdout:op 5048 completed, throughput=5MB/sec 2022-04-22T18:03:41.462 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bESD0YvREz1W5se off=11874666 len=234481 2022-04-22T18:03:41.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 5049 completed, throughput=5MB/sec 2022-04-22T18:03:41.466 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rX0IDxtn2embWWw off=206896 len=1681666 2022-04-22T18:03:41.469 INFO:tasks.workunit.client.0.smithi115.stdout:op 5050 completed, throughput=5MB/sec 2022-04-22T18:03:41.470 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IsX2kwYMJJ5Ir7u off=7872165 len=34296 2022-04-22T18:03:41.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 5051 completed, throughput=5MB/sec 2022-04-22T18:03:41.471 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kH8AGyD8EKu0jzW off=173859 len=1688036 2022-04-22T18:03:41.478 INFO:tasks.workunit.client.0.smithi115.stdout:op 5052 completed, throughput=5MB/sec 2022-04-22T18:03:41.478 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Aql61piHzkNUQ8p off=4350171 len=1922513 2022-04-22T18:03:41.487 INFO:tasks.workunit.client.0.smithi115.stdout:op 5053 completed, throughput=5MB/sec 2022-04-22T18:03:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:41 smithi121 bash[36902]: audit 2022-04-22T18:03:40.390183+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:41 smithi121 bash[36902]: audit 2022-04-22T18:03:40.784980+0000 mon.a (mon.0) 2869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:41 smithi115 bash[45495]: audit 2022-04-22T18:03:40.390183+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:41 smithi115 bash[45495]: audit 2022-04-22T18:03:40.784980+0000 mon.a (mon.0) 2869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:41 smithi115 bash[46934]: audit 2022-04-22T18:03:40.390183+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:41 smithi115 bash[46934]: audit 2022-04-22T18:03:40.784980+0000 mon.a (mon.0) 2869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:41.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:42.488 INFO:tasks.workunit.client.0.smithi115.stdout: 994: throughput=5MB/sec pending data=0 2022-04-22T18:03:42.488 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eXymiVMEsq3wX_I off=1957282 len=2095195 2022-04-22T18:03:42.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 5054 completed, throughput=5MB/sec 2022-04-22T18:03:42.537 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QSFdOFgDoWZosQ5 off=5937144 len=537936 2022-04-22T18:03:42.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:42 smithi121 bash[36902]: cluster 2022-04-22T18:03:40.796845+0000 mgr.y (mgr.34107) 1333 : cluster [DBG] pgmap v1909: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:03:42.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 5055 completed, throughput=5MB/sec 2022-04-22T18:03:42.539 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r6L_waxArROOpZg off=968019 len=1181684 2022-04-22T18:03:42.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 5056 completed, throughput=5MB/sec 2022-04-22T18:03:42.539 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W4svtHZqOX_TuzF off=10134595 len=1577799 2022-04-22T18:03:42.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 5057 completed, throughput=5MB/sec 2022-04-22T18:03:42.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:42 smithi115 bash[45495]: cluster 2022-04-22T18:03:40.796845+0000 mgr.y (mgr.34107) 1333 : cluster [DBG] pgmap v1909: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:03:42.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:42 smithi115 bash[46934]: cluster 2022-04-22T18:03:40.796845+0000 mgr.y (mgr.34107) 1333 : cluster [DBG] pgmap v1909: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:03:43.514 INFO:tasks.workunit.client.0.smithi115.stdout: 995: throughput=5MB/sec pending data=0 2022-04-22T18:03:43.514 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FW83wx59PsXte2M off=13438695 len=1132733 2022-04-22T18:03:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:43 smithi121 bash[36902]: audit 2022-04-22T18:03:42.777859+0000 mon.a (mon.0) 2870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:43.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 5058 completed, throughput=5MB/sec 2022-04-22T18:03:43.518 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bVMJUPNDxSC44z9 off=1783774 len=1171099 2022-04-22T18:03:43.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 5059 completed, throughput=5MB/sec 2022-04-22T18:03:43.523 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6om2t2__9KJvIs2 off=2358121 len=1570799 2022-04-22T18:03:43.530 INFO:tasks.workunit.client.0.smithi115.stdout:op 5060 completed, throughput=5MB/sec 2022-04-22T18:03:43.530 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-s9NOZhCGoWo1lR4 off=0 len=1833379 2022-04-22T18:03:43.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 5061 completed, throughput=5MB/sec 2022-04-22T18:03:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:43 smithi115 bash[45495]: audit 2022-04-22T18:03:42.777859+0000 mon.a (mon.0) 2870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:43 smithi115 bash[46934]: audit 2022-04-22T18:03:42.777859+0000 mon.a (mon.0) 2870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:43.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:44 smithi121 bash[36902]: cluster 2022-04-22T18:03:42.798012+0000 mgr.y (mgr.34107) 1334 : cluster [DBG] pgmap v1910: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:44.534 INFO:tasks.workunit.client.0.smithi115.stdout: 996: throughput=5MB/sec pending data=0 2022-04-22T18:03:44.534 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PHQu-7qgcWtNLJF off=9168364 len=724704 2022-04-22T18:03:44.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 5062 completed, throughput=5MB/sec 2022-04-22T18:03:44.537 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--5KID7xfs0yDZVy off=15231254 len=986864 2022-04-22T18:03:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:44 smithi115 bash[45495]: cluster 2022-04-22T18:03:42.798012+0000 mgr.y (mgr.34107) 1334 : cluster [DBG] pgmap v1910: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:44 smithi115 bash[46934]: cluster 2022-04-22T18:03:42.798012+0000 mgr.y (mgr.34107) 1334 : cluster [DBG] pgmap v1910: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:44.544 INFO:tasks.workunit.client.0.smithi115.stdout:op 5063 completed, throughput=5MB/sec 2022-04-22T18:03:44.545 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ESWiHSw8Kbd-pcO off=2766827 len=1557706 2022-04-22T18:03:44.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 5064 completed, throughput=5MB/sec 2022-04-22T18:03:44.552 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W187GFp5Q8-I5lv off=20225217 len=1563370 2022-04-22T18:03:44.560 INFO:tasks.workunit.client.0.smithi115.stdout:op 5065 completed, throughput=5MB/sec 2022-04-22T18:03:45.561 INFO:tasks.workunit.client.0.smithi115.stdout: 997: throughput=5MB/sec pending data=0 2022-04-22T18:03:45.561 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JGjK6ANzbYiwpsm off=2294753 len=1528369 2022-04-22T18:03:45.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 5066 completed, throughput=5MB/sec 2022-04-22T18:03:45.570 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CaTFVxv5wai8cX9 off=14902342 len=275619 2022-04-22T18:03:45.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 5067 completed, throughput=5MB/sec 2022-04-22T18:03:45.575 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f2JbwgGMmsfd9uM off=5444863 len=401242 2022-04-22T18:03:45.579 INFO:tasks.workunit.client.0.smithi115.stdout:op 5068 completed, throughput=5MB/sec 2022-04-22T18:03:45.579 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z4jUak-zUuOK5XV off=0 len=1109277 2022-04-22T18:03:45.586 INFO:tasks.workunit.client.0.smithi115.stdout:op 5069 completed, throughput=5MB/sec 2022-04-22T18:03:45.586 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DiRvszBX-0cdFgN off=1342890 len=1642267 2022-04-22T18:03:45.594 INFO:tasks.workunit.client.0.smithi115.stdout:op 5070 completed, throughput=5MB/sec 2022-04-22T18:03:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:46 smithi121 bash[36902]: cluster 2022-04-22T18:03:44.799148+0000 mgr.y (mgr.34107) 1335 : cluster [DBG] pgmap v1911: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:03:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:46 smithi121 bash[36902]: audit 2022-04-22T18:03:45.393191+0000 mon.a (mon.0) 2871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:46 smithi121 bash[36902]: audit 2022-04-22T18:03:45.788164+0000 mon.a (mon.0) 2872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[45495]: cluster 2022-04-22T18:03:44.799148+0000 mgr.y (mgr.34107) 1335 : cluster [DBG] pgmap v1911: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:03:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[45495]: audit 2022-04-22T18:03:45.393191+0000 mon.a (mon.0) 2871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[45495]: audit 2022-04-22T18:03:45.788164+0000 mon.a (mon.0) 2872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[46934]: cluster 2022-04-22T18:03:44.799148+0000 mgr.y (mgr.34107) 1335 : cluster [DBG] pgmap v1911: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:03:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[46934]: audit 2022-04-22T18:03:45.393191+0000 mon.a (mon.0) 2871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:46 smithi115 bash[46934]: audit 2022-04-22T18:03:45.788164+0000 mon.a (mon.0) 2872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:46.594 INFO:tasks.workunit.client.0.smithi115.stdout: 998: throughput=5MB/sec pending data=0 2022-04-22T18:03:46.594 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vOPeYqLjeBpGLrJ off=4717100 len=620855 2022-04-22T18:03:46.600 INFO:tasks.workunit.client.0.smithi115.stdout:op 5071 completed, throughput=5MB/sec 2022-04-22T18:03:46.601 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6M1joxQfz8cenLH off=3643130 len=1151060 2022-04-22T18:03:46.604 INFO:tasks.workunit.client.0.smithi115.stdout:op 5072 completed, throughput=5MB/sec 2022-04-22T18:03:46.604 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F_bij3eeRC1pH8D off=858271 len=1692785 2022-04-22T18:03:46.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 5073 completed, throughput=5MB/sec 2022-04-22T18:03:46.614 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SUcVpWyiZZGdwhu off=9977416 len=1035990 2022-04-22T18:03:46.619 INFO:tasks.workunit.client.0.smithi115.stdout:op 5074 completed, throughput=5MB/sec 2022-04-22T18:03:46.619 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rC4aSWfrOOOZgB2 off=4175035 len=230070 2022-04-22T18:03:46.620 INFO:tasks.workunit.client.0.smithi115.stdout:op 5075 completed, throughput=5MB/sec 2022-04-22T18:03:46.620 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UIULrcaSqetdR3q off=5707094 len=643078 2022-04-22T18:03:46.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 5076 completed, throughput=5MB/sec 2022-04-22T18:03:47.623 INFO:tasks.workunit.client.0.smithi115.stdout: 999: throughput=5MB/sec pending data=0 2022-04-22T18:03:47.623 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0-dVrpvcx3E9g-A off=11425404 len=164211 2022-04-22T18:03:47.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 5077 completed, throughput=5MB/sec 2022-04-22T18:03:47.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vK_pasHf6iva5Tp off=6488291 len=262172 2022-04-22T18:03:47.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5078 completed, throughput=5MB/sec 2022-04-22T18:03:47.625 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FkHD2gvVWV7nAa- off=11357235 len=103068 2022-04-22T18:03:47.630 INFO:tasks.workunit.client.0.smithi115.stdout:op 5079 completed, throughput=5MB/sec 2022-04-22T18:03:47.630 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e6I42aQ3mdtzgdM off=20034663 len=55400 2022-04-22T18:03:47.631 INFO:tasks.workunit.client.0.smithi115.stdout:op 5080 completed, throughput=5MB/sec 2022-04-22T18:03:47.631 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zyu0f7WrRI5sGV- off=4130079 len=2045097 2022-04-22T18:03:47.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 5081 completed, throughput=5MB/sec 2022-04-22T18:03:47.635 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R9NgaPIEciKw2Q2 off=1615560 len=1212579 2022-04-22T18:03:47.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 5082 completed, throughput=5MB/sec 2022-04-22T18:03:47.638 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sIUFfhy6r6mXBqM off=0 len=1446135 2022-04-22T18:03:47.647 INFO:tasks.workunit.client.0.smithi115.stdout:op 5083 completed, throughput=5MB/sec 2022-04-22T18:03:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:48 smithi121 bash[36902]: cluster 2022-04-22T18:03:46.800286+0000 mgr.y (mgr.34107) 1336 : cluster [DBG] pgmap v1912: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:48 smithi121 bash[36902]: audit 2022-04-22T18:03:47.781297+0000 mon.a (mon.0) 2873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:48 smithi115 bash[45495]: cluster 2022-04-22T18:03:46.800286+0000 mgr.y (mgr.34107) 1336 : cluster [DBG] pgmap v1912: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:48 smithi115 bash[45495]: audit 2022-04-22T18:03:47.781297+0000 mon.a (mon.0) 2873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:48 smithi115 bash[46934]: cluster 2022-04-22T18:03:46.800286+0000 mgr.y (mgr.34107) 1336 : cluster [DBG] pgmap v1912: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:03:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:48 smithi115 bash[46934]: audit 2022-04-22T18:03:47.781297+0000 mon.a (mon.0) 2873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:48.651 INFO:tasks.workunit.client.0.smithi115.stdout: 1000: throughput=5MB/sec pending data=0 2022-04-22T18:03:48.651 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hBqcnuCxHjWQzW0 off=4491408 len=267748 2022-04-22T18:03:48.651 INFO:tasks.workunit.client.0.smithi115.stdout:op 5084 completed, throughput=5MB/sec 2022-04-22T18:03:48.651 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DrTWFWYZyIgX1Md off=11466262 len=786827 2022-04-22T18:03:48.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 5085 completed, throughput=5MB/sec 2022-04-22T18:03:48.652 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8qKS5z2Y0ym4acp off=3490386 len=973530 2022-04-22T18:03:48.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 5086 completed, throughput=5MB/sec 2022-04-22T18:03:48.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GG9iVXaMTb2ERhh off=2099724 len=1138697 2022-04-22T18:03:48.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 5087 completed, throughput=5MB/sec 2022-04-22T18:03:48.662 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-34luFs9aSwHHC86 off=3488874 len=1382679 2022-04-22T18:03:48.664 INFO:tasks.workunit.client.0.smithi115.stdout:op 5088 completed, throughput=5MB/sec 2022-04-22T18:03:48.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-I7mBStvJ87Fu-93 off=790608 len=1957439 2022-04-22T18:03:48.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 5089 completed, throughput=5MB/sec 2022-04-22T18:03:49.669 INFO:tasks.workunit.client.0.smithi115.stdout: 1001: throughput=5MB/sec pending data=0 2022-04-22T18:03:49.669 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uuzE4ohAQX90w4z off=5229569 len=1218341 2022-04-22T18:03:49.672 INFO:tasks.workunit.client.0.smithi115.stdout:op 5090 completed, throughput=5MB/sec 2022-04-22T18:03:49.672 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tc4uggHepRI_yAS off=11744611 len=1660759 2022-04-22T18:03:49.676 INFO:tasks.workunit.client.0.smithi115.stdout:op 5091 completed, throughput=5MB/sec 2022-04-22T18:03:49.676 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0FxvEiSSRu9detf off=1850350 len=1241460 2022-04-22T18:03:49.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 5092 completed, throughput=5MB/sec 2022-04-22T18:03:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:50 smithi121 bash[36902]: cluster 2022-04-22T18:03:48.801463+0000 mgr.y (mgr.34107) 1337 : cluster [DBG] pgmap v1913: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:50 smithi115 bash[45495]: cluster 2022-04-22T18:03:48.801463+0000 mgr.y (mgr.34107) 1337 : cluster [DBG] pgmap v1913: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:50 smithi115 bash[46934]: cluster 2022-04-22T18:03:48.801463+0000 mgr.y (mgr.34107) 1337 : cluster [DBG] pgmap v1913: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 5 op/s 2022-04-22T18:03:50.680 INFO:tasks.workunit.client.0.smithi115.stdout: 1002: throughput=5MB/sec pending data=0 2022-04-22T18:03:50.696 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QyHzNltjYEtyZov off=11954540 len=344616 2022-04-22T18:03:50.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 5093 completed, throughput=5MB/sec 2022-04-22T18:03:50.698 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MwTTHJRv6vH_Zye off=1529040 len=42226 2022-04-22T18:03:50.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 5094 completed, throughput=5MB/sec 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O_qhOpevPQBf0PU off=19936730 len=382219 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 5095 completed, throughput=5MB/sec 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-orxVGQphSRHR61z off=4190304 len=1997516 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 5096 completed, throughput=5MB/sec 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-c_x46oVuOx8a7AQ off=1150424 len=120458 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:op 5097 completed, throughput=5MB/sec 2022-04-22T18:03:50.699 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-x2j93dRLVgLJrw2 off=2083284 len=1362878 2022-04-22T18:03:50.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 5098 completed, throughput=5MB/sec 2022-04-22T18:03:50.700 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pHjQIt0BmwqQSId off=11246407 len=1310032 2022-04-22T18:03:50.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 5099 completed, throughput=5MB/sec 2022-04-22T18:03:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:51 smithi121 bash[36902]: audit 2022-04-22T18:03:50.395907+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:51 smithi121 bash[36902]: audit 2022-04-22T18:03:50.791846+0000 mon.a (mon.0) 2875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:51 smithi115 bash[45495]: audit 2022-04-22T18:03:50.395907+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:51 smithi115 bash[45495]: audit 2022-04-22T18:03:50.791846+0000 mon.a (mon.0) 2875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:51 smithi115 bash[46934]: audit 2022-04-22T18:03:50.395907+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:51 smithi115 bash[46934]: audit 2022-04-22T18:03:50.791846+0000 mon.a (mon.0) 2875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:51.700 INFO:tasks.workunit.client.0.smithi115.stdout: 1003: throughput=5MB/sec pending data=0 2022-04-22T18:03:51.700 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GdFNsuaPm9ayP2O off=583908 len=1389171 2022-04-22T18:03:51.703 INFO:tasks.workunit.client.0.smithi115.stdout:op 5100 completed, throughput=5MB/sec 2022-04-22T18:03:51.704 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QUCxkuw0Bzpyf_N off=7537738 len=661842 2022-04-22T18:03:51.710 INFO:tasks.workunit.client.0.smithi115.stdout:op 5101 completed, throughput=5MB/sec 2022-04-22T18:03:51.710 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JyvO5_3uKkuxEUK off=12148110 len=367986 2022-04-22T18:03:51.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 5102 completed, throughput=5MB/sec 2022-04-22T18:03:51.712 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VnIyi3C6jgrsi0q off=12163595 len=1648036 2022-04-22T18:03:51.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 5103 completed, throughput=5MB/sec 2022-04-22T18:03:51.717 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AL_A3K-nBV_F_Zo off=11551424 len=839902 2022-04-22T18:03:51.719 INFO:tasks.workunit.client.0.smithi115.stdout:op 5104 completed, throughput=5MB/sec 2022-04-22T18:03:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:51.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:52 smithi121 bash[36902]: cluster 2022-04-22T18:03:50.802455+0000 mgr.y (mgr.34107) 1338 : cluster [DBG] pgmap v1914: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:52 smithi121 bash[36902]: audit 2022-04-22T18:03:51.740115+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:03:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:52 smithi121 bash[36902]: audit 2022-04-22T18:03:51.742938+0000 mon.a (mon.0) 2877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:03:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:52 smithi121 bash[36902]: audit 2022-04-22T18:03:51.744701+0000 mon.a (mon.0) 2878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:03:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:52 smithi121 bash[36902]: audit 2022-04-22T18:03:52.115827+0000 mon.a (mon.0) 2879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:03:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[45495]: cluster 2022-04-22T18:03:50.802455+0000 mgr.y (mgr.34107) 1338 : cluster [DBG] pgmap v1914: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[45495]: audit 2022-04-22T18:03:51.740115+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:03:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[45495]: audit 2022-04-22T18:03:51.742938+0000 mon.a (mon.0) 2877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:03:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[45495]: audit 2022-04-22T18:03:51.744701+0000 mon.a (mon.0) 2878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:03:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[45495]: audit 2022-04-22T18:03:52.115827+0000 mon.a (mon.0) 2879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:03:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[46934]: cluster 2022-04-22T18:03:50.802455+0000 mgr.y (mgr.34107) 1338 : cluster [DBG] pgmap v1914: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:03:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[46934]: audit 2022-04-22T18:03:51.740115+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:03:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[46934]: audit 2022-04-22T18:03:51.742938+0000 mon.a (mon.0) 2877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:03:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[46934]: audit 2022-04-22T18:03:51.744701+0000 mon.a (mon.0) 2878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:03:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:52 smithi115 bash[46934]: audit 2022-04-22T18:03:52.115827+0000 mon.a (mon.0) 2879 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:03:52.719 INFO:tasks.workunit.client.0.smithi115.stdout: 1004: throughput=5MB/sec pending data=0 2022-04-22T18:03:52.719 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DBq1a5QdTP_Ltmy off=13295362 len=1409002 2022-04-22T18:03:52.723 INFO:tasks.workunit.client.0.smithi115.stdout:op 5105 completed, throughput=5MB/sec 2022-04-22T18:03:52.723 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YHcbDRaxJ-8br4J off=7238853 len=1916467 2022-04-22T18:03:52.727 INFO:tasks.workunit.client.0.smithi115.stdout:op 5106 completed, throughput=5MB/sec 2022-04-22T18:03:52.727 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tLozese3vhaCE47 off=1206504 len=634955 2022-04-22T18:03:52.733 INFO:tasks.workunit.client.0.smithi115.stdout:op 5107 completed, throughput=5MB/sec 2022-04-22T18:03:52.733 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zKDI3g5rLwcA2hC off=5992993 len=651040 2022-04-22T18:03:52.735 INFO:tasks.workunit.client.0.smithi115.stdout:op 5108 completed, throughput=5MB/sec 2022-04-22T18:03:52.735 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qOvXZkG1s_VOBqL off=18918984 len=6503 2022-04-22T18:03:52.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 5109 completed, throughput=5MB/sec 2022-04-22T18:03:52.738 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JpApCbmkqjxK4x4 off=10748284 len=766513 2022-04-22T18:03:52.742 INFO:tasks.workunit.client.0.smithi115.stdout:op 5110 completed, throughput=5MB/sec 2022-04-22T18:03:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:53 smithi121 bash[36902]: audit 2022-04-22T18:03:52.784587+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:53 smithi115 bash[45495]: audit 2022-04-22T18:03:52.784587+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:53 smithi115 bash[46934]: audit 2022-04-22T18:03:52.784587+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:53.743 INFO:tasks.workunit.client.0.smithi115.stdout: 1005: throughput=5MB/sec pending data=0 2022-04-22T18:03:53.743 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QSTolNJ1ls-aIfR off=610921 len=1613175 2022-04-22T18:03:53.746 INFO:tasks.workunit.client.0.smithi115.stdout:op 5111 completed, throughput=5MB/sec 2022-04-22T18:03:53.746 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dDdioAw1V10Lw8P off=6486662 len=1846162 2022-04-22T18:03:53.750 INFO:tasks.workunit.client.0.smithi115.stdout:op 5112 completed, throughput=5MB/sec 2022-04-22T18:03:53.751 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hReUJvRkUfHN_0S off=14961133 len=1343965 2022-04-22T18:03:53.760 INFO:tasks.workunit.client.0.smithi115.stdout:op 5113 completed, throughput=5MB/sec 2022-04-22T18:03:53.760 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sA_bTxgpz6B4yJj off=449459 len=1783648 2022-04-22T18:03:53.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 5114 completed, throughput=5MB/sec 2022-04-22T18:03:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:03:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:03:53.619Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:03:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:54 smithi121 bash[36902]: cluster 2022-04-22T18:03:52.803553+0000 mgr.y (mgr.34107) 1339 : cluster [DBG] pgmap v1915: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:03:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:54 smithi115 bash[45495]: cluster 2022-04-22T18:03:52.803553+0000 mgr.y (mgr.34107) 1339 : cluster [DBG] pgmap v1915: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:03:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:54 smithi115 bash[46934]: cluster 2022-04-22T18:03:52.803553+0000 mgr.y (mgr.34107) 1339 : cluster [DBG] pgmap v1915: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:03:54.763 INFO:tasks.workunit.client.0.smithi115.stdout: 1006: throughput=5MB/sec pending data=0 2022-04-22T18:03:54.763 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LAb0RaYtRtvxGwx off=21591474 len=545053 2022-04-22T18:03:54.765 INFO:tasks.workunit.client.0.smithi115.stdout:op 5115 completed, throughput=5MB/sec 2022-04-22T18:03:54.765 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qQxXW9XhRQcdOY8 off=8987912 len=592278 2022-04-22T18:03:54.768 INFO:tasks.workunit.client.0.smithi115.stdout:op 5116 completed, throughput=5MB/sec 2022-04-22T18:03:54.768 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zge75bp2jb1vmIY off=1664666 len=615610 2022-04-22T18:03:54.770 INFO:tasks.workunit.client.0.smithi115.stdout:op 5117 completed, throughput=5MB/sec 2022-04-22T18:03:54.770 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0Zz8jnJOnkjIAMU off=2544597 len=1336064 2022-04-22T18:03:54.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 5118 completed, throughput=5MB/sec 2022-04-22T18:03:54.778 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-W1HAuogJGqmOArZ off=14720 len=762547 2022-04-22T18:03:54.780 INFO:tasks.workunit.client.0.smithi115.stdout:op 5119 completed, throughput=5MB/sec 2022-04-22T18:03:54.780 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-T33Agmws6BP4vHx off=14414880 len=1882552 2022-04-22T18:03:54.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 5120 completed, throughput=5MB/sec 2022-04-22T18:03:55.786 INFO:tasks.workunit.client.0.smithi115.stdout: 1007: throughput=5MB/sec pending data=0 2022-04-22T18:03:55.786 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mfDEBmHrxTfGQ8d off=12864241 len=800420 2022-04-22T18:03:55.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 5121 completed, throughput=5MB/sec 2022-04-22T18:03:55.793 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--YpglVihh8OFOrS off=3503080 len=879620 2022-04-22T18:03:55.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 5122 completed, throughput=5MB/sec 2022-04-22T18:03:55.800 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-klidM_bl69QagOq off=702996 len=737752 2022-04-22T18:03:55.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 5123 completed, throughput=5MB/sec 2022-04-22T18:03:55.801 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RuvFMyZ3ucXfWqC off=688335 len=1090772 2022-04-22T18:03:55.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 5124 completed, throughput=5MB/sec 2022-04-22T18:03:55.808 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mSWGg6XFCTJCYQU off=4287418 len=689387 2022-04-22T18:03:55.812 INFO:tasks.workunit.client.0.smithi115.stdout:op 5125 completed, throughput=5MB/sec 2022-04-22T18:03:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:56 smithi121 bash[36902]: cluster 2022-04-22T18:03:54.804569+0000 mgr.y (mgr.34107) 1340 : cluster [DBG] pgmap v1916: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:03:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:56 smithi121 bash[36902]: audit 2022-04-22T18:03:55.399138+0000 mon.a (mon.0) 2881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:56.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:56 smithi121 bash[36902]: audit 2022-04-22T18:03:55.795154+0000 mon.a (mon.0) 2882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[45495]: cluster 2022-04-22T18:03:54.804569+0000 mgr.y (mgr.34107) 1340 : cluster [DBG] pgmap v1916: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:03:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[45495]: audit 2022-04-22T18:03:55.399138+0000 mon.a (mon.0) 2881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[45495]: audit 2022-04-22T18:03:55.795154+0000 mon.a (mon.0) 2882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[46934]: cluster 2022-04-22T18:03:54.804569+0000 mgr.y (mgr.34107) 1340 : cluster [DBG] pgmap v1916: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:03:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[46934]: audit 2022-04-22T18:03:55.399138+0000 mon.a (mon.0) 2881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:03:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:56 smithi115 bash[46934]: audit 2022-04-22T18:03:55.795154+0000 mon.a (mon.0) 2882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:03:56.812 INFO:tasks.workunit.client.0.smithi115.stdout: 1008: throughput=5MB/sec pending data=0 2022-04-22T18:03:56.813 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F-UPHUevdaAPNA2 off=6119217 len=1792063 2022-04-22T18:03:56.816 INFO:tasks.workunit.client.0.smithi115.stdout:op 5126 completed, throughput=5MB/sec 2022-04-22T18:03:56.816 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xwpnQKIzlyCsOy- off=6471666 len=1253707 2022-04-22T18:03:56.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 5127 completed, throughput=5MB/sec 2022-04-22T18:03:56.823 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FMQNSgGeqSmKpEI off=4260996 len=1729571 2022-04-22T18:03:56.828 INFO:tasks.workunit.client.0.smithi115.stdout:op 5128 completed, throughput=5MB/sec 2022-04-22T18:03:57.829 INFO:tasks.workunit.client.0.smithi115.stdout: 1009: throughput=5MB/sec pending data=0 2022-04-22T18:03:57.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LIVSPYDlzq88njL off=10013858 len=685263 2022-04-22T18:03:57.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 5129 completed, throughput=5MB/sec 2022-04-22T18:03:57.836 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3nqTJP6D0xVuKEN off=1902222 len=1172089 2022-04-22T18:03:57.838 INFO:tasks.workunit.client.0.smithi115.stdout:op 5130 completed, throughput=5MB/sec 2022-04-22T18:03:57.838 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BKxbQPpWFR-HVxo off=19192386 len=479298 2022-04-22T18:03:57.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 5131 completed, throughput=5MB/sec 2022-04-22T18:03:57.844 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v34OVV3wWHYEVEl off=1275586 len=600101 2022-04-22T18:03:57.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 5132 completed, throughput=5MB/sec 2022-04-22T18:03:57.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sP4oEp3nO7ipr61 off=74917 len=38708 2022-04-22T18:03:57.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 5133 completed, throughput=5MB/sec 2022-04-22T18:03:57.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nPmcOOjhHZG9w6b off=12197470 len=79242 2022-04-22T18:03:57.849 INFO:tasks.workunit.client.0.smithi115.stdout:op 5134 completed, throughput=5MB/sec 2022-04-22T18:03:57.850 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OHUvDny_Y0Hz485 off=4162170 len=1930375 2022-04-22T18:03:57.855 INFO:tasks.workunit.client.0.smithi115.stdout:op 5135 completed, throughput=5MB/sec 2022-04-22T18:03:57.855 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CUjzVN0Uo_OJ-Tn off=6831376 len=59221 2022-04-22T18:03:57.858 INFO:tasks.workunit.client.0.smithi115.stdout:op 5136 completed, throughput=5MB/sec 2022-04-22T18:03:57.858 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Bg8bNAPTbtbI_pq off=11926124 len=1629341 2022-04-22T18:03:57.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 5137 completed, throughput=5MB/sec 2022-04-22T18:03:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:58 smithi121 bash[36902]: cluster 2022-04-22T18:03:56.805645+0000 mgr.y (mgr.34107) 1341 : cluster [DBG] pgmap v1917: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:03:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:03:58 smithi121 bash[36902]: audit 2022-04-22T18:03:57.787847+0000 mon.a (mon.0) 2883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:58 smithi115 bash[45495]: cluster 2022-04-22T18:03:56.805645+0000 mgr.y (mgr.34107) 1341 : cluster [DBG] pgmap v1917: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:03:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:03:58 smithi115 bash[45495]: audit 2022-04-22T18:03:57.787847+0000 mon.a (mon.0) 2883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:58 smithi115 bash[46934]: cluster 2022-04-22T18:03:56.805645+0000 mgr.y (mgr.34107) 1341 : cluster [DBG] pgmap v1917: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:03:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:03:58 smithi115 bash[46934]: audit 2022-04-22T18:03:57.787847+0000 mon.a (mon.0) 2883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:03:58.866 INFO:tasks.workunit.client.0.smithi115.stdout: 1010: throughput=5MB/sec pending data=0 2022-04-22T18:03:58.866 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D5E9ihXx_48WZXf off=14797869 len=1066993 2022-04-22T18:03:58.873 INFO:tasks.workunit.client.0.smithi115.stdout:op 5138 completed, throughput=5MB/sec 2022-04-22T18:03:58.873 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4Sjnvrylc2ZSDKF off=9468748 len=946775 2022-04-22T18:03:58.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 5139 completed, throughput=5MB/sec 2022-04-22T18:03:58.987 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ff4GfCCHiKfb_LT off=5388885 len=6805 2022-04-22T18:03:58.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 5140 completed, throughput=5MB/sec 2022-04-22T18:03:58.987 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YNS8KL5SonezL9U off=11774308 len=205526 2022-04-22T18:03:58.987 INFO:tasks.workunit.client.0.smithi115.stdout:op 5141 completed, throughput=5MB/sec 2022-04-22T18:03:58.987 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o-xmLJpOQDSayn3 off=5159439 len=1438518 2022-04-22T18:03:58.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 5142 completed, throughput=5MB/sec 2022-04-22T18:03:58.988 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o_WagNifWg6YxU8 off=5994340 len=1192 2022-04-22T18:03:58.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 5143 completed, throughput=5MB/sec 2022-04-22T18:03:58.989 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WsAm2gB5W66BOTK off=2779559 len=801265 2022-04-22T18:03:58.989 INFO:tasks.workunit.client.0.smithi115.stdout:op 5144 completed, throughput=5MB/sec 2022-04-22T18:03:59.008 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:03:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (27m) 6m ago 40m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (40m) 7m ago 40m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (29m) 6m ago 41m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (28m) 7m ago 41m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (36m) 7m ago 46m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (36m) 6m ago 48m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (35m) 6m ago 48m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (35m) 7m ago 46m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:03:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (35m) 6m ago 46m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (28m) 6m ago 41m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (28m) 7m ago 41m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (33m) 6m ago 45m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (33m) 6m ago 45m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (32m) 6m ago 44m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (32m) 6m ago 44m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (31m) 7m ago 43m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (30m) 7m ago 43m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (30m) 7m ago 42m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:03:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (29m) 7m ago 42m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:03:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (27m) 7m ago 41m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:59.758 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:03:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:03:59.891 INFO:tasks.workunit.client.0.smithi115.stdout: 1011: throughput=5MB/sec pending data=0 2022-04-22T18:03:59.891 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w51XwrytPhXxkqk off=4468194 len=175045 2022-04-22T18:03:59.896 INFO:tasks.workunit.client.0.smithi115.stdout:op 5145 completed, throughput=5MB/sec 2022-04-22T18:03:59.896 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8SVBNLdOjkvnd29 off=15691657 len=924720 2022-04-22T18:03:59.899 INFO:tasks.workunit.client.0.smithi115.stdout:op 5146 completed, throughput=5MB/sec 2022-04-22T18:03:59.900 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BPesmEzQzvj3ipE off=3605832 len=1216934 2022-04-22T18:03:59.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 5147 completed, throughput=5MB/sec 2022-04-22T18:03:59.907 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Io9UknhAiJvIuEe off=17635063 len=958429 2022-04-22T18:03:59.913 INFO:tasks.workunit.client.0.smithi115.stdout:op 5148 completed, throughput=5MB/sec 2022-04-22T18:03:59.913 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cFbrh4bbl8LyixT off=3499474 len=1150006 2022-04-22T18:03:59.918 INFO:tasks.workunit.client.0.smithi115.stdout:op 5149 completed, throughput=5MB/sec 2022-04-22T18:03:59.919 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0qOv0aozYCKzqBq off=611884 len=1367982 2022-04-22T18:03:59.921 INFO:tasks.workunit.client.0.smithi115.stdout:op 5150 completed, throughput=5MB/sec 2022-04-22T18:04:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:00 smithi121 bash[36902]: cluster 2022-04-22T18:03:58.806779+0000 mgr.y (mgr.34107) 1342 : cluster [DBG] pgmap v1918: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:04:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:00 smithi121 bash[36902]: audit 2022-04-22T18:03:58.985317+0000 mgr.y (mgr.34107) 1343 : audit [DBG] from='client.35805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:00 smithi121 bash[36902]: audit 2022-04-22T18:03:59.329245+0000 mgr.y (mgr.34107) 1344 : audit [DBG] from='client.45751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:00 smithi121 bash[36902]: audit 2022-04-22T18:03:59.751150+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? 172.21.15.115:0/694922756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[45495]: cluster 2022-04-22T18:03:58.806779+0000 mgr.y (mgr.34107) 1342 : cluster [DBG] pgmap v1918: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:04:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[45495]: audit 2022-04-22T18:03:58.985317+0000 mgr.y (mgr.34107) 1343 : audit [DBG] from='client.35805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[45495]: audit 2022-04-22T18:03:59.329245+0000 mgr.y (mgr.34107) 1344 : audit [DBG] from='client.45751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[45495]: audit 2022-04-22T18:03:59.751150+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? 172.21.15.115:0/694922756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[46934]: cluster 2022-04-22T18:03:58.806779+0000 mgr.y (mgr.34107) 1342 : cluster [DBG] pgmap v1918: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:04:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[46934]: audit 2022-04-22T18:03:58.985317+0000 mgr.y (mgr.34107) 1343 : audit [DBG] from='client.35805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[46934]: audit 2022-04-22T18:03:59.329245+0000 mgr.y (mgr.34107) 1344 : audit [DBG] from='client.45751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:00.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:00 smithi115 bash[46934]: audit 2022-04-22T18:03:59.751150+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? 172.21.15.115:0/694922756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:00.921 INFO:tasks.workunit.client.0.smithi115.stdout: 1012: throughput=5MB/sec pending data=0 2022-04-22T18:04:00.922 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EtKm-6E2vTPh7WU off=15118318 len=1569337 2022-04-22T18:04:00.929 INFO:tasks.workunit.client.0.smithi115.stdout:op 5151 completed, throughput=5MB/sec 2022-04-22T18:04:00.930 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JjgTC_RYQt54a2D off=4397287 len=2076498 2022-04-22T18:04:00.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 5152 completed, throughput=5MB/sec 2022-04-22T18:04:00.934 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-N2mSejuPFz5WcTi off=4865475 len=782209 2022-04-22T18:04:00.935 INFO:tasks.workunit.client.0.smithi115.stdout:op 5153 completed, throughput=5MB/sec 2022-04-22T18:04:00.935 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m8e5J-paaKWOXrv off=214418 len=1244575 2022-04-22T18:04:00.941 INFO:tasks.workunit.client.0.smithi115.stdout:op 5154 completed, throughput=5MB/sec 2022-04-22T18:04:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:01 smithi121 bash[36902]: audit 2022-04-22T18:04:00.402082+0000 mon.a (mon.0) 2885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:01 smithi121 bash[36902]: audit 2022-04-22T18:04:00.798407+0000 mon.a (mon.0) 2886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:01 smithi115 bash[45495]: audit 2022-04-22T18:04:00.402082+0000 mon.a (mon.0) 2885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:01 smithi115 bash[45495]: audit 2022-04-22T18:04:00.798407+0000 mon.a (mon.0) 2886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:01 smithi115 bash[46934]: audit 2022-04-22T18:04:00.402082+0000 mon.a (mon.0) 2885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:01 smithi115 bash[46934]: audit 2022-04-22T18:04:00.798407+0000 mon.a (mon.0) 2886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:01.941 INFO:tasks.workunit.client.0.smithi115.stdout: 1013: throughput=5MB/sec pending data=0 2022-04-22T18:04:01.970 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5BaYB4SZhL3TEHi off=1304951 len=1128433 2022-04-22T18:04:01.971 INFO:tasks.workunit.client.0.smithi115.stdout:op 5155 completed, throughput=5MB/sec 2022-04-22T18:04:01.971 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-URKmpzHbBs3R9Mc off=6365621 len=19054 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5156 completed, throughput=5MB/sec 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TaFVKhat646V0LG off=9416469 len=1661125 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5157 completed, throughput=5MB/sec 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w_FzgbBEy1nn2Wr off=10384567 len=391386 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5158 completed, throughput=5MB/sec 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bNfbEQgsr6f-D3S off=6641590 len=1715097 2022-04-22T18:04:01.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5159 completed, throughput=5MB/sec 2022-04-22T18:04:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:01.738Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:02 smithi121 bash[36902]: cluster 2022-04-22T18:04:00.807853+0000 mgr.y (mgr.34107) 1345 : cluster [DBG] pgmap v1919: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2022-04-22T18:04:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:02 smithi115 bash[45495]: cluster 2022-04-22T18:04:00.807853+0000 mgr.y (mgr.34107) 1345 : cluster [DBG] pgmap v1919: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2022-04-22T18:04:02.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:02 smithi115 bash[46934]: cluster 2022-04-22T18:04:00.807853+0000 mgr.y (mgr.34107) 1345 : cluster [DBG] pgmap v1919: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 4 op/s 2022-04-22T18:04:02.965 INFO:tasks.workunit.client.0.smithi115.stdout: 1014: throughput=5MB/sec pending data=0 2022-04-22T18:04:02.965 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RlWvIdtpw_Vxo_Y off=2495501 len=1384600 2022-04-22T18:04:02.970 INFO:tasks.workunit.client.0.smithi115.stdout:op 5160 completed, throughput=5MB/sec 2022-04-22T18:04:02.970 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-So3sgzaZX2js7IM off=4693670 len=2011498 2022-04-22T18:04:02.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 5161 completed, throughput=5MB/sec 2022-04-22T18:04:02.976 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q1vC87vri3E5erG off=6051626 len=771347 2022-04-22T18:04:02.978 INFO:tasks.workunit.client.0.smithi115.stdout:op 5162 completed, throughput=5MB/sec 2022-04-22T18:04:02.978 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jf0sQFIZH0MwcIb off=22950934 len=1191533 2022-04-22T18:04:02.981 INFO:tasks.workunit.client.0.smithi115.stdout:op 5163 completed, throughput=5MB/sec 2022-04-22T18:04:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:03 smithi121 bash[36902]: audit 2022-04-22T18:04:02.790899+0000 mon.a (mon.0) 2887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:03 smithi115 bash[45495]: audit 2022-04-22T18:04:02.790899+0000 mon.a (mon.0) 2887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:03 smithi115 bash[46934]: audit 2022-04-22T18:04:02.790899+0000 mon.a (mon.0) 2887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:03.981 INFO:tasks.workunit.client.0.smithi115.stdout: 1015: throughput=5MB/sec pending data=0 2022-04-22T18:04:03.981 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gIZPSK9vUW4-cJZ off=2547155 len=1032779 2022-04-22T18:04:03.988 INFO:tasks.workunit.client.0.smithi115.stdout:op 5164 completed, throughput=5MB/sec 2022-04-22T18:04:03.988 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Bo2adeT3-g7t1ke off=1425977 len=749865 2022-04-22T18:04:03.990 INFO:tasks.workunit.client.0.smithi115.stdout:op 5165 completed, throughput=5MB/sec 2022-04-22T18:04:03.990 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1_ED3sandI5-SU1 off=9371934 len=957519 2022-04-22T18:04:03.997 INFO:tasks.workunit.client.0.smithi115.stdout:op 5166 completed, throughput=5MB/sec 2022-04-22T18:04:03.997 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3MQtn4-jaWz28U9 off=2438043 len=351688 2022-04-22T18:04:04.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 5167 completed, throughput=5MB/sec 2022-04-22T18:04:04.001 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7gmFaUBNvTPmLOd off=14860380 len=1332390 2022-04-22T18:04:04.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 5168 completed, throughput=5MB/sec 2022-04-22T18:04:04.008 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-PqiOZnpBPJB-8ey off=3727739 len=702314 2022-04-22T18:04:04.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 5169 completed, throughput=5MB/sec 2022-04-22T18:04:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:03.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:04 smithi121 bash[36902]: cluster 2022-04-22T18:04:02.808891+0000 mgr.y (mgr.34107) 1346 : cluster [DBG] pgmap v1920: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2022-04-22T18:04:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:04 smithi115 bash[45495]: cluster 2022-04-22T18:04:02.808891+0000 mgr.y (mgr.34107) 1346 : cluster [DBG] pgmap v1920: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2022-04-22T18:04:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:04 smithi115 bash[46934]: cluster 2022-04-22T18:04:02.808891+0000 mgr.y (mgr.34107) 1346 : cluster [DBG] pgmap v1920: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2022-04-22T18:04:05.013 INFO:tasks.workunit.client.0.smithi115.stdout: 1016: throughput=5MB/sec pending data=0 2022-04-22T18:04:05.013 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ozO6dUiYOSUwi-S off=14308888 len=1636857 2022-04-22T18:04:05.017 INFO:tasks.workunit.client.0.smithi115.stdout:op 5170 completed, throughput=5MB/sec 2022-04-22T18:04:05.017 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8OzJrQde4afDU5g off=3694685 len=253361 2022-04-22T18:04:05.019 INFO:tasks.workunit.client.0.smithi115.stdout:op 5171 completed, throughput=5MB/sec 2022-04-22T18:04:05.019 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_JPFktm4OpKSdaK off=5555654 len=725708 2022-04-22T18:04:05.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 5172 completed, throughput=5MB/sec 2022-04-22T18:04:05.026 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Aka2XZ1yS8Y9Pqw off=1376002 len=741849 2022-04-22T18:04:05.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 5173 completed, throughput=5MB/sec 2022-04-22T18:04:05.032 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YLNccFjX3e-kLtV off=15101700 len=877824 2022-04-22T18:04:05.034 INFO:tasks.workunit.client.0.smithi115.stdout:op 5174 completed, throughput=5MB/sec 2022-04-22T18:04:05.034 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QwSEpMVII0e5tb4 off=8324355 len=996992 2022-04-22T18:04:05.041 INFO:tasks.workunit.client.0.smithi115.stdout:op 5175 completed, throughput=5MB/sec 2022-04-22T18:04:06.041 INFO:tasks.workunit.client.0.smithi115.stdout: 1017: throughput=5MB/sec pending data=0 2022-04-22T18:04:06.042 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AMYjhFtDaqOrn-9 off=6498894 len=755518 2022-04-22T18:04:06.048 INFO:tasks.workunit.client.0.smithi115.stdout:op 5176 completed, throughput=5MB/sec 2022-04-22T18:04:06.048 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TVaLFgnEx1Ey7tJ off=2890140 len=1717386 2022-04-22T18:04:06.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 5177 completed, throughput=5MB/sec 2022-04-22T18:04:06.058 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LTwG7dqN1xqrgCA off=5279006 len=133768 2022-04-22T18:04:06.061 INFO:tasks.workunit.client.0.smithi115.stdout:op 5178 completed, throughput=5MB/sec 2022-04-22T18:04:06.061 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_PgKOrwj4-2PBdY off=7198067 len=1159915 2022-04-22T18:04:06.067 INFO:tasks.workunit.client.0.smithi115.stdout:op 5179 completed, throughput=5MB/sec 2022-04-22T18:04:06.068 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EmluYb_Df5CssE8 off=2050723 len=1638773 2022-04-22T18:04:06.071 INFO:tasks.workunit.client.0.smithi115.stdout:op 5180 completed, throughput=5MB/sec 2022-04-22T18:04:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:06 smithi121 bash[36902]: cluster 2022-04-22T18:04:04.809995+0000 mgr.y (mgr.34107) 1347 : cluster [DBG] pgmap v1921: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:06 smithi121 bash[36902]: audit 2022-04-22T18:04:05.405022+0000 mon.a (mon.0) 2888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:06 smithi121 bash[36902]: audit 2022-04-22T18:04:05.801848+0000 mon.a (mon.0) 2889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[45495]: cluster 2022-04-22T18:04:04.809995+0000 mgr.y (mgr.34107) 1347 : cluster [DBG] pgmap v1921: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[45495]: audit 2022-04-22T18:04:05.405022+0000 mon.a (mon.0) 2888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[45495]: audit 2022-04-22T18:04:05.801848+0000 mon.a (mon.0) 2889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[46934]: cluster 2022-04-22T18:04:04.809995+0000 mgr.y (mgr.34107) 1347 : cluster [DBG] pgmap v1921: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[46934]: audit 2022-04-22T18:04:05.405022+0000 mon.a (mon.0) 2888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:06 smithi115 bash[46934]: audit 2022-04-22T18:04:05.801848+0000 mon.a (mon.0) 2889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:07.071 INFO:tasks.workunit.client.0.smithi115.stdout: 1018: throughput=5MB/sec pending data=0 2022-04-22T18:04:07.071 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-H8iTvJOFUmHwdLD off=434480 len=1393431 2022-04-22T18:04:07.074 INFO:tasks.workunit.client.0.smithi115.stdout:op 5181 completed, throughput=5MB/sec 2022-04-22T18:04:07.074 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oJwN_VwrqGDhktl off=13038690 len=702617 2022-04-22T18:04:07.076 INFO:tasks.workunit.client.0.smithi115.stdout:op 5182 completed, throughput=5MB/sec 2022-04-22T18:04:07.076 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZE_3W1ULQCcZZPO off=2102652 len=720901 2022-04-22T18:04:07.078 INFO:tasks.workunit.client.0.smithi115.stdout:op 5183 completed, throughput=5MB/sec 2022-04-22T18:04:07.079 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vtswGt91b5I4ZUQ off=2787369 len=1619617 2022-04-22T18:04:07.086 INFO:tasks.workunit.client.0.smithi115.stdout:op 5184 completed, throughput=5MB/sec 2022-04-22T18:04:07.086 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cI_0aSGHirFeJZu off=2386779 len=1545574 2022-04-22T18:04:07.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 5185 completed, throughput=5MB/sec 2022-04-22T18:04:08.093 INFO:tasks.workunit.client.0.smithi115.stdout: 1019: throughput=5MB/sec pending data=0 2022-04-22T18:04:08.094 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D2t9_JjFY3u_Y6M off=1085827 len=213832 2022-04-22T18:04:08.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 5186 completed, throughput=5MB/sec 2022-04-22T18:04:08.097 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5C5Fs__U5Was-rr off=665341 len=911912 2022-04-22T18:04:08.099 INFO:tasks.workunit.client.0.smithi115.stdout:op 5187 completed, throughput=5MB/sec 2022-04-22T18:04:08.099 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WNK9v4UtUsenaRy off=328692 len=67407 2022-04-22T18:04:08.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 5188 completed, throughput=5MB/sec 2022-04-22T18:04:08.102 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m2iw3CHkvn3CdnD off=2112936 len=732986 2022-04-22T18:04:08.108 INFO:tasks.workunit.client.0.smithi115.stdout:op 5189 completed, throughput=5MB/sec 2022-04-22T18:04:08.108 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fzk6B1eWZ0ZHsgQ off=10035221 len=1503942 2022-04-22T18:04:08.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 5190 completed, throughput=5MB/sec 2022-04-22T18:04:08.116 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jVBMx9iBSKKTxNr off=1834822 len=1366500 2022-04-22T18:04:08.123 INFO:tasks.workunit.client.0.smithi115.stdout:op 5191 completed, throughput=5MB/sec 2022-04-22T18:04:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:08 smithi121 bash[36902]: cluster 2022-04-22T18:04:06.811083+0000 mgr.y (mgr.34107) 1348 : cluster [DBG] pgmap v1922: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:08 smithi121 bash[36902]: audit 2022-04-22T18:04:07.794180+0000 mon.a (mon.0) 2890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:08 smithi115 bash[45495]: cluster 2022-04-22T18:04:06.811083+0000 mgr.y (mgr.34107) 1348 : cluster [DBG] pgmap v1922: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:08 smithi115 bash[45495]: audit 2022-04-22T18:04:07.794180+0000 mon.a (mon.0) 2890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:08 smithi115 bash[46934]: cluster 2022-04-22T18:04:06.811083+0000 mgr.y (mgr.34107) 1348 : cluster [DBG] pgmap v1922: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:08 smithi115 bash[46934]: audit 2022-04-22T18:04:07.794180+0000 mon.a (mon.0) 2890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:09.123 INFO:tasks.workunit.client.0.smithi115.stdout: 1020: throughput=5MB/sec pending data=0 2022-04-22T18:04:09.124 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3dw18h8XP-2mWPG off=9648437 len=535522 2022-04-22T18:04:09.130 INFO:tasks.workunit.client.0.smithi115.stdout:op 5192 completed, throughput=5MB/sec 2022-04-22T18:04:09.130 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_4HYw8QtOjwO5pc off=2286144 len=284305 2022-04-22T18:04:09.133 INFO:tasks.workunit.client.0.smithi115.stdout:op 5193 completed, throughput=5MB/sec 2022-04-22T18:04:09.134 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aoXA3eDWPCuWl-E off=4775087 len=1902459 2022-04-22T18:04:09.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 5194 completed, throughput=5MB/sec 2022-04-22T18:04:09.137 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LIIdmCN4p63jiV6 off=4043881 len=85034 2022-04-22T18:04:09.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 5195 completed, throughput=5MB/sec 2022-04-22T18:04:09.141 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9iQxGN49RYBe12U off=4555354 len=878585 2022-04-22T18:04:09.143 INFO:tasks.workunit.client.0.smithi115.stdout:op 5196 completed, throughput=5MB/sec 2022-04-22T18:04:09.143 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Mgc24l4mkukDQGi off=40854 len=40704 2022-04-22T18:04:09.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 5197 completed, throughput=5MB/sec 2022-04-22T18:04:09.144 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wpRU6R7HHZ_tC73 off=3819563 len=1022860 2022-04-22T18:04:09.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 5198 completed, throughput=5MB/sec 2022-04-22T18:04:09.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xO5k2M_7hMzyW2I off=9074931 len=1603628 2022-04-22T18:04:09.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 5199 completed, throughput=5MB/sec 2022-04-22T18:04:10.155 INFO:tasks.workunit.client.0.smithi115.stdout: 1021: throughput=5MB/sec pending data=0 2022-04-22T18:04:10.156 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Quh4Y1dw-RhBECo off=6621074 len=142709 2022-04-22T18:04:10.159 INFO:tasks.workunit.client.0.smithi115.stdout:op 5200 completed, throughput=5MB/sec 2022-04-22T18:04:10.159 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fVbvfESpOIW_0cH off=5726492 len=1901450 2022-04-22T18:04:10.168 INFO:tasks.workunit.client.0.smithi115.stdout:op 5201 completed, throughput=5MB/sec 2022-04-22T18:04:10.169 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CRa8NKmUKthTFhg off=5813347 len=1628504 2022-04-22T18:04:10.176 INFO:tasks.workunit.client.0.smithi115.stdout:op 5202 completed, throughput=5MB/sec 2022-04-22T18:04:10.177 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2oheWKyjBIT1QFX off=4164588 len=1813473 2022-04-22T18:04:10.186 INFO:tasks.workunit.client.0.smithi115.stdout:op 5203 completed, throughput=5MB/sec 2022-04-22T18:04:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:10 smithi121 bash[36902]: cluster 2022-04-22T18:04:08.812302+0000 mgr.y (mgr.34107) 1349 : cluster [DBG] pgmap v1923: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:10 smithi115 bash[45495]: cluster 2022-04-22T18:04:08.812302+0000 mgr.y (mgr.34107) 1349 : cluster [DBG] pgmap v1923: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:10 smithi115 bash[46934]: cluster 2022-04-22T18:04:08.812302+0000 mgr.y (mgr.34107) 1349 : cluster [DBG] pgmap v1923: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:11.187 INFO:tasks.workunit.client.0.smithi115.stdout: 1022: throughput=5MB/sec pending data=0 2022-04-22T18:04:11.187 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E8OazZK1u9-FRaS off=2146348 len=253722 2022-04-22T18:04:11.191 INFO:tasks.workunit.client.0.smithi115.stdout:op 5204 completed, throughput=5MB/sec 2022-04-22T18:04:11.192 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DVz3BZv05Yu8UHe off=3612342 len=707506 2022-04-22T18:04:11.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 5205 completed, throughput=5MB/sec 2022-04-22T18:04:11.195 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9Q4LUBdxOuTxPiM off=998371 len=1707765 2022-04-22T18:04:11.200 INFO:tasks.workunit.client.0.smithi115.stdout:op 5206 completed, throughput=5MB/sec 2022-04-22T18:04:11.200 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GoMWWgYrBCrflaP off=10360546 len=1885172 2022-04-22T18:04:11.204 INFO:tasks.workunit.client.0.smithi115.stdout:op 5207 completed, throughput=5MB/sec 2022-04-22T18:04:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:11 smithi121 bash[36902]: audit 2022-04-22T18:04:10.408418+0000 mon.a (mon.0) 2891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:11.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:11 smithi121 bash[36902]: audit 2022-04-22T18:04:10.805606+0000 mon.a (mon.0) 2892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:11 smithi115 bash[45495]: audit 2022-04-22T18:04:10.408418+0000 mon.a (mon.0) 2891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:11 smithi115 bash[45495]: audit 2022-04-22T18:04:10.805606+0000 mon.a (mon.0) 2892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:11 smithi115 bash[46934]: audit 2022-04-22T18:04:10.408418+0000 mon.a (mon.0) 2891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:11 smithi115 bash[46934]: audit 2022-04-22T18:04:10.805606+0000 mon.a (mon.0) 2892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:12.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:11.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:12.205 INFO:tasks.workunit.client.0.smithi115.stdout: 1023: throughput=5MB/sec pending data=0 2022-04-22T18:04:12.205 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-465Pf0gtmn-eQVW off=383983 len=1574492 2022-04-22T18:04:12.214 INFO:tasks.workunit.client.0.smithi115.stdout:op 5208 completed, throughput=5MB/sec 2022-04-22T18:04:12.214 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jLc4Q0o5i0NZVWb off=5991019 len=139503 2022-04-22T18:04:12.217 INFO:tasks.workunit.client.0.smithi115.stdout:op 5209 completed, throughput=5MB/sec 2022-04-22T18:04:12.217 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2evO0nXHdwYjirX off=2916084 len=199942 2022-04-22T18:04:12.218 INFO:tasks.workunit.client.0.smithi115.stdout:op 5210 completed, throughput=5MB/sec 2022-04-22T18:04:12.219 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QvxI3N3yi-6RtVq off=4929067 len=845573 2022-04-22T18:04:12.224 INFO:tasks.workunit.client.0.smithi115.stdout:op 5211 completed, throughput=5MB/sec 2022-04-22T18:04:12.224 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ADd-TjcCwyV_idM off=786299 len=544814 2022-04-22T18:04:12.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 5212 completed, throughput=5MB/sec 2022-04-22T18:04:12.228 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TB4mZa6xEs2N1hp off=0 len=1060613 2022-04-22T18:04:12.230 INFO:tasks.workunit.client.0.smithi115.stdout:op 5213 completed, throughput=5MB/sec 2022-04-22T18:04:12.273 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lk-HsjgJpT8NGP_ off=1336321 len=1303598 2022-04-22T18:04:12.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 5214 completed, throughput=5MB/sec 2022-04-22T18:04:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:12 smithi121 bash[36902]: cluster 2022-04-22T18:04:10.813437+0000 mgr.y (mgr.34107) 1350 : cluster [DBG] pgmap v1924: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:04:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:12 smithi115 bash[46934]: cluster 2022-04-22T18:04:10.813437+0000 mgr.y (mgr.34107) 1350 : cluster [DBG] pgmap v1924: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:04:12.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:12 smithi115 bash[45495]: cluster 2022-04-22T18:04:10.813437+0000 mgr.y (mgr.34107) 1350 : cluster [DBG] pgmap v1924: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:04:13.236 INFO:tasks.workunit.client.0.smithi115.stdout: 1024: throughput=5MB/sec pending data=0 2022-04-22T18:04:13.236 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bFrOEcnTJCqw5cg off=13581258 len=406012 2022-04-22T18:04:13.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 5215 completed, throughput=5MB/sec 2022-04-22T18:04:13.238 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AIu2jYJkrOvtLnN off=990637 len=142594 2022-04-22T18:04:13.239 INFO:tasks.workunit.client.0.smithi115.stdout:op 5216 completed, throughput=5MB/sec 2022-04-22T18:04:13.239 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e8kUSD_CzmExg1Q off=9363275 len=2087301 2022-04-22T18:04:13.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 5217 completed, throughput=5MB/sec 2022-04-22T18:04:13.246 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xs4eJAfy9m25Lzp off=5352681 len=596403 2022-04-22T18:04:13.251 INFO:tasks.workunit.client.0.smithi115.stdout:op 5218 completed, throughput=5MB/sec 2022-04-22T18:04:13.302 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1MCWqMJJWKlZ4Y1 off=0 len=1350568 2022-04-22T18:04:13.303 INFO:tasks.workunit.client.0.smithi115.stdout:op 5219 completed, throughput=5MB/sec 2022-04-22T18:04:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:13 smithi121 bash[36902]: audit 2022-04-22T18:04:12.797270+0000 mon.a (mon.0) 2893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:13 smithi115 bash[45495]: audit 2022-04-22T18:04:12.797270+0000 mon.a (mon.0) 2893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:13 smithi115 bash[46934]: audit 2022-04-22T18:04:12.797270+0000 mon.a (mon.0) 2893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:13.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:14.254 INFO:tasks.workunit.client.0.smithi115.stdout: 1025: throughput=5MB/sec pending data=0 2022-04-22T18:04:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SIMrxHKkzECbEFf off=15732092 len=1605900 2022-04-22T18:04:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:op 5220 completed, throughput=5MB/sec 2022-04-22T18:04:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2kZt0nSGPMpqouB off=494633 len=1685800 2022-04-22T18:04:14.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 5221 completed, throughput=5MB/sec 2022-04-22T18:04:14.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QCro2sz987R9vvb off=12357300 len=1484344 2022-04-22T18:04:14.277 INFO:tasks.workunit.client.0.smithi115.stdout:op 5222 completed, throughput=5MB/sec 2022-04-22T18:04:14.277 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v-UgjHooqun5WrM off=16555479 len=630874 2022-04-22T18:04:14.278 INFO:tasks.workunit.client.0.smithi115.stdout:op 5223 completed, throughput=5MB/sec 2022-04-22T18:04:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:14 smithi121 bash[36902]: cluster 2022-04-22T18:04:12.814555+0000 mgr.y (mgr.34107) 1351 : cluster [DBG] pgmap v1925: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:14 smithi121 bash[36902]: audit 2022-04-22T18:04:13.638269+0000 mon.a (mon.0) 2894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:04:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:14 smithi121 bash[36902]: audit 2022-04-22T18:04:13.638738+0000 mon.a (mon.0) 2895 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[45495]: cluster 2022-04-22T18:04:12.814555+0000 mgr.y (mgr.34107) 1351 : cluster [DBG] pgmap v1925: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[45495]: audit 2022-04-22T18:04:13.638269+0000 mon.a (mon.0) 2894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[45495]: audit 2022-04-22T18:04:13.638738+0000 mon.a (mon.0) 2895 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[46934]: cluster 2022-04-22T18:04:12.814555+0000 mgr.y (mgr.34107) 1351 : cluster [DBG] pgmap v1925: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[46934]: audit 2022-04-22T18:04:13.638269+0000 mon.a (mon.0) 2894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:04:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:14 smithi115 bash[46934]: audit 2022-04-22T18:04:13.638738+0000 mon.a (mon.0) 2895 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:04:15.279 INFO:tasks.workunit.client.0.smithi115.stdout: 1026: throughput=5MB/sec pending data=0 2022-04-22T18:04:15.279 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NNOqUwp6aMCqjip off=4490114 len=544995 2022-04-22T18:04:15.284 INFO:tasks.workunit.client.0.smithi115.stdout:op 5224 completed, throughput=5MB/sec 2022-04-22T18:04:15.284 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Yp1wco0IDGrVHrh off=710637 len=482950 2022-04-22T18:04:15.290 INFO:tasks.workunit.client.0.smithi115.stdout:op 5225 completed, throughput=5MB/sec 2022-04-22T18:04:15.290 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b5M5tH6CzRIgDm0 off=2977518 len=1092693 2022-04-22T18:04:15.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 5226 completed, throughput=5MB/sec 2022-04-22T18:04:15.291 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jIei-J-5Dk-r_E7 off=879768 len=222164 2022-04-22T18:04:15.295 INFO:tasks.workunit.client.0.smithi115.stdout:op 5227 completed, throughput=5MB/sec 2022-04-22T18:04:15.296 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kv5lBe4Sb9LkD7L off=2262849 len=892464 2022-04-22T18:04:15.297 INFO:tasks.workunit.client.0.smithi115.stdout:op 5228 completed, throughput=5MB/sec 2022-04-22T18:04:15.297 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SrDFrrh-Zl-IOKd off=3754356 len=856037 2022-04-22T18:04:15.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 5229 completed, throughput=5MB/sec 2022-04-22T18:04:15.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OXLfuP624al8xeV off=6337953 len=950623 2022-04-22T18:04:15.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 5230 completed, throughput=5MB/sec 2022-04-22T18:04:15.307 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YgNHBT3vUefxa1I off=4957473 len=65080 2022-04-22T18:04:15.310 INFO:tasks.workunit.client.0.smithi115.stdout:op 5231 completed, throughput=5MB/sec 2022-04-22T18:04:15.311 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oUHJikVu27ygRJy off=572948 len=1740440 2022-04-22T18:04:15.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 5232 completed, throughput=5MB/sec 2022-04-22T18:04:16.312 INFO:tasks.workunit.client.0.smithi115.stdout: 1027: throughput=5MB/sec pending data=0 2022-04-22T18:04:16.312 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KrgtAKa_ic0FmoL off=11875687 len=348716 2022-04-22T18:04:16.313 INFO:tasks.workunit.client.0.smithi115.stdout:op 5233 completed, throughput=5MB/sec 2022-04-22T18:04:16.313 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3l3bfo5yEuhCyen off=8221941 len=356186 2022-04-22T18:04:16.315 INFO:tasks.workunit.client.0.smithi115.stdout:op 5234 completed, throughput=5MB/sec 2022-04-22T18:04:16.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xfOqplxVGsdlPMS off=20052188 len=1279601 2022-04-22T18:04:16.324 INFO:tasks.workunit.client.0.smithi115.stdout:op 5235 completed, throughput=5MB/sec 2022-04-22T18:04:16.324 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jizMQfanm7ddtWy off=4080601 len=1615683 2022-04-22T18:04:16.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 5236 completed, throughput=5MB/sec 2022-04-22T18:04:16.330 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hUHbSfDx0E6tq7w off=2105258 len=1552781 2022-04-22T18:04:16.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 5237 completed, throughput=5MB/sec 2022-04-22T18:04:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:16 smithi121 bash[36902]: cluster 2022-04-22T18:04:14.815716+0000 mgr.y (mgr.34107) 1352 : cluster [DBG] pgmap v1926: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:16 smithi121 bash[36902]: audit 2022-04-22T18:04:15.411694+0000 mon.a (mon.0) 2896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:16 smithi121 bash[36902]: audit 2022-04-22T18:04:15.809188+0000 mon.a (mon.0) 2897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[45495]: cluster 2022-04-22T18:04:14.815716+0000 mgr.y (mgr.34107) 1352 : cluster [DBG] pgmap v1926: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[45495]: audit 2022-04-22T18:04:15.411694+0000 mon.a (mon.0) 2896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[45495]: audit 2022-04-22T18:04:15.809188+0000 mon.a (mon.0) 2897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[46934]: cluster 2022-04-22T18:04:14.815716+0000 mgr.y (mgr.34107) 1352 : cluster [DBG] pgmap v1926: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[46934]: audit 2022-04-22T18:04:15.411694+0000 mon.a (mon.0) 2896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:16 smithi115 bash[46934]: audit 2022-04-22T18:04:15.809188+0000 mon.a (mon.0) 2897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:17.338 INFO:tasks.workunit.client.0.smithi115.stdout: 1028: throughput=5MB/sec pending data=0 2022-04-22T18:04:17.338 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-emxZySouI6-4jvd off=8250492 len=742539 2022-04-22T18:04:17.340 INFO:tasks.workunit.client.0.smithi115.stdout:op 5238 completed, throughput=5MB/sec 2022-04-22T18:04:17.340 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xHjHAZz2FWmNuxA off=10822096 len=300199 2022-04-22T18:04:17.345 INFO:tasks.workunit.client.0.smithi115.stdout:op 5239 completed, throughput=5MB/sec 2022-04-22T18:04:17.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-StPQ5Kkhjcsubpt off=624712 len=1269778 2022-04-22T18:04:17.353 INFO:tasks.workunit.client.0.smithi115.stdout:op 5240 completed, throughput=5MB/sec 2022-04-22T18:04:17.353 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7iO47MwclYl0Y22 off=1182126 len=942346 2022-04-22T18:04:17.356 INFO:tasks.workunit.client.0.smithi115.stdout:op 5241 completed, throughput=5MB/sec 2022-04-22T18:04:17.356 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IZKeKRrS3g6WGW8 off=212485 len=934960 2022-04-22T18:04:17.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 5242 completed, throughput=5MB/sec 2022-04-22T18:04:18.359 INFO:tasks.workunit.client.0.smithi115.stdout: 1029: throughput=5MB/sec pending data=0 2022-04-22T18:04:18.359 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jakYJzCQ368LWmd off=246291 len=1514089 2022-04-22T18:04:18.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 5243 completed, throughput=5MB/sec 2022-04-22T18:04:18.362 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2htBPKfj71Aq7L6 off=563833 len=104498 2022-04-22T18:04:18.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 5244 completed, throughput=5MB/sec 2022-04-22T18:04:18.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IkjjkuLcPTPRF6T off=13830515 len=558407 2022-04-22T18:04:18.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 5245 completed, throughput=5MB/sec 2022-04-22T18:04:18.370 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zt-jTDEshMttd_L off=71381 len=77769 2022-04-22T18:04:18.374 INFO:tasks.workunit.client.0.smithi115.stdout:op 5246 completed, throughput=5MB/sec 2022-04-22T18:04:18.374 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YiE_IuQ9cKqrX3T off=3743266 len=652179 2022-04-22T18:04:18.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 5247 completed, throughput=5MB/sec 2022-04-22T18:04:18.376 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XzMzEZIBIBfD_3S off=1351506 len=593806 2022-04-22T18:04:18.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 5248 completed, throughput=5MB/sec 2022-04-22T18:04:18.381 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0cIsB4q-3wniI6b off=543067 len=528351 2022-04-22T18:04:18.385 INFO:tasks.workunit.client.0.smithi115.stdout:op 5249 completed, throughput=5MB/sec 2022-04-22T18:04:18.385 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OS0KsGlTtFPX2Iz off=378707 len=1907006 2022-04-22T18:04:18.392 INFO:tasks.workunit.client.0.smithi115.stdout:op 5250 completed, throughput=5MB/sec 2022-04-22T18:04:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:18 smithi121 bash[36902]: cluster 2022-04-22T18:04:16.816683+0000 mgr.y (mgr.34107) 1353 : cluster [DBG] pgmap v1927: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:18 smithi121 bash[36902]: audit 2022-04-22T18:04:17.800447+0000 mon.a (mon.0) 2898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:18 smithi115 bash[45495]: cluster 2022-04-22T18:04:16.816683+0000 mgr.y (mgr.34107) 1353 : cluster [DBG] pgmap v1927: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:18 smithi115 bash[45495]: audit 2022-04-22T18:04:17.800447+0000 mon.a (mon.0) 2898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:18 smithi115 bash[46934]: cluster 2022-04-22T18:04:16.816683+0000 mgr.y (mgr.34107) 1353 : cluster [DBG] pgmap v1927: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:18 smithi115 bash[46934]: audit 2022-04-22T18:04:17.800447+0000 mon.a (mon.0) 2898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:19.392 INFO:tasks.workunit.client.0.smithi115.stdout: 1030: throughput=5MB/sec pending data=0 2022-04-22T18:04:19.393 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_zrj55HR67OF2sb off=5131249 len=1865173 2022-04-22T18:04:19.404 INFO:tasks.workunit.client.0.smithi115.stdout:op 5251 completed, throughput=5MB/sec 2022-04-22T18:04:19.405 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Od9x9Ez-0qsFgJR off=8404664 len=537164 2022-04-22T18:04:19.406 INFO:tasks.workunit.client.0.smithi115.stdout:op 5252 completed, throughput=5MB/sec 2022-04-22T18:04:19.407 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TFDibMvko5l8w4t off=2106385 len=1521323 2022-04-22T18:04:19.409 INFO:tasks.workunit.client.0.smithi115.stdout:op 5253 completed, throughput=5MB/sec 2022-04-22T18:04:19.409 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ve3hL3SfYUzzlKJ off=8042258 len=1482072 2022-04-22T18:04:19.411 INFO:tasks.workunit.client.0.smithi115.stdout:op 5254 completed, throughput=5MB/sec 2022-04-22T18:04:20.412 INFO:tasks.workunit.client.0.smithi115.stdout: 1031: throughput=5MB/sec pending data=0 2022-04-22T18:04:20.412 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rVb4viTvlMCPV2T off=823272 len=1027447 2022-04-22T18:04:20.420 INFO:tasks.workunit.client.0.smithi115.stdout:op 5255 completed, throughput=5MB/sec 2022-04-22T18:04:20.420 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2Odb2X5i8iwDIJc off=12056497 len=229109 2022-04-22T18:04:20.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 5256 completed, throughput=5MB/sec 2022-04-22T18:04:20.422 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ReQWfAS3zHa8Cnh off=609065 len=442124 2022-04-22T18:04:20.427 INFO:tasks.workunit.client.0.smithi115.stdout:op 5257 completed, throughput=5MB/sec 2022-04-22T18:04:20.439 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--Pw5KIDZkmA7Dtd off=3243305 len=1883450 2022-04-22T18:04:20.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 5258 completed, throughput=5MB/sec 2022-04-22T18:04:20.440 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tSmCWmhIW834quo off=467484 len=1540421 2022-04-22T18:04:20.440 INFO:tasks.workunit.client.0.smithi115.stdout:op 5259 completed, throughput=5MB/sec 2022-04-22T18:04:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:20 smithi121 bash[36902]: cluster 2022-04-22T18:04:18.817890+0000 mgr.y (mgr.34107) 1354 : cluster [DBG] pgmap v1928: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:20 smithi115 bash[45495]: cluster 2022-04-22T18:04:18.817890+0000 mgr.y (mgr.34107) 1354 : cluster [DBG] pgmap v1928: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:20.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:20 smithi115 bash[46934]: cluster 2022-04-22T18:04:18.817890+0000 mgr.y (mgr.34107) 1354 : cluster [DBG] pgmap v1928: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:21.437 INFO:tasks.workunit.client.0.smithi115.stdout: 1032: throughput=5MB/sec pending data=0 2022-04-22T18:04:21.437 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ff0a18cwfDXcVI7 off=4786180 len=908565 2022-04-22T18:04:21.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 5260 completed, throughput=5MB/sec 2022-04-22T18:04:21.445 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-o-00aJX5icev7__ off=11405305 len=248457 2022-04-22T18:04:21.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 5261 completed, throughput=5MB/sec 2022-04-22T18:04:21.450 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-l2R-4PN1qi6hoZh off=21218667 len=1559865 2022-04-22T18:04:21.458 INFO:tasks.workunit.client.0.smithi115.stdout:op 5262 completed, throughput=5MB/sec 2022-04-22T18:04:21.458 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tUZZfTvyflVWR5F off=15362736 len=1903603 2022-04-22T18:04:21.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 5263 completed, throughput=5MB/sec 2022-04-22T18:04:21.467 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wu9fHUE9cemi3m- off=15627639 len=1975391 2022-04-22T18:04:21.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 5264 completed, throughput=5MB/sec 2022-04-22T18:04:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:21 smithi121 bash[36902]: audit 2022-04-22T18:04:20.414960+0000 mon.a (mon.0) 2899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:21.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:21 smithi121 bash[36902]: audit 2022-04-22T18:04:20.812878+0000 mon.a (mon.0) 2900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:21.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:21 smithi115 bash[45495]: audit 2022-04-22T18:04:20.414960+0000 mon.a (mon.0) 2899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:21 smithi115 bash[45495]: audit 2022-04-22T18:04:20.812878+0000 mon.a (mon.0) 2900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:21.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:21 smithi115 bash[46934]: audit 2022-04-22T18:04:20.414960+0000 mon.a (mon.0) 2899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:21.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:21 smithi115 bash[46934]: audit 2022-04-22T18:04:20.812878+0000 mon.a (mon.0) 2900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:22.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:21.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:22.470 INFO:tasks.workunit.client.0.smithi115.stdout: 1033: throughput=5MB/sec pending data=0 2022-04-22T18:04:22.471 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YnPT_lUhA05Nun5 off=2917144 len=1715133 2022-04-22T18:04:22.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 5265 completed, throughput=5MB/sec 2022-04-22T18:04:22.475 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YXJsnGTkTciMR57 off=1084965 len=1563474 2022-04-22T18:04:22.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 5266 completed, throughput=5MB/sec 2022-04-22T18:04:22.486 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rA8aSsV9ZJBpLVN off=1442593 len=261503 2022-04-22T18:04:22.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 5267 completed, throughput=5MB/sec 2022-04-22T18:04:22.490 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M_Bd7stV9oSGK6y off=483539 len=1337029 2022-04-22T18:04:22.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 5268 completed, throughput=5MB/sec 2022-04-22T18:04:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:22 smithi121 bash[36902]: cluster 2022-04-22T18:04:20.818982+0000 mgr.y (mgr.34107) 1355 : cluster [DBG] pgmap v1929: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:04:22.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:22 smithi115 bash[46934]: cluster 2022-04-22T18:04:20.818982+0000 mgr.y (mgr.34107) 1355 : cluster [DBG] pgmap v1929: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:04:22.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:22 smithi115 bash[45495]: cluster 2022-04-22T18:04:20.818982+0000 mgr.y (mgr.34107) 1355 : cluster [DBG] pgmap v1929: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:04:23.496 INFO:tasks.workunit.client.0.smithi115.stdout: 1034: throughput=5MB/sec pending data=0 2022-04-22T18:04:23.496 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dXjHffVQylDTbom off=6587656 len=684802 2022-04-22T18:04:23.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 5269 completed, throughput=5MB/sec 2022-04-22T18:04:23.503 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2ExNYUTF3CqSpFW off=11388094 len=151227 2022-04-22T18:04:23.504 INFO:tasks.workunit.client.0.smithi115.stdout:op 5270 completed, throughput=5MB/sec 2022-04-22T18:04:23.504 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ipWvp9pkbxSrske off=0 len=1170019 2022-04-22T18:04:23.509 INFO:tasks.workunit.client.0.smithi115.stdout:op 5271 completed, throughput=5MB/sec 2022-04-22T18:04:23.509 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5oPR4i3nuMCH0JK off=8546240 len=491585 2022-04-22T18:04:23.514 INFO:tasks.workunit.client.0.smithi115.stdout:op 5272 completed, throughput=5MB/sec 2022-04-22T18:04:23.514 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OlFN3KO4ky437j5 off=4594077 len=1397076 2022-04-22T18:04:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:23 smithi121 bash[36902]: audit 2022-04-22T18:04:22.803660+0000 mon.a (mon.0) 2901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:23.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 5273 completed, throughput=5MB/sec 2022-04-22T18:04:23.520 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b9EVoTHT1z0IvJX off=4826944 len=1703277 2022-04-22T18:04:23.527 INFO:tasks.workunit.client.0.smithi115.stdout:op 5274 completed, throughput=5MB/sec 2022-04-22T18:04:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:23 smithi115 bash[45495]: audit 2022-04-22T18:04:22.803660+0000 mon.a (mon.0) 2901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:23 smithi115 bash[46934]: audit 2022-04-22T18:04:22.803660+0000 mon.a (mon.0) 2901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:23.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:24 smithi121 bash[36902]: cluster 2022-04-22T18:04:22.820035+0000 mgr.y (mgr.34107) 1356 : cluster [DBG] pgmap v1930: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:24.527 INFO:tasks.workunit.client.0.smithi115.stdout: 1035: throughput=5MB/sec pending data=0 2022-04-22T18:04:24.527 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9Y1tB-nCdnnG0cS off=3913427 len=1967587 2022-04-22T18:04:24.531 INFO:tasks.workunit.client.0.smithi115.stdout:op 5275 completed, throughput=5MB/sec 2022-04-22T18:04:24.532 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GYTRtSbyw5_jY7q off=5580050 len=563230 2022-04-22T18:04:24.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 5276 completed, throughput=5MB/sec 2022-04-22T18:04:24.537 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tQ_9eIrbHZ0R8cV off=7440478 len=453936 2022-04-22T18:04:24.538 INFO:tasks.workunit.client.0.smithi115.stdout:op 5277 completed, throughput=5MB/sec 2022-04-22T18:04:24.538 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oDwW4iKJiWK9ssv off=1251005 len=594751 2022-04-22T18:04:24.543 INFO:tasks.workunit.client.0.smithi115.stdout:op 5278 completed, throughput=5MB/sec 2022-04-22T18:04:24.543 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v2QQYo7Eol-2yOC off=17392191 len=219881 2022-04-22T18:04:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:24 smithi115 bash[45495]: cluster 2022-04-22T18:04:22.820035+0000 mgr.y (mgr.34107) 1356 : cluster [DBG] pgmap v1930: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:24 smithi115 bash[46934]: cluster 2022-04-22T18:04:22.820035+0000 mgr.y (mgr.34107) 1356 : cluster [DBG] pgmap v1930: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:24.544 INFO:tasks.workunit.client.0.smithi115.stdout:op 5279 completed, throughput=5MB/sec 2022-04-22T18:04:24.545 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qjSYGwotyelGDVQ off=14811691 len=236714 2022-04-22T18:04:24.546 INFO:tasks.workunit.client.0.smithi115.stdout:op 5280 completed, throughput=5MB/sec 2022-04-22T18:04:24.546 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zz-Cv-1txmqtX2o off=7945507 len=674000 2022-04-22T18:04:24.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 5281 completed, throughput=5MB/sec 2022-04-22T18:04:25.549 INFO:tasks.workunit.client.0.smithi115.stdout: 1036: throughput=5MB/sec pending data=0 2022-04-22T18:04:25.549 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u5mnYfZtKrcfPA8 off=15749766 len=393693 2022-04-22T18:04:25.551 INFO:tasks.workunit.client.0.smithi115.stdout:op 5282 completed, throughput=5MB/sec 2022-04-22T18:04:25.551 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sfiKJvYmhWHpJnr off=13788225 len=2004953 2022-04-22T18:04:25.562 INFO:tasks.workunit.client.0.smithi115.stdout:op 5283 completed, throughput=5MB/sec 2022-04-22T18:04:25.563 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--wRockGW3DwbUrP off=1295176 len=306410 2022-04-22T18:04:25.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 5284 completed, throughput=5MB/sec 2022-04-22T18:04:25.564 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kBcdf4xbSqevDp2 off=2491560 len=127836 2022-04-22T18:04:25.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 5285 completed, throughput=5MB/sec 2022-04-22T18:04:25.565 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Tc85rPSAF_up8CW off=11954710 len=1508441 2022-04-22T18:04:25.572 INFO:tasks.workunit.client.0.smithi115.stdout:op 5286 completed, throughput=5MB/sec 2022-04-22T18:04:25.572 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VjAb5C1XS4HDOc8 off=9283177 len=1149101 2022-04-22T18:04:25.578 INFO:tasks.workunit.client.0.smithi115.stdout:op 5287 completed, throughput=5MB/sec 2022-04-22T18:04:26.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:26 smithi121 bash[36902]: cluster 2022-04-22T18:04:24.821133+0000 mgr.y (mgr.34107) 1357 : cluster [DBG] pgmap v1931: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:26.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:26 smithi121 bash[36902]: audit 2022-04-22T18:04:25.417856+0000 mon.a (mon.0) 2902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:26.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:26 smithi121 bash[36902]: audit 2022-04-22T18:04:25.816285+0000 mon.a (mon.0) 2903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[45495]: cluster 2022-04-22T18:04:24.821133+0000 mgr.y (mgr.34107) 1357 : cluster [DBG] pgmap v1931: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[45495]: audit 2022-04-22T18:04:25.417856+0000 mon.a (mon.0) 2902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[45495]: audit 2022-04-22T18:04:25.816285+0000 mon.a (mon.0) 2903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[46934]: cluster 2022-04-22T18:04:24.821133+0000 mgr.y (mgr.34107) 1357 : cluster [DBG] pgmap v1931: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[46934]: audit 2022-04-22T18:04:25.417856+0000 mon.a (mon.0) 2902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:26 smithi115 bash[46934]: audit 2022-04-22T18:04:25.816285+0000 mon.a (mon.0) 2903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:26.579 INFO:tasks.workunit.client.0.smithi115.stdout: 1037: throughput=5MB/sec pending data=0 2022-04-22T18:04:26.579 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ABK74llr7JjtWWx off=21310709 len=1362478 2022-04-22T18:04:26.588 INFO:tasks.workunit.client.0.smithi115.stdout:op 5288 completed, throughput=5MB/sec 2022-04-22T18:04:26.588 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lc87Ks9Af1G4fX- off=15224593 len=1836489 2022-04-22T18:04:26.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 5289 completed, throughput=5MB/sec 2022-04-22T18:04:26.592 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DTdT8zRdGF4yNLg off=2258288 len=526286 2022-04-22T18:04:26.597 INFO:tasks.workunit.client.0.smithi115.stdout:op 5290 completed, throughput=5MB/sec 2022-04-22T18:04:26.597 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fHUnY-gE-blAxxO off=14310701 len=2058338 2022-04-22T18:04:26.600 INFO:tasks.workunit.client.0.smithi115.stdout:op 5291 completed, throughput=5MB/sec 2022-04-22T18:04:27.600 INFO:tasks.workunit.client.0.smithi115.stdout: 1038: throughput=5MB/sec pending data=0 2022-04-22T18:04:27.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NeF3EkmKfHFMZBo off=5288570 len=553723 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5292 completed, throughput=5MB/sec 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LEAAqVzuCEr_B29 off=12272659 len=207642 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5293 completed, throughput=5MB/sec 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-D9qN_AFcBP3OIyj off=7423148 len=699627 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5294 completed, throughput=5MB/sec 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kvd9BXNEk5kdHOs off=3463922 len=1424430 2022-04-22T18:04:27.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5295 completed, throughput=5MB/sec 2022-04-22T18:04:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Nklrs-nTDETLRat off=12968745 len=713096 2022-04-22T18:04:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 5296 completed, throughput=5MB/sec 2022-04-22T18:04:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0Nlth5hzGCMHANS off=1711690 len=802815 2022-04-22T18:04:27.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 5297 completed, throughput=5MB/sec 2022-04-22T18:04:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:28 smithi121 bash[36902]: cluster 2022-04-22T18:04:26.822260+0000 mgr.y (mgr.34107) 1358 : cluster [DBG] pgmap v1932: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:28.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:28 smithi121 bash[36902]: audit 2022-04-22T18:04:27.806934+0000 mon.a (mon.0) 2904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:28 smithi115 bash[45495]: cluster 2022-04-22T18:04:26.822260+0000 mgr.y (mgr.34107) 1358 : cluster [DBG] pgmap v1932: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:28 smithi115 bash[45495]: audit 2022-04-22T18:04:27.806934+0000 mon.a (mon.0) 2904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:28 smithi115 bash[46934]: cluster 2022-04-22T18:04:26.822260+0000 mgr.y (mgr.34107) 1358 : cluster [DBG] pgmap v1932: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:28 smithi115 bash[46934]: audit 2022-04-22T18:04:27.806934+0000 mon.a (mon.0) 2904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:28.623 INFO:tasks.workunit.client.0.smithi115.stdout: 1039: throughput=5MB/sec pending data=0 2022-04-22T18:04:28.623 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-su9Ecnq-hBeawCD off=8685745 len=51110 2022-04-22T18:04:28.627 INFO:tasks.workunit.client.0.smithi115.stdout:op 5298 completed, throughput=5MB/sec 2022-04-22T18:04:28.627 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bAyRl-rCfohdQMk off=16409891 len=408259 2022-04-22T18:04:28.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 5299 completed, throughput=5MB/sec 2022-04-22T18:04:28.629 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6g-_f-sEphPh68y off=3017318 len=800748 2022-04-22T18:04:28.631 INFO:tasks.workunit.client.0.smithi115.stdout:op 5300 completed, throughput=5MB/sec 2022-04-22T18:04:28.632 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BvrizzJ5C-QY2WS off=3323160 len=1826404 2022-04-22T18:04:28.635 INFO:tasks.workunit.client.0.smithi115.stdout:op 5301 completed, throughput=5MB/sec 2022-04-22T18:04:28.635 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zy97rkAhIcvuQp- off=8421775 len=1888278 2022-04-22T18:04:28.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 5302 completed, throughput=5MB/sec 2022-04-22T18:04:28.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-erieZ1K_vPwrZGh off=686899 len=1687685 2022-04-22T18:04:28.642 INFO:tasks.workunit.client.0.smithi115.stdout:op 5303 completed, throughput=5MB/sec 2022-04-22T18:04:29.642 INFO:tasks.workunit.client.0.smithi115.stdout: 1040: throughput=5MB/sec pending data=0 2022-04-22T18:04:29.642 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yXBJgvpSelj042Z off=8364331 len=1795927 2022-04-22T18:04:29.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 5304 completed, throughput=5MB/sec 2022-04-22T18:04:29.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9YZim91vnwd45l2 off=1614272 len=1175505 2022-04-22T18:04:29.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 5305 completed, throughput=5MB/sec 2022-04-22T18:04:29.654 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4XHLqsPzIpKjsym off=1516258 len=1766777 2022-04-22T18:04:29.657 INFO:tasks.workunit.client.0.smithi115.stdout:op 5306 completed, throughput=5MB/sec 2022-04-22T18:04:30.131 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:04:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:04:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (27m) 6m ago 41m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (41m) 7m ago 41m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (29m) 6m ago 42m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (29m) 7m ago 42m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (36m) 7m ago 46m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (37m) 6m ago 48m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (36m) 6m ago 48m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (35m) 7m ago 46m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (35m) 6m ago 47m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (28m) 6m ago 41m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:04:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (28m) 7m ago 41m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (34m) 6m ago 46m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (33m) 6m ago 45m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (33m) 6m ago 45m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (32m) 6m ago 44m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (32m) 7m ago 44m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (31m) 7m ago 43m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (30m) 7m ago 43m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (29m) 7m ago 42m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:04:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (28m) 7m ago 41m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:04:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:30 smithi121 bash[36902]: cluster 2022-04-22T18:04:28.823453+0000 mgr.y (mgr.34107) 1359 : cluster [DBG] pgmap v1933: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:30 smithi115 bash[45495]: cluster 2022-04-22T18:04:28.823453+0000 mgr.y (mgr.34107) 1359 : cluster [DBG] pgmap v1933: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:30.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:30 smithi115 bash[46934]: cluster 2022-04-22T18:04:28.823453+0000 mgr.y (mgr.34107) 1359 : cluster [DBG] pgmap v1933: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:30.659 INFO:tasks.workunit.client.0.smithi115.stdout: 1041: throughput=5MB/sec pending data=0 2022-04-22T18:04:30.659 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F2GIVE14KDxcNZ7 off=6299033 len=668759 2022-04-22T18:04:30.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 5307 completed, throughput=5MB/sec 2022-04-22T18:04:30.663 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-e1YS3e90qg4ltGm off=1384096 len=1653619 2022-04-22T18:04:30.673 INFO:tasks.workunit.client.0.smithi115.stdout:op 5308 completed, throughput=5MB/sec 2022-04-22T18:04:30.673 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vVCgJrPesMRqRm4 off=4885745 len=1009453 2022-04-22T18:04:30.677 INFO:tasks.workunit.client.0.smithi115.stdout:op 5309 completed, throughput=5MB/sec 2022-04-22T18:04:30.677 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uiqxhWsQ6tnHzTo off=1598960 len=419422 2022-04-22T18:04:30.678 INFO:tasks.workunit.client.0.smithi115.stdout:op 5310 completed, throughput=5MB/sec 2022-04-22T18:04:30.678 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eRBudEMfHuf_r5J off=5439973 len=1532848 2022-04-22T18:04:30.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 5311 completed, throughput=5MB/sec 2022-04-22T18:04:30.888 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:04:30.889 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:04:30.890 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:04:30.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:04:30.890 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:04:30.890 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:04:30.891 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:04:30.892 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:04:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:31 smithi121 bash[36902]: audit 2022-04-22T18:04:30.110130+0000 mgr.y (mgr.34107) 1360 : audit [DBG] from='client.45763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:31 smithi121 bash[36902]: audit 2022-04-22T18:04:30.420696+0000 mon.a (mon.0) 2905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:31 smithi121 bash[36902]: audit 2022-04-22T18:04:30.449840+0000 mgr.y (mgr.34107) 1361 : audit [DBG] from='client.35829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:31 smithi121 bash[36902]: audit 2022-04-22T18:04:30.819945+0000 mon.a (mon.0) 2906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:31 smithi121 bash[36902]: audit 2022-04-22T18:04:30.881258+0000 mon.c (mon.1) 81 : audit [DBG] from='client.? 172.21.15.115:0/642521678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[45495]: audit 2022-04-22T18:04:30.110130+0000 mgr.y (mgr.34107) 1360 : audit [DBG] from='client.45763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[45495]: audit 2022-04-22T18:04:30.420696+0000 mon.a (mon.0) 2905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[45495]: audit 2022-04-22T18:04:30.449840+0000 mgr.y (mgr.34107) 1361 : audit [DBG] from='client.35829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[45495]: audit 2022-04-22T18:04:30.819945+0000 mon.a (mon.0) 2906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[45495]: audit 2022-04-22T18:04:30.881258+0000 mon.c (mon.1) 81 : audit [DBG] from='client.? 172.21.15.115:0/642521678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[46934]: audit 2022-04-22T18:04:30.110130+0000 mgr.y (mgr.34107) 1360 : audit [DBG] from='client.45763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[46934]: audit 2022-04-22T18:04:30.420696+0000 mon.a (mon.0) 2905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[46934]: audit 2022-04-22T18:04:30.449840+0000 mgr.y (mgr.34107) 1361 : audit [DBG] from='client.35829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:04:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[46934]: audit 2022-04-22T18:04:30.819945+0000 mon.a (mon.0) 2906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[46934]: audit 2022-04-22T18:04:30.881258+0000 mon.c (mon.1) 81 : audit [DBG] from='client.? 172.21.15.115:0/642521678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:04:31.685 INFO:tasks.workunit.client.0.smithi115.stdout: 1042: throughput=5MB/sec pending data=0 2022-04-22T18:04:31.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2P-wmyks390bNf7 off=5049905 len=378707 2022-04-22T18:04:31.690 INFO:tasks.workunit.client.0.smithi115.stdout:op 5312 completed, throughput=5MB/sec 2022-04-22T18:04:31.690 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xFVj5A6VtPtai_8 off=2221034 len=1757549 2022-04-22T18:04:31.696 INFO:tasks.workunit.client.0.smithi115.stdout:op 5313 completed, throughput=5MB/sec 2022-04-22T18:04:31.696 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1ttlqNNaN8BPB92 off=266959 len=399569 2022-04-22T18:04:31.701 INFO:tasks.workunit.client.0.smithi115.stdout:op 5314 completed, throughput=5MB/sec 2022-04-22T18:04:31.702 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_j8bcm_3SEyo36Y off=488836 len=85529 2022-04-22T18:04:31.705 INFO:tasks.workunit.client.0.smithi115.stdout:op 5315 completed, throughput=5MB/sec 2022-04-22T18:04:31.705 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G549ZmTXQlZLL7I off=14295879 len=147997 2022-04-22T18:04:31.709 INFO:tasks.workunit.client.0.smithi115.stdout:op 5316 completed, throughput=5MB/sec 2022-04-22T18:04:31.709 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iVGBz9Tf21-5Q2q off=6141259 len=1827828 2022-04-22T18:04:31.712 INFO:tasks.workunit.client.0.smithi115.stdout:op 5317 completed, throughput=5MB/sec 2022-04-22T18:04:32.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:31.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:32 smithi121 bash[36902]: cluster 2022-04-22T18:04:30.824394+0000 mgr.y (mgr.34107) 1362 : cluster [DBG] pgmap v1934: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:04:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:32 smithi115 bash[45495]: cluster 2022-04-22T18:04:30.824394+0000 mgr.y (mgr.34107) 1362 : cluster [DBG] pgmap v1934: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:04:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:32 smithi115 bash[46934]: cluster 2022-04-22T18:04:30.824394+0000 mgr.y (mgr.34107) 1362 : cluster [DBG] pgmap v1934: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:04:32.712 INFO:tasks.workunit.client.0.smithi115.stdout: 1043: throughput=5MB/sec pending data=0 2022-04-22T18:04:32.713 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-y44pM1OKWXl-y5A off=8448755 len=716594 2022-04-22T18:04:32.718 INFO:tasks.workunit.client.0.smithi115.stdout:op 5318 completed, throughput=5MB/sec 2022-04-22T18:04:32.718 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ibNbHLW73exyiNt off=257540 len=368750 2022-04-22T18:04:32.720 INFO:tasks.workunit.client.0.smithi115.stdout:op 5319 completed, throughput=5MB/sec 2022-04-22T18:04:32.720 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ot6HKA8bg4uTYGl off=2908086 len=1831687 2022-04-22T18:04:32.724 INFO:tasks.workunit.client.0.smithi115.stdout:op 5320 completed, throughput=5MB/sec 2022-04-22T18:04:32.724 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BqvskMG8Z5yi5Kr off=11759841 len=1311793 2022-04-22T18:04:32.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 5321 completed, throughput=5MB/sec 2022-04-22T18:04:32.728 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xMLYJi_mTDmkACx off=5656653 len=759459 2022-04-22T18:04:32.734 INFO:tasks.workunit.client.0.smithi115.stdout:op 5322 completed, throughput=5MB/sec 2022-04-22T18:04:32.734 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OuP8RvJvwlckzlx off=1205840 len=268943 2022-04-22T18:04:32.739 INFO:tasks.workunit.client.0.smithi115.stdout:op 5323 completed, throughput=5MB/sec 2022-04-22T18:04:32.739 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aTnVSw_3tUgZc47 off=9552381 len=1416993 2022-04-22T18:04:32.747 INFO:tasks.workunit.client.0.smithi115.stdout:op 5324 completed, throughput=5MB/sec 2022-04-22T18:04:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:33 smithi121 bash[36902]: audit 2022-04-22T18:04:32.810324+0000 mon.a (mon.0) 2907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:33 smithi115 bash[45495]: audit 2022-04-22T18:04:32.810324+0000 mon.a (mon.0) 2907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:33 smithi115 bash[46934]: audit 2022-04-22T18:04:32.810324+0000 mon.a (mon.0) 2907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:33.749 INFO:tasks.workunit.client.0.smithi115.stdout: 1044: throughput=5MB/sec pending data=0 2022-04-22T18:04:33.749 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GDGZJ8z1BuObs9F off=5100576 len=1580751 2022-04-22T18:04:33.751 INFO:tasks.workunit.client.0.smithi115.stdout:op 5325 completed, throughput=5MB/sec 2022-04-22T18:04:33.751 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BZaHEbQU3V-NH2H off=5747514 len=526554 2022-04-22T18:04:33.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5326 completed, throughput=5MB/sec 2022-04-22T18:04:33.753 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-A1v7ub8glwr7d12 off=2163332 len=1920208 2022-04-22T18:04:33.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 5327 completed, throughput=5MB/sec 2022-04-22T18:04:34.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:33.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:34 smithi121 bash[36902]: cluster 2022-04-22T18:04:32.825486+0000 mgr.y (mgr.34107) 1363 : cluster [DBG] pgmap v1935: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:34 smithi115 bash[45495]: cluster 2022-04-22T18:04:32.825486+0000 mgr.y (mgr.34107) 1363 : cluster [DBG] pgmap v1935: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:34 smithi115 bash[46934]: cluster 2022-04-22T18:04:32.825486+0000 mgr.y (mgr.34107) 1363 : cluster [DBG] pgmap v1935: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:04:34.766 INFO:tasks.workunit.client.0.smithi115.stdout: 1045: throughput=5MB/sec pending data=0 2022-04-22T18:04:34.766 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vXpjnDCxPuXVwLI off=15702207 len=660964 2022-04-22T18:04:34.770 INFO:tasks.workunit.client.0.smithi115.stdout:op 5328 completed, throughput=5MB/sec 2022-04-22T18:04:34.771 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HQ8JyFAVW1uiG97 off=11327211 len=377050 2022-04-22T18:04:34.771 INFO:tasks.workunit.client.0.smithi115.stdout:op 5329 completed, throughput=5MB/sec 2022-04-22T18:04:34.772 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2aEpHMObZMV-qZ4 off=11514449 len=192386 2022-04-22T18:04:34.773 INFO:tasks.workunit.client.0.smithi115.stdout:op 5330 completed, throughput=5MB/sec 2022-04-22T18:04:34.773 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IlcUsPIwoUm9ilz off=2390676 len=773125 2022-04-22T18:04:34.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 5331 completed, throughput=5MB/sec 2022-04-22T18:04:34.776 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SYZ_dh_wUbPkbgK off=5707672 len=530819 2022-04-22T18:04:34.781 INFO:tasks.workunit.client.0.smithi115.stdout:op 5332 completed, throughput=5MB/sec 2022-04-22T18:04:34.781 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FIvilhXmlmXjFC4 off=5034024 len=498175 2022-04-22T18:04:34.786 INFO:tasks.workunit.client.0.smithi115.stdout:op 5333 completed, throughput=5MB/sec 2022-04-22T18:04:34.786 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_P5r5Z3vwnMK39s off=17506872 len=2074692 2022-04-22T18:04:34.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 5334 completed, throughput=5MB/sec 2022-04-22T18:04:34.791 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rhWiYQZ9V5XXms8 off=9909854 len=551380 2022-04-22T18:04:34.794 INFO:tasks.workunit.client.0.smithi115.stdout:op 5335 completed, throughput=5MB/sec 2022-04-22T18:04:35.796 INFO:tasks.workunit.client.0.smithi115.stdout: 1046: throughput=5MB/sec pending data=0 2022-04-22T18:04:35.796 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zjMZIcywmAIWJ_d off=8817122 len=1192124 2022-04-22T18:04:35.799 INFO:tasks.workunit.client.0.smithi115.stdout:op 5336 completed, throughput=5MB/sec 2022-04-22T18:04:35.799 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YEOXgUlrKShTct4 off=9263932 len=1194085 2022-04-22T18:04:35.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 5337 completed, throughput=5MB/sec 2022-04-22T18:04:35.808 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5UVEltpyRtCvW0w off=12340664 len=53998 2022-04-22T18:04:35.808 INFO:tasks.workunit.client.0.smithi115.stdout:op 5338 completed, throughput=5MB/sec 2022-04-22T18:04:35.809 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gUaM6qinyr2tsT3 off=12014209 len=1062106 2022-04-22T18:04:35.814 INFO:tasks.workunit.client.0.smithi115.stdout:op 5339 completed, throughput=5MB/sec 2022-04-22T18:04:35.814 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QMQ4OQk5G8Amh_X off=1878260 len=1994006 2022-04-22T18:04:35.817 INFO:tasks.workunit.client.0.smithi115.stdout:op 5340 completed, throughput=5MB/sec 2022-04-22T18:04:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:36 smithi121 bash[36902]: cluster 2022-04-22T18:04:34.826581+0000 mgr.y (mgr.34107) 1364 : cluster [DBG] pgmap v1936: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:04:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:36 smithi121 bash[36902]: audit 2022-04-22T18:04:35.423225+0000 mon.a (mon.0) 2908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:36 smithi121 bash[36902]: audit 2022-04-22T18:04:35.823469+0000 mon.a (mon.0) 2909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[45495]: cluster 2022-04-22T18:04:34.826581+0000 mgr.y (mgr.34107) 1364 : cluster [DBG] pgmap v1936: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:04:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[45495]: audit 2022-04-22T18:04:35.423225+0000 mon.a (mon.0) 2908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[45495]: audit 2022-04-22T18:04:35.823469+0000 mon.a (mon.0) 2909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[46934]: cluster 2022-04-22T18:04:34.826581+0000 mgr.y (mgr.34107) 1364 : cluster [DBG] pgmap v1936: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:04:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[46934]: audit 2022-04-22T18:04:35.423225+0000 mon.a (mon.0) 2908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:36 smithi115 bash[46934]: audit 2022-04-22T18:04:35.823469+0000 mon.a (mon.0) 2909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:36.818 INFO:tasks.workunit.client.0.smithi115.stdout: 1047: throughput=5MB/sec pending data=0 2022-04-22T18:04:36.818 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JAV8_LN8IwLQjmz off=11373332 len=683624 2022-04-22T18:04:36.820 INFO:tasks.workunit.client.0.smithi115.stdout:op 5341 completed, throughput=5MB/sec 2022-04-22T18:04:36.820 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QeqR4B5EgTfM_tQ off=3162607 len=324357 2022-04-22T18:04:36.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 5342 completed, throughput=5MB/sec 2022-04-22T18:04:36.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ugsn2cWEeaBkfrX off=1964734 len=1697635 2022-04-22T18:04:36.834 INFO:tasks.workunit.client.0.smithi115.stdout:op 5343 completed, throughput=5MB/sec 2022-04-22T18:04:36.834 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-A3o_beoYuSr-lae off=822065 len=942235 2022-04-22T18:04:36.837 INFO:tasks.workunit.client.0.smithi115.stdout:op 5344 completed, throughput=5MB/sec 2022-04-22T18:04:36.837 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n9JBFXQ-SREfNqo off=3164767 len=158439 2022-04-22T18:04:36.838 INFO:tasks.workunit.client.0.smithi115.stdout:op 5345 completed, throughput=5MB/sec 2022-04-22T18:04:36.838 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JVLYoWZTP7tD6C2 off=363063 len=448147 2022-04-22T18:04:36.842 INFO:tasks.workunit.client.0.smithi115.stdout:op 5346 completed, throughput=5MB/sec 2022-04-22T18:04:36.842 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-g7vRc2gO2uspWfL off=3512293 len=663320 2022-04-22T18:04:36.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 5347 completed, throughput=5MB/sec 2022-04-22T18:04:36.845 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cFrzC01fPWZWnbF off=1358140 len=754453 2022-04-22T18:04:36.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 5348 completed, throughput=5MB/sec 2022-04-22T18:04:37.846 INFO:tasks.workunit.client.0.smithi115.stdout: 1048: throughput=5MB/sec pending data=0 2022-04-22T18:04:37.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-REPjKZLcnXUBKuG off=1031504 len=1486594 2022-04-22T18:04:37.854 INFO:tasks.workunit.client.0.smithi115.stdout:op 5349 completed, throughput=5MB/sec 2022-04-22T18:04:37.854 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-11lNbuBAQ91974l off=8562083 len=1155071 2022-04-22T18:04:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 5350 completed, throughput=5MB/sec 2022-04-22T18:04:37.860 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Xau_aOzXwDs__aE off=5401118 len=1514410 2022-04-22T18:04:37.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 5351 completed, throughput=5MB/sec 2022-04-22T18:04:37.863 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VchNfCdsGulrcq2 off=7737452 len=688260 2022-04-22T18:04:37.866 INFO:tasks.workunit.client.0.smithi115.stdout:op 5352 completed, throughput=5MB/sec 2022-04-22T18:04:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:38 smithi115 bash[46934]: cluster 2022-04-22T18:04:36.827340+0000 mgr.y (mgr.34107) 1365 : cluster [DBG] pgmap v1937: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:38 smithi115 bash[46934]: audit 2022-04-22T18:04:37.813565+0000 mon.a (mon.0) 2910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:38 smithi115 bash[45495]: cluster 2022-04-22T18:04:36.827340+0000 mgr.y (mgr.34107) 1365 : cluster [DBG] pgmap v1937: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:38 smithi115 bash[45495]: audit 2022-04-22T18:04:37.813565+0000 mon.a (mon.0) 2910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:38 smithi121 bash[36902]: cluster 2022-04-22T18:04:36.827340+0000 mgr.y (mgr.34107) 1365 : cluster [DBG] pgmap v1937: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:38 smithi121 bash[36902]: audit 2022-04-22T18:04:37.813565+0000 mon.a (mon.0) 2910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:38.866 INFO:tasks.workunit.client.0.smithi115.stdout: 1049: throughput=5MB/sec pending data=0 2022-04-22T18:04:38.866 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m8viPClFoNA2mGj off=1338 len=2001707 2022-04-22T18:04:38.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 5353 completed, throughput=5MB/sec 2022-04-22T18:04:38.870 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lyUm8RqzaGiXgEc off=7109296 len=618383 2022-04-22T18:04:38.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 5354 completed, throughput=5MB/sec 2022-04-22T18:04:38.877 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-52k7X3UtGxwQoJe off=11861237 len=685502 2022-04-22T18:04:38.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 5355 completed, throughput=5MB/sec 2022-04-22T18:04:38.879 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iIeuf51ghA0JHNp off=3683142 len=1791147 2022-04-22T18:04:38.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 5356 completed, throughput=5MB/sec 2022-04-22T18:04:38.882 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DuZ2bF3BwJ6k-Gv off=896587 len=471124 2022-04-22T18:04:38.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 5357 completed, throughput=5MB/sec 2022-04-22T18:04:39.887 INFO:tasks.workunit.client.0.smithi115.stdout: 1050: throughput=5MB/sec pending data=0 2022-04-22T18:04:39.888 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rjroJNRygZ0t4w3 off=23879581 len=1714223 2022-04-22T18:04:39.891 INFO:tasks.workunit.client.0.smithi115.stdout:op 5358 completed, throughput=5MB/sec 2022-04-22T18:04:39.891 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r7cj-9-2aqiDdDw off=10310546 len=1553581 2022-04-22T18:04:39.900 INFO:tasks.workunit.client.0.smithi115.stdout:op 5359 completed, throughput=5MB/sec 2022-04-22T18:04:39.900 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-14zaqRWWL8mC0Tt off=3078300 len=302655 2022-04-22T18:04:39.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 5360 completed, throughput=5MB/sec 2022-04-22T18:04:39.901 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XR3v0aMa3FVWBsC off=197869 len=1640740 2022-04-22T18:04:39.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 5361 completed, throughput=5MB/sec 2022-04-22T18:04:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:40 smithi115 bash[45495]: cluster 2022-04-22T18:04:38.828486+0000 mgr.y (mgr.34107) 1366 : cluster [DBG] pgmap v1938: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:40 smithi115 bash[46934]: cluster 2022-04-22T18:04:38.828486+0000 mgr.y (mgr.34107) 1366 : cluster [DBG] pgmap v1938: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:40 smithi121 bash[36902]: cluster 2022-04-22T18:04:38.828486+0000 mgr.y (mgr.34107) 1366 : cluster [DBG] pgmap v1938: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:40.909 INFO:tasks.workunit.client.0.smithi115.stdout: 1051: throughput=5MB/sec pending data=0 2022-04-22T18:04:40.910 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XpjMAYzm7j39CXS off=9600709 len=1480595 2022-04-22T18:04:40.914 INFO:tasks.workunit.client.0.smithi115.stdout:op 5362 completed, throughput=5MB/sec 2022-04-22T18:04:40.914 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-uDr5F-uKUiHhG0B off=13148744 len=42914 2022-04-22T18:04:40.915 INFO:tasks.workunit.client.0.smithi115.stdout:op 5363 completed, throughput=5MB/sec 2022-04-22T18:04:40.915 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Jdc9C1ATUa0rqVi off=92560 len=783635 2022-04-22T18:04:40.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 5364 completed, throughput=5MB/sec 2022-04-22T18:04:40.922 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-O5d7MpGnoQLDYln off=11701695 len=24799 2022-04-22T18:04:40.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 5365 completed, throughput=5MB/sec 2022-04-22T18:04:40.923 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lJaORkRHEgQfARV off=5438516 len=759726 2022-04-22T18:04:40.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 5366 completed, throughput=5MB/sec 2022-04-22T18:04:40.928 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f-zefgNPjov_HuA off=6610408 len=1007063 2022-04-22T18:04:40.934 INFO:tasks.workunit.client.0.smithi115.stdout:op 5367 completed, throughput=5MB/sec 2022-04-22T18:04:40.934 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1DbcZB95w6Q6Pq9 off=1343752 len=435927 2022-04-22T18:04:40.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 5368 completed, throughput=5MB/sec 2022-04-22T18:04:40.940 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ibNbHLW73exyiNt off=7061362 len=1110919 2022-04-22T18:04:40.942 INFO:tasks.workunit.client.0.smithi115.stdout:op 5369 completed, throughput=5MB/sec 2022-04-22T18:04:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:41 smithi115 bash[45495]: audit 2022-04-22T18:04:40.426263+0000 mon.a (mon.0) 2911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:41 smithi115 bash[45495]: audit 2022-04-22T18:04:40.827024+0000 mon.a (mon.0) 2912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:41 smithi115 bash[46934]: audit 2022-04-22T18:04:40.426263+0000 mon.a (mon.0) 2911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:41 smithi115 bash[46934]: audit 2022-04-22T18:04:40.827024+0000 mon.a (mon.0) 2912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:41 smithi121 bash[36902]: audit 2022-04-22T18:04:40.426263+0000 mon.a (mon.0) 2911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:41 smithi121 bash[36902]: audit 2022-04-22T18:04:40.827024+0000 mon.a (mon.0) 2912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:41.943 INFO:tasks.workunit.client.0.smithi115.stdout: 1052: throughput=5MB/sec pending data=0 2022-04-22T18:04:41.943 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--w9pVw17LLRud9t off=517328 len=574896 2022-04-22T18:04:41.949 INFO:tasks.workunit.client.0.smithi115.stdout:op 5370 completed, throughput=5MB/sec 2022-04-22T18:04:41.949 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6T1LuLavixPbEFz off=2719776 len=651643 2022-04-22T18:04:41.952 INFO:tasks.workunit.client.0.smithi115.stdout:op 5371 completed, throughput=5MB/sec 2022-04-22T18:04:41.952 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rT5XmL0CxXt7f_i off=2451799 len=37728 2022-04-22T18:04:41.953 INFO:tasks.workunit.client.0.smithi115.stdout:op 5372 completed, throughput=5MB/sec 2022-04-22T18:04:41.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-80sHnzBm9VL4D6N off=15832012 len=819649 2022-04-22T18:04:41.956 INFO:tasks.workunit.client.0.smithi115.stdout:op 5373 completed, throughput=5MB/sec 2022-04-22T18:04:41.956 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-f742rbVJDYUdMkz off=8632299 len=253867 2022-04-22T18:04:41.960 INFO:tasks.workunit.client.0.smithi115.stdout:op 5374 completed, throughput=5MB/sec 2022-04-22T18:04:41.960 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V31bnixbE9cj35T off=7209363 len=935571 2022-04-22T18:04:41.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 5375 completed, throughput=5MB/sec 2022-04-22T18:04:41.962 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KaUT3YwWPPMcvEf off=16057595 len=889561 2022-04-22T18:04:41.968 INFO:tasks.workunit.client.0.smithi115.stdout:op 5376 completed, throughput=5MB/sec 2022-04-22T18:04:41.968 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c5OlAATxl6dcbZq off=892375 len=228033 2022-04-22T18:04:41.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5377 completed, throughput=5MB/sec 2022-04-22T18:04:41.972 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Emf07EPJlZjsNpy off=2186468 len=1757834 2022-04-22T18:04:41.980 INFO:tasks.workunit.client.0.smithi115.stdout:op 5378 completed, throughput=5MB/sec 2022-04-22T18:04:42.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:41.738Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:42 smithi115 bash[45495]: cluster 2022-04-22T18:04:40.829502+0000 mgr.y (mgr.34107) 1367 : cluster [DBG] pgmap v1939: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:04:42.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:42 smithi115 bash[46934]: cluster 2022-04-22T18:04:40.829502+0000 mgr.y (mgr.34107) 1367 : cluster [DBG] pgmap v1939: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:04:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:42 smithi121 bash[36902]: cluster 2022-04-22T18:04:40.829502+0000 mgr.y (mgr.34107) 1367 : cluster [DBG] pgmap v1939: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:04:42.980 INFO:tasks.workunit.client.0.smithi115.stdout: 1053: throughput=5MB/sec pending data=0 2022-04-22T18:04:42.980 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jiTo1-BBYISTJUN off=1776065 len=1855121 2022-04-22T18:04:42.985 INFO:tasks.workunit.client.0.smithi115.stdout:op 5379 completed, throughput=5MB/sec 2022-04-22T18:04:42.985 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vT6h3162r7rTfhu off=648966 len=2046397 2022-04-22T18:04:42.993 INFO:tasks.workunit.client.0.smithi115.stdout:op 5380 completed, throughput=5MB/sec 2022-04-22T18:04:42.993 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Lf9SJTqocEVGgVc off=17199256 len=1672161 2022-04-22T18:04:42.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 5381 completed, throughput=5MB/sec 2022-04-22T18:04:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:43 smithi115 bash[45495]: audit 2022-04-22T18:04:42.816647+0000 mon.a (mon.0) 2913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:43 smithi115 bash[46934]: audit 2022-04-22T18:04:42.816647+0000 mon.a (mon.0) 2913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:43 smithi121 bash[36902]: audit 2022-04-22T18:04:42.816647+0000 mon.a (mon.0) 2913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:43.999 INFO:tasks.workunit.client.0.smithi115.stdout: 1054: throughput=5MB/sec pending data=0 2022-04-22T18:04:43.999 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BWyKI627AUCgdVE off=5212012 len=1645723 2022-04-22T18:04:44.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 5382 completed, throughput=5MB/sec 2022-04-22T18:04:44.008 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-p-nyKzfrfjl9SLd off=0 len=1230113 2022-04-22T18:04:44.012 INFO:tasks.workunit.client.0.smithi115.stdout:op 5383 completed, throughput=5MB/sec 2022-04-22T18:04:44.012 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BzSSSUXqthTu4kD off=2001864 len=740824 2022-04-22T18:04:44.014 INFO:tasks.workunit.client.0.smithi115.stdout:op 5384 completed, throughput=5MB/sec 2022-04-22T18:04:44.119 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CWSlyVBvW7vM_cY off=4867936 len=48991 2022-04-22T18:04:44.120 INFO:tasks.workunit.client.0.smithi115.stdout:op 5385 completed, throughput=5MB/sec 2022-04-22T18:04:44.121 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tWTDMK1l-4uCat4 off=2475431 len=1868130 2022-04-22T18:04:44.121 INFO:tasks.workunit.client.0.smithi115.stdout:op 5386 completed, throughput=5MB/sec 2022-04-22T18:04:44.121 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:43.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:44 smithi115 bash[45495]: cluster 2022-04-22T18:04:42.830502+0000 mgr.y (mgr.34107) 1368 : cluster [DBG] pgmap v1940: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:44 smithi115 bash[46934]: cluster 2022-04-22T18:04:42.830502+0000 mgr.y (mgr.34107) 1368 : cluster [DBG] pgmap v1940: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:44.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:44 smithi121 bash[36902]: cluster 2022-04-22T18:04:42.830502+0000 mgr.y (mgr.34107) 1368 : cluster [DBG] pgmap v1940: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:45.018 INFO:tasks.workunit.client.0.smithi115.stdout: 1055: throughput=5MB/sec pending data=0 2022-04-22T18:04:45.019 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ynZFsPE7aPFlI1w off=2542368 len=170627 2022-04-22T18:04:45.023 INFO:tasks.workunit.client.0.smithi115.stdout:op 5387 completed, throughput=5MB/sec 2022-04-22T18:04:45.023 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s3XP_upXbunDl53 off=14360651 len=892673 2022-04-22T18:04:45.030 INFO:tasks.workunit.client.0.smithi115.stdout:op 5388 completed, throughput=5MB/sec 2022-04-22T18:04:45.030 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cegHo8XFKgu68wn off=1562786 len=103241 2022-04-22T18:04:45.033 INFO:tasks.workunit.client.0.smithi115.stdout:op 5389 completed, throughput=5MB/sec 2022-04-22T18:04:45.034 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qQxXW9XhRQcdOY8 off=4400579 len=1696290 2022-04-22T18:04:45.039 INFO:tasks.workunit.client.0.smithi115.stdout:op 5390 completed, throughput=5MB/sec 2022-04-22T18:04:45.039 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WZS1d8HGX0X7nk1 off=9669152 len=169402 2022-04-22T18:04:45.042 INFO:tasks.workunit.client.0.smithi115.stdout:op 5391 completed, throughput=5MB/sec 2022-04-22T18:04:45.042 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CYn8vuDrXaXhILB off=13291046 len=190556 2022-04-22T18:04:45.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 5392 completed, throughput=5MB/sec 2022-04-22T18:04:45.043 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-u3ie0UMDBQPYyfN off=215597 len=1009743 2022-04-22T18:04:45.045 INFO:tasks.workunit.client.0.smithi115.stdout:op 5393 completed, throughput=5MB/sec 2022-04-22T18:04:46.045 INFO:tasks.workunit.client.0.smithi115.stdout: 1056: throughput=5MB/sec pending data=0 2022-04-22T18:04:46.046 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lBP8eHrDYZCvLNK off=2670109 len=670228 2022-04-22T18:04:46.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 5394 completed, throughput=5MB/sec 2022-04-22T18:04:46.054 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-atFyJhooTm0i0ox off=7084336 len=1515909 2022-04-22T18:04:46.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 5395 completed, throughput=5MB/sec 2022-04-22T18:04:46.054 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-vydIUr0an9CRtpP off=7073486 len=1482807 2022-04-22T18:04:46.057 INFO:tasks.workunit.client.0.smithi115.stdout:op 5396 completed, throughput=5MB/sec 2022-04-22T18:04:46.057 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-K1EbMFYnGx0yN3Y off=7623011 len=1356236 2022-04-22T18:04:46.064 INFO:tasks.workunit.client.0.smithi115.stdout:op 5397 completed, throughput=5MB/sec 2022-04-22T18:04:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[45495]: cluster 2022-04-22T18:04:44.831599+0000 mgr.y (mgr.34107) 1369 : cluster [DBG] pgmap v1941: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[45495]: audit 2022-04-22T18:04:45.429252+0000 mon.a (mon.0) 2914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[45495]: audit 2022-04-22T18:04:45.831493+0000 mon.a (mon.0) 2915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[46934]: cluster 2022-04-22T18:04:44.831599+0000 mgr.y (mgr.34107) 1369 : cluster [DBG] pgmap v1941: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[46934]: audit 2022-04-22T18:04:45.429252+0000 mon.a (mon.0) 2914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:46 smithi115 bash[46934]: audit 2022-04-22T18:04:45.831493+0000 mon.a (mon.0) 2915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:46 smithi121 bash[36902]: cluster 2022-04-22T18:04:44.831599+0000 mgr.y (mgr.34107) 1369 : cluster [DBG] pgmap v1941: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:46 smithi121 bash[36902]: audit 2022-04-22T18:04:45.429252+0000 mon.a (mon.0) 2914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:46 smithi121 bash[36902]: audit 2022-04-22T18:04:45.831493+0000 mon.a (mon.0) 2915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:47.064 INFO:tasks.workunit.client.0.smithi115.stdout: 1057: throughput=5MB/sec pending data=0 2022-04-22T18:04:47.064 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DU1YUeIHoiU_ZEN off=5716531 len=1576905 2022-04-22T18:04:47.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 5398 completed, throughput=5MB/sec 2022-04-22T18:04:47.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5GZCh2789MTS2OW off=260148 len=3399 2022-04-22T18:04:47.070 INFO:tasks.workunit.client.0.smithi115.stdout:op 5399 completed, throughput=5MB/sec 2022-04-22T18:04:47.070 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w-T6Elmosz7qN5d off=8774745 len=1632829 2022-04-22T18:04:47.080 INFO:tasks.workunit.client.0.smithi115.stdout:op 5400 completed, throughput=5MB/sec 2022-04-22T18:04:47.080 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IOfxEJrb809QdYx off=1796389 len=1804336 2022-04-22T18:04:47.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 5401 completed, throughput=5MB/sec 2022-04-22T18:04:47.090 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Bm0QPPFZ3wqGZ12 off=544753 len=875236 2022-04-22T18:04:47.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 5402 completed, throughput=5MB/sec 2022-04-22T18:04:48.093 INFO:tasks.workunit.client.0.smithi115.stdout: 1058: throughput=5MB/sec pending data=0 2022-04-22T18:04:48.093 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZAcNFwjEMr3FS2i off=7146465 len=1748084 2022-04-22T18:04:48.102 INFO:tasks.workunit.client.0.smithi115.stdout:op 5403 completed, throughput=5MB/sec 2022-04-22T18:04:48.103 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sVCPM_IQU6tbJvt off=827809 len=1842488 2022-04-22T18:04:48.107 INFO:tasks.workunit.client.0.smithi115.stdout:op 5404 completed, throughput=5MB/sec 2022-04-22T18:04:48.107 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wc4rAPiOJS0_WJ_ off=8946926 len=1204988 2022-04-22T18:04:48.109 INFO:tasks.workunit.client.0.smithi115.stdout:op 5405 completed, throughput=5MB/sec 2022-04-22T18:04:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:48 smithi121 bash[36902]: cluster 2022-04-22T18:04:46.832666+0000 mgr.y (mgr.34107) 1370 : cluster [DBG] pgmap v1942: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:48 smithi121 bash[36902]: audit 2022-04-22T18:04:47.820123+0000 mon.a (mon.0) 2916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:48 smithi115 bash[45495]: cluster 2022-04-22T18:04:46.832666+0000 mgr.y (mgr.34107) 1370 : cluster [DBG] pgmap v1942: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:48 smithi115 bash[45495]: audit 2022-04-22T18:04:47.820123+0000 mon.a (mon.0) 2916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:48 smithi115 bash[46934]: cluster 2022-04-22T18:04:46.832666+0000 mgr.y (mgr.34107) 1370 : cluster [DBG] pgmap v1942: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:04:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:48 smithi115 bash[46934]: audit 2022-04-22T18:04:47.820123+0000 mon.a (mon.0) 2916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:49.110 INFO:tasks.workunit.client.0.smithi115.stdout: 1059: throughput=5MB/sec pending data=0 2022-04-22T18:04:49.110 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0f09ksc0CSJPSUM off=2063585 len=1171457 2022-04-22T18:04:49.114 INFO:tasks.workunit.client.0.smithi115.stdout:op 5406 completed, throughput=5MB/sec 2022-04-22T18:04:49.114 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ovaihB8aj0aFanB off=11414387 len=1575918 2022-04-22T18:04:49.124 INFO:tasks.workunit.client.0.smithi115.stdout:op 5407 completed, throughput=5MB/sec 2022-04-22T18:04:49.124 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kmarwXAFh4Ct0vA off=11743037 len=199506 2022-04-22T18:04:49.125 INFO:tasks.workunit.client.0.smithi115.stdout:op 5408 completed, throughput=5MB/sec 2022-04-22T18:04:49.125 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ceEPARYlU4vtRZe off=754695 len=344873 2022-04-22T18:04:49.129 INFO:tasks.workunit.client.0.smithi115.stdout:op 5409 completed, throughput=5MB/sec 2022-04-22T18:04:49.130 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lnBJhOi8k_UJRAP off=10093929 len=1323760 2022-04-22T18:04:49.135 INFO:tasks.workunit.client.0.smithi115.stdout:op 5410 completed, throughput=5MB/sec 2022-04-22T18:04:49.135 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lxZbP7-dTh6tsV0 off=3382562 len=23561 2022-04-22T18:04:49.136 INFO:tasks.workunit.client.0.smithi115.stdout:op 5411 completed, throughput=5MB/sec 2022-04-22T18:04:49.136 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NYZwSKwqddK1W8w off=1682660 len=654274 2022-04-22T18:04:49.138 INFO:tasks.workunit.client.0.smithi115.stdout:op 5412 completed, throughput=5MB/sec 2022-04-22T18:04:49.138 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ec186yaXSq5wTkD off=17797831 len=1824132 2022-04-22T18:04:49.146 INFO:tasks.workunit.client.0.smithi115.stdout:op 5413 completed, throughput=5MB/sec 2022-04-22T18:04:50.146 INFO:tasks.workunit.client.0.smithi115.stdout: 1060: throughput=5MB/sec pending data=0 2022-04-22T18:04:50.147 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dYO0Sf3eoq8H7q5 off=3050316 len=855428 2022-04-22T18:04:50.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 5414 completed, throughput=5MB/sec 2022-04-22T18:04:50.148 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1i3VpnVJ3jJk-aQ off=13824171 len=779606 2022-04-22T18:04:50.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 5415 completed, throughput=5MB/sec 2022-04-22T18:04:50.150 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Tmf63Ibasn5rCIk off=743127 len=511382 2022-04-22T18:04:50.151 INFO:tasks.workunit.client.0.smithi115.stdout:op 5416 completed, throughput=5MB/sec 2022-04-22T18:04:50.151 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gSCeUEAZx7cC7Ln off=14056306 len=2044997 2022-04-22T18:04:50.154 INFO:tasks.workunit.client.0.smithi115.stdout:op 5417 completed, throughput=5MB/sec 2022-04-22T18:04:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:50 smithi121 bash[36902]: cluster 2022-04-22T18:04:48.833923+0000 mgr.y (mgr.34107) 1371 : cluster [DBG] pgmap v1943: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:50 smithi115 bash[45495]: cluster 2022-04-22T18:04:48.833923+0000 mgr.y (mgr.34107) 1371 : cluster [DBG] pgmap v1943: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:50 smithi115 bash[46934]: cluster 2022-04-22T18:04:48.833923+0000 mgr.y (mgr.34107) 1371 : cluster [DBG] pgmap v1943: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:04:51.155 INFO:tasks.workunit.client.0.smithi115.stdout: 1061: throughput=5MB/sec pending data=0 2022-04-22T18:04:51.173 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JB5yHP8wDBi528q off=6855847 len=142605 2022-04-22T18:04:51.174 INFO:tasks.workunit.client.0.smithi115.stdout:op 5418 completed, throughput=5MB/sec 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FZZBEGciSLEe-oN off=1481724 len=1489913 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 5419 completed, throughput=5MB/sec 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FuwR_ZrErBpQL2r off=8997021 len=1481973 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 5420 completed, throughput=5MB/sec 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z-wKFjJPEqsvYuF off=1023998 len=676958 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 5421 completed, throughput=5MB/sec 2022-04-22T18:04:51.175 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LoN98Sn6mxTnb0E off=9672958 len=1313798 2022-04-22T18:04:51.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 5422 completed, throughput=5MB/sec 2022-04-22T18:04:51.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:51 smithi115 bash[45495]: audit 2022-04-22T18:04:50.432294+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:51.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:51 smithi115 bash[45495]: audit 2022-04-22T18:04:50.836885+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:51.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:51 smithi115 bash[46934]: audit 2022-04-22T18:04:50.432294+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:51.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:51 smithi115 bash[46934]: audit 2022-04-22T18:04:50.836885+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:51 smithi121 bash[36902]: audit 2022-04-22T18:04:50.432294+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:51 smithi121 bash[36902]: audit 2022-04-22T18:04:50.836885+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:51.740Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:52.182 INFO:tasks.workunit.client.0.smithi115.stdout: 1062: throughput=5MB/sec pending data=0 2022-04-22T18:04:52.182 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lWmOHvfQndUyAZr off=6940957 len=1938284 2022-04-22T18:04:52.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 5423 completed, throughput=5MB/sec 2022-04-22T18:04:52.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4CUbfKzTBcoxoMF off=39410 len=1724215 2022-04-22T18:04:52.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 5424 completed, throughput=5MB/sec 2022-04-22T18:04:52.202 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--E5kGanNtIu_lEd off=2609843 len=1817043 2022-04-22T18:04:52.209 INFO:tasks.workunit.client.0.smithi115.stdout:op 5425 completed, throughput=5MB/sec 2022-04-22T18:04:52.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:52 smithi121 bash[36902]: cluster 2022-04-22T18:04:50.835159+0000 mgr.y (mgr.34107) 1372 : cluster [DBG] pgmap v1944: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:52.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:52 smithi121 bash[36902]: audit 2022-04-22T18:04:52.125111+0000 mon.a (mon.0) 2919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:04:52.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:52 smithi121 bash[36902]: audit 2022-04-22T18:04:52.128234+0000 mon.a (mon.0) 2920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:04:52.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:52 smithi121 bash[36902]: audit 2022-04-22T18:04:52.129912+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:04:52.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[46934]: cluster 2022-04-22T18:04:50.835159+0000 mgr.y (mgr.34107) 1372 : cluster [DBG] pgmap v1944: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:52.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[46934]: audit 2022-04-22T18:04:52.125111+0000 mon.a (mon.0) 2919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[46934]: audit 2022-04-22T18:04:52.128234+0000 mon.a (mon.0) 2920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[46934]: audit 2022-04-22T18:04:52.129912+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[45495]: cluster 2022-04-22T18:04:50.835159+0000 mgr.y (mgr.34107) 1372 : cluster [DBG] pgmap v1944: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[45495]: audit 2022-04-22T18:04:52.125111+0000 mon.a (mon.0) 2919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[45495]: audit 2022-04-22T18:04:52.128234+0000 mon.a (mon.0) 2920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:04:52.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:52 smithi115 bash[45495]: audit 2022-04-22T18:04:52.129912+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:04:53.209 INFO:tasks.workunit.client.0.smithi115.stdout: 1063: throughput=5MB/sec pending data=0 2022-04-22T18:04:53.210 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jHZu9aM9ejNvfyd off=606627 len=381749 2022-04-22T18:04:53.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 5426 completed, throughput=5MB/sec 2022-04-22T18:04:53.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KY6x7_S-91WA_qM off=652155 len=956833 2022-04-22T18:04:53.219 INFO:tasks.workunit.client.0.smithi115.stdout:op 5427 completed, throughput=5MB/sec 2022-04-22T18:04:53.219 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yrTXysYjrgl5fgl off=12169079 len=1248680 2022-04-22T18:04:53.222 INFO:tasks.workunit.client.0.smithi115.stdout:op 5428 completed, throughput=5MB/sec 2022-04-22T18:04:53.222 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lxWx8QtQM2Hn5bh off=10220824 len=654435 2022-04-22T18:04:53.224 INFO:tasks.workunit.client.0.smithi115.stdout:op 5429 completed, throughput=5MB/sec 2022-04-22T18:04:53.224 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0fDsMZ6RnOBAHHn off=643629 len=738475 2022-04-22T18:04:53.230 INFO:tasks.workunit.client.0.smithi115.stdout:op 5430 completed, throughput=5MB/sec 2022-04-22T18:04:53.230 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-QTc4qXCBZPin0vK off=600133 len=1757908 2022-04-22T18:04:53.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 5431 completed, throughput=5MB/sec 2022-04-22T18:04:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:53 smithi121 bash[36902]: audit 2022-04-22T18:04:52.497871+0000 mon.a (mon.0) 2922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:04:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:53 smithi121 bash[36902]: audit 2022-04-22T18:04:52.823015+0000 mon.a (mon.0) 2923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:53 smithi121 bash[36902]: cluster 2022-04-22T18:04:52.836208+0000 mgr.y (mgr.34107) 1373 : cluster [DBG] pgmap v1945: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:53.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:04:53.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:04:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[45495]: audit 2022-04-22T18:04:52.497871+0000 mon.a (mon.0) 2922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:04:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[45495]: audit 2022-04-22T18:04:52.823015+0000 mon.a (mon.0) 2923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[45495]: cluster 2022-04-22T18:04:52.836208+0000 mgr.y (mgr.34107) 1373 : cluster [DBG] pgmap v1945: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[46934]: audit 2022-04-22T18:04:52.497871+0000 mon.a (mon.0) 2922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:04:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[46934]: audit 2022-04-22T18:04:52.823015+0000 mon.a (mon.0) 2923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:53 smithi115 bash[46934]: cluster 2022-04-22T18:04:52.836208+0000 mgr.y (mgr.34107) 1373 : cluster [DBG] pgmap v1945: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:04:54.234 INFO:tasks.workunit.client.0.smithi115.stdout: 1064: throughput=5MB/sec pending data=0 2022-04-22T18:04:54.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8ED4J_Vl72cNB-r off=291247 len=1486987 2022-04-22T18:04:54.238 INFO:tasks.workunit.client.0.smithi115.stdout:op 5432 completed, throughput=5MB/sec 2022-04-22T18:04:54.238 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-a1JnXR-qy5Iwm3I off=2364633 len=1014282 2022-04-22T18:04:54.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 5433 completed, throughput=5MB/sec 2022-04-22T18:04:54.247 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qjSYGwotyelGDVQ off=15788641 len=864481 2022-04-22T18:04:54.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 5434 completed, throughput=5MB/sec 2022-04-22T18:04:54.250 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-bx_vKBO_RItmKtK off=180624 len=1263922 2022-04-22T18:04:54.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 5435 completed, throughput=5MB/sec 2022-04-22T18:04:55.257 INFO:tasks.workunit.client.0.smithi115.stdout: 1065: throughput=5MB/sec pending data=0 2022-04-22T18:04:55.257 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yejJRJh6_ZEzJjz off=356870 len=1922371 2022-04-22T18:04:55.266 INFO:tasks.workunit.client.0.smithi115.stdout:op 5436 completed, throughput=5MB/sec 2022-04-22T18:04:55.267 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TfOLzkwD1Cz6KmL off=2886944 len=1507002 2022-04-22T18:04:55.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 5437 completed, throughput=5MB/sec 2022-04-22T18:04:55.271 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yuSMJXWUlSAEjFR off=712215 len=561512 2022-04-22T18:04:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:op 5438 completed, throughput=5MB/sec 2022-04-22T18:04:55.272 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KRJ6TA9zV9TeSL5 off=22461769 len=135283 2022-04-22T18:04:55.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 5439 completed, throughput=5MB/sec 2022-04-22T18:04:55.276 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4I_nqKVMRCwOLcp off=14937593 len=1909476 2022-04-22T18:04:55.279 INFO:tasks.workunit.client.0.smithi115.stdout:op 5440 completed, throughput=5MB/sec 2022-04-22T18:04:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:55 smithi121 bash[36902]: cluster 2022-04-22T18:04:54.837310+0000 mgr.y (mgr.34107) 1374 : cluster [DBG] pgmap v1946: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:55 smithi121 bash[36902]: audit 2022-04-22T18:04:55.435539+0000 mon.a (mon.0) 2924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:55 smithi121 bash[36902]: audit 2022-04-22T18:04:55.839774+0000 mon.a (mon.0) 2925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:56.279 INFO:tasks.workunit.client.0.smithi115.stdout: 1066: throughput=5MB/sec pending data=0 2022-04-22T18:04:56.279 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kEof4FXOdnSJGUC off=1180309 len=1449834 2022-04-22T18:04:56.288 INFO:tasks.workunit.client.0.smithi115.stdout:op 5441 completed, throughput=5MB/sec 2022-04-22T18:04:56.289 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9H6ZicIIMbp3QC7 off=5232703 len=1996031 2022-04-22T18:04:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[46934]: cluster 2022-04-22T18:04:54.837310+0000 mgr.y (mgr.34107) 1374 : cluster [DBG] pgmap v1946: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[46934]: audit 2022-04-22T18:04:55.435539+0000 mon.a (mon.0) 2924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[46934]: audit 2022-04-22T18:04:55.839774+0000 mon.a (mon.0) 2925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:56.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[45495]: cluster 2022-04-22T18:04:54.837310+0000 mgr.y (mgr.34107) 1374 : cluster [DBG] pgmap v1946: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:04:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[45495]: audit 2022-04-22T18:04:55.435539+0000 mon.a (mon.0) 2924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:04:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:55 smithi115 bash[45495]: audit 2022-04-22T18:04:55.839774+0000 mon.a (mon.0) 2925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:04:56.294 INFO:tasks.workunit.client.0.smithi115.stdout:op 5442 completed, throughput=5MB/sec 2022-04-22T18:04:56.295 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S2JB-h4PBLT7P_7 off=2284007 len=74765 2022-04-22T18:04:56.295 INFO:tasks.workunit.client.0.smithi115.stdout:op 5443 completed, throughput=5MB/sec 2022-04-22T18:04:56.296 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XmKgjWYv_K2XSrI off=1854609 len=2085092 2022-04-22T18:04:56.300 INFO:tasks.workunit.client.0.smithi115.stdout:op 5444 completed, throughput=5MB/sec 2022-04-22T18:04:57.300 INFO:tasks.workunit.client.0.smithi115.stdout: 1067: throughput=5MB/sec pending data=0 2022-04-22T18:04:57.300 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QZqnef2zIk5OdHV off=16060920 len=553521 2022-04-22T18:04:57.306 INFO:tasks.workunit.client.0.smithi115.stdout:op 5445 completed, throughput=5MB/sec 2022-04-22T18:04:57.306 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GF7VxnIDVVd1Ng0 off=7571143 len=1834682 2022-04-22T18:04:57.311 INFO:tasks.workunit.client.0.smithi115.stdout:op 5446 completed, throughput=5MB/sec 2022-04-22T18:04:57.312 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z1aNe0LZoAuIyoY off=8837450 len=1039246 2022-04-22T18:04:57.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 5447 completed, throughput=5MB/sec 2022-04-22T18:04:57.319 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iIGDYVp0-5CoAxW off=334364 len=1364142 2022-04-22T18:04:57.322 INFO:tasks.workunit.client.0.smithi115.stdout:op 5448 completed, throughput=5MB/sec 2022-04-22T18:04:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:57 smithi121 bash[36902]: cluster 2022-04-22T18:04:56.838402+0000 mgr.y (mgr.34107) 1375 : cluster [DBG] pgmap v1947: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:57 smithi121 bash[36902]: audit 2022-04-22T18:04:57.826386+0000 mon.a (mon.0) 2926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:58.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:57 smithi115 bash[45495]: cluster 2022-04-22T18:04:56.838402+0000 mgr.y (mgr.34107) 1375 : cluster [DBG] pgmap v1947: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:58.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:57 smithi115 bash[45495]: audit 2022-04-22T18:04:57.826386+0000 mon.a (mon.0) 2926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:58.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:57 smithi115 bash[46934]: cluster 2022-04-22T18:04:56.838402+0000 mgr.y (mgr.34107) 1375 : cluster [DBG] pgmap v1947: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:04:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:57 smithi115 bash[46934]: audit 2022-04-22T18:04:57.826386+0000 mon.a (mon.0) 2926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:04:58.322 INFO:tasks.workunit.client.0.smithi115.stdout: 1068: throughput=5MB/sec pending data=0 2022-04-22T18:04:58.323 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-87aACHcKpogTJXH off=6668418 len=1784041 2022-04-22T18:04:58.326 INFO:tasks.workunit.client.0.smithi115.stdout:op 5449 completed, throughput=5MB/sec 2022-04-22T18:04:58.326 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LIVSPYDlzq88njL off=4969800 len=896598 2022-04-22T18:04:58.333 INFO:tasks.workunit.client.0.smithi115.stdout:op 5450 completed, throughput=5MB/sec 2022-04-22T18:04:58.333 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sL45o9AQ8XUsMHQ off=5638466 len=985335 2022-04-22T18:04:58.335 INFO:tasks.workunit.client.0.smithi115.stdout:op 5451 completed, throughput=5MB/sec 2022-04-22T18:04:58.335 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ARPXfDIOLzbScqK off=684155 len=1780626 2022-04-22T18:04:58.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 5452 completed, throughput=5MB/sec 2022-04-22T18:04:59.345 INFO:tasks.workunit.client.0.smithi115.stdout: 1069: throughput=5MB/sec pending data=0 2022-04-22T18:04:59.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-acs7aQFWgXG9r6z off=1969820 len=1669222 2022-04-22T18:04:59.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 5453 completed, throughput=5MB/sec 2022-04-22T18:04:59.355 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-43rdElk7Lg_pn4V off=3087805 len=1547544 2022-04-22T18:04:59.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 5454 completed, throughput=5MB/sec 2022-04-22T18:04:59.358 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vd9Mk39RmzPGAA4 off=14122227 len=1231432 2022-04-22T18:04:59.361 INFO:tasks.workunit.client.0.smithi115.stdout:op 5455 completed, throughput=5MB/sec 2022-04-22T18:04:59.361 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xl0Cy18ZvzrHyYk off=18343340 len=346778 2022-04-22T18:04:59.362 INFO:tasks.workunit.client.0.smithi115.stdout:op 5456 completed, throughput=5MB/sec 2022-04-22T18:05:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:04:59 smithi121 bash[36902]: cluster 2022-04-22T18:04:58.839674+0000 mgr.y (mgr.34107) 1376 : cluster [DBG] pgmap v1948: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:00.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:04:59 smithi115 bash[45495]: cluster 2022-04-22T18:04:58.839674+0000 mgr.y (mgr.34107) 1376 : cluster [DBG] pgmap v1948: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:00.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:04:59 smithi115 bash[46934]: cluster 2022-04-22T18:04:58.839674+0000 mgr.y (mgr.34107) 1376 : cluster [DBG] pgmap v1948: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:00.362 INFO:tasks.workunit.client.0.smithi115.stdout: 1070: throughput=5MB/sec pending data=0 2022-04-22T18:05:00.363 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WJOPCaiGcn84C3r off=8539895 len=914270 2022-04-22T18:05:00.365 INFO:tasks.workunit.client.0.smithi115.stdout:op 5457 completed, throughput=5MB/sec 2022-04-22T18:05:00.365 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--fuTS8Q0vpN-1aZ off=2352765 len=255175 2022-04-22T18:05:00.367 INFO:tasks.workunit.client.0.smithi115.stdout:op 5458 completed, throughput=5MB/sec 2022-04-22T18:05:00.367 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LvooAXRJwUEgDPI off=1187008 len=761229 2022-04-22T18:05:00.369 INFO:tasks.workunit.client.0.smithi115.stdout:op 5459 completed, throughput=5MB/sec 2022-04-22T18:05:00.369 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jVle4PuKIInCyVP off=803211 len=679788 2022-04-22T18:05:00.375 INFO:tasks.workunit.client.0.smithi115.stdout:op 5460 completed, throughput=5MB/sec 2022-04-22T18:05:00.375 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VJlOGkOWo8mgdO6 off=5476739 len=2005744 2022-04-22T18:05:00.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 5461 completed, throughput=5MB/sec 2022-04-22T18:05:00.379 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7vGtd67jLuyPe7I off=975269 len=1072766 2022-04-22T18:05:00.386 INFO:tasks.workunit.client.0.smithi115.stdout:op 5462 completed, throughput=5MB/sec 2022-04-22T18:05:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:00 smithi121 bash[36902]: audit 2022-04-22T18:05:00.438687+0000 mon.a (mon.0) 2927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:00 smithi121 bash[36902]: audit 2022-04-22T18:05:00.843170+0000 mon.a (mon.0) 2928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:01.271 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:05:01.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:00 smithi115 bash[45495]: audit 2022-04-22T18:05:00.438687+0000 mon.a (mon.0) 2927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:01.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:00 smithi115 bash[45495]: audit 2022-04-22T18:05:00.843170+0000 mon.a (mon.0) 2928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:01.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:00 smithi115 bash[46934]: audit 2022-04-22T18:05:00.438687+0000 mon.a (mon.0) 2927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:01.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:00 smithi115 bash[46934]: audit 2022-04-22T18:05:00.843170+0000 mon.a (mon.0) 2928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:01.386 INFO:tasks.workunit.client.0.smithi115.stdout: 1071: throughput=5MB/sec pending data=0 2022-04-22T18:05:01.386 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v2QQYo7Eol-2yOC off=1347075 len=565281 2022-04-22T18:05:01.389 INFO:tasks.workunit.client.0.smithi115.stdout:op 5463 completed, throughput=5MB/sec 2022-04-22T18:05:01.389 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YSuxv6f_T-AeZAc off=4599890 len=835489 2022-04-22T18:05:01.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 5464 completed, throughput=5MB/sec 2022-04-22T18:05:01.391 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YhOQnnFynR5HdFm off=6881011 len=1458128 2022-04-22T18:05:01.398 INFO:tasks.workunit.client.0.smithi115.stdout:op 5465 completed, throughput=5MB/sec 2022-04-22T18:05:01.399 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-S0FPTm9dFy08eDl off=1502185 len=183086 2022-04-22T18:05:01.402 INFO:tasks.workunit.client.0.smithi115.stdout:op 5466 completed, throughput=5MB/sec 2022-04-22T18:05:01.402 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jw4u0K2HkSL8qGX off=1908534 len=1789758 2022-04-22T18:05:01.405 INFO:tasks.workunit.client.0.smithi115.stdout:op 5467 completed, throughput=5MB/sec 2022-04-22T18:05:01.405 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RqNwG73w1q2oErc off=15116289 len=598820 2022-04-22T18:05:01.410 INFO:tasks.workunit.client.0.smithi115.stdout:op 5468 completed, throughput=5MB/sec 2022-04-22T18:05:01.614 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:05:01.614 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (28m) 7m ago 41m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:05:01.614 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (41m) 8m ago 41m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (30m) 7m ago 42m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (29m) 8m ago 42m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (37m) 8m ago 47m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (37m) 7m ago 49m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (36m) 7m ago 49m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (36m) 8m ago 47m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (36m) 7m ago 47m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (29m) 7m ago 42m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:05:01.615 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (29m) 8m ago 42m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (34m) 7m ago 46m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (34m) 7m ago 46m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (33m) 7m ago 45m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (33m) 7m ago 45m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (32m) 8m ago 44m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (31m) 8m ago 44m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (31m) 8m ago 43m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (30m) 8m ago 43m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:05:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (28m) 8m ago 42m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:02.012 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:05:02.013 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:05:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:01.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:01 smithi115 bash[45495]: cluster 2022-04-22T18:05:00.840874+0000 mgr.y (mgr.34107) 1377 : cluster [DBG] pgmap v1949: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:01 smithi115 bash[45495]: audit 2022-04-22T18:05:01.250994+0000 mgr.y (mgr.34107) 1378 : audit [DBG] from='client.45781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:01 smithi115 bash[46934]: cluster 2022-04-22T18:05:00.840874+0000 mgr.y (mgr.34107) 1377 : cluster [DBG] pgmap v1949: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:01 smithi115 bash[46934]: audit 2022-04-22T18:05:01.250994+0000 mgr.y (mgr.34107) 1378 : audit [DBG] from='client.45781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:01 smithi121 bash[36902]: cluster 2022-04-22T18:05:00.840874+0000 mgr.y (mgr.34107) 1377 : cluster [DBG] pgmap v1949: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:01 smithi121 bash[36902]: audit 2022-04-22T18:05:01.250994+0000 mgr.y (mgr.34107) 1378 : audit [DBG] from='client.45781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:02.410 INFO:tasks.workunit.client.0.smithi115.stdout: 1072: throughput=5MB/sec pending data=0 2022-04-22T18:05:02.410 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5AcNc6jdxjk5_fR off=10225362 len=1079823 2022-04-22T18:05:02.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 5469 completed, throughput=5MB/sec 2022-04-22T18:05:02.414 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JlpvwWOZynBUH07 off=4006212 len=1098747 2022-04-22T18:05:02.417 INFO:tasks.workunit.client.0.smithi115.stdout:op 5470 completed, throughput=5MB/sec 2022-04-22T18:05:02.417 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BcsAn_5HFb83SEO off=1222413 len=424407 2022-04-22T18:05:02.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 5471 completed, throughput=5MB/sec 2022-04-22T18:05:02.422 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WJMthBDvRChPORF off=899161 len=1610546 2022-04-22T18:05:02.424 INFO:tasks.workunit.client.0.smithi115.stdout:op 5472 completed, throughput=5MB/sec 2022-04-22T18:05:02.425 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qsuoPPnL0onI000 off=844922 len=1389332 2022-04-22T18:05:02.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 5473 completed, throughput=5MB/sec 2022-04-22T18:05:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:02 smithi121 bash[36902]: audit 2022-04-22T18:05:01.587736+0000 mgr.y (mgr.34107) 1379 : audit [DBG] from='client.45787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:03.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:02 smithi121 bash[36902]: audit 2022-04-22T18:05:02.005586+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.115:0/2921010491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:03.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:02 smithi121 bash[36902]: audit 2022-04-22T18:05:02.829803+0000 mon.a (mon.0) 2929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:03.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[45495]: audit 2022-04-22T18:05:01.587736+0000 mgr.y (mgr.34107) 1379 : audit [DBG] from='client.45787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:03.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[45495]: audit 2022-04-22T18:05:02.005586+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.115:0/2921010491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:03.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[45495]: audit 2022-04-22T18:05:02.829803+0000 mon.a (mon.0) 2929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:03.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[46934]: audit 2022-04-22T18:05:01.587736+0000 mgr.y (mgr.34107) 1379 : audit [DBG] from='client.45787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:03.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[46934]: audit 2022-04-22T18:05:02.005586+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.115:0/2921010491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:03.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:02 smithi115 bash[46934]: audit 2022-04-22T18:05:02.829803+0000 mon.a (mon.0) 2929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:03.428 INFO:tasks.workunit.client.0.smithi115.stdout: 1073: throughput=5MB/sec pending data=0 2022-04-22T18:05:03.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EEcAZovUiZbNzsv off=18704655 len=681549 2022-04-22T18:05:03.435 INFO:tasks.workunit.client.0.smithi115.stdout:op 5474 completed, throughput=5MB/sec 2022-04-22T18:05:03.435 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-36cH-GUztxN4D5j off=1986975 len=1928225 2022-04-22T18:05:03.446 INFO:tasks.workunit.client.0.smithi115.stdout:op 5475 completed, throughput=5MB/sec 2022-04-22T18:05:03.446 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KPlUKY01pCV16so off=2960589 len=1754275 2022-04-22T18:05:03.450 INFO:tasks.workunit.client.0.smithi115.stdout:op 5476 completed, throughput=5MB/sec 2022-04-22T18:05:03.450 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MeXS0pGjrQgwcmh off=20545050 len=1227337 2022-04-22T18:05:03.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 5477 completed, throughput=5MB/sec 2022-04-22T18:05:03.921 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:03.622Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:03 smithi121 bash[36902]: cluster 2022-04-22T18:05:02.841928+0000 mgr.y (mgr.34107) 1380 : cluster [DBG] pgmap v1950: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:04.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:03 smithi115 bash[45495]: cluster 2022-04-22T18:05:02.841928+0000 mgr.y (mgr.34107) 1380 : cluster [DBG] pgmap v1950: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:04.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:03 smithi115 bash[46934]: cluster 2022-04-22T18:05:02.841928+0000 mgr.y (mgr.34107) 1380 : cluster [DBG] pgmap v1950: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:04.456 INFO:tasks.workunit.client.0.smithi115.stdout: 1074: throughput=5MB/sec pending data=0 2022-04-22T18:05:04.457 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5tmC7RF1_K6IWGh off=10759196 len=1287056 2022-04-22T18:05:04.465 INFO:tasks.workunit.client.0.smithi115.stdout:op 5478 completed, throughput=5MB/sec 2022-04-22T18:05:04.465 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GvKo-7PnlbGgDbg off=11255903 len=740814 2022-04-22T18:05:04.470 INFO:tasks.workunit.client.0.smithi115.stdout:op 5479 completed, throughput=5MB/sec 2022-04-22T18:05:04.470 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aJsjEmU1YpumghH off=8679137 len=858567 2022-04-22T18:05:04.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 5480 completed, throughput=5MB/sec 2022-04-22T18:05:04.473 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_CPOj0zdQ2f8urR off=16310262 len=1508584 2022-04-22T18:05:04.479 INFO:tasks.workunit.client.0.smithi115.stdout:op 5481 completed, throughput=5MB/sec 2022-04-22T18:05:04.479 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mKJTZiYNyGUSPLF off=12558194 len=366748 2022-04-22T18:05:04.481 INFO:tasks.workunit.client.0.smithi115.stdout:op 5482 completed, throughput=5MB/sec 2022-04-22T18:05:05.481 INFO:tasks.workunit.client.0.smithi115.stdout: 1075: throughput=5MB/sec pending data=0 2022-04-22T18:05:05.481 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mGoiMhvqXrwCkx9 off=13266419 len=172038 2022-04-22T18:05:05.485 INFO:tasks.workunit.client.0.smithi115.stdout:op 5483 completed, throughput=5MB/sec 2022-04-22T18:05:05.486 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tUEYqOo9A8LgUsd off=6215726 len=490107 2022-04-22T18:05:05.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 5484 completed, throughput=5MB/sec 2022-04-22T18:05:05.490 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jiyV0jRyQKQL222 off=9322985 len=1532872 2022-04-22T18:05:05.500 INFO:tasks.workunit.client.0.smithi115.stdout:op 5485 completed, throughput=5MB/sec 2022-04-22T18:05:05.500 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kI2uE44Xk4rex8b off=4211556 len=826265 2022-04-22T18:05:05.502 INFO:tasks.workunit.client.0.smithi115.stdout:op 5486 completed, throughput=5MB/sec 2022-04-22T18:05:05.502 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hTcPpi28FDoVEWk off=3029917 len=1753620 2022-04-22T18:05:05.504 INFO:tasks.workunit.client.0.smithi115.stdout:op 5487 completed, throughput=5MB/sec 2022-04-22T18:05:05.505 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WZS1d8HGX0X7nk1 off=8491618 len=1972070 2022-04-22T18:05:05.512 INFO:tasks.workunit.client.0.smithi115.stdout:op 5488 completed, throughput=5MB/sec 2022-04-22T18:05:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:05 smithi121 bash[36902]: cluster 2022-04-22T18:05:04.843141+0000 mgr.y (mgr.34107) 1381 : cluster [DBG] pgmap v1951: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:05:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:05 smithi121 bash[36902]: audit 2022-04-22T18:05:05.441703+0000 mon.a (mon.0) 2930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:05 smithi121 bash[36902]: audit 2022-04-22T18:05:05.846737+0000 mon.a (mon.0) 2931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[45495]: cluster 2022-04-22T18:05:04.843141+0000 mgr.y (mgr.34107) 1381 : cluster [DBG] pgmap v1951: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:05:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[45495]: audit 2022-04-22T18:05:05.441703+0000 mon.a (mon.0) 2930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[45495]: audit 2022-04-22T18:05:05.846737+0000 mon.a (mon.0) 2931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[46934]: cluster 2022-04-22T18:05:04.843141+0000 mgr.y (mgr.34107) 1381 : cluster [DBG] pgmap v1951: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:05:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[46934]: audit 2022-04-22T18:05:05.441703+0000 mon.a (mon.0) 2930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:05 smithi115 bash[46934]: audit 2022-04-22T18:05:05.846737+0000 mon.a (mon.0) 2931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:06.512 INFO:tasks.workunit.client.0.smithi115.stdout: 1076: throughput=5MB/sec pending data=0 2022-04-22T18:05:06.513 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TF1-JputJW40afU off=711671 len=379775 2022-04-22T18:05:06.514 INFO:tasks.workunit.client.0.smithi115.stdout:op 5489 completed, throughput=5MB/sec 2022-04-22T18:05:06.514 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CtR1FBFRmZeE3Ex off=14684815 len=1143722 2022-04-22T18:05:06.517 INFO:tasks.workunit.client.0.smithi115.stdout:op 5490 completed, throughput=5MB/sec 2022-04-22T18:05:06.517 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JqgcvV4m3g1QQdU off=18050273 len=292238 2022-04-22T18:05:06.518 INFO:tasks.workunit.client.0.smithi115.stdout:op 5491 completed, throughput=5MB/sec 2022-04-22T18:05:06.518 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2Sq5_IvtVTfMQaY off=11892168 len=2014509 2022-04-22T18:05:06.522 INFO:tasks.workunit.client.0.smithi115.stdout:op 5492 completed, throughput=5MB/sec 2022-04-22T18:05:07.523 INFO:tasks.workunit.client.0.smithi115.stdout: 1077: throughput=5MB/sec pending data=0 2022-04-22T18:05:07.523 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fUhh7F9Ngbob0Pf off=5470552 len=364283 2022-04-22T18:05:07.528 INFO:tasks.workunit.client.0.smithi115.stdout:op 5493 completed, throughput=5MB/sec 2022-04-22T18:05:07.528 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IzBJ4aHXIr7-fkZ off=18270942 len=1510692 2022-04-22T18:05:07.537 INFO:tasks.workunit.client.0.smithi115.stdout:op 5494 completed, throughput=5MB/sec 2022-04-22T18:05:07.537 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WqFgN-246thbuWJ off=10657157 len=1869541 2022-04-22T18:05:07.540 INFO:tasks.workunit.client.0.smithi115.stdout:op 5495 completed, throughput=5MB/sec 2022-04-22T18:05:07.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-23yM2MADwPN6jtV off=1992636 len=146819 2022-04-22T18:05:07.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 5496 completed, throughput=5MB/sec 2022-04-22T18:05:07.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Dzus6HU3bpV06Kl off=2027062 len=2049774 2022-04-22T18:05:07.550 INFO:tasks.workunit.client.0.smithi115.stdout:op 5497 completed, throughput=5MB/sec 2022-04-22T18:05:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:07 smithi121 bash[36902]: cluster 2022-04-22T18:05:06.844294+0000 mgr.y (mgr.34107) 1382 : cluster [DBG] pgmap v1952: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:07 smithi121 bash[36902]: audit 2022-04-22T18:05:07.832750+0000 mon.a (mon.0) 2932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:08.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:07 smithi115 bash[45495]: cluster 2022-04-22T18:05:06.844294+0000 mgr.y (mgr.34107) 1382 : cluster [DBG] pgmap v1952: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:08.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:07 smithi115 bash[45495]: audit 2022-04-22T18:05:07.832750+0000 mon.a (mon.0) 2932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:08.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:07 smithi115 bash[46934]: cluster 2022-04-22T18:05:06.844294+0000 mgr.y (mgr.34107) 1382 : cluster [DBG] pgmap v1952: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:08.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:07 smithi115 bash[46934]: audit 2022-04-22T18:05:07.832750+0000 mon.a (mon.0) 2932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:08.550 INFO:tasks.workunit.client.0.smithi115.stdout: 1078: throughput=5MB/sec pending data=0 2022-04-22T18:05:08.550 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--k2vTt8kTLByi09 off=3032262 len=1024855 2022-04-22T18:05:08.559 INFO:tasks.workunit.client.0.smithi115.stdout:op 5498 completed, throughput=5MB/sec 2022-04-22T18:05:08.559 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JLSXlq1JfNokdw1 off=7679291 len=1826780 2022-04-22T18:05:08.562 INFO:tasks.workunit.client.0.smithi115.stdout:op 5499 completed, throughput=5MB/sec 2022-04-22T18:05:08.562 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5u-0jluSsYkplls off=4631639 len=1827307 2022-04-22T18:05:08.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 5500 completed, throughput=5MB/sec 2022-04-22T18:05:09.564 INFO:tasks.workunit.client.0.smithi115.stdout: 1079: throughput=5MB/sec pending data=0 2022-04-22T18:05:09.565 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--yFZmU7UiE5W0E8 off=11469334 len=1530245 2022-04-22T18:05:09.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 5501 completed, throughput=5MB/sec 2022-04-22T18:05:09.568 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TsF3RjbZ8NyXbnB off=1510944 len=1005997 2022-04-22T18:05:09.571 INFO:tasks.workunit.client.0.smithi115.stdout:op 5502 completed, throughput=5MB/sec 2022-04-22T18:05:09.571 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ouam6J57ykUOGF- off=11295416 len=572127 2022-04-22T18:05:09.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 5503 completed, throughput=5MB/sec 2022-04-22T18:05:09.577 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6xm9UnjhhIxgiX6 off=2840424 len=878041 2022-04-22T18:05:09.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 5504 completed, throughput=5MB/sec 2022-04-22T18:05:09.585 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uMwcu5lxqLGGS9z off=11392739 len=647934 2022-04-22T18:05:09.589 INFO:tasks.workunit.client.0.smithi115.stdout:op 5505 completed, throughput=5MB/sec 2022-04-22T18:05:09.589 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4hO1ies7PrEgjsd off=11676625 len=1599682 2022-04-22T18:05:09.592 INFO:tasks.workunit.client.0.smithi115.stdout:op 5506 completed, throughput=5MB/sec 2022-04-22T18:05:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:09 smithi121 bash[36902]: cluster 2022-04-22T18:05:08.845631+0000 mgr.y (mgr.34107) 1383 : cluster [DBG] pgmap v1953: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:05:10.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:09 smithi115 bash[45495]: cluster 2022-04-22T18:05:08.845631+0000 mgr.y (mgr.34107) 1383 : cluster [DBG] pgmap v1953: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:05:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:09 smithi115 bash[46934]: cluster 2022-04-22T18:05:08.845631+0000 mgr.y (mgr.34107) 1383 : cluster [DBG] pgmap v1953: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 4 op/s 2022-04-22T18:05:10.592 INFO:tasks.workunit.client.0.smithi115.stdout: 1080: throughput=5MB/sec pending data=0 2022-04-22T18:05:10.664 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NPNTRi1IHbkr2Rv off=143332 len=1193041 2022-04-22T18:05:10.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 5507 completed, throughput=5MB/sec 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3NdzyYbrOMHNhoe off=4555269 len=656899 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 5508 completed, throughput=5MB/sec 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gy0Nikg7mjfDOMC off=217035 len=465450 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 5509 completed, throughput=5MB/sec 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-K9hyqBTriFP4mah off=17448592 len=1251054 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 5510 completed, throughput=5MB/sec 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sh_k73oGFIWRGpE off=6574208 len=440799 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 5511 completed, throughput=5MB/sec 2022-04-22T18:05:10.666 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8GZ7b64ocG4jaUT off=5339021 len=694460 2022-04-22T18:05:10.667 INFO:tasks.workunit.client.0.smithi115.stdout:op 5512 completed, throughput=5MB/sec 2022-04-22T18:05:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:10 smithi121 bash[36902]: audit 2022-04-22T18:05:10.444599+0000 mon.a (mon.0) 2933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:11.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:10 smithi121 bash[36902]: audit 2022-04-22T18:05:10.850145+0000 mon.a (mon.0) 2934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:11.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:10 smithi115 bash[45495]: audit 2022-04-22T18:05:10.444599+0000 mon.a (mon.0) 2933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:11.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:10 smithi115 bash[45495]: audit 2022-04-22T18:05:10.850145+0000 mon.a (mon.0) 2934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:10 smithi115 bash[46934]: audit 2022-04-22T18:05:10.444599+0000 mon.a (mon.0) 2933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:10 smithi115 bash[46934]: audit 2022-04-22T18:05:10.850145+0000 mon.a (mon.0) 2934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:11.617 INFO:tasks.workunit.client.0.smithi115.stdout: 1081: throughput=5MB/sec pending data=0 2022-04-22T18:05:11.621 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-t7v57Efewl1ckCE off=6784522 len=1553804 2022-04-22T18:05:11.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 5513 completed, throughput=5MB/sec 2022-04-22T18:05:11.623 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9doPfq3A2SWk_Ly off=7210098 len=394384 2022-04-22T18:05:11.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 5514 completed, throughput=5MB/sec 2022-04-22T18:05:11.629 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-a0jki5Xs-eeJJJv off=12481679 len=1952533 2022-04-22T18:05:11.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 5515 completed, throughput=5MB/sec 2022-04-22T18:05:11.634 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9ic6geb6OUMLb_W off=5221804 len=1359019 2022-04-22T18:05:11.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 5516 completed, throughput=5MB/sec 2022-04-22T18:05:12.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:11 smithi115 bash[45495]: cluster 2022-04-22T18:05:10.846759+0000 mgr.y (mgr.34107) 1384 : cluster [DBG] pgmap v1954: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:12.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:11 smithi115 bash[46934]: cluster 2022-04-22T18:05:10.846759+0000 mgr.y (mgr.34107) 1384 : cluster [DBG] pgmap v1954: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:11.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:11 smithi121 bash[36902]: cluster 2022-04-22T18:05:10.846759+0000 mgr.y (mgr.34107) 1384 : cluster [DBG] pgmap v1954: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:12.642 INFO:tasks.workunit.client.0.smithi115.stdout: 1082: throughput=5MB/sec pending data=0 2022-04-22T18:05:12.642 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5PksEJXLoftipFc off=1935379 len=934141 2022-04-22T18:05:12.650 INFO:tasks.workunit.client.0.smithi115.stdout:op 5517 completed, throughput=5MB/sec 2022-04-22T18:05:12.650 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2HisR9MLBVUiSj2 off=6680309 len=877907 2022-04-22T18:05:12.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 5518 completed, throughput=5MB/sec 2022-04-22T18:05:12.652 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Zo61HSfCTa26zVm off=991106 len=535103 2022-04-22T18:05:12.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 5519 completed, throughput=5MB/sec 2022-04-22T18:05:12.655 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3wffABXMzRMmccB off=3024837 len=637678 2022-04-22T18:05:12.656 INFO:tasks.workunit.client.0.smithi115.stdout:op 5520 completed, throughput=5MB/sec 2022-04-22T18:05:12.656 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fG8KlaLCSMqNkFF off=6019740 len=614767 2022-04-22T18:05:12.661 INFO:tasks.workunit.client.0.smithi115.stdout:op 5521 completed, throughput=5MB/sec 2022-04-22T18:05:12.661 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-sP4oEp3nO7ipr61 off=4511360 len=1912822 2022-04-22T18:05:12.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 5522 completed, throughput=5MB/sec 2022-04-22T18:05:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:12 smithi121 bash[36902]: audit 2022-04-22T18:05:12.836121+0000 mon.a (mon.0) 2935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:13.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:12 smithi115 bash[45495]: audit 2022-04-22T18:05:12.836121+0000 mon.a (mon.0) 2935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:13.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:12 smithi115 bash[46934]: audit 2022-04-22T18:05:12.836121+0000 mon.a (mon.0) 2935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:13.665 INFO:tasks.workunit.client.0.smithi115.stdout: 1083: throughput=5MB/sec pending data=0 2022-04-22T18:05:13.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UQo0lAhSAw8-Fux off=2645300 len=553221 2022-04-22T18:05:13.668 INFO:tasks.workunit.client.0.smithi115.stdout:op 5523 completed, throughput=5MB/sec 2022-04-22T18:05:13.668 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tiSDSawuG0qGjOI off=6554069 len=2016654 2022-04-22T18:05:13.679 INFO:tasks.workunit.client.0.smithi115.stdout:op 5524 completed, throughput=5MB/sec 2022-04-22T18:05:13.679 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6SDNdYrCzDpFLjR off=5597709 len=84202 2022-04-22T18:05:13.680 INFO:tasks.workunit.client.0.smithi115.stdout:op 5525 completed, throughput=5MB/sec 2022-04-22T18:05:13.680 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Guld7jCBi-d9bHk off=1345717 len=1107192 2022-04-22T18:05:13.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 5526 completed, throughput=5MB/sec 2022-04-22T18:05:13.683 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P_BA8yZrPJL5Ay4 off=3474802 len=612106 2022-04-22T18:05:13.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 5527 completed, throughput=5MB/sec 2022-04-22T18:05:13.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-w8Ul7dat6o_Q5tn off=130303 len=1865589 2022-04-22T18:05:13.694 INFO:tasks.workunit.client.0.smithi115.stdout:op 5528 completed, throughput=5MB/sec 2022-04-22T18:05:13.952 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:13.624Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:13 smithi121 bash[36902]: cluster 2022-04-22T18:05:12.847872+0000 mgr.y (mgr.34107) 1385 : cluster [DBG] pgmap v1955: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:13 smithi121 bash[36902]: audit 2022-04-22T18:05:13.645916+0000 mon.a (mon.0) 2936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:05:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:13 smithi121 bash[36902]: audit 2022-04-22T18:05:13.646436+0000 mon.a (mon.0) 2937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:05:14.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[45495]: cluster 2022-04-22T18:05:12.847872+0000 mgr.y (mgr.34107) 1385 : cluster [DBG] pgmap v1955: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:14.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[45495]: audit 2022-04-22T18:05:13.645916+0000 mon.a (mon.0) 2936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:05:14.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[45495]: audit 2022-04-22T18:05:13.646436+0000 mon.a (mon.0) 2937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:05:14.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[46934]: cluster 2022-04-22T18:05:12.847872+0000 mgr.y (mgr.34107) 1385 : cluster [DBG] pgmap v1955: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:05:14.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[46934]: audit 2022-04-22T18:05:13.645916+0000 mon.a (mon.0) 2936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:05:14.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:13 smithi115 bash[46934]: audit 2022-04-22T18:05:13.646436+0000 mon.a (mon.0) 2937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:05:14.694 INFO:tasks.workunit.client.0.smithi115.stdout: 1084: throughput=5MB/sec pending data=0 2022-04-22T18:05:14.694 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hbn1mTmaKHtpJOY off=18541279 len=2071954 2022-04-22T18:05:14.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 5529 completed, throughput=5MB/sec 2022-04-22T18:05:14.707 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9GodEky_yVqKRH_ off=14112032 len=1309193 2022-04-22T18:05:14.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 5530 completed, throughput=5MB/sec 2022-04-22T18:05:14.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ksdUaa8OrgQukGx off=0 len=1392086 2022-04-22T18:05:14.717 INFO:tasks.workunit.client.0.smithi115.stdout:op 5531 completed, throughput=5MB/sec 2022-04-22T18:05:15.718 INFO:tasks.workunit.client.0.smithi115.stdout: 1085: throughput=5MB/sec pending data=0 2022-04-22T18:05:15.718 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-v7JeefhjAeUxR7P off=9462221 len=2074402 2022-04-22T18:05:15.728 INFO:tasks.workunit.client.0.smithi115.stdout:op 5532 completed, throughput=5MB/sec 2022-04-22T18:05:15.728 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CoN2wdOdnawqMx9 off=10490171 len=1879385 2022-04-22T18:05:15.736 INFO:tasks.workunit.client.0.smithi115.stdout:op 5533 completed, throughput=5MB/sec 2022-04-22T18:05:15.736 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hr9SuLJ_1-1knAV off=2783290 len=477583 2022-04-22T18:05:15.740 INFO:tasks.workunit.client.0.smithi115.stdout:op 5534 completed, throughput=5MB/sec 2022-04-22T18:05:15.741 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fj78frV1aRf-M7H off=11548890 len=679816 2022-04-22T18:05:15.745 INFO:tasks.workunit.client.0.smithi115.stdout:op 5535 completed, throughput=5MB/sec 2022-04-22T18:05:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:15 smithi121 bash[36902]: cluster 2022-04-22T18:05:14.849092+0000 mgr.y (mgr.34107) 1386 : cluster [DBG] pgmap v1956: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:15 smithi121 bash[36902]: audit 2022-04-22T18:05:15.447738+0000 mon.a (mon.0) 2938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:15 smithi121 bash[36902]: audit 2022-04-22T18:05:15.854082+0000 mon.a (mon.0) 2939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[46934]: cluster 2022-04-22T18:05:14.849092+0000 mgr.y (mgr.34107) 1386 : cluster [DBG] pgmap v1956: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[46934]: audit 2022-04-22T18:05:15.447738+0000 mon.a (mon.0) 2938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[46934]: audit 2022-04-22T18:05:15.854082+0000 mon.a (mon.0) 2939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[45495]: cluster 2022-04-22T18:05:14.849092+0000 mgr.y (mgr.34107) 1386 : cluster [DBG] pgmap v1956: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[45495]: audit 2022-04-22T18:05:15.447738+0000 mon.a (mon.0) 2938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:15 smithi115 bash[45495]: audit 2022-04-22T18:05:15.854082+0000 mon.a (mon.0) 2939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:16.746 INFO:tasks.workunit.client.0.smithi115.stdout: 1086: throughput=5MB/sec pending data=0 2022-04-22T18:05:16.746 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GoMWWgYrBCrflaP off=3695448 len=619838 2022-04-22T18:05:16.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 5536 completed, throughput=5MB/sec 2022-04-22T18:05:16.749 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gdijSzFZWKgE5hD off=5598821 len=264540 2022-04-22T18:05:16.754 INFO:tasks.workunit.client.0.smithi115.stdout:op 5537 completed, throughput=5MB/sec 2022-04-22T18:05:16.754 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z09d6HJW1kFqVQS off=3697189 len=144085 2022-04-22T18:05:16.755 INFO:tasks.workunit.client.0.smithi115.stdout:op 5538 completed, throughput=5MB/sec 2022-04-22T18:05:16.756 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jLoJGLd-5D2Ayur off=10066884 len=1014837 2022-04-22T18:05:16.762 INFO:tasks.workunit.client.0.smithi115.stdout:op 5539 completed, throughput=5MB/sec 2022-04-22T18:05:16.762 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mlXkM_GsI80cCzo off=20931628 len=1014464 2022-04-22T18:05:16.767 INFO:tasks.workunit.client.0.smithi115.stdout:op 5540 completed, throughput=5MB/sec 2022-04-22T18:05:16.767 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-t6uYUaeJMyZC8Ac off=2306960 len=28435 2022-04-22T18:05:16.770 INFO:tasks.workunit.client.0.smithi115.stdout:op 5541 completed, throughput=5MB/sec 2022-04-22T18:05:16.770 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-r8B8O-7EY3xc1o9 off=12766893 len=343290 2022-04-22T18:05:16.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 5542 completed, throughput=5MB/sec 2022-04-22T18:05:16.774 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VxGyObd0IZxYCPx off=1808843 len=2061891 2022-04-22T18:05:16.777 INFO:tasks.workunit.client.0.smithi115.stdout:op 5543 completed, throughput=5MB/sec 2022-04-22T18:05:17.778 INFO:tasks.workunit.client.0.smithi115.stdout: 1087: throughput=5MB/sec pending data=0 2022-04-22T18:05:17.778 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-IJV4R7ZEz-q41Ta off=7249180 len=1897841 2022-04-22T18:05:17.788 INFO:tasks.workunit.client.0.smithi115.stdout:op 5544 completed, throughput=5MB/sec 2022-04-22T18:05:17.789 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lJoNUWxpv-_2ZWy off=8396117 len=131869 2022-04-22T18:05:17.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 5545 completed, throughput=5MB/sec 2022-04-22T18:05:17.790 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1VdPANn5TB0Peoh off=1424803 len=55564 2022-04-22T18:05:17.791 INFO:tasks.workunit.client.0.smithi115.stdout:op 5546 completed, throughput=5MB/sec 2022-04-22T18:05:17.791 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-D8uCFF3vJ3BQ6-w off=20274083 len=1599924 2022-04-22T18:05:17.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 5547 completed, throughput=5MB/sec 2022-04-22T18:05:17.796 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Y8Axo1VtVBXvHop off=0 len=1327993 2022-04-22T18:05:17.802 INFO:tasks.workunit.client.0.smithi115.stdout:op 5548 completed, throughput=5MB/sec 2022-04-22T18:05:17.803 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7kWWZmsS0twajnC off=644715 len=44747 2022-04-22T18:05:17.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 5549 completed, throughput=5MB/sec 2022-04-22T18:05:17.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6tVf1tO_7XPcpVZ off=2049604 len=1508363 2022-04-22T18:05:17.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 5550 completed, throughput=5MB/sec 2022-04-22T18:05:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:17 smithi121 bash[36902]: cluster 2022-04-22T18:05:16.850339+0000 mgr.y (mgr.34107) 1387 : cluster [DBG] pgmap v1957: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:05:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:17 smithi121 bash[36902]: audit 2022-04-22T18:05:17.839441+0000 mon.a (mon.0) 2940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:17 smithi115 bash[45495]: cluster 2022-04-22T18:05:16.850339+0000 mgr.y (mgr.34107) 1387 : cluster [DBG] pgmap v1957: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:05:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:17 smithi115 bash[45495]: audit 2022-04-22T18:05:17.839441+0000 mon.a (mon.0) 2940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:17 smithi115 bash[46934]: cluster 2022-04-22T18:05:16.850339+0000 mgr.y (mgr.34107) 1387 : cluster [DBG] pgmap v1957: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:05:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:17 smithi115 bash[46934]: audit 2022-04-22T18:05:17.839441+0000 mon.a (mon.0) 2940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:18.811 INFO:tasks.workunit.client.0.smithi115.stdout: 1088: throughput=5MB/sec pending data=0 2022-04-22T18:05:18.812 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dzPn7Nh00m-5S2J off=4924424 len=601763 2022-04-22T18:05:18.818 INFO:tasks.workunit.client.0.smithi115.stdout:op 5551 completed, throughput=5MB/sec 2022-04-22T18:05:18.818 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V-f3Kkzltp7H3BE off=6110416 len=210727 2022-04-22T18:05:18.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 5552 completed, throughput=5MB/sec 2022-04-22T18:05:18.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HAly6H-wlLYlVph off=15740109 len=567388 2022-04-22T18:05:18.824 INFO:tasks.workunit.client.0.smithi115.stdout:op 5553 completed, throughput=5MB/sec 2022-04-22T18:05:18.824 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-X5kWRlVzxIHr1Zn off=8008246 len=341959 2022-04-22T18:05:18.829 INFO:tasks.workunit.client.0.smithi115.stdout:op 5554 completed, throughput=5MB/sec 2022-04-22T18:05:18.829 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5yYU99aNIsPOVBr off=1098835 len=1160681 2022-04-22T18:05:18.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 5555 completed, throughput=5MB/sec 2022-04-22T18:05:18.836 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ozg-UUL53JhAnze off=2500383 len=391763 2022-04-22T18:05:18.840 INFO:tasks.workunit.client.0.smithi115.stdout:op 5556 completed, throughput=5MB/sec 2022-04-22T18:05:18.840 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-StPQ5Kkhjcsubpt off=8970668 len=1361957 2022-04-22T18:05:18.842 INFO:tasks.workunit.client.0.smithi115.stdout:op 5557 completed, throughput=5MB/sec 2022-04-22T18:05:19.843 INFO:tasks.workunit.client.0.smithi115.stdout: 1089: throughput=5MB/sec pending data=0 2022-04-22T18:05:19.843 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rC4aSWfrOOOZgB2 off=3718381 len=423468 2022-04-22T18:05:19.845 INFO:tasks.workunit.client.0.smithi115.stdout:op 5558 completed, throughput=5MB/sec 2022-04-22T18:05:19.845 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LUEJLwFPnu7PGzI off=2290423 len=1331222 2022-04-22T18:05:19.853 INFO:tasks.workunit.client.0.smithi115.stdout:op 5559 completed, throughput=5MB/sec 2022-04-22T18:05:19.853 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s6d2R5CQSfJfv8f off=14048412 len=1023806 2022-04-22T18:05:19.860 INFO:tasks.workunit.client.0.smithi115.stdout:op 5560 completed, throughput=5MB/sec 2022-04-22T18:05:19.860 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tyHwD-dOqMknje1 off=16600822 len=953193 2022-04-22T18:05:19.863 INFO:tasks.workunit.client.0.smithi115.stdout:op 5561 completed, throughput=5MB/sec 2022-04-22T18:05:19.863 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-53wsAT953LmcjMB off=4340221 len=906153 2022-04-22T18:05:19.865 INFO:tasks.workunit.client.0.smithi115.stdout:op 5562 completed, throughput=5MB/sec 2022-04-22T18:05:19.865 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uDr5F-uKUiHhG0B off=307625 len=1736944 2022-04-22T18:05:19.874 INFO:tasks.workunit.client.0.smithi115.stdout:op 5563 completed, throughput=5MB/sec 2022-04-22T18:05:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:19 smithi121 bash[36902]: cluster 2022-04-22T18:05:18.851524+0000 mgr.y (mgr.34107) 1388 : cluster [DBG] pgmap v1958: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:20.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:19 smithi115 bash[45495]: cluster 2022-04-22T18:05:18.851524+0000 mgr.y (mgr.34107) 1388 : cluster [DBG] pgmap v1958: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:20.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:19 smithi115 bash[46934]: cluster 2022-04-22T18:05:18.851524+0000 mgr.y (mgr.34107) 1388 : cluster [DBG] pgmap v1958: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:05:20.874 INFO:tasks.workunit.client.0.smithi115.stdout: 1090: throughput=5MB/sec pending data=0 2022-04-22T18:05:20.875 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-epagiWqHY3vfnC6 off=3844097 len=162630 2022-04-22T18:05:20.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 5564 completed, throughput=5MB/sec 2022-04-22T18:05:20.876 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q2T5Sfg-vnms3Ph off=11412336 len=1704568 2022-04-22T18:05:20.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 5565 completed, throughput=5MB/sec 2022-04-22T18:05:20.887 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VFaNa-ea9OChYwS off=2042338 len=46976 2022-04-22T18:05:20.888 INFO:tasks.workunit.client.0.smithi115.stdout:op 5566 completed, throughput=5MB/sec 2022-04-22T18:05:20.888 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GAtGXrEbRzCGBxq off=4793545 len=797301 2022-04-22T18:05:20.893 INFO:tasks.workunit.client.0.smithi115.stdout:op 5567 completed, throughput=5MB/sec 2022-04-22T18:05:20.893 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iQlSYj91vUV5dEj off=4919990 len=702226 2022-04-22T18:05:20.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 5568 completed, throughput=5MB/sec 2022-04-22T18:05:20.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GiPvUqwr1jwM5Wb off=678911 len=1729584 2022-04-22T18:05:20.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 5569 completed, throughput=5MB/sec 2022-04-22T18:05:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:20 smithi121 bash[36902]: audit 2022-04-22T18:05:20.450670+0000 mon.a (mon.0) 2941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:20 smithi121 bash[36902]: audit 2022-04-22T18:05:20.857844+0000 mon.a (mon.0) 2942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:21.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:20 smithi115 bash[45495]: audit 2022-04-22T18:05:20.450670+0000 mon.a (mon.0) 2941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:21.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:20 smithi115 bash[45495]: audit 2022-04-22T18:05:20.857844+0000 mon.a (mon.0) 2942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:20 smithi115 bash[46934]: audit 2022-04-22T18:05:20.450670+0000 mon.a (mon.0) 2941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:20 smithi115 bash[46934]: audit 2022-04-22T18:05:20.857844+0000 mon.a (mon.0) 2942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:21.902 INFO:tasks.workunit.client.0.smithi115.stdout: 1091: throughput=5MB/sec pending data=0 2022-04-22T18:05:21.902 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h_ErsIRhOUNT8jW off=779932 len=1916646 2022-04-22T18:05:21.906 INFO:tasks.workunit.client.0.smithi115.stdout:op 5570 completed, throughput=5MB/sec 2022-04-22T18:05:21.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qEdK8OucdEvBApj off=7528349 len=425198 2022-04-22T18:05:21.909 INFO:tasks.workunit.client.0.smithi115.stdout:op 5571 completed, throughput=5MB/sec 2022-04-22T18:05:21.909 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7Zf4W4AMdhkQ-bl off=19496914 len=1640233 2022-04-22T18:05:21.912 INFO:tasks.workunit.client.0.smithi115.stdout:op 5572 completed, throughput=5MB/sec 2022-04-22T18:05:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:21 smithi115 bash[45495]: cluster 2022-04-22T18:05:20.852580+0000 mgr.y (mgr.34107) 1389 : cluster [DBG] pgmap v1959: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:22.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:21 smithi115 bash[46934]: cluster 2022-04-22T18:05:20.852580+0000 mgr.y (mgr.34107) 1389 : cluster [DBG] pgmap v1959: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:21.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:21 smithi121 bash[36902]: cluster 2022-04-22T18:05:20.852580+0000 mgr.y (mgr.34107) 1389 : cluster [DBG] pgmap v1959: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:22.914 INFO:tasks.workunit.client.0.smithi115.stdout: 1092: throughput=5MB/sec pending data=0 2022-04-22T18:05:22.914 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Of3MY2R2uy1INEP off=1021554 len=1821943 2022-04-22T18:05:22.922 INFO:tasks.workunit.client.0.smithi115.stdout:op 5573 completed, throughput=5MB/sec 2022-04-22T18:05:22.922 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O2vN3nJdOxjg-8W off=2227064 len=363518 2022-04-22T18:05:22.927 INFO:tasks.workunit.client.0.smithi115.stdout:op 5574 completed, throughput=5MB/sec 2022-04-22T18:05:22.927 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7Aj69pbl4dmXajh off=20214994 len=873718 2022-04-22T18:05:22.928 INFO:tasks.workunit.client.0.smithi115.stdout:op 5575 completed, throughput=5MB/sec 2022-04-22T18:05:22.929 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5kAsyl0IUzrbNGY off=23847693 len=935980 2022-04-22T18:05:22.933 INFO:tasks.workunit.client.0.smithi115.stdout:op 5576 completed, throughput=5MB/sec 2022-04-22T18:05:22.934 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-br5EyGNjZeXoLtY off=14817609 len=1583958 2022-04-22T18:05:22.939 INFO:tasks.workunit.client.0.smithi115.stdout:op 5577 completed, throughput=5MB/sec 2022-04-22T18:05:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:22 smithi121 bash[36902]: audit 2022-04-22T18:05:22.842470+0000 mon.a (mon.0) 2943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:22 smithi115 bash[45495]: audit 2022-04-22T18:05:22.842470+0000 mon.a (mon.0) 2943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:23.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:22 smithi115 bash[46934]: audit 2022-04-22T18:05:22.842470+0000 mon.a (mon.0) 2943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:23.939 INFO:tasks.workunit.client.0.smithi115.stdout: 1093: throughput=5MB/sec pending data=0 2022-04-22T18:05:23.939 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pIvlgmGaKefop21 off=19301752 len=1370698 2022-04-22T18:05:23.947 INFO:tasks.workunit.client.0.smithi115.stdout:op 5578 completed, throughput=5MB/sec 2022-04-22T18:05:23.947 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MEnO3I3zTgXPQaI off=5237205 len=2029367 2022-04-22T18:05:23.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 5579 completed, throughput=5MB/sec 2022-04-22T18:05:23.951 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-STWdOzKK2_uNuDG off=13827115 len=1967243 2022-04-22T18:05:23.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 5580 completed, throughput=5MB/sec 2022-04-22T18:05:23.979 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:23.624Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:23 smithi121 bash[36902]: cluster 2022-04-22T18:05:22.853621+0000 mgr.y (mgr.34107) 1390 : cluster [DBG] pgmap v1960: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:24.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:23 smithi115 bash[45495]: cluster 2022-04-22T18:05:22.853621+0000 mgr.y (mgr.34107) 1390 : cluster [DBG] pgmap v1960: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:24.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:23 smithi115 bash[46934]: cluster 2022-04-22T18:05:22.853621+0000 mgr.y (mgr.34107) 1390 : cluster [DBG] pgmap v1960: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:24.961 INFO:tasks.workunit.client.0.smithi115.stdout: 1094: throughput=5MB/sec pending data=0 2022-04-22T18:05:24.961 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7v1vCUUXasc_xuE off=9086380 len=820514 2022-04-22T18:05:24.965 INFO:tasks.workunit.client.0.smithi115.stdout:op 5581 completed, throughput=5MB/sec 2022-04-22T18:05:24.965 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VSRlFVndp2h9jTB off=20518084 len=788730 2022-04-22T18:05:24.966 INFO:tasks.workunit.client.0.smithi115.stdout:op 5582 completed, throughput=5MB/sec 2022-04-22T18:05:24.966 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Vjc3_zsXWXtAw4m off=2118271 len=1623787 2022-04-22T18:05:24.969 INFO:tasks.workunit.client.0.smithi115.stdout:op 5583 completed, throughput=5MB/sec 2022-04-22T18:05:24.969 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0aMq6ObBHz1z4xT off=1902534 len=703488 2022-04-22T18:05:24.974 INFO:tasks.workunit.client.0.smithi115.stdout:op 5584 completed, throughput=5MB/sec 2022-04-22T18:05:24.974 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TR5-MV4ZXLu9gzv off=4332631 len=709666 2022-04-22T18:05:24.979 INFO:tasks.workunit.client.0.smithi115.stdout:op 5585 completed, throughput=5MB/sec 2022-04-22T18:05:24.979 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aGGaCkCMyDXk6L6 off=5547214 len=1269378 2022-04-22T18:05:24.985 INFO:tasks.workunit.client.0.smithi115.stdout:op 5586 completed, throughput=5MB/sec 2022-04-22T18:05:25.985 INFO:tasks.workunit.client.0.smithi115.stdout: 1095: throughput=5MB/sec pending data=0 2022-04-22T18:05:25.985 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M-NY2749pY7GowH off=16572433 len=2024917 2022-04-22T18:05:25.998 INFO:tasks.workunit.client.0.smithi115.stdout:op 5587 completed, throughput=5MB/sec 2022-04-22T18:05:25.998 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zhGG2kjGfAVgWiU off=8212000 len=1473702 2022-04-22T18:05:26.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 5588 completed, throughput=5MB/sec 2022-04-22T18:05:26.001 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AC2DXija5-O0ced off=18418678 len=1117514 2022-04-22T18:05:26.007 INFO:tasks.workunit.client.0.smithi115.stdout:op 5589 completed, throughput=5MB/sec 2022-04-22T18:05:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:25 smithi121 bash[36902]: cluster 2022-04-22T18:05:24.854747+0000 mgr.y (mgr.34107) 1391 : cluster [DBG] pgmap v1961: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:25 smithi121 bash[36902]: audit 2022-04-22T18:05:25.453872+0000 mon.a (mon.0) 2944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:25 smithi121 bash[36902]: audit 2022-04-22T18:05:25.861315+0000 mon.a (mon.0) 2945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[45495]: cluster 2022-04-22T18:05:24.854747+0000 mgr.y (mgr.34107) 1391 : cluster [DBG] pgmap v1961: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[45495]: audit 2022-04-22T18:05:25.453872+0000 mon.a (mon.0) 2944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[45495]: audit 2022-04-22T18:05:25.861315+0000 mon.a (mon.0) 2945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[46934]: cluster 2022-04-22T18:05:24.854747+0000 mgr.y (mgr.34107) 1391 : cluster [DBG] pgmap v1961: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:05:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[46934]: audit 2022-04-22T18:05:25.453872+0000 mon.a (mon.0) 2944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:25 smithi115 bash[46934]: audit 2022-04-22T18:05:25.861315+0000 mon.a (mon.0) 2945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:27.008 INFO:tasks.workunit.client.0.smithi115.stdout: 1096: throughput=5MB/sec pending data=0 2022-04-22T18:05:27.008 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ro84uoC15ui1EdE off=914060 len=390712 2022-04-22T18:05:27.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 5590 completed, throughput=5MB/sec 2022-04-22T18:05:27.013 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_sB1LVMfhVxyQu8 off=291178 len=1985752 2022-04-22T18:05:27.024 INFO:tasks.workunit.client.0.smithi115.stdout:op 5591 completed, throughput=5MB/sec 2022-04-22T18:05:27.024 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-N5fvqhBEToda1Fs off=2617742 len=85444 2022-04-22T18:05:27.025 INFO:tasks.workunit.client.0.smithi115.stdout:op 5592 completed, throughput=5MB/sec 2022-04-22T18:05:27.025 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-61b7BQBH61J1M7d off=13370691 len=1400529 2022-04-22T18:05:27.032 INFO:tasks.workunit.client.0.smithi115.stdout:op 5593 completed, throughput=5MB/sec 2022-04-22T18:05:27.032 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-JmAIq3oubU6KgIE off=5373278 len=442523 2022-04-22T18:05:27.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 5594 completed, throughput=5MB/sec 2022-04-22T18:05:27.037 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BgnQ_JX-kHcveBT off=3221505 len=1548590 2022-04-22T18:05:27.039 INFO:tasks.workunit.client.0.smithi115.stdout:op 5595 completed, throughput=5MB/sec 2022-04-22T18:05:28.040 INFO:tasks.workunit.client.0.smithi115.stdout: 1097: throughput=5MB/sec pending data=0 2022-04-22T18:05:28.040 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rdXPUftV70Um_5d off=15869892 len=687152 2022-04-22T18:05:28.042 INFO:tasks.workunit.client.0.smithi115.stdout:op 5596 completed, throughput=5MB/sec 2022-04-22T18:05:28.042 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-izxT2EmwSVMk5A5 off=7084580 len=1063686 2022-04-22T18:05:28.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 5597 completed, throughput=5MB/sec 2022-04-22T18:05:28.050 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-p5IhLVVC9PP2cSt off=15112059 len=197879 2022-04-22T18:05:28.054 INFO:tasks.workunit.client.0.smithi115.stdout:op 5598 completed, throughput=5MB/sec 2022-04-22T18:05:28.055 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fzdfPeFTsQyhzDt off=1243459 len=94648 2022-04-22T18:05:28.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 5599 completed, throughput=5MB/sec 2022-04-22T18:05:28.058 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aoNNftndCEhN4ky off=1061917 len=633178 2022-04-22T18:05:28.063 INFO:tasks.workunit.client.0.smithi115.stdout:op 5600 completed, throughput=5MB/sec 2022-04-22T18:05:28.063 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Eh970rhikphvL2c off=1414103 len=876079 2022-04-22T18:05:28.068 INFO:tasks.workunit.client.0.smithi115.stdout:op 5601 completed, throughput=5MB/sec 2022-04-22T18:05:28.069 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-A5sl4REiVYH0PQe off=14360711 len=503340 2022-04-22T18:05:28.069 INFO:tasks.workunit.client.0.smithi115.stdout:op 5602 completed, throughput=5MB/sec 2022-04-22T18:05:28.070 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hviuPHM9r876jJx off=16159491 len=1241778 2022-04-22T18:05:28.075 INFO:tasks.workunit.client.0.smithi115.stdout:op 5603 completed, throughput=5MB/sec 2022-04-22T18:05:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:27 smithi121 bash[36902]: cluster 2022-04-22T18:05:26.855979+0000 mgr.y (mgr.34107) 1392 : cluster [DBG] pgmap v1962: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:05:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:27 smithi121 bash[36902]: audit 2022-04-22T18:05:27.845751+0000 mon.a (mon.0) 2946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:28.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:27 smithi115 bash[45495]: cluster 2022-04-22T18:05:26.855979+0000 mgr.y (mgr.34107) 1392 : cluster [DBG] pgmap v1962: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:05:28.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:27 smithi115 bash[45495]: audit 2022-04-22T18:05:27.845751+0000 mon.a (mon.0) 2946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:28.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:27 smithi115 bash[46934]: cluster 2022-04-22T18:05:26.855979+0000 mgr.y (mgr.34107) 1392 : cluster [DBG] pgmap v1962: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 5 op/s 2022-04-22T18:05:28.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:27 smithi115 bash[46934]: audit 2022-04-22T18:05:27.845751+0000 mon.a (mon.0) 2946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:29.075 INFO:tasks.workunit.client.0.smithi115.stdout: 1098: throughput=5MB/sec pending data=0 2022-04-22T18:05:29.075 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rcEbsJHToAAYv9t off=1354172 len=2004325 2022-04-22T18:05:29.087 INFO:tasks.workunit.client.0.smithi115.stdout:op 5604 completed, throughput=5MB/sec 2022-04-22T18:05:29.087 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-A_-6VhnADDU--CD off=647370 len=737730 2022-04-22T18:05:29.089 INFO:tasks.workunit.client.0.smithi115.stdout:op 5605 completed, throughput=5MB/sec 2022-04-22T18:05:29.089 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-hj7M5Uz9dbRcn80 off=7519645 len=1430403 2022-04-22T18:05:29.097 INFO:tasks.workunit.client.0.smithi115.stdout:op 5606 completed, throughput=5MB/sec 2022-04-22T18:05:29.097 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-6HQROF-QRbtZqN9 off=1577027 len=858889 2022-04-22T18:05:29.103 INFO:tasks.workunit.client.0.smithi115.stdout:op 5607 completed, throughput=5MB/sec 2022-04-22T18:05:30.103 INFO:tasks.workunit.client.0.smithi115.stdout: 1099: throughput=5MB/sec pending data=0 2022-04-22T18:05:30.103 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-he540X_bq8zWIJm off=9846340 len=907256 2022-04-22T18:05:30.110 INFO:tasks.workunit.client.0.smithi115.stdout:op 5608 completed, throughput=5MB/sec 2022-04-22T18:05:30.110 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-V_Yv86fyHyTl9lp off=13556716 len=179702 2022-04-22T18:05:30.112 INFO:tasks.workunit.client.0.smithi115.stdout:op 5609 completed, throughput=5MB/sec 2022-04-22T18:05:30.112 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KQNB_Po9OLD4Jg3 off=2072125 len=621809 2022-04-22T18:05:30.113 INFO:tasks.workunit.client.0.smithi115.stdout:op 5610 completed, throughput=5MB/sec 2022-04-22T18:05:30.114 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jf2vGj-JJ5xGNQY off=6019854 len=1553205 2022-04-22T18:05:30.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 5611 completed, throughput=5MB/sec 2022-04-22T18:05:30.117 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lHgnby7daNiSxsO off=5920951 len=1627459 2022-04-22T18:05:30.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 5612 completed, throughput=5MB/sec 2022-04-22T18:05:30.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj--zxWLhXH9Y96GuW off=18034153 len=1697869 2022-04-22T18:05:30.134 INFO:tasks.workunit.client.0.smithi115.stdout:op 5613 completed, throughput=5MB/sec 2022-04-22T18:05:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:29 smithi121 bash[36902]: cluster 2022-04-22T18:05:28.857134+0000 mgr.y (mgr.34107) 1393 : cluster [DBG] pgmap v1963: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:05:30.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:29 smithi115 bash[45495]: cluster 2022-04-22T18:05:28.857134+0000 mgr.y (mgr.34107) 1393 : cluster [DBG] pgmap v1963: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:05:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:29 smithi115 bash[46934]: cluster 2022-04-22T18:05:28.857134+0000 mgr.y (mgr.34107) 1393 : cluster [DBG] pgmap v1963: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:05:31.135 INFO:tasks.workunit.client.0.smithi115.stdout: 1100: throughput=5MB/sec pending data=0 2022-04-22T18:05:31.135 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9tkSsFi4sX557Xf off=2342359 len=783443 2022-04-22T18:05:31.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 5614 completed, throughput=5MB/sec 2022-04-22T18:05:31.137 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vQwAFV9YJTn1bkc off=8170898 len=1952529 2022-04-22T18:05:31.148 INFO:tasks.workunit.client.0.smithi115.stdout:op 5615 completed, throughput=5MB/sec 2022-04-22T18:05:31.149 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-VtRgXzzz7xAw8-p off=15823953 len=952806 2022-04-22T18:05:31.150 INFO:tasks.workunit.client.0.smithi115.stdout:op 5616 completed, throughput=5MB/sec 2022-04-22T18:05:31.150 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XV_jLrtfXvkHVCf off=4186208 len=1013301 2022-04-22T18:05:31.158 INFO:tasks.workunit.client.0.smithi115.stdout:op 5617 completed, throughput=5MB/sec 2022-04-22T18:05:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:30 smithi121 bash[36902]: audit 2022-04-22T18:05:30.457097+0000 mon.a (mon.0) 2947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:30 smithi121 bash[36902]: audit 2022-04-22T18:05:30.865010+0000 mon.a (mon.0) 2948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:31.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:30 smithi115 bash[45495]: audit 2022-04-22T18:05:30.457097+0000 mon.a (mon.0) 2947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:31.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:30 smithi115 bash[45495]: audit 2022-04-22T18:05:30.865010+0000 mon.a (mon.0) 2948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:31.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:30 smithi115 bash[46934]: audit 2022-04-22T18:05:30.457097+0000 mon.a (mon.0) 2947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:31.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:30 smithi115 bash[46934]: audit 2022-04-22T18:05:30.865010+0000 mon.a (mon.0) 2948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:32.000 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:31.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:32.158 INFO:tasks.workunit.client.0.smithi115.stdout: 1101: throughput=5MB/sec pending data=0 2022-04-22T18:05:32.159 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CGNTbOdzJcJRjQy off=7029594 len=540496 2022-04-22T18:05:32.161 INFO:tasks.workunit.client.0.smithi115.stdout:op 5618 completed, throughput=5MB/sec 2022-04-22T18:05:32.161 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4tpWtsekZvG9lm3 off=3595621 len=723157 2022-04-22T18:05:32.163 INFO:tasks.workunit.client.0.smithi115.stdout:op 5619 completed, throughput=5MB/sec 2022-04-22T18:05:32.163 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-WkhAkA1v_XiZGhO off=1270303 len=928026 2022-04-22T18:05:32.170 INFO:tasks.workunit.client.0.smithi115.stdout:op 5620 completed, throughput=5MB/sec 2022-04-22T18:05:32.170 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oro52Y0E_ktzox8 off=2988263 len=856544 2022-04-22T18:05:32.171 INFO:tasks.workunit.client.0.smithi115.stdout:op 5621 completed, throughput=5MB/sec 2022-04-22T18:05:32.171 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jIei-J-5Dk-r_E7 off=190208 len=1982762 2022-04-22T18:05:32.175 INFO:tasks.workunit.client.0.smithi115.stdout:op 5622 completed, throughput=5MB/sec 2022-04-22T18:05:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:31 smithi121 bash[36902]: cluster 2022-04-22T18:05:30.858168+0000 mgr.y (mgr.34107) 1394 : cluster [DBG] pgmap v1964: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:05:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:31 smithi115 bash[46934]: cluster 2022-04-22T18:05:30.858168+0000 mgr.y (mgr.34107) 1394 : cluster [DBG] pgmap v1964: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:05:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:31 smithi115 bash[45495]: cluster 2022-04-22T18:05:30.858168+0000 mgr.y (mgr.34107) 1394 : cluster [DBG] pgmap v1964: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 4 op/s 2022-04-22T18:05:32.395 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:05:32.743 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (28m) 7m ago 42m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (42m) 8m ago 42m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (30m) 7m ago 43m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (30m) 8m ago 43m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (37m) 8m ago 47m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (38m) 7m ago 49m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (37m) 7m ago 50m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (36m) 8m ago 48m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (36m) 7m ago 48m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:05:32.744 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (29m) 7m ago 42m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (29m) 8m ago 42m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (35m) 7m ago 47m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (34m) 7m ago 46m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (34m) 7m ago 46m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (33m) 7m ago 45m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (33m) 8m ago 45m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (32m) 8m ago 44m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (31m) 8m ago 44m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (30m) 8m ago 43m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:05:32.745 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (29m) 8m ago 42m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:05:33.152 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:05:33.152 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:05:33.152 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:05:33.152 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:33.152 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:05:33.153 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:05:33.154 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:05:33.154 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:05:33.175 INFO:tasks.workunit.client.0.smithi115.stdout: 1102: throughput=5MB/sec pending data=0 2022-04-22T18:05:33.176 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-7ehWdd5n42Sk7NG off=3120426 len=632465 2022-04-22T18:05:33.182 INFO:tasks.workunit.client.0.smithi115.stdout:op 5623 completed, throughput=5MB/sec 2022-04-22T18:05:33.182 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-8KVTDtopmxigq3V off=4243957 len=367110 2022-04-22T18:05:33.187 INFO:tasks.workunit.client.0.smithi115.stdout:op 5624 completed, throughput=5MB/sec 2022-04-22T18:05:33.187 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2tL482yoX5kK-AZ off=234442 len=931218 2022-04-22T18:05:33.190 INFO:tasks.workunit.client.0.smithi115.stdout:op 5625 completed, throughput=5MB/sec 2022-04-22T18:05:33.190 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GxQfe83U2csasCL off=684035 len=1852129 2022-04-22T18:05:33.193 INFO:tasks.workunit.client.0.smithi115.stdout:op 5626 completed, throughput=5MB/sec 2022-04-22T18:05:33.193 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zFzE_Z62zlMvBAa off=1591268 len=1500333 2022-04-22T18:05:33.202 INFO:tasks.workunit.client.0.smithi115.stdout:op 5627 completed, throughput=5MB/sec 2022-04-22T18:05:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:33 smithi121 bash[36902]: audit 2022-04-22T18:05:32.372790+0000 mgr.y (mgr.34107) 1395 : audit [DBG] from='client.35859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:33 smithi121 bash[36902]: audit 2022-04-22T18:05:32.848730+0000 mon.a (mon.0) 2949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:33 smithi115 bash[45495]: audit 2022-04-22T18:05:32.372790+0000 mgr.y (mgr.34107) 1395 : audit [DBG] from='client.35859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:33 smithi115 bash[45495]: audit 2022-04-22T18:05:32.848730+0000 mon.a (mon.0) 2949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:33 smithi115 bash[46934]: audit 2022-04-22T18:05:32.372790+0000 mgr.y (mgr.34107) 1395 : audit [DBG] from='client.35859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:33 smithi115 bash[46934]: audit 2022-04-22T18:05:32.848730+0000 mon.a (mon.0) 2949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:34.011 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:33.623Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:34.202 INFO:tasks.workunit.client.0.smithi115.stdout: 1103: throughput=5MB/sec pending data=0 2022-04-22T18:05:34.202 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1G7fR8nfNAQEKjO off=1452554 len=1669945 2022-04-22T18:05:34.205 INFO:tasks.workunit.client.0.smithi115.stdout:op 5628 completed, throughput=5MB/sec 2022-04-22T18:05:34.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-71wASS2b_bgBsLJ off=12167761 len=1234915 2022-04-22T18:05:34.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 5629 completed, throughput=5MB/sec 2022-04-22T18:05:34.215 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n4TwLDwDRtbO4ks off=854135 len=876369 2022-04-22T18:05:34.222 INFO:tasks.workunit.client.0.smithi115.stdout:op 5630 completed, throughput=5MB/sec 2022-04-22T18:05:34.222 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-EkgXFI5p9ON8bZF off=8985691 len=1351232 2022-04-22T18:05:34.229 INFO:tasks.workunit.client.0.smithi115.stdout:op 5631 completed, throughput=5MB/sec 2022-04-22T18:05:34.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DMAbo_tFbd8DssG off=7325229 len=225662 2022-04-22T18:05:34.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:34 smithi121 bash[36902]: audit 2022-04-22T18:05:32.719235+0000 mgr.y (mgr.34107) 1396 : audit [DBG] from='client.35865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:34.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:34 smithi121 bash[36902]: cluster 2022-04-22T18:05:32.859119+0000 mgr.y (mgr.34107) 1397 : cluster [DBG] pgmap v1965: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:34.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:34 smithi121 bash[36902]: audit 2022-04-22T18:05:33.146002+0000 mon.a (mon.0) 2950 : audit [DBG] from='client.? 172.21.15.115:0/2467151932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:34.276 INFO:tasks.workunit.client.0.smithi115.stdout:op 5632 completed, throughput=5MB/sec 2022-04-22T18:05:34.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[45495]: audit 2022-04-22T18:05:32.719235+0000 mgr.y (mgr.34107) 1396 : audit [DBG] from='client.35865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:34.384 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[45495]: cluster 2022-04-22T18:05:32.859119+0000 mgr.y (mgr.34107) 1397 : cluster [DBG] pgmap v1965: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:34.387 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[45495]: audit 2022-04-22T18:05:33.146002+0000 mon.a (mon.0) 2950 : audit [DBG] from='client.? 172.21.15.115:0/2467151932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:34.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[46934]: audit 2022-04-22T18:05:32.719235+0000 mgr.y (mgr.34107) 1396 : audit [DBG] from='client.35865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:05:34.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[46934]: cluster 2022-04-22T18:05:32.859119+0000 mgr.y (mgr.34107) 1397 : cluster [DBG] pgmap v1965: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:05:34.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:34 smithi115 bash[46934]: audit 2022-04-22T18:05:33.146002+0000 mon.a (mon.0) 2950 : audit [DBG] from='client.? 172.21.15.115:0/2467151932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:05:35.231 INFO:tasks.workunit.client.0.smithi115.stdout: 1104: throughput=5MB/sec pending data=0 2022-04-22T18:05:35.231 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-9TH-RqKZhOwCU9v off=990417 len=940671 2022-04-22T18:05:35.234 INFO:tasks.workunit.client.0.smithi115.stdout:op 5633 completed, throughput=5MB/sec 2022-04-22T18:05:35.234 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WPMvEkQjlJe7-Hp off=1834296 len=410750 2022-04-22T18:05:35.235 INFO:tasks.workunit.client.0.smithi115.stdout:op 5634 completed, throughput=5MB/sec 2022-04-22T18:05:35.236 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-AXHwwmcumEBWQaU off=4496839 len=405650 2022-04-22T18:05:35.241 INFO:tasks.workunit.client.0.smithi115.stdout:op 5635 completed, throughput=5MB/sec 2022-04-22T18:05:35.241 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-M-jAiR-cRrBdOSc off=5762303 len=411014 2022-04-22T18:05:35.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 5636 completed, throughput=5MB/sec 2022-04-22T18:05:35.242 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Fn5lBoUf3H-c2Yv off=17613666 len=1622635 2022-04-22T18:05:35.250 INFO:tasks.workunit.client.0.smithi115.stdout:op 5637 completed, throughput=5MB/sec 2022-04-22T18:05:35.250 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-USIgkRbWcNFmWSK off=18792914 len=1778481 2022-04-22T18:05:35.255 INFO:tasks.workunit.client.0.smithi115.stdout:op 5638 completed, throughput=5MB/sec 2022-04-22T18:05:36.255 INFO:tasks.workunit.client.0.smithi115.stdout: 1105: throughput=5MB/sec pending data=0 2022-04-22T18:05:36.255 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mZAeaopToIVv7iS off=2693761 len=1730975 2022-04-22T18:05:36.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 5639 completed, throughput=5MB/sec 2022-04-22T18:05:36.259 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kxC1SFwE-AvTdkr off=7470773 len=580539 2022-04-22T18:05:36.266 INFO:tasks.workunit.client.0.smithi115.stdout:op 5640 completed, throughput=5MB/sec 2022-04-22T18:05:36.266 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rT5XmL0CxXt7f_i off=7223772 len=156805 2022-04-22T18:05:36.271 INFO:tasks.workunit.client.0.smithi115.stdout:op 5641 completed, throughput=5MB/sec 2022-04-22T18:05:36.271 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MCLuFap8oYyDEEj off=2672220 len=1294251 2022-04-22T18:05:36.274 INFO:tasks.workunit.client.0.smithi115.stdout:op 5642 completed, throughput=5MB/sec 2022-04-22T18:05:36.274 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cJRpkUhBMinVcku off=9559619 len=873417 2022-04-22T18:05:36.280 INFO:tasks.workunit.client.0.smithi115.stdout:op 5643 completed, throughput=5MB/sec 2022-04-22T18:05:36.280 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-uY9uMjgJSAHr4G6 off=10557196 len=209088 2022-04-22T18:05:36.283 INFO:tasks.workunit.client.0.smithi115.stdout:op 5644 completed, throughput=5MB/sec 2022-04-22T18:05:36.284 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2TjU6D6cPBwyl-l off=14134136 len=1285465 2022-04-22T18:05:36.285 INFO:tasks.workunit.client.0.smithi115.stdout:op 5645 completed, throughput=5MB/sec 2022-04-22T18:05:36.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[45495]: cluster 2022-04-22T18:05:34.860236+0000 mgr.y (mgr.34107) 1398 : cluster [DBG] pgmap v1966: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:05:36.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[45495]: audit 2022-04-22T18:05:35.460354+0000 mon.a (mon.0) 2951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:36.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[45495]: audit 2022-04-22T18:05:35.868794+0000 mon.a (mon.0) 2952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:36.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[46934]: cluster 2022-04-22T18:05:34.860236+0000 mgr.y (mgr.34107) 1398 : cluster [DBG] pgmap v1966: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:05:36.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[46934]: audit 2022-04-22T18:05:35.460354+0000 mon.a (mon.0) 2951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:36.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:36 smithi115 bash[46934]: audit 2022-04-22T18:05:35.868794+0000 mon.a (mon.0) 2952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:36 smithi121 bash[36902]: cluster 2022-04-22T18:05:34.860236+0000 mgr.y (mgr.34107) 1398 : cluster [DBG] pgmap v1966: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 4 op/s 2022-04-22T18:05:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:36 smithi121 bash[36902]: audit 2022-04-22T18:05:35.460354+0000 mon.a (mon.0) 2951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:36 smithi121 bash[36902]: audit 2022-04-22T18:05:35.868794+0000 mon.a (mon.0) 2952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:37.286 INFO:tasks.workunit.client.0.smithi115.stdout: 1106: throughput=5MB/sec pending data=0 2022-04-22T18:05:37.286 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DyEAfETmSqhlxqd off=0 len=1168022 2022-04-22T18:05:37.294 INFO:tasks.workunit.client.0.smithi115.stdout:op 5646 completed, throughput=5MB/sec 2022-04-22T18:05:37.294 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-B4Tidv8nWc-5Afo off=5553440 len=1633663 2022-04-22T18:05:37.303 INFO:tasks.workunit.client.0.smithi115.stdout:op 5647 completed, throughput=5MB/sec 2022-04-22T18:05:37.304 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tyLAuKfBbt7MfNI off=2524261 len=2013108 2022-04-22T18:05:37.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 5648 completed, throughput=5MB/sec 2022-04-22T18:05:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:38 smithi115 bash[45495]: cluster 2022-04-22T18:05:36.861334+0000 mgr.y (mgr.34107) 1399 : cluster [DBG] pgmap v1967: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T18:05:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:38 smithi115 bash[45495]: audit 2022-04-22T18:05:37.851936+0000 mon.a (mon.0) 2953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:38 smithi115 bash[46934]: cluster 2022-04-22T18:05:36.861334+0000 mgr.y (mgr.34107) 1399 : cluster [DBG] pgmap v1967: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T18:05:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:38 smithi115 bash[46934]: audit 2022-04-22T18:05:37.851936+0000 mon.a (mon.0) 2953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:38.306 INFO:tasks.workunit.client.0.smithi115.stdout: 1107: throughput=5MB/sec pending data=0 2022-04-22T18:05:38.306 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oQ-Lna7QpE1Orwd off=4323796 len=151555 2022-04-22T18:05:38.307 INFO:tasks.workunit.client.0.smithi115.stdout:op 5649 completed, throughput=5MB/sec 2022-04-22T18:05:38.307 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-wmPWZvBPszeW73i off=5244603 len=1237652 2022-04-22T18:05:38.310 INFO:tasks.workunit.client.0.smithi115.stdout:op 5650 completed, throughput=5MB/sec 2022-04-22T18:05:38.310 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TfOLzkwD1Cz6KmL off=4806102 len=108752 2022-04-22T18:05:38.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 5651 completed, throughput=5MB/sec 2022-04-22T18:05:38.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Aw_SQPPWegvFW_j off=2502947 len=1179131 2022-04-22T18:05:38.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 5652 completed, throughput=5MB/sec 2022-04-22T18:05:38.322 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-NQyPC4L2sZ2AW4q off=888114 len=1812068 2022-04-22T18:05:38.331 INFO:tasks.workunit.client.0.smithi115.stdout:op 5653 completed, throughput=5MB/sec 2022-04-22T18:05:38.331 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2yotkxNOLefs7_1 off=22318997 len=161223 2022-04-22T18:05:38.335 INFO:tasks.workunit.client.0.smithi115.stdout:op 5654 completed, throughput=5MB/sec 2022-04-22T18:05:38.335 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rA1V7IWMrxNkr4R off=5601702 len=1204620 2022-04-22T18:05:38.338 INFO:tasks.workunit.client.0.smithi115.stdout:op 5655 completed, throughput=5MB/sec 2022-04-22T18:05:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:38 smithi121 bash[36902]: cluster 2022-04-22T18:05:36.861334+0000 mgr.y (mgr.34107) 1399 : cluster [DBG] pgmap v1967: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 5 op/s 2022-04-22T18:05:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:38 smithi121 bash[36902]: audit 2022-04-22T18:05:37.851936+0000 mon.a (mon.0) 2953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:39.339 INFO:tasks.workunit.client.0.smithi115.stdout: 1108: throughput=5MB/sec pending data=0 2022-04-22T18:05:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RhsT3ML_P4xfd4u off=2807428 len=712080 2022-04-22T18:05:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:op 5656 completed, throughput=5MB/sec 2022-04-22T18:05:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NCyiqZPIR1z2aWj off=13193070 len=1078319 2022-04-22T18:05:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:op 5657 completed, throughput=5MB/sec 2022-04-22T18:05:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rlbZ7rDTmVzUkYi off=4400435 len=165387 2022-04-22T18:05:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:op 5658 completed, throughput=5MB/sec 2022-04-22T18:05:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cVQ_YqO-wUl9HZe off=7781432 len=78256 2022-04-22T18:05:39.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 5659 completed, throughput=5MB/sec 2022-04-22T18:05:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FyCOFBcQqjZNRiS off=4034475 len=1867183 2022-04-22T18:05:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:op 5660 completed, throughput=5MB/sec 2022-04-22T18:05:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-G53o5FvP-KEdtzz off=2913553 len=1898714 2022-04-22T18:05:39.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 5661 completed, throughput=5MB/sec 2022-04-22T18:05:40.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:40 smithi115 bash[45495]: cluster 2022-04-22T18:05:38.862565+0000 mgr.y (mgr.34107) 1400 : cluster [DBG] pgmap v1968: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:40.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:40 smithi115 bash[46934]: cluster 2022-04-22T18:05:38.862565+0000 mgr.y (mgr.34107) 1400 : cluster [DBG] pgmap v1968: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:40.365 INFO:tasks.workunit.client.0.smithi115.stdout: 1109: throughput=5MB/sec pending data=0 2022-04-22T18:05:40.365 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-08xvhqK3V8UJzGC off=3709412 len=1288540 2022-04-22T18:05:40.374 INFO:tasks.workunit.client.0.smithi115.stdout:op 5662 completed, throughput=5MB/sec 2022-04-22T18:05:40.374 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dh_LlXUEYHxbvY7 off=10858178 len=810553 2022-04-22T18:05:40.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 5663 completed, throughput=5MB/sec 2022-04-22T18:05:40.376 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SRamw0c6hZjkjyu off=5733506 len=709815 2022-04-22T18:05:40.379 INFO:tasks.workunit.client.0.smithi115.stdout:op 5664 completed, throughput=5MB/sec 2022-04-22T18:05:40.379 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-o6ojgG-NJCMG586 off=1048461 len=740395 2022-04-22T18:05:40.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 5665 completed, throughput=5MB/sec 2022-04-22T18:05:40.381 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-na2NTqPeRIXzZqO off=15387760 len=1650556 2022-04-22T18:05:40.384 INFO:tasks.workunit.client.0.smithi115.stdout:op 5666 completed, throughput=5MB/sec 2022-04-22T18:05:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:40 smithi121 bash[36902]: cluster 2022-04-22T18:05:38.862565+0000 mgr.y (mgr.34107) 1400 : cluster [DBG] pgmap v1968: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:41 smithi115 bash[45495]: audit 2022-04-22T18:05:40.463594+0000 mon.a (mon.0) 2954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:41.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:41 smithi115 bash[45495]: audit 2022-04-22T18:05:40.872225+0000 mon.a (mon.0) 2955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:41 smithi115 bash[46934]: audit 2022-04-22T18:05:40.463594+0000 mon.a (mon.0) 2954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:41.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:41 smithi115 bash[46934]: audit 2022-04-22T18:05:40.872225+0000 mon.a (mon.0) 2955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:41.385 INFO:tasks.workunit.client.0.smithi115.stdout: 1110: throughput=5MB/sec pending data=0 2022-04-22T18:05:41.385 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wDc5XExGe_CReIr off=20014933 len=2076937 2022-04-22T18:05:41.394 INFO:tasks.workunit.client.0.smithi115.stdout:op 5667 completed, throughput=5MB/sec 2022-04-22T18:05:41.394 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-luGSaITc5tNuC0U off=8622935 len=912719 2022-04-22T18:05:41.396 INFO:tasks.workunit.client.0.smithi115.stdout:op 5668 completed, throughput=5MB/sec 2022-04-22T18:05:41.396 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-mv_Sez2Ehb6va_5 off=16417910 len=596456 2022-04-22T18:05:41.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 5669 completed, throughput=5MB/sec 2022-04-22T18:05:41.399 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bcgGe-8YdXAUQVl off=13016928 len=2042103 2022-04-22T18:05:41.400 INFO:tasks.workunit.client.0.smithi115.stdout:op 5670 completed, throughput=5MB/sec 2022-04-22T18:05:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:41 smithi121 bash[36902]: audit 2022-04-22T18:05:40.463594+0000 mon.a (mon.0) 2954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:41 smithi121 bash[36902]: audit 2022-04-22T18:05:40.872225+0000 mon.a (mon.0) 2955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:42.029 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:41.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:42.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:42 smithi115 bash[45495]: cluster 2022-04-22T18:05:40.863859+0000 mgr.y (mgr.34107) 1401 : cluster [DBG] pgmap v1969: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:42 smithi115 bash[46934]: cluster 2022-04-22T18:05:40.863859+0000 mgr.y (mgr.34107) 1401 : cluster [DBG] pgmap v1969: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:42.401 INFO:tasks.workunit.client.0.smithi115.stdout: 1111: throughput=5MB/sec pending data=0 2022-04-22T18:05:42.401 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Otrqxx2FASDsKAn off=3340420 len=1158125 2022-04-22T18:05:42.409 INFO:tasks.workunit.client.0.smithi115.stdout:op 5671 completed, throughput=5MB/sec 2022-04-22T18:05:42.409 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UG-4pfuv2P4t5m0 off=852672 len=1721848 2022-04-22T18:05:42.412 INFO:tasks.workunit.client.0.smithi115.stdout:op 5672 completed, throughput=5MB/sec 2022-04-22T18:05:42.413 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-domoWVd5c6yCAIX off=15644348 len=583816 2022-04-22T18:05:42.414 INFO:tasks.workunit.client.0.smithi115.stdout:op 5673 completed, throughput=5MB/sec 2022-04-22T18:05:42.415 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Ic_eUxlXca8BbF0 off=16222462 len=535697 2022-04-22T18:05:42.416 INFO:tasks.workunit.client.0.smithi115.stdout:op 5674 completed, throughput=5MB/sec 2022-04-22T18:05:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:42 smithi121 bash[36902]: cluster 2022-04-22T18:05:40.863859+0000 mgr.y (mgr.34107) 1401 : cluster [DBG] pgmap v1969: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:05:43.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:43 smithi115 bash[45495]: audit 2022-04-22T18:05:42.854876+0000 mon.a (mon.0) 2956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:43 smithi115 bash[46934]: audit 2022-04-22T18:05:42.854876+0000 mon.a (mon.0) 2956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:43.416 INFO:tasks.workunit.client.0.smithi115.stdout: 1112: throughput=5MB/sec pending data=0 2022-04-22T18:05:43.416 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6ZcfAuFFTfjEmhj off=9423999 len=377036 2022-04-22T18:05:43.419 INFO:tasks.workunit.client.0.smithi115.stdout:op 5675 completed, throughput=5MB/sec 2022-04-22T18:05:43.419 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-I2TEPqm-LVR0fBq off=138697 len=2074969 2022-04-22T18:05:43.430 INFO:tasks.workunit.client.0.smithi115.stdout:op 5676 completed, throughput=5MB/sec 2022-04-22T18:05:43.430 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SAYhbj79zbKeVYt off=5846758 len=2041811 2022-04-22T18:05:43.434 INFO:tasks.workunit.client.0.smithi115.stdout:op 5677 completed, throughput=5MB/sec 2022-04-22T18:05:43.434 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ATQEf_fOblfjrPe off=18168193 len=716236 2022-04-22T18:05:43.436 INFO:tasks.workunit.client.0.smithi115.stdout:op 5678 completed, throughput=5MB/sec 2022-04-22T18:05:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:43 smithi121 bash[36902]: audit 2022-04-22T18:05:42.854876+0000 mon.a (mon.0) 2956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:44.039 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:43.625Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:44.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:44 smithi115 bash[45495]: cluster 2022-04-22T18:05:42.864977+0000 mgr.y (mgr.34107) 1402 : cluster [DBG] pgmap v1970: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:44 smithi115 bash[46934]: cluster 2022-04-22T18:05:42.864977+0000 mgr.y (mgr.34107) 1402 : cluster [DBG] pgmap v1970: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:44.436 INFO:tasks.workunit.client.0.smithi115.stdout: 1113: throughput=5MB/sec pending data=0 2022-04-22T18:05:44.436 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SPo2lRpKNOwgW23 off=16651568 len=2065099 2022-04-22T18:05:44.445 INFO:tasks.workunit.client.0.smithi115.stdout:op 5679 completed, throughput=5MB/sec 2022-04-22T18:05:44.446 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BfTSy7_gyEBaCS2 off=6343122 len=645813 2022-04-22T18:05:44.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 5680 completed, throughput=5MB/sec 2022-04-22T18:05:44.447 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YzJA5Sh5Y0_LLUh off=3437963 len=847121 2022-04-22T18:05:44.453 INFO:tasks.workunit.client.0.smithi115.stdout:op 5681 completed, throughput=5MB/sec 2022-04-22T18:05:44.454 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-emxZySouI6-4jvd off=5099757 len=1534601 2022-04-22T18:05:44.455 INFO:tasks.workunit.client.0.smithi115.stdout:op 5682 completed, throughput=5MB/sec 2022-04-22T18:05:44.455 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KlXykJjfwbKOwB4 off=3221910 len=890218 2022-04-22T18:05:44.461 INFO:tasks.workunit.client.0.smithi115.stdout:op 5683 completed, throughput=5MB/sec 2022-04-22T18:05:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:44 smithi121 bash[36902]: cluster 2022-04-22T18:05:42.864977+0000 mgr.y (mgr.34107) 1402 : cluster [DBG] pgmap v1970: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:45.461 INFO:tasks.workunit.client.0.smithi115.stdout: 1114: throughput=5MB/sec pending data=0 2022-04-22T18:05:45.462 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tYompDGz5G8xI2c off=6164658 len=191743 2022-04-22T18:05:45.467 INFO:tasks.workunit.client.0.smithi115.stdout:op 5684 completed, throughput=5MB/sec 2022-04-22T18:05:45.467 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AUwLRQCA9ztJRlg off=4125106 len=367040 2022-04-22T18:05:45.468 INFO:tasks.workunit.client.0.smithi115.stdout:op 5685 completed, throughput=5MB/sec 2022-04-22T18:05:45.468 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-domoWVd5c6yCAIX off=6465507 len=1620680 2022-04-22T18:05:45.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 5686 completed, throughput=5MB/sec 2022-04-22T18:05:45.472 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-z0fjZWdIfwYwRQH off=2161562 len=1594106 2022-04-22T18:05:45.475 INFO:tasks.workunit.client.0.smithi115.stdout:op 5687 completed, throughput=5MB/sec 2022-04-22T18:05:45.475 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-3FPU0xkx7rxWfMo off=1188708 len=779366 2022-04-22T18:05:45.476 INFO:tasks.workunit.client.0.smithi115.stdout:op 5688 completed, throughput=5MB/sec 2022-04-22T18:05:45.476 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5qjH1Q47pe9Nt79 off=950962 len=1181184 2022-04-22T18:05:45.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 5689 completed, throughput=5MB/sec 2022-04-22T18:05:46.481 INFO:tasks.workunit.client.0.smithi115.stdout: 1115: throughput=5MB/sec pending data=0 2022-04-22T18:05:46.481 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jRiJXFjWdh_0VLH off=12000746 len=1035262 2022-04-22T18:05:46.489 INFO:tasks.workunit.client.0.smithi115.stdout:op 5690 completed, throughput=5MB/sec 2022-04-22T18:05:46.489 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-m_1YtaQ3iIoUz_N off=6965084 len=383762 2022-04-22T18:05:46.494 INFO:tasks.workunit.client.0.smithi115.stdout:op 5691 completed, throughput=5MB/sec 2022-04-22T18:05:46.494 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-LoJO7o-7CUsfk1m off=537067 len=1857426 2022-04-22T18:05:46.503 INFO:tasks.workunit.client.0.smithi115.stdout:op 5692 completed, throughput=5MB/sec 2022-04-22T18:05:46.503 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-haAbNWfk1-VIlc0 off=16816269 len=493497 2022-04-22T18:05:46.504 INFO:tasks.workunit.client.0.smithi115.stdout:op 5693 completed, throughput=5MB/sec 2022-04-22T18:05:46.504 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-XNuemQHN97X7tMK off=6544531 len=727517 2022-04-22T18:05:46.505 INFO:tasks.workunit.client.0.smithi115.stdout:op 5694 completed, throughput=5MB/sec 2022-04-22T18:05:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:46 smithi121 bash[36902]: cluster 2022-04-22T18:05:44.866114+0000 mgr.y (mgr.34107) 1403 : cluster [DBG] pgmap v1971: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:46 smithi121 bash[36902]: audit 2022-04-22T18:05:45.466663+0000 mon.a (mon.0) 2957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:46 smithi121 bash[36902]: audit 2022-04-22T18:05:45.875725+0000 mon.a (mon.0) 2958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[45495]: cluster 2022-04-22T18:05:44.866114+0000 mgr.y (mgr.34107) 1403 : cluster [DBG] pgmap v1971: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[45495]: audit 2022-04-22T18:05:45.466663+0000 mon.a (mon.0) 2957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[45495]: audit 2022-04-22T18:05:45.875725+0000 mon.a (mon.0) 2958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[46934]: cluster 2022-04-22T18:05:44.866114+0000 mgr.y (mgr.34107) 1403 : cluster [DBG] pgmap v1971: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[46934]: audit 2022-04-22T18:05:45.466663+0000 mon.a (mon.0) 2957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:46 smithi115 bash[46934]: audit 2022-04-22T18:05:45.875725+0000 mon.a (mon.0) 2958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:47.505 INFO:tasks.workunit.client.0.smithi115.stdout: 1116: throughput=5MB/sec pending data=0 2022-04-22T18:05:47.506 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-htZLrvEAtvuoca7 off=2730583 len=855647 2022-04-22T18:05:47.513 INFO:tasks.workunit.client.0.smithi115.stdout:op 5695 completed, throughput=5MB/sec 2022-04-22T18:05:47.513 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XUAL3D4reYmE3nW off=9609367 len=1283476 2022-04-22T18:05:47.521 INFO:tasks.workunit.client.0.smithi115.stdout:op 5696 completed, throughput=5MB/sec 2022-04-22T18:05:47.521 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_x6-rptK8vVAJKT off=5780372 len=994791 2022-04-22T18:05:47.523 INFO:tasks.workunit.client.0.smithi115.stdout:op 5697 completed, throughput=5MB/sec 2022-04-22T18:05:47.523 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DJtt49rkjGxNO2T off=9310463 len=171440 2022-04-22T18:05:47.524 INFO:tasks.workunit.client.0.smithi115.stdout:op 5698 completed, throughput=5MB/sec 2022-04-22T18:05:47.525 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fN65cjYdh9xRSOz off=13118433 len=1461923 2022-04-22T18:05:47.532 INFO:tasks.workunit.client.0.smithi115.stdout:op 5699 completed, throughput=5MB/sec 2022-04-22T18:05:47.532 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dEJQ4m29md-GCDi off=4763144 len=1096320 2022-04-22T18:05:47.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 5700 completed, throughput=5MB/sec 2022-04-22T18:05:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:48 smithi121 bash[36902]: cluster 2022-04-22T18:05:46.867267+0000 mgr.y (mgr.34107) 1404 : cluster [DBG] pgmap v1972: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:48 smithi121 bash[36902]: audit 2022-04-22T18:05:47.858366+0000 mon.a (mon.0) 2959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:48.540 INFO:tasks.workunit.client.0.smithi115.stdout: 1117: throughput=5MB/sec pending data=0 2022-04-22T18:05:48.540 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nH14QBSLMkR6vro off=485298 len=44948 2022-04-22T18:05:48.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 5701 completed, throughput=5MB/sec 2022-04-22T18:05:48.541 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hkKVWsPYFXkJ6a9 off=591598 len=1479278 2022-04-22T18:05:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:48 smithi115 bash[46934]: cluster 2022-04-22T18:05:46.867267+0000 mgr.y (mgr.34107) 1404 : cluster [DBG] pgmap v1972: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:48 smithi115 bash[46934]: audit 2022-04-22T18:05:47.858366+0000 mon.a (mon.0) 2959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:48 smithi115 bash[45495]: cluster 2022-04-22T18:05:46.867267+0000 mgr.y (mgr.34107) 1404 : cluster [DBG] pgmap v1972: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:48 smithi115 bash[45495]: audit 2022-04-22T18:05:47.858366+0000 mon.a (mon.0) 2959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:48.545 INFO:tasks.workunit.client.0.smithi115.stdout:op 5702 completed, throughput=5MB/sec 2022-04-22T18:05:48.545 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9CQnTDOUlz5p1H5 off=5580146 len=1267968 2022-04-22T18:05:48.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 5703 completed, throughput=5MB/sec 2022-04-22T18:05:48.554 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xTTL5kANAXin6F_ off=7200460 len=569506 2022-04-22T18:05:48.554 INFO:tasks.workunit.client.0.smithi115.stdout:op 5704 completed, throughput=5MB/sec 2022-04-22T18:05:48.555 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yDIM7aq3MxunO8I off=10309853 len=864683 2022-04-22T18:05:48.560 INFO:tasks.workunit.client.0.smithi115.stdout:op 5705 completed, throughput=5MB/sec 2022-04-22T18:05:48.561 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2_zNQxR2AFjt0nL off=0 len=1658004 2022-04-22T18:05:48.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 5706 completed, throughput=5MB/sec 2022-04-22T18:05:49.564 INFO:tasks.workunit.client.0.smithi115.stdout: 1118: throughput=5MB/sec pending data=0 2022-04-22T18:05:49.564 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mV8fDu8wWD7JpI0 off=1521672 len=257869 2022-04-22T18:05:49.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 5707 completed, throughput=5MB/sec 2022-04-22T18:05:49.570 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oqry-DIQlJBrlnl off=17023868 len=336789 2022-04-22T18:05:49.575 INFO:tasks.workunit.client.0.smithi115.stdout:op 5708 completed, throughput=5MB/sec 2022-04-22T18:05:49.575 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5tUi4RS8l04vagN off=3256032 len=625524 2022-04-22T18:05:49.577 INFO:tasks.workunit.client.0.smithi115.stdout:op 5709 completed, throughput=5MB/sec 2022-04-22T18:05:49.577 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tLozese3vhaCE47 off=5218 len=1788600 2022-04-22T18:05:49.585 INFO:tasks.workunit.client.0.smithi115.stdout:op 5710 completed, throughput=5MB/sec 2022-04-22T18:05:49.586 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iWbHuPvxnuq0Rj7 off=1555400 len=389645 2022-04-22T18:05:49.587 INFO:tasks.workunit.client.0.smithi115.stdout:op 5711 completed, throughput=5MB/sec 2022-04-22T18:05:49.587 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6p6N8E-27vgN2X2 off=2105420 len=1361495 2022-04-22T18:05:49.590 INFO:tasks.workunit.client.0.smithi115.stdout:op 5712 completed, throughput=5MB/sec 2022-04-22T18:05:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:50 smithi121 bash[36902]: cluster 2022-04-22T18:05:48.868341+0000 mgr.y (mgr.34107) 1405 : cluster [DBG] pgmap v1973: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:05:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:50 smithi115 bash[46934]: cluster 2022-04-22T18:05:48.868341+0000 mgr.y (mgr.34107) 1405 : cluster [DBG] pgmap v1973: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:05:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:50 smithi115 bash[45495]: cluster 2022-04-22T18:05:48.868341+0000 mgr.y (mgr.34107) 1405 : cluster [DBG] pgmap v1973: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 5 op/s 2022-04-22T18:05:50.591 INFO:tasks.workunit.client.0.smithi115.stdout: 1119: throughput=5MB/sec pending data=0 2022-04-22T18:05:50.613 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-d50WfOdEOM8otDK off=1182621 len=676501 2022-04-22T18:05:50.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 5713 completed, throughput=5MB/sec 2022-04-22T18:05:50.614 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_0q1MpKiPGaHrVq off=13003074 len=1566317 2022-04-22T18:05:50.614 INFO:tasks.workunit.client.0.smithi115.stdout:op 5714 completed, throughput=5MB/sec 2022-04-22T18:05:50.614 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-60Z7hLznviNi2sR off=2054242 len=1466070 2022-04-22T18:05:50.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 5715 completed, throughput=5MB/sec 2022-04-22T18:05:50.615 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RCeLr43fdj-fzme off=2094469 len=1545919 2022-04-22T18:05:50.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 5716 completed, throughput=5MB/sec 2022-04-22T18:05:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:51 smithi121 bash[36902]: audit 2022-04-22T18:05:50.469691+0000 mon.a (mon.0) 2960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:51 smithi121 bash[36902]: audit 2022-04-22T18:05:50.879232+0000 mon.a (mon.0) 2961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:51 smithi115 bash[46934]: audit 2022-04-22T18:05:50.469691+0000 mon.a (mon.0) 2960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:51 smithi115 bash[46934]: audit 2022-04-22T18:05:50.879232+0000 mon.a (mon.0) 2961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:51 smithi115 bash[45495]: audit 2022-04-22T18:05:50.469691+0000 mon.a (mon.0) 2960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:51 smithi115 bash[45495]: audit 2022-04-22T18:05:50.879232+0000 mon.a (mon.0) 2961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:51.616 INFO:tasks.workunit.client.0.smithi115.stdout: 1120: throughput=5MB/sec pending data=0 2022-04-22T18:05:51.617 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-N7XJv5AysUAqMPc off=5876836 len=267347 2022-04-22T18:05:51.618 INFO:tasks.workunit.client.0.smithi115.stdout:op 5717 completed, throughput=5MB/sec 2022-04-22T18:05:51.618 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-c664WdUeuWS5DEc off=5176582 len=313652 2022-04-22T18:05:51.623 INFO:tasks.workunit.client.0.smithi115.stdout:op 5718 completed, throughput=5MB/sec 2022-04-22T18:05:51.623 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W62Gp3P3EWzxyb3 off=1790974 len=113970 2022-04-22T18:05:51.628 INFO:tasks.workunit.client.0.smithi115.stdout:op 5719 completed, throughput=5MB/sec 2022-04-22T18:05:51.628 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xTzkzy90PUYM6XZ off=7303417 len=551985 2022-04-22T18:05:51.629 INFO:tasks.workunit.client.0.smithi115.stdout:op 5720 completed, throughput=5MB/sec 2022-04-22T18:05:51.630 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2iEdmg2X6Qt50gA off=2778782 len=192551 2022-04-22T18:05:51.633 INFO:tasks.workunit.client.0.smithi115.stdout:op 5721 completed, throughput=5MB/sec 2022-04-22T18:05:51.633 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BE-3QbCCqAPDOz9 off=14666805 len=1323662 2022-04-22T18:05:51.637 INFO:tasks.workunit.client.0.smithi115.stdout:op 5722 completed, throughput=5MB/sec 2022-04-22T18:05:51.638 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zo03TwvRggxrkiH off=14602550 len=910900 2022-04-22T18:05:51.639 INFO:tasks.workunit.client.0.smithi115.stdout:op 5723 completed, throughput=5MB/sec 2022-04-22T18:05:51.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ZdEVzztPoiafjFK off=3646713 len=1950158 2022-04-22T18:05:51.642 INFO:tasks.workunit.client.0.smithi115.stdout:op 5724 completed, throughput=5MB/sec 2022-04-22T18:05:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:51.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:52 smithi121 bash[36902]: cluster 2022-04-22T18:05:50.869385+0000 mgr.y (mgr.34107) 1406 : cluster [DBG] pgmap v1974: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:52.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:52 smithi115 bash[46934]: cluster 2022-04-22T18:05:50.869385+0000 mgr.y (mgr.34107) 1406 : cluster [DBG] pgmap v1974: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:52.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:52 smithi115 bash[45495]: cluster 2022-04-22T18:05:50.869385+0000 mgr.y (mgr.34107) 1406 : cluster [DBG] pgmap v1974: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:05:52.643 INFO:tasks.workunit.client.0.smithi115.stdout: 1121: throughput=5MB/sec pending data=0 2022-04-22T18:05:52.643 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zgS_e5uDD33kt6I off=10849351 len=915244 2022-04-22T18:05:52.649 INFO:tasks.workunit.client.0.smithi115.stdout:op 5725 completed, throughput=5MB/sec 2022-04-22T18:05:52.649 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0wu5D82tH6GizcY off=686514 len=482299 2022-04-22T18:05:52.654 INFO:tasks.workunit.client.0.smithi115.stdout:op 5726 completed, throughput=5MB/sec 2022-04-22T18:05:52.655 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3V7eHkctrPmomYB off=25728 len=612813 2022-04-22T18:05:52.659 INFO:tasks.workunit.client.0.smithi115.stdout:op 5727 completed, throughput=5MB/sec 2022-04-22T18:05:52.659 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-i1Kuhtkt_yA23L5 off=5934985 len=590213 2022-04-22T18:05:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 5728 completed, throughput=5MB/sec 2022-04-22T18:05:52.660 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eQCcYoCZW0GAm4G off=13664068 len=1304169 2022-04-22T18:05:52.665 INFO:tasks.workunit.client.0.smithi115.stdout:op 5729 completed, throughput=5MB/sec 2022-04-22T18:05:52.665 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-qYrpi4pEsgzsyJv off=2015667 len=1091167 2022-04-22T18:05:52.666 INFO:tasks.workunit.client.0.smithi115.stdout:op 5730 completed, throughput=5MB/sec 2022-04-22T18:05:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:53 smithi121 bash[36902]: audit 2022-04-22T18:05:52.505910+0000 mon.a (mon.0) 2962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:05:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:53 smithi121 bash[36902]: audit 2022-04-22T18:05:52.508878+0000 mon.a (mon.0) 2963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:05:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:53 smithi121 bash[36902]: audit 2022-04-22T18:05:52.510607+0000 mon.a (mon.0) 2964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:05:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:53 smithi121 bash[36902]: audit 2022-04-22T18:05:52.848519+0000 mon.a (mon.0) 2965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:05:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:53 smithi121 bash[36902]: audit 2022-04-22T18:05:52.861104+0000 mon.a (mon.0) 2966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:53.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[45495]: audit 2022-04-22T18:05:52.505910+0000 mon.a (mon.0) 2962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:05:53.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[45495]: audit 2022-04-22T18:05:52.508878+0000 mon.a (mon.0) 2963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:05:53.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[45495]: audit 2022-04-22T18:05:52.510607+0000 mon.a (mon.0) 2964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:05:53.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[45495]: audit 2022-04-22T18:05:52.848519+0000 mon.a (mon.0) 2965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:05:53.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[45495]: audit 2022-04-22T18:05:52.861104+0000 mon.a (mon.0) 2966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[46934]: audit 2022-04-22T18:05:52.505910+0000 mon.a (mon.0) 2962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:05:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[46934]: audit 2022-04-22T18:05:52.508878+0000 mon.a (mon.0) 2963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:05:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[46934]: audit 2022-04-22T18:05:52.510607+0000 mon.a (mon.0) 2964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:05:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[46934]: audit 2022-04-22T18:05:52.848519+0000 mon.a (mon.0) 2965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:05:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[46934]: audit 2022-04-22T18:05:52.861104+0000 mon.a (mon.0) 2966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:53.667 INFO:tasks.workunit.client.0.smithi115.stdout: 1122: throughput=5MB/sec pending data=0 2022-04-22T18:05:53.667 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BtOF5sUPw18NkYx off=14156114 len=1338727 2022-04-22T18:05:53.669 INFO:tasks.workunit.client.0.smithi115.stdout:op 5731 completed, throughput=5MB/sec 2022-04-22T18:05:53.669 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b567kkdK4tqSbNE off=871605 len=1028831 2022-04-22T18:05:53.672 INFO:tasks.workunit.client.0.smithi115.stdout:op 5732 completed, throughput=5MB/sec 2022-04-22T18:05:53.672 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0JaZKVtEN40aKuW off=11591783 len=938187 2022-04-22T18:05:53.674 INFO:tasks.workunit.client.0.smithi115.stdout:op 5733 completed, throughput=5MB/sec 2022-04-22T18:05:53.675 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9B8emFGDZ655vtI off=10797165 len=1605269 2022-04-22T18:05:53.683 INFO:tasks.workunit.client.0.smithi115.stdout:op 5734 completed, throughput=5MB/sec 2022-04-22T18:05:53.683 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-2it9giWeCfIxyGj off=13952294 len=769896 2022-04-22T18:05:53.685 INFO:tasks.workunit.client.0.smithi115.stdout:op 5735 completed, throughput=5MB/sec 2022-04-22T18:05:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:05:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:05:53.625Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:05:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:54 smithi121 bash[36902]: cluster 2022-04-22T18:05:52.870367+0000 mgr.y (mgr.34107) 1407 : cluster [DBG] pgmap v1975: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:54 smithi115 bash[45495]: cluster 2022-04-22T18:05:52.870367+0000 mgr.y (mgr.34107) 1407 : cluster [DBG] pgmap v1975: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:54 smithi115 bash[46934]: cluster 2022-04-22T18:05:52.870367+0000 mgr.y (mgr.34107) 1407 : cluster [DBG] pgmap v1975: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:54.685 INFO:tasks.workunit.client.0.smithi115.stdout: 1123: throughput=5MB/sec pending data=0 2022-04-22T18:05:54.685 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KvNVkQEsgxI_TM4 off=6490502 len=892495 2022-04-22T18:05:54.692 INFO:tasks.workunit.client.0.smithi115.stdout:op 5736 completed, throughput=5MB/sec 2022-04-22T18:05:54.692 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-FkHD2gvVWV7nAa- off=10285601 len=395664 2022-04-22T18:05:54.698 INFO:tasks.workunit.client.0.smithi115.stdout:op 5737 completed, throughput=5MB/sec 2022-04-22T18:05:54.698 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KK-YmV_2LOUIYFq off=431234 len=1400810 2022-04-22T18:05:54.706 INFO:tasks.workunit.client.0.smithi115.stdout:op 5738 completed, throughput=5MB/sec 2022-04-22T18:05:54.706 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3DeRsG2z-fflL2z off=255513 len=1661375 2022-04-22T18:05:54.714 INFO:tasks.workunit.client.0.smithi115.stdout:op 5739 completed, throughput=5MB/sec 2022-04-22T18:05:54.715 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TN58XTuexQ6tH_8 off=6083634 len=414036 2022-04-22T18:05:54.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 5740 completed, throughput=5MB/sec 2022-04-22T18:05:54.716 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-sFP5mt7jrP52p1- off=18394249 len=1101435 2022-04-22T18:05:54.721 INFO:tasks.workunit.client.0.smithi115.stdout:op 5741 completed, throughput=5MB/sec 2022-04-22T18:05:55.721 INFO:tasks.workunit.client.0.smithi115.stdout: 1124: throughput=5MB/sec pending data=0 2022-04-22T18:05:55.722 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-RYQO_tiSZ9bopNs off=16996705 len=1363440 2022-04-22T18:05:55.731 INFO:tasks.workunit.client.0.smithi115.stdout:op 5742 completed, throughput=5MB/sec 2022-04-22T18:05:55.731 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eZqyOmcjbH1nRg- off=1019195 len=2081099 2022-04-22T18:05:55.742 INFO:tasks.workunit.client.0.smithi115.stdout:op 5743 completed, throughput=5MB/sec 2022-04-22T18:05:55.742 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QJBcQEAr1OGSDOd off=22928913 len=1016595 2022-04-22T18:05:55.748 INFO:tasks.workunit.client.0.smithi115.stdout:op 5744 completed, throughput=5MB/sec 2022-04-22T18:05:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:56 smithi121 bash[36902]: cluster 2022-04-22T18:05:54.871436+0000 mgr.y (mgr.34107) 1408 : cluster [DBG] pgmap v1976: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:56 smithi121 bash[36902]: audit 2022-04-22T18:05:55.472828+0000 mon.a (mon.0) 2967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:56 smithi121 bash[36902]: audit 2022-04-22T18:05:55.883145+0000 mon.a (mon.0) 2968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[45495]: cluster 2022-04-22T18:05:54.871436+0000 mgr.y (mgr.34107) 1408 : cluster [DBG] pgmap v1976: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[45495]: audit 2022-04-22T18:05:55.472828+0000 mon.a (mon.0) 2967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[45495]: audit 2022-04-22T18:05:55.883145+0000 mon.a (mon.0) 2968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[46934]: cluster 2022-04-22T18:05:54.871436+0000 mgr.y (mgr.34107) 1408 : cluster [DBG] pgmap v1976: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:05:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[46934]: audit 2022-04-22T18:05:55.472828+0000 mon.a (mon.0) 2967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:05:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:56 smithi115 bash[46934]: audit 2022-04-22T18:05:55.883145+0000 mon.a (mon.0) 2968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:05:56.748 INFO:tasks.workunit.client.0.smithi115.stdout: 1125: throughput=5MB/sec pending data=0 2022-04-22T18:05:56.774 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kJx7ykhB-54hTY0 off=10202877 len=613399 2022-04-22T18:05:56.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 5745 completed, throughput=5MB/sec 2022-04-22T18:05:56.775 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tyLAuKfBbt7MfNI off=3978586 len=1482818 2022-04-22T18:05:56.775 INFO:tasks.workunit.client.0.smithi115.stdout:op 5746 completed, throughput=5MB/sec 2022-04-22T18:05:56.775 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r3mj4tZBzRFfIqP off=192159 len=1878196 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 5747 completed, throughput=5MB/sec 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-z4TsDK2SOtgwTCw off=2913173 len=424846 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 5748 completed, throughput=5MB/sec 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-5fXIOGnHVs6cwg8 off=4312966 len=823110 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:op 5749 completed, throughput=5MB/sec 2022-04-22T18:05:56.776 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vn9wlYC-6zygMWK off=19880749 len=832157 2022-04-22T18:05:56.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 5750 completed, throughput=5MB/sec 2022-04-22T18:05:57.779 INFO:tasks.workunit.client.0.smithi115.stdout: 1126: throughput=5MB/sec pending data=0 2022-04-22T18:05:57.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-5d5ew1JU_Ph4C82 off=2221205 len=649514 2022-04-22T18:05:57.785 INFO:tasks.workunit.client.0.smithi115.stdout:op 5751 completed, throughput=5MB/sec 2022-04-22T18:05:57.785 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0xUetVde6OI_gQK off=3928883 len=567933 2022-04-22T18:05:57.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 5752 completed, throughput=5MB/sec 2022-04-22T18:05:57.790 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ZTKvXENWfhsiXWY off=1295938 len=107347 2022-04-22T18:05:57.793 INFO:tasks.workunit.client.0.smithi115.stdout:op 5753 completed, throughput=5MB/sec 2022-04-22T18:05:57.793 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4l_RQQduz0nRwui off=289380 len=1564001 2022-04-22T18:05:57.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 5754 completed, throughput=5MB/sec 2022-04-22T18:05:57.796 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-F2U_Jn-y-aDPC5O off=8229690 len=1247161 2022-04-22T18:05:57.801 INFO:tasks.workunit.client.0.smithi115.stdout:op 5755 completed, throughput=5MB/sec 2022-04-22T18:05:57.801 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yINBxO2so3fGJzE off=578022 len=1047544 2022-04-22T18:05:57.809 INFO:tasks.workunit.client.0.smithi115.stdout:op 5756 completed, throughput=5MB/sec 2022-04-22T18:05:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:58 smithi121 bash[36902]: cluster 2022-04-22T18:05:56.872567+0000 mgr.y (mgr.34107) 1409 : cluster [DBG] pgmap v1977: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:05:58 smithi121 bash[36902]: audit 2022-04-22T18:05:57.863739+0000 mon.a (mon.0) 2969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:58 smithi115 bash[45495]: cluster 2022-04-22T18:05:56.872567+0000 mgr.y (mgr.34107) 1409 : cluster [DBG] pgmap v1977: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:05:58 smithi115 bash[45495]: audit 2022-04-22T18:05:57.863739+0000 mon.a (mon.0) 2969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:58 smithi115 bash[46934]: cluster 2022-04-22T18:05:56.872567+0000 mgr.y (mgr.34107) 1409 : cluster [DBG] pgmap v1977: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:05:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:05:58 smithi115 bash[46934]: audit 2022-04-22T18:05:57.863739+0000 mon.a (mon.0) 2969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:05:58.809 INFO:tasks.workunit.client.0.smithi115.stdout: 1127: throughput=5MB/sec pending data=0 2022-04-22T18:05:58.809 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-hR6D8PLyRw2u5lv off=13085689 len=259383 2022-04-22T18:05:58.811 INFO:tasks.workunit.client.0.smithi115.stdout:op 5757 completed, throughput=5MB/sec 2022-04-22T18:05:58.811 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VjkZD4lfqDDeXVz off=14480003 len=1895880 2022-04-22T18:05:58.823 INFO:tasks.workunit.client.0.smithi115.stdout:op 5758 completed, throughput=5MB/sec 2022-04-22T18:05:58.823 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0xwycwqVUvzT4Bz off=8173513 len=1365665 2022-04-22T18:05:58.830 INFO:tasks.workunit.client.0.smithi115.stdout:op 5759 completed, throughput=5MB/sec 2022-04-22T18:05:58.830 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-E-USYzZ2ZPCfj41 off=124516 len=1518959 2022-04-22T18:05:58.839 INFO:tasks.workunit.client.0.smithi115.stdout:op 5760 completed, throughput=5MB/sec 2022-04-22T18:05:59.840 INFO:tasks.workunit.client.0.smithi115.stdout: 1128: throughput=5MB/sec pending data=0 2022-04-22T18:05:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-S_-qfwmPNLwc9zk off=1687619 len=1991752 2022-04-22T18:05:59.844 INFO:tasks.workunit.client.0.smithi115.stdout:op 5761 completed, throughput=5MB/sec 2022-04-22T18:05:59.844 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DCKWdzLUsxFf04l off=3389640 len=1926169 2022-04-22T18:05:59.846 INFO:tasks.workunit.client.0.smithi115.stdout:op 5762 completed, throughput=5MB/sec 2022-04-22T18:05:59.846 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-qgGArOtjMAPL2Zj off=14737804 len=79473 2022-04-22T18:05:59.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 5763 completed, throughput=5MB/sec 2022-04-22T18:05:59.851 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-0EhRkFyWw3-jwB5 off=439804 len=1155868 2022-04-22T18:05:59.857 INFO:tasks.workunit.client.0.smithi115.stdout:op 5764 completed, throughput=5MB/sec 2022-04-22T18:06:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:00 smithi121 bash[36902]: cluster 2022-04-22T18:05:58.873734+0000 mgr.y (mgr.34107) 1410 : cluster [DBG] pgmap v1978: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:06:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:00 smithi115 bash[45495]: cluster 2022-04-22T18:05:58.873734+0000 mgr.y (mgr.34107) 1410 : cluster [DBG] pgmap v1978: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:06:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:00 smithi115 bash[46934]: cluster 2022-04-22T18:05:58.873734+0000 mgr.y (mgr.34107) 1410 : cluster [DBG] pgmap v1978: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 5 op/s 2022-04-22T18:06:00.857 INFO:tasks.workunit.client.0.smithi115.stdout: 1129: throughput=5MB/sec pending data=0 2022-04-22T18:06:00.858 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gSr_3moAFlLqgfF off=7378724 len=1822494 2022-04-22T18:06:00.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 5765 completed, throughput=5MB/sec 2022-04-22T18:06:00.869 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-geFC4BEtGOS82pI off=1555939 len=1019351 2022-04-22T18:06:00.875 INFO:tasks.workunit.client.0.smithi115.stdout:op 5766 completed, throughput=5MB/sec 2022-04-22T18:06:00.875 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NizmnJErrX7DNwE off=8315528 len=184198 2022-04-22T18:06:00.876 INFO:tasks.workunit.client.0.smithi115.stdout:op 5767 completed, throughput=5MB/sec 2022-04-22T18:06:00.876 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ColL9xIR8f0Emo4 off=18143853 len=1562734 2022-04-22T18:06:00.878 INFO:tasks.workunit.client.0.smithi115.stdout:op 5768 completed, throughput=5MB/sec 2022-04-22T18:06:00.878 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ILlLAmu9ogIUgqS off=5227315 len=1865046 2022-04-22T18:06:00.883 INFO:tasks.workunit.client.0.smithi115.stdout:op 5769 completed, throughput=5MB/sec 2022-04-22T18:06:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:01 smithi121 bash[36902]: audit 2022-04-22T18:06:00.475860+0000 mon.a (mon.0) 2970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:01 smithi121 bash[36902]: audit 2022-04-22T18:06:00.886524+0000 mon.a (mon.0) 2971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:01 smithi115 bash[45495]: audit 2022-04-22T18:06:00.475860+0000 mon.a (mon.0) 2970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:01 smithi115 bash[45495]: audit 2022-04-22T18:06:00.886524+0000 mon.a (mon.0) 2971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:01 smithi115 bash[46934]: audit 2022-04-22T18:06:00.475860+0000 mon.a (mon.0) 2970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:01 smithi115 bash[46934]: audit 2022-04-22T18:06:00.886524+0000 mon.a (mon.0) 2971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:01.884 INFO:tasks.workunit.client.0.smithi115.stdout: 1130: throughput=5MB/sec pending data=0 2022-04-22T18:06:01.884 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nWdf39EH9iYe9Lf off=2616085 len=1406269 2022-04-22T18:06:01.887 INFO:tasks.workunit.client.0.smithi115.stdout:op 5770 completed, throughput=5MB/sec 2022-04-22T18:06:01.887 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-okujDCayQSLM3o5 off=4293132 len=1420719 2022-04-22T18:06:01.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 5771 completed, throughput=5MB/sec 2022-04-22T18:06:01.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vbfOf9wzS6VmnoN off=6602717 len=806149 2022-04-22T18:06:01.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 5772 completed, throughput=5MB/sec 2022-04-22T18:06:01.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nzjnfuVm46S_e21 off=794135 len=103913 2022-04-22T18:06:01.898 INFO:tasks.workunit.client.0.smithi115.stdout:op 5773 completed, throughput=5MB/sec 2022-04-22T18:06:01.898 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-DbZFTaHWmrqrkOG off=11712995 len=378425 2022-04-22T18:06:01.899 INFO:tasks.workunit.client.0.smithi115.stdout:op 5774 completed, throughput=5MB/sec 2022-04-22T18:06:01.899 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lw1rJNGHJSuF5nD off=3120968 len=1032268 2022-04-22T18:06:01.901 INFO:tasks.workunit.client.0.smithi115.stdout:op 5775 completed, throughput=5MB/sec 2022-04-22T18:06:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:01.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:02 smithi121 bash[36902]: cluster 2022-04-22T18:06:00.874696+0000 mgr.y (mgr.34107) 1411 : cluster [DBG] pgmap v1979: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:02.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:02 smithi115 bash[45495]: cluster 2022-04-22T18:06:00.874696+0000 mgr.y (mgr.34107) 1411 : cluster [DBG] pgmap v1979: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:02.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:02 smithi115 bash[46934]: cluster 2022-04-22T18:06:00.874696+0000 mgr.y (mgr.34107) 1411 : cluster [DBG] pgmap v1979: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:02.905 INFO:tasks.workunit.client.0.smithi115.stdout: 1131: throughput=5MB/sec pending data=0 2022-04-22T18:06:02.906 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tEOGKRhl5OzY_yR off=4774731 len=391128 2022-04-22T18:06:02.906 INFO:tasks.workunit.client.0.smithi115.stdout:op 5776 completed, throughput=5MB/sec 2022-04-22T18:06:02.906 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gtvtC0JWY89JfXI off=8083460 len=743638 2022-04-22T18:06:02.906 INFO:tasks.workunit.client.0.smithi115.stdout:op 5777 completed, throughput=5MB/sec 2022-04-22T18:06:02.906 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q087anOTdsbjGU5 off=11132001 len=178777 2022-04-22T18:06:02.907 INFO:tasks.workunit.client.0.smithi115.stdout:op 5778 completed, throughput=5MB/sec 2022-04-22T18:06:02.907 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Hl8gCN4Me6XKx57 off=4558571 len=373558 2022-04-22T18:06:02.908 INFO:tasks.workunit.client.0.smithi115.stdout:op 5779 completed, throughput=5MB/sec 2022-04-22T18:06:02.908 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-GIqFzaJzoDeAuo5 off=0 len=1301703 2022-04-22T18:06:02.917 INFO:tasks.workunit.client.0.smithi115.stdout:op 5780 completed, throughput=5MB/sec 2022-04-22T18:06:02.917 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H5-2SooeQQ_MYE5 off=11414633 len=1854415 2022-04-22T18:06:02.926 INFO:tasks.workunit.client.0.smithi115.stdout:op 5781 completed, throughput=5MB/sec 2022-04-22T18:06:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:03 smithi121 bash[36902]: audit 2022-04-22T18:06:02.866694+0000 mon.a (mon.0) 2972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:03.549 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:06:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:03 smithi115 bash[45495]: audit 2022-04-22T18:06:02.866694+0000 mon.a (mon.0) 2972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:03 smithi115 bash[46934]: audit 2022-04-22T18:06:02.866694+0000 mon.a (mon.0) 2972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:03.869 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:06:03.869 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (29m) 8m ago 42m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:06:03.869 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (42m) 9m ago 42m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:06:03.869 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (31m) 8m ago 43m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:06:03.869 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (30m) 9m ago 43m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (38m) 9m ago 48m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (39m) 8m ago 50m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (38m) 8m ago 50m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (37m) 9m ago 48m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (37m) 8m ago 48m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (30m) 8m ago 43m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (30m) 9m ago 43m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (35m) 8m ago 47m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:06:03.870 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (35m) 8m ago 47m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (34m) 8m ago 46m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (34m) 8m ago 46m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (33m) 9m ago 45m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (32m) 9m ago 45m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (32m) 9m ago 44m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (31m) 9m ago 44m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:06:03.871 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (29m) 9m ago 43m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:06:03.927 INFO:tasks.workunit.client.0.smithi115.stdout: 1132: throughput=5MB/sec pending data=0 2022-04-22T18:06:03.927 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dZ4MgEKZQW7a5jF off=8370966 len=1305326 2022-04-22T18:06:03.935 INFO:tasks.workunit.client.0.smithi115.stdout:op 5782 completed, throughput=5MB/sec 2022-04-22T18:06:03.936 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yZmNRblb3QjKcmd off=2969735 len=1921474 2022-04-22T18:06:03.938 INFO:tasks.workunit.client.0.smithi115.stdout:op 5783 completed, throughput=5MB/sec 2022-04-22T18:06:03.938 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Q-bAb93Q64zJSFx off=5697361 len=631340 2022-04-22T18:06:03.944 INFO:tasks.workunit.client.0.smithi115.stdout:op 5784 completed, throughput=5MB/sec 2022-04-22T18:06:03.944 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-s9Ey-JWyUsvG3xi off=23414580 len=1952926 2022-04-22T18:06:03.951 INFO:tasks.workunit.client.0.smithi115.stdout:op 5785 completed, throughput=5MB/sec 2022-04-22T18:06:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:03.622Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:04.272 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:06:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:06:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:06:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:06:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:06:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:06:04.274 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:06:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:04 smithi121 bash[36902]: cluster 2022-04-22T18:06:02.875747+0000 mgr.y (mgr.34107) 1412 : cluster [DBG] pgmap v1980: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:04 smithi115 bash[45495]: cluster 2022-04-22T18:06:02.875747+0000 mgr.y (mgr.34107) 1412 : cluster [DBG] pgmap v1980: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:04 smithi115 bash[46934]: cluster 2022-04-22T18:06:02.875747+0000 mgr.y (mgr.34107) 1412 : cluster [DBG] pgmap v1980: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:04.952 INFO:tasks.workunit.client.0.smithi115.stdout: 1133: throughput=5MB/sec pending data=0 2022-04-22T18:06:04.953 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-m-E1lp44yi0GVwV off=12028128 len=1599929 2022-04-22T18:06:04.955 INFO:tasks.workunit.client.0.smithi115.stdout:op 5786 completed, throughput=5MB/sec 2022-04-22T18:06:04.956 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JqMp7jY2fWHeGmu off=4041235 len=1829959 2022-04-22T18:06:04.961 INFO:tasks.workunit.client.0.smithi115.stdout:op 5787 completed, throughput=5MB/sec 2022-04-22T18:06:04.961 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Q9l-kUT4w9WTYIJ off=2261092 len=1666331 2022-04-22T18:06:04.962 INFO:tasks.workunit.client.0.smithi115.stdout:op 5788 completed, throughput=5MB/sec 2022-04-22T18:06:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:05 smithi121 bash[36902]: audit 2022-04-22T18:06:03.504920+0000 mgr.y (mgr.34107) 1413 : audit [DBG] from='client.35877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:05 smithi121 bash[36902]: audit 2022-04-22T18:06:03.842540+0000 mgr.y (mgr.34107) 1414 : audit [DBG] from='client.45823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:05 smithi121 bash[36902]: audit 2022-04-22T18:06:04.266471+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.115:0/47420638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[45495]: audit 2022-04-22T18:06:03.504920+0000 mgr.y (mgr.34107) 1413 : audit [DBG] from='client.35877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[45495]: audit 2022-04-22T18:06:03.842540+0000 mgr.y (mgr.34107) 1414 : audit [DBG] from='client.45823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[45495]: audit 2022-04-22T18:06:04.266471+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.115:0/47420638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:05.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[46934]: audit 2022-04-22T18:06:03.504920+0000 mgr.y (mgr.34107) 1413 : audit [DBG] from='client.35877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[46934]: audit 2022-04-22T18:06:03.842540+0000 mgr.y (mgr.34107) 1414 : audit [DBG] from='client.45823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:05 smithi115 bash[46934]: audit 2022-04-22T18:06:04.266471+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.115:0/47420638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:05.962 INFO:tasks.workunit.client.0.smithi115.stdout: 1134: throughput=5MB/sec pending data=0 2022-04-22T18:06:05.962 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-v6Pi5aMs0HCLFtR off=15431748 len=1833984 2022-04-22T18:06:05.967 INFO:tasks.workunit.client.0.smithi115.stdout:op 5789 completed, throughput=5MB/sec 2022-04-22T18:06:05.967 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CBmYFrqU9HTuI3Y off=8226649 len=710737 2022-04-22T18:06:05.970 INFO:tasks.workunit.client.0.smithi115.stdout:op 5790 completed, throughput=5MB/sec 2022-04-22T18:06:05.970 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1SZLN4rlbJNqHbj off=19468797 len=875327 2022-04-22T18:06:05.972 INFO:tasks.workunit.client.0.smithi115.stdout:op 5791 completed, throughput=5MB/sec 2022-04-22T18:06:05.972 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-h_J_BMvdmro4NMV off=2424218 len=1695815 2022-04-22T18:06:05.976 INFO:tasks.workunit.client.0.smithi115.stdout:op 5792 completed, throughput=5MB/sec 2022-04-22T18:06:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:06 smithi121 bash[36902]: cluster 2022-04-22T18:06:04.876751+0000 mgr.y (mgr.34107) 1415 : cluster [DBG] pgmap v1981: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:06:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:06 smithi121 bash[36902]: audit 2022-04-22T18:06:05.478877+0000 mon.a (mon.0) 2973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:06 smithi121 bash[36902]: audit 2022-04-22T18:06:05.889540+0000 mon.a (mon.0) 2974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[45495]: cluster 2022-04-22T18:06:04.876751+0000 mgr.y (mgr.34107) 1415 : cluster [DBG] pgmap v1981: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:06:06.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[45495]: audit 2022-04-22T18:06:05.478877+0000 mon.a (mon.0) 2973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:06.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[45495]: audit 2022-04-22T18:06:05.889540+0000 mon.a (mon.0) 2974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:06.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[46934]: cluster 2022-04-22T18:06:04.876751+0000 mgr.y (mgr.34107) 1415 : cluster [DBG] pgmap v1981: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 5 op/s 2022-04-22T18:06:06.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[46934]: audit 2022-04-22T18:06:05.478877+0000 mon.a (mon.0) 2973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:06.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:06 smithi115 bash[46934]: audit 2022-04-22T18:06:05.889540+0000 mon.a (mon.0) 2974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:06.976 INFO:tasks.workunit.client.0.smithi115.stdout: 1135: throughput=5MB/sec pending data=0 2022-04-22T18:06:06.976 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-meotNQxY_fOcnh- off=13560242 len=1265395 2022-04-22T18:06:06.979 INFO:tasks.workunit.client.0.smithi115.stdout:op 5793 completed, throughput=5MB/sec 2022-04-22T18:06:06.980 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-QX4PN6JUSmy9Vem off=799002 len=510252 2022-04-22T18:06:06.985 INFO:tasks.workunit.client.0.smithi115.stdout:op 5794 completed, throughput=5MB/sec 2022-04-22T18:06:06.985 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mLY9E2eezX6_fdk off=14640349 len=201659 2022-04-22T18:06:06.990 INFO:tasks.workunit.client.0.smithi115.stdout:op 5795 completed, throughput=5MB/sec 2022-04-22T18:06:06.990 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_etHzyhSrrzA6je off=1412571 len=326986 2022-04-22T18:06:06.994 INFO:tasks.workunit.client.0.smithi115.stdout:op 5796 completed, throughput=5MB/sec 2022-04-22T18:06:06.994 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-34Jw9ejfxOKaalX off=5984912 len=899713 2022-04-22T18:06:07.000 INFO:tasks.workunit.client.0.smithi115.stdout:op 5797 completed, throughput=5MB/sec 2022-04-22T18:06:07.000 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-F2GIVE14KDxcNZ7 off=9907989 len=173415 2022-04-22T18:06:07.001 INFO:tasks.workunit.client.0.smithi115.stdout:op 5798 completed, throughput=5MB/sec 2022-04-22T18:06:07.001 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gfy0i9Gd2uevLTK off=16828644 len=1268894 2022-04-22T18:06:07.008 INFO:tasks.workunit.client.0.smithi115.stdout:op 5799 completed, throughput=5MB/sec 2022-04-22T18:06:07.008 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jFJs2JSZJWGDZw7 off=3728415 len=1208281 2022-04-22T18:06:07.013 INFO:tasks.workunit.client.0.smithi115.stdout:op 5800 completed, throughput=5MB/sec 2022-04-22T18:06:08.013 INFO:tasks.workunit.client.0.smithi115.stdout: 1136: throughput=5MB/sec pending data=0 2022-04-22T18:06:08.013 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nFkByPM8BBEbfV3 off=8939610 len=1532350 2022-04-22T18:06:08.021 INFO:tasks.workunit.client.0.smithi115.stdout:op 5801 completed, throughput=5MB/sec 2022-04-22T18:06:08.022 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-45-OxVVFQesSIRO off=6515759 len=1409823 2022-04-22T18:06:08.025 INFO:tasks.workunit.client.0.smithi115.stdout:op 5802 completed, throughput=5MB/sec 2022-04-22T18:06:08.025 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rMs7oO7VcmqLOgT off=178647 len=38709 2022-04-22T18:06:08.026 INFO:tasks.workunit.client.0.smithi115.stdout:op 5803 completed, throughput=5MB/sec 2022-04-22T18:06:08.026 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IRae_k9rEKeNp0X off=629183 len=501656 2022-04-22T18:06:08.027 INFO:tasks.workunit.client.0.smithi115.stdout:op 5804 completed, throughput=5MB/sec 2022-04-22T18:06:08.027 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-oWFVxBrFmd6OYAm off=20652369 len=1770241 2022-04-22T18:06:08.037 INFO:tasks.workunit.client.0.smithi115.stdout:op 5805 completed, throughput=5MB/sec 2022-04-22T18:06:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:08 smithi121 bash[36902]: cluster 2022-04-22T18:06:06.877871+0000 mgr.y (mgr.34107) 1416 : cluster [DBG] pgmap v1982: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:08 smithi121 bash[36902]: audit 2022-04-22T18:06:07.869913+0000 mon.a (mon.0) 2975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:08 smithi115 bash[45495]: cluster 2022-04-22T18:06:06.877871+0000 mgr.y (mgr.34107) 1416 : cluster [DBG] pgmap v1982: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:08 smithi115 bash[45495]: audit 2022-04-22T18:06:07.869913+0000 mon.a (mon.0) 2975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:08 smithi115 bash[46934]: cluster 2022-04-22T18:06:06.877871+0000 mgr.y (mgr.34107) 1416 : cluster [DBG] pgmap v1982: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 4 op/s 2022-04-22T18:06:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:08 smithi115 bash[46934]: audit 2022-04-22T18:06:07.869913+0000 mon.a (mon.0) 2975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:09.043 INFO:tasks.workunit.client.0.smithi115.stdout: 1137: throughput=5MB/sec pending data=0 2022-04-22T18:06:09.043 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BtOF5sUPw18NkYx off=1805403 len=1684170 2022-04-22T18:06:09.043 INFO:tasks.workunit.client.0.smithi115.stdout:op 5806 completed, throughput=5MB/sec 2022-04-22T18:06:09.043 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gWWEc8C-OH1T4s- off=9656593 len=1910205 2022-04-22T18:06:09.050 INFO:tasks.workunit.client.0.smithi115.stdout:op 5807 completed, throughput=5MB/sec 2022-04-22T18:06:09.050 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PP_912rvtWj8Cmj off=9737298 len=709842 2022-04-22T18:06:09.052 INFO:tasks.workunit.client.0.smithi115.stdout:op 5808 completed, throughput=5MB/sec 2022-04-22T18:06:09.053 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-509xt1HgjyDUOYD off=10746261 len=59266 2022-04-22T18:06:09.053 INFO:tasks.workunit.client.0.smithi115.stdout:op 5809 completed, throughput=5MB/sec 2022-04-22T18:06:09.053 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Hx_OB-q81JZH3cO off=9516941 len=1716421 2022-04-22T18:06:09.058 INFO:tasks.workunit.client.0.smithi115.stdout:op 5810 completed, throughput=5MB/sec 2022-04-22T18:06:10.059 INFO:tasks.workunit.client.0.smithi115.stdout: 1138: throughput=5MB/sec pending data=0 2022-04-22T18:06:10.059 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-R9KBKG7eSFw-jtO off=17754224 len=1443456 2022-04-22T18:06:10.062 INFO:tasks.workunit.client.0.smithi115.stdout:op 5811 completed, throughput=5MB/sec 2022-04-22T18:06:10.063 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Y2pUdvDZAdvyG8M off=4121643 len=1563034 2022-04-22T18:06:10.066 INFO:tasks.workunit.client.0.smithi115.stdout:op 5812 completed, throughput=5MB/sec 2022-04-22T18:06:10.066 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YLL3sAJN9YCHCWN off=6775057 len=1451742 2022-04-22T18:06:10.073 INFO:tasks.workunit.client.0.smithi115.stdout:op 5813 completed, throughput=5MB/sec 2022-04-22T18:06:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:10 smithi121 bash[36902]: cluster 2022-04-22T18:06:08.878954+0000 mgr.y (mgr.34107) 1417 : cluster [DBG] pgmap v1983: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:06:10.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:10 smithi115 bash[45495]: cluster 2022-04-22T18:06:08.878954+0000 mgr.y (mgr.34107) 1417 : cluster [DBG] pgmap v1983: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:06:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:10 smithi115 bash[46934]: cluster 2022-04-22T18:06:08.878954+0000 mgr.y (mgr.34107) 1417 : cluster [DBG] pgmap v1983: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 4 op/s 2022-04-22T18:06:11.073 INFO:tasks.workunit.client.0.smithi115.stdout: 1139: throughput=5MB/sec pending data=0 2022-04-22T18:06:11.074 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O-nzfQYTse6VlM3 off=518786 len=97984 2022-04-22T18:06:11.078 INFO:tasks.workunit.client.0.smithi115.stdout:op 5814 completed, throughput=5MB/sec 2022-04-22T18:06:11.078 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AdHwvL3k3TshcSw off=20474130 len=983776 2022-04-22T18:06:11.081 INFO:tasks.workunit.client.0.smithi115.stdout:op 5815 completed, throughput=5MB/sec 2022-04-22T18:06:11.081 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UX_PJ38Z6QX-sec off=791123 len=1787109 2022-04-22T18:06:11.084 INFO:tasks.workunit.client.0.smithi115.stdout:op 5816 completed, throughput=5MB/sec 2022-04-22T18:06:11.084 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fRkJvN0CnysFiV4 off=2526647 len=1351475 2022-04-22T18:06:11.092 INFO:tasks.workunit.client.0.smithi115.stdout:op 5817 completed, throughput=5MB/sec 2022-04-22T18:06:11.093 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rzrKjfEDHv2XRhi off=8840389 len=1446194 2022-04-22T18:06:11.100 INFO:tasks.workunit.client.0.smithi115.stdout:op 5818 completed, throughput=5MB/sec 2022-04-22T18:06:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:11 smithi121 bash[36902]: audit 2022-04-22T18:06:10.482061+0000 mon.a (mon.0) 2976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:11.537 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:11 smithi121 bash[36902]: audit 2022-04-22T18:06:10.893255+0000 mon.a (mon.0) 2977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:11 smithi115 bash[46934]: audit 2022-04-22T18:06:10.482061+0000 mon.a (mon.0) 2976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:11 smithi115 bash[46934]: audit 2022-04-22T18:06:10.893255+0000 mon.a (mon.0) 2977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:11 smithi115 bash[45495]: audit 2022-04-22T18:06:10.482061+0000 mon.a (mon.0) 2976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:11 smithi115 bash[45495]: audit 2022-04-22T18:06:10.893255+0000 mon.a (mon.0) 2977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:12.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:11.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:12.100 INFO:tasks.workunit.client.0.smithi115.stdout: 1140: throughput=5MB/sec pending data=0 2022-04-22T18:06:12.100 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-KB_AQGaiu7aLGhg off=8755578 len=159321 2022-04-22T18:06:12.105 INFO:tasks.workunit.client.0.smithi115.stdout:op 5819 completed, throughput=5MB/sec 2022-04-22T18:06:12.105 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_JxzktCzM7trofo off=13283221 len=1697500 2022-04-22T18:06:12.116 INFO:tasks.workunit.client.0.smithi115.stdout:op 5820 completed, throughput=5MB/sec 2022-04-22T18:06:12.117 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_8BeQTBW7Igv02d off=12231292 len=378041 2022-04-22T18:06:12.117 INFO:tasks.workunit.client.0.smithi115.stdout:op 5821 completed, throughput=5MB/sec 2022-04-22T18:06:12.118 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-UBJ9P4cM_zQKHyc off=6976613 len=2044102 2022-04-22T18:06:12.127 INFO:tasks.workunit.client.0.smithi115.stdout:op 5822 completed, throughput=5MB/sec 2022-04-22T18:06:12.127 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Zz-Cv-1txmqtX2o off=20092128 len=1002987 2022-04-22T18:06:12.134 INFO:tasks.workunit.client.0.smithi115.stdout:op 5823 completed, throughput=5MB/sec 2022-04-22T18:06:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:12 smithi121 bash[36902]: cluster 2022-04-22T18:06:10.880123+0000 mgr.y (mgr.34107) 1418 : cluster [DBG] pgmap v1984: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:06:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:12 smithi115 bash[45495]: cluster 2022-04-22T18:06:10.880123+0000 mgr.y (mgr.34107) 1418 : cluster [DBG] pgmap v1984: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:06:12.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:12 smithi115 bash[46934]: cluster 2022-04-22T18:06:10.880123+0000 mgr.y (mgr.34107) 1418 : cluster [DBG] pgmap v1984: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 4 op/s 2022-04-22T18:06:13.134 INFO:tasks.workunit.client.0.smithi115.stdout: 1141: throughput=5MB/sec pending data=0 2022-04-22T18:06:13.134 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YNAWVm5tVZ3cRaf off=5703390 len=1480607 2022-04-22T18:06:13.137 INFO:tasks.workunit.client.0.smithi115.stdout:op 5824 completed, throughput=5MB/sec 2022-04-22T18:06:13.138 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-f2JbwgGMmsfd9uM off=4630018 len=70322 2022-04-22T18:06:13.138 INFO:tasks.workunit.client.0.smithi115.stdout:op 5825 completed, throughput=5MB/sec 2022-04-22T18:06:13.139 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NYoRu9y8lV53YtB off=136141 len=168478 2022-04-22T18:06:13.140 INFO:tasks.workunit.client.0.smithi115.stdout:op 5826 completed, throughput=5MB/sec 2022-04-22T18:06:13.140 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-rbcOAuwhrz7Qpyx off=15300988 len=1696511 2022-04-22T18:06:13.144 INFO:tasks.workunit.client.0.smithi115.stdout:op 5827 completed, throughput=5MB/sec 2022-04-22T18:06:13.144 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ui-yoEbTvcwdH4L off=10459365 len=781689 2022-04-22T18:06:13.145 INFO:tasks.workunit.client.0.smithi115.stdout:op 5828 completed, throughput=5MB/sec 2022-04-22T18:06:13.145 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-j18FMk0F6UtXer9 off=1640582 len=1751093 2022-04-22T18:06:13.155 INFO:tasks.workunit.client.0.smithi115.stdout:op 5829 completed, throughput=5MB/sec 2022-04-22T18:06:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:13 smithi121 bash[36902]: audit 2022-04-22T18:06:12.872861+0000 mon.a (mon.0) 2978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:13 smithi115 bash[45495]: audit 2022-04-22T18:06:12.872861+0000 mon.a (mon.0) 2978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:13 smithi115 bash[46934]: audit 2022-04-22T18:06:12.872861+0000 mon.a (mon.0) 2978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:14.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:13.627Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:14.155 INFO:tasks.workunit.client.0.smithi115.stdout: 1142: throughput=5MB/sec pending data=0 2022-04-22T18:06:14.175 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CSp1PCj-4h1B39H off=6886559 len=954133 2022-04-22T18:06:14.176 INFO:tasks.workunit.client.0.smithi115.stdout:op 5830 completed, throughput=5MB/sec 2022-04-22T18:06:14.176 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-O6KQ9SAjxUTtzf7 off=1180940 len=161799 2022-04-22T18:06:14.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 5831 completed, throughput=5MB/sec 2022-04-22T18:06:14.177 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0j9uuTw0PdDhhMj off=4130838 len=1532332 2022-04-22T18:06:14.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 5832 completed, throughput=5MB/sec 2022-04-22T18:06:14.177 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gNIsKmv_6cDWE3e off=1865263 len=1529631 2022-04-22T18:06:14.177 INFO:tasks.workunit.client.0.smithi115.stdout:op 5833 completed, throughput=5MB/sec 2022-04-22T18:06:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:14 smithi121 bash[36902]: cluster 2022-04-22T18:06:12.881019+0000 mgr.y (mgr.34107) 1419 : cluster [DBG] pgmap v1985: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:14 smithi121 bash[36902]: audit 2022-04-22T18:06:13.661237+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:06:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:14 smithi121 bash[36902]: audit 2022-04-22T18:06:13.662441+0000 mon.a (mon.0) 2980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:06:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[45495]: cluster 2022-04-22T18:06:12.881019+0000 mgr.y (mgr.34107) 1419 : cluster [DBG] pgmap v1985: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[45495]: audit 2022-04-22T18:06:13.661237+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:06:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[45495]: audit 2022-04-22T18:06:13.662441+0000 mon.a (mon.0) 2980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:06:14.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[46934]: cluster 2022-04-22T18:06:12.881019+0000 mgr.y (mgr.34107) 1419 : cluster [DBG] pgmap v1985: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:14.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[46934]: audit 2022-04-22T18:06:13.661237+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:06:14.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:14 smithi115 bash[46934]: audit 2022-04-22T18:06:13.662441+0000 mon.a (mon.0) 2980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:06:15.176 INFO:tasks.workunit.client.0.smithi115.stdout: 1143: throughput=5MB/sec pending data=0 2022-04-22T18:06:15.176 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-rWAOpu01bEtl5Lf off=495794 len=1718608 2022-04-22T18:06:15.185 INFO:tasks.workunit.client.0.smithi115.stdout:op 5834 completed, throughput=5MB/sec 2022-04-22T18:06:15.185 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GJ0sAifooTfae-Y off=15810092 len=685773 2022-04-22T18:06:15.187 INFO:tasks.workunit.client.0.smithi115.stdout:op 5835 completed, throughput=5MB/sec 2022-04-22T18:06:15.187 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jSGF9zlIAyqkSL5 off=2982499 len=276074 2022-04-22T18:06:15.192 INFO:tasks.workunit.client.0.smithi115.stdout:op 5836 completed, throughput=5MB/sec 2022-04-22T18:06:15.192 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-nQO6SHFZrC5Uv1U off=9200959 len=1359095 2022-04-22T18:06:15.195 INFO:tasks.workunit.client.0.smithi115.stdout:op 5837 completed, throughput=5MB/sec 2022-04-22T18:06:15.195 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3bzKAFpYDbSifZZ off=10182314 len=2028043 2022-04-22T18:06:15.205 INFO:tasks.workunit.client.0.smithi115.stdout:op 5838 completed, throughput=5MB/sec 2022-04-22T18:06:16.206 INFO:tasks.workunit.client.0.smithi115.stdout: 1144: throughput=5MB/sec pending data=0 2022-04-22T18:06:16.206 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-A-7CxMEnLoLLiJr off=1325375 len=869744 2022-04-22T18:06:16.212 INFO:tasks.workunit.client.0.smithi115.stdout:op 5839 completed, throughput=5MB/sec 2022-04-22T18:06:16.212 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HKcgUF1oElXdpjd off=6622596 len=690985 2022-04-22T18:06:16.215 INFO:tasks.workunit.client.0.smithi115.stdout:op 5840 completed, throughput=5MB/sec 2022-04-22T18:06:16.215 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kGLBP5vZgvNwH53 off=654445 len=942480 2022-04-22T18:06:16.217 INFO:tasks.workunit.client.0.smithi115.stdout:op 5841 completed, throughput=5MB/sec 2022-04-22T18:06:16.217 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zx7JCjcKJJTFiuk off=14000720 len=1158426 2022-04-22T18:06:16.220 INFO:tasks.workunit.client.0.smithi115.stdout:op 5842 completed, throughput=5MB/sec 2022-04-22T18:06:16.221 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-u4h5X3W9ngTn-B7 off=12124749 len=1973332 2022-04-22T18:06:16.228 INFO:tasks.workunit.client.0.smithi115.stdout:op 5843 completed, throughput=5MB/sec 2022-04-22T18:06:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:16 smithi121 bash[36902]: cluster 2022-04-22T18:06:14.881986+0000 mgr.y (mgr.34107) 1420 : cluster [DBG] pgmap v1986: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:06:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:16 smithi121 bash[36902]: audit 2022-04-22T18:06:15.485193+0000 mon.a (mon.0) 2981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:16 smithi121 bash[36902]: audit 2022-04-22T18:06:15.896922+0000 mon.a (mon.0) 2982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[45495]: cluster 2022-04-22T18:06:14.881986+0000 mgr.y (mgr.34107) 1420 : cluster [DBG] pgmap v1986: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:06:16.630 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[45495]: audit 2022-04-22T18:06:15.485193+0000 mon.a (mon.0) 2981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:16.630 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[45495]: audit 2022-04-22T18:06:15.896922+0000 mon.a (mon.0) 2982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:16.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[46934]: cluster 2022-04-22T18:06:14.881986+0000 mgr.y (mgr.34107) 1420 : cluster [DBG] pgmap v1986: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 4 op/s 2022-04-22T18:06:16.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[46934]: audit 2022-04-22T18:06:15.485193+0000 mon.a (mon.0) 2981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:16.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:16 smithi115 bash[46934]: audit 2022-04-22T18:06:15.896922+0000 mon.a (mon.0) 2982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:17.229 INFO:tasks.workunit.client.0.smithi115.stdout: 1145: throughput=5MB/sec pending data=0 2022-04-22T18:06:17.229 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-E7eTzz03LMPetAS off=17072360 len=1059364 2022-04-22T18:06:17.231 INFO:tasks.workunit.client.0.smithi115.stdout:op 5844 completed, throughput=5MB/sec 2022-04-22T18:06:17.232 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gt2qmwofFHV_eq5 off=2469987 len=433911 2022-04-22T18:06:17.233 INFO:tasks.workunit.client.0.smithi115.stdout:op 5845 completed, throughput=5MB/sec 2022-04-22T18:06:17.233 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-U3lCxeQvg8exb-n off=1236222 len=701936 2022-04-22T18:06:17.236 INFO:tasks.workunit.client.0.smithi115.stdout:op 5846 completed, throughput=5MB/sec 2022-04-22T18:06:17.236 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zY2rmRAPxW07Bjg off=2408469 len=1072051 2022-04-22T18:06:17.242 INFO:tasks.workunit.client.0.smithi115.stdout:op 5847 completed, throughput=5MB/sec 2022-04-22T18:06:17.242 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fqeDIQDhaqMvm0O off=9907045 len=1305437 2022-04-22T18:06:17.243 INFO:tasks.workunit.client.0.smithi115.stdout:op 5848 completed, throughput=5MB/sec 2022-04-22T18:06:18.243 INFO:tasks.workunit.client.0.smithi115.stdout: 1146: throughput=5MB/sec pending data=0 2022-04-22T18:06:18.244 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-kTzIX-KnWmSGghr off=13716596 len=460279 2022-04-22T18:06:18.246 INFO:tasks.workunit.client.0.smithi115.stdout:op 5849 completed, throughput=5MB/sec 2022-04-22T18:06:18.246 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6Cq3_bHDYGZ0j5Y off=7499003 len=87179 2022-04-22T18:06:18.247 INFO:tasks.workunit.client.0.smithi115.stdout:op 5850 completed, throughput=5MB/sec 2022-04-22T18:06:18.248 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CZPyZTyOwEHLfk0 off=6393638 len=1505665 2022-04-22T18:06:18.256 INFO:tasks.workunit.client.0.smithi115.stdout:op 5851 completed, throughput=5MB/sec 2022-04-22T18:06:18.257 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JpHkLhE5AX2OBFG off=7907789 len=1267852 2022-04-22T18:06:18.259 INFO:tasks.workunit.client.0.smithi115.stdout:op 5852 completed, throughput=5MB/sec 2022-04-22T18:06:18.259 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Xe8PC-AK-Tdcj7W off=5272961 len=907944 2022-04-22T18:06:18.265 INFO:tasks.workunit.client.0.smithi115.stdout:op 5853 completed, throughput=5MB/sec 2022-04-22T18:06:18.266 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-t-7Rnm2rIGAQUTH off=16918354 len=1729415 2022-04-22T18:06:18.267 INFO:tasks.workunit.client.0.smithi115.stdout:op 5854 completed, throughput=5MB/sec 2022-04-22T18:06:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:18 smithi121 bash[36902]: cluster 2022-04-22T18:06:16.882909+0000 mgr.y (mgr.34107) 1421 : cluster [DBG] pgmap v1987: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:06:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:18 smithi121 bash[36902]: audit 2022-04-22T18:06:17.876090+0000 mon.a (mon.0) 2983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:18 smithi115 bash[45495]: cluster 2022-04-22T18:06:16.882909+0000 mgr.y (mgr.34107) 1421 : cluster [DBG] pgmap v1987: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:06:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:18 smithi115 bash[45495]: audit 2022-04-22T18:06:17.876090+0000 mon.a (mon.0) 2983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:18 smithi115 bash[46934]: cluster 2022-04-22T18:06:16.882909+0000 mgr.y (mgr.34107) 1421 : cluster [DBG] pgmap v1987: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 4 op/s 2022-04-22T18:06:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:18 smithi115 bash[46934]: audit 2022-04-22T18:06:17.876090+0000 mon.a (mon.0) 2983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:19.268 INFO:tasks.workunit.client.0.smithi115.stdout: 1147: throughput=5MB/sec pending data=0 2022-04-22T18:06:19.268 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-9AZM8wPmEb7JSQ_ off=20876092 len=215385 2022-04-22T18:06:19.273 INFO:tasks.workunit.client.0.smithi115.stdout:op 5855 completed, throughput=5MB/sec 2022-04-22T18:06:19.273 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-EtKm-6E2vTPh7WU off=5363538 len=1562162 2022-04-22T18:06:19.275 INFO:tasks.workunit.client.0.smithi115.stdout:op 5856 completed, throughput=5MB/sec 2022-04-22T18:06:19.275 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-HawOaM58EXRRyXe off=14595965 len=920848 2022-04-22T18:06:19.281 INFO:tasks.workunit.client.0.smithi115.stdout:op 5857 completed, throughput=5MB/sec 2022-04-22T18:06:19.281 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-lIXrf5JR5GIbZ59 off=11170564 len=545511 2022-04-22T18:06:19.286 INFO:tasks.workunit.client.0.smithi115.stdout:op 5858 completed, throughput=5MB/sec 2022-04-22T18:06:19.286 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-pQNJ2mzFOjKaU_u off=6839715 len=832092 2022-04-22T18:06:19.287 INFO:tasks.workunit.client.0.smithi115.stdout:op 5859 completed, throughput=5MB/sec 2022-04-22T18:06:19.288 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-r4ZeYJXDbc8YoM_ off=1879549 len=33658 2022-04-22T18:06:19.288 INFO:tasks.workunit.client.0.smithi115.stdout:op 5860 completed, throughput=5MB/sec 2022-04-22T18:06:19.288 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-CsSJpAzEJQnpgN9 off=5411807 len=6562 2022-04-22T18:06:19.291 INFO:tasks.workunit.client.0.smithi115.stdout:op 5861 completed, throughput=5MB/sec 2022-04-22T18:06:19.291 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8YXUodt6hFD1i26 off=7604752 len=627585 2022-04-22T18:06:19.292 INFO:tasks.workunit.client.0.smithi115.stdout:op 5862 completed, throughput=5MB/sec 2022-04-22T18:06:20.293 INFO:tasks.workunit.client.0.smithi115.stdout: 1148: throughput=5MB/sec pending data=0 2022-04-22T18:06:20.293 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-MwTTHJRv6vH_Zye off=1900240 len=1847028 2022-04-22T18:06:20.302 INFO:tasks.workunit.client.0.smithi115.stdout:op 5863 completed, throughput=5MB/sec 2022-04-22T18:06:20.302 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-4DIYwWBU881jsVs off=1418537 len=1116153 2022-04-22T18:06:20.304 INFO:tasks.workunit.client.0.smithi115.stdout:op 5864 completed, throughput=5MB/sec 2022-04-22T18:06:20.304 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-AsridPbhJlp10_K off=618718 len=681384 2022-04-22T18:06:20.305 INFO:tasks.workunit.client.0.smithi115.stdout:op 5865 completed, throughput=5MB/sec 2022-04-22T18:06:20.305 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ADd-TjcCwyV_idM off=5923529 len=1893086 2022-04-22T18:06:20.314 INFO:tasks.workunit.client.0.smithi115.stdout:op 5866 completed, throughput=5MB/sec 2022-04-22T18:06:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:20 smithi121 bash[36902]: cluster 2022-04-22T18:06:18.884051+0000 mgr.y (mgr.34107) 1422 : cluster [DBG] pgmap v1988: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:06:20.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:20 smithi115 bash[45495]: cluster 2022-04-22T18:06:18.884051+0000 mgr.y (mgr.34107) 1422 : cluster [DBG] pgmap v1988: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:06:20.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:20 smithi115 bash[46934]: cluster 2022-04-22T18:06:18.884051+0000 mgr.y (mgr.34107) 1422 : cluster [DBG] pgmap v1988: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 4 op/s 2022-04-22T18:06:21.314 INFO:tasks.workunit.client.0.smithi115.stdout: 1149: throughput=5MB/sec pending data=0 2022-04-22T18:06:21.315 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gADZd0o-hFE3i3Q off=910788 len=287398 2022-04-22T18:06:21.319 INFO:tasks.workunit.client.0.smithi115.stdout:op 5867 completed, throughput=5MB/sec 2022-04-22T18:06:21.319 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-IksC4AfZKvQk-PJ off=6798882 len=764401 2022-04-22T18:06:21.321 INFO:tasks.workunit.client.0.smithi115.stdout:op 5868 completed, throughput=5MB/sec 2022-04-22T18:06:21.321 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bDvd1mrhUcwHFdG off=4781932 len=1547183 2022-04-22T18:06:21.325 INFO:tasks.workunit.client.0.smithi115.stdout:op 5869 completed, throughput=5MB/sec 2022-04-22T18:06:21.325 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CMRfV5G3jH9UuRu off=1827026 len=1122075 2022-04-22T18:06:21.329 INFO:tasks.workunit.client.0.smithi115.stdout:op 5870 completed, throughput=5MB/sec 2022-04-22T18:06:21.329 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-k2q2MNpo8QGLs4- off=1385298 len=760734 2022-04-22T18:06:21.330 INFO:tasks.workunit.client.0.smithi115.stdout:op 5871 completed, throughput=5MB/sec 2022-04-22T18:06:21.330 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fN65cjYdh9xRSOz off=4517053 len=477466 2022-04-22T18:06:21.334 INFO:tasks.workunit.client.0.smithi115.stdout:op 5872 completed, throughput=5MB/sec 2022-04-22T18:06:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:21 smithi121 bash[36902]: audit 2022-04-22T18:06:20.488314+0000 mon.a (mon.0) 2984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:21 smithi121 bash[36902]: audit 2022-04-22T18:06:20.900628+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:21.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:21 smithi115 bash[45495]: audit 2022-04-22T18:06:20.488314+0000 mon.a (mon.0) 2984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:21.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:21 smithi115 bash[45495]: audit 2022-04-22T18:06:20.900628+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:21 smithi115 bash[46934]: audit 2022-04-22T18:06:20.488314+0000 mon.a (mon.0) 2984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:21 smithi115 bash[46934]: audit 2022-04-22T18:06:20.900628+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:22.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:21.747Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:22.335 INFO:tasks.workunit.client.0.smithi115.stdout: 1150: throughput=5MB/sec pending data=0 2022-04-22T18:06:22.335 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-p_-P_GQ2A_1qUr4 off=13207365 len=301709 2022-04-22T18:06:22.341 INFO:tasks.workunit.client.0.smithi115.stdout:op 5873 completed, throughput=5MB/sec 2022-04-22T18:06:22.341 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-yg3BskhSHWKpgFo off=2781459 len=1725979 2022-04-22T18:06:22.349 INFO:tasks.workunit.client.0.smithi115.stdout:op 5874 completed, throughput=5MB/sec 2022-04-22T18:06:22.349 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ivPp3yMUKz6IxiH off=2525299 len=847071 2022-04-22T18:06:22.355 INFO:tasks.workunit.client.0.smithi115.stdout:op 5875 completed, throughput=5MB/sec 2022-04-22T18:06:22.355 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-974D91-RknZFpRu off=9785114 len=135386 2022-04-22T18:06:22.358 INFO:tasks.workunit.client.0.smithi115.stdout:op 5876 completed, throughput=5MB/sec 2022-04-22T18:06:22.358 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_-tr_NnPrvFv7sw off=1261608 len=970591 2022-04-22T18:06:22.364 INFO:tasks.workunit.client.0.smithi115.stdout:op 5877 completed, throughput=5MB/sec 2022-04-22T18:06:22.364 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-q1DESoGKnHc4hWL off=454126 len=675745 2022-04-22T18:06:22.370 INFO:tasks.workunit.client.0.smithi115.stdout:op 5878 completed, throughput=5MB/sec 2022-04-22T18:06:22.370 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-y8kaq-LpnTBBgOr off=998152 len=621182 2022-04-22T18:06:22.372 INFO:tasks.workunit.client.0.smithi115.stdout:op 5879 completed, throughput=5MB/sec 2022-04-22T18:06:22.372 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-YtxIX-ISYRnrH0r off=2291529 len=403496 2022-04-22T18:06:22.376 INFO:tasks.workunit.client.0.smithi115.stdout:op 5880 completed, throughput=5MB/sec 2022-04-22T18:06:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:22 smithi121 bash[36902]: cluster 2022-04-22T18:06:20.885210+0000 mgr.y (mgr.34107) 1423 : cluster [DBG] pgmap v1989: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:06:22.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:22 smithi115 bash[45495]: cluster 2022-04-22T18:06:20.885210+0000 mgr.y (mgr.34107) 1423 : cluster [DBG] pgmap v1989: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:06:22.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:22 smithi115 bash[46934]: cluster 2022-04-22T18:06:20.885210+0000 mgr.y (mgr.34107) 1423 : cluster [DBG] pgmap v1989: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2022-04-22T18:06:23.377 INFO:tasks.workunit.client.0.smithi115.stdout: 1151: throughput=5MB/sec pending data=0 2022-04-22T18:06:23.377 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-tmiCr8AuNHyz6YB off=8832509 len=1047091 2022-04-22T18:06:23.381 INFO:tasks.workunit.client.0.smithi115.stdout:op 5881 completed, throughput=5MB/sec 2022-04-22T18:06:23.381 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-ygq1GauAhFVmhBr off=2929509 len=1899841 2022-04-22T18:06:23.391 INFO:tasks.workunit.client.0.smithi115.stdout:op 5882 completed, throughput=5MB/sec 2022-04-22T18:06:23.391 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-oqry-DIQlJBrlnl off=5752491 len=1612201 2022-04-22T18:06:23.395 INFO:tasks.workunit.client.0.smithi115.stdout:op 5883 completed, throughput=5MB/sec 2022-04-22T18:06:23.395 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Z9eT8J1vhOe7v21 off=3155830 len=88061 2022-04-22T18:06:23.399 INFO:tasks.workunit.client.0.smithi115.stdout:op 5884 completed, throughput=5MB/sec 2022-04-22T18:06:23.399 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-pm-p0kFWSg3e0Ng off=4221506 len=1968553 2022-04-22T18:06:23.408 INFO:tasks.workunit.client.0.smithi115.stdout:op 5885 completed, throughput=5MB/sec 2022-04-22T18:06:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:23 smithi121 bash[36902]: audit 2022-04-22T18:06:22.879261+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:23 smithi115 bash[45495]: audit 2022-04-22T18:06:22.879261+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:23.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:23 smithi115 bash[46934]: audit 2022-04-22T18:06:22.879261+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:23.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:24.409 INFO:tasks.workunit.client.0.smithi115.stdout: 1152: throughput=5MB/sec pending data=0 2022-04-22T18:06:24.420 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-fQ6TXUeC9rjjdM- off=13701753 len=20862 2022-04-22T18:06:24.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 5886 completed, throughput=5MB/sec 2022-04-22T18:06:24.421 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TDcUsRx-VKfOJVp off=1406747 len=1372376 2022-04-22T18:06:24.421 INFO:tasks.workunit.client.0.smithi115.stdout:op 5887 completed, throughput=5MB/sec 2022-04-22T18:06:24.422 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-LHYh0Vc6UwIWyl8 off=6815449 len=1898338 2022-04-22T18:06:24.422 INFO:tasks.workunit.client.0.smithi115.stdout:op 5888 completed, throughput=5MB/sec 2022-04-22T18:06:24.422 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-2HisR9MLBVUiSj2 off=8127236 len=1756237 2022-04-22T18:06:24.428 INFO:tasks.workunit.client.0.smithi115.stdout:op 5889 completed, throughput=5MB/sec 2022-04-22T18:06:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:24 smithi121 bash[36902]: cluster 2022-04-22T18:06:22.886132+0000 mgr.y (mgr.34107) 1424 : cluster [DBG] pgmap v1990: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:24 smithi115 bash[45495]: cluster 2022-04-22T18:06:22.886132+0000 mgr.y (mgr.34107) 1424 : cluster [DBG] pgmap v1990: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:24 smithi115 bash[46934]: cluster 2022-04-22T18:06:22.886132+0000 mgr.y (mgr.34107) 1424 : cluster [DBG] pgmap v1990: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 4 op/s 2022-04-22T18:06:25.428 INFO:tasks.workunit.client.0.smithi115.stdout: 1153: throughput=5MB/sec pending data=0 2022-04-22T18:06:25.429 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Ix_EdvPaVXUtuU7 off=3545258 len=1596142 2022-04-22T18:06:25.437 INFO:tasks.workunit.client.0.smithi115.stdout:op 5890 completed, throughput=5MB/sec 2022-04-22T18:06:25.437 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Z1uNMWP5Co5qZJd off=9526426 len=360564 2022-04-22T18:06:25.439 INFO:tasks.workunit.client.0.smithi115.stdout:op 5891 completed, throughput=5MB/sec 2022-04-22T18:06:25.439 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BE-3QbCCqAPDOz9 off=12443518 len=1200988 2022-04-22T18:06:25.447 INFO:tasks.workunit.client.0.smithi115.stdout:op 5892 completed, throughput=5MB/sec 2022-04-22T18:06:25.447 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-lFud9O2TOSlP_0D off=6790025 len=1678234 2022-04-22T18:06:25.449 INFO:tasks.workunit.client.0.smithi115.stdout:op 5893 completed, throughput=5MB/sec 2022-04-22T18:06:26.449 INFO:tasks.workunit.client.0.smithi115.stdout: 1154: throughput=5MB/sec pending data=0 2022-04-22T18:06:26.449 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-tfb43KZcdVJ0c2T off=9286253 len=1275558 2022-04-22T18:06:26.456 INFO:tasks.workunit.client.0.smithi115.stdout:op 5894 completed, throughput=5MB/sec 2022-04-22T18:06:26.456 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7q6N0n7kB-0Pcsk off=9844259 len=1544750 2022-04-22T18:06:26.459 INFO:tasks.workunit.client.0.smithi115.stdout:op 5895 completed, throughput=5MB/sec 2022-04-22T18:06:26.459 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X9M6FEXeSCd7xhx off=1432708 len=1740077 2022-04-22T18:06:26.464 INFO:tasks.workunit.client.0.smithi115.stdout:op 5896 completed, throughput=5MB/sec 2022-04-22T18:06:26.465 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-szEoAo6S_gw11Jp off=1041213 len=917406 2022-04-22T18:06:26.466 INFO:tasks.workunit.client.0.smithi115.stdout:op 5897 completed, throughput=5MB/sec 2022-04-22T18:06:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:26 smithi121 bash[36902]: cluster 2022-04-22T18:06:24.887234+0000 mgr.y (mgr.34107) 1425 : cluster [DBG] pgmap v1991: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:26 smithi121 bash[36902]: audit 2022-04-22T18:06:25.491546+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:26 smithi121 bash[36902]: audit 2022-04-22T18:06:25.904156+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:26.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[45495]: cluster 2022-04-22T18:06:24.887234+0000 mgr.y (mgr.34107) 1425 : cluster [DBG] pgmap v1991: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:26.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[45495]: audit 2022-04-22T18:06:25.491546+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:26.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[45495]: audit 2022-04-22T18:06:25.904156+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:26.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[46934]: cluster 2022-04-22T18:06:24.887234+0000 mgr.y (mgr.34107) 1425 : cluster [DBG] pgmap v1991: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:26.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[46934]: audit 2022-04-22T18:06:25.491546+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:26 smithi115 bash[46934]: audit 2022-04-22T18:06:25.904156+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:27.467 INFO:tasks.workunit.client.0.smithi115.stdout: 1155: throughput=5MB/sec pending data=0 2022-04-22T18:06:27.467 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-vozomCMeGF1NRz2 off=20824995 len=654725 2022-04-22T18:06:27.472 INFO:tasks.workunit.client.0.smithi115.stdout:op 5898 completed, throughput=5MB/sec 2022-04-22T18:06:27.473 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HszeawJT-iftbup off=3551811 len=1947219 2022-04-22T18:06:27.477 INFO:tasks.workunit.client.0.smithi115.stdout:op 5899 completed, throughput=5MB/sec 2022-04-22T18:06:27.477 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0bmFcDRWj3z3yEx off=1528032 len=740282 2022-04-22T18:06:27.480 INFO:tasks.workunit.client.0.smithi115.stdout:op 5900 completed, throughput=5MB/sec 2022-04-22T18:06:27.480 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-XIF-RvkeyqsaIP1 off=106102 len=892198 2022-04-22T18:06:27.486 INFO:tasks.workunit.client.0.smithi115.stdout:op 5901 completed, throughput=5MB/sec 2022-04-22T18:06:27.487 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-H7EA1lRYuJeCByJ off=8373815 len=45055 2022-04-22T18:06:27.490 INFO:tasks.workunit.client.0.smithi115.stdout:op 5902 completed, throughput=5MB/sec 2022-04-22T18:06:27.490 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-woK5CXQ-7GH-4aY off=11425086 len=1007444 2022-04-22T18:06:27.493 INFO:tasks.workunit.client.0.smithi115.stdout:op 5903 completed, throughput=5MB/sec 2022-04-22T18:06:28.493 INFO:tasks.workunit.client.0.smithi115.stdout: 1156: throughput=5MB/sec pending data=0 2022-04-22T18:06:28.493 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-ygS-1wmYhucXDJV off=6872332 len=1052233 2022-04-22T18:06:28.496 INFO:tasks.workunit.client.0.smithi115.stdout:op 5904 completed, throughput=5MB/sec 2022-04-22T18:06:28.496 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-A0JWwNiMYKfiw-h off=15343999 len=825674 2022-04-22T18:06:28.498 INFO:tasks.workunit.client.0.smithi115.stdout:op 5905 completed, throughput=5MB/sec 2022-04-22T18:06:28.498 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-_nC2PBm1t9jEeVN off=7443963 len=2055699 2022-04-22T18:06:28.506 INFO:tasks.workunit.client.0.smithi115.stdout:op 5906 completed, throughput=5MB/sec 2022-04-22T18:06:28.506 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-8cNz5J46L_NrwaX off=2957166 len=107001 2022-04-22T18:06:28.507 INFO:tasks.workunit.client.0.smithi115.stdout:op 5907 completed, throughput=5MB/sec 2022-04-22T18:06:28.507 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TVaLFgnEx1Ey7tJ off=743772 len=1525451 2022-04-22T18:06:28.515 INFO:tasks.workunit.client.0.smithi115.stdout:op 5908 completed, throughput=5MB/sec 2022-04-22T18:06:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:28 smithi121 bash[36902]: cluster 2022-04-22T18:06:26.888372+0000 mgr.y (mgr.34107) 1426 : cluster [DBG] pgmap v1992: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:28 smithi121 bash[36902]: audit 2022-04-22T18:06:27.882212+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:28 smithi115 bash[45495]: cluster 2022-04-22T18:06:26.888372+0000 mgr.y (mgr.34107) 1426 : cluster [DBG] pgmap v1992: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:28 smithi115 bash[45495]: audit 2022-04-22T18:06:27.882212+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:28 smithi115 bash[46934]: cluster 2022-04-22T18:06:26.888372+0000 mgr.y (mgr.34107) 1426 : cluster [DBG] pgmap v1992: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 5 op/s 2022-04-22T18:06:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:28 smithi115 bash[46934]: audit 2022-04-22T18:06:27.882212+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:29.516 INFO:tasks.workunit.client.0.smithi115.stdout: 1157: throughput=5MB/sec pending data=0 2022-04-22T18:06:29.516 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Gr5_Ip04CyYhglD off=5528614 len=1853276 2022-04-22T18:06:29.520 INFO:tasks.workunit.client.0.smithi115.stdout:op 5909 completed, throughput=5MB/sec 2022-04-22T18:06:29.521 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_HWgRhjZFd_Fptp off=3908453 len=1805482 2022-04-22T18:06:29.525 INFO:tasks.workunit.client.0.smithi115.stdout:op 5910 completed, throughput=5MB/sec 2022-04-22T18:06:29.525 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--zsl78pGQZ6PrAA off=20622412 len=991262 2022-04-22T18:06:29.527 INFO:tasks.workunit.client.0.smithi115.stdout:op 5911 completed, throughput=5MB/sec 2022-04-22T18:06:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:30 smithi121 bash[36902]: cluster 2022-04-22T18:06:28.889556+0000 mgr.y (mgr.34107) 1427 : cluster [DBG] pgmap v1993: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:06:30.527 INFO:tasks.workunit.client.0.smithi115.stdout: 1158: throughput=5MB/sec pending data=0 2022-04-22T18:06:30.527 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nc5W2DXx4Na-dsh off=149934 len=510745 2022-04-22T18:06:30.533 INFO:tasks.workunit.client.0.smithi115.stdout:op 5912 completed, throughput=5MB/sec 2022-04-22T18:06:30.533 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3NY0tBx7ahTZ5NM off=3069291 len=532778 2022-04-22T18:06:30.539 INFO:tasks.workunit.client.0.smithi115.stdout:op 5913 completed, throughput=5MB/sec 2022-04-22T18:06:30.539 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-CZPyZTyOwEHLfk0 off=4380287 len=562320 2022-04-22T18:06:30.541 INFO:tasks.workunit.client.0.smithi115.stdout:op 5914 completed, throughput=5MB/sec 2022-04-22T18:06:30.541 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xPUmNNJDWRoB2l- off=94028 len=158994 2022-04-22T18:06:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:30 smithi115 bash[45495]: cluster 2022-04-22T18:06:28.889556+0000 mgr.y (mgr.34107) 1427 : cluster [DBG] pgmap v1993: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:06:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:30 smithi115 bash[46934]: cluster 2022-04-22T18:06:28.889556+0000 mgr.y (mgr.34107) 1427 : cluster [DBG] pgmap v1993: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 5 op/s 2022-04-22T18:06:30.545 INFO:tasks.workunit.client.0.smithi115.stdout:op 5915 completed, throughput=5MB/sec 2022-04-22T18:06:30.545 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-YFOBh0B9UjL2OZx off=873432 len=1282784 2022-04-22T18:06:30.547 INFO:tasks.workunit.client.0.smithi115.stdout:op 5916 completed, throughput=5MB/sec 2022-04-22T18:06:30.547 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-zviGFnHMKGF7Nbb off=399994 len=55680 2022-04-22T18:06:30.548 INFO:tasks.workunit.client.0.smithi115.stdout:op 5917 completed, throughput=5MB/sec 2022-04-22T18:06:30.548 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-WZK4JxsDH98q3Jt off=1780607 len=1178820 2022-04-22T18:06:30.552 INFO:tasks.workunit.client.0.smithi115.stdout:op 5918 completed, throughput=5MB/sec 2022-04-22T18:06:30.552 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-1G2Cti-mOq8pp6p off=188489 len=364756 2022-04-22T18:06:30.553 INFO:tasks.workunit.client.0.smithi115.stdout:op 5919 completed, throughput=5MB/sec 2022-04-22T18:06:30.553 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xHKRecAqCJqIsdI off=15789520 len=585448 2022-04-22T18:06:30.558 INFO:tasks.workunit.client.0.smithi115.stdout:op 5920 completed, throughput=5MB/sec 2022-04-22T18:06:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:31 smithi121 bash[36902]: audit 2022-04-22T18:06:30.494553+0000 mon.a (mon.0) 2990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:31.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:31 smithi121 bash[36902]: audit 2022-04-22T18:06:30.908092+0000 mon.a (mon.0) 2991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:31 smithi115 bash[45495]: audit 2022-04-22T18:06:30.494553+0000 mon.a (mon.0) 2990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:31 smithi115 bash[45495]: audit 2022-04-22T18:06:30.908092+0000 mon.a (mon.0) 2991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:31 smithi115 bash[46934]: audit 2022-04-22T18:06:30.494553+0000 mon.a (mon.0) 2990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:31 smithi115 bash[46934]: audit 2022-04-22T18:06:30.908092+0000 mon.a (mon.0) 2991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:31.558 INFO:tasks.workunit.client.0.smithi115.stdout: 1159: throughput=5MB/sec pending data=0 2022-04-22T18:06:31.559 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DGBVi1v009nh_Pp off=3095728 len=383586 2022-04-22T18:06:31.564 INFO:tasks.workunit.client.0.smithi115.stdout:op 5921 completed, throughput=5MB/sec 2022-04-22T18:06:31.564 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-OmF0UKXDWMTxNZc off=154699 len=665470 2022-04-22T18:06:31.565 INFO:tasks.workunit.client.0.smithi115.stdout:op 5922 completed, throughput=5MB/sec 2022-04-22T18:06:31.566 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7WgTrDk98pBfb8t off=663025 len=507710 2022-04-22T18:06:31.568 INFO:tasks.workunit.client.0.smithi115.stdout:op 5923 completed, throughput=5MB/sec 2022-04-22T18:06:31.568 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Mu0A0EQ6RMGJ5Wa off=7310849 len=718713 2022-04-22T18:06:31.570 INFO:tasks.workunit.client.0.smithi115.stdout:op 5924 completed, throughput=5MB/sec 2022-04-22T18:06:31.570 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7Zf4W4AMdhkQ-bl off=1387934 len=1577109 2022-04-22T18:06:31.573 INFO:tasks.workunit.client.0.smithi115.stdout:op 5925 completed, throughput=5MB/sec 2022-04-22T18:06:31.573 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-n_ey1_YWtuT-jHm off=1072794 len=1774138 2022-04-22T18:06:31.580 INFO:tasks.workunit.client.0.smithi115.stdout:op 5926 completed, throughput=5MB/sec 2022-04-22T18:06:32.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:31.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:32 smithi121 bash[36902]: cluster 2022-04-22T18:06:30.890704+0000 mgr.y (mgr.34107) 1428 : cluster [DBG] pgmap v1994: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:06:32.621 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:32 smithi115 bash[45495]: cluster 2022-04-22T18:06:30.890704+0000 mgr.y (mgr.34107) 1428 : cluster [DBG] pgmap v1994: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:06:32.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:32 smithi115 bash[46934]: cluster 2022-04-22T18:06:30.890704+0000 mgr.y (mgr.34107) 1428 : cluster [DBG] pgmap v1994: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 5 op/s 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout: 1160: throughput=5MB/sec pending data=0 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-n9JBFXQ-SREfNqo off=1553127 len=1256953 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 5927 completed, throughput=5MB/sec 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dnG-w340oL42iqM off=19985294 len=1098698 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 5928 completed, throughput=5MB/sec 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-iIXtWgdqFI7lF7W off=2436244 len=431550 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:op 5929 completed, throughput=5MB/sec 2022-04-22T18:06:32.624 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-Db_C1S6wiGCCnxH off=14146047 len=218591 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5930 completed, throughput=5MB/sec 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-X39r-WtGViOqDlG off=3559432 len=620209 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5931 completed, throughput=5MB/sec 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NZvUN7-cwZiXTwB off=6048665 len=71620 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5932 completed, throughput=5MB/sec 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TroYF3FeBJcEVZG off=544956 len=1140517 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5933 completed, throughput=5MB/sec 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TWJAzU76ey0i7ag off=7619223 len=552221 2022-04-22T18:06:32.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5934 completed, throughput=5MB/sec 2022-04-22T18:06:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:33 smithi121 bash[36902]: audit 2022-04-22T18:06:32.885237+0000 mon.a (mon.0) 2992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:33 smithi115 bash[45495]: audit 2022-04-22T18:06:32.885237+0000 mon.a (mon.0) 2992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:33 smithi115 bash[46934]: audit 2022-04-22T18:06:32.885237+0000 mon.a (mon.0) 2992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:33.607 INFO:tasks.workunit.client.0.smithi115.stdout: 1161: throughput=5MB/sec pending data=0 2022-04-22T18:06:33.607 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wClaUkB73AU_W6N off=9661614 len=554784 2022-04-22T18:06:33.613 INFO:tasks.workunit.client.0.smithi115.stdout:op 5935 completed, throughput=5MB/sec 2022-04-22T18:06:33.614 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-cr19R67Puzpyu1v off=12161279 len=70975 2022-04-22T18:06:33.615 INFO:tasks.workunit.client.0.smithi115.stdout:op 5936 completed, throughput=5MB/sec 2022-04-22T18:06:33.615 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-L74OogsvLurszB1 off=3327230 len=1033722 2022-04-22T18:06:33.622 INFO:tasks.workunit.client.0.smithi115.stdout:op 5937 completed, throughput=5MB/sec 2022-04-22T18:06:33.622 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-0BK0uU1PWXkUPRB off=17180453 len=1799195 2022-04-22T18:06:33.625 INFO:tasks.workunit.client.0.smithi115.stdout:op 5938 completed, throughput=5MB/sec 2022-04-22T18:06:33.625 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-aMmlCefbfMkZjhJ off=11779238 len=963498 2022-04-22T18:06:33.627 INFO:tasks.workunit.client.0.smithi115.stdout:op 5939 completed, throughput=5MB/sec 2022-04-22T18:06:33.627 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-b0laMTcuv6tLPbq off=12314898 len=1404594 2022-04-22T18:06:33.630 INFO:tasks.workunit.client.0.smithi115.stdout:op 5940 completed, throughput=5MB/sec 2022-04-22T18:06:34.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:33.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:34 smithi121 bash[36902]: cluster 2022-04-22T18:06:32.891695+0000 mgr.y (mgr.34107) 1429 : cluster [DBG] pgmap v1995: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:06:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:34 smithi115 bash[46934]: cluster 2022-04-22T18:06:32.891695+0000 mgr.y (mgr.34107) 1429 : cluster [DBG] pgmap v1995: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:06:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:34 smithi115 bash[45495]: cluster 2022-04-22T18:06:32.891695+0000 mgr.y (mgr.34107) 1429 : cluster [DBG] pgmap v1995: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2022-04-22T18:06:34.630 INFO:tasks.workunit.client.0.smithi115.stdout: 1162: throughput=5MB/sec pending data=0 2022-04-22T18:06:34.630 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-wawP6qpQnTlrbvV off=10044520 len=161802 2022-04-22T18:06:34.634 INFO:tasks.workunit.client.0.smithi115.stdout:op 5941 completed, throughput=5MB/sec 2022-04-22T18:06:34.634 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-xShHAB1uAcSQH16 off=1670217 len=905061 2022-04-22T18:06:34.636 INFO:tasks.workunit.client.0.smithi115.stdout:op 5942 completed, throughput=5MB/sec 2022-04-22T18:06:34.636 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-KXrCLTehtT0GD-p off=2740539 len=1110866 2022-04-22T18:06:34.638 INFO:tasks.workunit.client.0.smithi115.stdout:op 5943 completed, throughput=5MB/sec 2022-04-22T18:06:34.639 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gFot79or9fmDH3E off=2772344 len=1967082 2022-04-22T18:06:34.641 INFO:tasks.workunit.client.0.smithi115.stdout:op 5944 completed, throughput=5MB/sec 2022-04-22T18:06:34.641 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-BFzESYEu-4NmPte off=1152675 len=535625 2022-04-22T18:06:34.642 INFO:tasks.workunit.client.0.smithi115.stdout:op 5945 completed, throughput=5MB/sec 2022-04-22T18:06:34.647 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (29m) 8m ago 43m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (43m) 9m ago 43m 25.0M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (31m) 8m ago 44m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (31m) 9m ago 44m 15.2M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (38m) 9m ago 48m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (39m) 8m ago 50m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:06:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (38m) 8m ago 51m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (37m) 9m ago 49m 91.9M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (37m) 8m ago 49m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (30m) 8m ago 43m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (30m) 9m ago 43m 9595k - 1dbe0e931976 f4b044642a96 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (36m) 8m ago 48m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (35m) 8m ago 47m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (35m) 8m ago 47m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (34m) 8m ago 46m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (34m) 9m ago 46m 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:06:34.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (33m) 9m ago 45m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:06:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (32m) 9m ago 45m 379M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:06:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (32m) 9m ago 45m 475M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:06:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (30m) 9m ago 43m 51.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:06:35.384 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:06:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:06:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:06:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:06:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:06:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:06:35.386 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:06:35.643 INFO:tasks.workunit.client.0.smithi115.stdout: 1163: throughput=5MB/sec pending data=0 2022-04-22T18:06:35.643 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-dzOO-jTxHJHS5HC off=10394929 len=816538 2022-04-22T18:06:35.645 INFO:tasks.workunit.client.0.smithi115.stdout:op 5946 completed, throughput=5MB/sec 2022-04-22T18:06:35.645 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M-NY2749pY7GowH off=15200256 len=620775 2022-04-22T18:06:35.652 INFO:tasks.workunit.client.0.smithi115.stdout:op 5947 completed, throughput=5MB/sec 2022-04-22T18:06:35.652 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-G8WCRxzbdTn_H-O off=7994318 len=1538851 2022-04-22T18:06:35.660 INFO:tasks.workunit.client.0.smithi115.stdout:op 5948 completed, throughput=5MB/sec 2022-04-22T18:06:35.660 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-GC66eNCVCRnkb0N off=7279514 len=975209 2022-04-22T18:06:35.662 INFO:tasks.workunit.client.0.smithi115.stdout:op 5949 completed, throughput=5MB/sec 2022-04-22T18:06:35.662 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj--RKNGzjWqgZ3i9M off=15783690 len=75738 2022-04-22T18:06:35.663 INFO:tasks.workunit.client.0.smithi115.stdout:op 5950 completed, throughput=5MB/sec 2022-04-22T18:06:35.663 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-cXKbF-a_C8d_xee off=1278752 len=1860854 2022-04-22T18:06:35.672 INFO:tasks.workunit.client.0.smithi115.stdout:op 5951 completed, throughput=5MB/sec 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: audit 2022-04-22T18:06:34.623744+0000 mgr.y (mgr.34107) 1430 : audit [DBG] from='client.45835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: cluster 2022-04-22T18:06:34.892791+0000 mgr.y (mgr.34107) 1431 : cluster [DBG] pgmap v1996: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: audit 2022-04-22T18:06:34.962428+0000 mgr.y (mgr.34107) 1432 : audit [DBG] from='client.35901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: audit 2022-04-22T18:06:35.378432+0000 mon.a (mon.0) 2993 : audit [DBG] from='client.? 172.21.15.115:0/3219427324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: audit 2022-04-22T18:06:35.497830+0000 mon.a (mon.0) 2994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:36 smithi121 bash[36902]: audit 2022-04-22T18:06:35.911832+0000 mon.a (mon.0) 2995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: audit 2022-04-22T18:06:34.623744+0000 mgr.y (mgr.34107) 1430 : audit [DBG] from='client.45835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: cluster 2022-04-22T18:06:34.892791+0000 mgr.y (mgr.34107) 1431 : cluster [DBG] pgmap v1996: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:06:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: audit 2022-04-22T18:06:34.962428+0000 mgr.y (mgr.34107) 1432 : audit [DBG] from='client.35901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: audit 2022-04-22T18:06:35.378432+0000 mon.a (mon.0) 2993 : audit [DBG] from='client.? 172.21.15.115:0/3219427324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: audit 2022-04-22T18:06:35.497830+0000 mon.a (mon.0) 2994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[45495]: audit 2022-04-22T18:06:35.911832+0000 mon.a (mon.0) 2995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: audit 2022-04-22T18:06:34.623744+0000 mgr.y (mgr.34107) 1430 : audit [DBG] from='client.45835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: cluster 2022-04-22T18:06:34.892791+0000 mgr.y (mgr.34107) 1431 : cluster [DBG] pgmap v1996: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 5 op/s 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: audit 2022-04-22T18:06:34.962428+0000 mgr.y (mgr.34107) 1432 : audit [DBG] from='client.35901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: audit 2022-04-22T18:06:35.378432+0000 mon.a (mon.0) 2993 : audit [DBG] from='client.? 172.21.15.115:0/3219427324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: audit 2022-04-22T18:06:35.497830+0000 mon.a (mon.0) 2994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:36 smithi115 bash[46934]: audit 2022-04-22T18:06:35.911832+0000 mon.a (mon.0) 2995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:36.673 INFO:tasks.workunit.client.0.smithi115.stdout: 1164: throughput=5MB/sec pending data=0 2022-04-22T18:06:36.673 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-dA8pk8HWDFc-KZS off=2479282 len=1155620 2022-04-22T18:06:36.680 INFO:tasks.workunit.client.0.smithi115.stdout:op 5952 completed, throughput=5MB/sec 2022-04-22T18:06:36.680 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-iwLS5IUoBqP8zQI off=3432781 len=781068 2022-04-22T18:06:36.687 INFO:tasks.workunit.client.0.smithi115.stdout:op 5953 completed, throughput=5MB/sec 2022-04-22T18:06:36.687 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-RizY67nFxGRhNYC off=4011123 len=1996186 2022-04-22T18:06:36.690 INFO:tasks.workunit.client.0.smithi115.stdout:op 5954 completed, throughput=5MB/sec 2022-04-22T18:06:36.690 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kYZso-bKUeTACDr off=14068501 len=610642 2022-04-22T18:06:36.695 INFO:tasks.workunit.client.0.smithi115.stdout:op 5955 completed, throughput=5MB/sec 2022-04-22T18:06:36.695 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-seoCjHl5m0BpnCT off=12530161 len=473770 2022-04-22T18:06:36.700 INFO:tasks.workunit.client.0.smithi115.stdout:op 5956 completed, throughput=5MB/sec 2022-04-22T18:06:37.700 INFO:tasks.workunit.client.0.smithi115.stdout: 1165: throughput=5MB/sec pending data=0 2022-04-22T18:06:37.701 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kH8AGyD8EKu0jzW off=1751343 len=482495 2022-04-22T18:06:37.707 INFO:tasks.workunit.client.0.smithi115.stdout:op 5957 completed, throughput=5MB/sec 2022-04-22T18:06:37.707 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-7sbbhZB1HE_Hm1H off=9589346 len=342125 2022-04-22T18:06:37.708 INFO:tasks.workunit.client.0.smithi115.stdout:op 5958 completed, throughput=5MB/sec 2022-04-22T18:06:37.708 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-po2G7mdDUNocHOM off=16286344 len=1421991 2022-04-22T18:06:37.716 INFO:tasks.workunit.client.0.smithi115.stdout:op 5959 completed, throughput=5MB/sec 2022-04-22T18:06:37.758 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-e92pvPKgcv6uLzF off=1594683 len=1786429 2022-04-22T18:06:37.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 5960 completed, throughput=5MB/sec 2022-04-22T18:06:37.759 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-bx7860KhfTThHvK off=15585826 len=1344715 2022-04-22T18:06:37.759 INFO:tasks.workunit.client.0.smithi115.stdout:op 5961 completed, throughput=5MB/sec 2022-04-22T18:06:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:38 smithi121 bash[36902]: cluster 2022-04-22T18:06:36.893854+0000 mgr.y (mgr.34107) 1433 : cluster [DBG] pgmap v1997: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:06:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:38 smithi121 bash[36902]: audit 2022-04-22T18:06:37.888612+0000 mon.a (mon.0) 2996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:38 smithi115 bash[45495]: cluster 2022-04-22T18:06:36.893854+0000 mgr.y (mgr.34107) 1433 : cluster [DBG] pgmap v1997: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:06:38.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:38 smithi115 bash[45495]: audit 2022-04-22T18:06:37.888612+0000 mon.a (mon.0) 2996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:38.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:38 smithi115 bash[46934]: cluster 2022-04-22T18:06:36.893854+0000 mgr.y (mgr.34107) 1433 : cluster [DBG] pgmap v1997: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.0 MiB/s wr, 5 op/s 2022-04-22T18:06:38.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:38 smithi115 bash[46934]: audit 2022-04-22T18:06:37.888612+0000 mon.a (mon.0) 2996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:38.724 INFO:tasks.workunit.client.0.smithi115.stdout: 1166: throughput=5MB/sec pending data=0 2022-04-22T18:06:38.724 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-4OaJKYU80WOHZQ7 off=706875 len=854650 2022-04-22T18:06:38.732 INFO:tasks.workunit.client.0.smithi115.stdout:op 5962 completed, throughput=5MB/sec 2022-04-22T18:06:38.732 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-C6LdihLmPdadJW8 off=20309141 len=1220587 2022-04-22T18:06:38.735 INFO:tasks.workunit.client.0.smithi115.stdout:op 5963 completed, throughput=5MB/sec 2022-04-22T18:06:38.735 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-jfRI2e386jUkX8p off=10590770 len=850737 2022-04-22T18:06:38.738 INFO:tasks.workunit.client.0.smithi115.stdout:op 5964 completed, throughput=5MB/sec 2022-04-22T18:06:38.738 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-aQ709jODHVQ77dn off=3076748 len=1209282 2022-04-22T18:06:38.744 INFO:tasks.workunit.client.0.smithi115.stdout:op 5965 completed, throughput=5MB/sec 2022-04-22T18:06:38.744 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-HeUHJdXuqIAS4RY off=129195 len=1423252 2022-04-22T18:06:38.746 INFO:tasks.workunit.client.0.smithi115.stdout:op 5966 completed, throughput=5MB/sec 2022-04-22T18:06:39.747 INFO:tasks.workunit.client.0.smithi115.stdout: 1167: throughput=5MB/sec pending data=0 2022-04-22T18:06:39.747 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-gkVVmrxTVpPrnNK off=2733126 len=711813 2022-04-22T18:06:39.753 INFO:tasks.workunit.client.0.smithi115.stdout:op 5967 completed, throughput=5MB/sec 2022-04-22T18:06:39.753 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-6AxA8bAh0KunFU2 off=19903030 len=904413 2022-04-22T18:06:39.756 INFO:tasks.workunit.client.0.smithi115.stdout:op 5968 completed, throughput=5MB/sec 2022-04-22T18:06:39.756 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-NF9cOMrWBZQ8Rpp off=2341089 len=410440 2022-04-22T18:06:39.758 INFO:tasks.workunit.client.0.smithi115.stdout:op 5969 completed, throughput=5MB/sec 2022-04-22T18:06:39.758 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-fk105pgv1nDfsx2 off=1379258 len=447363 2022-04-22T18:06:39.763 INFO:tasks.workunit.client.0.smithi115.stdout:op 5970 completed, throughput=5MB/sec 2022-04-22T18:06:39.763 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-w5DOdTpNHpmgvJn off=10825813 len=56625 2022-04-22T18:06:39.764 INFO:tasks.workunit.client.0.smithi115.stdout:op 5971 completed, throughput=5MB/sec 2022-04-22T18:06:39.764 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-Az45pnObxdYoeKS off=7958984 len=758736 2022-04-22T18:06:39.769 INFO:tasks.workunit.client.0.smithi115.stdout:op 5972 completed, throughput=5MB/sec 2022-04-22T18:06:39.769 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-3Jg3Cen1HjadWJF off=6243584 len=688284 2022-04-22T18:06:39.774 INFO:tasks.workunit.client.0.smithi115.stdout:op 5973 completed, throughput=5MB/sec 2022-04-22T18:06:39.774 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-TeqrnYnDZZ0So2a off=16622537 len=942078 2022-04-22T18:06:39.778 INFO:tasks.workunit.client.0.smithi115.stdout:op 5974 completed, throughput=5MB/sec 2022-04-22T18:06:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:40 smithi121 bash[36902]: cluster 2022-04-22T18:06:38.895114+0000 mgr.y (mgr.34107) 1434 : cluster [DBG] pgmap v1998: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:06:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:40 smithi115 bash[45495]: cluster 2022-04-22T18:06:38.895114+0000 mgr.y (mgr.34107) 1434 : cluster [DBG] pgmap v1998: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:06:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:40 smithi115 bash[46934]: cluster 2022-04-22T18:06:38.895114+0000 mgr.y (mgr.34107) 1434 : cluster [DBG] pgmap v1998: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.7 MiB/s wr, 5 op/s 2022-04-22T18:06:40.779 INFO:tasks.workunit.client.0.smithi115.stdout: 1168: throughput=5MB/sec pending data=0 2022-04-22T18:06:40.779 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-1pm12xYIMlOKoFt off=4615215 len=261857 2022-04-22T18:06:40.784 INFO:tasks.workunit.client.0.smithi115.stdout:op 5975 completed, throughput=5MB/sec 2022-04-22T18:06:40.784 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-DbdMz9pYKxI9PLz off=14593319 len=667813 2022-04-22T18:06:40.790 INFO:tasks.workunit.client.0.smithi115.stdout:op 5976 completed, throughput=5MB/sec 2022-04-22T18:06:40.790 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-zhcLNk850CHmZfb off=5090911 len=772097 2022-04-22T18:06:40.795 INFO:tasks.workunit.client.0.smithi115.stdout:op 5977 completed, throughput=5MB/sec 2022-04-22T18:06:40.795 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-gUaM6qinyr2tsT3 off=2587690 len=116089 2022-04-22T18:06:40.796 INFO:tasks.workunit.client.0.smithi115.stdout:op 5978 completed, throughput=5MB/sec 2022-04-22T18:06:40.796 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-xESxOfdPldW1uzF off=2299616 len=655961 2022-04-22T18:06:40.800 INFO:tasks.workunit.client.0.smithi115.stdout:op 5979 completed, throughput=5MB/sec 2022-04-22T18:06:40.800 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-M1iB-bnVuXDxfh8 off=5909548 len=83336 2022-04-22T18:06:40.804 INFO:tasks.workunit.client.0.smithi115.stdout:op 5980 completed, throughput=5MB/sec 2022-04-22T18:06:40.804 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-P-ElSPQInNCeow8 off=6434977 len=2042723 2022-04-22T18:06:40.807 INFO:tasks.workunit.client.0.smithi115.stdout:op 5981 completed, throughput=5MB/sec 2022-04-22T18:06:40.807 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-SI8j8NusJkbESFG off=5431985 len=212434 2022-04-22T18:06:40.809 INFO:tasks.workunit.client.0.smithi115.stdout:op 5982 completed, throughput=5MB/sec 2022-04-22T18:06:40.809 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-SIcbCxrUE6yZ_PQ off=20618381 len=823545 2022-04-22T18:06:40.816 INFO:tasks.workunit.client.0.smithi115.stdout:op 5983 completed, throughput=5MB/sec 2022-04-22T18:06:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:41 smithi121 bash[36902]: audit 2022-04-22T18:06:40.500819+0000 mon.a (mon.0) 2997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:41 smithi121 bash[36902]: audit 2022-04-22T18:06:40.915363+0000 mon.a (mon.0) 2998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:41 smithi115 bash[45495]: audit 2022-04-22T18:06:40.500819+0000 mon.a (mon.0) 2997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:41 smithi115 bash[45495]: audit 2022-04-22T18:06:40.915363+0000 mon.a (mon.0) 2998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:41 smithi115 bash[46934]: audit 2022-04-22T18:06:40.500819+0000 mon.a (mon.0) 2997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:41 smithi115 bash[46934]: audit 2022-04-22T18:06:40.915363+0000 mon.a (mon.0) 2998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:41.816 INFO:tasks.workunit.client.0.smithi115.stdout: 1169: throughput=5MB/sec pending data=0 2022-04-22T18:06:41.816 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-UHLs6CHhKSGPGcs off=1628702 len=1485443 2022-04-22T18:06:41.819 INFO:tasks.workunit.client.0.smithi115.stdout:op 5984 completed, throughput=5MB/sec 2022-04-22T18:06:41.819 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-BMLYny8h-QqTj-t off=1749150 len=1767402 2022-04-22T18:06:41.826 INFO:tasks.workunit.client.0.smithi115.stdout:op 5985 completed, throughput=5MB/sec 2022-04-22T18:06:41.826 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-W-fPME67Za2lEVt off=86706 len=106214 2022-04-22T18:06:41.831 INFO:tasks.workunit.client.0.smithi115.stdout:op 5986 completed, throughput=5MB/sec 2022-04-22T18:06:41.831 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-b30dTATPC-lcFAO off=922075 len=708786 2022-04-22T18:06:41.836 INFO:tasks.workunit.client.0.smithi115.stdout:op 5987 completed, throughput=5MB/sec 2022-04-22T18:06:41.836 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-TBP5LldiZEr6Za8 off=4880957 len=1848797 2022-04-22T18:06:41.841 INFO:tasks.workunit.client.0.smithi115.stdout:op 5988 completed, throughput=5MB/sec 2022-04-22T18:06:42.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:41.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:42 smithi121 bash[36902]: cluster 2022-04-22T18:06:40.896402+0000 mgr.y (mgr.34107) 1435 : cluster [DBG] pgmap v1999: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2022-04-22T18:06:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:42 smithi115 bash[45495]: cluster 2022-04-22T18:06:40.896402+0000 mgr.y (mgr.34107) 1435 : cluster [DBG] pgmap v1999: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2022-04-22T18:06:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:42 smithi115 bash[46934]: cluster 2022-04-22T18:06:40.896402+0000 mgr.y (mgr.34107) 1435 : cluster [DBG] pgmap v1999: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2022-04-22T18:06:42.841 INFO:tasks.workunit.client.0.smithi115.stdout: 1170: throughput=5MB/sec pending data=0 2022-04-22T18:06:42.842 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-kEdNu3vmsFglo4O off=7109165 len=677735 2022-04-22T18:06:42.848 INFO:tasks.workunit.client.0.smithi115.stdout:op 5989 completed, throughput=5MB/sec 2022-04-22T18:06:42.848 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-JpGWWvqhY9WvBiq off=11877137 len=555877 2022-04-22T18:06:42.851 INFO:tasks.workunit.client.0.smithi115.stdout:op 5990 completed, throughput=5MB/sec 2022-04-22T18:06:42.851 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-eh15g_JQqj3agC0 off=8471888 len=1017914 2022-04-22T18:06:42.858 INFO:tasks.workunit.client.0.smithi115.stdout:op 5991 completed, throughput=5MB/sec 2022-04-22T18:06:42.858 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-PzSbZwrmFbkPuoY off=4822264 len=137502 2022-04-22T18:06:42.859 INFO:tasks.workunit.client.0.smithi115.stdout:op 5992 completed, throughput=5MB/sec 2022-04-22T18:06:42.859 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-nCkyWKiX4BByBC4 off=8711870 len=1887294 2022-04-22T18:06:42.869 INFO:tasks.workunit.client.0.smithi115.stdout:op 5993 completed, throughput=5MB/sec 2022-04-22T18:06:42.869 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-MF9F1phXyllPEbL off=4460308 len=521814 2022-04-22T18:06:42.870 INFO:tasks.workunit.client.0.smithi115.stdout:op 5994 completed, throughput=5MB/sec 2022-04-22T18:06:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:43 smithi121 bash[36902]: audit 2022-04-22T18:06:42.891787+0000 mon.a (mon.0) 2999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:43 smithi115 bash[45495]: audit 2022-04-22T18:06:42.891787+0000 mon.a (mon.0) 2999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:43 smithi115 bash[46934]: audit 2022-04-22T18:06:42.891787+0000 mon.a (mon.0) 2999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:43.870 INFO:tasks.workunit.client.0.smithi115.stdout: 1171: throughput=5MB/sec pending data=0 2022-04-22T18:06:43.871 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-mLC1ZyF-g39VJnF off=2937611 len=1296556 2022-04-22T18:06:43.879 INFO:tasks.workunit.client.0.smithi115.stdout:op 5995 completed, throughput=5MB/sec 2022-04-22T18:06:43.879 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-_RE-UccB8phsZNS off=10185784 len=1643841 2022-04-22T18:06:43.882 INFO:tasks.workunit.client.0.smithi115.stdout:op 5996 completed, throughput=5MB/sec 2022-04-22T18:06:43.883 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-FPnqktMe0mvrNFV off=2795345 len=1289385 2022-04-22T18:06:43.885 INFO:tasks.workunit.client.0.smithi115.stdout:op 5997 completed, throughput=5MB/sec 2022-04-22T18:06:43.886 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-VO_mJnHI1UFijXv off=2307178 len=15211 2022-04-22T18:06:43.890 INFO:tasks.workunit.client.0.smithi115.stdout:op 5998 completed, throughput=5MB/sec 2022-04-22T18:06:43.890 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-OdnHUho3hsSEHjd off=7463967 len=243557 2022-04-22T18:06:43.894 INFO:tasks.workunit.client.0.smithi115.stdout:op 5999 completed, throughput=5MB/sec 2022-04-22T18:06:43.894 INFO:tasks.workunit.client.0.smithi115.stdout:WRITE : oid=obj-jE9RyhIkfkPHUKO off=3965177 len=61103 2022-04-22T18:06:43.897 INFO:tasks.workunit.client.0.smithi115.stdout:op 6000 completed, throughput=5MB/sec 2022-04-22T18:06:43.897 INFO:tasks.workunit.client.0.smithi115.stdout:READ : oid=obj-yrU8j_Z6IRl1Tq- off=9078680 len=1864654 2022-04-22T18:06:43.902 INFO:tasks.workunit.client.0.smithi115.stdout:op 6001 completed, throughput=5MB/sec 2022-04-22T18:06:44.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:43.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:44 smithi121 bash[36902]: cluster 2022-04-22T18:06:42.897278+0000 mgr.y (mgr.34107) 1436 : cluster [DBG] pgmap v2000: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:06:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:44 smithi115 bash[45495]: cluster 2022-04-22T18:06:42.897278+0000 mgr.y (mgr.34107) 1436 : cluster [DBG] pgmap v2000: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:06:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:44 smithi115 bash[46934]: cluster 2022-04-22T18:06:42.897278+0000 mgr.y (mgr.34107) 1436 : cluster [DBG] pgmap v2000: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 5 op/s 2022-04-22T18:06:44.902 INFO:tasks.workunit.client.0.smithi115.stdout:waiting for all operations to complete 2022-04-22T18:06:44.902 INFO:tasks.workunit.client.0.smithi115.stdout:cleaning up objects 2022-04-22T18:06:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:46 smithi121 bash[36902]: cluster 2022-04-22T18:06:44.898469+0000 mgr.y (mgr.34107) 1437 : cluster [DBG] pgmap v2001: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2022-04-22T18:06:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:46 smithi121 bash[36902]: audit 2022-04-22T18:06:45.504010+0000 mon.a (mon.0) 3000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:46 smithi121 bash[36902]: audit 2022-04-22T18:06:45.918664+0000 mon.a (mon.0) 3001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[45495]: cluster 2022-04-22T18:06:44.898469+0000 mgr.y (mgr.34107) 1437 : cluster [DBG] pgmap v2001: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2022-04-22T18:06:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[45495]: audit 2022-04-22T18:06:45.504010+0000 mon.a (mon.0) 3000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[45495]: audit 2022-04-22T18:06:45.918664+0000 mon.a (mon.0) 3001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[46934]: cluster 2022-04-22T18:06:44.898469+0000 mgr.y (mgr.34107) 1437 : cluster [DBG] pgmap v2001: 105 pgs: 105 active+clean; 126 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 7 op/s 2022-04-22T18:06:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[46934]: audit 2022-04-22T18:06:45.504010+0000 mon.a (mon.0) 3000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:46 smithi115 bash[46934]: audit 2022-04-22T18:06:45.918664+0000 mon.a (mon.0) 3001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:48 smithi121 bash[36902]: cluster 2022-04-22T18:06:46.899320+0000 mgr.y (mgr.34107) 1438 : cluster [DBG] pgmap v2002: 105 pgs: 105 active+clean; 124 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2022-04-22T18:06:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:48 smithi121 bash[36902]: audit 2022-04-22T18:06:47.894439+0000 mon.a (mon.0) 3002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:48 smithi115 bash[45495]: cluster 2022-04-22T18:06:46.899320+0000 mgr.y (mgr.34107) 1438 : cluster [DBG] pgmap v2002: 105 pgs: 105 active+clean; 124 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2022-04-22T18:06:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:48 smithi115 bash[45495]: audit 2022-04-22T18:06:47.894439+0000 mon.a (mon.0) 3002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:48 smithi115 bash[46934]: cluster 2022-04-22T18:06:46.899320+0000 mgr.y (mgr.34107) 1438 : cluster [DBG] pgmap v2002: 105 pgs: 105 active+clean; 124 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2022-04-22T18:06:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:48 smithi115 bash[46934]: audit 2022-04-22T18:06:47.894439+0000 mon.a (mon.0) 3002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:50 smithi121 bash[36902]: cluster 2022-04-22T18:06:48.900231+0000 mgr.y (mgr.34107) 1439 : cluster [DBG] pgmap v2003: 105 pgs: 105 active+clean; 119 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:06:50.608 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:50 smithi115 bash[45495]: cluster 2022-04-22T18:06:48.900231+0000 mgr.y (mgr.34107) 1439 : cluster [DBG] pgmap v2003: 105 pgs: 105 active+clean; 119 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:06:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:50 smithi115 bash[46934]: cluster 2022-04-22T18:06:48.900231+0000 mgr.y (mgr.34107) 1439 : cluster [DBG] pgmap v2003: 105 pgs: 105 active+clean; 119 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:06:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:51 smithi121 bash[36902]: audit 2022-04-22T18:06:50.506534+0000 mon.a (mon.0) 3003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:51 smithi121 bash[36902]: audit 2022-04-22T18:06:50.921397+0000 mon.a (mon.0) 3004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:51 smithi115 bash[45495]: audit 2022-04-22T18:06:50.506534+0000 mon.a (mon.0) 3003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:51 smithi115 bash[45495]: audit 2022-04-22T18:06:50.921397+0000 mon.a (mon.0) 3004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:51 smithi115 bash[46934]: audit 2022-04-22T18:06:50.506534+0000 mon.a (mon.0) 3003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:51 smithi115 bash[46934]: audit 2022-04-22T18:06:50.921397+0000 mon.a (mon.0) 3004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:51.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:52 smithi121 bash[36902]: cluster 2022-04-22T18:06:50.901190+0000 mgr.y (mgr.34107) 1440 : cluster [DBG] pgmap v2004: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 137 op/s 2022-04-22T18:06:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:52 smithi115 bash[45495]: cluster 2022-04-22T18:06:50.901190+0000 mgr.y (mgr.34107) 1440 : cluster [DBG] pgmap v2004: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 137 op/s 2022-04-22T18:06:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:52 smithi115 bash[46934]: cluster 2022-04-22T18:06:50.901190+0000 mgr.y (mgr.34107) 1440 : cluster [DBG] pgmap v2004: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 137 op/s 2022-04-22T18:06:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:53 smithi121 bash[36902]: audit 2022-04-22T18:06:52.856225+0000 mon.a (mon.0) 3005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:06:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:53 smithi121 bash[36902]: audit 2022-04-22T18:06:52.858289+0000 mon.a (mon.0) 3006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:06:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:53 smithi121 bash[36902]: audit 2022-04-22T18:06:52.859561+0000 mon.a (mon.0) 3007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:06:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:53 smithi121 bash[36902]: audit 2022-04-22T18:06:52.897312+0000 mon.a (mon.0) 3008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:53.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[45495]: audit 2022-04-22T18:06:52.856225+0000 mon.a (mon.0) 3005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[45495]: audit 2022-04-22T18:06:52.858289+0000 mon.a (mon.0) 3006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[45495]: audit 2022-04-22T18:06:52.859561+0000 mon.a (mon.0) 3007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[45495]: audit 2022-04-22T18:06:52.897312+0000 mon.a (mon.0) 3008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[46934]: audit 2022-04-22T18:06:52.856225+0000 mon.a (mon.0) 3005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[46934]: audit 2022-04-22T18:06:52.858289+0000 mon.a (mon.0) 3006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:06:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[46934]: audit 2022-04-22T18:06:52.859561+0000 mon.a (mon.0) 3007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:06:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[46934]: audit 2022-04-22T18:06:52.897312+0000 mon.a (mon.0) 3008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:06:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:06:53.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:06:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:54 smithi121 bash[36902]: cluster 2022-04-22T18:06:52.902057+0000 mgr.y (mgr.34107) 1441 : cluster [DBG] pgmap v2005: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 135 op/s 2022-04-22T18:06:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:54 smithi115 bash[46934]: cluster 2022-04-22T18:06:52.902057+0000 mgr.y (mgr.34107) 1441 : cluster [DBG] pgmap v2005: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 135 op/s 2022-04-22T18:06:54.583 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:54 smithi115 bash[45495]: cluster 2022-04-22T18:06:52.902057+0000 mgr.y (mgr.34107) 1441 : cluster [DBG] pgmap v2005: 105 pgs: 105 active+clean; 106 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 135 op/s 2022-04-22T18:06:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:56 smithi121 bash[36902]: cluster 2022-04-22T18:06:54.903120+0000 mgr.y (mgr.34107) 1442 : cluster [DBG] pgmap v2006: 105 pgs: 105 active+clean; 93 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 224 op/s 2022-04-22T18:06:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:56 smithi121 bash[36902]: audit 2022-04-22T18:06:55.509219+0000 mon.a (mon.0) 3009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:56 smithi121 bash[36902]: audit 2022-04-22T18:06:55.924454+0000 mon.a (mon.0) 3010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[45495]: cluster 2022-04-22T18:06:54.903120+0000 mgr.y (mgr.34107) 1442 : cluster [DBG] pgmap v2006: 105 pgs: 105 active+clean; 93 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 224 op/s 2022-04-22T18:06:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[45495]: audit 2022-04-22T18:06:55.509219+0000 mon.a (mon.0) 3009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[45495]: audit 2022-04-22T18:06:55.924454+0000 mon.a (mon.0) 3010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[46934]: cluster 2022-04-22T18:06:54.903120+0000 mgr.y (mgr.34107) 1442 : cluster [DBG] pgmap v2006: 105 pgs: 105 active+clean; 93 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 224 op/s 2022-04-22T18:06:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[46934]: audit 2022-04-22T18:06:55.509219+0000 mon.a (mon.0) 3009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:06:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:56 smithi115 bash[46934]: audit 2022-04-22T18:06:55.924454+0000 mon.a (mon.0) 3010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[45495]: cluster 2022-04-22T18:06:56.904130+0000 mgr.y (mgr.34107) 1443 : cluster [DBG] pgmap v2007: 105 pgs: 105 active+clean; 85 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 170 KiB/s wr, 274 op/s 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[45495]: audit 2022-04-22T18:06:57.286749+0000 mon.a (mon.0) 3011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[45495]: audit 2022-04-22T18:06:57.895819+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[45495]: audit 2022-04-22T18:06:57.899260+0000 mon.a (mon.0) 3013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[46934]: cluster 2022-04-22T18:06:56.904130+0000 mgr.y (mgr.34107) 1443 : cluster [DBG] pgmap v2007: 105 pgs: 105 active+clean; 85 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 170 KiB/s wr, 274 op/s 2022-04-22T18:06:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[46934]: audit 2022-04-22T18:06:57.286749+0000 mon.a (mon.0) 3011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[46934]: audit 2022-04-22T18:06:57.895819+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:06:58 smithi115 bash[46934]: audit 2022-04-22T18:06:57.899260+0000 mon.a (mon.0) 3013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:06:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:58 smithi121 bash[36902]: cluster 2022-04-22T18:06:56.904130+0000 mgr.y (mgr.34107) 1443 : cluster [DBG] pgmap v2007: 105 pgs: 105 active+clean; 85 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 170 KiB/s wr, 274 op/s 2022-04-22T18:06:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:58 smithi121 bash[36902]: audit 2022-04-22T18:06:57.286749+0000 mon.a (mon.0) 3011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:58 smithi121 bash[36902]: audit 2022-04-22T18:06:57.895819+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:06:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:06:58 smithi121 bash[36902]: audit 2022-04-22T18:06:57.899260+0000 mon.a (mon.0) 3013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:00 smithi121 bash[36902]: cluster 2022-04-22T18:06:58.905038+0000 mgr.y (mgr.34107) 1444 : cluster [DBG] pgmap v2008: 105 pgs: 105 active+clean; 77 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 25 KiB/s wr, 315 op/s 2022-04-22T18:07:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:00 smithi115 bash[45495]: cluster 2022-04-22T18:06:58.905038+0000 mgr.y (mgr.34107) 1444 : cluster [DBG] pgmap v2008: 105 pgs: 105 active+clean; 77 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 25 KiB/s wr, 315 op/s 2022-04-22T18:07:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:00 smithi115 bash[46934]: cluster 2022-04-22T18:06:58.905038+0000 mgr.y (mgr.34107) 1444 : cluster [DBG] pgmap v2008: 105 pgs: 105 active+clean; 77 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 152 KiB/s rd, 25 KiB/s wr, 315 op/s 2022-04-22T18:07:01.755 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:01.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:01.755 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:01 smithi115 bash[46934]: audit 2022-04-22T18:07:00.512087+0000 mon.a (mon.0) 3014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:01.755 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:01 smithi115 bash[46934]: audit 2022-04-22T18:07:00.927639+0000 mon.a (mon.0) 3015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:01.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:01 smithi115 bash[45495]: audit 2022-04-22T18:07:00.512087+0000 mon.a (mon.0) 3014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:01.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:01 smithi115 bash[45495]: audit 2022-04-22T18:07:00.927639+0000 mon.a (mon.0) 3015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:01 smithi121 bash[36902]: audit 2022-04-22T18:07:00.512087+0000 mon.a (mon.0) 3014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:01 smithi121 bash[36902]: audit 2022-04-22T18:07:00.927639+0000 mon.a (mon.0) 3015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:02.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:02 smithi121 bash[36902]: cluster 2022-04-22T18:07:00.906209+0000 mgr.y (mgr.34107) 1445 : cluster [DBG] pgmap v2009: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 364 op/s 2022-04-22T18:07:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:02 smithi115 bash[46934]: cluster 2022-04-22T18:07:00.906209+0000 mgr.y (mgr.34107) 1445 : cluster [DBG] pgmap v2009: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 364 op/s 2022-04-22T18:07:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:02 smithi115 bash[45495]: cluster 2022-04-22T18:07:00.906209+0000 mgr.y (mgr.34107) 1445 : cluster [DBG] pgmap v2009: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 364 op/s 2022-04-22T18:07:03.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:03 smithi121 bash[36902]: audit 2022-04-22T18:07:02.901494+0000 mon.a (mon.0) 3016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:03.643 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:03 smithi115 bash[45495]: audit 2022-04-22T18:07:02.901494+0000 mon.a (mon.0) 3016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:03.644 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:03.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:03.644 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:03 smithi115 bash[46934]: audit 2022-04-22T18:07:02.901494+0000 mon.a (mon.0) 3016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:04.738 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:04 smithi121 bash[36902]: cluster 2022-04-22T18:07:02.906998+0000 mgr.y (mgr.34107) 1446 : cluster [DBG] pgmap v2010: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 278 op/s 2022-04-22T18:07:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:04 smithi115 bash[45495]: cluster 2022-04-22T18:07:02.906998+0000 mgr.y (mgr.34107) 1446 : cluster [DBG] pgmap v2010: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 278 op/s 2022-04-22T18:07:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:04 smithi115 bash[46934]: cluster 2022-04-22T18:07:02.906998+0000 mgr.y (mgr.34107) 1446 : cluster [DBG] pgmap v2010: 105 pgs: 105 active+clean; 64 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 278 op/s 2022-04-22T18:07:05.783 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:07:06.166 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:07:06.166 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (30m) 9m ago 43m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:07:06.166 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (43m) 8s ago 43m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (32m) 9m ago 44m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (31m) 8s ago 44m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (39m) 8s ago 49m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (40m) 9m ago 51m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (39m) 9m ago 51m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (38m) 8s ago 49m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (38m) 9m ago 49m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (31m) 9m ago 44m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (31m) 8s ago 44m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:07:06.167 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (36m) 9m ago 48m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (36m) 9m ago 48m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (35m) 9m ago 47m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (35m) 9m ago 47m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (34m) 8s ago 46m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (33m) 8s ago 46m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (33m) 8s ago 46m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (32m) 8s ago 45m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:07:06.168 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (30m) 8s ago 44m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:07:06.610 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:07:06.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:07:06.612 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:07:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:06 smithi121 bash[36902]: cluster 2022-04-22T18:07:04.907670+0000 mgr.y (mgr.34107) 1447 : cluster [DBG] pgmap v2011: 105 pgs: 105 active+clean; 52 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 361 op/s 2022-04-22T18:07:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:06 smithi121 bash[36902]: audit 2022-04-22T18:07:05.514673+0000 mon.a (mon.0) 3017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:06 smithi121 bash[36902]: audit 2022-04-22T18:07:05.930247+0000 mon.a (mon.0) 3018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[45495]: cluster 2022-04-22T18:07:04.907670+0000 mgr.y (mgr.34107) 1447 : cluster [DBG] pgmap v2011: 105 pgs: 105 active+clean; 52 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 361 op/s 2022-04-22T18:07:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[45495]: audit 2022-04-22T18:07:05.514673+0000 mon.a (mon.0) 3017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[45495]: audit 2022-04-22T18:07:05.930247+0000 mon.a (mon.0) 3018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[46934]: cluster 2022-04-22T18:07:04.907670+0000 mgr.y (mgr.34107) 1447 : cluster [DBG] pgmap v2011: 105 pgs: 105 active+clean; 52 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 361 op/s 2022-04-22T18:07:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[46934]: audit 2022-04-22T18:07:05.514673+0000 mon.a (mon.0) 3017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:06 smithi115 bash[46934]: audit 2022-04-22T18:07:05.930247+0000 mon.a (mon.0) 3018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:07 smithi121 bash[36902]: audit 2022-04-22T18:07:05.761422+0000 mgr.y (mgr.34107) 1448 : audit [DBG] from='client.45853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:07 smithi121 bash[36902]: audit 2022-04-22T18:07:06.144045+0000 mgr.y (mgr.34107) 1449 : audit [DBG] from='client.35919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:07 smithi121 bash[36902]: audit 2022-04-22T18:07:06.605127+0000 mon.a (mon.0) 3019 : audit [DBG] from='client.? 172.21.15.115:0/915537855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:07 smithi121 bash[36902]: audit 2022-04-22T18:07:07.139423+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:07 smithi121 bash[36902]: audit 2022-04-22T18:07:07.146596+0000 mon.a (mon.0) 3021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[46934]: audit 2022-04-22T18:07:05.761422+0000 mgr.y (mgr.34107) 1448 : audit [DBG] from='client.45853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[46934]: audit 2022-04-22T18:07:06.144045+0000 mgr.y (mgr.34107) 1449 : audit [DBG] from='client.35919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[46934]: audit 2022-04-22T18:07:06.605127+0000 mon.a (mon.0) 3019 : audit [DBG] from='client.? 172.21.15.115:0/915537855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[46934]: audit 2022-04-22T18:07:07.139423+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[46934]: audit 2022-04-22T18:07:07.146596+0000 mon.a (mon.0) 3021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[45495]: audit 2022-04-22T18:07:05.761422+0000 mgr.y (mgr.34107) 1448 : audit [DBG] from='client.45853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[45495]: audit 2022-04-22T18:07:06.144045+0000 mgr.y (mgr.34107) 1449 : audit [DBG] from='client.35919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[45495]: audit 2022-04-22T18:07:06.605127+0000 mon.a (mon.0) 3019 : audit [DBG] from='client.? 172.21.15.115:0/915537855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[45495]: audit 2022-04-22T18:07:07.139423+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:07 smithi115 bash[45495]: audit 2022-04-22T18:07:07.146596+0000 mon.a (mon.0) 3021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:07:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:08 smithi121 bash[36902]: cluster 2022-04-22T18:07:06.908458+0000 mgr.y (mgr.34107) 1450 : cluster [DBG] pgmap v2012: 105 pgs: 105 active+clean; 46 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 321 op/s 2022-04-22T18:07:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:08 smithi121 bash[36902]: cephadm 2022-04-22T18:07:07.130747+0000 mgr.y (mgr.34107) 1451 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:07:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:08 smithi121 bash[36902]: audit 2022-04-22T18:07:07.904135+0000 mon.a (mon.0) 3022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[45495]: cluster 2022-04-22T18:07:06.908458+0000 mgr.y (mgr.34107) 1450 : cluster [DBG] pgmap v2012: 105 pgs: 105 active+clean; 46 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 321 op/s 2022-04-22T18:07:08.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[45495]: cephadm 2022-04-22T18:07:07.130747+0000 mgr.y (mgr.34107) 1451 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:07:08.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[45495]: audit 2022-04-22T18:07:07.904135+0000 mon.a (mon.0) 3022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:08.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[46934]: cluster 2022-04-22T18:07:06.908458+0000 mgr.y (mgr.34107) 1450 : cluster [DBG] pgmap v2012: 105 pgs: 105 active+clean; 46 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 321 op/s 2022-04-22T18:07:08.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[46934]: cephadm 2022-04-22T18:07:07.130747+0000 mgr.y (mgr.34107) 1451 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:07:08.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:08 smithi115 bash[46934]: audit 2022-04-22T18:07:07.904135+0000 mon.a (mon.0) 3022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:10 smithi121 bash[36902]: cluster 2022-04-22T18:07:08.909300+0000 mgr.y (mgr.34107) 1452 : cluster [DBG] pgmap v2013: 105 pgs: 105 active+clean; 38 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s 2022-04-22T18:07:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:10 smithi115 bash[45495]: cluster 2022-04-22T18:07:08.909300+0000 mgr.y (mgr.34107) 1452 : cluster [DBG] pgmap v2013: 105 pgs: 105 active+clean; 38 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s 2022-04-22T18:07:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:10 smithi115 bash[46934]: cluster 2022-04-22T18:07:08.909300+0000 mgr.y (mgr.34107) 1452 : cluster [DBG] pgmap v2013: 105 pgs: 105 active+clean; 38 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s 2022-04-22T18:07:11.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:11 smithi115 bash[45495]: audit 2022-04-22T18:07:10.517242+0000 mon.a (mon.0) 3023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:11.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:11 smithi115 bash[45495]: audit 2022-04-22T18:07:10.933458+0000 mon.a (mon.0) 3024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:11.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:11 smithi115 bash[46934]: audit 2022-04-22T18:07:10.517242+0000 mon.a (mon.0) 3023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:11.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:11 smithi115 bash[46934]: audit 2022-04-22T18:07:10.933458+0000 mon.a (mon.0) 3024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:11 smithi121 bash[36902]: audit 2022-04-22T18:07:10.517242+0000 mon.a (mon.0) 3023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:11 smithi121 bash[36902]: audit 2022-04-22T18:07:10.933458+0000 mon.a (mon.0) 3024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:11.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:12 smithi121 bash[36902]: cluster 2022-04-22T18:07:10.910281+0000 mgr.y (mgr.34107) 1453 : cluster [DBG] pgmap v2014: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 345 op/s 2022-04-22T18:07:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:12 smithi115 bash[46934]: cluster 2022-04-22T18:07:10.910281+0000 mgr.y (mgr.34107) 1453 : cluster [DBG] pgmap v2014: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 345 op/s 2022-04-22T18:07:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:12 smithi115 bash[45495]: cluster 2022-04-22T18:07:10.910281+0000 mgr.y (mgr.34107) 1453 : cluster [DBG] pgmap v2014: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 345 op/s 2022-04-22T18:07:13.636 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:13 smithi115 bash[45495]: audit 2022-04-22T18:07:12.907095+0000 mon.a (mon.0) 3025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:13.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:13 smithi115 bash[46934]: audit 2022-04-22T18:07:12.907095+0000 mon.a (mon.0) 3025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:13 smithi121 bash[36902]: audit 2022-04-22T18:07:12.907095+0000 mon.a (mon.0) 3025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:13.631Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:14 smithi121 bash[36902]: cluster 2022-04-22T18:07:12.911148+0000 mgr.y (mgr.34107) 1454 : cluster [DBG] pgmap v2015: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:14 smithi121 bash[36902]: audit 2022-04-22T18:07:13.671552+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:07:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:14 smithi121 bash[36902]: audit 2022-04-22T18:07:13.671929+0000 mon.a (mon.0) 3027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:07:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[45495]: cluster 2022-04-22T18:07:12.911148+0000 mgr.y (mgr.34107) 1454 : cluster [DBG] pgmap v2015: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[45495]: audit 2022-04-22T18:07:13.671552+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:07:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[45495]: audit 2022-04-22T18:07:13.671929+0000 mon.a (mon.0) 3027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:07:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[46934]: cluster 2022-04-22T18:07:12.911148+0000 mgr.y (mgr.34107) 1454 : cluster [DBG] pgmap v2015: 105 pgs: 105 active+clean; 26 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[46934]: audit 2022-04-22T18:07:13.671552+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:07:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:14 smithi115 bash[46934]: audit 2022-04-22T18:07:13.671929+0000 mon.a (mon.0) 3027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:07:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:16 smithi121 bash[36902]: cluster 2022-04-22T18:07:14.912178+0000 mgr.y (mgr.34107) 1455 : cluster [DBG] pgmap v2016: 105 pgs: 105 active+clean; 15 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 337 op/s 2022-04-22T18:07:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:16 smithi121 bash[36902]: audit 2022-04-22T18:07:15.519925+0000 mon.a (mon.0) 3028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:16 smithi121 bash[36902]: audit 2022-04-22T18:07:15.936301+0000 mon.a (mon.0) 3029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[45495]: cluster 2022-04-22T18:07:14.912178+0000 mgr.y (mgr.34107) 1455 : cluster [DBG] pgmap v2016: 105 pgs: 105 active+clean; 15 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 337 op/s 2022-04-22T18:07:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[45495]: audit 2022-04-22T18:07:15.519925+0000 mon.a (mon.0) 3028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[45495]: audit 2022-04-22T18:07:15.936301+0000 mon.a (mon.0) 3029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[46934]: cluster 2022-04-22T18:07:14.912178+0000 mgr.y (mgr.34107) 1455 : cluster [DBG] pgmap v2016: 105 pgs: 105 active+clean; 15 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 337 op/s 2022-04-22T18:07:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[46934]: audit 2022-04-22T18:07:15.519925+0000 mon.a (mon.0) 3028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:16 smithi115 bash[46934]: audit 2022-04-22T18:07:15.936301+0000 mon.a (mon.0) 3029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:17.043 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T18:07:17.043 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T18:07:17.055 INFO:tasks.workunit:Stopping ['rados/load-gen-big.sh'] on client.0... 2022-04-22T18:07:17.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-04-22T18:07:17.319 DEBUG:teuthology.parallel:result is None 2022-04-22T18:07:17.320 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:07:17.333 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:07:17.333 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:07:17.338 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:07:17.338 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:07:17.338 INFO:teuthology.task.print:**** done end rados_loadgenbig.yaml 2022-04-22T18:07:17.338 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:07:17.338 INFO:teuthology.task.print:**** done start rbd_import_export.yaml 2022-04-22T18:07:17.338 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2022-04-22T18:07:17.339 INFO:tasks.workunit:Pulling workunits from ref pacific 2022-04-22T18:07:17.340 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-22T18:07:17.340 DEBUG:teuthology.orchestra.run.smithi121:> stat -- /home/ubuntu/cephtest/mnt.1 2022-04-22T18:07:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T18:07:17.348 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-04-22T18:07:17.349 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -- /home/ubuntu/cephtest/mnt.1 2022-04-22T18:07:17.356 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.1 2022-04-22T18:07:17.356 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/mnt.1 && mkdir -- client.1 2022-04-22T18:07:17.406 INFO:tasks.workunit:timeout=3h 2022-04-22T18:07:17.406 INFO:tasks.workunit:cleanup=True 2022-04-22T18:07:17.407 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone --depth 1 --branch pacific https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 2022-04-22T18:07:17.414 INFO:tasks.workunit.client.1.smithi121.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2022-04-22T18:07:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:18 smithi121 bash[36902]: cluster 2022-04-22T18:07:16.913186+0000 mgr.y (mgr.34107) 1456 : cluster [DBG] pgmap v2017: 105 pgs: 105 active+clean; 8.6 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 297 op/s 2022-04-22T18:07:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:18 smithi121 bash[36902]: audit 2022-04-22T18:07:17.909862+0000 mon.a (mon.0) 3030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:18.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:18 smithi115 bash[45495]: cluster 2022-04-22T18:07:16.913186+0000 mgr.y (mgr.34107) 1456 : cluster [DBG] pgmap v2017: 105 pgs: 105 active+clean; 8.6 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 297 op/s 2022-04-22T18:07:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:18 smithi115 bash[45495]: audit 2022-04-22T18:07:17.909862+0000 mon.a (mon.0) 3030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:18 smithi115 bash[46934]: cluster 2022-04-22T18:07:16.913186+0000 mgr.y (mgr.34107) 1456 : cluster [DBG] pgmap v2017: 105 pgs: 105 active+clean; 8.6 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 297 op/s 2022-04-22T18:07:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:18 smithi115 bash[46934]: audit 2022-04-22T18:07:17.909862+0000 mon.a (mon.0) 3030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:20 smithi121 bash[36902]: cluster 2022-04-22T18:07:18.914292+0000 mgr.y (mgr.34107) 1457 : cluster [DBG] pgmap v2018: 105 pgs: 105 active+clean; 3.4 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 285 op/s 2022-04-22T18:07:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:20 smithi115 bash[45495]: cluster 2022-04-22T18:07:18.914292+0000 mgr.y (mgr.34107) 1457 : cluster [DBG] pgmap v2018: 105 pgs: 105 active+clean; 3.4 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 285 op/s 2022-04-22T18:07:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:20 smithi115 bash[46934]: cluster 2022-04-22T18:07:18.914292+0000 mgr.y (mgr.34107) 1457 : cluster [DBG] pgmap v2018: 105 pgs: 105 active+clean; 3.4 GiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 285 op/s 2022-04-22T18:07:21.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:21 smithi115 bash[45495]: audit 2022-04-22T18:07:20.522999+0000 mon.a (mon.0) 3031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:21.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:21 smithi115 bash[45495]: audit 2022-04-22T18:07:20.939462+0000 mon.a (mon.0) 3032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:21 smithi115 bash[46934]: audit 2022-04-22T18:07:20.522999+0000 mon.a (mon.0) 3031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:21 smithi115 bash[46934]: audit 2022-04-22T18:07:20.939462+0000 mon.a (mon.0) 3032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:21 smithi121 bash[36902]: audit 2022-04-22T18:07:20.522999+0000 mon.a (mon.0) 3031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:21 smithi121 bash[36902]: audit 2022-04-22T18:07:20.939462+0000 mon.a (mon.0) 3032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:21.751Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:22 smithi121 bash[36902]: cluster 2022-04-22T18:07:20.915613+0000 mgr.y (mgr.34107) 1458 : cluster [DBG] pgmap v2019: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:22 smithi115 bash[45495]: cluster 2022-04-22T18:07:20.915613+0000 mgr.y (mgr.34107) 1458 : cluster [DBG] pgmap v2019: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:22 smithi115 bash[46934]: cluster 2022-04-22T18:07:20.915613+0000 mgr.y (mgr.34107) 1458 : cluster [DBG] pgmap v2019: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s 2022-04-22T18:07:23.638 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:23 smithi115 bash[45495]: audit 2022-04-22T18:07:22.912968+0000 mon.a (mon.0) 3033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:23.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:23 smithi115 bash[46934]: audit 2022-04-22T18:07:22.912968+0000 mon.a (mon.0) 3033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:23 smithi121 bash[36902]: audit 2022-04-22T18:07:22.912968+0000 mon.a (mon.0) 3033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:24.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:23.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:24.431 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2022-04-22T18:07:24.457 INFO:tasks.workunit.client.1.smithi121.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-22T18:07:24.461 INFO:tasks.workunit.client.1.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2022-04-22T18:07:24.461 INFO:tasks.workunit.client.1.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-22T18:07:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:24 smithi121 bash[36902]: cluster 2022-04-22T18:07:22.916544+0000 mgr.y (mgr.34107) 1459 : cluster [DBG] pgmap v2020: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:24 smithi115 bash[45495]: cluster 2022-04-22T18:07:22.916544+0000 mgr.y (mgr.34107) 1459 : cluster [DBG] pgmap v2020: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:24 smithi115 bash[46934]: cluster 2022-04-22T18:07:22.916544+0000 mgr.y (mgr.34107) 1459 : cluster [DBG] pgmap v2020: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:25.254 INFO:tasks.workunit.client.1.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-22T18:07:25.342 INFO:tasks.workunit.client.1.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-22T18:07:25.406 INFO:tasks.workunit.client.1.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2022-04-22T18:07:25.409 INFO:tasks.workunit.client.1.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2022-04-22T18:07:25.409 INFO:tasks.workunit.client.1.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-22T18:07:25.462 INFO:tasks.workunit.client.1.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2022-04-22T18:07:25.486 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-04-22T18:07:25.486 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2022-04-22T18:07:25.534 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.1... 2022-04-22T18:07:25.536 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2022-04-22T18:07:25.536 DEBUG:teuthology.orchestra.run.smithi121:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/import_export.sh 2022-04-22T18:07:25.547 INFO:tasks.workunit.client.1.smithi121.stderr:+ export RBD_FORCE_ALLOW_V1=1 2022-04-22T18:07:25.547 INFO:tasks.workunit.client.1.smithi121.stderr:+ TMPDIR=/tmp/rbd_import_export_52777 2022-04-22T18:07:25.547 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm -rf /tmp/rbd_import_export_52777 2022-04-22T18:07:25.548 INFO:tasks.workunit.client.1.smithi121.stderr:+ mkdir /tmp/rbd_import_export_52777 2022-04-22T18:07:25.550 INFO:tasks.workunit.client.1.smithi121.stderr:+ trap rm -rf /tmp/rbd_import_export_52777 INT TERM EXIT 2022-04-22T18:07:25.550 INFO:tasks.workunit.client.1.smithi121.stderr:+ mkdir foo.52777 2022-04-22T18:07:25.551 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import foo.52777 foo.dir 2022-04-22T18:07:25.599 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: cannot import a directory 2022-04-22T18:07:25.599 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 0% complete...failed. 2022-04-22T18:07:25.599 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: import failed: (21) Is a directory 2022-04-22T18:07:25.602 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:25.603 INFO:tasks.workunit.client.1.smithi121.stderr:+ rmdir foo.52777 2022-04-22T18:07:25.613 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_52777/img bs=1k count=1 seek=10 2022-04-22T18:07:25.614 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records in 2022-04-22T18:07:25.614 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records out 2022-04-22T18:07:25.614 INFO:tasks.workunit.client.1.smithi121.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000113134 s, 9.1 MB/s 2022-04-22T18:07:25.614 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_52777/img bs=1k count=10 seek=100 2022-04-22T18:07:25.615 INFO:tasks.workunit.client.1.smithi121.stderr:10+0 records in 2022-04-22T18:07:25.616 INFO:tasks.workunit.client.1.smithi121.stderr:10+0 records out 2022-04-22T18:07:25.616 INFO:tasks.workunit.client.1.smithi121.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.000211598 s, 48.4 MB/s 2022-04-22T18:07:25.616 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_52777/img bs=1k count=100 seek=1000 2022-04-22T18:07:25.617 INFO:tasks.workunit.client.1.smithi121.stderr:70+1 records in 2022-04-22T18:07:25.617 INFO:tasks.workunit.client.1.smithi121.stderr:70+1 records out 2022-04-22T18:07:25.617 INFO:tasks.workunit.client.1.smithi121.stderr:72056 bytes (72 kB, 70 KiB) copied, 0.000294976 s, 244 MB/s 2022-04-22T18:07:25.617 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_52777/img bs=1k seek=10000 2022-04-22T18:07:25.619 INFO:tasks.workunit.client.1.smithi121.stderr:138+1 records in 2022-04-22T18:07:25.619 INFO:tasks.workunit.client.1.smithi121.stderr:138+1 records out 2022-04-22T18:07:25.619 INFO:tasks.workunit.client.1.smithi121.stderr:142144 bytes (142 kB, 139 KiB) copied, 0.00100992 s, 141 MB/s 2022-04-22T18:07:25.619 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_52777/img bs=1k seek=100000 2022-04-22T18:07:25.621 INFO:tasks.workunit.client.1.smithi121.stderr:74+1 records in 2022-04-22T18:07:25.621 INFO:tasks.workunit.client.1.smithi121.stderr:74+1 records out 2022-04-22T18:07:25.621 INFO:tasks.workunit.client.1.smithi121.stderr:76160 bytes (76 kB, 74 KiB) copied, 0.000311508 s, 244 MB/s 2022-04-22T18:07:25.621 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_52777/img bs=1k seek=1000000 2022-04-22T18:07:25.623 INFO:tasks.workunit.client.1.smithi121.stderr:194+1 records in 2022-04-22T18:07:25.623 INFO:tasks.workunit.client.1.smithi121.stderr:194+1 records out 2022-04-22T18:07:25.623 INFO:tasks.workunit.client.1.smithi121.stderr:199136 bytes (199 kB, 194 KiB) copied, 0.00145235 s, 137 MB/s 2022-04-22T18:07:25.623 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm testimg 2022-04-22T18:07:25.706 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 0% complete...failed. 2022-04-22T18:07:25.727 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: delete error: (2) No such file or directory 2022-04-22T18:07:25.727 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:25.727 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:26 smithi121 bash[36902]: cluster 2022-04-22T18:07:24.917653+0000 mgr.y (mgr.34107) 1460 : cluster [DBG] pgmap v2021: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:26 smithi121 bash[36902]: audit 2022-04-22T18:07:25.526057+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:26 smithi121 bash[36902]: audit 2022-04-22T18:07:25.943339+0000 mon.a (mon.0) 3035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[45495]: cluster 2022-04-22T18:07:24.917653+0000 mgr.y (mgr.34107) 1460 : cluster [DBG] pgmap v2021: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[45495]: audit 2022-04-22T18:07:25.526057+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[45495]: audit 2022-04-22T18:07:25.943339+0000 mon.a (mon.0) 3035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[46934]: cluster 2022-04-22T18:07:24.917653+0000 mgr.y (mgr.34107) 1460 : cluster [DBG] pgmap v2021: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 180 op/s 2022-04-22T18:07:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[46934]: audit 2022-04-22T18:07:25.526057+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:26 smithi115 bash[46934]: audit 2022-04-22T18:07:25.943339+0000 mon.a (mon.0) 3035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:27 smithi121 bash[36902]: cluster 2022-04-22T18:07:26.379496+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2022-04-22T18:07:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:27 smithi115 bash[45495]: cluster 2022-04-22T18:07:26.379496+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2022-04-22T18:07:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:27 smithi115 bash[46934]: cluster 2022-04-22T18:07:26.379496+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e1060: 8 total, 8 up, 8 in 2022-04-22T18:07:28.055 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2022-04-22T18:07:28.056 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:28.060 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg /tmp/rbd_import_export_52777/img2 2022-04-22T18:07:28.351 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:28.356 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg - 2022-04-22T18:07:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:28 smithi121 bash[36902]: cluster 2022-04-22T18:07:26.918477+0000 mgr.y (mgr.34107) 1461 : cluster [DBG] pgmap v2023: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 71 op/s 2022-04-22T18:07:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:28 smithi121 bash[36902]: cluster 2022-04-22T18:07:27.389271+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2022-04-22T18:07:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:28 smithi121 bash[36902]: audit 2022-04-22T18:07:27.915991+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[45495]: cluster 2022-04-22T18:07:26.918477+0000 mgr.y (mgr.34107) 1461 : cluster [DBG] pgmap v2023: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 71 op/s 2022-04-22T18:07:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[45495]: cluster 2022-04-22T18:07:27.389271+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2022-04-22T18:07:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[45495]: audit 2022-04-22T18:07:27.915991+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[46934]: cluster 2022-04-22T18:07:26.918477+0000 mgr.y (mgr.34107) 1461 : cluster [DBG] pgmap v2023: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 71 op/s 2022-04-22T18:07:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[46934]: cluster 2022-04-22T18:07:27.389271+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e1061: 8 total, 8 up, 8 in 2022-04-22T18:07:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:28 smithi115 bash[46934]: audit 2022-04-22T18:07:27.915991+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:29.260 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:29.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm testimg 2022-04-22T18:07:29.473 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-22T18:07:29.478 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img2 2022-04-22T18:07:30.232 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img3 2022-04-22T18:07:30.763 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img2 /tmp/rbd_import_export_52777/img3 2022-04-22T18:07:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:30 smithi121 bash[36902]: cluster 2022-04-22T18:07:28.919537+0000 mgr.y (mgr.34107) 1462 : cluster [DBG] pgmap v2025: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 28 KiB/s wr, 0 op/s 2022-04-22T18:07:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:30 smithi115 bash[45495]: cluster 2022-04-22T18:07:28.919537+0000 mgr.y (mgr.34107) 1462 : cluster [DBG] pgmap v2025: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 28 KiB/s wr, 0 op/s 2022-04-22T18:07:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:30 smithi115 bash[46934]: cluster 2022-04-22T18:07:28.919537+0000 mgr.y (mgr.34107) 1462 : cluster [DBG] pgmap v2025: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 28 KiB/s wr, 0 op/s 2022-04-22T18:07:30.934 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format - testimg 2022-04-22T18:07:31.519 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:31.519 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:31.523 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg /tmp/rbd_import_export_52777/img2 2022-04-22T18:07:31.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:31 smithi115 bash[46934]: audit 2022-04-22T18:07:30.529103+0000 mon.a (mon.0) 3039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:31.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:31 smithi115 bash[46934]: audit 2022-04-22T18:07:30.946931+0000 mon.a (mon.0) 3040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:31.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:31 smithi115 bash[45495]: audit 2022-04-22T18:07:30.529103+0000 mon.a (mon.0) 3039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:31.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:31 smithi115 bash[45495]: audit 2022-04-22T18:07:30.946931+0000 mon.a (mon.0) 3040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:31 smithi121 bash[36902]: audit 2022-04-22T18:07:30.529103+0000 mon.a (mon.0) 3039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:31 smithi121 bash[36902]: audit 2022-04-22T18:07:30.946931+0000 mon.a (mon.0) 3040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:31.885 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:31.889 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg - 2022-04-22T18:07:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:31.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:32 smithi121 bash[36902]: cluster 2022-04-22T18:07:30.920787+0000 mgr.y (mgr.34107) 1463 : cluster [DBG] pgmap v2026: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:32.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:32 smithi115 bash[45495]: cluster 2022-04-22T18:07:30.920787+0000 mgr.y (mgr.34107) 1463 : cluster [DBG] pgmap v2026: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:32 smithi115 bash[46934]: cluster 2022-04-22T18:07:30.920787+0000 mgr.y (mgr.34107) 1463 : cluster [DBG] pgmap v2026: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:32.839 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:32.846 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm testimg 2022-04-22T18:07:33.064 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-22T18:07:33.069 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img2 2022-04-22T18:07:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:33 smithi121 bash[36902]: audit 2022-04-22T18:07:32.919266+0000 mon.a (mon.0) 3041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:33.791 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img3 2022-04-22T18:07:33.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:33 smithi115 bash[45495]: audit 2022-04-22T18:07:32.919266+0000 mon.a (mon.0) 3041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:33.826 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:33.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:33.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:33 smithi115 bash[46934]: audit 2022-04-22T18:07:32.919266+0000 mon.a (mon.0) 3041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:34.271 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img2 /tmp/rbd_import_export_52777/img3 2022-04-22T18:07:34.486 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd help export 2022-04-22T18:07:34.487 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep -q export-format 2022-04-22T18:07:34.510 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_52777/img bs=1k count=10 seek=100 2022-04-22T18:07:34.511 INFO:tasks.workunit.client.1.smithi121.stderr:10+0 records in 2022-04-22T18:07:34.512 INFO:tasks.workunit.client.1.smithi121.stderr:10+0 records out 2022-04-22T18:07:34.512 INFO:tasks.workunit.client.1.smithi121.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.000240894 s, 42.5 MB/s 2022-04-22T18:07:34.512 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:34.616 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:34.654 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:34.654 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap create testimg@snap 2022-04-22T18:07:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:34 smithi121 bash[36902]: cluster 2022-04-22T18:07:32.921694+0000 mgr.y (mgr.34107) 1464 : cluster [DBG] pgmap v2027: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:34 smithi115 bash[45495]: cluster 2022-04-22T18:07:32.921694+0000 mgr.y (mgr.34107) 1464 : cluster [DBG] pgmap v2027: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:34 smithi115 bash[46934]: cluster 2022-04-22T18:07:32.921694+0000 mgr.y (mgr.34107) 1464 : cluster [DBG] pgmap v2027: 105 pgs: 105 active+clean; 3.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 65 KiB/s wr, 36 op/s 2022-04-22T18:07:35.437 INFO:tasks.workunit.client.1.smithi121.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2022-04-22T18:07:35.455 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd image-meta set testimg key1 value1 2022-04-22T18:07:35.544 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd image-meta list testimg 2022-04-22T18:07:35.610 INFO:tasks.workunit.client.1.smithi121.stderr:+ IMAGEMETA_BEFORE=There is 1 metadatum on this image: 2022-04-22T18:07:35.610 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:35.610 INFO:tasks.workunit.client.1.smithi121.stderr:Key Value 2022-04-22T18:07:35.610 INFO:tasks.workunit.client.1.smithi121.stderr:key1 value1 2022-04-22T18:07:35.611 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:35.673 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:35.678 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_52777/img_v2 testimg_import 2022-04-22T18:07:35.885 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2022-04-22T18:07:35.891 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import 2022-04-22T18:07:35.943 INFO:tasks.workunit.client.1.smithi121.stdout:rbd image 'testimg_import': 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: size 110 KiB in 1 objects 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: order 22 (4 MiB objects) 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: snapshot_count: 1 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: id: d9ac111ceffd 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: block_name_prefix: rbd_data.d9ac111ceffd 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: format: 2 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: op_features: 2022-04-22T18:07:35.944 INFO:tasks.workunit.client.1.smithi121.stdout: flags: 2022-04-22T18:07:35.945 INFO:tasks.workunit.client.1.smithi121.stdout: create_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:35.945 INFO:tasks.workunit.client.1.smithi121.stdout: access_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:35.945 INFO:tasks.workunit.client.1.smithi121.stdout: modify_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:35.949 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import@snap 2022-04-22T18:07:36.007 INFO:tasks.workunit.client.1.smithi121.stdout:rbd image 'testimg_import': 2022-04-22T18:07:36.008 INFO:tasks.workunit.client.1.smithi121.stdout: size 110 KiB in 1 objects 2022-04-22T18:07:36.008 INFO:tasks.workunit.client.1.smithi121.stdout: order 22 (4 MiB objects) 2022-04-22T18:07:36.008 INFO:tasks.workunit.client.1.smithi121.stdout: snapshot_count: 1 2022-04-22T18:07:36.008 INFO:tasks.workunit.client.1.smithi121.stdout: id: d9ac111ceffd 2022-04-22T18:07:36.008 INFO:tasks.workunit.client.1.smithi121.stdout: block_name_prefix: rbd_data.d9ac111ceffd 2022-04-22T18:07:36.009 INFO:tasks.workunit.client.1.smithi121.stdout: format: 2 2022-04-22T18:07:36.009 INFO:tasks.workunit.client.1.smithi121.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2022-04-22T18:07:36.009 INFO:tasks.workunit.client.1.smithi121.stdout: op_features: 2022-04-22T18:07:36.009 INFO:tasks.workunit.client.1.smithi121.stdout: flags: 2022-04-22T18:07:36.009 INFO:tasks.workunit.client.1.smithi121.stdout: create_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:36.010 INFO:tasks.workunit.client.1.smithi121.stdout: access_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:36.010 INFO:tasks.workunit.client.1.smithi121.stdout: modify_timestamp: Fri Apr 22 18:07:35 2022 2022-04-22T18:07:36.010 INFO:tasks.workunit.client.1.smithi121.stdout: protected: False 2022-04-22T18:07:36.014 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd image-meta list testimg_import 2022-04-22T18:07:36.080 INFO:tasks.workunit.client.1.smithi121.stderr:+ IMAGEMETA_AFTER=There is 1 metadatum on this image: 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr:Key Value 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr:key1 value1 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ There is 1 metadatum on this image: 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr:Key Value 2022-04-22T18:07:36.081 INFO:tasks.workunit.client.1.smithi121.stderr:key1 value1 = There is 1 metadatum on this image: 2022-04-22T18:07:36.082 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:36.082 INFO:tasks.workunit.client.1.smithi121.stderr:Key Value 2022-04-22T18:07:36.082 INFO:tasks.workunit.client.1.smithi121.stderr:key1 value1 ] 2022-04-22T18:07:36.082 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg_import /tmp/rbd_import_export_52777/img_import 2022-04-22T18:07:36.135 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete...done. 2022-04-22T18:07:36.138 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img_import 2022-04-22T18:07:36.139 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/img /tmp/rbd_import_export_52777/img_import 2022-04-22T18:07:36.140 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/img --format %b 2022-04-22T18:07:36.141 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=24 2022-04-22T18:07:36.141 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/img_import --format %b 2022-04-22T18:07:36.142 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=224 2022-04-22T18:07:36.142 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=200 2022-04-22T18:07:36.142 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=200 2022-04-22T18:07:36.142 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2022-04-22T18:07:36.142 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_52777/img_snap 2022-04-22T18:07:36.195 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete...done. 2022-04-22T18:07:36.201 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_52777/img_snap_import 2022-04-22T18:07:36.256 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete...done. 2022-04-22T18:07:36.262 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/img_snap /tmp/rbd_import_export_52777/img_snap_import 2022-04-22T18:07:36.262 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/img_snap /tmp/rbd_import_export_52777/img_snap_import 2022-04-22T18:07:36.263 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/img_snap --format %b 2022-04-22T18:07:36.265 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=224 2022-04-22T18:07:36.265 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/img_snap_import --format %b 2022-04-22T18:07:36.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=224 2022-04-22T18:07:36.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:36.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:36.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2022-04-22T18:07:36.267 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:36.268 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_import 2022-04-22T18:07:36.269 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_snap 2022-04-22T18:07:36.270 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_snap_import 2022-04-22T18:07:36.271 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap rm testimg_import@snap 2022-04-22T18:07:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:36 smithi121 bash[36902]: cluster 2022-04-22T18:07:34.922821+0000 mgr.y (mgr.34107) 1465 : cluster [DBG] pgmap v2028: 105 pgs: 105 active+clean; 5.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 270 KiB/s rd, 111 KiB/s wr, 77 op/s 2022-04-22T18:07:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:36 smithi121 bash[36902]: cluster 2022-04-22T18:07:35.413808+0000 mon.a (mon.0) 3042 : cluster [DBG] osdmap e1062: 8 total, 8 up, 8 in 2022-04-22T18:07:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:36 smithi121 bash[36902]: audit 2022-04-22T18:07:35.532167+0000 mon.a (mon.0) 3043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:36 smithi121 bash[36902]: cluster 2022-04-22T18:07:35.826229+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2022-04-22T18:07:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:36 smithi121 bash[36902]: audit 2022-04-22T18:07:35.950368+0000 mon.a (mon.0) 3045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[45495]: cluster 2022-04-22T18:07:34.922821+0000 mgr.y (mgr.34107) 1465 : cluster [DBG] pgmap v2028: 105 pgs: 105 active+clean; 5.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 270 KiB/s rd, 111 KiB/s wr, 77 op/s 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[45495]: cluster 2022-04-22T18:07:35.413808+0000 mon.a (mon.0) 3042 : cluster [DBG] osdmap e1062: 8 total, 8 up, 8 in 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[45495]: audit 2022-04-22T18:07:35.532167+0000 mon.a (mon.0) 3043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[45495]: cluster 2022-04-22T18:07:35.826229+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[45495]: audit 2022-04-22T18:07:35.950368+0000 mon.a (mon.0) 3045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[46934]: cluster 2022-04-22T18:07:34.922821+0000 mgr.y (mgr.34107) 1465 : cluster [DBG] pgmap v2028: 105 pgs: 105 active+clean; 5.0 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 270 KiB/s rd, 111 KiB/s wr, 77 op/s 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[46934]: cluster 2022-04-22T18:07:35.413808+0000 mon.a (mon.0) 3042 : cluster [DBG] osdmap e1062: 8 total, 8 up, 8 in 2022-04-22T18:07:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[46934]: audit 2022-04-22T18:07:35.532167+0000 mon.a (mon.0) 3043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[46934]: cluster 2022-04-22T18:07:35.826229+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e1063: 8 total, 8 up, 8 in 2022-04-22T18:07:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:36 smithi115 bash[46934]: audit 2022-04-22T18:07:35.950368+0000 mon.a (mon.0) 3045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:36.854 INFO:tasks.workunit.client.1.smithi121.stderr: Removing snap: 100% complete...done. 2022-04-22T18:07:36.872 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg_import 2022-04-22T18:07:37.010 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:07:37.095 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:37.099 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap rm testimg@snap 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (30m) 9m ago 44m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (44m) 40s ago 44m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (32m) 9m ago 45m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (32m) 40s ago 45m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (39m) 40s ago 49m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:07:37.353 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (40m) 9m ago 51m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (39m) 9m ago 52m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (38m) 40s ago 50m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (39m) 9m ago 50m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (31m) 9m ago 44m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (31m) 40s ago 44m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (37m) 9m ago 49m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (36m) 9m ago 48m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (36m) 9m ago 48m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (35m) 9m ago 47m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:07:37.354 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (35m) 40s ago 47m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:07:37.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (34m) 40s ago 46m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:07:37.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (33m) 40s ago 46m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:07:37.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (33m) 40s ago 46m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:07:37.355 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (31m) 40s ago 44m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:07:37.759 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:07:37.759 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:07:37.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:07:37.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:37.759 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:07:37.760 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:07:37.761 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:07:37.863 INFO:tasks.workunit.client.1.smithi121.stderr: Removing snap: 100% complete...done. 2022-04-22T18:07:37.881 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm testimg 2022-04-22T18:07:38.109 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:38.113 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --order 20 /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:38.224 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:38.225 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:38.229 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:37 smithi121 bash[36902]: cluster 2022-04-22T18:07:36.839843+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2022-04-22T18:07:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:37 smithi121 bash[36902]: cluster 2022-04-22T18:07:36.923663+0000 mgr.y (mgr.34107) 1466 : cluster [DBG] pgmap v2032: 105 pgs: 105 active+clean; 5.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 94 KiB/s wr, 110 op/s 2022-04-22T18:07:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:37 smithi121 bash[36902]: audit 2022-04-22T18:07:36.986879+0000 mgr.y (mgr.34107) 1467 : audit [DBG] from='client.36066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:37 smithi121 bash[36902]: audit 2022-04-22T18:07:37.329657+0000 mgr.y (mgr.34107) 1468 : audit [DBG] from='client.36078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:37 smithi121 bash[36902]: audit 2022-04-22T18:07:37.753439+0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.115:0/631625507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:38.284 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:38.290 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_52777/img_v2 testimg_import 2022-04-22T18:07:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[45495]: cluster 2022-04-22T18:07:36.839843+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[45495]: cluster 2022-04-22T18:07:36.923663+0000 mgr.y (mgr.34107) 1466 : cluster [DBG] pgmap v2032: 105 pgs: 105 active+clean; 5.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 94 KiB/s wr, 110 op/s 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[45495]: audit 2022-04-22T18:07:36.986879+0000 mgr.y (mgr.34107) 1467 : audit [DBG] from='client.36066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[45495]: audit 2022-04-22T18:07:37.329657+0000 mgr.y (mgr.34107) 1468 : audit [DBG] from='client.36078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[45495]: audit 2022-04-22T18:07:37.753439+0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.115:0/631625507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[46934]: cluster 2022-04-22T18:07:36.839843+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e1064: 8 total, 8 up, 8 in 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[46934]: cluster 2022-04-22T18:07:36.923663+0000 mgr.y (mgr.34107) 1466 : cluster [DBG] pgmap v2032: 105 pgs: 105 active+clean; 5.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 285 KiB/s rd, 94 KiB/s wr, 110 op/s 2022-04-22T18:07:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[46934]: audit 2022-04-22T18:07:36.986879+0000 mgr.y (mgr.34107) 1467 : audit [DBG] from='client.36066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[46934]: audit 2022-04-22T18:07:37.329657+0000 mgr.y (mgr.34107) 1468 : audit [DBG] from='client.36078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:07:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:37 smithi115 bash[46934]: audit 2022-04-22T18:07:37.753439+0000 mon.c (mon.1) 84 : audit [DBG] from='client.? 172.21.15.115:0/631625507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:07:38.417 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2022-04-22T18:07:38.431 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import 2022-04-22T18:07:38.431 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep order 2022-04-22T18:07:38.431 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $2} 2022-04-22T18:07:38.431 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep 20 2022-04-22T18:07:38.478 INFO:tasks.workunit.client.1.smithi121.stdout:20 2022-04-22T18:07:38.479 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:38.480 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg_import 2022-04-22T18:07:38.704 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:38.730 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg 2022-04-22T18:07:38.928 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:38.934 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:39.020 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:39.025 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg 2022-04-22T18:07:39.025 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep features 2022-04-22T18:07:39.084 INFO:tasks.workunit.client.1.smithi121.stderr:+ FEATURES_BEFORE= features: layering 2022-04-22T18:07:39.084 INFO:tasks.workunit.client.1.smithi121.stderr: op_features: 2022-04-22T18:07:39.084 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:39.142 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:39.147 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_52777/img_v2 testimg_import 2022-04-22T18:07:39.254 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2022-04-22T18:07:39.261 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import 2022-04-22T18:07:39.261 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep features 2022-04-22T18:07:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:38 smithi121 bash[36902]: cluster 2022-04-22T18:07:37.847597+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2022-04-22T18:07:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:38 smithi121 bash[36902]: audit 2022-04-22T18:07:37.921736+0000 mon.a (mon.0) 3048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:38 smithi115 bash[45495]: cluster 2022-04-22T18:07:37.847597+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2022-04-22T18:07:39.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:38 smithi115 bash[45495]: audit 2022-04-22T18:07:37.921736+0000 mon.a (mon.0) 3048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:39.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:38 smithi115 bash[46934]: cluster 2022-04-22T18:07:37.847597+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2022-04-22T18:07:39.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:38 smithi115 bash[46934]: audit 2022-04-22T18:07:37.921736+0000 mon.a (mon.0) 3048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:39.323 INFO:tasks.workunit.client.1.smithi121.stderr:+ FEATURES_AFTER= features: layering 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr: op_features: 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ features: layering 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr: op_features: != features: layering 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr: op_features: ] 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:39.324 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg_import 2022-04-22T18:07:39.481 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:39.487 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg 2022-04-22T18:07:39.662 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:39.667 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:39.783 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:39.787 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:39.844 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:39.850 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_52777/img_v2 testimg_import 2022-04-22T18:07:39.986 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2022-04-22T18:07:39.992 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import 2022-04-22T18:07:39.992 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep stripe unit 2022-04-22T18:07:39.992 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep -Ei (4 KiB|4096) 2022-04-22T18:07:40.049 INFO:tasks.workunit.client.1.smithi121.stdout: stripe unit: 4 KiB 2022-04-22T18:07:40.050 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import 2022-04-22T18:07:40.050 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep stripe count 2022-04-22T18:07:40.050 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $3} 2022-04-22T18:07:40.051 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep 1000 2022-04-22T18:07:40.107 INFO:tasks.workunit.client.1.smithi121.stdout:1000 2022-04-22T18:07:40.107 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/img_v2 2022-04-22T18:07:40.108 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg_import 2022-04-22T18:07:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:39 smithi121 bash[36902]: cluster 2022-04-22T18:07:38.924747+0000 mgr.y (mgr.34107) 1469 : cluster [DBG] pgmap v2034: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 177 KiB/s rd, 41 KiB/s wr, 182 op/s 2022-04-22T18:07:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:39 smithi115 bash[45495]: cluster 2022-04-22T18:07:38.924747+0000 mgr.y (mgr.34107) 1469 : cluster [DBG] pgmap v2034: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 177 KiB/s rd, 41 KiB/s wr, 182 op/s 2022-04-22T18:07:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:39 smithi115 bash[46934]: cluster 2022-04-22T18:07:38.924747+0000 mgr.y (mgr.34107) 1469 : cluster [DBG] pgmap v2034: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 177 KiB/s rd, 41 KiB/s wr, 182 op/s 2022-04-22T18:07:40.332 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-22T18:07:40.338 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg 2022-04-22T18:07:40.556 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-22T18:07:40.561 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_52777/img testimg 2022-04-22T18:07:40.670 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:40.675 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap create testimg@snap1 2022-04-22T18:07:40.883 INFO:tasks.workunit.client.1.smithi121.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2022-04-22T18:07:40.898 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap create testimg@snap2 2022-04-22T18:07:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:40 smithi121 bash[36902]: audit 2022-04-22T18:07:40.534851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:40 smithi115 bash[45495]: audit 2022-04-22T18:07:40.534851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:40 smithi115 bash[46934]: audit 2022-04-22T18:07:40.534851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:41.902 INFO:tasks.workunit.client.1.smithi121.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2022-04-22T18:07:41.918 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap protect testimg@snap2 2022-04-22T18:07:41.990 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_52777/snap_protect 2022-04-22T18:07:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:41.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[45495]: cluster 2022-04-22T18:07:40.864747+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2022-04-22T18:07:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[45495]: cluster 2022-04-22T18:07:40.925611+0000 mgr.y (mgr.34107) 1470 : cluster [DBG] pgmap v2036: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 49 KiB/s wr, 425 op/s 2022-04-22T18:07:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[45495]: audit 2022-04-22T18:07:40.953228+0000 mon.a (mon.0) 3051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:42.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[46934]: cluster 2022-04-22T18:07:40.864747+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2022-04-22T18:07:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[46934]: cluster 2022-04-22T18:07:40.925611+0000 mgr.y (mgr.34107) 1470 : cluster [DBG] pgmap v2036: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 49 KiB/s wr, 425 op/s 2022-04-22T18:07:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:41 smithi115 bash[46934]: audit 2022-04-22T18:07:40.953228+0000 mon.a (mon.0) 3051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:42.052 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:42.058 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_52777/snap_protect testimg_import 2022-04-22T18:07:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:41 smithi121 bash[36902]: cluster 2022-04-22T18:07:40.864747+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2022-04-22T18:07:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:41 smithi121 bash[36902]: cluster 2022-04-22T18:07:40.925611+0000 mgr.y (mgr.34107) 1470 : cluster [DBG] pgmap v2036: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 49 KiB/s wr, 425 op/s 2022-04-22T18:07:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:41 smithi121 bash[36902]: audit 2022-04-22T18:07:40.953228+0000 mon.a (mon.0) 3051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:42 smithi121 bash[36902]: cluster 2022-04-22T18:07:41.876722+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2022-04-22T18:07:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:42 smithi115 bash[46934]: cluster 2022-04-22T18:07:41.876722+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2022-04-22T18:07:43.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:42 smithi115 bash[45495]: cluster 2022-04-22T18:07:41.876722+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2022-04-22T18:07:43.897 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:43.635Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:43.952 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2022-04-22T18:07:43.957 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import@snap1 2022-04-22T18:07:43.957 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep protected: False 2022-04-22T18:07:44.015 INFO:tasks.workunit.client.1.smithi121.stdout: protected: False 2022-04-22T18:07:44.018 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info testimg_import@snap2 2022-04-22T18:07:44.019 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep protected: True 2022-04-22T18:07:44.078 INFO:tasks.workunit.client.1.smithi121.stdout: protected: True 2022-04-22T18:07:44.078 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/snap_protect 2022-04-22T18:07:44.079 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap unprotect testimg@snap2 2022-04-22T18:07:44.152 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap unprotect testimg_import@snap2 2022-04-22T18:07:44.229 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap purge testimg 2022-04-22T18:07:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:43 smithi121 bash[36902]: cluster 2022-04-22T18:07:42.876490+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2022-04-22T18:07:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:43 smithi121 bash[36902]: audit 2022-04-22T18:07:42.923800+0000 mon.a (mon.0) 3054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:43 smithi121 bash[36902]: cluster 2022-04-22T18:07:42.926438+0000 mgr.y (mgr.34107) 1471 : cluster [DBG] pgmap v2039: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 23 KiB/s wr, 370 op/s 2022-04-22T18:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[45495]: cluster 2022-04-22T18:07:42.876490+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2022-04-22T18:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[45495]: audit 2022-04-22T18:07:42.923800+0000 mon.a (mon.0) 3054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:44.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[45495]: cluster 2022-04-22T18:07:42.926438+0000 mgr.y (mgr.34107) 1471 : cluster [DBG] pgmap v2039: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 23 KiB/s wr, 370 op/s 2022-04-22T18:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[46934]: cluster 2022-04-22T18:07:42.876490+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2022-04-22T18:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[46934]: audit 2022-04-22T18:07:42.923800+0000 mon.a (mon.0) 3054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:43 smithi115 bash[46934]: cluster 2022-04-22T18:07:42.926438+0000 mgr.y (mgr.34107) 1471 : cluster [DBG] pgmap v2039: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 23 KiB/s wr, 370 op/s 2022-04-22T18:07:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:44 smithi121 bash[36902]: cluster 2022-04-22T18:07:43.897355+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2022-04-22T18:07:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:44 smithi115 bash[45495]: cluster 2022-04-22T18:07:43.897355+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2022-04-22T18:07:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:44 smithi115 bash[46934]: cluster 2022-04-22T18:07:43.897355+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2022-04-22T18:07:45.846 INFO:tasks.workunit.client.1.smithi121.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2022-04-22T18:07:46.116 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd snap purge testimg_import 2022-04-22T18:07:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:45 smithi121 bash[36902]: cluster 2022-04-22T18:07:44.901622+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2022-04-22T18:07:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:45 smithi121 bash[36902]: cluster 2022-04-22T18:07:44.927141+0000 mgr.y (mgr.34107) 1472 : cluster [DBG] pgmap v2042: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 24 KiB/s wr, 221 op/s 2022-04-22T18:07:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:45 smithi121 bash[36902]: audit 2022-04-22T18:07:45.538360+0000 mon.a (mon.0) 3057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:45 smithi121 bash[36902]: cluster 2022-04-22T18:07:45.833700+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2022-04-22T18:07:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[45495]: cluster 2022-04-22T18:07:44.901622+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[45495]: cluster 2022-04-22T18:07:44.927141+0000 mgr.y (mgr.34107) 1472 : cluster [DBG] pgmap v2042: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 24 KiB/s wr, 221 op/s 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[45495]: audit 2022-04-22T18:07:45.538360+0000 mon.a (mon.0) 3057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[45495]: cluster 2022-04-22T18:07:45.833700+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[46934]: cluster 2022-04-22T18:07:44.901622+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[46934]: cluster 2022-04-22T18:07:44.927141+0000 mgr.y (mgr.34107) 1472 : cluster [DBG] pgmap v2042: 105 pgs: 105 active+clean; 692 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 24 KiB/s wr, 221 op/s 2022-04-22T18:07:46.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[46934]: audit 2022-04-22T18:07:45.538360+0000 mon.a (mon.0) 3057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:46.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:45 smithi115 bash[46934]: cluster 2022-04-22T18:07:45.833700+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2022-04-22T18:07:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:46 smithi121 bash[36902]: audit 2022-04-22T18:07:45.956448+0000 mon.a (mon.0) 3059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:46 smithi115 bash[45495]: audit 2022-04-22T18:07:45.956448+0000 mon.a (mon.0) 3059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:46 smithi115 bash[46934]: audit 2022-04-22T18:07:45.956448+0000 mon.a (mon.0) 3059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:47.958 INFO:tasks.workunit.client.1.smithi121.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2022-04-22T18:07:47.973 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg 2022-04-22T18:07:48.196 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:48.202 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd remove testimg_import 2022-04-22T18:07:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:47 smithi121 bash[36902]: cluster 2022-04-22T18:07:46.922049+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2022-04-22T18:07:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:47 smithi121 bash[36902]: cluster 2022-04-22T18:07:46.927742+0000 mgr.y (mgr.34107) 1473 : cluster [DBG] pgmap v2045: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 28 KiB/s wr, 234 op/s 2022-04-22T18:07:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:47 smithi115 bash[45495]: cluster 2022-04-22T18:07:46.922049+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2022-04-22T18:07:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:47 smithi115 bash[45495]: cluster 2022-04-22T18:07:46.927742+0000 mgr.y (mgr.34107) 1473 : cluster [DBG] pgmap v2045: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 28 KiB/s wr, 234 op/s 2022-04-22T18:07:48.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:47 smithi115 bash[46934]: cluster 2022-04-22T18:07:46.922049+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2022-04-22T18:07:48.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:47 smithi115 bash[46934]: cluster 2022-04-22T18:07:46.927742+0000 mgr.y (mgr.34107) 1473 : cluster [DBG] pgmap v2045: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 28 KiB/s wr, 234 op/s 2022-04-22T18:07:48.436 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:48.442 INFO:tasks.workunit.client.1.smithi121.stderr:+ tiered=0 2022-04-22T18:07:48.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ ceph osd dump 2022-04-22T18:07:48.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep ^pool 2022-04-22T18:07:48.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ + grepgrep 'rbd' tier 2022-04-22T18:07:48.443 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:48.820 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_52777/sparse1 2022-04-22T18:07:48.832 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records in 2022-04-22T18:07:48.833 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records out 2022-04-22T18:07:48.833 INFO:tasks.workunit.client.1.smithi121.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010761 s, 97.4 MB/s 2022-04-22T18:07:48.833 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_52777/sparse2 2022-04-22T18:07:48.843 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records in 2022-04-22T18:07:48.843 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records out 2022-04-22T18:07:48.843 INFO:tasks.workunit.client.1.smithi121.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913469 s, 115 MB/s 2022-04-22T18:07:48.843 INFO:tasks.workunit.client.1.smithi121.stderr:+ truncate /tmp/rbd_import_export_52777/sparse2 -s 2M 2022-04-22T18:07:48.847 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse1 2022-04-22T18:07:48.925 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 0% complete...failed. 2022-04-22T18:07:48.926 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: delete error: (2) No such file or directory 2022-04-22T18:07:48.931 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:48.931 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_52777/sparse1 2022-04-22T18:07:49.050 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2022-04-22T18:07:49.051 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:49.051 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:49.056 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd ls -l 2022-04-22T18:07:49.056 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep+ sparse1 2022-04-22T18:07:49.056 INFO:tasks.workunit.client.1.smithi121.stderr:grep -Ei (2 MiB|2048k) 2022-04-22T18:07:49.113 INFO:tasks.workunit.client.1.smithi121.stdout:sparse1 2 MiB 2 2022-04-22T18:07:49.114 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse1 2022-04-22T18:07:49.115 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse1 2022-04-22T18:07:49.115 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse1 2022-04-22T18:07:49.115 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:49.115 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.dad853bd93bd 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.dad853bd93bd 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse1 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse1 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.dad853bd93bd\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse1 2022-04-22T18:07:49.174 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:49.175 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:49.233 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:49.233 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:49.233 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:49.233 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:49.233 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:48 smithi121 bash[36902]: audit 2022-04-22T18:07:47.927055+0000 mon.a (mon.0) 3061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:48 smithi121 bash[36902]: cluster 2022-04-22T18:07:47.941814+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2022-04-22T18:07:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:48 smithi121 bash[36902]: audit 2022-04-22T18:07:48.714801+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:48 smithi121 bash[36902]: audit 2022-04-22T18:07:48.806204+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[45495]: audit 2022-04-22T18:07:47.927055+0000 mon.a (mon.0) 3061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:49.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[45495]: cluster 2022-04-22T18:07:47.941814+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2022-04-22T18:07:49.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[45495]: audit 2022-04-22T18:07:48.714801+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[45495]: audit 2022-04-22T18:07:48.806204+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[46934]: audit 2022-04-22T18:07:47.927055+0000 mon.a (mon.0) 3061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:49.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[46934]: cluster 2022-04-22T18:07:47.941814+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2022-04-22T18:07:49.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[46934]: audit 2022-04-22T18:07:48.714801+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:48 smithi115 bash[46934]: audit 2022-04-22T18:07:48.806204+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.121:0/2246408780' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-04-22T18:07:49.321 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=1 2022-04-22T18:07:49.322 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 1 2022-04-22T18:07:49.322 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 1 = 1 ] 2022-04-22T18:07:49.322 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:49.379 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:49.384 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:49.384 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:49.388 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse1 --format %b 2022-04-22T18:07:49.389 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=2048 2022-04-22T18:07:49.389 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse1.out --format %b 2022-04-22T18:07:49.390 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=2048 2022-04-22T18:07:49.390 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:49.391 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:49.391 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2022-04-22T18:07:49.391 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:49.392 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse1 2022-04-22T18:07:49.614 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2022-04-22T18:07:49.620 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse2 2022-04-22T18:07:49.698 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 0% complete...failed. 2022-04-22T18:07:49.717 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: delete error: (2) No such file or directory 2022-04-22T18:07:49.718 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:49.718 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_52777/sparse2 2022-04-22T18:07:49.825 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2022-04-22T18:07:49.826 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:49.826 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:49.831 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd ls -l 2022-04-22T18:07:49.832 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep sparse2 2022-04-22T18:07:49.832 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep -Ei (2 MiB|2048k) 2022-04-22T18:07:49.890 INFO:tasks.workunit.client.1.smithi121.stdout:sparse2 2 MiB 2 2022-04-22T18:07:49.892 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse2 2022-04-22T18:07:49.892 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse2 2022-04-22T18:07:49.892 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse2 2022-04-22T18:07:49.892 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:49.892 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.db0bfa43f180 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.db0bfa43f180 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse2 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse2 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.db0bfa43f180\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:49.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:49.951 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse2 2022-04-22T18:07:49.951 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:49.951 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:50.007 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:50.007 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:50.007 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:50.007 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:50.007 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:50.101 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=0 2022-04-22T18:07:50.101 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 0 2022-04-22T18:07:50.101 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 0 = 0 ] 2022-04-22T18:07:50.101 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:50.150 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:50.155 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:50.155 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:50.159 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse2 --format %b 2022-04-22T18:07:50.160 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=2048 2022-04-22T18:07:50.160 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse2.out --format %b 2022-04-22T18:07:50.161 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=2048 2022-04-22T18:07:50.161 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:50.161 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:50.161 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2022-04-22T18:07:50.161 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:50.163 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse2 2022-04-22T18:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:49 smithi121 bash[36902]: cluster 2022-04-22T18:07:48.928501+0000 mgr.y (mgr.34107) 1474 : cluster [DBG] pgmap v2047: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 12 KiB/s wr, 117 op/s 2022-04-22T18:07:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:49 smithi115 bash[45495]: cluster 2022-04-22T18:07:48.928501+0000 mgr.y (mgr.34107) 1474 : cluster [DBG] pgmap v2047: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 12 KiB/s wr, 117 op/s 2022-04-22T18:07:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:49 smithi115 bash[46934]: cluster 2022-04-22T18:07:48.928501+0000 mgr.y (mgr.34107) 1474 : cluster [DBG] pgmap v2047: 105 pgs: 105 active+clean; 802 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 12 KiB/s wr, 117 op/s 2022-04-22T18:07:50.389 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2022-04-22T18:07:50.394 INFO:tasks.workunit.client.1.smithi121.stderr:+ truncate /tmp/rbd_import_export_52777/sparse1 -s 10M 2022-04-22T18:07:50.395 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format --order 20 - sparse1 2022-04-22T18:07:50.596 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:50.597 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:50.597 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:50.602 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd+ ls -l 2022-04-22T18:07:50.602 INFO:tasks.workunit.client.1.smithi121.stderr:grep sparse1 2022-04-22T18:07:50.602 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep -Ei (10 MiB|10240k) 2022-04-22T18:07:50.663 INFO:tasks.workunit.client.1.smithi121.stdout:sparse1 10 MiB 2 2022-04-22T18:07:50.665 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse1 2022-04-22T18:07:50.665 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse1 2022-04-22T18:07:50.665 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse1 2022-04-22T18:07:50.665 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:50.665 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:50.722 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.db38d3ee61bd 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.db38d3ee61bd 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse1 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse1 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.db38d3ee61bd\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse1 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:50.724 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:50.781 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:50.781 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:50.781 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:50.781 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:50.781 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:50.873 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=1 2022-04-22T18:07:50.873 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 1 2022-04-22T18:07:50.873 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 1 = 1 ] 2022-04-22T18:07:50.873 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:50.930 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:50.936 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:50.936 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/sparse1 /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:50.948 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse1 --format %b 2022-04-22T18:07:50.949 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=2048 2022-04-22T18:07:50.950 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse1.out --format %b 2022-04-22T18:07:50.952 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=2048 2022-04-22T18:07:50.952 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:50.953 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:50.953 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2022-04-22T18:07:50.953 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse1.out 2022-04-22T18:07:50.955 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse1 2022-04-22T18:07:51.176 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2022-04-22T18:07:51.182 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_52777/sparse2 oflag=append conv=notrunc 2022-04-22T18:07:51.200 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records in 2022-04-22T18:07:51.200 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records out 2022-04-22T18:07:51.200 INFO:tasks.workunit.client.1.smithi121.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0167732 s, 125 MB/s 2022-04-22T18:07:51.201 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format --order 20 - sparse2 2022-04-22T18:07:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:50 smithi121 bash[36902]: audit 2022-04-22T18:07:50.541327+0000 mon.a (mon.0) 3063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:50 smithi115 bash[45495]: audit 2022-04-22T18:07:50.541327+0000 mon.a (mon.0) 3063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:51.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:50 smithi115 bash[46934]: audit 2022-04-22T18:07:50.541327+0000 mon.a (mon.0) 3063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:51.379 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:51.380 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:51.380 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:51.385 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd ls -l 2022-04-22T18:07:51.386 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep sparse2 2022-04-22T18:07:51.386 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep -Ei (4 MiB|4096k) 2022-04-22T18:07:51.442 INFO:tasks.workunit.client.1.smithi121.stdout:sparse2 4 MiB 2 2022-04-22T18:07:51.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse2 2022-04-22T18:07:51.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse2 2022-04-22T18:07:51.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse2 2022-04-22T18:07:51.443 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:51.444 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:51.501 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.db655d16a967 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.db655d16a967 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse2 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse2 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.db655d16a967\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse2 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:51.503 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:51.554 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:51.554 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:51.554 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:51.555 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:51.555 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:51.641 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=0 2022-04-22T18:07:51.641 INFO:tasks.workunit.client.1.smithi121.stderr:2 2022-04-22T18:07:51.642 INFO:tasks.workunit.client.1.smithi121.stderr:3 2022-04-22T18:07:51.642 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 0 2 3 2022-04-22T18:07:51.642 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 0 2 3 = 0 2 3 ] 2022-04-22T18:07:51.642 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:51.696 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2022-04-22T18:07:51.745 INFO:tasks.workunit.client.1.smithi121.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_52777/sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:51.745 INFO:tasks.workunit.client.1.smithi121.stderr:+ cmp -l /tmp/rbd_import_export_52777/sparse2 /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:51.745 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse2 --format %b 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ origsize=6144 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse2.out --format %b 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ exportsize=6144 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ difference=0 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse2.out 2022-04-22T18:07:51.746 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse2 2022-04-22T18:07:51.935 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2022-04-22T18:07:51.941 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo partially-sparse file imports to partially-sparse image 2022-04-22T18:07:51.941 INFO:tasks.workunit.client.1.smithi121.stdout:partially-sparse file imports to partially-sparse image 2022-04-22T18:07:51.941 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_52777/sparse1 sparse 2022-04-22T18:07:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:51.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:51 smithi115 bash[46934]: cluster 2022-04-22T18:07:50.929616+0000 mgr.y (mgr.34107) 1475 : cluster [DBG] pgmap v2048: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 522 KiB/s wr, 209 op/s 2022-04-22T18:07:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:51 smithi115 bash[46934]: audit 2022-04-22T18:07:50.959771+0000 mon.a (mon.0) 3064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:51 smithi115 bash[45495]: cluster 2022-04-22T18:07:50.929616+0000 mgr.y (mgr.34107) 1475 : cluster [DBG] pgmap v2048: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 522 KiB/s wr, 209 op/s 2022-04-22T18:07:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:51 smithi115 bash[45495]: audit 2022-04-22T18:07:50.959771+0000 mon.a (mon.0) 3064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:52.067 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2022-04-22T18:07:52.069 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:52.069 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --order is deprecated, use --object-size 2022-04-22T18:07:52.075 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse 2022-04-22T18:07:52.075 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:52.076 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:52.076 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:52.076 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:52.127 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.db9256c7b79 2022-04-22T18:07:52.129 INFO:tasks.workunit.client.1.smithi121.stderr:+ + get_image_data_pool sparse 2022-04-22T18:07:52.129 INFO:tasks.workunit.client.1.smithi121.stderr:+ grepimage=sparse rbd_data.db9256c7b79 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.db9256c7b79\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:52.130 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:52.185 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:52.185 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:52.185 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:52.186 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:52.186 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:51 smithi121 bash[36902]: cluster 2022-04-22T18:07:50.929616+0000 mgr.y (mgr.34107) 1475 : cluster [DBG] pgmap v2048: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 522 KiB/s wr, 209 op/s 2022-04-22T18:07:52.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:51 smithi121 bash[36902]: audit 2022-04-22T18:07:50.959771+0000 mon.a (mon.0) 3064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:52.282 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=1 2022-04-22T18:07:52.282 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 1 2022-04-22T18:07:52.282 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 1 = 1 ] 2022-04-22T18:07:52.283 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse 2022-04-22T18:07:52.506 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2022-04-22T18:07:52.511 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo zeros import through stdin to sparse image 2022-04-22T18:07:52.512 INFO:tasks.workunit.client.1.smithi121.stdout:zeros import through stdin to sparse image 2022-04-22T18:07:52.512 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/dev/zero bs=1M count=4 2022-04-22T18:07:52.512 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd import --new-format - sparse 2022-04-22T18:07:52.598 INFO:tasks.workunit.client.1.smithi121.stderr:4+0 records in 2022-04-22T18:07:52.598 INFO:tasks.workunit.client.1.smithi121.stderr:4+0 records out 2022-04-22T18:07:52.598 INFO:tasks.workunit.client.1.smithi121.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0855586 s, 49.0 MB/s 2022-04-22T18:07:52.628 INFO:tasks.workunit.client.1.smithi121.stderr: Importing image: 100% complete...done. 2022-04-22T18:07:52.629 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:52.635 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse 2022-04-22T18:07:52.635 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:52.635 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:52.635 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:52.636 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:52.692 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.dbb3381a0f30 2022-04-22T18:07:52.693 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.dbb3381a0f30 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr:+ + + sort -u 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr:sed -e s/rbd_data.dbb3381a0f30\.// -eget_image_data_pool s/^0*\([0-9a-f]\)/\1/ sparse 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr: 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:52.694 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:52.695 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:52.748 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:52.748 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:52.748 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:52.748 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:52.748 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:52.839 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects= 2022-04-22T18:07:52.839 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 2022-04-22T18:07:52.839 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o = ] 2022-04-22T18:07:52.839 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse 2022-04-22T18:07:53.055 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:53.060 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo zeros export to sparse file 2022-04-22T18:07:53.061 INFO:tasks.workunit.client.1.smithi121.stdout:zeros export to sparse file 2022-04-22T18:07:53.061 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd create --new-format sparse --size 4 2022-04-22T18:07:53.127 INFO:tasks.workunit.client.1.smithi121.stderr:rbd: --new-format is deprecated, use --image-format 2 2022-04-22T18:07:53.131 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:53.131 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:53.132 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:53.189 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.dbd43aeeb111 2022-04-22T18:07:53.190 INFO:tasks.workunit.client.1.smithi121.stderr:+ dd if=/dev/zero bs=4M count=1 2022-04-22T18:07:53.190 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse 2022-04-22T18:07:53.190 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:53.191 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:53.191 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:53.191 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:53.247 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:53.247 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:53.247 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:53.247 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:53.248 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados -p rbd put rbd_data.dbd43aeeb111.000000000000 - 2022-04-22T18:07:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:52 smithi121 bash[36902]: audit 2022-04-22T18:07:52.929846+0000 mon.a (mon.0) 3065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:53.289 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records in 2022-04-22T18:07:53.289 INFO:tasks.workunit.client.1.smithi121.stderr:1+0 records out 2022-04-22T18:07:53.289 INFO:tasks.workunit.client.1.smithi121.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0981862 s, 42.7 MB/s 2022-04-22T18:07:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:52 smithi115 bash[45495]: audit 2022-04-22T18:07:52.929846+0000 mon.a (mon.0) 3065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:52 smithi115 bash[46934]: audit 2022-04-22T18:07:52.929846+0000 mon.a (mon.0) 3065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:53.314 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects sparse 2022-04-22T18:07:53.315 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:53.315 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:53.315 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep block_name_prefix 2022-04-22T18:07:53.315 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk {print $NF;} 2022-04-22T18:07:53.370 INFO:tasks.workunit.client.1.smithi121.stderr:+ prefix=rbd_data.dbd43aeeb111 2022-04-22T18:07:53.372 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep rbd_data.dbd43aeeb111 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ get_image_data_pool sparse 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ image=sparse 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ sed -e s/rbd_data.dbd43aeeb111\.// -e s/^0*\([0-9a-f]\)/\1/ 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ sort -u 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd info sparse 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ grep data_pool: 2022-04-22T18:07:53.373 INFO:tasks.workunit.client.1.smithi121.stderr:+ awk -F: { print $NF } 2022-04-22T18:07:53.426 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool= 2022-04-22T18:07:53.426 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ -z ] 2022-04-22T18:07:53.426 INFO:tasks.workunit.client.1.smithi121.stderr:+ data_pool=rbd 2022-04-22T18:07:53.426 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo rbd 2022-04-22T18:07:53.426 INFO:tasks.workunit.client.1.smithi121.stderr:+ rados ls -p rbd 2022-04-22T18:07:53.519 INFO:tasks.workunit.client.1.smithi121.stderr:+ objects=0 2022-04-22T18:07:53.519 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo 0 2022-04-22T18:07:53.519 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 -eq 1 -o 0 = 0 ] 2022-04-22T18:07:53.519 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse 2022-04-22T18:07:53.520 INFO:tasks.workunit.client.1.smithi121.stderr:rm: cannot remove '/tmp/rbd_import_export_52777/sparse': No such file or directory 2022-04-22T18:07:53.520 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:53.520 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd export sparse /tmp/rbd_import_export_52777/sparse 2022-04-22T18:07:53.577 INFO:tasks.workunit.client.1.smithi121.stderr: Exporting image: 100% complete...done. 2022-04-22T18:07:53.584 INFO:tasks.workunit.client.1.smithi121.stderr:+ stat /tmp/rbd_import_export_52777/sparse --format=%b 2022-04-22T18:07:53.585 INFO:tasks.workunit.client.1.smithi121.stderr:+ [ 0 = 0 ] 2022-04-22T18:07:53.585 INFO:tasks.workunit.client.1.smithi121.stderr:+ rbd rm sparse 2022-04-22T18:07:53.780 INFO:tasks.workunit.client.1.smithi121.stderr: Removing image: 100% complete...done. 2022-04-22T18:07:53.785 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm /tmp/rbd_import_export_52777/sparse /tmp/rbd_import_export_52777/sparse1 /tmp/rbd_import_export_52777/sparse2 /tmp/rbd_import_export_52777/sparse3 2022-04-22T18:07:53.789 INFO:tasks.workunit.client.1.smithi121.stderr:rm: cannot remove '/tmp/rbd_import_export_52777/sparse3': No such file or directory 2022-04-22T18:07:53.789 INFO:tasks.workunit.client.1.smithi121.stderr:+ true 2022-04-22T18:07:53.789 INFO:tasks.workunit.client.1.smithi121.stdout:OK 2022-04-22T18:07:53.790 INFO:tasks.workunit.client.1.smithi121.stderr:+ echo OK 2022-04-22T18:07:53.790 INFO:tasks.workunit.client.1.smithi121.stderr:+ rm -rf /tmp/rbd_import_export_52777 2022-04-22T18:07:53.791 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T18:07:53.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2022-04-22T18:07:53.806 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.1... 2022-04-22T18:07:53.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2022-04-22T18:07:53.967 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:07:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:07:53.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:07:54.041 DEBUG:teuthology.parallel:result is None 2022-04-22T18:07:54.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2022-04-22T18:07:54.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:53 smithi121 bash[36902]: cluster 2022-04-22T18:07:52.930676+0000 mgr.y (mgr.34107) 1476 : cluster [DBG] pgmap v2049: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 441 KiB/s wr, 177 op/s 2022-04-22T18:07:54.056 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2022-04-22T18:07:54.056 DEBUG:teuthology.orchestra.run.smithi121:> rmdir -- /home/ubuntu/cephtest/mnt.1 2022-04-22T18:07:54.061 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.1/client.1 2022-04-22T18:07:54.061 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:07:54.061 INFO:teuthology.task.print:**** done end rbd_import_export.yaml 2022-04-22T18:07:54.061 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:07:54.061 INFO:teuthology.task.print:**** done start test_rbd_api.yaml 2022-04-22T18:07:54.061 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2022-04-22T18:07:54.062 INFO:tasks.workunit:Pulling workunits from ref pacific 2022-04-22T18:07:54.063 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-22T18:07:54.063 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:07:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T18:07:54.071 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-04-22T18:07:54.071 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:07:54.076 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-04-22T18:07:54.076 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-04-22T18:07:54.083 INFO:tasks.workunit:timeout=3h 2022-04-22T18:07:54.083 INFO:tasks.workunit:cleanup=True 2022-04-22T18:07:54.084 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2022-04-22T18:07:54.091 INFO:tasks.workunit.client.0.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-04-22T18:07:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:53 smithi115 bash[45495]: cluster 2022-04-22T18:07:52.930676+0000 mgr.y (mgr.34107) 1476 : cluster [DBG] pgmap v2049: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 441 KiB/s wr, 177 op/s 2022-04-22T18:07:54.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:53 smithi115 bash[46934]: cluster 2022-04-22T18:07:52.930676+0000 mgr.y (mgr.34107) 1476 : cluster [DBG] pgmap v2049: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 441 KiB/s wr, 177 op/s 2022-04-22T18:07:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:55 smithi121 bash[36902]: cluster 2022-04-22T18:07:54.931772+0000 mgr.y (mgr.34107) 1477 : cluster [DBG] pgmap v2050: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 713 KiB/s rd, 654 KiB/s wr, 274 op/s 2022-04-22T18:07:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:55 smithi121 bash[36902]: audit 2022-04-22T18:07:55.544168+0000 mon.a (mon.0) 3066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:55 smithi121 bash[36902]: cluster 2022-04-22T18:07:55.836182+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2022-04-22T18:07:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:55 smithi121 bash[36902]: audit 2022-04-22T18:07:55.962990+0000 mon.a (mon.0) 3068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[45495]: cluster 2022-04-22T18:07:54.931772+0000 mgr.y (mgr.34107) 1477 : cluster [DBG] pgmap v2050: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 713 KiB/s rd, 654 KiB/s wr, 274 op/s 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[45495]: audit 2022-04-22T18:07:55.544168+0000 mon.a (mon.0) 3066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[45495]: cluster 2022-04-22T18:07:55.836182+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[45495]: audit 2022-04-22T18:07:55.962990+0000 mon.a (mon.0) 3068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[46934]: cluster 2022-04-22T18:07:54.931772+0000 mgr.y (mgr.34107) 1477 : cluster [DBG] pgmap v2050: 105 pgs: 105 active+clean; 1.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 713 KiB/s rd, 654 KiB/s wr, 274 op/s 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[46934]: audit 2022-04-22T18:07:55.544168+0000 mon.a (mon.0) 3066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:07:56.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[46934]: cluster 2022-04-22T18:07:55.836182+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2022-04-22T18:07:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:55 smithi115 bash[46934]: audit 2022-04-22T18:07:55.962990+0000 mon.a (mon.0) 3068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:07:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:57 smithi121 bash[36902]: cluster 2022-04-22T18:07:56.932772+0000 mgr.y (mgr.34107) 1478 : cluster [DBG] pgmap v2052: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 KiB/s rd, 812 KiB/s wr, 314 op/s 2022-04-22T18:07:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:57 smithi121 bash[36902]: audit 2022-04-22T18:07:57.932581+0000 mon.a (mon.0) 3069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:58.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:57 smithi115 bash[45495]: cluster 2022-04-22T18:07:56.932772+0000 mgr.y (mgr.34107) 1478 : cluster [DBG] pgmap v2052: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 KiB/s rd, 812 KiB/s wr, 314 op/s 2022-04-22T18:07:58.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:57 smithi115 bash[45495]: audit 2022-04-22T18:07:57.932581+0000 mon.a (mon.0) 3069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:07:58.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:57 smithi115 bash[46934]: cluster 2022-04-22T18:07:56.932772+0000 mgr.y (mgr.34107) 1478 : cluster [DBG] pgmap v2052: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 KiB/s rd, 812 KiB/s wr, 314 op/s 2022-04-22T18:07:58.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:57 smithi115 bash[46934]: audit 2022-04-22T18:07:57.932581+0000 mon.a (mon.0) 3069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:07:59 smithi121 bash[36902]: cluster 2022-04-22T18:07:58.933905+0000 mgr.y (mgr.34107) 1479 : cluster [DBG] pgmap v2053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 819 KiB/s rd, 1.1 MiB/s wr, 274 op/s 2022-04-22T18:08:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:07:59 smithi115 bash[45495]: cluster 2022-04-22T18:07:58.933905+0000 mgr.y (mgr.34107) 1479 : cluster [DBG] pgmap v2053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 819 KiB/s rd, 1.1 MiB/s wr, 274 op/s 2022-04-22T18:08:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:07:59 smithi115 bash[46934]: cluster 2022-04-22T18:07:58.933905+0000 mgr.y (mgr.34107) 1479 : cluster [DBG] pgmap v2053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 819 KiB/s rd, 1.1 MiB/s wr, 274 op/s 2022-04-22T18:08:01.155 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-04-22T18:08:01.203 INFO:tasks.workunit.client.0.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-22T18:08:01.206 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T18:08:01.206 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-22T18:08:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:00 smithi121 bash[36902]: audit 2022-04-22T18:08:00.547235+0000 mon.a (mon.0) 3070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:00 smithi121 bash[36902]: audit 2022-04-22T18:08:00.966864+0000 mon.a (mon.0) 3071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:01.271 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-22T18:08:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:00 smithi115 bash[46934]: audit 2022-04-22T18:08:00.547235+0000 mon.a (mon.0) 3070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:00 smithi115 bash[46934]: audit 2022-04-22T18:08:00.966864+0000 mon.a (mon.0) 3071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:01.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:00 smithi115 bash[45495]: audit 2022-04-22T18:08:00.547235+0000 mon.a (mon.0) 3070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:01.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:00 smithi115 bash[45495]: audit 2022-04-22T18:08:00.966864+0000 mon.a (mon.0) 3071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:01.341 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-22T18:08:01.395 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T18:08:01.396 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T18:08:01.396 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-22T18:08:01.453 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T18:08:01.542 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T18:08:01.542 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-04-22T18:08:01.550 INFO:tasks.workunit:Running workunits matching rbd/test_librbd.sh on client.0... 2022-04-22T18:08:01.551 INFO:tasks.workunit:Running workunit rbd/test_librbd.sh... 2022-04-22T18:08:01.551 DEBUG:teuthology.orchestra.run.smithi115:workunit test rbd/test_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-04-22T18:08:01.637 INFO:tasks.workunit.client.0.smithi115.stdout:seed 68765 2022-04-22T18:08:01.637 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] Running 371 tests from 15 test suites. 2022-04-22T18:08:01.638 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment set-up. 2022-04-22T18:08:01.638 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 128 tests from TestLibRBD 2022-04-22T18:08:01.674 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CreateAndStat 2022-04-22T18:08:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:01.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:02 smithi121 bash[36902]: cluster 2022-04-22T18:08:00.934925+0000 mgr.y (mgr.34107) 1480 : cluster [DBG] pgmap v2054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 545 KiB/s rd, 828 KiB/s wr, 185 op/s 2022-04-22T18:08:02.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:02 smithi115 bash[45495]: cluster 2022-04-22T18:08:00.934925+0000 mgr.y (mgr.34107) 1480 : cluster [DBG] pgmap v2054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 545 KiB/s rd, 828 KiB/s wr, 185 op/s 2022-04-22T18:08:02.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:02 smithi115 bash[46934]: cluster 2022-04-22T18:08:00.934925+0000 mgr.y (mgr.34107) 1480 : cluster [DBG] pgmap v2054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 545 KiB/s rd, 828 KiB/s wr, 185 op/s 2022-04-22T18:08:03.238 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:03 smithi121 bash[36902]: cluster 2022-04-22T18:08:02.203626+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2022-04-22T18:08:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:03 smithi121 bash[36902]: audit 2022-04-22T18:08:02.211998+0000 mon.a (mon.0) 3073 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:03 smithi121 bash[36902]: audit 2022-04-22T18:08:02.935492+0000 mon.a (mon.0) 3074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:03.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[45495]: cluster 2022-04-22T18:08:02.203626+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2022-04-22T18:08:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[45495]: audit 2022-04-22T18:08:02.211998+0000 mon.a (mon.0) 3073 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[45495]: audit 2022-04-22T18:08:02.935492+0000 mon.a (mon.0) 3074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[46934]: cluster 2022-04-22T18:08:02.203626+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2022-04-22T18:08:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[46934]: audit 2022-04-22T18:08:02.211998+0000 mon.a (mon.0) 3073 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[46934]: audit 2022-04-22T18:08:02.935492+0000 mon.a (mon.0) 3074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:03.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:04 smithi121 bash[36902]: cluster 2022-04-22T18:08:02.935773+0000 mgr.y (mgr.34107) 1481 : cluster [DBG] pgmap v2056: 137 pgs: 32 unknown, 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 340 KiB/s rd, 770 KiB/s wr, 108 op/s 2022-04-22T18:08:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:04 smithi121 bash[36902]: audit 2022-04-22T18:08:03.207183+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:04 smithi121 bash[36902]: cluster 2022-04-22T18:08:03.207690+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2022-04-22T18:08:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:04 smithi121 bash[36902]: cluster 2022-04-22T18:08:04.211818+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2022-04-22T18:08:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[45495]: cluster 2022-04-22T18:08:02.935773+0000 mgr.y (mgr.34107) 1481 : cluster [DBG] pgmap v2056: 137 pgs: 32 unknown, 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 340 KiB/s rd, 770 KiB/s wr, 108 op/s 2022-04-22T18:08:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[45495]: audit 2022-04-22T18:08:03.207183+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[45495]: cluster 2022-04-22T18:08:03.207690+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2022-04-22T18:08:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[45495]: cluster 2022-04-22T18:08:04.211818+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2022-04-22T18:08:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[46934]: cluster 2022-04-22T18:08:02.935773+0000 mgr.y (mgr.34107) 1481 : cluster [DBG] pgmap v2056: 137 pgs: 32 unknown, 105 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 340 KiB/s rd, 770 KiB/s wr, 108 op/s 2022-04-22T18:08:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[46934]: audit 2022-04-22T18:08:03.207183+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.115:0/1435405857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-1","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[46934]: cluster 2022-04-22T18:08:03.207690+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2022-04-22T18:08:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:04 smithi115 bash[46934]: cluster 2022-04-22T18:08:04.211818+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2022-04-22T18:08:05.245 INFO:tasks.workunit.client.0.smithi115.stdout:image has size 2097152 and order 22 2022-04-22T18:08:05.246 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CreateAndStat (3572 ms) 2022-04-22T18:08:05.246 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2022-04-22T18:08:05.246 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:05.246 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (0 ms) 2022-04-22T18:08:05.246 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2022-04-22T18:08:05.247 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.267 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CreateAndStatPP (21 ms) 2022-04-22T18:08:05.267 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.GetId 2022-04-22T18:08:05.267 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.296 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.GetId (29 ms) 2022-04-22T18:08:05.296 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.GetIdPP 2022-04-22T18:08:05.296 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.320 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.GetIdPP (25 ms) 2022-04-22T18:08:05.320 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2022-04-22T18:08:05.320 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.344 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (24 ms) 2022-04-22T18:08:05.344 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2022-04-22T18:08:05.345 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.362 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (17 ms) 2022-04-22T18:08:05.362 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (0 ms) 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.OpenAio 2022-04-22T18:08:05.363 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.381 INFO:tasks.workunit.client.0.smithi115.stdout:image has size 2097152 and order 22 2022-04-22T18:08:05.381 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.OpenAio (19 ms) 2022-04-22T18:08:05.381 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.OpenAioFail 2022-04-22T18:08:05.384 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.OpenAioFail (3 ms) 2022-04-22T18:08:05.384 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.OpenAioPP 2022-04-22T18:08:05.384 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.409 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.OpenAioPP (26 ms) 2022-04-22T18:08:05.410 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2022-04-22T18:08:05.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.OpenAioFailPP (4 ms) 2022-04-22T18:08:05.414 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2022-04-22T18:08:05.414 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.449 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ResizeAndStat (35 ms) 2022-04-22T18:08:05.449 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2022-04-22T18:08:05.449 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.480 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (31 ms) 2022-04-22T18:08:05.480 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2022-04-22T18:08:05.480 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.512 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (32 ms) 2022-04-22T18:08:05.512 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2022-04-22T18:08:05.512 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:05.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (37 ms) 2022-04-22T18:08:05.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2022-04-22T18:08:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:06 smithi121 bash[36902]: cluster 2022-04-22T18:08:04.936575+0000 mgr.y (mgr.34107) 1482 : cluster [DBG] pgmap v2059: 137 pgs: 12 creating+peering, 12 unknown, 113 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:08:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:06 smithi121 bash[36902]: cluster 2022-04-22T18:08:05.213016+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2022-04-22T18:08:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:06 smithi121 bash[36902]: audit 2022-04-22T18:08:05.550224+0000 mon.a (mon.0) 3079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:06 smithi121 bash[36902]: audit 2022-04-22T18:08:06.024604+0000 mon.a (mon.0) 3080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[45495]: cluster 2022-04-22T18:08:04.936575+0000 mgr.y (mgr.34107) 1482 : cluster [DBG] pgmap v2059: 137 pgs: 12 creating+peering, 12 unknown, 113 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[45495]: cluster 2022-04-22T18:08:05.213016+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[45495]: audit 2022-04-22T18:08:05.550224+0000 mon.a (mon.0) 3079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[45495]: audit 2022-04-22T18:08:06.024604+0000 mon.a (mon.0) 3080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[46934]: cluster 2022-04-22T18:08:04.936575+0000 mgr.y (mgr.34107) 1482 : cluster [DBG] pgmap v2059: 137 pgs: 12 creating+peering, 12 unknown, 113 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[46934]: cluster 2022-04-22T18:08:05.213016+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2022-04-22T18:08:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[46934]: audit 2022-04-22T18:08:05.550224+0000 mon.a (mon.0) 3079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:06 smithi115 bash[46934]: audit 2022-04-22T18:08:06.024604+0000 mon.a (mon.0) 3080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:07.254 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:07.478 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[45495]: cluster 2022-04-22T18:08:06.234633+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[45495]: audit 2022-04-22T18:08:06.239796+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[45495]: audit 2022-04-22T18:08:07.156181+0000 mon.a (mon.0) 3083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[45495]: audit 2022-04-22T18:08:07.159368+0000 mon.a (mon.0) 3084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[45495]: audit 2022-04-22T18:08:07.161464+0000 mon.a (mon.0) 3085 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[46934]: cluster 2022-04-22T18:08:06.234633+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2022-04-22T18:08:07.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[46934]: audit 2022-04-22T18:08:06.239796+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:07.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[46934]: audit 2022-04-22T18:08:07.156181+0000 mon.a (mon.0) 3083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:07.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[46934]: audit 2022-04-22T18:08:07.159368+0000 mon.a (mon.0) 3084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:07.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:07 smithi115 bash[46934]: audit 2022-04-22T18:08:07.161464+0000 mon.a (mon.0) 3085 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:07.492 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:07 smithi121 bash[36902]: cluster 2022-04-22T18:08:06.234633+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2022-04-22T18:08:07.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:07 smithi121 bash[36902]: audit 2022-04-22T18:08:06.239796+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:07.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:07 smithi121 bash[36902]: audit 2022-04-22T18:08:07.156181+0000 mon.a (mon.0) 3083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:07.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:07 smithi121 bash[36902]: audit 2022-04-22T18:08:07.159368+0000 mon.a (mon.0) 3084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:07.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:07 smithi121 bash[36902]: audit 2022-04-22T18:08:07.161464+0000 mon.a (mon.0) 3085 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:08.131 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:08:08.484 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:08:08.484 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (31m) 10m ago 45m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (44m) 71s ago 44m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (33m) 10m ago 46m 19.9M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (32m) 71s ago 45m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (40m) 71s ago 50m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (41m) 10m ago 52m 464M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (40m) 10m ago 52m 140M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (39m) 71s ago 50m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (39m) 10m ago 50m 95.4M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:08:08.485 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (32m) 10m ago 45m 9.83M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (32m) 71s ago 45m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (37m) 10m ago 49m 600M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (37m) 10m ago 49m 686M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (36m) 10m ago 48m 436M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (36m) 10m ago 48m 388M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (35m) 71s ago 47m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (34m) 71s ago 47m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (34m) 71s ago 47m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:08:08.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (33m) 71s ago 46m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:08:08.487 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (31m) 71s ago 45m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:08:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: cluster 2022-04-22T18:08:06.937520+0000 mgr.y (mgr.34107) 1483 : cluster [DBG] pgmap v2062: 169 pgs: 16 creating+peering, 30 unknown, 123 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2022-04-22T18:08:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: audit 2022-04-22T18:08:07.241360+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: cluster 2022-04-22T18:08:07.241512+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2022-04-22T18:08:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: audit 2022-04-22T18:08:07.488029+0000 mon.a (mon.0) 3088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: audit 2022-04-22T18:08:07.502242+0000 mon.a (mon.0) 3089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:08 smithi121 bash[36902]: audit 2022-04-22T18:08:07.939109+0000 mon.a (mon.0) 3090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: cluster 2022-04-22T18:08:06.937520+0000 mgr.y (mgr.34107) 1483 : cluster [DBG] pgmap v2062: 169 pgs: 16 creating+peering, 30 unknown, 123 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2022-04-22T18:08:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: audit 2022-04-22T18:08:07.241360+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: cluster 2022-04-22T18:08:07.241512+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: audit 2022-04-22T18:08:07.488029+0000 mon.a (mon.0) 3088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: audit 2022-04-22T18:08:07.502242+0000 mon.a (mon.0) 3089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[45495]: audit 2022-04-22T18:08:07.939109+0000 mon.a (mon.0) 3090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: cluster 2022-04-22T18:08:06.937520+0000 mgr.y (mgr.34107) 1483 : cluster [DBG] pgmap v2062: 169 pgs: 16 creating+peering, 30 unknown, 123 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: audit 2022-04-22T18:08:07.241360+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? 172.21.15.115:0/131805764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-2","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: cluster 2022-04-22T18:08:07.241512+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2022-04-22T18:08:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: audit 2022-04-22T18:08:07.488029+0000 mon.a (mon.0) 3088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: audit 2022-04-22T18:08:07.502242+0000 mon.a (mon.0) 3089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:08 smithi115 bash[46934]: audit 2022-04-22T18:08:07.939109+0000 mon.a (mon.0) 3090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:08.900 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:08:08.901 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:08:09.279 INFO:tasks.workunit.client.0.smithi115.stdout:image: image15 2022-04-22T18:08:09.279 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image15 2022-04-22T18:08:09.279 INFO:tasks.workunit.client.0.smithi115.stdout:found image15 2022-04-22T18:08:09.280 INFO:tasks.workunit.client.0.smithi115.stdout:erased image15 2022-04-22T18:08:09.280 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:09.328 INFO:tasks.workunit.client.0.smithi115.stdout:image: image15 2022-04-22T18:08:09.328 INFO:tasks.workunit.client.0.smithi115.stdout:image: image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image15 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:found image15 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:erased image15 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:found image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:erased image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:image: image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:found image16 2022-04-22T18:08:09.329 INFO:tasks.workunit.client.0.smithi115.stdout:erased image16 2022-04-22T18:08:09.330 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3779 ms) 2022-04-22T18:08:09.330 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2022-04-22T18:08:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:09 smithi121 bash[36902]: audit 2022-04-22T18:08:08.109980+0000 mgr.y (mgr.34107) 1484 : audit [DBG] from='client.46576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:09 smithi121 bash[36902]: cluster 2022-04-22T18:08:08.246311+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2022-04-22T18:08:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:09 smithi121 bash[36902]: audit 2022-04-22T18:08:08.460072+0000 mgr.y (mgr.34107) 1485 : audit [DBG] from='client.36651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:09 smithi121 bash[36902]: audit 2022-04-22T18:08:08.894336+0000 mon.c (mon.1) 85 : audit [DBG] from='client.? 172.21.15.115:0/3639577612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[45495]: audit 2022-04-22T18:08:08.109980+0000 mgr.y (mgr.34107) 1484 : audit [DBG] from='client.46576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[45495]: cluster 2022-04-22T18:08:08.246311+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2022-04-22T18:08:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[45495]: audit 2022-04-22T18:08:08.460072+0000 mgr.y (mgr.34107) 1485 : audit [DBG] from='client.36651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[45495]: audit 2022-04-22T18:08:08.894336+0000 mon.c (mon.1) 85 : audit [DBG] from='client.? 172.21.15.115:0/3639577612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[46934]: audit 2022-04-22T18:08:08.109980+0000 mgr.y (mgr.34107) 1484 : audit [DBG] from='client.46576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[46934]: cluster 2022-04-22T18:08:08.246311+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2022-04-22T18:08:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[46934]: audit 2022-04-22T18:08:08.460072+0000 mgr.y (mgr.34107) 1485 : audit [DBG] from='client.36651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:09 smithi115 bash[46934]: audit 2022-04-22T18:08:08.894336+0000 mon.c (mon.1) 85 : audit [DBG] from='client.? 172.21.15.115:0/3639577612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:10 smithi121 bash[36902]: cluster 2022-04-22T18:08:08.938378+0000 mgr.y (mgr.34107) 1486 : cluster [DBG] pgmap v2065: 169 pgs: 7 creating+peering, 17 unknown, 145 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2022-04-22T18:08:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:10 smithi121 bash[36902]: cluster 2022-04-22T18:08:09.258018+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2022-04-22T18:08:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:10 smithi115 bash[45495]: cluster 2022-04-22T18:08:08.938378+0000 mgr.y (mgr.34107) 1486 : cluster [DBG] pgmap v2065: 169 pgs: 7 creating+peering, 17 unknown, 145 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2022-04-22T18:08:10.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:10 smithi115 bash[45495]: cluster 2022-04-22T18:08:09.258018+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2022-04-22T18:08:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:10 smithi115 bash[46934]: cluster 2022-04-22T18:08:08.938378+0000 mgr.y (mgr.34107) 1486 : cluster [DBG] pgmap v2065: 169 pgs: 7 creating+peering, 17 unknown, 145 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2022-04-22T18:08:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:10 smithi115 bash[46934]: cluster 2022-04-22T18:08:09.258018+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2022-04-22T18:08:11.311 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[45495]: cluster 2022-04-22T18:08:10.267918+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2022-04-22T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[45495]: audit 2022-04-22T18:08:10.269708+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.115:0/3541630981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[45495]: audit 2022-04-22T18:08:10.270294+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[45495]: audit 2022-04-22T18:08:10.552876+0000 mon.a (mon.0) 3095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[45495]: audit 2022-04-22T18:08:10.973602+0000 mon.a (mon.0) 3096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[46934]: cluster 2022-04-22T18:08:10.267918+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2022-04-22T18:08:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[46934]: audit 2022-04-22T18:08:10.269708+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.115:0/3541630981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[46934]: audit 2022-04-22T18:08:10.270294+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[46934]: audit 2022-04-22T18:08:10.552876+0000 mon.a (mon.0) 3095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[46934]: audit 2022-04-22T18:08:10.973602+0000 mon.a (mon.0) 3096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:11 smithi121 bash[36902]: cluster 2022-04-22T18:08:10.267918+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2022-04-22T18:08:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:11 smithi121 bash[36902]: audit 2022-04-22T18:08:10.269708+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.115:0/3541630981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:11 smithi121 bash[36902]: audit 2022-04-22T18:08:10.270294+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:11 smithi121 bash[36902]: audit 2022-04-22T18:08:10.552876+0000 mon.a (mon.0) 3095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:11 smithi121 bash[36902]: audit 2022-04-22T18:08:10.973602+0000 mon.a (mon.0) 3096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:11.973 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:11.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[45495]: cluster 2022-04-22T18:08:10.939584+0000 mgr.y (mgr.34107) 1487 : cluster [DBG] pgmap v2068: 201 pgs: 12 creating+peering, 27 unknown, 162 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 32 op/s 2022-04-22T18:08:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[45495]: audit 2022-04-22T18:08:11.285668+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[45495]: cluster 2022-04-22T18:08:11.285981+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2022-04-22T18:08:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[45495]: audit 2022-04-22T18:08:11.627850+0000 mon.a (mon.0) 3099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:12.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[45495]: audit 2022-04-22T18:08:12.250207+0000 mon.a (mon.0) 3100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[46934]: cluster 2022-04-22T18:08:10.939584+0000 mgr.y (mgr.34107) 1487 : cluster [DBG] pgmap v2068: 201 pgs: 12 creating+peering, 27 unknown, 162 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 32 op/s 2022-04-22T18:08:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[46934]: audit 2022-04-22T18:08:11.285668+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[46934]: cluster 2022-04-22T18:08:11.285981+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2022-04-22T18:08:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[46934]: audit 2022-04-22T18:08:11.627850+0000 mon.a (mon.0) 3099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:12.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:12 smithi115 bash[46934]: audit 2022-04-22T18:08:12.250207+0000 mon.a (mon.0) 3100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:12 smithi121 bash[36902]: cluster 2022-04-22T18:08:10.939584+0000 mgr.y (mgr.34107) 1487 : cluster [DBG] pgmap v2068: 201 pgs: 12 creating+peering, 27 unknown, 162 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 32 op/s 2022-04-22T18:08:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:12 smithi121 bash[36902]: audit 2022-04-22T18:08:11.285668+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-3","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:12 smithi121 bash[36902]: cluster 2022-04-22T18:08:11.285981+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2022-04-22T18:08:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:12 smithi121 bash[36902]: audit 2022-04-22T18:08:11.627850+0000 mon.a (mon.0) 3099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:12 smithi121 bash[36902]: audit 2022-04-22T18:08:12.250207+0000 mon.a (mon.0) 3100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:13.316 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 1 2022-04-22T18:08:13.316 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:13.317 INFO:tasks.workunit.client.0.smithi115.stdout:image: image17 2022-04-22T18:08:13.317 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image17 2022-04-22T18:08:13.317 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:13.322 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 2 2022-04-22T18:08:13.322 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:13.322 INFO:tasks.workunit.client.0.smithi115.stdout:image: image17 2022-04-22T18:08:13.322 INFO:tasks.workunit.client.0.smithi115.stdout:image: image18 2022-04-22T18:08:13.322 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image17 2022-04-22T18:08:13.323 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image18 2022-04-22T18:08:13.342 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 1 2022-04-22T18:08:13.342 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:13.342 INFO:tasks.workunit.client.0.smithi115.stdout:image: image18 2022-04-22T18:08:13.343 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image18 2022-04-22T18:08:13.343 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (4013 ms) 2022-04-22T18:08:13.343 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCopy 2022-04-22T18:08:13.639 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:13 smithi115 bash[45495]: cluster 2022-04-22T18:08:12.285245+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2022-04-22T18:08:13.640 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:13 smithi115 bash[45495]: audit 2022-04-22T18:08:12.943127+0000 mon.a (mon.0) 3102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:13.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:13 smithi115 bash[46934]: cluster 2022-04-22T18:08:12.285245+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2022-04-22T18:08:13.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:13 smithi115 bash[46934]: audit 2022-04-22T18:08:12.943127+0000 mon.a (mon.0) 3102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:13 smithi121 bash[36902]: cluster 2022-04-22T18:08:12.285245+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2022-04-22T18:08:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:13 smithi121 bash[36902]: audit 2022-04-22T18:08:12.943127+0000 mon.a (mon.0) 3102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:13.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:14 smithi121 bash[36902]: cluster 2022-04-22T18:08:12.940508+0000 mgr.y (mgr.34107) 1488 : cluster [DBG] pgmap v2071: 201 pgs: 5 creating+peering, 24 unknown, 172 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 KiB/s wr, 23 op/s 2022-04-22T18:08:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:14 smithi121 bash[36902]: cluster 2022-04-22T18:08:13.286000+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2022-04-22T18:08:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:14 smithi121 bash[36902]: audit 2022-04-22T18:08:13.812521+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:08:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:14 smithi121 bash[36902]: audit 2022-04-22T18:08:13.813052+0000 mon.a (mon.0) 3105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:08:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[45495]: cluster 2022-04-22T18:08:12.940508+0000 mgr.y (mgr.34107) 1488 : cluster [DBG] pgmap v2071: 201 pgs: 5 creating+peering, 24 unknown, 172 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 KiB/s wr, 23 op/s 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[45495]: cluster 2022-04-22T18:08:13.286000+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[45495]: audit 2022-04-22T18:08:13.812521+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[45495]: audit 2022-04-22T18:08:13.813052+0000 mon.a (mon.0) 3105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[46934]: cluster 2022-04-22T18:08:12.940508+0000 mgr.y (mgr.34107) 1488 : cluster [DBG] pgmap v2071: 201 pgs: 5 creating+peering, 24 unknown, 172 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.0 KiB/s wr, 23 op/s 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[46934]: cluster 2022-04-22T18:08:13.286000+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[46934]: audit 2022-04-22T18:08:13.812521+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:08:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:14 smithi115 bash[46934]: audit 2022-04-22T18:08:13.813052+0000 mon.a (mon.0) 3105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:08:15.333 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:15 smithi115 bash[46934]: cluster 2022-04-22T18:08:14.311573+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2022-04-22T18:08:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:15 smithi115 bash[46934]: audit 2022-04-22T18:08:14.318898+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:15 smithi115 bash[45495]: cluster 2022-04-22T18:08:14.311573+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2022-04-22T18:08:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:15 smithi115 bash[45495]: audit 2022-04-22T18:08:14.318898+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:15 smithi121 bash[36902]: cluster 2022-04-22T18:08:14.311573+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2022-04-22T18:08:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:15 smithi121 bash[36902]: audit 2022-04-22T18:08:14.318898+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: cluster 2022-04-22T18:08:14.941599+0000 mgr.y (mgr.34107) 1489 : cluster [DBG] pgmap v2074: 233 pgs: 6 creating+peering, 34 unknown, 193 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:16.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: audit 2022-04-22T18:08:15.320671+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:16.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: cluster 2022-04-22T18:08:15.320841+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2022-04-22T18:08:16.796 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: audit 2022-04-22T18:08:15.555601+0000 mon.a (mon.0) 3110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:16.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: audit 2022-04-22T18:08:15.976656+0000 mon.a (mon.0) 3111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:16.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:16 smithi121 bash[36902]: cluster 2022-04-22T18:08:16.035150+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2022-04-22T18:08:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: cluster 2022-04-22T18:08:14.941599+0000 mgr.y (mgr.34107) 1489 : cluster [DBG] pgmap v2074: 233 pgs: 6 creating+peering, 34 unknown, 193 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: audit 2022-04-22T18:08:15.320671+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: cluster 2022-04-22T18:08:15.320841+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: audit 2022-04-22T18:08:15.555601+0000 mon.a (mon.0) 3110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: audit 2022-04-22T18:08:15.976656+0000 mon.a (mon.0) 3111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[45495]: cluster 2022-04-22T18:08:16.035150+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: cluster 2022-04-22T18:08:14.941599+0000 mgr.y (mgr.34107) 1489 : cluster [DBG] pgmap v2074: 233 pgs: 6 creating+peering, 34 unknown, 193 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: audit 2022-04-22T18:08:15.320671+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? 172.21.15.115:0/1203366710' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-4","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: cluster 2022-04-22T18:08:15.320841+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2022-04-22T18:08:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: audit 2022-04-22T18:08:15.555601+0000 mon.a (mon.0) 3110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: audit 2022-04-22T18:08:15.976656+0000 mon.a (mon.0) 3111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:16 smithi115 bash[46934]: cluster 2022-04-22T18:08:16.035150+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2022-04-22T18:08:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[46934]: cluster 2022-04-22T18:08:16.942746+0000 mgr.y (mgr.34107) 1490 : cluster [DBG] pgmap v2077: 233 pgs: 10 creating+peering, 15 unknown, 208 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[46934]: cluster 2022-04-22T18:08:17.036075+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2022-04-22T18:08:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[46934]: audit 2022-04-22T18:08:17.945428+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[45495]: cluster 2022-04-22T18:08:16.942746+0000 mgr.y (mgr.34107) 1490 : cluster [DBG] pgmap v2077: 233 pgs: 10 creating+peering, 15 unknown, 208 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[45495]: cluster 2022-04-22T18:08:17.036075+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2022-04-22T18:08:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:18 smithi115 bash[45495]: audit 2022-04-22T18:08:17.945428+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:18 smithi121 bash[36902]: cluster 2022-04-22T18:08:16.942746+0000 mgr.y (mgr.34107) 1490 : cluster [DBG] pgmap v2077: 233 pgs: 10 creating+peering, 15 unknown, 208 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2022-04-22T18:08:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:18 smithi121 bash[36902]: cluster 2022-04-22T18:08:17.036075+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2022-04-22T18:08:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:18 smithi121 bash[36902]: audit 2022-04-22T18:08:17.945428+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:19.522 INFO:tasks.workunit.client.0.smithi115.stdout:image: image19 2022-04-22T18:08:19.522 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:found image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:erased image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:image: image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:image: image20 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:found image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:erased image19 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image20 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:found image20 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:erased image20 2022-04-22T18:08:19.523 INFO:tasks.workunit.client.0.smithi115.stdout:0.00% done 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:100.00% done 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:image: image19 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:image: image20 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:image: image21 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image19 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:found image19 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:erased image19 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image20 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:found image20 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:erased image20 2022-04-22T18:08:19.524 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image21 2022-04-22T18:08:19.525 INFO:tasks.workunit.client.0.smithi115.stdout:found image21 2022-04-22T18:08:19.525 INFO:tasks.workunit.client.0.smithi115.stdout:erased image21 2022-04-22T18:08:19.525 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCopy (6181 ms) 2022-04-22T18:08:19.525 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCopyPP 2022-04-22T18:08:19.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:19 smithi115 bash[45495]: cluster 2022-04-22T18:08:18.328371+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2022-04-22T18:08:19.671 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:19 smithi115 bash[46934]: cluster 2022-04-22T18:08:18.328371+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2022-04-22T18:08:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:19 smithi121 bash[36902]: cluster 2022-04-22T18:08:18.328371+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2022-04-22T18:08:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:20 smithi121 bash[36902]: cluster 2022-04-22T18:08:18.943373+0000 mgr.y (mgr.34107) 1491 : cluster [DBG] pgmap v2080: 233 pgs: 10 creating+peering, 5 unknown, 218 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T18:08:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:20 smithi121 bash[36902]: cluster 2022-04-22T18:08:19.328517+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2022-04-22T18:08:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:20 smithi115 bash[45495]: cluster 2022-04-22T18:08:18.943373+0000 mgr.y (mgr.34107) 1491 : cluster [DBG] pgmap v2080: 233 pgs: 10 creating+peering, 5 unknown, 218 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T18:08:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:20 smithi115 bash[45495]: cluster 2022-04-22T18:08:19.328517+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2022-04-22T18:08:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:20 smithi115 bash[46934]: cluster 2022-04-22T18:08:18.943373+0000 mgr.y (mgr.34107) 1491 : cluster [DBG] pgmap v2080: 233 pgs: 10 creating+peering, 5 unknown, 218 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2022-04-22T18:08:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:20 smithi115 bash[46934]: cluster 2022-04-22T18:08:19.328517+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2022-04-22T18:08:21.060 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: cluster 2022-04-22T18:08:20.346128+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2022-04-22T18:08:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: audit 2022-04-22T18:08:20.351235+0000 mon.c (mon.1) 87 : audit [INF] from='client.? 172.21.15.115:0/511085426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: audit 2022-04-22T18:08:20.351760+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: audit 2022-04-22T18:08:20.558612+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:21.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: audit 2022-04-22T18:08:20.979939+0000 mon.a (mon.0) 3120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: audit 2022-04-22T18:08:21.035676+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[46934]: cluster 2022-04-22T18:08:21.035858+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: cluster 2022-04-22T18:08:20.346128+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: audit 2022-04-22T18:08:20.351235+0000 mon.c (mon.1) 87 : audit [INF] from='client.? 172.21.15.115:0/511085426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: audit 2022-04-22T18:08:20.351760+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: audit 2022-04-22T18:08:20.558612+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: audit 2022-04-22T18:08:20.979939+0000 mon.a (mon.0) 3120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:21.758 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: audit 2022-04-22T18:08:21.035676+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:21.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[45495]: cluster 2022-04-22T18:08:21.035858+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2022-04-22T18:08:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: cluster 2022-04-22T18:08:20.346128+0000 mon.a (mon.0) 3117 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: audit 2022-04-22T18:08:20.351235+0000 mon.c (mon.1) 87 : audit [INF] from='client.? 172.21.15.115:0/511085426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: audit 2022-04-22T18:08:20.351760+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: audit 2022-04-22T18:08:20.558612+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: audit 2022-04-22T18:08:20.979939+0000 mon.a (mon.0) 3120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: audit 2022-04-22T18:08:21.035676+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-5","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:21 smithi121 bash[36902]: cluster 2022-04-22T18:08:21.035858+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2022-04-22T18:08:22.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:21.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:22 smithi121 bash[36902]: cluster 2022-04-22T18:08:20.944580+0000 mgr.y (mgr.34107) 1492 : cluster [DBG] pgmap v2083: 265 pgs: 8 creating+peering, 24 unknown, 233 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 19 KiB/s wr, 76 op/s 2022-04-22T18:08:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:22 smithi121 bash[36902]: cluster 2022-04-22T18:08:22.052433+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2022-04-22T18:08:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:22 smithi115 bash[45495]: cluster 2022-04-22T18:08:20.944580+0000 mgr.y (mgr.34107) 1492 : cluster [DBG] pgmap v2083: 265 pgs: 8 creating+peering, 24 unknown, 233 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 19 KiB/s wr, 76 op/s 2022-04-22T18:08:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:22 smithi115 bash[45495]: cluster 2022-04-22T18:08:22.052433+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2022-04-22T18:08:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:22 smithi115 bash[46934]: cluster 2022-04-22T18:08:20.944580+0000 mgr.y (mgr.34107) 1492 : cluster [DBG] pgmap v2083: 265 pgs: 8 creating+peering, 24 unknown, 233 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 19 KiB/s wr, 76 op/s 2022-04-22T18:08:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:22 smithi115 bash[46934]: cluster 2022-04-22T18:08:22.052433+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2022-04-22T18:08:23.391 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 1 2022-04-22T18:08:23.391 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:23.391 INFO:tasks.workunit.client.0.smithi115.stdout:image: image22 2022-04-22T18:08:23.392 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image22 2022-04-22T18:08:23.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:23 smithi115 bash[45495]: audit 2022-04-22T18:08:22.948256+0000 mon.a (mon.0) 3124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:23.641 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:23 smithi115 bash[45495]: cluster 2022-04-22T18:08:23.055272+0000 mon.a (mon.0) 3125 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2022-04-22T18:08:23.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:23 smithi115 bash[46934]: audit 2022-04-22T18:08:22.948256+0000 mon.a (mon.0) 3124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:23.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:23 smithi115 bash[46934]: cluster 2022-04-22T18:08:23.055272+0000 mon.a (mon.0) 3125 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2022-04-22T18:08:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:23 smithi121 bash[36902]: audit 2022-04-22T18:08:22.948256+0000 mon.a (mon.0) 3124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:23 smithi121 bash[36902]: cluster 2022-04-22T18:08:23.055272+0000 mon.a (mon.0) 3125 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2022-04-22T18:08:24.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:23.636Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[45495]: cluster 2022-04-22T18:08:22.945694+0000 mgr.y (mgr.34107) 1493 : cluster [DBG] pgmap v2086: 265 pgs: 8 creating+peering, 17 unknown, 240 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 19 KiB/s wr, 80 op/s 2022-04-22T18:08:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[45495]: audit 2022-04-22T18:08:24.279499+0000 mon.a (mon.0) 3126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[45495]: audit 2022-04-22T18:08:24.288359+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[46934]: cluster 2022-04-22T18:08:22.945694+0000 mgr.y (mgr.34107) 1493 : cluster [DBG] pgmap v2086: 265 pgs: 8 creating+peering, 17 unknown, 240 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 19 KiB/s wr, 80 op/s 2022-04-22T18:08:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[46934]: audit 2022-04-22T18:08:24.279499+0000 mon.a (mon.0) 3126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:24 smithi115 bash[46934]: audit 2022-04-22T18:08:24.288359+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:24 smithi121 bash[36902]: cluster 2022-04-22T18:08:22.945694+0000 mgr.y (mgr.34107) 1493 : cluster [DBG] pgmap v2086: 265 pgs: 8 creating+peering, 17 unknown, 240 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 19 KiB/s wr, 80 op/s 2022-04-22T18:08:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:24 smithi121 bash[36902]: audit 2022-04-22T18:08:24.279499+0000 mon.a (mon.0) 3126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:24 smithi121 bash[36902]: audit 2022-04-22T18:08:24.288359+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:25.453 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 2 2022-04-22T18:08:25.454 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:25.454 INFO:tasks.workunit.client.0.smithi115.stdout:image: image22 2022-04-22T18:08:25.454 INFO:tasks.workunit.client.0.smithi115.stdout:image: image23 2022-04-22T18:08:25.454 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image22 2022-04-22T18:08:25.454 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image23 2022-04-22T18:08:25.566 INFO:tasks.workunit.client.0.smithi115.stdout:0.00% done 2022-04-22T18:08:25.566 INFO:tasks.workunit.client.0.smithi115.stdout:100.00% done 2022-04-22T18:08:25.566 INFO:tasks.workunit.client.0.smithi115.stdout:num images is: 3 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 3 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:image: image22 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:image: image23 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:image: image24 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image22 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image23 2022-04-22T18:08:25.567 INFO:tasks.workunit.client.0.smithi115.stdout:expected = image24 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCopyPP (6119 ms) 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestDeepCopy (0 ms) 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2022-04-22T18:08:25.642 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:25.643 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (0 ms) 2022-04-22T18:08:25.643 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2022-04-22T18:08:25.643 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:25 smithi121 bash[36902]: cephadm 2022-04-22T18:08:24.269895+0000 mgr.y (mgr.34107) 1494 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:08:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:25 smithi121 bash[36902]: cluster 2022-04-22T18:08:24.371264+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2022-04-22T18:08:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:25 smithi115 bash[45495]: cephadm 2022-04-22T18:08:24.269895+0000 mgr.y (mgr.34107) 1494 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:08:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:25 smithi115 bash[45495]: cluster 2022-04-22T18:08:24.371264+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2022-04-22T18:08:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:25 smithi115 bash[46934]: cephadm 2022-04-22T18:08:24.269895+0000 mgr.y (mgr.34107) 1494 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:08:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:25 smithi115 bash[46934]: cluster 2022-04-22T18:08:24.371264+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2022-04-22T18:08:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:26 smithi121 bash[36902]: cluster 2022-04-22T18:08:24.946899+0000 mgr.y (mgr.34107) 1495 : cluster [DBG] pgmap v2089: 265 pgs: 8 creating+peering, 3 unknown, 254 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 20 KiB/s wr, 136 op/s 2022-04-22T18:08:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:26 smithi121 bash[36902]: cluster 2022-04-22T18:08:25.397520+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2022-04-22T18:08:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:26 smithi121 bash[36902]: audit 2022-04-22T18:08:25.561090+0000 mon.a (mon.0) 3130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:26 smithi121 bash[36902]: audit 2022-04-22T18:08:25.983337+0000 mon.a (mon.0) 3131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:26 smithi121 bash[36902]: cluster 2022-04-22T18:08:26.040030+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2022-04-22T18:08:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[45495]: cluster 2022-04-22T18:08:24.946899+0000 mgr.y (mgr.34107) 1495 : cluster [DBG] pgmap v2089: 265 pgs: 8 creating+peering, 3 unknown, 254 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 20 KiB/s wr, 136 op/s 2022-04-22T18:08:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[45495]: cluster 2022-04-22T18:08:25.397520+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2022-04-22T18:08:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[45495]: audit 2022-04-22T18:08:25.561090+0000 mon.a (mon.0) 3130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[45495]: audit 2022-04-22T18:08:25.983337+0000 mon.a (mon.0) 3131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[45495]: cluster 2022-04-22T18:08:26.040030+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[46934]: cluster 2022-04-22T18:08:24.946899+0000 mgr.y (mgr.34107) 1495 : cluster [DBG] pgmap v2089: 265 pgs: 8 creating+peering, 3 unknown, 254 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 20 KiB/s wr, 136 op/s 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[46934]: cluster 2022-04-22T18:08:25.397520+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[46934]: audit 2022-04-22T18:08:25.561090+0000 mon.a (mon.0) 3130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[46934]: audit 2022-04-22T18:08:25.983337+0000 mon.a (mon.0) 3131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:26 smithi115 bash[46934]: cluster 2022-04-22T18:08:26.040030+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2022-04-22T18:08:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:28 smithi121 bash[36902]: cluster 2022-04-22T18:08:26.948107+0000 mgr.y (mgr.34107) 1496 : cluster [DBG] pgmap v2092: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 20 KiB/s wr, 132 op/s 2022-04-22T18:08:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:28 smithi121 bash[36902]: cluster 2022-04-22T18:08:27.054753+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2022-04-22T18:08:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:28 smithi121 bash[36902]: audit 2022-04-22T18:08:27.950845+0000 mon.a (mon.0) 3134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[46934]: cluster 2022-04-22T18:08:26.948107+0000 mgr.y (mgr.34107) 1496 : cluster [DBG] pgmap v2092: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 20 KiB/s wr, 132 op/s 2022-04-22T18:08:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[46934]: cluster 2022-04-22T18:08:27.054753+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2022-04-22T18:08:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[46934]: audit 2022-04-22T18:08:27.950845+0000 mon.a (mon.0) 3134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[45495]: cluster 2022-04-22T18:08:26.948107+0000 mgr.y (mgr.34107) 1496 : cluster [DBG] pgmap v2092: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 20 KiB/s wr, 132 op/s 2022-04-22T18:08:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[45495]: cluster 2022-04-22T18:08:27.054753+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2022-04-22T18:08:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:28 smithi115 bash[45495]: audit 2022-04-22T18:08:27.950845+0000 mon.a (mon.0) 3134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:29.094 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:29.095 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3453 ms) 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:08:29.096 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2022-04-22T18:08:29.097 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2022-04-22T18:08:29.097 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:29 smithi121 bash[36902]: cluster 2022-04-22T18:08:28.055100+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2022-04-22T18:08:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:29 smithi115 bash[46934]: cluster 2022-04-22T18:08:28.055100+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2022-04-22T18:08:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:29 smithi115 bash[45495]: cluster 2022-04-22T18:08:28.055100+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2022-04-22T18:08:30.092 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:30.093 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:30.093 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:30.093 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:30 smithi121 bash[36902]: cluster 2022-04-22T18:08:28.949370+0000 mgr.y (mgr.34107) 1497 : cluster [DBG] pgmap v2095: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 KiB/s wr, 36 op/s 2022-04-22T18:08:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:30 smithi121 bash[36902]: cluster 2022-04-22T18:08:29.075427+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2022-04-22T18:08:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:30 smithi115 bash[45495]: cluster 2022-04-22T18:08:28.949370+0000 mgr.y (mgr.34107) 1497 : cluster [DBG] pgmap v2095: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 KiB/s wr, 36 op/s 2022-04-22T18:08:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:30 smithi115 bash[45495]: cluster 2022-04-22T18:08:29.075427+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2022-04-22T18:08:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:30 smithi115 bash[46934]: cluster 2022-04-22T18:08:28.949370+0000 mgr.y (mgr.34107) 1497 : cluster [DBG] pgmap v2095: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 KiB/s wr, 36 op/s 2022-04-22T18:08:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:30 smithi115 bash[46934]: cluster 2022-04-22T18:08:29.075427+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2022-04-22T18:08:31.060 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:08:31.061 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:31.061 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:31.061 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:31.061 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:31.061 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:31 smithi121 bash[36902]: cluster 2022-04-22T18:08:30.075212+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2022-04-22T18:08:31.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:31 smithi121 bash[36902]: audit 2022-04-22T18:08:30.563984+0000 mon.a (mon.0) 3138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:31.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:31 smithi121 bash[36902]: audit 2022-04-22T18:08:30.986965+0000 mon.a (mon.0) 3139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:31.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:31 smithi121 bash[36902]: cluster 2022-04-22T18:08:31.041313+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2022-04-22T18:08:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[45495]: cluster 2022-04-22T18:08:30.075212+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[45495]: audit 2022-04-22T18:08:30.563984+0000 mon.a (mon.0) 3138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[45495]: audit 2022-04-22T18:08:30.986965+0000 mon.a (mon.0) 3139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[45495]: cluster 2022-04-22T18:08:31.041313+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[46934]: cluster 2022-04-22T18:08:30.075212+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[46934]: audit 2022-04-22T18:08:30.563984+0000 mon.a (mon.0) 3138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[46934]: audit 2022-04-22T18:08:30.986965+0000 mon.a (mon.0) 3139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[46934]: cluster 2022-04-22T18:08:31.041313+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e1104: 8 total, 8 up, 8 in 2022-04-22T18:08:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:31.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:32.074 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:32.099 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:32.100 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:32.100 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:32 smithi121 bash[36902]: cluster 2022-04-22T18:08:30.950621+0000 mgr.y (mgr.34107) 1498 : cluster [DBG] pgmap v2098: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.0 KiB/s wr, 80 op/s 2022-04-22T18:08:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:32 smithi121 bash[36902]: cluster 2022-04-22T18:08:32.057192+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e1105: 8 total, 8 up, 8 in 2022-04-22T18:08:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:32 smithi115 bash[45495]: cluster 2022-04-22T18:08:30.950621+0000 mgr.y (mgr.34107) 1498 : cluster [DBG] pgmap v2098: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.0 KiB/s wr, 80 op/s 2022-04-22T18:08:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:32 smithi115 bash[45495]: cluster 2022-04-22T18:08:32.057192+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e1105: 8 total, 8 up, 8 in 2022-04-22T18:08:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:32 smithi115 bash[46934]: cluster 2022-04-22T18:08:30.950621+0000 mgr.y (mgr.34107) 1498 : cluster [DBG] pgmap v2098: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.0 KiB/s wr, 80 op/s 2022-04-22T18:08:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:32 smithi115 bash[46934]: cluster 2022-04-22T18:08:32.057192+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e1105: 8 total, 8 up, 8 in 2022-04-22T18:08:33.075 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:08:33.076 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:08:33.079 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (3984 ms) 2022-04-22T18:08:33.079 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2022-04-22T18:08:33.079 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:33 smithi121 bash[36902]: audit 2022-04-22T18:08:32.953963+0000 mon.a (mon.0) 3142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:33 smithi121 bash[36902]: cluster 2022-04-22T18:08:33.059040+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2022-04-22T18:08:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:33 smithi115 bash[45495]: audit 2022-04-22T18:08:32.953963+0000 mon.a (mon.0) 3142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:33 smithi115 bash[45495]: cluster 2022-04-22T18:08:33.059040+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2022-04-22T18:08:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:33 smithi115 bash[46934]: audit 2022-04-22T18:08:32.953963+0000 mon.a (mon.0) 3142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:33 smithi115 bash[46934]: cluster 2022-04-22T18:08:33.059040+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e1106: 8 total, 8 up, 8 in 2022-04-22T18:08:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:33.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:34 smithi121 bash[36902]: cluster 2022-04-22T18:08:32.951590+0000 mgr.y (mgr.34107) 1499 : cluster [DBG] pgmap v2101: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 4.0 KiB/s wr, 50 op/s 2022-04-22T18:08:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:34 smithi115 bash[45495]: cluster 2022-04-22T18:08:32.951590+0000 mgr.y (mgr.34107) 1499 : cluster [DBG] pgmap v2101: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 4.0 KiB/s wr, 50 op/s 2022-04-22T18:08:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:34 smithi115 bash[46934]: cluster 2022-04-22T18:08:32.951590+0000 mgr.y (mgr.34107) 1499 : cluster [DBG] pgmap v2101: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 4.0 KiB/s wr, 50 op/s 2022-04-22T18:08:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:35 smithi121 bash[36902]: cluster 2022-04-22T18:08:34.094531+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2022-04-22T18:08:35.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:35 smithi121 bash[36902]: audit 2022-04-22T18:08:34.296815+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:35 smithi115 bash[45495]: cluster 2022-04-22T18:08:34.094531+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2022-04-22T18:08:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:35 smithi115 bash[45495]: audit 2022-04-22T18:08:34.296815+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:35 smithi115 bash[46934]: cluster 2022-04-22T18:08:34.094531+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e1107: 8 total, 8 up, 8 in 2022-04-22T18:08:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:35 smithi115 bash[46934]: audit 2022-04-22T18:08:34.296815+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:36 smithi121 bash[36902]: cluster 2022-04-22T18:08:34.952789+0000 mgr.y (mgr.34107) 1500 : cluster [DBG] pgmap v2104: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2022-04-22T18:08:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:36 smithi121 bash[36902]: cluster 2022-04-22T18:08:35.110295+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2022-04-22T18:08:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:36 smithi121 bash[36902]: audit 2022-04-22T18:08:35.566989+0000 mon.a (mon.0) 3147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:36 smithi121 bash[36902]: audit 2022-04-22T18:08:35.990570+0000 mon.a (mon.0) 3148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:36 smithi121 bash[36902]: cluster 2022-04-22T18:08:36.045442+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e1109: 8 total, 8 up, 8 in 2022-04-22T18:08:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[45495]: cluster 2022-04-22T18:08:34.952789+0000 mgr.y (mgr.34107) 1500 : cluster [DBG] pgmap v2104: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2022-04-22T18:08:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[45495]: cluster 2022-04-22T18:08:35.110295+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2022-04-22T18:08:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[45495]: audit 2022-04-22T18:08:35.566989+0000 mon.a (mon.0) 3147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[45495]: audit 2022-04-22T18:08:35.990570+0000 mon.a (mon.0) 3148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:36.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[45495]: cluster 2022-04-22T18:08:36.045442+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e1109: 8 total, 8 up, 8 in 2022-04-22T18:08:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[46934]: cluster 2022-04-22T18:08:34.952789+0000 mgr.y (mgr.34107) 1500 : cluster [DBG] pgmap v2104: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2022-04-22T18:08:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[46934]: cluster 2022-04-22T18:08:35.110295+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e1108: 8 total, 8 up, 8 in 2022-04-22T18:08:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[46934]: audit 2022-04-22T18:08:35.566989+0000 mon.a (mon.0) 3147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[46934]: audit 2022-04-22T18:08:35.990570+0000 mon.a (mon.0) 3148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:36 smithi115 bash[46934]: cluster 2022-04-22T18:08:36.045442+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e1109: 8 total, 8 up, 8 in 2022-04-22T18:08:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:38 smithi121 bash[36902]: cluster 2022-04-22T18:08:36.953936+0000 mgr.y (mgr.34107) 1501 : cluster [DBG] pgmap v2107: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 2 op/s 2022-04-22T18:08:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:38 smithi121 bash[36902]: cluster 2022-04-22T18:08:37.058784+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e1110: 8 total, 8 up, 8 in 2022-04-22T18:08:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:38 smithi121 bash[36902]: audit 2022-04-22T18:08:37.956826+0000 mon.a (mon.0) 3151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[45495]: cluster 2022-04-22T18:08:36.953936+0000 mgr.y (mgr.34107) 1501 : cluster [DBG] pgmap v2107: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 2 op/s 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[45495]: cluster 2022-04-22T18:08:37.058784+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e1110: 8 total, 8 up, 8 in 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[45495]: audit 2022-04-22T18:08:37.956826+0000 mon.a (mon.0) 3151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[46934]: cluster 2022-04-22T18:08:36.953936+0000 mgr.y (mgr.34107) 1501 : cluster [DBG] pgmap v2107: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 2 op/s 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[46934]: cluster 2022-04-22T18:08:37.058784+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e1110: 8 total, 8 up, 8 in 2022-04-22T18:08:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:38 smithi115 bash[46934]: audit 2022-04-22T18:08:37.956826+0000 mon.a (mon.0) 3151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:39.100 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:08:39.100 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:08:39.102 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6024 ms) 2022-04-22T18:08:39.102 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2022-04-22T18:08:39.102 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:39.293 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:08:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:39 smithi121 bash[36902]: cluster 2022-04-22T18:08:38.067468+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2022-04-22T18:08:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:39 smithi115 bash[45495]: cluster 2022-04-22T18:08:38.067468+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2022-04-22T18:08:39.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:39 smithi115 bash[46934]: cluster 2022-04-22T18:08:38.067468+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e1111: 8 total, 8 up, 8 in 2022-04-22T18:08:39.643 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:08:39.643 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (31m) 28s ago 45m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:08:39.643 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (45m) 102s ago 45m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:08:39.643 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (33m) 28s ago 46m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (33m) 102s ago 46m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (40m) 102s ago 50m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (41m) 28s ago 52m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (40m) 28s ago 53m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (39m) 102s ago 51m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (40m) 28s ago 51m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (32m) 28s ago 45m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (32m) 102s ago 45m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (38m) 28s ago 50m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:08:39.644 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (37m) 28s ago 49m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (37m) 28s ago 49m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (36m) 28s ago 48m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (36m) 102s ago 48m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (35m) 102s ago 48m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (34m) 102s ago 47m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (34m) 102s ago 47m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:08:39.645 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (32m) 102s ago 45m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:40.046 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:08:40.047 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:08:40.103 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:40.103 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:40.103 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:40.103 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:40 smithi121 bash[36902]: cluster 2022-04-22T18:08:38.955165+0000 mgr.y (mgr.34107) 1502 : cluster [DBG] pgmap v2110: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2022-04-22T18:08:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:40 smithi121 bash[36902]: cluster 2022-04-22T18:08:39.077658+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2022-04-22T18:08:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:40 smithi121 bash[36902]: audit 2022-04-22T18:08:39.273364+0000 mgr.y (mgr.34107) 1503 : audit [DBG] from='client.46612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:40 smithi121 bash[36902]: audit 2022-04-22T18:08:40.040389+0000 mon.a (mon.0) 3154 : audit [DBG] from='client.? 172.21.15.115:0/3144845544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[45495]: cluster 2022-04-22T18:08:38.955165+0000 mgr.y (mgr.34107) 1502 : cluster [DBG] pgmap v2110: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2022-04-22T18:08:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[45495]: cluster 2022-04-22T18:08:39.077658+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2022-04-22T18:08:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[45495]: audit 2022-04-22T18:08:39.273364+0000 mgr.y (mgr.34107) 1503 : audit [DBG] from='client.46612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[45495]: audit 2022-04-22T18:08:40.040389+0000 mon.a (mon.0) 3154 : audit [DBG] from='client.? 172.21.15.115:0/3144845544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[46934]: cluster 2022-04-22T18:08:38.955165+0000 mgr.y (mgr.34107) 1502 : cluster [DBG] pgmap v2110: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2022-04-22T18:08:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[46934]: cluster 2022-04-22T18:08:39.077658+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e1112: 8 total, 8 up, 8 in 2022-04-22T18:08:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[46934]: audit 2022-04-22T18:08:39.273364+0000 mgr.y (mgr.34107) 1503 : audit [DBG] from='client.46612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:40 smithi115 bash[46934]: audit 2022-04-22T18:08:40.040389+0000 mon.a (mon.0) 3154 : audit [DBG] from='client.? 172.21.15.115:0/3144845544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:08:41.069 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:08:41.069 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:41.070 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:08:41.070 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:41.070 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1 with size 2097152 2022-04-22T18:08:41.070 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:41.072 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:08:41.072 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:08:41.073 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1-rename 2022-04-22T18:08:41.073 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:08:41.073 INFO:tasks.workunit.client.0.smithi115.stdout:found snap1-rename with size 2097152 2022-04-22T18:08:41.073 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2 with size 4194304 2022-04-22T18:08:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:41 smithi121 bash[36902]: audit 2022-04-22T18:08:39.618322+0000 mgr.y (mgr.34107) 1504 : audit [DBG] from='client.36687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:41 smithi121 bash[36902]: cluster 2022-04-22T18:08:40.086598+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2022-04-22T18:08:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:41 smithi121 bash[36902]: audit 2022-04-22T18:08:40.570167+0000 mon.a (mon.0) 3156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:41 smithi121 bash[36902]: audit 2022-04-22T18:08:40.994796+0000 mon.a (mon.0) 3157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:41 smithi121 bash[36902]: cluster 2022-04-22T18:08:41.049753+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2022-04-22T18:08:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[45495]: audit 2022-04-22T18:08:39.618322+0000 mgr.y (mgr.34107) 1504 : audit [DBG] from='client.36687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[45495]: cluster 2022-04-22T18:08:40.086598+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[45495]: audit 2022-04-22T18:08:40.570167+0000 mon.a (mon.0) 3156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[45495]: audit 2022-04-22T18:08:40.994796+0000 mon.a (mon.0) 3157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[45495]: cluster 2022-04-22T18:08:41.049753+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[46934]: audit 2022-04-22T18:08:39.618322+0000 mgr.y (mgr.34107) 1504 : audit [DBG] from='client.36687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[46934]: cluster 2022-04-22T18:08:40.086598+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e1113: 8 total, 8 up, 8 in 2022-04-22T18:08:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[46934]: audit 2022-04-22T18:08:40.570167+0000 mon.a (mon.0) 3156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[46934]: audit 2022-04-22T18:08:40.994796+0000 mon.a (mon.0) 3157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[46934]: cluster 2022-04-22T18:08:41.049753+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e1114: 8 total, 8 up, 8 in 2022-04-22T18:08:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:41.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:42.086 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:08:42.087 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:08:42.087 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2-rename 2022-04-22T18:08:42.087 INFO:tasks.workunit.client.0.smithi115.stdout:found snap2-rename with size 4194304 2022-04-22T18:08:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:42 smithi121 bash[36902]: cluster 2022-04-22T18:08:40.956325+0000 mgr.y (mgr.34107) 1505 : cluster [DBG] pgmap v2113: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.5 KiB/s wr, 16 op/s 2022-04-22T18:08:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:42 smithi121 bash[36902]: cluster 2022-04-22T18:08:42.064387+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2022-04-22T18:08:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:42 smithi115 bash[46934]: cluster 2022-04-22T18:08:40.956325+0000 mgr.y (mgr.34107) 1505 : cluster [DBG] pgmap v2113: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.5 KiB/s wr, 16 op/s 2022-04-22T18:08:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:42 smithi115 bash[46934]: cluster 2022-04-22T18:08:42.064387+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2022-04-22T18:08:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:42 smithi115 bash[45495]: cluster 2022-04-22T18:08:40.956325+0000 mgr.y (mgr.34107) 1505 : cluster [DBG] pgmap v2113: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.5 KiB/s wr, 16 op/s 2022-04-22T18:08:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:42 smithi115 bash[45495]: cluster 2022-04-22T18:08:42.064387+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e1115: 8 total, 8 up, 8 in 2022-04-22T18:08:43.089 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:08:43.089 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:08:43.091 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3989 ms) 2022-04-22T18:08:43.091 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2022-04-22T18:08:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:43 smithi121 bash[36902]: audit 2022-04-22T18:08:42.960085+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:43 smithi121 bash[36902]: cluster 2022-04-22T18:08:43.065203+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2022-04-22T18:08:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:43 smithi115 bash[45495]: audit 2022-04-22T18:08:42.960085+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:43 smithi115 bash[45495]: cluster 2022-04-22T18:08:43.065203+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2022-04-22T18:08:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:43 smithi115 bash[46934]: audit 2022-04-22T18:08:42.960085+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:43 smithi115 bash[46934]: cluster 2022-04-22T18:08:43.065203+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e1116: 8 total, 8 up, 8 in 2022-04-22T18:08:44.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:43.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:44 smithi121 bash[36902]: cluster 2022-04-22T18:08:42.957243+0000 mgr.y (mgr.34107) 1506 : cluster [DBG] pgmap v2116: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 4 op/s 2022-04-22T18:08:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:44 smithi115 bash[45495]: cluster 2022-04-22T18:08:42.957243+0000 mgr.y (mgr.34107) 1506 : cluster [DBG] pgmap v2116: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 4 op/s 2022-04-22T18:08:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:44 smithi115 bash[46934]: cluster 2022-04-22T18:08:42.957243+0000 mgr.y (mgr.34107) 1506 : cluster [DBG] pgmap v2116: 265 pgs: 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 4 op/s 2022-04-22T18:08:45.138 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:45.138 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:45.138 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:45.138 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:45 smithi121 bash[36902]: cluster 2022-04-22T18:08:44.115277+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e1117: 8 total, 8 up, 8 in 2022-04-22T18:08:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:45 smithi121 bash[36902]: audit 2022-04-22T18:08:44.122105+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:45 smithi115 bash[46934]: cluster 2022-04-22T18:08:44.115277+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e1117: 8 total, 8 up, 8 in 2022-04-22T18:08:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:45 smithi115 bash[46934]: audit 2022-04-22T18:08:44.122105+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:45 smithi115 bash[45495]: cluster 2022-04-22T18:08:44.115277+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e1117: 8 total, 8 up, 8 in 2022-04-22T18:08:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:45 smithi115 bash[45495]: audit 2022-04-22T18:08:44.122105+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: cluster 2022-04-22T18:08:44.958387+0000 mgr.y (mgr.34107) 1507 : cluster [DBG] pgmap v2119: 297 pgs: 4 creating+peering, 28 unknown, 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: audit 2022-04-22T18:08:45.124614+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: cluster 2022-04-22T18:08:45.124900+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: audit 2022-04-22T18:08:45.573616+0000 mon.a (mon.0) 3166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: audit 2022-04-22T18:08:45.998343+0000 mon.a (mon.0) 3167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:46 smithi121 bash[36902]: cluster 2022-04-22T18:08:46.048114+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2022-04-22T18:08:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: cluster 2022-04-22T18:08:44.958387+0000 mgr.y (mgr.34107) 1507 : cluster [DBG] pgmap v2119: 297 pgs: 4 creating+peering, 28 unknown, 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: audit 2022-04-22T18:08:45.124614+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: cluster 2022-04-22T18:08:45.124900+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: audit 2022-04-22T18:08:45.573616+0000 mon.a (mon.0) 3166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: audit 2022-04-22T18:08:45.998343+0000 mon.a (mon.0) 3167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[45495]: cluster 2022-04-22T18:08:46.048114+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2022-04-22T18:08:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: cluster 2022-04-22T18:08:44.958387+0000 mgr.y (mgr.34107) 1507 : cluster [DBG] pgmap v2119: 297 pgs: 4 creating+peering, 28 unknown, 265 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2022-04-22T18:08:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: audit 2022-04-22T18:08:45.124614+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.115:0/4038768467' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-6","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: cluster 2022-04-22T18:08:45.124900+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2022-04-22T18:08:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: audit 2022-04-22T18:08:45.573616+0000 mon.a (mon.0) 3166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: audit 2022-04-22T18:08:45.998343+0000 mon.a (mon.0) 3167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:46.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:46 smithi115 bash[46934]: cluster 2022-04-22T18:08:46.048114+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2022-04-22T18:08:47.185 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4093 ms) 2022-04-22T18:08:47.185 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2022-04-22T18:08:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:48 smithi121 bash[36902]: cluster 2022-04-22T18:08:46.959631+0000 mgr.y (mgr.34107) 1508 : cluster [DBG] pgmap v2122: 297 pgs: 7 creating+peering, 16 unknown, 274 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2022-04-22T18:08:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:48 smithi121 bash[36902]: cluster 2022-04-22T18:08:47.052117+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2022-04-22T18:08:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:48 smithi121 bash[36902]: audit 2022-04-22T18:08:47.963498+0000 mon.a (mon.0) 3170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[45495]: cluster 2022-04-22T18:08:46.959631+0000 mgr.y (mgr.34107) 1508 : cluster [DBG] pgmap v2122: 297 pgs: 7 creating+peering, 16 unknown, 274 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2022-04-22T18:08:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[45495]: cluster 2022-04-22T18:08:47.052117+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2022-04-22T18:08:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[45495]: audit 2022-04-22T18:08:47.963498+0000 mon.a (mon.0) 3170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[46934]: cluster 2022-04-22T18:08:46.959631+0000 mgr.y (mgr.34107) 1508 : cluster [DBG] pgmap v2122: 297 pgs: 7 creating+peering, 16 unknown, 274 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2022-04-22T18:08:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[46934]: cluster 2022-04-22T18:08:47.052117+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2022-04-22T18:08:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:48 smithi115 bash[46934]: audit 2022-04-22T18:08:47.963498+0000 mon.a (mon.0) 3170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:49.183 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:49.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:49 smithi121 bash[36902]: cluster 2022-04-22T18:08:48.136809+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2022-04-22T18:08:49.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:49 smithi121 bash[36902]: audit 2022-04-22T18:08:48.141281+0000 mon.c (mon.1) 88 : audit [INF] from='client.? 172.21.15.115:0/2185023867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:49.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:49 smithi121 bash[36902]: audit 2022-04-22T18:08:48.144148+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[45495]: cluster 2022-04-22T18:08:48.136809+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2022-04-22T18:08:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[45495]: audit 2022-04-22T18:08:48.141281+0000 mon.c (mon.1) 88 : audit [INF] from='client.? 172.21.15.115:0/2185023867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[45495]: audit 2022-04-22T18:08:48.144148+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:49.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[46934]: cluster 2022-04-22T18:08:48.136809+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2022-04-22T18:08:49.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[46934]: audit 2022-04-22T18:08:48.141281+0000 mon.c (mon.1) 88 : audit [INF] from='client.? 172.21.15.115:0/2185023867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:49.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:49 smithi115 bash[46934]: audit 2022-04-22T18:08:48.144148+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: cluster 2022-04-22T18:08:48.960480+0000 mgr.y (mgr.34107) 1509 : cluster [DBG] pgmap v2125: 329 pgs: 7 creating+peering, 36 unknown, 286 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.143445+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: cluster 2022-04-22T18:08:49.143637+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.178339+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.229801+0000 mon.a (mon.0) 3176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.232649+0000 mon.a (mon.0) 3177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.234289+0000 mon.a (mon.0) 3178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:50 smithi121 bash[36902]: audit 2022-04-22T18:08:49.242811+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: cluster 2022-04-22T18:08:48.960480+0000 mgr.y (mgr.34107) 1509 : cluster [DBG] pgmap v2125: 329 pgs: 7 creating+peering, 36 unknown, 286 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.143445+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: cluster 2022-04-22T18:08:49.143637+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.178339+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.229801+0000 mon.a (mon.0) 3176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.232649+0000 mon.a (mon.0) 3177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.234289+0000 mon.a (mon.0) 3178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[45495]: audit 2022-04-22T18:08:49.242811+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: cluster 2022-04-22T18:08:48.960480+0000 mgr.y (mgr.34107) 1509 : cluster [DBG] pgmap v2125: 329 pgs: 7 creating+peering, 36 unknown, 286 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.143445+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-7","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: cluster 2022-04-22T18:08:49.143637+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.178339+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.229801+0000 mon.a (mon.0) 3176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.232649+0000 mon.a (mon.0) 3177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.234289+0000 mon.a (mon.0) 3178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:08:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:50 smithi115 bash[46934]: audit 2022-04-22T18:08:49.242811+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:51.094 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:08:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:51 smithi121 bash[36902]: cluster 2022-04-22T18:08:50.153336+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2022-04-22T18:08:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:51 smithi121 bash[36902]: audit 2022-04-22T18:08:50.576941+0000 mon.a (mon.0) 3181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:51 smithi121 bash[36902]: audit 2022-04-22T18:08:51.002172+0000 mon.a (mon.0) 3182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:51 smithi121 bash[36902]: cluster 2022-04-22T18:08:51.051888+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2022-04-22T18:08:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:51 smithi121 bash[36902]: audit 2022-04-22T18:08:51.095185+0000 mon.a (mon.0) 3184 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]: dispatch 2022-04-22T18:08:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[45495]: cluster 2022-04-22T18:08:50.153336+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[45495]: audit 2022-04-22T18:08:50.576941+0000 mon.a (mon.0) 3181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[45495]: audit 2022-04-22T18:08:51.002172+0000 mon.a (mon.0) 3182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[45495]: cluster 2022-04-22T18:08:51.051888+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[45495]: audit 2022-04-22T18:08:51.095185+0000 mon.a (mon.0) 3184 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]: dispatch 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[46934]: cluster 2022-04-22T18:08:50.153336+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[46934]: audit 2022-04-22T18:08:50.576941+0000 mon.a (mon.0) 3181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[46934]: audit 2022-04-22T18:08:51.002172+0000 mon.a (mon.0) 3182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[46934]: cluster 2022-04-22T18:08:51.051888+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2022-04-22T18:08:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[46934]: audit 2022-04-22T18:08:51.095185+0000 mon.a (mon.0) 3184 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]: dispatch 2022-04-22T18:08:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:51.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:52 smithi121 bash[36902]: cluster 2022-04-22T18:08:50.961882+0000 mgr.y (mgr.34107) 1510 : cluster [DBG] pgmap v2128: 329 pgs: 3 creating+peering, 17 unknown, 309 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:08:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:52 smithi121 bash[36902]: audit 2022-04-22T18:08:52.068049+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]': finished 2022-04-22T18:08:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:52 smithi121 bash[36902]: cluster 2022-04-22T18:08:52.068119+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2022-04-22T18:08:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[45495]: cluster 2022-04-22T18:08:50.961882+0000 mgr.y (mgr.34107) 1510 : cluster [DBG] pgmap v2128: 329 pgs: 3 creating+peering, 17 unknown, 309 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:08:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[45495]: audit 2022-04-22T18:08:52.068049+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]': finished 2022-04-22T18:08:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[45495]: cluster 2022-04-22T18:08:52.068119+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2022-04-22T18:08:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[46934]: cluster 2022-04-22T18:08:50.961882+0000 mgr.y (mgr.34107) 1510 : cluster [DBG] pgmap v2128: 329 pgs: 3 creating+peering, 17 unknown, 309 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:08:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[46934]: audit 2022-04-22T18:08:52.068049+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.115:0/2762600643' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi115-68765-7", "field": "max_bytes", "val": "10485760"}]': finished 2022-04-22T18:08:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:52 smithi115 bash[46934]: cluster 2022-04-22T18:08:52.068119+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2022-04-22T18:08:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:53 smithi121 bash[36902]: audit 2022-04-22T18:08:52.966573+0000 mon.a (mon.0) 3187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:53.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:53 smithi115 bash[45495]: audit 2022-04-22T18:08:52.966573+0000 mon.a (mon.0) 3187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:53 smithi115 bash[46934]: audit 2022-04-22T18:08:52.966573+0000 mon.a (mon.0) 3187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:08:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:08:53.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:08:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:54 smithi121 bash[36902]: cluster 2022-04-22T18:08:52.962885+0000 mgr.y (mgr.34107) 1511 : cluster [DBG] pgmap v2131: 329 pgs: 3 creating+peering, 10 unknown, 316 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2022-04-22T18:08:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:54 smithi115 bash[45495]: cluster 2022-04-22T18:08:52.962885+0000 mgr.y (mgr.34107) 1511 : cluster [DBG] pgmap v2131: 329 pgs: 3 creating+peering, 10 unknown, 316 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2022-04-22T18:08:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:54 smithi115 bash[46934]: cluster 2022-04-22T18:08:52.962885+0000 mgr.y (mgr.34107) 1511 : cluster [DBG] pgmap v2131: 329 pgs: 3 creating+peering, 10 unknown, 316 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2022-04-22T18:08:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:55 smithi121 bash[36902]: audit 2022-04-22T18:08:54.331692+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:55 smithi115 bash[45495]: audit 2022-04-22T18:08:54.331692+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:55 smithi115 bash[46934]: audit 2022-04-22T18:08:54.331692+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:08:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:56 smithi121 bash[36902]: cluster 2022-04-22T18:08:54.964951+0000 mgr.y (mgr.34107) 1512 : cluster [DBG] pgmap v2132: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 531 KiB/s wr, 11 op/s 2022-04-22T18:08:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:56 smithi121 bash[36902]: audit 2022-04-22T18:08:55.579940+0000 mon.a (mon.0) 3189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:56 smithi121 bash[36902]: audit 2022-04-22T18:08:56.005548+0000 mon.a (mon.0) 3190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:56 smithi121 bash[36902]: cluster 2022-04-22T18:08:56.053161+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2022-04-22T18:08:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[45495]: cluster 2022-04-22T18:08:54.964951+0000 mgr.y (mgr.34107) 1512 : cluster [DBG] pgmap v2132: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 531 KiB/s wr, 11 op/s 2022-04-22T18:08:56.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[45495]: audit 2022-04-22T18:08:55.579940+0000 mon.a (mon.0) 3189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[45495]: audit 2022-04-22T18:08:56.005548+0000 mon.a (mon.0) 3190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[45495]: cluster 2022-04-22T18:08:56.053161+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[46934]: cluster 2022-04-22T18:08:54.964951+0000 mgr.y (mgr.34107) 1512 : cluster [DBG] pgmap v2132: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 531 KiB/s wr, 11 op/s 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[46934]: audit 2022-04-22T18:08:55.579940+0000 mon.a (mon.0) 3189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[46934]: audit 2022-04-22T18:08:56.005548+0000 mon.a (mon.0) 3190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:08:56.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:56 smithi115 bash[46934]: cluster 2022-04-22T18:08:56.053161+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2022-04-22T18:08:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:58 smithi121 bash[36902]: cluster 2022-04-22T18:08:56.966146+0000 mgr.y (mgr.34107) 1513 : cluster [DBG] pgmap v2134: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 515 KiB/s wr, 10 op/s 2022-04-22T18:08:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:08:58 smithi121 bash[36902]: audit 2022-04-22T18:08:57.971459+0000 mon.a (mon.0) 3192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:58 smithi115 bash[45495]: cluster 2022-04-22T18:08:56.966146+0000 mgr.y (mgr.34107) 1513 : cluster [DBG] pgmap v2134: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 515 KiB/s wr, 10 op/s 2022-04-22T18:08:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:08:58 smithi115 bash[45495]: audit 2022-04-22T18:08:57.971459+0000 mon.a (mon.0) 3192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:08:58.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:58 smithi115 bash[46934]: cluster 2022-04-22T18:08:56.966146+0000 mgr.y (mgr.34107) 1513 : cluster [DBG] pgmap v2134: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 515 KiB/s wr, 10 op/s 2022-04-22T18:08:58.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:08:58 smithi115 bash[46934]: audit 2022-04-22T18:08:57.971459+0000 mon.a (mon.0) 3192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:00 smithi121 bash[36902]: cluster 2022-04-22T18:08:58.967994+0000 mgr.y (mgr.34107) 1514 : cluster [DBG] pgmap v2135: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 390 KiB/s wr, 7 op/s 2022-04-22T18:09:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:00 smithi115 bash[45495]: cluster 2022-04-22T18:08:58.967994+0000 mgr.y (mgr.34107) 1514 : cluster [DBG] pgmap v2135: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 390 KiB/s wr, 7 op/s 2022-04-22T18:09:00.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:00 smithi115 bash[46934]: cluster 2022-04-22T18:08:58.967994+0000 mgr.y (mgr.34107) 1514 : cluster [DBG] pgmap v2135: 329 pgs: 329 active+clean; 7.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 390 KiB/s wr, 7 op/s 2022-04-22T18:09:01.367 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[45495]: audit 2022-04-22T18:09:00.582913+0000 mon.a (mon.0) 3193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:01.367 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[45495]: audit 2022-04-22T18:09:01.008785+0000 mon.a (mon.0) 3194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:01.367 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[45495]: cluster 2022-04-22T18:09:01.059116+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2022-04-22T18:09:01.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[46934]: audit 2022-04-22T18:09:00.582913+0000 mon.a (mon.0) 3193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:01.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[46934]: audit 2022-04-22T18:09:01.008785+0000 mon.a (mon.0) 3194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:01.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[46934]: cluster 2022-04-22T18:09:01.059116+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2022-04-22T18:09:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:01 smithi121 bash[36902]: audit 2022-04-22T18:09:00.582913+0000 mon.a (mon.0) 3193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:01.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:01 smithi121 bash[36902]: audit 2022-04-22T18:09:01.008785+0000 mon.a (mon.0) 3194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:01.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:01 smithi121 bash[36902]: cluster 2022-04-22T18:09:01.059116+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2022-04-22T18:09:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:01.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:02 smithi121 bash[36902]: cluster 2022-04-22T18:09:00.970666+0000 mgr.y (mgr.34107) 1515 : cluster [DBG] pgmap v2136: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 805 KiB/s wr, 2 op/s 2022-04-22T18:09:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:02 smithi115 bash[45495]: cluster 2022-04-22T18:09:00.970666+0000 mgr.y (mgr.34107) 1515 : cluster [DBG] pgmap v2136: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 805 KiB/s wr, 2 op/s 2022-04-22T18:09:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:02 smithi115 bash[46934]: cluster 2022-04-22T18:09:00.970666+0000 mgr.y (mgr.34107) 1515 : cluster [DBG] pgmap v2136: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 805 KiB/s wr, 2 op/s 2022-04-22T18:09:03.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:03 smithi115 bash[45495]: audit 2022-04-22T18:09:02.974988+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:03.645 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:03 smithi115 bash[46934]: audit 2022-04-22T18:09:02.974988+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:03 smithi121 bash[36902]: audit 2022-04-22T18:09:02.974988+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:03.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:04 smithi121 bash[36902]: cluster 2022-04-22T18:09:02.972247+0000 mgr.y (mgr.34107) 1516 : cluster [DBG] pgmap v2138: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s 2022-04-22T18:09:04.802 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:04 smithi121 bash[36902]: audit 2022-04-22T18:09:04.369498+0000 mon.a (mon.0) 3197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:04 smithi115 bash[45495]: cluster 2022-04-22T18:09:02.972247+0000 mgr.y (mgr.34107) 1516 : cluster [DBG] pgmap v2138: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s 2022-04-22T18:09:04.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:04 smithi115 bash[45495]: audit 2022-04-22T18:09:04.369498+0000 mon.a (mon.0) 3197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:04.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:04 smithi115 bash[46934]: cluster 2022-04-22T18:09:02.972247+0000 mgr.y (mgr.34107) 1516 : cluster [DBG] pgmap v2138: 329 pgs: 329 active+clean; 12 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s 2022-04-22T18:09:04.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:04 smithi115 bash[46934]: audit 2022-04-22T18:09:04.369498+0000 mon.a (mon.0) 3197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:06.247 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RemoveFullTry (19062 ms) 2022-04-22T18:09:06.247 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2022-04-22T18:09:06.247 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:09:06.247 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2022-04-22T18:09:06.248 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestIO 2022-04-22T18:09:06.248 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: cluster 2022-04-22T18:09:04.974410+0000 mgr.y (mgr.34107) 1517 : cluster [DBG] pgmap v2139: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 918 KiB/s wr, 0 op/s 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: audit 2022-04-22T18:09:05.586213+0000 mon.a (mon.0) 3198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: audit 2022-04-22T18:09:06.012553+0000 mon.a (mon.0) 3199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: cluster 2022-04-22T18:09:06.050229+0000 mon.a (mon.0) 3200 : cluster [WRN] pool 'test-librbd-smithi115-68765-7' is full (reached quota's max_bytes: 10 MiB) 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: cluster 2022-04-22T18:09:06.050758+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2022-04-22T18:09:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:06 smithi121 bash[36902]: cluster 2022-04-22T18:09:06.063837+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: cluster 2022-04-22T18:09:04.974410+0000 mgr.y (mgr.34107) 1517 : cluster [DBG] pgmap v2139: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 918 KiB/s wr, 0 op/s 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: audit 2022-04-22T18:09:05.586213+0000 mon.a (mon.0) 3198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: audit 2022-04-22T18:09:06.012553+0000 mon.a (mon.0) 3199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: cluster 2022-04-22T18:09:06.050229+0000 mon.a (mon.0) 3200 : cluster [WRN] pool 'test-librbd-smithi115-68765-7' is full (reached quota's max_bytes: 10 MiB) 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: cluster 2022-04-22T18:09:06.050758+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2022-04-22T18:09:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[46934]: cluster 2022-04-22T18:09:06.063837+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: cluster 2022-04-22T18:09:04.974410+0000 mgr.y (mgr.34107) 1517 : cluster [DBG] pgmap v2139: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 918 KiB/s wr, 0 op/s 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: audit 2022-04-22T18:09:05.586213+0000 mon.a (mon.0) 3198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: audit 2022-04-22T18:09:06.012553+0000 mon.a (mon.0) 3199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: cluster 2022-04-22T18:09:06.050229+0000 mon.a (mon.0) 3200 : cluster [WRN] pool 'test-librbd-smithi115-68765-7' is full (reached quota's max_bytes: 10 MiB) 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: cluster 2022-04-22T18:09:06.050758+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2022-04-22T18:09:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:06 smithi115 bash[45495]: cluster 2022-04-22T18:09:06.063837+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2022-04-22T18:09:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:08 smithi121 bash[36902]: cluster 2022-04-22T18:09:06.975555+0000 mgr.y (mgr.34107) 1518 : cluster [DBG] pgmap v2141: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 1023 KiB/s wr, 1 op/s 2022-04-22T18:09:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:08 smithi121 bash[36902]: audit 2022-04-22T18:09:07.977758+0000 mon.a (mon.0) 3203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:08 smithi115 bash[45495]: cluster 2022-04-22T18:09:06.975555+0000 mgr.y (mgr.34107) 1518 : cluster [DBG] pgmap v2141: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 1023 KiB/s wr, 1 op/s 2022-04-22T18:09:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:08 smithi115 bash[45495]: audit 2022-04-22T18:09:07.977758+0000 mon.a (mon.0) 3203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:08 smithi115 bash[46934]: cluster 2022-04-22T18:09:06.975555+0000 mgr.y (mgr.34107) 1518 : cluster [DBG] pgmap v2141: 329 pgs: 329 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 1023 KiB/s wr, 1 op/s 2022-04-22T18:09:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:08 smithi115 bash[46934]: audit 2022-04-22T18:09:07.977758+0000 mon.a (mon.0) 3203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:10.423 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:09:10.577 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:10.578 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:10.579 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.580 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.581 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:10.582 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.583 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.584 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:10.584 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.584 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:10.584 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.585 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:10.586 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.587 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.588 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.589 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:10.590 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 512~512 = 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 1536~512 = 0 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:10.591 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.592 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:10.593 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.594 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.594 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:10.594 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.594 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.595 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.596 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.597 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:10.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:10.599 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.601 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.602 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.603 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.604 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.604 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.604 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.604 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.604 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.605 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.606 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:10.607 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:10.608 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:10.608 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:10.608 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:10.608 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:10 smithi121 bash[36902]: cluster 2022-04-22T18:09:08.976964+0000 mgr.y (mgr.34107) 1519 : cluster [DBG] pgmap v2142: 329 pgs: 329 active+clean; 13 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 896 KiB/s wr, 2 op/s 2022-04-22T18:09:10.768 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (32m) 59s ago 46m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (45m) 2m ago 45m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (34m) 59s ago 47m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (33m) 2m ago 47m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (41m) 2m ago 51m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (42m) 59s ago 53m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (41m) 59s ago 53m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:09:10.769 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (40m) 2m ago 51m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (40m) 59s ago 51m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (33m) 59s ago 46m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (33m) 2m ago 46m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (38m) 59s ago 50m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (38m) 59s ago 50m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (37m) 59s ago 49m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (37m) 59s ago 49m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (36m) 2m ago 49m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (36m) 2m ago 48m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (35m) 2m ago 48m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:09:10.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (34m) 2m ago 47m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:09:10.771 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (32m) 2m ago 46m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:09:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:10 smithi115 bash[45495]: cluster 2022-04-22T18:09:08.976964+0000 mgr.y (mgr.34107) 1519 : cluster [DBG] pgmap v2142: 329 pgs: 329 active+clean; 13 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 896 KiB/s wr, 2 op/s 2022-04-22T18:09:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:10 smithi115 bash[46934]: cluster 2022-04-22T18:09:08.976964+0000 mgr.y (mgr.34107) 1519 : cluster [DBG] pgmap v2142: 329 pgs: 329 active+clean; 13 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 896 KiB/s wr, 2 op/s 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:09:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:09:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:09:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:09:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:09:11.194 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:09:11.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: audit 2022-04-22T18:09:10.402627+0000 mgr.y (mgr.34107) 1520 : audit [DBG] from='client.46642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: audit 2022-04-22T18:09:10.589002+0000 mon.a (mon.0) 3204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: audit 2022-04-22T18:09:11.015541+0000 mon.a (mon.0) 3205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: cluster 2022-04-22T18:09:11.052703+0000 mon.a (mon.0) 3206 : cluster [INF] pool 'test-librbd-smithi115-68765-7' no longer out of quota; removing NO_QUOTA flag 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: cluster 2022-04-22T18:09:11.053116+0000 mon.a (mon.0) 3207 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: cluster 2022-04-22T18:09:11.059118+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2022-04-22T18:09:11.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[45495]: audit 2022-04-22T18:09:11.186508+0000 mon.c (mon.1) 89 : audit [DBG] from='client.? 172.21.15.115:0/579571303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: audit 2022-04-22T18:09:10.402627+0000 mgr.y (mgr.34107) 1520 : audit [DBG] from='client.46642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: audit 2022-04-22T18:09:10.589002+0000 mon.a (mon.0) 3204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: audit 2022-04-22T18:09:11.015541+0000 mon.a (mon.0) 3205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: cluster 2022-04-22T18:09:11.052703+0000 mon.a (mon.0) 3206 : cluster [INF] pool 'test-librbd-smithi115-68765-7' no longer out of quota; removing NO_QUOTA flag 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: cluster 2022-04-22T18:09:11.053116+0000 mon.a (mon.0) 3207 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2022-04-22T18:09:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: cluster 2022-04-22T18:09:11.059118+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2022-04-22T18:09:11.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[46934]: audit 2022-04-22T18:09:11.186508+0000 mon.c (mon.1) 89 : audit [DBG] from='client.? 172.21.15.115:0/579571303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: audit 2022-04-22T18:09:10.402627+0000 mgr.y (mgr.34107) 1520 : audit [DBG] from='client.46642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: audit 2022-04-22T18:09:10.589002+0000 mon.a (mon.0) 3204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: audit 2022-04-22T18:09:11.015541+0000 mon.a (mon.0) 3205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:11.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: cluster 2022-04-22T18:09:11.052703+0000 mon.a (mon.0) 3206 : cluster [INF] pool 'test-librbd-smithi115-68765-7' no longer out of quota; removing NO_QUOTA flag 2022-04-22T18:09:11.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: cluster 2022-04-22T18:09:11.053116+0000 mon.a (mon.0) 3207 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2022-04-22T18:09:11.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: cluster 2022-04-22T18:09:11.059118+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2022-04-22T18:09:11.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:11 smithi121 bash[36902]: audit 2022-04-22T18:09:11.186508+0000 mon.c (mon.1) 89 : audit [DBG] from='client.? 172.21.15.115:0/579571303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:11.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:12 smithi121 bash[36902]: audit 2022-04-22T18:09:10.749304+0000 mgr.y (mgr.34107) 1521 : audit [DBG] from='client.36717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:12 smithi121 bash[36902]: cluster 2022-04-22T18:09:10.978507+0000 mgr.y (mgr.34107) 1522 : cluster [DBG] pgmap v2143: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 809 KiB/s wr, 289 op/s 2022-04-22T18:09:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:12 smithi115 bash[45495]: audit 2022-04-22T18:09:10.749304+0000 mgr.y (mgr.34107) 1521 : audit [DBG] from='client.36717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:12 smithi115 bash[45495]: cluster 2022-04-22T18:09:10.978507+0000 mgr.y (mgr.34107) 1522 : cluster [DBG] pgmap v2143: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 809 KiB/s wr, 289 op/s 2022-04-22T18:09:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:12 smithi115 bash[46934]: audit 2022-04-22T18:09:10.749304+0000 mgr.y (mgr.34107) 1521 : audit [DBG] from='client.36717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:12 smithi115 bash[46934]: cluster 2022-04-22T18:09:10.978507+0000 mgr.y (mgr.34107) 1522 : cluster [DBG] pgmap v2143: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 809 KiB/s wr, 289 op/s 2022-04-22T18:09:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:13 smithi121 bash[36902]: audit 2022-04-22T18:09:12.981981+0000 mon.a (mon.0) 3209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:13.945 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:13 smithi115 bash[45495]: audit 2022-04-22T18:09:12.981981+0000 mon.a (mon.0) 3209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:13.946 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:13.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:13.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:13 smithi115 bash[46934]: audit 2022-04-22T18:09:12.981981+0000 mon.a (mon.0) 3209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:14.249 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.249 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.250 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.251 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.252 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.253 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.254 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.255 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.256 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.257 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.258 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.259 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.260 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:14.261 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:14.262 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestIO (8003 ms) 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2022-04-22T18:09:14.263 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:14 smithi121 bash[36902]: cluster 2022-04-22T18:09:12.979873+0000 mgr.y (mgr.34107) 1523 : cluster [DBG] pgmap v2145: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 192 KiB/s rd, 491 KiB/s wr, 358 op/s 2022-04-22T18:09:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:14 smithi121 bash[36902]: audit 2022-04-22T18:09:13.819768+0000 mon.a (mon.0) 3210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:09:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:14 smithi121 bash[36902]: audit 2022-04-22T18:09:13.822942+0000 mon.a (mon.0) 3211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:09:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:14 smithi121 bash[36902]: audit 2022-04-22T18:09:14.274283+0000 mon.a (mon.0) 3212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]: dispatch 2022-04-22T18:09:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[46934]: cluster 2022-04-22T18:09:12.979873+0000 mgr.y (mgr.34107) 1523 : cluster [DBG] pgmap v2145: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 192 KiB/s rd, 491 KiB/s wr, 358 op/s 2022-04-22T18:09:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[46934]: audit 2022-04-22T18:09:13.819768+0000 mon.a (mon.0) 3210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[46934]: audit 2022-04-22T18:09:13.822942+0000 mon.a (mon.0) 3211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[46934]: audit 2022-04-22T18:09:14.274283+0000 mon.a (mon.0) 3212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]: dispatch 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[45495]: cluster 2022-04-22T18:09:12.979873+0000 mgr.y (mgr.34107) 1523 : cluster [DBG] pgmap v2145: 329 pgs: 329 active+clean; 4.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 192 KiB/s rd, 491 KiB/s wr, 358 op/s 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[45495]: audit 2022-04-22T18:09:13.819768+0000 mon.a (mon.0) 3210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[45495]: audit 2022-04-22T18:09:13.822942+0000 mon.a (mon.0) 3211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:09:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:14 smithi115 bash[45495]: audit 2022-04-22T18:09:14.274283+0000 mon.a (mon.0) 3212 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]: dispatch 2022-04-22T18:09:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:15 smithi121 bash[36902]: audit 2022-04-22T18:09:14.415936+0000 mon.a (mon.0) 3213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]': finished 2022-04-22T18:09:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:15 smithi121 bash[36902]: cluster 2022-04-22T18:09:14.416092+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2022-04-22T18:09:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:15 smithi115 bash[46934]: audit 2022-04-22T18:09:14.415936+0000 mon.a (mon.0) 3213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]': finished 2022-04-22T18:09:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:15 smithi115 bash[46934]: cluster 2022-04-22T18:09:14.416092+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2022-04-22T18:09:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:15 smithi115 bash[45495]: audit 2022-04-22T18:09:14.415936+0000 mon.a (mon.0) 3213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "69.1", "id": [1, 4]}]': finished 2022-04-22T18:09:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:15 smithi115 bash[45495]: cluster 2022-04-22T18:09:14.416092+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2022-04-22T18:09:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:16 smithi121 bash[36902]: cluster 2022-04-22T18:09:14.981197+0000 mgr.y (mgr.34107) 1524 : cluster [DBG] pgmap v2147: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 523 KiB/s rd, 647 KiB/s wr, 1.02k op/s 2022-04-22T18:09:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:16 smithi121 bash[36902]: cluster 2022-04-22T18:09:15.440668+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2022-04-22T18:09:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:16 smithi121 bash[36902]: audit 2022-04-22T18:09:15.591683+0000 mon.a (mon.0) 3216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:16 smithi121 bash[36902]: audit 2022-04-22T18:09:16.018633+0000 mon.a (mon.0) 3217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[46934]: cluster 2022-04-22T18:09:14.981197+0000 mgr.y (mgr.34107) 1524 : cluster [DBG] pgmap v2147: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 523 KiB/s rd, 647 KiB/s wr, 1.02k op/s 2022-04-22T18:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[46934]: cluster 2022-04-22T18:09:15.440668+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2022-04-22T18:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[46934]: audit 2022-04-22T18:09:15.591683+0000 mon.a (mon.0) 3216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[46934]: audit 2022-04-22T18:09:16.018633+0000 mon.a (mon.0) 3217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[45495]: cluster 2022-04-22T18:09:14.981197+0000 mgr.y (mgr.34107) 1524 : cluster [DBG] pgmap v2147: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 523 KiB/s rd, 647 KiB/s wr, 1.02k op/s 2022-04-22T18:09:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[45495]: cluster 2022-04-22T18:09:15.440668+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2022-04-22T18:09:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[45495]: audit 2022-04-22T18:09:15.591683+0000 mon.a (mon.0) 3216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:16 smithi115 bash[45495]: audit 2022-04-22T18:09:16.018633+0000 mon.a (mon.0) 3217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:18 smithi121 bash[36902]: cluster 2022-04-22T18:09:16.982258+0000 mgr.y (mgr.34107) 1525 : cluster [DBG] pgmap v2149: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 443 KiB/s rd, 208 KiB/s wr, 884 op/s 2022-04-22T18:09:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:18 smithi121 bash[36902]: audit 2022-04-22T18:09:17.983343+0000 mon.a (mon.0) 3218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:18 smithi115 bash[45495]: cluster 2022-04-22T18:09:16.982258+0000 mgr.y (mgr.34107) 1525 : cluster [DBG] pgmap v2149: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 443 KiB/s rd, 208 KiB/s wr, 884 op/s 2022-04-22T18:09:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:18 smithi115 bash[45495]: audit 2022-04-22T18:09:17.983343+0000 mon.a (mon.0) 3218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:18 smithi115 bash[46934]: cluster 2022-04-22T18:09:16.982258+0000 mgr.y (mgr.34107) 1525 : cluster [DBG] pgmap v2149: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 443 KiB/s rd, 208 KiB/s wr, 884 op/s 2022-04-22T18:09:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:18 smithi115 bash[46934]: audit 2022-04-22T18:09:17.983343+0000 mon.a (mon.0) 3218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:20.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:20 smithi121 bash[36902]: cluster 2022-04-22T18:09:18.983443+0000 mgr.y (mgr.34107) 1526 : cluster [DBG] pgmap v2150: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 158 KiB/s wr, 670 op/s 2022-04-22T18:09:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:20 smithi115 bash[45495]: cluster 2022-04-22T18:09:18.983443+0000 mgr.y (mgr.34107) 1526 : cluster [DBG] pgmap v2150: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 158 KiB/s wr, 670 op/s 2022-04-22T18:09:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:20 smithi115 bash[46934]: cluster 2022-04-22T18:09:18.983443+0000 mgr.y (mgr.34107) 1526 : cluster [DBG] pgmap v2150: 329 pgs: 329 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 158 KiB/s wr, 670 op/s 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.610 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:21.611 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.611 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.611 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:21.611 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.611 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.612 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:21.613 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.614 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.615 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:21.616 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.617 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:21.618 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:21.619 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 512~512 = 512 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 1536~512 = 0 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.620 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:21.621 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.622 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:21.623 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.624 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.624 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:21.624 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.624 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:21.625 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.626 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.626 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:21.627 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.627 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.627 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:21.627 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.627 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.632 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.634 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.634 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:21.634 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.637 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.637 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.638 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:21.641 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.641 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.641 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.641 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.644 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.646 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.647 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.648 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.649 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.650 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:21.651 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:21.652 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:21.652 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:21.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:21 smithi115 bash[45495]: audit 2022-04-22T18:09:20.593937+0000 mon.a (mon.0) 3219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:21.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:21 smithi115 bash[45495]: audit 2022-04-22T18:09:21.021800+0000 mon.a (mon.0) 3220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:21.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:21 smithi115 bash[46934]: audit 2022-04-22T18:09:20.593937+0000 mon.a (mon.0) 3219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:21.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:21 smithi115 bash[46934]: audit 2022-04-22T18:09:21.021800+0000 mon.a (mon.0) 3220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:21 smithi121 bash[36902]: audit 2022-04-22T18:09:20.593937+0000 mon.a (mon.0) 3219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:21 smithi121 bash[36902]: audit 2022-04-22T18:09:21.021800+0000 mon.a (mon.0) 3220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:21.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:22 smithi121 bash[36902]: cluster 2022-04-22T18:09:20.985239+0000 mgr.y (mgr.34107) 1527 : cluster [DBG] pgmap v2151: 329 pgs: 329 active+clean; 6.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 332 KiB/s rd, 157 KiB/s wr, 695 op/s 2022-04-22T18:09:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:22 smithi115 bash[45495]: cluster 2022-04-22T18:09:20.985239+0000 mgr.y (mgr.34107) 1527 : cluster [DBG] pgmap v2151: 329 pgs: 329 active+clean; 6.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 332 KiB/s rd, 157 KiB/s wr, 695 op/s 2022-04-22T18:09:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:22 smithi115 bash[46934]: cluster 2022-04-22T18:09:20.985239+0000 mgr.y (mgr.34107) 1527 : cluster [DBG] pgmap v2151: 329 pgs: 329 active+clean; 6.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 332 KiB/s rd, 157 KiB/s wr, 695 op/s 2022-04-22T18:09:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:23 smithi121 bash[36902]: audit 2022-04-22T18:09:22.985751+0000 mon.a (mon.0) 3221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:23.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:23.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:23 smithi115 bash[45495]: audit 2022-04-22T18:09:22.985751+0000 mon.a (mon.0) 3221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:23 smithi115 bash[46934]: audit 2022-04-22T18:09:22.985751+0000 mon.a (mon.0) 3221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:24 smithi121 bash[36902]: cluster 2022-04-22T18:09:22.986580+0000 mgr.y (mgr.34107) 1528 : cluster [DBG] pgmap v2152: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 766 KiB/s wr, 590 op/s 2022-04-22T18:09:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:24 smithi115 bash[45495]: cluster 2022-04-22T18:09:22.986580+0000 mgr.y (mgr.34107) 1528 : cluster [DBG] pgmap v2152: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 766 KiB/s wr, 590 op/s 2022-04-22T18:09:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:24 smithi115 bash[46934]: cluster 2022-04-22T18:09:22.986580+0000 mgr.y (mgr.34107) 1528 : cluster [DBG] pgmap v2152: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 766 KiB/s wr, 590 op/s 2022-04-22T18:09:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:26 smithi121 bash[36902]: cluster 2022-04-22T18:09:24.988094+0000 mgr.y (mgr.34107) 1529 : cluster [DBG] pgmap v2153: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 475 KiB/s rd, 656 KiB/s wr, 506 op/s 2022-04-22T18:09:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:26 smithi121 bash[36902]: audit 2022-04-22T18:09:25.596706+0000 mon.a (mon.0) 3222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:26 smithi121 bash[36902]: audit 2022-04-22T18:09:26.025197+0000 mon.a (mon.0) 3223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[45495]: cluster 2022-04-22T18:09:24.988094+0000 mgr.y (mgr.34107) 1529 : cluster [DBG] pgmap v2153: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 475 KiB/s rd, 656 KiB/s wr, 506 op/s 2022-04-22T18:09:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[45495]: audit 2022-04-22T18:09:25.596706+0000 mon.a (mon.0) 3222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[45495]: audit 2022-04-22T18:09:26.025197+0000 mon.a (mon.0) 3223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[46934]: cluster 2022-04-22T18:09:24.988094+0000 mgr.y (mgr.34107) 1529 : cluster [DBG] pgmap v2153: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 475 KiB/s rd, 656 KiB/s wr, 506 op/s 2022-04-22T18:09:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[46934]: audit 2022-04-22T18:09:25.596706+0000 mon.a (mon.0) 3222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:26 smithi115 bash[46934]: audit 2022-04-22T18:09:26.025197+0000 mon.a (mon.0) 3223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:27.196 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.196 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.197 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.198 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.199 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.200 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.201 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.202 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.215 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.216 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.217 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.218 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.219 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:27.220 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:27.221 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:27.222 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 4 2022-04-22T18:09:27.222 INFO:tasks.workunit.client.0.smithi115.stdout:read: 4 2022-04-22T18:09:27.222 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (12946 ms) 2022-04-22T18:09:27.222 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2022-04-22T18:09:27.222 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:28 smithi121 bash[36902]: cluster 2022-04-22T18:09:26.990122+0000 mgr.y (mgr.34107) 1530 : cluster [DBG] pgmap v2154: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 751 KiB/s rd, 719 KiB/s wr, 754 op/s 2022-04-22T18:09:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:28 smithi121 bash[36902]: audit 2022-04-22T18:09:27.989490+0000 mon.a (mon.0) 3224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:28 smithi115 bash[45495]: cluster 2022-04-22T18:09:26.990122+0000 mgr.y (mgr.34107) 1530 : cluster [DBG] pgmap v2154: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 751 KiB/s rd, 719 KiB/s wr, 754 op/s 2022-04-22T18:09:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:28 smithi115 bash[45495]: audit 2022-04-22T18:09:27.989490+0000 mon.a (mon.0) 3224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:28 smithi115 bash[46934]: cluster 2022-04-22T18:09:26.990122+0000 mgr.y (mgr.34107) 1530 : cluster [DBG] pgmap v2154: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 751 KiB/s rd, 719 KiB/s wr, 754 op/s 2022-04-22T18:09:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:28 smithi115 bash[46934]: audit 2022-04-22T18:09:27.989490+0000 mon.a (mon.0) 3224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:30 smithi121 bash[36902]: cluster 2022-04-22T18:09:28.991142+0000 mgr.y (mgr.34107) 1531 : cluster [DBG] pgmap v2155: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 691 KiB/s wr, 725 op/s 2022-04-22T18:09:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:30 smithi115 bash[45495]: cluster 2022-04-22T18:09:28.991142+0000 mgr.y (mgr.34107) 1531 : cluster [DBG] pgmap v2155: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 691 KiB/s wr, 725 op/s 2022-04-22T18:09:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:30 smithi115 bash[46934]: cluster 2022-04-22T18:09:28.991142+0000 mgr.y (mgr.34107) 1531 : cluster [DBG] pgmap v2155: 329 pgs: 329 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 691 KiB/s wr, 725 op/s 2022-04-22T18:09:31.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:31 smithi115 bash[45495]: audit 2022-04-22T18:09:30.599742+0000 mon.a (mon.0) 3225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:31.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:31 smithi115 bash[45495]: audit 2022-04-22T18:09:31.028187+0000 mon.a (mon.0) 3226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:31.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:31 smithi115 bash[46934]: audit 2022-04-22T18:09:30.599742+0000 mon.a (mon.0) 3225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:31.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:31 smithi115 bash[46934]: audit 2022-04-22T18:09:31.028187+0000 mon.a (mon.0) 3226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:31 smithi121 bash[36902]: audit 2022-04-22T18:09:30.599742+0000 mon.a (mon.0) 3225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:31 smithi121 bash[36902]: audit 2022-04-22T18:09:31.028187+0000 mon.a (mon.0) 3226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:31.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:32 smithi121 bash[36902]: cluster 2022-04-22T18:09:30.992549+0000 mgr.y (mgr.34107) 1532 : cluster [DBG] pgmap v2156: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 692 KiB/s wr, 725 op/s 2022-04-22T18:09:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:32 smithi115 bash[46934]: cluster 2022-04-22T18:09:30.992549+0000 mgr.y (mgr.34107) 1532 : cluster [DBG] pgmap v2156: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 692 KiB/s wr, 725 op/s 2022-04-22T18:09:32.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:32 smithi115 bash[45495]: cluster 2022-04-22T18:09:30.992549+0000 mgr.y (mgr.34107) 1532 : cluster [DBG] pgmap v2156: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 KiB/s rd, 692 KiB/s wr, 725 op/s 2022-04-22T18:09:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:33 smithi121 bash[36902]: audit 2022-04-22T18:09:32.991521+0000 mon.a (mon.0) 3227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:33 smithi115 bash[45495]: audit 2022-04-22T18:09:32.991521+0000 mon.a (mon.0) 3227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:33 smithi115 bash[46934]: audit 2022-04-22T18:09:32.991521+0000 mon.a (mon.0) 3227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:33.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:33.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:34 smithi121 bash[36902]: cluster 2022-04-22T18:09:32.993532+0000 mgr.y (mgr.34107) 1533 : cluster [DBG] pgmap v2157: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 723 KiB/s rd, 691 KiB/s wr, 704 op/s 2022-04-22T18:09:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:34 smithi115 bash[45495]: cluster 2022-04-22T18:09:32.993532+0000 mgr.y (mgr.34107) 1533 : cluster [DBG] pgmap v2157: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 723 KiB/s rd, 691 KiB/s wr, 704 op/s 2022-04-22T18:09:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:34 smithi115 bash[46934]: cluster 2022-04-22T18:09:32.993532+0000 mgr.y (mgr.34107) 1533 : cluster [DBG] pgmap v2157: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 723 KiB/s rd, 691 KiB/s wr, 704 op/s 2022-04-22T18:09:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:36 smithi121 bash[36902]: cluster 2022-04-22T18:09:34.995542+0000 mgr.y (mgr.34107) 1534 : cluster [DBG] pgmap v2158: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:36 smithi121 bash[36902]: audit 2022-04-22T18:09:35.602610+0000 mon.a (mon.0) 3228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:36 smithi121 bash[36902]: audit 2022-04-22T18:09:36.031185+0000 mon.a (mon.0) 3229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[45495]: cluster 2022-04-22T18:09:34.995542+0000 mgr.y (mgr.34107) 1534 : cluster [DBG] pgmap v2158: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[45495]: audit 2022-04-22T18:09:35.602610+0000 mon.a (mon.0) 3228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[45495]: audit 2022-04-22T18:09:36.031185+0000 mon.a (mon.0) 3229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[46934]: cluster 2022-04-22T18:09:34.995542+0000 mgr.y (mgr.34107) 1534 : cluster [DBG] pgmap v2158: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[46934]: audit 2022-04-22T18:09:35.602610+0000 mon.a (mon.0) 3228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:36 smithi115 bash[46934]: audit 2022-04-22T18:09:36.031185+0000 mon.a (mon.0) 3229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:38 smithi121 bash[36902]: cluster 2022-04-22T18:09:36.997236+0000 mgr.y (mgr.34107) 1535 : cluster [DBG] pgmap v2159: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:38 smithi121 bash[36902]: audit 2022-04-22T18:09:37.993530+0000 mon.a (mon.0) 3230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:38 smithi115 bash[45495]: cluster 2022-04-22T18:09:36.997236+0000 mgr.y (mgr.34107) 1535 : cluster [DBG] pgmap v2159: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:38 smithi115 bash[45495]: audit 2022-04-22T18:09:37.993530+0000 mon.a (mon.0) 3230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:38 smithi115 bash[46934]: cluster 2022-04-22T18:09:36.997236+0000 mgr.y (mgr.34107) 1535 : cluster [DBG] pgmap v2159: 329 pgs: 329 active+clean; 15 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 145 KiB/s wr, 305 op/s 2022-04-22T18:09:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:38 smithi115 bash[46934]: audit 2022-04-22T18:09:37.993530+0000 mon.a (mon.0) 3230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:39.336 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:39.336 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:39.336 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:39.337 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:39.339 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:39.340 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.341 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:39.342 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:39.343 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:39.344 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:39.345 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 512~512 = 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 1536~512 = 0 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.346 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.347 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.348 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.348 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:39.348 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.349 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.349 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:39.349 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.349 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.350 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.351 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.351 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.351 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.352 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:39.353 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:39.354 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.356 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.357 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.358 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.359 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.360 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:39.361 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:39.362 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:40 smithi121 bash[36902]: cluster 2022-04-22T18:09:38.998601+0000 mgr.y (mgr.34107) 1536 : cluster [DBG] pgmap v2160: 329 pgs: 329 active+clean; 23 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 833 KiB/s wr, 274 op/s 2022-04-22T18:09:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:40 smithi115 bash[45495]: cluster 2022-04-22T18:09:38.998601+0000 mgr.y (mgr.34107) 1536 : cluster [DBG] pgmap v2160: 329 pgs: 329 active+clean; 23 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 833 KiB/s wr, 274 op/s 2022-04-22T18:09:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:40 smithi115 bash[46934]: cluster 2022-04-22T18:09:38.998601+0000 mgr.y (mgr.34107) 1536 : cluster [DBG] pgmap v2160: 329 pgs: 329 active+clean; 23 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 833 KiB/s wr, 274 op/s 2022-04-22T18:09:41.577 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:09:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:41 smithi115 bash[45495]: audit 2022-04-22T18:09:40.605188+0000 mon.a (mon.0) 3231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:41 smithi115 bash[45495]: audit 2022-04-22T18:09:41.033957+0000 mon.a (mon.0) 3232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:41.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:41.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:41 smithi115 bash[46934]: audit 2022-04-22T18:09:40.605188+0000 mon.a (mon.0) 3231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:41 smithi115 bash[46934]: audit 2022-04-22T18:09:41.033957+0000 mon.a (mon.0) 3232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:41.933 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:09:41.934 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (32m) 90s ago 46m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:09:41.934 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (46m) 2m ago 46m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:09:41.934 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (34m) 90s ago 47m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:09:41.934 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (34m) 2m ago 47m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (42m) 2m ago 51m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (42m) 90s ago 53m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (41m) 90s ago 54m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (40m) 2m ago 52m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (41m) 90s ago 52m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:09:41.935 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (33m) 90s ago 46m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (33m) 2m ago 46m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (39m) 90s ago 51m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (38m) 90s ago 50m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (38m) 90s ago 50m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (37m) 90s ago 49m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:09:41.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (37m) 2m ago 49m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:09:41.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (36m) 2m ago 49m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:09:41.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (35m) 2m ago 48m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:09:41.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (35m) 2m ago 48m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:09:41.937 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (33m) 2m ago 46m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:09:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:41 smithi121 bash[36902]: audit 2022-04-22T18:09:40.605188+0000 mon.a (mon.0) 3231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:41 smithi121 bash[36902]: audit 2022-04-22T18:09:41.033957+0000 mon.a (mon.0) 3232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:42.345 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:09:42.346 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:09:42.346 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:09:42.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:42.346 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:09:42.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:09:42.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:42.347 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:09:42.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:09:42.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:09:42.348 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:09:42.349 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:09:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:42 smithi115 bash[45495]: cluster 2022-04-22T18:09:41.000334+0000 mgr.y (mgr.34107) 1537 : cluster [DBG] pgmap v2161: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:42 smithi115 bash[45495]: audit 2022-04-22T18:09:42.340739+0000 mon.a (mon.0) 3233 : audit [DBG] from='client.? 172.21.15.115:0/759018213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:42 smithi115 bash[46934]: cluster 2022-04-22T18:09:41.000334+0000 mgr.y (mgr.34107) 1537 : cluster [DBG] pgmap v2161: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:42 smithi115 bash[46934]: audit 2022-04-22T18:09:42.340739+0000 mon.a (mon.0) 3233 : audit [DBG] from='client.? 172.21.15.115:0/759018213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:43.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:42 smithi121 bash[36902]: cluster 2022-04-22T18:09:41.000334+0000 mgr.y (mgr.34107) 1537 : cluster [DBG] pgmap v2161: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:43.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:42 smithi121 bash[36902]: audit 2022-04-22T18:09:42.340739+0000 mon.a (mon.0) 3233 : audit [DBG] from='client.? 172.21.15.115:0/759018213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:09:43.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[45495]: audit 2022-04-22T18:09:41.552438+0000 mgr.y (mgr.34107) 1538 : audit [DBG] from='client.36729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:43.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[45495]: audit 2022-04-22T18:09:41.908595+0000 mgr.y (mgr.34107) 1539 : audit [DBG] from='client.46666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:43.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[45495]: audit 2022-04-22T18:09:42.996262+0000 mon.a (mon.0) 3234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:43.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:43.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[46934]: audit 2022-04-22T18:09:41.552438+0000 mgr.y (mgr.34107) 1538 : audit [DBG] from='client.36729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[46934]: audit 2022-04-22T18:09:41.908595+0000 mgr.y (mgr.34107) 1539 : audit [DBG] from='client.46666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:43 smithi115 bash[46934]: audit 2022-04-22T18:09:42.996262+0000 mon.a (mon.0) 3234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:44.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:43 smithi121 bash[36902]: audit 2022-04-22T18:09:41.552438+0000 mgr.y (mgr.34107) 1538 : audit [DBG] from='client.36729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:44.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:43 smithi121 bash[36902]: audit 2022-04-22T18:09:41.908595+0000 mgr.y (mgr.34107) 1539 : audit [DBG] from='client.46666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:09:44.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:43 smithi121 bash[36902]: audit 2022-04-22T18:09:42.996262+0000 mon.a (mon.0) 3234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:44 smithi115 bash[45495]: cluster 2022-04-22T18:09:43.001650+0000 mgr.y (mgr.34107) 1540 : cluster [DBG] pgmap v2162: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:44 smithi115 bash[46934]: cluster 2022-04-22T18:09:43.001650+0000 mgr.y (mgr.34107) 1540 : cluster [DBG] pgmap v2162: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:44 smithi121 bash[36902]: cluster 2022-04-22T18:09:43.001650+0000 mgr.y (mgr.34107) 1540 : cluster [DBG] pgmap v2162: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 274 op/s 2022-04-22T18:09:45.238 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.239 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.240 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.241 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.242 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.243 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.244 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.245 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.246 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.247 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.248 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.249 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:45.250 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:45.251 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 4 2022-04-22T18:09:45.252 INFO:tasks.workunit.client.0.smithi115.stdout:read: 4 2022-04-22T18:09:45.252 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (18043 ms) 2022-04-22T18:09:45.252 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2022-04-22T18:09:45.252 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[45495]: cluster 2022-04-22T18:09:45.003040+0000 mgr.y (mgr.34107) 1541 : cluster [DBG] pgmap v2163: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 699 op/s 2022-04-22T18:09:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[45495]: audit 2022-04-22T18:09:45.608011+0000 mon.a (mon.0) 3235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[45495]: audit 2022-04-22T18:09:46.037034+0000 mon.a (mon.0) 3236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[46934]: cluster 2022-04-22T18:09:45.003040+0000 mgr.y (mgr.34107) 1541 : cluster [DBG] pgmap v2163: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 699 op/s 2022-04-22T18:09:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[46934]: audit 2022-04-22T18:09:45.608011+0000 mon.a (mon.0) 3235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:46 smithi115 bash[46934]: audit 2022-04-22T18:09:46.037034+0000 mon.a (mon.0) 3236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:46 smithi121 bash[36902]: cluster 2022-04-22T18:09:45.003040+0000 mgr.y (mgr.34107) 1541 : cluster [DBG] pgmap v2163: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 699 op/s 2022-04-22T18:09:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:46 smithi121 bash[36902]: audit 2022-04-22T18:09:45.608011+0000 mon.a (mon.0) 3235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:46 smithi121 bash[36902]: audit 2022-04-22T18:09:46.037034+0000 mon.a (mon.0) 3236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:48.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:48 smithi115 bash[45495]: cluster 2022-04-22T18:09:47.004794+0000 mgr.y (mgr.34107) 1542 : cluster [DBG] pgmap v2164: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 701 op/s 2022-04-22T18:09:48.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:48 smithi115 bash[45495]: audit 2022-04-22T18:09:47.998892+0000 mon.a (mon.0) 3237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:48.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:48 smithi115 bash[46934]: cluster 2022-04-22T18:09:47.004794+0000 mgr.y (mgr.34107) 1542 : cluster [DBG] pgmap v2164: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 701 op/s 2022-04-22T18:09:48.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:48 smithi115 bash[46934]: audit 2022-04-22T18:09:47.998892+0000 mon.a (mon.0) 3237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:48 smithi121 bash[36902]: cluster 2022-04-22T18:09:47.004794+0000 mgr.y (mgr.34107) 1542 : cluster [DBG] pgmap v2164: 329 pgs: 329 active+clean; 31 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 701 op/s 2022-04-22T18:09:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:48 smithi121 bash[36902]: audit 2022-04-22T18:09:47.998892+0000 mon.a (mon.0) 3237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:49.785 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.787 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.788 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:49.789 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:49.790 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.791 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.792 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:49.793 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.794 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 512~512 = 512 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.795 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 1536~512 = 0 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:49.797 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.798 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.799 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.800 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.801 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.803 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.804 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.805 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.806 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:49.807 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:49.808 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:49.808 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:49.808 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:49 smithi121 bash[36902]: audit 2022-04-22T18:09:49.251813+0000 mon.a (mon.0) 3238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:09:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:49 smithi121 bash[36902]: audit 2022-04-22T18:09:49.254130+0000 mon.a (mon.0) 3239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:09:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:49 smithi121 bash[36902]: audit 2022-04-22T18:09:49.255351+0000 mon.a (mon.0) 3240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:09:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[45495]: audit 2022-04-22T18:09:49.251813+0000 mon.a (mon.0) 3238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:09:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[45495]: audit 2022-04-22T18:09:49.254130+0000 mon.a (mon.0) 3239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:09:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[45495]: audit 2022-04-22T18:09:49.255351+0000 mon.a (mon.0) 3240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:09:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[46934]: audit 2022-04-22T18:09:49.251813+0000 mon.a (mon.0) 3238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:09:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[46934]: audit 2022-04-22T18:09:49.254130+0000 mon.a (mon.0) 3239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:09:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:49 smithi115 bash[46934]: audit 2022-04-22T18:09:49.255351+0000 mon.a (mon.0) 3240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:09:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:50 smithi121 bash[36902]: cluster 2022-04-22T18:09:49.006359+0000 mgr.y (mgr.34107) 1543 : cluster [DBG] pgmap v2165: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 713 op/s 2022-04-22T18:09:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:50 smithi121 bash[36902]: audit 2022-04-22T18:09:49.614966+0000 mon.a (mon.0) 3241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:50 smithi115 bash[45495]: cluster 2022-04-22T18:09:49.006359+0000 mgr.y (mgr.34107) 1543 : cluster [DBG] pgmap v2165: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 713 op/s 2022-04-22T18:09:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:50 smithi115 bash[45495]: audit 2022-04-22T18:09:49.614966+0000 mon.a (mon.0) 3241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:50 smithi115 bash[46934]: cluster 2022-04-22T18:09:49.006359+0000 mgr.y (mgr.34107) 1543 : cluster [DBG] pgmap v2165: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 713 op/s 2022-04-22T18:09:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:50 smithi115 bash[46934]: audit 2022-04-22T18:09:49.614966+0000 mon.a (mon.0) 3241 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:09:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:51 smithi121 bash[36902]: audit 2022-04-22T18:09:50.610644+0000 mon.a (mon.0) 3242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:51 smithi121 bash[36902]: cluster 2022-04-22T18:09:51.008293+0000 mgr.y (mgr.34107) 1544 : cluster [DBG] pgmap v2166: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1005 KiB/s wr, 837 op/s 2022-04-22T18:09:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:51 smithi121 bash[36902]: audit 2022-04-22T18:09:51.040192+0000 mon.a (mon.0) 3243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:51.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[45495]: audit 2022-04-22T18:09:50.610644+0000 mon.a (mon.0) 3242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[45495]: cluster 2022-04-22T18:09:51.008293+0000 mgr.y (mgr.34107) 1544 : cluster [DBG] pgmap v2166: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1005 KiB/s wr, 837 op/s 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[45495]: audit 2022-04-22T18:09:51.040192+0000 mon.a (mon.0) 3243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[46934]: audit 2022-04-22T18:09:50.610644+0000 mon.a (mon.0) 3242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[46934]: cluster 2022-04-22T18:09:51.008293+0000 mgr.y (mgr.34107) 1544 : cluster [DBG] pgmap v2166: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1005 KiB/s wr, 837 op/s 2022-04-22T18:09:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:51 smithi115 bash[46934]: audit 2022-04-22T18:09:51.040192+0000 mon.a (mon.0) 3243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:53 smithi121 bash[36902]: audit 2022-04-22T18:09:53.001735+0000 mon.a (mon.0) 3244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:53.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:53 smithi115 bash[45495]: audit 2022-04-22T18:09:53.001735+0000 mon.a (mon.0) 3244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:53 smithi115 bash[46934]: audit 2022-04-22T18:09:53.001735+0000 mon.a (mon.0) 3244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.821 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.822 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.823 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.824 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.825 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.826 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.827 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.828 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.829 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.830 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.831 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.832 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.834 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.835 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.836 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.837 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.838 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (8582 ms) 2022-04-22T18:09:53.839 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2022-04-22T18:09:53.839 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:09:53.839 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestDataPoolIO (0 ms) 2022-04-22T18:09:53.839 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2022-04-22T18:09:53.839 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:53.846 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (26 ms) 2022-04-22T18:09:53.847 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2022-04-22T18:09:53.847 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 0~1048576 = 0 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 0~4194304 = 0 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:aio discard: 3145728~1048576 = 0 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (25 ms) 2022-04-22T18:09:53.872 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestFUA 2022-04-22T18:09:53.875 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:53.959 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:53.960 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.961 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.962 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.963 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:53.964 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:53.965 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestFUA (88 ms) 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestIOPP 2022-04-22T18:09:53.966 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:53.975 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.975 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.975 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.976 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.977 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:53.978 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:53.997 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:53.997 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:53.998 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:53.998 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:53.998 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:53.999 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:54.000 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:54.000 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:54.001 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:54.001 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:54.001 INFO:tasks.workunit.client.0.smithi115.stdout:start compare and write 2022-04-22T18:09:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:compare and wrote: 512 2022-04-22T18:09:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.002 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:54.003 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.005 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.006 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:started aio compare and write 2022-04-22T18:09:54.007 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.008 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:finished aio compare and write 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:54.009 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:54.010 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:54.011 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:54.012 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:54.012 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 512~512 2022-04-22T18:09:54.012 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:09:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:09:53.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:09:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:54 smithi121 bash[36902]: cluster 2022-04-22T18:09:53.009658+0000 mgr.y (mgr.34107) 1545 : cluster [DBG] pgmap v2167: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 323 KiB/s wr, 837 op/s 2022-04-22T18:09:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:54 smithi115 bash[46934]: cluster 2022-04-22T18:09:53.009658+0000 mgr.y (mgr.34107) 1545 : cluster [DBG] pgmap v2167: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 323 KiB/s wr, 837 op/s 2022-04-22T18:09:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:54 smithi115 bash[45495]: cluster 2022-04-22T18:09:53.009658+0000 mgr.y (mgr.34107) 1545 : cluster [DBG] pgmap v2167: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 323 KiB/s wr, 837 op/s 2022-04-22T18:09:55.943 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:55.944 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:55.944 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:55.944 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:55.944 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:55.945 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:55.945 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.945 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.945 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:55.945 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.946 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.946 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:55.946 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.946 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.946 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:55.947 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.947 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.947 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.948 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.949 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:55.949 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.949 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.949 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:55.950 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.950 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.950 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:55.950 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.950 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.951 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:55.951 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.951 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.951 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.951 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.952 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:55.952 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.952 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.952 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:55.952 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.953 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:55.954 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.955 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.955 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:55.955 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.955 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.955 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:55.956 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.967 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.968 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.968 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.969 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:55.969 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.970 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.970 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:55.970 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.971 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.971 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:55.972 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.972 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.972 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:55.972 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.972 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.973 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.973 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.973 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.973 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.974 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.975 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.975 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:55.975 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:55.979 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:55.979 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.980 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.980 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.980 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.980 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:55.982 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:55.983 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.984 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.985 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:55.985 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:55.985 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:55.985 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.024 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.025 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.026 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.026 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.026 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.026 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.027 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.068 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.069 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.069 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.069 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.069 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.069 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.070 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.135 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.135 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.136 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.136 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.136 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.136 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.137 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.193 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.194 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.194 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.194 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.194 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.194 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.195 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.293 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.387 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.388 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.500 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.501 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:56 smithi121 bash[36902]: cluster 2022-04-22T18:09:55.011593+0000 mgr.y (mgr.34107) 1546 : cluster [DBG] pgmap v2168: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 327 KiB/s wr, 845 op/s 2022-04-22T18:09:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:56 smithi121 bash[36902]: audit 2022-04-22T18:09:55.613962+0000 mon.a (mon.0) 3245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:56 smithi121 bash[36902]: audit 2022-04-22T18:09:56.043532+0000 mon.a (mon.0) 3246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[45495]: cluster 2022-04-22T18:09:55.011593+0000 mgr.y (mgr.34107) 1546 : cluster [DBG] pgmap v2168: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 327 KiB/s wr, 845 op/s 2022-04-22T18:09:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[45495]: audit 2022-04-22T18:09:55.613962+0000 mon.a (mon.0) 3245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[45495]: audit 2022-04-22T18:09:56.043532+0000 mon.a (mon.0) 3246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[46934]: cluster 2022-04-22T18:09:55.011593+0000 mgr.y (mgr.34107) 1546 : cluster [DBG] pgmap v2168: 329 pgs: 329 active+clean; 35 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 327 KiB/s wr, 845 op/s 2022-04-22T18:09:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[46934]: audit 2022-04-22T18:09:55.613962+0000 mon.a (mon.0) 3245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:09:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:56 smithi115 bash[46934]: audit 2022-04-22T18:09:56.043532+0000 mon.a (mon.0) 3246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:09:56.597 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.598 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.599 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:56.599 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.599 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.599 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.737 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.781 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.859 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.859 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.860 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:56.860 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:56.860 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:56.860 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:56.860 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:56.999 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:56.999 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:56.999 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.000 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.000 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.000 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.000 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.134 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.135 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.312 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.312 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:57.312 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.313 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.313 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.313 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.313 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.482 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.670 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.676 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:57.676 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.676 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.677 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.677 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.677 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.856 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.856 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:57.856 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:57.856 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:57.856 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestIOPP (3897 ms) 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2022-04-22T18:09:57.857 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:09:57.887 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:57.887 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:57.887 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:57.888 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:57.888 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:09:57.888 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.888 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:57.888 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.889 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.889 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:57.889 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.889 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:57.890 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.891 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.891 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:57.891 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.891 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:57.891 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.893 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.895 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:57.895 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.895 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:57.895 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.895 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.896 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:57.896 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.896 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:09:57.896 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:57.897 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:57.897 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:09:57.897 INFO:tasks.workunit.client.0.smithi115.stdout:read: 512 2022-04-22T18:09:57.897 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.898 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:57.898 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:57.898 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:57.898 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:57.899 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:57.900 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:57.900 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:57.900 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.900 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:57.900 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:57.901 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:57.901 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:57.901 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:57.901 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:09:57.901 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:09:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:58 smithi121 bash[36902]: cluster 2022-04-22T18:09:57.013174+0000 mgr.y (mgr.34107) 1547 : cluster [DBG] pgmap v2169: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 180 KiB/s wr, 721 op/s 2022-04-22T18:09:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:09:58 smithi121 bash[36902]: audit 2022-04-22T18:09:58.004348+0000 mon.a (mon.0) 3247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:58 smithi115 bash[46934]: cluster 2022-04-22T18:09:57.013174+0000 mgr.y (mgr.34107) 1547 : cluster [DBG] pgmap v2169: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 180 KiB/s wr, 721 op/s 2022-04-22T18:09:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:09:58 smithi115 bash[46934]: audit 2022-04-22T18:09:58.004348+0000 mon.a (mon.0) 3247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:58 smithi115 bash[45495]: cluster 2022-04-22T18:09:57.013174+0000 mgr.y (mgr.34107) 1547 : cluster [DBG] pgmap v2169: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 432 KiB/s rd, 180 KiB/s wr, 721 op/s 2022-04-22T18:09:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:09:58 smithi115 bash[45495]: audit 2022-04-22T18:09:58.004348+0000 mon.a (mon.0) 3247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:09:59.597 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:09:59.597 INFO:tasks.workunit.client.0.smithi115.stdout:finished read 2022-04-22T18:09:59.597 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:59.597 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.597 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 0 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 16384 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.598 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 49152 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.599 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 65536 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 98304 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.600 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 114688 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.601 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 147456 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 163840 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.602 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 196608 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.603 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 212992 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.605 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.605 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.605 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.605 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.606 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.607 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.608 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.608 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.608 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.608 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.609 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.613 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.614 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.628 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.628 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.629 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.630 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.630 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.674 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.674 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.718 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.718 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.718 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.718 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.719 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.719 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.719 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.779 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.838 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.839 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.839 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.839 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.839 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.840 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:09:59.841 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.841 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.841 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:09:59.929 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:09:59.929 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:09:59.930 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:09:59.930 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:09:59.930 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:09:59.930 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:09:59.930 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.021 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.022 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.129 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.228 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.229 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.365 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.365 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.365 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.366 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.366 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.366 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.366 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.496 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.496 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.496 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.497 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.497 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.497 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.497 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:00 smithi121 bash[36902]: cluster 2022-04-22T18:09:59.014413+0000 mgr.y (mgr.34107) 1548 : cluster [DBG] pgmap v2170: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 445 KiB/s rd, 188 KiB/s wr, 746 op/s 2022-04-22T18:10:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:00 smithi121 bash[36902]: cluster 2022-04-22T18:10:00.000119+0000 mon.a (mon.0) 3248 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:10:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:00 smithi115 bash[46934]: cluster 2022-04-22T18:09:59.014413+0000 mgr.y (mgr.34107) 1548 : cluster [DBG] pgmap v2170: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 445 KiB/s rd, 188 KiB/s wr, 746 op/s 2022-04-22T18:10:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:00 smithi115 bash[46934]: cluster 2022-04-22T18:10:00.000119+0000 mon.a (mon.0) 3248 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:10:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:00 smithi115 bash[45495]: cluster 2022-04-22T18:09:59.014413+0000 mgr.y (mgr.34107) 1548 : cluster [DBG] pgmap v2170: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 445 KiB/s rd, 188 KiB/s wr, 746 op/s 2022-04-22T18:10:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:00 smithi115 bash[45495]: cluster 2022-04-22T18:10:00.000119+0000 mon.a (mon.0) 3248 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:10:00.649 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.650 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.650 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.650 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.651 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.651 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.651 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.800 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.800 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.800 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.800 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.801 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.803 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.804 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:00.980 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:00.980 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:00.980 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:00.981 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:00.981 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:00.981 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:00.981 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:01.147 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:01.147 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:01.147 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:01.147 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:01.148 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:01.148 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:01.148 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:01.351 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:01 smithi121 bash[36902]: audit 2022-04-22T18:10:00.616531+0000 mon.a (mon.0) 3249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:01 smithi121 bash[36902]: audit 2022-04-22T18:10:01.046586+0000 mon.a (mon.0) 3250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:finished writesame 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:to verify the data 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:verified 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:01.539 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:started writesame 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:expected fail, finished writesame 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (3681 ms) 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2022-04-22T18:10:01.540 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:01 smithi115 bash[45495]: audit 2022-04-22T18:10:00.616531+0000 mon.a (mon.0) 3249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:01 smithi115 bash[45495]: audit 2022-04-22T18:10:01.046586+0000 mon.a (mon.0) 3250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:01 smithi115 bash[46934]: audit 2022-04-22T18:10:00.616531+0000 mon.a (mon.0) 3249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:01 smithi115 bash[46934]: audit 2022-04-22T18:10:01.046586+0000 mon.a (mon.0) 3250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:01.764Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:02.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:02 smithi121 bash[36902]: cluster 2022-04-22T18:10:01.016190+0000 mgr.y (mgr.34107) 1549 : cluster [DBG] pgmap v2171: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 711 KiB/s rd, 359 KiB/s wr, 1.43k op/s 2022-04-22T18:10:02.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:02 smithi115 bash[45495]: cluster 2022-04-22T18:10:01.016190+0000 mgr.y (mgr.34107) 1549 : cluster [DBG] pgmap v2171: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 711 KiB/s rd, 359 KiB/s wr, 1.43k op/s 2022-04-22T18:10:02.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:02 smithi115 bash[46934]: cluster 2022-04-22T18:10:01.016190+0000 mgr.y (mgr.34107) 1549 : cluster [DBG] pgmap v2171: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 711 KiB/s rd, 359 KiB/s wr, 1.43k op/s 2022-04-22T18:10:03.151 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:10:03.151 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:10:03.151 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:snap: orig 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:found orig with size 2097152 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:write test data! 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 80 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:10:03.152 INFO:tasks.workunit.client.0.smithi115.stdout:snap: orig 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:snap: written 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:found orig with size 2097152 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:found written with size 2097152 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.153 INFO:tasks.workunit.client.0.smithi115.stdout:write to snapshot returned -30 2022-04-22T18:10:03.154 INFO:tasks.workunit.client.0.smithi115.stdout:Read-only file system 2022-04-22T18:10:03.171 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.171 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.172 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 80 2022-04-22T18:10:03.172 INFO:tasks.workunit.client.0.smithi115.stdout:opening testimg@orig 2022-04-22T18:10:03.172 INFO:tasks.workunit.client.0.smithi115.stdout:read: 80 2022-04-22T18:10:03.172 INFO:tasks.workunit.client.0.smithi115.stdout:write to snapshot returned -30 2022-04-22T18:10:03.172 INFO:tasks.workunit.client.0.smithi115.stdout:Read-only file system 2022-04-22T18:10:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:03 smithi121 bash[36902]: cluster 2022-04-22T18:10:02.104384+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2022-04-22T18:10:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:03 smithi121 bash[36902]: audit 2022-04-22T18:10:03.007633+0000 mon.a (mon.0) 3252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:03.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:03 smithi115 bash[45495]: cluster 2022-04-22T18:10:02.104384+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2022-04-22T18:10:03.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:03 smithi115 bash[45495]: audit 2022-04-22T18:10:03.007633+0000 mon.a (mon.0) 3252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:03 smithi115 bash[46934]: cluster 2022-04-22T18:10:02.104384+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2022-04-22T18:10:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:03 smithi115 bash[46934]: audit 2022-04-22T18:10:03.007633+0000 mon.a (mon.0) 3252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:03.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:04 smithi121 bash[36902]: cluster 2022-04-22T18:10:03.017021+0000 mgr.y (mgr.34107) 1550 : cluster [DBG] pgmap v2173: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 615 KiB/s rd, 302 KiB/s wr, 1.24k op/s 2022-04-22T18:10:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:04 smithi121 bash[36902]: cluster 2022-04-22T18:10:03.120008+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2022-04-22T18:10:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:04 smithi115 bash[45495]: cluster 2022-04-22T18:10:03.017021+0000 mgr.y (mgr.34107) 1550 : cluster [DBG] pgmap v2173: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 615 KiB/s rd, 302 KiB/s wr, 1.24k op/s 2022-04-22T18:10:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:04 smithi115 bash[45495]: cluster 2022-04-22T18:10:03.120008+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2022-04-22T18:10:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:04 smithi115 bash[46934]: cluster 2022-04-22T18:10:03.017021+0000 mgr.y (mgr.34107) 1550 : cluster [DBG] pgmap v2173: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 615 KiB/s rd, 302 KiB/s wr, 1.24k op/s 2022-04-22T18:10:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:04 smithi115 bash[46934]: cluster 2022-04-22T18:10:03.120008+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2022-04-22T18:10:05.178 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 2 2022-04-22T18:10:05.178 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 2 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:snap: orig 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:snap: written 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:found orig with size 2097152 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:found written with size 2097152 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 1 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 1 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:snap: orig 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:found orig with size 2097152 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:num snaps is: 0 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:expected: 0 2022-04-22T18:10:05.179 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3640 ms) 2022-04-22T18:10:05.180 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2022-04-22T18:10:05.180 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:05 smithi121 bash[36902]: cluster 2022-04-22T18:10:04.136041+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2022-04-22T18:10:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:05 smithi115 bash[46934]: cluster 2022-04-22T18:10:04.136041+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2022-04-22T18:10:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:05 smithi115 bash[45495]: cluster 2022-04-22T18:10:04.136041+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2022-04-22T18:10:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:06 smithi121 bash[36902]: cluster 2022-04-22T18:10:05.018332+0000 mgr.y (mgr.34107) 1551 : cluster [DBG] pgmap v2176: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 681 KiB/s wr, 2.74k op/s 2022-04-22T18:10:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:06 smithi121 bash[36902]: cluster 2022-04-22T18:10:05.152219+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2022-04-22T18:10:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:06 smithi121 bash[36902]: audit 2022-04-22T18:10:05.619155+0000 mon.a (mon.0) 3256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:06 smithi121 bash[36902]: audit 2022-04-22T18:10:06.050476+0000 mon.a (mon.0) 3257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[45495]: cluster 2022-04-22T18:10:05.018332+0000 mgr.y (mgr.34107) 1551 : cluster [DBG] pgmap v2176: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 681 KiB/s wr, 2.74k op/s 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[45495]: cluster 2022-04-22T18:10:05.152219+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[45495]: audit 2022-04-22T18:10:05.619155+0000 mon.a (mon.0) 3256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[45495]: audit 2022-04-22T18:10:06.050476+0000 mon.a (mon.0) 3257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[46934]: cluster 2022-04-22T18:10:05.018332+0000 mgr.y (mgr.34107) 1551 : cluster [DBG] pgmap v2176: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 681 KiB/s wr, 2.74k op/s 2022-04-22T18:10:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[46934]: cluster 2022-04-22T18:10:05.152219+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2022-04-22T18:10:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[46934]: audit 2022-04-22T18:10:05.619155+0000 mon.a (mon.0) 3256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:06.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:06 smithi115 bash[46934]: audit 2022-04-22T18:10:06.050476+0000 mon.a (mon.0) 3257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:07.195 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2017 ms) 2022-04-22T18:10:07.195 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestClone 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestClone (0 ms) 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestClone2 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestClone2 (0 ms) 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ListChildren 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ListChildren (0 ms) 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2022-04-22T18:10:07.196 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.197 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ListChildrenTiered (0 ms) 2022-04-22T18:10:07.197 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.LockingPP 2022-04-22T18:10:07.197 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.270 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.LockingPP (74 ms) 2022-04-22T18:10:07.270 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FlushAio 2022-04-22T18:10:07.270 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:07 smithi121 bash[36902]: cluster 2022-04-22T18:10:06.157564+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2022-04-22T18:10:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:07 smithi115 bash[45495]: cluster 2022-04-22T18:10:06.157564+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2022-04-22T18:10:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:07 smithi115 bash[46934]: cluster 2022-04-22T18:10:06.157564+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2022-04-22T18:10:07.556 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FlushAio (282 ms) 2022-04-22T18:10:07.556 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FlushAioPP 2022-04-22T18:10:07.556 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FlushAioPP (261 ms) 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapDiff 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapDiff (0 ms) 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2022-04-22T18:10:07.813 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (21 ms) 2022-04-22T18:10:07.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2022-04-22T18:10:07.834 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (24 ms) 2022-04-22T18:10:07.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2022-04-22T18:10:07.857 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.877 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ZeroLengthRead (20 ms) 2022-04-22T18:10:07.877 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2022-04-22T18:10:07.877 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.918 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.LargeCacheRead (40 ms) 2022-04-22T18:10:07.918 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestPendingAio 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestPendingAio (0 ms) 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (1 ms) 2022-04-22T18:10:07.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Flatten 2022-04-22T18:10:07.920 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:07.920 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Flatten (0 ms) 2022-04-22T18:10:07.920 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Sparsify 2022-04-22T18:10:07.920 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.953 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 32768 2022-04-22T18:10:07.954 INFO:tasks.workunit.client.0.smithi115.stdout:read: 32768 2022-04-22T18:10:07.954 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Sparsify (35 ms) 2022-04-22T18:10:07.954 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SparsifyPP 2022-04-22T18:10:07.954 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:07.987 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SparsifyPP (32 ms) 2022-04-22T18:10:07.987 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2022-04-22T18:10:07.987 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:08 smithi121 bash[36902]: cluster 2022-04-22T18:10:07.019156+0000 mgr.y (mgr.34107) 1552 : cluster [DBG] pgmap v2179: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1014 KiB/s rd, 506 KiB/s wr, 2.04k op/s 2022-04-22T18:10:08.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:08 smithi121 bash[36902]: cluster 2022-04-22T18:10:07.169304+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2022-04-22T18:10:08.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:08 smithi121 bash[36902]: audit 2022-04-22T18:10:08.010770+0000 mon.a (mon.0) 3260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[45495]: cluster 2022-04-22T18:10:07.019156+0000 mgr.y (mgr.34107) 1552 : cluster [DBG] pgmap v2179: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1014 KiB/s rd, 506 KiB/s wr, 2.04k op/s 2022-04-22T18:10:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[45495]: cluster 2022-04-22T18:10:07.169304+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2022-04-22T18:10:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[45495]: audit 2022-04-22T18:10:08.010770+0000 mon.a (mon.0) 3260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[46934]: cluster 2022-04-22T18:10:07.019156+0000 mgr.y (mgr.34107) 1552 : cluster [DBG] pgmap v2179: 329 pgs: 329 active+clean; 37 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1014 KiB/s rd, 506 KiB/s wr, 2.04k op/s 2022-04-22T18:10:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[46934]: cluster 2022-04-22T18:10:07.169304+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2022-04-22T18:10:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:08 smithi115 bash[46934]: audit 2022-04-22T18:10:08.010770+0000 mon.a (mon.0) 3260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:09 smithi121 bash[36902]: cluster 2022-04-22T18:10:08.174438+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2022-04-22T18:10:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:09 smithi115 bash[45495]: cluster 2022-04-22T18:10:08.174438+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2022-04-22T18:10:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:09 smithi115 bash[46934]: cluster 2022-04-22T18:10:08.174438+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2022-04-22T18:10:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:10 smithi121 bash[36902]: cluster 2022-04-22T18:10:09.020342+0000 mgr.y (mgr.34107) 1553 : cluster [DBG] pgmap v2182: 329 pgs: 329 active+clean; 39 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 544 KiB/s rd, 596 KiB/s wr, 102 op/s 2022-04-22T18:10:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:10 smithi121 bash[36902]: cluster 2022-04-22T18:10:09.181372+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2022-04-22T18:10:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:10 smithi115 bash[45495]: cluster 2022-04-22T18:10:09.020342+0000 mgr.y (mgr.34107) 1553 : cluster [DBG] pgmap v2182: 329 pgs: 329 active+clean; 39 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 544 KiB/s rd, 596 KiB/s wr, 102 op/s 2022-04-22T18:10:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:10 smithi115 bash[45495]: cluster 2022-04-22T18:10:09.181372+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2022-04-22T18:10:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:10 smithi115 bash[46934]: cluster 2022-04-22T18:10:09.020342+0000 mgr.y (mgr.34107) 1553 : cluster [DBG] pgmap v2182: 329 pgs: 329 active+clean; 39 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 544 KiB/s rd, 596 KiB/s wr, 102 op/s 2022-04-22T18:10:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:10 smithi115 bash[46934]: cluster 2022-04-22T18:10:09.181372+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2022-04-22T18:10:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:11 smithi121 bash[36902]: cluster 2022-04-22T18:10:10.190528+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2022-04-22T18:10:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:11 smithi121 bash[36902]: audit 2022-04-22T18:10:10.622351+0000 mon.a (mon.0) 3264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:11 smithi121 bash[36902]: audit 2022-04-22T18:10:11.054312+0000 mon.a (mon.0) 3265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[46934]: cluster 2022-04-22T18:10:10.190528+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2022-04-22T18:10:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[46934]: audit 2022-04-22T18:10:10.622351+0000 mon.a (mon.0) 3264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[46934]: audit 2022-04-22T18:10:11.054312+0000 mon.a (mon.0) 3265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[45495]: cluster 2022-04-22T18:10:10.190528+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2022-04-22T18:10:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[45495]: audit 2022-04-22T18:10:10.622351+0000 mon.a (mon.0) 3264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[45495]: audit 2022-04-22T18:10:11.054312+0000 mon.a (mon.0) 3265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:11.765Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:12.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapshotLimit (4247 ms) 2022-04-22T18:10:12.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2022-04-22T18:10:12.233 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:12 smithi121 bash[36902]: cluster 2022-04-22T18:10:11.021898+0000 mgr.y (mgr.34107) 1554 : cluster [DBG] pgmap v2185: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 665 KiB/s wr, 173 op/s 2022-04-22T18:10:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:12 smithi121 bash[36902]: cluster 2022-04-22T18:10:11.201895+0000 mon.a (mon.0) 3266 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2022-04-22T18:10:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:12 smithi115 bash[45495]: cluster 2022-04-22T18:10:11.021898+0000 mgr.y (mgr.34107) 1554 : cluster [DBG] pgmap v2185: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 665 KiB/s wr, 173 op/s 2022-04-22T18:10:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:12 smithi115 bash[45495]: cluster 2022-04-22T18:10:11.201895+0000 mon.a (mon.0) 3266 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2022-04-22T18:10:12.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:12 smithi115 bash[46934]: cluster 2022-04-22T18:10:11.021898+0000 mgr.y (mgr.34107) 1554 : cluster [DBG] pgmap v2185: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 665 KiB/s wr, 173 op/s 2022-04-22T18:10:12.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:12 smithi115 bash[46934]: cluster 2022-04-22T18:10:11.201895+0000 mon.a (mon.0) 3266 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2022-04-22T18:10:12.725 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:10:13.078 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:10:13.078 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (33m) 2m ago 47m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:10:13.078 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (46m) 3m ago 46m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:10:13.078 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (35m) 2m ago 48m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (34m) 3m ago 48m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (42m) 3m ago 52m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (43m) 2m ago 54m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (42m) 2m ago 54m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (41m) 3m ago 52m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (41m) 2m ago 52m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (34m) 2m ago 47m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (34m) 3m ago 47m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (40m) 2m ago 51m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:10:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (39m) 2m ago 51m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (38m) 2m ago 50m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (38m) 2m ago 50m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (37m) 3m ago 50m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (37m) 3m ago 49m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (36m) 3m ago 49m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (35m) 3m ago 48m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:10:13.080 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (33m) 3m ago 47m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:10:13.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:10:13.494 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:10:13.495 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:10:13.495 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:10:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:13 smithi121 bash[36902]: cluster 2022-04-22T18:10:12.216935+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2022-04-22T18:10:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:13 smithi121 bash[36902]: audit 2022-04-22T18:10:13.013572+0000 mon.a (mon.0) 3268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:13 smithi115 bash[45495]: cluster 2022-04-22T18:10:12.216935+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2022-04-22T18:10:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:13 smithi115 bash[45495]: audit 2022-04-22T18:10:13.013572+0000 mon.a (mon.0) 3268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:13 smithi115 bash[46934]: cluster 2022-04-22T18:10:12.216935+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2022-04-22T18:10:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:13 smithi115 bash[46934]: audit 2022-04-22T18:10:13.013572+0000 mon.a (mon.0) 3268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:13.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: audit 2022-04-22T18:10:12.701760+0000 mgr.y (mgr.34107) 1555 : audit [DBG] from='client.46678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: cluster 2022-04-22T18:10:13.022859+0000 mgr.y (mgr.34107) 1556 : cluster [DBG] pgmap v2188: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 72 KiB/s wr, 82 op/s 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: audit 2022-04-22T18:10:13.049566+0000 mgr.y (mgr.34107) 1557 : audit [DBG] from='client.36753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: cluster 2022-04-22T18:10:13.230984+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: audit 2022-04-22T18:10:13.487732+0000 mon.a (mon.0) 3270 : audit [DBG] from='client.? 172.21.15.115:0/1655379974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: audit 2022-04-22T18:10:13.826993+0000 mon.a (mon.0) 3271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:10:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:14 smithi121 bash[36902]: audit 2022-04-22T18:10:13.830543+0000 mon.a (mon.0) 3272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: audit 2022-04-22T18:10:12.701760+0000 mgr.y (mgr.34107) 1555 : audit [DBG] from='client.46678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: cluster 2022-04-22T18:10:13.022859+0000 mgr.y (mgr.34107) 1556 : cluster [DBG] pgmap v2188: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 72 KiB/s wr, 82 op/s 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: audit 2022-04-22T18:10:13.049566+0000 mgr.y (mgr.34107) 1557 : audit [DBG] from='client.36753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: cluster 2022-04-22T18:10:13.230984+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: audit 2022-04-22T18:10:13.487732+0000 mon.a (mon.0) 3270 : audit [DBG] from='client.? 172.21.15.115:0/1655379974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: audit 2022-04-22T18:10:13.826993+0000 mon.a (mon.0) 3271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[45495]: audit 2022-04-22T18:10:13.830543+0000 mon.a (mon.0) 3272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: audit 2022-04-22T18:10:12.701760+0000 mgr.y (mgr.34107) 1555 : audit [DBG] from='client.46678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: cluster 2022-04-22T18:10:13.022859+0000 mgr.y (mgr.34107) 1556 : cluster [DBG] pgmap v2188: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 72 KiB/s wr, 82 op/s 2022-04-22T18:10:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: audit 2022-04-22T18:10:13.049566+0000 mgr.y (mgr.34107) 1557 : audit [DBG] from='client.36753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: cluster 2022-04-22T18:10:13.230984+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2022-04-22T18:10:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: audit 2022-04-22T18:10:13.487732+0000 mon.a (mon.0) 3270 : audit [DBG] from='client.? 172.21.15.115:0/1655379974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: audit 2022-04-22T18:10:13.826993+0000 mon.a (mon.0) 3271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:10:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:14 smithi115 bash[46934]: audit 2022-04-22T18:10:13.830543+0000 mon.a (mon.0) 3272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:10:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:15 smithi121 bash[36902]: cluster 2022-04-22T18:10:14.234719+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2022-04-22T18:10:15.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:15 smithi115 bash[45495]: cluster 2022-04-22T18:10:14.234719+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2022-04-22T18:10:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:15 smithi115 bash[46934]: cluster 2022-04-22T18:10:14.234719+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2022-04-22T18:10:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:16 smithi121 bash[36902]: cluster 2022-04-22T18:10:15.023682+0000 mgr.y (mgr.34107) 1558 : cluster [DBG] pgmap v2191: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2022-04-22T18:10:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:16 smithi121 bash[36902]: cluster 2022-04-22T18:10:15.243009+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2022-04-22T18:10:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:16 smithi121 bash[36902]: audit 2022-04-22T18:10:15.625403+0000 mon.a (mon.0) 3275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:16 smithi121 bash[36902]: audit 2022-04-22T18:10:16.058093+0000 mon.a (mon.0) 3276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:16 smithi121 bash[36902]: cluster 2022-04-22T18:10:16.081704+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2022-04-22T18:10:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[45495]: cluster 2022-04-22T18:10:15.023682+0000 mgr.y (mgr.34107) 1558 : cluster [DBG] pgmap v2191: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2022-04-22T18:10:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[45495]: cluster 2022-04-22T18:10:15.243009+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[45495]: audit 2022-04-22T18:10:15.625403+0000 mon.a (mon.0) 3275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[45495]: audit 2022-04-22T18:10:16.058093+0000 mon.a (mon.0) 3276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[45495]: cluster 2022-04-22T18:10:16.081704+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[46934]: cluster 2022-04-22T18:10:15.023682+0000 mgr.y (mgr.34107) 1558 : cluster [DBG] pgmap v2191: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[46934]: cluster 2022-04-22T18:10:15.243009+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[46934]: audit 2022-04-22T18:10:15.625403+0000 mon.a (mon.0) 3275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[46934]: audit 2022-04-22T18:10:16.058093+0000 mon.a (mon.0) 3276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:16 smithi115 bash[46934]: cluster 2022-04-22T18:10:16.081704+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2022-04-22T18:10:17.116 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (4883 ms) 2022-04-22T18:10:17.116 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (0 ms) 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2022-04-22T18:10:17.117 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapRemove2 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapRemove2 (0 ms) 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.118 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (0 ms) 2022-04-22T18:10:17.119 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (0 ms) 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2022-04-22T18:10:17.120 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Metadata 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Metadata (0 ms) 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.MetadataPP 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.MetadataPP (0 ms) 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2022-04-22T18:10:17.121 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.122 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.UpdateFeatures (0 ms) 2022-04-22T18:10:17.122 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2022-04-22T18:10:17.122 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2022-04-22T18:10:17.122 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2022-04-22T18:10:17.122 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:17.136 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RebuildObjectMap (20 ms) 2022-04-22T18:10:17.136 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2022-04-22T18:10:17.136 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.136 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (0 ms) 2022-04-22T18:10:17.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2022-04-22T18:10:17.137 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CheckObjectMap (0 ms) 2022-04-22T18:10:17.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.BlockingAIO 2022-04-22T18:10:17.137 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:17.164 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.BlockingAIO (28 ms) 2022-04-22T18:10:17.165 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2022-04-22T18:10:17.165 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.165 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (0 ms) 2022-04-22T18:10:17.165 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2022-04-22T18:10:17.165 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2022-04-22T18:10:17.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2022-04-22T18:10:17.166 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:17.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2022-04-22T18:10:17.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2022-04-22T18:10:17.167 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:18.127 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (962 ms) 2022-04-22T18:10:18.127 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestImageOptions 2022-04-22T18:10:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:18 smithi121 bash[36902]: cluster 2022-04-22T18:10:17.025258+0000 mgr.y (mgr.34107) 1559 : cluster [DBG] pgmap v2194: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2022-04-22T18:10:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:18 smithi121 bash[36902]: cluster 2022-04-22T18:10:17.098189+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2022-04-22T18:10:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:18 smithi121 bash[36902]: audit 2022-04-22T18:10:18.016566+0000 mon.a (mon.0) 3279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[45495]: cluster 2022-04-22T18:10:17.025258+0000 mgr.y (mgr.34107) 1559 : cluster [DBG] pgmap v2194: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2022-04-22T18:10:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[45495]: cluster 2022-04-22T18:10:17.098189+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2022-04-22T18:10:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[45495]: audit 2022-04-22T18:10:18.016566+0000 mon.a (mon.0) 3279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[46934]: cluster 2022-04-22T18:10:17.025258+0000 mgr.y (mgr.34107) 1559 : cluster [DBG] pgmap v2194: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2022-04-22T18:10:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[46934]: cluster 2022-04-22T18:10:17.098189+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e1147: 8 total, 8 up, 8 in 2022-04-22T18:10:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:18 smithi115 bash[46934]: audit 2022-04-22T18:10:18.016566+0000 mon.a (mon.0) 3279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:19 smithi121 bash[36902]: cluster 2022-04-22T18:10:18.097900+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2022-04-22T18:10:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:19 smithi115 bash[46934]: cluster 2022-04-22T18:10:18.097900+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2022-04-22T18:10:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:19 smithi115 bash[45495]: cluster 2022-04-22T18:10:18.097900+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e1148: 8 total, 8 up, 8 in 2022-04-22T18:10:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:20 smithi121 bash[36902]: cluster 2022-04-22T18:10:19.026902+0000 mgr.y (mgr.34107) 1560 : cluster [DBG] pgmap v2197: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2022-04-22T18:10:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:20 smithi121 bash[36902]: cluster 2022-04-22T18:10:19.120720+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2022-04-22T18:10:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:20 smithi115 bash[45495]: cluster 2022-04-22T18:10:19.026902+0000 mgr.y (mgr.34107) 1560 : cluster [DBG] pgmap v2197: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2022-04-22T18:10:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:20 smithi115 bash[45495]: cluster 2022-04-22T18:10:19.120720+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2022-04-22T18:10:20.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:20 smithi115 bash[46934]: cluster 2022-04-22T18:10:19.026902+0000 mgr.y (mgr.34107) 1560 : cluster [DBG] pgmap v2197: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2022-04-22T18:10:20.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:20 smithi115 bash[46934]: cluster 2022-04-22T18:10:19.120720+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e1149: 8 total, 8 up, 8 in 2022-04-22T18:10:21.289 INFO:tasks.workunit.client.0.smithi115.stdout:0.00% done 2022-04-22T18:10:21.289 INFO:tasks.workunit.client.0.smithi115.stdout:100.00% done 2022-04-22T18:10:21.289 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestImageOptions (3163 ms) 2022-04-22T18:10:21.289 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2022-04-22T18:10:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:21 smithi121 bash[36902]: cluster 2022-04-22T18:10:20.126618+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2022-04-22T18:10:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:21 smithi121 bash[36902]: audit 2022-04-22T18:10:20.628631+0000 mon.a (mon.0) 3283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:21 smithi121 bash[36902]: audit 2022-04-22T18:10:21.061616+0000 mon.a (mon.0) 3284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:21 smithi121 bash[36902]: cluster 2022-04-22T18:10:21.081446+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e1151: 8 total, 8 up, 8 in 2022-04-22T18:10:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[45495]: cluster 2022-04-22T18:10:20.126618+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2022-04-22T18:10:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[45495]: audit 2022-04-22T18:10:20.628631+0000 mon.a (mon.0) 3283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[45495]: audit 2022-04-22T18:10:21.061616+0000 mon.a (mon.0) 3284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[45495]: cluster 2022-04-22T18:10:21.081446+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e1151: 8 total, 8 up, 8 in 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[46934]: cluster 2022-04-22T18:10:20.126618+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e1150: 8 total, 8 up, 8 in 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[46934]: audit 2022-04-22T18:10:20.628631+0000 mon.a (mon.0) 3283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[46934]: audit 2022-04-22T18:10:21.061616+0000 mon.a (mon.0) 3284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[46934]: cluster 2022-04-22T18:10:21.081446+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e1151: 8 total, 8 up, 8 in 2022-04-22T18:10:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:21.763Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:22.358 INFO:tasks.workunit.client.0.smithi115.stdout:0.00% done 2022-04-22T18:10:22.368 INFO:tasks.workunit.client.0.smithi115.stdout:100.00% done 2022-04-22T18:10:22.368 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1068 ms) 2022-04-22T18:10:22.368 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2022-04-22T18:10:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2022-04-22T18:10:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2022-04-22T18:10:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2022-04-22T18:10:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ImagePollIO 2022-04-22T18:10:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:22.410 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:10:22.410 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:10:22.410 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:10:22.411 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:10:22.411 INFO:tasks.workunit.client.0.smithi115.stdout:wrote: 512 2022-04-22T18:10:22.411 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.411 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:10:22.411 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:22.412 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:22.412 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:10:22.412 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.413 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:10:22.413 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:22.414 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:22.414 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:10:22.414 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.414 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:22.415 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:started write 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:write completion cb called! 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 0 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:finished write 2022-04-22T18:10:22.416 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.417 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:10:22.420 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:created completion 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:started read 2022-04-22T18:10:22.421 INFO:tasks.workunit.client.0.smithi115.stdout:read completion cb called! 2022-04-22T18:10:22.422 INFO:tasks.workunit.client.0.smithi115.stdout:return value is: 512 2022-04-22T18:10:22.422 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ImagePollIO (53 ms) 2022-04-22T18:10:22.422 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Mirror 2022-04-22T18:10:22.423 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:22.425 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:22.502 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Mirror (90 ms) 2022-04-22T18:10:22.502 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2022-04-22T18:10:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:22 smithi121 bash[36902]: cluster 2022-04-22T18:10:21.028272+0000 mgr.y (mgr.34107) 1561 : cluster [DBG] pgmap v2200: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2022-04-22T18:10:22.528 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (29 ms) 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.529 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ExclusiveLock (0 ms) 2022-04-22T18:10:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.BreakLock 2022-04-22T18:10:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.BreakLock (0 ms) 2022-04-22T18:10:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2022-04-22T18:10:22.530 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:22 smithi115 bash[46934]: cluster 2022-04-22T18:10:21.028272+0000 mgr.y (mgr.34107) 1561 : cluster [DBG] pgmap v2200: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2022-04-22T18:10:22.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:22 smithi115 bash[45495]: cluster 2022-04-22T18:10:21.028272+0000 mgr.y (mgr.34107) 1561 : cluster [DBG] pgmap v2200: 329 pgs: 329 active+clean; 41 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2022-04-22T18:10:22.547 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (19 ms) 2022-04-22T18:10:22.547 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2022-04-22T18:10:22.547 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.548 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestTrashPurge (0 ms) 2022-04-22T18:10:22.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2022-04-22T18:10:22.549 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:22.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2022-04-22T18:10:22.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestListWatchers 2022-04-22T18:10:22.549 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:22.572 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestListWatchers (24 ms) 2022-04-22T18:10:22.572 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2022-04-22T18:10:22.572 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: cluster 2022-04-22T18:10:22.150024+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2022-04-22T18:10:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.441472+0000 mon.a (mon.0) 3287 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]: dispatch 2022-04-22T18:10:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.448288+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]': finished 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.452092+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]: dispatch 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.458817+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]': finished 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.472326+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]: dispatch 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.477364+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]': finished 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.481666+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]: dispatch 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.485758+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]': finished 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.503203+0000 mon.a (mon.0) 3295 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.510108+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' 2022-04-22T18:10:23.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.510841+0000 mon.a (mon.0) 3297 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.511451+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:22.515611+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]': finished 2022-04-22T18:10:23.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:23 smithi121 bash[36902]: audit 2022-04-22T18:10:23.020005+0000 mon.a (mon.0) 3300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:23.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: cluster 2022-04-22T18:10:22.150024+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.441472+0000 mon.a (mon.0) 3287 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]: dispatch 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.448288+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]': finished 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.452092+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]: dispatch 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.458817+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]': finished 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.472326+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]: dispatch 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.477364+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]': finished 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.481666+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]: dispatch 2022-04-22T18:10:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.485758+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]': finished 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.503203+0000 mon.a (mon.0) 3295 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.510108+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.510841+0000 mon.a (mon.0) 3297 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.511451+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:22.515611+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]': finished 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[45495]: audit 2022-04-22T18:10:23.020005+0000 mon.a (mon.0) 3300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: cluster 2022-04-22T18:10:22.150024+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e1152: 8 total, 8 up, 8 in 2022-04-22T18:10:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.441472+0000 mon.a (mon.0) 3287 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]: dispatch 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.448288+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/uuid4"}]': finished 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.452092+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]: dispatch 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.458817+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/c106b46e-feed-468b-a6d4-322872e918ba"}]': finished 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.472326+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]: dispatch 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.477364+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/0a386461-147f-41c8-a1d8-33d030486051"}]': finished 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.481666+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]: dispatch 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.485758+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/3f3a3667-cd19-4401-92d7-384d235c7bb6"}]': finished 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.503203+0000 mon.a (mon.0) 3295 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.510108+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' 2022-04-22T18:10:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.510841+0000 mon.a (mon.0) 3297 : audit [DBG] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.511451+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]: dispatch 2022-04-22T18:10:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:22.515611+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? 172.21.15.115:0/1409503936' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/65/efdd9cee-ce50-4c1e-b746-9da1a2d5a7dc"}]': finished 2022-04-22T18:10:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[46934]: audit 2022-04-22T18:10:23.020005+0000 mon.a (mon.0) 3300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:23.546 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestSetSnapById (974 ms) 2022-04-22T18:10:23.546 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Namespaces 2022-04-22T18:10:23.587 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Namespaces (41 ms) 2022-04-22T18:10:23.587 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.NamespacesPP 2022-04-22T18:10:23.646 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:23.648 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.NamespacesPP (62 ms) 2022-04-22T18:10:23.648 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Migration 2022-04-22T18:10:23.648 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:23.648 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:24.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:23.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:24.236 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Migration (587 ms) 2022-04-22T18:10:24.236 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.MigrationPP 2022-04-22T18:10:24.236 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:24.236 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:24 smithi115 bash[45495]: cluster 2022-04-22T18:10:23.029378+0000 mgr.y (mgr.34107) 1562 : cluster [DBG] pgmap v2203: 329 pgs: 329 active+clean; 44 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 833 KiB/s wr, 14 op/s 2022-04-22T18:10:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:24 smithi115 bash[45495]: cluster 2022-04-22T18:10:23.527980+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2022-04-22T18:10:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:24 smithi115 bash[46934]: cluster 2022-04-22T18:10:23.029378+0000 mgr.y (mgr.34107) 1562 : cluster [DBG] pgmap v2203: 329 pgs: 329 active+clean; 44 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 833 KiB/s wr, 14 op/s 2022-04-22T18:10:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:24 smithi115 bash[46934]: cluster 2022-04-22T18:10:23.527980+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2022-04-22T18:10:24.830 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.MigrationPP (594 ms) 2022-04-22T18:10:24.830 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2022-04-22T18:10:24.831 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2022-04-22T18:10:24.832 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.832 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2022-04-22T18:10:24.832 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.PoolMetadata 2022-04-22T18:10:24.832 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.832 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.PoolMetadata (0 ms) 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.PoolMetadataPP (0 ms) 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.Config 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.Config (0 ms) 2022-04-22T18:10:24.833 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ConfigPP 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ConfigPP (0 ms) 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.PoolStatsPP (0 ms) 2022-04-22T18:10:24.834 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ImageSpec 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ImageSpec (0 ms) 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2022-04-22T18:10:24.835 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2022-04-22T18:10:24.836 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:24 smithi121 bash[36902]: cluster 2022-04-22T18:10:23.029378+0000 mgr.y (mgr.34107) 1562 : cluster [DBG] pgmap v2203: 329 pgs: 329 active+clean; 44 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 833 KiB/s wr, 14 op/s 2022-04-22T18:10:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:24 smithi121 bash[36902]: cluster 2022-04-22T18:10:23.527980+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e1153: 8 total, 8 up, 8 in 2022-04-22T18:10:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[45495]: cluster 2022-04-22T18:10:25.030853+0000 mgr.y (mgr.34107) 1563 : cluster [DBG] pgmap v2205: 329 pgs: 329 active+clean; 47 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[45495]: cluster 2022-04-22T18:10:25.538236+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[45495]: audit 2022-04-22T18:10:25.631931+0000 mon.a (mon.0) 3303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[45495]: audit 2022-04-22T18:10:26.065667+0000 mon.a (mon.0) 3304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[45495]: cluster 2022-04-22T18:10:26.087731+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1155: 8 total, 8 up, 8 in 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[46934]: cluster 2022-04-22T18:10:25.030853+0000 mgr.y (mgr.34107) 1563 : cluster [DBG] pgmap v2205: 329 pgs: 329 active+clean; 47 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[46934]: cluster 2022-04-22T18:10:25.538236+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[46934]: audit 2022-04-22T18:10:25.631931+0000 mon.a (mon.0) 3303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[46934]: audit 2022-04-22T18:10:26.065667+0000 mon.a (mon.0) 3304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:26 smithi115 bash[46934]: cluster 2022-04-22T18:10:26.087731+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1155: 8 total, 8 up, 8 in 2022-04-22T18:10:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:26 smithi121 bash[36902]: cluster 2022-04-22T18:10:25.030853+0000 mgr.y (mgr.34107) 1563 : cluster [DBG] pgmap v2205: 329 pgs: 329 active+clean; 47 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2022-04-22T18:10:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:26 smithi121 bash[36902]: cluster 2022-04-22T18:10:25.538236+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in 2022-04-22T18:10:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:26 smithi121 bash[36902]: audit 2022-04-22T18:10:25.631931+0000 mon.a (mon.0) 3303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:26 smithi121 bash[36902]: audit 2022-04-22T18:10:26.065667+0000 mon.a (mon.0) 3304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:26 smithi121 bash[36902]: cluster 2022-04-22T18:10:26.087731+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1155: 8 total, 8 up, 8 in 2022-04-22T18:10:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:28 smithi121 bash[36902]: cluster 2022-04-22T18:10:27.032348+0000 mgr.y (mgr.34107) 1564 : cluster [DBG] pgmap v2208: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 344 op/s 2022-04-22T18:10:28.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:28 smithi121 bash[36902]: cluster 2022-04-22T18:10:27.103654+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2022-04-22T18:10:28.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:28 smithi121 bash[36902]: audit 2022-04-22T18:10:28.023025+0000 mon.a (mon.0) 3307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[45495]: cluster 2022-04-22T18:10:27.032348+0000 mgr.y (mgr.34107) 1564 : cluster [DBG] pgmap v2208: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 344 op/s 2022-04-22T18:10:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[45495]: cluster 2022-04-22T18:10:27.103654+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2022-04-22T18:10:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[45495]: audit 2022-04-22T18:10:28.023025+0000 mon.a (mon.0) 3307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[46934]: cluster 2022-04-22T18:10:27.032348+0000 mgr.y (mgr.34107) 1564 : cluster [DBG] pgmap v2208: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 344 op/s 2022-04-22T18:10:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[46934]: cluster 2022-04-22T18:10:27.103654+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1156: 8 total, 8 up, 8 in 2022-04-22T18:10:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:28 smithi115 bash[46934]: audit 2022-04-22T18:10:28.023025+0000 mon.a (mon.0) 3307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:29 smithi121 bash[36902]: cluster 2022-04-22T18:10:28.112421+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e1157: 8 total, 8 up, 8 in 2022-04-22T18:10:29.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:29 smithi115 bash[45495]: cluster 2022-04-22T18:10:28.112421+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e1157: 8 total, 8 up, 8 in 2022-04-22T18:10:29.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:29 smithi115 bash[46934]: cluster 2022-04-22T18:10:28.112421+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e1157: 8 total, 8 up, 8 in 2022-04-22T18:10:30.179 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.QuiesceWatch (5350 ms) 2022-04-22T18:10:30.225 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2022-04-22T18:10:30.225 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:30 smithi121 bash[36902]: cluster 2022-04-22T18:10:29.033922+0000 mgr.y (mgr.34107) 1565 : cluster [DBG] pgmap v2211: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 471 KiB/s wr, 422 op/s 2022-04-22T18:10:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:30 smithi121 bash[36902]: cluster 2022-04-22T18:10:29.121627+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e1158: 8 total, 8 up, 8 in 2022-04-22T18:10:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:30 smithi115 bash[45495]: cluster 2022-04-22T18:10:29.033922+0000 mgr.y (mgr.34107) 1565 : cluster [DBG] pgmap v2211: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 471 KiB/s wr, 422 op/s 2022-04-22T18:10:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:30 smithi115 bash[45495]: cluster 2022-04-22T18:10:29.121627+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e1158: 8 total, 8 up, 8 in 2022-04-22T18:10:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:30 smithi115 bash[46934]: cluster 2022-04-22T18:10:29.033922+0000 mgr.y (mgr.34107) 1565 : cluster [DBG] pgmap v2211: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 471 KiB/s wr, 422 op/s 2022-04-22T18:10:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:30 smithi115 bash[46934]: cluster 2022-04-22T18:10:29.121627+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e1158: 8 total, 8 up, 8 in 2022-04-22T18:10:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:31 smithi121 bash[36902]: cluster 2022-04-22T18:10:30.127813+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e1159: 8 total, 8 up, 8 in 2022-04-22T18:10:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:31 smithi121 bash[36902]: audit 2022-04-22T18:10:30.635062+0000 mon.a (mon.0) 3311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:31 smithi121 bash[36902]: audit 2022-04-22T18:10:31.069447+0000 mon.a (mon.0) 3312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:31 smithi121 bash[36902]: cluster 2022-04-22T18:10:31.088418+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2022-04-22T18:10:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[45495]: cluster 2022-04-22T18:10:30.127813+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e1159: 8 total, 8 up, 8 in 2022-04-22T18:10:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[45495]: audit 2022-04-22T18:10:30.635062+0000 mon.a (mon.0) 3311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[45495]: audit 2022-04-22T18:10:31.069447+0000 mon.a (mon.0) 3312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[45495]: cluster 2022-04-22T18:10:31.088418+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[46934]: cluster 2022-04-22T18:10:30.127813+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e1159: 8 total, 8 up, 8 in 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[46934]: audit 2022-04-22T18:10:30.635062+0000 mon.a (mon.0) 3311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[46934]: audit 2022-04-22T18:10:31.069447+0000 mon.a (mon.0) 3312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[46934]: cluster 2022-04-22T18:10:31.088418+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1160: 8 total, 8 up, 8 in 2022-04-22T18:10:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:31.766Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:32 smithi121 bash[36902]: cluster 2022-04-22T18:10:31.035234+0000 mgr.y (mgr.34107) 1566 : cluster [DBG] pgmap v2214: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 9.0 KiB/s wr, 196 op/s 2022-04-22T18:10:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:32 smithi121 bash[36902]: cluster 2022-04-22T18:10:32.101213+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e1161: 8 total, 8 up, 8 in 2022-04-22T18:10:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:32 smithi115 bash[45495]: cluster 2022-04-22T18:10:31.035234+0000 mgr.y (mgr.34107) 1566 : cluster [DBG] pgmap v2214: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 9.0 KiB/s wr, 196 op/s 2022-04-22T18:10:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:32 smithi115 bash[45495]: cluster 2022-04-22T18:10:32.101213+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e1161: 8 total, 8 up, 8 in 2022-04-22T18:10:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:32 smithi115 bash[46934]: cluster 2022-04-22T18:10:31.035234+0000 mgr.y (mgr.34107) 1566 : cluster [DBG] pgmap v2214: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 155 KiB/s rd, 9.0 KiB/s wr, 196 op/s 2022-04-22T18:10:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:32 smithi115 bash[46934]: cluster 2022-04-22T18:10:32.101213+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e1161: 8 total, 8 up, 8 in 2022-04-22T18:10:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:33 smithi121 bash[36902]: audit 2022-04-22T18:10:33.026252+0000 mon.a (mon.0) 3315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:33 smithi121 bash[36902]: cluster 2022-04-22T18:10:33.102923+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1162: 8 total, 8 up, 8 in 2022-04-22T18:10:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:33 smithi115 bash[45495]: audit 2022-04-22T18:10:33.026252+0000 mon.a (mon.0) 3315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:33 smithi115 bash[45495]: cluster 2022-04-22T18:10:33.102923+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1162: 8 total, 8 up, 8 in 2022-04-22T18:10:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:33 smithi115 bash[46934]: audit 2022-04-22T18:10:33.026252+0000 mon.a (mon.0) 3315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:33 smithi115 bash[46934]: cluster 2022-04-22T18:10:33.102923+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1162: 8 total, 8 up, 8 in 2022-04-22T18:10:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:33.646Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:34 smithi121 bash[36902]: cluster 2022-04-22T18:10:33.036535+0000 mgr.y (mgr.34107) 1567 : cluster [DBG] pgmap v2217: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2022-04-22T18:10:34.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:34 smithi121 bash[36902]: cluster 2022-04-22T18:10:34.106778+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1163: 8 total, 8 up, 8 in 2022-04-22T18:10:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:34 smithi115 bash[45495]: cluster 2022-04-22T18:10:33.036535+0000 mgr.y (mgr.34107) 1567 : cluster [DBG] pgmap v2217: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2022-04-22T18:10:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:34 smithi115 bash[45495]: cluster 2022-04-22T18:10:34.106778+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1163: 8 total, 8 up, 8 in 2022-04-22T18:10:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:34 smithi115 bash[46934]: cluster 2022-04-22T18:10:33.036535+0000 mgr.y (mgr.34107) 1567 : cluster [DBG] pgmap v2217: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2022-04-22T18:10:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:34 smithi115 bash[46934]: cluster 2022-04-22T18:10:34.106778+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1163: 8 total, 8 up, 8 in 2022-04-22T18:10:36.154 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5974 ms) 2022-04-22T18:10:36.155 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2022-04-22T18:10:36.155 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:36 smithi121 bash[36902]: cluster 2022-04-22T18:10:35.037665+0000 mgr.y (mgr.34107) 1568 : cluster [DBG] pgmap v2220: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:36 smithi121 bash[36902]: cluster 2022-04-22T18:10:35.108359+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2022-04-22T18:10:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:36 smithi121 bash[36902]: audit 2022-04-22T18:10:35.638012+0000 mon.a (mon.0) 3319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:36 smithi121 bash[36902]: audit 2022-04-22T18:10:36.072892+0000 mon.a (mon.0) 3320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:36 smithi121 bash[36902]: cluster 2022-04-22T18:10:36.090260+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2022-04-22T18:10:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[45495]: cluster 2022-04-22T18:10:35.037665+0000 mgr.y (mgr.34107) 1568 : cluster [DBG] pgmap v2220: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[45495]: cluster 2022-04-22T18:10:35.108359+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[45495]: audit 2022-04-22T18:10:35.638012+0000 mon.a (mon.0) 3319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[45495]: audit 2022-04-22T18:10:36.072892+0000 mon.a (mon.0) 3320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[45495]: cluster 2022-04-22T18:10:36.090260+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[46934]: cluster 2022-04-22T18:10:35.037665+0000 mgr.y (mgr.34107) 1568 : cluster [DBG] pgmap v2220: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[46934]: cluster 2022-04-22T18:10:35.108359+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e1164: 8 total, 8 up, 8 in 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[46934]: audit 2022-04-22T18:10:35.638012+0000 mon.a (mon.0) 3319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[46934]: audit 2022-04-22T18:10:36.072892+0000 mon.a (mon.0) 3320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:36 smithi115 bash[46934]: cluster 2022-04-22T18:10:36.090260+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1165: 8 total, 8 up, 8 in 2022-04-22T18:10:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:38 smithi121 bash[36902]: cluster 2022-04-22T18:10:37.039132+0000 mgr.y (mgr.34107) 1569 : cluster [DBG] pgmap v2223: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2022-04-22T18:10:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:38 smithi121 bash[36902]: cluster 2022-04-22T18:10:37.118332+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2022-04-22T18:10:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:38 smithi121 bash[36902]: audit 2022-04-22T18:10:38.029324+0000 mon.a (mon.0) 3323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[45495]: cluster 2022-04-22T18:10:37.039132+0000 mgr.y (mgr.34107) 1569 : cluster [DBG] pgmap v2223: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2022-04-22T18:10:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[45495]: cluster 2022-04-22T18:10:37.118332+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2022-04-22T18:10:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[45495]: audit 2022-04-22T18:10:38.029324+0000 mon.a (mon.0) 3323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[46934]: cluster 2022-04-22T18:10:37.039132+0000 mgr.y (mgr.34107) 1569 : cluster [DBG] pgmap v2223: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2022-04-22T18:10:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[46934]: cluster 2022-04-22T18:10:37.118332+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1166: 8 total, 8 up, 8 in 2022-04-22T18:10:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:38 smithi115 bash[46934]: audit 2022-04-22T18:10:38.029324+0000 mon.a (mon.0) 3323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:39 smithi121 bash[36902]: cluster 2022-04-22T18:10:38.129077+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2022-04-22T18:10:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:39 smithi115 bash[45495]: cluster 2022-04-22T18:10:38.129077+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2022-04-22T18:10:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:39 smithi115 bash[46934]: cluster 2022-04-22T18:10:38.129077+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e1167: 8 total, 8 up, 8 in 2022-04-22T18:10:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:40 smithi121 bash[36902]: cluster 2022-04-22T18:10:39.040748+0000 mgr.y (mgr.34107) 1570 : cluster [DBG] pgmap v2226: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.5 KiB/s wr, 17 op/s 2022-04-22T18:10:40.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:40 smithi121 bash[36902]: cluster 2022-04-22T18:10:39.139395+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2022-04-22T18:10:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:40 smithi115 bash[45495]: cluster 2022-04-22T18:10:39.040748+0000 mgr.y (mgr.34107) 1570 : cluster [DBG] pgmap v2226: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.5 KiB/s wr, 17 op/s 2022-04-22T18:10:40.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:40 smithi115 bash[45495]: cluster 2022-04-22T18:10:39.139395+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2022-04-22T18:10:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:40 smithi115 bash[46934]: cluster 2022-04-22T18:10:39.040748+0000 mgr.y (mgr.34107) 1570 : cluster [DBG] pgmap v2226: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.5 KiB/s wr, 17 op/s 2022-04-22T18:10:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:40 smithi115 bash[46934]: cluster 2022-04-22T18:10:39.139395+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1168: 8 total, 8 up, 8 in 2022-04-22T18:10:41.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:41 smithi121 bash[36902]: cluster 2022-04-22T18:10:40.145600+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1169: 8 total, 8 up, 8 in 2022-04-22T18:10:41.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:41 smithi121 bash[36902]: audit 2022-04-22T18:10:40.641388+0000 mon.a (mon.0) 3327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:41.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:41 smithi121 bash[36902]: audit 2022-04-22T18:10:41.076166+0000 mon.a (mon.0) 3328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:41.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:41 smithi121 bash[36902]: cluster 2022-04-22T18:10:41.096075+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2022-04-22T18:10:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[45495]: cluster 2022-04-22T18:10:40.145600+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1169: 8 total, 8 up, 8 in 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[45495]: audit 2022-04-22T18:10:40.641388+0000 mon.a (mon.0) 3327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[45495]: audit 2022-04-22T18:10:41.076166+0000 mon.a (mon.0) 3328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[45495]: cluster 2022-04-22T18:10:41.096075+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[46934]: cluster 2022-04-22T18:10:40.145600+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1169: 8 total, 8 up, 8 in 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[46934]: audit 2022-04-22T18:10:40.641388+0000 mon.a (mon.0) 3327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[46934]: audit 2022-04-22T18:10:41.076166+0000 mon.a (mon.0) 3328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[46934]: cluster 2022-04-22T18:10:41.096075+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e1170: 8 total, 8 up, 8 in 2022-04-22T18:10:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:41.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:42.167 INFO:tasks.workunit.client.0.smithi115.stdout:10.00% done 2022-04-22T18:10:42.167 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.QuiesceWatchError (6013 ms) 2022-04-22T18:10:42.168 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2022-04-22T18:10:42.168 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:10:42.193 INFO:tasks.workunit.client.0.smithi115.stderr:test quiesce is not long enough to time out 2022-04-22T18:10:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:42 smithi121 bash[36902]: cluster 2022-04-22T18:10:41.042135+0000 mgr.y (mgr.34107) 1571 : cluster [DBG] pgmap v2229: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2022-04-22T18:10:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:42 smithi121 bash[36902]: cluster 2022-04-22T18:10:42.112011+0000 mon.a (mon.0) 3330 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2022-04-22T18:10:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:42 smithi115 bash[45495]: cluster 2022-04-22T18:10:41.042135+0000 mgr.y (mgr.34107) 1571 : cluster [DBG] pgmap v2229: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2022-04-22T18:10:42.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:42 smithi115 bash[45495]: cluster 2022-04-22T18:10:42.112011+0000 mon.a (mon.0) 3330 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2022-04-22T18:10:42.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:42 smithi115 bash[46934]: cluster 2022-04-22T18:10:41.042135+0000 mgr.y (mgr.34107) 1571 : cluster [DBG] pgmap v2229: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2022-04-22T18:10:42.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:42 smithi115 bash[46934]: cluster 2022-04-22T18:10:42.112011+0000 mon.a (mon.0) 3330 : cluster [DBG] osdmap e1171: 8 total, 8 up, 8 in 2022-04-22T18:10:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:43 smithi121 bash[36902]: audit 2022-04-22T18:10:43.032392+0000 mon.a (mon.0) 3331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:43 smithi115 bash[45495]: audit 2022-04-22T18:10:43.032392+0000 mon.a (mon.0) 3331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:43 smithi115 bash[46934]: audit 2022-04-22T18:10:43.032392+0000 mon.a (mon.0) 3331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:43.868 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:10:44.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:43.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:44.217 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (33m) 2m ago 47m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (47m) 3m ago 47m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (35m) 2m ago 48m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (35m) 3m ago 48m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (43m) 3m ago 52m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (43m) 2m ago 55m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:10:44.305 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (42m) 2m ago 55m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:10:44.306 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (41m) 3m ago 53m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:10:44.306 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (42m) 2m ago 53m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:10:44.306 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (34m) 2m ago 47m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:10:44.306 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (34m) 3m ago 47m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:10:44.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (40m) 2m ago 52m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (39m) 2m ago 51m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (39m) 2m ago 51m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (38m) 2m ago 50m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (38m) 3m ago 50m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (37m) 3m ago 50m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (36m) 3m ago 49m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (36m) 3m ago 49m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:10:44.307 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (34m) 3m ago 47m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:10:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:44 smithi121 bash[36902]: cluster 2022-04-22T18:10:43.043346+0000 mgr.y (mgr.34107) 1572 : cluster [DBG] pgmap v2232: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:10:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:44 smithi115 bash[46934]: cluster 2022-04-22T18:10:43.043346+0000 mgr.y (mgr.34107) 1572 : cluster [DBG] pgmap v2232: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:10:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:44 smithi115 bash[45495]: cluster 2022-04-22T18:10:43.043346+0000 mgr.y (mgr.34107) 1572 : cluster [DBG] pgmap v2232: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:10:44.625 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:10:44.625 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:44.626 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:10:44.627 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:10:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:45 smithi121 bash[36902]: audit 2022-04-22T18:10:43.849258+0000 mgr.y (mgr.34107) 1573 : audit [DBG] from='client.46696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:45 smithi121 bash[36902]: audit 2022-04-22T18:10:44.193581+0000 mgr.y (mgr.34107) 1574 : audit [DBG] from='client.36771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:45 smithi121 bash[36902]: audit 2022-04-22T18:10:44.620291+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.115:0/3324495000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[45495]: audit 2022-04-22T18:10:43.849258+0000 mgr.y (mgr.34107) 1573 : audit [DBG] from='client.46696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[45495]: audit 2022-04-22T18:10:44.193581+0000 mgr.y (mgr.34107) 1574 : audit [DBG] from='client.36771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[45495]: audit 2022-04-22T18:10:44.620291+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.115:0/3324495000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[46934]: audit 2022-04-22T18:10:43.849258+0000 mgr.y (mgr.34107) 1573 : audit [DBG] from='client.46696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[46934]: audit 2022-04-22T18:10:44.193581+0000 mgr.y (mgr.34107) 1574 : audit [DBG] from='client.36771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:10:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:45 smithi115 bash[46934]: audit 2022-04-22T18:10:44.620291+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.115:0/3324495000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:10:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:46 smithi121 bash[36902]: cluster 2022-04-22T18:10:45.045119+0000 mgr.y (mgr.34107) 1575 : cluster [DBG] pgmap v2233: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2022-04-22T18:10:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:46 smithi121 bash[36902]: audit 2022-04-22T18:10:45.644626+0000 mon.a (mon.0) 3332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:46 smithi121 bash[36902]: audit 2022-04-22T18:10:46.079666+0000 mon.a (mon.0) 3333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:46 smithi121 bash[36902]: cluster 2022-04-22T18:10:46.097195+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2022-04-22T18:10:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[45495]: cluster 2022-04-22T18:10:45.045119+0000 mgr.y (mgr.34107) 1575 : cluster [DBG] pgmap v2233: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2022-04-22T18:10:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[45495]: audit 2022-04-22T18:10:45.644626+0000 mon.a (mon.0) 3332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[45495]: audit 2022-04-22T18:10:46.079666+0000 mon.a (mon.0) 3333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[45495]: cluster 2022-04-22T18:10:46.097195+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2022-04-22T18:10:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[46934]: cluster 2022-04-22T18:10:45.045119+0000 mgr.y (mgr.34107) 1575 : cluster [DBG] pgmap v2233: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2022-04-22T18:10:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[46934]: audit 2022-04-22T18:10:45.644626+0000 mon.a (mon.0) 3332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[46934]: audit 2022-04-22T18:10:46.079666+0000 mon.a (mon.0) 3333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:46 smithi115 bash[46934]: cluster 2022-04-22T18:10:46.097195+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e1172: 8 total, 8 up, 8 in 2022-04-22T18:10:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:48 smithi121 bash[36902]: cluster 2022-04-22T18:10:47.046507+0000 mgr.y (mgr.34107) 1576 : cluster [DBG] pgmap v2235: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:48.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:48 smithi121 bash[36902]: audit 2022-04-22T18:10:48.035626+0000 mon.a (mon.0) 3335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:48 smithi115 bash[45495]: cluster 2022-04-22T18:10:47.046507+0000 mgr.y (mgr.34107) 1576 : cluster [DBG] pgmap v2235: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:48 smithi115 bash[45495]: audit 2022-04-22T18:10:48.035626+0000 mon.a (mon.0) 3335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:48 smithi115 bash[46934]: cluster 2022-04-22T18:10:47.046507+0000 mgr.y (mgr.34107) 1576 : cluster [DBG] pgmap v2235: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2022-04-22T18:10:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:48 smithi115 bash[46934]: audit 2022-04-22T18:10:48.035626+0000 mon.a (mon.0) 3335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:50 smithi121 bash[36902]: cluster 2022-04-22T18:10:49.048377+0000 mgr.y (mgr.34107) 1577 : cluster [DBG] pgmap v2236: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 900 B/s wr, 5 op/s 2022-04-22T18:10:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:50 smithi121 bash[36902]: audit 2022-04-22T18:10:49.623359+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:10:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:50 smithi121 bash[36902]: audit 2022-04-22T18:10:49.626279+0000 mon.a (mon.0) 3337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:10:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:50 smithi121 bash[36902]: audit 2022-04-22T18:10:49.627951+0000 mon.a (mon.0) 3338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:10:50.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:50 smithi121 bash[36902]: audit 2022-04-22T18:10:49.976006+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:10:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[45495]: cluster 2022-04-22T18:10:49.048377+0000 mgr.y (mgr.34107) 1577 : cluster [DBG] pgmap v2236: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 900 B/s wr, 5 op/s 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[45495]: audit 2022-04-22T18:10:49.623359+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[45495]: audit 2022-04-22T18:10:49.626279+0000 mon.a (mon.0) 3337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[45495]: audit 2022-04-22T18:10:49.627951+0000 mon.a (mon.0) 3338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[45495]: audit 2022-04-22T18:10:49.976006+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[46934]: cluster 2022-04-22T18:10:49.048377+0000 mgr.y (mgr.34107) 1577 : cluster [DBG] pgmap v2236: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 900 B/s wr, 5 op/s 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[46934]: audit 2022-04-22T18:10:49.623359+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:10:50.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[46934]: audit 2022-04-22T18:10:49.626279+0000 mon.a (mon.0) 3337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:10:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[46934]: audit 2022-04-22T18:10:49.627951+0000 mon.a (mon.0) 3338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:10:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:50 smithi115 bash[46934]: audit 2022-04-22T18:10:49.976006+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:10:51.115 INFO:tasks.workunit.client.0.smithi115.stderr:test quiesce is timed out 2022-04-22T18:10:51.117 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 0 2022-04-22T18:10:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:51 smithi121 bash[36902]: audit 2022-04-22T18:10:50.647787+0000 mon.a (mon.0) 3340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:51 smithi121 bash[36902]: audit 2022-04-22T18:10:51.083129+0000 mon.a (mon.0) 3341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:51 smithi121 bash[36902]: cluster 2022-04-22T18:10:51.099044+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2022-04-22T18:10:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[46934]: audit 2022-04-22T18:10:50.647787+0000 mon.a (mon.0) 3340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[46934]: audit 2022-04-22T18:10:51.083129+0000 mon.a (mon.0) 3341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[46934]: cluster 2022-04-22T18:10:51.099044+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2022-04-22T18:10:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[45495]: audit 2022-04-22T18:10:50.647787+0000 mon.a (mon.0) 3340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[45495]: audit 2022-04-22T18:10:51.083129+0000 mon.a (mon.0) 3341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[45495]: cluster 2022-04-22T18:10:51.099044+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1173: 8 total, 8 up, 8 in 2022-04-22T18:10:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:51.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:52.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 1 2022-04-22T18:10:52.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:52 smithi121 bash[36902]: cluster 2022-04-22T18:10:51.050257+0000 mgr.y (mgr.34107) 1578 : cluster [DBG] pgmap v2237: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 801 B/s wr, 4 op/s 2022-04-22T18:10:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:52 smithi115 bash[45495]: cluster 2022-04-22T18:10:51.050257+0000 mgr.y (mgr.34107) 1578 : cluster [DBG] pgmap v2237: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 801 B/s wr, 4 op/s 2022-04-22T18:10:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:52 smithi115 bash[46934]: cluster 2022-04-22T18:10:51.050257+0000 mgr.y (mgr.34107) 1578 : cluster [DBG] pgmap v2237: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 801 B/s wr, 4 op/s 2022-04-22T18:10:53.117 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 2 2022-04-22T18:10:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:53 smithi121 bash[36902]: audit 2022-04-22T18:10:53.038649+0000 mon.a (mon.0) 3343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:53 smithi115 bash[45495]: audit 2022-04-22T18:10:53.038649+0000 mon.a (mon.0) 3343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:53 smithi115 bash[46934]: audit 2022-04-22T18:10:53.038649+0000 mon.a (mon.0) 3343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:10:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:10:53.645Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:10:54.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 3 2022-04-22T18:10:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:54 smithi121 bash[36902]: cluster 2022-04-22T18:10:53.052029+0000 mgr.y (mgr.34107) 1579 : cluster [DBG] pgmap v2239: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:10:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:54 smithi115 bash[45495]: cluster 2022-04-22T18:10:53.052029+0000 mgr.y (mgr.34107) 1579 : cluster [DBG] pgmap v2239: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:10:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:54 smithi115 bash[46934]: cluster 2022-04-22T18:10:53.052029+0000 mgr.y (mgr.34107) 1579 : cluster [DBG] pgmap v2239: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2022-04-22T18:10:55.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 4 2022-04-22T18:10:56.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 5 2022-04-22T18:10:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:56 smithi121 bash[36902]: cluster 2022-04-22T18:10:55.054348+0000 mgr.y (mgr.34107) 1580 : cluster [DBG] pgmap v2240: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 228 B/s wr, 1 op/s 2022-04-22T18:10:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:56 smithi121 bash[36902]: audit 2022-04-22T18:10:55.650769+0000 mon.a (mon.0) 3344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:56 smithi121 bash[36902]: audit 2022-04-22T18:10:56.086343+0000 mon.a (mon.0) 3345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[45495]: cluster 2022-04-22T18:10:55.054348+0000 mgr.y (mgr.34107) 1580 : cluster [DBG] pgmap v2240: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 228 B/s wr, 1 op/s 2022-04-22T18:10:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[45495]: audit 2022-04-22T18:10:55.650769+0000 mon.a (mon.0) 3344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[45495]: audit 2022-04-22T18:10:56.086343+0000 mon.a (mon.0) 3345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[46934]: cluster 2022-04-22T18:10:55.054348+0000 mgr.y (mgr.34107) 1580 : cluster [DBG] pgmap v2240: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 228 B/s wr, 1 op/s 2022-04-22T18:10:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[46934]: audit 2022-04-22T18:10:55.650769+0000 mon.a (mon.0) 3344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:10:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:56 smithi115 bash[46934]: audit 2022-04-22T18:10:56.086343+0000 mon.a (mon.0) 3345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:10:57.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 6 2022-04-22T18:10:58.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 7 2022-04-22T18:10:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:58 smithi121 bash[36902]: cluster 2022-04-22T18:10:57.056621+0000 mgr.y (mgr.34107) 1581 : cluster [DBG] pgmap v2241: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:10:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:10:58 smithi121 bash[36902]: audit 2022-04-22T18:10:58.041845+0000 mon.a (mon.0) 3346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:58 smithi115 bash[45495]: cluster 2022-04-22T18:10:57.056621+0000 mgr.y (mgr.34107) 1581 : cluster [DBG] pgmap v2241: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:10:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:10:58 smithi115 bash[45495]: audit 2022-04-22T18:10:58.041845+0000 mon.a (mon.0) 3346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:58 smithi115 bash[46934]: cluster 2022-04-22T18:10:57.056621+0000 mgr.y (mgr.34107) 1581 : cluster [DBG] pgmap v2241: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:10:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:10:58 smithi115 bash[46934]: audit 2022-04-22T18:10:58.041845+0000 mon.a (mon.0) 3346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:10:59.118 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 8 2022-04-22T18:11:00.119 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 9 2022-04-22T18:11:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:00 smithi121 bash[36902]: cluster 2022-04-22T18:10:59.058633+0000 mgr.y (mgr.34107) 1582 : cluster [DBG] pgmap v2242: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:00 smithi115 bash[45495]: cluster 2022-04-22T18:10:59.058633+0000 mgr.y (mgr.34107) 1582 : cluster [DBG] pgmap v2242: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:00 smithi115 bash[46934]: cluster 2022-04-22T18:10:59.058633+0000 mgr.y (mgr.34107) 1582 : cluster [DBG] pgmap v2242: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:01.119 INFO:tasks.workunit.client.0.smithi115.stderr:waiting for timed out ... 10 2022-04-22T18:11:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:01 smithi121 bash[36902]: audit 2022-04-22T18:11:00.654057+0000 mon.a (mon.0) 3347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:01 smithi121 bash[36902]: audit 2022-04-22T18:11:01.089168+0000 mon.a (mon.0) 3348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:01 smithi115 bash[45495]: audit 2022-04-22T18:11:00.654057+0000 mon.a (mon.0) 3347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:01 smithi115 bash[45495]: audit 2022-04-22T18:11:01.089168+0000 mon.a (mon.0) 3348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:01 smithi115 bash[46934]: audit 2022-04-22T18:11:00.654057+0000 mon.a (mon.0) 3347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:01 smithi115 bash[46934]: audit 2022-04-22T18:11:01.089168+0000 mon.a (mon.0) 3348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:01.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:02 smithi121 bash[36902]: cluster 2022-04-22T18:11:01.060846+0000 mgr.y (mgr.34107) 1583 : cluster [DBG] pgmap v2243: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:02.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:02 smithi115 bash[45495]: cluster 2022-04-22T18:11:01.060846+0000 mgr.y (mgr.34107) 1583 : cluster [DBG] pgmap v2243: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:02.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:02 smithi115 bash[46934]: cluster 2022-04-22T18:11:01.060846+0000 mgr.y (mgr.34107) 1583 : cluster [DBG] pgmap v2243: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 102 B/s wr, 0 op/s 2022-04-22T18:11:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:03 smithi121 bash[36902]: audit 2022-04-22T18:11:03.044920+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:03.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:03 smithi115 bash[45495]: audit 2022-04-22T18:11:03.044920+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:03 smithi115 bash[46934]: audit 2022-04-22T18:11:03.044920+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:03.645Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:04 smithi121 bash[36902]: cluster 2022-04-22T18:11:03.062212+0000 mgr.y (mgr.34107) 1584 : cluster [DBG] pgmap v2244: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 427 B/s rd, 85 B/s wr, 0 op/s 2022-04-22T18:11:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:04 smithi115 bash[45495]: cluster 2022-04-22T18:11:03.062212+0000 mgr.y (mgr.34107) 1584 : cluster [DBG] pgmap v2244: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 427 B/s rd, 85 B/s wr, 0 op/s 2022-04-22T18:11:04.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:04 smithi115 bash[46934]: cluster 2022-04-22T18:11:03.062212+0000 mgr.y (mgr.34107) 1584 : cluster [DBG] pgmap v2244: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 427 B/s rd, 85 B/s wr, 0 op/s 2022-04-22T18:11:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:06 smithi121 bash[36902]: cluster 2022-04-22T18:11:05.063928+0000 mgr.y (mgr.34107) 1585 : cluster [DBG] pgmap v2245: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:06 smithi121 bash[36902]: audit 2022-04-22T18:11:05.657090+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:06 smithi121 bash[36902]: audit 2022-04-22T18:11:06.092611+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[45495]: cluster 2022-04-22T18:11:05.063928+0000 mgr.y (mgr.34107) 1585 : cluster [DBG] pgmap v2245: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[45495]: audit 2022-04-22T18:11:05.657090+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[45495]: audit 2022-04-22T18:11:06.092611+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[46934]: cluster 2022-04-22T18:11:05.063928+0000 mgr.y (mgr.34107) 1585 : cluster [DBG] pgmap v2245: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[46934]: audit 2022-04-22T18:11:05.657090+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:06 smithi115 bash[46934]: audit 2022-04-22T18:11:06.092611+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:08 smithi121 bash[36902]: cluster 2022-04-22T18:11:07.065913+0000 mgr.y (mgr.34107) 1586 : cluster [DBG] pgmap v2246: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:08 smithi121 bash[36902]: audit 2022-04-22T18:11:08.048124+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:08 smithi115 bash[45495]: cluster 2022-04-22T18:11:07.065913+0000 mgr.y (mgr.34107) 1586 : cluster [DBG] pgmap v2246: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:08 smithi115 bash[45495]: audit 2022-04-22T18:11:08.048124+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:08 smithi115 bash[46934]: cluster 2022-04-22T18:11:07.065913+0000 mgr.y (mgr.34107) 1586 : cluster [DBG] pgmap v2246: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:08 smithi115 bash[46934]: audit 2022-04-22T18:11:08.048124+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:10 smithi121 bash[36902]: cluster 2022-04-22T18:11:09.067800+0000 mgr.y (mgr.34107) 1587 : cluster [DBG] pgmap v2247: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:10 smithi115 bash[45495]: cluster 2022-04-22T18:11:09.067800+0000 mgr.y (mgr.34107) 1587 : cluster [DBG] pgmap v2247: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:10 smithi115 bash[46934]: cluster 2022-04-22T18:11:09.067800+0000 mgr.y (mgr.34107) 1587 : cluster [DBG] pgmap v2247: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:11 smithi115 bash[45495]: audit 2022-04-22T18:11:10.660467+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:11 smithi115 bash[45495]: audit 2022-04-22T18:11:11.096736+0000 mon.a (mon.0) 3354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:11 smithi115 bash[46934]: audit 2022-04-22T18:11:10.660467+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:11 smithi115 bash[46934]: audit 2022-04-22T18:11:11.096736+0000 mon.a (mon.0) 3354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:11 smithi121 bash[36902]: audit 2022-04-22T18:11:10.660467+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:11 smithi121 bash[36902]: audit 2022-04-22T18:11:11.096736+0000 mon.a (mon.0) 3354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:11.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:12.119 INFO:tasks.workunit.client.0.smithi115.stderr:test retry succeeds 2022-04-22T18:11:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:12 smithi115 bash[45495]: cluster 2022-04-22T18:11:11.069162+0000 mgr.y (mgr.34107) 1588 : cluster [DBG] pgmap v2248: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:12 smithi115 bash[46934]: cluster 2022-04-22T18:11:11.069162+0000 mgr.y (mgr.34107) 1588 : cluster [DBG] pgmap v2248: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:12 smithi121 bash[36902]: cluster 2022-04-22T18:11:11.069162+0000 mgr.y (mgr.34107) 1588 : cluster [DBG] pgmap v2248: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:13 smithi115 bash[45495]: cluster 2022-04-22T18:11:12.270887+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e1174: 8 total, 8 up, 8 in 2022-04-22T18:11:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:13 smithi115 bash[45495]: audit 2022-04-22T18:11:13.051154+0000 mon.a (mon.0) 3356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:13 smithi115 bash[46934]: cluster 2022-04-22T18:11:12.270887+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e1174: 8 total, 8 up, 8 in 2022-04-22T18:11:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:13 smithi115 bash[46934]: audit 2022-04-22T18:11:13.051154+0000 mon.a (mon.0) 3356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:13 smithi121 bash[36902]: cluster 2022-04-22T18:11:12.270887+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e1174: 8 total, 8 up, 8 in 2022-04-22T18:11:13.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:13 smithi121 bash[36902]: audit 2022-04-22T18:11:13.051154+0000 mon.a (mon.0) 3356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:13.646Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:14.355 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32189 ms) 2022-04-22T18:11:14.355 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.WriteZeroes 2022-04-22T18:11:14.355 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:11:14.420 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.WriteZeroes (64 ms) 2022-04-22T18:11:14.514 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2022-04-22T18:11:14.514 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:11:14.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (75 ms) 2022-04-22T18:11:14.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestLibRBD.ConcurentOperations 2022-04-22T18:11:14.515 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:11:14.515 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestLibRBD.ConcurentOperations (0 ms) 2022-04-22T18:11:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[45495]: cluster 2022-04-22T18:11:13.070504+0000 mgr.y (mgr.34107) 1589 : cluster [DBG] pgmap v2250: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[45495]: cluster 2022-04-22T18:11:13.288769+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[45495]: audit 2022-04-22T18:11:13.834960+0000 mon.a (mon.0) 3358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[45495]: audit 2022-04-22T18:11:13.842643+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[46934]: cluster 2022-04-22T18:11:13.070504+0000 mgr.y (mgr.34107) 1589 : cluster [DBG] pgmap v2250: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[46934]: cluster 2022-04-22T18:11:13.288769+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[46934]: audit 2022-04-22T18:11:13.834960+0000 mon.a (mon.0) 3358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:11:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:14 smithi115 bash[46934]: audit 2022-04-22T18:11:13.842643+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:11:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:14 smithi121 bash[36902]: cluster 2022-04-22T18:11:13.070504+0000 mgr.y (mgr.34107) 1589 : cluster [DBG] pgmap v2250: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:15.050 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:14 smithi121 bash[36902]: cluster 2022-04-22T18:11:13.288769+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e1175: 8 total, 8 up, 8 in 2022-04-22T18:11:15.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:14 smithi121 bash[36902]: audit 2022-04-22T18:11:13.834960+0000 mon.a (mon.0) 3358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:11:15.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:14 smithi121 bash[36902]: audit 2022-04-22T18:11:13.842643+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:11:15.051 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:11:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:11:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (34m) 3m ago 48m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (47m) 4m ago 47m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (36m) 3m ago 49m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (35m) 4m ago 49m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (43m) 4m ago 53m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (44m) 3m ago 55m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (43m) 3m ago 55m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:11:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (42m) 4m ago 53m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (42m) 3m ago 53m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (35m) 3m ago 48m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (35m) 4m ago 48m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (41m) 3m ago 52m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (40m) 3m ago 52m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:11:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (39m) 3m ago 52m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (39m) 3m ago 51m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (38m) 4m ago 51m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (38m) 4m ago 50m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (37m) 4m ago 50m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (36m) 4m ago 49m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:11:15.360 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (35m) 4m ago 48m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:11:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:15 smithi121 bash[36902]: cluster 2022-04-22T18:11:14.301750+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1176: 8 total, 8 up, 8 in 2022-04-22T18:11:15.770 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:11:15.772 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:11:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:15 smithi115 bash[45495]: cluster 2022-04-22T18:11:14.301750+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1176: 8 total, 8 up, 8 in 2022-04-22T18:11:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:15 smithi115 bash[46934]: cluster 2022-04-22T18:11:14.301750+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1176: 8 total, 8 up, 8 in 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: audit 2022-04-22T18:11:14.985445+0000 mgr.y (mgr.34107) 1590 : audit [DBG] from='client.46714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: cluster 2022-04-22T18:11:15.071798+0000 mgr.y (mgr.34107) 1591 : cluster [DBG] pgmap v2253: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: cluster 2022-04-22T18:11:15.310997+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: audit 2022-04-22T18:11:15.335106+0000 mgr.y (mgr.34107) 1592 : audit [DBG] from='client.46720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: audit 2022-04-22T18:11:15.663433+0000 mon.a (mon.0) 3362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: audit 2022-04-22T18:11:15.765658+0000 mon.a (mon.0) 3363 : audit [DBG] from='client.? 172.21.15.115:0/2834486157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:16 smithi121 bash[36902]: audit 2022-04-22T18:11:16.104483+0000 mon.a (mon.0) 3364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: audit 2022-04-22T18:11:14.985445+0000 mgr.y (mgr.34107) 1590 : audit [DBG] from='client.46714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: cluster 2022-04-22T18:11:15.071798+0000 mgr.y (mgr.34107) 1591 : cluster [DBG] pgmap v2253: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: cluster 2022-04-22T18:11:15.310997+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2022-04-22T18:11:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: audit 2022-04-22T18:11:15.335106+0000 mgr.y (mgr.34107) 1592 : audit [DBG] from='client.46720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: audit 2022-04-22T18:11:15.663433+0000 mon.a (mon.0) 3362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: audit 2022-04-22T18:11:15.765658+0000 mon.a (mon.0) 3363 : audit [DBG] from='client.? 172.21.15.115:0/2834486157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[45495]: audit 2022-04-22T18:11:16.104483+0000 mon.a (mon.0) 3364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: audit 2022-04-22T18:11:14.985445+0000 mgr.y (mgr.34107) 1590 : audit [DBG] from='client.46714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: cluster 2022-04-22T18:11:15.071798+0000 mgr.y (mgr.34107) 1591 : cluster [DBG] pgmap v2253: 329 pgs: 329 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: cluster 2022-04-22T18:11:15.310997+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e1177: 8 total, 8 up, 8 in 2022-04-22T18:11:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: audit 2022-04-22T18:11:15.335106+0000 mgr.y (mgr.34107) 1592 : audit [DBG] from='client.46720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: audit 2022-04-22T18:11:15.663433+0000 mon.a (mon.0) 3362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: audit 2022-04-22T18:11:15.765658+0000 mon.a (mon.0) 3363 : audit [DBG] from='client.? 172.21.15.115:0/2834486157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:16 smithi115 bash[46934]: audit 2022-04-22T18:11:16.104483+0000 mon.a (mon.0) 3364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:17 smithi121 bash[36902]: cluster 2022-04-22T18:11:16.317226+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2022-04-22T18:11:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:17 smithi115 bash[45495]: cluster 2022-04-22T18:11:16.317226+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2022-04-22T18:11:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:17 smithi115 bash[46934]: cluster 2022-04-22T18:11:16.317226+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e1178: 8 total, 8 up, 8 in 2022-04-22T18:11:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:18 smithi121 bash[36902]: cluster 2022-04-22T18:11:17.072669+0000 mgr.y (mgr.34107) 1593 : cluster [DBG] pgmap v2256: 265 pgs: 265 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2022-04-22T18:11:18.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:18 smithi121 bash[36902]: cluster 2022-04-22T18:11:17.329747+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e1179: 8 total, 8 up, 8 in 2022-04-22T18:11:18.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:18 smithi121 bash[36902]: audit 2022-04-22T18:11:18.054356+0000 mon.a (mon.0) 3367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:18.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[45495]: cluster 2022-04-22T18:11:17.072669+0000 mgr.y (mgr.34107) 1593 : cluster [DBG] pgmap v2256: 265 pgs: 265 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2022-04-22T18:11:18.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[45495]: cluster 2022-04-22T18:11:17.329747+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e1179: 8 total, 8 up, 8 in 2022-04-22T18:11:18.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[45495]: audit 2022-04-22T18:11:18.054356+0000 mon.a (mon.0) 3367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:18.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[46934]: cluster 2022-04-22T18:11:17.072669+0000 mgr.y (mgr.34107) 1593 : cluster [DBG] pgmap v2256: 265 pgs: 265 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2022-04-22T18:11:18.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[46934]: cluster 2022-04-22T18:11:17.329747+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e1179: 8 total, 8 up, 8 in 2022-04-22T18:11:18.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:18 smithi115 bash[46934]: audit 2022-04-22T18:11:18.054356+0000 mon.a (mon.0) 3367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:19 smithi121 bash[36902]: cluster 2022-04-22T18:11:18.341535+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1180: 8 total, 8 up, 8 in 2022-04-22T18:11:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:19 smithi115 bash[45495]: cluster 2022-04-22T18:11:18.341535+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1180: 8 total, 8 up, 8 in 2022-04-22T18:11:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:19 smithi115 bash[46934]: cluster 2022-04-22T18:11:18.341535+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1180: 8 total, 8 up, 8 in 2022-04-22T18:11:20.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:20 smithi121 bash[36902]: cluster 2022-04-22T18:11:19.073754+0000 mgr.y (mgr.34107) 1594 : cluster [DBG] pgmap v2259: 201 pgs: 201 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:20.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:20 smithi121 bash[36902]: cluster 2022-04-22T18:11:19.350897+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2022-04-22T18:11:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:20 smithi115 bash[45495]: cluster 2022-04-22T18:11:19.073754+0000 mgr.y (mgr.34107) 1594 : cluster [DBG] pgmap v2259: 201 pgs: 201 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:20 smithi115 bash[45495]: cluster 2022-04-22T18:11:19.350897+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2022-04-22T18:11:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:20 smithi115 bash[46934]: cluster 2022-04-22T18:11:19.073754+0000 mgr.y (mgr.34107) 1594 : cluster [DBG] pgmap v2259: 201 pgs: 201 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:20 smithi115 bash[46934]: cluster 2022-04-22T18:11:19.350897+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e1181: 8 total, 8 up, 8 in 2022-04-22T18:11:21.121 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 128 tests from TestLibRBD (192821 ms total) 2022-04-22T18:11:21.121 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:11:21.121 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from DiffIterateTest/0, where TypeParam = DiffIterateParams 2022-04-22T18:11:21.161 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterate 2022-04-22T18:11:22.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:21 smithi121 bash[36902]: cluster 2022-04-22T18:11:20.355848+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1182: 8 total, 8 up, 8 in 2022-04-22T18:11:22.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:21 smithi121 bash[36902]: audit 2022-04-22T18:11:20.666565+0000 mon.a (mon.0) 3371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:22.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:21 smithi121 bash[36902]: cluster 2022-04-22T18:11:21.110578+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2022-04-22T18:11:22.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:21 smithi121 bash[36902]: audit 2022-04-22T18:11:21.111482+0000 mon.a (mon.0) 3373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:21.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[45495]: cluster 2022-04-22T18:11:20.355848+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1182: 8 total, 8 up, 8 in 2022-04-22T18:11:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[45495]: audit 2022-04-22T18:11:20.666565+0000 mon.a (mon.0) 3371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[45495]: cluster 2022-04-22T18:11:21.110578+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2022-04-22T18:11:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[45495]: audit 2022-04-22T18:11:21.111482+0000 mon.a (mon.0) 3373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:22.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[46934]: cluster 2022-04-22T18:11:20.355848+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1182: 8 total, 8 up, 8 in 2022-04-22T18:11:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[46934]: audit 2022-04-22T18:11:20.666565+0000 mon.a (mon.0) 3371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[46934]: cluster 2022-04-22T18:11:21.110578+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e1183: 8 total, 8 up, 8 in 2022-04-22T18:11:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:21 smithi115 bash[46934]: audit 2022-04-22T18:11:21.111482+0000 mon.a (mon.0) 3373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:22 smithi121 bash[36902]: cluster 2022-04-22T18:11:21.074770+0000 mgr.y (mgr.34107) 1595 : cluster [DBG] pgmap v2262: 137 pgs: 137 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:22 smithi121 bash[36902]: cluster 2022-04-22T18:11:22.129315+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1184: 8 total, 8 up, 8 in 2022-04-22T18:11:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:22 smithi121 bash[36902]: audit 2022-04-22T18:11:22.132110+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:11:23.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[45495]: cluster 2022-04-22T18:11:21.074770+0000 mgr.y (mgr.34107) 1595 : cluster [DBG] pgmap v2262: 137 pgs: 137 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:23.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[45495]: cluster 2022-04-22T18:11:22.129315+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1184: 8 total, 8 up, 8 in 2022-04-22T18:11:23.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[45495]: audit 2022-04-22T18:11:22.132110+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:11:23.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[46934]: cluster 2022-04-22T18:11:21.074770+0000 mgr.y (mgr.34107) 1595 : cluster [DBG] pgmap v2262: 137 pgs: 137 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:23.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[46934]: cluster 2022-04-22T18:11:22.129315+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1184: 8 total, 8 up, 8 in 2022-04-22T18:11:23.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:22 smithi115 bash[46934]: audit 2022-04-22T18:11:22.132110+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:11:23.137 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:11:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:23 smithi121 bash[36902]: audit 2022-04-22T18:11:23.057623+0000 mon.a (mon.0) 3376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:23 smithi121 bash[36902]: cluster 2022-04-22T18:11:23.075627+0000 mgr.y (mgr.34107) 1596 : cluster [DBG] pgmap v2265: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:23 smithi121 bash[36902]: audit 2022-04-22T18:11:23.128833+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:11:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:23 smithi121 bash[36902]: cluster 2022-04-22T18:11:23.129057+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2022-04-22T18:11:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:23.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[45495]: audit 2022-04-22T18:11:23.057623+0000 mon.a (mon.0) 3376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[45495]: cluster 2022-04-22T18:11:23.075627+0000 mgr.y (mgr.34107) 1596 : cluster [DBG] pgmap v2265: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[45495]: audit 2022-04-22T18:11:23.128833+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:11:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[45495]: cluster 2022-04-22T18:11:23.129057+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2022-04-22T18:11:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[46934]: audit 2022-04-22T18:11:23.057623+0000 mon.a (mon.0) 3376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[46934]: cluster 2022-04-22T18:11:23.075627+0000 mgr.y (mgr.34107) 1596 : cluster [DBG] pgmap v2265: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[46934]: audit 2022-04-22T18:11:23.128833+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.115:0/1065566003' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-8","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:11:24.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:23 smithi115 bash[46934]: cluster 2022-04-22T18:11:23.129057+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1185: 8 total, 8 up, 8 in 2022-04-22T18:11:25.169 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [3270947~54781,4527565~54426,5603809~35039,5983545~60296,6423248~65512,8459541~19552,14586213~39344,15336445~6870,18471732~27178,20449996~90309] 2022-04-22T18:11:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:25 smithi121 bash[36902]: cluster 2022-04-22T18:11:24.130908+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1186: 8 total, 8 up, 8 in 2022-04-22T18:11:25.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:25 smithi115 bash[45495]: cluster 2022-04-22T18:11:24.130908+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1186: 8 total, 8 up, 8 in 2022-04-22T18:11:25.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:25 smithi115 bash[46934]: cluster 2022-04-22T18:11:24.130908+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1186: 8 total, 8 up, 8 in 2022-04-22T18:11:26.162 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [96518~41390,1127981~72829,6070827~48995,6151005~59489,8713162~95328,11476585~92055,11723742~71827,11813358~41462,12897199~53982,12986388~16955] 2022-04-22T18:11:26.170 INFO:tasks.workunit.client.0.smithi115.stdout: diff was [96518~41390,1127981~72829,6070827~48995,6151005~59489,8479093~3375727,12897199~53982,12986388~16955] 2022-04-22T18:11:26.172 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterate (5012 ms) 2022-04-22T18:11:26.173 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDiscard 2022-04-22T18:11:26.173 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:11:26.191 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~256 2022-04-22T18:11:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:26 smithi121 bash[36902]: cluster 2022-04-22T18:11:25.076430+0000 mgr.y (mgr.34107) 1597 : cluster [DBG] pgmap v2268: 137 pgs: 2 creating+peering, 14 unknown, 121 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:26 smithi121 bash[36902]: cluster 2022-04-22T18:11:25.143217+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2022-04-22T18:11:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:26 smithi121 bash[36902]: audit 2022-04-22T18:11:25.671267+0000 mon.a (mon.0) 3381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:26 smithi121 bash[36902]: audit 2022-04-22T18:11:26.112393+0000 mon.a (mon.0) 3382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[45495]: cluster 2022-04-22T18:11:25.076430+0000 mgr.y (mgr.34107) 1597 : cluster [DBG] pgmap v2268: 137 pgs: 2 creating+peering, 14 unknown, 121 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[45495]: cluster 2022-04-22T18:11:25.143217+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2022-04-22T18:11:26.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[45495]: audit 2022-04-22T18:11:25.671267+0000 mon.a (mon.0) 3381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:26.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[45495]: audit 2022-04-22T18:11:26.112393+0000 mon.a (mon.0) 3382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:26.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[46934]: cluster 2022-04-22T18:11:25.076430+0000 mgr.y (mgr.34107) 1597 : cluster [DBG] pgmap v2268: 137 pgs: 2 creating+peering, 14 unknown, 121 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:11:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[46934]: cluster 2022-04-22T18:11:25.143217+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e1187: 8 total, 8 up, 8 in 2022-04-22T18:11:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[46934]: audit 2022-04-22T18:11:25.671267+0000 mon.a (mon.0) 3381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:26 smithi115 bash[46934]: audit 2022-04-22T18:11:26.112393+0000 mon.a (mon.0) 3382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:27.170 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~256 2022-04-22T18:11:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:27 smithi121 bash[36902]: cluster 2022-04-22T18:11:26.145294+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2022-04-22T18:11:27.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:27 smithi115 bash[45495]: cluster 2022-04-22T18:11:26.145294+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2022-04-22T18:11:27.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:27 smithi115 bash[46934]: cluster 2022-04-22T18:11:26.145294+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1188: 8 total, 8 up, 8 in 2022-04-22T18:11:28.172 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~256 2022-04-22T18:11:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:28 smithi121 bash[36902]: cluster 2022-04-22T18:11:27.077529+0000 mgr.y (mgr.34107) 1598 : cluster [DBG] pgmap v2271: 137 pgs: 2 creating+peering, 135 active+clean; 7.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 29 KiB/s wr, 1 op/s 2022-04-22T18:11:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:28 smithi121 bash[36902]: cluster 2022-04-22T18:11:27.154146+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2022-04-22T18:11:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:28 smithi121 bash[36902]: audit 2022-04-22T18:11:28.060642+0000 mon.a (mon.0) 3385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[45495]: cluster 2022-04-22T18:11:27.077529+0000 mgr.y (mgr.34107) 1598 : cluster [DBG] pgmap v2271: 137 pgs: 2 creating+peering, 135 active+clean; 7.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 29 KiB/s wr, 1 op/s 2022-04-22T18:11:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[45495]: cluster 2022-04-22T18:11:27.154146+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2022-04-22T18:11:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[45495]: audit 2022-04-22T18:11:28.060642+0000 mon.a (mon.0) 3385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[46934]: cluster 2022-04-22T18:11:27.077529+0000 mgr.y (mgr.34107) 1598 : cluster [DBG] pgmap v2271: 137 pgs: 2 creating+peering, 135 active+clean; 7.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 29 KiB/s wr, 1 op/s 2022-04-22T18:11:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[46934]: cluster 2022-04-22T18:11:27.154146+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e1189: 8 total, 8 up, 8 in 2022-04-22T18:11:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:28 smithi115 bash[46934]: audit 2022-04-22T18:11:28.060642+0000 mon.a (mon.0) 3385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:29.185 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~256 2022-04-22T18:11:29.186 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateDiscard (3014 ms) 2022-04-22T18:11:29.186 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStress 2022-04-22T18:11:29.186 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:11:29.234 INFO:tasks.workunit.client.0.smithi115.stdout: i=0 exists [17899058~2112203,40386412~5021634,96020887~4716834,223277490~6462613,237548778~2648791,243987697~584250,258814963~2305310,313385671~6771406,348312696~1622459,371923159~2762070] wrote [17899058~2112203,40386412~5021634,96020887~4716834,223277490~6462613,237548778~2648791,243987697~584250,258814963~2305310,313385671~6771406,348312696~1622459,371923159~2762070] 2022-04-22T18:11:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:29 smithi121 bash[36902]: cluster 2022-04-22T18:11:28.158044+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2022-04-22T18:11:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:29 smithi115 bash[45495]: cluster 2022-04-22T18:11:28.158044+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2022-04-22T18:11:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:29 smithi115 bash[46934]: cluster 2022-04-22T18:11:28.158044+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1190: 8 total, 8 up, 8 in 2022-04-22T18:11:30.227 INFO:tasks.workunit.client.0.smithi115.stdout: i=1 exists [17899058~2112203,40386412~5021634,96020887~4716834,108197075~1053194,114591530~5246273,201127492~2532790,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,258814963~2305310,286656194~1285043,307796616~1366847,313385671~6771406,334157967~8042306,348312696~1622459,366024461~3666286,371923159~2762070] wrote [108197075~1053194,114591530~5246273,201127492~2532790,230775872~5166452,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,366024461~3666286] 2022-04-22T18:11:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:30 smithi121 bash[36902]: cluster 2022-04-22T18:11:29.078578+0000 mgr.y (mgr.34107) 1599 : cluster [DBG] pgmap v2274: 137 pgs: 137 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 231 KiB/s wr, 10 op/s 2022-04-22T18:11:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:30 smithi121 bash[36902]: cluster 2022-04-22T18:11:29.171343+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1191: 8 total, 8 up, 8 in 2022-04-22T18:11:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:30 smithi115 bash[45495]: cluster 2022-04-22T18:11:29.078578+0000 mgr.y (mgr.34107) 1599 : cluster [DBG] pgmap v2274: 137 pgs: 137 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 231 KiB/s wr, 10 op/s 2022-04-22T18:11:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:30 smithi115 bash[45495]: cluster 2022-04-22T18:11:29.171343+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1191: 8 total, 8 up, 8 in 2022-04-22T18:11:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:30 smithi115 bash[46934]: cluster 2022-04-22T18:11:29.078578+0000 mgr.y (mgr.34107) 1599 : cluster [DBG] pgmap v2274: 137 pgs: 137 active+clean; 16 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 231 KiB/s wr, 10 op/s 2022-04-22T18:11:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:30 smithi115 bash[46934]: cluster 2022-04-22T18:11:29.171343+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1191: 8 total, 8 up, 8 in 2022-04-22T18:11:31.249 INFO:tasks.workunit.client.0.smithi115.stdout: i=2 exists [4545896~8677095,17899058~2112203,40386412~5021634,64472393~5029107,85025446~8586200,96020887~4716834,108197075~1053194,110394263~3269370,114591530~5246273,201127492~2532790,203963897~1817490,207714256~37218,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,258814963~2305310,286656194~1285043,307796616~1366847,313385671~6771406,334157967~8042306,348312696~1622459,366024461~3666286,371923159~6688079] wrote [4545896~8677095,64472393~5029107,85025446~8586200,110394263~3269370,203963897~1817490,207714256~37218,373607831~5003407] 2022-04-22T18:11:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:31 smithi121 bash[36902]: cluster 2022-04-22T18:11:30.186265+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1192: 8 total, 8 up, 8 in 2022-04-22T18:11:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:31 smithi121 bash[36902]: audit 2022-04-22T18:11:30.674545+0000 mon.a (mon.0) 3389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:31 smithi121 bash[36902]: audit 2022-04-22T18:11:31.115198+0000 mon.a (mon.0) 3390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[45495]: cluster 2022-04-22T18:11:30.186265+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1192: 8 total, 8 up, 8 in 2022-04-22T18:11:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[45495]: audit 2022-04-22T18:11:30.674545+0000 mon.a (mon.0) 3389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[45495]: audit 2022-04-22T18:11:31.115198+0000 mon.a (mon.0) 3390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[46934]: cluster 2022-04-22T18:11:30.186265+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1192: 8 total, 8 up, 8 in 2022-04-22T18:11:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[46934]: audit 2022-04-22T18:11:30.674545+0000 mon.a (mon.0) 3389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[46934]: audit 2022-04-22T18:11:31.115198+0000 mon.a (mon.0) 3390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:31.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:32.268 INFO:tasks.workunit.client.0.smithi115.stdout: i=3 exists [358204~12864787,17899058~2759511,28320920~7655096,40386412~5021634,64472393~5029107,85025446~8586200,96020887~4716834,108197075~1053194,110394263~9443540,201127492~2532790,203876877~7726249,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,258814963~4080866,286656194~1285043,307796616~1366847,313385671~6771406,334157967~8042306,345801873~5672989,366024461~3666286,371923159~6688079] wrote [358204~4951814,18198685~2459884,28320920~7655096,113644391~5022109,203876877~7726249,259047789~3848040,345801873~5672989] 2022-04-22T18:11:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:32 smithi121 bash[36902]: cluster 2022-04-22T18:11:31.079623+0000 mgr.y (mgr.34107) 1600 : cluster [DBG] pgmap v2277: 137 pgs: 137 active+clean; 47 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.3 MiB/s wr, 15 op/s 2022-04-22T18:11:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:32 smithi121 bash[36902]: cluster 2022-04-22T18:11:31.196952+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1193: 8 total, 8 up, 8 in 2022-04-22T18:11:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:32 smithi115 bash[45495]: cluster 2022-04-22T18:11:31.079623+0000 mgr.y (mgr.34107) 1600 : cluster [DBG] pgmap v2277: 137 pgs: 137 active+clean; 47 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.3 MiB/s wr, 15 op/s 2022-04-22T18:11:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:32 smithi115 bash[45495]: cluster 2022-04-22T18:11:31.196952+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1193: 8 total, 8 up, 8 in 2022-04-22T18:11:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:32 smithi115 bash[46934]: cluster 2022-04-22T18:11:31.079623+0000 mgr.y (mgr.34107) 1600 : cluster [DBG] pgmap v2277: 137 pgs: 137 active+clean; 47 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.3 MiB/s wr, 15 op/s 2022-04-22T18:11:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:32 smithi115 bash[46934]: cluster 2022-04-22T18:11:31.196952+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1193: 8 total, 8 up, 8 in 2022-04-22T18:11:33.248 INFO:tasks.workunit.client.0.smithi115.stdout: i=4 exists [358204~12864787,17899058~2759511,28320920~7655096,38897810~6510236,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,96020887~4716834,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,201127492~10475634,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,246205123~585941,258814963~4080866,286656194~1285043,296496992~6738222,307796616~1366847,313385671~6771406,334157967~8042306,345801873~5672989,366024461~3666286,371923159~6688079,400957574~803290] wrote [38897810~4648802,48895278~616637,53996902~768604,138508106~6769628,153423329~4240055,167018155~6568931,201485895~2808311,246205123~585941,296496992~6738222,400957574~803290] 2022-04-22T18:11:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:33 smithi121 bash[36902]: cluster 2022-04-22T18:11:32.198630+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1194: 8 total, 8 up, 8 in 2022-04-22T18:11:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:33 smithi121 bash[36902]: audit 2022-04-22T18:11:33.063974+0000 mon.a (mon.0) 3393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:33 smithi115 bash[45495]: cluster 2022-04-22T18:11:32.198630+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1194: 8 total, 8 up, 8 in 2022-04-22T18:11:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:33 smithi115 bash[45495]: audit 2022-04-22T18:11:33.063974+0000 mon.a (mon.0) 3393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:33 smithi115 bash[46934]: cluster 2022-04-22T18:11:32.198630+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1194: 8 total, 8 up, 8 in 2022-04-22T18:11:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:33 smithi115 bash[46934]: audit 2022-04-22T18:11:33.063974+0000 mon.a (mon.0) 3393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:33.649Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:34.309 INFO:tasks.workunit.client.0.smithi115.stdout: i=5 exists [358204~12864787,17899058~2759511,26037455~19370591,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,96020887~4716834,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201127492~10475634,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,246205123~585941,258814963~6047597,267637720~6431302,286656194~1285043,296496992~6738222,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,371923159~6688079,400957574~803290] wrote [26037455~6429498,33517931~6305039,90117530~2000896,178500603~6261994,188617691~875329,259357208~5505352,267637720~6431302,313251498~7642355,344794546~6259927] 2022-04-22T18:11:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:34 smithi121 bash[36902]: cluster 2022-04-22T18:11:33.080698+0000 mgr.y (mgr.34107) 1601 : cluster [DBG] pgmap v2280: 137 pgs: 137 active+clean; 84 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 21 op/s 2022-04-22T18:11:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:34 smithi121 bash[36902]: cluster 2022-04-22T18:11:33.206672+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e1195: 8 total, 8 up, 8 in 2022-04-22T18:11:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:34 smithi115 bash[45495]: cluster 2022-04-22T18:11:33.080698+0000 mgr.y (mgr.34107) 1601 : cluster [DBG] pgmap v2280: 137 pgs: 137 active+clean; 84 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 21 op/s 2022-04-22T18:11:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:34 smithi115 bash[45495]: cluster 2022-04-22T18:11:33.206672+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e1195: 8 total, 8 up, 8 in 2022-04-22T18:11:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:34 smithi115 bash[46934]: cluster 2022-04-22T18:11:33.080698+0000 mgr.y (mgr.34107) 1601 : cluster [DBG] pgmap v2280: 137 pgs: 137 active+clean; 84 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 21 op/s 2022-04-22T18:11:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:34 smithi115 bash[46934]: cluster 2022-04-22T18:11:33.206672+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e1195: 8 total, 8 up, 8 in 2022-04-22T18:11:35.274 INFO:tasks.workunit.client.0.smithi115.stdout: i=6 exists [358204~12864787,17899058~2759511,26037455~20811111,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,96020887~4830093,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201127492~10475634,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,246205123~585941,258814963~6047597,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,371923159~6688079,400957574~803290] wrote [1338956~2896899,41311781~5536785,97579945~3271035,112352598~4921695,150002549~7133299,176312285~7243950,231897103~2828363,284997197~1999406,303020863~360114,304987623~322214] 2022-04-22T18:11:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:35 smithi121 bash[36902]: cluster 2022-04-22T18:11:34.228472+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1196: 8 total, 8 up, 8 in 2022-04-22T18:11:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:35 smithi121 bash[36902]: audit 2022-04-22T18:11:34.458241+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:35 smithi115 bash[45495]: cluster 2022-04-22T18:11:34.228472+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1196: 8 total, 8 up, 8 in 2022-04-22T18:11:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:35 smithi115 bash[45495]: audit 2022-04-22T18:11:34.458241+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:35 smithi115 bash[46934]: cluster 2022-04-22T18:11:34.228472+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1196: 8 total, 8 up, 8 in 2022-04-22T18:11:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:35 smithi115 bash[46934]: audit 2022-04-22T18:11:34.458241+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:36.172 INFO:tasks.workunit.client.0.smithi115.stdout: i=7 exists [358204~12864787,17899058~2759511,26037455~20811111,48895278~7075312,64472393~5029107,85025446~8586200,96020887~4830093,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201127492~10475634,223277490~6462613,230775872~5166452,237548778~2648791,243987697~584250,246205123~585941,249987954~5416159,258814963~6047597,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~6680316,363313257~866163,366024461~3666286,371923159~6688079,392565943~4758030,398340547~7179335] wrote [49317039~6653551,151838408~1554399,170889023~5779566,195704462~4510370,249987954~5416159,305945013~3184808,334977868~4808749,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:36 smithi121 bash[36902]: cluster 2022-04-22T18:11:35.081646+0000 mgr.y (mgr.34107) 1602 : cluster [DBG] pgmap v2283: 137 pgs: 137 active+clean; 199 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 27 op/s 2022-04-22T18:11:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:36 smithi121 bash[36902]: cluster 2022-04-22T18:11:35.232690+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e1197: 8 total, 8 up, 8 in 2022-04-22T18:11:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:36 smithi121 bash[36902]: audit 2022-04-22T18:11:35.677659+0000 mon.a (mon.0) 3398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:36 smithi121 bash[36902]: cluster 2022-04-22T18:11:36.112711+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2022-04-22T18:11:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:36 smithi121 bash[36902]: audit 2022-04-22T18:11:36.118790+0000 mon.a (mon.0) 3400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[45495]: cluster 2022-04-22T18:11:35.081646+0000 mgr.y (mgr.34107) 1602 : cluster [DBG] pgmap v2283: 137 pgs: 137 active+clean; 199 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 27 op/s 2022-04-22T18:11:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[45495]: cluster 2022-04-22T18:11:35.232690+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e1197: 8 total, 8 up, 8 in 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[45495]: audit 2022-04-22T18:11:35.677659+0000 mon.a (mon.0) 3398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[45495]: cluster 2022-04-22T18:11:36.112711+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[45495]: audit 2022-04-22T18:11:36.118790+0000 mon.a (mon.0) 3400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[46934]: cluster 2022-04-22T18:11:35.081646+0000 mgr.y (mgr.34107) 1602 : cluster [DBG] pgmap v2283: 137 pgs: 137 active+clean; 199 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 27 op/s 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[46934]: cluster 2022-04-22T18:11:35.232690+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e1197: 8 total, 8 up, 8 in 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[46934]: audit 2022-04-22T18:11:35.677659+0000 mon.a (mon.0) 3398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[46934]: cluster 2022-04-22T18:11:36.112711+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e1198: 8 total, 8 up, 8 in 2022-04-22T18:11:36.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:36 smithi115 bash[46934]: audit 2022-04-22T18:11:36.118790+0000 mon.a (mon.0) 3400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:37.161 INFO:tasks.workunit.client.0.smithi115.stdout: i=8 exists [358204~12864787,17899058~5322803,26037455~20811111,48895278~7075312,64472393~5029107,84696799~8914847,96020887~4830093,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,223277490~12664834,237548778~2648791,243987697~584250,246205123~585941,249987954~5416159,258814963~6047597,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,371923159~6688079,392565943~4758030,398340547~7179335] wrote [9813204~386511,18525521~4696340,66276849~2165525,84696799~4903774,103180346~4227001,122121191~142122,192776308~414349,228202958~6169148,346855254~6078964,357651108~274635] 2022-04-22T18:11:38.179 INFO:tasks.workunit.client.0.smithi115.stdout: i=9 exists [358204~12864787,17899058~5322803,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,96020887~4830093,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,223277490~12664834,237548778~2648791,243987697~584250,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,371923159~6688079,392565943~4758030,398340547~7179335] wrote [83486776~5630781,127473758~2717223,130291297~6389032,253852109~5582104,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680] 2022-04-22T18:11:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:38 smithi121 bash[36902]: cluster 2022-04-22T18:11:37.082358+0000 mgr.y (mgr.34107) 1603 : cluster [DBG] pgmap v2286: 137 pgs: 137 active+clean; 274 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:38 smithi121 bash[36902]: cluster 2022-04-22T18:11:37.125820+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2022-04-22T18:11:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:38 smithi121 bash[36902]: audit 2022-04-22T18:11:38.066839+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[45495]: cluster 2022-04-22T18:11:37.082358+0000 mgr.y (mgr.34107) 1603 : cluster [DBG] pgmap v2286: 137 pgs: 137 active+clean; 274 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[45495]: cluster 2022-04-22T18:11:37.125820+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2022-04-22T18:11:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[45495]: audit 2022-04-22T18:11:38.066839+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[46934]: cluster 2022-04-22T18:11:37.082358+0000 mgr.y (mgr.34107) 1603 : cluster [DBG] pgmap v2286: 137 pgs: 137 active+clean; 274 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[46934]: cluster 2022-04-22T18:11:37.125820+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e1199: 8 total, 8 up, 8 in 2022-04-22T18:11:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:38 smithi115 bash[46934]: audit 2022-04-22T18:11:38.066839+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:39.195 INFO:tasks.workunit.client.0.smithi115.stdout: i=10 exists [358204~12864787,17899058~7740391,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,96020887~4830093,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,223277490~12664834,237548778~2648791,243987697~584250,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~6377490,371923159~6688079,391705571~5618402,398340547~7179335] wrote [1168411~5790915,18904669~6734780,143455364~3634220,153231026~4721732,161603998~7772296,216742230~4059603,262818021~1653011,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:39 smithi121 bash[36902]: cluster 2022-04-22T18:11:38.133967+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1200: 8 total, 8 up, 8 in 2022-04-22T18:11:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:39 smithi115 bash[45495]: cluster 2022-04-22T18:11:38.133967+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1200: 8 total, 8 up, 8 in 2022-04-22T18:11:39.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:39 smithi115 bash[46934]: cluster 2022-04-22T18:11:38.133967+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1200: 8 total, 8 up, 8 in 2022-04-22T18:11:40.200 INFO:tasks.workunit.client.0.smithi115.stdout: i=11 exists [358204~12864787,17899058~7740391,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,96020887~4830093,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,223277490~12664834,237548778~2648791,243987697~584250,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,371923159~6688079,391705571~5618402,398340547~7179335] wrote [87052742~5424158,137917491~91474,167434704~1558571,208638260~425055,229497117~2058267,245411256~7036075,284555601~7142819,341936085~8126476,391779685~2212477] 2022-04-22T18:11:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:40 smithi121 bash[36902]: cluster 2022-04-22T18:11:39.083388+0000 mgr.y (mgr.34107) 1604 : cluster [DBG] pgmap v2289: 137 pgs: 137 active+clean; 400 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:40 smithi121 bash[36902]: cluster 2022-04-22T18:11:39.148728+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2022-04-22T18:11:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:40 smithi115 bash[45495]: cluster 2022-04-22T18:11:39.083388+0000 mgr.y (mgr.34107) 1604 : cluster [DBG] pgmap v2289: 137 pgs: 137 active+clean; 400 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:40 smithi115 bash[45495]: cluster 2022-04-22T18:11:39.148728+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2022-04-22T18:11:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:40 smithi115 bash[46934]: cluster 2022-04-22T18:11:39.083388+0000 mgr.y (mgr.34107) 1604 : cluster [DBG] pgmap v2289: 137 pgs: 137 active+clean; 400 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 31 op/s 2022-04-22T18:11:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:40 smithi115 bash[46934]: cluster 2022-04-22T18:11:39.148728+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1201: 8 total, 8 up, 8 in 2022-04-22T18:11:41.215 INFO:tasks.workunit.client.0.smithi115.stdout: i=12 exists [358204~12864787,17899058~7740391,26037455~20811111,48895278~12759917,64472393~5029107,83486776~10124870,96020887~4830093,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,223277490~12664834,237548778~2648791,243987697~584250,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,371923159~6688079,391705571~5618402,398340547~7179335] wrote [35608822~4750050,40893629~1490382,55300293~6354902,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,278521545~4205300,308833282~6613783] 2022-04-22T18:11:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:41 smithi121 bash[36902]: cluster 2022-04-22T18:11:40.154584+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2022-04-22T18:11:41.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:41 smithi121 bash[36902]: audit 2022-04-22T18:11:40.680916+0000 mon.a (mon.0) 3406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:41.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:41 smithi121 bash[36902]: audit 2022-04-22T18:11:41.122315+0000 mon.a (mon.0) 3407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[45495]: cluster 2022-04-22T18:11:40.154584+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2022-04-22T18:11:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[45495]: audit 2022-04-22T18:11:40.680916+0000 mon.a (mon.0) 3406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:41.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[45495]: audit 2022-04-22T18:11:41.122315+0000 mon.a (mon.0) 3407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[46934]: cluster 2022-04-22T18:11:40.154584+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e1202: 8 total, 8 up, 8 in 2022-04-22T18:11:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[46934]: audit 2022-04-22T18:11:40.680916+0000 mon.a (mon.0) 3406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[46934]: audit 2022-04-22T18:11:41.122315+0000 mon.a (mon.0) 3407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:41.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:42.236 INFO:tasks.workunit.client.0.smithi115.stdout: i=13 exists [358204~12864787,17899058~7740391,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~17364204,103180346~16657457,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,223277490~19273558,243987697~584250,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357651108~274635,363313257~6377490,371923159~6688079,385864957~173034,391705571~5618402,398340547~7179335] wrote [4849579~2102461,76803585~4111305,88716135~7490258,107119709~5777225,235175037~7376011,266448688~5166736,319260150~4273933,327866007~6887213,385864957~173034,401584875~2116936] 2022-04-22T18:11:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:42 smithi121 bash[36902]: cluster 2022-04-22T18:11:41.084269+0000 mgr.y (mgr.34107) 1605 : cluster [DBG] pgmap v2292: 137 pgs: 137 active+clean; 478 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 MiB/s wr, 21 op/s 2022-04-22T18:11:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:42 smithi121 bash[36902]: cluster 2022-04-22T18:11:41.164300+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e1203: 8 total, 8 up, 8 in 2022-04-22T18:11:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:42 smithi115 bash[45495]: cluster 2022-04-22T18:11:41.084269+0000 mgr.y (mgr.34107) 1605 : cluster [DBG] pgmap v2292: 137 pgs: 137 active+clean; 478 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 MiB/s wr, 21 op/s 2022-04-22T18:11:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:42 smithi115 bash[45495]: cluster 2022-04-22T18:11:41.164300+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e1203: 8 total, 8 up, 8 in 2022-04-22T18:11:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:42 smithi115 bash[46934]: cluster 2022-04-22T18:11:41.084269+0000 mgr.y (mgr.34107) 1605 : cluster [DBG] pgmap v2292: 137 pgs: 137 active+clean; 478 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41 MiB/s wr, 21 op/s 2022-04-22T18:11:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:42 smithi115 bash[46934]: cluster 2022-04-22T18:11:41.164300+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e1203: 8 total, 8 up, 8 in 2022-04-22T18:11:43.240 INFO:tasks.workunit.client.0.smithi115.stdout: i=14 exists [358204~12864787,17899058~7740391,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~17364204,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~20490731,243987697~584250,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,371923159~6688079,385864957~173034,391705571~5618402,398340547~7179335] wrote [41674998~2581801,56876848~4001363,91833690~4636567,127717081~4736765,150211175~5896403,203691436~7414116,222060317~3740269,274740903~2717085,357217629~5272584,401505364~2422986] 2022-04-22T18:11:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:43 smithi121 bash[36902]: cluster 2022-04-22T18:11:42.177771+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e1204: 8 total, 8 up, 8 in 2022-04-22T18:11:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:43 smithi121 bash[36902]: audit 2022-04-22T18:11:43.069745+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:43 smithi115 bash[45495]: cluster 2022-04-22T18:11:42.177771+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e1204: 8 total, 8 up, 8 in 2022-04-22T18:11:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:43 smithi115 bash[45495]: audit 2022-04-22T18:11:43.069745+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:43.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:43 smithi115 bash[46934]: cluster 2022-04-22T18:11:42.177771+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e1204: 8 total, 8 up, 8 in 2022-04-22T18:11:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:43 smithi115 bash[46934]: audit 2022-04-22T18:11:43.069745+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:44.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:43.649Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:44.237 INFO:tasks.workunit.client.0.smithi115.stdout: i=15 exists [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~20490731,243987697~584250,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,371923159~6688079,385864957~173034,391705571~5618402,398340547~7179335] wrote [13387288~5966260,76212653~2952306,94707726~7029722,132587116~3455386,230511465~5876747,248719919~4292401,265938071~1852873,296037928~4917453,316179792~2932051,402421085~2865295] 2022-04-22T18:11:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:44 smithi121 bash[36902]: cluster 2022-04-22T18:11:43.085268+0000 mgr.y (mgr.34107) 1606 : cluster [DBG] pgmap v2295: 137 pgs: 137 active+clean; 521 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 26 MiB/s wr, 23 op/s 2022-04-22T18:11:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:44 smithi121 bash[36902]: cluster 2022-04-22T18:11:43.185916+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2022-04-22T18:11:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:44 smithi115 bash[45495]: cluster 2022-04-22T18:11:43.085268+0000 mgr.y (mgr.34107) 1606 : cluster [DBG] pgmap v2295: 137 pgs: 137 active+clean; 521 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 26 MiB/s wr, 23 op/s 2022-04-22T18:11:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:44 smithi115 bash[45495]: cluster 2022-04-22T18:11:43.185916+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2022-04-22T18:11:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:44 smithi115 bash[46934]: cluster 2022-04-22T18:11:43.085268+0000 mgr.y (mgr.34107) 1606 : cluster [DBG] pgmap v2295: 137 pgs: 137 active+clean; 521 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 26 MiB/s wr, 23 op/s 2022-04-22T18:11:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:44 smithi115 bash[46934]: cluster 2022-04-22T18:11:43.185916+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e1205: 8 total, 8 up, 8 in 2022-04-22T18:11:45.249 INFO:tasks.workunit.client.0.smithi115.stdout: i=16 exists [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~22511630,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,371923159~6688079,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] wrote [35674797~2403115,171336895~678210,211455361~2113346,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,379125461~5171933,394912686~1855742] 2022-04-22T18:11:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:45 smithi121 bash[36902]: cluster 2022-04-22T18:11:44.186689+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e1206: 8 total, 8 up, 8 in 2022-04-22T18:11:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:45 smithi115 bash[45495]: cluster 2022-04-22T18:11:44.186689+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e1206: 8 total, 8 up, 8 in 2022-04-22T18:11:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:45 smithi115 bash[46934]: cluster 2022-04-22T18:11:44.186689+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e1206: 8 total, 8 up, 8 in 2022-04-22T18:11:46.150 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:11:46.264 INFO:tasks.workunit.client.0.smithi115.stdout: i=17 exists [358204~12864787,13387288~33461278,48895278~12759917,63641946~5859554,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22511630,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,371923159~6688079,379125461~11337910,391705571~13814311] wrote [8249801~2336322,21565899~6609152,63641946~4490210,88764093~5984638,151328086~10006739,195069328~4053541,223511505~6294523,383460721~7002650,395675089~5314611] 2022-04-22T18:11:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:46 smithi121 bash[36902]: cluster 2022-04-22T18:11:45.086208+0000 mgr.y (mgr.34107) 1607 : cluster [DBG] pgmap v2298: 137 pgs: 137 active+clean; 649 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 28 op/s 2022-04-22T18:11:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:46 smithi121 bash[36902]: cluster 2022-04-22T18:11:45.209833+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2022-04-22T18:11:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:46 smithi121 bash[36902]: audit 2022-04-22T18:11:45.684315+0000 mon.a (mon.0) 3414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:46 smithi121 bash[36902]: audit 2022-04-22T18:11:46.125648+0000 mon.a (mon.0) 3415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:46.523 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:11:46.523 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (34m) 3m ago 48m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:11:46.523 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (48m) 4m ago 48m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:11:46.524 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (36m) 3m ago 49m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:11:46.524 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (36m) 4m ago 49m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:11:46.524 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (44m) 4m ago 53m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:11:46.524 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (44m) 3m ago 56m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:11:46.524 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (43m) 3m ago 56m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (42m) 4m ago 54m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (43m) 3m ago 54m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (35m) 3m ago 48m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (35m) 4m ago 48m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (41m) 3m ago 53m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:11:46.525 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (40m) 3m ago 53m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (40m) 3m ago 52m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (39m) 3m ago 51m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (39m) 4m ago 51m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (38m) 4m ago 51m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (37m) 4m ago 50m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:11:46.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (37m) 4m ago 50m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:11:46.527 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (35m) 4m ago 48m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:11:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[46934]: cluster 2022-04-22T18:11:45.086208+0000 mgr.y (mgr.34107) 1607 : cluster [DBG] pgmap v2298: 137 pgs: 137 active+clean; 649 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 28 op/s 2022-04-22T18:11:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[46934]: cluster 2022-04-22T18:11:45.209833+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2022-04-22T18:11:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[46934]: audit 2022-04-22T18:11:45.684315+0000 mon.a (mon.0) 3414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:46.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[46934]: audit 2022-04-22T18:11:46.125648+0000 mon.a (mon.0) 3415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[45495]: cluster 2022-04-22T18:11:45.086208+0000 mgr.y (mgr.34107) 1607 : cluster [DBG] pgmap v2298: 137 pgs: 137 active+clean; 649 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 28 op/s 2022-04-22T18:11:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[45495]: cluster 2022-04-22T18:11:45.209833+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e1207: 8 total, 8 up, 8 in 2022-04-22T18:11:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[45495]: audit 2022-04-22T18:11:45.684315+0000 mon.a (mon.0) 3414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:46 smithi115 bash[45495]: audit 2022-04-22T18:11:46.125648+0000 mon.a (mon.0) 3415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:46.931 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:11:46.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:11:46.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:11:46.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:46.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:11:46.932 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:11:46.933 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:11:47.289 INFO:tasks.workunit.client.0.smithi115.stdout: i=18 exists [358204~12864787,13387288~33461278,48895278~12759917,63641946~17272944,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22511630,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,371923159~6688079,378967757~11495614,391705571~13814311] wrote [37541079~284740,58335599~1987181,69499952~7470379,209059456~1186466,222381315~4717533,247763752~2157508,307745298~7714921,330659402~5044208,378967757~5289639] 2022-04-22T18:11:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:47 smithi121 bash[36902]: audit 2022-04-22T18:11:46.129789+0000 mgr.y (mgr.34107) 1608 : audit [DBG] from='client.36819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:47 smithi121 bash[36902]: cluster 2022-04-22T18:11:46.210535+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1208: 8 total, 8 up, 8 in 2022-04-22T18:11:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:47 smithi121 bash[36902]: audit 2022-04-22T18:11:46.497980+0000 mgr.y (mgr.34107) 1609 : audit [DBG] from='client.36825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:47 smithi121 bash[36902]: audit 2022-04-22T18:11:46.921888+0000 mon.a (mon.0) 3417 : audit [DBG] from='client.? 172.21.15.115:0/3798739107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[45495]: audit 2022-04-22T18:11:46.129789+0000 mgr.y (mgr.34107) 1608 : audit [DBG] from='client.36819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[45495]: cluster 2022-04-22T18:11:46.210535+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1208: 8 total, 8 up, 8 in 2022-04-22T18:11:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[45495]: audit 2022-04-22T18:11:46.497980+0000 mgr.y (mgr.34107) 1609 : audit [DBG] from='client.36825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[45495]: audit 2022-04-22T18:11:46.921888+0000 mon.a (mon.0) 3417 : audit [DBG] from='client.? 172.21.15.115:0/3798739107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[46934]: audit 2022-04-22T18:11:46.129789+0000 mgr.y (mgr.34107) 1608 : audit [DBG] from='client.36819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[46934]: cluster 2022-04-22T18:11:46.210535+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1208: 8 total, 8 up, 8 in 2022-04-22T18:11:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[46934]: audit 2022-04-22T18:11:46.497980+0000 mgr.y (mgr.34107) 1609 : audit [DBG] from='client.36825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:11:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:47 smithi115 bash[46934]: audit 2022-04-22T18:11:46.921888+0000 mon.a (mon.0) 3417 : audit [DBG] from='client.? 172.21.15.115:0/3798739107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:11:48.301 INFO:tasks.workunit.client.0.smithi115.stdout: i=19 exists [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22511630,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,371923159~6688079,378967757~11495614,391705571~13814311] wrote [62715066~2807803,66594375~7588506,101731803~4113128,141133052~3130409,190798808~2972860,211468957~8070134,260709637~413875,273667814~6753815,350629815~1349947,361213496~6965178] 2022-04-22T18:11:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:48 smithi121 bash[36902]: cluster 2022-04-22T18:11:47.086883+0000 mgr.y (mgr.34107) 1610 : cluster [DBG] pgmap v2301: 137 pgs: 137 active+clean; 734 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 52 MiB/s wr, 34 op/s 2022-04-22T18:11:48.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:48 smithi121 bash[36902]: cluster 2022-04-22T18:11:47.232216+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e1209: 8 total, 8 up, 8 in 2022-04-22T18:11:48.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:48 smithi121 bash[36902]: audit 2022-04-22T18:11:48.072153+0000 mon.a (mon.0) 3419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:48.579 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[45495]: cluster 2022-04-22T18:11:47.086883+0000 mgr.y (mgr.34107) 1610 : cluster [DBG] pgmap v2301: 137 pgs: 137 active+clean; 734 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 52 MiB/s wr, 34 op/s 2022-04-22T18:11:48.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[45495]: cluster 2022-04-22T18:11:47.232216+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e1209: 8 total, 8 up, 8 in 2022-04-22T18:11:48.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[45495]: audit 2022-04-22T18:11:48.072153+0000 mon.a (mon.0) 3419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:48.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[46934]: cluster 2022-04-22T18:11:47.086883+0000 mgr.y (mgr.34107) 1610 : cluster [DBG] pgmap v2301: 137 pgs: 137 active+clean; 734 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 52 MiB/s wr, 34 op/s 2022-04-22T18:11:48.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[46934]: cluster 2022-04-22T18:11:47.232216+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e1209: 8 total, 8 up, 8 in 2022-04-22T18:11:48.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:48 smithi115 bash[46934]: audit 2022-04-22T18:11:48.072153+0000 mon.a (mon.0) 3419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:49.276 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 1 diff [108197075~1053194,114591530~5246273,201127492~2532790,230775872~5166452,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,366024461~3666286] 2022-04-22T18:11:49.276 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [108197075~1053194,114591530~5246273,201127492~2532790,230775872~5166452,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,366024461~3666286] 2022-04-22T18:11:49.288 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [104857600~4392669,113246208~6591595,197132288~6527994,230686720~5255604,285212672~2728565,306184192~2979271,315546770~2280443,331350016~10850257,364904448~4786299] 2022-04-22T18:11:49.288 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 2 diff [4545896~8677095,64472393~5029107,85025446~8586200,108197075~1053194,110394263~3269370,114591530~5246273,201127492~2532790,203963897~1817490,207714256~37218,230775872~5166452,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,366024461~3666286,373607831~5003407] 2022-04-22T18:11:49.288 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4545896~8677095,64472393~5029107,85025446~8586200,108197075~1053194,110394263~3269370,114591530~5246273,201127492~2532790,203963897~1817490,207714256~37218,230775872~5166452,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,366024461~3666286,373607831~5003407] 2022-04-22T18:11:49.297 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~9028687,62914560~6586940,83886080~9725566,104857600~14980203,197132288~10619186,230686720~5255604,285212672~2728565,306184192~2979271,315546770~2280443,331350016~10850257,364904448~4786299,373607831~5003407] 2022-04-22T18:11:49.297 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 3 diff [358204~12864787,18198685~2459884,28320920~7655096,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,201127492~2532790,203876877~7726249,230775872~5166452,259047789~3848040,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,345801873~5672989,366024461~3666286,373607831~5003407] 2022-04-22T18:11:49.297 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,28320920~7655096,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,201127492~2532790,203876877~7726249,230775872~5166452,259047789~3848040,286656194~1285043,307796616~1366847,315546770~2280443,334157967~8042306,345801873~5672989,366024461~3666286,373607831~5003407] 2022-04-22T18:11:49.307 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~10810192,62914560~6586940,83886080~9725566,104857600~14980203,197132288~14470838,230686720~5255604,259047789~3848040,285212672~2728565,306184192~2979271,315546770~2280443,331350016~10850257,343932928~7541934,364904448~4786299,373607831~5003407] 2022-04-22T18:11:49.307 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 4 diff [358204~12864787,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,201127492~10475634,230775872~5166452,246205123~585941,259047789~3848040,286656194~1285043,296496992~6738222,307796616~1366847,315546770~2280443,334157967~8042306,345801873~5672989,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.307 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,201127492~10475634,230775872~5166452,246205123~585941,259047789~3848040,286656194~1285043,296496992~6738222,307796616~1366847,315546770~2280443,334157967~8042306,345801873~5672989,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.314 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~10810192,38897810~4648802,46137344~3374571,50331648~4433858,62914560~6586940,83886080~9725566,104857600~14980203,138412032~6865702,150994944~6668440,163577856~10009230,197132288~14470838,230686720~5255604,244571947~2219117,259047789~3848040,285212672~2728565,293601280~9633934,306184192~2979271,315546770~2280443,331350016~10850257,343932928~7541934,364904448~4786299,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.314 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 5 diff [358204~12864787,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201127492~10475634,230775872~5166452,246205123~585941,259047789~5814771,267637720~6431302,286656194~1285043,296496992~6738222,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.314 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,108197075~1053194,110394263~9443540,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201127492~10475634,230775872~5166452,246205123~585941,259047789~5814771,267637720~6431302,286656194~1285043,296496992~6738222,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.322 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~18380788,46137344~3374571,50331648~4433858,62914560~6586940,83886080~9725566,104857600~14980203,138412032~6865702,150994944~6668440,163577856~10009230,176160768~13332252,197132288~14470838,230686720~5255604,244571947~2219117,259047789~15021233,285212672~2728565,293601280~9633934,306184192~2979271,313251498~7642355,331350016~10850257,343932928~7541934,364904448~4786299,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.323 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 6 diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,97579945~3271035,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201127492~10475634,230775872~5166452,246205123~585941,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.323 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,97579945~3271035,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201127492~10475634,230775872~5166452,246205123~585941,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,307796616~1366847,313251498~7642355,334157967~8042306,344794546~6680316,366024461~3666286,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.331 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~24346091,50331648~4433858,62914560~6586940,83886080~9725566,97579945~3271035,104857600~14980203,138412032~6865702,146800640~10862744,163577856~10009230,176160768~13332252,197132288~14470838,230686720~5255604,244571947~2219117,259047789~15021233,281018368~6922869,293601280~11708557,306184192~2979271,313251498~7642355,331350016~10850257,343932928~7541934,364904448~4786299,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.331 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 7 diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~7075312,64472393~5029107,85025446~8586200,97579945~3271035,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201127492~10475634,230775872~5166452,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~6680316,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.331 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~7075312,64472393~5029107,85025446~8586200,97579945~3271035,108197075~1053194,110394263~9443540,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201127492~10475634,230775872~5166452,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~6680316,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.340 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~30804766,62914560~6586940,83886080~9725566,97579945~3271035,104857600~14980203,138412032~6865702,146800640~10862744,163577856~25915164,192937984~18665142,230686720~5255604,244571947~2219117,247463936~7940177,259047789~15021233,281018368~6922869,293601280~15562183,313251498~7642355,331350016~10850257,343932928~7541934,360710144~3469276,364904448~4786299,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.340 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 8 diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,84696799~8914847,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,228202958~7739366,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.340 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,84696799~8914847,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,228202958~7739366,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~2944040,296496992~6883985,304987623~322214,305945013~3218450,313251498~7642355,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.347 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~5023176,25165824~30804766,62914560~6586940,83886080~9725566,97579945~22257858,121634816~628497,138412032~6865702,146800640~10862744,163577856~48025270,228202958~7739366,244571947~2219117,247463936~7940177,259047789~15021233,281018368~6922869,293601280~15562183,313251498~7642355,331350016~10850257,343932928~9001290,356515840~1409903,360710144~3469276,364904448~4786299,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.348 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 9 diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,228202958~7739366,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.348 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,228202958~7739366,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~866163,366024461~3666286,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.356 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~5023176,25165824~30804766,62914560~6586940,79691776~13919870,97579945~22257858,121634816~628497,125829120~10851209,138412032~6865702,146800640~10862744,163577856~48025270,228202958~7739366,244571947~2219117,247463936~26984598,281018368~6922869,289406976~19756487,313251498~8895273,327155712~15044561,343932928~9001290,356515840~1409903,360710144~3469276,364904448~4786299,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.356 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 10 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,228202958~7739366,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.356 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,228202958~7739366,246205123~585941,249987954~14874606,267637720~6810814,284997197~2944040,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~8042306,344794546~8139672,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.364 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~37771905,62914560~6586940,79691776~13919870,97579945~22257858,121634816~628497,125829120~10851209,138412032~19540726,159383552~52219574,213909504~6892329,228202958~7739366,244571947~2219117,247463936~26984598,281018368~6922869,289406976~19756487,313251498~8895273,327155712~15044561,343932928~9001290,356515840~1409903,360710144~8980603,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.364 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 11 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,228202958~7739366,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.364 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201127492~10475634,216742230~4059603,228202958~7739366,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5338639,313251498~8895273,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.373 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~37771905,62914560~6586940,79691776~13919870,97579945~22257858,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~7739366,244571947~29876587,281018368~28145095,313251498~8895273,327155712~25778506,356515840~1409903,360710144~8980603,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.373 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 12 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~7739366,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.373 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,108197075~1053194,110394263~9443540,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~7739366,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334157967~18776251,357651108~274635,363313257~6377490,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.382 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,79691776~13919870,97579945~22257858,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~7739366,244571947~29876587,276824064~45322707,327155712~25778506,356515840~1409903,360710144~8980603,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.382 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 13 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12719617,97579945~3271035,103180346~16657457,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~14348090,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357651108~274635,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.382 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12719617,97579945~3271035,103180346~16657457,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~14348090,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357651108~274635,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.390 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,75497472~20708921,97579945~22257858,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~14348090,244571947~29876587,276824064~46710019,327155712~25778506,356515840~1409903,360710144~8980603,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.390 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 14 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12983481,97579945~3271035,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.390 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12983481,97579945~3271035,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.399 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,75497472~20972785,97579945~22257858,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~14348090,244571947~78962136,327155712~25778506,356515840~13174907,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.400 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 15 diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.400 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.408 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~44340331,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~14348090,244571947~78962136,327155712~25778506,356515840~13174907,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.408 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 16 diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.408 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.416 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~44340331,121634816~628497,125829120~12179845,138412032~19540726,159383552~54185155,213909504~11891082,228202958~16074196,244571947~78962136,327155712~25778506,356515840~13174907,373607831~12430160,390070272~15449610] 2022-04-22T18:11:49.417 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 17 diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~5859554,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.417 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~5859554,76212653~4702237,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.425 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~44340331,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:49.425 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 18 diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~17272944,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.425 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~17272944,83486776~18250672,103180346~16657457,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~5272584,363313257~6377490,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.433 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~56923243,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:49.433 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 19 diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.433 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.441 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:49.578 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 2 diff [4545896~8677095,64472393~5029107,85025446~8586200,110394263~3269370,203963897~1817490,207714256~37218,373607831~5003407] 2022-04-22T18:11:49.578 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4545896~8677095,64472393~5029107,85025446~8586200,110394263~3269370,203963897~1817490,207714256~37218,373607831~5003407] 2022-04-22T18:11:49.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:49 smithi121 bash[36902]: cluster 2022-04-22T18:11:48.246229+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2022-04-22T18:11:49.587 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~9028687,62914560~6586940,83886080~9725566,109250269~4413364,203660282~4091192,373607831~5003407] 2022-04-22T18:11:49.587 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 3 diff [358204~12864787,18198685~2459884,28320920~7655096,64472393~5029107,85025446~8586200,110394263~8272237,203876877~7726249,259047789~3848040,345801873~5672989,373607831~5003407] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,28320920~7655096,64472393~5029107,85025446~8586200,110394263~8272237,203876877~7726249,259047789~3848040,345801873~5672989,373607831~5003407] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~10810192,62914560~6586940,83886080~9725566,109250269~9416231,203660282~7942844,259047789~3848040,343932928~7541934,373607831~5003407] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 4 diff [358204~12864787,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,110394263~8272237,138508106~6769628,153423329~4240055,167018155~6568931,201485895~10117231,246205123~585941,259047789~3848040,296496992~6738222,345801873~5672989,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,110394263~8272237,138508106~6769628,153423329~4240055,167018155~6568931,201485895~10117231,246205123~585941,259047789~3848040,296496992~6738222,345801873~5672989,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~10810192,38897810~4648802,46137344~3374571,50331648~4433858,62914560~6586940,83886080~9725566,109250269~9416231,138412032~6865702,150994944~6668440,163577856~10009230,201485895~10117231,244571947~2219117,259047789~3848040,293601280~9633934,343932928~7541934,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 5 diff [358204~12864787,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,110394263~8272237,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~10117231,246205123~585941,259047789~5814771,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6680316,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,110394263~8272237,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~10117231,246205123~585941,259047789~5814771,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6680316,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.588 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~18380788,46137344~3374571,50331648~4433858,62914560~6586940,83886080~9725566,109250269~9416231,138412032~6865702,150994944~6668440,163577856~10009230,176160768~13332252,201485895~10117231,244571947~2219117,259047789~15021233,293601280~9633934,313251498~7642355,343932928~7541934,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 6 diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,97579945~3271035,110394263~8272237,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~10117231,231897103~2828363,246205123~585941,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6680316,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,64472393~5029107,85025446~8586200,97579945~3271035,110394263~8272237,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~10117231,231897103~2828363,246205123~585941,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6680316,373607831~5003407,400957574~803290] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~24346091,50331648~4433858,62914560~6586940,83886080~9725566,97579945~3271035,109250269~9416231,138412032~6865702,146800640~10862744,163577856~10009230,176160768~13332252,201485895~10117231,231897103~2828363,244571947~2219117,259047789~15021233,281018368~5978235,293601280~11708557,313251498~7642355,343932928~7541934,373607831~5003407,398458880~3301984] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 7 diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~7075312,64472393~5029107,85025446~8586200,97579945~3271035,110394263~8272237,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~10117231,231897103~2828363,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6680316,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~2459884,26037455~20811111,48895278~7075312,64472393~5029107,85025446~8586200,97579945~3271035,110394263~8272237,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~10117231,231897103~2828363,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6680316,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~2459884,25165824~30804766,62914560~6586940,83886080~9725566,97579945~3271035,109250269~9416231,138412032~6865702,146800640~10862744,163577856~25915164,192937984~7276848,201485895~10117231,231897103~2828363,244571947~2219117,247463936~7940177,259047789~15021233,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,343932928~7541934,360710144~3469276,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 8 diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,84696799~8914847,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,84696799~8914847,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.589 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~5023176,25165824~30804766,62914560~6586940,83886080~9725566,97579945~9827402,109250269~9416231,121634816~628497,138412032~6865702,146800640~10862744,163577856~36636976,201485895~10117231,228202958~6522508,244571947~2219117,247463936~7940177,259047789~15021233,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,343932928~9001290,356515840~1409903,360710144~3469276,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 9 diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~5023176,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,373607831~5003407,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~5023176,25165824~30804766,62914560~6586940,79691776~13919870,97579945~9827402,109250269~9416231,121634816~628497,125829120~10851209,138412032~6865702,146800640~10862744,163577856~36636976,201485895~10117231,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,343932928~9001290,356515840~1409903,360710144~3469276,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 10 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~37771905,62914560~6586940,79691776~13919870,97579945~9827402,109250269~9416231,121634816~628497,125829120~10851209,138412032~19540726,159383552~40831280,201485895~10117231,213909504~6892329,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,343932928~9001290,356515840~1409903,360710144~6090564,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 11 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.590 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~7075312,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~37771905,62914560~6586940,79691776~13919870,97579945~9827402,109250269~9416231,121634816~628497,125829120~12179845,138412032~19540726,159383552~40831280,201485895~10117231,213909504~6892329,228202958~6522508,244571947~29876587,281018368~28111453,313251498~8895273,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 12 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,83486776~10124870,97579945~3271035,103180346~4227001,110394263~8272237,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,79691776~13919870,97579945~9827402,109250269~9416231,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~6522508,244571947~29876587,276824064~45322707,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,373607831~5003407,390070272~15449610] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 13 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12719617,97579945~3271035,103180346~15486154,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12719617,97579945~3271035,103180346~15486154,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,75497472~20708921,97579945~21086555,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~6522508,235175037~7376011,244571947~29876587,276824064~46710019,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 14 diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12983481,97579945~3271035,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.591 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,18198685~7440764,26037455~20811111,48895278~12759917,64472393~5029107,76803585~4111305,83486776~12983481,97579945~3271035,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:49 smithi115 bash[45495]: cluster 2022-04-22T18:11:48.246229+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2022-04-22T18:11:49.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:49 smithi115 bash[46934]: cluster 2022-04-22T18:11:48.246229+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1210: 8 total, 8 up, 8 in 2022-04-22T18:11:49.595 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~13222991,18198685~43456510,62914560~6586940,75497472~20972785,97579945~21086555,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~6522508,235175037~7376011,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 15 diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~43169028,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~14348090,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,373607831~5003407,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 16 diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~12252161,26037455~20811111,48895278~12759917,64472393~5029107,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~43169028,121634816~628497,125829120~12179845,138412032~19540726,159383552~54185155,213909504~11891082,228202958~16074196,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,373607831~12430160,390070272~15449610] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 17 diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~5859554,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.596 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~5859554,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~6586940,75497472~43169028,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,373607831~31912051] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 18 diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~17272944,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,63641946~17272944,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~61655195,62914560~55751940,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,373607831~31912051] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 19 diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 3 diff [358204~4951814,18198685~2459884,28320920~7655096,113644391~5022109,203876877~7726249,259047789~3848040,345801873~5672989] 2022-04-22T18:11:49.597 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,18198685~2459884,28320920~7655096,113644391~5022109,203876877~7726249,259047789~3848040,345801873~5672989] 2022-04-22T18:11:49.598 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,18198685~2459884,25165824~10810192,113644391~5022109,203876877~7726249,259047789~3848040,343932928~7541934] 2022-04-22T18:11:49.598 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 4 diff [358204~4951814,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,113644391~5022109,138508106~6769628,153423329~4240055,167018155~6568931,201485895~10117231,246205123~585941,259047789~3848040,296496992~6738222,345801873~5672989,400957574~803290] 2022-04-22T18:11:49.598 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,18198685~2459884,28320920~7655096,38897810~4648802,48895278~616637,53996902~768604,113644391~5022109,138508106~6769628,153423329~4240055,167018155~6568931,201485895~10117231,246205123~585941,259047789~3848040,296496992~6738222,345801873~5672989,400957574~803290] 2022-04-22T18:11:49.603 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,18198685~2459884,25165824~10810192,38897810~4648802,46137344~3374571,50331648~4433858,113644391~5022109,138412032~6865702,150994944~6668440,163577856~10009230,201485895~10117231,244571947~2219117,259047789~3848040,293601280~9633934,343932928~7541934,398458880~3301984] 2022-04-22T18:11:49.603 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 5 diff [358204~4951814,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,90117530~2000896,113644391~5022109,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~10117231,246205123~585941,259047789~5814771,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6680316,400957574~803290] 2022-04-22T18:11:49.603 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,18198685~2459884,26037455~17509157,48895278~616637,53996902~768604,90117530~2000896,113644391~5022109,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~10117231,246205123~585941,259047789~5814771,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6680316,400957574~803290] 2022-04-22T18:11:49.611 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,18198685~2459884,25165824~18380788,46137344~3374571,50331648~4433858,90117530~2000896,113644391~5022109,138412032~6865702,150994944~6668440,163577856~10009230,176160768~13332252,201485895~10117231,244571947~2219117,259047789~15021233,293601280~9633934,313251498~7642355,343932928~7541934,398458880~3301984] 2022-04-22T18:11:49.612 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 6 diff [358204~4951814,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,90117530~2000896,97579945~3271035,112352598~6313902,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~10117231,231897103~2828363,246205123~585941,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6680316,400957574~803290] 2022-04-22T18:11:49.612 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,18198685~2459884,26037455~20811111,48895278~616637,53996902~768604,90117530~2000896,97579945~3271035,112352598~6313902,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~10117231,231897103~2828363,246205123~585941,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6680316,400957574~803290] 2022-04-22T18:11:49.619 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,18198685~2459884,25165824~24346091,50331648~4433858,90117530~2000896,97579945~3271035,112352598~6313902,138412032~6865702,146800640~10862744,163577856~10009230,176160768~13332252,201485895~10117231,231897103~2828363,244571947~2219117,259047789~15021233,281018368~5978235,293601280~11708557,313251498~7642355,343932928~7541934,398458880~3301984] 2022-04-22T18:11:49.673 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 7 diff [358204~4951814,18198685~2459884,26037455~20811111,48895278~7075312,90117530~2000896,97579945~3271035,112352598~6313902,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~10117231,231897103~2828363,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6680316,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.673 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,18198685~2459884,26037455~20811111,48895278~7075312,90117530~2000896,97579945~3271035,112352598~6313902,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~10117231,231897103~2828363,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6680316,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,18198685~2459884,25165824~30804766,90117530~2000896,97579945~3271035,112352598~6313902,138412032~6865702,146800640~10862744,163577856~25915164,192937984~7276848,201485895~10117231,231897103~2828363,244571947~2219117,247463936~7940177,259047789~15021233,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,343932928~7541934,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 8 diff [358204~4951814,9813204~386511,18198685~5023176,26037455~20811111,48895278~7075312,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,9813204~386511,18198685~5023176,26037455~20811111,48895278~7075312,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~5416159,259047789~5814771,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,9813204~386511,18198685~5023176,25165824~30804766,66276849~2165525,84696799~4903774,90117530~2000896,97579945~9827402,112352598~6313902,121634816~628497,138412032~6865702,146800640~10862744,163577856~36636976,201485895~10117231,228202958~6522508,244571947~2219117,247463936~7940177,259047789~15021233,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,343932928~9001290,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 9 diff [358204~4951814,9813204~386511,18198685~5023176,26037455~20811111,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.676 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~4951814,9813204~386511,18198685~5023176,26037455~20811111,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~5310018,9813204~386511,18198685~5023176,25165824~30804766,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~6313902,121634816~628497,125829120~10851209,138412032~6865702,146800640~10862744,163577856~36636976,201485895~10117231,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,343932928~9001290,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 10 diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,18198685~37771905,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~6313902,121634816~628497,125829120~10851209,138412032~19540726,159383552~40831280,201485895~10117231,213909504~6892329,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,343932928~9001290,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 11 diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~7075312,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~7075312,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~10117231,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.677 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,18198685~37771905,66276849~2165525,79691776~12785124,97579945~9827402,112352598~6313902,121634816~628497,125829120~12179845,138412032~19540726,159383552~40831280,201485895~10117231,213909504~6892329,228202958~6522508,244571947~29876587,281018368~28111453,313251498~8895273,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.678 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 12 diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.679 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~6313902,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.679 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,18198685~43456510,66276849~2165525,79691776~12785124,97579945~9827402,112352598~6313902,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~6522508,244571947~29876587,276824064~45322707,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.679 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 13 diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~15486154,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.679 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~15486154,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.680 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,18198685~43456510,66276849~2165525,75497472~20708921,97579945~21086555,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~6892329,228202958~6522508,235175037~7376011,244571947~29876587,276824064~46710019,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.681 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 14 diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.681 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,18198685~7440764,26037455~20811111,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.685 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,18198685~43456510,66276849~2165525,75497472~20972785,97579945~21086555,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~6522508,235175037~7376011,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.685 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 15 diff [358204~6601122,9813204~386511,13387288~12252161,26037455~20811111,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.685 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,13387288~12252161,26037455~20811111,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15898664,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.693 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,13222991~48432204,66276849~2165525,75497472~43169028,121634816~628497,125829120~12179845,138412032~19540726,159383552~52219574,213909504~11891082,228202958~14348090,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.694 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 16 diff [358204~6601122,9813204~386511,13387288~12252161,26037455~20811111,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.694 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,9813204~386511,13387288~12252161,26037455~20811111,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~17864245,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.702 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,9813204~386511,13222991~48432204,66276849~2165525,75497472~43169028,121634816~628497,125829120~12179845,138412032~19540726,159383552~54185155,213909504~11891082,228202958~16074196,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~7426753,390070272~15449610] 2022-04-22T18:11:49.702 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 17 diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,63641946~4800428,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.702 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,63641946~4800428,76212653~4702237,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.710 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~48432204,63641946~4800428,75497472~43169028,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.710 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 18 diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,63641946~4800428,69499952~11414938,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.710 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,63641946~4800428,69499952~11414938,83486776~18250672,103180346~15486154,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~18499379,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.718 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~48432204,63641946~4800428,69499952~49166548,121634816~628497,125829120~12179845,138412032~75156675,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.719 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 19 diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.719 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.727 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:49.727 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 4 diff [38897810~4648802,48895278~616637,53996902~768604,138508106~6769628,153423329~4240055,167018155~6568931,201485895~2808311,246205123~585941,296496992~6738222,400957574~803290] 2022-04-22T18:11:49.727 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [38897810~4648802,48895278~616637,53996902~768604,138508106~6769628,153423329~4240055,167018155~6568931,201485895~2808311,246205123~585941,296496992~6738222,400957574~803290] 2022-04-22T18:11:49.736 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [38897810~4648802,46137344~3374571,50331648~4433858,138412032~6865702,150994944~6668440,163577856~10009230,201485895~2808311,244571947~2219117,293601280~9633934,398458880~3301984] 2022-04-22T18:11:49.736 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 5 diff [26037455~6429498,33517931~10028681,48895278~616637,53996902~768604,90117530~2000896,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~2808311,246205123~585941,259357208~5505352,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6259927,400957574~803290] 2022-04-22T18:11:49.736 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [26037455~6429498,33517931~10028681,48895278~616637,53996902~768604,90117530~2000896,138508106~6769628,153423329~4240055,167018155~6568931,178500603~6261994,188617691~875329,201485895~2808311,246205123~585941,259357208~5505352,267637720~6431302,296496992~6738222,313251498~7642355,344794546~6259927,400957574~803290] 2022-04-22T18:11:49.744 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [26037455~6429498,33517931~10028681,46137344~3374571,50331648~4433858,90117530~2000896,138412032~6865702,150994944~6668440,163577856~10009230,176160768~13332252,201485895~2808311,244571947~2219117,259357208~14711814,293601280~9633934,313251498~7642355,344794546~6259927,398458880~3301984] 2022-04-22T18:11:49.744 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 6 diff [1338956~2896899,26037455~6429498,33517931~13330635,48895278~616637,53996902~768604,90117530~2000896,97579945~3271035,112352598~4921695,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~2808311,231897103~2828363,246205123~585941,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6259927,400957574~803290] 2022-04-22T18:11:49.745 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,26037455~6429498,33517931~13330635,48895278~616637,53996902~768604,90117530~2000896,97579945~3271035,112352598~4921695,138508106~6769628,150002549~7660835,167018155~6568931,176312285~8450312,188617691~875329,201485895~2808311,231897103~2828363,246205123~585941,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,313251498~7642355,344794546~6259927,400957574~803290] 2022-04-22T18:11:49.752 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,26037455~6429498,33517931~15993984,50331648~4433858,90117530~2000896,97579945~3271035,112352598~4921695,138412032~6865702,146800640~10862744,163577856~10009230,176160768~13332252,201485895~2808311,231897103~2828363,244571947~2219117,259357208~14711814,281018368~5978235,293601280~11708557,313251498~7642355,344794546~6259927,398458880~3301984] 2022-04-22T18:11:49.752 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 7 diff [1338956~2896899,26037455~6429498,33517931~13330635,48895278~7075312,90117530~2000896,97579945~3271035,112352598~4921695,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~2808311,231897103~2828363,246205123~585941,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6259927,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.752 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,26037455~6429498,33517931~13330635,48895278~7075312,90117530~2000896,97579945~3271035,112352598~4921695,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,195704462~4510370,201485895~2808311,231897103~2828363,246205123~585941,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6259927,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.760 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,26037455~6429498,33517931~22452659,90117530~2000896,97579945~3271035,112352598~4921695,138412032~6865702,146800640~10862744,163577856~25915164,192937984~7276848,201485895~2808311,231897103~2828363,244571947~2219117,247463936~7940177,259357208~14711814,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,344794546~6259927,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.760 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 8 diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,228202958~6522508,246205123~585941,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.761 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,228202958~6522508,246205123~585941,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,296496992~6883985,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.769 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~22452659,66276849~2165525,84696799~4903774,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,138412032~6865702,146800640~10862744,163577856~36636976,201485895~2808311,228202958~6522508,244571947~2219117,247463936~7940177,259357208~14711814,281018368~5978235,293601280~15528541,313251498~7642355,334977868~4808749,344794546~8139672,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.769 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 9 diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.769 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,138508106~6769628,150002549~7660835,167018155~17744442,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.777 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~22452659,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,138412032~6865702,146800640~10862744,163577856~36636976,201485895~2808311,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,344794546~8139672,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.777 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 10 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.777 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,216742230~4059603,228202958~6522508,246205123~585941,249987954~14874606,267637720~6810814,284997197~1999406,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.786 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~22452659,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,138412032~19540726,159383552~40831280,201485895~2808311,213909504~6892329,228202958~6522508,244571947~2219117,247463936~26984598,281018368~5978235,289406976~19722845,313251498~8895273,327155712~5617531,334492946~5293671,344794546~8139672,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.786 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 11 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.786 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~7075312,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,138508106~8581478,150002549~7950209,161603998~23158599,188617691~875329,192776308~414349,195704462~4510370,201485895~2808311,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~11471758,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.795 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~22452659,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,138412032~19540726,159383552~40831280,201485895~2808311,208638260~425055,213909504~6892329,228202958~6522508,244571947~29876587,281018368~28111453,313251498~8895273,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.796 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 12 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~8589744,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.796 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~8589744,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~11471758,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.802 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~28137264,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,138412032~19540726,159383552~44910654,208638260~425055,213909504~6892329,228202958~6522508,244571947~29876587,276824064~45322707,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.802 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 13 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~8589744,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.802 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~8589744,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.810 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~28137264,66276849~2165525,75497472~20708921,97579945~19694348,121634816~628497,125829120~12179845,138412032~19540726,159383552~44910654,208638260~425055,213909504~6892329,228202958~6522508,235175037~7376011,244571947~29876587,276824064~46710019,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.810 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 14 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.810 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.818 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~28137264,66276849~2165525,75497472~20972785,97579945~19694348,121634816~628497,125829120~12179845,138412032~19540726,159383552~51722000,213909504~11891082,228202958~6522508,235175037~7376011,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.818 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 15 diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.818 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.826 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,26037455~6429498,33517931~28137264,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,138412032~19540726,159383552~51722000,213909504~11891082,228202958~14348090,244571947~78962136,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.826 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 16 diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.827 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~13330635,48895278~12759917,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~19444652,161603998~23158599,188617691~875329,192776308~414349,195704462~15401090,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.835 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,26037455~6429498,33517931~28137264,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,138412032~19540726,159383552~51722000,211455361~2113346,213909504~11891082,228202958~16074196,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~7426753,390070272~15449610] 2022-04-22T18:11:49.835 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 17 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~16036224,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.835 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~16036224,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.843 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~28137264,63641946~4800428,75497472~41776821,121634816~628497,125829120~12179845,138412032~72693520,211455361~2113346,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.843 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 18 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~16036224,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.843 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,192776308~414349,195069328~16036224,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.854 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~28137264,63641946~4800428,69499952~47774341,121634816~628497,125829120~12179845,138412032~72693520,211455361~2113346,213909504~30367650,244571947~78962136,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.854 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 19 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.854 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.859 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:49.859 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 5 diff [26037455~6429498,33517931~6305039,90117530~2000896,178500603~6261994,188617691~875329,259357208~5505352,267637720~6431302,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.860 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [26037455~6429498,33517931~6305039,90117530~2000896,178500603~6261994,188617691~875329,259357208~5505352,267637720~6431302,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.867 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [26037455~6429498,33517931~6305039,90117530~2000896,176160768~13332252,259357208~14711814,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.867 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 6 diff [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,90117530~2000896,97579945~3271035,112352598~4921695,150002549~7133299,176312285~8450312,188617691~875329,231897103~2828363,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.867 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,90117530~2000896,97579945~3271035,112352598~4921695,150002549~7133299,176312285~8450312,188617691~875329,231897103~2828363,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.875 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,90117530~2000896,97579945~3271035,112352598~4921695,146800640~10335208,176160768~13332252,231897103~2828363,259357208~14711814,281018368~5978235,303020863~2288974,313251498~7642355,344794546~6259927] 2022-04-22T18:11:49.875 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 7 diff [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,90117530~2000896,97579945~3271035,112352598~4921695,150002549~7133299,170889023~13873574,188617691~875329,195704462~4510370,231897103~2828363,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6259927,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.875 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,90117530~2000896,97579945~3271035,112352598~4921695,150002549~7133299,170889023~13873574,188617691~875329,195704462~4510370,231897103~2828363,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~6259927,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.882 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,90117530~2000896,97579945~3271035,112352598~4921695,146800640~10335208,170889023~18603997,192937984~7276848,231897103~2828363,247463936~7940177,259357208~14711814,281018368~5978235,303020863~6108958,313251498~7642355,334977868~4808749,344794546~6259927,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.883 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 8 diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,150002549~7133299,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,228202958~6522508,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.883 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,150002549~7133299,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,228202958~6522508,249987954~5416159,259357208~5505352,267637720~6431302,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,313251498~7642355,334977868~4808749,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.891 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,146800640~10335208,170889023~29325809,228202958~6522508,247463936~7940177,259357208~14711814,281018368~5978235,303020863~6108958,313251498~7642355,334977868~4808749,344794546~8139672,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.891 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 9 diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,150002549~7133299,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,228202958~6522508,249987954~14874606,267637720~6810814,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.891 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,150002549~7133299,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,228202958~6522508,249987954~14874606,267637720~6810814,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.900 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,146800640~10335208,170889023~29325809,228202958~6522508,247463936~26984598,281018368~5978235,289406976~8760300,303020863~6108958,313251498~8895273,327155712~5617531,334492946~5293671,344794546~8139672,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:49.900 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 10 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,150002549~7950209,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,216742230~4059603,228202958~6522508,249987954~14874606,267637720~6810814,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.900 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,90117530~2000896,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,150002549~7950209,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,216742230~4059603,228202958~6522508,249987954~14874606,267637720~6810814,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,344794546~8139672,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.909 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,79691776~9908797,90117530~2000896,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,143455364~14497394,159383552~9992742,170889023~29325809,213909504~6892329,228202958~6522508,247463936~26984598,281018368~5978235,289406976~8760300,303020863~6108958,313251498~8895273,327155712~5617531,334492946~5293671,344794546~8139672,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.909 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 11 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,150002549~7950209,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.909 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,150002549~7950209,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,284555601~7142819,291909219~6258057,303020863~360114,303824824~5304997,313251498~8895273,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.917 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6305039,41311781~5536785,49317039~6653551,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~29325809,208638260~425055,213909504~6892329,228202958~6522508,245411256~29037278,281018368~17148908,303020863~6108958,313251498~8895273,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.918 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 12 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.918 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,245411256~19451304,267637720~6810814,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~18321947,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.924 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~31633105,208638260~425055,213909504~6892329,228202958~6522508,245411256~29037278,276824064~21343212,303020863~19125908,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:49.925 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 13 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.925 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.932 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,75497472~20708921,97579945~19694348,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~31633105,208638260~425055,213909504~6892329,228202958~6522508,235175037~7376011,245411256~29037278,276824064~21343212,303020863~20513220,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.932 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 14 diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.932 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~19451304,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.940 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,75497472~20972785,97579945~19694348,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~31633105,203691436~7414116,213909504~11891082,228202958~6522508,235175037~7376011,245411256~52756020,303020863~20513220,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.940 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 15 diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.940 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~19451304,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303020863~360114,303824824~19709259,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.948 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~31633105,203691436~7414116,213909504~11891082,228202958~14348090,245411256~55544125,303020863~20513220,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:49.949 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 16 diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.949 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:49.956 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,26037455~6429498,33517931~6840941,40893629~5954937,49317039~12338156,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~31633105,203691436~7414116,211455361~2113346,213909504~11891082,228202958~16074196,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~10284868,378611238~7426753,390070272~15449610] 2022-04-22T18:11:49.957 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 17 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.957 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:49.968 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,75497472~41776821,121634816~628497,125829120~12179845,143455364~25920930,170889023~31633105,203691436~7414116,211455361~2113346,213909504~30367650,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.968 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 18 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.968 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~13873574,188617691~875329,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~19451304,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.979 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~12338156,63641946~4800428,69499952~47774341,121634816~628497,125829120~12179845,143455364~25920930,170889023~31633105,203691436~7414116,211455361~2113346,213909504~30367650,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:49.979 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 19 diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.980 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:49.987 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:49.988 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 6 diff [1338956~2896899,41311781~5536785,97579945~3271035,112352598~4921695,150002549~7133299,176312285~7243950,231897103~2828363,284997197~1999406,303020863~360114,304987623~322214] 2022-04-22T18:11:49.988 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,41311781~5536785,97579945~3271035,112352598~4921695,150002549~7133299,176312285~7243950,231897103~2828363,284997197~1999406,303020863~360114,304987623~322214] 2022-04-22T18:11:49.996 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,41311781~5536785,97579945~3271035,112352598~4921695,146800640~10335208,176312285~7243950,231897103~2828363,281018368~5978235,303020863~2288974] 2022-04-22T18:11:49.996 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 7 diff [1338956~2896899,41311781~5536785,49317039~6653551,97579945~3271035,112352598~4921695,150002549~7133299,170889023~12667212,195704462~4510370,231897103~2828363,249987954~5416159,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,334977868~4808749,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:49.996 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,41311781~5536785,49317039~6653551,97579945~3271035,112352598~4921695,150002549~7133299,170889023~12667212,195704462~4510370,231897103~2828363,249987954~5416159,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,334977868~4808749,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.008 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,41311781~5536785,49317039~6653551,97579945~3271035,112352598~4921695,146800640~10335208,170889023~12667212,192937984~7276848,231897103~2828363,247463936~7940177,281018368~5978235,303020863~6108958,334977868~4808749,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.008 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 8 diff [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,150002549~7133299,170889023~12667212,192776308~414349,195704462~4510370,228202958~6522508,249987954~5416159,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,334977868~4808749,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.008 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,150002549~7133299,170889023~12667212,192776308~414349,195704462~4510370,228202958~6522508,249987954~5416159,284997197~1999406,303020863~360114,304987623~322214,305945013~3184808,334977868~4808749,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.021 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,84696799~4903774,97579945~9827402,112352598~4921695,121634816~628497,146800640~10335208,170889023~12667212,189493020~10721812,228202958~6522508,247463936~7940177,281018368~5978235,303020863~6108958,334977868~4808749,346855254~6078964,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.022 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 9 diff [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,150002549~7133299,170889023~12667212,192776308~414349,195704462~4510370,228202958~6522508,249987954~9446259,268439604~6008930,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.022 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,150002549~7133299,170889023~12667212,192776308~414349,195704462~4510370,228202958~6522508,249987954~9446259,268439604~6008930,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.033 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1338956~2896899,9813204~386511,18525521~4696340,41311781~5536785,49317039~6653551,66276849~2165525,79691776~9908797,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,146800640~10335208,170889023~12667212,189493020~10721812,228202958~6522508,247463936~11970277,268439604~6008930,281018368~5978235,289406976~8760300,303020863~6108958,319043276~3103495,327155712~5617531,334492946~5293671,346855254~6078964,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.033 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 10 diff [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,150002549~7950209,161603998~7772296,170889023~12667212,192776308~414349,195704462~4510370,216742230~4059603,228202958~6522508,249987954~9446259,262818021~1653011,268439604~6008930,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.034 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,83486776~6113797,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,150002549~7950209,161603998~7772296,170889023~12667212,192776308~414349,195704462~4510370,216742230~4059603,228202958~6522508,249987954~9446259,262818021~1653011,268439604~6008930,284997197~1999406,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.042 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,79691776~9908797,97579945~9827402,112352598~4921695,121634816~628497,125829120~10851209,143455364~14497394,159383552~9992742,170889023~12667212,189493020~10721812,213909504~6892329,228202958~6522508,247463936~11970277,262818021~1653011,268439604~6008930,281018368~5978235,289406976~8760300,303020863~6108958,319043276~3103495,327155712~5617531,334492946~5293671,346855254~6078964,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.043 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 11 diff [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,150002549~7950209,161603998~7772296,170889023~12667212,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.043 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,150002549~7950209,161603998~7772296,170889023~12667212,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303020863~360114,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.053 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,41311781~5536785,49317039~6653551,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~10721812,208638260~425055,213909504~6892329,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,281018368~17148908,303020863~6108958,319043276~3103495,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.053 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 12 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.053 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,83486776~8990124,97579945~3271035,103180346~4227001,112352598~4921695,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.063 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,79691776~12785124,97579945~9827402,112352598~4921695,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~13029108,208638260~425055,213909504~6892329,228202958~6522508,245411256~14022957,262818021~1653011,268439604~6008930,276824064~21343212,303020863~12426202,319043276~3103495,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.064 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 13 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.064 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,97579945~3271035,103180346~14093947,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.076 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,75497472~20708921,97579945~19694348,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~13029108,208638260~425055,213909504~6892329,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,276824064~21343212,303020863~12426202,319043276~4490807,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.076 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 14 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.076 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,97579945~3271035,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303020863~360114,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.086 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,75497472~20972785,97579945~19694348,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~13029108,203691436~7414116,213909504~11891082,228202958~6522508,235175037~7376011,245411256~14022957,262818021~1653011,266448688~31718588,303020863~12426202,319043276~4490807,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.086 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 15 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~14022957,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303020863~360114,303824824~11622241,316179792~7354291,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.086 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~14022957,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303020863~360114,303824824~11622241,316179792~7354291,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.095 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~13029108,203691436~7414116,213909504~11891082,228202958~14348090,245411256~14022957,262818021~1653011,265938071~35017310,303020863~12426202,316179792~7354291,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.095 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 16 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.096 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~14497394,161603998~7772296,170889023~12667212,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.105 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~5954937,49317039~12338156,66276849~2165525,75497472~41776821,121634816~628497,125829120~12179845,143455364~14497394,159383552~9992742,170889023~12667212,189493020~13029108,203691436~7414116,211455361~2113346,213909504~11891082,228202958~16074196,245411256~14022957,262818021~1653011,265938071~35017310,301255591~133439,303020863~12426202,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~7426753,390070272~15449610] 2022-04-22T18:11:50.105 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 17 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~12667212,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:50.106 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~12667212,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:50.114 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,75497472~41776821,121634816~628497,125829120~12179845,143455364~25920930,170889023~12667212,189493020~13029108,203691436~7414116,211455361~2113346,213909504~30367650,245411256~14022957,262818021~1653011,265938071~35017310,301255591~133439,303020863~12426202,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:50.114 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 18 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~12667212,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.114 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~14093947,122121191~142122,127473758~9834087,137917491~91474,143455364~17879461,161603998~7772296,170889023~12667212,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.123 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~12338156,63641946~4800428,69499952~47774341,121634816~628497,125829120~12179845,143455364~25920930,170889023~12667212,189493020~13029108,203691436~7414116,211455361~2113346,213909504~30367650,245411256~14022957,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:50.123 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 19 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.123 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.133 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:50.133 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 7 diff [49317039~6653551,151838408~1554399,170889023~5779566,195704462~4510370,249987954~5416159,305945013~3184808,334977868~4808749,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.133 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [49317039~6653551,151838408~1554399,170889023~5779566,195704462~4510370,249987954~5416159,305945013~3184808,334977868~4808749,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.142 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [49317039~6653551,151838408~1554399,170889023~5779566,192937984~7276848,247463936~7940177,305309837~3819984,334977868~4808749,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.142 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 8 diff [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,84696799~4903774,103180346~4227001,122121191~142122,151838408~1554399,170889023~5779566,192776308~414349,195704462~4510370,228202958~6169148,249987954~5416159,305945013~3184808,334977868~4808749,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.143 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,84696799~4903774,103180346~4227001,122121191~142122,151838408~1554399,170889023~5779566,192776308~414349,195704462~4510370,228202958~6169148,249987954~5416159,305945013~3184808,334977868~4808749,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.151 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,84696799~4903774,100850980~6556367,121634816~628497,151838408~1554399,170889023~5779566,189493020~10721812,228202958~6169148,247463936~7940177,305309837~3819984,334977868~4808749,346855254~6078964,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.151 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 9 diff [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,151838408~1554399,170889023~5779566,192776308~414349,195704462~4510370,228202958~6169148,249987954~9446259,268439604~6008930,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.151 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,151838408~1554399,170889023~5779566,192776308~414349,195704462~4510370,228202958~6169148,249987954~9446259,268439604~6008930,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~866163,392565943~4758030,398340547~7179335] 2022-04-22T18:11:50.159 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [9813204~386511,18525521~4696340,49317039~6653551,66276849~2165525,79691776~9908797,100850980~6556367,121634816~628497,125829120~10851209,151838408~1554399,170889023~5779566,189493020~10721812,228202958~6169148,247463936~11970277,268439604~6008930,289406976~8760300,303824824~5304997,319043276~3103495,327155712~5617531,334492946~5293671,346855254~6078964,356515840~1409903,360710144~3469276,390070272~15449610] 2022-04-22T18:11:50.159 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 10 diff [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~4510370,216742230~4059603,228202958~6169148,249987954~9446259,262818021~1653011,268439604~6008930,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.160 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~4510370,216742230~4059603,228202958~6169148,249987954~9446259,262818021~1653011,268439604~6008930,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,346855254~6078964,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.168 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,79691776~9908797,100850980~6556367,121634816~628497,125829120~10851209,143455364~3634220,151838408~6114350,159383552~9992742,170889023~5779566,189493020~10721812,213909504~6892329,228202958~6169148,247463936~11970277,262818021~1653011,268439604~6008930,289406976~8760300,303824824~5304997,319043276~3103495,327155712~5617531,334492946~5293671,346855254~6078964,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.168 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 11 diff [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.168 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~4510370,208638260~425055,216742230~4059603,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~5304997,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.176 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,49317039~6653551,66276849~2165525,79691776~12785124,100850980~6556367,121634816~628497,125829120~12179845,143455364~3634220,151838408~6114350,159383552~9992742,170889023~5779566,189493020~10721812,208638260~425055,213909504~6892329,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,284555601~13611675,303824824~5304997,319043276~3103495,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.176 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 12 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.176 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~3103495,329868116~2905127,334492946~5293671,341936085~10998133,357651108~274635,363313257~3487451,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.185 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,79691776~12785124,100850980~6556367,121634816~628497,125829120~12179845,143455364~6678762,151838408~6114350,159383552~9992742,170889023~5779566,189493020~13029108,208638260~425055,213909504~6892329,228202958~6169148,245411256~14022957,262818021~1653011,268439604~6008930,276824064~5902781,284555601~13611675,303824824~11622241,319043276~3103495,327155712~5617531,334492946~5293671,341936085~10998133,356515840~1409903,360710144~6090564,390070272~15449610] 2022-04-22T18:11:50.186 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 13 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,103180346~9716588,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.186 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12719617,103180346~9716588,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,151838408~6114350,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,208638260~425055,216742230~4059603,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357651108~274635,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.193 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,49317039~12338156,66276849~2165525,75497472~20708921,100850980~12045954,121634816~628497,125829120~12179845,143455364~6678762,151838408~6114350,159383552~9992742,170889023~5779566,189493020~13029108,208638260~425055,213909504~6892329,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,276824064~5902781,284555601~13611675,303824824~11622241,319043276~4490807,327155712~12630905,341936085~10998133,356515840~1409903,360710144~6090564,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.194 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 14 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.194 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76803585~4111305,83486776~12983481,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~11622241,319043276~4490807,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.202 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,75497472~20972785,100850980~12045954,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,170889023~5779566,189493020~13029108,203691436~7414116,213909504~11891082,228202958~6169148,235175037~7376011,245411256~14022957,262818021~1653011,266448688~16278157,284555601~13611675,303824824~11622241,319043276~4490807,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.202 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 15 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~14022957,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~11622241,316179792~7354291,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.202 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~14022957,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~11622241,316179792~7354291,327866007~11920610,341936085~10998133,357217629~5272584,363313257~3487451,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.210 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,170889023~5779566,189493020~13029108,203691436~7414116,213909504~11891082,228202958~14348090,245411256~14022957,262818021~1653011,265938071~16788774,284555601~16399780,303824824~11622241,316179792~7354291,327155712~12630905,341936085~10998133,356515840~10284868,381681664~4356327,390070272~15449610] 2022-04-22T18:11:50.210 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 16 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.210 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,170889023~5779566,192776308~414349,195704462~6817666,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~5171933,385864957~173034,391705571~5618402,398340547~7179335] 2022-04-22T18:11:50.218 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~3363170,49317039~12338156,66276849~2165525,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,170889023~5779566,189493020~13029108,203691436~7414116,211455361~2113346,213909504~11891082,228202958~16074196,245411256~14022957,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11622241,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~7426753,390070272~15449610] 2022-04-22T18:11:50.218 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 17 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,170889023~5779566,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:50.219 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,170889023~5779566,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11622241,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,379125461~11337910,391705571~13814311] 2022-04-22T18:11:50.227 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~2113346,213909504~30367650,245411256~14022957,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11622241,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:50.227 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 18 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,170889023~5779566,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.227 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,69499952~11414938,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,170889023~5779566,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~14022957,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363313257~3487451,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.235 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~12338156,63641946~4800428,69499952~43396982,121634816~628497,125829120~12179845,143455364~6678762,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~2113346,213909504~30367650,245411256~14022957,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~10284868,378611238~26908644] 2022-04-22T18:11:50.235 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 19 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.235 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:50.245 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:50.245 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 8 diff [9813204~386511,18525521~4696340,66276849~2165525,84696799~4903774,103180346~4227001,122121191~142122,192776308~414349,228202958~6169148,346855254~6078964,357651108~274635] 2022-04-22T18:11:50.245 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [9813204~386511,18525521~4696340,66276849~2165525,84696799~4903774,103180346~4227001,122121191~142122,192776308~414349,228202958~6169148,346855254~6078964,357651108~274635] 2022-04-22T18:11:50.255 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [9813204~386511,18525521~4696340,66276849~2165525,84696799~4903774,100850980~6556367,121634816~628497,189493020~3697637,228202958~6169148,346855254~6078964,356515840~1409903] 2022-04-22T18:11:50.255 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 9 diff [9813204~386511,18525521~4696340,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,192776308~414349,228202958~6169148,253852109~5582104,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,346855254~6078964,357651108~274635] 2022-04-22T18:11:50.255 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [9813204~386511,18525521~4696340,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,192776308~414349,228202958~6169148,253852109~5582104,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,346855254~6078964,357651108~274635] 2022-04-22T18:11:50.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[46934]: cluster 2022-04-22T18:11:49.087869+0000 mgr.y (mgr.34107) 1611 : cluster [DBG] pgmap v2304: 137 pgs: 137 active+clean; 819 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 43 MiB/s wr, 32 op/s 2022-04-22T18:11:50.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[46934]: cluster 2022-04-22T18:11:49.260620+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2022-04-22T18:11:50.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[46934]: audit 2022-04-22T18:11:49.982350+0000 mon.a (mon.0) 3422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:11:50.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[46934]: audit 2022-04-22T18:11:49.984381+0000 mon.a (mon.0) 3423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:11:50.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[46934]: audit 2022-04-22T18:11:49.985954+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:11:50.265 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [9813204~386511,18525521~4696340,66276849~2165525,79691776~9908797,100850980~6556367,121634816~628497,125829120~10851209,189493020~3697637,228202958~6169148,253852109~5582104,268439604~6008930,289406976~8760300,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680,346855254~6078964,356515840~1409903] 2022-04-22T18:11:50.265 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 10 diff [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,153231026~4721732,161603998~7772296,192776308~414349,216742230~4059603,228202958~6169148,253852109~5582104,262818021~1653011,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,346855254~6078964,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.265 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,83486776~6113797,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,143455364~3634220,153231026~4721732,161603998~7772296,192776308~414349,216742230~4059603,228202958~6169148,253852109~5582104,262818021~1653011,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,346855254~6078964,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.272 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,79691776~9908797,100850980~6556367,121634816~628497,125829120~10851209,143455364~3634220,153231026~4721732,159383552~9992742,189493020~3697637,213909504~6892329,228202958~6169148,253852109~5582104,262818021~1653011,268439604~6008930,289406976~8760300,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,346855254~6078964,356515840~1409903,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.272 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 11 diff [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,192776308~414349,208638260~425055,216742230~4059603,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.273 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,192776308~414349,208638260~425055,216742230~4059603,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.281 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,66276849~2165525,79691776~12785124,100850980~6556367,121634816~628497,125829120~12179845,143455364~3634220,153231026~4721732,159383552~9992742,189493020~3697637,208638260~425055,213909504~6892329,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~13611675,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,341936085~10998133,356515840~1409903,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.281 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 12 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,192776308~414349,195989592~6532536,208638260~425055,216742230~4059603,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.281 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,83486776~8990124,103180346~4227001,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,192776308~414349,195989592~6532536,208638260~425055,216742230~4059603,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.290 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,79691776~12785124,100850980~6556367,121634816~628497,125829120~12179845,143455364~6678762,153231026~4721732,159383552~9992742,189493020~3697637,195989592~6532536,208638260~425055,213909504~6892329,228202958~6169148,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,276824064~5902781,284555601~13611675,303824824~3183847,308833282~6613783,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,341936085~10998133,356515840~1409903,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.290 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 13 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,76803585~4111305,83486776~12719617,103180346~9716588,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,192776308~414349,195989592~6532536,208638260~425055,216742230~4059603,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.291 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,76803585~4111305,83486776~12719617,103180346~9716588,122121191~142122,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,192776308~414349,195989592~6532536,208638260~425055,216742230~4059603,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~10998133,357651108~274635,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.302 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~1490382,55300293~6354902,66276849~2165525,75497472~20708921,100850980~12045954,121634816~628497,125829120~12179845,143455364~6678762,153231026~4721732,159383552~9992742,189493020~3697637,195989592~6532536,208638260~425055,213909504~6892329,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,276824064~5902781,284555601~13611675,303824824~3183847,308833282~6613783,319043276~4490807,327155712~8030914,336561845~688056,341936085~10998133,356515840~1409903,363341370~3459338,381681664~4356327,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.302 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 14 diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76803585~4111305,83486776~12983481,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,192776308~414349,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~10998133,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.302 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76803585~4111305,83486776~12983481,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,192776308~414349,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~10998133,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.315 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,18525521~7113928,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,75497472~20972785,100850980~12045954,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,189493020~3697637,195989592~6532536,203691436~7414116,213909504~11891082,228202958~6169148,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~16278157,284555601~13611675,303824824~3183847,308833282~6613783,319043276~4490807,327155712~8030914,336561845~688056,341936085~10998133,356515840~5974373,363341370~3459338,381681664~4356327,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.315 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 15 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,192776308~414349,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~7601064,253852109~5582104,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,336561845~688056,341936085~10998133,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.315 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,192776308~414349,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,228202958~14348090,245411256~7601064,253852109~5582104,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,336561845~688056,341936085~10998133,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.321 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,189493020~3697637,195989592~6532536,203691436~7414116,213909504~11891082,228202958~14348090,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,336561845~688056,341936085~10998133,356515840~5974373,363341370~3459338,381681664~4356327,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.321 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 16 diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,192776308~414349,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~10998133,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.321 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,9813204~386511,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,192776308~414349,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,228202958~16074196,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~10998133,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.330 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,9813204~386511,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,66276849~2165525,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,171336895~678210,189493020~3697637,195989592~6532536,203691436~7414116,211455361~2113346,213909504~11891082,228202958~16074196,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,335245198~6519073,341936085~10998133,356515840~5974373,363341370~3459338,378611238~7426753,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.330 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 17 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~10998133,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.330 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,76212653~4702237,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~10998133,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.347 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,75497472~37399462,121634816~628497,125829120~12179845,143455364~6678762,150211175~19165119,171336895~678210,189493020~3697637,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,335245198~6519073,341936085~10998133,356515840~5974373,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.348 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 18 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,69499952~11414938,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.348 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,69499952~11414938,83486776~18250672,103180346~9716588,122121191~142122,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,192776308~414349,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.356 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4800428,69499952~43396982,121634816~628497,125829120~12179845,143455364~6678762,150211175~19165119,171336895~678210,189493020~3697637,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~5974373,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.356 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 19 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.356 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.370 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.371 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 9 diff [83486776~5630781,127473758~2717223,130291297~6389032,253852109~5582104,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680] 2022-04-22T18:11:50.371 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [83486776~5630781,127473758~2717223,130291297~6389032,253852109~5582104,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680] 2022-04-22T18:11:50.381 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [79691776~9425781,125829120~10851209,253852109~5582104,268439604~6008930,289406976~8760300,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680] 2022-04-22T18:11:50.381 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 10 diff [1168411~5790915,18904669~6734780,83486776~5630781,127473758~2717223,130291297~6389032,143455364~3634220,153231026~4721732,161603998~7772296,216742230~4059603,253852109~5582104,262818021~1653011,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.381 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,83486776~5630781,127473758~2717223,130291297~6389032,143455364~3634220,153231026~4721732,161603998~7772296,216742230~4059603,253852109~5582104,262818021~1653011,268439604~6008930,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.389 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,79691776~9425781,125829120~10851209,143455364~3634220,153231026~4721732,159383552~9992742,213909504~6892329,253852109~5582104,262818021~1653011,268439604~6008930,289406976~8760300,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.390 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 11 diff [1168411~5790915,18904669~6734780,83486776~8990124,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.390 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,83486776~8990124,127473758~2717223,130291297~6389032,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~7142819,291909219~6258057,303824824~3183847,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.398 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,79691776~12785124,125829120~12179845,143455364~3634220,153231026~4721732,159383552~9992742,208638260~425055,213909504~6892329,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,284555601~13611675,303824824~3183847,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.398 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 12 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,83486776~8990124,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.399 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,83486776~8990124,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~3103495,329868116~2905127,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.409 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,79691776~12785124,125829120~12179845,143455364~6678762,153231026~4721732,159383552~9992742,195989592~6532536,208638260~425055,213909504~6892329,229497117~2058267,245411256~7036075,253852109~5582104,262818021~1653011,268439604~6008930,276824064~5902781,284555601~13611675,303824824~3183847,308833282~6613783,319043276~3103495,327155712~5617531,334492946~693680,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.409 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 13 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,83486776~12719617,107119709~5777225,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~8126476,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.409 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,83486776~12719617,107119709~5777225,127473758~2717223,130291297~7016548,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~8126476,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.418 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,75497472~20708921,107119709~5777225,125829120~12179845,143455364~6678762,153231026~4721732,159383552~9992742,195989592~6532536,208638260~425055,213909504~6892329,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,276824064~5902781,284555601~13611675,303824824~3183847,308833282~6613783,319043276~4490807,327155712~8030914,336561845~688056,341936085~8126476,363341370~3459338,381681664~4356327,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.457 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 14 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,83486776~12983481,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.458 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,83486776~12983481,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~7999846,274740903~2717085,278521545~4205300,284555601~7142819,291909219~6258057,303824824~3183847,308833282~6613783,319043276~4490807,327866007~7320619,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.462 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,75497472~20972785,107119709~5777225,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,195989592~6532536,203691436~7414116,213909504~11891082,229497117~2058267,235175037~7376011,245411256~7036075,253852109~5582104,262818021~1653011,266448688~16278157,284555601~13611675,303824824~3183847,308833282~6613783,319043276~4490807,327155712~8030914,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,381681664~4356327,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.462 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 15 diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~13053931,245411256~7601064,253852109~5582104,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.462 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~13053931,245411256~7601064,253852109~5582104,262818021~1653011,265938071~8510463,274740903~2717085,278521545~4205300,284555601~7142819,291909219~9046162,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.462 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,75497472~26239976,107119709~5777225,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,195989592~6532536,203691436~7414116,213909504~11891082,229497117~13053931,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,381681664~4356327,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.463 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 16 diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,229497117~14780037,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.463 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,229497117~14780037,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.463 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,75497472~26239976,107119709~5777225,125829120~12179845,143455364~6678762,150211175~7741583,159383552~9992742,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,213909504~11891082,229497117~14780037,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,378611238~7426753,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.463 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 17 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.464 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327866007~7320619,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.464 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,75497472~26239976,107119709~5777225,125829120~12179845,143455364~6678762,150211175~19165119,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,308833282~6613783,316179792~7354291,327155712~8030914,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.464 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 18 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.464 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,83486776~18250672,107119709~5777225,127473758~9834087,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,253852109~5582104,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.464 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~32237496,107119709~5777225,125829120~12179845,143455364~6678762,150211175~19165119,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,253852109~5582104,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,357217629~5272584,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.465 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 19 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.465 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.467 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.467 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 10 diff [1168411~5790915,18904669~6734780,143455364~3634220,153231026~4721732,161603998~7772296,216742230~4059603,262818021~1653011,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.468 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,143455364~3634220,153231026~4721732,161603998~7772296,216742230~4059603,262818021~1653011,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.473 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,143455364~3634220,153231026~4721732,159383552~9992742,213909504~6892329,262818021~1653011,336561845~688056,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.473 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 11 diff [1168411~5790915,18904669~6734780,87052742~5424158,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,262818021~1653011,284555601~7142819,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.474 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,87052742~5424158,137917491~91474,143455364~3634220,153231026~4721732,161603998~7772296,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,262818021~1653011,284555601~7142819,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.482 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,87052742~5424158,136680329~1328636,143455364~3634220,153231026~4721732,159383552~9992742,208638260~425055,213909504~6892329,229497117~2058267,245411256~7036075,262818021~1653011,284555601~7142819,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.482 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 12 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2273897,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,262818021~1653011,278521545~4205300,284555601~7142819,308833282~6613783,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.482 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2273897,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,245411256~7036075,262818021~1653011,278521545~4205300,284555601~7142819,308833282~6613783,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.490 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2975017,143455364~6678762,153231026~4721732,159383552~9992742,195989592~6532536,208638260~425055,213909504~6892329,229497117~2058267,245411256~7036075,262818021~1653011,276824064~5902781,284555601~7142819,308833282~6613783,336561845~688056,341936085~8126476,363341370~3459338,391705571~4418425] 2022-04-22T18:11:50.490 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 13 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,87052742~9153651,107119709~5777225,135033948~2273897,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.490 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,87052742~9153651,107119709~5777225,135033948~2273897,137917491~91474,143455364~6678762,153231026~4721732,161603998~7772296,195989592~6532536,208638260~425055,216742230~4059603,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,363341370~3459338,385864957~173034,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.498 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~1490382,55300293~6354902,75497472~5417418,87052742~9153651,107119709~5777225,135033948~2975017,143455364~6678762,153231026~4721732,159383552~9992742,195989592~6532536,208638260~425055,213909504~6892329,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,276824064~5902781,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,363341370~3459338,381681664~4356327,391705571~4418425,401584875~2116936] 2022-04-22T18:11:50.498 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 14 diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2273897,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,274740903~2717085,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.499 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2273897,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,274740903~2717085,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.507 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,18904669~6734780,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2975017,143455364~6678762,150211175~7741583,159383552~9992742,195989592~6532536,203691436~7414116,213909504~11891082,229497117~2058267,235175037~7376011,245411256~7036075,262818021~1653011,266448688~5166736,274448534~8278311,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,381681664~4356327,391705571~4418425,401505364~2422986] 2022-04-22T18:11:50.507 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 15 diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~13053931,245411256~7601064,262818021~1653011,265938071~5677353,274740903~2717085,278521545~4205300,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.508 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,195989592~6532536,203691436~7414116,216742230~4059603,222060317~3740269,229497117~13053931,245411256~7601064,262818021~1653011,265938071~5677353,274740903~2717085,278521545~4205300,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,385864957~173034,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.515 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,143455364~6678762,150211175~7741583,159383552~9992742,195989592~6532536,203691436~7414116,213909504~11891082,229497117~13053931,245411256~7601064,262818021~1653011,265938071~5677353,274448534~8278311,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,336561845~688056,341936085~8126476,357217629~5272584,363341370~3459338,381681664~4356327,391705571~4418425,401505364~3781016] 2022-04-22T18:11:50.515 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 16 diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,229497117~14780037,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.515 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,13387288~12252161,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~7741583,161603998~7772296,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,216742230~4059603,222060317~3740269,229497117~14780037,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~5171933,385864957~173034,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:50 smithi121 bash[36902]: cluster 2022-04-22T18:11:49.087869+0000 mgr.y (mgr.34107) 1611 : cluster [DBG] pgmap v2304: 137 pgs: 137 active+clean; 819 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 43 MiB/s wr, 32 op/s 2022-04-22T18:11:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:50 smithi121 bash[36902]: cluster 2022-04-22T18:11:49.260620+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2022-04-22T18:11:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:50 smithi121 bash[36902]: audit 2022-04-22T18:11:49.982350+0000 mon.a (mon.0) 3422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:11:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:50 smithi121 bash[36902]: audit 2022-04-22T18:11:49.984381+0000 mon.a (mon.0) 3423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:11:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:50 smithi121 bash[36902]: audit 2022-04-22T18:11:49.985954+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:11:50.523 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,13222991~12416458,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,143455364~6678762,150211175~7741583,159383552~9992742,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,213909504~11891082,229497117~14780037,245411256~7601064,258655884~338376,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,378611238~7426753,391705571~5062857,401505364~3781016] 2022-04-22T18:11:50.523 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 17 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.523 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,379125461~11337910,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.531 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,143455364~6678762,150211175~19165119,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,258655884~338376,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.532 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 18 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.532 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,143455364~6678762,150211175~11123650,161603998~7772296,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,216742230~4059603,222060317~22216837,245411256~7601064,258655884~338376,262818021~1653011,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,363341370~3459338,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.540 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,143455364~6678762,150211175~19165119,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,213909504~30367650,245411256~7601064,258655884~338376,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,363341370~3459338,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.540 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 19 diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.540 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[45495]: cluster 2022-04-22T18:11:49.087869+0000 mgr.y (mgr.34107) 1611 : cluster [DBG] pgmap v2304: 137 pgs: 137 active+clean; 819 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 43 MiB/s wr, 32 op/s 2022-04-22T18:11:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[45495]: cluster 2022-04-22T18:11:49.260620+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e1211: 8 total, 8 up, 8 in 2022-04-22T18:11:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[45495]: audit 2022-04-22T18:11:49.982350+0000 mon.a (mon.0) 3422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:11:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[45495]: audit 2022-04-22T18:11:49.984381+0000 mon.a (mon.0) 3423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:11:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:50 smithi115 bash[45495]: audit 2022-04-22T18:11:49.985954+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:11:50.548 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:50.549 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 11 diff [87052742~5424158,137917491~91474,167434704~1558571,208638260~425055,229497117~2058267,245411256~7036075,284555601~7142819,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.549 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [87052742~5424158,137917491~91474,167434704~1558571,208638260~425055,229497117~2058267,245411256~7036075,284555601~7142819,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.557 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [87052742~5424158,136680329~1328636,167434704~1558571,208638260~425055,229497117~2058267,245411256~7036075,284555601~7142819,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.557 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 12 diff [35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,245411256~7036075,278521545~4205300,284555601~7142819,308833282~6613783,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.557 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,245411256~7036075,278521545~4205300,284555601~7142819,308833282~6613783,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.565 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [35608822~4750050,40893629~1490382,55300293~6354902,87052742~5424158,135033948~2975017,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,245411256~7036075,276824064~5902781,284555601~7142819,308833282~6613783,341936085~8126476,391779685~2212477] 2022-04-22T18:11:50.565 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 13 diff [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,87052742~9153651,107119709~5777225,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,385864957~173034,391779685~2212477,401584875~2116936] 2022-04-22T18:11:50.565 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,87052742~9153651,107119709~5777225,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,385864957~173034,391779685~2212477,401584875~2116936] 2022-04-22T18:11:50.574 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,75497472~5417418,87052742~9153651,107119709~5777225,135033948~2975017,144417850~2152773,146974399~3159727,167434704~1558571,195989592~6532536,208638260~425055,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,276824064~5902781,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,381681664~4356327,391779685~2212477,401584875~2116936] 2022-04-22T18:11:50.574 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 14 diff [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,274740903~2717085,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,385864957~173034,391779685~2212477,401505364~2422986] 2022-04-22T18:11:50.574 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2273897,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,222060317~3740269,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,274740903~2717085,278521545~4205300,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,385864957~173034,391779685~2212477,401505364~2422986] 2022-04-22T18:11:50.582 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~9417515,107119709~5777225,127717081~4736765,135033948~2975017,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,220801833~4998753,229497117~2058267,235175037~7376011,245411256~7036075,266448688~5166736,274448534~8278311,284555601~7142819,308833282~6613783,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,381681664~4356327,391779685~2212477,401505364~2422986] 2022-04-22T18:11:50.583 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 15 diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,222060317~3740269,229497117~13053931,245411256~7601064,265938071~5677353,274740903~2717085,278521545~4205300,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,385864957~173034,391779685~2212477,401505364~3781016] 2022-04-22T18:11:50.583 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,222060317~3740269,229497117~13053931,245411256~7601064,265938071~5677353,274740903~2717085,278521545~4205300,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,385864957~173034,391779685~2212477,401505364~3781016] 2022-04-22T18:11:50.591 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,195989592~6532536,203691436~7414116,220801833~4998753,229497117~13053931,245411256~7601064,265938071~5677353,274448534~8278311,284555601~7142819,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,341936085~8126476,357217629~5272584,381681664~4356327,391779685~2212477,401505364~3781016] 2022-04-22T18:11:50.591 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 16 diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,222060317~3740269,229497117~14780037,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,379125461~5171933,385864957~173034,391779685~2212477,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.592 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,222060317~3740269,229497117~14780037,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,379125461~5171933,385864957~173034,391779685~2212477,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.598 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,144417850~2152773,146974399~3159727,150211175~5896403,167434704~1558571,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,220801833~4998753,229497117~14780037,245411256~7601064,258655884~338376,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,378611238~7426753,391779685~2212477,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.598 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 17 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~22216837,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,379125461~11337910,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.598 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~22216837,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,379125461~11337910,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.606 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,75497472~5417418,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,220801833~23475321,245411256~7601064,258655884~338376,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,341936085~8126476,357217629~5272584,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.606 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 18 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~22216837,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.606 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~22216837,245411256~7601064,258655884~338376,265938071~11754703,278521545~4205300,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.614 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,87052742~14684706,107119709~5777225,127717081~4736765,132587116~5421849,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,220801833~23475321,245411256~7601064,258655884~338376,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,357217629~5272584,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.614 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 19 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.614 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.621 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.622 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 12 diff [35608822~4750050,40893629~1490382,55300293~6354902,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,278521545~4205300,308833282~6613783] 2022-04-22T18:11:50.622 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [35608822~4750050,40893629~1490382,55300293~6354902,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,278521545~4205300,308833282~6613783] 2022-04-22T18:11:50.630 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [35608822~4750050,40893629~1490382,55300293~6354902,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,276824064~5902781,308833282~6613783] 2022-04-22T18:11:50.630 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 13 diff [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,88716135~7490258,107119709~5777225,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,235175037~7376011,266448688~5166736,278521545~4205300,308833282~6613783,319260150~4273933,327866007~6887213,385864957~173034,401584875~2116936] 2022-04-22T18:11:50.630 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,76803585~4111305,88716135~7490258,107119709~5777225,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,235175037~7376011,266448688~5166736,278521545~4205300,308833282~6613783,319260150~4273933,327866007~6887213,385864957~173034,401584875~2116936] 2022-04-22T18:11:50.638 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,35608822~4750050,40893629~1490382,55300293~6354902,75497472~5417418,88716135~7490258,107119709~5777225,135033948~2273897,144417850~2152773,146974399~3159727,195989592~6532536,235175037~7376011,266448688~5166736,276824064~5902781,308833282~6613783,319260150~4273933,327866007~6887213,381681664~4356327,401584875~2116936] 2022-04-22T18:11:50.638 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 14 diff [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,88716135~7754122,107119709~5777225,127717081~4736765,135033948~2273897,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,222060317~3740269,235175037~7376011,266448688~5166736,274740903~2717085,278521545~4205300,308833282~6613783,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~2422986] 2022-04-22T18:11:50.638 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,76803585~4111305,88716135~7754122,107119709~5777225,127717081~4736765,135033948~2273897,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,222060317~3740269,235175037~7376011,266448688~5166736,274740903~2717085,278521545~4205300,308833282~6613783,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~2422986] 2022-04-22T18:11:50.647 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,88716135~7754122,107119709~5777225,127717081~4736765,135033948~2273897,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,220801833~4998753,235175037~7376011,266448688~5166736,274448534~8278311,308833282~6613783,319260150~4273933,327866007~6887213,357217629~5272584,381681664~4356327,401505364~2422986] 2022-04-22T18:11:50.647 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 15 diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,222060317~3740269,230511465~12039583,248719919~4292401,265938071~5677353,274740903~2717085,278521545~4205300,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~3781016] 2022-04-22T18:11:50.647 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,222060317~3740269,230511465~12039583,248719919~4292401,265938071~5677353,274740903~2717085,278521545~4205300,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~3781016] 2022-04-22T18:11:50.655 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,195989592~6532536,203691436~7414116,220801833~4998753,230511465~12039583,248719919~4292401,265938071~5677353,274448534~8278311,296037928~4917453,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,381681664~4356327,401505364~3781016] 2022-04-22T18:11:50.655 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 16 diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,222060317~3740269,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~5171933,385864957~173034,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.655 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,35608822~4750050,40893629~3363170,55300293~6354902,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,222060317~3740269,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~5171933,385864957~173034,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.663 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,35608822~4750050,40893629~3363170,55300293~6354902,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~5896403,171336895~678210,195989592~6532536,203691436~7414116,211455361~2113346,220801833~4998753,230511465~13765689,248719919~4292401,258655884~338376,265938071~16788774,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,378611238~7426753,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.663 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 17 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.663 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.671 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,220801833~9004195,230511465~13765689,248719919~4292401,258655884~338376,265938071~16788774,296037928~4917453,301255591~133439,308833282~6613783,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.671 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 18 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.671 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,265938071~11754703,278521545~4205300,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.679 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~4720729,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,195069328~7452800,203691436~7414116,211455361~2113346,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.679 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 19 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.679 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.687 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.687 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 13 diff [4849579~2102461,76803585~4111305,88716135~7490258,107119709~5777225,235175037~7376011,266448688~5166736,319260150~4273933,327866007~6887213,385864957~173034,401584875~2116936] 2022-04-22T18:11:50.687 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,76803585~4111305,88716135~7490258,107119709~5777225,235175037~7376011,266448688~5166736,319260150~4273933,327866007~6887213,385864957~173034,401584875~2116936] 2022-04-22T18:11:50.695 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,75497472~5417418,88716135~7490258,107119709~5777225,235175037~7376011,266448688~5166736,319260150~4273933,327866007~6887213,381681664~4356327,401584875~2116936] 2022-04-22T18:11:50.696 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 14 diff [4849579~2102461,41674998~2581801,56876848~4001363,76803585~4111305,88716135~7754122,107119709~5777225,127717081~4736765,150211175~5896403,203691436~7414116,222060317~3740269,235175037~7376011,266448688~5166736,274740903~2717085,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~2422986] 2022-04-22T18:11:50.696 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,41674998~2581801,56876848~4001363,76803585~4111305,88716135~7754122,107119709~5777225,127717081~4736765,150211175~5896403,203691436~7414116,222060317~3740269,235175037~7376011,266448688~5166736,274740903~2717085,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~2422986] 2022-04-22T18:11:50.703 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,41674998~2581801,56876848~4001363,75497472~5417418,88716135~7754122,107119709~5777225,127717081~4736765,150211175~5896403,203691436~7414116,220801833~4998753,235175037~7376011,266448688~5166736,274448534~3009454,319260150~4273933,327866007~6887213,357217629~5272584,381681664~4356327,401505364~2422986] 2022-04-22T18:11:50.704 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 15 diff [4849579~2102461,13387288~5966260,41674998~2581801,56876848~4001363,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,222060317~3740269,230511465~12039583,248719919~4292401,265938071~5677353,274740903~2717085,296037928~4917453,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~3781016] 2022-04-22T18:11:50.704 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,41674998~2581801,56876848~4001363,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,222060317~3740269,230511465~12039583,248719919~4292401,265938071~5677353,274740903~2717085,296037928~4917453,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,385864957~173034,401505364~3781016] 2022-04-22T18:11:50.711 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,41674998~2581801,56876848~4001363,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,220801833~4998753,230511465~12039583,248719919~4292401,265938071~5677353,274448534~3009454,296037928~4917453,316179792~2932051,319260150~4273933,327866007~6887213,357217629~5272584,381681664~4356327,401505364~3781016] 2022-04-22T18:11:50.712 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 16 diff [4849579~2102461,13387288~5966260,35674797~2403115,41674998~2581801,56876848~4001363,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,222060317~3740269,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~5171933,385864957~173034,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.712 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,13387288~5966260,35674797~2403115,41674998~2581801,56876848~4001363,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,222060317~3740269,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~5171933,385864957~173034,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.720 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,13222991~6130557,35674797~2403115,41674998~2581801,56876848~4001363,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,220801833~4998753,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,378611238~7426753,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.720 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 17 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.720 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,76212653~4702237,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.729 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,75497472~5417418,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,220801833~9004195,230511465~13765689,248719919~4292401,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,316179792~2932051,319260150~4273933,327866007~6887213,335245198~6519073,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.729 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 18 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.729 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.736 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~11414938,88716135~13021313,107119709~5777225,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,265938071~11754703,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.736 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 19 diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.736 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.745 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.745 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 14 diff [41674998~2581801,56876848~4001363,91833690~4636567,127717081~4736765,150211175~5896403,203691436~7414116,222060317~3740269,274740903~2717085,357217629~5272584,401505364~2422986] 2022-04-22T18:11:50.745 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [41674998~2581801,56876848~4001363,91833690~4636567,127717081~4736765,150211175~5896403,203691436~7414116,222060317~3740269,274740903~2717085,357217629~5272584,401505364~2422986] 2022-04-22T18:11:50.753 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [41674998~2581801,56876848~4001363,91833690~4636567,127717081~4736765,150211175~5896403,203691436~7414116,220801833~4998753,274448534~3009454,357217629~5272584,401505364~2422986] 2022-04-22T18:11:50.753 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 15 diff [13387288~5966260,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,222060317~3740269,230511465~5876747,248719919~4292401,265938071~1852873,274740903~2717085,296037928~4917453,316179792~2932051,357217629~5272584,401505364~3781016] 2022-04-22T18:11:50.753 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [13387288~5966260,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,222060317~3740269,230511465~5876747,248719919~4292401,265938071~1852873,274740903~2717085,296037928~4917453,316179792~2932051,357217629~5272584,401505364~3781016] 2022-04-22T18:11:50.760 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [13222991~6130557,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,203691436~7414116,220801833~4998753,230511465~5876747,248719919~4292401,265938071~1852873,274448534~3009454,296037928~4917453,316179792~2932051,357217629~5272584,401505364~3781016] 2022-04-22T18:11:50.761 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 16 diff [13387288~5966260,35674797~2403115,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,222060317~3740269,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,379125461~5171933,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.761 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [13387288~5966260,35674797~2403115,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,222060317~3740269,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,379125461~5171933,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.768 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [13222991~6130557,35674797~2403115,41674998~2581801,56876848~4001363,76212653~2952306,91833690~9903758,127717081~4736765,132587116~3455386,150211175~5896403,171336895~678210,203691436~7414116,211455361~2113346,220801833~4998753,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,378611238~5686156,394912686~1855742,401505364~3781016] 2022-04-22T18:11:50.769 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 17 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,76212653~2952306,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.769 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,76212653~2952306,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,379125461~11337910,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.783 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,76212653~2952306,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,220801833~9004195,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.784 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 18 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~9665007,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.784 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~9665007,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,357217629~5272584,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.784 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,63641946~4490210,69499952~9665007,88764093~12973355,127717081~4736765,132587116~3455386,150211175~11123650,171336895~678210,195069328~4053541,203691436~7414116,211455361~2113346,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,357217629~5272584,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.785 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 19 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.785 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.792 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:50.792 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 15 diff [13387288~5966260,76212653~2952306,94707726~7029722,132587116~3455386,230511465~5876747,248719919~4292401,265938071~1852873,296037928~4917453,316179792~2932051,402421085~2865295] 2022-04-22T18:11:50.793 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [13387288~5966260,76212653~2952306,94707726~7029722,132587116~3455386,230511465~5876747,248719919~4292401,265938071~1852873,296037928~4917453,316179792~2932051,402421085~2865295] 2022-04-22T18:11:50.800 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [13222991~6130557,76212653~2952306,94707726~7029722,132587116~3455386,230511465~5876747,248719919~4292401,265938071~1852873,296037928~4917453,316179792~2932051,402421085~2865295] 2022-04-22T18:11:50.800 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 16 diff [13387288~5966260,35674797~2403115,76212653~2952306,94707726~7029722,132587116~3455386,171336895~678210,211455361~2113346,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,379125461~5171933,394912686~1855742,402421085~2865295] 2022-04-22T18:11:50.801 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [13387288~5966260,35674797~2403115,76212653~2952306,94707726~7029722,132587116~3455386,171336895~678210,211455361~2113346,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,379125461~5171933,394912686~1855742,402421085~2865295] 2022-04-22T18:11:50.808 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [13222991~6130557,35674797~2403115,76212653~2952306,94707726~7029722,132587116~3455386,171336895~678210,211455361~2113346,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,378611238~5686156,394912686~1855742,402421085~2865295] 2022-04-22T18:11:50.808 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 17 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,63641946~4490210,76212653~2952306,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,379125461~11337910,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.809 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,63641946~4490210,76212653~2952306,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,379125461~11337910,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.816 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,63641946~4490210,76212653~2952306,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,230511465~5876747,241784386~2492768,248719919~4292401,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,316179792~2932051,335245198~6519073,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.816 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 18 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~9665007,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.817 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~9665007,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.824 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~9665007,88764093~12973355,132587116~3455386,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,265938071~1852873,270495365~7197409,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.824 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 19 diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.825 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.832 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,61655195~17509764,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:50.832 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 16 diff [35674797~2403115,171336895~678210,211455361~2113346,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,379125461~5171933,394912686~1855742] 2022-04-22T18:11:50.832 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [35674797~2403115,171336895~678210,211455361~2113346,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,379125461~5171933,394912686~1855742] 2022-04-22T18:11:50.840 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [35674797~2403115,171336895~678210,211455361~2113346,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,378611238~5686156,394912686~1855742] 2022-04-22T18:11:50.840 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 17 diff [8249801~2336322,21565899~6609152,35674797~2403115,63641946~4490210,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,379125461~11337910,394912686~6077014] 2022-04-22T18:11:50.840 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,63641946~4490210,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,379125461~11337910,394912686~6077014] 2022-04-22T18:11:50.849 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,63641946~4490210,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,211455361~2113346,223511505~6294523,241784386~2492768,258655884~338376,270495365~7197409,301255591~133439,335245198~6519073,378611238~11852133,394912686~6077014] 2022-04-22T18:11:50.849 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 18 diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,270495365~7197409,301255591~133439,307745298~7714921,330659402~11104869,378967757~11495614,394912686~6077014] 2022-04-22T18:11:50.849 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,270495365~7197409,301255591~133439,307745298~7714921,330659402~11104869,378967757~11495614,394912686~6077014] 2022-04-22T18:11:50.856 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,171336895~678210,195069328~4053541,209059456~1186466,211455361~2113346,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,270495365~7197409,301255591~133439,307745298~7714921,330659402~11104869,378611238~11852133,394912686~6077014] 2022-04-22T18:11:50.856 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 19 diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:50.857 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:50.864 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014] 2022-04-22T18:11:50.864 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 17 diff [8249801~2336322,21565899~6609152,63641946~4490210,88764093~5984638,151328086~10006739,195069328~4053541,223511505~6294523,383460721~7002650,395675089~5314611] 2022-04-22T18:11:50.865 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,63641946~4490210,88764093~5984638,151328086~10006739,195069328~4053541,223511505~6294523,383460721~7002650,395675089~5314611] 2022-04-22T18:11:50.873 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,63641946~4490210,88764093~5984638,151328086~10006739,195069328~4053541,223511505~6294523,383460721~7002650,395675089~5314611] 2022-04-22T18:11:50.873 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 18 diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,195069328~4053541,209059456~1186466,222381315~7424713,247763752~2157508,307745298~7714921,330659402~5044208,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.873 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,195069328~4053541,209059456~1186466,222381315~7424713,247763752~2157508,307745298~7714921,330659402~5044208,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.880 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,63641946~4490210,69499952~7470379,88764093~5984638,151328086~10006739,195069328~4053541,209059456~1186466,222381315~7424713,247763752~2157508,307745298~7714921,330659402~5044208,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.880 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 19 diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.880 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.888 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:50.888 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to 18 diff [37541079~284740,58335599~1987181,69499952~7470379,209059456~1186466,222381315~4717533,247763752~2157508,307745298~7714921,330659402~5044208,378967757~5289639] 2022-04-22T18:11:50.888 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [37541079~284740,58335599~1987181,69499952~7470379,209059456~1186466,222381315~4717533,247763752~2157508,307745298~7714921,330659402~5044208,378967757~5289639] 2022-04-22T18:11:50.896 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [37541079~284740,58335599~1987181,69499952~7470379,209059456~1186466,222381315~4717533,247763752~2157508,307745298~7714921,330659402~5044208,378967757~5289639] 2022-04-22T18:11:50.896 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to 19 diff [37541079~284740,58335599~1987181,62715066~2807803,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:50.896 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [37541079~284740,58335599~1987181,62715066~2807803,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:50.904 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [37541079~284740,58335599~1987181,61655195~3867674,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:50.904 INFO:tasks.workunit.client.0.smithi115.stdout:from 18 to 19 diff [62715066~2807803,66594375~7588506,101731803~4113128,141133052~3130409,190798808~2972860,211468957~8070134,260709637~413875,273667814~6753815,350629815~1349947,361213496~6965178] 2022-04-22T18:11:50.904 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [62715066~2807803,66594375~7588506,101731803~4113128,141133052~3130409,190798808~2972860,211468957~8070134,260709637~413875,273667814~6753815,350629815~1349947,361213496~6965178] 2022-04-22T18:11:50.912 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [61655195~3867674,66594375~7588506,101731803~4113128,141133052~3130409,190798808~2972860,211468957~8070134,260709637~413875,273667814~6753815,350629815~1349947,361213496~6965178] 2022-04-22T18:11:51.424 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.424 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.441 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:51.442 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.442 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.450 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:51.450 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.450 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.458 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:51.458 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.459 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.467 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:51.467 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.467 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.475 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:51.475 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.475 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.484 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:51.484 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.485 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:51.492 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:51.492 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.492 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.500 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.501 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.501 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.509 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.509 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.509 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.518 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:51.518 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.518 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.527 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.527 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.527 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.536 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.536 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.536 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.546 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.546 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.546 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.552 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:51.553 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:51.553 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:51.561 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,61655195~17509764,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:51.561 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:51.561 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:51.570 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014] 2022-04-22T18:11:51.570 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to HEAD diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:51.570 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:51.578 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:51.578 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to HEAD diff [37541079~284740,58335599~1987181,62715066~2807803,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:51.578 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [37541079~284740,58335599~1987181,62715066~2807803,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:51.585 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [37541079~284740,58335599~1987181,61655195~3867674,66594375~10375956,101731803~4113128,141133052~3130409,190798808~2972860,209059456~1186466,211468957~8070134,222381315~4717533,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~5289639] 2022-04-22T18:11:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:51 smithi121 bash[36902]: audit 2022-04-22T18:11:50.397658+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:51.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:51 smithi121 bash[36902]: audit 2022-04-22T18:11:50.687192+0000 mon.a (mon.0) 3426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:51.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:51 smithi121 bash[36902]: audit 2022-04-22T18:11:51.128941+0000 mon.a (mon.0) 3427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:51.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[45495]: audit 2022-04-22T18:11:50.397658+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:51.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[45495]: audit 2022-04-22T18:11:50.687192+0000 mon.a (mon.0) 3426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:51.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[45495]: audit 2022-04-22T18:11:51.128941+0000 mon.a (mon.0) 3427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:51.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[46934]: audit 2022-04-22T18:11:50.397658+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:11:51.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[46934]: audit 2022-04-22T18:11:50.687192+0000 mon.a (mon.0) 3426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:51.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[46934]: audit 2022-04-22T18:11:51.128941+0000 mon.a (mon.0) 3427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:51.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:52.438 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.438 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.460 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:52.460 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.460 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.468 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:52.468 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.468 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.477 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:52.477 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.477 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.485 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:52.537 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.537 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.543 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:52.550 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.550 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.551 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:52.551 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.551 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:52.551 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:52.552 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.552 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.553 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.553 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.553 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.554 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.554 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.555 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.557 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.557 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.557 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.559 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.559 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.559 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.568 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:52.568 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:52.568 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:52.577 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,61655195~17509764,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:52.578 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:52.578 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:52.586 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014] 2022-04-22T18:11:52.587 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to HEAD diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:52.587 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:52.593 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,37541079~284740,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,190798808~2972860,195069328~4053541,209059456~1186466,211468957~8070134,222381315~7424713,247763752~2157508,260709637~413875,273667814~6753815,307745298~7714921,330659402~5044208,350629815~1349947,361213496~6965178,378967757~11495614,395675089~5314611] 2022-04-22T18:11:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:52 smithi121 bash[36902]: cluster 2022-04-22T18:11:51.089110+0000 mgr.y (mgr.34107) 1612 : cluster [DBG] pgmap v2306: 137 pgs: 137 active+clean; 858 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 35 MiB/s wr, 871 op/s 2022-04-22T18:11:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:52 smithi121 bash[36902]: cluster 2022-04-22T18:11:51.410556+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2022-04-22T18:11:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:52 smithi115 bash[46934]: cluster 2022-04-22T18:11:51.089110+0000 mgr.y (mgr.34107) 1612 : cluster [DBG] pgmap v2306: 137 pgs: 137 active+clean; 858 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 35 MiB/s wr, 871 op/s 2022-04-22T18:11:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:52 smithi115 bash[46934]: cluster 2022-04-22T18:11:51.410556+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2022-04-22T18:11:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:52 smithi115 bash[45495]: cluster 2022-04-22T18:11:51.089110+0000 mgr.y (mgr.34107) 1612 : cluster [DBG] pgmap v2306: 137 pgs: 137 active+clean; 858 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 35 MiB/s wr, 871 op/s 2022-04-22T18:11:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:52 smithi115 bash[45495]: cluster 2022-04-22T18:11:51.410556+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e1212: 8 total, 8 up, 8 in 2022-04-22T18:11:53.467 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.467 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.478 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:53.478 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.478 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.484 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:53.485 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.485 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.492 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:53.493 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.493 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.501 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:53.501 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.501 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.510 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:53.510 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.511 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.518 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:53.519 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.520 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:53.526 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:53.527 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.527 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.535 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.535 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.535 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.544 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.545 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.545 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.553 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:53.553 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.553 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.560 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.560 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.560 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.569 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.569 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.577 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.578 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.578 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.586 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:53.586 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:53.587 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:53.594 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,61655195~17509764,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:53.594 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:53.594 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,62715066~14255265,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014] 2022-04-22T18:11:53.602 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,21565899~6609152,35674797~2403115,58335599~1987181,61655195~15315136,88764093~5984638,101731803~4113128,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,241784386~2492768,247763752~2157508,258655884~338376,260709637~413875,270495365~9926264,301255591~133439,307745298~7714921,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014] 2022-04-22T18:11:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:53 smithi121 bash[36902]: cluster 2022-04-22T18:11:52.421061+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2022-04-22T18:11:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:53 smithi121 bash[36902]: audit 2022-04-22T18:11:53.075280+0000 mon.a (mon.0) 3430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:53.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:11:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:11:53.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:11:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:53 smithi115 bash[45495]: cluster 2022-04-22T18:11:52.421061+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2022-04-22T18:11:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:53 smithi115 bash[45495]: audit 2022-04-22T18:11:53.075280+0000 mon.a (mon.0) 3430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:53 smithi115 bash[46934]: cluster 2022-04-22T18:11:52.421061+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e1213: 8 total, 8 up, 8 in 2022-04-22T18:11:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:53 smithi115 bash[46934]: audit 2022-04-22T18:11:53.075280+0000 mon.a (mon.0) 3430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:54.469 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.470 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.491 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:54.491 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.491 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.500 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:54.500 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.500 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.508 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:54.508 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.508 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.516 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:54.517 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.517 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.525 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:54.525 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.525 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.533 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:54.533 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.533 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:54.541 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:54.541 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.542 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.549 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.549 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.549 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.557 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.557 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.558 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.565 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:54.566 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.566 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.574 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.574 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.575 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.582 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.582 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.582 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.593 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.593 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.593 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.599 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:54.599 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:54.599 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,58335599~1987181,62715066~16449893,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378967757~11495614,394912686~6077014,402421085~2865295] 2022-04-22T18:11:54.607 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,58335599~1987181,61655195~17509764,88764093~17080838,132587116~3455386,141133052~3130409,151328086~10006739,171336895~678210,190798808~2972860,195069328~4053541,209059456~1186466,211455361~8083730,222381315~7424713,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,361213496~6965178,378611238~11852133,394912686~6077014,402421085~2865295] 2022-04-22T18:11:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:54 smithi121 bash[36902]: cluster 2022-04-22T18:11:53.090086+0000 mgr.y (mgr.34107) 1613 : cluster [DBG] pgmap v2309: 137 pgs: 137 active+clean; 876 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 1.75k op/s 2022-04-22T18:11:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:54 smithi121 bash[36902]: cluster 2022-04-22T18:11:53.432601+0000 mon.a (mon.0) 3431 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2022-04-22T18:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:54 smithi115 bash[45495]: cluster 2022-04-22T18:11:53.090086+0000 mgr.y (mgr.34107) 1613 : cluster [DBG] pgmap v2309: 137 pgs: 137 active+clean; 876 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 1.75k op/s 2022-04-22T18:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:54 smithi115 bash[45495]: cluster 2022-04-22T18:11:53.432601+0000 mon.a (mon.0) 3431 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2022-04-22T18:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:54 smithi115 bash[46934]: cluster 2022-04-22T18:11:53.090086+0000 mgr.y (mgr.34107) 1613 : cluster [DBG] pgmap v2309: 137 pgs: 137 active+clean; 876 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 1.75k op/s 2022-04-22T18:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:54 smithi115 bash[46934]: cluster 2022-04-22T18:11:53.432601+0000 mon.a (mon.0) 3431 : cluster [DBG] osdmap e1214: 8 total, 8 up, 8 in 2022-04-22T18:11:55.475 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.475 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.502 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:55.502 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.502 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.510 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:55.510 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.510 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.519 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:55.520 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.523 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.527 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:55.527 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.527 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.534 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:55.534 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.535 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.544 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:55.545 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.545 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:55.552 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:55.553 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.553 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.559 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.559 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.559 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.566 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.567 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.567 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.574 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:55.574 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.575 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.583 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.583 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.583 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.591 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.591 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.591 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.599 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.599 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.599 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~16449893,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.607 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~17509764,88764093~17080838,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~5876747,241784386~2492768,247763752~5248568,258655884~338376,260709637~413875,265938071~1852873,270495365~9926264,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,330659402~11104869,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:55 smithi121 bash[36902]: cluster 2022-04-22T18:11:54.438818+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2022-04-22T18:11:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:55 smithi115 bash[45495]: cluster 2022-04-22T18:11:54.438818+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2022-04-22T18:11:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:55 smithi115 bash[46934]: cluster 2022-04-22T18:11:54.438818+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e1215: 8 total, 8 up, 8 in 2022-04-22T18:11:56.491 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.491 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.517 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:56.517 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.517 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.525 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:56.525 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.525 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.533 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:56.533 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.533 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.540 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:56.540 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.540 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.549 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:56.550 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.550 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.557 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:56.557 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.558 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:56.565 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:56.566 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.566 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.573 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.573 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.573 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.581 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.581 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.582 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.589 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:56.590 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.590 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.597 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.597 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.598 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.605 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.605 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.605 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.613 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35674797~2403115,41674998~2581801,56876848~4001363,61655195~19259695,88716135~17128796,107119709~5777225,127717081~4736765,132587116~3455386,141133052~3130409,150211175~11123650,171336895~678210,190798808~2972860,195069328~4053541,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~14483558,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:56 smithi121 bash[36902]: cluster 2022-04-22T18:11:55.091028+0000 mgr.y (mgr.34107) 1614 : cluster [DBG] pgmap v2312: 137 pgs: 137 active+clean; 880 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 4.09k op/s 2022-04-22T18:11:56.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:56 smithi121 bash[36902]: cluster 2022-04-22T18:11:55.447296+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2022-04-22T18:11:56.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:56 smithi121 bash[36902]: audit 2022-04-22T18:11:55.689789+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:56.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:56 smithi121 bash[36902]: audit 2022-04-22T18:11:56.132506+0000 mon.a (mon.0) 3435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:56.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[45495]: cluster 2022-04-22T18:11:55.091028+0000 mgr.y (mgr.34107) 1614 : cluster [DBG] pgmap v2312: 137 pgs: 137 active+clean; 880 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 4.09k op/s 2022-04-22T18:11:56.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[45495]: cluster 2022-04-22T18:11:55.447296+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2022-04-22T18:11:56.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[45495]: audit 2022-04-22T18:11:55.689789+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:56.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[45495]: audit 2022-04-22T18:11:56.132506+0000 mon.a (mon.0) 3435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:56.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[46934]: cluster 2022-04-22T18:11:55.091028+0000 mgr.y (mgr.34107) 1614 : cluster [DBG] pgmap v2312: 137 pgs: 137 active+clean; 880 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 4.09k op/s 2022-04-22T18:11:56.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[46934]: cluster 2022-04-22T18:11:55.447296+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e1216: 8 total, 8 up, 8 in 2022-04-22T18:11:56.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[46934]: audit 2022-04-22T18:11:55.689789+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:11:56.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:56 smithi115 bash[46934]: audit 2022-04-22T18:11:56.132506+0000 mon.a (mon.0) 3435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:11:57.492 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.492 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.529 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:57.529 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.530 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.540 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:57.540 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.540 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.545 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:57.545 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.546 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.554 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:57.554 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.554 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.562 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:57.562 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.562 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.568 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:57.569 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.569 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:57.576 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:57.576 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.577 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.584 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.585 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.585 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.594 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.595 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.595 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.600 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:57.603 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.604 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.610 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.610 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.610 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~7745711,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378967757~11495614,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.615 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,88716135~17128796,107119709~5777225,127717081~4736765,132587116~4720729,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~9004195,230511465~13765689,247763752~5248568,258655884~338376,260709637~413875,265938071~16788774,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,350629815~1349947,357217629~10961045,378611238~11852133,394912686~6077014,401505364~3781016] 2022-04-22T18:11:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:57 smithi121 bash[36902]: cluster 2022-04-22T18:11:56.459061+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2022-04-22T18:11:57.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:57 smithi115 bash[45495]: cluster 2022-04-22T18:11:56.459061+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2022-04-22T18:11:57.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:57 smithi115 bash[46934]: cluster 2022-04-22T18:11:56.459061+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e1217: 8 total, 8 up, 8 in 2022-04-22T18:11:58.505 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.505 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.538 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:58.539 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.539 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.546 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:58.547 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.547 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.554 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:58.554 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.555 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.562 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:58.562 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.563 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.569 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:58.570 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.570 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.577 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:58.578 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.578 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:58.586 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:58.586 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.586 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.594 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.594 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.594 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.602 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.603 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.603 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.609 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:58.609 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:58.610 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4849579~2102461,8249801~2336322,13387288~5966260,21565899~6609152,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:58.617 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4849579~2102461,8249801~2336322,13222991~6130557,21565899~6609152,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~3130409,144417850~2152773,146974399~3159727,150211175~11123650,167434704~1558571,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~8083730,220801833~23475321,245411256~7601064,258655884~338376,260709637~413875,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391779685~2212477,394912686~6077014,401505364~3781016] 2022-04-22T18:11:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:58 smithi121 bash[36902]: cluster 2022-04-22T18:11:57.091802+0000 mgr.y (mgr.34107) 1615 : cluster [DBG] pgmap v2315: 137 pgs: 137 active+clean; 868 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 3.96k op/s 2022-04-22T18:11:58.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:58 smithi121 bash[36902]: cluster 2022-04-22T18:11:57.468602+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e1218: 8 total, 8 up, 8 in 2022-04-22T18:11:58.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:58 smithi121 bash[36902]: audit 2022-04-22T18:11:58.078240+0000 mon.a (mon.0) 3438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:58.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[45495]: cluster 2022-04-22T18:11:57.091802+0000 mgr.y (mgr.34107) 1615 : cluster [DBG] pgmap v2315: 137 pgs: 137 active+clean; 868 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 3.96k op/s 2022-04-22T18:11:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[45495]: cluster 2022-04-22T18:11:57.468602+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e1218: 8 total, 8 up, 8 in 2022-04-22T18:11:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[45495]: audit 2022-04-22T18:11:58.078240+0000 mon.a (mon.0) 3438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:58.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[46934]: cluster 2022-04-22T18:11:57.091802+0000 mgr.y (mgr.34107) 1615 : cluster [DBG] pgmap v2315: 137 pgs: 137 active+clean; 868 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 3.96k op/s 2022-04-22T18:11:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[46934]: cluster 2022-04-22T18:11:57.468602+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e1218: 8 total, 8 up, 8 in 2022-04-22T18:11:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:58 smithi115 bash[46934]: audit 2022-04-22T18:11:58.078240+0000 mon.a (mon.0) 3438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:11:59.519 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.519 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.550 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:11:59.550 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.550 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.557 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:11:59.558 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.558 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.565 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:59.565 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.565 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.573 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:59.573 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.573 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.580 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:59.580 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.580 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.588 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:59.588 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.588 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:11:59.596 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:11:59.596 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.596 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.604 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.604 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.604 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.612 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.612 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.612 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,87052742~18792189,107119709~5777225,127717081~4736765,132587116~4720729,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.619 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~25614597,87052742~18792189,107119709~5777225,127717081~4736765,132587116~5421849,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,258655884~338376,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,296037928~4917453,301255591~133439,307745298~7714921,316179792~2932051,319260150~4273933,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:11:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:11:59 smithi121 bash[36902]: cluster 2022-04-22T18:11:58.473150+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2022-04-22T18:11:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:11:59 smithi115 bash[45495]: cluster 2022-04-22T18:11:58.473150+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2022-04-22T18:11:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:11:59 smithi115 bash[46934]: cluster 2022-04-22T18:11:58.473150+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e1219: 8 total, 8 up, 8 in 2022-04-22T18:12:00.540 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.541 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.563 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:00.564 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.564 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.570 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:00.571 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.571 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.578 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:00.579 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.579 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.587 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:00.587 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.587 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.594 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:00.595 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.595 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.602 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:00.602 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.602 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:00.612 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:00.612 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.612 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.618 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.618 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.619 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~22358155,107119709~5777225,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~8126476,350629815~1349947,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.626 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~50544638,107119709~5777225,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~8126476,350629815~1349947,357217629~10961045,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:12:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:00 smithi121 bash[36902]: cluster 2022-04-22T18:11:59.092636+0000 mgr.y (mgr.34107) 1616 : cluster [DBG] pgmap v2318: 137 pgs: 137 active+clean; 836 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.72k op/s 2022-04-22T18:12:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:00 smithi121 bash[36902]: cluster 2022-04-22T18:11:59.492442+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2022-04-22T18:12:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:00 smithi115 bash[45495]: cluster 2022-04-22T18:11:59.092636+0000 mgr.y (mgr.34107) 1616 : cluster [DBG] pgmap v2318: 137 pgs: 137 active+clean; 836 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.72k op/s 2022-04-22T18:12:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:00 smithi115 bash[45495]: cluster 2022-04-22T18:11:59.492442+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2022-04-22T18:12:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:00 smithi115 bash[46934]: cluster 2022-04-22T18:11:59.092636+0000 mgr.y (mgr.34107) 1616 : cluster [DBG] pgmap v2318: 137 pgs: 137 active+clean; 836 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.72k op/s 2022-04-22T18:12:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:00 smithi115 bash[46934]: cluster 2022-04-22T18:11:59.492442+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e1220: 8 total, 8 up, 8 in 2022-04-22T18:12:01.156 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.157 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.190 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:01.191 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.191 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.198 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:01.198 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.199 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.206 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:01.206 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.206 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.214 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:01.214 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.214 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.222 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:01.222 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.222 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.230 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:01.230 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.230 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:01.237 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:01.237 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:01.237 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,55300293~6354902,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,171336895~678210,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~9284129,401505364~3781016] 2022-04-22T18:12:01.245 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,55300293~57596641,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,171336895~678210,189493020~4278648,195069328~7452800,203691436~7414116,211455361~32821793,245411256~7601064,253852109~5582104,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~3183847,307745298~7714921,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~11852133,391705571~9284129,401505364~3781016] 2022-04-22T18:12:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:01 smithi121 bash[36902]: cluster 2022-04-22T18:12:00.504507+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2022-04-22T18:12:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:01 smithi121 bash[36902]: audit 2022-04-22T18:12:00.692963+0000 mon.a (mon.0) 3442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:01 smithi121 bash[36902]: cluster 2022-04-22T18:12:01.122733+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2022-04-22T18:12:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:01 smithi121 bash[36902]: audit 2022-04-22T18:12:01.135647+0000 mon.a (mon.0) 3444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:01.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[45495]: cluster 2022-04-22T18:12:00.504507+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[45495]: audit 2022-04-22T18:12:00.692963+0000 mon.a (mon.0) 3442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[45495]: cluster 2022-04-22T18:12:01.122733+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[45495]: audit 2022-04-22T18:12:01.135647+0000 mon.a (mon.0) 3444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[46934]: cluster 2022-04-22T18:12:00.504507+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e1221: 8 total, 8 up, 8 in 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[46934]: audit 2022-04-22T18:12:00.692963+0000 mon.a (mon.0) 3442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[46934]: cluster 2022-04-22T18:12:01.122733+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e1222: 8 total, 8 up, 8 in 2022-04-22T18:12:01.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[46934]: audit 2022-04-22T18:12:01.135647+0000 mon.a (mon.0) 3444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:01.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:02.156 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.156 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.196 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:02.234 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.234 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.236 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:02.236 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.237 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:02.238 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.238 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.238 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:02.238 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.238 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~3363170,49317039~12338156,62715066~18199824,83486776~29410158,122121191~142122,127473758~9834087,137917491~91474,141133052~9001074,150211175~11123650,161603998~7772296,170889023~5779566,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:02.241 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~3363170,49317039~63579895,121634816~628497,125829120~12179845,141133052~9001074,150211175~19165119,170889023~5779566,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~16399780,301255591~133439,303824824~11635395,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:02 smithi121 bash[36902]: cluster 2022-04-22T18:12:01.093614+0000 mgr.y (mgr.34107) 1617 : cluster [DBG] pgmap v2321: 137 pgs: 137 active+clean; 806 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.32k op/s 2022-04-22T18:12:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:02 smithi121 bash[36902]: cluster 2022-04-22T18:12:02.133568+0000 mon.a (mon.0) 3445 : cluster [DBG] osdmap e1223: 8 total, 8 up, 8 in 2022-04-22T18:12:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:02 smithi115 bash[45495]: cluster 2022-04-22T18:12:01.093614+0000 mgr.y (mgr.34107) 1617 : cluster [DBG] pgmap v2321: 137 pgs: 137 active+clean; 806 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.32k op/s 2022-04-22T18:12:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:02 smithi115 bash[45495]: cluster 2022-04-22T18:12:02.133568+0000 mon.a (mon.0) 3445 : cluster [DBG] osdmap e1223: 8 total, 8 up, 8 in 2022-04-22T18:12:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:02 smithi115 bash[46934]: cluster 2022-04-22T18:12:01.093614+0000 mgr.y (mgr.34107) 1617 : cluster [DBG] pgmap v2321: 137 pgs: 137 active+clean; 806 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.32k op/s 2022-04-22T18:12:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:02 smithi115 bash[46934]: cluster 2022-04-22T18:12:02.133568+0000 mon.a (mon.0) 3445 : cluster [DBG] osdmap e1223: 8 total, 8 up, 8 in 2022-04-22T18:12:03.163 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.163 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.197 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:03.197 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.197 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.204 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:03.204 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.204 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.212 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:03.212 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.212 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.219 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:03.219 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.219 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.227 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:03.227 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.227 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~14787763,35608822~4750050,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~12667212,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~14022957,260709637~413875,262818021~1653011,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~11635395,316179792~7354291,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:03.235 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~14952060,35608822~4750050,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~12667212,189493020~13029108,203691436~7414116,211455361~32821793,245411256~14022957,260709637~413875,262818021~1653011,265938071~35017310,301255591~133439,303020863~12439356,316179792~7354291,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:03.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:03.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:03.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:03 smithi115 bash[45495]: audit 2022-04-22T18:12:03.081177+0000 mon.a (mon.0) 3446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:03.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:03 smithi115 bash[45495]: cluster 2022-04-22T18:12:03.135919+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2022-04-22T18:12:03.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:03 smithi115 bash[46934]: audit 2022-04-22T18:12:03.081177+0000 mon.a (mon.0) 3446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:03.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:03 smithi115 bash[46934]: cluster 2022-04-22T18:12:03.135919+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2022-04-22T18:12:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:03 smithi121 bash[36902]: audit 2022-04-22T18:12:03.081177+0000 mon.a (mon.0) 3446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:03 smithi121 bash[36902]: cluster 2022-04-22T18:12:03.135919+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1224: 8 total, 8 up, 8 in 2022-04-22T18:12:04.179 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.179 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.193 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:04.193 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.193 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.201 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:04.201 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.201 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.209 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:04.209 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.210 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.216 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:04.217 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.217 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~6840941,40893629~5954937,49317039~12338156,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,141133052~20201773,161603998~7772296,170889023~13873574,188617691~875329,190798808~2972860,195069328~7452800,203691436~7414116,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~9046162,301255591~133439,303020863~360114,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:04.224 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~6840941,40893629~5954937,49317039~67957254,121634816~628497,125829120~12179845,141133052~28243242,170889023~31633105,203691436~7414116,211455361~32821793,245411256~55544125,301255591~133439,303020863~20513220,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:04 smithi115 bash[46934]: cluster 2022-04-22T18:12:03.094530+0000 mgr.y (mgr.34107) 1618 : cluster [DBG] pgmap v2324: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 810 op/s 2022-04-22T18:12:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:04 smithi115 bash[46934]: cluster 2022-04-22T18:12:04.139660+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e1225: 8 total, 8 up, 8 in 2022-04-22T18:12:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:04 smithi115 bash[45495]: cluster 2022-04-22T18:12:03.094530+0000 mgr.y (mgr.34107) 1618 : cluster [DBG] pgmap v2324: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 810 op/s 2022-04-22T18:12:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:04 smithi115 bash[45495]: cluster 2022-04-22T18:12:04.139660+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e1225: 8 total, 8 up, 8 in 2022-04-22T18:12:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:04 smithi121 bash[36902]: cluster 2022-04-22T18:12:03.094530+0000 mgr.y (mgr.34107) 1618 : cluster [DBG] pgmap v2324: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 810 op/s 2022-04-22T18:12:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:04 smithi121 bash[36902]: cluster 2022-04-22T18:12:04.139660+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e1225: 8 total, 8 up, 8 in 2022-04-22T18:12:05.163 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.163 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.196 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:05.196 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.196 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.203 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:05.203 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.203 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.210 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:05.210 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.210 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [1168411~5790915,8249801~2336322,13387288~19079665,33517931~13330635,48895278~12759917,62715066~18199824,83486776~33787517,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~16036224,211455361~9346472,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:05.217 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [1168411~5790915,8249801~2336322,13222991~19243962,33517931~83756362,121634816~628497,125829120~12179845,138412032~72693520,211455361~32821793,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:06.172 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.172 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.185 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:06.185 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.185 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.193 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:06.193 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.193 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~6601122,8249801~2336322,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,378967757~11495614,391705571~13814311] 2022-04-22T18:12:06.201 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~6959326,8249801~2336322,13222991~105443509,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,378611238~26908644] 2022-04-22T18:12:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:06 smithi121 bash[36902]: cluster 2022-04-22T18:12:05.095513+0000 mgr.y (mgr.34107) 1619 : cluster [DBG] pgmap v2327: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 695 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 816 op/s 2022-04-22T18:12:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:06 smithi121 bash[36902]: cluster 2022-04-22T18:12:05.141105+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e1226: 8 total, 8 up, 8 in 2022-04-22T18:12:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:06 smithi121 bash[36902]: audit 2022-04-22T18:12:05.696007+0000 mon.a (mon.0) 3450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:06 smithi121 bash[36902]: cluster 2022-04-22T18:12:06.124342+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2022-04-22T18:12:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[45495]: cluster 2022-04-22T18:12:05.095513+0000 mgr.y (mgr.34107) 1619 : cluster [DBG] pgmap v2327: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 695 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 816 op/s 2022-04-22T18:12:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[45495]: cluster 2022-04-22T18:12:05.141105+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e1226: 8 total, 8 up, 8 in 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[45495]: audit 2022-04-22T18:12:05.696007+0000 mon.a (mon.0) 3450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[45495]: cluster 2022-04-22T18:12:06.124342+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[46934]: cluster 2022-04-22T18:12:05.095513+0000 mgr.y (mgr.34107) 1619 : cluster [DBG] pgmap v2327: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 695 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 816 op/s 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[46934]: cluster 2022-04-22T18:12:05.141105+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e1226: 8 total, 8 up, 8 in 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[46934]: audit 2022-04-22T18:12:05.696007+0000 mon.a (mon.0) 3450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:06 smithi115 bash[46934]: cluster 2022-04-22T18:12:06.124342+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2022-04-22T18:12:07.200 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:07.200 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:07.210 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:07.210 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:07.210 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~35179724,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~13898264,341936085~10998133,357217629~10961045,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:07.217 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~118666500,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~14608559,341936085~10998133,356515840~11662834,373607831~31912051] 2022-04-22T18:12:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:07 smithi121 bash[36902]: audit 2022-04-22T18:12:06.152331+0000 mon.a (mon.0) 3452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:07.563 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:07 smithi115 bash[45495]: audit 2022-04-22T18:12:06.152331+0000 mon.a (mon.0) 3452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:07.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:07 smithi115 bash[46934]: audit 2022-04-22T18:12:06.152331+0000 mon.a (mon.0) 3452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:08.187 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:08.241 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [358204~12864787,13387288~33461278,48895278~12759917,62715066~18199824,83486776~36351027,122121191~142122,127473758~9834087,137917491~91474,138508106~22826719,161603998~23158599,188617691~875329,190798808~2972860,195069328~25732505,222060317~22216837,245411256~19451304,265938071~16788774,284555601~7142819,291909219~11471758,303824824~19709259,327866007~25068211,357217629~12473118,373607831~5003407,378967757~11495614,391705571~13814311] 2022-04-22T18:12:08.242 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~119837803,121634816~628497,125829120~12179845,138412032~105865122,244571947~78962136,327155712~25778506,356515840~13174907,373607831~31912051] 2022-04-22T18:12:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:08 smithi121 bash[36902]: cluster 2022-04-22T18:12:07.096610+0000 mgr.y (mgr.34107) 1620 : cluster [DBG] pgmap v2330: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 646 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 835 op/s 2022-04-22T18:12:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:08 smithi121 bash[36902]: cluster 2022-04-22T18:12:07.157296+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2022-04-22T18:12:08.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:08 smithi121 bash[36902]: audit 2022-04-22T18:12:08.083944+0000 mon.a (mon.0) 3454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[45495]: cluster 2022-04-22T18:12:07.096610+0000 mgr.y (mgr.34107) 1620 : cluster [DBG] pgmap v2330: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 646 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 835 op/s 2022-04-22T18:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[45495]: cluster 2022-04-22T18:12:07.157296+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2022-04-22T18:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[45495]: audit 2022-04-22T18:12:08.083944+0000 mon.a (mon.0) 3454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[46934]: cluster 2022-04-22T18:12:07.096610+0000 mgr.y (mgr.34107) 1620 : cluster [DBG] pgmap v2330: 137 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 133 active+clean; 646 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 835 op/s 2022-04-22T18:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[46934]: cluster 2022-04-22T18:12:07.157296+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2022-04-22T18:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:08 smithi115 bash[46934]: audit 2022-04-22T18:12:08.083944+0000 mon.a (mon.0) 3454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:09.223 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateStress (40036 ms) 2022-04-22T18:12:09.223 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateRegression6926 2022-04-22T18:12:09.225 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:09 smithi121 bash[36902]: cluster 2022-04-22T18:12:08.164509+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2022-04-22T18:12:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:09 smithi115 bash[45495]: cluster 2022-04-22T18:12:08.164509+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2022-04-22T18:12:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:09 smithi115 bash[46934]: cluster 2022-04-22T18:12:08.164509+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2022-04-22T18:12:10.206 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~256 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateRegression6926 (989 ms) 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParent 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateParent (0 ms) 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateIgnoreParent 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateIgnoreParent (0 ms) 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateCallbackError 2022-04-22T18:12:10.212 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:10.233 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [1950190~48680,2352984~59137,3022998~21729,7673058~66045,8739897~92547,11194331~72939,15199015~88928,16288610~58290,17602046~40591,18011350~59301] 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateCallbackError (30 ms) 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParentDiscard 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateParentDiscard (0 ms) 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnalignedSmall 2022-04-22T18:12:10.242 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:10.286 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 5000005~1234 2022-04-22T18:12:10.288 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnalignedSmall (47 ms) 2022-04-22T18:12:10.289 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnaligned 2022-04-22T18:12:10.289 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:10.368 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 8376263~12345 2022-04-22T18:12:10.369 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 8388608~4194304 2022-04-22T18:12:10.369 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 12582912~54321 2022-04-22T18:12:10.371 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnaligned (83 ms) 2022-04-22T18:12:10.371 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStriping 2022-04-22T18:12:10.372 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:12:10.372 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/0.DiffIterateStriping (0 ms) 2022-04-22T18:12:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:10 smithi121 bash[36902]: cluster 2022-04-22T18:12:09.097758+0000 mgr.y (mgr.34107) 1621 : cluster [DBG] pgmap v2333: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 546 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 471 op/s 2022-04-22T18:12:10.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:10 smithi121 bash[36902]: cluster 2022-04-22T18:12:09.182071+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2022-04-22T18:12:10.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:10 smithi115 bash[45495]: cluster 2022-04-22T18:12:09.097758+0000 mgr.y (mgr.34107) 1621 : cluster [DBG] pgmap v2333: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 546 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 471 op/s 2022-04-22T18:12:10.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:10 smithi115 bash[45495]: cluster 2022-04-22T18:12:09.182071+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2022-04-22T18:12:10.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:10 smithi115 bash[46934]: cluster 2022-04-22T18:12:09.097758+0000 mgr.y (mgr.34107) 1621 : cluster [DBG] pgmap v2333: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 546 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 471 op/s 2022-04-22T18:12:10.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:10 smithi115 bash[46934]: cluster 2022-04-22T18:12:09.182071+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2022-04-22T18:12:11.141 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from DiffIterateTest/0 (49211 ms total) 2022-04-22T18:12:11.141 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:12:11.141 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from DiffIterateTest/1, where TypeParam = DiffIterateParams 2022-04-22T18:12:11.178 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterate 2022-04-22T18:12:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:11 smithi121 bash[36902]: cluster 2022-04-22T18:12:10.188101+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2022-04-22T18:12:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:11 smithi121 bash[36902]: audit 2022-04-22T18:12:10.698650+0000 mon.a (mon.0) 3458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:11 smithi121 bash[36902]: cluster 2022-04-22T18:12:11.125860+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2022-04-22T18:12:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:11 smithi121 bash[36902]: audit 2022-04-22T18:12:11.148376+0000 mon.a (mon.0) 3460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[45495]: cluster 2022-04-22T18:12:10.188101+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2022-04-22T18:12:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[45495]: audit 2022-04-22T18:12:10.698650+0000 mon.a (mon.0) 3458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[45495]: cluster 2022-04-22T18:12:11.125860+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2022-04-22T18:12:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[45495]: audit 2022-04-22T18:12:11.148376+0000 mon.a (mon.0) 3460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[46934]: cluster 2022-04-22T18:12:10.188101+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2022-04-22T18:12:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[46934]: audit 2022-04-22T18:12:10.698650+0000 mon.a (mon.0) 3458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[46934]: cluster 2022-04-22T18:12:11.125860+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2022-04-22T18:12:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[46934]: audit 2022-04-22T18:12:11.148376+0000 mon.a (mon.0) 3460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:11.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:12 smithi121 bash[36902]: cluster 2022-04-22T18:12:11.098680+0000 mgr.y (mgr.34107) 1622 : cluster [DBG] pgmap v2336: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 483 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 290 op/s 2022-04-22T18:12:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:12 smithi115 bash[45495]: cluster 2022-04-22T18:12:11.098680+0000 mgr.y (mgr.34107) 1622 : cluster [DBG] pgmap v2336: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 483 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 290 op/s 2022-04-22T18:12:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:12 smithi115 bash[46934]: cluster 2022-04-22T18:12:11.098680+0000 mgr.y (mgr.34107) 1622 : cluster [DBG] pgmap v2336: 137 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 133 active+clean; 483 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 290 op/s 2022-04-22T18:12:13.235 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:13 smithi121 bash[36902]: cluster 2022-04-22T18:12:12.213658+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2022-04-22T18:12:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:13 smithi121 bash[36902]: audit 2022-04-22T18:12:12.215255+0000 mon.c (mon.1) 91 : audit [INF] from='client.? 172.21.15.115:0/2059020593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:13 smithi121 bash[36902]: audit 2022-04-22T18:12:12.216274+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:13 smithi121 bash[36902]: audit 2022-04-22T18:12:13.086702+0000 mon.a (mon.0) 3463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[45495]: cluster 2022-04-22T18:12:12.213658+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[45495]: audit 2022-04-22T18:12:12.215255+0000 mon.c (mon.1) 91 : audit [INF] from='client.? 172.21.15.115:0/2059020593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[45495]: audit 2022-04-22T18:12:12.216274+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[45495]: audit 2022-04-22T18:12:13.086702+0000 mon.a (mon.0) 3463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[46934]: cluster 2022-04-22T18:12:12.213658+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[46934]: audit 2022-04-22T18:12:12.215255+0000 mon.c (mon.1) 91 : audit [INF] from='client.? 172.21.15.115:0/2059020593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[46934]: audit 2022-04-22T18:12:12.216274+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:12:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[46934]: audit 2022-04-22T18:12:13.086702+0000 mon.a (mon.0) 3463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:13.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:14 smithi121 bash[36902]: cluster 2022-04-22T18:12:13.099727+0000 mgr.y (mgr.34107) 1623 : cluster [DBG] pgmap v2339: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:14 smithi121 bash[36902]: audit 2022-04-22T18:12:13.222393+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:12:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:14 smithi121 bash[36902]: cluster 2022-04-22T18:12:13.222716+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2022-04-22T18:12:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:14 smithi121 bash[36902]: audit 2022-04-22T18:12:13.843037+0000 mon.a (mon.0) 3466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:12:14.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:14 smithi121 bash[36902]: audit 2022-04-22T18:12:13.848770+0000 mon.a (mon.0) 3467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:12:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[45495]: cluster 2022-04-22T18:12:13.099727+0000 mgr.y (mgr.34107) 1623 : cluster [DBG] pgmap v2339: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[45495]: audit 2022-04-22T18:12:13.222393+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:12:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[45495]: cluster 2022-04-22T18:12:13.222716+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2022-04-22T18:12:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[45495]: audit 2022-04-22T18:12:13.843037+0000 mon.a (mon.0) 3466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:12:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[45495]: audit 2022-04-22T18:12:13.848770+0000 mon.a (mon.0) 3467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:12:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[46934]: cluster 2022-04-22T18:12:13.099727+0000 mgr.y (mgr.34107) 1623 : cluster [DBG] pgmap v2339: 137 pgs: 4 creating+peering, 28 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[46934]: audit 2022-04-22T18:12:13.222393+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-9","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:12:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[46934]: cluster 2022-04-22T18:12:13.222716+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2022-04-22T18:12:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[46934]: audit 2022-04-22T18:12:13.843037+0000 mon.a (mon.0) 3466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:12:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:14 smithi115 bash[46934]: audit 2022-04-22T18:12:13.848770+0000 mon.a (mon.0) 3467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:12:15.261 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [40023~10957,790607~8662,3887797~41687,5335012~62907,12350064~88656,13174679~97339,14614014~58262,14684765~28959,15141886~23932,17923220~26203] 2022-04-22T18:12:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:15 smithi121 bash[36902]: cluster 2022-04-22T18:12:14.229816+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2022-04-22T18:12:15.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:15 smithi121 bash[36902]: cluster 2022-04-22T18:12:15.228770+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2022-04-22T18:12:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:15 smithi115 bash[45495]: cluster 2022-04-22T18:12:14.229816+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2022-04-22T18:12:15.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:15 smithi115 bash[45495]: cluster 2022-04-22T18:12:15.228770+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2022-04-22T18:12:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:15 smithi115 bash[46934]: cluster 2022-04-22T18:12:14.229816+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2022-04-22T18:12:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:15 smithi115 bash[46934]: cluster 2022-04-22T18:12:15.228770+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2022-04-22T18:12:16.276 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [0~20971520] 2022-04-22T18:12:16.279 INFO:tasks.workunit.client.0.smithi115.stdout: diff was [0~20971520] 2022-04-22T18:12:16.281 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterate (5102 ms) 2022-04-22T18:12:16.281 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDiscard 2022-04-22T18:12:16.282 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:16.298 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~4194304 2022-04-22T18:12:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:16 smithi121 bash[36902]: cluster 2022-04-22T18:12:15.100631+0000 mgr.y (mgr.34107) 1624 : cluster [DBG] pgmap v2342: 137 pgs: 11 creating+peering, 8 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:16 smithi121 bash[36902]: audit 2022-04-22T18:12:15.702088+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:16 smithi121 bash[36902]: audit 2022-04-22T18:12:16.151271+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[45495]: cluster 2022-04-22T18:12:15.100631+0000 mgr.y (mgr.34107) 1624 : cluster [DBG] pgmap v2342: 137 pgs: 11 creating+peering, 8 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[45495]: audit 2022-04-22T18:12:15.702088+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[45495]: audit 2022-04-22T18:12:16.151271+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[46934]: cluster 2022-04-22T18:12:15.100631+0000 mgr.y (mgr.34107) 1624 : cluster [DBG] pgmap v2342: 137 pgs: 11 creating+peering, 8 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:12:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[46934]: audit 2022-04-22T18:12:15.702088+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:16 smithi115 bash[46934]: audit 2022-04-22T18:12:16.151271+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:17.284 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~4194304 2022-04-22T18:12:17.302 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:12:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:17 smithi121 bash[36902]: cluster 2022-04-22T18:12:16.255625+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2022-04-22T18:12:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:17 smithi115 bash[45495]: cluster 2022-04-22T18:12:16.255625+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2022-04-22T18:12:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:17 smithi115 bash[46934]: cluster 2022-04-22T18:12:16.255625+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2022-04-22T18:12:17.644 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:12:17.644 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (35m) 4m ago 49m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:12:17.644 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (48m) 5m ago 48m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:12:17.645 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (37m) 4m ago 50m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:12:17.645 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (36m) 5m ago 50m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:12:17.645 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (44m) 5m ago 54m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:12:17.645 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (45m) 4m ago 56m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:12:17.645 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (44m) 4m ago 56m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:12:17.646 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (43m) 5m ago 54m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:12:17.646 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (43m) 4m ago 54m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:12:17.646 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (36m) 4m ago 49m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:12:17.646 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (36m) 5m ago 49m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:12:17.646 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (42m) 4m ago 54m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (41m) 4m ago 53m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (40m) 4m ago 53m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (40m) 4m ago 52m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (39m) 5m ago 52m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (39m) 5m ago 51m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:12:17.647 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (38m) 5m ago 51m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:12:17.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (37m) 5m ago 50m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:12:17.648 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (36m) 5m ago 49m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:12:18.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:18.047 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:12:18.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:12:18.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:18.047 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:12:18.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:12:18.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:18.048 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:12:18.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:12:18.048 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:12:18.048 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:12:18.293 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~4194304 2022-04-22T18:12:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[45495]: cluster 2022-04-22T18:12:17.101297+0000 mgr.y (mgr.34107) 1625 : cluster [DBG] pgmap v2345: 137 pgs: 11 creating+peering, 126 active+clean; 8.0 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 38 KiB/s wr, 0 op/s 2022-04-22T18:12:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[45495]: cluster 2022-04-22T18:12:17.267327+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2022-04-22T18:12:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[45495]: audit 2022-04-22T18:12:17.283023+0000 mgr.y (mgr.34107) 1626 : audit [DBG] from='client.46786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[45495]: audit 2022-04-22T18:12:18.039730+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.115:0/2171002771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[45495]: audit 2022-04-22T18:12:18.089732+0000 mon.a (mon.0) 3474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[46934]: cluster 2022-04-22T18:12:17.101297+0000 mgr.y (mgr.34107) 1625 : cluster [DBG] pgmap v2345: 137 pgs: 11 creating+peering, 126 active+clean; 8.0 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 38 KiB/s wr, 0 op/s 2022-04-22T18:12:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[46934]: cluster 2022-04-22T18:12:17.267327+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2022-04-22T18:12:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[46934]: audit 2022-04-22T18:12:17.283023+0000 mgr.y (mgr.34107) 1626 : audit [DBG] from='client.46786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[46934]: audit 2022-04-22T18:12:18.039730+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.115:0/2171002771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:18 smithi115 bash[46934]: audit 2022-04-22T18:12:18.089732+0000 mon.a (mon.0) 3474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:18 smithi121 bash[36902]: cluster 2022-04-22T18:12:17.101297+0000 mgr.y (mgr.34107) 1625 : cluster [DBG] pgmap v2345: 137 pgs: 11 creating+peering, 126 active+clean; 8.0 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 38 KiB/s wr, 0 op/s 2022-04-22T18:12:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:18 smithi121 bash[36902]: cluster 2022-04-22T18:12:17.267327+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2022-04-22T18:12:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:18 smithi121 bash[36902]: audit 2022-04-22T18:12:17.283023+0000 mgr.y (mgr.34107) 1626 : audit [DBG] from='client.46786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:18 smithi121 bash[36902]: audit 2022-04-22T18:12:18.039730+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.115:0/2171002771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:18 smithi121 bash[36902]: audit 2022-04-22T18:12:18.089732+0000 mon.a (mon.0) 3474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:19.311 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~4194304 2022-04-22T18:12:19.312 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateDiscard (3032 ms) 2022-04-22T18:12:19.313 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStress 2022-04-22T18:12:19.313 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:19.363 INFO:tasks.workunit.client.0.smithi115.stdout: i=0 exists [11587921~4402029,16613655~5903897,37980807~6166171,75036797~5847340,133726068~4007938,181899924~260576,192108221~2903851,236713128~5547203,248515897~5509224,355062462~1037721] wrote [11587921~4402029,16613655~5903897,37980807~6166171,75036797~5847340,133726068~4007938,181899924~260576,192108221~2903851,236713128~5547203,248515897~5509224,355062462~1037721] 2022-04-22T18:12:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:19 smithi115 bash[45495]: audit 2022-04-22T18:12:17.620164+0000 mgr.y (mgr.34107) 1627 : audit [DBG] from='client.46792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:19 smithi115 bash[45495]: cluster 2022-04-22T18:12:18.274644+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2022-04-22T18:12:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:19 smithi115 bash[46934]: audit 2022-04-22T18:12:17.620164+0000 mgr.y (mgr.34107) 1627 : audit [DBG] from='client.46792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:19 smithi115 bash[46934]: cluster 2022-04-22T18:12:18.274644+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2022-04-22T18:12:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:19 smithi121 bash[36902]: audit 2022-04-22T18:12:17.620164+0000 mgr.y (mgr.34107) 1627 : audit [DBG] from='client.46792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:19.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:19 smithi121 bash[36902]: cluster 2022-04-22T18:12:18.274644+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2022-04-22T18:12:20.340 INFO:tasks.workunit.client.0.smithi115.stdout: i=1 exists [7271859~1719876,11587921~4402029,16613655~5903897,37980807~6166171,54525128~11701665,72220787~61817,75036797~5847340,87837226~4576052,128929450~8804556,162539959~1466436,170823811~161390,181899924~260576,192108221~2903851,204284047~3710223,236713128~10431734,248515897~5509224,355062462~1037721] wrote [7271859~1719876,54525128~11701665,72220787~61817,87837226~4576052,128929450~4960586,162539959~1466436,170823811~161390,204284047~3710223,242032303~5112559] 2022-04-22T18:12:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:20 smithi121 bash[36902]: cluster 2022-04-22T18:12:19.102279+0000 mgr.y (mgr.34107) 1628 : cluster [DBG] pgmap v2348: 137 pgs: 137 active+clean; 12 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 87 KiB/s wr, 14 op/s 2022-04-22T18:12:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:20 smithi121 bash[36902]: cluster 2022-04-22T18:12:19.294240+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2022-04-22T18:12:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:20 smithi115 bash[45495]: cluster 2022-04-22T18:12:19.102279+0000 mgr.y (mgr.34107) 1628 : cluster [DBG] pgmap v2348: 137 pgs: 137 active+clean; 12 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 87 KiB/s wr, 14 op/s 2022-04-22T18:12:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:20 smithi115 bash[45495]: cluster 2022-04-22T18:12:19.294240+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2022-04-22T18:12:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:20 smithi115 bash[46934]: cluster 2022-04-22T18:12:19.102279+0000 mgr.y (mgr.34107) 1628 : cluster [DBG] pgmap v2348: 137 pgs: 137 active+clean; 12 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 87 KiB/s wr, 14 op/s 2022-04-22T18:12:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:20 smithi115 bash[46934]: cluster 2022-04-22T18:12:19.294240+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2022-04-22T18:12:21.376 INFO:tasks.workunit.client.0.smithi115.stdout: i=2 exists [7271859~1719876,11587921~4402029,16613655~5903897,37980807~6166171,44558221~2809652,54525128~11701665,72220787~61817,75036797~5847340,87837226~4576052,100837533~894476,128929450~8804556,162539959~6193368,170823811~161390,181899924~260576,187651240~9021621,204284047~3710223,216733059~5572241,236713128~10431734,248515897~11186423,277708025~8166997,297649932~8133505,355062462~1037721,359073209~4595464] wrote [44558221~2809652,100837533~894476,163181900~5551427,187651240~9021621,216733059~5572241,251920030~7782290,277708025~8166997,297649932~8133505,359073209~4595464] 2022-04-22T18:12:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:21 smithi121 bash[36902]: cluster 2022-04-22T18:12:20.300787+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2022-04-22T18:12:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:21 smithi121 bash[36902]: audit 2022-04-22T18:12:20.705353+0000 mon.a (mon.0) 3478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:21 smithi121 bash[36902]: audit 2022-04-22T18:12:21.154969+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:21.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[45495]: cluster 2022-04-22T18:12:20.300787+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2022-04-22T18:12:21.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[45495]: audit 2022-04-22T18:12:20.705353+0000 mon.a (mon.0) 3478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:21.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[45495]: audit 2022-04-22T18:12:21.154969+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:21.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[46934]: cluster 2022-04-22T18:12:20.300787+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2022-04-22T18:12:21.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[46934]: audit 2022-04-22T18:12:20.705353+0000 mon.a (mon.0) 3478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:21.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[46934]: audit 2022-04-22T18:12:21.154969+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:21.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:22.389 INFO:tasks.workunit.client.0.smithi115.stdout: i=3 exists [7271859~1719876,9770533~6219417,16613655~5903897,24547202~1122964,37980807~9387066,48578067~5644097,54525128~11701665,70622547~720316,72220787~61817,75036797~5847340,87837226~4576052,96461900~7379890,128929450~8804556,162539959~6193368,170823811~161390,181899924~260576,187651240~9021621,204284047~3710223,216733059~5572241,236713128~10431734,248515897~11186423,272605505~13269517,297649932~8133505,346824863~11127399,359073209~4595464,370569711~6238729] wrote [9770533~5673665,24547202~1122964,43898979~2641262,48578067~5644097,70622547~720316,96461900~7379890,272605505~7484154,346824863~11127399,370569711~6238729] 2022-04-22T18:12:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:22 smithi121 bash[36902]: cluster 2022-04-22T18:12:21.103297+0000 mgr.y (mgr.34107) 1629 : cluster [DBG] pgmap v2351: 137 pgs: 137 active+clean; 55 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 21 op/s 2022-04-22T18:12:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:22 smithi121 bash[36902]: cluster 2022-04-22T18:12:21.318458+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2022-04-22T18:12:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:22 smithi115 bash[45495]: cluster 2022-04-22T18:12:21.103297+0000 mgr.y (mgr.34107) 1629 : cluster [DBG] pgmap v2351: 137 pgs: 137 active+clean; 55 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 21 op/s 2022-04-22T18:12:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:22 smithi115 bash[45495]: cluster 2022-04-22T18:12:21.318458+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2022-04-22T18:12:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:22 smithi115 bash[46934]: cluster 2022-04-22T18:12:21.103297+0000 mgr.y (mgr.34107) 1629 : cluster [DBG] pgmap v2351: 137 pgs: 137 active+clean; 55 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 21 op/s 2022-04-22T18:12:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:22 smithi115 bash[46934]: cluster 2022-04-22T18:12:21.318458+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2022-04-22T18:12:23.389 INFO:tasks.workunit.client.0.smithi115.stdout: i=4 exists [7271859~1719876,9770533~6219417,16613655~5903897,24547202~1122964,28144306~1126120,37980807~9387066,48578067~5644097,54525128~11701665,69134149~18590537,87837226~4576052,96461900~7379890,128929450~8804556,140479330~3829374,162539959~6193368,170823811~161390,181899924~260576,187651240~9021621,204284047~3710223,216733059~5572241,236713128~10431734,248515897~11186423,272605505~13269517,297649932~8133505,310164001~907360,318580171~2530324,346824863~11362933,359073209~4595464,370569711~6238729,383307066~2393898] wrote [28144306~1126120,69134149~6862794,79602393~8122293,140479330~3829374,252864104~6645568,310164001~907360,318580171~2530324,356943427~1244369,383307066~2393898] 2022-04-22T18:12:23.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:23 smithi115 bash[45495]: cluster 2022-04-22T18:12:22.330631+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2022-04-22T18:12:23.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:23 smithi115 bash[45495]: audit 2022-04-22T18:12:23.092664+0000 mon.a (mon.0) 3482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:23.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:23 smithi115 bash[46934]: cluster 2022-04-22T18:12:22.330631+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2022-04-22T18:12:23.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:23 smithi115 bash[46934]: audit 2022-04-22T18:12:23.092664+0000 mon.a (mon.0) 3482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:23 smithi121 bash[36902]: cluster 2022-04-22T18:12:22.330631+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2022-04-22T18:12:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:23 smithi121 bash[36902]: audit 2022-04-22T18:12:23.092664+0000 mon.a (mon.0) 3482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:24.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:23.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:24.408 INFO:tasks.workunit.client.0.smithi115.stdout: i=5 exists [537867~93432,7271859~1719876,9770533~6219417,16613655~5903897,23660168~4323299,28144306~1126120,37980807~9387066,48578067~5644097,54525128~11701665,69134149~23306671,96461900~7379890,128929450~8804556,140479330~9394804,162539959~6193368,170823811~161390,172955940~478166,181899924~260576,187651240~9021621,204284047~8630210,216733059~5572241,236713128~10431734,248515897~11186423,272605505~13269517,294639898~11143539,310164001~907360,318580171~2530324,346824863~11362933,359073209~4595464,370569711~6238729,383307066~2393898,394507543~7718500] wrote [537867~93432,23660168~4323299,85493230~6947590,142144166~7729968,172955940~478166,205657165~7257092,294639898~4646250,350381482~6555859,394507543~7718500] 2022-04-22T18:12:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:24 smithi121 bash[36902]: cluster 2022-04-22T18:12:23.104296+0000 mgr.y (mgr.34107) 1630 : cluster [DBG] pgmap v2354: 137 pgs: 137 active+clean; 90 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 14 MiB/s wr, 12 op/s 2022-04-22T18:12:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:24 smithi121 bash[36902]: cluster 2022-04-22T18:12:23.341271+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2022-04-22T18:12:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:24 smithi115 bash[45495]: cluster 2022-04-22T18:12:23.104296+0000 mgr.y (mgr.34107) 1630 : cluster [DBG] pgmap v2354: 137 pgs: 137 active+clean; 90 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 14 MiB/s wr, 12 op/s 2022-04-22T18:12:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:24 smithi115 bash[45495]: cluster 2022-04-22T18:12:23.341271+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2022-04-22T18:12:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:24 smithi115 bash[46934]: cluster 2022-04-22T18:12:23.104296+0000 mgr.y (mgr.34107) 1630 : cluster [DBG] pgmap v2354: 137 pgs: 137 active+clean; 90 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 14 MiB/s wr, 12 op/s 2022-04-22T18:12:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:24 smithi115 bash[46934]: cluster 2022-04-22T18:12:23.341271+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2022-04-22T18:12:25.389 INFO:tasks.workunit.client.0.smithi115.stdout: i=6 exists [537867~93432,7271859~1719876,9770533~6219417,16613655~5903897,23660168~4323299,28144306~1126120,37980807~9387066,48578067~5644097,54525128~11701665,69134149~23306671,96461900~7379890,128929450~8804556,140479330~9394804,162539959~6193368,170823811~161390,172955940~478166,181899924~260576,187651240~9021621,204284047~8630210,216733059~8398867,236713128~22989192,272605505~13269517,294639898~11143539,310164001~907360,318580171~2530324,331108357~2056746,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [45708349~391858,220330148~4801778,240269343~618809,246039340~4179696,331108357~2056746,354761607~4514188,370674568~4123474,391749451~2710677,407736278~6276381] 2022-04-22T18:12:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:25 smithi121 bash[36902]: cluster 2022-04-22T18:12:24.348959+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2022-04-22T18:12:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:25 smithi121 bash[36902]: audit 2022-04-22T18:12:24.492292+0000 mon.a (mon.0) 3485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:25 smithi115 bash[45495]: cluster 2022-04-22T18:12:24.348959+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2022-04-22T18:12:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:25 smithi115 bash[45495]: audit 2022-04-22T18:12:24.492292+0000 mon.a (mon.0) 3485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:25 smithi115 bash[46934]: cluster 2022-04-22T18:12:24.348959+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2022-04-22T18:12:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:25 smithi115 bash[46934]: audit 2022-04-22T18:12:24.492292+0000 mon.a (mon.0) 3485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:26.174 INFO:tasks.workunit.client.0.smithi115.stdout: i=7 exists [537867~93432,7271859~1719876,9770533~12747019,23660168~5610258,37980807~9387066,48578067~5644097,54525128~11701665,69134149~23306671,96461900~7379890,128929450~11239628,140479330~9394804,156031815~1650278,162539959~6193368,170823811~161390,172955940~478166,176265934~2934823,181899924~260576,187651240~9021621,204284047~8630210,216733059~8398867,236713128~22989192,272605505~13269517,290426012~1314510,294639898~13890871,310164001~907360,318580171~2530324,331108357~2056746,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [11926141~6348492,24626590~4407496,135368469~4800609,156031815~1650278,176265934~2934823,279295471~2009420,290426012~1314510,304017493~4513276,350395636~3435297,358115239~267541] 2022-04-22T18:12:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:26 smithi121 bash[36902]: cluster 2022-04-22T18:12:25.105310+0000 mgr.y (mgr.34107) 1631 : cluster [DBG] pgmap v2357: 137 pgs: 137 active+clean; 260 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 36 MiB/s wr, 32 op/s 2022-04-22T18:12:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:26 smithi121 bash[36902]: cluster 2022-04-22T18:12:25.357572+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2022-04-22T18:12:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:26 smithi121 bash[36902]: audit 2022-04-22T18:12:25.708937+0000 mon.a (mon.0) 3487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:26 smithi121 bash[36902]: cluster 2022-04-22T18:12:26.132632+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2022-04-22T18:12:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:26 smithi121 bash[36902]: audit 2022-04-22T18:12:26.158578+0000 mon.a (mon.0) 3489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[45495]: cluster 2022-04-22T18:12:25.105310+0000 mgr.y (mgr.34107) 1631 : cluster [DBG] pgmap v2357: 137 pgs: 137 active+clean; 260 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 36 MiB/s wr, 32 op/s 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[45495]: cluster 2022-04-22T18:12:25.357572+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[45495]: audit 2022-04-22T18:12:25.708937+0000 mon.a (mon.0) 3487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[45495]: cluster 2022-04-22T18:12:26.132632+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[45495]: audit 2022-04-22T18:12:26.158578+0000 mon.a (mon.0) 3489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[46934]: cluster 2022-04-22T18:12:25.105310+0000 mgr.y (mgr.34107) 1631 : cluster [DBG] pgmap v2357: 137 pgs: 137 active+clean; 260 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 36 MiB/s wr, 32 op/s 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[46934]: cluster 2022-04-22T18:12:25.357572+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2022-04-22T18:12:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[46934]: audit 2022-04-22T18:12:25.708937+0000 mon.a (mon.0) 3487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[46934]: cluster 2022-04-22T18:12:26.132632+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2022-04-22T18:12:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:26 smithi115 bash[46934]: audit 2022-04-22T18:12:26.158578+0000 mon.a (mon.0) 3489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:27.190 INFO:tasks.workunit.client.0.smithi115.stdout: i=8 exists [537867~93432,7271859~1719876,9770533~12747019,23660168~7437674,37980807~9387066,48578067~17648726,69134149~23306671,96461900~7379890,104506942~2679938,128929450~11239628,140479330~9394804,156031815~1650278,162539959~6193368,170823811~161390,172955940~478166,176265934~2934823,181899924~260576,187651240~9021621,204284047~8630210,216733059~8398867,232387454~2200988,236713128~22989192,268055099~17819923,290426012~1314510,294639898~13890871,310164001~907360,318580171~6930309,331108357~2056746,339378411~3205171,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [27400236~3697606,41005069~3866380,54030662~2380657,104506942~2679938,232387454~2200988,268055099~4891853,319057984~6452496,339378411~3205171] 2022-04-22T18:12:28.190 INFO:tasks.workunit.client.0.smithi115.stdout: i=9 exists [537867~93432,7271859~1719876,9770533~12747019,23660168~7437674,37980807~9387066,48578067~17648726,69134149~23306671,96461900~7379890,104506942~2679938,117910125~3229759,126331721~13837357,140479330~9394804,156031815~1650278,162539959~6193368,170127728~4595184,176265934~2934823,181899924~260576,187651240~9021621,204284047~8630210,216733059~8398867,230478800~4109642,236713128~22989192,268055099~17819923,290426012~1314510,294639898~13890871,309823923~3575102,318580171~6930309,331108357~2056746,339378411~3205171,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [44149506~3210250,49245423~1469163,117910125~3229759,126331721~4032080,133684696~5153155,170127728~4595184,192248466~3591435,230478800~3551223,309823923~3575102,359537914~1203910] 2022-04-22T18:12:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:28 smithi121 bash[36902]: cluster 2022-04-22T18:12:27.106379+0000 mgr.y (mgr.34107) 1632 : cluster [DBG] pgmap v2360: 137 pgs: 137 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 42 MiB/s wr, 35 op/s 2022-04-22T18:12:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:28 smithi121 bash[36902]: cluster 2022-04-22T18:12:27.143840+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2022-04-22T18:12:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:28 smithi121 bash[36902]: audit 2022-04-22T18:12:28.095531+0000 mon.a (mon.0) 3491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[45495]: cluster 2022-04-22T18:12:27.106379+0000 mgr.y (mgr.34107) 1632 : cluster [DBG] pgmap v2360: 137 pgs: 137 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 42 MiB/s wr, 35 op/s 2022-04-22T18:12:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[45495]: cluster 2022-04-22T18:12:27.143840+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2022-04-22T18:12:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[45495]: audit 2022-04-22T18:12:28.095531+0000 mon.a (mon.0) 3491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[46934]: cluster 2022-04-22T18:12:27.106379+0000 mgr.y (mgr.34107) 1632 : cluster [DBG] pgmap v2360: 137 pgs: 137 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 42 MiB/s wr, 35 op/s 2022-04-22T18:12:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[46934]: cluster 2022-04-22T18:12:27.143840+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2022-04-22T18:12:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:28 smithi115 bash[46934]: audit 2022-04-22T18:12:28.095531+0000 mon.a (mon.0) 3491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:29.204 INFO:tasks.workunit.client.0.smithi115.stdout: i=10 exists [537867~93432,7271859~15245693,23660168~7437674,37980807~9387066,48578067~17648726,69134149~23306671,96461900~7379890,104506942~2679938,109303801~2548682,117910125~3229759,126331721~13837357,140479330~9394804,156031815~1650278,160579830~8153497,170127728~4595184,176265934~2934823,181899924~260576,187651240~9021621,199840504~733999,204284047~8630210,216733059~8398867,230478800~29223520,268055099~17819923,290426012~1314510,294639898~13890871,309823923~3575102,318580171~6930309,331108357~2056746,339378411~3205171,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [7753393~2096457,60195734~1600357,109303801~2548682,129593787~3949462,160579830~7928318,199840504~733999,204311591~7405453,233007413~7963961,357899004~3210611,409686593~1721582] 2022-04-22T18:12:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:29 smithi121 bash[36902]: cluster 2022-04-22T18:12:28.146638+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2022-04-22T18:12:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:29 smithi115 bash[45495]: cluster 2022-04-22T18:12:28.146638+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2022-04-22T18:12:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:29 smithi115 bash[46934]: cluster 2022-04-22T18:12:28.146638+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2022-04-22T18:12:30.224 INFO:tasks.workunit.client.0.smithi115.stdout: i=11 exists [537867~93432,7271859~15245693,23660168~7555865,37980807~9387066,48578067~17648726,69134149~23306671,96461900~16044792,117910125~3229759,126257490~13911588,140479330~9394804,156031815~1650278,160579830~8153497,170127728~4595184,176265934~2934823,181899924~260576,187651240~9021621,199840504~733999,204284047~8630210,216733059~8398867,230478800~29223520,268055099~17819923,290426012~1314510,294639898~13890871,309823923~3575102,318580171~6930309,331108357~2056746,339378411~3205171,346824863~16843810,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [29383118~1832915,69259317~2205003,101226047~11280645,126257490~6037525,235709652~831810,250602604~5546445,303791829~1351743,350164616~4105806] 2022-04-22T18:12:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:30 smithi121 bash[36902]: cluster 2022-04-22T18:12:29.107523+0000 mgr.y (mgr.34107) 1633 : cluster [DBG] pgmap v2363: 137 pgs: 137 active+clean; 402 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 16 op/s 2022-04-22T18:12:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:30 smithi121 bash[36902]: cluster 2022-04-22T18:12:29.163934+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2022-04-22T18:12:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:30 smithi115 bash[45495]: cluster 2022-04-22T18:12:29.107523+0000 mgr.y (mgr.34107) 1633 : cluster [DBG] pgmap v2363: 137 pgs: 137 active+clean; 402 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 16 op/s 2022-04-22T18:12:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:30 smithi115 bash[45495]: cluster 2022-04-22T18:12:29.163934+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2022-04-22T18:12:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:30 smithi115 bash[46934]: cluster 2022-04-22T18:12:29.107523+0000 mgr.y (mgr.34107) 1633 : cluster [DBG] pgmap v2363: 137 pgs: 137 active+clean; 402 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 16 op/s 2022-04-22T18:12:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:30 smithi115 bash[46934]: cluster 2022-04-22T18:12:29.163934+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2022-04-22T18:12:31.248 INFO:tasks.workunit.client.0.smithi115.stdout: i=12 exists [537867~93432,7271859~15245693,23660168~7555865,37980807~9387066,48578067~17648726,69134149~23306671,96461900~16044792,117910125~3229759,126257490~13911588,140479330~9394804,156031815~1650278,160579830~14143082,176265934~2934823,181899924~260576,185116367~11556494,199840504~733999,204284047~8630210,216733059~8398867,230478800~29223520,268055099~17819923,290426012~1314510,294639898~13890871,309624073~6012010,318580171~6930309,331108357~2056746,339378411~3205171,346824863~16843810,363703729~6435264,370569711~6238729,383307066~2393898,391749451~2710677,394507543~7718500,407736278~6276381] wrote [15925586~4912519,52460540~2953439,61988179~970926,83382739~8051649,168086677~3293774,185116367~3788623,235663232~3821615,309624073~6012010,363703729~6435264] 2022-04-22T18:12:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:31 smithi121 bash[36902]: cluster 2022-04-22T18:12:30.172427+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2022-04-22T18:12:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:31 smithi121 bash[36902]: audit 2022-04-22T18:12:30.712110+0000 mon.a (mon.0) 3495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:31 smithi121 bash[36902]: audit 2022-04-22T18:12:31.165213+0000 mon.a (mon.0) 3496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[46934]: cluster 2022-04-22T18:12:30.172427+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2022-04-22T18:12:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[46934]: audit 2022-04-22T18:12:30.712110+0000 mon.a (mon.0) 3495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[46934]: audit 2022-04-22T18:12:31.165213+0000 mon.a (mon.0) 3496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[45495]: cluster 2022-04-22T18:12:30.172427+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2022-04-22T18:12:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[45495]: audit 2022-04-22T18:12:30.712110+0000 mon.a (mon.0) 3495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[45495]: audit 2022-04-22T18:12:31.165213+0000 mon.a (mon.0) 3496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:31.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:32.253 INFO:tasks.workunit.client.0.smithi115.stdout: i=13 exists [537867~93432,1951085~643036,7271859~15245693,23660168~7555865,37980807~9387066,48578067~17648726,69134149~23306671,96461900~16044792,117910125~3229759,126257490~13911588,140479330~9394804,156031815~1650278,160579830~14143082,176265934~2934823,181899924~260576,185116367~11782390,197353926~945089,199840504~733999,202861279~10052978,216733059~8398867,230478800~29223520,267735759~18139263,290426012~1314510,294639898~23502558,318580171~14584932,339378411~3205171,346824863~16843810,363703729~6435264,370569711~6238729,382902053~2821392,391749451~2710677,394507543~7718500,407736278~6276381] wrote [1951085~643036,190297638~6601119,197353926~945089,202861279~4522598,267735759~7819087,303999122~6576895,310840245~7302211,320687961~10690611,382902053~2821392] 2022-04-22T18:12:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:32 smithi121 bash[36902]: cluster 2022-04-22T18:12:31.108410+0000 mgr.y (mgr.34107) 1634 : cluster [DBG] pgmap v2366: 137 pgs: 137 active+clean; 497 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 42 MiB/s wr, 31 op/s 2022-04-22T18:12:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:32 smithi121 bash[36902]: cluster 2022-04-22T18:12:31.183893+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2022-04-22T18:12:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:32 smithi115 bash[45495]: cluster 2022-04-22T18:12:31.108410+0000 mgr.y (mgr.34107) 1634 : cluster [DBG] pgmap v2366: 137 pgs: 137 active+clean; 497 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 42 MiB/s wr, 31 op/s 2022-04-22T18:12:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:32 smithi115 bash[45495]: cluster 2022-04-22T18:12:31.183893+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2022-04-22T18:12:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:32 smithi115 bash[46934]: cluster 2022-04-22T18:12:31.108410+0000 mgr.y (mgr.34107) 1634 : cluster [DBG] pgmap v2366: 137 pgs: 137 active+clean; 497 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 42 MiB/s wr, 31 op/s 2022-04-22T18:12:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:32 smithi115 bash[46934]: cluster 2022-04-22T18:12:31.183893+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2022-04-22T18:12:33.252 INFO:tasks.workunit.client.0.smithi115.stdout: i=14 exists [537867~93432,1951085~643036,7271859~15245693,23660168~11204350,36527329~10840544,48578067~17648726,69134149~23306671,96461900~16044792,117910125~3229759,126257490~13911588,140479330~9394804,151168394~973029,156031815~1650278,160579830~14143082,176265934~2934823,181899924~260576,183141131~13757626,197353926~945089,199840504~733999,202861279~10052978,216733059~11877040,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,339378411~3205171,346824863~16843810,363703729~6435264,370569711~6238729,382902053~2821392,391749451~2710677,394507543~7718500,407736278~6276381] wrote [28124440~6740078,36527329~7999848,132653637~5753066,151168394~973029,162975759~2983484,183141131~2261286,222528134~6081965,246428521~1043816,260584465~7656673,276741098~1637647] 2022-04-22T18:12:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:33 smithi121 bash[36902]: cluster 2022-04-22T18:12:32.191148+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2022-04-22T18:12:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:33 smithi121 bash[36902]: audit 2022-04-22T18:12:33.098381+0000 mon.a (mon.0) 3499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:33 smithi115 bash[45495]: cluster 2022-04-22T18:12:32.191148+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2022-04-22T18:12:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:33 smithi115 bash[45495]: audit 2022-04-22T18:12:33.098381+0000 mon.a (mon.0) 3499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:33 smithi115 bash[46934]: cluster 2022-04-22T18:12:32.191148+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2022-04-22T18:12:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:33 smithi115 bash[46934]: audit 2022-04-22T18:12:33.098381+0000 mon.a (mon.0) 3499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:34.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:33.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:34.264 INFO:tasks.workunit.client.0.smithi115.stdout: i=15 exists [537867~93432,1951085~643036,7271859~27592659,36527329~10840544,48578067~17648726,69134149~23306671,96461900~16044792,117910125~3229759,126257490~13911588,140479330~9394804,151168394~973029,156031815~1650278,160579830~14530061,176265934~5440436,181899924~260576,183141131~13757626,197353926~945089,199840504~733999,202861279~10052978,216733059~11877040,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,339378411~3205171,346824863~16843810,363703729~6435264,370569711~15153734,391749451~2710677,394507543~7718500,407736278~6276381] wrote [8869241~4097104,22322704~6149628,89515870~2798162,106586364~2335962,169411010~5698881,177755668~3950702,244387008~5241594,297252051~2039024,376323253~7640449] 2022-04-22T18:12:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:34 smithi121 bash[36902]: cluster 2022-04-22T18:12:33.109263+0000 mgr.y (mgr.34107) 1635 : cluster [DBG] pgmap v2369: 137 pgs: 137 active+clean; 526 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 28 MiB/s wr, 26 op/s 2022-04-22T18:12:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:34 smithi121 bash[36902]: cluster 2022-04-22T18:12:33.203777+0000 mon.a (mon.0) 3500 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2022-04-22T18:12:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:34 smithi115 bash[45495]: cluster 2022-04-22T18:12:33.109263+0000 mgr.y (mgr.34107) 1635 : cluster [DBG] pgmap v2369: 137 pgs: 137 active+clean; 526 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 28 MiB/s wr, 26 op/s 2022-04-22T18:12:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:34 smithi115 bash[45495]: cluster 2022-04-22T18:12:33.203777+0000 mon.a (mon.0) 3500 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2022-04-22T18:12:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:34 smithi115 bash[46934]: cluster 2022-04-22T18:12:33.109263+0000 mgr.y (mgr.34107) 1635 : cluster [DBG] pgmap v2369: 137 pgs: 137 active+clean; 526 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 28 MiB/s wr, 26 op/s 2022-04-22T18:12:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:34 smithi115 bash[46934]: cluster 2022-04-22T18:12:33.203777+0000 mon.a (mon.0) 3500 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2022-04-22T18:12:35.260 INFO:tasks.workunit.client.0.smithi115.stdout: i=16 exists [537867~93432,1951085~643036,7271859~28423971,36527329~10840544,48578067~17648726,69134149~23306671,93229441~19277251,117910125~3229759,126257490~13911588,140479330~9394804,151168394~973029,156031815~1650278,160579830~14530061,176265934~5440436,181899924~260576,183141131~13757626,197353926~945089,199840504~733999,202861279~10052978,216733059~11877040,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,339378411~3205171,346824863~16843810,363703729~6435264,370569711~15153734,391749451~2710677,394507543~7718500,407736278~6276381] wrote [14674725~693941,34006797~1689033,42590738~2162114,59583537~3731923,93229441~6703124,164393032~4533752,234210977~660864,280525622~1571063,300653496~2697183,376493018~7539112] 2022-04-22T18:12:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:35 smithi121 bash[36902]: cluster 2022-04-22T18:12:34.209467+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2022-04-22T18:12:35.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:35 smithi115 bash[45495]: cluster 2022-04-22T18:12:34.209467+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2022-04-22T18:12:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:35 smithi115 bash[46934]: cluster 2022-04-22T18:12:34.209467+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2022-04-22T18:12:36.278 INFO:tasks.workunit.client.0.smithi115.stdout: i=17 exists [537867~93432,1951085~643036,4776580~30919250,36527329~10840544,48578067~17648726,69134149~23306671,93229441~19277251,117910125~3229759,126257490~13911588,140479330~11662093,156031815~1650278,160579830~14530061,176265934~5440436,181899924~260576,183141131~13757626,197353926~945089,199840504~733999,202861279~10052978,216733059~12940948,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,335923937~6659645,346824863~16843810,363703729~6435264,370569711~15153734,391749451~2710677,394507543~7718500,407736278~6276381] wrote [4776580~2525953,54250269~500086,101693003~2068871,148635147~2542055,225170315~4503692,250045696~7998601,277052988~2491159,320236191~7513247,335923937~4437411] 2022-04-22T18:12:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:36 smithi121 bash[36902]: cluster 2022-04-22T18:12:35.110282+0000 mgr.y (mgr.34107) 1636 : cluster [DBG] pgmap v2372: 137 pgs: 137 active+clean; 657 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 28 op/s 2022-04-22T18:12:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:36 smithi121 bash[36902]: cluster 2022-04-22T18:12:35.218400+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2022-04-22T18:12:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:36 smithi121 bash[36902]: audit 2022-04-22T18:12:35.715133+0000 mon.a (mon.0) 3503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:36 smithi121 bash[36902]: audit 2022-04-22T18:12:36.168815+0000 mon.a (mon.0) 3504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[45495]: cluster 2022-04-22T18:12:35.110282+0000 mgr.y (mgr.34107) 1636 : cluster [DBG] pgmap v2372: 137 pgs: 137 active+clean; 657 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 28 op/s 2022-04-22T18:12:36.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[45495]: cluster 2022-04-22T18:12:35.218400+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2022-04-22T18:12:36.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[45495]: audit 2022-04-22T18:12:35.715133+0000 mon.a (mon.0) 3503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:36.595 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[45495]: audit 2022-04-22T18:12:36.168815+0000 mon.a (mon.0) 3504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[46934]: cluster 2022-04-22T18:12:35.110282+0000 mgr.y (mgr.34107) 1636 : cluster [DBG] pgmap v2372: 137 pgs: 137 active+clean; 657 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 39 MiB/s wr, 28 op/s 2022-04-22T18:12:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[46934]: cluster 2022-04-22T18:12:35.218400+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2022-04-22T18:12:36.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[46934]: audit 2022-04-22T18:12:35.715133+0000 mon.a (mon.0) 3503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:36.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:36 smithi115 bash[46934]: audit 2022-04-22T18:12:36.168815+0000 mon.a (mon.0) 3504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:37.286 INFO:tasks.workunit.client.0.smithi115.stdout: i=18 exists [537867~93432,1951085~643036,4776580~30919250,36527329~10840544,48578067~17648726,68808681~23632139,93229441~19277251,117910125~3229759,126257490~13911588,140216272~15469183,156031815~1650278,159548006~823527,160579830~14530061,176265934~5440436,181899924~260576,183141131~13757626,197353926~945089,199840504~733999,202861279~10052978,215447184~14226823,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,335923937~6659645,346824863~16843810,363703729~6435264,370569711~15153734,391749451~2710677,394507543~7718500,407736278~6276381] wrote [4835663~545904,25453435~6548989,53553206~6383326,68808681~7231765,140216272~6209350,148199599~7485856,159548006~823527,215447184~4905087] 2022-04-22T18:12:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:37 smithi121 bash[36902]: cluster 2022-04-22T18:12:36.230587+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2022-04-22T18:12:37.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:37 smithi115 bash[45495]: cluster 2022-04-22T18:12:36.230587+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2022-04-22T18:12:37.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:37 smithi115 bash[46934]: cluster 2022-04-22T18:12:36.230587+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2022-04-22T18:12:38.293 INFO:tasks.workunit.client.0.smithi115.stdout: i=19 exists [537867~93432,1951085~643036,4776580~30919250,36527329~10840544,48578067~43862753,93229441~19277251,117910125~3229759,126257490~13911588,140216272~15469183,156031815~1650278,159548006~823527,160579830~14530061,176265934~20632823,197353926~945089,199840504~733999,202861279~10052978,213625896~16048111,230478800~29223520,260584465~25290557,290426012~1314510,294639898~23502558,318580171~14584932,335923937~6659645,346824863~23314130,370569711~15153734,391749451~2710677,394507543~7718500,407736278~6276381] wrote [49749071~1275726,57458664~1111690,65516556~5446500,74035482~2180239,176780953~7975801,213625896~6361002,314516215~3383505,348905001~686707,353802493~11422508] 2022-04-22T18:12:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:38 smithi121 bash[36902]: cluster 2022-04-22T18:12:37.111298+0000 mgr.y (mgr.34107) 1637 : cluster [DBG] pgmap v2375: 137 pgs: 137 active+clean; 717 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 32 op/s 2022-04-22T18:12:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:38 smithi121 bash[36902]: cluster 2022-04-22T18:12:37.239615+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2022-04-22T18:12:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:38 smithi121 bash[36902]: audit 2022-04-22T18:12:38.101433+0000 mon.a (mon.0) 3507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[45495]: cluster 2022-04-22T18:12:37.111298+0000 mgr.y (mgr.34107) 1637 : cluster [DBG] pgmap v2375: 137 pgs: 137 active+clean; 717 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 32 op/s 2022-04-22T18:12:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[45495]: cluster 2022-04-22T18:12:37.239615+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2022-04-22T18:12:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[45495]: audit 2022-04-22T18:12:38.101433+0000 mon.a (mon.0) 3507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[46934]: cluster 2022-04-22T18:12:37.111298+0000 mgr.y (mgr.34107) 1637 : cluster [DBG] pgmap v2375: 137 pgs: 137 active+clean; 717 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 32 op/s 2022-04-22T18:12:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[46934]: cluster 2022-04-22T18:12:37.239615+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2022-04-22T18:12:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:38 smithi115 bash[46934]: audit 2022-04-22T18:12:38.101433+0000 mon.a (mon.0) 3507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:39.269 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 1 diff [4194304~4194304,50331648~12582912,71303168~4194304,83886080~8388608,125829120~8388608,159383552~4194304,167772160~4194304,201326592~4194304,239075328~8388608] 2022-04-22T18:12:39.277 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,50331648~12582912,71303168~4194304,83886080~8388608,125829120~8388608,159383552~4194304,167772160~4194304,201326592~4194304,239075328~8388608] 2022-04-22T18:12:39.287 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~8388608,22517552~2648272,44146978~1990366,50331648~16777216,71303168~4194304,80884137~15584855,125829120~8388608,137734006~678026,159383552~12582912,182160500~2388876,195012072~2120216,201326592~8388608,239075328~8388608,254025121~1827423,356100183~415657] 2022-04-22T18:12:39.288 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 2 diff [4194304~4194304,41943040~4194304,50331648~12582912,71303168~4194304,83886080~8388608,100663296~4194304,125829120~8388608,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,276824064~8388608,293601280~8388608,356515840~8388608] 2022-04-22T18:12:39.288 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,41943040~4194304,50331648~12582912,71303168~4194304,83886080~8388608,100663296~4194304,125829120~8388608,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,276824064~8388608,293601280~8388608,356515840~8388608] 2022-04-22T18:12:39.295 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~8388608,22517552~2648272,41943040~25165824,71303168~4194304,80884137~15584855,100663296~4194304,125829120~8388608,137734006~678026,159383552~12582912,182160500~14971788,201326592~8388608,213909504~12582912,239075328~8388608,251658240~8388608,276824064~12582912,293601280~12582912,356100183~8804265] 2022-04-22T18:12:39.295 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 3 diff [4194304~12582912,20971520~4194304,41943040~20971520,67108864~8388608,83886080~16777216,125829120~8388608,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,268435456~16777216,293601280~8388608,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.296 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~4194304,41943040~20971520,67108864~8388608,83886080~16777216,125829120~8388608,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,268435456~16777216,293601280~8388608,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.304 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~12582912,20971520~8388608,41943040~33554432,80884137~23973463,125829120~8388608,137734006~678026,159383552~12582912,182160500~14971788,201326592~8388608,213909504~12582912,239075328~8388608,251658240~8388608,268435456~20971520,293601280~12582912,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.304 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 4 diff [4194304~12582912,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~4194304,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,268435456~16777216,293601280~8388608,306184192~4194304,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.304 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~4194304,159383552~12582912,184549376~12582912,201326592~4194304,213909504~8388608,239075328~8388608,251658240~8388608,268435456~16777216,293601280~8388608,306184192~4194304,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.313 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [4194304~12582912,20971520~8388608,41943040~62914560,125829120~8388608,137734006~9066634,159383552~12582912,182160500~14971788,201326592~8388608,213909504~12582912,239075328~8388608,251658240~8388608,268435456~20971520,293601280~29360128,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.313 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 5 diff [0~16777216,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~12582912,159383552~16777216,184549376~12582912,201326592~20971520,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.313 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~12582912,159383552~16777216,184549376~12582912,201326592~20971520,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.321 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~8388608,41943040~62914560,125829120~8388608,137734006~13260938,159383552~16777216,182160500~14971788,201326592~25165824,239075328~8388608,251658240~8388608,268435456~20971520,293601280~29360128,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.321 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 6 diff [0~16777216,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~12582912,159383552~16777216,184549376~12582912,201326592~25165824,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.321 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,41943040~20971520,67108864~33554432,125829120~8388608,138412032~12582912,159383552~16777216,184549376~12582912,201326592~25165824,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.330 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~8388608,41943040~62914560,125829120~8388608,137734006~13260938,159383552~16777216,182160500~14971788,201326592~25165824,239075328~20971520,268435456~20971520,293601280~29360128,327155712~8388608,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.330 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 7 diff [0~29360128,41943040~20971520,67108864~33554432,125829120~25165824,155189248~25165824,184549376~12582912,201326592~25165824,239075328~8388608,251658240~8388608,268435456~16777216,289406976~20971520,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.332 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,41943040~20971520,67108864~33554432,125829120~25165824,155189248~25165824,184549376~12582912,201326592~25165824,239075328~8388608,251658240~8388608,268435456~16777216,289406976~20971520,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.337 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~29360128,41943040~62914560,125829120~25165824,155189248~25165824,182160500~14971788,201326592~25165824,239075328~20971520,268435456~54525952,327155712~8388608,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.338 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 8 diff [0~29360128,37748736~67108864,125829120~25165824,155189248~25165824,184549376~12582912,201326592~25165824,230686720~4194304,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.338 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~67108864,125829120~25165824,155189248~25165824,184549376~12582912,201326592~25165824,230686720~4194304,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.348 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~71303168,125829120~25165824,155189248~25165824,182160500~14971788,201326592~25165824,230686720~4194304,239075328~20971520,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.348 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 9 diff [0~29360128,37748736~67108864,117440512~4194304,125829120~25165824,155189248~25165824,184549376~12582912,201326592~29360128,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.348 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~67108864,117440512~4194304,125829120~25165824,155189248~25165824,184549376~12582912,201326592~29360128,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.355 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,182160500~14971788,201326592~33554432,239075328~20971520,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.355 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 10 diff [0~16777216,20971520~8388608,37748736~67108864,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,184549376~62914560,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.356 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~67108864,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,184549376~62914560,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.363 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.363 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 11 diff [0~16777216,20971520~8388608,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.363 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.372 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.372 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 12 diff [0~29360128,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.372 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.379 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~113246208,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.380 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 13 diff [0~29360128,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.380 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~71303168,117440512~4194304,125829120~25165824,155189248~25165824,184549376~75497472,264241152~20971520,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.389 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~113246208,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.390 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 14 diff [0~109051904,117440512~4194304,125829120~205520896,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.390 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~109051904,117440512~4194304,125829120~205520896,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.396 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~251658240,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.396 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 15 diff [0~109051904,117440512~4194304,125829120~205520896,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.397 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~109051904,117440512~4194304,125829120~205520896,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.405 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.405 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 16 diff [0~113246208,117440512~4194304,125829120~205520896,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.405 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.413 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.413 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 17 diff [0~113246208,117440512~4194304,125829120~25165824,155189248~176160768,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.413 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~25165824,155189248~176160768,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.421 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.421 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 18 diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.421 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.429 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.429 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to 19 diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.429 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.437 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.437 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 2 diff [41943040~4194304,100663296~4194304,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,276824064~8388608,293601280~8388608,356515840~8388608] 2022-04-22T18:12:39.437 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [41943040~4194304,100663296~4194304,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,276824064~8388608,293601280~8388608,356515840~8388608] 2022-04-22T18:12:39.445 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [22517552~2648272,41943040~8388608,66226793~882071,80884137~3001943,92413278~4055714,100663296~4194304,137734006~678026,159383552~12582912,182160500~14971788,207994270~1720930,213909504~12582912,247144862~319074,251658240~8388608,276824064~12582912,293601280~12582912,356100183~8804265] 2022-04-22T18:12:39.445 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 3 diff [8388608~8388608,20971520~4194304,41943040~12582912,67108864~4194304,92274688~8388608,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,268435456~16777216,293601280~8388608,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.445 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~4194304,41943040~12582912,67108864~4194304,92274688~8388608,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,268435456~16777216,293601280~8388608,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.453 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8388608~8388608,20971520~8388608,41943040~12582912,66226793~9270679,80884137~3001943,92274688~12582912,137734006~678026,159383552~12582912,182160500~14971788,207994270~1720930,213909504~12582912,247144862~319074,251658240~8388608,268435456~20971520,293601280~12582912,343932928~20971520,369098752~8388608] 2022-04-22T18:12:39.453 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 4 diff [8388608~8388608,20971520~8388608,41943040~12582912,67108864~16777216,92274688~8388608,138412032~4194304,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,268435456~16777216,293601280~8388608,306184192~4194304,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.454 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,41943040~12582912,67108864~16777216,92274688~8388608,138412032~4194304,159383552~8388608,184549376~12582912,213909504~8388608,251658240~8388608,268435456~16777216,293601280~8388608,306184192~4194304,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.461 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8388608~8388608,20971520~8388608,41943040~12582912,66226793~21853591,92274688~12582912,137734006~9066634,159383552~12582912,182160500~14971788,207994270~1720930,213909504~12582912,247144862~319074,251658240~8388608,268435456~20971520,293601280~29360128,343932928~20971520,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.461 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 5 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,159383552~8388608,171966464~4194304,184549376~12582912,205520896~16777216,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.461 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,159383552~8388608,171966464~4194304,184549376~12582912,205520896~16777216,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.469 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,66226793~38630807,137734006~13260938,159383552~16777216,182160500~14971788,205520896~20971520,247144862~319074,251658240~8388608,268435456~20971520,293601280~29360128,343932928~20971520,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.469 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 6 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,159383552~8388608,171966464~4194304,184549376~12582912,205520896~20971520,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.469 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,159383552~8388608,171966464~4194304,184549376~12582912,205520896~20971520,239075328~8388608,251658240~8388608,268435456~16777216,293601280~16777216,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.477 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,66226793~38630807,137734006~13260938,159383552~16777216,182160500~14971788,205520896~20971520,239075328~20971520,268435456~20971520,293601280~29360128,327155712~8388608,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.477 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 7 diff [0~4194304,8388608~20971520,41943040~12582912,67108864~33554432,134217728~16777216,155189248~12582912,171966464~8388608,184549376~12582912,205520896~20971520,239075328~8388608,251658240~8388608,268435456~16777216,289406976~20971520,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.478 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,41943040~12582912,67108864~33554432,134217728~16777216,155189248~12582912,171966464~8388608,184549376~12582912,205520896~20971520,239075328~8388608,251658240~8388608,268435456~16777216,289406976~20971520,314572800~4194304,327155712~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.486 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~20971520,41943040~12582912,66226793~38630807,134217728~16777216,155189248~25165824,182160500~14971788,205520896~20971520,239075328~20971520,268435456~54525952,327155712~8388608,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.486 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 8 diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,134217728~16777216,155189248~12582912,171966464~8388608,184549376~12582912,205520896~20971520,230686720~4194304,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.486 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,134217728~16777216,155189248~12582912,171966464~8388608,184549376~12582912,205520896~20971520,230686720~4194304,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.494 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~42825111,134217728~16777216,155189248~25165824,182160500~14971788,205520896~20971520,230686720~4194304,239075328~20971520,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.494 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 9 diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,117440512~4194304,125829120~25165824,155189248~25165824,184549376~12582912,205520896~25165824,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.494 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,117440512~4194304,125829120~25165824,155189248~25165824,184549376~12582912,205520896~25165824,239075328~8388608,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.502 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~42825111,117440512~4194304,125829120~25165824,155189248~25165824,182160500~14971788,205520896~29360128,239075328~20971520,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.502 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 10 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~37748736,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~4194304,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.502 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~37748736,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~4194304,251658240~8388608,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.510 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~47019415,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.510 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 11 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.510 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~20971520,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:39 smithi121 bash[36902]: cluster 2022-04-22T18:12:38.248204+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2022-04-22T18:12:39.518 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~47019415,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~100663296,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.519 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 12 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.519 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.526 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~113246208,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.527 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.527 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~54525952,243269632~16777216,264241152~20971520,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.541 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~77886348,264241152~113246208,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.541 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 14 diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.541 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.542 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~251658240,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.542 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 15 diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.542 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:39 smithi115 bash[46934]: cluster 2022-04-22T18:12:38.248204+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2022-04-22T18:12:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:39 smithi115 bash[45495]: cluster 2022-04-22T18:12:38.248204+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2022-04-22T18:12:39.551 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.551 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 16 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.551 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~113246208,243269632~88080384,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.559 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.559 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 17 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~83886080,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.559 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~83886080,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.566 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.567 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 18 diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.567 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.575 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.575 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to 19 diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.575 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.583 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.583 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 3 diff [8388608~8388608,20971520~4194304,41943040~12582912,67108864~4194304,92274688~8388608,268435456~8388608,343932928~12582912,369098752~8388608] 2022-04-22T18:12:39.584 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~4194304,41943040~12582912,67108864~4194304,92274688~8388608,268435456~8388608,343932928~12582912,369098752~8388608] 2022-04-22T18:12:39.592 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8388608~8388608,20971520~8388608,41943040~12582912,66226793~9270679,80884137~3001943,92274688~12582912,137734006~678026,170985201~981263,182160500~2388876,196672861~459427,207994270~1720930,222305300~4187116,247144862~319074,259702320~344528,268435456~12582912,285875022~3531954,305783437~400755,343932928~16777216,363668673~1235775,369098752~8388608] 2022-04-22T18:12:39.592 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 4 diff [8388608~8388608,20971520~8388608,41943040~12582912,67108864~16777216,92274688~8388608,138412032~4194304,251658240~8388608,268435456~8388608,306184192~4194304,314572800~4194304,343932928~12582912,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.592 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,41943040~12582912,67108864~16777216,92274688~8388608,138412032~4194304,251658240~8388608,268435456~8388608,306184192~4194304,314572800~4194304,343932928~12582912,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.601 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [8388608~8388608,20971520~8388608,41943040~12582912,66226793~21853591,92274688~12582912,137734006~9066634,170985201~981263,182160500~2388876,196672861~459427,207994270~1720930,222305300~4187116,247144862~319074,251658240~8388608,268435456~12582912,285875022~3531954,305783437~17177971,343932928~16777216,363668673~1235775,369098752~8388608,381681664~4194304] 2022-04-22T18:12:39.601 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 5 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,171966464~4194304,205520896~8388608,251658240~8388608,268435456~8388608,293601280~8388608,306184192~4194304,314572800~4194304,343932928~12582912,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.602 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,171966464~4194304,205520896~8388608,251658240~8388608,268435456~8388608,293601280~8388608,306184192~4194304,314572800~4194304,343932928~12582912,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.606 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,66226793~38630807,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,222305300~4187116,247144862~319074,251658240~8388608,268435456~12582912,285875022~3531954,293601280~8388608,305783437~17177971,343932928~16777216,363668673~1235775,369098752~8388608,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.606 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 6 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,268435456~8388608,293601280~8388608,306184192~4194304,314572800~4194304,327155712~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.606 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,67108864~33554432,138412032~12582912,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,268435456~8388608,293601280~8388608,306184192~4194304,314572800~4194304,327155712~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.614 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~8388608,41943040~12582912,66226793~38630807,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~20971520,268435456~12582912,285875022~3531954,293601280~8388608,305783437~17177971,327155712~8388608,343932928~16777216,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.614 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 7 diff [0~4194304,8388608~20971520,41943040~12582912,67108864~33554432,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,268435456~12582912,289406976~20971520,314572800~4194304,327155712~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.614 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,41943040~12582912,67108864~33554432,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,268435456~12582912,289406976~20971520,314572800~4194304,327155712~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.623 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~20971520,41943040~12582912,66226793~38630807,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~20971520,268435456~16777216,285875022~37086386,327155712~8388608,343932928~16777216,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.623 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 8 diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.624 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~12582912,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.630 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~42825111,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,230686720~4194304,239075328~20971520,264241152~20971520,285875022~74835122,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.630 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 9 diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.630 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~20971520,37748736~16777216,67108864~37748736,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.638 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~42825111,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,182160500~2388876,188743680~8388608,205520896~8388608,218103808~16777216,239075328~20971520,264241152~20971520,285875022~79029426,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.693 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 10 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~37748736,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.693 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~37748736,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,251658240~8388608,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.700 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~47019415,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~79029426,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.700 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 11 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~16777216,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.700 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~16777216,264241152~16777216,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,343932928~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.700 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~47019415,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~79029426,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.701 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 12 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~16777216,264241152~16777216,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.701 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~16777216,264241152~16777216,289406976~33554432,327155712~4194304,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.701 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~91612338,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.701 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~29360128,218103808~20971520,243269632~16777216,264241152~16777216,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~41943040,117440512~4194304,125829120~25165824,155189248~25165824,184549376~29360128,218103808~20971520,243269632~16777216,264241152~16777216,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~75497472,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~91612338,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 14 diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,343932928~33554432,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~91612338,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 15 diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~41943040,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.702 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.703 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 16 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~88080384,218103808~20971520,243269632~41943040,289406976~41943040,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.703 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~88080384,218103808~20971520,243269632~41943040,289406976~41943040,335544320~4194304,343932928~41943040,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.703 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.703 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 17 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.703 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.704 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.704 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 18 diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.704 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.712 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.712 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to 19 diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.712 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.718 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.718 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 4 diff [25165824~4194304,67108864~16777216,138412032~4194304,251658240~8388608,306184192~4194304,314572800~4194304,356515840~4194304,381681664~4194304] 2022-04-22T18:12:39.718 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [25165824~4194304,67108864~16777216,138412032~4194304,251658240~8388608,306184192~4194304,314572800~4194304,356515840~4194304,381681664~4194304] 2022-04-22T18:12:39.726 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [25165824~4194304,66226793~21853591,103841790~1015810,137734006~9066634,170985201~981263,182160500~2388876,196672861~459427,207994270~1720930,222305300~4187116,247144862~319074,251658240~8388608,285875022~3531954,305783437~17177971,356515840~4194304,363668673~1235775,376808440~678920,381681664~4194304] 2022-04-22T18:12:39.726 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 5 diff [0~4194304,20971520~8388608,67108864~25165824,138412032~12582912,171966464~4194304,205520896~8388608,251658240~8388608,293601280~8388608,306184192~4194304,314572800~4194304,348127232~12582912,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.726 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,20971520~8388608,67108864~25165824,138412032~12582912,171966464~4194304,205520896~8388608,251658240~8388608,293601280~8388608,306184192~4194304,314572800~4194304,348127232~12582912,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.733 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,20971520~8388608,66226793~30242199,103841790~1015810,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,222305300~4187116,247144862~319074,251658240~8388608,285875022~3531954,293601280~8388608,305783437~17177971,348127232~12582912,363668673~1235775,376808440~678920,381681664~4194304,394264576~8388608] 2022-04-22T18:12:39.734 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 6 diff [0~4194304,20971520~8388608,41943040~4194304,67108864~25165824,138412032~12582912,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,293601280~8388608,306184192~4194304,314572800~4194304,327155712~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.734 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,20971520~8388608,41943040~4194304,67108864~25165824,138412032~12582912,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,293601280~8388608,306184192~4194304,314572800~4194304,327155712~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.742 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,20971520~8388608,41943040~4194304,66226793~30242199,103841790~1015810,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~20971520,285875022~3531954,293601280~8388608,305783437~17177971,327155712~8388608,348127232~12582912,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.742 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 7 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~4194304,67108864~25165824,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,276824064~4194304,289406976~20971520,314572800~4194304,327155712~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.742 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~4194304,67108864~25165824,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,251658240~8388608,276824064~4194304,289406976~20971520,314572800~4194304,327155712~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.750 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~20971520,41943040~4194304,66226793~30242199,103841790~1015810,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~20971520,276824064~8388608,285875022~37086386,327155712~8388608,348127232~12582912,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.750 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 8 diff [0~4194304,8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,67108864~25165824,100663296~4194304,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.751 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,67108864~25165824,100663296~4194304,134217728~16777216,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.757 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~8388608,50331648~8388608,66226793~30242199,100663296~8388608,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,230686720~4194304,239075328~20971520,264241152~20971520,285875022~58057906,348127232~12582912,363668673~1235775,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.758 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 9 diff [0~4194304,8388608~8388608,20971520~8388608,37748736~16777216,67108864~25165824,100663296~4194304,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.758 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,37748736~16777216,67108864~25165824,100663296~4194304,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.765 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~30242199,100663296~8388608,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,182160500~2388876,188743680~8388608,205520896~8388608,218103808~16777216,239075328~20971520,264241152~20971520,285875022~58057906,348127232~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.765 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 10 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~25165824,100663296~4194304,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.765 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~25165824,100663296~4194304,109051904~4194304,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,251658240~8388608,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.773 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~30242199,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~58057906,348127232~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.773 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 11 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.773 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~20971520,314572800~8388608,327155712~4194304,335544320~4194304,348127232~12582912,369098752~4194304,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.781 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~30242199,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~58057906,348127232~16777216,369098752~8388608,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.781 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 12 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~33554432,327155712~4194304,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.781 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~33554432,327155712~4194304,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.790 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~58720256,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~58057906,348127232~29360128,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.791 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,184549376~29360128,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~41943040,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.791 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,184549376~29360128,218103808~20971520,243269632~16777216,264241152~8388608,276824064~4194304,289406976~41943040,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.797 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~58720256,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~58057906,348127232~29360128,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.798 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 14 diff [0~54525952,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.798 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.807 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~96468992,100663296~12582912,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~29360128,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.807 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 15 diff [0~54525952,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.807 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~25165824,100663296~12582912,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.814 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~96468992,100663296~12582912,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.814 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 16 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.814 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~88080384,218103808~20971520,243269632~37748736,289406976~41943040,335544320~4194304,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.821 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.822 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 17 diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.822 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~46137344,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.829 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.829 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 18 diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.830 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~46137344,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.839 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.839 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to 19 diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.840 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.846 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.846 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 5 diff [0~4194304,20971520~8388608,83886080~8388608,138412032~8388608,171966464~4194304,205520896~8388608,293601280~8388608,348127232~8388608,394264576~8388608] 2022-04-22T18:12:39.847 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,20971520~8388608,83886080~8388608,138412032~8388608,171966464~4194304,205520896~8388608,293601280~8388608,348127232~8388608,394264576~8388608] 2022-04-22T18:12:39.855 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,20971520~8388608,66226793~882071,83886080~12582912,103841790~1015810,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,222305300~4187116,247144862~319074,259702320~344528,285875022~3531954,293601280~8388608,305783437~400755,311071361~3501439,321110495~1850913,348127232~12582912,363668673~1235775,376808440~678920,385700964~175004,394264576~8388608] 2022-04-22T18:12:39.855 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 6 diff [0~4194304,20971520~8388608,41943040~4194304,83886080~8388608,138412032~8388608,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,293601280~8388608,327155712~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.855 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,20971520~8388608,41943040~4194304,83886080~8388608,138412032~8388608,171966464~4194304,205520896~8388608,218103808~8388608,239075328~8388608,293601280~8388608,327155712~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.863 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,20971520~8388608,41943040~4194304,66226793~882071,83886080~12582912,103841790~1015810,137734006~13260938,170985201~5175567,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~12582912,259702320~344528,285875022~3531954,293601280~8388608,305783437~400755,311071361~3501439,321110495~1850913,327155712~8388608,348127232~12582912,363668673~1235775,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.864 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 7 diff [0~4194304,8388608~8388608,20971520~8388608,41943040~4194304,83886080~8388608,134217728~12582912,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,276824064~4194304,289406976~20971520,327155712~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.864 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,41943040~4194304,83886080~8388608,134217728~12582912,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,239075328~8388608,276824064~4194304,289406976~20971520,327155712~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.871 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~20971520,41943040~4194304,66226793~882071,83886080~12582912,103841790~1015810,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,239075328~12582912,259702320~344528,276824064~8388608,285875022~24503474,311071361~3501439,321110495~1850913,327155712~8388608,348127232~12582912,363668673~1235775,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.871 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 8 diff [0~4194304,8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,83886080~8388608,100663296~4194304,134217728~12582912,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.871 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,83886080~8388608,100663296~4194304,134217728~12582912,155189248~4194304,171966464~8388608,205520896~8388608,218103808~8388608,230686720~4194304,239075328~8388608,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.879 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~8388608,50331648~8388608,66226793~882071,83886080~12582912,100663296~8388608,134217728~16777216,155189248~4194304,170985201~9369871,182160500~2388876,196672861~459427,205520896~8388608,218103808~8388608,230686720~4194304,239075328~12582912,259702320~344528,264241152~20971520,285875022~24503474,311071361~3501439,318767104~25165824,348127232~12582912,363668673~1235775,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.880 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 9 diff [0~4194304,8388608~8388608,20971520~8388608,37748736~16777216,83886080~8388608,100663296~4194304,117440512~4194304,125829120~20971520,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.880 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~8388608,37748736~16777216,83886080~8388608,100663296~4194304,117440512~4194304,125829120~20971520,155189248~4194304,167772160~12582912,188743680~4194304,205520896~8388608,218103808~12582912,239075328~8388608,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.888 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~25165824,37748736~20971520,66226793~882071,83886080~12582912,100663296~8388608,117440512~4194304,125829120~25165824,155189248~4194304,167772160~12582912,182160500~2388876,188743680~8388608,205520896~8388608,218103808~16777216,239075328~12582912,259702320~344528,264241152~20971520,285875022~28697778,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.888 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 10 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,83886080~8388608,100663296~4194304,109051904~4194304,117440512~4194304,125829120~20971520,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.889 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,83886080~8388608,100663296~4194304,109051904~4194304,117440512~4194304,125829120~20971520,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~4194304,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.896 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~882071,83886080~12582912,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~33554432,259702320~344528,264241152~20971520,285875022~28697778,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.896 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 11 diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,83886080~8388608,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.896 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,83886080~8388608,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,188743680~4194304,197132288~16777216,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~20971520,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.904 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~25165824,66226793~9270679,83886080~12582912,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~28697778,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.904 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 12 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~25165824,318767104~12582912,335544320~4194304,348127232~25165824,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.904 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,184549376~8388608,197132288~16777216,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~25165824,318767104~12582912,335544320~4194304,348127232~25165824,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.912 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~37748736,79691776~16777216,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~58057906,348127232~29360128,385700964~175004,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.912 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,184549376~29360128,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~50331648,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.912 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,155189248~25165824,184549376~29360128,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~50331648,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.920 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~37748736,79691776~16777216,100663296~12582912,117440512~4194304,125829120~25165824,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~58057906,348127232~29360128,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.920 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 14 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.920 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~25165824,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.928 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~29360128,381681664~4194304,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.928 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 15 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.928 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.936 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.988 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 16 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.989 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~20971520,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.995 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.995 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 17 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.995 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~25165824,155189248~58720256,218103808~20971520,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.995 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~88080384,218103808~67108864,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.996 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 18 diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.996 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.996 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.996 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to 19 diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.996 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.997 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:39.997 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 6 diff [41943040~4194304,218103808~8388608,239075328~8388608,327155712~4194304,352321536~8388608,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:39.997 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [41943040~4194304,218103808~8388608,239075328~8388608,327155712~4194304,352321536~8388608,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:39.997 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,29270426~89702,41943040~4194304,66226793~882071,103841790~1015810,137734006~678026,149874134~1120810,170985201~981263,173434106~2726662,182160500~2388876,196672861~459427,212914257~995247,218103808~8388608,239075328~12582912,259702320~344528,285875022~3531954,305783437~400755,311071361~3501439,321110495~1850913,327155712~8388608,352321536~8388608,363668673~1235775,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:39.998 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 7 diff [8388608~8388608,20971520~8388608,41943040~4194304,134217728~8388608,155189248~4194304,176160768~4194304,218103808~8388608,239075328~8388608,276824064~4194304,289406976~4194304,301989888~8388608,327155712~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:39.998 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,41943040~4194304,134217728~8388608,155189248~4194304,176160768~4194304,218103808~8388608,239075328~8388608,276824064~4194304,289406976~4194304,301989888~8388608,327155712~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:39.998 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~20971520,41943040~4194304,66226793~882071,103841790~1015810,134217728~8388608,149874134~1120810,155189248~4194304,170985201~981263,173434106~6920966,182160500~2388876,196672861~459427,212914257~995247,218103808~8388608,239075328~12582912,259702320~344528,276824064~8388608,285875022~7726258,301989888~8388608,311071361~3501439,321110495~1850913,327155712~8388608,348127232~12582912,363668673~1235775,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:39.998 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 8 diff [8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,100663296~4194304,134217728~8388608,155189248~4194304,176160768~4194304,218103808~8388608,230686720~4194304,239075328~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:39.999 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,37748736~8388608,50331648~4194304,100663296~4194304,134217728~8388608,155189248~4194304,176160768~4194304,218103808~8388608,230686720~4194304,239075328~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.000 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~25165824,37748736~8388608,50331648~8388608,66226793~882071,100663296~8388608,134217728~8388608,149874134~1120810,155189248~4194304,170985201~981263,173434106~6920966,182160500~2388876,196672861~459427,212914257~995247,218103808~8388608,230686720~4194304,239075328~12582912,259702320~344528,264241152~20971520,285875022~7726258,301989888~8388608,311071361~3501439,318767104~25165824,348127232~12582912,363668673~1235775,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.000 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 9 diff [8388608~8388608,20971520~8388608,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,155189248~4194304,167772160~12582912,188743680~4194304,218103808~12582912,239075328~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.000 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,155189248~4194304,167772160~12582912,188743680~4194304,218103808~12582912,239075328~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.002 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~25165824,37748736~20971520,66226793~882071,100663296~8388608,117440512~4194304,125829120~16777216,149874134~1120810,155189248~4194304,167772160~12582912,182160500~2388876,188743680~8388608,212914257~995247,218103808~16777216,239075328~12582912,259702320~344528,264241152~20971520,285875022~7726258,301989888~12582912,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.002 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 10 diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,218103808~20971520,243269632~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.002 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,218103808~20971520,243269632~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.011 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~25165824,66226793~882071,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~2388876,188743680~25165824,218103808~33554432,259702320~344528,264241152~20971520,285875022~7726258,301989888~12582912,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.011 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 11 diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.011 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~12582912,335544320~4194304,348127232~12582912,369098752~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.019 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~25165824,66226793~9270679,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~2388876,188743680~25165824,218103808~41943040,264241152~20971520,285875022~7726258,301989888~12582912,318767104~25165824,348127232~16777216,369098752~8388608,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.019 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 12 diff [4194304~25165824,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~8388608,197132288~12582912,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~12582912,318767104~12582912,335544320~4194304,348127232~25165824,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.019 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~25165824,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~8388608,197132288~12582912,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~12582912,318767104~12582912,335544320~4194304,348127232~25165824,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.027 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~7726258,301989888~41943040,348127232~29360128,385700964~175004,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.027 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~29360128,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~37748736,348127232~25165824,381681664~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.027 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~29360128,218103808~20971520,243269632~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~37748736,348127232~25165824,381681664~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.035 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~31749004,218103808~41943040,264241152~20971520,285875022~7726258,301989888~41943040,348127232~29360128,381681664~4194304,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.035 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 14 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~4194304,301989888~37748736,348127232~25165824,381681664~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.035 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~4194304,301989888~37748736,348127232~25165824,381681664~4194304,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.043 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~64035370,218103808~67108864,285875022~7726258,301989888~41943040,348127232~29360128,381681664~4194304,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.043 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 15 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~8388608,301989888~37748736,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.043 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~8388608,301989888~37748736,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.051 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~16777216,149874134~64035370,218103808~67108864,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.052 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 16 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.052 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,150994944~62914560,218103808~20971520,243269632~12582912,260046848~20971520,289406976~50331648,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.059 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,149874134~64035370,218103808~67108864,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.059 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 17 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,146800640~4194304,155189248~58720256,218103808~20971520,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.060 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,146800640~4194304,155189248~58720256,218103808~20971520,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.067 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,146800640~67108864,218103808~67108864,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.068 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 18 diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.068 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.075 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.076 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to 19 diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.076 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:40.085 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.085 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 7 diff [8388608~8388608,20971520~8388608,134217728~8388608,155189248~4194304,176160768~4194304,276824064~4194304,289406976~4194304,301989888~8388608,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.085 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,134217728~8388608,155189248~4194304,176160768~4194304,276824064~4194304,289406976~4194304,301989888~8388608,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.093 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~20971520,66226793~882071,103841790~1015810,134217728~8388608,149874134~1120810,155189248~4194304,170985201~981263,173434106~6920966,182160500~2388876,196672861~459427,212914257~995247,225131926~1360490,259702320~344528,276824064~8388608,285875022~7726258,301989888~8388608,311071361~3501439,321110495~1850913,333165103~2379217,348127232~12582912,363668673~1235775,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.093 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 8 diff [8388608~8388608,20971520~8388608,37748736~4194304,50331648~4194304,100663296~4194304,134217728~8388608,155189248~4194304,176160768~4194304,230686720~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.094 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,37748736~4194304,50331648~4194304,100663296~4194304,134217728~8388608,155189248~4194304,176160768~4194304,230686720~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.101 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~25165824,37748736~8388608,50331648~8388608,66226793~882071,100663296~8388608,134217728~8388608,149874134~1120810,155189248~4194304,170985201~981263,173434106~6920966,182160500~2388876,196672861~459427,212914257~995247,225131926~1360490,230686720~4194304,259702320~344528,264241152~20971520,285875022~7726258,301989888~8388608,311071361~3501439,318767104~8388608,333165103~10767825,348127232~12582912,363668673~1235775,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.102 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 9 diff [8388608~8388608,20971520~8388608,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,155189248~4194304,167772160~12582912,188743680~4194304,226492416~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.102 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,155189248~4194304,167772160~12582912,188743680~4194304,226492416~4194304,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304] 2022-04-22T18:12:40.110 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,8388608~25165824,37748736~20971520,66226793~882071,100663296~8388608,117440512~4194304,125829120~16777216,149874134~1120810,155189248~4194304,167772160~12582912,182160500~2388876,188743680~8388608,212914257~995247,225131926~9749098,259702320~344528,264241152~20971520,285875022~7726258,301989888~12582912,318767104~8388608,333165103~10767825,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.110 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 10 diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,226492416~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.110 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,226492416~12582912,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.117 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~25165824,66226793~882071,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~2388876,188743680~25165824,225131926~18137706,259702320~344528,264241152~20971520,285875022~7726258,301989888~12582912,318767104~8388608,333165103~10767825,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.117 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 11 diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.117 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.125 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~25165824,66226793~9270679,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~2388876,188743680~25165824,225131926~18137706,247463936~12582912,264241152~20971520,285875022~7726258,301989888~12582912,318767104~8388608,333165103~10767825,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.125 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 12 diff [4194304~25165824,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~12582912,318767104~8388608,335544320~4194304,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.125 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~25165824,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~12582912,318767104~8388608,335544320~4194304,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.132 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~32923133,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~31749004,225131926~18137706,247463936~12582912,264241152~20971520,285875022~7726258,301989888~25165824,333165103~10767825,348127232~25165824,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.133 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 13 diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.133 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~29360128,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,155189248~25165824,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,276824064~4194304,289406976~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.142 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~33554432,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,155189248~25165824,182160500~31749004,225131926~18137706,247463936~12582912,264241152~20971520,285875022~7726258,301989888~41943040,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.142 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 14 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.142 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.148 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~64035370,222298112~62914560,285875022~7726258,301989888~41943040,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.148 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 15 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~8388608,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.149 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~8388608,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.156 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~16777216,149874134~64035370,222298112~62914560,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.156 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 16 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.156 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,150994944~62914560,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.164 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,149874134~64035370,222298112~62914560,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.164 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 17 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,146800640~4194304,155189248~58720256,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.164 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~16777216,146800640~4194304,155189248~58720256,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.172 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,146800640~67108864,222298112~62914560,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.172 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 18 diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.172 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~113246208,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.181 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.181 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to 19 diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.181 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.189 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.189 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 8 diff [25165824~4194304,37748736~4194304,50331648~4194304,100663296~4194304,230686720~4194304,264241152~8388608,318767104~8388608,335544320~4194304] 2022-04-22T18:12:40.189 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [25165824~4194304,37748736~4194304,50331648~4194304,100663296~4194304,230686720~4194304,264241152~8388608,318767104~8388608,335544320~4194304] 2022-04-22T18:12:40.196 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,25165824~8388608,37748736~8388608,50331648~8388608,66226793~882071,100663296~8388608,149874134~1120810,157682093~1701459,170985201~981263,173434106~2726662,179200757~1154315,182160500~2388876,196672861~459427,212914257~995247,225131926~1360490,230686720~4194304,259702320~344528,264241152~12582912,285875022~3531954,291740522~1860758,311071361~3501439,318767104~8388608,333165103~10767825,363668673~1235775,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.196 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 9 diff [25165824~4194304,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,167772160~8388608,188743680~4194304,226492416~4194304,264241152~8388608,306184192~4194304,318767104~8388608,335544320~4194304,356515840~4194304] 2022-04-22T18:12:40.196 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [25165824~4194304,37748736~16777216,100663296~4194304,117440512~4194304,125829120~12582912,167772160~8388608,188743680~4194304,226492416~4194304,264241152~8388608,306184192~4194304,318767104~8388608,335544320~4194304,356515840~4194304] 2022-04-22T18:12:40.215 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,25165824~8388608,37748736~20971520,66226793~882071,100663296~8388608,117440512~4194304,125829120~16777216,149874134~1120810,157682093~1701459,167772160~8388608,179200757~1154315,182160500~2388876,188743680~8388608,212914257~995247,225131926~9749098,259702320~344528,264241152~12582912,285875022~3531954,291740522~1860758,306184192~8388608,318767104~8388608,333165103~10767825,356515840~8388608,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.215 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 10 diff [4194304~4194304,25165824~4194304,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,264241152~8388608,306184192~4194304,318767104~8388608,335544320~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.216 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~4194304,37748736~16777216,58720256~4194304,100663296~4194304,109051904~4194304,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,264241152~8388608,306184192~4194304,318767104~8388608,335544320~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.216 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,25165824~8388608,37748736~25165824,66226793~882071,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~2388876,188743680~25165824,225131926~18137706,259702320~344528,264241152~12582912,285875022~3531954,291740522~1860758,306184192~8388608,318767104~8388608,333165103~10767825,356515840~8388608,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.216 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 11 diff [4194304~4194304,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.216 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,301989888~8388608,318767104~8388608,335544320~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.221 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,25165824~8388608,37748736~25165824,66226793~9270679,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~2388876,188743680~25165824,225131926~18137706,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~12582912,318767104~8388608,333165103~10767825,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.221 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 12 diff [4194304~4194304,12582912~8388608,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,301989888~12582912,318767104~8388608,335544320~4194304,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.221 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~8388608,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,264241152~8388608,301989888~12582912,318767104~8388608,335544320~4194304,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.229 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~20340221,25165824~8388608,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~31749004,225131926~18137706,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~25165824,333165103~10767825,348127232~25165824,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.229 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 13 diff [0~8388608,12582912~8388608,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.229 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,25165824~4194304,37748736~16777216,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.236 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,25165824~8388608,37748736~37748736,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~31749004,225131926~18137706,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~41943040,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.237 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 14 diff [0~8388608,12582912~8388608,25165824~29360128,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~16777216,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.237 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,25165824~29360128,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~16777216,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.244 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,25165824~50331648,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~5315114,157682093~18478675,179200757~34708747,222298112~58720256,285875022~3531954,291740522~1860758,301989888~41943040,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.244 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 15 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.244 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.252 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~16777216,149874134~5315114,157682093~56227411,222298112~58720256,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.252 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 16 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.253 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~4194304,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.263 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,149874134~5315114,157682093~56227411,222298112~62914560,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.263 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 17 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.263 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.270 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,146800640~8388608,157682093~56227411,222298112~62914560,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.270 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 18 diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.270 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.278 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.278 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to 19 diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.278 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.286 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.287 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 9 diff [41943040~8388608,117440512~4194304,125829120~12582912,167772160~8388608,188743680~4194304,226492416~4194304,306184192~4194304,356515840~4194304] 2022-04-22T18:12:40.287 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [41943040~8388608,117440512~4194304,125829120~12582912,167772160~8388608,188743680~4194304,226492416~4194304,306184192~4194304,356515840~4194304] 2022-04-22T18:12:40.295 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,31097842~2456590,41943040~12582912,66226793~882071,107186880~1865024,117440512~4194304,125829120~16777216,149874134~1120810,157682093~1701459,167772160~8388608,179200757~1154315,182160500~2388876,188743680~8388608,212914257~995247,225131926~9749098,259702320~344528,285875022~3531954,291740522~1860758,306184192~8388608,325510480~1645232,333165103~2379217,342583582~1349346,356515840~8388608,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.295 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 10 diff [4194304~4194304,41943040~8388608,58720256~4194304,109051904~4194304,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,306184192~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.295 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,41943040~8388608,58720256~4194304,109051904~4194304,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,306184192~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.303 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,31097842~2456590,41943040~12582912,58720256~4194304,66226793~882071,107186880~6059328,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~2388876,188743680~25165824,225131926~18137706,259702320~344528,285875022~3531954,291740522~1860758,306184192~8388608,325510480~1645232,333165103~2379217,342583582~1349346,356515840~8388608,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.303 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 11 diff [4194304~4194304,29360128~4194304,41943040~8388608,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,301989888~8388608,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.303 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,29360128~4194304,41943040~8388608,58720256~4194304,67108864~4194304,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,188743680~4194304,197132288~12582912,226492416~12582912,247463936~8388608,301989888~8388608,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.311 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,29360128~4194304,41943040~12582912,58720256~4194304,66226793~9270679,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~2388876,188743680~25165824,225131926~18137706,247463936~12582912,285875022~3531954,291740522~1860758,301989888~12582912,325510480~1645232,333165103~2379217,342583582~1349346,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.312 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 12 diff [4194304~4194304,12582912~8388608,29360128~4194304,41943040~12582912,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,301989888~12582912,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.312 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~8388608,29360128~4194304,41943040~12582912,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~8388608,197132288~12582912,226492416~12582912,247463936~8388608,301989888~12582912,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.320 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~20340221,29360128~4194304,41943040~33554432,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~31749004,225131926~18137706,247463936~12582912,285875022~3531954,291740522~1860758,301989888~16777216,325510480~1645232,333165103~2379217,342583582~1349346,348127232~25165824,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.320 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 13 diff [0~8388608,12582912~8388608,29360128~4194304,41943040~12582912,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.320 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,29360128~4194304,41943040~12582912,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,159383552~16777216,184549376~29360128,226492416~12582912,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.327 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,29360128~4194304,41943040~33554432,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~1120810,157682093~18478675,179200757~1154315,182160500~31749004,225131926~18137706,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.328 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 14 diff [0~8388608,12582912~8388608,25165824~29360128,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~16777216,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.328 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,25165824~29360128,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~16777216,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.336 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,25165824~50331648,79691776~12582912,100663296~12582912,117440512~4194304,125829120~16777216,149874134~5315114,157682093~18478675,179200757~34708747,222298112~58720256,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.336 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 15 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.336 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.344 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~16777216,100663296~12582912,117440512~4194304,125829120~16777216,149874134~5315114,157682093~56227411,222298112~58720256,285875022~3531954,291740522~43803798,342583582~1349346,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.344 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 16 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.344 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.352 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,149874134~5315114,157682093~56227411,222298112~62914560,285875022~3531954,291740522~43803798,342583582~1349346,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.352 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 17 diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.352 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~54525952,58720256~4194304,67108864~4194304,79691776~33554432,117440512~4194304,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.360 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~75497472,79691776~33554432,117440512~4194304,125829120~16777216,146800640~8388608,157682093~56227411,222298112~62914560,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.360 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 18 diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.360 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~62914560,67108864~8388608,79691776~33554432,117440512~4194304,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.368 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.368 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to 19 diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.368 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.380 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.380 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 10 diff [4194304~4194304,58720256~4194304,109051904~4194304,125829120~4194304,159383552~8388608,197132288~12582912,230686720~8388608,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.380 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,58720256~4194304,109051904~4194304,125829120~4194304,159383552~8388608,197132288~12582912,230686720~8388608,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.384 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,31097842~2456590,58720256~4194304,66226793~882071,107186880~6059328,121139884~494932,125829120~8388608,149874134~1120810,157682093~14284371,174722912~1437856,179200757~1154315,182160500~2388876,196672861~17236643,225131926~1360490,230686720~12582912,259702320~344528,285875022~3531954,291740522~1860758,313399025~1173775,325510480~1645232,333165103~2379217,342583582~1349346,356515840~8388608,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.384 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 11 diff [4194304~4194304,29360128~4194304,58720256~4194304,67108864~4194304,100663296~12582912,125829120~8388608,159383552~8388608,197132288~12582912,230686720~8388608,247463936~8388608,301989888~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.385 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,29360128~4194304,58720256~4194304,67108864~4194304,100663296~12582912,125829120~8388608,159383552~8388608,197132288~12582912,230686720~8388608,247463936~8388608,301989888~4194304,348127232~4194304,356515840~4194304,406847488~4194304] 2022-04-22T18:12:40.392 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~11951613,29360128~4194304,58720256~4194304,66226793~9270679,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~14284371,174722912~1437856,179200757~1154315,182160500~2388876,196672861~17236643,225131926~1360490,230686720~12582912,247463936~12582912,285875022~3531954,291740522~1860758,301989888~4194304,313399025~1173775,325510480~1645232,333165103~2379217,342583582~1349346,348127232~16777216,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.393 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 12 diff [4194304~4194304,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,159383552~12582912,184549376~4194304,197132288~12582912,230686720~8388608,247463936~8388608,301989888~12582912,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.393 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,159383552~12582912,184549376~4194304,197132288~12582912,230686720~8388608,247463936~8388608,301989888~12582912,348127232~4194304,356515840~12582912,406847488~4194304] 2022-04-22T18:12:40.400 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~20340221,29360128~4194304,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~14284371,174722912~1437856,179200757~1154315,182160500~10777484,196672861~17236643,225131926~1360490,230686720~12582912,247463936~12582912,285875022~3531954,291740522~1860758,301989888~16777216,325510480~1645232,333165103~2379217,342583582~1349346,348127232~25165824,376808440~678920,385700964~175004,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.400 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 13 diff [0~8388608,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,159383552~12582912,184549376~29360128,230686720~8388608,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.400 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,159383552~12582912,184549376~29360128,230686720~8388608,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.408 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,29360128~4194304,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~14284371,174722912~1437856,179200757~1154315,182160500~31749004,225131926~1360490,230686720~12582912,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.408 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 14 diff [0~8388608,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~12582912,150994944~4194304,159383552~12582912,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.408 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~8388608,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~12582912,150994944~4194304,159383552~12582912,180355072~33554432,222298112~16777216,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,356515840~12582912,381681664~4194304,406847488~4194304] 2022-04-22T18:12:40.416 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~20971520,25165824~20971520,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~12582912,149874134~5315114,157682093~14284371,174722912~1437856,179200757~34708747,222298112~58720256,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~25165824,376808440~678920,381681664~4194304,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.416 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 15 diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.416 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.423 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~25165824,79691776~16777216,100663296~12582912,121139884~494932,125829120~12582912,149874134~5315114,157682093~56227411,222298112~58720256,285875022~3531954,291740522~43803798,342583582~1349346,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.423 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 16 diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.424 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,150994944~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.432 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~25165824,79691776~33554432,121139884~494932,125829120~12582912,149874134~5315114,157682093~56227411,222298112~62914560,285875022~3531954,291740522~43803798,342583582~1349346,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.459 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 17 diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.459 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,146800640~4194304,159383552~54525952,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.463 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~25165824,79691776~33554432,121139884~494932,125829120~12582912,146800640~8388608,157682093~56227411,222298112~62914560,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.463 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 18 diff [0~41943040,50331648~12582912,67108864~8388608,79691776~33554432,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.463 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~12582912,67108864~8388608,79691776~33554432,125829120~29360128,159383552~79691776,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,356515840~12582912,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.464 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~62914560,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.464 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to 19 diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.464 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:40.465 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:40.465 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 11 diff [29360128~4194304,67108864~4194304,100663296~12582912,125829120~8388608,234881024~4194304,247463936~8388608,301989888~4194304,348127232~4194304] 2022-04-22T18:12:40.465 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [29360128~4194304,67108864~4194304,100663296~12582912,125829120~8388608,234881024~4194304,247463936~8388608,301989888~4194304,348127232~4194304] 2022-04-22T18:12:40.467 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,29360128~4194304,66226793~9270679,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~1701459,174722912~1437856,179200757~1154315,182160500~2388876,196672861~459427,200574503~752089,212914257~995247,225131926~1360490,234881024~4194304,247463936~12582912,285875022~3531954,291740522~1860758,301989888~4194304,313399025~1173775,325510480~1645232,333165103~2379217,342583582~1349346,348127232~8388608,363668673~1235775,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.467 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 12 diff [12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,167772160~4194304,184549376~4194304,234881024~4194304,247463936~8388608,301989888~12582912,348127232~4194304,360710144~8388608] 2022-04-22T18:12:40.467 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,167772160~4194304,184549376~4194304,234881024~4194304,247463936~8388608,301989888~12582912,348127232~4194304,360710144~8388608] 2022-04-22T18:12:40.471 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,12582912~8388608,29360128~4194304,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~1701459,167772160~4194304,174722912~1437856,179200757~1154315,182160500~10777484,196672861~459427,200574503~752089,212914257~995247,225131926~1360490,234881024~8388608,247463936~12582912,285875022~3531954,291740522~1860758,301989888~16777216,325510480~1645232,333165103~2379217,342583582~1349346,348127232~8388608,360710144~12582912,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.471 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 13 diff [0~4194304,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,167772160~4194304,184549376~25165824,234881024~4194304,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.472 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,12582912~8388608,29360128~4194304,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~8388608,167772160~4194304,184549376~25165824,234881024~4194304,247463936~8388608,264241152~8388608,301989888~29360128,348127232~4194304,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.480 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,12582912~8388608,29360128~4194304,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~8388608,149874134~1120810,157682093~1701459,167772160~4194304,174722912~1437856,179200757~1154315,182160500~27554700,212914257~995247,225131926~1360490,234881024~8388608,247463936~12582912,264241152~12582912,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~8388608,360710144~12582912,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.480 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 14 diff [0~4194304,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~12582912,150994944~4194304,159383552~4194304,167772160~4194304,180355072~29360128,222298112~8388608,234881024~4194304,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.480 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,67108864~4194304,79691776~8388608,100663296~12582912,125829120~12582912,150994944~4194304,159383552~4194304,167772160~4194304,180355072~29360128,222298112~8388608,234881024~4194304,243269632~12582912,260046848~16777216,301989888~29360128,348127232~4194304,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.488 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,12582912~8388608,25165824~20971520,50331648~25165824,79691776~12582912,100663296~12582912,121139884~494932,125829120~12582912,149874134~5315114,157682093~14284371,174722912~1437856,179200757~30514443,212914257~995247,222298112~8388608,234881024~46137344,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,348127232~8388608,360710144~12582912,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.488 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 15 diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,125829120~12582912,150994944~4194304,159383552~4194304,167772160~41943040,222298112~8388608,234881024~4194304,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.488 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,67108864~4194304,79691776~12582912,100663296~12582912,125829120~12582912,150994944~4194304,159383552~4194304,167772160~41943040,222298112~8388608,234881024~4194304,243269632~12582912,260046848~16777216,293601280~4194304,301989888~29360128,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.495 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~37748736,50331648~25165824,79691776~16777216,100663296~12582912,121139884~494932,125829120~12582912,149874134~5315114,157682093~52033107,212914257~995247,222298112~8388608,234881024~46137344,285875022~3531954,291740522~43803798,342583582~1349346,348127232~8388608,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.496 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 16 diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,150994944~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.496 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,150994944~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.504 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~37748736,50331648~25165824,79691776~33554432,121139884~494932,125829120~12582912,149874134~5315114,157682093~52033107,212914257~995247,222298112~62914560,285875022~3531954,291740522~43803798,342583582~1349346,348127232~8388608,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.504 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 17 diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,146800640~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.504 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~4194304,58720256~4194304,67108864~4194304,79691776~33554432,125829120~12582912,146800640~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:40 smithi121 bash[36902]: cluster 2022-04-22T18:12:39.112295+0000 mgr.y (mgr.34107) 1638 : cluster [DBG] pgmap v2378: 137 pgs: 137 active+clean; 795 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s wr, 18 op/s 2022-04-22T18:12:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:40 smithi121 bash[36902]: cluster 2022-04-22T18:12:39.256653+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2022-04-22T18:12:40.523 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~25165824,79691776~33554432,121139884~494932,125829120~12582912,146800640~8388608,157682093~52033107,212914257~995247,222298112~62914560,285875022~3531954,291740522~52192406,348127232~8388608,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.523 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 18 diff [0~41943040,50331648~12582912,67108864~8388608,79691776~33554432,125829120~29360128,159383552~12582912,176160768~33554432,213909504~25165824,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.523 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~12582912,67108864~8388608,79691776~33554432,125829120~29360128,159383552~12582912,176160768~33554432,213909504~25165824,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~4194304,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.523 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~62914560,121139884~494932,125829120~83886080,212914257~72298415,285875022~3531954,291740522~52192406,348127232~8388608,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.523 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to 19 diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:40.524 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:40.529 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.529 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 12 diff [12582912~8388608,50331648~4194304,58720256~4194304,79691776~8388608,167772160~4194304,184549376~4194304,234881024~4194304,306184192~8388608,360710144~8388608] 2022-04-22T18:12:40.529 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [12582912~8388608,50331648~4194304,58720256~4194304,79691776~8388608,167772160~4194304,184549376~4194304,234881024~4194304,306184192~8388608,360710144~8388608] 2022-04-22T18:12:40.537 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [631299~3563005,12582912~8388608,31216033~2338399,50331648~16777216,79691776~12582912,112506692~739516,121139884~494932,149874134~1120810,157682093~1701459,167772160~4194304,174722912~1437856,179200757~1154315,182160500~10777484,196672861~459427,200574503~752089,212914257~995247,225131926~1360490,234881024~8388608,259702320~344528,285875022~3531954,291740522~1860758,306184192~12582912,325510480~1645232,333165103~2379217,342583582~1349346,360710144~12582912,376808440~678920,385700964~175004,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.538 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 13 diff [0~4194304,12582912~8388608,50331648~4194304,58720256~4194304,79691776~8388608,167772160~4194304,184549376~25165824,234881024~4194304,264241152~8388608,301989888~29360128,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.538 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,12582912~8388608,50331648~4194304,58720256~4194304,79691776~8388608,167772160~4194304,184549376~25165824,234881024~4194304,264241152~8388608,301989888~29360128,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:40 smithi115 bash[45495]: cluster 2022-04-22T18:12:39.112295+0000 mgr.y (mgr.34107) 1638 : cluster [DBG] pgmap v2378: 137 pgs: 137 active+clean; 795 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s wr, 18 op/s 2022-04-22T18:12:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:40 smithi115 bash[45495]: cluster 2022-04-22T18:12:39.256653+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2022-04-22T18:12:40.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:40 smithi115 bash[46934]: cluster 2022-04-22T18:12:39.112295+0000 mgr.y (mgr.34107) 1638 : cluster [DBG] pgmap v2378: 137 pgs: 137 active+clean; 795 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s wr, 18 op/s 2022-04-22T18:12:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:40 smithi115 bash[46934]: cluster 2022-04-22T18:12:39.256653+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2022-04-22T18:12:40.545 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,12582912~8388608,31216033~2338399,50331648~16777216,79691776~12582912,112506692~739516,121139884~494932,149874134~1120810,157682093~1701459,167772160~4194304,174722912~1437856,179200757~1154315,182160500~27554700,212914257~995247,225131926~1360490,234881024~8388608,259702320~344528,264241152~12582912,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,360710144~12582912,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.545 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 14 diff [0~4194304,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,79691776~8388608,130023424~8388608,150994944~4194304,159383552~4194304,167772160~4194304,180355072~29360128,222298112~8388608,234881024~4194304,243269632~4194304,260046848~16777216,301989888~29360128,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.545 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,12582912~8388608,25165824~16777216,50331648~4194304,58720256~4194304,79691776~8388608,130023424~8388608,150994944~4194304,159383552~4194304,167772160~4194304,180355072~29360128,222298112~8388608,234881024~4194304,243269632~4194304,260046848~16777216,301989888~29360128,360710144~8388608,381681664~4194304] 2022-04-22T18:12:40.552 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,12582912~8388608,25165824~20971520,50331648~16777216,79691776~12582912,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~14284371,174722912~1437856,179200757~30514443,212914257~995247,222298112~8388608,234881024~16777216,259702320~21316048,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,360710144~12582912,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.553 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 15 diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,79691776~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~41943040,222298112~8388608,234881024~4194304,243269632~8388608,260046848~16777216,293601280~4194304,301989888~29360128,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.553 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,79691776~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~41943040,222298112~8388608,234881024~4194304,243269632~8388608,260046848~16777216,293601280~4194304,301989888~29360128,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.560 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~37748736,50331648~16777216,79691776~16777216,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~52033107,212914257~995247,222298112~8388608,234881024~16777216,259702320~21316048,285875022~3531954,291740522~43803798,342583582~1349346,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.560 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 16 diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,79691776~20971520,104857600~4194304,130023424~8388608,150994944~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~8388608,260046848~20971520,293601280~37748736,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.560 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~33554432,50331648~4194304,58720256~4194304,79691776~20971520,104857600~4194304,130023424~8388608,150994944~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~8388608,260046848~20971520,293601280~37748736,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.568 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~37748736,50331648~16777216,79691776~20971520,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~52033107,212914257~995247,222298112~29360128,259702320~25510352,285875022~3531954,291740522~43803798,342583582~1349346,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.568 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 17 diff [0~41943040,50331648~4194304,58720256~4194304,79691776~29360128,130023424~8388608,146800640~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.568 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~4194304,58720256~4194304,79691776~29360128,130023424~8388608,146800640~4194304,159383552~12582912,176160768~33554432,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.575 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~16777216,79691776~29360128,112506692~739516,121139884~494932,130023424~8388608,146800640~8388608,157682093~52033107,212914257~995247,222298112~62914560,285875022~3531954,291740522~52192406,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.576 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 18 diff [0~41943040,50331648~12582912,67108864~8388608,79691776~29360128,130023424~25165824,159383552~12582912,176160768~33554432,213909504~25165824,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.576 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,50331648~12582912,67108864~8388608,79691776~29360128,130023424~25165824,159383552~12582912,176160768~33554432,213909504~25165824,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,360710144~8388608,373293056~12582912] 2022-04-22T18:12:40.584 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~46137344,50331648~58720256,112506692~739516,121139884~494932,130023424~79691776,212914257~72298415,285875022~3531954,291740522~52192406,360710144~25165824,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.584 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to 19 diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:40.584 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:40.592 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.592 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 13 diff [0~4194304,188743680~20971520,264241152~8388608,301989888~29360128,381681664~4194304] 2022-04-22T18:12:40.592 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,188743680~20971520,264241152~8388608,301989888~29360128,381681664~4194304] 2022-04-22T18:12:40.600 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,31216033~2338399,66226793~882071,112506692~739516,121139884~494932,149874134~1120810,157682093~1701459,174722912~1437856,179200757~1154315,182160500~2388876,188743680~20971520,212914257~995247,225131926~1360490,259702320~344528,264241152~12582912,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.600 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 14 diff [0~4194304,25165824~16777216,130023424~8388608,150994944~4194304,159383552~4194304,180355072~4194304,188743680~20971520,222298112~8388608,243269632~4194304,260046848~16777216,301989888~29360128,381681664~4194304] 2022-04-22T18:12:40.600 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,25165824~16777216,130023424~8388608,150994944~4194304,159383552~4194304,180355072~4194304,188743680~20971520,222298112~8388608,243269632~4194304,260046848~16777216,301989888~29360128,381681664~4194304] 2022-04-22T18:12:40.608 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,25165824~20971520,66226793~882071,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~10090067,174722912~1437856,179200757~30514443,212914257~995247,222298112~8388608,243269632~8388608,259702320~21316048,285875022~3531954,291740522~1860758,301989888~33554432,342583582~1349346,376808440~678920,381681664~4194304,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.609 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 15 diff [0~4194304,8388608~4194304,20971520~20971520,88080384~4194304,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~16777216,188743680~20971520,222298112~8388608,243269632~8388608,260046848~16777216,293601280~4194304,301989888~29360128,373293056~12582912] 2022-04-22T18:12:40.609 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~4194304,20971520~20971520,88080384~4194304,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~16777216,188743680~20971520,222298112~8388608,243269632~8388608,260046848~16777216,293601280~4194304,301989888~29360128,373293056~12582912] 2022-04-22T18:12:40.617 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~25165824,66226793~882071,88080384~8388608,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~52033107,212914257~995247,222298112~8388608,243269632~8388608,259702320~21316048,285875022~3531954,291740522~43803798,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.617 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 16 diff [0~4194304,8388608~8388608,20971520~20971520,58720256~4194304,88080384~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~25165824,188743680~20971520,222298112~12582912,243269632~8388608,260046848~20971520,293601280~37748736,373293056~12582912] 2022-04-22T18:12:40.617 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~4194304,8388608~8388608,20971520~20971520,58720256~4194304,88080384~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~25165824,188743680~20971520,222298112~12582912,243269632~8388608,260046848~20971520,293601280~37748736,373293056~12582912] 2022-04-22T18:12:40.625 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~4194304,8388608~8388608,20971520~25165824,58720256~8388608,88080384~12582912,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~52033107,212914257~995247,222298112~12582912,243269632~8388608,259702320~25510352,285875022~3531954,291740522~43803798,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.625 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 17 diff [0~16777216,20971520~20971520,50331648~4194304,58720256~4194304,88080384~20971520,130023424~8388608,146800640~4194304,159383552~25165824,188743680~20971520,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,373293056~12582912] 2022-04-22T18:12:40.625 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,50331648~4194304,58720256~4194304,88080384~20971520,130023424~8388608,146800640~4194304,159383552~25165824,188743680~20971520,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,373293056~12582912] 2022-04-22T18:12:40.633 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~25165824,50331648~16777216,88080384~20971520,112506692~739516,121139884~494932,130023424~8388608,146800640~8388608,157682093~52033107,212914257~995247,222298112~12582912,243269632~41943040,285875022~3531954,291740522~52192406,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.633 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 18 diff [0~16777216,20971520~20971520,50331648~12582912,67108864~8388608,88080384~20971520,130023424~25165824,159383552~25165824,188743680~20971520,213909504~20971520,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,373293056~12582912] 2022-04-22T18:12:40.633 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,50331648~12582912,67108864~8388608,88080384~20971520,130023424~25165824,159383552~25165824,188743680~20971520,213909504~20971520,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,373293056~12582912] 2022-04-22T18:12:40.640 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~25165824,50331648~29360128,88080384~20971520,112506692~739516,121139884~494932,130023424~79691776,212914257~21966767,243269632~41943040,285875022~3531954,291740522~52192406,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.641 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to 19 diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:40.641 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:40.648 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.648 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 14 diff [25165824~16777216,130023424~8388608,150994944~4194304,159383552~4194304,180355072~4194304,222298112~8388608,243269632~4194304,260046848~8388608,272629760~4194304] 2022-04-22T18:12:40.649 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [25165824~16777216,130023424~8388608,150994944~4194304,159383552~4194304,180355072~4194304,222298112~8388608,243269632~4194304,260046848~8388608,272629760~4194304] 2022-04-22T18:12:40.657 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,25165824~20971520,66226793~882071,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~10090067,174722912~1437856,179200757~9542923,196898757~233531,200574503~752089,212914257~995247,222298112~8388608,243269632~8388608,259702320~8733136,272629760~8388608,285875022~3531954,291740522~1860758,333165103~2379217,342583582~1349346,376808440~678920,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.657 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 15 diff [8388608~4194304,20971520~20971520,88080384~4194304,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~16777216,222298112~8388608,243269632~8388608,260046848~8388608,272629760~4194304,293601280~4194304,373293056~8388608] 2022-04-22T18:12:40.657 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~4194304,20971520~20971520,88080384~4194304,104857600~4194304,130023424~8388608,150994944~4194304,159383552~4194304,167772160~16777216,222298112~8388608,243269632~8388608,260046848~8388608,272629760~4194304,293601280~4194304,373293056~8388608] 2022-04-22T18:12:40.665 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,8388608~8388608,20971520~25165824,66226793~882071,88080384~8388608,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~31061587,196898757~233531,200574503~752089,212914257~995247,222298112~8388608,243269632~8388608,259702320~8733136,272629760~8388608,285875022~3531954,291740522~10249366,333165103~2379217,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.665 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 16 diff [8388608~8388608,20971520~20971520,58720256~4194304,88080384~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~25165824,222298112~12582912,243269632~8388608,260046848~8388608,272629760~8388608,293601280~8388608,373293056~8388608] 2022-04-22T18:12:40.665 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~20971520,58720256~4194304,88080384~12582912,104857600~4194304,130023424~8388608,150994944~4194304,159383552~25165824,222298112~12582912,243269632~8388608,260046848~8388608,272629760~8388608,293601280~8388608,373293056~8388608] 2022-04-22T18:12:40.673 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,8388608~8388608,20971520~25165824,58720256~8388608,88080384~12582912,104857600~4194304,112506692~739516,121139884~494932,130023424~8388608,149874134~5315114,157682093~31061587,196898757~233531,200574503~752089,212914257~995247,222298112~12582912,243269632~8388608,259702320~8733136,272629760~12582912,285875022~3531954,291740522~14443670,333165103~2379217,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.673 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 17 diff [4194304~12582912,20971520~20971520,50331648~4194304,58720256~4194304,88080384~20971520,130023424~8388608,146800640~4194304,159383552~25165824,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.673 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,50331648~4194304,58720256~4194304,88080384~20971520,130023424~8388608,146800640~4194304,159383552~25165824,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.681 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~25165824,50331648~16777216,88080384~20971520,112506692~739516,121139884~494932,130023424~8388608,146800640~8388608,157682093~31061587,196898757~233531,200574503~752089,212914257~995247,222298112~12582912,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.681 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 18 diff [4194304~12582912,20971520~20971520,50331648~12582912,67108864~8388608,88080384~20971520,130023424~25165824,159383552~25165824,213909504~20971520,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.681 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,50331648~12582912,67108864~8388608,88080384~20971520,130023424~25165824,159383552~25165824,213909504~20971520,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.689 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~25165824,50331648~29360128,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,212914257~21966767,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.690 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to 19 diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.690 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.698 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.698 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 15 diff [8388608~4194304,20971520~8388608,88080384~4194304,104857600~4194304,167772160~12582912,243269632~8388608,293601280~4194304,373293056~8388608] 2022-04-22T18:12:40.699 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~4194304,20971520~8388608,88080384~4194304,104857600~4194304,167772160~12582912,243269632~8388608,293601280~4194304,373293056~8388608] 2022-04-22T18:12:40.705 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,8388608~8388608,20971520~8388608,66226793~882071,88080384~8388608,104857600~4194304,112506692~739516,121139884~494932,149874134~1120810,152141423~3047825,157682093~1701459,167772160~16777216,196898757~233531,200574503~752089,212914257~995247,243269632~8388608,259702320~344528,285875022~3531954,291740522~10249366,333165103~2379217,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.706 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 16 diff [8388608~8388608,20971520~8388608,33554432~4194304,41943040~4194304,58720256~4194304,88080384~12582912,104857600~4194304,163577856~16777216,230686720~4194304,243269632~8388608,276824064~4194304,293601280~8388608,373293056~8388608] 2022-04-22T18:12:40.706 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [8388608~8388608,20971520~8388608,33554432~4194304,41943040~4194304,58720256~4194304,88080384~12582912,104857600~4194304,163577856~16777216,230686720~4194304,243269632~8388608,276824064~4194304,293601280~8388608,373293056~8388608] 2022-04-22T18:12:40.713 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,8388608~8388608,20971520~8388608,33554432~4194304,41943040~4194304,58720256~8388608,88080384~12582912,104857600~4194304,112506692~739516,121139884~494932,149874134~1120810,152141423~3047825,157682093~1701459,163577856~20971520,196898757~233531,200574503~752089,212914257~995247,230686720~4194304,243269632~8388608,259702320~344528,276824064~8388608,285875022~3531954,291740522~14443670,333165103~2379217,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.713 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 17 diff [4194304~12582912,20971520~8388608,33554432~4194304,41943040~4194304,50331648~4194304,58720256~4194304,88080384~20971520,146800640~4194304,163577856~16777216,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.714 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~8388608,33554432~4194304,41943040~4194304,50331648~4194304,58720256~4194304,88080384~20971520,146800640~4194304,163577856~16777216,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.722 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~8388608,33554432~4194304,41943040~4194304,50331648~16777216,88080384~20971520,112506692~739516,121139884~494932,146800640~8388608,157682093~1701459,163577856~20971520,196898757~233531,200574503~752089,212914257~995247,222298112~12582912,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.761 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 18 diff [4194304~12582912,20971520~16777216,41943040~4194304,50331648~12582912,67108864~8388608,88080384~20971520,138412032~16777216,159383552~20971520,213909504~20971520,243269632~12582912,276824064~4194304,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.761 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~4194304,50331648~12582912,67108864~8388608,88080384~20971520,138412032~16777216,159383552~20971520,213909504~20971520,243269632~12582912,276824064~4194304,293601280~8388608,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.763 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~4194304,50331648~29360128,88080384~20971520,112506692~739516,121139884~494932,138412032~46137344,196898757~233531,200574503~752089,212914257~21966767,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.763 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to 19 diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~37748736,88080384~20971520,112506692~739516,121139884~494932,138412032~50331648,196898757~233531,200574503~752089,209715200~25165824,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 16 diff [12582912~4194304,33554432~4194304,41943040~4194304,58720256~4194304,92274688~8388608,163577856~8388608,230686720~4194304,276824064~4194304,297795584~4194304,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [12582912~4194304,33554432~4194304,41943040~4194304,58720256~4194304,92274688~8388608,163577856~8388608,230686720~4194304,276824064~4194304,297795584~4194304,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,12582912~4194304,33554432~4194304,41943040~4194304,58720256~8388608,92274688~8388608,112506692~739516,121139884~494932,149874134~1120810,152141423~3047825,157682093~1701459,163577856~8388608,175109891~1050877,196898757~233531,200574503~752089,212914257~995247,230686720~4194304,259702320~344528,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,333165103~2379217,342583582~1349346,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 17 diff [4194304~4194304,12582912~4194304,33554432~4194304,41943040~4194304,50331648~4194304,58720256~4194304,92274688~12582912,146800640~4194304,163577856~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,33554432~4194304,41943040~4194304,50331648~4194304,58720256~4194304,92274688~12582912,146800640~4194304,163577856~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,33554432~4194304,41943040~4194304,50331648~16777216,92274688~12582912,112506692~739516,121139884~494932,146800640~8388608,157682093~1701459,163577856~8388608,175109891~1050877,196898757~233531,200574503~752089,212914257~995247,222298112~12582912,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.764 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 18 diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~4194304,50331648~12582912,67108864~8388608,92274688~12582912,138412032~16777216,159383552~12582912,213909504~20971520,247463936~8388608,276824064~4194304,297795584~4194304,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.765 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~4194304,50331648~12582912,67108864~8388608,92274688~12582912,138412032~16777216,159383552~12582912,213909504~20971520,247463936~8388608,276824064~4194304,297795584~4194304,318767104~8388608,335544320~8388608,373293056~8388608] 2022-04-22T18:12:40.765 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,25165824~12582912,41943040~4194304,50331648~29360128,92274688~12582912,112506692~739516,121139884~494932,138412032~33554432,175109891~1050877,196898757~233531,200574503~752089,212914257~21966767,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,318767104~12582912,333165103~10767825,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.765 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to 19 diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.765 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:40.771 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,25165824~12582912,41943040~37748736,92274688~12582912,112506692~739516,121139884~494932,138412032~33554432,175109891~13633789,196898757~233531,200574503~752089,209715200~25165824,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.771 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 17 diff [4194304~4194304,50331648~4194304,100663296~4194304,146800640~4194304,222298112~8388608,247463936~8388608,276824064~4194304,318767104~8388608,335544320~8388608] 2022-04-22T18:12:40.771 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,50331648~4194304,100663296~4194304,146800640~4194304,222298112~8388608,247463936~8388608,276824064~4194304,318767104~8388608,335544320~8388608] 2022-04-22T18:12:40.779 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,50331648~8388608,66226793~882071,100663296~4194304,112506692~739516,121139884~494932,146800640~8388608,157682093~1701459,175109891~1050877,196898757~233531,200574503~752089,212914257~995247,222298112~8388608,247463936~12582912,276824064~4194304,285875022~3531954,291740522~1860758,318767104~12582912,333165103~10767825,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.779 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 18 diff [4194304~4194304,25165824~8388608,50331648~8388608,67108864~8388608,100663296~4194304,138412032~16777216,159383552~4194304,213909504~16777216,247463936~8388608,276824064~4194304,318767104~8388608,335544320~8388608] 2022-04-22T18:12:40.779 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,50331648~8388608,67108864~8388608,100663296~4194304,138412032~16777216,159383552~4194304,213909504~16777216,247463936~8388608,276824064~4194304,318767104~8388608,335544320~8388608] 2022-04-22T18:12:40.787 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,50331648~12582912,66226793~13464983,100663296~4194304,112506692~739516,121139884~494932,138412032~25165824,175109891~1050877,196898757~233531,200574503~752089,212914257~17772463,247463936~12582912,276824064~4194304,285875022~3531954,291740522~1860758,318767104~12582912,333165103~10767825,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.787 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to 19 diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:40.787 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:40.795 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,46137344~33554432,100663296~4194304,112506692~739516,121139884~494932,138412032~25165824,175109891~13633789,196898757~233531,200574503~752089,209715200~20971520,247463936~12582912,276824064~4194304,285875022~3531954,291740522~1860758,310378496~20971520,333165103~10767825,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.795 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to 18 diff [4194304~4194304,25165824~8388608,50331648~8388608,67108864~8388608,138412032~16777216,159383552~4194304,213909504~8388608] 2022-04-22T18:12:40.795 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,50331648~8388608,67108864~8388608,138412032~16777216,159383552~4194304,213909504~8388608] 2022-04-22T18:12:40.802 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,50331648~12582912,66226793~13464983,112506692~739516,121139884~494932,138412032~25165824,175109891~1050877,196898757~233531,200574503~752089,212914257~9383855,259702320~344528,285875022~3531954,291740522~1860758,333165103~2379217,342583582~1349346,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.803 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to 19 diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:40.803 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:40.810 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,46137344~33554432,112506692~739516,121139884~494932,138412032~25165824,175109891~13633789,196898757~233531,200574503~752089,209715200~12582912,259702320~344528,285875022~3531954,291740522~1860758,310378496~8388608,333165103~2379217,342583582~1349346,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:40.810 INFO:tasks.workunit.client.0.smithi115.stdout:from 18 to 19 diff [46137344~4194304,54525952~4194304,62914560~12582912,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:40.811 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [46137344~4194304,54525952~4194304,62914560~12582912,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:40.818 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~1600183,46137344~12582912,62914560~16777216,112506692~739516,121139884~494932,157682093~1701459,175109891~13633789,196898757~233531,200574503~752089,209715200~12582912,259702320~344528,285875022~3531954,291740522~1860758,310378496~8388608,333165103~2379217,342583582~1349346,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.277 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.278 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.298 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.298 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.298 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.306 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.306 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.306 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.315 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.315 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.315 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.324 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.324 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.324 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.332 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:41.333 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:41.333 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:41.341 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:41.341 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.341 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.349 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:41.349 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.349 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.357 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:41.357 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.357 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.365 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:41.365 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.365 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:41.373 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:41.374 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:41.374 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:41.383 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.383 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:41.383 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:41.391 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.391 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:41.392 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:41.400 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.400 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.400 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.408 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.408 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.409 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.417 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~37748736,88080384~20971520,112506692~739516,121139884~494932,138412032~50331648,196898757~233531,200574503~752089,209715200~25165824,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.417 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.417 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:41.425 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,25165824~12582912,41943040~37748736,92274688~12582912,112506692~739516,121139884~494932,138412032~33554432,175109891~13633789,196898757~233531,200574503~752089,209715200~25165824,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.426 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to HEAD diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:41.426 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:41.433 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,46137344~33554432,100663296~4194304,112506692~739516,121139884~494932,138412032~25165824,175109891~13633789,196898757~233531,200574503~752089,209715200~20971520,247463936~12582912,276824064~4194304,285875022~3531954,291740522~1860758,310378496~20971520,333165103~10767825,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.433 INFO:tasks.workunit.client.0.smithi115.stdout:from 17 to HEAD diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:41.434 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,310378496~4194304,348127232~16777216] 2022-04-22T18:12:41.441 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,46137344~33554432,112506692~739516,121139884~494932,138412032~25165824,175109891~13633789,196898757~233531,200574503~752089,209715200~12582912,259702320~344528,285875022~3531954,291740522~1860758,310378496~8388608,333165103~2379217,342583582~1349346,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:41 smithi115 bash[45495]: audit 2022-04-22T18:12:40.718526+0000 mon.a (mon.0) 3510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:41 smithi115 bash[45495]: audit 2022-04-22T18:12:41.172359+0000 mon.a (mon.0) 3511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:41 smithi115 bash[46934]: audit 2022-04-22T18:12:40.718526+0000 mon.a (mon.0) 3510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:41.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:41 smithi115 bash[46934]: audit 2022-04-22T18:12:41.172359+0000 mon.a (mon.0) 3511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:41 smithi121 bash[36902]: audit 2022-04-22T18:12:40.718526+0000 mon.a (mon.0) 3510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:41 smithi121 bash[36902]: audit 2022-04-22T18:12:41.172359+0000 mon.a (mon.0) 3511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:41.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:42.294 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.294 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.323 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.323 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.324 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.333 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.334 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.334 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.342 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.342 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.342 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.350 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.350 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.351 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.358 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:42.359 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:42.359 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:42.367 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:42.367 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.368 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.376 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:42.376 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.376 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.384 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:42.384 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.384 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.392 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:42.392 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.392 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:42.401 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:42.401 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:42.402 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:42.409 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.409 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:42.410 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:42.418 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.418 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:42.418 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:42.427 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.427 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.428 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.436 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.436 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.436 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.444 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~37748736,88080384~20971520,112506692~739516,121139884~494932,138412032~50331648,196898757~233531,200574503~752089,209715200~25165824,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.444 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.444 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:42.452 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,25165824~12582912,41943040~37748736,92274688~12582912,112506692~739516,121139884~494932,138412032~33554432,175109891~13633789,196898757~233531,200574503~752089,209715200~25165824,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.453 INFO:tasks.workunit.client.0.smithi115.stdout:from 16 to HEAD diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:42.453 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,25165824~8388608,46137344~12582912,62914560~12582912,100663296~4194304,138412032~16777216,159383552~4194304,176160768~8388608,209715200~8388608,222298112~8388608,247463936~8388608,276824064~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216] 2022-04-22T18:12:42.460 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,25165824~8388608,46137344~33554432,100663296~4194304,112506692~739516,121139884~494932,138412032~25165824,175109891~13633789,196898757~233531,200574503~752089,209715200~20971520,247463936~12582912,276824064~4194304,285875022~3531954,291740522~1860758,310378496~20971520,333165103~10767825,348127232~20971520,385723445~152523,402226043~427141,414012659~1223437] 2022-04-22T18:12:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:42 smithi115 bash[45495]: cluster 2022-04-22T18:12:41.113455+0000 mgr.y (mgr.34107) 1639 : cluster [DBG] pgmap v2380: 137 pgs: 137 active+clean; 860 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 34 MiB/s wr, 1.21k op/s 2022-04-22T18:12:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:42 smithi115 bash[45495]: cluster 2022-04-22T18:12:41.269160+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2022-04-22T18:12:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:42 smithi115 bash[46934]: cluster 2022-04-22T18:12:41.113455+0000 mgr.y (mgr.34107) 1639 : cluster [DBG] pgmap v2380: 137 pgs: 137 active+clean; 860 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 34 MiB/s wr, 1.21k op/s 2022-04-22T18:12:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:42 smithi115 bash[46934]: cluster 2022-04-22T18:12:41.269160+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2022-04-22T18:12:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:42 smithi121 bash[36902]: cluster 2022-04-22T18:12:41.113455+0000 mgr.y (mgr.34107) 1639 : cluster [DBG] pgmap v2380: 137 pgs: 137 active+clean; 860 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 34 MiB/s wr, 1.21k op/s 2022-04-22T18:12:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:42 smithi121 bash[36902]: cluster 2022-04-22T18:12:41.269160+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2022-04-22T18:12:43.306 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.306 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.336 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.336 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.336 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.345 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.345 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.346 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.351 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.352 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.352 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.360 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.361 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.361 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.368 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:43.369 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:43.369 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:43.376 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:43.377 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.377 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.385 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:43.385 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.385 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.394 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:43.394 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.394 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.401 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:43.402 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.402 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:43.409 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:43.410 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:43.410 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:43.417 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.417 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:43.417 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:43.425 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.425 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:43.425 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:43.433 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.434 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.434 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.449 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.450 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.450 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.455 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~37748736,88080384~20971520,112506692~739516,121139884~494932,138412032~50331648,196898757~233531,200574503~752089,209715200~25165824,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.455 INFO:tasks.workunit.client.0.smithi115.stdout:from 15 to HEAD diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.455 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~4194304,12582912~4194304,25165824~12582912,41943040~33554432,92274688~12582912,138412032~16777216,159383552~12582912,176160768~8388608,209715200~8388608,222298112~12582912,247463936~8388608,276824064~4194304,297795584~4194304,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:43.458 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~5794487,12582912~4194304,25165824~12582912,41943040~37748736,92274688~12582912,112506692~739516,121139884~494932,138412032~33554432,175109891~13633789,196898757~233531,200574503~752089,209715200~25165824,247463936~12582912,276824064~8388608,285875022~3531954,291740522~1860758,297795584~8388608,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:43 smithi115 bash[45495]: cluster 2022-04-22T18:12:42.276102+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2022-04-22T18:12:43.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:43 smithi115 bash[45495]: audit 2022-04-22T18:12:43.104333+0000 mon.a (mon.0) 3514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:43.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:43 smithi115 bash[46934]: cluster 2022-04-22T18:12:42.276102+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2022-04-22T18:12:43.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:43 smithi115 bash[46934]: audit 2022-04-22T18:12:43.104333+0000 mon.a (mon.0) 3514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:43 smithi121 bash[36902]: cluster 2022-04-22T18:12:42.276102+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2022-04-22T18:12:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:43 smithi121 bash[36902]: audit 2022-04-22T18:12:43.104333+0000 mon.a (mon.0) 3514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:43.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:44.320 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.321 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.346 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.346 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.346 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.353 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.353 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.353 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.361 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.361 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.362 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.370 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.371 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.371 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.380 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:44.381 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:44.381 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:44.386 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:44.386 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.386 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.395 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:44.395 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.395 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.401 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:44.401 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.402 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.410 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:44.410 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.410 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:44.417 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:44.418 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:44.418 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:44.425 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:44.426 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:44.426 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:44.435 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:44.436 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:44.436 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:44.441 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:44.442 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:44.442 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:44.450 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:44.451 INFO:tasks.workunit.client.0.smithi115.stdout:from 14 to HEAD diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:44.451 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~33554432,88080384~20971520,138412032~16777216,159383552~25165824,209715200~8388608,222298112~12582912,243269632~12582912,276824064~4194304,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:44.456 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~16777216,41943040~37748736,88080384~20971520,112506692~739516,121139884~494932,138412032~50331648,196898757~233531,200574503~752089,209715200~25165824,243269632~16777216,276824064~8388608,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:44 smithi121 bash[36902]: cluster 2022-04-22T18:12:43.114459+0000 mgr.y (mgr.34107) 1640 : cluster [DBG] pgmap v2383: 137 pgs: 137 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 19 MiB/s wr, 2.00k op/s 2022-04-22T18:12:44.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:44 smithi121 bash[36902]: cluster 2022-04-22T18:12:43.285452+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2022-04-22T18:12:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:44 smithi115 bash[45495]: cluster 2022-04-22T18:12:43.114459+0000 mgr.y (mgr.34107) 1640 : cluster [DBG] pgmap v2383: 137 pgs: 137 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 19 MiB/s wr, 2.00k op/s 2022-04-22T18:12:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:44 smithi115 bash[45495]: cluster 2022-04-22T18:12:43.285452+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2022-04-22T18:12:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:44 smithi115 bash[46934]: cluster 2022-04-22T18:12:43.114459+0000 mgr.y (mgr.34107) 1640 : cluster [DBG] pgmap v2383: 137 pgs: 137 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 19 MiB/s wr, 2.00k op/s 2022-04-22T18:12:44.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:44 smithi115 bash[46934]: cluster 2022-04-22T18:12:43.285452+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2022-04-22T18:12:45.334 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.334 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.369 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.369 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.370 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.377 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.377 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.378 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.385 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.385 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.385 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.392 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.393 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.393 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.400 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:45.400 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:45.401 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:45.408 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:45.408 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.409 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.416 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:45.416 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.417 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.428 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:45.428 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.428 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.434 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:45.434 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.434 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:45.441 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:45.441 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:45.441 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:45.449 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:45.449 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:45.449 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:45.457 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:45.457 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:45.457 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:45.465 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:45.465 INFO:tasks.workunit.client.0.smithi115.stdout:from 13 to HEAD diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:45.465 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [4194304~12582912,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~29360128,209715200~8388608,222298112~12582912,243269632~12582912,260046848~8388608,272629760~8388608,293601280~8388608,310378496~4194304,318767104~8388608,335544320~8388608,348127232~16777216,373293056~8388608] 2022-04-22T18:12:45.473 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [2594121~14183095,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~58720256,196898757~233531,200574503~752089,209715200~25165824,243269632~25165824,272629760~12582912,285875022~3531954,291740522~14443670,310378496~20971520,333165103~10767825,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:45 smithi121 bash[36902]: cluster 2022-04-22T18:12:44.290397+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2022-04-22T18:12:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:45 smithi115 bash[45495]: cluster 2022-04-22T18:12:44.290397+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2022-04-22T18:12:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:45 smithi115 bash[46934]: cluster 2022-04-22T18:12:44.290397+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2022-04-22T18:12:46.329 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.329 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.378 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.378 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.379 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.386 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.386 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.386 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.393 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.393 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.394 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.401 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.401 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.402 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.409 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:46.409 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:46.409 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:46.416 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:46.416 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.418 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.425 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:46.425 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.426 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.433 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:46.433 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.433 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.440 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:46.440 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.441 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:46.448 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:46.448 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:46.448 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:46.456 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:46.457 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:46.457 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:46.464 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:46.465 INFO:tasks.workunit.client.0.smithi115.stdout:from 12 to HEAD diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:46.465 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~16777216,20971520~20971520,46137344~29360128,88080384~20971520,130023424~25165824,159383552~58720256,222298112~12582912,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~16777216,373293056~12582912] 2022-04-22T18:12:46.472 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~16777216,20971520~58720256,88080384~20971520,112506692~739516,121139884~494932,130023424~104857600,243269632~41943040,285875022~3531954,291740522~52192406,348127232~20971520,373293056~12582912,402226043~427141,414012659~1223437] 2022-04-22T18:12:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:46 smithi121 bash[36902]: cluster 2022-04-22T18:12:45.115469+0000 mgr.y (mgr.34107) 1641 : cluster [DBG] pgmap v2386: 137 pgs: 137 active+clean; 864 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 4.02k op/s 2022-04-22T18:12:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:46 smithi121 bash[36902]: cluster 2022-04-22T18:12:45.308036+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2022-04-22T18:12:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:46 smithi121 bash[36902]: audit 2022-04-22T18:12:45.721382+0000 mon.a (mon.0) 3518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:46 smithi121 bash[36902]: audit 2022-04-22T18:12:46.175779+0000 mon.a (mon.0) 3519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[45495]: cluster 2022-04-22T18:12:45.115469+0000 mgr.y (mgr.34107) 1641 : cluster [DBG] pgmap v2386: 137 pgs: 137 active+clean; 864 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 4.02k op/s 2022-04-22T18:12:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[45495]: cluster 2022-04-22T18:12:45.308036+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2022-04-22T18:12:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[45495]: audit 2022-04-22T18:12:45.721382+0000 mon.a (mon.0) 3518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[45495]: audit 2022-04-22T18:12:46.175779+0000 mon.a (mon.0) 3519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[46934]: cluster 2022-04-22T18:12:45.115469+0000 mgr.y (mgr.34107) 1641 : cluster [DBG] pgmap v2386: 137 pgs: 137 active+clean; 864 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 4.02k op/s 2022-04-22T18:12:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[46934]: cluster 2022-04-22T18:12:45.308036+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2022-04-22T18:12:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[46934]: audit 2022-04-22T18:12:45.721382+0000 mon.a (mon.0) 3518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:46 smithi115 bash[46934]: audit 2022-04-22T18:12:46.175779+0000 mon.a (mon.0) 3519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:47.348 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.349 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.397 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.397 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.397 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.404 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.404 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.404 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.411 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.411 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.411 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.420 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.420 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.420 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.427 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:47.475 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:47.475 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:47.477 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.478 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:47.479 INFO:tasks.workunit.client.0.smithi115.stdout:from 11 to HEAD diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:47.480 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~29360128,130023424~25165824,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:47.483 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~109051904,112506692~739516,121139884~494932,130023424~155189248,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:47 smithi121 bash[36902]: cluster 2022-04-22T18:12:46.317924+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2022-04-22T18:12:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:47 smithi115 bash[45495]: cluster 2022-04-22T18:12:46.317924+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2022-04-22T18:12:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:47 smithi115 bash[46934]: cluster 2022-04-22T18:12:46.317924+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2022-04-22T18:12:48.363 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.363 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.404 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.405 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.405 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.412 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.412 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.413 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.421 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.421 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.421 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.428 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.428 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.428 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.435 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:48.436 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:48.436 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:48.443 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:48.444 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.444 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.452 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:48.452 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.452 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.460 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:48.460 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.461 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.468 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:48.468 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.468 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:48.472 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:12:48.476 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:48.477 INFO:tasks.workunit.client.0.smithi115.stdout:from 10 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:48.477 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~12582912,176160768~41943040,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912] 2022-04-22T18:12:48.483 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,414012659~1223437] 2022-04-22T18:12:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:48 smithi121 bash[36902]: cluster 2022-04-22T18:12:47.116331+0000 mgr.y (mgr.34107) 1642 : cluster [DBG] pgmap v2389: 137 pgs: 137 active+clean; 861 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 3.53k op/s 2022-04-22T18:12:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:48 smithi121 bash[36902]: cluster 2022-04-22T18:12:47.332021+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2022-04-22T18:12:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:48 smithi121 bash[36902]: audit 2022-04-22T18:12:48.107448+0000 mon.a (mon.0) 3522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[46934]: cluster 2022-04-22T18:12:47.116331+0000 mgr.y (mgr.34107) 1642 : cluster [DBG] pgmap v2389: 137 pgs: 137 active+clean; 861 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 3.53k op/s 2022-04-22T18:12:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[46934]: cluster 2022-04-22T18:12:47.332021+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2022-04-22T18:12:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[46934]: audit 2022-04-22T18:12:48.107448+0000 mon.a (mon.0) 3522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[45495]: cluster 2022-04-22T18:12:47.116331+0000 mgr.y (mgr.34107) 1642 : cluster [DBG] pgmap v2389: 137 pgs: 137 active+clean; 861 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 3.53k op/s 2022-04-22T18:12:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[45495]: cluster 2022-04-22T18:12:47.332021+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2022-04-22T18:12:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:48 smithi115 bash[45495]: audit 2022-04-22T18:12:48.107448+0000 mon.a (mon.0) 3522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:48.823 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (35m) 4m ago 49m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (49m) 5m ago 49m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (37m) 4m ago 50m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (37m) 5m ago 50m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (45m) 5m ago 54m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (45m) 4m ago 57m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (44m) 4m ago 57m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (43m) 5m ago 55m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (44m) 4m ago 55m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (36m) 4m ago 49m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:12:48.824 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (36m) 5m ago 49m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (42m) 4m ago 54m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (41m) 4m ago 54m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (41m) 4m ago 53m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (41m) 4m ago 53m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (40m) 5m ago 52m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (39m) 5m ago 52m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (38m) 5m ago 51m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (38m) 5m ago 51m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:12:48.825 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (36m) 5m ago 50m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:49.223 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:12:49.224 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:12:49.368 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.369 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.418 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.418 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.418 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.426 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.427 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.427 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.434 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.435 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.435 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.442 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.443 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.443 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.451 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:49.451 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:49.451 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:49.459 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:49.459 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.459 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.467 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:49.467 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.467 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.475 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:49.476 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.476 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.484 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:49.484 INFO:tasks.workunit.client.0.smithi115.stdout:from 9 to HEAD diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.484 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~41943040,46137344~29360128,79691776~33554432,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:49.492 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,121139884~494932,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:49 smithi121 bash[36902]: cluster 2022-04-22T18:12:48.338071+0000 mon.a (mon.0) 3523 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2022-04-22T18:12:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:49 smithi121 bash[36902]: audit 2022-04-22T18:12:48.448311+0000 mgr.y (mgr.34107) 1643 : audit [DBG] from='client.36873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:49 smithi121 bash[36902]: audit 2022-04-22T18:12:49.217566+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.115:0/3092641851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[45495]: cluster 2022-04-22T18:12:48.338071+0000 mon.a (mon.0) 3523 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2022-04-22T18:12:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[45495]: audit 2022-04-22T18:12:48.448311+0000 mgr.y (mgr.34107) 1643 : audit [DBG] from='client.36873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[45495]: audit 2022-04-22T18:12:49.217566+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.115:0/3092641851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[46934]: cluster 2022-04-22T18:12:48.338071+0000 mon.a (mon.0) 3523 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2022-04-22T18:12:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[46934]: audit 2022-04-22T18:12:48.448311+0000 mgr.y (mgr.34107) 1643 : audit [DBG] from='client.36873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:49 smithi115 bash[46934]: audit 2022-04-22T18:12:49.217566+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.115:0/3092641851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:12:50.376 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.376 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.437 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.437 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.437 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.447 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.447 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.447 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.456 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.456 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.456 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.465 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.465 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.465 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.473 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:50.473 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:50.473 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:50.481 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:50.482 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.482 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.490 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:50.491 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.491 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.499 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:50.499 INFO:tasks.workunit.client.0.smithi115.stdout:from 8 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.499 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:50.508 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:50.727 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[45495]: audit 2022-04-22T18:12:48.797401+0000 mgr.y (mgr.34107) 1644 : audit [DBG] from='client.36879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:50.727 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[45495]: cluster 2022-04-22T18:12:49.117294+0000 mgr.y (mgr.34107) 1645 : cluster [DBG] pgmap v2392: 137 pgs: 137 active+clean; 842 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.14k op/s 2022-04-22T18:12:50.727 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[45495]: cluster 2022-04-22T18:12:49.347908+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2022-04-22T18:12:50.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[46934]: audit 2022-04-22T18:12:48.797401+0000 mgr.y (mgr.34107) 1644 : audit [DBG] from='client.36879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:50.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[46934]: cluster 2022-04-22T18:12:49.117294+0000 mgr.y (mgr.34107) 1645 : cluster [DBG] pgmap v2392: 137 pgs: 137 active+clean; 842 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.14k op/s 2022-04-22T18:12:50.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:50 smithi115 bash[46934]: cluster 2022-04-22T18:12:49.347908+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2022-04-22T18:12:50.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:50 smithi121 bash[36902]: audit 2022-04-22T18:12:48.797401+0000 mgr.y (mgr.34107) 1644 : audit [DBG] from='client.36879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:12:50.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:50 smithi121 bash[36902]: cluster 2022-04-22T18:12:49.117294+0000 mgr.y (mgr.34107) 1645 : cluster [DBG] pgmap v2392: 137 pgs: 137 active+clean; 842 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.14k op/s 2022-04-22T18:12:50.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:50 smithi121 bash[36902]: cluster 2022-04-22T18:12:49.347908+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2022-04-22T18:12:51.177 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.177 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.215 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.216 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.216 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.224 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.224 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.224 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.231 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.231 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.231 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.239 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.239 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.240 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.246 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:51.247 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:51.247 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:51.254 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:51.254 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:51.254 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:51.261 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:51.261 INFO:tasks.workunit.client.0.smithi115.stdout:from 7 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:51.262 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~29360128,159383552~58720256,222298112~16777216,243269632~12582912,260046848~20971520,293601280~37748736,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:51.269 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~3531954,291740522~52192406,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: cluster 2022-04-22T18:12:50.358971+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1270: 8 total, 8 up, 8 in 2022-04-22T18:12:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:50.405138+0000 mon.a (mon.0) 3526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:12:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:50.406954+0000 mon.a (mon.0) 3527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:12:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:50.408045+0000 mon.a (mon.0) 3528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:12:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:50.724100+0000 mon.a (mon.0) 3529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:51.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:50.733188+0000 mon.a (mon.0) 3530 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:51.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: cluster 2022-04-22T18:12:51.145712+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2022-04-22T18:12:51.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:51 smithi121 bash[36902]: audit 2022-04-22T18:12:51.178950+0000 mon.a (mon.0) 3532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:51.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: cluster 2022-04-22T18:12:50.358971+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1270: 8 total, 8 up, 8 in 2022-04-22T18:12:51.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:50.405138+0000 mon.a (mon.0) 3526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:50.406954+0000 mon.a (mon.0) 3527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:50.408045+0000 mon.a (mon.0) 3528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:50.724100+0000 mon.a (mon.0) 3529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:50.733188+0000 mon.a (mon.0) 3530 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: cluster 2022-04-22T18:12:51.145712+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[45495]: audit 2022-04-22T18:12:51.178950+0000 mon.a (mon.0) 3532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:51.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: cluster 2022-04-22T18:12:50.358971+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1270: 8 total, 8 up, 8 in 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:50.405138+0000 mon.a (mon.0) 3526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:50.406954+0000 mon.a (mon.0) 3527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:50.408045+0000 mon.a (mon.0) 3528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:50.724100+0000 mon.a (mon.0) 3529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:50.733188+0000 mon.a (mon.0) 3530 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: cluster 2022-04-22T18:12:51.145712+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2022-04-22T18:12:51.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[46934]: audit 2022-04-22T18:12:51.178950+0000 mon.a (mon.0) 3532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:51.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:52.180 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.181 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.226 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.226 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.226 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.236 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.236 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.236 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.243 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.243 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.243 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.251 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.251 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.251 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.259 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:52.259 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:52.260 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:52.268 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:52.268 INFO:tasks.workunit.client.0.smithi115.stdout:from 6 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:52.268 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~92274688,222298112~16777216,243269632~12582912,260046848~20971520,289406976~41943040,335544320~8388608,348127232~20971520,373293056~12582912,406847488~4194304] 2022-04-22T18:12:52.275 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,402226043~427141,406847488~8388608] 2022-04-22T18:12:52.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:52 smithi121 bash[36902]: cluster 2022-04-22T18:12:51.118318+0000 mgr.y (mgr.34107) 1646 : cluster [DBG] pgmap v2395: 137 pgs: 137 active+clean; 796 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.35k op/s 2022-04-22T18:12:52.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:52 smithi121 bash[36902]: cluster 2022-04-22T18:12:52.158652+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2022-04-22T18:12:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:52 smithi115 bash[45495]: cluster 2022-04-22T18:12:51.118318+0000 mgr.y (mgr.34107) 1646 : cluster [DBG] pgmap v2395: 137 pgs: 137 active+clean; 796 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.35k op/s 2022-04-22T18:12:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:52 smithi115 bash[45495]: cluster 2022-04-22T18:12:52.158652+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2022-04-22T18:12:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:52 smithi115 bash[46934]: cluster 2022-04-22T18:12:51.118318+0000 mgr.y (mgr.34107) 1646 : cluster [DBG] pgmap v2395: 137 pgs: 137 active+clean; 796 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 1.35k op/s 2022-04-22T18:12:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:52 smithi115 bash[46934]: cluster 2022-04-22T18:12:52.158652+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2022-04-22T18:12:53.185 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.185 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.228 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.228 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.229 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.235 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.236 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.236 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.244 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.244 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.244 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.254 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.254 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.254 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.261 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:53.261 INFO:tasks.workunit.client.0.smithi115.stdout:from 5 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:53.261 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~4194304,406847488~8388608] 2022-04-22T18:12:53.268 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~8388608,402226043~427141,406847488~8388608] 2022-04-22T18:12:53.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:53 smithi115 bash[45495]: audit 2022-04-22T18:12:53.110215+0000 mon.a (mon.0) 3534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:53.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:53 smithi115 bash[45495]: cluster 2022-04-22T18:12:53.161389+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2022-04-22T18:12:53.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:53 smithi115 bash[46934]: audit 2022-04-22T18:12:53.110215+0000 mon.a (mon.0) 3534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:53.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:53 smithi115 bash[46934]: cluster 2022-04-22T18:12:53.161389+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2022-04-22T18:12:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:53 smithi121 bash[36902]: audit 2022-04-22T18:12:53.110215+0000 mon.a (mon.0) 3534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:53.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:53 smithi121 bash[36902]: cluster 2022-04-22T18:12:53.161389+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2022-04-22T18:12:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:12:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:12:53.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:12:54.183 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.184 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.230 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.230 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.230 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.239 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.240 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.240 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.247 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.247 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.248 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.255 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.255 INFO:tasks.workunit.client.0.smithi115.stdout:from 4 to HEAD diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.255 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~75497472,79691776~33554432,117440512~4194304,125829120~113246208,243269632~37748736,289406976~54525952,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.262 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:54 smithi121 bash[36902]: cluster 2022-04-22T18:12:53.119217+0000 mgr.y (mgr.34107) 1647 : cluster [DBG] pgmap v2398: 137 pgs: 137 active+clean; 773 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 909 op/s 2022-04-22T18:12:54.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:54 smithi121 bash[36902]: cluster 2022-04-22T18:12:54.164309+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2022-04-22T18:12:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:54 smithi115 bash[45495]: cluster 2022-04-22T18:12:53.119217+0000 mgr.y (mgr.34107) 1647 : cluster [DBG] pgmap v2398: 137 pgs: 137 active+clean; 773 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 909 op/s 2022-04-22T18:12:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:54 smithi115 bash[45495]: cluster 2022-04-22T18:12:54.164309+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2022-04-22T18:12:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:54 smithi115 bash[46934]: cluster 2022-04-22T18:12:53.119217+0000 mgr.y (mgr.34107) 1647 : cluster [DBG] pgmap v2398: 137 pgs: 137 active+clean; 773 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 909 op/s 2022-04-22T18:12:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:54 smithi115 bash[46934]: cluster 2022-04-22T18:12:54.164309+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2022-04-22T18:12:55.185 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.185 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.230 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.230 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.230 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.237 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.238 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.238 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.245 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.245 INFO:tasks.workunit.client.0.smithi115.stdout:from 3 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.245 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~37748736,289406976~41943040,335544320~8388608,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:55.252 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~58057906,348127232~37748736,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.172 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.172 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.213 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.213 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.214 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.221 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.221 INFO:tasks.workunit.client.0.smithi115.stdout:from 2 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.221 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~41943040,289406976~41943040,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.228 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~159383552,285875022~100000946,390070272~12582912,406847488~8388608] 2022-04-22T18:12:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:56 smithi121 bash[36902]: cluster 2022-04-22T18:12:55.120295+0000 mgr.y (mgr.34107) 1648 : cluster [DBG] pgmap v2401: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 698 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 828 op/s 2022-04-22T18:12:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:56 smithi121 bash[36902]: cluster 2022-04-22T18:12:55.168092+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2022-04-22T18:12:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:56 smithi121 bash[36902]: audit 2022-04-22T18:12:55.726910+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:56 smithi121 bash[36902]: cluster 2022-04-22T18:12:56.145178+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2022-04-22T18:12:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[45495]: cluster 2022-04-22T18:12:55.120295+0000 mgr.y (mgr.34107) 1648 : cluster [DBG] pgmap v2401: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 698 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 828 op/s 2022-04-22T18:12:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[45495]: cluster 2022-04-22T18:12:55.168092+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[45495]: audit 2022-04-22T18:12:55.726910+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[45495]: cluster 2022-04-22T18:12:56.145178+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[46934]: cluster 2022-04-22T18:12:55.120295+0000 mgr.y (mgr.34107) 1648 : cluster [DBG] pgmap v2401: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 698 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 828 op/s 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[46934]: cluster 2022-04-22T18:12:55.168092+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[46934]: audit 2022-04-22T18:12:55.726910+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:12:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:56 smithi115 bash[46934]: cluster 2022-04-22T18:12:56.145178+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2022-04-22T18:12:57.190 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.191 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.239 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.239 INFO:tasks.workunit.client.0.smithi115.stdout:from 1 to HEAD diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.239 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~113246208,243269632~88080384,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.246 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:57 smithi121 bash[36902]: audit 2022-04-22T18:12:56.181490+0000 mon.a (mon.0) 3540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:57 smithi115 bash[45495]: audit 2022-04-22T18:12:56.181490+0000 mon.a (mon.0) 3540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:57.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:57 smithi115 bash[46934]: audit 2022-04-22T18:12:56.181490+0000 mon.a (mon.0) 3540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:12:58.218 INFO:tasks.workunit.client.0.smithi115.stdout:from 0 to HEAD diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:58.218 INFO:tasks.workunit.client.0.smithi115.stdout: limited diff [0~113246208,117440512~4194304,125829120~205520896,335544320~50331648,390070272~12582912,406847488~8388608] 2022-04-22T18:12:58.245 INFO:tasks.workunit.client.0.smithi115.stdout: actual was [0~113246208,117440512~4194304,125829120~260046848,390070272~12582912,406847488~8388608] 2022-04-22T18:12:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:58 smithi121 bash[36902]: cluster 2022-04-22T18:12:57.121278+0000 mgr.y (mgr.34107) 1649 : cluster [DBG] pgmap v2404: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 662 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 788 op/s 2022-04-22T18:12:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:58 smithi121 bash[36902]: cluster 2022-04-22T18:12:57.177006+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2022-04-22T18:12:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:58 smithi121 bash[36902]: audit 2022-04-22T18:12:58.113377+0000 mon.a (mon.0) 3542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[45495]: cluster 2022-04-22T18:12:57.121278+0000 mgr.y (mgr.34107) 1649 : cluster [DBG] pgmap v2404: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 662 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 788 op/s 2022-04-22T18:12:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[45495]: cluster 2022-04-22T18:12:57.177006+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2022-04-22T18:12:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[45495]: audit 2022-04-22T18:12:58.113377+0000 mon.a (mon.0) 3542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[46934]: cluster 2022-04-22T18:12:57.121278+0000 mgr.y (mgr.34107) 1649 : cluster [DBG] pgmap v2404: 137 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 130 active+clean; 662 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 788 op/s 2022-04-22T18:12:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[46934]: cluster 2022-04-22T18:12:57.177006+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2022-04-22T18:12:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:58 smithi115 bash[46934]: audit 2022-04-22T18:12:58.113377+0000 mon.a (mon.0) 3542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:12:59.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateStress (39919 ms) 2022-04-22T18:12:59.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateRegression6926 2022-04-22T18:12:59.232 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:12:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:12:59 smithi121 bash[36902]: cluster 2022-04-22T18:12:58.185784+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2022-04-22T18:12:59.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:12:59 smithi115 bash[45495]: cluster 2022-04-22T18:12:58.185784+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2022-04-22T18:12:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:12:59 smithi115 bash[46934]: cluster 2022-04-22T18:12:58.185784+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2022-04-22T18:13:00.217 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 0~4194304 2022-04-22T18:13:00.222 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateRegression6926 (991 ms) 2022-04-22T18:13:00.222 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParent 2022-04-22T18:13:00.222 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:00.222 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateParent (0 ms) 2022-04-22T18:13:00.223 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateIgnoreParent 2022-04-22T18:13:00.223 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:00.223 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateIgnoreParent (0 ms) 2022-04-22T18:13:00.223 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateCallbackError 2022-04-22T18:13:00.223 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:13:00.240 INFO:tasks.workunit.client.0.smithi115.stdout: wrote [812659~102328,1419722~61567,3149091~46475,3693868~2446,5744151~31001,8879885~73786,10852585~81953,11528966~25617,15627151~82645,16794045~23347] 2022-04-22T18:13:00.250 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateCallbackError (27 ms) 2022-04-22T18:13:00.250 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParentDiscard 2022-04-22T18:13:00.250 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:00.250 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateParentDiscard (0 ms) 2022-04-22T18:13:00.251 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnalignedSmall 2022-04-22T18:13:00.251 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:13:00.303 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 5000005~1234 2022-04-22T18:13:00.306 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnalignedSmall (55 ms) 2022-04-22T18:13:00.306 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnaligned 2022-04-22T18:13:00.306 INFO:tasks.workunit.client.0.smithi115.stdout:using old format 2022-04-22T18:13:00.389 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 8376263~12345 2022-04-22T18:13:00.391 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 8388608~4194304 2022-04-22T18:13:00.391 INFO:tasks.workunit.client.0.smithi115.stdout:iterate_cb 12582912~54321 2022-04-22T18:13:00.395 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnaligned (91 ms) 2022-04-22T18:13:00.396 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStriping 2022-04-22T18:13:00.396 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:00.396 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] DiffIterateTest/1.DiffIterateStriping (0 ms) 2022-04-22T18:13:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:00 smithi121 bash[36902]: cluster 2022-04-22T18:12:59.122518+0000 mgr.y (mgr.34107) 1650 : cluster [DBG] pgmap v2407: 137 pgs: 137 active+clean; 587 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 352 op/s 2022-04-22T18:13:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:00 smithi121 bash[36902]: cluster 2022-04-22T18:12:59.198746+0000 mon.a (mon.0) 3544 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2022-04-22T18:13:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:00 smithi115 bash[45495]: cluster 2022-04-22T18:12:59.122518+0000 mgr.y (mgr.34107) 1650 : cluster [DBG] pgmap v2407: 137 pgs: 137 active+clean; 587 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 352 op/s 2022-04-22T18:13:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:00 smithi115 bash[45495]: cluster 2022-04-22T18:12:59.198746+0000 mon.a (mon.0) 3544 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2022-04-22T18:13:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:00 smithi115 bash[46934]: cluster 2022-04-22T18:12:59.122518+0000 mgr.y (mgr.34107) 1650 : cluster [DBG] pgmap v2407: 137 pgs: 137 active+clean; 587 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 352 op/s 2022-04-22T18:13:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:00 smithi115 bash[46934]: cluster 2022-04-22T18:12:59.198746+0000 mon.a (mon.0) 3544 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2022-04-22T18:13:01.159 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 11 tests from DiffIterateTest/1 (49217 ms total) 2022-04-22T18:13:01.160 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:13:01.160 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 16 tests from TestImageWatcher 2022-04-22T18:13:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:01 smithi121 bash[36902]: cluster 2022-04-22T18:13:00.203233+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2022-04-22T18:13:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:01 smithi121 bash[36902]: audit 2022-04-22T18:13:00.730096+0000 mon.a (mon.0) 3546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:01 smithi121 bash[36902]: cluster 2022-04-22T18:13:01.148673+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2022-04-22T18:13:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:01 smithi121 bash[36902]: audit 2022-04-22T18:13:01.184714+0000 mon.a (mon.0) 3548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:01.774Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[45495]: cluster 2022-04-22T18:13:00.203233+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2022-04-22T18:13:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[45495]: audit 2022-04-22T18:13:00.730096+0000 mon.a (mon.0) 3546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[45495]: cluster 2022-04-22T18:13:01.148673+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2022-04-22T18:13:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[45495]: audit 2022-04-22T18:13:01.184714+0000 mon.a (mon.0) 3548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[46934]: cluster 2022-04-22T18:13:00.203233+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2022-04-22T18:13:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[46934]: audit 2022-04-22T18:13:00.730096+0000 mon.a (mon.0) 3546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[46934]: cluster 2022-04-22T18:13:01.148673+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2022-04-22T18:13:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:01 smithi115 bash[46934]: audit 2022-04-22T18:13:01.184714+0000 mon.a (mon.0) 3548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:02 smithi121 bash[36902]: cluster 2022-04-22T18:13:01.123583+0000 mgr.y (mgr.34107) 1651 : cluster [DBG] pgmap v2410: 137 pgs: 137 active+clean; 479 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 534 KiB/s wr, 291 op/s 2022-04-22T18:13:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:02 smithi121 bash[36902]: cluster 2022-04-22T18:13:02.165675+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2022-04-22T18:13:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:02 smithi121 bash[36902]: audit 2022-04-22T18:13:02.168489+0000 mon.c (mon.1) 94 : audit [INF] from='client.? 172.21.15.115:0/1870499768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:02 smithi121 bash[36902]: audit 2022-04-22T18:13:02.169109+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[46934]: cluster 2022-04-22T18:13:01.123583+0000 mgr.y (mgr.34107) 1651 : cluster [DBG] pgmap v2410: 137 pgs: 137 active+clean; 479 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 534 KiB/s wr, 291 op/s 2022-04-22T18:13:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[46934]: cluster 2022-04-22T18:13:02.165675+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2022-04-22T18:13:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[46934]: audit 2022-04-22T18:13:02.168489+0000 mon.c (mon.1) 94 : audit [INF] from='client.? 172.21.15.115:0/1870499768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[46934]: audit 2022-04-22T18:13:02.169109+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[45495]: cluster 2022-04-22T18:13:01.123583+0000 mgr.y (mgr.34107) 1651 : cluster [DBG] pgmap v2410: 137 pgs: 137 active+clean; 479 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 534 KiB/s wr, 291 op/s 2022-04-22T18:13:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[45495]: cluster 2022-04-22T18:13:02.165675+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2022-04-22T18:13:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[45495]: audit 2022-04-22T18:13:02.168489+0000 mon.c (mon.1) 94 : audit [INF] from='client.? 172.21.15.115:0/1870499768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:02 smithi115 bash[45495]: audit 2022-04-22T18:13:02.169109+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:03.174 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2022-04-22T18:13:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:03 smithi121 bash[36902]: audit 2022-04-22T18:13:03.116486+0000 mon.a (mon.0) 3551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:03 smithi121 bash[36902]: cluster 2022-04-22T18:13:03.124481+0000 mgr.y (mgr.34107) 1652 : cluster [DBG] pgmap v2413: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:03 smithi121 bash[36902]: audit 2022-04-22T18:13:03.166723+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:03 smithi121 bash[36902]: cluster 2022-04-22T18:13:03.166984+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2022-04-22T18:13:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:03.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[45495]: audit 2022-04-22T18:13:03.116486+0000 mon.a (mon.0) 3551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[45495]: cluster 2022-04-22T18:13:03.124481+0000 mgr.y (mgr.34107) 1652 : cluster [DBG] pgmap v2413: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[45495]: audit 2022-04-22T18:13:03.166723+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[45495]: cluster 2022-04-22T18:13:03.166984+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2022-04-22T18:13:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[46934]: audit 2022-04-22T18:13:03.116486+0000 mon.a (mon.0) 3551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[46934]: cluster 2022-04-22T18:13:03.124481+0000 mgr.y (mgr.34107) 1652 : cluster [DBG] pgmap v2413: 137 pgs: 2 creating+peering, 30 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[46934]: audit 2022-04-22T18:13:03.166723+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-10","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:03 smithi115 bash[46934]: cluster 2022-04-22T18:13:03.166984+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2022-04-22T18:13:05.198 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.198 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (2023 ms) 2022-04-22T18:13:05.199 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2022-04-22T18:13:05.204 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.204 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyFlatten (6 ms) 2022-04-22T18:13:05.205 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyResize 2022-04-22T18:13:05.209 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.209 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyResize (6 ms) 2022-04-22T18:13:05.209 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2022-04-22T18:13:05.215 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.215 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (5 ms) 2022-04-22T18:13:05.215 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2022-04-22T18:13:05.220 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.220 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (5 ms) 2022-04-22T18:13:05.220 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2022-04-22T18:13:05.226 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.226 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (7 ms) 2022-04-22T18:13:05.226 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2022-04-22T18:13:05.232 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapRename (6 ms) 2022-04-22T18:13:05.232 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2022-04-22T18:13:05.237 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.238 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (5 ms) 2022-04-22T18:13:05.238 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2022-04-22T18:13:05.248 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.248 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (6 ms) 2022-04-22T18:13:05.248 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2022-04-22T18:13:05.251 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.251 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (6 ms) 2022-04-22T18:13:05.251 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2022-04-22T18:13:05.257 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.257 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (7 ms) 2022-04-22T18:13:05.257 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyRename 2022-04-22T18:13:05.262 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.263 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyRename (6 ms) 2022-04-22T18:13:05.263 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2022-04-22T18:13:05.268 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.268 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (6 ms) 2022-04-22T18:13:05.268 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2022-04-22T18:13:05.275 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (6 ms) 2022-04-22T18:13:05.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2022-04-22T18:13:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (6 ms) 2022-04-22T18:13:05.280 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2022-04-22T18:13:05.286 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:05.286 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (5 ms) 2022-04-22T18:13:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:05 smithi121 bash[36902]: cluster 2022-04-22T18:13:04.168282+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e1284: 8 total, 8 up, 8 in 2022-04-22T18:13:05.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:05 smithi115 bash[45495]: cluster 2022-04-22T18:13:04.168282+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e1284: 8 total, 8 up, 8 in 2022-04-22T18:13:05.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:05 smithi115 bash[46934]: cluster 2022-04-22T18:13:04.168282+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e1284: 8 total, 8 up, 8 in 2022-04-22T18:13:06.199 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 16 tests from TestImageWatcher (2111 ms total) 2022-04-22T18:13:06.200 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:13:06.200 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 32 tests from TestInternal 2022-04-22T18:13:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:06 smithi121 bash[36902]: cluster 2022-04-22T18:13:05.125495+0000 mgr.y (mgr.34107) 1653 : cluster [DBG] pgmap v2416: 137 pgs: 2 creating+peering, 11 unknown, 124 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:06 smithi121 bash[36902]: cluster 2022-04-22T18:13:05.176274+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2022-04-22T18:13:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:06 smithi121 bash[36902]: audit 2022-04-22T18:13:05.733264+0000 mon.a (mon.0) 3556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[45495]: cluster 2022-04-22T18:13:05.125495+0000 mgr.y (mgr.34107) 1653 : cluster [DBG] pgmap v2416: 137 pgs: 2 creating+peering, 11 unknown, 124 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[45495]: cluster 2022-04-22T18:13:05.176274+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2022-04-22T18:13:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[45495]: audit 2022-04-22T18:13:05.733264+0000 mon.a (mon.0) 3556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[46934]: cluster 2022-04-22T18:13:05.125495+0000 mgr.y (mgr.34107) 1653 : cluster [DBG] pgmap v2416: 137 pgs: 2 creating+peering, 11 unknown, 124 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[46934]: cluster 2022-04-22T18:13:05.176274+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2022-04-22T18:13:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:06 smithi115 bash[46934]: audit 2022-04-22T18:13:05.733264+0000 mon.a (mon.0) 3556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:07 smithi121 bash[36902]: cluster 2022-04-22T18:13:06.189401+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2022-04-22T18:13:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:07 smithi121 bash[36902]: audit 2022-04-22T18:13:06.191295+0000 mon.a (mon.0) 3558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:07 smithi115 bash[45495]: cluster 2022-04-22T18:13:06.189401+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2022-04-22T18:13:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:07 smithi115 bash[45495]: audit 2022-04-22T18:13:06.191295+0000 mon.a (mon.0) 3558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:07.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:07 smithi115 bash[46934]: cluster 2022-04-22T18:13:06.189401+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2022-04-22T18:13:07.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:07 smithi115 bash[46934]: audit 2022-04-22T18:13:06.191295+0000 mon.a (mon.0) 3558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:08.216 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.OpenByID 2022-04-22T18:13:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:08 smithi121 bash[36902]: cluster 2022-04-22T18:13:07.126455+0000 mgr.y (mgr.34107) 1654 : cluster [DBG] pgmap v2419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:08 smithi121 bash[36902]: cluster 2022-04-22T18:13:07.204226+0000 mon.a (mon.0) 3559 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2022-04-22T18:13:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:08 smithi121 bash[36902]: audit 2022-04-22T18:13:07.209813+0000 mon.c (mon.1) 95 : audit [INF] from='client.? 172.21.15.115:0/2860474988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:08 smithi121 bash[36902]: audit 2022-04-22T18:13:07.210546+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:08 smithi121 bash[36902]: audit 2022-04-22T18:13:08.119604+0000 mon.a (mon.0) 3561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[45495]: cluster 2022-04-22T18:13:07.126455+0000 mgr.y (mgr.34107) 1654 : cluster [DBG] pgmap v2419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[45495]: cluster 2022-04-22T18:13:07.204226+0000 mon.a (mon.0) 3559 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[45495]: audit 2022-04-22T18:13:07.209813+0000 mon.c (mon.1) 95 : audit [INF] from='client.? 172.21.15.115:0/2860474988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[45495]: audit 2022-04-22T18:13:07.210546+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[45495]: audit 2022-04-22T18:13:08.119604+0000 mon.a (mon.0) 3561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[46934]: cluster 2022-04-22T18:13:07.126455+0000 mgr.y (mgr.34107) 1654 : cluster [DBG] pgmap v2419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[46934]: cluster 2022-04-22T18:13:07.204226+0000 mon.a (mon.0) 3559 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2022-04-22T18:13:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[46934]: audit 2022-04-22T18:13:07.209813+0000 mon.c (mon.1) 95 : audit [INF] from='client.? 172.21.15.115:0/2860474988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[46934]: audit 2022-04-22T18:13:07.210546+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:08 smithi115 bash[46934]: audit 2022-04-22T18:13:08.119604+0000 mon.a (mon.0) 3561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:09 smithi121 bash[36902]: audit 2022-04-22T18:13:08.204894+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:09 smithi121 bash[36902]: cluster 2022-04-22T18:13:08.206077+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2022-04-22T18:13:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:09 smithi121 bash[36902]: cluster 2022-04-22T18:13:09.208637+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2022-04-22T18:13:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[45495]: audit 2022-04-22T18:13:08.204894+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[45495]: cluster 2022-04-22T18:13:08.206077+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2022-04-22T18:13:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[45495]: cluster 2022-04-22T18:13:09.208637+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2022-04-22T18:13:09.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[46934]: audit 2022-04-22T18:13:08.204894+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-11","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:09.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[46934]: cluster 2022-04-22T18:13:08.206077+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2022-04-22T18:13:09.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:09 smithi115 bash[46934]: cluster 2022-04-22T18:13:09.208637+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2022-04-22T18:13:10.224 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.224 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.OpenByID (2008 ms) 2022-04-22T18:13:10.224 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.OpenSnapDNE 2022-04-22T18:13:10.245 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.OpenSnapDNE (22 ms) 2022-04-22T18:13:10.245 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2022-04-22T18:13:10.250 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.251 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (5 ms) 2022-04-22T18:13:10.251 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.ResizeLocksImage 2022-04-22T18:13:10.258 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.259 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.ResizeLocksImage (8 ms) 2022-04-22T18:13:10.259 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2022-04-22T18:13:10.269 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.269 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (11 ms) 2022-04-22T18:13:10.270 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2022-04-22T18:13:10.275 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapCreateLocksImage (6 ms) 2022-04-22T18:13:10.275 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2022-04-22T18:13:10.280 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.280 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (5 ms) 2022-04-22T18:13:10.280 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2022-04-22T18:13:10.285 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.285 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (5 ms) 2022-04-22T18:13:10.286 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2022-04-22T18:13:10.292 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.292 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (7 ms) 2022-04-22T18:13:10.292 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2022-04-22T18:13:10.298 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.298 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapSetReleasesLock (5 ms) 2022-04-22T18:13:10.298 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.FlattenLocksImage 2022-04-22T18:13:10.304 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.304 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.FlattenLocksImage (7 ms) 2022-04-22T18:13:10.305 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2022-04-22T18:13:10.311 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.311 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (6 ms) 2022-04-22T18:13:10.311 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2022-04-22T18:13:10.317 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.317 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.AioWriteRequestsLock (6 ms) 2022-04-22T18:13:10.317 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2022-04-22T18:13:10.323 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.323 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (7 ms) 2022-04-22T18:13:10.323 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.CancelAsyncResize 2022-04-22T18:13:10.329 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.329 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.CancelAsyncResize (5 ms) 2022-04-22T18:13:10.329 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.MultipleResize 2022-04-22T18:13:10.399 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.MultipleResize (71 ms) 2022-04-22T18:13:10.406 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.Metadata 2022-04-22T18:13:10.406 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.406 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.Metadata (6 ms) 2022-04-22T18:13:10.406 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.MetadataConfApply 2022-04-22T18:13:10.412 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.412 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.MetadataConfApply (7 ms) 2022-04-22T18:13:10.412 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapshotCopyup 2022-04-22T18:13:10.418 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.418 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapshotCopyup (6 ms) 2022-04-22T18:13:10.418 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2022-04-22T18:13:10.424 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.424 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (5 ms) 2022-04-22T18:13:10.424 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2022-04-22T18:13:10.430 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.430 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (6 ms) 2022-04-22T18:13:10.430 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.ResizeCopyup 2022-04-22T18:13:10.436 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.436 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.ResizeCopyup (7 ms) 2022-04-22T18:13:10.436 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.DiscardCopyup 2022-04-22T18:13:10.443 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.443 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.DiscardCopyup (7 ms) 2022-04-22T18:13:10.443 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.ImageOptions 2022-04-22T18:13:10.449 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.ImageOptions (6 ms) 2022-04-22T18:13:10.449 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.WriteFullCopyup 2022-04-22T18:13:10.455 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.455 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.WriteFullCopyup (6 ms) 2022-04-22T18:13:10.456 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2022-04-22T18:13:10.461 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.462 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (6 ms) 2022-04-22T18:13:10.462 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.TestCoR 2022-04-22T18:13:10.468 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.468 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.TestCoR (6 ms) 2022-04-22T18:13:10.468 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2022-04-22T18:13:10.474 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.474 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (6 ms) 2022-04-22T18:13:10.474 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2022-04-22T18:13:10.480 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.480 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.PoolMetadataConfApply (7 ms) 2022-04-22T18:13:10.480 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.Sparsify 2022-04-22T18:13:10.513 INFO:tasks.workunit.client.0.smithi115.stdout:sparsify_supported=1 2022-04-22T18:13:10.513 INFO:tasks.workunit.client.0.smithi115.stdout:sparse_read_supported=1 2022-04-22T18:13:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:10 smithi121 bash[36902]: cluster 2022-04-22T18:13:09.127515+0000 mgr.y (mgr.34107) 1655 : cluster [DBG] pgmap v2422: 137 pgs: 8 creating+peering, 16 unknown, 113 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:10 smithi121 bash[36902]: cluster 2022-04-22T18:13:10.209022+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2022-04-22T18:13:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:10 smithi115 bash[45495]: cluster 2022-04-22T18:13:09.127515+0000 mgr.y (mgr.34107) 1655 : cluster [DBG] pgmap v2422: 137 pgs: 8 creating+peering, 16 unknown, 113 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:10 smithi115 bash[45495]: cluster 2022-04-22T18:13:10.209022+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2022-04-22T18:13:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:10 smithi115 bash[46934]: cluster 2022-04-22T18:13:09.127515+0000 mgr.y (mgr.34107) 1655 : cluster [DBG] pgmap v2422: 137 pgs: 8 creating+peering, 16 unknown, 113 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:10.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:10 smithi115 bash[46934]: cluster 2022-04-22T18:13:10.209022+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2022-04-22T18:13:10.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.Sparsify (68 ms) 2022-04-22T18:13:10.549 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.SparsifyClone 2022-04-22T18:13:10.554 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.555 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.SparsifyClone (6 ms) 2022-04-22T18:13:10.555 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestInternal.MissingDataPool 2022-04-22T18:13:10.560 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:10.561 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestInternal.MissingDataPool (6 ms) 2022-04-22T18:13:11.251 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 32 tests from TestInternal (2345 ms total) 2022-04-22T18:13:11.252 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:13:11.252 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 51 tests from TestMirroring 2022-04-22T18:13:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:11 smithi121 bash[36902]: audit 2022-04-22T18:13:10.736454+0000 mon.a (mon.0) 3566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:11 smithi121 bash[36902]: audit 2022-04-22T18:13:11.195737+0000 mon.a (mon.0) 3567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:11 smithi115 bash[45495]: audit 2022-04-22T18:13:10.736454+0000 mon.a (mon.0) 3566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:11 smithi115 bash[45495]: audit 2022-04-22T18:13:11.195737+0000 mon.a (mon.0) 3567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:11 smithi115 bash[46934]: audit 2022-04-22T18:13:10.736454+0000 mon.a (mon.0) 3566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:11 smithi115 bash[46934]: audit 2022-04-22T18:13:11.195737+0000 mon.a (mon.0) 3567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:11.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:12 smithi121 bash[36902]: cluster 2022-04-22T18:13:11.128512+0000 mgr.y (mgr.34107) 1656 : cluster [DBG] pgmap v2425: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 12 KiB/s wr, 29 op/s 2022-04-22T18:13:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:12 smithi121 bash[36902]: cluster 2022-04-22T18:13:11.240941+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2022-04-22T18:13:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:12 smithi115 bash[45495]: cluster 2022-04-22T18:13:11.128512+0000 mgr.y (mgr.34107) 1656 : cluster [DBG] pgmap v2425: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 12 KiB/s wr, 29 op/s 2022-04-22T18:13:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:12 smithi115 bash[45495]: cluster 2022-04-22T18:13:11.240941+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2022-04-22T18:13:12.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:12 smithi115 bash[46934]: cluster 2022-04-22T18:13:11.128512+0000 mgr.y (mgr.34107) 1656 : cluster [DBG] pgmap v2425: 137 pgs: 8 creating+peering, 7 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 12 KiB/s wr, 29 op/s 2022-04-22T18:13:12.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:12 smithi115 bash[46934]: cluster 2022-04-22T18:13:11.240941+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2022-04-22T18:13:13.271 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2022-04-22T18:13:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:13 smithi121 bash[36902]: cluster 2022-04-22T18:13:12.250340+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2022-04-22T18:13:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:13 smithi121 bash[36902]: audit 2022-04-22T18:13:12.255571+0000 mon.c (mon.1) 96 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:13 smithi121 bash[36902]: audit 2022-04-22T18:13:12.256910+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:13 smithi121 bash[36902]: audit 2022-04-22T18:13:13.122687+0000 mon.a (mon.0) 3571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[45495]: cluster 2022-04-22T18:13:12.250340+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2022-04-22T18:13:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[45495]: audit 2022-04-22T18:13:12.255571+0000 mon.c (mon.1) 96 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[45495]: audit 2022-04-22T18:13:12.256910+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[45495]: audit 2022-04-22T18:13:13.122687+0000 mon.a (mon.0) 3571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[46934]: cluster 2022-04-22T18:13:12.250340+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2022-04-22T18:13:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[46934]: audit 2022-04-22T18:13:12.255571+0000 mon.c (mon.1) 96 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[46934]: audit 2022-04-22T18:13:12.256910+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[46934]: audit 2022-04-22T18:13:13.122687+0000 mon.a (mon.0) 3571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:14.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:13.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[45495]: cluster 2022-04-22T18:13:13.129395+0000 mgr.y (mgr.34107) 1657 : cluster [DBG] pgmap v2428: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[45495]: audit 2022-04-22T18:13:13.261664+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[45495]: cluster 2022-04-22T18:13:13.262033+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[45495]: audit 2022-04-22T18:13:13.849211+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[45495]: audit 2022-04-22T18:13:13.854514+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[46934]: cluster 2022-04-22T18:13:13.129395+0000 mgr.y (mgr.34107) 1657 : cluster [DBG] pgmap v2428: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[46934]: audit 2022-04-22T18:13:13.261664+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[46934]: cluster 2022-04-22T18:13:13.262033+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2022-04-22T18:13:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[46934]: audit 2022-04-22T18:13:13.849211+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:13:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:14 smithi115 bash[46934]: audit 2022-04-22T18:13:13.854514+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:13:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:14 smithi121 bash[36902]: cluster 2022-04-22T18:13:13.129395+0000 mgr.y (mgr.34107) 1657 : cluster [DBG] pgmap v2428: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:14 smithi121 bash[36902]: audit 2022-04-22T18:13:13.261664+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-12","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:14 smithi121 bash[36902]: cluster 2022-04-22T18:13:13.262033+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2022-04-22T18:13:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:14 smithi121 bash[36902]: audit 2022-04-22T18:13:13.849211+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:13:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:14 smithi121 bash[36902]: audit 2022-04-22T18:13:13.854514+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:13:15.509 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (2238 ms) 2022-04-22T18:13:15.509 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2022-04-22T18:13:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:15 smithi115 bash[45495]: cluster 2022-04-22T18:13:14.263228+0000 mon.a (mon.0) 3576 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2022-04-22T18:13:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:15 smithi115 bash[46934]: cluster 2022-04-22T18:13:14.263228+0000 mon.a (mon.0) 3576 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2022-04-22T18:13:15.719 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (211 ms) 2022-04-22T18:13:15.720 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2022-04-22T18:13:15.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:15 smithi121 bash[36902]: cluster 2022-04-22T18:13:14.263228+0000 mon.a (mon.0) 3576 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2022-04-22T18:13:15.886 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (166 ms) 2022-04-22T18:13:15.899 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2022-04-22T18:13:16.107 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (222 ms) 2022-04-22T18:13:16.108 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2022-04-22T18:13:16.298 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (190 ms) 2022-04-22T18:13:16.298 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2022-04-22T18:13:16.489 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (191 ms) 2022-04-22T18:13:16.489 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2022-04-22T18:13:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[45495]: cluster 2022-04-22T18:13:15.130508+0000 mgr.y (mgr.34107) 1658 : cluster [DBG] pgmap v2431: 137 pgs: 10 creating+peering, 9 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[45495]: cluster 2022-04-22T18:13:15.283191+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2022-04-22T18:13:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[45495]: audit 2022-04-22T18:13:15.739786+0000 mon.a (mon.0) 3578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[45495]: audit 2022-04-22T18:13:16.199023+0000 mon.a (mon.0) 3579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[46934]: cluster 2022-04-22T18:13:15.130508+0000 mgr.y (mgr.34107) 1658 : cluster [DBG] pgmap v2431: 137 pgs: 10 creating+peering, 9 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[46934]: cluster 2022-04-22T18:13:15.283191+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2022-04-22T18:13:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[46934]: audit 2022-04-22T18:13:15.739786+0000 mon.a (mon.0) 3578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:16 smithi115 bash[46934]: audit 2022-04-22T18:13:16.199023+0000 mon.a (mon.0) 3579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:16.727 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (238 ms) 2022-04-22T18:13:16.727 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2022-04-22T18:13:16.727 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:16.728 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2022-04-22T18:13:16.728 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2022-04-22T18:13:16.728 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:16.728 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableJournalingWithPeer (0 ms) 2022-04-22T18:13:16.728 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2022-04-22T18:13:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:16 smithi121 bash[36902]: cluster 2022-04-22T18:13:15.130508+0000 mgr.y (mgr.34107) 1658 : cluster [DBG] pgmap v2431: 137 pgs: 10 creating+peering, 9 unknown, 118 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:16 smithi121 bash[36902]: cluster 2022-04-22T18:13:15.283191+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2022-04-22T18:13:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:16 smithi121 bash[36902]: audit 2022-04-22T18:13:15.739786+0000 mon.a (mon.0) 3578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:16 smithi121 bash[36902]: audit 2022-04-22T18:13:16.199023+0000 mon.a (mon.0) 3579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:16.875 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (148 ms) 2022-04-22T18:13:16.875 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2022-04-22T18:13:17.046 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (169 ms) 2022-04-22T18:13:17.079 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2022-04-22T18:13:18.379 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (1335 ms) 2022-04-22T18:13:18.379 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2022-04-22T18:13:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:18 smithi121 bash[36902]: cluster 2022-04-22T18:13:17.131837+0000 mgr.y (mgr.34107) 1659 : cluster [DBG] pgmap v2433: 137 pgs: 10 creating+peering, 127 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 8.8 KiB/s wr, 92 op/s 2022-04-22T18:13:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:18 smithi121 bash[36902]: cluster 2022-04-22T18:13:17.292964+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2022-04-22T18:13:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:18 smithi121 bash[36902]: audit 2022-04-22T18:13:18.125829+0000 mon.a (mon.0) 3581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[45495]: cluster 2022-04-22T18:13:17.131837+0000 mgr.y (mgr.34107) 1659 : cluster [DBG] pgmap v2433: 137 pgs: 10 creating+peering, 127 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 8.8 KiB/s wr, 92 op/s 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[45495]: cluster 2022-04-22T18:13:17.292964+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[45495]: audit 2022-04-22T18:13:18.125829+0000 mon.a (mon.0) 3581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[46934]: cluster 2022-04-22T18:13:17.131837+0000 mgr.y (mgr.34107) 1659 : cluster [DBG] pgmap v2433: 137 pgs: 10 creating+peering, 127 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 8.8 KiB/s wr, 92 op/s 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[46934]: cluster 2022-04-22T18:13:17.292964+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2022-04-22T18:13:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:18 smithi115 bash[46934]: audit 2022-04-22T18:13:18.125829+0000 mon.a (mon.0) 3581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:19.595 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:13:19.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:19 smithi121 bash[36902]: cluster 2022-04-22T18:13:18.302678+0000 mon.a (mon.0) 3582 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2022-04-22T18:13:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:19 smithi115 bash[45495]: cluster 2022-04-22T18:13:18.302678+0000 mon.a (mon.0) 3582 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2022-04-22T18:13:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:19 smithi115 bash[46934]: cluster 2022-04-22T18:13:18.302678+0000 mon.a (mon.0) 3582 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2022-04-22T18:13:19.945 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:13:19.945 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (36m) 5m ago 50m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:13:19.951 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (49m) 6m ago 49m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:13:19.969 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (38m) 5m ago 51m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:13:19.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (37m) 6m ago 51m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:13:19.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (45m) 6m ago 55m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:13:19.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (46m) 5m ago 57m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:13:19.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (45m) 5m ago 57m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:13:19.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (44m) 6m ago 55m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:13:19.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (44m) 5m ago 55m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:13:19.980 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (37m) 5m ago 50m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:13:19.981 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (37m) 6m ago 50m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:13:19.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (43m) 5m ago 55m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:13:19.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (42m) 5m ago 54m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:13:19.982 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (41m) 5m ago 54m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:13:19.982 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (41m) 5m ago 53m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:13:19.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (40m) 6m ago 53m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:13:19.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (40m) 6m ago 52m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:13:19.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (39m) 6m ago 52m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:13:19.983 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (38m) 6m ago 51m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:13:19.984 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (37m) 6m ago 50m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:13:20.351 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:13:20.352 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:13:20.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:13:20.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:20.352 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:13:20.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:13:20.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:13:20.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:13:20.363 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:13:20.363 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:13:20.397 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (2019 ms) 2022-04-22T18:13:20.398 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2022-04-22T18:13:20.578 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (180 ms) 2022-04-22T18:13:20.578 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2022-04-22T18:13:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:20 smithi121 bash[36902]: cluster 2022-04-22T18:13:19.132948+0000 mgr.y (mgr.34107) 1660 : cluster [DBG] pgmap v2436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 248 KiB/s rd, 27 KiB/s wr, 356 op/s 2022-04-22T18:13:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:20 smithi121 bash[36902]: cluster 2022-04-22T18:13:19.311544+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2022-04-22T18:13:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:20 smithi121 bash[36902]: audit 2022-04-22T18:13:19.574351+0000 mgr.y (mgr.34107) 1661 : audit [DBG] from='client.46855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:20.768 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (191 ms) 2022-04-22T18:13:20.769 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2022-04-22T18:13:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[46934]: cluster 2022-04-22T18:13:19.132948+0000 mgr.y (mgr.34107) 1660 : cluster [DBG] pgmap v2436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 248 KiB/s rd, 27 KiB/s wr, 356 op/s 2022-04-22T18:13:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[46934]: cluster 2022-04-22T18:13:19.311544+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2022-04-22T18:13:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[46934]: audit 2022-04-22T18:13:19.574351+0000 mgr.y (mgr.34107) 1661 : audit [DBG] from='client.46855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[45495]: cluster 2022-04-22T18:13:19.132948+0000 mgr.y (mgr.34107) 1660 : cluster [DBG] pgmap v2436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 248 KiB/s rd, 27 KiB/s wr, 356 op/s 2022-04-22T18:13:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[45495]: cluster 2022-04-22T18:13:19.311544+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2022-04-22T18:13:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:20 smithi115 bash[45495]: audit 2022-04-22T18:13:19.574351+0000 mgr.y (mgr.34107) 1661 : audit [DBG] from='client.46855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:20.999 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (230 ms) 2022-04-22T18:13:20.999 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2022-04-22T18:13:21.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (167 ms) 2022-04-22T18:13:21.167 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2022-04-22T18:13:21.325 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (160 ms) 2022-04-22T18:13:21.326 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2022-04-22T18:13:21.517 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (191 ms) 2022-04-22T18:13:21.518 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2022-04-22T18:13:21.734 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (218 ms) 2022-04-22T18:13:21.778 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2022-04-22T18:13:21.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:21 smithi121 bash[36902]: audit 2022-04-22T18:13:19.917184+0000 mgr.y (mgr.34107) 1662 : audit [DBG] from='client.36933 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:21.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:21 smithi121 bash[36902]: cluster 2022-04-22T18:13:20.320717+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:21 smithi121 bash[36902]: audit 2022-04-22T18:13:20.344882+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.115:0/1241690222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:21 smithi121 bash[36902]: audit 2022-04-22T18:13:20.742063+0000 mon.a (mon.0) 3586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:21 smithi121 bash[36902]: audit 2022-04-22T18:13:21.202212+0000 mon.a (mon.0) 3587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[45495]: audit 2022-04-22T18:13:19.917184+0000 mgr.y (mgr.34107) 1662 : audit [DBG] from='client.36933 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[45495]: cluster 2022-04-22T18:13:20.320717+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[45495]: audit 2022-04-22T18:13:20.344882+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.115:0/1241690222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[45495]: audit 2022-04-22T18:13:20.742063+0000 mon.a (mon.0) 3586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:21.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[45495]: audit 2022-04-22T18:13:21.202212+0000 mon.a (mon.0) 3587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:21.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[46934]: audit 2022-04-22T18:13:19.917184+0000 mgr.y (mgr.34107) 1662 : audit [DBG] from='client.36933 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:21.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[46934]: cluster 2022-04-22T18:13:20.320717+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2022-04-22T18:13:21.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[46934]: audit 2022-04-22T18:13:20.344882+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.115:0/1241690222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:21.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[46934]: audit 2022-04-22T18:13:20.742063+0000 mon.a (mon.0) 3586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:21.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[46934]: audit 2022-04-22T18:13:21.202212+0000 mon.a (mon.0) 3587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:21.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:22 smithi121 bash[36902]: cluster 2022-04-22T18:13:21.133866+0000 mgr.y (mgr.34107) 1663 : cluster [DBG] pgmap v2439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 47 KiB/s wr, 502 op/s 2022-04-22T18:13:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:22 smithi115 bash[45495]: cluster 2022-04-22T18:13:21.133866+0000 mgr.y (mgr.34107) 1663 : cluster [DBG] pgmap v2439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 47 KiB/s wr, 502 op/s 2022-04-22T18:13:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:22 smithi115 bash[46934]: cluster 2022-04-22T18:13:21.133866+0000 mgr.y (mgr.34107) 1663 : cluster [DBG] pgmap v2439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 47 KiB/s wr, 502 op/s 2022-04-22T18:13:23.467 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (1732 ms) 2022-04-22T18:13:23.467 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2022-04-22T18:13:23.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:23 smithi115 bash[45495]: cluster 2022-04-22T18:13:22.333556+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2022-04-22T18:13:23.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:23 smithi115 bash[45495]: audit 2022-04-22T18:13:23.128849+0000 mon.a (mon.0) 3589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:23.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:23 smithi115 bash[46934]: cluster 2022-04-22T18:13:22.333556+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2022-04-22T18:13:23.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:23 smithi115 bash[46934]: audit 2022-04-22T18:13:23.128849+0000 mon.a (mon.0) 3589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:23.708 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (241 ms) 2022-04-22T18:13:23.708 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2022-04-22T18:13:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:23 smithi121 bash[36902]: cluster 2022-04-22T18:13:22.333556+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2022-04-22T18:13:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:23 smithi121 bash[36902]: audit 2022-04-22T18:13:23.128849+0000 mon.a (mon.0) 3589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:23.929 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (220 ms) 2022-04-22T18:13:23.930 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2022-04-22T18:13:24.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:23.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:24.152 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (224 ms) 2022-04-22T18:13:24.152 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2022-04-22T18:13:24.496 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (345 ms) 2022-04-22T18:13:24.496 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2022-04-22T18:13:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:24 smithi121 bash[36902]: cluster 2022-04-22T18:13:23.134834+0000 mgr.y (mgr.34107) 1664 : cluster [DBG] pgmap v2441: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 339 KiB/s rd, 40 KiB/s wr, 478 op/s 2022-04-22T18:13:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:24 smithi121 bash[36902]: cluster 2022-04-22T18:13:23.351930+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2022-04-22T18:13:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:24 smithi115 bash[45495]: cluster 2022-04-22T18:13:23.134834+0000 mgr.y (mgr.34107) 1664 : cluster [DBG] pgmap v2441: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 339 KiB/s rd, 40 KiB/s wr, 478 op/s 2022-04-22T18:13:24.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:24 smithi115 bash[45495]: cluster 2022-04-22T18:13:23.351930+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2022-04-22T18:13:24.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:24 smithi115 bash[46934]: cluster 2022-04-22T18:13:23.134834+0000 mgr.y (mgr.34107) 1664 : cluster [DBG] pgmap v2441: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 339 KiB/s rd, 40 KiB/s wr, 478 op/s 2022-04-22T18:13:24.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:24 smithi115 bash[46934]: cluster 2022-04-22T18:13:23.351930+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2022-04-22T18:13:24.862 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (365 ms) 2022-04-22T18:13:24.862 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2022-04-22T18:13:25.151 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (289 ms) 2022-04-22T18:13:25.152 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2022-04-22T18:13:25.458 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (307 ms) 2022-04-22T18:13:25.458 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2022-04-22T18:13:25.788 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (331 ms) 2022-04-22T18:13:25.788 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2022-04-22T18:13:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:25 smithi115 bash[46934]: audit 2022-04-22T18:13:24.525200+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:25 smithi115 bash[45495]: audit 2022-04-22T18:13:24.525200+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:25 smithi121 bash[36902]: audit 2022-04-22T18:13:24.525200+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:26.175 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (387 ms) 2022-04-22T18:13:26.175 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2022-04-22T18:13:26.349 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (174 ms) 2022-04-22T18:13:26.350 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2022-04-22T18:13:26.511 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (161 ms) 2022-04-22T18:13:26.511 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2022-04-22T18:13:26.666 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (155 ms) 2022-04-22T18:13:26.666 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2022-04-22T18:13:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[46934]: cluster 2022-04-22T18:13:25.135780+0000 mgr.y (mgr.34107) 1665 : cluster [DBG] pgmap v2443: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 373 KiB/s rd, 52 KiB/s wr, 532 op/s 2022-04-22T18:13:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[46934]: audit 2022-04-22T18:13:25.744873+0000 mon.a (mon.0) 3592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[46934]: cluster 2022-04-22T18:13:26.156009+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[46934]: audit 2022-04-22T18:13:26.205592+0000 mon.a (mon.0) 3594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[45495]: cluster 2022-04-22T18:13:25.135780+0000 mgr.y (mgr.34107) 1665 : cluster [DBG] pgmap v2443: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 373 KiB/s rd, 52 KiB/s wr, 532 op/s 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[45495]: audit 2022-04-22T18:13:25.744873+0000 mon.a (mon.0) 3592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[45495]: cluster 2022-04-22T18:13:26.156009+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2022-04-22T18:13:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:26 smithi115 bash[45495]: audit 2022-04-22T18:13:26.205592+0000 mon.a (mon.0) 3594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:26.905 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (238 ms) 2022-04-22T18:13:26.905 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2022-04-22T18:13:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:26 smithi121 bash[36902]: cluster 2022-04-22T18:13:25.135780+0000 mgr.y (mgr.34107) 1665 : cluster [DBG] pgmap v2443: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 373 KiB/s rd, 52 KiB/s wr, 532 op/s 2022-04-22T18:13:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:26 smithi121 bash[36902]: audit 2022-04-22T18:13:25.744873+0000 mon.a (mon.0) 3592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:26 smithi121 bash[36902]: cluster 2022-04-22T18:13:26.156009+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2022-04-22T18:13:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:26 smithi121 bash[36902]: audit 2022-04-22T18:13:26.205592+0000 mon.a (mon.0) 3594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:27.146 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (241 ms) 2022-04-22T18:13:27.146 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2022-04-22T18:13:27.376 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (230 ms) 2022-04-22T18:13:27.376 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2022-04-22T18:13:27.618 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (242 ms) 2022-04-22T18:13:27.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.MirrorStatusList 2022-04-22T18:13:27.784 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.MirrorStatusList (166 ms) 2022-04-22T18:13:27.784 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2022-04-22T18:13:27.982 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.RemoveBootstrapped (199 ms) 2022-04-22T18:13:27.982 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2022-04-22T18:13:28.679 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.AioPromoteDemote (697 ms) 2022-04-22T18:13:28.679 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.AioGetInfo 2022-04-22T18:13:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:28 smithi115 bash[45495]: cluster 2022-04-22T18:13:27.136661+0000 mgr.y (mgr.34107) 1666 : cluster [DBG] pgmap v2445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 60 KiB/s wr, 677 op/s 2022-04-22T18:13:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:28 smithi115 bash[45495]: audit 2022-04-22T18:13:28.131777+0000 mon.a (mon.0) 3595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:28 smithi115 bash[46934]: cluster 2022-04-22T18:13:27.136661+0000 mgr.y (mgr.34107) 1666 : cluster [DBG] pgmap v2445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 60 KiB/s wr, 677 op/s 2022-04-22T18:13:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:28 smithi115 bash[46934]: audit 2022-04-22T18:13:28.131777+0000 mon.a (mon.0) 3595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:28 smithi121 bash[36902]: cluster 2022-04-22T18:13:27.136661+0000 mgr.y (mgr.34107) 1666 : cluster [DBG] pgmap v2445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 60 KiB/s wr, 677 op/s 2022-04-22T18:13:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:28 smithi121 bash[36902]: audit 2022-04-22T18:13:28.131777+0000 mon.a (mon.0) 3595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:29.438 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.AioGetInfo (758 ms) 2022-04-22T18:13:29.438 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.AioGetStatus 2022-04-22T18:13:29.898 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.AioGetStatus (460 ms) 2022-04-22T18:13:29.922 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.SiteName 2022-04-22T18:13:29.924 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.SiteName (12 ms) 2022-04-22T18:13:29.925 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.Bootstrap 2022-04-22T18:13:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: cluster 2022-04-22T18:13:29.137749+0000 mgr.y (mgr.34107) 1667 : cluster [DBG] pgmap v2446: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 700 KiB/s rd, 85 KiB/s wr, 964 op/s 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.900084+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.901134+0000 mon.c (mon.1) 97 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.901622+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.902061+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.905938+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:13:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:30 smithi121 bash[36902]: audit 2022-04-22T18:13:29.906636+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: cluster 2022-04-22T18:13:29.137749+0000 mgr.y (mgr.34107) 1667 : cluster [DBG] pgmap v2446: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 700 KiB/s rd, 85 KiB/s wr, 964 op/s 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.900084+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.901134+0000 mon.c (mon.1) 97 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.901622+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.902061+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.905938+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[46934]: audit 2022-04-22T18:13:29.906636+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: cluster 2022-04-22T18:13:29.137749+0000 mgr.y (mgr.34107) 1667 : cluster [DBG] pgmap v2446: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 700 KiB/s rd, 85 KiB/s wr, 964 op/s 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.900084+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.901134+0000 mon.c (mon.1) 97 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.901622+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.902061+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.905938+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:13:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:30 smithi115 bash[45495]: audit 2022-04-22T18:13:29.906636+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:31.623 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.Bootstrap (1712 ms) 2022-04-22T18:13:31.623 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.PeerDirection 2022-04-22T18:13:31.639 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.PeerDirection (17 ms) 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.Snapshot 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.Snapshot (0 ms) 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.SnapshotRemoveOnDisable 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.SnapshotRemoveOnDisable (0 ms) 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.SnapshotUnlinkPeer 2022-04-22T18:13:31.640 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.SnapshotUnlinkPeer (0 ms) 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.SnapshotImageState 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.SnapshotImageState (0 ms) 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.SnapshotPromoteDemote 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.SnapshotPromoteDemote (0 ms) 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroring.AioSnapshotCreate 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroring.AioSnapshotCreate (0 ms) 2022-04-22T18:13:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:31 smithi121 bash[36902]: audit 2022-04-22T18:13:30.747925+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:31 smithi121 bash[36902]: cluster 2022-04-22T18:13:31.154822+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2022-04-22T18:13:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:31 smithi121 bash[36902]: audit 2022-04-22T18:13:31.208729+0000 mon.a (mon.0) 3601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:31.776Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[45495]: audit 2022-04-22T18:13:30.747925+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[45495]: cluster 2022-04-22T18:13:31.154822+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2022-04-22T18:13:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[45495]: audit 2022-04-22T18:13:31.208729+0000 mon.a (mon.0) 3601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[46934]: audit 2022-04-22T18:13:30.747925+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[46934]: cluster 2022-04-22T18:13:31.154822+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2022-04-22T18:13:32.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:31 smithi115 bash[46934]: audit 2022-04-22T18:13:31.208729+0000 mon.a (mon.0) 3601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:32.649 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 51 tests from TestMirroring (18369 ms total) 2022-04-22T18:13:32.649 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:13:32.650 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 31 tests from TestDeepCopy 2022-04-22T18:13:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: cluster 2022-04-22T18:13:31.138789+0000 mgr.y (mgr.34107) 1668 : cluster [DBG] pgmap v2447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 832 KiB/s rd, 125 KiB/s wr, 1.16k op/s 2022-04-22T18:13:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.589461+0000 mon.c (mon.1) 100 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:13:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.595637+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:33.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.596819+0000 mon.c (mon.1) 101 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.597427+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.602884+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2022-04-22T18:13:33.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.603788+0000 mon.c (mon.1) 102 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:33.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.625790+0000 mon.c (mon.1) 103 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.626275+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:32 smithi121 bash[36902]: audit 2022-04-22T18:13:31.631143+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]': finished 2022-04-22T18:13:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: cluster 2022-04-22T18:13:31.138789+0000 mgr.y (mgr.34107) 1668 : cluster [DBG] pgmap v2447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 832 KiB/s rd, 125 KiB/s wr, 1.16k op/s 2022-04-22T18:13:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.589461+0000 mon.c (mon.1) 100 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:13:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.595637+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.596819+0000 mon.c (mon.1) 101 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.597427+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.602884+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2022-04-22T18:13:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.603788+0000 mon.c (mon.1) 102 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.625790+0000 mon.c (mon.1) 103 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.626275+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[45495]: audit 2022-04-22T18:13:31.631143+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]': finished 2022-04-22T18:13:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: cluster 2022-04-22T18:13:31.138789+0000 mgr.y (mgr.34107) 1668 : cluster [DBG] pgmap v2447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 832 KiB/s rd, 125 KiB/s wr, 1.16k op/s 2022-04-22T18:13:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.589461+0000 mon.c (mon.1) 100 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:13:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.595637+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.596819+0000 mon.c (mon.1) 101 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.597427+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.602884+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2022-04-22T18:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.603788+0000 mon.c (mon.1) 102 : audit [DBG] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:13:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.625790+0000 mon.c (mon.1) 103 : audit [INF] from='client.? 172.21.15.115:0/2028310712' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.626275+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]: dispatch 2022-04-22T18:13:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:32 smithi115 bash[46934]: audit 2022-04-22T18:13:31.631143+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/76/27216e1d-aaa4-454d-8dc4-d8e0deb68133"}]': finished 2022-04-22T18:13:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:33 smithi121 bash[36902]: cluster 2022-04-22T18:13:32.637603+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2022-04-22T18:13:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:33 smithi121 bash[36902]: audit 2022-04-22T18:13:33.134611+0000 mon.a (mon.0) 3608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:33 smithi121 bash[36902]: cluster 2022-04-22T18:13:33.139428+0000 mgr.y (mgr.34107) 1669 : cluster [DBG] pgmap v2450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:33.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[45495]: cluster 2022-04-22T18:13:32.637603+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2022-04-22T18:13:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[45495]: audit 2022-04-22T18:13:33.134611+0000 mon.a (mon.0) 3608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[45495]: cluster 2022-04-22T18:13:33.139428+0000 mgr.y (mgr.34107) 1669 : cluster [DBG] pgmap v2450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:34.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[46934]: cluster 2022-04-22T18:13:32.637603+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2022-04-22T18:13:34.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[46934]: audit 2022-04-22T18:13:33.134611+0000 mon.a (mon.0) 3608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:34.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:33 smithi115 bash[46934]: cluster 2022-04-22T18:13:33.139428+0000 mgr.y (mgr.34107) 1669 : cluster [DBG] pgmap v2450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:34.659 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Empty 2022-04-22T18:13:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:34 smithi121 bash[36902]: cluster 2022-04-22T18:13:33.647724+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2022-04-22T18:13:35.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:34 smithi121 bash[36902]: audit 2022-04-22T18:13:33.663506+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:35.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:34 smithi115 bash[45495]: cluster 2022-04-22T18:13:33.647724+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2022-04-22T18:13:35.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:34 smithi115 bash[45495]: audit 2022-04-22T18:13:33.663506+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:34 smithi115 bash[46934]: cluster 2022-04-22T18:13:33.647724+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2022-04-22T18:13:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:34 smithi115 bash[46934]: audit 2022-04-22T18:13:33.663506+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:13:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:35 smithi121 bash[36902]: audit 2022-04-22T18:13:34.651111+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:35 smithi121 bash[36902]: cluster 2022-04-22T18:13:34.651272+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2022-04-22T18:13:36.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:35 smithi121 bash[36902]: cluster 2022-04-22T18:13:35.140284+0000 mgr.y (mgr.34107) 1670 : cluster [DBG] pgmap v2453: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:36.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[45495]: audit 2022-04-22T18:13:34.651111+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:36.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[45495]: cluster 2022-04-22T18:13:34.651272+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2022-04-22T18:13:36.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[45495]: cluster 2022-04-22T18:13:35.140284+0000 mgr.y (mgr.34107) 1670 : cluster [DBG] pgmap v2453: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[46934]: audit 2022-04-22T18:13:34.651111+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? 172.21.15.115:0/3256766543' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-13","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:13:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[46934]: cluster 2022-04-22T18:13:34.651272+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e1306: 8 total, 8 up, 8 in 2022-04-22T18:13:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:35 smithi115 bash[46934]: cluster 2022-04-22T18:13:35.140284+0000 mgr.y (mgr.34107) 1670 : cluster [DBG] pgmap v2453: 137 pgs: 6 creating+peering, 22 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:13:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:36 smithi121 bash[36902]: cluster 2022-04-22T18:13:35.669907+0000 mon.a (mon.0) 3613 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2022-04-22T18:13:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:36 smithi121 bash[36902]: audit 2022-04-22T18:13:35.751564+0000 mon.a (mon.0) 3614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:36 smithi121 bash[36902]: audit 2022-04-22T18:13:36.212090+0000 mon.a (mon.0) 3615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[45495]: cluster 2022-04-22T18:13:35.669907+0000 mon.a (mon.0) 3613 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2022-04-22T18:13:37.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[45495]: audit 2022-04-22T18:13:35.751564+0000 mon.a (mon.0) 3614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:37.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[45495]: audit 2022-04-22T18:13:36.212090+0000 mon.a (mon.0) 3615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[46934]: cluster 2022-04-22T18:13:35.669907+0000 mon.a (mon.0) 3613 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2022-04-22T18:13:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[46934]: audit 2022-04-22T18:13:35.751564+0000 mon.a (mon.0) 3614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:36 smithi115 bash[46934]: audit 2022-04-22T18:13:36.212090+0000 mon.a (mon.0) 3615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:38.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:37 smithi121 bash[36902]: cluster 2022-04-22T18:13:36.679660+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2022-04-22T18:13:38.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:37 smithi121 bash[36902]: cluster 2022-04-22T18:13:37.141374+0000 mgr.y (mgr.34107) 1671 : cluster [DBG] pgmap v2456: 137 pgs: 6 creating+peering, 9 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:13:38.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:37 smithi115 bash[45495]: cluster 2022-04-22T18:13:36.679660+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2022-04-22T18:13:38.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:37 smithi115 bash[45495]: cluster 2022-04-22T18:13:37.141374+0000 mgr.y (mgr.34107) 1671 : cluster [DBG] pgmap v2456: 137 pgs: 6 creating+peering, 9 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:13:38.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:37 smithi115 bash[46934]: cluster 2022-04-22T18:13:36.679660+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2022-04-22T18:13:38.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:37 smithi115 bash[46934]: cluster 2022-04-22T18:13:37.141374+0000 mgr.y (mgr.34107) 1671 : cluster [DBG] pgmap v2456: 137 pgs: 6 creating+peering, 9 unknown, 122 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:13:38.859 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Empty (4200 ms) 2022-04-22T18:13:38.859 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.NoSnaps 2022-04-22T18:13:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:38 smithi121 bash[36902]: cluster 2022-04-22T18:13:37.696685+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2022-04-22T18:13:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:38 smithi121 bash[36902]: audit 2022-04-22T18:13:38.137418+0000 mon.a (mon.0) 3618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:38 smithi115 bash[46934]: cluster 2022-04-22T18:13:37.696685+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2022-04-22T18:13:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:38 smithi115 bash[46934]: audit 2022-04-22T18:13:38.137418+0000 mon.a (mon.0) 3618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:39.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:38 smithi115 bash[45495]: cluster 2022-04-22T18:13:37.696685+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2022-04-22T18:13:39.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:38 smithi115 bash[45495]: audit 2022-04-22T18:13:38.137418+0000 mon.a (mon.0) 3618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:39.185 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.NoSnaps (327 ms) 2022-04-22T18:13:39.186 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Snaps 2022-04-22T18:13:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:39 smithi121 bash[36902]: cluster 2022-04-22T18:13:38.700600+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2022-04-22T18:13:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:39 smithi121 bash[36902]: cluster 2022-04-22T18:13:39.142419+0000 mgr.y (mgr.34107) 1672 : cluster [DBG] pgmap v2459: 137 pgs: 137 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2022-04-22T18:13:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:39 smithi115 bash[45495]: cluster 2022-04-22T18:13:38.700600+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2022-04-22T18:13:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:39 smithi115 bash[45495]: cluster 2022-04-22T18:13:39.142419+0000 mgr.y (mgr.34107) 1672 : cluster [DBG] pgmap v2459: 137 pgs: 137 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2022-04-22T18:13:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:39 smithi115 bash[46934]: cluster 2022-04-22T18:13:38.700600+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2022-04-22T18:13:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:39 smithi115 bash[46934]: cluster 2022-04-22T18:13:39.142419+0000 mgr.y (mgr.34107) 1672 : cluster [DBG] pgmap v2459: 137 pgs: 137 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 3 op/s 2022-04-22T18:13:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:40 smithi121 bash[36902]: cluster 2022-04-22T18:13:39.720092+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2022-04-22T18:13:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:40 smithi115 bash[45495]: cluster 2022-04-22T18:13:39.720092+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2022-04-22T18:13:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:40 smithi115 bash[46934]: cluster 2022-04-22T18:13:39.720092+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2022-04-22T18:13:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:41 smithi121 bash[36902]: cluster 2022-04-22T18:13:40.724831+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2022-04-22T18:13:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:41 smithi121 bash[36902]: audit 2022-04-22T18:13:40.754741+0000 mon.a (mon.0) 3622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:41 smithi121 bash[36902]: cluster 2022-04-22T18:13:41.143429+0000 mgr.y (mgr.34107) 1673 : cluster [DBG] pgmap v2462: 137 pgs: 137 active+clean; 25 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 9 op/s 2022-04-22T18:13:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:41 smithi121 bash[36902]: audit 2022-04-22T18:13:41.215770+0000 mon.a (mon.0) 3623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:41.776Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[45495]: cluster 2022-04-22T18:13:40.724831+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[45495]: audit 2022-04-22T18:13:40.754741+0000 mon.a (mon.0) 3622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[45495]: cluster 2022-04-22T18:13:41.143429+0000 mgr.y (mgr.34107) 1673 : cluster [DBG] pgmap v2462: 137 pgs: 137 active+clean; 25 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 9 op/s 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[45495]: audit 2022-04-22T18:13:41.215770+0000 mon.a (mon.0) 3623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[46934]: cluster 2022-04-22T18:13:40.724831+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2022-04-22T18:13:42.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[46934]: audit 2022-04-22T18:13:40.754741+0000 mon.a (mon.0) 3622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[46934]: cluster 2022-04-22T18:13:41.143429+0000 mgr.y (mgr.34107) 1673 : cluster [DBG] pgmap v2462: 137 pgs: 137 active+clean; 25 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 9 op/s 2022-04-22T18:13:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:41 smithi115 bash[46934]: audit 2022-04-22T18:13:41.215770+0000 mon.a (mon.0) 3623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:42 smithi121 bash[36902]: cluster 2022-04-22T18:13:41.740551+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2022-04-22T18:13:43.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:42 smithi115 bash[45495]: cluster 2022-04-22T18:13:41.740551+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2022-04-22T18:13:43.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:42 smithi115 bash[46934]: cluster 2022-04-22T18:13:41.740551+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2022-04-22T18:13:43.182 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Snaps (3996 ms) 2022-04-22T18:13:43.182 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.SnapDiscard 2022-04-22T18:13:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:43 smithi121 bash[36902]: cluster 2022-04-22T18:13:42.748180+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2022-04-22T18:13:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:43 smithi121 bash[36902]: audit 2022-04-22T18:13:43.140399+0000 mon.a (mon.0) 3626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:43 smithi121 bash[36902]: cluster 2022-04-22T18:13:43.144230+0000 mgr.y (mgr.34107) 1674 : cluster [DBG] pgmap v2465: 137 pgs: 137 active+clean; 45 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 43 op/s 2022-04-22T18:13:44.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:43.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:44.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[45495]: cluster 2022-04-22T18:13:42.748180+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2022-04-22T18:13:44.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[45495]: audit 2022-04-22T18:13:43.140399+0000 mon.a (mon.0) 3626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:44.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[45495]: cluster 2022-04-22T18:13:43.144230+0000 mgr.y (mgr.34107) 1674 : cluster [DBG] pgmap v2465: 137 pgs: 137 active+clean; 45 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 43 op/s 2022-04-22T18:13:44.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[46934]: cluster 2022-04-22T18:13:42.748180+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2022-04-22T18:13:44.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[46934]: audit 2022-04-22T18:13:43.140399+0000 mon.a (mon.0) 3626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:44.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:43 smithi115 bash[46934]: cluster 2022-04-22T18:13:43.144230+0000 mgr.y (mgr.34107) 1674 : cluster [DBG] pgmap v2465: 137 pgs: 137 active+clean; 45 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 43 op/s 2022-04-22T18:13:44.957 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.SnapDiscard (1776 ms) 2022-04-22T18:13:44.958 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneDiscard 2022-04-22T18:13:44.985 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:44 smithi121 bash[36902]: cluster 2022-04-22T18:13:43.759463+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2022-04-22T18:13:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:44 smithi121 bash[36902]: audit 2022-04-22T18:13:44.548633+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:45.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:44 smithi115 bash[45495]: cluster 2022-04-22T18:13:43.759463+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2022-04-22T18:13:45.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:44 smithi115 bash[45495]: audit 2022-04-22T18:13:44.548633+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:45.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:44 smithi115 bash[46934]: cluster 2022-04-22T18:13:43.759463+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2022-04-22T18:13:45.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:44 smithi115 bash[46934]: audit 2022-04-22T18:13:44.548633+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:45.112 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneDiscard (155 ms) 2022-04-22T18:13:45.113 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneShrink 2022-04-22T18:13:45.136 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.265 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneShrink (152 ms) 2022-04-22T18:13:45.285 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneExpand 2022-04-22T18:13:45.288 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneExpand (148 ms) 2022-04-22T18:13:45.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneHideParent 2022-04-22T18:13:45.438 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.564 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneHideParent (151 ms) 2022-04-22T18:13:45.565 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Clone 2022-04-22T18:13:45.589 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Clone (158 ms) 2022-04-22T18:13:45.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneFlatten 2022-04-22T18:13:45.748 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:45.873 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneFlatten (152 ms) 2022-04-22T18:13:45.873 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Stress 2022-04-22T18:13:45.895 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 54435552~1956920 2022-04-22T18:13:45.907 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 25895425~7382472 2022-04-22T18:13:45.918 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 61071632~4832983 2022-04-22T18:13:45.929 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 55107581~3727987 2022-04-22T18:13:45.932 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 52487869~1648112 2022-04-22T18:13:45.939 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 45996738~3412594 2022-04-22T18:13:45.945 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 63134943~2930678 2022-04-22T18:13:45.954 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 77545667~6340413 2022-04-22T18:13:45.956 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap0 2022-04-22T18:13:46.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[45495]: cluster 2022-04-22T18:13:44.768599+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e1316: 8 total, 8 up, 8 in 2022-04-22T18:13:46.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[45495]: cluster 2022-04-22T18:13:45.145241+0000 mgr.y (mgr.34107) 1675 : cluster [DBG] pgmap v2468: 137 pgs: 137 active+clean; 65 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 10 MiB/s wr, 78 op/s 2022-04-22T18:13:46.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[45495]: audit 2022-04-22T18:13:45.757483+0000 mon.a (mon.0) 3630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[46934]: cluster 2022-04-22T18:13:44.768599+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e1316: 8 total, 8 up, 8 in 2022-04-22T18:13:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[46934]: cluster 2022-04-22T18:13:45.145241+0000 mgr.y (mgr.34107) 1675 : cluster [DBG] pgmap v2468: 137 pgs: 137 active+clean; 65 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 10 MiB/s wr, 78 op/s 2022-04-22T18:13:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:45 smithi115 bash[46934]: audit 2022-04-22T18:13:45.757483+0000 mon.a (mon.0) 3630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:46.182 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 72756512~4694722 2022-04-22T18:13:46.182 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 78513615~5274688 2022-04-22T18:13:46.183 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 79004579~1275411 2022-04-22T18:13:46.205 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 71586466~1762133 2022-04-22T18:13:46.206 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 67889748~2213735 2022-04-22T18:13:46.206 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 77856077~4844534 2022-04-22T18:13:46.209 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 11704964~425287 2022-04-22T18:13:46.212 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 54860839~4315526 2022-04-22T18:13:46.218 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:13:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:45 smithi121 bash[36902]: cluster 2022-04-22T18:13:44.768599+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e1316: 8 total, 8 up, 8 in 2022-04-22T18:13:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:45 smithi121 bash[36902]: cluster 2022-04-22T18:13:45.145241+0000 mgr.y (mgr.34107) 1675 : cluster [DBG] pgmap v2468: 137 pgs: 137 active+clean; 65 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 10 MiB/s wr, 78 op/s 2022-04-22T18:13:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:45 smithi121 bash[36902]: audit 2022-04-22T18:13:45.757483+0000 mon.a (mon.0) 3630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:47.201 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 68082430~5045127 2022-04-22T18:13:47.202 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 17161723~4661597 2022-04-22T18:13:47.205 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 65029965~5749499 2022-04-22T18:13:47.232 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 42716525~2540728 2022-04-22T18:13:47.233 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 22248622~1974799 2022-04-22T18:13:47.233 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 25092428~5188172 2022-04-22T18:13:47.233 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 46308453~1566386 2022-04-22T18:13:47.234 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 40768466~4621327 2022-04-22T18:13:47.251 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:13:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:47 smithi121 bash[36902]: cluster 2022-04-22T18:13:46.165732+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2022-04-22T18:13:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:47 smithi121 bash[36902]: audit 2022-04-22T18:13:46.219307+0000 mon.a (mon.0) 3632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:47.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:47 smithi115 bash[45495]: cluster 2022-04-22T18:13:46.165732+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2022-04-22T18:13:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:47 smithi115 bash[45495]: audit 2022-04-22T18:13:46.219307+0000 mon.a (mon.0) 3632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:47 smithi115 bash[46934]: cluster 2022-04-22T18:13:46.165732+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1317: 8 total, 8 up, 8 in 2022-04-22T18:13:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:47 smithi115 bash[46934]: audit 2022-04-22T18:13:46.219307+0000 mon.a (mon.0) 3632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:48.204 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167021022 2022-04-22T18:13:48.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:48 smithi121 bash[36902]: cluster 2022-04-22T18:13:47.146304+0000 mgr.y (mgr.34107) 1676 : cluster [DBG] pgmap v2470: 137 pgs: 137 active+clean; 69 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 8.3 MiB/s wr, 122 op/s 2022-04-22T18:13:48.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:48 smithi121 bash[36902]: cluster 2022-04-22T18:13:47.183718+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1318: 8 total, 8 up, 8 in 2022-04-22T18:13:48.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:48 smithi121 bash[36902]: audit 2022-04-22T18:13:48.143414+0000 mon.a (mon.0) 3634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:48.708 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 43570999~100838 2022-04-22T18:13:48.708 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 117386034~1010995 2022-04-22T18:13:48.708 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 68023516~6972336 2022-04-22T18:13:48.708 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 6951438~1034367 2022-04-22T18:13:48.708 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 146079831~2217167 2022-04-22T18:13:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 34916735~5257860 2022-04-22T18:13:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 126866271~6660943 2022-04-22T18:13:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 90575702~7719925 2022-04-22T18:13:48.709 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap3 2022-04-22T18:13:48.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[45495]: cluster 2022-04-22T18:13:47.146304+0000 mgr.y (mgr.34107) 1676 : cluster [DBG] pgmap v2470: 137 pgs: 137 active+clean; 69 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 8.3 MiB/s wr, 122 op/s 2022-04-22T18:13:48.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[45495]: cluster 2022-04-22T18:13:47.183718+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1318: 8 total, 8 up, 8 in 2022-04-22T18:13:48.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[45495]: audit 2022-04-22T18:13:48.143414+0000 mon.a (mon.0) 3634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:48.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[46934]: cluster 2022-04-22T18:13:47.146304+0000 mgr.y (mgr.34107) 1676 : cluster [DBG] pgmap v2470: 137 pgs: 137 active+clean; 69 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 8.3 MiB/s wr, 122 op/s 2022-04-22T18:13:48.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[46934]: cluster 2022-04-22T18:13:47.183718+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1318: 8 total, 8 up, 8 in 2022-04-22T18:13:48.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:48 smithi115 bash[46934]: audit 2022-04-22T18:13:48.143414+0000 mon.a (mon.0) 3634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:49 smithi121 bash[36902]: cluster 2022-04-22T18:13:48.193615+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2022-04-22T18:13:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:49 smithi115 bash[45495]: cluster 2022-04-22T18:13:48.193615+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2022-04-22T18:13:49.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:49 smithi115 bash[46934]: cluster 2022-04-22T18:13:48.193615+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e1319: 8 total, 8 up, 8 in 2022-04-22T18:13:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:50 smithi121 bash[36902]: cluster 2022-04-22T18:13:49.147381+0000 mgr.y (mgr.34107) 1677 : cluster [DBG] pgmap v2473: 137 pgs: 137 active+clean; 100 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 MiB/s rd, 7.2 MiB/s wr, 131 op/s 2022-04-22T18:13:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:50 smithi121 bash[36902]: cluster 2022-04-22T18:13:49.200464+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1320: 8 total, 8 up, 8 in 2022-04-22T18:13:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:50 smithi115 bash[45495]: cluster 2022-04-22T18:13:49.147381+0000 mgr.y (mgr.34107) 1677 : cluster [DBG] pgmap v2473: 137 pgs: 137 active+clean; 100 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 MiB/s rd, 7.2 MiB/s wr, 131 op/s 2022-04-22T18:13:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:50 smithi115 bash[45495]: cluster 2022-04-22T18:13:49.200464+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1320: 8 total, 8 up, 8 in 2022-04-22T18:13:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:50 smithi115 bash[46934]: cluster 2022-04-22T18:13:49.147381+0000 mgr.y (mgr.34107) 1677 : cluster [DBG] pgmap v2473: 137 pgs: 137 active+clean; 100 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 MiB/s rd, 7.2 MiB/s wr, 131 op/s 2022-04-22T18:13:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:50 smithi115 bash[46934]: cluster 2022-04-22T18:13:49.200464+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1320: 8 total, 8 up, 8 in 2022-04-22T18:13:50.738 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:13:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:13:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (36m) 5m ago 50m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:13:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (50m) 6m ago 50m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (38m) 5m ago 51m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (38m) 6m ago 51m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (46m) 6m ago 55m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (46m) 5m ago 58m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (45m) 5m ago 58m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:13:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (44m) 6m ago 56m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:13:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (45m) 5m ago 56m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:13:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (38m) 5m ago 50m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:13:51.077 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (37m) 6m ago 50m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:13:51.077 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (43m) 5m ago 55m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:13:51.077 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (42m) 5m ago 55m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:13:51.077 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (42m) 5m ago 54m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:13:51.077 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (42m) 5m ago 54m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:13:51.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (41m) 6m ago 53m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:13:51.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (40m) 6m ago 53m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:13:51.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (39m) 6m ago 52m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:13:51.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (39m) 6m ago 52m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:13:51.078 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (37m) 6m ago 51m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:13:51.485 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:51.486 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:13:51.487 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: cluster 2022-04-22T18:13:50.210254+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e1321: 8 total, 8 up, 8 in 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: audit 2022-04-22T18:13:50.739889+0000 mon.a (mon.0) 3638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: audit 2022-04-22T18:13:50.741626+0000 mon.a (mon.0) 3639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: audit 2022-04-22T18:13:50.742593+0000 mon.a (mon.0) 3640 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: audit 2022-04-22T18:13:50.759867+0000 mon.a (mon.0) 3641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:51 smithi121 bash[36902]: audit 2022-04-22T18:13:51.102924+0000 mon.a (mon.0) 3642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: cluster 2022-04-22T18:13:50.210254+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e1321: 8 total, 8 up, 8 in 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: audit 2022-04-22T18:13:50.739889+0000 mon.a (mon.0) 3638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: audit 2022-04-22T18:13:50.741626+0000 mon.a (mon.0) 3639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: audit 2022-04-22T18:13:50.742593+0000 mon.a (mon.0) 3640 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: audit 2022-04-22T18:13:50.759867+0000 mon.a (mon.0) 3641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[46934]: audit 2022-04-22T18:13:51.102924+0000 mon.a (mon.0) 3642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: cluster 2022-04-22T18:13:50.210254+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e1321: 8 total, 8 up, 8 in 2022-04-22T18:13:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: audit 2022-04-22T18:13:50.739889+0000 mon.a (mon.0) 3638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:13:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: audit 2022-04-22T18:13:50.741626+0000 mon.a (mon.0) 3639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:13:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: audit 2022-04-22T18:13:50.742593+0000 mon.a (mon.0) 3640 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:13:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: audit 2022-04-22T18:13:50.759867+0000 mon.a (mon.0) 3641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[45495]: audit 2022-04-22T18:13:51.102924+0000 mon.a (mon.0) 3642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:13:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:51.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: audit 2022-04-22T18:13:50.716906+0000 mgr.y (mgr.34107) 1678 : audit [DBG] from='client.46891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: audit 2022-04-22T18:13:51.051341+0000 mgr.y (mgr.34107) 1679 : audit [DBG] from='client.36969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: cluster 2022-04-22T18:13:51.148316+0000 mgr.y (mgr.34107) 1680 : cluster [DBG] pgmap v2476: 137 pgs: 137 active+clean; 116 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 9.4 MiB/s wr, 76 op/s 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: cluster 2022-04-22T18:13:51.213120+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1322: 8 total, 8 up, 8 in 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: audit 2022-04-22T18:13:51.227621+0000 mon.a (mon.0) 3644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:52 smithi121 bash[36902]: audit 2022-04-22T18:13:51.480365+0000 mon.a (mon.0) 3645 : audit [DBG] from='client.? 172.21.15.115:0/2797594996' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: audit 2022-04-22T18:13:50.716906+0000 mgr.y (mgr.34107) 1678 : audit [DBG] from='client.46891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: audit 2022-04-22T18:13:51.051341+0000 mgr.y (mgr.34107) 1679 : audit [DBG] from='client.36969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: cluster 2022-04-22T18:13:51.148316+0000 mgr.y (mgr.34107) 1680 : cluster [DBG] pgmap v2476: 137 pgs: 137 active+clean; 116 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 9.4 MiB/s wr, 76 op/s 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: cluster 2022-04-22T18:13:51.213120+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1322: 8 total, 8 up, 8 in 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: audit 2022-04-22T18:13:51.227621+0000 mon.a (mon.0) 3644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[45495]: audit 2022-04-22T18:13:51.480365+0000 mon.a (mon.0) 3645 : audit [DBG] from='client.? 172.21.15.115:0/2797594996' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: audit 2022-04-22T18:13:50.716906+0000 mgr.y (mgr.34107) 1678 : audit [DBG] from='client.46891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: audit 2022-04-22T18:13:51.051341+0000 mgr.y (mgr.34107) 1679 : audit [DBG] from='client.36969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: cluster 2022-04-22T18:13:51.148316+0000 mgr.y (mgr.34107) 1680 : cluster [DBG] pgmap v2476: 137 pgs: 137 active+clean; 116 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 9.4 MiB/s wr, 76 op/s 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: cluster 2022-04-22T18:13:51.213120+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1322: 8 total, 8 up, 8 in 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: audit 2022-04-22T18:13:51.227621+0000 mon.a (mon.0) 3644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:52 smithi115 bash[46934]: audit 2022-04-22T18:13:51.480365+0000 mon.a (mon.0) 3645 : audit [DBG] from='client.? 172.21.15.115:0/2797594996' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:13:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:53 smithi121 bash[36902]: cluster 2022-04-22T18:13:52.232319+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2022-04-22T18:13:53.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:53 smithi121 bash[36902]: audit 2022-04-22T18:13:53.146371+0000 mon.a (mon.0) 3647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:53 smithi115 bash[45495]: cluster 2022-04-22T18:13:52.232319+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2022-04-22T18:13:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:53 smithi115 bash[45495]: audit 2022-04-22T18:13:53.146371+0000 mon.a (mon.0) 3647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:53 smithi115 bash[46934]: cluster 2022-04-22T18:13:52.232319+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1323: 8 total, 8 up, 8 in 2022-04-22T18:13:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:53 smithi115 bash[46934]: audit 2022-04-22T18:13:53.146371+0000 mon.a (mon.0) 3647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:13:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:13:53.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:13:54.165 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Stress (8292 ms) 2022-04-22T18:13:54.166 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.NoSnaps_LargerDstObjSize 2022-04-22T18:13:54.488 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.NoSnaps_LargerDstObjSize (322 ms) 2022-04-22T18:13:54.488 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Snaps_LargerDstObjSize 2022-04-22T18:13:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:54 smithi121 bash[36902]: cluster 2022-04-22T18:13:53.149664+0000 mgr.y (mgr.34107) 1681 : cluster [DBG] pgmap v2479: 137 pgs: 137 active+clean; 121 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2022-04-22T18:13:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:54 smithi121 bash[36902]: cluster 2022-04-22T18:13:53.244334+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e1324: 8 total, 8 up, 8 in 2022-04-22T18:13:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:54 smithi115 bash[46934]: cluster 2022-04-22T18:13:53.149664+0000 mgr.y (mgr.34107) 1681 : cluster [DBG] pgmap v2479: 137 pgs: 137 active+clean; 121 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2022-04-22T18:13:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:54 smithi115 bash[46934]: cluster 2022-04-22T18:13:53.244334+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e1324: 8 total, 8 up, 8 in 2022-04-22T18:13:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:54 smithi115 bash[45495]: cluster 2022-04-22T18:13:53.149664+0000 mgr.y (mgr.34107) 1681 : cluster [DBG] pgmap v2479: 137 pgs: 137 active+clean; 121 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 4.4 MiB/s wr, 24 op/s 2022-04-22T18:13:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:54 smithi115 bash[45495]: cluster 2022-04-22T18:13:53.244334+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e1324: 8 total, 8 up, 8 in 2022-04-22T18:13:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[45495]: cluster 2022-04-22T18:13:55.150813+0000 mgr.y (mgr.34107) 1682 : cluster [DBG] pgmap v2481: 137 pgs: 137 active+clean; 162 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 8.5 MiB/s wr, 56 op/s 2022-04-22T18:13:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[45495]: cluster 2022-04-22T18:13:55.250610+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2022-04-22T18:13:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[45495]: audit 2022-04-22T18:13:55.762533+0000 mon.a (mon.0) 3650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[45495]: audit 2022-04-22T18:13:56.228236+0000 mon.a (mon.0) 3651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[46934]: cluster 2022-04-22T18:13:55.150813+0000 mgr.y (mgr.34107) 1682 : cluster [DBG] pgmap v2481: 137 pgs: 137 active+clean; 162 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 8.5 MiB/s wr, 56 op/s 2022-04-22T18:13:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[46934]: cluster 2022-04-22T18:13:55.250610+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2022-04-22T18:13:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[46934]: audit 2022-04-22T18:13:55.762533+0000 mon.a (mon.0) 3650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:56 smithi115 bash[46934]: audit 2022-04-22T18:13:56.228236+0000 mon.a (mon.0) 3651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:56 smithi121 bash[36902]: cluster 2022-04-22T18:13:55.150813+0000 mgr.y (mgr.34107) 1682 : cluster [DBG] pgmap v2481: 137 pgs: 137 active+clean; 162 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 8.5 MiB/s wr, 56 op/s 2022-04-22T18:13:56.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:56 smithi121 bash[36902]: cluster 2022-04-22T18:13:55.250610+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1325: 8 total, 8 up, 8 in 2022-04-22T18:13:56.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:56 smithi121 bash[36902]: audit 2022-04-22T18:13:55.762533+0000 mon.a (mon.0) 3650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:13:56.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:56 smithi121 bash[36902]: audit 2022-04-22T18:13:56.228236+0000 mon.a (mon.0) 3651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:13:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:57 smithi115 bash[45495]: cluster 2022-04-22T18:13:56.256707+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2022-04-22T18:13:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:57 smithi115 bash[46934]: cluster 2022-04-22T18:13:56.256707+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2022-04-22T18:13:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:57 smithi121 bash[36902]: cluster 2022-04-22T18:13:56.256707+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1326: 8 total, 8 up, 8 in 2022-04-22T18:13:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[45495]: cluster 2022-04-22T18:13:57.151902+0000 mgr.y (mgr.34107) 1683 : cluster [DBG] pgmap v2484: 137 pgs: 137 active+clean; 208 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 15 MiB/s wr, 78 op/s 2022-04-22T18:13:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[45495]: cluster 2022-04-22T18:13:57.280812+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2022-04-22T18:13:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[45495]: audit 2022-04-22T18:13:58.149449+0000 mon.a (mon.0) 3654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[46934]: cluster 2022-04-22T18:13:57.151902+0000 mgr.y (mgr.34107) 1683 : cluster [DBG] pgmap v2484: 137 pgs: 137 active+clean; 208 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 15 MiB/s wr, 78 op/s 2022-04-22T18:13:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[46934]: cluster 2022-04-22T18:13:57.280812+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2022-04-22T18:13:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:58 smithi115 bash[46934]: audit 2022-04-22T18:13:58.149449+0000 mon.a (mon.0) 3654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:58.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Snaps_LargerDstObjSize (4234 ms) 2022-04-22T18:13:58.722 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Clone_LargerDstObjSize 2022-04-22T18:13:58.746 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:58 smithi121 bash[36902]: cluster 2022-04-22T18:13:57.151902+0000 mgr.y (mgr.34107) 1683 : cluster [DBG] pgmap v2484: 137 pgs: 137 active+clean; 208 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 15 MiB/s wr, 78 op/s 2022-04-22T18:13:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:58 smithi121 bash[36902]: cluster 2022-04-22T18:13:57.280812+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1327: 8 total, 8 up, 8 in 2022-04-22T18:13:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:58 smithi121 bash[36902]: audit 2022-04-22T18:13:58.149449+0000 mon.a (mon.0) 3654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:13:58.865 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Clone_LargerDstObjSize (140 ms) 2022-04-22T18:13:58.865 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneFlatten_LargerDstObjSize 2022-04-22T18:13:58.886 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:13:59.006 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneFlatten_LargerDstObjSize (143 ms) 2022-04-22T18:13:59.006 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Stress_LargerDstObjSize 2022-04-22T18:13:59.029 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 25047989~2314225 2022-04-22T18:13:59.043 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 44358103~6626626 2022-04-22T18:13:59.047 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 16026230~4640766 2022-04-22T18:13:59.068 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 20001711~6993503 2022-04-22T18:13:59.074 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 33193694~1040894 2022-04-22T18:13:59.078 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 3258061~4166829 2022-04-22T18:13:59.081 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 17813297~3989985 2022-04-22T18:13:59.097 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 35816753~1615812 2022-04-22T18:13:59.105 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap0 2022-04-22T18:13:59.318 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 72305117~947130 2022-04-22T18:13:59.320 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 6027947~5249347 2022-04-22T18:13:59.323 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 60420609~6051287 2022-04-22T18:13:59.323 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 50824159~3493641 2022-04-22T18:13:59.324 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 56479156~3622766 2022-04-22T18:13:59.326 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 15500172~3248948 2022-04-22T18:13:59.329 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 44181144~4778180 2022-04-22T18:13:59.329 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 13048954~1037181 2022-04-22T18:13:59.364 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:13:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:13:59 smithi121 bash[36902]: cluster 2022-04-22T18:13:58.281842+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1328: 8 total, 8 up, 8 in 2022-04-22T18:13:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:13:59 smithi115 bash[45495]: cluster 2022-04-22T18:13:58.281842+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1328: 8 total, 8 up, 8 in 2022-04-22T18:13:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:13:59 smithi115 bash[46934]: cluster 2022-04-22T18:13:58.281842+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1328: 8 total, 8 up, 8 in 2022-04-22T18:14:00.322 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 146058580 2022-04-22T18:14:00.331 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 111261277~912504 2022-04-22T18:14:00.331 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 82602711~2057140 2022-04-22T18:14:00.331 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 728394~766031 2022-04-22T18:14:00.332 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 127885088~8303828 2022-04-22T18:14:00.332 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 73996645~5686388 2022-04-22T18:14:00.332 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 91834789~5639905 2022-04-22T18:14:00.333 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 29784735~3901762 2022-04-22T18:14:00.335 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 115750885~4756354 2022-04-22T18:14:00.372 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:14:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:00 smithi121 bash[36902]: cluster 2022-04-22T18:13:59.152876+0000 mgr.y (mgr.34107) 1684 : cluster [DBG] pgmap v2487: 137 pgs: 137 active+clean; 233 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 15 MiB/s wr, 80 op/s 2022-04-22T18:14:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:00 smithi121 bash[36902]: cluster 2022-04-22T18:13:59.300308+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2022-04-22T18:14:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:00 smithi115 bash[45495]: cluster 2022-04-22T18:13:59.152876+0000 mgr.y (mgr.34107) 1684 : cluster [DBG] pgmap v2487: 137 pgs: 137 active+clean; 233 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 15 MiB/s wr, 80 op/s 2022-04-22T18:14:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:00 smithi115 bash[45495]: cluster 2022-04-22T18:13:59.300308+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2022-04-22T18:14:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:00 smithi115 bash[46934]: cluster 2022-04-22T18:13:59.152876+0000 mgr.y (mgr.34107) 1684 : cluster [DBG] pgmap v2487: 137 pgs: 137 active+clean; 233 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 15 MiB/s wr, 80 op/s 2022-04-22T18:14:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:00 smithi115 bash[46934]: cluster 2022-04-22T18:13:59.300308+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1329: 8 total, 8 up, 8 in 2022-04-22T18:14:01.335 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 119903366~2419033 2022-04-22T18:14:01.337 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 21688993~7720015 2022-04-22T18:14:01.339 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 139972960~6085620 2022-04-22T18:14:01.340 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 92663013~6160493 2022-04-22T18:14:01.341 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 116482494~1868488 2022-04-22T18:14:01.341 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 113612614~5845758 2022-04-22T18:14:01.362 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 57177523~486113 2022-04-22T18:14:01.363 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 111747390~7441339 2022-04-22T18:14:01.367 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap3 2022-04-22T18:14:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:01 smithi121 bash[36902]: cluster 2022-04-22T18:14:00.310524+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2022-04-22T18:14:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:01 smithi121 bash[36902]: audit 2022-04-22T18:14:00.765778+0000 mon.a (mon.0) 3658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:01 smithi121 bash[36902]: audit 2022-04-22T18:14:01.231887+0000 mon.a (mon.0) 3659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:01.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[45495]: cluster 2022-04-22T18:14:00.310524+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2022-04-22T18:14:01.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[45495]: audit 2022-04-22T18:14:00.765778+0000 mon.a (mon.0) 3658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:01.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[45495]: audit 2022-04-22T18:14:01.231887+0000 mon.a (mon.0) 3659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:01.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[46934]: cluster 2022-04-22T18:14:00.310524+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1330: 8 total, 8 up, 8 in 2022-04-22T18:14:01.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[46934]: audit 2022-04-22T18:14:00.765778+0000 mon.a (mon.0) 3658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:01.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[46934]: audit 2022-04-22T18:14:01.231887+0000 mon.a (mon.0) 3659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:01.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:02.339 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 109789506 2022-04-22T18:14:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:02 smithi121 bash[36902]: cluster 2022-04-22T18:14:01.153981+0000 mgr.y (mgr.34107) 1685 : cluster [DBG] pgmap v2490: 137 pgs: 137 active+clean; 251 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 31 MiB/s rd, 8.3 MiB/s wr, 62 op/s 2022-04-22T18:14:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:02 smithi121 bash[36902]: cluster 2022-04-22T18:14:01.320174+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2022-04-22T18:14:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:02 smithi115 bash[45495]: cluster 2022-04-22T18:14:01.153981+0000 mgr.y (mgr.34107) 1685 : cluster [DBG] pgmap v2490: 137 pgs: 137 active+clean; 251 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 31 MiB/s rd, 8.3 MiB/s wr, 62 op/s 2022-04-22T18:14:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:02 smithi115 bash[45495]: cluster 2022-04-22T18:14:01.320174+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2022-04-22T18:14:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:02 smithi115 bash[46934]: cluster 2022-04-22T18:14:01.153981+0000 mgr.y (mgr.34107) 1685 : cluster [DBG] pgmap v2490: 137 pgs: 137 active+clean; 251 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 31 MiB/s rd, 8.3 MiB/s wr, 62 op/s 2022-04-22T18:14:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:02 smithi115 bash[46934]: cluster 2022-04-22T18:14:01.320174+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e1331: 8 total, 8 up, 8 in 2022-04-22T18:14:03.666 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:03 smithi115 bash[46934]: cluster 2022-04-22T18:14:02.328646+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2022-04-22T18:14:03.666 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:03 smithi115 bash[46934]: audit 2022-04-22T18:14:03.152724+0000 mon.a (mon.0) 3662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:03.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:03 smithi115 bash[45495]: cluster 2022-04-22T18:14:02.328646+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2022-04-22T18:14:03.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:03 smithi115 bash[45495]: audit 2022-04-22T18:14:03.152724+0000 mon.a (mon.0) 3662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:03 smithi121 bash[36902]: cluster 2022-04-22T18:14:02.328646+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2022-04-22T18:14:03.803 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:03 smithi121 bash[36902]: audit 2022-04-22T18:14:03.152724+0000 mon.a (mon.0) 3662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:03.660Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:04.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:04 smithi121 bash[36902]: cluster 2022-04-22T18:14:03.155119+0000 mgr.y (mgr.34107) 1686 : cluster [DBG] pgmap v2493: 137 pgs: 137 active+clean; 296 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 64 op/s 2022-04-22T18:14:04.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:04 smithi121 bash[36902]: cluster 2022-04-22T18:14:03.336619+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2022-04-22T18:14:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:04 smithi115 bash[45495]: cluster 2022-04-22T18:14:03.155119+0000 mgr.y (mgr.34107) 1686 : cluster [DBG] pgmap v2493: 137 pgs: 137 active+clean; 296 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 64 op/s 2022-04-22T18:14:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:04 smithi115 bash[45495]: cluster 2022-04-22T18:14:03.336619+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2022-04-22T18:14:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:04 smithi115 bash[46934]: cluster 2022-04-22T18:14:03.155119+0000 mgr.y (mgr.34107) 1686 : cluster [DBG] pgmap v2493: 137 pgs: 137 active+clean; 296 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 12 MiB/s wr, 64 op/s 2022-04-22T18:14:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:04 smithi115 bash[46934]: cluster 2022-04-22T18:14:03.336619+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2022-04-22T18:14:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:05 smithi121 bash[36902]: cluster 2022-04-22T18:14:04.345674+0000 mon.a (mon.0) 3664 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2022-04-22T18:14:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:05 smithi115 bash[45495]: cluster 2022-04-22T18:14:04.345674+0000 mon.a (mon.0) 3664 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2022-04-22T18:14:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:05 smithi115 bash[46934]: cluster 2022-04-22T18:14:04.345674+0000 mon.a (mon.0) 3664 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2022-04-22T18:14:06.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:06 smithi121 bash[36902]: cluster 2022-04-22T18:14:05.156150+0000 mgr.y (mgr.34107) 1687 : cluster [DBG] pgmap v2496: 137 pgs: 137 active+clean; 315 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 13 MiB/s wr, 52 op/s 2022-04-22T18:14:06.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:06 smithi121 bash[36902]: cluster 2022-04-22T18:14:05.357853+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2022-04-22T18:14:06.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:06 smithi121 bash[36902]: audit 2022-04-22T18:14:05.768815+0000 mon.a (mon.0) 3666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:06.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:06 smithi121 bash[36902]: audit 2022-04-22T18:14:06.235428+0000 mon.a (mon.0) 3667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[45495]: cluster 2022-04-22T18:14:05.156150+0000 mgr.y (mgr.34107) 1687 : cluster [DBG] pgmap v2496: 137 pgs: 137 active+clean; 315 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 13 MiB/s wr, 52 op/s 2022-04-22T18:14:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[45495]: cluster 2022-04-22T18:14:05.357853+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2022-04-22T18:14:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[45495]: audit 2022-04-22T18:14:05.768815+0000 mon.a (mon.0) 3666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[45495]: audit 2022-04-22T18:14:06.235428+0000 mon.a (mon.0) 3667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[46934]: cluster 2022-04-22T18:14:05.156150+0000 mgr.y (mgr.34107) 1687 : cluster [DBG] pgmap v2496: 137 pgs: 137 active+clean; 315 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 13 MiB/s wr, 52 op/s 2022-04-22T18:14:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[46934]: cluster 2022-04-22T18:14:05.357853+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2022-04-22T18:14:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[46934]: audit 2022-04-22T18:14:05.768815+0000 mon.a (mon.0) 3666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:06 smithi115 bash[46934]: audit 2022-04-22T18:14:06.235428+0000 mon.a (mon.0) 3667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:07.319 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Stress_LargerDstObjSize (8313 ms) 2022-04-22T18:14:07.324 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.NoSnaps_SmallerDstObjSize 2022-04-22T18:14:07.605 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.NoSnaps_SmallerDstObjSize (287 ms) 2022-04-22T18:14:07.606 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Snaps_SmallerDstObjSize 2022-04-22T18:14:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:07 smithi121 bash[36902]: cluster 2022-04-22T18:14:06.372101+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2022-04-22T18:14:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:07 smithi115 bash[45495]: cluster 2022-04-22T18:14:06.372101+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2022-04-22T18:14:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:07 smithi115 bash[46934]: cluster 2022-04-22T18:14:06.372101+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2022-04-22T18:14:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:08 smithi121 bash[36902]: cluster 2022-04-22T18:14:07.157271+0000 mgr.y (mgr.34107) 1688 : cluster [DBG] pgmap v2499: 137 pgs: 137 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2022-04-22T18:14:08.832 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:08 smithi121 bash[36902]: audit 2022-04-22T18:14:08.155843+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:08.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:08 smithi115 bash[45495]: cluster 2022-04-22T18:14:07.157271+0000 mgr.y (mgr.34107) 1688 : cluster [DBG] pgmap v2499: 137 pgs: 137 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2022-04-22T18:14:08.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:08 smithi115 bash[45495]: audit 2022-04-22T18:14:08.155843+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:08.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:08 smithi115 bash[46934]: cluster 2022-04-22T18:14:07.157271+0000 mgr.y (mgr.34107) 1688 : cluster [DBG] pgmap v2499: 137 pgs: 137 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2022-04-22T18:14:08.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:08 smithi115 bash[46934]: audit 2022-04-22T18:14:08.155843+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:09 smithi121 bash[36902]: cluster 2022-04-22T18:14:08.399949+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2022-04-22T18:14:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:09 smithi115 bash[45495]: cluster 2022-04-22T18:14:08.399949+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2022-04-22T18:14:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:09 smithi115 bash[46934]: cluster 2022-04-22T18:14:08.399949+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2022-04-22T18:14:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:10 smithi121 bash[36902]: cluster 2022-04-22T18:14:09.158242+0000 mgr.y (mgr.34107) 1689 : cluster [DBG] pgmap v2501: 137 pgs: 137 active+clean; 391 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 9.7 MiB/s wr, 67 op/s 2022-04-22T18:14:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:10 smithi121 bash[36902]: cluster 2022-04-22T18:14:09.422413+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2022-04-22T18:14:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:10 smithi115 bash[45495]: cluster 2022-04-22T18:14:09.158242+0000 mgr.y (mgr.34107) 1689 : cluster [DBG] pgmap v2501: 137 pgs: 137 active+clean; 391 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 9.7 MiB/s wr, 67 op/s 2022-04-22T18:14:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:10 smithi115 bash[45495]: cluster 2022-04-22T18:14:09.422413+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2022-04-22T18:14:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:10 smithi115 bash[46934]: cluster 2022-04-22T18:14:09.158242+0000 mgr.y (mgr.34107) 1689 : cluster [DBG] pgmap v2501: 137 pgs: 137 active+clean; 391 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 9.7 MiB/s wr, 67 op/s 2022-04-22T18:14:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:10 smithi115 bash[46934]: cluster 2022-04-22T18:14:09.422413+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2022-04-22T18:14:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:11 smithi121 bash[36902]: cluster 2022-04-22T18:14:10.427517+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2022-04-22T18:14:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:11 smithi121 bash[36902]: audit 2022-04-22T18:14:10.772032+0000 mon.a (mon.0) 3673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:11 smithi121 bash[36902]: audit 2022-04-22T18:14:11.238844+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:11.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[45495]: cluster 2022-04-22T18:14:10.427517+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2022-04-22T18:14:11.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[45495]: audit 2022-04-22T18:14:10.772032+0000 mon.a (mon.0) 3673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:11.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[45495]: audit 2022-04-22T18:14:11.238844+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:11.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[46934]: cluster 2022-04-22T18:14:10.427517+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2022-04-22T18:14:11.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[46934]: audit 2022-04-22T18:14:10.772032+0000 mon.a (mon.0) 3673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:11.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[46934]: audit 2022-04-22T18:14:11.238844+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:11.858 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Snaps_SmallerDstObjSize (4253 ms) 2022-04-22T18:14:11.858 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Clone_SmallerDstObjSize 2022-04-22T18:14:11.885 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:12.015 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Clone_SmallerDstObjSize (157 ms) 2022-04-22T18:14:12.015 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneFlatten_SmallerDstObjSize 2022-04-22T18:14:12.036 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:11.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:12.155 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneFlatten_SmallerDstObjSize (141 ms) 2022-04-22T18:14:12.155 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Stress_SmallerDstObjSize 2022-04-22T18:14:12.178 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 4677471~331907 2022-04-22T18:14:12.218 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 2794762~307598 2022-04-22T18:14:12.218 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 70328007~4008879 2022-04-22T18:14:12.219 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 26265011~491774 2022-04-22T18:14:12.219 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 13688959~991029 2022-04-22T18:14:12.219 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 59721275~2967675 2022-04-22T18:14:12.219 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 26273726~2757502 2022-04-22T18:14:12.228 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 53998545~1487132 2022-04-22T18:14:12.230 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap0 2022-04-22T18:14:12.458 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 144304718 2022-04-22T18:14:12.464 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 16627513~6633432 2022-04-22T18:14:12.466 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 68214941~2675663 2022-04-22T18:14:12.468 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 58008684~6455314 2022-04-22T18:14:12.469 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 27082179~4868479 2022-04-22T18:14:12.470 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 123726507~1541099 2022-04-22T18:14:12.470 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 6777436~1692365 2022-04-22T18:14:12.472 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 5214433~4277862 2022-04-22T18:14:12.488 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 122025973~6669668 2022-04-22T18:14:12.505 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap1 2022-04-22T18:14:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:12 smithi121 bash[36902]: cluster 2022-04-22T18:14:11.159138+0000 mgr.y (mgr.34107) 1690 : cluster [DBG] pgmap v2504: 137 pgs: 137 active+clean; 426 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 MiB/s rd, 14 MiB/s wr, 81 op/s 2022-04-22T18:14:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:12 smithi121 bash[36902]: cluster 2022-04-22T18:14:11.443646+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2022-04-22T18:14:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:12 smithi115 bash[45495]: cluster 2022-04-22T18:14:11.159138+0000 mgr.y (mgr.34107) 1690 : cluster [DBG] pgmap v2504: 137 pgs: 137 active+clean; 426 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 MiB/s rd, 14 MiB/s wr, 81 op/s 2022-04-22T18:14:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:12 smithi115 bash[45495]: cluster 2022-04-22T18:14:11.443646+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2022-04-22T18:14:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:12 smithi115 bash[46934]: cluster 2022-04-22T18:14:11.159138+0000 mgr.y (mgr.34107) 1690 : cluster [DBG] pgmap v2504: 137 pgs: 137 active+clean; 426 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 MiB/s rd, 14 MiB/s wr, 81 op/s 2022-04-22T18:14:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:12 smithi115 bash[46934]: cluster 2022-04-22T18:14:11.443646+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2022-04-22T18:14:13.467 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 154776661 2022-04-22T18:14:13.472 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 222897~7804604 2022-04-22T18:14:13.473 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 56170738~3389126 2022-04-22T18:14:13.475 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 98631489~2222553 2022-04-22T18:14:13.475 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 55655856~1408044 2022-04-22T18:14:13.499 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 21322740~7464703 2022-04-22T18:14:13.500 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 145143760~258344 2022-04-22T18:14:13.501 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 116287833~4593558 2022-04-22T18:14:13.503 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 4312678~2587092 2022-04-22T18:14:13.533 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap2 2022-04-22T18:14:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:13 smithi121 bash[36902]: cluster 2022-04-22T18:14:12.447874+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2022-04-22T18:14:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:13 smithi121 bash[36902]: audit 2022-04-22T18:14:13.158796+0000 mon.a (mon.0) 3677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:13 smithi115 bash[45495]: cluster 2022-04-22T18:14:12.447874+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2022-04-22T18:14:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:13 smithi115 bash[45495]: audit 2022-04-22T18:14:13.158796+0000 mon.a (mon.0) 3677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:13.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:13.661Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:13 smithi115 bash[46934]: cluster 2022-04-22T18:14:12.447874+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2022-04-22T18:14:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:13 smithi115 bash[46934]: audit 2022-04-22T18:14:13.158796+0000 mon.a (mon.0) 3677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:14 smithi121 bash[36902]: cluster 2022-04-22T18:14:13.160212+0000 mgr.y (mgr.34107) 1691 : cluster [DBG] pgmap v2507: 137 pgs: 137 active+clean; 454 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 13 MiB/s wr, 109 op/s 2022-04-22T18:14:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:14 smithi121 bash[36902]: cluster 2022-04-22T18:14:13.458118+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2022-04-22T18:14:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:14 smithi121 bash[36902]: audit 2022-04-22T18:14:13.854509+0000 mon.a (mon.0) 3679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:14:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:14 smithi121 bash[36902]: audit 2022-04-22T18:14:13.860731+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:14:14.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:14 smithi121 bash[36902]: audit 2022-04-22T18:14:14.446278+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]: dispatch 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[45495]: cluster 2022-04-22T18:14:13.160212+0000 mgr.y (mgr.34107) 1691 : cluster [DBG] pgmap v2507: 137 pgs: 137 active+clean; 454 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 13 MiB/s wr, 109 op/s 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[45495]: cluster 2022-04-22T18:14:13.458118+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[45495]: audit 2022-04-22T18:14:13.854509+0000 mon.a (mon.0) 3679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[45495]: audit 2022-04-22T18:14:13.860731+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[45495]: audit 2022-04-22T18:14:14.446278+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]: dispatch 2022-04-22T18:14:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[46934]: cluster 2022-04-22T18:14:13.160212+0000 mgr.y (mgr.34107) 1691 : cluster [DBG] pgmap v2507: 137 pgs: 137 active+clean; 454 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 13 MiB/s wr, 109 op/s 2022-04-22T18:14:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[46934]: cluster 2022-04-22T18:14:13.458118+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2022-04-22T18:14:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[46934]: audit 2022-04-22T18:14:13.854509+0000 mon.a (mon.0) 3679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:14:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[46934]: audit 2022-04-22T18:14:13.860731+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:14:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:14 smithi115 bash[46934]: audit 2022-04-22T18:14:14.446278+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]: dispatch 2022-04-22T18:14:15.746 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 199849021 2022-04-22T18:14:15.760 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 175529139~6557650 2022-04-22T18:14:15.760 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 117150653~2776757 2022-04-22T18:14:15.766 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 160373598~4280098 2022-04-22T18:14:15.766 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 94398429~1702337 2022-04-22T18:14:15.766 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 136155514~6493396 2022-04-22T18:14:15.766 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 65076039~4754813 2022-04-22T18:14:15.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:15 smithi121 bash[36902]: audit 2022-04-22T18:14:14.461114+0000 mon.a (mon.0) 3682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]': finished 2022-04-22T18:14:15.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:15 smithi121 bash[36902]: cluster 2022-04-22T18:14:14.461431+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2022-04-22T18:14:15.779 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 106788495~2273336 2022-04-22T18:14:15.784 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 104324252~8234632 2022-04-22T18:14:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:15 smithi115 bash[46934]: audit 2022-04-22T18:14:14.461114+0000 mon.a (mon.0) 3682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]': finished 2022-04-22T18:14:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:15 smithi115 bash[46934]: cluster 2022-04-22T18:14:14.461431+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2022-04-22T18:14:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:15 smithi115 bash[45495]: audit 2022-04-22T18:14:14.461114+0000 mon.a (mon.0) 3682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "77.f", "id": [6, 0]}]': finished 2022-04-22T18:14:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:15 smithi115 bash[45495]: cluster 2022-04-22T18:14:14.461431+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2022-04-22T18:14:15.813 INFO:tasks.workunit.client.0.smithi115.stdout:snap: snap3 2022-04-22T18:14:16.514 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 176227839 2022-04-22T18:14:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:16 smithi121 bash[36902]: cluster 2022-04-22T18:14:15.161300+0000 mgr.y (mgr.34107) 1692 : cluster [DBG] pgmap v2510: 137 pgs: 137 active+clean; 499 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 15 MiB/s wr, 107 op/s 2022-04-22T18:14:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:16 smithi121 bash[36902]: cluster 2022-04-22T18:14:15.480193+0000 mon.a (mon.0) 3684 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2022-04-22T18:14:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:16 smithi121 bash[36902]: audit 2022-04-22T18:14:15.774977+0000 mon.a (mon.0) 3685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:16 smithi121 bash[36902]: audit 2022-04-22T18:14:16.242631+0000 mon.a (mon.0) 3686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[45495]: cluster 2022-04-22T18:14:15.161300+0000 mgr.y (mgr.34107) 1692 : cluster [DBG] pgmap v2510: 137 pgs: 137 active+clean; 499 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 15 MiB/s wr, 107 op/s 2022-04-22T18:14:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[45495]: cluster 2022-04-22T18:14:15.480193+0000 mon.a (mon.0) 3684 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2022-04-22T18:14:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[45495]: audit 2022-04-22T18:14:15.774977+0000 mon.a (mon.0) 3685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[45495]: audit 2022-04-22T18:14:16.242631+0000 mon.a (mon.0) 3686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[46934]: cluster 2022-04-22T18:14:15.161300+0000 mgr.y (mgr.34107) 1692 : cluster [DBG] pgmap v2510: 137 pgs: 137 active+clean; 499 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 15 MiB/s wr, 107 op/s 2022-04-22T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[46934]: cluster 2022-04-22T18:14:15.480193+0000 mon.a (mon.0) 3684 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2022-04-22T18:14:16.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[46934]: audit 2022-04-22T18:14:15.774977+0000 mon.a (mon.0) 3685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:16.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:16 smithi115 bash[46934]: audit 2022-04-22T18:14:16.242631+0000 mon.a (mon.0) 3686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:17 smithi121 bash[36902]: cluster 2022-04-22T18:14:16.501104+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2022-04-22T18:14:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:17 smithi115 bash[45495]: cluster 2022-04-22T18:14:16.501104+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2022-04-22T18:14:17.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:17 smithi115 bash[46934]: cluster 2022-04-22T18:14:16.501104+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2022-04-22T18:14:18.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[45495]: cluster 2022-04-22T18:14:17.162478+0000 mgr.y (mgr.34107) 1693 : cluster [DBG] pgmap v2513: 137 pgs: 137 active+clean; 535 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 14 MiB/s wr, 50 op/s 2022-04-22T18:14:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[45495]: cluster 2022-04-22T18:14:17.511227+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2022-04-22T18:14:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[45495]: audit 2022-04-22T18:14:18.161894+0000 mon.a (mon.0) 3689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[46934]: cluster 2022-04-22T18:14:17.162478+0000 mgr.y (mgr.34107) 1693 : cluster [DBG] pgmap v2513: 137 pgs: 137 active+clean; 535 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 14 MiB/s wr, 50 op/s 2022-04-22T18:14:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[46934]: cluster 2022-04-22T18:14:17.511227+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2022-04-22T18:14:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:18 smithi115 bash[46934]: audit 2022-04-22T18:14:18.161894+0000 mon.a (mon.0) 3689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:19.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:18 smithi121 bash[36902]: cluster 2022-04-22T18:14:17.162478+0000 mgr.y (mgr.34107) 1693 : cluster [DBG] pgmap v2513: 137 pgs: 137 active+clean; 535 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 14 MiB/s wr, 50 op/s 2022-04-22T18:14:19.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:18 smithi121 bash[36902]: cluster 2022-04-22T18:14:17.511227+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2022-04-22T18:14:19.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:18 smithi121 bash[36902]: audit 2022-04-22T18:14:18.161894+0000 mon.a (mon.0) 3689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:19 smithi115 bash[45495]: cluster 2022-04-22T18:14:18.513123+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2022-04-22T18:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:19 smithi115 bash[46934]: cluster 2022-04-22T18:14:18.513123+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2022-04-22T18:14:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:19 smithi121 bash[36902]: cluster 2022-04-22T18:14:18.513123+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2022-04-22T18:14:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:20 smithi115 bash[45495]: cluster 2022-04-22T18:14:19.163542+0000 mgr.y (mgr.34107) 1694 : cluster [DBG] pgmap v2516: 137 pgs: 137 active+clean; 556 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 10 MiB/s wr, 37 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:20 smithi115 bash[45495]: cluster 2022-04-22T18:14:19.532098+0000 mon.a (mon.0) 3691 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2022-04-22T18:14:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:20 smithi115 bash[46934]: cluster 2022-04-22T18:14:19.163542+0000 mgr.y (mgr.34107) 1694 : cluster [DBG] pgmap v2516: 137 pgs: 137 active+clean; 556 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 10 MiB/s wr, 37 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:20 smithi115 bash[46934]: cluster 2022-04-22T18:14:19.532098+0000 mon.a (mon.0) 3691 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2022-04-22T18:14:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:20 smithi121 bash[36902]: cluster 2022-04-22T18:14:19.163542+0000 mgr.y (mgr.34107) 1694 : cluster [DBG] pgmap v2516: 137 pgs: 137 active+clean; 556 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 10 MiB/s wr, 37 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:20 smithi121 bash[36902]: cluster 2022-04-22T18:14:19.532098+0000 mon.a (mon.0) 3691 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2022-04-22T18:14:21.658 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Stress_SmallerDstObjSize (9502 ms) 2022-04-22T18:14:21.659 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2022-04-22T18:14:21.680 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:21.788 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize (131 ms) 2022-04-22T18:14:21.788 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2022-04-22T18:14:21.809 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:21.885 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:14:21.913 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Snaps_StrippingLargerDstObjSize (124 ms) 2022-04-22T18:14:21.913 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2022-04-22T18:14:21.934 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:21 smithi121 bash[36902]: cluster 2022-04-22T18:14:20.533431+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2022-04-22T18:14:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:21 smithi121 bash[36902]: audit 2022-04-22T18:14:20.777291+0000 mon.a (mon.0) 3693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:21 smithi121 bash[36902]: audit 2022-04-22T18:14:21.245960+0000 mon.a (mon.0) 3694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:22.038 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Clone_StrippingLargerDstObjSize (126 ms) 2022-04-22T18:14:22.038 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2022-04-22T18:14:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:21.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:22.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[45495]: cluster 2022-04-22T18:14:20.533431+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2022-04-22T18:14:22.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[45495]: audit 2022-04-22T18:14:20.777291+0000 mon.a (mon.0) 3693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:22.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[45495]: audit 2022-04-22T18:14:21.245960+0000 mon.a (mon.0) 3694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:22.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[46934]: cluster 2022-04-22T18:14:20.533431+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2022-04-22T18:14:22.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[46934]: audit 2022-04-22T18:14:20.777291+0000 mon.a (mon.0) 3693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:22.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:21 smithi115 bash[46934]: audit 2022-04-22T18:14:21.245960+0000 mon.a (mon.0) 3694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:22.059 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.163 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize (124 ms) 2022-04-22T18:14:22.266 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2022-04-22T18:14:22.267 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.267 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (37m) 6m ago 51m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (50m) 7m ago 50m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (39m) 6m ago 52m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (38m) 7m ago 52m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (46m) 7m ago 56m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (47m) 6m ago 58m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (46m) 6m ago 58m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (45m) 7m ago 56m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:14:22.268 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (45m) 6m ago 56m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (38m) 6m ago 51m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (38m) 7m ago 51m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (44m) 6m ago 56m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (43m) 6m ago 55m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (43m) 6m ago 55m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (42m) 6m ago 54m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (41m) 7m ago 54m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (41m) 7m ago 53m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (40m) 7m ago 53m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (39m) 7m ago 52m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:14:22.269 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (38m) 7m ago 51m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:14:22.286 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Stress_StrippingLargerDstObjSize (124 ms) 2022-04-22T18:14:22.286 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2022-04-22T18:14:22.308 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.412 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize (126 ms) 2022-04-22T18:14:22.413 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2022-04-22T18:14:22.436 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.540 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize (127 ms) 2022-04-22T18:14:22.540 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2022-04-22T18:14:22.564 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.666 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Clone_StrippingSmallerDstObjSize (126 ms) 2022-04-22T18:14:22.666 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2022-04-22T18:14:22.685 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.697 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:14:22.697 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:14:22.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:14:22.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:14:22.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:22.699 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:14:22.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:14:22.699 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:14:22.699 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:14:22.794 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize (128 ms) 2022-04-22T18:14:22.794 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2022-04-22T18:14:22.818 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:22.935 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestDeepCopy.Stress_StrippingSmallerDstObjSize (140 ms) 2022-04-22T18:14:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:22 smithi121 bash[36902]: cluster 2022-04-22T18:14:21.164373+0000 mgr.y (mgr.34107) 1695 : cluster [DBG] pgmap v2519: 137 pgs: 137 active+clean; 564 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 27 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:23.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:22 smithi115 bash[45495]: cluster 2022-04-22T18:14:21.164373+0000 mgr.y (mgr.34107) 1695 : cluster [DBG] pgmap v2519: 137 pgs: 137 active+clean; 564 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 27 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:23.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:22 smithi115 bash[46934]: cluster 2022-04-22T18:14:21.164373+0000 mgr.y (mgr.34107) 1695 : cluster [DBG] pgmap v2519: 137 pgs: 137 active+clean; 564 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 27 op/s; 1.9 MiB/s, 2 keys/s, 1 objects/s recovering 2022-04-22T18:14:23.577 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 31 tests from TestDeepCopy (48275 ms total) 2022-04-22T18:14:23.577 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:14:23.577 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 6 tests from TestGroup 2022-04-22T18:14:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:23 smithi121 bash[36902]: audit 2022-04-22T18:14:21.859254+0000 mgr.y (mgr.34107) 1696 : audit [DBG] from='client.36981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:23 smithi121 bash[36902]: audit 2022-04-22T18:14:22.225338+0000 mgr.y (mgr.34107) 1697 : audit [DBG] from='client.46915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:23 smithi121 bash[36902]: audit 2022-04-22T18:14:22.692021+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.115:0/2575051841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:24.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:23 smithi121 bash[36902]: audit 2022-04-22T18:14:23.168290+0000 mon.a (mon.0) 3695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:23.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:24.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[45495]: audit 2022-04-22T18:14:21.859254+0000 mgr.y (mgr.34107) 1696 : audit [DBG] from='client.36981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[45495]: audit 2022-04-22T18:14:22.225338+0000 mgr.y (mgr.34107) 1697 : audit [DBG] from='client.46915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[45495]: audit 2022-04-22T18:14:22.692021+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.115:0/2575051841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[45495]: audit 2022-04-22T18:14:23.168290+0000 mon.a (mon.0) 3695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[46934]: audit 2022-04-22T18:14:21.859254+0000 mgr.y (mgr.34107) 1696 : audit [DBG] from='client.36981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[46934]: audit 2022-04-22T18:14:22.225338+0000 mgr.y (mgr.34107) 1697 : audit [DBG] from='client.46915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[46934]: audit 2022-04-22T18:14:22.692021+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.115:0/2575051841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:23 smithi115 bash[46934]: audit 2022-04-22T18:14:23.168290+0000 mon.a (mon.0) 3695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:25.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:24 smithi121 bash[36902]: cluster 2022-04-22T18:14:23.165675+0000 mgr.y (mgr.34107) 1698 : cluster [DBG] pgmap v2520: 137 pgs: 137 active+clean; 587 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 25 MiB/s rd, 7.7 MiB/s wr, 98 op/s; 1.3 MiB/s, 1 keys/s, 1 objects/s recovering 2022-04-22T18:14:25.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:24 smithi121 bash[36902]: cluster 2022-04-22T18:14:23.565444+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2022-04-22T18:14:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:24 smithi115 bash[45495]: cluster 2022-04-22T18:14:23.165675+0000 mgr.y (mgr.34107) 1698 : cluster [DBG] pgmap v2520: 137 pgs: 137 active+clean; 587 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 25 MiB/s rd, 7.7 MiB/s wr, 98 op/s; 1.3 MiB/s, 1 keys/s, 1 objects/s recovering 2022-04-22T18:14:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:24 smithi115 bash[45495]: cluster 2022-04-22T18:14:23.565444+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2022-04-22T18:14:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:24 smithi115 bash[46934]: cluster 2022-04-22T18:14:23.165675+0000 mgr.y (mgr.34107) 1698 : cluster [DBG] pgmap v2520: 137 pgs: 137 active+clean; 587 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 25 MiB/s rd, 7.7 MiB/s wr, 98 op/s; 1.3 MiB/s, 1 keys/s, 1 objects/s recovering 2022-04-22T18:14:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:24 smithi115 bash[46934]: cluster 2022-04-22T18:14:23.565444+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2022-04-22T18:14:25.590 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.group_create 2022-04-22T18:14:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:25 smithi121 bash[36902]: cluster 2022-04-22T18:14:24.571616+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2022-04-22T18:14:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:25 smithi121 bash[36902]: audit 2022-04-22T18:14:24.577834+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.115:0/2919909165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:25 smithi121 bash[36902]: audit 2022-04-22T18:14:24.578430+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:26.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[45495]: cluster 2022-04-22T18:14:24.571616+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2022-04-22T18:14:26.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[45495]: audit 2022-04-22T18:14:24.577834+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.115:0/2919909165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:26.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[45495]: audit 2022-04-22T18:14:24.578430+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[46934]: cluster 2022-04-22T18:14:24.571616+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2022-04-22T18:14:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[46934]: audit 2022-04-22T18:14:24.577834+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.115:0/2919909165' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:25 smithi115 bash[46934]: audit 2022-04-22T18:14:24.578430+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:26 smithi121 bash[36902]: cluster 2022-04-22T18:14:25.166668+0000 mgr.y (mgr.34107) 1699 : cluster [DBG] pgmap v2523: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:27.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:26 smithi121 bash[36902]: audit 2022-04-22T18:14:25.581966+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:27.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:26 smithi121 bash[36902]: cluster 2022-04-22T18:14:25.582206+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2022-04-22T18:14:27.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:26 smithi121 bash[36902]: audit 2022-04-22T18:14:25.779980+0000 mon.a (mon.0) 3701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:27.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:26 smithi121 bash[36902]: audit 2022-04-22T18:14:26.249132+0000 mon.a (mon.0) 3702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[45495]: cluster 2022-04-22T18:14:25.166668+0000 mgr.y (mgr.34107) 1699 : cluster [DBG] pgmap v2523: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[45495]: audit 2022-04-22T18:14:25.581966+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[45495]: cluster 2022-04-22T18:14:25.582206+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[45495]: audit 2022-04-22T18:14:25.779980+0000 mon.a (mon.0) 3701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[45495]: audit 2022-04-22T18:14:26.249132+0000 mon.a (mon.0) 3702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[46934]: cluster 2022-04-22T18:14:25.166668+0000 mgr.y (mgr.34107) 1699 : cluster [DBG] pgmap v2523: 137 pgs: 5 creating+peering, 27 unknown, 105 active+clean; 4.6 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[46934]: audit 2022-04-22T18:14:25.581966+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-14","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[46934]: cluster 2022-04-22T18:14:25.582206+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2022-04-22T18:14:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[46934]: audit 2022-04-22T18:14:25.779980+0000 mon.a (mon.0) 3701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:26 smithi115 bash[46934]: audit 2022-04-22T18:14:26.249132+0000 mon.a (mon.0) 3702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:27.638 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.group_create (2048 ms) 2022-04-22T18:14:27.638 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.group_createPP 2022-04-22T18:14:27.661 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.group_createPP (24 ms) 2022-04-22T18:14:27.662 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.add_image 2022-04-22T18:14:27.668 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:27.668 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.add_image (6 ms) 2022-04-22T18:14:27.668 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.add_imagePP 2022-04-22T18:14:27.675 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:27.675 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.add_imagePP (6 ms) 2022-04-22T18:14:27.675 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.add_snapshot 2022-04-22T18:14:27.682 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:27.682 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.add_snapshot (6 ms) 2022-04-22T18:14:27.682 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestGroup.add_snapshotPP 2022-04-22T18:14:27.691 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:27.691 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestGroup.add_snapshotPP (7 ms) 2022-04-22T18:14:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:27 smithi121 bash[36902]: cluster 2022-04-22T18:14:26.588757+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2022-04-22T18:14:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:27 smithi115 bash[45495]: cluster 2022-04-22T18:14:26.588757+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2022-04-22T18:14:28.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:27 smithi115 bash[46934]: cluster 2022-04-22T18:14:26.588757+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2022-04-22T18:14:28.615 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 6 tests from TestGroup (2098 ms total) 2022-04-22T18:14:28.615 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:14:28.615 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 57 tests from TestMigration 2022-04-22T18:14:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:28 smithi121 bash[36902]: cluster 2022-04-22T18:14:27.167758+0000 mgr.y (mgr.34107) 1700 : cluster [DBG] pgmap v2526: 137 pgs: 5 creating+peering, 13 unknown, 119 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:28 smithi121 bash[36902]: cluster 2022-04-22T18:14:27.597653+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2022-04-22T18:14:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:28 smithi121 bash[36902]: audit 2022-04-22T18:14:28.169591+0000 mon.a (mon.0) 3705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[46934]: cluster 2022-04-22T18:14:27.167758+0000 mgr.y (mgr.34107) 1700 : cluster [DBG] pgmap v2526: 137 pgs: 5 creating+peering, 13 unknown, 119 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:29.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[46934]: cluster 2022-04-22T18:14:27.597653+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2022-04-22T18:14:29.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[46934]: audit 2022-04-22T18:14:28.169591+0000 mon.a (mon.0) 3705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:29.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[45495]: cluster 2022-04-22T18:14:27.167758+0000 mgr.y (mgr.34107) 1700 : cluster [DBG] pgmap v2526: 137 pgs: 5 creating+peering, 13 unknown, 119 active+clean; 4.6 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:14:29.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[45495]: cluster 2022-04-22T18:14:27.597653+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2022-04-22T18:14:29.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:28 smithi115 bash[45495]: audit 2022-04-22T18:14:28.169591+0000 mon.a (mon.0) 3705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:29 smithi121 bash[36902]: cluster 2022-04-22T18:14:28.606313+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2022-04-22T18:14:30.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:29 smithi121 bash[36902]: cluster 2022-04-22T18:14:29.168633+0000 mgr.y (mgr.34107) 1701 : cluster [DBG] pgmap v2529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:30.085 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:29 smithi115 bash[45495]: cluster 2022-04-22T18:14:28.606313+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2022-04-22T18:14:30.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:29 smithi115 bash[45495]: cluster 2022-04-22T18:14:29.168633+0000 mgr.y (mgr.34107) 1701 : cluster [DBG] pgmap v2529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:30.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:29 smithi115 bash[46934]: cluster 2022-04-22T18:14:28.606313+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2022-04-22T18:14:30.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:29 smithi115 bash[46934]: cluster 2022-04-22T18:14:29.168633+0000 mgr.y (mgr.34107) 1701 : cluster [DBG] pgmap v2529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:31 smithi121 bash[36902]: cluster 2022-04-22T18:14:29.627798+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2022-04-22T18:14:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:31 smithi121 bash[36902]: audit 2022-04-22T18:14:29.629824+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:31 smithi115 bash[46934]: cluster 2022-04-22T18:14:29.627798+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2022-04-22T18:14:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:31 smithi115 bash[46934]: audit 2022-04-22T18:14:29.629824+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:31 smithi115 bash[45495]: cluster 2022-04-22T18:14:29.627798+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2022-04-22T18:14:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:31 smithi115 bash[45495]: audit 2022-04-22T18:14:29.629824+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:14:31.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Empty 2022-04-22T18:14:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:31.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: audit 2022-04-22T18:14:30.624860+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: cluster 2022-04-22T18:14:30.625054+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: audit 2022-04-22T18:14:31.077650+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: cluster 2022-04-22T18:14:31.169630+0000 mgr.y (mgr.34107) 1702 : cluster [DBG] pgmap v2532: 137 pgs: 11 creating+peering, 17 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: audit 2022-04-22T18:14:31.252393+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:32 smithi121 bash[36902]: cluster 2022-04-22T18:14:31.627596+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2022-04-22T18:14:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: audit 2022-04-22T18:14:30.624860+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: cluster 2022-04-22T18:14:30.625054+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2022-04-22T18:14:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: audit 2022-04-22T18:14:31.077650+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: cluster 2022-04-22T18:14:31.169630+0000 mgr.y (mgr.34107) 1702 : cluster [DBG] pgmap v2532: 137 pgs: 11 creating+peering, 17 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: audit 2022-04-22T18:14:31.252393+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[45495]: cluster 2022-04-22T18:14:31.627596+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: audit 2022-04-22T18:14:30.624860+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.115:0/3637883312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-15","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: cluster 2022-04-22T18:14:30.625054+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: audit 2022-04-22T18:14:31.077650+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: cluster 2022-04-22T18:14:31.169630+0000 mgr.y (mgr.34107) 1702 : cluster [DBG] pgmap v2532: 137 pgs: 11 creating+peering, 17 unknown, 109 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: audit 2022-04-22T18:14:31.252393+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:32 smithi115 bash[46934]: cluster 2022-04-22T18:14:31.627596+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2022-04-22T18:14:33.690 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:33.700 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:33 smithi121 bash[36902]: cluster 2022-04-22T18:14:32.631035+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2022-04-22T18:14:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:33 smithi121 bash[36902]: cluster 2022-04-22T18:14:33.170657+0000 mgr.y (mgr.34107) 1703 : cluster [DBG] pgmap v2535: 169 pgs: 22 creating+peering, 19 unknown, 128 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:33 smithi121 bash[36902]: audit 2022-04-22T18:14:33.172790+0000 mon.a (mon.0) 3715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:33.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[45495]: cluster 2022-04-22T18:14:32.631035+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2022-04-22T18:14:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[45495]: cluster 2022-04-22T18:14:33.170657+0000 mgr.y (mgr.34107) 1703 : cluster [DBG] pgmap v2535: 169 pgs: 22 creating+peering, 19 unknown, 128 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:34.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[45495]: audit 2022-04-22T18:14:33.172790+0000 mon.a (mon.0) 3715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:34.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[46934]: cluster 2022-04-22T18:14:32.631035+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2022-04-22T18:14:34.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[46934]: cluster 2022-04-22T18:14:33.170657+0000 mgr.y (mgr.34107) 1703 : cluster [DBG] pgmap v2535: 169 pgs: 22 creating+peering, 19 unknown, 128 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:34.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:33 smithi115 bash[46934]: audit 2022-04-22T18:14:33.172790+0000 mon.a (mon.0) 3715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:34 smithi121 bash[36902]: cluster 2022-04-22T18:14:33.637674+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2022-04-22T18:14:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:34 smithi115 bash[45495]: cluster 2022-04-22T18:14:33.637674+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2022-04-22T18:14:35.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:34 smithi115 bash[46934]: cluster 2022-04-22T18:14:33.637674+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2022-04-22T18:14:35.893 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:35 smithi121 bash[36902]: cluster 2022-04-22T18:14:34.654128+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2022-04-22T18:14:36.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:35 smithi121 bash[36902]: cluster 2022-04-22T18:14:35.171828+0000 mgr.y (mgr.34107) 1704 : cluster [DBG] pgmap v2538: 169 pgs: 11 creating+peering, 3 unknown, 155 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:36.026 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:36.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:35 smithi115 bash[45495]: cluster 2022-04-22T18:14:34.654128+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2022-04-22T18:14:36.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:35 smithi115 bash[45495]: cluster 2022-04-22T18:14:35.171828+0000 mgr.y (mgr.34107) 1704 : cluster [DBG] pgmap v2538: 169 pgs: 11 creating+peering, 3 unknown, 155 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:36.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:35 smithi115 bash[46934]: cluster 2022-04-22T18:14:34.654128+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2022-04-22T18:14:36.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:35 smithi115 bash[46934]: cluster 2022-04-22T18:14:35.171828+0000 mgr.y (mgr.34107) 1704 : cluster [DBG] pgmap v2538: 169 pgs: 11 creating+peering, 3 unknown, 155 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:14:36.181 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Empty (4540 ms) 2022-04-22T18:14:36.181 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.OtherName 2022-04-22T18:14:36.216 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:36.224 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:36.439 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:36.541 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:36.687 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.OtherName (506 ms) 2022-04-22T18:14:36.688 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.OtherPool 2022-04-22T18:14:36.724 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:36.735 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:36 smithi121 bash[36902]: cluster 2022-04-22T18:14:35.661415+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2022-04-22T18:14:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:36 smithi121 bash[36902]: audit 2022-04-22T18:14:35.787363+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:36 smithi121 bash[36902]: audit 2022-04-22T18:14:36.255831+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:37.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[45495]: cluster 2022-04-22T18:14:35.661415+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2022-04-22T18:14:37.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[45495]: audit 2022-04-22T18:14:35.787363+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:37.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[45495]: audit 2022-04-22T18:14:36.255831+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[46934]: cluster 2022-04-22T18:14:35.661415+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2022-04-22T18:14:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[46934]: audit 2022-04-22T18:14:35.787363+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:36 smithi115 bash[46934]: audit 2022-04-22T18:14:36.255831+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:37 smithi121 bash[36902]: cluster 2022-04-22T18:14:37.173182+0000 mgr.y (mgr.34107) 1705 : cluster [DBG] pgmap v2540: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.0 KiB/s wr, 73 op/s 2022-04-22T18:14:38.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:37 smithi115 bash[45495]: cluster 2022-04-22T18:14:37.173182+0000 mgr.y (mgr.34107) 1705 : cluster [DBG] pgmap v2540: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.0 KiB/s wr, 73 op/s 2022-04-22T18:14:38.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:37 smithi115 bash[46934]: cluster 2022-04-22T18:14:37.173182+0000 mgr.y (mgr.34107) 1705 : cluster [DBG] pgmap v2540: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.0 KiB/s wr, 73 op/s 2022-04-22T18:14:38.875 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:38.981 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:38 smithi121 bash[36902]: cluster 2022-04-22T18:14:37.670477+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2022-04-22T18:14:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:38 smithi121 bash[36902]: audit 2022-04-22T18:14:38.176136+0000 mon.a (mon.0) 3722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:38 smithi115 bash[45495]: cluster 2022-04-22T18:14:37.670477+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2022-04-22T18:14:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:38 smithi115 bash[45495]: audit 2022-04-22T18:14:38.176136+0000 mon.a (mon.0) 3722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:39.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:38 smithi115 bash[46934]: cluster 2022-04-22T18:14:37.670477+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e1363: 8 total, 8 up, 8 in 2022-04-22T18:14:39.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:38 smithi115 bash[46934]: audit 2022-04-22T18:14:38.176136+0000 mon.a (mon.0) 3722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:39.126 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.OtherPool (2440 ms) 2022-04-22T18:14:39.127 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.OtherNamespace 2022-04-22T18:14:39.162 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:39.175 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:39.389 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:39.499 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:39.641 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.OtherNamespace (515 ms) 2022-04-22T18:14:39.642 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.DataPool 2022-04-22T18:14:39.676 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:39.687 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:39.927 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:39 smithi121 bash[36902]: cluster 2022-04-22T18:14:38.676934+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2022-04-22T18:14:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:39 smithi121 bash[36902]: cluster 2022-04-22T18:14:39.174039+0000 mgr.y (mgr.34107) 1706 : cluster [DBG] pgmap v2543: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 8.4 KiB/s wr, 96 op/s 2022-04-22T18:14:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:39 smithi115 bash[45495]: cluster 2022-04-22T18:14:38.676934+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2022-04-22T18:14:40.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:39 smithi115 bash[45495]: cluster 2022-04-22T18:14:39.174039+0000 mgr.y (mgr.34107) 1706 : cluster [DBG] pgmap v2543: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 8.4 KiB/s wr, 96 op/s 2022-04-22T18:14:40.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:39 smithi115 bash[46934]: cluster 2022-04-22T18:14:38.676934+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1364: 8 total, 8 up, 8 in 2022-04-22T18:14:40.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:39 smithi115 bash[46934]: cluster 2022-04-22T18:14:39.174039+0000 mgr.y (mgr.34107) 1706 : cluster [DBG] pgmap v2543: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 8.4 KiB/s wr, 96 op/s 2022-04-22T18:14:40.058 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:40.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.DataPool (568 ms) 2022-04-22T18:14:40.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortAfterPrepare 2022-04-22T18:14:40.245 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:40.255 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:40.499 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:14:40.673 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortAfterPrepare (463 ms) 2022-04-22T18:14:40.673 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortAfterFailedPrepare 2022-04-22T18:14:40.701 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:40.709 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:40.828 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortAfterFailedPrepare (155 ms) 2022-04-22T18:14:40.829 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortAfterExecute 2022-04-22T18:14:40.866 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:40.876 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:41.125 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:40 smithi121 bash[36902]: audit 2022-04-22T18:14:40.790464+0000 mon.a (mon.0) 3724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:41.267 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:14:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:40 smithi115 bash[45495]: audit 2022-04-22T18:14:40.790464+0000 mon.a (mon.0) 3724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:40 smithi115 bash[46934]: audit 2022-04-22T18:14:40.790464+0000 mon.a (mon.0) 3724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:41.450 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortAfterExecute (623 ms) 2022-04-22T18:14:41.451 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.OtherPoolAbortAfterExecute 2022-04-22T18:14:41.485 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:41.493 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:41.685 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:41.780 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:14:41.943 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.OtherPoolAbortAfterExecute (492 ms) 2022-04-22T18:14:41.943 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.OtherNamespaceAbortAfterExecute 2022-04-22T18:14:41.974 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:41.986 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:41.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:42.182 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:42.279 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:14:42.466 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.OtherNamespaceAbortAfterExecute (523 ms) 2022-04-22T18:14:42.466 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.MirroringSamePool 2022-04-22T18:14:42.496 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.503 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.508 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.MirroringSamePool (43 ms) 2022-04-22T18:14:42.508 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.MirroringAbort 2022-04-22T18:14:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:42 smithi121 bash[36902]: cluster 2022-04-22T18:14:41.175113+0000 mgr.y (mgr.34107) 1707 : cluster [DBG] pgmap v2544: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 22 KiB/s wr, 305 op/s 2022-04-22T18:14:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:42 smithi121 bash[36902]: cluster 2022-04-22T18:14:41.188873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1365: 8 total, 8 up, 8 in 2022-04-22T18:14:42.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:42 smithi121 bash[36902]: audit 2022-04-22T18:14:41.258833+0000 mon.a (mon.0) 3726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:42.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:42 smithi121 bash[36902]: cluster 2022-04-22T18:14:41.842748+0000 mon.a (mon.0) 3727 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2022-04-22T18:14:42.540 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[46934]: cluster 2022-04-22T18:14:41.175113+0000 mgr.y (mgr.34107) 1707 : cluster [DBG] pgmap v2544: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 22 KiB/s wr, 305 op/s 2022-04-22T18:14:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[46934]: cluster 2022-04-22T18:14:41.188873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1365: 8 total, 8 up, 8 in 2022-04-22T18:14:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[46934]: audit 2022-04-22T18:14:41.258833+0000 mon.a (mon.0) 3726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[46934]: cluster 2022-04-22T18:14:41.842748+0000 mon.a (mon.0) 3727 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2022-04-22T18:14:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[45495]: cluster 2022-04-22T18:14:41.175113+0000 mgr.y (mgr.34107) 1707 : cluster [DBG] pgmap v2544: 169 pgs: 169 active+clean; 4.6 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 22 KiB/s wr, 305 op/s 2022-04-22T18:14:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[45495]: cluster 2022-04-22T18:14:41.188873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1365: 8 total, 8 up, 8 in 2022-04-22T18:14:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[45495]: audit 2022-04-22T18:14:41.258833+0000 mon.a (mon.0) 3726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:42.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:42 smithi115 bash[45495]: cluster 2022-04-22T18:14:41.842748+0000 mon.a (mon.0) 3727 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2022-04-22T18:14:42.546 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.551 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.MirroringAbort (42 ms) 2022-04-22T18:14:42.551 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.MirroringOtherPoolDisabled 2022-04-22T18:14:42.578 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.588 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.593 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.MirroringOtherPoolDisabled (42 ms) 2022-04-22T18:14:42.593 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.MirroringOtherPoolEnabled 2022-04-22T18:14:42.624 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.634 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.639 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.MirroringOtherPoolEnabled (46 ms) 2022-04-22T18:14:42.639 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.MirroringPool 2022-04-22T18:14:42.671 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.683 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.688 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.MirroringPool (49 ms) 2022-04-22T18:14:42.688 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Group 2022-04-22T18:14:42.718 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.727 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.732 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Group (45 ms) 2022-04-22T18:14:42.732 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.GroupAbort 2022-04-22T18:14:42.766 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.772 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:14:42.777 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.GroupAbort (45 ms) 2022-04-22T18:14:42.777 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnaps 2022-04-22T18:14:42.817 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:42.824 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:42.889 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:42.958 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:43.247 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:43.419 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:43 smithi121 bash[36902]: audit 2022-04-22T18:14:43.179024+0000 mon.a (mon.0) 3728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:43.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:43 smithi115 bash[46934]: audit 2022-04-22T18:14:43.179024+0000 mon.a (mon.0) 3728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:43.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:43 smithi115 bash[45495]: audit 2022-04-22T18:14:43.179024+0000 mon.a (mon.0) 3728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:43.612 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnaps (834 ms) 2022-04-22T18:14:43.612 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsOtherPool 2022-04-22T18:14:43.649 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:43.674 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:43.733 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:43.789 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:43.798 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:43.867 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:44.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:43.660Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:44.120 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:44.275 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:44.461 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsOtherPool (849 ms) 2022-04-22T18:14:44.480 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsDataPool 2022-04-22T18:14:44.494 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:44.502 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:44 smithi121 bash[36902]: cluster 2022-04-22T18:14:43.176369+0000 mgr.y (mgr.34107) 1708 : cluster [DBG] pgmap v2546: 169 pgs: 169 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 985 KiB/s rd, 2.0 MiB/s wr, 419 op/s 2022-04-22T18:14:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:44 smithi115 bash[45495]: cluster 2022-04-22T18:14:43.176369+0000 mgr.y (mgr.34107) 1708 : cluster [DBG] pgmap v2546: 169 pgs: 169 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 985 KiB/s rd, 2.0 MiB/s wr, 419 op/s 2022-04-22T18:14:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:44 smithi115 bash[46934]: cluster 2022-04-22T18:14:43.176369+0000 mgr.y (mgr.34107) 1708 : cluster [DBG] pgmap v2546: 169 pgs: 169 active+clean; 17 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 985 KiB/s rd, 2.0 MiB/s wr, 419 op/s 2022-04-22T18:14:44.565 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:44.623 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:44.933 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:45.123 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:45.325 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsDataPool (864 ms) 2022-04-22T18:14:45.351 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsShrinkAfterPrepare 2022-04-22T18:14:45.353 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:45.359 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:45.425 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:45.473 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:45.770 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:14:45.784 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:45.950 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:45 smithi121 bash[36902]: audit 2022-04-22T18:14:44.585639+0000 mon.a (mon.0) 3729 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:46.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:45 smithi115 bash[45495]: audit 2022-04-22T18:14:44.585639+0000 mon.a (mon.0) 3729 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:45 smithi115 bash[46934]: audit 2022-04-22T18:14:44.585639+0000 mon.a (mon.0) 3729 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:46.109 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsShrinkAfterPrepare (785 ms) 2022-04-22T18:14:46.110 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroBeforePrepare 2022-04-22T18:14:46.143 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:46.150 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:46.218 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:46.271 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 0 2022-04-22T18:14:46.288 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:46.429 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:46.556 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:46.625 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroBeforePrepare (516 ms) 2022-04-22T18:14:46.625 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroAfterPrepare 2022-04-22T18:14:46.662 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:46.669 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:46.737 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:46.799 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:46 smithi121 bash[36902]: cluster 2022-04-22T18:14:45.177962+0000 mgr.y (mgr.34107) 1709 : cluster [DBG] pgmap v2547: 169 pgs: 169 active+clean; 49 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 501 op/s 2022-04-22T18:14:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:46 smithi121 bash[36902]: audit 2022-04-22T18:14:45.793636+0000 mon.a (mon.0) 3730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:46 smithi121 bash[36902]: cluster 2022-04-22T18:14:46.188393+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e1366: 8 total, 8 up, 8 in 2022-04-22T18:14:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:46 smithi121 bash[36902]: audit 2022-04-22T18:14:46.262082+0000 mon.a (mon.0) 3732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:47.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[45495]: cluster 2022-04-22T18:14:45.177962+0000 mgr.y (mgr.34107) 1709 : cluster [DBG] pgmap v2547: 169 pgs: 169 active+clean; 49 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 501 op/s 2022-04-22T18:14:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[45495]: audit 2022-04-22T18:14:45.793636+0000 mon.a (mon.0) 3730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[45495]: cluster 2022-04-22T18:14:46.188393+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e1366: 8 total, 8 up, 8 in 2022-04-22T18:14:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[45495]: audit 2022-04-22T18:14:46.262082+0000 mon.a (mon.0) 3732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[46934]: cluster 2022-04-22T18:14:45.177962+0000 mgr.y (mgr.34107) 1709 : cluster [DBG] pgmap v2547: 169 pgs: 169 active+clean; 49 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 501 op/s 2022-04-22T18:14:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[46934]: audit 2022-04-22T18:14:45.793636+0000 mon.a (mon.0) 3730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[46934]: cluster 2022-04-22T18:14:46.188393+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e1366: 8 total, 8 up, 8 in 2022-04-22T18:14:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:46 smithi115 bash[46934]: audit 2022-04-22T18:14:46.262082+0000 mon.a (mon.0) 3732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:47.099 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 0 2022-04-22T18:14:47.121 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:47.235 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:47.321 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroAfterPrepare (694 ms) 2022-04-22T18:14:47.376 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsExpandAfterPrepare 2022-04-22T18:14:47.377 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:47.377 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:47.431 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:47.483 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:47.777 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:14:47.783 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:47.967 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:48.237 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsExpandAfterPrepare (917 ms) 2022-04-22T18:14:48.238 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.NoSnapsSnapAfterPrepare 2022-04-22T18:14:48.274 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:48.284 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:48.346 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:14:48.403 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:48.692 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: after_prepare_snap 2022-04-22T18:14:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:48 smithi121 bash[36902]: cluster 2022-04-22T18:14:47.178997+0000 mgr.y (mgr.34107) 1710 : cluster [DBG] pgmap v2549: 169 pgs: 169 active+clean; 101 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 18 MiB/s wr, 572 op/s 2022-04-22T18:14:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:48 smithi121 bash[36902]: audit 2022-04-22T18:14:48.181616+0000 mon.a (mon.0) 3733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:48 smithi115 bash[45495]: cluster 2022-04-22T18:14:47.178997+0000 mgr.y (mgr.34107) 1710 : cluster [DBG] pgmap v2549: 169 pgs: 169 active+clean; 101 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 18 MiB/s wr, 572 op/s 2022-04-22T18:14:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:48 smithi115 bash[45495]: audit 2022-04-22T18:14:48.181616+0000 mon.a (mon.0) 3733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:48 smithi115 bash[46934]: cluster 2022-04-22T18:14:47.178997+0000 mgr.y (mgr.34107) 1710 : cluster [DBG] pgmap v2549: 169 pgs: 169 active+clean; 101 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 18 MiB/s wr, 572 op/s 2022-04-22T18:14:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:48 smithi115 bash[46934]: audit 2022-04-22T18:14:48.181616+0000 mon.a (mon.0) 3733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:49 smithi121 bash[36902]: cluster 2022-04-22T18:14:49.180364+0000 mgr.y (mgr.34107) 1711 : cluster [DBG] pgmap v2550: 169 pgs: 169 active+clean; 145 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 30 MiB/s wr, 565 op/s 2022-04-22T18:14:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:49 smithi121 bash[36902]: cluster 2022-04-22T18:14:49.612777+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2022-04-22T18:14:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:49 smithi115 bash[46934]: cluster 2022-04-22T18:14:49.180364+0000 mgr.y (mgr.34107) 1711 : cluster [DBG] pgmap v2550: 169 pgs: 169 active+clean; 145 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 30 MiB/s wr, 565 op/s 2022-04-22T18:14:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:49 smithi115 bash[46934]: cluster 2022-04-22T18:14:49.612777+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2022-04-22T18:14:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:49 smithi115 bash[45495]: cluster 2022-04-22T18:14:49.180364+0000 mgr.y (mgr.34107) 1711 : cluster [DBG] pgmap v2550: 169 pgs: 169 active+clean; 145 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 30 MiB/s wr, 565 op/s 2022-04-22T18:14:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:49 smithi115 bash[45495]: cluster 2022-04-22T18:14:49.612777+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e1367: 8 total, 8 up, 8 in 2022-04-22T18:14:50.629 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:14:50.666 INFO:tasks.workunit.client.0.smithi115.stdout:write: * 0~1000 2022-04-22T18:14:50.670 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:14:50.857 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:14:51.128 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.NoSnapsSnapAfterPrepare (2891 ms) 2022-04-22T18:14:51.129 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Snaps 2022-04-22T18:14:51.165 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:51.207 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:14:51.232 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: cluster 2022-04-22T18:14:50.614117+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e1368: 8 total, 8 up, 8 in 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:50.796533+0000 mon.a (mon.0) 3736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:51.110482+0000 mon.a (mon.0) 3737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:51.112933+0000 mon.a (mon.0) 3738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:51.114561+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: cluster 2022-04-22T18:14:51.181540+0000 mgr.y (mgr.34107) 1712 : cluster [DBG] pgmap v2553: 169 pgs: 169 active+clean; 193 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 50 MiB/s wr, 623 op/s 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:51.264430+0000 mon.a (mon.0) 3740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:51 smithi121 bash[36902]: audit 2022-04-22T18:14:51.473187+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:51.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: cluster 2022-04-22T18:14:50.614117+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e1368: 8 total, 8 up, 8 in 2022-04-22T18:14:51.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:50.796533+0000 mon.a (mon.0) 3736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:51.110482+0000 mon.a (mon.0) 3737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:14:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:51.112933+0000 mon.a (mon.0) 3738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:14:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:51.114561+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:14:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: cluster 2022-04-22T18:14:51.181540+0000 mgr.y (mgr.34107) 1712 : cluster [DBG] pgmap v2553: 169 pgs: 169 active+clean; 193 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 50 MiB/s wr, 623 op/s 2022-04-22T18:14:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:51.264430+0000 mon.a (mon.0) 3740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[45495]: audit 2022-04-22T18:14:51.473187+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: cluster 2022-04-22T18:14:50.614117+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e1368: 8 total, 8 up, 8 in 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:50.796533+0000 mon.a (mon.0) 3736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:51.110482+0000 mon.a (mon.0) 3737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:51.112933+0000 mon.a (mon.0) 3738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:51.114561+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: cluster 2022-04-22T18:14:51.181540+0000 mgr.y (mgr.34107) 1712 : cluster [DBG] pgmap v2553: 169 pgs: 169 active+clean; 193 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 50 MiB/s wr, 623 op/s 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:51.264430+0000 mon.a (mon.0) 3740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[46934]: audit 2022-04-22T18:14:51.473187+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:14:52.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:51.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:53.081 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:14:53.222 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:14:53.244 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:14:53.245 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:14:53.245 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:14:53.444 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:14:53.444 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (37m) 6m ago 51m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:14:53.444 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (51m) 7m ago 51m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (39m) 6m ago 52m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (39m) 7m ago 52m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (47m) 7m ago 57m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (47m) 6m ago 59m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (46m) 6m ago 59m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (45m) 7m ago 57m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (46m) 6m ago 57m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (39m) 6m ago 51m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (38m) 7m ago 51m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:14:53.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (44m) 6m ago 56m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (43m) 6m ago 56m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (43m) 6m ago 55m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (43m) 6m ago 55m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (42m) 7m ago 54m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (41m) 7m ago 54m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (41m) 7m ago 53m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (40m) 7m ago 53m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:14:53.446 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (38m) 7m ago 52m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:14:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:53 smithi121 bash[36902]: cluster 2022-04-22T18:14:52.200209+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e1369: 8 total, 8 up, 8 in 2022-04-22T18:14:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:53 smithi121 bash[36902]: audit 2022-04-22T18:14:53.184740+0000 mon.a (mon.0) 3743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:53 smithi115 bash[45495]: cluster 2022-04-22T18:14:52.200209+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e1369: 8 total, 8 up, 8 in 2022-04-22T18:14:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:53 smithi115 bash[45495]: audit 2022-04-22T18:14:53.184740+0000 mon.a (mon.0) 3743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:53 smithi115 bash[46934]: cluster 2022-04-22T18:14:52.200209+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e1369: 8 total, 8 up, 8 in 2022-04-22T18:14:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:53 smithi115 bash[46934]: audit 2022-04-22T18:14:53.184740+0000 mon.a (mon.0) 3743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:53.849 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:14:53.849 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:53.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:14:53.851 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:14:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:14:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:14:53.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:14:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:54 smithi121 bash[36902]: audit 2022-04-22T18:14:53.059367+0000 mgr.y (mgr.34107) 1713 : audit [DBG] from='client.46951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:54 smithi121 bash[36902]: cluster 2022-04-22T18:14:53.182571+0000 mgr.y (mgr.34107) 1714 : cluster [DBG] pgmap v2555: 169 pgs: 169 active+clean; 201 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 40 MiB/s wr, 481 op/s 2022-04-22T18:14:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:54 smithi121 bash[36902]: cluster 2022-04-22T18:14:53.201191+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e1370: 8 total, 8 up, 8 in 2022-04-22T18:14:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:54 smithi121 bash[36902]: audit 2022-04-22T18:14:53.420410+0000 mgr.y (mgr.34107) 1715 : audit [DBG] from='client.37029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:54 smithi121 bash[36902]: audit 2022-04-22T18:14:53.843998+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? 172.21.15.115:0/2776960023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[45495]: audit 2022-04-22T18:14:53.059367+0000 mgr.y (mgr.34107) 1713 : audit [DBG] from='client.46951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[45495]: cluster 2022-04-22T18:14:53.182571+0000 mgr.y (mgr.34107) 1714 : cluster [DBG] pgmap v2555: 169 pgs: 169 active+clean; 201 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 40 MiB/s wr, 481 op/s 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[45495]: cluster 2022-04-22T18:14:53.201191+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e1370: 8 total, 8 up, 8 in 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[45495]: audit 2022-04-22T18:14:53.420410+0000 mgr.y (mgr.34107) 1715 : audit [DBG] from='client.37029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[45495]: audit 2022-04-22T18:14:53.843998+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? 172.21.15.115:0/2776960023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[46934]: audit 2022-04-22T18:14:53.059367+0000 mgr.y (mgr.34107) 1713 : audit [DBG] from='client.46951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[46934]: cluster 2022-04-22T18:14:53.182571+0000 mgr.y (mgr.34107) 1714 : cluster [DBG] pgmap v2555: 169 pgs: 169 active+clean; 201 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 40 MiB/s wr, 481 op/s 2022-04-22T18:14:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[46934]: cluster 2022-04-22T18:14:53.201191+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e1370: 8 total, 8 up, 8 in 2022-04-22T18:14:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[46934]: audit 2022-04-22T18:14:53.420410+0000 mgr.y (mgr.34107) 1715 : audit [DBG] from='client.37029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:14:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:54 smithi115 bash[46934]: audit 2022-04-22T18:14:53.843998+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? 172.21.15.115:0/2776960023' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:14:55.244 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:14:55.247 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:14:55.247 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:14:55.259 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:14:55.269 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:14:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:55 smithi121 bash[36902]: cluster 2022-04-22T18:14:54.220437+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2022-04-22T18:14:55.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:55 smithi115 bash[45495]: cluster 2022-04-22T18:14:54.220437+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2022-04-22T18:14:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:55 smithi115 bash[46934]: cluster 2022-04-22T18:14:54.220437+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e1371: 8 total, 8 up, 8 in 2022-04-22T18:14:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:56 smithi121 bash[36902]: cluster 2022-04-22T18:14:55.183746+0000 mgr.y (mgr.34107) 1716 : cluster [DBG] pgmap v2558: 169 pgs: 169 active+clean; 225 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 25 MiB/s rd, 18 MiB/s wr, 136 op/s 2022-04-22T18:14:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:56 smithi121 bash[36902]: cluster 2022-04-22T18:14:55.232261+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2022-04-22T18:14:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:56 smithi121 bash[36902]: audit 2022-04-22T18:14:55.799583+0000 mon.a (mon.0) 3747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[46934]: cluster 2022-04-22T18:14:55.183746+0000 mgr.y (mgr.34107) 1716 : cluster [DBG] pgmap v2558: 169 pgs: 169 active+clean; 225 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 25 MiB/s rd, 18 MiB/s wr, 136 op/s 2022-04-22T18:14:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[46934]: cluster 2022-04-22T18:14:55.232261+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2022-04-22T18:14:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[46934]: audit 2022-04-22T18:14:55.799583+0000 mon.a (mon.0) 3747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:56.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[45495]: cluster 2022-04-22T18:14:55.183746+0000 mgr.y (mgr.34107) 1716 : cluster [DBG] pgmap v2558: 169 pgs: 169 active+clean; 225 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 25 MiB/s rd, 18 MiB/s wr, 136 op/s 2022-04-22T18:14:56.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[45495]: cluster 2022-04-22T18:14:55.232261+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e1372: 8 total, 8 up, 8 in 2022-04-22T18:14:56.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:56 smithi115 bash[45495]: audit 2022-04-22T18:14:55.799583+0000 mon.a (mon.0) 3747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:14:57.272 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:14:57.300 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:14:57.304 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:14:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:57 smithi121 bash[36902]: cluster 2022-04-22T18:14:56.247703+0000 mon.a (mon.0) 3748 : cluster [DBG] osdmap e1373: 8 total, 8 up, 8 in 2022-04-22T18:14:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:57 smithi121 bash[36902]: audit 2022-04-22T18:14:56.267794+0000 mon.a (mon.0) 3749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:57 smithi115 bash[45495]: cluster 2022-04-22T18:14:56.247703+0000 mon.a (mon.0) 3748 : cluster [DBG] osdmap e1373: 8 total, 8 up, 8 in 2022-04-22T18:14:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:57 smithi115 bash[45495]: audit 2022-04-22T18:14:56.267794+0000 mon.a (mon.0) 3749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:57 smithi115 bash[46934]: cluster 2022-04-22T18:14:56.247703+0000 mon.a (mon.0) 3748 : cluster [DBG] osdmap e1373: 8 total, 8 up, 8 in 2022-04-22T18:14:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:57 smithi115 bash[46934]: audit 2022-04-22T18:14:56.267794+0000 mon.a (mon.0) 3749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:14:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:58 smithi121 bash[36902]: cluster 2022-04-22T18:14:57.184794+0000 mgr.y (mgr.34107) 1717 : cluster [DBG] pgmap v2561: 169 pgs: 169 active+clean; 241 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 125 op/s 2022-04-22T18:14:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:58 smithi121 bash[36902]: cluster 2022-04-22T18:14:57.255489+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e1374: 8 total, 8 up, 8 in 2022-04-22T18:14:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:58 smithi121 bash[36902]: audit 2022-04-22T18:14:58.187614+0000 mon.a (mon.0) 3751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[45495]: cluster 2022-04-22T18:14:57.184794+0000 mgr.y (mgr.34107) 1717 : cluster [DBG] pgmap v2561: 169 pgs: 169 active+clean; 241 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 125 op/s 2022-04-22T18:14:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[45495]: cluster 2022-04-22T18:14:57.255489+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e1374: 8 total, 8 up, 8 in 2022-04-22T18:14:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[45495]: audit 2022-04-22T18:14:58.187614+0000 mon.a (mon.0) 3751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[46934]: cluster 2022-04-22T18:14:57.184794+0000 mgr.y (mgr.34107) 1717 : cluster [DBG] pgmap v2561: 169 pgs: 169 active+clean; 241 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 11 MiB/s wr, 125 op/s 2022-04-22T18:14:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[46934]: cluster 2022-04-22T18:14:57.255489+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e1374: 8 total, 8 up, 8 in 2022-04-22T18:14:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:58 smithi115 bash[46934]: audit 2022-04-22T18:14:58.187614+0000 mon.a (mon.0) 3751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:14:59.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:14:59 smithi115 bash[45495]: cluster 2022-04-22T18:14:58.264450+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e1375: 8 total, 8 up, 8 in 2022-04-22T18:14:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:14:59 smithi115 bash[46934]: cluster 2022-04-22T18:14:58.264450+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e1375: 8 total, 8 up, 8 in 2022-04-22T18:14:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:14:59 smithi121 bash[36902]: cluster 2022-04-22T18:14:58.264450+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e1375: 8 total, 8 up, 8 in 2022-04-22T18:15:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:00 smithi115 bash[45495]: cluster 2022-04-22T18:14:59.185853+0000 mgr.y (mgr.34107) 1718 : cluster [DBG] pgmap v2564: 169 pgs: 169 active+clean; 253 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T18:15:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:00 smithi115 bash[45495]: cluster 2022-04-22T18:14:59.268434+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2022-04-22T18:15:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:00 smithi115 bash[46934]: cluster 2022-04-22T18:14:59.185853+0000 mgr.y (mgr.34107) 1718 : cluster [DBG] pgmap v2564: 169 pgs: 169 active+clean; 253 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T18:15:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:00 smithi115 bash[46934]: cluster 2022-04-22T18:14:59.268434+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2022-04-22T18:15:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:00 smithi121 bash[36902]: cluster 2022-04-22T18:14:59.185853+0000 mgr.y (mgr.34107) 1718 : cluster [DBG] pgmap v2564: 169 pgs: 169 active+clean; 253 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 7 op/s 2022-04-22T18:15:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:00 smithi121 bash[36902]: cluster 2022-04-22T18:14:59.268434+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e1376: 8 total, 8 up, 8 in 2022-04-22T18:15:01.053 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:15:01.281 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:15:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[45495]: cluster 2022-04-22T18:15:00.282768+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2022-04-22T18:15:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[45495]: audit 2022-04-22T18:15:00.803074+0000 mon.a (mon.0) 3755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[45495]: audit 2022-04-22T18:15:01.270435+0000 mon.a (mon.0) 3756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[46934]: cluster 2022-04-22T18:15:00.282768+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2022-04-22T18:15:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[46934]: audit 2022-04-22T18:15:00.803074+0000 mon.a (mon.0) 3755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[46934]: audit 2022-04-22T18:15:01.270435+0000 mon.a (mon.0) 3756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:01 smithi121 bash[36902]: cluster 2022-04-22T18:15:00.282768+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e1377: 8 total, 8 up, 8 in 2022-04-22T18:15:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:01 smithi121 bash[36902]: audit 2022-04-22T18:15:00.803074+0000 mon.a (mon.0) 3755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:01 smithi121 bash[36902]: audit 2022-04-22T18:15:01.270435+0000 mon.a (mon.0) 3756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:01.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:02.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:02 smithi121 bash[36902]: cluster 2022-04-22T18:15:01.187014+0000 mgr.y (mgr.34107) 1719 : cluster [DBG] pgmap v2567: 169 pgs: 169 active+clean; 253 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:15:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:02 smithi115 bash[45495]: cluster 2022-04-22T18:15:01.187014+0000 mgr.y (mgr.34107) 1719 : cluster [DBG] pgmap v2567: 169 pgs: 169 active+clean; 253 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:15:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:02 smithi115 bash[46934]: cluster 2022-04-22T18:15:01.187014+0000 mgr.y (mgr.34107) 1719 : cluster [DBG] pgmap v2567: 169 pgs: 169 active+clean; 253 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 52 op/s 2022-04-22T18:15:03.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:03 smithi115 bash[45495]: cluster 2022-04-22T18:15:02.294135+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2022-04-22T18:15:03.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:03 smithi115 bash[45495]: audit 2022-04-22T18:15:03.190223+0000 mon.a (mon.0) 3758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:03.669 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:03 smithi115 bash[46934]: cluster 2022-04-22T18:15:02.294135+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2022-04-22T18:15:03.669 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:03 smithi115 bash[46934]: audit 2022-04-22T18:15:03.190223+0000 mon.a (mon.0) 3758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:03 smithi121 bash[36902]: cluster 2022-04-22T18:15:02.294135+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e1378: 8 total, 8 up, 8 in 2022-04-22T18:15:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:03 smithi121 bash[36902]: audit 2022-04-22T18:15:03.190223+0000 mon.a (mon.0) 3758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:03.663Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:04 smithi121 bash[36902]: cluster 2022-04-22T18:15:03.188150+0000 mgr.y (mgr.34107) 1720 : cluster [DBG] pgmap v2569: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 52 op/s 2022-04-22T18:15:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:04 smithi121 bash[36902]: cluster 2022-04-22T18:15:03.310709+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2022-04-22T18:15:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:04 smithi115 bash[45495]: cluster 2022-04-22T18:15:03.188150+0000 mgr.y (mgr.34107) 1720 : cluster [DBG] pgmap v2569: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 52 op/s 2022-04-22T18:15:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:04 smithi115 bash[45495]: cluster 2022-04-22T18:15:03.310709+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2022-04-22T18:15:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:04 smithi115 bash[46934]: cluster 2022-04-22T18:15:03.188150+0000 mgr.y (mgr.34107) 1720 : cluster [DBG] pgmap v2569: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 52 op/s 2022-04-22T18:15:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:04 smithi115 bash[46934]: cluster 2022-04-22T18:15:03.310709+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e1379: 8 total, 8 up, 8 in 2022-04-22T18:15:04.882 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Snaps (13753 ms) 2022-04-22T18:15:04.882 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsOtherPool 2022-04-22T18:15:04.918 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:04.928 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:05.003 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:15:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:05 smithi121 bash[36902]: cluster 2022-04-22T18:15:04.318041+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2022-04-22T18:15:05.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:05 smithi115 bash[45495]: cluster 2022-04-22T18:15:04.318041+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2022-04-22T18:15:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:05 smithi115 bash[46934]: cluster 2022-04-22T18:15:04.318041+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e1380: 8 total, 8 up, 8 in 2022-04-22T18:15:06.370 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:15:06.396 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:15:06.396 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:15:06.396 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:15:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:06 smithi121 bash[36902]: cluster 2022-04-22T18:15:05.189453+0000 mgr.y (mgr.34107) 1721 : cluster [DBG] pgmap v2572: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 3.3 MiB/s wr, 89 op/s 2022-04-22T18:15:06.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:06 smithi121 bash[36902]: cluster 2022-04-22T18:15:05.343657+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2022-04-22T18:15:06.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:06 smithi121 bash[36902]: audit 2022-04-22T18:15:05.805936+0000 mon.a (mon.0) 3762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:06.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:06 smithi121 bash[36902]: audit 2022-04-22T18:15:06.273590+0000 mon.a (mon.0) 3763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[45495]: cluster 2022-04-22T18:15:05.189453+0000 mgr.y (mgr.34107) 1721 : cluster [DBG] pgmap v2572: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 3.3 MiB/s wr, 89 op/s 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[45495]: cluster 2022-04-22T18:15:05.343657+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[45495]: audit 2022-04-22T18:15:05.805936+0000 mon.a (mon.0) 3762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[45495]: audit 2022-04-22T18:15:06.273590+0000 mon.a (mon.0) 3763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[46934]: cluster 2022-04-22T18:15:05.189453+0000 mgr.y (mgr.34107) 1721 : cluster [DBG] pgmap v2572: 169 pgs: 169 active+clean; 261 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 3.3 MiB/s wr, 89 op/s 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[46934]: cluster 2022-04-22T18:15:05.343657+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e1381: 8 total, 8 up, 8 in 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[46934]: audit 2022-04-22T18:15:05.805936+0000 mon.a (mon.0) 3762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:06 smithi115 bash[46934]: audit 2022-04-22T18:15:06.273590+0000 mon.a (mon.0) 3763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:07 smithi121 bash[36902]: cluster 2022-04-22T18:15:06.351132+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2022-04-22T18:15:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:07 smithi115 bash[45495]: cluster 2022-04-22T18:15:06.351132+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2022-04-22T18:15:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:07 smithi115 bash[46934]: cluster 2022-04-22T18:15:06.351132+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e1382: 8 total, 8 up, 8 in 2022-04-22T18:15:08.379 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:15:08.383 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:15:08.384 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:15:08.397 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:15:08.407 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:15:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:08 smithi121 bash[36902]: cluster 2022-04-22T18:15:07.190664+0000 mgr.y (mgr.34107) 1722 : cluster [DBG] pgmap v2575: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 6.0 MiB/s wr, 181 op/s 2022-04-22T18:15:08.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:08 smithi121 bash[36902]: cluster 2022-04-22T18:15:07.359182+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2022-04-22T18:15:08.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:08 smithi121 bash[36902]: audit 2022-04-22T18:15:08.193740+0000 mon.a (mon.0) 3766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[45495]: cluster 2022-04-22T18:15:07.190664+0000 mgr.y (mgr.34107) 1722 : cluster [DBG] pgmap v2575: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 6.0 MiB/s wr, 181 op/s 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[45495]: cluster 2022-04-22T18:15:07.359182+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[45495]: audit 2022-04-22T18:15:08.193740+0000 mon.a (mon.0) 3766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[46934]: cluster 2022-04-22T18:15:07.190664+0000 mgr.y (mgr.34107) 1722 : cluster [DBG] pgmap v2575: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 6.0 MiB/s wr, 181 op/s 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[46934]: cluster 2022-04-22T18:15:07.359182+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e1383: 8 total, 8 up, 8 in 2022-04-22T18:15:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:08 smithi115 bash[46934]: audit 2022-04-22T18:15:08.193740+0000 mon.a (mon.0) 3766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:09 smithi121 bash[36902]: cluster 2022-04-22T18:15:08.364031+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e1384: 8 total, 8 up, 8 in 2022-04-22T18:15:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:09 smithi115 bash[45495]: cluster 2022-04-22T18:15:08.364031+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e1384: 8 total, 8 up, 8 in 2022-04-22T18:15:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:09 smithi115 bash[46934]: cluster 2022-04-22T18:15:08.364031+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e1384: 8 total, 8 up, 8 in 2022-04-22T18:15:10.400 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:10.426 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:15:10.430 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:10.436 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:15:10.513 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:15:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:10 smithi121 bash[36902]: cluster 2022-04-22T18:15:09.191756+0000 mgr.y (mgr.34107) 1723 : cluster [DBG] pgmap v2578: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.0 MiB/s wr, 127 op/s 2022-04-22T18:15:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:10 smithi121 bash[36902]: cluster 2022-04-22T18:15:09.379570+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2022-04-22T18:15:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:10 smithi115 bash[45495]: cluster 2022-04-22T18:15:09.191756+0000 mgr.y (mgr.34107) 1723 : cluster [DBG] pgmap v2578: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.0 MiB/s wr, 127 op/s 2022-04-22T18:15:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:10 smithi115 bash[45495]: cluster 2022-04-22T18:15:09.379570+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2022-04-22T18:15:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:10 smithi115 bash[46934]: cluster 2022-04-22T18:15:09.191756+0000 mgr.y (mgr.34107) 1723 : cluster [DBG] pgmap v2578: 169 pgs: 169 active+clean; 273 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.0 MiB/s wr, 127 op/s 2022-04-22T18:15:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:10 smithi115 bash[46934]: cluster 2022-04-22T18:15:09.379570+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e1385: 8 total, 8 up, 8 in 2022-04-22T18:15:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:11 smithi121 bash[36902]: cluster 2022-04-22T18:15:10.385198+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2022-04-22T18:15:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:11 smithi121 bash[36902]: audit 2022-04-22T18:15:10.808855+0000 mon.a (mon.0) 3770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:11 smithi121 bash[36902]: cluster 2022-04-22T18:15:11.201767+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2022-04-22T18:15:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:11 smithi121 bash[36902]: audit 2022-04-22T18:15:11.277353+0000 mon.a (mon.0) 3772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[45495]: cluster 2022-04-22T18:15:10.385198+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[45495]: audit 2022-04-22T18:15:10.808855+0000 mon.a (mon.0) 3770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[45495]: cluster 2022-04-22T18:15:11.201767+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[45495]: audit 2022-04-22T18:15:11.277353+0000 mon.a (mon.0) 3772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[46934]: cluster 2022-04-22T18:15:10.385198+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[46934]: audit 2022-04-22T18:15:10.808855+0000 mon.a (mon.0) 3770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[46934]: cluster 2022-04-22T18:15:11.201767+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2022-04-22T18:15:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[46934]: audit 2022-04-22T18:15:11.277353+0000 mon.a (mon.0) 3772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:11.784Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:12 smithi121 bash[36902]: cluster 2022-04-22T18:15:11.192773+0000 mgr.y (mgr.34107) 1724 : cluster [DBG] pgmap v2581: 169 pgs: 169 active+clean; 301 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 20 op/s 2022-04-22T18:15:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:12 smithi121 bash[36902]: cluster 2022-04-22T18:15:12.212554+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2022-04-22T18:15:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:12 smithi115 bash[45495]: cluster 2022-04-22T18:15:11.192773+0000 mgr.y (mgr.34107) 1724 : cluster [DBG] pgmap v2581: 169 pgs: 169 active+clean; 301 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 20 op/s 2022-04-22T18:15:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:12 smithi115 bash[45495]: cluster 2022-04-22T18:15:12.212554+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2022-04-22T18:15:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:12 smithi115 bash[46934]: cluster 2022-04-22T18:15:11.192773+0000 mgr.y (mgr.34107) 1724 : cluster [DBG] pgmap v2581: 169 pgs: 169 active+clean; 301 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 20 op/s 2022-04-22T18:15:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:12 smithi115 bash[46934]: cluster 2022-04-22T18:15:12.212554+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2022-04-22T18:15:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:13 smithi115 bash[45495]: audit 2022-04-22T18:15:13.196866+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:13.700 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:13 smithi115 bash[45495]: cluster 2022-04-22T18:15:13.214523+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2022-04-22T18:15:13.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:13 smithi115 bash[46934]: audit 2022-04-22T18:15:13.196866+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:13.701 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:13 smithi115 bash[46934]: cluster 2022-04-22T18:15:13.214523+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2022-04-22T18:15:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:13 smithi121 bash[36902]: audit 2022-04-22T18:15:13.196866+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:13 smithi121 bash[36902]: cluster 2022-04-22T18:15:13.214523+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2022-04-22T18:15:13.947 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:15:14.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:13.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:14.145 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:15:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:14 smithi121 bash[36902]: cluster 2022-04-22T18:15:13.193913+0000 mgr.y (mgr.34107) 1725 : cluster [DBG] pgmap v2584: 169 pgs: 169 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 14 MiB/s wr, 30 op/s 2022-04-22T18:15:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:14 smithi121 bash[36902]: audit 2022-04-22T18:15:13.861328+0000 mon.a (mon.0) 3776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:15:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:14 smithi121 bash[36902]: audit 2022-04-22T18:15:13.867587+0000 mon.a (mon.0) 3777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:15:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[45495]: cluster 2022-04-22T18:15:13.193913+0000 mgr.y (mgr.34107) 1725 : cluster [DBG] pgmap v2584: 169 pgs: 169 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 14 MiB/s wr, 30 op/s 2022-04-22T18:15:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[45495]: audit 2022-04-22T18:15:13.861328+0000 mon.a (mon.0) 3776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:15:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[45495]: audit 2022-04-22T18:15:13.867587+0000 mon.a (mon.0) 3777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:15:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[46934]: cluster 2022-04-22T18:15:13.193913+0000 mgr.y (mgr.34107) 1725 : cluster [DBG] pgmap v2584: 169 pgs: 169 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 14 MiB/s wr, 30 op/s 2022-04-22T18:15:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[46934]: audit 2022-04-22T18:15:13.861328+0000 mon.a (mon.0) 3776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:15:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:14 smithi115 bash[46934]: audit 2022-04-22T18:15:13.867587+0000 mon.a (mon.0) 3777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:15:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:15 smithi121 bash[36902]: cluster 2022-04-22T18:15:14.416121+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2022-04-22T18:15:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:15 smithi121 bash[36902]: audit 2022-04-22T18:15:14.469255+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]: dispatch 2022-04-22T18:15:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:15 smithi115 bash[45495]: cluster 2022-04-22T18:15:14.416121+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2022-04-22T18:15:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:15 smithi115 bash[45495]: audit 2022-04-22T18:15:14.469255+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]: dispatch 2022-04-22T18:15:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:15 smithi115 bash[46934]: cluster 2022-04-22T18:15:14.416121+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2022-04-22T18:15:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:15 smithi115 bash[46934]: audit 2022-04-22T18:15:14.469255+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]: dispatch 2022-04-22T18:15:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:16 smithi121 bash[36902]: cluster 2022-04-22T18:15:15.195077+0000 mgr.y (mgr.34107) 1726 : cluster [DBG] pgmap v2587: 169 pgs: 169 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 66 op/s 2022-04-22T18:15:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:16 smithi121 bash[36902]: audit 2022-04-22T18:15:15.435406+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]': finished 2022-04-22T18:15:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:16 smithi121 bash[36902]: cluster 2022-04-22T18:15:15.435565+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2022-04-22T18:15:16.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:16 smithi121 bash[36902]: audit 2022-04-22T18:15:15.811939+0000 mon.a (mon.0) 3782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:16.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:16 smithi121 bash[36902]: audit 2022-04-22T18:15:16.281244+0000 mon.a (mon.0) 3783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:16.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[46934]: cluster 2022-04-22T18:15:15.195077+0000 mgr.y (mgr.34107) 1726 : cluster [DBG] pgmap v2587: 169 pgs: 169 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 66 op/s 2022-04-22T18:15:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[46934]: audit 2022-04-22T18:15:15.435406+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]': finished 2022-04-22T18:15:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[46934]: cluster 2022-04-22T18:15:15.435565+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2022-04-22T18:15:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[46934]: audit 2022-04-22T18:15:15.811939+0000 mon.a (mon.0) 3782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[46934]: audit 2022-04-22T18:15:16.281244+0000 mon.a (mon.0) 3783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[45495]: cluster 2022-04-22T18:15:15.195077+0000 mgr.y (mgr.34107) 1726 : cluster [DBG] pgmap v2587: 169 pgs: 169 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 66 op/s 2022-04-22T18:15:16.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[45495]: audit 2022-04-22T18:15:15.435406+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "80.f", "id": [4, 6]}]': finished 2022-04-22T18:15:16.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[45495]: cluster 2022-04-22T18:15:15.435565+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2022-04-22T18:15:16.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[45495]: audit 2022-04-22T18:15:15.811939+0000 mon.a (mon.0) 3782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:16.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:16 smithi115 bash[45495]: audit 2022-04-22T18:15:16.281244+0000 mon.a (mon.0) 3783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:16.988 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsOtherPool (12106 ms) 2022-04-22T18:15:16.988 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsDataPool 2022-04-22T18:15:17.020 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:17.028 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:17.101 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:15:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:17 smithi121 bash[36902]: cluster 2022-04-22T18:15:16.439907+0000 mon.a (mon.0) 3784 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2022-04-22T18:15:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:17 smithi115 bash[45495]: cluster 2022-04-22T18:15:16.439907+0000 mon.a (mon.0) 3784 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2022-04-22T18:15:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:17 smithi115 bash[46934]: cluster 2022-04-22T18:15:16.439907+0000 mon.a (mon.0) 3784 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2022-04-22T18:15:18.486 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:15:18.501 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:15:18.501 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:15:18.501 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:15:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:18 smithi121 bash[36902]: cluster 2022-04-22T18:15:17.196354+0000 mgr.y (mgr.34107) 1727 : cluster [DBG] pgmap v2590: 169 pgs: 169 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 5.0 MiB/s wr, 72 op/s 2022-04-22T18:15:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:18 smithi121 bash[36902]: cluster 2022-04-22T18:15:17.468098+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2022-04-22T18:15:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:18 smithi121 bash[36902]: audit 2022-04-22T18:15:18.200026+0000 mon.a (mon.0) 3786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:18.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[45495]: cluster 2022-04-22T18:15:17.196354+0000 mgr.y (mgr.34107) 1727 : cluster [DBG] pgmap v2590: 169 pgs: 169 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 5.0 MiB/s wr, 72 op/s 2022-04-22T18:15:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[45495]: cluster 2022-04-22T18:15:17.468098+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2022-04-22T18:15:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[45495]: audit 2022-04-22T18:15:18.200026+0000 mon.a (mon.0) 3786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[46934]: cluster 2022-04-22T18:15:17.196354+0000 mgr.y (mgr.34107) 1727 : cluster [DBG] pgmap v2590: 169 pgs: 169 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 5.0 MiB/s wr, 72 op/s 2022-04-22T18:15:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[46934]: cluster 2022-04-22T18:15:17.468098+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2022-04-22T18:15:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:18 smithi115 bash[46934]: audit 2022-04-22T18:15:18.200026+0000 mon.a (mon.0) 3786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:19 smithi121 bash[36902]: cluster 2022-04-22T18:15:18.470249+0000 mon.a (mon.0) 3787 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2022-04-22T18:15:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:19 smithi115 bash[45495]: cluster 2022-04-22T18:15:18.470249+0000 mon.a (mon.0) 3787 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2022-04-22T18:15:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:19 smithi115 bash[46934]: cluster 2022-04-22T18:15:18.470249+0000 mon.a (mon.0) 3787 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2022-04-22T18:15:20.508 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:15:20.511 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:15:20.512 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:15:20.518 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:15:20.531 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:15:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:20 smithi121 bash[36902]: cluster 2022-04-22T18:15:19.197422+0000 mgr.y (mgr.34107) 1728 : cluster [DBG] pgmap v2593: 169 pgs: 169 active+clean; 365 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 67 MiB/s rd, 10 MiB/s wr, 160 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:20 smithi121 bash[36902]: cluster 2022-04-22T18:15:19.488070+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2022-04-22T18:15:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:20 smithi115 bash[45495]: cluster 2022-04-22T18:15:19.197422+0000 mgr.y (mgr.34107) 1728 : cluster [DBG] pgmap v2593: 169 pgs: 169 active+clean; 365 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 67 MiB/s rd, 10 MiB/s wr, 160 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:20 smithi115 bash[45495]: cluster 2022-04-22T18:15:19.488070+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2022-04-22T18:15:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:20 smithi115 bash[46934]: cluster 2022-04-22T18:15:19.197422+0000 mgr.y (mgr.34107) 1728 : cluster [DBG] pgmap v2593: 169 pgs: 169 active+clean; 365 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 67 MiB/s rd, 10 MiB/s wr, 160 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:20 smithi115 bash[46934]: cluster 2022-04-22T18:15:19.488070+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2022-04-22T18:15:21.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[45495]: cluster 2022-04-22T18:15:20.495912+0000 mon.a (mon.0) 3789 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2022-04-22T18:15:21.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[45495]: audit 2022-04-22T18:15:20.815230+0000 mon.a (mon.0) 3790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:21.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[45495]: cluster 2022-04-22T18:15:21.207176+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2022-04-22T18:15:21.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[45495]: audit 2022-04-22T18:15:21.284895+0000 mon.a (mon.0) 3792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:21.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[46934]: cluster 2022-04-22T18:15:20.495912+0000 mon.a (mon.0) 3789 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2022-04-22T18:15:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[46934]: audit 2022-04-22T18:15:20.815230+0000 mon.a (mon.0) 3790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[46934]: cluster 2022-04-22T18:15:21.207176+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2022-04-22T18:15:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[46934]: audit 2022-04-22T18:15:21.284895+0000 mon.a (mon.0) 3792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:21 smithi121 bash[36902]: cluster 2022-04-22T18:15:20.495912+0000 mon.a (mon.0) 3789 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2022-04-22T18:15:22.045 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:21 smithi121 bash[36902]: audit 2022-04-22T18:15:20.815230+0000 mon.a (mon.0) 3790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:22.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:21 smithi121 bash[36902]: cluster 2022-04-22T18:15:21.207176+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2022-04-22T18:15:22.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:21 smithi121 bash[36902]: audit 2022-04-22T18:15:21.284895+0000 mon.a (mon.0) 3792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:21.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:22.235 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:22.263 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:15:22.265 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:15:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:22 smithi115 bash[45495]: cluster 2022-04-22T18:15:21.198482+0000 mgr.y (mgr.34107) 1729 : cluster [DBG] pgmap v2596: 169 pgs: 169 active+clean; 369 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 13 MiB/s wr, 147 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:22 smithi115 bash[45495]: cluster 2022-04-22T18:15:22.221312+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2022-04-22T18:15:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:22 smithi115 bash[46934]: cluster 2022-04-22T18:15:21.198482+0000 mgr.y (mgr.34107) 1729 : cluster [DBG] pgmap v2596: 169 pgs: 169 active+clean; 369 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 13 MiB/s wr, 147 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:22 smithi115 bash[46934]: cluster 2022-04-22T18:15:22.221312+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2022-04-22T18:15:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:22 smithi121 bash[36902]: cluster 2022-04-22T18:15:21.198482+0000 mgr.y (mgr.34107) 1729 : cluster [DBG] pgmap v2596: 169 pgs: 169 active+clean; 369 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 13 MiB/s wr, 147 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2022-04-22T18:15:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:22 smithi121 bash[36902]: cluster 2022-04-22T18:15:22.221312+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2022-04-22T18:15:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:23 smithi115 bash[45495]: audit 2022-04-22T18:15:23.202834+0000 mon.a (mon.0) 3794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:23 smithi115 bash[45495]: cluster 2022-04-22T18:15:23.222205+0000 mon.a (mon.0) 3795 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2022-04-22T18:15:23.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:23.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:23 smithi115 bash[46934]: audit 2022-04-22T18:15:23.202834+0000 mon.a (mon.0) 3794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:23 smithi115 bash[46934]: cluster 2022-04-22T18:15:23.222205+0000 mon.a (mon.0) 3795 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2022-04-22T18:15:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:23 smithi121 bash[36902]: audit 2022-04-22T18:15:23.202834+0000 mon.a (mon.0) 3794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:23 smithi121 bash[36902]: cluster 2022-04-22T18:15:23.222205+0000 mon.a (mon.0) 3795 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2022-04-22T18:15:24.223 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:15:24.575 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (38m) 7m ago 52m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (51m) 8m ago 51m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (40m) 7m ago 53m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (39m) 8m ago 53m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (47m) 8m ago 57m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (48m) 7m ago 59m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (47m) 7m ago 59m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:15:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (46m) 8m ago 57m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (46m) 7m ago 57m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (39m) 7m ago 52m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (39m) 8m ago 52m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (45m) 7m ago 57m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (44m) 7m ago 56m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (44m) 7m ago 56m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (43m) 7m ago 55m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (42m) 8m ago 55m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (42m) 8m ago 54m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:15:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (41m) 8m ago 54m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:15:24.578 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (40m) 8m ago 53m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:15:24.578 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (39m) 8m ago 52m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:15:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:24 smithi115 bash[45495]: cluster 2022-04-22T18:15:23.199643+0000 mgr.y (mgr.34107) 1730 : cluster [DBG] pgmap v2599: 169 pgs: 169 active+clean; 377 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2022-04-22T18:15:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:24 smithi115 bash[45495]: cluster 2022-04-22T18:15:24.225409+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2022-04-22T18:15:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:24 smithi115 bash[46934]: cluster 2022-04-22T18:15:23.199643+0000 mgr.y (mgr.34107) 1730 : cluster [DBG] pgmap v2599: 169 pgs: 169 active+clean; 377 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2022-04-22T18:15:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:24 smithi115 bash[46934]: cluster 2022-04-22T18:15:24.225409+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2022-04-22T18:15:24.981 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:15:24.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:15:24.983 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:15:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:24 smithi121 bash[36902]: cluster 2022-04-22T18:15:23.199643+0000 mgr.y (mgr.34107) 1730 : cluster [DBG] pgmap v2599: 169 pgs: 169 active+clean; 377 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2022-04-22T18:15:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:24 smithi121 bash[36902]: cluster 2022-04-22T18:15:24.225409+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2022-04-22T18:15:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[45495]: audit 2022-04-22T18:15:24.203423+0000 mgr.y (mgr.34107) 1731 : audit [DBG] from='client.37041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[45495]: audit 2022-04-22T18:15:24.550248+0000 mgr.y (mgr.34107) 1732 : audit [DBG] from='client.46975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[45495]: audit 2022-04-22T18:15:24.976096+0000 mon.a (mon.0) 3797 : audit [DBG] from='client.? 172.21.15.115:0/2269233505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[45495]: cluster 2022-04-22T18:15:25.227214+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2022-04-22T18:15:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[46934]: audit 2022-04-22T18:15:24.203423+0000 mgr.y (mgr.34107) 1731 : audit [DBG] from='client.37041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[46934]: audit 2022-04-22T18:15:24.550248+0000 mgr.y (mgr.34107) 1732 : audit [DBG] from='client.46975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[46934]: audit 2022-04-22T18:15:24.976096+0000 mon.a (mon.0) 3797 : audit [DBG] from='client.? 172.21.15.115:0/2269233505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:25 smithi115 bash[46934]: cluster 2022-04-22T18:15:25.227214+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2022-04-22T18:15:26.000 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:15:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:25 smithi121 bash[36902]: audit 2022-04-22T18:15:24.203423+0000 mgr.y (mgr.34107) 1731 : audit [DBG] from='client.37041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:25 smithi121 bash[36902]: audit 2022-04-22T18:15:24.550248+0000 mgr.y (mgr.34107) 1732 : audit [DBG] from='client.46975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:25 smithi121 bash[36902]: audit 2022-04-22T18:15:24.976096+0000 mon.a (mon.0) 3797 : audit [DBG] from='client.? 172.21.15.115:0/2269233505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:26.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:25 smithi121 bash[36902]: cluster 2022-04-22T18:15:25.227214+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2022-04-22T18:15:26.195 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[45495]: cluster 2022-04-22T18:15:25.200828+0000 mgr.y (mgr.34107) 1733 : cluster [DBG] pgmap v2602: 169 pgs: 169 active+clean; 397 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 MiB/s wr, 25 op/s 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[45495]: audit 2022-04-22T18:15:25.818290+0000 mon.a (mon.0) 3799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[45495]: cluster 2022-04-22T18:15:26.206307+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[45495]: audit 2022-04-22T18:15:26.288453+0000 mon.a (mon.0) 3801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[46934]: cluster 2022-04-22T18:15:25.200828+0000 mgr.y (mgr.34107) 1733 : cluster [DBG] pgmap v2602: 169 pgs: 169 active+clean; 397 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 MiB/s wr, 25 op/s 2022-04-22T18:15:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[46934]: audit 2022-04-22T18:15:25.818290+0000 mon.a (mon.0) 3799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[46934]: cluster 2022-04-22T18:15:26.206307+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2022-04-22T18:15:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:26 smithi115 bash[46934]: audit 2022-04-22T18:15:26.288453+0000 mon.a (mon.0) 3801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:26 smithi121 bash[36902]: cluster 2022-04-22T18:15:25.200828+0000 mgr.y (mgr.34107) 1733 : cluster [DBG] pgmap v2602: 169 pgs: 169 active+clean; 397 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 MiB/s wr, 25 op/s 2022-04-22T18:15:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:26 smithi121 bash[36902]: audit 2022-04-22T18:15:25.818290+0000 mon.a (mon.0) 3799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:26 smithi121 bash[36902]: cluster 2022-04-22T18:15:26.206307+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2022-04-22T18:15:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:26 smithi121 bash[36902]: audit 2022-04-22T18:15:26.288453+0000 mon.a (mon.0) 3801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:28 smithi121 bash[36902]: cluster 2022-04-22T18:15:27.201961+0000 mgr.y (mgr.34107) 1734 : cluster [DBG] pgmap v2605: 169 pgs: 169 active+clean; 401 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 153 op/s 2022-04-22T18:15:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:28 smithi121 bash[36902]: cluster 2022-04-22T18:15:27.219439+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2022-04-22T18:15:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:28 smithi121 bash[36902]: audit 2022-04-22T18:15:28.206239+0000 mon.a (mon.0) 3803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[45495]: cluster 2022-04-22T18:15:27.201961+0000 mgr.y (mgr.34107) 1734 : cluster [DBG] pgmap v2605: 169 pgs: 169 active+clean; 401 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 153 op/s 2022-04-22T18:15:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[45495]: cluster 2022-04-22T18:15:27.219439+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2022-04-22T18:15:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[45495]: audit 2022-04-22T18:15:28.206239+0000 mon.a (mon.0) 3803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[46934]: cluster 2022-04-22T18:15:27.201961+0000 mgr.y (mgr.34107) 1734 : cluster [DBG] pgmap v2605: 169 pgs: 169 active+clean; 401 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 153 op/s 2022-04-22T18:15:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[46934]: cluster 2022-04-22T18:15:27.219439+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2022-04-22T18:15:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:28 smithi115 bash[46934]: audit 2022-04-22T18:15:28.206239+0000 mon.a (mon.0) 3803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:29 smithi121 bash[36902]: cluster 2022-04-22T18:15:28.225902+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2022-04-22T18:15:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:29 smithi115 bash[45495]: cluster 2022-04-22T18:15:28.225902+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2022-04-22T18:15:29.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:29 smithi115 bash[46934]: cluster 2022-04-22T18:15:28.225902+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2022-04-22T18:15:29.823 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsDataPool (12835 ms) 2022-04-22T18:15:29.823 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsShrinkAfterPrepare 2022-04-22T18:15:29.856 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:29.863 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:29.926 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:15:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:30 smithi121 bash[36902]: cluster 2022-04-22T18:15:29.203020+0000 mgr.y (mgr.34107) 1735 : cluster [DBG] pgmap v2608: 169 pgs: 169 active+clean; 405 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 1.0 MiB/s wr, 197 op/s 2022-04-22T18:15:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:30 smithi121 bash[36902]: cluster 2022-04-22T18:15:29.234219+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2022-04-22T18:15:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:30 smithi115 bash[45495]: cluster 2022-04-22T18:15:29.203020+0000 mgr.y (mgr.34107) 1735 : cluster [DBG] pgmap v2608: 169 pgs: 169 active+clean; 405 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 1.0 MiB/s wr, 197 op/s 2022-04-22T18:15:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:30 smithi115 bash[45495]: cluster 2022-04-22T18:15:29.234219+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2022-04-22T18:15:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:30 smithi115 bash[46934]: cluster 2022-04-22T18:15:29.203020+0000 mgr.y (mgr.34107) 1735 : cluster [DBG] pgmap v2608: 169 pgs: 169 active+clean; 405 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 1.0 MiB/s wr, 197 op/s 2022-04-22T18:15:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:30 smithi115 bash[46934]: cluster 2022-04-22T18:15:29.234219+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2022-04-22T18:15:31.264 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:15:31.276 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:15:31.276 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:15:31.276 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:15:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:31 smithi121 bash[36902]: cluster 2022-04-22T18:15:30.242852+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2022-04-22T18:15:31.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:31 smithi121 bash[36902]: audit 2022-04-22T18:15:30.821184+0000 mon.a (mon.0) 3807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:31 smithi115 bash[45495]: cluster 2022-04-22T18:15:30.242852+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2022-04-22T18:15:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:31 smithi115 bash[45495]: audit 2022-04-22T18:15:30.821184+0000 mon.a (mon.0) 3807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:31 smithi115 bash[46934]: cluster 2022-04-22T18:15:30.242852+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2022-04-22T18:15:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:31 smithi115 bash[46934]: audit 2022-04-22T18:15:30.821184+0000 mon.a (mon.0) 3807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:32.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:31.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:32 smithi121 bash[36902]: cluster 2022-04-22T18:15:31.204128+0000 mgr.y (mgr.34107) 1736 : cluster [DBG] pgmap v2611: 169 pgs: 169 active+clean; 409 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 6.0 MiB/s wr, 74 op/s 2022-04-22T18:15:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:32 smithi121 bash[36902]: cluster 2022-04-22T18:15:31.247997+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2022-04-22T18:15:32.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:32 smithi121 bash[36902]: audit 2022-04-22T18:15:31.291572+0000 mon.a (mon.0) 3809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[46934]: cluster 2022-04-22T18:15:31.204128+0000 mgr.y (mgr.34107) 1736 : cluster [DBG] pgmap v2611: 169 pgs: 169 active+clean; 409 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 6.0 MiB/s wr, 74 op/s 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[46934]: cluster 2022-04-22T18:15:31.247997+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[46934]: audit 2022-04-22T18:15:31.291572+0000 mon.a (mon.0) 3809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[45495]: cluster 2022-04-22T18:15:31.204128+0000 mgr.y (mgr.34107) 1736 : cluster [DBG] pgmap v2611: 169 pgs: 169 active+clean; 409 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 57 MiB/s rd, 6.0 MiB/s wr, 74 op/s 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[45495]: cluster 2022-04-22T18:15:31.247997+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2022-04-22T18:15:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:32 smithi115 bash[45495]: audit 2022-04-22T18:15:31.291572+0000 mon.a (mon.0) 3809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:33.288 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:15:33.295 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:15:33.295 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:15:33.308 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:15:33.326 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:15:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:33 smithi115 bash[45495]: cluster 2022-04-22T18:15:32.267228+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2022-04-22T18:15:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:33 smithi115 bash[45495]: audit 2022-04-22T18:15:33.209218+0000 mon.a (mon.0) 3811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:33 smithi115 bash[46934]: cluster 2022-04-22T18:15:32.267228+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2022-04-22T18:15:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:33 smithi115 bash[46934]: audit 2022-04-22T18:15:33.209218+0000 mon.a (mon.0) 3811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:33 smithi121 bash[36902]: cluster 2022-04-22T18:15:32.267228+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2022-04-22T18:15:33.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:33 smithi121 bash[36902]: audit 2022-04-22T18:15:33.209218+0000 mon.a (mon.0) 3811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:34.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:33.664Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:34 smithi115 bash[46934]: cluster 2022-04-22T18:15:33.205287+0000 mgr.y (mgr.34107) 1737 : cluster [DBG] pgmap v2614: 169 pgs: 169 active+clean; 425 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 51 MiB/s rd, 11 MiB/s wr, 37 op/s 2022-04-22T18:15:34.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:34 smithi115 bash[46934]: cluster 2022-04-22T18:15:33.277069+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2022-04-22T18:15:34.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:34 smithi115 bash[45495]: cluster 2022-04-22T18:15:33.205287+0000 mgr.y (mgr.34107) 1737 : cluster [DBG] pgmap v2614: 169 pgs: 169 active+clean; 425 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 51 MiB/s rd, 11 MiB/s wr, 37 op/s 2022-04-22T18:15:34.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:34 smithi115 bash[45495]: cluster 2022-04-22T18:15:33.277069+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2022-04-22T18:15:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:34 smithi121 bash[36902]: cluster 2022-04-22T18:15:33.205287+0000 mgr.y (mgr.34107) 1737 : cluster [DBG] pgmap v2614: 169 pgs: 169 active+clean; 425 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 51 MiB/s rd, 11 MiB/s wr, 37 op/s 2022-04-22T18:15:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:34 smithi121 bash[36902]: cluster 2022-04-22T18:15:33.277069+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2022-04-22T18:15:35.313 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:35.346 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:15:35.351 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:15:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:35 smithi121 bash[36902]: cluster 2022-04-22T18:15:34.285239+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2022-04-22T18:15:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:35 smithi115 bash[45495]: cluster 2022-04-22T18:15:34.285239+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2022-04-22T18:15:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:35 smithi115 bash[46934]: cluster 2022-04-22T18:15:34.285239+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2022-04-22T18:15:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:36 smithi121 bash[36902]: cluster 2022-04-22T18:15:35.206486+0000 mgr.y (mgr.34107) 1738 : cluster [DBG] pgmap v2617: 169 pgs: 169 active+clean; 441 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 28 op/s 2022-04-22T18:15:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:36 smithi121 bash[36902]: cluster 2022-04-22T18:15:35.297511+0000 mon.a (mon.0) 3814 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2022-04-22T18:15:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:36 smithi121 bash[36902]: audit 2022-04-22T18:15:35.824364+0000 mon.a (mon.0) 3815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:36 smithi121 bash[36902]: cluster 2022-04-22T18:15:36.211665+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2022-04-22T18:15:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[45495]: cluster 2022-04-22T18:15:35.206486+0000 mgr.y (mgr.34107) 1738 : cluster [DBG] pgmap v2617: 169 pgs: 169 active+clean; 441 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 28 op/s 2022-04-22T18:15:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[45495]: cluster 2022-04-22T18:15:35.297511+0000 mon.a (mon.0) 3814 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2022-04-22T18:15:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[45495]: audit 2022-04-22T18:15:35.824364+0000 mon.a (mon.0) 3815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[45495]: cluster 2022-04-22T18:15:36.211665+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2022-04-22T18:15:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[46934]: cluster 2022-04-22T18:15:35.206486+0000 mgr.y (mgr.34107) 1738 : cluster [DBG] pgmap v2617: 169 pgs: 169 active+clean; 441 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 28 op/s 2022-04-22T18:15:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[46934]: cluster 2022-04-22T18:15:35.297511+0000 mon.a (mon.0) 3814 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2022-04-22T18:15:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[46934]: audit 2022-04-22T18:15:35.824364+0000 mon.a (mon.0) 3815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:36 smithi115 bash[46934]: cluster 2022-04-22T18:15:36.211665+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2022-04-22T18:15:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:37 smithi121 bash[36902]: audit 2022-04-22T18:15:36.298106+0000 mon.a (mon.0) 3817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:37 smithi121 bash[36902]: cluster 2022-04-22T18:15:37.225098+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2022-04-22T18:15:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:37 smithi115 bash[45495]: audit 2022-04-22T18:15:36.298106+0000 mon.a (mon.0) 3817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:37 smithi115 bash[45495]: cluster 2022-04-22T18:15:37.225098+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2022-04-22T18:15:37.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:37 smithi115 bash[46934]: audit 2022-04-22T18:15:36.298106+0000 mon.a (mon.0) 3817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:37 smithi115 bash[46934]: cluster 2022-04-22T18:15:37.225098+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2022-04-22T18:15:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:38 smithi121 bash[36902]: cluster 2022-04-22T18:15:37.207741+0000 mgr.y (mgr.34107) 1739 : cluster [DBG] pgmap v2620: 169 pgs: 169 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 6.0 MiB/s wr, 23 op/s 2022-04-22T18:15:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:38 smithi121 bash[36902]: audit 2022-04-22T18:15:38.212200+0000 mon.a (mon.0) 3819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:38 smithi121 bash[36902]: cluster 2022-04-22T18:15:38.227777+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2022-04-22T18:15:38.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[45495]: cluster 2022-04-22T18:15:37.207741+0000 mgr.y (mgr.34107) 1739 : cluster [DBG] pgmap v2620: 169 pgs: 169 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 6.0 MiB/s wr, 23 op/s 2022-04-22T18:15:38.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[45495]: audit 2022-04-22T18:15:38.212200+0000 mon.a (mon.0) 3819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:38.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[45495]: cluster 2022-04-22T18:15:38.227777+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2022-04-22T18:15:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[46934]: cluster 2022-04-22T18:15:37.207741+0000 mgr.y (mgr.34107) 1739 : cluster [DBG] pgmap v2620: 169 pgs: 169 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 6.0 MiB/s wr, 23 op/s 2022-04-22T18:15:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[46934]: audit 2022-04-22T18:15:38.212200+0000 mon.a (mon.0) 3819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:38 smithi115 bash[46934]: cluster 2022-04-22T18:15:38.227777+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2022-04-22T18:15:38.997 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:15:39.018 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:15:39.219 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:15:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:40 smithi121 bash[36902]: cluster 2022-04-22T18:15:39.208889+0000 mgr.y (mgr.34107) 1740 : cluster [DBG] pgmap v2623: 169 pgs: 169 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2022-04-22T18:15:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:40 smithi121 bash[36902]: cluster 2022-04-22T18:15:39.312490+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2022-04-22T18:15:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:40 smithi115 bash[45495]: cluster 2022-04-22T18:15:39.208889+0000 mgr.y (mgr.34107) 1740 : cluster [DBG] pgmap v2623: 169 pgs: 169 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2022-04-22T18:15:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:40 smithi115 bash[45495]: cluster 2022-04-22T18:15:39.312490+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2022-04-22T18:15:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:40 smithi115 bash[46934]: cluster 2022-04-22T18:15:39.208889+0000 mgr.y (mgr.34107) 1740 : cluster [DBG] pgmap v2623: 169 pgs: 169 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2022-04-22T18:15:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:40 smithi115 bash[46934]: cluster 2022-04-22T18:15:39.312490+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2022-04-22T18:15:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:41 smithi121 bash[36902]: cluster 2022-04-22T18:15:40.317091+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2022-04-22T18:15:41.819 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:41 smithi121 bash[36902]: audit 2022-04-22T18:15:40.827768+0000 mon.a (mon.0) 3823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:41.819 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:41 smithi121 bash[36902]: audit 2022-04-22T18:15:41.298348+0000 mon.a (mon.0) 3824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:41.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[45495]: cluster 2022-04-22T18:15:40.317091+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2022-04-22T18:15:41.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[45495]: audit 2022-04-22T18:15:40.827768+0000 mon.a (mon.0) 3823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:41.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[45495]: audit 2022-04-22T18:15:41.298348+0000 mon.a (mon.0) 3824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:41.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[46934]: cluster 2022-04-22T18:15:40.317091+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2022-04-22T18:15:41.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[46934]: audit 2022-04-22T18:15:40.827768+0000 mon.a (mon.0) 3823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:41.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[46934]: audit 2022-04-22T18:15:41.298348+0000 mon.a (mon.0) 3824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:41.867 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsShrinkAfterPrepare (12044 ms) 2022-04-22T18:15:41.867 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroBeforePrepare 2022-04-22T18:15:41.902 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:41.909 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:41.977 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:15:42.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:41.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:42 smithi121 bash[36902]: cluster 2022-04-22T18:15:41.209928+0000 mgr.y (mgr.34107) 1741 : cluster [DBG] pgmap v2626: 169 pgs: 169 active+clean; 465 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 71 op/s 2022-04-22T18:15:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:42 smithi121 bash[36902]: cluster 2022-04-22T18:15:41.335831+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2022-04-22T18:15:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:42 smithi115 bash[45495]: cluster 2022-04-22T18:15:41.209928+0000 mgr.y (mgr.34107) 1741 : cluster [DBG] pgmap v2626: 169 pgs: 169 active+clean; 465 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 71 op/s 2022-04-22T18:15:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:42 smithi115 bash[45495]: cluster 2022-04-22T18:15:41.335831+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2022-04-22T18:15:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:42 smithi115 bash[46934]: cluster 2022-04-22T18:15:41.209928+0000 mgr.y (mgr.34107) 1741 : cluster [DBG] pgmap v2626: 169 pgs: 169 active+clean; 465 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 71 op/s 2022-04-22T18:15:42.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:42 smithi115 bash[46934]: cluster 2022-04-22T18:15:41.335831+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2022-04-22T18:15:43.376 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:15:43.394 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:15:43.395 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:15:43.395 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:15:43.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:43 smithi115 bash[45495]: cluster 2022-04-22T18:15:42.354331+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2022-04-22T18:15:43.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:43 smithi115 bash[45495]: audit 2022-04-22T18:15:43.214958+0000 mon.a (mon.0) 3827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:43.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:43 smithi115 bash[46934]: cluster 2022-04-22T18:15:42.354331+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2022-04-22T18:15:43.673 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:43 smithi115 bash[46934]: audit 2022-04-22T18:15:43.214958+0000 mon.a (mon.0) 3827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:43 smithi121 bash[36902]: cluster 2022-04-22T18:15:42.354331+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2022-04-22T18:15:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:43 smithi121 bash[36902]: audit 2022-04-22T18:15:43.214958+0000 mon.a (mon.0) 3827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:44.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:43.666Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:44 smithi121 bash[36902]: cluster 2022-04-22T18:15:43.211173+0000 mgr.y (mgr.34107) 1742 : cluster [DBG] pgmap v2629: 169 pgs: 169 active+clean; 469 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s rd, 8.0 MiB/s wr, 222 op/s 2022-04-22T18:15:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:44 smithi121 bash[36902]: cluster 2022-04-22T18:15:43.356825+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2022-04-22T18:15:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:44 smithi115 bash[45495]: cluster 2022-04-22T18:15:43.211173+0000 mgr.y (mgr.34107) 1742 : cluster [DBG] pgmap v2629: 169 pgs: 169 active+clean; 469 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s rd, 8.0 MiB/s wr, 222 op/s 2022-04-22T18:15:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:44 smithi115 bash[45495]: cluster 2022-04-22T18:15:43.356825+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2022-04-22T18:15:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:44 smithi115 bash[46934]: cluster 2022-04-22T18:15:43.211173+0000 mgr.y (mgr.34107) 1742 : cluster [DBG] pgmap v2629: 169 pgs: 169 active+clean; 469 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s rd, 8.0 MiB/s wr, 222 op/s 2022-04-22T18:15:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:44 smithi115 bash[46934]: cluster 2022-04-22T18:15:43.356825+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2022-04-22T18:15:45.398 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:15:45.406 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:15:45.406 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:15:45.418 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:15:45.427 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:15:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:45 smithi121 bash[36902]: cluster 2022-04-22T18:15:44.374514+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2022-04-22T18:15:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:45 smithi115 bash[45495]: cluster 2022-04-22T18:15:44.374514+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2022-04-22T18:15:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:45 smithi115 bash[46934]: cluster 2022-04-22T18:15:44.374514+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2022-04-22T18:15:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:46 smithi121 bash[36902]: cluster 2022-04-22T18:15:45.212442+0000 mgr.y (mgr.34107) 1743 : cluster [DBG] pgmap v2632: 169 pgs: 169 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s rd, 9.0 MiB/s wr, 181 op/s 2022-04-22T18:15:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:46 smithi121 bash[36902]: cluster 2022-04-22T18:15:45.379939+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2022-04-22T18:15:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:46 smithi121 bash[36902]: audit 2022-04-22T18:15:45.830776+0000 mon.a (mon.0) 3831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:46 smithi121 bash[36902]: cluster 2022-04-22T18:15:46.218661+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2022-04-22T18:15:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:46 smithi121 bash[36902]: audit 2022-04-22T18:15:46.302108+0000 mon.a (mon.0) 3833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[45495]: cluster 2022-04-22T18:15:45.212442+0000 mgr.y (mgr.34107) 1743 : cluster [DBG] pgmap v2632: 169 pgs: 169 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s rd, 9.0 MiB/s wr, 181 op/s 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[45495]: cluster 2022-04-22T18:15:45.379939+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[45495]: audit 2022-04-22T18:15:45.830776+0000 mon.a (mon.0) 3831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[45495]: cluster 2022-04-22T18:15:46.218661+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[45495]: audit 2022-04-22T18:15:46.302108+0000 mon.a (mon.0) 3833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[46934]: cluster 2022-04-22T18:15:45.212442+0000 mgr.y (mgr.34107) 1743 : cluster [DBG] pgmap v2632: 169 pgs: 169 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s rd, 9.0 MiB/s wr, 181 op/s 2022-04-22T18:15:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[46934]: cluster 2022-04-22T18:15:45.379939+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2022-04-22T18:15:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[46934]: audit 2022-04-22T18:15:45.830776+0000 mon.a (mon.0) 3831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[46934]: cluster 2022-04-22T18:15:46.218661+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2022-04-22T18:15:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:46 smithi115 bash[46934]: audit 2022-04-22T18:15:46.302108+0000 mon.a (mon.0) 3833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:47.247 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:47.277 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:15:47.283 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 0 2022-04-22T18:15:47.328 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:15:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:48 smithi121 bash[36902]: cluster 2022-04-22T18:15:47.213645+0000 mgr.y (mgr.34107) 1744 : cluster [DBG] pgmap v2635: 169 pgs: 169 active+clean; 489 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 14 op/s 2022-04-22T18:15:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:48 smithi121 bash[36902]: cluster 2022-04-22T18:15:47.233630+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2022-04-22T18:15:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:48 smithi121 bash[36902]: audit 2022-04-22T18:15:48.218087+0000 mon.a (mon.0) 3835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[45495]: cluster 2022-04-22T18:15:47.213645+0000 mgr.y (mgr.34107) 1744 : cluster [DBG] pgmap v2635: 169 pgs: 169 active+clean; 489 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 14 op/s 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[45495]: cluster 2022-04-22T18:15:47.233630+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[45495]: audit 2022-04-22T18:15:48.218087+0000 mon.a (mon.0) 3835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[46934]: cluster 2022-04-22T18:15:47.213645+0000 mgr.y (mgr.34107) 1744 : cluster [DBG] pgmap v2635: 169 pgs: 169 active+clean; 489 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 14 op/s 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[46934]: cluster 2022-04-22T18:15:47.233630+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2022-04-22T18:15:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:48 smithi115 bash[46934]: audit 2022-04-22T18:15:48.218087+0000 mon.a (mon.0) 3835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:49 smithi121 bash[36902]: cluster 2022-04-22T18:15:48.234725+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2022-04-22T18:15:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:49 smithi115 bash[45495]: cluster 2022-04-22T18:15:48.234725+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2022-04-22T18:15:49.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:49 smithi115 bash[46934]: cluster 2022-04-22T18:15:48.234725+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2022-04-22T18:15:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:50 smithi121 bash[36902]: cluster 2022-04-22T18:15:49.214827+0000 mgr.y (mgr.34107) 1745 : cluster [DBG] pgmap v2638: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2022-04-22T18:15:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:50 smithi121 bash[36902]: cluster 2022-04-22T18:15:49.251835+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2022-04-22T18:15:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:50 smithi115 bash[45495]: cluster 2022-04-22T18:15:49.214827+0000 mgr.y (mgr.34107) 1745 : cluster [DBG] pgmap v2638: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2022-04-22T18:15:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:50 smithi115 bash[45495]: cluster 2022-04-22T18:15:49.251835+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2022-04-22T18:15:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:50 smithi115 bash[46934]: cluster 2022-04-22T18:15:49.214827+0000 mgr.y (mgr.34107) 1745 : cluster [DBG] pgmap v2638: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2022-04-22T18:15:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:50 smithi115 bash[46934]: cluster 2022-04-22T18:15:49.251835+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2022-04-22T18:15:50.914 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:15:51.138 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:15:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[45495]: cluster 2022-04-22T18:15:50.264317+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2022-04-22T18:15:51.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[45495]: audit 2022-04-22T18:15:50.833503+0000 mon.a (mon.0) 3839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:51.529 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[45495]: cluster 2022-04-22T18:15:51.224249+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2022-04-22T18:15:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[46934]: cluster 2022-04-22T18:15:50.264317+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2022-04-22T18:15:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[46934]: audit 2022-04-22T18:15:50.833503+0000 mon.a (mon.0) 3839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:51.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[46934]: cluster 2022-04-22T18:15:51.224249+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2022-04-22T18:15:51.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:51 smithi121 bash[36902]: cluster 2022-04-22T18:15:50.264317+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2022-04-22T18:15:51.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:51 smithi121 bash[36902]: audit 2022-04-22T18:15:50.833503+0000 mon.a (mon.0) 3839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:51.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:51 smithi121 bash[36902]: cluster 2022-04-22T18:15:51.224249+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2022-04-22T18:15:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:51.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: cluster 2022-04-22T18:15:51.215964+0000 mgr.y (mgr.34107) 1746 : cluster [DBG] pgmap v2641: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 25 op/s 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: audit 2022-04-22T18:15:51.305912+0000 mon.a (mon.0) 3841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: audit 2022-04-22T18:15:51.480802+0000 mon.a (mon.0) 3842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: audit 2022-04-22T18:15:51.483709+0000 mon.a (mon.0) 3843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: audit 2022-04-22T18:15:51.485450+0000 mon.a (mon.0) 3844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: audit 2022-04-22T18:15:51.848538+0000 mon.a (mon.0) 3845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:15:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[45495]: cluster 2022-04-22T18:15:52.238498+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: cluster 2022-04-22T18:15:51.215964+0000 mgr.y (mgr.34107) 1746 : cluster [DBG] pgmap v2641: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 25 op/s 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: audit 2022-04-22T18:15:51.305912+0000 mon.a (mon.0) 3841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: audit 2022-04-22T18:15:51.480802+0000 mon.a (mon.0) 3842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: audit 2022-04-22T18:15:51.483709+0000 mon.a (mon.0) 3843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: audit 2022-04-22T18:15:51.485450+0000 mon.a (mon.0) 3844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: audit 2022-04-22T18:15:51.848538+0000 mon.a (mon.0) 3845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:15:52.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:52 smithi115 bash[46934]: cluster 2022-04-22T18:15:52.238498+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: cluster 2022-04-22T18:15:51.215964+0000 mgr.y (mgr.34107) 1746 : cluster [DBG] pgmap v2641: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 25 op/s 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: audit 2022-04-22T18:15:51.305912+0000 mon.a (mon.0) 3841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: audit 2022-04-22T18:15:51.480802+0000 mon.a (mon.0) 3842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: audit 2022-04-22T18:15:51.483709+0000 mon.a (mon.0) 3843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: audit 2022-04-22T18:15:51.485450+0000 mon.a (mon.0) 3844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: audit 2022-04-22T18:15:51.848538+0000 mon.a (mon.0) 3845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:15:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:52 smithi121 bash[36902]: cluster 2022-04-22T18:15:52.238498+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2022-04-22T18:15:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:53 smithi115 bash[46934]: audit 2022-04-22T18:15:53.221139+0000 mon.a (mon.0) 3847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:53 smithi115 bash[46934]: cluster 2022-04-22T18:15:53.241919+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2022-04-22T18:15:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:53 smithi115 bash[45495]: audit 2022-04-22T18:15:53.221139+0000 mon.a (mon.0) 3847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:53 smithi115 bash[45495]: cluster 2022-04-22T18:15:53.241919+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2022-04-22T18:15:53.718 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsShrinkToZeroBeforePrepare (11851 ms) 2022-04-22T18:15:53.718 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroAfterPrepare 2022-04-22T18:15:53.753 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:53.761 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:15:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:53 smithi121 bash[36902]: audit 2022-04-22T18:15:53.221139+0000 mon.a (mon.0) 3847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:53 smithi121 bash[36902]: cluster 2022-04-22T18:15:53.241919+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2022-04-22T18:15:53.819 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:15:54.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:15:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:15:53.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:15:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:54 smithi115 bash[45495]: cluster 2022-04-22T18:15:53.217112+0000 mgr.y (mgr.34107) 1747 : cluster [DBG] pgmap v2644: 169 pgs: 169 active+clean; 505 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2022-04-22T18:15:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:54 smithi115 bash[46934]: cluster 2022-04-22T18:15:53.217112+0000 mgr.y (mgr.34107) 1747 : cluster [DBG] pgmap v2644: 169 pgs: 169 active+clean; 505 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2022-04-22T18:15:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:54 smithi121 bash[36902]: cluster 2022-04-22T18:15:53.217112+0000 mgr.y (mgr.34107) 1747 : cluster [DBG] pgmap v2644: 169 pgs: 169 active+clean; 505 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2022-04-22T18:15:55.311 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:15:55.326 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:15:55.326 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:15:55.326 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:15:55.376 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:15:55.713 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:15:55.759 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (39m) 7m ago 52m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:15:55.759 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (52m) 8m ago 52m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:15:55.759 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (40m) 7m ago 53m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:15:55.759 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (40m) 8m ago 53m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:15:55.759 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (48m) 8m ago 58m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (48m) 7m ago 60m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (47m) 7m ago 60m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (46m) 8m ago 58m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (47m) 7m ago 58m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (40m) 7m ago 53m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (40m) 8m ago 53m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (45m) 7m ago 57m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (45m) 7m ago 57m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:15:55.760 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (44m) 7m ago 56m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (44m) 7m ago 56m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (43m) 8m ago 55m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (42m) 8m ago 55m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (42m) 8m ago 54m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (41m) 8m ago 54m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:15:55.761 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (39m) 8m ago 53m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:15:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:55 smithi121 bash[36902]: cluster 2022-04-22T18:15:54.284820+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2022-04-22T18:15:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:55 smithi115 bash[45495]: cluster 2022-04-22T18:15:54.284820+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2022-04-22T18:15:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:55 smithi115 bash[46934]: cluster 2022-04-22T18:15:54.284820+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e1430: 8 total, 8 up, 8 in 2022-04-22T18:15:56.126 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:15:56.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:15:56.128 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:15:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:56 smithi121 bash[36902]: cluster 2022-04-22T18:15:55.217972+0000 mgr.y (mgr.34107) 1748 : cluster [DBG] pgmap v2647: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 41 MiB/s rd, 5.0 MiB/s wr, 207 op/s 2022-04-22T18:15:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:56 smithi121 bash[36902]: cluster 2022-04-22T18:15:55.299267+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2022-04-22T18:15:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:56 smithi121 bash[36902]: audit 2022-04-22T18:15:55.356246+0000 mgr.y (mgr.34107) 1749 : audit [DBG] from='client.37059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:56 smithi121 bash[36902]: audit 2022-04-22T18:15:55.836064+0000 mon.a (mon.0) 3851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:56 smithi121 bash[36902]: audit 2022-04-22T18:15:56.121307+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? 172.21.15.115:0/3389719985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[46934]: cluster 2022-04-22T18:15:55.217972+0000 mgr.y (mgr.34107) 1748 : cluster [DBG] pgmap v2647: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 41 MiB/s rd, 5.0 MiB/s wr, 207 op/s 2022-04-22T18:15:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[46934]: cluster 2022-04-22T18:15:55.299267+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2022-04-22T18:15:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[46934]: audit 2022-04-22T18:15:55.356246+0000 mgr.y (mgr.34107) 1749 : audit [DBG] from='client.37059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[46934]: audit 2022-04-22T18:15:55.836064+0000 mon.a (mon.0) 3851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[46934]: audit 2022-04-22T18:15:56.121307+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? 172.21.15.115:0/3389719985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[45495]: cluster 2022-04-22T18:15:55.217972+0000 mgr.y (mgr.34107) 1748 : cluster [DBG] pgmap v2647: 169 pgs: 169 active+clean; 509 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 41 MiB/s rd, 5.0 MiB/s wr, 207 op/s 2022-04-22T18:15:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[45495]: cluster 2022-04-22T18:15:55.299267+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e1431: 8 total, 8 up, 8 in 2022-04-22T18:15:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[45495]: audit 2022-04-22T18:15:55.356246+0000 mgr.y (mgr.34107) 1749 : audit [DBG] from='client.37059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[45495]: audit 2022-04-22T18:15:55.836064+0000 mon.a (mon.0) 3851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:15:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:56 smithi115 bash[45495]: audit 2022-04-22T18:15:56.121307+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? 172.21.15.115:0/3389719985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:15:57.323 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:15:57.325 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:15:57.325 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:15:57.335 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:15:57.353 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:15:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:57 smithi121 bash[36902]: audit 2022-04-22T18:15:55.689908+0000 mgr.y (mgr.34107) 1750 : audit [DBG] from='client.46993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:57 smithi121 bash[36902]: cluster 2022-04-22T18:15:56.302521+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2022-04-22T18:15:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:57 smithi121 bash[36902]: audit 2022-04-22T18:15:56.309597+0000 mon.a (mon.0) 3853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[46934]: audit 2022-04-22T18:15:55.689908+0000 mgr.y (mgr.34107) 1750 : audit [DBG] from='client.46993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[46934]: cluster 2022-04-22T18:15:56.302521+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2022-04-22T18:15:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[46934]: audit 2022-04-22T18:15:56.309597+0000 mon.a (mon.0) 3853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:57.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[45495]: audit 2022-04-22T18:15:55.689908+0000 mgr.y (mgr.34107) 1750 : audit [DBG] from='client.46993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:15:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[45495]: cluster 2022-04-22T18:15:56.302521+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e1432: 8 total, 8 up, 8 in 2022-04-22T18:15:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:57 smithi115 bash[45495]: audit 2022-04-22T18:15:56.309597+0000 mon.a (mon.0) 3853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:15:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:58 smithi121 bash[36902]: cluster 2022-04-22T18:15:57.219162+0000 mgr.y (mgr.34107) 1751 : cluster [DBG] pgmap v2650: 169 pgs: 169 active+clean; 517 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 30 op/s 2022-04-22T18:15:58.835 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:58 smithi121 bash[36902]: cluster 2022-04-22T18:15:57.306797+0000 mon.a (mon.0) 3854 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2022-04-22T18:15:58.835 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:58 smithi121 bash[36902]: audit 2022-04-22T18:15:58.224292+0000 mon.a (mon.0) 3855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:58.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[46934]: cluster 2022-04-22T18:15:57.219162+0000 mgr.y (mgr.34107) 1751 : cluster [DBG] pgmap v2650: 169 pgs: 169 active+clean; 517 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 30 op/s 2022-04-22T18:15:58.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[46934]: cluster 2022-04-22T18:15:57.306797+0000 mon.a (mon.0) 3854 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2022-04-22T18:15:58.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[46934]: audit 2022-04-22T18:15:58.224292+0000 mon.a (mon.0) 3855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:58.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[45495]: cluster 2022-04-22T18:15:57.219162+0000 mgr.y (mgr.34107) 1751 : cluster [DBG] pgmap v2650: 169 pgs: 169 active+clean; 517 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 30 op/s 2022-04-22T18:15:58.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[45495]: cluster 2022-04-22T18:15:57.306797+0000 mon.a (mon.0) 3854 : cluster [DBG] osdmap e1433: 8 total, 8 up, 8 in 2022-04-22T18:15:58.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:58 smithi115 bash[45495]: audit 2022-04-22T18:15:58.224292+0000 mon.a (mon.0) 3855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:15:59.345 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:15:59.373 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:15:59.391 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:15:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:15:59 smithi121 bash[36902]: cluster 2022-04-22T18:15:58.317267+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2022-04-22T18:15:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:15:59 smithi115 bash[45495]: cluster 2022-04-22T18:15:58.317267+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2022-04-22T18:15:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:15:59 smithi115 bash[46934]: cluster 2022-04-22T18:15:58.317267+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e1434: 8 total, 8 up, 8 in 2022-04-22T18:16:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:00 smithi121 bash[36902]: cluster 2022-04-22T18:15:59.220275+0000 mgr.y (mgr.34107) 1752 : cluster [DBG] pgmap v2653: 169 pgs: 169 active+clean; 537 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 27 op/s 2022-04-22T18:16:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:00 smithi121 bash[36902]: cluster 2022-04-22T18:15:59.330947+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2022-04-22T18:16:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:00 smithi115 bash[45495]: cluster 2022-04-22T18:15:59.220275+0000 mgr.y (mgr.34107) 1752 : cluster [DBG] pgmap v2653: 169 pgs: 169 active+clean; 537 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 27 op/s 2022-04-22T18:16:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:00 smithi115 bash[45495]: cluster 2022-04-22T18:15:59.330947+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2022-04-22T18:16:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:00 smithi115 bash[46934]: cluster 2022-04-22T18:15:59.220275+0000 mgr.y (mgr.34107) 1752 : cluster [DBG] pgmap v2653: 169 pgs: 169 active+clean; 537 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 27 op/s 2022-04-22T18:16:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:00 smithi115 bash[46934]: cluster 2022-04-22T18:15:59.330947+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e1435: 8 total, 8 up, 8 in 2022-04-22T18:16:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:01 smithi121 bash[36902]: cluster 2022-04-22T18:16:00.352696+0000 mon.a (mon.0) 3858 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2022-04-22T18:16:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:01 smithi121 bash[36902]: audit 2022-04-22T18:16:00.838891+0000 mon.a (mon.0) 3859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:01 smithi121 bash[36902]: cluster 2022-04-22T18:16:01.225667+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2022-04-22T18:16:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:01 smithi121 bash[36902]: audit 2022-04-22T18:16:01.313573+0000 mon.a (mon.0) 3861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[45495]: cluster 2022-04-22T18:16:00.352696+0000 mon.a (mon.0) 3858 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2022-04-22T18:16:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[45495]: audit 2022-04-22T18:16:00.838891+0000 mon.a (mon.0) 3859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[45495]: cluster 2022-04-22T18:16:01.225667+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2022-04-22T18:16:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[45495]: audit 2022-04-22T18:16:01.313573+0000 mon.a (mon.0) 3861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[46934]: cluster 2022-04-22T18:16:00.352696+0000 mon.a (mon.0) 3858 : cluster [DBG] osdmap e1436: 8 total, 8 up, 8 in 2022-04-22T18:16:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[46934]: audit 2022-04-22T18:16:00.838891+0000 mon.a (mon.0) 3859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[46934]: cluster 2022-04-22T18:16:01.225667+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e1437: 8 total, 8 up, 8 in 2022-04-22T18:16:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[46934]: audit 2022-04-22T18:16:01.313573+0000 mon.a (mon.0) 3861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:02.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:01.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:02 smithi121 bash[36902]: cluster 2022-04-22T18:16:01.221514+0000 mgr.y (mgr.34107) 1753 : cluster [DBG] pgmap v2656: 169 pgs: 169 active+clean; 553 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 31 op/s 2022-04-22T18:16:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:02 smithi121 bash[36902]: cluster 2022-04-22T18:16:02.237340+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2022-04-22T18:16:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:02 smithi115 bash[45495]: cluster 2022-04-22T18:16:01.221514+0000 mgr.y (mgr.34107) 1753 : cluster [DBG] pgmap v2656: 169 pgs: 169 active+clean; 553 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 31 op/s 2022-04-22T18:16:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:02 smithi115 bash[45495]: cluster 2022-04-22T18:16:02.237340+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2022-04-22T18:16:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:02 smithi115 bash[46934]: cluster 2022-04-22T18:16:01.221514+0000 mgr.y (mgr.34107) 1753 : cluster [DBG] pgmap v2656: 169 pgs: 169 active+clean; 553 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 9.0 MiB/s wr, 31 op/s 2022-04-22T18:16:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:02 smithi115 bash[46934]: cluster 2022-04-22T18:16:02.237340+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e1438: 8 total, 8 up, 8 in 2022-04-22T18:16:03.020 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 0 2022-04-22T18:16:03.095 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:03.256 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:03.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:03 smithi115 bash[45495]: audit 2022-04-22T18:16:03.227060+0000 mon.a (mon.0) 3863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:03.673 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:03 smithi115 bash[46934]: audit 2022-04-22T18:16:03.227060+0000 mon.a (mon.0) 3863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:03 smithi121 bash[36902]: audit 2022-04-22T18:16:03.227060+0000 mon.a (mon.0) 3863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:04.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:03.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:04 smithi121 bash[36902]: cluster 2022-04-22T18:16:03.222518+0000 mgr.y (mgr.34107) 1754 : cluster [DBG] pgmap v2659: 169 pgs: 169 active+clean; 569 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 123 op/s 2022-04-22T18:16:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:04 smithi121 bash[36902]: cluster 2022-04-22T18:16:03.363798+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e1439: 8 total, 8 up, 8 in 2022-04-22T18:16:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:04 smithi115 bash[46934]: cluster 2022-04-22T18:16:03.222518+0000 mgr.y (mgr.34107) 1754 : cluster [DBG] pgmap v2659: 169 pgs: 169 active+clean; 569 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 123 op/s 2022-04-22T18:16:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:04 smithi115 bash[46934]: cluster 2022-04-22T18:16:03.363798+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e1439: 8 total, 8 up, 8 in 2022-04-22T18:16:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:04 smithi115 bash[45495]: cluster 2022-04-22T18:16:03.222518+0000 mgr.y (mgr.34107) 1754 : cluster [DBG] pgmap v2659: 169 pgs: 169 active+clean; 569 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 MiB/s rd, 6.0 MiB/s wr, 123 op/s 2022-04-22T18:16:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:04 smithi115 bash[45495]: cluster 2022-04-22T18:16:03.363798+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e1439: 8 total, 8 up, 8 in 2022-04-22T18:16:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:05 smithi121 bash[36902]: cluster 2022-04-22T18:16:04.369129+0000 mon.a (mon.0) 3865 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2022-04-22T18:16:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:05 smithi115 bash[45495]: cluster 2022-04-22T18:16:04.369129+0000 mon.a (mon.0) 3865 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2022-04-22T18:16:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:05 smithi115 bash[46934]: cluster 2022-04-22T18:16:04.369129+0000 mon.a (mon.0) 3865 : cluster [DBG] osdmap e1440: 8 total, 8 up, 8 in 2022-04-22T18:16:05.873 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsShrinkToZeroAfterPrepare (12155 ms) 2022-04-22T18:16:05.873 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare 2022-04-22T18:16:05.908 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:05.916 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:16:05.989 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:06 smithi121 bash[36902]: cluster 2022-04-22T18:16:05.223659+0000 mgr.y (mgr.34107) 1755 : cluster [DBG] pgmap v2662: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2022-04-22T18:16:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:06 smithi121 bash[36902]: cluster 2022-04-22T18:16:05.383701+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e1441: 8 total, 8 up, 8 in 2022-04-22T18:16:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:06 smithi121 bash[36902]: audit 2022-04-22T18:16:05.841907+0000 mon.a (mon.0) 3867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:06 smithi121 bash[36902]: audit 2022-04-22T18:16:06.317430+0000 mon.a (mon.0) 3868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[45495]: cluster 2022-04-22T18:16:05.223659+0000 mgr.y (mgr.34107) 1755 : cluster [DBG] pgmap v2662: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[45495]: cluster 2022-04-22T18:16:05.383701+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e1441: 8 total, 8 up, 8 in 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[45495]: audit 2022-04-22T18:16:05.841907+0000 mon.a (mon.0) 3867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[45495]: audit 2022-04-22T18:16:06.317430+0000 mon.a (mon.0) 3868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[46934]: cluster 2022-04-22T18:16:05.223659+0000 mgr.y (mgr.34107) 1755 : cluster [DBG] pgmap v2662: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[46934]: cluster 2022-04-22T18:16:05.383701+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e1441: 8 total, 8 up, 8 in 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[46934]: audit 2022-04-22T18:16:05.841907+0000 mon.a (mon.0) 3867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:06 smithi115 bash[46934]: audit 2022-04-22T18:16:06.317430+0000 mon.a (mon.0) 3868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:07.418 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:16:07.435 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:16:07.435 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:16:07.435 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:16:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:07 smithi121 bash[36902]: cluster 2022-04-22T18:16:06.393660+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e1442: 8 total, 8 up, 8 in 2022-04-22T18:16:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:07 smithi115 bash[45495]: cluster 2022-04-22T18:16:06.393660+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e1442: 8 total, 8 up, 8 in 2022-04-22T18:16:07.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:07 smithi115 bash[46934]: cluster 2022-04-22T18:16:06.393660+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e1442: 8 total, 8 up, 8 in 2022-04-22T18:16:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:08 smithi121 bash[36902]: cluster 2022-04-22T18:16:07.224813+0000 mgr.y (mgr.34107) 1756 : cluster [DBG] pgmap v2665: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 28 MiB/s rd, 4.0 MiB/s wr, 66 op/s 2022-04-22T18:16:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:08 smithi121 bash[36902]: cluster 2022-04-22T18:16:07.405055+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2022-04-22T18:16:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:08 smithi121 bash[36902]: audit 2022-04-22T18:16:08.229875+0000 mon.a (mon.0) 3871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[46934]: cluster 2022-04-22T18:16:07.224813+0000 mgr.y (mgr.34107) 1756 : cluster [DBG] pgmap v2665: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 28 MiB/s rd, 4.0 MiB/s wr, 66 op/s 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[46934]: cluster 2022-04-22T18:16:07.405055+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[46934]: audit 2022-04-22T18:16:08.229875+0000 mon.a (mon.0) 3871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[45495]: cluster 2022-04-22T18:16:07.224813+0000 mgr.y (mgr.34107) 1756 : cluster [DBG] pgmap v2665: 169 pgs: 169 active+clean; 573 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 28 MiB/s rd, 4.0 MiB/s wr, 66 op/s 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[45495]: cluster 2022-04-22T18:16:07.405055+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e1443: 8 total, 8 up, 8 in 2022-04-22T18:16:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:08 smithi115 bash[45495]: audit 2022-04-22T18:16:08.229875+0000 mon.a (mon.0) 3871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:09.438 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:16:09.469 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:16:09.469 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:16:09.469 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:16:09.469 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:16:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:09 smithi121 bash[36902]: cluster 2022-04-22T18:16:08.411499+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e1444: 8 total, 8 up, 8 in 2022-04-22T18:16:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:09 smithi115 bash[45495]: cluster 2022-04-22T18:16:08.411499+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e1444: 8 total, 8 up, 8 in 2022-04-22T18:16:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:09 smithi115 bash[46934]: cluster 2022-04-22T18:16:08.411499+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e1444: 8 total, 8 up, 8 in 2022-04-22T18:16:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:10 smithi121 bash[36902]: cluster 2022-04-22T18:16:09.225898+0000 mgr.y (mgr.34107) 1757 : cluster [DBG] pgmap v2668: 169 pgs: 169 active+clean; 593 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2022-04-22T18:16:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:10 smithi121 bash[36902]: cluster 2022-04-22T18:16:09.422928+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e1445: 8 total, 8 up, 8 in 2022-04-22T18:16:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:10 smithi115 bash[45495]: cluster 2022-04-22T18:16:09.225898+0000 mgr.y (mgr.34107) 1757 : cluster [DBG] pgmap v2668: 169 pgs: 169 active+clean; 593 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2022-04-22T18:16:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:10 smithi115 bash[45495]: cluster 2022-04-22T18:16:09.422928+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e1445: 8 total, 8 up, 8 in 2022-04-22T18:16:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:10 smithi115 bash[46934]: cluster 2022-04-22T18:16:09.225898+0000 mgr.y (mgr.34107) 1757 : cluster [DBG] pgmap v2668: 169 pgs: 169 active+clean; 593 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2022-04-22T18:16:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:10 smithi115 bash[46934]: cluster 2022-04-22T18:16:09.422928+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e1445: 8 total, 8 up, 8 in 2022-04-22T18:16:11.248 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:11.281 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:16:11.285 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:11 smithi121 bash[36902]: cluster 2022-04-22T18:16:10.430730+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e1446: 8 total, 8 up, 8 in 2022-04-22T18:16:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:11 smithi121 bash[36902]: audit 2022-04-22T18:16:10.844887+0000 mon.a (mon.0) 3875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:11 smithi121 bash[36902]: cluster 2022-04-22T18:16:11.230322+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2022-04-22T18:16:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:11 smithi121 bash[36902]: audit 2022-04-22T18:16:11.320814+0000 mon.a (mon.0) 3877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:11.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[45495]: cluster 2022-04-22T18:16:10.430730+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e1446: 8 total, 8 up, 8 in 2022-04-22T18:16:11.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[45495]: audit 2022-04-22T18:16:10.844887+0000 mon.a (mon.0) 3875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[45495]: cluster 2022-04-22T18:16:11.230322+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2022-04-22T18:16:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[45495]: audit 2022-04-22T18:16:11.320814+0000 mon.a (mon.0) 3877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[46934]: cluster 2022-04-22T18:16:10.430730+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e1446: 8 total, 8 up, 8 in 2022-04-22T18:16:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[46934]: audit 2022-04-22T18:16:10.844887+0000 mon.a (mon.0) 3875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[46934]: cluster 2022-04-22T18:16:11.230322+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e1447: 8 total, 8 up, 8 in 2022-04-22T18:16:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[46934]: audit 2022-04-22T18:16:11.320814+0000 mon.a (mon.0) 3877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:12.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:11.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:12 smithi121 bash[36902]: cluster 2022-04-22T18:16:11.226765+0000 mgr.y (mgr.34107) 1758 : cluster [DBG] pgmap v2672: 169 pgs: 169 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 10 MiB/s wr, 88 op/s 2022-04-22T18:16:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:12 smithi121 bash[36902]: cluster 2022-04-22T18:16:12.239586+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2022-04-22T18:16:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:12 smithi115 bash[45495]: cluster 2022-04-22T18:16:11.226765+0000 mgr.y (mgr.34107) 1758 : cluster [DBG] pgmap v2672: 169 pgs: 169 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 10 MiB/s wr, 88 op/s 2022-04-22T18:16:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:12 smithi115 bash[45495]: cluster 2022-04-22T18:16:12.239586+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2022-04-22T18:16:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:12 smithi115 bash[46934]: cluster 2022-04-22T18:16:11.226765+0000 mgr.y (mgr.34107) 1758 : cluster [DBG] pgmap v2672: 169 pgs: 169 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 10 MiB/s wr, 88 op/s 2022-04-22T18:16:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:12 smithi115 bash[46934]: cluster 2022-04-22T18:16:12.239586+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e1448: 8 total, 8 up, 8 in 2022-04-22T18:16:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:13 smithi121 bash[36902]: audit 2022-04-22T18:16:13.232996+0000 mon.a (mon.0) 3879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:13 smithi121 bash[36902]: cluster 2022-04-22T18:16:13.243150+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e1449: 8 total, 8 up, 8 in 2022-04-22T18:16:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:13 smithi115 bash[46934]: audit 2022-04-22T18:16:13.232996+0000 mon.a (mon.0) 3879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:13 smithi115 bash[46934]: cluster 2022-04-22T18:16:13.243150+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e1449: 8 total, 8 up, 8 in 2022-04-22T18:16:13.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:13.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:13 smithi115 bash[45495]: audit 2022-04-22T18:16:13.232996+0000 mon.a (mon.0) 3879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:13 smithi115 bash[45495]: cluster 2022-04-22T18:16:13.243150+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e1449: 8 total, 8 up, 8 in 2022-04-22T18:16:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:14 smithi121 bash[36902]: cluster 2022-04-22T18:16:13.228166+0000 mgr.y (mgr.34107) 1759 : cluster [DBG] pgmap v2674: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.0 MiB/s wr, 35 op/s 2022-04-22T18:16:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:14 smithi121 bash[36902]: audit 2022-04-22T18:16:13.866867+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:16:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:14 smithi121 bash[36902]: audit 2022-04-22T18:16:13.872913+0000 mon.a (mon.0) 3882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:16:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:14 smithi121 bash[36902]: cluster 2022-04-22T18:16:14.248388+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[45495]: cluster 2022-04-22T18:16:13.228166+0000 mgr.y (mgr.34107) 1759 : cluster [DBG] pgmap v2674: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.0 MiB/s wr, 35 op/s 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[45495]: audit 2022-04-22T18:16:13.866867+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[45495]: audit 2022-04-22T18:16:13.872913+0000 mon.a (mon.0) 3882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[45495]: cluster 2022-04-22T18:16:14.248388+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[46934]: cluster 2022-04-22T18:16:13.228166+0000 mgr.y (mgr.34107) 1759 : cluster [DBG] pgmap v2674: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.0 MiB/s wr, 35 op/s 2022-04-22T18:16:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[46934]: audit 2022-04-22T18:16:13.866867+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:16:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[46934]: audit 2022-04-22T18:16:13.872913+0000 mon.a (mon.0) 3882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:16:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:14 smithi115 bash[46934]: cluster 2022-04-22T18:16:14.248388+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e1450: 8 total, 8 up, 8 in 2022-04-22T18:16:14.971 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:16:14.979 INFO:tasks.workunit.client.0.smithi115.stdout:write: * 83886080~1000 2022-04-22T18:16:15.031 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:15.194 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:16 smithi121 bash[36902]: cluster 2022-04-22T18:16:15.229332+0000 mgr.y (mgr.34107) 1760 : cluster [DBG] pgmap v2677: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2022-04-22T18:16:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:16 smithi121 bash[36902]: cluster 2022-04-22T18:16:15.458764+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2022-04-22T18:16:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:16 smithi121 bash[36902]: audit 2022-04-22T18:16:15.848129+0000 mon.a (mon.0) 3885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:16 smithi121 bash[36902]: cluster 2022-04-22T18:16:16.236264+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e1452: 8 total, 8 up, 8 in 2022-04-22T18:16:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:16 smithi121 bash[36902]: audit 2022-04-22T18:16:16.323870+0000 mon.a (mon.0) 3887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[45495]: cluster 2022-04-22T18:16:15.229332+0000 mgr.y (mgr.34107) 1760 : cluster [DBG] pgmap v2677: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2022-04-22T18:16:16.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[45495]: cluster 2022-04-22T18:16:15.458764+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2022-04-22T18:16:16.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[45495]: audit 2022-04-22T18:16:15.848129+0000 mon.a (mon.0) 3885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[45495]: cluster 2022-04-22T18:16:16.236264+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e1452: 8 total, 8 up, 8 in 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[45495]: audit 2022-04-22T18:16:16.323870+0000 mon.a (mon.0) 3887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[46934]: cluster 2022-04-22T18:16:15.229332+0000 mgr.y (mgr.34107) 1760 : cluster [DBG] pgmap v2677: 169 pgs: 169 active+clean; 621 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[46934]: cluster 2022-04-22T18:16:15.458764+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e1451: 8 total, 8 up, 8 in 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[46934]: audit 2022-04-22T18:16:15.848129+0000 mon.a (mon.0) 3885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:16.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[46934]: cluster 2022-04-22T18:16:16.236264+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e1452: 8 total, 8 up, 8 in 2022-04-22T18:16:16.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:16 smithi115 bash[46934]: audit 2022-04-22T18:16:16.323870+0000 mon.a (mon.0) 3887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:17.891 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare (12018 ms) 2022-04-22T18:16:17.891 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare2 2022-04-22T18:16:17.925 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:18.052 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 41943040~10 2022-04-22T18:16:18.053 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:18 smithi121 bash[36902]: cluster 2022-04-22T18:16:17.230540+0000 mgr.y (mgr.34107) 1761 : cluster [DBG] pgmap v2680: 169 pgs: 169 active+clean; 617 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 1024 KiB/s wr, 10 op/s 2022-04-22T18:16:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:18 smithi121 bash[36902]: cluster 2022-04-22T18:16:17.251965+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e1453: 8 total, 8 up, 8 in 2022-04-22T18:16:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:18 smithi121 bash[36902]: audit 2022-04-22T18:16:18.236031+0000 mon.a (mon.0) 3889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[45495]: cluster 2022-04-22T18:16:17.230540+0000 mgr.y (mgr.34107) 1761 : cluster [DBG] pgmap v2680: 169 pgs: 169 active+clean; 617 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 1024 KiB/s wr, 10 op/s 2022-04-22T18:16:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[45495]: cluster 2022-04-22T18:16:17.251965+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e1453: 8 total, 8 up, 8 in 2022-04-22T18:16:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[45495]: audit 2022-04-22T18:16:18.236031+0000 mon.a (mon.0) 3889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[46934]: cluster 2022-04-22T18:16:17.230540+0000 mgr.y (mgr.34107) 1761 : cluster [DBG] pgmap v2680: 169 pgs: 169 active+clean; 617 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 1024 KiB/s wr, 10 op/s 2022-04-22T18:16:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[46934]: cluster 2022-04-22T18:16:17.251965+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e1453: 8 total, 8 up, 8 in 2022-04-22T18:16:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:18 smithi115 bash[46934]: audit 2022-04-22T18:16:18.236031+0000 mon.a (mon.0) 3889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:19.487 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:16:19.513 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:19 smithi121 bash[36902]: cluster 2022-04-22T18:16:18.470066+0000 mon.a (mon.0) 3890 : cluster [DBG] osdmap e1454: 8 total, 8 up, 8 in 2022-04-22T18:16:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:19 smithi115 bash[45495]: cluster 2022-04-22T18:16:18.470066+0000 mon.a (mon.0) 3890 : cluster [DBG] osdmap e1454: 8 total, 8 up, 8 in 2022-04-22T18:16:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:19 smithi115 bash[46934]: cluster 2022-04-22T18:16:18.470066+0000 mon.a (mon.0) 3890 : cluster [DBG] osdmap e1454: 8 total, 8 up, 8 in 2022-04-22T18:16:20.747 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:20.755 INFO:tasks.workunit.client.0.smithi115.stdout:write: Y 41943040~5 2022-04-22T18:16:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:20 smithi121 bash[36902]: cluster 2022-04-22T18:16:19.231570+0000 mgr.y (mgr.34107) 1762 : cluster [DBG] pgmap v2683: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2022-04-22T18:16:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:20 smithi121 bash[36902]: cluster 2022-04-22T18:16:19.475849+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e1455: 8 total, 8 up, 8 in 2022-04-22T18:16:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:20 smithi115 bash[46934]: cluster 2022-04-22T18:16:19.231570+0000 mgr.y (mgr.34107) 1762 : cluster [DBG] pgmap v2683: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2022-04-22T18:16:20.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:20 smithi115 bash[46934]: cluster 2022-04-22T18:16:19.475849+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e1455: 8 total, 8 up, 8 in 2022-04-22T18:16:20.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:20 smithi115 bash[45495]: cluster 2022-04-22T18:16:19.231570+0000 mgr.y (mgr.34107) 1762 : cluster [DBG] pgmap v2683: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2022-04-22T18:16:20.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:20 smithi115 bash[45495]: cluster 2022-04-22T18:16:19.475849+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e1455: 8 total, 8 up, 8 in 2022-04-22T18:16:20.961 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:21.083 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:21.696 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare2 (3805 ms) 2022-04-22T18:16:21.696 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsSnapAfterPrepare 2022-04-22T18:16:21.735 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:21.743 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:16:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:21 smithi121 bash[36902]: cluster 2022-04-22T18:16:20.483239+0000 mon.a (mon.0) 3892 : cluster [DBG] osdmap e1456: 8 total, 8 up, 8 in 2022-04-22T18:16:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:21 smithi121 bash[36902]: audit 2022-04-22T18:16:20.851005+0000 mon.a (mon.0) 3893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:21 smithi121 bash[36902]: audit 2022-04-22T18:16:21.327354+0000 mon.a (mon.0) 3894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[45495]: cluster 2022-04-22T18:16:20.483239+0000 mon.a (mon.0) 3892 : cluster [DBG] osdmap e1456: 8 total, 8 up, 8 in 2022-04-22T18:16:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[45495]: audit 2022-04-22T18:16:20.851005+0000 mon.a (mon.0) 3893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[45495]: audit 2022-04-22T18:16:21.327354+0000 mon.a (mon.0) 3894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[46934]: cluster 2022-04-22T18:16:20.483239+0000 mon.a (mon.0) 3892 : cluster [DBG] osdmap e1456: 8 total, 8 up, 8 in 2022-04-22T18:16:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[46934]: audit 2022-04-22T18:16:20.851005+0000 mon.a (mon.0) 3893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[46934]: audit 2022-04-22T18:16:21.327354+0000 mon.a (mon.0) 3894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:21.822 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:22.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:21.784Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:22 smithi121 bash[36902]: cluster 2022-04-22T18:16:21.232505+0000 mgr.y (mgr.34107) 1763 : cluster [DBG] pgmap v2686: 169 pgs: 169 active+clean; 625 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 50 MiB/s rd, 5.0 MiB/s wr, 228 op/s 2022-04-22T18:16:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:22 smithi121 bash[36902]: cluster 2022-04-22T18:16:21.493676+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e1457: 8 total, 8 up, 8 in 2022-04-22T18:16:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:22 smithi115 bash[45495]: cluster 2022-04-22T18:16:21.232505+0000 mgr.y (mgr.34107) 1763 : cluster [DBG] pgmap v2686: 169 pgs: 169 active+clean; 625 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 50 MiB/s rd, 5.0 MiB/s wr, 228 op/s 2022-04-22T18:16:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:22 smithi115 bash[45495]: cluster 2022-04-22T18:16:21.493676+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e1457: 8 total, 8 up, 8 in 2022-04-22T18:16:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:22 smithi115 bash[46934]: cluster 2022-04-22T18:16:21.232505+0000 mgr.y (mgr.34107) 1763 : cluster [DBG] pgmap v2686: 169 pgs: 169 active+clean; 625 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 50 MiB/s rd, 5.0 MiB/s wr, 228 op/s 2022-04-22T18:16:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:22 smithi115 bash[46934]: cluster 2022-04-22T18:16:21.493676+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e1457: 8 total, 8 up, 8 in 2022-04-22T18:16:23.524 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:16:23.532 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:16:23.533 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:16:23.533 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:16:23.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:23.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:23 smithi115 bash[45495]: cluster 2022-04-22T18:16:22.502643+0000 mon.a (mon.0) 3896 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2022-04-22T18:16:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:23 smithi115 bash[45495]: audit 2022-04-22T18:16:23.239107+0000 mon.a (mon.0) 3897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:23 smithi115 bash[46934]: cluster 2022-04-22T18:16:22.502643+0000 mon.a (mon.0) 3896 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2022-04-22T18:16:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:23 smithi115 bash[46934]: audit 2022-04-22T18:16:23.239107+0000 mon.a (mon.0) 3897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:23 smithi121 bash[36902]: cluster 2022-04-22T18:16:22.502643+0000 mon.a (mon.0) 3896 : cluster [DBG] osdmap e1458: 8 total, 8 up, 8 in 2022-04-22T18:16:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:23 smithi121 bash[36902]: audit 2022-04-22T18:16:23.239107+0000 mon.a (mon.0) 3897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:24 smithi115 bash[45495]: cluster 2022-04-22T18:16:23.233747+0000 mgr.y (mgr.34107) 1764 : cluster [DBG] pgmap v2689: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.0 MiB/s wr, 63 op/s 2022-04-22T18:16:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:24 smithi115 bash[45495]: cluster 2022-04-22T18:16:23.508462+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e1459: 8 total, 8 up, 8 in 2022-04-22T18:16:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:24 smithi115 bash[46934]: cluster 2022-04-22T18:16:23.233747+0000 mgr.y (mgr.34107) 1764 : cluster [DBG] pgmap v2689: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.0 MiB/s wr, 63 op/s 2022-04-22T18:16:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:24 smithi115 bash[46934]: cluster 2022-04-22T18:16:23.508462+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e1459: 8 total, 8 up, 8 in 2022-04-22T18:16:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:24 smithi121 bash[36902]: cluster 2022-04-22T18:16:23.233747+0000 mgr.y (mgr.34107) 1764 : cluster [DBG] pgmap v2689: 169 pgs: 169 active+clean; 629 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.0 MiB/s wr, 63 op/s 2022-04-22T18:16:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:24 smithi121 bash[36902]: cluster 2022-04-22T18:16:23.508462+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e1459: 8 total, 8 up, 8 in 2022-04-22T18:16:25.552 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:16:25.554 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:16:25.554 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:16:25.563 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:16:25.571 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:16:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:25 smithi115 bash[45495]: cluster 2022-04-22T18:16:24.526277+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e1460: 8 total, 8 up, 8 in 2022-04-22T18:16:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:25 smithi115 bash[46934]: cluster 2022-04-22T18:16:24.526277+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e1460: 8 total, 8 up, 8 in 2022-04-22T18:16:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:25 smithi121 bash[36902]: cluster 2022-04-22T18:16:24.526277+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e1460: 8 total, 8 up, 8 in 2022-04-22T18:16:26.505 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:16:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[46934]: cluster 2022-04-22T18:16:25.235096+0000 mgr.y (mgr.34107) 1765 : cluster [DBG] pgmap v2692: 169 pgs: 169 active+clean; 637 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 173 op/s 2022-04-22T18:16:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[46934]: cluster 2022-04-22T18:16:25.538013+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e1461: 8 total, 8 up, 8 in 2022-04-22T18:16:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[46934]: audit 2022-04-22T18:16:25.853742+0000 mon.a (mon.0) 3901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[46934]: cluster 2022-04-22T18:16:26.235085+0000 mon.a (mon.0) 3902 : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2022-04-22T18:16:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[46934]: audit 2022-04-22T18:16:26.330883+0000 mon.a (mon.0) 3903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[45495]: cluster 2022-04-22T18:16:25.235096+0000 mgr.y (mgr.34107) 1765 : cluster [DBG] pgmap v2692: 169 pgs: 169 active+clean; 637 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 173 op/s 2022-04-22T18:16:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[45495]: cluster 2022-04-22T18:16:25.538013+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e1461: 8 total, 8 up, 8 in 2022-04-22T18:16:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[45495]: audit 2022-04-22T18:16:25.853742+0000 mon.a (mon.0) 3901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[45495]: cluster 2022-04-22T18:16:26.235085+0000 mon.a (mon.0) 3902 : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2022-04-22T18:16:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:26 smithi115 bash[45495]: audit 2022-04-22T18:16:26.330883+0000 mon.a (mon.0) 3903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:26.853 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:16:26.853 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (39m) 8m ago 53m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:16:26.853 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (53m) 9m ago 53m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (41m) 8m ago 54m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (41m) 9m ago 54m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (48m) 9m ago 58m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (49m) 8m ago 60m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (48m) 8m ago 60m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:16:26.854 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (47m) 9m ago 58m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (47m) 8m ago 59m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (40m) 8m ago 53m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (40m) 9m ago 53m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (46m) 8m ago 58m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (45m) 8m ago 57m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (45m) 8m ago 57m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:16:26.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (44m) 8m ago 56m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:16:26.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (43m) 9m ago 56m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:16:26.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (43m) 9m ago 55m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:16:26.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (42m) 9m ago 55m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:16:26.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (41m) 9m ago 54m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:16:26.856 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (40m) 9m ago 53m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:16:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:26 smithi121 bash[36902]: cluster 2022-04-22T18:16:25.235096+0000 mgr.y (mgr.34107) 1765 : cluster [DBG] pgmap v2692: 169 pgs: 169 active+clean; 637 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 173 op/s 2022-04-22T18:16:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:26 smithi121 bash[36902]: cluster 2022-04-22T18:16:25.538013+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e1461: 8 total, 8 up, 8 in 2022-04-22T18:16:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:26 smithi121 bash[36902]: audit 2022-04-22T18:16:25.853742+0000 mon.a (mon.0) 3901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:26 smithi121 bash[36902]: cluster 2022-04-22T18:16:26.235085+0000 mon.a (mon.0) 3902 : cluster [DBG] osdmap e1462: 8 total, 8 up, 8 in 2022-04-22T18:16:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:26 smithi121 bash[36902]: audit 2022-04-22T18:16:26.330883+0000 mon.a (mon.0) 3903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:27.257 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:16:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:16:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:16:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:16:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:16:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:16:27.318 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:16:27.324 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:27.324 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:16:27.324 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:27 smithi121 bash[36902]: audit 2022-04-22T18:16:26.482946+0000 mgr.y (mgr.34107) 1766 : audit [DBG] from='client.37077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:27 smithi121 bash[36902]: cluster 2022-04-22T18:16:27.243900+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2022-04-22T18:16:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:27 smithi121 bash[36902]: audit 2022-04-22T18:16:27.250983+0000 mon.a (mon.0) 3905 : audit [DBG] from='client.? 172.21.15.115:0/1174505510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:16:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[45495]: audit 2022-04-22T18:16:26.482946+0000 mgr.y (mgr.34107) 1766 : audit [DBG] from='client.37077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[45495]: cluster 2022-04-22T18:16:27.243900+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2022-04-22T18:16:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[45495]: audit 2022-04-22T18:16:27.250983+0000 mon.a (mon.0) 3905 : audit [DBG] from='client.? 172.21.15.115:0/1174505510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:16:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[46934]: audit 2022-04-22T18:16:26.482946+0000 mgr.y (mgr.34107) 1766 : audit [DBG] from='client.37077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[46934]: cluster 2022-04-22T18:16:27.243900+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e1463: 8 total, 8 up, 8 in 2022-04-22T18:16:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:27 smithi115 bash[46934]: audit 2022-04-22T18:16:27.250983+0000 mon.a (mon.0) 3905 : audit [DBG] from='client.? 172.21.15.115:0/1174505510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:16:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:28 smithi121 bash[36902]: audit 2022-04-22T18:16:26.828388+0000 mgr.y (mgr.34107) 1767 : audit [DBG] from='client.37083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:28 smithi121 bash[36902]: cluster 2022-04-22T18:16:27.236481+0000 mgr.y (mgr.34107) 1768 : cluster [DBG] pgmap v2695: 169 pgs: 169 active+clean; 645 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 KiB/s rd, 3.0 MiB/s wr, 123 op/s 2022-04-22T18:16:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:28 smithi121 bash[36902]: audit 2022-04-22T18:16:28.242002+0000 mon.a (mon.0) 3906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:28 smithi121 bash[36902]: cluster 2022-04-22T18:16:28.248878+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2022-04-22T18:16:29.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[45495]: audit 2022-04-22T18:16:26.828388+0000 mgr.y (mgr.34107) 1767 : audit [DBG] from='client.37083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:29.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[45495]: cluster 2022-04-22T18:16:27.236481+0000 mgr.y (mgr.34107) 1768 : cluster [DBG] pgmap v2695: 169 pgs: 169 active+clean; 645 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 KiB/s rd, 3.0 MiB/s wr, 123 op/s 2022-04-22T18:16:29.112 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[45495]: audit 2022-04-22T18:16:28.242002+0000 mon.a (mon.0) 3906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:29.112 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[45495]: cluster 2022-04-22T18:16:28.248878+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2022-04-22T18:16:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[46934]: audit 2022-04-22T18:16:26.828388+0000 mgr.y (mgr.34107) 1767 : audit [DBG] from='client.37083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[46934]: cluster 2022-04-22T18:16:27.236481+0000 mgr.y (mgr.34107) 1768 : cluster [DBG] pgmap v2695: 169 pgs: 169 active+clean; 645 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 KiB/s rd, 3.0 MiB/s wr, 123 op/s 2022-04-22T18:16:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[46934]: audit 2022-04-22T18:16:28.242002+0000 mon.a (mon.0) 3906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:28 smithi115 bash[46934]: cluster 2022-04-22T18:16:28.248878+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e1464: 8 total, 8 up, 8 in 2022-04-22T18:16:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:30 smithi121 bash[36902]: cluster 2022-04-22T18:16:29.237520+0000 mgr.y (mgr.34107) 1769 : cluster [DBG] pgmap v2698: 169 pgs: 169 active+clean; 669 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 7.0 MiB/s wr, 12 op/s 2022-04-22T18:16:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:30 smithi121 bash[36902]: cluster 2022-04-22T18:16:29.251996+0000 mon.a (mon.0) 3908 : cluster [DBG] osdmap e1465: 8 total, 8 up, 8 in 2022-04-22T18:16:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:30 smithi115 bash[45495]: cluster 2022-04-22T18:16:29.237520+0000 mgr.y (mgr.34107) 1769 : cluster [DBG] pgmap v2698: 169 pgs: 169 active+clean; 669 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 7.0 MiB/s wr, 12 op/s 2022-04-22T18:16:30.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:30 smithi115 bash[45495]: cluster 2022-04-22T18:16:29.251996+0000 mon.a (mon.0) 3908 : cluster [DBG] osdmap e1465: 8 total, 8 up, 8 in 2022-04-22T18:16:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:30 smithi115 bash[46934]: cluster 2022-04-22T18:16:29.237520+0000 mgr.y (mgr.34107) 1769 : cluster [DBG] pgmap v2698: 169 pgs: 169 active+clean; 669 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 7.0 MiB/s wr, 12 op/s 2022-04-22T18:16:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:30 smithi115 bash[46934]: cluster 2022-04-22T18:16:29.251996+0000 mon.a (mon.0) 3908 : cluster [DBG] osdmap e1465: 8 total, 8 up, 8 in 2022-04-22T18:16:31.235 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: after_prepare_snap 2022-04-22T18:16:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[45495]: cluster 2022-04-22T18:16:30.254875+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e1466: 8 total, 8 up, 8 in 2022-04-22T18:16:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[45495]: audit 2022-04-22T18:16:30.856449+0000 mon.a (mon.0) 3910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[45495]: cluster 2022-04-22T18:16:31.238027+0000 mon.a (mon.0) 3911 : cluster [DBG] osdmap e1467: 8 total, 8 up, 8 in 2022-04-22T18:16:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[46934]: cluster 2022-04-22T18:16:30.254875+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e1466: 8 total, 8 up, 8 in 2022-04-22T18:16:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[46934]: audit 2022-04-22T18:16:30.856449+0000 mon.a (mon.0) 3910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[46934]: cluster 2022-04-22T18:16:31.238027+0000 mon.a (mon.0) 3911 : cluster [DBG] osdmap e1467: 8 total, 8 up, 8 in 2022-04-22T18:16:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:31 smithi121 bash[36902]: cluster 2022-04-22T18:16:30.254875+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e1466: 8 total, 8 up, 8 in 2022-04-22T18:16:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:31 smithi121 bash[36902]: audit 2022-04-22T18:16:30.856449+0000 mon.a (mon.0) 3910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:31 smithi121 bash[36902]: cluster 2022-04-22T18:16:31.238027+0000 mon.a (mon.0) 3911 : cluster [DBG] osdmap e1467: 8 total, 8 up, 8 in 2022-04-22T18:16:32.266 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:31.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[45495]: cluster 2022-04-22T18:16:31.238219+0000 mgr.y (mgr.34107) 1770 : cluster [DBG] pgmap v2702: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.0 MiB/s wr, 17 op/s 2022-04-22T18:16:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[45495]: audit 2022-04-22T18:16:31.334255+0000 mon.a (mon.0) 3912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[45495]: cluster 2022-04-22T18:16:32.246865+0000 mon.a (mon.0) 3913 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2022-04-22T18:16:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[46934]: cluster 2022-04-22T18:16:31.238219+0000 mgr.y (mgr.34107) 1770 : cluster [DBG] pgmap v2702: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.0 MiB/s wr, 17 op/s 2022-04-22T18:16:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[46934]: audit 2022-04-22T18:16:31.334255+0000 mon.a (mon.0) 3912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:32 smithi115 bash[46934]: cluster 2022-04-22T18:16:32.246865+0000 mon.a (mon.0) 3913 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2022-04-22T18:16:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:32 smithi121 bash[36902]: cluster 2022-04-22T18:16:31.238219+0000 mgr.y (mgr.34107) 1770 : cluster [DBG] pgmap v2702: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.0 MiB/s wr, 17 op/s 2022-04-22T18:16:32.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:32 smithi121 bash[36902]: audit 2022-04-22T18:16:31.334255+0000 mon.a (mon.0) 3912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:32.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:32 smithi121 bash[36902]: cluster 2022-04-22T18:16:32.246865+0000 mon.a (mon.0) 3913 : cluster [DBG] osdmap e1468: 8 total, 8 up, 8 in 2022-04-22T18:16:33.264 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:16:33.305 INFO:tasks.workunit.client.0.smithi115.stdout:write: * 0~1000 2022-04-22T18:16:33.311 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:33.492 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:34.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:33.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:34 smithi121 bash[36902]: cluster 2022-04-22T18:16:33.239441+0000 mgr.y (mgr.34107) 1771 : cluster [DBG] pgmap v2704: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 1.0 MiB/s wr, 91 op/s 2022-04-22T18:16:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:34 smithi121 bash[36902]: cluster 2022-04-22T18:16:33.250449+0000 mon.a (mon.0) 3914 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2022-04-22T18:16:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:34 smithi121 bash[36902]: audit 2022-04-22T18:16:33.251021+0000 mon.a (mon.0) 3915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[45495]: cluster 2022-04-22T18:16:33.239441+0000 mgr.y (mgr.34107) 1771 : cluster [DBG] pgmap v2704: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 1.0 MiB/s wr, 91 op/s 2022-04-22T18:16:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[45495]: cluster 2022-04-22T18:16:33.250449+0000 mon.a (mon.0) 3914 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2022-04-22T18:16:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[45495]: audit 2022-04-22T18:16:33.251021+0000 mon.a (mon.0) 3915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[46934]: cluster 2022-04-22T18:16:33.239441+0000 mgr.y (mgr.34107) 1771 : cluster [DBG] pgmap v2704: 169 pgs: 169 active+clean; 677 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 1.0 MiB/s wr, 91 op/s 2022-04-22T18:16:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[46934]: cluster 2022-04-22T18:16:33.250449+0000 mon.a (mon.0) 3914 : cluster [DBG] osdmap e1469: 8 total, 8 up, 8 in 2022-04-22T18:16:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:34 smithi115 bash[46934]: audit 2022-04-22T18:16:33.251021+0000 mon.a (mon.0) 3915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:35 smithi115 bash[45495]: cluster 2022-04-22T18:16:34.272637+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e1470: 8 total, 8 up, 8 in 2022-04-22T18:16:35.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:35 smithi115 bash[46934]: cluster 2022-04-22T18:16:34.272637+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e1470: 8 total, 8 up, 8 in 2022-04-22T18:16:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:35 smithi121 bash[36902]: cluster 2022-04-22T18:16:34.272637+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e1470: 8 total, 8 up, 8 in 2022-04-22T18:16:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[46934]: cluster 2022-04-22T18:16:35.240592+0000 mgr.y (mgr.34107) 1772 : cluster [DBG] pgmap v2707: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 2.0 MiB/s wr, 128 op/s 2022-04-22T18:16:36.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[46934]: cluster 2022-04-22T18:16:35.280440+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e1471: 8 total, 8 up, 8 in 2022-04-22T18:16:36.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[46934]: audit 2022-04-22T18:16:35.859637+0000 mon.a (mon.0) 3918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:36.631 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[45495]: cluster 2022-04-22T18:16:35.240592+0000 mgr.y (mgr.34107) 1772 : cluster [DBG] pgmap v2707: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 2.0 MiB/s wr, 128 op/s 2022-04-22T18:16:36.631 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[45495]: cluster 2022-04-22T18:16:35.280440+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e1471: 8 total, 8 up, 8 in 2022-04-22T18:16:36.631 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:36 smithi115 bash[45495]: audit 2022-04-22T18:16:35.859637+0000 mon.a (mon.0) 3918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:36 smithi121 bash[36902]: cluster 2022-04-22T18:16:35.240592+0000 mgr.y (mgr.34107) 1772 : cluster [DBG] pgmap v2707: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 2.0 MiB/s wr, 128 op/s 2022-04-22T18:16:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:36 smithi121 bash[36902]: cluster 2022-04-22T18:16:35.280440+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e1471: 8 total, 8 up, 8 in 2022-04-22T18:16:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:36 smithi121 bash[36902]: audit 2022-04-22T18:16:35.859637+0000 mon.a (mon.0) 3918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:36.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsSnapAfterPrepare (15223 ms) 2022-04-22T18:16:36.919 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapsSnapExpandAfterPrepare 2022-04-22T18:16:36.953 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:36.961 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:16:37.038 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:37 smithi121 bash[36902]: cluster 2022-04-22T18:16:36.287232+0000 mon.a (mon.0) 3919 : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2022-04-22T18:16:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:37 smithi121 bash[36902]: audit 2022-04-22T18:16:36.337404+0000 mon.a (mon.0) 3920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:37 smithi115 bash[45495]: cluster 2022-04-22T18:16:36.287232+0000 mon.a (mon.0) 3919 : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2022-04-22T18:16:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:37 smithi115 bash[45495]: audit 2022-04-22T18:16:36.337404+0000 mon.a (mon.0) 3920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:37 smithi115 bash[46934]: cluster 2022-04-22T18:16:36.287232+0000 mon.a (mon.0) 3919 : cluster [DBG] osdmap e1472: 8 total, 8 up, 8 in 2022-04-22T18:16:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:37 smithi115 bash[46934]: audit 2022-04-22T18:16:36.337404+0000 mon.a (mon.0) 3920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:38.328 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:16:38.342 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:16:38.342 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:16:38.342 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:16:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:38 smithi121 bash[36902]: cluster 2022-04-22T18:16:37.241850+0000 mgr.y (mgr.34107) 1773 : cluster [DBG] pgmap v2710: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2022-04-22T18:16:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:38 smithi121 bash[36902]: cluster 2022-04-22T18:16:37.305398+0000 mon.a (mon.0) 3921 : cluster [DBG] osdmap e1473: 8 total, 8 up, 8 in 2022-04-22T18:16:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:38 smithi121 bash[36902]: audit 2022-04-22T18:16:38.248844+0000 mon.a (mon.0) 3922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[45495]: cluster 2022-04-22T18:16:37.241850+0000 mgr.y (mgr.34107) 1773 : cluster [DBG] pgmap v2710: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2022-04-22T18:16:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[45495]: cluster 2022-04-22T18:16:37.305398+0000 mon.a (mon.0) 3921 : cluster [DBG] osdmap e1473: 8 total, 8 up, 8 in 2022-04-22T18:16:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[45495]: audit 2022-04-22T18:16:38.248844+0000 mon.a (mon.0) 3922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[46934]: cluster 2022-04-22T18:16:37.241850+0000 mgr.y (mgr.34107) 1773 : cluster [DBG] pgmap v2710: 169 pgs: 169 active+clean; 681 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2022-04-22T18:16:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[46934]: cluster 2022-04-22T18:16:37.305398+0000 mon.a (mon.0) 3921 : cluster [DBG] osdmap e1473: 8 total, 8 up, 8 in 2022-04-22T18:16:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:38 smithi115 bash[46934]: audit 2022-04-22T18:16:38.248844+0000 mon.a (mon.0) 3922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:39 smithi121 bash[36902]: cluster 2022-04-22T18:16:38.311376+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2022-04-22T18:16:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:39 smithi115 bash[45495]: cluster 2022-04-22T18:16:38.311376+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2022-04-22T18:16:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:39 smithi115 bash[46934]: cluster 2022-04-22T18:16:38.311376+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e1474: 8 total, 8 up, 8 in 2022-04-22T18:16:40.339 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:16:40.346 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:16:40.346 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:16:40.357 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:16:40.367 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:16:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:40 smithi121 bash[36902]: cluster 2022-04-22T18:16:39.242951+0000 mgr.y (mgr.34107) 1774 : cluster [DBG] pgmap v2713: 169 pgs: 169 active+clean; 685 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 161 op/s 2022-04-22T18:16:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:40 smithi121 bash[36902]: cluster 2022-04-22T18:16:39.320552+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2022-04-22T18:16:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:40 smithi115 bash[45495]: cluster 2022-04-22T18:16:39.242951+0000 mgr.y (mgr.34107) 1774 : cluster [DBG] pgmap v2713: 169 pgs: 169 active+clean; 685 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 161 op/s 2022-04-22T18:16:40.901 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:40 smithi115 bash[45495]: cluster 2022-04-22T18:16:39.320552+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2022-04-22T18:16:40.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:40 smithi115 bash[46934]: cluster 2022-04-22T18:16:39.242951+0000 mgr.y (mgr.34107) 1774 : cluster [DBG] pgmap v2713: 169 pgs: 169 active+clean; 685 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 161 op/s 2022-04-22T18:16:40.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:40 smithi115 bash[46934]: cluster 2022-04-22T18:16:39.320552+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e1475: 8 total, 8 up, 8 in 2022-04-22T18:16:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:41 smithi121 bash[36902]: cluster 2022-04-22T18:16:40.322197+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2022-04-22T18:16:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:41 smithi121 bash[36902]: audit 2022-04-22T18:16:40.862784+0000 mon.a (mon.0) 3926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:41 smithi121 bash[36902]: cluster 2022-04-22T18:16:41.245164+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2022-04-22T18:16:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[45495]: cluster 2022-04-22T18:16:40.322197+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2022-04-22T18:16:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[45495]: audit 2022-04-22T18:16:40.862784+0000 mon.a (mon.0) 3926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[45495]: cluster 2022-04-22T18:16:41.245164+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2022-04-22T18:16:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[46934]: cluster 2022-04-22T18:16:40.322197+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e1476: 8 total, 8 up, 8 in 2022-04-22T18:16:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[46934]: audit 2022-04-22T18:16:40.862784+0000 mon.a (mon.0) 3926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[46934]: cluster 2022-04-22T18:16:41.245164+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e1477: 8 total, 8 up, 8 in 2022-04-22T18:16:42.269 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:42.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:41.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:42.301 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:16:42.305 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:42 smithi121 bash[36902]: cluster 2022-04-22T18:16:41.243849+0000 mgr.y (mgr.34107) 1775 : cluster [DBG] pgmap v2717: 169 pgs: 169 active+clean; 701 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 43 MiB/s rd, 9.1 MiB/s wr, 36 op/s 2022-04-22T18:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:42 smithi121 bash[36902]: audit 2022-04-22T18:16:41.340242+0000 mon.a (mon.0) 3928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:42 smithi121 bash[36902]: cluster 2022-04-22T18:16:42.255381+0000 mon.a (mon.0) 3929 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[45495]: cluster 2022-04-22T18:16:41.243849+0000 mgr.y (mgr.34107) 1775 : cluster [DBG] pgmap v2717: 169 pgs: 169 active+clean; 701 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 43 MiB/s rd, 9.1 MiB/s wr, 36 op/s 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[45495]: audit 2022-04-22T18:16:41.340242+0000 mon.a (mon.0) 3928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[45495]: cluster 2022-04-22T18:16:42.255381+0000 mon.a (mon.0) 3929 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[46934]: cluster 2022-04-22T18:16:41.243849+0000 mgr.y (mgr.34107) 1775 : cluster [DBG] pgmap v2717: 169 pgs: 169 active+clean; 701 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 43 MiB/s rd, 9.1 MiB/s wr, 36 op/s 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[46934]: audit 2022-04-22T18:16:41.340242+0000 mon.a (mon.0) 3928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:42 smithi115 bash[46934]: cluster 2022-04-22T18:16:42.255381+0000 mon.a (mon.0) 3929 : cluster [DBG] osdmap e1478: 8 total, 8 up, 8 in 2022-04-22T18:16:43.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:43 smithi115 bash[45495]: audit 2022-04-22T18:16:43.251775+0000 mon.a (mon.0) 3930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:43.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:43 smithi115 bash[46934]: audit 2022-04-22T18:16:43.251775+0000 mon.a (mon.0) 3930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:43 smithi121 bash[36902]: audit 2022-04-22T18:16:43.251775+0000 mon.a (mon.0) 3930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:44.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:43.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:44 smithi121 bash[36902]: cluster 2022-04-22T18:16:43.244984+0000 mgr.y (mgr.34107) 1776 : cluster [DBG] pgmap v2719: 169 pgs: 169 active+clean; 725 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 35 op/s 2022-04-22T18:16:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:44 smithi121 bash[36902]: cluster 2022-04-22T18:16:43.354958+0000 mon.a (mon.0) 3931 : cluster [DBG] osdmap e1479: 8 total, 8 up, 8 in 2022-04-22T18:16:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:44 smithi115 bash[45495]: cluster 2022-04-22T18:16:43.244984+0000 mgr.y (mgr.34107) 1776 : cluster [DBG] pgmap v2719: 169 pgs: 169 active+clean; 725 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 35 op/s 2022-04-22T18:16:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:44 smithi115 bash[45495]: cluster 2022-04-22T18:16:43.354958+0000 mon.a (mon.0) 3931 : cluster [DBG] osdmap e1479: 8 total, 8 up, 8 in 2022-04-22T18:16:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:44 smithi115 bash[46934]: cluster 2022-04-22T18:16:43.244984+0000 mgr.y (mgr.34107) 1776 : cluster [DBG] pgmap v2719: 169 pgs: 169 active+clean; 725 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 13 MiB/s rd, 10 MiB/s wr, 35 op/s 2022-04-22T18:16:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:44 smithi115 bash[46934]: cluster 2022-04-22T18:16:43.354958+0000 mon.a (mon.0) 3931 : cluster [DBG] osdmap e1479: 8 total, 8 up, 8 in 2022-04-22T18:16:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:45 smithi121 bash[36902]: cluster 2022-04-22T18:16:44.359190+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e1480: 8 total, 8 up, 8 in 2022-04-22T18:16:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:45 smithi115 bash[45495]: cluster 2022-04-22T18:16:44.359190+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e1480: 8 total, 8 up, 8 in 2022-04-22T18:16:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:45 smithi115 bash[46934]: cluster 2022-04-22T18:16:44.359190+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e1480: 8 total, 8 up, 8 in 2022-04-22T18:16:46.164 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: after_prepare_snap 2022-04-22T18:16:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:46 smithi121 bash[36902]: cluster 2022-04-22T18:16:45.246218+0000 mgr.y (mgr.34107) 1777 : cluster [DBG] pgmap v2722: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 6.0 MiB/s wr, 25 op/s 2022-04-22T18:16:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:46 smithi121 bash[36902]: cluster 2022-04-22T18:16:45.372745+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2022-04-22T18:16:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:46 smithi121 bash[36902]: audit 2022-04-22T18:16:45.865749+0000 mon.a (mon.0) 3934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:46 smithi121 bash[36902]: cluster 2022-04-22T18:16:46.249087+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e1482: 8 total, 8 up, 8 in 2022-04-22T18:16:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:46 smithi121 bash[36902]: audit 2022-04-22T18:16:46.343468+0000 mon.a (mon.0) 3936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[45495]: cluster 2022-04-22T18:16:45.246218+0000 mgr.y (mgr.34107) 1777 : cluster [DBG] pgmap v2722: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 6.0 MiB/s wr, 25 op/s 2022-04-22T18:16:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[45495]: cluster 2022-04-22T18:16:45.372745+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2022-04-22T18:16:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[45495]: audit 2022-04-22T18:16:45.865749+0000 mon.a (mon.0) 3934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[45495]: cluster 2022-04-22T18:16:46.249087+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e1482: 8 total, 8 up, 8 in 2022-04-22T18:16:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[45495]: audit 2022-04-22T18:16:46.343468+0000 mon.a (mon.0) 3936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[46934]: cluster 2022-04-22T18:16:45.246218+0000 mgr.y (mgr.34107) 1777 : cluster [DBG] pgmap v2722: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 6.0 MiB/s wr, 25 op/s 2022-04-22T18:16:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[46934]: cluster 2022-04-22T18:16:45.372745+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e1481: 8 total, 8 up, 8 in 2022-04-22T18:16:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[46934]: audit 2022-04-22T18:16:45.865749+0000 mon.a (mon.0) 3934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[46934]: cluster 2022-04-22T18:16:46.249087+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e1482: 8 total, 8 up, 8 in 2022-04-22T18:16:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:46 smithi115 bash[46934]: audit 2022-04-22T18:16:46.343468+0000 mon.a (mon.0) 3936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:47.275 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:16:47.293 INFO:tasks.workunit.client.0.smithi115.stdout:write: * 83886080~1000 2022-04-22T18:16:47.301 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:47.480 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:48 smithi121 bash[36902]: cluster 2022-04-22T18:16:47.247404+0000 mgr.y (mgr.34107) 1778 : cluster [DBG] pgmap v2725: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 35 op/s 2022-04-22T18:16:48.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:48 smithi121 bash[36902]: cluster 2022-04-22T18:16:47.262436+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2022-04-22T18:16:48.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:48 smithi121 bash[36902]: audit 2022-04-22T18:16:48.254882+0000 mon.a (mon.0) 3938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:48.637 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[45495]: cluster 2022-04-22T18:16:47.247404+0000 mgr.y (mgr.34107) 1778 : cluster [DBG] pgmap v2725: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 35 op/s 2022-04-22T18:16:48.637 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[45495]: cluster 2022-04-22T18:16:47.262436+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2022-04-22T18:16:48.637 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[45495]: audit 2022-04-22T18:16:48.254882+0000 mon.a (mon.0) 3938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:48.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[46934]: cluster 2022-04-22T18:16:47.247404+0000 mgr.y (mgr.34107) 1778 : cluster [DBG] pgmap v2725: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 35 op/s 2022-04-22T18:16:48.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[46934]: cluster 2022-04-22T18:16:47.262436+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e1483: 8 total, 8 up, 8 in 2022-04-22T18:16:48.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:48 smithi115 bash[46934]: audit 2022-04-22T18:16:48.254882+0000 mon.a (mon.0) 3938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:49 smithi121 bash[36902]: cluster 2022-04-22T18:16:48.378917+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e1484: 8 total, 8 up, 8 in 2022-04-22T18:16:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:49 smithi115 bash[45495]: cluster 2022-04-22T18:16:48.378917+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e1484: 8 total, 8 up, 8 in 2022-04-22T18:16:49.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:49 smithi115 bash[46934]: cluster 2022-04-22T18:16:48.378917+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e1484: 8 total, 8 up, 8 in 2022-04-22T18:16:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:50 smithi121 bash[36902]: cluster 2022-04-22T18:16:49.248486+0000 mgr.y (mgr.34107) 1779 : cluster [DBG] pgmap v2728: 169 pgs: 169 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 210 op/s 2022-04-22T18:16:50.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:50 smithi121 bash[36902]: cluster 2022-04-22T18:16:49.390057+0000 mon.a (mon.0) 3940 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2022-04-22T18:16:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:50 smithi115 bash[46934]: cluster 2022-04-22T18:16:49.248486+0000 mgr.y (mgr.34107) 1779 : cluster [DBG] pgmap v2728: 169 pgs: 169 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 210 op/s 2022-04-22T18:16:50.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:50 smithi115 bash[46934]: cluster 2022-04-22T18:16:49.390057+0000 mon.a (mon.0) 3940 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2022-04-22T18:16:50.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:50 smithi115 bash[45495]: cluster 2022-04-22T18:16:49.248486+0000 mgr.y (mgr.34107) 1779 : cluster [DBG] pgmap v2728: 169 pgs: 169 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 210 op/s 2022-04-22T18:16:50.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:50 smithi115 bash[45495]: cluster 2022-04-22T18:16:49.390057+0000 mon.a (mon.0) 3940 : cluster [DBG] osdmap e1485: 8 total, 8 up, 8 in 2022-04-22T18:16:51.157 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapsSnapExpandAfterPrepare (14238 ms) 2022-04-22T18:16:51.157 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Clone 2022-04-22T18:16:51.194 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:51.202 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:16:51.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Clone (50 ms) 2022-04-22T18:16:51.210 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneParent 2022-04-22T18:16:51.244 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:51.251 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:16:51.256 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneParent (50 ms) 2022-04-22T18:16:51.257 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneUpdateAfterPrepare 2022-04-22T18:16:51.294 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:51.301 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:16:51.307 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneUpdateAfterPrepare (50 ms) 2022-04-22T18:16:51.307 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.TriggerAssertSnapcSeq 2022-04-22T18:16:51.340 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:51.349 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 41943040~10 2022-04-22T18:16:51.359 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:51 smithi121 bash[36902]: cluster 2022-04-22T18:16:50.392649+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e1486: 8 total, 8 up, 8 in 2022-04-22T18:16:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:51 smithi121 bash[36902]: audit 2022-04-22T18:16:50.868695+0000 mon.a (mon.0) 3942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:51 smithi121 bash[36902]: audit 2022-04-22T18:16:51.347113+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[45495]: cluster 2022-04-22T18:16:50.392649+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e1486: 8 total, 8 up, 8 in 2022-04-22T18:16:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[45495]: audit 2022-04-22T18:16:50.868695+0000 mon.a (mon.0) 3942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[45495]: audit 2022-04-22T18:16:51.347113+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[46934]: cluster 2022-04-22T18:16:50.392649+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e1486: 8 total, 8 up, 8 in 2022-04-22T18:16:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[46934]: audit 2022-04-22T18:16:50.868695+0000 mon.a (mon.0) 3942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[46934]: audit 2022-04-22T18:16:51.347113+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:52.200 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:51.789Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: cluster 2022-04-22T18:16:51.249324+0000 mgr.y (mgr.34107) 1780 : cluster [DBG] pgmap v2731: 169 pgs: 169 active+clean; 749 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 200 op/s 2022-04-22T18:16:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: audit 2022-04-22T18:16:51.856534+0000 mon.a (mon.0) 3944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:16:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: audit 2022-04-22T18:16:51.859499+0000 mon.a (mon.0) 3945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:16:52.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: audit 2022-04-22T18:16:51.861242+0000 mon.a (mon.0) 3946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:16:52.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: audit 2022-04-22T18:16:52.212387+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:16:52.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:52 smithi121 bash[36902]: cluster 2022-04-22T18:16:52.251920+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2022-04-22T18:16:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: cluster 2022-04-22T18:16:51.249324+0000 mgr.y (mgr.34107) 1780 : cluster [DBG] pgmap v2731: 169 pgs: 169 active+clean; 749 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 200 op/s 2022-04-22T18:16:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: audit 2022-04-22T18:16:51.856534+0000 mon.a (mon.0) 3944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:16:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: audit 2022-04-22T18:16:51.859499+0000 mon.a (mon.0) 3945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:16:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: audit 2022-04-22T18:16:51.861242+0000 mon.a (mon.0) 3946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:16:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: audit 2022-04-22T18:16:52.212387+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:16:52.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[45495]: cluster 2022-04-22T18:16:52.251920+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: cluster 2022-04-22T18:16:51.249324+0000 mgr.y (mgr.34107) 1780 : cluster [DBG] pgmap v2731: 169 pgs: 169 active+clean; 749 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 5.0 MiB/s wr, 200 op/s 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: audit 2022-04-22T18:16:51.856534+0000 mon.a (mon.0) 3944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: audit 2022-04-22T18:16:51.859499+0000 mon.a (mon.0) 3945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: audit 2022-04-22T18:16:51.861242+0000 mon.a (mon.0) 3946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: audit 2022-04-22T18:16:52.212387+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:16:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:52 smithi115 bash[46934]: cluster 2022-04-22T18:16:52.251920+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e1487: 8 total, 8 up, 8 in 2022-04-22T18:16:53.271 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 41943041~10 2022-04-22T18:16:53.277 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:16:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:16:53.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:16:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:54 smithi121 bash[36902]: cluster 2022-04-22T18:16:53.250539+0000 mgr.y (mgr.34107) 1781 : cluster [DBG] pgmap v2733: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2022-04-22T18:16:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:54 smithi121 bash[36902]: cluster 2022-04-22T18:16:53.258603+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2022-04-22T18:16:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:54 smithi121 bash[36902]: audit 2022-04-22T18:16:53.260228+0000 mon.a (mon.0) 3950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:54.518 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 41943042~10 2022-04-22T18:16:54.533 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:16:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[45495]: cluster 2022-04-22T18:16:53.250539+0000 mgr.y (mgr.34107) 1781 : cluster [DBG] pgmap v2733: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2022-04-22T18:16:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[45495]: cluster 2022-04-22T18:16:53.258603+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2022-04-22T18:16:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[45495]: audit 2022-04-22T18:16:53.260228+0000 mon.a (mon.0) 3950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[46934]: cluster 2022-04-22T18:16:53.250539+0000 mgr.y (mgr.34107) 1781 : cluster [DBG] pgmap v2733: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2022-04-22T18:16:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[46934]: cluster 2022-04-22T18:16:53.258603+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e1488: 8 total, 8 up, 8 in 2022-04-22T18:16:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:54 smithi115 bash[46934]: audit 2022-04-22T18:16:53.260228+0000 mon.a (mon.0) 3950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:55 smithi115 bash[45495]: cluster 2022-04-22T18:16:54.266061+0000 mon.a (mon.0) 3951 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2022-04-22T18:16:55.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:55 smithi115 bash[46934]: cluster 2022-04-22T18:16:54.266061+0000 mon.a (mon.0) 3951 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2022-04-22T18:16:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:55 smithi121 bash[36902]: cluster 2022-04-22T18:16:54.266061+0000 mon.a (mon.0) 3951 : cluster [DBG] osdmap e1489: 8 total, 8 up, 8 in 2022-04-22T18:16:56.267 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:16:56.300 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:16:56.383 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:16:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[45495]: cluster 2022-04-22T18:16:55.251880+0000 mgr.y (mgr.34107) 1782 : cluster [DBG] pgmap v2736: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2022-04-22T18:16:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[45495]: cluster 2022-04-22T18:16:55.276803+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2022-04-22T18:16:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[45495]: audit 2022-04-22T18:16:55.871421+0000 mon.a (mon.0) 3953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:56.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[45495]: cluster 2022-04-22T18:16:56.251379+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2022-04-22T18:16:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[46934]: cluster 2022-04-22T18:16:55.251880+0000 mgr.y (mgr.34107) 1782 : cluster [DBG] pgmap v2736: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2022-04-22T18:16:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[46934]: cluster 2022-04-22T18:16:55.276803+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2022-04-22T18:16:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[46934]: audit 2022-04-22T18:16:55.871421+0000 mon.a (mon.0) 3953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:56 smithi115 bash[46934]: cluster 2022-04-22T18:16:56.251379+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2022-04-22T18:16:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:56 smithi121 bash[36902]: cluster 2022-04-22T18:16:55.251880+0000 mgr.y (mgr.34107) 1782 : cluster [DBG] pgmap v2736: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 34 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2022-04-22T18:16:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:56 smithi121 bash[36902]: cluster 2022-04-22T18:16:55.276803+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e1490: 8 total, 8 up, 8 in 2022-04-22T18:16:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:56 smithi121 bash[36902]: audit 2022-04-22T18:16:55.871421+0000 mon.a (mon.0) 3953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:16:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:56 smithi121 bash[36902]: cluster 2022-04-22T18:16:56.251379+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e1491: 8 total, 8 up, 8 in 2022-04-22T18:16:57.509 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.TriggerAssertSnapcSeq (6202 ms) 2022-04-22T18:16:57.509 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.SnapTrimBeforePrepare 2022-04-22T18:16:57.536 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:16:57.542 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 41943040~10 2022-04-22T18:16:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:57 smithi115 bash[45495]: audit 2022-04-22T18:16:56.350652+0000 mon.a (mon.0) 3955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:57 smithi115 bash[46934]: audit 2022-04-22T18:16:56.350652+0000 mon.a (mon.0) 3955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:57.550 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:16:57.663 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:16:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:57 smithi121 bash[36902]: audit 2022-04-22T18:16:56.350652+0000 mon.a (mon.0) 3955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:16:58.004 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (40m) 8m ago 53m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (53m) 10m ago 53m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (42m) 8m ago 54m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (41m) 10m ago 54m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (49m) 10m ago 59m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (49m) 8m ago 61m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:16:58.128 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (48m) 8m ago 61m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (47m) 10m ago 59m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (48m) 8m ago 59m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (41m) 8m ago 54m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (41m) 10m ago 54m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (46m) 8m ago 58m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (46m) 8m ago 58m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (45m) 8m ago 57m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (45m) 8m ago 57m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (44m) 10m ago 56m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:16:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (43m) 10m ago 56m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:16:58.130 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (43m) 10m ago 55m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:16:58.130 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (42m) 10m ago 55m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:16:58.130 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (40m) 10m ago 54m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:16:58.418 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:16:58.418 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:16:58.418 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:16:58.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:58.418 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:16:58.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:16:58.420 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:16:58.420 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:16:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[45495]: cluster 2022-04-22T18:16:57.253101+0000 mgr.y (mgr.34107) 1783 : cluster [DBG] pgmap v2739: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 5.0 KiB/s wr, 51 op/s 2022-04-22T18:16:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[45495]: cluster 2022-04-22T18:16:57.290220+0000 mon.a (mon.0) 3956 : cluster [DBG] osdmap e1492: 8 total, 8 up, 8 in 2022-04-22T18:16:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[45495]: audit 2022-04-22T18:16:58.261769+0000 mon.a (mon.0) 3957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[46934]: cluster 2022-04-22T18:16:57.253101+0000 mgr.y (mgr.34107) 1783 : cluster [DBG] pgmap v2739: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 5.0 KiB/s wr, 51 op/s 2022-04-22T18:16:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[46934]: cluster 2022-04-22T18:16:57.290220+0000 mon.a (mon.0) 3956 : cluster [DBG] osdmap e1492: 8 total, 8 up, 8 in 2022-04-22T18:16:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:58 smithi115 bash[46934]: audit 2022-04-22T18:16:58.261769+0000 mon.a (mon.0) 3957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:58 smithi121 bash[36902]: cluster 2022-04-22T18:16:57.253101+0000 mgr.y (mgr.34107) 1783 : cluster [DBG] pgmap v2739: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 20 MiB/s rd, 5.0 KiB/s wr, 51 op/s 2022-04-22T18:16:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:58 smithi121 bash[36902]: cluster 2022-04-22T18:16:57.290220+0000 mon.a (mon.0) 3956 : cluster [DBG] osdmap e1492: 8 total, 8 up, 8 in 2022-04-22T18:16:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:58 smithi121 bash[36902]: audit 2022-04-22T18:16:58.261769+0000 mon.a (mon.0) 3957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:16:59.319 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:16:59.338 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:16:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:59 smithi121 bash[36902]: audit 2022-04-22T18:16:57.641024+0000 mgr.y (mgr.34107) 1784 : audit [DBG] from='client.37095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:59 smithi121 bash[36902]: audit 2022-04-22T18:16:57.984474+0000 mgr.y (mgr.34107) 1785 : audit [DBG] from='client.37101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:59 smithi121 bash[36902]: cluster 2022-04-22T18:16:58.294134+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e1493: 8 total, 8 up, 8 in 2022-04-22T18:16:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:16:59 smithi121 bash[36902]: audit 2022-04-22T18:16:58.412169+0000 mon.c (mon.1) 108 : audit [DBG] from='client.? 172.21.15.115:0/754480671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:16:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[45495]: audit 2022-04-22T18:16:57.641024+0000 mgr.y (mgr.34107) 1784 : audit [DBG] from='client.37095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[45495]: audit 2022-04-22T18:16:57.984474+0000 mgr.y (mgr.34107) 1785 : audit [DBG] from='client.37101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[45495]: cluster 2022-04-22T18:16:58.294134+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e1493: 8 total, 8 up, 8 in 2022-04-22T18:16:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[45495]: audit 2022-04-22T18:16:58.412169+0000 mon.c (mon.1) 108 : audit [DBG] from='client.? 172.21.15.115:0/754480671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:16:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[46934]: audit 2022-04-22T18:16:57.641024+0000 mgr.y (mgr.34107) 1784 : audit [DBG] from='client.37095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[46934]: audit 2022-04-22T18:16:57.984474+0000 mgr.y (mgr.34107) 1785 : audit [DBG] from='client.37101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:16:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[46934]: cluster 2022-04-22T18:16:58.294134+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e1493: 8 total, 8 up, 8 in 2022-04-22T18:16:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:16:59 smithi115 bash[46934]: audit 2022-04-22T18:16:58.412169+0000 mon.c (mon.1) 108 : audit [DBG] from='client.? 172.21.15.115:0/754480671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:17:00.520 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:00.527 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:17:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:00 smithi121 bash[36902]: cluster 2022-04-22T18:16:59.254105+0000 mgr.y (mgr.34107) 1786 : cluster [DBG] pgmap v2742: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.2 KiB/s wr, 21 op/s 2022-04-22T18:17:00.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:00 smithi121 bash[36902]: cluster 2022-04-22T18:16:59.304471+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2022-04-22T18:17:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:00 smithi115 bash[45495]: cluster 2022-04-22T18:16:59.254105+0000 mgr.y (mgr.34107) 1786 : cluster [DBG] pgmap v2742: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.2 KiB/s wr, 21 op/s 2022-04-22T18:17:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:00 smithi115 bash[45495]: cluster 2022-04-22T18:16:59.304471+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2022-04-22T18:17:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:00 smithi115 bash[46934]: cluster 2022-04-22T18:16:59.254105+0000 mgr.y (mgr.34107) 1786 : cluster [DBG] pgmap v2742: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.2 KiB/s wr, 21 op/s 2022-04-22T18:17:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:00 smithi115 bash[46934]: cluster 2022-04-22T18:16:59.304471+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e1494: 8 total, 8 up, 8 in 2022-04-22T18:17:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:01 smithi121 bash[36902]: cluster 2022-04-22T18:17:00.313981+0000 mon.a (mon.0) 3960 : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2022-04-22T18:17:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:01 smithi121 bash[36902]: audit 2022-04-22T18:17:00.874368+0000 mon.a (mon.0) 3961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:01.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:01 smithi115 bash[45495]: cluster 2022-04-22T18:17:00.313981+0000 mon.a (mon.0) 3960 : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2022-04-22T18:17:01.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:01 smithi115 bash[45495]: audit 2022-04-22T18:17:00.874368+0000 mon.a (mon.0) 3961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:01.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:01 smithi115 bash[46934]: cluster 2022-04-22T18:17:00.313981+0000 mon.a (mon.0) 3960 : cluster [DBG] osdmap e1495: 8 total, 8 up, 8 in 2022-04-22T18:17:01.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:01 smithi115 bash[46934]: audit 2022-04-22T18:17:00.874368+0000 mon.a (mon.0) 3961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:01.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:02.341 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 41943040~10 2022-04-22T18:17:02.353 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap4 2022-04-22T18:17:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:02 smithi121 bash[36902]: cluster 2022-04-22T18:17:01.255213+0000 mgr.y (mgr.34107) 1787 : cluster [DBG] pgmap v2745: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s rd, 10 KiB/s wr, 115 op/s 2022-04-22T18:17:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:02 smithi121 bash[36902]: cluster 2022-04-22T18:17:01.324829+0000 mon.a (mon.0) 3962 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2022-04-22T18:17:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:02 smithi121 bash[36902]: audit 2022-04-22T18:17:01.353957+0000 mon.a (mon.0) 3963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[45495]: cluster 2022-04-22T18:17:01.255213+0000 mgr.y (mgr.34107) 1787 : cluster [DBG] pgmap v2745: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s rd, 10 KiB/s wr, 115 op/s 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[45495]: cluster 2022-04-22T18:17:01.324829+0000 mon.a (mon.0) 3962 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[45495]: audit 2022-04-22T18:17:01.353957+0000 mon.a (mon.0) 3963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[46934]: cluster 2022-04-22T18:17:01.255213+0000 mgr.y (mgr.34107) 1787 : cluster [DBG] pgmap v2745: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s rd, 10 KiB/s wr, 115 op/s 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[46934]: cluster 2022-04-22T18:17:01.324829+0000 mon.a (mon.0) 3962 : cluster [DBG] osdmap e1496: 8 total, 8 up, 8 in 2022-04-22T18:17:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:02 smithi115 bash[46934]: audit 2022-04-22T18:17:01.353957+0000 mon.a (mon.0) 3963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:03.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:03 smithi115 bash[45495]: cluster 2022-04-22T18:17:02.330015+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e1497: 8 total, 8 up, 8 in 2022-04-22T18:17:03.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:03 smithi115 bash[45495]: audit 2022-04-22T18:17:03.264094+0000 mon.a (mon.0) 3965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:03.679 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:03 smithi115 bash[46934]: cluster 2022-04-22T18:17:02.330015+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e1497: 8 total, 8 up, 8 in 2022-04-22T18:17:03.679 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:03 smithi115 bash[46934]: audit 2022-04-22T18:17:03.264094+0000 mon.a (mon.0) 3965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:03 smithi121 bash[36902]: cluster 2022-04-22T18:17:02.330015+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e1497: 8 total, 8 up, 8 in 2022-04-22T18:17:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:03 smithi121 bash[36902]: audit 2022-04-22T18:17:03.264094+0000 mon.a (mon.0) 3965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:03.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:04.378 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 41943040 2022-04-22T18:17:04.410 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:17:04.491 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:17:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:04 smithi121 bash[36902]: cluster 2022-04-22T18:17:03.256473+0000 mgr.y (mgr.34107) 1788 : cluster [DBG] pgmap v2748: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 107 KiB/s rd, 11 KiB/s wr, 153 op/s 2022-04-22T18:17:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:04 smithi121 bash[36902]: cluster 2022-04-22T18:17:03.349218+0000 mon.a (mon.0) 3966 : cluster [DBG] osdmap e1498: 8 total, 8 up, 8 in 2022-04-22T18:17:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:04 smithi115 bash[45495]: cluster 2022-04-22T18:17:03.256473+0000 mgr.y (mgr.34107) 1788 : cluster [DBG] pgmap v2748: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 107 KiB/s rd, 11 KiB/s wr, 153 op/s 2022-04-22T18:17:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:04 smithi115 bash[45495]: cluster 2022-04-22T18:17:03.349218+0000 mon.a (mon.0) 3966 : cluster [DBG] osdmap e1498: 8 total, 8 up, 8 in 2022-04-22T18:17:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:04 smithi115 bash[46934]: cluster 2022-04-22T18:17:03.256473+0000 mgr.y (mgr.34107) 1788 : cluster [DBG] pgmap v2748: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 107 KiB/s rd, 11 KiB/s wr, 153 op/s 2022-04-22T18:17:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:04 smithi115 bash[46934]: cluster 2022-04-22T18:17:03.349218+0000 mon.a (mon.0) 3966 : cluster [DBG] osdmap e1498: 8 total, 8 up, 8 in 2022-04-22T18:17:05.700 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.SnapTrimBeforePrepare (8185 ms) 2022-04-22T18:17:05.700 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortInUseImage 2022-04-22T18:17:05.728 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:05.736 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:17:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:05 smithi121 bash[36902]: cluster 2022-04-22T18:17:04.363812+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2022-04-22T18:17:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:05 smithi115 bash[45495]: cluster 2022-04-22T18:17:04.363812+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2022-04-22T18:17:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:05 smithi115 bash[46934]: cluster 2022-04-22T18:17:04.363812+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e1499: 8 total, 8 up, 8 in 2022-04-22T18:17:06.040 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortInUseImage (346 ms) 2022-04-22T18:17:06.040 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortWithoutSnapshots 2022-04-22T18:17:06.073 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:06.081 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:17:06.151 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:17:06.206 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:17:06.511 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:17:06.560 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:17:06.616 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:17:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:06 smithi121 bash[36902]: cluster 2022-04-22T18:17:05.257780+0000 mgr.y (mgr.34107) 1789 : cluster [DBG] pgmap v2751: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 57 op/s 2022-04-22T18:17:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:06 smithi121 bash[36902]: cluster 2022-04-22T18:17:05.384891+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e1500: 8 total, 8 up, 8 in 2022-04-22T18:17:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:06 smithi121 bash[36902]: audit 2022-04-22T18:17:05.877466+0000 mon.a (mon.0) 3969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:06 smithi121 bash[36902]: cluster 2022-04-22T18:17:06.254092+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2022-04-22T18:17:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:06 smithi121 bash[36902]: audit 2022-04-22T18:17:06.357451+0000 mon.a (mon.0) 3971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[45495]: cluster 2022-04-22T18:17:05.257780+0000 mgr.y (mgr.34107) 1789 : cluster [DBG] pgmap v2751: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 57 op/s 2022-04-22T18:17:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[45495]: cluster 2022-04-22T18:17:05.384891+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e1500: 8 total, 8 up, 8 in 2022-04-22T18:17:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[45495]: audit 2022-04-22T18:17:05.877466+0000 mon.a (mon.0) 3969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[45495]: cluster 2022-04-22T18:17:06.254092+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2022-04-22T18:17:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[45495]: audit 2022-04-22T18:17:06.357451+0000 mon.a (mon.0) 3971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[46934]: cluster 2022-04-22T18:17:05.257780+0000 mgr.y (mgr.34107) 1789 : cluster [DBG] pgmap v2751: 169 pgs: 169 active+clean; 733 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 57 op/s 2022-04-22T18:17:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[46934]: cluster 2022-04-22T18:17:05.384891+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e1500: 8 total, 8 up, 8 in 2022-04-22T18:17:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[46934]: audit 2022-04-22T18:17:05.877466+0000 mon.a (mon.0) 3969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[46934]: cluster 2022-04-22T18:17:06.254092+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e1501: 8 total, 8 up, 8 in 2022-04-22T18:17:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:06 smithi115 bash[46934]: audit 2022-04-22T18:17:06.357451+0000 mon.a (mon.0) 3971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:06.893 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortWithoutSnapshots (853 ms) 2022-04-22T18:17:06.928 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.AbortWithSnapshots 2022-04-22T18:17:06.928 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:06.933 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:17:06.993 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:17:08.425 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 8388609~8388609 2022-04-22T18:17:08.437 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 0~1000 2022-04-22T18:17:08.437 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 1010~1000 2022-04-22T18:17:08.437 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:17:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:08 smithi121 bash[36902]: cluster 2022-04-22T18:17:07.258940+0000 mgr.y (mgr.34107) 1790 : cluster [DBG] pgmap v2754: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2022-04-22T18:17:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:08 smithi121 bash[36902]: cluster 2022-04-22T18:17:07.395979+0000 mon.a (mon.0) 3972 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2022-04-22T18:17:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:08 smithi121 bash[36902]: audit 2022-04-22T18:17:08.267120+0000 mon.a (mon.0) 3973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[45495]: cluster 2022-04-22T18:17:07.258940+0000 mgr.y (mgr.34107) 1790 : cluster [DBG] pgmap v2754: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2022-04-22T18:17:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[45495]: cluster 2022-04-22T18:17:07.395979+0000 mon.a (mon.0) 3972 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2022-04-22T18:17:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[45495]: audit 2022-04-22T18:17:08.267120+0000 mon.a (mon.0) 3973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[46934]: cluster 2022-04-22T18:17:07.258940+0000 mgr.y (mgr.34107) 1790 : cluster [DBG] pgmap v2754: 169 pgs: 169 active+clean; 741 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2022-04-22T18:17:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[46934]: cluster 2022-04-22T18:17:07.395979+0000 mon.a (mon.0) 3972 : cluster [DBG] osdmap e1502: 8 total, 8 up, 8 in 2022-04-22T18:17:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:08 smithi115 bash[46934]: audit 2022-04-22T18:17:08.267120+0000 mon.a (mon.0) 3973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:09 smithi121 bash[36902]: cluster 2022-04-22T18:17:08.410692+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2022-04-22T18:17:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:09 smithi115 bash[45495]: cluster 2022-04-22T18:17:08.410692+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2022-04-22T18:17:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:09 smithi115 bash[46934]: cluster 2022-04-22T18:17:08.410692+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e1503: 8 total, 8 up, 8 in 2022-04-22T18:17:10.439 INFO:tasks.workunit.client.0.smithi115.stdout:write: X 8388609~1000 2022-04-22T18:17:10.442 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16777228~1000 2022-04-22T18:17:10.442 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 167772160 2022-04-22T18:17:10.458 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 83886079~8388609 2022-04-22T18:17:10.467 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:17:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:10 smithi121 bash[36902]: cluster 2022-04-22T18:17:09.259951+0000 mgr.y (mgr.34107) 1791 : cluster [DBG] pgmap v2757: 169 pgs: 169 active+clean; 753 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 9.1 MiB/s rd, 10 MiB/s wr, 156 op/s 2022-04-22T18:17:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:10 smithi121 bash[36902]: cluster 2022-04-22T18:17:09.418963+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e1504: 8 total, 8 up, 8 in 2022-04-22T18:17:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:10 smithi115 bash[45495]: cluster 2022-04-22T18:17:09.259951+0000 mgr.y (mgr.34107) 1791 : cluster [DBG] pgmap v2757: 169 pgs: 169 active+clean; 753 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 9.1 MiB/s rd, 10 MiB/s wr, 156 op/s 2022-04-22T18:17:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:10 smithi115 bash[45495]: cluster 2022-04-22T18:17:09.418963+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e1504: 8 total, 8 up, 8 in 2022-04-22T18:17:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:10 smithi115 bash[46934]: cluster 2022-04-22T18:17:09.259951+0000 mgr.y (mgr.34107) 1791 : cluster [DBG] pgmap v2757: 169 pgs: 169 active+clean; 753 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 9.1 MiB/s rd, 10 MiB/s wr, 156 op/s 2022-04-22T18:17:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:10 smithi115 bash[46934]: cluster 2022-04-22T18:17:09.418963+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e1504: 8 total, 8 up, 8 in 2022-04-22T18:17:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:11 smithi121 bash[36902]: cluster 2022-04-22T18:17:10.422225+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2022-04-22T18:17:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:11 smithi121 bash[36902]: audit 2022-04-22T18:17:10.880764+0000 mon.a (mon.0) 3977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:11 smithi121 bash[36902]: audit 2022-04-22T18:17:11.360907+0000 mon.a (mon.0) 3978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[46934]: cluster 2022-04-22T18:17:10.422225+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[46934]: audit 2022-04-22T18:17:10.880764+0000 mon.a (mon.0) 3977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[46934]: audit 2022-04-22T18:17:11.360907+0000 mon.a (mon.0) 3978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[45495]: cluster 2022-04-22T18:17:10.422225+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e1505: 8 total, 8 up, 8 in 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[45495]: audit 2022-04-22T18:17:10.880764+0000 mon.a (mon.0) 3977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[45495]: audit 2022-04-22T18:17:11.360907+0000 mon.a (mon.0) 3978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:12.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:11.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:12.468 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:12.489 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83886079~1 2022-04-22T18:17:12.493 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:17:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:12 smithi121 bash[36902]: cluster 2022-04-22T18:17:11.261041+0000 mgr.y (mgr.34107) 1792 : cluster [DBG] pgmap v2760: 169 pgs: 169 active+clean; 789 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 24 MiB/s wr, 265 op/s 2022-04-22T18:17:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:12 smithi121 bash[36902]: cluster 2022-04-22T18:17:11.440460+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2022-04-22T18:17:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:12 smithi115 bash[45495]: cluster 2022-04-22T18:17:11.261041+0000 mgr.y (mgr.34107) 1792 : cluster [DBG] pgmap v2760: 169 pgs: 169 active+clean; 789 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 24 MiB/s wr, 265 op/s 2022-04-22T18:17:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:12 smithi115 bash[45495]: cluster 2022-04-22T18:17:11.440460+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2022-04-22T18:17:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:12 smithi115 bash[46934]: cluster 2022-04-22T18:17:11.261041+0000 mgr.y (mgr.34107) 1792 : cluster [DBG] pgmap v2760: 169 pgs: 169 active+clean; 789 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 19 MiB/s rd, 24 MiB/s wr, 265 op/s 2022-04-22T18:17:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:12 smithi115 bash[46934]: cluster 2022-04-22T18:17:11.440460+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e1506: 8 total, 8 up, 8 in 2022-04-22T18:17:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:13 smithi121 bash[36902]: cluster 2022-04-22T18:17:12.449652+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2022-04-22T18:17:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:13 smithi121 bash[36902]: audit 2022-04-22T18:17:13.270044+0000 mon.a (mon.0) 3981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:13 smithi115 bash[46934]: cluster 2022-04-22T18:17:12.449652+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2022-04-22T18:17:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:13 smithi115 bash[46934]: audit 2022-04-22T18:17:13.270044+0000 mon.a (mon.0) 3981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:13 smithi115 bash[45495]: cluster 2022-04-22T18:17:12.449652+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e1507: 8 total, 8 up, 8 in 2022-04-22T18:17:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:13 smithi115 bash[45495]: audit 2022-04-22T18:17:13.270044+0000 mon.a (mon.0) 3981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:13.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:13.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:14 smithi121 bash[36902]: cluster 2022-04-22T18:17:13.262333+0000 mgr.y (mgr.34107) 1793 : cluster [DBG] pgmap v2763: 169 pgs: 169 active+clean; 805 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 20 MiB/s wr, 138 op/s 2022-04-22T18:17:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:14 smithi121 bash[36902]: cluster 2022-04-22T18:17:13.468145+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2022-04-22T18:17:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:14 smithi121 bash[36902]: audit 2022-04-22T18:17:13.873082+0000 mon.a (mon.0) 3983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:17:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:14 smithi121 bash[36902]: audit 2022-04-22T18:17:13.878445+0000 mon.a (mon.0) 3984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[45495]: cluster 2022-04-22T18:17:13.262333+0000 mgr.y (mgr.34107) 1793 : cluster [DBG] pgmap v2763: 169 pgs: 169 active+clean; 805 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 20 MiB/s wr, 138 op/s 2022-04-22T18:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[45495]: cluster 2022-04-22T18:17:13.468145+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2022-04-22T18:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[45495]: audit 2022-04-22T18:17:13.873082+0000 mon.a (mon.0) 3983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[45495]: audit 2022-04-22T18:17:13.878445+0000 mon.a (mon.0) 3984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:17:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[46934]: cluster 2022-04-22T18:17:13.262333+0000 mgr.y (mgr.34107) 1793 : cluster [DBG] pgmap v2763: 169 pgs: 169 active+clean; 805 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 20 MiB/s wr, 138 op/s 2022-04-22T18:17:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[46934]: cluster 2022-04-22T18:17:13.468145+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e1508: 8 total, 8 up, 8 in 2022-04-22T18:17:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[46934]: audit 2022-04-22T18:17:13.873082+0000 mon.a (mon.0) 3983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:17:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:14 smithi115 bash[46934]: audit 2022-04-22T18:17:13.878445+0000 mon.a (mon.0) 3984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:17:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:15 smithi121 bash[36902]: cluster 2022-04-22T18:17:14.469923+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e1509: 8 total, 8 up, 8 in 2022-04-22T18:17:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:15 smithi115 bash[45495]: cluster 2022-04-22T18:17:14.469923+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e1509: 8 total, 8 up, 8 in 2022-04-22T18:17:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:15 smithi115 bash[46934]: cluster 2022-04-22T18:17:14.469923+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e1509: 8 total, 8 up, 8 in 2022-04-22T18:17:16.243 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:17:16.337 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:17:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:16 smithi121 bash[36902]: cluster 2022-04-22T18:17:15.263576+0000 mgr.y (mgr.34107) 1794 : cluster [DBG] pgmap v2766: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 27 op/s 2022-04-22T18:17:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:16 smithi121 bash[36902]: cluster 2022-04-22T18:17:15.481171+0000 mon.a (mon.0) 3986 : cluster [DBG] osdmap e1510: 8 total, 8 up, 8 in 2022-04-22T18:17:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:16 smithi121 bash[36902]: audit 2022-04-22T18:17:15.883556+0000 mon.a (mon.0) 3987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:16 smithi121 bash[36902]: cluster 2022-04-22T18:17:16.256090+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2022-04-22T18:17:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:16 smithi121 bash[36902]: audit 2022-04-22T18:17:16.364134+0000 mon.a (mon.0) 3989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[46934]: cluster 2022-04-22T18:17:15.263576+0000 mgr.y (mgr.34107) 1794 : cluster [DBG] pgmap v2766: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 27 op/s 2022-04-22T18:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[46934]: cluster 2022-04-22T18:17:15.481171+0000 mon.a (mon.0) 3986 : cluster [DBG] osdmap e1510: 8 total, 8 up, 8 in 2022-04-22T18:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[46934]: audit 2022-04-22T18:17:15.883556+0000 mon.a (mon.0) 3987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[46934]: cluster 2022-04-22T18:17:16.256090+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2022-04-22T18:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[46934]: audit 2022-04-22T18:17:16.364134+0000 mon.a (mon.0) 3989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[45495]: cluster 2022-04-22T18:17:15.263576+0000 mgr.y (mgr.34107) 1794 : cluster [DBG] pgmap v2766: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 27 op/s 2022-04-22T18:17:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[45495]: cluster 2022-04-22T18:17:15.481171+0000 mon.a (mon.0) 3986 : cluster [DBG] osdmap e1510: 8 total, 8 up, 8 in 2022-04-22T18:17:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[45495]: audit 2022-04-22T18:17:15.883556+0000 mon.a (mon.0) 3987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[45495]: cluster 2022-04-22T18:17:16.256090+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e1511: 8 total, 8 up, 8 in 2022-04-22T18:17:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:16 smithi115 bash[45495]: audit 2022-04-22T18:17:16.364134+0000 mon.a (mon.0) 3989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:17.288 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 0~8388609 2022-04-22T18:17:17.309 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 16777218~8388609 2022-04-22T18:17:17.380 INFO:tasks.workunit.client.0.smithi115.stdout:migration_abort 2022-04-22T18:17:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:18 smithi115 bash[45495]: cluster 2022-04-22T18:17:17.264365+0000 mgr.y (mgr.34107) 1795 : cluster [DBG] pgmap v2769: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 21 op/s 2022-04-22T18:17:18.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:18 smithi115 bash[45495]: cluster 2022-04-22T18:17:17.270361+0000 mon.a (mon.0) 3990 : cluster [DBG] osdmap e1512: 8 total, 8 up, 8 in 2022-04-22T18:17:18.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:18 smithi115 bash[46934]: cluster 2022-04-22T18:17:17.264365+0000 mgr.y (mgr.34107) 1795 : cluster [DBG] pgmap v2769: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 21 op/s 2022-04-22T18:17:18.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:18 smithi115 bash[46934]: cluster 2022-04-22T18:17:17.270361+0000 mon.a (mon.0) 3990 : cluster [DBG] osdmap e1512: 8 total, 8 up, 8 in 2022-04-22T18:17:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:18 smithi121 bash[36902]: cluster 2022-04-22T18:17:17.264365+0000 mgr.y (mgr.34107) 1795 : cluster [DBG] pgmap v2769: 169 pgs: 169 active+clean; 821 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 21 op/s 2022-04-22T18:17:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:18 smithi121 bash[36902]: cluster 2022-04-22T18:17:17.270361+0000 mon.a (mon.0) 3990 : cluster [DBG] osdmap e1512: 8 total, 8 up, 8 in 2022-04-22T18:17:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:19 smithi115 bash[46934]: audit 2022-04-22T18:17:18.274482+0000 mon.a (mon.0) 3991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:19 smithi115 bash[46934]: cluster 2022-04-22T18:17:18.283579+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2022-04-22T18:17:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:19 smithi115 bash[45495]: audit 2022-04-22T18:17:18.274482+0000 mon.a (mon.0) 3991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:19 smithi115 bash[45495]: cluster 2022-04-22T18:17:18.283579+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2022-04-22T18:17:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:19 smithi121 bash[36902]: audit 2022-04-22T18:17:18.274482+0000 mon.a (mon.0) 3991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:19 smithi121 bash[36902]: cluster 2022-04-22T18:17:18.283579+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e1513: 8 total, 8 up, 8 in 2022-04-22T18:17:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:20 smithi121 bash[36902]: cluster 2022-04-22T18:17:19.265446+0000 mgr.y (mgr.34107) 1796 : cluster [DBG] pgmap v2772: 169 pgs: 169 active+clean; 833 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 5.0 MiB/s wr, 36 op/s 2022-04-22T18:17:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:20 smithi121 bash[36902]: cluster 2022-04-22T18:17:19.286639+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2022-04-22T18:17:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:20 smithi115 bash[45495]: cluster 2022-04-22T18:17:19.265446+0000 mgr.y (mgr.34107) 1796 : cluster [DBG] pgmap v2772: 169 pgs: 169 active+clean; 833 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 5.0 MiB/s wr, 36 op/s 2022-04-22T18:17:20.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:20 smithi115 bash[45495]: cluster 2022-04-22T18:17:19.286639+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2022-04-22T18:17:20.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:20 smithi115 bash[46934]: cluster 2022-04-22T18:17:19.265446+0000 mgr.y (mgr.34107) 1796 : cluster [DBG] pgmap v2772: 169 pgs: 169 active+clean; 833 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 5.0 MiB/s wr, 36 op/s 2022-04-22T18:17:20.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:20 smithi115 bash[46934]: cluster 2022-04-22T18:17:19.286639+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e1514: 8 total, 8 up, 8 in 2022-04-22T18:17:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:21 smithi121 bash[36902]: cluster 2022-04-22T18:17:20.291780+0000 mon.a (mon.0) 3994 : cluster [DBG] osdmap e1515: 8 total, 8 up, 8 in 2022-04-22T18:17:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:21 smithi121 bash[36902]: audit 2022-04-22T18:17:20.886309+0000 mon.a (mon.0) 3995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:21 smithi115 bash[45495]: cluster 2022-04-22T18:17:20.291780+0000 mon.a (mon.0) 3994 : cluster [DBG] osdmap e1515: 8 total, 8 up, 8 in 2022-04-22T18:17:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:21 smithi115 bash[45495]: audit 2022-04-22T18:17:20.886309+0000 mon.a (mon.0) 3995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:21.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:21 smithi115 bash[46934]: cluster 2022-04-22T18:17:20.291780+0000 mon.a (mon.0) 3994 : cluster [DBG] osdmap e1515: 8 total, 8 up, 8 in 2022-04-22T18:17:21.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:21 smithi115 bash[46934]: audit 2022-04-22T18:17:20.886309+0000 mon.a (mon.0) 3995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:21.939 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.AbortWithSnapshots (15046 ms) 2022-04-22T18:17:21.940 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV1Parent 2022-04-22T18:17:21.974 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:21.981 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:21.987 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV1Parent (48 ms) 2022-04-22T18:17:21.987 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV2Parent 2022-04-22T18:17:22.021 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.031 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.038 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV2Parent (50 ms) 2022-04-22T18:17:22.039 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV1ParentAbort 2022-04-22T18:17:22.072 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.079 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.084 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV1ParentAbort (47 ms) 2022-04-22T18:17:22.084 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV2ParentAbort 2022-04-22T18:17:22.123 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.131 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.137 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV2ParentAbort (53 ms) 2022-04-22T18:17:22.138 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach 2022-04-22T18:17:22.171 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.181 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.188 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach (48 ms) 2022-04-22T18:17:22.188 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixParentReattach 2022-04-22T18:17:22.217 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.224 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.229 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixParentReattach (44 ms) 2022-04-22T18:17:22.230 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV1ParentAbortRelinkNotNeeded 2022-04-22T18:17:22.268 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.274 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.279 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV1ParentAbortRelinkNotNeeded (51 ms) 2022-04-22T18:17:22.279 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach 2022-04-22T18:17:22.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:21.791Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:22.319 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.325 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.329 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach (49 ms) 2022-04-22T18:17:22.329 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixParentReattach 2022-04-22T18:17:22.369 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.377 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.382 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixParentReattach (53 ms) 2022-04-22T18:17:22.382 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.CloneV2ParentAbortRelinkNotNeeded 2022-04-22T18:17:22.417 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.428 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:17:22.433 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.CloneV2ParentAbortRelinkNotNeeded (51 ms) 2022-04-22T18:17:22.433 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.StressNoMigrate 2022-04-22T18:17:22.464 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:22.475 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 58408475~2970230 2022-04-22T18:17:22.516 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 4209776~3107318 2022-04-22T18:17:22.521 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 55528930~5519173 2022-04-22T18:17:22.559 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 17808693~6003341 2022-04-22T18:17:22.563 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 43574688~1184698 2022-04-22T18:17:22.578 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 5102305~2327584 2022-04-22T18:17:22.583 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 64869256~6222888 2022-04-22T18:17:22.640 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 73375963~7526749 2022-04-22T18:17:22.645 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap0 2022-04-22T18:17:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:22 smithi121 bash[36902]: cluster 2022-04-22T18:17:21.266398+0000 mgr.y (mgr.34107) 1797 : cluster [DBG] pgmap v2775: 169 pgs: 169 active+clean; 889 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 28 MiB/s rd, 22 MiB/s wr, 155 op/s 2022-04-22T18:17:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:22 smithi121 bash[36902]: cluster 2022-04-22T18:17:21.315012+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2022-04-22T18:17:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:22 smithi121 bash[36902]: audit 2022-04-22T18:17:21.368041+0000 mon.a (mon.0) 3997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[45495]: cluster 2022-04-22T18:17:21.266398+0000 mgr.y (mgr.34107) 1797 : cluster [DBG] pgmap v2775: 169 pgs: 169 active+clean; 889 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 28 MiB/s rd, 22 MiB/s wr, 155 op/s 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[45495]: cluster 2022-04-22T18:17:21.315012+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[45495]: audit 2022-04-22T18:17:21.368041+0000 mon.a (mon.0) 3997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[46934]: cluster 2022-04-22T18:17:21.266398+0000 mgr.y (mgr.34107) 1797 : cluster [DBG] pgmap v2775: 169 pgs: 169 active+clean; 889 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 28 MiB/s rd, 22 MiB/s wr, 155 op/s 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[46934]: cluster 2022-04-22T18:17:21.315012+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e1516: 8 total, 8 up, 8 in 2022-04-22T18:17:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:22 smithi115 bash[46934]: audit 2022-04-22T18:17:21.368041+0000 mon.a (mon.0) 3997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:23.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:23 smithi115 bash[45495]: audit 2022-04-22T18:17:23.276454+0000 mon.a (mon.0) 3998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:23.679 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:23 smithi115 bash[46934]: audit 2022-04-22T18:17:23.276454+0000 mon.a (mon.0) 3998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:23 smithi121 bash[36902]: audit 2022-04-22T18:17:23.276454+0000 mon.a (mon.0) 3998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:23.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:24.340 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 114374149 2022-04-22T18:17:24.351 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 19779949~3627783 2022-04-22T18:17:24.360 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 60197376~7626454 2022-04-22T18:17:24.360 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 72963732~112249 2022-04-22T18:17:24.361 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 60168108~8118133 2022-04-22T18:17:24.375 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 42361890~5398035 2022-04-22T18:17:24.379 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 83161757~2588696 2022-04-22T18:17:24.379 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 18634860~3019623 2022-04-22T18:17:24.381 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 36155651~1219778 2022-04-22T18:17:24.382 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:17:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:24 smithi121 bash[36902]: cluster 2022-04-22T18:17:23.267819+0000 mgr.y (mgr.34107) 1798 : cluster [DBG] pgmap v2777: 169 pgs: 169 active+clean; 906 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 30 MiB/s rd, 17 MiB/s wr, 136 op/s 2022-04-22T18:17:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:24 smithi121 bash[36902]: cluster 2022-04-22T18:17:23.320328+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2022-04-22T18:17:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:24 smithi115 bash[45495]: cluster 2022-04-22T18:17:23.267819+0000 mgr.y (mgr.34107) 1798 : cluster [DBG] pgmap v2777: 169 pgs: 169 active+clean; 906 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 30 MiB/s rd, 17 MiB/s wr, 136 op/s 2022-04-22T18:17:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:24 smithi115 bash[45495]: cluster 2022-04-22T18:17:23.320328+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2022-04-22T18:17:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:24 smithi115 bash[46934]: cluster 2022-04-22T18:17:23.267819+0000 mgr.y (mgr.34107) 1798 : cluster [DBG] pgmap v2777: 169 pgs: 169 active+clean; 906 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 30 MiB/s rd, 17 MiB/s wr, 136 op/s 2022-04-22T18:17:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:24 smithi115 bash[46934]: cluster 2022-04-22T18:17:23.320328+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e1517: 8 total, 8 up, 8 in 2022-04-22T18:17:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:25 smithi121 bash[36902]: cluster 2022-04-22T18:17:24.324290+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e1518: 8 total, 8 up, 8 in 2022-04-22T18:17:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:25 smithi115 bash[45495]: cluster 2022-04-22T18:17:24.324290+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e1518: 8 total, 8 up, 8 in 2022-04-22T18:17:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:25 smithi115 bash[46934]: cluster 2022-04-22T18:17:24.324290+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e1518: 8 total, 8 up, 8 in 2022-04-22T18:17:26.281 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 189842926 2022-04-22T18:17:26.293 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 135902254~2935615 2022-04-22T18:17:26.299 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 13000127~7770428 2022-04-22T18:17:26.299 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 179663304~626477 2022-04-22T18:17:26.300 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 68259319~834011 2022-04-22T18:17:26.300 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 102428580~7858404 2022-04-22T18:17:26.307 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 104060442~4727857 2022-04-22T18:17:26.354 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 6157148~5294397 2022-04-22T18:17:26.358 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 180693012~5589398 2022-04-22T18:17:26.358 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:17:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:26 smithi121 bash[36902]: cluster 2022-04-22T18:17:25.269057+0000 mgr.y (mgr.34107) 1799 : cluster [DBG] pgmap v2780: 169 pgs: 169 active+clean; 897 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 29 MiB/s rd, 10 MiB/s wr, 68 op/s 2022-04-22T18:17:26.828 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:26 smithi121 bash[36902]: cluster 2022-04-22T18:17:25.348379+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2022-04-22T18:17:26.828 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:26 smithi121 bash[36902]: audit 2022-04-22T18:17:25.889511+0000 mon.a (mon.0) 4002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:26.828 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:26 smithi121 bash[36902]: cluster 2022-04-22T18:17:26.261726+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2022-04-22T18:17:26.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[45495]: cluster 2022-04-22T18:17:25.269057+0000 mgr.y (mgr.34107) 1799 : cluster [DBG] pgmap v2780: 169 pgs: 169 active+clean; 897 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 29 MiB/s rd, 10 MiB/s wr, 68 op/s 2022-04-22T18:17:26.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[45495]: cluster 2022-04-22T18:17:25.348379+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2022-04-22T18:17:26.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[45495]: audit 2022-04-22T18:17:25.889511+0000 mon.a (mon.0) 4002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:26.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[45495]: cluster 2022-04-22T18:17:26.261726+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2022-04-22T18:17:26.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[46934]: cluster 2022-04-22T18:17:25.269057+0000 mgr.y (mgr.34107) 1799 : cluster [DBG] pgmap v2780: 169 pgs: 169 active+clean; 897 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 29 MiB/s rd, 10 MiB/s wr, 68 op/s 2022-04-22T18:17:26.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[46934]: cluster 2022-04-22T18:17:25.348379+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e1519: 8 total, 8 up, 8 in 2022-04-22T18:17:26.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[46934]: audit 2022-04-22T18:17:25.889511+0000 mon.a (mon.0) 4002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:26.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:26 smithi115 bash[46934]: cluster 2022-04-22T18:17:26.261726+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e1520: 8 total, 8 up, 8 in 2022-04-22T18:17:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:27 smithi121 bash[36902]: audit 2022-04-22T18:17:26.370473+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:27 smithi115 bash[45495]: audit 2022-04-22T18:17:26.370473+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:27 smithi115 bash[46934]: audit 2022-04-22T18:17:26.370473+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:28.380 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 125831858 2022-04-22T18:17:28.417 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 114561785~7320749 2022-04-22T18:17:28.430 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 71397970~5528538 2022-04-22T18:17:28.434 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 8089854~4942689 2022-04-22T18:17:28.441 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 91845654~5861735 2022-04-22T18:17:28.443 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 116097823~5287553 2022-04-22T18:17:28.494 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 120131213~5700645 2022-04-22T18:17:28.501 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 84106352~5431856 2022-04-22T18:17:28.504 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 55106992~5662477 2022-04-22T18:17:28.505 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:17:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:28 smithi121 bash[36902]: cluster 2022-04-22T18:17:27.270283+0000 mgr.y (mgr.34107) 1800 : cluster [DBG] pgmap v2783: 169 pgs: 169 active+clean; 911 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 20 MiB/s rd, 8.7 MiB/s wr, 56 op/s 2022-04-22T18:17:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:28 smithi121 bash[36902]: cluster 2022-04-22T18:17:27.363848+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e1521: 8 total, 8 up, 8 in 2022-04-22T18:17:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:28 smithi121 bash[36902]: audit 2022-04-22T18:17:28.279518+0000 mon.a (mon.0) 4006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[45495]: cluster 2022-04-22T18:17:27.270283+0000 mgr.y (mgr.34107) 1800 : cluster [DBG] pgmap v2783: 169 pgs: 169 active+clean; 911 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 20 MiB/s rd, 8.7 MiB/s wr, 56 op/s 2022-04-22T18:17:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[45495]: cluster 2022-04-22T18:17:27.363848+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e1521: 8 total, 8 up, 8 in 2022-04-22T18:17:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[45495]: audit 2022-04-22T18:17:28.279518+0000 mon.a (mon.0) 4006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[46934]: cluster 2022-04-22T18:17:27.270283+0000 mgr.y (mgr.34107) 1800 : cluster [DBG] pgmap v2783: 169 pgs: 169 active+clean; 911 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 20 MiB/s rd, 8.7 MiB/s wr, 56 op/s 2022-04-22T18:17:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[46934]: cluster 2022-04-22T18:17:27.363848+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e1521: 8 total, 8 up, 8 in 2022-04-22T18:17:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:28 smithi115 bash[46934]: audit 2022-04-22T18:17:28.279518+0000 mon.a (mon.0) 4006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:28.813 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:17:29.166 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:17:29.166 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (40m) 9m ago 54m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:17:29.166 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (54m) 10m ago 54m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:17:29.166 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (42m) 9m ago 55m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (42m) 10m ago 55m 17.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (49m) 10m ago 59m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (50m) 9m ago 61m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (49m) 9m ago 61m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (48m) 10m ago 59m 97.1M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (48m) 9m ago 60m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (41m) 9m ago 54m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (41m) 10m ago 54m 11.8M - 1dbe0e931976 f4b044642a96 2022-04-22T18:17:29.167 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (47m) 9m ago 59m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (46m) 9m ago 58m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (46m) 9m ago 58m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (45m) 9m ago 57m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (45m) 10m ago 57m 486M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (44m) 10m ago 56m 607M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (43m) 10m ago 56m 382M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (42m) 10m ago 55m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:17:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (41m) 10m ago 54m 57.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:17:29.578 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:17:29.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:17:29.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:17:29.581 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:17:29.581 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:17:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:29 smithi121 bash[36902]: cluster 2022-04-22T18:17:28.369636+0000 mon.a (mon.0) 4007 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2022-04-22T18:17:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:29 smithi115 bash[45495]: cluster 2022-04-22T18:17:28.369636+0000 mon.a (mon.0) 4007 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2022-04-22T18:17:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:29 smithi115 bash[46934]: cluster 2022-04-22T18:17:28.369636+0000 mon.a (mon.0) 4007 : cluster [DBG] osdmap e1522: 8 total, 8 up, 8 in 2022-04-22T18:17:30.411 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 106922132 2022-04-22T18:17:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:30 smithi121 bash[36902]: audit 2022-04-22T18:17:28.792208+0000 mgr.y (mgr.34107) 1801 : audit [DBG] from='client.37113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:30 smithi121 bash[36902]: audit 2022-04-22T18:17:29.140688+0000 mgr.y (mgr.34107) 1802 : audit [DBG] from='client.47047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:30 smithi121 bash[36902]: cluster 2022-04-22T18:17:29.271350+0000 mgr.y (mgr.34107) 1803 : cluster [DBG] pgmap v2786: 169 pgs: 169 active+clean; 952 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 11 MiB/s rd, 11 MiB/s wr, 35 op/s 2022-04-22T18:17:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:30 smithi121 bash[36902]: cluster 2022-04-22T18:17:29.390582+0000 mon.a (mon.0) 4008 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2022-04-22T18:17:30.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:30 smithi121 bash[36902]: audit 2022-04-22T18:17:29.572555+0000 mon.c (mon.1) 109 : audit [DBG] from='client.? 172.21.15.115:0/1465919406' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:17:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[46934]: audit 2022-04-22T18:17:28.792208+0000 mgr.y (mgr.34107) 1801 : audit [DBG] from='client.37113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[46934]: audit 2022-04-22T18:17:29.140688+0000 mgr.y (mgr.34107) 1802 : audit [DBG] from='client.47047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[46934]: cluster 2022-04-22T18:17:29.271350+0000 mgr.y (mgr.34107) 1803 : cluster [DBG] pgmap v2786: 169 pgs: 169 active+clean; 952 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 11 MiB/s rd, 11 MiB/s wr, 35 op/s 2022-04-22T18:17:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[46934]: cluster 2022-04-22T18:17:29.390582+0000 mon.a (mon.0) 4008 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2022-04-22T18:17:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[46934]: audit 2022-04-22T18:17:29.572555+0000 mon.c (mon.1) 109 : audit [DBG] from='client.? 172.21.15.115:0/1465919406' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:17:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[45495]: audit 2022-04-22T18:17:28.792208+0000 mgr.y (mgr.34107) 1801 : audit [DBG] from='client.37113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[45495]: audit 2022-04-22T18:17:29.140688+0000 mgr.y (mgr.34107) 1802 : audit [DBG] from='client.47047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:17:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[45495]: cluster 2022-04-22T18:17:29.271350+0000 mgr.y (mgr.34107) 1803 : cluster [DBG] pgmap v2786: 169 pgs: 169 active+clean; 952 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 11 MiB/s rd, 11 MiB/s wr, 35 op/s 2022-04-22T18:17:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[45495]: cluster 2022-04-22T18:17:29.390582+0000 mon.a (mon.0) 4008 : cluster [DBG] osdmap e1523: 8 total, 8 up, 8 in 2022-04-22T18:17:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:30 smithi115 bash[45495]: audit 2022-04-22T18:17:29.572555+0000 mon.c (mon.1) 109 : audit [DBG] from='client.? 172.21.15.115:0/1465919406' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:17:31.333 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.StressNoMigrate (8900 ms) 2022-04-22T18:17:31.333 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Stress 2022-04-22T18:17:31.365 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:31.372 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 56324525~4749711 2022-04-22T18:17:31.411 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 68197824~5174129 2022-04-22T18:17:31.417 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 60800174~1001415 2022-04-22T18:17:31.428 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 73699677~1055993 2022-04-22T18:17:31.432 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 48779388~6612162 2022-04-22T18:17:31.485 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 22321519~3903853 2022-04-22T18:17:31.491 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 26209607~3957621 2022-04-22T18:17:31.533 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 25677526~1485871 2022-04-22T18:17:31.537 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap0 2022-04-22T18:17:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:31 smithi121 bash[36902]: cluster 2022-04-22T18:17:30.392244+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2022-04-22T18:17:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:31 smithi121 bash[36902]: audit 2022-04-22T18:17:30.892641+0000 mon.a (mon.0) 4010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:31 smithi121 bash[36902]: cluster 2022-04-22T18:17:31.265230+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2022-04-22T18:17:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:31 smithi121 bash[36902]: audit 2022-04-22T18:17:31.373042+0000 mon.a (mon.0) 4012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[45495]: cluster 2022-04-22T18:17:30.392244+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2022-04-22T18:17:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[45495]: audit 2022-04-22T18:17:30.892641+0000 mon.a (mon.0) 4010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[45495]: cluster 2022-04-22T18:17:31.265230+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2022-04-22T18:17:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[45495]: audit 2022-04-22T18:17:31.373042+0000 mon.a (mon.0) 4012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[46934]: cluster 2022-04-22T18:17:30.392244+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e1524: 8 total, 8 up, 8 in 2022-04-22T18:17:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[46934]: audit 2022-04-22T18:17:30.892641+0000 mon.a (mon.0) 4010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[46934]: cluster 2022-04-22T18:17:31.265230+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e1525: 8 total, 8 up, 8 in 2022-04-22T18:17:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[46934]: audit 2022-04-22T18:17:31.373042+0000 mon.a (mon.0) 4012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:32.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:31.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:32 smithi121 bash[36902]: cluster 2022-04-22T18:17:31.271956+0000 mgr.y (mgr.34107) 1804 : cluster [DBG] pgmap v2790: 169 pgs: 169 active+clean; 990 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 769 KiB/s rd, 15 MiB/s wr, 18 op/s 2022-04-22T18:17:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:32 smithi115 bash[45495]: cluster 2022-04-22T18:17:31.271956+0000 mgr.y (mgr.34107) 1804 : cluster [DBG] pgmap v2790: 169 pgs: 169 active+clean; 990 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 769 KiB/s rd, 15 MiB/s wr, 18 op/s 2022-04-22T18:17:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:32 smithi115 bash[46934]: cluster 2022-04-22T18:17:31.271956+0000 mgr.y (mgr.34107) 1804 : cluster [DBG] pgmap v2790: 169 pgs: 169 active+clean; 990 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 769 KiB/s rd, 15 MiB/s wr, 18 op/s 2022-04-22T18:17:33.446 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 75582878~7347606 2022-04-22T18:17:33.459 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 64760767~6059963 2022-04-22T18:17:33.459 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 12727612~5786051 2022-04-22T18:17:33.463 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 22810919~3083759 2022-04-22T18:17:33.464 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 16286043~1612873 2022-04-22T18:17:33.516 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 16290435~2013061 2022-04-22T18:17:33.523 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 7215037~6031908 2022-04-22T18:17:33.528 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 3636177~4262930 2022-04-22T18:17:33.543 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:33 smithi121 bash[36902]: cluster 2022-04-22T18:17:32.423080+0000 mon.a (mon.0) 4013 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2022-04-22T18:17:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:33 smithi121 bash[36902]: audit 2022-04-22T18:17:33.282749+0000 mon.a (mon.0) 4014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:33 smithi115 bash[45495]: cluster 2022-04-22T18:17:32.423080+0000 mon.a (mon.0) 4013 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2022-04-22T18:17:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:33 smithi115 bash[45495]: audit 2022-04-22T18:17:33.282749+0000 mon.a (mon.0) 4014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:33 smithi115 bash[46934]: cluster 2022-04-22T18:17:32.423080+0000 mon.a (mon.0) 4013 : cluster [DBG] osdmap e1526: 8 total, 8 up, 8 in 2022-04-22T18:17:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:33 smithi115 bash[46934]: audit 2022-04-22T18:17:33.282749+0000 mon.a (mon.0) 4014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:33.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:33.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:34 smithi121 bash[36902]: cluster 2022-04-22T18:17:33.273073+0000 mgr.y (mgr.34107) 1805 : cluster [DBG] pgmap v2792: 169 pgs: 169 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 33 MiB/s rd, 18 MiB/s wr, 51 op/s 2022-04-22T18:17:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:34 smithi121 bash[36902]: cluster 2022-04-22T18:17:33.434726+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2022-04-22T18:17:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:34 smithi115 bash[45495]: cluster 2022-04-22T18:17:33.273073+0000 mgr.y (mgr.34107) 1805 : cluster [DBG] pgmap v2792: 169 pgs: 169 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 33 MiB/s rd, 18 MiB/s wr, 51 op/s 2022-04-22T18:17:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:34 smithi115 bash[45495]: cluster 2022-04-22T18:17:33.434726+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2022-04-22T18:17:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:34 smithi115 bash[46934]: cluster 2022-04-22T18:17:33.273073+0000 mgr.y (mgr.34107) 1805 : cluster [DBG] pgmap v2792: 169 pgs: 169 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 33 MiB/s rd, 18 MiB/s wr, 51 op/s 2022-04-22T18:17:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:34 smithi115 bash[46934]: cluster 2022-04-22T18:17:33.434726+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e1527: 8 total, 8 up, 8 in 2022-04-22T18:17:35.463 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 29803416~5351891 2022-04-22T18:17:35.478 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 48350924~6080477 2022-04-22T18:17:35.478 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 54635679~2839797 2022-04-22T18:17:35.482 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 31909475~4855624 2022-04-22T18:17:35.507 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 42367678~1459727 2022-04-22T18:17:35.508 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 28111422~7787434 2022-04-22T18:17:35.513 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 42114276~936340 2022-04-22T18:17:35.514 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 9410355~1904597 2022-04-22T18:17:35.515 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:17:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:35 smithi121 bash[36902]: cluster 2022-04-22T18:17:34.445100+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2022-04-22T18:17:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:35 smithi115 bash[45495]: cluster 2022-04-22T18:17:34.445100+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2022-04-22T18:17:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:35 smithi115 bash[46934]: cluster 2022-04-22T18:17:34.445100+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e1528: 8 total, 8 up, 8 in 2022-04-22T18:17:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:36 smithi121 bash[36902]: cluster 2022-04-22T18:17:35.274263+0000 mgr.y (mgr.34107) 1806 : cluster [DBG] pgmap v2795: 169 pgs: 169 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 49 MiB/s rd, 20 MiB/s wr, 62 op/s 2022-04-22T18:17:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:36 smithi121 bash[36902]: cluster 2022-04-22T18:17:35.448030+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e1529: 8 total, 8 up, 8 in 2022-04-22T18:17:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:36 smithi121 bash[36902]: audit 2022-04-22T18:17:35.895288+0000 mon.a (mon.0) 4018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:36 smithi121 bash[36902]: audit 2022-04-22T18:17:36.377632+0000 mon.a (mon.0) 4019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[45495]: cluster 2022-04-22T18:17:35.274263+0000 mgr.y (mgr.34107) 1806 : cluster [DBG] pgmap v2795: 169 pgs: 169 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 49 MiB/s rd, 20 MiB/s wr, 62 op/s 2022-04-22T18:17:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[45495]: cluster 2022-04-22T18:17:35.448030+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e1529: 8 total, 8 up, 8 in 2022-04-22T18:17:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[45495]: audit 2022-04-22T18:17:35.895288+0000 mon.a (mon.0) 4018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[45495]: audit 2022-04-22T18:17:36.377632+0000 mon.a (mon.0) 4019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[46934]: cluster 2022-04-22T18:17:35.274263+0000 mgr.y (mgr.34107) 1806 : cluster [DBG] pgmap v2795: 169 pgs: 169 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 49 MiB/s rd, 20 MiB/s wr, 62 op/s 2022-04-22T18:17:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[46934]: cluster 2022-04-22T18:17:35.448030+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e1529: 8 total, 8 up, 8 in 2022-04-22T18:17:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[46934]: audit 2022-04-22T18:17:35.895288+0000 mon.a (mon.0) 4018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:36 smithi115 bash[46934]: audit 2022-04-22T18:17:36.377632+0000 mon.a (mon.0) 4019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:37.484 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 31973622~7945108 2022-04-22T18:17:37.501 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 44583300~2181596 2022-04-22T18:17:37.502 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 45737509~4452894 2022-04-22T18:17:37.532 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 65186211~2149087 2022-04-22T18:17:37.532 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 41735427~3307688 2022-04-22T18:17:37.535 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 7570618~5292384 2022-04-22T18:17:37.536 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 76401807~3428564 2022-04-22T18:17:37.539 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 24843372~4533901 2022-04-22T18:17:37.539 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:17:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:37 smithi121 bash[36902]: cluster 2022-04-22T18:17:36.458882+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2022-04-22T18:17:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:37 smithi115 bash[45495]: cluster 2022-04-22T18:17:36.458882+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2022-04-22T18:17:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:37 smithi115 bash[46934]: cluster 2022-04-22T18:17:36.458882+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e1530: 8 total, 8 up, 8 in 2022-04-22T18:17:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:38 smithi121 bash[36902]: cluster 2022-04-22T18:17:37.275557+0000 mgr.y (mgr.34107) 1807 : cluster [DBG] pgmap v2798: 169 pgs: 169 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 15 MiB/s wr, 34 op/s 2022-04-22T18:17:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:38 smithi121 bash[36902]: cluster 2022-04-22T18:17:37.470362+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2022-04-22T18:17:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:38 smithi121 bash[36902]: audit 2022-04-22T18:17:38.285813+0000 mon.a (mon.0) 4022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[45495]: cluster 2022-04-22T18:17:37.275557+0000 mgr.y (mgr.34107) 1807 : cluster [DBG] pgmap v2798: 169 pgs: 169 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 15 MiB/s wr, 34 op/s 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[45495]: cluster 2022-04-22T18:17:37.470362+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[45495]: audit 2022-04-22T18:17:38.285813+0000 mon.a (mon.0) 4022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[46934]: cluster 2022-04-22T18:17:37.275557+0000 mgr.y (mgr.34107) 1807 : cluster [DBG] pgmap v2798: 169 pgs: 169 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s rd, 15 MiB/s wr, 34 op/s 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[46934]: cluster 2022-04-22T18:17:37.470362+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e1531: 8 total, 8 up, 8 in 2022-04-22T18:17:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:38 smithi115 bash[46934]: audit 2022-04-22T18:17:38.285813+0000 mon.a (mon.0) 4022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:39.519 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:17:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:39 smithi121 bash[36902]: cluster 2022-04-22T18:17:38.489962+0000 mon.a (mon.0) 4023 : cluster [DBG] osdmap e1532: 8 total, 8 up, 8 in 2022-04-22T18:17:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:39 smithi115 bash[45495]: cluster 2022-04-22T18:17:38.489962+0000 mon.a (mon.0) 4023 : cluster [DBG] osdmap e1532: 8 total, 8 up, 8 in 2022-04-22T18:17:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:39 smithi115 bash[46934]: cluster 2022-04-22T18:17:38.489962+0000 mon.a (mon.0) 4023 : cluster [DBG] osdmap e1532: 8 total, 8 up, 8 in 2022-04-22T18:17:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:40 smithi121 bash[36902]: cluster 2022-04-22T18:17:39.276707+0000 mgr.y (mgr.34107) 1808 : cluster [DBG] pgmap v2801: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:40 smithi121 bash[36902]: cluster 2022-04-22T18:17:39.501317+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e1533: 8 total, 8 up, 8 in 2022-04-22T18:17:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:40 smithi115 bash[45495]: cluster 2022-04-22T18:17:39.276707+0000 mgr.y (mgr.34107) 1808 : cluster [DBG] pgmap v2801: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:40 smithi115 bash[45495]: cluster 2022-04-22T18:17:39.501317+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e1533: 8 total, 8 up, 8 in 2022-04-22T18:17:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:40 smithi115 bash[46934]: cluster 2022-04-22T18:17:39.276707+0000 mgr.y (mgr.34107) 1808 : cluster [DBG] pgmap v2801: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:40 smithi115 bash[46934]: cluster 2022-04-22T18:17:39.501317+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e1533: 8 total, 8 up, 8 in 2022-04-22T18:17:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[45495]: cluster 2022-04-22T18:17:40.503716+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e1534: 8 total, 8 up, 8 in 2022-04-22T18:17:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[45495]: audit 2022-04-22T18:17:40.898267+0000 mon.a (mon.0) 4026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[45495]: audit 2022-04-22T18:17:41.381370+0000 mon.a (mon.0) 4027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[46934]: cluster 2022-04-22T18:17:40.503716+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e1534: 8 total, 8 up, 8 in 2022-04-22T18:17:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[46934]: audit 2022-04-22T18:17:40.898267+0000 mon.a (mon.0) 4026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[46934]: audit 2022-04-22T18:17:41.381370+0000 mon.a (mon.0) 4027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:41 smithi121 bash[36902]: cluster 2022-04-22T18:17:40.503716+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e1534: 8 total, 8 up, 8 in 2022-04-22T18:17:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:41 smithi121 bash[36902]: audit 2022-04-22T18:17:40.898267+0000 mon.a (mon.0) 4026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:41 smithi121 bash[36902]: audit 2022-04-22T18:17:41.381370+0000 mon.a (mon.0) 4027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:42.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:41.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:42 smithi115 bash[45495]: cluster 2022-04-22T18:17:41.278205+0000 mgr.y (mgr.34107) 1809 : cluster [DBG] pgmap v2804: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:42 smithi115 bash[45495]: cluster 2022-04-22T18:17:41.514676+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2022-04-22T18:17:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:42 smithi115 bash[46934]: cluster 2022-04-22T18:17:41.278205+0000 mgr.y (mgr.34107) 1809 : cluster [DBG] pgmap v2804: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:42 smithi115 bash[46934]: cluster 2022-04-22T18:17:41.514676+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2022-04-22T18:17:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:42 smithi121 bash[36902]: cluster 2022-04-22T18:17:41.278205+0000 mgr.y (mgr.34107) 1809 : cluster [DBG] pgmap v2804: 169 pgs: 169 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 22 op/s 2022-04-22T18:17:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:42 smithi121 bash[36902]: cluster 2022-04-22T18:17:41.514676+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e1535: 8 total, 8 up, 8 in 2022-04-22T18:17:43.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:43.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:43 smithi115 bash[45495]: cluster 2022-04-22T18:17:42.531076+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2022-04-22T18:17:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:43 smithi115 bash[45495]: audit 2022-04-22T18:17:43.289193+0000 mon.a (mon.0) 4030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:43 smithi115 bash[46934]: cluster 2022-04-22T18:17:42.531076+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2022-04-22T18:17:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:43 smithi115 bash[46934]: audit 2022-04-22T18:17:43.289193+0000 mon.a (mon.0) 4030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:43 smithi121 bash[36902]: cluster 2022-04-22T18:17:42.531076+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e1536: 8 total, 8 up, 8 in 2022-04-22T18:17:44.065 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:43 smithi121 bash[36902]: audit 2022-04-22T18:17:43.289193+0000 mon.a (mon.0) 4030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:44.507 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:17:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:44 smithi115 bash[46934]: cluster 2022-04-22T18:17:43.279438+0000 mgr.y (mgr.34107) 1810 : cluster [DBG] pgmap v2807: 169 pgs: 169 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 25 op/s 2022-04-22T18:17:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:44 smithi115 bash[46934]: cluster 2022-04-22T18:17:43.540750+0000 mon.a (mon.0) 4031 : cluster [DBG] osdmap e1537: 8 total, 8 up, 8 in 2022-04-22T18:17:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:44 smithi115 bash[45495]: cluster 2022-04-22T18:17:43.279438+0000 mgr.y (mgr.34107) 1810 : cluster [DBG] pgmap v2807: 169 pgs: 169 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 25 op/s 2022-04-22T18:17:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:44 smithi115 bash[45495]: cluster 2022-04-22T18:17:43.540750+0000 mon.a (mon.0) 4031 : cluster [DBG] osdmap e1537: 8 total, 8 up, 8 in 2022-04-22T18:17:44.795 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:17:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:44 smithi121 bash[36902]: cluster 2022-04-22T18:17:43.279438+0000 mgr.y (mgr.34107) 1810 : cluster [DBG] pgmap v2807: 169 pgs: 169 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 25 op/s 2022-04-22T18:17:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:44 smithi121 bash[36902]: cluster 2022-04-22T18:17:43.540750+0000 mon.a (mon.0) 4031 : cluster [DBG] osdmap e1537: 8 total, 8 up, 8 in 2022-04-22T18:17:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:46 smithi121 bash[36902]: cluster 2022-04-22T18:17:45.280816+0000 mgr.y (mgr.34107) 1811 : cluster [DBG] pgmap v2809: 169 pgs: 169 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 27 MiB/s rd, 7.8 MiB/s wr, 158 op/s 2022-04-22T18:17:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:46 smithi121 bash[36902]: cluster 2022-04-22T18:17:45.551317+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e1538: 8 total, 8 up, 8 in 2022-04-22T18:17:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:46 smithi121 bash[36902]: audit 2022-04-22T18:17:45.901445+0000 mon.a (mon.0) 4033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:46 smithi121 bash[36902]: audit 2022-04-22T18:17:46.384898+0000 mon.a (mon.0) 4034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[45495]: cluster 2022-04-22T18:17:45.280816+0000 mgr.y (mgr.34107) 1811 : cluster [DBG] pgmap v2809: 169 pgs: 169 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 27 MiB/s rd, 7.8 MiB/s wr, 158 op/s 2022-04-22T18:17:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[45495]: cluster 2022-04-22T18:17:45.551317+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e1538: 8 total, 8 up, 8 in 2022-04-22T18:17:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[45495]: audit 2022-04-22T18:17:45.901445+0000 mon.a (mon.0) 4033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[45495]: audit 2022-04-22T18:17:46.384898+0000 mon.a (mon.0) 4034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[46934]: cluster 2022-04-22T18:17:45.280816+0000 mgr.y (mgr.34107) 1811 : cluster [DBG] pgmap v2809: 169 pgs: 169 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 27 MiB/s rd, 7.8 MiB/s wr, 158 op/s 2022-04-22T18:17:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[46934]: cluster 2022-04-22T18:17:45.551317+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e1538: 8 total, 8 up, 8 in 2022-04-22T18:17:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[46934]: audit 2022-04-22T18:17:45.901445+0000 mon.a (mon.0) 4033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:46 smithi115 bash[46934]: audit 2022-04-22T18:17:46.384898+0000 mon.a (mon.0) 4034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:47 smithi121 bash[36902]: cluster 2022-04-22T18:17:46.560654+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2022-04-22T18:17:48.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:47 smithi115 bash[45495]: cluster 2022-04-22T18:17:46.560654+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2022-04-22T18:17:48.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:47 smithi115 bash[46934]: cluster 2022-04-22T18:17:46.560654+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e1539: 8 total, 8 up, 8 in 2022-04-22T18:17:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:48 smithi121 bash[36902]: cluster 2022-04-22T18:17:47.282001+0000 mgr.y (mgr.34107) 1812 : cluster [DBG] pgmap v2812: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 52 MiB/s rd, 8.5 MiB/s wr, 196 op/s 2022-04-22T18:17:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:48 smithi121 bash[36902]: cluster 2022-04-22T18:17:47.572746+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2022-04-22T18:17:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:48 smithi121 bash[36902]: audit 2022-04-22T18:17:48.292093+0000 mon.a (mon.0) 4037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[45495]: cluster 2022-04-22T18:17:47.282001+0000 mgr.y (mgr.34107) 1812 : cluster [DBG] pgmap v2812: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 52 MiB/s rd, 8.5 MiB/s wr, 196 op/s 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[45495]: cluster 2022-04-22T18:17:47.572746+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[45495]: audit 2022-04-22T18:17:48.292093+0000 mon.a (mon.0) 4037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[46934]: cluster 2022-04-22T18:17:47.282001+0000 mgr.y (mgr.34107) 1812 : cluster [DBG] pgmap v2812: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 52 MiB/s rd, 8.5 MiB/s wr, 196 op/s 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[46934]: cluster 2022-04-22T18:17:47.572746+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e1540: 8 total, 8 up, 8 in 2022-04-22T18:17:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:48 smithi115 bash[46934]: audit 2022-04-22T18:17:48.292093+0000 mon.a (mon.0) 4037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:49.354 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Stress (18021 ms) 2022-04-22T18:17:49.355 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.Stress2 2022-04-22T18:17:49.395 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:17:49.407 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 32574626~2900235 2022-04-22T18:17:49.435 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 54691610~872389 2022-04-22T18:17:49.439 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 67380079~7833440 2022-04-22T18:17:49.497 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 42324212~7305561 2022-04-22T18:17:49.501 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 51153198~442452 2022-04-22T18:17:49.508 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 73782516~1402919 2022-04-22T18:17:49.512 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 33376542~378050 2022-04-22T18:17:49.518 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 44961350~466640 2022-04-22T18:17:49.522 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap0 2022-04-22T18:17:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:49 smithi121 bash[36902]: cluster 2022-04-22T18:17:48.579445+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e1541: 8 total, 8 up, 8 in 2022-04-22T18:17:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:49 smithi115 bash[46934]: cluster 2022-04-22T18:17:48.579445+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e1541: 8 total, 8 up, 8 in 2022-04-22T18:17:50.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:49 smithi115 bash[45495]: cluster 2022-04-22T18:17:48.579445+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e1541: 8 total, 8 up, 8 in 2022-04-22T18:17:50.608 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 5167398~4261109 2022-04-22T18:17:50.617 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 7568797~2352351 2022-04-22T18:17:50.639 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 74753599~4959785 2022-04-22T18:17:50.643 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 8388350~6750807 2022-04-22T18:17:50.643 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 36450533~160462 2022-04-22T18:17:50.644 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 74254138~8065898 2022-04-22T18:17:50.671 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 10583244~5449278 2022-04-22T18:17:50.675 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 54801120~1656902 2022-04-22T18:17:50.675 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:17:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:50 smithi121 bash[36902]: cluster 2022-04-22T18:17:49.282864+0000 mgr.y (mgr.34107) 1813 : cluster [DBG] pgmap v2815: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 47 MiB/s rd, 7.2 MiB/s wr, 76 op/s 2022-04-22T18:17:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:50 smithi121 bash[36902]: cluster 2022-04-22T18:17:49.590304+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e1542: 8 total, 8 up, 8 in 2022-04-22T18:17:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:50 smithi115 bash[45495]: cluster 2022-04-22T18:17:49.282864+0000 mgr.y (mgr.34107) 1813 : cluster [DBG] pgmap v2815: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 47 MiB/s rd, 7.2 MiB/s wr, 76 op/s 2022-04-22T18:17:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:50 smithi115 bash[45495]: cluster 2022-04-22T18:17:49.590304+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e1542: 8 total, 8 up, 8 in 2022-04-22T18:17:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:50 smithi115 bash[46934]: cluster 2022-04-22T18:17:49.282864+0000 mgr.y (mgr.34107) 1813 : cluster [DBG] pgmap v2815: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 47 MiB/s rd, 7.2 MiB/s wr, 76 op/s 2022-04-22T18:17:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:50 smithi115 bash[46934]: cluster 2022-04-22T18:17:49.590304+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e1542: 8 total, 8 up, 8 in 2022-04-22T18:17:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:51 smithi121 bash[36902]: cluster 2022-04-22T18:17:50.591547+0000 mon.a (mon.0) 4040 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2022-04-22T18:17:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:51 smithi121 bash[36902]: audit 2022-04-22T18:17:50.904611+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:51 smithi121 bash[36902]: audit 2022-04-22T18:17:51.388593+0000 mon.a (mon.0) 4042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:51.795Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[45495]: cluster 2022-04-22T18:17:50.591547+0000 mon.a (mon.0) 4040 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2022-04-22T18:17:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[45495]: audit 2022-04-22T18:17:50.904611+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[45495]: audit 2022-04-22T18:17:51.388593+0000 mon.a (mon.0) 4042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[46934]: cluster 2022-04-22T18:17:50.591547+0000 mon.a (mon.0) 4040 : cluster [DBG] osdmap e1543: 8 total, 8 up, 8 in 2022-04-22T18:17:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[46934]: audit 2022-04-22T18:17:50.904611+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:51 smithi115 bash[46934]: audit 2022-04-22T18:17:51.388593+0000 mon.a (mon.0) 4042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:52.633 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 106680160 2022-04-22T18:17:52.641 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 46379649~3914304 2022-04-22T18:17:52.647 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 38773269~4933817 2022-04-22T18:17:52.647 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 71590855~3780693 2022-04-22T18:17:52.651 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 42560393~5698396 2022-04-22T18:17:52.685 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 81766330~6133418 2022-04-22T18:17:52.689 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 74812208~2532377 2022-04-22T18:17:52.690 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 11361966~1406902 2022-04-22T18:17:52.691 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 82726837~5930518 2022-04-22T18:17:52.722 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:17:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:52 smithi121 bash[36902]: cluster 2022-04-22T18:17:51.284006+0000 mgr.y (mgr.34107) 1814 : cluster [DBG] pgmap v2818: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s rd, 4.8 MiB/s wr, 55 op/s 2022-04-22T18:17:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:52 smithi121 bash[36902]: cluster 2022-04-22T18:17:51.612078+0000 mon.a (mon.0) 4043 : cluster [DBG] osdmap e1544: 8 total, 8 up, 8 in 2022-04-22T18:17:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:52 smithi121 bash[36902]: audit 2022-04-22T18:17:52.220367+0000 mon.a (mon.0) 4044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:17:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:52 smithi121 bash[36902]: audit 2022-04-22T18:17:52.223452+0000 mon.a (mon.0) 4045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:17:53.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:52 smithi121 bash[36902]: audit 2022-04-22T18:17:52.225265+0000 mon.a (mon.0) 4046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:17:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[45495]: cluster 2022-04-22T18:17:51.284006+0000 mgr.y (mgr.34107) 1814 : cluster [DBG] pgmap v2818: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s rd, 4.8 MiB/s wr, 55 op/s 2022-04-22T18:17:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[45495]: cluster 2022-04-22T18:17:51.612078+0000 mon.a (mon.0) 4043 : cluster [DBG] osdmap e1544: 8 total, 8 up, 8 in 2022-04-22T18:17:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[45495]: audit 2022-04-22T18:17:52.220367+0000 mon.a (mon.0) 4044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:17:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[45495]: audit 2022-04-22T18:17:52.223452+0000 mon.a (mon.0) 4045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:17:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[45495]: audit 2022-04-22T18:17:52.225265+0000 mon.a (mon.0) 4046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:17:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[46934]: cluster 2022-04-22T18:17:51.284006+0000 mgr.y (mgr.34107) 1814 : cluster [DBG] pgmap v2818: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s rd, 4.8 MiB/s wr, 55 op/s 2022-04-22T18:17:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[46934]: cluster 2022-04-22T18:17:51.612078+0000 mon.a (mon.0) 4043 : cluster [DBG] osdmap e1544: 8 total, 8 up, 8 in 2022-04-22T18:17:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[46934]: audit 2022-04-22T18:17:52.220367+0000 mon.a (mon.0) 4044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:17:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[46934]: audit 2022-04-22T18:17:52.223452+0000 mon.a (mon.0) 4045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:17:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:52 smithi115 bash[46934]: audit 2022-04-22T18:17:52.225265+0000 mon.a (mon.0) 4046 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:17:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:53 smithi121 bash[36902]: cluster 2022-04-22T18:17:52.616102+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2022-04-22T18:17:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:53 smithi121 bash[36902]: audit 2022-04-22T18:17:53.295581+0000 mon.a (mon.0) 4048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:17:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:17:53.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:17:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:53 smithi115 bash[45495]: cluster 2022-04-22T18:17:52.616102+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2022-04-22T18:17:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:53 smithi115 bash[45495]: audit 2022-04-22T18:17:53.295581+0000 mon.a (mon.0) 4048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:53 smithi115 bash[46934]: cluster 2022-04-22T18:17:52.616102+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e1545: 8 total, 8 up, 8 in 2022-04-22T18:17:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:53 smithi115 bash[46934]: audit 2022-04-22T18:17:53.295581+0000 mon.a (mon.0) 4048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:54.658 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 142386272 2022-04-22T18:17:54.666 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 15145186~3015381 2022-04-22T18:17:54.672 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 95670595~2501438 2022-04-22T18:17:54.673 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 69947784~7231975 2022-04-22T18:17:54.680 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 72061702~1932510 2022-04-22T18:17:54.715 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 114841956~4373365 2022-04-22T18:17:54.717 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 59092422~373510 2022-04-22T18:17:54.718 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 13556679~1217668 2022-04-22T18:17:54.720 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 76341246~2390023 2022-04-22T18:17:54.720 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:17:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:54 smithi121 bash[36902]: cluster 2022-04-22T18:17:53.285322+0000 mgr.y (mgr.34107) 1815 : cluster [DBG] pgmap v2821: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 66 MiB/s rd, 12 MiB/s wr, 66 op/s 2022-04-22T18:17:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:54 smithi121 bash[36902]: cluster 2022-04-22T18:17:53.635272+0000 mon.a (mon.0) 4049 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2022-04-22T18:17:55.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:54 smithi115 bash[45495]: cluster 2022-04-22T18:17:53.285322+0000 mgr.y (mgr.34107) 1815 : cluster [DBG] pgmap v2821: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 66 MiB/s rd, 12 MiB/s wr, 66 op/s 2022-04-22T18:17:55.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:54 smithi115 bash[45495]: cluster 2022-04-22T18:17:53.635272+0000 mon.a (mon.0) 4049 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2022-04-22T18:17:55.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:54 smithi115 bash[46934]: cluster 2022-04-22T18:17:53.285322+0000 mgr.y (mgr.34107) 1815 : cluster [DBG] pgmap v2821: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 66 MiB/s rd, 12 MiB/s wr, 66 op/s 2022-04-22T18:17:55.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:54 smithi115 bash[46934]: cluster 2022-04-22T18:17:53.635272+0000 mon.a (mon.0) 4049 : cluster [DBG] osdmap e1546: 8 total, 8 up, 8 in 2022-04-22T18:17:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:55 smithi121 bash[36902]: cluster 2022-04-22T18:17:54.643797+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2022-04-22T18:17:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:55 smithi121 bash[36902]: cluster 2022-04-22T18:17:55.286570+0000 mgr.y (mgr.34107) 1816 : cluster [DBG] pgmap v2824: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 17 MiB/s wr, 46 op/s 2022-04-22T18:17:56.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:55 smithi115 bash[45495]: cluster 2022-04-22T18:17:54.643797+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2022-04-22T18:17:56.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:55 smithi115 bash[45495]: cluster 2022-04-22T18:17:55.286570+0000 mgr.y (mgr.34107) 1816 : cluster [DBG] pgmap v2824: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 17 MiB/s wr, 46 op/s 2022-04-22T18:17:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:55 smithi115 bash[46934]: cluster 2022-04-22T18:17:54.643797+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e1547: 8 total, 8 up, 8 in 2022-04-22T18:17:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:55 smithi115 bash[46934]: cluster 2022-04-22T18:17:55.286570+0000 mgr.y (mgr.34107) 1816 : cluster [DBG] pgmap v2824: 169 pgs: 169 active+clean; 1.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 17 MiB/s wr, 46 op/s 2022-04-22T18:17:56.295 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:17:56.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:56 smithi121 bash[36902]: cluster 2022-04-22T18:17:55.660560+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e1548: 8 total, 8 up, 8 in 2022-04-22T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:56 smithi121 bash[36902]: audit 2022-04-22T18:17:55.907572+0000 mon.a (mon.0) 4052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:56 smithi121 bash[36902]: cluster 2022-04-22T18:17:56.273795+0000 mon.a (mon.0) 4053 : cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2022-04-22T18:17:56.985 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:56 smithi121 bash[36902]: audit 2022-04-22T18:17:56.392092+0000 mon.a (mon.0) 4054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[45495]: cluster 2022-04-22T18:17:55.660560+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e1548: 8 total, 8 up, 8 in 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[45495]: audit 2022-04-22T18:17:55.907572+0000 mon.a (mon.0) 4052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[45495]: cluster 2022-04-22T18:17:56.273795+0000 mon.a (mon.0) 4053 : cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[45495]: audit 2022-04-22T18:17:56.392092+0000 mon.a (mon.0) 4054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[46934]: cluster 2022-04-22T18:17:55.660560+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e1548: 8 total, 8 up, 8 in 2022-04-22T18:17:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[46934]: audit 2022-04-22T18:17:55.907572+0000 mon.a (mon.0) 4052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:17:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[46934]: cluster 2022-04-22T18:17:56.273795+0000 mon.a (mon.0) 4053 : cluster [DBG] osdmap e1549: 8 total, 8 up, 8 in 2022-04-22T18:17:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:56 smithi115 bash[46934]: audit 2022-04-22T18:17:56.392092+0000 mon.a (mon.0) 4054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:17:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:57 smithi121 bash[36902]: audit 2022-04-22T18:17:56.658538+0000 mon.a (mon.0) 4055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.066 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:57 smithi121 bash[36902]: audit 2022-04-22T18:17:56.979242+0000 mon.a (mon.0) 4056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.066 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:57 smithi121 bash[36902]: cluster 2022-04-22T18:17:57.286539+0000 mon.a (mon.0) 4057 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2022-04-22T18:17:58.066 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:57 smithi121 bash[36902]: cluster 2022-04-22T18:17:57.287552+0000 mgr.y (mgr.34107) 1817 : cluster [DBG] pgmap v2828: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 137 KiB/s rd, 14 MiB/s wr, 20 op/s 2022-04-22T18:17:58.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[45495]: audit 2022-04-22T18:17:56.658538+0000 mon.a (mon.0) 4055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[45495]: audit 2022-04-22T18:17:56.979242+0000 mon.a (mon.0) 4056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[45495]: cluster 2022-04-22T18:17:57.286539+0000 mon.a (mon.0) 4057 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2022-04-22T18:17:58.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[45495]: cluster 2022-04-22T18:17:57.287552+0000 mgr.y (mgr.34107) 1817 : cluster [DBG] pgmap v2828: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 137 KiB/s rd, 14 MiB/s wr, 20 op/s 2022-04-22T18:17:58.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[46934]: audit 2022-04-22T18:17:56.658538+0000 mon.a (mon.0) 4055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[46934]: audit 2022-04-22T18:17:56.979242+0000 mon.a (mon.0) 4056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:17:58.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[46934]: cluster 2022-04-22T18:17:57.286539+0000 mon.a (mon.0) 4057 : cluster [DBG] osdmap e1550: 8 total, 8 up, 8 in 2022-04-22T18:17:58.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:57 smithi115 bash[46934]: cluster 2022-04-22T18:17:57.287552+0000 mgr.y (mgr.34107) 1817 : cluster [DBG] pgmap v2828: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 137 KiB/s rd, 14 MiB/s wr, 20 op/s 2022-04-22T18:17:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:59 smithi121 bash[36902]: cluster 2022-04-22T18:17:58.294048+0000 mon.a (mon.0) 4058 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2022-04-22T18:17:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:17:59 smithi121 bash[36902]: audit 2022-04-22T18:17:58.298107+0000 mon.a (mon.0) 4059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:59 smithi115 bash[45495]: cluster 2022-04-22T18:17:58.294048+0000 mon.a (mon.0) 4058 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2022-04-22T18:17:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:17:59 smithi115 bash[45495]: audit 2022-04-22T18:17:58.298107+0000 mon.a (mon.0) 4059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:59 smithi115 bash[46934]: cluster 2022-04-22T18:17:58.294048+0000 mon.a (mon.0) 4058 : cluster [DBG] osdmap e1551: 8 total, 8 up, 8 in 2022-04-22T18:17:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:17:59 smithi115 bash[46934]: audit 2022-04-22T18:17:58.298107+0000 mon.a (mon.0) 4059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:17:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (41m) 9m ago 54m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (54m) 3s ago 54m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (43m) 9m ago 55m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (42m) 3s ago 55m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (50m) 3s ago 60m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:18:00.299 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (50m) 9m ago 62m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (49m) 9m ago 62m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (49m) 3s ago 60m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (49m) 9m ago 60m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (42m) 9m ago 55m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (42m) 3s ago 55m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (47m) 9m ago 59m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (47m) 9m ago 59m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (46m) 9m ago 58m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (46m) 9m ago 58m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (45m) 3s ago 57m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:18:00.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (44m) 3s ago 57m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:18:00.301 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (44m) 3s ago 56m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:18:00.301 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (43m) 3s ago 56m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:18:00.301 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (41m) 3s ago 55m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:18:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:00.730 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:18:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:18:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:00 smithi121 bash[36902]: cluster 2022-04-22T18:17:59.288722+0000 mgr.y (mgr.34107) 1818 : cluster [DBG] pgmap v2830: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 5.7 KiB/s rd, 8.3 MiB/s wr, 12 op/s 2022-04-22T18:18:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:00 smithi121 bash[36902]: cluster 2022-04-22T18:17:59.299407+0000 mon.a (mon.0) 4060 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2022-04-22T18:18:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:00 smithi115 bash[45495]: cluster 2022-04-22T18:17:59.288722+0000 mgr.y (mgr.34107) 1818 : cluster [DBG] pgmap v2830: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 5.7 KiB/s rd, 8.3 MiB/s wr, 12 op/s 2022-04-22T18:18:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:00 smithi115 bash[45495]: cluster 2022-04-22T18:17:59.299407+0000 mon.a (mon.0) 4060 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2022-04-22T18:18:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:00 smithi115 bash[46934]: cluster 2022-04-22T18:17:59.288722+0000 mgr.y (mgr.34107) 1818 : cluster [DBG] pgmap v2830: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 5.7 KiB/s rd, 8.3 MiB/s wr, 12 op/s 2022-04-22T18:18:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:00 smithi115 bash[46934]: cluster 2022-04-22T18:17:59.299407+0000 mon.a (mon.0) 4060 : cluster [DBG] osdmap e1552: 8 total, 8 up, 8 in 2022-04-22T18:18:01.269 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 29124197~6170716 2022-04-22T18:18:01.328 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 28094565~5893080 2022-04-22T18:18:01.332 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 121125683~1263055 2022-04-22T18:18:01.349 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 111738272~2380222 2022-04-22T18:18:01.369 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 87091884~4431881 2022-04-22T18:18:01.402 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 77720703~7956694 2022-04-22T18:18:01.415 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 70719094~6758775 2022-04-22T18:18:01.498 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 2706042~2817908 2022-04-22T18:18:01.514 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user0 2022-04-22T18:18:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:01 smithi121 bash[36902]: audit 2022-04-22T18:17:59.923978+0000 mgr.y (mgr.34107) 1819 : audit [DBG] from='client.47059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:01 smithi121 bash[36902]: audit 2022-04-22T18:18:00.266892+0000 mgr.y (mgr.34107) 1820 : audit [DBG] from='client.47065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:01 smithi121 bash[36902]: cluster 2022-04-22T18:18:00.310844+0000 mon.a (mon.0) 4061 : cluster [DBG] osdmap e1553: 8 total, 8 up, 8 in 2022-04-22T18:18:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:01 smithi121 bash[36902]: audit 2022-04-22T18:18:00.724224+0000 mon.c (mon.1) 110 : audit [DBG] from='client.? 172.21.15.115:0/927155331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:01 smithi121 bash[36902]: audit 2022-04-22T18:18:00.910145+0000 mon.a (mon.0) 4062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[46934]: audit 2022-04-22T18:17:59.923978+0000 mgr.y (mgr.34107) 1819 : audit [DBG] from='client.47059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[46934]: audit 2022-04-22T18:18:00.266892+0000 mgr.y (mgr.34107) 1820 : audit [DBG] from='client.47065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[46934]: cluster 2022-04-22T18:18:00.310844+0000 mon.a (mon.0) 4061 : cluster [DBG] osdmap e1553: 8 total, 8 up, 8 in 2022-04-22T18:18:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[46934]: audit 2022-04-22T18:18:00.724224+0000 mon.c (mon.1) 110 : audit [DBG] from='client.? 172.21.15.115:0/927155331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[46934]: audit 2022-04-22T18:18:00.910145+0000 mon.a (mon.0) 4062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[45495]: audit 2022-04-22T18:17:59.923978+0000 mgr.y (mgr.34107) 1819 : audit [DBG] from='client.47059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[45495]: audit 2022-04-22T18:18:00.266892+0000 mgr.y (mgr.34107) 1820 : audit [DBG] from='client.47065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[45495]: cluster 2022-04-22T18:18:00.310844+0000 mon.a (mon.0) 4061 : cluster [DBG] osdmap e1553: 8 total, 8 up, 8 in 2022-04-22T18:18:01.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[45495]: audit 2022-04-22T18:18:00.724224+0000 mon.c (mon.1) 110 : audit [DBG] from='client.? 172.21.15.115:0/927155331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:01.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[45495]: audit 2022-04-22T18:18:00.910145+0000 mon.a (mon.0) 4062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:02.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:01.795Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:02 smithi121 bash[36902]: cluster 2022-04-22T18:18:01.289390+0000 mgr.y (mgr.34107) 1821 : cluster [DBG] pgmap v2833: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2022-04-22T18:18:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:02 smithi121 bash[36902]: audit 2022-04-22T18:18:01.395121+0000 mon.a (mon.0) 4063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:02 smithi115 bash[45495]: cluster 2022-04-22T18:18:01.289390+0000 mgr.y (mgr.34107) 1821 : cluster [DBG] pgmap v2833: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2022-04-22T18:18:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:02 smithi115 bash[45495]: audit 2022-04-22T18:18:01.395121+0000 mon.a (mon.0) 4063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:02 smithi115 bash[46934]: cluster 2022-04-22T18:18:01.289390+0000 mgr.y (mgr.34107) 1821 : cluster [DBG] pgmap v2833: 169 pgs: 169 active+clean; 1.2 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2022-04-22T18:18:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:02 smithi115 bash[46934]: audit 2022-04-22T18:18:01.395121+0000 mon.a (mon.0) 4063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:03.358 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 21187457~5993751 2022-04-22T18:18:03.379 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 858037~6749949 2022-04-22T18:18:03.379 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 46658322~5842055 2022-04-22T18:18:03.387 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 101351086~3127065 2022-04-22T18:18:03.388 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 122885566~1786488 2022-04-22T18:18:03.389 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 56708800~2114773 2022-04-22T18:18:03.421 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 35863889~2107612 2022-04-22T18:18:03.444 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 74275169~1958620 2022-04-22T18:18:03.445 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user1 2022-04-22T18:18:03.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:03 smithi115 bash[46934]: cluster 2022-04-22T18:18:02.330395+0000 mon.a (mon.0) 4064 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2022-04-22T18:18:03.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:03 smithi115 bash[46934]: audit 2022-04-22T18:18:03.300375+0000 mon.a (mon.0) 4065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:03.683 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:03 smithi115 bash[45495]: cluster 2022-04-22T18:18:02.330395+0000 mon.a (mon.0) 4064 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2022-04-22T18:18:03.683 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:03 smithi115 bash[45495]: audit 2022-04-22T18:18:03.300375+0000 mon.a (mon.0) 4065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:03 smithi121 bash[36902]: cluster 2022-04-22T18:18:02.330395+0000 mon.a (mon.0) 4064 : cluster [DBG] osdmap e1554: 8 total, 8 up, 8 in 2022-04-22T18:18:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:03 smithi121 bash[36902]: audit 2022-04-22T18:18:03.300375+0000 mon.a (mon.0) 4065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:03.676Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:04 smithi121 bash[36902]: cluster 2022-04-22T18:18:03.290560+0000 mgr.y (mgr.34107) 1822 : cluster [DBG] pgmap v2835: 169 pgs: 169 active+clean; 1.3 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 34 MiB/s rd, 5.1 MiB/s wr, 41 op/s 2022-04-22T18:18:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:04 smithi121 bash[36902]: cluster 2022-04-22T18:18:03.344105+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2022-04-22T18:18:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:04 smithi115 bash[45495]: cluster 2022-04-22T18:18:03.290560+0000 mgr.y (mgr.34107) 1822 : cluster [DBG] pgmap v2835: 169 pgs: 169 active+clean; 1.3 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 34 MiB/s rd, 5.1 MiB/s wr, 41 op/s 2022-04-22T18:18:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:04 smithi115 bash[45495]: cluster 2022-04-22T18:18:03.344105+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2022-04-22T18:18:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:04 smithi115 bash[46934]: cluster 2022-04-22T18:18:03.290560+0000 mgr.y (mgr.34107) 1822 : cluster [DBG] pgmap v2835: 169 pgs: 169 active+clean; 1.3 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 34 MiB/s rd, 5.1 MiB/s wr, 41 op/s 2022-04-22T18:18:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:04 smithi115 bash[46934]: cluster 2022-04-22T18:18:03.344105+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e1555: 8 total, 8 up, 8 in 2022-04-22T18:18:05.374 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 261633250 2022-04-22T18:18:05.398 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 231374324~6888913 2022-04-22T18:18:05.422 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 222259691~4186688 2022-04-22T18:18:05.422 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 45328427~7270790 2022-04-22T18:18:05.435 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 232435929~94567 2022-04-22T18:18:05.450 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 126863666~3341956 2022-04-22T18:18:05.452 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 93563480~1502617 2022-04-22T18:18:05.452 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 207236558~6735570 2022-04-22T18:18:05.463 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 169696267~7597408 2022-04-22T18:18:05.468 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user2 2022-04-22T18:18:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:05 smithi121 bash[36902]: cluster 2022-04-22T18:18:04.351225+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e1556: 8 total, 8 up, 8 in 2022-04-22T18:18:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:05 smithi115 bash[45495]: cluster 2022-04-22T18:18:04.351225+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e1556: 8 total, 8 up, 8 in 2022-04-22T18:18:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:05 smithi115 bash[46934]: cluster 2022-04-22T18:18:04.351225+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e1556: 8 total, 8 up, 8 in 2022-04-22T18:18:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:06 smithi121 bash[36902]: cluster 2022-04-22T18:18:05.291803+0000 mgr.y (mgr.34107) 1823 : cluster [DBG] pgmap v2838: 169 pgs: 169 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 MiB/s rd, 10 MiB/s wr, 108 op/s 2022-04-22T18:18:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:06 smithi121 bash[36902]: cluster 2022-04-22T18:18:05.354002+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2022-04-22T18:18:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:06 smithi121 bash[36902]: audit 2022-04-22T18:18:05.913810+0000 mon.a (mon.0) 4069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[45495]: cluster 2022-04-22T18:18:05.291803+0000 mgr.y (mgr.34107) 1823 : cluster [DBG] pgmap v2838: 169 pgs: 169 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 MiB/s rd, 10 MiB/s wr, 108 op/s 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[45495]: cluster 2022-04-22T18:18:05.354002+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[45495]: audit 2022-04-22T18:18:05.913810+0000 mon.a (mon.0) 4069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[46934]: cluster 2022-04-22T18:18:05.291803+0000 mgr.y (mgr.34107) 1823 : cluster [DBG] pgmap v2838: 169 pgs: 169 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 MiB/s rd, 10 MiB/s wr, 108 op/s 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[46934]: cluster 2022-04-22T18:18:05.354002+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e1557: 8 total, 8 up, 8 in 2022-04-22T18:18:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:06 smithi115 bash[46934]: audit 2022-04-22T18:18:05.913810+0000 mon.a (mon.0) 4069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:07.402 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 189664787~5050855 2022-04-22T18:18:07.435 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 108756913~1772479 2022-04-22T18:18:07.436 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 257775453~1229498 2022-04-22T18:18:07.438 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 140080336~4992586 2022-04-22T18:18:07.439 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 9345043~8224825 2022-04-22T18:18:07.461 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 254268620~7364630 2022-04-22T18:18:07.465 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 242922316~7115973 2022-04-22T18:18:07.470 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 131024795~3962380 2022-04-22T18:18:07.473 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user3 2022-04-22T18:18:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:07 smithi121 bash[36902]: cluster 2022-04-22T18:18:06.373465+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e1558: 8 total, 8 up, 8 in 2022-04-22T18:18:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:07 smithi121 bash[36902]: audit 2022-04-22T18:18:06.398295+0000 mon.a (mon.0) 4071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:07 smithi115 bash[45495]: cluster 2022-04-22T18:18:06.373465+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e1558: 8 total, 8 up, 8 in 2022-04-22T18:18:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:07 smithi115 bash[45495]: audit 2022-04-22T18:18:06.398295+0000 mon.a (mon.0) 4071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:07 smithi115 bash[46934]: cluster 2022-04-22T18:18:06.373465+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e1558: 8 total, 8 up, 8 in 2022-04-22T18:18:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:07 smithi115 bash[46934]: audit 2022-04-22T18:18:06.398295+0000 mon.a (mon.0) 4071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:08 smithi121 bash[36902]: cluster 2022-04-22T18:18:07.292925+0000 mgr.y (mgr.34107) 1824 : cluster [DBG] pgmap v2841: 169 pgs: 169 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 15 MiB/s rd, 20 MiB/s wr, 116 op/s 2022-04-22T18:18:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:08 smithi121 bash[36902]: cluster 2022-04-22T18:18:07.384938+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e1559: 8 total, 8 up, 8 in 2022-04-22T18:18:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:08 smithi121 bash[36902]: audit 2022-04-22T18:18:08.303580+0000 mon.a (mon.0) 4073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[45495]: cluster 2022-04-22T18:18:07.292925+0000 mgr.y (mgr.34107) 1824 : cluster [DBG] pgmap v2841: 169 pgs: 169 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 15 MiB/s rd, 20 MiB/s wr, 116 op/s 2022-04-22T18:18:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[45495]: cluster 2022-04-22T18:18:07.384938+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e1559: 8 total, 8 up, 8 in 2022-04-22T18:18:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[45495]: audit 2022-04-22T18:18:08.303580+0000 mon.a (mon.0) 4073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[46934]: cluster 2022-04-22T18:18:07.292925+0000 mgr.y (mgr.34107) 1824 : cluster [DBG] pgmap v2841: 169 pgs: 169 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 15 MiB/s rd, 20 MiB/s wr, 116 op/s 2022-04-22T18:18:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[46934]: cluster 2022-04-22T18:18:07.384938+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e1559: 8 total, 8 up, 8 in 2022-04-22T18:18:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:08 smithi115 bash[46934]: audit 2022-04-22T18:18:08.303580+0000 mon.a (mon.0) 4073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:09.414 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 247995027 2022-04-22T18:18:09.465 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 24799502~6199875 2022-04-22T18:18:09.471 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 26349470~3099937 2022-04-22T18:18:09.501 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 49599005~6199875 2022-04-22T18:18:09.505 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 51148973~3099937 2022-04-22T18:18:09.535 INFO:tasks.workunit.client.0.smithi115.stdout:write: 3 74398508~6199875 2022-04-22T18:18:09.539 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 75948476~3099937 2022-04-22T18:18:09.580 INFO:tasks.workunit.client.0.smithi115.stdout:write: 4 99198010~6199875 2022-04-22T18:18:09.584 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 100747978~3099937 2022-04-22T18:18:09.614 INFO:tasks.workunit.client.0.smithi115.stdout:write: 5 123997513~6199875 2022-04-22T18:18:09.619 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 125547481~3099937 2022-04-22T18:18:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:09 smithi121 bash[36902]: cluster 2022-04-22T18:18:08.390509+0000 mon.a (mon.0) 4074 : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2022-04-22T18:18:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:09 smithi115 bash[45495]: cluster 2022-04-22T18:18:08.390509+0000 mon.a (mon.0) 4074 : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2022-04-22T18:18:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:09 smithi115 bash[46934]: cluster 2022-04-22T18:18:08.390509+0000 mon.a (mon.0) 4074 : cluster [DBG] osdmap e1560: 8 total, 8 up, 8 in 2022-04-22T18:18:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:10 smithi121 bash[36902]: cluster 2022-04-22T18:18:09.294069+0000 mgr.y (mgr.34107) 1825 : cluster [DBG] pgmap v2844: 169 pgs: 169 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.5 MiB/s rd, 26 MiB/s wr, 31 op/s 2022-04-22T18:18:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:10 smithi121 bash[36902]: cluster 2022-04-22T18:18:09.401235+0000 mon.a (mon.0) 4075 : cluster [DBG] osdmap e1561: 8 total, 8 up, 8 in 2022-04-22T18:18:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:10 smithi115 bash[45495]: cluster 2022-04-22T18:18:09.294069+0000 mgr.y (mgr.34107) 1825 : cluster [DBG] pgmap v2844: 169 pgs: 169 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.5 MiB/s rd, 26 MiB/s wr, 31 op/s 2022-04-22T18:18:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:10 smithi115 bash[45495]: cluster 2022-04-22T18:18:09.401235+0000 mon.a (mon.0) 4075 : cluster [DBG] osdmap e1561: 8 total, 8 up, 8 in 2022-04-22T18:18:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:10 smithi115 bash[46934]: cluster 2022-04-22T18:18:09.294069+0000 mgr.y (mgr.34107) 1825 : cluster [DBG] pgmap v2844: 169 pgs: 169 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.5 MiB/s rd, 26 MiB/s wr, 31 op/s 2022-04-22T18:18:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:10 smithi115 bash[46934]: cluster 2022-04-22T18:18:09.401235+0000 mon.a (mon.0) 4075 : cluster [DBG] osdmap e1561: 8 total, 8 up, 8 in 2022-04-22T18:18:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:11 smithi121 bash[36902]: audit 2022-04-22T18:18:10.916877+0000 mon.a (mon.0) 4076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:11.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:11 smithi115 bash[45495]: audit 2022-04-22T18:18:10.916877+0000 mon.a (mon.0) 4076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:11 smithi115 bash[46934]: audit 2022-04-22T18:18:10.916877+0000 mon.a (mon.0) 4076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:11.923 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:18:12.110 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:18:12.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:11.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:12 smithi121 bash[36902]: cluster 2022-04-22T18:18:11.295323+0000 mgr.y (mgr.34107) 1826 : cluster [DBG] pgmap v2846: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 21 MiB/s rd, 26 MiB/s wr, 111 op/s 2022-04-22T18:18:12.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:12 smithi121 bash[36902]: audit 2022-04-22T18:18:11.402851+0000 mon.a (mon.0) 4077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:12.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:12 smithi115 bash[45495]: cluster 2022-04-22T18:18:11.295323+0000 mgr.y (mgr.34107) 1826 : cluster [DBG] pgmap v2846: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 21 MiB/s rd, 26 MiB/s wr, 111 op/s 2022-04-22T18:18:12.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:12 smithi115 bash[45495]: audit 2022-04-22T18:18:11.402851+0000 mon.a (mon.0) 4077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:12 smithi115 bash[46934]: cluster 2022-04-22T18:18:11.295323+0000 mgr.y (mgr.34107) 1826 : cluster [DBG] pgmap v2846: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 21 MiB/s rd, 26 MiB/s wr, 111 op/s 2022-04-22T18:18:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:12 smithi115 bash[46934]: audit 2022-04-22T18:18:11.402851+0000 mon.a (mon.0) 4077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:13.683 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:13 smithi115 bash[45495]: cluster 2022-04-22T18:18:12.411239+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2022-04-22T18:18:13.684 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:13 smithi115 bash[45495]: audit 2022-04-22T18:18:13.306428+0000 mon.a (mon.0) 4079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:13.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:13 smithi115 bash[46934]: cluster 2022-04-22T18:18:12.411239+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2022-04-22T18:18:13.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:13 smithi115 bash[46934]: audit 2022-04-22T18:18:13.306428+0000 mon.a (mon.0) 4079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:13 smithi121 bash[36902]: cluster 2022-04-22T18:18:12.411239+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e1562: 8 total, 8 up, 8 in 2022-04-22T18:18:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:13 smithi121 bash[36902]: audit 2022-04-22T18:18:13.306428+0000 mon.a (mon.0) 4079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:14.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:13.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:14 smithi121 bash[36902]: cluster 2022-04-22T18:18:13.296548+0000 mgr.y (mgr.34107) 1827 : cluster [DBG] pgmap v2848: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 24 MiB/s wr, 135 op/s 2022-04-22T18:18:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:14 smithi121 bash[36902]: cluster 2022-04-22T18:18:13.422687+0000 mon.a (mon.0) 4080 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2022-04-22T18:18:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:14 smithi121 bash[36902]: audit 2022-04-22T18:18:13.879339+0000 mon.a (mon.0) 4081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:18:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:14 smithi121 bash[36902]: audit 2022-04-22T18:18:13.885308+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:18:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[45495]: cluster 2022-04-22T18:18:13.296548+0000 mgr.y (mgr.34107) 1827 : cluster [DBG] pgmap v2848: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 24 MiB/s wr, 135 op/s 2022-04-22T18:18:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[45495]: cluster 2022-04-22T18:18:13.422687+0000 mon.a (mon.0) 4080 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2022-04-22T18:18:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[45495]: audit 2022-04-22T18:18:13.879339+0000 mon.a (mon.0) 4081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:18:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[45495]: audit 2022-04-22T18:18:13.885308+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:18:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[46934]: cluster 2022-04-22T18:18:13.296548+0000 mgr.y (mgr.34107) 1827 : cluster [DBG] pgmap v2848: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 24 MiB/s wr, 135 op/s 2022-04-22T18:18:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[46934]: cluster 2022-04-22T18:18:13.422687+0000 mon.a (mon.0) 4080 : cluster [DBG] osdmap e1563: 8 total, 8 up, 8 in 2022-04-22T18:18:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[46934]: audit 2022-04-22T18:18:13.879339+0000 mon.a (mon.0) 4081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:18:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:14 smithi115 bash[46934]: audit 2022-04-22T18:18:13.885308+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:18:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:15 smithi121 bash[36902]: cluster 2022-04-22T18:18:14.432046+0000 mon.a (mon.0) 4083 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2022-04-22T18:18:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:15 smithi115 bash[45495]: cluster 2022-04-22T18:18:14.432046+0000 mon.a (mon.0) 4083 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2022-04-22T18:18:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:15 smithi115 bash[46934]: cluster 2022-04-22T18:18:14.432046+0000 mon.a (mon.0) 4083 : cluster [DBG] osdmap e1564: 8 total, 8 up, 8 in 2022-04-22T18:18:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:16 smithi121 bash[36902]: cluster 2022-04-22T18:18:15.297744+0000 mgr.y (mgr.34107) 1828 : cluster [DBG] pgmap v2851: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s rd, 16 MiB/s wr, 262 op/s 2022-04-22T18:18:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:16 smithi121 bash[36902]: cluster 2022-04-22T18:18:15.440037+0000 mon.a (mon.0) 4084 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2022-04-22T18:18:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:16 smithi121 bash[36902]: audit 2022-04-22T18:18:15.920140+0000 mon.a (mon.0) 4085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:16 smithi121 bash[36902]: audit 2022-04-22T18:18:16.406057+0000 mon.a (mon.0) 4086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[45495]: cluster 2022-04-22T18:18:15.297744+0000 mgr.y (mgr.34107) 1828 : cluster [DBG] pgmap v2851: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s rd, 16 MiB/s wr, 262 op/s 2022-04-22T18:18:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[45495]: cluster 2022-04-22T18:18:15.440037+0000 mon.a (mon.0) 4084 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2022-04-22T18:18:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[45495]: audit 2022-04-22T18:18:15.920140+0000 mon.a (mon.0) 4085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[45495]: audit 2022-04-22T18:18:16.406057+0000 mon.a (mon.0) 4086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[46934]: cluster 2022-04-22T18:18:15.297744+0000 mgr.y (mgr.34107) 1828 : cluster [DBG] pgmap v2851: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s rd, 16 MiB/s wr, 262 op/s 2022-04-22T18:18:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[46934]: cluster 2022-04-22T18:18:15.440037+0000 mon.a (mon.0) 4084 : cluster [DBG] osdmap e1565: 8 total, 8 up, 8 in 2022-04-22T18:18:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[46934]: audit 2022-04-22T18:18:15.920140+0000 mon.a (mon.0) 4085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:16 smithi115 bash[46934]: audit 2022-04-22T18:18:16.406057+0000 mon.a (mon.0) 4086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:17.607 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.Stress2 (28252 ms) 2022-04-22T18:18:17.607 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMigration.StressLive 2022-04-22T18:18:17.648 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 83886080 2022-04-22T18:18:17.655 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 57853420~4583428 2022-04-22T18:18:17.703 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 78991766~4894314 2022-04-22T18:18:17.708 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 44496745~635781 2022-04-22T18:18:17.718 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 66852406~5094741 2022-04-22T18:18:17.724 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 10191652~1762505 2022-04-22T18:18:17.741 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 48756400~6777564 2022-04-22T18:18:17.747 INFO:tasks.workunit.client.0.smithi115.stdout:write: A 58881691~1246054 2022-04-22T18:18:17.761 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 69584416~1302566 2022-04-22T18:18:17.766 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap0 2022-04-22T18:18:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:18 smithi121 bash[36902]: cluster 2022-04-22T18:18:17.298639+0000 mgr.y (mgr.34107) 1829 : cluster [DBG] pgmap v2853: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2022-04-22T18:18:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:18 smithi121 bash[36902]: audit 2022-04-22T18:18:18.309315+0000 mon.a (mon.0) 4087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:18 smithi115 bash[45495]: cluster 2022-04-22T18:18:17.298639+0000 mgr.y (mgr.34107) 1829 : cluster [DBG] pgmap v2853: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2022-04-22T18:18:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:18 smithi115 bash[45495]: audit 2022-04-22T18:18:18.309315+0000 mon.a (mon.0) 4087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:18 smithi115 bash[46934]: cluster 2022-04-22T18:18:17.298639+0000 mgr.y (mgr.34107) 1829 : cluster [DBG] pgmap v2853: 169 pgs: 169 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2022-04-22T18:18:18.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:18 smithi115 bash[46934]: audit 2022-04-22T18:18:18.309315+0000 mon.a (mon.0) 4087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:19.479 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 140512880 2022-04-22T18:18:19.491 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 119309330~7881562 2022-04-22T18:18:19.515 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 2641104~5208588 2022-04-22T18:18:19.515 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 41632945~3188333 2022-04-22T18:18:19.518 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 2949017~7640073 2022-04-22T18:18:19.530 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 134221890~5502226 2022-04-22T18:18:19.533 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 24777002~3250966 2022-04-22T18:18:19.544 INFO:tasks.workunit.client.0.smithi115.stdout:write: B 47639627~5773000 2022-04-22T18:18:19.557 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 128078258~5083846 2022-04-22T18:18:19.563 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap1 2022-04-22T18:18:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:19 smithi121 bash[36902]: cluster 2022-04-22T18:18:18.455858+0000 mon.a (mon.0) 4088 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2022-04-22T18:18:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:19 smithi115 bash[45495]: cluster 2022-04-22T18:18:18.455858+0000 mon.a (mon.0) 4088 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2022-04-22T18:18:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:19 smithi115 bash[46934]: cluster 2022-04-22T18:18:18.455858+0000 mon.a (mon.0) 4088 : cluster [DBG] osdmap e1566: 8 total, 8 up, 8 in 2022-04-22T18:18:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:20 smithi121 bash[36902]: cluster 2022-04-22T18:18:19.299748+0000 mgr.y (mgr.34107) 1830 : cluster [DBG] pgmap v2855: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2022-04-22T18:18:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:20 smithi121 bash[36902]: cluster 2022-04-22T18:18:19.466296+0000 mon.a (mon.0) 4089 : cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2022-04-22T18:18:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:20 smithi115 bash[45495]: cluster 2022-04-22T18:18:19.299748+0000 mgr.y (mgr.34107) 1830 : cluster [DBG] pgmap v2855: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2022-04-22T18:18:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:20 smithi115 bash[45495]: cluster 2022-04-22T18:18:19.466296+0000 mon.a (mon.0) 4089 : cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2022-04-22T18:18:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:20 smithi115 bash[46934]: cluster 2022-04-22T18:18:19.299748+0000 mgr.y (mgr.34107) 1830 : cluster [DBG] pgmap v2855: 169 pgs: 169 active+clean; 1.5 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2022-04-22T18:18:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:20 smithi115 bash[46934]: cluster 2022-04-22T18:18:19.466296+0000 mon.a (mon.0) 4089 : cluster [DBG] osdmap e1567: 8 total, 8 up, 8 in 2022-04-22T18:18:21.300 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 114771532 2022-04-22T18:18:21.330 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 10906638~4832411 2022-04-22T18:18:21.342 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 8882228~5463170 2022-04-22T18:18:21.360 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 110068789~4702743 2022-04-22T18:18:21.363 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 19675365~5019669 2022-04-22T18:18:21.513 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 81438039~4013561 2022-04-22T18:18:21.514 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 78925050~35741 2022-04-22T18:18:21.518 INFO:tasks.workunit.client.0.smithi115.stdout:write: C 23167847~4228779 2022-04-22T18:18:21.518 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 12866987~8018469 2022-04-22T18:18:21.518 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap2 2022-04-22T18:18:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:21 smithi121 bash[36902]: cluster 2022-04-22T18:18:20.474615+0000 mon.a (mon.0) 4090 : cluster [DBG] osdmap e1568: 8 total, 8 up, 8 in 2022-04-22T18:18:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:21 smithi121 bash[36902]: audit 2022-04-22T18:18:20.923004+0000 mon.a (mon.0) 4091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:21 smithi121 bash[36902]: cluster 2022-04-22T18:18:21.282284+0000 mon.a (mon.0) 4092 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2022-04-22T18:18:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:21 smithi121 bash[36902]: audit 2022-04-22T18:18:21.409221+0000 mon.a (mon.0) 4093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[45495]: cluster 2022-04-22T18:18:20.474615+0000 mon.a (mon.0) 4090 : cluster [DBG] osdmap e1568: 8 total, 8 up, 8 in 2022-04-22T18:18:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[45495]: audit 2022-04-22T18:18:20.923004+0000 mon.a (mon.0) 4091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[45495]: cluster 2022-04-22T18:18:21.282284+0000 mon.a (mon.0) 4092 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2022-04-22T18:18:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[45495]: audit 2022-04-22T18:18:21.409221+0000 mon.a (mon.0) 4093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:21.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[46934]: cluster 2022-04-22T18:18:20.474615+0000 mon.a (mon.0) 4090 : cluster [DBG] osdmap e1568: 8 total, 8 up, 8 in 2022-04-22T18:18:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[46934]: audit 2022-04-22T18:18:20.923004+0000 mon.a (mon.0) 4091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[46934]: cluster 2022-04-22T18:18:21.282284+0000 mon.a (mon.0) 4092 : cluster [DBG] osdmap e1569: 8 total, 8 up, 8 in 2022-04-22T18:18:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[46934]: audit 2022-04-22T18:18:21.409221+0000 mon.a (mon.0) 4093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:22.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:21.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:22 smithi121 bash[36902]: cluster 2022-04-22T18:18:21.300370+0000 mgr.y (mgr.34107) 1831 : cluster [DBG] pgmap v2859: 169 pgs: 169 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s rd, 11 MiB/s wr, 108 op/s 2022-04-22T18:18:22.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:22 smithi121 bash[36902]: cluster 2022-04-22T18:18:22.299291+0000 mon.a (mon.0) 4094 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2022-04-22T18:18:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:22 smithi115 bash[45495]: cluster 2022-04-22T18:18:21.300370+0000 mgr.y (mgr.34107) 1831 : cluster [DBG] pgmap v2859: 169 pgs: 169 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s rd, 11 MiB/s wr, 108 op/s 2022-04-22T18:18:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:22 smithi115 bash[45495]: cluster 2022-04-22T18:18:22.299291+0000 mon.a (mon.0) 4094 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2022-04-22T18:18:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:22 smithi115 bash[46934]: cluster 2022-04-22T18:18:21.300370+0000 mgr.y (mgr.34107) 1831 : cluster [DBG] pgmap v2859: 169 pgs: 169 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s rd, 11 MiB/s wr, 108 op/s 2022-04-22T18:18:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:22 smithi115 bash[46934]: cluster 2022-04-22T18:18:22.299291+0000 mon.a (mon.0) 4094 : cluster [DBG] osdmap e1570: 8 total, 8 up, 8 in 2022-04-22T18:18:23.322 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 93947878 2022-04-22T18:18:23.344 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 45930252~4984712 2022-04-22T18:18:23.358 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 79291651~1580300 2022-04-22T18:18:23.359 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 31715596~1469778 2022-04-22T18:18:23.360 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 62661970~8049013 2022-04-22T18:18:23.360 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 28259543~4277053 2022-04-22T18:18:23.387 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 45766215~1132877 2022-04-22T18:18:23.388 INFO:tasks.workunit.client.0.smithi115.stdout:write: D 47998960~2075271 2022-04-22T18:18:23.389 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 62934975~4981709 2022-04-22T18:18:23.390 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: snap3 2022-04-22T18:18:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:23.679Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:24 smithi121 bash[36902]: cluster 2022-04-22T18:18:23.301664+0000 mgr.y (mgr.34107) 1832 : cluster [DBG] pgmap v2861: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 16 MiB/s wr, 86 op/s 2022-04-22T18:18:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:24 smithi121 bash[36902]: cluster 2022-04-22T18:18:23.306154+0000 mon.a (mon.0) 4095 : cluster [DBG] osdmap e1571: 8 total, 8 up, 8 in 2022-04-22T18:18:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:24 smithi121 bash[36902]: audit 2022-04-22T18:18:23.311964+0000 mon.a (mon.0) 4096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[45495]: cluster 2022-04-22T18:18:23.301664+0000 mgr.y (mgr.34107) 1832 : cluster [DBG] pgmap v2861: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 16 MiB/s wr, 86 op/s 2022-04-22T18:18:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[45495]: cluster 2022-04-22T18:18:23.306154+0000 mon.a (mon.0) 4095 : cluster [DBG] osdmap e1571: 8 total, 8 up, 8 in 2022-04-22T18:18:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[45495]: audit 2022-04-22T18:18:23.311964+0000 mon.a (mon.0) 4096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[46934]: cluster 2022-04-22T18:18:23.301664+0000 mgr.y (mgr.34107) 1832 : cluster [DBG] pgmap v2861: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 83 MiB/s rd, 16 MiB/s wr, 86 op/s 2022-04-22T18:18:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[46934]: cluster 2022-04-22T18:18:23.306154+0000 mon.a (mon.0) 4095 : cluster [DBG] osdmap e1571: 8 total, 8 up, 8 in 2022-04-22T18:18:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:24 smithi115 bash[46934]: audit 2022-04-22T18:18:23.311964+0000 mon.a (mon.0) 4096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:25.337 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 98703392 2022-04-22T18:18:25.350 INFO:tasks.workunit.client.0.smithi115.stdout:migration_prepare 2022-04-22T18:18:25.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:25 smithi121 bash[36902]: cluster 2022-04-22T18:18:24.314085+0000 mon.a (mon.0) 4097 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2022-04-22T18:18:25.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:25 smithi115 bash[45495]: cluster 2022-04-22T18:18:24.314085+0000 mon.a (mon.0) 4097 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2022-04-22T18:18:25.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:25 smithi115 bash[46934]: cluster 2022-04-22T18:18:24.314085+0000 mon.a (mon.0) 4097 : cluster [DBG] osdmap e1572: 8 total, 8 up, 8 in 2022-04-22T18:18:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:26 smithi121 bash[36902]: cluster 2022-04-22T18:18:25.302915+0000 mgr.y (mgr.34107) 1833 : cluster [DBG] pgmap v2864: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 14 MiB/s rd, 12 MiB/s wr, 32 op/s 2022-04-22T18:18:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:26 smithi121 bash[36902]: cluster 2022-04-22T18:18:25.319553+0000 mon.a (mon.0) 4098 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2022-04-22T18:18:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:26 smithi121 bash[36902]: audit 2022-04-22T18:18:25.926170+0000 mon.a (mon.0) 4099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:26 smithi121 bash[36902]: cluster 2022-04-22T18:18:26.283638+0000 mon.a (mon.0) 4100 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2022-04-22T18:18:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[45495]: cluster 2022-04-22T18:18:25.302915+0000 mgr.y (mgr.34107) 1833 : cluster [DBG] pgmap v2864: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 14 MiB/s rd, 12 MiB/s wr, 32 op/s 2022-04-22T18:18:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[45495]: cluster 2022-04-22T18:18:25.319553+0000 mon.a (mon.0) 4098 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2022-04-22T18:18:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[45495]: audit 2022-04-22T18:18:25.926170+0000 mon.a (mon.0) 4099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[45495]: cluster 2022-04-22T18:18:26.283638+0000 mon.a (mon.0) 4100 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2022-04-22T18:18:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[46934]: cluster 2022-04-22T18:18:25.302915+0000 mgr.y (mgr.34107) 1833 : cluster [DBG] pgmap v2864: 169 pgs: 169 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 14 MiB/s rd, 12 MiB/s wr, 32 op/s 2022-04-22T18:18:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[46934]: cluster 2022-04-22T18:18:25.319553+0000 mon.a (mon.0) 4098 : cluster [DBG] osdmap e1573: 8 total, 8 up, 8 in 2022-04-22T18:18:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[46934]: audit 2022-04-22T18:18:25.926170+0000 mon.a (mon.0) 4099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:26 smithi115 bash[46934]: cluster 2022-04-22T18:18:26.283638+0000 mon.a (mon.0) 4100 : cluster [DBG] osdmap e1574: 8 total, 8 up, 8 in 2022-04-22T18:18:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:27 smithi121 bash[36902]: audit 2022-04-22T18:18:26.412312+0000 mon.a (mon.0) 4101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:27 smithi121 bash[36902]: cluster 2022-04-22T18:18:27.296879+0000 mon.a (mon.0) 4102 : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2022-04-22T18:18:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:27 smithi115 bash[46934]: audit 2022-04-22T18:18:26.412312+0000 mon.a (mon.0) 4101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:27 smithi115 bash[46934]: cluster 2022-04-22T18:18:27.296879+0000 mon.a (mon.0) 4102 : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2022-04-22T18:18:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:27 smithi115 bash[45495]: audit 2022-04-22T18:18:26.412312+0000 mon.a (mon.0) 4101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:27 smithi115 bash[45495]: cluster 2022-04-22T18:18:27.296879+0000 mon.a (mon.0) 4102 : cluster [DBG] osdmap e1575: 8 total, 8 up, 8 in 2022-04-22T18:18:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:28 smithi121 bash[36902]: cluster 2022-04-22T18:18:27.303495+0000 mgr.y (mgr.34107) 1834 : cluster [DBG] pgmap v2868: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 22 op/s 2022-04-22T18:18:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:28 smithi121 bash[36902]: cluster 2022-04-22T18:18:28.306025+0000 mon.a (mon.0) 4103 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2022-04-22T18:18:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:28 smithi121 bash[36902]: audit 2022-04-22T18:18:28.313969+0000 mon.a (mon.0) 4104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[46934]: cluster 2022-04-22T18:18:27.303495+0000 mgr.y (mgr.34107) 1834 : cluster [DBG] pgmap v2868: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 22 op/s 2022-04-22T18:18:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[46934]: cluster 2022-04-22T18:18:28.306025+0000 mon.a (mon.0) 4103 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2022-04-22T18:18:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[46934]: audit 2022-04-22T18:18:28.313969+0000 mon.a (mon.0) 4104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[45495]: cluster 2022-04-22T18:18:27.303495+0000 mgr.y (mgr.34107) 1834 : cluster [DBG] pgmap v2868: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 22 op/s 2022-04-22T18:18:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[45495]: cluster 2022-04-22T18:18:28.306025+0000 mon.a (mon.0) 4103 : cluster [DBG] osdmap e1576: 8 total, 8 up, 8 in 2022-04-22T18:18:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:28 smithi115 bash[45495]: audit 2022-04-22T18:18:28.313969+0000 mon.a (mon.0) 4104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:30.340 INFO:tasks.workunit.client.0.smithi115.stdout:migration_execute 2022-04-22T18:18:30.340 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 82179842~3001959 2022-04-22T18:18:30.373 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 19323937~7244641 2022-04-22T18:18:30.400 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 86697662~7124893 2022-04-22T18:18:30.453 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 2416072~530515 2022-04-22T18:18:30.467 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 66720248~6809077 2022-04-22T18:18:30.544 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 6530589~4013824 2022-04-22T18:18:30.560 INFO:tasks.workunit.client.0.smithi115.stdout:write: a 64973832~2978571 2022-04-22T18:18:30.580 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 91815767~2055885 2022-04-22T18:18:30.583 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user0 2022-04-22T18:18:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:30 smithi121 bash[36902]: cluster 2022-04-22T18:18:29.304599+0000 mgr.y (mgr.34107) 1835 : cluster [DBG] pgmap v2870: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.5 MiB/s wr, 7 op/s 2022-04-22T18:18:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:30 smithi121 bash[36902]: cluster 2022-04-22T18:18:29.309918+0000 mon.a (mon.0) 4105 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2022-04-22T18:18:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:30 smithi115 bash[46934]: cluster 2022-04-22T18:18:29.304599+0000 mgr.y (mgr.34107) 1835 : cluster [DBG] pgmap v2870: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.5 MiB/s wr, 7 op/s 2022-04-22T18:18:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:30 smithi115 bash[46934]: cluster 2022-04-22T18:18:29.309918+0000 mon.a (mon.0) 4105 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2022-04-22T18:18:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:30 smithi115 bash[45495]: cluster 2022-04-22T18:18:29.304599+0000 mgr.y (mgr.34107) 1835 : cluster [DBG] pgmap v2870: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.5 MiB/s wr, 7 op/s 2022-04-22T18:18:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:30 smithi115 bash[45495]: cluster 2022-04-22T18:18:29.309918+0000 mon.a (mon.0) 4105 : cluster [DBG] osdmap e1577: 8 total, 8 up, 8 in 2022-04-22T18:18:31.108 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:18:31.463 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:18:31.463 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (41m) 10m ago 55m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T18:18:31.463 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (55m) 34s ago 55m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:18:31.463 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (43m) 10m ago 56m 22.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:18:31.463 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (43m) 34s ago 56m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (50m) 34s ago 60m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (51m) 10m ago 62m 467M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (50m) 10m ago 62m 153M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (49m) 34s ago 60m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (49m) 10m ago 61m 101M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (42m) 10m ago 55m 13.1M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (42m) 34s ago 55m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (48m) 10m ago 60m 604M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (47m) 10m ago 59m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (47m) 10m ago 59m 439M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:18:31.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (46m) 10m ago 58m 393M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:18:31.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (46m) 34s ago 58m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:18:31.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (45m) 34s ago 57m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:18:31.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (44m) 34s ago 57m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:18:31.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (43m) 34s ago 56m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:18:31.465 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (42m) 34s ago 55m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:18:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:31 smithi121 bash[36902]: audit 2022-04-22T18:18:30.929340+0000 mon.a (mon.0) 4106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:31 smithi115 bash[45495]: audit 2022-04-22T18:18:30.929340+0000 mon.a (mon.0) 4106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:31.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:31 smithi115 bash[46934]: audit 2022-04-22T18:18:30.929340+0000 mon.a (mon.0) 4106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:31.871 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:18:31.871 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:18:31.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:18:31.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:18:31.873 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:18:31.873 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:18:31.873 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:18:31.873 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:18:32.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:31.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:32.358 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 154284544 2022-04-22T18:18:32.371 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 42831512~1001529 2022-04-22T18:18:32.382 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 32912247~2379834 2022-04-22T18:18:32.382 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 89537415~7112916 2022-04-22T18:18:32.390 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 23334235~7360594 2022-04-22T18:18:32.390 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 49515551~1089392 2022-04-22T18:18:32.392 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 116451517~8338021 2022-04-22T18:18:32.401 INFO:tasks.workunit.client.0.smithi115.stdout:write: b 46703820~5249540 2022-04-22T18:18:32.425 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 12823397~6063647 2022-04-22T18:18:32.425 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user1 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: audit 2022-04-22T18:18:31.087417+0000 mgr.y (mgr.34107) 1836 : audit [DBG] from='client.47077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: cluster 2022-04-22T18:18:31.305946+0000 mgr.y (mgr.34107) 1837 : cluster [DBG] pgmap v2872: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 20 MiB/s rd, 3.6 MiB/s wr, 69 op/s 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: cluster 2022-04-22T18:18:31.324973+0000 mon.a (mon.0) 4107 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: audit 2022-04-22T18:18:31.415801+0000 mon.a (mon.0) 4108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: audit 2022-04-22T18:18:31.435753+0000 mgr.y (mgr.34107) 1838 : audit [DBG] from='client.47083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:32 smithi121 bash[36902]: audit 2022-04-22T18:18:31.865272+0000 mon.a (mon.0) 4109 : audit [DBG] from='client.? 172.21.15.115:0/2062933593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: audit 2022-04-22T18:18:31.087417+0000 mgr.y (mgr.34107) 1836 : audit [DBG] from='client.47077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: cluster 2022-04-22T18:18:31.305946+0000 mgr.y (mgr.34107) 1837 : cluster [DBG] pgmap v2872: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 20 MiB/s rd, 3.6 MiB/s wr, 69 op/s 2022-04-22T18:18:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: cluster 2022-04-22T18:18:31.324973+0000 mon.a (mon.0) 4107 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2022-04-22T18:18:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: audit 2022-04-22T18:18:31.415801+0000 mon.a (mon.0) 4108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: audit 2022-04-22T18:18:31.435753+0000 mgr.y (mgr.34107) 1838 : audit [DBG] from='client.47083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[45495]: audit 2022-04-22T18:18:31.865272+0000 mon.a (mon.0) 4109 : audit [DBG] from='client.? 172.21.15.115:0/2062933593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: audit 2022-04-22T18:18:31.087417+0000 mgr.y (mgr.34107) 1836 : audit [DBG] from='client.47077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: cluster 2022-04-22T18:18:31.305946+0000 mgr.y (mgr.34107) 1837 : cluster [DBG] pgmap v2872: 169 pgs: 169 active+clean; 1.6 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 20 MiB/s rd, 3.6 MiB/s wr, 69 op/s 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: cluster 2022-04-22T18:18:31.324973+0000 mon.a (mon.0) 4107 : cluster [DBG] osdmap e1578: 8 total, 8 up, 8 in 2022-04-22T18:18:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: audit 2022-04-22T18:18:31.415801+0000 mon.a (mon.0) 4108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: audit 2022-04-22T18:18:31.435753+0000 mgr.y (mgr.34107) 1838 : audit [DBG] from='client.47083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:18:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:32 smithi115 bash[46934]: audit 2022-04-22T18:18:31.865272+0000 mon.a (mon.0) 4109 : audit [DBG] from='client.? 172.21.15.115:0/2062933593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:18:33.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:33 smithi115 bash[45495]: cluster 2022-04-22T18:18:32.341534+0000 mon.a (mon.0) 4110 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2022-04-22T18:18:33.686 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:33 smithi115 bash[45495]: audit 2022-04-22T18:18:33.316375+0000 mon.a (mon.0) 4111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:33.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:33 smithi115 bash[46934]: cluster 2022-04-22T18:18:32.341534+0000 mon.a (mon.0) 4110 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2022-04-22T18:18:33.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:33 smithi115 bash[46934]: audit 2022-04-22T18:18:33.316375+0000 mon.a (mon.0) 4111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:33 smithi121 bash[36902]: cluster 2022-04-22T18:18:32.341534+0000 mon.a (mon.0) 4110 : cluster [DBG] osdmap e1579: 8 total, 8 up, 8 in 2022-04-22T18:18:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:33 smithi121 bash[36902]: audit 2022-04-22T18:18:33.316375+0000 mon.a (mon.0) 4111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:34.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:33.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:34.371 INFO:tasks.workunit.client.0.smithi115.stdout:resize: 175711196 2022-04-22T18:18:34.384 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 82551565~2997348 2022-04-22T18:18:34.444 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 78264005~7764996 2022-04-22T18:18:34.444 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 139406228~2269177 2022-04-22T18:18:34.444 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 76852332~6224566 2022-04-22T18:18:34.445 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 135008716~4000141 2022-04-22T18:18:34.445 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 26183048~1418011 2022-04-22T18:18:34.445 INFO:tasks.workunit.client.0.smithi115.stdout:write: c 172293688~3417508 2022-04-22T18:18:34.445 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 167428754~688457 2022-04-22T18:18:34.445 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user2 2022-04-22T18:18:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:34 smithi121 bash[36902]: cluster 2022-04-22T18:18:33.307189+0000 mgr.y (mgr.34107) 1839 : cluster [DBG] pgmap v2875: 169 pgs: 169 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 47 MiB/s rd, 11 MiB/s wr, 95 op/s 2022-04-22T18:18:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:34 smithi121 bash[36902]: cluster 2022-04-22T18:18:33.349847+0000 mon.a (mon.0) 4112 : cluster [DBG] osdmap e1580: 8 total, 8 up, 8 in 2022-04-22T18:18:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:34 smithi115 bash[45495]: cluster 2022-04-22T18:18:33.307189+0000 mgr.y (mgr.34107) 1839 : cluster [DBG] pgmap v2875: 169 pgs: 169 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 47 MiB/s rd, 11 MiB/s wr, 95 op/s 2022-04-22T18:18:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:34 smithi115 bash[45495]: cluster 2022-04-22T18:18:33.349847+0000 mon.a (mon.0) 4112 : cluster [DBG] osdmap e1580: 8 total, 8 up, 8 in 2022-04-22T18:18:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:34 smithi115 bash[46934]: cluster 2022-04-22T18:18:33.307189+0000 mgr.y (mgr.34107) 1839 : cluster [DBG] pgmap v2875: 169 pgs: 169 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 47 MiB/s rd, 11 MiB/s wr, 95 op/s 2022-04-22T18:18:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:34 smithi115 bash[46934]: cluster 2022-04-22T18:18:33.349847+0000 mon.a (mon.0) 4112 : cluster [DBG] osdmap e1580: 8 total, 8 up, 8 in 2022-04-22T18:18:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:35 smithi121 bash[36902]: cluster 2022-04-22T18:18:34.354144+0000 mon.a (mon.0) 4113 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2022-04-22T18:18:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:35 smithi115 bash[45495]: cluster 2022-04-22T18:18:34.354144+0000 mon.a (mon.0) 4113 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2022-04-22T18:18:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:35 smithi115 bash[46934]: cluster 2022-04-22T18:18:34.354144+0000 mon.a (mon.0) 4113 : cluster [DBG] osdmap e1581: 8 total, 8 up, 8 in 2022-04-22T18:18:36.389 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 68030310~7801373 2022-04-22T18:18:36.406 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 109264248~6670110 2022-04-22T18:18:36.408 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 55713849~2010838 2022-04-22T18:18:36.411 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 65992083~161757 2022-04-22T18:18:36.412 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 55243066~7484737 2022-04-22T18:18:36.472 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 170764330~4946866 2022-04-22T18:18:36.473 INFO:tasks.workunit.client.0.smithi115.stdout:write: d 114905814~4691415 2022-04-22T18:18:36.480 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 143715376~6199051 2022-04-22T18:18:36.487 INFO:tasks.workunit.client.0.smithi115.stdout:snap_create: user3 2022-04-22T18:18:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:36 smithi121 bash[36902]: cluster 2022-04-22T18:18:35.308418+0000 mgr.y (mgr.34107) 1840 : cluster [DBG] pgmap v2878: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 44 MiB/s rd, 26 MiB/s wr, 208 op/s 2022-04-22T18:18:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:36 smithi121 bash[36902]: cluster 2022-04-22T18:18:35.373352+0000 mon.a (mon.0) 4114 : cluster [DBG] osdmap e1582: 8 total, 8 up, 8 in 2022-04-22T18:18:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:36 smithi121 bash[36902]: audit 2022-04-22T18:18:35.932234+0000 mon.a (mon.0) 4115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[45495]: cluster 2022-04-22T18:18:35.308418+0000 mgr.y (mgr.34107) 1840 : cluster [DBG] pgmap v2878: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 44 MiB/s rd, 26 MiB/s wr, 208 op/s 2022-04-22T18:18:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[45495]: cluster 2022-04-22T18:18:35.373352+0000 mon.a (mon.0) 4114 : cluster [DBG] osdmap e1582: 8 total, 8 up, 8 in 2022-04-22T18:18:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[45495]: audit 2022-04-22T18:18:35.932234+0000 mon.a (mon.0) 4115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:36.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[46934]: cluster 2022-04-22T18:18:35.308418+0000 mgr.y (mgr.34107) 1840 : cluster [DBG] pgmap v2878: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 44 MiB/s rd, 26 MiB/s wr, 208 op/s 2022-04-22T18:18:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[46934]: cluster 2022-04-22T18:18:35.373352+0000 mon.a (mon.0) 4114 : cluster [DBG] osdmap e1582: 8 total, 8 up, 8 in 2022-04-22T18:18:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:36 smithi115 bash[46934]: audit 2022-04-22T18:18:35.932234+0000 mon.a (mon.0) 4115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:37 smithi121 bash[36902]: cluster 2022-04-22T18:18:36.376508+0000 mon.a (mon.0) 4116 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2022-04-22T18:18:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:37 smithi121 bash[36902]: audit 2022-04-22T18:18:36.418877+0000 mon.a (mon.0) 4117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:37 smithi115 bash[45495]: cluster 2022-04-22T18:18:36.376508+0000 mon.a (mon.0) 4116 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2022-04-22T18:18:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:37 smithi115 bash[45495]: audit 2022-04-22T18:18:36.418877+0000 mon.a (mon.0) 4117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:37 smithi115 bash[46934]: cluster 2022-04-22T18:18:36.376508+0000 mon.a (mon.0) 4116 : cluster [DBG] osdmap e1583: 8 total, 8 up, 8 in 2022-04-22T18:18:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:37 smithi115 bash[46934]: audit 2022-04-22T18:18:36.418877+0000 mon.a (mon.0) 4117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:38.433 INFO:tasks.workunit.client.0.smithi115.stdout:write: 1 17571119~4392779 2022-04-22T18:18:38.444 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 18669313~2196389 2022-04-22T18:18:38.472 INFO:tasks.workunit.client.0.smithi115.stdout:write: 2 35142239~4392779 2022-04-22T18:18:38.475 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 36240433~2196389 2022-04-22T18:18:38.495 INFO:tasks.workunit.client.0.smithi115.stdout:write: 3 52713358~4392779 2022-04-22T18:18:38.498 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 53811552~2196389 2022-04-22T18:18:38.520 INFO:tasks.workunit.client.0.smithi115.stdout:write: 4 70284478~4392779 2022-04-22T18:18:38.523 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 71382672~2196389 2022-04-22T18:18:38.542 INFO:tasks.workunit.client.0.smithi115.stdout:write: 5 87855598~4392779 2022-04-22T18:18:38.545 INFO:tasks.workunit.client.0.smithi115.stdout:discard: 88953792~2196389 2022-04-22T18:18:38.625 INFO:tasks.workunit.client.0.smithi115.stdout:migration_commit 2022-04-22T18:18:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:38 smithi121 bash[36902]: cluster 2022-04-22T18:18:37.309606+0000 mgr.y (mgr.34107) 1841 : cluster [DBG] pgmap v2881: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 10 MiB/s rd, 17 MiB/s wr, 174 op/s 2022-04-22T18:18:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:38 smithi121 bash[36902]: cluster 2022-04-22T18:18:37.394556+0000 mon.a (mon.0) 4118 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2022-04-22T18:18:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:38 smithi121 bash[36902]: audit 2022-04-22T18:18:38.319415+0000 mon.a (mon.0) 4119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[45495]: cluster 2022-04-22T18:18:37.309606+0000 mgr.y (mgr.34107) 1841 : cluster [DBG] pgmap v2881: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 10 MiB/s rd, 17 MiB/s wr, 174 op/s 2022-04-22T18:18:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[45495]: cluster 2022-04-22T18:18:37.394556+0000 mon.a (mon.0) 4118 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2022-04-22T18:18:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[45495]: audit 2022-04-22T18:18:38.319415+0000 mon.a (mon.0) 4119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[46934]: cluster 2022-04-22T18:18:37.309606+0000 mgr.y (mgr.34107) 1841 : cluster [DBG] pgmap v2881: 169 pgs: 169 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 10 MiB/s rd, 17 MiB/s wr, 174 op/s 2022-04-22T18:18:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[46934]: cluster 2022-04-22T18:18:37.394556+0000 mon.a (mon.0) 4118 : cluster [DBG] osdmap e1584: 8 total, 8 up, 8 in 2022-04-22T18:18:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:38 smithi115 bash[46934]: audit 2022-04-22T18:18:38.319415+0000 mon.a (mon.0) 4119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:39 smithi121 bash[36902]: cluster 2022-04-22T18:18:38.410709+0000 mon.a (mon.0) 4120 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2022-04-22T18:18:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:39 smithi115 bash[45495]: cluster 2022-04-22T18:18:38.410709+0000 mon.a (mon.0) 4120 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2022-04-22T18:18:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:39 smithi115 bash[46934]: cluster 2022-04-22T18:18:38.410709+0000 mon.a (mon.0) 4120 : cluster [DBG] osdmap e1585: 8 total, 8 up, 8 in 2022-04-22T18:18:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:40 smithi121 bash[36902]: cluster 2022-04-22T18:18:39.310764+0000 mgr.y (mgr.34107) 1842 : cluster [DBG] pgmap v2884: 169 pgs: 169 active+clean; 1.8 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 8 op/s 2022-04-22T18:18:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:40 smithi121 bash[36902]: cluster 2022-04-22T18:18:39.417728+0000 mon.a (mon.0) 4121 : cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2022-04-22T18:18:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:40 smithi115 bash[45495]: cluster 2022-04-22T18:18:39.310764+0000 mgr.y (mgr.34107) 1842 : cluster [DBG] pgmap v2884: 169 pgs: 169 active+clean; 1.8 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 8 op/s 2022-04-22T18:18:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:40 smithi115 bash[45495]: cluster 2022-04-22T18:18:39.417728+0000 mon.a (mon.0) 4121 : cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2022-04-22T18:18:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:40 smithi115 bash[46934]: cluster 2022-04-22T18:18:39.310764+0000 mgr.y (mgr.34107) 1842 : cluster [DBG] pgmap v2884: 169 pgs: 169 active+clean; 1.8 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 8 op/s 2022-04-22T18:18:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:40 smithi115 bash[46934]: cluster 2022-04-22T18:18:39.417728+0000 mon.a (mon.0) 4121 : cluster [DBG] osdmap e1586: 8 total, 8 up, 8 in 2022-04-22T18:18:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:41 smithi121 bash[36902]: cluster 2022-04-22T18:18:40.423995+0000 mon.a (mon.0) 4122 : cluster [DBG] osdmap e1587: 8 total, 8 up, 8 in 2022-04-22T18:18:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:41 smithi121 bash[36902]: audit 2022-04-22T18:18:40.935364+0000 mon.a (mon.0) 4123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:41 smithi115 bash[45495]: cluster 2022-04-22T18:18:40.423995+0000 mon.a (mon.0) 4122 : cluster [DBG] osdmap e1587: 8 total, 8 up, 8 in 2022-04-22T18:18:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:41 smithi115 bash[45495]: audit 2022-04-22T18:18:40.935364+0000 mon.a (mon.0) 4123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:41 smithi115 bash[46934]: cluster 2022-04-22T18:18:40.423995+0000 mon.a (mon.0) 4122 : cluster [DBG] osdmap e1587: 8 total, 8 up, 8 in 2022-04-22T18:18:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:41 smithi115 bash[46934]: audit 2022-04-22T18:18:40.935364+0000 mon.a (mon.0) 4123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:42.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:41.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:42 smithi121 bash[36902]: cluster 2022-04-22T18:18:41.311950+0000 mgr.y (mgr.34107) 1843 : cluster [DBG] pgmap v2887: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 96 KiB/s rd, 19 MiB/s wr, 144 op/s 2022-04-22T18:18:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:42 smithi121 bash[36902]: audit 2022-04-22T18:18:41.422727+0000 mon.a (mon.0) 4124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:42 smithi121 bash[36902]: cluster 2022-04-22T18:18:41.437515+0000 mon.a (mon.0) 4125 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2022-04-22T18:18:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[46934]: cluster 2022-04-22T18:18:41.311950+0000 mgr.y (mgr.34107) 1843 : cluster [DBG] pgmap v2887: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 96 KiB/s rd, 19 MiB/s wr, 144 op/s 2022-04-22T18:18:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[46934]: audit 2022-04-22T18:18:41.422727+0000 mon.a (mon.0) 4124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[46934]: cluster 2022-04-22T18:18:41.437515+0000 mon.a (mon.0) 4125 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2022-04-22T18:18:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[45495]: cluster 2022-04-22T18:18:41.311950+0000 mgr.y (mgr.34107) 1843 : cluster [DBG] pgmap v2887: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 96 KiB/s rd, 19 MiB/s wr, 144 op/s 2022-04-22T18:18:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[45495]: audit 2022-04-22T18:18:41.422727+0000 mon.a (mon.0) 4124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:42 smithi115 bash[45495]: cluster 2022-04-22T18:18:41.437515+0000 mon.a (mon.0) 4125 : cluster [DBG] osdmap e1588: 8 total, 8 up, 8 in 2022-04-22T18:18:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:43 smithi121 bash[36902]: cluster 2022-04-22T18:18:42.438956+0000 mon.a (mon.0) 4126 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2022-04-22T18:18:43.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:43 smithi121 bash[36902]: audit 2022-04-22T18:18:43.322367+0000 mon.a (mon.0) 4127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:43 smithi115 bash[45495]: cluster 2022-04-22T18:18:42.438956+0000 mon.a (mon.0) 4126 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2022-04-22T18:18:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:43 smithi115 bash[45495]: audit 2022-04-22T18:18:43.322367+0000 mon.a (mon.0) 4127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:43.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:43.680Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:43 smithi115 bash[46934]: cluster 2022-04-22T18:18:42.438956+0000 mon.a (mon.0) 4126 : cluster [DBG] osdmap e1589: 8 total, 8 up, 8 in 2022-04-22T18:18:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:43 smithi115 bash[46934]: audit 2022-04-22T18:18:43.322367+0000 mon.a (mon.0) 4127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:44.353 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMigration.StressLive (26747 ms) 2022-04-22T18:18:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:44 smithi121 bash[36902]: cluster 2022-04-22T18:18:43.313178+0000 mgr.y (mgr.34107) 1844 : cluster [DBG] pgmap v2890: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 14 MiB/s wr, 151 op/s 2022-04-22T18:18:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:44 smithi115 bash[45495]: cluster 2022-04-22T18:18:43.313178+0000 mgr.y (mgr.34107) 1844 : cluster [DBG] pgmap v2890: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 14 MiB/s wr, 151 op/s 2022-04-22T18:18:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:44 smithi115 bash[46934]: cluster 2022-04-22T18:18:43.313178+0000 mgr.y (mgr.34107) 1844 : cluster [DBG] pgmap v2890: 169 pgs: 169 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 14 MiB/s wr, 151 op/s 2022-04-22T18:18:45.479 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 57 tests from TestMigration (252712 ms total) 2022-04-22T18:18:45.479 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:18:45.480 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from TestMirroringWatcher 2022-04-22T18:18:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:45 smithi121 bash[36902]: cluster 2022-04-22T18:18:44.450571+0000 mon.a (mon.0) 4128 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2022-04-22T18:18:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:45 smithi115 bash[45495]: cluster 2022-04-22T18:18:44.450571+0000 mon.a (mon.0) 4128 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2022-04-22T18:18:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:45 smithi115 bash[46934]: cluster 2022-04-22T18:18:44.450571+0000 mon.a (mon.0) 4128 : cluster [DBG] osdmap e1590: 8 total, 8 up, 8 in 2022-04-22T18:18:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:46 smithi121 bash[36902]: cluster 2022-04-22T18:18:45.314316+0000 mgr.y (mgr.34107) 1845 : cluster [DBG] pgmap v2892: 137 pgs: 137 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2022-04-22T18:18:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:46 smithi121 bash[36902]: cluster 2022-04-22T18:18:45.454563+0000 mon.a (mon.0) 4129 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2022-04-22T18:18:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:46 smithi121 bash[36902]: cluster 2022-04-22T18:18:45.466274+0000 mon.a (mon.0) 4130 : cluster [DBG] osdmap e1591: 8 total, 8 up, 8 in 2022-04-22T18:18:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:46 smithi121 bash[36902]: audit 2022-04-22T18:18:45.968377+0000 mon.a (mon.0) 4131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:46 smithi121 bash[36902]: audit 2022-04-22T18:18:46.426964+0000 mon.a (mon.0) 4132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[45495]: cluster 2022-04-22T18:18:45.314316+0000 mgr.y (mgr.34107) 1845 : cluster [DBG] pgmap v2892: 137 pgs: 137 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2022-04-22T18:18:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[45495]: cluster 2022-04-22T18:18:45.454563+0000 mon.a (mon.0) 4129 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2022-04-22T18:18:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[45495]: cluster 2022-04-22T18:18:45.466274+0000 mon.a (mon.0) 4130 : cluster [DBG] osdmap e1591: 8 total, 8 up, 8 in 2022-04-22T18:18:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[45495]: audit 2022-04-22T18:18:45.968377+0000 mon.a (mon.0) 4131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[45495]: audit 2022-04-22T18:18:46.426964+0000 mon.a (mon.0) 4132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[46934]: cluster 2022-04-22T18:18:45.314316+0000 mgr.y (mgr.34107) 1845 : cluster [DBG] pgmap v2892: 137 pgs: 137 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2022-04-22T18:18:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[46934]: cluster 2022-04-22T18:18:45.454563+0000 mon.a (mon.0) 4129 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2022-04-22T18:18:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[46934]: cluster 2022-04-22T18:18:45.466274+0000 mon.a (mon.0) 4130 : cluster [DBG] osdmap e1591: 8 total, 8 up, 8 in 2022-04-22T18:18:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[46934]: audit 2022-04-22T18:18:45.968377+0000 mon.a (mon.0) 4131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:46.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:46 smithi115 bash[46934]: audit 2022-04-22T18:18:46.426964+0000 mon.a (mon.0) 4132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:47.520 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroringWatcher.ModeUpdated 2022-04-22T18:18:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:47 smithi121 bash[36902]: cluster 2022-04-22T18:18:46.492042+0000 mon.a (mon.0) 4133 : cluster [DBG] osdmap e1592: 8 total, 8 up, 8 in 2022-04-22T18:18:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:47 smithi121 bash[36902]: audit 2022-04-22T18:18:46.499493+0000 mon.c (mon.1) 111 : audit [INF] from='client.? 172.21.15.115:0/848938393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:47 smithi121 bash[36902]: audit 2022-04-22T18:18:46.501598+0000 mon.a (mon.0) 4134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[46934]: cluster 2022-04-22T18:18:46.492042+0000 mon.a (mon.0) 4133 : cluster [DBG] osdmap e1592: 8 total, 8 up, 8 in 2022-04-22T18:18:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[46934]: audit 2022-04-22T18:18:46.499493+0000 mon.c (mon.1) 111 : audit [INF] from='client.? 172.21.15.115:0/848938393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[46934]: audit 2022-04-22T18:18:46.501598+0000 mon.a (mon.0) 4134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[45495]: cluster 2022-04-22T18:18:46.492042+0000 mon.a (mon.0) 4133 : cluster [DBG] osdmap e1592: 8 total, 8 up, 8 in 2022-04-22T18:18:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[45495]: audit 2022-04-22T18:18:46.499493+0000 mon.c (mon.1) 111 : audit [INF] from='client.? 172.21.15.115:0/848938393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:47 smithi115 bash[45495]: audit 2022-04-22T18:18:46.501598+0000 mon.a (mon.0) 4134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[45495]: cluster 2022-04-22T18:18:47.315453+0000 mgr.y (mgr.34107) 1846 : cluster [DBG] pgmap v2895: 137 pgs: 12 creating+peering, 20 unknown, 105 active+clean; 4.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2022-04-22T18:18:48.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[45495]: audit 2022-04-22T18:18:47.494342+0000 mon.a (mon.0) 4135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[45495]: cluster 2022-04-22T18:18:47.494516+0000 mon.a (mon.0) 4136 : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2022-04-22T18:18:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[45495]: audit 2022-04-22T18:18:48.325440+0000 mon.a (mon.0) 4137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[46934]: cluster 2022-04-22T18:18:47.315453+0000 mgr.y (mgr.34107) 1846 : cluster [DBG] pgmap v2895: 137 pgs: 12 creating+peering, 20 unknown, 105 active+clean; 4.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2022-04-22T18:18:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[46934]: audit 2022-04-22T18:18:47.494342+0000 mon.a (mon.0) 4135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[46934]: cluster 2022-04-22T18:18:47.494516+0000 mon.a (mon.0) 4136 : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2022-04-22T18:18:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:48 smithi115 bash[46934]: audit 2022-04-22T18:18:48.325440+0000 mon.a (mon.0) 4137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:48 smithi121 bash[36902]: cluster 2022-04-22T18:18:47.315453+0000 mgr.y (mgr.34107) 1846 : cluster [DBG] pgmap v2895: 137 pgs: 12 creating+peering, 20 unknown, 105 active+clean; 4.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2022-04-22T18:18:49.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:48 smithi121 bash[36902]: audit 2022-04-22T18:18:47.494342+0000 mon.a (mon.0) 4135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-17","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:49.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:48 smithi121 bash[36902]: cluster 2022-04-22T18:18:47.494516+0000 mon.a (mon.0) 4136 : cluster [DBG] osdmap e1593: 8 total, 8 up, 8 in 2022-04-22T18:18:49.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:48 smithi121 bash[36902]: audit 2022-04-22T18:18:48.325440+0000 mon.a (mon.0) 4137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:49.551 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroringWatcher.ModeUpdated (2031 ms) 2022-04-22T18:18:49.552 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestMirroringWatcher.ImageStatusUpdated 2022-04-22T18:18:49.581 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestMirroringWatcher.ImageStatusUpdated (30 ms) 2022-04-22T18:18:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:49 smithi115 bash[46934]: cluster 2022-04-22T18:18:48.512686+0000 mon.a (mon.0) 4138 : cluster [DBG] osdmap e1594: 8 total, 8 up, 8 in 2022-04-22T18:18:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:49 smithi115 bash[46934]: cluster 2022-04-22T18:18:49.512711+0000 mon.a (mon.0) 4139 : cluster [DBG] osdmap e1595: 8 total, 8 up, 8 in 2022-04-22T18:18:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:49 smithi115 bash[45495]: cluster 2022-04-22T18:18:48.512686+0000 mon.a (mon.0) 4138 : cluster [DBG] osdmap e1594: 8 total, 8 up, 8 in 2022-04-22T18:18:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:49 smithi115 bash[45495]: cluster 2022-04-22T18:18:49.512711+0000 mon.a (mon.0) 4139 : cluster [DBG] osdmap e1595: 8 total, 8 up, 8 in 2022-04-22T18:18:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:49 smithi121 bash[36902]: cluster 2022-04-22T18:18:48.512686+0000 mon.a (mon.0) 4138 : cluster [DBG] osdmap e1594: 8 total, 8 up, 8 in 2022-04-22T18:18:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:49 smithi121 bash[36902]: cluster 2022-04-22T18:18:49.512711+0000 mon.a (mon.0) 4139 : cluster [DBG] osdmap e1595: 8 total, 8 up, 8 in 2022-04-22T18:18:50.547 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from TestMirroringWatcher (2061 ms total) 2022-04-22T18:18:50.547 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:18:50.547 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from TestObjectMap 2022-04-22T18:18:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:50 smithi121 bash[36902]: cluster 2022-04-22T18:18:49.316479+0000 mgr.y (mgr.34107) 1847 : cluster [DBG] pgmap v2898: 137 pgs: 12 creating+peering, 13 unknown, 112 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2022-04-22T18:18:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:50 smithi115 bash[45495]: cluster 2022-04-22T18:18:49.316479+0000 mgr.y (mgr.34107) 1847 : cluster [DBG] pgmap v2898: 137 pgs: 12 creating+peering, 13 unknown, 112 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2022-04-22T18:18:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:50 smithi115 bash[46934]: cluster 2022-04-22T18:18:49.316479+0000 mgr.y (mgr.34107) 1847 : cluster [DBG] pgmap v2898: 137 pgs: 12 creating+peering, 13 unknown, 112 active+clean; 4.6 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2022-04-22T18:18:51.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[45495]: cluster 2022-04-22T18:18:50.533902+0000 mon.a (mon.0) 4140 : cluster [DBG] osdmap e1596: 8 total, 8 up, 8 in 2022-04-22T18:18:51.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[45495]: audit 2022-04-22T18:18:51.021777+0000 mon.a (mon.0) 4141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:51.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[45495]: audit 2022-04-22T18:18:51.430812+0000 mon.a (mon.0) 4142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:51.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[46934]: cluster 2022-04-22T18:18:50.533902+0000 mon.a (mon.0) 4140 : cluster [DBG] osdmap e1596: 8 total, 8 up, 8 in 2022-04-22T18:18:51.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[46934]: audit 2022-04-22T18:18:51.021777+0000 mon.a (mon.0) 4141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:51.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[46934]: audit 2022-04-22T18:18:51.430812+0000 mon.a (mon.0) 4142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:51 smithi121 bash[36902]: cluster 2022-04-22T18:18:50.533902+0000 mon.a (mon.0) 4140 : cluster [DBG] osdmap e1596: 8 total, 8 up, 8 in 2022-04-22T18:18:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:51 smithi121 bash[36902]: audit 2022-04-22T18:18:51.021777+0000 mon.a (mon.0) 4141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:51 smithi121 bash[36902]: audit 2022-04-22T18:18:51.430812+0000 mon.a (mon.0) 4142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:52.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:51.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:52.579 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenCorrupt 2022-04-22T18:18:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:52 smithi121 bash[36902]: cluster 2022-04-22T18:18:51.317433+0000 mgr.y (mgr.34107) 1848 : cluster [DBG] pgmap v2901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:18:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:52 smithi121 bash[36902]: cluster 2022-04-22T18:18:51.556857+0000 mon.a (mon.0) 4143 : cluster [DBG] osdmap e1597: 8 total, 8 up, 8 in 2022-04-22T18:18:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:52 smithi121 bash[36902]: audit 2022-04-22T18:18:51.560864+0000 mon.c (mon.1) 112 : audit [INF] from='client.? 172.21.15.115:0/1808915313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:52 smithi121 bash[36902]: audit 2022-04-22T18:18:51.562482+0000 mon.a (mon.0) 4144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[45495]: cluster 2022-04-22T18:18:51.317433+0000 mgr.y (mgr.34107) 1848 : cluster [DBG] pgmap v2901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:18:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[45495]: cluster 2022-04-22T18:18:51.556857+0000 mon.a (mon.0) 4143 : cluster [DBG] osdmap e1597: 8 total, 8 up, 8 in 2022-04-22T18:18:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[45495]: audit 2022-04-22T18:18:51.560864+0000 mon.c (mon.1) 112 : audit [INF] from='client.? 172.21.15.115:0/1808915313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[45495]: audit 2022-04-22T18:18:51.562482+0000 mon.a (mon.0) 4144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[46934]: cluster 2022-04-22T18:18:51.317433+0000 mgr.y (mgr.34107) 1848 : cluster [DBG] pgmap v2901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:18:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[46934]: cluster 2022-04-22T18:18:51.556857+0000 mon.a (mon.0) 4143 : cluster [DBG] osdmap e1597: 8 total, 8 up, 8 in 2022-04-22T18:18:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[46934]: audit 2022-04-22T18:18:51.560864+0000 mon.c (mon.1) 112 : audit [INF] from='client.? 172.21.15.115:0/1808915313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:52 smithi115 bash[46934]: audit 2022-04-22T18:18:51.562482+0000 mon.a (mon.0) 4144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:53 smithi121 bash[36902]: audit 2022-04-22T18:18:52.558922+0000 mon.a (mon.0) 4145 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:53 smithi121 bash[36902]: cluster 2022-04-22T18:18:52.559149+0000 mon.a (mon.0) 4146 : cluster [DBG] osdmap e1598: 8 total, 8 up, 8 in 2022-04-22T18:18:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:53 smithi121 bash[36902]: audit 2022-04-22T18:18:53.328402+0000 mon.a (mon.0) 4147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:18:53.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:18:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[45495]: audit 2022-04-22T18:18:52.558922+0000 mon.a (mon.0) 4145 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[45495]: cluster 2022-04-22T18:18:52.559149+0000 mon.a (mon.0) 4146 : cluster [DBG] osdmap e1598: 8 total, 8 up, 8 in 2022-04-22T18:18:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[45495]: audit 2022-04-22T18:18:53.328402+0000 mon.a (mon.0) 4147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[46934]: audit 2022-04-22T18:18:52.558922+0000 mon.a (mon.0) 4145 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-18","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[46934]: cluster 2022-04-22T18:18:52.559149+0000 mon.a (mon.0) 4146 : cluster [DBG] osdmap e1598: 8 total, 8 up, 8 in 2022-04-22T18:18:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:53 smithi115 bash[46934]: audit 2022-04-22T18:18:53.328402+0000 mon.a (mon.0) 4147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:54.609 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:18:54.610 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenCorrupt (2030 ms) 2022-04-22T18:18:54.610 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenTooSmall 2022-04-22T18:18:54.621 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:18:54.621 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenTooSmall (12 ms) 2022-04-22T18:18:54.622 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestObjectMap.InvalidateFlagOnDisk 2022-04-22T18:18:54.625 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:18:54.626 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestObjectMap.InvalidateFlagOnDisk (4 ms) 2022-04-22T18:18:54.626 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall 2022-04-22T18:18:54.630 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:18:54.630 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall (5 ms) 2022-04-22T18:18:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:54 smithi121 bash[36902]: cluster 2022-04-22T18:18:53.318480+0000 mgr.y (mgr.34107) 1849 : cluster [DBG] pgmap v2904: 137 pgs: 9 creating+peering, 12 unknown, 116 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:55.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:54 smithi121 bash[36902]: cluster 2022-04-22T18:18:53.585072+0000 mon.a (mon.0) 4148 : cluster [DBG] osdmap e1599: 8 total, 8 up, 8 in 2022-04-22T18:18:55.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:54 smithi115 bash[45495]: cluster 2022-04-22T18:18:53.318480+0000 mgr.y (mgr.34107) 1849 : cluster [DBG] pgmap v2904: 137 pgs: 9 creating+peering, 12 unknown, 116 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:55.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:54 smithi115 bash[45495]: cluster 2022-04-22T18:18:53.585072+0000 mon.a (mon.0) 4148 : cluster [DBG] osdmap e1599: 8 total, 8 up, 8 in 2022-04-22T18:18:55.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:54 smithi115 bash[46934]: cluster 2022-04-22T18:18:53.318480+0000 mgr.y (mgr.34107) 1849 : cluster [DBG] pgmap v2904: 137 pgs: 9 creating+peering, 12 unknown, 116 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:55.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:54 smithi115 bash[46934]: cluster 2022-04-22T18:18:53.585072+0000 mon.a (mon.0) 4148 : cluster [DBG] osdmap e1599: 8 total, 8 up, 8 in 2022-04-22T18:18:55.620 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 4 tests from TestObjectMap (2051 ms total) 2022-04-22T18:18:55.620 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:18:55.620 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 1 test from TestOperations 2022-04-22T18:18:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:55 smithi121 bash[36902]: cluster 2022-04-22T18:18:54.588714+0000 mon.a (mon.0) 4149 : cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2022-04-22T18:18:56.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:55 smithi115 bash[45495]: cluster 2022-04-22T18:18:54.588714+0000 mon.a (mon.0) 4149 : cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2022-04-22T18:18:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:55 smithi115 bash[46934]: cluster 2022-04-22T18:18:54.588714+0000 mon.a (mon.0) 4149 : cluster [DBG] osdmap e1600: 8 total, 8 up, 8 in 2022-04-22T18:18:57.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[45495]: cluster 2022-04-22T18:18:55.319563+0000 mgr.y (mgr.34107) 1850 : cluster [DBG] pgmap v2907: 137 pgs: 9 creating+peering, 5 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[45495]: cluster 2022-04-22T18:18:55.610374+0000 mon.a (mon.0) 4150 : cluster [DBG] osdmap e1601: 8 total, 8 up, 8 in 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[45495]: audit 2022-04-22T18:18:56.110096+0000 mon.a (mon.0) 4151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[45495]: audit 2022-04-22T18:18:56.433980+0000 mon.a (mon.0) 4152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[46934]: cluster 2022-04-22T18:18:55.319563+0000 mgr.y (mgr.34107) 1850 : cluster [DBG] pgmap v2907: 137 pgs: 9 creating+peering, 5 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[46934]: cluster 2022-04-22T18:18:55.610374+0000 mon.a (mon.0) 4150 : cluster [DBG] osdmap e1601: 8 total, 8 up, 8 in 2022-04-22T18:18:57.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[46934]: audit 2022-04-22T18:18:56.110096+0000 mon.a (mon.0) 4151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:57.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:56 smithi115 bash[46934]: audit 2022-04-22T18:18:56.433980+0000 mon.a (mon.0) 4152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:56 smithi121 bash[36902]: cluster 2022-04-22T18:18:55.319563+0000 mgr.y (mgr.34107) 1850 : cluster [DBG] pgmap v2907: 137 pgs: 9 creating+peering, 5 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:56 smithi121 bash[36902]: cluster 2022-04-22T18:18:55.610374+0000 mon.a (mon.0) 4150 : cluster [DBG] osdmap e1601: 8 total, 8 up, 8 in 2022-04-22T18:18:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:56 smithi121 bash[36902]: audit 2022-04-22T18:18:56.110096+0000 mon.a (mon.0) 4151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:18:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:56 smithi121 bash[36902]: audit 2022-04-22T18:18:56.433980+0000 mon.a (mon.0) 4152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:18:57.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: cluster 2022-04-22T18:18:56.621031+0000 mon.a (mon.0) 4153 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2022-04-22T18:18:57.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:56.627830+0000 mon.c (mon.1) 113 : audit [INF] from='client.? 172.21.15.115:0/1499758291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:57.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:56.628624+0000 mon.a (mon.0) 4154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:57.643 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestOperations.DisableJournalingCorrupt 2022-04-22T18:18:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:56.987982+0000 mon.a (mon.0) 4155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:18:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:56.990815+0000 mon.a (mon.0) 4156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:18:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:56.992422+0000 mon.a (mon.0) 4157 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:18:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:57.295608+0000 mon.a (mon.0) 4158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:57 smithi121 bash[36902]: audit 2022-04-22T18:18:57.325502+0000 mon.a (mon.0) 4159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:58.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: cluster 2022-04-22T18:18:56.621031+0000 mon.a (mon.0) 4153 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2022-04-22T18:18:58.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:56.627830+0000 mon.c (mon.1) 113 : audit [INF] from='client.? 172.21.15.115:0/1499758291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:56.628624+0000 mon.a (mon.0) 4154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:56.987982+0000 mon.a (mon.0) 4155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:56.990815+0000 mon.a (mon.0) 4156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:56.992422+0000 mon.a (mon.0) 4157 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:57.295608+0000 mon.a (mon.0) 4158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[45495]: audit 2022-04-22T18:18:57.325502+0000 mon.a (mon.0) 4159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: cluster 2022-04-22T18:18:56.621031+0000 mon.a (mon.0) 4153 : cluster [DBG] osdmap e1602: 8 total, 8 up, 8 in 2022-04-22T18:18:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:56.627830+0000 mon.c (mon.1) 113 : audit [INF] from='client.? 172.21.15.115:0/1499758291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:56.628624+0000 mon.a (mon.0) 4154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:56.987982+0000 mon.a (mon.0) 4155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:56.990815+0000 mon.a (mon.0) 4156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:56.992422+0000 mon.a (mon.0) 4157 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:57.295608+0000 mon.a (mon.0) 4158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:57 smithi115 bash[46934]: audit 2022-04-22T18:18:57.325502+0000 mon.a (mon.0) 4159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:18:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:58 smithi121 bash[36902]: cluster 2022-04-22T18:18:57.320695+0000 mgr.y (mgr.34107) 1851 : cluster [DBG] pgmap v2910: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:58 smithi121 bash[36902]: audit 2022-04-22T18:18:57.625484+0000 mon.a (mon.0) 4160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:58 smithi121 bash[36902]: cluster 2022-04-22T18:18:57.625732+0000 mon.a (mon.0) 4161 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2022-04-22T18:18:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:58 smithi121 bash[36902]: audit 2022-04-22T18:18:58.331418+0000 mon.a (mon.0) 4162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:59.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[45495]: cluster 2022-04-22T18:18:57.320695+0000 mgr.y (mgr.34107) 1851 : cluster [DBG] pgmap v2910: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:59.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[45495]: audit 2022-04-22T18:18:57.625484+0000 mon.a (mon.0) 4160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[45495]: cluster 2022-04-22T18:18:57.625732+0000 mon.a (mon.0) 4161 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2022-04-22T18:18:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[45495]: audit 2022-04-22T18:18:58.331418+0000 mon.a (mon.0) 4162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[46934]: cluster 2022-04-22T18:18:57.320695+0000 mgr.y (mgr.34107) 1851 : cluster [DBG] pgmap v2910: 137 pgs: 8 creating+peering, 24 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:18:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[46934]: audit 2022-04-22T18:18:57.625484+0000 mon.a (mon.0) 4160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-19","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:18:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[46934]: cluster 2022-04-22T18:18:57.625732+0000 mon.a (mon.0) 4161 : cluster [DBG] osdmap e1603: 8 total, 8 up, 8 in 2022-04-22T18:18:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:58 smithi115 bash[46934]: audit 2022-04-22T18:18:58.331418+0000 mon.a (mon.0) 4162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:18:59.674 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:18:59.675 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestOperations.DisableJournalingCorrupt (2031 ms) 2022-04-22T18:19:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:18:59 smithi121 bash[36902]: cluster 2022-04-22T18:18:58.642092+0000 mon.a (mon.0) 4163 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2022-04-22T18:19:00.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:18:59 smithi115 bash[45495]: cluster 2022-04-22T18:18:58.642092+0000 mon.a (mon.0) 4163 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2022-04-22T18:19:00.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:18:59 smithi115 bash[46934]: cluster 2022-04-22T18:18:58.642092+0000 mon.a (mon.0) 4163 : cluster [DBG] osdmap e1604: 8 total, 8 up, 8 in 2022-04-22T18:19:00.668 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 1 test from TestOperations (2031 ms total) 2022-04-22T18:19:00.668 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:00.669 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from TestTrash 2022-04-22T18:19:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:00 smithi121 bash[36902]: cluster 2022-04-22T18:18:59.321747+0000 mgr.y (mgr.34107) 1852 : cluster [DBG] pgmap v2913: 137 pgs: 8 creating+peering, 14 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:00 smithi121 bash[36902]: cluster 2022-04-22T18:18:59.651234+0000 mon.a (mon.0) 4164 : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2022-04-22T18:19:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:00 smithi121 bash[36902]: audit 2022-04-22T18:18:59.681128+0000 mon.a (mon.0) 4165 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[45495]: cluster 2022-04-22T18:18:59.321747+0000 mgr.y (mgr.34107) 1852 : cluster [DBG] pgmap v2913: 137 pgs: 8 creating+peering, 14 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[45495]: cluster 2022-04-22T18:18:59.651234+0000 mon.a (mon.0) 4164 : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2022-04-22T18:19:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[45495]: audit 2022-04-22T18:18:59.681128+0000 mon.a (mon.0) 4165 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[46934]: cluster 2022-04-22T18:18:59.321747+0000 mgr.y (mgr.34107) 1852 : cluster [DBG] pgmap v2913: 137 pgs: 8 creating+peering, 14 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[46934]: cluster 2022-04-22T18:18:59.651234+0000 mon.a (mon.0) 4164 : cluster [DBG] osdmap e1605: 8 total, 8 up, 8 in 2022-04-22T18:19:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:00 smithi115 bash[46934]: audit 2022-04-22T18:18:59.681128+0000 mon.a (mon.0) 4165 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:01 smithi121 bash[36902]: cluster 2022-04-22T18:19:00.654652+0000 mon.a (mon.0) 4166 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2022-04-22T18:19:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:01 smithi121 bash[36902]: audit 2022-04-22T18:19:01.166863+0000 mon.a (mon.0) 4167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:01 smithi121 bash[36902]: cluster 2022-04-22T18:19:01.322638+0000 mgr.y (mgr.34107) 1853 : cluster [DBG] pgmap v2916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:01 smithi121 bash[36902]: audit 2022-04-22T18:19:01.401036+0000 mon.a (mon.0) 4168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:01 smithi121 bash[36902]: audit 2022-04-22T18:19:01.437860+0000 mon.a (mon.0) 4169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:01.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[45495]: cluster 2022-04-22T18:19:00.654652+0000 mon.a (mon.0) 4166 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2022-04-22T18:19:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[45495]: audit 2022-04-22T18:19:01.166863+0000 mon.a (mon.0) 4167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:02.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[45495]: cluster 2022-04-22T18:19:01.322638+0000 mgr.y (mgr.34107) 1853 : cluster [DBG] pgmap v2916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[45495]: audit 2022-04-22T18:19:01.401036+0000 mon.a (mon.0) 4168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[45495]: audit 2022-04-22T18:19:01.437860+0000 mon.a (mon.0) 4169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[46934]: cluster 2022-04-22T18:19:00.654652+0000 mon.a (mon.0) 4166 : cluster [DBG] osdmap e1606: 8 total, 8 up, 8 in 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[46934]: audit 2022-04-22T18:19:01.166863+0000 mon.a (mon.0) 4167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[46934]: cluster 2022-04-22T18:19:01.322638+0000 mgr.y (mgr.34107) 1853 : cluster [DBG] pgmap v2916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[46934]: audit 2022-04-22T18:19:01.401036+0000 mon.a (mon.0) 4168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:01 smithi115 bash[46934]: audit 2022-04-22T18:19:01.437860+0000 mon.a (mon.0) 4169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:02.248 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:19:02.584 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:19:02.584 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (42m) 1s ago 55m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:19:02.584 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (55m) 65s ago 55m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (44m) 1s ago 56m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (43m) 65s ago 56m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (51m) 65s ago 61m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (52m) 1s ago 63m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (50m) 1s ago 63m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (50m) 65s ago 61m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (50m) 1s ago 61m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (43m) 1s ago 56m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (43m) 65s ago 56m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (48m) 1s ago 60m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:19:02.585 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (48m) 1s ago 60m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (47m) 1s ago 59m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (47m) 1s ago 59m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (46m) 65s ago 58m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (45m) 65s ago 58m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (45m) 65s ago 57m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (44m) 65s ago 57m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:19:02.586 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (42m) 65s ago 56m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:19:02.674 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestTrash.UserRemovingSource 2022-04-22T18:19:03.004 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:03.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:19:03.006 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:19:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:02 smithi121 bash[36902]: cluster 2022-04-22T18:19:01.663774+0000 mon.a (mon.0) 4170 : cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2022-04-22T18:19:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:02 smithi121 bash[36902]: audit 2022-04-22T18:19:01.666703+0000 mon.a (mon.0) 4171 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:02 smithi121 bash[36902]: audit 2022-04-22T18:19:01.726408+0000 mon.a (mon.0) 4172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:02 smithi121 bash[36902]: audit 2022-04-22T18:19:02.226497+0000 mgr.y (mgr.34107) 1854 : audit [DBG] from='client.37209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:02 smithi121 bash[36902]: audit 2022-04-22T18:19:02.563763+0000 mgr.y (mgr.34107) 1855 : audit [DBG] from='client.47149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[45495]: cluster 2022-04-22T18:19:01.663774+0000 mon.a (mon.0) 4170 : cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2022-04-22T18:19:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[45495]: audit 2022-04-22T18:19:01.666703+0000 mon.a (mon.0) 4171 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[45495]: audit 2022-04-22T18:19:01.726408+0000 mon.a (mon.0) 4172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[45495]: audit 2022-04-22T18:19:02.226497+0000 mgr.y (mgr.34107) 1854 : audit [DBG] from='client.37209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[45495]: audit 2022-04-22T18:19:02.563763+0000 mgr.y (mgr.34107) 1855 : audit [DBG] from='client.47149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[46934]: cluster 2022-04-22T18:19:01.663774+0000 mon.a (mon.0) 4170 : cluster [DBG] osdmap e1607: 8 total, 8 up, 8 in 2022-04-22T18:19:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[46934]: audit 2022-04-22T18:19:01.666703+0000 mon.a (mon.0) 4171 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[46934]: audit 2022-04-22T18:19:01.726408+0000 mon.a (mon.0) 4172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[46934]: audit 2022-04-22T18:19:02.226497+0000 mgr.y (mgr.34107) 1854 : audit [DBG] from='client.37209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:02 smithi115 bash[46934]: audit 2022-04-22T18:19:02.563763+0000 mgr.y (mgr.34107) 1855 : audit [DBG] from='client.47149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:03 smithi121 bash[36902]: audit 2022-04-22T18:19:02.665989+0000 mon.a (mon.0) 4173 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:03 smithi121 bash[36902]: cluster 2022-04-22T18:19:02.666280+0000 mon.a (mon.0) 4174 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2022-04-22T18:19:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:03 smithi121 bash[36902]: audit 2022-04-22T18:19:02.998453+0000 mon.c (mon.1) 114 : audit [DBG] from='client.? 172.21.15.115:0/1925823878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:03 smithi121 bash[36902]: cluster 2022-04-22T18:19:03.323805+0000 mgr.y (mgr.34107) 1856 : cluster [DBG] pgmap v2919: 137 pgs: 11 creating+peering, 11 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:03 smithi121 bash[36902]: audit 2022-04-22T18:19:03.334305+0000 mon.a (mon.0) 4175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:03.682Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[45495]: audit 2022-04-22T18:19:02.665989+0000 mon.a (mon.0) 4173 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[45495]: cluster 2022-04-22T18:19:02.666280+0000 mon.a (mon.0) 4174 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[45495]: audit 2022-04-22T18:19:02.998453+0000 mon.c (mon.1) 114 : audit [DBG] from='client.? 172.21.15.115:0/1925823878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[45495]: cluster 2022-04-22T18:19:03.323805+0000 mgr.y (mgr.34107) 1856 : cluster [DBG] pgmap v2919: 137 pgs: 11 creating+peering, 11 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[45495]: audit 2022-04-22T18:19:03.334305+0000 mon.a (mon.0) 4175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[46934]: audit 2022-04-22T18:19:02.665989+0000 mon.a (mon.0) 4173 : audit [INF] from='client.? 172.21.15.115:0/635423145' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-20","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[46934]: cluster 2022-04-22T18:19:02.666280+0000 mon.a (mon.0) 4174 : cluster [DBG] osdmap e1608: 8 total, 8 up, 8 in 2022-04-22T18:19:04.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[46934]: audit 2022-04-22T18:19:02.998453+0000 mon.c (mon.1) 114 : audit [DBG] from='client.? 172.21.15.115:0/1925823878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[46934]: cluster 2022-04-22T18:19:03.323805+0000 mgr.y (mgr.34107) 1856 : cluster [DBG] pgmap v2919: 137 pgs: 11 creating+peering, 11 unknown, 115 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:03 smithi115 bash[46934]: audit 2022-04-22T18:19:03.334305+0000 mon.a (mon.0) 4175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:04.720 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:04.720 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestTrash.UserRemovingSource (2045 ms) 2022-04-22T18:19:04.720 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestTrash.RestoreMirroringSource 2022-04-22T18:19:04.725 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:04.726 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestTrash.RestoreMirroringSource (6 ms) 2022-04-22T18:19:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:04 smithi121 bash[36902]: cluster 2022-04-22T18:19:03.689416+0000 mon.a (mon.0) 4176 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2022-04-22T18:19:05.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:04 smithi115 bash[45495]: cluster 2022-04-22T18:19:03.689416+0000 mon.a (mon.0) 4176 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2022-04-22T18:19:05.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:04 smithi115 bash[46934]: cluster 2022-04-22T18:19:03.689416+0000 mon.a (mon.0) 4176 : cluster [DBG] osdmap e1609: 8 total, 8 up, 8 in 2022-04-22T18:19:05.724 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 2 tests from TestTrash (2051 ms total) 2022-04-22T18:19:05.724 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:05.725 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 3 tests from TestJournalEntries 2022-04-22T18:19:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:05 smithi121 bash[36902]: cluster 2022-04-22T18:19:04.691321+0000 mon.a (mon.0) 4177 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2022-04-22T18:19:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:05 smithi121 bash[36902]: cluster 2022-04-22T18:19:05.324975+0000 mgr.y (mgr.34107) 1857 : cluster [DBG] pgmap v2922: 137 pgs: 11 creating+peering, 3 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:05 smithi115 bash[46934]: cluster 2022-04-22T18:19:04.691321+0000 mon.a (mon.0) 4177 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2022-04-22T18:19:06.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:05 smithi115 bash[46934]: cluster 2022-04-22T18:19:05.324975+0000 mgr.y (mgr.34107) 1857 : cluster [DBG] pgmap v2922: 137 pgs: 11 creating+peering, 3 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:06.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:05 smithi115 bash[45495]: cluster 2022-04-22T18:19:04.691321+0000 mon.a (mon.0) 4177 : cluster [DBG] osdmap e1610: 8 total, 8 up, 8 in 2022-04-22T18:19:06.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:05 smithi115 bash[45495]: cluster 2022-04-22T18:19:05.324975+0000 mgr.y (mgr.34107) 1857 : cluster [DBG] pgmap v2922: 137 pgs: 11 creating+peering, 3 unknown, 123 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:06 smithi121 bash[36902]: cluster 2022-04-22T18:19:05.715971+0000 mon.a (mon.0) 4178 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2022-04-22T18:19:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:06 smithi121 bash[36902]: audit 2022-04-22T18:19:06.213938+0000 mon.a (mon.0) 4179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:06 smithi121 bash[36902]: audit 2022-04-22T18:19:06.441181+0000 mon.a (mon.0) 4180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:07.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[45495]: cluster 2022-04-22T18:19:05.715971+0000 mon.a (mon.0) 4178 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2022-04-22T18:19:07.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[45495]: audit 2022-04-22T18:19:06.213938+0000 mon.a (mon.0) 4179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:07.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[45495]: audit 2022-04-22T18:19:06.441181+0000 mon.a (mon.0) 4180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[46934]: cluster 2022-04-22T18:19:05.715971+0000 mon.a (mon.0) 4178 : cluster [DBG] osdmap e1611: 8 total, 8 up, 8 in 2022-04-22T18:19:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[46934]: audit 2022-04-22T18:19:06.213938+0000 mon.a (mon.0) 4179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:06 smithi115 bash[46934]: audit 2022-04-22T18:19:06.441181+0000 mon.a (mon.0) 4180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:07.742 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalEntries.AioWrite 2022-04-22T18:19:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:07 smithi121 bash[36902]: cluster 2022-04-22T18:19:06.719876+0000 mon.a (mon.0) 4181 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2022-04-22T18:19:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:07 smithi121 bash[36902]: audit 2022-04-22T18:19:06.725387+0000 mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.115:0/1464724528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:07 smithi121 bash[36902]: audit 2022-04-22T18:19:06.732802+0000 mon.a (mon.0) 4182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:07 smithi121 bash[36902]: cluster 2022-04-22T18:19:07.326127+0000 mgr.y (mgr.34107) 1858 : cluster [DBG] pgmap v2925: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:08.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[45495]: cluster 2022-04-22T18:19:06.719876+0000 mon.a (mon.0) 4181 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2022-04-22T18:19:08.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[45495]: audit 2022-04-22T18:19:06.725387+0000 mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.115:0/1464724528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[45495]: audit 2022-04-22T18:19:06.732802+0000 mon.a (mon.0) 4182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[45495]: cluster 2022-04-22T18:19:07.326127+0000 mgr.y (mgr.34107) 1858 : cluster [DBG] pgmap v2925: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[46934]: cluster 2022-04-22T18:19:06.719876+0000 mon.a (mon.0) 4181 : cluster [DBG] osdmap e1612: 8 total, 8 up, 8 in 2022-04-22T18:19:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[46934]: audit 2022-04-22T18:19:06.725387+0000 mon.c (mon.1) 115 : audit [INF] from='client.? 172.21.15.115:0/1464724528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[46934]: audit 2022-04-22T18:19:06.732802+0000 mon.a (mon.0) 4182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:07 smithi115 bash[46934]: cluster 2022-04-22T18:19:07.326127+0000 mgr.y (mgr.34107) 1858 : cluster [DBG] pgmap v2925: 137 pgs: 7 creating+peering, 25 unknown, 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:08 smithi121 bash[36902]: audit 2022-04-22T18:19:07.733241+0000 mon.a (mon.0) 4183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:08 smithi121 bash[36902]: cluster 2022-04-22T18:19:07.733436+0000 mon.a (mon.0) 4184 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2022-04-22T18:19:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:08 smithi121 bash[36902]: audit 2022-04-22T18:19:08.337548+0000 mon.a (mon.0) 4185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:09.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[45495]: audit 2022-04-22T18:19:07.733241+0000 mon.a (mon.0) 4183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:09.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[45495]: cluster 2022-04-22T18:19:07.733436+0000 mon.a (mon.0) 4184 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2022-04-22T18:19:09.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[45495]: audit 2022-04-22T18:19:08.337548+0000 mon.a (mon.0) 4185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[46934]: audit 2022-04-22T18:19:07.733241+0000 mon.a (mon.0) 4183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-21","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[46934]: cluster 2022-04-22T18:19:07.733436+0000 mon.a (mon.0) 4184 : cluster [DBG] osdmap e1613: 8 total, 8 up, 8 in 2022-04-22T18:19:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:08 smithi115 bash[46934]: audit 2022-04-22T18:19:08.337548+0000 mon.a (mon.0) 4185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:09.778 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:09.778 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalEntries.AioWrite (2036 ms) 2022-04-22T18:19:09.778 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalEntries.AioDiscard 2022-04-22T18:19:09.783 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:09.783 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalEntries.AioDiscard (5 ms) 2022-04-22T18:19:09.783 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalEntries.AioFlush 2022-04-22T18:19:09.792 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:09.792 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalEntries.AioFlush (9 ms) 2022-04-22T18:19:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:09 smithi121 bash[36902]: cluster 2022-04-22T18:19:08.739519+0000 mon.a (mon.0) 4186 : cluster [DBG] osdmap e1614: 8 total, 8 up, 8 in 2022-04-22T18:19:10.122 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:09 smithi121 bash[36902]: cluster 2022-04-22T18:19:09.327218+0000 mgr.y (mgr.34107) 1859 : cluster [DBG] pgmap v2928: 137 pgs: 7 creating+peering, 12 unknown, 118 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:10.122 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:09 smithi121 bash[36902]: audit 2022-04-22T18:19:09.705377+0000 mon.a (mon.0) 4187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:10.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[45495]: cluster 2022-04-22T18:19:08.739519+0000 mon.a (mon.0) 4186 : cluster [DBG] osdmap e1614: 8 total, 8 up, 8 in 2022-04-22T18:19:10.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[45495]: cluster 2022-04-22T18:19:09.327218+0000 mgr.y (mgr.34107) 1859 : cluster [DBG] pgmap v2928: 137 pgs: 7 creating+peering, 12 unknown, 118 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:10.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[45495]: audit 2022-04-22T18:19:09.705377+0000 mon.a (mon.0) 4187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:10.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[46934]: cluster 2022-04-22T18:19:08.739519+0000 mon.a (mon.0) 4186 : cluster [DBG] osdmap e1614: 8 total, 8 up, 8 in 2022-04-22T18:19:10.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[46934]: cluster 2022-04-22T18:19:09.327218+0000 mgr.y (mgr.34107) 1859 : cluster [DBG] pgmap v2928: 137 pgs: 7 creating+peering, 12 unknown, 118 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:09 smithi115 bash[46934]: audit 2022-04-22T18:19:09.705377+0000 mon.a (mon.0) 4187 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:10.775 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 3 tests from TestJournalEntries (2050 ms total) 2022-04-22T18:19:10.775 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:10.776 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 16 tests from TestJournalReplay 2022-04-22T18:19:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:10 smithi121 bash[36902]: cluster 2022-04-22T18:19:09.760129+0000 mon.a (mon.0) 4188 : cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2022-04-22T18:19:11.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:10 smithi115 bash[46934]: cluster 2022-04-22T18:19:09.760129+0000 mon.a (mon.0) 4188 : cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2022-04-22T18:19:11.117 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:10 smithi115 bash[45495]: cluster 2022-04-22T18:19:09.760129+0000 mon.a (mon.0) 4188 : cluster [DBG] osdmap e1615: 8 total, 8 up, 8 in 2022-04-22T18:19:12.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:11.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:12.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[46934]: cluster 2022-04-22T18:19:10.767418+0000 mon.a (mon.0) 4189 : cluster [DBG] osdmap e1616: 8 total, 8 up, 8 in 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[46934]: audit 2022-04-22T18:19:11.261830+0000 mon.a (mon.0) 4190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[46934]: cluster 2022-04-22T18:19:11.328152+0000 mgr.y (mgr.34107) 1860 : cluster [DBG] pgmap v2931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[46934]: audit 2022-04-22T18:19:11.444736+0000 mon.a (mon.0) 4191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[45495]: cluster 2022-04-22T18:19:10.767418+0000 mon.a (mon.0) 4189 : cluster [DBG] osdmap e1616: 8 total, 8 up, 8 in 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[45495]: audit 2022-04-22T18:19:11.261830+0000 mon.a (mon.0) 4190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[45495]: cluster 2022-04-22T18:19:11.328152+0000 mgr.y (mgr.34107) 1860 : cluster [DBG] pgmap v2931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:11 smithi115 bash[45495]: audit 2022-04-22T18:19:11.444736+0000 mon.a (mon.0) 4191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:11 smithi121 bash[36902]: cluster 2022-04-22T18:19:10.767418+0000 mon.a (mon.0) 4189 : cluster [DBG] osdmap e1616: 8 total, 8 up, 8 in 2022-04-22T18:19:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:11 smithi121 bash[36902]: audit 2022-04-22T18:19:11.261830+0000 mon.a (mon.0) 4190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:11 smithi121 bash[36902]: cluster 2022-04-22T18:19:11.328152+0000 mgr.y (mgr.34107) 1860 : cluster [DBG] pgmap v2931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:11 smithi121 bash[36902]: audit 2022-04-22T18:19:11.444736+0000 mon.a (mon.0) 4191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:12.796 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.AioDiscardEvent 2022-04-22T18:19:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:12 smithi121 bash[36902]: cluster 2022-04-22T18:19:11.777077+0000 mon.a (mon.0) 4192 : cluster [DBG] osdmap e1617: 8 total, 8 up, 8 in 2022-04-22T18:19:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:12 smithi121 bash[36902]: audit 2022-04-22T18:19:11.779375+0000 mon.a (mon.0) 4193 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:13.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:12 smithi115 bash[46934]: cluster 2022-04-22T18:19:11.777077+0000 mon.a (mon.0) 4192 : cluster [DBG] osdmap e1617: 8 total, 8 up, 8 in 2022-04-22T18:19:13.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:12 smithi115 bash[46934]: audit 2022-04-22T18:19:11.779375+0000 mon.a (mon.0) 4193 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:13.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:12 smithi115 bash[45495]: cluster 2022-04-22T18:19:11.777077+0000 mon.a (mon.0) 4192 : cluster [DBG] osdmap e1617: 8 total, 8 up, 8 in 2022-04-22T18:19:13.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:12 smithi115 bash[45495]: audit 2022-04-22T18:19:11.779375+0000 mon.a (mon.0) 4193 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:14.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:13.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[45495]: audit 2022-04-22T18:19:12.783021+0000 mon.a (mon.0) 4194 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[45495]: cluster 2022-04-22T18:19:12.783486+0000 mon.a (mon.0) 4195 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[45495]: cluster 2022-04-22T18:19:13.329212+0000 mgr.y (mgr.34107) 1861 : cluster [DBG] pgmap v2934: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[45495]: audit 2022-04-22T18:19:13.340859+0000 mon.a (mon.0) 4196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[45495]: cluster 2022-04-22T18:19:13.785586+0000 mon.a (mon.0) 4197 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[46934]: audit 2022-04-22T18:19:12.783021+0000 mon.a (mon.0) 4194 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:14.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[46934]: cluster 2022-04-22T18:19:12.783486+0000 mon.a (mon.0) 4195 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2022-04-22T18:19:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[46934]: cluster 2022-04-22T18:19:13.329212+0000 mgr.y (mgr.34107) 1861 : cluster [DBG] pgmap v2934: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[46934]: audit 2022-04-22T18:19:13.340859+0000 mon.a (mon.0) 4196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:13 smithi115 bash[46934]: cluster 2022-04-22T18:19:13.785586+0000 mon.a (mon.0) 4197 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2022-04-22T18:19:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:13 smithi121 bash[36902]: audit 2022-04-22T18:19:12.783021+0000 mon.a (mon.0) 4194 : audit [INF] from='client.? 172.21.15.115:0/2681245073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi115-68765-22","app": "rados","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:13 smithi121 bash[36902]: cluster 2022-04-22T18:19:12.783486+0000 mon.a (mon.0) 4195 : cluster [DBG] osdmap e1618: 8 total, 8 up, 8 in 2022-04-22T18:19:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:13 smithi121 bash[36902]: cluster 2022-04-22T18:19:13.329212+0000 mgr.y (mgr.34107) 1861 : cluster [DBG] pgmap v2934: 137 pgs: 8 creating+peering, 18 unknown, 111 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:13 smithi121 bash[36902]: audit 2022-04-22T18:19:13.340859+0000 mon.a (mon.0) 4196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:13 smithi121 bash[36902]: cluster 2022-04-22T18:19:13.785586+0000 mon.a (mon.0) 4197 : cluster [DBG] osdmap e1619: 8 total, 8 up, 8 in 2022-04-22T18:19:14.818 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.818 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.AioDiscardEvent (2023 ms) 2022-04-22T18:19:14.818 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.AioWriteEvent 2022-04-22T18:19:14.823 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.823 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.AioWriteEvent (5 ms) 2022-04-22T18:19:14.823 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.AioFlushEvent 2022-04-22T18:19:14.835 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.835 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.AioFlushEvent (12 ms) 2022-04-22T18:19:14.836 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapCreate 2022-04-22T18:19:14.840 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.840 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapCreate (5 ms) 2022-04-22T18:19:14.840 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapProtect 2022-04-22T18:19:14.845 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.845 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapProtect (5 ms) 2022-04-22T18:19:14.845 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapUnprotect 2022-04-22T18:19:14.852 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.852 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapUnprotect (7 ms) 2022-04-22T18:19:14.852 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapRename 2022-04-22T18:19:14.857 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapRename (5 ms) 2022-04-22T18:19:14.857 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapRollback 2022-04-22T18:19:14.864 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.864 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapRollback (7 ms) 2022-04-22T18:19:14.864 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.SnapRemove 2022-04-22T18:19:14.870 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.870 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.SnapRemove (6 ms) 2022-04-22T18:19:14.870 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.Rename 2022-04-22T18:19:14.875 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.875 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.Rename (5 ms) 2022-04-22T18:19:14.875 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.Resize 2022-04-22T18:19:14.881 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.881 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.Resize (6 ms) 2022-04-22T18:19:14.881 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.Flatten 2022-04-22T18:19:14.886 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.886 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.Flatten (5 ms) 2022-04-22T18:19:14.886 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.UpdateFeatures 2022-04-22T18:19:14.892 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.892 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.UpdateFeatures (6 ms) 2022-04-22T18:19:14.892 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.MetadataSet 2022-04-22T18:19:14.898 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.898 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.MetadataSet (6 ms) 2022-04-22T18:19:14.898 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.MetadataRemove 2022-04-22T18:19:14.904 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.905 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.MetadataRemove (6 ms) 2022-04-22T18:19:14.905 INFO:tasks.workunit.client.0.smithi115.stdout:[ RUN ] TestJournalReplay.ObjectPosition 2022-04-22T18:19:14.910 INFO:tasks.workunit.client.0.smithi115.stdout:SKIPPING 2022-04-22T18:19:14.910 INFO:tasks.workunit.client.0.smithi115.stdout:[ OK ] TestJournalReplay.ObjectPosition (6 ms) 2022-04-22T18:19:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:14 smithi121 bash[36902]: audit 2022-04-22T18:19:13.885916+0000 mon.a (mon.0) 4198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:19:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:14 smithi121 bash[36902]: audit 2022-04-22T18:19:13.891971+0000 mon.a (mon.0) 4199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:19:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:14 smithi121 bash[36902]: cluster 2022-04-22T18:19:14.790630+0000 mon.a (mon.0) 4200 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2022-04-22T18:19:15.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[45495]: audit 2022-04-22T18:19:13.885916+0000 mon.a (mon.0) 4198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:19:15.424 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[45495]: audit 2022-04-22T18:19:13.891971+0000 mon.a (mon.0) 4199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:19:15.424 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[45495]: cluster 2022-04-22T18:19:14.790630+0000 mon.a (mon.0) 4200 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2022-04-22T18:19:15.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[46934]: audit 2022-04-22T18:19:13.885916+0000 mon.a (mon.0) 4198 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:19:15.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[46934]: audit 2022-04-22T18:19:13.891971+0000 mon.a (mon.0) 4199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:19:15.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:14 smithi115 bash[46934]: cluster 2022-04-22T18:19:14.790630+0000 mon.a (mon.0) 4200 : cluster [DBG] osdmap e1620: 8 total, 8 up, 8 in 2022-04-22T18:19:15.827 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] 16 tests from TestJournalReplay (2115 ms total) 2022-04-22T18:19:15.827 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout:[----------] Global test environment tear-down 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout:[==========] 371 tests from 15 test suites ran. (674189 ms total) 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout:[ PASSED ] 371 tests. 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout: YOU HAVE 2 DISABLED TESTS 2022-04-22T18:19:15.828 INFO:tasks.workunit.client.0.smithi115.stdout: 2022-04-22T18:19:15.842 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T18:19:15.842 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T18:19:15.851 INFO:tasks.workunit:Stopping ['rbd/test_librbd.sh'] on client.0... 2022-04-22T18:19:15.852 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-04-22T18:19:16.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:15 smithi115 bash[45495]: cluster 2022-04-22T18:19:15.330271+0000 mgr.y (mgr.34107) 1862 : cluster [DBG] pgmap v2937: 137 pgs: 8 creating+peering, 9 unknown, 120 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2022-04-22T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:15 smithi115 bash[46934]: cluster 2022-04-22T18:19:15.330271+0000 mgr.y (mgr.34107) 1862 : cluster [DBG] pgmap v2937: 137 pgs: 8 creating+peering, 9 unknown, 120 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2022-04-22T18:19:16.074 DEBUG:teuthology.parallel:result is None 2022-04-22T18:19:16.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:19:16.083 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:19:16.084 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:19:16.088 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:19:16.088 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:19:16.088 INFO:teuthology.task.print:**** done end test_rbd_api.yaml 2022-04-22T18:19:16.088 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:19:16.088 INFO:teuthology.task.print:**** done start test_rbd_python.yaml 2022-04-22T18:19:16.089 INFO:teuthology.task.full_sequential:In full_sequential, running task workunit... 2022-04-22T18:19:16.089 INFO:tasks.workunit:Pulling workunits from ref pacific 2022-04-22T18:19:16.089 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-22T18:19:16.090 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:19:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-22T18:19:16.138 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-04-22T18:19:16.139 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:19:16.143 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-04-22T18:19:16.144 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-04-22T18:19:16.148 INFO:tasks.workunit:timeout=3h 2022-04-22T18:19:16.148 INFO:tasks.workunit:cleanup=True 2022-04-22T18:19:16.149 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch pacific https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2022-04-22T18:19:16.200 INFO:tasks.workunit.client.0.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-04-22T18:19:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:15 smithi121 bash[36902]: cluster 2022-04-22T18:19:15.330271+0000 mgr.y (mgr.34107) 1862 : cluster [DBG] pgmap v2937: 137 pgs: 8 creating+peering, 9 unknown, 120 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2022-04-22T18:19:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:16 smithi121 bash[36902]: cluster 2022-04-22T18:19:15.816704+0000 mon.a (mon.0) 4201 : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2022-04-22T18:19:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:16 smithi121 bash[36902]: audit 2022-04-22T18:19:15.966292+0000 mon.a (mon.0) 4202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:16 smithi121 bash[36902]: audit 2022-04-22T18:19:16.448249+0000 mon.a (mon.0) 4203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:17.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[45495]: cluster 2022-04-22T18:19:15.816704+0000 mon.a (mon.0) 4201 : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2022-04-22T18:19:17.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[45495]: audit 2022-04-22T18:19:15.966292+0000 mon.a (mon.0) 4202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:17.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[45495]: audit 2022-04-22T18:19:16.448249+0000 mon.a (mon.0) 4203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[46934]: cluster 2022-04-22T18:19:15.816704+0000 mon.a (mon.0) 4201 : cluster [DBG] osdmap e1621: 8 total, 8 up, 8 in 2022-04-22T18:19:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[46934]: audit 2022-04-22T18:19:15.966292+0000 mon.a (mon.0) 4202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:17.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:16 smithi115 bash[46934]: audit 2022-04-22T18:19:16.448249+0000 mon.a (mon.0) 4203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:17 smithi121 bash[36902]: cluster 2022-04-22T18:19:17.331335+0000 mgr.y (mgr.34107) 1863 : cluster [DBG] pgmap v2939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:17 smithi115 bash[46934]: cluster 2022-04-22T18:19:17.331335+0000 mgr.y (mgr.34107) 1863 : cluster [DBG] pgmap v2939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:17 smithi115 bash[45495]: cluster 2022-04-22T18:19:17.331335+0000 mgr.y (mgr.34107) 1863 : cluster [DBG] pgmap v2939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:18 smithi121 bash[36902]: audit 2022-04-22T18:19:18.343718+0000 mon.a (mon.0) 4204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:19.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:18 smithi115 bash[45495]: audit 2022-04-22T18:19:18.343718+0000 mon.a (mon.0) 4204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:19.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:18 smithi115 bash[46934]: audit 2022-04-22T18:19:18.343718+0000 mon.a (mon.0) 4204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:19 smithi121 bash[36902]: cluster 2022-04-22T18:19:19.332456+0000 mgr.y (mgr.34107) 1864 : cluster [DBG] pgmap v2940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:20.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:19 smithi115 bash[45495]: cluster 2022-04-22T18:19:19.332456+0000 mgr.y (mgr.34107) 1864 : cluster [DBG] pgmap v2940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:20.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:19 smithi115 bash[46934]: cluster 2022-04-22T18:19:19.332456+0000 mgr.y (mgr.34107) 1864 : cluster [DBG] pgmap v2940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:21.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:21 smithi115 bash[45495]: audit 2022-04-22T18:19:20.969608+0000 mon.a (mon.0) 4205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:21.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:21 smithi115 bash[46934]: audit 2022-04-22T18:19:20.969608+0000 mon.a (mon.0) 4205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:21 smithi121 bash[36902]: audit 2022-04-22T18:19:20.969608+0000 mon.a (mon.0) 4205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:22.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:21.799Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:22 smithi121 bash[36902]: cluster 2022-04-22T18:19:21.333501+0000 mgr.y (mgr.34107) 1865 : cluster [DBG] pgmap v2941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:22 smithi121 bash[36902]: audit 2022-04-22T18:19:21.451766+0000 mon.a (mon.0) 4206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:22 smithi115 bash[45495]: cluster 2022-04-22T18:19:21.333501+0000 mgr.y (mgr.34107) 1865 : cluster [DBG] pgmap v2941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:22 smithi115 bash[45495]: audit 2022-04-22T18:19:21.451766+0000 mon.a (mon.0) 4206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:22 smithi115 bash[46934]: cluster 2022-04-22T18:19:21.333501+0000 mgr.y (mgr.34107) 1865 : cluster [DBG] pgmap v2941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:22 smithi115 bash[46934]: audit 2022-04-22T18:19:21.451766+0000 mon.a (mon.0) 4206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:23.411 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-04-22T18:19:23.416 INFO:tasks.workunit.client.0.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-22T18:19:23.419 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T18:19:23.419 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-22T18:19:23.480 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-22T18:19:23.545 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-22T18:19:23.600 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-22T18:19:23.601 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T18:19:23.601 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-22T18:19:23.658 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-22T18:19:23.664 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-04-22T18:19:23.665 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-04-22T18:19:23.672 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2022-04-22T18:19:23.673 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2022-04-22T18:19:23.673 DEBUG:teuthology.orchestra.run.smithi115:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-04-22T18:19:23.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:23 smithi115 bash[45495]: audit 2022-04-22T18:19:23.346305+0000 mon.a (mon.0) 4207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:23.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:23 smithi115 bash[46934]: audit 2022-04-22T18:19:23.346305+0000 mon.a (mon.0) 4207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:23.700 INFO:tasks.workunit.client.0.smithi115.stderr:+ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-04-22T18:19:23.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2022-04-22T18:19:23.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ [ -n ] 2022-04-22T18:19:23.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ python3 -m nose -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2022-04-22T18:19:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:23 smithi121 bash[36902]: audit 2022-04-22T18:19:23.346305+0000 mon.a (mon.0) 4207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:23.682Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:24 smithi121 bash[36902]: cluster 2022-04-22T18:19:23.334636+0000 mgr.y (mgr.34107) 1866 : cluster [DBG] pgmap v2942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:24 smithi115 bash[45495]: cluster 2022-04-22T18:19:23.334636+0000 mgr.y (mgr.34107) 1866 : cluster [DBG] pgmap v2942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:24 smithi115 bash[46934]: cluster 2022-04-22T18:19:23.334636+0000 mgr.y (mgr.34107) 1866 : cluster [DBG] pgmap v2942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:25 smithi121 bash[36902]: audit 2022-04-22T18:19:24.946857+0000 mon.a (mon.0) 4208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:25 smithi121 bash[36902]: cluster 2022-04-22T18:19:25.335818+0000 mgr.y (mgr.34107) 1867 : cluster [DBG] pgmap v2943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:25 smithi121 bash[36902]: cluster 2022-04-22T18:19:25.412184+0000 mon.a (mon.0) 4209 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2022-04-22T18:19:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:25 smithi121 bash[36902]: audit 2022-04-22T18:19:25.414200+0000 mon.c (mon.1) 116 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:25 smithi121 bash[36902]: audit 2022-04-22T18:19:25.415540+0000 mon.a (mon.0) 4210 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[45495]: audit 2022-04-22T18:19:24.946857+0000 mon.a (mon.0) 4208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[45495]: cluster 2022-04-22T18:19:25.335818+0000 mgr.y (mgr.34107) 1867 : cluster [DBG] pgmap v2943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[45495]: cluster 2022-04-22T18:19:25.412184+0000 mon.a (mon.0) 4209 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2022-04-22T18:19:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[45495]: audit 2022-04-22T18:19:25.414200+0000 mon.c (mon.1) 116 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:26.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[45495]: audit 2022-04-22T18:19:25.415540+0000 mon.a (mon.0) 4210 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[46934]: audit 2022-04-22T18:19:24.946857+0000 mon.a (mon.0) 4208 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[46934]: cluster 2022-04-22T18:19:25.335818+0000 mgr.y (mgr.34107) 1867 : cluster [DBG] pgmap v2943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[46934]: cluster 2022-04-22T18:19:25.412184+0000 mon.a (mon.0) 4209 : cluster [DBG] osdmap e1622: 8 total, 8 up, 8 in 2022-04-22T18:19:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[46934]: audit 2022-04-22T18:19:25.414200+0000 mon.c (mon.1) 116 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:25 smithi115 bash[46934]: audit 2022-04-22T18:19:25.415540+0000 mon.a (mon.0) 4210 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]: dispatch 2022-04-22T18:19:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:26 smithi121 bash[36902]: audit 2022-04-22T18:19:25.972365+0000 mon.a (mon.0) 4211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:26 smithi121 bash[36902]: audit 2022-04-22T18:19:26.413774+0000 mon.a (mon.0) 4212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:26 smithi121 bash[36902]: cluster 2022-04-22T18:19:26.414432+0000 mon.a (mon.0) 4213 : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2022-04-22T18:19:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:26 smithi121 bash[36902]: audit 2022-04-22T18:19:26.455127+0000 mon.a (mon.0) 4214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:27.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[45495]: audit 2022-04-22T18:19:25.972365+0000 mon.a (mon.0) 4211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:27.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[45495]: audit 2022-04-22T18:19:26.413774+0000 mon.a (mon.0) 4212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:27.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[45495]: cluster 2022-04-22T18:19:26.414432+0000 mon.a (mon.0) 4213 : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2022-04-22T18:19:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[45495]: audit 2022-04-22T18:19:26.455127+0000 mon.a (mon.0) 4214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[46934]: audit 2022-04-22T18:19:25.972365+0000 mon.a (mon.0) 4211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[46934]: audit 2022-04-22T18:19:26.413774+0000 mon.a (mon.0) 4212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-1","app": "rbd","yes_i_really_mean_it": true}]': finished 2022-04-22T18:19:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[46934]: cluster 2022-04-22T18:19:26.414432+0000 mon.a (mon.0) 4213 : cluster [DBG] osdmap e1623: 8 total, 8 up, 8 in 2022-04-22T18:19:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:26 smithi115 bash[46934]: audit 2022-04-22T18:19:26.455127+0000 mon.a (mon.0) 4214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:28 smithi121 bash[36902]: cluster 2022-04-22T18:19:27.336924+0000 mgr.y (mgr.34107) 1868 : cluster [DBG] pgmap v2946: 137 pgs: 3 creating+peering, 20 unknown, 114 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:28 smithi121 bash[36902]: cluster 2022-04-22T18:19:27.415699+0000 mon.a (mon.0) 4215 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2022-04-22T18:19:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:28 smithi121 bash[36902]: audit 2022-04-22T18:19:28.348997+0000 mon.a (mon.0) 4216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[45495]: cluster 2022-04-22T18:19:27.336924+0000 mgr.y (mgr.34107) 1868 : cluster [DBG] pgmap v2946: 137 pgs: 3 creating+peering, 20 unknown, 114 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[45495]: cluster 2022-04-22T18:19:27.415699+0000 mon.a (mon.0) 4215 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2022-04-22T18:19:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[45495]: audit 2022-04-22T18:19:28.348997+0000 mon.a (mon.0) 4216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[46934]: cluster 2022-04-22T18:19:27.336924+0000 mgr.y (mgr.34107) 1868 : cluster [DBG] pgmap v2946: 137 pgs: 3 creating+peering, 20 unknown, 114 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2022-04-22T18:19:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[46934]: cluster 2022-04-22T18:19:27.415699+0000 mon.a (mon.0) 4215 : cluster [DBG] osdmap e1624: 8 total, 8 up, 8 in 2022-04-22T18:19:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:28 smithi115 bash[46934]: audit 2022-04-22T18:19:28.348997+0000 mon.a (mon.0) 4216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:29 smithi121 bash[36902]: cluster 2022-04-22T18:19:28.420479+0000 mon.a (mon.0) 4217 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2022-04-22T18:19:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:29 smithi115 bash[46934]: cluster 2022-04-22T18:19:28.420479+0000 mon.a (mon.0) 4217 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2022-04-22T18:19:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:29 smithi115 bash[45495]: cluster 2022-04-22T18:19:28.420479+0000 mon.a (mon.0) 4217 : cluster [DBG] osdmap e1625: 8 total, 8 up, 8 in 2022-04-22T18:19:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:30 smithi121 bash[36902]: cluster 2022-04-22T18:19:29.337949+0000 mgr.y (mgr.34107) 1869 : cluster [DBG] pgmap v2949: 137 pgs: 3 creating+peering, 8 unknown, 126 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2022-04-22T18:19:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:30 smithi121 bash[36902]: cluster 2022-04-22T18:19:29.440227+0000 mon.a (mon.0) 4218 : cluster [DBG] osdmap e1626: 8 total, 8 up, 8 in 2022-04-22T18:19:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:30 smithi115 bash[46934]: cluster 2022-04-22T18:19:29.337949+0000 mgr.y (mgr.34107) 1869 : cluster [DBG] pgmap v2949: 137 pgs: 3 creating+peering, 8 unknown, 126 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2022-04-22T18:19:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:30 smithi115 bash[46934]: cluster 2022-04-22T18:19:29.440227+0000 mon.a (mon.0) 4218 : cluster [DBG] osdmap e1626: 8 total, 8 up, 8 in 2022-04-22T18:19:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:30 smithi115 bash[45495]: cluster 2022-04-22T18:19:29.337949+0000 mgr.y (mgr.34107) 1869 : cluster [DBG] pgmap v2949: 137 pgs: 3 creating+peering, 8 unknown, 126 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2022-04-22T18:19:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:30 smithi115 bash[45495]: cluster 2022-04-22T18:19:29.440227+0000 mon.a (mon.0) 4218 : cluster [DBG] osdmap e1626: 8 total, 8 up, 8 in 2022-04-22T18:19:31.582 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_basic ... ok 2022-04-22T18:19:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:31 smithi121 bash[36902]: audit 2022-04-22T18:19:30.975502+0000 mon.a (mon.0) 4219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:31 smithi115 bash[45495]: audit 2022-04-22T18:19:30.975502+0000 mon.a (mon.0) 4219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:31.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:31 smithi115 bash[46934]: audit 2022-04-22T18:19:30.975502+0000 mon.a (mon.0) 4219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:32.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:31.798Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:32 smithi121 bash[36902]: cluster 2022-04-22T18:19:31.339156+0000 mgr.y (mgr.34107) 1870 : cluster [DBG] pgmap v2951: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 6.3 KiB/s wr, 152 op/s 2022-04-22T18:19:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:32 smithi121 bash[36902]: cluster 2022-04-22T18:19:31.445416+0000 mon.a (mon.0) 4220 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2022-04-22T18:19:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:32 smithi121 bash[36902]: audit 2022-04-22T18:19:31.458298+0000 mon.a (mon.0) 4221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[46934]: cluster 2022-04-22T18:19:31.339156+0000 mgr.y (mgr.34107) 1870 : cluster [DBG] pgmap v2951: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 6.3 KiB/s wr, 152 op/s 2022-04-22T18:19:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[46934]: cluster 2022-04-22T18:19:31.445416+0000 mon.a (mon.0) 4220 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2022-04-22T18:19:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[46934]: audit 2022-04-22T18:19:31.458298+0000 mon.a (mon.0) 4221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[45495]: cluster 2022-04-22T18:19:31.339156+0000 mgr.y (mgr.34107) 1870 : cluster [DBG] pgmap v2951: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 6.3 KiB/s wr, 152 op/s 2022-04-22T18:19:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[45495]: cluster 2022-04-22T18:19:31.445416+0000 mon.a (mon.0) 4220 : cluster [DBG] osdmap e1627: 8 total, 8 up, 8 in 2022-04-22T18:19:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:32 smithi115 bash[45495]: audit 2022-04-22T18:19:31.458298+0000 mon.a (mon.0) 4221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:33.381 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:19:33.626 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_drops_cache ... ok 2022-04-22T18:19:33.729 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (42m) 32s ago 56m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (56m) 97s ago 56m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (44m) 32s ago 57m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (44m) 97s ago 57m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (51m) 97s ago 61m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:19:33.730 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (52m) 32s ago 63m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (51m) 32s ago 64m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (50m) 97s ago 62m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (50m) 32s ago 62m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (43m) 32s ago 56m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (43m) 97s ago 56m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:19:33.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (49m) 32s ago 61m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (48m) 32s ago 60m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (48m) 32s ago 60m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (47m) 32s ago 59m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (47m) 97s ago 59m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (46m) 97s ago 58m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (45m) 97s ago 58m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:19:33.732 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (45m) 97s ago 57m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:19:33.733 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (43m) 97s ago 56m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:19:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:33 smithi121 bash[36902]: cluster 2022-04-22T18:19:32.463859+0000 mon.a (mon.0) 4222 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2022-04-22T18:19:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:33 smithi121 bash[36902]: audit 2022-04-22T18:19:33.351878+0000 mon.a (mon.0) 4223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:33 smithi115 bash[46934]: cluster 2022-04-22T18:19:32.463859+0000 mon.a (mon.0) 4222 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2022-04-22T18:19:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:33 smithi115 bash[46934]: audit 2022-04-22T18:19:33.351878+0000 mon.a (mon.0) 4223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:33.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:33.682Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:33 smithi115 bash[45495]: cluster 2022-04-22T18:19:32.463859+0000 mon.a (mon.0) 4222 : cluster [DBG] osdmap e1628: 8 total, 8 up, 8 in 2022-04-22T18:19:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:33 smithi115 bash[45495]: audit 2022-04-22T18:19:33.351878+0000 mon.a (mon.0) 4223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:19:34.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:19:34.142 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:19:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:34 smithi121 bash[36902]: cluster 2022-04-22T18:19:33.340198+0000 mgr.y (mgr.34107) 1871 : cluster [DBG] pgmap v2954: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 294 KiB/s rd, 20 KiB/s wr, 389 op/s 2022-04-22T18:19:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:34 smithi121 bash[36902]: audit 2022-04-22T18:19:33.361052+0000 mgr.y (mgr.34107) 1872 : audit [DBG] from='client.47191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:34 smithi121 bash[36902]: cluster 2022-04-22T18:19:33.474283+0000 mon.a (mon.0) 4224 : cluster [DBG] osdmap e1629: 8 total, 8 up, 8 in 2022-04-22T18:19:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:34 smithi121 bash[36902]: audit 2022-04-22T18:19:34.134485+0000 mon.a (mon.0) 4225 : audit [DBG] from='client.? 172.21.15.115:0/2113639154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[45495]: cluster 2022-04-22T18:19:33.340198+0000 mgr.y (mgr.34107) 1871 : cluster [DBG] pgmap v2954: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 294 KiB/s rd, 20 KiB/s wr, 389 op/s 2022-04-22T18:19:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[45495]: audit 2022-04-22T18:19:33.361052+0000 mgr.y (mgr.34107) 1872 : audit [DBG] from='client.47191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[45495]: cluster 2022-04-22T18:19:33.474283+0000 mon.a (mon.0) 4224 : cluster [DBG] osdmap e1629: 8 total, 8 up, 8 in 2022-04-22T18:19:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[45495]: audit 2022-04-22T18:19:34.134485+0000 mon.a (mon.0) 4225 : audit [DBG] from='client.? 172.21.15.115:0/2113639154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[46934]: cluster 2022-04-22T18:19:33.340198+0000 mgr.y (mgr.34107) 1871 : cluster [DBG] pgmap v2954: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 294 KiB/s rd, 20 KiB/s wr, 389 op/s 2022-04-22T18:19:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[46934]: audit 2022-04-22T18:19:33.361052+0000 mgr.y (mgr.34107) 1872 : audit [DBG] from='client.47191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[46934]: cluster 2022-04-22T18:19:33.474283+0000 mon.a (mon.0) 4224 : cluster [DBG] osdmap e1629: 8 total, 8 up, 8 in 2022-04-22T18:19:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:34 smithi115 bash[46934]: audit 2022-04-22T18:19:34.134485+0000 mon.a (mon.0) 4225 : audit [DBG] from='client.? 172.21.15.115:0/2113639154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:19:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:35 smithi121 bash[36902]: audit 2022-04-22T18:19:33.703097+0000 mgr.y (mgr.34107) 1873 : audit [DBG] from='client.47197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:35 smithi121 bash[36902]: cluster 2022-04-22T18:19:34.487849+0000 mon.a (mon.0) 4226 : cluster [DBG] osdmap e1630: 8 total, 8 up, 8 in 2022-04-22T18:19:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:35 smithi115 bash[45495]: audit 2022-04-22T18:19:33.703097+0000 mgr.y (mgr.34107) 1873 : audit [DBG] from='client.47197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:35 smithi115 bash[45495]: cluster 2022-04-22T18:19:34.487849+0000 mon.a (mon.0) 4226 : cluster [DBG] osdmap e1630: 8 total, 8 up, 8 in 2022-04-22T18:19:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:35 smithi115 bash[46934]: audit 2022-04-22T18:19:33.703097+0000 mgr.y (mgr.34107) 1873 : audit [DBG] from='client.47197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:19:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:35 smithi115 bash[46934]: cluster 2022-04-22T18:19:34.487849+0000 mon.a (mon.0) 4226 : cluster [DBG] osdmap e1630: 8 total, 8 up, 8 in 2022-04-22T18:19:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:36 smithi121 bash[36902]: cluster 2022-04-22T18:19:35.341308+0000 mgr.y (mgr.34107) 1874 : cluster [DBG] pgmap v2957: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 25 KiB/s wr, 439 op/s 2022-04-22T18:19:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:36 smithi121 bash[36902]: cluster 2022-04-22T18:19:35.500705+0000 mon.a (mon.0) 4227 : cluster [DBG] osdmap e1631: 8 total, 8 up, 8 in 2022-04-22T18:19:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:36 smithi121 bash[36902]: audit 2022-04-22T18:19:35.978700+0000 mon.a (mon.0) 4228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:36 smithi121 bash[36902]: cluster 2022-04-22T18:19:36.329032+0000 mon.a (mon.0) 4229 : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2022-04-22T18:19:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:36 smithi121 bash[36902]: audit 2022-04-22T18:19:36.460962+0000 mon.a (mon.0) 4230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[45495]: cluster 2022-04-22T18:19:35.341308+0000 mgr.y (mgr.34107) 1874 : cluster [DBG] pgmap v2957: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 25 KiB/s wr, 439 op/s 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[45495]: cluster 2022-04-22T18:19:35.500705+0000 mon.a (mon.0) 4227 : cluster [DBG] osdmap e1631: 8 total, 8 up, 8 in 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[45495]: audit 2022-04-22T18:19:35.978700+0000 mon.a (mon.0) 4228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[45495]: cluster 2022-04-22T18:19:36.329032+0000 mon.a (mon.0) 4229 : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[45495]: audit 2022-04-22T18:19:36.460962+0000 mon.a (mon.0) 4230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[46934]: cluster 2022-04-22T18:19:35.341308+0000 mgr.y (mgr.34107) 1874 : cluster [DBG] pgmap v2957: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 25 KiB/s wr, 439 op/s 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[46934]: cluster 2022-04-22T18:19:35.500705+0000 mon.a (mon.0) 4227 : cluster [DBG] osdmap e1631: 8 total, 8 up, 8 in 2022-04-22T18:19:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[46934]: audit 2022-04-22T18:19:35.978700+0000 mon.a (mon.0) 4228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[46934]: cluster 2022-04-22T18:19:36.329032+0000 mon.a (mon.0) 4229 : cluster [DBG] osdmap e1632: 8 total, 8 up, 8 in 2022-04-22T18:19:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:36 smithi115 bash[46934]: audit 2022-04-22T18:19:36.460962+0000 mon.a (mon.0) 4230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:37.493 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_errors ... ok 2022-04-22T18:19:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:38 smithi121 bash[36902]: cluster 2022-04-22T18:19:37.342485+0000 mgr.y (mgr.34107) 1875 : cluster [DBG] pgmap v2961: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 12 KiB/s wr, 221 op/s 2022-04-22T18:19:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:38 smithi121 bash[36902]: cluster 2022-04-22T18:19:37.346082+0000 mon.a (mon.0) 4231 : cluster [DBG] osdmap e1633: 8 total, 8 up, 8 in 2022-04-22T18:19:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:38 smithi115 bash[45495]: cluster 2022-04-22T18:19:37.342485+0000 mgr.y (mgr.34107) 1875 : cluster [DBG] pgmap v2961: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 12 KiB/s wr, 221 op/s 2022-04-22T18:19:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:38 smithi115 bash[45495]: cluster 2022-04-22T18:19:37.346082+0000 mon.a (mon.0) 4231 : cluster [DBG] osdmap e1633: 8 total, 8 up, 8 in 2022-04-22T18:19:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:38 smithi115 bash[46934]: cluster 2022-04-22T18:19:37.342485+0000 mgr.y (mgr.34107) 1875 : cluster [DBG] pgmap v2961: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 12 KiB/s wr, 221 op/s 2022-04-22T18:19:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:38 smithi115 bash[46934]: cluster 2022-04-22T18:19:37.346082+0000 mon.a (mon.0) 4231 : cluster [DBG] osdmap e1633: 8 total, 8 up, 8 in 2022-04-22T18:19:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:39 smithi121 bash[36902]: cluster 2022-04-22T18:19:38.354126+0000 mon.a (mon.0) 4232 : cluster [DBG] osdmap e1634: 8 total, 8 up, 8 in 2022-04-22T18:19:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:39 smithi121 bash[36902]: audit 2022-04-22T18:19:38.355698+0000 mon.a (mon.0) 4233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:39 smithi115 bash[45495]: cluster 2022-04-22T18:19:38.354126+0000 mon.a (mon.0) 4232 : cluster [DBG] osdmap e1634: 8 total, 8 up, 8 in 2022-04-22T18:19:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:39 smithi115 bash[45495]: audit 2022-04-22T18:19:38.355698+0000 mon.a (mon.0) 4233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:39 smithi115 bash[46934]: cluster 2022-04-22T18:19:38.354126+0000 mon.a (mon.0) 4232 : cluster [DBG] osdmap e1634: 8 total, 8 up, 8 in 2022-04-22T18:19:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:39 smithi115 bash[46934]: audit 2022-04-22T18:19:38.355698+0000 mon.a (mon.0) 4233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:40.537 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_larger_order ... ok 2022-04-22T18:19:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:40 smithi121 bash[36902]: cluster 2022-04-22T18:19:39.343376+0000 mgr.y (mgr.34107) 1876 : cluster [DBG] pgmap v2963: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 238 KiB/s rd, 16 KiB/s wr, 315 op/s 2022-04-22T18:19:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:40 smithi121 bash[36902]: audit 2022-04-22T18:19:39.972371+0000 mon.a (mon.0) 4234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:40 smithi115 bash[45495]: cluster 2022-04-22T18:19:39.343376+0000 mgr.y (mgr.34107) 1876 : cluster [DBG] pgmap v2963: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 238 KiB/s rd, 16 KiB/s wr, 315 op/s 2022-04-22T18:19:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:40 smithi115 bash[45495]: audit 2022-04-22T18:19:39.972371+0000 mon.a (mon.0) 4234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:40 smithi115 bash[46934]: cluster 2022-04-22T18:19:39.343376+0000 mgr.y (mgr.34107) 1876 : cluster [DBG] pgmap v2963: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 238 KiB/s rd, 16 KiB/s wr, 315 op/s 2022-04-22T18:19:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:40 smithi115 bash[46934]: audit 2022-04-22T18:19:39.972371+0000 mon.a (mon.0) 4234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:19:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:41 smithi121 bash[36902]: cluster 2022-04-22T18:19:40.374565+0000 mon.a (mon.0) 4235 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2022-04-22T18:19:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:41 smithi121 bash[36902]: audit 2022-04-22T18:19:40.981874+0000 mon.a (mon.0) 4236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:41 smithi121 bash[36902]: cluster 2022-04-22T18:19:41.329562+0000 mon.a (mon.0) 4237 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2022-04-22T18:19:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[45495]: cluster 2022-04-22T18:19:40.374565+0000 mon.a (mon.0) 4235 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2022-04-22T18:19:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[45495]: audit 2022-04-22T18:19:40.981874+0000 mon.a (mon.0) 4236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:41.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[45495]: cluster 2022-04-22T18:19:41.329562+0000 mon.a (mon.0) 4237 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2022-04-22T18:19:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[46934]: cluster 2022-04-22T18:19:40.374565+0000 mon.a (mon.0) 4235 : cluster [DBG] osdmap e1635: 8 total, 8 up, 8 in 2022-04-22T18:19:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[46934]: audit 2022-04-22T18:19:40.981874+0000 mon.a (mon.0) 4236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[46934]: cluster 2022-04-22T18:19:41.329562+0000 mon.a (mon.0) 4237 : cluster [DBG] osdmap e1636: 8 total, 8 up, 8 in 2022-04-22T18:19:42.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:41.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:42 smithi121 bash[36902]: cluster 2022-04-22T18:19:41.343907+0000 mgr.y (mgr.34107) 1877 : cluster [DBG] pgmap v2966: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 19 KiB/s wr, 398 op/s 2022-04-22T18:19:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:42 smithi121 bash[36902]: audit 2022-04-22T18:19:41.464411+0000 mon.a (mon.0) 4238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:42 smithi115 bash[45495]: cluster 2022-04-22T18:19:41.343907+0000 mgr.y (mgr.34107) 1877 : cluster [DBG] pgmap v2966: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 19 KiB/s wr, 398 op/s 2022-04-22T18:19:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:42 smithi115 bash[45495]: audit 2022-04-22T18:19:41.464411+0000 mon.a (mon.0) 4238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:42 smithi115 bash[46934]: cluster 2022-04-22T18:19:41.343907+0000 mgr.y (mgr.34107) 1877 : cluster [DBG] pgmap v2966: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 19 KiB/s wr, 398 op/s 2022-04-22T18:19:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:42 smithi115 bash[46934]: audit 2022-04-22T18:19:41.464411+0000 mon.a (mon.0) 4238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:43.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:43 smithi115 bash[45495]: cluster 2022-04-22T18:19:42.381775+0000 mon.a (mon.0) 4239 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2022-04-22T18:19:43.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:43 smithi115 bash[45495]: audit 2022-04-22T18:19:43.359578+0000 mon.a (mon.0) 4240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:43.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:43 smithi115 bash[46934]: cluster 2022-04-22T18:19:42.381775+0000 mon.a (mon.0) 4239 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2022-04-22T18:19:43.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:43 smithi115 bash[46934]: audit 2022-04-22T18:19:43.359578+0000 mon.a (mon.0) 4240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:43 smithi121 bash[36902]: cluster 2022-04-22T18:19:42.381775+0000 mon.a (mon.0) 4239 : cluster [DBG] osdmap e1637: 8 total, 8 up, 8 in 2022-04-22T18:19:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:43 smithi121 bash[36902]: audit 2022-04-22T18:19:43.359578+0000 mon.a (mon.0) 4240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:44.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:43.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:44.558 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_multi_level ... ok 2022-04-22T18:19:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:44 smithi121 bash[36902]: cluster 2022-04-22T18:19:43.344921+0000 mgr.y (mgr.34107) 1878 : cluster [DBG] pgmap v2968: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 346 KiB/s rd, 26 KiB/s wr, 470 op/s 2022-04-22T18:19:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:44 smithi121 bash[36902]: cluster 2022-04-22T18:19:43.395746+0000 mon.a (mon.0) 4241 : cluster [DBG] osdmap e1638: 8 total, 8 up, 8 in 2022-04-22T18:19:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:44 smithi115 bash[45495]: cluster 2022-04-22T18:19:43.344921+0000 mgr.y (mgr.34107) 1878 : cluster [DBG] pgmap v2968: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 346 KiB/s rd, 26 KiB/s wr, 470 op/s 2022-04-22T18:19:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:44 smithi115 bash[45495]: cluster 2022-04-22T18:19:43.395746+0000 mon.a (mon.0) 4241 : cluster [DBG] osdmap e1638: 8 total, 8 up, 8 in 2022-04-22T18:19:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:44 smithi115 bash[46934]: cluster 2022-04-22T18:19:43.344921+0000 mgr.y (mgr.34107) 1878 : cluster [DBG] pgmap v2968: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 346 KiB/s rd, 26 KiB/s wr, 470 op/s 2022-04-22T18:19:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:44 smithi115 bash[46934]: cluster 2022-04-22T18:19:43.395746+0000 mon.a (mon.0) 4241 : cluster [DBG] osdmap e1638: 8 total, 8 up, 8 in 2022-04-22T18:19:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:45 smithi121 bash[36902]: cluster 2022-04-22T18:19:44.413350+0000 mon.a (mon.0) 4242 : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2022-04-22T18:19:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:45 smithi115 bash[45495]: cluster 2022-04-22T18:19:44.413350+0000 mon.a (mon.0) 4242 : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2022-04-22T18:19:45.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:45 smithi115 bash[46934]: cluster 2022-04-22T18:19:44.413350+0000 mon.a (mon.0) 4242 : cluster [DBG] osdmap e1639: 8 total, 8 up, 8 in 2022-04-22T18:19:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:46 smithi121 bash[36902]: cluster 2022-04-22T18:19:45.346031+0000 mgr.y (mgr.34107) 1879 : cluster [DBG] pgmap v2971: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 27 KiB/s wr, 449 op/s 2022-04-22T18:19:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:46 smithi121 bash[36902]: cluster 2022-04-22T18:19:45.417477+0000 mon.a (mon.0) 4243 : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2022-04-22T18:19:46.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:46 smithi121 bash[36902]: audit 2022-04-22T18:19:45.984935+0000 mon.a (mon.0) 4244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:46.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:46 smithi121 bash[36902]: cluster 2022-04-22T18:19:46.330372+0000 mon.a (mon.0) 4245 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2022-04-22T18:19:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[45495]: cluster 2022-04-22T18:19:45.346031+0000 mgr.y (mgr.34107) 1879 : cluster [DBG] pgmap v2971: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 27 KiB/s wr, 449 op/s 2022-04-22T18:19:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[45495]: cluster 2022-04-22T18:19:45.417477+0000 mon.a (mon.0) 4243 : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[45495]: audit 2022-04-22T18:19:45.984935+0000 mon.a (mon.0) 4244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[45495]: cluster 2022-04-22T18:19:46.330372+0000 mon.a (mon.0) 4245 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[46934]: cluster 2022-04-22T18:19:45.346031+0000 mgr.y (mgr.34107) 1879 : cluster [DBG] pgmap v2971: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 327 KiB/s rd, 27 KiB/s wr, 449 op/s 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[46934]: cluster 2022-04-22T18:19:45.417477+0000 mon.a (mon.0) 4243 : cluster [DBG] osdmap e1640: 8 total, 8 up, 8 in 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[46934]: audit 2022-04-22T18:19:45.984935+0000 mon.a (mon.0) 4244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:46 smithi115 bash[46934]: cluster 2022-04-22T18:19:46.330372+0000 mon.a (mon.0) 4245 : cluster [DBG] osdmap e1641: 8 total, 8 up, 8 in 2022-04-22T18:19:47.583 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_smaller_order ... ok 2022-04-22T18:19:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:47 smithi121 bash[36902]: audit 2022-04-22T18:19:46.467814+0000 mon.a (mon.0) 4246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:47.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:47 smithi115 bash[45495]: audit 2022-04-22T18:19:46.467814+0000 mon.a (mon.0) 4246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:47 smithi115 bash[46934]: audit 2022-04-22T18:19:46.467814+0000 mon.a (mon.0) 4246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:48 smithi121 bash[36902]: cluster 2022-04-22T18:19:47.347164+0000 mgr.y (mgr.34107) 1880 : cluster [DBG] pgmap v2974: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 212 KiB/s rd, 16 KiB/s wr, 293 op/s 2022-04-22T18:19:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:48 smithi121 bash[36902]: cluster 2022-04-22T18:19:47.429472+0000 mon.a (mon.0) 4247 : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2022-04-22T18:19:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:48 smithi121 bash[36902]: audit 2022-04-22T18:19:48.362375+0000 mon.a (mon.0) 4248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:48.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[46934]: cluster 2022-04-22T18:19:47.347164+0000 mgr.y (mgr.34107) 1880 : cluster [DBG] pgmap v2974: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 212 KiB/s rd, 16 KiB/s wr, 293 op/s 2022-04-22T18:19:48.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[46934]: cluster 2022-04-22T18:19:47.429472+0000 mon.a (mon.0) 4247 : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2022-04-22T18:19:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[46934]: audit 2022-04-22T18:19:48.362375+0000 mon.a (mon.0) 4248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[45495]: cluster 2022-04-22T18:19:47.347164+0000 mgr.y (mgr.34107) 1880 : cluster [DBG] pgmap v2974: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 212 KiB/s rd, 16 KiB/s wr, 293 op/s 2022-04-22T18:19:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[45495]: cluster 2022-04-22T18:19:47.429472+0000 mon.a (mon.0) 4247 : cluster [DBG] osdmap e1642: 8 total, 8 up, 8 in 2022-04-22T18:19:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:48 smithi115 bash[45495]: audit 2022-04-22T18:19:48.362375+0000 mon.a (mon.0) 4248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:49.605 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_flatten_with_progress ... ok 2022-04-22T18:19:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:49 smithi121 bash[36902]: cluster 2022-04-22T18:19:48.439035+0000 mon.a (mon.0) 4249 : cluster [DBG] osdmap e1643: 8 total, 8 up, 8 in 2022-04-22T18:19:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:49 smithi115 bash[45495]: cluster 2022-04-22T18:19:48.439035+0000 mon.a (mon.0) 4249 : cluster [DBG] osdmap e1643: 8 total, 8 up, 8 in 2022-04-22T18:19:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:49 smithi115 bash[46934]: cluster 2022-04-22T18:19:48.439035+0000 mon.a (mon.0) 4249 : cluster [DBG] osdmap e1643: 8 total, 8 up, 8 in 2022-04-22T18:19:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:50 smithi121 bash[36902]: cluster 2022-04-22T18:19:49.348214+0000 mgr.y (mgr.34107) 1881 : cluster [DBG] pgmap v2977: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 359 KiB/s rd, 26 KiB/s wr, 488 op/s 2022-04-22T18:19:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:50 smithi121 bash[36902]: cluster 2022-04-22T18:19:49.449010+0000 mon.a (mon.0) 4250 : cluster [DBG] osdmap e1644: 8 total, 8 up, 8 in 2022-04-22T18:19:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:50 smithi115 bash[45495]: cluster 2022-04-22T18:19:49.348214+0000 mgr.y (mgr.34107) 1881 : cluster [DBG] pgmap v2977: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 359 KiB/s rd, 26 KiB/s wr, 488 op/s 2022-04-22T18:19:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:50 smithi115 bash[45495]: cluster 2022-04-22T18:19:49.449010+0000 mon.a (mon.0) 4250 : cluster [DBG] osdmap e1644: 8 total, 8 up, 8 in 2022-04-22T18:19:50.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:50 smithi115 bash[46934]: cluster 2022-04-22T18:19:49.348214+0000 mgr.y (mgr.34107) 1881 : cluster [DBG] pgmap v2977: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 359 KiB/s rd, 26 KiB/s wr, 488 op/s 2022-04-22T18:19:50.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:50 smithi115 bash[46934]: cluster 2022-04-22T18:19:49.449010+0000 mon.a (mon.0) 4250 : cluster [DBG] osdmap e1644: 8 total, 8 up, 8 in 2022-04-22T18:19:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:51 smithi121 bash[36902]: cluster 2022-04-22T18:19:50.456668+0000 mon.a (mon.0) 4251 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2022-04-22T18:19:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:51 smithi121 bash[36902]: audit 2022-04-22T18:19:50.987540+0000 mon.a (mon.0) 4252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:51 smithi121 bash[36902]: cluster 2022-04-22T18:19:51.334235+0000 mon.a (mon.0) 4253 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2022-04-22T18:19:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[45495]: cluster 2022-04-22T18:19:50.456668+0000 mon.a (mon.0) 4251 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2022-04-22T18:19:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[45495]: audit 2022-04-22T18:19:50.987540+0000 mon.a (mon.0) 4252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[45495]: cluster 2022-04-22T18:19:51.334235+0000 mon.a (mon.0) 4253 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2022-04-22T18:19:51.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[46934]: cluster 2022-04-22T18:19:50.456668+0000 mon.a (mon.0) 4251 : cluster [DBG] osdmap e1645: 8 total, 8 up, 8 in 2022-04-22T18:19:51.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[46934]: audit 2022-04-22T18:19:50.987540+0000 mon.a (mon.0) 4252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:51.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[46934]: cluster 2022-04-22T18:19:51.334235+0000 mon.a (mon.0) 4253 : cluster [DBG] osdmap e1646: 8 total, 8 up, 8 in 2022-04-22T18:19:52.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:51.799Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:52 smithi121 bash[36902]: cluster 2022-04-22T18:19:51.349076+0000 mgr.y (mgr.34107) 1882 : cluster [DBG] pgmap v2981: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 441 KiB/s rd, 26 KiB/s wr, 578 op/s 2022-04-22T18:19:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:52 smithi121 bash[36902]: audit 2022-04-22T18:19:51.470505+0000 mon.a (mon.0) 4254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:52 smithi115 bash[45495]: cluster 2022-04-22T18:19:51.349076+0000 mgr.y (mgr.34107) 1882 : cluster [DBG] pgmap v2981: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 441 KiB/s rd, 26 KiB/s wr, 578 op/s 2022-04-22T18:19:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:52 smithi115 bash[45495]: audit 2022-04-22T18:19:51.470505+0000 mon.a (mon.0) 4254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:52 smithi115 bash[46934]: cluster 2022-04-22T18:19:51.349076+0000 mgr.y (mgr.34107) 1882 : cluster [DBG] pgmap v2981: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 441 KiB/s rd, 26 KiB/s wr, 578 op/s 2022-04-22T18:19:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:52 smithi115 bash[46934]: audit 2022-04-22T18:19:51.470505+0000 mon.a (mon.0) 4254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:53 smithi121 bash[36902]: audit 2022-04-22T18:19:53.365141+0000 mon.a (mon.0) 4255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:53 smithi115 bash[45495]: audit 2022-04-22T18:19:53.365141+0000 mon.a (mon.0) 4255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:53.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:19:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:19:53.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:19:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:53 smithi115 bash[46934]: audit 2022-04-22T18:19:53.365141+0000 mon.a (mon.0) 4255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:54 smithi121 bash[36902]: cluster 2022-04-22T18:19:53.350511+0000 mgr.y (mgr.34107) 1883 : cluster [DBG] pgmap v2982: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 586 KiB/s rd, 20 KiB/s wr, 729 op/s 2022-04-22T18:19:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:54 smithi115 bash[45495]: cluster 2022-04-22T18:19:53.350511+0000 mgr.y (mgr.34107) 1883 : cluster [DBG] pgmap v2982: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 586 KiB/s rd, 20 KiB/s wr, 729 op/s 2022-04-22T18:19:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:54 smithi115 bash[46934]: cluster 2022-04-22T18:19:53.350511+0000 mgr.y (mgr.34107) 1883 : cluster [DBG] pgmap v2982: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 586 KiB/s rd, 20 KiB/s wr, 729 op/s 2022-04-22T18:19:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:56 smithi121 bash[36902]: cluster 2022-04-22T18:19:55.351606+0000 mgr.y (mgr.34107) 1884 : cluster [DBG] pgmap v2983: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 710 KiB/s rd, 21 KiB/s wr, 870 op/s 2022-04-22T18:19:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:56 smithi121 bash[36902]: audit 2022-04-22T18:19:55.990221+0000 mon.a (mon.0) 4256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:56 smithi121 bash[36902]: cluster 2022-04-22T18:19:56.332821+0000 mon.a (mon.0) 4257 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2022-04-22T18:19:56.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[45495]: cluster 2022-04-22T18:19:55.351606+0000 mgr.y (mgr.34107) 1884 : cluster [DBG] pgmap v2983: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 710 KiB/s rd, 21 KiB/s wr, 870 op/s 2022-04-22T18:19:56.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[45495]: audit 2022-04-22T18:19:55.990221+0000 mon.a (mon.0) 4256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:56.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[45495]: cluster 2022-04-22T18:19:56.332821+0000 mon.a (mon.0) 4257 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2022-04-22T18:19:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[46934]: cluster 2022-04-22T18:19:55.351606+0000 mgr.y (mgr.34107) 1884 : cluster [DBG] pgmap v2983: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 710 KiB/s rd, 21 KiB/s wr, 870 op/s 2022-04-22T18:19:56.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[46934]: audit 2022-04-22T18:19:55.990221+0000 mon.a (mon.0) 4256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:19:56.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:56 smithi115 bash[46934]: cluster 2022-04-22T18:19:56.332821+0000 mon.a (mon.0) 4257 : cluster [DBG] osdmap e1647: 8 total, 8 up, 8 in 2022-04-22T18:19:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:57 smithi121 bash[36902]: audit 2022-04-22T18:19:56.475780+0000 mon.a (mon.0) 4258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:57.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:57 smithi115 bash[45495]: audit 2022-04-22T18:19:56.475780+0000 mon.a (mon.0) 4258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:57 smithi115 bash[46934]: audit 2022-04-22T18:19:56.475780+0000 mon.a (mon.0) 4258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:19:58.640 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_list_children ... ok 2022-04-22T18:19:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:58 smithi121 bash[36902]: cluster 2022-04-22T18:19:57.352300+0000 mgr.y (mgr.34107) 1885 : cluster [DBG] pgmap v2985: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 16 KiB/s wr, 1.60k op/s 2022-04-22T18:19:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:58 smithi121 bash[36902]: audit 2022-04-22T18:19:58.367889+0000 mon.a (mon.0) 4259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:58 smithi115 bash[45495]: cluster 2022-04-22T18:19:57.352300+0000 mgr.y (mgr.34107) 1885 : cluster [DBG] pgmap v2985: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 16 KiB/s wr, 1.60k op/s 2022-04-22T18:19:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:58 smithi115 bash[45495]: audit 2022-04-22T18:19:58.367889+0000 mon.a (mon.0) 4259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:58 smithi115 bash[46934]: cluster 2022-04-22T18:19:57.352300+0000 mgr.y (mgr.34107) 1885 : cluster [DBG] pgmap v2985: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 16 KiB/s wr, 1.60k op/s 2022-04-22T18:19:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:58 smithi115 bash[46934]: audit 2022-04-22T18:19:58.367889+0000 mon.a (mon.0) 4259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:19:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:19:59 smithi121 bash[36902]: cluster 2022-04-22T18:19:58.490729+0000 mon.a (mon.0) 4260 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2022-04-22T18:19:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:19:59 smithi115 bash[45495]: cluster 2022-04-22T18:19:58.490729+0000 mon.a (mon.0) 4260 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2022-04-22T18:19:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:19:59 smithi115 bash[46934]: cluster 2022-04-22T18:19:58.490729+0000 mon.a (mon.0) 4260 : cluster [DBG] osdmap e1648: 8 total, 8 up, 8 in 2022-04-22T18:20:00.645 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_read ... ok 2022-04-22T18:20:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:00 smithi121 bash[36902]: cluster 2022-04-22T18:19:59.353487+0000 mgr.y (mgr.34107) 1886 : cluster [DBG] pgmap v2987: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 19 KiB/s wr, 1.47k op/s 2022-04-22T18:20:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:00 smithi121 bash[36902]: cluster 2022-04-22T18:19:59.501535+0000 mon.a (mon.0) 4261 : cluster [DBG] osdmap e1649: 8 total, 8 up, 8 in 2022-04-22T18:20:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:00 smithi121 bash[36902]: cluster 2022-04-22T18:20:00.000188+0000 mon.a (mon.0) 4262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:20:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[45495]: cluster 2022-04-22T18:19:59.353487+0000 mgr.y (mgr.34107) 1886 : cluster [DBG] pgmap v2987: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 19 KiB/s wr, 1.47k op/s 2022-04-22T18:20:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[45495]: cluster 2022-04-22T18:19:59.501535+0000 mon.a (mon.0) 4261 : cluster [DBG] osdmap e1649: 8 total, 8 up, 8 in 2022-04-22T18:20:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[45495]: cluster 2022-04-22T18:20:00.000188+0000 mon.a (mon.0) 4262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:20:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[46934]: cluster 2022-04-22T18:19:59.353487+0000 mgr.y (mgr.34107) 1886 : cluster [DBG] pgmap v2987: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 19 KiB/s wr, 1.47k op/s 2022-04-22T18:20:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[46934]: cluster 2022-04-22T18:19:59.501535+0000 mon.a (mon.0) 4261 : cluster [DBG] osdmap e1649: 8 total, 8 up, 8 in 2022-04-22T18:20:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:00 smithi115 bash[46934]: cluster 2022-04-22T18:20:00.000188+0000 mon.a (mon.0) 4262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:20:01.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[45495]: cluster 2022-04-22T18:20:00.508819+0000 mon.a (mon.0) 4263 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2022-04-22T18:20:01.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[45495]: audit 2022-04-22T18:20:00.995820+0000 mon.a (mon.0) 4264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:01.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[45495]: audit 2022-04-22T18:20:01.478355+0000 mon.a (mon.0) 4265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:01.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[46934]: cluster 2022-04-22T18:20:00.508819+0000 mon.a (mon.0) 4263 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2022-04-22T18:20:01.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[46934]: audit 2022-04-22T18:20:00.995820+0000 mon.a (mon.0) 4264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:01.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[46934]: audit 2022-04-22T18:20:01.478355+0000 mon.a (mon.0) 4265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:01 smithi121 bash[36902]: cluster 2022-04-22T18:20:00.508819+0000 mon.a (mon.0) 4263 : cluster [DBG] osdmap e1650: 8 total, 8 up, 8 in 2022-04-22T18:20:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:01 smithi121 bash[36902]: audit 2022-04-22T18:20:00.995820+0000 mon.a (mon.0) 4264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:01.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:01 smithi121 bash[36902]: audit 2022-04-22T18:20:01.478355+0000 mon.a (mon.0) 4265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:02.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:01.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: cluster 2022-04-22T18:20:01.354518+0000 mgr.y (mgr.34107) 1887 : cluster [DBG] pgmap v2990: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 14 KiB/s wr, 1.59k op/s 2022-04-22T18:20:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: cluster 2022-04-22T18:20:01.517282+0000 mon.a (mon.0) 4266 : cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2022-04-22T18:20:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: audit 2022-04-22T18:20:01.735278+0000 mon.a (mon.0) 4267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: audit 2022-04-22T18:20:01.738547+0000 mon.a (mon.0) 4268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: audit 2022-04-22T18:20:01.740435+0000 mon.a (mon.0) 4269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[45495]: audit 2022-04-22T18:20:02.100161+0000 mon.a (mon.0) 4270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: cluster 2022-04-22T18:20:01.354518+0000 mgr.y (mgr.34107) 1887 : cluster [DBG] pgmap v2990: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 14 KiB/s wr, 1.59k op/s 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: cluster 2022-04-22T18:20:01.517282+0000 mon.a (mon.0) 4266 : cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: audit 2022-04-22T18:20:01.735278+0000 mon.a (mon.0) 4267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:20:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: audit 2022-04-22T18:20:01.738547+0000 mon.a (mon.0) 4268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:20:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: audit 2022-04-22T18:20:01.740435+0000 mon.a (mon.0) 4269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:20:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:02 smithi115 bash[46934]: audit 2022-04-22T18:20:02.100161+0000 mon.a (mon.0) 4270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: cluster 2022-04-22T18:20:01.354518+0000 mgr.y (mgr.34107) 1887 : cluster [DBG] pgmap v2990: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 14 KiB/s wr, 1.59k op/s 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: cluster 2022-04-22T18:20:01.517282+0000 mon.a (mon.0) 4266 : cluster [DBG] osdmap e1651: 8 total, 8 up, 8 in 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: audit 2022-04-22T18:20:01.735278+0000 mon.a (mon.0) 4267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: audit 2022-04-22T18:20:01.738547+0000 mon.a (mon.0) 4268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: audit 2022-04-22T18:20:01.740435+0000 mon.a (mon.0) 4269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:20:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:02 smithi121 bash[36902]: audit 2022-04-22T18:20:02.100161+0000 mon.a (mon.0) 4270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:20:03.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:03.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:03 smithi115 bash[45495]: cluster 2022-04-22T18:20:02.528606+0000 mon.a (mon.0) 4271 : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2022-04-22T18:20:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:03 smithi115 bash[45495]: audit 2022-04-22T18:20:03.370957+0000 mon.a (mon.0) 4272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:03.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:03 smithi115 bash[46934]: cluster 2022-04-22T18:20:02.528606+0000 mon.a (mon.0) 4271 : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2022-04-22T18:20:03.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:03 smithi115 bash[46934]: audit 2022-04-22T18:20:03.370957+0000 mon.a (mon.0) 4272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:03 smithi121 bash[36902]: cluster 2022-04-22T18:20:02.528606+0000 mon.a (mon.0) 4271 : cluster [DBG] osdmap e1652: 8 total, 8 up, 8 in 2022-04-22T18:20:04.039 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:03 smithi121 bash[36902]: audit 2022-04-22T18:20:03.370957+0000 mon.a (mon.0) 4272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:04.517 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:20:04.686 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_resize_flatten_multi_level ... ok 2022-04-22T18:20:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:04 smithi115 bash[45495]: cluster 2022-04-22T18:20:03.355563+0000 mgr.y (mgr.34107) 1888 : cluster [DBG] pgmap v2993: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 547 KiB/s rd, 15 KiB/s wr, 684 op/s 2022-04-22T18:20:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:04 smithi115 bash[45495]: cluster 2022-04-22T18:20:03.541056+0000 mon.a (mon.0) 4273 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2022-04-22T18:20:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:04 smithi115 bash[46934]: cluster 2022-04-22T18:20:03.355563+0000 mgr.y (mgr.34107) 1888 : cluster [DBG] pgmap v2993: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 547 KiB/s rd, 15 KiB/s wr, 684 op/s 2022-04-22T18:20:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:04 smithi115 bash[46934]: cluster 2022-04-22T18:20:03.541056+0000 mon.a (mon.0) 4273 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (43m) 63s ago 56m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (56m) 2m ago 56m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (45m) 63s ago 57m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (44m) 2m ago 57m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (52m) 2m ago 62m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:20:04.883 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (53m) 63s ago 64m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (52m) 63s ago 64m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (51m) 2m ago 62m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (51m) 63s ago 62m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (44m) 63s ago 57m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (44m) 2m ago 57m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (49m) 63s ago 61m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (49m) 63s ago 61m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (48m) 63s ago 60m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (48m) 63s ago 60m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:20:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (47m) 2m ago 59m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:20:04.885 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (46m) 2m ago 59m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:20:04.885 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (46m) 2m ago 58m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:20:04.885 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (45m) 2m ago 58m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:20:04.885 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (43m) 2m ago 57m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:20:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:04 smithi121 bash[36902]: cluster 2022-04-22T18:20:03.355563+0000 mgr.y (mgr.34107) 1888 : cluster [DBG] pgmap v2993: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 547 KiB/s rd, 15 KiB/s wr, 684 op/s 2022-04-22T18:20:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:04 smithi121 bash[36902]: cluster 2022-04-22T18:20:03.541056+0000 mon.a (mon.0) 4273 : cluster [DBG] osdmap e1653: 8 total, 8 up, 8 in 2022-04-22T18:20:05.287 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:20:05.288 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:20:05.289 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:20:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:05 smithi121 bash[36902]: audit 2022-04-22T18:20:04.496577+0000 mgr.y (mgr.34107) 1889 : audit [DBG] from='client.37275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:06.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:05 smithi121 bash[36902]: cluster 2022-04-22T18:20:04.547442+0000 mon.a (mon.0) 4274 : cluster [DBG] osdmap e1654: 8 total, 8 up, 8 in 2022-04-22T18:20:06.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:05 smithi121 bash[36902]: audit 2022-04-22T18:20:05.281444+0000 mon.c (mon.1) 117 : audit [DBG] from='client.? 172.21.15.115:0/332017691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:06.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[45495]: audit 2022-04-22T18:20:04.496577+0000 mgr.y (mgr.34107) 1889 : audit [DBG] from='client.37275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:06.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[45495]: cluster 2022-04-22T18:20:04.547442+0000 mon.a (mon.0) 4274 : cluster [DBG] osdmap e1654: 8 total, 8 up, 8 in 2022-04-22T18:20:06.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[45495]: audit 2022-04-22T18:20:05.281444+0000 mon.c (mon.1) 117 : audit [DBG] from='client.? 172.21.15.115:0/332017691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[46934]: audit 2022-04-22T18:20:04.496577+0000 mgr.y (mgr.34107) 1889 : audit [DBG] from='client.37275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[46934]: cluster 2022-04-22T18:20:04.547442+0000 mon.a (mon.0) 4274 : cluster [DBG] osdmap e1654: 8 total, 8 up, 8 in 2022-04-22T18:20:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:05 smithi115 bash[46934]: audit 2022-04-22T18:20:05.281444+0000 mon.c (mon.1) 117 : audit [DBG] from='client.? 172.21.15.115:0/332017691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:06.493 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_resize_io ... ok 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: audit 2022-04-22T18:20:04.857665+0000 mgr.y (mgr.34107) 1890 : audit [DBG] from='client.37281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: cluster 2022-04-22T18:20:05.356555+0000 mgr.y (mgr.34107) 1891 : cluster [DBG] pgmap v2996: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 402 KiB/s rd, 19 KiB/s wr, 515 op/s 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: cluster 2022-04-22T18:20:05.556497+0000 mon.a (mon.0) 4275 : cluster [DBG] osdmap e1655: 8 total, 8 up, 8 in 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: audit 2022-04-22T18:20:05.998731+0000 mon.a (mon.0) 4276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: cluster 2022-04-22T18:20:06.340426+0000 mon.a (mon.0) 4277 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2022-04-22T18:20:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:06 smithi121 bash[36902]: audit 2022-04-22T18:20:06.481755+0000 mon.a (mon.0) 4278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: audit 2022-04-22T18:20:04.857665+0000 mgr.y (mgr.34107) 1890 : audit [DBG] from='client.37281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: cluster 2022-04-22T18:20:05.356555+0000 mgr.y (mgr.34107) 1891 : cluster [DBG] pgmap v2996: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 402 KiB/s rd, 19 KiB/s wr, 515 op/s 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: cluster 2022-04-22T18:20:05.556497+0000 mon.a (mon.0) 4275 : cluster [DBG] osdmap e1655: 8 total, 8 up, 8 in 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: audit 2022-04-22T18:20:05.998731+0000 mon.a (mon.0) 4276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: cluster 2022-04-22T18:20:06.340426+0000 mon.a (mon.0) 4277 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[45495]: audit 2022-04-22T18:20:06.481755+0000 mon.a (mon.0) 4278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: audit 2022-04-22T18:20:04.857665+0000 mgr.y (mgr.34107) 1890 : audit [DBG] from='client.37281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: cluster 2022-04-22T18:20:05.356555+0000 mgr.y (mgr.34107) 1891 : cluster [DBG] pgmap v2996: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 402 KiB/s rd, 19 KiB/s wr, 515 op/s 2022-04-22T18:20:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: cluster 2022-04-22T18:20:05.556497+0000 mon.a (mon.0) 4275 : cluster [DBG] osdmap e1655: 8 total, 8 up, 8 in 2022-04-22T18:20:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: audit 2022-04-22T18:20:05.998731+0000 mon.a (mon.0) 4276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: cluster 2022-04-22T18:20:06.340426+0000 mon.a (mon.0) 4277 : cluster [DBG] osdmap e1656: 8 total, 8 up, 8 in 2022-04-22T18:20:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:06 smithi115 bash[46934]: audit 2022-04-22T18:20:06.481755+0000 mon.a (mon.0) 4278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:08.524 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_resize_stat ... ok 2022-04-22T18:20:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:08 smithi121 bash[36902]: cluster 2022-04-22T18:20:07.355767+0000 mon.a (mon.0) 4279 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2022-04-22T18:20:08.835 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:08 smithi121 bash[36902]: cluster 2022-04-22T18:20:07.357395+0000 mgr.y (mgr.34107) 1892 : cluster [DBG] pgmap v3000: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 306 KiB/s rd, 17 KiB/s wr, 396 op/s 2022-04-22T18:20:08.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:08 smithi115 bash[45495]: cluster 2022-04-22T18:20:07.355767+0000 mon.a (mon.0) 4279 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2022-04-22T18:20:08.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:08 smithi115 bash[45495]: cluster 2022-04-22T18:20:07.357395+0000 mgr.y (mgr.34107) 1892 : cluster [DBG] pgmap v3000: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 306 KiB/s rd, 17 KiB/s wr, 396 op/s 2022-04-22T18:20:08.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:08 smithi115 bash[46934]: cluster 2022-04-22T18:20:07.355767+0000 mon.a (mon.0) 4279 : cluster [DBG] osdmap e1657: 8 total, 8 up, 8 in 2022-04-22T18:20:08.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:08 smithi115 bash[46934]: cluster 2022-04-22T18:20:07.357395+0000 mgr.y (mgr.34107) 1892 : cluster [DBG] pgmap v3000: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 306 KiB/s rd, 17 KiB/s wr, 396 op/s 2022-04-22T18:20:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:09 smithi121 bash[36902]: cluster 2022-04-22T18:20:08.368161+0000 mon.a (mon.0) 4280 : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2022-04-22T18:20:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:09 smithi121 bash[36902]: audit 2022-04-22T18:20:08.373455+0000 mon.a (mon.0) 4281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:09 smithi115 bash[45495]: cluster 2022-04-22T18:20:08.368161+0000 mon.a (mon.0) 4280 : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2022-04-22T18:20:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:09 smithi115 bash[45495]: audit 2022-04-22T18:20:08.373455+0000 mon.a (mon.0) 4281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:09 smithi115 bash[46934]: cluster 2022-04-22T18:20:08.368161+0000 mon.a (mon.0) 4280 : cluster [DBG] osdmap e1658: 8 total, 8 up, 8 in 2022-04-22T18:20:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:09 smithi115 bash[46934]: audit 2022-04-22T18:20:08.373455+0000 mon.a (mon.0) 4281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:10.590 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_stat ... ok 2022-04-22T18:20:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:10 smithi121 bash[36902]: cluster 2022-04-22T18:20:09.358483+0000 mgr.y (mgr.34107) 1893 : cluster [DBG] pgmap v3002: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 25 KiB/s wr, 404 op/s 2022-04-22T18:20:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:10 smithi121 bash[36902]: cluster 2022-04-22T18:20:09.373821+0000 mon.a (mon.0) 4282 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2022-04-22T18:20:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:10 smithi115 bash[45495]: cluster 2022-04-22T18:20:09.358483+0000 mgr.y (mgr.34107) 1893 : cluster [DBG] pgmap v3002: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 25 KiB/s wr, 404 op/s 2022-04-22T18:20:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:10 smithi115 bash[45495]: cluster 2022-04-22T18:20:09.373821+0000 mon.a (mon.0) 4282 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2022-04-22T18:20:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:10 smithi115 bash[46934]: cluster 2022-04-22T18:20:09.358483+0000 mgr.y (mgr.34107) 1893 : cluster [DBG] pgmap v3002: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 296 KiB/s rd, 25 KiB/s wr, 404 op/s 2022-04-22T18:20:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:10 smithi115 bash[46934]: cluster 2022-04-22T18:20:09.373821+0000 mon.a (mon.0) 4282 : cluster [DBG] osdmap e1659: 8 total, 8 up, 8 in 2022-04-22T18:20:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:11 smithi121 bash[36902]: cluster 2022-04-22T18:20:10.420471+0000 mon.a (mon.0) 4283 : cluster [DBG] osdmap e1660: 8 total, 8 up, 8 in 2022-04-22T18:20:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:11 smithi121 bash[36902]: audit 2022-04-22T18:20:11.001632+0000 mon.a (mon.0) 4284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:11 smithi121 bash[36902]: cluster 2022-04-22T18:20:11.342134+0000 mon.a (mon.0) 4285 : cluster [DBG] osdmap e1661: 8 total, 8 up, 8 in 2022-04-22T18:20:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[45495]: cluster 2022-04-22T18:20:10.420471+0000 mon.a (mon.0) 4283 : cluster [DBG] osdmap e1660: 8 total, 8 up, 8 in 2022-04-22T18:20:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[45495]: audit 2022-04-22T18:20:11.001632+0000 mon.a (mon.0) 4284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[45495]: cluster 2022-04-22T18:20:11.342134+0000 mon.a (mon.0) 4285 : cluster [DBG] osdmap e1661: 8 total, 8 up, 8 in 2022-04-22T18:20:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[46934]: cluster 2022-04-22T18:20:10.420471+0000 mon.a (mon.0) 4283 : cluster [DBG] osdmap e1660: 8 total, 8 up, 8 in 2022-04-22T18:20:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[46934]: audit 2022-04-22T18:20:11.001632+0000 mon.a (mon.0) 4284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[46934]: cluster 2022-04-22T18:20:11.342134+0000 mon.a (mon.0) 4285 : cluster [DBG] osdmap e1661: 8 total, 8 up, 8 in 2022-04-22T18:20:12.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:11.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:12 smithi121 bash[36902]: cluster 2022-04-22T18:20:11.359190+0000 mgr.y (mgr.34107) 1894 : cluster [DBG] pgmap v3006: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 27 KiB/s wr, 466 op/s 2022-04-22T18:20:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:12 smithi121 bash[36902]: audit 2022-04-22T18:20:11.485431+0000 mon.a (mon.0) 4286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:12 smithi115 bash[45495]: cluster 2022-04-22T18:20:11.359190+0000 mgr.y (mgr.34107) 1894 : cluster [DBG] pgmap v3006: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 27 KiB/s wr, 466 op/s 2022-04-22T18:20:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:12 smithi115 bash[45495]: audit 2022-04-22T18:20:11.485431+0000 mon.a (mon.0) 4286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:12 smithi115 bash[46934]: cluster 2022-04-22T18:20:11.359190+0000 mgr.y (mgr.34107) 1894 : cluster [DBG] pgmap v3006: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 27 KiB/s wr, 466 op/s 2022-04-22T18:20:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:12 smithi115 bash[46934]: audit 2022-04-22T18:20:11.485431+0000 mon.a (mon.0) 4286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:13 smithi121 bash[36902]: cluster 2022-04-22T18:20:12.426032+0000 mon.a (mon.0) 4287 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2022-04-22T18:20:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:13 smithi121 bash[36902]: audit 2022-04-22T18:20:13.376704+0000 mon.a (mon.0) 4288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:13 smithi115 bash[45495]: cluster 2022-04-22T18:20:12.426032+0000 mon.a (mon.0) 4287 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2022-04-22T18:20:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:13 smithi115 bash[45495]: audit 2022-04-22T18:20:13.376704+0000 mon.a (mon.0) 4288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:13.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:13.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:13 smithi115 bash[46934]: cluster 2022-04-22T18:20:12.426032+0000 mon.a (mon.0) 4287 : cluster [DBG] osdmap e1662: 8 total, 8 up, 8 in 2022-04-22T18:20:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:13 smithi115 bash[46934]: audit 2022-04-22T18:20:13.376704+0000 mon.a (mon.0) 4288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:14.610 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_stripe_unit_and_count ... ok 2022-04-22T18:20:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:14 smithi121 bash[36902]: cluster 2022-04-22T18:20:13.360375+0000 mgr.y (mgr.34107) 1895 : cluster [DBG] pgmap v3008: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 18 KiB/s wr, 314 op/s 2022-04-22T18:20:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:14 smithi121 bash[36902]: cluster 2022-04-22T18:20:13.444023+0000 mon.a (mon.0) 4289 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2022-04-22T18:20:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:14 smithi121 bash[36902]: audit 2022-04-22T18:20:13.891708+0000 mon.a (mon.0) 4290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:20:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:14 smithi121 bash[36902]: audit 2022-04-22T18:20:13.898147+0000 mon.a (mon.0) 4291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:20:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[45495]: cluster 2022-04-22T18:20:13.360375+0000 mgr.y (mgr.34107) 1895 : cluster [DBG] pgmap v3008: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 18 KiB/s wr, 314 op/s 2022-04-22T18:20:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[45495]: cluster 2022-04-22T18:20:13.444023+0000 mon.a (mon.0) 4289 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[45495]: audit 2022-04-22T18:20:13.891708+0000 mon.a (mon.0) 4290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[45495]: audit 2022-04-22T18:20:13.898147+0000 mon.a (mon.0) 4291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[46934]: cluster 2022-04-22T18:20:13.360375+0000 mgr.y (mgr.34107) 1895 : cluster [DBG] pgmap v3008: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 18 KiB/s wr, 314 op/s 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[46934]: cluster 2022-04-22T18:20:13.444023+0000 mon.a (mon.0) 4289 : cluster [DBG] osdmap e1663: 8 total, 8 up, 8 in 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[46934]: audit 2022-04-22T18:20:13.891708+0000 mon.a (mon.0) 4290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:20:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:14 smithi115 bash[46934]: audit 2022-04-22T18:20:13.898147+0000 mon.a (mon.0) 4291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:20:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:15 smithi121 bash[36902]: cluster 2022-04-22T18:20:14.456168+0000 mon.a (mon.0) 4292 : cluster [DBG] osdmap e1664: 8 total, 8 up, 8 in 2022-04-22T18:20:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:15 smithi115 bash[45495]: cluster 2022-04-22T18:20:14.456168+0000 mon.a (mon.0) 4292 : cluster [DBG] osdmap e1664: 8 total, 8 up, 8 in 2022-04-22T18:20:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:15 smithi115 bash[46934]: cluster 2022-04-22T18:20:14.456168+0000 mon.a (mon.0) 4292 : cluster [DBG] osdmap e1664: 8 total, 8 up, 8 in 2022-04-22T18:20:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:16 smithi121 bash[36902]: cluster 2022-04-22T18:20:15.361641+0000 mgr.y (mgr.34107) 1896 : cluster [DBG] pgmap v3011: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 164 KiB/s rd, 14 KiB/s wr, 230 op/s 2022-04-22T18:20:16.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:16 smithi121 bash[36902]: cluster 2022-04-22T18:20:15.461298+0000 mon.a (mon.0) 4293 : cluster [DBG] osdmap e1665: 8 total, 8 up, 8 in 2022-04-22T18:20:16.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:16 smithi121 bash[36902]: audit 2022-04-22T18:20:16.004510+0000 mon.a (mon.0) 4294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:16.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:16 smithi121 bash[36902]: cluster 2022-04-22T18:20:16.350423+0000 mon.a (mon.0) 4295 : cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2022-04-22T18:20:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[45495]: cluster 2022-04-22T18:20:15.361641+0000 mgr.y (mgr.34107) 1896 : cluster [DBG] pgmap v3011: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 164 KiB/s rd, 14 KiB/s wr, 230 op/s 2022-04-22T18:20:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[45495]: cluster 2022-04-22T18:20:15.461298+0000 mon.a (mon.0) 4293 : cluster [DBG] osdmap e1665: 8 total, 8 up, 8 in 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[45495]: audit 2022-04-22T18:20:16.004510+0000 mon.a (mon.0) 4294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[45495]: cluster 2022-04-22T18:20:16.350423+0000 mon.a (mon.0) 4295 : cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[46934]: cluster 2022-04-22T18:20:15.361641+0000 mgr.y (mgr.34107) 1896 : cluster [DBG] pgmap v3011: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 164 KiB/s rd, 14 KiB/s wr, 230 op/s 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[46934]: cluster 2022-04-22T18:20:15.461298+0000 mon.a (mon.0) 4293 : cluster [DBG] osdmap e1665: 8 total, 8 up, 8 in 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[46934]: audit 2022-04-22T18:20:16.004510+0000 mon.a (mon.0) 4294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:16 smithi115 bash[46934]: cluster 2022-04-22T18:20:16.350423+0000 mon.a (mon.0) 4295 : cluster [DBG] osdmap e1666: 8 total, 8 up, 8 in 2022-04-22T18:20:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:17 smithi121 bash[36902]: audit 2022-04-22T18:20:16.488465+0000 mon.a (mon.0) 4296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:17 smithi115 bash[45495]: audit 2022-04-22T18:20:16.488465+0000 mon.a (mon.0) 4296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:17 smithi115 bash[46934]: audit 2022-04-22T18:20:16.488465+0000 mon.a (mon.0) 4296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:18.620 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_trash_snapshot ... ok 2022-04-22T18:20:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:18 smithi121 bash[36902]: cluster 2022-04-22T18:20:17.362775+0000 mgr.y (mgr.34107) 1897 : cluster [DBG] pgmap v3014: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 254 KiB/s rd, 15 KiB/s wr, 336 op/s 2022-04-22T18:20:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:18 smithi121 bash[36902]: cluster 2022-04-22T18:20:17.471030+0000 mon.a (mon.0) 4297 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2022-04-22T18:20:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:18 smithi121 bash[36902]: audit 2022-04-22T18:20:18.379706+0000 mon.a (mon.0) 4298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[45495]: cluster 2022-04-22T18:20:17.362775+0000 mgr.y (mgr.34107) 1897 : cluster [DBG] pgmap v3014: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 254 KiB/s rd, 15 KiB/s wr, 336 op/s 2022-04-22T18:20:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[45495]: cluster 2022-04-22T18:20:17.471030+0000 mon.a (mon.0) 4297 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2022-04-22T18:20:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[45495]: audit 2022-04-22T18:20:18.379706+0000 mon.a (mon.0) 4298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[46934]: cluster 2022-04-22T18:20:17.362775+0000 mgr.y (mgr.34107) 1897 : cluster [DBG] pgmap v3014: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 254 KiB/s rd, 15 KiB/s wr, 336 op/s 2022-04-22T18:20:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[46934]: cluster 2022-04-22T18:20:17.471030+0000 mon.a (mon.0) 4297 : cluster [DBG] osdmap e1667: 8 total, 8 up, 8 in 2022-04-22T18:20:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:18 smithi115 bash[46934]: audit 2022-04-22T18:20:18.379706+0000 mon.a (mon.0) 4298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:19 smithi121 bash[36902]: cluster 2022-04-22T18:20:18.488625+0000 mon.a (mon.0) 4299 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2022-04-22T18:20:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:19 smithi115 bash[45495]: cluster 2022-04-22T18:20:18.488625+0000 mon.a (mon.0) 4299 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2022-04-22T18:20:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:19 smithi115 bash[46934]: cluster 2022-04-22T18:20:18.488625+0000 mon.a (mon.0) 4299 : cluster [DBG] osdmap e1668: 8 total, 8 up, 8 in 2022-04-22T18:20:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:20 smithi121 bash[36902]: cluster 2022-04-22T18:20:19.363825+0000 mgr.y (mgr.34107) 1898 : cluster [DBG] pgmap v3017: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 15 KiB/s wr, 298 op/s 2022-04-22T18:20:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:20 smithi121 bash[36902]: cluster 2022-04-22T18:20:19.500348+0000 mon.a (mon.0) 4300 : cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2022-04-22T18:20:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:20 smithi115 bash[45495]: cluster 2022-04-22T18:20:19.363825+0000 mgr.y (mgr.34107) 1898 : cluster [DBG] pgmap v3017: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 15 KiB/s wr, 298 op/s 2022-04-22T18:20:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:20 smithi115 bash[45495]: cluster 2022-04-22T18:20:19.500348+0000 mon.a (mon.0) 4300 : cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2022-04-22T18:20:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:20 smithi115 bash[46934]: cluster 2022-04-22T18:20:19.363825+0000 mgr.y (mgr.34107) 1898 : cluster [DBG] pgmap v3017: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 15 KiB/s wr, 298 op/s 2022-04-22T18:20:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:20 smithi115 bash[46934]: cluster 2022-04-22T18:20:19.500348+0000 mon.a (mon.0) 4300 : cluster [DBG] osdmap e1669: 8 total, 8 up, 8 in 2022-04-22T18:20:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: cluster 2022-04-22T18:20:20.511430+0000 mon.a (mon.0) 4301 : cluster [DBG] osdmap e1670: 8 total, 8 up, 8 in 2022-04-22T18:20:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: audit 2022-04-22T18:20:20.519514+0000 mon.c (mon.1) 118 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: audit 2022-04-22T18:20:20.521436+0000 mon.a (mon.0) 4302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: audit 2022-04-22T18:20:21.006704+0000 mon.a (mon.0) 4303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: audit 2022-04-22T18:20:21.350001+0000 mon.a (mon.0) 4304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]': finished 2022-04-22T18:20:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: cluster 2022-04-22T18:20:21.350172+0000 mon.a (mon.0) 4305 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2022-04-22T18:20:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[45495]: audit 2022-04-22T18:20:21.491819+0000 mon.a (mon.0) 4306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: cluster 2022-04-22T18:20:20.511430+0000 mon.a (mon.0) 4301 : cluster [DBG] osdmap e1670: 8 total, 8 up, 8 in 2022-04-22T18:20:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: audit 2022-04-22T18:20:20.519514+0000 mon.c (mon.1) 118 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: audit 2022-04-22T18:20:20.521436+0000 mon.a (mon.0) 4302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: audit 2022-04-22T18:20:21.006704+0000 mon.a (mon.0) 4303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: audit 2022-04-22T18:20:21.350001+0000 mon.a (mon.0) 4304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]': finished 2022-04-22T18:20:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: cluster 2022-04-22T18:20:21.350172+0000 mon.a (mon.0) 4305 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2022-04-22T18:20:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[46934]: audit 2022-04-22T18:20:21.491819+0000 mon.a (mon.0) 4306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: cluster 2022-04-22T18:20:20.511430+0000 mon.a (mon.0) 4301 : cluster [DBG] osdmap e1670: 8 total, 8 up, 8 in 2022-04-22T18:20:22.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: audit 2022-04-22T18:20:20.519514+0000 mon.c (mon.1) 118 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: audit 2022-04-22T18:20:20.521436+0000 mon.a (mon.0) 4302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]: dispatch 2022-04-22T18:20:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: audit 2022-04-22T18:20:21.006704+0000 mon.a (mon.0) 4303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: audit 2022-04-22T18:20:21.350001+0000 mon.a (mon.0) 4304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rbd-api-smithi115-73548-2","app": "rbd"}]': finished 2022-04-22T18:20:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: cluster 2022-04-22T18:20:21.350172+0000 mon.a (mon.0) 4305 : cluster [DBG] osdmap e1671: 8 total, 8 up, 8 in 2022-04-22T18:20:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:21 smithi121 bash[36902]: audit 2022-04-22T18:20:21.491819+0000 mon.a (mon.0) 4306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:22.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:21.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:22 smithi115 bash[45495]: cluster 2022-04-22T18:20:21.364448+0000 mgr.y (mgr.34107) 1899 : cluster [DBG] pgmap v3021: 169 pgs: 4 creating+peering, 28 unknown, 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 9.8 KiB/s wr, 175 op/s 2022-04-22T18:20:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:22 smithi115 bash[45495]: cluster 2022-04-22T18:20:22.366679+0000 mon.a (mon.0) 4307 : cluster [DBG] osdmap e1672: 8 total, 8 up, 8 in 2022-04-22T18:20:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:22 smithi115 bash[46934]: cluster 2022-04-22T18:20:21.364448+0000 mgr.y (mgr.34107) 1899 : cluster [DBG] pgmap v3021: 169 pgs: 4 creating+peering, 28 unknown, 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 9.8 KiB/s wr, 175 op/s 2022-04-22T18:20:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:22 smithi115 bash[46934]: cluster 2022-04-22T18:20:22.366679+0000 mon.a (mon.0) 4307 : cluster [DBG] osdmap e1672: 8 total, 8 up, 8 in 2022-04-22T18:20:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:22 smithi121 bash[36902]: cluster 2022-04-22T18:20:21.364448+0000 mgr.y (mgr.34107) 1899 : cluster [DBG] pgmap v3021: 169 pgs: 4 creating+peering, 28 unknown, 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 9.8 KiB/s wr, 175 op/s 2022-04-22T18:20:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:22 smithi121 bash[36902]: cluster 2022-04-22T18:20:22.366679+0000 mon.a (mon.0) 4307 : cluster [DBG] osdmap e1672: 8 total, 8 up, 8 in 2022-04-22T18:20:24.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:23.687Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:24 smithi121 bash[36902]: cluster 2022-04-22T18:20:23.365634+0000 mgr.y (mgr.34107) 1900 : cluster [DBG] pgmap v3023: 169 pgs: 4 creating+peering, 4 unknown, 161 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 9.2 KiB/s wr, 192 op/s 2022-04-22T18:20:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:24 smithi121 bash[36902]: cluster 2022-04-22T18:20:23.371899+0000 mon.a (mon.0) 4308 : cluster [DBG] osdmap e1673: 8 total, 8 up, 8 in 2022-04-22T18:20:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:24 smithi121 bash[36902]: audit 2022-04-22T18:20:23.382480+0000 mon.a (mon.0) 4309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[45495]: cluster 2022-04-22T18:20:23.365634+0000 mgr.y (mgr.34107) 1900 : cluster [DBG] pgmap v3023: 169 pgs: 4 creating+peering, 4 unknown, 161 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 9.2 KiB/s wr, 192 op/s 2022-04-22T18:20:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[45495]: cluster 2022-04-22T18:20:23.371899+0000 mon.a (mon.0) 4308 : cluster [DBG] osdmap e1673: 8 total, 8 up, 8 in 2022-04-22T18:20:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[45495]: audit 2022-04-22T18:20:23.382480+0000 mon.a (mon.0) 4309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[46934]: cluster 2022-04-22T18:20:23.365634+0000 mgr.y (mgr.34107) 1900 : cluster [DBG] pgmap v3023: 169 pgs: 4 creating+peering, 4 unknown, 161 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 9.2 KiB/s wr, 192 op/s 2022-04-22T18:20:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[46934]: cluster 2022-04-22T18:20:23.371899+0000 mon.a (mon.0) 4308 : cluster [DBG] osdmap e1673: 8 total, 8 up, 8 in 2022-04-22T18:20:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:24 smithi115 bash[46934]: audit 2022-04-22T18:20:23.382480+0000 mon.a (mon.0) 4309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:25.572 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_unprotect_with_children ... ok 2022-04-22T18:20:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:25 smithi121 bash[36902]: cluster 2022-04-22T18:20:24.379713+0000 mon.a (mon.0) 4310 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2022-04-22T18:20:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:25 smithi115 bash[45495]: cluster 2022-04-22T18:20:24.379713+0000 mon.a (mon.0) 4310 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2022-04-22T18:20:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:25 smithi115 bash[46934]: cluster 2022-04-22T18:20:24.379713+0000 mon.a (mon.0) 4310 : cluster [DBG] osdmap e1674: 8 total, 8 up, 8 in 2022-04-22T18:20:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:26 smithi121 bash[36902]: cluster 2022-04-22T18:20:25.366852+0000 mgr.y (mgr.34107) 1901 : cluster [DBG] pgmap v3026: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.0 KiB/s wr, 148 op/s 2022-04-22T18:20:26.900 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:26 smithi121 bash[36902]: cluster 2022-04-22T18:20:25.398181+0000 mon.a (mon.0) 4311 : cluster [DBG] osdmap e1675: 8 total, 8 up, 8 in 2022-04-22T18:20:26.900 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:26 smithi121 bash[36902]: audit 2022-04-22T18:20:26.009567+0000 mon.a (mon.0) 4312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:26.900 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:26 smithi121 bash[36902]: cluster 2022-04-22T18:20:26.350114+0000 mon.a (mon.0) 4313 : cluster [DBG] osdmap e1676: 8 total, 8 up, 8 in 2022-04-22T18:20:26.901 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[45495]: cluster 2022-04-22T18:20:25.366852+0000 mgr.y (mgr.34107) 1901 : cluster [DBG] pgmap v3026: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.0 KiB/s wr, 148 op/s 2022-04-22T18:20:26.901 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[45495]: cluster 2022-04-22T18:20:25.398181+0000 mon.a (mon.0) 4311 : cluster [DBG] osdmap e1675: 8 total, 8 up, 8 in 2022-04-22T18:20:26.901 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[45495]: audit 2022-04-22T18:20:26.009567+0000 mon.a (mon.0) 4312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:26.902 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[45495]: cluster 2022-04-22T18:20:26.350114+0000 mon.a (mon.0) 4313 : cluster [DBG] osdmap e1676: 8 total, 8 up, 8 in 2022-04-22T18:20:26.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[46934]: cluster 2022-04-22T18:20:25.366852+0000 mgr.y (mgr.34107) 1901 : cluster [DBG] pgmap v3026: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.0 KiB/s wr, 148 op/s 2022-04-22T18:20:26.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[46934]: cluster 2022-04-22T18:20:25.398181+0000 mon.a (mon.0) 4311 : cluster [DBG] osdmap e1675: 8 total, 8 up, 8 in 2022-04-22T18:20:26.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[46934]: audit 2022-04-22T18:20:26.009567+0000 mon.a (mon.0) 4312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:26.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:26 smithi115 bash[46934]: cluster 2022-04-22T18:20:26.350114+0000 mon.a (mon.0) 4313 : cluster [DBG] osdmap e1676: 8 total, 8 up, 8 in 2022-04-22T18:20:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:27 smithi121 bash[36902]: audit 2022-04-22T18:20:26.495311+0000 mon.a (mon.0) 4314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:27 smithi115 bash[45495]: audit 2022-04-22T18:20:26.495311+0000 mon.a (mon.0) 4314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:27 smithi115 bash[46934]: audit 2022-04-22T18:20:26.495311+0000 mon.a (mon.0) 4314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:28 smithi121 bash[36902]: cluster 2022-04-22T18:20:27.367866+0000 mgr.y (mgr.34107) 1902 : cluster [DBG] pgmap v3029: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 64 op/s 2022-04-22T18:20:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:28 smithi121 bash[36902]: cluster 2022-04-22T18:20:27.412970+0000 mon.a (mon.0) 4315 : cluster [DBG] osdmap e1677: 8 total, 8 up, 8 in 2022-04-22T18:20:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:28 smithi121 bash[36902]: audit 2022-04-22T18:20:28.384776+0000 mon.a (mon.0) 4316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[45495]: cluster 2022-04-22T18:20:27.367866+0000 mgr.y (mgr.34107) 1902 : cluster [DBG] pgmap v3029: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 64 op/s 2022-04-22T18:20:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[45495]: cluster 2022-04-22T18:20:27.412970+0000 mon.a (mon.0) 4315 : cluster [DBG] osdmap e1677: 8 total, 8 up, 8 in 2022-04-22T18:20:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[45495]: audit 2022-04-22T18:20:28.384776+0000 mon.a (mon.0) 4316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[46934]: cluster 2022-04-22T18:20:27.367866+0000 mgr.y (mgr.34107) 1902 : cluster [DBG] pgmap v3029: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 64 op/s 2022-04-22T18:20:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[46934]: cluster 2022-04-22T18:20:27.412970+0000 mon.a (mon.0) 4315 : cluster [DBG] osdmap e1677: 8 total, 8 up, 8 in 2022-04-22T18:20:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:28 smithi115 bash[46934]: audit 2022-04-22T18:20:28.384776+0000 mon.a (mon.0) 4316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:29.582 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_unprotected ... ok 2022-04-22T18:20:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:29 smithi121 bash[36902]: cluster 2022-04-22T18:20:28.413273+0000 mon.a (mon.0) 4317 : cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2022-04-22T18:20:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:29 smithi115 bash[45495]: cluster 2022-04-22T18:20:28.413273+0000 mon.a (mon.0) 4317 : cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2022-04-22T18:20:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:29 smithi115 bash[46934]: cluster 2022-04-22T18:20:28.413273+0000 mon.a (mon.0) 4317 : cluster [DBG] osdmap e1678: 8 total, 8 up, 8 in 2022-04-22T18:20:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:30 smithi121 bash[36902]: cluster 2022-04-22T18:20:29.368806+0000 mgr.y (mgr.34107) 1903 : cluster [DBG] pgmap v3032: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.7 KiB/s wr, 141 op/s 2022-04-22T18:20:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:30 smithi121 bash[36902]: cluster 2022-04-22T18:20:29.430615+0000 mon.a (mon.0) 4318 : cluster [DBG] osdmap e1679: 8 total, 8 up, 8 in 2022-04-22T18:20:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:30 smithi115 bash[45495]: cluster 2022-04-22T18:20:29.368806+0000 mgr.y (mgr.34107) 1903 : cluster [DBG] pgmap v3032: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.7 KiB/s wr, 141 op/s 2022-04-22T18:20:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:30 smithi115 bash[45495]: cluster 2022-04-22T18:20:29.430615+0000 mon.a (mon.0) 4318 : cluster [DBG] osdmap e1679: 8 total, 8 up, 8 in 2022-04-22T18:20:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:30 smithi115 bash[46934]: cluster 2022-04-22T18:20:29.368806+0000 mgr.y (mgr.34107) 1903 : cluster [DBG] pgmap v3032: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.7 KiB/s wr, 141 op/s 2022-04-22T18:20:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:30 smithi115 bash[46934]: cluster 2022-04-22T18:20:29.430615+0000 mon.a (mon.0) 4318 : cluster [DBG] osdmap e1679: 8 total, 8 up, 8 in 2022-04-22T18:20:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:31 smithi121 bash[36902]: cluster 2022-04-22T18:20:30.443864+0000 mon.a (mon.0) 4319 : cluster [DBG] osdmap e1680: 8 total, 8 up, 8 in 2022-04-22T18:20:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:31 smithi121 bash[36902]: audit 2022-04-22T18:20:31.012257+0000 mon.a (mon.0) 4320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:31 smithi115 bash[45495]: cluster 2022-04-22T18:20:30.443864+0000 mon.a (mon.0) 4319 : cluster [DBG] osdmap e1680: 8 total, 8 up, 8 in 2022-04-22T18:20:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:31 smithi115 bash[45495]: audit 2022-04-22T18:20:31.012257+0000 mon.a (mon.0) 4320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:31 smithi115 bash[46934]: cluster 2022-04-22T18:20:30.443864+0000 mon.a (mon.0) 4319 : cluster [DBG] osdmap e1680: 8 total, 8 up, 8 in 2022-04-22T18:20:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:31 smithi115 bash[46934]: audit 2022-04-22T18:20:31.012257+0000 mon.a (mon.0) 4320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:32.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:31.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:32.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:32 smithi121 bash[36902]: cluster 2022-04-22T18:20:31.369821+0000 mgr.y (mgr.34107) 1904 : cluster [DBG] pgmap v3035: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 171 op/s 2022-04-22T18:20:32.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:32 smithi121 bash[36902]: cluster 2022-04-22T18:20:31.455307+0000 mon.a (mon.0) 4321 : cluster [DBG] osdmap e1681: 8 total, 8 up, 8 in 2022-04-22T18:20:32.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:32 smithi121 bash[36902]: audit 2022-04-22T18:20:31.498654+0000 mon.a (mon.0) 4322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[45495]: cluster 2022-04-22T18:20:31.369821+0000 mgr.y (mgr.34107) 1904 : cluster [DBG] pgmap v3035: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 171 op/s 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[45495]: cluster 2022-04-22T18:20:31.455307+0000 mon.a (mon.0) 4321 : cluster [DBG] osdmap e1681: 8 total, 8 up, 8 in 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[45495]: audit 2022-04-22T18:20:31.498654+0000 mon.a (mon.0) 4322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[46934]: cluster 2022-04-22T18:20:31.369821+0000 mgr.y (mgr.34107) 1904 : cluster [DBG] pgmap v3035: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 171 op/s 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[46934]: cluster 2022-04-22T18:20:31.455307+0000 mon.a (mon.0) 4321 : cluster [DBG] osdmap e1681: 8 total, 8 up, 8 in 2022-04-22T18:20:32.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:32 smithi115 bash[46934]: audit 2022-04-22T18:20:31.498654+0000 mon.a (mon.0) 4322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:33.641 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_with_params ... ok 2022-04-22T18:20:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:33 smithi121 bash[36902]: cluster 2022-04-22T18:20:32.464147+0000 mon.a (mon.0) 4323 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2022-04-22T18:20:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:33 smithi121 bash[36902]: audit 2022-04-22T18:20:33.387857+0000 mon.a (mon.0) 4324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:33.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:33.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:33 smithi115 bash[45495]: cluster 2022-04-22T18:20:32.464147+0000 mon.a (mon.0) 4323 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2022-04-22T18:20:33.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:33 smithi115 bash[45495]: audit 2022-04-22T18:20:33.387857+0000 mon.a (mon.0) 4324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:33 smithi115 bash[46934]: cluster 2022-04-22T18:20:32.464147+0000 mon.a (mon.0) 4323 : cluster [DBG] osdmap e1682: 8 total, 8 up, 8 in 2022-04-22T18:20:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:33 smithi115 bash[46934]: audit 2022-04-22T18:20:33.387857+0000 mon.a (mon.0) 4324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:34 smithi121 bash[36902]: cluster 2022-04-22T18:20:33.371025+0000 mgr.y (mgr.34107) 1905 : cluster [DBG] pgmap v3038: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 16 KiB/s wr, 278 op/s 2022-04-22T18:20:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:34 smithi121 bash[36902]: cluster 2022-04-22T18:20:33.481164+0000 mon.a (mon.0) 4325 : cluster [DBG] osdmap e1683: 8 total, 8 up, 8 in 2022-04-22T18:20:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:34 smithi115 bash[45495]: cluster 2022-04-22T18:20:33.371025+0000 mgr.y (mgr.34107) 1905 : cluster [DBG] pgmap v3038: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 16 KiB/s wr, 278 op/s 2022-04-22T18:20:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:34 smithi115 bash[45495]: cluster 2022-04-22T18:20:33.481164+0000 mon.a (mon.0) 4325 : cluster [DBG] osdmap e1683: 8 total, 8 up, 8 in 2022-04-22T18:20:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:34 smithi115 bash[46934]: cluster 2022-04-22T18:20:33.371025+0000 mgr.y (mgr.34107) 1905 : cluster [DBG] pgmap v3038: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 16 KiB/s wr, 278 op/s 2022-04-22T18:20:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:34 smithi115 bash[46934]: cluster 2022-04-22T18:20:33.481164+0000 mon.a (mon.0) 4325 : cluster [DBG] osdmap e1683: 8 total, 8 up, 8 in 2022-04-22T18:20:35.688 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:20:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:35 smithi121 bash[36902]: cluster 2022-04-22T18:20:34.481462+0000 mon.a (mon.0) 4326 : cluster [DBG] osdmap e1684: 8 total, 8 up, 8 in 2022-04-22T18:20:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:35 smithi115 bash[45495]: cluster 2022-04-22T18:20:34.481462+0000 mon.a (mon.0) 4326 : cluster [DBG] osdmap e1684: 8 total, 8 up, 8 in 2022-04-22T18:20:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:35 smithi115 bash[46934]: cluster 2022-04-22T18:20:34.481462+0000 mon.a (mon.0) 4326 : cluster [DBG] osdmap e1684: 8 total, 8 up, 8 in 2022-04-22T18:20:36.036 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (43m) 94s ago 57m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (57m) 2m ago 57m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (45m) 94s ago 58m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (45m) 2m ago 58m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (52m) 2m ago 62m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (53m) 94s ago 64m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (52m) 94s ago 65m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:20:36.037 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (51m) 2m ago 63m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (52m) 94s ago 63m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (44m) 94s ago 57m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (44m) 2m ago 57m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (50m) 94s ago 62m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (49m) 94s ago 61m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (49m) 94s ago 61m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (48m) 94s ago 60m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (48m) 2m ago 60m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (47m) 2m ago 59m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:20:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (46m) 2m ago 59m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:20:36.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (46m) 2m ago 59m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:20:36.039 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (44m) 2m ago 57m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:20:36.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:20:36.476 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: cluster 2022-04-22T18:20:35.372184+0000 mgr.y (mgr.34107) 1906 : cluster [DBG] pgmap v3041: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 15 KiB/s wr, 275 op/s 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: cluster 2022-04-22T18:20:35.497348+0000 mon.a (mon.0) 4327 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: audit 2022-04-22T18:20:35.662596+0000 mgr.y (mgr.34107) 1907 : audit [DBG] from='client.37293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: audit 2022-04-22T18:20:36.014929+0000 mon.a (mon.0) 4328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: cluster 2022-04-22T18:20:36.354516+0000 mon.a (mon.0) 4329 : cluster [DBG] osdmap e1686: 8 total, 8 up, 8 in 2022-04-22T18:20:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:36 smithi121 bash[36902]: audit 2022-04-22T18:20:36.468990+0000 mon.c (mon.1) 119 : audit [DBG] from='client.? 172.21.15.115:0/583128716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: cluster 2022-04-22T18:20:35.372184+0000 mgr.y (mgr.34107) 1906 : cluster [DBG] pgmap v3041: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 15 KiB/s wr, 275 op/s 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: cluster 2022-04-22T18:20:35.497348+0000 mon.a (mon.0) 4327 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: audit 2022-04-22T18:20:35.662596+0000 mgr.y (mgr.34107) 1907 : audit [DBG] from='client.37293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: audit 2022-04-22T18:20:36.014929+0000 mon.a (mon.0) 4328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: cluster 2022-04-22T18:20:36.354516+0000 mon.a (mon.0) 4329 : cluster [DBG] osdmap e1686: 8 total, 8 up, 8 in 2022-04-22T18:20:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[45495]: audit 2022-04-22T18:20:36.468990+0000 mon.c (mon.1) 119 : audit [DBG] from='client.? 172.21.15.115:0/583128716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: cluster 2022-04-22T18:20:35.372184+0000 mgr.y (mgr.34107) 1906 : cluster [DBG] pgmap v3041: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 15 KiB/s wr, 275 op/s 2022-04-22T18:20:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: cluster 2022-04-22T18:20:35.497348+0000 mon.a (mon.0) 4327 : cluster [DBG] osdmap e1685: 8 total, 8 up, 8 in 2022-04-22T18:20:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: audit 2022-04-22T18:20:35.662596+0000 mgr.y (mgr.34107) 1907 : audit [DBG] from='client.37293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: audit 2022-04-22T18:20:36.014929+0000 mon.a (mon.0) 4328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: cluster 2022-04-22T18:20:36.354516+0000 mon.a (mon.0) 4329 : cluster [DBG] osdmap e1686: 8 total, 8 up, 8 in 2022-04-22T18:20:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:36 smithi115 bash[46934]: audit 2022-04-22T18:20:36.468990+0000 mon.c (mon.1) 119 : audit [DBG] from='client.? 172.21.15.115:0/583128716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:20:37.642 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_with_params2 ... ok 2022-04-22T18:20:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:37 smithi121 bash[36902]: audit 2022-04-22T18:20:36.010129+0000 mgr.y (mgr.34107) 1908 : audit [DBG] from='client.37299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:37.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:37 smithi121 bash[36902]: audit 2022-04-22T18:20:36.501258+0000 mon.a (mon.0) 4330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:37 smithi115 bash[45495]: audit 2022-04-22T18:20:36.010129+0000 mgr.y (mgr.34107) 1908 : audit [DBG] from='client.37299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:37 smithi115 bash[45495]: audit 2022-04-22T18:20:36.501258+0000 mon.a (mon.0) 4330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:37 smithi115 bash[46934]: audit 2022-04-22T18:20:36.010129+0000 mgr.y (mgr.34107) 1908 : audit [DBG] from='client.37299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:20:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:37 smithi115 bash[46934]: audit 2022-04-22T18:20:36.501258+0000 mon.a (mon.0) 4330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[45495]: cluster 2022-04-22T18:20:37.373295+0000 mgr.y (mgr.34107) 1909 : cluster [DBG] pgmap v3044: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 11 KiB/s wr, 245 op/s 2022-04-22T18:20:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[45495]: cluster 2022-04-22T18:20:37.505829+0000 mon.a (mon.0) 4331 : cluster [DBG] osdmap e1687: 8 total, 8 up, 8 in 2022-04-22T18:20:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[45495]: audit 2022-04-22T18:20:38.390921+0000 mon.a (mon.0) 4332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[46934]: cluster 2022-04-22T18:20:37.373295+0000 mgr.y (mgr.34107) 1909 : cluster [DBG] pgmap v3044: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 11 KiB/s wr, 245 op/s 2022-04-22T18:20:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[46934]: cluster 2022-04-22T18:20:37.505829+0000 mon.a (mon.0) 4331 : cluster [DBG] osdmap e1687: 8 total, 8 up, 8 in 2022-04-22T18:20:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:38 smithi115 bash[46934]: audit 2022-04-22T18:20:38.390921+0000 mon.a (mon.0) 4332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:38 smithi121 bash[36902]: cluster 2022-04-22T18:20:37.373295+0000 mgr.y (mgr.34107) 1909 : cluster [DBG] pgmap v3044: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 11 KiB/s wr, 245 op/s 2022-04-22T18:20:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:38 smithi121 bash[36902]: cluster 2022-04-22T18:20:37.505829+0000 mon.a (mon.0) 4331 : cluster [DBG] osdmap e1687: 8 total, 8 up, 8 in 2022-04-22T18:20:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:38 smithi121 bash[36902]: audit 2022-04-22T18:20:38.390921+0000 mon.a (mon.0) 4332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:39.677 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_with_params3 ... SKIP 2022-04-22T18:20:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:39 smithi115 bash[45495]: cluster 2022-04-22T18:20:38.526266+0000 mon.a (mon.0) 4333 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2022-04-22T18:20:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:39 smithi115 bash[46934]: cluster 2022-04-22T18:20:38.526266+0000 mon.a (mon.0) 4333 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2022-04-22T18:20:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:39 smithi121 bash[36902]: cluster 2022-04-22T18:20:38.526266+0000 mon.a (mon.0) 4333 : cluster [DBG] osdmap e1688: 8 total, 8 up, 8 in 2022-04-22T18:20:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:40 smithi115 bash[45495]: cluster 2022-04-22T18:20:39.374333+0000 mgr.y (mgr.34107) 1910 : cluster [DBG] pgmap v3047: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 15 KiB/s wr, 294 op/s 2022-04-22T18:20:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:40 smithi115 bash[45495]: cluster 2022-04-22T18:20:39.534649+0000 mon.a (mon.0) 4334 : cluster [DBG] osdmap e1689: 8 total, 8 up, 8 in 2022-04-22T18:20:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:40 smithi115 bash[46934]: cluster 2022-04-22T18:20:39.374333+0000 mgr.y (mgr.34107) 1910 : cluster [DBG] pgmap v3047: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 15 KiB/s wr, 294 op/s 2022-04-22T18:20:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:40 smithi115 bash[46934]: cluster 2022-04-22T18:20:39.534649+0000 mon.a (mon.0) 4334 : cluster [DBG] osdmap e1689: 8 total, 8 up, 8 in 2022-04-22T18:20:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:40 smithi121 bash[36902]: cluster 2022-04-22T18:20:39.374333+0000 mgr.y (mgr.34107) 1910 : cluster [DBG] pgmap v3047: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 15 KiB/s wr, 294 op/s 2022-04-22T18:20:41.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:40 smithi121 bash[36902]: cluster 2022-04-22T18:20:39.534649+0000 mon.a (mon.0) 4334 : cluster [DBG] osdmap e1689: 8 total, 8 up, 8 in 2022-04-22T18:20:41.512 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestClone.test_write ... ok 2022-04-22T18:20:41.707 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_acquire_release_lock ... ok 2022-04-22T18:20:41.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[45495]: cluster 2022-04-22T18:20:40.544147+0000 mon.a (mon.0) 4335 : cluster [DBG] osdmap e1690: 8 total, 8 up, 8 in 2022-04-22T18:20:41.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[45495]: audit 2022-04-22T18:20:41.017492+0000 mon.a (mon.0) 4336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:41.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[45495]: cluster 2022-04-22T18:20:41.356905+0000 mon.a (mon.0) 4337 : cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2022-04-22T18:20:41.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[45495]: audit 2022-04-22T18:20:41.504040+0000 mon.a (mon.0) 4338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:41.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[46934]: cluster 2022-04-22T18:20:40.544147+0000 mon.a (mon.0) 4335 : cluster [DBG] osdmap e1690: 8 total, 8 up, 8 in 2022-04-22T18:20:41.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[46934]: audit 2022-04-22T18:20:41.017492+0000 mon.a (mon.0) 4336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:41.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[46934]: cluster 2022-04-22T18:20:41.356905+0000 mon.a (mon.0) 4337 : cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2022-04-22T18:20:41.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[46934]: audit 2022-04-22T18:20:41.504040+0000 mon.a (mon.0) 4338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:41 smithi121 bash[36902]: cluster 2022-04-22T18:20:40.544147+0000 mon.a (mon.0) 4335 : cluster [DBG] osdmap e1690: 8 total, 8 up, 8 in 2022-04-22T18:20:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:41 smithi121 bash[36902]: audit 2022-04-22T18:20:41.017492+0000 mon.a (mon.0) 4336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:41 smithi121 bash[36902]: cluster 2022-04-22T18:20:41.356905+0000 mon.a (mon.0) 4337 : cluster [DBG] osdmap e1691: 8 total, 8 up, 8 in 2022-04-22T18:20:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:41 smithi121 bash[36902]: audit 2022-04-22T18:20:41.504040+0000 mon.a (mon.0) 4338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:42.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:41.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:42.699 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_break_lock ... ok 2022-04-22T18:20:42.949 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_discard ... ok 2022-04-22T18:20:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:42 smithi121 bash[36902]: cluster 2022-04-22T18:20:41.374909+0000 mgr.y (mgr.34107) 1911 : cluster [DBG] pgmap v3051: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 83 KiB/s rd, 12 KiB/s wr, 127 op/s 2022-04-22T18:20:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:42 smithi121 bash[36902]: audit 2022-04-22T18:20:41.811081+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.115:0/983571928' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:42 smithi121 bash[36902]: audit 2022-04-22T18:20:41.811685+0000 mon.a (mon.0) 4339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:43.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[45495]: cluster 2022-04-22T18:20:41.374909+0000 mgr.y (mgr.34107) 1911 : cluster [DBG] pgmap v3051: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 83 KiB/s rd, 12 KiB/s wr, 127 op/s 2022-04-22T18:20:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[45495]: audit 2022-04-22T18:20:41.811081+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.115:0/983571928' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[45495]: audit 2022-04-22T18:20:41.811685+0000 mon.a (mon.0) 4339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[46934]: cluster 2022-04-22T18:20:41.374909+0000 mgr.y (mgr.34107) 1911 : cluster [DBG] pgmap v3051: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 83 KiB/s rd, 12 KiB/s wr, 127 op/s 2022-04-22T18:20:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[46934]: audit 2022-04-22T18:20:41.811081+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.115:0/983571928' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:42 smithi115 bash[46934]: audit 2022-04-22T18:20:41.811685+0000 mon.a (mon.0) 4339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]: dispatch 2022-04-22T18:20:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:43 smithi121 bash[36902]: audit 2022-04-22T18:20:42.550841+0000 mon.a (mon.0) 4340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]': finished 2022-04-22T18:20:44.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:43 smithi121 bash[36902]: cluster 2022-04-22T18:20:42.551037+0000 mon.a (mon.0) 4341 : cluster [DBG] osdmap e1692: 8 total, 8 up, 8 in 2022-04-22T18:20:44.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:43 smithi121 bash[36902]: audit 2022-04-22T18:20:43.394026+0000 mon.a (mon.0) 4342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:43.685Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[45495]: audit 2022-04-22T18:20:42.550841+0000 mon.a (mon.0) 4340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]': finished 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[45495]: cluster 2022-04-22T18:20:42.551037+0000 mon.a (mon.0) 4341 : cluster [DBG] osdmap e1692: 8 total, 8 up, 8 in 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[45495]: audit 2022-04-22T18:20:43.394026+0000 mon.a (mon.0) 4342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[46934]: audit 2022-04-22T18:20:42.550841+0000 mon.a (mon.0) 4340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.115:0/1437297017"}]': finished 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[46934]: cluster 2022-04-22T18:20:42.551037+0000 mon.a (mon.0) 4341 : cluster [DBG] osdmap e1692: 8 total, 8 up, 8 in 2022-04-22T18:20:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:43 smithi115 bash[46934]: audit 2022-04-22T18:20:43.394026+0000 mon.a (mon.0) 4342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:44.725 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_flatten ... ok 2022-04-22T18:20:44.981 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_resize ... ok 2022-04-22T18:20:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:44 smithi121 bash[36902]: cluster 2022-04-22T18:20:43.376076+0000 mgr.y (mgr.34107) 1912 : cluster [DBG] pgmap v3053: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 263 KiB/s rd, 17 KiB/s wr, 356 op/s 2022-04-22T18:20:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:44 smithi121 bash[36902]: cluster 2022-04-22T18:20:43.566828+0000 mon.a (mon.0) 4343 : cluster [DBG] osdmap e1693: 8 total, 8 up, 8 in 2022-04-22T18:20:45.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:44 smithi115 bash[46934]: cluster 2022-04-22T18:20:43.376076+0000 mgr.y (mgr.34107) 1912 : cluster [DBG] pgmap v3053: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 263 KiB/s rd, 17 KiB/s wr, 356 op/s 2022-04-22T18:20:45.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:44 smithi115 bash[46934]: cluster 2022-04-22T18:20:43.566828+0000 mon.a (mon.0) 4343 : cluster [DBG] osdmap e1693: 8 total, 8 up, 8 in 2022-04-22T18:20:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:44 smithi115 bash[45495]: cluster 2022-04-22T18:20:43.376076+0000 mgr.y (mgr.34107) 1912 : cluster [DBG] pgmap v3053: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 263 KiB/s rd, 17 KiB/s wr, 356 op/s 2022-04-22T18:20:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:44 smithi115 bash[45495]: cluster 2022-04-22T18:20:43.566828+0000 mon.a (mon.0) 4343 : cluster [DBG] osdmap e1693: 8 total, 8 up, 8 in 2022-04-22T18:20:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:45 smithi121 bash[36902]: cluster 2022-04-22T18:20:44.577273+0000 mon.a (mon.0) 4344 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2022-04-22T18:20:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:45 smithi115 bash[45495]: cluster 2022-04-22T18:20:44.577273+0000 mon.a (mon.0) 4344 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2022-04-22T18:20:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:45 smithi115 bash[46934]: cluster 2022-04-22T18:20:44.577273+0000 mon.a (mon.0) 4344 : cluster [DBG] osdmap e1694: 8 total, 8 up, 8 in 2022-04-22T18:20:46.523 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_snap_create ... ok 2022-04-22T18:20:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:46 smithi121 bash[36902]: cluster 2022-04-22T18:20:45.377270+0000 mgr.y (mgr.34107) 1913 : cluster [DBG] pgmap v3056: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 23 KiB/s wr, 573 op/s 2022-04-22T18:20:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:46 smithi121 bash[36902]: cluster 2022-04-22T18:20:45.588259+0000 mon.a (mon.0) 4345 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2022-04-22T18:20:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:46 smithi121 bash[36902]: audit 2022-04-22T18:20:46.020446+0000 mon.a (mon.0) 4346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:46 smithi121 bash[36902]: cluster 2022-04-22T18:20:46.358766+0000 mon.a (mon.0) 4347 : cluster [DBG] osdmap e1696: 8 total, 8 up, 8 in 2022-04-22T18:20:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:46 smithi121 bash[36902]: audit 2022-04-22T18:20:46.507327+0000 mon.a (mon.0) 4348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[45495]: cluster 2022-04-22T18:20:45.377270+0000 mgr.y (mgr.34107) 1913 : cluster [DBG] pgmap v3056: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 23 KiB/s wr, 573 op/s 2022-04-22T18:20:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[45495]: cluster 2022-04-22T18:20:45.588259+0000 mon.a (mon.0) 4345 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2022-04-22T18:20:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[45495]: audit 2022-04-22T18:20:46.020446+0000 mon.a (mon.0) 4346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[45495]: cluster 2022-04-22T18:20:46.358766+0000 mon.a (mon.0) 4347 : cluster [DBG] osdmap e1696: 8 total, 8 up, 8 in 2022-04-22T18:20:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[45495]: audit 2022-04-22T18:20:46.507327+0000 mon.a (mon.0) 4348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[46934]: cluster 2022-04-22T18:20:45.377270+0000 mgr.y (mgr.34107) 1913 : cluster [DBG] pgmap v3056: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 431 KiB/s rd, 23 KiB/s wr, 573 op/s 2022-04-22T18:20:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[46934]: cluster 2022-04-22T18:20:45.588259+0000 mon.a (mon.0) 4345 : cluster [DBG] osdmap e1695: 8 total, 8 up, 8 in 2022-04-22T18:20:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[46934]: audit 2022-04-22T18:20:46.020446+0000 mon.a (mon.0) 4346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[46934]: cluster 2022-04-22T18:20:46.358766+0000 mon.a (mon.0) 4347 : cluster [DBG] osdmap e1696: 8 total, 8 up, 8 in 2022-04-22T18:20:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:46 smithi115 bash[46934]: audit 2022-04-22T18:20:46.507327+0000 mon.a (mon.0) 4348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:48.750 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_snap_rollback ... ok 2022-04-22T18:20:48.983 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_follower_write ... ok 2022-04-22T18:20:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:48 smithi121 bash[36902]: cluster 2022-04-22T18:20:47.378371+0000 mgr.y (mgr.34107) 1914 : cluster [DBG] pgmap v3059: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 265 KiB/s rd, 15 KiB/s wr, 355 op/s 2022-04-22T18:20:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:48 smithi121 bash[36902]: cluster 2022-04-22T18:20:47.595591+0000 mon.a (mon.0) 4349 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2022-04-22T18:20:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:48 smithi121 bash[36902]: audit 2022-04-22T18:20:48.397265+0000 mon.a (mon.0) 4350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[45495]: cluster 2022-04-22T18:20:47.378371+0000 mgr.y (mgr.34107) 1914 : cluster [DBG] pgmap v3059: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 265 KiB/s rd, 15 KiB/s wr, 355 op/s 2022-04-22T18:20:49.161 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[45495]: cluster 2022-04-22T18:20:47.595591+0000 mon.a (mon.0) 4349 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2022-04-22T18:20:49.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[45495]: audit 2022-04-22T18:20:48.397265+0000 mon.a (mon.0) 4350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:49.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[46934]: cluster 2022-04-22T18:20:47.378371+0000 mgr.y (mgr.34107) 1914 : cluster [DBG] pgmap v3059: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 265 KiB/s rd, 15 KiB/s wr, 355 op/s 2022-04-22T18:20:49.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[46934]: cluster 2022-04-22T18:20:47.595591+0000 mon.a (mon.0) 4349 : cluster [DBG] osdmap e1697: 8 total, 8 up, 8 in 2022-04-22T18:20:49.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:48 smithi115 bash[46934]: audit 2022-04-22T18:20:48.397265+0000 mon.a (mon.0) 4350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:49.192 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_ownership ... ok 2022-04-22T18:20:49.378 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_read_only_leadership ... ok 2022-04-22T18:20:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:49 smithi121 bash[36902]: cluster 2022-04-22T18:20:48.605838+0000 mon.a (mon.0) 4351 : cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2022-04-22T18:20:50.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:49 smithi115 bash[45495]: cluster 2022-04-22T18:20:48.605838+0000 mon.a (mon.0) 4351 : cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2022-04-22T18:20:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:49 smithi115 bash[46934]: cluster 2022-04-22T18:20:48.605838+0000 mon.a (mon.0) 4351 : cluster [DBG] osdmap e1698: 8 total, 8 up, 8 in 2022-04-22T18:20:50.779 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestExclusiveLock.test_snapshot_leadership ... ok 2022-04-22T18:20:50.975 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_add ... ok 2022-04-22T18:20:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:50 smithi121 bash[36902]: cluster 2022-04-22T18:20:49.379310+0000 mgr.y (mgr.34107) 1915 : cluster [DBG] pgmap v3062: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 279 op/s 2022-04-22T18:20:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:50 smithi121 bash[36902]: cluster 2022-04-22T18:20:49.616436+0000 mon.a (mon.0) 4352 : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2022-04-22T18:20:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:50 smithi115 bash[45495]: cluster 2022-04-22T18:20:49.379310+0000 mgr.y (mgr.34107) 1915 : cluster [DBG] pgmap v3062: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 279 op/s 2022-04-22T18:20:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:50 smithi115 bash[45495]: cluster 2022-04-22T18:20:49.616436+0000 mon.a (mon.0) 4352 : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2022-04-22T18:20:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:50 smithi115 bash[46934]: cluster 2022-04-22T18:20:49.379310+0000 mgr.y (mgr.34107) 1915 : cluster [DBG] pgmap v3062: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 279 op/s 2022-04-22T18:20:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:50 smithi115 bash[46934]: cluster 2022-04-22T18:20:49.616436+0000 mon.a (mon.0) 4352 : cluster [DBG] osdmap e1699: 8 total, 8 up, 8 in 2022-04-22T18:20:51.171 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_list ... ok 2022-04-22T18:20:51.340 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_list_empty ... ok 2022-04-22T18:20:51.561 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_list_move_to_trash ... ok 2022-04-22T18:20:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:51 smithi121 bash[36902]: cluster 2022-04-22T18:20:50.623736+0000 mon.a (mon.0) 4353 : cluster [DBG] osdmap e1700: 8 total, 8 up, 8 in 2022-04-22T18:20:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:51 smithi121 bash[36902]: audit 2022-04-22T18:20:51.023085+0000 mon.a (mon.0) 4354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:51 smithi121 bash[36902]: cluster 2022-04-22T18:20:51.359045+0000 mon.a (mon.0) 4355 : cluster [DBG] osdmap e1701: 8 total, 8 up, 8 in 2022-04-22T18:20:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:51 smithi121 bash[36902]: audit 2022-04-22T18:20:51.510457+0000 mon.a (mon.0) 4356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:51.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:52.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[45495]: cluster 2022-04-22T18:20:50.623736+0000 mon.a (mon.0) 4353 : cluster [DBG] osdmap e1700: 8 total, 8 up, 8 in 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[45495]: audit 2022-04-22T18:20:51.023085+0000 mon.a (mon.0) 4354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[45495]: cluster 2022-04-22T18:20:51.359045+0000 mon.a (mon.0) 4355 : cluster [DBG] osdmap e1701: 8 total, 8 up, 8 in 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[45495]: audit 2022-04-22T18:20:51.510457+0000 mon.a (mon.0) 4356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[46934]: cluster 2022-04-22T18:20:50.623736+0000 mon.a (mon.0) 4353 : cluster [DBG] osdmap e1700: 8 total, 8 up, 8 in 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[46934]: audit 2022-04-22T18:20:51.023085+0000 mon.a (mon.0) 4354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[46934]: cluster 2022-04-22T18:20:51.359045+0000 mon.a (mon.0) 4355 : cluster [DBG] osdmap e1701: 8 total, 8 up, 8 in 2022-04-22T18:20:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:51 smithi115 bash[46934]: audit 2022-04-22T18:20:51.510457+0000 mon.a (mon.0) 4356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:52 smithi121 bash[36902]: cluster 2022-04-22T18:20:51.380009+0000 mgr.y (mgr.34107) 1916 : cluster [DBG] pgmap v3066: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 18 KiB/s wr, 251 op/s 2022-04-22T18:20:53.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:52 smithi115 bash[46934]: cluster 2022-04-22T18:20:51.380009+0000 mgr.y (mgr.34107) 1916 : cluster [DBG] pgmap v3066: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 18 KiB/s wr, 251 op/s 2022-04-22T18:20:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:52 smithi115 bash[45495]: cluster 2022-04-22T18:20:51.380009+0000 mgr.y (mgr.34107) 1916 : cluster [DBG] pgmap v3066: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 18 KiB/s wr, 251 op/s 2022-04-22T18:20:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:53 smithi121 bash[36902]: audit 2022-04-22T18:20:53.400427+0000 mon.a (mon.0) 4357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:20:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:20:53.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:20:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:53 smithi115 bash[45495]: audit 2022-04-22T18:20:53.400427+0000 mon.a (mon.0) 4357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:53 smithi115 bash[46934]: audit 2022-04-22T18:20:53.400427+0000 mon.a (mon.0) 4357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:54.767 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_many_images ... ok 2022-04-22T18:20:55.006 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_image_remove ... ok 2022-04-22T18:20:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:54 smithi121 bash[36902]: cluster 2022-04-22T18:20:53.381026+0000 mgr.y (mgr.34107) 1917 : cluster [DBG] pgmap v3067: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 369 KiB/s rd, 40 KiB/s wr, 518 op/s 2022-04-22T18:20:55.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:54 smithi115 bash[45495]: cluster 2022-04-22T18:20:53.381026+0000 mgr.y (mgr.34107) 1917 : cluster [DBG] pgmap v3067: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 369 KiB/s rd, 40 KiB/s wr, 518 op/s 2022-04-22T18:20:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:54 smithi115 bash[46934]: cluster 2022-04-22T18:20:53.381026+0000 mgr.y (mgr.34107) 1917 : cluster [DBG] pgmap v3067: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 369 KiB/s rd, 40 KiB/s wr, 518 op/s 2022-04-22T18:20:55.176 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap ... ok 2022-04-22T18:20:55.388 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap_flags ... ok 2022-04-22T18:20:55.718 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap_list_many ... ok 2022-04-22T18:20:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:56 smithi121 bash[36902]: cluster 2022-04-22T18:20:55.382186+0000 mgr.y (mgr.34107) 1918 : cluster [DBG] pgmap v3068: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 428 KiB/s rd, 57 KiB/s wr, 629 op/s 2022-04-22T18:20:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:56 smithi121 bash[36902]: audit 2022-04-22T18:20:56.026380+0000 mon.a (mon.0) 4358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:56 smithi121 bash[36902]: cluster 2022-04-22T18:20:56.365773+0000 mon.a (mon.0) 4359 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2022-04-22T18:20:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:56 smithi121 bash[36902]: audit 2022-04-22T18:20:56.513809+0000 mon.a (mon.0) 4360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:57.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[45495]: cluster 2022-04-22T18:20:55.382186+0000 mgr.y (mgr.34107) 1918 : cluster [DBG] pgmap v3068: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 428 KiB/s rd, 57 KiB/s wr, 629 op/s 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[45495]: audit 2022-04-22T18:20:56.026380+0000 mon.a (mon.0) 4358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[45495]: cluster 2022-04-22T18:20:56.365773+0000 mon.a (mon.0) 4359 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[45495]: audit 2022-04-22T18:20:56.513809+0000 mon.a (mon.0) 4360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[46934]: cluster 2022-04-22T18:20:55.382186+0000 mgr.y (mgr.34107) 1918 : cluster [DBG] pgmap v3068: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 428 KiB/s rd, 57 KiB/s wr, 629 op/s 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[46934]: audit 2022-04-22T18:20:56.026380+0000 mon.a (mon.0) 4358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:20:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[46934]: cluster 2022-04-22T18:20:56.365773+0000 mon.a (mon.0) 4359 : cluster [DBG] osdmap e1702: 8 total, 8 up, 8 in 2022-04-22T18:20:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:56 smithi115 bash[46934]: audit 2022-04-22T18:20:56.513809+0000 mon.a (mon.0) 4360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:20:57.551 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap_namespace ... ok 2022-04-22T18:20:57.753 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap_rename ... ok 2022-04-22T18:20:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:58 smithi121 bash[36902]: cluster 2022-04-22T18:20:57.378937+0000 mon.a (mon.0) 4361 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2022-04-22T18:20:58.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:58 smithi121 bash[36902]: cluster 2022-04-22T18:20:57.382833+0000 mgr.y (mgr.34107) 1919 : cluster [DBG] pgmap v3071: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 61 KiB/s wr, 804 op/s 2022-04-22T18:20:58.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:58 smithi115 bash[45495]: cluster 2022-04-22T18:20:57.378937+0000 mon.a (mon.0) 4361 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2022-04-22T18:20:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:58 smithi115 bash[45495]: cluster 2022-04-22T18:20:57.382833+0000 mgr.y (mgr.34107) 1919 : cluster [DBG] pgmap v3071: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 61 KiB/s wr, 804 op/s 2022-04-22T18:20:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:58 smithi115 bash[46934]: cluster 2022-04-22T18:20:57.378937+0000 mon.a (mon.0) 4361 : cluster [DBG] osdmap e1703: 8 total, 8 up, 8 in 2022-04-22T18:20:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:58 smithi115 bash[46934]: cluster 2022-04-22T18:20:57.382833+0000 mgr.y (mgr.34107) 1919 : cluster [DBG] pgmap v3071: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 562 KiB/s rd, 61 KiB/s wr, 804 op/s 2022-04-22T18:20:59.565 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestGroups.test_group_snap_rollback ... ok 2022-04-22T18:20:59.697 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_access_timestamp ... ok 2022-04-22T18:20:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:59 smithi121 bash[36902]: cluster 2022-04-22T18:20:58.387417+0000 mon.a (mon.0) 4362 : cluster [DBG] osdmap e1704: 8 total, 8 up, 8 in 2022-04-22T18:20:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:20:59 smithi121 bash[36902]: audit 2022-04-22T18:20:58.403084+0000 mon.a (mon.0) 4363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:59 smithi115 bash[46934]: cluster 2022-04-22T18:20:58.387417+0000 mon.a (mon.0) 4362 : cluster [DBG] osdmap e1704: 8 total, 8 up, 8 in 2022-04-22T18:20:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:20:59 smithi115 bash[46934]: audit 2022-04-22T18:20:58.403084+0000 mon.a (mon.0) 4363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:59 smithi115 bash[45495]: cluster 2022-04-22T18:20:58.387417+0000 mon.a (mon.0) 4362 : cluster [DBG] osdmap e1704: 8 total, 8 up, 8 in 2022-04-22T18:20:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:20:59 smithi115 bash[45495]: audit 2022-04-22T18:20:58.403084+0000 mon.a (mon.0) 4363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:20:59.859 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_aio_discard ... ok 2022-04-22T18:21:00.009 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_aio_flush ... ok 2022-04-22T18:21:00.176 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_aio_read ... ok 2022-04-22T18:21:00.344 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_aio_write ... ok 2022-04-22T18:21:00.509 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_aio_write_zeroes ... ok 2022-04-22T18:21:00.660 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_block_name_prefix ... ok 2022-04-22T18:21:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:00 smithi121 bash[36902]: cluster 2022-04-22T18:20:59.383871+0000 mgr.y (mgr.34107) 1920 : cluster [DBG] pgmap v3073: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 573 KiB/s rd, 60 KiB/s wr, 822 op/s 2022-04-22T18:21:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:00 smithi121 bash[36902]: cluster 2022-04-22T18:20:59.393865+0000 mon.a (mon.0) 4364 : cluster [DBG] osdmap e1705: 8 total, 8 up, 8 in 2022-04-22T18:21:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:00 smithi115 bash[45495]: cluster 2022-04-22T18:20:59.383871+0000 mgr.y (mgr.34107) 1920 : cluster [DBG] pgmap v3073: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 573 KiB/s rd, 60 KiB/s wr, 822 op/s 2022-04-22T18:21:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:00 smithi115 bash[45495]: cluster 2022-04-22T18:20:59.393865+0000 mon.a (mon.0) 4364 : cluster [DBG] osdmap e1705: 8 total, 8 up, 8 in 2022-04-22T18:21:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:00 smithi115 bash[46934]: cluster 2022-04-22T18:20:59.383871+0000 mgr.y (mgr.34107) 1920 : cluster [DBG] pgmap v3073: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 573 KiB/s rd, 60 KiB/s wr, 822 op/s 2022-04-22T18:21:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:00 smithi115 bash[46934]: cluster 2022-04-22T18:20:59.393865+0000 mon.a (mon.0) 4364 : cluster [DBG] osdmap e1705: 8 total, 8 up, 8 in 2022-04-22T18:21:00.884 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_config_list ... ok 2022-04-22T18:21:01.276 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_copy ... ok 2022-04-22T18:21:01.683 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_copy2 ... ok 2022-04-22T18:21:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:01 smithi121 bash[36902]: audit 2022-04-22T18:21:01.028956+0000 mon.a (mon.0) 4365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:01.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:01 smithi115 bash[45495]: audit 2022-04-22T18:21:01.028956+0000 mon.a (mon.0) 4365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:01 smithi115 bash[46934]: audit 2022-04-22T18:21:01.028956+0000 mon.a (mon.0) 4365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:01.828 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_copy3 ... SKIP 2022-04-22T18:21:02.136 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:01.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:02 smithi121 bash[36902]: cluster 2022-04-22T18:21:01.385174+0000 mgr.y (mgr.34107) 1921 : cluster [DBG] pgmap v3075: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 513 KiB/s rd, 40 KiB/s wr, 704 op/s 2022-04-22T18:21:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:02 smithi121 bash[36902]: audit 2022-04-22T18:21:01.517186+0000 mon.a (mon.0) 4366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:02 smithi121 bash[36902]: audit 2022-04-22T18:21:02.109004+0000 mon.a (mon.0) 4367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:21:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:02 smithi121 bash[36902]: audit 2022-04-22T18:21:02.112250+0000 mon.a (mon.0) 4368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:21:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:02 smithi121 bash[36902]: audit 2022-04-22T18:21:02.113953+0000 mon.a (mon.0) 4369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:21:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[45495]: cluster 2022-04-22T18:21:01.385174+0000 mgr.y (mgr.34107) 1921 : cluster [DBG] pgmap v3075: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 513 KiB/s rd, 40 KiB/s wr, 704 op/s 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[45495]: audit 2022-04-22T18:21:01.517186+0000 mon.a (mon.0) 4366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[45495]: audit 2022-04-22T18:21:02.109004+0000 mon.a (mon.0) 4367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[45495]: audit 2022-04-22T18:21:02.112250+0000 mon.a (mon.0) 4368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[45495]: audit 2022-04-22T18:21:02.113953+0000 mon.a (mon.0) 4369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[46934]: cluster 2022-04-22T18:21:01.385174+0000 mgr.y (mgr.34107) 1921 : cluster [DBG] pgmap v3075: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 513 KiB/s rd, 40 KiB/s wr, 704 op/s 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[46934]: audit 2022-04-22T18:21:01.517186+0000 mon.a (mon.0) 4366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[46934]: audit 2022-04-22T18:21:02.109004+0000 mon.a (mon.0) 4367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:21:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[46934]: audit 2022-04-22T18:21:02.112250+0000 mon.a (mon.0) 4368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:21:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:02 smithi115 bash[46934]: audit 2022-04-22T18:21:02.113953+0000 mon.a (mon.0) 4369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:21:03.573 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_create_snap ... ok 2022-04-22T18:21:03.694 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:03 smithi115 bash[45495]: cluster 2022-04-22T18:21:02.399020+0000 mon.a (mon.0) 4370 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2022-04-22T18:21:03.694 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:03 smithi115 bash[45495]: audit 2022-04-22T18:21:02.472140+0000 mon.a (mon.0) 4371 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:21:03.695 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:03 smithi115 bash[46934]: cluster 2022-04-22T18:21:02.399020+0000 mon.a (mon.0) 4370 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2022-04-22T18:21:03.695 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:03 smithi115 bash[46934]: audit 2022-04-22T18:21:02.472140+0000 mon.a (mon.0) 4371 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:21:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:03 smithi121 bash[36902]: cluster 2022-04-22T18:21:02.399020+0000 mon.a (mon.0) 4370 : cluster [DBG] osdmap e1706: 8 total, 8 up, 8 in 2022-04-22T18:21:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:03 smithi121 bash[36902]: audit 2022-04-22T18:21:02.472140+0000 mon.a (mon.0) 4371 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:21:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:03.688Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:04 smithi121 bash[36902]: cluster 2022-04-22T18:21:03.386294+0000 mgr.y (mgr.34107) 1922 : cluster [DBG] pgmap v3077: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 563 KiB/s rd, 730 KiB/s wr, 768 op/s 2022-04-22T18:21:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:04 smithi121 bash[36902]: audit 2022-04-22T18:21:03.410380+0000 mon.a (mon.0) 4372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:04 smithi121 bash[36902]: cluster 2022-04-22T18:21:03.419229+0000 mon.a (mon.0) 4373 : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2022-04-22T18:21:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[45495]: cluster 2022-04-22T18:21:03.386294+0000 mgr.y (mgr.34107) 1922 : cluster [DBG] pgmap v3077: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 563 KiB/s rd, 730 KiB/s wr, 768 op/s 2022-04-22T18:21:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[45495]: audit 2022-04-22T18:21:03.410380+0000 mon.a (mon.0) 4372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[45495]: cluster 2022-04-22T18:21:03.419229+0000 mon.a (mon.0) 4373 : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2022-04-22T18:21:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[46934]: cluster 2022-04-22T18:21:03.386294+0000 mgr.y (mgr.34107) 1922 : cluster [DBG] pgmap v3077: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 563 KiB/s rd, 730 KiB/s wr, 768 op/s 2022-04-22T18:21:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[46934]: audit 2022-04-22T18:21:03.410380+0000 mon.a (mon.0) 4372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:04 smithi115 bash[46934]: cluster 2022-04-22T18:21:03.419229+0000 mon.a (mon.0) 4373 : cluster [DBG] osdmap e1707: 8 total, 8 up, 8 in 2022-04-22T18:21:05.644 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_create_snap_exists ... ok 2022-04-22T18:21:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:05 smithi121 bash[36902]: cluster 2022-04-22T18:21:04.479260+0000 mon.a (mon.0) 4374 : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2022-04-22T18:21:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:05 smithi115 bash[45495]: cluster 2022-04-22T18:21:04.479260+0000 mon.a (mon.0) 4374 : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2022-04-22T18:21:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:05 smithi115 bash[46934]: cluster 2022-04-22T18:21:04.479260+0000 mon.a (mon.0) 4374 : cluster [DBG] osdmap e1708: 8 total, 8 up, 8 in 2022-04-22T18:21:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:06 smithi121 bash[36902]: cluster 2022-04-22T18:21:05.387341+0000 mgr.y (mgr.34107) 1923 : cluster [DBG] pgmap v3080: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2022-04-22T18:21:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:06 smithi121 bash[36902]: cluster 2022-04-22T18:21:05.486276+0000 mon.a (mon.0) 4375 : cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2022-04-22T18:21:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:06 smithi121 bash[36902]: audit 2022-04-22T18:21:06.032159+0000 mon.a (mon.0) 4376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:06 smithi121 bash[36902]: cluster 2022-04-22T18:21:06.366343+0000 mon.a (mon.0) 4377 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2022-04-22T18:21:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[46934]: cluster 2022-04-22T18:21:05.387341+0000 mgr.y (mgr.34107) 1923 : cluster [DBG] pgmap v3080: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2022-04-22T18:21:06.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[46934]: cluster 2022-04-22T18:21:05.486276+0000 mon.a (mon.0) 4375 : cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2022-04-22T18:21:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[46934]: audit 2022-04-22T18:21:06.032159+0000 mon.a (mon.0) 4376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[46934]: cluster 2022-04-22T18:21:06.366343+0000 mon.a (mon.0) 4377 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2022-04-22T18:21:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[45495]: cluster 2022-04-22T18:21:05.387341+0000 mgr.y (mgr.34107) 1923 : cluster [DBG] pgmap v3080: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2022-04-22T18:21:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[45495]: cluster 2022-04-22T18:21:05.486276+0000 mon.a (mon.0) 4375 : cluster [DBG] osdmap e1709: 8 total, 8 up, 8 in 2022-04-22T18:21:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[45495]: audit 2022-04-22T18:21:06.032159+0000 mon.a (mon.0) 4376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:06 smithi115 bash[45495]: cluster 2022-04-22T18:21:06.366343+0000 mon.a (mon.0) 4377 : cluster [DBG] osdmap e1710: 8 total, 8 up, 8 in 2022-04-22T18:21:06.847 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:21:07.191 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:21:07.191 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (44m) 2m ago 57m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:21:07.191 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (57m) 3m ago 57m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:21:07.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (46m) 2m ago 58m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:21:07.192 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (45m) 3m ago 58m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:21:07.192 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (53m) 3m ago 63m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (54m) 2m ago 65m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (53m) 2m ago 65m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (52m) 3m ago 63m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (52m) 2m ago 63m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (45m) 2m ago 58m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (45m) 3m ago 58m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (50m) 2m ago 62m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (50m) 2m ago 62m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (49m) 2m ago 61m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:21:07.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (49m) 2m ago 61m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:21:07.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (48m) 3m ago 60m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:21:07.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (47m) 3m ago 60m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:21:07.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (47m) 3m ago 60m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:21:07.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (46m) 3m ago 59m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:21:07.194 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (44m) 3m ago 58m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:21:07.597 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:21:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:21:07.599 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:21:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:07 smithi121 bash[36902]: audit 2022-04-22T18:21:06.520357+0000 mon.a (mon.0) 4378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:07 smithi121 bash[36902]: cluster 2022-04-22T18:21:07.378842+0000 mon.a (mon.0) 4379 : cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2022-04-22T18:21:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:07 smithi115 bash[45495]: audit 2022-04-22T18:21:06.520357+0000 mon.a (mon.0) 4378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:07 smithi115 bash[45495]: cluster 2022-04-22T18:21:07.378842+0000 mon.a (mon.0) 4379 : cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2022-04-22T18:21:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:07 smithi115 bash[46934]: audit 2022-04-22T18:21:06.520357+0000 mon.a (mon.0) 4378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:07 smithi115 bash[46934]: cluster 2022-04-22T18:21:07.378842+0000 mon.a (mon.0) 4379 : cluster [DBG] osdmap e1711: 8 total, 8 up, 8 in 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: audit 2022-04-22T18:21:06.822533+0000 mgr.y (mgr.34107) 1924 : audit [DBG] from='client.47317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: audit 2022-04-22T18:21:07.164096+0000 mgr.y (mgr.34107) 1925 : audit [DBG] from='client.47323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: cluster 2022-04-22T18:21:07.387887+0000 mgr.y (mgr.34107) 1926 : cluster [DBG] pgmap v3084: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 1.0 MiB/s wr, 317 op/s 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: audit 2022-04-22T18:21:07.591171+0000 mon.a (mon.0) 4380 : audit [DBG] from='client.? 172.21.15.115:0/1241461148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: cluster 2022-04-22T18:21:08.387970+0000 mon.a (mon.0) 4381 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2022-04-22T18:21:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:08 smithi121 bash[36902]: audit 2022-04-22T18:21:08.409917+0000 mon.a (mon.0) 4382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: audit 2022-04-22T18:21:06.822533+0000 mgr.y (mgr.34107) 1924 : audit [DBG] from='client.47317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: audit 2022-04-22T18:21:07.164096+0000 mgr.y (mgr.34107) 1925 : audit [DBG] from='client.47323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: cluster 2022-04-22T18:21:07.387887+0000 mgr.y (mgr.34107) 1926 : cluster [DBG] pgmap v3084: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 1.0 MiB/s wr, 317 op/s 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: audit 2022-04-22T18:21:07.591171+0000 mon.a (mon.0) 4380 : audit [DBG] from='client.? 172.21.15.115:0/1241461148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: cluster 2022-04-22T18:21:08.387970+0000 mon.a (mon.0) 4381 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2022-04-22T18:21:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[45495]: audit 2022-04-22T18:21:08.409917+0000 mon.a (mon.0) 4382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: audit 2022-04-22T18:21:06.822533+0000 mgr.y (mgr.34107) 1924 : audit [DBG] from='client.47317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: audit 2022-04-22T18:21:07.164096+0000 mgr.y (mgr.34107) 1925 : audit [DBG] from='client.47323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: cluster 2022-04-22T18:21:07.387887+0000 mgr.y (mgr.34107) 1926 : cluster [DBG] pgmap v3084: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 1.0 MiB/s wr, 317 op/s 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: audit 2022-04-22T18:21:07.591171+0000 mon.a (mon.0) 4380 : audit [DBG] from='client.? 172.21.15.115:0/1241461148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: cluster 2022-04-22T18:21:08.387970+0000 mon.a (mon.0) 4381 : cluster [DBG] osdmap e1712: 8 total, 8 up, 8 in 2022-04-22T18:21:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:08 smithi115 bash[46934]: audit 2022-04-22T18:21:08.409917+0000 mon.a (mon.0) 4382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:10 smithi121 bash[36902]: cluster 2022-04-22T18:21:09.388836+0000 mgr.y (mgr.34107) 1927 : cluster [DBG] pgmap v3087: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 5.1 KiB/s wr, 99 op/s 2022-04-22T18:21:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:10 smithi121 bash[36902]: cluster 2022-04-22T18:21:09.392268+0000 mon.a (mon.0) 4383 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2022-04-22T18:21:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:10 smithi115 bash[45495]: cluster 2022-04-22T18:21:09.388836+0000 mgr.y (mgr.34107) 1927 : cluster [DBG] pgmap v3087: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 5.1 KiB/s wr, 99 op/s 2022-04-22T18:21:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:10 smithi115 bash[45495]: cluster 2022-04-22T18:21:09.392268+0000 mon.a (mon.0) 4383 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2022-04-22T18:21:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:10 smithi115 bash[46934]: cluster 2022-04-22T18:21:09.388836+0000 mgr.y (mgr.34107) 1927 : cluster [DBG] pgmap v3087: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 5.1 KiB/s wr, 99 op/s 2022-04-22T18:21:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:10 smithi115 bash[46934]: cluster 2022-04-22T18:21:09.392268+0000 mon.a (mon.0) 4383 : cluster [DBG] osdmap e1713: 8 total, 8 up, 8 in 2022-04-22T18:21:11.522 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_create_snap_flags ... ok 2022-04-22T18:21:11.688 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_create_timestamp ... ok 2022-04-22T18:21:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:11 smithi121 bash[36902]: cluster 2022-04-22T18:21:10.398473+0000 mon.a (mon.0) 4384 : cluster [DBG] osdmap e1714: 8 total, 8 up, 8 in 2022-04-22T18:21:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:11 smithi121 bash[36902]: audit 2022-04-22T18:21:11.035203+0000 mon.a (mon.0) 4385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:11 smithi121 bash[36902]: cluster 2022-04-22T18:21:11.368485+0000 mon.a (mon.0) 4386 : cluster [DBG] osdmap e1715: 8 total, 8 up, 8 in 2022-04-22T18:21:11.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[46934]: cluster 2022-04-22T18:21:10.398473+0000 mon.a (mon.0) 4384 : cluster [DBG] osdmap e1714: 8 total, 8 up, 8 in 2022-04-22T18:21:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[46934]: audit 2022-04-22T18:21:11.035203+0000 mon.a (mon.0) 4385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[46934]: cluster 2022-04-22T18:21:11.368485+0000 mon.a (mon.0) 4386 : cluster [DBG] osdmap e1715: 8 total, 8 up, 8 in 2022-04-22T18:21:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[45495]: cluster 2022-04-22T18:21:10.398473+0000 mon.a (mon.0) 4384 : cluster [DBG] osdmap e1714: 8 total, 8 up, 8 in 2022-04-22T18:21:11.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[45495]: audit 2022-04-22T18:21:11.035203+0000 mon.a (mon.0) 4385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:11.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[45495]: cluster 2022-04-22T18:21:11.368485+0000 mon.a (mon.0) 4386 : cluster [DBG] osdmap e1715: 8 total, 8 up, 8 in 2022-04-22T18:21:11.833 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_create_with_params ... SKIP 2022-04-22T18:21:12.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:11.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:12 smithi121 bash[36902]: cluster 2022-04-22T18:21:11.389407+0000 mgr.y (mgr.34107) 1928 : cluster [DBG] pgmap v3090: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 7.5 KiB/s wr, 96 op/s 2022-04-22T18:21:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:12 smithi121 bash[36902]: audit 2022-04-22T18:21:11.523813+0000 mon.a (mon.0) 4387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:12 smithi115 bash[45495]: cluster 2022-04-22T18:21:11.389407+0000 mgr.y (mgr.34107) 1928 : cluster [DBG] pgmap v3090: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 7.5 KiB/s wr, 96 op/s 2022-04-22T18:21:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:12 smithi115 bash[45495]: audit 2022-04-22T18:21:11.523813+0000 mon.a (mon.0) 4387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:12.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:12 smithi115 bash[46934]: cluster 2022-04-22T18:21:11.389407+0000 mgr.y (mgr.34107) 1928 : cluster [DBG] pgmap v3090: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 7.5 KiB/s wr, 96 op/s 2022-04-22T18:21:12.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:12 smithi115 bash[46934]: audit 2022-04-22T18:21:11.523813+0000 mon.a (mon.0) 4387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:13.697 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:13 smithi115 bash[45495]: cluster 2022-04-22T18:21:12.404046+0000 mon.a (mon.0) 4388 : cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2022-04-22T18:21:13.697 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:13 smithi115 bash[46934]: cluster 2022-04-22T18:21:12.404046+0000 mon.a (mon.0) 4388 : cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2022-04-22T18:21:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:13 smithi121 bash[36902]: cluster 2022-04-22T18:21:12.404046+0000 mon.a (mon.0) 4388 : cluster [DBG] osdmap e1716: 8 total, 8 up, 8 in 2022-04-22T18:21:14.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:13.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:14 smithi121 bash[36902]: cluster 2022-04-22T18:21:13.390500+0000 mgr.y (mgr.34107) 1929 : cluster [DBG] pgmap v3092: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 8.2 KiB/s wr, 91 op/s 2022-04-22T18:21:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:14 smithi121 bash[36902]: cluster 2022-04-22T18:21:13.421057+0000 mon.a (mon.0) 4389 : cluster [DBG] osdmap e1717: 8 total, 8 up, 8 in 2022-04-22T18:21:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:14 smithi121 bash[36902]: audit 2022-04-22T18:21:13.421531+0000 mon.a (mon.0) 4390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:14 smithi121 bash[36902]: audit 2022-04-22T18:21:13.901759+0000 mon.a (mon.0) 4391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:21:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:14 smithi121 bash[36902]: audit 2022-04-22T18:21:13.905129+0000 mon.a (mon.0) 4392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:21:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[46934]: cluster 2022-04-22T18:21:13.390500+0000 mgr.y (mgr.34107) 1929 : cluster [DBG] pgmap v3092: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 8.2 KiB/s wr, 91 op/s 2022-04-22T18:21:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[46934]: cluster 2022-04-22T18:21:13.421057+0000 mon.a (mon.0) 4389 : cluster [DBG] osdmap e1717: 8 total, 8 up, 8 in 2022-04-22T18:21:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[46934]: audit 2022-04-22T18:21:13.421531+0000 mon.a (mon.0) 4390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[46934]: audit 2022-04-22T18:21:13.901759+0000 mon.a (mon.0) 4391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:21:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[46934]: audit 2022-04-22T18:21:13.905129+0000 mon.a (mon.0) 4392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:21:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[45495]: cluster 2022-04-22T18:21:13.390500+0000 mgr.y (mgr.34107) 1929 : cluster [DBG] pgmap v3092: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 8.2 KiB/s wr, 91 op/s 2022-04-22T18:21:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[45495]: cluster 2022-04-22T18:21:13.421057+0000 mon.a (mon.0) 4389 : cluster [DBG] osdmap e1717: 8 total, 8 up, 8 in 2022-04-22T18:21:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[45495]: audit 2022-04-22T18:21:13.421531+0000 mon.a (mon.0) 4390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[45495]: audit 2022-04-22T18:21:13.901759+0000 mon.a (mon.0) 4391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:21:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:14 smithi115 bash[45495]: audit 2022-04-22T18:21:13.905129+0000 mon.a (mon.0) 4392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:21:15.733 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_deep_copy ... ok 2022-04-22T18:21:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:15 smithi121 bash[36902]: cluster 2022-04-22T18:21:14.422878+0000 mon.a (mon.0) 4393 : cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2022-04-22T18:21:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:15 smithi115 bash[45495]: cluster 2022-04-22T18:21:14.422878+0000 mon.a (mon.0) 4393 : cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2022-04-22T18:21:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:15 smithi115 bash[46934]: cluster 2022-04-22T18:21:14.422878+0000 mon.a (mon.0) 4393 : cluster [DBG] osdmap e1718: 8 total, 8 up, 8 in 2022-04-22T18:21:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:16 smithi121 bash[36902]: cluster 2022-04-22T18:21:15.391562+0000 mgr.y (mgr.34107) 1930 : cluster [DBG] pgmap v3095: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 9.7 KiB/s wr, 163 op/s 2022-04-22T18:21:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:16 smithi121 bash[36902]: cluster 2022-04-22T18:21:15.432914+0000 mon.a (mon.0) 4394 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2022-04-22T18:21:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:16 smithi121 bash[36902]: audit 2022-04-22T18:21:16.038150+0000 mon.a (mon.0) 4395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:16 smithi121 bash[36902]: cluster 2022-04-22T18:21:16.371808+0000 mon.a (mon.0) 4396 : cluster [DBG] osdmap e1720: 8 total, 8 up, 8 in 2022-04-22T18:21:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[46934]: cluster 2022-04-22T18:21:15.391562+0000 mgr.y (mgr.34107) 1930 : cluster [DBG] pgmap v3095: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 9.7 KiB/s wr, 163 op/s 2022-04-22T18:21:16.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[46934]: cluster 2022-04-22T18:21:15.432914+0000 mon.a (mon.0) 4394 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2022-04-22T18:21:16.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[46934]: audit 2022-04-22T18:21:16.038150+0000 mon.a (mon.0) 4395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:16.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[46934]: cluster 2022-04-22T18:21:16.371808+0000 mon.a (mon.0) 4396 : cluster [DBG] osdmap e1720: 8 total, 8 up, 8 in 2022-04-22T18:21:16.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[45495]: cluster 2022-04-22T18:21:15.391562+0000 mgr.y (mgr.34107) 1930 : cluster [DBG] pgmap v3095: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 9.7 KiB/s wr, 163 op/s 2022-04-22T18:21:16.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[45495]: cluster 2022-04-22T18:21:15.432914+0000 mon.a (mon.0) 4394 : cluster [DBG] osdmap e1719: 8 total, 8 up, 8 in 2022-04-22T18:21:16.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[45495]: audit 2022-04-22T18:21:16.038150+0000 mon.a (mon.0) 4395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:16.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:16 smithi115 bash[45495]: cluster 2022-04-22T18:21:16.371808+0000 mon.a (mon.0) 4396 : cluster [DBG] osdmap e1720: 8 total, 8 up, 8 in 2022-04-22T18:21:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:17 smithi121 bash[36902]: audit 2022-04-22T18:21:16.527171+0000 mon.a (mon.0) 4397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:17 smithi115 bash[45495]: audit 2022-04-22T18:21:16.527171+0000 mon.a (mon.0) 4397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:17 smithi115 bash[46934]: audit 2022-04-22T18:21:16.527171+0000 mon.a (mon.0) 4397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:18 smithi121 bash[36902]: cluster 2022-04-22T18:21:17.392870+0000 mgr.y (mgr.34107) 1931 : cluster [DBG] pgmap v3098: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 9.7 KiB/s wr, 194 op/s 2022-04-22T18:21:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:18 smithi121 bash[36902]: cluster 2022-04-22T18:21:17.446576+0000 mon.a (mon.0) 4398 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2022-04-22T18:21:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:18 smithi121 bash[36902]: audit 2022-04-22T18:21:18.414935+0000 mon.a (mon.0) 4399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[45495]: cluster 2022-04-22T18:21:17.392870+0000 mgr.y (mgr.34107) 1931 : cluster [DBG] pgmap v3098: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 9.7 KiB/s wr, 194 op/s 2022-04-22T18:21:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[45495]: cluster 2022-04-22T18:21:17.446576+0000 mon.a (mon.0) 4398 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2022-04-22T18:21:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[45495]: audit 2022-04-22T18:21:18.414935+0000 mon.a (mon.0) 4399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[46934]: cluster 2022-04-22T18:21:17.392870+0000 mgr.y (mgr.34107) 1931 : cluster [DBG] pgmap v3098: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 9.7 KiB/s wr, 194 op/s 2022-04-22T18:21:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[46934]: cluster 2022-04-22T18:21:17.446576+0000 mon.a (mon.0) 4398 : cluster [DBG] osdmap e1721: 8 total, 8 up, 8 in 2022-04-22T18:21:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:18 smithi115 bash[46934]: audit 2022-04-22T18:21:18.414935+0000 mon.a (mon.0) 4399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:19 smithi121 bash[36902]: cluster 2022-04-22T18:21:18.453905+0000 mon.a (mon.0) 4400 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2022-04-22T18:21:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:19 smithi115 bash[45495]: cluster 2022-04-22T18:21:18.453905+0000 mon.a (mon.0) 4400 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2022-04-22T18:21:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:19 smithi115 bash[46934]: cluster 2022-04-22T18:21:18.453905+0000 mon.a (mon.0) 4400 : cluster [DBG] osdmap e1722: 8 total, 8 up, 8 in 2022-04-22T18:21:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:20 smithi121 bash[36902]: cluster 2022-04-22T18:21:19.394009+0000 mgr.y (mgr.34107) 1932 : cluster [DBG] pgmap v3101: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 14 KiB/s wr, 228 op/s 2022-04-22T18:21:20.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:20 smithi121 bash[36902]: cluster 2022-04-22T18:21:19.466629+0000 mon.a (mon.0) 4401 : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2022-04-22T18:21:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:20 smithi115 bash[45495]: cluster 2022-04-22T18:21:19.394009+0000 mgr.y (mgr.34107) 1932 : cluster [DBG] pgmap v3101: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 14 KiB/s wr, 228 op/s 2022-04-22T18:21:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:20 smithi115 bash[45495]: cluster 2022-04-22T18:21:19.466629+0000 mon.a (mon.0) 4401 : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2022-04-22T18:21:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:20 smithi115 bash[46934]: cluster 2022-04-22T18:21:19.394009+0000 mgr.y (mgr.34107) 1932 : cluster [DBG] pgmap v3101: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 14 KiB/s wr, 228 op/s 2022-04-22T18:21:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:20 smithi115 bash[46934]: cluster 2022-04-22T18:21:19.466629+0000 mon.a (mon.0) 4401 : cluster [DBG] osdmap e1723: 8 total, 8 up, 8 in 2022-04-22T18:21:21.534 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_deep_copy_clone ... ok 2022-04-22T18:21:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:21 smithi121 bash[36902]: cluster 2022-04-22T18:21:20.477146+0000 mon.a (mon.0) 4402 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2022-04-22T18:21:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:21 smithi121 bash[36902]: audit 2022-04-22T18:21:21.040964+0000 mon.a (mon.0) 4403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:21 smithi121 bash[36902]: cluster 2022-04-22T18:21:21.374379+0000 mon.a (mon.0) 4404 : cluster [DBG] osdmap e1725: 8 total, 8 up, 8 in 2022-04-22T18:21:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[45495]: cluster 2022-04-22T18:21:20.477146+0000 mon.a (mon.0) 4402 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2022-04-22T18:21:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[45495]: audit 2022-04-22T18:21:21.040964+0000 mon.a (mon.0) 4403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[45495]: cluster 2022-04-22T18:21:21.374379+0000 mon.a (mon.0) 4404 : cluster [DBG] osdmap e1725: 8 total, 8 up, 8 in 2022-04-22T18:21:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[46934]: cluster 2022-04-22T18:21:20.477146+0000 mon.a (mon.0) 4402 : cluster [DBG] osdmap e1724: 8 total, 8 up, 8 in 2022-04-22T18:21:21.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[46934]: audit 2022-04-22T18:21:21.040964+0000 mon.a (mon.0) 4403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:21.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[46934]: cluster 2022-04-22T18:21:21.374379+0000 mon.a (mon.0) 4404 : cluster [DBG] osdmap e1725: 8 total, 8 up, 8 in 2022-04-22T18:21:22.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:21.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:22 smithi121 bash[36902]: cluster 2022-04-22T18:21:21.394645+0000 mgr.y (mgr.34107) 1933 : cluster [DBG] pgmap v3105: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 167 op/s 2022-04-22T18:21:22.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:22 smithi121 bash[36902]: audit 2022-04-22T18:21:21.530730+0000 mon.a (mon.0) 4405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:22 smithi115 bash[45495]: cluster 2022-04-22T18:21:21.394645+0000 mgr.y (mgr.34107) 1933 : cluster [DBG] pgmap v3105: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 167 op/s 2022-04-22T18:21:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:22 smithi115 bash[45495]: audit 2022-04-22T18:21:21.530730+0000 mon.a (mon.0) 4405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:22 smithi115 bash[46934]: cluster 2022-04-22T18:21:21.394645+0000 mgr.y (mgr.34107) 1933 : cluster [DBG] pgmap v3105: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 167 op/s 2022-04-22T18:21:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:22 smithi115 bash[46934]: audit 2022-04-22T18:21:21.530730+0000 mon.a (mon.0) 4405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:23.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:23 smithi121 bash[36902]: cluster 2022-04-22T18:21:22.484411+0000 mon.a (mon.0) 4406 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2022-04-22T18:21:23.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:23 smithi121 bash[36902]: audit 2022-04-22T18:21:23.417890+0000 mon.a (mon.0) 4407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:23.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:23.689Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:23 smithi115 bash[45495]: cluster 2022-04-22T18:21:22.484411+0000 mon.a (mon.0) 4406 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2022-04-22T18:21:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:23 smithi115 bash[45495]: audit 2022-04-22T18:21:23.417890+0000 mon.a (mon.0) 4407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:23 smithi115 bash[46934]: cluster 2022-04-22T18:21:22.484411+0000 mon.a (mon.0) 4406 : cluster [DBG] osdmap e1726: 8 total, 8 up, 8 in 2022-04-22T18:21:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:23 smithi115 bash[46934]: audit 2022-04-22T18:21:23.417890+0000 mon.a (mon.0) 4407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:24 smithi121 bash[36902]: cluster 2022-04-22T18:21:23.395707+0000 mgr.y (mgr.34107) 1934 : cluster [DBG] pgmap v3107: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 10 KiB/s wr, 187 op/s 2022-04-22T18:21:24.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:24 smithi121 bash[36902]: cluster 2022-04-22T18:21:23.496130+0000 mon.a (mon.0) 4408 : cluster [DBG] osdmap e1727: 8 total, 8 up, 8 in 2022-04-22T18:21:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:24 smithi115 bash[45495]: cluster 2022-04-22T18:21:23.395707+0000 mgr.y (mgr.34107) 1934 : cluster [DBG] pgmap v3107: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 10 KiB/s wr, 187 op/s 2022-04-22T18:21:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:24 smithi115 bash[45495]: cluster 2022-04-22T18:21:23.496130+0000 mon.a (mon.0) 4408 : cluster [DBG] osdmap e1727: 8 total, 8 up, 8 in 2022-04-22T18:21:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:24 smithi115 bash[46934]: cluster 2022-04-22T18:21:23.395707+0000 mgr.y (mgr.34107) 1934 : cluster [DBG] pgmap v3107: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 10 KiB/s wr, 187 op/s 2022-04-22T18:21:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:24 smithi115 bash[46934]: cluster 2022-04-22T18:21:23.496130+0000 mon.a (mon.0) 4408 : cluster [DBG] osdmap e1727: 8 total, 8 up, 8 in 2022-04-22T18:21:25.664 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_diff_iterate ... ok 2022-04-22T18:21:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:25 smithi121 bash[36902]: cluster 2022-04-22T18:21:24.500166+0000 mon.a (mon.0) 4409 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2022-04-22T18:21:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:25 smithi115 bash[46934]: cluster 2022-04-22T18:21:24.500166+0000 mon.a (mon.0) 4409 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2022-04-22T18:21:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:25 smithi115 bash[45495]: cluster 2022-04-22T18:21:24.500166+0000 mon.a (mon.0) 4409 : cluster [DBG] osdmap e1728: 8 total, 8 up, 8 in 2022-04-22T18:21:26.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[46934]: cluster 2022-04-22T18:21:25.396749+0000 mgr.y (mgr.34107) 1935 : cluster [DBG] pgmap v3110: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 193 op/s 2022-04-22T18:21:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[46934]: cluster 2022-04-22T18:21:25.521008+0000 mon.a (mon.0) 4410 : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2022-04-22T18:21:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[46934]: audit 2022-04-22T18:21:26.043992+0000 mon.a (mon.0) 4411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[46934]: cluster 2022-04-22T18:21:26.379203+0000 mon.a (mon.0) 4412 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2022-04-22T18:21:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[45495]: cluster 2022-04-22T18:21:25.396749+0000 mgr.y (mgr.34107) 1935 : cluster [DBG] pgmap v3110: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 193 op/s 2022-04-22T18:21:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[45495]: cluster 2022-04-22T18:21:25.521008+0000 mon.a (mon.0) 4410 : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2022-04-22T18:21:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[45495]: audit 2022-04-22T18:21:26.043992+0000 mon.a (mon.0) 4411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:26 smithi115 bash[45495]: cluster 2022-04-22T18:21:26.379203+0000 mon.a (mon.0) 4412 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2022-04-22T18:21:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:26 smithi121 bash[36902]: cluster 2022-04-22T18:21:25.396749+0000 mgr.y (mgr.34107) 1935 : cluster [DBG] pgmap v3110: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 193 op/s 2022-04-22T18:21:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:26 smithi121 bash[36902]: cluster 2022-04-22T18:21:25.521008+0000 mon.a (mon.0) 4410 : cluster [DBG] osdmap e1729: 8 total, 8 up, 8 in 2022-04-22T18:21:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:26 smithi121 bash[36902]: audit 2022-04-22T18:21:26.043992+0000 mon.a (mon.0) 4411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:26 smithi121 bash[36902]: cluster 2022-04-22T18:21:26.379203+0000 mon.a (mon.0) 4412 : cluster [DBG] osdmap e1730: 8 total, 8 up, 8 in 2022-04-22T18:21:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:27 smithi115 bash[46934]: audit 2022-04-22T18:21:26.533579+0000 mon.a (mon.0) 4413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:27.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:27 smithi115 bash[45495]: audit 2022-04-22T18:21:26.533579+0000 mon.a (mon.0) 4413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:27 smithi121 bash[36902]: audit 2022-04-22T18:21:26.533579+0000 mon.a (mon.0) 4413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:28 smithi115 bash[46934]: cluster 2022-04-22T18:21:27.397753+0000 mgr.y (mgr.34107) 1936 : cluster [DBG] pgmap v3113: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 5.5 KiB/s wr, 40 op/s 2022-04-22T18:21:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:28 smithi115 bash[46934]: audit 2022-04-22T18:21:28.420675+0000 mon.a (mon.0) 4414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:28 smithi115 bash[45495]: cluster 2022-04-22T18:21:27.397753+0000 mgr.y (mgr.34107) 1936 : cluster [DBG] pgmap v3113: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 5.5 KiB/s wr, 40 op/s 2022-04-22T18:21:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:28 smithi115 bash[45495]: audit 2022-04-22T18:21:28.420675+0000 mon.a (mon.0) 4414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:28 smithi121 bash[36902]: cluster 2022-04-22T18:21:27.397753+0000 mgr.y (mgr.34107) 1936 : cluster [DBG] pgmap v3113: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 5.5 KiB/s wr, 40 op/s 2022-04-22T18:21:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:28 smithi121 bash[36902]: audit 2022-04-22T18:21:28.420675+0000 mon.a (mon.0) 4414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:30 smithi115 bash[46934]: cluster 2022-04-22T18:21:29.398962+0000 mgr.y (mgr.34107) 1937 : cluster [DBG] pgmap v3114: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 6.9 KiB/s wr, 76 op/s 2022-04-22T18:21:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:30 smithi115 bash[45495]: cluster 2022-04-22T18:21:29.398962+0000 mgr.y (mgr.34107) 1937 : cluster [DBG] pgmap v3114: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 6.9 KiB/s wr, 76 op/s 2022-04-22T18:21:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:30 smithi121 bash[36902]: cluster 2022-04-22T18:21:29.398962+0000 mgr.y (mgr.34107) 1937 : cluster [DBG] pgmap v3114: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 6.9 KiB/s wr, 76 op/s 2022-04-22T18:21:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:31 smithi115 bash[45495]: audit 2022-04-22T18:21:31.046616+0000 mon.a (mon.0) 4415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:31 smithi115 bash[45495]: cluster 2022-04-22T18:21:31.377864+0000 mon.a (mon.0) 4416 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2022-04-22T18:21:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:31 smithi115 bash[46934]: audit 2022-04-22T18:21:31.046616+0000 mon.a (mon.0) 4415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:31 smithi115 bash[46934]: cluster 2022-04-22T18:21:31.377864+0000 mon.a (mon.0) 4416 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2022-04-22T18:21:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:31 smithi121 bash[36902]: audit 2022-04-22T18:21:31.046616+0000 mon.a (mon.0) 4415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:31 smithi121 bash[36902]: cluster 2022-04-22T18:21:31.377864+0000 mon.a (mon.0) 4416 : cluster [DBG] osdmap e1731: 8 total, 8 up, 8 in 2022-04-22T18:21:32.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:31.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:32 smithi121 bash[36902]: cluster 2022-04-22T18:21:31.399743+0000 mgr.y (mgr.34107) 1938 : cluster [DBG] pgmap v3116: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 4.8 KiB/s wr, 64 op/s 2022-04-22T18:21:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:32 smithi121 bash[36902]: audit 2022-04-22T18:21:31.542015+0000 mon.a (mon.0) 4417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:32 smithi115 bash[45495]: cluster 2022-04-22T18:21:31.399743+0000 mgr.y (mgr.34107) 1938 : cluster [DBG] pgmap v3116: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 4.8 KiB/s wr, 64 op/s 2022-04-22T18:21:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:32 smithi115 bash[45495]: audit 2022-04-22T18:21:31.542015+0000 mon.a (mon.0) 4417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:32 smithi115 bash[46934]: cluster 2022-04-22T18:21:31.399743+0000 mgr.y (mgr.34107) 1938 : cluster [DBG] pgmap v3116: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 4.8 KiB/s wr, 64 op/s 2022-04-22T18:21:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:32 smithi115 bash[46934]: audit 2022-04-22T18:21:31.542015+0000 mon.a (mon.0) 4417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:33.379 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_encryption_luks1 ... ok 2022-04-22T18:21:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:33 smithi121 bash[36902]: audit 2022-04-22T18:21:33.423604+0000 mon.a (mon.0) 4418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:34.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:33 smithi115 bash[46934]: audit 2022-04-22T18:21:33.423604+0000 mon.a (mon.0) 4418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:34.060 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:33.690Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:34.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:33 smithi115 bash[45495]: audit 2022-04-22T18:21:33.423604+0000 mon.a (mon.0) 4418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:34 smithi121 bash[36902]: cluster 2022-04-22T18:21:33.400859+0000 mgr.y (mgr.34107) 1939 : cluster [DBG] pgmap v3117: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.3 KiB/s wr, 53 op/s 2022-04-22T18:21:35.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:34 smithi115 bash[45495]: cluster 2022-04-22T18:21:33.400859+0000 mgr.y (mgr.34107) 1939 : cluster [DBG] pgmap v3117: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.3 KiB/s wr, 53 op/s 2022-04-22T18:21:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:34 smithi115 bash[46934]: cluster 2022-04-22T18:21:33.400859+0000 mgr.y (mgr.34107) 1939 : cluster [DBG] pgmap v3117: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.3 KiB/s wr, 53 op/s 2022-04-22T18:21:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:36 smithi121 bash[36902]: cluster 2022-04-22T18:21:35.401710+0000 mgr.y (mgr.34107) 1940 : cluster [DBG] pgmap v3118: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 108 KiB/s rd, 457 KiB/s wr, 59 op/s 2022-04-22T18:21:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:36 smithi121 bash[36902]: audit 2022-04-22T18:21:36.048216+0000 mon.a (mon.0) 4419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:36 smithi121 bash[36902]: cluster 2022-04-22T18:21:36.383397+0000 mon.a (mon.0) 4420 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2022-04-22T18:21:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:36 smithi121 bash[36902]: audit 2022-04-22T18:21:36.539802+0000 mon.a (mon.0) 4421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[45495]: cluster 2022-04-22T18:21:35.401710+0000 mgr.y (mgr.34107) 1940 : cluster [DBG] pgmap v3118: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 108 KiB/s rd, 457 KiB/s wr, 59 op/s 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[45495]: audit 2022-04-22T18:21:36.048216+0000 mon.a (mon.0) 4419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[45495]: cluster 2022-04-22T18:21:36.383397+0000 mon.a (mon.0) 4420 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[45495]: audit 2022-04-22T18:21:36.539802+0000 mon.a (mon.0) 4421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[46934]: cluster 2022-04-22T18:21:35.401710+0000 mgr.y (mgr.34107) 1940 : cluster [DBG] pgmap v3118: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 108 KiB/s rd, 457 KiB/s wr, 59 op/s 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[46934]: audit 2022-04-22T18:21:36.048216+0000 mon.a (mon.0) 4419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[46934]: cluster 2022-04-22T18:21:36.383397+0000 mon.a (mon.0) 4420 : cluster [DBG] osdmap e1732: 8 total, 8 up, 8 in 2022-04-22T18:21:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:36 smithi115 bash[46934]: audit 2022-04-22T18:21:36.539802+0000 mon.a (mon.0) 4421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:21:38.360 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:21:38.360 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (44m) 2m ago 58m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:21:38.360 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (58m) 3m ago 58m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (46m) 2m ago 59m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (46m) 3m ago 59m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (53m) 3m ago 63m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (54m) 2m ago 65m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (53m) 2m ago 66m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (52m) 3m ago 64m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:21:38.361 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (53m) 2m ago 64m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (45m) 2m ago 58m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (45m) 3m ago 58m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (51m) 2m ago 63m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (50m) 2m ago 62m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (50m) 2m ago 62m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (49m) 2m ago 61m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (49m) 3m ago 61m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (48m) 3m ago 61m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (47m) 3m ago 60m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:21:38.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (47m) 3m ago 60m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:21:38.363 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (45m) 3m ago 58m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:21:38.827 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:21:38.828 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:21:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:38 smithi121 bash[36902]: cluster 2022-04-22T18:21:37.402752+0000 mgr.y (mgr.34107) 1941 : cluster [DBG] pgmap v3120: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 514 KiB/s wr, 35 op/s 2022-04-22T18:21:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:38 smithi121 bash[36902]: audit 2022-04-22T18:21:38.426566+0000 mon.a (mon.0) 4422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:39.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:38 smithi115 bash[45495]: cluster 2022-04-22T18:21:37.402752+0000 mgr.y (mgr.34107) 1941 : cluster [DBG] pgmap v3120: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 514 KiB/s wr, 35 op/s 2022-04-22T18:21:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:38 smithi115 bash[45495]: audit 2022-04-22T18:21:38.426566+0000 mon.a (mon.0) 4422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:38 smithi115 bash[46934]: cluster 2022-04-22T18:21:37.402752+0000 mgr.y (mgr.34107) 1941 : cluster [DBG] pgmap v3120: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 514 KiB/s wr, 35 op/s 2022-04-22T18:21:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:38 smithi115 bash[46934]: audit 2022-04-22T18:21:38.426566+0000 mon.a (mon.0) 4422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:39 smithi121 bash[36902]: audit 2022-04-22T18:21:37.956429+0000 mgr.y (mgr.34107) 1942 : audit [DBG] from='client.47335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:39 smithi121 bash[36902]: audit 2022-04-22T18:21:38.333746+0000 mgr.y (mgr.34107) 1943 : audit [DBG] from='client.47341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:39 smithi121 bash[36902]: audit 2022-04-22T18:21:38.820870+0000 mon.c (mon.1) 121 : audit [DBG] from='client.? 172.21.15.115:0/62040188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:40.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[46934]: audit 2022-04-22T18:21:37.956429+0000 mgr.y (mgr.34107) 1942 : audit [DBG] from='client.47335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[46934]: audit 2022-04-22T18:21:38.333746+0000 mgr.y (mgr.34107) 1943 : audit [DBG] from='client.47341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[46934]: audit 2022-04-22T18:21:38.820870+0000 mon.c (mon.1) 121 : audit [DBG] from='client.? 172.21.15.115:0/62040188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[45495]: audit 2022-04-22T18:21:37.956429+0000 mgr.y (mgr.34107) 1942 : audit [DBG] from='client.47335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[45495]: audit 2022-04-22T18:21:38.333746+0000 mgr.y (mgr.34107) 1943 : audit [DBG] from='client.47341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:21:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:39 smithi115 bash[45495]: audit 2022-04-22T18:21:38.820870+0000 mon.c (mon.1) 121 : audit [DBG] from='client.? 172.21.15.115:0/62040188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:21:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:40 smithi121 bash[36902]: cluster 2022-04-22T18:21:39.404055+0000 mgr.y (mgr.34107) 1944 : cluster [DBG] pgmap v3121: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 513 KiB/s wr, 43 op/s 2022-04-22T18:21:41.620 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:40 smithi115 bash[45495]: cluster 2022-04-22T18:21:39.404055+0000 mgr.y (mgr.34107) 1944 : cluster [DBG] pgmap v3121: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 513 KiB/s wr, 43 op/s 2022-04-22T18:21:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:40 smithi115 bash[46934]: cluster 2022-04-22T18:21:39.404055+0000 mgr.y (mgr.34107) 1944 : cluster [DBG] pgmap v3121: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 513 KiB/s wr, 43 op/s 2022-04-22T18:21:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:41 smithi121 bash[36902]: audit 2022-04-22T18:21:41.050991+0000 mon.a (mon.0) 4423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:41 smithi121 bash[36902]: audit 2022-04-22T18:21:41.542410+0000 mon.a (mon.0) 4424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:41 smithi115 bash[45495]: audit 2022-04-22T18:21:41.050991+0000 mon.a (mon.0) 4423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:42.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:41 smithi115 bash[45495]: audit 2022-04-22T18:21:41.542410+0000 mon.a (mon.0) 4424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:41 smithi115 bash[46934]: audit 2022-04-22T18:21:41.050991+0000 mon.a (mon.0) 4423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:42.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:41 smithi115 bash[46934]: audit 2022-04-22T18:21:41.542410+0000 mon.a (mon.0) 4424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:41.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:42 smithi121 bash[36902]: cluster 2022-04-22T18:21:41.404880+0000 mgr.y (mgr.34107) 1945 : cluster [DBG] pgmap v3122: 137 pgs: 137 active+clean; 8.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 822 KiB/s wr, 38 op/s 2022-04-22T18:21:43.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:42 smithi115 bash[45495]: cluster 2022-04-22T18:21:41.404880+0000 mgr.y (mgr.34107) 1945 : cluster [DBG] pgmap v3122: 137 pgs: 137 active+clean; 8.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 822 KiB/s wr, 38 op/s 2022-04-22T18:21:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:42 smithi115 bash[46934]: cluster 2022-04-22T18:21:41.404880+0000 mgr.y (mgr.34107) 1945 : cluster [DBG] pgmap v3122: 137 pgs: 137 active+clean; 8.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 822 KiB/s wr, 38 op/s 2022-04-22T18:21:43.214 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_encryption_luks2 ... ok 2022-04-22T18:21:43.358 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_flags ... ok 2022-04-22T18:21:43.509 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_id ... ok 2022-04-22T18:21:43.681 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_image_auto_close ... ok 2022-04-22T18:21:43.884 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_image_config_set_and_get_and_remove ... ok 2022-04-22T18:21:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:43 smithi121 bash[36902]: audit 2022-04-22T18:21:43.429339+0000 mon.a (mon.0) 4425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:43 smithi115 bash[45495]: audit 2022-04-22T18:21:43.429339+0000 mon.a (mon.0) 4425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:43.690Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:43 smithi115 bash[46934]: audit 2022-04-22T18:21:43.429339+0000 mon.a (mon.0) 4425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:44.060 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_invalidate_cache ... ok 2022-04-22T18:21:44.205 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_large_read ... ok 2022-04-22T18:21:44.451 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_large_write ... ok 2022-04-22T18:21:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:44 smithi121 bash[36902]: cluster 2022-04-22T18:21:43.405837+0000 mgr.y (mgr.34107) 1946 : cluster [DBG] pgmap v3123: 137 pgs: 137 active+clean; 21 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 2.0 MiB/s wr, 41 op/s 2022-04-22T18:21:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:44 smithi115 bash[45495]: cluster 2022-04-22T18:21:43.405837+0000 mgr.y (mgr.34107) 1946 : cluster [DBG] pgmap v3123: 137 pgs: 137 active+clean; 21 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 2.0 MiB/s wr, 41 op/s 2022-04-22T18:21:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:44 smithi115 bash[46934]: cluster 2022-04-22T18:21:43.405837+0000 mgr.y (mgr.34107) 1946 : cluster [DBG] pgmap v3123: 137 pgs: 137 active+clean; 21 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 2.0 MiB/s wr, 41 op/s 2022-04-22T18:21:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:45 smithi121 bash[36902]: cluster 2022-04-22T18:21:44.596227+0000 mon.a (mon.0) 4426 : cluster [DBG] osdmap e1733: 8 total, 8 up, 8 in 2022-04-22T18:21:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:45 smithi115 bash[45495]: cluster 2022-04-22T18:21:44.596227+0000 mon.a (mon.0) 4426 : cluster [DBG] osdmap e1733: 8 total, 8 up, 8 in 2022-04-22T18:21:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:45 smithi115 bash[46934]: cluster 2022-04-22T18:21:44.596227+0000 mon.a (mon.0) 4426 : cluster [DBG] osdmap e1733: 8 total, 8 up, 8 in 2022-04-22T18:21:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:46 smithi121 bash[36902]: cluster 2022-04-22T18:21:45.406776+0000 mgr.y (mgr.34107) 1947 : cluster [DBG] pgmap v3125: 137 pgs: 137 active+clean; 17 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 51 op/s 2022-04-22T18:21:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:46 smithi121 bash[36902]: cluster 2022-04-22T18:21:45.606785+0000 mon.a (mon.0) 4427 : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2022-04-22T18:21:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:46 smithi121 bash[36902]: audit 2022-04-22T18:21:46.053407+0000 mon.a (mon.0) 4428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:46 smithi121 bash[36902]: audit 2022-04-22T18:21:46.545641+0000 mon.a (mon.0) 4429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[45495]: cluster 2022-04-22T18:21:45.406776+0000 mgr.y (mgr.34107) 1947 : cluster [DBG] pgmap v3125: 137 pgs: 137 active+clean; 17 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 51 op/s 2022-04-22T18:21:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[45495]: cluster 2022-04-22T18:21:45.606785+0000 mon.a (mon.0) 4427 : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2022-04-22T18:21:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[45495]: audit 2022-04-22T18:21:46.053407+0000 mon.a (mon.0) 4428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[45495]: audit 2022-04-22T18:21:46.545641+0000 mon.a (mon.0) 4429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:47.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[46934]: cluster 2022-04-22T18:21:45.406776+0000 mgr.y (mgr.34107) 1947 : cluster [DBG] pgmap v3125: 137 pgs: 137 active+clean; 17 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 51 op/s 2022-04-22T18:21:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[46934]: cluster 2022-04-22T18:21:45.606785+0000 mon.a (mon.0) 4427 : cluster [DBG] osdmap e1734: 8 total, 8 up, 8 in 2022-04-22T18:21:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[46934]: audit 2022-04-22T18:21:46.053407+0000 mon.a (mon.0) 4428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:46 smithi115 bash[46934]: audit 2022-04-22T18:21:46.545641+0000 mon.a (mon.0) 4429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:47 smithi121 bash[36902]: cluster 2022-04-22T18:21:46.612926+0000 mon.a (mon.0) 4430 : cluster [DBG] osdmap e1735: 8 total, 8 up, 8 in 2022-04-22T18:21:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:47 smithi115 bash[45495]: cluster 2022-04-22T18:21:46.612926+0000 mon.a (mon.0) 4430 : cluster [DBG] osdmap e1735: 8 total, 8 up, 8 in 2022-04-22T18:21:48.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:47 smithi115 bash[46934]: cluster 2022-04-22T18:21:46.612926+0000 mon.a (mon.0) 4430 : cluster [DBG] osdmap e1735: 8 total, 8 up, 8 in 2022-04-22T18:21:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:48 smithi121 bash[36902]: cluster 2022-04-22T18:21:47.407958+0000 mgr.y (mgr.34107) 1948 : cluster [DBG] pgmap v3128: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 3.3 MiB/s wr, 150 op/s 2022-04-22T18:21:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:48 smithi121 bash[36902]: cluster 2022-04-22T18:21:47.624412+0000 mon.a (mon.0) 4431 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2022-04-22T18:21:49.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:48 smithi121 bash[36902]: audit 2022-04-22T18:21:48.432200+0000 mon.a (mon.0) 4432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[45495]: cluster 2022-04-22T18:21:47.407958+0000 mgr.y (mgr.34107) 1948 : cluster [DBG] pgmap v3128: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 3.3 MiB/s wr, 150 op/s 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[45495]: cluster 2022-04-22T18:21:47.624412+0000 mon.a (mon.0) 4431 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[45495]: audit 2022-04-22T18:21:48.432200+0000 mon.a (mon.0) 4432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[46934]: cluster 2022-04-22T18:21:47.407958+0000 mgr.y (mgr.34107) 1948 : cluster [DBG] pgmap v3128: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 3.3 MiB/s wr, 150 op/s 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[46934]: cluster 2022-04-22T18:21:47.624412+0000 mon.a (mon.0) 4431 : cluster [DBG] osdmap e1736: 8 total, 8 up, 8 in 2022-04-22T18:21:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:48 smithi115 bash[46934]: audit 2022-04-22T18:21:48.432200+0000 mon.a (mon.0) 4432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:49 smithi121 bash[36902]: cluster 2022-04-22T18:21:48.637597+0000 mon.a (mon.0) 4433 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2022-04-22T18:21:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:49 smithi115 bash[46934]: cluster 2022-04-22T18:21:48.637597+0000 mon.a (mon.0) 4433 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2022-04-22T18:21:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:49 smithi115 bash[45495]: cluster 2022-04-22T18:21:48.637597+0000 mon.a (mon.0) 4433 : cluster [DBG] osdmap e1737: 8 total, 8 up, 8 in 2022-04-22T18:21:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:50 smithi121 bash[36902]: cluster 2022-04-22T18:21:49.408975+0000 mgr.y (mgr.34107) 1949 : cluster [DBG] pgmap v3131: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 281 KiB/s rd, 2.0 MiB/s wr, 382 op/s 2022-04-22T18:21:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:50 smithi121 bash[36902]: cluster 2022-04-22T18:21:49.652002+0000 mon.a (mon.0) 4434 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2022-04-22T18:21:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:50 smithi115 bash[45495]: cluster 2022-04-22T18:21:49.408975+0000 mgr.y (mgr.34107) 1949 : cluster [DBG] pgmap v3131: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 281 KiB/s rd, 2.0 MiB/s wr, 382 op/s 2022-04-22T18:21:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:50 smithi115 bash[45495]: cluster 2022-04-22T18:21:49.652002+0000 mon.a (mon.0) 4434 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2022-04-22T18:21:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:50 smithi115 bash[46934]: cluster 2022-04-22T18:21:49.408975+0000 mgr.y (mgr.34107) 1949 : cluster [DBG] pgmap v3131: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 281 KiB/s rd, 2.0 MiB/s wr, 382 op/s 2022-04-22T18:21:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:50 smithi115 bash[46934]: cluster 2022-04-22T18:21:49.652002+0000 mon.a (mon.0) 4434 : cluster [DBG] osdmap e1738: 8 total, 8 up, 8 in 2022-04-22T18:21:51.811 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_limit_snaps ... ok 2022-04-22T18:21:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:51 smithi121 bash[36902]: cluster 2022-04-22T18:21:50.657282+0000 mon.a (mon.0) 4435 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2022-04-22T18:21:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:51 smithi121 bash[36902]: audit 2022-04-22T18:21:51.056370+0000 mon.a (mon.0) 4436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:51 smithi121 bash[36902]: audit 2022-04-22T18:21:51.549403+0000 mon.a (mon.0) 4437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:52.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:51.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[45495]: cluster 2022-04-22T18:21:50.657282+0000 mon.a (mon.0) 4435 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[45495]: audit 2022-04-22T18:21:51.056370+0000 mon.a (mon.0) 4436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[45495]: audit 2022-04-22T18:21:51.549403+0000 mon.a (mon.0) 4437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[46934]: cluster 2022-04-22T18:21:50.657282+0000 mon.a (mon.0) 4435 : cluster [DBG] osdmap e1739: 8 total, 8 up, 8 in 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[46934]: audit 2022-04-22T18:21:51.056370+0000 mon.a (mon.0) 4436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:52.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:51 smithi115 bash[46934]: audit 2022-04-22T18:21:51.549403+0000 mon.a (mon.0) 4437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:52.131 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_list_lockers ... ok 2022-04-22T18:21:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:52 smithi121 bash[36902]: cluster 2022-04-22T18:21:51.409808+0000 mgr.y (mgr.34107) 1950 : cluster [DBG] pgmap v3134: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 16 KiB/s wr, 233 op/s 2022-04-22T18:21:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:52 smithi121 bash[36902]: cluster 2022-04-22T18:21:51.670683+0000 mon.a (mon.0) 4438 : cluster [DBG] osdmap e1740: 8 total, 8 up, 8 in 2022-04-22T18:21:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:52 smithi115 bash[45495]: cluster 2022-04-22T18:21:51.409808+0000 mgr.y (mgr.34107) 1950 : cluster [DBG] pgmap v3134: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 16 KiB/s wr, 233 op/s 2022-04-22T18:21:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:52 smithi115 bash[45495]: cluster 2022-04-22T18:21:51.670683+0000 mon.a (mon.0) 4438 : cluster [DBG] osdmap e1740: 8 total, 8 up, 8 in 2022-04-22T18:21:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:52 smithi115 bash[46934]: cluster 2022-04-22T18:21:51.409808+0000 mgr.y (mgr.34107) 1950 : cluster [DBG] pgmap v3134: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 16 KiB/s wr, 233 op/s 2022-04-22T18:21:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:52 smithi115 bash[46934]: cluster 2022-04-22T18:21:51.670683+0000 mon.a (mon.0) 4438 : cluster [DBG] osdmap e1740: 8 total, 8 up, 8 in 2022-04-22T18:21:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:53 smithi121 bash[36902]: cluster 2022-04-22T18:21:52.683060+0000 mon.a (mon.0) 4439 : cluster [DBG] osdmap e1741: 8 total, 8 up, 8 in 2022-04-22T18:21:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:53 smithi121 bash[36902]: cluster 2022-04-22T18:21:53.410828+0000 mgr.y (mgr.34107) 1951 : cluster [DBG] pgmap v3137: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2022-04-22T18:21:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:53 smithi121 bash[36902]: audit 2022-04-22T18:21:53.435607+0000 mon.a (mon.0) 4440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:54.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:21:53.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[45495]: cluster 2022-04-22T18:21:52.683060+0000 mon.a (mon.0) 4439 : cluster [DBG] osdmap e1741: 8 total, 8 up, 8 in 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[45495]: cluster 2022-04-22T18:21:53.410828+0000 mgr.y (mgr.34107) 1951 : cluster [DBG] pgmap v3137: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[45495]: audit 2022-04-22T18:21:53.435607+0000 mon.a (mon.0) 4440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[46934]: cluster 2022-04-22T18:21:52.683060+0000 mon.a (mon.0) 4439 : cluster [DBG] osdmap e1741: 8 total, 8 up, 8 in 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[46934]: cluster 2022-04-22T18:21:53.410828+0000 mgr.y (mgr.34107) 1951 : cluster [DBG] pgmap v3137: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 3.0 KiB/s wr, 146 op/s 2022-04-22T18:21:54.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:53 smithi115 bash[46934]: audit 2022-04-22T18:21:53.435607+0000 mon.a (mon.0) 4440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:54 smithi121 bash[36902]: cluster 2022-04-22T18:21:53.696307+0000 mon.a (mon.0) 4441 : cluster [DBG] osdmap e1742: 8 total, 8 up, 8 in 2022-04-22T18:21:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:54 smithi115 bash[45495]: cluster 2022-04-22T18:21:53.696307+0000 mon.a (mon.0) 4441 : cluster [DBG] osdmap e1742: 8 total, 8 up, 8 in 2022-04-22T18:21:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:54 smithi115 bash[46934]: cluster 2022-04-22T18:21:53.696307+0000 mon.a (mon.0) 4441 : cluster [DBG] osdmap e1742: 8 total, 8 up, 8 in 2022-04-22T18:21:55.867 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_list_snaps ... ok 2022-04-22T18:21:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:55 smithi121 bash[36902]: cluster 2022-04-22T18:21:54.704305+0000 mon.a (mon.0) 4442 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2022-04-22T18:21:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:55 smithi121 bash[36902]: cluster 2022-04-22T18:21:55.411833+0000 mgr.y (mgr.34107) 1952 : cluster [DBG] pgmap v3140: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 158 KiB/s rd, 5.7 KiB/s wr, 202 op/s 2022-04-22T18:21:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:55 smithi115 bash[45495]: cluster 2022-04-22T18:21:54.704305+0000 mon.a (mon.0) 4442 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2022-04-22T18:21:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:55 smithi115 bash[45495]: cluster 2022-04-22T18:21:55.411833+0000 mgr.y (mgr.34107) 1952 : cluster [DBG] pgmap v3140: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 158 KiB/s rd, 5.7 KiB/s wr, 202 op/s 2022-04-22T18:21:56.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:55 smithi115 bash[46934]: cluster 2022-04-22T18:21:54.704305+0000 mon.a (mon.0) 4442 : cluster [DBG] osdmap e1743: 8 total, 8 up, 8 in 2022-04-22T18:21:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:55 smithi115 bash[46934]: cluster 2022-04-22T18:21:55.411833+0000 mgr.y (mgr.34107) 1952 : cluster [DBG] pgmap v3140: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 158 KiB/s rd, 5.7 KiB/s wr, 202 op/s 2022-04-22T18:21:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:56 smithi121 bash[36902]: cluster 2022-04-22T18:21:55.715160+0000 mon.a (mon.0) 4443 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2022-04-22T18:21:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:56 smithi121 bash[36902]: audit 2022-04-22T18:21:56.059382+0000 mon.a (mon.0) 4444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:56 smithi121 bash[36902]: cluster 2022-04-22T18:21:56.389190+0000 mon.a (mon.0) 4445 : cluster [DBG] osdmap e1745: 8 total, 8 up, 8 in 2022-04-22T18:21:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:56 smithi121 bash[36902]: audit 2022-04-22T18:21:56.553163+0000 mon.a (mon.0) 4446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[45495]: cluster 2022-04-22T18:21:55.715160+0000 mon.a (mon.0) 4443 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[45495]: audit 2022-04-22T18:21:56.059382+0000 mon.a (mon.0) 4444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[45495]: cluster 2022-04-22T18:21:56.389190+0000 mon.a (mon.0) 4445 : cluster [DBG] osdmap e1745: 8 total, 8 up, 8 in 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[45495]: audit 2022-04-22T18:21:56.553163+0000 mon.a (mon.0) 4446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[46934]: cluster 2022-04-22T18:21:55.715160+0000 mon.a (mon.0) 4443 : cluster [DBG] osdmap e1744: 8 total, 8 up, 8 in 2022-04-22T18:21:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[46934]: audit 2022-04-22T18:21:56.059382+0000 mon.a (mon.0) 4444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:21:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[46934]: cluster 2022-04-22T18:21:56.389190+0000 mon.a (mon.0) 4445 : cluster [DBG] osdmap e1745: 8 total, 8 up, 8 in 2022-04-22T18:21:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:56 smithi115 bash[46934]: audit 2022-04-22T18:21:56.553163+0000 mon.a (mon.0) 4446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:21:57.558 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_list_snaps_iterator_auto_close ... ok 2022-04-22T18:21:57.751 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_lock_unlock ... ok 2022-04-22T18:21:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:58 smithi121 bash[36902]: cluster 2022-04-22T18:21:57.402458+0000 mon.a (mon.0) 4447 : cluster [DBG] osdmap e1746: 8 total, 8 up, 8 in 2022-04-22T18:21:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:58 smithi121 bash[36902]: cluster 2022-04-22T18:21:57.412350+0000 mgr.y (mgr.34107) 1953 : cluster [DBG] pgmap v3144: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 6.7 KiB/s wr, 144 op/s 2022-04-22T18:21:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:58 smithi115 bash[45495]: cluster 2022-04-22T18:21:57.402458+0000 mon.a (mon.0) 4447 : cluster [DBG] osdmap e1746: 8 total, 8 up, 8 in 2022-04-22T18:21:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:58 smithi115 bash[45495]: cluster 2022-04-22T18:21:57.412350+0000 mgr.y (mgr.34107) 1953 : cluster [DBG] pgmap v3144: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 6.7 KiB/s wr, 144 op/s 2022-04-22T18:21:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:58 smithi115 bash[46934]: cluster 2022-04-22T18:21:57.402458+0000 mon.a (mon.0) 4447 : cluster [DBG] osdmap e1746: 8 total, 8 up, 8 in 2022-04-22T18:21:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:58 smithi115 bash[46934]: cluster 2022-04-22T18:21:57.412350+0000 mgr.y (mgr.34107) 1953 : cluster [DBG] pgmap v3144: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 6.7 KiB/s wr, 144 op/s 2022-04-22T18:21:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:59 smithi121 bash[36902]: cluster 2022-04-22T18:21:58.409353+0000 mon.a (mon.0) 4448 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2022-04-22T18:21:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:21:59 smithi121 bash[36902]: audit 2022-04-22T18:21:58.438357+0000 mon.a (mon.0) 4449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:59 smithi115 bash[45495]: cluster 2022-04-22T18:21:58.409353+0000 mon.a (mon.0) 4448 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2022-04-22T18:21:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:21:59 smithi115 bash[45495]: audit 2022-04-22T18:21:58.438357+0000 mon.a (mon.0) 4449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:21:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:59 smithi115 bash[46934]: cluster 2022-04-22T18:21:58.409353+0000 mon.a (mon.0) 4448 : cluster [DBG] osdmap e1747: 8 total, 8 up, 8 in 2022-04-22T18:21:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:21:59 smithi115 bash[46934]: audit 2022-04-22T18:21:58.438357+0000 mon.a (mon.0) 4449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:00 smithi121 bash[36902]: cluster 2022-04-22T18:21:59.412973+0000 mgr.y (mgr.34107) 1954 : cluster [DBG] pgmap v3147: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.8 KiB/s wr, 147 op/s 2022-04-22T18:22:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:00 smithi121 bash[36902]: cluster 2022-04-22T18:21:59.413327+0000 mon.a (mon.0) 4450 : cluster [DBG] osdmap e1748: 8 total, 8 up, 8 in 2022-04-22T18:22:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:00 smithi115 bash[45495]: cluster 2022-04-22T18:21:59.412973+0000 mgr.y (mgr.34107) 1954 : cluster [DBG] pgmap v3147: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.8 KiB/s wr, 147 op/s 2022-04-22T18:22:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:00 smithi115 bash[45495]: cluster 2022-04-22T18:21:59.413327+0000 mon.a (mon.0) 4450 : cluster [DBG] osdmap e1748: 8 total, 8 up, 8 in 2022-04-22T18:22:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:00 smithi115 bash[46934]: cluster 2022-04-22T18:21:59.412973+0000 mgr.y (mgr.34107) 1954 : cluster [DBG] pgmap v3147: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.8 KiB/s wr, 147 op/s 2022-04-22T18:22:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:00 smithi115 bash[46934]: cluster 2022-04-22T18:21:59.413327+0000 mon.a (mon.0) 4450 : cluster [DBG] osdmap e1748: 8 total, 8 up, 8 in 2022-04-22T18:22:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:01 smithi121 bash[36902]: cluster 2022-04-22T18:22:00.430784+0000 mon.a (mon.0) 4451 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2022-04-22T18:22:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:01 smithi121 bash[36902]: audit 2022-04-22T18:22:01.062696+0000 mon.a (mon.0) 4452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:01 smithi121 bash[36902]: cluster 2022-04-22T18:22:01.389947+0000 mon.a (mon.0) 4453 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[45495]: cluster 2022-04-22T18:22:00.430784+0000 mon.a (mon.0) 4451 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[45495]: audit 2022-04-22T18:22:01.062696+0000 mon.a (mon.0) 4452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[45495]: cluster 2022-04-22T18:22:01.389947+0000 mon.a (mon.0) 4453 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[46934]: cluster 2022-04-22T18:22:00.430784+0000 mon.a (mon.0) 4451 : cluster [DBG] osdmap e1749: 8 total, 8 up, 8 in 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[46934]: audit 2022-04-22T18:22:01.062696+0000 mon.a (mon.0) 4452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[46934]: cluster 2022-04-22T18:22:01.389947+0000 mon.a (mon.0) 4453 : cluster [DBG] osdmap e1750: 8 total, 8 up, 8 in 2022-04-22T18:22:02.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:01.811Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:02 smithi121 bash[36902]: cluster 2022-04-22T18:22:01.413881+0000 mgr.y (mgr.34107) 1955 : cluster [DBG] pgmap v3150: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 104 KiB/s rd, 8.2 KiB/s wr, 148 op/s 2022-04-22T18:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:02 smithi121 bash[36902]: audit 2022-04-22T18:22:01.556762+0000 mon.a (mon.0) 4454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:02.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:02 smithi121 bash[36902]: cluster 2022-04-22T18:22:02.406422+0000 mon.a (mon.0) 4455 : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[46934]: cluster 2022-04-22T18:22:01.413881+0000 mgr.y (mgr.34107) 1955 : cluster [DBG] pgmap v3150: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 104 KiB/s rd, 8.2 KiB/s wr, 148 op/s 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[46934]: audit 2022-04-22T18:22:01.556762+0000 mon.a (mon.0) 4454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[46934]: cluster 2022-04-22T18:22:02.406422+0000 mon.a (mon.0) 4455 : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[45495]: cluster 2022-04-22T18:22:01.413881+0000 mgr.y (mgr.34107) 1955 : cluster [DBG] pgmap v3150: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 104 KiB/s rd, 8.2 KiB/s wr, 148 op/s 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[45495]: audit 2022-04-22T18:22:01.556762+0000 mon.a (mon.0) 4454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:02 smithi115 bash[45495]: cluster 2022-04-22T18:22:02.406422+0000 mon.a (mon.0) 4455 : cluster [DBG] osdmap e1751: 8 total, 8 up, 8 in 2022-04-22T18:22:03.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[46934]: audit 2022-04-22T18:22:02.478679+0000 mon.a (mon.0) 4456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:22:03.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[46934]: audit 2022-04-22T18:22:02.481387+0000 mon.a (mon.0) 4457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:22:03.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[46934]: audit 2022-04-22T18:22:02.482766+0000 mon.a (mon.0) 4458 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:22:03.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[46934]: audit 2022-04-22T18:22:02.854817+0000 mon.a (mon.0) 4459 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:22:03.700 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[45495]: audit 2022-04-22T18:22:02.478679+0000 mon.a (mon.0) 4456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:22:03.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[45495]: audit 2022-04-22T18:22:02.481387+0000 mon.a (mon.0) 4457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:22:03.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[45495]: audit 2022-04-22T18:22:02.482766+0000 mon.a (mon.0) 4458 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:22:03.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[45495]: audit 2022-04-22T18:22:02.854817+0000 mon.a (mon.0) 4459 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:22:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:03 smithi121 bash[36902]: audit 2022-04-22T18:22:02.478679+0000 mon.a (mon.0) 4456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:22:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:03 smithi121 bash[36902]: audit 2022-04-22T18:22:02.481387+0000 mon.a (mon.0) 4457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:22:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:03 smithi121 bash[36902]: audit 2022-04-22T18:22:02.482766+0000 mon.a (mon.0) 4458 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:22:03.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:03 smithi121 bash[36902]: audit 2022-04-22T18:22:02.854817+0000 mon.a (mon.0) 4459 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:22:04.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:03.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:04 smithi121 bash[36902]: cluster 2022-04-22T18:22:03.415117+0000 mgr.y (mgr.34107) 1956 : cluster [DBG] pgmap v3152: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 93 op/s 2022-04-22T18:22:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:04 smithi121 bash[36902]: cluster 2022-04-22T18:22:03.439196+0000 mon.a (mon.0) 4460 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2022-04-22T18:22:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:04 smithi121 bash[36902]: audit 2022-04-22T18:22:03.441607+0000 mon.a (mon.0) 4461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[46934]: cluster 2022-04-22T18:22:03.415117+0000 mgr.y (mgr.34107) 1956 : cluster [DBG] pgmap v3152: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 93 op/s 2022-04-22T18:22:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[46934]: cluster 2022-04-22T18:22:03.439196+0000 mon.a (mon.0) 4460 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2022-04-22T18:22:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[46934]: audit 2022-04-22T18:22:03.441607+0000 mon.a (mon.0) 4461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[45495]: cluster 2022-04-22T18:22:03.415117+0000 mgr.y (mgr.34107) 1956 : cluster [DBG] pgmap v3152: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 93 op/s 2022-04-22T18:22:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[45495]: cluster 2022-04-22T18:22:03.439196+0000 mon.a (mon.0) 4460 : cluster [DBG] osdmap e1752: 8 total, 8 up, 8 in 2022-04-22T18:22:04.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:04 smithi115 bash[45495]: audit 2022-04-22T18:22:03.441607+0000 mon.a (mon.0) 4461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:05 smithi121 bash[36902]: cluster 2022-04-22T18:22:04.456430+0000 mon.a (mon.0) 4462 : cluster [DBG] osdmap e1753: 8 total, 8 up, 8 in 2022-04-22T18:22:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:05 smithi115 bash[45495]: cluster 2022-04-22T18:22:04.456430+0000 mon.a (mon.0) 4462 : cluster [DBG] osdmap e1753: 8 total, 8 up, 8 in 2022-04-22T18:22:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:05 smithi115 bash[46934]: cluster 2022-04-22T18:22:04.456430+0000 mon.a (mon.0) 4462 : cluster [DBG] osdmap e1753: 8 total, 8 up, 8 in 2022-04-22T18:22:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:06 smithi121 bash[36902]: cluster 2022-04-22T18:22:05.416276+0000 mgr.y (mgr.34107) 1957 : cluster [DBG] pgmap v3155: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 3.7 KiB/s wr, 84 op/s 2022-04-22T18:22:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:06 smithi121 bash[36902]: cluster 2022-04-22T18:22:05.464927+0000 mon.a (mon.0) 4463 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2022-04-22T18:22:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:06 smithi121 bash[36902]: audit 2022-04-22T18:22:06.066051+0000 mon.a (mon.0) 4464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:06 smithi121 bash[36902]: cluster 2022-04-22T18:22:06.393555+0000 mon.a (mon.0) 4465 : cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2022-04-22T18:22:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[45495]: cluster 2022-04-22T18:22:05.416276+0000 mgr.y (mgr.34107) 1957 : cluster [DBG] pgmap v3155: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 3.7 KiB/s wr, 84 op/s 2022-04-22T18:22:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[45495]: cluster 2022-04-22T18:22:05.464927+0000 mon.a (mon.0) 4463 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2022-04-22T18:22:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[45495]: audit 2022-04-22T18:22:06.066051+0000 mon.a (mon.0) 4464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[45495]: cluster 2022-04-22T18:22:06.393555+0000 mon.a (mon.0) 4465 : cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2022-04-22T18:22:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[46934]: cluster 2022-04-22T18:22:05.416276+0000 mgr.y (mgr.34107) 1957 : cluster [DBG] pgmap v3155: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 3.7 KiB/s wr, 84 op/s 2022-04-22T18:22:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[46934]: cluster 2022-04-22T18:22:05.464927+0000 mon.a (mon.0) 4463 : cluster [DBG] osdmap e1754: 8 total, 8 up, 8 in 2022-04-22T18:22:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[46934]: audit 2022-04-22T18:22:06.066051+0000 mon.a (mon.0) 4464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:06 smithi115 bash[46934]: cluster 2022-04-22T18:22:06.393555+0000 mon.a (mon.0) 4465 : cluster [DBG] osdmap e1755: 8 total, 8 up, 8 in 2022-04-22T18:22:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:07 smithi121 bash[36902]: audit 2022-04-22T18:22:06.560682+0000 mon.a (mon.0) 4466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:07 smithi121 bash[36902]: cluster 2022-04-22T18:22:07.401788+0000 mon.a (mon.0) 4467 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2022-04-22T18:22:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:07 smithi115 bash[45495]: audit 2022-04-22T18:22:06.560682+0000 mon.a (mon.0) 4466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:07 smithi115 bash[45495]: cluster 2022-04-22T18:22:07.401788+0000 mon.a (mon.0) 4467 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2022-04-22T18:22:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:07 smithi115 bash[46934]: audit 2022-04-22T18:22:06.560682+0000 mon.a (mon.0) 4466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:07.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:07 smithi115 bash[46934]: cluster 2022-04-22T18:22:07.401788+0000 mon.a (mon.0) 4467 : cluster [DBG] osdmap e1756: 8 total, 8 up, 8 in 2022-04-22T18:22:08.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:08 smithi121 bash[36902]: cluster 2022-04-22T18:22:07.416994+0000 mgr.y (mgr.34107) 1958 : cluster [DBG] pgmap v3159: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 80 op/s 2022-04-22T18:22:08.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:08 smithi121 bash[36902]: cluster 2022-04-22T18:22:08.411499+0000 mon.a (mon.0) 4468 : cluster [DBG] osdmap e1757: 8 total, 8 up, 8 in 2022-04-22T18:22:08.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:08 smithi121 bash[36902]: audit 2022-04-22T18:22:08.446338+0000 mon.a (mon.0) 4469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[45495]: cluster 2022-04-22T18:22:07.416994+0000 mgr.y (mgr.34107) 1958 : cluster [DBG] pgmap v3159: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 80 op/s 2022-04-22T18:22:08.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[45495]: cluster 2022-04-22T18:22:08.411499+0000 mon.a (mon.0) 4468 : cluster [DBG] osdmap e1757: 8 total, 8 up, 8 in 2022-04-22T18:22:08.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[45495]: audit 2022-04-22T18:22:08.446338+0000 mon.a (mon.0) 4469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:08.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[46934]: cluster 2022-04-22T18:22:07.416994+0000 mgr.y (mgr.34107) 1958 : cluster [DBG] pgmap v3159: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 80 op/s 2022-04-22T18:22:08.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[46934]: cluster 2022-04-22T18:22:08.411499+0000 mon.a (mon.0) 4468 : cluster [DBG] osdmap e1757: 8 total, 8 up, 8 in 2022-04-22T18:22:08.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:08 smithi115 bash[46934]: audit 2022-04-22T18:22:08.446338+0000 mon.a (mon.0) 4469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:09.203 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:22:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:22:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (45m) 3m ago 59m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:22:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (58m) 4m ago 58m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:22:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (47m) 3m ago 60m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:22:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (46m) 4m ago 59m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:22:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (54m) 4m ago 64m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:22:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (55m) 3m ago 66m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:22:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (54m) 3m ago 66m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:22:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (53m) 4m ago 64m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:22:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (53m) 3m ago 64m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:22:09.558 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (46m) 3m ago 59m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:22:09.558 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (46m) 4m ago 59m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:22:09.558 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (51m) 3m ago 63m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:22:09.558 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (51m) 3m ago 63m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:22:09.558 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (50m) 3m ago 62m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:22:09.559 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (50m) 3m ago 62m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:22:09.559 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (49m) 4m ago 61m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:22:09.559 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (48m) 4m ago 61m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:22:09.559 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (48m) 4m ago 61m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:22:09.559 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (47m) 4m ago 60m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:22:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (45m) 4m ago 59m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:22:09.949 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:22:09.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:22:09.951 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:22:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:10 smithi121 bash[36902]: audit 2022-04-22T18:22:09.181763+0000 mgr.y (mgr.34107) 1959 : audit [DBG] from='client.37419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:10 smithi121 bash[36902]: cluster 2022-04-22T18:22:09.417581+0000 mgr.y (mgr.34107) 1960 : cluster [DBG] pgmap v3162: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 776 B/s wr, 2 op/s 2022-04-22T18:22:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:10 smithi121 bash[36902]: cluster 2022-04-22T18:22:09.417704+0000 mon.a (mon.0) 4470 : cluster [DBG] osdmap e1758: 8 total, 8 up, 8 in 2022-04-22T18:22:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:10 smithi121 bash[36902]: audit 2022-04-22T18:22:09.527722+0000 mgr.y (mgr.34107) 1961 : audit [DBG] from='client.47359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:10 smithi121 bash[36902]: audit 2022-04-22T18:22:09.942665+0000 mon.c (mon.1) 122 : audit [DBG] from='client.? 172.21.15.115:0/1680226667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[45495]: audit 2022-04-22T18:22:09.181763+0000 mgr.y (mgr.34107) 1959 : audit [DBG] from='client.37419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[45495]: cluster 2022-04-22T18:22:09.417581+0000 mgr.y (mgr.34107) 1960 : cluster [DBG] pgmap v3162: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 776 B/s wr, 2 op/s 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[45495]: cluster 2022-04-22T18:22:09.417704+0000 mon.a (mon.0) 4470 : cluster [DBG] osdmap e1758: 8 total, 8 up, 8 in 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[45495]: audit 2022-04-22T18:22:09.527722+0000 mgr.y (mgr.34107) 1961 : audit [DBG] from='client.47359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[45495]: audit 2022-04-22T18:22:09.942665+0000 mon.c (mon.1) 122 : audit [DBG] from='client.? 172.21.15.115:0/1680226667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[46934]: audit 2022-04-22T18:22:09.181763+0000 mgr.y (mgr.34107) 1959 : audit [DBG] from='client.37419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[46934]: cluster 2022-04-22T18:22:09.417581+0000 mgr.y (mgr.34107) 1960 : cluster [DBG] pgmap v3162: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 776 B/s wr, 2 op/s 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[46934]: cluster 2022-04-22T18:22:09.417704+0000 mon.a (mon.0) 4470 : cluster [DBG] osdmap e1758: 8 total, 8 up, 8 in 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[46934]: audit 2022-04-22T18:22:09.527722+0000 mgr.y (mgr.34107) 1961 : audit [DBG] from='client.47359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:10 smithi115 bash[46934]: audit 2022-04-22T18:22:09.942665+0000 mon.c (mon.1) 122 : audit [DBG] from='client.? 172.21.15.115:0/1680226667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:11 smithi121 bash[36902]: cluster 2022-04-22T18:22:10.424846+0000 mon.a (mon.0) 4471 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2022-04-22T18:22:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:11 smithi121 bash[36902]: audit 2022-04-22T18:22:11.069202+0000 mon.a (mon.0) 4472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:11 smithi115 bash[45495]: cluster 2022-04-22T18:22:10.424846+0000 mon.a (mon.0) 4471 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2022-04-22T18:22:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:11 smithi115 bash[45495]: audit 2022-04-22T18:22:11.069202+0000 mon.a (mon.0) 4472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:11 smithi115 bash[46934]: cluster 2022-04-22T18:22:10.424846+0000 mon.a (mon.0) 4471 : cluster [DBG] osdmap e1759: 8 total, 8 up, 8 in 2022-04-22T18:22:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:11 smithi115 bash[46934]: audit 2022-04-22T18:22:11.069202+0000 mon.a (mon.0) 4472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:12.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:11.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:12 smithi121 bash[36902]: cluster 2022-04-22T18:22:11.418408+0000 mgr.y (mgr.34107) 1962 : cluster [DBG] pgmap v3164: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 3.7 KiB/s wr, 123 op/s 2022-04-22T18:22:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:12 smithi121 bash[36902]: cluster 2022-04-22T18:22:11.431718+0000 mon.a (mon.0) 4473 : cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2022-04-22T18:22:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:12 smithi121 bash[36902]: audit 2022-04-22T18:22:11.564399+0000 mon.a (mon.0) 4474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[45495]: cluster 2022-04-22T18:22:11.418408+0000 mgr.y (mgr.34107) 1962 : cluster [DBG] pgmap v3164: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 3.7 KiB/s wr, 123 op/s 2022-04-22T18:22:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[45495]: cluster 2022-04-22T18:22:11.431718+0000 mon.a (mon.0) 4473 : cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2022-04-22T18:22:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[45495]: audit 2022-04-22T18:22:11.564399+0000 mon.a (mon.0) 4474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[46934]: cluster 2022-04-22T18:22:11.418408+0000 mgr.y (mgr.34107) 1962 : cluster [DBG] pgmap v3164: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 3.7 KiB/s wr, 123 op/s 2022-04-22T18:22:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[46934]: cluster 2022-04-22T18:22:11.431718+0000 mon.a (mon.0) 4473 : cluster [DBG] osdmap e1760: 8 total, 8 up, 8 in 2022-04-22T18:22:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:12 smithi115 bash[46934]: audit 2022-04-22T18:22:11.564399+0000 mon.a (mon.0) 4474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:13.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:13 smithi121 bash[36902]: cluster 2022-04-22T18:22:12.442166+0000 mon.a (mon.0) 4475 : cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2022-04-22T18:22:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:13 smithi115 bash[45495]: cluster 2022-04-22T18:22:12.442166+0000 mon.a (mon.0) 4475 : cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2022-04-22T18:22:13.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:13.694Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:13 smithi115 bash[46934]: cluster 2022-04-22T18:22:12.442166+0000 mon.a (mon.0) 4475 : cluster [DBG] osdmap e1761: 8 total, 8 up, 8 in 2022-04-22T18:22:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:14 smithi121 bash[36902]: cluster 2022-04-22T18:22:13.419552+0000 mgr.y (mgr.34107) 1963 : cluster [DBG] pgmap v3167: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2022-04-22T18:22:14.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:14 smithi121 bash[36902]: cluster 2022-04-22T18:22:13.450453+0000 mon.a (mon.0) 4476 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2022-04-22T18:22:14.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:14 smithi121 bash[36902]: audit 2022-04-22T18:22:13.451113+0000 mon.a (mon.0) 4477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:14.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:14 smithi121 bash[36902]: audit 2022-04-22T18:22:13.910941+0000 mon.a (mon.0) 4478 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:22:14.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:14 smithi121 bash[36902]: audit 2022-04-22T18:22:13.914207+0000 mon.a (mon.0) 4479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:22:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[46934]: cluster 2022-04-22T18:22:13.419552+0000 mgr.y (mgr.34107) 1963 : cluster [DBG] pgmap v3167: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2022-04-22T18:22:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[46934]: cluster 2022-04-22T18:22:13.450453+0000 mon.a (mon.0) 4476 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2022-04-22T18:22:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[46934]: audit 2022-04-22T18:22:13.451113+0000 mon.a (mon.0) 4477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[46934]: audit 2022-04-22T18:22:13.910941+0000 mon.a (mon.0) 4478 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:22:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[46934]: audit 2022-04-22T18:22:13.914207+0000 mon.a (mon.0) 4479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:22:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[45495]: cluster 2022-04-22T18:22:13.419552+0000 mgr.y (mgr.34107) 1963 : cluster [DBG] pgmap v3167: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 93 KiB/s rd, 3.7 KiB/s wr, 124 op/s 2022-04-22T18:22:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[45495]: cluster 2022-04-22T18:22:13.450453+0000 mon.a (mon.0) 4476 : cluster [DBG] osdmap e1762: 8 total, 8 up, 8 in 2022-04-22T18:22:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[45495]: audit 2022-04-22T18:22:13.451113+0000 mon.a (mon.0) 4477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[45495]: audit 2022-04-22T18:22:13.910941+0000 mon.a (mon.0) 4478 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:22:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:14 smithi115 bash[45495]: audit 2022-04-22T18:22:13.914207+0000 mon.a (mon.0) 4479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:22:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:15 smithi121 bash[36902]: cluster 2022-04-22T18:22:14.452050+0000 mon.a (mon.0) 4480 : cluster [DBG] osdmap e1763: 8 total, 8 up, 8 in 2022-04-22T18:22:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:15 smithi115 bash[45495]: cluster 2022-04-22T18:22:14.452050+0000 mon.a (mon.0) 4480 : cluster [DBG] osdmap e1763: 8 total, 8 up, 8 in 2022-04-22T18:22:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:15 smithi115 bash[46934]: cluster 2022-04-22T18:22:14.452050+0000 mon.a (mon.0) 4480 : cluster [DBG] osdmap e1763: 8 total, 8 up, 8 in 2022-04-22T18:22:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:16 smithi121 bash[36902]: cluster 2022-04-22T18:22:15.420507+0000 mgr.y (mgr.34107) 1964 : cluster [DBG] pgmap v3170: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2022-04-22T18:22:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:16 smithi121 bash[36902]: cluster 2022-04-22T18:22:15.469321+0000 mon.a (mon.0) 4481 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2022-04-22T18:22:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:16 smithi121 bash[36902]: audit 2022-04-22T18:22:16.072502+0000 mon.a (mon.0) 4482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[45495]: cluster 2022-04-22T18:22:15.420507+0000 mgr.y (mgr.34107) 1964 : cluster [DBG] pgmap v3170: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2022-04-22T18:22:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[45495]: cluster 2022-04-22T18:22:15.469321+0000 mon.a (mon.0) 4481 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2022-04-22T18:22:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[45495]: audit 2022-04-22T18:22:16.072502+0000 mon.a (mon.0) 4482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[46934]: cluster 2022-04-22T18:22:15.420507+0000 mgr.y (mgr.34107) 1964 : cluster [DBG] pgmap v3170: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.7 KiB/s wr, 168 op/s 2022-04-22T18:22:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[46934]: cluster 2022-04-22T18:22:15.469321+0000 mon.a (mon.0) 4481 : cluster [DBG] osdmap e1764: 8 total, 8 up, 8 in 2022-04-22T18:22:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:16 smithi115 bash[46934]: audit 2022-04-22T18:22:16.072502+0000 mon.a (mon.0) 4482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:17 smithi121 bash[36902]: cluster 2022-04-22T18:22:16.476758+0000 mon.a (mon.0) 4483 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2022-04-22T18:22:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:17 smithi121 bash[36902]: audit 2022-04-22T18:22:16.568035+0000 mon.a (mon.0) 4484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:17 smithi115 bash[45495]: cluster 2022-04-22T18:22:16.476758+0000 mon.a (mon.0) 4483 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2022-04-22T18:22:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:17 smithi115 bash[45495]: audit 2022-04-22T18:22:16.568035+0000 mon.a (mon.0) 4484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:17 smithi115 bash[46934]: cluster 2022-04-22T18:22:16.476758+0000 mon.a (mon.0) 4483 : cluster [DBG] osdmap e1765: 8 total, 8 up, 8 in 2022-04-22T18:22:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:17 smithi115 bash[46934]: audit 2022-04-22T18:22:16.568035+0000 mon.a (mon.0) 4484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:18 smithi121 bash[36902]: cluster 2022-04-22T18:22:17.421260+0000 mgr.y (mgr.34107) 1965 : cluster [DBG] pgmap v3173: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.5 KiB/s wr, 168 op/s 2022-04-22T18:22:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:18 smithi121 bash[36902]: cluster 2022-04-22T18:22:17.486329+0000 mon.a (mon.0) 4485 : cluster [DBG] osdmap e1766: 8 total, 8 up, 8 in 2022-04-22T18:22:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:18 smithi121 bash[36902]: audit 2022-04-22T18:22:18.453132+0000 mon.a (mon.0) 4486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[45495]: cluster 2022-04-22T18:22:17.421260+0000 mgr.y (mgr.34107) 1965 : cluster [DBG] pgmap v3173: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.5 KiB/s wr, 168 op/s 2022-04-22T18:22:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[45495]: cluster 2022-04-22T18:22:17.486329+0000 mon.a (mon.0) 4485 : cluster [DBG] osdmap e1766: 8 total, 8 up, 8 in 2022-04-22T18:22:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[45495]: audit 2022-04-22T18:22:18.453132+0000 mon.a (mon.0) 4486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[46934]: cluster 2022-04-22T18:22:17.421260+0000 mgr.y (mgr.34107) 1965 : cluster [DBG] pgmap v3173: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s rd, 3.5 KiB/s wr, 168 op/s 2022-04-22T18:22:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[46934]: cluster 2022-04-22T18:22:17.486329+0000 mon.a (mon.0) 4485 : cluster [DBG] osdmap e1766: 8 total, 8 up, 8 in 2022-04-22T18:22:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:18 smithi115 bash[46934]: audit 2022-04-22T18:22:18.453132+0000 mon.a (mon.0) 4486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:19 smithi115 bash[45495]: cluster 2022-04-22T18:22:18.506001+0000 mon.a (mon.0) 4487 : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2022-04-22T18:22:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:19 smithi115 bash[46934]: cluster 2022-04-22T18:22:18.506001+0000 mon.a (mon.0) 4487 : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2022-04-22T18:22:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:19 smithi121 bash[36902]: cluster 2022-04-22T18:22:18.506001+0000 mon.a (mon.0) 4487 : cluster [DBG] osdmap e1767: 8 total, 8 up, 8 in 2022-04-22T18:22:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:20 smithi115 bash[45495]: cluster 2022-04-22T18:22:19.422198+0000 mgr.y (mgr.34107) 1966 : cluster [DBG] pgmap v3176: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:20 smithi115 bash[45495]: cluster 2022-04-22T18:22:19.514148+0000 mon.a (mon.0) 4488 : cluster [DBG] osdmap e1768: 8 total, 8 up, 8 in 2022-04-22T18:22:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:20 smithi115 bash[46934]: cluster 2022-04-22T18:22:19.422198+0000 mgr.y (mgr.34107) 1966 : cluster [DBG] pgmap v3176: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:20 smithi115 bash[46934]: cluster 2022-04-22T18:22:19.514148+0000 mon.a (mon.0) 4488 : cluster [DBG] osdmap e1768: 8 total, 8 up, 8 in 2022-04-22T18:22:21.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:20 smithi121 bash[36902]: cluster 2022-04-22T18:22:19.422198+0000 mgr.y (mgr.34107) 1966 : cluster [DBG] pgmap v3176: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:21.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:20 smithi121 bash[36902]: cluster 2022-04-22T18:22:19.514148+0000 mon.a (mon.0) 4488 : cluster [DBG] osdmap e1768: 8 total, 8 up, 8 in 2022-04-22T18:22:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:21 smithi115 bash[45495]: cluster 2022-04-22T18:22:20.523367+0000 mon.a (mon.0) 4489 : cluster [DBG] osdmap e1769: 8 total, 8 up, 8 in 2022-04-22T18:22:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:21 smithi115 bash[45495]: audit 2022-04-22T18:22:21.075522+0000 mon.a (mon.0) 4490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:21 smithi115 bash[46934]: cluster 2022-04-22T18:22:20.523367+0000 mon.a (mon.0) 4489 : cluster [DBG] osdmap e1769: 8 total, 8 up, 8 in 2022-04-22T18:22:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:21 smithi115 bash[46934]: audit 2022-04-22T18:22:21.075522+0000 mon.a (mon.0) 4490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:21 smithi121 bash[36902]: cluster 2022-04-22T18:22:20.523367+0000 mon.a (mon.0) 4489 : cluster [DBG] osdmap e1769: 8 total, 8 up, 8 in 2022-04-22T18:22:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:21 smithi121 bash[36902]: audit 2022-04-22T18:22:21.075522+0000 mon.a (mon.0) 4490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:21.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[45495]: cluster 2022-04-22T18:22:21.423269+0000 mgr.y (mgr.34107) 1967 : cluster [DBG] pgmap v3179: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 3.2 KiB/s wr, 210 op/s 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[45495]: cluster 2022-04-22T18:22:21.532226+0000 mon.a (mon.0) 4491 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[45495]: audit 2022-04-22T18:22:21.571396+0000 mon.a (mon.0) 4492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[46934]: cluster 2022-04-22T18:22:21.423269+0000 mgr.y (mgr.34107) 1967 : cluster [DBG] pgmap v3179: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 3.2 KiB/s wr, 210 op/s 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[46934]: cluster 2022-04-22T18:22:21.532226+0000 mon.a (mon.0) 4491 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2022-04-22T18:22:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:22 smithi115 bash[46934]: audit 2022-04-22T18:22:21.571396+0000 mon.a (mon.0) 4492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:22 smithi121 bash[36902]: cluster 2022-04-22T18:22:21.423269+0000 mgr.y (mgr.34107) 1967 : cluster [DBG] pgmap v3179: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 156 KiB/s rd, 3.2 KiB/s wr, 210 op/s 2022-04-22T18:22:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:22 smithi121 bash[36902]: cluster 2022-04-22T18:22:21.532226+0000 mon.a (mon.0) 4491 : cluster [DBG] osdmap e1770: 8 total, 8 up, 8 in 2022-04-22T18:22:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:22 smithi121 bash[36902]: audit 2022-04-22T18:22:21.571396+0000 mon.a (mon.0) 4492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:23 smithi121 bash[36902]: cluster 2022-04-22T18:22:22.544105+0000 mon.a (mon.0) 4493 : cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2022-04-22T18:22:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:23 smithi121 bash[36902]: audit 2022-04-22T18:22:23.456260+0000 mon.a (mon.0) 4494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:23.694Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:23 smithi115 bash[45495]: cluster 2022-04-22T18:22:22.544105+0000 mon.a (mon.0) 4493 : cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2022-04-22T18:22:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:23 smithi115 bash[45495]: audit 2022-04-22T18:22:23.456260+0000 mon.a (mon.0) 4494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:24.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:23 smithi115 bash[46934]: cluster 2022-04-22T18:22:22.544105+0000 mon.a (mon.0) 4493 : cluster [DBG] osdmap e1771: 8 total, 8 up, 8 in 2022-04-22T18:22:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:23 smithi115 bash[46934]: audit 2022-04-22T18:22:23.456260+0000 mon.a (mon.0) 4494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:24 smithi121 bash[36902]: cluster 2022-04-22T18:22:23.424422+0000 mgr.y (mgr.34107) 1968 : cluster [DBG] pgmap v3182: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 157 KiB/s rd, 3.2 KiB/s wr, 211 op/s 2022-04-22T18:22:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:24 smithi121 bash[36902]: cluster 2022-04-22T18:22:23.556094+0000 mon.a (mon.0) 4495 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2022-04-22T18:22:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:24 smithi115 bash[45495]: cluster 2022-04-22T18:22:23.424422+0000 mgr.y (mgr.34107) 1968 : cluster [DBG] pgmap v3182: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 157 KiB/s rd, 3.2 KiB/s wr, 211 op/s 2022-04-22T18:22:25.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:24 smithi115 bash[45495]: cluster 2022-04-22T18:22:23.556094+0000 mon.a (mon.0) 4495 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2022-04-22T18:22:25.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:24 smithi115 bash[46934]: cluster 2022-04-22T18:22:23.424422+0000 mgr.y (mgr.34107) 1968 : cluster [DBG] pgmap v3182: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 157 KiB/s rd, 3.2 KiB/s wr, 211 op/s 2022-04-22T18:22:25.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:24 smithi115 bash[46934]: cluster 2022-04-22T18:22:23.556094+0000 mon.a (mon.0) 4495 : cluster [DBG] osdmap e1772: 8 total, 8 up, 8 in 2022-04-22T18:22:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:25 smithi121 bash[36902]: cluster 2022-04-22T18:22:24.561695+0000 mon.a (mon.0) 4496 : cluster [DBG] osdmap e1773: 8 total, 8 up, 8 in 2022-04-22T18:22:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:25 smithi115 bash[45495]: cluster 2022-04-22T18:22:24.561695+0000 mon.a (mon.0) 4496 : cluster [DBG] osdmap e1773: 8 total, 8 up, 8 in 2022-04-22T18:22:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:25 smithi115 bash[46934]: cluster 2022-04-22T18:22:24.561695+0000 mon.a (mon.0) 4496 : cluster [DBG] osdmap e1773: 8 total, 8 up, 8 in 2022-04-22T18:22:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:26 smithi121 bash[36902]: cluster 2022-04-22T18:22:25.425436+0000 mgr.y (mgr.34107) 1969 : cluster [DBG] pgmap v3185: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.0 KiB/s wr, 256 op/s 2022-04-22T18:22:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:26 smithi121 bash[36902]: cluster 2022-04-22T18:22:25.579960+0000 mon.a (mon.0) 4497 : cluster [DBG] osdmap e1774: 8 total, 8 up, 8 in 2022-04-22T18:22:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:26 smithi121 bash[36902]: audit 2022-04-22T18:22:26.078900+0000 mon.a (mon.0) 4498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[45495]: cluster 2022-04-22T18:22:25.425436+0000 mgr.y (mgr.34107) 1969 : cluster [DBG] pgmap v3185: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.0 KiB/s wr, 256 op/s 2022-04-22T18:22:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[45495]: cluster 2022-04-22T18:22:25.579960+0000 mon.a (mon.0) 4497 : cluster [DBG] osdmap e1774: 8 total, 8 up, 8 in 2022-04-22T18:22:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[45495]: audit 2022-04-22T18:22:26.078900+0000 mon.a (mon.0) 4498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[46934]: cluster 2022-04-22T18:22:25.425436+0000 mgr.y (mgr.34107) 1969 : cluster [DBG] pgmap v3185: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.0 KiB/s wr, 256 op/s 2022-04-22T18:22:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[46934]: cluster 2022-04-22T18:22:25.579960+0000 mon.a (mon.0) 4497 : cluster [DBG] osdmap e1774: 8 total, 8 up, 8 in 2022-04-22T18:22:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:26 smithi115 bash[46934]: audit 2022-04-22T18:22:26.078900+0000 mon.a (mon.0) 4498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:27 smithi121 bash[36902]: audit 2022-04-22T18:22:26.574554+0000 mon.a (mon.0) 4499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:28.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:27 smithi121 bash[36902]: cluster 2022-04-22T18:22:26.589919+0000 mon.a (mon.0) 4500 : cluster [DBG] osdmap e1775: 8 total, 8 up, 8 in 2022-04-22T18:22:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:27 smithi115 bash[45495]: audit 2022-04-22T18:22:26.574554+0000 mon.a (mon.0) 4499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:27 smithi115 bash[45495]: cluster 2022-04-22T18:22:26.589919+0000 mon.a (mon.0) 4500 : cluster [DBG] osdmap e1775: 8 total, 8 up, 8 in 2022-04-22T18:22:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:27 smithi115 bash[46934]: audit 2022-04-22T18:22:26.574554+0000 mon.a (mon.0) 4499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:28.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:27 smithi115 bash[46934]: cluster 2022-04-22T18:22:26.589919+0000 mon.a (mon.0) 4500 : cluster [DBG] osdmap e1775: 8 total, 8 up, 8 in 2022-04-22T18:22:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:28 smithi121 bash[36902]: cluster 2022-04-22T18:22:27.426558+0000 mgr.y (mgr.34107) 1970 : cluster [DBG] pgmap v3188: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 3.5 KiB/s wr, 255 op/s 2022-04-22T18:22:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:28 smithi121 bash[36902]: cluster 2022-04-22T18:22:27.597672+0000 mon.a (mon.0) 4501 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2022-04-22T18:22:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:28 smithi121 bash[36902]: audit 2022-04-22T18:22:28.459590+0000 mon.a (mon.0) 4502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[45495]: cluster 2022-04-22T18:22:27.426558+0000 mgr.y (mgr.34107) 1970 : cluster [DBG] pgmap v3188: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 3.5 KiB/s wr, 255 op/s 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[45495]: cluster 2022-04-22T18:22:27.597672+0000 mon.a (mon.0) 4501 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[45495]: audit 2022-04-22T18:22:28.459590+0000 mon.a (mon.0) 4502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[46934]: cluster 2022-04-22T18:22:27.426558+0000 mgr.y (mgr.34107) 1970 : cluster [DBG] pgmap v3188: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 3.5 KiB/s wr, 255 op/s 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[46934]: cluster 2022-04-22T18:22:27.597672+0000 mon.a (mon.0) 4501 : cluster [DBG] osdmap e1776: 8 total, 8 up, 8 in 2022-04-22T18:22:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:28 smithi115 bash[46934]: audit 2022-04-22T18:22:28.459590+0000 mon.a (mon.0) 4502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:29 smithi121 bash[36902]: cluster 2022-04-22T18:22:28.613927+0000 mon.a (mon.0) 4503 : cluster [DBG] osdmap e1777: 8 total, 8 up, 8 in 2022-04-22T18:22:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:29 smithi115 bash[45495]: cluster 2022-04-22T18:22:28.613927+0000 mon.a (mon.0) 4503 : cluster [DBG] osdmap e1777: 8 total, 8 up, 8 in 2022-04-22T18:22:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:29 smithi115 bash[46934]: cluster 2022-04-22T18:22:28.613927+0000 mon.a (mon.0) 4503 : cluster [DBG] osdmap e1777: 8 total, 8 up, 8 in 2022-04-22T18:22:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:30 smithi121 bash[36902]: cluster 2022-04-22T18:22:29.427614+0000 mgr.y (mgr.34107) 1971 : cluster [DBG] pgmap v3191: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:30 smithi121 bash[36902]: cluster 2022-04-22T18:22:29.627272+0000 mon.a (mon.0) 4504 : cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2022-04-22T18:22:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:30 smithi115 bash[45495]: cluster 2022-04-22T18:22:29.427614+0000 mgr.y (mgr.34107) 1971 : cluster [DBG] pgmap v3191: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:30 smithi115 bash[45495]: cluster 2022-04-22T18:22:29.627272+0000 mon.a (mon.0) 4504 : cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2022-04-22T18:22:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:30 smithi115 bash[46934]: cluster 2022-04-22T18:22:29.427614+0000 mgr.y (mgr.34107) 1971 : cluster [DBG] pgmap v3191: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:22:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:30 smithi115 bash[46934]: cluster 2022-04-22T18:22:29.627272+0000 mon.a (mon.0) 4504 : cluster [DBG] osdmap e1778: 8 total, 8 up, 8 in 2022-04-22T18:22:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:31 smithi121 bash[36902]: cluster 2022-04-22T18:22:30.634081+0000 mon.a (mon.0) 4505 : cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2022-04-22T18:22:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:31 smithi121 bash[36902]: audit 2022-04-22T18:22:31.082118+0000 mon.a (mon.0) 4506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:31 smithi121 bash[36902]: audit 2022-04-22T18:22:31.578189+0000 mon.a (mon.0) 4507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:31.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[45495]: cluster 2022-04-22T18:22:30.634081+0000 mon.a (mon.0) 4505 : cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2022-04-22T18:22:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[45495]: audit 2022-04-22T18:22:31.082118+0000 mon.a (mon.0) 4506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[45495]: audit 2022-04-22T18:22:31.578189+0000 mon.a (mon.0) 4507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[46934]: cluster 2022-04-22T18:22:30.634081+0000 mon.a (mon.0) 4505 : cluster [DBG] osdmap e1779: 8 total, 8 up, 8 in 2022-04-22T18:22:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[46934]: audit 2022-04-22T18:22:31.082118+0000 mon.a (mon.0) 4506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:31 smithi115 bash[46934]: audit 2022-04-22T18:22:31.578189+0000 mon.a (mon.0) 4507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:32 smithi121 bash[36902]: cluster 2022-04-22T18:22:31.428552+0000 mgr.y (mgr.34107) 1972 : cluster [DBG] pgmap v3194: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 3.2 KiB/s wr, 298 op/s 2022-04-22T18:22:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:32 smithi121 bash[36902]: cluster 2022-04-22T18:22:31.639574+0000 mon.a (mon.0) 4508 : cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2022-04-22T18:22:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:32 smithi115 bash[45495]: cluster 2022-04-22T18:22:31.428552+0000 mgr.y (mgr.34107) 1972 : cluster [DBG] pgmap v3194: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 3.2 KiB/s wr, 298 op/s 2022-04-22T18:22:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:32 smithi115 bash[45495]: cluster 2022-04-22T18:22:31.639574+0000 mon.a (mon.0) 4508 : cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2022-04-22T18:22:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:32 smithi115 bash[46934]: cluster 2022-04-22T18:22:31.428552+0000 mgr.y (mgr.34107) 1972 : cluster [DBG] pgmap v3194: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 3.2 KiB/s wr, 298 op/s 2022-04-22T18:22:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:32 smithi115 bash[46934]: cluster 2022-04-22T18:22:31.639574+0000 mon.a (mon.0) 4508 : cluster [DBG] osdmap e1780: 8 total, 8 up, 8 in 2022-04-22T18:22:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:33 smithi121 bash[36902]: cluster 2022-04-22T18:22:32.665512+0000 mon.a (mon.0) 4509 : cluster [DBG] osdmap e1781: 8 total, 8 up, 8 in 2022-04-22T18:22:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:33 smithi121 bash[36902]: audit 2022-04-22T18:22:33.462551+0000 mon.a (mon.0) 4510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:33.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:33 smithi115 bash[45495]: cluster 2022-04-22T18:22:32.665512+0000 mon.a (mon.0) 4509 : cluster [DBG] osdmap e1781: 8 total, 8 up, 8 in 2022-04-22T18:22:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:33 smithi115 bash[45495]: audit 2022-04-22T18:22:33.462551+0000 mon.a (mon.0) 4510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:33 smithi115 bash[46934]: cluster 2022-04-22T18:22:32.665512+0000 mon.a (mon.0) 4509 : cluster [DBG] osdmap e1781: 8 total, 8 up, 8 in 2022-04-22T18:22:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:33 smithi115 bash[46934]: audit 2022-04-22T18:22:33.462551+0000 mon.a (mon.0) 4510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:34 smithi121 bash[36902]: cluster 2022-04-22T18:22:33.429777+0000 mgr.y (mgr.34107) 1973 : cluster [DBG] pgmap v3197: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 2.7 KiB/s wr, 298 op/s 2022-04-22T18:22:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:34 smithi121 bash[36902]: cluster 2022-04-22T18:22:33.677997+0000 mon.a (mon.0) 4511 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2022-04-22T18:22:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:34 smithi115 bash[45495]: cluster 2022-04-22T18:22:33.429777+0000 mgr.y (mgr.34107) 1973 : cluster [DBG] pgmap v3197: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 2.7 KiB/s wr, 298 op/s 2022-04-22T18:22:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:34 smithi115 bash[45495]: cluster 2022-04-22T18:22:33.677997+0000 mon.a (mon.0) 4511 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2022-04-22T18:22:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:34 smithi115 bash[46934]: cluster 2022-04-22T18:22:33.429777+0000 mgr.y (mgr.34107) 1973 : cluster [DBG] pgmap v3197: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 2.7 KiB/s wr, 298 op/s 2022-04-22T18:22:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:34 smithi115 bash[46934]: cluster 2022-04-22T18:22:33.677997+0000 mon.a (mon.0) 4511 : cluster [DBG] osdmap e1782: 8 total, 8 up, 8 in 2022-04-22T18:22:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:35 smithi121 bash[36902]: cluster 2022-04-22T18:22:34.683801+0000 mon.a (mon.0) 4512 : cluster [DBG] osdmap e1783: 8 total, 8 up, 8 in 2022-04-22T18:22:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:35 smithi115 bash[45495]: cluster 2022-04-22T18:22:34.683801+0000 mon.a (mon.0) 4512 : cluster [DBG] osdmap e1783: 8 total, 8 up, 8 in 2022-04-22T18:22:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:35 smithi115 bash[46934]: cluster 2022-04-22T18:22:34.683801+0000 mon.a (mon.0) 4512 : cluster [DBG] osdmap e1783: 8 total, 8 up, 8 in 2022-04-22T18:22:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:36 smithi121 bash[36902]: cluster 2022-04-22T18:22:35.430817+0000 mgr.y (mgr.34107) 1974 : cluster [DBG] pgmap v3200: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 3.5 KiB/s wr, 343 op/s 2022-04-22T18:22:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:36 smithi121 bash[36902]: cluster 2022-04-22T18:22:35.693776+0000 mon.a (mon.0) 4513 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2022-04-22T18:22:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:36 smithi121 bash[36902]: audit 2022-04-22T18:22:36.085333+0000 mon.a (mon.0) 4514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:37.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:36 smithi121 bash[36902]: audit 2022-04-22T18:22:36.581900+0000 mon.a (mon.0) 4515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[45495]: cluster 2022-04-22T18:22:35.430817+0000 mgr.y (mgr.34107) 1974 : cluster [DBG] pgmap v3200: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 3.5 KiB/s wr, 343 op/s 2022-04-22T18:22:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[45495]: cluster 2022-04-22T18:22:35.693776+0000 mon.a (mon.0) 4513 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2022-04-22T18:22:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[45495]: audit 2022-04-22T18:22:36.085333+0000 mon.a (mon.0) 4514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[45495]: audit 2022-04-22T18:22:36.581900+0000 mon.a (mon.0) 4515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[46934]: cluster 2022-04-22T18:22:35.430817+0000 mgr.y (mgr.34107) 1974 : cluster [DBG] pgmap v3200: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 3.5 KiB/s wr, 343 op/s 2022-04-22T18:22:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[46934]: cluster 2022-04-22T18:22:35.693776+0000 mon.a (mon.0) 4513 : cluster [DBG] osdmap e1784: 8 total, 8 up, 8 in 2022-04-22T18:22:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[46934]: audit 2022-04-22T18:22:36.085333+0000 mon.a (mon.0) 4514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:36 smithi115 bash[46934]: audit 2022-04-22T18:22:36.581900+0000 mon.a (mon.0) 4515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:37 smithi121 bash[36902]: cluster 2022-04-22T18:22:36.702094+0000 mon.a (mon.0) 4516 : cluster [DBG] osdmap e1785: 8 total, 8 up, 8 in 2022-04-22T18:22:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:37 smithi121 bash[36902]: cluster 2022-04-22T18:22:37.431901+0000 mgr.y (mgr.34107) 1975 : cluster [DBG] pgmap v3203: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 4.5 KiB/s wr, 344 op/s 2022-04-22T18:22:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:37 smithi115 bash[45495]: cluster 2022-04-22T18:22:36.702094+0000 mon.a (mon.0) 4516 : cluster [DBG] osdmap e1785: 8 total, 8 up, 8 in 2022-04-22T18:22:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:37 smithi115 bash[45495]: cluster 2022-04-22T18:22:37.431901+0000 mgr.y (mgr.34107) 1975 : cluster [DBG] pgmap v3203: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 4.5 KiB/s wr, 344 op/s 2022-04-22T18:22:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:37 smithi115 bash[46934]: cluster 2022-04-22T18:22:36.702094+0000 mon.a (mon.0) 4516 : cluster [DBG] osdmap e1785: 8 total, 8 up, 8 in 2022-04-22T18:22:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:37 smithi115 bash[46934]: cluster 2022-04-22T18:22:37.431901+0000 mgr.y (mgr.34107) 1975 : cluster [DBG] pgmap v3203: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 4.5 KiB/s wr, 344 op/s 2022-04-22T18:22:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:38 smithi121 bash[36902]: cluster 2022-04-22T18:22:37.720156+0000 mon.a (mon.0) 4517 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2022-04-22T18:22:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:38 smithi121 bash[36902]: audit 2022-04-22T18:22:38.465846+0000 mon.a (mon.0) 4518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:38 smithi115 bash[45495]: cluster 2022-04-22T18:22:37.720156+0000 mon.a (mon.0) 4517 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2022-04-22T18:22:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:38 smithi115 bash[45495]: audit 2022-04-22T18:22:38.465846+0000 mon.a (mon.0) 4518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:38 smithi115 bash[46934]: cluster 2022-04-22T18:22:37.720156+0000 mon.a (mon.0) 4517 : cluster [DBG] osdmap e1786: 8 total, 8 up, 8 in 2022-04-22T18:22:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:38 smithi115 bash[46934]: audit 2022-04-22T18:22:38.465846+0000 mon.a (mon.0) 4518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:39 smithi121 bash[36902]: cluster 2022-04-22T18:22:38.724707+0000 mon.a (mon.0) 4519 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2022-04-22T18:22:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:39 smithi121 bash[36902]: cluster 2022-04-22T18:22:39.433031+0000 mgr.y (mgr.34107) 1976 : cluster [DBG] pgmap v3206: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:22:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:39 smithi115 bash[46934]: cluster 2022-04-22T18:22:38.724707+0000 mon.a (mon.0) 4519 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2022-04-22T18:22:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:39 smithi115 bash[46934]: cluster 2022-04-22T18:22:39.433031+0000 mgr.y (mgr.34107) 1976 : cluster [DBG] pgmap v3206: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:22:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:39 smithi115 bash[45495]: cluster 2022-04-22T18:22:38.724707+0000 mon.a (mon.0) 4519 : cluster [DBG] osdmap e1787: 8 total, 8 up, 8 in 2022-04-22T18:22:40.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:39 smithi115 bash[45495]: cluster 2022-04-22T18:22:39.433031+0000 mgr.y (mgr.34107) 1976 : cluster [DBG] pgmap v3206: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:22:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (45m) 3m ago 59m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (59m) 4m ago 59m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (47m) 3m ago 60m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (47m) 4m ago 60m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (54m) 4m ago 64m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:22:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (55m) 3m ago 66m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (54m) 3m ago 67m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (53m) 4m ago 65m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (54m) 3m ago 65m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (46m) 3m ago 59m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (46m) 4m ago 59m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (52m) 3m ago 64m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (51m) 3m ago 63m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (51m) 3m ago 63m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (50m) 3m ago 62m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:22:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (50m) 4m ago 62m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:22:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (49m) 4m ago 62m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:22:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (48m) 4m ago 61m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:22:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (48m) 4m ago 61m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:22:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (46m) 4m ago 59m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:22:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:40 smithi121 bash[36902]: cluster 2022-04-22T18:22:39.745673+0000 mon.a (mon.0) 4520 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2022-04-22T18:22:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:40 smithi121 bash[36902]: audit 2022-04-22T18:22:40.312683+0000 mgr.y (mgr.34107) 1977 : audit [DBG] from='client.37437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:40 smithi121 bash[36902]: audit 2022-04-22T18:22:40.654593+0000 mgr.y (mgr.34107) 1978 : audit [DBG] from='client.37443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[46934]: cluster 2022-04-22T18:22:39.745673+0000 mon.a (mon.0) 4520 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2022-04-22T18:22:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[46934]: audit 2022-04-22T18:22:40.312683+0000 mgr.y (mgr.34107) 1977 : audit [DBG] from='client.37437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[46934]: audit 2022-04-22T18:22:40.654593+0000 mgr.y (mgr.34107) 1978 : audit [DBG] from='client.37443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[45495]: cluster 2022-04-22T18:22:39.745673+0000 mon.a (mon.0) 4520 : cluster [DBG] osdmap e1788: 8 total, 8 up, 8 in 2022-04-22T18:22:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[45495]: audit 2022-04-22T18:22:40.312683+0000 mgr.y (mgr.34107) 1977 : audit [DBG] from='client.37437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:40 smithi115 bash[45495]: audit 2022-04-22T18:22:40.654593+0000 mgr.y (mgr.34107) 1978 : audit [DBG] from='client.37443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:22:41.091 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:22:41.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:22:41.092 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:22:41.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:41.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:22:41.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:22:41.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:41.094 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:22:41.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:22:41.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:41.094 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:22:41.095 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:22:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:41 smithi121 bash[36902]: cluster 2022-04-22T18:22:40.746584+0000 mon.a (mon.0) 4521 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2022-04-22T18:22:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:41 smithi121 bash[36902]: audit 2022-04-22T18:22:41.084374+0000 mon.a (mon.0) 4522 : audit [DBG] from='client.? 172.21.15.115:0/2661240922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:41 smithi121 bash[36902]: audit 2022-04-22T18:22:41.088466+0000 mon.a (mon.0) 4523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:41 smithi121 bash[36902]: cluster 2022-04-22T18:22:41.434138+0000 mgr.y (mgr.34107) 1979 : cluster [DBG] pgmap v3209: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 3.0 KiB/s wr, 385 op/s 2022-04-22T18:22:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:41 smithi121 bash[36902]: audit 2022-04-22T18:22:41.585886+0000 mon.a (mon.0) 4524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:41.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[45495]: cluster 2022-04-22T18:22:40.746584+0000 mon.a (mon.0) 4521 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2022-04-22T18:22:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[45495]: audit 2022-04-22T18:22:41.084374+0000 mon.a (mon.0) 4522 : audit [DBG] from='client.? 172.21.15.115:0/2661240922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[45495]: audit 2022-04-22T18:22:41.088466+0000 mon.a (mon.0) 4523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[45495]: cluster 2022-04-22T18:22:41.434138+0000 mgr.y (mgr.34107) 1979 : cluster [DBG] pgmap v3209: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 3.0 KiB/s wr, 385 op/s 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[45495]: audit 2022-04-22T18:22:41.585886+0000 mon.a (mon.0) 4524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[46934]: cluster 2022-04-22T18:22:40.746584+0000 mon.a (mon.0) 4521 : cluster [DBG] osdmap e1789: 8 total, 8 up, 8 in 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[46934]: audit 2022-04-22T18:22:41.084374+0000 mon.a (mon.0) 4522 : audit [DBG] from='client.? 172.21.15.115:0/2661240922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[46934]: audit 2022-04-22T18:22:41.088466+0000 mon.a (mon.0) 4523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[46934]: cluster 2022-04-22T18:22:41.434138+0000 mgr.y (mgr.34107) 1979 : cluster [DBG] pgmap v3209: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 287 KiB/s rd, 3.0 KiB/s wr, 385 op/s 2022-04-22T18:22:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:41 smithi115 bash[46934]: audit 2022-04-22T18:22:41.585886+0000 mon.a (mon.0) 4524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:42 smithi115 bash[45495]: cluster 2022-04-22T18:22:41.771287+0000 mon.a (mon.0) 4525 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2022-04-22T18:22:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:42 smithi115 bash[46934]: cluster 2022-04-22T18:22:41.771287+0000 mon.a (mon.0) 4525 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2022-04-22T18:22:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:42 smithi121 bash[36902]: cluster 2022-04-22T18:22:41.771287+0000 mon.a (mon.0) 4525 : cluster [DBG] osdmap e1790: 8 total, 8 up, 8 in 2022-04-22T18:22:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:43.695Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[45495]: cluster 2022-04-22T18:22:42.785196+0000 mon.a (mon.0) 4526 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2022-04-22T18:22:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[45495]: cluster 2022-04-22T18:22:43.435255+0000 mgr.y (mgr.34107) 1980 : cluster [DBG] pgmap v3212: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 3.2 KiB/s wr, 386 op/s 2022-04-22T18:22:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[45495]: audit 2022-04-22T18:22:43.468607+0000 mon.a (mon.0) 4527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[46934]: cluster 2022-04-22T18:22:42.785196+0000 mon.a (mon.0) 4526 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2022-04-22T18:22:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[46934]: cluster 2022-04-22T18:22:43.435255+0000 mgr.y (mgr.34107) 1980 : cluster [DBG] pgmap v3212: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 3.2 KiB/s wr, 386 op/s 2022-04-22T18:22:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:43 smithi115 bash[46934]: audit 2022-04-22T18:22:43.468607+0000 mon.a (mon.0) 4527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:43 smithi121 bash[36902]: cluster 2022-04-22T18:22:42.785196+0000 mon.a (mon.0) 4526 : cluster [DBG] osdmap e1791: 8 total, 8 up, 8 in 2022-04-22T18:22:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:43 smithi121 bash[36902]: cluster 2022-04-22T18:22:43.435255+0000 mgr.y (mgr.34107) 1980 : cluster [DBG] pgmap v3212: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 289 KiB/s rd, 3.2 KiB/s wr, 386 op/s 2022-04-22T18:22:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:43 smithi121 bash[36902]: audit 2022-04-22T18:22:43.468607+0000 mon.a (mon.0) 4527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:44 smithi121 bash[36902]: cluster 2022-04-22T18:22:43.796859+0000 mon.a (mon.0) 4528 : cluster [DBG] osdmap e1792: 8 total, 8 up, 8 in 2022-04-22T18:22:45.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:44 smithi115 bash[45495]: cluster 2022-04-22T18:22:43.796859+0000 mon.a (mon.0) 4528 : cluster [DBG] osdmap e1792: 8 total, 8 up, 8 in 2022-04-22T18:22:45.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:44 smithi115 bash[46934]: cluster 2022-04-22T18:22:43.796859+0000 mon.a (mon.0) 4528 : cluster [DBG] osdmap e1792: 8 total, 8 up, 8 in 2022-04-22T18:22:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:45 smithi121 bash[36902]: cluster 2022-04-22T18:22:44.797289+0000 mon.a (mon.0) 4529 : cluster [DBG] osdmap e1793: 8 total, 8 up, 8 in 2022-04-22T18:22:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:45 smithi121 bash[36902]: cluster 2022-04-22T18:22:45.436470+0000 mgr.y (mgr.34107) 1981 : cluster [DBG] pgmap v3215: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:46.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:45 smithi115 bash[46934]: cluster 2022-04-22T18:22:44.797289+0000 mon.a (mon.0) 4529 : cluster [DBG] osdmap e1793: 8 total, 8 up, 8 in 2022-04-22T18:22:46.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:45 smithi115 bash[46934]: cluster 2022-04-22T18:22:45.436470+0000 mgr.y (mgr.34107) 1981 : cluster [DBG] pgmap v3215: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:45 smithi115 bash[45495]: cluster 2022-04-22T18:22:44.797289+0000 mon.a (mon.0) 4529 : cluster [DBG] osdmap e1793: 8 total, 8 up, 8 in 2022-04-22T18:22:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:45 smithi115 bash[45495]: cluster 2022-04-22T18:22:45.436470+0000 mgr.y (mgr.34107) 1981 : cluster [DBG] pgmap v3215: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:46 smithi121 bash[36902]: cluster 2022-04-22T18:22:45.821238+0000 mon.a (mon.0) 4530 : cluster [DBG] osdmap e1794: 8 total, 8 up, 8 in 2022-04-22T18:22:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:46 smithi121 bash[36902]: audit 2022-04-22T18:22:46.091596+0000 mon.a (mon.0) 4531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:46 smithi121 bash[36902]: audit 2022-04-22T18:22:46.589799+0000 mon.a (mon.0) 4532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[45495]: cluster 2022-04-22T18:22:45.821238+0000 mon.a (mon.0) 4530 : cluster [DBG] osdmap e1794: 8 total, 8 up, 8 in 2022-04-22T18:22:47.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[45495]: audit 2022-04-22T18:22:46.091596+0000 mon.a (mon.0) 4531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:47.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[45495]: audit 2022-04-22T18:22:46.589799+0000 mon.a (mon.0) 4532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:47.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[46934]: cluster 2022-04-22T18:22:45.821238+0000 mon.a (mon.0) 4530 : cluster [DBG] osdmap e1794: 8 total, 8 up, 8 in 2022-04-22T18:22:47.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[46934]: audit 2022-04-22T18:22:46.091596+0000 mon.a (mon.0) 4531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:47.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:46 smithi115 bash[46934]: audit 2022-04-22T18:22:46.589799+0000 mon.a (mon.0) 4532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:47 smithi121 bash[36902]: cluster 2022-04-22T18:22:46.825284+0000 mon.a (mon.0) 4533 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2022-04-22T18:22:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:47 smithi121 bash[36902]: cluster 2022-04-22T18:22:47.437561+0000 mgr.y (mgr.34107) 1982 : cluster [DBG] pgmap v3218: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:47 smithi115 bash[45495]: cluster 2022-04-22T18:22:46.825284+0000 mon.a (mon.0) 4533 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2022-04-22T18:22:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:47 smithi115 bash[45495]: cluster 2022-04-22T18:22:47.437561+0000 mgr.y (mgr.34107) 1982 : cluster [DBG] pgmap v3218: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:47 smithi115 bash[46934]: cluster 2022-04-22T18:22:46.825284+0000 mon.a (mon.0) 4533 : cluster [DBG] osdmap e1795: 8 total, 8 up, 8 in 2022-04-22T18:22:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:47 smithi115 bash[46934]: cluster 2022-04-22T18:22:47.437561+0000 mgr.y (mgr.34107) 1982 : cluster [DBG] pgmap v3218: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 3.5 KiB/s wr, 430 op/s 2022-04-22T18:22:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:48 smithi121 bash[36902]: cluster 2022-04-22T18:22:47.838345+0000 mon.a (mon.0) 4534 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2022-04-22T18:22:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:48 smithi121 bash[36902]: audit 2022-04-22T18:22:48.471774+0000 mon.a (mon.0) 4535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:49.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:48 smithi115 bash[45495]: cluster 2022-04-22T18:22:47.838345+0000 mon.a (mon.0) 4534 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2022-04-22T18:22:49.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:48 smithi115 bash[45495]: audit 2022-04-22T18:22:48.471774+0000 mon.a (mon.0) 4535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:49.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:48 smithi115 bash[46934]: cluster 2022-04-22T18:22:47.838345+0000 mon.a (mon.0) 4534 : cluster [DBG] osdmap e1796: 8 total, 8 up, 8 in 2022-04-22T18:22:49.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:48 smithi115 bash[46934]: audit 2022-04-22T18:22:48.471774+0000 mon.a (mon.0) 4535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:49 smithi121 bash[36902]: cluster 2022-04-22T18:22:48.847565+0000 mon.a (mon.0) 4536 : cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2022-04-22T18:22:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:49 smithi121 bash[36902]: cluster 2022-04-22T18:22:49.438565+0000 mgr.y (mgr.34107) 1983 : cluster [DBG] pgmap v3221: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:22:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:49 smithi115 bash[45495]: cluster 2022-04-22T18:22:48.847565+0000 mon.a (mon.0) 4536 : cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2022-04-22T18:22:50.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:49 smithi115 bash[45495]: cluster 2022-04-22T18:22:49.438565+0000 mgr.y (mgr.34107) 1983 : cluster [DBG] pgmap v3221: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:22:50.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:49 smithi115 bash[46934]: cluster 2022-04-22T18:22:48.847565+0000 mon.a (mon.0) 4536 : cluster [DBG] osdmap e1797: 8 total, 8 up, 8 in 2022-04-22T18:22:50.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:49 smithi115 bash[46934]: cluster 2022-04-22T18:22:49.438565+0000 mgr.y (mgr.34107) 1983 : cluster [DBG] pgmap v3221: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:22:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:50 smithi121 bash[36902]: cluster 2022-04-22T18:22:49.855617+0000 mon.a (mon.0) 4537 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2022-04-22T18:22:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:50 smithi115 bash[45495]: cluster 2022-04-22T18:22:49.855617+0000 mon.a (mon.0) 4537 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2022-04-22T18:22:51.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:50 smithi115 bash[46934]: cluster 2022-04-22T18:22:49.855617+0000 mon.a (mon.0) 4537 : cluster [DBG] osdmap e1798: 8 total, 8 up, 8 in 2022-04-22T18:22:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:51 smithi121 bash[36902]: cluster 2022-04-22T18:22:50.862576+0000 mon.a (mon.0) 4538 : cluster [DBG] osdmap e1799: 8 total, 8 up, 8 in 2022-04-22T18:22:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:51 smithi121 bash[36902]: audit 2022-04-22T18:22:51.094681+0000 mon.a (mon.0) 4539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:51 smithi121 bash[36902]: cluster 2022-04-22T18:22:51.439589+0000 mgr.y (mgr.34107) 1984 : cluster [DBG] pgmap v3224: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 352 KiB/s rd, 3.7 KiB/s wr, 473 op/s 2022-04-22T18:22:52.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:51 smithi121 bash[36902]: audit 2022-04-22T18:22:51.593435+0000 mon.a (mon.0) 4540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:51.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:52.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[46934]: cluster 2022-04-22T18:22:50.862576+0000 mon.a (mon.0) 4538 : cluster [DBG] osdmap e1799: 8 total, 8 up, 8 in 2022-04-22T18:22:52.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[46934]: audit 2022-04-22T18:22:51.094681+0000 mon.a (mon.0) 4539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[46934]: cluster 2022-04-22T18:22:51.439589+0000 mgr.y (mgr.34107) 1984 : cluster [DBG] pgmap v3224: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 352 KiB/s rd, 3.7 KiB/s wr, 473 op/s 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[46934]: audit 2022-04-22T18:22:51.593435+0000 mon.a (mon.0) 4540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[45495]: cluster 2022-04-22T18:22:50.862576+0000 mon.a (mon.0) 4538 : cluster [DBG] osdmap e1799: 8 total, 8 up, 8 in 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[45495]: audit 2022-04-22T18:22:51.094681+0000 mon.a (mon.0) 4539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[45495]: cluster 2022-04-22T18:22:51.439589+0000 mgr.y (mgr.34107) 1984 : cluster [DBG] pgmap v3224: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 352 KiB/s rd, 3.7 KiB/s wr, 473 op/s 2022-04-22T18:22:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:51 smithi115 bash[45495]: audit 2022-04-22T18:22:51.593435+0000 mon.a (mon.0) 4540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:52 smithi121 bash[36902]: cluster 2022-04-22T18:22:51.867974+0000 mon.a (mon.0) 4541 : cluster [DBG] osdmap e1800: 8 total, 8 up, 8 in 2022-04-22T18:22:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:52 smithi115 bash[45495]: cluster 2022-04-22T18:22:51.867974+0000 mon.a (mon.0) 4541 : cluster [DBG] osdmap e1800: 8 total, 8 up, 8 in 2022-04-22T18:22:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:52 smithi115 bash[46934]: cluster 2022-04-22T18:22:51.867974+0000 mon.a (mon.0) 4541 : cluster [DBG] osdmap e1800: 8 total, 8 up, 8 in 2022-04-22T18:22:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:22:53.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:22:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[45495]: cluster 2022-04-22T18:22:52.894553+0000 mon.a (mon.0) 4542 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2022-04-22T18:22:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[45495]: cluster 2022-04-22T18:22:53.440646+0000 mgr.y (mgr.34107) 1985 : cluster [DBG] pgmap v3227: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 353 KiB/s rd, 3.5 KiB/s wr, 474 op/s 2022-04-22T18:22:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[45495]: audit 2022-04-22T18:22:53.474613+0000 mon.a (mon.0) 4543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[46934]: cluster 2022-04-22T18:22:52.894553+0000 mon.a (mon.0) 4542 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2022-04-22T18:22:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[46934]: cluster 2022-04-22T18:22:53.440646+0000 mgr.y (mgr.34107) 1985 : cluster [DBG] pgmap v3227: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 353 KiB/s rd, 3.5 KiB/s wr, 474 op/s 2022-04-22T18:22:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:53 smithi115 bash[46934]: audit 2022-04-22T18:22:53.474613+0000 mon.a (mon.0) 4543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:53 smithi121 bash[36902]: cluster 2022-04-22T18:22:52.894553+0000 mon.a (mon.0) 4542 : cluster [DBG] osdmap e1801: 8 total, 8 up, 8 in 2022-04-22T18:22:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:53 smithi121 bash[36902]: cluster 2022-04-22T18:22:53.440646+0000 mgr.y (mgr.34107) 1985 : cluster [DBG] pgmap v3227: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 353 KiB/s rd, 3.5 KiB/s wr, 474 op/s 2022-04-22T18:22:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:53 smithi121 bash[36902]: audit 2022-04-22T18:22:53.474613+0000 mon.a (mon.0) 4543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:54 smithi121 bash[36902]: cluster 2022-04-22T18:22:53.900427+0000 mon.a (mon.0) 4544 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2022-04-22T18:22:55.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:54 smithi115 bash[45495]: cluster 2022-04-22T18:22:53.900427+0000 mon.a (mon.0) 4544 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2022-04-22T18:22:55.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:54 smithi115 bash[46934]: cluster 2022-04-22T18:22:53.900427+0000 mon.a (mon.0) 4544 : cluster [DBG] osdmap e1802: 8 total, 8 up, 8 in 2022-04-22T18:22:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:55 smithi121 bash[36902]: cluster 2022-04-22T18:22:54.908119+0000 mon.a (mon.0) 4545 : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2022-04-22T18:22:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:55 smithi121 bash[36902]: cluster 2022-04-22T18:22:55.441656+0000 mgr.y (mgr.34107) 1986 : cluster [DBG] pgmap v3230: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 3.0 KiB/s wr, 517 op/s 2022-04-22T18:22:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:55 smithi115 bash[45495]: cluster 2022-04-22T18:22:54.908119+0000 mon.a (mon.0) 4545 : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2022-04-22T18:22:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:55 smithi115 bash[45495]: cluster 2022-04-22T18:22:55.441656+0000 mgr.y (mgr.34107) 1986 : cluster [DBG] pgmap v3230: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 3.0 KiB/s wr, 517 op/s 2022-04-22T18:22:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:55 smithi115 bash[46934]: cluster 2022-04-22T18:22:54.908119+0000 mon.a (mon.0) 4545 : cluster [DBG] osdmap e1803: 8 total, 8 up, 8 in 2022-04-22T18:22:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:55 smithi115 bash[46934]: cluster 2022-04-22T18:22:55.441656+0000 mgr.y (mgr.34107) 1986 : cluster [DBG] pgmap v3230: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 3.0 KiB/s wr, 517 op/s 2022-04-22T18:22:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:56 smithi121 bash[36902]: cluster 2022-04-22T18:22:55.923863+0000 mon.a (mon.0) 4546 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2022-04-22T18:22:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:56 smithi121 bash[36902]: audit 2022-04-22T18:22:56.097846+0000 mon.a (mon.0) 4547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:56 smithi121 bash[36902]: audit 2022-04-22T18:22:56.597198+0000 mon.a (mon.0) 4548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:57.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[45495]: cluster 2022-04-22T18:22:55.923863+0000 mon.a (mon.0) 4546 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2022-04-22T18:22:57.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[45495]: audit 2022-04-22T18:22:56.097846+0000 mon.a (mon.0) 4547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:57.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[45495]: audit 2022-04-22T18:22:56.597198+0000 mon.a (mon.0) 4548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[46934]: cluster 2022-04-22T18:22:55.923863+0000 mon.a (mon.0) 4546 : cluster [DBG] osdmap e1804: 8 total, 8 up, 8 in 2022-04-22T18:22:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[46934]: audit 2022-04-22T18:22:56.097846+0000 mon.a (mon.0) 4547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:22:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:56 smithi115 bash[46934]: audit 2022-04-22T18:22:56.597198+0000 mon.a (mon.0) 4548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:22:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:57 smithi121 bash[36902]: cluster 2022-04-22T18:22:56.925624+0000 mon.a (mon.0) 4549 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2022-04-22T18:22:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:57 smithi121 bash[36902]: cluster 2022-04-22T18:22:57.442659+0000 mgr.y (mgr.34107) 1987 : cluster [DBG] pgmap v3233: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 3.5 KiB/s wr, 518 op/s 2022-04-22T18:22:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:57 smithi115 bash[45495]: cluster 2022-04-22T18:22:56.925624+0000 mon.a (mon.0) 4549 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2022-04-22T18:22:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:57 smithi115 bash[45495]: cluster 2022-04-22T18:22:57.442659+0000 mgr.y (mgr.34107) 1987 : cluster [DBG] pgmap v3233: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 3.5 KiB/s wr, 518 op/s 2022-04-22T18:22:58.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:57 smithi115 bash[46934]: cluster 2022-04-22T18:22:56.925624+0000 mon.a (mon.0) 4549 : cluster [DBG] osdmap e1805: 8 total, 8 up, 8 in 2022-04-22T18:22:58.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:57 smithi115 bash[46934]: cluster 2022-04-22T18:22:57.442659+0000 mgr.y (mgr.34107) 1987 : cluster [DBG] pgmap v3233: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 3.5 KiB/s wr, 518 op/s 2022-04-22T18:22:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:58 smithi121 bash[36902]: cluster 2022-04-22T18:22:57.945059+0000 mon.a (mon.0) 4550 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2022-04-22T18:22:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:58 smithi121 bash[36902]: audit 2022-04-22T18:22:58.477690+0000 mon.a (mon.0) 4551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:59.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:58 smithi115 bash[45495]: cluster 2022-04-22T18:22:57.945059+0000 mon.a (mon.0) 4550 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2022-04-22T18:22:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:58 smithi115 bash[45495]: audit 2022-04-22T18:22:58.477690+0000 mon.a (mon.0) 4551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:22:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:58 smithi115 bash[46934]: cluster 2022-04-22T18:22:57.945059+0000 mon.a (mon.0) 4550 : cluster [DBG] osdmap e1806: 8 total, 8 up, 8 in 2022-04-22T18:22:59.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:58 smithi115 bash[46934]: audit 2022-04-22T18:22:58.477690+0000 mon.a (mon.0) 4551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:59 smithi121 bash[36902]: cluster 2022-04-22T18:22:58.951419+0000 mon.a (mon.0) 4552 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2022-04-22T18:23:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:22:59 smithi121 bash[36902]: cluster 2022-04-22T18:22:59.443512+0000 mgr.y (mgr.34107) 1988 : cluster [DBG] pgmap v3236: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:59 smithi115 bash[45495]: cluster 2022-04-22T18:22:58.951419+0000 mon.a (mon.0) 4552 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2022-04-22T18:23:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:22:59 smithi115 bash[45495]: cluster 2022-04-22T18:22:59.443512+0000 mgr.y (mgr.34107) 1988 : cluster [DBG] pgmap v3236: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:00.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:59 smithi115 bash[46934]: cluster 2022-04-22T18:22:58.951419+0000 mon.a (mon.0) 4552 : cluster [DBG] osdmap e1807: 8 total, 8 up, 8 in 2022-04-22T18:23:00.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:22:59 smithi115 bash[46934]: cluster 2022-04-22T18:22:59.443512+0000 mgr.y (mgr.34107) 1988 : cluster [DBG] pgmap v3236: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:00 smithi121 bash[36902]: cluster 2022-04-22T18:22:59.965551+0000 mon.a (mon.0) 4553 : cluster [DBG] osdmap e1808: 8 total, 8 up, 8 in 2022-04-22T18:23:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:00 smithi115 bash[45495]: cluster 2022-04-22T18:22:59.965551+0000 mon.a (mon.0) 4553 : cluster [DBG] osdmap e1808: 8 total, 8 up, 8 in 2022-04-22T18:23:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:00 smithi115 bash[46934]: cluster 2022-04-22T18:22:59.965551+0000 mon.a (mon.0) 4553 : cluster [DBG] osdmap e1808: 8 total, 8 up, 8 in 2022-04-22T18:23:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:01 smithi121 bash[36902]: cluster 2022-04-22T18:23:00.968053+0000 mon.a (mon.0) 4554 : cluster [DBG] osdmap e1809: 8 total, 8 up, 8 in 2022-04-22T18:23:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:01 smithi121 bash[36902]: audit 2022-04-22T18:23:01.100941+0000 mon.a (mon.0) 4555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:01 smithi121 bash[36902]: cluster 2022-04-22T18:23:01.444271+0000 mgr.y (mgr.34107) 1989 : cluster [DBG] pgmap v3239: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 417 KiB/s rd, 3.5 KiB/s wr, 560 op/s 2022-04-22T18:23:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:01 smithi121 bash[36902]: audit 2022-04-22T18:23:01.600819+0000 mon.a (mon.0) 4556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:02.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:01.816Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[45495]: cluster 2022-04-22T18:23:00.968053+0000 mon.a (mon.0) 4554 : cluster [DBG] osdmap e1809: 8 total, 8 up, 8 in 2022-04-22T18:23:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[45495]: audit 2022-04-22T18:23:01.100941+0000 mon.a (mon.0) 4555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[45495]: cluster 2022-04-22T18:23:01.444271+0000 mgr.y (mgr.34107) 1989 : cluster [DBG] pgmap v3239: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 417 KiB/s rd, 3.5 KiB/s wr, 560 op/s 2022-04-22T18:23:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[45495]: audit 2022-04-22T18:23:01.600819+0000 mon.a (mon.0) 4556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[46934]: cluster 2022-04-22T18:23:00.968053+0000 mon.a (mon.0) 4554 : cluster [DBG] osdmap e1809: 8 total, 8 up, 8 in 2022-04-22T18:23:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[46934]: audit 2022-04-22T18:23:01.100941+0000 mon.a (mon.0) 4555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[46934]: cluster 2022-04-22T18:23:01.444271+0000 mgr.y (mgr.34107) 1989 : cluster [DBG] pgmap v3239: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 417 KiB/s rd, 3.5 KiB/s wr, 560 op/s 2022-04-22T18:23:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:01 smithi115 bash[46934]: audit 2022-04-22T18:23:01.600819+0000 mon.a (mon.0) 4556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:03.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[46934]: cluster 2022-04-22T18:23:01.979349+0000 mon.a (mon.0) 4557 : cluster [DBG] osdmap e1810: 8 total, 8 up, 8 in 2022-04-22T18:23:03.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[46934]: audit 2022-04-22T18:23:02.861730+0000 mon.a (mon.0) 4558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:23:03.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[46934]: audit 2022-04-22T18:23:02.864526+0000 mon.a (mon.0) 4559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:23:03.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[46934]: audit 2022-04-22T18:23:02.866338+0000 mon.a (mon.0) 4560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:23:03.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[45495]: cluster 2022-04-22T18:23:01.979349+0000 mon.a (mon.0) 4557 : cluster [DBG] osdmap e1810: 8 total, 8 up, 8 in 2022-04-22T18:23:03.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[45495]: audit 2022-04-22T18:23:02.861730+0000 mon.a (mon.0) 4558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:23:03.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[45495]: audit 2022-04-22T18:23:02.864526+0000 mon.a (mon.0) 4559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:23:03.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:02 smithi115 bash[45495]: audit 2022-04-22T18:23:02.866338+0000 mon.a (mon.0) 4560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:23:03.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:02 smithi121 bash[36902]: cluster 2022-04-22T18:23:01.979349+0000 mon.a (mon.0) 4557 : cluster [DBG] osdmap e1810: 8 total, 8 up, 8 in 2022-04-22T18:23:03.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:02 smithi121 bash[36902]: audit 2022-04-22T18:23:02.861730+0000 mon.a (mon.0) 4558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:23:03.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:02 smithi121 bash[36902]: audit 2022-04-22T18:23:02.864526+0000 mon.a (mon.0) 4559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:23:03.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:02 smithi121 bash[36902]: audit 2022-04-22T18:23:02.866338+0000 mon.a (mon.0) 4560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:23:04.004 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:03.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:04 smithi121 bash[36902]: cluster 2022-04-22T18:23:03.005350+0000 mon.a (mon.0) 4561 : cluster [DBG] osdmap e1811: 8 total, 8 up, 8 in 2022-04-22T18:23:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:04 smithi121 bash[36902]: audit 2022-04-22T18:23:03.227141+0000 mon.a (mon.0) 4562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:23:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:04 smithi121 bash[36902]: cluster 2022-04-22T18:23:03.445280+0000 mgr.y (mgr.34107) 1990 : cluster [DBG] pgmap v3242: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2022-04-22T18:23:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:04 smithi121 bash[36902]: audit 2022-04-22T18:23:03.480324+0000 mon.a (mon.0) 4563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[45495]: cluster 2022-04-22T18:23:03.005350+0000 mon.a (mon.0) 4561 : cluster [DBG] osdmap e1811: 8 total, 8 up, 8 in 2022-04-22T18:23:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[45495]: audit 2022-04-22T18:23:03.227141+0000 mon.a (mon.0) 4562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:23:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[45495]: cluster 2022-04-22T18:23:03.445280+0000 mgr.y (mgr.34107) 1990 : cluster [DBG] pgmap v3242: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2022-04-22T18:23:04.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[45495]: audit 2022-04-22T18:23:03.480324+0000 mon.a (mon.0) 4563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:04.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[46934]: cluster 2022-04-22T18:23:03.005350+0000 mon.a (mon.0) 4561 : cluster [DBG] osdmap e1811: 8 total, 8 up, 8 in 2022-04-22T18:23:04.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[46934]: audit 2022-04-22T18:23:03.227141+0000 mon.a (mon.0) 4562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:23:04.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[46934]: cluster 2022-04-22T18:23:03.445280+0000 mgr.y (mgr.34107) 1990 : cluster [DBG] pgmap v3242: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 419 KiB/s rd, 3.7 KiB/s wr, 562 op/s 2022-04-22T18:23:04.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:03 smithi115 bash[46934]: audit 2022-04-22T18:23:03.480324+0000 mon.a (mon.0) 4563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:05.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:05 smithi115 bash[46934]: cluster 2022-04-22T18:23:04.012126+0000 mon.a (mon.0) 4564 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2022-04-22T18:23:05.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:05 smithi115 bash[45495]: cluster 2022-04-22T18:23:04.012126+0000 mon.a (mon.0) 4564 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2022-04-22T18:23:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:05 smithi121 bash[36902]: cluster 2022-04-22T18:23:04.012126+0000 mon.a (mon.0) 4564 : cluster [DBG] osdmap e1812: 8 total, 8 up, 8 in 2022-04-22T18:23:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:06 smithi115 bash[45495]: cluster 2022-04-22T18:23:05.021268+0000 mon.a (mon.0) 4565 : cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2022-04-22T18:23:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:06 smithi115 bash[45495]: cluster 2022-04-22T18:23:05.446367+0000 mgr.y (mgr.34107) 1991 : cluster [DBG] pgmap v3245: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.5 KiB/s wr, 607 op/s 2022-04-22T18:23:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:06 smithi115 bash[46934]: cluster 2022-04-22T18:23:05.021268+0000 mon.a (mon.0) 4565 : cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2022-04-22T18:23:06.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:06 smithi115 bash[46934]: cluster 2022-04-22T18:23:05.446367+0000 mgr.y (mgr.34107) 1991 : cluster [DBG] pgmap v3245: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.5 KiB/s wr, 607 op/s 2022-04-22T18:23:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:06 smithi121 bash[36902]: cluster 2022-04-22T18:23:05.021268+0000 mon.a (mon.0) 4565 : cluster [DBG] osdmap e1813: 8 total, 8 up, 8 in 2022-04-22T18:23:06.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:06 smithi121 bash[36902]: cluster 2022-04-22T18:23:05.446367+0000 mgr.y (mgr.34107) 1991 : cluster [DBG] pgmap v3245: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.5 KiB/s wr, 607 op/s 2022-04-22T18:23:07.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[46934]: cluster 2022-04-22T18:23:06.033330+0000 mon.a (mon.0) 4566 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2022-04-22T18:23:07.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[46934]: audit 2022-04-22T18:23:06.104551+0000 mon.a (mon.0) 4567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:07.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[46934]: audit 2022-04-22T18:23:06.604536+0000 mon.a (mon.0) 4568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:07.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[45495]: cluster 2022-04-22T18:23:06.033330+0000 mon.a (mon.0) 4566 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2022-04-22T18:23:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[45495]: audit 2022-04-22T18:23:06.104551+0000 mon.a (mon.0) 4567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:07 smithi115 bash[45495]: audit 2022-04-22T18:23:06.604536+0000 mon.a (mon.0) 4568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:07 smithi121 bash[36902]: cluster 2022-04-22T18:23:06.033330+0000 mon.a (mon.0) 4566 : cluster [DBG] osdmap e1814: 8 total, 8 up, 8 in 2022-04-22T18:23:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:07 smithi121 bash[36902]: audit 2022-04-22T18:23:06.104551+0000 mon.a (mon.0) 4567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:07 smithi121 bash[36902]: audit 2022-04-22T18:23:06.604536+0000 mon.a (mon.0) 4568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:08 smithi121 bash[36902]: cluster 2022-04-22T18:23:07.042288+0000 mon.a (mon.0) 4569 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2022-04-22T18:23:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:08 smithi121 bash[36902]: cluster 2022-04-22T18:23:07.447047+0000 mgr.y (mgr.34107) 1992 : cluster [DBG] pgmap v3248: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.0 KiB/s wr, 607 op/s 2022-04-22T18:23:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:08 smithi115 bash[45495]: cluster 2022-04-22T18:23:07.042288+0000 mon.a (mon.0) 4569 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2022-04-22T18:23:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:08 smithi115 bash[45495]: cluster 2022-04-22T18:23:07.447047+0000 mgr.y (mgr.34107) 1992 : cluster [DBG] pgmap v3248: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.0 KiB/s wr, 607 op/s 2022-04-22T18:23:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:08 smithi115 bash[46934]: cluster 2022-04-22T18:23:07.042288+0000 mon.a (mon.0) 4569 : cluster [DBG] osdmap e1815: 8 total, 8 up, 8 in 2022-04-22T18:23:08.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:08 smithi115 bash[46934]: cluster 2022-04-22T18:23:07.447047+0000 mgr.y (mgr.34107) 1992 : cluster [DBG] pgmap v3248: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 452 KiB/s rd, 3.0 KiB/s wr, 607 op/s 2022-04-22T18:23:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:09 smithi121 bash[36902]: cluster 2022-04-22T18:23:08.051217+0000 mon.a (mon.0) 4570 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2022-04-22T18:23:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:09 smithi121 bash[36902]: audit 2022-04-22T18:23:08.483557+0000 mon.a (mon.0) 4571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:09 smithi115 bash[45495]: cluster 2022-04-22T18:23:08.051217+0000 mon.a (mon.0) 4570 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2022-04-22T18:23:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:09 smithi115 bash[45495]: audit 2022-04-22T18:23:08.483557+0000 mon.a (mon.0) 4571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:09 smithi115 bash[46934]: cluster 2022-04-22T18:23:08.051217+0000 mon.a (mon.0) 4570 : cluster [DBG] osdmap e1816: 8 total, 8 up, 8 in 2022-04-22T18:23:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:09 smithi115 bash[46934]: audit 2022-04-22T18:23:08.483557+0000 mon.a (mon.0) 4571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:10 smithi121 bash[36902]: cluster 2022-04-22T18:23:09.059705+0000 mon.a (mon.0) 4572 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2022-04-22T18:23:10.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:10 smithi121 bash[36902]: cluster 2022-04-22T18:23:09.447855+0000 mgr.y (mgr.34107) 1993 : cluster [DBG] pgmap v3251: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:10 smithi115 bash[45495]: cluster 2022-04-22T18:23:09.059705+0000 mon.a (mon.0) 4572 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2022-04-22T18:23:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:10 smithi115 bash[45495]: cluster 2022-04-22T18:23:09.447855+0000 mgr.y (mgr.34107) 1993 : cluster [DBG] pgmap v3251: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:10 smithi115 bash[46934]: cluster 2022-04-22T18:23:09.059705+0000 mon.a (mon.0) 4572 : cluster [DBG] osdmap e1817: 8 total, 8 up, 8 in 2022-04-22T18:23:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:10 smithi115 bash[46934]: cluster 2022-04-22T18:23:09.447855+0000 mgr.y (mgr.34107) 1993 : cluster [DBG] pgmap v3251: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:11.461 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:23:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:11 smithi121 bash[36902]: cluster 2022-04-22T18:23:10.072298+0000 mon.a (mon.0) 4573 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2022-04-22T18:23:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:11 smithi115 bash[45495]: cluster 2022-04-22T18:23:10.072298+0000 mon.a (mon.0) 4573 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2022-04-22T18:23:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:11 smithi115 bash[46934]: cluster 2022-04-22T18:23:10.072298+0000 mon.a (mon.0) 4573 : cluster [DBG] osdmap e1818: 8 total, 8 up, 8 in 2022-04-22T18:23:11.809 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:23:11.809 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (46m) 4m ago 60m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:23:11.809 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (59m) 5m ago 59m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:23:11.809 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (48m) 4m ago 61m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (47m) 5m ago 61m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (55m) 5m ago 65m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (56m) 4m ago 67m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (55m) 4m ago 67m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (54m) 5m ago 65m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (54m) 4m ago 65m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (47m) 4m ago 60m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (47m) 5m ago 60m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:23:11.810 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (52m) 4m ago 64m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (52m) 4m ago 64m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (51m) 4m ago 63m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (51m) 4m ago 63m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (50m) 5m ago 63m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (50m) 5m ago 62m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (49m) 5m ago 62m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (48m) 5m ago 61m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:23:11.811 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (46m) 5m ago 60m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:23:12.086 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:11.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:23:12.223 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:23:12.224 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:23:12.225 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:23:12.225 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:23:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:12 smithi121 bash[36902]: cluster 2022-04-22T18:23:11.081852+0000 mon.a (mon.0) 4574 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2022-04-22T18:23:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:12 smithi121 bash[36902]: audit 2022-04-22T18:23:11.107722+0000 mon.a (mon.0) 4575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:12 smithi121 bash[36902]: audit 2022-04-22T18:23:11.436562+0000 mgr.y (mgr.34107) 1994 : audit [DBG] from='client.37455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:12 smithi121 bash[36902]: cluster 2022-04-22T18:23:11.448456+0000 mgr.y (mgr.34107) 1995 : cluster [DBG] pgmap v3254: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 384 KiB/s rd, 3.2 KiB/s wr, 518 op/s 2022-04-22T18:23:12.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:12 smithi121 bash[36902]: audit 2022-04-22T18:23:11.608146+0000 mon.a (mon.0) 4576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:12.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[45495]: cluster 2022-04-22T18:23:11.081852+0000 mon.a (mon.0) 4574 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2022-04-22T18:23:12.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[45495]: audit 2022-04-22T18:23:11.107722+0000 mon.a (mon.0) 4575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:12.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[45495]: audit 2022-04-22T18:23:11.436562+0000 mgr.y (mgr.34107) 1994 : audit [DBG] from='client.37455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[45495]: cluster 2022-04-22T18:23:11.448456+0000 mgr.y (mgr.34107) 1995 : cluster [DBG] pgmap v3254: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 384 KiB/s rd, 3.2 KiB/s wr, 518 op/s 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[45495]: audit 2022-04-22T18:23:11.608146+0000 mon.a (mon.0) 4576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[46934]: cluster 2022-04-22T18:23:11.081852+0000 mon.a (mon.0) 4574 : cluster [DBG] osdmap e1819: 8 total, 8 up, 8 in 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[46934]: audit 2022-04-22T18:23:11.107722+0000 mon.a (mon.0) 4575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[46934]: audit 2022-04-22T18:23:11.436562+0000 mgr.y (mgr.34107) 1994 : audit [DBG] from='client.37455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[46934]: cluster 2022-04-22T18:23:11.448456+0000 mgr.y (mgr.34107) 1995 : cluster [DBG] pgmap v3254: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 384 KiB/s rd, 3.2 KiB/s wr, 518 op/s 2022-04-22T18:23:12.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:12 smithi115 bash[46934]: audit 2022-04-22T18:23:11.608146+0000 mon.a (mon.0) 4576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:13 smithi121 bash[36902]: audit 2022-04-22T18:23:11.784758+0000 mgr.y (mgr.34107) 1996 : audit [DBG] from='client.47392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:13 smithi121 bash[36902]: cluster 2022-04-22T18:23:12.091352+0000 mon.a (mon.0) 4577 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2022-04-22T18:23:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:13 smithi121 bash[36902]: audit 2022-04-22T18:23:12.216145+0000 mon.c (mon.1) 123 : audit [DBG] from='client.? 172.21.15.115:0/3782258712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[45495]: audit 2022-04-22T18:23:11.784758+0000 mgr.y (mgr.34107) 1996 : audit [DBG] from='client.47392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[45495]: cluster 2022-04-22T18:23:12.091352+0000 mon.a (mon.0) 4577 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2022-04-22T18:23:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[45495]: audit 2022-04-22T18:23:12.216145+0000 mon.c (mon.1) 123 : audit [DBG] from='client.? 172.21.15.115:0/3782258712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:13.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[46934]: audit 2022-04-22T18:23:11.784758+0000 mgr.y (mgr.34107) 1996 : audit [DBG] from='client.47392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[46934]: cluster 2022-04-22T18:23:12.091352+0000 mon.a (mon.0) 4577 : cluster [DBG] osdmap e1820: 8 total, 8 up, 8 in 2022-04-22T18:23:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[46934]: audit 2022-04-22T18:23:12.216145+0000 mon.c (mon.1) 123 : audit [DBG] from='client.? 172.21.15.115:0/3782258712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:13.697Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:14 smithi121 bash[36902]: cluster 2022-04-22T18:23:13.101146+0000 mon.a (mon.0) 4578 : cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2022-04-22T18:23:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:14 smithi121 bash[36902]: cluster 2022-04-22T18:23:13.449258+0000 mgr.y (mgr.34107) 1997 : cluster [DBG] pgmap v3257: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 2.7 KiB/s wr, 519 op/s 2022-04-22T18:23:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:14 smithi121 bash[36902]: audit 2022-04-22T18:23:13.486721+0000 mon.a (mon.0) 4579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:14 smithi121 bash[36902]: audit 2022-04-22T18:23:13.923182+0000 mon.a (mon.0) 4580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:23:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:14 smithi121 bash[36902]: audit 2022-04-22T18:23:13.924156+0000 mon.a (mon.0) 4581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:23:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[45495]: cluster 2022-04-22T18:23:13.101146+0000 mon.a (mon.0) 4578 : cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2022-04-22T18:23:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[45495]: cluster 2022-04-22T18:23:13.449258+0000 mgr.y (mgr.34107) 1997 : cluster [DBG] pgmap v3257: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 2.7 KiB/s wr, 519 op/s 2022-04-22T18:23:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[45495]: audit 2022-04-22T18:23:13.486721+0000 mon.a (mon.0) 4579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[45495]: audit 2022-04-22T18:23:13.923182+0000 mon.a (mon.0) 4580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[45495]: audit 2022-04-22T18:23:13.924156+0000 mon.a (mon.0) 4581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[46934]: cluster 2022-04-22T18:23:13.101146+0000 mon.a (mon.0) 4578 : cluster [DBG] osdmap e1821: 8 total, 8 up, 8 in 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[46934]: cluster 2022-04-22T18:23:13.449258+0000 mgr.y (mgr.34107) 1997 : cluster [DBG] pgmap v3257: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 386 KiB/s rd, 2.7 KiB/s wr, 519 op/s 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[46934]: audit 2022-04-22T18:23:13.486721+0000 mon.a (mon.0) 4579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[46934]: audit 2022-04-22T18:23:13.923182+0000 mon.a (mon.0) 4580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:23:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:14 smithi115 bash[46934]: audit 2022-04-22T18:23:13.924156+0000 mon.a (mon.0) 4581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:23:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:15 smithi121 bash[36902]: cluster 2022-04-22T18:23:14.110900+0000 mon.a (mon.0) 4582 : cluster [DBG] osdmap e1822: 8 total, 8 up, 8 in 2022-04-22T18:23:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:15 smithi115 bash[45495]: cluster 2022-04-22T18:23:14.110900+0000 mon.a (mon.0) 4582 : cluster [DBG] osdmap e1822: 8 total, 8 up, 8 in 2022-04-22T18:23:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:15 smithi115 bash[46934]: cluster 2022-04-22T18:23:14.110900+0000 mon.a (mon.0) 4582 : cluster [DBG] osdmap e1822: 8 total, 8 up, 8 in 2022-04-22T18:23:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:16 smithi121 bash[36902]: cluster 2022-04-22T18:23:15.126187+0000 mon.a (mon.0) 4583 : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2022-04-22T18:23:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:16 smithi121 bash[36902]: cluster 2022-04-22T18:23:15.450088+0000 mgr.y (mgr.34107) 1998 : cluster [DBG] pgmap v3260: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:16 smithi121 bash[36902]: audit 2022-04-22T18:23:16.110595+0000 mon.a (mon.0) 4584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[45495]: cluster 2022-04-22T18:23:15.126187+0000 mon.a (mon.0) 4583 : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2022-04-22T18:23:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[45495]: cluster 2022-04-22T18:23:15.450088+0000 mgr.y (mgr.34107) 1998 : cluster [DBG] pgmap v3260: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[45495]: audit 2022-04-22T18:23:16.110595+0000 mon.a (mon.0) 4584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[46934]: cluster 2022-04-22T18:23:15.126187+0000 mon.a (mon.0) 4583 : cluster [DBG] osdmap e1823: 8 total, 8 up, 8 in 2022-04-22T18:23:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[46934]: cluster 2022-04-22T18:23:15.450088+0000 mgr.y (mgr.34107) 1998 : cluster [DBG] pgmap v3260: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:16 smithi115 bash[46934]: audit 2022-04-22T18:23:16.110595+0000 mon.a (mon.0) 4584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:17 smithi121 bash[36902]: cluster 2022-04-22T18:23:16.136500+0000 mon.a (mon.0) 4585 : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2022-04-22T18:23:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:17 smithi121 bash[36902]: audit 2022-04-22T18:23:16.611519+0000 mon.a (mon.0) 4586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:17 smithi115 bash[45495]: cluster 2022-04-22T18:23:16.136500+0000 mon.a (mon.0) 4585 : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2022-04-22T18:23:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:17 smithi115 bash[45495]: audit 2022-04-22T18:23:16.611519+0000 mon.a (mon.0) 4586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:17 smithi115 bash[46934]: cluster 2022-04-22T18:23:16.136500+0000 mon.a (mon.0) 4585 : cluster [DBG] osdmap e1824: 8 total, 8 up, 8 in 2022-04-22T18:23:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:17 smithi115 bash[46934]: audit 2022-04-22T18:23:16.611519+0000 mon.a (mon.0) 4586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:18 smithi121 bash[36902]: cluster 2022-04-22T18:23:17.140690+0000 mon.a (mon.0) 4587 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2022-04-22T18:23:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:18 smithi121 bash[36902]: cluster 2022-04-22T18:23:17.450762+0000 mgr.y (mgr.34107) 1999 : cluster [DBG] pgmap v3263: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:18 smithi115 bash[45495]: cluster 2022-04-22T18:23:17.140690+0000 mon.a (mon.0) 4587 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2022-04-22T18:23:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:18 smithi115 bash[45495]: cluster 2022-04-22T18:23:17.450762+0000 mgr.y (mgr.34107) 1999 : cluster [DBG] pgmap v3263: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:18 smithi115 bash[46934]: cluster 2022-04-22T18:23:17.140690+0000 mon.a (mon.0) 4587 : cluster [DBG] osdmap e1825: 8 total, 8 up, 8 in 2022-04-22T18:23:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:18 smithi115 bash[46934]: cluster 2022-04-22T18:23:17.450762+0000 mgr.y (mgr.34107) 1999 : cluster [DBG] pgmap v3263: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 KiB/s rd, 3.5 KiB/s wr, 688 op/s 2022-04-22T18:23:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:19 smithi121 bash[36902]: cluster 2022-04-22T18:23:18.162837+0000 mon.a (mon.0) 4588 : cluster [DBG] osdmap e1826: 8 total, 8 up, 8 in 2022-04-22T18:23:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:19 smithi121 bash[36902]: audit 2022-04-22T18:23:18.490266+0000 mon.a (mon.0) 4589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:19 smithi115 bash[45495]: cluster 2022-04-22T18:23:18.162837+0000 mon.a (mon.0) 4588 : cluster [DBG] osdmap e1826: 8 total, 8 up, 8 in 2022-04-22T18:23:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:19 smithi115 bash[45495]: audit 2022-04-22T18:23:18.490266+0000 mon.a (mon.0) 4589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:19 smithi115 bash[46934]: cluster 2022-04-22T18:23:18.162837+0000 mon.a (mon.0) 4588 : cluster [DBG] osdmap e1826: 8 total, 8 up, 8 in 2022-04-22T18:23:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:19 smithi115 bash[46934]: audit 2022-04-22T18:23:18.490266+0000 mon.a (mon.0) 4589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:20 smithi121 bash[36902]: cluster 2022-04-22T18:23:19.169330+0000 mon.a (mon.0) 4590 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2022-04-22T18:23:20.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:20 smithi121 bash[36902]: cluster 2022-04-22T18:23:19.451575+0000 mgr.y (mgr.34107) 2000 : cluster [DBG] pgmap v3266: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:20 smithi115 bash[45495]: cluster 2022-04-22T18:23:19.169330+0000 mon.a (mon.0) 4590 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2022-04-22T18:23:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:20 smithi115 bash[45495]: cluster 2022-04-22T18:23:19.451575+0000 mgr.y (mgr.34107) 2000 : cluster [DBG] pgmap v3266: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:20.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:20 smithi115 bash[46934]: cluster 2022-04-22T18:23:19.169330+0000 mon.a (mon.0) 4590 : cluster [DBG] osdmap e1827: 8 total, 8 up, 8 in 2022-04-22T18:23:20.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:20 smithi115 bash[46934]: cluster 2022-04-22T18:23:19.451575+0000 mgr.y (mgr.34107) 2000 : cluster [DBG] pgmap v3266: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2022-04-22T18:23:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:21 smithi121 bash[36902]: cluster 2022-04-22T18:23:20.183221+0000 mon.a (mon.0) 4591 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2022-04-22T18:23:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:21 smithi121 bash[36902]: audit 2022-04-22T18:23:21.113879+0000 mon.a (mon.0) 4592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:21 smithi115 bash[45495]: cluster 2022-04-22T18:23:20.183221+0000 mon.a (mon.0) 4591 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2022-04-22T18:23:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:21 smithi115 bash[45495]: audit 2022-04-22T18:23:21.113879+0000 mon.a (mon.0) 4592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:21 smithi115 bash[46934]: cluster 2022-04-22T18:23:20.183221+0000 mon.a (mon.0) 4591 : cluster [DBG] osdmap e1828: 8 total, 8 up, 8 in 2022-04-22T18:23:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:21 smithi115 bash[46934]: audit 2022-04-22T18:23:21.113879+0000 mon.a (mon.0) 4592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:22.199 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:21.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:22 smithi121 bash[36902]: cluster 2022-04-22T18:23:21.196365+0000 mon.a (mon.0) 4593 : cluster [DBG] osdmap e1829: 8 total, 8 up, 8 in 2022-04-22T18:23:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:22 smithi121 bash[36902]: cluster 2022-04-22T18:23:21.452520+0000 mgr.y (mgr.34107) 2001 : cluster [DBG] pgmap v3269: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 542 KiB/s rd, 3.5 KiB/s wr, 730 op/s 2022-04-22T18:23:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:22 smithi121 bash[36902]: audit 2022-04-22T18:23:21.614937+0000 mon.a (mon.0) 4594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:22.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[45495]: cluster 2022-04-22T18:23:21.196365+0000 mon.a (mon.0) 4593 : cluster [DBG] osdmap e1829: 8 total, 8 up, 8 in 2022-04-22T18:23:22.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[45495]: cluster 2022-04-22T18:23:21.452520+0000 mgr.y (mgr.34107) 2001 : cluster [DBG] pgmap v3269: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 542 KiB/s rd, 3.5 KiB/s wr, 730 op/s 2022-04-22T18:23:22.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[45495]: audit 2022-04-22T18:23:21.614937+0000 mon.a (mon.0) 4594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[46934]: cluster 2022-04-22T18:23:21.196365+0000 mon.a (mon.0) 4593 : cluster [DBG] osdmap e1829: 8 total, 8 up, 8 in 2022-04-22T18:23:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[46934]: cluster 2022-04-22T18:23:21.452520+0000 mgr.y (mgr.34107) 2001 : cluster [DBG] pgmap v3269: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 542 KiB/s rd, 3.5 KiB/s wr, 730 op/s 2022-04-22T18:23:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:22 smithi115 bash[46934]: audit 2022-04-22T18:23:21.614937+0000 mon.a (mon.0) 4594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:23 smithi121 bash[36902]: cluster 2022-04-22T18:23:22.203773+0000 mon.a (mon.0) 4595 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2022-04-22T18:23:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:23 smithi115 bash[45495]: cluster 2022-04-22T18:23:22.203773+0000 mon.a (mon.0) 4595 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2022-04-22T18:23:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:23 smithi115 bash[46934]: cluster 2022-04-22T18:23:22.203773+0000 mon.a (mon.0) 4595 : cluster [DBG] osdmap e1830: 8 total, 8 up, 8 in 2022-04-22T18:23:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:23.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:24 smithi121 bash[36902]: cluster 2022-04-22T18:23:23.211040+0000 mon.a (mon.0) 4596 : cluster [DBG] osdmap e1831: 8 total, 8 up, 8 in 2022-04-22T18:23:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:24 smithi121 bash[36902]: cluster 2022-04-22T18:23:23.453427+0000 mgr.y (mgr.34107) 2002 : cluster [DBG] pgmap v3272: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 543 KiB/s rd, 3.2 KiB/s wr, 731 op/s 2022-04-22T18:23:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:24 smithi121 bash[36902]: audit 2022-04-22T18:23:23.493555+0000 mon.a (mon.0) 4597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[45495]: cluster 2022-04-22T18:23:23.211040+0000 mon.a (mon.0) 4596 : cluster [DBG] osdmap e1831: 8 total, 8 up, 8 in 2022-04-22T18:23:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[45495]: cluster 2022-04-22T18:23:23.453427+0000 mgr.y (mgr.34107) 2002 : cluster [DBG] pgmap v3272: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 543 KiB/s rd, 3.2 KiB/s wr, 731 op/s 2022-04-22T18:23:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[45495]: audit 2022-04-22T18:23:23.493555+0000 mon.a (mon.0) 4597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[46934]: cluster 2022-04-22T18:23:23.211040+0000 mon.a (mon.0) 4596 : cluster [DBG] osdmap e1831: 8 total, 8 up, 8 in 2022-04-22T18:23:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[46934]: cluster 2022-04-22T18:23:23.453427+0000 mgr.y (mgr.34107) 2002 : cluster [DBG] pgmap v3272: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 543 KiB/s rd, 3.2 KiB/s wr, 731 op/s 2022-04-22T18:23:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:24 smithi115 bash[46934]: audit 2022-04-22T18:23:23.493555+0000 mon.a (mon.0) 4597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:25 smithi121 bash[36902]: cluster 2022-04-22T18:23:24.222712+0000 mon.a (mon.0) 4598 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2022-04-22T18:23:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:25 smithi115 bash[45495]: cluster 2022-04-22T18:23:24.222712+0000 mon.a (mon.0) 4598 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2022-04-22T18:23:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:25 smithi115 bash[46934]: cluster 2022-04-22T18:23:24.222712+0000 mon.a (mon.0) 4598 : cluster [DBG] osdmap e1832: 8 total, 8 up, 8 in 2022-04-22T18:23:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:26 smithi121 bash[36902]: cluster 2022-04-22T18:23:25.237715+0000 mon.a (mon.0) 4599 : cluster [DBG] osdmap e1833: 8 total, 8 up, 8 in 2022-04-22T18:23:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:26 smithi121 bash[36902]: cluster 2022-04-22T18:23:25.454321+0000 mgr.y (mgr.34107) 2003 : cluster [DBG] pgmap v3275: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 3.0 KiB/s wr, 775 op/s 2022-04-22T18:23:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:26 smithi121 bash[36902]: audit 2022-04-22T18:23:26.116998+0000 mon.a (mon.0) 4600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[46934]: cluster 2022-04-22T18:23:25.237715+0000 mon.a (mon.0) 4599 : cluster [DBG] osdmap e1833: 8 total, 8 up, 8 in 2022-04-22T18:23:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[46934]: cluster 2022-04-22T18:23:25.454321+0000 mgr.y (mgr.34107) 2003 : cluster [DBG] pgmap v3275: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 3.0 KiB/s wr, 775 op/s 2022-04-22T18:23:26.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[46934]: audit 2022-04-22T18:23:26.116998+0000 mon.a (mon.0) 4600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[45495]: cluster 2022-04-22T18:23:25.237715+0000 mon.a (mon.0) 4599 : cluster [DBG] osdmap e1833: 8 total, 8 up, 8 in 2022-04-22T18:23:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[45495]: cluster 2022-04-22T18:23:25.454321+0000 mgr.y (mgr.34107) 2003 : cluster [DBG] pgmap v3275: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 3.0 KiB/s wr, 775 op/s 2022-04-22T18:23:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:26 smithi115 bash[45495]: audit 2022-04-22T18:23:26.116998+0000 mon.a (mon.0) 4600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:27 smithi121 bash[36902]: cluster 2022-04-22T18:23:26.249617+0000 mon.a (mon.0) 4601 : cluster [DBG] osdmap e1834: 8 total, 8 up, 8 in 2022-04-22T18:23:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:27 smithi121 bash[36902]: audit 2022-04-22T18:23:26.618892+0000 mon.a (mon.0) 4602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:27 smithi115 bash[45495]: cluster 2022-04-22T18:23:26.249617+0000 mon.a (mon.0) 4601 : cluster [DBG] osdmap e1834: 8 total, 8 up, 8 in 2022-04-22T18:23:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:27 smithi115 bash[45495]: audit 2022-04-22T18:23:26.618892+0000 mon.a (mon.0) 4602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:27 smithi115 bash[46934]: cluster 2022-04-22T18:23:26.249617+0000 mon.a (mon.0) 4601 : cluster [DBG] osdmap e1834: 8 total, 8 up, 8 in 2022-04-22T18:23:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:27 smithi115 bash[46934]: audit 2022-04-22T18:23:26.618892+0000 mon.a (mon.0) 4602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:28 smithi115 bash[45495]: cluster 2022-04-22T18:23:27.250251+0000 mon.a (mon.0) 4603 : cluster [DBG] osdmap e1835: 8 total, 8 up, 8 in 2022-04-22T18:23:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:28 smithi115 bash[45495]: cluster 2022-04-22T18:23:27.455166+0000 mgr.y (mgr.34107) 2004 : cluster [DBG] pgmap v3278: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 4.0 KiB/s wr, 776 op/s 2022-04-22T18:23:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:28 smithi115 bash[46934]: cluster 2022-04-22T18:23:27.250251+0000 mon.a (mon.0) 4603 : cluster [DBG] osdmap e1835: 8 total, 8 up, 8 in 2022-04-22T18:23:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:28 smithi115 bash[46934]: cluster 2022-04-22T18:23:27.455166+0000 mgr.y (mgr.34107) 2004 : cluster [DBG] pgmap v3278: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 4.0 KiB/s wr, 776 op/s 2022-04-22T18:23:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:28 smithi121 bash[36902]: cluster 2022-04-22T18:23:27.250251+0000 mon.a (mon.0) 4603 : cluster [DBG] osdmap e1835: 8 total, 8 up, 8 in 2022-04-22T18:23:28.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:28 smithi121 bash[36902]: cluster 2022-04-22T18:23:27.455166+0000 mgr.y (mgr.34107) 2004 : cluster [DBG] pgmap v3278: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 575 KiB/s rd, 4.0 KiB/s wr, 776 op/s 2022-04-22T18:23:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:29 smithi115 bash[45495]: cluster 2022-04-22T18:23:28.266891+0000 mon.a (mon.0) 4604 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2022-04-22T18:23:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:29 smithi115 bash[45495]: audit 2022-04-22T18:23:28.496828+0000 mon.a (mon.0) 4605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:29 smithi115 bash[46934]: cluster 2022-04-22T18:23:28.266891+0000 mon.a (mon.0) 4604 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2022-04-22T18:23:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:29 smithi115 bash[46934]: audit 2022-04-22T18:23:28.496828+0000 mon.a (mon.0) 4605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:29 smithi121 bash[36902]: cluster 2022-04-22T18:23:28.266891+0000 mon.a (mon.0) 4604 : cluster [DBG] osdmap e1836: 8 total, 8 up, 8 in 2022-04-22T18:23:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:29 smithi121 bash[36902]: audit 2022-04-22T18:23:28.496828+0000 mon.a (mon.0) 4605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:30 smithi115 bash[45495]: cluster 2022-04-22T18:23:29.274275+0000 mon.a (mon.0) 4606 : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2022-04-22T18:23:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:30 smithi115 bash[45495]: cluster 2022-04-22T18:23:29.456023+0000 mgr.y (mgr.34107) 2005 : cluster [DBG] pgmap v3281: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:23:30.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:30 smithi115 bash[46934]: cluster 2022-04-22T18:23:29.274275+0000 mon.a (mon.0) 4606 : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2022-04-22T18:23:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:30 smithi115 bash[46934]: cluster 2022-04-22T18:23:29.456023+0000 mgr.y (mgr.34107) 2005 : cluster [DBG] pgmap v3281: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:23:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:30 smithi121 bash[36902]: cluster 2022-04-22T18:23:29.274275+0000 mon.a (mon.0) 4606 : cluster [DBG] osdmap e1837: 8 total, 8 up, 8 in 2022-04-22T18:23:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:30 smithi121 bash[36902]: cluster 2022-04-22T18:23:29.456023+0000 mgr.y (mgr.34107) 2005 : cluster [DBG] pgmap v3281: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:23:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:31 smithi115 bash[45495]: cluster 2022-04-22T18:23:30.285889+0000 mon.a (mon.0) 4607 : cluster [DBG] osdmap e1838: 8 total, 8 up, 8 in 2022-04-22T18:23:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:31 smithi115 bash[45495]: audit 2022-04-22T18:23:31.120404+0000 mon.a (mon.0) 4608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:31 smithi115 bash[46934]: cluster 2022-04-22T18:23:30.285889+0000 mon.a (mon.0) 4607 : cluster [DBG] osdmap e1838: 8 total, 8 up, 8 in 2022-04-22T18:23:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:31 smithi115 bash[46934]: audit 2022-04-22T18:23:31.120404+0000 mon.a (mon.0) 4608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:31 smithi121 bash[36902]: cluster 2022-04-22T18:23:30.285889+0000 mon.a (mon.0) 4607 : cluster [DBG] osdmap e1838: 8 total, 8 up, 8 in 2022-04-22T18:23:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:31 smithi121 bash[36902]: audit 2022-04-22T18:23:31.120404+0000 mon.a (mon.0) 4608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:32.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:31.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:32 smithi121 bash[36902]: cluster 2022-04-22T18:23:31.302414+0000 mon.a (mon.0) 4609 : cluster [DBG] osdmap e1839: 8 total, 8 up, 8 in 2022-04-22T18:23:32.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:32 smithi121 bash[36902]: cluster 2022-04-22T18:23:31.456812+0000 mgr.y (mgr.34107) 2006 : cluster [DBG] pgmap v3284: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 607 KiB/s rd, 3.2 KiB/s wr, 817 op/s 2022-04-22T18:23:32.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:32 smithi121 bash[36902]: audit 2022-04-22T18:23:31.622646+0000 mon.a (mon.0) 4610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[45495]: cluster 2022-04-22T18:23:31.302414+0000 mon.a (mon.0) 4609 : cluster [DBG] osdmap e1839: 8 total, 8 up, 8 in 2022-04-22T18:23:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[45495]: cluster 2022-04-22T18:23:31.456812+0000 mgr.y (mgr.34107) 2006 : cluster [DBG] pgmap v3284: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 607 KiB/s rd, 3.2 KiB/s wr, 817 op/s 2022-04-22T18:23:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[45495]: audit 2022-04-22T18:23:31.622646+0000 mon.a (mon.0) 4610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[46934]: cluster 2022-04-22T18:23:31.302414+0000 mon.a (mon.0) 4609 : cluster [DBG] osdmap e1839: 8 total, 8 up, 8 in 2022-04-22T18:23:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[46934]: cluster 2022-04-22T18:23:31.456812+0000 mgr.y (mgr.34107) 2006 : cluster [DBG] pgmap v3284: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 607 KiB/s rd, 3.2 KiB/s wr, 817 op/s 2022-04-22T18:23:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:32 smithi115 bash[46934]: audit 2022-04-22T18:23:31.622646+0000 mon.a (mon.0) 4610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:33.705 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:33 smithi115 bash[45495]: cluster 2022-04-22T18:23:32.310363+0000 mon.a (mon.0) 4611 : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2022-04-22T18:23:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:33 smithi115 bash[46934]: cluster 2022-04-22T18:23:32.310363+0000 mon.a (mon.0) 4611 : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2022-04-22T18:23:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:33 smithi121 bash[36902]: cluster 2022-04-22T18:23:32.310363+0000 mon.a (mon.0) 4611 : cluster [DBG] osdmap e1840: 8 total, 8 up, 8 in 2022-04-22T18:23:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:33.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:34 smithi121 bash[36902]: cluster 2022-04-22T18:23:33.319093+0000 mon.a (mon.0) 4612 : cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2022-04-22T18:23:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:34 smithi121 bash[36902]: cluster 2022-04-22T18:23:33.457803+0000 mgr.y (mgr.34107) 2007 : cluster [DBG] pgmap v3287: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 608 KiB/s rd, 3.0 KiB/s wr, 818 op/s 2022-04-22T18:23:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:34 smithi121 bash[36902]: audit 2022-04-22T18:23:33.500162+0000 mon.a (mon.0) 4613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[46934]: cluster 2022-04-22T18:23:33.319093+0000 mon.a (mon.0) 4612 : cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2022-04-22T18:23:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[46934]: cluster 2022-04-22T18:23:33.457803+0000 mgr.y (mgr.34107) 2007 : cluster [DBG] pgmap v3287: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 608 KiB/s rd, 3.0 KiB/s wr, 818 op/s 2022-04-22T18:23:34.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[46934]: audit 2022-04-22T18:23:33.500162+0000 mon.a (mon.0) 4613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:34.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[45495]: cluster 2022-04-22T18:23:33.319093+0000 mon.a (mon.0) 4612 : cluster [DBG] osdmap e1841: 8 total, 8 up, 8 in 2022-04-22T18:23:34.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[45495]: cluster 2022-04-22T18:23:33.457803+0000 mgr.y (mgr.34107) 2007 : cluster [DBG] pgmap v3287: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 608 KiB/s rd, 3.0 KiB/s wr, 818 op/s 2022-04-22T18:23:34.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:34 smithi115 bash[45495]: audit 2022-04-22T18:23:33.500162+0000 mon.a (mon.0) 4613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:35 smithi121 bash[36902]: cluster 2022-04-22T18:23:34.329473+0000 mon.a (mon.0) 4614 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2022-04-22T18:23:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:35 smithi115 bash[45495]: cluster 2022-04-22T18:23:34.329473+0000 mon.a (mon.0) 4614 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2022-04-22T18:23:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:35 smithi115 bash[46934]: cluster 2022-04-22T18:23:34.329473+0000 mon.a (mon.0) 4614 : cluster [DBG] osdmap e1842: 8 total, 8 up, 8 in 2022-04-22T18:23:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:36 smithi121 bash[36902]: cluster 2022-04-22T18:23:35.340861+0000 mon.a (mon.0) 4615 : cluster [DBG] osdmap e1843: 8 total, 8 up, 8 in 2022-04-22T18:23:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:36 smithi121 bash[36902]: cluster 2022-04-22T18:23:35.458716+0000 mgr.y (mgr.34107) 2008 : cluster [DBG] pgmap v3290: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.5 KiB/s wr, 863 op/s 2022-04-22T18:23:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:36 smithi121 bash[36902]: audit 2022-04-22T18:23:36.123389+0000 mon.a (mon.0) 4616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[45495]: cluster 2022-04-22T18:23:35.340861+0000 mon.a (mon.0) 4615 : cluster [DBG] osdmap e1843: 8 total, 8 up, 8 in 2022-04-22T18:23:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[45495]: cluster 2022-04-22T18:23:35.458716+0000 mgr.y (mgr.34107) 2008 : cluster [DBG] pgmap v3290: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.5 KiB/s wr, 863 op/s 2022-04-22T18:23:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[45495]: audit 2022-04-22T18:23:36.123389+0000 mon.a (mon.0) 4616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[46934]: cluster 2022-04-22T18:23:35.340861+0000 mon.a (mon.0) 4615 : cluster [DBG] osdmap e1843: 8 total, 8 up, 8 in 2022-04-22T18:23:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[46934]: cluster 2022-04-22T18:23:35.458716+0000 mgr.y (mgr.34107) 2008 : cluster [DBG] pgmap v3290: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.5 KiB/s wr, 863 op/s 2022-04-22T18:23:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:36 smithi115 bash[46934]: audit 2022-04-22T18:23:36.123389+0000 mon.a (mon.0) 4616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:37 smithi121 bash[36902]: cluster 2022-04-22T18:23:36.347772+0000 mon.a (mon.0) 4617 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2022-04-22T18:23:37.846 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:37 smithi121 bash[36902]: audit 2022-04-22T18:23:36.626586+0000 mon.a (mon.0) 4618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:37.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:37 smithi115 bash[45495]: cluster 2022-04-22T18:23:36.347772+0000 mon.a (mon.0) 4617 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2022-04-22T18:23:37.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:37 smithi115 bash[45495]: audit 2022-04-22T18:23:36.626586+0000 mon.a (mon.0) 4618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:37.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:37 smithi115 bash[46934]: cluster 2022-04-22T18:23:36.347772+0000 mon.a (mon.0) 4617 : cluster [DBG] osdmap e1844: 8 total, 8 up, 8 in 2022-04-22T18:23:37.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:37 smithi115 bash[46934]: audit 2022-04-22T18:23:36.626586+0000 mon.a (mon.0) 4618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:38 smithi121 bash[36902]: cluster 2022-04-22T18:23:37.355072+0000 mon.a (mon.0) 4619 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2022-04-22T18:23:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:38 smithi121 bash[36902]: cluster 2022-04-22T18:23:37.459619+0000 mgr.y (mgr.34107) 2009 : cluster [DBG] pgmap v3293: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2022-04-22T18:23:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:38 smithi115 bash[45495]: cluster 2022-04-22T18:23:37.355072+0000 mon.a (mon.0) 4619 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2022-04-22T18:23:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:38 smithi115 bash[45495]: cluster 2022-04-22T18:23:37.459619+0000 mgr.y (mgr.34107) 2009 : cluster [DBG] pgmap v3293: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2022-04-22T18:23:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:38 smithi115 bash[46934]: cluster 2022-04-22T18:23:37.355072+0000 mon.a (mon.0) 4619 : cluster [DBG] osdmap e1845: 8 total, 8 up, 8 in 2022-04-22T18:23:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:38 smithi115 bash[46934]: cluster 2022-04-22T18:23:37.459619+0000 mgr.y (mgr.34107) 2009 : cluster [DBG] pgmap v3293: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 640 KiB/s rd, 3.7 KiB/s wr, 863 op/s 2022-04-22T18:23:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:39 smithi121 bash[36902]: cluster 2022-04-22T18:23:38.370689+0000 mon.a (mon.0) 4620 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2022-04-22T18:23:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:39 smithi121 bash[36902]: audit 2022-04-22T18:23:38.503440+0000 mon.a (mon.0) 4621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:39 smithi115 bash[45495]: cluster 2022-04-22T18:23:38.370689+0000 mon.a (mon.0) 4620 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2022-04-22T18:23:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:39 smithi115 bash[45495]: audit 2022-04-22T18:23:38.503440+0000 mon.a (mon.0) 4621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:39 smithi115 bash[46934]: cluster 2022-04-22T18:23:38.370689+0000 mon.a (mon.0) 4620 : cluster [DBG] osdmap e1846: 8 total, 8 up, 8 in 2022-04-22T18:23:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:39 smithi115 bash[46934]: audit 2022-04-22T18:23:38.503440+0000 mon.a (mon.0) 4621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:40 smithi121 bash[36902]: cluster 2022-04-22T18:23:39.379726+0000 mon.a (mon.0) 4622 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2022-04-22T18:23:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:40 smithi121 bash[36902]: cluster 2022-04-22T18:23:39.460430+0000 mgr.y (mgr.34107) 2010 : cluster [DBG] pgmap v3296: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:40 smithi115 bash[45495]: cluster 2022-04-22T18:23:39.379726+0000 mon.a (mon.0) 4622 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2022-04-22T18:23:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:40 smithi115 bash[45495]: cluster 2022-04-22T18:23:39.460430+0000 mgr.y (mgr.34107) 2010 : cluster [DBG] pgmap v3296: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:40 smithi115 bash[46934]: cluster 2022-04-22T18:23:39.379726+0000 mon.a (mon.0) 4622 : cluster [DBG] osdmap e1847: 8 total, 8 up, 8 in 2022-04-22T18:23:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:40 smithi115 bash[46934]: cluster 2022-04-22T18:23:39.460430+0000 mgr.y (mgr.34107) 2010 : cluster [DBG] pgmap v3296: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:23:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:41 smithi121 bash[36902]: cluster 2022-04-22T18:23:40.389583+0000 mon.a (mon.0) 4623 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2022-04-22T18:23:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:41 smithi121 bash[36902]: audit 2022-04-22T18:23:41.125907+0000 mon.a (mon.0) 4624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:41.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:41 smithi115 bash[45495]: cluster 2022-04-22T18:23:40.389583+0000 mon.a (mon.0) 4623 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2022-04-22T18:23:41.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:41 smithi115 bash[45495]: audit 2022-04-22T18:23:41.125907+0000 mon.a (mon.0) 4624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:41 smithi115 bash[46934]: cluster 2022-04-22T18:23:40.389583+0000 mon.a (mon.0) 4623 : cluster [DBG] osdmap e1848: 8 total, 8 up, 8 in 2022-04-22T18:23:41.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:41 smithi115 bash[46934]: audit 2022-04-22T18:23:41.125907+0000 mon.a (mon.0) 4624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:42.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:41.819Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:42.605 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:23:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:42 smithi121 bash[36902]: cluster 2022-04-22T18:23:41.390486+0000 mon.a (mon.0) 4625 : cluster [DBG] osdmap e1849: 8 total, 8 up, 8 in 2022-04-22T18:23:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:42 smithi121 bash[36902]: cluster 2022-04-22T18:23:41.461306+0000 mgr.y (mgr.34107) 2011 : cluster [DBG] pgmap v3299: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 672 KiB/s rd, 3.5 KiB/s wr, 905 op/s 2022-04-22T18:23:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:42 smithi121 bash[36902]: audit 2022-04-22T18:23:41.630368+0000 mon.a (mon.0) 4626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[46934]: cluster 2022-04-22T18:23:41.390486+0000 mon.a (mon.0) 4625 : cluster [DBG] osdmap e1849: 8 total, 8 up, 8 in 2022-04-22T18:23:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[46934]: cluster 2022-04-22T18:23:41.461306+0000 mgr.y (mgr.34107) 2011 : cluster [DBG] pgmap v3299: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 672 KiB/s rd, 3.5 KiB/s wr, 905 op/s 2022-04-22T18:23:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[46934]: audit 2022-04-22T18:23:41.630368+0000 mon.a (mon.0) 4626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[45495]: cluster 2022-04-22T18:23:41.390486+0000 mon.a (mon.0) 4625 : cluster [DBG] osdmap e1849: 8 total, 8 up, 8 in 2022-04-22T18:23:42.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[45495]: cluster 2022-04-22T18:23:41.461306+0000 mgr.y (mgr.34107) 2011 : cluster [DBG] pgmap v3299: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 672 KiB/s rd, 3.5 KiB/s wr, 905 op/s 2022-04-22T18:23:42.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:42 smithi115 bash[45495]: audit 2022-04-22T18:23:41.630368+0000 mon.a (mon.0) 4626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:42.950 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:23:42.950 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (46m) 4m ago 60m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:23:42.950 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (60m) 5m ago 60m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (48m) 4m ago 61m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (48m) 5m ago 61m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (56m) 5m ago 65m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (56m) 4m ago 68m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (55m) 4m ago 68m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (54m) 5m ago 66m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (55m) 4m ago 66m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (47m) 4m ago 60m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (47m) 5m ago 60m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:23:42.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (53m) 4m ago 65m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (52m) 4m ago 64m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (52m) 4m ago 64m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (51m) 4m ago 63m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (51m) 5m ago 63m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (50m) 5m ago 63m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (49m) 5m ago 62m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (49m) 5m ago 62m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:23:42.952 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (47m) 5m ago 60m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:23:43.364 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:23:43.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:23:43.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:23:43.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:23:43.366 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:23:43.367 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:23:43.706 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[45495]: cluster 2022-04-22T18:23:42.440612+0000 mon.a (mon.0) 4627 : cluster [DBG] osdmap e1850: 8 total, 8 up, 8 in 2022-04-22T18:23:43.707 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[45495]: audit 2022-04-22T18:23:42.581941+0000 mgr.y (mgr.34107) 2012 : audit [DBG] from='client.47404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:43.707 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[45495]: audit 2022-04-22T18:23:43.356741+0000 mon.c (mon.1) 124 : audit [DBG] from='client.? 172.21.15.115:0/618676416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:43.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[46934]: cluster 2022-04-22T18:23:42.440612+0000 mon.a (mon.0) 4627 : cluster [DBG] osdmap e1850: 8 total, 8 up, 8 in 2022-04-22T18:23:43.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[46934]: audit 2022-04-22T18:23:42.581941+0000 mgr.y (mgr.34107) 2012 : audit [DBG] from='client.47404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:43.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[46934]: audit 2022-04-22T18:23:43.356741+0000 mon.c (mon.1) 124 : audit [DBG] from='client.? 172.21.15.115:0/618676416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:43 smithi121 bash[36902]: cluster 2022-04-22T18:23:42.440612+0000 mon.a (mon.0) 4627 : cluster [DBG] osdmap e1850: 8 total, 8 up, 8 in 2022-04-22T18:23:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:43 smithi121 bash[36902]: audit 2022-04-22T18:23:42.581941+0000 mgr.y (mgr.34107) 2012 : audit [DBG] from='client.47404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:43 smithi121 bash[36902]: audit 2022-04-22T18:23:43.356741+0000 mon.c (mon.1) 124 : audit [DBG] from='client.? 172.21.15.115:0/618676416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:23:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:43.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:44 smithi121 bash[36902]: audit 2022-04-22T18:23:42.922149+0000 mgr.y (mgr.34107) 2013 : audit [DBG] from='client.37479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:44 smithi121 bash[36902]: cluster 2022-04-22T18:23:43.454831+0000 mon.a (mon.0) 4628 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2022-04-22T18:23:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:44 smithi121 bash[36902]: cluster 2022-04-22T18:23:43.461861+0000 mgr.y (mgr.34107) 2014 : cluster [DBG] pgmap v3302: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 674 KiB/s rd, 3.2 KiB/s wr, 906 op/s 2022-04-22T18:23:44.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:44 smithi121 bash[36902]: audit 2022-04-22T18:23:43.506339+0000 mon.a (mon.0) 4629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[45495]: audit 2022-04-22T18:23:42.922149+0000 mgr.y (mgr.34107) 2013 : audit [DBG] from='client.37479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[45495]: cluster 2022-04-22T18:23:43.454831+0000 mon.a (mon.0) 4628 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2022-04-22T18:23:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[45495]: cluster 2022-04-22T18:23:43.461861+0000 mgr.y (mgr.34107) 2014 : cluster [DBG] pgmap v3302: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 674 KiB/s rd, 3.2 KiB/s wr, 906 op/s 2022-04-22T18:23:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[45495]: audit 2022-04-22T18:23:43.506339+0000 mon.a (mon.0) 4629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:44.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[46934]: audit 2022-04-22T18:23:42.922149+0000 mgr.y (mgr.34107) 2013 : audit [DBG] from='client.37479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:23:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[46934]: cluster 2022-04-22T18:23:43.454831+0000 mon.a (mon.0) 4628 : cluster [DBG] osdmap e1851: 8 total, 8 up, 8 in 2022-04-22T18:23:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[46934]: cluster 2022-04-22T18:23:43.461861+0000 mgr.y (mgr.34107) 2014 : cluster [DBG] pgmap v3302: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 674 KiB/s rd, 3.2 KiB/s wr, 906 op/s 2022-04-22T18:23:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:44 smithi115 bash[46934]: audit 2022-04-22T18:23:43.506339+0000 mon.a (mon.0) 4629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:45 smithi121 bash[36902]: cluster 2022-04-22T18:23:44.466735+0000 mon.a (mon.0) 4630 : cluster [DBG] osdmap e1852: 8 total, 8 up, 8 in 2022-04-22T18:23:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:45 smithi115 bash[45495]: cluster 2022-04-22T18:23:44.466735+0000 mon.a (mon.0) 4630 : cluster [DBG] osdmap e1852: 8 total, 8 up, 8 in 2022-04-22T18:23:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:45 smithi115 bash[46934]: cluster 2022-04-22T18:23:44.466735+0000 mon.a (mon.0) 4630 : cluster [DBG] osdmap e1852: 8 total, 8 up, 8 in 2022-04-22T18:23:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:46 smithi121 bash[36902]: cluster 2022-04-22T18:23:45.462827+0000 mgr.y (mgr.34107) 2015 : cluster [DBG] pgmap v3304: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.2 KiB/s wr, 932 op/s 2022-04-22T18:23:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:46 smithi121 bash[36902]: cluster 2022-04-22T18:23:45.473428+0000 mon.a (mon.0) 4631 : cluster [DBG] osdmap e1853: 8 total, 8 up, 8 in 2022-04-22T18:23:46.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:46 smithi121 bash[36902]: audit 2022-04-22T18:23:46.128884+0000 mon.a (mon.0) 4632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[45495]: cluster 2022-04-22T18:23:45.462827+0000 mgr.y (mgr.34107) 2015 : cluster [DBG] pgmap v3304: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.2 KiB/s wr, 932 op/s 2022-04-22T18:23:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[45495]: cluster 2022-04-22T18:23:45.473428+0000 mon.a (mon.0) 4631 : cluster [DBG] osdmap e1853: 8 total, 8 up, 8 in 2022-04-22T18:23:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[45495]: audit 2022-04-22T18:23:46.128884+0000 mon.a (mon.0) 4632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[46934]: cluster 2022-04-22T18:23:45.462827+0000 mgr.y (mgr.34107) 2015 : cluster [DBG] pgmap v3304: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 693 KiB/s rd, 3.2 KiB/s wr, 932 op/s 2022-04-22T18:23:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[46934]: cluster 2022-04-22T18:23:45.473428+0000 mon.a (mon.0) 4631 : cluster [DBG] osdmap e1853: 8 total, 8 up, 8 in 2022-04-22T18:23:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:46 smithi115 bash[46934]: audit 2022-04-22T18:23:46.128884+0000 mon.a (mon.0) 4632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:47 smithi121 bash[36902]: cluster 2022-04-22T18:23:46.474336+0000 mon.a (mon.0) 4633 : cluster [DBG] osdmap e1854: 8 total, 8 up, 8 in 2022-04-22T18:23:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:47 smithi121 bash[36902]: audit 2022-04-22T18:23:46.634307+0000 mon.a (mon.0) 4634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:47 smithi115 bash[45495]: cluster 2022-04-22T18:23:46.474336+0000 mon.a (mon.0) 4633 : cluster [DBG] osdmap e1854: 8 total, 8 up, 8 in 2022-04-22T18:23:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:47 smithi115 bash[45495]: audit 2022-04-22T18:23:46.634307+0000 mon.a (mon.0) 4634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:47 smithi115 bash[46934]: cluster 2022-04-22T18:23:46.474336+0000 mon.a (mon.0) 4633 : cluster [DBG] osdmap e1854: 8 total, 8 up, 8 in 2022-04-22T18:23:47.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:47 smithi115 bash[46934]: audit 2022-04-22T18:23:46.634307+0000 mon.a (mon.0) 4634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:48 smithi121 bash[36902]: cluster 2022-04-22T18:23:47.463967+0000 mgr.y (mgr.34107) 2016 : cluster [DBG] pgmap v3307: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 704 KiB/s rd, 3.5 KiB/s wr, 948 op/s 2022-04-22T18:23:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:48 smithi121 bash[36902]: cluster 2022-04-22T18:23:47.497009+0000 mon.a (mon.0) 4635 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2022-04-22T18:23:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:48 smithi115 bash[45495]: cluster 2022-04-22T18:23:47.463967+0000 mgr.y (mgr.34107) 2016 : cluster [DBG] pgmap v3307: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 704 KiB/s rd, 3.5 KiB/s wr, 948 op/s 2022-04-22T18:23:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:48 smithi115 bash[45495]: cluster 2022-04-22T18:23:47.497009+0000 mon.a (mon.0) 4635 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2022-04-22T18:23:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:48 smithi115 bash[46934]: cluster 2022-04-22T18:23:47.463967+0000 mgr.y (mgr.34107) 2016 : cluster [DBG] pgmap v3307: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 704 KiB/s rd, 3.5 KiB/s wr, 948 op/s 2022-04-22T18:23:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:48 smithi115 bash[46934]: cluster 2022-04-22T18:23:47.497009+0000 mon.a (mon.0) 4635 : cluster [DBG] osdmap e1855: 8 total, 8 up, 8 in 2022-04-22T18:23:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:49 smithi115 bash[45495]: cluster 2022-04-22T18:23:48.509716+0000 mon.a (mon.0) 4636 : cluster [DBG] osdmap e1856: 8 total, 8 up, 8 in 2022-04-22T18:23:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:49 smithi115 bash[45495]: audit 2022-04-22T18:23:48.510703+0000 mon.a (mon.0) 4637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:49 smithi115 bash[46934]: cluster 2022-04-22T18:23:48.509716+0000 mon.a (mon.0) 4636 : cluster [DBG] osdmap e1856: 8 total, 8 up, 8 in 2022-04-22T18:23:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:49 smithi115 bash[46934]: audit 2022-04-22T18:23:48.510703+0000 mon.a (mon.0) 4637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:49 smithi121 bash[36902]: cluster 2022-04-22T18:23:48.509716+0000 mon.a (mon.0) 4636 : cluster [DBG] osdmap e1856: 8 total, 8 up, 8 in 2022-04-22T18:23:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:49 smithi121 bash[36902]: audit 2022-04-22T18:23:48.510703+0000 mon.a (mon.0) 4637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:50 smithi115 bash[45495]: cluster 2022-04-22T18:23:49.464939+0000 mgr.y (mgr.34107) 2017 : cluster [DBG] pgmap v3310: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:23:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:50 smithi115 bash[45495]: cluster 2022-04-22T18:23:49.517841+0000 mon.a (mon.0) 4638 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2022-04-22T18:23:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:50 smithi115 bash[46934]: cluster 2022-04-22T18:23:49.464939+0000 mgr.y (mgr.34107) 2017 : cluster [DBG] pgmap v3310: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:23:50.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:50 smithi115 bash[46934]: cluster 2022-04-22T18:23:49.517841+0000 mon.a (mon.0) 4638 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2022-04-22T18:23:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:50 smithi121 bash[36902]: cluster 2022-04-22T18:23:49.464939+0000 mgr.y (mgr.34107) 2017 : cluster [DBG] pgmap v3310: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2022-04-22T18:23:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:50 smithi121 bash[36902]: cluster 2022-04-22T18:23:49.517841+0000 mon.a (mon.0) 4638 : cluster [DBG] osdmap e1857: 8 total, 8 up, 8 in 2022-04-22T18:23:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:51 smithi115 bash[46934]: cluster 2022-04-22T18:23:50.521091+0000 mon.a (mon.0) 4639 : cluster [DBG] osdmap e1858: 8 total, 8 up, 8 in 2022-04-22T18:23:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:51 smithi115 bash[46934]: audit 2022-04-22T18:23:51.131831+0000 mon.a (mon.0) 4640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:51 smithi115 bash[45495]: cluster 2022-04-22T18:23:50.521091+0000 mon.a (mon.0) 4639 : cluster [DBG] osdmap e1858: 8 total, 8 up, 8 in 2022-04-22T18:23:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:51 smithi115 bash[45495]: audit 2022-04-22T18:23:51.131831+0000 mon.a (mon.0) 4640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:51 smithi121 bash[36902]: cluster 2022-04-22T18:23:50.521091+0000 mon.a (mon.0) 4639 : cluster [DBG] osdmap e1858: 8 total, 8 up, 8 in 2022-04-22T18:23:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:51 smithi121 bash[36902]: audit 2022-04-22T18:23:51.131831+0000 mon.a (mon.0) 4640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:51.819Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:52 smithi121 bash[36902]: cluster 2022-04-22T18:23:51.465942+0000 mgr.y (mgr.34107) 2018 : cluster [DBG] pgmap v3313: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 738 KiB/s rd, 3.5 KiB/s wr, 992 op/s 2022-04-22T18:23:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:52 smithi121 bash[36902]: cluster 2022-04-22T18:23:51.543567+0000 mon.a (mon.0) 4641 : cluster [DBG] osdmap e1859: 8 total, 8 up, 8 in 2022-04-22T18:23:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:52 smithi121 bash[36902]: audit 2022-04-22T18:23:51.637883+0000 mon.a (mon.0) 4642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[45495]: cluster 2022-04-22T18:23:51.465942+0000 mgr.y (mgr.34107) 2018 : cluster [DBG] pgmap v3313: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 738 KiB/s rd, 3.5 KiB/s wr, 992 op/s 2022-04-22T18:23:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[45495]: cluster 2022-04-22T18:23:51.543567+0000 mon.a (mon.0) 4641 : cluster [DBG] osdmap e1859: 8 total, 8 up, 8 in 2022-04-22T18:23:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[45495]: audit 2022-04-22T18:23:51.637883+0000 mon.a (mon.0) 4642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[46934]: cluster 2022-04-22T18:23:51.465942+0000 mgr.y (mgr.34107) 2018 : cluster [DBG] pgmap v3313: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 738 KiB/s rd, 3.5 KiB/s wr, 992 op/s 2022-04-22T18:23:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[46934]: cluster 2022-04-22T18:23:51.543567+0000 mon.a (mon.0) 4641 : cluster [DBG] osdmap e1859: 8 total, 8 up, 8 in 2022-04-22T18:23:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:52 smithi115 bash[46934]: audit 2022-04-22T18:23:51.637883+0000 mon.a (mon.0) 4642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:53 smithi121 bash[36902]: cluster 2022-04-22T18:23:52.553230+0000 mon.a (mon.0) 4643 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2022-04-22T18:23:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:53 smithi121 bash[36902]: audit 2022-04-22T18:23:53.514331+0000 mon.a (mon.0) 4644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:53 smithi115 bash[45495]: cluster 2022-04-22T18:23:52.553230+0000 mon.a (mon.0) 4643 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2022-04-22T18:23:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:53 smithi115 bash[45495]: audit 2022-04-22T18:23:53.514331+0000 mon.a (mon.0) 4644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:23:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:23:53.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:23:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:53 smithi115 bash[46934]: cluster 2022-04-22T18:23:52.553230+0000 mon.a (mon.0) 4643 : cluster [DBG] osdmap e1860: 8 total, 8 up, 8 in 2022-04-22T18:23:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:53 smithi115 bash[46934]: audit 2022-04-22T18:23:53.514331+0000 mon.a (mon.0) 4644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:54 smithi121 bash[36902]: cluster 2022-04-22T18:23:53.467076+0000 mgr.y (mgr.34107) 2019 : cluster [DBG] pgmap v3316: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 739 KiB/s rd, 3.0 KiB/s wr, 993 op/s 2022-04-22T18:23:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:54 smithi121 bash[36902]: cluster 2022-04-22T18:23:53.562851+0000 mon.a (mon.0) 4645 : cluster [DBG] osdmap e1861: 8 total, 8 up, 8 in 2022-04-22T18:23:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:54 smithi115 bash[45495]: cluster 2022-04-22T18:23:53.467076+0000 mgr.y (mgr.34107) 2019 : cluster [DBG] pgmap v3316: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 739 KiB/s rd, 3.0 KiB/s wr, 993 op/s 2022-04-22T18:23:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:54 smithi115 bash[45495]: cluster 2022-04-22T18:23:53.562851+0000 mon.a (mon.0) 4645 : cluster [DBG] osdmap e1861: 8 total, 8 up, 8 in 2022-04-22T18:23:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:54 smithi115 bash[46934]: cluster 2022-04-22T18:23:53.467076+0000 mgr.y (mgr.34107) 2019 : cluster [DBG] pgmap v3316: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 739 KiB/s rd, 3.0 KiB/s wr, 993 op/s 2022-04-22T18:23:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:54 smithi115 bash[46934]: cluster 2022-04-22T18:23:53.562851+0000 mon.a (mon.0) 4645 : cluster [DBG] osdmap e1861: 8 total, 8 up, 8 in 2022-04-22T18:23:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:55 smithi121 bash[36902]: cluster 2022-04-22T18:23:54.574112+0000 mon.a (mon.0) 4646 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2022-04-22T18:23:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:55 smithi115 bash[45495]: cluster 2022-04-22T18:23:54.574112+0000 mon.a (mon.0) 4646 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2022-04-22T18:23:56.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:55 smithi115 bash[46934]: cluster 2022-04-22T18:23:54.574112+0000 mon.a (mon.0) 4646 : cluster [DBG] osdmap e1862: 8 total, 8 up, 8 in 2022-04-22T18:23:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:56 smithi121 bash[36902]: cluster 2022-04-22T18:23:55.468207+0000 mgr.y (mgr.34107) 2020 : cluster [DBG] pgmap v3319: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:56 smithi121 bash[36902]: cluster 2022-04-22T18:23:55.583502+0000 mon.a (mon.0) 4647 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2022-04-22T18:23:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:56 smithi121 bash[36902]: audit 2022-04-22T18:23:56.134665+0000 mon.a (mon.0) 4648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[46934]: cluster 2022-04-22T18:23:55.468207+0000 mgr.y (mgr.34107) 2020 : cluster [DBG] pgmap v3319: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[46934]: cluster 2022-04-22T18:23:55.583502+0000 mon.a (mon.0) 4647 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2022-04-22T18:23:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[46934]: audit 2022-04-22T18:23:56.134665+0000 mon.a (mon.0) 4648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[45495]: cluster 2022-04-22T18:23:55.468207+0000 mgr.y (mgr.34107) 2020 : cluster [DBG] pgmap v3319: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[45495]: cluster 2022-04-22T18:23:55.583502+0000 mon.a (mon.0) 4647 : cluster [DBG] osdmap e1863: 8 total, 8 up, 8 in 2022-04-22T18:23:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:56 smithi115 bash[45495]: audit 2022-04-22T18:23:56.134665+0000 mon.a (mon.0) 4648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:23:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:57 smithi121 bash[36902]: cluster 2022-04-22T18:23:56.594694+0000 mon.a (mon.0) 4649 : cluster [DBG] osdmap e1864: 8 total, 8 up, 8 in 2022-04-22T18:23:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:57 smithi121 bash[36902]: audit 2022-04-22T18:23:56.641254+0000 mon.a (mon.0) 4650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:57 smithi115 bash[45495]: cluster 2022-04-22T18:23:56.594694+0000 mon.a (mon.0) 4649 : cluster [DBG] osdmap e1864: 8 total, 8 up, 8 in 2022-04-22T18:23:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:57 smithi115 bash[45495]: audit 2022-04-22T18:23:56.641254+0000 mon.a (mon.0) 4650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:57 smithi115 bash[46934]: cluster 2022-04-22T18:23:56.594694+0000 mon.a (mon.0) 4649 : cluster [DBG] osdmap e1864: 8 total, 8 up, 8 in 2022-04-22T18:23:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:57 smithi115 bash[46934]: audit 2022-04-22T18:23:56.641254+0000 mon.a (mon.0) 4650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:23:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:58 smithi121 bash[36902]: cluster 2022-04-22T18:23:57.469430+0000 mgr.y (mgr.34107) 2021 : cluster [DBG] pgmap v3322: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:58 smithi121 bash[36902]: cluster 2022-04-22T18:23:57.609162+0000 mon.a (mon.0) 4651 : cluster [DBG] osdmap e1865: 8 total, 8 up, 8 in 2022-04-22T18:23:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:58 smithi121 bash[36902]: audit 2022-04-22T18:23:58.517581+0000 mon.a (mon.0) 4652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[45495]: cluster 2022-04-22T18:23:57.469430+0000 mgr.y (mgr.34107) 2021 : cluster [DBG] pgmap v3322: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[45495]: cluster 2022-04-22T18:23:57.609162+0000 mon.a (mon.0) 4651 : cluster [DBG] osdmap e1865: 8 total, 8 up, 8 in 2022-04-22T18:23:59.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[45495]: audit 2022-04-22T18:23:58.517581+0000 mon.a (mon.0) 4652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:23:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[46934]: cluster 2022-04-22T18:23:57.469430+0000 mgr.y (mgr.34107) 2021 : cluster [DBG] pgmap v3322: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 3.5 KiB/s wr, 1.04k op/s 2022-04-22T18:23:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[46934]: cluster 2022-04-22T18:23:57.609162+0000 mon.a (mon.0) 4651 : cluster [DBG] osdmap e1865: 8 total, 8 up, 8 in 2022-04-22T18:23:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:58 smithi115 bash[46934]: audit 2022-04-22T18:23:58.517581+0000 mon.a (mon.0) 4652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:23:59 smithi121 bash[36902]: cluster 2022-04-22T18:23:58.616235+0000 mon.a (mon.0) 4653 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2022-04-22T18:24:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:23:59 smithi115 bash[46934]: cluster 2022-04-22T18:23:58.616235+0000 mon.a (mon.0) 4653 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2022-04-22T18:24:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:23:59 smithi115 bash[45495]: cluster 2022-04-22T18:23:58.616235+0000 mon.a (mon.0) 4653 : cluster [DBG] osdmap e1866: 8 total, 8 up, 8 in 2022-04-22T18:24:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:00 smithi121 bash[36902]: cluster 2022-04-22T18:23:59.470484+0000 mgr.y (mgr.34107) 2022 : cluster [DBG] pgmap v3325: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:00 smithi121 bash[36902]: cluster 2022-04-22T18:23:59.625316+0000 mon.a (mon.0) 4654 : cluster [DBG] osdmap e1867: 8 total, 8 up, 8 in 2022-04-22T18:24:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:00 smithi115 bash[46934]: cluster 2022-04-22T18:23:59.470484+0000 mgr.y (mgr.34107) 2022 : cluster [DBG] pgmap v3325: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:00 smithi115 bash[46934]: cluster 2022-04-22T18:23:59.625316+0000 mon.a (mon.0) 4654 : cluster [DBG] osdmap e1867: 8 total, 8 up, 8 in 2022-04-22T18:24:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:00 smithi115 bash[45495]: cluster 2022-04-22T18:23:59.470484+0000 mgr.y (mgr.34107) 2022 : cluster [DBG] pgmap v3325: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:00 smithi115 bash[45495]: cluster 2022-04-22T18:23:59.625316+0000 mon.a (mon.0) 4654 : cluster [DBG] osdmap e1867: 8 total, 8 up, 8 in 2022-04-22T18:24:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:01 smithi121 bash[36902]: cluster 2022-04-22T18:24:00.628869+0000 mon.a (mon.0) 4655 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2022-04-22T18:24:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:01 smithi121 bash[36902]: audit 2022-04-22T18:24:01.138032+0000 mon.a (mon.0) 4656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:01 smithi115 bash[45495]: cluster 2022-04-22T18:24:00.628869+0000 mon.a (mon.0) 4655 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2022-04-22T18:24:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:01 smithi115 bash[45495]: audit 2022-04-22T18:24:01.138032+0000 mon.a (mon.0) 4656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:01.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:01 smithi115 bash[46934]: cluster 2022-04-22T18:24:00.628869+0000 mon.a (mon.0) 4655 : cluster [DBG] osdmap e1868: 8 total, 8 up, 8 in 2022-04-22T18:24:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:01 smithi115 bash[46934]: audit 2022-04-22T18:24:01.138032+0000 mon.a (mon.0) 4656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:02 smithi121 bash[36902]: cluster 2022-04-22T18:24:01.471529+0000 mgr.y (mgr.34107) 2023 : cluster [DBG] pgmap v3328: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 802 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2022-04-22T18:24:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:02 smithi121 bash[36902]: cluster 2022-04-22T18:24:01.643061+0000 mon.a (mon.0) 4657 : cluster [DBG] osdmap e1869: 8 total, 8 up, 8 in 2022-04-22T18:24:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:02 smithi121 bash[36902]: audit 2022-04-22T18:24:01.644520+0000 mon.a (mon.0) 4658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[45495]: cluster 2022-04-22T18:24:01.471529+0000 mgr.y (mgr.34107) 2023 : cluster [DBG] pgmap v3328: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 802 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2022-04-22T18:24:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[45495]: cluster 2022-04-22T18:24:01.643061+0000 mon.a (mon.0) 4657 : cluster [DBG] osdmap e1869: 8 total, 8 up, 8 in 2022-04-22T18:24:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[45495]: audit 2022-04-22T18:24:01.644520+0000 mon.a (mon.0) 4658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[46934]: cluster 2022-04-22T18:24:01.471529+0000 mgr.y (mgr.34107) 2023 : cluster [DBG] pgmap v3328: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 802 KiB/s rd, 3.7 KiB/s wr, 1.08k op/s 2022-04-22T18:24:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[46934]: cluster 2022-04-22T18:24:01.643061+0000 mon.a (mon.0) 4657 : cluster [DBG] osdmap e1869: 8 total, 8 up, 8 in 2022-04-22T18:24:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:02 smithi115 bash[46934]: audit 2022-04-22T18:24:01.644520+0000 mon.a (mon.0) 4658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: cluster 2022-04-22T18:24:02.654457+0000 mon.a (mon.0) 4659 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: audit 2022-04-22T18:24:03.235521+0000 mon.a (mon.0) 4660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: audit 2022-04-22T18:24:03.238574+0000 mon.a (mon.0) 4661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: audit 2022-04-22T18:24:03.240433+0000 mon.a (mon.0) 4662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: audit 2022-04-22T18:24:03.520526+0000 mon.a (mon.0) 4663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:03 smithi121 bash[36902]: audit 2022-04-22T18:24:03.595382+0000 mon.a (mon.0) 4664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:24:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:03.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: cluster 2022-04-22T18:24:02.654457+0000 mon.a (mon.0) 4659 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2022-04-22T18:24:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: audit 2022-04-22T18:24:03.235521+0000 mon.a (mon.0) 4660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: audit 2022-04-22T18:24:03.238574+0000 mon.a (mon.0) 4661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: audit 2022-04-22T18:24:03.240433+0000 mon.a (mon.0) 4662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: audit 2022-04-22T18:24:03.520526+0000 mon.a (mon.0) 4663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[45495]: audit 2022-04-22T18:24:03.595382+0000 mon.a (mon.0) 4664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: cluster 2022-04-22T18:24:02.654457+0000 mon.a (mon.0) 4659 : cluster [DBG] osdmap e1870: 8 total, 8 up, 8 in 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: audit 2022-04-22T18:24:03.235521+0000 mon.a (mon.0) 4660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:24:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: audit 2022-04-22T18:24:03.238574+0000 mon.a (mon.0) 4661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:24:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: audit 2022-04-22T18:24:03.240433+0000 mon.a (mon.0) 4662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:24:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: audit 2022-04-22T18:24:03.520526+0000 mon.a (mon.0) 4663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:03 smithi115 bash[46934]: audit 2022-04-22T18:24:03.595382+0000 mon.a (mon.0) 4664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:24:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:04 smithi121 bash[36902]: cluster 2022-04-22T18:24:03.472245+0000 mgr.y (mgr.34107) 2024 : cluster [DBG] pgmap v3331: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 803 KiB/s rd, 3.2 KiB/s wr, 1.08k op/s 2022-04-22T18:24:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:04 smithi121 bash[36902]: cluster 2022-04-22T18:24:03.666477+0000 mon.a (mon.0) 4665 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2022-04-22T18:24:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:04 smithi115 bash[45495]: cluster 2022-04-22T18:24:03.472245+0000 mgr.y (mgr.34107) 2024 : cluster [DBG] pgmap v3331: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 803 KiB/s rd, 3.2 KiB/s wr, 1.08k op/s 2022-04-22T18:24:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:04 smithi115 bash[45495]: cluster 2022-04-22T18:24:03.666477+0000 mon.a (mon.0) 4665 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2022-04-22T18:24:05.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:04 smithi115 bash[46934]: cluster 2022-04-22T18:24:03.472245+0000 mgr.y (mgr.34107) 2024 : cluster [DBG] pgmap v3331: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 803 KiB/s rd, 3.2 KiB/s wr, 1.08k op/s 2022-04-22T18:24:05.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:04 smithi115 bash[46934]: cluster 2022-04-22T18:24:03.666477+0000 mon.a (mon.0) 4665 : cluster [DBG] osdmap e1871: 8 total, 8 up, 8 in 2022-04-22T18:24:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:05 smithi121 bash[36902]: cluster 2022-04-22T18:24:04.673511+0000 mon.a (mon.0) 4666 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2022-04-22T18:24:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:05 smithi115 bash[45495]: cluster 2022-04-22T18:24:04.673511+0000 mon.a (mon.0) 4666 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2022-04-22T18:24:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:05 smithi115 bash[46934]: cluster 2022-04-22T18:24:04.673511+0000 mon.a (mon.0) 4666 : cluster [DBG] osdmap e1872: 8 total, 8 up, 8 in 2022-04-22T18:24:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:06 smithi121 bash[36902]: cluster 2022-04-22T18:24:05.473326+0000 mgr.y (mgr.34107) 2025 : cluster [DBG] pgmap v3334: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.2 KiB/s wr, 1.12k op/s 2022-04-22T18:24:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:06 smithi121 bash[36902]: cluster 2022-04-22T18:24:05.686530+0000 mon.a (mon.0) 4667 : cluster [DBG] osdmap e1873: 8 total, 8 up, 8 in 2022-04-22T18:24:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:06 smithi121 bash[36902]: audit 2022-04-22T18:24:06.141192+0000 mon.a (mon.0) 4668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:06 smithi121 bash[36902]: audit 2022-04-22T18:24:06.648876+0000 mon.a (mon.0) 4669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[45495]: cluster 2022-04-22T18:24:05.473326+0000 mgr.y (mgr.34107) 2025 : cluster [DBG] pgmap v3334: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.2 KiB/s wr, 1.12k op/s 2022-04-22T18:24:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[45495]: cluster 2022-04-22T18:24:05.686530+0000 mon.a (mon.0) 4667 : cluster [DBG] osdmap e1873: 8 total, 8 up, 8 in 2022-04-22T18:24:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[45495]: audit 2022-04-22T18:24:06.141192+0000 mon.a (mon.0) 4668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[45495]: audit 2022-04-22T18:24:06.648876+0000 mon.a (mon.0) 4669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[46934]: cluster 2022-04-22T18:24:05.473326+0000 mgr.y (mgr.34107) 2025 : cluster [DBG] pgmap v3334: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.2 KiB/s wr, 1.12k op/s 2022-04-22T18:24:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[46934]: cluster 2022-04-22T18:24:05.686530+0000 mon.a (mon.0) 4667 : cluster [DBG] osdmap e1873: 8 total, 8 up, 8 in 2022-04-22T18:24:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[46934]: audit 2022-04-22T18:24:06.141192+0000 mon.a (mon.0) 4668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:06 smithi115 bash[46934]: audit 2022-04-22T18:24:06.648876+0000 mon.a (mon.0) 4669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:07 smithi121 bash[36902]: cluster 2022-04-22T18:24:06.688374+0000 mon.a (mon.0) 4670 : cluster [DBG] osdmap e1874: 8 total, 8 up, 8 in 2022-04-22T18:24:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:07 smithi115 bash[46934]: cluster 2022-04-22T18:24:06.688374+0000 mon.a (mon.0) 4670 : cluster [DBG] osdmap e1874: 8 total, 8 up, 8 in 2022-04-22T18:24:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:07 smithi115 bash[45495]: cluster 2022-04-22T18:24:06.688374+0000 mon.a (mon.0) 4670 : cluster [DBG] osdmap e1874: 8 total, 8 up, 8 in 2022-04-22T18:24:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:08 smithi121 bash[36902]: cluster 2022-04-22T18:24:07.474458+0000 mgr.y (mgr.34107) 2026 : cluster [DBG] pgmap v3337: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.5 KiB/s wr, 1.12k op/s 2022-04-22T18:24:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:08 smithi121 bash[36902]: cluster 2022-04-22T18:24:07.704574+0000 mon.a (mon.0) 4671 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2022-04-22T18:24:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:08 smithi121 bash[36902]: audit 2022-04-22T18:24:08.523411+0000 mon.a (mon.0) 4672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[45495]: cluster 2022-04-22T18:24:07.474458+0000 mgr.y (mgr.34107) 2026 : cluster [DBG] pgmap v3337: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.5 KiB/s wr, 1.12k op/s 2022-04-22T18:24:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[45495]: cluster 2022-04-22T18:24:07.704574+0000 mon.a (mon.0) 4671 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2022-04-22T18:24:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[45495]: audit 2022-04-22T18:24:08.523411+0000 mon.a (mon.0) 4672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[46934]: cluster 2022-04-22T18:24:07.474458+0000 mgr.y (mgr.34107) 2026 : cluster [DBG] pgmap v3337: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 836 KiB/s rd, 3.5 KiB/s wr, 1.12k op/s 2022-04-22T18:24:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[46934]: cluster 2022-04-22T18:24:07.704574+0000 mon.a (mon.0) 4671 : cluster [DBG] osdmap e1875: 8 total, 8 up, 8 in 2022-04-22T18:24:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:08 smithi115 bash[46934]: audit 2022-04-22T18:24:08.523411+0000 mon.a (mon.0) 4672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:09 smithi121 bash[36902]: cluster 2022-04-22T18:24:08.712787+0000 mon.a (mon.0) 4673 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2022-04-22T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:09 smithi121 bash[36902]: cluster 2022-04-22T18:24:09.475498+0000 mgr.y (mgr.34107) 2027 : cluster [DBG] pgmap v3340: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:24:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:09 smithi115 bash[45495]: cluster 2022-04-22T18:24:08.712787+0000 mon.a (mon.0) 4673 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2022-04-22T18:24:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:09 smithi115 bash[45495]: cluster 2022-04-22T18:24:09.475498+0000 mgr.y (mgr.34107) 2027 : cluster [DBG] pgmap v3340: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:24:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:09 smithi115 bash[46934]: cluster 2022-04-22T18:24:08.712787+0000 mon.a (mon.0) 4673 : cluster [DBG] osdmap e1876: 8 total, 8 up, 8 in 2022-04-22T18:24:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:09 smithi115 bash[46934]: cluster 2022-04-22T18:24:09.475498+0000 mgr.y (mgr.34107) 2027 : cluster [DBG] pgmap v3340: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-04-22T18:24:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:10 smithi121 bash[36902]: cluster 2022-04-22T18:24:09.724031+0000 mon.a (mon.0) 4674 : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2022-04-22T18:24:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:10 smithi115 bash[45495]: cluster 2022-04-22T18:24:09.724031+0000 mon.a (mon.0) 4674 : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2022-04-22T18:24:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:10 smithi115 bash[46934]: cluster 2022-04-22T18:24:09.724031+0000 mon.a (mon.0) 4674 : cluster [DBG] osdmap e1877: 8 total, 8 up, 8 in 2022-04-22T18:24:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:11 smithi121 bash[36902]: cluster 2022-04-22T18:24:10.726921+0000 mon.a (mon.0) 4675 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2022-04-22T18:24:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:11 smithi121 bash[36902]: audit 2022-04-22T18:24:11.144482+0000 mon.a (mon.0) 4676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:11 smithi121 bash[36902]: cluster 2022-04-22T18:24:11.476539+0000 mgr.y (mgr.34107) 2028 : cluster [DBG] pgmap v3343: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 868 KiB/s rd, 3.5 KiB/s wr, 1.17k op/s 2022-04-22T18:24:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:11 smithi121 bash[36902]: audit 2022-04-22T18:24:11.652314+0000 mon.a (mon.0) 4677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:11.819Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[45495]: cluster 2022-04-22T18:24:10.726921+0000 mon.a (mon.0) 4675 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2022-04-22T18:24:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[45495]: audit 2022-04-22T18:24:11.144482+0000 mon.a (mon.0) 4676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[45495]: cluster 2022-04-22T18:24:11.476539+0000 mgr.y (mgr.34107) 2028 : cluster [DBG] pgmap v3343: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 868 KiB/s rd, 3.5 KiB/s wr, 1.17k op/s 2022-04-22T18:24:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[45495]: audit 2022-04-22T18:24:11.652314+0000 mon.a (mon.0) 4677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[46934]: cluster 2022-04-22T18:24:10.726921+0000 mon.a (mon.0) 4675 : cluster [DBG] osdmap e1878: 8 total, 8 up, 8 in 2022-04-22T18:24:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[46934]: audit 2022-04-22T18:24:11.144482+0000 mon.a (mon.0) 4676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[46934]: cluster 2022-04-22T18:24:11.476539+0000 mgr.y (mgr.34107) 2028 : cluster [DBG] pgmap v3343: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 868 KiB/s rd, 3.5 KiB/s wr, 1.17k op/s 2022-04-22T18:24:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:11 smithi115 bash[46934]: audit 2022-04-22T18:24:11.652314+0000 mon.a (mon.0) 4677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:12 smithi121 bash[36902]: cluster 2022-04-22T18:24:11.747110+0000 mon.a (mon.0) 4678 : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2022-04-22T18:24:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:12 smithi115 bash[45495]: cluster 2022-04-22T18:24:11.747110+0000 mon.a (mon.0) 4678 : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2022-04-22T18:24:13.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:12 smithi115 bash[46934]: cluster 2022-04-22T18:24:11.747110+0000 mon.a (mon.0) 4678 : cluster [DBG] osdmap e1879: 8 total, 8 up, 8 in 2022-04-22T18:24:13.741 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:24:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:13.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[45495]: cluster 2022-04-22T18:24:12.756435+0000 mon.a (mon.0) 4679 : cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2022-04-22T18:24:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[45495]: cluster 2022-04-22T18:24:13.477532+0000 mgr.y (mgr.34107) 2029 : cluster [DBG] pgmap v3346: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 869 KiB/s rd, 3.0 KiB/s wr, 1.17k op/s 2022-04-22T18:24:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[45495]: audit 2022-04-22T18:24:13.526186+0000 mon.a (mon.0) 4680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[46934]: cluster 2022-04-22T18:24:12.756435+0000 mon.a (mon.0) 4679 : cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2022-04-22T18:24:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[46934]: cluster 2022-04-22T18:24:13.477532+0000 mgr.y (mgr.34107) 2029 : cluster [DBG] pgmap v3346: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 869 KiB/s rd, 3.0 KiB/s wr, 1.17k op/s 2022-04-22T18:24:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:13 smithi115 bash[46934]: audit 2022-04-22T18:24:13.526186+0000 mon.a (mon.0) 4680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:14.087 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (47m) 5m ago 61m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (60m) 6m ago 60m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (49m) 5m ago 62m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (48m) 6m ago 62m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (56m) 6m ago 66m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (57m) 5m ago 68m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:24:14.088 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (56m) 5m ago 68m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (55m) 6m ago 66m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (55m) 5m ago 66m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (48m) 5m ago 61m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (48m) 6m ago 61m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (54m) 5m ago 65m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (53m) 5m ago 65m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (52m) 5m ago 64m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (52m) 5m ago 64m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:24:14.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (51m) 6m ago 64m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:24:14.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (51m) 6m ago 63m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:24:14.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (50m) 6m ago 63m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:24:14.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (49m) 6m ago 62m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:24:14.090 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (48m) 6m ago 61m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:24:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:13 smithi121 bash[36902]: cluster 2022-04-22T18:24:12.756435+0000 mon.a (mon.0) 4679 : cluster [DBG] osdmap e1880: 8 total, 8 up, 8 in 2022-04-22T18:24:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:13 smithi121 bash[36902]: cluster 2022-04-22T18:24:13.477532+0000 mgr.y (mgr.34107) 2029 : cluster [DBG] pgmap v3346: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 869 KiB/s rd, 3.0 KiB/s wr, 1.17k op/s 2022-04-22T18:24:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:13 smithi121 bash[36902]: audit 2022-04-22T18:24:13.526186+0000 mon.a (mon.0) 4680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:14.494 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:24:14.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:24:14.504 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:24:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: audit 2022-04-22T18:24:13.718560+0000 mgr.y (mgr.34107) 2030 : audit [DBG] from='client.47422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: cluster 2022-04-22T18:24:13.767878+0000 mon.a (mon.0) 4681 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: audit 2022-04-22T18:24:13.938991+0000 mon.a (mon.0) 4682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: audit 2022-04-22T18:24:13.939767+0000 mon.a (mon.0) 4683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: audit 2022-04-22T18:24:14.064737+0000 mgr.y (mgr.34107) 2031 : audit [DBG] from='client.47428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[45495]: audit 2022-04-22T18:24:14.487249+0000 mon.a (mon.0) 4684 : audit [DBG] from='client.? 172.21.15.115:0/1455925085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: audit 2022-04-22T18:24:13.718560+0000 mgr.y (mgr.34107) 2030 : audit [DBG] from='client.47422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: cluster 2022-04-22T18:24:13.767878+0000 mon.a (mon.0) 4681 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2022-04-22T18:24:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: audit 2022-04-22T18:24:13.938991+0000 mon.a (mon.0) 4682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:24:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: audit 2022-04-22T18:24:13.939767+0000 mon.a (mon.0) 4683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:24:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: audit 2022-04-22T18:24:14.064737+0000 mgr.y (mgr.34107) 2031 : audit [DBG] from='client.47428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:14 smithi115 bash[46934]: audit 2022-04-22T18:24:14.487249+0000 mon.a (mon.0) 4684 : audit [DBG] from='client.? 172.21.15.115:0/1455925085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: audit 2022-04-22T18:24:13.718560+0000 mgr.y (mgr.34107) 2030 : audit [DBG] from='client.47422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: cluster 2022-04-22T18:24:13.767878+0000 mon.a (mon.0) 4681 : cluster [DBG] osdmap e1881: 8 total, 8 up, 8 in 2022-04-22T18:24:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: audit 2022-04-22T18:24:13.938991+0000 mon.a (mon.0) 4682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:24:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: audit 2022-04-22T18:24:13.939767+0000 mon.a (mon.0) 4683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:24:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: audit 2022-04-22T18:24:14.064737+0000 mgr.y (mgr.34107) 2031 : audit [DBG] from='client.47428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:15.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:14 smithi121 bash[36902]: audit 2022-04-22T18:24:14.487249+0000 mon.a (mon.0) 4684 : audit [DBG] from='client.? 172.21.15.115:0/1455925085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:15 smithi115 bash[45495]: cluster 2022-04-22T18:24:14.776446+0000 mon.a (mon.0) 4685 : cluster [DBG] osdmap e1882: 8 total, 8 up, 8 in 2022-04-22T18:24:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:15 smithi115 bash[45495]: cluster 2022-04-22T18:24:15.478600+0000 mgr.y (mgr.34107) 2032 : cluster [DBG] pgmap v3349: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.2 KiB/s wr, 1.22k op/s 2022-04-22T18:24:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:15 smithi115 bash[46934]: cluster 2022-04-22T18:24:14.776446+0000 mon.a (mon.0) 4685 : cluster [DBG] osdmap e1882: 8 total, 8 up, 8 in 2022-04-22T18:24:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:15 smithi115 bash[46934]: cluster 2022-04-22T18:24:15.478600+0000 mgr.y (mgr.34107) 2032 : cluster [DBG] pgmap v3349: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.2 KiB/s wr, 1.22k op/s 2022-04-22T18:24:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:15 smithi121 bash[36902]: cluster 2022-04-22T18:24:14.776446+0000 mon.a (mon.0) 4685 : cluster [DBG] osdmap e1882: 8 total, 8 up, 8 in 2022-04-22T18:24:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:15 smithi121 bash[36902]: cluster 2022-04-22T18:24:15.478600+0000 mgr.y (mgr.34107) 2032 : cluster [DBG] pgmap v3349: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.2 KiB/s wr, 1.22k op/s 2022-04-22T18:24:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[46934]: cluster 2022-04-22T18:24:15.785457+0000 mon.a (mon.0) 4686 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2022-04-22T18:24:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[46934]: audit 2022-04-22T18:24:16.147460+0000 mon.a (mon.0) 4687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[46934]: audit 2022-04-22T18:24:16.656159+0000 mon.a (mon.0) 4688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[45495]: cluster 2022-04-22T18:24:15.785457+0000 mon.a (mon.0) 4686 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2022-04-22T18:24:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[45495]: audit 2022-04-22T18:24:16.147460+0000 mon.a (mon.0) 4687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:16 smithi115 bash[45495]: audit 2022-04-22T18:24:16.656159+0000 mon.a (mon.0) 4688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:16 smithi121 bash[36902]: cluster 2022-04-22T18:24:15.785457+0000 mon.a (mon.0) 4686 : cluster [DBG] osdmap e1883: 8 total, 8 up, 8 in 2022-04-22T18:24:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:16 smithi121 bash[36902]: audit 2022-04-22T18:24:16.147460+0000 mon.a (mon.0) 4687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:16 smithi121 bash[36902]: audit 2022-04-22T18:24:16.656159+0000 mon.a (mon.0) 4688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:17 smithi121 bash[36902]: cluster 2022-04-22T18:24:16.798726+0000 mon.a (mon.0) 4689 : cluster [DBG] osdmap e1884: 8 total, 8 up, 8 in 2022-04-22T18:24:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:17 smithi121 bash[36902]: cluster 2022-04-22T18:24:17.479716+0000 mgr.y (mgr.34107) 2033 : cluster [DBG] pgmap v3352: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.5 KiB/s wr, 1.22k op/s 2022-04-22T18:24:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:17 smithi115 bash[45495]: cluster 2022-04-22T18:24:16.798726+0000 mon.a (mon.0) 4689 : cluster [DBG] osdmap e1884: 8 total, 8 up, 8 in 2022-04-22T18:24:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:17 smithi115 bash[45495]: cluster 2022-04-22T18:24:17.479716+0000 mgr.y (mgr.34107) 2033 : cluster [DBG] pgmap v3352: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.5 KiB/s wr, 1.22k op/s 2022-04-22T18:24:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:17 smithi115 bash[46934]: cluster 2022-04-22T18:24:16.798726+0000 mon.a (mon.0) 4689 : cluster [DBG] osdmap e1884: 8 total, 8 up, 8 in 2022-04-22T18:24:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:17 smithi115 bash[46934]: cluster 2022-04-22T18:24:17.479716+0000 mgr.y (mgr.34107) 2033 : cluster [DBG] pgmap v3352: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 902 KiB/s rd, 3.5 KiB/s wr, 1.22k op/s 2022-04-22T18:24:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:18 smithi121 bash[36902]: cluster 2022-04-22T18:24:17.810172+0000 mon.a (mon.0) 4690 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2022-04-22T18:24:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:18 smithi121 bash[36902]: audit 2022-04-22T18:24:18.529252+0000 mon.a (mon.0) 4691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:19.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:18 smithi115 bash[45495]: cluster 2022-04-22T18:24:17.810172+0000 mon.a (mon.0) 4690 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2022-04-22T18:24:19.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:18 smithi115 bash[45495]: audit 2022-04-22T18:24:18.529252+0000 mon.a (mon.0) 4691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:19.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:18 smithi115 bash[46934]: cluster 2022-04-22T18:24:17.810172+0000 mon.a (mon.0) 4690 : cluster [DBG] osdmap e1885: 8 total, 8 up, 8 in 2022-04-22T18:24:19.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:18 smithi115 bash[46934]: audit 2022-04-22T18:24:18.529252+0000 mon.a (mon.0) 4691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:19 smithi121 bash[36902]: cluster 2022-04-22T18:24:18.818535+0000 mon.a (mon.0) 4692 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2022-04-22T18:24:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:19 smithi121 bash[36902]: cluster 2022-04-22T18:24:19.480718+0000 mgr.y (mgr.34107) 2034 : cluster [DBG] pgmap v3355: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:19 smithi115 bash[45495]: cluster 2022-04-22T18:24:18.818535+0000 mon.a (mon.0) 4692 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2022-04-22T18:24:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:19 smithi115 bash[45495]: cluster 2022-04-22T18:24:19.480718+0000 mgr.y (mgr.34107) 2034 : cluster [DBG] pgmap v3355: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:19 smithi115 bash[46934]: cluster 2022-04-22T18:24:18.818535+0000 mon.a (mon.0) 4692 : cluster [DBG] osdmap e1886: 8 total, 8 up, 8 in 2022-04-22T18:24:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:19 smithi115 bash[46934]: cluster 2022-04-22T18:24:19.480718+0000 mgr.y (mgr.34107) 2034 : cluster [DBG] pgmap v3355: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-04-22T18:24:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:20 smithi121 bash[36902]: cluster 2022-04-22T18:24:19.831386+0000 mon.a (mon.0) 4693 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2022-04-22T18:24:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:20 smithi115 bash[45495]: cluster 2022-04-22T18:24:19.831386+0000 mon.a (mon.0) 4693 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2022-04-22T18:24:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:20 smithi115 bash[46934]: cluster 2022-04-22T18:24:19.831386+0000 mon.a (mon.0) 4693 : cluster [DBG] osdmap e1887: 8 total, 8 up, 8 in 2022-04-22T18:24:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:21 smithi121 bash[36902]: cluster 2022-04-22T18:24:20.841091+0000 mon.a (mon.0) 4694 : cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2022-04-22T18:24:22.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:21 smithi121 bash[36902]: audit 2022-04-22T18:24:21.150398+0000 mon.a (mon.0) 4695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:22.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:21 smithi121 bash[36902]: cluster 2022-04-22T18:24:21.481706+0000 mgr.y (mgr.34107) 2035 : cluster [DBG] pgmap v3358: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 933 KiB/s rd, 3.2 KiB/s wr, 1.26k op/s 2022-04-22T18:24:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:21 smithi121 bash[36902]: audit 2022-04-22T18:24:21.660026+0000 mon.a (mon.0) 4696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:21.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[45495]: cluster 2022-04-22T18:24:20.841091+0000 mon.a (mon.0) 4694 : cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2022-04-22T18:24:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[45495]: audit 2022-04-22T18:24:21.150398+0000 mon.a (mon.0) 4695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[45495]: cluster 2022-04-22T18:24:21.481706+0000 mgr.y (mgr.34107) 2035 : cluster [DBG] pgmap v3358: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 933 KiB/s rd, 3.2 KiB/s wr, 1.26k op/s 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[45495]: audit 2022-04-22T18:24:21.660026+0000 mon.a (mon.0) 4696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[46934]: cluster 2022-04-22T18:24:20.841091+0000 mon.a (mon.0) 4694 : cluster [DBG] osdmap e1888: 8 total, 8 up, 8 in 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[46934]: audit 2022-04-22T18:24:21.150398+0000 mon.a (mon.0) 4695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[46934]: cluster 2022-04-22T18:24:21.481706+0000 mgr.y (mgr.34107) 2035 : cluster [DBG] pgmap v3358: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 933 KiB/s rd, 3.2 KiB/s wr, 1.26k op/s 2022-04-22T18:24:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:21 smithi115 bash[46934]: audit 2022-04-22T18:24:21.660026+0000 mon.a (mon.0) 4696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:22 smithi121 bash[36902]: cluster 2022-04-22T18:24:21.844457+0000 mon.a (mon.0) 4697 : cluster [DBG] osdmap e1889: 8 total, 8 up, 8 in 2022-04-22T18:24:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:22 smithi115 bash[45495]: cluster 2022-04-22T18:24:21.844457+0000 mon.a (mon.0) 4697 : cluster [DBG] osdmap e1889: 8 total, 8 up, 8 in 2022-04-22T18:24:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:22 smithi115 bash[46934]: cluster 2022-04-22T18:24:21.844457+0000 mon.a (mon.0) 4697 : cluster [DBG] osdmap e1889: 8 total, 8 up, 8 in 2022-04-22T18:24:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:23.701Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[45495]: cluster 2022-04-22T18:24:22.860359+0000 mon.a (mon.0) 4698 : cluster [DBG] osdmap e1890: 8 total, 8 up, 8 in 2022-04-22T18:24:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[45495]: cluster 2022-04-22T18:24:23.482805+0000 mgr.y (mgr.34107) 2036 : cluster [DBG] pgmap v3361: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2022-04-22T18:24:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[45495]: audit 2022-04-22T18:24:23.532231+0000 mon.a (mon.0) 4699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[46934]: cluster 2022-04-22T18:24:22.860359+0000 mon.a (mon.0) 4698 : cluster [DBG] osdmap e1890: 8 total, 8 up, 8 in 2022-04-22T18:24:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[46934]: cluster 2022-04-22T18:24:23.482805+0000 mgr.y (mgr.34107) 2036 : cluster [DBG] pgmap v3361: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2022-04-22T18:24:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:23 smithi115 bash[46934]: audit 2022-04-22T18:24:23.532231+0000 mon.a (mon.0) 4699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:23 smithi121 bash[36902]: cluster 2022-04-22T18:24:22.860359+0000 mon.a (mon.0) 4698 : cluster [DBG] osdmap e1890: 8 total, 8 up, 8 in 2022-04-22T18:24:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:23 smithi121 bash[36902]: cluster 2022-04-22T18:24:23.482805+0000 mgr.y (mgr.34107) 2036 : cluster [DBG] pgmap v3361: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 935 KiB/s rd, 3.7 KiB/s wr, 1.26k op/s 2022-04-22T18:24:24.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:23 smithi121 bash[36902]: audit 2022-04-22T18:24:23.532231+0000 mon.a (mon.0) 4699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:24 smithi121 bash[36902]: cluster 2022-04-22T18:24:23.871231+0000 mon.a (mon.0) 4700 : cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2022-04-22T18:24:25.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:24 smithi115 bash[45495]: cluster 2022-04-22T18:24:23.871231+0000 mon.a (mon.0) 4700 : cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2022-04-22T18:24:25.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:24 smithi115 bash[46934]: cluster 2022-04-22T18:24:23.871231+0000 mon.a (mon.0) 4700 : cluster [DBG] osdmap e1891: 8 total, 8 up, 8 in 2022-04-22T18:24:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:25 smithi121 bash[36902]: cluster 2022-04-22T18:24:24.878592+0000 mon.a (mon.0) 4701 : cluster [DBG] osdmap e1892: 8 total, 8 up, 8 in 2022-04-22T18:24:26.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:25 smithi121 bash[36902]: cluster 2022-04-22T18:24:25.483881+0000 mgr.y (mgr.34107) 2037 : cluster [DBG] pgmap v3364: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2022-04-22T18:24:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:25 smithi115 bash[46934]: cluster 2022-04-22T18:24:24.878592+0000 mon.a (mon.0) 4701 : cluster [DBG] osdmap e1892: 8 total, 8 up, 8 in 2022-04-22T18:24:26.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:25 smithi115 bash[46934]: cluster 2022-04-22T18:24:25.483881+0000 mgr.y (mgr.34107) 2037 : cluster [DBG] pgmap v3364: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2022-04-22T18:24:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:25 smithi115 bash[45495]: cluster 2022-04-22T18:24:24.878592+0000 mon.a (mon.0) 4701 : cluster [DBG] osdmap e1892: 8 total, 8 up, 8 in 2022-04-22T18:24:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:25 smithi115 bash[45495]: cluster 2022-04-22T18:24:25.483881+0000 mgr.y (mgr.34107) 2037 : cluster [DBG] pgmap v3364: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.7 KiB/s wr, 1.30k op/s 2022-04-22T18:24:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:26 smithi121 bash[36902]: cluster 2022-04-22T18:24:25.888165+0000 mon.a (mon.0) 4702 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2022-04-22T18:24:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:26 smithi121 bash[36902]: audit 2022-04-22T18:24:26.153672+0000 mon.a (mon.0) 4703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:26 smithi121 bash[36902]: audit 2022-04-22T18:24:26.663737+0000 mon.a (mon.0) 4704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[46934]: cluster 2022-04-22T18:24:25.888165+0000 mon.a (mon.0) 4702 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2022-04-22T18:24:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[46934]: audit 2022-04-22T18:24:26.153672+0000 mon.a (mon.0) 4703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[46934]: audit 2022-04-22T18:24:26.663737+0000 mon.a (mon.0) 4704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[45495]: cluster 2022-04-22T18:24:25.888165+0000 mon.a (mon.0) 4702 : cluster [DBG] osdmap e1893: 8 total, 8 up, 8 in 2022-04-22T18:24:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[45495]: audit 2022-04-22T18:24:26.153672+0000 mon.a (mon.0) 4703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:26 smithi115 bash[45495]: audit 2022-04-22T18:24:26.663737+0000 mon.a (mon.0) 4704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:27 smithi121 bash[36902]: cluster 2022-04-22T18:24:26.895362+0000 mon.a (mon.0) 4705 : cluster [DBG] osdmap e1894: 8 total, 8 up, 8 in 2022-04-22T18:24:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:27 smithi121 bash[36902]: cluster 2022-04-22T18:24:27.484890+0000 mgr.y (mgr.34107) 2038 : cluster [DBG] pgmap v3367: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.5 KiB/s wr, 1.30k op/s 2022-04-22T18:24:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:27 smithi115 bash[45495]: cluster 2022-04-22T18:24:26.895362+0000 mon.a (mon.0) 4705 : cluster [DBG] osdmap e1894: 8 total, 8 up, 8 in 2022-04-22T18:24:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:27 smithi115 bash[45495]: cluster 2022-04-22T18:24:27.484890+0000 mgr.y (mgr.34107) 2038 : cluster [DBG] pgmap v3367: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.5 KiB/s wr, 1.30k op/s 2022-04-22T18:24:28.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:27 smithi115 bash[46934]: cluster 2022-04-22T18:24:26.895362+0000 mon.a (mon.0) 4705 : cluster [DBG] osdmap e1894: 8 total, 8 up, 8 in 2022-04-22T18:24:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:27 smithi115 bash[46934]: cluster 2022-04-22T18:24:27.484890+0000 mgr.y (mgr.34107) 2038 : cluster [DBG] pgmap v3367: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 967 KiB/s rd, 3.5 KiB/s wr, 1.30k op/s 2022-04-22T18:24:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:28 smithi121 bash[36902]: cluster 2022-04-22T18:24:27.907337+0000 mon.a (mon.0) 4706 : cluster [DBG] osdmap e1895: 8 total, 8 up, 8 in 2022-04-22T18:24:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:28 smithi121 bash[36902]: audit 2022-04-22T18:24:28.535599+0000 mon.a (mon.0) 4707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:29.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:28 smithi115 bash[45495]: cluster 2022-04-22T18:24:27.907337+0000 mon.a (mon.0) 4706 : cluster [DBG] osdmap e1895: 8 total, 8 up, 8 in 2022-04-22T18:24:29.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:28 smithi115 bash[45495]: audit 2022-04-22T18:24:28.535599+0000 mon.a (mon.0) 4707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:29.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:28 smithi115 bash[46934]: cluster 2022-04-22T18:24:27.907337+0000 mon.a (mon.0) 4706 : cluster [DBG] osdmap e1895: 8 total, 8 up, 8 in 2022-04-22T18:24:29.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:28 smithi115 bash[46934]: audit 2022-04-22T18:24:28.535599+0000 mon.a (mon.0) 4707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:29 smithi121 bash[36902]: cluster 2022-04-22T18:24:28.915847+0000 mon.a (mon.0) 4708 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2022-04-22T18:24:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:29 smithi121 bash[36902]: cluster 2022-04-22T18:24:29.485732+0000 mgr.y (mgr.34107) 2039 : cluster [DBG] pgmap v3370: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:29 smithi115 bash[45495]: cluster 2022-04-22T18:24:28.915847+0000 mon.a (mon.0) 4708 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2022-04-22T18:24:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:29 smithi115 bash[45495]: cluster 2022-04-22T18:24:29.485732+0000 mgr.y (mgr.34107) 2039 : cluster [DBG] pgmap v3370: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:29 smithi115 bash[46934]: cluster 2022-04-22T18:24:28.915847+0000 mon.a (mon.0) 4708 : cluster [DBG] osdmap e1896: 8 total, 8 up, 8 in 2022-04-22T18:24:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:29 smithi115 bash[46934]: cluster 2022-04-22T18:24:29.485732+0000 mgr.y (mgr.34107) 2039 : cluster [DBG] pgmap v3370: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:30 smithi121 bash[36902]: cluster 2022-04-22T18:24:29.927498+0000 mon.a (mon.0) 4709 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2022-04-22T18:24:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:30 smithi115 bash[45495]: cluster 2022-04-22T18:24:29.927498+0000 mon.a (mon.0) 4709 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2022-04-22T18:24:31.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:30 smithi115 bash[46934]: cluster 2022-04-22T18:24:29.927498+0000 mon.a (mon.0) 4709 : cluster [DBG] osdmap e1897: 8 total, 8 up, 8 in 2022-04-22T18:24:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:31 smithi121 bash[36902]: cluster 2022-04-22T18:24:30.940064+0000 mon.a (mon.0) 4710 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2022-04-22T18:24:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:31 smithi121 bash[36902]: audit 2022-04-22T18:24:31.156647+0000 mon.a (mon.0) 4711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:31 smithi121 bash[36902]: cluster 2022-04-22T18:24:31.486539+0000 mgr.y (mgr.34107) 2040 : cluster [DBG] pgmap v3373: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 998 KiB/s rd, 3.0 KiB/s wr, 1.35k op/s 2022-04-22T18:24:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:31 smithi121 bash[36902]: audit 2022-04-22T18:24:31.667494+0000 mon.a (mon.0) 4712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:32.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:31.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[45495]: cluster 2022-04-22T18:24:30.940064+0000 mon.a (mon.0) 4710 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2022-04-22T18:24:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[45495]: audit 2022-04-22T18:24:31.156647+0000 mon.a (mon.0) 4711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[45495]: cluster 2022-04-22T18:24:31.486539+0000 mgr.y (mgr.34107) 2040 : cluster [DBG] pgmap v3373: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 998 KiB/s rd, 3.0 KiB/s wr, 1.35k op/s 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[45495]: audit 2022-04-22T18:24:31.667494+0000 mon.a (mon.0) 4712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[46934]: cluster 2022-04-22T18:24:30.940064+0000 mon.a (mon.0) 4710 : cluster [DBG] osdmap e1898: 8 total, 8 up, 8 in 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[46934]: audit 2022-04-22T18:24:31.156647+0000 mon.a (mon.0) 4711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[46934]: cluster 2022-04-22T18:24:31.486539+0000 mgr.y (mgr.34107) 2040 : cluster [DBG] pgmap v3373: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 998 KiB/s rd, 3.0 KiB/s wr, 1.35k op/s 2022-04-22T18:24:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:31 smithi115 bash[46934]: audit 2022-04-22T18:24:31.667494+0000 mon.a (mon.0) 4712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:32 smithi121 bash[36902]: cluster 2022-04-22T18:24:31.946175+0000 mon.a (mon.0) 4713 : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2022-04-22T18:24:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:32 smithi115 bash[45495]: cluster 2022-04-22T18:24:31.946175+0000 mon.a (mon.0) 4713 : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2022-04-22T18:24:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:32 smithi115 bash[46934]: cluster 2022-04-22T18:24:31.946175+0000 mon.a (mon.0) 4713 : cluster [DBG] osdmap e1899: 8 total, 8 up, 8 in 2022-04-22T18:24:33.973 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:33.701Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:33 smithi121 bash[36902]: cluster 2022-04-22T18:24:32.958200+0000 mon.a (mon.0) 4714 : cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2022-04-22T18:24:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:33 smithi121 bash[36902]: cluster 2022-04-22T18:24:33.487664+0000 mgr.y (mgr.34107) 2041 : cluster [DBG] pgmap v3376: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2022-04-22T18:24:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:33 smithi121 bash[36902]: audit 2022-04-22T18:24:33.538819+0000 mon.a (mon.0) 4715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:34.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[45495]: cluster 2022-04-22T18:24:32.958200+0000 mon.a (mon.0) 4714 : cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2022-04-22T18:24:34.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[45495]: cluster 2022-04-22T18:24:33.487664+0000 mgr.y (mgr.34107) 2041 : cluster [DBG] pgmap v3376: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2022-04-22T18:24:34.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[45495]: audit 2022-04-22T18:24:33.538819+0000 mon.a (mon.0) 4715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[46934]: cluster 2022-04-22T18:24:32.958200+0000 mon.a (mon.0) 4714 : cluster [DBG] osdmap e1900: 8 total, 8 up, 8 in 2022-04-22T18:24:34.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[46934]: cluster 2022-04-22T18:24:33.487664+0000 mgr.y (mgr.34107) 2041 : cluster [DBG] pgmap v3376: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 999 KiB/s rd, 3.7 KiB/s wr, 1.35k op/s 2022-04-22T18:24:34.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:33 smithi115 bash[46934]: audit 2022-04-22T18:24:33.538819+0000 mon.a (mon.0) 4715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:34 smithi121 bash[36902]: cluster 2022-04-22T18:24:33.973241+0000 mon.a (mon.0) 4716 : cluster [DBG] osdmap e1901: 8 total, 8 up, 8 in 2022-04-22T18:24:35.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:34 smithi115 bash[46934]: cluster 2022-04-22T18:24:33.973241+0000 mon.a (mon.0) 4716 : cluster [DBG] osdmap e1901: 8 total, 8 up, 8 in 2022-04-22T18:24:35.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:34 smithi115 bash[45495]: cluster 2022-04-22T18:24:33.973241+0000 mon.a (mon.0) 4716 : cluster [DBG] osdmap e1901: 8 total, 8 up, 8 in 2022-04-22T18:24:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:35 smithi121 bash[36902]: cluster 2022-04-22T18:24:34.978455+0000 mon.a (mon.0) 4717 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2022-04-22T18:24:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:35 smithi121 bash[36902]: cluster 2022-04-22T18:24:35.488625+0000 mgr.y (mgr.34107) 2042 : cluster [DBG] pgmap v3379: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2022-04-22T18:24:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:35 smithi115 bash[45495]: cluster 2022-04-22T18:24:34.978455+0000 mon.a (mon.0) 4717 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2022-04-22T18:24:36.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:35 smithi115 bash[45495]: cluster 2022-04-22T18:24:35.488625+0000 mgr.y (mgr.34107) 2042 : cluster [DBG] pgmap v3379: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2022-04-22T18:24:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:35 smithi115 bash[46934]: cluster 2022-04-22T18:24:34.978455+0000 mon.a (mon.0) 4717 : cluster [DBG] osdmap e1902: 8 total, 8 up, 8 in 2022-04-22T18:24:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:35 smithi115 bash[46934]: cluster 2022-04-22T18:24:35.488625+0000 mgr.y (mgr.34107) 2042 : cluster [DBG] pgmap v3379: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.7 KiB/s wr, 1.39k op/s 2022-04-22T18:24:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:37 smithi121 bash[36902]: cluster 2022-04-22T18:24:36.003279+0000 mon.a (mon.0) 4718 : cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2022-04-22T18:24:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:37 smithi121 bash[36902]: audit 2022-04-22T18:24:36.159829+0000 mon.a (mon.0) 4719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:37 smithi121 bash[36902]: audit 2022-04-22T18:24:36.671377+0000 mon.a (mon.0) 4720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[45495]: cluster 2022-04-22T18:24:36.003279+0000 mon.a (mon.0) 4718 : cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2022-04-22T18:24:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[45495]: audit 2022-04-22T18:24:36.159829+0000 mon.a (mon.0) 4719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[45495]: audit 2022-04-22T18:24:36.671377+0000 mon.a (mon.0) 4720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:37.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[46934]: cluster 2022-04-22T18:24:36.003279+0000 mon.a (mon.0) 4718 : cluster [DBG] osdmap e1903: 8 total, 8 up, 8 in 2022-04-22T18:24:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[46934]: audit 2022-04-22T18:24:36.159829+0000 mon.a (mon.0) 4719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:36 smithi115 bash[46934]: audit 2022-04-22T18:24:36.671377+0000 mon.a (mon.0) 4720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:38 smithi121 bash[36902]: cluster 2022-04-22T18:24:37.008278+0000 mon.a (mon.0) 4721 : cluster [DBG] osdmap e1904: 8 total, 8 up, 8 in 2022-04-22T18:24:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:38 smithi121 bash[36902]: cluster 2022-04-22T18:24:37.489436+0000 mgr.y (mgr.34107) 2043 : cluster [DBG] pgmap v3382: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 KiB/s wr, 1.39k op/s 2022-04-22T18:24:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:38 smithi115 bash[45495]: cluster 2022-04-22T18:24:37.008278+0000 mon.a (mon.0) 4721 : cluster [DBG] osdmap e1904: 8 total, 8 up, 8 in 2022-04-22T18:24:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:38 smithi115 bash[45495]: cluster 2022-04-22T18:24:37.489436+0000 mgr.y (mgr.34107) 2043 : cluster [DBG] pgmap v3382: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 KiB/s wr, 1.39k op/s 2022-04-22T18:24:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:38 smithi115 bash[46934]: cluster 2022-04-22T18:24:37.008278+0000 mon.a (mon.0) 4721 : cluster [DBG] osdmap e1904: 8 total, 8 up, 8 in 2022-04-22T18:24:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:38 smithi115 bash[46934]: cluster 2022-04-22T18:24:37.489436+0000 mgr.y (mgr.34107) 2043 : cluster [DBG] pgmap v3382: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.0 KiB/s wr, 1.39k op/s 2022-04-22T18:24:39.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:39 smithi115 bash[45495]: cluster 2022-04-22T18:24:38.016442+0000 mon.a (mon.0) 4722 : cluster [DBG] osdmap e1905: 8 total, 8 up, 8 in 2022-04-22T18:24:39.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:39 smithi115 bash[45495]: audit 2022-04-22T18:24:38.542165+0000 mon.a (mon.0) 4723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:39 smithi115 bash[46934]: cluster 2022-04-22T18:24:38.016442+0000 mon.a (mon.0) 4722 : cluster [DBG] osdmap e1905: 8 total, 8 up, 8 in 2022-04-22T18:24:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:39 smithi115 bash[46934]: audit 2022-04-22T18:24:38.542165+0000 mon.a (mon.0) 4723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:39 smithi121 bash[36902]: cluster 2022-04-22T18:24:38.016442+0000 mon.a (mon.0) 4722 : cluster [DBG] osdmap e1905: 8 total, 8 up, 8 in 2022-04-22T18:24:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:39 smithi121 bash[36902]: audit 2022-04-22T18:24:38.542165+0000 mon.a (mon.0) 4723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:40 smithi121 bash[36902]: cluster 2022-04-22T18:24:39.028222+0000 mon.a (mon.0) 4724 : cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2022-04-22T18:24:40.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:40 smithi121 bash[36902]: cluster 2022-04-22T18:24:39.490241+0000 mgr.y (mgr.34107) 2044 : cluster [DBG] pgmap v3385: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:40 smithi115 bash[45495]: cluster 2022-04-22T18:24:39.028222+0000 mon.a (mon.0) 4724 : cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2022-04-22T18:24:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:40 smithi115 bash[45495]: cluster 2022-04-22T18:24:39.490241+0000 mgr.y (mgr.34107) 2044 : cluster [DBG] pgmap v3385: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:40 smithi115 bash[46934]: cluster 2022-04-22T18:24:39.028222+0000 mon.a (mon.0) 4724 : cluster [DBG] osdmap e1906: 8 total, 8 up, 8 in 2022-04-22T18:24:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:40 smithi115 bash[46934]: cluster 2022-04-22T18:24:39.490241+0000 mgr.y (mgr.34107) 2044 : cluster [DBG] pgmap v3385: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:41.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:41 smithi115 bash[45495]: cluster 2022-04-22T18:24:40.043891+0000 mon.a (mon.0) 4725 : cluster [DBG] osdmap e1907: 8 total, 8 up, 8 in 2022-04-22T18:24:41.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:41 smithi115 bash[46934]: cluster 2022-04-22T18:24:40.043891+0000 mon.a (mon.0) 4725 : cluster [DBG] osdmap e1907: 8 total, 8 up, 8 in 2022-04-22T18:24:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:41 smithi121 bash[36902]: cluster 2022-04-22T18:24:40.043891+0000 mon.a (mon.0) 4725 : cluster [DBG] osdmap e1907: 8 total, 8 up, 8 in 2022-04-22T18:24:42.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:41.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[45495]: cluster 2022-04-22T18:24:41.049357+0000 mon.a (mon.0) 4726 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[45495]: audit 2022-04-22T18:24:41.162912+0000 mon.a (mon.0) 4727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[45495]: cluster 2022-04-22T18:24:41.491014+0000 mgr.y (mgr.34107) 2045 : cluster [DBG] pgmap v3388: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 861 KiB/s rd, 4.0 KiB/s wr, 1.15k op/s 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[45495]: audit 2022-04-22T18:24:41.675162+0000 mon.a (mon.0) 4728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[46934]: cluster 2022-04-22T18:24:41.049357+0000 mon.a (mon.0) 4726 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[46934]: audit 2022-04-22T18:24:41.162912+0000 mon.a (mon.0) 4727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[46934]: cluster 2022-04-22T18:24:41.491014+0000 mgr.y (mgr.34107) 2045 : cluster [DBG] pgmap v3388: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 861 KiB/s rd, 4.0 KiB/s wr, 1.15k op/s 2022-04-22T18:24:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:42 smithi115 bash[46934]: audit 2022-04-22T18:24:41.675162+0000 mon.a (mon.0) 4728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:42 smithi121 bash[36902]: cluster 2022-04-22T18:24:41.049357+0000 mon.a (mon.0) 4726 : cluster [DBG] osdmap e1908: 8 total, 8 up, 8 in 2022-04-22T18:24:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:42 smithi121 bash[36902]: audit 2022-04-22T18:24:41.162912+0000 mon.a (mon.0) 4727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:42 smithi121 bash[36902]: cluster 2022-04-22T18:24:41.491014+0000 mgr.y (mgr.34107) 2045 : cluster [DBG] pgmap v3388: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 861 KiB/s rd, 4.0 KiB/s wr, 1.15k op/s 2022-04-22T18:24:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:42 smithi121 bash[36902]: audit 2022-04-22T18:24:41.675162+0000 mon.a (mon.0) 4728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:43 smithi121 bash[36902]: cluster 2022-04-22T18:24:42.058736+0000 mon.a (mon.0) 4729 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2022-04-22T18:24:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:43 smithi115 bash[45495]: cluster 2022-04-22T18:24:42.058736+0000 mon.a (mon.0) 4729 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2022-04-22T18:24:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:43 smithi115 bash[46934]: cluster 2022-04-22T18:24:42.058736+0000 mon.a (mon.0) 4729 : cluster [DBG] osdmap e1909: 8 total, 8 up, 8 in 2022-04-22T18:24:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:43.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:44 smithi121 bash[36902]: cluster 2022-04-22T18:24:43.069179+0000 mon.a (mon.0) 4730 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2022-04-22T18:24:44.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:44 smithi121 bash[36902]: cluster 2022-04-22T18:24:43.491913+0000 mgr.y (mgr.34107) 2046 : cluster [DBG] pgmap v3391: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 862 KiB/s rd, 3.5 KiB/s wr, 1.15k op/s 2022-04-22T18:24:44.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:44 smithi121 bash[36902]: audit 2022-04-22T18:24:43.545464+0000 mon.a (mon.0) 4731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[45495]: cluster 2022-04-22T18:24:43.069179+0000 mon.a (mon.0) 4730 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2022-04-22T18:24:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[45495]: cluster 2022-04-22T18:24:43.491913+0000 mgr.y (mgr.34107) 2046 : cluster [DBG] pgmap v3391: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 862 KiB/s rd, 3.5 KiB/s wr, 1.15k op/s 2022-04-22T18:24:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[45495]: audit 2022-04-22T18:24:43.545464+0000 mon.a (mon.0) 4731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[46934]: cluster 2022-04-22T18:24:43.069179+0000 mon.a (mon.0) 4730 : cluster [DBG] osdmap e1910: 8 total, 8 up, 8 in 2022-04-22T18:24:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[46934]: cluster 2022-04-22T18:24:43.491913+0000 mgr.y (mgr.34107) 2046 : cluster [DBG] pgmap v3391: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 862 KiB/s rd, 3.5 KiB/s wr, 1.15k op/s 2022-04-22T18:24:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:44 smithi115 bash[46934]: audit 2022-04-22T18:24:43.545464+0000 mon.a (mon.0) 4731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:44.864 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:24:45.216 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:24:45.216 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (47m) 5m ago 61m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:24:45.216 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (61m) 6m ago 61m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (49m) 5m ago 62m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (49m) 6m ago 62m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (57m) 6m ago 66m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (57m) 5m ago 69m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (56m) 5m ago 69m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (55m) 6m ago 67m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (56m) 5m ago 67m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (48m) 5m ago 61m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (48m) 6m ago 61m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (54m) 5m ago 66m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:24:45.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (53m) 5m ago 65m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (53m) 5m ago 65m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (52m) 5m ago 64m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (52m) 6m ago 64m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (51m) 6m ago 64m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (50m) 6m ago 63m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (50m) 6m ago 63m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:24:45.218 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (48m) 6m ago 61m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:24:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:45 smithi121 bash[36902]: cluster 2022-04-22T18:24:44.084111+0000 mon.a (mon.0) 4732 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2022-04-22T18:24:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:45 smithi115 bash[46934]: cluster 2022-04-22T18:24:44.084111+0000 mon.a (mon.0) 4732 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2022-04-22T18:24:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:45 smithi115 bash[45495]: cluster 2022-04-22T18:24:44.084111+0000 mon.a (mon.0) 4732 : cluster [DBG] osdmap e1911: 8 total, 8 up, 8 in 2022-04-22T18:24:45.615 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:45.616 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:24:45.617 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:24:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:46 smithi121 bash[36902]: audit 2022-04-22T18:24:44.841777+0000 mgr.y (mgr.34107) 2047 : audit [DBG] from='client.47440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:46 smithi121 bash[36902]: cluster 2022-04-22T18:24:45.088702+0000 mon.a (mon.0) 4733 : cluster [DBG] osdmap e1912: 8 total, 8 up, 8 in 2022-04-22T18:24:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:46 smithi121 bash[36902]: audit 2022-04-22T18:24:45.190406+0000 mgr.y (mgr.34107) 2048 : audit [DBG] from='client.47446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:46 smithi121 bash[36902]: cluster 2022-04-22T18:24:45.492702+0000 mgr.y (mgr.34107) 2049 : cluster [DBG] pgmap v3394: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:46 smithi121 bash[36902]: audit 2022-04-22T18:24:45.608412+0000 mon.c (mon.1) 125 : audit [DBG] from='client.? 172.21.15.115:0/723596892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[45495]: audit 2022-04-22T18:24:44.841777+0000 mgr.y (mgr.34107) 2047 : audit [DBG] from='client.47440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[45495]: cluster 2022-04-22T18:24:45.088702+0000 mon.a (mon.0) 4733 : cluster [DBG] osdmap e1912: 8 total, 8 up, 8 in 2022-04-22T18:24:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[45495]: audit 2022-04-22T18:24:45.190406+0000 mgr.y (mgr.34107) 2048 : audit [DBG] from='client.47446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[45495]: cluster 2022-04-22T18:24:45.492702+0000 mgr.y (mgr.34107) 2049 : cluster [DBG] pgmap v3394: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:46.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[45495]: audit 2022-04-22T18:24:45.608412+0000 mon.c (mon.1) 125 : audit [DBG] from='client.? 172.21.15.115:0/723596892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[46934]: audit 2022-04-22T18:24:44.841777+0000 mgr.y (mgr.34107) 2047 : audit [DBG] from='client.47440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[46934]: cluster 2022-04-22T18:24:45.088702+0000 mon.a (mon.0) 4733 : cluster [DBG] osdmap e1912: 8 total, 8 up, 8 in 2022-04-22T18:24:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[46934]: audit 2022-04-22T18:24:45.190406+0000 mgr.y (mgr.34107) 2048 : audit [DBG] from='client.47446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:24:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[46934]: cluster 2022-04-22T18:24:45.492702+0000 mgr.y (mgr.34107) 2049 : cluster [DBG] pgmap v3394: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:46 smithi115 bash[46934]: audit 2022-04-22T18:24:45.608412+0000 mon.c (mon.1) 125 : audit [DBG] from='client.? 172.21.15.115:0/723596892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:24:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:47 smithi121 bash[36902]: cluster 2022-04-22T18:24:46.103184+0000 mon.a (mon.0) 4734 : cluster [DBG] osdmap e1913: 8 total, 8 up, 8 in 2022-04-22T18:24:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:47 smithi121 bash[36902]: audit 2022-04-22T18:24:46.166064+0000 mon.a (mon.0) 4735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:47 smithi121 bash[36902]: audit 2022-04-22T18:24:46.678958+0000 mon.a (mon.0) 4736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[45495]: cluster 2022-04-22T18:24:46.103184+0000 mon.a (mon.0) 4734 : cluster [DBG] osdmap e1913: 8 total, 8 up, 8 in 2022-04-22T18:24:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[45495]: audit 2022-04-22T18:24:46.166064+0000 mon.a (mon.0) 4735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[45495]: audit 2022-04-22T18:24:46.678958+0000 mon.a (mon.0) 4736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[46934]: cluster 2022-04-22T18:24:46.103184+0000 mon.a (mon.0) 4734 : cluster [DBG] osdmap e1913: 8 total, 8 up, 8 in 2022-04-22T18:24:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[46934]: audit 2022-04-22T18:24:46.166064+0000 mon.a (mon.0) 4735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:47 smithi115 bash[46934]: audit 2022-04-22T18:24:46.678958+0000 mon.a (mon.0) 4736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:48 smithi121 bash[36902]: cluster 2022-04-22T18:24:47.111794+0000 mon.a (mon.0) 4737 : cluster [DBG] osdmap e1914: 8 total, 8 up, 8 in 2022-04-22T18:24:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:48 smithi121 bash[36902]: cluster 2022-04-22T18:24:47.493474+0000 mgr.y (mgr.34107) 2050 : cluster [DBG] pgmap v3397: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:48 smithi115 bash[45495]: cluster 2022-04-22T18:24:47.111794+0000 mon.a (mon.0) 4737 : cluster [DBG] osdmap e1914: 8 total, 8 up, 8 in 2022-04-22T18:24:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:48 smithi115 bash[45495]: cluster 2022-04-22T18:24:47.493474+0000 mgr.y (mgr.34107) 2050 : cluster [DBG] pgmap v3397: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:48 smithi115 bash[46934]: cluster 2022-04-22T18:24:47.111794+0000 mon.a (mon.0) 4737 : cluster [DBG] osdmap e1914: 8 total, 8 up, 8 in 2022-04-22T18:24:48.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:48 smithi115 bash[46934]: cluster 2022-04-22T18:24:47.493474+0000 mgr.y (mgr.34107) 2050 : cluster [DBG] pgmap v3397: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.0 KiB/s wr, 1.46k op/s 2022-04-22T18:24:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:49 smithi121 bash[36902]: cluster 2022-04-22T18:24:48.119548+0000 mon.a (mon.0) 4738 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2022-04-22T18:24:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:49 smithi121 bash[36902]: audit 2022-04-22T18:24:48.548472+0000 mon.a (mon.0) 4739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:49 smithi115 bash[45495]: cluster 2022-04-22T18:24:48.119548+0000 mon.a (mon.0) 4738 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2022-04-22T18:24:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:49 smithi115 bash[45495]: audit 2022-04-22T18:24:48.548472+0000 mon.a (mon.0) 4739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:49 smithi115 bash[46934]: cluster 2022-04-22T18:24:48.119548+0000 mon.a (mon.0) 4738 : cluster [DBG] osdmap e1915: 8 total, 8 up, 8 in 2022-04-22T18:24:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:49 smithi115 bash[46934]: audit 2022-04-22T18:24:48.548472+0000 mon.a (mon.0) 4739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:50 smithi121 bash[36902]: cluster 2022-04-22T18:24:49.128076+0000 mon.a (mon.0) 4740 : cluster [DBG] osdmap e1916: 8 total, 8 up, 8 in 2022-04-22T18:24:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:50 smithi121 bash[36902]: cluster 2022-04-22T18:24:49.494340+0000 mgr.y (mgr.34107) 2051 : cluster [DBG] pgmap v3400: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:50 smithi115 bash[45495]: cluster 2022-04-22T18:24:49.128076+0000 mon.a (mon.0) 4740 : cluster [DBG] osdmap e1916: 8 total, 8 up, 8 in 2022-04-22T18:24:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:50 smithi115 bash[45495]: cluster 2022-04-22T18:24:49.494340+0000 mgr.y (mgr.34107) 2051 : cluster [DBG] pgmap v3400: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:50 smithi115 bash[46934]: cluster 2022-04-22T18:24:49.128076+0000 mon.a (mon.0) 4740 : cluster [DBG] osdmap e1916: 8 total, 8 up, 8 in 2022-04-22T18:24:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:50 smithi115 bash[46934]: cluster 2022-04-22T18:24:49.494340+0000 mgr.y (mgr.34107) 2051 : cluster [DBG] pgmap v3400: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:24:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:51 smithi121 bash[36902]: cluster 2022-04-22T18:24:50.137500+0000 mon.a (mon.0) 4741 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2022-04-22T18:24:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:51 smithi115 bash[45495]: cluster 2022-04-22T18:24:50.137500+0000 mon.a (mon.0) 4741 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2022-04-22T18:24:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:51 smithi115 bash[46934]: cluster 2022-04-22T18:24:50.137500+0000 mon.a (mon.0) 4741 : cluster [DBG] osdmap e1917: 8 total, 8 up, 8 in 2022-04-22T18:24:52.163 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:51.823Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:52.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[46934]: cluster 2022-04-22T18:24:51.147124+0000 mon.a (mon.0) 4742 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2022-04-22T18:24:52.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[46934]: audit 2022-04-22T18:24:51.168826+0000 mon.a (mon.0) 4743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:52.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[46934]: cluster 2022-04-22T18:24:51.495171+0000 mgr.y (mgr.34107) 2052 : cluster [DBG] pgmap v3403: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 1.51k op/s 2022-04-22T18:24:52.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[46934]: audit 2022-04-22T18:24:51.682665+0000 mon.a (mon.0) 4744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:52.165 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[45495]: cluster 2022-04-22T18:24:51.147124+0000 mon.a (mon.0) 4742 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2022-04-22T18:24:52.165 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[45495]: audit 2022-04-22T18:24:51.168826+0000 mon.a (mon.0) 4743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:52.165 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[45495]: cluster 2022-04-22T18:24:51.495171+0000 mgr.y (mgr.34107) 2052 : cluster [DBG] pgmap v3403: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 1.51k op/s 2022-04-22T18:24:52.165 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:52 smithi115 bash[45495]: audit 2022-04-22T18:24:51.682665+0000 mon.a (mon.0) 4744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:52 smithi121 bash[36902]: cluster 2022-04-22T18:24:51.147124+0000 mon.a (mon.0) 4742 : cluster [DBG] osdmap e1918: 8 total, 8 up, 8 in 2022-04-22T18:24:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:52 smithi121 bash[36902]: audit 2022-04-22T18:24:51.168826+0000 mon.a (mon.0) 4743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:52 smithi121 bash[36902]: cluster 2022-04-22T18:24:51.495171+0000 mgr.y (mgr.34107) 2052 : cluster [DBG] pgmap v3403: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 1.51k op/s 2022-04-22T18:24:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:52 smithi121 bash[36902]: audit 2022-04-22T18:24:51.682665+0000 mon.a (mon.0) 4744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:53 smithi121 bash[36902]: cluster 2022-04-22T18:24:52.149867+0000 mon.a (mon.0) 4745 : cluster [DBG] osdmap e1919: 8 total, 8 up, 8 in 2022-04-22T18:24:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:53 smithi115 bash[45495]: cluster 2022-04-22T18:24:52.149867+0000 mon.a (mon.0) 4745 : cluster [DBG] osdmap e1919: 8 total, 8 up, 8 in 2022-04-22T18:24:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:53 smithi115 bash[46934]: cluster 2022-04-22T18:24:52.149867+0000 mon.a (mon.0) 4745 : cluster [DBG] osdmap e1919: 8 total, 8 up, 8 in 2022-04-22T18:24:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:24:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:24:53.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:24:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:54 smithi121 bash[36902]: cluster 2022-04-22T18:24:53.170921+0000 mon.a (mon.0) 4746 : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2022-04-22T18:24:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:54 smithi121 bash[36902]: cluster 2022-04-22T18:24:53.496135+0000 mgr.y (mgr.34107) 2053 : cluster [DBG] pgmap v3406: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.5 KiB/s wr, 1.51k op/s 2022-04-22T18:24:54.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:54 smithi121 bash[36902]: audit 2022-04-22T18:24:53.551636+0000 mon.a (mon.0) 4747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[45495]: cluster 2022-04-22T18:24:53.170921+0000 mon.a (mon.0) 4746 : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2022-04-22T18:24:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[45495]: cluster 2022-04-22T18:24:53.496135+0000 mgr.y (mgr.34107) 2053 : cluster [DBG] pgmap v3406: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.5 KiB/s wr, 1.51k op/s 2022-04-22T18:24:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[45495]: audit 2022-04-22T18:24:53.551636+0000 mon.a (mon.0) 4747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[46934]: cluster 2022-04-22T18:24:53.170921+0000 mon.a (mon.0) 4746 : cluster [DBG] osdmap e1920: 8 total, 8 up, 8 in 2022-04-22T18:24:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[46934]: cluster 2022-04-22T18:24:53.496135+0000 mgr.y (mgr.34107) 2053 : cluster [DBG] pgmap v3406: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.5 KiB/s wr, 1.51k op/s 2022-04-22T18:24:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:54 smithi115 bash[46934]: audit 2022-04-22T18:24:53.551636+0000 mon.a (mon.0) 4747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:55 smithi121 bash[36902]: cluster 2022-04-22T18:24:54.180734+0000 mon.a (mon.0) 4748 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2022-04-22T18:24:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:55 smithi115 bash[46934]: cluster 2022-04-22T18:24:54.180734+0000 mon.a (mon.0) 4748 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2022-04-22T18:24:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:55 smithi115 bash[45495]: cluster 2022-04-22T18:24:54.180734+0000 mon.a (mon.0) 4748 : cluster [DBG] osdmap e1921: 8 total, 8 up, 8 in 2022-04-22T18:24:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:56 smithi121 bash[36902]: cluster 2022-04-22T18:24:55.189210+0000 mon.a (mon.0) 4749 : cluster [DBG] osdmap e1922: 8 total, 8 up, 8 in 2022-04-22T18:24:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:56 smithi121 bash[36902]: cluster 2022-04-22T18:24:55.497034+0000 mgr.y (mgr.34107) 2054 : cluster [DBG] pgmap v3409: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 KiB/s wr, 1.56k op/s 2022-04-22T18:24:56.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:56 smithi121 bash[36902]: audit 2022-04-22T18:24:56.171703+0000 mon.a (mon.0) 4750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[45495]: cluster 2022-04-22T18:24:55.189210+0000 mon.a (mon.0) 4749 : cluster [DBG] osdmap e1922: 8 total, 8 up, 8 in 2022-04-22T18:24:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[45495]: cluster 2022-04-22T18:24:55.497034+0000 mgr.y (mgr.34107) 2054 : cluster [DBG] pgmap v3409: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 KiB/s wr, 1.56k op/s 2022-04-22T18:24:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[45495]: audit 2022-04-22T18:24:56.171703+0000 mon.a (mon.0) 4750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[46934]: cluster 2022-04-22T18:24:55.189210+0000 mon.a (mon.0) 4749 : cluster [DBG] osdmap e1922: 8 total, 8 up, 8 in 2022-04-22T18:24:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[46934]: cluster 2022-04-22T18:24:55.497034+0000 mgr.y (mgr.34107) 2054 : cluster [DBG] pgmap v3409: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 4.0 KiB/s wr, 1.56k op/s 2022-04-22T18:24:56.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:56 smithi115 bash[46934]: audit 2022-04-22T18:24:56.171703+0000 mon.a (mon.0) 4750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:24:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:57 smithi121 bash[36902]: cluster 2022-04-22T18:24:56.203155+0000 mon.a (mon.0) 4751 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2022-04-22T18:24:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:57 smithi121 bash[36902]: audit 2022-04-22T18:24:56.686461+0000 mon.a (mon.0) 4752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:57 smithi115 bash[45495]: cluster 2022-04-22T18:24:56.203155+0000 mon.a (mon.0) 4751 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2022-04-22T18:24:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:57 smithi115 bash[45495]: audit 2022-04-22T18:24:56.686461+0000 mon.a (mon.0) 4752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:57 smithi115 bash[46934]: cluster 2022-04-22T18:24:56.203155+0000 mon.a (mon.0) 4751 : cluster [DBG] osdmap e1923: 8 total, 8 up, 8 in 2022-04-22T18:24:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:57 smithi115 bash[46934]: audit 2022-04-22T18:24:56.686461+0000 mon.a (mon.0) 4752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:24:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:58 smithi121 bash[36902]: cluster 2022-04-22T18:24:57.207052+0000 mon.a (mon.0) 4753 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2022-04-22T18:24:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:58 smithi121 bash[36902]: cluster 2022-04-22T18:24:57.497852+0000 mgr.y (mgr.34107) 2055 : cluster [DBG] pgmap v3412: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2022-04-22T18:24:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:58 smithi115 bash[45495]: cluster 2022-04-22T18:24:57.207052+0000 mon.a (mon.0) 4753 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2022-04-22T18:24:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:58 smithi115 bash[45495]: cluster 2022-04-22T18:24:57.497852+0000 mgr.y (mgr.34107) 2055 : cluster [DBG] pgmap v3412: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2022-04-22T18:24:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:58 smithi115 bash[46934]: cluster 2022-04-22T18:24:57.207052+0000 mon.a (mon.0) 4753 : cluster [DBG] osdmap e1924: 8 total, 8 up, 8 in 2022-04-22T18:24:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:58 smithi115 bash[46934]: cluster 2022-04-22T18:24:57.497852+0000 mgr.y (mgr.34107) 2055 : cluster [DBG] pgmap v3412: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 KiB/s wr, 1.56k op/s 2022-04-22T18:24:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:59 smithi121 bash[36902]: cluster 2022-04-22T18:24:58.215913+0000 mon.a (mon.0) 4754 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2022-04-22T18:24:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:24:59 smithi121 bash[36902]: audit 2022-04-22T18:24:58.554683+0000 mon.a (mon.0) 4755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:59 smithi115 bash[45495]: cluster 2022-04-22T18:24:58.215913+0000 mon.a (mon.0) 4754 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2022-04-22T18:24:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:24:59 smithi115 bash[45495]: audit 2022-04-22T18:24:58.554683+0000 mon.a (mon.0) 4755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:24:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:59 smithi115 bash[46934]: cluster 2022-04-22T18:24:58.215913+0000 mon.a (mon.0) 4754 : cluster [DBG] osdmap e1925: 8 total, 8 up, 8 in 2022-04-22T18:24:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:24:59 smithi115 bash[46934]: audit 2022-04-22T18:24:58.554683+0000 mon.a (mon.0) 4755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:00.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:00 smithi121 bash[36902]: cluster 2022-04-22T18:24:59.224388+0000 mon.a (mon.0) 4756 : cluster [DBG] osdmap e1926: 8 total, 8 up, 8 in 2022-04-22T18:25:00.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:00 smithi121 bash[36902]: cluster 2022-04-22T18:24:59.498752+0000 mgr.y (mgr.34107) 2056 : cluster [DBG] pgmap v3415: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:00 smithi115 bash[45495]: cluster 2022-04-22T18:24:59.224388+0000 mon.a (mon.0) 4756 : cluster [DBG] osdmap e1926: 8 total, 8 up, 8 in 2022-04-22T18:25:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:00 smithi115 bash[45495]: cluster 2022-04-22T18:24:59.498752+0000 mgr.y (mgr.34107) 2056 : cluster [DBG] pgmap v3415: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:00.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:00 smithi115 bash[46934]: cluster 2022-04-22T18:24:59.224388+0000 mon.a (mon.0) 4756 : cluster [DBG] osdmap e1926: 8 total, 8 up, 8 in 2022-04-22T18:25:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:00 smithi115 bash[46934]: cluster 2022-04-22T18:24:59.498752+0000 mgr.y (mgr.34107) 2056 : cluster [DBG] pgmap v3415: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:01.489 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:01 smithi121 bash[36902]: cluster 2022-04-22T18:25:00.232622+0000 mon.a (mon.0) 4757 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2022-04-22T18:25:01.489 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:01 smithi121 bash[36902]: audit 2022-04-22T18:25:01.174949+0000 mon.a (mon.0) 4758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:01 smithi115 bash[45495]: cluster 2022-04-22T18:25:00.232622+0000 mon.a (mon.0) 4757 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2022-04-22T18:25:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:01 smithi115 bash[45495]: audit 2022-04-22T18:25:01.174949+0000 mon.a (mon.0) 4758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:01 smithi115 bash[46934]: cluster 2022-04-22T18:25:00.232622+0000 mon.a (mon.0) 4757 : cluster [DBG] osdmap e1927: 8 total, 8 up, 8 in 2022-04-22T18:25:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:01 smithi115 bash[46934]: audit 2022-04-22T18:25:01.174949+0000 mon.a (mon.0) 4758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:01.823Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:02 smithi121 bash[36902]: cluster 2022-04-22T18:25:01.239075+0000 mon.a (mon.0) 4759 : cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2022-04-22T18:25:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:02 smithi121 bash[36902]: cluster 2022-04-22T18:25:01.499695+0000 mgr.y (mgr.34107) 2057 : cluster [DBG] pgmap v3418: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.60k op/s 2022-04-22T18:25:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:02 smithi121 bash[36902]: audit 2022-04-22T18:25:01.690646+0000 mon.a (mon.0) 4760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[45495]: cluster 2022-04-22T18:25:01.239075+0000 mon.a (mon.0) 4759 : cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2022-04-22T18:25:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[45495]: cluster 2022-04-22T18:25:01.499695+0000 mgr.y (mgr.34107) 2057 : cluster [DBG] pgmap v3418: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.60k op/s 2022-04-22T18:25:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[45495]: audit 2022-04-22T18:25:01.690646+0000 mon.a (mon.0) 4760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:02.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[46934]: cluster 2022-04-22T18:25:01.239075+0000 mon.a (mon.0) 4759 : cluster [DBG] osdmap e1928: 8 total, 8 up, 8 in 2022-04-22T18:25:02.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[46934]: cluster 2022-04-22T18:25:01.499695+0000 mgr.y (mgr.34107) 2057 : cluster [DBG] pgmap v3418: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.60k op/s 2022-04-22T18:25:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:02 smithi115 bash[46934]: audit 2022-04-22T18:25:01.690646+0000 mon.a (mon.0) 4760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:03 smithi121 bash[36902]: cluster 2022-04-22T18:25:02.252080+0000 mon.a (mon.0) 4761 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2022-04-22T18:25:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:03 smithi115 bash[45495]: cluster 2022-04-22T18:25:02.252080+0000 mon.a (mon.0) 4761 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2022-04-22T18:25:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:03 smithi115 bash[46934]: cluster 2022-04-22T18:25:02.252080+0000 mon.a (mon.0) 4761 : cluster [DBG] osdmap e1929: 8 total, 8 up, 8 in 2022-04-22T18:25:03.942 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:03.704Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: cluster 2022-04-22T18:25:03.259860+0000 mon.a (mon.0) 4762 : cluster [DBG] osdmap e1930: 8 total, 8 up, 8 in 2022-04-22T18:25:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: cluster 2022-04-22T18:25:03.500551+0000 mgr.y (mgr.34107) 2058 : cluster [DBG] pgmap v3421: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.60k op/s 2022-04-22T18:25:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: audit 2022-04-22T18:25:03.557680+0000 mon.a (mon.0) 4763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: audit 2022-04-22T18:25:03.606536+0000 mon.a (mon.0) 4764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: audit 2022-04-22T18:25:03.609494+0000 mon.a (mon.0) 4765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: audit 2022-04-22T18:25:03.611129+0000 mon.a (mon.0) 4766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[45495]: audit 2022-04-22T18:25:03.960452+0000 mon.a (mon.0) 4767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: cluster 2022-04-22T18:25:03.259860+0000 mon.a (mon.0) 4762 : cluster [DBG] osdmap e1930: 8 total, 8 up, 8 in 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: cluster 2022-04-22T18:25:03.500551+0000 mgr.y (mgr.34107) 2058 : cluster [DBG] pgmap v3421: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.60k op/s 2022-04-22T18:25:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: audit 2022-04-22T18:25:03.557680+0000 mon.a (mon.0) 4763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: audit 2022-04-22T18:25:03.606536+0000 mon.a (mon.0) 4764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:25:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: audit 2022-04-22T18:25:03.609494+0000 mon.a (mon.0) 4765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:25:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: audit 2022-04-22T18:25:03.611129+0000 mon.a (mon.0) 4766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:25:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:04 smithi115 bash[46934]: audit 2022-04-22T18:25:03.960452+0000 mon.a (mon.0) 4767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:25:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: cluster 2022-04-22T18:25:03.259860+0000 mon.a (mon.0) 4762 : cluster [DBG] osdmap e1930: 8 total, 8 up, 8 in 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: cluster 2022-04-22T18:25:03.500551+0000 mgr.y (mgr.34107) 2058 : cluster [DBG] pgmap v3421: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.60k op/s 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: audit 2022-04-22T18:25:03.557680+0000 mon.a (mon.0) 4763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: audit 2022-04-22T18:25:03.606536+0000 mon.a (mon.0) 4764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: audit 2022-04-22T18:25:03.609494+0000 mon.a (mon.0) 4765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: audit 2022-04-22T18:25:03.611129+0000 mon.a (mon.0) 4766 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:25:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:04 smithi121 bash[36902]: audit 2022-04-22T18:25:03.960452+0000 mon.a (mon.0) 4767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:25:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:05 smithi115 bash[45495]: cluster 2022-04-22T18:25:04.277122+0000 mon.a (mon.0) 4768 : cluster [DBG] osdmap e1931: 8 total, 8 up, 8 in 2022-04-22T18:25:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:05 smithi115 bash[46934]: cluster 2022-04-22T18:25:04.277122+0000 mon.a (mon.0) 4768 : cluster [DBG] osdmap e1931: 8 total, 8 up, 8 in 2022-04-22T18:25:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:05 smithi121 bash[36902]: cluster 2022-04-22T18:25:04.277122+0000 mon.a (mon.0) 4768 : cluster [DBG] osdmap e1931: 8 total, 8 up, 8 in 2022-04-22T18:25:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[45495]: cluster 2022-04-22T18:25:05.281771+0000 mon.a (mon.0) 4769 : cluster [DBG] osdmap e1932: 8 total, 8 up, 8 in 2022-04-22T18:25:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[45495]: cluster 2022-04-22T18:25:05.501435+0000 mgr.y (mgr.34107) 2059 : cluster [DBG] pgmap v3424: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.65k op/s 2022-04-22T18:25:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[45495]: audit 2022-04-22T18:25:06.178143+0000 mon.a (mon.0) 4770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[46934]: cluster 2022-04-22T18:25:05.281771+0000 mon.a (mon.0) 4769 : cluster [DBG] osdmap e1932: 8 total, 8 up, 8 in 2022-04-22T18:25:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[46934]: cluster 2022-04-22T18:25:05.501435+0000 mgr.y (mgr.34107) 2059 : cluster [DBG] pgmap v3424: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.65k op/s 2022-04-22T18:25:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:06 smithi115 bash[46934]: audit 2022-04-22T18:25:06.178143+0000 mon.a (mon.0) 4770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:06 smithi121 bash[36902]: cluster 2022-04-22T18:25:05.281771+0000 mon.a (mon.0) 4769 : cluster [DBG] osdmap e1932: 8 total, 8 up, 8 in 2022-04-22T18:25:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:06 smithi121 bash[36902]: cluster 2022-04-22T18:25:05.501435+0000 mgr.y (mgr.34107) 2059 : cluster [DBG] pgmap v3424: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.5 KiB/s wr, 1.65k op/s 2022-04-22T18:25:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:06 smithi121 bash[36902]: audit 2022-04-22T18:25:06.178143+0000 mon.a (mon.0) 4770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:07 smithi121 bash[36902]: cluster 2022-04-22T18:25:06.292385+0000 mon.a (mon.0) 4771 : cluster [DBG] osdmap e1933: 8 total, 8 up, 8 in 2022-04-22T18:25:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:07 smithi121 bash[36902]: audit 2022-04-22T18:25:06.694568+0000 mon.a (mon.0) 4772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:07 smithi115 bash[45495]: cluster 2022-04-22T18:25:06.292385+0000 mon.a (mon.0) 4771 : cluster [DBG] osdmap e1933: 8 total, 8 up, 8 in 2022-04-22T18:25:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:07 smithi115 bash[45495]: audit 2022-04-22T18:25:06.694568+0000 mon.a (mon.0) 4772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:07 smithi115 bash[46934]: cluster 2022-04-22T18:25:06.292385+0000 mon.a (mon.0) 4771 : cluster [DBG] osdmap e1933: 8 total, 8 up, 8 in 2022-04-22T18:25:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:07 smithi115 bash[46934]: audit 2022-04-22T18:25:06.694568+0000 mon.a (mon.0) 4772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:08 smithi121 bash[36902]: cluster 2022-04-22T18:25:07.304387+0000 mon.a (mon.0) 4773 : cluster [DBG] osdmap e1934: 8 total, 8 up, 8 in 2022-04-22T18:25:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:08 smithi121 bash[36902]: cluster 2022-04-22T18:25:07.502296+0000 mgr.y (mgr.34107) 2060 : cluster [DBG] pgmap v3427: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.64k op/s 2022-04-22T18:25:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:08 smithi115 bash[45495]: cluster 2022-04-22T18:25:07.304387+0000 mon.a (mon.0) 4773 : cluster [DBG] osdmap e1934: 8 total, 8 up, 8 in 2022-04-22T18:25:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:08 smithi115 bash[45495]: cluster 2022-04-22T18:25:07.502296+0000 mgr.y (mgr.34107) 2060 : cluster [DBG] pgmap v3427: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.64k op/s 2022-04-22T18:25:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:08 smithi115 bash[46934]: cluster 2022-04-22T18:25:07.304387+0000 mon.a (mon.0) 4773 : cluster [DBG] osdmap e1934: 8 total, 8 up, 8 in 2022-04-22T18:25:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:08 smithi115 bash[46934]: cluster 2022-04-22T18:25:07.502296+0000 mgr.y (mgr.34107) 2060 : cluster [DBG] pgmap v3427: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.64k op/s 2022-04-22T18:25:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:09 smithi121 bash[36902]: cluster 2022-04-22T18:25:08.318947+0000 mon.a (mon.0) 4774 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2022-04-22T18:25:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:09 smithi121 bash[36902]: audit 2022-04-22T18:25:08.560658+0000 mon.a (mon.0) 4775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:09 smithi115 bash[45495]: cluster 2022-04-22T18:25:08.318947+0000 mon.a (mon.0) 4774 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2022-04-22T18:25:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:09 smithi115 bash[45495]: audit 2022-04-22T18:25:08.560658+0000 mon.a (mon.0) 4775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:09 smithi115 bash[46934]: cluster 2022-04-22T18:25:08.318947+0000 mon.a (mon.0) 4774 : cluster [DBG] osdmap e1935: 8 total, 8 up, 8 in 2022-04-22T18:25:09.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:09 smithi115 bash[46934]: audit 2022-04-22T18:25:08.560658+0000 mon.a (mon.0) 4775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:10 smithi121 bash[36902]: cluster 2022-04-22T18:25:09.325503+0000 mon.a (mon.0) 4776 : cluster [DBG] osdmap e1936: 8 total, 8 up, 8 in 2022-04-22T18:25:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:10 smithi121 bash[36902]: cluster 2022-04-22T18:25:09.503115+0000 mgr.y (mgr.34107) 2061 : cluster [DBG] pgmap v3430: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:10 smithi115 bash[45495]: cluster 2022-04-22T18:25:09.325503+0000 mon.a (mon.0) 4776 : cluster [DBG] osdmap e1936: 8 total, 8 up, 8 in 2022-04-22T18:25:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:10 smithi115 bash[45495]: cluster 2022-04-22T18:25:09.503115+0000 mgr.y (mgr.34107) 2061 : cluster [DBG] pgmap v3430: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:10 smithi115 bash[46934]: cluster 2022-04-22T18:25:09.325503+0000 mon.a (mon.0) 4776 : cluster [DBG] osdmap e1936: 8 total, 8 up, 8 in 2022-04-22T18:25:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:10 smithi115 bash[46934]: cluster 2022-04-22T18:25:09.503115+0000 mgr.y (mgr.34107) 2061 : cluster [DBG] pgmap v3430: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2022-04-22T18:25:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:11 smithi121 bash[36902]: cluster 2022-04-22T18:25:10.344394+0000 mon.a (mon.0) 4777 : cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2022-04-22T18:25:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:11 smithi121 bash[36902]: audit 2022-04-22T18:25:11.181261+0000 mon.a (mon.0) 4778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:11 smithi115 bash[45495]: cluster 2022-04-22T18:25:10.344394+0000 mon.a (mon.0) 4777 : cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2022-04-22T18:25:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:11 smithi115 bash[45495]: audit 2022-04-22T18:25:11.181261+0000 mon.a (mon.0) 4778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:11 smithi115 bash[46934]: cluster 2022-04-22T18:25:10.344394+0000 mon.a (mon.0) 4777 : cluster [DBG] osdmap e1937: 8 total, 8 up, 8 in 2022-04-22T18:25:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:11 smithi115 bash[46934]: audit 2022-04-22T18:25:11.181261+0000 mon.a (mon.0) 4778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:11.825Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:12 smithi121 bash[36902]: cluster 2022-04-22T18:25:11.351222+0000 mon.a (mon.0) 4779 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2022-04-22T18:25:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:12 smithi121 bash[36902]: cluster 2022-04-22T18:25:11.503888+0000 mgr.y (mgr.34107) 2062 : cluster [DBG] pgmap v3433: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.69k op/s 2022-04-22T18:25:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:12 smithi121 bash[36902]: audit 2022-04-22T18:25:11.698349+0000 mon.a (mon.0) 4780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[45495]: cluster 2022-04-22T18:25:11.351222+0000 mon.a (mon.0) 4779 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2022-04-22T18:25:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[45495]: cluster 2022-04-22T18:25:11.503888+0000 mgr.y (mgr.34107) 2062 : cluster [DBG] pgmap v3433: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.69k op/s 2022-04-22T18:25:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[45495]: audit 2022-04-22T18:25:11.698349+0000 mon.a (mon.0) 4780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:12.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[46934]: cluster 2022-04-22T18:25:11.351222+0000 mon.a (mon.0) 4779 : cluster [DBG] osdmap e1938: 8 total, 8 up, 8 in 2022-04-22T18:25:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[46934]: cluster 2022-04-22T18:25:11.503888+0000 mgr.y (mgr.34107) 2062 : cluster [DBG] pgmap v3433: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 KiB/s wr, 1.69k op/s 2022-04-22T18:25:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:12 smithi115 bash[46934]: audit 2022-04-22T18:25:11.698349+0000 mon.a (mon.0) 4780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:13.710 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:13 smithi115 bash[45495]: cluster 2022-04-22T18:25:12.356273+0000 mon.a (mon.0) 4781 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2022-04-22T18:25:13.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:13 smithi115 bash[46934]: cluster 2022-04-22T18:25:12.356273+0000 mon.a (mon.0) 4781 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2022-04-22T18:25:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:13 smithi121 bash[36902]: cluster 2022-04-22T18:25:12.356273+0000 mon.a (mon.0) 4781 : cluster [DBG] osdmap e1939: 8 total, 8 up, 8 in 2022-04-22T18:25:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:13.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:14 smithi121 bash[36902]: cluster 2022-04-22T18:25:13.376711+0000 mon.a (mon.0) 4782 : cluster [DBG] osdmap e1940: 8 total, 8 up, 8 in 2022-04-22T18:25:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:14 smithi121 bash[36902]: cluster 2022-04-22T18:25:13.504726+0000 mgr.y (mgr.34107) 2063 : cluster [DBG] pgmap v3436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:14 smithi121 bash[36902]: audit 2022-04-22T18:25:13.563790+0000 mon.a (mon.0) 4783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:14 smithi121 bash[36902]: audit 2022-04-22T18:25:13.954980+0000 mon.a (mon.0) 4784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:25:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:14 smithi121 bash[36902]: audit 2022-04-22T18:25:13.955551+0000 mon.a (mon.0) 4785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:25:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[46934]: cluster 2022-04-22T18:25:13.376711+0000 mon.a (mon.0) 4782 : cluster [DBG] osdmap e1940: 8 total, 8 up, 8 in 2022-04-22T18:25:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[46934]: cluster 2022-04-22T18:25:13.504726+0000 mgr.y (mgr.34107) 2063 : cluster [DBG] pgmap v3436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[46934]: audit 2022-04-22T18:25:13.563790+0000 mon.a (mon.0) 4783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[46934]: audit 2022-04-22T18:25:13.954980+0000 mon.a (mon.0) 4784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:25:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[46934]: audit 2022-04-22T18:25:13.955551+0000 mon.a (mon.0) 4785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:25:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[45495]: cluster 2022-04-22T18:25:13.376711+0000 mon.a (mon.0) 4782 : cluster [DBG] osdmap e1940: 8 total, 8 up, 8 in 2022-04-22T18:25:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[45495]: cluster 2022-04-22T18:25:13.504726+0000 mgr.y (mgr.34107) 2063 : cluster [DBG] pgmap v3436: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[45495]: audit 2022-04-22T18:25:13.563790+0000 mon.a (mon.0) 4783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[45495]: audit 2022-04-22T18:25:13.954980+0000 mon.a (mon.0) 4784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:25:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:14 smithi115 bash[45495]: audit 2022-04-22T18:25:13.955551+0000 mon.a (mon.0) 4785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:25:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:15 smithi121 bash[36902]: cluster 2022-04-22T18:25:14.388708+0000 mon.a (mon.0) 4786 : cluster [DBG] osdmap e1941: 8 total, 8 up, 8 in 2022-04-22T18:25:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:15 smithi115 bash[45495]: cluster 2022-04-22T18:25:14.388708+0000 mon.a (mon.0) 4786 : cluster [DBG] osdmap e1941: 8 total, 8 up, 8 in 2022-04-22T18:25:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:15 smithi115 bash[46934]: cluster 2022-04-22T18:25:14.388708+0000 mon.a (mon.0) 4786 : cluster [DBG] osdmap e1941: 8 total, 8 up, 8 in 2022-04-22T18:25:15.992 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:25:16.332 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:25:16.332 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (48m) 6m ago 62m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (61m) 7m ago 61m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (50m) 6m ago 63m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (49m) 7m ago 63m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (57m) 7m ago 67m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (58m) 6m ago 69m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (57m) 6m ago 69m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (56m) 7m ago 67m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (56m) 6m ago 67m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:25:16.333 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (49m) 6m ago 62m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (49m) 7m ago 62m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (55m) 6m ago 67m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (54m) 6m ago 66m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (53m) 6m ago 66m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (53m) 6m ago 65m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (52m) 7m ago 65m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (52m) 7m ago 64m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (51m) 7m ago 64m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (50m) 7m ago 63m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:25:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (49m) 7m ago 62m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:25:16.745 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:25:16.745 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:25:16.745 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:25:16.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:16.746 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:25:16.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:25:16.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:25:16.747 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:25:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:16 smithi121 bash[36902]: cluster 2022-04-22T18:25:15.390419+0000 mon.a (mon.0) 4787 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2022-04-22T18:25:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:16 smithi121 bash[36902]: cluster 2022-04-22T18:25:15.505478+0000 mgr.y (mgr.34107) 2064 : cluster [DBG] pgmap v3439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2022-04-22T18:25:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:16 smithi121 bash[36902]: audit 2022-04-22T18:25:16.184395+0000 mon.a (mon.0) 4788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[45495]: cluster 2022-04-22T18:25:15.390419+0000 mon.a (mon.0) 4787 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2022-04-22T18:25:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[45495]: cluster 2022-04-22T18:25:15.505478+0000 mgr.y (mgr.34107) 2064 : cluster [DBG] pgmap v3439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2022-04-22T18:25:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[45495]: audit 2022-04-22T18:25:16.184395+0000 mon.a (mon.0) 4788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:16.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[46934]: cluster 2022-04-22T18:25:15.390419+0000 mon.a (mon.0) 4787 : cluster [DBG] osdmap e1942: 8 total, 8 up, 8 in 2022-04-22T18:25:16.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[46934]: cluster 2022-04-22T18:25:15.505478+0000 mgr.y (mgr.34107) 2064 : cluster [DBG] pgmap v3439: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 KiB/s wr, 1.74k op/s 2022-04-22T18:25:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:16 smithi115 bash[46934]: audit 2022-04-22T18:25:16.184395+0000 mon.a (mon.0) 4788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:17 smithi121 bash[36902]: audit 2022-04-22T18:25:15.969815+0000 mgr.y (mgr.34107) 2065 : audit [DBG] from='client.37527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:17 smithi121 bash[36902]: audit 2022-04-22T18:25:16.310511+0000 mgr.y (mgr.34107) 2066 : audit [DBG] from='client.37533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:17 smithi121 bash[36902]: cluster 2022-04-22T18:25:16.398386+0000 mon.a (mon.0) 4789 : cluster [DBG] osdmap e1943: 8 total, 8 up, 8 in 2022-04-22T18:25:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:17 smithi121 bash[36902]: audit 2022-04-22T18:25:16.701873+0000 mon.a (mon.0) 4790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:17 smithi121 bash[36902]: audit 2022-04-22T18:25:16.738145+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.115:0/3626416840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[45495]: audit 2022-04-22T18:25:15.969815+0000 mgr.y (mgr.34107) 2065 : audit [DBG] from='client.37527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[45495]: audit 2022-04-22T18:25:16.310511+0000 mgr.y (mgr.34107) 2066 : audit [DBG] from='client.37533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[45495]: cluster 2022-04-22T18:25:16.398386+0000 mon.a (mon.0) 4789 : cluster [DBG] osdmap e1943: 8 total, 8 up, 8 in 2022-04-22T18:25:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[45495]: audit 2022-04-22T18:25:16.701873+0000 mon.a (mon.0) 4790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[45495]: audit 2022-04-22T18:25:16.738145+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.115:0/3626416840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[46934]: audit 2022-04-22T18:25:15.969815+0000 mgr.y (mgr.34107) 2065 : audit [DBG] from='client.37527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[46934]: audit 2022-04-22T18:25:16.310511+0000 mgr.y (mgr.34107) 2066 : audit [DBG] from='client.37533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[46934]: cluster 2022-04-22T18:25:16.398386+0000 mon.a (mon.0) 4789 : cluster [DBG] osdmap e1943: 8 total, 8 up, 8 in 2022-04-22T18:25:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[46934]: audit 2022-04-22T18:25:16.701873+0000 mon.a (mon.0) 4790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:17 smithi115 bash[46934]: audit 2022-04-22T18:25:16.738145+0000 mon.c (mon.1) 126 : audit [DBG] from='client.? 172.21.15.115:0/3626416840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:18 smithi121 bash[36902]: cluster 2022-04-22T18:25:17.475923+0000 mon.a (mon.0) 4791 : cluster [DBG] osdmap e1944: 8 total, 8 up, 8 in 2022-04-22T18:25:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:18 smithi121 bash[36902]: cluster 2022-04-22T18:25:17.506131+0000 mgr.y (mgr.34107) 2067 : cluster [DBG] pgmap v3442: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 1.73k op/s 2022-04-22T18:25:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:18 smithi115 bash[45495]: cluster 2022-04-22T18:25:17.475923+0000 mon.a (mon.0) 4791 : cluster [DBG] osdmap e1944: 8 total, 8 up, 8 in 2022-04-22T18:25:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:18 smithi115 bash[45495]: cluster 2022-04-22T18:25:17.506131+0000 mgr.y (mgr.34107) 2067 : cluster [DBG] pgmap v3442: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 1.73k op/s 2022-04-22T18:25:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:18 smithi115 bash[46934]: cluster 2022-04-22T18:25:17.475923+0000 mon.a (mon.0) 4791 : cluster [DBG] osdmap e1944: 8 total, 8 up, 8 in 2022-04-22T18:25:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:18 smithi115 bash[46934]: cluster 2022-04-22T18:25:17.506131+0000 mgr.y (mgr.34107) 2067 : cluster [DBG] pgmap v3442: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 1.73k op/s 2022-04-22T18:25:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:19 smithi121 bash[36902]: cluster 2022-04-22T18:25:18.490538+0000 mon.a (mon.0) 4792 : cluster [DBG] osdmap e1945: 8 total, 8 up, 8 in 2022-04-22T18:25:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:19 smithi121 bash[36902]: audit 2022-04-22T18:25:18.566528+0000 mon.a (mon.0) 4793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:19 smithi115 bash[45495]: cluster 2022-04-22T18:25:18.490538+0000 mon.a (mon.0) 4792 : cluster [DBG] osdmap e1945: 8 total, 8 up, 8 in 2022-04-22T18:25:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:19 smithi115 bash[45495]: audit 2022-04-22T18:25:18.566528+0000 mon.a (mon.0) 4793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:19 smithi115 bash[46934]: cluster 2022-04-22T18:25:18.490538+0000 mon.a (mon.0) 4792 : cluster [DBG] osdmap e1945: 8 total, 8 up, 8 in 2022-04-22T18:25:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:19 smithi115 bash[46934]: audit 2022-04-22T18:25:18.566528+0000 mon.a (mon.0) 4793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:20 smithi121 bash[36902]: cluster 2022-04-22T18:25:19.496313+0000 mon.a (mon.0) 4794 : cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2022-04-22T18:25:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:20 smithi121 bash[36902]: cluster 2022-04-22T18:25:19.506738+0000 mgr.y (mgr.34107) 2068 : cluster [DBG] pgmap v3445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2022-04-22T18:25:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:20 smithi115 bash[45495]: cluster 2022-04-22T18:25:19.496313+0000 mon.a (mon.0) 4794 : cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2022-04-22T18:25:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:20 smithi115 bash[45495]: cluster 2022-04-22T18:25:19.506738+0000 mgr.y (mgr.34107) 2068 : cluster [DBG] pgmap v3445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2022-04-22T18:25:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:20 smithi115 bash[46934]: cluster 2022-04-22T18:25:19.496313+0000 mon.a (mon.0) 4794 : cluster [DBG] osdmap e1946: 8 total, 8 up, 8 in 2022-04-22T18:25:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:20 smithi115 bash[46934]: cluster 2022-04-22T18:25:19.506738+0000 mgr.y (mgr.34107) 2068 : cluster [DBG] pgmap v3445: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2022-04-22T18:25:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:21 smithi121 bash[36902]: cluster 2022-04-22T18:25:20.512417+0000 mon.a (mon.0) 4795 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2022-04-22T18:25:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:21 smithi121 bash[36902]: audit 2022-04-22T18:25:21.187324+0000 mon.a (mon.0) 4796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:21 smithi115 bash[45495]: cluster 2022-04-22T18:25:20.512417+0000 mon.a (mon.0) 4795 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2022-04-22T18:25:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:21 smithi115 bash[45495]: audit 2022-04-22T18:25:21.187324+0000 mon.a (mon.0) 4796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:21 smithi115 bash[46934]: cluster 2022-04-22T18:25:20.512417+0000 mon.a (mon.0) 4795 : cluster [DBG] osdmap e1947: 8 total, 8 up, 8 in 2022-04-22T18:25:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:21 smithi115 bash[46934]: audit 2022-04-22T18:25:21.187324+0000 mon.a (mon.0) 4796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:21.826Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[45495]: cluster 2022-04-22T18:25:21.507665+0000 mgr.y (mgr.34107) 2069 : cluster [DBG] pgmap v3447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.0 KiB/s wr, 1.76k op/s 2022-04-22T18:25:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[45495]: cluster 2022-04-22T18:25:21.514802+0000 mon.a (mon.0) 4797 : cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2022-04-22T18:25:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[45495]: audit 2022-04-22T18:25:21.705508+0000 mon.a (mon.0) 4798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[46934]: cluster 2022-04-22T18:25:21.507665+0000 mgr.y (mgr.34107) 2069 : cluster [DBG] pgmap v3447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.0 KiB/s wr, 1.76k op/s 2022-04-22T18:25:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[46934]: cluster 2022-04-22T18:25:21.514802+0000 mon.a (mon.0) 4797 : cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2022-04-22T18:25:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:22 smithi115 bash[46934]: audit 2022-04-22T18:25:21.705508+0000 mon.a (mon.0) 4798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:22 smithi121 bash[36902]: cluster 2022-04-22T18:25:21.507665+0000 mgr.y (mgr.34107) 2069 : cluster [DBG] pgmap v3447: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.0 KiB/s wr, 1.76k op/s 2022-04-22T18:25:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:22 smithi121 bash[36902]: cluster 2022-04-22T18:25:21.514802+0000 mon.a (mon.0) 4797 : cluster [DBG] osdmap e1948: 8 total, 8 up, 8 in 2022-04-22T18:25:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:22 smithi121 bash[36902]: audit 2022-04-22T18:25:21.705508+0000 mon.a (mon.0) 4798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:23 smithi115 bash[45495]: cluster 2022-04-22T18:25:22.527954+0000 mon.a (mon.0) 4799 : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2022-04-22T18:25:23.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:23.704Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:23 smithi115 bash[46934]: cluster 2022-04-22T18:25:22.527954+0000 mon.a (mon.0) 4799 : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2022-04-22T18:25:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:23 smithi121 bash[36902]: cluster 2022-04-22T18:25:22.527954+0000 mon.a (mon.0) 4799 : cluster [DBG] osdmap e1949: 8 total, 8 up, 8 in 2022-04-22T18:25:24.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[45495]: cluster 2022-04-22T18:25:23.508765+0000 mgr.y (mgr.34107) 2070 : cluster [DBG] pgmap v3450: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 KiB/s wr, 1.78k op/s 2022-04-22T18:25:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[45495]: cluster 2022-04-22T18:25:23.542125+0000 mon.a (mon.0) 4800 : cluster [DBG] osdmap e1950: 8 total, 8 up, 8 in 2022-04-22T18:25:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[45495]: audit 2022-04-22T18:25:23.569474+0000 mon.a (mon.0) 4801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[46934]: cluster 2022-04-22T18:25:23.508765+0000 mgr.y (mgr.34107) 2070 : cluster [DBG] pgmap v3450: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 KiB/s wr, 1.78k op/s 2022-04-22T18:25:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[46934]: cluster 2022-04-22T18:25:23.542125+0000 mon.a (mon.0) 4800 : cluster [DBG] osdmap e1950: 8 total, 8 up, 8 in 2022-04-22T18:25:24.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:24 smithi115 bash[46934]: audit 2022-04-22T18:25:23.569474+0000 mon.a (mon.0) 4801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:24 smithi121 bash[36902]: cluster 2022-04-22T18:25:23.508765+0000 mgr.y (mgr.34107) 2070 : cluster [DBG] pgmap v3450: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 KiB/s wr, 1.78k op/s 2022-04-22T18:25:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:24 smithi121 bash[36902]: cluster 2022-04-22T18:25:23.542125+0000 mon.a (mon.0) 4800 : cluster [DBG] osdmap e1950: 8 total, 8 up, 8 in 2022-04-22T18:25:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:24 smithi121 bash[36902]: audit 2022-04-22T18:25:23.569474+0000 mon.a (mon.0) 4801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:25 smithi121 bash[36902]: cluster 2022-04-22T18:25:24.549401+0000 mon.a (mon.0) 4802 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2022-04-22T18:25:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:25 smithi115 bash[45495]: cluster 2022-04-22T18:25:24.549401+0000 mon.a (mon.0) 4802 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2022-04-22T18:25:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:25 smithi115 bash[46934]: cluster 2022-04-22T18:25:24.549401+0000 mon.a (mon.0) 4802 : cluster [DBG] osdmap e1951: 8 total, 8 up, 8 in 2022-04-22T18:25:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:26 smithi121 bash[36902]: cluster 2022-04-22T18:25:25.509835+0000 mgr.y (mgr.34107) 2071 : cluster [DBG] pgmap v3453: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 KiB/s wr, 1.78k op/s 2022-04-22T18:25:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:26 smithi121 bash[36902]: cluster 2022-04-22T18:25:25.557439+0000 mon.a (mon.0) 4803 : cluster [DBG] osdmap e1952: 8 total, 8 up, 8 in 2022-04-22T18:25:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:26 smithi121 bash[36902]: audit 2022-04-22T18:25:26.190781+0000 mon.a (mon.0) 4804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:26 smithi121 bash[36902]: cluster 2022-04-22T18:25:26.477137+0000 mon.a (mon.0) 4805 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2022-04-22T18:25:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[46934]: cluster 2022-04-22T18:25:25.509835+0000 mgr.y (mgr.34107) 2071 : cluster [DBG] pgmap v3453: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 KiB/s wr, 1.78k op/s 2022-04-22T18:25:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[46934]: cluster 2022-04-22T18:25:25.557439+0000 mon.a (mon.0) 4803 : cluster [DBG] osdmap e1952: 8 total, 8 up, 8 in 2022-04-22T18:25:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[46934]: audit 2022-04-22T18:25:26.190781+0000 mon.a (mon.0) 4804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[46934]: cluster 2022-04-22T18:25:26.477137+0000 mon.a (mon.0) 4805 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2022-04-22T18:25:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[45495]: cluster 2022-04-22T18:25:25.509835+0000 mgr.y (mgr.34107) 2071 : cluster [DBG] pgmap v3453: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 KiB/s wr, 1.78k op/s 2022-04-22T18:25:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[45495]: cluster 2022-04-22T18:25:25.557439+0000 mon.a (mon.0) 4803 : cluster [DBG] osdmap e1952: 8 total, 8 up, 8 in 2022-04-22T18:25:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[45495]: audit 2022-04-22T18:25:26.190781+0000 mon.a (mon.0) 4804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:26 smithi115 bash[45495]: cluster 2022-04-22T18:25:26.477137+0000 mon.a (mon.0) 4805 : cluster [DBG] osdmap e1953: 8 total, 8 up, 8 in 2022-04-22T18:25:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:27 smithi121 bash[36902]: audit 2022-04-22T18:25:26.709585+0000 mon.a (mon.0) 4806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:27 smithi121 bash[36902]: cluster 2022-04-22T18:25:27.493990+0000 mon.a (mon.0) 4807 : cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2022-04-22T18:25:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:27 smithi115 bash[45495]: audit 2022-04-22T18:25:26.709585+0000 mon.a (mon.0) 4806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:27 smithi115 bash[45495]: cluster 2022-04-22T18:25:27.493990+0000 mon.a (mon.0) 4807 : cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2022-04-22T18:25:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:27 smithi115 bash[46934]: audit 2022-04-22T18:25:26.709585+0000 mon.a (mon.0) 4806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:28.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:27 smithi115 bash[46934]: cluster 2022-04-22T18:25:27.493990+0000 mon.a (mon.0) 4807 : cluster [DBG] osdmap e1954: 8 total, 8 up, 8 in 2022-04-22T18:25:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:28 smithi121 bash[36902]: cluster 2022-04-22T18:25:27.510470+0000 mgr.y (mgr.34107) 2072 : cluster [DBG] pgmap v3457: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.79k op/s 2022-04-22T18:25:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:28 smithi115 bash[45495]: cluster 2022-04-22T18:25:27.510470+0000 mgr.y (mgr.34107) 2072 : cluster [DBG] pgmap v3457: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.79k op/s 2022-04-22T18:25:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:28 smithi115 bash[46934]: cluster 2022-04-22T18:25:27.510470+0000 mgr.y (mgr.34107) 2072 : cluster [DBG] pgmap v3457: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.79k op/s 2022-04-22T18:25:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:29 smithi121 bash[36902]: cluster 2022-04-22T18:25:28.576251+0000 mon.a (mon.0) 4808 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2022-04-22T18:25:30.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:29 smithi121 bash[36902]: audit 2022-04-22T18:25:28.577020+0000 mon.a (mon.0) 4809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:29 smithi115 bash[45495]: cluster 2022-04-22T18:25:28.576251+0000 mon.a (mon.0) 4808 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2022-04-22T18:25:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:29 smithi115 bash[45495]: audit 2022-04-22T18:25:28.577020+0000 mon.a (mon.0) 4809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:29 smithi115 bash[46934]: cluster 2022-04-22T18:25:28.576251+0000 mon.a (mon.0) 4808 : cluster [DBG] osdmap e1955: 8 total, 8 up, 8 in 2022-04-22T18:25:30.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:29 smithi115 bash[46934]: audit 2022-04-22T18:25:28.577020+0000 mon.a (mon.0) 4809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:30 smithi121 bash[36902]: cluster 2022-04-22T18:25:29.511601+0000 mgr.y (mgr.34107) 2073 : cluster [DBG] pgmap v3459: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:30 smithi121 bash[36902]: cluster 2022-04-22T18:25:29.588018+0000 mon.a (mon.0) 4810 : cluster [DBG] osdmap e1956: 8 total, 8 up, 8 in 2022-04-22T18:25:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:30 smithi115 bash[45495]: cluster 2022-04-22T18:25:29.511601+0000 mgr.y (mgr.34107) 2073 : cluster [DBG] pgmap v3459: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:30 smithi115 bash[45495]: cluster 2022-04-22T18:25:29.588018+0000 mon.a (mon.0) 4810 : cluster [DBG] osdmap e1956: 8 total, 8 up, 8 in 2022-04-22T18:25:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:30 smithi115 bash[46934]: cluster 2022-04-22T18:25:29.511601+0000 mgr.y (mgr.34107) 2073 : cluster [DBG] pgmap v3459: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:30 smithi115 bash[46934]: cluster 2022-04-22T18:25:29.588018+0000 mon.a (mon.0) 4810 : cluster [DBG] osdmap e1956: 8 total, 8 up, 8 in 2022-04-22T18:25:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:31 smithi121 bash[36902]: cluster 2022-04-22T18:25:30.593676+0000 mon.a (mon.0) 4811 : cluster [DBG] osdmap e1957: 8 total, 8 up, 8 in 2022-04-22T18:25:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:31 smithi121 bash[36902]: audit 2022-04-22T18:25:31.193810+0000 mon.a (mon.0) 4812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:31 smithi121 bash[36902]: cluster 2022-04-22T18:25:31.477379+0000 mon.a (mon.0) 4813 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2022-04-22T18:25:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:31.826Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[45495]: cluster 2022-04-22T18:25:30.593676+0000 mon.a (mon.0) 4811 : cluster [DBG] osdmap e1957: 8 total, 8 up, 8 in 2022-04-22T18:25:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[45495]: audit 2022-04-22T18:25:31.193810+0000 mon.a (mon.0) 4812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[45495]: cluster 2022-04-22T18:25:31.477379+0000 mon.a (mon.0) 4813 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2022-04-22T18:25:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[46934]: cluster 2022-04-22T18:25:30.593676+0000 mon.a (mon.0) 4811 : cluster [DBG] osdmap e1957: 8 total, 8 up, 8 in 2022-04-22T18:25:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[46934]: audit 2022-04-22T18:25:31.193810+0000 mon.a (mon.0) 4812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:31 smithi115 bash[46934]: cluster 2022-04-22T18:25:31.477379+0000 mon.a (mon.0) 4813 : cluster [DBG] osdmap e1958: 8 total, 8 up, 8 in 2022-04-22T18:25:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:32 smithi121 bash[36902]: cluster 2022-04-22T18:25:31.512368+0000 mgr.y (mgr.34107) 2074 : cluster [DBG] pgmap v3463: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2022-04-22T18:25:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:32 smithi121 bash[36902]: audit 2022-04-22T18:25:31.713375+0000 mon.a (mon.0) 4814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:32 smithi121 bash[36902]: cluster 2022-04-22T18:25:32.497576+0000 mon.a (mon.0) 4815 : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2022-04-22T18:25:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[45495]: cluster 2022-04-22T18:25:31.512368+0000 mgr.y (mgr.34107) 2074 : cluster [DBG] pgmap v3463: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2022-04-22T18:25:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[45495]: audit 2022-04-22T18:25:31.713375+0000 mon.a (mon.0) 4814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[45495]: cluster 2022-04-22T18:25:32.497576+0000 mon.a (mon.0) 4815 : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2022-04-22T18:25:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[46934]: cluster 2022-04-22T18:25:31.512368+0000 mgr.y (mgr.34107) 2074 : cluster [DBG] pgmap v3463: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.0 KiB/s wr, 1.73k op/s 2022-04-22T18:25:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[46934]: audit 2022-04-22T18:25:31.713375+0000 mon.a (mon.0) 4814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:32 smithi115 bash[46934]: cluster 2022-04-22T18:25:32.497576+0000 mon.a (mon.0) 4815 : cluster [DBG] osdmap e1959: 8 total, 8 up, 8 in 2022-04-22T18:25:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:33.705Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:34 smithi121 bash[36902]: cluster 2022-04-22T18:25:33.500501+0000 mon.a (mon.0) 4816 : cluster [DBG] osdmap e1960: 8 total, 8 up, 8 in 2022-04-22T18:25:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:34 smithi121 bash[36902]: cluster 2022-04-22T18:25:33.512954+0000 mgr.y (mgr.34107) 2075 : cluster [DBG] pgmap v3466: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.1 KiB/s wr, 1.77k op/s 2022-04-22T18:25:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:34 smithi121 bash[36902]: audit 2022-04-22T18:25:33.575677+0000 mon.a (mon.0) 4817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:34.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[45495]: cluster 2022-04-22T18:25:33.500501+0000 mon.a (mon.0) 4816 : cluster [DBG] osdmap e1960: 8 total, 8 up, 8 in 2022-04-22T18:25:34.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[45495]: cluster 2022-04-22T18:25:33.512954+0000 mgr.y (mgr.34107) 2075 : cluster [DBG] pgmap v3466: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.1 KiB/s wr, 1.77k op/s 2022-04-22T18:25:34.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[45495]: audit 2022-04-22T18:25:33.575677+0000 mon.a (mon.0) 4817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[46934]: cluster 2022-04-22T18:25:33.500501+0000 mon.a (mon.0) 4816 : cluster [DBG] osdmap e1960: 8 total, 8 up, 8 in 2022-04-22T18:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[46934]: cluster 2022-04-22T18:25:33.512954+0000 mgr.y (mgr.34107) 2075 : cluster [DBG] pgmap v3466: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.1 KiB/s wr, 1.77k op/s 2022-04-22T18:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:34 smithi115 bash[46934]: audit 2022-04-22T18:25:33.575677+0000 mon.a (mon.0) 4817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:35 smithi121 bash[36902]: cluster 2022-04-22T18:25:34.505689+0000 mon.a (mon.0) 4818 : cluster [DBG] osdmap e1961: 8 total, 8 up, 8 in 2022-04-22T18:25:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:35 smithi115 bash[45495]: cluster 2022-04-22T18:25:34.505689+0000 mon.a (mon.0) 4818 : cluster [DBG] osdmap e1961: 8 total, 8 up, 8 in 2022-04-22T18:25:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:35 smithi115 bash[46934]: cluster 2022-04-22T18:25:34.505689+0000 mon.a (mon.0) 4818 : cluster [DBG] osdmap e1961: 8 total, 8 up, 8 in 2022-04-22T18:25:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[45495]: cluster 2022-04-22T18:25:35.513541+0000 mgr.y (mgr.34107) 2076 : cluster [DBG] pgmap v3469: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[45495]: cluster 2022-04-22T18:25:35.513844+0000 mon.a (mon.0) 4819 : cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2022-04-22T18:25:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[45495]: audit 2022-04-22T18:25:36.197080+0000 mon.a (mon.0) 4820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[45495]: cluster 2022-04-22T18:25:36.480145+0000 mon.a (mon.0) 4821 : cluster [DBG] osdmap e1963: 8 total, 8 up, 8 in 2022-04-22T18:25:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[46934]: cluster 2022-04-22T18:25:35.513541+0000 mgr.y (mgr.34107) 2076 : cluster [DBG] pgmap v3469: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[46934]: cluster 2022-04-22T18:25:35.513844+0000 mon.a (mon.0) 4819 : cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2022-04-22T18:25:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[46934]: audit 2022-04-22T18:25:36.197080+0000 mon.a (mon.0) 4820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:36 smithi115 bash[46934]: cluster 2022-04-22T18:25:36.480145+0000 mon.a (mon.0) 4821 : cluster [DBG] osdmap e1963: 8 total, 8 up, 8 in 2022-04-22T18:25:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:36 smithi121 bash[36902]: cluster 2022-04-22T18:25:35.513541+0000 mgr.y (mgr.34107) 2076 : cluster [DBG] pgmap v3469: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.69k op/s 2022-04-22T18:25:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:36 smithi121 bash[36902]: cluster 2022-04-22T18:25:35.513844+0000 mon.a (mon.0) 4819 : cluster [DBG] osdmap e1962: 8 total, 8 up, 8 in 2022-04-22T18:25:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:36 smithi121 bash[36902]: audit 2022-04-22T18:25:36.197080+0000 mon.a (mon.0) 4820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:36 smithi121 bash[36902]: cluster 2022-04-22T18:25:36.480145+0000 mon.a (mon.0) 4821 : cluster [DBG] osdmap e1963: 8 total, 8 up, 8 in 2022-04-22T18:25:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:37 smithi115 bash[45495]: audit 2022-04-22T18:25:36.716938+0000 mon.a (mon.0) 4822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:37 smithi115 bash[46934]: audit 2022-04-22T18:25:36.716938+0000 mon.a (mon.0) 4822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:37 smithi121 bash[36902]: audit 2022-04-22T18:25:36.716938+0000 mon.a (mon.0) 4822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:38 smithi115 bash[45495]: cluster 2022-04-22T18:25:37.514585+0000 mgr.y (mgr.34107) 2077 : cluster [DBG] pgmap v3471: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2022-04-22T18:25:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:38 smithi115 bash[45495]: cluster 2022-04-22T18:25:37.529538+0000 mon.a (mon.0) 4823 : cluster [DBG] osdmap e1964: 8 total, 8 up, 8 in 2022-04-22T18:25:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:38 smithi115 bash[46934]: cluster 2022-04-22T18:25:37.514585+0000 mgr.y (mgr.34107) 2077 : cluster [DBG] pgmap v3471: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2022-04-22T18:25:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:38 smithi115 bash[46934]: cluster 2022-04-22T18:25:37.529538+0000 mon.a (mon.0) 4823 : cluster [DBG] osdmap e1964: 8 total, 8 up, 8 in 2022-04-22T18:25:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:38 smithi121 bash[36902]: cluster 2022-04-22T18:25:37.514585+0000 mgr.y (mgr.34107) 2077 : cluster [DBG] pgmap v3471: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.68k op/s 2022-04-22T18:25:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:38 smithi121 bash[36902]: cluster 2022-04-22T18:25:37.529538+0000 mon.a (mon.0) 4823 : cluster [DBG] osdmap e1964: 8 total, 8 up, 8 in 2022-04-22T18:25:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:39 smithi121 bash[36902]: cluster 2022-04-22T18:25:38.577221+0000 mon.a (mon.0) 4824 : cluster [DBG] osdmap e1965: 8 total, 8 up, 8 in 2022-04-22T18:25:40.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:39 smithi121 bash[36902]: audit 2022-04-22T18:25:38.580607+0000 mon.a (mon.0) 4825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:39 smithi115 bash[45495]: cluster 2022-04-22T18:25:38.577221+0000 mon.a (mon.0) 4824 : cluster [DBG] osdmap e1965: 8 total, 8 up, 8 in 2022-04-22T18:25:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:39 smithi115 bash[45495]: audit 2022-04-22T18:25:38.580607+0000 mon.a (mon.0) 4825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:39 smithi115 bash[46934]: cluster 2022-04-22T18:25:38.577221+0000 mon.a (mon.0) 4824 : cluster [DBG] osdmap e1965: 8 total, 8 up, 8 in 2022-04-22T18:25:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:39 smithi115 bash[46934]: audit 2022-04-22T18:25:38.580607+0000 mon.a (mon.0) 4825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:40 smithi121 bash[36902]: cluster 2022-04-22T18:25:39.515582+0000 mgr.y (mgr.34107) 2078 : cluster [DBG] pgmap v3474: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:40 smithi121 bash[36902]: cluster 2022-04-22T18:25:39.587730+0000 mon.a (mon.0) 4826 : cluster [DBG] osdmap e1966: 8 total, 8 up, 8 in 2022-04-22T18:25:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:40 smithi115 bash[45495]: cluster 2022-04-22T18:25:39.515582+0000 mgr.y (mgr.34107) 2078 : cluster [DBG] pgmap v3474: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:40 smithi115 bash[45495]: cluster 2022-04-22T18:25:39.587730+0000 mon.a (mon.0) 4826 : cluster [DBG] osdmap e1966: 8 total, 8 up, 8 in 2022-04-22T18:25:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:40 smithi115 bash[46934]: cluster 2022-04-22T18:25:39.515582+0000 mgr.y (mgr.34107) 2078 : cluster [DBG] pgmap v3474: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:25:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:40 smithi115 bash[46934]: cluster 2022-04-22T18:25:39.587730+0000 mon.a (mon.0) 4826 : cluster [DBG] osdmap e1966: 8 total, 8 up, 8 in 2022-04-22T18:25:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:41 smithi121 bash[36902]: cluster 2022-04-22T18:25:40.592727+0000 mon.a (mon.0) 4827 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2022-04-22T18:25:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:41 smithi121 bash[36902]: audit 2022-04-22T18:25:41.200058+0000 mon.a (mon.0) 4828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:41 smithi121 bash[36902]: cluster 2022-04-22T18:25:41.482378+0000 mon.a (mon.0) 4829 : cluster [DBG] osdmap e1968: 8 total, 8 up, 8 in 2022-04-22T18:25:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:41.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[45495]: cluster 2022-04-22T18:25:40.592727+0000 mon.a (mon.0) 4827 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2022-04-22T18:25:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[45495]: audit 2022-04-22T18:25:41.200058+0000 mon.a (mon.0) 4828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[45495]: cluster 2022-04-22T18:25:41.482378+0000 mon.a (mon.0) 4829 : cluster [DBG] osdmap e1968: 8 total, 8 up, 8 in 2022-04-22T18:25:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[46934]: cluster 2022-04-22T18:25:40.592727+0000 mon.a (mon.0) 4827 : cluster [DBG] osdmap e1967: 8 total, 8 up, 8 in 2022-04-22T18:25:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[46934]: audit 2022-04-22T18:25:41.200058+0000 mon.a (mon.0) 4828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:41 smithi115 bash[46934]: cluster 2022-04-22T18:25:41.482378+0000 mon.a (mon.0) 4829 : cluster [DBG] osdmap e1968: 8 total, 8 up, 8 in 2022-04-22T18:25:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:42 smithi121 bash[36902]: cluster 2022-04-22T18:25:41.516384+0000 mgr.y (mgr.34107) 2079 : cluster [DBG] pgmap v3478: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.65k op/s 2022-04-22T18:25:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:42 smithi121 bash[36902]: audit 2022-04-22T18:25:41.720855+0000 mon.a (mon.0) 4830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:42 smithi121 bash[36902]: cluster 2022-04-22T18:25:42.503826+0000 mon.a (mon.0) 4831 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2022-04-22T18:25:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[45495]: cluster 2022-04-22T18:25:41.516384+0000 mgr.y (mgr.34107) 2079 : cluster [DBG] pgmap v3478: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.65k op/s 2022-04-22T18:25:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[45495]: audit 2022-04-22T18:25:41.720855+0000 mon.a (mon.0) 4830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[45495]: cluster 2022-04-22T18:25:42.503826+0000 mon.a (mon.0) 4831 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2022-04-22T18:25:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[46934]: cluster 2022-04-22T18:25:41.516384+0000 mgr.y (mgr.34107) 2079 : cluster [DBG] pgmap v3478: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.65k op/s 2022-04-22T18:25:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[46934]: audit 2022-04-22T18:25:41.720855+0000 mon.a (mon.0) 4830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:42 smithi115 bash[46934]: cluster 2022-04-22T18:25:42.503826+0000 mon.a (mon.0) 4831 : cluster [DBG] osdmap e1969: 8 total, 8 up, 8 in 2022-04-22T18:25:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:43.705Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[45495]: cluster 2022-04-22T18:25:43.508249+0000 mon.a (mon.0) 4832 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2022-04-22T18:25:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[45495]: cluster 2022-04-22T18:25:43.516942+0000 mgr.y (mgr.34107) 2080 : cluster [DBG] pgmap v3481: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.1 KiB/s wr, 1.67k op/s 2022-04-22T18:25:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[45495]: audit 2022-04-22T18:25:43.581164+0000 mon.a (mon.0) 4833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[46934]: cluster 2022-04-22T18:25:43.508249+0000 mon.a (mon.0) 4832 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2022-04-22T18:25:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[46934]: cluster 2022-04-22T18:25:43.516942+0000 mgr.y (mgr.34107) 2080 : cluster [DBG] pgmap v3481: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.1 KiB/s wr, 1.67k op/s 2022-04-22T18:25:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:44 smithi115 bash[46934]: audit 2022-04-22T18:25:43.581164+0000 mon.a (mon.0) 4833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:44 smithi121 bash[36902]: cluster 2022-04-22T18:25:43.508249+0000 mon.a (mon.0) 4832 : cluster [DBG] osdmap e1970: 8 total, 8 up, 8 in 2022-04-22T18:25:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:44 smithi121 bash[36902]: cluster 2022-04-22T18:25:43.516942+0000 mgr.y (mgr.34107) 2080 : cluster [DBG] pgmap v3481: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.1 KiB/s wr, 1.67k op/s 2022-04-22T18:25:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:44 smithi121 bash[36902]: audit 2022-04-22T18:25:43.581164+0000 mon.a (mon.0) 4833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:45 smithi115 bash[45495]: cluster 2022-04-22T18:25:44.521534+0000 mon.a (mon.0) 4834 : cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2022-04-22T18:25:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:45 smithi115 bash[46934]: cluster 2022-04-22T18:25:44.521534+0000 mon.a (mon.0) 4834 : cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2022-04-22T18:25:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:45 smithi121 bash[36902]: cluster 2022-04-22T18:25:44.521534+0000 mon.a (mon.0) 4834 : cluster [DBG] osdmap e1971: 8 total, 8 up, 8 in 2022-04-22T18:25:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[45495]: cluster 2022-04-22T18:25:45.517824+0000 mgr.y (mgr.34107) 2081 : cluster [DBG] pgmap v3483: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[45495]: cluster 2022-04-22T18:25:45.525733+0000 mon.a (mon.0) 4835 : cluster [DBG] osdmap e1972: 8 total, 8 up, 8 in 2022-04-22T18:25:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[45495]: audit 2022-04-22T18:25:46.203002+0000 mon.a (mon.0) 4836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:46.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[45495]: cluster 2022-04-22T18:25:46.485800+0000 mon.a (mon.0) 4837 : cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2022-04-22T18:25:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[46934]: cluster 2022-04-22T18:25:45.517824+0000 mgr.y (mgr.34107) 2081 : cluster [DBG] pgmap v3483: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[46934]: cluster 2022-04-22T18:25:45.525733+0000 mon.a (mon.0) 4835 : cluster [DBG] osdmap e1972: 8 total, 8 up, 8 in 2022-04-22T18:25:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[46934]: audit 2022-04-22T18:25:46.203002+0000 mon.a (mon.0) 4836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:46.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:46 smithi115 bash[46934]: cluster 2022-04-22T18:25:46.485800+0000 mon.a (mon.0) 4837 : cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2022-04-22T18:25:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:46 smithi121 bash[36902]: cluster 2022-04-22T18:25:45.517824+0000 mgr.y (mgr.34107) 2081 : cluster [DBG] pgmap v3483: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:46 smithi121 bash[36902]: cluster 2022-04-22T18:25:45.525733+0000 mon.a (mon.0) 4835 : cluster [DBG] osdmap e1972: 8 total, 8 up, 8 in 2022-04-22T18:25:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:46 smithi121 bash[36902]: audit 2022-04-22T18:25:46.203002+0000 mon.a (mon.0) 4836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:46 smithi121 bash[36902]: cluster 2022-04-22T18:25:46.485800+0000 mon.a (mon.0) 4837 : cluster [DBG] osdmap e1973: 8 total, 8 up, 8 in 2022-04-22T18:25:47.125 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (48m) 6m ago 62m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (62m) 7m ago 62m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (50m) 6m ago 63m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (50m) 7m ago 63m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (58m) 7m ago 67m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:25:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (58m) 6m ago 70m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (57m) 6m ago 70m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (56m) 7m ago 68m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (57m) 6m ago 68m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (49m) 6m ago 62m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (49m) 7m ago 62m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (55m) 6m ago 67m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (54m) 6m ago 67m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (54m) 6m ago 66m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (53m) 6m ago 66m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:25:47.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (53m) 7m ago 65m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:25:47.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (52m) 7m ago 65m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:25:47.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (51m) 7m ago 64m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:25:47.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (51m) 7m ago 64m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:25:47.472 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (49m) 7m ago 62m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:25:47.887 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:25:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:25:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:25:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:25:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:25:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:25:47.889 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:25:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:47 smithi121 bash[36902]: audit 2022-04-22T18:25:46.724767+0000 mon.a (mon.0) 4838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:47 smithi115 bash[45495]: audit 2022-04-22T18:25:46.724767+0000 mon.a (mon.0) 4838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:48.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:47 smithi115 bash[46934]: audit 2022-04-22T18:25:46.724767+0000 mon.a (mon.0) 4838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[45495]: audit 2022-04-22T18:25:47.101138+0000 mgr.y (mgr.34107) 2082 : audit [DBG] from='client.37545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[45495]: audit 2022-04-22T18:25:47.442670+0000 mgr.y (mgr.34107) 2083 : audit [DBG] from='client.47482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[45495]: cluster 2022-04-22T18:25:47.518976+0000 mgr.y (mgr.34107) 2084 : cluster [DBG] pgmap v3486: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[45495]: cluster 2022-04-22T18:25:47.544634+0000 mon.a (mon.0) 4839 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[45495]: audit 2022-04-22T18:25:47.880407+0000 mon.c (mon.1) 127 : audit [DBG] from='client.? 172.21.15.115:0/3122057589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[46934]: audit 2022-04-22T18:25:47.101138+0000 mgr.y (mgr.34107) 2082 : audit [DBG] from='client.37545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[46934]: audit 2022-04-22T18:25:47.442670+0000 mgr.y (mgr.34107) 2083 : audit [DBG] from='client.47482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[46934]: cluster 2022-04-22T18:25:47.518976+0000 mgr.y (mgr.34107) 2084 : cluster [DBG] pgmap v3486: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[46934]: cluster 2022-04-22T18:25:47.544634+0000 mon.a (mon.0) 4839 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2022-04-22T18:25:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:48 smithi115 bash[46934]: audit 2022-04-22T18:25:47.880407+0000 mon.c (mon.1) 127 : audit [DBG] from='client.? 172.21.15.115:0/3122057589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:48 smithi121 bash[36902]: audit 2022-04-22T18:25:47.101138+0000 mgr.y (mgr.34107) 2082 : audit [DBG] from='client.37545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:48 smithi121 bash[36902]: audit 2022-04-22T18:25:47.442670+0000 mgr.y (mgr.34107) 2083 : audit [DBG] from='client.47482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:25:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:48 smithi121 bash[36902]: cluster 2022-04-22T18:25:47.518976+0000 mgr.y (mgr.34107) 2084 : cluster [DBG] pgmap v3486: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.0 KiB/s wr, 1.59k op/s 2022-04-22T18:25:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:48 smithi121 bash[36902]: cluster 2022-04-22T18:25:47.544634+0000 mon.a (mon.0) 4839 : cluster [DBG] osdmap e1974: 8 total, 8 up, 8 in 2022-04-22T18:25:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:48 smithi121 bash[36902]: audit 2022-04-22T18:25:47.880407+0000 mon.c (mon.1) 127 : audit [DBG] from='client.? 172.21.15.115:0/3122057589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:25:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:49 smithi121 bash[36902]: cluster 2022-04-22T18:25:48.551178+0000 mon.a (mon.0) 4840 : cluster [DBG] osdmap e1975: 8 total, 8 up, 8 in 2022-04-22T18:25:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:49 smithi121 bash[36902]: audit 2022-04-22T18:25:48.583333+0000 mon.a (mon.0) 4841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:49 smithi115 bash[45495]: cluster 2022-04-22T18:25:48.551178+0000 mon.a (mon.0) 4840 : cluster [DBG] osdmap e1975: 8 total, 8 up, 8 in 2022-04-22T18:25:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:49 smithi115 bash[45495]: audit 2022-04-22T18:25:48.583333+0000 mon.a (mon.0) 4841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:49 smithi115 bash[46934]: cluster 2022-04-22T18:25:48.551178+0000 mon.a (mon.0) 4840 : cluster [DBG] osdmap e1975: 8 total, 8 up, 8 in 2022-04-22T18:25:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:49 smithi115 bash[46934]: audit 2022-04-22T18:25:48.583333+0000 mon.a (mon.0) 4841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:50 smithi121 bash[36902]: cluster 2022-04-22T18:25:49.519661+0000 mgr.y (mgr.34107) 2085 : cluster [DBG] pgmap v3489: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:25:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:50 smithi121 bash[36902]: cluster 2022-04-22T18:25:49.557431+0000 mon.a (mon.0) 4842 : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2022-04-22T18:25:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:50 smithi115 bash[45495]: cluster 2022-04-22T18:25:49.519661+0000 mgr.y (mgr.34107) 2085 : cluster [DBG] pgmap v3489: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:25:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:50 smithi115 bash[45495]: cluster 2022-04-22T18:25:49.557431+0000 mon.a (mon.0) 4842 : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2022-04-22T18:25:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:50 smithi115 bash[46934]: cluster 2022-04-22T18:25:49.519661+0000 mgr.y (mgr.34107) 2085 : cluster [DBG] pgmap v3489: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:25:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:50 smithi115 bash[46934]: cluster 2022-04-22T18:25:49.557431+0000 mon.a (mon.0) 4842 : cluster [DBG] osdmap e1976: 8 total, 8 up, 8 in 2022-04-22T18:25:51.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[46934]: cluster 2022-04-22T18:25:50.566279+0000 mon.a (mon.0) 4843 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2022-04-22T18:25:51.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[46934]: audit 2022-04-22T18:25:51.205385+0000 mon.a (mon.0) 4844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:51.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[46934]: cluster 2022-04-22T18:25:51.488530+0000 mon.a (mon.0) 4845 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2022-04-22T18:25:51.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[45495]: cluster 2022-04-22T18:25:50.566279+0000 mon.a (mon.0) 4843 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2022-04-22T18:25:51.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[45495]: audit 2022-04-22T18:25:51.205385+0000 mon.a (mon.0) 4844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:51.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[45495]: cluster 2022-04-22T18:25:51.488530+0000 mon.a (mon.0) 4845 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2022-04-22T18:25:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:51 smithi121 bash[36902]: cluster 2022-04-22T18:25:50.566279+0000 mon.a (mon.0) 4843 : cluster [DBG] osdmap e1977: 8 total, 8 up, 8 in 2022-04-22T18:25:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:51 smithi121 bash[36902]: audit 2022-04-22T18:25:51.205385+0000 mon.a (mon.0) 4844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:51 smithi121 bash[36902]: cluster 2022-04-22T18:25:51.488530+0000 mon.a (mon.0) 4845 : cluster [DBG] osdmap e1978: 8 total, 8 up, 8 in 2022-04-22T18:25:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:51.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:52 smithi121 bash[36902]: cluster 2022-04-22T18:25:51.520192+0000 mgr.y (mgr.34107) 2086 : cluster [DBG] pgmap v3493: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.56k op/s 2022-04-22T18:25:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:52 smithi121 bash[36902]: audit 2022-04-22T18:25:51.728659+0000 mon.a (mon.0) 4846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:52 smithi121 bash[36902]: cluster 2022-04-22T18:25:52.506079+0000 mon.a (mon.0) 4847 : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2022-04-22T18:25:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[45495]: cluster 2022-04-22T18:25:51.520192+0000 mgr.y (mgr.34107) 2086 : cluster [DBG] pgmap v3493: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.56k op/s 2022-04-22T18:25:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[45495]: audit 2022-04-22T18:25:51.728659+0000 mon.a (mon.0) 4846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[45495]: cluster 2022-04-22T18:25:52.506079+0000 mon.a (mon.0) 4847 : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2022-04-22T18:25:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[46934]: cluster 2022-04-22T18:25:51.520192+0000 mgr.y (mgr.34107) 2086 : cluster [DBG] pgmap v3493: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.56k op/s 2022-04-22T18:25:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[46934]: audit 2022-04-22T18:25:51.728659+0000 mon.a (mon.0) 4846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:52 smithi115 bash[46934]: cluster 2022-04-22T18:25:52.506079+0000 mon.a (mon.0) 4847 : cluster [DBG] osdmap e1979: 8 total, 8 up, 8 in 2022-04-22T18:25:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:25:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:25:53.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:25:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:54 smithi121 bash[36902]: cluster 2022-04-22T18:25:53.521404+0000 mgr.y (mgr.34107) 2087 : cluster [DBG] pgmap v3495: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2022-04-22T18:25:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:54 smithi121 bash[36902]: cluster 2022-04-22T18:25:53.580717+0000 mon.a (mon.0) 4848 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2022-04-22T18:25:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:54 smithi121 bash[36902]: audit 2022-04-22T18:25:53.585122+0000 mon.a (mon.0) 4849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[46934]: cluster 2022-04-22T18:25:53.521404+0000 mgr.y (mgr.34107) 2087 : cluster [DBG] pgmap v3495: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2022-04-22T18:25:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[46934]: cluster 2022-04-22T18:25:53.580717+0000 mon.a (mon.0) 4848 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2022-04-22T18:25:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[46934]: audit 2022-04-22T18:25:53.585122+0000 mon.a (mon.0) 4849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[45495]: cluster 2022-04-22T18:25:53.521404+0000 mgr.y (mgr.34107) 2087 : cluster [DBG] pgmap v3495: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.55k op/s 2022-04-22T18:25:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[45495]: cluster 2022-04-22T18:25:53.580717+0000 mon.a (mon.0) 4848 : cluster [DBG] osdmap e1980: 8 total, 8 up, 8 in 2022-04-22T18:25:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:54 smithi115 bash[45495]: audit 2022-04-22T18:25:53.585122+0000 mon.a (mon.0) 4849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:55 smithi121 bash[36902]: cluster 2022-04-22T18:25:54.588390+0000 mon.a (mon.0) 4850 : cluster [DBG] osdmap e1981: 8 total, 8 up, 8 in 2022-04-22T18:25:56.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:55 smithi115 bash[46934]: cluster 2022-04-22T18:25:54.588390+0000 mon.a (mon.0) 4850 : cluster [DBG] osdmap e1981: 8 total, 8 up, 8 in 2022-04-22T18:25:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:55 smithi115 bash[45495]: cluster 2022-04-22T18:25:54.588390+0000 mon.a (mon.0) 4850 : cluster [DBG] osdmap e1981: 8 total, 8 up, 8 in 2022-04-22T18:25:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:56 smithi121 bash[36902]: cluster 2022-04-22T18:25:55.522559+0000 mgr.y (mgr.34107) 2088 : cluster [DBG] pgmap v3498: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2022-04-22T18:25:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:56 smithi121 bash[36902]: cluster 2022-04-22T18:25:55.598799+0000 mon.a (mon.0) 4851 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2022-04-22T18:25:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:56 smithi121 bash[36902]: audit 2022-04-22T18:25:56.208615+0000 mon.a (mon.0) 4852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:56 smithi121 bash[36902]: cluster 2022-04-22T18:25:56.491135+0000 mon.a (mon.0) 4853 : cluster [DBG] osdmap e1983: 8 total, 8 up, 8 in 2022-04-22T18:25:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[46934]: cluster 2022-04-22T18:25:55.522559+0000 mgr.y (mgr.34107) 2088 : cluster [DBG] pgmap v3498: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2022-04-22T18:25:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[46934]: cluster 2022-04-22T18:25:55.598799+0000 mon.a (mon.0) 4851 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2022-04-22T18:25:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[46934]: audit 2022-04-22T18:25:56.208615+0000 mon.a (mon.0) 4852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[46934]: cluster 2022-04-22T18:25:56.491135+0000 mon.a (mon.0) 4853 : cluster [DBG] osdmap e1983: 8 total, 8 up, 8 in 2022-04-22T18:25:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[45495]: cluster 2022-04-22T18:25:55.522559+0000 mgr.y (mgr.34107) 2088 : cluster [DBG] pgmap v3498: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.50k op/s 2022-04-22T18:25:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[45495]: cluster 2022-04-22T18:25:55.598799+0000 mon.a (mon.0) 4851 : cluster [DBG] osdmap e1982: 8 total, 8 up, 8 in 2022-04-22T18:25:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[45495]: audit 2022-04-22T18:25:56.208615+0000 mon.a (mon.0) 4852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:25:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:56 smithi115 bash[45495]: cluster 2022-04-22T18:25:56.491135+0000 mon.a (mon.0) 4853 : cluster [DBG] osdmap e1983: 8 total, 8 up, 8 in 2022-04-22T18:25:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:57 smithi121 bash[36902]: audit 2022-04-22T18:25:56.732246+0000 mon.a (mon.0) 4854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:57 smithi121 bash[36902]: cluster 2022-04-22T18:25:57.508344+0000 mon.a (mon.0) 4855 : cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2022-04-22T18:25:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:57 smithi115 bash[45495]: audit 2022-04-22T18:25:56.732246+0000 mon.a (mon.0) 4854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:57 smithi115 bash[45495]: cluster 2022-04-22T18:25:57.508344+0000 mon.a (mon.0) 4855 : cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2022-04-22T18:25:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:57 smithi115 bash[46934]: audit 2022-04-22T18:25:56.732246+0000 mon.a (mon.0) 4854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:25:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:57 smithi115 bash[46934]: cluster 2022-04-22T18:25:57.508344+0000 mon.a (mon.0) 4855 : cluster [DBG] osdmap e1984: 8 total, 8 up, 8 in 2022-04-22T18:25:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:58 smithi121 bash[36902]: cluster 2022-04-22T18:25:57.523192+0000 mgr.y (mgr.34107) 2089 : cluster [DBG] pgmap v3502: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 KiB/s wr, 1.54k op/s 2022-04-22T18:25:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:58 smithi121 bash[36902]: cluster 2022-04-22T18:25:58.512027+0000 mon.a (mon.0) 4856 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2022-04-22T18:25:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:25:58 smithi121 bash[36902]: audit 2022-04-22T18:25:58.587282+0000 mon.a (mon.0) 4857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[45495]: cluster 2022-04-22T18:25:57.523192+0000 mgr.y (mgr.34107) 2089 : cluster [DBG] pgmap v3502: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 KiB/s wr, 1.54k op/s 2022-04-22T18:25:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[45495]: cluster 2022-04-22T18:25:58.512027+0000 mon.a (mon.0) 4856 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2022-04-22T18:25:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[45495]: audit 2022-04-22T18:25:58.587282+0000 mon.a (mon.0) 4857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:25:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[46934]: cluster 2022-04-22T18:25:57.523192+0000 mgr.y (mgr.34107) 2089 : cluster [DBG] pgmap v3502: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 KiB/s wr, 1.54k op/s 2022-04-22T18:25:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[46934]: cluster 2022-04-22T18:25:58.512027+0000 mon.a (mon.0) 4856 : cluster [DBG] osdmap e1985: 8 total, 8 up, 8 in 2022-04-22T18:25:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:25:58 smithi115 bash[46934]: audit 2022-04-22T18:25:58.587282+0000 mon.a (mon.0) 4857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:00 smithi115 bash[45495]: cluster 2022-04-22T18:25:59.516974+0000 mon.a (mon.0) 4858 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2022-04-22T18:26:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:00 smithi115 bash[45495]: cluster 2022-04-22T18:25:59.523727+0000 mgr.y (mgr.34107) 2090 : cluster [DBG] pgmap v3505: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:26:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:00 smithi115 bash[46934]: cluster 2022-04-22T18:25:59.516974+0000 mon.a (mon.0) 4858 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2022-04-22T18:26:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:00 smithi115 bash[46934]: cluster 2022-04-22T18:25:59.523727+0000 mgr.y (mgr.34107) 2090 : cluster [DBG] pgmap v3505: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:26:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:00 smithi121 bash[36902]: cluster 2022-04-22T18:25:59.516974+0000 mon.a (mon.0) 4858 : cluster [DBG] osdmap e1986: 8 total, 8 up, 8 in 2022-04-22T18:26:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:00 smithi121 bash[36902]: cluster 2022-04-22T18:25:59.523727+0000 mgr.y (mgr.34107) 2090 : cluster [DBG] pgmap v3505: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:26:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[45495]: cluster 2022-04-22T18:26:00.527943+0000 mon.a (mon.0) 4859 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2022-04-22T18:26:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[45495]: audit 2022-04-22T18:26:01.211774+0000 mon.a (mon.0) 4860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[45495]: cluster 2022-04-22T18:26:01.492392+0000 mon.a (mon.0) 4861 : cluster [DBG] osdmap e1988: 8 total, 8 up, 8 in 2022-04-22T18:26:01.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[46934]: cluster 2022-04-22T18:26:00.527943+0000 mon.a (mon.0) 4859 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2022-04-22T18:26:01.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[46934]: audit 2022-04-22T18:26:01.211774+0000 mon.a (mon.0) 4860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:01.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[46934]: cluster 2022-04-22T18:26:01.492392+0000 mon.a (mon.0) 4861 : cluster [DBG] osdmap e1988: 8 total, 8 up, 8 in 2022-04-22T18:26:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:01 smithi121 bash[36902]: cluster 2022-04-22T18:26:00.527943+0000 mon.a (mon.0) 4859 : cluster [DBG] osdmap e1987: 8 total, 8 up, 8 in 2022-04-22T18:26:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:01 smithi121 bash[36902]: audit 2022-04-22T18:26:01.211774+0000 mon.a (mon.0) 4860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:01 smithi121 bash[36902]: cluster 2022-04-22T18:26:01.492392+0000 mon.a (mon.0) 4861 : cluster [DBG] osdmap e1988: 8 total, 8 up, 8 in 2022-04-22T18:26:02.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:01.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:02 smithi115 bash[45495]: cluster 2022-04-22T18:26:01.524353+0000 mgr.y (mgr.34107) 2091 : cluster [DBG] pgmap v3508: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2022-04-22T18:26:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:02 smithi115 bash[45495]: audit 2022-04-22T18:26:01.736018+0000 mon.a (mon.0) 4862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:02 smithi115 bash[46934]: cluster 2022-04-22T18:26:01.524353+0000 mgr.y (mgr.34107) 2091 : cluster [DBG] pgmap v3508: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2022-04-22T18:26:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:02 smithi115 bash[46934]: audit 2022-04-22T18:26:01.736018+0000 mon.a (mon.0) 4862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:02 smithi121 bash[36902]: cluster 2022-04-22T18:26:01.524353+0000 mgr.y (mgr.34107) 2091 : cluster [DBG] pgmap v3508: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.47k op/s 2022-04-22T18:26:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:02 smithi121 bash[36902]: audit 2022-04-22T18:26:01.736018+0000 mon.a (mon.0) 4862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:03 smithi115 bash[45495]: cluster 2022-04-22T18:26:02.533440+0000 mon.a (mon.0) 4863 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2022-04-22T18:26:03.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:03 smithi115 bash[46934]: cluster 2022-04-22T18:26:02.533440+0000 mon.a (mon.0) 4863 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2022-04-22T18:26:03.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:03.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:04.001 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:03 smithi121 bash[36902]: cluster 2022-04-22T18:26:02.533440+0000 mon.a (mon.0) 4863 : cluster [DBG] osdmap e1989: 8 total, 8 up, 8 in 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: cluster 2022-04-22T18:26:03.525346+0000 mgr.y (mgr.34107) 2092 : cluster [DBG] pgmap v3510: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: cluster 2022-04-22T18:26:03.544653+0000 mon.a (mon.0) 4864 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: audit 2022-04-22T18:26:03.589667+0000 mon.a (mon.0) 4865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: audit 2022-04-22T18:26:03.967947+0000 mon.a (mon.0) 4866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: audit 2022-04-22T18:26:03.970940+0000 mon.a (mon.0) 4867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: audit 2022-04-22T18:26:03.972729+0000 mon.a (mon.0) 4868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:26:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:04 smithi121 bash[36902]: audit 2022-04-22T18:26:04.337904+0000 mon.a (mon.0) 4869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:26:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: cluster 2022-04-22T18:26:03.525346+0000 mgr.y (mgr.34107) 2092 : cluster [DBG] pgmap v3510: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2022-04-22T18:26:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: cluster 2022-04-22T18:26:03.544653+0000 mon.a (mon.0) 4864 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2022-04-22T18:26:04.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: audit 2022-04-22T18:26:03.589667+0000 mon.a (mon.0) 4865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: audit 2022-04-22T18:26:03.967947+0000 mon.a (mon.0) 4866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: audit 2022-04-22T18:26:03.970940+0000 mon.a (mon.0) 4867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: audit 2022-04-22T18:26:03.972729+0000 mon.a (mon.0) 4868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[45495]: audit 2022-04-22T18:26:04.337904+0000 mon.a (mon.0) 4869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: cluster 2022-04-22T18:26:03.525346+0000 mgr.y (mgr.34107) 2092 : cluster [DBG] pgmap v3510: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 KiB/s wr, 1.46k op/s 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: cluster 2022-04-22T18:26:03.544653+0000 mon.a (mon.0) 4864 : cluster [DBG] osdmap e1990: 8 total, 8 up, 8 in 2022-04-22T18:26:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: audit 2022-04-22T18:26:03.589667+0000 mon.a (mon.0) 4865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: audit 2022-04-22T18:26:03.967947+0000 mon.a (mon.0) 4866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:26:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: audit 2022-04-22T18:26:03.970940+0000 mon.a (mon.0) 4867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:26:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: audit 2022-04-22T18:26:03.972729+0000 mon.a (mon.0) 4868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:26:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:04 smithi115 bash[46934]: audit 2022-04-22T18:26:04.337904+0000 mon.a (mon.0) 4869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:26:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:05 smithi121 bash[36902]: cluster 2022-04-22T18:26:04.588393+0000 mon.a (mon.0) 4870 : cluster [DBG] osdmap e1991: 8 total, 8 up, 8 in 2022-04-22T18:26:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:05 smithi115 bash[45495]: cluster 2022-04-22T18:26:04.588393+0000 mon.a (mon.0) 4870 : cluster [DBG] osdmap e1991: 8 total, 8 up, 8 in 2022-04-22T18:26:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:05 smithi115 bash[46934]: cluster 2022-04-22T18:26:04.588393+0000 mon.a (mon.0) 4870 : cluster [DBG] osdmap e1991: 8 total, 8 up, 8 in 2022-04-22T18:26:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:06 smithi121 bash[36902]: cluster 2022-04-22T18:26:05.526429+0000 mgr.y (mgr.34107) 2093 : cluster [DBG] pgmap v3513: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.41k op/s 2022-04-22T18:26:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:06 smithi121 bash[36902]: cluster 2022-04-22T18:26:05.596691+0000 mon.a (mon.0) 4871 : cluster [DBG] osdmap e1992: 8 total, 8 up, 8 in 2022-04-22T18:26:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:06 smithi121 bash[36902]: audit 2022-04-22T18:26:06.214920+0000 mon.a (mon.0) 4872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:06 smithi121 bash[36902]: cluster 2022-04-22T18:26:06.494481+0000 mon.a (mon.0) 4873 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2022-04-22T18:26:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[46934]: cluster 2022-04-22T18:26:05.526429+0000 mgr.y (mgr.34107) 2093 : cluster [DBG] pgmap v3513: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.41k op/s 2022-04-22T18:26:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[46934]: cluster 2022-04-22T18:26:05.596691+0000 mon.a (mon.0) 4871 : cluster [DBG] osdmap e1992: 8 total, 8 up, 8 in 2022-04-22T18:26:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[46934]: audit 2022-04-22T18:26:06.214920+0000 mon.a (mon.0) 4872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:07.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[46934]: cluster 2022-04-22T18:26:06.494481+0000 mon.a (mon.0) 4873 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2022-04-22T18:26:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[45495]: cluster 2022-04-22T18:26:05.526429+0000 mgr.y (mgr.34107) 2093 : cluster [DBG] pgmap v3513: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.41k op/s 2022-04-22T18:26:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[45495]: cluster 2022-04-22T18:26:05.596691+0000 mon.a (mon.0) 4871 : cluster [DBG] osdmap e1992: 8 total, 8 up, 8 in 2022-04-22T18:26:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[45495]: audit 2022-04-22T18:26:06.214920+0000 mon.a (mon.0) 4872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:06 smithi115 bash[45495]: cluster 2022-04-22T18:26:06.494481+0000 mon.a (mon.0) 4873 : cluster [DBG] osdmap e1993: 8 total, 8 up, 8 in 2022-04-22T18:26:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:07 smithi121 bash[36902]: audit 2022-04-22T18:26:06.739691+0000 mon.a (mon.0) 4874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:07 smithi121 bash[36902]: cluster 2022-04-22T18:26:07.508409+0000 mon.a (mon.0) 4875 : cluster [DBG] osdmap e1994: 8 total, 8 up, 8 in 2022-04-22T18:26:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:07 smithi115 bash[46934]: audit 2022-04-22T18:26:06.739691+0000 mon.a (mon.0) 4874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:07 smithi115 bash[46934]: cluster 2022-04-22T18:26:07.508409+0000 mon.a (mon.0) 4875 : cluster [DBG] osdmap e1994: 8 total, 8 up, 8 in 2022-04-22T18:26:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:07 smithi115 bash[45495]: audit 2022-04-22T18:26:06.739691+0000 mon.a (mon.0) 4874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:07 smithi115 bash[45495]: cluster 2022-04-22T18:26:07.508409+0000 mon.a (mon.0) 4875 : cluster [DBG] osdmap e1994: 8 total, 8 up, 8 in 2022-04-22T18:26:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:08 smithi121 bash[36902]: cluster 2022-04-22T18:26:07.527078+0000 mgr.y (mgr.34107) 2094 : cluster [DBG] pgmap v3517: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.43k op/s 2022-04-22T18:26:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:08 smithi121 bash[36902]: cluster 2022-04-22T18:26:08.515974+0000 mon.a (mon.0) 4876 : cluster [DBG] osdmap e1995: 8 total, 8 up, 8 in 2022-04-22T18:26:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:08 smithi121 bash[36902]: audit 2022-04-22T18:26:08.592115+0000 mon.a (mon.0) 4877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[45495]: cluster 2022-04-22T18:26:07.527078+0000 mgr.y (mgr.34107) 2094 : cluster [DBG] pgmap v3517: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.43k op/s 2022-04-22T18:26:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[45495]: cluster 2022-04-22T18:26:08.515974+0000 mon.a (mon.0) 4876 : cluster [DBG] osdmap e1995: 8 total, 8 up, 8 in 2022-04-22T18:26:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[45495]: audit 2022-04-22T18:26:08.592115+0000 mon.a (mon.0) 4877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[46934]: cluster 2022-04-22T18:26:07.527078+0000 mgr.y (mgr.34107) 2094 : cluster [DBG] pgmap v3517: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 5.0 KiB/s wr, 1.43k op/s 2022-04-22T18:26:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[46934]: cluster 2022-04-22T18:26:08.515974+0000 mon.a (mon.0) 4876 : cluster [DBG] osdmap e1995: 8 total, 8 up, 8 in 2022-04-22T18:26:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:08 smithi115 bash[46934]: audit 2022-04-22T18:26:08.592115+0000 mon.a (mon.0) 4877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:10 smithi115 bash[45495]: cluster 2022-04-22T18:26:09.524172+0000 mon.a (mon.0) 4878 : cluster [DBG] osdmap e1996: 8 total, 8 up, 8 in 2022-04-22T18:26:10.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:10 smithi115 bash[45495]: cluster 2022-04-22T18:26:09.527685+0000 mgr.y (mgr.34107) 2095 : cluster [DBG] pgmap v3520: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:10.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:10 smithi115 bash[46934]: cluster 2022-04-22T18:26:09.524172+0000 mon.a (mon.0) 4878 : cluster [DBG] osdmap e1996: 8 total, 8 up, 8 in 2022-04-22T18:26:10.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:10 smithi115 bash[46934]: cluster 2022-04-22T18:26:09.527685+0000 mgr.y (mgr.34107) 2095 : cluster [DBG] pgmap v3520: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:10 smithi121 bash[36902]: cluster 2022-04-22T18:26:09.524172+0000 mon.a (mon.0) 4878 : cluster [DBG] osdmap e1996: 8 total, 8 up, 8 in 2022-04-22T18:26:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:10 smithi121 bash[36902]: cluster 2022-04-22T18:26:09.527685+0000 mgr.y (mgr.34107) 2095 : cluster [DBG] pgmap v3520: 137 pgs: 137 active+clean; 4.6 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[45495]: cluster 2022-04-22T18:26:10.531484+0000 mon.a (mon.0) 4879 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2022-04-22T18:26:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[45495]: audit 2022-04-22T18:26:11.218286+0000 mon.a (mon.0) 4880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[45495]: cluster 2022-04-22T18:26:11.497407+0000 mon.a (mon.0) 4881 : cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2022-04-22T18:26:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[46934]: cluster 2022-04-22T18:26:10.531484+0000 mon.a (mon.0) 4879 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2022-04-22T18:26:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[46934]: audit 2022-04-22T18:26:11.218286+0000 mon.a (mon.0) 4880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[46934]: cluster 2022-04-22T18:26:11.497407+0000 mon.a (mon.0) 4881 : cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2022-04-22T18:26:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:11 smithi121 bash[36902]: cluster 2022-04-22T18:26:10.531484+0000 mon.a (mon.0) 4879 : cluster [DBG] osdmap e1997: 8 total, 8 up, 8 in 2022-04-22T18:26:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:11 smithi121 bash[36902]: audit 2022-04-22T18:26:11.218286+0000 mon.a (mon.0) 4880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:11 smithi121 bash[36902]: cluster 2022-04-22T18:26:11.497407+0000 mon.a (mon.0) 4881 : cluster [DBG] osdmap e1998: 8 total, 8 up, 8 in 2022-04-22T18:26:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:11.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:12 smithi115 bash[45495]: cluster 2022-04-22T18:26:11.528249+0000 mgr.y (mgr.34107) 2096 : cluster [DBG] pgmap v3523: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:12 smithi115 bash[45495]: audit 2022-04-22T18:26:11.743684+0000 mon.a (mon.0) 4882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:12 smithi115 bash[46934]: cluster 2022-04-22T18:26:11.528249+0000 mgr.y (mgr.34107) 2096 : cluster [DBG] pgmap v3523: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:12 smithi115 bash[46934]: audit 2022-04-22T18:26:11.743684+0000 mon.a (mon.0) 4882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:12 smithi121 bash[36902]: cluster 2022-04-22T18:26:11.528249+0000 mgr.y (mgr.34107) 2096 : cluster [DBG] pgmap v3523: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:12 smithi121 bash[36902]: audit 2022-04-22T18:26:11.743684+0000 mon.a (mon.0) 4882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:13 smithi121 bash[36902]: cluster 2022-04-22T18:26:12.540317+0000 mon.a (mon.0) 4883 : cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2022-04-22T18:26:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:13 smithi115 bash[45495]: cluster 2022-04-22T18:26:12.540317+0000 mon.a (mon.0) 4883 : cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2022-04-22T18:26:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:13 smithi115 bash[46934]: cluster 2022-04-22T18:26:12.540317+0000 mon.a (mon.0) 4883 : cluster [DBG] osdmap e1999: 8 total, 8 up, 8 in 2022-04-22T18:26:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:13.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:14 smithi121 bash[36902]: cluster 2022-04-22T18:26:13.529184+0000 mgr.y (mgr.34107) 2097 : cluster [DBG] pgmap v3525: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1020 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:14 smithi121 bash[36902]: cluster 2022-04-22T18:26:13.547521+0000 mon.a (mon.0) 4884 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2022-04-22T18:26:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:14 smithi121 bash[36902]: audit 2022-04-22T18:26:13.594628+0000 mon.a (mon.0) 4885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:14 smithi121 bash[36902]: audit 2022-04-22T18:26:13.971822+0000 mon.a (mon.0) 4886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:26:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:14 smithi121 bash[36902]: audit 2022-04-22T18:26:13.972304+0000 mon.a (mon.0) 4887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:26:15.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[46934]: cluster 2022-04-22T18:26:13.529184+0000 mgr.y (mgr.34107) 2097 : cluster [DBG] pgmap v3525: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1020 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:15.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[46934]: cluster 2022-04-22T18:26:13.547521+0000 mon.a (mon.0) 4884 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2022-04-22T18:26:15.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[46934]: audit 2022-04-22T18:26:13.594628+0000 mon.a (mon.0) 4885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[46934]: audit 2022-04-22T18:26:13.971822+0000 mon.a (mon.0) 4886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[46934]: audit 2022-04-22T18:26:13.972304+0000 mon.a (mon.0) 4887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[45495]: cluster 2022-04-22T18:26:13.529184+0000 mgr.y (mgr.34107) 2097 : cluster [DBG] pgmap v3525: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1020 KiB/s rd, 5.0 KiB/s wr, 1.38k op/s 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[45495]: cluster 2022-04-22T18:26:13.547521+0000 mon.a (mon.0) 4884 : cluster [DBG] osdmap e2000: 8 total, 8 up, 8 in 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[45495]: audit 2022-04-22T18:26:13.594628+0000 mon.a (mon.0) 4885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[45495]: audit 2022-04-22T18:26:13.971822+0000 mon.a (mon.0) 4886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:26:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:14 smithi115 bash[45495]: audit 2022-04-22T18:26:13.972304+0000 mon.a (mon.0) 4887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:26:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:15 smithi121 bash[36902]: cluster 2022-04-22T18:26:14.557557+0000 mon.a (mon.0) 4888 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2022-04-22T18:26:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:15 smithi115 bash[45495]: cluster 2022-04-22T18:26:14.557557+0000 mon.a (mon.0) 4888 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2022-04-22T18:26:16.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:15 smithi115 bash[46934]: cluster 2022-04-22T18:26:14.557557+0000 mon.a (mon.0) 4888 : cluster [DBG] osdmap e2001: 8 total, 8 up, 8 in 2022-04-22T18:26:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:16 smithi121 bash[36902]: cluster 2022-04-22T18:26:15.530327+0000 mgr.y (mgr.34107) 2098 : cluster [DBG] pgmap v3528: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 978 KiB/s rd, 5.0 KiB/s wr, 1.32k op/s 2022-04-22T18:26:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:16 smithi121 bash[36902]: cluster 2022-04-22T18:26:15.568767+0000 mon.a (mon.0) 4889 : cluster [DBG] osdmap e2002: 8 total, 8 up, 8 in 2022-04-22T18:26:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:16 smithi121 bash[36902]: audit 2022-04-22T18:26:16.221351+0000 mon.a (mon.0) 4890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:16 smithi121 bash[36902]: cluster 2022-04-22T18:26:16.499824+0000 mon.a (mon.0) 4891 : cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2022-04-22T18:26:17.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[45495]: cluster 2022-04-22T18:26:15.530327+0000 mgr.y (mgr.34107) 2098 : cluster [DBG] pgmap v3528: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 978 KiB/s rd, 5.0 KiB/s wr, 1.32k op/s 2022-04-22T18:26:17.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[45495]: cluster 2022-04-22T18:26:15.568767+0000 mon.a (mon.0) 4889 : cluster [DBG] osdmap e2002: 8 total, 8 up, 8 in 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[45495]: audit 2022-04-22T18:26:16.221351+0000 mon.a (mon.0) 4890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[45495]: cluster 2022-04-22T18:26:16.499824+0000 mon.a (mon.0) 4891 : cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[46934]: cluster 2022-04-22T18:26:15.530327+0000 mgr.y (mgr.34107) 2098 : cluster [DBG] pgmap v3528: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 978 KiB/s rd, 5.0 KiB/s wr, 1.32k op/s 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[46934]: cluster 2022-04-22T18:26:15.568767+0000 mon.a (mon.0) 4889 : cluster [DBG] osdmap e2002: 8 total, 8 up, 8 in 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[46934]: audit 2022-04-22T18:26:16.221351+0000 mon.a (mon.0) 4890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:16 smithi115 bash[46934]: cluster 2022-04-22T18:26:16.499824+0000 mon.a (mon.0) 4891 : cluster [DBG] osdmap e2003: 8 total, 8 up, 8 in 2022-04-22T18:26:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:17 smithi121 bash[36902]: audit 2022-04-22T18:26:16.747328+0000 mon.a (mon.0) 4892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:17 smithi121 bash[36902]: cluster 2022-04-22T18:26:17.514179+0000 mon.a (mon.0) 4893 : cluster [DBG] osdmap e2004: 8 total, 8 up, 8 in 2022-04-22T18:26:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:17 smithi115 bash[45495]: audit 2022-04-22T18:26:16.747328+0000 mon.a (mon.0) 4892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:17 smithi115 bash[45495]: cluster 2022-04-22T18:26:17.514179+0000 mon.a (mon.0) 4893 : cluster [DBG] osdmap e2004: 8 total, 8 up, 8 in 2022-04-22T18:26:18.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:17 smithi115 bash[46934]: audit 2022-04-22T18:26:16.747328+0000 mon.a (mon.0) 4892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:18.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:17 smithi115 bash[46934]: cluster 2022-04-22T18:26:17.514179+0000 mon.a (mon.0) 4893 : cluster [DBG] osdmap e2004: 8 total, 8 up, 8 in 2022-04-22T18:26:18.271 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (49m) 7m ago 63m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (62m) 8m ago 62m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (51m) 7m ago 64m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (50m) 8m ago 64m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (58m) 8m ago 68m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:26:18.622 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (59m) 7m ago 70m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (58m) 7m ago 70m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (57m) 8m ago 68m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (57m) 7m ago 68m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (50m) 7m ago 63m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (50m) 8m ago 63m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (56m) 7m ago 68m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (55m) 7m ago 67m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (54m) 7m ago 67m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (54m) 7m ago 66m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:26:18.623 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (53m) 8m ago 66m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:26:18.624 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (53m) 8m ago 65m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:26:18.624 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (52m) 8m ago 65m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:26:18.624 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (51m) 8m ago 64m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:26:18.624 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (50m) 8m ago 63m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:26:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:18 smithi121 bash[36902]: cluster 2022-04-22T18:26:17.531195+0000 mgr.y (mgr.34107) 2099 : cluster [DBG] pgmap v3532: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 992 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:26:19.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:26:19.032 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:26:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:18 smithi115 bash[45495]: cluster 2022-04-22T18:26:17.531195+0000 mgr.y (mgr.34107) 2099 : cluster [DBG] pgmap v3532: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 992 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2022-04-22T18:26:19.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:18 smithi115 bash[46934]: cluster 2022-04-22T18:26:17.531195+0000 mgr.y (mgr.34107) 2099 : cluster [DBG] pgmap v3532: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 992 KiB/s rd, 5.0 KiB/s wr, 1.34k op/s 2022-04-22T18:26:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:19 smithi121 bash[36902]: audit 2022-04-22T18:26:18.248483+0000 mgr.y (mgr.34107) 2100 : audit [DBG] from='client.37560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:19 smithi121 bash[36902]: cluster 2022-04-22T18:26:18.573566+0000 mon.a (mon.0) 4894 : cluster [DBG] osdmap e2005: 8 total, 8 up, 8 in 2022-04-22T18:26:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:19 smithi121 bash[36902]: audit 2022-04-22T18:26:18.596866+0000 mgr.y (mgr.34107) 2101 : audit [DBG] from='client.37566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:19 smithi121 bash[36902]: audit 2022-04-22T18:26:18.597330+0000 mon.a (mon.0) 4895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:19 smithi121 bash[36902]: audit 2022-04-22T18:26:19.023903+0000 mon.a (mon.0) 4896 : audit [DBG] from='client.? 172.21.15.115:0/1258413847' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[46934]: audit 2022-04-22T18:26:18.248483+0000 mgr.y (mgr.34107) 2100 : audit [DBG] from='client.37560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[46934]: cluster 2022-04-22T18:26:18.573566+0000 mon.a (mon.0) 4894 : cluster [DBG] osdmap e2005: 8 total, 8 up, 8 in 2022-04-22T18:26:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[46934]: audit 2022-04-22T18:26:18.596866+0000 mgr.y (mgr.34107) 2101 : audit [DBG] from='client.37566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[46934]: audit 2022-04-22T18:26:18.597330+0000 mon.a (mon.0) 4895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[46934]: audit 2022-04-22T18:26:19.023903+0000 mon.a (mon.0) 4896 : audit [DBG] from='client.? 172.21.15.115:0/1258413847' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[45495]: audit 2022-04-22T18:26:18.248483+0000 mgr.y (mgr.34107) 2100 : audit [DBG] from='client.37560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[45495]: cluster 2022-04-22T18:26:18.573566+0000 mon.a (mon.0) 4894 : cluster [DBG] osdmap e2005: 8 total, 8 up, 8 in 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[45495]: audit 2022-04-22T18:26:18.596866+0000 mgr.y (mgr.34107) 2101 : audit [DBG] from='client.37566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[45495]: audit 2022-04-22T18:26:18.597330+0000 mon.a (mon.0) 4895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:19 smithi115 bash[45495]: audit 2022-04-22T18:26:19.023903+0000 mon.a (mon.0) 4896 : audit [DBG] from='client.? 172.21.15.115:0/1258413847' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:20 smithi121 bash[36902]: cluster 2022-04-22T18:26:19.532292+0000 mgr.y (mgr.34107) 2102 : cluster [DBG] pgmap v3534: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:20 smithi121 bash[36902]: cluster 2022-04-22T18:26:19.591063+0000 mon.a (mon.0) 4897 : cluster [DBG] osdmap e2006: 8 total, 8 up, 8 in 2022-04-22T18:26:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:20 smithi115 bash[45495]: cluster 2022-04-22T18:26:19.532292+0000 mgr.y (mgr.34107) 2102 : cluster [DBG] pgmap v3534: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:20 smithi115 bash[45495]: cluster 2022-04-22T18:26:19.591063+0000 mon.a (mon.0) 4897 : cluster [DBG] osdmap e2006: 8 total, 8 up, 8 in 2022-04-22T18:26:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:20 smithi115 bash[46934]: cluster 2022-04-22T18:26:19.532292+0000 mgr.y (mgr.34107) 2102 : cluster [DBG] pgmap v3534: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:20 smithi115 bash[46934]: cluster 2022-04-22T18:26:19.591063+0000 mon.a (mon.0) 4897 : cluster [DBG] osdmap e2006: 8 total, 8 up, 8 in 2022-04-22T18:26:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:21 smithi121 bash[36902]: cluster 2022-04-22T18:26:20.596457+0000 mon.a (mon.0) 4898 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2022-04-22T18:26:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:21 smithi121 bash[36902]: audit 2022-04-22T18:26:21.224640+0000 mon.a (mon.0) 4899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:21 smithi121 bash[36902]: cluster 2022-04-22T18:26:21.502349+0000 mon.a (mon.0) 4900 : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2022-04-22T18:26:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:21.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[45495]: cluster 2022-04-22T18:26:20.596457+0000 mon.a (mon.0) 4898 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2022-04-22T18:26:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[45495]: audit 2022-04-22T18:26:21.224640+0000 mon.a (mon.0) 4899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[45495]: cluster 2022-04-22T18:26:21.502349+0000 mon.a (mon.0) 4900 : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2022-04-22T18:26:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[46934]: cluster 2022-04-22T18:26:20.596457+0000 mon.a (mon.0) 4898 : cluster [DBG] osdmap e2007: 8 total, 8 up, 8 in 2022-04-22T18:26:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[46934]: audit 2022-04-22T18:26:21.224640+0000 mon.a (mon.0) 4899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:21 smithi115 bash[46934]: cluster 2022-04-22T18:26:21.502349+0000 mon.a (mon.0) 4900 : cluster [DBG] osdmap e2008: 8 total, 8 up, 8 in 2022-04-22T18:26:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:22 smithi121 bash[36902]: cluster 2022-04-22T18:26:21.532830+0000 mgr.y (mgr.34107) 2103 : cluster [DBG] pgmap v3538: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 955 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2022-04-22T18:26:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:22 smithi121 bash[36902]: audit 2022-04-22T18:26:21.751083+0000 mon.a (mon.0) 4901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:22 smithi121 bash[36902]: cluster 2022-04-22T18:26:22.518229+0000 mon.a (mon.0) 4902 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2022-04-22T18:26:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[45495]: cluster 2022-04-22T18:26:21.532830+0000 mgr.y (mgr.34107) 2103 : cluster [DBG] pgmap v3538: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 955 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2022-04-22T18:26:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[45495]: audit 2022-04-22T18:26:21.751083+0000 mon.a (mon.0) 4901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[45495]: cluster 2022-04-22T18:26:22.518229+0000 mon.a (mon.0) 4902 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2022-04-22T18:26:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[46934]: cluster 2022-04-22T18:26:21.532830+0000 mgr.y (mgr.34107) 2103 : cluster [DBG] pgmap v3538: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 955 KiB/s rd, 5.0 KiB/s wr, 1.29k op/s 2022-04-22T18:26:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[46934]: audit 2022-04-22T18:26:21.751083+0000 mon.a (mon.0) 4901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:22 smithi115 bash[46934]: cluster 2022-04-22T18:26:22.518229+0000 mon.a (mon.0) 4902 : cluster [DBG] osdmap e2009: 8 total, 8 up, 8 in 2022-04-22T18:26:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:23.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[46934]: cluster 2022-04-22T18:26:23.524829+0000 mon.a (mon.0) 4903 : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2022-04-22T18:26:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[46934]: cluster 2022-04-22T18:26:23.533313+0000 mgr.y (mgr.34107) 2104 : cluster [DBG] pgmap v3541: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 971 KiB/s rd, 5.1 KiB/s wr, 1.31k op/s 2022-04-22T18:26:24.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[46934]: audit 2022-04-22T18:26:23.600074+0000 mon.a (mon.0) 4904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[45495]: cluster 2022-04-22T18:26:23.524829+0000 mon.a (mon.0) 4903 : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2022-04-22T18:26:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[45495]: cluster 2022-04-22T18:26:23.533313+0000 mgr.y (mgr.34107) 2104 : cluster [DBG] pgmap v3541: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 971 KiB/s rd, 5.1 KiB/s wr, 1.31k op/s 2022-04-22T18:26:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:24 smithi115 bash[45495]: audit 2022-04-22T18:26:23.600074+0000 mon.a (mon.0) 4904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:24 smithi121 bash[36902]: cluster 2022-04-22T18:26:23.524829+0000 mon.a (mon.0) 4903 : cluster [DBG] osdmap e2010: 8 total, 8 up, 8 in 2022-04-22T18:26:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:24 smithi121 bash[36902]: cluster 2022-04-22T18:26:23.533313+0000 mgr.y (mgr.34107) 2104 : cluster [DBG] pgmap v3541: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 971 KiB/s rd, 5.1 KiB/s wr, 1.31k op/s 2022-04-22T18:26:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:24 smithi121 bash[36902]: audit 2022-04-22T18:26:23.600074+0000 mon.a (mon.0) 4904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:25 smithi115 bash[45495]: cluster 2022-04-22T18:26:24.530154+0000 mon.a (mon.0) 4905 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2022-04-22T18:26:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:25 smithi115 bash[46934]: cluster 2022-04-22T18:26:24.530154+0000 mon.a (mon.0) 4905 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2022-04-22T18:26:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:25 smithi121 bash[36902]: cluster 2022-04-22T18:26:24.530154+0000 mon.a (mon.0) 4905 : cluster [DBG] osdmap e2011: 8 total, 8 up, 8 in 2022-04-22T18:26:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:26 smithi121 bash[36902]: cluster 2022-04-22T18:26:25.533949+0000 mgr.y (mgr.34107) 2105 : cluster [DBG] pgmap v3544: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 924 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2022-04-22T18:26:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:26 smithi121 bash[36902]: cluster 2022-04-22T18:26:25.536846+0000 mon.a (mon.0) 4906 : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2022-04-22T18:26:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:26 smithi121 bash[36902]: audit 2022-04-22T18:26:26.227921+0000 mon.a (mon.0) 4907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:26 smithi121 bash[36902]: cluster 2022-04-22T18:26:26.504787+0000 mon.a (mon.0) 4908 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2022-04-22T18:26:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[45495]: cluster 2022-04-22T18:26:25.533949+0000 mgr.y (mgr.34107) 2105 : cluster [DBG] pgmap v3544: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 924 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2022-04-22T18:26:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[45495]: cluster 2022-04-22T18:26:25.536846+0000 mon.a (mon.0) 4906 : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2022-04-22T18:26:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[45495]: audit 2022-04-22T18:26:26.227921+0000 mon.a (mon.0) 4907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[45495]: cluster 2022-04-22T18:26:26.504787+0000 mon.a (mon.0) 4908 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2022-04-22T18:26:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[46934]: cluster 2022-04-22T18:26:25.533949+0000 mgr.y (mgr.34107) 2105 : cluster [DBG] pgmap v3544: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 924 KiB/s rd, 5.0 KiB/s wr, 1.25k op/s 2022-04-22T18:26:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[46934]: cluster 2022-04-22T18:26:25.536846+0000 mon.a (mon.0) 4906 : cluster [DBG] osdmap e2012: 8 total, 8 up, 8 in 2022-04-22T18:26:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[46934]: audit 2022-04-22T18:26:26.227921+0000 mon.a (mon.0) 4907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:26 smithi115 bash[46934]: cluster 2022-04-22T18:26:26.504787+0000 mon.a (mon.0) 4908 : cluster [DBG] osdmap e2013: 8 total, 8 up, 8 in 2022-04-22T18:26:28.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:27 smithi121 bash[36902]: audit 2022-04-22T18:26:26.754851+0000 mon.a (mon.0) 4909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:27 smithi115 bash[46934]: audit 2022-04-22T18:26:26.754851+0000 mon.a (mon.0) 4909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:27 smithi115 bash[45495]: audit 2022-04-22T18:26:26.754851+0000 mon.a (mon.0) 4909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:28 smithi121 bash[36902]: cluster 2022-04-22T18:26:27.535148+0000 mgr.y (mgr.34107) 2106 : cluster [DBG] pgmap v3546: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2022-04-22T18:26:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:28 smithi121 bash[36902]: cluster 2022-04-22T18:26:27.556712+0000 mon.a (mon.0) 4910 : cluster [DBG] osdmap e2014: 8 total, 8 up, 8 in 2022-04-22T18:26:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:28 smithi115 bash[46934]: cluster 2022-04-22T18:26:27.535148+0000 mgr.y (mgr.34107) 2106 : cluster [DBG] pgmap v3546: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2022-04-22T18:26:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:28 smithi115 bash[46934]: cluster 2022-04-22T18:26:27.556712+0000 mon.a (mon.0) 4910 : cluster [DBG] osdmap e2014: 8 total, 8 up, 8 in 2022-04-22T18:26:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:28 smithi115 bash[45495]: cluster 2022-04-22T18:26:27.535148+0000 mgr.y (mgr.34107) 2106 : cluster [DBG] pgmap v3546: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 920 KiB/s rd, 5.0 KiB/s wr, 1.24k op/s 2022-04-22T18:26:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:28 smithi115 bash[45495]: cluster 2022-04-22T18:26:27.556712+0000 mon.a (mon.0) 4910 : cluster [DBG] osdmap e2014: 8 total, 8 up, 8 in 2022-04-22T18:26:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:29 smithi121 bash[36902]: cluster 2022-04-22T18:26:28.562782+0000 mon.a (mon.0) 4911 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2022-04-22T18:26:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:29 smithi121 bash[36902]: audit 2022-04-22T18:26:28.602790+0000 mon.a (mon.0) 4912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:29 smithi115 bash[46934]: cluster 2022-04-22T18:26:28.562782+0000 mon.a (mon.0) 4911 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2022-04-22T18:26:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:29 smithi115 bash[46934]: audit 2022-04-22T18:26:28.602790+0000 mon.a (mon.0) 4912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:30.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:29 smithi115 bash[45495]: cluster 2022-04-22T18:26:28.562782+0000 mon.a (mon.0) 4911 : cluster [DBG] osdmap e2015: 8 total, 8 up, 8 in 2022-04-22T18:26:30.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:29 smithi115 bash[45495]: audit 2022-04-22T18:26:28.602790+0000 mon.a (mon.0) 4912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:30 smithi121 bash[36902]: cluster 2022-04-22T18:26:29.536269+0000 mgr.y (mgr.34107) 2107 : cluster [DBG] pgmap v3549: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:26:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:30 smithi121 bash[36902]: cluster 2022-04-22T18:26:29.577096+0000 mon.a (mon.0) 4913 : cluster [DBG] osdmap e2016: 8 total, 8 up, 8 in 2022-04-22T18:26:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:30 smithi115 bash[46934]: cluster 2022-04-22T18:26:29.536269+0000 mgr.y (mgr.34107) 2107 : cluster [DBG] pgmap v3549: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:26:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:30 smithi115 bash[46934]: cluster 2022-04-22T18:26:29.577096+0000 mon.a (mon.0) 4913 : cluster [DBG] osdmap e2016: 8 total, 8 up, 8 in 2022-04-22T18:26:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:30 smithi115 bash[45495]: cluster 2022-04-22T18:26:29.536269+0000 mgr.y (mgr.34107) 2107 : cluster [DBG] pgmap v3549: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:26:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:30 smithi115 bash[45495]: cluster 2022-04-22T18:26:29.577096+0000 mon.a (mon.0) 4913 : cluster [DBG] osdmap e2016: 8 total, 8 up, 8 in 2022-04-22T18:26:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:31 smithi121 bash[36902]: cluster 2022-04-22T18:26:30.578989+0000 mon.a (mon.0) 4914 : cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2022-04-22T18:26:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:31 smithi121 bash[36902]: audit 2022-04-22T18:26:31.230972+0000 mon.a (mon.0) 4915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:31 smithi121 bash[36902]: cluster 2022-04-22T18:26:31.508700+0000 mon.a (mon.0) 4916 : cluster [DBG] osdmap e2018: 8 total, 8 up, 8 in 2022-04-22T18:26:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:31.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[45495]: cluster 2022-04-22T18:26:30.578989+0000 mon.a (mon.0) 4914 : cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2022-04-22T18:26:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[45495]: audit 2022-04-22T18:26:31.230972+0000 mon.a (mon.0) 4915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[45495]: cluster 2022-04-22T18:26:31.508700+0000 mon.a (mon.0) 4916 : cluster [DBG] osdmap e2018: 8 total, 8 up, 8 in 2022-04-22T18:26:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[46934]: cluster 2022-04-22T18:26:30.578989+0000 mon.a (mon.0) 4914 : cluster [DBG] osdmap e2017: 8 total, 8 up, 8 in 2022-04-22T18:26:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[46934]: audit 2022-04-22T18:26:31.230972+0000 mon.a (mon.0) 4915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:31 smithi115 bash[46934]: cluster 2022-04-22T18:26:31.508700+0000 mon.a (mon.0) 4916 : cluster [DBG] osdmap e2018: 8 total, 8 up, 8 in 2022-04-22T18:26:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:32 smithi121 bash[36902]: cluster 2022-04-22T18:26:31.536966+0000 mgr.y (mgr.34107) 2108 : cluster [DBG] pgmap v3553: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 894 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:32 smithi121 bash[36902]: audit 2022-04-22T18:26:31.758850+0000 mon.a (mon.0) 4917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:32 smithi121 bash[36902]: cluster 2022-04-22T18:26:32.524957+0000 mon.a (mon.0) 4918 : cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2022-04-22T18:26:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[45495]: cluster 2022-04-22T18:26:31.536966+0000 mgr.y (mgr.34107) 2108 : cluster [DBG] pgmap v3553: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 894 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[45495]: audit 2022-04-22T18:26:31.758850+0000 mon.a (mon.0) 4917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[45495]: cluster 2022-04-22T18:26:32.524957+0000 mon.a (mon.0) 4918 : cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2022-04-22T18:26:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[46934]: cluster 2022-04-22T18:26:31.536966+0000 mgr.y (mgr.34107) 2108 : cluster [DBG] pgmap v3553: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 894 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[46934]: audit 2022-04-22T18:26:31.758850+0000 mon.a (mon.0) 4917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:32 smithi115 bash[46934]: cluster 2022-04-22T18:26:32.524957+0000 mon.a (mon.0) 4918 : cluster [DBG] osdmap e2019: 8 total, 8 up, 8 in 2022-04-22T18:26:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:33.704Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:34 smithi121 bash[36902]: cluster 2022-04-22T18:26:33.538006+0000 mgr.y (mgr.34107) 2109 : cluster [DBG] pgmap v3555: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:34 smithi121 bash[36902]: cluster 2022-04-22T18:26:33.603254+0000 mon.a (mon.0) 4919 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2022-04-22T18:26:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:34 smithi121 bash[36902]: audit 2022-04-22T18:26:33.606662+0000 mon.a (mon.0) 4920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[45495]: cluster 2022-04-22T18:26:33.538006+0000 mgr.y (mgr.34107) 2109 : cluster [DBG] pgmap v3555: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[45495]: cluster 2022-04-22T18:26:33.603254+0000 mon.a (mon.0) 4919 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2022-04-22T18:26:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[45495]: audit 2022-04-22T18:26:33.606662+0000 mon.a (mon.0) 4920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[46934]: cluster 2022-04-22T18:26:33.538006+0000 mgr.y (mgr.34107) 2109 : cluster [DBG] pgmap v3555: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 892 KiB/s rd, 5.0 KiB/s wr, 1.21k op/s 2022-04-22T18:26:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[46934]: cluster 2022-04-22T18:26:33.603254+0000 mon.a (mon.0) 4919 : cluster [DBG] osdmap e2020: 8 total, 8 up, 8 in 2022-04-22T18:26:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:34 smithi115 bash[46934]: audit 2022-04-22T18:26:33.606662+0000 mon.a (mon.0) 4920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:35 smithi121 bash[36902]: cluster 2022-04-22T18:26:34.612824+0000 mon.a (mon.0) 4921 : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2022-04-22T18:26:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:35 smithi115 bash[45495]: cluster 2022-04-22T18:26:34.612824+0000 mon.a (mon.0) 4921 : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2022-04-22T18:26:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:35 smithi115 bash[46934]: cluster 2022-04-22T18:26:34.612824+0000 mon.a (mon.0) 4921 : cluster [DBG] osdmap e2021: 8 total, 8 up, 8 in 2022-04-22T18:26:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:36 smithi121 bash[36902]: cluster 2022-04-22T18:26:35.538988+0000 mgr.y (mgr.34107) 2110 : cluster [DBG] pgmap v3558: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2022-04-22T18:26:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:36 smithi121 bash[36902]: cluster 2022-04-22T18:26:35.625447+0000 mon.a (mon.0) 4922 : cluster [DBG] osdmap e2022: 8 total, 8 up, 8 in 2022-04-22T18:26:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:36 smithi121 bash[36902]: audit 2022-04-22T18:26:36.234001+0000 mon.a (mon.0) 4923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:36 smithi121 bash[36902]: cluster 2022-04-22T18:26:36.509697+0000 mon.a (mon.0) 4924 : cluster [DBG] osdmap e2023: 8 total, 8 up, 8 in 2022-04-22T18:26:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[46934]: cluster 2022-04-22T18:26:35.538988+0000 mgr.y (mgr.34107) 2110 : cluster [DBG] pgmap v3558: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2022-04-22T18:26:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[46934]: cluster 2022-04-22T18:26:35.625447+0000 mon.a (mon.0) 4922 : cluster [DBG] osdmap e2022: 8 total, 8 up, 8 in 2022-04-22T18:26:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[46934]: audit 2022-04-22T18:26:36.234001+0000 mon.a (mon.0) 4923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[46934]: cluster 2022-04-22T18:26:36.509697+0000 mon.a (mon.0) 4924 : cluster [DBG] osdmap e2023: 8 total, 8 up, 8 in 2022-04-22T18:26:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[45495]: cluster 2022-04-22T18:26:35.538988+0000 mgr.y (mgr.34107) 2110 : cluster [DBG] pgmap v3558: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 851 KiB/s rd, 5.0 KiB/s wr, 1.15k op/s 2022-04-22T18:26:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[45495]: cluster 2022-04-22T18:26:35.625447+0000 mon.a (mon.0) 4922 : cluster [DBG] osdmap e2022: 8 total, 8 up, 8 in 2022-04-22T18:26:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[45495]: audit 2022-04-22T18:26:36.234001+0000 mon.a (mon.0) 4923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:36 smithi115 bash[45495]: cluster 2022-04-22T18:26:36.509697+0000 mon.a (mon.0) 4924 : cluster [DBG] osdmap e2023: 8 total, 8 up, 8 in 2022-04-22T18:26:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:37 smithi121 bash[36902]: audit 2022-04-22T18:26:36.762666+0000 mon.a (mon.0) 4925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:37 smithi121 bash[36902]: cluster 2022-04-22T18:26:37.520726+0000 mon.a (mon.0) 4926 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2022-04-22T18:26:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:37 smithi115 bash[45495]: audit 2022-04-22T18:26:36.762666+0000 mon.a (mon.0) 4925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:37 smithi115 bash[45495]: cluster 2022-04-22T18:26:37.520726+0000 mon.a (mon.0) 4926 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2022-04-22T18:26:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:37 smithi115 bash[46934]: audit 2022-04-22T18:26:36.762666+0000 mon.a (mon.0) 4925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:37 smithi115 bash[46934]: cluster 2022-04-22T18:26:37.520726+0000 mon.a (mon.0) 4926 : cluster [DBG] osdmap e2024: 8 total, 8 up, 8 in 2022-04-22T18:26:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:38 smithi121 bash[36902]: cluster 2022-04-22T18:26:37.540061+0000 mgr.y (mgr.34107) 2111 : cluster [DBG] pgmap v3562: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 871 KiB/s rd, 5.1 KiB/s wr, 1.18k op/s 2022-04-22T18:26:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:38 smithi121 bash[36902]: cluster 2022-04-22T18:26:38.531843+0000 mon.a (mon.0) 4927 : cluster [DBG] osdmap e2025: 8 total, 8 up, 8 in 2022-04-22T18:26:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:38 smithi121 bash[36902]: audit 2022-04-22T18:26:38.609935+0000 mon.a (mon.0) 4928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[45495]: cluster 2022-04-22T18:26:37.540061+0000 mgr.y (mgr.34107) 2111 : cluster [DBG] pgmap v3562: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 871 KiB/s rd, 5.1 KiB/s wr, 1.18k op/s 2022-04-22T18:26:39.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[45495]: cluster 2022-04-22T18:26:38.531843+0000 mon.a (mon.0) 4927 : cluster [DBG] osdmap e2025: 8 total, 8 up, 8 in 2022-04-22T18:26:39.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[45495]: audit 2022-04-22T18:26:38.609935+0000 mon.a (mon.0) 4928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[46934]: cluster 2022-04-22T18:26:37.540061+0000 mgr.y (mgr.34107) 2111 : cluster [DBG] pgmap v3562: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 871 KiB/s rd, 5.1 KiB/s wr, 1.18k op/s 2022-04-22T18:26:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[46934]: cluster 2022-04-22T18:26:38.531843+0000 mon.a (mon.0) 4927 : cluster [DBG] osdmap e2025: 8 total, 8 up, 8 in 2022-04-22T18:26:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:38 smithi115 bash[46934]: audit 2022-04-22T18:26:38.609935+0000 mon.a (mon.0) 4928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:40 smithi121 bash[36902]: cluster 2022-04-22T18:26:39.534191+0000 mon.a (mon.0) 4929 : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2022-04-22T18:26:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:40 smithi121 bash[36902]: cluster 2022-04-22T18:26:39.540713+0000 mgr.y (mgr.34107) 2112 : cluster [DBG] pgmap v3565: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:40 smithi115 bash[45495]: cluster 2022-04-22T18:26:39.534191+0000 mon.a (mon.0) 4929 : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2022-04-22T18:26:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:40 smithi115 bash[45495]: cluster 2022-04-22T18:26:39.540713+0000 mgr.y (mgr.34107) 2112 : cluster [DBG] pgmap v3565: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:40 smithi115 bash[46934]: cluster 2022-04-22T18:26:39.534191+0000 mon.a (mon.0) 4929 : cluster [DBG] osdmap e2026: 8 total, 8 up, 8 in 2022-04-22T18:26:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:40 smithi115 bash[46934]: cluster 2022-04-22T18:26:39.540713+0000 mgr.y (mgr.34107) 2112 : cluster [DBG] pgmap v3565: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:26:41.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[45495]: cluster 2022-04-22T18:26:40.545165+0000 mon.a (mon.0) 4930 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2022-04-22T18:26:41.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[45495]: audit 2022-04-22T18:26:41.237265+0000 mon.a (mon.0) 4931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:41.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[45495]: cluster 2022-04-22T18:26:41.511021+0000 mon.a (mon.0) 4932 : cluster [DBG] osdmap e2028: 8 total, 8 up, 8 in 2022-04-22T18:26:41.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[46934]: cluster 2022-04-22T18:26:40.545165+0000 mon.a (mon.0) 4930 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2022-04-22T18:26:41.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[46934]: audit 2022-04-22T18:26:41.237265+0000 mon.a (mon.0) 4931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:41.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[46934]: cluster 2022-04-22T18:26:41.511021+0000 mon.a (mon.0) 4932 : cluster [DBG] osdmap e2028: 8 total, 8 up, 8 in 2022-04-22T18:26:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:41 smithi121 bash[36902]: cluster 2022-04-22T18:26:40.545165+0000 mon.a (mon.0) 4930 : cluster [DBG] osdmap e2027: 8 total, 8 up, 8 in 2022-04-22T18:26:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:41 smithi121 bash[36902]: audit 2022-04-22T18:26:41.237265+0000 mon.a (mon.0) 4931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:41 smithi121 bash[36902]: cluster 2022-04-22T18:26:41.511021+0000 mon.a (mon.0) 4932 : cluster [DBG] osdmap e2028: 8 total, 8 up, 8 in 2022-04-22T18:26:42.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:41.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:42 smithi121 bash[36902]: cluster 2022-04-22T18:26:41.541271+0000 mgr.y (mgr.34107) 2113 : cluster [DBG] pgmap v3568: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 825 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:42 smithi121 bash[36902]: audit 2022-04-22T18:26:41.766635+0000 mon.a (mon.0) 4933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:42 smithi115 bash[45495]: cluster 2022-04-22T18:26:41.541271+0000 mgr.y (mgr.34107) 2113 : cluster [DBG] pgmap v3568: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 825 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:42 smithi115 bash[45495]: audit 2022-04-22T18:26:41.766635+0000 mon.a (mon.0) 4933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:42 smithi115 bash[46934]: cluster 2022-04-22T18:26:41.541271+0000 mgr.y (mgr.34107) 2113 : cluster [DBG] pgmap v3568: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 825 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:42 smithi115 bash[46934]: audit 2022-04-22T18:26:41.766635+0000 mon.a (mon.0) 4933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:43 smithi121 bash[36902]: cluster 2022-04-22T18:26:42.559021+0000 mon.a (mon.0) 4934 : cluster [DBG] osdmap e2029: 8 total, 8 up, 8 in 2022-04-22T18:26:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:43 smithi115 bash[45495]: cluster 2022-04-22T18:26:42.559021+0000 mon.a (mon.0) 4934 : cluster [DBG] osdmap e2029: 8 total, 8 up, 8 in 2022-04-22T18:26:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:43 smithi115 bash[46934]: cluster 2022-04-22T18:26:42.559021+0000 mon.a (mon.0) 4934 : cluster [DBG] osdmap e2029: 8 total, 8 up, 8 in 2022-04-22T18:26:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:43.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:44 smithi121 bash[36902]: cluster 2022-04-22T18:26:43.542373+0000 mgr.y (mgr.34107) 2114 : cluster [DBG] pgmap v3570: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 823 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:44 smithi121 bash[36902]: cluster 2022-04-22T18:26:43.567703+0000 mon.a (mon.0) 4935 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2022-04-22T18:26:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:44 smithi121 bash[36902]: audit 2022-04-22T18:26:43.612195+0000 mon.a (mon.0) 4936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[46934]: cluster 2022-04-22T18:26:43.542373+0000 mgr.y (mgr.34107) 2114 : cluster [DBG] pgmap v3570: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 823 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[46934]: cluster 2022-04-22T18:26:43.567703+0000 mon.a (mon.0) 4935 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2022-04-22T18:26:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[46934]: audit 2022-04-22T18:26:43.612195+0000 mon.a (mon.0) 4936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[45495]: cluster 2022-04-22T18:26:43.542373+0000 mgr.y (mgr.34107) 2114 : cluster [DBG] pgmap v3570: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 823 KiB/s rd, 5.0 KiB/s wr, 1.11k op/s 2022-04-22T18:26:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[45495]: cluster 2022-04-22T18:26:43.567703+0000 mon.a (mon.0) 4935 : cluster [DBG] osdmap e2030: 8 total, 8 up, 8 in 2022-04-22T18:26:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:44 smithi115 bash[45495]: audit 2022-04-22T18:26:43.612195+0000 mon.a (mon.0) 4936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:45 smithi121 bash[36902]: cluster 2022-04-22T18:26:44.575463+0000 mon.a (mon.0) 4937 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2022-04-22T18:26:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:45 smithi115 bash[45495]: cluster 2022-04-22T18:26:44.575463+0000 mon.a (mon.0) 4937 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2022-04-22T18:26:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:45 smithi115 bash[46934]: cluster 2022-04-22T18:26:44.575463+0000 mon.a (mon.0) 4937 : cluster [DBG] osdmap e2031: 8 total, 8 up, 8 in 2022-04-22T18:26:47.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:46 smithi121 bash[36902]: cluster 2022-04-22T18:26:45.543408+0000 mgr.y (mgr.34107) 2115 : cluster [DBG] pgmap v3573: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 784 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2022-04-22T18:26:47.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:46 smithi121 bash[36902]: cluster 2022-04-22T18:26:45.585418+0000 mon.a (mon.0) 4938 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2022-04-22T18:26:47.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:46 smithi121 bash[36902]: audit 2022-04-22T18:26:46.240438+0000 mon.a (mon.0) 4939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:47.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:46 smithi121 bash[36902]: cluster 2022-04-22T18:26:46.512539+0000 mon.a (mon.0) 4940 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2022-04-22T18:26:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[45495]: cluster 2022-04-22T18:26:45.543408+0000 mgr.y (mgr.34107) 2115 : cluster [DBG] pgmap v3573: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 784 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2022-04-22T18:26:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[45495]: cluster 2022-04-22T18:26:45.585418+0000 mon.a (mon.0) 4938 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2022-04-22T18:26:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[45495]: audit 2022-04-22T18:26:46.240438+0000 mon.a (mon.0) 4939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[45495]: cluster 2022-04-22T18:26:46.512539+0000 mon.a (mon.0) 4940 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2022-04-22T18:26:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[46934]: cluster 2022-04-22T18:26:45.543408+0000 mgr.y (mgr.34107) 2115 : cluster [DBG] pgmap v3573: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 784 KiB/s rd, 5.0 KiB/s wr, 1.06k op/s 2022-04-22T18:26:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[46934]: cluster 2022-04-22T18:26:45.585418+0000 mon.a (mon.0) 4938 : cluster [DBG] osdmap e2032: 8 total, 8 up, 8 in 2022-04-22T18:26:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[46934]: audit 2022-04-22T18:26:46.240438+0000 mon.a (mon.0) 4939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:47.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:46 smithi115 bash[46934]: cluster 2022-04-22T18:26:46.512539+0000 mon.a (mon.0) 4940 : cluster [DBG] osdmap e2033: 8 total, 8 up, 8 in 2022-04-22T18:26:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:47 smithi121 bash[36902]: audit 2022-04-22T18:26:46.770852+0000 mon.a (mon.0) 4941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:47 smithi121 bash[36902]: cluster 2022-04-22T18:26:47.527606+0000 mon.a (mon.0) 4942 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2022-04-22T18:26:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:47 smithi115 bash[45495]: audit 2022-04-22T18:26:46.770852+0000 mon.a (mon.0) 4941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:47 smithi115 bash[45495]: cluster 2022-04-22T18:26:47.527606+0000 mon.a (mon.0) 4942 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2022-04-22T18:26:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:47 smithi115 bash[46934]: audit 2022-04-22T18:26:46.770852+0000 mon.a (mon.0) 4941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:47 smithi115 bash[46934]: cluster 2022-04-22T18:26:47.527606+0000 mon.a (mon.0) 4942 : cluster [DBG] osdmap e2034: 8 total, 8 up, 8 in 2022-04-22T18:26:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:48 smithi121 bash[36902]: cluster 2022-04-22T18:26:47.544129+0000 mgr.y (mgr.34107) 2116 : cluster [DBG] pgmap v3577: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 797 KiB/s rd, 5.0 KiB/s wr, 1.08k op/s 2022-04-22T18:26:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:48 smithi115 bash[45495]: cluster 2022-04-22T18:26:47.544129+0000 mgr.y (mgr.34107) 2116 : cluster [DBG] pgmap v3577: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 797 KiB/s rd, 5.0 KiB/s wr, 1.08k op/s 2022-04-22T18:26:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:48 smithi115 bash[46934]: cluster 2022-04-22T18:26:47.544129+0000 mgr.y (mgr.34107) 2116 : cluster [DBG] pgmap v3577: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 797 KiB/s rd, 5.0 KiB/s wr, 1.08k op/s 2022-04-22T18:26:49.402 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:26:49.763 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (49m) 7m ago 63m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (63m) 8m ago 63m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (51m) 7m ago 64m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (51m) 8m ago 64m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (59m) 8m ago 68m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (59m) 7m ago 71m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (58m) 7m ago 71m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (57m) 8m ago 69m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:26:49.764 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (58m) 7m ago 69m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (50m) 7m ago 63m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (50m) 8m ago 63m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (56m) 7m ago 68m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (55m) 7m ago 68m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (55m) 7m ago 67m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (55m) 7m ago 67m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (54m) 8m ago 66m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (53m) 8m ago 66m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:26:49.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (52m) 8m ago 65m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:26:49.766 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (52m) 8m ago 65m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:26:49.766 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (50m) 8m ago 64m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:26:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:49 smithi121 bash[36902]: cluster 2022-04-22T18:26:48.591732+0000 mon.a (mon.0) 4943 : cluster [DBG] osdmap e2035: 8 total, 8 up, 8 in 2022-04-22T18:26:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:49 smithi121 bash[36902]: audit 2022-04-22T18:26:48.614374+0000 mon.a (mon.0) 4944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:49 smithi115 bash[45495]: cluster 2022-04-22T18:26:48.591732+0000 mon.a (mon.0) 4943 : cluster [DBG] osdmap e2035: 8 total, 8 up, 8 in 2022-04-22T18:26:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:49 smithi115 bash[45495]: audit 2022-04-22T18:26:48.614374+0000 mon.a (mon.0) 4944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:49 smithi115 bash[46934]: cluster 2022-04-22T18:26:48.591732+0000 mon.a (mon.0) 4943 : cluster [DBG] osdmap e2035: 8 total, 8 up, 8 in 2022-04-22T18:26:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:49 smithi115 bash[46934]: audit 2022-04-22T18:26:48.614374+0000 mon.a (mon.0) 4944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:50.174 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:26:50.174 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:26:50.174 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:26:50.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:50.174 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:26:50.175 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:26:50.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:26:50.176 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:26:50.176 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:26:50.176 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:26:50.176 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:26:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:50 smithi121 bash[36902]: audit 2022-04-22T18:26:49.379731+0000 mgr.y (mgr.34107) 2117 : audit [DBG] from='client.37578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:50 smithi121 bash[36902]: cluster 2022-04-22T18:26:49.545190+0000 mgr.y (mgr.34107) 2118 : cluster [DBG] pgmap v3579: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:26:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:50 smithi121 bash[36902]: cluster 2022-04-22T18:26:49.606709+0000 mon.a (mon.0) 4945 : cluster [DBG] osdmap e2036: 8 total, 8 up, 8 in 2022-04-22T18:26:51.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:50 smithi121 bash[36902]: audit 2022-04-22T18:26:50.166812+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.115:0/1818444375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[45495]: audit 2022-04-22T18:26:49.379731+0000 mgr.y (mgr.34107) 2117 : audit [DBG] from='client.37578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[45495]: cluster 2022-04-22T18:26:49.545190+0000 mgr.y (mgr.34107) 2118 : cluster [DBG] pgmap v3579: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:26:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[45495]: cluster 2022-04-22T18:26:49.606709+0000 mon.a (mon.0) 4945 : cluster [DBG] osdmap e2036: 8 total, 8 up, 8 in 2022-04-22T18:26:51.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[45495]: audit 2022-04-22T18:26:50.166812+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.115:0/1818444375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[46934]: audit 2022-04-22T18:26:49.379731+0000 mgr.y (mgr.34107) 2117 : audit [DBG] from='client.37578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[46934]: cluster 2022-04-22T18:26:49.545190+0000 mgr.y (mgr.34107) 2118 : cluster [DBG] pgmap v3579: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:26:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[46934]: cluster 2022-04-22T18:26:49.606709+0000 mon.a (mon.0) 4945 : cluster [DBG] osdmap e2036: 8 total, 8 up, 8 in 2022-04-22T18:26:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:50 smithi115 bash[46934]: audit 2022-04-22T18:26:50.166812+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.115:0/1818444375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:26:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:51 smithi121 bash[36902]: audit 2022-04-22T18:26:49.738321+0000 mgr.y (mgr.34107) 2119 : audit [DBG] from='client.37584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:51 smithi121 bash[36902]: cluster 2022-04-22T18:26:50.619993+0000 mon.a (mon.0) 4946 : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2022-04-22T18:26:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:51 smithi121 bash[36902]: audit 2022-04-22T18:26:51.243627+0000 mon.a (mon.0) 4947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:51 smithi121 bash[36902]: cluster 2022-04-22T18:26:51.513766+0000 mon.a (mon.0) 4948 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2022-04-22T18:26:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:51.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:52.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[45495]: audit 2022-04-22T18:26:49.738321+0000 mgr.y (mgr.34107) 2119 : audit [DBG] from='client.37584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[45495]: cluster 2022-04-22T18:26:50.619993+0000 mon.a (mon.0) 4946 : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[45495]: audit 2022-04-22T18:26:51.243627+0000 mon.a (mon.0) 4947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[45495]: cluster 2022-04-22T18:26:51.513766+0000 mon.a (mon.0) 4948 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[46934]: audit 2022-04-22T18:26:49.738321+0000 mgr.y (mgr.34107) 2119 : audit [DBG] from='client.37584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[46934]: cluster 2022-04-22T18:26:50.619993+0000 mon.a (mon.0) 4946 : cluster [DBG] osdmap e2037: 8 total, 8 up, 8 in 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[46934]: audit 2022-04-22T18:26:51.243627+0000 mon.a (mon.0) 4947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:51 smithi115 bash[46934]: cluster 2022-04-22T18:26:51.513766+0000 mon.a (mon.0) 4948 : cluster [DBG] osdmap e2038: 8 total, 8 up, 8 in 2022-04-22T18:26:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:52 smithi121 bash[36902]: cluster 2022-04-22T18:26:51.545984+0000 mgr.y (mgr.34107) 2120 : cluster [DBG] pgmap v3583: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 759 KiB/s rd, 5.0 KiB/s wr, 1.03k op/s 2022-04-22T18:26:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:52 smithi121 bash[36902]: audit 2022-04-22T18:26:51.774452+0000 mon.a (mon.0) 4949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:52 smithi121 bash[36902]: cluster 2022-04-22T18:26:52.534209+0000 mon.a (mon.0) 4950 : cluster [DBG] osdmap e2039: 8 total, 8 up, 8 in 2022-04-22T18:26:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[46934]: cluster 2022-04-22T18:26:51.545984+0000 mgr.y (mgr.34107) 2120 : cluster [DBG] pgmap v3583: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 759 KiB/s rd, 5.0 KiB/s wr, 1.03k op/s 2022-04-22T18:26:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[46934]: audit 2022-04-22T18:26:51.774452+0000 mon.a (mon.0) 4949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[46934]: cluster 2022-04-22T18:26:52.534209+0000 mon.a (mon.0) 4950 : cluster [DBG] osdmap e2039: 8 total, 8 up, 8 in 2022-04-22T18:26:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[45495]: cluster 2022-04-22T18:26:51.545984+0000 mgr.y (mgr.34107) 2120 : cluster [DBG] pgmap v3583: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 759 KiB/s rd, 5.0 KiB/s wr, 1.03k op/s 2022-04-22T18:26:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[45495]: audit 2022-04-22T18:26:51.774452+0000 mon.a (mon.0) 4949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:52 smithi115 bash[45495]: cluster 2022-04-22T18:26:52.534209+0000 mon.a (mon.0) 4950 : cluster [DBG] osdmap e2039: 8 total, 8 up, 8 in 2022-04-22T18:26:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:26:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:26:53.709Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:26:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:54 smithi121 bash[36902]: cluster 2022-04-22T18:26:53.536914+0000 mon.a (mon.0) 4951 : cluster [DBG] osdmap e2040: 8 total, 8 up, 8 in 2022-04-22T18:26:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:54 smithi121 bash[36902]: cluster 2022-04-22T18:26:53.546536+0000 mgr.y (mgr.34107) 2121 : cluster [DBG] pgmap v3586: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 5.1 KiB/s wr, 1.04k op/s 2022-04-22T18:26:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:54 smithi121 bash[36902]: audit 2022-04-22T18:26:53.617420+0000 mon.a (mon.0) 4952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[45495]: cluster 2022-04-22T18:26:53.536914+0000 mon.a (mon.0) 4951 : cluster [DBG] osdmap e2040: 8 total, 8 up, 8 in 2022-04-22T18:26:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[45495]: cluster 2022-04-22T18:26:53.546536+0000 mgr.y (mgr.34107) 2121 : cluster [DBG] pgmap v3586: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 5.1 KiB/s wr, 1.04k op/s 2022-04-22T18:26:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[45495]: audit 2022-04-22T18:26:53.617420+0000 mon.a (mon.0) 4952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[46934]: cluster 2022-04-22T18:26:53.536914+0000 mon.a (mon.0) 4951 : cluster [DBG] osdmap e2040: 8 total, 8 up, 8 in 2022-04-22T18:26:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[46934]: cluster 2022-04-22T18:26:53.546536+0000 mgr.y (mgr.34107) 2121 : cluster [DBG] pgmap v3586: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 771 KiB/s rd, 5.1 KiB/s wr, 1.04k op/s 2022-04-22T18:26:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:54 smithi115 bash[46934]: audit 2022-04-22T18:26:53.617420+0000 mon.a (mon.0) 4952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:26:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:55 smithi121 bash[36902]: cluster 2022-04-22T18:26:54.542469+0000 mon.a (mon.0) 4953 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2022-04-22T18:26:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:55 smithi115 bash[45495]: cluster 2022-04-22T18:26:54.542469+0000 mon.a (mon.0) 4953 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2022-04-22T18:26:56.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:55 smithi115 bash[46934]: cluster 2022-04-22T18:26:54.542469+0000 mon.a (mon.0) 4953 : cluster [DBG] osdmap e2041: 8 total, 8 up, 8 in 2022-04-22T18:26:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:56 smithi121 bash[36902]: cluster 2022-04-22T18:26:55.547427+0000 mgr.y (mgr.34107) 2122 : cluster [DBG] pgmap v3588: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 721 KiB/s rd, 5.0 KiB/s wr, 975 op/s 2022-04-22T18:26:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:56 smithi121 bash[36902]: cluster 2022-04-22T18:26:55.552326+0000 mon.a (mon.0) 4954 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2022-04-22T18:26:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:56 smithi121 bash[36902]: audit 2022-04-22T18:26:56.246689+0000 mon.a (mon.0) 4955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:56 smithi121 bash[36902]: cluster 2022-04-22T18:26:56.518811+0000 mon.a (mon.0) 4956 : cluster [DBG] osdmap e2043: 8 total, 8 up, 8 in 2022-04-22T18:26:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[45495]: cluster 2022-04-22T18:26:55.547427+0000 mgr.y (mgr.34107) 2122 : cluster [DBG] pgmap v3588: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 721 KiB/s rd, 5.0 KiB/s wr, 975 op/s 2022-04-22T18:26:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[45495]: cluster 2022-04-22T18:26:55.552326+0000 mon.a (mon.0) 4954 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2022-04-22T18:26:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[45495]: audit 2022-04-22T18:26:56.246689+0000 mon.a (mon.0) 4955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[45495]: cluster 2022-04-22T18:26:56.518811+0000 mon.a (mon.0) 4956 : cluster [DBG] osdmap e2043: 8 total, 8 up, 8 in 2022-04-22T18:26:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[46934]: cluster 2022-04-22T18:26:55.547427+0000 mgr.y (mgr.34107) 2122 : cluster [DBG] pgmap v3588: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 721 KiB/s rd, 5.0 KiB/s wr, 975 op/s 2022-04-22T18:26:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[46934]: cluster 2022-04-22T18:26:55.552326+0000 mon.a (mon.0) 4954 : cluster [DBG] osdmap e2042: 8 total, 8 up, 8 in 2022-04-22T18:26:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[46934]: audit 2022-04-22T18:26:56.246689+0000 mon.a (mon.0) 4955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:26:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:56 smithi115 bash[46934]: cluster 2022-04-22T18:26:56.518811+0000 mon.a (mon.0) 4956 : cluster [DBG] osdmap e2043: 8 total, 8 up, 8 in 2022-04-22T18:26:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:57 smithi121 bash[36902]: audit 2022-04-22T18:26:56.778275+0000 mon.a (mon.0) 4957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:57 smithi115 bash[45495]: audit 2022-04-22T18:26:56.778275+0000 mon.a (mon.0) 4957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:57 smithi115 bash[46934]: audit 2022-04-22T18:26:56.778275+0000 mon.a (mon.0) 4957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:26:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:58 smithi121 bash[36902]: cluster 2022-04-22T18:26:57.548539+0000 mgr.y (mgr.34107) 2123 : cluster [DBG] pgmap v3591: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 726 KiB/s rd, 5.0 KiB/s wr, 981 op/s 2022-04-22T18:26:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:58 smithi121 bash[36902]: cluster 2022-04-22T18:26:57.563873+0000 mon.a (mon.0) 4958 : cluster [DBG] osdmap e2044: 8 total, 8 up, 8 in 2022-04-22T18:26:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:58 smithi115 bash[45495]: cluster 2022-04-22T18:26:57.548539+0000 mgr.y (mgr.34107) 2123 : cluster [DBG] pgmap v3591: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 726 KiB/s rd, 5.0 KiB/s wr, 981 op/s 2022-04-22T18:26:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:58 smithi115 bash[45495]: cluster 2022-04-22T18:26:57.563873+0000 mon.a (mon.0) 4958 : cluster [DBG] osdmap e2044: 8 total, 8 up, 8 in 2022-04-22T18:26:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:58 smithi115 bash[46934]: cluster 2022-04-22T18:26:57.548539+0000 mgr.y (mgr.34107) 2123 : cluster [DBG] pgmap v3591: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 726 KiB/s rd, 5.0 KiB/s wr, 981 op/s 2022-04-22T18:26:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:58 smithi115 bash[46934]: cluster 2022-04-22T18:26:57.563873+0000 mon.a (mon.0) 4958 : cluster [DBG] osdmap e2044: 8 total, 8 up, 8 in 2022-04-22T18:27:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:59 smithi121 bash[36902]: cluster 2022-04-22T18:26:58.611429+0000 mon.a (mon.0) 4959 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2022-04-22T18:27:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:26:59 smithi121 bash[36902]: audit 2022-04-22T18:26:58.620037+0000 mon.a (mon.0) 4960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:59 smithi115 bash[45495]: cluster 2022-04-22T18:26:58.611429+0000 mon.a (mon.0) 4959 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2022-04-22T18:27:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:26:59 smithi115 bash[45495]: audit 2022-04-22T18:26:58.620037+0000 mon.a (mon.0) 4960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:59 smithi115 bash[46934]: cluster 2022-04-22T18:26:58.611429+0000 mon.a (mon.0) 4959 : cluster [DBG] osdmap e2045: 8 total, 8 up, 8 in 2022-04-22T18:27:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:26:59 smithi115 bash[46934]: audit 2022-04-22T18:26:58.620037+0000 mon.a (mon.0) 4960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:00 smithi121 bash[36902]: cluster 2022-04-22T18:26:59.549540+0000 mgr.y (mgr.34107) 2124 : cluster [DBG] pgmap v3594: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:27:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:00 smithi121 bash[36902]: cluster 2022-04-22T18:26:59.617793+0000 mon.a (mon.0) 4961 : cluster [DBG] osdmap e2046: 8 total, 8 up, 8 in 2022-04-22T18:27:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:00 smithi115 bash[45495]: cluster 2022-04-22T18:26:59.549540+0000 mgr.y (mgr.34107) 2124 : cluster [DBG] pgmap v3594: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:27:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:00 smithi115 bash[45495]: cluster 2022-04-22T18:26:59.617793+0000 mon.a (mon.0) 4961 : cluster [DBG] osdmap e2046: 8 total, 8 up, 8 in 2022-04-22T18:27:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:00 smithi115 bash[46934]: cluster 2022-04-22T18:26:59.549540+0000 mgr.y (mgr.34107) 2124 : cluster [DBG] pgmap v3594: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2022-04-22T18:27:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:00 smithi115 bash[46934]: cluster 2022-04-22T18:26:59.617793+0000 mon.a (mon.0) 4961 : cluster [DBG] osdmap e2046: 8 total, 8 up, 8 in 2022-04-22T18:27:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:01 smithi121 bash[36902]: cluster 2022-04-22T18:27:00.626830+0000 mon.a (mon.0) 4962 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2022-04-22T18:27:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:01 smithi121 bash[36902]: audit 2022-04-22T18:27:01.249790+0000 mon.a (mon.0) 4963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:01 smithi121 bash[36902]: cluster 2022-04-22T18:27:01.521821+0000 mon.a (mon.0) 4964 : cluster [DBG] osdmap e2048: 8 total, 8 up, 8 in 2022-04-22T18:27:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:01.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[45495]: cluster 2022-04-22T18:27:00.626830+0000 mon.a (mon.0) 4962 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2022-04-22T18:27:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[45495]: audit 2022-04-22T18:27:01.249790+0000 mon.a (mon.0) 4963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[45495]: cluster 2022-04-22T18:27:01.521821+0000 mon.a (mon.0) 4964 : cluster [DBG] osdmap e2048: 8 total, 8 up, 8 in 2022-04-22T18:27:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[46934]: cluster 2022-04-22T18:27:00.626830+0000 mon.a (mon.0) 4962 : cluster [DBG] osdmap e2047: 8 total, 8 up, 8 in 2022-04-22T18:27:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[46934]: audit 2022-04-22T18:27:01.249790+0000 mon.a (mon.0) 4963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:01 smithi115 bash[46934]: cluster 2022-04-22T18:27:01.521821+0000 mon.a (mon.0) 4964 : cluster [DBG] osdmap e2048: 8 total, 8 up, 8 in 2022-04-22T18:27:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:02 smithi121 bash[36902]: cluster 2022-04-22T18:27:01.550147+0000 mgr.y (mgr.34107) 2125 : cluster [DBG] pgmap v3598: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 696 KiB/s rd, 5.0 KiB/s wr, 942 op/s 2022-04-22T18:27:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:02 smithi121 bash[36902]: audit 2022-04-22T18:27:01.781826+0000 mon.a (mon.0) 4965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:02 smithi121 bash[36902]: cluster 2022-04-22T18:27:02.540141+0000 mon.a (mon.0) 4966 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2022-04-22T18:27:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[46934]: cluster 2022-04-22T18:27:01.550147+0000 mgr.y (mgr.34107) 2125 : cluster [DBG] pgmap v3598: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 696 KiB/s rd, 5.0 KiB/s wr, 942 op/s 2022-04-22T18:27:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[46934]: audit 2022-04-22T18:27:01.781826+0000 mon.a (mon.0) 4965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[46934]: cluster 2022-04-22T18:27:02.540141+0000 mon.a (mon.0) 4966 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2022-04-22T18:27:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[45495]: cluster 2022-04-22T18:27:01.550147+0000 mgr.y (mgr.34107) 2125 : cluster [DBG] pgmap v3598: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 696 KiB/s rd, 5.0 KiB/s wr, 942 op/s 2022-04-22T18:27:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[45495]: audit 2022-04-22T18:27:01.781826+0000 mon.a (mon.0) 4965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:02 smithi115 bash[45495]: cluster 2022-04-22T18:27:02.540141+0000 mon.a (mon.0) 4966 : cluster [DBG] osdmap e2049: 8 total, 8 up, 8 in 2022-04-22T18:27:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:03.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:04.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: cluster 2022-04-22T18:27:03.544286+0000 mon.a (mon.0) 4967 : cluster [DBG] osdmap e2050: 8 total, 8 up, 8 in 2022-04-22T18:27:04.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: cluster 2022-04-22T18:27:03.550742+0000 mgr.y (mgr.34107) 2126 : cluster [DBG] pgmap v3601: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 707 KiB/s rd, 5.1 KiB/s wr, 956 op/s 2022-04-22T18:27:04.687 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: audit 2022-04-22T18:27:03.622438+0000 mon.a (mon.0) 4968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: audit 2022-04-22T18:27:04.348937+0000 mon.a (mon.0) 4969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: audit 2022-04-22T18:27:04.351854+0000 mon.a (mon.0) 4970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[45495]: audit 2022-04-22T18:27:04.353567+0000 mon.a (mon.0) 4971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: cluster 2022-04-22T18:27:03.544286+0000 mon.a (mon.0) 4967 : cluster [DBG] osdmap e2050: 8 total, 8 up, 8 in 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: cluster 2022-04-22T18:27:03.550742+0000 mgr.y (mgr.34107) 2126 : cluster [DBG] pgmap v3601: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 707 KiB/s rd, 5.1 KiB/s wr, 956 op/s 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: audit 2022-04-22T18:27:03.622438+0000 mon.a (mon.0) 4968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:04.688 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: audit 2022-04-22T18:27:04.348937+0000 mon.a (mon.0) 4969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:27:04.689 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: audit 2022-04-22T18:27:04.351854+0000 mon.a (mon.0) 4970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:27:04.689 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:04 smithi115 bash[46934]: audit 2022-04-22T18:27:04.353567+0000 mon.a (mon.0) 4971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: cluster 2022-04-22T18:27:03.544286+0000 mon.a (mon.0) 4967 : cluster [DBG] osdmap e2050: 8 total, 8 up, 8 in 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: cluster 2022-04-22T18:27:03.550742+0000 mgr.y (mgr.34107) 2126 : cluster [DBG] pgmap v3601: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 707 KiB/s rd, 5.1 KiB/s wr, 956 op/s 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: audit 2022-04-22T18:27:03.622438+0000 mon.a (mon.0) 4968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: audit 2022-04-22T18:27:04.348937+0000 mon.a (mon.0) 4969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: audit 2022-04-22T18:27:04.351854+0000 mon.a (mon.0) 4970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:27:04.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:04 smithi121 bash[36902]: audit 2022-04-22T18:27:04.353567+0000 mon.a (mon.0) 4971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:27:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:05 smithi121 bash[36902]: cluster 2022-04-22T18:27:04.554777+0000 mon.a (mon.0) 4972 : cluster [DBG] osdmap e2051: 8 total, 8 up, 8 in 2022-04-22T18:27:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:05 smithi121 bash[36902]: audit 2022-04-22T18:27:04.707741+0000 mon.a (mon.0) 4973 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:27:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:05 smithi115 bash[45495]: cluster 2022-04-22T18:27:04.554777+0000 mon.a (mon.0) 4972 : cluster [DBG] osdmap e2051: 8 total, 8 up, 8 in 2022-04-22T18:27:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:05 smithi115 bash[45495]: audit 2022-04-22T18:27:04.707741+0000 mon.a (mon.0) 4973 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:27:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:05 smithi115 bash[46934]: cluster 2022-04-22T18:27:04.554777+0000 mon.a (mon.0) 4972 : cluster [DBG] osdmap e2051: 8 total, 8 up, 8 in 2022-04-22T18:27:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:05 smithi115 bash[46934]: audit 2022-04-22T18:27:04.707741+0000 mon.a (mon.0) 4973 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:27:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:06 smithi121 bash[36902]: cluster 2022-04-22T18:27:05.551902+0000 mgr.y (mgr.34107) 2127 : cluster [DBG] pgmap v3603: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 657 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2022-04-22T18:27:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:06 smithi121 bash[36902]: cluster 2022-04-22T18:27:05.564233+0000 mon.a (mon.0) 4974 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2022-04-22T18:27:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:06 smithi121 bash[36902]: audit 2022-04-22T18:27:06.252913+0000 mon.a (mon.0) 4975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:06 smithi121 bash[36902]: cluster 2022-04-22T18:27:06.524024+0000 mon.a (mon.0) 4976 : cluster [DBG] osdmap e2053: 8 total, 8 up, 8 in 2022-04-22T18:27:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[45495]: cluster 2022-04-22T18:27:05.551902+0000 mgr.y (mgr.34107) 2127 : cluster [DBG] pgmap v3603: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 657 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2022-04-22T18:27:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[45495]: cluster 2022-04-22T18:27:05.564233+0000 mon.a (mon.0) 4974 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2022-04-22T18:27:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[45495]: audit 2022-04-22T18:27:06.252913+0000 mon.a (mon.0) 4975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[45495]: cluster 2022-04-22T18:27:06.524024+0000 mon.a (mon.0) 4976 : cluster [DBG] osdmap e2053: 8 total, 8 up, 8 in 2022-04-22T18:27:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[46934]: cluster 2022-04-22T18:27:05.551902+0000 mgr.y (mgr.34107) 2127 : cluster [DBG] pgmap v3603: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 657 KiB/s rd, 5.0 KiB/s wr, 889 op/s 2022-04-22T18:27:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[46934]: cluster 2022-04-22T18:27:05.564233+0000 mon.a (mon.0) 4974 : cluster [DBG] osdmap e2052: 8 total, 8 up, 8 in 2022-04-22T18:27:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[46934]: audit 2022-04-22T18:27:06.252913+0000 mon.a (mon.0) 4975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:06 smithi115 bash[46934]: cluster 2022-04-22T18:27:06.524024+0000 mon.a (mon.0) 4976 : cluster [DBG] osdmap e2053: 8 total, 8 up, 8 in 2022-04-22T18:27:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:07 smithi121 bash[36902]: audit 2022-04-22T18:27:06.785322+0000 mon.a (mon.0) 4977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:07 smithi115 bash[45495]: audit 2022-04-22T18:27:06.785322+0000 mon.a (mon.0) 4977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:07 smithi115 bash[46934]: audit 2022-04-22T18:27:06.785322+0000 mon.a (mon.0) 4977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:08 smithi121 bash[36902]: cluster 2022-04-22T18:27:07.553095+0000 mgr.y (mgr.34107) 2128 : cluster [DBG] pgmap v3606: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 660 KiB/s rd, 5.0 KiB/s wr, 893 op/s 2022-04-22T18:27:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:08 smithi121 bash[36902]: cluster 2022-04-22T18:27:07.578414+0000 mon.a (mon.0) 4978 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2022-04-22T18:27:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:08 smithi115 bash[45495]: cluster 2022-04-22T18:27:07.553095+0000 mgr.y (mgr.34107) 2128 : cluster [DBG] pgmap v3606: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 660 KiB/s rd, 5.0 KiB/s wr, 893 op/s 2022-04-22T18:27:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:08 smithi115 bash[45495]: cluster 2022-04-22T18:27:07.578414+0000 mon.a (mon.0) 4978 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2022-04-22T18:27:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:08 smithi115 bash[46934]: cluster 2022-04-22T18:27:07.553095+0000 mgr.y (mgr.34107) 2128 : cluster [DBG] pgmap v3606: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 660 KiB/s rd, 5.0 KiB/s wr, 893 op/s 2022-04-22T18:27:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:08 smithi115 bash[46934]: cluster 2022-04-22T18:27:07.578414+0000 mon.a (mon.0) 4978 : cluster [DBG] osdmap e2054: 8 total, 8 up, 8 in 2022-04-22T18:27:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:09 smithi121 bash[36902]: cluster 2022-04-22T18:27:08.588003+0000 mon.a (mon.0) 4979 : cluster [DBG] osdmap e2055: 8 total, 8 up, 8 in 2022-04-22T18:27:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:09 smithi121 bash[36902]: audit 2022-04-22T18:27:08.625283+0000 mon.a (mon.0) 4980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:09 smithi115 bash[45495]: cluster 2022-04-22T18:27:08.588003+0000 mon.a (mon.0) 4979 : cluster [DBG] osdmap e2055: 8 total, 8 up, 8 in 2022-04-22T18:27:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:09 smithi115 bash[45495]: audit 2022-04-22T18:27:08.625283+0000 mon.a (mon.0) 4980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:10.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:09 smithi115 bash[46934]: cluster 2022-04-22T18:27:08.588003+0000 mon.a (mon.0) 4979 : cluster [DBG] osdmap e2055: 8 total, 8 up, 8 in 2022-04-22T18:27:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:09 smithi115 bash[46934]: audit 2022-04-22T18:27:08.625283+0000 mon.a (mon.0) 4980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:10 smithi121 bash[36902]: cluster 2022-04-22T18:27:09.554179+0000 mgr.y (mgr.34107) 2129 : cluster [DBG] pgmap v3609: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:27:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:10 smithi121 bash[36902]: cluster 2022-04-22T18:27:09.602659+0000 mon.a (mon.0) 4981 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2022-04-22T18:27:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:10 smithi115 bash[45495]: cluster 2022-04-22T18:27:09.554179+0000 mgr.y (mgr.34107) 2129 : cluster [DBG] pgmap v3609: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:27:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:10 smithi115 bash[45495]: cluster 2022-04-22T18:27:09.602659+0000 mon.a (mon.0) 4981 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2022-04-22T18:27:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:10 smithi115 bash[46934]: cluster 2022-04-22T18:27:09.554179+0000 mgr.y (mgr.34107) 2129 : cluster [DBG] pgmap v3609: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2022-04-22T18:27:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:10 smithi115 bash[46934]: cluster 2022-04-22T18:27:09.602659+0000 mon.a (mon.0) 4981 : cluster [DBG] osdmap e2056: 8 total, 8 up, 8 in 2022-04-22T18:27:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:11 smithi121 bash[36902]: cluster 2022-04-22T18:27:10.606536+0000 mon.a (mon.0) 4982 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2022-04-22T18:27:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:11 smithi121 bash[36902]: audit 2022-04-22T18:27:11.256012+0000 mon.a (mon.0) 4983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:11 smithi121 bash[36902]: cluster 2022-04-22T18:27:11.529158+0000 mon.a (mon.0) 4984 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2022-04-22T18:27:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:11.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[45495]: cluster 2022-04-22T18:27:10.606536+0000 mon.a (mon.0) 4982 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2022-04-22T18:27:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[45495]: audit 2022-04-22T18:27:11.256012+0000 mon.a (mon.0) 4983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[45495]: cluster 2022-04-22T18:27:11.529158+0000 mon.a (mon.0) 4984 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2022-04-22T18:27:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[46934]: cluster 2022-04-22T18:27:10.606536+0000 mon.a (mon.0) 4982 : cluster [DBG] osdmap e2057: 8 total, 8 up, 8 in 2022-04-22T18:27:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[46934]: audit 2022-04-22T18:27:11.256012+0000 mon.a (mon.0) 4983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:11 smithi115 bash[46934]: cluster 2022-04-22T18:27:11.529158+0000 mon.a (mon.0) 4984 : cluster [DBG] osdmap e2058: 8 total, 8 up, 8 in 2022-04-22T18:27:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:12 smithi121 bash[36902]: cluster 2022-04-22T18:27:11.555002+0000 mgr.y (mgr.34107) 2130 : cluster [DBG] pgmap v3613: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 632 KiB/s rd, 5.0 KiB/s wr, 857 op/s 2022-04-22T18:27:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:12 smithi121 bash[36902]: audit 2022-04-22T18:27:11.788987+0000 mon.a (mon.0) 4985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:12 smithi121 bash[36902]: cluster 2022-04-22T18:27:12.547680+0000 mon.a (mon.0) 4986 : cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2022-04-22T18:27:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[45495]: cluster 2022-04-22T18:27:11.555002+0000 mgr.y (mgr.34107) 2130 : cluster [DBG] pgmap v3613: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 632 KiB/s rd, 5.0 KiB/s wr, 857 op/s 2022-04-22T18:27:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[45495]: audit 2022-04-22T18:27:11.788987+0000 mon.a (mon.0) 4985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[45495]: cluster 2022-04-22T18:27:12.547680+0000 mon.a (mon.0) 4986 : cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2022-04-22T18:27:13.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[46934]: cluster 2022-04-22T18:27:11.555002+0000 mgr.y (mgr.34107) 2130 : cluster [DBG] pgmap v3613: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 632 KiB/s rd, 5.0 KiB/s wr, 857 op/s 2022-04-22T18:27:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[46934]: audit 2022-04-22T18:27:11.788987+0000 mon.a (mon.0) 4985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:12 smithi115 bash[46934]: cluster 2022-04-22T18:27:12.547680+0000 mon.a (mon.0) 4986 : cluster [DBG] osdmap e2059: 8 total, 8 up, 8 in 2022-04-22T18:27:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:13.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:14 smithi121 bash[36902]: cluster 2022-04-22T18:27:13.554288+0000 mon.a (mon.0) 4987 : cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2022-04-22T18:27:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:14 smithi121 bash[36902]: cluster 2022-04-22T18:27:13.559956+0000 mgr.y (mgr.34107) 2131 : cluster [DBG] pgmap v3616: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 635 KiB/s rd, 5.0 KiB/s wr, 859 op/s 2022-04-22T18:27:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:14 smithi121 bash[36902]: audit 2022-04-22T18:27:13.628098+0000 mon.a (mon.0) 4988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:14 smithi121 bash[36902]: audit 2022-04-22T18:27:13.986891+0000 mon.a (mon.0) 4989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:27:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:14 smithi121 bash[36902]: audit 2022-04-22T18:27:13.987354+0000 mon.a (mon.0) 4990 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:27:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[45495]: cluster 2022-04-22T18:27:13.554288+0000 mon.a (mon.0) 4987 : cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2022-04-22T18:27:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[45495]: cluster 2022-04-22T18:27:13.559956+0000 mgr.y (mgr.34107) 2131 : cluster [DBG] pgmap v3616: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 635 KiB/s rd, 5.0 KiB/s wr, 859 op/s 2022-04-22T18:27:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[45495]: audit 2022-04-22T18:27:13.628098+0000 mon.a (mon.0) 4988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[45495]: audit 2022-04-22T18:27:13.986891+0000 mon.a (mon.0) 4989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:27:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[45495]: audit 2022-04-22T18:27:13.987354+0000 mon.a (mon.0) 4990 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:27:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[46934]: cluster 2022-04-22T18:27:13.554288+0000 mon.a (mon.0) 4987 : cluster [DBG] osdmap e2060: 8 total, 8 up, 8 in 2022-04-22T18:27:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[46934]: cluster 2022-04-22T18:27:13.559956+0000 mgr.y (mgr.34107) 2131 : cluster [DBG] pgmap v3616: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 635 KiB/s rd, 5.0 KiB/s wr, 859 op/s 2022-04-22T18:27:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[46934]: audit 2022-04-22T18:27:13.628098+0000 mon.a (mon.0) 4988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[46934]: audit 2022-04-22T18:27:13.986891+0000 mon.a (mon.0) 4989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:27:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:14 smithi115 bash[46934]: audit 2022-04-22T18:27:13.987354+0000 mon.a (mon.0) 4990 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:27:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:15 smithi121 bash[36902]: cluster 2022-04-22T18:27:14.560501+0000 mon.a (mon.0) 4991 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2022-04-22T18:27:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:15 smithi115 bash[45495]: cluster 2022-04-22T18:27:14.560501+0000 mon.a (mon.0) 4991 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2022-04-22T18:27:16.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:15 smithi115 bash[46934]: cluster 2022-04-22T18:27:14.560501+0000 mon.a (mon.0) 4991 : cluster [DBG] osdmap e2061: 8 total, 8 up, 8 in 2022-04-22T18:27:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:16 smithi121 bash[36902]: cluster 2022-04-22T18:27:15.561040+0000 mgr.y (mgr.34107) 2132 : cluster [DBG] pgmap v3618: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2022-04-22T18:27:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:16 smithi121 bash[36902]: cluster 2022-04-22T18:27:15.567748+0000 mon.a (mon.0) 4992 : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2022-04-22T18:27:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:16 smithi121 bash[36902]: audit 2022-04-22T18:27:16.259060+0000 mon.a (mon.0) 4993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:16 smithi121 bash[36902]: cluster 2022-04-22T18:27:16.530156+0000 mon.a (mon.0) 4994 : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2022-04-22T18:27:17.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[45495]: cluster 2022-04-22T18:27:15.561040+0000 mgr.y (mgr.34107) 2132 : cluster [DBG] pgmap v3618: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2022-04-22T18:27:17.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[45495]: cluster 2022-04-22T18:27:15.567748+0000 mon.a (mon.0) 4992 : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2022-04-22T18:27:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[45495]: audit 2022-04-22T18:27:16.259060+0000 mon.a (mon.0) 4993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[45495]: cluster 2022-04-22T18:27:16.530156+0000 mon.a (mon.0) 4994 : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2022-04-22T18:27:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[46934]: cluster 2022-04-22T18:27:15.561040+0000 mgr.y (mgr.34107) 2132 : cluster [DBG] pgmap v3618: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 KiB/s rd, 5.0 KiB/s wr, 802 op/s 2022-04-22T18:27:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[46934]: cluster 2022-04-22T18:27:15.567748+0000 mon.a (mon.0) 4992 : cluster [DBG] osdmap e2062: 8 total, 8 up, 8 in 2022-04-22T18:27:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[46934]: audit 2022-04-22T18:27:16.259060+0000 mon.a (mon.0) 4993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:16 smithi115 bash[46934]: cluster 2022-04-22T18:27:16.530156+0000 mon.a (mon.0) 4994 : cluster [DBG] osdmap e2063: 8 total, 8 up, 8 in 2022-04-22T18:27:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:17 smithi121 bash[36902]: audit 2022-04-22T18:27:16.792745+0000 mon.a (mon.0) 4995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:17 smithi115 bash[45495]: audit 2022-04-22T18:27:16.792745+0000 mon.a (mon.0) 4995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:18.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:17 smithi115 bash[46934]: audit 2022-04-22T18:27:16.792745+0000 mon.a (mon.0) 4995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:18 smithi121 bash[36902]: cluster 2022-04-22T18:27:17.562176+0000 mgr.y (mgr.34107) 2133 : cluster [DBG] pgmap v3621: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 596 KiB/s rd, 5.0 KiB/s wr, 807 op/s 2022-04-22T18:27:19.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:18 smithi121 bash[36902]: cluster 2022-04-22T18:27:17.579073+0000 mon.a (mon.0) 4996 : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2022-04-22T18:27:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:18 smithi115 bash[45495]: cluster 2022-04-22T18:27:17.562176+0000 mgr.y (mgr.34107) 2133 : cluster [DBG] pgmap v3621: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 596 KiB/s rd, 5.0 KiB/s wr, 807 op/s 2022-04-22T18:27:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:18 smithi115 bash[45495]: cluster 2022-04-22T18:27:17.579073+0000 mon.a (mon.0) 4996 : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2022-04-22T18:27:19.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:18 smithi115 bash[46934]: cluster 2022-04-22T18:27:17.562176+0000 mgr.y (mgr.34107) 2133 : cluster [DBG] pgmap v3621: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 596 KiB/s rd, 5.0 KiB/s wr, 807 op/s 2022-04-22T18:27:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:18 smithi115 bash[46934]: cluster 2022-04-22T18:27:17.579073+0000 mon.a (mon.0) 4996 : cluster [DBG] osdmap e2064: 8 total, 8 up, 8 in 2022-04-22T18:27:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:19 smithi121 bash[36902]: cluster 2022-04-22T18:27:18.587528+0000 mon.a (mon.0) 4997 : cluster [DBG] osdmap e2065: 8 total, 8 up, 8 in 2022-04-22T18:27:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:19 smithi121 bash[36902]: audit 2022-04-22T18:27:18.631440+0000 mon.a (mon.0) 4998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:20.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:19 smithi115 bash[45495]: cluster 2022-04-22T18:27:18.587528+0000 mon.a (mon.0) 4997 : cluster [DBG] osdmap e2065: 8 total, 8 up, 8 in 2022-04-22T18:27:20.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:19 smithi115 bash[45495]: audit 2022-04-22T18:27:18.631440+0000 mon.a (mon.0) 4998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:19 smithi115 bash[46934]: cluster 2022-04-22T18:27:18.587528+0000 mon.a (mon.0) 4997 : cluster [DBG] osdmap e2065: 8 total, 8 up, 8 in 2022-04-22T18:27:20.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:19 smithi115 bash[46934]: audit 2022-04-22T18:27:18.631440+0000 mon.a (mon.0) 4998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:20.544 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (50m) 8m ago 64m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (63m) 9m ago 63m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (52m) 8m ago 65m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (51m) 9m ago 65m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (59m) 9m ago 69m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:27:20.885 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (60m) 8m ago 71m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (59m) 8m ago 71m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (58m) 9m ago 69m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (58m) 8m ago 69m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (51m) 8m ago 64m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (51m) 9m ago 64m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (57m) 8m ago 69m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (56m) 8m ago 68m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (55m) 8m ago 68m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (55m) 8m ago 67m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:27:20.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (54m) 9m ago 67m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:27:20.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (54m) 9m ago 66m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:27:20.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (53m) 9m ago 66m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:27:20.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (52m) 9m ago 65m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:27:20.887 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (51m) 9m ago 64m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:27:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:20 smithi121 bash[36902]: cluster 2022-04-22T18:27:19.563212+0000 mgr.y (mgr.34107) 2134 : cluster [DBG] pgmap v3624: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:20 smithi121 bash[36902]: cluster 2022-04-22T18:27:19.599650+0000 mon.a (mon.0) 4999 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2022-04-22T18:27:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:20 smithi115 bash[45495]: cluster 2022-04-22T18:27:19.563212+0000 mgr.y (mgr.34107) 2134 : cluster [DBG] pgmap v3624: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:20 smithi115 bash[45495]: cluster 2022-04-22T18:27:19.599650+0000 mon.a (mon.0) 4999 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2022-04-22T18:27:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:20 smithi115 bash[46934]: cluster 2022-04-22T18:27:19.563212+0000 mgr.y (mgr.34107) 2134 : cluster [DBG] pgmap v3624: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:20 smithi115 bash[46934]: cluster 2022-04-22T18:27:19.599650+0000 mon.a (mon.0) 4999 : cluster [DBG] osdmap e2066: 8 total, 8 up, 8 in 2022-04-22T18:27:21.291 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:27:21.291 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:27:21.291 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:27:21.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:21.292 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:27:21.293 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:27:21.293 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:27:21.293 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:27:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:21 smithi121 bash[36902]: audit 2022-04-22T18:27:20.522492+0000 mgr.y (mgr.34107) 2135 : audit [DBG] from='client.47530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:21 smithi121 bash[36902]: cluster 2022-04-22T18:27:20.609377+0000 mon.a (mon.0) 5000 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2022-04-22T18:27:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:21 smithi121 bash[36902]: audit 2022-04-22T18:27:21.262129+0000 mon.a (mon.0) 5001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:21 smithi121 bash[36902]: audit 2022-04-22T18:27:21.284272+0000 mon.a (mon.0) 5002 : audit [DBG] from='client.? 172.21.15.115:0/3653802800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:21 smithi121 bash[36902]: cluster 2022-04-22T18:27:21.532353+0000 mon.a (mon.0) 5003 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2022-04-22T18:27:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:21.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[45495]: audit 2022-04-22T18:27:20.522492+0000 mgr.y (mgr.34107) 2135 : audit [DBG] from='client.47530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[45495]: cluster 2022-04-22T18:27:20.609377+0000 mon.a (mon.0) 5000 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[45495]: audit 2022-04-22T18:27:21.262129+0000 mon.a (mon.0) 5001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[45495]: audit 2022-04-22T18:27:21.284272+0000 mon.a (mon.0) 5002 : audit [DBG] from='client.? 172.21.15.115:0/3653802800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[45495]: cluster 2022-04-22T18:27:21.532353+0000 mon.a (mon.0) 5003 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[46934]: audit 2022-04-22T18:27:20.522492+0000 mgr.y (mgr.34107) 2135 : audit [DBG] from='client.47530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[46934]: cluster 2022-04-22T18:27:20.609377+0000 mon.a (mon.0) 5000 : cluster [DBG] osdmap e2067: 8 total, 8 up, 8 in 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[46934]: audit 2022-04-22T18:27:21.262129+0000 mon.a (mon.0) 5001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[46934]: audit 2022-04-22T18:27:21.284272+0000 mon.a (mon.0) 5002 : audit [DBG] from='client.? 172.21.15.115:0/3653802800' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:21 smithi115 bash[46934]: cluster 2022-04-22T18:27:21.532353+0000 mon.a (mon.0) 5003 : cluster [DBG] osdmap e2068: 8 total, 8 up, 8 in 2022-04-22T18:27:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:22 smithi121 bash[36902]: audit 2022-04-22T18:27:20.858375+0000 mgr.y (mgr.34107) 2136 : audit [DBG] from='client.37602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:22 smithi121 bash[36902]: cluster 2022-04-22T18:27:21.564021+0000 mgr.y (mgr.34107) 2137 : cluster [DBG] pgmap v3628: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 565 KiB/s rd, 5.0 KiB/s wr, 767 op/s 2022-04-22T18:27:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:22 smithi121 bash[36902]: audit 2022-04-22T18:27:21.796425+0000 mon.a (mon.0) 5004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:22 smithi121 bash[36902]: cluster 2022-04-22T18:27:22.551150+0000 mon.a (mon.0) 5005 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2022-04-22T18:27:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[45495]: audit 2022-04-22T18:27:20.858375+0000 mgr.y (mgr.34107) 2136 : audit [DBG] from='client.37602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[45495]: cluster 2022-04-22T18:27:21.564021+0000 mgr.y (mgr.34107) 2137 : cluster [DBG] pgmap v3628: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 565 KiB/s rd, 5.0 KiB/s wr, 767 op/s 2022-04-22T18:27:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[45495]: audit 2022-04-22T18:27:21.796425+0000 mon.a (mon.0) 5004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[45495]: cluster 2022-04-22T18:27:22.551150+0000 mon.a (mon.0) 5005 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2022-04-22T18:27:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[46934]: audit 2022-04-22T18:27:20.858375+0000 mgr.y (mgr.34107) 2136 : audit [DBG] from='client.37602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[46934]: cluster 2022-04-22T18:27:21.564021+0000 mgr.y (mgr.34107) 2137 : cluster [DBG] pgmap v3628: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 565 KiB/s rd, 5.0 KiB/s wr, 767 op/s 2022-04-22T18:27:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[46934]: audit 2022-04-22T18:27:21.796425+0000 mon.a (mon.0) 5004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:22 smithi115 bash[46934]: cluster 2022-04-22T18:27:22.551150+0000 mon.a (mon.0) 5005 : cluster [DBG] osdmap e2069: 8 total, 8 up, 8 in 2022-04-22T18:27:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:23.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:24 smithi121 bash[36902]: cluster 2022-04-22T18:27:23.554757+0000 mon.a (mon.0) 5006 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2022-04-22T18:27:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:24 smithi121 bash[36902]: cluster 2022-04-22T18:27:23.564580+0000 mgr.y (mgr.34107) 2138 : cluster [DBG] pgmap v3631: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 568 KiB/s rd, 5.0 KiB/s wr, 770 op/s 2022-04-22T18:27:25.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:24 smithi121 bash[36902]: audit 2022-04-22T18:27:23.634096+0000 mon.a (mon.0) 5007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[45495]: cluster 2022-04-22T18:27:23.554757+0000 mon.a (mon.0) 5006 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2022-04-22T18:27:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[45495]: cluster 2022-04-22T18:27:23.564580+0000 mgr.y (mgr.34107) 2138 : cluster [DBG] pgmap v3631: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 568 KiB/s rd, 5.0 KiB/s wr, 770 op/s 2022-04-22T18:27:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[45495]: audit 2022-04-22T18:27:23.634096+0000 mon.a (mon.0) 5007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[46934]: cluster 2022-04-22T18:27:23.554757+0000 mon.a (mon.0) 5006 : cluster [DBG] osdmap e2070: 8 total, 8 up, 8 in 2022-04-22T18:27:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[46934]: cluster 2022-04-22T18:27:23.564580+0000 mgr.y (mgr.34107) 2138 : cluster [DBG] pgmap v3631: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 568 KiB/s rd, 5.0 KiB/s wr, 770 op/s 2022-04-22T18:27:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:24 smithi115 bash[46934]: audit 2022-04-22T18:27:23.634096+0000 mon.a (mon.0) 5007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:25 smithi121 bash[36902]: cluster 2022-04-22T18:27:24.562962+0000 mon.a (mon.0) 5008 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2022-04-22T18:27:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:25 smithi115 bash[45495]: cluster 2022-04-22T18:27:24.562962+0000 mon.a (mon.0) 5008 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2022-04-22T18:27:26.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:25 smithi115 bash[46934]: cluster 2022-04-22T18:27:24.562962+0000 mon.a (mon.0) 5008 : cluster [DBG] osdmap e2071: 8 total, 8 up, 8 in 2022-04-22T18:27:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:26 smithi121 bash[36902]: cluster 2022-04-22T18:27:25.567337+0000 mgr.y (mgr.34107) 2139 : cluster [DBG] pgmap v3634: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 532 KiB/s rd, 5.0 KiB/s wr, 720 op/s 2022-04-22T18:27:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:26 smithi121 bash[36902]: cluster 2022-04-22T18:27:25.567445+0000 mon.a (mon.0) 5009 : cluster [DBG] osdmap e2072: 8 total, 8 up, 8 in 2022-04-22T18:27:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:26 smithi121 bash[36902]: audit 2022-04-22T18:27:26.265312+0000 mon.a (mon.0) 5010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:27.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:26 smithi121 bash[36902]: cluster 2022-04-22T18:27:26.533410+0000 mon.a (mon.0) 5011 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2022-04-22T18:27:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[45495]: cluster 2022-04-22T18:27:25.567337+0000 mgr.y (mgr.34107) 2139 : cluster [DBG] pgmap v3634: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 532 KiB/s rd, 5.0 KiB/s wr, 720 op/s 2022-04-22T18:27:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[45495]: cluster 2022-04-22T18:27:25.567445+0000 mon.a (mon.0) 5009 : cluster [DBG] osdmap e2072: 8 total, 8 up, 8 in 2022-04-22T18:27:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[45495]: audit 2022-04-22T18:27:26.265312+0000 mon.a (mon.0) 5010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[45495]: cluster 2022-04-22T18:27:26.533410+0000 mon.a (mon.0) 5011 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2022-04-22T18:27:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[46934]: cluster 2022-04-22T18:27:25.567337+0000 mgr.y (mgr.34107) 2139 : cluster [DBG] pgmap v3634: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 532 KiB/s rd, 5.0 KiB/s wr, 720 op/s 2022-04-22T18:27:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[46934]: cluster 2022-04-22T18:27:25.567445+0000 mon.a (mon.0) 5009 : cluster [DBG] osdmap e2072: 8 total, 8 up, 8 in 2022-04-22T18:27:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[46934]: audit 2022-04-22T18:27:26.265312+0000 mon.a (mon.0) 5010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:26 smithi115 bash[46934]: cluster 2022-04-22T18:27:26.533410+0000 mon.a (mon.0) 5011 : cluster [DBG] osdmap e2073: 8 total, 8 up, 8 in 2022-04-22T18:27:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:27 smithi121 bash[36902]: audit 2022-04-22T18:27:26.800180+0000 mon.a (mon.0) 5012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:27 smithi115 bash[45495]: audit 2022-04-22T18:27:26.800180+0000 mon.a (mon.0) 5012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:27 smithi115 bash[46934]: audit 2022-04-22T18:27:26.800180+0000 mon.a (mon.0) 5012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:28 smithi121 bash[36902]: cluster 2022-04-22T18:27:27.568444+0000 mgr.y (mgr.34107) 2140 : cluster [DBG] pgmap v3636: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2022-04-22T18:27:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:28 smithi121 bash[36902]: cluster 2022-04-22T18:27:27.585063+0000 mon.a (mon.0) 5013 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2022-04-22T18:27:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:28 smithi115 bash[45495]: cluster 2022-04-22T18:27:27.568444+0000 mgr.y (mgr.34107) 2140 : cluster [DBG] pgmap v3636: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2022-04-22T18:27:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:28 smithi115 bash[45495]: cluster 2022-04-22T18:27:27.585063+0000 mon.a (mon.0) 5013 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2022-04-22T18:27:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:28 smithi115 bash[46934]: cluster 2022-04-22T18:27:27.568444+0000 mgr.y (mgr.34107) 2140 : cluster [DBG] pgmap v3636: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 531 KiB/s rd, 5.0 KiB/s wr, 719 op/s 2022-04-22T18:27:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:28 smithi115 bash[46934]: cluster 2022-04-22T18:27:27.585063+0000 mon.a (mon.0) 5013 : cluster [DBG] osdmap e2074: 8 total, 8 up, 8 in 2022-04-22T18:27:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:29 smithi121 bash[36902]: cluster 2022-04-22T18:27:28.600369+0000 mon.a (mon.0) 5014 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2022-04-22T18:27:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:29 smithi121 bash[36902]: audit 2022-04-22T18:27:28.636027+0000 mon.a (mon.0) 5015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:29 smithi115 bash[45495]: cluster 2022-04-22T18:27:28.600369+0000 mon.a (mon.0) 5014 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2022-04-22T18:27:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:29 smithi115 bash[45495]: audit 2022-04-22T18:27:28.636027+0000 mon.a (mon.0) 5015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:29 smithi115 bash[46934]: cluster 2022-04-22T18:27:28.600369+0000 mon.a (mon.0) 5014 : cluster [DBG] osdmap e2075: 8 total, 8 up, 8 in 2022-04-22T18:27:30.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:29 smithi115 bash[46934]: audit 2022-04-22T18:27:28.636027+0000 mon.a (mon.0) 5015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:30 smithi121 bash[36902]: cluster 2022-04-22T18:27:29.569422+0000 mgr.y (mgr.34107) 2141 : cluster [DBG] pgmap v3639: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:27:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:30 smithi121 bash[36902]: cluster 2022-04-22T18:27:29.618124+0000 mon.a (mon.0) 5016 : cluster [DBG] osdmap e2076: 8 total, 8 up, 8 in 2022-04-22T18:27:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:30 smithi115 bash[45495]: cluster 2022-04-22T18:27:29.569422+0000 mgr.y (mgr.34107) 2141 : cluster [DBG] pgmap v3639: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:27:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:30 smithi115 bash[45495]: cluster 2022-04-22T18:27:29.618124+0000 mon.a (mon.0) 5016 : cluster [DBG] osdmap e2076: 8 total, 8 up, 8 in 2022-04-22T18:27:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:30 smithi115 bash[46934]: cluster 2022-04-22T18:27:29.569422+0000 mgr.y (mgr.34107) 2141 : cluster [DBG] pgmap v3639: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 9 op/s 2022-04-22T18:27:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:30 smithi115 bash[46934]: cluster 2022-04-22T18:27:29.618124+0000 mon.a (mon.0) 5016 : cluster [DBG] osdmap e2076: 8 total, 8 up, 8 in 2022-04-22T18:27:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:31 smithi121 bash[36902]: cluster 2022-04-22T18:27:30.629031+0000 mon.a (mon.0) 5017 : cluster [DBG] osdmap e2077: 8 total, 8 up, 8 in 2022-04-22T18:27:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:31 smithi121 bash[36902]: audit 2022-04-22T18:27:31.268454+0000 mon.a (mon.0) 5018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:31 smithi121 bash[36902]: cluster 2022-04-22T18:27:31.537725+0000 mon.a (mon.0) 5019 : cluster [DBG] osdmap e2078: 8 total, 8 up, 8 in 2022-04-22T18:27:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:31.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[45495]: cluster 2022-04-22T18:27:30.629031+0000 mon.a (mon.0) 5017 : cluster [DBG] osdmap e2077: 8 total, 8 up, 8 in 2022-04-22T18:27:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[45495]: audit 2022-04-22T18:27:31.268454+0000 mon.a (mon.0) 5018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[45495]: cluster 2022-04-22T18:27:31.537725+0000 mon.a (mon.0) 5019 : cluster [DBG] osdmap e2078: 8 total, 8 up, 8 in 2022-04-22T18:27:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[46934]: cluster 2022-04-22T18:27:30.629031+0000 mon.a (mon.0) 5017 : cluster [DBG] osdmap e2077: 8 total, 8 up, 8 in 2022-04-22T18:27:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[46934]: audit 2022-04-22T18:27:31.268454+0000 mon.a (mon.0) 5018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:31 smithi115 bash[46934]: cluster 2022-04-22T18:27:31.537725+0000 mon.a (mon.0) 5019 : cluster [DBG] osdmap e2078: 8 total, 8 up, 8 in 2022-04-22T18:27:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:32 smithi121 bash[36902]: cluster 2022-04-22T18:27:31.570255+0000 mgr.y (mgr.34107) 2142 : cluster [DBG] pgmap v3643: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 5.0 KiB/s wr, 679 op/s 2022-04-22T18:27:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:32 smithi121 bash[36902]: audit 2022-04-22T18:27:31.803683+0000 mon.a (mon.0) 5020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:32 smithi121 bash[36902]: cluster 2022-04-22T18:27:32.553166+0000 mon.a (mon.0) 5021 : cluster [DBG] osdmap e2079: 8 total, 8 up, 8 in 2022-04-22T18:27:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[45495]: cluster 2022-04-22T18:27:31.570255+0000 mgr.y (mgr.34107) 2142 : cluster [DBG] pgmap v3643: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 5.0 KiB/s wr, 679 op/s 2022-04-22T18:27:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[45495]: audit 2022-04-22T18:27:31.803683+0000 mon.a (mon.0) 5020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[45495]: cluster 2022-04-22T18:27:32.553166+0000 mon.a (mon.0) 5021 : cluster [DBG] osdmap e2079: 8 total, 8 up, 8 in 2022-04-22T18:27:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[46934]: cluster 2022-04-22T18:27:31.570255+0000 mgr.y (mgr.34107) 2142 : cluster [DBG] pgmap v3643: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 500 KiB/s rd, 5.0 KiB/s wr, 679 op/s 2022-04-22T18:27:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[46934]: audit 2022-04-22T18:27:31.803683+0000 mon.a (mon.0) 5020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:32 smithi115 bash[46934]: cluster 2022-04-22T18:27:32.553166+0000 mon.a (mon.0) 5021 : cluster [DBG] osdmap e2079: 8 total, 8 up, 8 in 2022-04-22T18:27:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:33.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:34 smithi121 bash[36902]: cluster 2022-04-22T18:27:33.558447+0000 mon.a (mon.0) 5022 : cluster [DBG] osdmap e2080: 8 total, 8 up, 8 in 2022-04-22T18:27:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:34 smithi121 bash[36902]: cluster 2022-04-22T18:27:33.570781+0000 mgr.y (mgr.34107) 2143 : cluster [DBG] pgmap v3646: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 5.1 KiB/s wr, 684 op/s 2022-04-22T18:27:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:34 smithi121 bash[36902]: audit 2022-04-22T18:27:33.638546+0000 mon.a (mon.0) 5023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[45495]: cluster 2022-04-22T18:27:33.558447+0000 mon.a (mon.0) 5022 : cluster [DBG] osdmap e2080: 8 total, 8 up, 8 in 2022-04-22T18:27:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[45495]: cluster 2022-04-22T18:27:33.570781+0000 mgr.y (mgr.34107) 2143 : cluster [DBG] pgmap v3646: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 5.1 KiB/s wr, 684 op/s 2022-04-22T18:27:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[45495]: audit 2022-04-22T18:27:33.638546+0000 mon.a (mon.0) 5023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[46934]: cluster 2022-04-22T18:27:33.558447+0000 mon.a (mon.0) 5022 : cluster [DBG] osdmap e2080: 8 total, 8 up, 8 in 2022-04-22T18:27:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[46934]: cluster 2022-04-22T18:27:33.570781+0000 mgr.y (mgr.34107) 2143 : cluster [DBG] pgmap v3646: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 504 KiB/s rd, 5.1 KiB/s wr, 684 op/s 2022-04-22T18:27:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:34 smithi115 bash[46934]: audit 2022-04-22T18:27:33.638546+0000 mon.a (mon.0) 5023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:35 smithi121 bash[36902]: cluster 2022-04-22T18:27:34.566203+0000 mon.a (mon.0) 5024 : cluster [DBG] osdmap e2081: 8 total, 8 up, 8 in 2022-04-22T18:27:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:35 smithi115 bash[45495]: cluster 2022-04-22T18:27:34.566203+0000 mon.a (mon.0) 5024 : cluster [DBG] osdmap e2081: 8 total, 8 up, 8 in 2022-04-22T18:27:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:35 smithi115 bash[46934]: cluster 2022-04-22T18:27:34.566203+0000 mon.a (mon.0) 5024 : cluster [DBG] osdmap e2081: 8 total, 8 up, 8 in 2022-04-22T18:27:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:36 smithi121 bash[36902]: cluster 2022-04-22T18:27:35.571756+0000 mgr.y (mgr.34107) 2144 : cluster [DBG] pgmap v3648: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 464 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2022-04-22T18:27:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:36 smithi121 bash[36902]: cluster 2022-04-22T18:27:35.574354+0000 mon.a (mon.0) 5025 : cluster [DBG] osdmap e2082: 8 total, 8 up, 8 in 2022-04-22T18:27:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:36 smithi121 bash[36902]: audit 2022-04-22T18:27:36.271668+0000 mon.a (mon.0) 5026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:36 smithi121 bash[36902]: cluster 2022-04-22T18:27:36.548563+0000 mon.a (mon.0) 5027 : cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2022-04-22T18:27:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[45495]: cluster 2022-04-22T18:27:35.571756+0000 mgr.y (mgr.34107) 2144 : cluster [DBG] pgmap v3648: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 464 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2022-04-22T18:27:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[45495]: cluster 2022-04-22T18:27:35.574354+0000 mon.a (mon.0) 5025 : cluster [DBG] osdmap e2082: 8 total, 8 up, 8 in 2022-04-22T18:27:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[45495]: audit 2022-04-22T18:27:36.271668+0000 mon.a (mon.0) 5026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[45495]: cluster 2022-04-22T18:27:36.548563+0000 mon.a (mon.0) 5027 : cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2022-04-22T18:27:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[46934]: cluster 2022-04-22T18:27:35.571756+0000 mgr.y (mgr.34107) 2144 : cluster [DBG] pgmap v3648: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 464 KiB/s rd, 5.0 KiB/s wr, 628 op/s 2022-04-22T18:27:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[46934]: cluster 2022-04-22T18:27:35.574354+0000 mon.a (mon.0) 5025 : cluster [DBG] osdmap e2082: 8 total, 8 up, 8 in 2022-04-22T18:27:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[46934]: audit 2022-04-22T18:27:36.271668+0000 mon.a (mon.0) 5026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:36 smithi115 bash[46934]: cluster 2022-04-22T18:27:36.548563+0000 mon.a (mon.0) 5027 : cluster [DBG] osdmap e2083: 8 total, 8 up, 8 in 2022-04-22T18:27:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:37 smithi121 bash[36902]: audit 2022-04-22T18:27:36.807555+0000 mon.a (mon.0) 5028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:37 smithi115 bash[45495]: audit 2022-04-22T18:27:36.807555+0000 mon.a (mon.0) 5028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:37 smithi115 bash[46934]: audit 2022-04-22T18:27:36.807555+0000 mon.a (mon.0) 5028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:38 smithi121 bash[36902]: cluster 2022-04-22T18:27:37.572817+0000 mgr.y (mgr.34107) 2145 : cluster [DBG] pgmap v3651: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 466 KiB/s rd, 5.0 KiB/s wr, 632 op/s 2022-04-22T18:27:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:38 smithi121 bash[36902]: cluster 2022-04-22T18:27:37.602059+0000 mon.a (mon.0) 5029 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2022-04-22T18:27:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:38 smithi115 bash[45495]: cluster 2022-04-22T18:27:37.572817+0000 mgr.y (mgr.34107) 2145 : cluster [DBG] pgmap v3651: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 466 KiB/s rd, 5.0 KiB/s wr, 632 op/s 2022-04-22T18:27:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:38 smithi115 bash[45495]: cluster 2022-04-22T18:27:37.602059+0000 mon.a (mon.0) 5029 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2022-04-22T18:27:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:38 smithi115 bash[46934]: cluster 2022-04-22T18:27:37.572817+0000 mgr.y (mgr.34107) 2145 : cluster [DBG] pgmap v3651: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 466 KiB/s rd, 5.0 KiB/s wr, 632 op/s 2022-04-22T18:27:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:38 smithi115 bash[46934]: cluster 2022-04-22T18:27:37.602059+0000 mon.a (mon.0) 5029 : cluster [DBG] osdmap e2084: 8 total, 8 up, 8 in 2022-04-22T18:27:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:39 smithi121 bash[36902]: cluster 2022-04-22T18:27:38.606476+0000 mon.a (mon.0) 5030 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2022-04-22T18:27:40.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:39 smithi121 bash[36902]: audit 2022-04-22T18:27:38.641522+0000 mon.a (mon.0) 5031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:39 smithi115 bash[45495]: cluster 2022-04-22T18:27:38.606476+0000 mon.a (mon.0) 5030 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2022-04-22T18:27:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:39 smithi115 bash[45495]: audit 2022-04-22T18:27:38.641522+0000 mon.a (mon.0) 5031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:39 smithi115 bash[46934]: cluster 2022-04-22T18:27:38.606476+0000 mon.a (mon.0) 5030 : cluster [DBG] osdmap e2085: 8 total, 8 up, 8 in 2022-04-22T18:27:40.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:39 smithi115 bash[46934]: audit 2022-04-22T18:27:38.641522+0000 mon.a (mon.0) 5031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:40 smithi121 bash[36902]: cluster 2022-04-22T18:27:39.573830+0000 mgr.y (mgr.34107) 2146 : cluster [DBG] pgmap v3654: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:40 smithi121 bash[36902]: cluster 2022-04-22T18:27:39.615021+0000 mon.a (mon.0) 5032 : cluster [DBG] osdmap e2086: 8 total, 8 up, 8 in 2022-04-22T18:27:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:40 smithi115 bash[45495]: cluster 2022-04-22T18:27:39.573830+0000 mgr.y (mgr.34107) 2146 : cluster [DBG] pgmap v3654: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:40 smithi115 bash[45495]: cluster 2022-04-22T18:27:39.615021+0000 mon.a (mon.0) 5032 : cluster [DBG] osdmap e2086: 8 total, 8 up, 8 in 2022-04-22T18:27:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:40 smithi115 bash[46934]: cluster 2022-04-22T18:27:39.573830+0000 mgr.y (mgr.34107) 2146 : cluster [DBG] pgmap v3654: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2022-04-22T18:27:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:40 smithi115 bash[46934]: cluster 2022-04-22T18:27:39.615021+0000 mon.a (mon.0) 5032 : cluster [DBG] osdmap e2086: 8 total, 8 up, 8 in 2022-04-22T18:27:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:41 smithi121 bash[36902]: cluster 2022-04-22T18:27:40.619605+0000 mon.a (mon.0) 5033 : cluster [DBG] osdmap e2087: 8 total, 8 up, 8 in 2022-04-22T18:27:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:41 smithi121 bash[36902]: audit 2022-04-22T18:27:41.274902+0000 mon.a (mon.0) 5034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:41 smithi121 bash[36902]: cluster 2022-04-22T18:27:41.546807+0000 mon.a (mon.0) 5035 : cluster [DBG] osdmap e2088: 8 total, 8 up, 8 in 2022-04-22T18:27:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:41.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[45495]: cluster 2022-04-22T18:27:40.619605+0000 mon.a (mon.0) 5033 : cluster [DBG] osdmap e2087: 8 total, 8 up, 8 in 2022-04-22T18:27:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[45495]: audit 2022-04-22T18:27:41.274902+0000 mon.a (mon.0) 5034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[45495]: cluster 2022-04-22T18:27:41.546807+0000 mon.a (mon.0) 5035 : cluster [DBG] osdmap e2088: 8 total, 8 up, 8 in 2022-04-22T18:27:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[46934]: cluster 2022-04-22T18:27:40.619605+0000 mon.a (mon.0) 5033 : cluster [DBG] osdmap e2087: 8 total, 8 up, 8 in 2022-04-22T18:27:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[46934]: audit 2022-04-22T18:27:41.274902+0000 mon.a (mon.0) 5034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:41 smithi115 bash[46934]: cluster 2022-04-22T18:27:41.546807+0000 mon.a (mon.0) 5035 : cluster [DBG] osdmap e2088: 8 total, 8 up, 8 in 2022-04-22T18:27:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:42 smithi121 bash[36902]: cluster 2022-04-22T18:27:41.574464+0000 mgr.y (mgr.34107) 2147 : cluster [DBG] pgmap v3658: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 587 op/s 2022-04-22T18:27:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:42 smithi121 bash[36902]: audit 2022-04-22T18:27:41.811058+0000 mon.a (mon.0) 5036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:42 smithi115 bash[45495]: cluster 2022-04-22T18:27:41.574464+0000 mgr.y (mgr.34107) 2147 : cluster [DBG] pgmap v3658: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 587 op/s 2022-04-22T18:27:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:42 smithi115 bash[45495]: audit 2022-04-22T18:27:41.811058+0000 mon.a (mon.0) 5036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:42 smithi115 bash[46934]: cluster 2022-04-22T18:27:41.574464+0000 mgr.y (mgr.34107) 2147 : cluster [DBG] pgmap v3658: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 587 op/s 2022-04-22T18:27:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:42 smithi115 bash[46934]: audit 2022-04-22T18:27:41.811058+0000 mon.a (mon.0) 5036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:43 smithi121 bash[36902]: cluster 2022-04-22T18:27:42.633709+0000 mon.a (mon.0) 5037 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2022-04-22T18:27:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:43 smithi115 bash[45495]: cluster 2022-04-22T18:27:42.633709+0000 mon.a (mon.0) 5037 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2022-04-22T18:27:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:43.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:44.045 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 18:27:43 smithi115 bash[42914]: debug 2022-04-22T18:27:43.645+0000 7f8eea62c700 -1 mgr.server handle_report got status from non-daemon mon.c 2022-04-22T18:27:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:43 smithi115 bash[46934]: cluster 2022-04-22T18:27:42.633709+0000 mon.a (mon.0) 5037 : cluster [DBG] osdmap e2089: 8 total, 8 up, 8 in 2022-04-22T18:27:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:44 smithi121 bash[36902]: cluster 2022-04-22T18:27:43.575546+0000 mgr.y (mgr.34107) 2148 : cluster [DBG] pgmap v3660: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 588 op/s 2022-04-22T18:27:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:44 smithi121 bash[36902]: audit 2022-04-22T18:27:43.646234+0000 mon.a (mon.0) 5038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:45.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:44 smithi121 bash[36902]: cluster 2022-04-22T18:27:43.658823+0000 mon.a (mon.0) 5039 : cluster [DBG] osdmap e2090: 8 total, 8 up, 8 in 2022-04-22T18:27:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[45495]: cluster 2022-04-22T18:27:43.575546+0000 mgr.y (mgr.34107) 2148 : cluster [DBG] pgmap v3660: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 588 op/s 2022-04-22T18:27:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[45495]: audit 2022-04-22T18:27:43.646234+0000 mon.a (mon.0) 5038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[45495]: cluster 2022-04-22T18:27:43.658823+0000 mon.a (mon.0) 5039 : cluster [DBG] osdmap e2090: 8 total, 8 up, 8 in 2022-04-22T18:27:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[46934]: cluster 2022-04-22T18:27:43.575546+0000 mgr.y (mgr.34107) 2148 : cluster [DBG] pgmap v3660: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 434 KiB/s rd, 5.0 KiB/s wr, 588 op/s 2022-04-22T18:27:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[46934]: audit 2022-04-22T18:27:43.646234+0000 mon.a (mon.0) 5038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:27:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:44 smithi115 bash[46934]: cluster 2022-04-22T18:27:43.658823+0000 mon.a (mon.0) 5039 : cluster [DBG] osdmap e2090: 8 total, 8 up, 8 in 2022-04-22T18:27:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:45 smithi121 bash[36902]: cluster 2022-04-22T18:27:44.667284+0000 mon.a (mon.0) 5040 : cluster [DBG] osdmap e2091: 8 total, 8 up, 8 in 2022-04-22T18:27:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:45 smithi115 bash[45495]: cluster 2022-04-22T18:27:44.667284+0000 mon.a (mon.0) 5040 : cluster [DBG] osdmap e2091: 8 total, 8 up, 8 in 2022-04-22T18:27:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:45 smithi115 bash[46934]: cluster 2022-04-22T18:27:44.667284+0000 mon.a (mon.0) 5040 : cluster [DBG] osdmap e2091: 8 total, 8 up, 8 in 2022-04-22T18:27:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:46 smithi121 bash[36902]: cluster 2022-04-22T18:27:45.576661+0000 mgr.y (mgr.34107) 2149 : cluster [DBG] pgmap v3663: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 399 KiB/s rd, 5.0 KiB/s wr, 540 op/s 2022-04-22T18:27:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:46 smithi121 bash[36902]: cluster 2022-04-22T18:27:45.675459+0000 mon.a (mon.0) 5041 : cluster [DBG] osdmap e2092: 8 total, 8 up, 8 in 2022-04-22T18:27:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:46 smithi121 bash[36902]: audit 2022-04-22T18:27:46.278324+0000 mon.a (mon.0) 5042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:46 smithi121 bash[36902]: cluster 2022-04-22T18:27:46.545328+0000 mon.a (mon.0) 5043 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2022-04-22T18:27:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[45495]: cluster 2022-04-22T18:27:45.576661+0000 mgr.y (mgr.34107) 2149 : cluster [DBG] pgmap v3663: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 399 KiB/s rd, 5.0 KiB/s wr, 540 op/s 2022-04-22T18:27:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[45495]: cluster 2022-04-22T18:27:45.675459+0000 mon.a (mon.0) 5041 : cluster [DBG] osdmap e2092: 8 total, 8 up, 8 in 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[45495]: audit 2022-04-22T18:27:46.278324+0000 mon.a (mon.0) 5042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[45495]: cluster 2022-04-22T18:27:46.545328+0000 mon.a (mon.0) 5043 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[46934]: cluster 2022-04-22T18:27:45.576661+0000 mgr.y (mgr.34107) 2149 : cluster [DBG] pgmap v3663: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 399 KiB/s rd, 5.0 KiB/s wr, 540 op/s 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[46934]: cluster 2022-04-22T18:27:45.675459+0000 mon.a (mon.0) 5041 : cluster [DBG] osdmap e2092: 8 total, 8 up, 8 in 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[46934]: audit 2022-04-22T18:27:46.278324+0000 mon.a (mon.0) 5042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:46 smithi115 bash[46934]: cluster 2022-04-22T18:27:46.545328+0000 mon.a (mon.0) 5043 : cluster [DBG] osdmap e2093: 8 total, 8 up, 8 in 2022-04-22T18:27:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:47 smithi121 bash[36902]: audit 2022-04-22T18:27:46.814802+0000 mon.a (mon.0) 5044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:47 smithi121 bash[36902]: cluster 2022-04-22T18:27:47.557348+0000 mon.a (mon.0) 5045 : cluster [DBG] osdmap e2094: 8 total, 8 up, 8 in 2022-04-22T18:27:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:47 smithi115 bash[45495]: audit 2022-04-22T18:27:46.814802+0000 mon.a (mon.0) 5044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:48.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:47 smithi115 bash[45495]: cluster 2022-04-22T18:27:47.557348+0000 mon.a (mon.0) 5045 : cluster [DBG] osdmap e2094: 8 total, 8 up, 8 in 2022-04-22T18:27:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:47 smithi115 bash[46934]: audit 2022-04-22T18:27:46.814802+0000 mon.a (mon.0) 5044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:47 smithi115 bash[46934]: cluster 2022-04-22T18:27:47.557348+0000 mon.a (mon.0) 5045 : cluster [DBG] osdmap e2094: 8 total, 8 up, 8 in 2022-04-22T18:27:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:48 smithi121 bash[36902]: cluster 2022-04-22T18:27:47.577437+0000 mgr.y (mgr.34107) 2150 : cluster [DBG] pgmap v3667: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 410 KiB/s rd, 5.1 KiB/s wr, 554 op/s 2022-04-22T18:27:49.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:48 smithi121 bash[36902]: cluster 2022-04-22T18:27:48.567799+0000 mon.a (mon.0) 5046 : cluster [DBG] osdmap e2095: 8 total, 8 up, 8 in 2022-04-22T18:27:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:48 smithi115 bash[45495]: cluster 2022-04-22T18:27:47.577437+0000 mgr.y (mgr.34107) 2150 : cluster [DBG] pgmap v3667: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 410 KiB/s rd, 5.1 KiB/s wr, 554 op/s 2022-04-22T18:27:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:48 smithi115 bash[45495]: cluster 2022-04-22T18:27:48.567799+0000 mon.a (mon.0) 5046 : cluster [DBG] osdmap e2095: 8 total, 8 up, 8 in 2022-04-22T18:27:49.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:48 smithi115 bash[46934]: cluster 2022-04-22T18:27:47.577437+0000 mgr.y (mgr.34107) 2150 : cluster [DBG] pgmap v3667: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 410 KiB/s rd, 5.1 KiB/s wr, 554 op/s 2022-04-22T18:27:49.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:48 smithi115 bash[46934]: cluster 2022-04-22T18:27:48.567799+0000 mon.a (mon.0) 5046 : cluster [DBG] osdmap e2095: 8 total, 8 up, 8 in 2022-04-22T18:27:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:50 smithi121 bash[36902]: cluster 2022-04-22T18:27:49.573098+0000 mon.a (mon.0) 5047 : cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2022-04-22T18:27:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:50 smithi121 bash[36902]: cluster 2022-04-22T18:27:49.578179+0000 mgr.y (mgr.34107) 2151 : cluster [DBG] pgmap v3670: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:27:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:50 smithi115 bash[45495]: cluster 2022-04-22T18:27:49.573098+0000 mon.a (mon.0) 5047 : cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2022-04-22T18:27:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:50 smithi115 bash[45495]: cluster 2022-04-22T18:27:49.578179+0000 mgr.y (mgr.34107) 2151 : cluster [DBG] pgmap v3670: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:27:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:50 smithi115 bash[46934]: cluster 2022-04-22T18:27:49.573098+0000 mon.a (mon.0) 5047 : cluster [DBG] osdmap e2096: 8 total, 8 up, 8 in 2022-04-22T18:27:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:50 smithi115 bash[46934]: cluster 2022-04-22T18:27:49.578179+0000 mgr.y (mgr.34107) 2151 : cluster [DBG] pgmap v3670: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2022-04-22T18:27:51.672 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:27:51.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[45495]: cluster 2022-04-22T18:27:50.583679+0000 mon.a (mon.0) 5048 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2022-04-22T18:27:51.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[45495]: audit 2022-04-22T18:27:51.281546+0000 mon.a (mon.0) 5049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:51.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[45495]: cluster 2022-04-22T18:27:51.546775+0000 mon.a (mon.0) 5050 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2022-04-22T18:27:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[46934]: cluster 2022-04-22T18:27:50.583679+0000 mon.a (mon.0) 5048 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2022-04-22T18:27:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[46934]: audit 2022-04-22T18:27:51.281546+0000 mon.a (mon.0) 5049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[46934]: cluster 2022-04-22T18:27:51.546775+0000 mon.a (mon.0) 5050 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2022-04-22T18:27:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:51 smithi121 bash[36902]: cluster 2022-04-22T18:27:50.583679+0000 mon.a (mon.0) 5048 : cluster [DBG] osdmap e2097: 8 total, 8 up, 8 in 2022-04-22T18:27:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:51 smithi121 bash[36902]: audit 2022-04-22T18:27:51.281546+0000 mon.a (mon.0) 5049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:51 smithi121 bash[36902]: cluster 2022-04-22T18:27:51.546775+0000 mon.a (mon.0) 5050 : cluster [DBG] osdmap e2098: 8 total, 8 up, 8 in 2022-04-22T18:27:52.021 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:27:52.021 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (50m) 8m ago 64m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (64m) 9m ago 64m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (52m) 8m ago 65m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (52m) 9m ago 65m 19.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (60m) 9m ago 69m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (60m) 8m ago 72m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (59m) 8m ago 72m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (58m) 9m ago 70m 119M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (59m) 8m ago 70m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:27:52.022 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (52m) 8m ago 64m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (51m) 9m ago 64m 9.93M - 1dbe0e931976 f4b044642a96 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (57m) 8m ago 69m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (56m) 8m ago 69m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (56m) 8m ago 68m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (56m) 8m ago 68m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (55m) 9m ago 67m 555M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (54m) 9m ago 67m 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (54m) 9m ago 66m 463M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:27:52.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (53m) 9m ago 66m 610M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:27:52.024 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (51m) 9m ago 65m 58.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:27:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:51.831Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:52.421 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:27:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:27:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:27:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:27:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:27:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:27:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:27:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:27:52.423 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:27:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:52 smithi121 bash[36902]: cluster 2022-04-22T18:27:51.578730+0000 mgr.y (mgr.34107) 2152 : cluster [DBG] pgmap v3673: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 366 KiB/s rd, 5.0 KiB/s wr, 496 op/s 2022-04-22T18:27:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:52 smithi121 bash[36902]: audit 2022-04-22T18:27:51.648275+0000 mgr.y (mgr.34107) 2153 : audit [DBG] from='client.37614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:52 smithi121 bash[36902]: audit 2022-04-22T18:27:51.818501+0000 mon.a (mon.0) 5051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:52 smithi121 bash[36902]: audit 2022-04-22T18:27:52.414015+0000 mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.115:0/2875082460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[45495]: cluster 2022-04-22T18:27:51.578730+0000 mgr.y (mgr.34107) 2152 : cluster [DBG] pgmap v3673: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 366 KiB/s rd, 5.0 KiB/s wr, 496 op/s 2022-04-22T18:27:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[45495]: audit 2022-04-22T18:27:51.648275+0000 mgr.y (mgr.34107) 2153 : audit [DBG] from='client.37614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[45495]: audit 2022-04-22T18:27:51.818501+0000 mon.a (mon.0) 5051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[45495]: audit 2022-04-22T18:27:52.414015+0000 mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.115:0/2875082460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[46934]: cluster 2022-04-22T18:27:51.578730+0000 mgr.y (mgr.34107) 2152 : cluster [DBG] pgmap v3673: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 366 KiB/s rd, 5.0 KiB/s wr, 496 op/s 2022-04-22T18:27:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[46934]: audit 2022-04-22T18:27:51.648275+0000 mgr.y (mgr.34107) 2153 : audit [DBG] from='client.37614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[46934]: audit 2022-04-22T18:27:51.818501+0000 mon.a (mon.0) 5051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:52 smithi115 bash[46934]: audit 2022-04-22T18:27:52.414015+0000 mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.115:0/2875082460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:27:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:53 smithi121 bash[36902]: audit 2022-04-22T18:27:51.998941+0000 mgr.y (mgr.34107) 2154 : audit [DBG] from='client.47554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:53 smithi121 bash[36902]: cluster 2022-04-22T18:27:52.589522+0000 mon.a (mon.0) 5052 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2022-04-22T18:27:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:27:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:27:53.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:27:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:53 smithi115 bash[45495]: audit 2022-04-22T18:27:51.998941+0000 mgr.y (mgr.34107) 2154 : audit [DBG] from='client.47554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:53 smithi115 bash[45495]: cluster 2022-04-22T18:27:52.589522+0000 mon.a (mon.0) 5052 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2022-04-22T18:27:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:53 smithi115 bash[46934]: audit 2022-04-22T18:27:51.998941+0000 mgr.y (mgr.34107) 2154 : audit [DBG] from='client.47554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:27:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:53 smithi115 bash[46934]: cluster 2022-04-22T18:27:52.589522+0000 mon.a (mon.0) 5052 : cluster [DBG] osdmap e2099: 8 total, 8 up, 8 in 2022-04-22T18:27:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:54 smithi121 bash[36902]: cluster 2022-04-22T18:27:53.579890+0000 mgr.y (mgr.34107) 2155 : cluster [DBG] pgmap v3675: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 5.0 KiB/s wr, 499 op/s 2022-04-22T18:27:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:54 smithi121 bash[36902]: cluster 2022-04-22T18:27:53.601940+0000 mon.a (mon.0) 5053 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2022-04-22T18:27:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:54 smithi115 bash[46934]: cluster 2022-04-22T18:27:53.579890+0000 mgr.y (mgr.34107) 2155 : cluster [DBG] pgmap v3675: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 5.0 KiB/s wr, 499 op/s 2022-04-22T18:27:55.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:54 smithi115 bash[46934]: cluster 2022-04-22T18:27:53.601940+0000 mon.a (mon.0) 5053 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2022-04-22T18:27:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:54 smithi115 bash[45495]: cluster 2022-04-22T18:27:53.579890+0000 mgr.y (mgr.34107) 2155 : cluster [DBG] pgmap v3675: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 368 KiB/s rd, 5.0 KiB/s wr, 499 op/s 2022-04-22T18:27:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:54 smithi115 bash[45495]: cluster 2022-04-22T18:27:53.601940+0000 mon.a (mon.0) 5053 : cluster [DBG] osdmap e2100: 8 total, 8 up, 8 in 2022-04-22T18:27:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:55 smithi121 bash[36902]: cluster 2022-04-22T18:27:54.607046+0000 mon.a (mon.0) 5054 : cluster [DBG] osdmap e2101: 8 total, 8 up, 8 in 2022-04-22T18:27:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:55 smithi115 bash[45495]: cluster 2022-04-22T18:27:54.607046+0000 mon.a (mon.0) 5054 : cluster [DBG] osdmap e2101: 8 total, 8 up, 8 in 2022-04-22T18:27:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:55 smithi115 bash[46934]: cluster 2022-04-22T18:27:54.607046+0000 mon.a (mon.0) 5054 : cluster [DBG] osdmap e2101: 8 total, 8 up, 8 in 2022-04-22T18:27:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:56 smithi121 bash[36902]: cluster 2022-04-22T18:27:55.580916+0000 mgr.y (mgr.34107) 2156 : cluster [DBG] pgmap v3678: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 5.0 KiB/s wr, 454 op/s 2022-04-22T18:27:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:56 smithi121 bash[36902]: cluster 2022-04-22T18:27:55.615912+0000 mon.a (mon.0) 5055 : cluster [DBG] osdmap e2102: 8 total, 8 up, 8 in 2022-04-22T18:27:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:56 smithi121 bash[36902]: audit 2022-04-22T18:27:56.284604+0000 mon.a (mon.0) 5056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:56 smithi121 bash[36902]: cluster 2022-04-22T18:27:56.548437+0000 mon.a (mon.0) 5057 : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2022-04-22T18:27:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[46934]: cluster 2022-04-22T18:27:55.580916+0000 mgr.y (mgr.34107) 2156 : cluster [DBG] pgmap v3678: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 5.0 KiB/s wr, 454 op/s 2022-04-22T18:27:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[46934]: cluster 2022-04-22T18:27:55.615912+0000 mon.a (mon.0) 5055 : cluster [DBG] osdmap e2102: 8 total, 8 up, 8 in 2022-04-22T18:27:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[46934]: audit 2022-04-22T18:27:56.284604+0000 mon.a (mon.0) 5056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[46934]: cluster 2022-04-22T18:27:56.548437+0000 mon.a (mon.0) 5057 : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2022-04-22T18:27:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[45495]: cluster 2022-04-22T18:27:55.580916+0000 mgr.y (mgr.34107) 2156 : cluster [DBG] pgmap v3678: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 335 KiB/s rd, 5.0 KiB/s wr, 454 op/s 2022-04-22T18:27:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[45495]: cluster 2022-04-22T18:27:55.615912+0000 mon.a (mon.0) 5055 : cluster [DBG] osdmap e2102: 8 total, 8 up, 8 in 2022-04-22T18:27:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[45495]: audit 2022-04-22T18:27:56.284604+0000 mon.a (mon.0) 5056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:27:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:56 smithi115 bash[45495]: cluster 2022-04-22T18:27:56.548437+0000 mon.a (mon.0) 5057 : cluster [DBG] osdmap e2103: 8 total, 8 up, 8 in 2022-04-22T18:27:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:57 smithi121 bash[36902]: audit 2022-04-22T18:27:56.822053+0000 mon.a (mon.0) 5058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:57 smithi121 bash[36902]: cluster 2022-04-22T18:27:57.564889+0000 mon.a (mon.0) 5059 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2022-04-22T18:27:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:57 smithi115 bash[46934]: audit 2022-04-22T18:27:56.822053+0000 mon.a (mon.0) 5058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:58.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:57 smithi115 bash[46934]: cluster 2022-04-22T18:27:57.564889+0000 mon.a (mon.0) 5059 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2022-04-22T18:27:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:57 smithi115 bash[45495]: audit 2022-04-22T18:27:56.822053+0000 mon.a (mon.0) 5058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:27:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:57 smithi115 bash[45495]: cluster 2022-04-22T18:27:57.564889+0000 mon.a (mon.0) 5059 : cluster [DBG] osdmap e2104: 8 total, 8 up, 8 in 2022-04-22T18:27:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:58 smithi121 bash[36902]: cluster 2022-04-22T18:27:57.581548+0000 mgr.y (mgr.34107) 2157 : cluster [DBG] pgmap v3682: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 338 KiB/s rd, 5.0 KiB/s wr, 457 op/s 2022-04-22T18:27:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:58 smithi115 bash[45495]: cluster 2022-04-22T18:27:57.581548+0000 mgr.y (mgr.34107) 2157 : cluster [DBG] pgmap v3682: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 338 KiB/s rd, 5.0 KiB/s wr, 457 op/s 2022-04-22T18:27:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:58 smithi115 bash[46934]: cluster 2022-04-22T18:27:57.581548+0000 mgr.y (mgr.34107) 2157 : cluster [DBG] pgmap v3682: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 338 KiB/s rd, 5.0 KiB/s wr, 457 op/s 2022-04-22T18:28:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:27:59 smithi121 bash[36902]: cluster 2022-04-22T18:27:58.623809+0000 mon.a (mon.0) 5060 : cluster [DBG] osdmap e2105: 8 total, 8 up, 8 in 2022-04-22T18:28:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:27:59 smithi115 bash[45495]: cluster 2022-04-22T18:27:58.623809+0000 mon.a (mon.0) 5060 : cluster [DBG] osdmap e2105: 8 total, 8 up, 8 in 2022-04-22T18:28:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:27:59 smithi115 bash[46934]: cluster 2022-04-22T18:27:58.623809+0000 mon.a (mon.0) 5060 : cluster [DBG] osdmap e2105: 8 total, 8 up, 8 in 2022-04-22T18:28:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:00 smithi121 bash[36902]: cluster 2022-04-22T18:27:59.582546+0000 mgr.y (mgr.34107) 2158 : cluster [DBG] pgmap v3684: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2022-04-22T18:28:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:00 smithi121 bash[36902]: cluster 2022-04-22T18:27:59.641024+0000 mon.a (mon.0) 5061 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2022-04-22T18:28:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:00 smithi115 bash[45495]: cluster 2022-04-22T18:27:59.582546+0000 mgr.y (mgr.34107) 2158 : cluster [DBG] pgmap v3684: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2022-04-22T18:28:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:00 smithi115 bash[45495]: cluster 2022-04-22T18:27:59.641024+0000 mon.a (mon.0) 5061 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2022-04-22T18:28:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:00 smithi115 bash[46934]: cluster 2022-04-22T18:27:59.582546+0000 mgr.y (mgr.34107) 2158 : cluster [DBG] pgmap v3684: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2022-04-22T18:28:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:00 smithi115 bash[46934]: cluster 2022-04-22T18:27:59.641024+0000 mon.a (mon.0) 5061 : cluster [DBG] osdmap e2106: 8 total, 8 up, 8 in 2022-04-22T18:28:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:01 smithi121 bash[36902]: cluster 2022-04-22T18:28:00.648444+0000 mon.a (mon.0) 5062 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2022-04-22T18:28:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:01 smithi121 bash[36902]: audit 2022-04-22T18:28:01.287693+0000 mon.a (mon.0) 5063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:01 smithi121 bash[36902]: cluster 2022-04-22T18:28:01.554546+0000 mon.a (mon.0) 5064 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2022-04-22T18:28:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:01.834Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[46934]: cluster 2022-04-22T18:28:00.648444+0000 mon.a (mon.0) 5062 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2022-04-22T18:28:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[46934]: audit 2022-04-22T18:28:01.287693+0000 mon.a (mon.0) 5063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[46934]: cluster 2022-04-22T18:28:01.554546+0000 mon.a (mon.0) 5064 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2022-04-22T18:28:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[45495]: cluster 2022-04-22T18:28:00.648444+0000 mon.a (mon.0) 5062 : cluster [DBG] osdmap e2107: 8 total, 8 up, 8 in 2022-04-22T18:28:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[45495]: audit 2022-04-22T18:28:01.287693+0000 mon.a (mon.0) 5063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:01 smithi115 bash[45495]: cluster 2022-04-22T18:28:01.554546+0000 mon.a (mon.0) 5064 : cluster [DBG] osdmap e2108: 8 total, 8 up, 8 in 2022-04-22T18:28:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:02 smithi121 bash[36902]: cluster 2022-04-22T18:28:01.583251+0000 mgr.y (mgr.34107) 2159 : cluster [DBG] pgmap v3688: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 303 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:02 smithi121 bash[36902]: audit 2022-04-22T18:28:01.825549+0000 mon.a (mon.0) 5065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:02 smithi121 bash[36902]: cluster 2022-04-22T18:28:02.570215+0000 mon.a (mon.0) 5066 : cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2022-04-22T18:28:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[45495]: cluster 2022-04-22T18:28:01.583251+0000 mgr.y (mgr.34107) 2159 : cluster [DBG] pgmap v3688: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 303 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[45495]: audit 2022-04-22T18:28:01.825549+0000 mon.a (mon.0) 5065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[45495]: cluster 2022-04-22T18:28:02.570215+0000 mon.a (mon.0) 5066 : cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2022-04-22T18:28:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[46934]: cluster 2022-04-22T18:28:01.583251+0000 mgr.y (mgr.34107) 2159 : cluster [DBG] pgmap v3688: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 303 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[46934]: audit 2022-04-22T18:28:01.825549+0000 mon.a (mon.0) 5065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:02 smithi115 bash[46934]: cluster 2022-04-22T18:28:02.570215+0000 mon.a (mon.0) 5066 : cluster [DBG] osdmap e2109: 8 total, 8 up, 8 in 2022-04-22T18:28:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:03.714Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:04 smithi121 bash[36902]: cluster 2022-04-22T18:28:03.584405+0000 mgr.y (mgr.34107) 2160 : cluster [DBG] pgmap v3690: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:05.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:04 smithi121 bash[36902]: cluster 2022-04-22T18:28:03.589501+0000 mon.a (mon.0) 5067 : cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2022-04-22T18:28:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:04 smithi115 bash[45495]: cluster 2022-04-22T18:28:03.584405+0000 mgr.y (mgr.34107) 2160 : cluster [DBG] pgmap v3690: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:05.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:04 smithi115 bash[45495]: cluster 2022-04-22T18:28:03.589501+0000 mon.a (mon.0) 5067 : cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2022-04-22T18:28:05.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:04 smithi115 bash[46934]: cluster 2022-04-22T18:28:03.584405+0000 mgr.y (mgr.34107) 2160 : cluster [DBG] pgmap v3690: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 5.0 KiB/s wr, 413 op/s 2022-04-22T18:28:05.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:04 smithi115 bash[46934]: cluster 2022-04-22T18:28:03.589501+0000 mon.a (mon.0) 5067 : cluster [DBG] osdmap e2110: 8 total, 8 up, 8 in 2022-04-22T18:28:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:05 smithi121 bash[36902]: cluster 2022-04-22T18:28:04.594503+0000 mon.a (mon.0) 5068 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2022-04-22T18:28:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:05 smithi121 bash[36902]: audit 2022-04-22T18:28:04.717129+0000 mon.a (mon.0) 5069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:28:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:05 smithi121 bash[36902]: audit 2022-04-22T18:28:04.719133+0000 mon.a (mon.0) 5070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:28:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:05 smithi121 bash[36902]: audit 2022-04-22T18:28:04.720383+0000 mon.a (mon.0) 5071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:28:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[45495]: cluster 2022-04-22T18:28:04.594503+0000 mon.a (mon.0) 5068 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2022-04-22T18:28:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[45495]: audit 2022-04-22T18:28:04.717129+0000 mon.a (mon.0) 5069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[45495]: audit 2022-04-22T18:28:04.719133+0000 mon.a (mon.0) 5070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[45495]: audit 2022-04-22T18:28:04.720383+0000 mon.a (mon.0) 5071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[46934]: cluster 2022-04-22T18:28:04.594503+0000 mon.a (mon.0) 5068 : cluster [DBG] osdmap e2111: 8 total, 8 up, 8 in 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[46934]: audit 2022-04-22T18:28:04.717129+0000 mon.a (mon.0) 5069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[46934]: audit 2022-04-22T18:28:04.719133+0000 mon.a (mon.0) 5070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:28:06.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:05 smithi115 bash[46934]: audit 2022-04-22T18:28:04.720383+0000 mon.a (mon.0) 5071 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:28:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:06 smithi121 bash[36902]: cluster 2022-04-22T18:28:05.585463+0000 mgr.y (mgr.34107) 2161 : cluster [DBG] pgmap v3693: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 269 KiB/s rd, 5.0 KiB/s wr, 365 op/s 2022-04-22T18:28:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:06 smithi121 bash[36902]: cluster 2022-04-22T18:28:05.609338+0000 mon.a (mon.0) 5072 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2022-04-22T18:28:07.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:06 smithi121 bash[36902]: audit 2022-04-22T18:28:06.291312+0000 mon.a (mon.0) 5073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:07.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:06 smithi121 bash[36902]: cluster 2022-04-22T18:28:06.561034+0000 mon.a (mon.0) 5074 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2022-04-22T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[45495]: cluster 2022-04-22T18:28:05.585463+0000 mgr.y (mgr.34107) 2161 : cluster [DBG] pgmap v3693: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 269 KiB/s rd, 5.0 KiB/s wr, 365 op/s 2022-04-22T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[45495]: cluster 2022-04-22T18:28:05.609338+0000 mon.a (mon.0) 5072 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2022-04-22T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[45495]: audit 2022-04-22T18:28:06.291312+0000 mon.a (mon.0) 5073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[45495]: cluster 2022-04-22T18:28:06.561034+0000 mon.a (mon.0) 5074 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2022-04-22T18:28:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[46934]: cluster 2022-04-22T18:28:05.585463+0000 mgr.y (mgr.34107) 2161 : cluster [DBG] pgmap v3693: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 269 KiB/s rd, 5.0 KiB/s wr, 365 op/s 2022-04-22T18:28:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[46934]: cluster 2022-04-22T18:28:05.609338+0000 mon.a (mon.0) 5072 : cluster [DBG] osdmap e2112: 8 total, 8 up, 8 in 2022-04-22T18:28:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[46934]: audit 2022-04-22T18:28:06.291312+0000 mon.a (mon.0) 5073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:06 smithi115 bash[46934]: cluster 2022-04-22T18:28:06.561034+0000 mon.a (mon.0) 5074 : cluster [DBG] osdmap e2113: 8 total, 8 up, 8 in 2022-04-22T18:28:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:07 smithi121 bash[36902]: audit 2022-04-22T18:28:06.829233+0000 mon.a (mon.0) 5075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:07 smithi121 bash[36902]: cluster 2022-04-22T18:28:07.574183+0000 mon.a (mon.0) 5076 : cluster [DBG] osdmap e2114: 8 total, 8 up, 8 in 2022-04-22T18:28:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:07 smithi115 bash[45495]: audit 2022-04-22T18:28:06.829233+0000 mon.a (mon.0) 5075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:07 smithi115 bash[45495]: cluster 2022-04-22T18:28:07.574183+0000 mon.a (mon.0) 5076 : cluster [DBG] osdmap e2114: 8 total, 8 up, 8 in 2022-04-22T18:28:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:07 smithi115 bash[46934]: audit 2022-04-22T18:28:06.829233+0000 mon.a (mon.0) 5075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:07 smithi115 bash[46934]: cluster 2022-04-22T18:28:07.574183+0000 mon.a (mon.0) 5076 : cluster [DBG] osdmap e2114: 8 total, 8 up, 8 in 2022-04-22T18:28:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:08 smithi121 bash[36902]: cluster 2022-04-22T18:28:07.586070+0000 mgr.y (mgr.34107) 2162 : cluster [DBG] pgmap v3697: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 271 KiB/s rd, 5.0 KiB/s wr, 367 op/s 2022-04-22T18:28:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:08 smithi115 bash[45495]: cluster 2022-04-22T18:28:07.586070+0000 mgr.y (mgr.34107) 2162 : cluster [DBG] pgmap v3697: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 271 KiB/s rd, 5.0 KiB/s wr, 367 op/s 2022-04-22T18:28:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:08 smithi115 bash[46934]: cluster 2022-04-22T18:28:07.586070+0000 mgr.y (mgr.34107) 2162 : cluster [DBG] pgmap v3697: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 271 KiB/s rd, 5.0 KiB/s wr, 367 op/s 2022-04-22T18:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:09 smithi121 bash[36902]: cluster 2022-04-22T18:28:08.619237+0000 mon.a (mon.0) 5077 : cluster [DBG] osdmap e2115: 8 total, 8 up, 8 in 2022-04-22T18:28:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:09 smithi121 bash[36902]: audit 2022-04-22T18:28:09.113684+0000 mon.a (mon.0) 5078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:09 smithi121 bash[36902]: audit 2022-04-22T18:28:09.472131+0000 mon.a (mon.0) 5079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[45495]: cluster 2022-04-22T18:28:08.619237+0000 mon.a (mon.0) 5077 : cluster [DBG] osdmap e2115: 8 total, 8 up, 8 in 2022-04-22T18:28:10.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[45495]: audit 2022-04-22T18:28:09.113684+0000 mon.a (mon.0) 5078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:10.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[45495]: audit 2022-04-22T18:28:09.472131+0000 mon.a (mon.0) 5079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[46934]: cluster 2022-04-22T18:28:08.619237+0000 mon.a (mon.0) 5077 : cluster [DBG] osdmap e2115: 8 total, 8 up, 8 in 2022-04-22T18:28:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[46934]: audit 2022-04-22T18:28:09.113684+0000 mon.a (mon.0) 5078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:09 smithi115 bash[46934]: audit 2022-04-22T18:28:09.472131+0000 mon.a (mon.0) 5079 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:28:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:10 smithi121 bash[36902]: cluster 2022-04-22T18:28:09.587054+0000 mgr.y (mgr.34107) 2163 : cluster [DBG] pgmap v3699: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:10 smithi121 bash[36902]: cluster 2022-04-22T18:28:09.622815+0000 mon.a (mon.0) 5080 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2022-04-22T18:28:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:10 smithi115 bash[45495]: cluster 2022-04-22T18:28:09.587054+0000 mgr.y (mgr.34107) 2163 : cluster [DBG] pgmap v3699: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:10 smithi115 bash[45495]: cluster 2022-04-22T18:28:09.622815+0000 mon.a (mon.0) 5080 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2022-04-22T18:28:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:10 smithi115 bash[46934]: cluster 2022-04-22T18:28:09.587054+0000 mgr.y (mgr.34107) 2163 : cluster [DBG] pgmap v3699: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:10 smithi115 bash[46934]: cluster 2022-04-22T18:28:09.622815+0000 mon.a (mon.0) 5080 : cluster [DBG] osdmap e2116: 8 total, 8 up, 8 in 2022-04-22T18:28:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:11 smithi121 bash[36902]: cluster 2022-04-22T18:28:10.638249+0000 mon.a (mon.0) 5081 : cluster [DBG] osdmap e2117: 8 total, 8 up, 8 in 2022-04-22T18:28:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:11 smithi121 bash[36902]: audit 2022-04-22T18:28:11.294393+0000 mon.a (mon.0) 5082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:11 smithi121 bash[36902]: cluster 2022-04-22T18:28:11.563192+0000 mon.a (mon.0) 5083 : cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2022-04-22T18:28:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:11.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[45495]: cluster 2022-04-22T18:28:10.638249+0000 mon.a (mon.0) 5081 : cluster [DBG] osdmap e2117: 8 total, 8 up, 8 in 2022-04-22T18:28:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[45495]: audit 2022-04-22T18:28:11.294393+0000 mon.a (mon.0) 5082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[45495]: cluster 2022-04-22T18:28:11.563192+0000 mon.a (mon.0) 5083 : cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2022-04-22T18:28:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[46934]: cluster 2022-04-22T18:28:10.638249+0000 mon.a (mon.0) 5081 : cluster [DBG] osdmap e2117: 8 total, 8 up, 8 in 2022-04-22T18:28:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[46934]: audit 2022-04-22T18:28:11.294393+0000 mon.a (mon.0) 5082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:11 smithi115 bash[46934]: cluster 2022-04-22T18:28:11.563192+0000 mon.a (mon.0) 5083 : cluster [DBG] osdmap e2118: 8 total, 8 up, 8 in 2022-04-22T18:28:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:12 smithi121 bash[36902]: cluster 2022-04-22T18:28:11.587831+0000 mgr.y (mgr.34107) 2164 : cluster [DBG] pgmap v3703: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 237 KiB/s rd, 5.0 KiB/s wr, 322 op/s 2022-04-22T18:28:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:12 smithi121 bash[36902]: audit 2022-04-22T18:28:11.832925+0000 mon.a (mon.0) 5084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:12 smithi121 bash[36902]: cluster 2022-04-22T18:28:12.581578+0000 mon.a (mon.0) 5085 : cluster [DBG] osdmap e2119: 8 total, 8 up, 8 in 2022-04-22T18:28:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[45495]: cluster 2022-04-22T18:28:11.587831+0000 mgr.y (mgr.34107) 2164 : cluster [DBG] pgmap v3703: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 237 KiB/s rd, 5.0 KiB/s wr, 322 op/s 2022-04-22T18:28:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[45495]: audit 2022-04-22T18:28:11.832925+0000 mon.a (mon.0) 5084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[45495]: cluster 2022-04-22T18:28:12.581578+0000 mon.a (mon.0) 5085 : cluster [DBG] osdmap e2119: 8 total, 8 up, 8 in 2022-04-22T18:28:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[46934]: cluster 2022-04-22T18:28:11.587831+0000 mgr.y (mgr.34107) 2164 : cluster [DBG] pgmap v3703: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 237 KiB/s rd, 5.0 KiB/s wr, 322 op/s 2022-04-22T18:28:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[46934]: audit 2022-04-22T18:28:11.832925+0000 mon.a (mon.0) 5084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:12 smithi115 bash[46934]: cluster 2022-04-22T18:28:12.581578+0000 mon.a (mon.0) 5085 : cluster [DBG] osdmap e2119: 8 total, 8 up, 8 in 2022-04-22T18:28:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:13.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:14 smithi121 bash[36902]: cluster 2022-04-22T18:28:13.584333+0000 mon.a (mon.0) 5086 : cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2022-04-22T18:28:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:14 smithi121 bash[36902]: cluster 2022-04-22T18:28:13.588466+0000 mgr.y (mgr.34107) 2165 : cluster [DBG] pgmap v3706: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 240 KiB/s rd, 5.0 KiB/s wr, 327 op/s 2022-04-22T18:28:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:14 smithi121 bash[36902]: audit 2022-04-22T18:28:14.001840+0000 mon.a (mon.0) 5087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:28:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:14 smithi121 bash[36902]: audit 2022-04-22T18:28:14.002375+0000 mon.a (mon.0) 5088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:28:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[45495]: cluster 2022-04-22T18:28:13.584333+0000 mon.a (mon.0) 5086 : cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[45495]: cluster 2022-04-22T18:28:13.588466+0000 mgr.y (mgr.34107) 2165 : cluster [DBG] pgmap v3706: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 240 KiB/s rd, 5.0 KiB/s wr, 327 op/s 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[45495]: audit 2022-04-22T18:28:14.001840+0000 mon.a (mon.0) 5087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[45495]: audit 2022-04-22T18:28:14.002375+0000 mon.a (mon.0) 5088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[46934]: cluster 2022-04-22T18:28:13.584333+0000 mon.a (mon.0) 5086 : cluster [DBG] osdmap e2120: 8 total, 8 up, 8 in 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[46934]: cluster 2022-04-22T18:28:13.588466+0000 mgr.y (mgr.34107) 2165 : cluster [DBG] pgmap v3706: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 240 KiB/s rd, 5.0 KiB/s wr, 327 op/s 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[46934]: audit 2022-04-22T18:28:14.001840+0000 mon.a (mon.0) 5087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:28:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:14 smithi115 bash[46934]: audit 2022-04-22T18:28:14.002375+0000 mon.a (mon.0) 5088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:28:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:15 smithi121 bash[36902]: cluster 2022-04-22T18:28:14.590849+0000 mon.a (mon.0) 5089 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2022-04-22T18:28:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:15 smithi115 bash[45495]: cluster 2022-04-22T18:28:14.590849+0000 mon.a (mon.0) 5089 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2022-04-22T18:28:16.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:15 smithi115 bash[46934]: cluster 2022-04-22T18:28:14.590849+0000 mon.a (mon.0) 5089 : cluster [DBG] osdmap e2121: 8 total, 8 up, 8 in 2022-04-22T18:28:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:16 smithi121 bash[36902]: cluster 2022-04-22T18:28:15.589496+0000 mgr.y (mgr.34107) 2166 : cluster [DBG] pgmap v3708: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:16 smithi121 bash[36902]: cluster 2022-04-22T18:28:15.602321+0000 mon.a (mon.0) 5090 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2022-04-22T18:28:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:16 smithi121 bash[36902]: audit 2022-04-22T18:28:16.297674+0000 mon.a (mon.0) 5091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:16 smithi121 bash[36902]: cluster 2022-04-22T18:28:16.564352+0000 mon.a (mon.0) 5092 : cluster [DBG] osdmap e2123: 8 total, 8 up, 8 in 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[45495]: cluster 2022-04-22T18:28:15.589496+0000 mgr.y (mgr.34107) 2166 : cluster [DBG] pgmap v3708: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[45495]: cluster 2022-04-22T18:28:15.602321+0000 mon.a (mon.0) 5090 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[45495]: audit 2022-04-22T18:28:16.297674+0000 mon.a (mon.0) 5091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[45495]: cluster 2022-04-22T18:28:16.564352+0000 mon.a (mon.0) 5092 : cluster [DBG] osdmap e2123: 8 total, 8 up, 8 in 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[46934]: cluster 2022-04-22T18:28:15.589496+0000 mgr.y (mgr.34107) 2166 : cluster [DBG] pgmap v3708: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[46934]: cluster 2022-04-22T18:28:15.602321+0000 mon.a (mon.0) 5090 : cluster [DBG] osdmap e2122: 8 total, 8 up, 8 in 2022-04-22T18:28:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[46934]: audit 2022-04-22T18:28:16.297674+0000 mon.a (mon.0) 5091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:16 smithi115 bash[46934]: cluster 2022-04-22T18:28:16.564352+0000 mon.a (mon.0) 5092 : cluster [DBG] osdmap e2123: 8 total, 8 up, 8 in 2022-04-22T18:28:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:17 smithi121 bash[36902]: audit 2022-04-22T18:28:16.836576+0000 mon.a (mon.0) 5093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:17 smithi115 bash[45495]: audit 2022-04-22T18:28:16.836576+0000 mon.a (mon.0) 5093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:18.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:17 smithi115 bash[46934]: audit 2022-04-22T18:28:16.836576+0000 mon.a (mon.0) 5093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:18 smithi121 bash[36902]: cluster 2022-04-22T18:28:17.590595+0000 mgr.y (mgr.34107) 2167 : cluster [DBG] pgmap v3711: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:19.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:18 smithi121 bash[36902]: cluster 2022-04-22T18:28:17.616274+0000 mon.a (mon.0) 5094 : cluster [DBG] osdmap e2124: 8 total, 8 up, 8 in 2022-04-22T18:28:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:18 smithi115 bash[45495]: cluster 2022-04-22T18:28:17.590595+0000 mgr.y (mgr.34107) 2167 : cluster [DBG] pgmap v3711: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:18 smithi115 bash[45495]: cluster 2022-04-22T18:28:17.616274+0000 mon.a (mon.0) 5094 : cluster [DBG] osdmap e2124: 8 total, 8 up, 8 in 2022-04-22T18:28:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:18 smithi115 bash[46934]: cluster 2022-04-22T18:28:17.590595+0000 mgr.y (mgr.34107) 2167 : cluster [DBG] pgmap v3711: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 5.0 KiB/s wr, 281 op/s 2022-04-22T18:28:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:18 smithi115 bash[46934]: cluster 2022-04-22T18:28:17.616274+0000 mon.a (mon.0) 5094 : cluster [DBG] osdmap e2124: 8 total, 8 up, 8 in 2022-04-22T18:28:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:19 smithi121 bash[36902]: cluster 2022-04-22T18:28:18.622215+0000 mon.a (mon.0) 5095 : cluster [DBG] osdmap e2125: 8 total, 8 up, 8 in 2022-04-22T18:28:20.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:19 smithi115 bash[45495]: cluster 2022-04-22T18:28:18.622215+0000 mon.a (mon.0) 5095 : cluster [DBG] osdmap e2125: 8 total, 8 up, 8 in 2022-04-22T18:28:20.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:19 smithi115 bash[46934]: cluster 2022-04-22T18:28:18.622215+0000 mon.a (mon.0) 5095 : cluster [DBG] osdmap e2125: 8 total, 8 up, 8 in 2022-04-22T18:28:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:20 smithi121 bash[36902]: cluster 2022-04-22T18:28:19.591684+0000 mgr.y (mgr.34107) 2168 : cluster [DBG] pgmap v3714: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:20 smithi121 bash[36902]: cluster 2022-04-22T18:28:19.640059+0000 mon.a (mon.0) 5096 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2022-04-22T18:28:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:20 smithi115 bash[45495]: cluster 2022-04-22T18:28:19.591684+0000 mgr.y (mgr.34107) 2168 : cluster [DBG] pgmap v3714: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:20 smithi115 bash[45495]: cluster 2022-04-22T18:28:19.640059+0000 mon.a (mon.0) 5096 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2022-04-22T18:28:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:20 smithi115 bash[46934]: cluster 2022-04-22T18:28:19.591684+0000 mgr.y (mgr.34107) 2168 : cluster [DBG] pgmap v3714: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:20 smithi115 bash[46934]: cluster 2022-04-22T18:28:19.640059+0000 mon.a (mon.0) 5096 : cluster [DBG] osdmap e2126: 8 total, 8 up, 8 in 2022-04-22T18:28:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:21 smithi121 bash[36902]: cluster 2022-04-22T18:28:20.642262+0000 mon.a (mon.0) 5097 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2022-04-22T18:28:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:21 smithi121 bash[36902]: audit 2022-04-22T18:28:21.300750+0000 mon.a (mon.0) 5098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:21 smithi121 bash[36902]: cluster 2022-04-22T18:28:21.571160+0000 mon.a (mon.0) 5099 : cluster [DBG] osdmap e2128: 8 total, 8 up, 8 in 2022-04-22T18:28:22.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:21.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[45495]: cluster 2022-04-22T18:28:20.642262+0000 mon.a (mon.0) 5097 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2022-04-22T18:28:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[45495]: audit 2022-04-22T18:28:21.300750+0000 mon.a (mon.0) 5098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[45495]: cluster 2022-04-22T18:28:21.571160+0000 mon.a (mon.0) 5099 : cluster [DBG] osdmap e2128: 8 total, 8 up, 8 in 2022-04-22T18:28:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[46934]: cluster 2022-04-22T18:28:20.642262+0000 mon.a (mon.0) 5097 : cluster [DBG] osdmap e2127: 8 total, 8 up, 8 in 2022-04-22T18:28:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[46934]: audit 2022-04-22T18:28:21.300750+0000 mon.a (mon.0) 5098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:21 smithi115 bash[46934]: cluster 2022-04-22T18:28:21.571160+0000 mon.a (mon.0) 5099 : cluster [DBG] osdmap e2128: 8 total, 8 up, 8 in 2022-04-22T18:28:22.813 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:28:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:22 smithi121 bash[36902]: cluster 2022-04-22T18:28:21.592553+0000 mgr.y (mgr.34107) 2169 : cluster [DBG] pgmap v3718: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 173 KiB/s rd, 5.0 KiB/s wr, 235 op/s 2022-04-22T18:28:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:22 smithi121 bash[36902]: audit 2022-04-22T18:28:21.840271+0000 mon.a (mon.0) 5100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:22 smithi121 bash[36902]: cluster 2022-04-22T18:28:22.585326+0000 mon.a (mon.0) 5101 : cluster [DBG] osdmap e2129: 8 total, 8 up, 8 in 2022-04-22T18:28:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[45495]: cluster 2022-04-22T18:28:21.592553+0000 mgr.y (mgr.34107) 2169 : cluster [DBG] pgmap v3718: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 173 KiB/s rd, 5.0 KiB/s wr, 235 op/s 2022-04-22T18:28:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[45495]: audit 2022-04-22T18:28:21.840271+0000 mon.a (mon.0) 5100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:23.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[45495]: cluster 2022-04-22T18:28:22.585326+0000 mon.a (mon.0) 5101 : cluster [DBG] osdmap e2129: 8 total, 8 up, 8 in 2022-04-22T18:28:23.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[46934]: cluster 2022-04-22T18:28:21.592553+0000 mgr.y (mgr.34107) 2169 : cluster [DBG] pgmap v3718: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 173 KiB/s rd, 5.0 KiB/s wr, 235 op/s 2022-04-22T18:28:23.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[46934]: audit 2022-04-22T18:28:21.840271+0000 mon.a (mon.0) 5100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:23.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:22 smithi115 bash[46934]: cluster 2022-04-22T18:28:22.585326+0000 mon.a (mon.0) 5101 : cluster [DBG] osdmap e2129: 8 total, 8 up, 8 in 2022-04-22T18:28:23.153 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:28:23.153 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (51m) 9m ago 65m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (64m) 14s ago 64m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (53m) 9m ago 66m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (52m) 14s ago 66m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (60m) 14s ago 70m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (61m) 9m ago 72m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (60m) 9m ago 72m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (59m) 14s ago 70m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (59m) 9m ago 70m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:28:23.154 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (52m) 9m ago 65m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (52m) 14s ago 65m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (58m) 9m ago 70m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (57m) 9m ago 69m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (57m) 9m ago 69m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (56m) 9m ago 68m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (55m) 14s ago 68m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (55m) 14s ago 67m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (54m) 14s ago 67m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:28:23.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (53m) 14s ago 66m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:28:23.156 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (52m) 14s ago 65m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:28:23.558 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:28:23.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:28:23.560 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:28:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:23 smithi121 bash[36902]: audit 2022-04-22T18:28:23.550935+0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.115:0/4157791509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:23 smithi121 bash[36902]: cluster 2022-04-22T18:28:23.589121+0000 mon.a (mon.0) 5102 : cluster [DBG] osdmap e2130: 8 total, 8 up, 8 in 2022-04-22T18:28:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:23.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:24.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:23 smithi115 bash[45495]: audit 2022-04-22T18:28:23.550935+0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.115:0/4157791509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:23 smithi115 bash[45495]: cluster 2022-04-22T18:28:23.589121+0000 mon.a (mon.0) 5102 : cluster [DBG] osdmap e2130: 8 total, 8 up, 8 in 2022-04-22T18:28:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:23 smithi115 bash[46934]: audit 2022-04-22T18:28:23.550935+0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.115:0/4157791509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:23 smithi115 bash[46934]: cluster 2022-04-22T18:28:23.589121+0000 mon.a (mon.0) 5102 : cluster [DBG] osdmap e2130: 8 total, 8 up, 8 in 2022-04-22T18:28:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:24 smithi121 bash[36902]: audit 2022-04-22T18:28:22.790677+0000 mgr.y (mgr.34107) 2170 : audit [DBG] from='client.37632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:24 smithi121 bash[36902]: audit 2022-04-22T18:28:23.132114+0000 mgr.y (mgr.34107) 2171 : audit [DBG] from='client.37638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:24 smithi121 bash[36902]: cluster 2022-04-22T18:28:23.593165+0000 mgr.y (mgr.34107) 2172 : cluster [DBG] pgmap v3721: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 176 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2022-04-22T18:28:25.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:24 smithi121 bash[36902]: cluster 2022-04-22T18:28:24.598883+0000 mon.a (mon.0) 5103 : cluster [DBG] osdmap e2131: 8 total, 8 up, 8 in 2022-04-22T18:28:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[45495]: audit 2022-04-22T18:28:22.790677+0000 mgr.y (mgr.34107) 2170 : audit [DBG] from='client.37632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[45495]: audit 2022-04-22T18:28:23.132114+0000 mgr.y (mgr.34107) 2171 : audit [DBG] from='client.37638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[45495]: cluster 2022-04-22T18:28:23.593165+0000 mgr.y (mgr.34107) 2172 : cluster [DBG] pgmap v3721: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 176 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2022-04-22T18:28:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[45495]: cluster 2022-04-22T18:28:24.598883+0000 mon.a (mon.0) 5103 : cluster [DBG] osdmap e2131: 8 total, 8 up, 8 in 2022-04-22T18:28:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[46934]: audit 2022-04-22T18:28:22.790677+0000 mgr.y (mgr.34107) 2170 : audit [DBG] from='client.37632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[46934]: audit 2022-04-22T18:28:23.132114+0000 mgr.y (mgr.34107) 2171 : audit [DBG] from='client.37638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[46934]: cluster 2022-04-22T18:28:23.593165+0000 mgr.y (mgr.34107) 2172 : cluster [DBG] pgmap v3721: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 176 KiB/s rd, 5.1 KiB/s wr, 240 op/s 2022-04-22T18:28:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:24 smithi115 bash[46934]: cluster 2022-04-22T18:28:24.598883+0000 mon.a (mon.0) 5103 : cluster [DBG] osdmap e2131: 8 total, 8 up, 8 in 2022-04-22T18:28:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:26 smithi121 bash[36902]: cluster 2022-04-22T18:28:25.594191+0000 mgr.y (mgr.34107) 2173 : cluster [DBG] pgmap v3723: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2022-04-22T18:28:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:26 smithi121 bash[36902]: cluster 2022-04-22T18:28:25.606558+0000 mon.a (mon.0) 5104 : cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2022-04-22T18:28:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:26 smithi121 bash[36902]: audit 2022-04-22T18:28:26.303747+0000 mon.a (mon.0) 5105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:26 smithi121 bash[36902]: cluster 2022-04-22T18:28:26.570304+0000 mon.a (mon.0) 5106 : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2022-04-22T18:28:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[46934]: cluster 2022-04-22T18:28:25.594191+0000 mgr.y (mgr.34107) 2173 : cluster [DBG] pgmap v3723: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2022-04-22T18:28:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[46934]: cluster 2022-04-22T18:28:25.606558+0000 mon.a (mon.0) 5104 : cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2022-04-22T18:28:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[46934]: audit 2022-04-22T18:28:26.303747+0000 mon.a (mon.0) 5105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[46934]: cluster 2022-04-22T18:28:26.570304+0000 mon.a (mon.0) 5106 : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2022-04-22T18:28:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[45495]: cluster 2022-04-22T18:28:25.594191+0000 mgr.y (mgr.34107) 2173 : cluster [DBG] pgmap v3723: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2022-04-22T18:28:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[45495]: cluster 2022-04-22T18:28:25.606558+0000 mon.a (mon.0) 5104 : cluster [DBG] osdmap e2132: 8 total, 8 up, 8 in 2022-04-22T18:28:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[45495]: audit 2022-04-22T18:28:26.303747+0000 mon.a (mon.0) 5105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:26 smithi115 bash[45495]: cluster 2022-04-22T18:28:26.570304+0000 mon.a (mon.0) 5106 : cluster [DBG] osdmap e2133: 8 total, 8 up, 8 in 2022-04-22T18:28:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:27 smithi121 bash[36902]: audit 2022-04-22T18:28:26.843893+0000 mon.a (mon.0) 5107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:27 smithi115 bash[45495]: audit 2022-04-22T18:28:26.843893+0000 mon.a (mon.0) 5107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:27 smithi115 bash[46934]: audit 2022-04-22T18:28:26.843893+0000 mon.a (mon.0) 5107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:28 smithi121 bash[36902]: cluster 2022-04-22T18:28:27.595305+0000 mgr.y (mgr.34107) 2174 : cluster [DBG] pgmap v3726: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 192 op/s 2022-04-22T18:28:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:28 smithi121 bash[36902]: cluster 2022-04-22T18:28:27.612873+0000 mon.a (mon.0) 5108 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2022-04-22T18:28:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:28 smithi115 bash[45495]: cluster 2022-04-22T18:28:27.595305+0000 mgr.y (mgr.34107) 2174 : cluster [DBG] pgmap v3726: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 192 op/s 2022-04-22T18:28:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:28 smithi115 bash[45495]: cluster 2022-04-22T18:28:27.612873+0000 mon.a (mon.0) 5108 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2022-04-22T18:28:29.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:28 smithi115 bash[46934]: cluster 2022-04-22T18:28:27.595305+0000 mgr.y (mgr.34107) 2174 : cluster [DBG] pgmap v3726: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 140 KiB/s rd, 5.0 KiB/s wr, 192 op/s 2022-04-22T18:28:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:28 smithi115 bash[46934]: cluster 2022-04-22T18:28:27.612873+0000 mon.a (mon.0) 5108 : cluster [DBG] osdmap e2134: 8 total, 8 up, 8 in 2022-04-22T18:28:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:29 smithi121 bash[36902]: cluster 2022-04-22T18:28:28.614056+0000 mon.a (mon.0) 5109 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2022-04-22T18:28:30.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:29 smithi115 bash[45495]: cluster 2022-04-22T18:28:28.614056+0000 mon.a (mon.0) 5109 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2022-04-22T18:28:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:29 smithi115 bash[46934]: cluster 2022-04-22T18:28:28.614056+0000 mon.a (mon.0) 5109 : cluster [DBG] osdmap e2135: 8 total, 8 up, 8 in 2022-04-22T18:28:31.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:30 smithi121 bash[36902]: cluster 2022-04-22T18:28:29.596459+0000 mgr.y (mgr.34107) 2175 : cluster [DBG] pgmap v3729: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:31.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:30 smithi121 bash[36902]: cluster 2022-04-22T18:28:29.631593+0000 mon.a (mon.0) 5110 : cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2022-04-22T18:28:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:30 smithi115 bash[45495]: cluster 2022-04-22T18:28:29.596459+0000 mgr.y (mgr.34107) 2175 : cluster [DBG] pgmap v3729: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:30 smithi115 bash[45495]: cluster 2022-04-22T18:28:29.631593+0000 mon.a (mon.0) 5110 : cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2022-04-22T18:28:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:30 smithi115 bash[46934]: cluster 2022-04-22T18:28:29.596459+0000 mgr.y (mgr.34107) 2175 : cluster [DBG] pgmap v3729: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:30 smithi115 bash[46934]: cluster 2022-04-22T18:28:29.631593+0000 mon.a (mon.0) 5110 : cluster [DBG] osdmap e2136: 8 total, 8 up, 8 in 2022-04-22T18:28:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:31 smithi121 bash[36902]: cluster 2022-04-22T18:28:30.635934+0000 mon.a (mon.0) 5111 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2022-04-22T18:28:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:31 smithi121 bash[36902]: audit 2022-04-22T18:28:31.306867+0000 mon.a (mon.0) 5112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:31 smithi121 bash[36902]: cluster 2022-04-22T18:28:31.572483+0000 mon.a (mon.0) 5113 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2022-04-22T18:28:32.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:31.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[45495]: cluster 2022-04-22T18:28:30.635934+0000 mon.a (mon.0) 5111 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[45495]: audit 2022-04-22T18:28:31.306867+0000 mon.a (mon.0) 5112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[45495]: cluster 2022-04-22T18:28:31.572483+0000 mon.a (mon.0) 5113 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[46934]: cluster 2022-04-22T18:28:30.635934+0000 mon.a (mon.0) 5111 : cluster [DBG] osdmap e2137: 8 total, 8 up, 8 in 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[46934]: audit 2022-04-22T18:28:31.306867+0000 mon.a (mon.0) 5112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:32.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:31 smithi115 bash[46934]: cluster 2022-04-22T18:28:31.572483+0000 mon.a (mon.0) 5113 : cluster [DBG] osdmap e2138: 8 total, 8 up, 8 in 2022-04-22T18:28:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:32 smithi121 bash[36902]: cluster 2022-04-22T18:28:31.597403+0000 mgr.y (mgr.34107) 2176 : cluster [DBG] pgmap v3733: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.0 KiB/s wr, 149 op/s 2022-04-22T18:28:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:32 smithi121 bash[36902]: audit 2022-04-22T18:28:31.847438+0000 mon.a (mon.0) 5114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:32 smithi121 bash[36902]: cluster 2022-04-22T18:28:32.588693+0000 mon.a (mon.0) 5115 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2022-04-22T18:28:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[45495]: cluster 2022-04-22T18:28:31.597403+0000 mgr.y (mgr.34107) 2176 : cluster [DBG] pgmap v3733: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.0 KiB/s wr, 149 op/s 2022-04-22T18:28:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[45495]: audit 2022-04-22T18:28:31.847438+0000 mon.a (mon.0) 5114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[45495]: cluster 2022-04-22T18:28:32.588693+0000 mon.a (mon.0) 5115 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2022-04-22T18:28:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[46934]: cluster 2022-04-22T18:28:31.597403+0000 mgr.y (mgr.34107) 2176 : cluster [DBG] pgmap v3733: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.0 KiB/s wr, 149 op/s 2022-04-22T18:28:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[46934]: audit 2022-04-22T18:28:31.847438+0000 mon.a (mon.0) 5114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:32 smithi115 bash[46934]: cluster 2022-04-22T18:28:32.588693+0000 mon.a (mon.0) 5115 : cluster [DBG] osdmap e2139: 8 total, 8 up, 8 in 2022-04-22T18:28:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:33.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:34 smithi121 bash[36902]: cluster 2022-04-22T18:28:33.596393+0000 mon.a (mon.0) 5116 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2022-04-22T18:28:35.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:34 smithi121 bash[36902]: cluster 2022-04-22T18:28:33.598225+0000 mgr.y (mgr.34107) 2177 : cluster [DBG] pgmap v3736: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 111 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2022-04-22T18:28:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:34 smithi115 bash[45495]: cluster 2022-04-22T18:28:33.596393+0000 mon.a (mon.0) 5116 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2022-04-22T18:28:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:34 smithi115 bash[45495]: cluster 2022-04-22T18:28:33.598225+0000 mgr.y (mgr.34107) 2177 : cluster [DBG] pgmap v3736: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 111 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2022-04-22T18:28:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:34 smithi115 bash[46934]: cluster 2022-04-22T18:28:33.596393+0000 mon.a (mon.0) 5116 : cluster [DBG] osdmap e2140: 8 total, 8 up, 8 in 2022-04-22T18:28:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:34 smithi115 bash[46934]: cluster 2022-04-22T18:28:33.598225+0000 mgr.y (mgr.34107) 2177 : cluster [DBG] pgmap v3736: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 111 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2022-04-22T18:28:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:35 smithi121 bash[36902]: cluster 2022-04-22T18:28:34.604054+0000 mon.a (mon.0) 5117 : cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2022-04-22T18:28:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:35 smithi115 bash[45495]: cluster 2022-04-22T18:28:34.604054+0000 mon.a (mon.0) 5117 : cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2022-04-22T18:28:36.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:35 smithi115 bash[46934]: cluster 2022-04-22T18:28:34.604054+0000 mon.a (mon.0) 5117 : cluster [DBG] osdmap e2141: 8 total, 8 up, 8 in 2022-04-22T18:28:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:36 smithi121 bash[36902]: cluster 2022-04-22T18:28:35.599283+0000 mgr.y (mgr.34107) 2178 : cluster [DBG] pgmap v3738: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:36 smithi121 bash[36902]: cluster 2022-04-22T18:28:35.609749+0000 mon.a (mon.0) 5118 : cluster [DBG] osdmap e2142: 8 total, 8 up, 8 in 2022-04-22T18:28:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:36 smithi121 bash[36902]: audit 2022-04-22T18:28:36.309852+0000 mon.a (mon.0) 5119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:36 smithi121 bash[36902]: cluster 2022-04-22T18:28:36.575947+0000 mon.a (mon.0) 5120 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2022-04-22T18:28:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[46934]: cluster 2022-04-22T18:28:35.599283+0000 mgr.y (mgr.34107) 2178 : cluster [DBG] pgmap v3738: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[46934]: cluster 2022-04-22T18:28:35.609749+0000 mon.a (mon.0) 5118 : cluster [DBG] osdmap e2142: 8 total, 8 up, 8 in 2022-04-22T18:28:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[46934]: audit 2022-04-22T18:28:36.309852+0000 mon.a (mon.0) 5119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:37.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[46934]: cluster 2022-04-22T18:28:36.575947+0000 mon.a (mon.0) 5120 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2022-04-22T18:28:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[45495]: cluster 2022-04-22T18:28:35.599283+0000 mgr.y (mgr.34107) 2178 : cluster [DBG] pgmap v3738: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[45495]: cluster 2022-04-22T18:28:35.609749+0000 mon.a (mon.0) 5118 : cluster [DBG] osdmap e2142: 8 total, 8 up, 8 in 2022-04-22T18:28:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[45495]: audit 2022-04-22T18:28:36.309852+0000 mon.a (mon.0) 5119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:37.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:36 smithi115 bash[45495]: cluster 2022-04-22T18:28:36.575947+0000 mon.a (mon.0) 5120 : cluster [DBG] osdmap e2143: 8 total, 8 up, 8 in 2022-04-22T18:28:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:37 smithi121 bash[36902]: audit 2022-04-22T18:28:36.850671+0000 mon.a (mon.0) 5121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:37 smithi115 bash[45495]: audit 2022-04-22T18:28:36.850671+0000 mon.a (mon.0) 5121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:38.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:37 smithi115 bash[46934]: audit 2022-04-22T18:28:36.850671+0000 mon.a (mon.0) 5121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:38 smithi121 bash[36902]: cluster 2022-04-22T18:28:37.600426+0000 mgr.y (mgr.34107) 2179 : cluster [DBG] pgmap v3741: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:38 smithi121 bash[36902]: cluster 2022-04-22T18:28:37.629514+0000 mon.a (mon.0) 5122 : cluster [DBG] osdmap e2144: 8 total, 8 up, 8 in 2022-04-22T18:28:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:38 smithi115 bash[45495]: cluster 2022-04-22T18:28:37.600426+0000 mgr.y (mgr.34107) 2179 : cluster [DBG] pgmap v3741: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:38 smithi115 bash[45495]: cluster 2022-04-22T18:28:37.629514+0000 mon.a (mon.0) 5122 : cluster [DBG] osdmap e2144: 8 total, 8 up, 8 in 2022-04-22T18:28:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:38 smithi115 bash[46934]: cluster 2022-04-22T18:28:37.600426+0000 mgr.y (mgr.34107) 2179 : cluster [DBG] pgmap v3741: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 76 KiB/s rd, 5.0 KiB/s wr, 106 op/s 2022-04-22T18:28:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:38 smithi115 bash[46934]: cluster 2022-04-22T18:28:37.629514+0000 mon.a (mon.0) 5122 : cluster [DBG] osdmap e2144: 8 total, 8 up, 8 in 2022-04-22T18:28:39.803 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_many_snaps ... ok 2022-04-22T18:28:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:39 smithi121 bash[36902]: cluster 2022-04-22T18:28:38.636147+0000 mon.a (mon.0) 5123 : cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2022-04-22T18:28:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:39 smithi115 bash[45495]: cluster 2022-04-22T18:28:38.636147+0000 mon.a (mon.0) 5123 : cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2022-04-22T18:28:40.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:39 smithi115 bash[46934]: cluster 2022-04-22T18:28:38.636147+0000 mon.a (mon.0) 5123 : cluster [DBG] osdmap e2145: 8 total, 8 up, 8 in 2022-04-22T18:28:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:40 smithi121 bash[36902]: cluster 2022-04-22T18:28:39.601401+0000 mgr.y (mgr.34107) 2180 : cluster [DBG] pgmap v3744: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:40 smithi121 bash[36902]: cluster 2022-04-22T18:28:39.646272+0000 mon.a (mon.0) 5124 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2022-04-22T18:28:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:40 smithi115 bash[45495]: cluster 2022-04-22T18:28:39.601401+0000 mgr.y (mgr.34107) 2180 : cluster [DBG] pgmap v3744: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:40 smithi115 bash[45495]: cluster 2022-04-22T18:28:39.646272+0000 mon.a (mon.0) 5124 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2022-04-22T18:28:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:40 smithi115 bash[46934]: cluster 2022-04-22T18:28:39.601401+0000 mgr.y (mgr.34107) 2180 : cluster [DBG] pgmap v3744: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2022-04-22T18:28:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:40 smithi115 bash[46934]: cluster 2022-04-22T18:28:39.646272+0000 mon.a (mon.0) 5124 : cluster [DBG] osdmap e2146: 8 total, 8 up, 8 in 2022-04-22T18:28:41.427 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_metadata ... ok 2022-04-22T18:28:41.600 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_modify_timestamp ... ok 2022-04-22T18:28:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:41 smithi121 bash[36902]: audit 2022-04-22T18:28:41.313111+0000 mon.a (mon.0) 5125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:42.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:41 smithi121 bash[36902]: cluster 2022-04-22T18:28:41.574321+0000 mon.a (mon.0) 5126 : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2022-04-22T18:28:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:41 smithi115 bash[45495]: audit 2022-04-22T18:28:41.313111+0000 mon.a (mon.0) 5125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:41 smithi115 bash[45495]: cluster 2022-04-22T18:28:41.574321+0000 mon.a (mon.0) 5126 : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2022-04-22T18:28:42.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:41.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:41 smithi115 bash[46934]: audit 2022-04-22T18:28:41.313111+0000 mon.a (mon.0) 5125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:41 smithi115 bash[46934]: cluster 2022-04-22T18:28:41.574321+0000 mon.a (mon.0) 5126 : cluster [DBG] osdmap e2147: 8 total, 8 up, 8 in 2022-04-22T18:28:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:42 smithi121 bash[36902]: cluster 2022-04-22T18:28:41.601980+0000 mgr.y (mgr.34107) 2181 : cluster [DBG] pgmap v3747: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 4.2 KiB/s wr, 82 op/s 2022-04-22T18:28:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:42 smithi121 bash[36902]: audit 2022-04-22T18:28:41.854227+0000 mon.a (mon.0) 5127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:42 smithi121 bash[36902]: cluster 2022-04-22T18:28:42.593787+0000 mon.a (mon.0) 5128 : cluster [DBG] osdmap e2148: 8 total, 8 up, 8 in 2022-04-22T18:28:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[45495]: cluster 2022-04-22T18:28:41.601980+0000 mgr.y (mgr.34107) 2181 : cluster [DBG] pgmap v3747: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 4.2 KiB/s wr, 82 op/s 2022-04-22T18:28:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[45495]: audit 2022-04-22T18:28:41.854227+0000 mon.a (mon.0) 5127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[45495]: cluster 2022-04-22T18:28:42.593787+0000 mon.a (mon.0) 5128 : cluster [DBG] osdmap e2148: 8 total, 8 up, 8 in 2022-04-22T18:28:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[46934]: cluster 2022-04-22T18:28:41.601980+0000 mgr.y (mgr.34107) 2181 : cluster [DBG] pgmap v3747: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 4.2 KiB/s wr, 82 op/s 2022-04-22T18:28:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[46934]: audit 2022-04-22T18:28:41.854227+0000 mon.a (mon.0) 5127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:42 smithi115 bash[46934]: cluster 2022-04-22T18:28:42.593787+0000 mon.a (mon.0) 5128 : cluster [DBG] osdmap e2148: 8 total, 8 up, 8 in 2022-04-22T18:28:43.752 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_protect_snap ... ok 2022-04-22T18:28:43.922 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_read ... ok 2022-04-22T18:28:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:43.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:44.083 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_read_bad_offset ... ok 2022-04-22T18:28:44.234 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_read_with_fadvise_flags ... ok 2022-04-22T18:28:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:44 smithi121 bash[36902]: cluster 2022-04-22T18:28:43.597283+0000 mon.a (mon.0) 5129 : cluster [DBG] osdmap e2149: 8 total, 8 up, 8 in 2022-04-22T18:28:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:44 smithi121 bash[36902]: cluster 2022-04-22T18:28:43.603275+0000 mgr.y (mgr.34107) 2182 : cluster [DBG] pgmap v3750: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 877 op/s 2022-04-22T18:28:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:44 smithi115 bash[45495]: cluster 2022-04-22T18:28:43.597283+0000 mon.a (mon.0) 5129 : cluster [DBG] osdmap e2149: 8 total, 8 up, 8 in 2022-04-22T18:28:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:44 smithi115 bash[45495]: cluster 2022-04-22T18:28:43.603275+0000 mgr.y (mgr.34107) 2182 : cluster [DBG] pgmap v3750: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 877 op/s 2022-04-22T18:28:45.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:44 smithi115 bash[46934]: cluster 2022-04-22T18:28:43.597283+0000 mon.a (mon.0) 5129 : cluster [DBG] osdmap e2149: 8 total, 8 up, 8 in 2022-04-22T18:28:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:44 smithi115 bash[46934]: cluster 2022-04-22T18:28:43.603275+0000 mgr.y (mgr.34107) 2182 : cluster [DBG] pgmap v3750: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 44 KiB/s wr, 877 op/s 2022-04-22T18:28:45.768 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_snap ... ok 2022-04-22T18:28:46.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:45 smithi121 bash[36902]: cluster 2022-04-22T18:28:44.605424+0000 mon.a (mon.0) 5130 : cluster [DBG] osdmap e2150: 8 total, 8 up, 8 in 2022-04-22T18:28:46.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:45 smithi115 bash[45495]: cluster 2022-04-22T18:28:44.605424+0000 mon.a (mon.0) 5130 : cluster [DBG] osdmap e2150: 8 total, 8 up, 8 in 2022-04-22T18:28:46.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:45 smithi115 bash[46934]: cluster 2022-04-22T18:28:44.605424+0000 mon.a (mon.0) 5130 : cluster [DBG] osdmap e2150: 8 total, 8 up, 8 in 2022-04-22T18:28:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:46 smithi121 bash[36902]: cluster 2022-04-22T18:28:45.604330+0000 mgr.y (mgr.34107) 2183 : cluster [DBG] pgmap v3752: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 41 KiB/s wr, 839 op/s 2022-04-22T18:28:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:46 smithi121 bash[36902]: cluster 2022-04-22T18:28:45.612908+0000 mon.a (mon.0) 5131 : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2022-04-22T18:28:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:46 smithi121 bash[36902]: audit 2022-04-22T18:28:46.316371+0000 mon.a (mon.0) 5132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:46 smithi121 bash[36902]: cluster 2022-04-22T18:28:46.577251+0000 mon.a (mon.0) 5133 : cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2022-04-22T18:28:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[45495]: cluster 2022-04-22T18:28:45.604330+0000 mgr.y (mgr.34107) 2183 : cluster [DBG] pgmap v3752: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 41 KiB/s wr, 839 op/s 2022-04-22T18:28:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[45495]: cluster 2022-04-22T18:28:45.612908+0000 mon.a (mon.0) 5131 : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2022-04-22T18:28:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[45495]: audit 2022-04-22T18:28:46.316371+0000 mon.a (mon.0) 5132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[45495]: cluster 2022-04-22T18:28:46.577251+0000 mon.a (mon.0) 5133 : cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2022-04-22T18:28:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[46934]: cluster 2022-04-22T18:28:45.604330+0000 mgr.y (mgr.34107) 2183 : cluster [DBG] pgmap v3752: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 41 KiB/s wr, 839 op/s 2022-04-22T18:28:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[46934]: cluster 2022-04-22T18:28:45.612908+0000 mon.a (mon.0) 5131 : cluster [DBG] osdmap e2151: 8 total, 8 up, 8 in 2022-04-22T18:28:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[46934]: audit 2022-04-22T18:28:46.316371+0000 mon.a (mon.0) 5132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:46 smithi115 bash[46934]: cluster 2022-04-22T18:28:46.577251+0000 mon.a (mon.0) 5133 : cluster [DBG] osdmap e2152: 8 total, 8 up, 8 in 2022-04-22T18:28:47.788 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_snap2 ... ok 2022-04-22T18:28:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:47 smithi121 bash[36902]: audit 2022-04-22T18:28:46.857718+0000 mon.a (mon.0) 5134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:48.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:47 smithi115 bash[45495]: audit 2022-04-22T18:28:46.857718+0000 mon.a (mon.0) 5134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:48.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:47 smithi115 bash[46934]: audit 2022-04-22T18:28:46.857718+0000 mon.a (mon.0) 5134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:48 smithi121 bash[36902]: cluster 2022-04-22T18:28:47.605416+0000 mgr.y (mgr.34107) 2184 : cluster [DBG] pgmap v3755: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 5.7 KiB/s wr, 161 op/s 2022-04-22T18:28:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:48 smithi121 bash[36902]: cluster 2022-04-22T18:28:47.621788+0000 mon.a (mon.0) 5135 : cluster [DBG] osdmap e2153: 8 total, 8 up, 8 in 2022-04-22T18:28:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:48 smithi115 bash[46934]: cluster 2022-04-22T18:28:47.605416+0000 mgr.y (mgr.34107) 2184 : cluster [DBG] pgmap v3755: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 5.7 KiB/s wr, 161 op/s 2022-04-22T18:28:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:48 smithi115 bash[46934]: cluster 2022-04-22T18:28:47.621788+0000 mon.a (mon.0) 5135 : cluster [DBG] osdmap e2153: 8 total, 8 up, 8 in 2022-04-22T18:28:49.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:48 smithi115 bash[45495]: cluster 2022-04-22T18:28:47.605416+0000 mgr.y (mgr.34107) 2184 : cluster [DBG] pgmap v3755: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 5.7 KiB/s wr, 161 op/s 2022-04-22T18:28:49.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:48 smithi115 bash[45495]: cluster 2022-04-22T18:28:47.621788+0000 mon.a (mon.0) 5135 : cluster [DBG] osdmap e2153: 8 total, 8 up, 8 in 2022-04-22T18:28:49.801 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_snap_by_id ... ok 2022-04-22T18:28:49.968 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_snap_not_found ... ok 2022-04-22T18:28:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:49 smithi121 bash[36902]: cluster 2022-04-22T18:28:48.641670+0000 mon.a (mon.0) 5136 : cluster [DBG] osdmap e2154: 8 total, 8 up, 8 in 2022-04-22T18:28:50.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:49 smithi115 bash[45495]: cluster 2022-04-22T18:28:48.641670+0000 mon.a (mon.0) 5136 : cluster [DBG] osdmap e2154: 8 total, 8 up, 8 in 2022-04-22T18:28:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:49 smithi115 bash[46934]: cluster 2022-04-22T18:28:48.641670+0000 mon.a (mon.0) 5136 : cluster [DBG] osdmap e2154: 8 total, 8 up, 8 in 2022-04-22T18:28:50.170 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_with_exclusive_lock ... ok 2022-04-22T18:28:50.324 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_with_snap ... SKIP 2022-04-22T18:28:50.487 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_remove_with_watcher ... SKIP 2022-04-22T18:28:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:50 smithi121 bash[36902]: cluster 2022-04-22T18:28:49.606452+0000 mgr.y (mgr.34107) 2185 : cluster [DBG] pgmap v3758: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 154 KiB/s rd, 11 KiB/s wr, 213 op/s 2022-04-22T18:28:51.039 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:50 smithi121 bash[36902]: cluster 2022-04-22T18:28:49.661098+0000 mon.a (mon.0) 5137 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2022-04-22T18:28:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:50 smithi115 bash[45495]: cluster 2022-04-22T18:28:49.606452+0000 mgr.y (mgr.34107) 2185 : cluster [DBG] pgmap v3758: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 154 KiB/s rd, 11 KiB/s wr, 213 op/s 2022-04-22T18:28:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:50 smithi115 bash[45495]: cluster 2022-04-22T18:28:49.661098+0000 mon.a (mon.0) 5137 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2022-04-22T18:28:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:50 smithi115 bash[46934]: cluster 2022-04-22T18:28:49.606452+0000 mgr.y (mgr.34107) 2185 : cluster [DBG] pgmap v3758: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 154 KiB/s rd, 11 KiB/s wr, 213 op/s 2022-04-22T18:28:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:50 smithi115 bash[46934]: cluster 2022-04-22T18:28:49.661098+0000 mon.a (mon.0) 5137 : cluster [DBG] osdmap e2155: 8 total, 8 up, 8 in 2022-04-22T18:28:51.747 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_rename_snap ... ok 2022-04-22T18:28:51.935 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_resize ... ok 2022-04-22T18:28:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:51 smithi121 bash[36902]: cluster 2022-04-22T18:28:50.670982+0000 mon.a (mon.0) 5138 : cluster [DBG] osdmap e2156: 8 total, 8 up, 8 in 2022-04-22T18:28:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:51 smithi121 bash[36902]: audit 2022-04-22T18:28:51.319366+0000 mon.a (mon.0) 5139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:51 smithi121 bash[36902]: cluster 2022-04-22T18:28:51.585714+0000 mon.a (mon.0) 5140 : cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2022-04-22T18:28:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:51.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[46934]: cluster 2022-04-22T18:28:50.670982+0000 mon.a (mon.0) 5138 : cluster [DBG] osdmap e2156: 8 total, 8 up, 8 in 2022-04-22T18:28:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[46934]: audit 2022-04-22T18:28:51.319366+0000 mon.a (mon.0) 5139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[46934]: cluster 2022-04-22T18:28:51.585714+0000 mon.a (mon.0) 5140 : cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2022-04-22T18:28:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[45495]: cluster 2022-04-22T18:28:50.670982+0000 mon.a (mon.0) 5138 : cluster [DBG] osdmap e2156: 8 total, 8 up, 8 in 2022-04-22T18:28:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[45495]: audit 2022-04-22T18:28:51.319366+0000 mon.a (mon.0) 5139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:51 smithi115 bash[45495]: cluster 2022-04-22T18:28:51.585714+0000 mon.a (mon.0) 5140 : cluster [DBG] osdmap e2157: 8 total, 8 up, 8 in 2022-04-22T18:28:52.126 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_resize_allow_shrink_False ... ok 2022-04-22T18:28:52.356 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_resize_bytes ... ok 2022-04-22T18:28:52.567 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_resize_down ... ok 2022-04-22T18:28:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:52 smithi121 bash[36902]: cluster 2022-04-22T18:28:51.607084+0000 mgr.y (mgr.34107) 2186 : cluster [DBG] pgmap v3762: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 201 op/s 2022-04-22T18:28:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:52 smithi121 bash[36902]: audit 2022-04-22T18:28:51.861576+0000 mon.a (mon.0) 5141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:52 smithi115 bash[45495]: cluster 2022-04-22T18:28:51.607084+0000 mgr.y (mgr.34107) 2186 : cluster [DBG] pgmap v3762: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 201 op/s 2022-04-22T18:28:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:52 smithi115 bash[45495]: audit 2022-04-22T18:28:51.861576+0000 mon.a (mon.0) 5141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:52 smithi115 bash[46934]: cluster 2022-04-22T18:28:51.607084+0000 mgr.y (mgr.34107) 2186 : cluster [DBG] pgmap v3762: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 143 KiB/s rd, 11 KiB/s wr, 201 op/s 2022-04-22T18:28:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:52 smithi115 bash[46934]: audit 2022-04-22T18:28:51.861576+0000 mon.a (mon.0) 5141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:53.938 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:28:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:28:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:28:53.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:28:54.289 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:28:54.289 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (51m) 9m ago 65m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T18:28:54.289 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (65m) 45s ago 65m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (53m) 9m ago 66m 23.2M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (53m) 45s ago 66m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (61m) 45s ago 71m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (61m) 9m ago 73m 471M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (60m) 9m ago 73m 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (59m) 45s ago 71m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (60m) 9m ago 71m 121M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (53m) 9m ago 65m 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (52m) 45s ago 65m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:28:54.290 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (58m) 9m ago 70m 665M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (57m) 9m ago 70m 741M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (57m) 9m ago 69m 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (57m) 9m ago 69m 440M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (56m) 45s ago 68m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (55m) 45s ago 68m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (55m) 45s ago 67m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (54m) 45s ago 67m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:28:54.291 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (52m) 45s ago 66m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:28:54.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:28:54.705 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:28:54.706 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:28:54.764 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_rollback_to_snap ... ok 2022-04-22T18:28:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:54 smithi121 bash[36902]: cluster 2022-04-22T18:28:53.598979+0000 mon.a (mon.0) 5142 : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2022-04-22T18:28:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:54 smithi121 bash[36902]: cluster 2022-04-22T18:28:53.607799+0000 mgr.y (mgr.34107) 2187 : cluster [DBG] pgmap v3764: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 19 KiB/s wr, 408 op/s 2022-04-22T18:28:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:54 smithi115 bash[45495]: cluster 2022-04-22T18:28:53.598979+0000 mon.a (mon.0) 5142 : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2022-04-22T18:28:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:54 smithi115 bash[45495]: cluster 2022-04-22T18:28:53.607799+0000 mgr.y (mgr.34107) 2187 : cluster [DBG] pgmap v3764: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 19 KiB/s wr, 408 op/s 2022-04-22T18:28:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:54 smithi115 bash[46934]: cluster 2022-04-22T18:28:53.598979+0000 mon.a (mon.0) 5142 : cluster [DBG] osdmap e2158: 8 total, 8 up, 8 in 2022-04-22T18:28:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:54 smithi115 bash[46934]: cluster 2022-04-22T18:28:53.607799+0000 mgr.y (mgr.34107) 2187 : cluster [DBG] pgmap v3764: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 301 KiB/s rd, 19 KiB/s wr, 408 op/s 2022-04-22T18:28:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:55 smithi121 bash[36902]: audit 2022-04-22T18:28:53.916226+0000 mgr.y (mgr.34107) 2188 : audit [DBG] from='client.47584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:55 smithi121 bash[36902]: audit 2022-04-22T18:28:54.256901+0000 mgr.y (mgr.34107) 2189 : audit [DBG] from='client.47590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:55 smithi121 bash[36902]: cluster 2022-04-22T18:28:54.607822+0000 mon.a (mon.0) 5143 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2022-04-22T18:28:56.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:55 smithi121 bash[36902]: audit 2022-04-22T18:28:54.696578+0000 mon.c (mon.1) 131 : audit [DBG] from='client.? 172.21.15.115:0/2494092288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:56.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[45495]: audit 2022-04-22T18:28:53.916226+0000 mgr.y (mgr.34107) 2188 : audit [DBG] from='client.47584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[45495]: audit 2022-04-22T18:28:54.256901+0000 mgr.y (mgr.34107) 2189 : audit [DBG] from='client.47590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[45495]: cluster 2022-04-22T18:28:54.607822+0000 mon.a (mon.0) 5143 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[45495]: audit 2022-04-22T18:28:54.696578+0000 mon.c (mon.1) 131 : audit [DBG] from='client.? 172.21.15.115:0/2494092288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[46934]: audit 2022-04-22T18:28:53.916226+0000 mgr.y (mgr.34107) 2188 : audit [DBG] from='client.47584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[46934]: audit 2022-04-22T18:28:54.256901+0000 mgr.y (mgr.34107) 2189 : audit [DBG] from='client.47590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[46934]: cluster 2022-04-22T18:28:54.607822+0000 mon.a (mon.0) 5143 : cluster [DBG] osdmap e2159: 8 total, 8 up, 8 in 2022-04-22T18:28:56.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:55 smithi115 bash[46934]: audit 2022-04-22T18:28:54.696578+0000 mon.c (mon.1) 131 : audit [DBG] from='client.? 172.21.15.115:0/2494092288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:28:56.742 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_rollback_to_snap_sparse ... ok 2022-04-22T18:28:57.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:56 smithi121 bash[36902]: cluster 2022-04-22T18:28:55.608801+0000 mgr.y (mgr.34107) 2190 : cluster [DBG] pgmap v3766: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 383 KiB/s rd, 25 KiB/s wr, 511 op/s 2022-04-22T18:28:57.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:56 smithi121 bash[36902]: cluster 2022-04-22T18:28:55.615545+0000 mon.a (mon.0) 5144 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2022-04-22T18:28:57.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:56 smithi121 bash[36902]: audit 2022-04-22T18:28:56.322364+0000 mon.a (mon.0) 5145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:57.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:56 smithi121 bash[36902]: cluster 2022-04-22T18:28:56.585520+0000 mon.a (mon.0) 5146 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2022-04-22T18:28:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[45495]: cluster 2022-04-22T18:28:55.608801+0000 mgr.y (mgr.34107) 2190 : cluster [DBG] pgmap v3766: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 383 KiB/s rd, 25 KiB/s wr, 511 op/s 2022-04-22T18:28:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[45495]: cluster 2022-04-22T18:28:55.615545+0000 mon.a (mon.0) 5144 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2022-04-22T18:28:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[45495]: audit 2022-04-22T18:28:56.322364+0000 mon.a (mon.0) 5145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:57.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[45495]: cluster 2022-04-22T18:28:56.585520+0000 mon.a (mon.0) 5146 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2022-04-22T18:28:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[46934]: cluster 2022-04-22T18:28:55.608801+0000 mgr.y (mgr.34107) 2190 : cluster [DBG] pgmap v3766: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 383 KiB/s rd, 25 KiB/s wr, 511 op/s 2022-04-22T18:28:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[46934]: cluster 2022-04-22T18:28:55.615545+0000 mon.a (mon.0) 5144 : cluster [DBG] osdmap e2160: 8 total, 8 up, 8 in 2022-04-22T18:28:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[46934]: audit 2022-04-22T18:28:56.322364+0000 mon.a (mon.0) 5145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:28:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:56 smithi115 bash[46934]: cluster 2022-04-22T18:28:56.585520+0000 mon.a (mon.0) 5146 : cluster [DBG] osdmap e2161: 8 total, 8 up, 8 in 2022-04-22T18:28:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:57 smithi121 bash[36902]: audit 2022-04-22T18:28:56.865106+0000 mon.a (mon.0) 5147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:57 smithi115 bash[45495]: audit 2022-04-22T18:28:56.865106+0000 mon.a (mon.0) 5147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:57 smithi115 bash[46934]: audit 2022-04-22T18:28:56.865106+0000 mon.a (mon.0) 5147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:28:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:58 smithi121 bash[36902]: cluster 2022-04-22T18:28:57.609957+0000 mgr.y (mgr.34107) 2191 : cluster [DBG] pgmap v3769: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 280 op/s 2022-04-22T18:28:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:58 smithi121 bash[36902]: cluster 2022-04-22T18:28:57.629489+0000 mon.a (mon.0) 5148 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2022-04-22T18:28:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:58 smithi115 bash[45495]: cluster 2022-04-22T18:28:57.609957+0000 mgr.y (mgr.34107) 2191 : cluster [DBG] pgmap v3769: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 280 op/s 2022-04-22T18:28:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:58 smithi115 bash[45495]: cluster 2022-04-22T18:28:57.629489+0000 mon.a (mon.0) 5148 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2022-04-22T18:28:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:58 smithi115 bash[46934]: cluster 2022-04-22T18:28:57.609957+0000 mgr.y (mgr.34107) 2191 : cluster [DBG] pgmap v3769: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 202 KiB/s rd, 17 KiB/s wr, 280 op/s 2022-04-22T18:28:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:58 smithi115 bash[46934]: cluster 2022-04-22T18:28:57.629489+0000 mon.a (mon.0) 5148 : cluster [DBG] osdmap e2162: 8 total, 8 up, 8 in 2022-04-22T18:29:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:28:59 smithi121 bash[36902]: cluster 2022-04-22T18:28:58.643135+0000 mon.a (mon.0) 5149 : cluster [DBG] osdmap e2163: 8 total, 8 up, 8 in 2022-04-22T18:29:00.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:28:59 smithi115 bash[45495]: cluster 2022-04-22T18:28:58.643135+0000 mon.a (mon.0) 5149 : cluster [DBG] osdmap e2163: 8 total, 8 up, 8 in 2022-04-22T18:29:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:28:59 smithi115 bash[46934]: cluster 2022-04-22T18:28:58.643135+0000 mon.a (mon.0) 5149 : cluster [DBG] osdmap e2163: 8 total, 8 up, 8 in 2022-04-22T18:29:00.823 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_rollback_with_resize ... ok 2022-04-22T18:29:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:00 smithi121 bash[36902]: cluster 2022-04-22T18:28:59.611012+0000 mgr.y (mgr.34107) 2192 : cluster [DBG] pgmap v3772: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 47 KiB/s rd, 11 KiB/s wr, 85 op/s 2022-04-22T18:29:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:00 smithi121 bash[36902]: cluster 2022-04-22T18:28:59.649263+0000 mon.a (mon.0) 5150 : cluster [DBG] osdmap e2164: 8 total, 8 up, 8 in 2022-04-22T18:29:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:00 smithi115 bash[45495]: cluster 2022-04-22T18:28:59.611012+0000 mgr.y (mgr.34107) 2192 : cluster [DBG] pgmap v3772: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 47 KiB/s rd, 11 KiB/s wr, 85 op/s 2022-04-22T18:29:01.383 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:00 smithi115 bash[45495]: cluster 2022-04-22T18:28:59.649263+0000 mon.a (mon.0) 5150 : cluster [DBG] osdmap e2164: 8 total, 8 up, 8 in 2022-04-22T18:29:01.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:00 smithi115 bash[46934]: cluster 2022-04-22T18:28:59.611012+0000 mgr.y (mgr.34107) 2192 : cluster [DBG] pgmap v3772: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 47 KiB/s rd, 11 KiB/s wr, 85 op/s 2022-04-22T18:29:01.386 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:00 smithi115 bash[46934]: cluster 2022-04-22T18:28:59.649263+0000 mon.a (mon.0) 5150 : cluster [DBG] osdmap e2164: 8 total, 8 up, 8 in 2022-04-22T18:29:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:01 smithi121 bash[36902]: cluster 2022-04-22T18:29:00.658226+0000 mon.a (mon.0) 5151 : cluster [DBG] osdmap e2165: 8 total, 8 up, 8 in 2022-04-22T18:29:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:01 smithi121 bash[36902]: audit 2022-04-22T18:29:01.325616+0000 mon.a (mon.0) 5152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:01 smithi121 bash[36902]: cluster 2022-04-22T18:29:01.589372+0000 mon.a (mon.0) 5153 : cluster [DBG] osdmap e2166: 8 total, 8 up, 8 in 2022-04-22T18:29:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[45495]: cluster 2022-04-22T18:29:00.658226+0000 mon.a (mon.0) 5151 : cluster [DBG] osdmap e2165: 8 total, 8 up, 8 in 2022-04-22T18:29:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[45495]: audit 2022-04-22T18:29:01.325616+0000 mon.a (mon.0) 5152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[45495]: cluster 2022-04-22T18:29:01.589372+0000 mon.a (mon.0) 5153 : cluster [DBG] osdmap e2166: 8 total, 8 up, 8 in 2022-04-22T18:29:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:01.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[46934]: cluster 2022-04-22T18:29:00.658226+0000 mon.a (mon.0) 5151 : cluster [DBG] osdmap e2165: 8 total, 8 up, 8 in 2022-04-22T18:29:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[46934]: audit 2022-04-22T18:29:01.325616+0000 mon.a (mon.0) 5152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:01 smithi115 bash[46934]: cluster 2022-04-22T18:29:01.589372+0000 mon.a (mon.0) 5153 : cluster [DBG] osdmap e2166: 8 total, 8 up, 8 in 2022-04-22T18:29:02.751 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_no_snap ... ok 2022-04-22T18:29:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:02 smithi121 bash[36902]: cluster 2022-04-22T18:29:01.611701+0000 mgr.y (mgr.34107) 2193 : cluster [DBG] pgmap v3776: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 110 KiB/s rd, 16 KiB/s wr, 157 op/s 2022-04-22T18:29:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:02 smithi121 bash[36902]: audit 2022-04-22T18:29:01.868921+0000 mon.a (mon.0) 5154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:02 smithi121 bash[36902]: cluster 2022-04-22T18:29:02.603538+0000 mon.a (mon.0) 5155 : cluster [DBG] osdmap e2167: 8 total, 8 up, 8 in 2022-04-22T18:29:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[45495]: cluster 2022-04-22T18:29:01.611701+0000 mgr.y (mgr.34107) 2193 : cluster [DBG] pgmap v3776: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 110 KiB/s rd, 16 KiB/s wr, 157 op/s 2022-04-22T18:29:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[45495]: audit 2022-04-22T18:29:01.868921+0000 mon.a (mon.0) 5154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[45495]: cluster 2022-04-22T18:29:02.603538+0000 mon.a (mon.0) 5155 : cluster [DBG] osdmap e2167: 8 total, 8 up, 8 in 2022-04-22T18:29:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[46934]: cluster 2022-04-22T18:29:01.611701+0000 mgr.y (mgr.34107) 2193 : cluster [DBG] pgmap v3776: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 110 KiB/s rd, 16 KiB/s wr, 157 op/s 2022-04-22T18:29:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[46934]: audit 2022-04-22T18:29:01.868921+0000 mon.a (mon.0) 5154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:02 smithi115 bash[46934]: cluster 2022-04-22T18:29:02.603538+0000 mon.a (mon.0) 5155 : cluster [DBG] osdmap e2167: 8 total, 8 up, 8 in 2022-04-22T18:29:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:03.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:04.824 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_snap ... ok 2022-04-22T18:29:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:04 smithi121 bash[36902]: cluster 2022-04-22T18:29:03.612770+0000 mgr.y (mgr.34107) 2194 : cluster [DBG] pgmap v3778: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 14 KiB/s wr, 143 op/s 2022-04-22T18:29:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:04 smithi121 bash[36902]: cluster 2022-04-22T18:29:03.669016+0000 mon.a (mon.0) 5156 : cluster [DBG] osdmap e2168: 8 total, 8 up, 8 in 2022-04-22T18:29:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:04 smithi115 bash[45495]: cluster 2022-04-22T18:29:03.612770+0000 mgr.y (mgr.34107) 2194 : cluster [DBG] pgmap v3778: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 14 KiB/s wr, 143 op/s 2022-04-22T18:29:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:04 smithi115 bash[45495]: cluster 2022-04-22T18:29:03.669016+0000 mon.a (mon.0) 5156 : cluster [DBG] osdmap e2168: 8 total, 8 up, 8 in 2022-04-22T18:29:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:04 smithi115 bash[46934]: cluster 2022-04-22T18:29:03.612770+0000 mgr.y (mgr.34107) 2194 : cluster [DBG] pgmap v3778: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 102 KiB/s rd, 14 KiB/s wr, 143 op/s 2022-04-22T18:29:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:04 smithi115 bash[46934]: cluster 2022-04-22T18:29:03.669016+0000 mon.a (mon.0) 5156 : cluster [DBG] osdmap e2168: 8 total, 8 up, 8 in 2022-04-22T18:29:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:05 smithi121 bash[36902]: cluster 2022-04-22T18:29:04.681026+0000 mon.a (mon.0) 5157 : cluster [DBG] osdmap e2169: 8 total, 8 up, 8 in 2022-04-22T18:29:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:05 smithi115 bash[45495]: cluster 2022-04-22T18:29:04.681026+0000 mon.a (mon.0) 5157 : cluster [DBG] osdmap e2169: 8 total, 8 up, 8 in 2022-04-22T18:29:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:05 smithi115 bash[46934]: cluster 2022-04-22T18:29:04.681026+0000 mon.a (mon.0) 5157 : cluster [DBG] osdmap e2169: 8 total, 8 up, 8 in 2022-04-22T18:29:06.741 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_snap_by_id ... ok 2022-04-22T18:29:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:06 smithi121 bash[36902]: cluster 2022-04-22T18:29:05.613931+0000 mgr.y (mgr.34107) 2195 : cluster [DBG] pgmap v3781: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 81 KiB/s rd, 10 KiB/s wr, 119 op/s 2022-04-22T18:29:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:06 smithi121 bash[36902]: cluster 2022-04-22T18:29:05.693314+0000 mon.a (mon.0) 5158 : cluster [DBG] osdmap e2170: 8 total, 8 up, 8 in 2022-04-22T18:29:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:06 smithi121 bash[36902]: audit 2022-04-22T18:29:06.328804+0000 mon.a (mon.0) 5159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:06 smithi121 bash[36902]: cluster 2022-04-22T18:29:06.590169+0000 mon.a (mon.0) 5160 : cluster [DBG] osdmap e2171: 8 total, 8 up, 8 in 2022-04-22T18:29:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[45495]: cluster 2022-04-22T18:29:05.613931+0000 mgr.y (mgr.34107) 2195 : cluster [DBG] pgmap v3781: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 81 KiB/s rd, 10 KiB/s wr, 119 op/s 2022-04-22T18:29:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[45495]: cluster 2022-04-22T18:29:05.693314+0000 mon.a (mon.0) 5158 : cluster [DBG] osdmap e2170: 8 total, 8 up, 8 in 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[45495]: audit 2022-04-22T18:29:06.328804+0000 mon.a (mon.0) 5159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[45495]: cluster 2022-04-22T18:29:06.590169+0000 mon.a (mon.0) 5160 : cluster [DBG] osdmap e2171: 8 total, 8 up, 8 in 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[46934]: cluster 2022-04-22T18:29:05.613931+0000 mgr.y (mgr.34107) 2195 : cluster [DBG] pgmap v3781: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 81 KiB/s rd, 10 KiB/s wr, 119 op/s 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[46934]: cluster 2022-04-22T18:29:05.693314+0000 mon.a (mon.0) 5158 : cluster [DBG] osdmap e2170: 8 total, 8 up, 8 in 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[46934]: audit 2022-04-22T18:29:06.328804+0000 mon.a (mon.0) 5159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:06 smithi115 bash[46934]: cluster 2022-04-22T18:29:06.590169+0000 mon.a (mon.0) 5160 : cluster [DBG] osdmap e2171: 8 total, 8 up, 8 in 2022-04-22T18:29:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:07 smithi121 bash[36902]: audit 2022-04-22T18:29:06.872405+0000 mon.a (mon.0) 5161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:07 smithi115 bash[45495]: audit 2022-04-22T18:29:06.872405+0000 mon.a (mon.0) 5161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:07 smithi115 bash[46934]: audit 2022-04-22T18:29:06.872405+0000 mon.a (mon.0) 5161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:08.867 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_snap_deleted ... ok 2022-04-22T18:29:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:08 smithi121 bash[36902]: cluster 2022-04-22T18:29:07.615080+0000 mgr.y (mgr.34107) 2196 : cluster [DBG] pgmap v3784: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 12 KiB/s wr, 202 op/s 2022-04-22T18:29:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:08 smithi121 bash[36902]: cluster 2022-04-22T18:29:07.706033+0000 mon.a (mon.0) 5162 : cluster [DBG] osdmap e2172: 8 total, 8 up, 8 in 2022-04-22T18:29:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:08 smithi115 bash[45495]: cluster 2022-04-22T18:29:07.615080+0000 mgr.y (mgr.34107) 2196 : cluster [DBG] pgmap v3784: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 12 KiB/s wr, 202 op/s 2022-04-22T18:29:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:08 smithi115 bash[45495]: cluster 2022-04-22T18:29:07.706033+0000 mon.a (mon.0) 5162 : cluster [DBG] osdmap e2172: 8 total, 8 up, 8 in 2022-04-22T18:29:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:08 smithi115 bash[46934]: cluster 2022-04-22T18:29:07.615080+0000 mgr.y (mgr.34107) 2196 : cluster [DBG] pgmap v3784: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 12 KiB/s wr, 202 op/s 2022-04-22T18:29:09.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:08 smithi115 bash[46934]: cluster 2022-04-22T18:29:07.706033+0000 mon.a (mon.0) 5162 : cluster [DBG] osdmap e2172: 8 total, 8 up, 8 in 2022-04-22T18:29:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:09 smithi121 bash[36902]: cluster 2022-04-22T18:29:08.707881+0000 mon.a (mon.0) 5163 : cluster [DBG] osdmap e2173: 8 total, 8 up, 8 in 2022-04-22T18:29:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:09 smithi121 bash[36902]: audit 2022-04-22T18:29:09.480902+0000 mon.a (mon.0) 5164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:29:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:09 smithi121 bash[36902]: audit 2022-04-22T18:29:09.483972+0000 mon.a (mon.0) 5165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:29:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:09 smithi121 bash[36902]: audit 2022-04-22T18:29:09.485874+0000 mon.a (mon.0) 5166 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:29:09.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[46934]: cluster 2022-04-22T18:29:08.707881+0000 mon.a (mon.0) 5163 : cluster [DBG] osdmap e2173: 8 total, 8 up, 8 in 2022-04-22T18:29:09.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[46934]: audit 2022-04-22T18:29:09.480902+0000 mon.a (mon.0) 5164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:29:09.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[46934]: audit 2022-04-22T18:29:09.483972+0000 mon.a (mon.0) 5165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:29:09.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[46934]: audit 2022-04-22T18:29:09.485874+0000 mon.a (mon.0) 5166 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:29:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[45495]: cluster 2022-04-22T18:29:08.707881+0000 mon.a (mon.0) 5163 : cluster [DBG] osdmap e2173: 8 total, 8 up, 8 in 2022-04-22T18:29:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[45495]: audit 2022-04-22T18:29:09.480902+0000 mon.a (mon.0) 5164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:29:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[45495]: audit 2022-04-22T18:29:09.483972+0000 mon.a (mon.0) 5165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:29:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:09 smithi115 bash[45495]: audit 2022-04-22T18:29:09.485874+0000 mon.a (mon.0) 5166 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:29:11.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:10 smithi121 bash[36902]: cluster 2022-04-22T18:29:09.615836+0000 mgr.y (mgr.34107) 2197 : cluster [DBG] pgmap v3787: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 11 KiB/s wr, 140 op/s 2022-04-22T18:29:11.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:10 smithi121 bash[36902]: cluster 2022-04-22T18:29:09.723571+0000 mon.a (mon.0) 5167 : cluster [DBG] osdmap e2174: 8 total, 8 up, 8 in 2022-04-22T18:29:11.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:10 smithi121 bash[36902]: audit 2022-04-22T18:29:09.797431+0000 mon.a (mon.0) 5168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:11.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:10 smithi121 bash[36902]: audit 2022-04-22T18:29:09.811340+0000 mon.a (mon.0) 5169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[45495]: cluster 2022-04-22T18:29:09.615836+0000 mgr.y (mgr.34107) 2197 : cluster [DBG] pgmap v3787: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 11 KiB/s wr, 140 op/s 2022-04-22T18:29:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[45495]: cluster 2022-04-22T18:29:09.723571+0000 mon.a (mon.0) 5167 : cluster [DBG] osdmap e2174: 8 total, 8 up, 8 in 2022-04-22T18:29:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[45495]: audit 2022-04-22T18:29:09.797431+0000 mon.a (mon.0) 5168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[45495]: audit 2022-04-22T18:29:09.811340+0000 mon.a (mon.0) 5169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[46934]: cluster 2022-04-22T18:29:09.615836+0000 mgr.y (mgr.34107) 2197 : cluster [DBG] pgmap v3787: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 99 KiB/s rd, 11 KiB/s wr, 140 op/s 2022-04-22T18:29:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[46934]: cluster 2022-04-22T18:29:09.723571+0000 mon.a (mon.0) 5167 : cluster [DBG] osdmap e2174: 8 total, 8 up, 8 in 2022-04-22T18:29:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[46934]: audit 2022-04-22T18:29:09.797431+0000 mon.a (mon.0) 5168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:10 smithi115 bash[46934]: audit 2022-04-22T18:29:09.811340+0000 mon.a (mon.0) 5169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:11 smithi121 bash[36902]: cluster 2022-04-22T18:29:10.733890+0000 mon.a (mon.0) 5170 : cluster [DBG] osdmap e2175: 8 total, 8 up, 8 in 2022-04-22T18:29:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:11 smithi121 bash[36902]: audit 2022-04-22T18:29:11.332001+0000 mon.a (mon.0) 5171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:12.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:11 smithi121 bash[36902]: cluster 2022-04-22T18:29:11.594069+0000 mon.a (mon.0) 5172 : cluster [DBG] osdmap e2176: 8 total, 8 up, 8 in 2022-04-22T18:29:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[45495]: cluster 2022-04-22T18:29:10.733890+0000 mon.a (mon.0) 5170 : cluster [DBG] osdmap e2175: 8 total, 8 up, 8 in 2022-04-22T18:29:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[45495]: audit 2022-04-22T18:29:11.332001+0000 mon.a (mon.0) 5171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[45495]: cluster 2022-04-22T18:29:11.594069+0000 mon.a (mon.0) 5172 : cluster [DBG] osdmap e2176: 8 total, 8 up, 8 in 2022-04-22T18:29:12.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:11.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[46934]: cluster 2022-04-22T18:29:10.733890+0000 mon.a (mon.0) 5170 : cluster [DBG] osdmap e2175: 8 total, 8 up, 8 in 2022-04-22T18:29:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[46934]: audit 2022-04-22T18:29:11.332001+0000 mon.a (mon.0) 5171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:11 smithi115 bash[46934]: cluster 2022-04-22T18:29:11.594069+0000 mon.a (mon.0) 5172 : cluster [DBG] osdmap e2176: 8 total, 8 up, 8 in 2022-04-22T18:29:12.767 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_snap_recreated ... ok 2022-04-22T18:29:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:12 smithi121 bash[36902]: cluster 2022-04-22T18:29:11.616629+0000 mgr.y (mgr.34107) 2198 : cluster [DBG] pgmap v3791: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 13 KiB/s wr, 123 op/s 2022-04-22T18:29:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:12 smithi121 bash[36902]: audit 2022-04-22T18:29:11.876273+0000 mon.a (mon.0) 5173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:12 smithi121 bash[36902]: cluster 2022-04-22T18:29:12.610163+0000 mon.a (mon.0) 5174 : cluster [DBG] osdmap e2177: 8 total, 8 up, 8 in 2022-04-22T18:29:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[45495]: cluster 2022-04-22T18:29:11.616629+0000 mgr.y (mgr.34107) 2198 : cluster [DBG] pgmap v3791: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 13 KiB/s wr, 123 op/s 2022-04-22T18:29:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[45495]: audit 2022-04-22T18:29:11.876273+0000 mon.a (mon.0) 5173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[45495]: cluster 2022-04-22T18:29:12.610163+0000 mon.a (mon.0) 5174 : cluster [DBG] osdmap e2177: 8 total, 8 up, 8 in 2022-04-22T18:29:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[46934]: cluster 2022-04-22T18:29:11.616629+0000 mgr.y (mgr.34107) 2198 : cluster [DBG] pgmap v3791: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 83 KiB/s rd, 13 KiB/s wr, 123 op/s 2022-04-22T18:29:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[46934]: audit 2022-04-22T18:29:11.876273+0000 mon.a (mon.0) 5173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:12 smithi115 bash[46934]: cluster 2022-04-22T18:29:12.610163+0000 mon.a (mon.0) 5174 : cluster [DBG] osdmap e2177: 8 total, 8 up, 8 in 2022-04-22T18:29:13.986 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:13.718Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:14.906 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_set_snap_sparse ... ok 2022-04-22T18:29:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: cluster 2022-04-22T18:29:13.617732+0000 mgr.y (mgr.34107) 2199 : cluster [DBG] pgmap v3793: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 78 KiB/s rd, 12 KiB/s wr, 113 op/s 2022-04-22T18:29:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: cluster 2022-04-22T18:29:13.745829+0000 mon.a (mon.0) 5175 : cluster [DBG] osdmap e2178: 8 total, 8 up, 8 in 2022-04-22T18:29:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: audit 2022-04-22T18:29:13.940004+0000 mon.a (mon.0) 5176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: audit 2022-04-22T18:29:14.016891+0000 mon.a (mon.0) 5177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:29:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: audit 2022-04-22T18:29:14.019387+0000 mon.a (mon.0) 5178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:29:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:14 smithi121 bash[36902]: audit 2022-04-22T18:29:14.298293+0000 mon.a (mon.0) 5179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: cluster 2022-04-22T18:29:13.617732+0000 mgr.y (mgr.34107) 2199 : cluster [DBG] pgmap v3793: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 78 KiB/s rd, 12 KiB/s wr, 113 op/s 2022-04-22T18:29:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: cluster 2022-04-22T18:29:13.745829+0000 mon.a (mon.0) 5175 : cluster [DBG] osdmap e2178: 8 total, 8 up, 8 in 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: audit 2022-04-22T18:29:13.940004+0000 mon.a (mon.0) 5176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: audit 2022-04-22T18:29:14.016891+0000 mon.a (mon.0) 5177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: audit 2022-04-22T18:29:14.019387+0000 mon.a (mon.0) 5178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[45495]: audit 2022-04-22T18:29:14.298293+0000 mon.a (mon.0) 5179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: cluster 2022-04-22T18:29:13.617732+0000 mgr.y (mgr.34107) 2199 : cluster [DBG] pgmap v3793: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 78 KiB/s rd, 12 KiB/s wr, 113 op/s 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: cluster 2022-04-22T18:29:13.745829+0000 mon.a (mon.0) 5175 : cluster [DBG] osdmap e2178: 8 total, 8 up, 8 in 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: audit 2022-04-22T18:29:13.940004+0000 mon.a (mon.0) 5176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: audit 2022-04-22T18:29:14.016891+0000 mon.a (mon.0) 5177 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:29:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: audit 2022-04-22T18:29:14.019387+0000 mon.a (mon.0) 5178 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:29:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:14 smithi115 bash[46934]: audit 2022-04-22T18:29:14.298293+0000 mon.a (mon.0) 5179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:29:16.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:15 smithi115 bash[45495]: cluster 2022-04-22T18:29:14.758638+0000 mon.a (mon.0) 5180 : cluster [DBG] osdmap e2179: 8 total, 8 up, 8 in 2022-04-22T18:29:16.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:15 smithi115 bash[46934]: cluster 2022-04-22T18:29:14.758638+0000 mon.a (mon.0) 5180 : cluster [DBG] osdmap e2179: 8 total, 8 up, 8 in 2022-04-22T18:29:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:15 smithi121 bash[36902]: cluster 2022-04-22T18:29:14.758638+0000 mon.a (mon.0) 5180 : cluster [DBG] osdmap e2179: 8 total, 8 up, 8 in 2022-04-22T18:29:17.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[45495]: cluster 2022-04-22T18:29:15.618840+0000 mgr.y (mgr.34107) 2200 : cluster [DBG] pgmap v3796: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[45495]: cluster 2022-04-22T18:29:15.768601+0000 mon.a (mon.0) 5181 : cluster [DBG] osdmap e2180: 8 total, 8 up, 8 in 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[45495]: audit 2022-04-22T18:29:16.335279+0000 mon.a (mon.0) 5182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[45495]: cluster 2022-04-22T18:29:16.594045+0000 mon.a (mon.0) 5183 : cluster [DBG] osdmap e2181: 8 total, 8 up, 8 in 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[46934]: cluster 2022-04-22T18:29:15.618840+0000 mgr.y (mgr.34107) 2200 : cluster [DBG] pgmap v3796: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[46934]: cluster 2022-04-22T18:29:15.768601+0000 mon.a (mon.0) 5181 : cluster [DBG] osdmap e2180: 8 total, 8 up, 8 in 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[46934]: audit 2022-04-22T18:29:16.335279+0000 mon.a (mon.0) 5182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:17.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:16 smithi115 bash[46934]: cluster 2022-04-22T18:29:16.594045+0000 mon.a (mon.0) 5183 : cluster [DBG] osdmap e2181: 8 total, 8 up, 8 in 2022-04-22T18:29:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:16 smithi121 bash[36902]: cluster 2022-04-22T18:29:15.618840+0000 mgr.y (mgr.34107) 2200 : cluster [DBG] pgmap v3796: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2022-04-22T18:29:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:16 smithi121 bash[36902]: cluster 2022-04-22T18:29:15.768601+0000 mon.a (mon.0) 5181 : cluster [DBG] osdmap e2180: 8 total, 8 up, 8 in 2022-04-22T18:29:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:16 smithi121 bash[36902]: audit 2022-04-22T18:29:16.335279+0000 mon.a (mon.0) 5182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:16 smithi121 bash[36902]: cluster 2022-04-22T18:29:16.594045+0000 mon.a (mon.0) 5183 : cluster [DBG] osdmap e2181: 8 total, 8 up, 8 in 2022-04-22T18:29:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:17 smithi115 bash[45495]: audit 2022-04-22T18:29:16.880153+0000 mon.a (mon.0) 5184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:17 smithi115 bash[45495]: cluster 2022-04-22T18:29:17.604380+0000 mon.a (mon.0) 5185 : cluster [DBG] osdmap e2182: 8 total, 8 up, 8 in 2022-04-22T18:29:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:17 smithi115 bash[46934]: audit 2022-04-22T18:29:16.880153+0000 mon.a (mon.0) 5184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:17 smithi115 bash[46934]: cluster 2022-04-22T18:29:17.604380+0000 mon.a (mon.0) 5185 : cluster [DBG] osdmap e2182: 8 total, 8 up, 8 in 2022-04-22T18:29:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:17 smithi121 bash[36902]: audit 2022-04-22T18:29:16.880153+0000 mon.a (mon.0) 5184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:17 smithi121 bash[36902]: cluster 2022-04-22T18:29:17.604380+0000 mon.a (mon.0) 5185 : cluster [DBG] osdmap e2182: 8 total, 8 up, 8 in 2022-04-22T18:29:18.767 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_size ... ok 2022-04-22T18:29:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:18 smithi115 bash[45495]: cluster 2022-04-22T18:29:17.619620+0000 mgr.y (mgr.34107) 2201 : cluster [DBG] pgmap v3800: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.3 KiB/s wr, 102 op/s 2022-04-22T18:29:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:18 smithi115 bash[45495]: cluster 2022-04-22T18:29:18.612410+0000 mon.a (mon.0) 5186 : cluster [DBG] osdmap e2183: 8 total, 8 up, 8 in 2022-04-22T18:29:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:18 smithi115 bash[46934]: cluster 2022-04-22T18:29:17.619620+0000 mgr.y (mgr.34107) 2201 : cluster [DBG] pgmap v3800: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.3 KiB/s wr, 102 op/s 2022-04-22T18:29:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:18 smithi115 bash[46934]: cluster 2022-04-22T18:29:18.612410+0000 mon.a (mon.0) 5186 : cluster [DBG] osdmap e2183: 8 total, 8 up, 8 in 2022-04-22T18:29:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:18 smithi121 bash[36902]: cluster 2022-04-22T18:29:17.619620+0000 mgr.y (mgr.34107) 2201 : cluster [DBG] pgmap v3800: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.3 KiB/s wr, 102 op/s 2022-04-22T18:29:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:18 smithi121 bash[36902]: cluster 2022-04-22T18:29:18.612410+0000 mon.a (mon.0) 5186 : cluster [DBG] osdmap e2183: 8 total, 8 up, 8 in 2022-04-22T18:29:20.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:19 smithi115 bash[45495]: cluster 2022-04-22T18:29:19.620565+0000 mgr.y (mgr.34107) 2202 : cluster [DBG] pgmap v3802: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 6.2 KiB/s wr, 88 op/s 2022-04-22T18:29:20.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:19 smithi115 bash[46934]: cluster 2022-04-22T18:29:19.620565+0000 mgr.y (mgr.34107) 2202 : cluster [DBG] pgmap v3802: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 6.2 KiB/s wr, 88 op/s 2022-04-22T18:29:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:19 smithi121 bash[36902]: cluster 2022-04-22T18:29:19.620565+0000 mgr.y (mgr.34107) 2202 : cluster [DBG] pgmap v3802: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 6.2 KiB/s wr, 88 op/s 2022-04-22T18:29:20.934 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_snap_exists ... ok 2022-04-22T18:29:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:20 smithi121 bash[36902]: cluster 2022-04-22T18:29:19.786880+0000 mon.a (mon.0) 5187 : cluster [DBG] osdmap e2184: 8 total, 8 up, 8 in 2022-04-22T18:29:21.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:20 smithi115 bash[45495]: cluster 2022-04-22T18:29:19.786880+0000 mon.a (mon.0) 5187 : cluster [DBG] osdmap e2184: 8 total, 8 up, 8 in 2022-04-22T18:29:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:20 smithi115 bash[46934]: cluster 2022-04-22T18:29:19.786880+0000 mon.a (mon.0) 5187 : cluster [DBG] osdmap e2184: 8 total, 8 up, 8 in 2022-04-22T18:29:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:21 smithi121 bash[36902]: cluster 2022-04-22T18:29:20.788325+0000 mon.a (mon.0) 5188 : cluster [DBG] osdmap e2185: 8 total, 8 up, 8 in 2022-04-22T18:29:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:21 smithi121 bash[36902]: audit 2022-04-22T18:29:21.338464+0000 mon.a (mon.0) 5189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:21 smithi121 bash[36902]: cluster 2022-04-22T18:29:21.594856+0000 mon.a (mon.0) 5190 : cluster [DBG] osdmap e2186: 8 total, 8 up, 8 in 2022-04-22T18:29:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:21 smithi121 bash[36902]: cluster 2022-04-22T18:29:21.621215+0000 mgr.y (mgr.34107) 2203 : cluster [DBG] pgmap v3806: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 90 op/s 2022-04-22T18:29:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:21.839Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:22.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[45495]: cluster 2022-04-22T18:29:20.788325+0000 mon.a (mon.0) 5188 : cluster [DBG] osdmap e2185: 8 total, 8 up, 8 in 2022-04-22T18:29:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[45495]: audit 2022-04-22T18:29:21.338464+0000 mon.a (mon.0) 5189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[45495]: cluster 2022-04-22T18:29:21.594856+0000 mon.a (mon.0) 5190 : cluster [DBG] osdmap e2186: 8 total, 8 up, 8 in 2022-04-22T18:29:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[45495]: cluster 2022-04-22T18:29:21.621215+0000 mgr.y (mgr.34107) 2203 : cluster [DBG] pgmap v3806: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 90 op/s 2022-04-22T18:29:22.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[46934]: cluster 2022-04-22T18:29:20.788325+0000 mon.a (mon.0) 5188 : cluster [DBG] osdmap e2185: 8 total, 8 up, 8 in 2022-04-22T18:29:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[46934]: audit 2022-04-22T18:29:21.338464+0000 mon.a (mon.0) 5189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[46934]: cluster 2022-04-22T18:29:21.594856+0000 mon.a (mon.0) 5190 : cluster [DBG] osdmap e2186: 8 total, 8 up, 8 in 2022-04-22T18:29:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:21 smithi115 bash[46934]: cluster 2022-04-22T18:29:21.621215+0000 mgr.y (mgr.34107) 2203 : cluster [DBG] pgmap v3806: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 90 op/s 2022-04-22T18:29:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:22 smithi121 bash[36902]: audit 2022-04-22T18:29:21.883785+0000 mon.a (mon.0) 5191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:22 smithi121 bash[36902]: cluster 2022-04-22T18:29:22.610783+0000 mon.a (mon.0) 5192 : cluster [DBG] osdmap e2187: 8 total, 8 up, 8 in 2022-04-22T18:29:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:22 smithi115 bash[46934]: audit 2022-04-22T18:29:21.883785+0000 mon.a (mon.0) 5191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:22 smithi115 bash[46934]: cluster 2022-04-22T18:29:22.610783+0000 mon.a (mon.0) 5192 : cluster [DBG] osdmap e2187: 8 total, 8 up, 8 in 2022-04-22T18:29:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:22 smithi115 bash[45495]: audit 2022-04-22T18:29:21.883785+0000 mon.a (mon.0) 5191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:22 smithi115 bash[45495]: cluster 2022-04-22T18:29:22.610783+0000 mon.a (mon.0) 5192 : cluster [DBG] osdmap e2187: 8 total, 8 up, 8 in 2022-04-22T18:29:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:23.718Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:24 smithi121 bash[36902]: cluster 2022-04-22T18:29:23.622341+0000 mgr.y (mgr.34107) 2204 : cluster [DBG] pgmap v3808: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 6.0 KiB/s wr, 130 op/s 2022-04-22T18:29:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:24 smithi121 bash[36902]: cluster 2022-04-22T18:29:23.627076+0000 mon.a (mon.0) 5193 : cluster [DBG] osdmap e2188: 8 total, 8 up, 8 in 2022-04-22T18:29:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:24 smithi115 bash[45495]: cluster 2022-04-22T18:29:23.622341+0000 mgr.y (mgr.34107) 2204 : cluster [DBG] pgmap v3808: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 6.0 KiB/s wr, 130 op/s 2022-04-22T18:29:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:24 smithi115 bash[45495]: cluster 2022-04-22T18:29:23.627076+0000 mon.a (mon.0) 5193 : cluster [DBG] osdmap e2188: 8 total, 8 up, 8 in 2022-04-22T18:29:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:24 smithi115 bash[46934]: cluster 2022-04-22T18:29:23.622341+0000 mgr.y (mgr.34107) 2204 : cluster [DBG] pgmap v3808: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 6.0 KiB/s wr, 130 op/s 2022-04-22T18:29:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:24 smithi115 bash[46934]: cluster 2022-04-22T18:29:23.627076+0000 mon.a (mon.0) 5193 : cluster [DBG] osdmap e2188: 8 total, 8 up, 8 in 2022-04-22T18:29:25.078 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:29:25.419 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:29:25.420 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (52m) 11s ago 66m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:29:25.420 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (66m) 76s ago 66m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:29:25.420 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (54m) 11s ago 67m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:29:25.420 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (53m) 76s ago 67m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:29:25.420 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (61m) 76s ago 71m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (62m) 11s ago 73m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (61m) 11s ago 73m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (60m) 76s ago 71m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (60m) 11s ago 72m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (53m) 11s ago 66m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (53m) 76s ago 66m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (59m) 11s ago 71m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (58m) 11s ago 70m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (58m) 11s ago 70m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:29:25.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (57m) 11s ago 69m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:29:25.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (56m) 76s ago 69m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:29:25.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (56m) 76s ago 68m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:29:25.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (55m) 76s ago 68m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:29:25.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (54m) 76s ago 67m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:29:25.422 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (53m) 76s ago 66m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:29:25.836 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:29:25.836 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:29:25.836 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:29:25.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:25.836 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:29:25.837 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:29:25.838 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:29:25.838 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:29:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:25 smithi121 bash[36902]: cluster 2022-04-22T18:29:24.637931+0000 mon.a (mon.0) 5194 : cluster [DBG] osdmap e2189: 8 total, 8 up, 8 in 2022-04-22T18:29:26.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:25 smithi115 bash[45495]: cluster 2022-04-22T18:29:24.637931+0000 mon.a (mon.0) 5194 : cluster [DBG] osdmap e2189: 8 total, 8 up, 8 in 2022-04-22T18:29:26.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:25 smithi115 bash[46934]: cluster 2022-04-22T18:29:24.637931+0000 mon.a (mon.0) 5194 : cluster [DBG] osdmap e2189: 8 total, 8 up, 8 in 2022-04-22T18:29:26.757 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_snap_get_id ... ok 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: audit 2022-04-22T18:29:25.054142+0000 mgr.y (mgr.34107) 2205 : audit [DBG] from='client.47602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: audit 2022-04-22T18:29:25.394078+0000 mgr.y (mgr.34107) 2206 : audit [DBG] from='client.47608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: cluster 2022-04-22T18:29:25.623542+0000 mgr.y (mgr.34107) 2207 : cluster [DBG] pgmap v3811: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 92 KiB/s rd, 6.0 KiB/s wr, 122 op/s 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: cluster 2022-04-22T18:29:25.638674+0000 mon.a (mon.0) 5195 : cluster [DBG] osdmap e2190: 8 total, 8 up, 8 in 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: audit 2022-04-22T18:29:25.828913+0000 mon.a (mon.0) 5196 : audit [DBG] from='client.? 172.21.15.115:0/1943161821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: audit 2022-04-22T18:29:26.341691+0000 mon.a (mon.0) 5197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:27.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:26 smithi121 bash[36902]: cluster 2022-04-22T18:29:26.598938+0000 mon.a (mon.0) 5198 : cluster [DBG] osdmap e2191: 8 total, 8 up, 8 in 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: audit 2022-04-22T18:29:25.054142+0000 mgr.y (mgr.34107) 2205 : audit [DBG] from='client.47602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: audit 2022-04-22T18:29:25.394078+0000 mgr.y (mgr.34107) 2206 : audit [DBG] from='client.47608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: cluster 2022-04-22T18:29:25.623542+0000 mgr.y (mgr.34107) 2207 : cluster [DBG] pgmap v3811: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 92 KiB/s rd, 6.0 KiB/s wr, 122 op/s 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: cluster 2022-04-22T18:29:25.638674+0000 mon.a (mon.0) 5195 : cluster [DBG] osdmap e2190: 8 total, 8 up, 8 in 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: audit 2022-04-22T18:29:25.828913+0000 mon.a (mon.0) 5196 : audit [DBG] from='client.? 172.21.15.115:0/1943161821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: audit 2022-04-22T18:29:26.341691+0000 mon.a (mon.0) 5197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[45495]: cluster 2022-04-22T18:29:26.598938+0000 mon.a (mon.0) 5198 : cluster [DBG] osdmap e2191: 8 total, 8 up, 8 in 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: audit 2022-04-22T18:29:25.054142+0000 mgr.y (mgr.34107) 2205 : audit [DBG] from='client.47602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: audit 2022-04-22T18:29:25.394078+0000 mgr.y (mgr.34107) 2206 : audit [DBG] from='client.47608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: cluster 2022-04-22T18:29:25.623542+0000 mgr.y (mgr.34107) 2207 : cluster [DBG] pgmap v3811: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 92 KiB/s rd, 6.0 KiB/s wr, 122 op/s 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: cluster 2022-04-22T18:29:25.638674+0000 mon.a (mon.0) 5195 : cluster [DBG] osdmap e2190: 8 total, 8 up, 8 in 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: audit 2022-04-22T18:29:25.828913+0000 mon.a (mon.0) 5196 : audit [DBG] from='client.? 172.21.15.115:0/1943161821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: audit 2022-04-22T18:29:26.341691+0000 mon.a (mon.0) 5197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:26 smithi115 bash[46934]: cluster 2022-04-22T18:29:26.598938+0000 mon.a (mon.0) 5198 : cluster [DBG] osdmap e2191: 8 total, 8 up, 8 in 2022-04-22T18:29:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:27 smithi121 bash[36902]: audit 2022-04-22T18:29:26.886952+0000 mon.a (mon.0) 5199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:28.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:27 smithi115 bash[45495]: audit 2022-04-22T18:29:26.886952+0000 mon.a (mon.0) 5199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:28.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:27 smithi115 bash[46934]: audit 2022-04-22T18:29:26.886952+0000 mon.a (mon.0) 5199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:28 smithi121 bash[36902]: cluster 2022-04-22T18:29:27.624605+0000 mgr.y (mgr.34107) 2208 : cluster [DBG] pgmap v3814: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 21 op/s 2022-04-22T18:29:29.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:28 smithi121 bash[36902]: cluster 2022-04-22T18:29:27.654434+0000 mon.a (mon.0) 5200 : cluster [DBG] osdmap e2192: 8 total, 8 up, 8 in 2022-04-22T18:29:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:28 smithi115 bash[45495]: cluster 2022-04-22T18:29:27.624605+0000 mgr.y (mgr.34107) 2208 : cluster [DBG] pgmap v3814: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 21 op/s 2022-04-22T18:29:29.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:28 smithi115 bash[45495]: cluster 2022-04-22T18:29:27.654434+0000 mon.a (mon.0) 5200 : cluster [DBG] osdmap e2192: 8 total, 8 up, 8 in 2022-04-22T18:29:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:28 smithi115 bash[46934]: cluster 2022-04-22T18:29:27.624605+0000 mgr.y (mgr.34107) 2208 : cluster [DBG] pgmap v3814: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 21 op/s 2022-04-22T18:29:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:28 smithi115 bash[46934]: cluster 2022-04-22T18:29:27.654434+0000 mon.a (mon.0) 5200 : cluster [DBG] osdmap e2192: 8 total, 8 up, 8 in 2022-04-22T18:29:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:29 smithi121 bash[36902]: cluster 2022-04-22T18:29:28.661784+0000 mon.a (mon.0) 5201 : cluster [DBG] osdmap e2193: 8 total, 8 up, 8 in 2022-04-22T18:29:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:29 smithi115 bash[45495]: cluster 2022-04-22T18:29:28.661784+0000 mon.a (mon.0) 5201 : cluster [DBG] osdmap e2193: 8 total, 8 up, 8 in 2022-04-22T18:29:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:29 smithi115 bash[46934]: cluster 2022-04-22T18:29:28.661784+0000 mon.a (mon.0) 5201 : cluster [DBG] osdmap e2193: 8 total, 8 up, 8 in 2022-04-22T18:29:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:30 smithi121 bash[36902]: cluster 2022-04-22T18:29:29.625613+0000 mgr.y (mgr.34107) 2209 : cluster [DBG] pgmap v3817: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.5 KiB/s wr, 87 op/s 2022-04-22T18:29:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:30 smithi121 bash[36902]: cluster 2022-04-22T18:29:29.676992+0000 mon.a (mon.0) 5202 : cluster [DBG] osdmap e2194: 8 total, 8 up, 8 in 2022-04-22T18:29:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:30 smithi115 bash[45495]: cluster 2022-04-22T18:29:29.625613+0000 mgr.y (mgr.34107) 2209 : cluster [DBG] pgmap v3817: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.5 KiB/s wr, 87 op/s 2022-04-22T18:29:31.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:30 smithi115 bash[45495]: cluster 2022-04-22T18:29:29.676992+0000 mon.a (mon.0) 5202 : cluster [DBG] osdmap e2194: 8 total, 8 up, 8 in 2022-04-22T18:29:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:30 smithi115 bash[46934]: cluster 2022-04-22T18:29:29.625613+0000 mgr.y (mgr.34107) 2209 : cluster [DBG] pgmap v3817: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 5.5 KiB/s wr, 87 op/s 2022-04-22T18:29:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:30 smithi115 bash[46934]: cluster 2022-04-22T18:29:29.676992+0000 mon.a (mon.0) 5202 : cluster [DBG] osdmap e2194: 8 total, 8 up, 8 in 2022-04-22T18:29:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:31 smithi121 bash[36902]: cluster 2022-04-22T18:29:30.683194+0000 mon.a (mon.0) 5203 : cluster [DBG] osdmap e2195: 8 total, 8 up, 8 in 2022-04-22T18:29:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:31 smithi121 bash[36902]: audit 2022-04-22T18:29:31.344729+0000 mon.a (mon.0) 5204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:32.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:31 smithi121 bash[36902]: cluster 2022-04-22T18:29:31.603812+0000 mon.a (mon.0) 5205 : cluster [DBG] osdmap e2196: 8 total, 8 up, 8 in 2022-04-22T18:29:32.084 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:31.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:32.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[45495]: cluster 2022-04-22T18:29:30.683194+0000 mon.a (mon.0) 5203 : cluster [DBG] osdmap e2195: 8 total, 8 up, 8 in 2022-04-22T18:29:32.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[45495]: audit 2022-04-22T18:29:31.344729+0000 mon.a (mon.0) 5204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:32.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[45495]: cluster 2022-04-22T18:29:31.603812+0000 mon.a (mon.0) 5205 : cluster [DBG] osdmap e2196: 8 total, 8 up, 8 in 2022-04-22T18:29:32.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[46934]: cluster 2022-04-22T18:29:30.683194+0000 mon.a (mon.0) 5203 : cluster [DBG] osdmap e2195: 8 total, 8 up, 8 in 2022-04-22T18:29:32.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[46934]: audit 2022-04-22T18:29:31.344729+0000 mon.a (mon.0) 5204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:32.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:31 smithi115 bash[46934]: cluster 2022-04-22T18:29:31.603812+0000 mon.a (mon.0) 5205 : cluster [DBG] osdmap e2196: 8 total, 8 up, 8 in 2022-04-22T18:29:32.782 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_snap_get_name ... ok 2022-04-22T18:29:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:32 smithi121 bash[36902]: cluster 2022-04-22T18:29:31.626353+0000 mgr.y (mgr.34107) 2210 : cluster [DBG] pgmap v3821: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.3 KiB/s wr, 80 op/s 2022-04-22T18:29:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:32 smithi121 bash[36902]: audit 2022-04-22T18:29:31.890795+0000 mon.a (mon.0) 5206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:32 smithi121 bash[36902]: cluster 2022-04-22T18:29:32.621503+0000 mon.a (mon.0) 5207 : cluster [DBG] osdmap e2197: 8 total, 8 up, 8 in 2022-04-22T18:29:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[45495]: cluster 2022-04-22T18:29:31.626353+0000 mgr.y (mgr.34107) 2210 : cluster [DBG] pgmap v3821: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.3 KiB/s wr, 80 op/s 2022-04-22T18:29:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[45495]: audit 2022-04-22T18:29:31.890795+0000 mon.a (mon.0) 5206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[45495]: cluster 2022-04-22T18:29:32.621503+0000 mon.a (mon.0) 5207 : cluster [DBG] osdmap e2197: 8 total, 8 up, 8 in 2022-04-22T18:29:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[46934]: cluster 2022-04-22T18:29:31.626353+0000 mgr.y (mgr.34107) 2210 : cluster [DBG] pgmap v3821: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.3 KiB/s wr, 80 op/s 2022-04-22T18:29:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[46934]: audit 2022-04-22T18:29:31.890795+0000 mon.a (mon.0) 5206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:32 smithi115 bash[46934]: cluster 2022-04-22T18:29:32.621503+0000 mon.a (mon.0) 5207 : cluster [DBG] osdmap e2197: 8 total, 8 up, 8 in 2022-04-22T18:29:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:33.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:34.854 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_snap_timestamp ... ok 2022-04-22T18:29:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:34 smithi121 bash[36902]: cluster 2022-04-22T18:29:33.627437+0000 mgr.y (mgr.34107) 2211 : cluster [DBG] pgmap v3823: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 KiB/s rd, 6.2 KiB/s wr, 65 op/s 2022-04-22T18:29:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:34 smithi121 bash[36902]: cluster 2022-04-22T18:29:33.692599+0000 mon.a (mon.0) 5208 : cluster [DBG] osdmap e2198: 8 total, 8 up, 8 in 2022-04-22T18:29:35.037 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_sparsify ... ok 2022-04-22T18:29:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:34 smithi115 bash[45495]: cluster 2022-04-22T18:29:33.627437+0000 mgr.y (mgr.34107) 2211 : cluster [DBG] pgmap v3823: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 KiB/s rd, 6.2 KiB/s wr, 65 op/s 2022-04-22T18:29:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:34 smithi115 bash[45495]: cluster 2022-04-22T18:29:33.692599+0000 mon.a (mon.0) 5208 : cluster [DBG] osdmap e2198: 8 total, 8 up, 8 in 2022-04-22T18:29:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:34 smithi115 bash[46934]: cluster 2022-04-22T18:29:33.627437+0000 mgr.y (mgr.34107) 2211 : cluster [DBG] pgmap v3823: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 KiB/s rd, 6.2 KiB/s wr, 65 op/s 2022-04-22T18:29:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:34 smithi115 bash[46934]: cluster 2022-04-22T18:29:33.692599+0000 mon.a (mon.0) 5208 : cluster [DBG] osdmap e2198: 8 total, 8 up, 8 in 2022-04-22T18:29:35.199 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_stat ... ok 2022-04-22T18:29:35.350 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_update_features ... SKIP 2022-04-22T18:29:35.502 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_use_after_close ... ok 2022-04-22T18:29:35.657 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_watchers_list ... ok 2022-04-22T18:29:35.837 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_write ... ok 2022-04-22T18:29:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:35 smithi121 bash[36902]: cluster 2022-04-22T18:29:34.705228+0000 mon.a (mon.0) 5209 : cluster [DBG] osdmap e2199: 8 total, 8 up, 8 in 2022-04-22T18:29:36.028 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_write_read ... ok 2022-04-22T18:29:36.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:35 smithi115 bash[45495]: cluster 2022-04-22T18:29:34.705228+0000 mon.a (mon.0) 5209 : cluster [DBG] osdmap e2199: 8 total, 8 up, 8 in 2022-04-22T18:29:36.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:35 smithi115 bash[46934]: cluster 2022-04-22T18:29:34.705228+0000 mon.a (mon.0) 5209 : cluster [DBG] osdmap e2199: 8 total, 8 up, 8 in 2022-04-22T18:29:36.215 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_write_with_fadvise_flags ... ok 2022-04-22T18:29:36.404 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_write_zeroes ... ok 2022-04-22T18:29:36.592 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImage.test_write_zeroes_thick_provision ... ok 2022-04-22T18:29:36.776 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImageId.test_read ... ok 2022-04-22T18:29:36.981 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImageId.test_resize ... ok 2022-04-22T18:29:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:36 smithi121 bash[36902]: cluster 2022-04-22T18:29:35.628340+0000 mgr.y (mgr.34107) 2212 : cluster [DBG] pgmap v3826: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2022-04-22T18:29:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:36 smithi121 bash[36902]: audit 2022-04-22T18:29:36.347945+0000 mon.a (mon.0) 5210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:36 smithi121 bash[36902]: cluster 2022-04-22T18:29:36.603189+0000 mon.a (mon.0) 5211 : cluster [DBG] osdmap e2200: 8 total, 8 up, 8 in 2022-04-22T18:29:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[45495]: cluster 2022-04-22T18:29:35.628340+0000 mgr.y (mgr.34107) 2212 : cluster [DBG] pgmap v3826: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2022-04-22T18:29:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[45495]: audit 2022-04-22T18:29:36.347945+0000 mon.a (mon.0) 5210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[45495]: cluster 2022-04-22T18:29:36.603189+0000 mon.a (mon.0) 5211 : cluster [DBG] osdmap e2200: 8 total, 8 up, 8 in 2022-04-22T18:29:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[46934]: cluster 2022-04-22T18:29:35.628340+0000 mgr.y (mgr.34107) 2212 : cluster [DBG] pgmap v3826: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2022-04-22T18:29:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[46934]: audit 2022-04-22T18:29:36.347945+0000 mon.a (mon.0) 5210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:36 smithi115 bash[46934]: cluster 2022-04-22T18:29:36.603189+0000 mon.a (mon.0) 5211 : cluster [DBG] osdmap e2200: 8 total, 8 up, 8 in 2022-04-22T18:29:37.177 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestImageId.test_write ... ok 2022-04-22T18:29:37.563 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMigration.test_migrate_abort ... ok 2022-04-22T18:29:37.962 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMigration.test_migrate_abort_with_progress ... ok 2022-04-22T18:29:38.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:37 smithi121 bash[36902]: audit 2022-04-22T18:29:36.894360+0000 mon.a (mon.0) 5212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:38.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:37 smithi115 bash[45495]: audit 2022-04-22T18:29:36.894360+0000 mon.a (mon.0) 5212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:38.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:37 smithi115 bash[46934]: audit 2022-04-22T18:29:36.894360+0000 mon.a (mon.0) 5212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:38.533 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMigration.test_migration ... ok 2022-04-22T18:29:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:38 smithi121 bash[36902]: cluster 2022-04-22T18:29:37.629278+0000 mgr.y (mgr.34107) 2213 : cluster [DBG] pgmap v3828: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 8.0 KiB/s wr, 224 op/s 2022-04-22T18:29:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:38 smithi115 bash[45495]: cluster 2022-04-22T18:29:37.629278+0000 mgr.y (mgr.34107) 2213 : cluster [DBG] pgmap v3828: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 8.0 KiB/s wr, 224 op/s 2022-04-22T18:29:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:38 smithi115 bash[46934]: cluster 2022-04-22T18:29:37.629278+0000 mgr.y (mgr.34107) 2213 : cluster [DBG] pgmap v3828: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 166 KiB/s rd, 8.0 KiB/s wr, 224 op/s 2022-04-22T18:29:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:39 smithi121 bash[36902]: cluster 2022-04-22T18:29:38.724561+0000 mon.a (mon.0) 5213 : cluster [DBG] osdmap e2201: 8 total, 8 up, 8 in 2022-04-22T18:29:40.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:39 smithi115 bash[45495]: cluster 2022-04-22T18:29:38.724561+0000 mon.a (mon.0) 5213 : cluster [DBG] osdmap e2201: 8 total, 8 up, 8 in 2022-04-22T18:29:40.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:39 smithi115 bash[46934]: cluster 2022-04-22T18:29:38.724561+0000 mon.a (mon.0) 5213 : cluster [DBG] osdmap e2201: 8 total, 8 up, 8 in 2022-04-22T18:29:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:40 smithi121 bash[36902]: cluster 2022-04-22T18:29:39.630258+0000 mgr.y (mgr.34107) 2214 : cluster [DBG] pgmap v3830: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 305 KiB/s rd, 26 KiB/s wr, 426 op/s 2022-04-22T18:29:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:40 smithi121 bash[36902]: cluster 2022-04-22T18:29:39.740121+0000 mon.a (mon.0) 5214 : cluster [DBG] osdmap e2202: 8 total, 8 up, 8 in 2022-04-22T18:29:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:40 smithi115 bash[45495]: cluster 2022-04-22T18:29:39.630258+0000 mgr.y (mgr.34107) 2214 : cluster [DBG] pgmap v3830: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 305 KiB/s rd, 26 KiB/s wr, 426 op/s 2022-04-22T18:29:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:40 smithi115 bash[45495]: cluster 2022-04-22T18:29:39.740121+0000 mon.a (mon.0) 5214 : cluster [DBG] osdmap e2202: 8 total, 8 up, 8 in 2022-04-22T18:29:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:40 smithi115 bash[46934]: cluster 2022-04-22T18:29:39.630258+0000 mgr.y (mgr.34107) 2214 : cluster [DBG] pgmap v3830: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 305 KiB/s rd, 26 KiB/s wr, 426 op/s 2022-04-22T18:29:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:40 smithi115 bash[46934]: cluster 2022-04-22T18:29:39.740121+0000 mon.a (mon.0) 5214 : cluster [DBG] osdmap e2202: 8 total, 8 up, 8 in 2022-04-22T18:29:41.901 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMigration.test_migration_import ... ok 2022-04-22T18:29:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:41 smithi121 bash[36902]: cluster 2022-04-22T18:29:40.753130+0000 mon.a (mon.0) 5215 : cluster [DBG] osdmap e2203: 8 total, 8 up, 8 in 2022-04-22T18:29:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:41 smithi121 bash[36902]: audit 2022-04-22T18:29:41.350977+0000 mon.a (mon.0) 5216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:41 smithi121 bash[36902]: cluster 2022-04-22T18:29:41.605783+0000 mon.a (mon.0) 5217 : cluster [DBG] osdmap e2204: 8 total, 8 up, 8 in 2022-04-22T18:29:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:41.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:42.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[45495]: cluster 2022-04-22T18:29:40.753130+0000 mon.a (mon.0) 5215 : cluster [DBG] osdmap e2203: 8 total, 8 up, 8 in 2022-04-22T18:29:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[45495]: audit 2022-04-22T18:29:41.350977+0000 mon.a (mon.0) 5216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[45495]: cluster 2022-04-22T18:29:41.605783+0000 mon.a (mon.0) 5217 : cluster [DBG] osdmap e2204: 8 total, 8 up, 8 in 2022-04-22T18:29:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[46934]: cluster 2022-04-22T18:29:40.753130+0000 mon.a (mon.0) 5215 : cluster [DBG] osdmap e2203: 8 total, 8 up, 8 in 2022-04-22T18:29:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[46934]: audit 2022-04-22T18:29:41.350977+0000 mon.a (mon.0) 5216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:41 smithi115 bash[46934]: cluster 2022-04-22T18:29:41.605783+0000 mon.a (mon.0) 5217 : cluster [DBG] osdmap e2204: 8 total, 8 up, 8 in 2022-04-22T18:29:42.367 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMigration.test_migration_with_progress ... ok 2022-04-22T18:29:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:42 smithi121 bash[36902]: cluster 2022-04-22T18:29:41.630952+0000 mgr.y (mgr.34107) 2215 : cluster [DBG] pgmap v3834: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 566 KiB/s rd, 45 KiB/s wr, 786 op/s 2022-04-22T18:29:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:42 smithi121 bash[36902]: audit 2022-04-22T18:29:41.897255+0000 mon.a (mon.0) 5218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:42 smithi115 bash[45495]: cluster 2022-04-22T18:29:41.630952+0000 mgr.y (mgr.34107) 2215 : cluster [DBG] pgmap v3834: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 566 KiB/s rd, 45 KiB/s wr, 786 op/s 2022-04-22T18:29:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:42 smithi115 bash[45495]: audit 2022-04-22T18:29:41.897255+0000 mon.a (mon.0) 5218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:42 smithi115 bash[46934]: cluster 2022-04-22T18:29:41.630952+0000 mgr.y (mgr.34107) 2215 : cluster [DBG] pgmap v3834: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 566 KiB/s rd, 45 KiB/s wr, 786 op/s 2022-04-22T18:29:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:42 smithi115 bash[46934]: audit 2022-04-22T18:29:41.897255+0000 mon.a (mon.0) 5218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:44.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:43 smithi115 bash[45495]: cluster 2022-04-22T18:29:42.764249+0000 mon.a (mon.0) 5219 : cluster [DBG] osdmap e2205: 8 total, 8 up, 8 in 2022-04-22T18:29:44.051 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:43.720Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:44.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:43 smithi115 bash[46934]: cluster 2022-04-22T18:29:42.764249+0000 mon.a (mon.0) 5219 : cluster [DBG] osdmap e2205: 8 total, 8 up, 8 in 2022-04-22T18:29:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:43 smithi121 bash[36902]: cluster 2022-04-22T18:29:42.764249+0000 mon.a (mon.0) 5219 : cluster [DBG] osdmap e2205: 8 total, 8 up, 8 in 2022-04-22T18:29:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[45495]: cluster 2022-04-22T18:29:43.632197+0000 mgr.y (mgr.34107) 2216 : cluster [DBG] pgmap v3836: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 572 KiB/s rd, 30 KiB/s wr, 755 op/s 2022-04-22T18:29:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[45495]: cluster 2022-04-22T18:29:43.775803+0000 mon.a (mon.0) 5220 : cluster [DBG] osdmap e2206: 8 total, 8 up, 8 in 2022-04-22T18:29:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[45495]: audit 2022-04-22T18:29:43.805346+0000 mon.c (mon.1) 132 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[45495]: audit 2022-04-22T18:29:43.805905+0000 mon.a (mon.0) 5221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[45495]: audit 2022-04-22T18:29:43.812517+0000 mon.a (mon.0) 5222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]': finished 2022-04-22T18:29:45.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[46934]: cluster 2022-04-22T18:29:43.632197+0000 mgr.y (mgr.34107) 2216 : cluster [DBG] pgmap v3836: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 572 KiB/s rd, 30 KiB/s wr, 755 op/s 2022-04-22T18:29:45.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[46934]: cluster 2022-04-22T18:29:43.775803+0000 mon.a (mon.0) 5220 : cluster [DBG] osdmap e2206: 8 total, 8 up, 8 in 2022-04-22T18:29:45.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[46934]: audit 2022-04-22T18:29:43.805346+0000 mon.c (mon.1) 132 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[46934]: audit 2022-04-22T18:29:43.805905+0000 mon.a (mon.0) 5221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:44 smithi115 bash[46934]: audit 2022-04-22T18:29:43.812517+0000 mon.a (mon.0) 5222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]': finished 2022-04-22T18:29:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:44 smithi121 bash[36902]: cluster 2022-04-22T18:29:43.632197+0000 mgr.y (mgr.34107) 2216 : cluster [DBG] pgmap v3836: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 572 KiB/s rd, 30 KiB/s wr, 755 op/s 2022-04-22T18:29:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:44 smithi121 bash[36902]: cluster 2022-04-22T18:29:43.775803+0000 mon.a (mon.0) 5220 : cluster [DBG] osdmap e2206: 8 total, 8 up, 8 in 2022-04-22T18:29:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:44 smithi121 bash[36902]: audit 2022-04-22T18:29:43.805346+0000 mon.c (mon.1) 132 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:44 smithi121 bash[36902]: audit 2022-04-22T18:29:43.805905+0000 mon.a (mon.0) 5221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]: dispatch 2022-04-22T18:29:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:44 smithi121 bash[36902]: audit 2022-04-22T18:29:43.812517+0000 mon.a (mon.0) 5222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/8d669866-14a2-41b2-81f9-ff802a68f0b5"}]': finished 2022-04-22T18:29:45.930 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_aio_mirror_image_create_snapshot ... ok 2022-04-22T18:29:46.096 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_mirror_image ... SKIP 2022-04-22T18:29:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:45 smithi121 bash[36902]: cluster 2022-04-22T18:29:44.817236+0000 mon.a (mon.0) 5223 : cluster [DBG] osdmap e2207: 8 total, 8 up, 8 in 2022-04-22T18:29:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:45 smithi121 bash[36902]: cluster 2022-04-22T18:29:45.633308+0000 mgr.y (mgr.34107) 2217 : cluster [DBG] pgmap v3839: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 19 KiB/s wr, 469 op/s 2022-04-22T18:29:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:45 smithi115 bash[45495]: cluster 2022-04-22T18:29:44.817236+0000 mon.a (mon.0) 5223 : cluster [DBG] osdmap e2207: 8 total, 8 up, 8 in 2022-04-22T18:29:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:45 smithi115 bash[45495]: cluster 2022-04-22T18:29:45.633308+0000 mgr.y (mgr.34107) 2217 : cluster [DBG] pgmap v3839: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 19 KiB/s wr, 469 op/s 2022-04-22T18:29:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:45 smithi115 bash[46934]: cluster 2022-04-22T18:29:44.817236+0000 mon.a (mon.0) 5223 : cluster [DBG] osdmap e2207: 8 total, 8 up, 8 in 2022-04-22T18:29:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:45 smithi115 bash[46934]: cluster 2022-04-22T18:29:45.633308+0000 mgr.y (mgr.34107) 2217 : cluster [DBG] pgmap v3839: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 361 KiB/s rd, 19 KiB/s wr, 469 op/s 2022-04-22T18:29:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:46 smithi121 bash[36902]: cluster 2022-04-22T18:29:45.826854+0000 mon.a (mon.0) 5224 : cluster [DBG] osdmap e2208: 8 total, 8 up, 8 in 2022-04-22T18:29:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:46 smithi121 bash[36902]: audit 2022-04-22T18:29:46.354143+0000 mon.a (mon.0) 5225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:46 smithi121 bash[36902]: cluster 2022-04-22T18:29:46.607226+0000 mon.a (mon.0) 5226 : cluster [DBG] osdmap e2209: 8 total, 8 up, 8 in 2022-04-22T18:29:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[45495]: cluster 2022-04-22T18:29:45.826854+0000 mon.a (mon.0) 5224 : cluster [DBG] osdmap e2208: 8 total, 8 up, 8 in 2022-04-22T18:29:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[45495]: audit 2022-04-22T18:29:46.354143+0000 mon.a (mon.0) 5225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[45495]: cluster 2022-04-22T18:29:46.607226+0000 mon.a (mon.0) 5226 : cluster [DBG] osdmap e2209: 8 total, 8 up, 8 in 2022-04-22T18:29:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[46934]: cluster 2022-04-22T18:29:45.826854+0000 mon.a (mon.0) 5224 : cluster [DBG] osdmap e2208: 8 total, 8 up, 8 in 2022-04-22T18:29:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[46934]: audit 2022-04-22T18:29:46.354143+0000 mon.a (mon.0) 5225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:46 smithi115 bash[46934]: cluster 2022-04-22T18:29:46.607226+0000 mon.a (mon.0) 5226 : cluster [DBG] osdmap e2209: 8 total, 8 up, 8 in 2022-04-22T18:29:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:47 smithi121 bash[36902]: audit 2022-04-22T18:29:46.900537+0000 mon.a (mon.0) 5227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:47 smithi121 bash[36902]: cluster 2022-04-22T18:29:47.621355+0000 mon.a (mon.0) 5228 : cluster [DBG] osdmap e2210: 8 total, 8 up, 8 in 2022-04-22T18:29:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:47 smithi121 bash[36902]: cluster 2022-04-22T18:29:47.633904+0000 mgr.y (mgr.34107) 2218 : cluster [DBG] pgmap v3843: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 228 KiB/s rd, 13 KiB/s wr, 295 op/s 2022-04-22T18:29:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[45495]: audit 2022-04-22T18:29:46.900537+0000 mon.a (mon.0) 5227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[45495]: cluster 2022-04-22T18:29:47.621355+0000 mon.a (mon.0) 5228 : cluster [DBG] osdmap e2210: 8 total, 8 up, 8 in 2022-04-22T18:29:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[45495]: cluster 2022-04-22T18:29:47.633904+0000 mgr.y (mgr.34107) 2218 : cluster [DBG] pgmap v3843: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 228 KiB/s rd, 13 KiB/s wr, 295 op/s 2022-04-22T18:29:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[46934]: audit 2022-04-22T18:29:46.900537+0000 mon.a (mon.0) 5227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[46934]: cluster 2022-04-22T18:29:47.621355+0000 mon.a (mon.0) 5228 : cluster [DBG] osdmap e2210: 8 total, 8 up, 8 in 2022-04-22T18:29:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:47 smithi115 bash[46934]: cluster 2022-04-22T18:29:47.633904+0000 mgr.y (mgr.34107) 2218 : cluster [DBG] pgmap v3843: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 228 KiB/s rd, 13 KiB/s wr, 295 op/s 2022-04-22T18:29:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: cluster 2022-04-22T18:29:48.628690+0000 mon.a (mon.0) 5229 : cluster [DBG] osdmap e2211: 8 total, 8 up, 8 in 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.665448+0000 mon.c (mon.1) 133 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.666043+0000 mon.a (mon.0) 5230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.670572+0000 mon.a (mon.0) 5231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]': finished 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.733252+0000 mon.c (mon.1) 134 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.733914+0000 mon.a (mon.0) 5232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:49 smithi121 bash[36902]: audit 2022-04-22T18:29:48.739487+0000 mon.a (mon.0) 5233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]': finished 2022-04-22T18:29:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: cluster 2022-04-22T18:29:48.628690+0000 mon.a (mon.0) 5229 : cluster [DBG] osdmap e2211: 8 total, 8 up, 8 in 2022-04-22T18:29:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.665448+0000 mon.c (mon.1) 133 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.666043+0000 mon.a (mon.0) 5230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.670572+0000 mon.a (mon.0) 5231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]': finished 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.733252+0000 mon.c (mon.1) 134 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.733914+0000 mon.a (mon.0) 5232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[46934]: audit 2022-04-22T18:29:48.739487+0000 mon.a (mon.0) 5233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]': finished 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: cluster 2022-04-22T18:29:48.628690+0000 mon.a (mon.0) 5229 : cluster [DBG] osdmap e2211: 8 total, 8 up, 8 in 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.665448+0000 mon.c (mon.1) 133 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.666043+0000 mon.a (mon.0) 5230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]: dispatch 2022-04-22T18:29:50.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.670572+0000 mon.a (mon.0) 5231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/17bf5bd6-5378-4477-83f8-ce31ecda13ec"}]': finished 2022-04-22T18:29:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.733252+0000 mon.c (mon.1) 134 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.733914+0000 mon.a (mon.0) 5232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]: dispatch 2022-04-22T18:29:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:49 smithi115 bash[45495]: audit 2022-04-22T18:29:48.739487+0000 mon.a (mon.0) 5233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/cb451773-70e5-47dc-93ce-539e97071d80"}]': finished 2022-04-22T18:29:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:50 smithi121 bash[36902]: cluster 2022-04-22T18:29:49.634511+0000 mgr.y (mgr.34107) 2219 : cluster [DBG] pgmap v3845: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 171 KiB/s rd, 13 KiB/s wr, 222 op/s 2022-04-22T18:29:51.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:50 smithi121 bash[36902]: cluster 2022-04-22T18:29:49.743464+0000 mon.a (mon.0) 5234 : cluster [DBG] osdmap e2212: 8 total, 8 up, 8 in 2022-04-22T18:29:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:50 smithi115 bash[45495]: cluster 2022-04-22T18:29:49.634511+0000 mgr.y (mgr.34107) 2219 : cluster [DBG] pgmap v3845: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 171 KiB/s rd, 13 KiB/s wr, 222 op/s 2022-04-22T18:29:51.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:50 smithi115 bash[45495]: cluster 2022-04-22T18:29:49.743464+0000 mon.a (mon.0) 5234 : cluster [DBG] osdmap e2212: 8 total, 8 up, 8 in 2022-04-22T18:29:51.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:50 smithi115 bash[46934]: cluster 2022-04-22T18:29:49.634511+0000 mgr.y (mgr.34107) 2219 : cluster [DBG] pgmap v3845: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 171 KiB/s rd, 13 KiB/s wr, 222 op/s 2022-04-22T18:29:51.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:50 smithi115 bash[46934]: cluster 2022-04-22T18:29:49.743464+0000 mon.a (mon.0) 5234 : cluster [DBG] osdmap e2212: 8 total, 8 up, 8 in 2022-04-22T18:29:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:51 smithi121 bash[36902]: cluster 2022-04-22T18:29:50.748242+0000 mon.a (mon.0) 5235 : cluster [DBG] osdmap e2213: 8 total, 8 up, 8 in 2022-04-22T18:29:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:51 smithi121 bash[36902]: audit 2022-04-22T18:29:51.357339+0000 mon.a (mon.0) 5236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:51 smithi121 bash[36902]: cluster 2022-04-22T18:29:51.611756+0000 mon.a (mon.0) 5237 : cluster [DBG] osdmap e2214: 8 total, 8 up, 8 in 2022-04-22T18:29:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:51.842Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[45495]: cluster 2022-04-22T18:29:50.748242+0000 mon.a (mon.0) 5235 : cluster [DBG] osdmap e2213: 8 total, 8 up, 8 in 2022-04-22T18:29:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[45495]: audit 2022-04-22T18:29:51.357339+0000 mon.a (mon.0) 5236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[45495]: cluster 2022-04-22T18:29:51.611756+0000 mon.a (mon.0) 5237 : cluster [DBG] osdmap e2214: 8 total, 8 up, 8 in 2022-04-22T18:29:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[46934]: cluster 2022-04-22T18:29:50.748242+0000 mon.a (mon.0) 5235 : cluster [DBG] osdmap e2213: 8 total, 8 up, 8 in 2022-04-22T18:29:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[46934]: audit 2022-04-22T18:29:51.357339+0000 mon.a (mon.0) 5236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:51 smithi115 bash[46934]: cluster 2022-04-22T18:29:51.611756+0000 mon.a (mon.0) 5237 : cluster [DBG] osdmap e2214: 8 total, 8 up, 8 in 2022-04-22T18:29:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:52 smithi121 bash[36902]: cluster 2022-04-22T18:29:51.635090+0000 mgr.y (mgr.34107) 2220 : cluster [DBG] pgmap v3849: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 162 KiB/s rd, 11 KiB/s wr, 216 op/s 2022-04-22T18:29:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:52 smithi121 bash[36902]: audit 2022-04-22T18:29:51.904245+0000 mon.a (mon.0) 5238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:52 smithi121 bash[36902]: cluster 2022-04-22T18:29:52.629059+0000 mon.a (mon.0) 5239 : cluster [DBG] osdmap e2215: 8 total, 8 up, 8 in 2022-04-22T18:29:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[45495]: cluster 2022-04-22T18:29:51.635090+0000 mgr.y (mgr.34107) 2220 : cluster [DBG] pgmap v3849: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 162 KiB/s rd, 11 KiB/s wr, 216 op/s 2022-04-22T18:29:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[45495]: audit 2022-04-22T18:29:51.904245+0000 mon.a (mon.0) 5238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[45495]: cluster 2022-04-22T18:29:52.629059+0000 mon.a (mon.0) 5239 : cluster [DBG] osdmap e2215: 8 total, 8 up, 8 in 2022-04-22T18:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[46934]: cluster 2022-04-22T18:29:51.635090+0000 mgr.y (mgr.34107) 2220 : cluster [DBG] pgmap v3849: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 162 KiB/s rd, 11 KiB/s wr, 216 op/s 2022-04-22T18:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[46934]: audit 2022-04-22T18:29:51.904245+0000 mon.a (mon.0) 5238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:52 smithi115 bash[46934]: cluster 2022-04-22T18:29:52.629059+0000 mon.a (mon.0) 5239 : cluster [DBG] osdmap e2215: 8 total, 8 up, 8 in 2022-04-22T18:29:53.730 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_mirror_image_create_snapshot ... ok 2022-04-22T18:29:53.921 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_mirror_image_status ... SKIP 2022-04-22T18:29:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:29:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:29:53.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:29:54.138 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_mirror_peer ... ok 2022-04-22T18:29:54.350 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_mirror_peer_bootstrap ... ok 2022-04-22T18:29:54.541 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_site_name ... ok 2022-04-22T18:29:54.724 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestMirroring.test_uuid ... ok 2022-04-22T18:29:54.873 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_get ... ok 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: cluster 2022-04-22T18:29:53.635710+0000 mgr.y (mgr.34107) 2221 : cluster [DBG] pgmap v3852: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 116 KiB/s rd, 8.0 KiB/s wr, 158 op/s 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: cluster 2022-04-22T18:29:53.636640+0000 mon.a (mon.0) 5240 : cluster [DBG] osdmap e2216: 8 total, 8 up, 8 in 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:53.980601+0000 mon.a (mon.0) 5241 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:53.981383+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:53.982018+0000 mon.c (mon.1) 136 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:53.982541+0000 mon.a (mon.0) 5242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:53.986273+0000 mon.a (mon.0) 5243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]': finished 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.196911+0000 mon.c (mon.1) 137 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.197484+0000 mon.c (mon.1) 138 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.197915+0000 mon.a (mon.0) 5244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.198893+0000 mon.c (mon.1) 139 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.393782+0000 mon.a (mon.0) 5245 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.396395+0000 mon.c (mon.1) 140 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.396914+0000 mon.c (mon.1) 141 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.397348+0000 mon.a (mon.0) 5246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.401273+0000 mon.a (mon.0) 5247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:29:55.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:54 smithi121 bash[36902]: audit 2022-04-22T18:29:54.401995+0000 mon.c (mon.1) 142 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: cluster 2022-04-22T18:29:53.635710+0000 mgr.y (mgr.34107) 2221 : cluster [DBG] pgmap v3852: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 116 KiB/s rd, 8.0 KiB/s wr, 158 op/s 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: cluster 2022-04-22T18:29:53.636640+0000 mon.a (mon.0) 5240 : cluster [DBG] osdmap e2216: 8 total, 8 up, 8 in 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:53.980601+0000 mon.a (mon.0) 5241 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:53.981383+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:53.982018+0000 mon.c (mon.1) 136 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:53.982541+0000 mon.a (mon.0) 5242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:53.986273+0000 mon.a (mon.0) 5243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]': finished 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.196911+0000 mon.c (mon.1) 137 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.197484+0000 mon.c (mon.1) 138 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.197915+0000 mon.a (mon.0) 5244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.198893+0000 mon.c (mon.1) 139 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.393782+0000 mon.a (mon.0) 5245 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.396395+0000 mon.c (mon.1) 140 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.396914+0000 mon.c (mon.1) 141 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.397348+0000 mon.a (mon.0) 5246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.401273+0000 mon.a (mon.0) 5247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[45495]: audit 2022-04-22T18:29:54.401995+0000 mon.c (mon.1) 142 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: cluster 2022-04-22T18:29:53.635710+0000 mgr.y (mgr.34107) 2221 : cluster [DBG] pgmap v3852: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 116 KiB/s rd, 8.0 KiB/s wr, 158 op/s 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: cluster 2022-04-22T18:29:53.636640+0000 mon.a (mon.0) 5240 : cluster [DBG] osdmap e2216: 8 total, 8 up, 8 in 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:53.980601+0000 mon.a (mon.0) 5241 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:53.981383+0000 mon.c (mon.1) 135 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:53.982018+0000 mon.c (mon.1) 136 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:53.982541+0000 mon.a (mon.0) 5242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]: dispatch 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:53.986273+0000 mon.a (mon.0) 5243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/87/af37394a-2f9a-4174-9ee4-4663d72ed92b"}]': finished 2022-04-22T18:29:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.196911+0000 mon.c (mon.1) 137 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.197484+0000 mon.c (mon.1) 138 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.197915+0000 mon.a (mon.0) 5244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.198893+0000 mon.c (mon.1) 139 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.393782+0000 mon.a (mon.0) 5245 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.396395+0000 mon.c (mon.1) 140 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.396914+0000 mon.c (mon.1) 141 : audit [INF] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.397348+0000 mon.a (mon.0) 5246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.401273+0000 mon.a (mon.0) 5247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2022-04-22T18:29:55.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:54 smithi115 bash[46934]: audit 2022-04-22T18:29:54.401995+0000 mon.c (mon.1) 142 : audit [DBG] from='client.? 172.21.15.115:0/1084105178' entity='client.admin' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: dispatch 2022-04-22T18:29:55.142 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_list ... ok 2022-04-22T18:29:55.287 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_move ... ok 2022-04-22T18:29:55.573 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_purge ... ok 2022-04-22T18:29:55.716 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_remove ... ok 2022-04-22T18:29:55.875 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_remove_denied ... ok 2022-04-22T18:29:56.020 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_remove_with_progress ... ok 2022-04-22T18:29:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:29:56.246 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.TestTrash.test_restore ... ok 2022-04-22T18:29:56.247 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_features_from_string ... ok 2022-04-22T18:29:56.247 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_features_to_string ... ok 2022-04-22T18:29:56.440 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_open_by_id ... ok 2022-04-22T18:29:56.593 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:29:56.593 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (53m) 42s ago 66m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:29:56.593 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (66m) 107s ago 66m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (54m) 42s ago 67m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (54m) 107s ago 67m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (62m) 107s ago 72m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (62m) 42s ago 74m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (61m) 42s ago 74m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (60m) 107s ago 72m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (61m) 42s ago 72m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (54m) 42s ago 67m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (54m) 107s ago 67m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:29:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (59m) 42s ago 71m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (59m) 42s ago 71m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (58m) 42s ago 70m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (58m) 42s ago 70m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (57m) 107s ago 69m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (56m) 107s ago 69m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (56m) 107s ago 68m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (55m) 107s ago 68m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:29:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (53m) 107s ago 67m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:29:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:56 smithi121 bash[36902]: cluster 2022-04-22T18:29:55.636945+0000 mgr.y (mgr.34107) 2222 : cluster [DBG] pgmap v3853: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 6.3 KiB/s wr, 94 op/s 2022-04-22T18:29:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:56 smithi121 bash[36902]: audit 2022-04-22T18:29:56.360037+0000 mon.a (mon.0) 5248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:56 smithi121 bash[36902]: cluster 2022-04-22T18:29:56.612279+0000 mon.a (mon.0) 5249 : cluster [DBG] osdmap e2217: 8 total, 8 up, 8 in 2022-04-22T18:29:57.024 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:29:57.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:29:57.047 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:29:57.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[46934]: cluster 2022-04-22T18:29:55.636945+0000 mgr.y (mgr.34107) 2222 : cluster [DBG] pgmap v3853: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 6.3 KiB/s wr, 94 op/s 2022-04-22T18:29:57.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[46934]: audit 2022-04-22T18:29:56.360037+0000 mon.a (mon.0) 5248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:57.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[46934]: cluster 2022-04-22T18:29:56.612279+0000 mon.a (mon.0) 5249 : cluster [DBG] osdmap e2217: 8 total, 8 up, 8 in 2022-04-22T18:29:57.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[45495]: cluster 2022-04-22T18:29:55.636945+0000 mgr.y (mgr.34107) 2222 : cluster [DBG] pgmap v3853: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 60 KiB/s rd, 6.3 KiB/s wr, 94 op/s 2022-04-22T18:29:57.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[45495]: audit 2022-04-22T18:29:56.360037+0000 mon.a (mon.0) 5248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:29:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:56 smithi115 bash[45495]: cluster 2022-04-22T18:29:56.612279+0000 mon.a (mon.0) 5249 : cluster [DBG] osdmap e2217: 8 total, 8 up, 8 in 2022-04-22T18:29:57.869 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_pool_stats ... ok 2022-04-22T18:29:57.869 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_version ... ok 2022-04-22T18:29:58.015 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_create ... ok 2022-04-22T18:29:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:57 smithi121 bash[36902]: audit 2022-04-22T18:29:56.210012+0000 mgr.y (mgr.34107) 2223 : audit [DBG] from='client.47668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:57 smithi121 bash[36902]: audit 2022-04-22T18:29:56.567124+0000 mgr.y (mgr.34107) 2224 : audit [DBG] from='client.47686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:57 smithi121 bash[36902]: audit 2022-04-22T18:29:56.907502+0000 mon.a (mon.0) 5250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:58.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:57 smithi121 bash[36902]: audit 2022-04-22T18:29:57.017279+0000 mon.c (mon.1) 143 : audit [DBG] from='client.? 172.21.15.115:0/239637446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[45495]: audit 2022-04-22T18:29:56.210012+0000 mgr.y (mgr.34107) 2223 : audit [DBG] from='client.47668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[45495]: audit 2022-04-22T18:29:56.567124+0000 mgr.y (mgr.34107) 2224 : audit [DBG] from='client.47686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[45495]: audit 2022-04-22T18:29:56.907502+0000 mon.a (mon.0) 5250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[45495]: audit 2022-04-22T18:29:57.017279+0000 mon.c (mon.1) 143 : audit [DBG] from='client.? 172.21.15.115:0/239637446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[46934]: audit 2022-04-22T18:29:56.210012+0000 mgr.y (mgr.34107) 2223 : audit [DBG] from='client.47668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[46934]: audit 2022-04-22T18:29:56.567124+0000 mgr.y (mgr.34107) 2224 : audit [DBG] from='client.47686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[46934]: audit 2022-04-22T18:29:56.907502+0000 mon.a (mon.0) 5250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:57 smithi115 bash[46934]: audit 2022-04-22T18:29:57.017279+0000 mon.c (mon.1) 143 : audit [DBG] from='client.? 172.21.15.115:0/239637446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:29:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:58 smithi121 bash[36902]: cluster 2022-04-22T18:29:57.638086+0000 mgr.y (mgr.34107) 2225 : cluster [DBG] pgmap v3855: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 178 KiB/s rd, 15 KiB/s wr, 264 op/s 2022-04-22T18:29:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:29:58 smithi121 bash[36902]: cluster 2022-04-22T18:29:57.704959+0000 mon.a (mon.0) 5251 : cluster [DBG] osdmap e2218: 8 total, 8 up, 8 in 2022-04-22T18:29:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:58 smithi115 bash[45495]: cluster 2022-04-22T18:29:57.638086+0000 mgr.y (mgr.34107) 2225 : cluster [DBG] pgmap v3855: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 178 KiB/s rd, 15 KiB/s wr, 264 op/s 2022-04-22T18:29:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:29:58 smithi115 bash[45495]: cluster 2022-04-22T18:29:57.704959+0000 mon.a (mon.0) 5251 : cluster [DBG] osdmap e2218: 8 total, 8 up, 8 in 2022-04-22T18:29:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:58 smithi115 bash[46934]: cluster 2022-04-22T18:29:57.638086+0000 mgr.y (mgr.34107) 2225 : cluster [DBG] pgmap v3855: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 178 KiB/s rd, 15 KiB/s wr, 264 op/s 2022-04-22T18:29:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:29:58 smithi115 bash[46934]: cluster 2022-04-22T18:29:57.704959+0000 mon.a (mon.0) 5251 : cluster [DBG] osdmap e2218: 8 total, 8 up, 8 in 2022-04-22T18:29:59.413 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_create_defaults ... ok 2022-04-22T18:29:59.645 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_context_manager ... ok 2022-04-22T18:30:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:00 smithi121 bash[36902]: cluster 2022-04-22T18:29:59.638921+0000 mgr.y (mgr.34107) 2226 : cluster [DBG] pgmap v3857: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 35 KiB/s wr, 549 op/s 2022-04-22T18:30:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:00 smithi121 bash[36902]: cluster 2022-04-22T18:30:00.000171+0000 mon.a (mon.0) 5252 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:30:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:00 smithi115 bash[45495]: cluster 2022-04-22T18:29:59.638921+0000 mgr.y (mgr.34107) 2226 : cluster [DBG] pgmap v3857: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 35 KiB/s wr, 549 op/s 2022-04-22T18:30:01.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:00 smithi115 bash[45495]: cluster 2022-04-22T18:30:00.000171+0000 mon.a (mon.0) 5252 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:30:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:00 smithi115 bash[46934]: cluster 2022-04-22T18:29:59.638921+0000 mgr.y (mgr.34107) 2226 : cluster [DBG] pgmap v3857: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 35 KiB/s wr, 549 op/s 2022-04-22T18:30:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:00 smithi115 bash[46934]: cluster 2022-04-22T18:30:00.000171+0000 mon.a (mon.0) 5252 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:30:01.769 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_open_read_only ... ok 2022-04-22T18:30:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:01 smithi121 bash[36902]: cluster 2022-04-22T18:30:00.717727+0000 mon.a (mon.0) 5253 : cluster [DBG] osdmap e2219: 8 total, 8 up, 8 in 2022-04-22T18:30:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:01 smithi121 bash[36902]: audit 2022-04-22T18:30:01.362517+0000 mon.a (mon.0) 5254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:01 smithi121 bash[36902]: cluster 2022-04-22T18:30:01.614644+0000 mon.a (mon.0) 5255 : cluster [DBG] osdmap e2220: 8 total, 8 up, 8 in 2022-04-22T18:30:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:01.841Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[45495]: cluster 2022-04-22T18:30:00.717727+0000 mon.a (mon.0) 5253 : cluster [DBG] osdmap e2219: 8 total, 8 up, 8 in 2022-04-22T18:30:02.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[45495]: audit 2022-04-22T18:30:01.362517+0000 mon.a (mon.0) 5254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[45495]: cluster 2022-04-22T18:30:01.614644+0000 mon.a (mon.0) 5255 : cluster [DBG] osdmap e2220: 8 total, 8 up, 8 in 2022-04-22T18:30:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[46934]: cluster 2022-04-22T18:30:00.717727+0000 mon.a (mon.0) 5253 : cluster [DBG] osdmap e2219: 8 total, 8 up, 8 in 2022-04-22T18:30:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[46934]: audit 2022-04-22T18:30:01.362517+0000 mon.a (mon.0) 5254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:01 smithi115 bash[46934]: cluster 2022-04-22T18:30:01.614644+0000 mon.a (mon.0) 5255 : cluster [DBG] osdmap e2220: 8 total, 8 up, 8 in 2022-04-22T18:30:02.631 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_open_dne ... ok 2022-04-22T18:30:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:02 smithi121 bash[36902]: cluster 2022-04-22T18:30:01.639577+0000 mgr.y (mgr.34107) 2227 : cluster [DBG] pgmap v3860: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 42 KiB/s wr, 745 op/s 2022-04-22T18:30:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:02 smithi121 bash[36902]: audit 2022-04-22T18:30:01.910710+0000 mon.a (mon.0) 5256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:02 smithi115 bash[45495]: cluster 2022-04-22T18:30:01.639577+0000 mgr.y (mgr.34107) 2227 : cluster [DBG] pgmap v3860: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 42 KiB/s wr, 745 op/s 2022-04-22T18:30:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:02 smithi115 bash[45495]: audit 2022-04-22T18:30:01.910710+0000 mon.a (mon.0) 5256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:02 smithi115 bash[46934]: cluster 2022-04-22T18:30:01.639577+0000 mgr.y (mgr.34107) 2227 : cluster [DBG] pgmap v3860: 137 pgs: 137 active+clean; 4.6 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 531 KiB/s rd, 42 KiB/s wr, 745 op/s 2022-04-22T18:30:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:02 smithi115 bash[46934]: audit 2022-04-22T18:30:01.910710+0000 mon.a (mon.0) 5256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:03.505 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_open_readonly_dne ... ok 2022-04-22T18:30:03.700 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_aio_open ... ok 2022-04-22T18:30:03.724 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_remove_dne ... ok 2022-04-22T18:30:03.728 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_list_empty ... ok 2022-04-22T18:30:03.898 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_list ... ok 2022-04-22T18:30:04.094 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_remove_with_progress ... ok 2022-04-22T18:30:04.094 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:03.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:04.169 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_remove_canceled ... ok 2022-04-22T18:30:04.660 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_pool_metadata ... ok 2022-04-22T18:30:04.685 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_config_list ... ok 2022-04-22T18:30:04.700 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_pool_config_set_and_get_and_remove ... ok 2022-04-22T18:30:04.756 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_namespaces ... ok 2022-04-22T18:30:04.770 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_create_group ... ok 2022-04-22T18:30:04.787 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_rename_group ... ok 2022-04-22T18:30:04.788 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_list_groups_empty ... ok 2022-04-22T18:30:04.803 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_list_groups ... ok 2022-04-22T18:30:04.817 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_list_groups_after_removed ... ok 2022-04-22T18:30:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: cluster 2022-04-22T18:30:03.640735+0000 mgr.y (mgr.34107) 2228 : cluster [DBG] pgmap v3861: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 37 KiB/s wr, 667 op/s 2022-04-22T18:30:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: audit 2022-04-22T18:30:04.667591+0000 mon.a (mon.0) 5257 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:30:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: audit 2022-04-22T18:30:04.697798+0000 mon.a (mon.0) 5258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:30:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: audit 2022-04-22T18:30:04.700807+0000 mon.a (mon.0) 5259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:30:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: audit 2022-04-22T18:30:04.702537+0000 mon.a (mon.0) 5260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:30:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:04 smithi121 bash[36902]: audit 2022-04-22T18:30:04.712393+0000 mon.a (mon.0) 5261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:30:05.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: cluster 2022-04-22T18:30:03.640735+0000 mgr.y (mgr.34107) 2228 : cluster [DBG] pgmap v3861: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 37 KiB/s wr, 667 op/s 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: audit 2022-04-22T18:30:04.667591+0000 mon.a (mon.0) 5257 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: audit 2022-04-22T18:30:04.697798+0000 mon.a (mon.0) 5258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: audit 2022-04-22T18:30:04.700807+0000 mon.a (mon.0) 5259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: audit 2022-04-22T18:30:04.702537+0000 mon.a (mon.0) 5260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[45495]: audit 2022-04-22T18:30:04.712393+0000 mon.a (mon.0) 5261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:30:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: cluster 2022-04-22T18:30:03.640735+0000 mgr.y (mgr.34107) 2228 : cluster [DBG] pgmap v3861: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 483 KiB/s rd, 37 KiB/s wr, 667 op/s 2022-04-22T18:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: audit 2022-04-22T18:30:04.667591+0000 mon.a (mon.0) 5257 : audit [INF] from='client.? ' entity='client.admin' 2022-04-22T18:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: audit 2022-04-22T18:30:04.697798+0000 mon.a (mon.0) 5258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: audit 2022-04-22T18:30:04.700807+0000 mon.a (mon.0) 5259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: audit 2022-04-22T18:30:04.702537+0000 mon.a (mon.0) 5260 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:04 smithi115 bash[46934]: audit 2022-04-22T18:30:04.712393+0000 mon.a (mon.0) 5261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:30:05.751 INFO:tasks.workunit.client.0.smithi115.stderr:test_rbd.test_rename ... ok 2022-04-22T18:30:05.751 INFO:tasks.workunit.client.0.smithi115.stderr: 2022-04-22T18:30:05.751 INFO:tasks.workunit.client.0.smithi115.stderr:---------------------------------------------------------------------- 2022-04-22T18:30:05.751 INFO:tasks.workunit.client.0.smithi115.stderr:Ran 171 tests in 641.263s 2022-04-22T18:30:05.751 INFO:tasks.workunit.client.0.smithi115.stderr: 2022-04-22T18:30:05.752 INFO:tasks.workunit.client.0.smithi115.stderr:OK (SKIP=8) 2022-04-22T18:30:05.790 INFO:tasks.workunit.client.0.smithi115.stderr:+ exit 0 2022-04-22T18:30:05.791 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-22T18:30:05.791 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-22T18:30:05.801 INFO:tasks.workunit:Stopping ['rbd/test_librbd_python.sh'] on client.0... 2022-04-22T18:30:05.801 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-04-22T18:30:06.013 DEBUG:teuthology.parallel:result is None 2022-04-22T18:30:06.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:30:06.026 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:30:06.026 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.0 2022-04-22T18:30:06.031 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2022-04-22T18:30:06.031 INFO:teuthology.task.full_sequential:In full_sequential, running task print... 2022-04-22T18:30:06.031 INFO:teuthology.task.print:**** done end test_rbd_python.yaml 2022-04-22T18:30:06.031 DEBUG:teuthology.parallel:result is None 2022-04-22T18:30:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:06 smithi121 bash[36902]: cluster 2022-04-22T18:30:05.641950+0000 mgr.y (mgr.34107) 2229 : cluster [DBG] pgmap v3862: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 399 KiB/s rd, 31 KiB/s wr, 553 op/s 2022-04-22T18:30:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:06 smithi121 bash[36902]: cluster 2022-04-22T18:30:05.736677+0000 mon.a (mon.0) 5262 : cluster [DBG] osdmap e2221: 8 total, 8 up, 8 in 2022-04-22T18:30:07.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:06 smithi121 bash[36902]: audit 2022-04-22T18:30:06.365839+0000 mon.a (mon.0) 5263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:07.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[45495]: cluster 2022-04-22T18:30:05.641950+0000 mgr.y (mgr.34107) 2229 : cluster [DBG] pgmap v3862: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 399 KiB/s rd, 31 KiB/s wr, 553 op/s 2022-04-22T18:30:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[45495]: cluster 2022-04-22T18:30:05.736677+0000 mon.a (mon.0) 5262 : cluster [DBG] osdmap e2221: 8 total, 8 up, 8 in 2022-04-22T18:30:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[45495]: audit 2022-04-22T18:30:06.365839+0000 mon.a (mon.0) 5263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[46934]: cluster 2022-04-22T18:30:05.641950+0000 mgr.y (mgr.34107) 2229 : cluster [DBG] pgmap v3862: 137 pgs: 137 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 399 KiB/s rd, 31 KiB/s wr, 553 op/s 2022-04-22T18:30:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[46934]: cluster 2022-04-22T18:30:05.736677+0000 mon.a (mon.0) 5262 : cluster [DBG] osdmap e2221: 8 total, 8 up, 8 in 2022-04-22T18:30:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:06 smithi115 bash[46934]: audit 2022-04-22T18:30:06.365839+0000 mon.a (mon.0) 5263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:07 smithi121 bash[36902]: audit 2022-04-22T18:30:06.914266+0000 mon.a (mon.0) 5264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:08.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:07 smithi115 bash[45495]: audit 2022-04-22T18:30:06.914266+0000 mon.a (mon.0) 5264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:08.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:07 smithi115 bash[46934]: audit 2022-04-22T18:30:06.914266+0000 mon.a (mon.0) 5264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:08 smithi121 bash[36902]: cluster 2022-04-22T18:30:07.643047+0000 mgr.y (mgr.34107) 2230 : cluster [DBG] pgmap v3864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:08 smithi115 bash[45495]: cluster 2022-04-22T18:30:07.643047+0000 mgr.y (mgr.34107) 2230 : cluster [DBG] pgmap v3864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:08 smithi115 bash[46934]: cluster 2022-04-22T18:30:07.643047+0000 mgr.y (mgr.34107) 2230 : cluster [DBG] pgmap v3864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:10 smithi121 bash[36902]: cluster 2022-04-22T18:30:09.644179+0000 mgr.y (mgr.34107) 2231 : cluster [DBG] pgmap v3865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:10 smithi115 bash[45495]: cluster 2022-04-22T18:30:09.644179+0000 mgr.y (mgr.34107) 2231 : cluster [DBG] pgmap v3865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:10 smithi115 bash[46934]: cluster 2022-04-22T18:30:09.644179+0000 mgr.y (mgr.34107) 2231 : cluster [DBG] pgmap v3865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:11 smithi121 bash[36902]: audit 2022-04-22T18:30:11.369071+0000 mon.a (mon.0) 5265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:11 smithi115 bash[45495]: audit 2022-04-22T18:30:11.369071+0000 mon.a (mon.0) 5265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:12.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:11.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:11 smithi115 bash[46934]: audit 2022-04-22T18:30:11.369071+0000 mon.a (mon.0) 5265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:12 smithi121 bash[36902]: cluster 2022-04-22T18:30:11.645178+0000 mgr.y (mgr.34107) 2232 : cluster [DBG] pgmap v3866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:12 smithi121 bash[36902]: audit 2022-04-22T18:30:11.918346+0000 mon.a (mon.0) 5266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:12 smithi115 bash[45495]: cluster 2022-04-22T18:30:11.645178+0000 mgr.y (mgr.34107) 2232 : cluster [DBG] pgmap v3866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:12 smithi115 bash[45495]: audit 2022-04-22T18:30:11.918346+0000 mon.a (mon.0) 5266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:12 smithi115 bash[46934]: cluster 2022-04-22T18:30:11.645178+0000 mgr.y (mgr.34107) 2232 : cluster [DBG] pgmap v3866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:12 smithi115 bash[46934]: audit 2022-04-22T18:30:11.918346+0000 mon.a (mon.0) 5266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:13.721Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:14 smithi121 bash[36902]: cluster 2022-04-22T18:30:13.646352+0000 mgr.y (mgr.34107) 2233 : cluster [DBG] pgmap v3867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:14 smithi121 bash[36902]: audit 2022-04-22T18:30:14.027330+0000 mon.a (mon.0) 5267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:30:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:14 smithi121 bash[36902]: audit 2022-04-22T18:30:14.030734+0000 mon.a (mon.0) 5268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:30:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[45495]: cluster 2022-04-22T18:30:13.646352+0000 mgr.y (mgr.34107) 2233 : cluster [DBG] pgmap v3867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[45495]: audit 2022-04-22T18:30:14.027330+0000 mon.a (mon.0) 5267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:30:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[45495]: audit 2022-04-22T18:30:14.030734+0000 mon.a (mon.0) 5268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:30:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[46934]: cluster 2022-04-22T18:30:13.646352+0000 mgr.y (mgr.34107) 2233 : cluster [DBG] pgmap v3867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[46934]: audit 2022-04-22T18:30:14.027330+0000 mon.a (mon.0) 5267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:30:15.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:14 smithi115 bash[46934]: audit 2022-04-22T18:30:14.030734+0000 mon.a (mon.0) 5268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:30:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:16 smithi121 bash[36902]: cluster 2022-04-22T18:30:15.647365+0000 mgr.y (mgr.34107) 2234 : cluster [DBG] pgmap v3868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:16 smithi121 bash[36902]: audit 2022-04-22T18:30:16.372077+0000 mon.a (mon.0) 5269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:16 smithi115 bash[45495]: cluster 2022-04-22T18:30:15.647365+0000 mgr.y (mgr.34107) 2234 : cluster [DBG] pgmap v3868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:16 smithi115 bash[45495]: audit 2022-04-22T18:30:16.372077+0000 mon.a (mon.0) 5269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:16 smithi115 bash[46934]: cluster 2022-04-22T18:30:15.647365+0000 mgr.y (mgr.34107) 2234 : cluster [DBG] pgmap v3868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:16 smithi115 bash[46934]: audit 2022-04-22T18:30:16.372077+0000 mon.a (mon.0) 5269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:18.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:17 smithi115 bash[45495]: audit 2022-04-22T18:30:16.921945+0000 mon.a (mon.0) 5270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:18.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:17 smithi115 bash[46934]: audit 2022-04-22T18:30:16.921945+0000 mon.a (mon.0) 5270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:17 smithi121 bash[36902]: audit 2022-04-22T18:30:16.921945+0000 mon.a (mon.0) 5270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:18 smithi115 bash[45495]: cluster 2022-04-22T18:30:17.648509+0000 mgr.y (mgr.34107) 2235 : cluster [DBG] pgmap v3869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:18 smithi115 bash[46934]: cluster 2022-04-22T18:30:17.648509+0000 mgr.y (mgr.34107) 2235 : cluster [DBG] pgmap v3869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:18 smithi121 bash[36902]: cluster 2022-04-22T18:30:17.648509+0000 mgr.y (mgr.34107) 2235 : cluster [DBG] pgmap v3869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:20 smithi115 bash[45495]: cluster 2022-04-22T18:30:19.649673+0000 mgr.y (mgr.34107) 2236 : cluster [DBG] pgmap v3870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:20 smithi115 bash[46934]: cluster 2022-04-22T18:30:19.649673+0000 mgr.y (mgr.34107) 2236 : cluster [DBG] pgmap v3870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:20 smithi121 bash[36902]: cluster 2022-04-22T18:30:19.649673+0000 mgr.y (mgr.34107) 2236 : cluster [DBG] pgmap v3870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:21 smithi115 bash[45495]: audit 2022-04-22T18:30:21.375087+0000 mon.a (mon.0) 5271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:21.844Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:21 smithi115 bash[46934]: audit 2022-04-22T18:30:21.375087+0000 mon.a (mon.0) 5271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:21 smithi121 bash[36902]: audit 2022-04-22T18:30:21.375087+0000 mon.a (mon.0) 5271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:22 smithi115 bash[45495]: cluster 2022-04-22T18:30:21.650770+0000 mgr.y (mgr.34107) 2237 : cluster [DBG] pgmap v3871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:22 smithi115 bash[45495]: audit 2022-04-22T18:30:21.925411+0000 mon.a (mon.0) 5272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:22 smithi115 bash[46934]: cluster 2022-04-22T18:30:21.650770+0000 mgr.y (mgr.34107) 2237 : cluster [DBG] pgmap v3871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:22 smithi115 bash[46934]: audit 2022-04-22T18:30:21.925411+0000 mon.a (mon.0) 5272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:22 smithi121 bash[36902]: cluster 2022-04-22T18:30:21.650770+0000 mgr.y (mgr.34107) 2237 : cluster [DBG] pgmap v3871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:22 smithi121 bash[36902]: audit 2022-04-22T18:30:21.925411+0000 mon.a (mon.0) 5272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:23.723Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:24 smithi115 bash[45495]: cluster 2022-04-22T18:30:23.651973+0000 mgr.y (mgr.34107) 2238 : cluster [DBG] pgmap v3872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:25.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:24 smithi115 bash[46934]: cluster 2022-04-22T18:30:23.651973+0000 mgr.y (mgr.34107) 2238 : cluster [DBG] pgmap v3872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:24 smithi121 bash[36902]: cluster 2022-04-22T18:30:23.651973+0000 mgr.y (mgr.34107) 2238 : cluster [DBG] pgmap v3872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:25 smithi121 bash[36902]: cluster 2022-04-22T18:30:25.653053+0000 mgr.y (mgr.34107) 2239 : cluster [DBG] pgmap v3873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:25 smithi115 bash[45495]: cluster 2022-04-22T18:30:25.653053+0000 mgr.y (mgr.34107) 2239 : cluster [DBG] pgmap v3873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:25 smithi115 bash[46934]: cluster 2022-04-22T18:30:25.653053+0000 mgr.y (mgr.34107) 2239 : cluster [DBG] pgmap v3873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:26 smithi121 bash[36902]: audit 2022-04-22T18:30:26.378373+0000 mon.a (mon.0) 5273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:27.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:26 smithi115 bash[46934]: audit 2022-04-22T18:30:26.378373+0000 mon.a (mon.0) 5273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:26 smithi115 bash[45495]: audit 2022-04-22T18:30:26.378373+0000 mon.a (mon.0) 5273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:27.400 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:30:27.753 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:30:27.753 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (53m) 73s ago 67m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (67m) 2m ago 67m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (55m) 73s ago 68m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (55m) 2m ago 68m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (62m) 2m ago 72m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (63m) 73s ago 74m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (62m) 73s ago 74m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (61m) 2m ago 72m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (61m) 73s ago 73m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:30:27.754 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (54m) 73s ago 67m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (54m) 2m ago 67m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (60m) 73s ago 72m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (59m) 73s ago 71m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (59m) 73s ago 71m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (58m) 73s ago 70m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (57m) 2m ago 70m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (57m) 2m ago 69m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (56m) 2m ago 69m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (55m) 2m ago 68m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:30:27.755 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (54m) 2m ago 67m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:30:28.166 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:28.167 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:30:28.168 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:30:28.168 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:30:28.168 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:30:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:27 smithi121 bash[36902]: audit 2022-04-22T18:30:26.928748+0000 mon.a (mon.0) 5274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:27 smithi121 bash[36902]: audit 2022-04-22T18:30:27.377896+0000 mgr.y (mgr.34107) 2240 : audit [DBG] from='client.37782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:27 smithi121 bash[36902]: cluster 2022-04-22T18:30:27.654134+0000 mgr.y (mgr.34107) 2241 : cluster [DBG] pgmap v3874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:27 smithi121 bash[36902]: audit 2022-04-22T18:30:27.722835+0000 mgr.y (mgr.34107) 2242 : audit [DBG] from='client.37788 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[45495]: audit 2022-04-22T18:30:26.928748+0000 mon.a (mon.0) 5274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[45495]: audit 2022-04-22T18:30:27.377896+0000 mgr.y (mgr.34107) 2240 : audit [DBG] from='client.37782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[45495]: cluster 2022-04-22T18:30:27.654134+0000 mgr.y (mgr.34107) 2241 : cluster [DBG] pgmap v3874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[45495]: audit 2022-04-22T18:30:27.722835+0000 mgr.y (mgr.34107) 2242 : audit [DBG] from='client.37788 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[46934]: audit 2022-04-22T18:30:26.928748+0000 mon.a (mon.0) 5274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[46934]: audit 2022-04-22T18:30:27.377896+0000 mgr.y (mgr.34107) 2240 : audit [DBG] from='client.37782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[46934]: cluster 2022-04-22T18:30:27.654134+0000 mgr.y (mgr.34107) 2241 : cluster [DBG] pgmap v3874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:27 smithi115 bash[46934]: audit 2022-04-22T18:30:27.722835+0000 mgr.y (mgr.34107) 2242 : audit [DBG] from='client.37788 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:28 smithi121 bash[36902]: audit 2022-04-22T18:30:28.158483+0000 mon.a (mon.0) 5275 : audit [DBG] from='client.? 172.21.15.115:0/3636518657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:30:29.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:28 smithi115 bash[45495]: audit 2022-04-22T18:30:28.158483+0000 mon.a (mon.0) 5275 : audit [DBG] from='client.? 172.21.15.115:0/3636518657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:30:29.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:28 smithi115 bash[46934]: audit 2022-04-22T18:30:28.158483+0000 mon.a (mon.0) 5275 : audit [DBG] from='client.? 172.21.15.115:0/3636518657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:30:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:29 smithi121 bash[36902]: cluster 2022-04-22T18:30:29.655351+0000 mgr.y (mgr.34107) 2243 : cluster [DBG] pgmap v3875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:29 smithi115 bash[45495]: cluster 2022-04-22T18:30:29.655351+0000 mgr.y (mgr.34107) 2243 : cluster [DBG] pgmap v3875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:29 smithi115 bash[46934]: cluster 2022-04-22T18:30:29.655351+0000 mgr.y (mgr.34107) 2243 : cluster [DBG] pgmap v3875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:31 smithi121 bash[36902]: audit 2022-04-22T18:30:31.381565+0000 mon.a (mon.0) 5276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:31 smithi115 bash[45495]: audit 2022-04-22T18:30:31.381565+0000 mon.a (mon.0) 5276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:31 smithi115 bash[46934]: audit 2022-04-22T18:30:31.381565+0000 mon.a (mon.0) 5276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:31.844Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:32 smithi121 bash[36902]: cluster 2022-04-22T18:30:31.656398+0000 mgr.y (mgr.34107) 2244 : cluster [DBG] pgmap v3876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:32 smithi121 bash[36902]: audit 2022-04-22T18:30:31.932499+0000 mon.a (mon.0) 5277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:32 smithi115 bash[45495]: cluster 2022-04-22T18:30:31.656398+0000 mgr.y (mgr.34107) 2244 : cluster [DBG] pgmap v3876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:32 smithi115 bash[45495]: audit 2022-04-22T18:30:31.932499+0000 mon.a (mon.0) 5277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:32 smithi115 bash[46934]: cluster 2022-04-22T18:30:31.656398+0000 mgr.y (mgr.34107) 2244 : cluster [DBG] pgmap v3876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:32 smithi115 bash[46934]: audit 2022-04-22T18:30:31.932499+0000 mon.a (mon.0) 5277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:33.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:34 smithi121 bash[36902]: cluster 2022-04-22T18:30:33.657496+0000 mgr.y (mgr.34107) 2245 : cluster [DBG] pgmap v3877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:35.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:34 smithi115 bash[45495]: cluster 2022-04-22T18:30:33.657496+0000 mgr.y (mgr.34107) 2245 : cluster [DBG] pgmap v3877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:35.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:34 smithi115 bash[46934]: cluster 2022-04-22T18:30:33.657496+0000 mgr.y (mgr.34107) 2245 : cluster [DBG] pgmap v3877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:36 smithi121 bash[36902]: cluster 2022-04-22T18:30:35.658589+0000 mgr.y (mgr.34107) 2246 : cluster [DBG] pgmap v3878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:36 smithi121 bash[36902]: audit 2022-04-22T18:30:36.384450+0000 mon.a (mon.0) 5278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:36 smithi115 bash[45495]: cluster 2022-04-22T18:30:35.658589+0000 mgr.y (mgr.34107) 2246 : cluster [DBG] pgmap v3878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:36 smithi115 bash[45495]: audit 2022-04-22T18:30:36.384450+0000 mon.a (mon.0) 5278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:36 smithi115 bash[46934]: cluster 2022-04-22T18:30:35.658589+0000 mgr.y (mgr.34107) 2246 : cluster [DBG] pgmap v3878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:36 smithi115 bash[46934]: audit 2022-04-22T18:30:36.384450+0000 mon.a (mon.0) 5278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:37 smithi121 bash[36902]: audit 2022-04-22T18:30:36.936303+0000 mon.a (mon.0) 5279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:37 smithi115 bash[45495]: audit 2022-04-22T18:30:36.936303+0000 mon.a (mon.0) 5279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:37 smithi115 bash[46934]: audit 2022-04-22T18:30:36.936303+0000 mon.a (mon.0) 5279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:38 smithi121 bash[36902]: cluster 2022-04-22T18:30:37.659730+0000 mgr.y (mgr.34107) 2247 : cluster [DBG] pgmap v3879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:38 smithi115 bash[45495]: cluster 2022-04-22T18:30:37.659730+0000 mgr.y (mgr.34107) 2247 : cluster [DBG] pgmap v3879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:38 smithi115 bash[46934]: cluster 2022-04-22T18:30:37.659730+0000 mgr.y (mgr.34107) 2247 : cluster [DBG] pgmap v3879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:40 smithi121 bash[36902]: cluster 2022-04-22T18:30:39.660829+0000 mgr.y (mgr.34107) 2248 : cluster [DBG] pgmap v3880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:40 smithi115 bash[45495]: cluster 2022-04-22T18:30:39.660829+0000 mgr.y (mgr.34107) 2248 : cluster [DBG] pgmap v3880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:40 smithi115 bash[46934]: cluster 2022-04-22T18:30:39.660829+0000 mgr.y (mgr.34107) 2248 : cluster [DBG] pgmap v3880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:41 smithi121 bash[36902]: audit 2022-04-22T18:30:41.387553+0000 mon.a (mon.0) 5280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:42.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:41.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:41 smithi115 bash[45495]: audit 2022-04-22T18:30:41.387553+0000 mon.a (mon.0) 5280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:41 smithi115 bash[46934]: audit 2022-04-22T18:30:41.387553+0000 mon.a (mon.0) 5280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:42 smithi121 bash[36902]: cluster 2022-04-22T18:30:41.661952+0000 mgr.y (mgr.34107) 2249 : cluster [DBG] pgmap v3881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:43.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:42 smithi121 bash[36902]: audit 2022-04-22T18:30:41.939972+0000 mon.a (mon.0) 5281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:42 smithi115 bash[45495]: cluster 2022-04-22T18:30:41.661952+0000 mgr.y (mgr.34107) 2249 : cluster [DBG] pgmap v3881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:42 smithi115 bash[45495]: audit 2022-04-22T18:30:41.939972+0000 mon.a (mon.0) 5281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:42 smithi115 bash[46934]: cluster 2022-04-22T18:30:41.661952+0000 mgr.y (mgr.34107) 2249 : cluster [DBG] pgmap v3881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:42 smithi115 bash[46934]: audit 2022-04-22T18:30:41.939972+0000 mon.a (mon.0) 5281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:43.723Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:44 smithi121 bash[36902]: cluster 2022-04-22T18:30:43.663156+0000 mgr.y (mgr.34107) 2250 : cluster [DBG] pgmap v3882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:44 smithi115 bash[45495]: cluster 2022-04-22T18:30:43.663156+0000 mgr.y (mgr.34107) 2250 : cluster [DBG] pgmap v3882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:44 smithi115 bash[46934]: cluster 2022-04-22T18:30:43.663156+0000 mgr.y (mgr.34107) 2250 : cluster [DBG] pgmap v3882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:46 smithi121 bash[36902]: cluster 2022-04-22T18:30:45.664324+0000 mgr.y (mgr.34107) 2251 : cluster [DBG] pgmap v3883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:46 smithi121 bash[36902]: audit 2022-04-22T18:30:46.390827+0000 mon.a (mon.0) 5282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:46 smithi115 bash[45495]: cluster 2022-04-22T18:30:45.664324+0000 mgr.y (mgr.34107) 2251 : cluster [DBG] pgmap v3883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:46 smithi115 bash[45495]: audit 2022-04-22T18:30:46.390827+0000 mon.a (mon.0) 5282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:46 smithi115 bash[46934]: cluster 2022-04-22T18:30:45.664324+0000 mgr.y (mgr.34107) 2251 : cluster [DBG] pgmap v3883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:46 smithi115 bash[46934]: audit 2022-04-22T18:30:46.390827+0000 mon.a (mon.0) 5282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:47 smithi121 bash[36902]: audit 2022-04-22T18:30:46.943571+0000 mon.a (mon.0) 5283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:47 smithi115 bash[45495]: audit 2022-04-22T18:30:46.943571+0000 mon.a (mon.0) 5283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:47 smithi115 bash[46934]: audit 2022-04-22T18:30:46.943571+0000 mon.a (mon.0) 5283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:49.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:48 smithi121 bash[36902]: cluster 2022-04-22T18:30:47.665679+0000 mgr.y (mgr.34107) 2252 : cluster [DBG] pgmap v3884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:49.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:48 smithi115 bash[45495]: cluster 2022-04-22T18:30:47.665679+0000 mgr.y (mgr.34107) 2252 : cluster [DBG] pgmap v3884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:48 smithi115 bash[46934]: cluster 2022-04-22T18:30:47.665679+0000 mgr.y (mgr.34107) 2252 : cluster [DBG] pgmap v3884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:51.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:50 smithi115 bash[45495]: cluster 2022-04-22T18:30:49.666730+0000 mgr.y (mgr.34107) 2253 : cluster [DBG] pgmap v3885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:51.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:50 smithi115 bash[46934]: cluster 2022-04-22T18:30:49.666730+0000 mgr.y (mgr.34107) 2253 : cluster [DBG] pgmap v3885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:50 smithi121 bash[36902]: cluster 2022-04-22T18:30:49.666730+0000 mgr.y (mgr.34107) 2253 : cluster [DBG] pgmap v3885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:52.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:51.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:51 smithi115 bash[45495]: audit 2022-04-22T18:30:51.393956+0000 mon.a (mon.0) 5284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:51 smithi115 bash[46934]: audit 2022-04-22T18:30:51.393956+0000 mon.a (mon.0) 5284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:51 smithi121 bash[36902]: audit 2022-04-22T18:30:51.393956+0000 mon.a (mon.0) 5284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:52 smithi115 bash[45495]: cluster 2022-04-22T18:30:51.667725+0000 mgr.y (mgr.34107) 2254 : cluster [DBG] pgmap v3886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:52 smithi115 bash[45495]: audit 2022-04-22T18:30:51.947194+0000 mon.a (mon.0) 5285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:52 smithi115 bash[46934]: cluster 2022-04-22T18:30:51.667725+0000 mgr.y (mgr.34107) 2254 : cluster [DBG] pgmap v3886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:52 smithi115 bash[46934]: audit 2022-04-22T18:30:51.947194+0000 mon.a (mon.0) 5285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:52 smithi121 bash[36902]: cluster 2022-04-22T18:30:51.667725+0000 mgr.y (mgr.34107) 2254 : cluster [DBG] pgmap v3886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:52 smithi121 bash[36902]: audit 2022-04-22T18:30:51.947194+0000 mon.a (mon.0) 5285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:30:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:30:53.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:30:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:54 smithi115 bash[45495]: cluster 2022-04-22T18:30:53.668738+0000 mgr.y (mgr.34107) 2255 : cluster [DBG] pgmap v3887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:55.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:54 smithi115 bash[46934]: cluster 2022-04-22T18:30:53.668738+0000 mgr.y (mgr.34107) 2255 : cluster [DBG] pgmap v3887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:54 smithi121 bash[36902]: cluster 2022-04-22T18:30:53.668738+0000 mgr.y (mgr.34107) 2255 : cluster [DBG] pgmap v3887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:56 smithi115 bash[45495]: cluster 2022-04-22T18:30:55.669759+0000 mgr.y (mgr.34107) 2256 : cluster [DBG] pgmap v3888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:56 smithi115 bash[45495]: audit 2022-04-22T18:30:56.396954+0000 mon.a (mon.0) 5286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:56 smithi115 bash[46934]: cluster 2022-04-22T18:30:55.669759+0000 mgr.y (mgr.34107) 2256 : cluster [DBG] pgmap v3888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:56 smithi115 bash[46934]: audit 2022-04-22T18:30:56.396954+0000 mon.a (mon.0) 5286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:56 smithi121 bash[36902]: cluster 2022-04-22T18:30:55.669759+0000 mgr.y (mgr.34107) 2256 : cluster [DBG] pgmap v3888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:56 smithi121 bash[36902]: audit 2022-04-22T18:30:56.396954+0000 mon.a (mon.0) 5286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:30:58.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:57 smithi115 bash[45495]: audit 2022-04-22T18:30:56.950720+0000 mon.a (mon.0) 5287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:57 smithi115 bash[46934]: audit 2022-04-22T18:30:56.950720+0000 mon.a (mon.0) 5287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:57 smithi121 bash[36902]: audit 2022-04-22T18:30:56.950720+0000 mon.a (mon.0) 5287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:30:58.545 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:30:58.896 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (54m) 104s ago 67m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (67m) 2m ago 67m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (56m) 104s ago 68m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (55m) 2m ago 68m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (63m) 2m ago 73m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:30:58.902 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (63m) 104s ago 75m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:30:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (62m) 104s ago 75m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:30:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (62m) 2m ago 73m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:30:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (62m) 104s ago 73m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:30:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (55m) 104s ago 68m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:30:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (55m) 2m ago 68m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:30:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (60m) 104s ago 72m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:30:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (60m) 104s ago 72m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:30:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (59m) 104s ago 71m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:30:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (59m) 104s ago 71m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:30:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (58m) 2m ago 70m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:30:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (57m) 2m ago 70m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:30:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (57m) 2m ago 69m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:30:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (56m) 2m ago 69m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:30:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (54m) 2m ago 68m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:30:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:58 smithi121 bash[36902]: cluster 2022-04-22T18:30:57.670870+0000 mgr.y (mgr.34107) 2257 : cluster [DBG] pgmap v3889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:58 smithi121 bash[36902]: audit 2022-04-22T18:30:58.523653+0000 mgr.y (mgr.34107) 2258 : audit [DBG] from='client.37800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:59.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:58 smithi115 bash[45495]: cluster 2022-04-22T18:30:57.670870+0000 mgr.y (mgr.34107) 2257 : cluster [DBG] pgmap v3889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:59.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:58 smithi115 bash[45495]: audit 2022-04-22T18:30:58.523653+0000 mgr.y (mgr.34107) 2258 : audit [DBG] from='client.37800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:58 smithi115 bash[46934]: cluster 2022-04-22T18:30:57.670870+0000 mgr.y (mgr.34107) 2257 : cluster [DBG] pgmap v3889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:30:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:58 smithi115 bash[46934]: audit 2022-04-22T18:30:58.523653+0000 mgr.y (mgr.34107) 2258 : audit [DBG] from='client.37800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:30:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:30:59.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:30:59.304 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:30:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:31:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:59 smithi121 bash[36902]: audit 2022-04-22T18:30:58.867603+0000 mgr.y (mgr.34107) 2259 : audit [DBG] from='client.47740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:59 smithi121 bash[36902]: audit 2022-04-22T18:30:59.295346+0000 mon.c (mon.1) 144 : audit [DBG] from='client.? 172.21.15.115:0/1084109300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:30:59 smithi121 bash[36902]: cluster 2022-04-22T18:30:59.672025+0000 mgr.y (mgr.34107) 2260 : cluster [DBG] pgmap v3890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[45495]: audit 2022-04-22T18:30:58.867603+0000 mgr.y (mgr.34107) 2259 : audit [DBG] from='client.47740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[45495]: audit 2022-04-22T18:30:59.295346+0000 mon.c (mon.1) 144 : audit [DBG] from='client.? 172.21.15.115:0/1084109300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[45495]: cluster 2022-04-22T18:30:59.672025+0000 mgr.y (mgr.34107) 2260 : cluster [DBG] pgmap v3890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[46934]: audit 2022-04-22T18:30:58.867603+0000 mgr.y (mgr.34107) 2259 : audit [DBG] from='client.47740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[46934]: audit 2022-04-22T18:30:59.295346+0000 mon.c (mon.1) 144 : audit [DBG] from='client.? 172.21.15.115:0/1084109300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:30:59 smithi115 bash[46934]: cluster 2022-04-22T18:30:59.672025+0000 mgr.y (mgr.34107) 2260 : cluster [DBG] pgmap v3890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:01 smithi121 bash[36902]: audit 2022-04-22T18:31:01.400184+0000 mon.a (mon.0) 5288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:01 smithi115 bash[45495]: audit 2022-04-22T18:31:01.400184+0000 mon.a (mon.0) 5288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:01.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:01 smithi115 bash[46934]: audit 2022-04-22T18:31:01.400184+0000 mon.a (mon.0) 5288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:02.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:01.847Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:02 smithi121 bash[36902]: cluster 2022-04-22T18:31:01.673133+0000 mgr.y (mgr.34107) 2261 : cluster [DBG] pgmap v3891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:02 smithi121 bash[36902]: audit 2022-04-22T18:31:01.953976+0000 mon.a (mon.0) 5289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:03.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:02 smithi115 bash[45495]: cluster 2022-04-22T18:31:01.673133+0000 mgr.y (mgr.34107) 2261 : cluster [DBG] pgmap v3891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:02 smithi115 bash[45495]: audit 2022-04-22T18:31:01.953976+0000 mon.a (mon.0) 5289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:02 smithi115 bash[46934]: cluster 2022-04-22T18:31:01.673133+0000 mgr.y (mgr.34107) 2261 : cluster [DBG] pgmap v3891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:02 smithi115 bash[46934]: audit 2022-04-22T18:31:01.953976+0000 mon.a (mon.0) 5289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:03.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:04 smithi121 bash[36902]: cluster 2022-04-22T18:31:03.674370+0000 mgr.y (mgr.34107) 2262 : cluster [DBG] pgmap v3892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:04 smithi121 bash[36902]: audit 2022-04-22T18:31:04.721334+0000 mon.a (mon.0) 5290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:31:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:04 smithi121 bash[36902]: audit 2022-04-22T18:31:04.725042+0000 mon.a (mon.0) 5291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:31:05.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:04 smithi121 bash[36902]: audit 2022-04-22T18:31:04.726715+0000 mon.a (mon.0) 5292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[45495]: cluster 2022-04-22T18:31:03.674370+0000 mgr.y (mgr.34107) 2262 : cluster [DBG] pgmap v3892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[45495]: audit 2022-04-22T18:31:04.721334+0000 mon.a (mon.0) 5290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[45495]: audit 2022-04-22T18:31:04.725042+0000 mon.a (mon.0) 5291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[45495]: audit 2022-04-22T18:31:04.726715+0000 mon.a (mon.0) 5292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[46934]: cluster 2022-04-22T18:31:03.674370+0000 mgr.y (mgr.34107) 2262 : cluster [DBG] pgmap v3892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[46934]: audit 2022-04-22T18:31:04.721334+0000 mon.a (mon.0) 5290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:31:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[46934]: audit 2022-04-22T18:31:04.725042+0000 mon.a (mon.0) 5291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:31:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:04 smithi115 bash[46934]: audit 2022-04-22T18:31:04.726715+0000 mon.a (mon.0) 5292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:31:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:06 smithi121 bash[36902]: audit 2022-04-22T18:31:05.097845+0000 mon.a (mon.0) 5293 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:31:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:06 smithi121 bash[36902]: cluster 2022-04-22T18:31:05.675395+0000 mgr.y (mgr.34107) 2263 : cluster [DBG] pgmap v3893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:06 smithi115 bash[45495]: audit 2022-04-22T18:31:05.097845+0000 mon.a (mon.0) 5293 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:31:06.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:06 smithi115 bash[45495]: cluster 2022-04-22T18:31:05.675395+0000 mgr.y (mgr.34107) 2263 : cluster [DBG] pgmap v3893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:06 smithi115 bash[46934]: audit 2022-04-22T18:31:05.097845+0000 mon.a (mon.0) 5293 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:31:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:06 smithi115 bash[46934]: cluster 2022-04-22T18:31:05.675395+0000 mgr.y (mgr.34107) 2263 : cluster [DBG] pgmap v3893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:07 smithi121 bash[36902]: audit 2022-04-22T18:31:06.402914+0000 mon.a (mon.0) 5294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:07 smithi121 bash[36902]: audit 2022-04-22T18:31:06.957443+0000 mon.a (mon.0) 5295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:07 smithi115 bash[45495]: audit 2022-04-22T18:31:06.402914+0000 mon.a (mon.0) 5294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:07 smithi115 bash[45495]: audit 2022-04-22T18:31:06.957443+0000 mon.a (mon.0) 5295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:07 smithi115 bash[46934]: audit 2022-04-22T18:31:06.402914+0000 mon.a (mon.0) 5294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:07 smithi115 bash[46934]: audit 2022-04-22T18:31:06.957443+0000 mon.a (mon.0) 5295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:08 smithi121 bash[36902]: cluster 2022-04-22T18:31:07.676600+0000 mgr.y (mgr.34107) 2264 : cluster [DBG] pgmap v3894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:08 smithi115 bash[45495]: cluster 2022-04-22T18:31:07.676600+0000 mgr.y (mgr.34107) 2264 : cluster [DBG] pgmap v3894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:08 smithi115 bash[46934]: cluster 2022-04-22T18:31:07.676600+0000 mgr.y (mgr.34107) 2264 : cluster [DBG] pgmap v3894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:10 smithi121 bash[36902]: cluster 2022-04-22T18:31:09.677826+0000 mgr.y (mgr.34107) 2265 : cluster [DBG] pgmap v3895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:10 smithi115 bash[45495]: cluster 2022-04-22T18:31:09.677826+0000 mgr.y (mgr.34107) 2265 : cluster [DBG] pgmap v3895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:11.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:10 smithi115 bash[46934]: cluster 2022-04-22T18:31:09.677826+0000 mgr.y (mgr.34107) 2265 : cluster [DBG] pgmap v3895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:11 smithi121 bash[36902]: audit 2022-04-22T18:31:11.406034+0000 mon.a (mon.0) 5296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:12.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:11 smithi115 bash[45495]: audit 2022-04-22T18:31:11.406034+0000 mon.a (mon.0) 5296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:12.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:11.847Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:12.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:11 smithi115 bash[46934]: audit 2022-04-22T18:31:11.406034+0000 mon.a (mon.0) 5296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:12 smithi121 bash[36902]: cluster 2022-04-22T18:31:11.678966+0000 mgr.y (mgr.34107) 2266 : cluster [DBG] pgmap v3896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:12 smithi121 bash[36902]: audit 2022-04-22T18:31:11.960957+0000 mon.a (mon.0) 5297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:12 smithi115 bash[45495]: cluster 2022-04-22T18:31:11.678966+0000 mgr.y (mgr.34107) 2266 : cluster [DBG] pgmap v3896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:12 smithi115 bash[45495]: audit 2022-04-22T18:31:11.960957+0000 mon.a (mon.0) 5297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:12 smithi115 bash[46934]: cluster 2022-04-22T18:31:11.678966+0000 mgr.y (mgr.34107) 2266 : cluster [DBG] pgmap v3896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:12 smithi115 bash[46934]: audit 2022-04-22T18:31:11.960957+0000 mon.a (mon.0) 5297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:13.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:14 smithi121 bash[36902]: cluster 2022-04-22T18:31:13.680060+0000 mgr.y (mgr.34107) 2267 : cluster [DBG] pgmap v3897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:14 smithi121 bash[36902]: audit 2022-04-22T18:31:14.036439+0000 mon.a (mon.0) 5298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:31:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:14 smithi121 bash[36902]: audit 2022-04-22T18:31:14.039779+0000 mon.a (mon.0) 5299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:31:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[45495]: cluster 2022-04-22T18:31:13.680060+0000 mgr.y (mgr.34107) 2267 : cluster [DBG] pgmap v3897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[45495]: audit 2022-04-22T18:31:14.036439+0000 mon.a (mon.0) 5298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:31:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[45495]: audit 2022-04-22T18:31:14.039779+0000 mon.a (mon.0) 5299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:31:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[46934]: cluster 2022-04-22T18:31:13.680060+0000 mgr.y (mgr.34107) 2267 : cluster [DBG] pgmap v3897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[46934]: audit 2022-04-22T18:31:14.036439+0000 mon.a (mon.0) 5298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:31:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:14 smithi115 bash[46934]: audit 2022-04-22T18:31:14.039779+0000 mon.a (mon.0) 5299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:31:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:16 smithi121 bash[36902]: cluster 2022-04-22T18:31:15.681007+0000 mgr.y (mgr.34107) 2268 : cluster [DBG] pgmap v3898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:17.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:16 smithi121 bash[36902]: audit 2022-04-22T18:31:16.408989+0000 mon.a (mon.0) 5300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:16 smithi115 bash[46934]: cluster 2022-04-22T18:31:15.681007+0000 mgr.y (mgr.34107) 2268 : cluster [DBG] pgmap v3898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:16 smithi115 bash[46934]: audit 2022-04-22T18:31:16.408989+0000 mon.a (mon.0) 5300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:16 smithi115 bash[45495]: cluster 2022-04-22T18:31:15.681007+0000 mgr.y (mgr.34107) 2268 : cluster [DBG] pgmap v3898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:16 smithi115 bash[45495]: audit 2022-04-22T18:31:16.408989+0000 mon.a (mon.0) 5300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:17 smithi121 bash[36902]: audit 2022-04-22T18:31:16.964380+0000 mon.a (mon.0) 5301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:17 smithi115 bash[45495]: audit 2022-04-22T18:31:16.964380+0000 mon.a (mon.0) 5301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:17 smithi115 bash[46934]: audit 2022-04-22T18:31:16.964380+0000 mon.a (mon.0) 5301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:18 smithi115 bash[45495]: cluster 2022-04-22T18:31:17.682123+0000 mgr.y (mgr.34107) 2269 : cluster [DBG] pgmap v3899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:18 smithi115 bash[46934]: cluster 2022-04-22T18:31:17.682123+0000 mgr.y (mgr.34107) 2269 : cluster [DBG] pgmap v3899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:18 smithi121 bash[36902]: cluster 2022-04-22T18:31:17.682123+0000 mgr.y (mgr.34107) 2269 : cluster [DBG] pgmap v3899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:20 smithi115 bash[45495]: cluster 2022-04-22T18:31:19.683213+0000 mgr.y (mgr.34107) 2270 : cluster [DBG] pgmap v3900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:20 smithi115 bash[46934]: cluster 2022-04-22T18:31:19.683213+0000 mgr.y (mgr.34107) 2270 : cluster [DBG] pgmap v3900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:20 smithi121 bash[36902]: cluster 2022-04-22T18:31:19.683213+0000 mgr.y (mgr.34107) 2270 : cluster [DBG] pgmap v3900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:21 smithi115 bash[45495]: audit 2022-04-22T18:31:21.412274+0000 mon.a (mon.0) 5302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:21.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:21 smithi115 bash[46934]: audit 2022-04-22T18:31:21.412274+0000 mon.a (mon.0) 5302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:21 smithi121 bash[36902]: audit 2022-04-22T18:31:21.412274+0000 mon.a (mon.0) 5302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:22 smithi115 bash[45495]: cluster 2022-04-22T18:31:21.684324+0000 mgr.y (mgr.34107) 2271 : cluster [DBG] pgmap v3901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:22 smithi115 bash[45495]: audit 2022-04-22T18:31:21.968203+0000 mon.a (mon.0) 5303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:22 smithi115 bash[46934]: cluster 2022-04-22T18:31:21.684324+0000 mgr.y (mgr.34107) 2271 : cluster [DBG] pgmap v3901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:22 smithi115 bash[46934]: audit 2022-04-22T18:31:21.968203+0000 mon.a (mon.0) 5303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:22 smithi121 bash[36902]: cluster 2022-04-22T18:31:21.684324+0000 mgr.y (mgr.34107) 2271 : cluster [DBG] pgmap v3901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:23.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:22 smithi121 bash[36902]: audit 2022-04-22T18:31:21.968203+0000 mon.a (mon.0) 5303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:23.727Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:25.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:24 smithi115 bash[45495]: cluster 2022-04-22T18:31:23.685471+0000 mgr.y (mgr.34107) 2272 : cluster [DBG] pgmap v3902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:25.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:24 smithi115 bash[46934]: cluster 2022-04-22T18:31:23.685471+0000 mgr.y (mgr.34107) 2272 : cluster [DBG] pgmap v3902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:24 smithi121 bash[36902]: cluster 2022-04-22T18:31:23.685471+0000 mgr.y (mgr.34107) 2272 : cluster [DBG] pgmap v3902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:27.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:26 smithi115 bash[45495]: cluster 2022-04-22T18:31:25.686490+0000 mgr.y (mgr.34107) 2273 : cluster [DBG] pgmap v3903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:26 smithi115 bash[45495]: audit 2022-04-22T18:31:26.415401+0000 mon.a (mon.0) 5304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:26 smithi115 bash[46934]: cluster 2022-04-22T18:31:25.686490+0000 mgr.y (mgr.34107) 2273 : cluster [DBG] pgmap v3903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:26 smithi115 bash[46934]: audit 2022-04-22T18:31:26.415401+0000 mon.a (mon.0) 5304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:26 smithi121 bash[36902]: cluster 2022-04-22T18:31:25.686490+0000 mgr.y (mgr.34107) 2273 : cluster [DBG] pgmap v3903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:27.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:26 smithi121 bash[36902]: audit 2022-04-22T18:31:26.415401+0000 mon.a (mon.0) 5304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:27 smithi121 bash[36902]: audit 2022-04-22T18:31:26.972117+0000 mon.a (mon.0) 5305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:27 smithi115 bash[45495]: audit 2022-04-22T18:31:26.972117+0000 mon.a (mon.0) 5305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:27 smithi115 bash[46934]: audit 2022-04-22T18:31:26.972117+0000 mon.a (mon.0) 5305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:28 smithi121 bash[36902]: cluster 2022-04-22T18:31:27.687619+0000 mgr.y (mgr.34107) 2274 : cluster [DBG] pgmap v3904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:29.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:28 smithi115 bash[45495]: cluster 2022-04-22T18:31:27.687619+0000 mgr.y (mgr.34107) 2274 : cluster [DBG] pgmap v3904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:29.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:28 smithi115 bash[46934]: cluster 2022-04-22T18:31:27.687619+0000 mgr.y (mgr.34107) 2274 : cluster [DBG] pgmap v3904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:29.678 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:31:30.019 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:31:30.019 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (54m) 2m ago 68m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:31:30.019 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (68m) 3m ago 68m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:31:30.050 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (56m) 2m ago 69m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:31:30.050 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (56m) 3m ago 69m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:31:30.050 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (63m) 3m ago 73m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (64m) 2m ago 75m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (63m) 2m ago 75m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (62m) 3m ago 73m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (62m) 2m ago 74m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (55m) 2m ago 68m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (55m) 3m ago 68m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (61m) 2m ago 73m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (60m) 2m ago 72m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (60m) 2m ago 72m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:31:30.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (59m) 2m ago 71m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:31:30.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (59m) 3m ago 71m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:31:30.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (58m) 3m ago 70m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:31:30.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (57m) 3m ago 70m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:31:30.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (56m) 3m ago 69m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:31:30.052 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (55m) 3m ago 68m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:31:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:29 smithi121 bash[36902]: audit 2022-04-22T18:31:29.654299+0000 mgr.y (mgr.34107) 2275 : audit [DBG] from='client.37818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:30.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:29 smithi121 bash[36902]: cluster 2022-04-22T18:31:29.688790+0000 mgr.y (mgr.34107) 2276 : cluster [DBG] pgmap v3905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:29 smithi115 bash[45495]: audit 2022-04-22T18:31:29.654299+0000 mgr.y (mgr.34107) 2275 : audit [DBG] from='client.37818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:29 smithi115 bash[45495]: cluster 2022-04-22T18:31:29.688790+0000 mgr.y (mgr.34107) 2276 : cluster [DBG] pgmap v3905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:29 smithi115 bash[46934]: audit 2022-04-22T18:31:29.654299+0000 mgr.y (mgr.34107) 2275 : audit [DBG] from='client.37818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:29 smithi115 bash[46934]: cluster 2022-04-22T18:31:29.688790+0000 mgr.y (mgr.34107) 2276 : cluster [DBG] pgmap v3905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:30.427 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:31:30.427 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:31:30.427 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:31:30.428 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:31:30.429 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:31:30.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:31:30.429 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:31:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:30 smithi121 bash[36902]: audit 2022-04-22T18:31:29.997290+0000 mgr.y (mgr.34107) 2277 : audit [DBG] from='client.37824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:30 smithi121 bash[36902]: audit 2022-04-22T18:31:30.420040+0000 mon.a (mon.0) 5306 : audit [DBG] from='client.? 172.21.15.115:0/378046910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:30 smithi115 bash[45495]: audit 2022-04-22T18:31:29.997290+0000 mgr.y (mgr.34107) 2277 : audit [DBG] from='client.37824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:30 smithi115 bash[45495]: audit 2022-04-22T18:31:30.420040+0000 mon.a (mon.0) 5306 : audit [DBG] from='client.? 172.21.15.115:0/378046910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:30 smithi115 bash[46934]: audit 2022-04-22T18:31:29.997290+0000 mgr.y (mgr.34107) 2277 : audit [DBG] from='client.37824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:31:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:30 smithi115 bash[46934]: audit 2022-04-22T18:31:30.420040+0000 mon.a (mon.0) 5306 : audit [DBG] from='client.? 172.21.15.115:0/378046910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:31:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:31 smithi121 bash[36902]: audit 2022-04-22T18:31:31.418373+0000 mon.a (mon.0) 5307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:31 smithi121 bash[36902]: cluster 2022-04-22T18:31:31.689857+0000 mgr.y (mgr.34107) 2278 : cluster [DBG] pgmap v3906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:32.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:31.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:31 smithi115 bash[45495]: audit 2022-04-22T18:31:31.418373+0000 mon.a (mon.0) 5307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:31 smithi115 bash[45495]: cluster 2022-04-22T18:31:31.689857+0000 mgr.y (mgr.34107) 2278 : cluster [DBG] pgmap v3906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:31 smithi115 bash[46934]: audit 2022-04-22T18:31:31.418373+0000 mon.a (mon.0) 5307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:31 smithi115 bash[46934]: cluster 2022-04-22T18:31:31.689857+0000 mgr.y (mgr.34107) 2278 : cluster [DBG] pgmap v3906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:32 smithi121 bash[36902]: audit 2022-04-22T18:31:31.976023+0000 mon.a (mon.0) 5308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:32 smithi115 bash[45495]: audit 2022-04-22T18:31:31.976023+0000 mon.a (mon.0) 5308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:32 smithi115 bash[46934]: audit 2022-04-22T18:31:31.976023+0000 mon.a (mon.0) 5308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:33 smithi115 bash[45495]: cluster 2022-04-22T18:31:33.691100+0000 mgr.y (mgr.34107) 2279 : cluster [DBG] pgmap v3907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:34.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:33 smithi115 bash[46934]: cluster 2022-04-22T18:31:33.691100+0000 mgr.y (mgr.34107) 2279 : cluster [DBG] pgmap v3907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:33.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:33 smithi121 bash[36902]: cluster 2022-04-22T18:31:33.691100+0000 mgr.y (mgr.34107) 2279 : cluster [DBG] pgmap v3907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:36 smithi121 bash[36902]: cluster 2022-04-22T18:31:35.692223+0000 mgr.y (mgr.34107) 2280 : cluster [DBG] pgmap v3908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:37.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:36 smithi121 bash[36902]: audit 2022-04-22T18:31:36.421867+0000 mon.a (mon.0) 5309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:36 smithi115 bash[45495]: cluster 2022-04-22T18:31:35.692223+0000 mgr.y (mgr.34107) 2280 : cluster [DBG] pgmap v3908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:37.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:36 smithi115 bash[45495]: audit 2022-04-22T18:31:36.421867+0000 mon.a (mon.0) 5309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:36 smithi115 bash[46934]: cluster 2022-04-22T18:31:35.692223+0000 mgr.y (mgr.34107) 2280 : cluster [DBG] pgmap v3908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:37.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:36 smithi115 bash[46934]: audit 2022-04-22T18:31:36.421867+0000 mon.a (mon.0) 5309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:37 smithi121 bash[36902]: audit 2022-04-22T18:31:36.979805+0000 mon.a (mon.0) 5310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:38.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:37 smithi115 bash[45495]: audit 2022-04-22T18:31:36.979805+0000 mon.a (mon.0) 5310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:37 smithi115 bash[46934]: audit 2022-04-22T18:31:36.979805+0000 mon.a (mon.0) 5310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:39.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:38 smithi115 bash[45495]: cluster 2022-04-22T18:31:37.693378+0000 mgr.y (mgr.34107) 2281 : cluster [DBG] pgmap v3909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:38 smithi115 bash[46934]: cluster 2022-04-22T18:31:37.693378+0000 mgr.y (mgr.34107) 2281 : cluster [DBG] pgmap v3909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:38 smithi121 bash[36902]: cluster 2022-04-22T18:31:37.693378+0000 mgr.y (mgr.34107) 2281 : cluster [DBG] pgmap v3909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:40 smithi115 bash[45495]: cluster 2022-04-22T18:31:39.694526+0000 mgr.y (mgr.34107) 2282 : cluster [DBG] pgmap v3910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:40 smithi115 bash[46934]: cluster 2022-04-22T18:31:39.694526+0000 mgr.y (mgr.34107) 2282 : cluster [DBG] pgmap v3910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:40 smithi121 bash[36902]: cluster 2022-04-22T18:31:39.694526+0000 mgr.y (mgr.34107) 2282 : cluster [DBG] pgmap v3910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:42.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:41 smithi115 bash[46934]: audit 2022-04-22T18:31:41.424719+0000 mon.a (mon.0) 5311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:41 smithi115 bash[45495]: audit 2022-04-22T18:31:41.424719+0000 mon.a (mon.0) 5311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:42.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:41.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:41 smithi121 bash[36902]: audit 2022-04-22T18:31:41.424719+0000 mon.a (mon.0) 5311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:42 smithi115 bash[45495]: cluster 2022-04-22T18:31:41.695659+0000 mgr.y (mgr.34107) 2283 : cluster [DBG] pgmap v3911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:42 smithi115 bash[45495]: audit 2022-04-22T18:31:41.983548+0000 mon.a (mon.0) 5312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:42 smithi115 bash[46934]: cluster 2022-04-22T18:31:41.695659+0000 mgr.y (mgr.34107) 2283 : cluster [DBG] pgmap v3911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:42 smithi115 bash[46934]: audit 2022-04-22T18:31:41.983548+0000 mon.a (mon.0) 5312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:42 smithi121 bash[36902]: cluster 2022-04-22T18:31:41.695659+0000 mgr.y (mgr.34107) 2283 : cluster [DBG] pgmap v3911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:42 smithi121 bash[36902]: audit 2022-04-22T18:31:41.983548+0000 mon.a (mon.0) 5312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:43.727Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:44 smithi115 bash[45495]: cluster 2022-04-22T18:31:43.696866+0000 mgr.y (mgr.34107) 2284 : cluster [DBG] pgmap v3912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:44 smithi115 bash[46934]: cluster 2022-04-22T18:31:43.696866+0000 mgr.y (mgr.34107) 2284 : cluster [DBG] pgmap v3912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:44 smithi121 bash[36902]: cluster 2022-04-22T18:31:43.696866+0000 mgr.y (mgr.34107) 2284 : cluster [DBG] pgmap v3912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:47.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:46 smithi115 bash[45495]: cluster 2022-04-22T18:31:45.697901+0000 mgr.y (mgr.34107) 2285 : cluster [DBG] pgmap v3913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:46 smithi115 bash[45495]: audit 2022-04-22T18:31:46.427557+0000 mon.a (mon.0) 5313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:46 smithi115 bash[46934]: cluster 2022-04-22T18:31:45.697901+0000 mgr.y (mgr.34107) 2285 : cluster [DBG] pgmap v3913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:46 smithi115 bash[46934]: audit 2022-04-22T18:31:46.427557+0000 mon.a (mon.0) 5313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:46 smithi121 bash[36902]: cluster 2022-04-22T18:31:45.697901+0000 mgr.y (mgr.34107) 2285 : cluster [DBG] pgmap v3913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:46 smithi121 bash[36902]: audit 2022-04-22T18:31:46.427557+0000 mon.a (mon.0) 5313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:47 smithi115 bash[45495]: audit 2022-04-22T18:31:46.986950+0000 mon.a (mon.0) 5314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:47 smithi115 bash[46934]: audit 2022-04-22T18:31:46.986950+0000 mon.a (mon.0) 5314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:47 smithi121 bash[36902]: audit 2022-04-22T18:31:46.986950+0000 mon.a (mon.0) 5314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:49.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:48 smithi115 bash[46934]: cluster 2022-04-22T18:31:47.699048+0000 mgr.y (mgr.34107) 2286 : cluster [DBG] pgmap v3914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:49.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:48 smithi115 bash[45495]: cluster 2022-04-22T18:31:47.699048+0000 mgr.y (mgr.34107) 2286 : cluster [DBG] pgmap v3914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:48 smithi121 bash[36902]: cluster 2022-04-22T18:31:47.699048+0000 mgr.y (mgr.34107) 2286 : cluster [DBG] pgmap v3914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:50 smithi121 bash[36902]: cluster 2022-04-22T18:31:49.700163+0000 mgr.y (mgr.34107) 2287 : cluster [DBG] pgmap v3915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:50 smithi115 bash[45495]: cluster 2022-04-22T18:31:49.700163+0000 mgr.y (mgr.34107) 2287 : cluster [DBG] pgmap v3915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:51.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:50 smithi115 bash[46934]: cluster 2022-04-22T18:31:49.700163+0000 mgr.y (mgr.34107) 2287 : cluster [DBG] pgmap v3915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:51 smithi121 bash[36902]: audit 2022-04-22T18:31:51.430388+0000 mon.a (mon.0) 5315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:51.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:52.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:51 smithi115 bash[45495]: audit 2022-04-22T18:31:51.430388+0000 mon.a (mon.0) 5315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:52.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:51 smithi115 bash[46934]: audit 2022-04-22T18:31:51.430388+0000 mon.a (mon.0) 5315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:52 smithi121 bash[36902]: cluster 2022-04-22T18:31:51.701291+0000 mgr.y (mgr.34107) 2288 : cluster [DBG] pgmap v3916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:52 smithi121 bash[36902]: audit 2022-04-22T18:31:51.990685+0000 mon.a (mon.0) 5316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:52 smithi115 bash[45495]: cluster 2022-04-22T18:31:51.701291+0000 mgr.y (mgr.34107) 2288 : cluster [DBG] pgmap v3916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:52 smithi115 bash[45495]: audit 2022-04-22T18:31:51.990685+0000 mon.a (mon.0) 5316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:52 smithi115 bash[46934]: cluster 2022-04-22T18:31:51.701291+0000 mgr.y (mgr.34107) 2288 : cluster [DBG] pgmap v3916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:52 smithi115 bash[46934]: audit 2022-04-22T18:31:51.990685+0000 mon.a (mon.0) 5316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:53 smithi115 bash[45495]: cluster 2022-04-22T18:31:53.702456+0000 mgr.y (mgr.34107) 2289 : cluster [DBG] pgmap v3917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:31:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:31:53.729Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:31:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:53 smithi115 bash[46934]: cluster 2022-04-22T18:31:53.702456+0000 mgr.y (mgr.34107) 2289 : cluster [DBG] pgmap v3917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:53 smithi121 bash[36902]: cluster 2022-04-22T18:31:53.702456+0000 mgr.y (mgr.34107) 2289 : cluster [DBG] pgmap v3917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:57.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:56 smithi115 bash[46934]: cluster 2022-04-22T18:31:55.703497+0000 mgr.y (mgr.34107) 2290 : cluster [DBG] pgmap v3918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:56 smithi115 bash[46934]: audit 2022-04-22T18:31:56.433444+0000 mon.a (mon.0) 5317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:56 smithi115 bash[45495]: cluster 2022-04-22T18:31:55.703497+0000 mgr.y (mgr.34107) 2290 : cluster [DBG] pgmap v3918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:56 smithi115 bash[45495]: audit 2022-04-22T18:31:56.433444+0000 mon.a (mon.0) 5317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:56 smithi121 bash[36902]: cluster 2022-04-22T18:31:55.703497+0000 mgr.y (mgr.34107) 2290 : cluster [DBG] pgmap v3918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:56 smithi121 bash[36902]: audit 2022-04-22T18:31:56.433444+0000 mon.a (mon.0) 5317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:31:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:57 smithi115 bash[46934]: audit 2022-04-22T18:31:56.994279+0000 mon.a (mon.0) 5318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:57 smithi115 bash[45495]: audit 2022-04-22T18:31:56.994279+0000 mon.a (mon.0) 5318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:57 smithi121 bash[36902]: audit 2022-04-22T18:31:56.994279+0000 mon.a (mon.0) 5318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:31:59.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:31:58 smithi115 bash[45495]: cluster 2022-04-22T18:31:57.704650+0000 mgr.y (mgr.34107) 2291 : cluster [DBG] pgmap v3919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:31:58 smithi115 bash[46934]: cluster 2022-04-22T18:31:57.704650+0000 mgr.y (mgr.34107) 2291 : cluster [DBG] pgmap v3919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:31:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:31:58 smithi121 bash[36902]: cluster 2022-04-22T18:31:57.704650+0000 mgr.y (mgr.34107) 2291 : cluster [DBG] pgmap v3919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:00.802 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:32:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:00 smithi115 bash[46934]: cluster 2022-04-22T18:31:59.705814+0000 mgr.y (mgr.34107) 2292 : cluster [DBG] pgmap v3920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:00 smithi115 bash[45495]: cluster 2022-04-22T18:31:59.705814+0000 mgr.y (mgr.34107) 2292 : cluster [DBG] pgmap v3920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (55m) 2m ago 68m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (68m) 3m ago 68m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (57m) 2m ago 69m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (56m) 3m ago 69m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (64m) 3m ago 74m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:32:01.137 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (64m) 2m ago 76m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (63m) 2m ago 76m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (63m) 3m ago 74m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (63m) 2m ago 74m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (56m) 2m ago 69m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (56m) 3m ago 69m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (61m) 2m ago 73m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (61m) 2m ago 73m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (60m) 2m ago 72m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (60m) 2m ago 72m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:32:01.138 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (59m) 3m ago 71m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:32:01.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (58m) 3m ago 71m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:32:01.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (58m) 3m ago 70m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:32:01.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (57m) 3m ago 70m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:32:01.139 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (55m) 3m ago 69m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:32:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:00 smithi121 bash[36902]: cluster 2022-04-22T18:31:59.705814+0000 mgr.y (mgr.34107) 2292 : cluster [DBG] pgmap v3920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:01.542 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:32:01.543 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:32:02.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:01.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[45495]: audit 2022-04-22T18:32:00.777879+0000 mgr.y (mgr.34107) 2293 : audit [DBG] from='client.47770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[45495]: audit 2022-04-22T18:32:01.436457+0000 mon.a (mon.0) 5319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[45495]: audit 2022-04-22T18:32:01.534434+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.115:0/3944890479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[46934]: audit 2022-04-22T18:32:00.777879+0000 mgr.y (mgr.34107) 2293 : audit [DBG] from='client.47770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:02.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[46934]: audit 2022-04-22T18:32:01.436457+0000 mon.a (mon.0) 5319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:01 smithi115 bash[46934]: audit 2022-04-22T18:32:01.534434+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.115:0/3944890479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:01 smithi121 bash[36902]: audit 2022-04-22T18:32:00.777879+0000 mgr.y (mgr.34107) 2293 : audit [DBG] from='client.47770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:01 smithi121 bash[36902]: audit 2022-04-22T18:32:01.436457+0000 mon.a (mon.0) 5319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:01 smithi121 bash[36902]: audit 2022-04-22T18:32:01.534434+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.115:0/3944890479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[46934]: audit 2022-04-22T18:32:01.114036+0000 mgr.y (mgr.34107) 2294 : audit [DBG] from='client.47776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[46934]: cluster 2022-04-22T18:32:01.706933+0000 mgr.y (mgr.34107) 2295 : cluster [DBG] pgmap v3921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[46934]: audit 2022-04-22T18:32:01.998271+0000 mon.a (mon.0) 5320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[45495]: audit 2022-04-22T18:32:01.114036+0000 mgr.y (mgr.34107) 2294 : audit [DBG] from='client.47776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[45495]: cluster 2022-04-22T18:32:01.706933+0000 mgr.y (mgr.34107) 2295 : cluster [DBG] pgmap v3921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:02 smithi115 bash[45495]: audit 2022-04-22T18:32:01.998271+0000 mon.a (mon.0) 5320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:02 smithi121 bash[36902]: audit 2022-04-22T18:32:01.114036+0000 mgr.y (mgr.34107) 2294 : audit [DBG] from='client.47776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:02 smithi121 bash[36902]: cluster 2022-04-22T18:32:01.706933+0000 mgr.y (mgr.34107) 2295 : cluster [DBG] pgmap v3921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:02 smithi121 bash[36902]: audit 2022-04-22T18:32:01.998271+0000 mon.a (mon.0) 5320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:04.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:03.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:04 smithi115 bash[46934]: cluster 2022-04-22T18:32:03.708120+0000 mgr.y (mgr.34107) 2296 : cluster [DBG] pgmap v3922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:04 smithi115 bash[45495]: cluster 2022-04-22T18:32:03.708120+0000 mgr.y (mgr.34107) 2296 : cluster [DBG] pgmap v3922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:05.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:04 smithi121 bash[36902]: cluster 2022-04-22T18:32:03.708120+0000 mgr.y (mgr.34107) 2296 : cluster [DBG] pgmap v3922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:05 smithi121 bash[36902]: audit 2022-04-22T18:32:05.106630+0000 mon.a (mon.0) 5321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:32:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:05 smithi121 bash[36902]: audit 2022-04-22T18:32:05.111085+0000 mon.a (mon.0) 5322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:32:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:05 smithi121 bash[36902]: audit 2022-04-22T18:32:05.113008+0000 mon.a (mon.0) 5323 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:32:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:05 smithi121 bash[36902]: audit 2022-04-22T18:32:05.479061+0000 mon.a (mon.0) 5324 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:32:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[45495]: audit 2022-04-22T18:32:05.106630+0000 mon.a (mon.0) 5321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:32:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[45495]: audit 2022-04-22T18:32:05.111085+0000 mon.a (mon.0) 5322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:32:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[45495]: audit 2022-04-22T18:32:05.113008+0000 mon.a (mon.0) 5323 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:32:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[45495]: audit 2022-04-22T18:32:05.479061+0000 mon.a (mon.0) 5324 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:32:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[46934]: audit 2022-04-22T18:32:05.106630+0000 mon.a (mon.0) 5321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:32:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[46934]: audit 2022-04-22T18:32:05.111085+0000 mon.a (mon.0) 5322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:32:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[46934]: audit 2022-04-22T18:32:05.113008+0000 mon.a (mon.0) 5323 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:32:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:05 smithi115 bash[46934]: audit 2022-04-22T18:32:05.479061+0000 mon.a (mon.0) 5324 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:32:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:06 smithi121 bash[36902]: cluster 2022-04-22T18:32:05.709255+0000 mgr.y (mgr.34107) 2297 : cluster [DBG] pgmap v3923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:06 smithi121 bash[36902]: audit 2022-04-22T18:32:06.439077+0000 mon.a (mon.0) 5325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:06 smithi115 bash[45495]: cluster 2022-04-22T18:32:05.709255+0000 mgr.y (mgr.34107) 2297 : cluster [DBG] pgmap v3923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:06 smithi115 bash[45495]: audit 2022-04-22T18:32:06.439077+0000 mon.a (mon.0) 5325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:06 smithi115 bash[46934]: cluster 2022-04-22T18:32:05.709255+0000 mgr.y (mgr.34107) 2297 : cluster [DBG] pgmap v3923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:06 smithi115 bash[46934]: audit 2022-04-22T18:32:06.439077+0000 mon.a (mon.0) 5325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:07 smithi121 bash[36902]: audit 2022-04-22T18:32:07.001909+0000 mon.a (mon.0) 5326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:07 smithi115 bash[45495]: audit 2022-04-22T18:32:07.001909+0000 mon.a (mon.0) 5326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:07 smithi115 bash[46934]: audit 2022-04-22T18:32:07.001909+0000 mon.a (mon.0) 5326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:08 smithi121 bash[36902]: cluster 2022-04-22T18:32:07.710415+0000 mgr.y (mgr.34107) 2298 : cluster [DBG] pgmap v3924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:09.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:08 smithi115 bash[45495]: cluster 2022-04-22T18:32:07.710415+0000 mgr.y (mgr.34107) 2298 : cluster [DBG] pgmap v3924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:09.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:08 smithi115 bash[46934]: cluster 2022-04-22T18:32:07.710415+0000 mgr.y (mgr.34107) 2298 : cluster [DBG] pgmap v3924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:09 smithi121 bash[36902]: cluster 2022-04-22T18:32:09.711688+0000 mgr.y (mgr.34107) 2299 : cluster [DBG] pgmap v3925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:10.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:09 smithi115 bash[45495]: cluster 2022-04-22T18:32:09.711688+0000 mgr.y (mgr.34107) 2299 : cluster [DBG] pgmap v3925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:09 smithi115 bash[46934]: cluster 2022-04-22T18:32:09.711688+0000 mgr.y (mgr.34107) 2299 : cluster [DBG] pgmap v3925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:11 smithi121 bash[36902]: audit 2022-04-22T18:32:11.442001+0000 mon.a (mon.0) 5327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:11 smithi115 bash[45495]: audit 2022-04-22T18:32:11.442001+0000 mon.a (mon.0) 5327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:11 smithi115 bash[46934]: audit 2022-04-22T18:32:11.442001+0000 mon.a (mon.0) 5327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:11.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:12 smithi121 bash[36902]: cluster 2022-04-22T18:32:11.712963+0000 mgr.y (mgr.34107) 2300 : cluster [DBG] pgmap v3926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:12 smithi121 bash[36902]: audit 2022-04-22T18:32:12.005350+0000 mon.a (mon.0) 5328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:12 smithi115 bash[45495]: cluster 2022-04-22T18:32:11.712963+0000 mgr.y (mgr.34107) 2300 : cluster [DBG] pgmap v3926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:12 smithi115 bash[45495]: audit 2022-04-22T18:32:12.005350+0000 mon.a (mon.0) 5328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:12 smithi115 bash[46934]: cluster 2022-04-22T18:32:11.712963+0000 mgr.y (mgr.34107) 2300 : cluster [DBG] pgmap v3926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:12 smithi115 bash[46934]: audit 2022-04-22T18:32:12.005350+0000 mon.a (mon.0) 5328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:14.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:13.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:15.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[45495]: cluster 2022-04-22T18:32:13.714097+0000 mgr.y (mgr.34107) 2301 : cluster [DBG] pgmap v3927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[45495]: audit 2022-04-22T18:32:14.043235+0000 mon.a (mon.0) 5329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:32:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[45495]: audit 2022-04-22T18:32:14.046494+0000 mon.a (mon.0) 5330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:32:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[46934]: cluster 2022-04-22T18:32:13.714097+0000 mgr.y (mgr.34107) 2301 : cluster [DBG] pgmap v3927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[46934]: audit 2022-04-22T18:32:14.043235+0000 mon.a (mon.0) 5329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:32:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:14 smithi115 bash[46934]: audit 2022-04-22T18:32:14.046494+0000 mon.a (mon.0) 5330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:32:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:14 smithi121 bash[36902]: cluster 2022-04-22T18:32:13.714097+0000 mgr.y (mgr.34107) 2301 : cluster [DBG] pgmap v3927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:14 smithi121 bash[36902]: audit 2022-04-22T18:32:14.043235+0000 mon.a (mon.0) 5329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:32:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:14 smithi121 bash[36902]: audit 2022-04-22T18:32:14.046494+0000 mon.a (mon.0) 5330 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:32:17.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:16 smithi115 bash[45495]: cluster 2022-04-22T18:32:15.715103+0000 mgr.y (mgr.34107) 2302 : cluster [DBG] pgmap v3928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:16 smithi115 bash[45495]: audit 2022-04-22T18:32:16.445316+0000 mon.a (mon.0) 5331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:16 smithi115 bash[46934]: cluster 2022-04-22T18:32:15.715103+0000 mgr.y (mgr.34107) 2302 : cluster [DBG] pgmap v3928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:16 smithi115 bash[46934]: audit 2022-04-22T18:32:16.445316+0000 mon.a (mon.0) 5331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:16 smithi121 bash[36902]: cluster 2022-04-22T18:32:15.715103+0000 mgr.y (mgr.34107) 2302 : cluster [DBG] pgmap v3928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:16 smithi121 bash[36902]: audit 2022-04-22T18:32:16.445316+0000 mon.a (mon.0) 5331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:17 smithi115 bash[45495]: audit 2022-04-22T18:32:17.009208+0000 mon.a (mon.0) 5332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:17 smithi115 bash[46934]: audit 2022-04-22T18:32:17.009208+0000 mon.a (mon.0) 5332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:17 smithi121 bash[36902]: audit 2022-04-22T18:32:17.009208+0000 mon.a (mon.0) 5332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:19.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:18 smithi115 bash[45495]: cluster 2022-04-22T18:32:17.716352+0000 mgr.y (mgr.34107) 2303 : cluster [DBG] pgmap v3929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:19.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:18 smithi115 bash[46934]: cluster 2022-04-22T18:32:17.716352+0000 mgr.y (mgr.34107) 2303 : cluster [DBG] pgmap v3929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:18 smithi121 bash[36902]: cluster 2022-04-22T18:32:17.716352+0000 mgr.y (mgr.34107) 2303 : cluster [DBG] pgmap v3929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:20 smithi121 bash[36902]: cluster 2022-04-22T18:32:19.717457+0000 mgr.y (mgr.34107) 2304 : cluster [DBG] pgmap v3930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:21.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:20 smithi115 bash[45495]: cluster 2022-04-22T18:32:19.717457+0000 mgr.y (mgr.34107) 2304 : cluster [DBG] pgmap v3930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:21.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:20 smithi115 bash[46934]: cluster 2022-04-22T18:32:19.717457+0000 mgr.y (mgr.34107) 2304 : cluster [DBG] pgmap v3930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:21 smithi121 bash[36902]: audit 2022-04-22T18:32:21.448390+0000 mon.a (mon.0) 5333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:21 smithi115 bash[45495]: audit 2022-04-22T18:32:21.448390+0000 mon.a (mon.0) 5333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:21.853Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:21 smithi115 bash[46934]: audit 2022-04-22T18:32:21.448390+0000 mon.a (mon.0) 5333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:22 smithi121 bash[36902]: cluster 2022-04-22T18:32:21.718664+0000 mgr.y (mgr.34107) 2305 : cluster [DBG] pgmap v3931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:22 smithi121 bash[36902]: audit 2022-04-22T18:32:22.012750+0000 mon.a (mon.0) 5334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:22 smithi115 bash[45495]: cluster 2022-04-22T18:32:21.718664+0000 mgr.y (mgr.34107) 2305 : cluster [DBG] pgmap v3931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:22 smithi115 bash[45495]: audit 2022-04-22T18:32:22.012750+0000 mon.a (mon.0) 5334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:23.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:22 smithi115 bash[46934]: cluster 2022-04-22T18:32:21.718664+0000 mgr.y (mgr.34107) 2305 : cluster [DBG] pgmap v3931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:23.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:22 smithi115 bash[46934]: audit 2022-04-22T18:32:22.012750+0000 mon.a (mon.0) 5334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:23.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:24 smithi121 bash[36902]: cluster 2022-04-22T18:32:23.719842+0000 mgr.y (mgr.34107) 2306 : cluster [DBG] pgmap v3932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:25.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:24 smithi115 bash[45495]: cluster 2022-04-22T18:32:23.719842+0000 mgr.y (mgr.34107) 2306 : cluster [DBG] pgmap v3932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:25.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:24 smithi115 bash[46934]: cluster 2022-04-22T18:32:23.719842+0000 mgr.y (mgr.34107) 2306 : cluster [DBG] pgmap v3932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:26 smithi121 bash[36902]: cluster 2022-04-22T18:32:25.720898+0000 mgr.y (mgr.34107) 2307 : cluster [DBG] pgmap v3933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:26 smithi121 bash[36902]: audit 2022-04-22T18:32:26.451299+0000 mon.a (mon.0) 5335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:26 smithi115 bash[45495]: cluster 2022-04-22T18:32:25.720898+0000 mgr.y (mgr.34107) 2307 : cluster [DBG] pgmap v3933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:26 smithi115 bash[45495]: audit 2022-04-22T18:32:26.451299+0000 mon.a (mon.0) 5335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:26 smithi115 bash[46934]: cluster 2022-04-22T18:32:25.720898+0000 mgr.y (mgr.34107) 2307 : cluster [DBG] pgmap v3933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:26 smithi115 bash[46934]: audit 2022-04-22T18:32:26.451299+0000 mon.a (mon.0) 5335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:27 smithi121 bash[36902]: audit 2022-04-22T18:32:27.016430+0000 mon.a (mon.0) 5336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:27 smithi121 bash[36902]: cluster 2022-04-22T18:32:27.722038+0000 mgr.y (mgr.34107) 2308 : cluster [DBG] pgmap v3934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:27 smithi115 bash[45495]: audit 2022-04-22T18:32:27.016430+0000 mon.a (mon.0) 5336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:27 smithi115 bash[45495]: cluster 2022-04-22T18:32:27.722038+0000 mgr.y (mgr.34107) 2308 : cluster [DBG] pgmap v3934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:27 smithi115 bash[46934]: audit 2022-04-22T18:32:27.016430+0000 mon.a (mon.0) 5336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:27 smithi115 bash[46934]: cluster 2022-04-22T18:32:27.722038+0000 mgr.y (mgr.34107) 2308 : cluster [DBG] pgmap v3934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:31.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:30 smithi115 bash[45495]: cluster 2022-04-22T18:32:29.723100+0000 mgr.y (mgr.34107) 2309 : cluster [DBG] pgmap v3935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:31.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:30 smithi115 bash[46934]: cluster 2022-04-22T18:32:29.723100+0000 mgr.y (mgr.34107) 2309 : cluster [DBG] pgmap v3935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:30 smithi121 bash[36902]: cluster 2022-04-22T18:32:29.723100+0000 mgr.y (mgr.34107) 2309 : cluster [DBG] pgmap v3935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:31.923 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:32:32.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:31 smithi115 bash[46934]: audit 2022-04-22T18:32:31.454465+0000 mon.a (mon.0) 5337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:32.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:31.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:32.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:31 smithi115 bash[45495]: audit 2022-04-22T18:32:31.454465+0000 mon.a (mon.0) 5337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (55m) 3m ago 69m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (69m) 4m ago 69m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (57m) 3m ago 70m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (57m) 4m ago 70m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:32:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (64m) 4m ago 74m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (65m) 3m ago 76m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (64m) 3m ago 77m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (63m) 4m ago 75m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (63m) 3m ago 75m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (56m) 3m ago 69m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (56m) 4m ago 69m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:32:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (62m) 3m ago 74m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (61m) 3m ago 73m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (61m) 3m ago 73m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (60m) 3m ago 72m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (60m) 4m ago 72m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (59m) 4m ago 71m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (58m) 4m ago 71m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:32:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (57m) 4m ago 70m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:32:32.266 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (56m) 4m ago 69m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:32:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:31 smithi121 bash[36902]: audit 2022-04-22T18:32:31.454465+0000 mon.a (mon.0) 5337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:32:32.670 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:32:32.670 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:32:32.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:32.670 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:32:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:32:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:32:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:32:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:32:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:32:32.672 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:32:33.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[45495]: cluster 2022-04-22T18:32:31.724211+0000 mgr.y (mgr.34107) 2310 : cluster [DBG] pgmap v3936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[45495]: audit 2022-04-22T18:32:32.019991+0000 mon.a (mon.0) 5338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[45495]: audit 2022-04-22T18:32:32.662823+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.115:0/975263345' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[46934]: cluster 2022-04-22T18:32:31.724211+0000 mgr.y (mgr.34107) 2310 : cluster [DBG] pgmap v3936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[46934]: audit 2022-04-22T18:32:32.019991+0000 mon.a (mon.0) 5338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:32 smithi115 bash[46934]: audit 2022-04-22T18:32:32.662823+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.115:0/975263345' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:32 smithi121 bash[36902]: cluster 2022-04-22T18:32:31.724211+0000 mgr.y (mgr.34107) 2310 : cluster [DBG] pgmap v3936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:32 smithi121 bash[36902]: audit 2022-04-22T18:32:32.019991+0000 mon.a (mon.0) 5338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:32 smithi121 bash[36902]: audit 2022-04-22T18:32:32.662823+0000 mon.c (mon.1) 146 : audit [DBG] from='client.? 172.21.15.115:0/975263345' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:32:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:33.731Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:34.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:33 smithi115 bash[45495]: audit 2022-04-22T18:32:31.898963+0000 mgr.y (mgr.34107) 2311 : audit [DBG] from='client.37854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:34.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:33 smithi115 bash[45495]: audit 2022-04-22T18:32:32.234570+0000 mgr.y (mgr.34107) 2312 : audit [DBG] from='client.47794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:34.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:33 smithi115 bash[46934]: audit 2022-04-22T18:32:31.898963+0000 mgr.y (mgr.34107) 2311 : audit [DBG] from='client.37854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:33 smithi115 bash[46934]: audit 2022-04-22T18:32:32.234570+0000 mgr.y (mgr.34107) 2312 : audit [DBG] from='client.47794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:33 smithi121 bash[36902]: audit 2022-04-22T18:32:31.898963+0000 mgr.y (mgr.34107) 2311 : audit [DBG] from='client.37854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:33 smithi121 bash[36902]: audit 2022-04-22T18:32:32.234570+0000 mgr.y (mgr.34107) 2312 : audit [DBG] from='client.47794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:32:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:34 smithi121 bash[36902]: cluster 2022-04-22T18:32:33.725337+0000 mgr.y (mgr.34107) 2313 : cluster [DBG] pgmap v3937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:35.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:34 smithi115 bash[45495]: cluster 2022-04-22T18:32:33.725337+0000 mgr.y (mgr.34107) 2313 : cluster [DBG] pgmap v3937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:34 smithi115 bash[46934]: cluster 2022-04-22T18:32:33.725337+0000 mgr.y (mgr.34107) 2313 : cluster [DBG] pgmap v3937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:36 smithi121 bash[36902]: cluster 2022-04-22T18:32:35.726479+0000 mgr.y (mgr.34107) 2314 : cluster [DBG] pgmap v3938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:36 smithi121 bash[36902]: audit 2022-04-22T18:32:36.457550+0000 mon.a (mon.0) 5339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:36 smithi115 bash[45495]: cluster 2022-04-22T18:32:35.726479+0000 mgr.y (mgr.34107) 2314 : cluster [DBG] pgmap v3938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:36 smithi115 bash[45495]: audit 2022-04-22T18:32:36.457550+0000 mon.a (mon.0) 5339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:36 smithi115 bash[46934]: cluster 2022-04-22T18:32:35.726479+0000 mgr.y (mgr.34107) 2314 : cluster [DBG] pgmap v3938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:36 smithi115 bash[46934]: audit 2022-04-22T18:32:36.457550+0000 mon.a (mon.0) 5339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:37 smithi121 bash[36902]: audit 2022-04-22T18:32:37.023444+0000 mon.a (mon.0) 5340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:37 smithi115 bash[45495]: audit 2022-04-22T18:32:37.023444+0000 mon.a (mon.0) 5340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:37 smithi115 bash[46934]: audit 2022-04-22T18:32:37.023444+0000 mon.a (mon.0) 5340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:38 smithi121 bash[36902]: cluster 2022-04-22T18:32:37.727631+0000 mgr.y (mgr.34107) 2315 : cluster [DBG] pgmap v3939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:39.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:38 smithi115 bash[45495]: cluster 2022-04-22T18:32:37.727631+0000 mgr.y (mgr.34107) 2315 : cluster [DBG] pgmap v3939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:38 smithi115 bash[46934]: cluster 2022-04-22T18:32:37.727631+0000 mgr.y (mgr.34107) 2315 : cluster [DBG] pgmap v3939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:40.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:39 smithi121 bash[36902]: cluster 2022-04-22T18:32:39.728794+0000 mgr.y (mgr.34107) 2316 : cluster [DBG] pgmap v3940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:39 smithi115 bash[45495]: cluster 2022-04-22T18:32:39.728794+0000 mgr.y (mgr.34107) 2316 : cluster [DBG] pgmap v3940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:39 smithi115 bash[46934]: cluster 2022-04-22T18:32:39.728794+0000 mgr.y (mgr.34107) 2316 : cluster [DBG] pgmap v3940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:41.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:41 smithi115 bash[45495]: audit 2022-04-22T18:32:41.460544+0000 mon.a (mon.0) 5341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:41.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:41 smithi115 bash[46934]: audit 2022-04-22T18:32:41.460544+0000 mon.a (mon.0) 5341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:41 smithi121 bash[36902]: audit 2022-04-22T18:32:41.460544+0000 mon.a (mon.0) 5341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:42.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:41.855Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:42 smithi121 bash[36902]: cluster 2022-04-22T18:32:41.729920+0000 mgr.y (mgr.34107) 2317 : cluster [DBG] pgmap v3941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:42 smithi121 bash[36902]: audit 2022-04-22T18:32:42.026990+0000 mon.a (mon.0) 5342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:42 smithi115 bash[45495]: cluster 2022-04-22T18:32:41.729920+0000 mgr.y (mgr.34107) 2317 : cluster [DBG] pgmap v3941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:42 smithi115 bash[45495]: audit 2022-04-22T18:32:42.026990+0000 mon.a (mon.0) 5342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:42 smithi115 bash[46934]: cluster 2022-04-22T18:32:41.729920+0000 mgr.y (mgr.34107) 2317 : cluster [DBG] pgmap v3941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:42 smithi115 bash[46934]: audit 2022-04-22T18:32:42.026990+0000 mon.a (mon.0) 5342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:44.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:43.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:45.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:44 smithi115 bash[45495]: cluster 2022-04-22T18:32:43.731088+0000 mgr.y (mgr.34107) 2318 : cluster [DBG] pgmap v3942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:45.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:44 smithi115 bash[46934]: cluster 2022-04-22T18:32:43.731088+0000 mgr.y (mgr.34107) 2318 : cluster [DBG] pgmap v3942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:45.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:44 smithi121 bash[36902]: cluster 2022-04-22T18:32:43.731088+0000 mgr.y (mgr.34107) 2318 : cluster [DBG] pgmap v3942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:46 smithi121 bash[36902]: cluster 2022-04-22T18:32:45.732131+0000 mgr.y (mgr.34107) 2319 : cluster [DBG] pgmap v3943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:46 smithi121 bash[36902]: audit 2022-04-22T18:32:46.463704+0000 mon.a (mon.0) 5343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:46 smithi115 bash[45495]: cluster 2022-04-22T18:32:45.732131+0000 mgr.y (mgr.34107) 2319 : cluster [DBG] pgmap v3943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:46 smithi115 bash[45495]: audit 2022-04-22T18:32:46.463704+0000 mon.a (mon.0) 5343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:46 smithi115 bash[46934]: cluster 2022-04-22T18:32:45.732131+0000 mgr.y (mgr.34107) 2319 : cluster [DBG] pgmap v3943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:46 smithi115 bash[46934]: audit 2022-04-22T18:32:46.463704+0000 mon.a (mon.0) 5343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:47 smithi121 bash[36902]: audit 2022-04-22T18:32:47.030741+0000 mon.a (mon.0) 5344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:47 smithi115 bash[45495]: audit 2022-04-22T18:32:47.030741+0000 mon.a (mon.0) 5344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:47 smithi115 bash[46934]: audit 2022-04-22T18:32:47.030741+0000 mon.a (mon.0) 5344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:48 smithi121 bash[36902]: cluster 2022-04-22T18:32:47.733450+0000 mgr.y (mgr.34107) 2320 : cluster [DBG] pgmap v3944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:48 smithi115 bash[45495]: cluster 2022-04-22T18:32:47.733450+0000 mgr.y (mgr.34107) 2320 : cluster [DBG] pgmap v3944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:49.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:48 smithi115 bash[46934]: cluster 2022-04-22T18:32:47.733450+0000 mgr.y (mgr.34107) 2320 : cluster [DBG] pgmap v3944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:50 smithi121 bash[36902]: cluster 2022-04-22T18:32:49.734633+0000 mgr.y (mgr.34107) 2321 : cluster [DBG] pgmap v3945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:51.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:50 smithi115 bash[45495]: cluster 2022-04-22T18:32:49.734633+0000 mgr.y (mgr.34107) 2321 : cluster [DBG] pgmap v3945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:50 smithi115 bash[46934]: cluster 2022-04-22T18:32:49.734633+0000 mgr.y (mgr.34107) 2321 : cluster [DBG] pgmap v3945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:51 smithi121 bash[36902]: audit 2022-04-22T18:32:51.466677+0000 mon.a (mon.0) 5345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:51 smithi115 bash[45495]: audit 2022-04-22T18:32:51.466677+0000 mon.a (mon.0) 5345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:51.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:52.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:51 smithi115 bash[46934]: audit 2022-04-22T18:32:51.466677+0000 mon.a (mon.0) 5345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:52 smithi121 bash[36902]: cluster 2022-04-22T18:32:51.735719+0000 mgr.y (mgr.34107) 2322 : cluster [DBG] pgmap v3946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:52 smithi121 bash[36902]: audit 2022-04-22T18:32:52.034514+0000 mon.a (mon.0) 5346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:52 smithi115 bash[45495]: cluster 2022-04-22T18:32:51.735719+0000 mgr.y (mgr.34107) 2322 : cluster [DBG] pgmap v3946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:52 smithi115 bash[45495]: audit 2022-04-22T18:32:52.034514+0000 mon.a (mon.0) 5346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:52 smithi115 bash[46934]: cluster 2022-04-22T18:32:51.735719+0000 mgr.y (mgr.34107) 2322 : cluster [DBG] pgmap v3946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:52 smithi115 bash[46934]: audit 2022-04-22T18:32:52.034514+0000 mon.a (mon.0) 5346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:53 smithi115 bash[45495]: cluster 2022-04-22T18:32:53.736699+0000 mgr.y (mgr.34107) 2323 : cluster [DBG] pgmap v3947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:32:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:32:53.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:32:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:53 smithi115 bash[46934]: cluster 2022-04-22T18:32:53.736699+0000 mgr.y (mgr.34107) 2323 : cluster [DBG] pgmap v3947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:53 smithi121 bash[36902]: cluster 2022-04-22T18:32:53.736699+0000 mgr.y (mgr.34107) 2323 : cluster [DBG] pgmap v3947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:56 smithi121 bash[36902]: cluster 2022-04-22T18:32:55.737737+0000 mgr.y (mgr.34107) 2324 : cluster [DBG] pgmap v3948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:56 smithi121 bash[36902]: audit 2022-04-22T18:32:56.469813+0000 mon.a (mon.0) 5347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:56 smithi115 bash[45495]: cluster 2022-04-22T18:32:55.737737+0000 mgr.y (mgr.34107) 2324 : cluster [DBG] pgmap v3948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:56 smithi115 bash[45495]: audit 2022-04-22T18:32:56.469813+0000 mon.a (mon.0) 5347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:56 smithi115 bash[46934]: cluster 2022-04-22T18:32:55.737737+0000 mgr.y (mgr.34107) 2324 : cluster [DBG] pgmap v3948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:56 smithi115 bash[46934]: audit 2022-04-22T18:32:56.469813+0000 mon.a (mon.0) 5347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:32:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:57 smithi121 bash[36902]: audit 2022-04-22T18:32:57.038099+0000 mon.a (mon.0) 5348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:57 smithi115 bash[45495]: audit 2022-04-22T18:32:57.038099+0000 mon.a (mon.0) 5348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:57 smithi115 bash[46934]: audit 2022-04-22T18:32:57.038099+0000 mon.a (mon.0) 5348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:32:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:32:58 smithi121 bash[36902]: cluster 2022-04-22T18:32:57.738804+0000 mgr.y (mgr.34107) 2325 : cluster [DBG] pgmap v3949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:32:58 smithi115 bash[45495]: cluster 2022-04-22T18:32:57.738804+0000 mgr.y (mgr.34107) 2325 : cluster [DBG] pgmap v3949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:32:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:32:58 smithi115 bash[46934]: cluster 2022-04-22T18:32:57.738804+0000 mgr.y (mgr.34107) 2325 : cluster [DBG] pgmap v3949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:00 smithi121 bash[36902]: cluster 2022-04-22T18:32:59.739935+0000 mgr.y (mgr.34107) 2326 : cluster [DBG] pgmap v3950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:01.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:00 smithi115 bash[45495]: cluster 2022-04-22T18:32:59.739935+0000 mgr.y (mgr.34107) 2326 : cluster [DBG] pgmap v3950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:00 smithi115 bash[46934]: cluster 2022-04-22T18:32:59.739935+0000 mgr.y (mgr.34107) 2326 : cluster [DBG] pgmap v3950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:01 smithi121 bash[36902]: audit 2022-04-22T18:33:01.472896+0000 mon.a (mon.0) 5349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:02.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:01 smithi115 bash[45495]: audit 2022-04-22T18:33:01.472896+0000 mon.a (mon.0) 5349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:01.855Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:01 smithi115 bash[46934]: audit 2022-04-22T18:33:01.472896+0000 mon.a (mon.0) 5349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:03.052 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:33:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:02 smithi121 bash[36902]: cluster 2022-04-22T18:33:01.740980+0000 mgr.y (mgr.34107) 2327 : cluster [DBG] pgmap v3951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:02 smithi121 bash[36902]: audit 2022-04-22T18:33:02.041754+0000 mon.a (mon.0) 5350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:03.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:02 smithi115 bash[45495]: cluster 2022-04-22T18:33:01.740980+0000 mgr.y (mgr.34107) 2327 : cluster [DBG] pgmap v3951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:02 smithi115 bash[45495]: audit 2022-04-22T18:33:02.041754+0000 mon.a (mon.0) 5350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:02 smithi115 bash[46934]: cluster 2022-04-22T18:33:01.740980+0000 mgr.y (mgr.34107) 2327 : cluster [DBG] pgmap v3951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:02 smithi115 bash[46934]: audit 2022-04-22T18:33:02.041754+0000 mon.a (mon.0) 5350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:03.395 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:33:03.407 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (56m) 3m ago 69m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:33:03.407 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (69m) 4m ago 69m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:33:03.407 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (58m) 3m ago 70m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:33:03.407 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (57m) 4m ago 70m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:33:03.407 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (65m) 4m ago 75m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (66m) 3m ago 77m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (65m) 3m ago 77m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (64m) 4m ago 75m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (64m) 3m ago 75m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (57m) 3m ago 70m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (57m) 4m ago 70m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (62m) 3m ago 74m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (62m) 3m ago 74m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (61m) 3m ago 73m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:33:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (61m) 3m ago 73m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:33:03.409 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (60m) 4m ago 72m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:33:03.409 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (59m) 4m ago 72m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:33:03.409 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (59m) 4m ago 71m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:33:03.409 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (58m) 4m ago 71m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:33:03.409 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (56m) 4m ago 70m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:33:03.796 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:33:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:33:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:33:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:03.906 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:33:03.907 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:33:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:03 smithi115 bash[45495]: audit 2022-04-22T18:33:03.788601+0000 mon.a (mon.0) 5351 : audit [DBG] from='client.? 172.21.15.115:0/3671071822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:03.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:03 smithi115 bash[46934]: audit 2022-04-22T18:33:03.788601+0000 mon.a (mon.0) 5351 : audit [DBG] from='client.? 172.21.15.115:0/3671071822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:04.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:03 smithi121 bash[36902]: audit 2022-04-22T18:33:03.788601+0000 mon.a (mon.0) 5351 : audit [DBG] from='client.? 172.21.15.115:0/3671071822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:04 smithi121 bash[36902]: audit 2022-04-22T18:33:03.029726+0000 mgr.y (mgr.34107) 2328 : audit [DBG] from='client.47806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:04 smithi121 bash[36902]: audit 2022-04-22T18:33:03.367970+0000 mgr.y (mgr.34107) 2329 : audit [DBG] from='client.47812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:04 smithi121 bash[36902]: cluster 2022-04-22T18:33:03.742136+0000 mgr.y (mgr.34107) 2330 : cluster [DBG] pgmap v3952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:05.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[45495]: audit 2022-04-22T18:33:03.029726+0000 mgr.y (mgr.34107) 2328 : audit [DBG] from='client.47806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[45495]: audit 2022-04-22T18:33:03.367970+0000 mgr.y (mgr.34107) 2329 : audit [DBG] from='client.47812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[45495]: cluster 2022-04-22T18:33:03.742136+0000 mgr.y (mgr.34107) 2330 : cluster [DBG] pgmap v3952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:05.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[46934]: audit 2022-04-22T18:33:03.029726+0000 mgr.y (mgr.34107) 2328 : audit [DBG] from='client.47806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[46934]: audit 2022-04-22T18:33:03.367970+0000 mgr.y (mgr.34107) 2329 : audit [DBG] from='client.47812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:04 smithi115 bash[46934]: cluster 2022-04-22T18:33:03.742136+0000 mgr.y (mgr.34107) 2330 : cluster [DBG] pgmap v3952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:05 smithi121 bash[36902]: audit 2022-04-22T18:33:05.487985+0000 mon.a (mon.0) 5352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:33:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:05 smithi121 bash[36902]: audit 2022-04-22T18:33:05.490997+0000 mon.a (mon.0) 5353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:33:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:05 smithi121 bash[36902]: audit 2022-04-22T18:33:05.492796+0000 mon.a (mon.0) 5354 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:33:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:05 smithi121 bash[36902]: cluster 2022-04-22T18:33:05.743070+0000 mgr.y (mgr.34107) 2331 : cluster [DBG] pgmap v3953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:06.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[45495]: audit 2022-04-22T18:33:05.487985+0000 mon.a (mon.0) 5352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:33:06.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[45495]: audit 2022-04-22T18:33:05.490997+0000 mon.a (mon.0) 5353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:33:06.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[45495]: audit 2022-04-22T18:33:05.492796+0000 mon.a (mon.0) 5354 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:33:06.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[45495]: cluster 2022-04-22T18:33:05.743070+0000 mgr.y (mgr.34107) 2331 : cluster [DBG] pgmap v3953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:06.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[46934]: audit 2022-04-22T18:33:05.487985+0000 mon.a (mon.0) 5352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:33:06.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[46934]: audit 2022-04-22T18:33:05.490997+0000 mon.a (mon.0) 5353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:33:06.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[46934]: audit 2022-04-22T18:33:05.492796+0000 mon.a (mon.0) 5354 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:33:06.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:05 smithi115 bash[46934]: cluster 2022-04-22T18:33:05.743070+0000 mgr.y (mgr.34107) 2331 : cluster [DBG] pgmap v3953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:06 smithi121 bash[36902]: audit 2022-04-22T18:33:05.847229+0000 mon.a (mon.0) 5355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:33:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:06 smithi121 bash[36902]: audit 2022-04-22T18:33:06.476082+0000 mon.a (mon.0) 5356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:07.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:06 smithi115 bash[45495]: audit 2022-04-22T18:33:05.847229+0000 mon.a (mon.0) 5355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:33:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:06 smithi115 bash[45495]: audit 2022-04-22T18:33:06.476082+0000 mon.a (mon.0) 5356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:06 smithi115 bash[46934]: audit 2022-04-22T18:33:05.847229+0000 mon.a (mon.0) 5355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:33:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:06 smithi115 bash[46934]: audit 2022-04-22T18:33:06.476082+0000 mon.a (mon.0) 5356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:07 smithi121 bash[36902]: audit 2022-04-22T18:33:07.045430+0000 mon.a (mon.0) 5357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:07 smithi121 bash[36902]: cluster 2022-04-22T18:33:07.744188+0000 mgr.y (mgr.34107) 2332 : cluster [DBG] pgmap v3954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:07 smithi115 bash[45495]: audit 2022-04-22T18:33:07.045430+0000 mon.a (mon.0) 5357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:07 smithi115 bash[45495]: cluster 2022-04-22T18:33:07.744188+0000 mgr.y (mgr.34107) 2332 : cluster [DBG] pgmap v3954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:07 smithi115 bash[46934]: audit 2022-04-22T18:33:07.045430+0000 mon.a (mon.0) 5357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:07 smithi115 bash[46934]: cluster 2022-04-22T18:33:07.744188+0000 mgr.y (mgr.34107) 2332 : cluster [DBG] pgmap v3954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:10 smithi121 bash[36902]: cluster 2022-04-22T18:33:09.745303+0000 mgr.y (mgr.34107) 2333 : cluster [DBG] pgmap v3955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:11.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:10 smithi115 bash[45495]: cluster 2022-04-22T18:33:09.745303+0000 mgr.y (mgr.34107) 2333 : cluster [DBG] pgmap v3955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:10 smithi115 bash[46934]: cluster 2022-04-22T18:33:09.745303+0000 mgr.y (mgr.34107) 2333 : cluster [DBG] pgmap v3955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:11 smithi121 bash[36902]: audit 2022-04-22T18:33:11.479118+0000 mon.a (mon.0) 5358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:11 smithi115 bash[45495]: audit 2022-04-22T18:33:11.479118+0000 mon.a (mon.0) 5358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:11 smithi115 bash[46934]: audit 2022-04-22T18:33:11.479118+0000 mon.a (mon.0) 5358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:11.856Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:12 smithi121 bash[36902]: cluster 2022-04-22T18:33:11.746427+0000 mgr.y (mgr.34107) 2334 : cluster [DBG] pgmap v3956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:12 smithi121 bash[36902]: audit 2022-04-22T18:33:12.049120+0000 mon.a (mon.0) 5359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:12 smithi115 bash[45495]: cluster 2022-04-22T18:33:11.746427+0000 mgr.y (mgr.34107) 2334 : cluster [DBG] pgmap v3956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:12 smithi115 bash[45495]: audit 2022-04-22T18:33:12.049120+0000 mon.a (mon.0) 5359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:12 smithi115 bash[46934]: cluster 2022-04-22T18:33:11.746427+0000 mgr.y (mgr.34107) 2334 : cluster [DBG] pgmap v3956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:12 smithi115 bash[46934]: audit 2022-04-22T18:33:12.049120+0000 mon.a (mon.0) 5359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:14.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:13.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:14 smithi121 bash[36902]: cluster 2022-04-22T18:33:13.747560+0000 mgr.y (mgr.34107) 2335 : cluster [DBG] pgmap v3957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:14 smithi121 bash[36902]: audit 2022-04-22T18:33:14.049935+0000 mon.a (mon.0) 5360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:33:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:14 smithi121 bash[36902]: audit 2022-04-22T18:33:14.053281+0000 mon.a (mon.0) 5361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:33:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[45495]: cluster 2022-04-22T18:33:13.747560+0000 mgr.y (mgr.34107) 2335 : cluster [DBG] pgmap v3957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[45495]: audit 2022-04-22T18:33:14.049935+0000 mon.a (mon.0) 5360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:33:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[45495]: audit 2022-04-22T18:33:14.053281+0000 mon.a (mon.0) 5361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:33:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[46934]: cluster 2022-04-22T18:33:13.747560+0000 mgr.y (mgr.34107) 2335 : cluster [DBG] pgmap v3957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[46934]: audit 2022-04-22T18:33:14.049935+0000 mon.a (mon.0) 5360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:33:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:14 smithi115 bash[46934]: audit 2022-04-22T18:33:14.053281+0000 mon.a (mon.0) 5361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:33:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:15 smithi121 bash[36902]: cluster 2022-04-22T18:33:15.748525+0000 mgr.y (mgr.34107) 2336 : cluster [DBG] pgmap v3958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:15 smithi115 bash[46934]: cluster 2022-04-22T18:33:15.748525+0000 mgr.y (mgr.34107) 2336 : cluster [DBG] pgmap v3958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:16.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:15 smithi115 bash[45495]: cluster 2022-04-22T18:33:15.748525+0000 mgr.y (mgr.34107) 2336 : cluster [DBG] pgmap v3958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:16 smithi121 bash[36902]: audit 2022-04-22T18:33:16.482181+0000 mon.a (mon.0) 5362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:17.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:16 smithi115 bash[45495]: audit 2022-04-22T18:33:16.482181+0000 mon.a (mon.0) 5362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:17.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:16 smithi115 bash[46934]: audit 2022-04-22T18:33:16.482181+0000 mon.a (mon.0) 5362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:17 smithi121 bash[36902]: audit 2022-04-22T18:33:17.052349+0000 mon.a (mon.0) 5363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:17 smithi121 bash[36902]: cluster 2022-04-22T18:33:17.749538+0000 mgr.y (mgr.34107) 2337 : cluster [DBG] pgmap v3959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:17 smithi115 bash[45495]: audit 2022-04-22T18:33:17.052349+0000 mon.a (mon.0) 5363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:17 smithi115 bash[45495]: cluster 2022-04-22T18:33:17.749538+0000 mgr.y (mgr.34107) 2337 : cluster [DBG] pgmap v3959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:17 smithi115 bash[46934]: audit 2022-04-22T18:33:17.052349+0000 mon.a (mon.0) 5363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:17 smithi115 bash[46934]: cluster 2022-04-22T18:33:17.749538+0000 mgr.y (mgr.34107) 2337 : cluster [DBG] pgmap v3959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:21.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:20 smithi121 bash[36902]: cluster 2022-04-22T18:33:19.750762+0000 mgr.y (mgr.34107) 2338 : cluster [DBG] pgmap v3960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:21.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:20 smithi115 bash[45495]: cluster 2022-04-22T18:33:19.750762+0000 mgr.y (mgr.34107) 2338 : cluster [DBG] pgmap v3960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:21.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:20 smithi115 bash[46934]: cluster 2022-04-22T18:33:19.750762+0000 mgr.y (mgr.34107) 2338 : cluster [DBG] pgmap v3960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:21 smithi121 bash[36902]: audit 2022-04-22T18:33:21.485223+0000 mon.a (mon.0) 5364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:22.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:21 smithi115 bash[45495]: audit 2022-04-22T18:33:21.485223+0000 mon.a (mon.0) 5364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:21.857Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:22.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:21 smithi115 bash[46934]: audit 2022-04-22T18:33:21.485223+0000 mon.a (mon.0) 5364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:22 smithi121 bash[36902]: cluster 2022-04-22T18:33:21.751683+0000 mgr.y (mgr.34107) 2339 : cluster [DBG] pgmap v3961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:22 smithi121 bash[36902]: audit 2022-04-22T18:33:22.055634+0000 mon.a (mon.0) 5365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:23.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:22 smithi115 bash[45495]: cluster 2022-04-22T18:33:21.751683+0000 mgr.y (mgr.34107) 2339 : cluster [DBG] pgmap v3961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:22 smithi115 bash[45495]: audit 2022-04-22T18:33:22.055634+0000 mon.a (mon.0) 5365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:22 smithi115 bash[46934]: cluster 2022-04-22T18:33:21.751683+0000 mgr.y (mgr.34107) 2339 : cluster [DBG] pgmap v3961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:22 smithi115 bash[46934]: audit 2022-04-22T18:33:22.055634+0000 mon.a (mon.0) 5365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:24.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:23.736Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:24 smithi121 bash[36902]: cluster 2022-04-22T18:33:23.752728+0000 mgr.y (mgr.34107) 2340 : cluster [DBG] pgmap v3962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:25.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:24 smithi115 bash[45495]: cluster 2022-04-22T18:33:23.752728+0000 mgr.y (mgr.34107) 2340 : cluster [DBG] pgmap v3962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:25.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:24 smithi115 bash[46934]: cluster 2022-04-22T18:33:23.752728+0000 mgr.y (mgr.34107) 2340 : cluster [DBG] pgmap v3962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:25 smithi121 bash[36902]: cluster 2022-04-22T18:33:25.753722+0000 mgr.y (mgr.34107) 2341 : cluster [DBG] pgmap v3963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:25 smithi115 bash[45495]: cluster 2022-04-22T18:33:25.753722+0000 mgr.y (mgr.34107) 2341 : cluster [DBG] pgmap v3963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:25 smithi115 bash[46934]: cluster 2022-04-22T18:33:25.753722+0000 mgr.y (mgr.34107) 2341 : cluster [DBG] pgmap v3963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:26 smithi121 bash[36902]: audit 2022-04-22T18:33:26.488545+0000 mon.a (mon.0) 5366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:26 smithi115 bash[45495]: audit 2022-04-22T18:33:26.488545+0000 mon.a (mon.0) 5366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:26 smithi115 bash[46934]: audit 2022-04-22T18:33:26.488545+0000 mon.a (mon.0) 5366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:27 smithi121 bash[36902]: audit 2022-04-22T18:33:27.059841+0000 mon.a (mon.0) 5367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:27 smithi121 bash[36902]: cluster 2022-04-22T18:33:27.754783+0000 mgr.y (mgr.34107) 2342 : cluster [DBG] pgmap v3964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:28.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:27 smithi115 bash[45495]: audit 2022-04-22T18:33:27.059841+0000 mon.a (mon.0) 5367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:27 smithi115 bash[45495]: cluster 2022-04-22T18:33:27.754783+0000 mgr.y (mgr.34107) 2342 : cluster [DBG] pgmap v3964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:27 smithi115 bash[46934]: audit 2022-04-22T18:33:27.059841+0000 mon.a (mon.0) 5367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:27 smithi115 bash[46934]: cluster 2022-04-22T18:33:27.754783+0000 mgr.y (mgr.34107) 2342 : cluster [DBG] pgmap v3964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:30 smithi121 bash[36902]: cluster 2022-04-22T18:33:29.755854+0000 mgr.y (mgr.34107) 2343 : cluster [DBG] pgmap v3965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:30 smithi115 bash[45495]: cluster 2022-04-22T18:33:29.755854+0000 mgr.y (mgr.34107) 2343 : cluster [DBG] pgmap v3965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:30 smithi115 bash[46934]: cluster 2022-04-22T18:33:29.755854+0000 mgr.y (mgr.34107) 2343 : cluster [DBG] pgmap v3965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:31 smithi121 bash[36902]: audit 2022-04-22T18:33:31.491691+0000 mon.a (mon.0) 5368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:31 smithi115 bash[45495]: audit 2022-04-22T18:33:31.491691+0000 mon.a (mon.0) 5368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:31.858Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:31 smithi115 bash[46934]: audit 2022-04-22T18:33:31.491691+0000 mon.a (mon.0) 5368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:32 smithi121 bash[36902]: cluster 2022-04-22T18:33:31.756824+0000 mgr.y (mgr.34107) 2344 : cluster [DBG] pgmap v3966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:32 smithi121 bash[36902]: audit 2022-04-22T18:33:32.063417+0000 mon.a (mon.0) 5369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:32 smithi115 bash[45495]: cluster 2022-04-22T18:33:31.756824+0000 mgr.y (mgr.34107) 2344 : cluster [DBG] pgmap v3966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:32 smithi115 bash[45495]: audit 2022-04-22T18:33:32.063417+0000 mon.a (mon.0) 5369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:32 smithi115 bash[46934]: cluster 2022-04-22T18:33:31.756824+0000 mgr.y (mgr.34107) 2344 : cluster [DBG] pgmap v3966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:32 smithi115 bash[46934]: audit 2022-04-22T18:33:32.063417+0000 mon.a (mon.0) 5369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:33 smithi115 bash[45495]: cluster 2022-04-22T18:33:33.758038+0000 mgr.y (mgr.34107) 2345 : cluster [DBG] pgmap v3967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:33.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:34.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:33 smithi115 bash[46934]: cluster 2022-04-22T18:33:33.758038+0000 mgr.y (mgr.34107) 2345 : cluster [DBG] pgmap v3967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:34.181 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:33:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:33 smithi121 bash[36902]: cluster 2022-04-22T18:33:33.758038+0000 mgr.y (mgr.34107) 2345 : cluster [DBG] pgmap v3967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (56m) 4m ago 70m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (70m) 5m ago 70m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (58m) 4m ago 71m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (58m) 5m ago 71m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:33:34.520 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (65m) 5m ago 75m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (66m) 4m ago 77m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (65m) 4m ago 78m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (64m) 5m ago 76m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (64m) 4m ago 76m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (57m) 4m ago 70m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (57m) 5m ago 70m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (63m) 4m ago 75m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (62m) 4m ago 74m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (62m) 4m ago 74m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:33:34.521 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (61m) 4m ago 73m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:33:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (61m) 5m ago 73m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:33:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (60m) 5m ago 72m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:33:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (59m) 5m ago 72m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:33:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (59m) 5m ago 72m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:33:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (57m) 5m ago 70m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:33:34.924 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:33:34.924 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:33:34.925 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:33:34.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:33:34.926 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:33:34.926 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:33:34.926 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:33:34.926 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:33:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:34 smithi121 bash[36902]: audit 2022-04-22T18:33:34.150942+0000 mgr.y (mgr.34107) 2346 : audit [DBG] from='client.37890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:34 smithi121 bash[36902]: audit 2022-04-22T18:33:34.492699+0000 mgr.y (mgr.34107) 2347 : audit [DBG] from='client.37896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:35.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:34 smithi115 bash[45495]: audit 2022-04-22T18:33:34.150942+0000 mgr.y (mgr.34107) 2346 : audit [DBG] from='client.37890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:35.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:34 smithi115 bash[45495]: audit 2022-04-22T18:33:34.492699+0000 mgr.y (mgr.34107) 2347 : audit [DBG] from='client.37896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:34 smithi115 bash[46934]: audit 2022-04-22T18:33:34.150942+0000 mgr.y (mgr.34107) 2346 : audit [DBG] from='client.37890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:34 smithi115 bash[46934]: audit 2022-04-22T18:33:34.492699+0000 mgr.y (mgr.34107) 2347 : audit [DBG] from='client.37896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:33:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:35 smithi121 bash[36902]: audit 2022-04-22T18:33:34.916907+0000 mon.c (mon.1) 147 : audit [DBG] from='client.? 172.21.15.115:0/1446158194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:35 smithi121 bash[36902]: cluster 2022-04-22T18:33:35.759194+0000 mgr.y (mgr.34107) 2348 : cluster [DBG] pgmap v3968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:35 smithi115 bash[45495]: audit 2022-04-22T18:33:34.916907+0000 mon.c (mon.1) 147 : audit [DBG] from='client.? 172.21.15.115:0/1446158194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:35 smithi115 bash[45495]: cluster 2022-04-22T18:33:35.759194+0000 mgr.y (mgr.34107) 2348 : cluster [DBG] pgmap v3968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:35 smithi115 bash[46934]: audit 2022-04-22T18:33:34.916907+0000 mon.c (mon.1) 147 : audit [DBG] from='client.? 172.21.15.115:0/1446158194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:33:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:35 smithi115 bash[46934]: cluster 2022-04-22T18:33:35.759194+0000 mgr.y (mgr.34107) 2348 : cluster [DBG] pgmap v3968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:36 smithi121 bash[36902]: audit 2022-04-22T18:33:36.494697+0000 mon.a (mon.0) 5370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:36 smithi115 bash[45495]: audit 2022-04-22T18:33:36.494697+0000 mon.a (mon.0) 5370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:36 smithi115 bash[46934]: audit 2022-04-22T18:33:36.494697+0000 mon.a (mon.0) 5370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:37 smithi121 bash[36902]: audit 2022-04-22T18:33:37.066949+0000 mon.a (mon.0) 5371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:37 smithi121 bash[36902]: cluster 2022-04-22T18:33:37.760330+0000 mgr.y (mgr.34107) 2349 : cluster [DBG] pgmap v3969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:38.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:37 smithi115 bash[45495]: audit 2022-04-22T18:33:37.066949+0000 mon.a (mon.0) 5371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:38.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:37 smithi115 bash[45495]: cluster 2022-04-22T18:33:37.760330+0000 mgr.y (mgr.34107) 2349 : cluster [DBG] pgmap v3969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:37 smithi115 bash[46934]: audit 2022-04-22T18:33:37.066949+0000 mon.a (mon.0) 5371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:37 smithi115 bash[46934]: cluster 2022-04-22T18:33:37.760330+0000 mgr.y (mgr.34107) 2349 : cluster [DBG] pgmap v3969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:40 smithi121 bash[36902]: cluster 2022-04-22T18:33:39.761455+0000 mgr.y (mgr.34107) 2350 : cluster [DBG] pgmap v3970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:41.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:40 smithi115 bash[45495]: cluster 2022-04-22T18:33:39.761455+0000 mgr.y (mgr.34107) 2350 : cluster [DBG] pgmap v3970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:41.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:40 smithi115 bash[46934]: cluster 2022-04-22T18:33:39.761455+0000 mgr.y (mgr.34107) 2350 : cluster [DBG] pgmap v3970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:41 smithi121 bash[36902]: audit 2022-04-22T18:33:41.497708+0000 mon.a (mon.0) 5372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:42.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:41 smithi115 bash[46934]: audit 2022-04-22T18:33:41.497708+0000 mon.a (mon.0) 5372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:42.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:41.858Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:41 smithi115 bash[45495]: audit 2022-04-22T18:33:41.497708+0000 mon.a (mon.0) 5372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:42 smithi121 bash[36902]: cluster 2022-04-22T18:33:41.762535+0000 mgr.y (mgr.34107) 2351 : cluster [DBG] pgmap v3971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:42 smithi121 bash[36902]: audit 2022-04-22T18:33:42.070849+0000 mon.a (mon.0) 5373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:42 smithi115 bash[45495]: cluster 2022-04-22T18:33:41.762535+0000 mgr.y (mgr.34107) 2351 : cluster [DBG] pgmap v3971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:42 smithi115 bash[45495]: audit 2022-04-22T18:33:42.070849+0000 mon.a (mon.0) 5373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:42 smithi115 bash[46934]: cluster 2022-04-22T18:33:41.762535+0000 mgr.y (mgr.34107) 2351 : cluster [DBG] pgmap v3971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:42 smithi115 bash[46934]: audit 2022-04-22T18:33:42.070849+0000 mon.a (mon.0) 5373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:43 smithi115 bash[45495]: cluster 2022-04-22T18:33:43.763705+0000 mgr.y (mgr.34107) 2352 : cluster [DBG] pgmap v3972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:44.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:43.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:44.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:43 smithi115 bash[46934]: cluster 2022-04-22T18:33:43.763705+0000 mgr.y (mgr.34107) 2352 : cluster [DBG] pgmap v3972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:43 smithi121 bash[36902]: cluster 2022-04-22T18:33:43.763705+0000 mgr.y (mgr.34107) 2352 : cluster [DBG] pgmap v3972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:46 smithi121 bash[36902]: cluster 2022-04-22T18:33:45.764682+0000 mgr.y (mgr.34107) 2353 : cluster [DBG] pgmap v3973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:46 smithi121 bash[36902]: audit 2022-04-22T18:33:46.500845+0000 mon.a (mon.0) 5374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:46 smithi115 bash[45495]: cluster 2022-04-22T18:33:45.764682+0000 mgr.y (mgr.34107) 2353 : cluster [DBG] pgmap v3973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:46 smithi115 bash[45495]: audit 2022-04-22T18:33:46.500845+0000 mon.a (mon.0) 5374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:46 smithi115 bash[46934]: cluster 2022-04-22T18:33:45.764682+0000 mgr.y (mgr.34107) 2353 : cluster [DBG] pgmap v3973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:46 smithi115 bash[46934]: audit 2022-04-22T18:33:46.500845+0000 mon.a (mon.0) 5374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:47 smithi121 bash[36902]: audit 2022-04-22T18:33:47.074327+0000 mon.a (mon.0) 5375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:47 smithi121 bash[36902]: cluster 2022-04-22T18:33:47.765806+0000 mgr.y (mgr.34107) 2354 : cluster [DBG] pgmap v3974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:47 smithi115 bash[45495]: audit 2022-04-22T18:33:47.074327+0000 mon.a (mon.0) 5375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:47 smithi115 bash[45495]: cluster 2022-04-22T18:33:47.765806+0000 mgr.y (mgr.34107) 2354 : cluster [DBG] pgmap v3974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:47 smithi115 bash[46934]: audit 2022-04-22T18:33:47.074327+0000 mon.a (mon.0) 5375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:47 smithi115 bash[46934]: cluster 2022-04-22T18:33:47.765806+0000 mgr.y (mgr.34107) 2354 : cluster [DBG] pgmap v3974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:50.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:49 smithi121 bash[36902]: cluster 2022-04-22T18:33:49.766850+0000 mgr.y (mgr.34107) 2355 : cluster [DBG] pgmap v3975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:49 smithi115 bash[45495]: cluster 2022-04-22T18:33:49.766850+0000 mgr.y (mgr.34107) 2355 : cluster [DBG] pgmap v3975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:50.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:49 smithi115 bash[46934]: cluster 2022-04-22T18:33:49.766850+0000 mgr.y (mgr.34107) 2355 : cluster [DBG] pgmap v3975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:51.867 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:51 smithi115 bash[45495]: audit 2022-04-22T18:33:51.504100+0000 mon.a (mon.0) 5376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:51.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:51 smithi115 bash[46934]: audit 2022-04-22T18:33:51.504100+0000 mon.a (mon.0) 5376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:51 smithi121 bash[36902]: audit 2022-04-22T18:33:51.504100+0000 mon.a (mon.0) 5376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:51.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:52 smithi121 bash[36902]: cluster 2022-04-22T18:33:51.767910+0000 mgr.y (mgr.34107) 2356 : cluster [DBG] pgmap v3976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:52 smithi121 bash[36902]: audit 2022-04-22T18:33:52.078143+0000 mon.a (mon.0) 5377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:52 smithi115 bash[45495]: cluster 2022-04-22T18:33:51.767910+0000 mgr.y (mgr.34107) 2356 : cluster [DBG] pgmap v3976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:52 smithi115 bash[45495]: audit 2022-04-22T18:33:52.078143+0000 mon.a (mon.0) 5377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:52 smithi115 bash[46934]: cluster 2022-04-22T18:33:51.767910+0000 mgr.y (mgr.34107) 2356 : cluster [DBG] pgmap v3976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:52 smithi115 bash[46934]: audit 2022-04-22T18:33:52.078143+0000 mon.a (mon.0) 5377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:53 smithi115 bash[45495]: cluster 2022-04-22T18:33:53.768981+0000 mgr.y (mgr.34107) 2357 : cluster [DBG] pgmap v3977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:33:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:33:53.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:33:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:53 smithi115 bash[46934]: cluster 2022-04-22T18:33:53.768981+0000 mgr.y (mgr.34107) 2357 : cluster [DBG] pgmap v3977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:53 smithi121 bash[36902]: cluster 2022-04-22T18:33:53.768981+0000 mgr.y (mgr.34107) 2357 : cluster [DBG] pgmap v3977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:55 smithi121 bash[36902]: cluster 2022-04-22T18:33:55.769952+0000 mgr.y (mgr.34107) 2358 : cluster [DBG] pgmap v3978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:55 smithi115 bash[45495]: cluster 2022-04-22T18:33:55.769952+0000 mgr.y (mgr.34107) 2358 : cluster [DBG] pgmap v3978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:55 smithi115 bash[46934]: cluster 2022-04-22T18:33:55.769952+0000 mgr.y (mgr.34107) 2358 : cluster [DBG] pgmap v3978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:57.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:56 smithi121 bash[36902]: audit 2022-04-22T18:33:56.507094+0000 mon.a (mon.0) 5378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:57.386 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:56 smithi115 bash[45495]: audit 2022-04-22T18:33:56.507094+0000 mon.a (mon.0) 5378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:57.411 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:56 smithi115 bash[46934]: audit 2022-04-22T18:33:56.507094+0000 mon.a (mon.0) 5378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:33:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:57 smithi121 bash[36902]: audit 2022-04-22T18:33:57.081563+0000 mon.a (mon.0) 5379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:57 smithi121 bash[36902]: cluster 2022-04-22T18:33:57.771121+0000 mgr.y (mgr.34107) 2359 : cluster [DBG] pgmap v3979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:57 smithi115 bash[45495]: audit 2022-04-22T18:33:57.081563+0000 mon.a (mon.0) 5379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:57 smithi115 bash[45495]: cluster 2022-04-22T18:33:57.771121+0000 mgr.y (mgr.34107) 2359 : cluster [DBG] pgmap v3979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:33:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:57 smithi115 bash[46934]: audit 2022-04-22T18:33:57.081563+0000 mon.a (mon.0) 5379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:33:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:57 smithi115 bash[46934]: cluster 2022-04-22T18:33:57.771121+0000 mgr.y (mgr.34107) 2359 : cluster [DBG] pgmap v3979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:33:59 smithi121 bash[36902]: cluster 2022-04-22T18:33:59.772284+0000 mgr.y (mgr.34107) 2360 : cluster [DBG] pgmap v3980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:33:59 smithi115 bash[45495]: cluster 2022-04-22T18:33:59.772284+0000 mgr.y (mgr.34107) 2360 : cluster [DBG] pgmap v3980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:33:59 smithi115 bash[46934]: cluster 2022-04-22T18:33:59.772284+0000 mgr.y (mgr.34107) 2360 : cluster [DBG] pgmap v3980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:01.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:01 smithi115 bash[45495]: audit 2022-04-22T18:34:01.510225+0000 mon.a (mon.0) 5380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:01.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:01 smithi115 bash[46934]: audit 2022-04-22T18:34:01.510225+0000 mon.a (mon.0) 5380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:01 smithi121 bash[36902]: audit 2022-04-22T18:34:01.510225+0000 mon.a (mon.0) 5380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:01.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:02 smithi121 bash[36902]: cluster 2022-04-22T18:34:01.773378+0000 mgr.y (mgr.34107) 2361 : cluster [DBG] pgmap v3981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:03.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:02 smithi121 bash[36902]: audit 2022-04-22T18:34:02.085303+0000 mon.a (mon.0) 5381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:03.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:02 smithi115 bash[45495]: cluster 2022-04-22T18:34:01.773378+0000 mgr.y (mgr.34107) 2361 : cluster [DBG] pgmap v3981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:03.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:02 smithi115 bash[45495]: audit 2022-04-22T18:34:02.085303+0000 mon.a (mon.0) 5381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:03.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:02 smithi115 bash[46934]: cluster 2022-04-22T18:34:01.773378+0000 mgr.y (mgr.34107) 2361 : cluster [DBG] pgmap v3981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:03.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:02 smithi115 bash[46934]: audit 2022-04-22T18:34:02.085303+0000 mon.a (mon.0) 5381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:04.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:03 smithi115 bash[45495]: cluster 2022-04-22T18:34:03.774588+0000 mgr.y (mgr.34107) 2362 : cluster [DBG] pgmap v3982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:03 smithi115 bash[46934]: cluster 2022-04-22T18:34:03.774588+0000 mgr.y (mgr.34107) 2362 : cluster [DBG] pgmap v3982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:03.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:03 smithi121 bash[36902]: cluster 2022-04-22T18:34:03.774588+0000 mgr.y (mgr.34107) 2362 : cluster [DBG] pgmap v3982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:05.294 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:34:05.631 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:34:05.632 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (57m) 4m ago 70m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:34:05.633 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (70m) 5m ago 70m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:34:05.633 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (59m) 4m ago 71m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:34:05.633 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (58m) 5m ago 71m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:34:05.633 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (66m) 5m ago 76m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:34:05.633 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (67m) 4m ago 78m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:34:05.634 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (66m) 4m ago 78m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:34:05.634 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (65m) 5m ago 76m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:34:05.634 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (65m) 4m ago 76m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:34:05.634 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (58m) 4m ago 71m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:34:05.635 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (58m) 5m ago 71m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:34:05.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (63m) 4m ago 75m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:34:05.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (63m) 4m ago 75m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:34:05.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (62m) 4m ago 74m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:34:05.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (62m) 4m ago 74m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:34:05.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (61m) 5m ago 73m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:34:05.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (60m) 5m ago 73m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:34:05.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (60m) 5m ago 72m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:34:05.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (59m) 5m ago 72m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:34:05.636 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (57m) 5m ago 71m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:34:06.047 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:34:06.047 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:34:06.047 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:34:06.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:06.048 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:34:06.049 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:34:06.049 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:34:06.049 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:34:06.173 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: audit 2022-04-22T18:34:05.270272+0000 mgr.y (mgr.34107) 2363 : audit [DBG] from='client.37908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.173 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: audit 2022-04-22T18:34:05.604729+0000 mgr.y (mgr.34107) 2364 : audit [DBG] from='client.47848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.173 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: cluster 2022-04-22T18:34:05.775676+0000 mgr.y (mgr.34107) 2365 : cluster [DBG] pgmap v3983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: audit 2022-04-22T18:34:05.857377+0000 mon.a (mon.0) 5382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: audit 2022-04-22T18:34:05.859924+0000 mon.a (mon.0) 5383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[45495]: audit 2022-04-22T18:34:05.861350+0000 mon.a (mon.0) 5384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: audit 2022-04-22T18:34:05.270272+0000 mgr.y (mgr.34107) 2363 : audit [DBG] from='client.37908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: audit 2022-04-22T18:34:05.604729+0000 mgr.y (mgr.34107) 2364 : audit [DBG] from='client.47848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: cluster 2022-04-22T18:34:05.775676+0000 mgr.y (mgr.34107) 2365 : cluster [DBG] pgmap v3983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:06.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: audit 2022-04-22T18:34:05.857377+0000 mon.a (mon.0) 5382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:34:06.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: audit 2022-04-22T18:34:05.859924+0000 mon.a (mon.0) 5383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:34:06.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:05 smithi115 bash[46934]: audit 2022-04-22T18:34:05.861350+0000 mon.a (mon.0) 5384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:34:06.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: audit 2022-04-22T18:34:05.270272+0000 mgr.y (mgr.34107) 2363 : audit [DBG] from='client.37908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: audit 2022-04-22T18:34:05.604729+0000 mgr.y (mgr.34107) 2364 : audit [DBG] from='client.47848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:06.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: cluster 2022-04-22T18:34:05.775676+0000 mgr.y (mgr.34107) 2365 : cluster [DBG] pgmap v3983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:06.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: audit 2022-04-22T18:34:05.857377+0000 mon.a (mon.0) 5382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:34:06.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: audit 2022-04-22T18:34:05.859924+0000 mon.a (mon.0) 5383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:34:06.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:05 smithi121 bash[36902]: audit 2022-04-22T18:34:05.861350+0000 mon.a (mon.0) 5384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:34:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:06 smithi121 bash[36902]: audit 2022-04-22T18:34:06.039212+0000 mon.a (mon.0) 5385 : audit [DBG] from='client.? 172.21.15.115:0/260288308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:06 smithi121 bash[36902]: audit 2022-04-22T18:34:06.218025+0000 mon.a (mon.0) 5386 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:34:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:06 smithi121 bash[36902]: audit 2022-04-22T18:34:06.513014+0000 mon.a (mon.0) 5387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:07.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[46934]: audit 2022-04-22T18:34:06.039212+0000 mon.a (mon.0) 5385 : audit [DBG] from='client.? 172.21.15.115:0/260288308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[46934]: audit 2022-04-22T18:34:06.218025+0000 mon.a (mon.0) 5386 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:34:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[46934]: audit 2022-04-22T18:34:06.513014+0000 mon.a (mon.0) 5387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[45495]: audit 2022-04-22T18:34:06.039212+0000 mon.a (mon.0) 5385 : audit [DBG] from='client.? 172.21.15.115:0/260288308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[45495]: audit 2022-04-22T18:34:06.218025+0000 mon.a (mon.0) 5386 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:34:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:06 smithi115 bash[45495]: audit 2022-04-22T18:34:06.513014+0000 mon.a (mon.0) 5387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:08.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:07 smithi121 bash[36902]: audit 2022-04-22T18:34:07.088982+0000 mon.a (mon.0) 5388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:08.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:07 smithi121 bash[36902]: cluster 2022-04-22T18:34:07.776792+0000 mgr.y (mgr.34107) 2366 : cluster [DBG] pgmap v3984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:07 smithi115 bash[45495]: audit 2022-04-22T18:34:07.088982+0000 mon.a (mon.0) 5388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:07 smithi115 bash[45495]: cluster 2022-04-22T18:34:07.776792+0000 mgr.y (mgr.34107) 2366 : cluster [DBG] pgmap v3984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:07 smithi115 bash[46934]: audit 2022-04-22T18:34:07.088982+0000 mon.a (mon.0) 5388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:07 smithi115 bash[46934]: cluster 2022-04-22T18:34:07.776792+0000 mgr.y (mgr.34107) 2366 : cluster [DBG] pgmap v3984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:09 smithi121 bash[36902]: cluster 2022-04-22T18:34:09.777876+0000 mgr.y (mgr.34107) 2367 : cluster [DBG] pgmap v3985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:09 smithi115 bash[45495]: cluster 2022-04-22T18:34:09.777876+0000 mgr.y (mgr.34107) 2367 : cluster [DBG] pgmap v3985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:09 smithi115 bash[46934]: cluster 2022-04-22T18:34:09.777876+0000 mgr.y (mgr.34107) 2367 : cluster [DBG] pgmap v3985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:11.869 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:11 smithi115 bash[45495]: audit 2022-04-22T18:34:11.515965+0000 mon.a (mon.0) 5389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:11 smithi115 bash[46934]: audit 2022-04-22T18:34:11.515965+0000 mon.a (mon.0) 5389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:11 smithi121 bash[36902]: audit 2022-04-22T18:34:11.515965+0000 mon.a (mon.0) 5389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:11.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:12 smithi121 bash[36902]: cluster 2022-04-22T18:34:11.778983+0000 mgr.y (mgr.34107) 2368 : cluster [DBG] pgmap v3986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:12 smithi121 bash[36902]: audit 2022-04-22T18:34:12.092481+0000 mon.a (mon.0) 5390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:12 smithi115 bash[45495]: cluster 2022-04-22T18:34:11.778983+0000 mgr.y (mgr.34107) 2368 : cluster [DBG] pgmap v3986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:12 smithi115 bash[45495]: audit 2022-04-22T18:34:12.092481+0000 mon.a (mon.0) 5390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:12 smithi115 bash[46934]: cluster 2022-04-22T18:34:11.778983+0000 mgr.y (mgr.34107) 2368 : cluster [DBG] pgmap v3986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:12 smithi115 bash[46934]: audit 2022-04-22T18:34:12.092481+0000 mon.a (mon.0) 5390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:14.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:13 smithi115 bash[45495]: cluster 2022-04-22T18:34:13.780096+0000 mgr.y (mgr.34107) 2369 : cluster [DBG] pgmap v3987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:13.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:13 smithi115 bash[46934]: cluster 2022-04-22T18:34:13.780096+0000 mgr.y (mgr.34107) 2369 : cluster [DBG] pgmap v3987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:13 smithi121 bash[36902]: cluster 2022-04-22T18:34:13.780096+0000 mgr.y (mgr.34107) 2369 : cluster [DBG] pgmap v3987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:14 smithi121 bash[36902]: audit 2022-04-22T18:34:14.054943+0000 mon.a (mon.0) 5391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:34:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:14 smithi121 bash[36902]: audit 2022-04-22T18:34:14.059162+0000 mon.a (mon.0) 5392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:34:15.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:14 smithi115 bash[45495]: audit 2022-04-22T18:34:14.054943+0000 mon.a (mon.0) 5391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:34:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:14 smithi115 bash[45495]: audit 2022-04-22T18:34:14.059162+0000 mon.a (mon.0) 5392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:34:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:14 smithi115 bash[46934]: audit 2022-04-22T18:34:14.054943+0000 mon.a (mon.0) 5391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:34:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:14 smithi115 bash[46934]: audit 2022-04-22T18:34:14.059162+0000 mon.a (mon.0) 5392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:34:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:15 smithi121 bash[36902]: cluster 2022-04-22T18:34:15.781126+0000 mgr.y (mgr.34107) 2370 : cluster [DBG] pgmap v3988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:15 smithi115 bash[45495]: cluster 2022-04-22T18:34:15.781126+0000 mgr.y (mgr.34107) 2370 : cluster [DBG] pgmap v3988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:15 smithi115 bash[46934]: cluster 2022-04-22T18:34:15.781126+0000 mgr.y (mgr.34107) 2370 : cluster [DBG] pgmap v3988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:16 smithi121 bash[36902]: audit 2022-04-22T18:34:16.519091+0000 mon.a (mon.0) 5393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:17.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:16 smithi115 bash[45495]: audit 2022-04-22T18:34:16.519091+0000 mon.a (mon.0) 5393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:17.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:16 smithi115 bash[46934]: audit 2022-04-22T18:34:16.519091+0000 mon.a (mon.0) 5393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:17 smithi121 bash[36902]: audit 2022-04-22T18:34:17.096132+0000 mon.a (mon.0) 5394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:17 smithi121 bash[36902]: cluster 2022-04-22T18:34:17.782208+0000 mgr.y (mgr.34107) 2371 : cluster [DBG] pgmap v3989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:17 smithi115 bash[45495]: audit 2022-04-22T18:34:17.096132+0000 mon.a (mon.0) 5394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:17 smithi115 bash[45495]: cluster 2022-04-22T18:34:17.782208+0000 mgr.y (mgr.34107) 2371 : cluster [DBG] pgmap v3989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:17 smithi115 bash[46934]: audit 2022-04-22T18:34:17.096132+0000 mon.a (mon.0) 5394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:17 smithi115 bash[46934]: cluster 2022-04-22T18:34:17.782208+0000 mgr.y (mgr.34107) 2371 : cluster [DBG] pgmap v3989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:19 smithi121 bash[36902]: cluster 2022-04-22T18:34:19.783312+0000 mgr.y (mgr.34107) 2372 : cluster [DBG] pgmap v3990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:19 smithi115 bash[45495]: cluster 2022-04-22T18:34:19.783312+0000 mgr.y (mgr.34107) 2372 : cluster [DBG] pgmap v3990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:19 smithi115 bash[46934]: cluster 2022-04-22T18:34:19.783312+0000 mgr.y (mgr.34107) 2372 : cluster [DBG] pgmap v3990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:21.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:21 smithi115 bash[45495]: audit 2022-04-22T18:34:21.522196+0000 mon.a (mon.0) 5395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:21 smithi115 bash[46934]: audit 2022-04-22T18:34:21.522196+0000 mon.a (mon.0) 5395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:21 smithi121 bash[36902]: audit 2022-04-22T18:34:21.522196+0000 mon.a (mon.0) 5395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:21.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:22 smithi121 bash[36902]: cluster 2022-04-22T18:34:21.784407+0000 mgr.y (mgr.34107) 2373 : cluster [DBG] pgmap v3991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:22 smithi121 bash[36902]: audit 2022-04-22T18:34:22.099800+0000 mon.a (mon.0) 5396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:22 smithi115 bash[45495]: cluster 2022-04-22T18:34:21.784407+0000 mgr.y (mgr.34107) 2373 : cluster [DBG] pgmap v3991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:22 smithi115 bash[45495]: audit 2022-04-22T18:34:22.099800+0000 mon.a (mon.0) 5396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:22 smithi115 bash[46934]: cluster 2022-04-22T18:34:21.784407+0000 mgr.y (mgr.34107) 2373 : cluster [DBG] pgmap v3991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:22 smithi115 bash[46934]: audit 2022-04-22T18:34:22.099800+0000 mon.a (mon.0) 5396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:23 smithi115 bash[45495]: cluster 2022-04-22T18:34:23.785535+0000 mgr.y (mgr.34107) 2374 : cluster [DBG] pgmap v3992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:23.741Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:23 smithi115 bash[46934]: cluster 2022-04-22T18:34:23.785535+0000 mgr.y (mgr.34107) 2374 : cluster [DBG] pgmap v3992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:23 smithi121 bash[36902]: cluster 2022-04-22T18:34:23.785535+0000 mgr.y (mgr.34107) 2374 : cluster [DBG] pgmap v3992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:25 smithi121 bash[36902]: cluster 2022-04-22T18:34:25.786636+0000 mgr.y (mgr.34107) 2375 : cluster [DBG] pgmap v3993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:25 smithi115 bash[45495]: cluster 2022-04-22T18:34:25.786636+0000 mgr.y (mgr.34107) 2375 : cluster [DBG] pgmap v3993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:25 smithi115 bash[46934]: cluster 2022-04-22T18:34:25.786636+0000 mgr.y (mgr.34107) 2375 : cluster [DBG] pgmap v3993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:26 smithi121 bash[36902]: audit 2022-04-22T18:34:26.525296+0000 mon.a (mon.0) 5397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:26 smithi115 bash[45495]: audit 2022-04-22T18:34:26.525296+0000 mon.a (mon.0) 5397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:26 smithi115 bash[46934]: audit 2022-04-22T18:34:26.525296+0000 mon.a (mon.0) 5397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:27 smithi121 bash[36902]: audit 2022-04-22T18:34:27.103293+0000 mon.a (mon.0) 5398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:28.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:27 smithi121 bash[36902]: cluster 2022-04-22T18:34:27.787821+0000 mgr.y (mgr.34107) 2376 : cluster [DBG] pgmap v3994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:27 smithi115 bash[45495]: audit 2022-04-22T18:34:27.103293+0000 mon.a (mon.0) 5398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:27 smithi115 bash[45495]: cluster 2022-04-22T18:34:27.787821+0000 mgr.y (mgr.34107) 2376 : cluster [DBG] pgmap v3994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:27 smithi115 bash[46934]: audit 2022-04-22T18:34:27.103293+0000 mon.a (mon.0) 5398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:27 smithi115 bash[46934]: cluster 2022-04-22T18:34:27.787821+0000 mgr.y (mgr.34107) 2376 : cluster [DBG] pgmap v3994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:29 smithi121 bash[36902]: cluster 2022-04-22T18:34:29.788913+0000 mgr.y (mgr.34107) 2377 : cluster [DBG] pgmap v3995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:29 smithi115 bash[45495]: cluster 2022-04-22T18:34:29.788913+0000 mgr.y (mgr.34107) 2377 : cluster [DBG] pgmap v3995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:29 smithi115 bash[46934]: cluster 2022-04-22T18:34:29.788913+0000 mgr.y (mgr.34107) 2377 : cluster [DBG] pgmap v3995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:31 smithi115 bash[46934]: audit 2022-04-22T18:34:31.528220+0000 mon.a (mon.0) 5399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:31.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:31 smithi115 bash[45495]: audit 2022-04-22T18:34:31.528220+0000 mon.a (mon.0) 5399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:31 smithi121 bash[36902]: audit 2022-04-22T18:34:31.528220+0000 mon.a (mon.0) 5399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:31.862Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:32 smithi121 bash[36902]: cluster 2022-04-22T18:34:31.789987+0000 mgr.y (mgr.34107) 2378 : cluster [DBG] pgmap v3996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:32 smithi121 bash[36902]: audit 2022-04-22T18:34:32.106768+0000 mon.a (mon.0) 5400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:32 smithi115 bash[46934]: cluster 2022-04-22T18:34:31.789987+0000 mgr.y (mgr.34107) 2378 : cluster [DBG] pgmap v3996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:32 smithi115 bash[46934]: audit 2022-04-22T18:34:32.106768+0000 mon.a (mon.0) 5400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:32 smithi115 bash[45495]: cluster 2022-04-22T18:34:31.789987+0000 mgr.y (mgr.34107) 2378 : cluster [DBG] pgmap v3996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:32 smithi115 bash[45495]: audit 2022-04-22T18:34:32.106768+0000 mon.a (mon.0) 5400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:33 smithi115 bash[45495]: cluster 2022-04-22T18:34:33.791060+0000 mgr.y (mgr.34107) 2379 : cluster [DBG] pgmap v3997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:34.062 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:33.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:34.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:33 smithi115 bash[46934]: cluster 2022-04-22T18:34:33.791060+0000 mgr.y (mgr.34107) 2379 : cluster [DBG] pgmap v3997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:33 smithi121 bash[36902]: cluster 2022-04-22T18:34:33.791060+0000 mgr.y (mgr.34107) 2379 : cluster [DBG] pgmap v3997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:35 smithi121 bash[36902]: cluster 2022-04-22T18:34:35.792112+0000 mgr.y (mgr.34107) 2380 : cluster [DBG] pgmap v3998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:36.374 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:35 smithi115 bash[45495]: cluster 2022-04-22T18:34:35.792112+0000 mgr.y (mgr.34107) 2380 : cluster [DBG] pgmap v3998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:36.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:35 smithi115 bash[46934]: cluster 2022-04-22T18:34:35.792112+0000 mgr.y (mgr.34107) 2380 : cluster [DBG] pgmap v3998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:36.432 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:34:36.781 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:34:36.781 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (57m) 5m ago 71m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:34:36.781 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (71m) 6m ago 71m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:34:36.786 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (59m) 5m ago 72m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:34:36.786 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (59m) 6m ago 72m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (66m) 6m ago 76m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (67m) 5m ago 78m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (66m) 5m ago 79m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (65m) 6m ago 77m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (66m) 5m ago 77m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (58m) 5m ago 71m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (58m) 6m ago 71m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (64m) 5m ago 76m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (63m) 5m ago 75m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (63m) 5m ago 75m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:34:36.787 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (62m) 5m ago 74m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:34:36.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (62m) 6m ago 74m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:34:36.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (61m) 6m ago 73m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:34:36.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (60m) 6m ago 73m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:34:36.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (60m) 6m ago 73m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:34:36.788 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (58m) 6m ago 71m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:34:37.187 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:34:37.187 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:34:37.187 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:34:37.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:37.188 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:34:37.188 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:34:37.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:37.190 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:34:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:34:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:36 smithi121 bash[36902]: audit 2022-04-22T18:34:36.408678+0000 mgr.y (mgr.34107) 2381 : audit [DBG] from='client.47860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:36 smithi121 bash[36902]: audit 2022-04-22T18:34:36.531009+0000 mon.a (mon.0) 5401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:37.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:36 smithi121 bash[36902]: audit 2022-04-22T18:34:36.752508+0000 mgr.y (mgr.34107) 2382 : audit [DBG] from='client.47866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:37.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[45495]: audit 2022-04-22T18:34:36.408678+0000 mgr.y (mgr.34107) 2381 : audit [DBG] from='client.47860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[45495]: audit 2022-04-22T18:34:36.531009+0000 mon.a (mon.0) 5401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[45495]: audit 2022-04-22T18:34:36.752508+0000 mgr.y (mgr.34107) 2382 : audit [DBG] from='client.47866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[46934]: audit 2022-04-22T18:34:36.408678+0000 mgr.y (mgr.34107) 2381 : audit [DBG] from='client.47860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[46934]: audit 2022-04-22T18:34:36.531009+0000 mon.a (mon.0) 5401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:36 smithi115 bash[46934]: audit 2022-04-22T18:34:36.752508+0000 mgr.y (mgr.34107) 2382 : audit [DBG] from='client.47866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:34:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:37 smithi121 bash[36902]: audit 2022-04-22T18:34:37.110425+0000 mon.a (mon.0) 5402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:37 smithi121 bash[36902]: audit 2022-04-22T18:34:37.179784+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.115:0/257615801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:37 smithi121 bash[36902]: cluster 2022-04-22T18:34:37.793246+0000 mgr.y (mgr.34107) 2383 : cluster [DBG] pgmap v3999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:38.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[45495]: audit 2022-04-22T18:34:37.110425+0000 mon.a (mon.0) 5402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[45495]: audit 2022-04-22T18:34:37.179784+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.115:0/257615801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[45495]: cluster 2022-04-22T18:34:37.793246+0000 mgr.y (mgr.34107) 2383 : cluster [DBG] pgmap v3999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[46934]: audit 2022-04-22T18:34:37.110425+0000 mon.a (mon.0) 5402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[46934]: audit 2022-04-22T18:34:37.179784+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.115:0/257615801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:34:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:37 smithi115 bash[46934]: cluster 2022-04-22T18:34:37.793246+0000 mgr.y (mgr.34107) 2383 : cluster [DBG] pgmap v3999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:39 smithi121 bash[36902]: cluster 2022-04-22T18:34:39.794347+0000 mgr.y (mgr.34107) 2384 : cluster [DBG] pgmap v4000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:39 smithi115 bash[45495]: cluster 2022-04-22T18:34:39.794347+0000 mgr.y (mgr.34107) 2384 : cluster [DBG] pgmap v4000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:39 smithi115 bash[46934]: cluster 2022-04-22T18:34:39.794347+0000 mgr.y (mgr.34107) 2384 : cluster [DBG] pgmap v4000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:41.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:41 smithi115 bash[45495]: audit 2022-04-22T18:34:41.533282+0000 mon.a (mon.0) 5403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:41 smithi115 bash[46934]: audit 2022-04-22T18:34:41.533282+0000 mon.a (mon.0) 5403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:41 smithi121 bash[36902]: audit 2022-04-22T18:34:41.533282+0000 mon.a (mon.0) 5403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:42.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:41.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:42 smithi121 bash[36902]: cluster 2022-04-22T18:34:41.795511+0000 mgr.y (mgr.34107) 2385 : cluster [DBG] pgmap v4001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:42 smithi121 bash[36902]: audit 2022-04-22T18:34:42.114122+0000 mon.a (mon.0) 5404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:42 smithi115 bash[46934]: cluster 2022-04-22T18:34:41.795511+0000 mgr.y (mgr.34107) 2385 : cluster [DBG] pgmap v4001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:42 smithi115 bash[46934]: audit 2022-04-22T18:34:42.114122+0000 mon.a (mon.0) 5404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:42 smithi115 bash[45495]: cluster 2022-04-22T18:34:41.795511+0000 mgr.y (mgr.34107) 2385 : cluster [DBG] pgmap v4001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:42 smithi115 bash[45495]: audit 2022-04-22T18:34:42.114122+0000 mon.a (mon.0) 5404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:43 smithi115 bash[45495]: cluster 2022-04-22T18:34:43.796672+0000 mgr.y (mgr.34107) 2386 : cluster [DBG] pgmap v4002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:44.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:43.740Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:43 smithi115 bash[46934]: cluster 2022-04-22T18:34:43.796672+0000 mgr.y (mgr.34107) 2386 : cluster [DBG] pgmap v4002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:43 smithi121 bash[36902]: cluster 2022-04-22T18:34:43.796672+0000 mgr.y (mgr.34107) 2386 : cluster [DBG] pgmap v4002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:45 smithi121 bash[36902]: cluster 2022-04-22T18:34:45.797743+0000 mgr.y (mgr.34107) 2387 : cluster [DBG] pgmap v4003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:45 smithi115 bash[45495]: cluster 2022-04-22T18:34:45.797743+0000 mgr.y (mgr.34107) 2387 : cluster [DBG] pgmap v4003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:45 smithi115 bash[46934]: cluster 2022-04-22T18:34:45.797743+0000 mgr.y (mgr.34107) 2387 : cluster [DBG] pgmap v4003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:46 smithi121 bash[36902]: audit 2022-04-22T18:34:46.536241+0000 mon.a (mon.0) 5405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:46 smithi115 bash[45495]: audit 2022-04-22T18:34:46.536241+0000 mon.a (mon.0) 5405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:46 smithi115 bash[46934]: audit 2022-04-22T18:34:46.536241+0000 mon.a (mon.0) 5405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:47 smithi121 bash[36902]: audit 2022-04-22T18:34:47.117866+0000 mon.a (mon.0) 5406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:47 smithi121 bash[36902]: cluster 2022-04-22T18:34:47.798828+0000 mgr.y (mgr.34107) 2388 : cluster [DBG] pgmap v4004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:48.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:47 smithi115 bash[45495]: audit 2022-04-22T18:34:47.117866+0000 mon.a (mon.0) 5406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:47 smithi115 bash[45495]: cluster 2022-04-22T18:34:47.798828+0000 mgr.y (mgr.34107) 2388 : cluster [DBG] pgmap v4004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:47 smithi115 bash[46934]: audit 2022-04-22T18:34:47.117866+0000 mon.a (mon.0) 5406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:47 smithi115 bash[46934]: cluster 2022-04-22T18:34:47.798828+0000 mgr.y (mgr.34107) 2388 : cluster [DBG] pgmap v4004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:49 smithi121 bash[36902]: cluster 2022-04-22T18:34:49.799958+0000 mgr.y (mgr.34107) 2389 : cluster [DBG] pgmap v4005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:50.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:49 smithi115 bash[45495]: cluster 2022-04-22T18:34:49.799958+0000 mgr.y (mgr.34107) 2389 : cluster [DBG] pgmap v4005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:50.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:49 smithi115 bash[46934]: cluster 2022-04-22T18:34:49.799958+0000 mgr.y (mgr.34107) 2389 : cluster [DBG] pgmap v4005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:51.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:51 smithi115 bash[45495]: audit 2022-04-22T18:34:51.539307+0000 mon.a (mon.0) 5407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:51.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:51 smithi115 bash[46934]: audit 2022-04-22T18:34:51.539307+0000 mon.a (mon.0) 5407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:51 smithi121 bash[36902]: audit 2022-04-22T18:34:51.539307+0000 mon.a (mon.0) 5407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:52.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:51.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:52 smithi121 bash[36902]: cluster 2022-04-22T18:34:51.801150+0000 mgr.y (mgr.34107) 2390 : cluster [DBG] pgmap v4006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:52 smithi121 bash[36902]: audit 2022-04-22T18:34:52.121641+0000 mon.a (mon.0) 5408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:53.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:52 smithi115 bash[45495]: cluster 2022-04-22T18:34:51.801150+0000 mgr.y (mgr.34107) 2390 : cluster [DBG] pgmap v4006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:52 smithi115 bash[45495]: audit 2022-04-22T18:34:52.121641+0000 mon.a (mon.0) 5408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:52 smithi115 bash[46934]: cluster 2022-04-22T18:34:51.801150+0000 mgr.y (mgr.34107) 2390 : cluster [DBG] pgmap v4006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:52 smithi115 bash[46934]: audit 2022-04-22T18:34:52.121641+0000 mon.a (mon.0) 5408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:54.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:53 smithi115 bash[45495]: cluster 2022-04-22T18:34:53.802187+0000 mgr.y (mgr.34107) 2391 : cluster [DBG] pgmap v4007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:34:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:34:53.741Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:34:54.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:53 smithi115 bash[46934]: cluster 2022-04-22T18:34:53.802187+0000 mgr.y (mgr.34107) 2391 : cluster [DBG] pgmap v4007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:53 smithi121 bash[36902]: cluster 2022-04-22T18:34:53.802187+0000 mgr.y (mgr.34107) 2391 : cluster [DBG] pgmap v4007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:55 smithi121 bash[36902]: cluster 2022-04-22T18:34:55.803265+0000 mgr.y (mgr.34107) 2392 : cluster [DBG] pgmap v4008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:56.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:55 smithi115 bash[45495]: cluster 2022-04-22T18:34:55.803265+0000 mgr.y (mgr.34107) 2392 : cluster [DBG] pgmap v4008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:55 smithi115 bash[46934]: cluster 2022-04-22T18:34:55.803265+0000 mgr.y (mgr.34107) 2392 : cluster [DBG] pgmap v4008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:56 smithi121 bash[36902]: audit 2022-04-22T18:34:56.542615+0000 mon.a (mon.0) 5409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:56 smithi115 bash[45495]: audit 2022-04-22T18:34:56.542615+0000 mon.a (mon.0) 5409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:56 smithi115 bash[46934]: audit 2022-04-22T18:34:56.542615+0000 mon.a (mon.0) 5409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:34:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:57 smithi121 bash[36902]: audit 2022-04-22T18:34:57.125349+0000 mon.a (mon.0) 5410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:57 smithi121 bash[36902]: cluster 2022-04-22T18:34:57.804366+0000 mgr.y (mgr.34107) 2393 : cluster [DBG] pgmap v4009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:57 smithi115 bash[45495]: audit 2022-04-22T18:34:57.125349+0000 mon.a (mon.0) 5410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:57 smithi115 bash[45495]: cluster 2022-04-22T18:34:57.804366+0000 mgr.y (mgr.34107) 2393 : cluster [DBG] pgmap v4009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:34:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:57 smithi115 bash[46934]: audit 2022-04-22T18:34:57.125349+0000 mon.a (mon.0) 5410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:34:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:57 smithi115 bash[46934]: cluster 2022-04-22T18:34:57.804366+0000 mgr.y (mgr.34107) 2393 : cluster [DBG] pgmap v4009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:34:59 smithi121 bash[36902]: cluster 2022-04-22T18:34:59.805510+0000 mgr.y (mgr.34107) 2394 : cluster [DBG] pgmap v4010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:00.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:34:59 smithi115 bash[45495]: cluster 2022-04-22T18:34:59.805510+0000 mgr.y (mgr.34107) 2394 : cluster [DBG] pgmap v4010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:34:59 smithi115 bash[46934]: cluster 2022-04-22T18:34:59.805510+0000 mgr.y (mgr.34107) 2394 : cluster [DBG] pgmap v4010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:01 smithi121 bash[36902]: audit 2022-04-22T18:35:01.545886+0000 mon.a (mon.0) 5411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:01.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:01 smithi115 bash[45495]: audit 2022-04-22T18:35:01.545886+0000 mon.a (mon.0) 5411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:01.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:01 smithi115 bash[46934]: audit 2022-04-22T18:35:01.545886+0000 mon.a (mon.0) 5411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:01.874 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:01.865Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:02 smithi121 bash[36902]: cluster 2022-04-22T18:35:01.806608+0000 mgr.y (mgr.34107) 2395 : cluster [DBG] pgmap v4011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:02 smithi121 bash[36902]: audit 2022-04-22T18:35:02.128990+0000 mon.a (mon.0) 5412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:02 smithi115 bash[45495]: cluster 2022-04-22T18:35:01.806608+0000 mgr.y (mgr.34107) 2395 : cluster [DBG] pgmap v4011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:02 smithi115 bash[45495]: audit 2022-04-22T18:35:02.128990+0000 mon.a (mon.0) 5412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:02 smithi115 bash[46934]: cluster 2022-04-22T18:35:01.806608+0000 mgr.y (mgr.34107) 2395 : cluster [DBG] pgmap v4011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:02 smithi115 bash[46934]: audit 2022-04-22T18:35:02.128990+0000 mon.a (mon.0) 5412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:03 smithi115 bash[45495]: cluster 2022-04-22T18:35:03.807808+0000 mgr.y (mgr.34107) 2396 : cluster [DBG] pgmap v4012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:03.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:03 smithi115 bash[46934]: cluster 2022-04-22T18:35:03.807808+0000 mgr.y (mgr.34107) 2396 : cluster [DBG] pgmap v4012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:03 smithi121 bash[36902]: cluster 2022-04-22T18:35:03.807808+0000 mgr.y (mgr.34107) 2396 : cluster [DBG] pgmap v4012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:06.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:05 smithi121 bash[36902]: cluster 2022-04-22T18:35:05.808796+0000 mgr.y (mgr.34107) 2397 : cluster [DBG] pgmap v4013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:05 smithi115 bash[45495]: cluster 2022-04-22T18:35:05.808796+0000 mgr.y (mgr.34107) 2397 : cluster [DBG] pgmap v4013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:05 smithi115 bash[46934]: cluster 2022-04-22T18:35:05.808796+0000 mgr.y (mgr.34107) 2397 : cluster [DBG] pgmap v4013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:06 smithi121 bash[36902]: audit 2022-04-22T18:35:06.227603+0000 mon.a (mon.0) 5413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:35:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:06 smithi121 bash[36902]: audit 2022-04-22T18:35:06.230428+0000 mon.a (mon.0) 5414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:35:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:06 smithi121 bash[36902]: audit 2022-04-22T18:35:06.231412+0000 mon.a (mon.0) 5415 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:35:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:06 smithi121 bash[36902]: audit 2022-04-22T18:35:06.548780+0000 mon.a (mon.0) 5416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:07.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:06 smithi121 bash[36902]: audit 2022-04-22T18:35:06.586559+0000 mon.a (mon.0) 5417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:35:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[45495]: audit 2022-04-22T18:35:06.227603+0000 mon.a (mon.0) 5413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:35:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[45495]: audit 2022-04-22T18:35:06.230428+0000 mon.a (mon.0) 5414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:35:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[45495]: audit 2022-04-22T18:35:06.231412+0000 mon.a (mon.0) 5415 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[45495]: audit 2022-04-22T18:35:06.548780+0000 mon.a (mon.0) 5416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[45495]: audit 2022-04-22T18:35:06.586559+0000 mon.a (mon.0) 5417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[46934]: audit 2022-04-22T18:35:06.227603+0000 mon.a (mon.0) 5413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[46934]: audit 2022-04-22T18:35:06.230428+0000 mon.a (mon.0) 5414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[46934]: audit 2022-04-22T18:35:06.231412+0000 mon.a (mon.0) 5415 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:35:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[46934]: audit 2022-04-22T18:35:06.548780+0000 mon.a (mon.0) 5416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:07.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:06 smithi115 bash[46934]: audit 2022-04-22T18:35:06.586559+0000 mon.a (mon.0) 5417 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:35:07.558 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (58m) 5m ago 71m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (71m) 6m ago 71m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (60m) 5m ago 73m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (59m) 6m ago 72m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (67m) 6m ago 77m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:35:07.901 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (68m) 5m ago 79m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (67m) 5m ago 79m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (66m) 6m ago 77m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (66m) 5m ago 77m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (59m) 5m ago 72m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (59m) 6m ago 72m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (64m) 5m ago 76m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (64m) 5m ago 76m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:35:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (63m) 5m ago 75m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (63m) 5m ago 75m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (62m) 6m ago 74m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (61m) 6m ago 74m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (61m) 6m ago 74m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (60m) 6m ago 73m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:35:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (58m) 6m ago 72m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:35:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:07 smithi121 bash[36902]: audit 2022-04-22T18:35:07.132714+0000 mon.a (mon.0) 5418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:07 smithi121 bash[36902]: audit 2022-04-22T18:35:07.534639+0000 mgr.y (mgr.34107) 2398 : audit [DBG] from='client.37944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:07 smithi121 bash[36902]: cluster 2022-04-22T18:35:07.809548+0000 mgr.y (mgr.34107) 2399 : cluster [DBG] pgmap v4014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:08.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[45495]: audit 2022-04-22T18:35:07.132714+0000 mon.a (mon.0) 5418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[45495]: audit 2022-04-22T18:35:07.534639+0000 mgr.y (mgr.34107) 2398 : audit [DBG] from='client.37944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[45495]: cluster 2022-04-22T18:35:07.809548+0000 mgr.y (mgr.34107) 2399 : cluster [DBG] pgmap v4014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[46934]: audit 2022-04-22T18:35:07.132714+0000 mon.a (mon.0) 5418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[46934]: audit 2022-04-22T18:35:07.534639+0000 mgr.y (mgr.34107) 2398 : audit [DBG] from='client.37944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:07 smithi115 bash[46934]: cluster 2022-04-22T18:35:07.809548+0000 mgr.y (mgr.34107) 2399 : cluster [DBG] pgmap v4014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:08.314 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:35:08.314 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:35:08.314 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:35:08.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:08.316 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:35:08.316 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:35:08.316 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:35:08.316 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:35:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:08 smithi121 bash[36902]: audit 2022-04-22T18:35:07.874478+0000 mgr.y (mgr.34107) 2400 : audit [DBG] from='client.47884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:08 smithi121 bash[36902]: audit 2022-04-22T18:35:08.307090+0000 mon.a (mon.0) 5419 : audit [DBG] from='client.? 172.21.15.115:0/1279836684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:09.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:08 smithi115 bash[45495]: audit 2022-04-22T18:35:07.874478+0000 mgr.y (mgr.34107) 2400 : audit [DBG] from='client.47884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:09.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:08 smithi115 bash[45495]: audit 2022-04-22T18:35:08.307090+0000 mon.a (mon.0) 5419 : audit [DBG] from='client.? 172.21.15.115:0/1279836684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:09.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:08 smithi115 bash[46934]: audit 2022-04-22T18:35:07.874478+0000 mgr.y (mgr.34107) 2400 : audit [DBG] from='client.47884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:09.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:08 smithi115 bash[46934]: audit 2022-04-22T18:35:08.307090+0000 mon.a (mon.0) 5419 : audit [DBG] from='client.? 172.21.15.115:0/1279836684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:10.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:09 smithi121 bash[36902]: cluster 2022-04-22T18:35:09.810534+0000 mgr.y (mgr.34107) 2401 : cluster [DBG] pgmap v4015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:09 smithi115 bash[45495]: cluster 2022-04-22T18:35:09.810534+0000 mgr.y (mgr.34107) 2401 : cluster [DBG] pgmap v4015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:09 smithi115 bash[46934]: cluster 2022-04-22T18:35:09.810534+0000 mgr.y (mgr.34107) 2401 : cluster [DBG] pgmap v4015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:11.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:11 smithi115 bash[45495]: audit 2022-04-22T18:35:11.551309+0000 mon.a (mon.0) 5420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:11 smithi115 bash[46934]: audit 2022-04-22T18:35:11.551309+0000 mon.a (mon.0) 5420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:11 smithi121 bash[36902]: audit 2022-04-22T18:35:11.551309+0000 mon.a (mon.0) 5420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:11.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:12 smithi121 bash[36902]: cluster 2022-04-22T18:35:11.811655+0000 mgr.y (mgr.34107) 2402 : cluster [DBG] pgmap v4016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:12 smithi121 bash[36902]: audit 2022-04-22T18:35:12.136411+0000 mon.a (mon.0) 5421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:12 smithi115 bash[45495]: cluster 2022-04-22T18:35:11.811655+0000 mgr.y (mgr.34107) 2402 : cluster [DBG] pgmap v4016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:13.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:12 smithi115 bash[45495]: audit 2022-04-22T18:35:12.136411+0000 mon.a (mon.0) 5421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:12 smithi115 bash[46934]: cluster 2022-04-22T18:35:11.811655+0000 mgr.y (mgr.34107) 2402 : cluster [DBG] pgmap v4016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:12 smithi115 bash[46934]: audit 2022-04-22T18:35:12.136411+0000 mon.a (mon.0) 5421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:13 smithi115 bash[45495]: cluster 2022-04-22T18:35:13.812834+0000 mgr.y (mgr.34107) 2403 : cluster [DBG] pgmap v4017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:13.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:13 smithi115 bash[46934]: cluster 2022-04-22T18:35:13.812834+0000 mgr.y (mgr.34107) 2403 : cluster [DBG] pgmap v4017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:13 smithi121 bash[36902]: cluster 2022-04-22T18:35:13.812834+0000 mgr.y (mgr.34107) 2403 : cluster [DBG] pgmap v4017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:14 smithi121 bash[36902]: audit 2022-04-22T18:35:14.059642+0000 mon.a (mon.0) 5422 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:35:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:14 smithi121 bash[36902]: audit 2022-04-22T18:35:14.064889+0000 mon.a (mon.0) 5423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:35:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:14 smithi115 bash[45495]: audit 2022-04-22T18:35:14.059642+0000 mon.a (mon.0) 5422 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:35:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:14 smithi115 bash[45495]: audit 2022-04-22T18:35:14.064889+0000 mon.a (mon.0) 5423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:35:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:14 smithi115 bash[46934]: audit 2022-04-22T18:35:14.059642+0000 mon.a (mon.0) 5422 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:35:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:14 smithi115 bash[46934]: audit 2022-04-22T18:35:14.064889+0000 mon.a (mon.0) 5423 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:35:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:15 smithi121 bash[36902]: cluster 2022-04-22T18:35:15.813899+0000 mgr.y (mgr.34107) 2404 : cluster [DBG] pgmap v4018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:15 smithi115 bash[45495]: cluster 2022-04-22T18:35:15.813899+0000 mgr.y (mgr.34107) 2404 : cluster [DBG] pgmap v4018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:15 smithi115 bash[46934]: cluster 2022-04-22T18:35:15.813899+0000 mgr.y (mgr.34107) 2404 : cluster [DBG] pgmap v4018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:16 smithi121 bash[36902]: audit 2022-04-22T18:35:16.554203+0000 mon.a (mon.0) 5424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:17.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:16 smithi115 bash[45495]: audit 2022-04-22T18:35:16.554203+0000 mon.a (mon.0) 5424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:17.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:16 smithi115 bash[46934]: audit 2022-04-22T18:35:16.554203+0000 mon.a (mon.0) 5424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:17 smithi121 bash[36902]: audit 2022-04-22T18:35:17.140003+0000 mon.a (mon.0) 5425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:17 smithi121 bash[36902]: cluster 2022-04-22T18:35:17.815045+0000 mgr.y (mgr.34107) 2405 : cluster [DBG] pgmap v4019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:18.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:17 smithi115 bash[45495]: audit 2022-04-22T18:35:17.140003+0000 mon.a (mon.0) 5425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:18.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:17 smithi115 bash[45495]: cluster 2022-04-22T18:35:17.815045+0000 mgr.y (mgr.34107) 2405 : cluster [DBG] pgmap v4019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:18.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:17 smithi115 bash[46934]: audit 2022-04-22T18:35:17.140003+0000 mon.a (mon.0) 5425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:18.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:17 smithi115 bash[46934]: cluster 2022-04-22T18:35:17.815045+0000 mgr.y (mgr.34107) 2405 : cluster [DBG] pgmap v4019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:19 smithi121 bash[36902]: cluster 2022-04-22T18:35:19.816191+0000 mgr.y (mgr.34107) 2406 : cluster [DBG] pgmap v4020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:20.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:19 smithi115 bash[45495]: cluster 2022-04-22T18:35:19.816191+0000 mgr.y (mgr.34107) 2406 : cluster [DBG] pgmap v4020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:19 smithi115 bash[46934]: cluster 2022-04-22T18:35:19.816191+0000 mgr.y (mgr.34107) 2406 : cluster [DBG] pgmap v4020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:21.877 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:21 smithi115 bash[45495]: audit 2022-04-22T18:35:21.557537+0000 mon.a (mon.0) 5426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:21.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:21 smithi115 bash[46934]: audit 2022-04-22T18:35:21.557537+0000 mon.a (mon.0) 5426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:21 smithi121 bash[36902]: audit 2022-04-22T18:35:21.557537+0000 mon.a (mon.0) 5426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:22.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:21.865Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:22 smithi121 bash[36902]: cluster 2022-04-22T18:35:21.817296+0000 mgr.y (mgr.34107) 2407 : cluster [DBG] pgmap v4021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:22 smithi121 bash[36902]: audit 2022-04-22T18:35:22.143796+0000 mon.a (mon.0) 5427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:23.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:22 smithi115 bash[45495]: cluster 2022-04-22T18:35:21.817296+0000 mgr.y (mgr.34107) 2407 : cluster [DBG] pgmap v4021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:22 smithi115 bash[45495]: audit 2022-04-22T18:35:22.143796+0000 mon.a (mon.0) 5427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:22 smithi115 bash[46934]: cluster 2022-04-22T18:35:21.817296+0000 mgr.y (mgr.34107) 2407 : cluster [DBG] pgmap v4021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:22 smithi115 bash[46934]: audit 2022-04-22T18:35:22.143796+0000 mon.a (mon.0) 5427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:23 smithi115 bash[45495]: cluster 2022-04-22T18:35:23.818442+0000 mgr.y (mgr.34107) 2408 : cluster [DBG] pgmap v4022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:23.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:23 smithi115 bash[46934]: cluster 2022-04-22T18:35:23.818442+0000 mgr.y (mgr.34107) 2408 : cluster [DBG] pgmap v4022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:23 smithi121 bash[36902]: cluster 2022-04-22T18:35:23.818442+0000 mgr.y (mgr.34107) 2408 : cluster [DBG] pgmap v4022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:25 smithi121 bash[36902]: cluster 2022-04-22T18:35:25.819497+0000 mgr.y (mgr.34107) 2409 : cluster [DBG] pgmap v4023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:26.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:25 smithi115 bash[45495]: cluster 2022-04-22T18:35:25.819497+0000 mgr.y (mgr.34107) 2409 : cluster [DBG] pgmap v4023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:25 smithi115 bash[46934]: cluster 2022-04-22T18:35:25.819497+0000 mgr.y (mgr.34107) 2409 : cluster [DBG] pgmap v4023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:26 smithi121 bash[36902]: audit 2022-04-22T18:35:26.560719+0000 mon.a (mon.0) 5428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:26 smithi115 bash[45495]: audit 2022-04-22T18:35:26.560719+0000 mon.a (mon.0) 5428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:26 smithi115 bash[46934]: audit 2022-04-22T18:35:26.560719+0000 mon.a (mon.0) 5428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:27 smithi121 bash[36902]: audit 2022-04-22T18:35:27.147250+0000 mon.a (mon.0) 5429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:27 smithi121 bash[36902]: cluster 2022-04-22T18:35:27.820552+0000 mgr.y (mgr.34107) 2410 : cluster [DBG] pgmap v4024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:27 smithi115 bash[45495]: audit 2022-04-22T18:35:27.147250+0000 mon.a (mon.0) 5429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:27 smithi115 bash[45495]: cluster 2022-04-22T18:35:27.820552+0000 mgr.y (mgr.34107) 2410 : cluster [DBG] pgmap v4024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:27 smithi115 bash[46934]: audit 2022-04-22T18:35:27.147250+0000 mon.a (mon.0) 5429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:27 smithi115 bash[46934]: cluster 2022-04-22T18:35:27.820552+0000 mgr.y (mgr.34107) 2410 : cluster [DBG] pgmap v4024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:29 smithi121 bash[36902]: cluster 2022-04-22T18:35:29.821660+0000 mgr.y (mgr.34107) 2411 : cluster [DBG] pgmap v4025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:30.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:29 smithi115 bash[45495]: cluster 2022-04-22T18:35:29.821660+0000 mgr.y (mgr.34107) 2411 : cluster [DBG] pgmap v4025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:29 smithi115 bash[46934]: cluster 2022-04-22T18:35:29.821660+0000 mgr.y (mgr.34107) 2411 : cluster [DBG] pgmap v4025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:31.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:31 smithi115 bash[45495]: audit 2022-04-22T18:35:31.563718+0000 mon.a (mon.0) 5430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:31.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:31 smithi115 bash[46934]: audit 2022-04-22T18:35:31.563718+0000 mon.a (mon.0) 5430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:31 smithi121 bash[36902]: audit 2022-04-22T18:35:31.563718+0000 mon.a (mon.0) 5430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:31.867Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:32 smithi121 bash[36902]: cluster 2022-04-22T18:35:31.822740+0000 mgr.y (mgr.34107) 2412 : cluster [DBG] pgmap v4026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:32 smithi121 bash[36902]: audit 2022-04-22T18:35:32.150493+0000 mon.a (mon.0) 5431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:32 smithi115 bash[45495]: cluster 2022-04-22T18:35:31.822740+0000 mgr.y (mgr.34107) 2412 : cluster [DBG] pgmap v4026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:33.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:32 smithi115 bash[45495]: audit 2022-04-22T18:35:32.150493+0000 mon.a (mon.0) 5431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:32 smithi115 bash[46934]: cluster 2022-04-22T18:35:31.822740+0000 mgr.y (mgr.34107) 2412 : cluster [DBG] pgmap v4026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:33.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:32 smithi115 bash[46934]: audit 2022-04-22T18:35:32.150493+0000 mon.a (mon.0) 5431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:34.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:33 smithi115 bash[45495]: cluster 2022-04-22T18:35:33.823978+0000 mgr.y (mgr.34107) 2413 : cluster [DBG] pgmap v4027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:33.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:34.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:33 smithi115 bash[46934]: cluster 2022-04-22T18:35:33.823978+0000 mgr.y (mgr.34107) 2413 : cluster [DBG] pgmap v4027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:33 smithi121 bash[36902]: cluster 2022-04-22T18:35:33.823978+0000 mgr.y (mgr.34107) 2413 : cluster [DBG] pgmap v4027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:35 smithi121 bash[36902]: cluster 2022-04-22T18:35:35.825033+0000 mgr.y (mgr.34107) 2414 : cluster [DBG] pgmap v4028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:35 smithi115 bash[45495]: cluster 2022-04-22T18:35:35.825033+0000 mgr.y (mgr.34107) 2414 : cluster [DBG] pgmap v4028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:35 smithi115 bash[46934]: cluster 2022-04-22T18:35:35.825033+0000 mgr.y (mgr.34107) 2414 : cluster [DBG] pgmap v4028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:36 smithi121 bash[36902]: audit 2022-04-22T18:35:36.566863+0000 mon.a (mon.0) 5432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:36 smithi115 bash[45495]: audit 2022-04-22T18:35:36.566863+0000 mon.a (mon.0) 5432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:36 smithi115 bash[46934]: audit 2022-04-22T18:35:36.566863+0000 mon.a (mon.0) 5432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:37 smithi121 bash[36902]: audit 2022-04-22T18:35:37.154233+0000 mon.a (mon.0) 5433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:37 smithi121 bash[36902]: cluster 2022-04-22T18:35:37.826133+0000 mgr.y (mgr.34107) 2415 : cluster [DBG] pgmap v4029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:37 smithi115 bash[45495]: audit 2022-04-22T18:35:37.154233+0000 mon.a (mon.0) 5433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:37 smithi115 bash[45495]: cluster 2022-04-22T18:35:37.826133+0000 mgr.y (mgr.34107) 2415 : cluster [DBG] pgmap v4029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:37 smithi115 bash[46934]: audit 2022-04-22T18:35:37.154233+0000 mon.a (mon.0) 5433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:37 smithi115 bash[46934]: cluster 2022-04-22T18:35:37.826133+0000 mgr.y (mgr.34107) 2415 : cluster [DBG] pgmap v4029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:35:39.036 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:35:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (58m) 6m ago 72m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:35:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (72m) 7m ago 72m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:35:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (60m) 6m ago 73m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:35:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (60m) 7m ago 73m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:35:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (67m) 7m ago 77m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:35:39.038 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (68m) 6m ago 79m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:35:39.038 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (67m) 6m ago 80m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:35:39.038 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (66m) 7m ago 78m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:35:39.038 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (67m) 6m ago 78m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:35:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (59m) 6m ago 72m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:35:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (59m) 7m ago 72m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:35:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (65m) 6m ago 77m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:35:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (64m) 6m ago 76m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:35:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (64m) 6m ago 76m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:35:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (63m) 6m ago 75m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:35:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (63m) 7m ago 75m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:35:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (62m) 7m ago 75m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:35:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (61m) 7m ago 74m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:35:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (61m) 7m ago 74m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:35:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (59m) 7m ago 72m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:35:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:38 smithi121 bash[36902]: audit 2022-04-22T18:35:38.672234+0000 mgr.y (mgr.34107) 2416 : audit [DBG] from='client.37962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:38 smithi115 bash[45495]: audit 2022-04-22T18:35:38.672234+0000 mgr.y (mgr.34107) 2416 : audit [DBG] from='client.37962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:39.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:38 smithi115 bash[46934]: audit 2022-04-22T18:35:38.672234+0000 mgr.y (mgr.34107) 2416 : audit [DBG] from='client.37962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:35:39.441 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:35:39.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:35:39.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:35:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:35:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:35:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:35:39.443 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:35:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:39 smithi121 bash[36902]: audit 2022-04-22T18:35:39.014755+0000 mgr.y (mgr.34107) 2417 : audit [DBG] from='client.37968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:39 smithi121 bash[36902]: audit 2022-04-22T18:35:39.433637+0000 mon.c (mon.1) 149 : audit [DBG] from='client.? 172.21.15.115:0/2556295486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:39 smithi121 bash[36902]: cluster 2022-04-22T18:35:39.827383+0000 mgr.y (mgr.34107) 2418 : cluster [DBG] pgmap v4030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[45495]: audit 2022-04-22T18:35:39.014755+0000 mgr.y (mgr.34107) 2417 : audit [DBG] from='client.37968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[45495]: audit 2022-04-22T18:35:39.433637+0000 mon.c (mon.1) 149 : audit [DBG] from='client.? 172.21.15.115:0/2556295486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[45495]: cluster 2022-04-22T18:35:39.827383+0000 mgr.y (mgr.34107) 2418 : cluster [DBG] pgmap v4030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[46934]: audit 2022-04-22T18:35:39.014755+0000 mgr.y (mgr.34107) 2417 : audit [DBG] from='client.37968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:35:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[46934]: audit 2022-04-22T18:35:39.433637+0000 mon.c (mon.1) 149 : audit [DBG] from='client.? 172.21.15.115:0/2556295486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:35:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:39 smithi115 bash[46934]: cluster 2022-04-22T18:35:39.827383+0000 mgr.y (mgr.34107) 2418 : cluster [DBG] pgmap v4030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:41 smithi121 bash[36902]: audit 2022-04-22T18:35:41.570290+0000 mon.a (mon.0) 5434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:42.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:41 smithi115 bash[45495]: audit 2022-04-22T18:35:41.570290+0000 mon.a (mon.0) 5434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:42.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:41.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:41 smithi115 bash[46934]: audit 2022-04-22T18:35:41.570290+0000 mon.a (mon.0) 5434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:42 smithi115 bash[45495]: cluster 2022-04-22T18:35:41.828623+0000 mgr.y (mgr.34107) 2419 : cluster [DBG] pgmap v4031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:42 smithi115 bash[45495]: audit 2022-04-22T18:35:42.157930+0000 mon.a (mon.0) 5435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:42 smithi115 bash[46934]: cluster 2022-04-22T18:35:41.828623+0000 mgr.y (mgr.34107) 2419 : cluster [DBG] pgmap v4031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:42 smithi115 bash[46934]: audit 2022-04-22T18:35:42.157930+0000 mon.a (mon.0) 5435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:42 smithi121 bash[36902]: cluster 2022-04-22T18:35:41.828623+0000 mgr.y (mgr.34107) 2419 : cluster [DBG] pgmap v4031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:42 smithi121 bash[36902]: audit 2022-04-22T18:35:42.157930+0000 mon.a (mon.0) 5435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:43 smithi115 bash[45495]: cluster 2022-04-22T18:35:43.829799+0000 mgr.y (mgr.34107) 2420 : cluster [DBG] pgmap v4032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:43.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:43 smithi115 bash[46934]: cluster 2022-04-22T18:35:43.829799+0000 mgr.y (mgr.34107) 2420 : cluster [DBG] pgmap v4032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:43 smithi121 bash[36902]: cluster 2022-04-22T18:35:43.829799+0000 mgr.y (mgr.34107) 2420 : cluster [DBG] pgmap v4032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:45 smithi121 bash[36902]: cluster 2022-04-22T18:35:45.830822+0000 mgr.y (mgr.34107) 2421 : cluster [DBG] pgmap v4033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:46.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:45 smithi115 bash[45495]: cluster 2022-04-22T18:35:45.830822+0000 mgr.y (mgr.34107) 2421 : cluster [DBG] pgmap v4033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:45 smithi115 bash[46934]: cluster 2022-04-22T18:35:45.830822+0000 mgr.y (mgr.34107) 2421 : cluster [DBG] pgmap v4033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:46 smithi121 bash[36902]: audit 2022-04-22T18:35:46.573377+0000 mon.a (mon.0) 5436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:46 smithi115 bash[45495]: audit 2022-04-22T18:35:46.573377+0000 mon.a (mon.0) 5436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:46 smithi115 bash[46934]: audit 2022-04-22T18:35:46.573377+0000 mon.a (mon.0) 5436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:47 smithi121 bash[36902]: audit 2022-04-22T18:35:47.161663+0000 mon.a (mon.0) 5437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:47 smithi121 bash[36902]: cluster 2022-04-22T18:35:47.831885+0000 mgr.y (mgr.34107) 2422 : cluster [DBG] pgmap v4034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:47 smithi115 bash[45495]: audit 2022-04-22T18:35:47.161663+0000 mon.a (mon.0) 5437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:47 smithi115 bash[45495]: cluster 2022-04-22T18:35:47.831885+0000 mgr.y (mgr.34107) 2422 : cluster [DBG] pgmap v4034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:47 smithi115 bash[46934]: audit 2022-04-22T18:35:47.161663+0000 mon.a (mon.0) 5437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:47 smithi115 bash[46934]: cluster 2022-04-22T18:35:47.831885+0000 mgr.y (mgr.34107) 2422 : cluster [DBG] pgmap v4034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:50.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:49 smithi121 bash[36902]: cluster 2022-04-22T18:35:49.833090+0000 mgr.y (mgr.34107) 2423 : cluster [DBG] pgmap v4035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:50.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:49 smithi115 bash[46934]: cluster 2022-04-22T18:35:49.833090+0000 mgr.y (mgr.34107) 2423 : cluster [DBG] pgmap v4035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:50.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:49 smithi115 bash[45495]: cluster 2022-04-22T18:35:49.833090+0000 mgr.y (mgr.34107) 2423 : cluster [DBG] pgmap v4035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:51 smithi121 bash[36902]: audit 2022-04-22T18:35:51.576314+0000 mon.a (mon.0) 5438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:52.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:51.867Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:52.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:51 smithi115 bash[45495]: audit 2022-04-22T18:35:51.576314+0000 mon.a (mon.0) 5438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:52.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:51 smithi115 bash[46934]: audit 2022-04-22T18:35:51.576314+0000 mon.a (mon.0) 5438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:52 smithi121 bash[36902]: cluster 2022-04-22T18:35:51.834268+0000 mgr.y (mgr.34107) 2424 : cluster [DBG] pgmap v4036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:53.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:52 smithi121 bash[36902]: audit 2022-04-22T18:35:52.164921+0000 mon.a (mon.0) 5439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:52 smithi115 bash[46934]: cluster 2022-04-22T18:35:51.834268+0000 mgr.y (mgr.34107) 2424 : cluster [DBG] pgmap v4036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:52 smithi115 bash[46934]: audit 2022-04-22T18:35:52.164921+0000 mon.a (mon.0) 5439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:52 smithi115 bash[45495]: cluster 2022-04-22T18:35:51.834268+0000 mgr.y (mgr.34107) 2424 : cluster [DBG] pgmap v4036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:52 smithi115 bash[45495]: audit 2022-04-22T18:35:52.164921+0000 mon.a (mon.0) 5439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:54.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:53 smithi115 bash[45495]: cluster 2022-04-22T18:35:53.835451+0000 mgr.y (mgr.34107) 2425 : cluster [DBG] pgmap v4037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:35:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:35:53.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:35:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:53 smithi115 bash[46934]: cluster 2022-04-22T18:35:53.835451+0000 mgr.y (mgr.34107) 2425 : cluster [DBG] pgmap v4037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:53 smithi121 bash[36902]: cluster 2022-04-22T18:35:53.835451+0000 mgr.y (mgr.34107) 2425 : cluster [DBG] pgmap v4037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:55 smithi121 bash[36902]: cluster 2022-04-22T18:35:55.836475+0000 mgr.y (mgr.34107) 2426 : cluster [DBG] pgmap v4038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:55 smithi115 bash[45495]: cluster 2022-04-22T18:35:55.836475+0000 mgr.y (mgr.34107) 2426 : cluster [DBG] pgmap v4038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:56.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:55 smithi115 bash[46934]: cluster 2022-04-22T18:35:55.836475+0000 mgr.y (mgr.34107) 2426 : cluster [DBG] pgmap v4038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:56 smithi121 bash[36902]: audit 2022-04-22T18:35:56.579369+0000 mon.a (mon.0) 5440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:56 smithi115 bash[46934]: audit 2022-04-22T18:35:56.579369+0000 mon.a (mon.0) 5440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:56 smithi115 bash[45495]: audit 2022-04-22T18:35:56.579369+0000 mon.a (mon.0) 5440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:35:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:57 smithi121 bash[36902]: audit 2022-04-22T18:35:57.167912+0000 mon.a (mon.0) 5441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:57 smithi121 bash[36902]: cluster 2022-04-22T18:35:57.837520+0000 mgr.y (mgr.34107) 2427 : cluster [DBG] pgmap v4039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:57 smithi115 bash[46934]: audit 2022-04-22T18:35:57.167912+0000 mon.a (mon.0) 5441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:57 smithi115 bash[46934]: cluster 2022-04-22T18:35:57.837520+0000 mgr.y (mgr.34107) 2427 : cluster [DBG] pgmap v4039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:35:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:57 smithi115 bash[45495]: audit 2022-04-22T18:35:57.167912+0000 mon.a (mon.0) 5441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:35:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:57 smithi115 bash[45495]: cluster 2022-04-22T18:35:57.837520+0000 mgr.y (mgr.34107) 2427 : cluster [DBG] pgmap v4039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:00.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:35:59 smithi121 bash[36902]: cluster 2022-04-22T18:35:59.838623+0000 mgr.y (mgr.34107) 2428 : cluster [DBG] pgmap v4040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:35:59 smithi115 bash[45495]: cluster 2022-04-22T18:35:59.838623+0000 mgr.y (mgr.34107) 2428 : cluster [DBG] pgmap v4040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:00.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:35:59 smithi115 bash[46934]: cluster 2022-04-22T18:35:59.838623+0000 mgr.y (mgr.34107) 2428 : cluster [DBG] pgmap v4040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:01 smithi121 bash[36902]: audit 2022-04-22T18:36:01.582496+0000 mon.a (mon.0) 5442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:02.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:01 smithi115 bash[46934]: audit 2022-04-22T18:36:01.582496+0000 mon.a (mon.0) 5442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:02.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:01.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:01 smithi115 bash[45495]: audit 2022-04-22T18:36:01.582496+0000 mon.a (mon.0) 5442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:02 smithi121 bash[36902]: cluster 2022-04-22T18:36:01.839886+0000 mgr.y (mgr.34107) 2429 : cluster [DBG] pgmap v4041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:02 smithi121 bash[36902]: audit 2022-04-22T18:36:02.171416+0000 mon.a (mon.0) 5443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:02 smithi115 bash[45495]: cluster 2022-04-22T18:36:01.839886+0000 mgr.y (mgr.34107) 2429 : cluster [DBG] pgmap v4041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:02 smithi115 bash[45495]: audit 2022-04-22T18:36:02.171416+0000 mon.a (mon.0) 5443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:02 smithi115 bash[46934]: cluster 2022-04-22T18:36:01.839886+0000 mgr.y (mgr.34107) 2429 : cluster [DBG] pgmap v4041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:02 smithi115 bash[46934]: audit 2022-04-22T18:36:02.171416+0000 mon.a (mon.0) 5443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:03.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:04.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:03 smithi115 bash[45495]: cluster 2022-04-22T18:36:03.841028+0000 mgr.y (mgr.34107) 2430 : cluster [DBG] pgmap v4042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:03 smithi115 bash[46934]: cluster 2022-04-22T18:36:03.841028+0000 mgr.y (mgr.34107) 2430 : cluster [DBG] pgmap v4042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:03 smithi121 bash[36902]: cluster 2022-04-22T18:36:03.841028+0000 mgr.y (mgr.34107) 2430 : cluster [DBG] pgmap v4042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:05 smithi121 bash[36902]: cluster 2022-04-22T18:36:05.842054+0000 mgr.y (mgr.34107) 2431 : cluster [DBG] pgmap v4043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:05 smithi115 bash[45495]: cluster 2022-04-22T18:36:05.842054+0000 mgr.y (mgr.34107) 2431 : cluster [DBG] pgmap v4043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:05 smithi115 bash[46934]: cluster 2022-04-22T18:36:05.842054+0000 mgr.y (mgr.34107) 2431 : cluster [DBG] pgmap v4043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:06 smithi121 bash[36902]: audit 2022-04-22T18:36:06.585243+0000 mon.a (mon.0) 5444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:06 smithi121 bash[36902]: audit 2022-04-22T18:36:06.594297+0000 mon.a (mon.0) 5445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:36:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:06 smithi121 bash[36902]: audit 2022-04-22T18:36:06.596924+0000 mon.a (mon.0) 5446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:36:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:06 smithi121 bash[36902]: audit 2022-04-22T18:36:06.598610+0000 mon.a (mon.0) 5447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:36:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[45495]: audit 2022-04-22T18:36:06.585243+0000 mon.a (mon.0) 5444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[45495]: audit 2022-04-22T18:36:06.594297+0000 mon.a (mon.0) 5445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:36:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[45495]: audit 2022-04-22T18:36:06.596924+0000 mon.a (mon.0) 5446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:36:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[45495]: audit 2022-04-22T18:36:06.598610+0000 mon.a (mon.0) 5447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:36:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[46934]: audit 2022-04-22T18:36:06.585243+0000 mon.a (mon.0) 5444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[46934]: audit 2022-04-22T18:36:06.594297+0000 mon.a (mon.0) 5445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:36:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[46934]: audit 2022-04-22T18:36:06.596924+0000 mon.a (mon.0) 5446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:36:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:06 smithi115 bash[46934]: audit 2022-04-22T18:36:06.598610+0000 mon.a (mon.0) 5447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:36:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:07 smithi121 bash[36902]: audit 2022-04-22T18:36:06.934546+0000 mon.a (mon.0) 5448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:36:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:07 smithi121 bash[36902]: audit 2022-04-22T18:36:07.174691+0000 mon.a (mon.0) 5449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:07 smithi121 bash[36902]: cluster 2022-04-22T18:36:07.843210+0000 mgr.y (mgr.34107) 2432 : cluster [DBG] pgmap v4044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[45495]: audit 2022-04-22T18:36:06.934546+0000 mon.a (mon.0) 5448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:36:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[45495]: audit 2022-04-22T18:36:07.174691+0000 mon.a (mon.0) 5449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[45495]: cluster 2022-04-22T18:36:07.843210+0000 mgr.y (mgr.34107) 2432 : cluster [DBG] pgmap v4044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[46934]: audit 2022-04-22T18:36:06.934546+0000 mon.a (mon.0) 5448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:36:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[46934]: audit 2022-04-22T18:36:07.174691+0000 mon.a (mon.0) 5449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:07 smithi115 bash[46934]: cluster 2022-04-22T18:36:07.843210+0000 mgr.y (mgr.34107) 2432 : cluster [DBG] pgmap v4044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:09.813 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:36:10.159 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:36:10.234 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (59m) 6m ago 73m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (72m) 8m ago 72m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (61m) 6m ago 74m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (60m) 8m ago 73m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (68m) 8m ago 78m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (69m) 6m ago 80m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (68m) 6m ago 80m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (67m) 8m ago 78m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (67m) 6m ago 78m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:36:10.235 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (60m) 6m ago 73m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (60m) 8m ago 73m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (65m) 6m ago 77m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (65m) 6m ago 77m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (64m) 6m ago 76m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (64m) 6m ago 76m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (63m) 8m ago 75m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (63m) 8m ago 75m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (62m) 8m ago 75m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (61m) 8m ago 74m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:36:10.236 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (59m) 8m ago 73m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:36:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:09 smithi121 bash[36902]: audit 2022-04-22T18:36:09.788518+0000 mgr.y (mgr.34107) 2433 : audit [DBG] from='client.47914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:09 smithi121 bash[36902]: cluster 2022-04-22T18:36:09.843973+0000 mgr.y (mgr.34107) 2434 : cluster [DBG] pgmap v4045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:09 smithi115 bash[45495]: audit 2022-04-22T18:36:09.788518+0000 mgr.y (mgr.34107) 2433 : audit [DBG] from='client.47914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:09 smithi115 bash[45495]: cluster 2022-04-22T18:36:09.843973+0000 mgr.y (mgr.34107) 2434 : cluster [DBG] pgmap v4045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:09 smithi115 bash[46934]: audit 2022-04-22T18:36:09.788518+0000 mgr.y (mgr.34107) 2433 : audit [DBG] from='client.47914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:09 smithi115 bash[46934]: cluster 2022-04-22T18:36:09.843973+0000 mgr.y (mgr.34107) 2434 : cluster [DBG] pgmap v4045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:36:10.568 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:36:10.568 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:36:10.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:10.569 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:36:10.569 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:36:10.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:10.569 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:36:10.570 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:36:10.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:10.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:36:10.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:36:10.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:10.572 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:36:10.572 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:36:10.572 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:36:10.572 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:36:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:10 smithi121 bash[36902]: audit 2022-04-22T18:36:10.125533+0000 mgr.y (mgr.34107) 2435 : audit [DBG] from='client.37986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:10 smithi121 bash[36902]: audit 2022-04-22T18:36:10.560269+0000 mon.a (mon.0) 5450 : audit [DBG] from='client.? 172.21.15.115:0/3646048669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:11.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:10 smithi115 bash[46934]: audit 2022-04-22T18:36:10.125533+0000 mgr.y (mgr.34107) 2435 : audit [DBG] from='client.37986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:10 smithi115 bash[46934]: audit 2022-04-22T18:36:10.560269+0000 mon.a (mon.0) 5450 : audit [DBG] from='client.? 172.21.15.115:0/3646048669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:10 smithi115 bash[45495]: audit 2022-04-22T18:36:10.125533+0000 mgr.y (mgr.34107) 2435 : audit [DBG] from='client.37986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:10 smithi115 bash[45495]: audit 2022-04-22T18:36:10.560269+0000 mon.a (mon.0) 5450 : audit [DBG] from='client.? 172.21.15.115:0/3646048669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:11 smithi121 bash[36902]: audit 2022-04-22T18:36:11.588183+0000 mon.a (mon.0) 5451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:11 smithi121 bash[36902]: cluster 2022-04-22T18:36:11.845014+0000 mgr.y (mgr.34107) 2436 : cluster [DBG] pgmap v4046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:11.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:11 smithi115 bash[45495]: audit 2022-04-22T18:36:11.588183+0000 mon.a (mon.0) 5451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:11 smithi115 bash[45495]: cluster 2022-04-22T18:36:11.845014+0000 mgr.y (mgr.34107) 2436 : cluster [DBG] pgmap v4046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:11 smithi115 bash[46934]: audit 2022-04-22T18:36:11.588183+0000 mon.a (mon.0) 5451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:11 smithi115 bash[46934]: cluster 2022-04-22T18:36:11.845014+0000 mgr.y (mgr.34107) 2436 : cluster [DBG] pgmap v4046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:12 smithi121 bash[36902]: audit 2022-04-22T18:36:12.177913+0000 mon.a (mon.0) 5452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:12 smithi115 bash[46934]: audit 2022-04-22T18:36:12.177913+0000 mon.a (mon.0) 5452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:12 smithi115 bash[45495]: audit 2022-04-22T18:36:12.177913+0000 mon.a (mon.0) 5452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:13 smithi115 bash[45495]: cluster 2022-04-22T18:36:13.846078+0000 mgr.y (mgr.34107) 2437 : cluster [DBG] pgmap v4047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:13.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:13 smithi115 bash[46934]: cluster 2022-04-22T18:36:13.846078+0000 mgr.y (mgr.34107) 2437 : cluster [DBG] pgmap v4047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:13 smithi121 bash[36902]: cluster 2022-04-22T18:36:13.846078+0000 mgr.y (mgr.34107) 2437 : cluster [DBG] pgmap v4047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:15.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:14 smithi121 bash[36902]: audit 2022-04-22T18:36:14.064221+0000 mon.a (mon.0) 5453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:36:15.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:14 smithi121 bash[36902]: audit 2022-04-22T18:36:14.070078+0000 mon.a (mon.0) 5454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:36:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:14 smithi115 bash[45495]: audit 2022-04-22T18:36:14.064221+0000 mon.a (mon.0) 5453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:36:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:14 smithi115 bash[45495]: audit 2022-04-22T18:36:14.070078+0000 mon.a (mon.0) 5454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:36:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:14 smithi115 bash[46934]: audit 2022-04-22T18:36:14.064221+0000 mon.a (mon.0) 5453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:36:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:14 smithi115 bash[46934]: audit 2022-04-22T18:36:14.070078+0000 mon.a (mon.0) 5454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:36:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:15 smithi121 bash[36902]: cluster 2022-04-22T18:36:15.847093+0000 mgr.y (mgr.34107) 2438 : cluster [DBG] pgmap v4048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:15 smithi115 bash[45495]: cluster 2022-04-22T18:36:15.847093+0000 mgr.y (mgr.34107) 2438 : cluster [DBG] pgmap v4048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:15 smithi115 bash[46934]: cluster 2022-04-22T18:36:15.847093+0000 mgr.y (mgr.34107) 2438 : cluster [DBG] pgmap v4048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:16 smithi121 bash[36902]: audit 2022-04-22T18:36:16.591456+0000 mon.a (mon.0) 5455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:17.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:16 smithi115 bash[45495]: audit 2022-04-22T18:36:16.591456+0000 mon.a (mon.0) 5455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:17.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:16 smithi115 bash[46934]: audit 2022-04-22T18:36:16.591456+0000 mon.a (mon.0) 5455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:17 smithi121 bash[36902]: audit 2022-04-22T18:36:17.181636+0000 mon.a (mon.0) 5456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:17 smithi121 bash[36902]: cluster 2022-04-22T18:36:17.848251+0000 mgr.y (mgr.34107) 2439 : cluster [DBG] pgmap v4049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:17 smithi115 bash[45495]: audit 2022-04-22T18:36:17.181636+0000 mon.a (mon.0) 5456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:17 smithi115 bash[45495]: cluster 2022-04-22T18:36:17.848251+0000 mgr.y (mgr.34107) 2439 : cluster [DBG] pgmap v4049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:17 smithi115 bash[46934]: audit 2022-04-22T18:36:17.181636+0000 mon.a (mon.0) 5456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:17 smithi115 bash[46934]: cluster 2022-04-22T18:36:17.848251+0000 mgr.y (mgr.34107) 2439 : cluster [DBG] pgmap v4049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:19 smithi121 bash[36902]: cluster 2022-04-22T18:36:19.849387+0000 mgr.y (mgr.34107) 2440 : cluster [DBG] pgmap v4050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:20.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:19 smithi115 bash[45495]: cluster 2022-04-22T18:36:19.849387+0000 mgr.y (mgr.34107) 2440 : cluster [DBG] pgmap v4050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:19 smithi115 bash[46934]: cluster 2022-04-22T18:36:19.849387+0000 mgr.y (mgr.34107) 2440 : cluster [DBG] pgmap v4050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:21 smithi121 bash[36902]: audit 2022-04-22T18:36:21.594533+0000 mon.a (mon.0) 5457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:22.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:21 smithi115 bash[45495]: audit 2022-04-22T18:36:21.594533+0000 mon.a (mon.0) 5457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:21.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:22.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:21 smithi115 bash[46934]: audit 2022-04-22T18:36:21.594533+0000 mon.a (mon.0) 5457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:22 smithi121 bash[36902]: cluster 2022-04-22T18:36:21.850468+0000 mgr.y (mgr.34107) 2441 : cluster [DBG] pgmap v4051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:22 smithi121 bash[36902]: audit 2022-04-22T18:36:22.185515+0000 mon.a (mon.0) 5458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:22 smithi115 bash[45495]: cluster 2022-04-22T18:36:21.850468+0000 mgr.y (mgr.34107) 2441 : cluster [DBG] pgmap v4051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:23.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:22 smithi115 bash[45495]: audit 2022-04-22T18:36:22.185515+0000 mon.a (mon.0) 5458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:22 smithi115 bash[46934]: cluster 2022-04-22T18:36:21.850468+0000 mgr.y (mgr.34107) 2441 : cluster [DBG] pgmap v4051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:23.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:22 smithi115 bash[46934]: audit 2022-04-22T18:36:22.185515+0000 mon.a (mon.0) 5458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:24.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:23.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:24 smithi121 bash[36902]: cluster 2022-04-22T18:36:23.851578+0000 mgr.y (mgr.34107) 2442 : cluster [DBG] pgmap v4052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:25.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:24 smithi115 bash[45495]: cluster 2022-04-22T18:36:23.851578+0000 mgr.y (mgr.34107) 2442 : cluster [DBG] pgmap v4052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:25.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:24 smithi115 bash[46934]: cluster 2022-04-22T18:36:23.851578+0000 mgr.y (mgr.34107) 2442 : cluster [DBG] pgmap v4052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:26 smithi121 bash[36902]: cluster 2022-04-22T18:36:25.852558+0000 mgr.y (mgr.34107) 2443 : cluster [DBG] pgmap v4053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:26 smithi121 bash[36902]: audit 2022-04-22T18:36:26.597543+0000 mon.a (mon.0) 5459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:26 smithi115 bash[45495]: cluster 2022-04-22T18:36:25.852558+0000 mgr.y (mgr.34107) 2443 : cluster [DBG] pgmap v4053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:26 smithi115 bash[45495]: audit 2022-04-22T18:36:26.597543+0000 mon.a (mon.0) 5459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:26 smithi115 bash[46934]: cluster 2022-04-22T18:36:25.852558+0000 mgr.y (mgr.34107) 2443 : cluster [DBG] pgmap v4053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:26 smithi115 bash[46934]: audit 2022-04-22T18:36:26.597543+0000 mon.a (mon.0) 5459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:27 smithi121 bash[36902]: audit 2022-04-22T18:36:27.189307+0000 mon.a (mon.0) 5460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:27 smithi115 bash[45495]: audit 2022-04-22T18:36:27.189307+0000 mon.a (mon.0) 5460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:27 smithi115 bash[46934]: audit 2022-04-22T18:36:27.189307+0000 mon.a (mon.0) 5460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:28 smithi121 bash[36902]: cluster 2022-04-22T18:36:27.853645+0000 mgr.y (mgr.34107) 2444 : cluster [DBG] pgmap v4054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:29.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:28 smithi115 bash[45495]: cluster 2022-04-22T18:36:27.853645+0000 mgr.y (mgr.34107) 2444 : cluster [DBG] pgmap v4054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:29.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:28 smithi115 bash[46934]: cluster 2022-04-22T18:36:27.853645+0000 mgr.y (mgr.34107) 2444 : cluster [DBG] pgmap v4054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:30 smithi121 bash[36902]: cluster 2022-04-22T18:36:29.854740+0000 mgr.y (mgr.34107) 2445 : cluster [DBG] pgmap v4055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:31.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:30 smithi115 bash[45495]: cluster 2022-04-22T18:36:29.854740+0000 mgr.y (mgr.34107) 2445 : cluster [DBG] pgmap v4055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:31.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:30 smithi115 bash[46934]: cluster 2022-04-22T18:36:29.854740+0000 mgr.y (mgr.34107) 2445 : cluster [DBG] pgmap v4055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:31 smithi121 bash[36902]: audit 2022-04-22T18:36:31.600531+0000 mon.a (mon.0) 5461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:31 smithi115 bash[45495]: audit 2022-04-22T18:36:31.600531+0000 mon.a (mon.0) 5461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:31.869Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:31 smithi115 bash[46934]: audit 2022-04-22T18:36:31.600531+0000 mon.a (mon.0) 5461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:32 smithi121 bash[36902]: cluster 2022-04-22T18:36:31.855771+0000 mgr.y (mgr.34107) 2446 : cluster [DBG] pgmap v4056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:32 smithi121 bash[36902]: audit 2022-04-22T18:36:32.193005+0000 mon.a (mon.0) 5462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:32 smithi115 bash[45495]: cluster 2022-04-22T18:36:31.855771+0000 mgr.y (mgr.34107) 2446 : cluster [DBG] pgmap v4056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:32 smithi115 bash[45495]: audit 2022-04-22T18:36:32.193005+0000 mon.a (mon.0) 5462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:32 smithi115 bash[46934]: cluster 2022-04-22T18:36:31.855771+0000 mgr.y (mgr.34107) 2446 : cluster [DBG] pgmap v4056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:32 smithi115 bash[46934]: audit 2022-04-22T18:36:32.193005+0000 mon.a (mon.0) 5462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:34.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:33.747Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:34 smithi121 bash[36902]: cluster 2022-04-22T18:36:33.856917+0000 mgr.y (mgr.34107) 2447 : cluster [DBG] pgmap v4057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:35.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:34 smithi115 bash[45495]: cluster 2022-04-22T18:36:33.856917+0000 mgr.y (mgr.34107) 2447 : cluster [DBG] pgmap v4057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:35.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:34 smithi115 bash[46934]: cluster 2022-04-22T18:36:33.856917+0000 mgr.y (mgr.34107) 2447 : cluster [DBG] pgmap v4057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:36 smithi121 bash[36902]: cluster 2022-04-22T18:36:35.857961+0000 mgr.y (mgr.34107) 2448 : cluster [DBG] pgmap v4058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:37.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:36 smithi121 bash[36902]: audit 2022-04-22T18:36:36.603679+0000 mon.a (mon.0) 5463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:36 smithi115 bash[45495]: cluster 2022-04-22T18:36:35.857961+0000 mgr.y (mgr.34107) 2448 : cluster [DBG] pgmap v4058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:36 smithi115 bash[45495]: audit 2022-04-22T18:36:36.603679+0000 mon.a (mon.0) 5463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:36 smithi115 bash[46934]: cluster 2022-04-22T18:36:35.857961+0000 mgr.y (mgr.34107) 2448 : cluster [DBG] pgmap v4058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:36 smithi115 bash[46934]: audit 2022-04-22T18:36:36.603679+0000 mon.a (mon.0) 5463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:37 smithi121 bash[36902]: audit 2022-04-22T18:36:37.196670+0000 mon.a (mon.0) 5464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:37 smithi115 bash[45495]: audit 2022-04-22T18:36:37.196670+0000 mon.a (mon.0) 5464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:37 smithi115 bash[46934]: audit 2022-04-22T18:36:37.196670+0000 mon.a (mon.0) 5464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:38 smithi121 bash[36902]: cluster 2022-04-22T18:36:37.859051+0000 mgr.y (mgr.34107) 2449 : cluster [DBG] pgmap v4059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:38 smithi115 bash[45495]: cluster 2022-04-22T18:36:37.859051+0000 mgr.y (mgr.34107) 2449 : cluster [DBG] pgmap v4059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:38 smithi115 bash[46934]: cluster 2022-04-22T18:36:37.859051+0000 mgr.y (mgr.34107) 2449 : cluster [DBG] pgmap v4059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:40.944 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:36:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:40 smithi121 bash[36902]: cluster 2022-04-22T18:36:39.859960+0000 mgr.y (mgr.34107) 2450 : cluster [DBG] pgmap v4060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:41.289 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:36:41.290 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (59m) 7m ago 73m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:36:41.290 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (73m) 8m ago 73m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:36:41.290 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (61m) 7m ago 74m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:36:41.290 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (61m) 8m ago 74m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:36:41.291 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (69m) 8m ago 78m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:36:41.291 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (69m) 7m ago 80m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:36:41.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (68m) 7m ago 81m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:36:41.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (67m) 8m ago 79m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:36:41.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (68m) 7m ago 79m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:36:41.292 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (60m) 7m ago 73m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:36:41.292 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (60m) 8m ago 73m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:36:41.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (66m) 7m ago 78m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:36:41.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (65m) 7m ago 77m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:36:41.293 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (65m) 7m ago 77m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:36:41.293 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (64m) 7m ago 76m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:36:41.293 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (64m) 8m ago 76m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:36:41.293 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (63m) 8m ago 76m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:36:41.294 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (62m) 8m ago 75m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:36:41.294 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (62m) 8m ago 75m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:36:41.294 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (60m) 8m ago 73m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:36:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:40 smithi115 bash[45495]: cluster 2022-04-22T18:36:39.859960+0000 mgr.y (mgr.34107) 2450 : cluster [DBG] pgmap v4060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:41.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:40 smithi115 bash[46934]: cluster 2022-04-22T18:36:39.859960+0000 mgr.y (mgr.34107) 2450 : cluster [DBG] pgmap v4060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:41.691 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:36:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:36:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:41 smithi121 bash[36902]: audit 2022-04-22T18:36:40.918625+0000 mgr.y (mgr.34107) 2451 : audit [DBG] from='client.47932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:41 smithi121 bash[36902]: audit 2022-04-22T18:36:41.261217+0000 mgr.y (mgr.34107) 2452 : audit [DBG] from='client.38004 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:41 smithi121 bash[36902]: audit 2022-04-22T18:36:41.606426+0000 mon.a (mon.0) 5465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:41 smithi121 bash[36902]: audit 2022-04-22T18:36:41.683151+0000 mon.a (mon.0) 5466 : audit [DBG] from='client.? 172.21.15.115:0/161714638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:42.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:41.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[45495]: audit 2022-04-22T18:36:40.918625+0000 mgr.y (mgr.34107) 2451 : audit [DBG] from='client.47932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[45495]: audit 2022-04-22T18:36:41.261217+0000 mgr.y (mgr.34107) 2452 : audit [DBG] from='client.38004 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[45495]: audit 2022-04-22T18:36:41.606426+0000 mon.a (mon.0) 5465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[45495]: audit 2022-04-22T18:36:41.683151+0000 mon.a (mon.0) 5466 : audit [DBG] from='client.? 172.21.15.115:0/161714638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[46934]: audit 2022-04-22T18:36:40.918625+0000 mgr.y (mgr.34107) 2451 : audit [DBG] from='client.47932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[46934]: audit 2022-04-22T18:36:41.261217+0000 mgr.y (mgr.34107) 2452 : audit [DBG] from='client.38004 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:36:42.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[46934]: audit 2022-04-22T18:36:41.606426+0000 mon.a (mon.0) 5465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:41 smithi115 bash[46934]: audit 2022-04-22T18:36:41.683151+0000 mon.a (mon.0) 5466 : audit [DBG] from='client.? 172.21.15.115:0/161714638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:36:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:42 smithi121 bash[36902]: cluster 2022-04-22T18:36:41.861004+0000 mgr.y (mgr.34107) 2453 : cluster [DBG] pgmap v4061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:42 smithi121 bash[36902]: audit 2022-04-22T18:36:42.200259+0000 mon.a (mon.0) 5467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:42 smithi115 bash[45495]: cluster 2022-04-22T18:36:41.861004+0000 mgr.y (mgr.34107) 2453 : cluster [DBG] pgmap v4061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:42 smithi115 bash[45495]: audit 2022-04-22T18:36:42.200259+0000 mon.a (mon.0) 5467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:42 smithi115 bash[46934]: cluster 2022-04-22T18:36:41.861004+0000 mgr.y (mgr.34107) 2453 : cluster [DBG] pgmap v4061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:42 smithi115 bash[46934]: audit 2022-04-22T18:36:42.200259+0000 mon.a (mon.0) 5467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:43.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:44 smithi121 bash[36902]: cluster 2022-04-22T18:36:43.862085+0000 mgr.y (mgr.34107) 2454 : cluster [DBG] pgmap v4062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:45.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:44 smithi115 bash[45495]: cluster 2022-04-22T18:36:43.862085+0000 mgr.y (mgr.34107) 2454 : cluster [DBG] pgmap v4062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:45.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:44 smithi115 bash[46934]: cluster 2022-04-22T18:36:43.862085+0000 mgr.y (mgr.34107) 2454 : cluster [DBG] pgmap v4062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:46 smithi121 bash[36902]: cluster 2022-04-22T18:36:45.863080+0000 mgr.y (mgr.34107) 2455 : cluster [DBG] pgmap v4063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:46 smithi121 bash[36902]: audit 2022-04-22T18:36:46.609556+0000 mon.a (mon.0) 5468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:46 smithi115 bash[45495]: cluster 2022-04-22T18:36:45.863080+0000 mgr.y (mgr.34107) 2455 : cluster [DBG] pgmap v4063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:46 smithi115 bash[45495]: audit 2022-04-22T18:36:46.609556+0000 mon.a (mon.0) 5468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:46 smithi115 bash[46934]: cluster 2022-04-22T18:36:45.863080+0000 mgr.y (mgr.34107) 2455 : cluster [DBG] pgmap v4063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:46 smithi115 bash[46934]: audit 2022-04-22T18:36:46.609556+0000 mon.a (mon.0) 5468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:47 smithi121 bash[36902]: audit 2022-04-22T18:36:47.204024+0000 mon.a (mon.0) 5469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:47 smithi115 bash[45495]: audit 2022-04-22T18:36:47.204024+0000 mon.a (mon.0) 5469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:47 smithi115 bash[46934]: audit 2022-04-22T18:36:47.204024+0000 mon.a (mon.0) 5469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:48 smithi121 bash[36902]: cluster 2022-04-22T18:36:47.864187+0000 mgr.y (mgr.34107) 2456 : cluster [DBG] pgmap v4064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:49.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:48 smithi115 bash[46934]: cluster 2022-04-22T18:36:47.864187+0000 mgr.y (mgr.34107) 2456 : cluster [DBG] pgmap v4064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:48 smithi115 bash[45495]: cluster 2022-04-22T18:36:47.864187+0000 mgr.y (mgr.34107) 2456 : cluster [DBG] pgmap v4064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:50 smithi121 bash[36902]: cluster 2022-04-22T18:36:49.865262+0000 mgr.y (mgr.34107) 2457 : cluster [DBG] pgmap v4065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:50 smithi115 bash[45495]: cluster 2022-04-22T18:36:49.865262+0000 mgr.y (mgr.34107) 2457 : cluster [DBG] pgmap v4065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:50 smithi115 bash[46934]: cluster 2022-04-22T18:36:49.865262+0000 mgr.y (mgr.34107) 2457 : cluster [DBG] pgmap v4065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:52 smithi121 bash[36902]: audit 2022-04-22T18:36:51.612447+0000 mon.a (mon.0) 5470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:51 smithi115 bash[45495]: audit 2022-04-22T18:36:51.612447+0000 mon.a (mon.0) 5470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:52.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:51 smithi115 bash[46934]: audit 2022-04-22T18:36:51.612447+0000 mon.a (mon.0) 5470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:51.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:53 smithi121 bash[36902]: cluster 2022-04-22T18:36:51.866297+0000 mgr.y (mgr.34107) 2458 : cluster [DBG] pgmap v4066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:53.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:53 smithi121 bash[36902]: audit 2022-04-22T18:36:52.207610+0000 mon.a (mon.0) 5471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:52 smithi115 bash[46934]: cluster 2022-04-22T18:36:51.866297+0000 mgr.y (mgr.34107) 2458 : cluster [DBG] pgmap v4066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:52 smithi115 bash[46934]: audit 2022-04-22T18:36:52.207610+0000 mon.a (mon.0) 5471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:52 smithi115 bash[45495]: cluster 2022-04-22T18:36:51.866297+0000 mgr.y (mgr.34107) 2458 : cluster [DBG] pgmap v4066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:52 smithi115 bash[45495]: audit 2022-04-22T18:36:52.207610+0000 mon.a (mon.0) 5471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:54.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:36:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:36:53.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:36:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:55 smithi121 bash[36902]: cluster 2022-04-22T18:36:53.867397+0000 mgr.y (mgr.34107) 2459 : cluster [DBG] pgmap v4067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:55.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:54 smithi115 bash[45495]: cluster 2022-04-22T18:36:53.867397+0000 mgr.y (mgr.34107) 2459 : cluster [DBG] pgmap v4067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:55.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:54 smithi115 bash[46934]: cluster 2022-04-22T18:36:53.867397+0000 mgr.y (mgr.34107) 2459 : cluster [DBG] pgmap v4067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:57 smithi121 bash[36902]: cluster 2022-04-22T18:36:55.868483+0000 mgr.y (mgr.34107) 2460 : cluster [DBG] pgmap v4068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:57 smithi121 bash[36902]: audit 2022-04-22T18:36:56.615614+0000 mon.a (mon.0) 5472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:57 smithi115 bash[45495]: cluster 2022-04-22T18:36:55.868483+0000 mgr.y (mgr.34107) 2460 : cluster [DBG] pgmap v4068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:57 smithi115 bash[45495]: audit 2022-04-22T18:36:56.615614+0000 mon.a (mon.0) 5472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:57 smithi115 bash[46934]: cluster 2022-04-22T18:36:55.868483+0000 mgr.y (mgr.34107) 2460 : cluster [DBG] pgmap v4068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:57 smithi115 bash[46934]: audit 2022-04-22T18:36:56.615614+0000 mon.a (mon.0) 5472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:36:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:58 smithi115 bash[45495]: audit 2022-04-22T18:36:57.211542+0000 mon.a (mon.0) 5473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:58 smithi115 bash[46934]: audit 2022-04-22T18:36:57.211542+0000 mon.a (mon.0) 5473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:58 smithi121 bash[36902]: audit 2022-04-22T18:36:57.211542+0000 mon.a (mon.0) 5473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:36:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:36:59 smithi115 bash[45495]: cluster 2022-04-22T18:36:57.869550+0000 mgr.y (mgr.34107) 2461 : cluster [DBG] pgmap v4069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:36:59 smithi115 bash[46934]: cluster 2022-04-22T18:36:57.869550+0000 mgr.y (mgr.34107) 2461 : cluster [DBG] pgmap v4069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:36:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:36:59 smithi121 bash[36902]: cluster 2022-04-22T18:36:57.869550+0000 mgr.y (mgr.34107) 2461 : cluster [DBG] pgmap v4069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:01 smithi115 bash[45495]: cluster 2022-04-22T18:36:59.870647+0000 mgr.y (mgr.34107) 2462 : cluster [DBG] pgmap v4070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:01 smithi115 bash[46934]: cluster 2022-04-22T18:36:59.870647+0000 mgr.y (mgr.34107) 2462 : cluster [DBG] pgmap v4070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:01.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:01 smithi121 bash[36902]: cluster 2022-04-22T18:36:59.870647+0000 mgr.y (mgr.34107) 2462 : cluster [DBG] pgmap v4070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:02 smithi115 bash[46934]: audit 2022-04-22T18:37:01.618600+0000 mon.a (mon.0) 5474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:02.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:02 smithi115 bash[45495]: audit 2022-04-22T18:37:01.618600+0000 mon.a (mon.0) 5474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:01.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:02 smithi121 bash[36902]: audit 2022-04-22T18:37:01.618600+0000 mon.a (mon.0) 5474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:03 smithi115 bash[45495]: cluster 2022-04-22T18:37:01.871585+0000 mgr.y (mgr.34107) 2463 : cluster [DBG] pgmap v4071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:03 smithi115 bash[45495]: audit 2022-04-22T18:37:02.215188+0000 mon.a (mon.0) 5475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:03 smithi115 bash[46934]: cluster 2022-04-22T18:37:01.871585+0000 mgr.y (mgr.34107) 2463 : cluster [DBG] pgmap v4071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:03 smithi115 bash[46934]: audit 2022-04-22T18:37:02.215188+0000 mon.a (mon.0) 5475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:03 smithi121 bash[36902]: cluster 2022-04-22T18:37:01.871585+0000 mgr.y (mgr.34107) 2463 : cluster [DBG] pgmap v4071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:03 smithi121 bash[36902]: audit 2022-04-22T18:37:02.215188+0000 mon.a (mon.0) 5475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:03.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:05 smithi115 bash[45495]: cluster 2022-04-22T18:37:03.872609+0000 mgr.y (mgr.34107) 2464 : cluster [DBG] pgmap v4072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:05.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:05 smithi115 bash[46934]: cluster 2022-04-22T18:37:03.872609+0000 mgr.y (mgr.34107) 2464 : cluster [DBG] pgmap v4072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:05 smithi121 bash[36902]: cluster 2022-04-22T18:37:03.872609+0000 mgr.y (mgr.34107) 2464 : cluster [DBG] pgmap v4072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:07 smithi121 bash[36902]: cluster 2022-04-22T18:37:05.873668+0000 mgr.y (mgr.34107) 2465 : cluster [DBG] pgmap v4073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:07 smithi121 bash[36902]: audit 2022-04-22T18:37:06.622518+0000 mon.a (mon.0) 5476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:07 smithi121 bash[36902]: audit 2022-04-22T18:37:06.943003+0000 mon.a (mon.0) 5477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:37:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:07 smithi121 bash[36902]: audit 2022-04-22T18:37:06.946067+0000 mon.a (mon.0) 5478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:37:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:07 smithi121 bash[36902]: audit 2022-04-22T18:37:06.947797+0000 mon.a (mon.0) 5479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:37:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[45495]: cluster 2022-04-22T18:37:05.873668+0000 mgr.y (mgr.34107) 2465 : cluster [DBG] pgmap v4073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:07.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[45495]: audit 2022-04-22T18:37:06.622518+0000 mon.a (mon.0) 5476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:07.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[45495]: audit 2022-04-22T18:37:06.943003+0000 mon.a (mon.0) 5477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:37:07.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[45495]: audit 2022-04-22T18:37:06.946067+0000 mon.a (mon.0) 5478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:37:07.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[45495]: audit 2022-04-22T18:37:06.947797+0000 mon.a (mon.0) 5479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:37:07.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[46934]: cluster 2022-04-22T18:37:05.873668+0000 mgr.y (mgr.34107) 2465 : cluster [DBG] pgmap v4073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:07.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[46934]: audit 2022-04-22T18:37:06.622518+0000 mon.a (mon.0) 5476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:07.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[46934]: audit 2022-04-22T18:37:06.943003+0000 mon.a (mon.0) 5477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:37:07.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[46934]: audit 2022-04-22T18:37:06.946067+0000 mon.a (mon.0) 5478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:37:07.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:07 smithi115 bash[46934]: audit 2022-04-22T18:37:06.947797+0000 mon.a (mon.0) 5479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:37:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:08 smithi121 bash[36902]: audit 2022-04-22T18:37:07.218611+0000 mon.a (mon.0) 5480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:08 smithi121 bash[36902]: audit 2022-04-22T18:37:07.530016+0000 mon.a (mon.0) 5481 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:08 smithi115 bash[45495]: audit 2022-04-22T18:37:07.218611+0000 mon.a (mon.0) 5480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:08 smithi115 bash[45495]: audit 2022-04-22T18:37:07.530016+0000 mon.a (mon.0) 5481 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:08 smithi115 bash[46934]: audit 2022-04-22T18:37:07.218611+0000 mon.a (mon.0) 5480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:08 smithi115 bash[46934]: audit 2022-04-22T18:37:07.530016+0000 mon.a (mon.0) 5481 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:09.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:09 smithi115 bash[45495]: cluster 2022-04-22T18:37:07.874707+0000 mgr.y (mgr.34107) 2466 : cluster [DBG] pgmap v4074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:09.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:09 smithi115 bash[46934]: cluster 2022-04-22T18:37:07.874707+0000 mgr.y (mgr.34107) 2466 : cluster [DBG] pgmap v4074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:09 smithi121 bash[36902]: cluster 2022-04-22T18:37:07.874707+0000 mgr.y (mgr.34107) 2466 : cluster [DBG] pgmap v4074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:11.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:11 smithi115 bash[45495]: cluster 2022-04-22T18:37:09.875806+0000 mgr.y (mgr.34107) 2467 : cluster [DBG] pgmap v4075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:11 smithi115 bash[46934]: cluster 2022-04-22T18:37:09.875806+0000 mgr.y (mgr.34107) 2467 : cluster [DBG] pgmap v4075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:11.829 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:11 smithi121 bash[36902]: cluster 2022-04-22T18:37:09.875806+0000 mgr.y (mgr.34107) 2467 : cluster [DBG] pgmap v4075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:12.065 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:37:12.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:11.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:12.659 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:37:12.659 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (60m) 7m ago 74m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:37:12.659 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (73m) 9m ago 73m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (62m) 7m ago 75m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (61m) 9m ago 75m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (69m) 9m ago 79m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (70m) 7m ago 81m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (69m) 7m ago 81m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (68m) 9m ago 79m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (68m) 7m ago 79m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (61m) 7m ago 74m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:37:12.660 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (61m) 9m ago 74m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (66m) 7m ago 78m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (66m) 7m ago 78m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (65m) 7m ago 77m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (65m) 7m ago 77m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (64m) 9m ago 77m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (64m) 9m ago 76m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (63m) 9m ago 76m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (62m) 9m ago 75m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:37:12.661 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (60m) 9m ago 74m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:37:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:12 smithi115 bash[45495]: audit 2022-04-22T18:37:11.625617+0000 mon.a (mon.0) 5482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:12 smithi115 bash[45495]: audit 2022-04-22T18:37:12.221855+0000 mon.a (mon.0) 5483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:12 smithi115 bash[46934]: audit 2022-04-22T18:37:11.625617+0000 mon.a (mon.0) 5482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:12 smithi115 bash[46934]: audit 2022-04-22T18:37:12.221855+0000 mon.a (mon.0) 5483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:37:12.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:37:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:37:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:37:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:12 smithi121 bash[36902]: audit 2022-04-22T18:37:11.625617+0000 mon.a (mon.0) 5482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:12 smithi121 bash[36902]: audit 2022-04-22T18:37:12.221855+0000 mon.a (mon.0) 5483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:13.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[46934]: cluster 2022-04-22T18:37:11.876773+0000 mgr.y (mgr.34107) 2468 : cluster [DBG] pgmap v4076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[46934]: audit 2022-04-22T18:37:12.042103+0000 mgr.y (mgr.34107) 2469 : audit [DBG] from='client.38016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[46934]: audit 2022-04-22T18:37:12.382110+0000 mgr.y (mgr.34107) 2470 : audit [DBG] from='client.38022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[46934]: audit 2022-04-22T18:37:12.811288+0000 mon.b (mon.2) 14 : audit [DBG] from='client.? 172.21.15.115:0/2402845065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[45495]: cluster 2022-04-22T18:37:11.876773+0000 mgr.y (mgr.34107) 2468 : cluster [DBG] pgmap v4076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[45495]: audit 2022-04-22T18:37:12.042103+0000 mgr.y (mgr.34107) 2469 : audit [DBG] from='client.38016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[45495]: audit 2022-04-22T18:37:12.382110+0000 mgr.y (mgr.34107) 2470 : audit [DBG] from='client.38022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:13 smithi115 bash[45495]: audit 2022-04-22T18:37:12.811288+0000 mon.b (mon.2) 14 : audit [DBG] from='client.? 172.21.15.115:0/2402845065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:13 smithi121 bash[36902]: cluster 2022-04-22T18:37:11.876773+0000 mgr.y (mgr.34107) 2468 : cluster [DBG] pgmap v4076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:13 smithi121 bash[36902]: audit 2022-04-22T18:37:12.042103+0000 mgr.y (mgr.34107) 2469 : audit [DBG] from='client.38016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:13 smithi121 bash[36902]: audit 2022-04-22T18:37:12.382110+0000 mgr.y (mgr.34107) 2470 : audit [DBG] from='client.38022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:14.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:13 smithi121 bash[36902]: audit 2022-04-22T18:37:12.811288+0000 mon.b (mon.2) 14 : audit [DBG] from='client.? 172.21.15.115:0/2402845065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:14 smithi115 bash[46934]: audit 2022-04-22T18:37:14.068907+0000 mon.a (mon.0) 5484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:37:15.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:14 smithi115 bash[46934]: audit 2022-04-22T18:37:14.075266+0000 mon.a (mon.0) 5485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:37:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:14 smithi115 bash[45495]: audit 2022-04-22T18:37:14.068907+0000 mon.a (mon.0) 5484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:37:15.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:14 smithi115 bash[45495]: audit 2022-04-22T18:37:14.075266+0000 mon.a (mon.0) 5485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:37:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:14 smithi121 bash[36902]: audit 2022-04-22T18:37:14.068907+0000 mon.a (mon.0) 5484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:37:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:14 smithi121 bash[36902]: audit 2022-04-22T18:37:14.075266+0000 mon.a (mon.0) 5485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:37:16.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:15 smithi115 bash[45495]: cluster 2022-04-22T18:37:13.877949+0000 mgr.y (mgr.34107) 2471 : cluster [DBG] pgmap v4077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:16.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:15 smithi115 bash[46934]: cluster 2022-04-22T18:37:13.877949+0000 mgr.y (mgr.34107) 2471 : cluster [DBG] pgmap v4077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:16.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:15 smithi121 bash[36902]: cluster 2022-04-22T18:37:13.877949+0000 mgr.y (mgr.34107) 2471 : cluster [DBG] pgmap v4077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:17 smithi121 bash[36902]: cluster 2022-04-22T18:37:15.879026+0000 mgr.y (mgr.34107) 2472 : cluster [DBG] pgmap v4078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:17 smithi121 bash[36902]: cephadm 2022-04-22T18:37:16.111594+0000 mgr.y (mgr.34107) 2473 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:37:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:17 smithi121 bash[36902]: audit 2022-04-22T18:37:16.122270+0000 mon.a (mon.0) 5486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:17 smithi121 bash[36902]: audit 2022-04-22T18:37:16.128591+0000 mon.a (mon.0) 5487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:17 smithi121 bash[36902]: audit 2022-04-22T18:37:16.628905+0000 mon.a (mon.0) 5488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[45495]: cluster 2022-04-22T18:37:15.879026+0000 mgr.y (mgr.34107) 2472 : cluster [DBG] pgmap v4078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[45495]: cephadm 2022-04-22T18:37:16.111594+0000 mgr.y (mgr.34107) 2473 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:37:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[45495]: audit 2022-04-22T18:37:16.122270+0000 mon.a (mon.0) 5486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[45495]: audit 2022-04-22T18:37:16.128591+0000 mon.a (mon.0) 5487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[45495]: audit 2022-04-22T18:37:16.628905+0000 mon.a (mon.0) 5488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[46934]: cluster 2022-04-22T18:37:15.879026+0000 mgr.y (mgr.34107) 2472 : cluster [DBG] pgmap v4078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[46934]: cephadm 2022-04-22T18:37:16.111594+0000 mgr.y (mgr.34107) 2473 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T18:37:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[46934]: audit 2022-04-22T18:37:16.122270+0000 mon.a (mon.0) 5486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[46934]: audit 2022-04-22T18:37:16.128591+0000 mon.a (mon.0) 5487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:37:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:17 smithi115 bash[46934]: audit 2022-04-22T18:37:16.628905+0000 mon.a (mon.0) 5488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:18 smithi121 bash[36902]: audit 2022-04-22T18:37:17.224590+0000 mon.a (mon.0) 5489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:18 smithi115 bash[45495]: audit 2022-04-22T18:37:17.224590+0000 mon.a (mon.0) 5489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:18 smithi115 bash[46934]: audit 2022-04-22T18:37:17.224590+0000 mon.a (mon.0) 5489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:19 smithi121 bash[36902]: cluster 2022-04-22T18:37:17.880111+0000 mgr.y (mgr.34107) 2474 : cluster [DBG] pgmap v4079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:19 smithi115 bash[45495]: cluster 2022-04-22T18:37:17.880111+0000 mgr.y (mgr.34107) 2474 : cluster [DBG] pgmap v4079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:19 smithi115 bash[46934]: cluster 2022-04-22T18:37:17.880111+0000 mgr.y (mgr.34107) 2474 : cluster [DBG] pgmap v4079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:21 smithi121 bash[36902]: cluster 2022-04-22T18:37:19.881222+0000 mgr.y (mgr.34107) 2475 : cluster [DBG] pgmap v4080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:21 smithi115 bash[45495]: cluster 2022-04-22T18:37:19.881222+0000 mgr.y (mgr.34107) 2475 : cluster [DBG] pgmap v4080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:21 smithi115 bash[46934]: cluster 2022-04-22T18:37:19.881222+0000 mgr.y (mgr.34107) 2475 : cluster [DBG] pgmap v4080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:22.145 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:21.873Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:22 smithi121 bash[36902]: audit 2022-04-22T18:37:21.631892+0000 mon.a (mon.0) 5490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:22.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:22 smithi115 bash[45495]: audit 2022-04-22T18:37:21.631892+0000 mon.a (mon.0) 5490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:22 smithi115 bash[46934]: audit 2022-04-22T18:37:21.631892+0000 mon.a (mon.0) 5490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:23 smithi121 bash[36902]: cluster 2022-04-22T18:37:21.882370+0000 mgr.y (mgr.34107) 2476 : cluster [DBG] pgmap v4081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:23 smithi121 bash[36902]: audit 2022-04-22T18:37:22.228365+0000 mon.a (mon.0) 5491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:23 smithi115 bash[45495]: cluster 2022-04-22T18:37:21.882370+0000 mgr.y (mgr.34107) 2476 : cluster [DBG] pgmap v4081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:23.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:23 smithi115 bash[45495]: audit 2022-04-22T18:37:22.228365+0000 mon.a (mon.0) 5491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:23.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:23 smithi115 bash[46934]: cluster 2022-04-22T18:37:21.882370+0000 mgr.y (mgr.34107) 2476 : cluster [DBG] pgmap v4081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:23.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:23 smithi115 bash[46934]: audit 2022-04-22T18:37:22.228365+0000 mon.a (mon.0) 5491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:23.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:25 smithi121 bash[36902]: cluster 2022-04-22T18:37:23.883561+0000 mgr.y (mgr.34107) 2477 : cluster [DBG] pgmap v4082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:25 smithi115 bash[45495]: cluster 2022-04-22T18:37:23.883561+0000 mgr.y (mgr.34107) 2477 : cluster [DBG] pgmap v4082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:25 smithi115 bash[46934]: cluster 2022-04-22T18:37:23.883561+0000 mgr.y (mgr.34107) 2477 : cluster [DBG] pgmap v4082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:27 smithi121 bash[36902]: cluster 2022-04-22T18:37:25.884523+0000 mgr.y (mgr.34107) 2478 : cluster [DBG] pgmap v4083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:27 smithi121 bash[36902]: audit 2022-04-22T18:37:26.634924+0000 mon.a (mon.0) 5492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:27 smithi115 bash[45495]: cluster 2022-04-22T18:37:25.884523+0000 mgr.y (mgr.34107) 2478 : cluster [DBG] pgmap v4083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:27 smithi115 bash[45495]: audit 2022-04-22T18:37:26.634924+0000 mon.a (mon.0) 5492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:27 smithi115 bash[46934]: cluster 2022-04-22T18:37:25.884523+0000 mgr.y (mgr.34107) 2478 : cluster [DBG] pgmap v4083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:27 smithi115 bash[46934]: audit 2022-04-22T18:37:26.634924+0000 mon.a (mon.0) 5492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:28 smithi121 bash[36902]: audit 2022-04-22T18:37:27.234845+0000 mon.a (mon.0) 5493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:28 smithi115 bash[45495]: audit 2022-04-22T18:37:27.234845+0000 mon.a (mon.0) 5493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:28 smithi115 bash[46934]: audit 2022-04-22T18:37:27.234845+0000 mon.a (mon.0) 5493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:29 smithi121 bash[36902]: cluster 2022-04-22T18:37:27.885608+0000 mgr.y (mgr.34107) 2479 : cluster [DBG] pgmap v4084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:29 smithi115 bash[45495]: cluster 2022-04-22T18:37:27.885608+0000 mgr.y (mgr.34107) 2479 : cluster [DBG] pgmap v4084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:29 smithi115 bash[46934]: cluster 2022-04-22T18:37:27.885608+0000 mgr.y (mgr.34107) 2479 : cluster [DBG] pgmap v4084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:31 smithi121 bash[36902]: cluster 2022-04-22T18:37:29.886720+0000 mgr.y (mgr.34107) 2480 : cluster [DBG] pgmap v4085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:31 smithi115 bash[45495]: cluster 2022-04-22T18:37:29.886720+0000 mgr.y (mgr.34107) 2480 : cluster [DBG] pgmap v4085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:31 smithi115 bash[46934]: cluster 2022-04-22T18:37:29.886720+0000 mgr.y (mgr.34107) 2480 : cluster [DBG] pgmap v4085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:32.179 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:31.874Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:32 smithi121 bash[36902]: audit 2022-04-22T18:37:31.637998+0000 mon.a (mon.0) 5494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:32 smithi115 bash[45495]: audit 2022-04-22T18:37:31.637998+0000 mon.a (mon.0) 5494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:32.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:32 smithi115 bash[46934]: audit 2022-04-22T18:37:31.637998+0000 mon.a (mon.0) 5494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:33 smithi121 bash[36902]: cluster 2022-04-22T18:37:31.887882+0000 mgr.y (mgr.34107) 2481 : cluster [DBG] pgmap v4086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:33 smithi121 bash[36902]: audit 2022-04-22T18:37:32.238723+0000 mon.a (mon.0) 5495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:33 smithi115 bash[45495]: cluster 2022-04-22T18:37:31.887882+0000 mgr.y (mgr.34107) 2481 : cluster [DBG] pgmap v4086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:33 smithi115 bash[45495]: audit 2022-04-22T18:37:32.238723+0000 mon.a (mon.0) 5495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:33 smithi115 bash[46934]: cluster 2022-04-22T18:37:31.887882+0000 mgr.y (mgr.34107) 2481 : cluster [DBG] pgmap v4086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:33 smithi115 bash[46934]: audit 2022-04-22T18:37:32.238723+0000 mon.a (mon.0) 5495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:33.751Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:35 smithi121 bash[36902]: cluster 2022-04-22T18:37:33.888991+0000 mgr.y (mgr.34107) 2482 : cluster [DBG] pgmap v4087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:35 smithi115 bash[45495]: cluster 2022-04-22T18:37:33.888991+0000 mgr.y (mgr.34107) 2482 : cluster [DBG] pgmap v4087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:35 smithi115 bash[46934]: cluster 2022-04-22T18:37:33.888991+0000 mgr.y (mgr.34107) 2482 : cluster [DBG] pgmap v4087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:37 smithi121 bash[36902]: cluster 2022-04-22T18:37:35.889807+0000 mgr.y (mgr.34107) 2483 : cluster [DBG] pgmap v4088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:37.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:37 smithi121 bash[36902]: audit 2022-04-22T18:37:36.641278+0000 mon.a (mon.0) 5496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:37 smithi115 bash[45495]: cluster 2022-04-22T18:37:35.889807+0000 mgr.y (mgr.34107) 2483 : cluster [DBG] pgmap v4088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:37 smithi115 bash[45495]: audit 2022-04-22T18:37:36.641278+0000 mon.a (mon.0) 5496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:37 smithi115 bash[46934]: cluster 2022-04-22T18:37:35.889807+0000 mgr.y (mgr.34107) 2483 : cluster [DBG] pgmap v4088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:37 smithi115 bash[46934]: audit 2022-04-22T18:37:36.641278+0000 mon.a (mon.0) 5496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:38 smithi121 bash[36902]: audit 2022-04-22T18:37:37.242389+0000 mon.a (mon.0) 5497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:38 smithi115 bash[45495]: audit 2022-04-22T18:37:37.242389+0000 mon.a (mon.0) 5497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:38 smithi115 bash[46934]: audit 2022-04-22T18:37:37.242389+0000 mon.a (mon.0) 5497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:39 smithi121 bash[36902]: cluster 2022-04-22T18:37:37.890892+0000 mgr.y (mgr.34107) 2484 : cluster [DBG] pgmap v4089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:39 smithi115 bash[45495]: cluster 2022-04-22T18:37:37.890892+0000 mgr.y (mgr.34107) 2484 : cluster [DBG] pgmap v4089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:39 smithi115 bash[46934]: cluster 2022-04-22T18:37:37.890892+0000 mgr.y (mgr.34107) 2484 : cluster [DBG] pgmap v4089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:41 smithi121 bash[36902]: cluster 2022-04-22T18:37:39.891922+0000 mgr.y (mgr.34107) 2485 : cluster [DBG] pgmap v4090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:41 smithi115 bash[45495]: cluster 2022-04-22T18:37:39.891922+0000 mgr.y (mgr.34107) 2485 : cluster [DBG] pgmap v4090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:41 smithi115 bash[46934]: cluster 2022-04-22T18:37:39.891922+0000 mgr.y (mgr.34107) 2485 : cluster [DBG] pgmap v4090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:42.216 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:41.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:42 smithi121 bash[36902]: audit 2022-04-22T18:37:41.644286+0000 mon.a (mon.0) 5498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:42.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:42 smithi115 bash[45495]: audit 2022-04-22T18:37:41.644286+0000 mon.a (mon.0) 5498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:42.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:42 smithi115 bash[46934]: audit 2022-04-22T18:37:41.644286+0000 mon.a (mon.0) 5498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:43.191 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:37:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:43 smithi121 bash[36902]: cluster 2022-04-22T18:37:41.892997+0000 mgr.y (mgr.34107) 2486 : cluster [DBG] pgmap v4091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:43.562 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:43 smithi121 bash[36902]: audit 2022-04-22T18:37:42.245924+0000 mon.a (mon.0) 5499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (60m) 8m ago 74m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (74m) 9m ago 74m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (62m) 8m ago 75m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (62m) 9m ago 75m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:37:43.563 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (70m) 9m ago 79m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (70m) 8m ago 82m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (69m) 8m ago 82m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (68m) 9m ago 80m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (69m) 8m ago 80m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (61m) 8m ago 74m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (61m) 9m ago 74m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (67m) 8m ago 79m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (66m) 8m ago 78m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (66m) 8m ago 78m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:37:43.564 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (65m) 8m ago 77m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:37:43.565 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (65m) 9m ago 77m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:37:43.565 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (64m) 9m ago 77m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:37:43.565 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (63m) 9m ago 76m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:37:43.565 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (63m) 9m ago 76m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:37:43.565 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (61m) 9m ago 74m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:37:43.565 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:43 smithi115 bash[45495]: cluster 2022-04-22T18:37:41.892997+0000 mgr.y (mgr.34107) 2486 : cluster [DBG] pgmap v4091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:43.565 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:43 smithi115 bash[45495]: audit 2022-04-22T18:37:42.245924+0000 mon.a (mon.0) 5499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:43.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:43 smithi115 bash[46934]: cluster 2022-04-22T18:37:41.892997+0000 mgr.y (mgr.34107) 2486 : cluster [DBG] pgmap v4091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:43.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:43 smithi115 bash[46934]: audit 2022-04-22T18:37:42.245924+0000 mon.a (mon.0) 5499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:43.949 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:37:43.949 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:37:43.949 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:37:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:37:43.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:37:43.951 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:37:43.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:37:43.951 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:37:43.951 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:37:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:43.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:44 smithi121 bash[36902]: audit 2022-04-22T18:37:43.169276+0000 mgr.y (mgr.34107) 2487 : audit [DBG] from='client.38034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:44 smithi121 bash[36902]: audit 2022-04-22T18:37:43.514353+0000 mgr.y (mgr.34107) 2488 : audit [DBG] from='client.47974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:44 smithi121 bash[36902]: audit 2022-04-22T18:37:43.941608+0000 mon.c (mon.1) 150 : audit [DBG] from='client.? 172.21.15.115:0/1187667214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[45495]: audit 2022-04-22T18:37:43.169276+0000 mgr.y (mgr.34107) 2487 : audit [DBG] from='client.38034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[45495]: audit 2022-04-22T18:37:43.514353+0000 mgr.y (mgr.34107) 2488 : audit [DBG] from='client.47974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[45495]: audit 2022-04-22T18:37:43.941608+0000 mon.c (mon.1) 150 : audit [DBG] from='client.? 172.21.15.115:0/1187667214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[46934]: audit 2022-04-22T18:37:43.169276+0000 mgr.y (mgr.34107) 2487 : audit [DBG] from='client.38034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[46934]: audit 2022-04-22T18:37:43.514353+0000 mgr.y (mgr.34107) 2488 : audit [DBG] from='client.47974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:37:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:44 smithi115 bash[46934]: audit 2022-04-22T18:37:43.941608+0000 mon.c (mon.1) 150 : audit [DBG] from='client.? 172.21.15.115:0/1187667214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:37:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:45 smithi121 bash[36902]: cluster 2022-04-22T18:37:43.894108+0000 mgr.y (mgr.34107) 2489 : cluster [DBG] pgmap v4092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:45 smithi115 bash[45495]: cluster 2022-04-22T18:37:43.894108+0000 mgr.y (mgr.34107) 2489 : cluster [DBG] pgmap v4092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:45 smithi115 bash[46934]: cluster 2022-04-22T18:37:43.894108+0000 mgr.y (mgr.34107) 2489 : cluster [DBG] pgmap v4092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:47 smithi121 bash[36902]: cluster 2022-04-22T18:37:45.895110+0000 mgr.y (mgr.34107) 2490 : cluster [DBG] pgmap v4093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:47 smithi121 bash[36902]: audit 2022-04-22T18:37:46.647103+0000 mon.a (mon.0) 5500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:47 smithi115 bash[45495]: cluster 2022-04-22T18:37:45.895110+0000 mgr.y (mgr.34107) 2490 : cluster [DBG] pgmap v4093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:47 smithi115 bash[45495]: audit 2022-04-22T18:37:46.647103+0000 mon.a (mon.0) 5500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:47 smithi115 bash[46934]: cluster 2022-04-22T18:37:45.895110+0000 mgr.y (mgr.34107) 2490 : cluster [DBG] pgmap v4093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:47 smithi115 bash[46934]: audit 2022-04-22T18:37:46.647103+0000 mon.a (mon.0) 5500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:48 smithi121 bash[36902]: audit 2022-04-22T18:37:47.249377+0000 mon.a (mon.0) 5501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:48 smithi115 bash[45495]: audit 2022-04-22T18:37:47.249377+0000 mon.a (mon.0) 5501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:48 smithi115 bash[46934]: audit 2022-04-22T18:37:47.249377+0000 mon.a (mon.0) 5501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:49 smithi121 bash[36902]: cluster 2022-04-22T18:37:47.896369+0000 mgr.y (mgr.34107) 2491 : cluster [DBG] pgmap v4094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:49 smithi115 bash[45495]: cluster 2022-04-22T18:37:47.896369+0000 mgr.y (mgr.34107) 2491 : cluster [DBG] pgmap v4094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:49 smithi115 bash[46934]: cluster 2022-04-22T18:37:47.896369+0000 mgr.y (mgr.34107) 2491 : cluster [DBG] pgmap v4094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:51 smithi121 bash[36902]: cluster 2022-04-22T18:37:49.897392+0000 mgr.y (mgr.34107) 2492 : cluster [DBG] pgmap v4095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:51 smithi115 bash[46934]: cluster 2022-04-22T18:37:49.897392+0000 mgr.y (mgr.34107) 2492 : cluster [DBG] pgmap v4095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:51 smithi115 bash[45495]: cluster 2022-04-22T18:37:49.897392+0000 mgr.y (mgr.34107) 2492 : cluster [DBG] pgmap v4095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:52.248 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:51.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:52 smithi121 bash[36902]: audit 2022-04-22T18:37:51.650121+0000 mon.a (mon.0) 5502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:52.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:52 smithi115 bash[45495]: audit 2022-04-22T18:37:51.650121+0000 mon.a (mon.0) 5502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:52.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:52 smithi115 bash[46934]: audit 2022-04-22T18:37:51.650121+0000 mon.a (mon.0) 5502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:53 smithi121 bash[36902]: cluster 2022-04-22T18:37:51.898330+0000 mgr.y (mgr.34107) 2493 : cluster [DBG] pgmap v4096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:53 smithi121 bash[36902]: audit 2022-04-22T18:37:52.252807+0000 mon.a (mon.0) 5503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:53 smithi115 bash[46934]: cluster 2022-04-22T18:37:51.898330+0000 mgr.y (mgr.34107) 2493 : cluster [DBG] pgmap v4096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:53 smithi115 bash[46934]: audit 2022-04-22T18:37:52.252807+0000 mon.a (mon.0) 5503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:53 smithi115 bash[45495]: cluster 2022-04-22T18:37:51.898330+0000 mgr.y (mgr.34107) 2493 : cluster [DBG] pgmap v4096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:53 smithi115 bash[45495]: audit 2022-04-22T18:37:52.252807+0000 mon.a (mon.0) 5503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:37:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:37:53.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:55 smithi121 bash[36902]: cluster 2022-04-22T18:37:53.899374+0000 mgr.y (mgr.34107) 2494 : cluster [DBG] pgmap v4097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:55 smithi115 bash[46934]: cluster 2022-04-22T18:37:53.899374+0000 mgr.y (mgr.34107) 2494 : cluster [DBG] pgmap v4097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:55 smithi115 bash[45495]: cluster 2022-04-22T18:37:53.899374+0000 mgr.y (mgr.34107) 2494 : cluster [DBG] pgmap v4097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:57 smithi121 bash[36902]: cluster 2022-04-22T18:37:55.900261+0000 mgr.y (mgr.34107) 2495 : cluster [DBG] pgmap v4098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:57 smithi121 bash[36902]: audit 2022-04-22T18:37:56.653210+0000 mon.a (mon.0) 5504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:57 smithi115 bash[45495]: cluster 2022-04-22T18:37:55.900261+0000 mgr.y (mgr.34107) 2495 : cluster [DBG] pgmap v4098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:57 smithi115 bash[45495]: audit 2022-04-22T18:37:56.653210+0000 mon.a (mon.0) 5504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:57 smithi115 bash[46934]: cluster 2022-04-22T18:37:55.900261+0000 mgr.y (mgr.34107) 2495 : cluster [DBG] pgmap v4098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:57 smithi115 bash[46934]: audit 2022-04-22T18:37:56.653210+0000 mon.a (mon.0) 5504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:37:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:58 smithi115 bash[45495]: audit 2022-04-22T18:37:57.256132+0000 mon.a (mon.0) 5505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:58 smithi115 bash[46934]: audit 2022-04-22T18:37:57.256132+0000 mon.a (mon.0) 5505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:58 smithi121 bash[36902]: audit 2022-04-22T18:37:57.256132+0000 mon.a (mon.0) 5505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:37:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:37:59 smithi115 bash[45495]: cluster 2022-04-22T18:37:57.901423+0000 mgr.y (mgr.34107) 2496 : cluster [DBG] pgmap v4099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:37:59 smithi115 bash[46934]: cluster 2022-04-22T18:37:57.901423+0000 mgr.y (mgr.34107) 2496 : cluster [DBG] pgmap v4099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:37:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:37:59 smithi121 bash[36902]: cluster 2022-04-22T18:37:57.901423+0000 mgr.y (mgr.34107) 2496 : cluster [DBG] pgmap v4099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:01 smithi115 bash[45495]: cluster 2022-04-22T18:37:59.902559+0000 mgr.y (mgr.34107) 2497 : cluster [DBG] pgmap v4100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:01 smithi115 bash[46934]: cluster 2022-04-22T18:37:59.902559+0000 mgr.y (mgr.34107) 2497 : cluster [DBG] pgmap v4100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:01 smithi121 bash[36902]: cluster 2022-04-22T18:37:59.902559+0000 mgr.y (mgr.34107) 2497 : cluster [DBG] pgmap v4100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:02.275 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:01.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:02.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:02 smithi115 bash[45495]: audit 2022-04-22T18:38:01.656147+0000 mon.a (mon.0) 5506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:02.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:02 smithi115 bash[45495]: audit 2022-04-22T18:38:02.259243+0000 mon.a (mon.0) 5507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:02 smithi115 bash[46934]: audit 2022-04-22T18:38:01.656147+0000 mon.a (mon.0) 5506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:02 smithi115 bash[46934]: audit 2022-04-22T18:38:02.259243+0000 mon.a (mon.0) 5507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:02 smithi121 bash[36902]: audit 2022-04-22T18:38:01.656147+0000 mon.a (mon.0) 5506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:02 smithi121 bash[36902]: audit 2022-04-22T18:38:02.259243+0000 mon.a (mon.0) 5507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:04.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:03 smithi115 bash[46934]: cluster 2022-04-22T18:38:01.903693+0000 mgr.y (mgr.34107) 2498 : cluster [DBG] pgmap v4101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:03.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:03 smithi115 bash[45495]: cluster 2022-04-22T18:38:01.903693+0000 mgr.y (mgr.34107) 2498 : cluster [DBG] pgmap v4101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:03 smithi121 bash[36902]: cluster 2022-04-22T18:38:01.903693+0000 mgr.y (mgr.34107) 2498 : cluster [DBG] pgmap v4101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:05 smithi121 bash[36902]: cluster 2022-04-22T18:38:03.904814+0000 mgr.y (mgr.34107) 2499 : cluster [DBG] pgmap v4102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:05 smithi115 bash[45495]: cluster 2022-04-22T18:38:03.904814+0000 mgr.y (mgr.34107) 2499 : cluster [DBG] pgmap v4102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:05 smithi115 bash[46934]: cluster 2022-04-22T18:38:03.904814+0000 mgr.y (mgr.34107) 2499 : cluster [DBG] pgmap v4102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:06 smithi121 bash[36902]: audit 2022-04-22T18:38:06.659066+0000 mon.a (mon.0) 5508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:06 smithi115 bash[45495]: audit 2022-04-22T18:38:06.659066+0000 mon.a (mon.0) 5508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:06 smithi115 bash[46934]: audit 2022-04-22T18:38:06.659066+0000 mon.a (mon.0) 5508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:07 smithi121 bash[36902]: cluster 2022-04-22T18:38:05.905862+0000 mgr.y (mgr.34107) 2500 : cluster [DBG] pgmap v4103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:07 smithi121 bash[36902]: audit 2022-04-22T18:38:07.262965+0000 mon.a (mon.0) 5509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:07 smithi115 bash[45495]: cluster 2022-04-22T18:38:05.905862+0000 mgr.y (mgr.34107) 2500 : cluster [DBG] pgmap v4103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:07 smithi115 bash[45495]: audit 2022-04-22T18:38:07.262965+0000 mon.a (mon.0) 5509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:07 smithi115 bash[46934]: cluster 2022-04-22T18:38:05.905862+0000 mgr.y (mgr.34107) 2500 : cluster [DBG] pgmap v4103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:07 smithi115 bash[46934]: audit 2022-04-22T18:38:07.262965+0000 mon.a (mon.0) 5509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:09 smithi121 bash[36902]: cluster 2022-04-22T18:38:07.906963+0000 mgr.y (mgr.34107) 2501 : cluster [DBG] pgmap v4104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:10.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:09 smithi115 bash[45495]: cluster 2022-04-22T18:38:07.906963+0000 mgr.y (mgr.34107) 2501 : cluster [DBG] pgmap v4104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:09 smithi115 bash[46934]: cluster 2022-04-22T18:38:07.906963+0000 mgr.y (mgr.34107) 2501 : cluster [DBG] pgmap v4104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:11 smithi121 bash[36902]: cluster 2022-04-22T18:38:09.908103+0000 mgr.y (mgr.34107) 2502 : cluster [DBG] pgmap v4105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:11 smithi121 bash[36902]: audit 2022-04-22T18:38:11.662325+0000 mon.a (mon.0) 5510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:11 smithi115 bash[45495]: cluster 2022-04-22T18:38:09.908103+0000 mgr.y (mgr.34107) 2502 : cluster [DBG] pgmap v4105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:11 smithi115 bash[45495]: audit 2022-04-22T18:38:11.662325+0000 mon.a (mon.0) 5510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:11.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:11 smithi115 bash[46934]: cluster 2022-04-22T18:38:09.908103+0000 mgr.y (mgr.34107) 2502 : cluster [DBG] pgmap v4105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:12.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:11 smithi115 bash[46934]: audit 2022-04-22T18:38:11.662325+0000 mon.a (mon.0) 5510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:12 smithi121 bash[36902]: audit 2022-04-22T18:38:12.266673+0000 mon.a (mon.0) 5511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:12 smithi115 bash[45495]: audit 2022-04-22T18:38:12.266673+0000 mon.a (mon.0) 5511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:13.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:12 smithi115 bash[46934]: audit 2022-04-22T18:38:12.266673+0000 mon.a (mon.0) 5511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:13 smithi115 bash[45495]: cluster 2022-04-22T18:38:11.909252+0000 mgr.y (mgr.34107) 2503 : cluster [DBG] pgmap v4106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:13.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:14.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:13 smithi115 bash[46934]: cluster 2022-04-22T18:38:11.909252+0000 mgr.y (mgr.34107) 2503 : cluster [DBG] pgmap v4106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:13 smithi121 bash[36902]: cluster 2022-04-22T18:38:11.909252+0000 mgr.y (mgr.34107) 2503 : cluster [DBG] pgmap v4106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (61m) 9m ago 75m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (74m) 10m ago 74m 25.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (63m) 9m ago 76m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (62m) 10m ago 76m 20.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:38:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (70m) 10m ago 80m 413M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (71m) 9m ago 82m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (70m) 9m ago 82m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (69m) 10m ago 80m 144M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (69m) 9m ago 80m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (62m) 9m ago 75m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (62m) 10m ago 75m 12.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (68m) 9m ago 79m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (67m) 9m ago 79m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (66m) 9m ago 78m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:38:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (66m) 9m ago 78m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:38:14.672 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (65m) 10m ago 78m 563M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:38:14.672 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (65m) 10m ago 77m 689M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:38:14.672 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (64m) 10m ago 77m 476M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:38:14.672 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (63m) 10m ago 76m 618M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:38:14.672 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (62m) 10m ago 75m 61.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:38:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:38:15.070 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:38:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:14 smithi121 bash[36902]: audit 2022-04-22T18:38:14.073360+0000 mon.a (mon.0) 5512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:38:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:14 smithi121 bash[36902]: audit 2022-04-22T18:38:14.080879+0000 mon.a (mon.0) 5513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:38:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:14 smithi115 bash[45495]: audit 2022-04-22T18:38:14.073360+0000 mon.a (mon.0) 5512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:38:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:14 smithi115 bash[45495]: audit 2022-04-22T18:38:14.080879+0000 mon.a (mon.0) 5513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:38:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:14 smithi115 bash[46934]: audit 2022-04-22T18:38:14.073360+0000 mon.a (mon.0) 5512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:38:15.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:14 smithi115 bash[46934]: audit 2022-04-22T18:38:14.080879+0000 mon.a (mon.0) 5513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:38:16.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[46934]: cluster 2022-04-22T18:38:13.910415+0000 mgr.y (mgr.34107) 2504 : cluster [DBG] pgmap v4107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:16.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[46934]: audit 2022-04-22T18:38:14.301161+0000 mgr.y (mgr.34107) 2505 : audit [DBG] from='client.38052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[46934]: audit 2022-04-22T18:38:14.642616+0000 mgr.y (mgr.34107) 2506 : audit [DBG] from='client.38058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[46934]: audit 2022-04-22T18:38:15.060821+0000 mon.a (mon.0) 5514 : audit [DBG] from='client.? 172.21.15.115:0/2495915115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:16.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[45495]: cluster 2022-04-22T18:38:13.910415+0000 mgr.y (mgr.34107) 2504 : cluster [DBG] pgmap v4107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:16.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[45495]: audit 2022-04-22T18:38:14.301161+0000 mgr.y (mgr.34107) 2505 : audit [DBG] from='client.38052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[45495]: audit 2022-04-22T18:38:14.642616+0000 mgr.y (mgr.34107) 2506 : audit [DBG] from='client.38058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:15 smithi115 bash[45495]: audit 2022-04-22T18:38:15.060821+0000 mon.a (mon.0) 5514 : audit [DBG] from='client.? 172.21.15.115:0/2495915115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:15 smithi121 bash[36902]: cluster 2022-04-22T18:38:13.910415+0000 mgr.y (mgr.34107) 2504 : cluster [DBG] pgmap v4107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:15 smithi121 bash[36902]: audit 2022-04-22T18:38:14.301161+0000 mgr.y (mgr.34107) 2505 : audit [DBG] from='client.38052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:15 smithi121 bash[36902]: audit 2022-04-22T18:38:14.642616+0000 mgr.y (mgr.34107) 2506 : audit [DBG] from='client.38058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:15 smithi121 bash[36902]: audit 2022-04-22T18:38:15.060821+0000 mon.a (mon.0) 5514 : audit [DBG] from='client.? 172.21.15.115:0/2495915115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[45495]: audit 2022-04-22T18:38:16.137206+0000 mon.a (mon.0) 5515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:38:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[45495]: audit 2022-04-22T18:38:16.140367+0000 mon.a (mon.0) 5516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:38:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[45495]: audit 2022-04-22T18:38:16.142287+0000 mon.a (mon.0) 5517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:38:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[45495]: audit 2022-04-22T18:38:16.665537+0000 mon.a (mon.0) 5518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[45495]: audit 2022-04-22T18:38:16.734604+0000 mon.a (mon.0) 5519 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[46934]: audit 2022-04-22T18:38:16.137206+0000 mon.a (mon.0) 5515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[46934]: audit 2022-04-22T18:38:16.140367+0000 mon.a (mon.0) 5516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[46934]: audit 2022-04-22T18:38:16.142287+0000 mon.a (mon.0) 5517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[46934]: audit 2022-04-22T18:38:16.665537+0000 mon.a (mon.0) 5518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:16 smithi115 bash[46934]: audit 2022-04-22T18:38:16.734604+0000 mon.a (mon.0) 5519 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:16 smithi121 bash[36902]: audit 2022-04-22T18:38:16.137206+0000 mon.a (mon.0) 5515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:38:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:16 smithi121 bash[36902]: audit 2022-04-22T18:38:16.140367+0000 mon.a (mon.0) 5516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:38:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:16 smithi121 bash[36902]: audit 2022-04-22T18:38:16.142287+0000 mon.a (mon.0) 5517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:38:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:16 smithi121 bash[36902]: audit 2022-04-22T18:38:16.665537+0000 mon.a (mon.0) 5518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:16 smithi121 bash[36902]: audit 2022-04-22T18:38:16.734604+0000 mon.a (mon.0) 5519 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:17 smithi121 bash[36902]: cluster 2022-04-22T18:38:15.911383+0000 mgr.y (mgr.34107) 2507 : cluster [DBG] pgmap v4108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:17 smithi121 bash[36902]: audit 2022-04-22T18:38:17.270259+0000 mon.a (mon.0) 5520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:17 smithi115 bash[46934]: cluster 2022-04-22T18:38:15.911383+0000 mgr.y (mgr.34107) 2507 : cluster [DBG] pgmap v4108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:17 smithi115 bash[46934]: audit 2022-04-22T18:38:17.270259+0000 mon.a (mon.0) 5520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:17 smithi115 bash[45495]: cluster 2022-04-22T18:38:15.911383+0000 mgr.y (mgr.34107) 2507 : cluster [DBG] pgmap v4108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:18.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:17 smithi115 bash[45495]: audit 2022-04-22T18:38:17.270259+0000 mon.a (mon.0) 5520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:20.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:19 smithi121 bash[36902]: cluster 2022-04-22T18:38:17.912416+0000 mgr.y (mgr.34107) 2508 : cluster [DBG] pgmap v4109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:20.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:19 smithi115 bash[45495]: cluster 2022-04-22T18:38:17.912416+0000 mgr.y (mgr.34107) 2508 : cluster [DBG] pgmap v4109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:19 smithi115 bash[46934]: cluster 2022-04-22T18:38:17.912416+0000 mgr.y (mgr.34107) 2508 : cluster [DBG] pgmap v4109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:21 smithi121 bash[36902]: cluster 2022-04-22T18:38:19.913529+0000 mgr.y (mgr.34107) 2509 : cluster [DBG] pgmap v4110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:21 smithi121 bash[36902]: audit 2022-04-22T18:38:20.688072+0000 mon.a (mon.0) 5521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:21 smithi121 bash[36902]: audit 2022-04-22T18:38:21.010014+0000 mon.a (mon.0) 5522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:21 smithi121 bash[36902]: audit 2022-04-22T18:38:21.668041+0000 mon.a (mon.0) 5523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:22.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:21.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[45495]: cluster 2022-04-22T18:38:19.913529+0000 mgr.y (mgr.34107) 2509 : cluster [DBG] pgmap v4110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[45495]: audit 2022-04-22T18:38:20.688072+0000 mon.a (mon.0) 5521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[45495]: audit 2022-04-22T18:38:21.010014+0000 mon.a (mon.0) 5522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[45495]: audit 2022-04-22T18:38:21.668041+0000 mon.a (mon.0) 5523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[46934]: cluster 2022-04-22T18:38:19.913529+0000 mgr.y (mgr.34107) 2509 : cluster [DBG] pgmap v4110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[46934]: audit 2022-04-22T18:38:20.688072+0000 mon.a (mon.0) 5521 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[46934]: audit 2022-04-22T18:38:21.010014+0000 mon.a (mon.0) 5522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:38:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:21 smithi115 bash[46934]: audit 2022-04-22T18:38:21.668041+0000 mon.a (mon.0) 5523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:22 smithi121 bash[36902]: audit 2022-04-22T18:38:22.274003+0000 mon.a (mon.0) 5524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:23.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:22 smithi115 bash[45495]: audit 2022-04-22T18:38:22.274003+0000 mon.a (mon.0) 5524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:23.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:22 smithi115 bash[46934]: audit 2022-04-22T18:38:22.274003+0000 mon.a (mon.0) 5524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:23 smithi115 bash[45495]: cluster 2022-04-22T18:38:21.914646+0000 mgr.y (mgr.34107) 2510 : cluster [DBG] pgmap v4111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:24.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:23.755Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:23 smithi115 bash[46934]: cluster 2022-04-22T18:38:21.914646+0000 mgr.y (mgr.34107) 2510 : cluster [DBG] pgmap v4111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:23 smithi121 bash[36902]: cluster 2022-04-22T18:38:21.914646+0000 mgr.y (mgr.34107) 2510 : cluster [DBG] pgmap v4111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:25 smithi121 bash[36902]: cluster 2022-04-22T18:38:23.915788+0000 mgr.y (mgr.34107) 2511 : cluster [DBG] pgmap v4112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:25 smithi115 bash[45495]: cluster 2022-04-22T18:38:23.915788+0000 mgr.y (mgr.34107) 2511 : cluster [DBG] pgmap v4112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:25 smithi115 bash[46934]: cluster 2022-04-22T18:38:23.915788+0000 mgr.y (mgr.34107) 2511 : cluster [DBG] pgmap v4112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:26 smithi121 bash[36902]: audit 2022-04-22T18:38:26.671043+0000 mon.a (mon.0) 5525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:26 smithi115 bash[45495]: audit 2022-04-22T18:38:26.671043+0000 mon.a (mon.0) 5525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:26 smithi115 bash[46934]: audit 2022-04-22T18:38:26.671043+0000 mon.a (mon.0) 5525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:27 smithi121 bash[36902]: cluster 2022-04-22T18:38:25.916734+0000 mgr.y (mgr.34107) 2512 : cluster [DBG] pgmap v4113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:27 smithi121 bash[36902]: audit 2022-04-22T18:38:27.277611+0000 mon.a (mon.0) 5526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:27 smithi115 bash[45495]: cluster 2022-04-22T18:38:25.916734+0000 mgr.y (mgr.34107) 2512 : cluster [DBG] pgmap v4113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:27 smithi115 bash[45495]: audit 2022-04-22T18:38:27.277611+0000 mon.a (mon.0) 5526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:27 smithi115 bash[46934]: cluster 2022-04-22T18:38:25.916734+0000 mgr.y (mgr.34107) 2512 : cluster [DBG] pgmap v4113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:27 smithi115 bash[46934]: audit 2022-04-22T18:38:27.277611+0000 mon.a (mon.0) 5526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:29 smithi121 bash[36902]: cluster 2022-04-22T18:38:27.917834+0000 mgr.y (mgr.34107) 2513 : cluster [DBG] pgmap v4114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:29 smithi115 bash[45495]: cluster 2022-04-22T18:38:27.917834+0000 mgr.y (mgr.34107) 2513 : cluster [DBG] pgmap v4114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:29 smithi115 bash[46934]: cluster 2022-04-22T18:38:27.917834+0000 mgr.y (mgr.34107) 2513 : cluster [DBG] pgmap v4114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:31 smithi121 bash[36902]: cluster 2022-04-22T18:38:29.919003+0000 mgr.y (mgr.34107) 2514 : cluster [DBG] pgmap v4115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:31 smithi121 bash[36902]: audit 2022-04-22T18:38:31.673968+0000 mon.a (mon.0) 5527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:31 smithi115 bash[45495]: cluster 2022-04-22T18:38:29.919003+0000 mgr.y (mgr.34107) 2514 : cluster [DBG] pgmap v4115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:31 smithi115 bash[45495]: audit 2022-04-22T18:38:31.673968+0000 mon.a (mon.0) 5527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:31.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:32.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:31 smithi115 bash[46934]: cluster 2022-04-22T18:38:29.919003+0000 mgr.y (mgr.34107) 2514 : cluster [DBG] pgmap v4115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:32.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:31 smithi115 bash[46934]: audit 2022-04-22T18:38:31.673968+0000 mon.a (mon.0) 5527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:32 smithi121 bash[36902]: audit 2022-04-22T18:38:32.281516+0000 mon.a (mon.0) 5528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:33.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:32 smithi115 bash[45495]: audit 2022-04-22T18:38:32.281516+0000 mon.a (mon.0) 5528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:32 smithi115 bash[46934]: audit 2022-04-22T18:38:32.281516+0000 mon.a (mon.0) 5528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:34.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:33 smithi115 bash[45495]: cluster 2022-04-22T18:38:31.920171+0000 mgr.y (mgr.34107) 2515 : cluster [DBG] pgmap v4116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:33.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:33 smithi115 bash[46934]: cluster 2022-04-22T18:38:31.920171+0000 mgr.y (mgr.34107) 2515 : cluster [DBG] pgmap v4116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:33 smithi121 bash[36902]: cluster 2022-04-22T18:38:31.920171+0000 mgr.y (mgr.34107) 2515 : cluster [DBG] pgmap v4116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:35 smithi121 bash[36902]: cluster 2022-04-22T18:38:33.921363+0000 mgr.y (mgr.34107) 2516 : cluster [DBG] pgmap v4117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:35 smithi115 bash[45495]: cluster 2022-04-22T18:38:33.921363+0000 mgr.y (mgr.34107) 2516 : cluster [DBG] pgmap v4117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:35 smithi115 bash[46934]: cluster 2022-04-22T18:38:33.921363+0000 mgr.y (mgr.34107) 2516 : cluster [DBG] pgmap v4117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:36 smithi121 bash[36902]: audit 2022-04-22T18:38:36.676958+0000 mon.a (mon.0) 5529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:36 smithi115 bash[45495]: audit 2022-04-22T18:38:36.676958+0000 mon.a (mon.0) 5529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:36 smithi115 bash[46934]: audit 2022-04-22T18:38:36.676958+0000 mon.a (mon.0) 5529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:37 smithi121 bash[36902]: cluster 2022-04-22T18:38:35.922315+0000 mgr.y (mgr.34107) 2517 : cluster [DBG] pgmap v4118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:37 smithi121 bash[36902]: audit 2022-04-22T18:38:37.285154+0000 mon.a (mon.0) 5530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:37 smithi115 bash[45495]: cluster 2022-04-22T18:38:35.922315+0000 mgr.y (mgr.34107) 2517 : cluster [DBG] pgmap v4118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:37 smithi115 bash[45495]: audit 2022-04-22T18:38:37.285154+0000 mon.a (mon.0) 5530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:37 smithi115 bash[46934]: cluster 2022-04-22T18:38:35.922315+0000 mgr.y (mgr.34107) 2517 : cluster [DBG] pgmap v4118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:37 smithi115 bash[46934]: audit 2022-04-22T18:38:37.285154+0000 mon.a (mon.0) 5530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:39 smithi121 bash[36902]: cluster 2022-04-22T18:38:37.923393+0000 mgr.y (mgr.34107) 2518 : cluster [DBG] pgmap v4119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:39 smithi115 bash[45495]: cluster 2022-04-22T18:38:37.923393+0000 mgr.y (mgr.34107) 2518 : cluster [DBG] pgmap v4119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:39 smithi115 bash[46934]: cluster 2022-04-22T18:38:37.923393+0000 mgr.y (mgr.34107) 2518 : cluster [DBG] pgmap v4119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:41 smithi121 bash[36902]: cluster 2022-04-22T18:38:39.924536+0000 mgr.y (mgr.34107) 2519 : cluster [DBG] pgmap v4120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:41 smithi121 bash[36902]: audit 2022-04-22T18:38:41.679919+0000 mon.a (mon.0) 5531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:42.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:41.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:41 smithi115 bash[46934]: cluster 2022-04-22T18:38:39.924536+0000 mgr.y (mgr.34107) 2519 : cluster [DBG] pgmap v4120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:42.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:41 smithi115 bash[46934]: audit 2022-04-22T18:38:41.679919+0000 mon.a (mon.0) 5531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:41 smithi115 bash[45495]: cluster 2022-04-22T18:38:39.924536+0000 mgr.y (mgr.34107) 2519 : cluster [DBG] pgmap v4120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:41 smithi115 bash[45495]: audit 2022-04-22T18:38:41.679919+0000 mon.a (mon.0) 5531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:42 smithi121 bash[36902]: audit 2022-04-22T18:38:42.288508+0000 mon.a (mon.0) 5532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:42 smithi115 bash[45495]: audit 2022-04-22T18:38:42.288508+0000 mon.a (mon.0) 5532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:42 smithi115 bash[46934]: audit 2022-04-22T18:38:42.288508+0000 mon.a (mon.0) 5532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:43 smithi115 bash[45495]: cluster 2022-04-22T18:38:41.925651+0000 mgr.y (mgr.34107) 2520 : cluster [DBG] pgmap v4121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:44.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:43 smithi115 bash[46934]: cluster 2022-04-22T18:38:41.925651+0000 mgr.y (mgr.34107) 2520 : cluster [DBG] pgmap v4121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:44.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:43.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:43 smithi121 bash[36902]: cluster 2022-04-22T18:38:41.925651+0000 mgr.y (mgr.34107) 2520 : cluster [DBG] pgmap v4121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (61m) 9m ago 75m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (75m) 25s ago 75m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (63m) 9m ago 76m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (63m) 25s ago 76m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:38:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (71m) 25s ago 80m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (71m) 9m ago 83m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (70m) 9m ago 83m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (69m) 25s ago 81m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (70m) 9m ago 81m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (62m) 9m ago 75m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (62m) 25s ago 75m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (68m) 9m ago 80m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (67m) 9m ago 80m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (67m) 9m ago 79m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:38:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (66m) 9m ago 78m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:38:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (66m) 25s ago 78m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:38:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (65m) 25s ago 78m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:38:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (64m) 25s ago 77m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:38:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (64m) 25s ago 77m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:38:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (62m) 25s ago 75m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:38:46.203 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:38:46.203 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:38:46.203 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:38:46.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:46.203 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:38:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:38:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:38:46.205 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:38:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:45 smithi121 bash[36902]: cluster 2022-04-22T18:38:43.926804+0000 mgr.y (mgr.34107) 2521 : cluster [DBG] pgmap v4122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:45 smithi115 bash[45495]: cluster 2022-04-22T18:38:43.926804+0000 mgr.y (mgr.34107) 2521 : cluster [DBG] pgmap v4122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:45 smithi115 bash[46934]: cluster 2022-04-22T18:38:43.926804+0000 mgr.y (mgr.34107) 2521 : cluster [DBG] pgmap v4122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:46 smithi121 bash[36902]: audit 2022-04-22T18:38:45.425883+0000 mgr.y (mgr.34107) 2522 : audit [DBG] from='client.48004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:46 smithi121 bash[36902]: audit 2022-04-22T18:38:45.768950+0000 mgr.y (mgr.34107) 2523 : audit [DBG] from='client.48010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:46 smithi121 bash[36902]: audit 2022-04-22T18:38:46.195297+0000 mon.a (mon.0) 5533 : audit [DBG] from='client.? 172.21.15.115:0/4269801150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:47.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:46 smithi121 bash[36902]: audit 2022-04-22T18:38:46.682787+0000 mon.a (mon.0) 5534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[45495]: audit 2022-04-22T18:38:45.425883+0000 mgr.y (mgr.34107) 2522 : audit [DBG] from='client.48004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[45495]: audit 2022-04-22T18:38:45.768950+0000 mgr.y (mgr.34107) 2523 : audit [DBG] from='client.48010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[45495]: audit 2022-04-22T18:38:46.195297+0000 mon.a (mon.0) 5533 : audit [DBG] from='client.? 172.21.15.115:0/4269801150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[45495]: audit 2022-04-22T18:38:46.682787+0000 mon.a (mon.0) 5534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[46934]: audit 2022-04-22T18:38:45.425883+0000 mgr.y (mgr.34107) 2522 : audit [DBG] from='client.48004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[46934]: audit 2022-04-22T18:38:45.768950+0000 mgr.y (mgr.34107) 2523 : audit [DBG] from='client.48010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:38:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[46934]: audit 2022-04-22T18:38:46.195297+0000 mon.a (mon.0) 5533 : audit [DBG] from='client.? 172.21.15.115:0/4269801150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:38:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:46 smithi115 bash[46934]: audit 2022-04-22T18:38:46.682787+0000 mon.a (mon.0) 5534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:47 smithi121 bash[36902]: cluster 2022-04-22T18:38:45.927372+0000 mgr.y (mgr.34107) 2524 : cluster [DBG] pgmap v4123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:47 smithi121 bash[36902]: audit 2022-04-22T18:38:47.292097+0000 mon.a (mon.0) 5535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:47 smithi115 bash[45495]: cluster 2022-04-22T18:38:45.927372+0000 mgr.y (mgr.34107) 2524 : cluster [DBG] pgmap v4123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:47 smithi115 bash[45495]: audit 2022-04-22T18:38:47.292097+0000 mon.a (mon.0) 5535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:47 smithi115 bash[46934]: cluster 2022-04-22T18:38:45.927372+0000 mgr.y (mgr.34107) 2524 : cluster [DBG] pgmap v4123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:47 smithi115 bash[46934]: audit 2022-04-22T18:38:47.292097+0000 mon.a (mon.0) 5535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:48 smithi121 bash[36902]: cluster 2022-04-22T18:38:47.928443+0000 mgr.y (mgr.34107) 2525 : cluster [DBG] pgmap v4124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:48 smithi115 bash[45495]: cluster 2022-04-22T18:38:47.928443+0000 mgr.y (mgr.34107) 2525 : cluster [DBG] pgmap v4124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:49.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:48 smithi115 bash[46934]: cluster 2022-04-22T18:38:47.928443+0000 mgr.y (mgr.34107) 2525 : cluster [DBG] pgmap v4124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:50 smithi121 bash[36902]: cluster 2022-04-22T18:38:49.929556+0000 mgr.y (mgr.34107) 2526 : cluster [DBG] pgmap v4125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:50 smithi115 bash[45495]: cluster 2022-04-22T18:38:49.929556+0000 mgr.y (mgr.34107) 2526 : cluster [DBG] pgmap v4125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:50 smithi115 bash[46934]: cluster 2022-04-22T18:38:49.929556+0000 mgr.y (mgr.34107) 2526 : cluster [DBG] pgmap v4125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:51 smithi121 bash[36902]: audit 2022-04-22T18:38:51.685702+0000 mon.a (mon.0) 5536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:51 smithi115 bash[45495]: audit 2022-04-22T18:38:51.685702+0000 mon.a (mon.0) 5536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:52.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:51 smithi115 bash[46934]: audit 2022-04-22T18:38:51.685702+0000 mon.a (mon.0) 5536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:51.879Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:52 smithi121 bash[36902]: cluster 2022-04-22T18:38:51.930659+0000 mgr.y (mgr.34107) 2527 : cluster [DBG] pgmap v4126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:52 smithi121 bash[36902]: audit 2022-04-22T18:38:52.295924+0000 mon.a (mon.0) 5537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:52 smithi115 bash[45495]: cluster 2022-04-22T18:38:51.930659+0000 mgr.y (mgr.34107) 2527 : cluster [DBG] pgmap v4126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:52 smithi115 bash[45495]: audit 2022-04-22T18:38:52.295924+0000 mon.a (mon.0) 5537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:52 smithi115 bash[46934]: cluster 2022-04-22T18:38:51.930659+0000 mgr.y (mgr.34107) 2527 : cluster [DBG] pgmap v4126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:52 smithi115 bash[46934]: audit 2022-04-22T18:38:52.295924+0000 mon.a (mon.0) 5537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:38:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:38:53.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:38:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:55 smithi121 bash[36902]: cluster 2022-04-22T18:38:53.931827+0000 mgr.y (mgr.34107) 2528 : cluster [DBG] pgmap v4127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:55.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:54 smithi115 bash[45495]: cluster 2022-04-22T18:38:53.931827+0000 mgr.y (mgr.34107) 2528 : cluster [DBG] pgmap v4127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:55.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:54 smithi115 bash[46934]: cluster 2022-04-22T18:38:53.931827+0000 mgr.y (mgr.34107) 2528 : cluster [DBG] pgmap v4127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:57 smithi121 bash[36902]: cluster 2022-04-22T18:38:55.932788+0000 mgr.y (mgr.34107) 2529 : cluster [DBG] pgmap v4128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:57 smithi121 bash[36902]: audit 2022-04-22T18:38:56.688770+0000 mon.a (mon.0) 5538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:56 smithi115 bash[45495]: cluster 2022-04-22T18:38:55.932788+0000 mgr.y (mgr.34107) 2529 : cluster [DBG] pgmap v4128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:56 smithi115 bash[45495]: audit 2022-04-22T18:38:56.688770+0000 mon.a (mon.0) 5538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:56 smithi115 bash[46934]: cluster 2022-04-22T18:38:55.932788+0000 mgr.y (mgr.34107) 2529 : cluster [DBG] pgmap v4128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:56 smithi115 bash[46934]: audit 2022-04-22T18:38:56.688770+0000 mon.a (mon.0) 5538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:38:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:58 smithi121 bash[36902]: audit 2022-04-22T18:38:57.299744+0000 mon.a (mon.0) 5539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:58 smithi115 bash[45495]: audit 2022-04-22T18:38:57.299744+0000 mon.a (mon.0) 5539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:58 smithi115 bash[46934]: audit 2022-04-22T18:38:57.299744+0000 mon.a (mon.0) 5539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:38:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:38:59 smithi115 bash[45495]: cluster 2022-04-22T18:38:57.933870+0000 mgr.y (mgr.34107) 2530 : cluster [DBG] pgmap v4129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:38:59 smithi115 bash[46934]: cluster 2022-04-22T18:38:57.933870+0000 mgr.y (mgr.34107) 2530 : cluster [DBG] pgmap v4129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:38:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:38:59 smithi121 bash[36902]: cluster 2022-04-22T18:38:57.933870+0000 mgr.y (mgr.34107) 2530 : cluster [DBG] pgmap v4129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:01 smithi115 bash[45495]: cluster 2022-04-22T18:38:59.935007+0000 mgr.y (mgr.34107) 2531 : cluster [DBG] pgmap v4130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:01 smithi115 bash[46934]: cluster 2022-04-22T18:38:59.935007+0000 mgr.y (mgr.34107) 2531 : cluster [DBG] pgmap v4130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:01 smithi121 bash[36902]: cluster 2022-04-22T18:38:59.935007+0000 mgr.y (mgr.34107) 2531 : cluster [DBG] pgmap v4130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:02.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:02 smithi115 bash[45495]: audit 2022-04-22T18:39:01.691777+0000 mon.a (mon.0) 5540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:02 smithi115 bash[46934]: audit 2022-04-22T18:39:01.691777+0000 mon.a (mon.0) 5540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:01.879Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:02 smithi121 bash[36902]: audit 2022-04-22T18:39:01.691777+0000 mon.a (mon.0) 5540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:03 smithi115 bash[45495]: cluster 2022-04-22T18:39:01.936200+0000 mgr.y (mgr.34107) 2532 : cluster [DBG] pgmap v4131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:03.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:03 smithi115 bash[45495]: audit 2022-04-22T18:39:02.303413+0000 mon.a (mon.0) 5541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:03 smithi115 bash[46934]: cluster 2022-04-22T18:39:01.936200+0000 mgr.y (mgr.34107) 2532 : cluster [DBG] pgmap v4131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:03 smithi115 bash[46934]: audit 2022-04-22T18:39:02.303413+0000 mon.a (mon.0) 5541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:03 smithi121 bash[36902]: cluster 2022-04-22T18:39:01.936200+0000 mgr.y (mgr.34107) 2532 : cluster [DBG] pgmap v4131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:03 smithi121 bash[36902]: audit 2022-04-22T18:39:02.303413+0000 mon.a (mon.0) 5541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:03.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:05 smithi115 bash[45495]: cluster 2022-04-22T18:39:03.937403+0000 mgr.y (mgr.34107) 2533 : cluster [DBG] pgmap v4132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:05.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:05 smithi115 bash[46934]: cluster 2022-04-22T18:39:03.937403+0000 mgr.y (mgr.34107) 2533 : cluster [DBG] pgmap v4132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:05 smithi121 bash[36902]: cluster 2022-04-22T18:39:03.937403+0000 mgr.y (mgr.34107) 2533 : cluster [DBG] pgmap v4132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:07 smithi115 bash[45495]: cluster 2022-04-22T18:39:05.938331+0000 mgr.y (mgr.34107) 2534 : cluster [DBG] pgmap v4133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:07.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:07 smithi115 bash[45495]: audit 2022-04-22T18:39:06.694750+0000 mon.a (mon.0) 5542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:07 smithi115 bash[46934]: cluster 2022-04-22T18:39:05.938331+0000 mgr.y (mgr.34107) 2534 : cluster [DBG] pgmap v4133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:07 smithi115 bash[46934]: audit 2022-04-22T18:39:06.694750+0000 mon.a (mon.0) 5542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:07 smithi121 bash[36902]: cluster 2022-04-22T18:39:05.938331+0000 mgr.y (mgr.34107) 2534 : cluster [DBG] pgmap v4133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:07 smithi121 bash[36902]: audit 2022-04-22T18:39:06.694750+0000 mon.a (mon.0) 5542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:08 smithi115 bash[45495]: audit 2022-04-22T18:39:07.306982+0000 mon.a (mon.0) 5543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:08.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:08 smithi115 bash[46934]: audit 2022-04-22T18:39:07.306982+0000 mon.a (mon.0) 5543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:08 smithi121 bash[36902]: audit 2022-04-22T18:39:07.306982+0000 mon.a (mon.0) 5543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:09.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:09 smithi115 bash[45495]: cluster 2022-04-22T18:39:07.939430+0000 mgr.y (mgr.34107) 2535 : cluster [DBG] pgmap v4134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:09.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:09 smithi115 bash[46934]: cluster 2022-04-22T18:39:07.939430+0000 mgr.y (mgr.34107) 2535 : cluster [DBG] pgmap v4134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:09 smithi121 bash[36902]: cluster 2022-04-22T18:39:07.939430+0000 mgr.y (mgr.34107) 2535 : cluster [DBG] pgmap v4134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:11 smithi121 bash[36902]: cluster 2022-04-22T18:39:09.940553+0000 mgr.y (mgr.34107) 2536 : cluster [DBG] pgmap v4135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:11 smithi115 bash[45495]: cluster 2022-04-22T18:39:09.940553+0000 mgr.y (mgr.34107) 2536 : cluster [DBG] pgmap v4135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:11 smithi115 bash[46934]: cluster 2022-04-22T18:39:09.940553+0000 mgr.y (mgr.34107) 2536 : cluster [DBG] pgmap v4135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:12 smithi115 bash[45495]: audit 2022-04-22T18:39:11.697832+0000 mon.a (mon.0) 5544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:11.880Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:12.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:12 smithi115 bash[46934]: audit 2022-04-22T18:39:11.697832+0000 mon.a (mon.0) 5544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:12 smithi121 bash[36902]: audit 2022-04-22T18:39:11.697832+0000 mon.a (mon.0) 5544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:13 smithi121 bash[36902]: cluster 2022-04-22T18:39:11.941637+0000 mgr.y (mgr.34107) 2537 : cluster [DBG] pgmap v4136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:13.562 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:13 smithi121 bash[36902]: audit 2022-04-22T18:39:12.310854+0000 mon.a (mon.0) 5545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:13.563 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:13 smithi115 bash[45495]: cluster 2022-04-22T18:39:11.941637+0000 mgr.y (mgr.34107) 2537 : cluster [DBG] pgmap v4136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:13.563 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:13 smithi115 bash[45495]: audit 2022-04-22T18:39:12.310854+0000 mon.a (mon.0) 5545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:13.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:13 smithi115 bash[46934]: cluster 2022-04-22T18:39:11.941637+0000 mgr.y (mgr.34107) 2537 : cluster [DBG] pgmap v4136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:13.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:13 smithi115 bash[46934]: audit 2022-04-22T18:39:12.310854+0000 mon.a (mon.0) 5545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:13.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:15 smithi121 bash[36902]: cluster 2022-04-22T18:39:13.942916+0000 mgr.y (mgr.34107) 2538 : cluster [DBG] pgmap v4137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:15 smithi121 bash[36902]: audit 2022-04-22T18:39:14.077089+0000 mon.a (mon.0) 5546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:39:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:15 smithi121 bash[36902]: audit 2022-04-22T18:39:14.086586+0000 mon.a (mon.0) 5547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:39:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[45495]: cluster 2022-04-22T18:39:13.942916+0000 mgr.y (mgr.34107) 2538 : cluster [DBG] pgmap v4137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[45495]: audit 2022-04-22T18:39:14.077089+0000 mon.a (mon.0) 5546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:39:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[45495]: audit 2022-04-22T18:39:14.086586+0000 mon.a (mon.0) 5547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:39:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[46934]: cluster 2022-04-22T18:39:13.942916+0000 mgr.y (mgr.34107) 2538 : cluster [DBG] pgmap v4137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[46934]: audit 2022-04-22T18:39:14.077089+0000 mon.a (mon.0) 5546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:39:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:15 smithi115 bash[46934]: audit 2022-04-22T18:39:14.086586+0000 mon.a (mon.0) 5547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:39:16.575 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:39:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (62m) 10m ago 76m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (75m) 56s ago 75m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (64m) 10m ago 77m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (63m) 56s ago 77m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (71m) 56s ago 81m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (72m) 10m ago 83m 473M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (71m) 10m ago 83m 221M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (70m) 56s ago 81m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (70m) 10m ago 81m 147M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (63m) 10m ago 76m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (63m) 56s ago 76m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (69m) 10m ago 81m 692M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (68m) 10m ago 80m 769M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (67m) 10m ago 80m 520M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (67m) 10m ago 79m 472M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (66m) 56s ago 79m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (66m) 56s ago 78m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (65m) 56s ago 78m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (64m) 56s ago 77m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (63m) 56s ago 76m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:39:17.334 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:39:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:17 smithi121 bash[36902]: cluster 2022-04-22T18:39:15.943942+0000 mgr.y (mgr.34107) 2539 : cluster [DBG] pgmap v4138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:17 smithi121 bash[36902]: audit 2022-04-22T18:39:16.551664+0000 mgr.y (mgr.34107) 2540 : audit [DBG] from='client.38088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:17 smithi121 bash[36902]: audit 2022-04-22T18:39:16.700643+0000 mon.a (mon.0) 5548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[45495]: cluster 2022-04-22T18:39:15.943942+0000 mgr.y (mgr.34107) 2539 : cluster [DBG] pgmap v4138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[45495]: audit 2022-04-22T18:39:16.551664+0000 mgr.y (mgr.34107) 2540 : audit [DBG] from='client.38088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[45495]: audit 2022-04-22T18:39:16.700643+0000 mon.a (mon.0) 5548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[46934]: cluster 2022-04-22T18:39:15.943942+0000 mgr.y (mgr.34107) 2539 : cluster [DBG] pgmap v4138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[46934]: audit 2022-04-22T18:39:16.551664+0000 mgr.y (mgr.34107) 2540 : audit [DBG] from='client.38088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:17 smithi115 bash[46934]: audit 2022-04-22T18:39:16.700643+0000 mon.a (mon.0) 5548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:18 smithi121 bash[36902]: audit 2022-04-22T18:39:16.894827+0000 mgr.y (mgr.34107) 2541 : audit [DBG] from='client.38094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:18 smithi121 bash[36902]: audit 2022-04-22T18:39:17.314235+0000 mon.a (mon.0) 5549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:18 smithi121 bash[36902]: audit 2022-04-22T18:39:17.325226+0000 mon.a (mon.0) 5550 : audit [DBG] from='client.? 172.21.15.115:0/3635253093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[45495]: audit 2022-04-22T18:39:16.894827+0000 mgr.y (mgr.34107) 2541 : audit [DBG] from='client.38094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[45495]: audit 2022-04-22T18:39:17.314235+0000 mon.a (mon.0) 5549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[45495]: audit 2022-04-22T18:39:17.325226+0000 mon.a (mon.0) 5550 : audit [DBG] from='client.? 172.21.15.115:0/3635253093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[46934]: audit 2022-04-22T18:39:16.894827+0000 mgr.y (mgr.34107) 2541 : audit [DBG] from='client.38094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[46934]: audit 2022-04-22T18:39:17.314235+0000 mon.a (mon.0) 5549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:18 smithi115 bash[46934]: audit 2022-04-22T18:39:17.325226+0000 mon.a (mon.0) 5550 : audit [DBG] from='client.? 172.21.15.115:0/3635253093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:19 smithi121 bash[36902]: cluster 2022-04-22T18:39:17.945054+0000 mgr.y (mgr.34107) 2542 : cluster [DBG] pgmap v4139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:19 smithi115 bash[45495]: cluster 2022-04-22T18:39:17.945054+0000 mgr.y (mgr.34107) 2542 : cluster [DBG] pgmap v4139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:19 smithi115 bash[46934]: cluster 2022-04-22T18:39:17.945054+0000 mgr.y (mgr.34107) 2542 : cluster [DBG] pgmap v4139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[46934]: cluster 2022-04-22T18:39:19.946199+0000 mgr.y (mgr.34107) 2543 : cluster [DBG] pgmap v4140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[46934]: audit 2022-04-22T18:39:21.020086+0000 mon.a (mon.0) 5551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[46934]: audit 2022-04-22T18:39:21.022924+0000 mon.a (mon.0) 5552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[46934]: audit 2022-04-22T18:39:21.024471+0000 mon.a (mon.0) 5553 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[45495]: cluster 2022-04-22T18:39:19.946199+0000 mgr.y (mgr.34107) 2543 : cluster [DBG] pgmap v4140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:21.335 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[45495]: audit 2022-04-22T18:39:21.020086+0000 mon.a (mon.0) 5551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:39:21.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[45495]: audit 2022-04-22T18:39:21.022924+0000 mon.a (mon.0) 5552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:39:21.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[45495]: audit 2022-04-22T18:39:21.024471+0000 mon.a (mon.0) 5553 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:39:21.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:21 smithi121 bash[36902]: cluster 2022-04-22T18:39:19.946199+0000 mgr.y (mgr.34107) 2543 : cluster [DBG] pgmap v4140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:21.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:21 smithi121 bash[36902]: audit 2022-04-22T18:39:21.020086+0000 mon.a (mon.0) 5551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:39:21.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:21 smithi121 bash[36902]: audit 2022-04-22T18:39:21.022924+0000 mon.a (mon.0) 5552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:39:21.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:21 smithi121 bash[36902]: audit 2022-04-22T18:39:21.024471+0000 mon.a (mon.0) 5553 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:39:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:21.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:22 smithi121 bash[36902]: audit 2022-04-22T18:39:21.345070+0000 mon.a (mon.0) 5554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:22 smithi121 bash[36902]: audit 2022-04-22T18:39:21.361303+0000 mon.a (mon.0) 5555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:22 smithi121 bash[36902]: audit 2022-04-22T18:39:21.703398+0000 mon.a (mon.0) 5556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:22 smithi121 bash[36902]: audit 2022-04-22T18:39:22.317905+0000 mon.a (mon.0) 5557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[45495]: audit 2022-04-22T18:39:21.345070+0000 mon.a (mon.0) 5554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[45495]: audit 2022-04-22T18:39:21.361303+0000 mon.a (mon.0) 5555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[45495]: audit 2022-04-22T18:39:21.703398+0000 mon.a (mon.0) 5556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[45495]: audit 2022-04-22T18:39:22.317905+0000 mon.a (mon.0) 5557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[46934]: audit 2022-04-22T18:39:21.345070+0000 mon.a (mon.0) 5554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[46934]: audit 2022-04-22T18:39:21.361303+0000 mon.a (mon.0) 5555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[46934]: audit 2022-04-22T18:39:21.703398+0000 mon.a (mon.0) 5556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:22 smithi115 bash[46934]: audit 2022-04-22T18:39:22.317905+0000 mon.a (mon.0) 5557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:23 smithi115 bash[45495]: cluster 2022-04-22T18:39:21.947277+0000 mgr.y (mgr.34107) 2544 : cluster [DBG] pgmap v4141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:24.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:23 smithi115 bash[46934]: cluster 2022-04-22T18:39:21.947277+0000 mgr.y (mgr.34107) 2544 : cluster [DBG] pgmap v4141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:24.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:23.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:23 smithi121 bash[36902]: cluster 2022-04-22T18:39:21.947277+0000 mgr.y (mgr.34107) 2544 : cluster [DBG] pgmap v4141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:25 smithi121 bash[36902]: cluster 2022-04-22T18:39:23.948437+0000 mgr.y (mgr.34107) 2545 : cluster [DBG] pgmap v4142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:25 smithi121 bash[36902]: audit 2022-04-22T18:39:25.498271+0000 mon.a (mon.0) 5558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:25 smithi115 bash[45495]: cluster 2022-04-22T18:39:23.948437+0000 mgr.y (mgr.34107) 2545 : cluster [DBG] pgmap v4142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:25 smithi115 bash[45495]: audit 2022-04-22T18:39:25.498271+0000 mon.a (mon.0) 5558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:25 smithi115 bash[46934]: cluster 2022-04-22T18:39:23.948437+0000 mgr.y (mgr.34107) 2545 : cluster [DBG] pgmap v4142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:25 smithi115 bash[46934]: audit 2022-04-22T18:39:25.498271+0000 mon.a (mon.0) 5558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:26 smithi121 bash[36902]: audit 2022-04-22T18:39:26.706440+0000 mon.a (mon.0) 5559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:26 smithi115 bash[45495]: audit 2022-04-22T18:39:26.706440+0000 mon.a (mon.0) 5559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:26 smithi115 bash[46934]: audit 2022-04-22T18:39:26.706440+0000 mon.a (mon.0) 5559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:27 smithi121 bash[36902]: cluster 2022-04-22T18:39:25.949409+0000 mgr.y (mgr.34107) 2546 : cluster [DBG] pgmap v4143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:27 smithi121 bash[36902]: audit 2022-04-22T18:39:27.321560+0000 mon.a (mon.0) 5560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:27 smithi115 bash[45495]: cluster 2022-04-22T18:39:25.949409+0000 mgr.y (mgr.34107) 2546 : cluster [DBG] pgmap v4143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:27 smithi115 bash[45495]: audit 2022-04-22T18:39:27.321560+0000 mon.a (mon.0) 5560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:27 smithi115 bash[46934]: cluster 2022-04-22T18:39:25.949409+0000 mgr.y (mgr.34107) 2546 : cluster [DBG] pgmap v4143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:27 smithi115 bash[46934]: audit 2022-04-22T18:39:27.321560+0000 mon.a (mon.0) 5560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:30.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:29 smithi115 bash[45495]: cluster 2022-04-22T18:39:27.950439+0000 mgr.y (mgr.34107) 2547 : cluster [DBG] pgmap v4144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:30.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:29 smithi115 bash[46934]: cluster 2022-04-22T18:39:27.950439+0000 mgr.y (mgr.34107) 2547 : cluster [DBG] pgmap v4144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:29 smithi121 bash[36902]: cluster 2022-04-22T18:39:27.950439+0000 mgr.y (mgr.34107) 2547 : cluster [DBG] pgmap v4144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:31 smithi121 bash[36902]: cluster 2022-04-22T18:39:29.951663+0000 mgr.y (mgr.34107) 2548 : cluster [DBG] pgmap v4145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:31 smithi121 bash[36902]: audit 2022-04-22T18:39:31.709548+0000 mon.a (mon.0) 5561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:31.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:31 smithi115 bash[45495]: cluster 2022-04-22T18:39:29.951663+0000 mgr.y (mgr.34107) 2548 : cluster [DBG] pgmap v4145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:31 smithi115 bash[45495]: audit 2022-04-22T18:39:31.709548+0000 mon.a (mon.0) 5561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:32.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:31 smithi115 bash[46934]: cluster 2022-04-22T18:39:29.951663+0000 mgr.y (mgr.34107) 2548 : cluster [DBG] pgmap v4145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:32.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:31 smithi115 bash[46934]: audit 2022-04-22T18:39:31.709548+0000 mon.a (mon.0) 5561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:32 smithi121 bash[36902]: audit 2022-04-22T18:39:32.325284+0000 mon.a (mon.0) 5562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:32 smithi115 bash[45495]: audit 2022-04-22T18:39:32.325284+0000 mon.a (mon.0) 5562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:32 smithi115 bash[46934]: audit 2022-04-22T18:39:32.325284+0000 mon.a (mon.0) 5562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:34.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:33 smithi115 bash[46934]: cluster 2022-04-22T18:39:31.952771+0000 mgr.y (mgr.34107) 2549 : cluster [DBG] pgmap v4146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:34.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:33 smithi115 bash[45495]: cluster 2022-04-22T18:39:31.952771+0000 mgr.y (mgr.34107) 2549 : cluster [DBG] pgmap v4146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:34.039 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:33.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:33 smithi121 bash[36902]: cluster 2022-04-22T18:39:31.952771+0000 mgr.y (mgr.34107) 2549 : cluster [DBG] pgmap v4146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:35 smithi121 bash[36902]: cluster 2022-04-22T18:39:33.953963+0000 mgr.y (mgr.34107) 2550 : cluster [DBG] pgmap v4147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:35 smithi121 bash[36902]: audit 2022-04-22T18:39:34.902143+0000 mon.a (mon.0) 5563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:35 smithi121 bash[36902]: audit 2022-04-22T18:39:34.910999+0000 mon.a (mon.0) 5564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[45495]: cluster 2022-04-22T18:39:33.953963+0000 mgr.y (mgr.34107) 2550 : cluster [DBG] pgmap v4147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[45495]: audit 2022-04-22T18:39:34.902143+0000 mon.a (mon.0) 5563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[45495]: audit 2022-04-22T18:39:34.910999+0000 mon.a (mon.0) 5564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[46934]: cluster 2022-04-22T18:39:33.953963+0000 mgr.y (mgr.34107) 2550 : cluster [DBG] pgmap v4147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[46934]: audit 2022-04-22T18:39:34.902143+0000 mon.a (mon.0) 5563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:35 smithi115 bash[46934]: audit 2022-04-22T18:39:34.910999+0000 mon.a (mon.0) 5564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:39:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:36 smithi121 bash[36902]: cephadm 2022-04-22T18:39:34.889707+0000 mgr.y (mgr.34107) 2551 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:39:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:36 smithi121 bash[36902]: audit 2022-04-22T18:39:36.712254+0000 mon.a (mon.0) 5565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:36 smithi115 bash[45495]: cephadm 2022-04-22T18:39:34.889707+0000 mgr.y (mgr.34107) 2551 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:39:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:36 smithi115 bash[45495]: audit 2022-04-22T18:39:36.712254+0000 mon.a (mon.0) 5565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:36 smithi115 bash[46934]: cephadm 2022-04-22T18:39:34.889707+0000 mgr.y (mgr.34107) 2551 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T18:39:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:36 smithi115 bash[46934]: audit 2022-04-22T18:39:36.712254+0000 mon.a (mon.0) 5565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:37 smithi121 bash[36902]: cluster 2022-04-22T18:39:35.954969+0000 mgr.y (mgr.34107) 2552 : cluster [DBG] pgmap v4148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:37 smithi121 bash[36902]: audit 2022-04-22T18:39:37.328790+0000 mon.a (mon.0) 5566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:37 smithi115 bash[45495]: cluster 2022-04-22T18:39:35.954969+0000 mgr.y (mgr.34107) 2552 : cluster [DBG] pgmap v4148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:37 smithi115 bash[45495]: audit 2022-04-22T18:39:37.328790+0000 mon.a (mon.0) 5566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:37 smithi115 bash[46934]: cluster 2022-04-22T18:39:35.954969+0000 mgr.y (mgr.34107) 2552 : cluster [DBG] pgmap v4148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:37 smithi115 bash[46934]: audit 2022-04-22T18:39:37.328790+0000 mon.a (mon.0) 5566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:39 smithi121 bash[36902]: cluster 2022-04-22T18:39:37.956119+0000 mgr.y (mgr.34107) 2553 : cluster [DBG] pgmap v4149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:39 smithi115 bash[45495]: cluster 2022-04-22T18:39:37.956119+0000 mgr.y (mgr.34107) 2553 : cluster [DBG] pgmap v4149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:39 smithi115 bash[46934]: cluster 2022-04-22T18:39:37.956119+0000 mgr.y (mgr.34107) 2553 : cluster [DBG] pgmap v4149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:41 smithi121 bash[36902]: cluster 2022-04-22T18:39:39.957156+0000 mgr.y (mgr.34107) 2554 : cluster [DBG] pgmap v4150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:42.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:41 smithi121 bash[36902]: audit 2022-04-22T18:39:41.715700+0000 mon.a (mon.0) 5567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:41.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:41 smithi115 bash[45495]: cluster 2022-04-22T18:39:39.957156+0000 mgr.y (mgr.34107) 2554 : cluster [DBG] pgmap v4150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:41 smithi115 bash[45495]: audit 2022-04-22T18:39:41.715700+0000 mon.a (mon.0) 5567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:42.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:41 smithi115 bash[46934]: cluster 2022-04-22T18:39:39.957156+0000 mgr.y (mgr.34107) 2554 : cluster [DBG] pgmap v4150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:42.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:41 smithi115 bash[46934]: audit 2022-04-22T18:39:41.715700+0000 mon.a (mon.0) 5567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:42 smithi121 bash[36902]: audit 2022-04-22T18:39:42.332333+0000 mon.a (mon.0) 5568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:42 smithi115 bash[45495]: audit 2022-04-22T18:39:42.332333+0000 mon.a (mon.0) 5568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:42 smithi115 bash[46934]: audit 2022-04-22T18:39:42.332333+0000 mon.a (mon.0) 5568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:44.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:43 smithi115 bash[45495]: cluster 2022-04-22T18:39:41.958218+0000 mgr.y (mgr.34107) 2555 : cluster [DBG] pgmap v4151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:43.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:43 smithi115 bash[46934]: cluster 2022-04-22T18:39:41.958218+0000 mgr.y (mgr.34107) 2555 : cluster [DBG] pgmap v4151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:43 smithi121 bash[36902]: cluster 2022-04-22T18:39:41.958218+0000 mgr.y (mgr.34107) 2555 : cluster [DBG] pgmap v4151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:45 smithi121 bash[36902]: cluster 2022-04-22T18:39:43.959378+0000 mgr.y (mgr.34107) 2556 : cluster [DBG] pgmap v4152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:45 smithi115 bash[45495]: cluster 2022-04-22T18:39:43.959378+0000 mgr.y (mgr.34107) 2556 : cluster [DBG] pgmap v4152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:46.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:45 smithi115 bash[46934]: cluster 2022-04-22T18:39:43.959378+0000 mgr.y (mgr.34107) 2556 : cluster [DBG] pgmap v4152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:46 smithi121 bash[36902]: audit 2022-04-22T18:39:46.718874+0000 mon.a (mon.0) 5569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:46 smithi115 bash[45495]: audit 2022-04-22T18:39:46.718874+0000 mon.a (mon.0) 5569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:46 smithi115 bash[46934]: audit 2022-04-22T18:39:46.718874+0000 mon.a (mon.0) 5569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:47.707 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (62m) 22s ago 76m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (76m) 87s ago 76m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (64m) 22s ago 77m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (64m) 87s ago 77m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (72m) 87s ago 81m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (72m) 22s ago 84m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:39:48.053 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (71m) 22s ago 84m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (70m) 87s ago 82m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (71m) 22s ago 82m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (63m) 22s ago 76m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (63m) 87s ago 76m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (69m) 22s ago 81m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (68m) 22s ago 81m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (68m) 22s ago 80m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (68m) 22s ago 80m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:39:48.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (67m) 87s ago 79m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:39:48.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (66m) 87s ago 79m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:39:48.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (65m) 87s ago 78m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:39:48.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (65m) 87s ago 78m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:39:48.055 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (63m) 87s ago 76m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:39:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:47 smithi121 bash[36902]: cluster 2022-04-22T18:39:45.960418+0000 mgr.y (mgr.34107) 2557 : cluster [DBG] pgmap v4153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:47 smithi121 bash[36902]: audit 2022-04-22T18:39:47.336138+0000 mon.a (mon.0) 5570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:47 smithi121 bash[36902]: audit 2022-04-22T18:39:47.684109+0000 mgr.y (mgr.34107) 2558 : audit [DBG] from='client.38106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[46934]: cluster 2022-04-22T18:39:45.960418+0000 mgr.y (mgr.34107) 2557 : cluster [DBG] pgmap v4153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[46934]: audit 2022-04-22T18:39:47.336138+0000 mon.a (mon.0) 5570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[46934]: audit 2022-04-22T18:39:47.684109+0000 mgr.y (mgr.34107) 2558 : audit [DBG] from='client.38106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[45495]: cluster 2022-04-22T18:39:45.960418+0000 mgr.y (mgr.34107) 2557 : cluster [DBG] pgmap v4153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[45495]: audit 2022-04-22T18:39:47.336138+0000 mon.a (mon.0) 5570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:48.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:47 smithi115 bash[45495]: audit 2022-04-22T18:39:47.684109+0000 mgr.y (mgr.34107) 2558 : audit [DBG] from='client.38106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:39:48.462 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:39:48.463 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:39:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:48 smithi121 bash[36902]: cluster 2022-04-22T18:39:47.961390+0000 mgr.y (mgr.34107) 2559 : cluster [DBG] pgmap v4154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:48 smithi121 bash[36902]: audit 2022-04-22T18:39:48.031556+0000 mgr.y (mgr.34107) 2560 : audit [DBG] from='client.48046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:48 smithi121 bash[36902]: audit 2022-04-22T18:39:48.453880+0000 mon.c (mon.1) 151 : audit [DBG] from='client.? 172.21.15.115:0/2299528434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[45495]: cluster 2022-04-22T18:39:47.961390+0000 mgr.y (mgr.34107) 2559 : cluster [DBG] pgmap v4154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[45495]: audit 2022-04-22T18:39:48.031556+0000 mgr.y (mgr.34107) 2560 : audit [DBG] from='client.48046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[45495]: audit 2022-04-22T18:39:48.453880+0000 mon.c (mon.1) 151 : audit [DBG] from='client.? 172.21.15.115:0/2299528434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[46934]: cluster 2022-04-22T18:39:47.961390+0000 mgr.y (mgr.34107) 2559 : cluster [DBG] pgmap v4154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[46934]: audit 2022-04-22T18:39:48.031556+0000 mgr.y (mgr.34107) 2560 : audit [DBG] from='client.48046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:39:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:48 smithi115 bash[46934]: audit 2022-04-22T18:39:48.453880+0000 mon.c (mon.1) 151 : audit [DBG] from='client.? 172.21.15.115:0/2299528434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:39:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:51 smithi115 bash[45495]: cluster 2022-04-22T18:39:49.962413+0000 mgr.y (mgr.34107) 2561 : cluster [DBG] pgmap v4155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:51 smithi115 bash[46934]: cluster 2022-04-22T18:39:49.962413+0000 mgr.y (mgr.34107) 2561 : cluster [DBG] pgmap v4155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:51 smithi121 bash[36902]: cluster 2022-04-22T18:39:49.962413+0000 mgr.y (mgr.34107) 2561 : cluster [DBG] pgmap v4155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:52 smithi115 bash[45495]: audit 2022-04-22T18:39:51.721987+0000 mon.a (mon.0) 5571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:51.883Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:52.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:52 smithi115 bash[46934]: audit 2022-04-22T18:39:51.721987+0000 mon.a (mon.0) 5571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:52 smithi121 bash[36902]: audit 2022-04-22T18:39:51.721987+0000 mon.a (mon.0) 5571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:53 smithi115 bash[45495]: cluster 2022-04-22T18:39:51.963506+0000 mgr.y (mgr.34107) 2562 : cluster [DBG] pgmap v4156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:53 smithi115 bash[45495]: audit 2022-04-22T18:39:52.339870+0000 mon.a (mon.0) 5572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:53 smithi115 bash[46934]: cluster 2022-04-22T18:39:51.963506+0000 mgr.y (mgr.34107) 2562 : cluster [DBG] pgmap v4156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:53 smithi115 bash[46934]: audit 2022-04-22T18:39:52.339870+0000 mon.a (mon.0) 5572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:53 smithi121 bash[36902]: cluster 2022-04-22T18:39:51.963506+0000 mgr.y (mgr.34107) 2562 : cluster [DBG] pgmap v4156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:53 smithi121 bash[36902]: audit 2022-04-22T18:39:52.339870+0000 mon.a (mon.0) 5572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:39:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:39:53.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:39:55.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:55 smithi115 bash[45495]: cluster 2022-04-22T18:39:53.964662+0000 mgr.y (mgr.34107) 2563 : cluster [DBG] pgmap v4157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:55.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:55 smithi115 bash[46934]: cluster 2022-04-22T18:39:53.964662+0000 mgr.y (mgr.34107) 2563 : cluster [DBG] pgmap v4157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:55 smithi121 bash[36902]: cluster 2022-04-22T18:39:53.964662+0000 mgr.y (mgr.34107) 2563 : cluster [DBG] pgmap v4157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:57 smithi115 bash[45495]: cluster 2022-04-22T18:39:55.965626+0000 mgr.y (mgr.34107) 2564 : cluster [DBG] pgmap v4158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:57 smithi115 bash[45495]: audit 2022-04-22T18:39:56.725160+0000 mon.a (mon.0) 5573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:57.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:57 smithi115 bash[46934]: cluster 2022-04-22T18:39:55.965626+0000 mgr.y (mgr.34107) 2564 : cluster [DBG] pgmap v4158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:57 smithi115 bash[46934]: audit 2022-04-22T18:39:56.725160+0000 mon.a (mon.0) 5573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:57 smithi121 bash[36902]: cluster 2022-04-22T18:39:55.965626+0000 mgr.y (mgr.34107) 2564 : cluster [DBG] pgmap v4158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:57 smithi121 bash[36902]: audit 2022-04-22T18:39:56.725160+0000 mon.a (mon.0) 5573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:39:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:58 smithi115 bash[45495]: audit 2022-04-22T18:39:57.343948+0000 mon.a (mon.0) 5574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:58 smithi115 bash[46934]: audit 2022-04-22T18:39:57.343948+0000 mon.a (mon.0) 5574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:58 smithi121 bash[36902]: audit 2022-04-22T18:39:57.343948+0000 mon.a (mon.0) 5574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:39:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:39:59 smithi121 bash[36902]: cluster 2022-04-22T18:39:57.966584+0000 mgr.y (mgr.34107) 2565 : cluster [DBG] pgmap v4159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:39:59 smithi115 bash[45495]: cluster 2022-04-22T18:39:57.966584+0000 mgr.y (mgr.34107) 2565 : cluster [DBG] pgmap v4159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:39:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:39:59 smithi115 bash[46934]: cluster 2022-04-22T18:39:57.966584+0000 mgr.y (mgr.34107) 2565 : cluster [DBG] pgmap v4159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:00 smithi121 bash[36902]: cluster 2022-04-22T18:40:00.000160+0000 mon.a (mon.0) 5575 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:40:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:00 smithi115 bash[45495]: cluster 2022-04-22T18:40:00.000160+0000 mon.a (mon.0) 5575 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:40:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:00 smithi115 bash[46934]: cluster 2022-04-22T18:40:00.000160+0000 mon.a (mon.0) 5575 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:40:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:01 smithi121 bash[36902]: cluster 2022-04-22T18:39:59.967686+0000 mgr.y (mgr.34107) 2566 : cluster [DBG] pgmap v4160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:01.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:01 smithi115 bash[45495]: cluster 2022-04-22T18:39:59.967686+0000 mgr.y (mgr.34107) 2566 : cluster [DBG] pgmap v4160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:01 smithi115 bash[46934]: cluster 2022-04-22T18:39:59.967686+0000 mgr.y (mgr.34107) 2566 : cluster [DBG] pgmap v4160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:02.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:02 smithi115 bash[45495]: audit 2022-04-22T18:40:01.728264+0000 mon.a (mon.0) 5576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:01.883Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:02 smithi115 bash[46934]: audit 2022-04-22T18:40:01.728264+0000 mon.a (mon.0) 5576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:02 smithi121 bash[36902]: audit 2022-04-22T18:40:01.728264+0000 mon.a (mon.0) 5576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:03 smithi121 bash[36902]: cluster 2022-04-22T18:40:01.968731+0000 mgr.y (mgr.34107) 2567 : cluster [DBG] pgmap v4161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:03 smithi121 bash[36902]: audit 2022-04-22T18:40:02.347569+0000 mon.a (mon.0) 5577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:03 smithi115 bash[45495]: cluster 2022-04-22T18:40:01.968731+0000 mgr.y (mgr.34107) 2567 : cluster [DBG] pgmap v4161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:03 smithi115 bash[45495]: audit 2022-04-22T18:40:02.347569+0000 mon.a (mon.0) 5577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:03 smithi115 bash[46934]: cluster 2022-04-22T18:40:01.968731+0000 mgr.y (mgr.34107) 2567 : cluster [DBG] pgmap v4161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:03 smithi115 bash[46934]: audit 2022-04-22T18:40:02.347569+0000 mon.a (mon.0) 5577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:03.763Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:05 smithi121 bash[36902]: cluster 2022-04-22T18:40:03.969868+0000 mgr.y (mgr.34107) 2568 : cluster [DBG] pgmap v4162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:05 smithi115 bash[45495]: cluster 2022-04-22T18:40:03.969868+0000 mgr.y (mgr.34107) 2568 : cluster [DBG] pgmap v4162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:05 smithi115 bash[46934]: cluster 2022-04-22T18:40:03.969868+0000 mgr.y (mgr.34107) 2568 : cluster [DBG] pgmap v4162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:07 smithi121 bash[36902]: cluster 2022-04-22T18:40:05.970820+0000 mgr.y (mgr.34107) 2569 : cluster [DBG] pgmap v4163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:07.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:07 smithi121 bash[36902]: audit 2022-04-22T18:40:06.731322+0000 mon.a (mon.0) 5578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:07 smithi115 bash[45495]: cluster 2022-04-22T18:40:05.970820+0000 mgr.y (mgr.34107) 2569 : cluster [DBG] pgmap v4163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:07 smithi115 bash[45495]: audit 2022-04-22T18:40:06.731322+0000 mon.a (mon.0) 5578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:07 smithi115 bash[46934]: cluster 2022-04-22T18:40:05.970820+0000 mgr.y (mgr.34107) 2569 : cluster [DBG] pgmap v4163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:07 smithi115 bash[46934]: audit 2022-04-22T18:40:06.731322+0000 mon.a (mon.0) 5578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:08 smithi121 bash[36902]: audit 2022-04-22T18:40:07.351114+0000 mon.a (mon.0) 5579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:08 smithi115 bash[45495]: audit 2022-04-22T18:40:07.351114+0000 mon.a (mon.0) 5579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:08 smithi115 bash[46934]: audit 2022-04-22T18:40:07.351114+0000 mon.a (mon.0) 5579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:09 smithi121 bash[36902]: cluster 2022-04-22T18:40:07.971897+0000 mgr.y (mgr.34107) 2570 : cluster [DBG] pgmap v4164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:09 smithi115 bash[45495]: cluster 2022-04-22T18:40:07.971897+0000 mgr.y (mgr.34107) 2570 : cluster [DBG] pgmap v4164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:09 smithi115 bash[46934]: cluster 2022-04-22T18:40:07.971897+0000 mgr.y (mgr.34107) 2570 : cluster [DBG] pgmap v4164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:11 smithi121 bash[36902]: cluster 2022-04-22T18:40:09.972975+0000 mgr.y (mgr.34107) 2571 : cluster [DBG] pgmap v4165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:11 smithi115 bash[45495]: cluster 2022-04-22T18:40:09.972975+0000 mgr.y (mgr.34107) 2571 : cluster [DBG] pgmap v4165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:11 smithi115 bash[46934]: cluster 2022-04-22T18:40:09.972975+0000 mgr.y (mgr.34107) 2571 : cluster [DBG] pgmap v4165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:12.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:12 smithi115 bash[45495]: audit 2022-04-22T18:40:11.734458+0000 mon.a (mon.0) 5580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:11.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:12.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:12 smithi115 bash[46934]: audit 2022-04-22T18:40:11.734458+0000 mon.a (mon.0) 5580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:12 smithi121 bash[36902]: audit 2022-04-22T18:40:11.734458+0000 mon.a (mon.0) 5580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:13 smithi121 bash[36902]: cluster 2022-04-22T18:40:11.973987+0000 mgr.y (mgr.34107) 2572 : cluster [DBG] pgmap v4166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:13.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:13 smithi121 bash[36902]: audit 2022-04-22T18:40:12.354725+0000 mon.a (mon.0) 5581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:13 smithi115 bash[45495]: cluster 2022-04-22T18:40:11.973987+0000 mgr.y (mgr.34107) 2572 : cluster [DBG] pgmap v4166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:13 smithi115 bash[45495]: audit 2022-04-22T18:40:12.354725+0000 mon.a (mon.0) 5581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:13 smithi115 bash[46934]: cluster 2022-04-22T18:40:11.973987+0000 mgr.y (mgr.34107) 2572 : cluster [DBG] pgmap v4166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:13 smithi115 bash[46934]: audit 2022-04-22T18:40:12.354725+0000 mon.a (mon.0) 5581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:13.763Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:15 smithi121 bash[36902]: cluster 2022-04-22T18:40:13.975152+0000 mgr.y (mgr.34107) 2573 : cluster [DBG] pgmap v4167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:15 smithi121 bash[36902]: audit 2022-04-22T18:40:14.081397+0000 mon.a (mon.0) 5582 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:40:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:15 smithi121 bash[36902]: audit 2022-04-22T18:40:14.092863+0000 mon.a (mon.0) 5583 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:40:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[45495]: cluster 2022-04-22T18:40:13.975152+0000 mgr.y (mgr.34107) 2573 : cluster [DBG] pgmap v4167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[45495]: audit 2022-04-22T18:40:14.081397+0000 mon.a (mon.0) 5582 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:40:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[45495]: audit 2022-04-22T18:40:14.092863+0000 mon.a (mon.0) 5583 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:40:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[46934]: cluster 2022-04-22T18:40:13.975152+0000 mgr.y (mgr.34107) 2573 : cluster [DBG] pgmap v4167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[46934]: audit 2022-04-22T18:40:14.081397+0000 mon.a (mon.0) 5582 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:40:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:15 smithi115 bash[46934]: audit 2022-04-22T18:40:14.092863+0000 mon.a (mon.0) 5583 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:40:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:17 smithi121 bash[36902]: cluster 2022-04-22T18:40:15.976074+0000 mgr.y (mgr.34107) 2574 : cluster [DBG] pgmap v4168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:17 smithi121 bash[36902]: audit 2022-04-22T18:40:16.737397+0000 mon.a (mon.0) 5584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:17 smithi115 bash[45495]: cluster 2022-04-22T18:40:15.976074+0000 mgr.y (mgr.34107) 2574 : cluster [DBG] pgmap v4168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:17 smithi115 bash[45495]: audit 2022-04-22T18:40:16.737397+0000 mon.a (mon.0) 5584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:17 smithi115 bash[46934]: cluster 2022-04-22T18:40:15.976074+0000 mgr.y (mgr.34107) 2574 : cluster [DBG] pgmap v4168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:17 smithi115 bash[46934]: audit 2022-04-22T18:40:16.737397+0000 mon.a (mon.0) 5584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:18 smithi121 bash[36902]: audit 2022-04-22T18:40:17.358392+0000 mon.a (mon.0) 5585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:18 smithi115 bash[45495]: audit 2022-04-22T18:40:17.358392+0000 mon.a (mon.0) 5585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:18 smithi115 bash[46934]: audit 2022-04-22T18:40:17.358392+0000 mon.a (mon.0) 5585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:18.839 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:40:19.190 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:40:19.190 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (63m) 53s ago 77m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:40:19.190 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (76m) 118s ago 76m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:40:19.190 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (65m) 53s ago 78m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (64m) 118s ago 78m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (72m) 118s ago 82m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (73m) 53s ago 84m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (72m) 53s ago 84m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (71m) 118s ago 82m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (71m) 53s ago 82m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (64m) 53s ago 77m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (64m) 118s ago 77m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:40:19.191 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (70m) 53s ago 82m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (69m) 53s ago 81m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (68m) 53s ago 81m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (68m) 53s ago 80m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (67m) 118s ago 80m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (67m) 118s ago 79m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (66m) 118s ago 79m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (65m) 118s ago 78m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:40:19.192 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (64m) 118s ago 77m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:40:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:19 smithi121 bash[36902]: cluster 2022-04-22T18:40:17.977130+0000 mgr.y (mgr.34107) 2575 : cluster [DBG] pgmap v4169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:19 smithi121 bash[36902]: audit 2022-04-22T18:40:18.815558+0000 mgr.y (mgr.34107) 2576 : audit [DBG] from='client.48058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:19 smithi115 bash[46934]: cluster 2022-04-22T18:40:17.977130+0000 mgr.y (mgr.34107) 2575 : cluster [DBG] pgmap v4169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:19.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:19 smithi115 bash[46934]: audit 2022-04-22T18:40:18.815558+0000 mgr.y (mgr.34107) 2576 : audit [DBG] from='client.48058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:19.579 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:19 smithi115 bash[45495]: cluster 2022-04-22T18:40:17.977130+0000 mgr.y (mgr.34107) 2575 : cluster [DBG] pgmap v4169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:19.579 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:19 smithi115 bash[45495]: audit 2022-04-22T18:40:18.815558+0000 mgr.y (mgr.34107) 2576 : audit [DBG] from='client.48058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:40:19.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:40:19.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:40:19.589 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:40:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:20 smithi121 bash[36902]: audit 2022-04-22T18:40:19.163218+0000 mgr.y (mgr.34107) 2577 : audit [DBG] from='client.38130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:20 smithi121 bash[36902]: audit 2022-04-22T18:40:19.579361+0000 mon.a (mon.0) 5586 : audit [DBG] from='client.? 172.21.15.115:0/537228480' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:20 smithi115 bash[45495]: audit 2022-04-22T18:40:19.163218+0000 mgr.y (mgr.34107) 2577 : audit [DBG] from='client.38130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:20 smithi115 bash[45495]: audit 2022-04-22T18:40:19.579361+0000 mon.a (mon.0) 5586 : audit [DBG] from='client.? 172.21.15.115:0/537228480' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:20 smithi115 bash[46934]: audit 2022-04-22T18:40:19.163218+0000 mgr.y (mgr.34107) 2577 : audit [DBG] from='client.38130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:20 smithi115 bash[46934]: audit 2022-04-22T18:40:19.579361+0000 mon.a (mon.0) 5586 : audit [DBG] from='client.? 172.21.15.115:0/537228480' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:21 smithi121 bash[36902]: cluster 2022-04-22T18:40:19.978227+0000 mgr.y (mgr.34107) 2578 : cluster [DBG] pgmap v4170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:21 smithi115 bash[46934]: cluster 2022-04-22T18:40:19.978227+0000 mgr.y (mgr.34107) 2578 : cluster [DBG] pgmap v4170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:21 smithi115 bash[45495]: cluster 2022-04-22T18:40:19.978227+0000 mgr.y (mgr.34107) 2578 : cluster [DBG] pgmap v4170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:22.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:22 smithi115 bash[45495]: audit 2022-04-22T18:40:21.740549+0000 mon.a (mon.0) 5587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:21.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:22.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:22 smithi115 bash[46934]: audit 2022-04-22T18:40:21.740549+0000 mon.a (mon.0) 5587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:22 smithi121 bash[36902]: audit 2022-04-22T18:40:21.740549+0000 mon.a (mon.0) 5587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:23 smithi121 bash[36902]: cluster 2022-04-22T18:40:21.979316+0000 mgr.y (mgr.34107) 2579 : cluster [DBG] pgmap v4171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:23.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:23 smithi121 bash[36902]: audit 2022-04-22T18:40:22.362189+0000 mon.a (mon.0) 5588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:23 smithi115 bash[45495]: cluster 2022-04-22T18:40:21.979316+0000 mgr.y (mgr.34107) 2579 : cluster [DBG] pgmap v4171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:23 smithi115 bash[45495]: audit 2022-04-22T18:40:22.362189+0000 mon.a (mon.0) 5588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:23 smithi115 bash[46934]: cluster 2022-04-22T18:40:21.979316+0000 mgr.y (mgr.34107) 2579 : cluster [DBG] pgmap v4171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:23 smithi115 bash[46934]: audit 2022-04-22T18:40:22.362189+0000 mon.a (mon.0) 5588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:23.763Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:25 smithi121 bash[36902]: cluster 2022-04-22T18:40:23.980451+0000 mgr.y (mgr.34107) 2580 : cluster [DBG] pgmap v4172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:25 smithi115 bash[46934]: cluster 2022-04-22T18:40:23.980451+0000 mgr.y (mgr.34107) 2580 : cluster [DBG] pgmap v4172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:25 smithi115 bash[45495]: cluster 2022-04-22T18:40:23.980451+0000 mgr.y (mgr.34107) 2580 : cluster [DBG] pgmap v4172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:27 smithi121 bash[36902]: cluster 2022-04-22T18:40:25.981455+0000 mgr.y (mgr.34107) 2581 : cluster [DBG] pgmap v4173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:27 smithi121 bash[36902]: audit 2022-04-22T18:40:26.743620+0000 mon.a (mon.0) 5589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:27 smithi115 bash[46934]: cluster 2022-04-22T18:40:25.981455+0000 mgr.y (mgr.34107) 2581 : cluster [DBG] pgmap v4173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:27 smithi115 bash[46934]: audit 2022-04-22T18:40:26.743620+0000 mon.a (mon.0) 5589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:27 smithi115 bash[45495]: cluster 2022-04-22T18:40:25.981455+0000 mgr.y (mgr.34107) 2581 : cluster [DBG] pgmap v4173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:27 smithi115 bash[45495]: audit 2022-04-22T18:40:26.743620+0000 mon.a (mon.0) 5589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:28 smithi121 bash[36902]: audit 2022-04-22T18:40:27.365946+0000 mon.a (mon.0) 5590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:28 smithi115 bash[46934]: audit 2022-04-22T18:40:27.365946+0000 mon.a (mon.0) 5590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:28 smithi115 bash[45495]: audit 2022-04-22T18:40:27.365946+0000 mon.a (mon.0) 5590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:29 smithi121 bash[36902]: cluster 2022-04-22T18:40:27.982545+0000 mgr.y (mgr.34107) 2582 : cluster [DBG] pgmap v4174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:29 smithi115 bash[45495]: cluster 2022-04-22T18:40:27.982545+0000 mgr.y (mgr.34107) 2582 : cluster [DBG] pgmap v4174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:29 smithi115 bash[46934]: cluster 2022-04-22T18:40:27.982545+0000 mgr.y (mgr.34107) 2582 : cluster [DBG] pgmap v4174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:31 smithi121 bash[36902]: cluster 2022-04-22T18:40:29.983665+0000 mgr.y (mgr.34107) 2583 : cluster [DBG] pgmap v4175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:31 smithi115 bash[45495]: cluster 2022-04-22T18:40:29.983665+0000 mgr.y (mgr.34107) 2583 : cluster [DBG] pgmap v4175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:31 smithi115 bash[46934]: cluster 2022-04-22T18:40:29.983665+0000 mgr.y (mgr.34107) 2583 : cluster [DBG] pgmap v4175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:32.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:31.885Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:32 smithi121 bash[36902]: audit 2022-04-22T18:40:31.746539+0000 mon.a (mon.0) 5591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:32.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:32 smithi115 bash[45495]: audit 2022-04-22T18:40:31.746539+0000 mon.a (mon.0) 5591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:32.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:32 smithi115 bash[46934]: audit 2022-04-22T18:40:31.746539+0000 mon.a (mon.0) 5591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:33 smithi121 bash[36902]: cluster 2022-04-22T18:40:31.984735+0000 mgr.y (mgr.34107) 2584 : cluster [DBG] pgmap v4176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:33 smithi121 bash[36902]: audit 2022-04-22T18:40:32.369684+0000 mon.a (mon.0) 5592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:33 smithi115 bash[45495]: cluster 2022-04-22T18:40:31.984735+0000 mgr.y (mgr.34107) 2584 : cluster [DBG] pgmap v4176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:33 smithi115 bash[45495]: audit 2022-04-22T18:40:32.369684+0000 mon.a (mon.0) 5592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:33 smithi115 bash[46934]: cluster 2022-04-22T18:40:31.984735+0000 mgr.y (mgr.34107) 2584 : cluster [DBG] pgmap v4176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:33 smithi115 bash[46934]: audit 2022-04-22T18:40:32.369684+0000 mon.a (mon.0) 5592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:33.764Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:35 smithi121 bash[36902]: cluster 2022-04-22T18:40:33.985889+0000 mgr.y (mgr.34107) 2585 : cluster [DBG] pgmap v4177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:35 smithi121 bash[36902]: audit 2022-04-22T18:40:34.921193+0000 mon.a (mon.0) 5593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:40:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:35 smithi121 bash[36902]: audit 2022-04-22T18:40:34.924025+0000 mon.a (mon.0) 5594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:40:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:35 smithi121 bash[36902]: audit 2022-04-22T18:40:34.925595+0000 mon.a (mon.0) 5595 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:40:35.272 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[45495]: cluster 2022-04-22T18:40:33.985889+0000 mgr.y (mgr.34107) 2585 : cluster [DBG] pgmap v4177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:35.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[45495]: audit 2022-04-22T18:40:34.921193+0000 mon.a (mon.0) 5593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:40:35.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[45495]: audit 2022-04-22T18:40:34.924025+0000 mon.a (mon.0) 5594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:40:35.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[45495]: audit 2022-04-22T18:40:34.925595+0000 mon.a (mon.0) 5595 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:40:35.273 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[46934]: cluster 2022-04-22T18:40:33.985889+0000 mgr.y (mgr.34107) 2585 : cluster [DBG] pgmap v4177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:35.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[46934]: audit 2022-04-22T18:40:34.921193+0000 mon.a (mon.0) 5593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:40:35.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[46934]: audit 2022-04-22T18:40:34.924025+0000 mon.a (mon.0) 5594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:40:35.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:35 smithi115 bash[46934]: audit 2022-04-22T18:40:34.925595+0000 mon.a (mon.0) 5595 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:40:36.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:36 smithi115 bash[45495]: audit 2022-04-22T18:40:35.278978+0000 mon.a (mon.0) 5596 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:40:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:36 smithi115 bash[46934]: audit 2022-04-22T18:40:35.278978+0000 mon.a (mon.0) 5596 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:40:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:36 smithi121 bash[36902]: audit 2022-04-22T18:40:35.278978+0000 mon.a (mon.0) 5596 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:40:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:37 smithi115 bash[46934]: cluster 2022-04-22T18:40:35.986873+0000 mgr.y (mgr.34107) 2586 : cluster [DBG] pgmap v4178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:37 smithi115 bash[46934]: audit 2022-04-22T18:40:36.749409+0000 mon.a (mon.0) 5597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:37 smithi115 bash[45495]: cluster 2022-04-22T18:40:35.986873+0000 mgr.y (mgr.34107) 2586 : cluster [DBG] pgmap v4178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:37 smithi115 bash[45495]: audit 2022-04-22T18:40:36.749409+0000 mon.a (mon.0) 5597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:37 smithi121 bash[36902]: cluster 2022-04-22T18:40:35.986873+0000 mgr.y (mgr.34107) 2586 : cluster [DBG] pgmap v4178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:37 smithi121 bash[36902]: audit 2022-04-22T18:40:36.749409+0000 mon.a (mon.0) 5597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:38 smithi121 bash[36902]: audit 2022-04-22T18:40:37.373117+0000 mon.a (mon.0) 5598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:38.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:38 smithi115 bash[45495]: audit 2022-04-22T18:40:37.373117+0000 mon.a (mon.0) 5598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:38 smithi115 bash[46934]: audit 2022-04-22T18:40:37.373117+0000 mon.a (mon.0) 5598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:39 smithi121 bash[36902]: cluster 2022-04-22T18:40:37.988032+0000 mgr.y (mgr.34107) 2587 : cluster [DBG] pgmap v4179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:39 smithi115 bash[45495]: cluster 2022-04-22T18:40:37.988032+0000 mgr.y (mgr.34107) 2587 : cluster [DBG] pgmap v4179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:39 smithi115 bash[46934]: cluster 2022-04-22T18:40:37.988032+0000 mgr.y (mgr.34107) 2587 : cluster [DBG] pgmap v4179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:41 smithi121 bash[36902]: cluster 2022-04-22T18:40:39.989266+0000 mgr.y (mgr.34107) 2588 : cluster [DBG] pgmap v4180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:41.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:41 smithi115 bash[45495]: cluster 2022-04-22T18:40:39.989266+0000 mgr.y (mgr.34107) 2588 : cluster [DBG] pgmap v4180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:41 smithi115 bash[46934]: cluster 2022-04-22T18:40:39.989266+0000 mgr.y (mgr.34107) 2588 : cluster [DBG] pgmap v4180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:41.886Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:42 smithi121 bash[36902]: audit 2022-04-22T18:40:41.752623+0000 mon.a (mon.0) 5599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:42.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:42 smithi115 bash[45495]: audit 2022-04-22T18:40:41.752623+0000 mon.a (mon.0) 5599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:42.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:42 smithi115 bash[46934]: audit 2022-04-22T18:40:41.752623+0000 mon.a (mon.0) 5599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:43 smithi121 bash[36902]: cluster 2022-04-22T18:40:41.990428+0000 mgr.y (mgr.34107) 2589 : cluster [DBG] pgmap v4181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:43 smithi121 bash[36902]: audit 2022-04-22T18:40:42.376758+0000 mon.a (mon.0) 5600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:43.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:43 smithi115 bash[45495]: cluster 2022-04-22T18:40:41.990428+0000 mgr.y (mgr.34107) 2589 : cluster [DBG] pgmap v4181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:43.773 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:43 smithi115 bash[45495]: audit 2022-04-22T18:40:42.376758+0000 mon.a (mon.0) 5600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:43.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:43 smithi115 bash[46934]: cluster 2022-04-22T18:40:41.990428+0000 mgr.y (mgr.34107) 2589 : cluster [DBG] pgmap v4181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:43.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:43 smithi115 bash[46934]: audit 2022-04-22T18:40:42.376758+0000 mon.a (mon.0) 5600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:43.765Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:45 smithi121 bash[36902]: cluster 2022-04-22T18:40:43.991660+0000 mgr.y (mgr.34107) 2590 : cluster [DBG] pgmap v4182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:45.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:45 smithi115 bash[45495]: cluster 2022-04-22T18:40:43.991660+0000 mgr.y (mgr.34107) 2590 : cluster [DBG] pgmap v4182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:45 smithi115 bash[46934]: cluster 2022-04-22T18:40:43.991660+0000 mgr.y (mgr.34107) 2590 : cluster [DBG] pgmap v4182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:47 smithi121 bash[36902]: cluster 2022-04-22T18:40:45.992759+0000 mgr.y (mgr.34107) 2591 : cluster [DBG] pgmap v4183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:47 smithi121 bash[36902]: audit 2022-04-22T18:40:46.755699+0000 mon.a (mon.0) 5601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:47.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:47 smithi115 bash[45495]: cluster 2022-04-22T18:40:45.992759+0000 mgr.y (mgr.34107) 2591 : cluster [DBG] pgmap v4183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:47.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:47 smithi115 bash[45495]: audit 2022-04-22T18:40:46.755699+0000 mon.a (mon.0) 5601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:47.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:47 smithi115 bash[46934]: cluster 2022-04-22T18:40:45.992759+0000 mgr.y (mgr.34107) 2591 : cluster [DBG] pgmap v4183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:47 smithi115 bash[46934]: audit 2022-04-22T18:40:46.755699+0000 mon.a (mon.0) 5601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:48 smithi121 bash[36902]: audit 2022-04-22T18:40:47.380330+0000 mon.a (mon.0) 5602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:48 smithi115 bash[45495]: audit 2022-04-22T18:40:47.380330+0000 mon.a (mon.0) 5602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:48 smithi115 bash[46934]: audit 2022-04-22T18:40:47.380330+0000 mon.a (mon.0) 5602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:49 smithi121 bash[36902]: cluster 2022-04-22T18:40:47.993846+0000 mgr.y (mgr.34107) 2592 : cluster [DBG] pgmap v4184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:49 smithi115 bash[46934]: cluster 2022-04-22T18:40:47.993846+0000 mgr.y (mgr.34107) 2592 : cluster [DBG] pgmap v4184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:49.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:49 smithi115 bash[45495]: cluster 2022-04-22T18:40:47.993846+0000 mgr.y (mgr.34107) 2592 : cluster [DBG] pgmap v4184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:49.960 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:40:50.305 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:40:50.473 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (63m) 84s ago 77m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:40:50.473 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (77m) 2m ago 77m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:40:50.473 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (65m) 84s ago 78m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (65m) 2m ago 78m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (73m) 2m ago 82m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (73m) 84s ago 85m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (72m) 84s ago 85m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (71m) 2m ago 83m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (72m) 84s ago 83m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (64m) 84s ago 77m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (64m) 2m ago 77m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:40:50.474 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (70m) 84s ago 82m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (69m) 84s ago 82m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (69m) 84s ago 81m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (69m) 84s ago 81m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (68m) 2m ago 80m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (67m) 2m ago 80m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (66m) 2m ago 79m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (66m) 2m ago 79m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:40:50.475 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (64m) 2m ago 78m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:40:50.707 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:40:50.707 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:40:50.707 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:40:50.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:50.707 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:40:50.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:40:50.709 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:40:50.709 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:40:50.709 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:40:50.709 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:40:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:51 smithi121 bash[36902]: audit 2022-04-22T18:40:49.936462+0000 mgr.y (mgr.34107) 2593 : audit [DBG] from='client.38142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:51 smithi121 bash[36902]: cluster 2022-04-22T18:40:49.994901+0000 mgr.y (mgr.34107) 2594 : cluster [DBG] pgmap v4185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:51 smithi121 bash[36902]: audit 2022-04-22T18:40:50.283121+0000 mgr.y (mgr.34107) 2595 : audit [DBG] from='client.48082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:51 smithi121 bash[36902]: audit 2022-04-22T18:40:50.697528+0000 mon.c (mon.1) 152 : audit [DBG] from='client.? 172.21.15.115:0/3673901558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[45495]: audit 2022-04-22T18:40:49.936462+0000 mgr.y (mgr.34107) 2593 : audit [DBG] from='client.38142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[45495]: cluster 2022-04-22T18:40:49.994901+0000 mgr.y (mgr.34107) 2594 : cluster [DBG] pgmap v4185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[45495]: audit 2022-04-22T18:40:50.283121+0000 mgr.y (mgr.34107) 2595 : audit [DBG] from='client.48082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[45495]: audit 2022-04-22T18:40:50.697528+0000 mon.c (mon.1) 152 : audit [DBG] from='client.? 172.21.15.115:0/3673901558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[46934]: audit 2022-04-22T18:40:49.936462+0000 mgr.y (mgr.34107) 2593 : audit [DBG] from='client.38142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[46934]: cluster 2022-04-22T18:40:49.994901+0000 mgr.y (mgr.34107) 2594 : cluster [DBG] pgmap v4185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[46934]: audit 2022-04-22T18:40:50.283121+0000 mgr.y (mgr.34107) 2595 : audit [DBG] from='client.48082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:40:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[46934]: audit 2022-04-22T18:40:50.697528+0000 mon.c (mon.1) 152 : audit [DBG] from='client.? 172.21.15.115:0/3673901558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:40:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:51.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:52 smithi121 bash[36902]: audit 2022-04-22T18:40:51.758803+0000 mon.a (mon.0) 5603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:52.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:52 smithi115 bash[45495]: audit 2022-04-22T18:40:51.758803+0000 mon.a (mon.0) 5603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:52.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:52 smithi115 bash[46934]: audit 2022-04-22T18:40:51.758803+0000 mon.a (mon.0) 5603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:53 smithi121 bash[36902]: cluster 2022-04-22T18:40:51.996020+0000 mgr.y (mgr.34107) 2596 : cluster [DBG] pgmap v4186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:53 smithi121 bash[36902]: audit 2022-04-22T18:40:52.384000+0000 mon.a (mon.0) 5604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:53.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:53 smithi115 bash[46934]: cluster 2022-04-22T18:40:51.996020+0000 mgr.y (mgr.34107) 2596 : cluster [DBG] pgmap v4186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:53.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:53 smithi115 bash[46934]: audit 2022-04-22T18:40:52.384000+0000 mon.a (mon.0) 5604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:53.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:53 smithi115 bash[45495]: cluster 2022-04-22T18:40:51.996020+0000 mgr.y (mgr.34107) 2596 : cluster [DBG] pgmap v4186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:53.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:53 smithi115 bash[45495]: audit 2022-04-22T18:40:52.384000+0000 mon.a (mon.0) 5604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:40:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:40:53.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:40:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:55 smithi121 bash[36902]: cluster 2022-04-22T18:40:53.997111+0000 mgr.y (mgr.34107) 2597 : cluster [DBG] pgmap v4187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:55 smithi115 bash[46934]: cluster 2022-04-22T18:40:53.997111+0000 mgr.y (mgr.34107) 2597 : cluster [DBG] pgmap v4187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:55 smithi115 bash[45495]: cluster 2022-04-22T18:40:53.997111+0000 mgr.y (mgr.34107) 2597 : cluster [DBG] pgmap v4187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:57 smithi121 bash[36902]: cluster 2022-04-22T18:40:55.997779+0000 mgr.y (mgr.34107) 2598 : cluster [DBG] pgmap v4188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:57 smithi121 bash[36902]: audit 2022-04-22T18:40:56.761830+0000 mon.a (mon.0) 5605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:57.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:57 smithi115 bash[46934]: cluster 2022-04-22T18:40:55.997779+0000 mgr.y (mgr.34107) 2598 : cluster [DBG] pgmap v4188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:57.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:57 smithi115 bash[46934]: audit 2022-04-22T18:40:56.761830+0000 mon.a (mon.0) 5605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:57 smithi115 bash[45495]: cluster 2022-04-22T18:40:55.997779+0000 mgr.y (mgr.34107) 2598 : cluster [DBG] pgmap v4188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:57 smithi115 bash[45495]: audit 2022-04-22T18:40:56.761830+0000 mon.a (mon.0) 5605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:40:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:58 smithi121 bash[36902]: audit 2022-04-22T18:40:57.387730+0000 mon.a (mon.0) 5606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:58 smithi115 bash[46934]: audit 2022-04-22T18:40:57.387730+0000 mon.a (mon.0) 5606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:58.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:58 smithi115 bash[45495]: audit 2022-04-22T18:40:57.387730+0000 mon.a (mon.0) 5606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:40:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:40:59 smithi121 bash[36902]: cluster 2022-04-22T18:40:57.998767+0000 mgr.y (mgr.34107) 2599 : cluster [DBG] pgmap v4189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:40:59 smithi115 bash[46934]: cluster 2022-04-22T18:40:57.998767+0000 mgr.y (mgr.34107) 2599 : cluster [DBG] pgmap v4189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:40:59.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:40:59 smithi115 bash[45495]: cluster 2022-04-22T18:40:57.998767+0000 mgr.y (mgr.34107) 2599 : cluster [DBG] pgmap v4189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:01 smithi121 bash[36902]: cluster 2022-04-22T18:40:59.999877+0000 mgr.y (mgr.34107) 2600 : cluster [DBG] pgmap v4190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:01.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:01 smithi115 bash[46934]: cluster 2022-04-22T18:40:59.999877+0000 mgr.y (mgr.34107) 2600 : cluster [DBG] pgmap v4190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:01 smithi115 bash[45495]: cluster 2022-04-22T18:40:59.999877+0000 mgr.y (mgr.34107) 2600 : cluster [DBG] pgmap v4190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:01.883Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:02 smithi121 bash[36902]: audit 2022-04-22T18:41:01.764917+0000 mon.a (mon.0) 5607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:02 smithi115 bash[46934]: audit 2022-04-22T18:41:01.764917+0000 mon.a (mon.0) 5607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:02.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:02 smithi115 bash[45495]: audit 2022-04-22T18:41:01.764917+0000 mon.a (mon.0) 5607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:03 smithi121 bash[36902]: cluster 2022-04-22T18:41:02.000991+0000 mgr.y (mgr.34107) 2601 : cluster [DBG] pgmap v4191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:03 smithi121 bash[36902]: audit 2022-04-22T18:41:02.391424+0000 mon.a (mon.0) 5608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:03.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:03 smithi115 bash[45495]: cluster 2022-04-22T18:41:02.000991+0000 mgr.y (mgr.34107) 2601 : cluster [DBG] pgmap v4191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:03.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:03 smithi115 bash[45495]: audit 2022-04-22T18:41:02.391424+0000 mon.a (mon.0) 5608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:03.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:03 smithi115 bash[46934]: cluster 2022-04-22T18:41:02.000991+0000 mgr.y (mgr.34107) 2601 : cluster [DBG] pgmap v4191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:03.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:03 smithi115 bash[46934]: audit 2022-04-22T18:41:02.391424+0000 mon.a (mon.0) 5608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:03.766Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:05 smithi121 bash[36902]: cluster 2022-04-22T18:41:04.002044+0000 mgr.y (mgr.34107) 2602 : cluster [DBG] pgmap v4192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:05.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:05 smithi115 bash[45495]: cluster 2022-04-22T18:41:04.002044+0000 mgr.y (mgr.34107) 2602 : cluster [DBG] pgmap v4192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:05 smithi115 bash[46934]: cluster 2022-04-22T18:41:04.002044+0000 mgr.y (mgr.34107) 2602 : cluster [DBG] pgmap v4192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:07.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:07 smithi121 bash[36902]: cluster 2022-04-22T18:41:06.002660+0000 mgr.y (mgr.34107) 2603 : cluster [DBG] pgmap v4193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:07.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:07 smithi121 bash[36902]: audit 2022-04-22T18:41:06.768076+0000 mon.a (mon.0) 5609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:07 smithi115 bash[45495]: cluster 2022-04-22T18:41:06.002660+0000 mgr.y (mgr.34107) 2603 : cluster [DBG] pgmap v4193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:07 smithi115 bash[45495]: audit 2022-04-22T18:41:06.768076+0000 mon.a (mon.0) 5609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:07 smithi115 bash[46934]: cluster 2022-04-22T18:41:06.002660+0000 mgr.y (mgr.34107) 2603 : cluster [DBG] pgmap v4193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:07 smithi115 bash[46934]: audit 2022-04-22T18:41:06.768076+0000 mon.a (mon.0) 5609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:08 smithi121 bash[36902]: audit 2022-04-22T18:41:07.395021+0000 mon.a (mon.0) 5610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:08.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:08 smithi115 bash[45495]: audit 2022-04-22T18:41:07.395021+0000 mon.a (mon.0) 5610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:08 smithi115 bash[46934]: audit 2022-04-22T18:41:07.395021+0000 mon.a (mon.0) 5610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:09.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:09 smithi121 bash[36902]: cluster 2022-04-22T18:41:08.003734+0000 mgr.y (mgr.34107) 2604 : cluster [DBG] pgmap v4194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:09.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:09 smithi115 bash[45495]: cluster 2022-04-22T18:41:08.003734+0000 mgr.y (mgr.34107) 2604 : cluster [DBG] pgmap v4194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:09 smithi115 bash[46934]: cluster 2022-04-22T18:41:08.003734+0000 mgr.y (mgr.34107) 2604 : cluster [DBG] pgmap v4194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:11 smithi121 bash[36902]: cluster 2022-04-22T18:41:10.004805+0000 mgr.y (mgr.34107) 2605 : cluster [DBG] pgmap v4195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:11 smithi115 bash[46934]: cluster 2022-04-22T18:41:10.004805+0000 mgr.y (mgr.34107) 2605 : cluster [DBG] pgmap v4195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:11.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:11 smithi115 bash[45495]: cluster 2022-04-22T18:41:10.004805+0000 mgr.y (mgr.34107) 2605 : cluster [DBG] pgmap v4195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:11.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:12 smithi121 bash[36902]: audit 2022-04-22T18:41:11.771023+0000 mon.a (mon.0) 5611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:12 smithi115 bash[46934]: audit 2022-04-22T18:41:11.771023+0000 mon.a (mon.0) 5611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:12 smithi115 bash[45495]: audit 2022-04-22T18:41:11.771023+0000 mon.a (mon.0) 5611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:13 smithi121 bash[36902]: cluster 2022-04-22T18:41:12.005951+0000 mgr.y (mgr.34107) 2606 : cluster [DBG] pgmap v4196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:13 smithi121 bash[36902]: audit 2022-04-22T18:41:12.398483+0000 mon.a (mon.0) 5612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:13 smithi115 bash[45495]: cluster 2022-04-22T18:41:12.005951+0000 mgr.y (mgr.34107) 2606 : cluster [DBG] pgmap v4196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:13.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:13 smithi115 bash[45495]: audit 2022-04-22T18:41:12.398483+0000 mon.a (mon.0) 5612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:13.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:13 smithi115 bash[46934]: cluster 2022-04-22T18:41:12.005951+0000 mgr.y (mgr.34107) 2606 : cluster [DBG] pgmap v4196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:13.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:13 smithi115 bash[46934]: audit 2022-04-22T18:41:12.398483+0000 mon.a (mon.0) 5612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:13.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:14 smithi121 bash[36902]: audit 2022-04-22T18:41:14.085792+0000 mon.a (mon.0) 5613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:41:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:14 smithi121 bash[36902]: audit 2022-04-22T18:41:14.097655+0000 mon.a (mon.0) 5614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:41:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:14 smithi115 bash[45495]: audit 2022-04-22T18:41:14.085792+0000 mon.a (mon.0) 5613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:41:14.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:14 smithi115 bash[45495]: audit 2022-04-22T18:41:14.097655+0000 mon.a (mon.0) 5614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:41:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:14 smithi115 bash[46934]: audit 2022-04-22T18:41:14.085792+0000 mon.a (mon.0) 5613 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:41:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:14 smithi115 bash[46934]: audit 2022-04-22T18:41:14.097655+0000 mon.a (mon.0) 5614 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:15 smithi121 bash[36902]: cluster 2022-04-22T18:41:14.006999+0000 mgr.y (mgr.34107) 2607 : cluster [DBG] pgmap v4197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:15 smithi115 bash[45495]: cluster 2022-04-22T18:41:14.006999+0000 mgr.y (mgr.34107) 2607 : cluster [DBG] pgmap v4197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:15 smithi115 bash[46934]: cluster 2022-04-22T18:41:14.006999+0000 mgr.y (mgr.34107) 2607 : cluster [DBG] pgmap v4197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:17 smithi121 bash[36902]: cluster 2022-04-22T18:41:16.007660+0000 mgr.y (mgr.34107) 2608 : cluster [DBG] pgmap v4198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:17 smithi121 bash[36902]: audit 2022-04-22T18:41:16.774252+0000 mon.a (mon.0) 5615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:17 smithi121 bash[36902]: audit 2022-04-22T18:41:17.401844+0000 mon.a (mon.0) 5616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[45495]: cluster 2022-04-22T18:41:16.007660+0000 mgr.y (mgr.34107) 2608 : cluster [DBG] pgmap v4198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:17.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[45495]: audit 2022-04-22T18:41:16.774252+0000 mon.a (mon.0) 5615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:17.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[45495]: audit 2022-04-22T18:41:17.401844+0000 mon.a (mon.0) 5616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:17.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[46934]: cluster 2022-04-22T18:41:16.007660+0000 mgr.y (mgr.34107) 2608 : cluster [DBG] pgmap v4198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:17.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[46934]: audit 2022-04-22T18:41:16.774252+0000 mon.a (mon.0) 5615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:17.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:17 smithi115 bash[46934]: audit 2022-04-22T18:41:17.401844+0000 mon.a (mon.0) 5616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:19 smithi121 bash[36902]: cluster 2022-04-22T18:41:18.008672+0000 mgr.y (mgr.34107) 2609 : cluster [DBG] pgmap v4199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:19.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:19 smithi115 bash[45495]: cluster 2022-04-22T18:41:18.008672+0000 mgr.y (mgr.34107) 2609 : cluster [DBG] pgmap v4199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:19.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:19 smithi115 bash[46934]: cluster 2022-04-22T18:41:18.008672+0000 mgr.y (mgr.34107) 2609 : cluster [DBG] pgmap v4199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:21.079 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (64m) 115s ago 78m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (77m) 3m ago 77m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (66m) 115s ago 79m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (65m) 3m ago 79m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (73m) 3m ago 83m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:41:21.432 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (74m) 115s ago 85m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (73m) 115s ago 85m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (72m) 3m ago 83m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (72m) 115s ago 83m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (65m) 115s ago 78m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (65m) 3m ago 78m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (71m) 115s ago 83m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (70m) 115s ago 82m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (70m) 115s ago 82m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (69m) 115s ago 81m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:41:21.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (68m) 3m ago 81m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:41:21.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (68m) 3m ago 80m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:41:21.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (67m) 3m ago 80m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:41:21.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (66m) 3m ago 79m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:41:21.434 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (65m) 3m ago 78m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:41:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:21 smithi121 bash[36902]: cluster 2022-04-22T18:41:20.009749+0000 mgr.y (mgr.34107) 2610 : cluster [DBG] pgmap v4200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:21 smithi115 bash[45495]: cluster 2022-04-22T18:41:20.009749+0000 mgr.y (mgr.34107) 2610 : cluster [DBG] pgmap v4200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:21.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:21 smithi115 bash[46934]: cluster 2022-04-22T18:41:20.009749+0000 mgr.y (mgr.34107) 2610 : cluster [DBG] pgmap v4200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:21.839 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:41:21.840 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:41:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:21.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:22 smithi121 bash[36902]: audit 2022-04-22T18:41:21.055387+0000 mgr.y (mgr.34107) 2611 : audit [DBG] from='client.38160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:22 smithi121 bash[36902]: audit 2022-04-22T18:41:21.403038+0000 mgr.y (mgr.34107) 2612 : audit [DBG] from='client.38166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:22 smithi121 bash[36902]: audit 2022-04-22T18:41:21.776925+0000 mon.a (mon.0) 5617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:22 smithi121 bash[36902]: audit 2022-04-22T18:41:21.830707+0000 mon.a (mon.0) 5618 : audit [DBG] from='client.? 172.21.15.115:0/3819712744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:22 smithi121 bash[36902]: audit 2022-04-22T18:41:22.405650+0000 mon.a (mon.0) 5619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[45495]: audit 2022-04-22T18:41:21.055387+0000 mgr.y (mgr.34107) 2611 : audit [DBG] from='client.38160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[45495]: audit 2022-04-22T18:41:21.403038+0000 mgr.y (mgr.34107) 2612 : audit [DBG] from='client.38166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[45495]: audit 2022-04-22T18:41:21.776925+0000 mon.a (mon.0) 5617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[45495]: audit 2022-04-22T18:41:21.830707+0000 mon.a (mon.0) 5618 : audit [DBG] from='client.? 172.21.15.115:0/3819712744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[45495]: audit 2022-04-22T18:41:22.405650+0000 mon.a (mon.0) 5619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:22.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[46934]: audit 2022-04-22T18:41:21.055387+0000 mgr.y (mgr.34107) 2611 : audit [DBG] from='client.38160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[46934]: audit 2022-04-22T18:41:21.403038+0000 mgr.y (mgr.34107) 2612 : audit [DBG] from='client.38166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:22.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[46934]: audit 2022-04-22T18:41:21.776925+0000 mon.a (mon.0) 5617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:22.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[46934]: audit 2022-04-22T18:41:21.830707+0000 mon.a (mon.0) 5618 : audit [DBG] from='client.? 172.21.15.115:0/3819712744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:22.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:22 smithi115 bash[46934]: audit 2022-04-22T18:41:22.405650+0000 mon.a (mon.0) 5619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:24.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:23 smithi115 bash[45495]: cluster 2022-04-22T18:41:22.010851+0000 mgr.y (mgr.34107) 2613 : cluster [DBG] pgmap v4201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:23.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:23 smithi115 bash[46934]: cluster 2022-04-22T18:41:22.010851+0000 mgr.y (mgr.34107) 2613 : cluster [DBG] pgmap v4201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:23 smithi121 bash[36902]: cluster 2022-04-22T18:41:22.010851+0000 mgr.y (mgr.34107) 2613 : cluster [DBG] pgmap v4201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:25 smithi121 bash[36902]: cluster 2022-04-22T18:41:24.011944+0000 mgr.y (mgr.34107) 2614 : cluster [DBG] pgmap v4202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:25 smithi115 bash[45495]: cluster 2022-04-22T18:41:24.011944+0000 mgr.y (mgr.34107) 2614 : cluster [DBG] pgmap v4202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:25 smithi115 bash[46934]: cluster 2022-04-22T18:41:24.011944+0000 mgr.y (mgr.34107) 2614 : cluster [DBG] pgmap v4202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:26 smithi121 bash[36902]: audit 2022-04-22T18:41:26.779713+0000 mon.a (mon.0) 5620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:27.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:26 smithi115 bash[45495]: audit 2022-04-22T18:41:26.779713+0000 mon.a (mon.0) 5620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:27.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:26 smithi115 bash[46934]: audit 2022-04-22T18:41:26.779713+0000 mon.a (mon.0) 5620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:27 smithi121 bash[36902]: cluster 2022-04-22T18:41:26.012851+0000 mgr.y (mgr.34107) 2615 : cluster [DBG] pgmap v4203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:27 smithi121 bash[36902]: audit 2022-04-22T18:41:27.409342+0000 mon.a (mon.0) 5621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:27 smithi115 bash[45495]: cluster 2022-04-22T18:41:26.012851+0000 mgr.y (mgr.34107) 2615 : cluster [DBG] pgmap v4203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:27 smithi115 bash[45495]: audit 2022-04-22T18:41:27.409342+0000 mon.a (mon.0) 5621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:28.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:27 smithi115 bash[46934]: cluster 2022-04-22T18:41:26.012851+0000 mgr.y (mgr.34107) 2615 : cluster [DBG] pgmap v4203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:27 smithi115 bash[46934]: audit 2022-04-22T18:41:27.409342+0000 mon.a (mon.0) 5621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:29 smithi121 bash[36902]: cluster 2022-04-22T18:41:28.013978+0000 mgr.y (mgr.34107) 2616 : cluster [DBG] pgmap v4204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:29 smithi115 bash[45495]: cluster 2022-04-22T18:41:28.013978+0000 mgr.y (mgr.34107) 2616 : cluster [DBG] pgmap v4204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:30.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:29 smithi115 bash[46934]: cluster 2022-04-22T18:41:28.013978+0000 mgr.y (mgr.34107) 2616 : cluster [DBG] pgmap v4204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:31 smithi121 bash[36902]: cluster 2022-04-22T18:41:30.015048+0000 mgr.y (mgr.34107) 2617 : cluster [DBG] pgmap v4205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:31 smithi121 bash[36902]: audit 2022-04-22T18:41:31.782619+0000 mon.a (mon.0) 5622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:32.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:31.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:32.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:31 smithi115 bash[45495]: cluster 2022-04-22T18:41:30.015048+0000 mgr.y (mgr.34107) 2617 : cluster [DBG] pgmap v4205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:32.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:31 smithi115 bash[45495]: audit 2022-04-22T18:41:31.782619+0000 mon.a (mon.0) 5622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:31 smithi115 bash[46934]: cluster 2022-04-22T18:41:30.015048+0000 mgr.y (mgr.34107) 2617 : cluster [DBG] pgmap v4205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:31 smithi115 bash[46934]: audit 2022-04-22T18:41:31.782619+0000 mon.a (mon.0) 5622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:32 smithi121 bash[36902]: audit 2022-04-22T18:41:32.413140+0000 mon.a (mon.0) 5623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:33.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:32 smithi115 bash[45495]: audit 2022-04-22T18:41:32.413140+0000 mon.a (mon.0) 5623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:32 smithi115 bash[46934]: audit 2022-04-22T18:41:32.413140+0000 mon.a (mon.0) 5623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:33.905 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:33.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:33.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:33 smithi115 bash[46934]: cluster 2022-04-22T18:41:32.016287+0000 mgr.y (mgr.34107) 2618 : cluster [DBG] pgmap v4206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:33.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:33 smithi115 bash[45495]: cluster 2022-04-22T18:41:32.016287+0000 mgr.y (mgr.34107) 2618 : cluster [DBG] pgmap v4206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:33 smithi121 bash[36902]: cluster 2022-04-22T18:41:32.016287+0000 mgr.y (mgr.34107) 2618 : cluster [DBG] pgmap v4206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:35 smithi121 bash[36902]: cluster 2022-04-22T18:41:34.017347+0000 mgr.y (mgr.34107) 2619 : cluster [DBG] pgmap v4207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:35 smithi121 bash[36902]: audit 2022-04-22T18:41:35.287448+0000 mon.a (mon.0) 5624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:41:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:35 smithi121 bash[36902]: audit 2022-04-22T18:41:35.290342+0000 mon.a (mon.0) 5625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:41:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:35 smithi121 bash[36902]: audit 2022-04-22T18:41:35.292218+0000 mon.a (mon.0) 5626 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:41:36.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:35 smithi121 bash[36902]: audit 2022-04-22T18:41:35.662282+0000 mon.a (mon.0) 5627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:41:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[45495]: cluster 2022-04-22T18:41:34.017347+0000 mgr.y (mgr.34107) 2619 : cluster [DBG] pgmap v4207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[45495]: audit 2022-04-22T18:41:35.287448+0000 mon.a (mon.0) 5624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:41:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[45495]: audit 2022-04-22T18:41:35.290342+0000 mon.a (mon.0) 5625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[45495]: audit 2022-04-22T18:41:35.292218+0000 mon.a (mon.0) 5626 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[45495]: audit 2022-04-22T18:41:35.662282+0000 mon.a (mon.0) 5627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[46934]: cluster 2022-04-22T18:41:34.017347+0000 mgr.y (mgr.34107) 2619 : cluster [DBG] pgmap v4207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[46934]: audit 2022-04-22T18:41:35.287448+0000 mon.a (mon.0) 5624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[46934]: audit 2022-04-22T18:41:35.290342+0000 mon.a (mon.0) 5625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[46934]: audit 2022-04-22T18:41:35.292218+0000 mon.a (mon.0) 5626 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:41:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:35 smithi115 bash[46934]: audit 2022-04-22T18:41:35.662282+0000 mon.a (mon.0) 5627 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:41:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:36 smithi121 bash[36902]: audit 2022-04-22T18:41:36.785550+0000 mon.a (mon.0) 5628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:36 smithi115 bash[45495]: audit 2022-04-22T18:41:36.785550+0000 mon.a (mon.0) 5628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:36 smithi115 bash[46934]: audit 2022-04-22T18:41:36.785550+0000 mon.a (mon.0) 5628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:37 smithi121 bash[36902]: cluster 2022-04-22T18:41:36.018330+0000 mgr.y (mgr.34107) 2620 : cluster [DBG] pgmap v4208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:37 smithi121 bash[36902]: audit 2022-04-22T18:41:37.416645+0000 mon.a (mon.0) 5629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:37 smithi115 bash[45495]: cluster 2022-04-22T18:41:36.018330+0000 mgr.y (mgr.34107) 2620 : cluster [DBG] pgmap v4208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:37 smithi115 bash[45495]: audit 2022-04-22T18:41:37.416645+0000 mon.a (mon.0) 5629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:37 smithi115 bash[46934]: cluster 2022-04-22T18:41:36.018330+0000 mgr.y (mgr.34107) 2620 : cluster [DBG] pgmap v4208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:37 smithi115 bash[46934]: audit 2022-04-22T18:41:37.416645+0000 mon.a (mon.0) 5629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:38 smithi121 bash[36902]: cluster 2022-04-22T18:41:38.019210+0000 mgr.y (mgr.34107) 2621 : cluster [DBG] pgmap v4209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:38 smithi115 bash[45495]: cluster 2022-04-22T18:41:38.019210+0000 mgr.y (mgr.34107) 2621 : cluster [DBG] pgmap v4209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:38 smithi115 bash[46934]: cluster 2022-04-22T18:41:38.019210+0000 mgr.y (mgr.34107) 2621 : cluster [DBG] pgmap v4209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:41 smithi121 bash[36902]: cluster 2022-04-22T18:41:40.020324+0000 mgr.y (mgr.34107) 2622 : cluster [DBG] pgmap v4210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:41 smithi115 bash[45495]: cluster 2022-04-22T18:41:40.020324+0000 mgr.y (mgr.34107) 2622 : cluster [DBG] pgmap v4210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:41 smithi115 bash[46934]: cluster 2022-04-22T18:41:40.020324+0000 mgr.y (mgr.34107) 2622 : cluster [DBG] pgmap v4210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:42.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:42 smithi115 bash[45495]: audit 2022-04-22T18:41:41.788608+0000 mon.a (mon.0) 5630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:42.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:42 smithi115 bash[46934]: audit 2022-04-22T18:41:41.788608+0000 mon.a (mon.0) 5630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:41.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:42.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:42 smithi121 bash[36902]: audit 2022-04-22T18:41:41.788608+0000 mon.a (mon.0) 5630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:43 smithi121 bash[36902]: cluster 2022-04-22T18:41:42.021406+0000 mgr.y (mgr.34107) 2623 : cluster [DBG] pgmap v4211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:43 smithi121 bash[36902]: audit 2022-04-22T18:41:42.420284+0000 mon.a (mon.0) 5631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:43 smithi115 bash[45495]: cluster 2022-04-22T18:41:42.021406+0000 mgr.y (mgr.34107) 2623 : cluster [DBG] pgmap v4211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:43 smithi115 bash[45495]: audit 2022-04-22T18:41:42.420284+0000 mon.a (mon.0) 5631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:43 smithi115 bash[46934]: cluster 2022-04-22T18:41:42.021406+0000 mgr.y (mgr.34107) 2623 : cluster [DBG] pgmap v4211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:43 smithi115 bash[46934]: audit 2022-04-22T18:41:42.420284+0000 mon.a (mon.0) 5631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:43.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:45 smithi121 bash[36902]: cluster 2022-04-22T18:41:44.022479+0000 mgr.y (mgr.34107) 2624 : cluster [DBG] pgmap v4212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:45 smithi115 bash[45495]: cluster 2022-04-22T18:41:44.022479+0000 mgr.y (mgr.34107) 2624 : cluster [DBG] pgmap v4212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:45 smithi115 bash[46934]: cluster 2022-04-22T18:41:44.022479+0000 mgr.y (mgr.34107) 2624 : cluster [DBG] pgmap v4212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:47 smithi121 bash[36902]: cluster 2022-04-22T18:41:46.023278+0000 mgr.y (mgr.34107) 2625 : cluster [DBG] pgmap v4213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:47 smithi121 bash[36902]: audit 2022-04-22T18:41:46.791784+0000 mon.a (mon.0) 5632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:47 smithi115 bash[45495]: cluster 2022-04-22T18:41:46.023278+0000 mgr.y (mgr.34107) 2625 : cluster [DBG] pgmap v4213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:47 smithi115 bash[45495]: audit 2022-04-22T18:41:46.791784+0000 mon.a (mon.0) 5632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:47 smithi115 bash[46934]: cluster 2022-04-22T18:41:46.023278+0000 mgr.y (mgr.34107) 2625 : cluster [DBG] pgmap v4213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:47 smithi115 bash[46934]: audit 2022-04-22T18:41:46.791784+0000 mon.a (mon.0) 5632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:48 smithi121 bash[36902]: audit 2022-04-22T18:41:47.424106+0000 mon.a (mon.0) 5633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:48 smithi115 bash[45495]: audit 2022-04-22T18:41:47.424106+0000 mon.a (mon.0) 5633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:48 smithi115 bash[46934]: audit 2022-04-22T18:41:47.424106+0000 mon.a (mon.0) 5633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:49 smithi121 bash[36902]: cluster 2022-04-22T18:41:48.024354+0000 mgr.y (mgr.34107) 2626 : cluster [DBG] pgmap v4214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:49 smithi115 bash[45495]: cluster 2022-04-22T18:41:48.024354+0000 mgr.y (mgr.34107) 2626 : cluster [DBG] pgmap v4214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:49 smithi115 bash[46934]: cluster 2022-04-22T18:41:48.024354+0000 mgr.y (mgr.34107) 2626 : cluster [DBG] pgmap v4214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:51 smithi121 bash[36902]: cluster 2022-04-22T18:41:50.025525+0000 mgr.y (mgr.34107) 2627 : cluster [DBG] pgmap v4215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:51 smithi115 bash[45495]: cluster 2022-04-22T18:41:50.025525+0000 mgr.y (mgr.34107) 2627 : cluster [DBG] pgmap v4215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:51.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:51 smithi115 bash[46934]: cluster 2022-04-22T18:41:50.025525+0000 mgr.y (mgr.34107) 2627 : cluster [DBG] pgmap v4215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:52.221 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:41:52.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:52 smithi115 bash[45495]: audit 2022-04-22T18:41:51.795066+0000 mon.a (mon.0) 5634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:51.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:52.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:52 smithi115 bash[46934]: audit 2022-04-22T18:41:51.795066+0000 mon.a (mon.0) 5634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:52 smithi121 bash[36902]: audit 2022-04-22T18:41:51.795066+0000 mon.a (mon.0) 5634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:41:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (64m) 2m ago 78m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:41:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (78m) 3m ago 78m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:41:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (66m) 2m ago 79m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:41:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (66m) 3m ago 79m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (74m) 3m ago 83m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (74m) 2m ago 86m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (73m) 2m ago 86m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (72m) 3m ago 84m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (73m) 2m ago 84m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:41:52.574 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (66m) 2m ago 78m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (65m) 3m ago 78m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (71m) 2m ago 83m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (70m) 2m ago 83m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (70m) 2m ago 82m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (70m) 2m ago 82m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (69m) 3m ago 81m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (68m) 3m ago 81m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (68m) 3m ago 80m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:41:52.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (67m) 3m ago 80m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:41:52.576 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (65m) 3m ago 79m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:41:52.979 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:41:52.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:41:52.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:41:52.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:41:52.981 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:41:52.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:41:52.982 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:41:52.982 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:53 smithi121 bash[36902]: cluster 2022-04-22T18:41:52.026668+0000 mgr.y (mgr.34107) 2628 : cluster [DBG] pgmap v4216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:53 smithi121 bash[36902]: audit 2022-04-22T18:41:52.197899+0000 mgr.y (mgr.34107) 2629 : audit [DBG] from='client.38178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:53 smithi121 bash[36902]: audit 2022-04-22T18:41:52.427959+0000 mon.a (mon.0) 5635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:53 smithi121 bash[36902]: audit 2022-04-22T18:41:52.543831+0000 mgr.y (mgr.34107) 2630 : audit [DBG] from='client.48118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:53 smithi121 bash[36902]: audit 2022-04-22T18:41:52.971344+0000 mon.a (mon.0) 5636 : audit [DBG] from='client.? 172.21.15.115:0/3938295534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[45495]: cluster 2022-04-22T18:41:52.026668+0000 mgr.y (mgr.34107) 2628 : cluster [DBG] pgmap v4216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[45495]: audit 2022-04-22T18:41:52.197899+0000 mgr.y (mgr.34107) 2629 : audit [DBG] from='client.38178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[45495]: audit 2022-04-22T18:41:52.427959+0000 mon.a (mon.0) 5635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[45495]: audit 2022-04-22T18:41:52.543831+0000 mgr.y (mgr.34107) 2630 : audit [DBG] from='client.48118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[45495]: audit 2022-04-22T18:41:52.971344+0000 mon.a (mon.0) 5636 : audit [DBG] from='client.? 172.21.15.115:0/3938295534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[46934]: cluster 2022-04-22T18:41:52.026668+0000 mgr.y (mgr.34107) 2628 : cluster [DBG] pgmap v4216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[46934]: audit 2022-04-22T18:41:52.197899+0000 mgr.y (mgr.34107) 2629 : audit [DBG] from='client.38178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[46934]: audit 2022-04-22T18:41:52.427959+0000 mon.a (mon.0) 5635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[46934]: audit 2022-04-22T18:41:52.543831+0000 mgr.y (mgr.34107) 2630 : audit [DBG] from='client.48118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:41:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[46934]: audit 2022-04-22T18:41:52.971344+0000 mon.a (mon.0) 5636 : audit [DBG] from='client.? 172.21.15.115:0/3938295534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:41:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:41:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:41:53.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:41:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:55 smithi121 bash[36902]: cluster 2022-04-22T18:41:54.027748+0000 mgr.y (mgr.34107) 2631 : cluster [DBG] pgmap v4217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:55 smithi115 bash[46934]: cluster 2022-04-22T18:41:54.027748+0000 mgr.y (mgr.34107) 2631 : cluster [DBG] pgmap v4217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:55 smithi115 bash[45495]: cluster 2022-04-22T18:41:54.027748+0000 mgr.y (mgr.34107) 2631 : cluster [DBG] pgmap v4217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:57 smithi121 bash[36902]: cluster 2022-04-22T18:41:56.028562+0000 mgr.y (mgr.34107) 2632 : cluster [DBG] pgmap v4218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:57 smithi121 bash[36902]: audit 2022-04-22T18:41:56.798262+0000 mon.a (mon.0) 5637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:57 smithi115 bash[46934]: cluster 2022-04-22T18:41:56.028562+0000 mgr.y (mgr.34107) 2632 : cluster [DBG] pgmap v4218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:57 smithi115 bash[46934]: audit 2022-04-22T18:41:56.798262+0000 mon.a (mon.0) 5637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:57 smithi115 bash[45495]: cluster 2022-04-22T18:41:56.028562+0000 mgr.y (mgr.34107) 2632 : cluster [DBG] pgmap v4218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:57 smithi115 bash[45495]: audit 2022-04-22T18:41:56.798262+0000 mon.a (mon.0) 5637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:41:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:58 smithi121 bash[36902]: audit 2022-04-22T18:41:57.431430+0000 mon.a (mon.0) 5638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:58 smithi115 bash[45495]: audit 2022-04-22T18:41:57.431430+0000 mon.a (mon.0) 5638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:58 smithi115 bash[46934]: audit 2022-04-22T18:41:57.431430+0000 mon.a (mon.0) 5638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:41:59.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:41:59 smithi121 bash[36902]: cluster 2022-04-22T18:41:58.029585+0000 mgr.y (mgr.34107) 2633 : cluster [DBG] pgmap v4219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:41:59 smithi115 bash[45495]: cluster 2022-04-22T18:41:58.029585+0000 mgr.y (mgr.34107) 2633 : cluster [DBG] pgmap v4219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:41:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:41:59 smithi115 bash[46934]: cluster 2022-04-22T18:41:58.029585+0000 mgr.y (mgr.34107) 2633 : cluster [DBG] pgmap v4219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:01 smithi121 bash[36902]: cluster 2022-04-22T18:42:00.030725+0000 mgr.y (mgr.34107) 2634 : cluster [DBG] pgmap v4220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:01 smithi115 bash[45495]: cluster 2022-04-22T18:42:00.030725+0000 mgr.y (mgr.34107) 2634 : cluster [DBG] pgmap v4220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:01 smithi115 bash[46934]: cluster 2022-04-22T18:42:00.030725+0000 mgr.y (mgr.34107) 2634 : cluster [DBG] pgmap v4220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:02.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:02 smithi115 bash[45495]: audit 2022-04-22T18:42:01.801459+0000 mon.a (mon.0) 5639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:01.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:02 smithi115 bash[46934]: audit 2022-04-22T18:42:01.801459+0000 mon.a (mon.0) 5639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:02 smithi121 bash[36902]: audit 2022-04-22T18:42:01.801459+0000 mon.a (mon.0) 5639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:03 smithi121 bash[36902]: cluster 2022-04-22T18:42:02.031921+0000 mgr.y (mgr.34107) 2635 : cluster [DBG] pgmap v4221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:03 smithi121 bash[36902]: audit 2022-04-22T18:42:02.435255+0000 mon.a (mon.0) 5640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:03 smithi115 bash[45495]: cluster 2022-04-22T18:42:02.031921+0000 mgr.y (mgr.34107) 2635 : cluster [DBG] pgmap v4221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:03 smithi115 bash[45495]: audit 2022-04-22T18:42:02.435255+0000 mon.a (mon.0) 5640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:03 smithi115 bash[46934]: cluster 2022-04-22T18:42:02.031921+0000 mgr.y (mgr.34107) 2635 : cluster [DBG] pgmap v4221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:03 smithi115 bash[46934]: audit 2022-04-22T18:42:02.435255+0000 mon.a (mon.0) 5640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:03.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:05 smithi121 bash[36902]: cluster 2022-04-22T18:42:04.032956+0000 mgr.y (mgr.34107) 2636 : cluster [DBG] pgmap v4222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:05 smithi115 bash[45495]: cluster 2022-04-22T18:42:04.032956+0000 mgr.y (mgr.34107) 2636 : cluster [DBG] pgmap v4222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:05.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:05 smithi115 bash[46934]: cluster 2022-04-22T18:42:04.032956+0000 mgr.y (mgr.34107) 2636 : cluster [DBG] pgmap v4222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:07 smithi121 bash[36902]: cluster 2022-04-22T18:42:06.033896+0000 mgr.y (mgr.34107) 2637 : cluster [DBG] pgmap v4223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:07 smithi121 bash[36902]: audit 2022-04-22T18:42:06.804650+0000 mon.a (mon.0) 5641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:07 smithi115 bash[45495]: cluster 2022-04-22T18:42:06.033896+0000 mgr.y (mgr.34107) 2637 : cluster [DBG] pgmap v4223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:07 smithi115 bash[45495]: audit 2022-04-22T18:42:06.804650+0000 mon.a (mon.0) 5641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:07 smithi115 bash[46934]: cluster 2022-04-22T18:42:06.033896+0000 mgr.y (mgr.34107) 2637 : cluster [DBG] pgmap v4223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:07 smithi115 bash[46934]: audit 2022-04-22T18:42:06.804650+0000 mon.a (mon.0) 5641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:08 smithi121 bash[36902]: audit 2022-04-22T18:42:07.438769+0000 mon.a (mon.0) 5642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:08 smithi115 bash[45495]: audit 2022-04-22T18:42:07.438769+0000 mon.a (mon.0) 5642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:08 smithi115 bash[46934]: audit 2022-04-22T18:42:07.438769+0000 mon.a (mon.0) 5642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:09 smithi121 bash[36902]: cluster 2022-04-22T18:42:08.035037+0000 mgr.y (mgr.34107) 2638 : cluster [DBG] pgmap v4224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:09 smithi115 bash[45495]: cluster 2022-04-22T18:42:08.035037+0000 mgr.y (mgr.34107) 2638 : cluster [DBG] pgmap v4224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:09 smithi115 bash[46934]: cluster 2022-04-22T18:42:08.035037+0000 mgr.y (mgr.34107) 2638 : cluster [DBG] pgmap v4224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:11 smithi121 bash[36902]: cluster 2022-04-22T18:42:10.036161+0000 mgr.y (mgr.34107) 2639 : cluster [DBG] pgmap v4225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:11 smithi115 bash[46934]: cluster 2022-04-22T18:42:10.036161+0000 mgr.y (mgr.34107) 2639 : cluster [DBG] pgmap v4225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:11 smithi115 bash[45495]: cluster 2022-04-22T18:42:10.036161+0000 mgr.y (mgr.34107) 2639 : cluster [DBG] pgmap v4225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:12.175 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:11.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:12 smithi121 bash[36902]: audit 2022-04-22T18:42:11.807588+0000 mon.a (mon.0) 5643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:12 smithi115 bash[45495]: audit 2022-04-22T18:42:11.807588+0000 mon.a (mon.0) 5643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:12.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:12 smithi115 bash[46934]: audit 2022-04-22T18:42:11.807588+0000 mon.a (mon.0) 5643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:13 smithi121 bash[36902]: cluster 2022-04-22T18:42:12.037317+0000 mgr.y (mgr.34107) 2640 : cluster [DBG] pgmap v4226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:13 smithi121 bash[36902]: audit 2022-04-22T18:42:12.442507+0000 mon.a (mon.0) 5644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:13 smithi115 bash[45495]: cluster 2022-04-22T18:42:12.037317+0000 mgr.y (mgr.34107) 2640 : cluster [DBG] pgmap v4226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:13 smithi115 bash[45495]: audit 2022-04-22T18:42:12.442507+0000 mon.a (mon.0) 5644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:13 smithi115 bash[46934]: cluster 2022-04-22T18:42:12.037317+0000 mgr.y (mgr.34107) 2640 : cluster [DBG] pgmap v4226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:13 smithi115 bash[46934]: audit 2022-04-22T18:42:12.442507+0000 mon.a (mon.0) 5644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:13.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:14 smithi121 bash[36902]: audit 2022-04-22T18:42:14.092853+0000 mon.a (mon.0) 5645 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:42:14.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:14 smithi121 bash[36902]: audit 2022-04-22T18:42:14.103796+0000 mon.a (mon.0) 5646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:42:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:14 smithi115 bash[45495]: audit 2022-04-22T18:42:14.092853+0000 mon.a (mon.0) 5645 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:42:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:14 smithi115 bash[45495]: audit 2022-04-22T18:42:14.103796+0000 mon.a (mon.0) 5646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:42:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:14 smithi115 bash[46934]: audit 2022-04-22T18:42:14.092853+0000 mon.a (mon.0) 5645 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:42:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:14 smithi115 bash[46934]: audit 2022-04-22T18:42:14.103796+0000 mon.a (mon.0) 5646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:42:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:15 smithi121 bash[36902]: cluster 2022-04-22T18:42:14.038496+0000 mgr.y (mgr.34107) 2641 : cluster [DBG] pgmap v4227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:15 smithi115 bash[45495]: cluster 2022-04-22T18:42:14.038496+0000 mgr.y (mgr.34107) 2641 : cluster [DBG] pgmap v4227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:15 smithi115 bash[46934]: cluster 2022-04-22T18:42:14.038496+0000 mgr.y (mgr.34107) 2641 : cluster [DBG] pgmap v4227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:17 smithi121 bash[36902]: cluster 2022-04-22T18:42:16.039260+0000 mgr.y (mgr.34107) 2642 : cluster [DBG] pgmap v4228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:17 smithi121 bash[36902]: audit 2022-04-22T18:42:16.810767+0000 mon.a (mon.0) 5647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:17 smithi115 bash[45495]: cluster 2022-04-22T18:42:16.039260+0000 mgr.y (mgr.34107) 2642 : cluster [DBG] pgmap v4228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:17 smithi115 bash[45495]: audit 2022-04-22T18:42:16.810767+0000 mon.a (mon.0) 5647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:17 smithi115 bash[46934]: cluster 2022-04-22T18:42:16.039260+0000 mgr.y (mgr.34107) 2642 : cluster [DBG] pgmap v4228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:17 smithi115 bash[46934]: audit 2022-04-22T18:42:16.810767+0000 mon.a (mon.0) 5647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:18 smithi121 bash[36902]: audit 2022-04-22T18:42:17.446024+0000 mon.a (mon.0) 5648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:18 smithi115 bash[45495]: audit 2022-04-22T18:42:17.446024+0000 mon.a (mon.0) 5648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:18 smithi115 bash[46934]: audit 2022-04-22T18:42:17.446024+0000 mon.a (mon.0) 5648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:19 smithi121 bash[36902]: cluster 2022-04-22T18:42:18.040406+0000 mgr.y (mgr.34107) 2643 : cluster [DBG] pgmap v4229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:19 smithi115 bash[45495]: cluster 2022-04-22T18:42:18.040406+0000 mgr.y (mgr.34107) 2643 : cluster [DBG] pgmap v4229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:19 smithi115 bash[46934]: cluster 2022-04-22T18:42:18.040406+0000 mgr.y (mgr.34107) 2643 : cluster [DBG] pgmap v4229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:21 smithi121 bash[36902]: cluster 2022-04-22T18:42:20.041399+0000 mgr.y (mgr.34107) 2644 : cluster [DBG] pgmap v4230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:21 smithi115 bash[45495]: cluster 2022-04-22T18:42:20.041399+0000 mgr.y (mgr.34107) 2644 : cluster [DBG] pgmap v4230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:21 smithi115 bash[46934]: cluster 2022-04-22T18:42:20.041399+0000 mgr.y (mgr.34107) 2644 : cluster [DBG] pgmap v4230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:22.209 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:21.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:22 smithi121 bash[36902]: audit 2022-04-22T18:42:21.814074+0000 mon.a (mon.0) 5649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:22 smithi115 bash[45495]: audit 2022-04-22T18:42:21.814074+0000 mon.a (mon.0) 5649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:22 smithi115 bash[46934]: audit 2022-04-22T18:42:21.814074+0000 mon.a (mon.0) 5649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:23.360 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:42:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:23 smithi121 bash[36902]: cluster 2022-04-22T18:42:22.042657+0000 mgr.y (mgr.34107) 2645 : cluster [DBG] pgmap v4231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:23 smithi121 bash[36902]: audit 2022-04-22T18:42:22.449995+0000 mon.a (mon.0) 5650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:23 smithi115 bash[46934]: cluster 2022-04-22T18:42:22.042657+0000 mgr.y (mgr.34107) 2645 : cluster [DBG] pgmap v4231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:23 smithi115 bash[46934]: audit 2022-04-22T18:42:22.449995+0000 mon.a (mon.0) 5650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:23 smithi115 bash[45495]: cluster 2022-04-22T18:42:22.042657+0000 mgr.y (mgr.34107) 2645 : cluster [DBG] pgmap v4231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:23 smithi115 bash[45495]: audit 2022-04-22T18:42:22.449995+0000 mon.a (mon.0) 5650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:23.703 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (65m) 2m ago 79m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (78m) 4m ago 78m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (67m) 2m ago 80m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (66m) 4m ago 80m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (74m) 4m ago 84m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (75m) 2m ago 86m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (74m) 2m ago 86m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:42:23.704 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (73m) 4m ago 84m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (73m) 2m ago 84m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (66m) 2m ago 79m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (66m) 4m ago 79m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (72m) 2m ago 84m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (71m) 2m ago 83m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (71m) 2m ago 83m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (70m) 2m ago 82m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (69m) 4m ago 82m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (69m) 4m ago 81m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:42:23.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (68m) 4m ago 81m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:42:23.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (67m) 4m ago 80m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:42:23.706 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (66m) 4m ago 79m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:42:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:23.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:24.108 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:42:24.108 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:42:24.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:42:24.110 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:42:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:24 smithi121 bash[36902]: audit 2022-04-22T18:42:23.335232+0000 mgr.y (mgr.34107) 2646 : audit [DBG] from='client.48130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:24 smithi121 bash[36902]: audit 2022-04-22T18:42:23.676707+0000 mgr.y (mgr.34107) 2647 : audit [DBG] from='client.38202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:24 smithi121 bash[36902]: audit 2022-04-22T18:42:24.100088+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.115:0/2332243161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[46934]: audit 2022-04-22T18:42:23.335232+0000 mgr.y (mgr.34107) 2646 : audit [DBG] from='client.48130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[46934]: audit 2022-04-22T18:42:23.676707+0000 mgr.y (mgr.34107) 2647 : audit [DBG] from='client.38202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[46934]: audit 2022-04-22T18:42:24.100088+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.115:0/2332243161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[45495]: audit 2022-04-22T18:42:23.335232+0000 mgr.y (mgr.34107) 2646 : audit [DBG] from='client.48130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[45495]: audit 2022-04-22T18:42:23.676707+0000 mgr.y (mgr.34107) 2647 : audit [DBG] from='client.38202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:24 smithi115 bash[45495]: audit 2022-04-22T18:42:24.100088+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.115:0/2332243161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:25 smithi121 bash[36902]: cluster 2022-04-22T18:42:24.043809+0000 mgr.y (mgr.34107) 2648 : cluster [DBG] pgmap v4232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:25 smithi115 bash[45495]: cluster 2022-04-22T18:42:24.043809+0000 mgr.y (mgr.34107) 2648 : cluster [DBG] pgmap v4232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:25 smithi115 bash[46934]: cluster 2022-04-22T18:42:24.043809+0000 mgr.y (mgr.34107) 2648 : cluster [DBG] pgmap v4232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:27 smithi121 bash[36902]: cluster 2022-04-22T18:42:26.044680+0000 mgr.y (mgr.34107) 2649 : cluster [DBG] pgmap v4233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:27 smithi121 bash[36902]: audit 2022-04-22T18:42:26.817011+0000 mon.a (mon.0) 5651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:27 smithi115 bash[45495]: cluster 2022-04-22T18:42:26.044680+0000 mgr.y (mgr.34107) 2649 : cluster [DBG] pgmap v4233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:27 smithi115 bash[45495]: audit 2022-04-22T18:42:26.817011+0000 mon.a (mon.0) 5651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:27 smithi115 bash[46934]: cluster 2022-04-22T18:42:26.044680+0000 mgr.y (mgr.34107) 2649 : cluster [DBG] pgmap v4233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:27 smithi115 bash[46934]: audit 2022-04-22T18:42:26.817011+0000 mon.a (mon.0) 5651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:28 smithi121 bash[36902]: audit 2022-04-22T18:42:27.453819+0000 mon.a (mon.0) 5652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:28 smithi115 bash[45495]: audit 2022-04-22T18:42:27.453819+0000 mon.a (mon.0) 5652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:28 smithi115 bash[46934]: audit 2022-04-22T18:42:27.453819+0000 mon.a (mon.0) 5652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:29 smithi121 bash[36902]: cluster 2022-04-22T18:42:28.045783+0000 mgr.y (mgr.34107) 2650 : cluster [DBG] pgmap v4234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:29 smithi115 bash[45495]: cluster 2022-04-22T18:42:28.045783+0000 mgr.y (mgr.34107) 2650 : cluster [DBG] pgmap v4234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:29 smithi115 bash[46934]: cluster 2022-04-22T18:42:28.045783+0000 mgr.y (mgr.34107) 2650 : cluster [DBG] pgmap v4234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:31 smithi121 bash[36902]: cluster 2022-04-22T18:42:30.046666+0000 mgr.y (mgr.34107) 2651 : cluster [DBG] pgmap v4235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:31.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:31 smithi115 bash[45495]: cluster 2022-04-22T18:42:30.046666+0000 mgr.y (mgr.34107) 2651 : cluster [DBG] pgmap v4235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:31 smithi115 bash[46934]: cluster 2022-04-22T18:42:30.046666+0000 mgr.y (mgr.34107) 2651 : cluster [DBG] pgmap v4235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:32.241 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:31.892Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:32 smithi121 bash[36902]: audit 2022-04-22T18:42:31.819970+0000 mon.a (mon.0) 5653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:32 smithi115 bash[45495]: audit 2022-04-22T18:42:31.819970+0000 mon.a (mon.0) 5653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:32.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:32 smithi115 bash[46934]: audit 2022-04-22T18:42:31.819970+0000 mon.a (mon.0) 5653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:33 smithi121 bash[36902]: cluster 2022-04-22T18:42:32.047903+0000 mgr.y (mgr.34107) 2652 : cluster [DBG] pgmap v4236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:33 smithi121 bash[36902]: audit 2022-04-22T18:42:32.457561+0000 mon.a (mon.0) 5654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:33 smithi115 bash[46934]: cluster 2022-04-22T18:42:32.047903+0000 mgr.y (mgr.34107) 2652 : cluster [DBG] pgmap v4236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:33 smithi115 bash[46934]: audit 2022-04-22T18:42:32.457561+0000 mon.a (mon.0) 5654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:33 smithi115 bash[45495]: cluster 2022-04-22T18:42:32.047903+0000 mgr.y (mgr.34107) 2652 : cluster [DBG] pgmap v4236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:33 smithi115 bash[45495]: audit 2022-04-22T18:42:32.457561+0000 mon.a (mon.0) 5654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:33.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:35 smithi121 bash[36902]: cluster 2022-04-22T18:42:34.049011+0000 mgr.y (mgr.34107) 2653 : cluster [DBG] pgmap v4237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:35 smithi115 bash[45495]: cluster 2022-04-22T18:42:34.049011+0000 mgr.y (mgr.34107) 2653 : cluster [DBG] pgmap v4237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:35 smithi115 bash[46934]: cluster 2022-04-22T18:42:34.049011+0000 mgr.y (mgr.34107) 2653 : cluster [DBG] pgmap v4237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[45495]: audit 2022-04-22T18:42:35.671718+0000 mon.a (mon.0) 5655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:42:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[45495]: audit 2022-04-22T18:42:35.674843+0000 mon.a (mon.0) 5656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[45495]: audit 2022-04-22T18:42:35.676852+0000 mon.a (mon.0) 5657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[45495]: audit 2022-04-22T18:42:36.041949+0000 mon.a (mon.0) 5658 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[46934]: audit 2022-04-22T18:42:35.671718+0000 mon.a (mon.0) 5655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[46934]: audit 2022-04-22T18:42:35.674843+0000 mon.a (mon.0) 5656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[46934]: audit 2022-04-22T18:42:35.676852+0000 mon.a (mon.0) 5657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:42:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:36 smithi115 bash[46934]: audit 2022-04-22T18:42:36.041949+0000 mon.a (mon.0) 5658 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:42:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:36 smithi121 bash[36902]: audit 2022-04-22T18:42:35.671718+0000 mon.a (mon.0) 5655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:42:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:36 smithi121 bash[36902]: audit 2022-04-22T18:42:35.674843+0000 mon.a (mon.0) 5656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:42:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:36 smithi121 bash[36902]: audit 2022-04-22T18:42:35.676852+0000 mon.a (mon.0) 5657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:42:36.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:36 smithi121 bash[36902]: audit 2022-04-22T18:42:36.041949+0000 mon.a (mon.0) 5658 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:42:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:37 smithi121 bash[36902]: cluster 2022-04-22T18:42:36.049623+0000 mgr.y (mgr.34107) 2654 : cluster [DBG] pgmap v4238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:37 smithi121 bash[36902]: audit 2022-04-22T18:42:36.823216+0000 mon.a (mon.0) 5659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:37 smithi115 bash[45495]: cluster 2022-04-22T18:42:36.049623+0000 mgr.y (mgr.34107) 2654 : cluster [DBG] pgmap v4238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:37 smithi115 bash[45495]: audit 2022-04-22T18:42:36.823216+0000 mon.a (mon.0) 5659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:37 smithi115 bash[46934]: cluster 2022-04-22T18:42:36.049623+0000 mgr.y (mgr.34107) 2654 : cluster [DBG] pgmap v4238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:37 smithi115 bash[46934]: audit 2022-04-22T18:42:36.823216+0000 mon.a (mon.0) 5659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:38 smithi121 bash[36902]: audit 2022-04-22T18:42:37.461282+0000 mon.a (mon.0) 5660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:38 smithi115 bash[45495]: audit 2022-04-22T18:42:37.461282+0000 mon.a (mon.0) 5660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:38 smithi115 bash[46934]: audit 2022-04-22T18:42:37.461282+0000 mon.a (mon.0) 5660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:39 smithi115 bash[45495]: cluster 2022-04-22T18:42:38.050813+0000 mgr.y (mgr.34107) 2655 : cluster [DBG] pgmap v4239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:39 smithi115 bash[46934]: cluster 2022-04-22T18:42:38.050813+0000 mgr.y (mgr.34107) 2655 : cluster [DBG] pgmap v4239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:39 smithi121 bash[36902]: cluster 2022-04-22T18:42:38.050813+0000 mgr.y (mgr.34107) 2655 : cluster [DBG] pgmap v4239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:41 smithi115 bash[45495]: cluster 2022-04-22T18:42:40.051808+0000 mgr.y (mgr.34107) 2656 : cluster [DBG] pgmap v4240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:41 smithi115 bash[46934]: cluster 2022-04-22T18:42:40.051808+0000 mgr.y (mgr.34107) 2656 : cluster [DBG] pgmap v4240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:41 smithi121 bash[36902]: cluster 2022-04-22T18:42:40.051808+0000 mgr.y (mgr.34107) 2656 : cluster [DBG] pgmap v4240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:42.270 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:41.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:42 smithi115 bash[45495]: audit 2022-04-22T18:42:41.826164+0000 mon.a (mon.0) 5661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:42.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:42 smithi115 bash[46934]: audit 2022-04-22T18:42:41.826164+0000 mon.a (mon.0) 5661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:42 smithi121 bash[36902]: audit 2022-04-22T18:42:41.826164+0000 mon.a (mon.0) 5661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:43 smithi115 bash[45495]: cluster 2022-04-22T18:42:42.052972+0000 mgr.y (mgr.34107) 2657 : cluster [DBG] pgmap v4241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:43 smithi115 bash[45495]: audit 2022-04-22T18:42:42.465157+0000 mon.a (mon.0) 5662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:43 smithi115 bash[46934]: cluster 2022-04-22T18:42:42.052972+0000 mgr.y (mgr.34107) 2657 : cluster [DBG] pgmap v4241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:43 smithi115 bash[46934]: audit 2022-04-22T18:42:42.465157+0000 mon.a (mon.0) 5662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:43 smithi121 bash[36902]: cluster 2022-04-22T18:42:42.052972+0000 mgr.y (mgr.34107) 2657 : cluster [DBG] pgmap v4241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:43 smithi121 bash[36902]: audit 2022-04-22T18:42:42.465157+0000 mon.a (mon.0) 5662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:43.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:45 smithi115 bash[45495]: cluster 2022-04-22T18:42:44.054077+0000 mgr.y (mgr.34107) 2658 : cluster [DBG] pgmap v4242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:45 smithi115 bash[46934]: cluster 2022-04-22T18:42:44.054077+0000 mgr.y (mgr.34107) 2658 : cluster [DBG] pgmap v4242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:45 smithi121 bash[36902]: cluster 2022-04-22T18:42:44.054077+0000 mgr.y (mgr.34107) 2658 : cluster [DBG] pgmap v4242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:47 smithi115 bash[45495]: cluster 2022-04-22T18:42:46.055180+0000 mgr.y (mgr.34107) 2659 : cluster [DBG] pgmap v4243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:47 smithi115 bash[45495]: audit 2022-04-22T18:42:46.829049+0000 mon.a (mon.0) 5663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:47 smithi115 bash[46934]: cluster 2022-04-22T18:42:46.055180+0000 mgr.y (mgr.34107) 2659 : cluster [DBG] pgmap v4243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:47 smithi115 bash[46934]: audit 2022-04-22T18:42:46.829049+0000 mon.a (mon.0) 5663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:47 smithi121 bash[36902]: cluster 2022-04-22T18:42:46.055180+0000 mgr.y (mgr.34107) 2659 : cluster [DBG] pgmap v4243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:47 smithi121 bash[36902]: audit 2022-04-22T18:42:46.829049+0000 mon.a (mon.0) 5663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:48 smithi115 bash[45495]: audit 2022-04-22T18:42:47.468939+0000 mon.a (mon.0) 5664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:48 smithi115 bash[46934]: audit 2022-04-22T18:42:47.468939+0000 mon.a (mon.0) 5664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:48 smithi121 bash[36902]: audit 2022-04-22T18:42:47.468939+0000 mon.a (mon.0) 5664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:49 smithi115 bash[45495]: cluster 2022-04-22T18:42:48.056299+0000 mgr.y (mgr.34107) 2660 : cluster [DBG] pgmap v4244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:49 smithi115 bash[45495]: audit 2022-04-22T18:42:48.649060+0000 mon.a (mon.0) 5665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:49.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:49 smithi115 bash[46934]: cluster 2022-04-22T18:42:48.056299+0000 mgr.y (mgr.34107) 2660 : cluster [DBG] pgmap v4244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:49.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:49 smithi115 bash[46934]: audit 2022-04-22T18:42:48.649060+0000 mon.a (mon.0) 5665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:49 smithi121 bash[36902]: cluster 2022-04-22T18:42:48.056299+0000 mgr.y (mgr.34107) 2660 : cluster [DBG] pgmap v4244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:49 smithi121 bash[36902]: audit 2022-04-22T18:42:48.649060+0000 mon.a (mon.0) 5665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:51 smithi121 bash[36902]: cluster 2022-04-22T18:42:50.057216+0000 mgr.y (mgr.34107) 2661 : cluster [DBG] pgmap v4245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:51.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:51 smithi115 bash[46934]: cluster 2022-04-22T18:42:50.057216+0000 mgr.y (mgr.34107) 2661 : cluster [DBG] pgmap v4245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:51 smithi115 bash[45495]: cluster 2022-04-22T18:42:50.057216+0000 mgr.y (mgr.34107) 2661 : cluster [DBG] pgmap v4245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:51.894Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:52 smithi121 bash[36902]: audit 2022-04-22T18:42:51.832059+0000 mon.a (mon.0) 5666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:52.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:52 smithi115 bash[45495]: audit 2022-04-22T18:42:51.832059+0000 mon.a (mon.0) 5666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:52.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:52 smithi115 bash[46934]: audit 2022-04-22T18:42:51.832059+0000 mon.a (mon.0) 5666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:53 smithi121 bash[36902]: cluster 2022-04-22T18:42:52.058216+0000 mgr.y (mgr.34107) 2662 : cluster [DBG] pgmap v4246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:53 smithi121 bash[36902]: audit 2022-04-22T18:42:52.472299+0000 mon.a (mon.0) 5667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:53.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:53 smithi115 bash[45495]: cluster 2022-04-22T18:42:52.058216+0000 mgr.y (mgr.34107) 2662 : cluster [DBG] pgmap v4246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:53.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:53 smithi115 bash[45495]: audit 2022-04-22T18:42:52.472299+0000 mon.a (mon.0) 5667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:53.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:53 smithi115 bash[46934]: cluster 2022-04-22T18:42:52.058216+0000 mgr.y (mgr.34107) 2662 : cluster [DBG] pgmap v4246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:53.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:53 smithi115 bash[46934]: audit 2022-04-22T18:42:52.472299+0000 mon.a (mon.0) 5667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:42:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:42:53.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:42:54.487 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:42:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:54 smithi121 bash[36902]: audit 2022-04-22T18:42:53.652285+0000 mon.a (mon.0) 5668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:54.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:54 smithi115 bash[46934]: audit 2022-04-22T18:42:53.652285+0000 mon.a (mon.0) 5668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:54.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:54 smithi115 bash[45495]: audit 2022-04-22T18:42:53.652285+0000 mon.a (mon.0) 5668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:54.840 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:42:54.925 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (65m) 3m ago 79m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:42:54.925 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (79m) 4m ago 79m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:42:54.925 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (67m) 3m ago 80m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:42:54.926 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (67m) 4m ago 80m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:42:54.926 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (75m) 4m ago 85m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:42:54.927 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (75m) 3m ago 87m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:42:54.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (74m) 3m ago 87m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:42:54.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (73m) 4m ago 85m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:42:54.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (74m) 3m ago 85m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:42:54.928 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (67m) 3m ago 80m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:42:54.928 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (67m) 4m ago 79m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:42:54.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (72m) 3m ago 84m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:42:54.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (72m) 3m ago 84m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:42:54.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (71m) 3m ago 83m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:42:54.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (71m) 3m ago 83m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:42:54.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (70m) 4m ago 82m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:42:54.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (69m) 4m ago 82m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:42:54.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (69m) 4m ago 81m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:42:54.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (68m) 4m ago 81m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:42:54.930 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (66m) 4m ago 80m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:42:55.242 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:42:55.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:42:55.244 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:42:55.244 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:55 smithi121 bash[36902]: cluster 2022-04-22T18:42:54.059225+0000 mgr.y (mgr.34107) 2663 : cluster [DBG] pgmap v4247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:55 smithi121 bash[36902]: audit 2022-04-22T18:42:54.462000+0000 mgr.y (mgr.34107) 2664 : audit [DBG] from='client.48148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:55 smithi121 bash[36902]: audit 2022-04-22T18:42:54.812745+0000 mgr.y (mgr.34107) 2665 : audit [DBG] from='client.48154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:55 smithi121 bash[36902]: audit 2022-04-22T18:42:55.233436+0000 mon.a (mon.0) 5669 : audit [DBG] from='client.? 172.21.15.115:0/3093937075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[45495]: cluster 2022-04-22T18:42:54.059225+0000 mgr.y (mgr.34107) 2663 : cluster [DBG] pgmap v4247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[45495]: audit 2022-04-22T18:42:54.462000+0000 mgr.y (mgr.34107) 2664 : audit [DBG] from='client.48148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[45495]: audit 2022-04-22T18:42:54.812745+0000 mgr.y (mgr.34107) 2665 : audit [DBG] from='client.48154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[45495]: audit 2022-04-22T18:42:55.233436+0000 mon.a (mon.0) 5669 : audit [DBG] from='client.? 172.21.15.115:0/3093937075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[46934]: cluster 2022-04-22T18:42:54.059225+0000 mgr.y (mgr.34107) 2663 : cluster [DBG] pgmap v4247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[46934]: audit 2022-04-22T18:42:54.462000+0000 mgr.y (mgr.34107) 2664 : audit [DBG] from='client.48148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[46934]: audit 2022-04-22T18:42:54.812745+0000 mgr.y (mgr.34107) 2665 : audit [DBG] from='client.48154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:42:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:55 smithi115 bash[46934]: audit 2022-04-22T18:42:55.233436+0000 mon.a (mon.0) 5669 : audit [DBG] from='client.? 172.21.15.115:0/3093937075' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:42:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:57 smithi121 bash[36902]: cluster 2022-04-22T18:42:56.060298+0000 mgr.y (mgr.34107) 2666 : cluster [DBG] pgmap v4248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:57 smithi121 bash[36902]: audit 2022-04-22T18:42:56.835384+0000 mon.a (mon.0) 5670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:57.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:57 smithi115 bash[46934]: cluster 2022-04-22T18:42:56.060298+0000 mgr.y (mgr.34107) 2666 : cluster [DBG] pgmap v4248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:57 smithi115 bash[46934]: audit 2022-04-22T18:42:56.835384+0000 mon.a (mon.0) 5670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:57 smithi115 bash[45495]: cluster 2022-04-22T18:42:56.060298+0000 mgr.y (mgr.34107) 2666 : cluster [DBG] pgmap v4248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:57 smithi115 bash[45495]: audit 2022-04-22T18:42:56.835384+0000 mon.a (mon.0) 5670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:42:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:58 smithi121 bash[36902]: audit 2022-04-22T18:42:57.475815+0000 mon.a (mon.0) 5671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:58 smithi115 bash[46934]: audit 2022-04-22T18:42:57.475815+0000 mon.a (mon.0) 5671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:58 smithi115 bash[45495]: audit 2022-04-22T18:42:57.475815+0000 mon.a (mon.0) 5671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:42:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:59 smithi121 bash[36902]: cluster 2022-04-22T18:42:58.061351+0000 mgr.y (mgr.34107) 2667 : cluster [DBG] pgmap v4249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:42:59 smithi121 bash[36902]: audit 2022-04-22T18:42:58.655550+0000 mon.a (mon.0) 5672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:59 smithi115 bash[45495]: cluster 2022-04-22T18:42:58.061351+0000 mgr.y (mgr.34107) 2667 : cluster [DBG] pgmap v4249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:42:59 smithi115 bash[45495]: audit 2022-04-22T18:42:58.655550+0000 mon.a (mon.0) 5672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:42:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:59 smithi115 bash[46934]: cluster 2022-04-22T18:42:58.061351+0000 mgr.y (mgr.34107) 2667 : cluster [DBG] pgmap v4249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:42:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:42:59 smithi115 bash[46934]: audit 2022-04-22T18:42:58.655550+0000 mon.a (mon.0) 5672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:01 smithi121 bash[36902]: cluster 2022-04-22T18:43:00.062269+0000 mgr.y (mgr.34107) 2668 : cluster [DBG] pgmap v4250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:01.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:01 smithi115 bash[45495]: cluster 2022-04-22T18:43:00.062269+0000 mgr.y (mgr.34107) 2668 : cluster [DBG] pgmap v4250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:01 smithi115 bash[46934]: cluster 2022-04-22T18:43:00.062269+0000 mgr.y (mgr.34107) 2668 : cluster [DBG] pgmap v4250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:02.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:01.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:02 smithi121 bash[36902]: audit 2022-04-22T18:43:01.838737+0000 mon.a (mon.0) 5673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:02.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:02 smithi115 bash[45495]: audit 2022-04-22T18:43:01.838737+0000 mon.a (mon.0) 5673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:02.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:02 smithi115 bash[46934]: audit 2022-04-22T18:43:01.838737+0000 mon.a (mon.0) 5673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:03 smithi121 bash[36902]: cluster 2022-04-22T18:43:02.063433+0000 mgr.y (mgr.34107) 2669 : cluster [DBG] pgmap v4251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:03 smithi121 bash[36902]: audit 2022-04-22T18:43:02.479515+0000 mon.a (mon.0) 5674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:03.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:03 smithi115 bash[46934]: cluster 2022-04-22T18:43:02.063433+0000 mgr.y (mgr.34107) 2669 : cluster [DBG] pgmap v4251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:03.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:03 smithi115 bash[46934]: audit 2022-04-22T18:43:02.479515+0000 mon.a (mon.0) 5674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:03.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:03 smithi115 bash[45495]: cluster 2022-04-22T18:43:02.063433+0000 mgr.y (mgr.34107) 2669 : cluster [DBG] pgmap v4251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:03.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:03 smithi115 bash[45495]: audit 2022-04-22T18:43:02.479515+0000 mon.a (mon.0) 5674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:04.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:03.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:04 smithi121 bash[36902]: audit 2022-04-22T18:43:03.658523+0000 mon.a (mon.0) 5675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:04.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:04 smithi115 bash[46934]: audit 2022-04-22T18:43:03.658523+0000 mon.a (mon.0) 5675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:04.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:04 smithi115 bash[45495]: audit 2022-04-22T18:43:03.658523+0000 mon.a (mon.0) 5675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:05 smithi121 bash[36902]: cluster 2022-04-22T18:43:04.064455+0000 mgr.y (mgr.34107) 2670 : cluster [DBG] pgmap v4252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:05 smithi115 bash[46934]: cluster 2022-04-22T18:43:04.064455+0000 mgr.y (mgr.34107) 2670 : cluster [DBG] pgmap v4252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:05.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:05 smithi115 bash[45495]: cluster 2022-04-22T18:43:04.064455+0000 mgr.y (mgr.34107) 2670 : cluster [DBG] pgmap v4252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:07 smithi121 bash[36902]: cluster 2022-04-22T18:43:06.065543+0000 mgr.y (mgr.34107) 2671 : cluster [DBG] pgmap v4253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:07 smithi121 bash[36902]: audit 2022-04-22T18:43:06.841966+0000 mon.a (mon.0) 5676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:07.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:07 smithi115 bash[46934]: cluster 2022-04-22T18:43:06.065543+0000 mgr.y (mgr.34107) 2671 : cluster [DBG] pgmap v4253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:07 smithi115 bash[46934]: audit 2022-04-22T18:43:06.841966+0000 mon.a (mon.0) 5676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:07 smithi115 bash[45495]: cluster 2022-04-22T18:43:06.065543+0000 mgr.y (mgr.34107) 2671 : cluster [DBG] pgmap v4253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:07 smithi115 bash[45495]: audit 2022-04-22T18:43:06.841966+0000 mon.a (mon.0) 5676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:08.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:08 smithi121 bash[36902]: audit 2022-04-22T18:43:07.482948+0000 mon.a (mon.0) 5677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:08 smithi115 bash[46934]: audit 2022-04-22T18:43:07.482948+0000 mon.a (mon.0) 5677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:08 smithi115 bash[45495]: audit 2022-04-22T18:43:07.482948+0000 mon.a (mon.0) 5677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:09 smithi121 bash[36902]: cluster 2022-04-22T18:43:08.066587+0000 mgr.y (mgr.34107) 2672 : cluster [DBG] pgmap v4254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:09 smithi121 bash[36902]: audit 2022-04-22T18:43:08.661786+0000 mon.a (mon.0) 5678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:09.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:09 smithi115 bash[45495]: cluster 2022-04-22T18:43:08.066587+0000 mgr.y (mgr.34107) 2672 : cluster [DBG] pgmap v4254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:09 smithi115 bash[45495]: audit 2022-04-22T18:43:08.661786+0000 mon.a (mon.0) 5678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:09 smithi115 bash[46934]: cluster 2022-04-22T18:43:08.066587+0000 mgr.y (mgr.34107) 2672 : cluster [DBG] pgmap v4254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:09 smithi115 bash[46934]: audit 2022-04-22T18:43:08.661786+0000 mon.a (mon.0) 5678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:11 smithi121 bash[36902]: cluster 2022-04-22T18:43:10.067561+0000 mgr.y (mgr.34107) 2673 : cluster [DBG] pgmap v4255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:11 smithi115 bash[46934]: cluster 2022-04-22T18:43:10.067561+0000 mgr.y (mgr.34107) 2673 : cluster [DBG] pgmap v4255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:11 smithi115 bash[45495]: cluster 2022-04-22T18:43:10.067561+0000 mgr.y (mgr.34107) 2673 : cluster [DBG] pgmap v4255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:11.897Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:12 smithi121 bash[36902]: audit 2022-04-22T18:43:11.845063+0000 mon.a (mon.0) 5679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:12 smithi115 bash[46934]: audit 2022-04-22T18:43:11.845063+0000 mon.a (mon.0) 5679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:12.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:12 smithi115 bash[45495]: audit 2022-04-22T18:43:11.845063+0000 mon.a (mon.0) 5679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:13 smithi121 bash[36902]: cluster 2022-04-22T18:43:12.068785+0000 mgr.y (mgr.34107) 2674 : cluster [DBG] pgmap v4256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:13 smithi121 bash[36902]: audit 2022-04-22T18:43:12.486726+0000 mon.a (mon.0) 5680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:13.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:13 smithi115 bash[45495]: cluster 2022-04-22T18:43:12.068785+0000 mgr.y (mgr.34107) 2674 : cluster [DBG] pgmap v4256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:13.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:13 smithi115 bash[45495]: audit 2022-04-22T18:43:12.486726+0000 mon.a (mon.0) 5680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:13.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:13 smithi115 bash[46934]: cluster 2022-04-22T18:43:12.068785+0000 mgr.y (mgr.34107) 2674 : cluster [DBG] pgmap v4256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:13.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:13 smithi115 bash[46934]: audit 2022-04-22T18:43:12.486726+0000 mon.a (mon.0) 5680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:14.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:13.774Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:14 smithi121 bash[36902]: audit 2022-04-22T18:43:13.664920+0000 mon.a (mon.0) 5681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:14 smithi121 bash[36902]: audit 2022-04-22T18:43:14.102699+0000 mon.a (mon.0) 5682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:43:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:14 smithi121 bash[36902]: audit 2022-04-22T18:43:14.108944+0000 mon.a (mon.0) 5683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:43:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[45495]: audit 2022-04-22T18:43:13.664920+0000 mon.a (mon.0) 5681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[45495]: audit 2022-04-22T18:43:14.102699+0000 mon.a (mon.0) 5682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:43:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[45495]: audit 2022-04-22T18:43:14.108944+0000 mon.a (mon.0) 5683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:43:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[46934]: audit 2022-04-22T18:43:13.664920+0000 mon.a (mon.0) 5681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[46934]: audit 2022-04-22T18:43:14.102699+0000 mon.a (mon.0) 5682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:43:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:14 smithi115 bash[46934]: audit 2022-04-22T18:43:14.108944+0000 mon.a (mon.0) 5683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:43:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:15 smithi121 bash[36902]: cluster 2022-04-22T18:43:14.070000+0000 mgr.y (mgr.34107) 2675 : cluster [DBG] pgmap v4257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:15 smithi115 bash[46934]: cluster 2022-04-22T18:43:14.070000+0000 mgr.y (mgr.34107) 2675 : cluster [DBG] pgmap v4257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:15 smithi115 bash[45495]: cluster 2022-04-22T18:43:14.070000+0000 mgr.y (mgr.34107) 2675 : cluster [DBG] pgmap v4257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:17 smithi121 bash[36902]: cluster 2022-04-22T18:43:16.071022+0000 mgr.y (mgr.34107) 2676 : cluster [DBG] pgmap v4258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:17 smithi121 bash[36902]: audit 2022-04-22T18:43:16.848298+0000 mon.a (mon.0) 5684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:17 smithi115 bash[46934]: cluster 2022-04-22T18:43:16.071022+0000 mgr.y (mgr.34107) 2676 : cluster [DBG] pgmap v4258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:17 smithi115 bash[46934]: audit 2022-04-22T18:43:16.848298+0000 mon.a (mon.0) 5684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:17 smithi115 bash[45495]: cluster 2022-04-22T18:43:16.071022+0000 mgr.y (mgr.34107) 2676 : cluster [DBG] pgmap v4258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:17 smithi115 bash[45495]: audit 2022-04-22T18:43:16.848298+0000 mon.a (mon.0) 5684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:18 smithi121 bash[36902]: audit 2022-04-22T18:43:17.490393+0000 mon.a (mon.0) 5685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:18 smithi115 bash[45495]: audit 2022-04-22T18:43:17.490393+0000 mon.a (mon.0) 5685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:18 smithi115 bash[46934]: audit 2022-04-22T18:43:17.490393+0000 mon.a (mon.0) 5685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:19 smithi121 bash[36902]: cluster 2022-04-22T18:43:18.072132+0000 mgr.y (mgr.34107) 2677 : cluster [DBG] pgmap v4259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:19 smithi121 bash[36902]: audit 2022-04-22T18:43:18.667983+0000 mon.a (mon.0) 5686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:19 smithi115 bash[46934]: cluster 2022-04-22T18:43:18.072132+0000 mgr.y (mgr.34107) 2677 : cluster [DBG] pgmap v4259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:19 smithi115 bash[46934]: audit 2022-04-22T18:43:18.667983+0000 mon.a (mon.0) 5686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:19 smithi115 bash[45495]: cluster 2022-04-22T18:43:18.072132+0000 mgr.y (mgr.34107) 2677 : cluster [DBG] pgmap v4259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:19 smithi115 bash[45495]: audit 2022-04-22T18:43:18.667983+0000 mon.a (mon.0) 5686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:21 smithi121 bash[36902]: cluster 2022-04-22T18:43:20.073123+0000 mgr.y (mgr.34107) 2678 : cluster [DBG] pgmap v4260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:21 smithi115 bash[45495]: cluster 2022-04-22T18:43:20.073123+0000 mgr.y (mgr.34107) 2678 : cluster [DBG] pgmap v4260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:21.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:21 smithi115 bash[46934]: cluster 2022-04-22T18:43:20.073123+0000 mgr.y (mgr.34107) 2678 : cluster [DBG] pgmap v4260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:22.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:21.898Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:22 smithi121 bash[36902]: audit 2022-04-22T18:43:21.851347+0000 mon.a (mon.0) 5687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:22 smithi115 bash[45495]: audit 2022-04-22T18:43:21.851347+0000 mon.a (mon.0) 5687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:22 smithi115 bash[46934]: audit 2022-04-22T18:43:21.851347+0000 mon.a (mon.0) 5687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:23 smithi121 bash[36902]: cluster 2022-04-22T18:43:22.074294+0000 mgr.y (mgr.34107) 2679 : cluster [DBG] pgmap v4261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:23 smithi121 bash[36902]: audit 2022-04-22T18:43:22.493961+0000 mon.a (mon.0) 5688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:23.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:23 smithi115 bash[45495]: cluster 2022-04-22T18:43:22.074294+0000 mgr.y (mgr.34107) 2679 : cluster [DBG] pgmap v4261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:23.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:23 smithi115 bash[45495]: audit 2022-04-22T18:43:22.493961+0000 mon.a (mon.0) 5688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:23.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:23 smithi115 bash[46934]: cluster 2022-04-22T18:43:22.074294+0000 mgr.y (mgr.34107) 2679 : cluster [DBG] pgmap v4261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:23.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:23 smithi115 bash[46934]: audit 2022-04-22T18:43:22.493961+0000 mon.a (mon.0) 5688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:23.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:24 smithi121 bash[36902]: audit 2022-04-22T18:43:23.670783+0000 mon.a (mon.0) 5689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:24.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:24 smithi115 bash[45495]: audit 2022-04-22T18:43:23.670783+0000 mon.a (mon.0) 5689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:24.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:24 smithi115 bash[46934]: audit 2022-04-22T18:43:23.670783+0000 mon.a (mon.0) 5689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:25.612 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:43:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:25 smithi121 bash[36902]: cluster 2022-04-22T18:43:24.075395+0000 mgr.y (mgr.34107) 2680 : cluster [DBG] pgmap v4262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:25 smithi115 bash[46934]: cluster 2022-04-22T18:43:24.075395+0000 mgr.y (mgr.34107) 2680 : cluster [DBG] pgmap v4262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:25 smithi115 bash[45495]: cluster 2022-04-22T18:43:24.075395+0000 mgr.y (mgr.34107) 2680 : cluster [DBG] pgmap v4262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:25.949 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (66m) 4m ago 80m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (80m) 5m ago 80m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (68m) 4m ago 81m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (68m) 5m ago 81m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (75m) 5m ago 85m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:43:25.950 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (76m) 4m ago 87m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (75m) 4m ago 87m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (74m) 5m ago 85m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (74m) 4m ago 86m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (67m) 4m ago 80m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (67m) 5m ago 80m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (73m) 4m ago 85m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:43:25.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (72m) 4m ago 84m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:43:25.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (72m) 4m ago 84m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:43:25.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (71m) 4m ago 83m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:43:25.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (70m) 5m ago 83m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:43:25.952 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (70m) 5m ago 82m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:43:25.953 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (69m) 5m ago 82m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:43:25.953 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (68m) 5m ago 81m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:43:25.953 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (67m) 5m ago 80m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:43:26.362 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:43:26.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:26.363 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:43:26.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:43:26.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:43:26.365 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:43:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:26 smithi121 bash[36902]: audit 2022-04-22T18:43:25.587287+0000 mgr.y (mgr.34107) 2681 : audit [DBG] from='client.48166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:26 smithi121 bash[36902]: audit 2022-04-22T18:43:25.921599+0000 mgr.y (mgr.34107) 2682 : audit [DBG] from='client.38238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:26 smithi121 bash[36902]: audit 2022-04-22T18:43:26.353493+0000 mon.c (mon.1) 154 : audit [DBG] from='client.? 172.21.15.115:0/4275658709' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:26.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[45495]: audit 2022-04-22T18:43:25.587287+0000 mgr.y (mgr.34107) 2681 : audit [DBG] from='client.48166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[45495]: audit 2022-04-22T18:43:25.921599+0000 mgr.y (mgr.34107) 2682 : audit [DBG] from='client.38238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[45495]: audit 2022-04-22T18:43:26.353493+0000 mon.c (mon.1) 154 : audit [DBG] from='client.? 172.21.15.115:0/4275658709' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:26.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[46934]: audit 2022-04-22T18:43:25.587287+0000 mgr.y (mgr.34107) 2681 : audit [DBG] from='client.48166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[46934]: audit 2022-04-22T18:43:25.921599+0000 mgr.y (mgr.34107) 2682 : audit [DBG] from='client.38238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:26.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:26 smithi115 bash[46934]: audit 2022-04-22T18:43:26.353493+0000 mon.c (mon.1) 154 : audit [DBG] from='client.? 172.21.15.115:0/4275658709' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:27 smithi121 bash[36902]: cluster 2022-04-22T18:43:26.076329+0000 mgr.y (mgr.34107) 2683 : cluster [DBG] pgmap v4263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:27 smithi121 bash[36902]: audit 2022-04-22T18:43:26.854372+0000 mon.a (mon.0) 5690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:27.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:27 smithi115 bash[45495]: cluster 2022-04-22T18:43:26.076329+0000 mgr.y (mgr.34107) 2683 : cluster [DBG] pgmap v4263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:27.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:27 smithi115 bash[45495]: audit 2022-04-22T18:43:26.854372+0000 mon.a (mon.0) 5690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:27 smithi115 bash[46934]: cluster 2022-04-22T18:43:26.076329+0000 mgr.y (mgr.34107) 2683 : cluster [DBG] pgmap v4263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:27 smithi115 bash[46934]: audit 2022-04-22T18:43:26.854372+0000 mon.a (mon.0) 5690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:28 smithi121 bash[36902]: audit 2022-04-22T18:43:27.497556+0000 mon.a (mon.0) 5691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:28 smithi115 bash[45495]: audit 2022-04-22T18:43:27.497556+0000 mon.a (mon.0) 5691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:28 smithi115 bash[46934]: audit 2022-04-22T18:43:27.497556+0000 mon.a (mon.0) 5691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:29 smithi121 bash[36902]: cluster 2022-04-22T18:43:28.077404+0000 mgr.y (mgr.34107) 2684 : cluster [DBG] pgmap v4264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:29 smithi121 bash[36902]: audit 2022-04-22T18:43:28.673968+0000 mon.a (mon.0) 5692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:29 smithi115 bash[45495]: cluster 2022-04-22T18:43:28.077404+0000 mgr.y (mgr.34107) 2684 : cluster [DBG] pgmap v4264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:29 smithi115 bash[45495]: audit 2022-04-22T18:43:28.673968+0000 mon.a (mon.0) 5692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:29 smithi115 bash[46934]: cluster 2022-04-22T18:43:28.077404+0000 mgr.y (mgr.34107) 2684 : cluster [DBG] pgmap v4264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:29 smithi115 bash[46934]: audit 2022-04-22T18:43:28.673968+0000 mon.a (mon.0) 5692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:31 smithi121 bash[36902]: cluster 2022-04-22T18:43:30.078379+0000 mgr.y (mgr.34107) 2685 : cluster [DBG] pgmap v4265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:31 smithi115 bash[45495]: cluster 2022-04-22T18:43:30.078379+0000 mgr.y (mgr.34107) 2685 : cluster [DBG] pgmap v4265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:31 smithi115 bash[46934]: cluster 2022-04-22T18:43:30.078379+0000 mgr.y (mgr.34107) 2685 : cluster [DBG] pgmap v4265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:32.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:31.899Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:32 smithi121 bash[36902]: audit 2022-04-22T18:43:31.857519+0000 mon.a (mon.0) 5693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:32 smithi115 bash[45495]: audit 2022-04-22T18:43:31.857519+0000 mon.a (mon.0) 5693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:32 smithi115 bash[46934]: audit 2022-04-22T18:43:31.857519+0000 mon.a (mon.0) 5693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:33 smithi121 bash[36902]: cluster 2022-04-22T18:43:32.079517+0000 mgr.y (mgr.34107) 2686 : cluster [DBG] pgmap v4266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:33 smithi121 bash[36902]: audit 2022-04-22T18:43:32.501193+0000 mon.a (mon.0) 5694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:33 smithi115 bash[46934]: cluster 2022-04-22T18:43:32.079517+0000 mgr.y (mgr.34107) 2686 : cluster [DBG] pgmap v4266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:33 smithi115 bash[46934]: audit 2022-04-22T18:43:32.501193+0000 mon.a (mon.0) 5694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:33.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:33 smithi115 bash[45495]: cluster 2022-04-22T18:43:32.079517+0000 mgr.y (mgr.34107) 2686 : cluster [DBG] pgmap v4266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:33.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:33 smithi115 bash[45495]: audit 2022-04-22T18:43:32.501193+0000 mon.a (mon.0) 5694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:33.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:34 smithi121 bash[36902]: audit 2022-04-22T18:43:33.676995+0000 mon.a (mon.0) 5695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:34.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:34 smithi115 bash[45495]: audit 2022-04-22T18:43:33.676995+0000 mon.a (mon.0) 5695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:34.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:34 smithi115 bash[46934]: audit 2022-04-22T18:43:33.676995+0000 mon.a (mon.0) 5695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:35 smithi121 bash[36902]: cluster 2022-04-22T18:43:34.080563+0000 mgr.y (mgr.34107) 2687 : cluster [DBG] pgmap v4267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:35 smithi115 bash[45495]: cluster 2022-04-22T18:43:34.080563+0000 mgr.y (mgr.34107) 2687 : cluster [DBG] pgmap v4267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:35 smithi115 bash[46934]: cluster 2022-04-22T18:43:34.080563+0000 mgr.y (mgr.34107) 2687 : cluster [DBG] pgmap v4267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:36 smithi121 bash[36902]: audit 2022-04-22T18:43:36.051113+0000 mon.a (mon.0) 5696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:43:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:36 smithi121 bash[36902]: audit 2022-04-22T18:43:36.054168+0000 mon.a (mon.0) 5697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:43:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:36 smithi121 bash[36902]: audit 2022-04-22T18:43:36.055948+0000 mon.a (mon.0) 5698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:43:36.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:36 smithi121 bash[36902]: audit 2022-04-22T18:43:36.406363+0000 mon.a (mon.0) 5699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:43:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[45495]: audit 2022-04-22T18:43:36.051113+0000 mon.a (mon.0) 5696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[45495]: audit 2022-04-22T18:43:36.054168+0000 mon.a (mon.0) 5697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[45495]: audit 2022-04-22T18:43:36.055948+0000 mon.a (mon.0) 5698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[45495]: audit 2022-04-22T18:43:36.406363+0000 mon.a (mon.0) 5699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[46934]: audit 2022-04-22T18:43:36.051113+0000 mon.a (mon.0) 5696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[46934]: audit 2022-04-22T18:43:36.054168+0000 mon.a (mon.0) 5697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[46934]: audit 2022-04-22T18:43:36.055948+0000 mon.a (mon.0) 5698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:43:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:36 smithi115 bash[46934]: audit 2022-04-22T18:43:36.406363+0000 mon.a (mon.0) 5699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:43:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:37 smithi121 bash[36902]: cluster 2022-04-22T18:43:36.081471+0000 mgr.y (mgr.34107) 2688 : cluster [DBG] pgmap v4268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:37 smithi121 bash[36902]: audit 2022-04-22T18:43:36.860543+0000 mon.a (mon.0) 5700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:37 smithi115 bash[45495]: cluster 2022-04-22T18:43:36.081471+0000 mgr.y (mgr.34107) 2688 : cluster [DBG] pgmap v4268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:37 smithi115 bash[45495]: audit 2022-04-22T18:43:36.860543+0000 mon.a (mon.0) 5700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:37 smithi115 bash[46934]: cluster 2022-04-22T18:43:36.081471+0000 mgr.y (mgr.34107) 2688 : cluster [DBG] pgmap v4268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:37 smithi115 bash[46934]: audit 2022-04-22T18:43:36.860543+0000 mon.a (mon.0) 5700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:38.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:38 smithi121 bash[36902]: audit 2022-04-22T18:43:37.504940+0000 mon.a (mon.0) 5701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:38 smithi115 bash[45495]: audit 2022-04-22T18:43:37.504940+0000 mon.a (mon.0) 5701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:38 smithi115 bash[46934]: audit 2022-04-22T18:43:37.504940+0000 mon.a (mon.0) 5701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:39 smithi121 bash[36902]: cluster 2022-04-22T18:43:38.082582+0000 mgr.y (mgr.34107) 2689 : cluster [DBG] pgmap v4269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:39 smithi121 bash[36902]: audit 2022-04-22T18:43:38.680287+0000 mon.a (mon.0) 5702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:39 smithi115 bash[45495]: cluster 2022-04-22T18:43:38.082582+0000 mgr.y (mgr.34107) 2689 : cluster [DBG] pgmap v4269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:39 smithi115 bash[45495]: audit 2022-04-22T18:43:38.680287+0000 mon.a (mon.0) 5702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:39 smithi115 bash[46934]: cluster 2022-04-22T18:43:38.082582+0000 mgr.y (mgr.34107) 2689 : cluster [DBG] pgmap v4269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:39 smithi115 bash[46934]: audit 2022-04-22T18:43:38.680287+0000 mon.a (mon.0) 5702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:41 smithi121 bash[36902]: cluster 2022-04-22T18:43:40.083614+0000 mgr.y (mgr.34107) 2690 : cluster [DBG] pgmap v4270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:41 smithi115 bash[45495]: cluster 2022-04-22T18:43:40.083614+0000 mgr.y (mgr.34107) 2690 : cluster [DBG] pgmap v4270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:41 smithi115 bash[46934]: cluster 2022-04-22T18:43:40.083614+0000 mgr.y (mgr.34107) 2690 : cluster [DBG] pgmap v4270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:42.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:41.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:42 smithi121 bash[36902]: audit 2022-04-22T18:43:41.863877+0000 mon.a (mon.0) 5703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:42.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:42 smithi115 bash[45495]: audit 2022-04-22T18:43:41.863877+0000 mon.a (mon.0) 5703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:42 smithi115 bash[46934]: audit 2022-04-22T18:43:41.863877+0000 mon.a (mon.0) 5703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:43 smithi121 bash[36902]: cluster 2022-04-22T18:43:42.084705+0000 mgr.y (mgr.34107) 2691 : cluster [DBG] pgmap v4271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:43 smithi121 bash[36902]: audit 2022-04-22T18:43:42.508299+0000 mon.a (mon.0) 5704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:43.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:43 smithi115 bash[45495]: cluster 2022-04-22T18:43:42.084705+0000 mgr.y (mgr.34107) 2691 : cluster [DBG] pgmap v4271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:43.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:43 smithi115 bash[45495]: audit 2022-04-22T18:43:42.508299+0000 mon.a (mon.0) 5704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:43.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:43 smithi115 bash[46934]: cluster 2022-04-22T18:43:42.084705+0000 mgr.y (mgr.34107) 2691 : cluster [DBG] pgmap v4271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:43.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:43 smithi115 bash[46934]: audit 2022-04-22T18:43:42.508299+0000 mon.a (mon.0) 5704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:43.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:44.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:44 smithi115 bash[45495]: audit 2022-04-22T18:43:43.683454+0000 mon.a (mon.0) 5705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:44 smithi115 bash[46934]: audit 2022-04-22T18:43:43.683454+0000 mon.a (mon.0) 5705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:44 smithi121 bash[36902]: audit 2022-04-22T18:43:43.683454+0000 mon.a (mon.0) 5705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:45.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:45 smithi115 bash[45495]: cluster 2022-04-22T18:43:44.085794+0000 mgr.y (mgr.34107) 2692 : cluster [DBG] pgmap v4272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:45 smithi115 bash[46934]: cluster 2022-04-22T18:43:44.085794+0000 mgr.y (mgr.34107) 2692 : cluster [DBG] pgmap v4272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:45 smithi121 bash[36902]: cluster 2022-04-22T18:43:44.085794+0000 mgr.y (mgr.34107) 2692 : cluster [DBG] pgmap v4272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:47.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[45495]: cluster 2022-04-22T18:43:46.086910+0000 mgr.y (mgr.34107) 2693 : cluster [DBG] pgmap v4273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[45495]: audit 2022-04-22T18:43:46.866855+0000 mon.a (mon.0) 5706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[45495]: audit 2022-04-22T18:43:47.511653+0000 mon.a (mon.0) 5707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[46934]: cluster 2022-04-22T18:43:46.086910+0000 mgr.y (mgr.34107) 2693 : cluster [DBG] pgmap v4273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[46934]: audit 2022-04-22T18:43:46.866855+0000 mon.a (mon.0) 5706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:47 smithi115 bash[46934]: audit 2022-04-22T18:43:47.511653+0000 mon.a (mon.0) 5707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:47 smithi121 bash[36902]: cluster 2022-04-22T18:43:46.086910+0000 mgr.y (mgr.34107) 2693 : cluster [DBG] pgmap v4273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:47 smithi121 bash[36902]: audit 2022-04-22T18:43:46.866855+0000 mon.a (mon.0) 5706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:47 smithi121 bash[36902]: audit 2022-04-22T18:43:47.511653+0000 mon.a (mon.0) 5707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:49.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:49 smithi115 bash[45495]: cluster 2022-04-22T18:43:48.088139+0000 mgr.y (mgr.34107) 2694 : cluster [DBG] pgmap v4274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:49 smithi115 bash[45495]: audit 2022-04-22T18:43:48.686521+0000 mon.a (mon.0) 5708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:49 smithi115 bash[46934]: cluster 2022-04-22T18:43:48.088139+0000 mgr.y (mgr.34107) 2694 : cluster [DBG] pgmap v4274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:49 smithi115 bash[46934]: audit 2022-04-22T18:43:48.686521+0000 mon.a (mon.0) 5708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:49 smithi121 bash[36902]: cluster 2022-04-22T18:43:48.088139+0000 mgr.y (mgr.34107) 2694 : cluster [DBG] pgmap v4274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:49 smithi121 bash[36902]: audit 2022-04-22T18:43:48.686521+0000 mon.a (mon.0) 5708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:51 smithi115 bash[45495]: cluster 2022-04-22T18:43:50.089187+0000 mgr.y (mgr.34107) 2695 : cluster [DBG] pgmap v4275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:51 smithi115 bash[46934]: cluster 2022-04-22T18:43:50.089187+0000 mgr.y (mgr.34107) 2695 : cluster [DBG] pgmap v4275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:51 smithi121 bash[36902]: cluster 2022-04-22T18:43:50.089187+0000 mgr.y (mgr.34107) 2695 : cluster [DBG] pgmap v4275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:51.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:52.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:52 smithi115 bash[46934]: audit 2022-04-22T18:43:51.870059+0000 mon.a (mon.0) 5709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:52.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:52 smithi115 bash[46934]: audit 2022-04-22T18:43:52.515511+0000 mon.a (mon.0) 5710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:52.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:52 smithi115 bash[45495]: audit 2022-04-22T18:43:51.870059+0000 mon.a (mon.0) 5709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:52.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:52 smithi115 bash[45495]: audit 2022-04-22T18:43:52.515511+0000 mon.a (mon.0) 5710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:52 smithi121 bash[36902]: audit 2022-04-22T18:43:51.870059+0000 mon.a (mon.0) 5709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:52 smithi121 bash[36902]: audit 2022-04-22T18:43:52.515511+0000 mon.a (mon.0) 5710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:43:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:43:53.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:43:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:53 smithi115 bash[45495]: cluster 2022-04-22T18:43:52.090368+0000 mgr.y (mgr.34107) 2696 : cluster [DBG] pgmap v4276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:53 smithi115 bash[45495]: audit 2022-04-22T18:43:53.689675+0000 mon.a (mon.0) 5711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:53 smithi115 bash[46934]: cluster 2022-04-22T18:43:52.090368+0000 mgr.y (mgr.34107) 2696 : cluster [DBG] pgmap v4276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:53 smithi115 bash[46934]: audit 2022-04-22T18:43:53.689675+0000 mon.a (mon.0) 5711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:53 smithi121 bash[36902]: cluster 2022-04-22T18:43:52.090368+0000 mgr.y (mgr.34107) 2696 : cluster [DBG] pgmap v4276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:54.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:53 smithi121 bash[36902]: audit 2022-04-22T18:43:53.689675+0000 mon.a (mon.0) 5711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:55 smithi121 bash[36902]: cluster 2022-04-22T18:43:54.091466+0000 mgr.y (mgr.34107) 2697 : cluster [DBG] pgmap v4277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:56.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:55 smithi115 bash[45495]: cluster 2022-04-22T18:43:54.091466+0000 mgr.y (mgr.34107) 2697 : cluster [DBG] pgmap v4277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:55 smithi115 bash[46934]: cluster 2022-04-22T18:43:54.091466+0000 mgr.y (mgr.34107) 2697 : cluster [DBG] pgmap v4277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:56.734 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:43:57.084 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:43:57.084 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (67m) 4m ago 80m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:43:57.084 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (80m) 5m ago 80m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (68m) 4m ago 81m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (68m) 5m ago 81m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (76m) 5m ago 86m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (76m) 4m ago 88m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (75m) 4m ago 88m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (74m) 5m ago 86m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (75m) 4m ago 86m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:43:57.085 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (68m) 4m ago 81m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (68m) 5m ago 81m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (73m) 4m ago 85m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (73m) 4m ago 85m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (72m) 4m ago 84m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (72m) 4m ago 84m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (71m) 5m ago 83m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (70m) 5m ago 83m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (70m) 5m ago 82m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:43:57.086 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (69m) 5m ago 82m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:43:57.087 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (67m) 5m ago 81m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:43:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:56 smithi121 bash[36902]: audit 2022-04-22T18:43:56.873283+0000 mon.a (mon.0) 5712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:57.377 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:56 smithi115 bash[45495]: audit 2022-04-22T18:43:56.873283+0000 mon.a (mon.0) 5712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:57.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:56 smithi115 bash[46934]: audit 2022-04-22T18:43:56.873283+0000 mon.a (mon.0) 5712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:43:57.484 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:43:57.484 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:43:57.484 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:43:57.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:43:57.485 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:43:57.486 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:43:57.486 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:43:57.486 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:43:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:57 smithi121 bash[36902]: cluster 2022-04-22T18:43:56.092524+0000 mgr.y (mgr.34107) 2698 : cluster [DBG] pgmap v4278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:57 smithi121 bash[36902]: audit 2022-04-22T18:43:56.710123+0000 mgr.y (mgr.34107) 2699 : audit [DBG] from='client.48184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:57 smithi121 bash[36902]: audit 2022-04-22T18:43:57.475453+0000 mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.115:0/1343227400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:57 smithi121 bash[36902]: audit 2022-04-22T18:43:57.519152+0000 mon.a (mon.0) 5713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[45495]: cluster 2022-04-22T18:43:56.092524+0000 mgr.y (mgr.34107) 2698 : cluster [DBG] pgmap v4278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[45495]: audit 2022-04-22T18:43:56.710123+0000 mgr.y (mgr.34107) 2699 : audit [DBG] from='client.48184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[45495]: audit 2022-04-22T18:43:57.475453+0000 mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.115:0/1343227400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[45495]: audit 2022-04-22T18:43:57.519152+0000 mon.a (mon.0) 5713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[46934]: cluster 2022-04-22T18:43:56.092524+0000 mgr.y (mgr.34107) 2698 : cluster [DBG] pgmap v4278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:43:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[46934]: audit 2022-04-22T18:43:56.710123+0000 mgr.y (mgr.34107) 2699 : audit [DBG] from='client.48184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[46934]: audit 2022-04-22T18:43:57.475453+0000 mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.115:0/1343227400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:43:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:57 smithi115 bash[46934]: audit 2022-04-22T18:43:57.519152+0000 mon.a (mon.0) 5713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:43:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:58 smithi121 bash[36902]: audit 2022-04-22T18:43:57.061275+0000 mgr.y (mgr.34107) 2700 : audit [DBG] from='client.38256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:58 smithi121 bash[36902]: audit 2022-04-22T18:43:58.692725+0000 mon.a (mon.0) 5714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:58 smithi115 bash[45495]: audit 2022-04-22T18:43:57.061275+0000 mgr.y (mgr.34107) 2700 : audit [DBG] from='client.38256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:58 smithi115 bash[45495]: audit 2022-04-22T18:43:58.692725+0000 mon.a (mon.0) 5714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:43:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:58 smithi115 bash[46934]: audit 2022-04-22T18:43:57.061275+0000 mgr.y (mgr.34107) 2700 : audit [DBG] from='client.38256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:43:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:58 smithi115 bash[46934]: audit 2022-04-22T18:43:58.692725+0000 mon.a (mon.0) 5714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:43:59 smithi121 bash[36902]: cluster 2022-04-22T18:43:58.093635+0000 mgr.y (mgr.34107) 2701 : cluster [DBG] pgmap v4279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:00.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:43:59 smithi115 bash[45495]: cluster 2022-04-22T18:43:58.093635+0000 mgr.y (mgr.34107) 2701 : cluster [DBG] pgmap v4279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:43:59 smithi115 bash[46934]: cluster 2022-04-22T18:43:58.093635+0000 mgr.y (mgr.34107) 2701 : cluster [DBG] pgmap v4279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:01 smithi121 bash[36902]: cluster 2022-04-22T18:44:00.094625+0000 mgr.y (mgr.34107) 2702 : cluster [DBG] pgmap v4280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:01 smithi121 bash[36902]: audit 2022-04-22T18:44:01.876433+0000 mon.a (mon.0) 5715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:01 smithi115 bash[46934]: cluster 2022-04-22T18:44:00.094625+0000 mgr.y (mgr.34107) 2702 : cluster [DBG] pgmap v4280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:01 smithi115 bash[46934]: audit 2022-04-22T18:44:01.876433+0000 mon.a (mon.0) 5715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:01.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:02.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:01 smithi115 bash[45495]: cluster 2022-04-22T18:44:00.094625+0000 mgr.y (mgr.34107) 2702 : cluster [DBG] pgmap v4280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:02.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:01 smithi115 bash[45495]: audit 2022-04-22T18:44:01.876433+0000 mon.a (mon.0) 5715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:02 smithi121 bash[36902]: audit 2022-04-22T18:44:02.522765+0000 mon.a (mon.0) 5716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:02 smithi115 bash[46934]: audit 2022-04-22T18:44:02.522765+0000 mon.a (mon.0) 5716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:02 smithi115 bash[45495]: audit 2022-04-22T18:44:02.522765+0000 mon.a (mon.0) 5716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:04.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:03.776Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:03 smithi115 bash[45495]: cluster 2022-04-22T18:44:02.095799+0000 mgr.y (mgr.34107) 2703 : cluster [DBG] pgmap v4281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:03 smithi115 bash[45495]: audit 2022-04-22T18:44:03.695761+0000 mon.a (mon.0) 5717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:03 smithi115 bash[46934]: cluster 2022-04-22T18:44:02.095799+0000 mgr.y (mgr.34107) 2703 : cluster [DBG] pgmap v4281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:03 smithi115 bash[46934]: audit 2022-04-22T18:44:03.695761+0000 mon.a (mon.0) 5717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:03 smithi121 bash[36902]: cluster 2022-04-22T18:44:02.095799+0000 mgr.y (mgr.34107) 2703 : cluster [DBG] pgmap v4281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:03 smithi121 bash[36902]: audit 2022-04-22T18:44:03.695761+0000 mon.a (mon.0) 5717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:04 smithi121 bash[36902]: cluster 2022-04-22T18:44:04.096814+0000 mgr.y (mgr.34107) 2704 : cluster [DBG] pgmap v4282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:04 smithi115 bash[45495]: cluster 2022-04-22T18:44:04.096814+0000 mgr.y (mgr.34107) 2704 : cluster [DBG] pgmap v4282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:04 smithi115 bash[46934]: cluster 2022-04-22T18:44:04.096814+0000 mgr.y (mgr.34107) 2704 : cluster [DBG] pgmap v4282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:07 smithi121 bash[36902]: cluster 2022-04-22T18:44:06.097946+0000 mgr.y (mgr.34107) 2705 : cluster [DBG] pgmap v4283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:07 smithi121 bash[36902]: audit 2022-04-22T18:44:06.879715+0000 mon.a (mon.0) 5718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:07 smithi115 bash[45495]: cluster 2022-04-22T18:44:06.097946+0000 mgr.y (mgr.34107) 2705 : cluster [DBG] pgmap v4283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:07 smithi115 bash[45495]: audit 2022-04-22T18:44:06.879715+0000 mon.a (mon.0) 5718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:07 smithi115 bash[46934]: cluster 2022-04-22T18:44:06.097946+0000 mgr.y (mgr.34107) 2705 : cluster [DBG] pgmap v4283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:07 smithi115 bash[46934]: audit 2022-04-22T18:44:06.879715+0000 mon.a (mon.0) 5718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:08 smithi121 bash[36902]: audit 2022-04-22T18:44:07.526325+0000 mon.a (mon.0) 5719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:08 smithi115 bash[45495]: audit 2022-04-22T18:44:07.526325+0000 mon.a (mon.0) 5719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:08 smithi115 bash[46934]: audit 2022-04-22T18:44:07.526325+0000 mon.a (mon.0) 5719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:09 smithi121 bash[36902]: cluster 2022-04-22T18:44:08.099066+0000 mgr.y (mgr.34107) 2706 : cluster [DBG] pgmap v4284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:09 smithi121 bash[36902]: audit 2022-04-22T18:44:08.698893+0000 mon.a (mon.0) 5720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:09 smithi115 bash[45495]: cluster 2022-04-22T18:44:08.099066+0000 mgr.y (mgr.34107) 2706 : cluster [DBG] pgmap v4284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:09 smithi115 bash[45495]: audit 2022-04-22T18:44:08.698893+0000 mon.a (mon.0) 5720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:09 smithi115 bash[46934]: cluster 2022-04-22T18:44:08.099066+0000 mgr.y (mgr.34107) 2706 : cluster [DBG] pgmap v4284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:09 smithi115 bash[46934]: audit 2022-04-22T18:44:08.698893+0000 mon.a (mon.0) 5720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:11 smithi121 bash[36902]: cluster 2022-04-22T18:44:10.100017+0000 mgr.y (mgr.34107) 2707 : cluster [DBG] pgmap v4285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:11 smithi115 bash[45495]: cluster 2022-04-22T18:44:10.100017+0000 mgr.y (mgr.34107) 2707 : cluster [DBG] pgmap v4285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:11 smithi115 bash[46934]: cluster 2022-04-22T18:44:10.100017+0000 mgr.y (mgr.34107) 2707 : cluster [DBG] pgmap v4285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:12.182 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:11.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:12 smithi121 bash[36902]: audit 2022-04-22T18:44:11.882714+0000 mon.a (mon.0) 5721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:12 smithi115 bash[45495]: audit 2022-04-22T18:44:11.882714+0000 mon.a (mon.0) 5721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:12.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:12 smithi115 bash[46934]: audit 2022-04-22T18:44:11.882714+0000 mon.a (mon.0) 5721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:13 smithi121 bash[36902]: cluster 2022-04-22T18:44:12.101070+0000 mgr.y (mgr.34107) 2708 : cluster [DBG] pgmap v4286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:13 smithi121 bash[36902]: audit 2022-04-22T18:44:12.530124+0000 mon.a (mon.0) 5722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:13 smithi115 bash[45495]: cluster 2022-04-22T18:44:12.101070+0000 mgr.y (mgr.34107) 2708 : cluster [DBG] pgmap v4286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:13 smithi115 bash[45495]: audit 2022-04-22T18:44:12.530124+0000 mon.a (mon.0) 5722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:13 smithi115 bash[46934]: cluster 2022-04-22T18:44:12.101070+0000 mgr.y (mgr.34107) 2708 : cluster [DBG] pgmap v4286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:13 smithi115 bash[46934]: audit 2022-04-22T18:44:12.530124+0000 mon.a (mon.0) 5722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:14.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:13.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:14 smithi121 bash[36902]: audit 2022-04-22T18:44:13.702225+0000 mon.a (mon.0) 5723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:14 smithi121 bash[36902]: audit 2022-04-22T18:44:14.114593+0000 mon.a (mon.0) 5724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:44:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:14 smithi121 bash[36902]: audit 2022-04-22T18:44:14.115080+0000 mon.a (mon.0) 5725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:44:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[45495]: audit 2022-04-22T18:44:13.702225+0000 mon.a (mon.0) 5723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[45495]: audit 2022-04-22T18:44:14.114593+0000 mon.a (mon.0) 5724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:44:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[45495]: audit 2022-04-22T18:44:14.115080+0000 mon.a (mon.0) 5725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:44:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[46934]: audit 2022-04-22T18:44:13.702225+0000 mon.a (mon.0) 5723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[46934]: audit 2022-04-22T18:44:14.114593+0000 mon.a (mon.0) 5724 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:44:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:14 smithi115 bash[46934]: audit 2022-04-22T18:44:14.115080+0000 mon.a (mon.0) 5725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:44:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:15 smithi121 bash[36902]: cluster 2022-04-22T18:44:14.102299+0000 mgr.y (mgr.34107) 2709 : cluster [DBG] pgmap v4287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:15 smithi115 bash[45495]: cluster 2022-04-22T18:44:14.102299+0000 mgr.y (mgr.34107) 2709 : cluster [DBG] pgmap v4287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:15 smithi115 bash[46934]: cluster 2022-04-22T18:44:14.102299+0000 mgr.y (mgr.34107) 2709 : cluster [DBG] pgmap v4287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:17 smithi121 bash[36902]: cluster 2022-04-22T18:44:16.103510+0000 mgr.y (mgr.34107) 2710 : cluster [DBG] pgmap v4288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:17 smithi121 bash[36902]: audit 2022-04-22T18:44:16.885576+0000 mon.a (mon.0) 5726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:17 smithi115 bash[45495]: cluster 2022-04-22T18:44:16.103510+0000 mgr.y (mgr.34107) 2710 : cluster [DBG] pgmap v4288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:17 smithi115 bash[45495]: audit 2022-04-22T18:44:16.885576+0000 mon.a (mon.0) 5726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:17 smithi115 bash[46934]: cluster 2022-04-22T18:44:16.103510+0000 mgr.y (mgr.34107) 2710 : cluster [DBG] pgmap v4288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:17 smithi115 bash[46934]: audit 2022-04-22T18:44:16.885576+0000 mon.a (mon.0) 5726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:18 smithi121 bash[36902]: audit 2022-04-22T18:44:17.533712+0000 mon.a (mon.0) 5727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:18 smithi115 bash[45495]: audit 2022-04-22T18:44:17.533712+0000 mon.a (mon.0) 5727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:18 smithi115 bash[46934]: audit 2022-04-22T18:44:17.533712+0000 mon.a (mon.0) 5727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:19 smithi121 bash[36902]: cluster 2022-04-22T18:44:18.104609+0000 mgr.y (mgr.34107) 2711 : cluster [DBG] pgmap v4289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:19 smithi121 bash[36902]: audit 2022-04-22T18:44:18.705572+0000 mon.a (mon.0) 5728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:19 smithi115 bash[45495]: cluster 2022-04-22T18:44:18.104609+0000 mgr.y (mgr.34107) 2711 : cluster [DBG] pgmap v4289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:19 smithi115 bash[45495]: audit 2022-04-22T18:44:18.705572+0000 mon.a (mon.0) 5728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:19 smithi115 bash[46934]: cluster 2022-04-22T18:44:18.104609+0000 mgr.y (mgr.34107) 2711 : cluster [DBG] pgmap v4289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:19 smithi115 bash[46934]: audit 2022-04-22T18:44:18.705572+0000 mon.a (mon.0) 5728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:21.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:21 smithi121 bash[36902]: cluster 2022-04-22T18:44:20.105381+0000 mgr.y (mgr.34107) 2712 : cluster [DBG] pgmap v4290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:21 smithi115 bash[45495]: cluster 2022-04-22T18:44:20.105381+0000 mgr.y (mgr.34107) 2712 : cluster [DBG] pgmap v4290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:21 smithi115 bash[46934]: cluster 2022-04-22T18:44:20.105381+0000 mgr.y (mgr.34107) 2712 : cluster [DBG] pgmap v4290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:22.218 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:21.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:22 smithi121 bash[36902]: audit 2022-04-22T18:44:21.888705+0000 mon.a (mon.0) 5729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:22.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:22 smithi115 bash[46934]: audit 2022-04-22T18:44:21.888705+0000 mon.a (mon.0) 5729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:22.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:22 smithi115 bash[45495]: audit 2022-04-22T18:44:21.888705+0000 mon.a (mon.0) 5729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:23 smithi121 bash[36902]: cluster 2022-04-22T18:44:22.106498+0000 mgr.y (mgr.34107) 2713 : cluster [DBG] pgmap v4291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:23 smithi121 bash[36902]: audit 2022-04-22T18:44:22.537466+0000 mon.a (mon.0) 5730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:23 smithi115 bash[45495]: cluster 2022-04-22T18:44:22.106498+0000 mgr.y (mgr.34107) 2713 : cluster [DBG] pgmap v4291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:23 smithi115 bash[45495]: audit 2022-04-22T18:44:22.537466+0000 mon.a (mon.0) 5730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:23 smithi115 bash[46934]: cluster 2022-04-22T18:44:22.106498+0000 mgr.y (mgr.34107) 2713 : cluster [DBG] pgmap v4291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:23 smithi115 bash[46934]: audit 2022-04-22T18:44:22.537466+0000 mon.a (mon.0) 5730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:23.779Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:24 smithi121 bash[36902]: audit 2022-04-22T18:44:23.708615+0000 mon.a (mon.0) 5731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:24 smithi115 bash[45495]: audit 2022-04-22T18:44:23.708615+0000 mon.a (mon.0) 5731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:24 smithi115 bash[46934]: audit 2022-04-22T18:44:23.708615+0000 mon.a (mon.0) 5731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:25 smithi121 bash[36902]: cluster 2022-04-22T18:44:24.107636+0000 mgr.y (mgr.34107) 2714 : cluster [DBG] pgmap v4292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:25 smithi115 bash[45495]: cluster 2022-04-22T18:44:24.107636+0000 mgr.y (mgr.34107) 2714 : cluster [DBG] pgmap v4292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:25 smithi115 bash[46934]: cluster 2022-04-22T18:44:24.107636+0000 mgr.y (mgr.34107) 2714 : cluster [DBG] pgmap v4292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:27 smithi121 bash[36902]: cluster 2022-04-22T18:44:26.108626+0000 mgr.y (mgr.34107) 2715 : cluster [DBG] pgmap v4293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:27.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:27 smithi121 bash[36902]: audit 2022-04-22T18:44:26.891639+0000 mon.a (mon.0) 5732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:27 smithi115 bash[46934]: cluster 2022-04-22T18:44:26.108626+0000 mgr.y (mgr.34107) 2715 : cluster [DBG] pgmap v4293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:27 smithi115 bash[46934]: audit 2022-04-22T18:44:26.891639+0000 mon.a (mon.0) 5732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:27 smithi115 bash[45495]: cluster 2022-04-22T18:44:26.108626+0000 mgr.y (mgr.34107) 2715 : cluster [DBG] pgmap v4293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:27 smithi115 bash[45495]: audit 2022-04-22T18:44:26.891639+0000 mon.a (mon.0) 5732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:27.861 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:44:28.206 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:44:28.206 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (67m) 5m ago 81m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (81m) 6m ago 81m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (69m) 5m ago 82m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (69m) 6m ago 82m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (76m) 6m ago 86m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (77m) 5m ago 88m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (76m) 5m ago 88m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (75m) 6m ago 86m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:44:28.207 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (75m) 5m ago 87m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (68m) 5m ago 81m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (68m) 6m ago 81m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (74m) 5m ago 86m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (73m) 5m ago 85m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (73m) 5m ago 85m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (72m) 5m ago 84m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (71m) 6m ago 84m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (71m) 6m ago 83m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (70m) 6m ago 83m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:44:28.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (69m) 6m ago 82m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:44:28.209 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (68m) 6m ago 81m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:44:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:28 smithi121 bash[36902]: audit 2022-04-22T18:44:27.541340+0000 mon.a (mon.0) 5733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:28.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:28 smithi121 bash[36902]: audit 2022-04-22T18:44:27.838136+0000 mgr.y (mgr.34107) 2716 : audit [DBG] from='client.38268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:28 smithi115 bash[46934]: audit 2022-04-22T18:44:27.541340+0000 mon.a (mon.0) 5733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:28 smithi115 bash[46934]: audit 2022-04-22T18:44:27.838136+0000 mgr.y (mgr.34107) 2716 : audit [DBG] from='client.38268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:28.621 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:28 smithi115 bash[45495]: audit 2022-04-22T18:44:27.541340+0000 mon.a (mon.0) 5733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:28.621 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:28 smithi115 bash[45495]: audit 2022-04-22T18:44:27.838136+0000 mgr.y (mgr.34107) 2716 : audit [DBG] from='client.38268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:44:28.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:44:28.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:44:28.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:28.641 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:44:28.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:44:28.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:28.641 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:44:28.642 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:44:28.643 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:44:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:29 smithi121 bash[36902]: cluster 2022-04-22T18:44:28.109370+0000 mgr.y (mgr.34107) 2717 : cluster [DBG] pgmap v4294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:29 smithi121 bash[36902]: audit 2022-04-22T18:44:28.180766+0000 mgr.y (mgr.34107) 2718 : audit [DBG] from='client.48208 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:29 smithi121 bash[36902]: audit 2022-04-22T18:44:28.606276+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.115:0/4227923817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:44:29.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:29 smithi121 bash[36902]: audit 2022-04-22T18:44:28.711954+0000 mon.a (mon.0) 5734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[45495]: cluster 2022-04-22T18:44:28.109370+0000 mgr.y (mgr.34107) 2717 : cluster [DBG] pgmap v4294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[45495]: audit 2022-04-22T18:44:28.180766+0000 mgr.y (mgr.34107) 2718 : audit [DBG] from='client.48208 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[45495]: audit 2022-04-22T18:44:28.606276+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.115:0/4227923817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[45495]: audit 2022-04-22T18:44:28.711954+0000 mon.a (mon.0) 5734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[46934]: cluster 2022-04-22T18:44:28.109370+0000 mgr.y (mgr.34107) 2717 : cluster [DBG] pgmap v4294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[46934]: audit 2022-04-22T18:44:28.180766+0000 mgr.y (mgr.34107) 2718 : audit [DBG] from='client.48208 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[46934]: audit 2022-04-22T18:44:28.606276+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.115:0/4227923817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:29 smithi115 bash[46934]: audit 2022-04-22T18:44:28.711954+0000 mon.a (mon.0) 5734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:31 smithi121 bash[36902]: cluster 2022-04-22T18:44:30.110298+0000 mgr.y (mgr.34107) 2719 : cluster [DBG] pgmap v4295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:31 smithi115 bash[45495]: cluster 2022-04-22T18:44:30.110298+0000 mgr.y (mgr.34107) 2719 : cluster [DBG] pgmap v4295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:31 smithi115 bash[46934]: cluster 2022-04-22T18:44:30.110298+0000 mgr.y (mgr.34107) 2719 : cluster [DBG] pgmap v4295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:32.257 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:31.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:32 smithi121 bash[36902]: audit 2022-04-22T18:44:31.894713+0000 mon.a (mon.0) 5735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:32.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:32 smithi115 bash[45495]: audit 2022-04-22T18:44:31.894713+0000 mon.a (mon.0) 5735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:32 smithi115 bash[46934]: audit 2022-04-22T18:44:31.894713+0000 mon.a (mon.0) 5735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:33 smithi121 bash[36902]: cluster 2022-04-22T18:44:32.111463+0000 mgr.y (mgr.34107) 2720 : cluster [DBG] pgmap v4296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:33.529 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:33 smithi121 bash[36902]: audit 2022-04-22T18:44:32.545176+0000 mon.a (mon.0) 5736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:33 smithi115 bash[45495]: cluster 2022-04-22T18:44:32.111463+0000 mgr.y (mgr.34107) 2720 : cluster [DBG] pgmap v4296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:33 smithi115 bash[45495]: audit 2022-04-22T18:44:32.545176+0000 mon.a (mon.0) 5736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:33 smithi115 bash[46934]: cluster 2022-04-22T18:44:32.111463+0000 mgr.y (mgr.34107) 2720 : cluster [DBG] pgmap v4296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:33 smithi115 bash[46934]: audit 2022-04-22T18:44:32.545176+0000 mon.a (mon.0) 5736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:34.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:33.779Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:34 smithi115 bash[45495]: audit 2022-04-22T18:44:33.715022+0000 mon.a (mon.0) 5737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:34 smithi115 bash[46934]: audit 2022-04-22T18:44:33.715022+0000 mon.a (mon.0) 5737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:34 smithi121 bash[36902]: audit 2022-04-22T18:44:33.715022+0000 mon.a (mon.0) 5737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:35 smithi115 bash[46934]: cluster 2022-04-22T18:44:34.112589+0000 mgr.y (mgr.34107) 2721 : cluster [DBG] pgmap v4297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:35 smithi115 bash[45495]: cluster 2022-04-22T18:44:34.112589+0000 mgr.y (mgr.34107) 2721 : cluster [DBG] pgmap v4297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:35 smithi121 bash[36902]: cluster 2022-04-22T18:44:34.112589+0000 mgr.y (mgr.34107) 2721 : cluster [DBG] pgmap v4297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: cluster 2022-04-22T18:44:36.113611+0000 mgr.y (mgr.34107) 2722 : cluster [DBG] pgmap v4298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: audit 2022-04-22T18:44:36.414895+0000 mon.a (mon.0) 5738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:44:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: audit 2022-04-22T18:44:36.417967+0000 mon.a (mon.0) 5739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:44:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: audit 2022-04-22T18:44:36.419610+0000 mon.a (mon.0) 5740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:44:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: audit 2022-04-22T18:44:36.794967+0000 mon.a (mon.0) 5741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:44:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[46934]: audit 2022-04-22T18:44:36.897840+0000 mon.a (mon.0) 5742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: cluster 2022-04-22T18:44:36.113611+0000 mgr.y (mgr.34107) 2722 : cluster [DBG] pgmap v4298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: audit 2022-04-22T18:44:36.414895+0000 mon.a (mon.0) 5738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: audit 2022-04-22T18:44:36.417967+0000 mon.a (mon.0) 5739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: audit 2022-04-22T18:44:36.419610+0000 mon.a (mon.0) 5740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: audit 2022-04-22T18:44:36.794967+0000 mon.a (mon.0) 5741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:44:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:37 smithi115 bash[45495]: audit 2022-04-22T18:44:36.897840+0000 mon.a (mon.0) 5742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: cluster 2022-04-22T18:44:36.113611+0000 mgr.y (mgr.34107) 2722 : cluster [DBG] pgmap v4298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: audit 2022-04-22T18:44:36.414895+0000 mon.a (mon.0) 5738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:44:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: audit 2022-04-22T18:44:36.417967+0000 mon.a (mon.0) 5739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:44:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: audit 2022-04-22T18:44:36.419610+0000 mon.a (mon.0) 5740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:44:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: audit 2022-04-22T18:44:36.794967+0000 mon.a (mon.0) 5741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:44:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:37 smithi121 bash[36902]: audit 2022-04-22T18:44:36.897840+0000 mon.a (mon.0) 5742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:38 smithi115 bash[45495]: audit 2022-04-22T18:44:37.548835+0000 mon.a (mon.0) 5743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:38 smithi115 bash[46934]: audit 2022-04-22T18:44:37.548835+0000 mon.a (mon.0) 5743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:38 smithi121 bash[36902]: audit 2022-04-22T18:44:37.548835+0000 mon.a (mon.0) 5743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:39.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:39 smithi115 bash[45495]: cluster 2022-04-22T18:44:38.114706+0000 mgr.y (mgr.34107) 2723 : cluster [DBG] pgmap v4299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:39.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:39 smithi115 bash[45495]: audit 2022-04-22T18:44:38.718077+0000 mon.a (mon.0) 5744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:39.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:39 smithi115 bash[46934]: cluster 2022-04-22T18:44:38.114706+0000 mgr.y (mgr.34107) 2723 : cluster [DBG] pgmap v4299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:39.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:39 smithi115 bash[46934]: audit 2022-04-22T18:44:38.718077+0000 mon.a (mon.0) 5744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:39 smithi121 bash[36902]: cluster 2022-04-22T18:44:38.114706+0000 mgr.y (mgr.34107) 2723 : cluster [DBG] pgmap v4299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:39 smithi121 bash[36902]: audit 2022-04-22T18:44:38.718077+0000 mon.a (mon.0) 5744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:41 smithi115 bash[45495]: cluster 2022-04-22T18:44:40.115925+0000 mgr.y (mgr.34107) 2724 : cluster [DBG] pgmap v4300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:41 smithi115 bash[46934]: cluster 2022-04-22T18:44:40.115925+0000 mgr.y (mgr.34107) 2724 : cluster [DBG] pgmap v4300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:41 smithi121 bash[36902]: cluster 2022-04-22T18:44:40.115925+0000 mgr.y (mgr.34107) 2724 : cluster [DBG] pgmap v4300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:42.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:41.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:42 smithi121 bash[36902]: audit 2022-04-22T18:44:41.900822+0000 mon.a (mon.0) 5745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:42 smithi115 bash[45495]: audit 2022-04-22T18:44:41.900822+0000 mon.a (mon.0) 5745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:42 smithi115 bash[46934]: audit 2022-04-22T18:44:41.900822+0000 mon.a (mon.0) 5745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:43 smithi121 bash[36902]: cluster 2022-04-22T18:44:42.117152+0000 mgr.y (mgr.34107) 2725 : cluster [DBG] pgmap v4301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:43 smithi121 bash[36902]: audit 2022-04-22T18:44:42.552607+0000 mon.a (mon.0) 5746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:43.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:43 smithi115 bash[45495]: cluster 2022-04-22T18:44:42.117152+0000 mgr.y (mgr.34107) 2725 : cluster [DBG] pgmap v4301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:43.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:43 smithi115 bash[45495]: audit 2022-04-22T18:44:42.552607+0000 mon.a (mon.0) 5746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:43.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:43 smithi115 bash[46934]: cluster 2022-04-22T18:44:42.117152+0000 mgr.y (mgr.34107) 2725 : cluster [DBG] pgmap v4301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:43.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:43 smithi115 bash[46934]: audit 2022-04-22T18:44:42.552607+0000 mon.a (mon.0) 5746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:43.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:44 smithi121 bash[36902]: audit 2022-04-22T18:44:43.721284+0000 mon.a (mon.0) 5747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:44.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:44 smithi115 bash[45495]: audit 2022-04-22T18:44:43.721284+0000 mon.a (mon.0) 5747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:44 smithi115 bash[46934]: audit 2022-04-22T18:44:43.721284+0000 mon.a (mon.0) 5747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:45 smithi121 bash[36902]: cluster 2022-04-22T18:44:44.118186+0000 mgr.y (mgr.34107) 2726 : cluster [DBG] pgmap v4302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:45.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:45 smithi115 bash[45495]: cluster 2022-04-22T18:44:44.118186+0000 mgr.y (mgr.34107) 2726 : cluster [DBG] pgmap v4302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:45 smithi115 bash[46934]: cluster 2022-04-22T18:44:44.118186+0000 mgr.y (mgr.34107) 2726 : cluster [DBG] pgmap v4302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:47 smithi121 bash[36902]: cluster 2022-04-22T18:44:46.119179+0000 mgr.y (mgr.34107) 2727 : cluster [DBG] pgmap v4303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:47 smithi121 bash[36902]: audit 2022-04-22T18:44:46.904023+0000 mon.a (mon.0) 5748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:47.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:47 smithi115 bash[45495]: cluster 2022-04-22T18:44:46.119179+0000 mgr.y (mgr.34107) 2727 : cluster [DBG] pgmap v4303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:47 smithi115 bash[45495]: audit 2022-04-22T18:44:46.904023+0000 mon.a (mon.0) 5748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:47 smithi115 bash[46934]: cluster 2022-04-22T18:44:46.119179+0000 mgr.y (mgr.34107) 2727 : cluster [DBG] pgmap v4303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:47 smithi115 bash[46934]: audit 2022-04-22T18:44:46.904023+0000 mon.a (mon.0) 5748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:48 smithi121 bash[36902]: audit 2022-04-22T18:44:47.556297+0000 mon.a (mon.0) 5749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:48 smithi115 bash[45495]: audit 2022-04-22T18:44:47.556297+0000 mon.a (mon.0) 5749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:48 smithi115 bash[46934]: audit 2022-04-22T18:44:47.556297+0000 mon.a (mon.0) 5749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:49 smithi121 bash[36902]: cluster 2022-04-22T18:44:48.120323+0000 mgr.y (mgr.34107) 2728 : cluster [DBG] pgmap v4304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:49.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:49 smithi121 bash[36902]: audit 2022-04-22T18:44:48.724323+0000 mon.a (mon.0) 5750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:49.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:49 smithi115 bash[45495]: cluster 2022-04-22T18:44:48.120323+0000 mgr.y (mgr.34107) 2728 : cluster [DBG] pgmap v4304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:49 smithi115 bash[45495]: audit 2022-04-22T18:44:48.724323+0000 mon.a (mon.0) 5750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:49 smithi115 bash[46934]: cluster 2022-04-22T18:44:48.120323+0000 mgr.y (mgr.34107) 2728 : cluster [DBG] pgmap v4304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:49 smithi115 bash[46934]: audit 2022-04-22T18:44:48.724323+0000 mon.a (mon.0) 5750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:51 smithi121 bash[36902]: cluster 2022-04-22T18:44:50.121330+0000 mgr.y (mgr.34107) 2729 : cluster [DBG] pgmap v4305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:51.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:51 smithi115 bash[45495]: cluster 2022-04-22T18:44:50.121330+0000 mgr.y (mgr.34107) 2729 : cluster [DBG] pgmap v4305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:51 smithi115 bash[46934]: cluster 2022-04-22T18:44:50.121330+0000 mgr.y (mgr.34107) 2729 : cluster [DBG] pgmap v4305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:52.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:51.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:52 smithi121 bash[36902]: audit 2022-04-22T18:44:51.906945+0000 mon.a (mon.0) 5751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:52.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:52 smithi115 bash[45495]: audit 2022-04-22T18:44:51.906945+0000 mon.a (mon.0) 5751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:52.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:52 smithi115 bash[46934]: audit 2022-04-22T18:44:51.906945+0000 mon.a (mon.0) 5751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:53 smithi121 bash[36902]: cluster 2022-04-22T18:44:52.122395+0000 mgr.y (mgr.34107) 2730 : cluster [DBG] pgmap v4306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:53 smithi121 bash[36902]: audit 2022-04-22T18:44:52.560088+0000 mon.a (mon.0) 5752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:53.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:53 smithi115 bash[45495]: cluster 2022-04-22T18:44:52.122395+0000 mgr.y (mgr.34107) 2730 : cluster [DBG] pgmap v4306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:53.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:53 smithi115 bash[45495]: audit 2022-04-22T18:44:52.560088+0000 mon.a (mon.0) 5752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:53.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:53 smithi115 bash[46934]: cluster 2022-04-22T18:44:52.122395+0000 mgr.y (mgr.34107) 2730 : cluster [DBG] pgmap v4306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:53.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:53 smithi115 bash[46934]: audit 2022-04-22T18:44:52.560088+0000 mon.a (mon.0) 5752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:54.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:44:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:44:53.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:44:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:54 smithi121 bash[36902]: audit 2022-04-22T18:44:53.727220+0000 mon.a (mon.0) 5753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:54 smithi115 bash[45495]: audit 2022-04-22T18:44:53.727220+0000 mon.a (mon.0) 5753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:54 smithi115 bash[46934]: audit 2022-04-22T18:44:53.727220+0000 mon.a (mon.0) 5753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:55 smithi121 bash[36902]: cluster 2022-04-22T18:44:54.123566+0000 mgr.y (mgr.34107) 2731 : cluster [DBG] pgmap v4307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:55 smithi115 bash[45495]: cluster 2022-04-22T18:44:54.123566+0000 mgr.y (mgr.34107) 2731 : cluster [DBG] pgmap v4307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:55.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:55 smithi115 bash[46934]: cluster 2022-04-22T18:44:54.123566+0000 mgr.y (mgr.34107) 2731 : cluster [DBG] pgmap v4307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:57 smithi121 bash[36902]: cluster 2022-04-22T18:44:56.124554+0000 mgr.y (mgr.34107) 2732 : cluster [DBG] pgmap v4308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:57 smithi121 bash[36902]: audit 2022-04-22T18:44:56.909403+0000 mon.a (mon.0) 5754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:57.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:57 smithi115 bash[45495]: cluster 2022-04-22T18:44:56.124554+0000 mgr.y (mgr.34107) 2732 : cluster [DBG] pgmap v4308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:57 smithi115 bash[45495]: audit 2022-04-22T18:44:56.909403+0000 mon.a (mon.0) 5754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:57 smithi115 bash[46934]: cluster 2022-04-22T18:44:56.124554+0000 mgr.y (mgr.34107) 2732 : cluster [DBG] pgmap v4308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:57 smithi115 bash[46934]: audit 2022-04-22T18:44:56.909403+0000 mon.a (mon.0) 5754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:44:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:58 smithi121 bash[36902]: audit 2022-04-22T18:44:57.563841+0000 mon.a (mon.0) 5755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:58.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:58 smithi115 bash[45495]: audit 2022-04-22T18:44:57.563841+0000 mon.a (mon.0) 5755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:58 smithi115 bash[46934]: audit 2022-04-22T18:44:57.563841+0000 mon.a (mon.0) 5755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:44:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:44:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:44:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (68m) 5m ago 81m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:44:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (81m) 6m ago 81m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:44:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (70m) 5m ago 82m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:44:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (69m) 6m ago 82m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:44:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (77m) 6m ago 87m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:44:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (77m) 5m ago 89m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:44:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (76m) 5m ago 89m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:44:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (76m) 6m ago 87m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (76m) 5m ago 87m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (69m) 5m ago 82m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (69m) 6m ago 82m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (74m) 5m ago 86m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (74m) 5m ago 86m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:44:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (73m) 5m ago 85m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:44:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (73m) 5m ago 85m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:44:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (72m) 6m ago 84m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:44:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (71m) 6m ago 84m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:44:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (71m) 6m ago 83m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:44:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (70m) 6m ago 83m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:44:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (68m) 6m ago 82m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:44:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:44:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:44:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:44:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:44:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:44:59.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:44:59.748 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:44:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:44:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:59 smithi121 bash[36902]: cluster 2022-04-22T18:44:58.125656+0000 mgr.y (mgr.34107) 2733 : cluster [DBG] pgmap v4309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:44:59 smithi121 bash[36902]: audit 2022-04-22T18:44:58.730483+0000 mon.a (mon.0) 5756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:59 smithi115 bash[45495]: cluster 2022-04-22T18:44:58.125656+0000 mgr.y (mgr.34107) 2733 : cluster [DBG] pgmap v4309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:44:59 smithi115 bash[45495]: audit 2022-04-22T18:44:58.730483+0000 mon.a (mon.0) 5756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:44:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:59 smithi115 bash[46934]: cluster 2022-04-22T18:44:58.125656+0000 mgr.y (mgr.34107) 2733 : cluster [DBG] pgmap v4309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:44:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:44:59 smithi115 bash[46934]: audit 2022-04-22T18:44:58.730483+0000 mon.a (mon.0) 5756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:00 smithi121 bash[36902]: audit 2022-04-22T18:44:58.958639+0000 mgr.y (mgr.34107) 2734 : audit [DBG] from='client.38286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:00 smithi121 bash[36902]: audit 2022-04-22T18:44:59.306973+0000 mgr.y (mgr.34107) 2735 : audit [DBG] from='client.48226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:00 smithi121 bash[36902]: audit 2022-04-22T18:44:59.737339+0000 mon.a (mon.0) 5757 : audit [DBG] from='client.? 172.21.15.115:0/2352679036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[45495]: audit 2022-04-22T18:44:58.958639+0000 mgr.y (mgr.34107) 2734 : audit [DBG] from='client.38286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[45495]: audit 2022-04-22T18:44:59.306973+0000 mgr.y (mgr.34107) 2735 : audit [DBG] from='client.48226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[45495]: audit 2022-04-22T18:44:59.737339+0000 mon.a (mon.0) 5757 : audit [DBG] from='client.? 172.21.15.115:0/2352679036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[46934]: audit 2022-04-22T18:44:58.958639+0000 mgr.y (mgr.34107) 2734 : audit [DBG] from='client.38286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[46934]: audit 2022-04-22T18:44:59.306973+0000 mgr.y (mgr.34107) 2735 : audit [DBG] from='client.48226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:00 smithi115 bash[46934]: audit 2022-04-22T18:44:59.737339+0000 mon.a (mon.0) 5757 : audit [DBG] from='client.? 172.21.15.115:0/2352679036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:01 smithi121 bash[36902]: cluster 2022-04-22T18:45:00.126726+0000 mgr.y (mgr.34107) 2736 : cluster [DBG] pgmap v4310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:01 smithi115 bash[45495]: cluster 2022-04-22T18:45:00.126726+0000 mgr.y (mgr.34107) 2736 : cluster [DBG] pgmap v4310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:01 smithi115 bash[46934]: cluster 2022-04-22T18:45:00.126726+0000 mgr.y (mgr.34107) 2736 : cluster [DBG] pgmap v4310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:01.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:02 smithi121 bash[36902]: audit 2022-04-22T18:45:01.912572+0000 mon.a (mon.0) 5758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:02.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:02 smithi115 bash[45495]: audit 2022-04-22T18:45:01.912572+0000 mon.a (mon.0) 5758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:02.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:02 smithi115 bash[46934]: audit 2022-04-22T18:45:01.912572+0000 mon.a (mon.0) 5758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:03 smithi121 bash[36902]: cluster 2022-04-22T18:45:02.127937+0000 mgr.y (mgr.34107) 2737 : cluster [DBG] pgmap v4311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:03 smithi121 bash[36902]: audit 2022-04-22T18:45:02.567851+0000 mon.a (mon.0) 5759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:03.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:03 smithi115 bash[45495]: cluster 2022-04-22T18:45:02.127937+0000 mgr.y (mgr.34107) 2737 : cluster [DBG] pgmap v4311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:03.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:03 smithi115 bash[45495]: audit 2022-04-22T18:45:02.567851+0000 mon.a (mon.0) 5759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:03.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:03 smithi115 bash[46934]: cluster 2022-04-22T18:45:02.127937+0000 mgr.y (mgr.34107) 2737 : cluster [DBG] pgmap v4311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:03.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:03 smithi115 bash[46934]: audit 2022-04-22T18:45:02.567851+0000 mon.a (mon.0) 5759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:04.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:03.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:04 smithi121 bash[36902]: audit 2022-04-22T18:45:03.733142+0000 mon.a (mon.0) 5760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:04.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:04 smithi115 bash[45495]: audit 2022-04-22T18:45:03.733142+0000 mon.a (mon.0) 5760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:04 smithi115 bash[46934]: audit 2022-04-22T18:45:03.733142+0000 mon.a (mon.0) 5760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:05 smithi121 bash[36902]: cluster 2022-04-22T18:45:04.129088+0000 mgr.y (mgr.34107) 2738 : cluster [DBG] pgmap v4312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:05.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:05 smithi115 bash[45495]: cluster 2022-04-22T18:45:04.129088+0000 mgr.y (mgr.34107) 2738 : cluster [DBG] pgmap v4312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:05.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:05 smithi115 bash[46934]: cluster 2022-04-22T18:45:04.129088+0000 mgr.y (mgr.34107) 2738 : cluster [DBG] pgmap v4312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:07 smithi121 bash[36902]: cluster 2022-04-22T18:45:06.130122+0000 mgr.y (mgr.34107) 2739 : cluster [DBG] pgmap v4313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:07 smithi121 bash[36902]: audit 2022-04-22T18:45:06.915222+0000 mon.a (mon.0) 5761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:07 smithi115 bash[45495]: cluster 2022-04-22T18:45:06.130122+0000 mgr.y (mgr.34107) 2739 : cluster [DBG] pgmap v4313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:07 smithi115 bash[45495]: audit 2022-04-22T18:45:06.915222+0000 mon.a (mon.0) 5761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:07 smithi115 bash[46934]: cluster 2022-04-22T18:45:06.130122+0000 mgr.y (mgr.34107) 2739 : cluster [DBG] pgmap v4313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:07 smithi115 bash[46934]: audit 2022-04-22T18:45:06.915222+0000 mon.a (mon.0) 5761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:08 smithi121 bash[36902]: audit 2022-04-22T18:45:07.571329+0000 mon.a (mon.0) 5762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:08.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:08 smithi115 bash[45495]: audit 2022-04-22T18:45:07.571329+0000 mon.a (mon.0) 5762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:08 smithi115 bash[46934]: audit 2022-04-22T18:45:07.571329+0000 mon.a (mon.0) 5762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:09 smithi121 bash[36902]: cluster 2022-04-22T18:45:08.131155+0000 mgr.y (mgr.34107) 2740 : cluster [DBG] pgmap v4314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:09 smithi121 bash[36902]: audit 2022-04-22T18:45:08.736234+0000 mon.a (mon.0) 5763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:09 smithi115 bash[45495]: cluster 2022-04-22T18:45:08.131155+0000 mgr.y (mgr.34107) 2740 : cluster [DBG] pgmap v4314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:09 smithi115 bash[45495]: audit 2022-04-22T18:45:08.736234+0000 mon.a (mon.0) 5763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:09 smithi115 bash[46934]: cluster 2022-04-22T18:45:08.131155+0000 mgr.y (mgr.34107) 2740 : cluster [DBG] pgmap v4314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:09 smithi115 bash[46934]: audit 2022-04-22T18:45:08.736234+0000 mon.a (mon.0) 5763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:11 smithi121 bash[36902]: cluster 2022-04-22T18:45:10.132220+0000 mgr.y (mgr.34107) 2741 : cluster [DBG] pgmap v4315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:11.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:11 smithi115 bash[45495]: cluster 2022-04-22T18:45:10.132220+0000 mgr.y (mgr.34107) 2741 : cluster [DBG] pgmap v4315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:11 smithi115 bash[46934]: cluster 2022-04-22T18:45:10.132220+0000 mgr.y (mgr.34107) 2741 : cluster [DBG] pgmap v4315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:11.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:12 smithi121 bash[36902]: audit 2022-04-22T18:45:11.918394+0000 mon.a (mon.0) 5764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:12 smithi115 bash[45495]: audit 2022-04-22T18:45:11.918394+0000 mon.a (mon.0) 5764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:12.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:12 smithi115 bash[46934]: audit 2022-04-22T18:45:11.918394+0000 mon.a (mon.0) 5764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:13 smithi121 bash[36902]: cluster 2022-04-22T18:45:12.133305+0000 mgr.y (mgr.34107) 2742 : cluster [DBG] pgmap v4316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:13 smithi121 bash[36902]: audit 2022-04-22T18:45:12.574836+0000 mon.a (mon.0) 5765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:13 smithi115 bash[45495]: cluster 2022-04-22T18:45:12.133305+0000 mgr.y (mgr.34107) 2742 : cluster [DBG] pgmap v4316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:13 smithi115 bash[45495]: audit 2022-04-22T18:45:12.574836+0000 mon.a (mon.0) 5765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:13 smithi115 bash[46934]: cluster 2022-04-22T18:45:12.133305+0000 mgr.y (mgr.34107) 2742 : cluster [DBG] pgmap v4316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:13 smithi115 bash[46934]: audit 2022-04-22T18:45:12.574836+0000 mon.a (mon.0) 5765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:14.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:13.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:14 smithi121 bash[36902]: audit 2022-04-22T18:45:13.739244+0000 mon.a (mon.0) 5766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:14 smithi121 bash[36902]: audit 2022-04-22T18:45:14.127194+0000 mon.a (mon.0) 5767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:45:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:14 smithi121 bash[36902]: audit 2022-04-22T18:45:14.128618+0000 mon.a (mon.0) 5768 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:45:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[45495]: audit 2022-04-22T18:45:13.739244+0000 mon.a (mon.0) 5766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[45495]: audit 2022-04-22T18:45:14.127194+0000 mon.a (mon.0) 5767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:45:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[45495]: audit 2022-04-22T18:45:14.128618+0000 mon.a (mon.0) 5768 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:45:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[46934]: audit 2022-04-22T18:45:13.739244+0000 mon.a (mon.0) 5766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[46934]: audit 2022-04-22T18:45:14.127194+0000 mon.a (mon.0) 5767 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:45:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:14 smithi115 bash[46934]: audit 2022-04-22T18:45:14.128618+0000 mon.a (mon.0) 5768 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:45:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:15 smithi121 bash[36902]: cluster 2022-04-22T18:45:14.134257+0000 mgr.y (mgr.34107) 2743 : cluster [DBG] pgmap v4317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:15 smithi115 bash[45495]: cluster 2022-04-22T18:45:14.134257+0000 mgr.y (mgr.34107) 2743 : cluster [DBG] pgmap v4317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:15.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:15 smithi115 bash[46934]: cluster 2022-04-22T18:45:14.134257+0000 mgr.y (mgr.34107) 2743 : cluster [DBG] pgmap v4317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:17 smithi121 bash[36902]: cluster 2022-04-22T18:45:16.135256+0000 mgr.y (mgr.34107) 2744 : cluster [DBG] pgmap v4318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:17 smithi121 bash[36902]: audit 2022-04-22T18:45:16.921388+0000 mon.a (mon.0) 5769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:17 smithi115 bash[45495]: cluster 2022-04-22T18:45:16.135256+0000 mgr.y (mgr.34107) 2744 : cluster [DBG] pgmap v4318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:17 smithi115 bash[45495]: audit 2022-04-22T18:45:16.921388+0000 mon.a (mon.0) 5769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:17 smithi115 bash[46934]: cluster 2022-04-22T18:45:16.135256+0000 mgr.y (mgr.34107) 2744 : cluster [DBG] pgmap v4318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:17 smithi115 bash[46934]: audit 2022-04-22T18:45:16.921388+0000 mon.a (mon.0) 5769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:18 smithi121 bash[36902]: audit 2022-04-22T18:45:17.578622+0000 mon.a (mon.0) 5770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:18 smithi115 bash[45495]: audit 2022-04-22T18:45:17.578622+0000 mon.a (mon.0) 5770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:18 smithi115 bash[46934]: audit 2022-04-22T18:45:17.578622+0000 mon.a (mon.0) 5770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:19 smithi121 bash[36902]: cluster 2022-04-22T18:45:18.136350+0000 mgr.y (mgr.34107) 2745 : cluster [DBG] pgmap v4319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:19 smithi121 bash[36902]: audit 2022-04-22T18:45:18.742269+0000 mon.a (mon.0) 5771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:19 smithi115 bash[45495]: cluster 2022-04-22T18:45:18.136350+0000 mgr.y (mgr.34107) 2745 : cluster [DBG] pgmap v4319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:19 smithi115 bash[45495]: audit 2022-04-22T18:45:18.742269+0000 mon.a (mon.0) 5771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:19 smithi115 bash[46934]: cluster 2022-04-22T18:45:18.136350+0000 mgr.y (mgr.34107) 2745 : cluster [DBG] pgmap v4319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:19 smithi115 bash[46934]: audit 2022-04-22T18:45:18.742269+0000 mon.a (mon.0) 5771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:21 smithi121 bash[36902]: cluster 2022-04-22T18:45:20.137536+0000 mgr.y (mgr.34107) 2746 : cluster [DBG] pgmap v4320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:21 smithi115 bash[45495]: cluster 2022-04-22T18:45:20.137536+0000 mgr.y (mgr.34107) 2746 : cluster [DBG] pgmap v4320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:21 smithi115 bash[46934]: cluster 2022-04-22T18:45:20.137536+0000 mgr.y (mgr.34107) 2746 : cluster [DBG] pgmap v4320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:21.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:22.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:22 smithi121 bash[36902]: audit 2022-04-22T18:45:21.924491+0000 mon.a (mon.0) 5772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:22 smithi115 bash[45495]: audit 2022-04-22T18:45:21.924491+0000 mon.a (mon.0) 5772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:22 smithi115 bash[46934]: audit 2022-04-22T18:45:21.924491+0000 mon.a (mon.0) 5772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:23 smithi121 bash[36902]: cluster 2022-04-22T18:45:22.138714+0000 mgr.y (mgr.34107) 2747 : cluster [DBG] pgmap v4321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:23.773 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:23 smithi121 bash[36902]: audit 2022-04-22T18:45:22.582177+0000 mon.a (mon.0) 5773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:23 smithi115 bash[45495]: cluster 2022-04-22T18:45:22.138714+0000 mgr.y (mgr.34107) 2747 : cluster [DBG] pgmap v4321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:23 smithi115 bash[45495]: audit 2022-04-22T18:45:22.582177+0000 mon.a (mon.0) 5773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:23 smithi115 bash[46934]: cluster 2022-04-22T18:45:22.138714+0000 mgr.y (mgr.34107) 2747 : cluster [DBG] pgmap v4321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:23 smithi115 bash[46934]: audit 2022-04-22T18:45:22.582177+0000 mon.a (mon.0) 5773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:24.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:23.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:24 smithi121 bash[36902]: audit 2022-04-22T18:45:23.745604+0000 mon.a (mon.0) 5774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:24.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:24 smithi115 bash[46934]: audit 2022-04-22T18:45:23.745604+0000 mon.a (mon.0) 5774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:24.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:24 smithi115 bash[45495]: audit 2022-04-22T18:45:23.745604+0000 mon.a (mon.0) 5774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:25 smithi121 bash[36902]: cluster 2022-04-22T18:45:24.139970+0000 mgr.y (mgr.34107) 2748 : cluster [DBG] pgmap v4322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:25 smithi115 bash[45495]: cluster 2022-04-22T18:45:24.139970+0000 mgr.y (mgr.34107) 2748 : cluster [DBG] pgmap v4322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:25 smithi115 bash[46934]: cluster 2022-04-22T18:45:24.139970+0000 mgr.y (mgr.34107) 2748 : cluster [DBG] pgmap v4322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:27 smithi121 bash[36902]: cluster 2022-04-22T18:45:26.141082+0000 mgr.y (mgr.34107) 2749 : cluster [DBG] pgmap v4323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:27 smithi121 bash[36902]: audit 2022-04-22T18:45:26.927096+0000 mon.a (mon.0) 5775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:27 smithi115 bash[46934]: cluster 2022-04-22T18:45:26.141082+0000 mgr.y (mgr.34107) 2749 : cluster [DBG] pgmap v4323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:27 smithi115 bash[46934]: audit 2022-04-22T18:45:26.927096+0000 mon.a (mon.0) 5775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:27 smithi115 bash[45495]: cluster 2022-04-22T18:45:26.141082+0000 mgr.y (mgr.34107) 2749 : cluster [DBG] pgmap v4323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:27 smithi115 bash[45495]: audit 2022-04-22T18:45:26.927096+0000 mon.a (mon.0) 5775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:28 smithi121 bash[36902]: audit 2022-04-22T18:45:27.586006+0000 mon.a (mon.0) 5776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:28 smithi115 bash[46934]: audit 2022-04-22T18:45:27.586006+0000 mon.a (mon.0) 5776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:28 smithi115 bash[45495]: audit 2022-04-22T18:45:27.586006+0000 mon.a (mon.0) 5776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:29 smithi121 bash[36902]: cluster 2022-04-22T18:45:28.142199+0000 mgr.y (mgr.34107) 2750 : cluster [DBG] pgmap v4324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:29 smithi121 bash[36902]: audit 2022-04-22T18:45:28.748725+0000 mon.a (mon.0) 5777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:29 smithi115 bash[45495]: cluster 2022-04-22T18:45:28.142199+0000 mgr.y (mgr.34107) 2750 : cluster [DBG] pgmap v4324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:29 smithi115 bash[45495]: audit 2022-04-22T18:45:28.748725+0000 mon.a (mon.0) 5777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:29 smithi115 bash[46934]: cluster 2022-04-22T18:45:28.142199+0000 mgr.y (mgr.34107) 2750 : cluster [DBG] pgmap v4324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:29 smithi115 bash[46934]: audit 2022-04-22T18:45:28.748725+0000 mon.a (mon.0) 5777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:30.125 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (68m) 6m ago 82m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (82m) 7m ago 82m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (70m) 6m ago 83m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (70m) 7m ago 83m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (77m) 7m ago 87m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:45:30.476 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (78m) 6m ago 89m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (77m) 6m ago 89m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (76m) 7m ago 87m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (76m) 6m ago 88m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (69m) 6m ago 82m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (69m) 7m ago 82m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (75m) 6m ago 87m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (74m) 6m ago 86m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (74m) 6m ago 86m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (73m) 6m ago 85m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:45:30.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (73m) 7m ago 85m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:45:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (72m) 7m ago 84m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:45:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (71m) 7m ago 84m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:45:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (70m) 7m ago 83m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:45:30.478 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (69m) 7m ago 82m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:45:30.883 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:45:30.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:45:30.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:45:30.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:45:30.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:45:30.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:45:30.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:45:30.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:45:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:31 smithi121 bash[36902]: audit 2022-04-22T18:45:30.101537+0000 mgr.y (mgr.34107) 2751 : audit [DBG] from='client.38304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:31 smithi121 bash[36902]: cluster 2022-04-22T18:45:30.143224+0000 mgr.y (mgr.34107) 2752 : cluster [DBG] pgmap v4325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:31 smithi121 bash[36902]: audit 2022-04-22T18:45:30.447882+0000 mgr.y (mgr.34107) 2753 : audit [DBG] from='client.48244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:31 smithi121 bash[36902]: audit 2022-04-22T18:45:30.874147+0000 mon.c (mon.1) 157 : audit [DBG] from='client.? 172.21.15.115:0/536905789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[45495]: audit 2022-04-22T18:45:30.101537+0000 mgr.y (mgr.34107) 2751 : audit [DBG] from='client.38304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[45495]: cluster 2022-04-22T18:45:30.143224+0000 mgr.y (mgr.34107) 2752 : cluster [DBG] pgmap v4325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[45495]: audit 2022-04-22T18:45:30.447882+0000 mgr.y (mgr.34107) 2753 : audit [DBG] from='client.48244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[45495]: audit 2022-04-22T18:45:30.874147+0000 mon.c (mon.1) 157 : audit [DBG] from='client.? 172.21.15.115:0/536905789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[46934]: audit 2022-04-22T18:45:30.101537+0000 mgr.y (mgr.34107) 2751 : audit [DBG] from='client.38304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[46934]: cluster 2022-04-22T18:45:30.143224+0000 mgr.y (mgr.34107) 2752 : cluster [DBG] pgmap v4325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[46934]: audit 2022-04-22T18:45:30.447882+0000 mgr.y (mgr.34107) 2753 : audit [DBG] from='client.48244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:45:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[46934]: audit 2022-04-22T18:45:30.874147+0000 mon.c (mon.1) 157 : audit [DBG] from='client.? 172.21.15.115:0/536905789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:45:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:31.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:32 smithi121 bash[36902]: audit 2022-04-22T18:45:31.930278+0000 mon.a (mon.0) 5778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:32 smithi115 bash[45495]: audit 2022-04-22T18:45:31.930278+0000 mon.a (mon.0) 5778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:32 smithi115 bash[46934]: audit 2022-04-22T18:45:31.930278+0000 mon.a (mon.0) 5778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:33 smithi121 bash[36902]: cluster 2022-04-22T18:45:32.144400+0000 mgr.y (mgr.34107) 2754 : cluster [DBG] pgmap v4326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:33 smithi121 bash[36902]: audit 2022-04-22T18:45:32.589621+0000 mon.a (mon.0) 5779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:33 smithi115 bash[46934]: cluster 2022-04-22T18:45:32.144400+0000 mgr.y (mgr.34107) 2754 : cluster [DBG] pgmap v4326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:33 smithi115 bash[46934]: audit 2022-04-22T18:45:32.589621+0000 mon.a (mon.0) 5779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:33 smithi115 bash[45495]: cluster 2022-04-22T18:45:32.144400+0000 mgr.y (mgr.34107) 2754 : cluster [DBG] pgmap v4326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:33 smithi115 bash[45495]: audit 2022-04-22T18:45:32.589621+0000 mon.a (mon.0) 5779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:34.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:33.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:34 smithi121 bash[36902]: audit 2022-04-22T18:45:33.751837+0000 mon.a (mon.0) 5780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:34.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:34 smithi115 bash[46934]: audit 2022-04-22T18:45:33.751837+0000 mon.a (mon.0) 5780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:34 smithi115 bash[45495]: audit 2022-04-22T18:45:33.751837+0000 mon.a (mon.0) 5780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:35 smithi121 bash[36902]: cluster 2022-04-22T18:45:34.145528+0000 mgr.y (mgr.34107) 2755 : cluster [DBG] pgmap v4327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:35 smithi115 bash[45495]: cluster 2022-04-22T18:45:34.145528+0000 mgr.y (mgr.34107) 2755 : cluster [DBG] pgmap v4327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:35 smithi115 bash[46934]: cluster 2022-04-22T18:45:34.145528+0000 mgr.y (mgr.34107) 2755 : cluster [DBG] pgmap v4327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: cluster 2022-04-22T18:45:36.146594+0000 mgr.y (mgr.34107) 2756 : cluster [DBG] pgmap v4328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: audit 2022-04-22T18:45:36.805088+0000 mon.a (mon.0) 5781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:45:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: audit 2022-04-22T18:45:36.808339+0000 mon.a (mon.0) 5782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:45:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: audit 2022-04-22T18:45:36.810346+0000 mon.a (mon.0) 5783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:45:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: audit 2022-04-22T18:45:36.933394+0000 mon.a (mon.0) 5784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:37 smithi121 bash[36902]: audit 2022-04-22T18:45:37.176216+0000 mon.a (mon.0) 5785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: cluster 2022-04-22T18:45:36.146594+0000 mgr.y (mgr.34107) 2756 : cluster [DBG] pgmap v4328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: audit 2022-04-22T18:45:36.805088+0000 mon.a (mon.0) 5781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: audit 2022-04-22T18:45:36.808339+0000 mon.a (mon.0) 5782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: audit 2022-04-22T18:45:36.810346+0000 mon.a (mon.0) 5783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: audit 2022-04-22T18:45:36.933394+0000 mon.a (mon.0) 5784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[45495]: audit 2022-04-22T18:45:37.176216+0000 mon.a (mon.0) 5785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: cluster 2022-04-22T18:45:36.146594+0000 mgr.y (mgr.34107) 2756 : cluster [DBG] pgmap v4328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: audit 2022-04-22T18:45:36.805088+0000 mon.a (mon.0) 5781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: audit 2022-04-22T18:45:36.808339+0000 mon.a (mon.0) 5782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: audit 2022-04-22T18:45:36.810346+0000 mon.a (mon.0) 5783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: audit 2022-04-22T18:45:36.933394+0000 mon.a (mon.0) 5784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:37 smithi115 bash[46934]: audit 2022-04-22T18:45:37.176216+0000 mon.a (mon.0) 5785 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:45:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:38 smithi121 bash[36902]: audit 2022-04-22T18:45:37.593195+0000 mon.a (mon.0) 5786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:38 smithi115 bash[45495]: audit 2022-04-22T18:45:37.593195+0000 mon.a (mon.0) 5786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:38 smithi115 bash[46934]: audit 2022-04-22T18:45:37.593195+0000 mon.a (mon.0) 5786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:39 smithi121 bash[36902]: cluster 2022-04-22T18:45:38.147783+0000 mgr.y (mgr.34107) 2757 : cluster [DBG] pgmap v4329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:39 smithi121 bash[36902]: audit 2022-04-22T18:45:38.754909+0000 mon.a (mon.0) 5787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:39 smithi115 bash[45495]: cluster 2022-04-22T18:45:38.147783+0000 mgr.y (mgr.34107) 2757 : cluster [DBG] pgmap v4329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:39 smithi115 bash[45495]: audit 2022-04-22T18:45:38.754909+0000 mon.a (mon.0) 5787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:39 smithi115 bash[46934]: cluster 2022-04-22T18:45:38.147783+0000 mgr.y (mgr.34107) 2757 : cluster [DBG] pgmap v4329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:39 smithi115 bash[46934]: audit 2022-04-22T18:45:38.754909+0000 mon.a (mon.0) 5787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:41 smithi115 bash[45495]: cluster 2022-04-22T18:45:40.148787+0000 mgr.y (mgr.34107) 2758 : cluster [DBG] pgmap v4330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:41 smithi115 bash[46934]: cluster 2022-04-22T18:45:40.148787+0000 mgr.y (mgr.34107) 2758 : cluster [DBG] pgmap v4330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:41 smithi121 bash[36902]: cluster 2022-04-22T18:45:40.148787+0000 mgr.y (mgr.34107) 2758 : cluster [DBG] pgmap v4330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:41.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:42 smithi115 bash[45495]: audit 2022-04-22T18:45:41.936249+0000 mon.a (mon.0) 5788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:42 smithi115 bash[46934]: audit 2022-04-22T18:45:41.936249+0000 mon.a (mon.0) 5788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:42 smithi121 bash[36902]: audit 2022-04-22T18:45:41.936249+0000 mon.a (mon.0) 5788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:43 smithi115 bash[45495]: cluster 2022-04-22T18:45:42.149790+0000 mgr.y (mgr.34107) 2759 : cluster [DBG] pgmap v4331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:43 smithi115 bash[45495]: audit 2022-04-22T18:45:42.596920+0000 mon.a (mon.0) 5789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:43 smithi115 bash[46934]: cluster 2022-04-22T18:45:42.149790+0000 mgr.y (mgr.34107) 2759 : cluster [DBG] pgmap v4331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:43 smithi115 bash[46934]: audit 2022-04-22T18:45:42.596920+0000 mon.a (mon.0) 5789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:43 smithi121 bash[36902]: cluster 2022-04-22T18:45:42.149790+0000 mgr.y (mgr.34107) 2759 : cluster [DBG] pgmap v4331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:43 smithi121 bash[36902]: audit 2022-04-22T18:45:42.596920+0000 mon.a (mon.0) 5789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:44.045 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:43.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:44 smithi115 bash[45495]: audit 2022-04-22T18:45:43.758250+0000 mon.a (mon.0) 5790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:44 smithi115 bash[46934]: audit 2022-04-22T18:45:43.758250+0000 mon.a (mon.0) 5790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:44 smithi121 bash[36902]: audit 2022-04-22T18:45:43.758250+0000 mon.a (mon.0) 5790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:45 smithi115 bash[45495]: cluster 2022-04-22T18:45:44.150767+0000 mgr.y (mgr.34107) 2760 : cluster [DBG] pgmap v4332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:45 smithi115 bash[46934]: cluster 2022-04-22T18:45:44.150767+0000 mgr.y (mgr.34107) 2760 : cluster [DBG] pgmap v4332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:45 smithi121 bash[36902]: cluster 2022-04-22T18:45:44.150767+0000 mgr.y (mgr.34107) 2760 : cluster [DBG] pgmap v4332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:47 smithi115 bash[45495]: cluster 2022-04-22T18:45:46.151869+0000 mgr.y (mgr.34107) 2761 : cluster [DBG] pgmap v4333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:47 smithi115 bash[45495]: audit 2022-04-22T18:45:46.939237+0000 mon.a (mon.0) 5791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:47 smithi115 bash[46934]: cluster 2022-04-22T18:45:46.151869+0000 mgr.y (mgr.34107) 2761 : cluster [DBG] pgmap v4333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:47 smithi115 bash[46934]: audit 2022-04-22T18:45:46.939237+0000 mon.a (mon.0) 5791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:47 smithi121 bash[36902]: cluster 2022-04-22T18:45:46.151869+0000 mgr.y (mgr.34107) 2761 : cluster [DBG] pgmap v4333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:47 smithi121 bash[36902]: audit 2022-04-22T18:45:46.939237+0000 mon.a (mon.0) 5791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:48 smithi115 bash[45495]: audit 2022-04-22T18:45:47.600489+0000 mon.a (mon.0) 5792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:48 smithi115 bash[46934]: audit 2022-04-22T18:45:47.600489+0000 mon.a (mon.0) 5792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:48 smithi121 bash[36902]: audit 2022-04-22T18:45:47.600489+0000 mon.a (mon.0) 5792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:49 smithi115 bash[45495]: cluster 2022-04-22T18:45:48.152893+0000 mgr.y (mgr.34107) 2762 : cluster [DBG] pgmap v4334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:49 smithi115 bash[45495]: audit 2022-04-22T18:45:48.761333+0000 mon.a (mon.0) 5793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:49 smithi115 bash[46934]: cluster 2022-04-22T18:45:48.152893+0000 mgr.y (mgr.34107) 2762 : cluster [DBG] pgmap v4334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:49 smithi115 bash[46934]: audit 2022-04-22T18:45:48.761333+0000 mon.a (mon.0) 5793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:49 smithi121 bash[36902]: cluster 2022-04-22T18:45:48.152893+0000 mgr.y (mgr.34107) 2762 : cluster [DBG] pgmap v4334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:49 smithi121 bash[36902]: audit 2022-04-22T18:45:48.761333+0000 mon.a (mon.0) 5793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:51.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:51 smithi115 bash[46934]: cluster 2022-04-22T18:45:50.153946+0000 mgr.y (mgr.34107) 2763 : cluster [DBG] pgmap v4335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:51.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:51 smithi115 bash[45495]: cluster 2022-04-22T18:45:50.153946+0000 mgr.y (mgr.34107) 2763 : cluster [DBG] pgmap v4335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:51 smithi121 bash[36902]: cluster 2022-04-22T18:45:50.153946+0000 mgr.y (mgr.34107) 2763 : cluster [DBG] pgmap v4335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:51.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:52 smithi121 bash[36902]: audit 2022-04-22T18:45:51.942219+0000 mon.a (mon.0) 5794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:52 smithi115 bash[45495]: audit 2022-04-22T18:45:51.942219+0000 mon.a (mon.0) 5794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:52 smithi115 bash[46934]: audit 2022-04-22T18:45:51.942219+0000 mon.a (mon.0) 5794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:53 smithi121 bash[36902]: cluster 2022-04-22T18:45:52.155046+0000 mgr.y (mgr.34107) 2764 : cluster [DBG] pgmap v4336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:53 smithi121 bash[36902]: audit 2022-04-22T18:45:52.604280+0000 mon.a (mon.0) 5795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:54.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:45:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:45:53.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:45:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:53 smithi115 bash[45495]: cluster 2022-04-22T18:45:52.155046+0000 mgr.y (mgr.34107) 2764 : cluster [DBG] pgmap v4336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:53 smithi115 bash[45495]: audit 2022-04-22T18:45:52.604280+0000 mon.a (mon.0) 5795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:53 smithi115 bash[46934]: cluster 2022-04-22T18:45:52.155046+0000 mgr.y (mgr.34107) 2764 : cluster [DBG] pgmap v4336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:53 smithi115 bash[46934]: audit 2022-04-22T18:45:52.604280+0000 mon.a (mon.0) 5795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:54 smithi121 bash[36902]: audit 2022-04-22T18:45:53.764652+0000 mon.a (mon.0) 5796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:54 smithi115 bash[45495]: audit 2022-04-22T18:45:53.764652+0000 mon.a (mon.0) 5796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:55.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:54 smithi115 bash[46934]: audit 2022-04-22T18:45:53.764652+0000 mon.a (mon.0) 5796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:45:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:55 smithi121 bash[36902]: cluster 2022-04-22T18:45:54.156143+0000 mgr.y (mgr.34107) 2765 : cluster [DBG] pgmap v4337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:56.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:55 smithi115 bash[45495]: cluster 2022-04-22T18:45:54.156143+0000 mgr.y (mgr.34107) 2765 : cluster [DBG] pgmap v4337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:56.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:55 smithi115 bash[46934]: cluster 2022-04-22T18:45:54.156143+0000 mgr.y (mgr.34107) 2765 : cluster [DBG] pgmap v4337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:57 smithi121 bash[36902]: cluster 2022-04-22T18:45:56.157149+0000 mgr.y (mgr.34107) 2766 : cluster [DBG] pgmap v4338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:57 smithi121 bash[36902]: audit 2022-04-22T18:45:56.945098+0000 mon.a (mon.0) 5797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:57 smithi115 bash[46934]: cluster 2022-04-22T18:45:56.157149+0000 mgr.y (mgr.34107) 2766 : cluster [DBG] pgmap v4338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:57 smithi115 bash[46934]: audit 2022-04-22T18:45:56.945098+0000 mon.a (mon.0) 5797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:57 smithi115 bash[45495]: cluster 2022-04-22T18:45:56.157149+0000 mgr.y (mgr.34107) 2766 : cluster [DBG] pgmap v4338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:45:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:57 smithi115 bash[45495]: audit 2022-04-22T18:45:56.945098+0000 mon.a (mon.0) 5797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:45:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:58 smithi121 bash[36902]: audit 2022-04-22T18:45:57.607775+0000 mon.a (mon.0) 5798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:58 smithi115 bash[46934]: audit 2022-04-22T18:45:57.607775+0000 mon.a (mon.0) 5798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:45:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:58 smithi115 bash[45495]: audit 2022-04-22T18:45:57.607775+0000 mon.a (mon.0) 5798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:59 smithi121 bash[36902]: cluster 2022-04-22T18:45:58.158330+0000 mgr.y (mgr.34107) 2767 : cluster [DBG] pgmap v4339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:45:59 smithi121 bash[36902]: audit 2022-04-22T18:45:58.767560+0000 mon.a (mon.0) 5799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:59 smithi115 bash[45495]: cluster 2022-04-22T18:45:58.158330+0000 mgr.y (mgr.34107) 2767 : cluster [DBG] pgmap v4339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:45:59 smithi115 bash[45495]: audit 2022-04-22T18:45:58.767560+0000 mon.a (mon.0) 5799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:59 smithi115 bash[46934]: cluster 2022-04-22T18:45:58.158330+0000 mgr.y (mgr.34107) 2767 : cluster [DBG] pgmap v4339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:45:59 smithi115 bash[46934]: audit 2022-04-22T18:45:58.767560+0000 mon.a (mon.0) 5799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:01.253 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:46:01.598 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:46:01.598 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (69m) 6m ago 82m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (82m) 7m ago 82m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (71m) 6m ago 83m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (70m) 7m ago 83m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (78m) 7m ago 88m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (78m) 6m ago 90m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (77m) 6m ago 90m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (77m) 7m ago 88m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (77m) 6m ago 88m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (70m) 6m ago 83m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:46:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (70m) 7m ago 83m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (75m) 6m ago 87m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (75m) 6m ago 87m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (74m) 6m ago 86m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (74m) 6m ago 86m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (73m) 7m ago 85m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (72m) 7m ago 85m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (72m) 7m ago 84m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (71m) 7m ago 84m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:46:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (69m) 7m ago 83m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:46:01.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:01 smithi115 bash[46934]: cluster 2022-04-22T18:46:00.159456+0000 mgr.y (mgr.34107) 2768 : cluster [DBG] pgmap v4340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:01.918 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:01 smithi115 bash[45495]: cluster 2022-04-22T18:46:00.159456+0000 mgr.y (mgr.34107) 2768 : cluster [DBG] pgmap v4340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:02.009 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:46:02.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:46:02.011 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:46:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:01 smithi121 bash[36902]: cluster 2022-04-22T18:46:00.159456+0000 mgr.y (mgr.34107) 2768 : cluster [DBG] pgmap v4340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:01.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:02 smithi121 bash[36902]: audit 2022-04-22T18:46:01.230393+0000 mgr.y (mgr.34107) 2769 : audit [DBG] from='client.48256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:02 smithi121 bash[36902]: audit 2022-04-22T18:46:01.570488+0000 mgr.y (mgr.34107) 2770 : audit [DBG] from='client.48262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:02 smithi121 bash[36902]: audit 2022-04-22T18:46:01.948735+0000 mon.a (mon.0) 5800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:02 smithi121 bash[36902]: audit 2022-04-22T18:46:02.001024+0000 mon.a (mon.0) 5801 : audit [DBG] from='client.? 172.21.15.115:0/1011785503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[45495]: audit 2022-04-22T18:46:01.230393+0000 mgr.y (mgr.34107) 2769 : audit [DBG] from='client.48256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[45495]: audit 2022-04-22T18:46:01.570488+0000 mgr.y (mgr.34107) 2770 : audit [DBG] from='client.48262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[45495]: audit 2022-04-22T18:46:01.948735+0000 mon.a (mon.0) 5800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[45495]: audit 2022-04-22T18:46:02.001024+0000 mon.a (mon.0) 5801 : audit [DBG] from='client.? 172.21.15.115:0/1011785503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[46934]: audit 2022-04-22T18:46:01.230393+0000 mgr.y (mgr.34107) 2769 : audit [DBG] from='client.48256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[46934]: audit 2022-04-22T18:46:01.570488+0000 mgr.y (mgr.34107) 2770 : audit [DBG] from='client.48262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[46934]: audit 2022-04-22T18:46:01.948735+0000 mon.a (mon.0) 5800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:02 smithi115 bash[46934]: audit 2022-04-22T18:46:02.001024+0000 mon.a (mon.0) 5801 : audit [DBG] from='client.? 172.21.15.115:0/1011785503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:03 smithi121 bash[36902]: cluster 2022-04-22T18:46:02.160494+0000 mgr.y (mgr.34107) 2771 : cluster [DBG] pgmap v4341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:03 smithi121 bash[36902]: audit 2022-04-22T18:46:02.611062+0000 mon.a (mon.0) 5802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:04.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:03.784Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:04.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:03 smithi115 bash[45495]: cluster 2022-04-22T18:46:02.160494+0000 mgr.y (mgr.34107) 2771 : cluster [DBG] pgmap v4341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:04.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:03 smithi115 bash[45495]: audit 2022-04-22T18:46:02.611062+0000 mon.a (mon.0) 5802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:04.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:03 smithi115 bash[46934]: cluster 2022-04-22T18:46:02.160494+0000 mgr.y (mgr.34107) 2771 : cluster [DBG] pgmap v4341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:04.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:03 smithi115 bash[46934]: audit 2022-04-22T18:46:02.611062+0000 mon.a (mon.0) 5802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:04 smithi121 bash[36902]: audit 2022-04-22T18:46:03.770873+0000 mon.a (mon.0) 5803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:04 smithi115 bash[45495]: audit 2022-04-22T18:46:03.770873+0000 mon.a (mon.0) 5803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:04 smithi115 bash[46934]: audit 2022-04-22T18:46:03.770873+0000 mon.a (mon.0) 5803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:05 smithi121 bash[36902]: cluster 2022-04-22T18:46:04.161626+0000 mgr.y (mgr.34107) 2772 : cluster [DBG] pgmap v4342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:05 smithi115 bash[45495]: cluster 2022-04-22T18:46:04.161626+0000 mgr.y (mgr.34107) 2772 : cluster [DBG] pgmap v4342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:05 smithi115 bash[46934]: cluster 2022-04-22T18:46:04.161626+0000 mgr.y (mgr.34107) 2772 : cluster [DBG] pgmap v4342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:07 smithi121 bash[36902]: cluster 2022-04-22T18:46:06.162676+0000 mgr.y (mgr.34107) 2773 : cluster [DBG] pgmap v4343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:07 smithi121 bash[36902]: audit 2022-04-22T18:46:06.950849+0000 mon.a (mon.0) 5804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:07 smithi115 bash[46934]: cluster 2022-04-22T18:46:06.162676+0000 mgr.y (mgr.34107) 2773 : cluster [DBG] pgmap v4343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:07 smithi115 bash[46934]: audit 2022-04-22T18:46:06.950849+0000 mon.a (mon.0) 5804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:07 smithi115 bash[45495]: cluster 2022-04-22T18:46:06.162676+0000 mgr.y (mgr.34107) 2773 : cluster [DBG] pgmap v4343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:07 smithi115 bash[45495]: audit 2022-04-22T18:46:06.950849+0000 mon.a (mon.0) 5804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:08 smithi121 bash[36902]: audit 2022-04-22T18:46:07.614471+0000 mon.a (mon.0) 5805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:08 smithi115 bash[45495]: audit 2022-04-22T18:46:07.614471+0000 mon.a (mon.0) 5805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:08 smithi115 bash[46934]: audit 2022-04-22T18:46:07.614471+0000 mon.a (mon.0) 5805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:09 smithi121 bash[36902]: cluster 2022-04-22T18:46:08.163897+0000 mgr.y (mgr.34107) 2774 : cluster [DBG] pgmap v4344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:09 smithi121 bash[36902]: audit 2022-04-22T18:46:08.774002+0000 mon.a (mon.0) 5806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:09 smithi115 bash[45495]: cluster 2022-04-22T18:46:08.163897+0000 mgr.y (mgr.34107) 2774 : cluster [DBG] pgmap v4344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:09 smithi115 bash[45495]: audit 2022-04-22T18:46:08.774002+0000 mon.a (mon.0) 5806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:09 smithi115 bash[46934]: cluster 2022-04-22T18:46:08.163897+0000 mgr.y (mgr.34107) 2774 : cluster [DBG] pgmap v4344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:09 smithi115 bash[46934]: audit 2022-04-22T18:46:08.774002+0000 mon.a (mon.0) 5806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:11.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:11 smithi115 bash[46934]: cluster 2022-04-22T18:46:10.164951+0000 mgr.y (mgr.34107) 2775 : cluster [DBG] pgmap v4345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:11.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:11 smithi115 bash[45495]: cluster 2022-04-22T18:46:10.164951+0000 mgr.y (mgr.34107) 2775 : cluster [DBG] pgmap v4345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:11 smithi121 bash[36902]: cluster 2022-04-22T18:46:10.164951+0000 mgr.y (mgr.34107) 2775 : cluster [DBG] pgmap v4345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:11.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:12 smithi121 bash[36902]: audit 2022-04-22T18:46:11.958009+0000 mon.a (mon.0) 5807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:12 smithi121 bash[36902]: audit 2022-04-22T18:46:12.618021+0000 mon.a (mon.0) 5808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:12 smithi115 bash[45495]: audit 2022-04-22T18:46:11.958009+0000 mon.a (mon.0) 5807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:12 smithi115 bash[45495]: audit 2022-04-22T18:46:12.618021+0000 mon.a (mon.0) 5808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:12 smithi115 bash[46934]: audit 2022-04-22T18:46:11.958009+0000 mon.a (mon.0) 5807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:12 smithi115 bash[46934]: audit 2022-04-22T18:46:12.618021+0000 mon.a (mon.0) 5808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:14.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:13.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:13 smithi115 bash[45495]: cluster 2022-04-22T18:46:12.166046+0000 mgr.y (mgr.34107) 2776 : cluster [DBG] pgmap v4346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:13 smithi115 bash[45495]: audit 2022-04-22T18:46:13.777292+0000 mon.a (mon.0) 5809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:13 smithi115 bash[46934]: cluster 2022-04-22T18:46:12.166046+0000 mgr.y (mgr.34107) 2776 : cluster [DBG] pgmap v4346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:13 smithi115 bash[46934]: audit 2022-04-22T18:46:13.777292+0000 mon.a (mon.0) 5809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:13 smithi121 bash[36902]: cluster 2022-04-22T18:46:12.166046+0000 mgr.y (mgr.34107) 2776 : cluster [DBG] pgmap v4346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:13 smithi121 bash[36902]: audit 2022-04-22T18:46:13.777292+0000 mon.a (mon.0) 5809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:14 smithi121 bash[36902]: audit 2022-04-22T18:46:14.139212+0000 mon.a (mon.0) 5810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:46:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:14 smithi121 bash[36902]: audit 2022-04-22T18:46:14.139975+0000 mon.a (mon.0) 5811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:46:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:14 smithi115 bash[46934]: audit 2022-04-22T18:46:14.139212+0000 mon.a (mon.0) 5810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:46:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:14 smithi115 bash[46934]: audit 2022-04-22T18:46:14.139975+0000 mon.a (mon.0) 5811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:46:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:14 smithi115 bash[45495]: audit 2022-04-22T18:46:14.139212+0000 mon.a (mon.0) 5810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:46:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:14 smithi115 bash[45495]: audit 2022-04-22T18:46:14.139975+0000 mon.a (mon.0) 5811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:46:16.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:15 smithi121 bash[36902]: cluster 2022-04-22T18:46:14.167171+0000 mgr.y (mgr.34107) 2777 : cluster [DBG] pgmap v4347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:15 smithi115 bash[45495]: cluster 2022-04-22T18:46:14.167171+0000 mgr.y (mgr.34107) 2777 : cluster [DBG] pgmap v4347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:15 smithi115 bash[46934]: cluster 2022-04-22T18:46:14.167171+0000 mgr.y (mgr.34107) 2777 : cluster [DBG] pgmap v4347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:17 smithi121 bash[36902]: cluster 2022-04-22T18:46:16.168286+0000 mgr.y (mgr.34107) 2778 : cluster [DBG] pgmap v4348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:17 smithi121 bash[36902]: audit 2022-04-22T18:46:16.960073+0000 mon.a (mon.0) 5812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:18.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:17 smithi121 bash[36902]: audit 2022-04-22T18:46:17.621384+0000 mon.a (mon.0) 5813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[46934]: cluster 2022-04-22T18:46:16.168286+0000 mgr.y (mgr.34107) 2778 : cluster [DBG] pgmap v4348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[46934]: audit 2022-04-22T18:46:16.960073+0000 mon.a (mon.0) 5812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[46934]: audit 2022-04-22T18:46:17.621384+0000 mon.a (mon.0) 5813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[45495]: cluster 2022-04-22T18:46:16.168286+0000 mgr.y (mgr.34107) 2778 : cluster [DBG] pgmap v4348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[45495]: audit 2022-04-22T18:46:16.960073+0000 mon.a (mon.0) 5812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:17 smithi115 bash[45495]: audit 2022-04-22T18:46:17.621384+0000 mon.a (mon.0) 5813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:18 smithi121 bash[36902]: audit 2022-04-22T18:46:18.780245+0000 mon.a (mon.0) 5814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:19.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:18 smithi115 bash[45495]: audit 2022-04-22T18:46:18.780245+0000 mon.a (mon.0) 5814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:18 smithi115 bash[46934]: audit 2022-04-22T18:46:18.780245+0000 mon.a (mon.0) 5814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:19 smithi121 bash[36902]: cluster 2022-04-22T18:46:18.169443+0000 mgr.y (mgr.34107) 2779 : cluster [DBG] pgmap v4349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:19 smithi115 bash[45495]: cluster 2022-04-22T18:46:18.169443+0000 mgr.y (mgr.34107) 2779 : cluster [DBG] pgmap v4349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:19 smithi115 bash[46934]: cluster 2022-04-22T18:46:18.169443+0000 mgr.y (mgr.34107) 2779 : cluster [DBG] pgmap v4349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:20 smithi121 bash[36902]: cluster 2022-04-22T18:46:20.170374+0000 mgr.y (mgr.34107) 2780 : cluster [DBG] pgmap v4350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:20 smithi115 bash[46934]: cluster 2022-04-22T18:46:20.170374+0000 mgr.y (mgr.34107) 2780 : cluster [DBG] pgmap v4350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:20 smithi115 bash[45495]: cluster 2022-04-22T18:46:20.170374+0000 mgr.y (mgr.34107) 2780 : cluster [DBG] pgmap v4350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:22.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:21.911Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:22 smithi121 bash[36902]: audit 2022-04-22T18:46:21.961331+0000 mon.a (mon.0) 5815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:22 smithi121 bash[36902]: audit 2022-04-22T18:46:22.624984+0000 mon.a (mon.0) 5816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:22 smithi115 bash[45495]: audit 2022-04-22T18:46:21.961331+0000 mon.a (mon.0) 5815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:22 smithi115 bash[45495]: audit 2022-04-22T18:46:22.624984+0000 mon.a (mon.0) 5816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:22 smithi115 bash[46934]: audit 2022-04-22T18:46:21.961331+0000 mon.a (mon.0) 5815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:22 smithi115 bash[46934]: audit 2022-04-22T18:46:22.624984+0000 mon.a (mon.0) 5816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:23 smithi121 bash[36902]: cluster 2022-04-22T18:46:22.171368+0000 mgr.y (mgr.34107) 2781 : cluster [DBG] pgmap v4351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:23 smithi121 bash[36902]: audit 2022-04-22T18:46:23.783201+0000 mon.a (mon.0) 5817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:23.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:23 smithi115 bash[45495]: cluster 2022-04-22T18:46:22.171368+0000 mgr.y (mgr.34107) 2781 : cluster [DBG] pgmap v4351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:23 smithi115 bash[45495]: audit 2022-04-22T18:46:23.783201+0000 mon.a (mon.0) 5817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:23 smithi115 bash[46934]: cluster 2022-04-22T18:46:22.171368+0000 mgr.y (mgr.34107) 2781 : cluster [DBG] pgmap v4351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:23 smithi115 bash[46934]: audit 2022-04-22T18:46:23.783201+0000 mon.a (mon.0) 5817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:25 smithi121 bash[36902]: cluster 2022-04-22T18:46:24.172367+0000 mgr.y (mgr.34107) 2782 : cluster [DBG] pgmap v4352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:25 smithi115 bash[45495]: cluster 2022-04-22T18:46:24.172367+0000 mgr.y (mgr.34107) 2782 : cluster [DBG] pgmap v4352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:25 smithi115 bash[46934]: cluster 2022-04-22T18:46:24.172367+0000 mgr.y (mgr.34107) 2782 : cluster [DBG] pgmap v4352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:27 smithi121 bash[36902]: cluster 2022-04-22T18:46:26.173555+0000 mgr.y (mgr.34107) 2783 : cluster [DBG] pgmap v4353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:27 smithi121 bash[36902]: audit 2022-04-22T18:46:26.964360+0000 mon.a (mon.0) 5818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:27 smithi121 bash[36902]: audit 2022-04-22T18:46:27.628777+0000 mon.a (mon.0) 5819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[45495]: cluster 2022-04-22T18:46:26.173555+0000 mgr.y (mgr.34107) 2783 : cluster [DBG] pgmap v4353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[45495]: audit 2022-04-22T18:46:26.964360+0000 mon.a (mon.0) 5818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[45495]: audit 2022-04-22T18:46:27.628777+0000 mon.a (mon.0) 5819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[46934]: cluster 2022-04-22T18:46:26.173555+0000 mgr.y (mgr.34107) 2783 : cluster [DBG] pgmap v4353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[46934]: audit 2022-04-22T18:46:26.964360+0000 mon.a (mon.0) 5818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:27 smithi115 bash[46934]: audit 2022-04-22T18:46:27.628777+0000 mon.a (mon.0) 5819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:28 smithi121 bash[36902]: audit 2022-04-22T18:46:28.786358+0000 mon.a (mon.0) 5820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:28 smithi115 bash[45495]: audit 2022-04-22T18:46:28.786358+0000 mon.a (mon.0) 5820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:28 smithi115 bash[46934]: audit 2022-04-22T18:46:28.786358+0000 mon.a (mon.0) 5820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:29 smithi121 bash[36902]: cluster 2022-04-22T18:46:28.174833+0000 mgr.y (mgr.34107) 2784 : cluster [DBG] pgmap v4354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:29 smithi115 bash[45495]: cluster 2022-04-22T18:46:28.174833+0000 mgr.y (mgr.34107) 2784 : cluster [DBG] pgmap v4354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:29 smithi115 bash[46934]: cluster 2022-04-22T18:46:28.174833+0000 mgr.y (mgr.34107) 2784 : cluster [DBG] pgmap v4354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:31 smithi115 bash[45495]: cluster 2022-04-22T18:46:30.175941+0000 mgr.y (mgr.34107) 2785 : cluster [DBG] pgmap v4355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:31 smithi115 bash[46934]: cluster 2022-04-22T18:46:30.175941+0000 mgr.y (mgr.34107) 2785 : cluster [DBG] pgmap v4355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:31 smithi121 bash[36902]: cluster 2022-04-22T18:46:30.175941+0000 mgr.y (mgr.34107) 2785 : cluster [DBG] pgmap v4355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:31.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:32.391 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:46:32.734 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:46:32.740 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (69m) 7m ago 83m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:46:32.740 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (83m) 8m ago 83m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (71m) 7m ago 84m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (71m) 8m ago 84m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (78m) 8m ago 88m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (79m) 7m ago 90m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (78m) 7m ago 91m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (77m) 8m ago 89m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (77m) 7m ago 89m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (70m) 7m ago 83m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:46:32.741 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (70m) 8m ago 83m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (76m) 7m ago 88m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (75m) 7m ago 87m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (75m) 7m ago 87m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (74m) 7m ago 86m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (74m) 8m ago 86m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (73m) 8m ago 85m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (72m) 8m ago 85m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (71m) 8m ago 84m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:46:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (70m) 8m ago 83m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:33.143 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:46:33.144 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:46:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:32 smithi121 bash[36902]: audit 2022-04-22T18:46:31.967009+0000 mon.a (mon.0) 5821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:32 smithi121 bash[36902]: audit 2022-04-22T18:46:32.632423+0000 mon.a (mon.0) 5822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:32 smithi115 bash[45495]: audit 2022-04-22T18:46:31.967009+0000 mon.a (mon.0) 5821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:32 smithi115 bash[45495]: audit 2022-04-22T18:46:32.632423+0000 mon.a (mon.0) 5822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:32 smithi115 bash[46934]: audit 2022-04-22T18:46:31.967009+0000 mon.a (mon.0) 5821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:32 smithi115 bash[46934]: audit 2022-04-22T18:46:32.632423+0000 mon.a (mon.0) 5822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:33 smithi121 bash[36902]: cluster 2022-04-22T18:46:32.177017+0000 mgr.y (mgr.34107) 2786 : cluster [DBG] pgmap v4356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:33 smithi121 bash[36902]: audit 2022-04-22T18:46:32.365371+0000 mgr.y (mgr.34107) 2787 : audit [DBG] from='client.48274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:33 smithi121 bash[36902]: audit 2022-04-22T18:46:32.704639+0000 mgr.y (mgr.34107) 2788 : audit [DBG] from='client.38346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:33 smithi121 bash[36902]: audit 2022-04-22T18:46:33.133925+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.115:0/439041495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:33 smithi121 bash[36902]: audit 2022-04-22T18:46:33.789382+0000 mon.a (mon.0) 5823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:33.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[46934]: cluster 2022-04-22T18:46:32.177017+0000 mgr.y (mgr.34107) 2786 : cluster [DBG] pgmap v4356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[46934]: audit 2022-04-22T18:46:32.365371+0000 mgr.y (mgr.34107) 2787 : audit [DBG] from='client.48274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[46934]: audit 2022-04-22T18:46:32.704639+0000 mgr.y (mgr.34107) 2788 : audit [DBG] from='client.38346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[46934]: audit 2022-04-22T18:46:33.133925+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.115:0/439041495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[46934]: audit 2022-04-22T18:46:33.789382+0000 mon.a (mon.0) 5823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[45495]: cluster 2022-04-22T18:46:32.177017+0000 mgr.y (mgr.34107) 2786 : cluster [DBG] pgmap v4356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[45495]: audit 2022-04-22T18:46:32.365371+0000 mgr.y (mgr.34107) 2787 : audit [DBG] from='client.48274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[45495]: audit 2022-04-22T18:46:32.704639+0000 mgr.y (mgr.34107) 2788 : audit [DBG] from='client.38346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[45495]: audit 2022-04-22T18:46:33.133925+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.115:0/439041495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:46:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:33 smithi115 bash[45495]: audit 2022-04-22T18:46:33.789382+0000 mon.a (mon.0) 5823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:35 smithi115 bash[45495]: cluster 2022-04-22T18:46:34.178079+0000 mgr.y (mgr.34107) 2789 : cluster [DBG] pgmap v4357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:35 smithi115 bash[46934]: cluster 2022-04-22T18:46:34.178079+0000 mgr.y (mgr.34107) 2789 : cluster [DBG] pgmap v4357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:35 smithi121 bash[36902]: cluster 2022-04-22T18:46:34.178079+0000 mgr.y (mgr.34107) 2789 : cluster [DBG] pgmap v4357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: cluster 2022-04-22T18:46:36.179076+0000 mgr.y (mgr.34107) 2790 : cluster [DBG] pgmap v4358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:36.969626+0000 mon.a (mon.0) 5824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:37.185002+0000 mon.a (mon.0) 5825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:37.187791+0000 mon.a (mon.0) 5826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:37.189476+0000 mon.a (mon.0) 5827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:37.533198+0000 mon.a (mon.0) 5828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:46:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:37 smithi121 bash[36902]: audit 2022-04-22T18:46:37.635660+0000 mon.a (mon.0) 5829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: cluster 2022-04-22T18:46:36.179076+0000 mgr.y (mgr.34107) 2790 : cluster [DBG] pgmap v4358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:36.969626+0000 mon.a (mon.0) 5824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:37.185002+0000 mon.a (mon.0) 5825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:37.187791+0000 mon.a (mon.0) 5826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:37.189476+0000 mon.a (mon.0) 5827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:37.533198+0000 mon.a (mon.0) 5828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[45495]: audit 2022-04-22T18:46:37.635660+0000 mon.a (mon.0) 5829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: cluster 2022-04-22T18:46:36.179076+0000 mgr.y (mgr.34107) 2790 : cluster [DBG] pgmap v4358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:36.969626+0000 mon.a (mon.0) 5824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:37.185002+0000 mon.a (mon.0) 5825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:46:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:37.187791+0000 mon.a (mon.0) 5826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:46:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:37.189476+0000 mon.a (mon.0) 5827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:46:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:37.533198+0000 mon.a (mon.0) 5828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:46:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:37 smithi115 bash[46934]: audit 2022-04-22T18:46:37.635660+0000 mon.a (mon.0) 5829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:38 smithi121 bash[36902]: audit 2022-04-22T18:46:38.791932+0000 mon.a (mon.0) 5830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:38 smithi115 bash[45495]: audit 2022-04-22T18:46:38.791932+0000 mon.a (mon.0) 5830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:38 smithi115 bash[46934]: audit 2022-04-22T18:46:38.791932+0000 mon.a (mon.0) 5830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:39 smithi121 bash[36902]: cluster 2022-04-22T18:46:38.180206+0000 mgr.y (mgr.34107) 2791 : cluster [DBG] pgmap v4359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:39 smithi115 bash[45495]: cluster 2022-04-22T18:46:38.180206+0000 mgr.y (mgr.34107) 2791 : cluster [DBG] pgmap v4359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:39 smithi115 bash[46934]: cluster 2022-04-22T18:46:38.180206+0000 mgr.y (mgr.34107) 2791 : cluster [DBG] pgmap v4359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:40 smithi121 bash[36902]: cluster 2022-04-22T18:46:40.181276+0000 mgr.y (mgr.34107) 2792 : cluster [DBG] pgmap v4360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:40 smithi115 bash[45495]: cluster 2022-04-22T18:46:40.181276+0000 mgr.y (mgr.34107) 2792 : cluster [DBG] pgmap v4360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:40 smithi115 bash[46934]: cluster 2022-04-22T18:46:40.181276+0000 mgr.y (mgr.34107) 2792 : cluster [DBG] pgmap v4360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:41.911Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:42 smithi121 bash[36902]: audit 2022-04-22T18:46:41.972088+0000 mon.a (mon.0) 5831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:42 smithi121 bash[36902]: audit 2022-04-22T18:46:42.639082+0000 mon.a (mon.0) 5832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:42 smithi115 bash[45495]: audit 2022-04-22T18:46:41.972088+0000 mon.a (mon.0) 5831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:42 smithi115 bash[45495]: audit 2022-04-22T18:46:42.639082+0000 mon.a (mon.0) 5832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:42 smithi115 bash[46934]: audit 2022-04-22T18:46:41.972088+0000 mon.a (mon.0) 5831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:42 smithi115 bash[46934]: audit 2022-04-22T18:46:42.639082+0000 mon.a (mon.0) 5832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:43 smithi121 bash[36902]: cluster 2022-04-22T18:46:42.182354+0000 mgr.y (mgr.34107) 2793 : cluster [DBG] pgmap v4361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:43 smithi121 bash[36902]: audit 2022-04-22T18:46:43.795266+0000 mon.a (mon.0) 5833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:43 smithi115 bash[45495]: cluster 2022-04-22T18:46:42.182354+0000 mgr.y (mgr.34107) 2793 : cluster [DBG] pgmap v4361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:43 smithi115 bash[45495]: audit 2022-04-22T18:46:43.795266+0000 mon.a (mon.0) 5833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:43.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:43 smithi115 bash[46934]: cluster 2022-04-22T18:46:42.182354+0000 mgr.y (mgr.34107) 2793 : cluster [DBG] pgmap v4361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:43 smithi115 bash[46934]: audit 2022-04-22T18:46:43.795266+0000 mon.a (mon.0) 5833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:44 smithi121 bash[36902]: cluster 2022-04-22T18:46:44.183537+0000 mgr.y (mgr.34107) 2794 : cluster [DBG] pgmap v4362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:44 smithi115 bash[45495]: cluster 2022-04-22T18:46:44.183537+0000 mgr.y (mgr.34107) 2794 : cluster [DBG] pgmap v4362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:44 smithi115 bash[46934]: cluster 2022-04-22T18:46:44.183537+0000 mgr.y (mgr.34107) 2794 : cluster [DBG] pgmap v4362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:47 smithi121 bash[36902]: cluster 2022-04-22T18:46:46.184588+0000 mgr.y (mgr.34107) 2795 : cluster [DBG] pgmap v4363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:47 smithi121 bash[36902]: audit 2022-04-22T18:46:46.974285+0000 mon.a (mon.0) 5834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:47 smithi121 bash[36902]: audit 2022-04-22T18:46:47.642738+0000 mon.a (mon.0) 5835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[45495]: cluster 2022-04-22T18:46:46.184588+0000 mgr.y (mgr.34107) 2795 : cluster [DBG] pgmap v4363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[45495]: audit 2022-04-22T18:46:46.974285+0000 mon.a (mon.0) 5834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[45495]: audit 2022-04-22T18:46:47.642738+0000 mon.a (mon.0) 5835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[46934]: cluster 2022-04-22T18:46:46.184588+0000 mgr.y (mgr.34107) 2795 : cluster [DBG] pgmap v4363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[46934]: audit 2022-04-22T18:46:46.974285+0000 mon.a (mon.0) 5834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:47 smithi115 bash[46934]: audit 2022-04-22T18:46:47.642738+0000 mon.a (mon.0) 5835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:48 smithi121 bash[36902]: audit 2022-04-22T18:46:48.798087+0000 mon.a (mon.0) 5836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:48 smithi115 bash[45495]: audit 2022-04-22T18:46:48.798087+0000 mon.a (mon.0) 5836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:48 smithi115 bash[46934]: audit 2022-04-22T18:46:48.798087+0000 mon.a (mon.0) 5836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:49 smithi121 bash[36902]: cluster 2022-04-22T18:46:48.185774+0000 mgr.y (mgr.34107) 2796 : cluster [DBG] pgmap v4364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:49 smithi115 bash[45495]: cluster 2022-04-22T18:46:48.185774+0000 mgr.y (mgr.34107) 2796 : cluster [DBG] pgmap v4364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:49 smithi115 bash[46934]: cluster 2022-04-22T18:46:48.185774+0000 mgr.y (mgr.34107) 2796 : cluster [DBG] pgmap v4364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:50 smithi121 bash[36902]: cluster 2022-04-22T18:46:50.186788+0000 mgr.y (mgr.34107) 2797 : cluster [DBG] pgmap v4365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:50 smithi115 bash[45495]: cluster 2022-04-22T18:46:50.186788+0000 mgr.y (mgr.34107) 2797 : cluster [DBG] pgmap v4365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:50 smithi115 bash[46934]: cluster 2022-04-22T18:46:50.186788+0000 mgr.y (mgr.34107) 2797 : cluster [DBG] pgmap v4365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:51.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:52 smithi121 bash[36902]: audit 2022-04-22T18:46:51.976797+0000 mon.a (mon.0) 5837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:52 smithi121 bash[36902]: audit 2022-04-22T18:46:52.646384+0000 mon.a (mon.0) 5838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:52 smithi115 bash[45495]: audit 2022-04-22T18:46:51.976797+0000 mon.a (mon.0) 5837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:52 smithi115 bash[45495]: audit 2022-04-22T18:46:52.646384+0000 mon.a (mon.0) 5838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:52 smithi115 bash[46934]: audit 2022-04-22T18:46:51.976797+0000 mon.a (mon.0) 5837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:52 smithi115 bash[46934]: audit 2022-04-22T18:46:52.646384+0000 mon.a (mon.0) 5838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:53 smithi121 bash[36902]: cluster 2022-04-22T18:46:52.187864+0000 mgr.y (mgr.34107) 2798 : cluster [DBG] pgmap v4366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:53 smithi121 bash[36902]: audit 2022-04-22T18:46:53.801136+0000 mon.a (mon.0) 5839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:46:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:46:53.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:46:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:53 smithi115 bash[45495]: cluster 2022-04-22T18:46:52.187864+0000 mgr.y (mgr.34107) 2798 : cluster [DBG] pgmap v4366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:53 smithi115 bash[45495]: audit 2022-04-22T18:46:53.801136+0000 mon.a (mon.0) 5839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:53 smithi115 bash[46934]: cluster 2022-04-22T18:46:52.187864+0000 mgr.y (mgr.34107) 2798 : cluster [DBG] pgmap v4366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:53 smithi115 bash[46934]: audit 2022-04-22T18:46:53.801136+0000 mon.a (mon.0) 5839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:54 smithi121 bash[36902]: cluster 2022-04-22T18:46:54.188929+0000 mgr.y (mgr.34107) 2799 : cluster [DBG] pgmap v4367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:54 smithi115 bash[45495]: cluster 2022-04-22T18:46:54.188929+0000 mgr.y (mgr.34107) 2799 : cluster [DBG] pgmap v4367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:54 smithi115 bash[46934]: cluster 2022-04-22T18:46:54.188929+0000 mgr.y (mgr.34107) 2799 : cluster [DBG] pgmap v4367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:57 smithi121 bash[36902]: cluster 2022-04-22T18:46:56.190185+0000 mgr.y (mgr.34107) 2800 : cluster [DBG] pgmap v4368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:57 smithi121 bash[36902]: audit 2022-04-22T18:46:56.979277+0000 mon.a (mon.0) 5840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:58.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:57 smithi121 bash[36902]: audit 2022-04-22T18:46:57.649829+0000 mon.a (mon.0) 5841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[46934]: cluster 2022-04-22T18:46:56.190185+0000 mgr.y (mgr.34107) 2800 : cluster [DBG] pgmap v4368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[46934]: audit 2022-04-22T18:46:56.979277+0000 mon.a (mon.0) 5840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[46934]: audit 2022-04-22T18:46:57.649829+0000 mon.a (mon.0) 5841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[45495]: cluster 2022-04-22T18:46:56.190185+0000 mgr.y (mgr.34107) 2800 : cluster [DBG] pgmap v4368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:46:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[45495]: audit 2022-04-22T18:46:56.979277+0000 mon.a (mon.0) 5840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:46:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:57 smithi115 bash[45495]: audit 2022-04-22T18:46:57.649829+0000 mon.a (mon.0) 5841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:46:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:58 smithi121 bash[36902]: audit 2022-04-22T18:46:58.804143+0000 mon.a (mon.0) 5842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:58 smithi115 bash[45495]: audit 2022-04-22T18:46:58.804143+0000 mon.a (mon.0) 5842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:46:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:58 smithi115 bash[46934]: audit 2022-04-22T18:46:58.804143+0000 mon.a (mon.0) 5842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:46:59 smithi121 bash[36902]: cluster 2022-04-22T18:46:58.191300+0000 mgr.y (mgr.34107) 2801 : cluster [DBG] pgmap v4369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:46:59 smithi115 bash[45495]: cluster 2022-04-22T18:46:58.191300+0000 mgr.y (mgr.34107) 2801 : cluster [DBG] pgmap v4369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:46:59 smithi115 bash[46934]: cluster 2022-04-22T18:46:58.191300+0000 mgr.y (mgr.34107) 2801 : cluster [DBG] pgmap v4369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:01 smithi121 bash[36902]: cluster 2022-04-22T18:47:00.192404+0000 mgr.y (mgr.34107) 2802 : cluster [DBG] pgmap v4370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:00 smithi115 bash[45495]: cluster 2022-04-22T18:47:00.192404+0000 mgr.y (mgr.34107) 2802 : cluster [DBG] pgmap v4370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:00 smithi115 bash[46934]: cluster 2022-04-22T18:47:00.192404+0000 mgr.y (mgr.34107) 2802 : cluster [DBG] pgmap v4370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:01.913Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:03.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:02 smithi121 bash[36902]: audit 2022-04-22T18:47:01.981629+0000 mon.a (mon.0) 5843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:03.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:02 smithi121 bash[36902]: audit 2022-04-22T18:47:02.653383+0000 mon.a (mon.0) 5844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:02 smithi115 bash[46934]: audit 2022-04-22T18:47:01.981629+0000 mon.a (mon.0) 5843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:02 smithi115 bash[46934]: audit 2022-04-22T18:47:02.653383+0000 mon.a (mon.0) 5844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:02 smithi115 bash[45495]: audit 2022-04-22T18:47:01.981629+0000 mon.a (mon.0) 5843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:02 smithi115 bash[45495]: audit 2022-04-22T18:47:02.653383+0000 mon.a (mon.0) 5844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:03.518 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:47:03.861 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:47:03.861 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (70m) 7m ago 83m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:47:03.861 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (83m) 8m ago 83m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:47:03.862 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (72m) 7m ago 84m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:47:03.862 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (71m) 8m ago 84m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:47:03.863 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (79m) 8m ago 89m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:47:03.863 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (80m) 7m ago 91m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:47:03.863 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (79m) 7m ago 91m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:47:03.863 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (78m) 8m ago 89m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:47:03.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (78m) 7m ago 89m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:47:03.864 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (71m) 7m ago 84m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:47:03.864 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (71m) 8m ago 84m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:47:03.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (76m) 7m ago 88m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:47:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (76m) 7m ago 88m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:47:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (75m) 7m ago 87m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:47:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (75m) 7m ago 87m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:47:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (74m) 8m ago 86m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:47:03.867 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (73m) 8m ago 86m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:47:03.867 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (73m) 8m ago 85m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:47:03.867 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (72m) 8m ago 85m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:47:03.867 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (70m) 8m ago 84m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:47:04.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:03.787Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[45495]: cluster 2022-04-22T18:47:02.193506+0000 mgr.y (mgr.34107) 2803 : cluster [DBG] pgmap v4371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[45495]: audit 2022-04-22T18:47:03.494349+0000 mgr.y (mgr.34107) 2804 : audit [DBG] from='client.38358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[45495]: audit 2022-04-22T18:47:03.807331+0000 mon.a (mon.0) 5845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[45495]: audit 2022-04-22T18:47:03.833177+0000 mgr.y (mgr.34107) 2805 : audit [DBG] from='client.48298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[46934]: cluster 2022-04-22T18:47:02.193506+0000 mgr.y (mgr.34107) 2803 : cluster [DBG] pgmap v4371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[46934]: audit 2022-04-22T18:47:03.494349+0000 mgr.y (mgr.34107) 2804 : audit [DBG] from='client.38358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[46934]: audit 2022-04-22T18:47:03.807331+0000 mon.a (mon.0) 5845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:03 smithi115 bash[46934]: audit 2022-04-22T18:47:03.833177+0000 mgr.y (mgr.34107) 2805 : audit [DBG] from='client.48298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:03 smithi121 bash[36902]: cluster 2022-04-22T18:47:02.193506+0000 mgr.y (mgr.34107) 2803 : cluster [DBG] pgmap v4371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:03 smithi121 bash[36902]: audit 2022-04-22T18:47:03.494349+0000 mgr.y (mgr.34107) 2804 : audit [DBG] from='client.38358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:03 smithi121 bash[36902]: audit 2022-04-22T18:47:03.807331+0000 mon.a (mon.0) 5845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:03 smithi121 bash[36902]: audit 2022-04-22T18:47:03.833177+0000 mgr.y (mgr.34107) 2805 : audit [DBG] from='client.48298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:04.276 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:47:04.323 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:47:04.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:47:04.328 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:47:04.329 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:47:04.329 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:47:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:05 smithi121 bash[36902]: cluster 2022-04-22T18:47:04.195140+0000 mgr.y (mgr.34107) 2806 : cluster [DBG] pgmap v4372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:05 smithi121 bash[36902]: audit 2022-04-22T18:47:04.266867+0000 mon.a (mon.0) 5846 : audit [DBG] from='client.? 172.21.15.115:0/57066546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:04 smithi115 bash[45495]: cluster 2022-04-22T18:47:04.195140+0000 mgr.y (mgr.34107) 2806 : cluster [DBG] pgmap v4372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:04 smithi115 bash[45495]: audit 2022-04-22T18:47:04.266867+0000 mon.a (mon.0) 5846 : audit [DBG] from='client.? 172.21.15.115:0/57066546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:04 smithi115 bash[46934]: cluster 2022-04-22T18:47:04.195140+0000 mgr.y (mgr.34107) 2806 : cluster [DBG] pgmap v4372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:04 smithi115 bash[46934]: audit 2022-04-22T18:47:04.266867+0000 mon.a (mon.0) 5846 : audit [DBG] from='client.? 172.21.15.115:0/57066546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:07 smithi121 bash[36902]: cluster 2022-04-22T18:47:06.196215+0000 mgr.y (mgr.34107) 2807 : cluster [DBG] pgmap v4373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:07 smithi121 bash[36902]: audit 2022-04-22T18:47:06.984202+0000 mon.a (mon.0) 5847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:07 smithi121 bash[36902]: audit 2022-04-22T18:47:07.656904+0000 mon.a (mon.0) 5848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[45495]: cluster 2022-04-22T18:47:06.196215+0000 mgr.y (mgr.34107) 2807 : cluster [DBG] pgmap v4373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[45495]: audit 2022-04-22T18:47:06.984202+0000 mon.a (mon.0) 5847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[45495]: audit 2022-04-22T18:47:07.656904+0000 mon.a (mon.0) 5848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[46934]: cluster 2022-04-22T18:47:06.196215+0000 mgr.y (mgr.34107) 2807 : cluster [DBG] pgmap v4373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[46934]: audit 2022-04-22T18:47:06.984202+0000 mon.a (mon.0) 5847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:07 smithi115 bash[46934]: audit 2022-04-22T18:47:07.656904+0000 mon.a (mon.0) 5848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:08 smithi121 bash[36902]: audit 2022-04-22T18:47:08.810105+0000 mon.a (mon.0) 5849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:08 smithi115 bash[45495]: audit 2022-04-22T18:47:08.810105+0000 mon.a (mon.0) 5849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:08 smithi115 bash[46934]: audit 2022-04-22T18:47:08.810105+0000 mon.a (mon.0) 5849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:10.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:09 smithi121 bash[36902]: cluster 2022-04-22T18:47:08.197382+0000 mgr.y (mgr.34107) 2808 : cluster [DBG] pgmap v4374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:09 smithi115 bash[45495]: cluster 2022-04-22T18:47:08.197382+0000 mgr.y (mgr.34107) 2808 : cluster [DBG] pgmap v4374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:09 smithi115 bash[46934]: cluster 2022-04-22T18:47:08.197382+0000 mgr.y (mgr.34107) 2808 : cluster [DBG] pgmap v4374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:11 smithi121 bash[36902]: cluster 2022-04-22T18:47:10.198457+0000 mgr.y (mgr.34107) 2809 : cluster [DBG] pgmap v4375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:10 smithi115 bash[45495]: cluster 2022-04-22T18:47:10.198457+0000 mgr.y (mgr.34107) 2809 : cluster [DBG] pgmap v4375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:10 smithi115 bash[46934]: cluster 2022-04-22T18:47:10.198457+0000 mgr.y (mgr.34107) 2809 : cluster [DBG] pgmap v4375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:11.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:12 smithi121 bash[36902]: audit 2022-04-22T18:47:11.986557+0000 mon.a (mon.0) 5850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:12 smithi121 bash[36902]: audit 2022-04-22T18:47:12.660733+0000 mon.a (mon.0) 5851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:12 smithi115 bash[46934]: audit 2022-04-22T18:47:11.986557+0000 mon.a (mon.0) 5850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:12 smithi115 bash[46934]: audit 2022-04-22T18:47:12.660733+0000 mon.a (mon.0) 5851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:12 smithi115 bash[45495]: audit 2022-04-22T18:47:11.986557+0000 mon.a (mon.0) 5850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:12 smithi115 bash[45495]: audit 2022-04-22T18:47:12.660733+0000 mon.a (mon.0) 5851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:13 smithi121 bash[36902]: cluster 2022-04-22T18:47:12.199171+0000 mgr.y (mgr.34107) 2810 : cluster [DBG] pgmap v4376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:13 smithi121 bash[36902]: audit 2022-04-22T18:47:13.813235+0000 mon.a (mon.0) 5852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:13.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:14.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:13 smithi115 bash[45495]: cluster 2022-04-22T18:47:12.199171+0000 mgr.y (mgr.34107) 2810 : cluster [DBG] pgmap v4376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:14.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:13 smithi115 bash[45495]: audit 2022-04-22T18:47:13.813235+0000 mon.a (mon.0) 5852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:14.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:13 smithi115 bash[46934]: cluster 2022-04-22T18:47:12.199171+0000 mgr.y (mgr.34107) 2810 : cluster [DBG] pgmap v4376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:14.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:13 smithi115 bash[46934]: audit 2022-04-22T18:47:13.813235+0000 mon.a (mon.0) 5852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:15 smithi121 bash[36902]: audit 2022-04-22T18:47:14.152767+0000 mon.a (mon.0) 5853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:47:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:15 smithi121 bash[36902]: audit 2022-04-22T18:47:14.153904+0000 mon.a (mon.0) 5854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:47:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:15 smithi121 bash[36902]: cluster 2022-04-22T18:47:14.200204+0000 mgr.y (mgr.34107) 2811 : cluster [DBG] pgmap v4377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[46934]: audit 2022-04-22T18:47:14.152767+0000 mon.a (mon.0) 5853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:47:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[46934]: audit 2022-04-22T18:47:14.153904+0000 mon.a (mon.0) 5854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:47:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[46934]: cluster 2022-04-22T18:47:14.200204+0000 mgr.y (mgr.34107) 2811 : cluster [DBG] pgmap v4377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[45495]: audit 2022-04-22T18:47:14.152767+0000 mon.a (mon.0) 5853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:47:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[45495]: audit 2022-04-22T18:47:14.153904+0000 mon.a (mon.0) 5854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:47:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:14 smithi115 bash[45495]: cluster 2022-04-22T18:47:14.200204+0000 mgr.y (mgr.34107) 2811 : cluster [DBG] pgmap v4377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:17 smithi121 bash[36902]: cluster 2022-04-22T18:47:16.201212+0000 mgr.y (mgr.34107) 2812 : cluster [DBG] pgmap v4378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:17 smithi121 bash[36902]: audit 2022-04-22T18:47:16.988970+0000 mon.a (mon.0) 5855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:17 smithi121 bash[36902]: audit 2022-04-22T18:47:17.663948+0000 mon.a (mon.0) 5856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[46934]: cluster 2022-04-22T18:47:16.201212+0000 mgr.y (mgr.34107) 2812 : cluster [DBG] pgmap v4378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[46934]: audit 2022-04-22T18:47:16.988970+0000 mon.a (mon.0) 5855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[46934]: audit 2022-04-22T18:47:17.663948+0000 mon.a (mon.0) 5856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[45495]: cluster 2022-04-22T18:47:16.201212+0000 mgr.y (mgr.34107) 2812 : cluster [DBG] pgmap v4378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[45495]: audit 2022-04-22T18:47:16.988970+0000 mon.a (mon.0) 5855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:17 smithi115 bash[45495]: audit 2022-04-22T18:47:17.663948+0000 mon.a (mon.0) 5856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:19.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:18 smithi121 bash[36902]: audit 2022-04-22T18:47:18.816181+0000 mon.a (mon.0) 5857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:18 smithi115 bash[45495]: audit 2022-04-22T18:47:18.816181+0000 mon.a (mon.0) 5857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:18 smithi115 bash[46934]: audit 2022-04-22T18:47:18.816181+0000 mon.a (mon.0) 5857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:19 smithi121 bash[36902]: cluster 2022-04-22T18:47:18.202366+0000 mgr.y (mgr.34107) 2813 : cluster [DBG] pgmap v4379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:19 smithi115 bash[45495]: cluster 2022-04-22T18:47:18.202366+0000 mgr.y (mgr.34107) 2813 : cluster [DBG] pgmap v4379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:19 smithi115 bash[46934]: cluster 2022-04-22T18:47:18.202366+0000 mgr.y (mgr.34107) 2813 : cluster [DBG] pgmap v4379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:21 smithi121 bash[36902]: cluster 2022-04-22T18:47:20.203227+0000 mgr.y (mgr.34107) 2814 : cluster [DBG] pgmap v4380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:20 smithi115 bash[45495]: cluster 2022-04-22T18:47:20.203227+0000 mgr.y (mgr.34107) 2814 : cluster [DBG] pgmap v4380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:20 smithi115 bash[46934]: cluster 2022-04-22T18:47:20.203227+0000 mgr.y (mgr.34107) 2814 : cluster [DBG] pgmap v4380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:21.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:22 smithi121 bash[36902]: audit 2022-04-22T18:47:21.991620+0000 mon.a (mon.0) 5858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:22 smithi121 bash[36902]: audit 2022-04-22T18:47:22.667749+0000 mon.a (mon.0) 5859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:22 smithi115 bash[45495]: audit 2022-04-22T18:47:21.991620+0000 mon.a (mon.0) 5858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:22 smithi115 bash[45495]: audit 2022-04-22T18:47:22.667749+0000 mon.a (mon.0) 5859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:22 smithi115 bash[46934]: audit 2022-04-22T18:47:21.991620+0000 mon.a (mon.0) 5858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:22 smithi115 bash[46934]: audit 2022-04-22T18:47:22.667749+0000 mon.a (mon.0) 5859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:23 smithi121 bash[36902]: cluster 2022-04-22T18:47:22.204239+0000 mgr.y (mgr.34107) 2815 : cluster [DBG] pgmap v4381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:23 smithi121 bash[36902]: audit 2022-04-22T18:47:23.819161+0000 mon.a (mon.0) 5860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:23.791Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:23 smithi115 bash[45495]: cluster 2022-04-22T18:47:22.204239+0000 mgr.y (mgr.34107) 2815 : cluster [DBG] pgmap v4381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:23 smithi115 bash[45495]: audit 2022-04-22T18:47:23.819161+0000 mon.a (mon.0) 5860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:23 smithi115 bash[46934]: cluster 2022-04-22T18:47:22.204239+0000 mgr.y (mgr.34107) 2815 : cluster [DBG] pgmap v4381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:23 smithi115 bash[46934]: audit 2022-04-22T18:47:23.819161+0000 mon.a (mon.0) 5860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:25 smithi121 bash[36902]: cluster 2022-04-22T18:47:24.205260+0000 mgr.y (mgr.34107) 2816 : cluster [DBG] pgmap v4382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:24 smithi115 bash[45495]: cluster 2022-04-22T18:47:24.205260+0000 mgr.y (mgr.34107) 2816 : cluster [DBG] pgmap v4382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:24 smithi115 bash[46934]: cluster 2022-04-22T18:47:24.205260+0000 mgr.y (mgr.34107) 2816 : cluster [DBG] pgmap v4382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:27 smithi121 bash[36902]: cluster 2022-04-22T18:47:26.206339+0000 mgr.y (mgr.34107) 2817 : cluster [DBG] pgmap v4383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:27 smithi121 bash[36902]: audit 2022-04-22T18:47:26.994544+0000 mon.a (mon.0) 5861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:27 smithi121 bash[36902]: audit 2022-04-22T18:47:27.671318+0000 mon.a (mon.0) 5862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[46934]: cluster 2022-04-22T18:47:26.206339+0000 mgr.y (mgr.34107) 2817 : cluster [DBG] pgmap v4383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[46934]: audit 2022-04-22T18:47:26.994544+0000 mon.a (mon.0) 5861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[46934]: audit 2022-04-22T18:47:27.671318+0000 mon.a (mon.0) 5862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[45495]: cluster 2022-04-22T18:47:26.206339+0000 mgr.y (mgr.34107) 2817 : cluster [DBG] pgmap v4383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[45495]: audit 2022-04-22T18:47:26.994544+0000 mon.a (mon.0) 5861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:27 smithi115 bash[45495]: audit 2022-04-22T18:47:27.671318+0000 mon.a (mon.0) 5862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:28 smithi121 bash[36902]: audit 2022-04-22T18:47:28.822199+0000 mon.a (mon.0) 5863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:28 smithi115 bash[46934]: audit 2022-04-22T18:47:28.822199+0000 mon.a (mon.0) 5863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:28 smithi115 bash[45495]: audit 2022-04-22T18:47:28.822199+0000 mon.a (mon.0) 5863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:29 smithi121 bash[36902]: cluster 2022-04-22T18:47:28.207223+0000 mgr.y (mgr.34107) 2818 : cluster [DBG] pgmap v4384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:29 smithi115 bash[45495]: cluster 2022-04-22T18:47:28.207223+0000 mgr.y (mgr.34107) 2818 : cluster [DBG] pgmap v4384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:29 smithi115 bash[46934]: cluster 2022-04-22T18:47:28.207223+0000 mgr.y (mgr.34107) 2818 : cluster [DBG] pgmap v4384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:31 smithi121 bash[36902]: cluster 2022-04-22T18:47:30.208209+0000 mgr.y (mgr.34107) 2819 : cluster [DBG] pgmap v4385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:30 smithi115 bash[45495]: cluster 2022-04-22T18:47:30.208209+0000 mgr.y (mgr.34107) 2819 : cluster [DBG] pgmap v4385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:30 smithi115 bash[46934]: cluster 2022-04-22T18:47:30.208209+0000 mgr.y (mgr.34107) 2819 : cluster [DBG] pgmap v4385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:31.918Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:32 smithi121 bash[36902]: audit 2022-04-22T18:47:31.997294+0000 mon.a (mon.0) 5864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:32 smithi121 bash[36902]: audit 2022-04-22T18:47:32.674775+0000 mon.a (mon.0) 5865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:32 smithi115 bash[45495]: audit 2022-04-22T18:47:31.997294+0000 mon.a (mon.0) 5864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:32 smithi115 bash[45495]: audit 2022-04-22T18:47:32.674775+0000 mon.a (mon.0) 5865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:32 smithi115 bash[46934]: audit 2022-04-22T18:47:31.997294+0000 mon.a (mon.0) 5864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:32 smithi115 bash[46934]: audit 2022-04-22T18:47:32.674775+0000 mon.a (mon.0) 5865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:33 smithi121 bash[36902]: cluster 2022-04-22T18:47:32.209237+0000 mgr.y (mgr.34107) 2820 : cluster [DBG] pgmap v4386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:33 smithi121 bash[36902]: audit 2022-04-22T18:47:33.825150+0000 mon.a (mon.0) 5866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:33.792Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:33 smithi115 bash[45495]: cluster 2022-04-22T18:47:32.209237+0000 mgr.y (mgr.34107) 2820 : cluster [DBG] pgmap v4386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:33 smithi115 bash[45495]: audit 2022-04-22T18:47:33.825150+0000 mon.a (mon.0) 5866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:33 smithi115 bash[46934]: cluster 2022-04-22T18:47:32.209237+0000 mgr.y (mgr.34107) 2820 : cluster [DBG] pgmap v4386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:33 smithi115 bash[46934]: audit 2022-04-22T18:47:33.825150+0000 mon.a (mon.0) 5866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:34.651 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:47:35.002 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:47:35.002 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (70m) 8m ago 84m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:47:35.002 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (84m) 9m ago 84m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:47:35.002 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (72m) 8m ago 85m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (72m) 9m ago 85m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (79m) 9m ago 89m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (80m) 8m ago 91m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (79m) 8m ago 92m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (78m) 9m ago 90m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (78m) 8m ago 90m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (71m) 8m ago 84m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (71m) 9m ago 84m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (77m) 8m ago 89m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (76m) 8m ago 88m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:47:35.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (76m) 8m ago 88m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (75m) 8m ago 87m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (75m) 9m ago 87m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (74m) 9m ago 86m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (73m) 9m ago 86m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (73m) 9m ago 86m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:47:35.004 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (71m) 9m ago 84m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:47:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:35 smithi121 bash[36902]: cluster 2022-04-22T18:47:34.210362+0000 mgr.y (mgr.34107) 2821 : cluster [DBG] pgmap v4387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:35 smithi121 bash[36902]: audit 2022-04-22T18:47:34.627028+0000 mgr.y (mgr.34107) 2822 : audit [DBG] from='client.38376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:35 smithi121 bash[36902]: audit 2022-04-22T18:47:34.974076+0000 mgr.y (mgr.34107) 2823 : audit [DBG] from='client.38382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[46934]: cluster 2022-04-22T18:47:34.210362+0000 mgr.y (mgr.34107) 2821 : cluster [DBG] pgmap v4387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[46934]: audit 2022-04-22T18:47:34.627028+0000 mgr.y (mgr.34107) 2822 : audit [DBG] from='client.38376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[46934]: audit 2022-04-22T18:47:34.974076+0000 mgr.y (mgr.34107) 2823 : audit [DBG] from='client.38382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[45495]: cluster 2022-04-22T18:47:34.210362+0000 mgr.y (mgr.34107) 2821 : cluster [DBG] pgmap v4387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:35.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[45495]: audit 2022-04-22T18:47:34.627028+0000 mgr.y (mgr.34107) 2822 : audit [DBG] from='client.38376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:34 smithi115 bash[45495]: audit 2022-04-22T18:47:34.974076+0000 mgr.y (mgr.34107) 2823 : audit [DBG] from='client.38382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:47:35.406 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:47:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:47:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:47:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:47:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:47:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:47:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:47:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:47:35.408 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:47:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:36 smithi121 bash[36902]: audit 2022-04-22T18:47:35.397313+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.115:0/3706100975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:35 smithi115 bash[45495]: audit 2022-04-22T18:47:35.397313+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.115:0/3706100975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:35 smithi115 bash[46934]: audit 2022-04-22T18:47:35.397313+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.115:0/3706100975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:47:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: cluster 2022-04-22T18:47:36.211453+0000 mgr.y (mgr.34107) 2824 : cluster [DBG] pgmap v4388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.004518+0000 mon.a (mon.0) 5867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.541643+0000 mon.a (mon.0) 5868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.544830+0000 mon.a (mon.0) 5869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.546457+0000 mon.a (mon.0) 5870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.678131+0000 mon.a (mon.0) 5871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:37 smithi121 bash[36902]: audit 2022-04-22T18:47:37.925529+0000 mon.a (mon.0) 5872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:47:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: cluster 2022-04-22T18:47:36.211453+0000 mgr.y (mgr.34107) 2824 : cluster [DBG] pgmap v4388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.004518+0000 mon.a (mon.0) 5867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.541643+0000 mon.a (mon.0) 5868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:47:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.544830+0000 mon.a (mon.0) 5869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:47:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.546457+0000 mon.a (mon.0) 5870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.678131+0000 mon.a (mon.0) 5871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[45495]: audit 2022-04-22T18:47:37.925529+0000 mon.a (mon.0) 5872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: cluster 2022-04-22T18:47:36.211453+0000 mgr.y (mgr.34107) 2824 : cluster [DBG] pgmap v4388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.004518+0000 mon.a (mon.0) 5867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.541643+0000 mon.a (mon.0) 5868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.544830+0000 mon.a (mon.0) 5869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.546457+0000 mon.a (mon.0) 5870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:47:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.678131+0000 mon.a (mon.0) 5871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:37 smithi115 bash[46934]: audit 2022-04-22T18:47:37.925529+0000 mon.a (mon.0) 5872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:47:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:38 smithi121 bash[36902]: audit 2022-04-22T18:47:38.828319+0000 mon.a (mon.0) 5873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:38 smithi115 bash[45495]: audit 2022-04-22T18:47:38.828319+0000 mon.a (mon.0) 5873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:38 smithi115 bash[46934]: audit 2022-04-22T18:47:38.828319+0000 mon.a (mon.0) 5873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:40 smithi121 bash[36902]: cluster 2022-04-22T18:47:38.212525+0000 mgr.y (mgr.34107) 2825 : cluster [DBG] pgmap v4389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:39 smithi115 bash[45495]: cluster 2022-04-22T18:47:38.212525+0000 mgr.y (mgr.34107) 2825 : cluster [DBG] pgmap v4389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:39 smithi115 bash[46934]: cluster 2022-04-22T18:47:38.212525+0000 mgr.y (mgr.34107) 2825 : cluster [DBG] pgmap v4389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:41.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:41 smithi121 bash[36902]: cluster 2022-04-22T18:47:40.213591+0000 mgr.y (mgr.34107) 2826 : cluster [DBG] pgmap v4390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:41 smithi115 bash[45495]: cluster 2022-04-22T18:47:40.213591+0000 mgr.y (mgr.34107) 2826 : cluster [DBG] pgmap v4390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:41 smithi115 bash[46934]: cluster 2022-04-22T18:47:40.213591+0000 mgr.y (mgr.34107) 2826 : cluster [DBG] pgmap v4390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:41.917Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:42 smithi121 bash[36902]: cluster 2022-04-22T18:47:42.214643+0000 mgr.y (mgr.34107) 2827 : cluster [DBG] pgmap v4391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:42 smithi121 bash[36902]: audit 2022-04-22T18:47:42.681726+0000 mon.a (mon.0) 5874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:42 smithi115 bash[45495]: cluster 2022-04-22T18:47:42.214643+0000 mgr.y (mgr.34107) 2827 : cluster [DBG] pgmap v4391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:42 smithi115 bash[45495]: audit 2022-04-22T18:47:42.681726+0000 mon.a (mon.0) 5874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:42 smithi115 bash[46934]: cluster 2022-04-22T18:47:42.214643+0000 mgr.y (mgr.34107) 2827 : cluster [DBG] pgmap v4391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:42 smithi115 bash[46934]: audit 2022-04-22T18:47:42.681726+0000 mon.a (mon.0) 5874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:44 smithi121 bash[36902]: audit 2022-04-22T18:47:43.831432+0000 mon.a (mon.0) 5875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:44 smithi115 bash[45495]: audit 2022-04-22T18:47:43.831432+0000 mon.a (mon.0) 5875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:43.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:44 smithi115 bash[46934]: audit 2022-04-22T18:47:43.831432+0000 mon.a (mon.0) 5875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:45 smithi115 bash[45495]: cluster 2022-04-22T18:47:44.215752+0000 mgr.y (mgr.34107) 2828 : cluster [DBG] pgmap v4392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:45 smithi115 bash[46934]: cluster 2022-04-22T18:47:44.215752+0000 mgr.y (mgr.34107) 2828 : cluster [DBG] pgmap v4392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:45 smithi121 bash[36902]: cluster 2022-04-22T18:47:44.215752+0000 mgr.y (mgr.34107) 2828 : cluster [DBG] pgmap v4392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:47 smithi115 bash[45495]: cluster 2022-04-22T18:47:46.216857+0000 mgr.y (mgr.34107) 2829 : cluster [DBG] pgmap v4393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:47 smithi115 bash[46934]: cluster 2022-04-22T18:47:46.216857+0000 mgr.y (mgr.34107) 2829 : cluster [DBG] pgmap v4393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:47 smithi121 bash[36902]: cluster 2022-04-22T18:47:46.216857+0000 mgr.y (mgr.34107) 2829 : cluster [DBG] pgmap v4393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:48 smithi115 bash[45495]: audit 2022-04-22T18:47:47.685121+0000 mon.a (mon.0) 5876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:48 smithi115 bash[46934]: audit 2022-04-22T18:47:47.685121+0000 mon.a (mon.0) 5876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:48 smithi121 bash[36902]: audit 2022-04-22T18:47:47.685121+0000 mon.a (mon.0) 5876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:49 smithi115 bash[45495]: cluster 2022-04-22T18:47:48.218009+0000 mgr.y (mgr.34107) 2830 : cluster [DBG] pgmap v4394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:49 smithi115 bash[45495]: audit 2022-04-22T18:47:48.834270+0000 mon.a (mon.0) 5877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:49.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:49 smithi115 bash[46934]: cluster 2022-04-22T18:47:48.218009+0000 mgr.y (mgr.34107) 2830 : cluster [DBG] pgmap v4394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:49.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:49 smithi115 bash[46934]: audit 2022-04-22T18:47:48.834270+0000 mon.a (mon.0) 5877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:49 smithi121 bash[36902]: cluster 2022-04-22T18:47:48.218009+0000 mgr.y (mgr.34107) 2830 : cluster [DBG] pgmap v4394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:49 smithi121 bash[36902]: audit 2022-04-22T18:47:48.834270+0000 mon.a (mon.0) 5877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:51 smithi121 bash[36902]: cluster 2022-04-22T18:47:50.218969+0000 mgr.y (mgr.34107) 2831 : cluster [DBG] pgmap v4395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:51 smithi115 bash[45495]: cluster 2022-04-22T18:47:50.218969+0000 mgr.y (mgr.34107) 2831 : cluster [DBG] pgmap v4395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:51 smithi115 bash[46934]: cluster 2022-04-22T18:47:50.218969+0000 mgr.y (mgr.34107) 2831 : cluster [DBG] pgmap v4395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:51.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:53 smithi121 bash[36902]: cluster 2022-04-22T18:47:52.220076+0000 mgr.y (mgr.34107) 2832 : cluster [DBG] pgmap v4396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:53 smithi121 bash[36902]: audit 2022-04-22T18:47:52.689619+0000 mon.a (mon.0) 5878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:52 smithi115 bash[45495]: cluster 2022-04-22T18:47:52.220076+0000 mgr.y (mgr.34107) 2832 : cluster [DBG] pgmap v4396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:52 smithi115 bash[45495]: audit 2022-04-22T18:47:52.689619+0000 mon.a (mon.0) 5878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:52 smithi115 bash[46934]: cluster 2022-04-22T18:47:52.220076+0000 mgr.y (mgr.34107) 2832 : cluster [DBG] pgmap v4396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:52 smithi115 bash[46934]: audit 2022-04-22T18:47:52.689619+0000 mon.a (mon.0) 5878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:54 smithi121 bash[36902]: audit 2022-04-22T18:47:53.837332+0000 mon.a (mon.0) 5879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:54 smithi115 bash[45495]: audit 2022-04-22T18:47:53.837332+0000 mon.a (mon.0) 5879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:47:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:47:53.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:47:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:54 smithi115 bash[46934]: audit 2022-04-22T18:47:53.837332+0000 mon.a (mon.0) 5879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:55 smithi115 bash[45495]: cluster 2022-04-22T18:47:54.221150+0000 mgr.y (mgr.34107) 2833 : cluster [DBG] pgmap v4397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:55 smithi115 bash[46934]: cluster 2022-04-22T18:47:54.221150+0000 mgr.y (mgr.34107) 2833 : cluster [DBG] pgmap v4397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:55 smithi121 bash[36902]: cluster 2022-04-22T18:47:54.221150+0000 mgr.y (mgr.34107) 2833 : cluster [DBG] pgmap v4397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:57 smithi115 bash[45495]: cluster 2022-04-22T18:47:56.222282+0000 mgr.y (mgr.34107) 2834 : cluster [DBG] pgmap v4398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:57 smithi115 bash[46934]: cluster 2022-04-22T18:47:56.222282+0000 mgr.y (mgr.34107) 2834 : cluster [DBG] pgmap v4398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:57 smithi121 bash[36902]: cluster 2022-04-22T18:47:56.222282+0000 mgr.y (mgr.34107) 2834 : cluster [DBG] pgmap v4398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:58 smithi115 bash[45495]: audit 2022-04-22T18:47:57.693225+0000 mon.a (mon.0) 5880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:58 smithi115 bash[46934]: audit 2022-04-22T18:47:57.693225+0000 mon.a (mon.0) 5880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:58 smithi121 bash[36902]: audit 2022-04-22T18:47:57.693225+0000 mon.a (mon.0) 5880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:47:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:59 smithi121 bash[36902]: cluster 2022-04-22T18:47:58.223485+0000 mgr.y (mgr.34107) 2835 : cluster [DBG] pgmap v4399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:47:59 smithi121 bash[36902]: audit 2022-04-22T18:47:58.840465+0000 mon.a (mon.0) 5881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:59 smithi115 bash[45495]: cluster 2022-04-22T18:47:58.223485+0000 mgr.y (mgr.34107) 2835 : cluster [DBG] pgmap v4399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:47:59 smithi115 bash[45495]: audit 2022-04-22T18:47:58.840465+0000 mon.a (mon.0) 5881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:47:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:59 smithi115 bash[46934]: cluster 2022-04-22T18:47:58.223485+0000 mgr.y (mgr.34107) 2835 : cluster [DBG] pgmap v4399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:47:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:47:59 smithi115 bash[46934]: audit 2022-04-22T18:47:58.840465+0000 mon.a (mon.0) 5881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:01 smithi121 bash[36902]: cluster 2022-04-22T18:48:00.224571+0000 mgr.y (mgr.34107) 2836 : cluster [DBG] pgmap v4400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:01 smithi115 bash[45495]: cluster 2022-04-22T18:48:00.224571+0000 mgr.y (mgr.34107) 2836 : cluster [DBG] pgmap v4400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:01 smithi115 bash[46934]: cluster 2022-04-22T18:48:00.224571+0000 mgr.y (mgr.34107) 2836 : cluster [DBG] pgmap v4400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:01.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:03 smithi121 bash[36902]: cluster 2022-04-22T18:48:02.225610+0000 mgr.y (mgr.34107) 2837 : cluster [DBG] pgmap v4401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:03 smithi121 bash[36902]: audit 2022-04-22T18:48:02.696957+0000 mon.a (mon.0) 5882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:03 smithi115 bash[45495]: cluster 2022-04-22T18:48:02.225610+0000 mgr.y (mgr.34107) 2837 : cluster [DBG] pgmap v4401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:03 smithi115 bash[45495]: audit 2022-04-22T18:48:02.696957+0000 mon.a (mon.0) 5882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:03 smithi115 bash[46934]: cluster 2022-04-22T18:48:02.225610+0000 mgr.y (mgr.34107) 2837 : cluster [DBG] pgmap v4401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:03 smithi115 bash[46934]: audit 2022-04-22T18:48:02.696957+0000 mon.a (mon.0) 5882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:04 smithi121 bash[36902]: audit 2022-04-22T18:48:03.843720+0000 mon.a (mon.0) 5883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:04 smithi115 bash[45495]: audit 2022-04-22T18:48:03.843720+0000 mon.a (mon.0) 5883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:03.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:04 smithi115 bash[46934]: audit 2022-04-22T18:48:03.843720+0000 mon.a (mon.0) 5883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:05.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:05 smithi115 bash[45495]: cluster 2022-04-22T18:48:04.226704+0000 mgr.y (mgr.34107) 2838 : cluster [DBG] pgmap v4402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:05.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:05 smithi115 bash[46934]: cluster 2022-04-22T18:48:04.226704+0000 mgr.y (mgr.34107) 2838 : cluster [DBG] pgmap v4402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:05 smithi121 bash[36902]: cluster 2022-04-22T18:48:04.226704+0000 mgr.y (mgr.34107) 2838 : cluster [DBG] pgmap v4402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:05.790 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:48:06.145 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:48:06.145 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (71m) 8m ago 84m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:48:06.145 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (84m) 9m ago 84m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:48:06.145 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (73m) 8m ago 85m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (72m) 9m ago 85m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (80m) 9m ago 90m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (81m) 8m ago 92m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (80m) 8m ago 92m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (79m) 9m ago 90m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (79m) 8m ago 90m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (72m) 8m ago 85m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (72m) 9m ago 85m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:48:06.146 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (77m) 8m ago 89m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (77m) 8m ago 89m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (76m) 8m ago 88m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (76m) 8m ago 88m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (75m) 9m ago 87m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (74m) 9m ago 87m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (74m) 9m ago 87m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (73m) 9m ago 86m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:48:06.147 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (71m) 9m ago 85m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:48:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:06 smithi115 bash[46934]: audit 2022-04-22T18:48:05.764358+0000 mgr.y (mgr.34107) 2839 : audit [DBG] from='client.48328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:06 smithi115 bash[45495]: audit 2022-04-22T18:48:05.764358+0000 mgr.y (mgr.34107) 2839 : audit [DBG] from='client.48328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:06 smithi121 bash[36902]: audit 2022-04-22T18:48:05.764358+0000 mgr.y (mgr.34107) 2839 : audit [DBG] from='client.48328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:06.537 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:48:06.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:48:06.539 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:48:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[45495]: audit 2022-04-22T18:48:06.112247+0000 mgr.y (mgr.34107) 2840 : audit [DBG] from='client.48334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[45495]: cluster 2022-04-22T18:48:06.227726+0000 mgr.y (mgr.34107) 2841 : cluster [DBG] pgmap v4403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[45495]: audit 2022-04-22T18:48:06.528377+0000 mon.c (mon.1) 160 : audit [DBG] from='client.? 172.21.15.115:0/2894511041' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:07.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[46934]: audit 2022-04-22T18:48:06.112247+0000 mgr.y (mgr.34107) 2840 : audit [DBG] from='client.48334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:07.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[46934]: cluster 2022-04-22T18:48:06.227726+0000 mgr.y (mgr.34107) 2841 : cluster [DBG] pgmap v4403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:07.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:07 smithi115 bash[46934]: audit 2022-04-22T18:48:06.528377+0000 mon.c (mon.1) 160 : audit [DBG] from='client.? 172.21.15.115:0/2894511041' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:07 smithi121 bash[36902]: audit 2022-04-22T18:48:06.112247+0000 mgr.y (mgr.34107) 2840 : audit [DBG] from='client.48334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:07 smithi121 bash[36902]: cluster 2022-04-22T18:48:06.227726+0000 mgr.y (mgr.34107) 2841 : cluster [DBG] pgmap v4403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:07 smithi121 bash[36902]: audit 2022-04-22T18:48:06.528377+0000 mon.c (mon.1) 160 : audit [DBG] from='client.? 172.21.15.115:0/2894511041' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:08 smithi115 bash[46934]: audit 2022-04-22T18:48:07.700170+0000 mon.a (mon.0) 5884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:08 smithi115 bash[45495]: audit 2022-04-22T18:48:07.700170+0000 mon.a (mon.0) 5884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:08 smithi121 bash[36902]: audit 2022-04-22T18:48:07.700170+0000 mon.a (mon.0) 5884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:09 smithi115 bash[46934]: cluster 2022-04-22T18:48:08.228813+0000 mgr.y (mgr.34107) 2842 : cluster [DBG] pgmap v4404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:09 smithi115 bash[46934]: audit 2022-04-22T18:48:08.846851+0000 mon.a (mon.0) 5885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:09.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:09 smithi115 bash[45495]: cluster 2022-04-22T18:48:08.228813+0000 mgr.y (mgr.34107) 2842 : cluster [DBG] pgmap v4404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:09.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:09 smithi115 bash[45495]: audit 2022-04-22T18:48:08.846851+0000 mon.a (mon.0) 5885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:09 smithi121 bash[36902]: cluster 2022-04-22T18:48:08.228813+0000 mgr.y (mgr.34107) 2842 : cluster [DBG] pgmap v4404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:09 smithi121 bash[36902]: audit 2022-04-22T18:48:08.846851+0000 mon.a (mon.0) 5885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:11.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:11 smithi115 bash[45495]: cluster 2022-04-22T18:48:10.229902+0000 mgr.y (mgr.34107) 2843 : cluster [DBG] pgmap v4405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:11 smithi115 bash[46934]: cluster 2022-04-22T18:48:10.229902+0000 mgr.y (mgr.34107) 2843 : cluster [DBG] pgmap v4405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:11 smithi121 bash[36902]: cluster 2022-04-22T18:48:10.229902+0000 mgr.y (mgr.34107) 2843 : cluster [DBG] pgmap v4405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:11.921Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:13 smithi121 bash[36902]: cluster 2022-04-22T18:48:12.230958+0000 mgr.y (mgr.34107) 2844 : cluster [DBG] pgmap v4406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:13 smithi121 bash[36902]: audit 2022-04-22T18:48:12.703916+0000 mon.a (mon.0) 5886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:13 smithi115 bash[46934]: cluster 2022-04-22T18:48:12.230958+0000 mgr.y (mgr.34107) 2844 : cluster [DBG] pgmap v4406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:13 smithi115 bash[46934]: audit 2022-04-22T18:48:12.703916+0000 mon.a (mon.0) 5886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:13 smithi115 bash[45495]: cluster 2022-04-22T18:48:12.230958+0000 mgr.y (mgr.34107) 2844 : cluster [DBG] pgmap v4406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:13 smithi115 bash[45495]: audit 2022-04-22T18:48:12.703916+0000 mon.a (mon.0) 5886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:14 smithi121 bash[36902]: audit 2022-04-22T18:48:13.849739+0000 mon.a (mon.0) 5887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:14 smithi115 bash[45495]: audit 2022-04-22T18:48:13.849739+0000 mon.a (mon.0) 5887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:13.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:14 smithi115 bash[46934]: audit 2022-04-22T18:48:13.849739+0000 mon.a (mon.0) 5887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[45495]: audit 2022-04-22T18:48:14.163304+0000 mon.a (mon.0) 5888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:48:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[45495]: audit 2022-04-22T18:48:14.164820+0000 mon.a (mon.0) 5889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:48:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[45495]: cluster 2022-04-22T18:48:14.232134+0000 mgr.y (mgr.34107) 2845 : cluster [DBG] pgmap v4407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[46934]: audit 2022-04-22T18:48:14.163304+0000 mon.a (mon.0) 5888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:48:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[46934]: audit 2022-04-22T18:48:14.164820+0000 mon.a (mon.0) 5889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:48:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:15 smithi115 bash[46934]: cluster 2022-04-22T18:48:14.232134+0000 mgr.y (mgr.34107) 2845 : cluster [DBG] pgmap v4407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:15 smithi121 bash[36902]: audit 2022-04-22T18:48:14.163304+0000 mon.a (mon.0) 5888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:48:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:15 smithi121 bash[36902]: audit 2022-04-22T18:48:14.164820+0000 mon.a (mon.0) 5889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:48:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:15 smithi121 bash[36902]: cluster 2022-04-22T18:48:14.232134+0000 mgr.y (mgr.34107) 2845 : cluster [DBG] pgmap v4407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:17 smithi121 bash[36902]: cluster 2022-04-22T18:48:16.233227+0000 mgr.y (mgr.34107) 2846 : cluster [DBG] pgmap v4408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:17 smithi115 bash[45495]: cluster 2022-04-22T18:48:16.233227+0000 mgr.y (mgr.34107) 2846 : cluster [DBG] pgmap v4408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:17 smithi115 bash[46934]: cluster 2022-04-22T18:48:16.233227+0000 mgr.y (mgr.34107) 2846 : cluster [DBG] pgmap v4408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:18 smithi121 bash[36902]: audit 2022-04-22T18:48:17.707458+0000 mon.a (mon.0) 5890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:18 smithi115 bash[45495]: audit 2022-04-22T18:48:17.707458+0000 mon.a (mon.0) 5890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:18 smithi115 bash[46934]: audit 2022-04-22T18:48:17.707458+0000 mon.a (mon.0) 5890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:19 smithi121 bash[36902]: cluster 2022-04-22T18:48:18.234326+0000 mgr.y (mgr.34107) 2847 : cluster [DBG] pgmap v4409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:19 smithi121 bash[36902]: audit 2022-04-22T18:48:18.852832+0000 mon.a (mon.0) 5891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:19 smithi115 bash[45495]: cluster 2022-04-22T18:48:18.234326+0000 mgr.y (mgr.34107) 2847 : cluster [DBG] pgmap v4409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:19 smithi115 bash[45495]: audit 2022-04-22T18:48:18.852832+0000 mon.a (mon.0) 5891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:19 smithi115 bash[46934]: cluster 2022-04-22T18:48:18.234326+0000 mgr.y (mgr.34107) 2847 : cluster [DBG] pgmap v4409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:19 smithi115 bash[46934]: audit 2022-04-22T18:48:18.852832+0000 mon.a (mon.0) 5891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:21 smithi121 bash[36902]: cluster 2022-04-22T18:48:20.235414+0000 mgr.y (mgr.34107) 2848 : cluster [DBG] pgmap v4410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:21 smithi115 bash[46934]: cluster 2022-04-22T18:48:20.235414+0000 mgr.y (mgr.34107) 2848 : cluster [DBG] pgmap v4410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:21 smithi115 bash[45495]: cluster 2022-04-22T18:48:20.235414+0000 mgr.y (mgr.34107) 2848 : cluster [DBG] pgmap v4410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:21.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:23 smithi115 bash[46934]: cluster 2022-04-22T18:48:22.236549+0000 mgr.y (mgr.34107) 2849 : cluster [DBG] pgmap v4411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:23 smithi115 bash[46934]: audit 2022-04-22T18:48:22.711036+0000 mon.a (mon.0) 5892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:23 smithi115 bash[45495]: cluster 2022-04-22T18:48:22.236549+0000 mgr.y (mgr.34107) 2849 : cluster [DBG] pgmap v4411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:23 smithi115 bash[45495]: audit 2022-04-22T18:48:22.711036+0000 mon.a (mon.0) 5892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:23 smithi121 bash[36902]: cluster 2022-04-22T18:48:22.236549+0000 mgr.y (mgr.34107) 2849 : cluster [DBG] pgmap v4411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:23 smithi121 bash[36902]: audit 2022-04-22T18:48:22.711036+0000 mon.a (mon.0) 5892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:24 smithi115 bash[45495]: audit 2022-04-22T18:48:23.855877+0000 mon.a (mon.0) 5893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:24 smithi115 bash[46934]: audit 2022-04-22T18:48:23.855877+0000 mon.a (mon.0) 5893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:23.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:24 smithi121 bash[36902]: audit 2022-04-22T18:48:23.855877+0000 mon.a (mon.0) 5893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:25 smithi115 bash[45495]: cluster 2022-04-22T18:48:24.237678+0000 mgr.y (mgr.34107) 2850 : cluster [DBG] pgmap v4412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:25 smithi115 bash[46934]: cluster 2022-04-22T18:48:24.237678+0000 mgr.y (mgr.34107) 2850 : cluster [DBG] pgmap v4412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:25 smithi121 bash[36902]: cluster 2022-04-22T18:48:24.237678+0000 mgr.y (mgr.34107) 2850 : cluster [DBG] pgmap v4412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:27.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:27 smithi121 bash[36902]: cluster 2022-04-22T18:48:26.238758+0000 mgr.y (mgr.34107) 2851 : cluster [DBG] pgmap v4413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:27 smithi115 bash[45495]: cluster 2022-04-22T18:48:26.238758+0000 mgr.y (mgr.34107) 2851 : cluster [DBG] pgmap v4413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:27 smithi115 bash[46934]: cluster 2022-04-22T18:48:26.238758+0000 mgr.y (mgr.34107) 2851 : cluster [DBG] pgmap v4413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:28 smithi121 bash[36902]: audit 2022-04-22T18:48:27.714739+0000 mon.a (mon.0) 5894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:28 smithi115 bash[45495]: audit 2022-04-22T18:48:27.714739+0000 mon.a (mon.0) 5894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:28 smithi115 bash[46934]: audit 2022-04-22T18:48:27.714739+0000 mon.a (mon.0) 5894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:29 smithi121 bash[36902]: cluster 2022-04-22T18:48:28.239907+0000 mgr.y (mgr.34107) 2852 : cluster [DBG] pgmap v4414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:29 smithi121 bash[36902]: audit 2022-04-22T18:48:28.859228+0000 mon.a (mon.0) 5895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:29 smithi115 bash[45495]: cluster 2022-04-22T18:48:28.239907+0000 mgr.y (mgr.34107) 2852 : cluster [DBG] pgmap v4414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:29 smithi115 bash[45495]: audit 2022-04-22T18:48:28.859228+0000 mon.a (mon.0) 5895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:29 smithi115 bash[46934]: cluster 2022-04-22T18:48:28.239907+0000 mgr.y (mgr.34107) 2852 : cluster [DBG] pgmap v4414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:29 smithi115 bash[46934]: audit 2022-04-22T18:48:28.859228+0000 mon.a (mon.0) 5895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:31 smithi121 bash[36902]: cluster 2022-04-22T18:48:30.240994+0000 mgr.y (mgr.34107) 2853 : cluster [DBG] pgmap v4415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:31 smithi115 bash[45495]: cluster 2022-04-22T18:48:30.240994+0000 mgr.y (mgr.34107) 2853 : cluster [DBG] pgmap v4415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:31 smithi115 bash[46934]: cluster 2022-04-22T18:48:30.240994+0000 mgr.y (mgr.34107) 2853 : cluster [DBG] pgmap v4415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:31.921Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:33 smithi115 bash[45495]: cluster 2022-04-22T18:48:32.242096+0000 mgr.y (mgr.34107) 2854 : cluster [DBG] pgmap v4416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:33 smithi115 bash[45495]: audit 2022-04-22T18:48:32.721083+0000 mon.a (mon.0) 5896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:33 smithi115 bash[46934]: cluster 2022-04-22T18:48:32.242096+0000 mgr.y (mgr.34107) 2854 : cluster [DBG] pgmap v4416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:33 smithi115 bash[46934]: audit 2022-04-22T18:48:32.721083+0000 mon.a (mon.0) 5896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:33.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:33 smithi121 bash[36902]: cluster 2022-04-22T18:48:32.242096+0000 mgr.y (mgr.34107) 2854 : cluster [DBG] pgmap v4416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:33.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:33 smithi121 bash[36902]: audit 2022-04-22T18:48:32.721083+0000 mon.a (mon.0) 5896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:34 smithi115 bash[46934]: audit 2022-04-22T18:48:33.862303+0000 mon.a (mon.0) 5897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:33.795Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:34 smithi115 bash[45495]: audit 2022-04-22T18:48:33.862303+0000 mon.a (mon.0) 5897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:34 smithi121 bash[36902]: audit 2022-04-22T18:48:33.862303+0000 mon.a (mon.0) 5897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:35 smithi115 bash[45495]: cluster 2022-04-22T18:48:34.243197+0000 mgr.y (mgr.34107) 2855 : cluster [DBG] pgmap v4417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:35 smithi115 bash[46934]: cluster 2022-04-22T18:48:34.243197+0000 mgr.y (mgr.34107) 2855 : cluster [DBG] pgmap v4417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:35 smithi121 bash[36902]: cluster 2022-04-22T18:48:34.243197+0000 mgr.y (mgr.34107) 2855 : cluster [DBG] pgmap v4417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:36.915 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:48:37.262 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (71m) 9m ago 85m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (85m) 10m ago 85m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (73m) 9m ago 86m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (73m) 10m ago 86m 21.7M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (80m) 10m ago 90m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (81m) 9m ago 92m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (80m) 9m ago 93m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (79m) 10m ago 91m 155M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (80m) 9m ago 91m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:48:37.263 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (72m) 9m ago 85m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (72m) 10m ago 85m 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (78m) 9m ago 90m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (77m) 9m ago 89m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (77m) 9m ago 89m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (76m) 9m ago 88m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (76m) 10m ago 88m 567M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (75m) 10m ago 87m 697M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (74m) 10m ago 87m 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (74m) 10m ago 87m 629M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:48:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (72m) 10m ago 85m 63.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:48:37.670 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:48:37.745 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:37.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:48:37.747 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:48:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:37 smithi121 bash[36902]: cluster 2022-04-22T18:48:36.244321+0000 mgr.y (mgr.34107) 2856 : cluster [DBG] pgmap v4418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:37 smithi121 bash[36902]: audit 2022-04-22T18:48:36.890553+0000 mgr.y (mgr.34107) 2857 : audit [DBG] from='client.48346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:37 smithi115 bash[45495]: cluster 2022-04-22T18:48:36.244321+0000 mgr.y (mgr.34107) 2856 : cluster [DBG] pgmap v4418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:37 smithi115 bash[45495]: audit 2022-04-22T18:48:36.890553+0000 mgr.y (mgr.34107) 2857 : audit [DBG] from='client.48346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:37 smithi115 bash[46934]: cluster 2022-04-22T18:48:36.244321+0000 mgr.y (mgr.34107) 2856 : cluster [DBG] pgmap v4418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:37 smithi115 bash[46934]: audit 2022-04-22T18:48:36.890553+0000 mgr.y (mgr.34107) 2857 : audit [DBG] from='client.48346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.235334+0000 mgr.y (mgr.34107) 2858 : audit [DBG] from='client.38418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.661295+0000 mon.c (mon.1) 161 : audit [DBG] from='client.? 172.21.15.115:0/1415046733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.724906+0000 mon.a (mon.0) 5898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.932106+0000 mon.a (mon.0) 5899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.935190+0000 mon.a (mon.0) 5900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[45495]: audit 2022-04-22T18:48:37.937009+0000 mon.a (mon.0) 5901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.235334+0000 mgr.y (mgr.34107) 2858 : audit [DBG] from='client.38418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.661295+0000 mon.c (mon.1) 161 : audit [DBG] from='client.? 172.21.15.115:0/1415046733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:38.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.724906+0000 mon.a (mon.0) 5898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.932106+0000 mon.a (mon.0) 5899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:48:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.935190+0000 mon.a (mon.0) 5900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:48:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:38 smithi115 bash[46934]: audit 2022-04-22T18:48:37.937009+0000 mon.a (mon.0) 5901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:48:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.235334+0000 mgr.y (mgr.34107) 2858 : audit [DBG] from='client.38418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:48:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.661295+0000 mon.c (mon.1) 161 : audit [DBG] from='client.? 172.21.15.115:0/1415046733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:48:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.724906+0000 mon.a (mon.0) 5898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.932106+0000 mon.a (mon.0) 5899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:48:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.935190+0000 mon.a (mon.0) 5900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:48:38.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:38 smithi121 bash[36902]: audit 2022-04-22T18:48:37.937009+0000 mon.a (mon.0) 5901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:48:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:39 smithi121 bash[36902]: cluster 2022-04-22T18:48:38.245444+0000 mgr.y (mgr.34107) 2859 : cluster [DBG] pgmap v4419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:39 smithi121 bash[36902]: audit 2022-04-22T18:48:38.865406+0000 mon.a (mon.0) 5902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:39 smithi115 bash[45495]: cluster 2022-04-22T18:48:38.245444+0000 mgr.y (mgr.34107) 2859 : cluster [DBG] pgmap v4419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:39 smithi115 bash[45495]: audit 2022-04-22T18:48:38.865406+0000 mon.a (mon.0) 5902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:39 smithi115 bash[46934]: cluster 2022-04-22T18:48:38.245444+0000 mgr.y (mgr.34107) 2859 : cluster [DBG] pgmap v4419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:39 smithi115 bash[46934]: audit 2022-04-22T18:48:38.865406+0000 mon.a (mon.0) 5902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:41 smithi121 bash[36902]: cluster 2022-04-22T18:48:40.246492+0000 mgr.y (mgr.34107) 2860 : cluster [DBG] pgmap v4420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:41 smithi115 bash[46934]: cluster 2022-04-22T18:48:40.246492+0000 mgr.y (mgr.34107) 2860 : cluster [DBG] pgmap v4420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:41 smithi115 bash[45495]: cluster 2022-04-22T18:48:40.246492+0000 mgr.y (mgr.34107) 2860 : cluster [DBG] pgmap v4420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:41.921Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:43 smithi121 bash[36902]: cluster 2022-04-22T18:48:42.247592+0000 mgr.y (mgr.34107) 2861 : cluster [DBG] pgmap v4421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:43.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:43 smithi121 bash[36902]: audit 2022-04-22T18:48:42.320949+0000 mon.a (mon.0) 5903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:43 smithi121 bash[36902]: audit 2022-04-22T18:48:42.666663+0000 mon.a (mon.0) 5904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:43 smithi121 bash[36902]: audit 2022-04-22T18:48:42.728025+0000 mon.a (mon.0) 5905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[45495]: cluster 2022-04-22T18:48:42.247592+0000 mgr.y (mgr.34107) 2861 : cluster [DBG] pgmap v4421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[45495]: audit 2022-04-22T18:48:42.320949+0000 mon.a (mon.0) 5903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[45495]: audit 2022-04-22T18:48:42.666663+0000 mon.a (mon.0) 5904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[45495]: audit 2022-04-22T18:48:42.728025+0000 mon.a (mon.0) 5905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[46934]: cluster 2022-04-22T18:48:42.247592+0000 mgr.y (mgr.34107) 2861 : cluster [DBG] pgmap v4421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[46934]: audit 2022-04-22T18:48:42.320949+0000 mon.a (mon.0) 5903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[46934]: audit 2022-04-22T18:48:42.666663+0000 mon.a (mon.0) 5904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:48:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[46934]: audit 2022-04-22T18:48:42.728025+0000 mon.a (mon.0) 5905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:43.795Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:44 smithi121 bash[36902]: audit 2022-04-22T18:48:43.868713+0000 mon.a (mon.0) 5906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:44 smithi115 bash[45495]: audit 2022-04-22T18:48:43.868713+0000 mon.a (mon.0) 5906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:44 smithi115 bash[46934]: audit 2022-04-22T18:48:43.868713+0000 mon.a (mon.0) 5906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:45 smithi121 bash[36902]: cluster 2022-04-22T18:48:44.248616+0000 mgr.y (mgr.34107) 2862 : cluster [DBG] pgmap v4422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:45 smithi115 bash[45495]: cluster 2022-04-22T18:48:44.248616+0000 mgr.y (mgr.34107) 2862 : cluster [DBG] pgmap v4422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:45 smithi115 bash[46934]: cluster 2022-04-22T18:48:44.248616+0000 mgr.y (mgr.34107) 2862 : cluster [DBG] pgmap v4422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:47 smithi121 bash[36902]: cluster 2022-04-22T18:48:46.249709+0000 mgr.y (mgr.34107) 2863 : cluster [DBG] pgmap v4423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:47 smithi115 bash[45495]: cluster 2022-04-22T18:48:46.249709+0000 mgr.y (mgr.34107) 2863 : cluster [DBG] pgmap v4423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:47 smithi115 bash[46934]: cluster 2022-04-22T18:48:46.249709+0000 mgr.y (mgr.34107) 2863 : cluster [DBG] pgmap v4423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:48 smithi121 bash[36902]: audit 2022-04-22T18:48:47.731423+0000 mon.a (mon.0) 5907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:48 smithi115 bash[45495]: audit 2022-04-22T18:48:47.731423+0000 mon.a (mon.0) 5907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:48 smithi115 bash[46934]: audit 2022-04-22T18:48:47.731423+0000 mon.a (mon.0) 5907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:49 smithi121 bash[36902]: cluster 2022-04-22T18:48:48.250817+0000 mgr.y (mgr.34107) 2864 : cluster [DBG] pgmap v4424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:49 smithi121 bash[36902]: audit 2022-04-22T18:48:48.871935+0000 mon.a (mon.0) 5908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:49 smithi115 bash[45495]: cluster 2022-04-22T18:48:48.250817+0000 mgr.y (mgr.34107) 2864 : cluster [DBG] pgmap v4424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:49 smithi115 bash[45495]: audit 2022-04-22T18:48:48.871935+0000 mon.a (mon.0) 5908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:49 smithi115 bash[46934]: cluster 2022-04-22T18:48:48.250817+0000 mgr.y (mgr.34107) 2864 : cluster [DBG] pgmap v4424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:49 smithi115 bash[46934]: audit 2022-04-22T18:48:48.871935+0000 mon.a (mon.0) 5908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:51 smithi121 bash[36902]: cluster 2022-04-22T18:48:50.251922+0000 mgr.y (mgr.34107) 2865 : cluster [DBG] pgmap v4425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:51 smithi115 bash[45495]: cluster 2022-04-22T18:48:50.251922+0000 mgr.y (mgr.34107) 2865 : cluster [DBG] pgmap v4425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:51 smithi115 bash[46934]: cluster 2022-04-22T18:48:50.251922+0000 mgr.y (mgr.34107) 2865 : cluster [DBG] pgmap v4425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:51.922Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:53 smithi115 bash[45495]: cluster 2022-04-22T18:48:52.253073+0000 mgr.y (mgr.34107) 2866 : cluster [DBG] pgmap v4426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:53 smithi115 bash[45495]: audit 2022-04-22T18:48:52.734390+0000 mon.a (mon.0) 5909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:53 smithi115 bash[46934]: cluster 2022-04-22T18:48:52.253073+0000 mgr.y (mgr.34107) 2866 : cluster [DBG] pgmap v4426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:53 smithi115 bash[46934]: audit 2022-04-22T18:48:52.734390+0000 mon.a (mon.0) 5909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:53 smithi121 bash[36902]: cluster 2022-04-22T18:48:52.253073+0000 mgr.y (mgr.34107) 2866 : cluster [DBG] pgmap v4426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:53 smithi121 bash[36902]: audit 2022-04-22T18:48:52.734390+0000 mon.a (mon.0) 5909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:54 smithi115 bash[45495]: audit 2022-04-22T18:48:53.874967+0000 mon.a (mon.0) 5910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:48:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:48:53.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:48:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:54 smithi115 bash[46934]: audit 2022-04-22T18:48:53.874967+0000 mon.a (mon.0) 5910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:54 smithi121 bash[36902]: audit 2022-04-22T18:48:53.874967+0000 mon.a (mon.0) 5910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:55 smithi121 bash[36902]: cluster 2022-04-22T18:48:54.254177+0000 mgr.y (mgr.34107) 2867 : cluster [DBG] pgmap v4427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:55 smithi115 bash[45495]: cluster 2022-04-22T18:48:54.254177+0000 mgr.y (mgr.34107) 2867 : cluster [DBG] pgmap v4427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:55 smithi115 bash[46934]: cluster 2022-04-22T18:48:54.254177+0000 mgr.y (mgr.34107) 2867 : cluster [DBG] pgmap v4427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:57 smithi121 bash[36902]: cluster 2022-04-22T18:48:56.255303+0000 mgr.y (mgr.34107) 2868 : cluster [DBG] pgmap v4428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:57 smithi115 bash[45495]: cluster 2022-04-22T18:48:56.255303+0000 mgr.y (mgr.34107) 2868 : cluster [DBG] pgmap v4428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:57 smithi115 bash[46934]: cluster 2022-04-22T18:48:56.255303+0000 mgr.y (mgr.34107) 2868 : cluster [DBG] pgmap v4428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:58 smithi121 bash[36902]: audit 2022-04-22T18:48:57.737994+0000 mon.a (mon.0) 5911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:58 smithi115 bash[45495]: audit 2022-04-22T18:48:57.737994+0000 mon.a (mon.0) 5911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:58 smithi115 bash[46934]: audit 2022-04-22T18:48:57.737994+0000 mon.a (mon.0) 5911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:48:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:59 smithi121 bash[36902]: cluster 2022-04-22T18:48:58.256385+0000 mgr.y (mgr.34107) 2869 : cluster [DBG] pgmap v4429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:59.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:48:59 smithi121 bash[36902]: audit 2022-04-22T18:48:58.877923+0000 mon.a (mon.0) 5912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:59 smithi115 bash[45495]: cluster 2022-04-22T18:48:58.256385+0000 mgr.y (mgr.34107) 2869 : cluster [DBG] pgmap v4429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:48:59 smithi115 bash[45495]: audit 2022-04-22T18:48:58.877923+0000 mon.a (mon.0) 5912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:48:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:59 smithi115 bash[46934]: cluster 2022-04-22T18:48:58.256385+0000 mgr.y (mgr.34107) 2869 : cluster [DBG] pgmap v4429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:48:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:48:59 smithi115 bash[46934]: audit 2022-04-22T18:48:58.877923+0000 mon.a (mon.0) 5912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:01 smithi121 bash[36902]: cluster 2022-04-22T18:49:00.257437+0000 mgr.y (mgr.34107) 2870 : cluster [DBG] pgmap v4430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:01 smithi115 bash[45495]: cluster 2022-04-22T18:49:00.257437+0000 mgr.y (mgr.34107) 2870 : cluster [DBG] pgmap v4430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:01 smithi115 bash[46934]: cluster 2022-04-22T18:49:00.257437+0000 mgr.y (mgr.34107) 2870 : cluster [DBG] pgmap v4430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:01.923Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:03 smithi115 bash[45495]: cluster 2022-04-22T18:49:02.258537+0000 mgr.y (mgr.34107) 2871 : cluster [DBG] pgmap v4431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:03 smithi115 bash[45495]: audit 2022-04-22T18:49:02.741822+0000 mon.a (mon.0) 5913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:03 smithi115 bash[46934]: cluster 2022-04-22T18:49:02.258537+0000 mgr.y (mgr.34107) 2871 : cluster [DBG] pgmap v4431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:03 smithi115 bash[46934]: audit 2022-04-22T18:49:02.741822+0000 mon.a (mon.0) 5913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:03 smithi121 bash[36902]: cluster 2022-04-22T18:49:02.258537+0000 mgr.y (mgr.34107) 2871 : cluster [DBG] pgmap v4431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:03 smithi121 bash[36902]: audit 2022-04-22T18:49:02.741822+0000 mon.a (mon.0) 5913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:04 smithi115 bash[45495]: audit 2022-04-22T18:49:03.881292+0000 mon.a (mon.0) 5914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:03.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:04 smithi115 bash[46934]: audit 2022-04-22T18:49:03.881292+0000 mon.a (mon.0) 5914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:04.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:04 smithi121 bash[36902]: audit 2022-04-22T18:49:03.881292+0000 mon.a (mon.0) 5914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:05 smithi115 bash[45495]: cluster 2022-04-22T18:49:04.259701+0000 mgr.y (mgr.34107) 2872 : cluster [DBG] pgmap v4432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:05 smithi115 bash[46934]: cluster 2022-04-22T18:49:04.259701+0000 mgr.y (mgr.34107) 2872 : cluster [DBG] pgmap v4432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:05 smithi121 bash[36902]: cluster 2022-04-22T18:49:04.259701+0000 mgr.y (mgr.34107) 2872 : cluster [DBG] pgmap v4432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:07 smithi121 bash[36902]: cluster 2022-04-22T18:49:06.260835+0000 mgr.y (mgr.34107) 2873 : cluster [DBG] pgmap v4433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:07 smithi115 bash[45495]: cluster 2022-04-22T18:49:06.260835+0000 mgr.y (mgr.34107) 2873 : cluster [DBG] pgmap v4433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:07 smithi115 bash[46934]: cluster 2022-04-22T18:49:06.260835+0000 mgr.y (mgr.34107) 2873 : cluster [DBG] pgmap v4433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:49:08.377 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:49:08.377 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (72m) 9m ago 86m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (85m) 26s ago 85m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (74m) 9m ago 87m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (73m) 26s ago 86m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (81m) 26s ago 91m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (82m) 9m ago 93m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (81m) 9m ago 93m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (80m) 26s ago 91m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (80m) 9m ago 91m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (73m) 9m ago 86m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:49:08.378 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (73m) 26s ago 86m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (78m) 9m ago 90m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (78m) 9m ago 90m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (77m) 9m ago 89m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (77m) 9m ago 89m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (76m) 26s ago 88m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (75m) 26s ago 88m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (75m) 26s ago 88m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (74m) 26s ago 87m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:49:08.379 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (72m) 26s ago 86m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:49:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:08 smithi121 bash[36902]: audit 2022-04-22T18:49:07.745581+0000 mon.a (mon.0) 5915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:08.777 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:49:08.778 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:49:08.779 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:49:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:08 smithi115 bash[45495]: audit 2022-04-22T18:49:07.745581+0000 mon.a (mon.0) 5915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:08 smithi115 bash[46934]: audit 2022-04-22T18:49:07.745581+0000 mon.a (mon.0) 5915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:09 smithi121 bash[36902]: audit 2022-04-22T18:49:08.010272+0000 mgr.y (mgr.34107) 2874 : audit [DBG] from='client.38430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:09 smithi121 bash[36902]: cluster 2022-04-22T18:49:08.261615+0000 mgr.y (mgr.34107) 2875 : cluster [DBG] pgmap v4434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:09 smithi121 bash[36902]: audit 2022-04-22T18:49:08.349669+0000 mgr.y (mgr.34107) 2876 : audit [DBG] from='client.38436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:09 smithi121 bash[36902]: audit 2022-04-22T18:49:08.768787+0000 mon.c (mon.1) 162 : audit [DBG] from='client.? 172.21.15.115:0/3955663629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:09 smithi121 bash[36902]: audit 2022-04-22T18:49:08.884415+0000 mon.a (mon.0) 5916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[46934]: audit 2022-04-22T18:49:08.010272+0000 mgr.y (mgr.34107) 2874 : audit [DBG] from='client.38430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[46934]: cluster 2022-04-22T18:49:08.261615+0000 mgr.y (mgr.34107) 2875 : cluster [DBG] pgmap v4434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[46934]: audit 2022-04-22T18:49:08.349669+0000 mgr.y (mgr.34107) 2876 : audit [DBG] from='client.38436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[46934]: audit 2022-04-22T18:49:08.768787+0000 mon.c (mon.1) 162 : audit [DBG] from='client.? 172.21.15.115:0/3955663629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[46934]: audit 2022-04-22T18:49:08.884415+0000 mon.a (mon.0) 5916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[45495]: audit 2022-04-22T18:49:08.010272+0000 mgr.y (mgr.34107) 2874 : audit [DBG] from='client.38430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[45495]: cluster 2022-04-22T18:49:08.261615+0000 mgr.y (mgr.34107) 2875 : cluster [DBG] pgmap v4434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[45495]: audit 2022-04-22T18:49:08.349669+0000 mgr.y (mgr.34107) 2876 : audit [DBG] from='client.38436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[45495]: audit 2022-04-22T18:49:08.768787+0000 mon.c (mon.1) 162 : audit [DBG] from='client.? 172.21.15.115:0/3955663629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:09 smithi115 bash[45495]: audit 2022-04-22T18:49:08.884415+0000 mon.a (mon.0) 5916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:11 smithi121 bash[36902]: cluster 2022-04-22T18:49:10.262641+0000 mgr.y (mgr.34107) 2877 : cluster [DBG] pgmap v4435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:11 smithi115 bash[46934]: cluster 2022-04-22T18:49:10.262641+0000 mgr.y (mgr.34107) 2877 : cluster [DBG] pgmap v4435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:11 smithi115 bash[45495]: cluster 2022-04-22T18:49:10.262641+0000 mgr.y (mgr.34107) 2877 : cluster [DBG] pgmap v4435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:11.923Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:13 smithi115 bash[46934]: cluster 2022-04-22T18:49:12.263760+0000 mgr.y (mgr.34107) 2878 : cluster [DBG] pgmap v4436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:13 smithi115 bash[46934]: audit 2022-04-22T18:49:12.749526+0000 mon.a (mon.0) 5917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:13 smithi115 bash[45495]: cluster 2022-04-22T18:49:12.263760+0000 mgr.y (mgr.34107) 2878 : cluster [DBG] pgmap v4436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:13 smithi115 bash[45495]: audit 2022-04-22T18:49:12.749526+0000 mon.a (mon.0) 5917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:13 smithi121 bash[36902]: cluster 2022-04-22T18:49:12.263760+0000 mgr.y (mgr.34107) 2878 : cluster [DBG] pgmap v4436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:13.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:13 smithi121 bash[36902]: audit 2022-04-22T18:49:12.749526+0000 mon.a (mon.0) 5917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:14 smithi115 bash[46934]: audit 2022-04-22T18:49:13.887373+0000 mon.a (mon.0) 5918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:13.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:14.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:14 smithi115 bash[45495]: audit 2022-04-22T18:49:13.887373+0000 mon.a (mon.0) 5918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:14 smithi121 bash[36902]: audit 2022-04-22T18:49:13.887373+0000 mon.a (mon.0) 5918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[46934]: audit 2022-04-22T18:49:14.177046+0000 mon.a (mon.0) 5919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:49:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[46934]: audit 2022-04-22T18:49:14.178229+0000 mon.a (mon.0) 5920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:49:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[46934]: cluster 2022-04-22T18:49:14.264779+0000 mgr.y (mgr.34107) 2879 : cluster [DBG] pgmap v4437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[45495]: audit 2022-04-22T18:49:14.177046+0000 mon.a (mon.0) 5919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:49:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[45495]: audit 2022-04-22T18:49:14.178229+0000 mon.a (mon.0) 5920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:49:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:15 smithi115 bash[45495]: cluster 2022-04-22T18:49:14.264779+0000 mgr.y (mgr.34107) 2879 : cluster [DBG] pgmap v4437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:15 smithi121 bash[36902]: audit 2022-04-22T18:49:14.177046+0000 mon.a (mon.0) 5919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:15 smithi121 bash[36902]: audit 2022-04-22T18:49:14.178229+0000 mon.a (mon.0) 5920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:15 smithi121 bash[36902]: cluster 2022-04-22T18:49:14.264779+0000 mgr.y (mgr.34107) 2879 : cluster [DBG] pgmap v4437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:17.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:17 smithi121 bash[36902]: cluster 2022-04-22T18:49:16.265867+0000 mgr.y (mgr.34107) 2880 : cluster [DBG] pgmap v4438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:17 smithi115 bash[45495]: cluster 2022-04-22T18:49:16.265867+0000 mgr.y (mgr.34107) 2880 : cluster [DBG] pgmap v4438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:17 smithi115 bash[46934]: cluster 2022-04-22T18:49:16.265867+0000 mgr.y (mgr.34107) 2880 : cluster [DBG] pgmap v4438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:18 smithi121 bash[36902]: audit 2022-04-22T18:49:17.753425+0000 mon.a (mon.0) 5921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:18 smithi115 bash[46934]: audit 2022-04-22T18:49:17.753425+0000 mon.a (mon.0) 5921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:18 smithi115 bash[45495]: audit 2022-04-22T18:49:17.753425+0000 mon.a (mon.0) 5921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:19 smithi121 bash[36902]: cluster 2022-04-22T18:49:18.267008+0000 mgr.y (mgr.34107) 2881 : cluster [DBG] pgmap v4439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:19 smithi121 bash[36902]: audit 2022-04-22T18:49:18.890451+0000 mon.a (mon.0) 5922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:19 smithi115 bash[45495]: cluster 2022-04-22T18:49:18.267008+0000 mgr.y (mgr.34107) 2881 : cluster [DBG] pgmap v4439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:19 smithi115 bash[45495]: audit 2022-04-22T18:49:18.890451+0000 mon.a (mon.0) 5922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:19 smithi115 bash[46934]: cluster 2022-04-22T18:49:18.267008+0000 mgr.y (mgr.34107) 2881 : cluster [DBG] pgmap v4439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:19 smithi115 bash[46934]: audit 2022-04-22T18:49:18.890451+0000 mon.a (mon.0) 5922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:21 smithi121 bash[36902]: cluster 2022-04-22T18:49:20.268074+0000 mgr.y (mgr.34107) 2882 : cluster [DBG] pgmap v4440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:21 smithi115 bash[45495]: cluster 2022-04-22T18:49:20.268074+0000 mgr.y (mgr.34107) 2882 : cluster [DBG] pgmap v4440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:21 smithi115 bash[46934]: cluster 2022-04-22T18:49:20.268074+0000 mgr.y (mgr.34107) 2882 : cluster [DBG] pgmap v4440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:21.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:23 smithi115 bash[46934]: cluster 2022-04-22T18:49:22.269573+0000 mgr.y (mgr.34107) 2883 : cluster [DBG] pgmap v4441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:23 smithi115 bash[46934]: audit 2022-04-22T18:49:22.756754+0000 mon.a (mon.0) 5923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:23 smithi115 bash[45495]: cluster 2022-04-22T18:49:22.269573+0000 mgr.y (mgr.34107) 2883 : cluster [DBG] pgmap v4441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:23 smithi115 bash[45495]: audit 2022-04-22T18:49:22.756754+0000 mon.a (mon.0) 5923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:23 smithi121 bash[36902]: cluster 2022-04-22T18:49:22.269573+0000 mgr.y (mgr.34107) 2883 : cluster [DBG] pgmap v4441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:23 smithi121 bash[36902]: audit 2022-04-22T18:49:22.756754+0000 mon.a (mon.0) 5923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:24 smithi115 bash[46934]: audit 2022-04-22T18:49:23.893677+0000 mon.a (mon.0) 5924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:23.798Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:24.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:24 smithi115 bash[45495]: audit 2022-04-22T18:49:23.893677+0000 mon.a (mon.0) 5924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:24 smithi121 bash[36902]: audit 2022-04-22T18:49:23.893677+0000 mon.a (mon.0) 5924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:25 smithi115 bash[45495]: cluster 2022-04-22T18:49:24.270695+0000 mgr.y (mgr.34107) 2884 : cluster [DBG] pgmap v4442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:25 smithi115 bash[46934]: cluster 2022-04-22T18:49:24.270695+0000 mgr.y (mgr.34107) 2884 : cluster [DBG] pgmap v4442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:25 smithi121 bash[36902]: cluster 2022-04-22T18:49:24.270695+0000 mgr.y (mgr.34107) 2884 : cluster [DBG] pgmap v4442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:27 smithi121 bash[36902]: cluster 2022-04-22T18:49:26.271802+0000 mgr.y (mgr.34107) 2885 : cluster [DBG] pgmap v4443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:27 smithi115 bash[45495]: cluster 2022-04-22T18:49:26.271802+0000 mgr.y (mgr.34107) 2885 : cluster [DBG] pgmap v4443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:27 smithi115 bash[46934]: cluster 2022-04-22T18:49:26.271802+0000 mgr.y (mgr.34107) 2885 : cluster [DBG] pgmap v4443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:28 smithi121 bash[36902]: audit 2022-04-22T18:49:27.760370+0000 mon.a (mon.0) 5925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:28 smithi115 bash[45495]: audit 2022-04-22T18:49:27.760370+0000 mon.a (mon.0) 5925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:28 smithi115 bash[46934]: audit 2022-04-22T18:49:27.760370+0000 mon.a (mon.0) 5925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:29 smithi121 bash[36902]: cluster 2022-04-22T18:49:28.272878+0000 mgr.y (mgr.34107) 2886 : cluster [DBG] pgmap v4444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:29 smithi121 bash[36902]: audit 2022-04-22T18:49:28.896850+0000 mon.a (mon.0) 5926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:29.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:29 smithi115 bash[45495]: cluster 2022-04-22T18:49:28.272878+0000 mgr.y (mgr.34107) 2886 : cluster [DBG] pgmap v4444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:29.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:29 smithi115 bash[45495]: audit 2022-04-22T18:49:28.896850+0000 mon.a (mon.0) 5926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:29.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:29 smithi115 bash[46934]: cluster 2022-04-22T18:49:28.272878+0000 mgr.y (mgr.34107) 2886 : cluster [DBG] pgmap v4444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:29.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:29 smithi115 bash[46934]: audit 2022-04-22T18:49:28.896850+0000 mon.a (mon.0) 5926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:31 smithi121 bash[36902]: cluster 2022-04-22T18:49:30.273732+0000 mgr.y (mgr.34107) 2887 : cluster [DBG] pgmap v4445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:31 smithi115 bash[45495]: cluster 2022-04-22T18:49:30.273732+0000 mgr.y (mgr.34107) 2887 : cluster [DBG] pgmap v4445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:31 smithi115 bash[46934]: cluster 2022-04-22T18:49:30.273732+0000 mgr.y (mgr.34107) 2887 : cluster [DBG] pgmap v4445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:31.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:33 smithi115 bash[45495]: cluster 2022-04-22T18:49:32.274818+0000 mgr.y (mgr.34107) 2888 : cluster [DBG] pgmap v4446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:33 smithi115 bash[45495]: audit 2022-04-22T18:49:32.763796+0000 mon.a (mon.0) 5927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:33 smithi115 bash[46934]: cluster 2022-04-22T18:49:32.274818+0000 mgr.y (mgr.34107) 2888 : cluster [DBG] pgmap v4446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:33 smithi115 bash[46934]: audit 2022-04-22T18:49:32.763796+0000 mon.a (mon.0) 5927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:33 smithi121 bash[36902]: cluster 2022-04-22T18:49:32.274818+0000 mgr.y (mgr.34107) 2888 : cluster [DBG] pgmap v4446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:33 smithi121 bash[36902]: audit 2022-04-22T18:49:32.763796+0000 mon.a (mon.0) 5927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:34 smithi115 bash[45495]: audit 2022-04-22T18:49:33.899959+0000 mon.a (mon.0) 5928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:34 smithi115 bash[46934]: audit 2022-04-22T18:49:33.899959+0000 mon.a (mon.0) 5928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:33.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:34 smithi121 bash[36902]: audit 2022-04-22T18:49:33.899959+0000 mon.a (mon.0) 5928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:35 smithi121 bash[36902]: cluster 2022-04-22T18:49:34.275908+0000 mgr.y (mgr.34107) 2889 : cluster [DBG] pgmap v4447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:35 smithi115 bash[45495]: cluster 2022-04-22T18:49:34.275908+0000 mgr.y (mgr.34107) 2889 : cluster [DBG] pgmap v4447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:35 smithi115 bash[46934]: cluster 2022-04-22T18:49:34.275908+0000 mgr.y (mgr.34107) 2889 : cluster [DBG] pgmap v4447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:37 smithi121 bash[36902]: cluster 2022-04-22T18:49:36.276872+0000 mgr.y (mgr.34107) 2890 : cluster [DBG] pgmap v4448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:37 smithi115 bash[45495]: cluster 2022-04-22T18:49:36.276872+0000 mgr.y (mgr.34107) 2890 : cluster [DBG] pgmap v4448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:37 smithi115 bash[46934]: cluster 2022-04-22T18:49:36.276872+0000 mgr.y (mgr.34107) 2890 : cluster [DBG] pgmap v4448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:38 smithi121 bash[36902]: audit 2022-04-22T18:49:37.767640+0000 mon.a (mon.0) 5929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:38 smithi115 bash[46934]: audit 2022-04-22T18:49:37.767640+0000 mon.a (mon.0) 5929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:38 smithi115 bash[45495]: audit 2022-04-22T18:49:37.767640+0000 mon.a (mon.0) 5929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:39.158 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:49:39.502 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:49:39.502 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (72m) 10m ago 86m 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (86m) 57s ago 86m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (74m) 10m ago 87m 23.3M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (74m) 57s ago 87m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (81m) 57s ago 91m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (82m) 10m ago 93m 474M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (81m) 10m ago 94m 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (80m) 57s ago 92m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:49:39.503 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (81m) 10m ago 92m 157M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (73m) 10m ago 86m 11.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (73m) 57s ago 86m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (79m) 10m ago 91m 696M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (78m) 10m ago 90m 774M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (78m) 10m ago 90m 524M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (77m) 10m ago 89m 478M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:49:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (77m) 57s ago 89m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:49:39.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (76m) 57s ago 89m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:49:39.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (75m) 57s ago 88m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:49:39.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (75m) 57s ago 88m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:49:39.505 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (73m) 57s ago 86m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:49:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:39 smithi121 bash[36902]: cluster 2022-04-22T18:49:38.277904+0000 mgr.y (mgr.34107) 2891 : cluster [DBG] pgmap v4449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:39.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:39 smithi121 bash[36902]: audit 2022-04-22T18:49:38.902956+0000 mon.a (mon.0) 5930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:39 smithi115 bash[45495]: cluster 2022-04-22T18:49:38.277904+0000 mgr.y (mgr.34107) 2891 : cluster [DBG] pgmap v4449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:39 smithi115 bash[45495]: audit 2022-04-22T18:49:38.902956+0000 mon.a (mon.0) 5930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:39 smithi115 bash[46934]: cluster 2022-04-22T18:49:38.277904+0000 mgr.y (mgr.34107) 2891 : cluster [DBG] pgmap v4449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:39 smithi115 bash[46934]: audit 2022-04-22T18:49:38.902956+0000 mon.a (mon.0) 5930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:49:39.904 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:39.905 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:49:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:49:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:40 smithi121 bash[36902]: audit 2022-04-22T18:49:39.134187+0000 mgr.y (mgr.34107) 2892 : audit [DBG] from='client.48382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:40 smithi121 bash[36902]: audit 2022-04-22T18:49:39.473061+0000 mgr.y (mgr.34107) 2893 : audit [DBG] from='client.48388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:40 smithi121 bash[36902]: audit 2022-04-22T18:49:39.895662+0000 mon.a (mon.0) 5931 : audit [DBG] from='client.? 172.21.15.115:0/3714129181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[45495]: audit 2022-04-22T18:49:39.134187+0000 mgr.y (mgr.34107) 2892 : audit [DBG] from='client.48382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[45495]: audit 2022-04-22T18:49:39.473061+0000 mgr.y (mgr.34107) 2893 : audit [DBG] from='client.48388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[45495]: audit 2022-04-22T18:49:39.895662+0000 mon.a (mon.0) 5931 : audit [DBG] from='client.? 172.21.15.115:0/3714129181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[46934]: audit 2022-04-22T18:49:39.134187+0000 mgr.y (mgr.34107) 2892 : audit [DBG] from='client.48382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[46934]: audit 2022-04-22T18:49:39.473061+0000 mgr.y (mgr.34107) 2893 : audit [DBG] from='client.48388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:49:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:40 smithi115 bash[46934]: audit 2022-04-22T18:49:39.895662+0000 mon.a (mon.0) 5931 : audit [DBG] from='client.? 172.21.15.115:0/3714129181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:49:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:41 smithi121 bash[36902]: cluster 2022-04-22T18:49:40.278960+0000 mgr.y (mgr.34107) 2894 : cluster [DBG] pgmap v4450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:41 smithi115 bash[45495]: cluster 2022-04-22T18:49:40.278960+0000 mgr.y (mgr.34107) 2894 : cluster [DBG] pgmap v4450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:41 smithi115 bash[46934]: cluster 2022-04-22T18:49:40.278960+0000 mgr.y (mgr.34107) 2894 : cluster [DBG] pgmap v4450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:41.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: cluster 2022-04-22T18:49:42.280067+0000 mgr.y (mgr.34107) 2895 : cluster [DBG] pgmap v4451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:42.674431+0000 mon.a (mon.0) 5932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:42.677521+0000 mon.a (mon.0) 5933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:42.679217+0000 mon.a (mon.0) 5934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:42.771412+0000 mon.a (mon.0) 5935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:43.000604+0000 mon.a (mon.0) 5936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:43 smithi121 bash[36902]: audit 2022-04-22T18:49:43.007301+0000 mon.a (mon.0) 5937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: cluster 2022-04-22T18:49:42.280067+0000 mgr.y (mgr.34107) 2895 : cluster [DBG] pgmap v4451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:42.674431+0000 mon.a (mon.0) 5932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:49:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:42.677521+0000 mon.a (mon.0) 5933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:49:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:42.679217+0000 mon.a (mon.0) 5934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:49:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:42.771412+0000 mon.a (mon.0) 5935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:43.000604+0000 mon.a (mon.0) 5936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[46934]: audit 2022-04-22T18:49:43.007301+0000 mon.a (mon.0) 5937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: cluster 2022-04-22T18:49:42.280067+0000 mgr.y (mgr.34107) 2895 : cluster [DBG] pgmap v4451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:42.674431+0000 mon.a (mon.0) 5932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:42.677521+0000 mon.a (mon.0) 5933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:42.679217+0000 mon.a (mon.0) 5934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:42.771412+0000 mon.a (mon.0) 5935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:43.000604+0000 mon.a (mon.0) 5936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[45495]: audit 2022-04-22T18:49:43.007301+0000 mon.a (mon.0) 5937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:44.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:44 smithi115 bash[45495]: audit 2022-04-22T18:49:43.905511+0000 mon.a (mon.0) 5938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:44.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:44 smithi115 bash[46934]: audit 2022-04-22T18:49:43.905511+0000 mon.a (mon.0) 5938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:43.801Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:44 smithi121 bash[36902]: audit 2022-04-22T18:49:43.905511+0000 mon.a (mon.0) 5938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:45 smithi115 bash[45495]: cluster 2022-04-22T18:49:44.280918+0000 mgr.y (mgr.34107) 2896 : cluster [DBG] pgmap v4452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:45 smithi115 bash[46934]: cluster 2022-04-22T18:49:44.280918+0000 mgr.y (mgr.34107) 2896 : cluster [DBG] pgmap v4452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:45 smithi121 bash[36902]: cluster 2022-04-22T18:49:44.280918+0000 mgr.y (mgr.34107) 2896 : cluster [DBG] pgmap v4452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:47.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:47 smithi115 bash[45495]: cluster 2022-04-22T18:49:46.281573+0000 mgr.y (mgr.34107) 2897 : cluster [DBG] pgmap v4453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:47.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:47 smithi115 bash[45495]: audit 2022-04-22T18:49:47.109820+0000 mon.a (mon.0) 5939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:47.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:47 smithi115 bash[46934]: cluster 2022-04-22T18:49:46.281573+0000 mgr.y (mgr.34107) 2897 : cluster [DBG] pgmap v4453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:47.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:47 smithi115 bash[46934]: audit 2022-04-22T18:49:47.109820+0000 mon.a (mon.0) 5939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:47 smithi121 bash[36902]: cluster 2022-04-22T18:49:46.281573+0000 mgr.y (mgr.34107) 2897 : cluster [DBG] pgmap v4453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:47 smithi121 bash[36902]: audit 2022-04-22T18:49:47.109820+0000 mon.a (mon.0) 5939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:48 smithi121 bash[36902]: audit 2022-04-22T18:49:47.463231+0000 mon.a (mon.0) 5940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:48 smithi121 bash[36902]: audit 2022-04-22T18:49:47.774956+0000 mon.a (mon.0) 5941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:48 smithi115 bash[45495]: audit 2022-04-22T18:49:47.463231+0000 mon.a (mon.0) 5940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:48 smithi115 bash[45495]: audit 2022-04-22T18:49:47.774956+0000 mon.a (mon.0) 5941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:48 smithi115 bash[46934]: audit 2022-04-22T18:49:47.463231+0000 mon.a (mon.0) 5940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:49:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:48 smithi115 bash[46934]: audit 2022-04-22T18:49:47.774956+0000 mon.a (mon.0) 5941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:49 smithi121 bash[36902]: cluster 2022-04-22T18:49:48.282632+0000 mgr.y (mgr.34107) 2898 : cluster [DBG] pgmap v4454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:49 smithi121 bash[36902]: audit 2022-04-22T18:49:48.908416+0000 mon.a (mon.0) 5942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:49 smithi115 bash[45495]: cluster 2022-04-22T18:49:48.282632+0000 mgr.y (mgr.34107) 2898 : cluster [DBG] pgmap v4454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:49 smithi115 bash[45495]: audit 2022-04-22T18:49:48.908416+0000 mon.a (mon.0) 5942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:49 smithi115 bash[46934]: cluster 2022-04-22T18:49:48.282632+0000 mgr.y (mgr.34107) 2898 : cluster [DBG] pgmap v4454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:49 smithi115 bash[46934]: audit 2022-04-22T18:49:48.908416+0000 mon.a (mon.0) 5942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:51 smithi121 bash[36902]: cluster 2022-04-22T18:49:50.283912+0000 mgr.y (mgr.34107) 2899 : cluster [DBG] pgmap v4455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:51 smithi115 bash[45495]: cluster 2022-04-22T18:49:50.283912+0000 mgr.y (mgr.34107) 2899 : cluster [DBG] pgmap v4455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:51 smithi115 bash[46934]: cluster 2022-04-22T18:49:50.283912+0000 mgr.y (mgr.34107) 2899 : cluster [DBG] pgmap v4455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:51.925Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:53 smithi115 bash[45495]: cluster 2022-04-22T18:49:52.285094+0000 mgr.y (mgr.34107) 2900 : cluster [DBG] pgmap v4456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:53 smithi115 bash[45495]: audit 2022-04-22T18:49:52.778869+0000 mon.a (mon.0) 5943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:53 smithi115 bash[46934]: cluster 2022-04-22T18:49:52.285094+0000 mgr.y (mgr.34107) 2900 : cluster [DBG] pgmap v4456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:53 smithi115 bash[46934]: audit 2022-04-22T18:49:52.778869+0000 mon.a (mon.0) 5943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:53 smithi121 bash[36902]: cluster 2022-04-22T18:49:52.285094+0000 mgr.y (mgr.34107) 2900 : cluster [DBG] pgmap v4456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:53 smithi121 bash[36902]: audit 2022-04-22T18:49:52.778869+0000 mon.a (mon.0) 5943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:54 smithi115 bash[45495]: audit 2022-04-22T18:49:53.911510+0000 mon.a (mon.0) 5944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:54.319 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:49:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:49:53.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:49:54.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:54 smithi115 bash[46934]: audit 2022-04-22T18:49:53.911510+0000 mon.a (mon.0) 5944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:54 smithi121 bash[36902]: audit 2022-04-22T18:49:53.911510+0000 mon.a (mon.0) 5944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:55 smithi121 bash[36902]: cluster 2022-04-22T18:49:54.285816+0000 mgr.y (mgr.34107) 2901 : cluster [DBG] pgmap v4457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:55 smithi115 bash[45495]: cluster 2022-04-22T18:49:54.285816+0000 mgr.y (mgr.34107) 2901 : cluster [DBG] pgmap v4457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:55 smithi115 bash[46934]: cluster 2022-04-22T18:49:54.285816+0000 mgr.y (mgr.34107) 2901 : cluster [DBG] pgmap v4457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:57 smithi121 bash[36902]: cluster 2022-04-22T18:49:56.286851+0000 mgr.y (mgr.34107) 2902 : cluster [DBG] pgmap v4458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:57 smithi115 bash[45495]: cluster 2022-04-22T18:49:56.286851+0000 mgr.y (mgr.34107) 2902 : cluster [DBG] pgmap v4458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:57 smithi115 bash[46934]: cluster 2022-04-22T18:49:56.286851+0000 mgr.y (mgr.34107) 2902 : cluster [DBG] pgmap v4458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:58 smithi121 bash[36902]: audit 2022-04-22T18:49:57.782693+0000 mon.a (mon.0) 5945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:58 smithi115 bash[45495]: audit 2022-04-22T18:49:57.782693+0000 mon.a (mon.0) 5945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:58 smithi115 bash[46934]: audit 2022-04-22T18:49:57.782693+0000 mon.a (mon.0) 5945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:49:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:59 smithi121 bash[36902]: cluster 2022-04-22T18:49:58.287986+0000 mgr.y (mgr.34107) 2903 : cluster [DBG] pgmap v4459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:49:59 smithi121 bash[36902]: audit 2022-04-22T18:49:58.914627+0000 mon.a (mon.0) 5946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:59 smithi115 bash[45495]: cluster 2022-04-22T18:49:58.287986+0000 mgr.y (mgr.34107) 2903 : cluster [DBG] pgmap v4459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:49:59 smithi115 bash[45495]: audit 2022-04-22T18:49:58.914627+0000 mon.a (mon.0) 5946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:49:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:59 smithi115 bash[46934]: cluster 2022-04-22T18:49:58.287986+0000 mgr.y (mgr.34107) 2903 : cluster [DBG] pgmap v4459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:49:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:49:59 smithi115 bash[46934]: audit 2022-04-22T18:49:58.914627+0000 mon.a (mon.0) 5946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:00 smithi121 bash[36902]: cluster 2022-04-22T18:50:00.000157+0000 mon.a (mon.0) 5947 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:50:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:00 smithi115 bash[45495]: cluster 2022-04-22T18:50:00.000157+0000 mon.a (mon.0) 5947 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:50:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:00 smithi115 bash[46934]: cluster 2022-04-22T18:50:00.000157+0000 mon.a (mon.0) 5947 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T18:50:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:01 smithi121 bash[36902]: cluster 2022-04-22T18:50:00.289066+0000 mgr.y (mgr.34107) 2904 : cluster [DBG] pgmap v4460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:01 smithi115 bash[45495]: cluster 2022-04-22T18:50:00.289066+0000 mgr.y (mgr.34107) 2904 : cluster [DBG] pgmap v4460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:01 smithi115 bash[46934]: cluster 2022-04-22T18:50:00.289066+0000 mgr.y (mgr.34107) 2904 : cluster [DBG] pgmap v4460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:01.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:03 smithi121 bash[36902]: cluster 2022-04-22T18:50:02.289971+0000 mgr.y (mgr.34107) 2905 : cluster [DBG] pgmap v4461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:03 smithi121 bash[36902]: audit 2022-04-22T18:50:02.786386+0000 mon.a (mon.0) 5948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:03 smithi115 bash[45495]: cluster 2022-04-22T18:50:02.289971+0000 mgr.y (mgr.34107) 2905 : cluster [DBG] pgmap v4461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:03 smithi115 bash[45495]: audit 2022-04-22T18:50:02.786386+0000 mon.a (mon.0) 5948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:03 smithi115 bash[46934]: cluster 2022-04-22T18:50:02.289971+0000 mgr.y (mgr.34107) 2905 : cluster [DBG] pgmap v4461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:03 smithi115 bash[46934]: audit 2022-04-22T18:50:02.786386+0000 mon.a (mon.0) 5948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:04 smithi115 bash[45495]: audit 2022-04-22T18:50:03.917847+0000 mon.a (mon.0) 5949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:03.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:04 smithi115 bash[46934]: audit 2022-04-22T18:50:03.917847+0000 mon.a (mon.0) 5949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:04 smithi121 bash[36902]: audit 2022-04-22T18:50:03.917847+0000 mon.a (mon.0) 5949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:05 smithi121 bash[36902]: cluster 2022-04-22T18:50:04.291062+0000 mgr.y (mgr.34107) 2906 : cluster [DBG] pgmap v4462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:05 smithi115 bash[45495]: cluster 2022-04-22T18:50:04.291062+0000 mgr.y (mgr.34107) 2906 : cluster [DBG] pgmap v4462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:05 smithi115 bash[46934]: cluster 2022-04-22T18:50:04.291062+0000 mgr.y (mgr.34107) 2906 : cluster [DBG] pgmap v4462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:07 smithi121 bash[36902]: cluster 2022-04-22T18:50:06.292086+0000 mgr.y (mgr.34107) 2907 : cluster [DBG] pgmap v4463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:07 smithi115 bash[45495]: cluster 2022-04-22T18:50:06.292086+0000 mgr.y (mgr.34107) 2907 : cluster [DBG] pgmap v4463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:07 smithi115 bash[46934]: cluster 2022-04-22T18:50:06.292086+0000 mgr.y (mgr.34107) 2907 : cluster [DBG] pgmap v4463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:08 smithi121 bash[36902]: audit 2022-04-22T18:50:07.790313+0000 mon.a (mon.0) 5950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:08 smithi115 bash[46934]: audit 2022-04-22T18:50:07.790313+0000 mon.a (mon.0) 5950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:08 smithi115 bash[45495]: audit 2022-04-22T18:50:07.790313+0000 mon.a (mon.0) 5950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:09 smithi121 bash[36902]: cluster 2022-04-22T18:50:08.293226+0000 mgr.y (mgr.34107) 2908 : cluster [DBG] pgmap v4464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:09 smithi121 bash[36902]: audit 2022-04-22T18:50:08.920936+0000 mon.a (mon.0) 5951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:09 smithi115 bash[45495]: cluster 2022-04-22T18:50:08.293226+0000 mgr.y (mgr.34107) 2908 : cluster [DBG] pgmap v4464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:09 smithi115 bash[45495]: audit 2022-04-22T18:50:08.920936+0000 mon.a (mon.0) 5951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:09 smithi115 bash[46934]: cluster 2022-04-22T18:50:08.293226+0000 mgr.y (mgr.34107) 2908 : cluster [DBG] pgmap v4464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:09 smithi115 bash[46934]: audit 2022-04-22T18:50:08.920936+0000 mon.a (mon.0) 5951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:10.278 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:50:10.619 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:50:10.623 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (73m) 23s ago 87m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:50:10.623 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (86m) 88s ago 86m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:50:10.623 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (75m) 23s ago 88m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (74m) 88s ago 88m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (82m) 88s ago 92m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (83m) 23s ago 94m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (82m) 23s ago 94m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (81m) 88s ago 92m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:50:10.624 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (81m) 23s ago 92m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (74m) 23s ago 87m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (74m) 88s ago 87m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (79m) 23s ago 91m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (79m) 23s ago 91m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (78m) 23s ago 90m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (78m) 23s ago 90m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (77m) 88s ago 90m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (77m) 88s ago 89m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:50:10.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (76m) 88s ago 89m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:50:10.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (75m) 88s ago 88m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:50:10.626 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (73m) 88s ago 87m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:50:11.017 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:50:11.017 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:50:11.017 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:50:11.017 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:11.017 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:50:11.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:50:11.019 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:50:11.019 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:50:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:11 smithi121 bash[36902]: audit 2022-04-22T18:50:10.253794+0000 mgr.y (mgr.34107) 2909 : audit [DBG] from='client.48400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:11 smithi121 bash[36902]: cluster 2022-04-22T18:50:10.294391+0000 mgr.y (mgr.34107) 2910 : cluster [DBG] pgmap v4465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:11 smithi121 bash[36902]: audit 2022-04-22T18:50:10.590484+0000 mgr.y (mgr.34107) 2911 : audit [DBG] from='client.48406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:11 smithi121 bash[36902]: audit 2022-04-22T18:50:11.008100+0000 mon.a (mon.0) 5952 : audit [DBG] from='client.? 172.21.15.115:0/1823794679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[46934]: audit 2022-04-22T18:50:10.253794+0000 mgr.y (mgr.34107) 2909 : audit [DBG] from='client.48400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[46934]: cluster 2022-04-22T18:50:10.294391+0000 mgr.y (mgr.34107) 2910 : cluster [DBG] pgmap v4465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[46934]: audit 2022-04-22T18:50:10.590484+0000 mgr.y (mgr.34107) 2911 : audit [DBG] from='client.48406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[46934]: audit 2022-04-22T18:50:11.008100+0000 mon.a (mon.0) 5952 : audit [DBG] from='client.? 172.21.15.115:0/1823794679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[45495]: audit 2022-04-22T18:50:10.253794+0000 mgr.y (mgr.34107) 2909 : audit [DBG] from='client.48400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[45495]: cluster 2022-04-22T18:50:10.294391+0000 mgr.y (mgr.34107) 2910 : cluster [DBG] pgmap v4465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[45495]: audit 2022-04-22T18:50:10.590484+0000 mgr.y (mgr.34107) 2911 : audit [DBG] from='client.48406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[45495]: audit 2022-04-22T18:50:11.008100+0000 mon.a (mon.0) 5952 : audit [DBG] from='client.? 172.21.15.115:0/1823794679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:11.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:13 smithi121 bash[36902]: cluster 2022-04-22T18:50:12.295568+0000 mgr.y (mgr.34107) 2912 : cluster [DBG] pgmap v4466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:13 smithi121 bash[36902]: audit 2022-04-22T18:50:12.794087+0000 mon.a (mon.0) 5953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:13 smithi115 bash[46934]: cluster 2022-04-22T18:50:12.295568+0000 mgr.y (mgr.34107) 2912 : cluster [DBG] pgmap v4466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:13 smithi115 bash[46934]: audit 2022-04-22T18:50:12.794087+0000 mon.a (mon.0) 5953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:13 smithi115 bash[45495]: cluster 2022-04-22T18:50:12.295568+0000 mgr.y (mgr.34107) 2912 : cluster [DBG] pgmap v4466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:13 smithi115 bash[45495]: audit 2022-04-22T18:50:12.794087+0000 mon.a (mon.0) 5953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:14 smithi115 bash[46934]: audit 2022-04-22T18:50:13.924233+0000 mon.a (mon.0) 5954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:13.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:14.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:14 smithi115 bash[45495]: audit 2022-04-22T18:50:13.924233+0000 mon.a (mon.0) 5954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:14 smithi121 bash[36902]: audit 2022-04-22T18:50:13.924233+0000 mon.a (mon.0) 5954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:15 smithi121 bash[36902]: audit 2022-04-22T18:50:14.186965+0000 mon.a (mon.0) 5955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:50:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:15 smithi121 bash[36902]: audit 2022-04-22T18:50:14.187436+0000 mon.a (mon.0) 5956 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:50:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:15 smithi121 bash[36902]: cluster 2022-04-22T18:50:14.296532+0000 mgr.y (mgr.34107) 2913 : cluster [DBG] pgmap v4467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[45495]: audit 2022-04-22T18:50:14.186965+0000 mon.a (mon.0) 5955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:50:15.577 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[45495]: audit 2022-04-22T18:50:14.187436+0000 mon.a (mon.0) 5956 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:50:15.577 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[45495]: cluster 2022-04-22T18:50:14.296532+0000 mgr.y (mgr.34107) 2913 : cluster [DBG] pgmap v4467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:15.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[46934]: audit 2022-04-22T18:50:14.186965+0000 mon.a (mon.0) 5955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:50:15.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[46934]: audit 2022-04-22T18:50:14.187436+0000 mon.a (mon.0) 5956 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:50:15.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:15 smithi115 bash[46934]: cluster 2022-04-22T18:50:14.296532+0000 mgr.y (mgr.34107) 2913 : cluster [DBG] pgmap v4467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:17 smithi121 bash[36902]: cluster 2022-04-22T18:50:16.297625+0000 mgr.y (mgr.34107) 2914 : cluster [DBG] pgmap v4468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:17 smithi115 bash[45495]: cluster 2022-04-22T18:50:16.297625+0000 mgr.y (mgr.34107) 2914 : cluster [DBG] pgmap v4468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:17 smithi115 bash[46934]: cluster 2022-04-22T18:50:16.297625+0000 mgr.y (mgr.34107) 2914 : cluster [DBG] pgmap v4468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:18 smithi121 bash[36902]: audit 2022-04-22T18:50:17.797663+0000 mon.a (mon.0) 5957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:18 smithi115 bash[45495]: audit 2022-04-22T18:50:17.797663+0000 mon.a (mon.0) 5957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:18 smithi115 bash[46934]: audit 2022-04-22T18:50:17.797663+0000 mon.a (mon.0) 5957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:19 smithi121 bash[36902]: cluster 2022-04-22T18:50:18.298607+0000 mgr.y (mgr.34107) 2915 : cluster [DBG] pgmap v4469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:19 smithi121 bash[36902]: audit 2022-04-22T18:50:18.927182+0000 mon.a (mon.0) 5958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:19 smithi115 bash[46934]: cluster 2022-04-22T18:50:18.298607+0000 mgr.y (mgr.34107) 2915 : cluster [DBG] pgmap v4469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:19 smithi115 bash[46934]: audit 2022-04-22T18:50:18.927182+0000 mon.a (mon.0) 5958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:19 smithi115 bash[45495]: cluster 2022-04-22T18:50:18.298607+0000 mgr.y (mgr.34107) 2915 : cluster [DBG] pgmap v4469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:19 smithi115 bash[45495]: audit 2022-04-22T18:50:18.927182+0000 mon.a (mon.0) 5958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:21 smithi121 bash[36902]: cluster 2022-04-22T18:50:20.299718+0000 mgr.y (mgr.34107) 2916 : cluster [DBG] pgmap v4470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:21 smithi115 bash[45495]: cluster 2022-04-22T18:50:20.299718+0000 mgr.y (mgr.34107) 2916 : cluster [DBG] pgmap v4470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:21 smithi115 bash[46934]: cluster 2022-04-22T18:50:20.299718+0000 mgr.y (mgr.34107) 2916 : cluster [DBG] pgmap v4470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:21.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:23 smithi121 bash[36902]: cluster 2022-04-22T18:50:22.300736+0000 mgr.y (mgr.34107) 2917 : cluster [DBG] pgmap v4471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:23 smithi121 bash[36902]: audit 2022-04-22T18:50:22.801303+0000 mon.a (mon.0) 5959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:23 smithi115 bash[46934]: cluster 2022-04-22T18:50:22.300736+0000 mgr.y (mgr.34107) 2917 : cluster [DBG] pgmap v4471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:23 smithi115 bash[46934]: audit 2022-04-22T18:50:22.801303+0000 mon.a (mon.0) 5959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:23 smithi115 bash[45495]: cluster 2022-04-22T18:50:22.300736+0000 mgr.y (mgr.34107) 2917 : cluster [DBG] pgmap v4471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:23 smithi115 bash[45495]: audit 2022-04-22T18:50:22.801303+0000 mon.a (mon.0) 5959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:24.067 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:23.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:24 smithi121 bash[36902]: audit 2022-04-22T18:50:23.930517+0000 mon.a (mon.0) 5960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:24 smithi115 bash[45495]: audit 2022-04-22T18:50:23.930517+0000 mon.a (mon.0) 5960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:24 smithi115 bash[46934]: audit 2022-04-22T18:50:23.930517+0000 mon.a (mon.0) 5960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:25 smithi121 bash[36902]: cluster 2022-04-22T18:50:24.301884+0000 mgr.y (mgr.34107) 2918 : cluster [DBG] pgmap v4472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:25 smithi115 bash[45495]: cluster 2022-04-22T18:50:24.301884+0000 mgr.y (mgr.34107) 2918 : cluster [DBG] pgmap v4472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:25 smithi115 bash[46934]: cluster 2022-04-22T18:50:24.301884+0000 mgr.y (mgr.34107) 2918 : cluster [DBG] pgmap v4472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:27 smithi121 bash[36902]: cluster 2022-04-22T18:50:26.303059+0000 mgr.y (mgr.34107) 2919 : cluster [DBG] pgmap v4473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:27 smithi115 bash[45495]: cluster 2022-04-22T18:50:26.303059+0000 mgr.y (mgr.34107) 2919 : cluster [DBG] pgmap v4473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:27 smithi115 bash[46934]: cluster 2022-04-22T18:50:26.303059+0000 mgr.y (mgr.34107) 2919 : cluster [DBG] pgmap v4473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:28 smithi121 bash[36902]: audit 2022-04-22T18:50:27.804812+0000 mon.a (mon.0) 5961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:28 smithi115 bash[45495]: audit 2022-04-22T18:50:27.804812+0000 mon.a (mon.0) 5961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:28 smithi115 bash[46934]: audit 2022-04-22T18:50:27.804812+0000 mon.a (mon.0) 5961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:29 smithi121 bash[36902]: cluster 2022-04-22T18:50:28.304240+0000 mgr.y (mgr.34107) 2920 : cluster [DBG] pgmap v4474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:29 smithi121 bash[36902]: audit 2022-04-22T18:50:28.933738+0000 mon.a (mon.0) 5962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:29 smithi115 bash[45495]: cluster 2022-04-22T18:50:28.304240+0000 mgr.y (mgr.34107) 2920 : cluster [DBG] pgmap v4474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:29 smithi115 bash[45495]: audit 2022-04-22T18:50:28.933738+0000 mon.a (mon.0) 5962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:29 smithi115 bash[46934]: cluster 2022-04-22T18:50:28.304240+0000 mgr.y (mgr.34107) 2920 : cluster [DBG] pgmap v4474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:29 smithi115 bash[46934]: audit 2022-04-22T18:50:28.933738+0000 mon.a (mon.0) 5962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:31 smithi121 bash[36902]: cluster 2022-04-22T18:50:30.305318+0000 mgr.y (mgr.34107) 2921 : cluster [DBG] pgmap v4475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:31 smithi115 bash[45495]: cluster 2022-04-22T18:50:30.305318+0000 mgr.y (mgr.34107) 2921 : cluster [DBG] pgmap v4475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:31 smithi115 bash[46934]: cluster 2022-04-22T18:50:30.305318+0000 mgr.y (mgr.34107) 2921 : cluster [DBG] pgmap v4475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:31.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:33 smithi121 bash[36902]: cluster 2022-04-22T18:50:32.306456+0000 mgr.y (mgr.34107) 2922 : cluster [DBG] pgmap v4476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:33 smithi121 bash[36902]: audit 2022-04-22T18:50:32.808559+0000 mon.a (mon.0) 5963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:33 smithi115 bash[45495]: cluster 2022-04-22T18:50:32.306456+0000 mgr.y (mgr.34107) 2922 : cluster [DBG] pgmap v4476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:33 smithi115 bash[45495]: audit 2022-04-22T18:50:32.808559+0000 mon.a (mon.0) 5963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:33 smithi115 bash[46934]: cluster 2022-04-22T18:50:32.306456+0000 mgr.y (mgr.34107) 2922 : cluster [DBG] pgmap v4476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:33 smithi115 bash[46934]: audit 2022-04-22T18:50:32.808559+0000 mon.a (mon.0) 5963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:34.066 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:33.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:34 smithi121 bash[36902]: audit 2022-04-22T18:50:33.936698+0000 mon.a (mon.0) 5964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:34 smithi115 bash[45495]: audit 2022-04-22T18:50:33.936698+0000 mon.a (mon.0) 5964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:34 smithi115 bash[46934]: audit 2022-04-22T18:50:33.936698+0000 mon.a (mon.0) 5964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:35.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:35 smithi121 bash[36902]: cluster 2022-04-22T18:50:34.307579+0000 mgr.y (mgr.34107) 2923 : cluster [DBG] pgmap v4477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:35 smithi115 bash[45495]: cluster 2022-04-22T18:50:34.307579+0000 mgr.y (mgr.34107) 2923 : cluster [DBG] pgmap v4477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:35 smithi115 bash[46934]: cluster 2022-04-22T18:50:34.307579+0000 mgr.y (mgr.34107) 2923 : cluster [DBG] pgmap v4477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:37 smithi121 bash[36902]: cluster 2022-04-22T18:50:36.308637+0000 mgr.y (mgr.34107) 2924 : cluster [DBG] pgmap v4478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:37 smithi115 bash[45495]: cluster 2022-04-22T18:50:36.308637+0000 mgr.y (mgr.34107) 2924 : cluster [DBG] pgmap v4478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:37 smithi115 bash[46934]: cluster 2022-04-22T18:50:36.308637+0000 mgr.y (mgr.34107) 2924 : cluster [DBG] pgmap v4478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:38 smithi121 bash[36902]: audit 2022-04-22T18:50:37.812264+0000 mon.a (mon.0) 5965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:38 smithi115 bash[45495]: audit 2022-04-22T18:50:37.812264+0000 mon.a (mon.0) 5965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:38 smithi115 bash[46934]: audit 2022-04-22T18:50:37.812264+0000 mon.a (mon.0) 5965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:39 smithi121 bash[36902]: cluster 2022-04-22T18:50:38.309752+0000 mgr.y (mgr.34107) 2925 : cluster [DBG] pgmap v4479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:39 smithi121 bash[36902]: audit 2022-04-22T18:50:38.939999+0000 mon.a (mon.0) 5966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:39 smithi115 bash[46934]: cluster 2022-04-22T18:50:38.309752+0000 mgr.y (mgr.34107) 2925 : cluster [DBG] pgmap v4479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:39 smithi115 bash[46934]: audit 2022-04-22T18:50:38.939999+0000 mon.a (mon.0) 5966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:39 smithi115 bash[45495]: cluster 2022-04-22T18:50:38.309752+0000 mgr.y (mgr.34107) 2925 : cluster [DBG] pgmap v4479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:39 smithi115 bash[45495]: audit 2022-04-22T18:50:38.939999+0000 mon.a (mon.0) 5966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:50:41.732 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:50:41.733 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (73m) 54s ago 87m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:50:41.733 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (87m) 119s ago 87m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:50:41.733 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (75m) 54s ago 88m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:50:41.733 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (75m) 119s ago 88m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:50:41.734 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (83m) 119s ago 92m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:50:41.734 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (83m) 54s ago 94m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:50:41.734 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (82m) 54s ago 95m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:50:41.734 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (81m) 119s ago 93m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:50:41.735 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (82m) 54s ago 93m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:50:41.735 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (74m) 54s ago 87m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:50:41.735 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (74m) 119s ago 87m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:50:41.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (80m) 54s ago 92m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (79m) 54s ago 91m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (79m) 54s ago 91m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (78m) 54s ago 90m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (78m) 119s ago 90m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (77m) 119s ago 90m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:50:41.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (76m) 119s ago 89m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:50:41.737 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (76m) 119s ago 89m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:50:41.737 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (74m) 119s ago 87m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:50:41.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:41 smithi121 bash[36902]: cluster 2022-04-22T18:50:40.310886+0000 mgr.y (mgr.34107) 2926 : cluster [DBG] pgmap v4480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:41 smithi115 bash[45495]: cluster 2022-04-22T18:50:40.310886+0000 mgr.y (mgr.34107) 2926 : cluster [DBG] pgmap v4480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:41 smithi115 bash[46934]: cluster 2022-04-22T18:50:40.310886+0000 mgr.y (mgr.34107) 2926 : cluster [DBG] pgmap v4480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:42.149 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:50:42.149 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:50:42.149 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:50:42.150 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:50:42.151 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:50:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:41.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:42 smithi121 bash[36902]: audit 2022-04-22T18:50:41.362659+0000 mgr.y (mgr.34107) 2927 : audit [DBG] from='client.48418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:42 smithi121 bash[36902]: audit 2022-04-22T18:50:41.702115+0000 mgr.y (mgr.34107) 2928 : audit [DBG] from='client.38490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:42 smithi121 bash[36902]: audit 2022-04-22T18:50:42.139761+0000 mon.a (mon.0) 5967 : audit [DBG] from='client.? 172.21.15.115:0/2944824272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:42.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[45495]: audit 2022-04-22T18:50:41.362659+0000 mgr.y (mgr.34107) 2927 : audit [DBG] from='client.48418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[45495]: audit 2022-04-22T18:50:41.702115+0000 mgr.y (mgr.34107) 2928 : audit [DBG] from='client.38490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[45495]: audit 2022-04-22T18:50:42.139761+0000 mon.a (mon.0) 5967 : audit [DBG] from='client.? 172.21.15.115:0/2944824272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[46934]: audit 2022-04-22T18:50:41.362659+0000 mgr.y (mgr.34107) 2927 : audit [DBG] from='client.48418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[46934]: audit 2022-04-22T18:50:41.702115+0000 mgr.y (mgr.34107) 2928 : audit [DBG] from='client.38490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:50:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:42 smithi115 bash[46934]: audit 2022-04-22T18:50:42.139761+0000 mon.a (mon.0) 5967 : audit [DBG] from='client.? 172.21.15.115:0/2944824272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:50:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:43 smithi121 bash[36902]: cluster 2022-04-22T18:50:42.311933+0000 mgr.y (mgr.34107) 2929 : cluster [DBG] pgmap v4481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:43.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:43 smithi121 bash[36902]: audit 2022-04-22T18:50:42.815858+0000 mon.a (mon.0) 5968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:43 smithi115 bash[45495]: cluster 2022-04-22T18:50:42.311933+0000 mgr.y (mgr.34107) 2929 : cluster [DBG] pgmap v4481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:43 smithi115 bash[45495]: audit 2022-04-22T18:50:42.815858+0000 mon.a (mon.0) 5968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:43.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:43 smithi115 bash[46934]: cluster 2022-04-22T18:50:42.311933+0000 mgr.y (mgr.34107) 2929 : cluster [DBG] pgmap v4481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:43.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:43 smithi115 bash[46934]: audit 2022-04-22T18:50:42.815858+0000 mon.a (mon.0) 5968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:43.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:44 smithi121 bash[36902]: audit 2022-04-22T18:50:43.943138+0000 mon.a (mon.0) 5969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:44 smithi115 bash[45495]: audit 2022-04-22T18:50:43.943138+0000 mon.a (mon.0) 5969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:44 smithi115 bash[46934]: audit 2022-04-22T18:50:43.943138+0000 mon.a (mon.0) 5969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:45 smithi121 bash[36902]: cluster 2022-04-22T18:50:44.313093+0000 mgr.y (mgr.34107) 2930 : cluster [DBG] pgmap v4482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:45 smithi115 bash[45495]: cluster 2022-04-22T18:50:44.313093+0000 mgr.y (mgr.34107) 2930 : cluster [DBG] pgmap v4482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:45 smithi115 bash[46934]: cluster 2022-04-22T18:50:44.313093+0000 mgr.y (mgr.34107) 2930 : cluster [DBG] pgmap v4482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:47 smithi121 bash[36902]: cluster 2022-04-22T18:50:46.314175+0000 mgr.y (mgr.34107) 2931 : cluster [DBG] pgmap v4483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:47 smithi115 bash[45495]: cluster 2022-04-22T18:50:46.314175+0000 mgr.y (mgr.34107) 2931 : cluster [DBG] pgmap v4483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:47 smithi115 bash[46934]: cluster 2022-04-22T18:50:46.314175+0000 mgr.y (mgr.34107) 2931 : cluster [DBG] pgmap v4483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:48 smithi121 bash[36902]: audit 2022-04-22T18:50:47.470772+0000 mon.a (mon.0) 5970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:50:48.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:48 smithi121 bash[36902]: audit 2022-04-22T18:50:47.473761+0000 mon.a (mon.0) 5971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:50:48.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:48 smithi121 bash[36902]: audit 2022-04-22T18:50:47.475605+0000 mon.a (mon.0) 5972 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:50:48.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:48 smithi121 bash[36902]: audit 2022-04-22T18:50:47.819447+0000 mon.a (mon.0) 5973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:48.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:48 smithi121 bash[36902]: audit 2022-04-22T18:50:47.839513+0000 mon.a (mon.0) 5974 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[45495]: audit 2022-04-22T18:50:47.470772+0000 mon.a (mon.0) 5970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[45495]: audit 2022-04-22T18:50:47.473761+0000 mon.a (mon.0) 5971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[45495]: audit 2022-04-22T18:50:47.475605+0000 mon.a (mon.0) 5972 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[45495]: audit 2022-04-22T18:50:47.819447+0000 mon.a (mon.0) 5973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[45495]: audit 2022-04-22T18:50:47.839513+0000 mon.a (mon.0) 5974 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[46934]: audit 2022-04-22T18:50:47.470772+0000 mon.a (mon.0) 5970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[46934]: audit 2022-04-22T18:50:47.473761+0000 mon.a (mon.0) 5971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[46934]: audit 2022-04-22T18:50:47.475605+0000 mon.a (mon.0) 5972 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[46934]: audit 2022-04-22T18:50:47.819447+0000 mon.a (mon.0) 5973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:48 smithi115 bash[46934]: audit 2022-04-22T18:50:47.839513+0000 mon.a (mon.0) 5974 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:50:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:49 smithi121 bash[36902]: cluster 2022-04-22T18:50:48.315337+0000 mgr.y (mgr.34107) 2932 : cluster [DBG] pgmap v4484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:49 smithi121 bash[36902]: audit 2022-04-22T18:50:48.946212+0000 mon.a (mon.0) 5975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:49 smithi115 bash[46934]: cluster 2022-04-22T18:50:48.315337+0000 mgr.y (mgr.34107) 2932 : cluster [DBG] pgmap v4484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:49 smithi115 bash[46934]: audit 2022-04-22T18:50:48.946212+0000 mon.a (mon.0) 5975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:49 smithi115 bash[45495]: cluster 2022-04-22T18:50:48.315337+0000 mgr.y (mgr.34107) 2932 : cluster [DBG] pgmap v4484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:49 smithi115 bash[45495]: audit 2022-04-22T18:50:48.946212+0000 mon.a (mon.0) 5975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:51 smithi121 bash[36902]: cluster 2022-04-22T18:50:50.316370+0000 mgr.y (mgr.34107) 2933 : cluster [DBG] pgmap v4485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:52.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:51 smithi115 bash[45495]: cluster 2022-04-22T18:50:50.316370+0000 mgr.y (mgr.34107) 2933 : cluster [DBG] pgmap v4485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:51.930Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:52.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:51 smithi115 bash[46934]: cluster 2022-04-22T18:50:50.316370+0000 mgr.y (mgr.34107) 2933 : cluster [DBG] pgmap v4485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:53 smithi121 bash[36902]: cluster 2022-04-22T18:50:52.317490+0000 mgr.y (mgr.34107) 2934 : cluster [DBG] pgmap v4486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:53 smithi121 bash[36902]: audit 2022-04-22T18:50:52.822754+0000 mon.a (mon.0) 5976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:53 smithi115 bash[45495]: cluster 2022-04-22T18:50:52.317490+0000 mgr.y (mgr.34107) 2934 : cluster [DBG] pgmap v4486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:53 smithi115 bash[45495]: audit 2022-04-22T18:50:52.822754+0000 mon.a (mon.0) 5976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:53 smithi115 bash[46934]: cluster 2022-04-22T18:50:52.317490+0000 mgr.y (mgr.34107) 2934 : cluster [DBG] pgmap v4486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:53 smithi115 bash[46934]: audit 2022-04-22T18:50:52.822754+0000 mon.a (mon.0) 5976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:54.072 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:50:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:50:53.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:50:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:54 smithi121 bash[36902]: audit 2022-04-22T18:50:53.949390+0000 mon.a (mon.0) 5977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:54 smithi115 bash[45495]: audit 2022-04-22T18:50:53.949390+0000 mon.a (mon.0) 5977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:54 smithi115 bash[46934]: audit 2022-04-22T18:50:53.949390+0000 mon.a (mon.0) 5977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:55 smithi121 bash[36902]: cluster 2022-04-22T18:50:54.318540+0000 mgr.y (mgr.34107) 2935 : cluster [DBG] pgmap v4487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:55 smithi115 bash[45495]: cluster 2022-04-22T18:50:54.318540+0000 mgr.y (mgr.34107) 2935 : cluster [DBG] pgmap v4487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:55 smithi115 bash[46934]: cluster 2022-04-22T18:50:54.318540+0000 mgr.y (mgr.34107) 2935 : cluster [DBG] pgmap v4487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:57 smithi121 bash[36902]: cluster 2022-04-22T18:50:56.319640+0000 mgr.y (mgr.34107) 2936 : cluster [DBG] pgmap v4488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:57 smithi115 bash[45495]: cluster 2022-04-22T18:50:56.319640+0000 mgr.y (mgr.34107) 2936 : cluster [DBG] pgmap v4488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:57 smithi115 bash[46934]: cluster 2022-04-22T18:50:56.319640+0000 mgr.y (mgr.34107) 2936 : cluster [DBG] pgmap v4488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:58 smithi121 bash[36902]: audit 2022-04-22T18:50:57.826377+0000 mon.a (mon.0) 5978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:58 smithi115 bash[45495]: audit 2022-04-22T18:50:57.826377+0000 mon.a (mon.0) 5978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:58 smithi115 bash[46934]: audit 2022-04-22T18:50:57.826377+0000 mon.a (mon.0) 5978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:50:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:59 smithi121 bash[36902]: cluster 2022-04-22T18:50:58.320717+0000 mgr.y (mgr.34107) 2937 : cluster [DBG] pgmap v4489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:50:59 smithi121 bash[36902]: audit 2022-04-22T18:50:58.952525+0000 mon.a (mon.0) 5979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:59 smithi115 bash[45495]: cluster 2022-04-22T18:50:58.320717+0000 mgr.y (mgr.34107) 2937 : cluster [DBG] pgmap v4489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:50:59 smithi115 bash[45495]: audit 2022-04-22T18:50:58.952525+0000 mon.a (mon.0) 5979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:50:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:59 smithi115 bash[46934]: cluster 2022-04-22T18:50:58.320717+0000 mgr.y (mgr.34107) 2937 : cluster [DBG] pgmap v4489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:50:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:50:59 smithi115 bash[46934]: audit 2022-04-22T18:50:58.952525+0000 mon.a (mon.0) 5979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:01 smithi121 bash[36902]: cluster 2022-04-22T18:51:00.321837+0000 mgr.y (mgr.34107) 2938 : cluster [DBG] pgmap v4490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:01 smithi115 bash[45495]: cluster 2022-04-22T18:51:00.321837+0000 mgr.y (mgr.34107) 2938 : cluster [DBG] pgmap v4490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:01 smithi115 bash[46934]: cluster 2022-04-22T18:51:00.321837+0000 mgr.y (mgr.34107) 2938 : cluster [DBG] pgmap v4490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:01.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:03 smithi121 bash[36902]: cluster 2022-04-22T18:51:02.322916+0000 mgr.y (mgr.34107) 2939 : cluster [DBG] pgmap v4491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:03 smithi121 bash[36902]: audit 2022-04-22T18:51:02.830250+0000 mon.a (mon.0) 5980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:03 smithi115 bash[46934]: cluster 2022-04-22T18:51:02.322916+0000 mgr.y (mgr.34107) 2939 : cluster [DBG] pgmap v4491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:03 smithi115 bash[46934]: audit 2022-04-22T18:51:02.830250+0000 mon.a (mon.0) 5980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:03 smithi115 bash[45495]: cluster 2022-04-22T18:51:02.322916+0000 mgr.y (mgr.34107) 2939 : cluster [DBG] pgmap v4491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:03 smithi115 bash[45495]: audit 2022-04-22T18:51:02.830250+0000 mon.a (mon.0) 5980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:04.073 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:03.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:04 smithi121 bash[36902]: audit 2022-04-22T18:51:03.955590+0000 mon.a (mon.0) 5981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:04 smithi115 bash[46934]: audit 2022-04-22T18:51:03.955590+0000 mon.a (mon.0) 5981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:04 smithi115 bash[45495]: audit 2022-04-22T18:51:03.955590+0000 mon.a (mon.0) 5981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:05 smithi121 bash[36902]: cluster 2022-04-22T18:51:04.324062+0000 mgr.y (mgr.34107) 2940 : cluster [DBG] pgmap v4492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:05 smithi115 bash[46934]: cluster 2022-04-22T18:51:04.324062+0000 mgr.y (mgr.34107) 2940 : cluster [DBG] pgmap v4492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:05.588 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:05 smithi115 bash[45495]: cluster 2022-04-22T18:51:04.324062+0000 mgr.y (mgr.34107) 2940 : cluster [DBG] pgmap v4492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:07 smithi121 bash[36902]: cluster 2022-04-22T18:51:06.325127+0000 mgr.y (mgr.34107) 2941 : cluster [DBG] pgmap v4493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:07 smithi115 bash[45495]: cluster 2022-04-22T18:51:06.325127+0000 mgr.y (mgr.34107) 2941 : cluster [DBG] pgmap v4493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:07 smithi115 bash[46934]: cluster 2022-04-22T18:51:06.325127+0000 mgr.y (mgr.34107) 2941 : cluster [DBG] pgmap v4493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:08 smithi121 bash[36902]: audit 2022-04-22T18:51:07.834116+0000 mon.a (mon.0) 5982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:08 smithi115 bash[45495]: audit 2022-04-22T18:51:07.834116+0000 mon.a (mon.0) 5982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:08 smithi115 bash[46934]: audit 2022-04-22T18:51:07.834116+0000 mon.a (mon.0) 5982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:09 smithi121 bash[36902]: cluster 2022-04-22T18:51:08.326255+0000 mgr.y (mgr.34107) 2942 : cluster [DBG] pgmap v4494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:09 smithi121 bash[36902]: audit 2022-04-22T18:51:08.958514+0000 mon.a (mon.0) 5983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:09 smithi115 bash[45495]: cluster 2022-04-22T18:51:08.326255+0000 mgr.y (mgr.34107) 2942 : cluster [DBG] pgmap v4494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:09 smithi115 bash[45495]: audit 2022-04-22T18:51:08.958514+0000 mon.a (mon.0) 5983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:09 smithi115 bash[46934]: cluster 2022-04-22T18:51:08.326255+0000 mgr.y (mgr.34107) 2942 : cluster [DBG] pgmap v4494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:09 smithi115 bash[46934]: audit 2022-04-22T18:51:08.958514+0000 mon.a (mon.0) 5983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:11 smithi121 bash[36902]: cluster 2022-04-22T18:51:10.327347+0000 mgr.y (mgr.34107) 2943 : cluster [DBG] pgmap v4495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:11 smithi115 bash[46934]: cluster 2022-04-22T18:51:10.327347+0000 mgr.y (mgr.34107) 2943 : cluster [DBG] pgmap v4495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:11 smithi115 bash[45495]: cluster 2022-04-22T18:51:10.327347+0000 mgr.y (mgr.34107) 2943 : cluster [DBG] pgmap v4495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:11.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:12.525 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:51:12.862 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:51:12.862 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (74m) 85s ago 88m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:51:12.862 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (87m) 2m ago 87m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (76m) 85s ago 89m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (75m) 2m ago 89m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (83m) 2m ago 93m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (84m) 85s ago 95m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (83m) 85s ago 95m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (82m) 2m ago 93m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (82m) 85s ago 93m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:51:12.863 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (75m) 85s ago 88m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (75m) 2m ago 88m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (81m) 85s ago 92m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (80m) 85s ago 92m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (79m) 85s ago 91m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (79m) 85s ago 91m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (78m) 2m ago 91m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (78m) 2m ago 90m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (77m) 2m ago 90m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (76m) 2m ago 89m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:51:12.864 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (74m) 2m ago 88m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:51:13.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:51:13.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:51:13.269 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:51:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:13 smithi121 bash[36902]: cluster 2022-04-22T18:51:12.328341+0000 mgr.y (mgr.34107) 2944 : cluster [DBG] pgmap v4496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:13.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:13 smithi121 bash[36902]: audit 2022-04-22T18:51:12.499287+0000 mgr.y (mgr.34107) 2945 : audit [DBG] from='client.48436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:13.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:13 smithi121 bash[36902]: audit 2022-04-22T18:51:12.837641+0000 mon.a (mon.0) 5984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:13.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:13 smithi121 bash[36902]: audit 2022-04-22T18:51:12.839804+0000 mgr.y (mgr.34107) 2946 : audit [DBG] from='client.38508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:13.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[45495]: cluster 2022-04-22T18:51:12.328341+0000 mgr.y (mgr.34107) 2944 : cluster [DBG] pgmap v4496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[45495]: audit 2022-04-22T18:51:12.499287+0000 mgr.y (mgr.34107) 2945 : audit [DBG] from='client.48436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[45495]: audit 2022-04-22T18:51:12.837641+0000 mon.a (mon.0) 5984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[45495]: audit 2022-04-22T18:51:12.839804+0000 mgr.y (mgr.34107) 2946 : audit [DBG] from='client.38508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[46934]: cluster 2022-04-22T18:51:12.328341+0000 mgr.y (mgr.34107) 2944 : cluster [DBG] pgmap v4496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[46934]: audit 2022-04-22T18:51:12.499287+0000 mgr.y (mgr.34107) 2945 : audit [DBG] from='client.48436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[46934]: audit 2022-04-22T18:51:12.837641+0000 mon.a (mon.0) 5984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[46934]: audit 2022-04-22T18:51:12.839804+0000 mgr.y (mgr.34107) 2946 : audit [DBG] from='client.38508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:14.078 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:13.807Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:14 smithi121 bash[36902]: audit 2022-04-22T18:51:13.258034+0000 mon.c (mon.1) 163 : audit [DBG] from='client.? 172.21.15.115:0/2321584016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:14 smithi121 bash[36902]: audit 2022-04-22T18:51:13.961786+0000 mon.a (mon.0) 5985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:14 smithi115 bash[45495]: audit 2022-04-22T18:51:13.258034+0000 mon.c (mon.1) 163 : audit [DBG] from='client.? 172.21.15.115:0/2321584016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:14 smithi115 bash[45495]: audit 2022-04-22T18:51:13.961786+0000 mon.a (mon.0) 5985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:14 smithi115 bash[46934]: audit 2022-04-22T18:51:13.258034+0000 mon.c (mon.1) 163 : audit [DBG] from='client.? 172.21.15.115:0/2321584016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:14 smithi115 bash[46934]: audit 2022-04-22T18:51:13.961786+0000 mon.a (mon.0) 5985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:15 smithi121 bash[36902]: audit 2022-04-22T18:51:14.206184+0000 mon.a (mon.0) 5986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:51:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:15 smithi121 bash[36902]: audit 2022-04-22T18:51:14.206740+0000 mon.a (mon.0) 5987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:51:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:15 smithi121 bash[36902]: cluster 2022-04-22T18:51:14.329380+0000 mgr.y (mgr.34107) 2947 : cluster [DBG] pgmap v4497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[45495]: audit 2022-04-22T18:51:14.206184+0000 mon.a (mon.0) 5986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:51:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[45495]: audit 2022-04-22T18:51:14.206740+0000 mon.a (mon.0) 5987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:51:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[45495]: cluster 2022-04-22T18:51:14.329380+0000 mgr.y (mgr.34107) 2947 : cluster [DBG] pgmap v4497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[46934]: audit 2022-04-22T18:51:14.206184+0000 mon.a (mon.0) 5986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:51:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[46934]: audit 2022-04-22T18:51:14.206740+0000 mon.a (mon.0) 5987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:51:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:15 smithi115 bash[46934]: cluster 2022-04-22T18:51:14.329380+0000 mgr.y (mgr.34107) 2947 : cluster [DBG] pgmap v4497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:17.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:17 smithi121 bash[36902]: cluster 2022-04-22T18:51:16.330365+0000 mgr.y (mgr.34107) 2948 : cluster [DBG] pgmap v4498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:17 smithi115 bash[45495]: cluster 2022-04-22T18:51:16.330365+0000 mgr.y (mgr.34107) 2948 : cluster [DBG] pgmap v4498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:17 smithi115 bash[46934]: cluster 2022-04-22T18:51:16.330365+0000 mgr.y (mgr.34107) 2948 : cluster [DBG] pgmap v4498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:18 smithi121 bash[36902]: audit 2022-04-22T18:51:17.841593+0000 mon.a (mon.0) 5988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:18.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:18 smithi115 bash[45495]: audit 2022-04-22T18:51:17.841593+0000 mon.a (mon.0) 5988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:18 smithi115 bash[46934]: audit 2022-04-22T18:51:17.841593+0000 mon.a (mon.0) 5988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:19 smithi121 bash[36902]: cluster 2022-04-22T18:51:18.331540+0000 mgr.y (mgr.34107) 2949 : cluster [DBG] pgmap v4499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:19 smithi121 bash[36902]: audit 2022-04-22T18:51:18.965083+0000 mon.a (mon.0) 5989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:19 smithi115 bash[45495]: cluster 2022-04-22T18:51:18.331540+0000 mgr.y (mgr.34107) 2949 : cluster [DBG] pgmap v4499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:19 smithi115 bash[45495]: audit 2022-04-22T18:51:18.965083+0000 mon.a (mon.0) 5989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:19 smithi115 bash[46934]: cluster 2022-04-22T18:51:18.331540+0000 mgr.y (mgr.34107) 2949 : cluster [DBG] pgmap v4499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:19 smithi115 bash[46934]: audit 2022-04-22T18:51:18.965083+0000 mon.a (mon.0) 5989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:21 smithi121 bash[36902]: cluster 2022-04-22T18:51:20.332661+0000 mgr.y (mgr.34107) 2950 : cluster [DBG] pgmap v4500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:21 smithi115 bash[45495]: cluster 2022-04-22T18:51:20.332661+0000 mgr.y (mgr.34107) 2950 : cluster [DBG] pgmap v4500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:21 smithi115 bash[46934]: cluster 2022-04-22T18:51:20.332661+0000 mgr.y (mgr.34107) 2950 : cluster [DBG] pgmap v4500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:21.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:23 smithi121 bash[36902]: cluster 2022-04-22T18:51:22.333801+0000 mgr.y (mgr.34107) 2951 : cluster [DBG] pgmap v4501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:23 smithi121 bash[36902]: audit 2022-04-22T18:51:22.845365+0000 mon.a (mon.0) 5990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:23.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:23 smithi115 bash[45495]: cluster 2022-04-22T18:51:22.333801+0000 mgr.y (mgr.34107) 2951 : cluster [DBG] pgmap v4501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:23.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:23 smithi115 bash[45495]: audit 2022-04-22T18:51:22.845365+0000 mon.a (mon.0) 5990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:23.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:23 smithi115 bash[46934]: cluster 2022-04-22T18:51:22.333801+0000 mgr.y (mgr.34107) 2951 : cluster [DBG] pgmap v4501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:23.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:23 smithi115 bash[46934]: audit 2022-04-22T18:51:22.845365+0000 mon.a (mon.0) 5990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:24.081 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:23.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:24.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:24 smithi121 bash[36902]: audit 2022-04-22T18:51:23.968264+0000 mon.a (mon.0) 5991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:24 smithi115 bash[45495]: audit 2022-04-22T18:51:23.968264+0000 mon.a (mon.0) 5991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:24 smithi115 bash[46934]: audit 2022-04-22T18:51:23.968264+0000 mon.a (mon.0) 5991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:25 smithi121 bash[36902]: cluster 2022-04-22T18:51:24.334918+0000 mgr.y (mgr.34107) 2952 : cluster [DBG] pgmap v4502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:25 smithi115 bash[45495]: cluster 2022-04-22T18:51:24.334918+0000 mgr.y (mgr.34107) 2952 : cluster [DBG] pgmap v4502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:25 smithi115 bash[46934]: cluster 2022-04-22T18:51:24.334918+0000 mgr.y (mgr.34107) 2952 : cluster [DBG] pgmap v4502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:27 smithi121 bash[36902]: cluster 2022-04-22T18:51:26.336039+0000 mgr.y (mgr.34107) 2953 : cluster [DBG] pgmap v4503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:27 smithi115 bash[45495]: cluster 2022-04-22T18:51:26.336039+0000 mgr.y (mgr.34107) 2953 : cluster [DBG] pgmap v4503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:27 smithi115 bash[46934]: cluster 2022-04-22T18:51:26.336039+0000 mgr.y (mgr.34107) 2953 : cluster [DBG] pgmap v4503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:28 smithi121 bash[36902]: audit 2022-04-22T18:51:27.848999+0000 mon.a (mon.0) 5992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:28 smithi115 bash[45495]: audit 2022-04-22T18:51:27.848999+0000 mon.a (mon.0) 5992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:28 smithi115 bash[46934]: audit 2022-04-22T18:51:27.848999+0000 mon.a (mon.0) 5992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:29 smithi121 bash[36902]: cluster 2022-04-22T18:51:28.337211+0000 mgr.y (mgr.34107) 2954 : cluster [DBG] pgmap v4504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:29.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:29 smithi121 bash[36902]: audit 2022-04-22T18:51:28.971331+0000 mon.a (mon.0) 5993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:29.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:29 smithi115 bash[45495]: cluster 2022-04-22T18:51:28.337211+0000 mgr.y (mgr.34107) 2954 : cluster [DBG] pgmap v4504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:29.868 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:29 smithi115 bash[45495]: audit 2022-04-22T18:51:28.971331+0000 mon.a (mon.0) 5993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:29 smithi115 bash[46934]: cluster 2022-04-22T18:51:28.337211+0000 mgr.y (mgr.34107) 2954 : cluster [DBG] pgmap v4504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:29 smithi115 bash[46934]: audit 2022-04-22T18:51:28.971331+0000 mon.a (mon.0) 5993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:31 smithi121 bash[36902]: cluster 2022-04-22T18:51:30.338335+0000 mgr.y (mgr.34107) 2955 : cluster [DBG] pgmap v4505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:31 smithi115 bash[45495]: cluster 2022-04-22T18:51:30.338335+0000 mgr.y (mgr.34107) 2955 : cluster [DBG] pgmap v4505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:31 smithi115 bash[46934]: cluster 2022-04-22T18:51:30.338335+0000 mgr.y (mgr.34107) 2955 : cluster [DBG] pgmap v4505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:31.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:33 smithi121 bash[36902]: cluster 2022-04-22T18:51:32.339416+0000 mgr.y (mgr.34107) 2956 : cluster [DBG] pgmap v4506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:33 smithi121 bash[36902]: audit 2022-04-22T18:51:32.852428+0000 mon.a (mon.0) 5994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:33 smithi115 bash[45495]: cluster 2022-04-22T18:51:32.339416+0000 mgr.y (mgr.34107) 2956 : cluster [DBG] pgmap v4506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:33 smithi115 bash[45495]: audit 2022-04-22T18:51:32.852428+0000 mon.a (mon.0) 5994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:33 smithi115 bash[46934]: cluster 2022-04-22T18:51:32.339416+0000 mgr.y (mgr.34107) 2956 : cluster [DBG] pgmap v4506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:33 smithi115 bash[46934]: audit 2022-04-22T18:51:32.852428+0000 mon.a (mon.0) 5994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:34.084 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:33.807Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:34 smithi121 bash[36902]: audit 2022-04-22T18:51:33.974302+0000 mon.a (mon.0) 5995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:34 smithi115 bash[45495]: audit 2022-04-22T18:51:33.974302+0000 mon.a (mon.0) 5995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:34 smithi115 bash[46934]: audit 2022-04-22T18:51:33.974302+0000 mon.a (mon.0) 5995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:35 smithi121 bash[36902]: cluster 2022-04-22T18:51:34.340519+0000 mgr.y (mgr.34107) 2957 : cluster [DBG] pgmap v4507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:35.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:35 smithi115 bash[45495]: cluster 2022-04-22T18:51:34.340519+0000 mgr.y (mgr.34107) 2957 : cluster [DBG] pgmap v4507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:35.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:35 smithi115 bash[46934]: cluster 2022-04-22T18:51:34.340519+0000 mgr.y (mgr.34107) 2957 : cluster [DBG] pgmap v4507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:37 smithi121 bash[36902]: cluster 2022-04-22T18:51:36.341604+0000 mgr.y (mgr.34107) 2958 : cluster [DBG] pgmap v4508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:37 smithi115 bash[46934]: cluster 2022-04-22T18:51:36.341604+0000 mgr.y (mgr.34107) 2958 : cluster [DBG] pgmap v4508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:37 smithi115 bash[45495]: cluster 2022-04-22T18:51:36.341604+0000 mgr.y (mgr.34107) 2958 : cluster [DBG] pgmap v4508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:38 smithi121 bash[36902]: audit 2022-04-22T18:51:37.856223+0000 mon.a (mon.0) 5996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:38 smithi115 bash[45495]: audit 2022-04-22T18:51:37.856223+0000 mon.a (mon.0) 5996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:38 smithi115 bash[46934]: audit 2022-04-22T18:51:37.856223+0000 mon.a (mon.0) 5996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:39 smithi121 bash[36902]: cluster 2022-04-22T18:51:38.342754+0000 mgr.y (mgr.34107) 2959 : cluster [DBG] pgmap v4509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:39 smithi121 bash[36902]: audit 2022-04-22T18:51:38.977453+0000 mon.a (mon.0) 5997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:39 smithi115 bash[45495]: cluster 2022-04-22T18:51:38.342754+0000 mgr.y (mgr.34107) 2959 : cluster [DBG] pgmap v4509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:39 smithi115 bash[45495]: audit 2022-04-22T18:51:38.977453+0000 mon.a (mon.0) 5997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:39 smithi115 bash[46934]: cluster 2022-04-22T18:51:38.342754+0000 mgr.y (mgr.34107) 2959 : cluster [DBG] pgmap v4509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:39 smithi115 bash[46934]: audit 2022-04-22T18:51:38.977453+0000 mon.a (mon.0) 5997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:41 smithi121 bash[36902]: cluster 2022-04-22T18:51:40.343960+0000 mgr.y (mgr.34107) 2960 : cluster [DBG] pgmap v4510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:41 smithi115 bash[45495]: cluster 2022-04-22T18:51:40.343960+0000 mgr.y (mgr.34107) 2960 : cluster [DBG] pgmap v4510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:41 smithi115 bash[46934]: cluster 2022-04-22T18:51:40.343960+0000 mgr.y (mgr.34107) 2960 : cluster [DBG] pgmap v4510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:41.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:43 smithi121 bash[36902]: cluster 2022-04-22T18:51:42.345079+0000 mgr.y (mgr.34107) 2961 : cluster [DBG] pgmap v4511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:43.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:43 smithi121 bash[36902]: audit 2022-04-22T18:51:42.859907+0000 mon.a (mon.0) 5998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:43 smithi115 bash[46934]: cluster 2022-04-22T18:51:42.345079+0000 mgr.y (mgr.34107) 2961 : cluster [DBG] pgmap v4511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:43 smithi115 bash[46934]: audit 2022-04-22T18:51:42.859907+0000 mon.a (mon.0) 5998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:43.608 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:43 smithi115 bash[45495]: cluster 2022-04-22T18:51:42.345079+0000 mgr.y (mgr.34107) 2961 : cluster [DBG] pgmap v4511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:43.608 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:43 smithi115 bash[45495]: audit 2022-04-22T18:51:42.859907+0000 mon.a (mon.0) 5998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:43.644 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:51:43.991 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:51:43.991 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (74m) 116s ago 88m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:51:43.991 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (88m) 3m ago 88m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:51:43.991 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (76m) 116s ago 89m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:51:43.991 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (76m) 3m ago 89m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (84m) 3m ago 93m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (84m) 116s ago 96m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (83m) 116s ago 96m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (82m) 3m ago 94m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (83m) 116s ago 94m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (75m) 116s ago 88m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (75m) 3m ago 88m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (81m) 116s ago 93m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (80m) 116s ago 92m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:51:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (80m) 116s ago 92m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (79m) 116s ago 91m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (79m) 3m ago 91m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (78m) 3m ago 91m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (77m) 3m ago 90m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (77m) 3m ago 90m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:51:43.993 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (75m) 3m ago 88m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:51:44.091 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:43.807Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:44.400 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:51:44.402 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:51:44.406 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:51:44.406 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:51:44.406 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:51:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:44 smithi121 bash[36902]: audit 2022-04-22T18:51:43.618536+0000 mgr.y (mgr.34107) 2962 : audit [DBG] from='client.38520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:44 smithi121 bash[36902]: audit 2022-04-22T18:51:43.963588+0000 mgr.y (mgr.34107) 2963 : audit [DBG] from='client.48460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:44 smithi121 bash[36902]: audit 2022-04-22T18:51:43.980407+0000 mon.a (mon.0) 5999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[45495]: audit 2022-04-22T18:51:43.618536+0000 mgr.y (mgr.34107) 2962 : audit [DBG] from='client.38520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[45495]: audit 2022-04-22T18:51:43.963588+0000 mgr.y (mgr.34107) 2963 : audit [DBG] from='client.48460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[45495]: audit 2022-04-22T18:51:43.980407+0000 mon.a (mon.0) 5999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[46934]: audit 2022-04-22T18:51:43.618536+0000 mgr.y (mgr.34107) 2962 : audit [DBG] from='client.38520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[46934]: audit 2022-04-22T18:51:43.963588+0000 mgr.y (mgr.34107) 2963 : audit [DBG] from='client.48460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:51:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:44 smithi115 bash[46934]: audit 2022-04-22T18:51:43.980407+0000 mon.a (mon.0) 5999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:45 smithi121 bash[36902]: cluster 2022-04-22T18:51:44.346144+0000 mgr.y (mgr.34107) 2964 : cluster [DBG] pgmap v4512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:45 smithi121 bash[36902]: audit 2022-04-22T18:51:44.391396+0000 mon.a (mon.0) 6000 : audit [DBG] from='client.? 172.21.15.115:0/38231170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:45 smithi115 bash[45495]: cluster 2022-04-22T18:51:44.346144+0000 mgr.y (mgr.34107) 2964 : cluster [DBG] pgmap v4512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:45 smithi115 bash[45495]: audit 2022-04-22T18:51:44.391396+0000 mon.a (mon.0) 6000 : audit [DBG] from='client.? 172.21.15.115:0/38231170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:45 smithi115 bash[46934]: cluster 2022-04-22T18:51:44.346144+0000 mgr.y (mgr.34107) 2964 : cluster [DBG] pgmap v4512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:45 smithi115 bash[46934]: audit 2022-04-22T18:51:44.391396+0000 mon.a (mon.0) 6000 : audit [DBG] from='client.? 172.21.15.115:0/38231170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:51:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:47 smithi121 bash[36902]: cluster 2022-04-22T18:51:46.347213+0000 mgr.y (mgr.34107) 2965 : cluster [DBG] pgmap v4513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:47 smithi115 bash[45495]: cluster 2022-04-22T18:51:46.347213+0000 mgr.y (mgr.34107) 2965 : cluster [DBG] pgmap v4513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:47 smithi115 bash[46934]: cluster 2022-04-22T18:51:46.347213+0000 mgr.y (mgr.34107) 2965 : cluster [DBG] pgmap v4513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:48 smithi121 bash[36902]: audit 2022-04-22T18:51:47.849431+0000 mon.a (mon.0) 6001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:48 smithi121 bash[36902]: audit 2022-04-22T18:51:47.852629+0000 mon.a (mon.0) 6002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:48 smithi121 bash[36902]: audit 2022-04-22T18:51:47.854621+0000 mon.a (mon.0) 6003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:48 smithi121 bash[36902]: audit 2022-04-22T18:51:47.863294+0000 mon.a (mon.0) 6004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:48 smithi121 bash[36902]: audit 2022-04-22T18:51:48.233703+0000 mon.a (mon.0) 6005 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:51:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[45495]: audit 2022-04-22T18:51:47.849431+0000 mon.a (mon.0) 6001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:51:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[45495]: audit 2022-04-22T18:51:47.852629+0000 mon.a (mon.0) 6002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:51:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[45495]: audit 2022-04-22T18:51:47.854621+0000 mon.a (mon.0) 6003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[45495]: audit 2022-04-22T18:51:47.863294+0000 mon.a (mon.0) 6004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[45495]: audit 2022-04-22T18:51:48.233703+0000 mon.a (mon.0) 6005 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[46934]: audit 2022-04-22T18:51:47.849431+0000 mon.a (mon.0) 6001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[46934]: audit 2022-04-22T18:51:47.852629+0000 mon.a (mon.0) 6002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[46934]: audit 2022-04-22T18:51:47.854621+0000 mon.a (mon.0) 6003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[46934]: audit 2022-04-22T18:51:47.863294+0000 mon.a (mon.0) 6004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:48 smithi115 bash[46934]: audit 2022-04-22T18:51:48.233703+0000 mon.a (mon.0) 6005 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:51:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:49 smithi121 bash[36902]: cluster 2022-04-22T18:51:48.348353+0000 mgr.y (mgr.34107) 2966 : cluster [DBG] pgmap v4514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:49 smithi121 bash[36902]: audit 2022-04-22T18:51:48.983453+0000 mon.a (mon.0) 6006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:49 smithi115 bash[45495]: cluster 2022-04-22T18:51:48.348353+0000 mgr.y (mgr.34107) 2966 : cluster [DBG] pgmap v4514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:49.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:49 smithi115 bash[45495]: audit 2022-04-22T18:51:48.983453+0000 mon.a (mon.0) 6006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:49.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:49 smithi115 bash[46934]: cluster 2022-04-22T18:51:48.348353+0000 mgr.y (mgr.34107) 2966 : cluster [DBG] pgmap v4514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:49.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:49 smithi115 bash[46934]: audit 2022-04-22T18:51:48.983453+0000 mon.a (mon.0) 6006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:51 smithi121 bash[36902]: cluster 2022-04-22T18:51:50.349455+0000 mgr.y (mgr.34107) 2967 : cluster [DBG] pgmap v4515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:51 smithi115 bash[45495]: cluster 2022-04-22T18:51:50.349455+0000 mgr.y (mgr.34107) 2967 : cluster [DBG] pgmap v4515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:51 smithi115 bash[46934]: cluster 2022-04-22T18:51:50.349455+0000 mgr.y (mgr.34107) 2967 : cluster [DBG] pgmap v4515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:51.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:53.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:53 smithi121 bash[36902]: cluster 2022-04-22T18:51:52.350576+0000 mgr.y (mgr.34107) 2968 : cluster [DBG] pgmap v4516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:53.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:53 smithi121 bash[36902]: audit 2022-04-22T18:51:52.867095+0000 mon.a (mon.0) 6007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:53 smithi115 bash[45495]: cluster 2022-04-22T18:51:52.350576+0000 mgr.y (mgr.34107) 2968 : cluster [DBG] pgmap v4516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:53 smithi115 bash[45495]: audit 2022-04-22T18:51:52.867095+0000 mon.a (mon.0) 6007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:53 smithi115 bash[46934]: cluster 2022-04-22T18:51:52.350576+0000 mgr.y (mgr.34107) 2968 : cluster [DBG] pgmap v4516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:53 smithi115 bash[46934]: audit 2022-04-22T18:51:52.867095+0000 mon.a (mon.0) 6007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:54.091 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:51:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:51:53.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:51:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:54 smithi121 bash[36902]: audit 2022-04-22T18:51:53.986960+0000 mon.a (mon.0) 6008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:54 smithi115 bash[45495]: audit 2022-04-22T18:51:53.986960+0000 mon.a (mon.0) 6008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:54 smithi115 bash[46934]: audit 2022-04-22T18:51:53.986960+0000 mon.a (mon.0) 6008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:55.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:55 smithi121 bash[36902]: cluster 2022-04-22T18:51:54.351782+0000 mgr.y (mgr.34107) 2969 : cluster [DBG] pgmap v4517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:55 smithi115 bash[45495]: cluster 2022-04-22T18:51:54.351782+0000 mgr.y (mgr.34107) 2969 : cluster [DBG] pgmap v4517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:55.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:55 smithi115 bash[46934]: cluster 2022-04-22T18:51:54.351782+0000 mgr.y (mgr.34107) 2969 : cluster [DBG] pgmap v4517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:57 smithi121 bash[36902]: cluster 2022-04-22T18:51:56.352912+0000 mgr.y (mgr.34107) 2970 : cluster [DBG] pgmap v4518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:57.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:57 smithi115 bash[45495]: cluster 2022-04-22T18:51:56.352912+0000 mgr.y (mgr.34107) 2970 : cluster [DBG] pgmap v4518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:57 smithi115 bash[46934]: cluster 2022-04-22T18:51:56.352912+0000 mgr.y (mgr.34107) 2970 : cluster [DBG] pgmap v4518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:58.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:58 smithi121 bash[36902]: audit 2022-04-22T18:51:57.870837+0000 mon.a (mon.0) 6009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:58 smithi115 bash[45495]: audit 2022-04-22T18:51:57.870837+0000 mon.a (mon.0) 6009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:58 smithi115 bash[46934]: audit 2022-04-22T18:51:57.870837+0000 mon.a (mon.0) 6009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:51:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:59 smithi121 bash[36902]: cluster 2022-04-22T18:51:58.354046+0000 mgr.y (mgr.34107) 2971 : cluster [DBG] pgmap v4519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:51:59 smithi121 bash[36902]: audit 2022-04-22T18:51:58.990074+0000 mon.a (mon.0) 6010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:59 smithi115 bash[45495]: cluster 2022-04-22T18:51:58.354046+0000 mgr.y (mgr.34107) 2971 : cluster [DBG] pgmap v4519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:51:59 smithi115 bash[45495]: audit 2022-04-22T18:51:58.990074+0000 mon.a (mon.0) 6010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:51:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:59 smithi115 bash[46934]: cluster 2022-04-22T18:51:58.354046+0000 mgr.y (mgr.34107) 2971 : cluster [DBG] pgmap v4519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:51:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:51:59 smithi115 bash[46934]: audit 2022-04-22T18:51:58.990074+0000 mon.a (mon.0) 6010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:01 smithi121 bash[36902]: cluster 2022-04-22T18:52:00.355158+0000 mgr.y (mgr.34107) 2972 : cluster [DBG] pgmap v4520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:01 smithi115 bash[45495]: cluster 2022-04-22T18:52:00.355158+0000 mgr.y (mgr.34107) 2972 : cluster [DBG] pgmap v4520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:01 smithi115 bash[46934]: cluster 2022-04-22T18:52:00.355158+0000 mgr.y (mgr.34107) 2972 : cluster [DBG] pgmap v4520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:01.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:03 smithi121 bash[36902]: cluster 2022-04-22T18:52:02.356364+0000 mgr.y (mgr.34107) 2973 : cluster [DBG] pgmap v4521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:03 smithi121 bash[36902]: audit 2022-04-22T18:52:02.874681+0000 mon.a (mon.0) 6011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:03 smithi115 bash[45495]: cluster 2022-04-22T18:52:02.356364+0000 mgr.y (mgr.34107) 2973 : cluster [DBG] pgmap v4521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:03 smithi115 bash[45495]: audit 2022-04-22T18:52:02.874681+0000 mon.a (mon.0) 6011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:03 smithi115 bash[46934]: cluster 2022-04-22T18:52:02.356364+0000 mgr.y (mgr.34107) 2973 : cluster [DBG] pgmap v4521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:03 smithi115 bash[46934]: audit 2022-04-22T18:52:02.874681+0000 mon.a (mon.0) 6011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:04.095 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:03.811Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:04 smithi121 bash[36902]: audit 2022-04-22T18:52:03.993285+0000 mon.a (mon.0) 6012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:04 smithi115 bash[46934]: audit 2022-04-22T18:52:03.993285+0000 mon.a (mon.0) 6012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:04.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:04 smithi115 bash[45495]: audit 2022-04-22T18:52:03.993285+0000 mon.a (mon.0) 6012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:05 smithi121 bash[36902]: cluster 2022-04-22T18:52:04.357454+0000 mgr.y (mgr.34107) 2974 : cluster [DBG] pgmap v4522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:05 smithi115 bash[45495]: cluster 2022-04-22T18:52:04.357454+0000 mgr.y (mgr.34107) 2974 : cluster [DBG] pgmap v4522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:05 smithi115 bash[46934]: cluster 2022-04-22T18:52:04.357454+0000 mgr.y (mgr.34107) 2974 : cluster [DBG] pgmap v4522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:07 smithi121 bash[36902]: cluster 2022-04-22T18:52:06.358506+0000 mgr.y (mgr.34107) 2975 : cluster [DBG] pgmap v4523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:07 smithi115 bash[45495]: cluster 2022-04-22T18:52:06.358506+0000 mgr.y (mgr.34107) 2975 : cluster [DBG] pgmap v4523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:07 smithi115 bash[46934]: cluster 2022-04-22T18:52:06.358506+0000 mgr.y (mgr.34107) 2975 : cluster [DBG] pgmap v4523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:08 smithi121 bash[36902]: audit 2022-04-22T18:52:07.878313+0000 mon.a (mon.0) 6013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:08 smithi115 bash[45495]: audit 2022-04-22T18:52:07.878313+0000 mon.a (mon.0) 6013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:08 smithi115 bash[46934]: audit 2022-04-22T18:52:07.878313+0000 mon.a (mon.0) 6013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:09 smithi121 bash[36902]: cluster 2022-04-22T18:52:08.359629+0000 mgr.y (mgr.34107) 2976 : cluster [DBG] pgmap v4524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:09 smithi121 bash[36902]: audit 2022-04-22T18:52:08.996237+0000 mon.a (mon.0) 6014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:09 smithi115 bash[45495]: cluster 2022-04-22T18:52:08.359629+0000 mgr.y (mgr.34107) 2976 : cluster [DBG] pgmap v4524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:09 smithi115 bash[45495]: audit 2022-04-22T18:52:08.996237+0000 mon.a (mon.0) 6014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:09 smithi115 bash[46934]: cluster 2022-04-22T18:52:08.359629+0000 mgr.y (mgr.34107) 2976 : cluster [DBG] pgmap v4524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:09 smithi115 bash[46934]: audit 2022-04-22T18:52:08.996237+0000 mon.a (mon.0) 6014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:11 smithi121 bash[36902]: cluster 2022-04-22T18:52:10.360689+0000 mgr.y (mgr.34107) 2977 : cluster [DBG] pgmap v4525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:11 smithi115 bash[45495]: cluster 2022-04-22T18:52:10.360689+0000 mgr.y (mgr.34107) 2977 : cluster [DBG] pgmap v4525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:11.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:11 smithi115 bash[46934]: cluster 2022-04-22T18:52:10.360689+0000 mgr.y (mgr.34107) 2977 : cluster [DBG] pgmap v4525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:11.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:13 smithi121 bash[36902]: cluster 2022-04-22T18:52:12.361792+0000 mgr.y (mgr.34107) 2978 : cluster [DBG] pgmap v4526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:13 smithi121 bash[36902]: audit 2022-04-22T18:52:12.882041+0000 mon.a (mon.0) 6015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:13 smithi115 bash[45495]: cluster 2022-04-22T18:52:12.361792+0000 mgr.y (mgr.34107) 2978 : cluster [DBG] pgmap v4526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:13 smithi115 bash[45495]: audit 2022-04-22T18:52:12.882041+0000 mon.a (mon.0) 6015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:13 smithi115 bash[46934]: cluster 2022-04-22T18:52:12.361792+0000 mgr.y (mgr.34107) 2978 : cluster [DBG] pgmap v4526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:13 smithi115 bash[46934]: audit 2022-04-22T18:52:12.882041+0000 mon.a (mon.0) 6015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:14.099 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:13.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:14.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:14 smithi121 bash[36902]: audit 2022-04-22T18:52:13.999309+0000 mon.a (mon.0) 6016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:14 smithi115 bash[45495]: audit 2022-04-22T18:52:13.999309+0000 mon.a (mon.0) 6016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:14 smithi115 bash[46934]: audit 2022-04-22T18:52:13.999309+0000 mon.a (mon.0) 6016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:14.774 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:52:15.121 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:52:15.121 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (75m) 2m ago 89m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:52:15.121 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (88m) 3m ago 88m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:52:15.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (77m) 2m ago 90m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:52:15.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (76m) 3m ago 90m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:52:15.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (84m) 3m ago 94m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:52:15.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (85m) 2m ago 96m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:52:15.124 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (84m) 2m ago 96m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (83m) 3m ago 94m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (83m) 2m ago 94m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (76m) 2m ago 89m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (76m) 3m ago 89m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (82m) 2m ago 93m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:52:15.125 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (81m) 2m ago 93m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:52:15.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (80m) 2m ago 93m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:52:15.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (80m) 2m ago 92m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:52:15.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (79m) 3m ago 92m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:52:15.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (79m) 3m ago 91m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:52:15.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (78m) 3m ago 91m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:52:15.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (77m) 3m ago 90m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:52:15.133 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (76m) 3m ago 89m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:52:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:15 smithi121 bash[36902]: audit 2022-04-22T18:52:14.217718+0000 mon.a (mon.0) 6017 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:52:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:15 smithi121 bash[36902]: audit 2022-04-22T18:52:14.218472+0000 mon.a (mon.0) 6018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:52:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:15 smithi121 bash[36902]: cluster 2022-04-22T18:52:14.362838+0000 mgr.y (mgr.34107) 2979 : cluster [DBG] pgmap v4527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:15 smithi121 bash[36902]: audit 2022-04-22T18:52:14.750688+0000 mgr.y (mgr.34107) 2980 : audit [DBG] from='client.48472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:15.522 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:52:15.522 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:15.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:52:15.524 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:52:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[45495]: audit 2022-04-22T18:52:14.217718+0000 mon.a (mon.0) 6017 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:52:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[45495]: audit 2022-04-22T18:52:14.218472+0000 mon.a (mon.0) 6018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:52:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[45495]: cluster 2022-04-22T18:52:14.362838+0000 mgr.y (mgr.34107) 2979 : cluster [DBG] pgmap v4527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[45495]: audit 2022-04-22T18:52:14.750688+0000 mgr.y (mgr.34107) 2980 : audit [DBG] from='client.48472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[46934]: audit 2022-04-22T18:52:14.217718+0000 mon.a (mon.0) 6017 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:52:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[46934]: audit 2022-04-22T18:52:14.218472+0000 mon.a (mon.0) 6018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:52:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[46934]: cluster 2022-04-22T18:52:14.362838+0000 mgr.y (mgr.34107) 2979 : cluster [DBG] pgmap v4527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:15 smithi115 bash[46934]: audit 2022-04-22T18:52:14.750688+0000 mgr.y (mgr.34107) 2980 : audit [DBG] from='client.48472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:16 smithi121 bash[36902]: audit 2022-04-22T18:52:15.095104+0000 mgr.y (mgr.34107) 2981 : audit [DBG] from='client.48478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:16 smithi121 bash[36902]: audit 2022-04-22T18:52:15.513763+0000 mon.a (mon.0) 6019 : audit [DBG] from='client.? 172.21.15.115:0/1074615824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:16 smithi115 bash[45495]: audit 2022-04-22T18:52:15.095104+0000 mgr.y (mgr.34107) 2981 : audit [DBG] from='client.48478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:16 smithi115 bash[45495]: audit 2022-04-22T18:52:15.513763+0000 mon.a (mon.0) 6019 : audit [DBG] from='client.? 172.21.15.115:0/1074615824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:16 smithi115 bash[46934]: audit 2022-04-22T18:52:15.095104+0000 mgr.y (mgr.34107) 2981 : audit [DBG] from='client.48478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:16 smithi115 bash[46934]: audit 2022-04-22T18:52:15.513763+0000 mon.a (mon.0) 6019 : audit [DBG] from='client.? 172.21.15.115:0/1074615824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:17 smithi121 bash[36902]: cluster 2022-04-22T18:52:16.363921+0000 mgr.y (mgr.34107) 2982 : cluster [DBG] pgmap v4528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:17 smithi115 bash[45495]: cluster 2022-04-22T18:52:16.363921+0000 mgr.y (mgr.34107) 2982 : cluster [DBG] pgmap v4528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:17 smithi115 bash[46934]: cluster 2022-04-22T18:52:16.363921+0000 mgr.y (mgr.34107) 2982 : cluster [DBG] pgmap v4528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:18 smithi121 bash[36902]: audit 2022-04-22T18:52:17.885903+0000 mon.a (mon.0) 6020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:18 smithi115 bash[45495]: audit 2022-04-22T18:52:17.885903+0000 mon.a (mon.0) 6020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:18 smithi115 bash[46934]: audit 2022-04-22T18:52:17.885903+0000 mon.a (mon.0) 6020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:19 smithi121 bash[36902]: cluster 2022-04-22T18:52:18.365019+0000 mgr.y (mgr.34107) 2983 : cluster [DBG] pgmap v4529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:19.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:19 smithi121 bash[36902]: audit 2022-04-22T18:52:19.002329+0000 mon.a (mon.0) 6021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:19 smithi115 bash[45495]: cluster 2022-04-22T18:52:18.365019+0000 mgr.y (mgr.34107) 2983 : cluster [DBG] pgmap v4529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:19 smithi115 bash[45495]: audit 2022-04-22T18:52:19.002329+0000 mon.a (mon.0) 6021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:19 smithi115 bash[46934]: cluster 2022-04-22T18:52:18.365019+0000 mgr.y (mgr.34107) 2983 : cluster [DBG] pgmap v4529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:19 smithi115 bash[46934]: audit 2022-04-22T18:52:19.002329+0000 mon.a (mon.0) 6021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:21 smithi121 bash[36902]: cluster 2022-04-22T18:52:20.366134+0000 mgr.y (mgr.34107) 2984 : cluster [DBG] pgmap v4530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:21 smithi115 bash[45495]: cluster 2022-04-22T18:52:20.366134+0000 mgr.y (mgr.34107) 2984 : cluster [DBG] pgmap v4530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:21 smithi115 bash[46934]: cluster 2022-04-22T18:52:20.366134+0000 mgr.y (mgr.34107) 2984 : cluster [DBG] pgmap v4530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:21.936Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:23 smithi121 bash[36902]: cluster 2022-04-22T18:52:22.367236+0000 mgr.y (mgr.34107) 2985 : cluster [DBG] pgmap v4531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:23 smithi121 bash[36902]: audit 2022-04-22T18:52:22.889590+0000 mon.a (mon.0) 6022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:23 smithi115 bash[45495]: cluster 2022-04-22T18:52:22.367236+0000 mgr.y (mgr.34107) 2985 : cluster [DBG] pgmap v4531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:23 smithi115 bash[45495]: audit 2022-04-22T18:52:22.889590+0000 mon.a (mon.0) 6022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:23 smithi115 bash[46934]: cluster 2022-04-22T18:52:22.367236+0000 mgr.y (mgr.34107) 2985 : cluster [DBG] pgmap v4531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:23 smithi115 bash[46934]: audit 2022-04-22T18:52:22.889590+0000 mon.a (mon.0) 6022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:24.104 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:23.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:24 smithi121 bash[36902]: audit 2022-04-22T18:52:24.005425+0000 mon.a (mon.0) 6023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:24 smithi115 bash[45495]: audit 2022-04-22T18:52:24.005425+0000 mon.a (mon.0) 6023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:24 smithi115 bash[46934]: audit 2022-04-22T18:52:24.005425+0000 mon.a (mon.0) 6023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:25.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:25 smithi121 bash[36902]: cluster 2022-04-22T18:52:24.368359+0000 mgr.y (mgr.34107) 2986 : cluster [DBG] pgmap v4532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:25 smithi115 bash[45495]: cluster 2022-04-22T18:52:24.368359+0000 mgr.y (mgr.34107) 2986 : cluster [DBG] pgmap v4532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:25 smithi115 bash[46934]: cluster 2022-04-22T18:52:24.368359+0000 mgr.y (mgr.34107) 2986 : cluster [DBG] pgmap v4532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:27 smithi121 bash[36902]: cluster 2022-04-22T18:52:26.369606+0000 mgr.y (mgr.34107) 2987 : cluster [DBG] pgmap v4533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:27 smithi115 bash[45495]: cluster 2022-04-22T18:52:26.369606+0000 mgr.y (mgr.34107) 2987 : cluster [DBG] pgmap v4533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:27 smithi115 bash[46934]: cluster 2022-04-22T18:52:26.369606+0000 mgr.y (mgr.34107) 2987 : cluster [DBG] pgmap v4533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:28 smithi121 bash[36902]: audit 2022-04-22T18:52:27.893315+0000 mon.a (mon.0) 6024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:28 smithi115 bash[46934]: audit 2022-04-22T18:52:27.893315+0000 mon.a (mon.0) 6024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:28 smithi115 bash[45495]: audit 2022-04-22T18:52:27.893315+0000 mon.a (mon.0) 6024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:29 smithi121 bash[36902]: cluster 2022-04-22T18:52:28.370803+0000 mgr.y (mgr.34107) 2988 : cluster [DBG] pgmap v4534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:29 smithi121 bash[36902]: audit 2022-04-22T18:52:29.008435+0000 mon.a (mon.0) 6025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:29 smithi115 bash[46934]: cluster 2022-04-22T18:52:28.370803+0000 mgr.y (mgr.34107) 2988 : cluster [DBG] pgmap v4534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:29 smithi115 bash[46934]: audit 2022-04-22T18:52:29.008435+0000 mon.a (mon.0) 6025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:29 smithi115 bash[45495]: cluster 2022-04-22T18:52:28.370803+0000 mgr.y (mgr.34107) 2988 : cluster [DBG] pgmap v4534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:29 smithi115 bash[45495]: audit 2022-04-22T18:52:29.008435+0000 mon.a (mon.0) 6025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:31 smithi121 bash[36902]: cluster 2022-04-22T18:52:30.372076+0000 mgr.y (mgr.34107) 2989 : cluster [DBG] pgmap v4535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:31 smithi115 bash[46934]: cluster 2022-04-22T18:52:30.372076+0000 mgr.y (mgr.34107) 2989 : cluster [DBG] pgmap v4535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:31 smithi115 bash[45495]: cluster 2022-04-22T18:52:30.372076+0000 mgr.y (mgr.34107) 2989 : cluster [DBG] pgmap v4535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:31.936Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:33 smithi121 bash[36902]: cluster 2022-04-22T18:52:32.373157+0000 mgr.y (mgr.34107) 2990 : cluster [DBG] pgmap v4536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:33.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:33 smithi121 bash[36902]: audit 2022-04-22T18:52:32.897046+0000 mon.a (mon.0) 6026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:33 smithi115 bash[45495]: cluster 2022-04-22T18:52:32.373157+0000 mgr.y (mgr.34107) 2990 : cluster [DBG] pgmap v4536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:33 smithi115 bash[45495]: audit 2022-04-22T18:52:32.897046+0000 mon.a (mon.0) 6026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:33 smithi115 bash[46934]: cluster 2022-04-22T18:52:32.373157+0000 mgr.y (mgr.34107) 2990 : cluster [DBG] pgmap v4536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:33 smithi115 bash[46934]: audit 2022-04-22T18:52:32.897046+0000 mon.a (mon.0) 6026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:34.108 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:33.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:34.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:34 smithi121 bash[36902]: audit 2022-04-22T18:52:34.011888+0000 mon.a (mon.0) 6027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:34 smithi115 bash[45495]: audit 2022-04-22T18:52:34.011888+0000 mon.a (mon.0) 6027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:34 smithi115 bash[46934]: audit 2022-04-22T18:52:34.011888+0000 mon.a (mon.0) 6027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:35 smithi121 bash[36902]: cluster 2022-04-22T18:52:34.374239+0000 mgr.y (mgr.34107) 2991 : cluster [DBG] pgmap v4537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:35 smithi115 bash[45495]: cluster 2022-04-22T18:52:34.374239+0000 mgr.y (mgr.34107) 2991 : cluster [DBG] pgmap v4537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:35 smithi115 bash[46934]: cluster 2022-04-22T18:52:34.374239+0000 mgr.y (mgr.34107) 2991 : cluster [DBG] pgmap v4537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:37 smithi121 bash[36902]: cluster 2022-04-22T18:52:36.375331+0000 mgr.y (mgr.34107) 2992 : cluster [DBG] pgmap v4538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:37 smithi115 bash[45495]: cluster 2022-04-22T18:52:36.375331+0000 mgr.y (mgr.34107) 2992 : cluster [DBG] pgmap v4538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:37 smithi115 bash[46934]: cluster 2022-04-22T18:52:36.375331+0000 mgr.y (mgr.34107) 2992 : cluster [DBG] pgmap v4538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:38 smithi121 bash[36902]: audit 2022-04-22T18:52:37.900718+0000 mon.a (mon.0) 6028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:38 smithi115 bash[45495]: audit 2022-04-22T18:52:37.900718+0000 mon.a (mon.0) 6028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:38 smithi115 bash[46934]: audit 2022-04-22T18:52:37.900718+0000 mon.a (mon.0) 6028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:39 smithi121 bash[36902]: cluster 2022-04-22T18:52:38.376407+0000 mgr.y (mgr.34107) 2993 : cluster [DBG] pgmap v4539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:39 smithi121 bash[36902]: audit 2022-04-22T18:52:39.015051+0000 mon.a (mon.0) 6029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:39 smithi115 bash[45495]: cluster 2022-04-22T18:52:38.376407+0000 mgr.y (mgr.34107) 2993 : cluster [DBG] pgmap v4539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:39 smithi115 bash[45495]: audit 2022-04-22T18:52:39.015051+0000 mon.a (mon.0) 6029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:39 smithi115 bash[46934]: cluster 2022-04-22T18:52:38.376407+0000 mgr.y (mgr.34107) 2993 : cluster [DBG] pgmap v4539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:39 smithi115 bash[46934]: audit 2022-04-22T18:52:39.015051+0000 mon.a (mon.0) 6029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:41 smithi121 bash[36902]: cluster 2022-04-22T18:52:40.377564+0000 mgr.y (mgr.34107) 2994 : cluster [DBG] pgmap v4540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:41 smithi115 bash[45495]: cluster 2022-04-22T18:52:40.377564+0000 mgr.y (mgr.34107) 2994 : cluster [DBG] pgmap v4540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:41 smithi115 bash[46934]: cluster 2022-04-22T18:52:40.377564+0000 mgr.y (mgr.34107) 2994 : cluster [DBG] pgmap v4540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:41.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:43 smithi121 bash[36902]: cluster 2022-04-22T18:52:42.378670+0000 mgr.y (mgr.34107) 2995 : cluster [DBG] pgmap v4541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:43 smithi121 bash[36902]: audit 2022-04-22T18:52:42.904548+0000 mon.a (mon.0) 6030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:43 smithi115 bash[45495]: cluster 2022-04-22T18:52:42.378670+0000 mgr.y (mgr.34107) 2995 : cluster [DBG] pgmap v4541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:43 smithi115 bash[45495]: audit 2022-04-22T18:52:42.904548+0000 mon.a (mon.0) 6030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:43 smithi115 bash[46934]: cluster 2022-04-22T18:52:42.378670+0000 mgr.y (mgr.34107) 2995 : cluster [DBG] pgmap v4541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:43 smithi115 bash[46934]: audit 2022-04-22T18:52:42.904548+0000 mon.a (mon.0) 6030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:44.105 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:43.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:44 smithi121 bash[36902]: audit 2022-04-22T18:52:44.018397+0000 mon.a (mon.0) 6031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:44 smithi115 bash[45495]: audit 2022-04-22T18:52:44.018397+0000 mon.a (mon.0) 6031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:44 smithi115 bash[46934]: audit 2022-04-22T18:52:44.018397+0000 mon.a (mon.0) 6031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:45 smithi121 bash[36902]: cluster 2022-04-22T18:52:44.379785+0000 mgr.y (mgr.34107) 2996 : cluster [DBG] pgmap v4542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:45 smithi115 bash[45495]: cluster 2022-04-22T18:52:44.379785+0000 mgr.y (mgr.34107) 2996 : cluster [DBG] pgmap v4542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:45 smithi115 bash[46934]: cluster 2022-04-22T18:52:44.379785+0000 mgr.y (mgr.34107) 2996 : cluster [DBG] pgmap v4542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:45.893 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:52:46.238 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:52:46.238 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (75m) 2m ago 89m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (89m) 4m ago 89m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (77m) 2m ago 90m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (77m) 4m ago 90m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (85m) 4m ago 94m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (85m) 2m ago 97m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:52:46.239 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (84m) 2m ago 97m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (83m) 4m ago 95m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (84m) 2m ago 95m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (76m) 2m ago 89m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (76m) 4m ago 89m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (82m) 2m ago 94m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (81m) 2m ago 94m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (81m) 2m ago 93m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (80m) 2m ago 92m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (80m) 4m ago 92m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:52:46.240 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (79m) 4m ago 92m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:52:46.241 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (78m) 4m ago 91m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:52:46.241 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (78m) 4m ago 91m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:52:46.241 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (76m) 4m ago 89m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:52:46.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:46 smithi121 bash[36902]: audit 2022-04-22T18:52:45.870343+0000 mgr.y (mgr.34107) 2997 : audit [DBG] from='client.38556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:46 smithi115 bash[46934]: audit 2022-04-22T18:52:45.870343+0000 mgr.y (mgr.34107) 2997 : audit [DBG] from='client.38556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:46.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:46 smithi115 bash[45495]: audit 2022-04-22T18:52:45.870343+0000 mgr.y (mgr.34107) 2997 : audit [DBG] from='client.38556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:46.640 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:52:46.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:52:46.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:52:46.643 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:52:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:47 smithi121 bash[36902]: audit 2022-04-22T18:52:46.215554+0000 mgr.y (mgr.34107) 2998 : audit [DBG] from='client.48496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:47 smithi121 bash[36902]: cluster 2022-04-22T18:52:46.380724+0000 mgr.y (mgr.34107) 2999 : cluster [DBG] pgmap v4543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:47.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:47 smithi121 bash[36902]: audit 2022-04-22T18:52:46.631279+0000 mon.a (mon.0) 6032 : audit [DBG] from='client.? 172.21.15.115:0/2570199929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[45495]: audit 2022-04-22T18:52:46.215554+0000 mgr.y (mgr.34107) 2998 : audit [DBG] from='client.48496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[45495]: cluster 2022-04-22T18:52:46.380724+0000 mgr.y (mgr.34107) 2999 : cluster [DBG] pgmap v4543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[45495]: audit 2022-04-22T18:52:46.631279+0000 mon.a (mon.0) 6032 : audit [DBG] from='client.? 172.21.15.115:0/2570199929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:47.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[46934]: audit 2022-04-22T18:52:46.215554+0000 mgr.y (mgr.34107) 2998 : audit [DBG] from='client.48496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:52:47.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[46934]: cluster 2022-04-22T18:52:46.380724+0000 mgr.y (mgr.34107) 2999 : cluster [DBG] pgmap v4543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:47.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:47 smithi115 bash[46934]: audit 2022-04-22T18:52:46.631279+0000 mon.a (mon.0) 6032 : audit [DBG] from='client.? 172.21.15.115:0/2570199929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:52:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:48 smithi121 bash[36902]: audit 2022-04-22T18:52:47.908561+0000 mon.a (mon.0) 6033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:48 smithi115 bash[45495]: audit 2022-04-22T18:52:47.908561+0000 mon.a (mon.0) 6033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:48 smithi115 bash[46934]: audit 2022-04-22T18:52:47.908561+0000 mon.a (mon.0) 6033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: audit 2022-04-22T18:52:48.243420+0000 mon.a (mon.0) 6034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:52:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: audit 2022-04-22T18:52:48.246129+0000 mon.a (mon.0) 6035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:52:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: audit 2022-04-22T18:52:48.247811+0000 mon.a (mon.0) 6036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:52:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: cluster 2022-04-22T18:52:48.381546+0000 mgr.y (mgr.34107) 3000 : cluster [DBG] pgmap v4544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: audit 2022-04-22T18:52:48.586193+0000 mon.a (mon.0) 6037 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:52:49.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:49 smithi121 bash[36902]: audit 2022-04-22T18:52:49.021329+0000 mon.a (mon.0) 6038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: audit 2022-04-22T18:52:48.243420+0000 mon.a (mon.0) 6034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:52:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: audit 2022-04-22T18:52:48.246129+0000 mon.a (mon.0) 6035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:52:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: audit 2022-04-22T18:52:48.247811+0000 mon.a (mon.0) 6036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:52:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: cluster 2022-04-22T18:52:48.381546+0000 mgr.y (mgr.34107) 3000 : cluster [DBG] pgmap v4544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: audit 2022-04-22T18:52:48.586193+0000 mon.a (mon.0) 6037 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:52:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[45495]: audit 2022-04-22T18:52:49.021329+0000 mon.a (mon.0) 6038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: audit 2022-04-22T18:52:48.243420+0000 mon.a (mon.0) 6034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:52:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: audit 2022-04-22T18:52:48.246129+0000 mon.a (mon.0) 6035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:52:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: audit 2022-04-22T18:52:48.247811+0000 mon.a (mon.0) 6036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:52:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: cluster 2022-04-22T18:52:48.381546+0000 mgr.y (mgr.34107) 3000 : cluster [DBG] pgmap v4544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: audit 2022-04-22T18:52:48.586193+0000 mon.a (mon.0) 6037 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:52:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:49 smithi115 bash[46934]: audit 2022-04-22T18:52:49.021329+0000 mon.a (mon.0) 6038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:51.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:51 smithi121 bash[36902]: cluster 2022-04-22T18:52:50.382625+0000 mgr.y (mgr.34107) 3001 : cluster [DBG] pgmap v4545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:51 smithi115 bash[45495]: cluster 2022-04-22T18:52:50.382625+0000 mgr.y (mgr.34107) 3001 : cluster [DBG] pgmap v4545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:51 smithi115 bash[46934]: cluster 2022-04-22T18:52:50.382625+0000 mgr.y (mgr.34107) 3001 : cluster [DBG] pgmap v4545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:51.939Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:53 smithi121 bash[36902]: cluster 2022-04-22T18:52:52.383819+0000 mgr.y (mgr.34107) 3002 : cluster [DBG] pgmap v4546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:53 smithi121 bash[36902]: audit 2022-04-22T18:52:52.912356+0000 mon.a (mon.0) 6039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:53 smithi115 bash[45495]: cluster 2022-04-22T18:52:52.383819+0000 mgr.y (mgr.34107) 3002 : cluster [DBG] pgmap v4546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:53 smithi115 bash[45495]: audit 2022-04-22T18:52:52.912356+0000 mon.a (mon.0) 6039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:53 smithi115 bash[46934]: cluster 2022-04-22T18:52:52.383819+0000 mgr.y (mgr.34107) 3002 : cluster [DBG] pgmap v4546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:53 smithi115 bash[46934]: audit 2022-04-22T18:52:52.912356+0000 mon.a (mon.0) 6039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:54.113 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:52:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:52:53.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:52:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:54 smithi121 bash[36902]: audit 2022-04-22T18:52:54.024476+0000 mon.a (mon.0) 6040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:54 smithi115 bash[45495]: audit 2022-04-22T18:52:54.024476+0000 mon.a (mon.0) 6040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:54 smithi115 bash[46934]: audit 2022-04-22T18:52:54.024476+0000 mon.a (mon.0) 6040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:55 smithi121 bash[36902]: cluster 2022-04-22T18:52:54.385070+0000 mgr.y (mgr.34107) 3003 : cluster [DBG] pgmap v4547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:55 smithi115 bash[45495]: cluster 2022-04-22T18:52:54.385070+0000 mgr.y (mgr.34107) 3003 : cluster [DBG] pgmap v4547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:55 smithi115 bash[46934]: cluster 2022-04-22T18:52:54.385070+0000 mgr.y (mgr.34107) 3003 : cluster [DBG] pgmap v4547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:57 smithi121 bash[36902]: cluster 2022-04-22T18:52:56.386217+0000 mgr.y (mgr.34107) 3004 : cluster [DBG] pgmap v4548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:57 smithi115 bash[45495]: cluster 2022-04-22T18:52:56.386217+0000 mgr.y (mgr.34107) 3004 : cluster [DBG] pgmap v4548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:57 smithi115 bash[46934]: cluster 2022-04-22T18:52:56.386217+0000 mgr.y (mgr.34107) 3004 : cluster [DBG] pgmap v4548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:58 smithi121 bash[36902]: audit 2022-04-22T18:52:57.916022+0000 mon.a (mon.0) 6041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:58 smithi115 bash[45495]: audit 2022-04-22T18:52:57.916022+0000 mon.a (mon.0) 6041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:58 smithi115 bash[46934]: audit 2022-04-22T18:52:57.916022+0000 mon.a (mon.0) 6041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:52:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:59 smithi121 bash[36902]: cluster 2022-04-22T18:52:58.387315+0000 mgr.y (mgr.34107) 3005 : cluster [DBG] pgmap v4549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:52:59 smithi121 bash[36902]: audit 2022-04-22T18:52:59.027667+0000 mon.a (mon.0) 6042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:59 smithi115 bash[45495]: cluster 2022-04-22T18:52:58.387315+0000 mgr.y (mgr.34107) 3005 : cluster [DBG] pgmap v4549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:52:59 smithi115 bash[45495]: audit 2022-04-22T18:52:59.027667+0000 mon.a (mon.0) 6042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:52:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:59 smithi115 bash[46934]: cluster 2022-04-22T18:52:58.387315+0000 mgr.y (mgr.34107) 3005 : cluster [DBG] pgmap v4549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:52:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:52:59 smithi115 bash[46934]: audit 2022-04-22T18:52:59.027667+0000 mon.a (mon.0) 6042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:01.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:01 smithi121 bash[36902]: cluster 2022-04-22T18:53:00.388382+0000 mgr.y (mgr.34107) 3006 : cluster [DBG] pgmap v4550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:01 smithi115 bash[45495]: cluster 2022-04-22T18:53:00.388382+0000 mgr.y (mgr.34107) 3006 : cluster [DBG] pgmap v4550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:01 smithi115 bash[46934]: cluster 2022-04-22T18:53:00.388382+0000 mgr.y (mgr.34107) 3006 : cluster [DBG] pgmap v4550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:01.939Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:03 smithi121 bash[36902]: cluster 2022-04-22T18:53:02.389508+0000 mgr.y (mgr.34107) 3007 : cluster [DBG] pgmap v4551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:03 smithi121 bash[36902]: audit 2022-04-22T18:53:02.919687+0000 mon.a (mon.0) 6043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:03.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:03 smithi115 bash[45495]: cluster 2022-04-22T18:53:02.389508+0000 mgr.y (mgr.34107) 3007 : cluster [DBG] pgmap v4551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:03.557 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:03 smithi115 bash[45495]: audit 2022-04-22T18:53:02.919687+0000 mon.a (mon.0) 6043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:03.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:03 smithi115 bash[46934]: cluster 2022-04-22T18:53:02.389508+0000 mgr.y (mgr.34107) 3007 : cluster [DBG] pgmap v4551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:03.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:03 smithi115 bash[46934]: audit 2022-04-22T18:53:02.919687+0000 mon.a (mon.0) 6043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:04.113 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:03.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:04 smithi121 bash[36902]: audit 2022-04-22T18:53:04.030754+0000 mon.a (mon.0) 6044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:04 smithi115 bash[45495]: audit 2022-04-22T18:53:04.030754+0000 mon.a (mon.0) 6044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:04 smithi115 bash[46934]: audit 2022-04-22T18:53:04.030754+0000 mon.a (mon.0) 6044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:05 smithi121 bash[36902]: cluster 2022-04-22T18:53:04.390652+0000 mgr.y (mgr.34107) 3008 : cluster [DBG] pgmap v4552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:05 smithi115 bash[45495]: cluster 2022-04-22T18:53:04.390652+0000 mgr.y (mgr.34107) 3008 : cluster [DBG] pgmap v4552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:05 smithi115 bash[46934]: cluster 2022-04-22T18:53:04.390652+0000 mgr.y (mgr.34107) 3008 : cluster [DBG] pgmap v4552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:07 smithi121 bash[36902]: cluster 2022-04-22T18:53:06.391890+0000 mgr.y (mgr.34107) 3009 : cluster [DBG] pgmap v4553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:07 smithi115 bash[45495]: cluster 2022-04-22T18:53:06.391890+0000 mgr.y (mgr.34107) 3009 : cluster [DBG] pgmap v4553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:07 smithi115 bash[46934]: cluster 2022-04-22T18:53:06.391890+0000 mgr.y (mgr.34107) 3009 : cluster [DBG] pgmap v4553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:08 smithi121 bash[36902]: audit 2022-04-22T18:53:07.923543+0000 mon.a (mon.0) 6045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:08 smithi115 bash[45495]: audit 2022-04-22T18:53:07.923543+0000 mon.a (mon.0) 6045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:08 smithi115 bash[46934]: audit 2022-04-22T18:53:07.923543+0000 mon.a (mon.0) 6045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:09 smithi121 bash[36902]: cluster 2022-04-22T18:53:08.392983+0000 mgr.y (mgr.34107) 3010 : cluster [DBG] pgmap v4554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:09 smithi121 bash[36902]: audit 2022-04-22T18:53:09.034058+0000 mon.a (mon.0) 6046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:09 smithi115 bash[45495]: cluster 2022-04-22T18:53:08.392983+0000 mgr.y (mgr.34107) 3010 : cluster [DBG] pgmap v4554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:09 smithi115 bash[45495]: audit 2022-04-22T18:53:09.034058+0000 mon.a (mon.0) 6046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:09 smithi115 bash[46934]: cluster 2022-04-22T18:53:08.392983+0000 mgr.y (mgr.34107) 3010 : cluster [DBG] pgmap v4554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:09 smithi115 bash[46934]: audit 2022-04-22T18:53:09.034058+0000 mon.a (mon.0) 6046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:11 smithi121 bash[36902]: cluster 2022-04-22T18:53:10.393967+0000 mgr.y (mgr.34107) 3011 : cluster [DBG] pgmap v4555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:11 smithi115 bash[45495]: cluster 2022-04-22T18:53:10.393967+0000 mgr.y (mgr.34107) 3011 : cluster [DBG] pgmap v4555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:11 smithi115 bash[46934]: cluster 2022-04-22T18:53:10.393967+0000 mgr.y (mgr.34107) 3011 : cluster [DBG] pgmap v4555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:12.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:11.940Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:13 smithi121 bash[36902]: cluster 2022-04-22T18:53:12.395163+0000 mgr.y (mgr.34107) 3012 : cluster [DBG] pgmap v4556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:13 smithi121 bash[36902]: audit 2022-04-22T18:53:12.927280+0000 mon.a (mon.0) 6047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:13 smithi115 bash[45495]: cluster 2022-04-22T18:53:12.395163+0000 mgr.y (mgr.34107) 3012 : cluster [DBG] pgmap v4556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:13 smithi115 bash[45495]: audit 2022-04-22T18:53:12.927280+0000 mon.a (mon.0) 6047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:13 smithi115 bash[46934]: cluster 2022-04-22T18:53:12.395163+0000 mgr.y (mgr.34107) 3012 : cluster [DBG] pgmap v4556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:13 smithi115 bash[46934]: audit 2022-04-22T18:53:12.927280+0000 mon.a (mon.0) 6047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:14.115 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:13.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:14.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:14 smithi121 bash[36902]: audit 2022-04-22T18:53:14.037210+0000 mon.a (mon.0) 6048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:14 smithi115 bash[45495]: audit 2022-04-22T18:53:14.037210+0000 mon.a (mon.0) 6048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:14 smithi115 bash[46934]: audit 2022-04-22T18:53:14.037210+0000 mon.a (mon.0) 6048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:15 smithi121 bash[36902]: audit 2022-04-22T18:53:14.230135+0000 mon.a (mon.0) 6049 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:53:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:15 smithi121 bash[36902]: audit 2022-04-22T18:53:14.230584+0000 mon.a (mon.0) 6050 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:53:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:15 smithi121 bash[36902]: cluster 2022-04-22T18:53:14.396289+0000 mgr.y (mgr.34107) 3013 : cluster [DBG] pgmap v4557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[45495]: audit 2022-04-22T18:53:14.230135+0000 mon.a (mon.0) 6049 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[45495]: audit 2022-04-22T18:53:14.230584+0000 mon.a (mon.0) 6050 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[45495]: cluster 2022-04-22T18:53:14.396289+0000 mgr.y (mgr.34107) 3013 : cluster [DBG] pgmap v4557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[46934]: audit 2022-04-22T18:53:14.230135+0000 mon.a (mon.0) 6049 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[46934]: audit 2022-04-22T18:53:14.230584+0000 mon.a (mon.0) 6050 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:53:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:15 smithi115 bash[46934]: cluster 2022-04-22T18:53:14.396289+0000 mgr.y (mgr.34107) 3013 : cluster [DBG] pgmap v4557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:17.011 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (76m) 3m ago 90m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (89m) 4m ago 89m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (78m) 3m ago 91m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (77m) 4m ago 91m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:53:17.351 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (85m) 4m ago 95m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (86m) 3m ago 97m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (85m) 3m ago 97m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (84m) 4m ago 95m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (84m) 3m ago 95m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (77m) 3m ago 90m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (77m) 4m ago 90m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (83m) 3m ago 95m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (82m) 3m ago 94m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (81m) 3m ago 94m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:53:17.352 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (81m) 3m ago 93m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:53:17.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (80m) 4m ago 93m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:53:17.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (80m) 4m ago 92m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:53:17.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (79m) 4m ago 92m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:53:17.353 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (78m) 4m ago 91m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:53:17.353 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (77m) 4m ago 90m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:53:17.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:53:17.752 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:53:17.753 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:53:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:17 smithi121 bash[36902]: cluster 2022-04-22T18:53:16.397351+0000 mgr.y (mgr.34107) 3014 : cluster [DBG] pgmap v4558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:17 smithi121 bash[36902]: audit 2022-04-22T18:53:16.987112+0000 mgr.y (mgr.34107) 3015 : audit [DBG] from='client.38574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:17 smithi115 bash[45495]: cluster 2022-04-22T18:53:16.397351+0000 mgr.y (mgr.34107) 3014 : cluster [DBG] pgmap v4558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:17 smithi115 bash[45495]: audit 2022-04-22T18:53:16.987112+0000 mgr.y (mgr.34107) 3015 : audit [DBG] from='client.38574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:17 smithi115 bash[46934]: cluster 2022-04-22T18:53:16.397351+0000 mgr.y (mgr.34107) 3014 : cluster [DBG] pgmap v4558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:17 smithi115 bash[46934]: audit 2022-04-22T18:53:16.987112+0000 mgr.y (mgr.34107) 3015 : audit [DBG] from='client.38574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:18 smithi121 bash[36902]: audit 2022-04-22T18:53:17.320789+0000 mgr.y (mgr.34107) 3016 : audit [DBG] from='client.48514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:18 smithi121 bash[36902]: audit 2022-04-22T18:53:17.742234+0000 mon.a (mon.0) 6051 : audit [DBG] from='client.? 172.21.15.115:0/1829849830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:18.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:18 smithi121 bash[36902]: audit 2022-04-22T18:53:17.931339+0000 mon.a (mon.0) 6052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[45495]: audit 2022-04-22T18:53:17.320789+0000 mgr.y (mgr.34107) 3016 : audit [DBG] from='client.48514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[45495]: audit 2022-04-22T18:53:17.742234+0000 mon.a (mon.0) 6051 : audit [DBG] from='client.? 172.21.15.115:0/1829849830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[45495]: audit 2022-04-22T18:53:17.931339+0000 mon.a (mon.0) 6052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[46934]: audit 2022-04-22T18:53:17.320789+0000 mgr.y (mgr.34107) 3016 : audit [DBG] from='client.48514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[46934]: audit 2022-04-22T18:53:17.742234+0000 mon.a (mon.0) 6051 : audit [DBG] from='client.? 172.21.15.115:0/1829849830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:18 smithi115 bash[46934]: audit 2022-04-22T18:53:17.931339+0000 mon.a (mon.0) 6052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:19 smithi121 bash[36902]: cluster 2022-04-22T18:53:18.398423+0000 mgr.y (mgr.34107) 3017 : cluster [DBG] pgmap v4559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:19 smithi121 bash[36902]: audit 2022-04-22T18:53:19.040152+0000 mon.a (mon.0) 6053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:19 smithi115 bash[45495]: cluster 2022-04-22T18:53:18.398423+0000 mgr.y (mgr.34107) 3017 : cluster [DBG] pgmap v4559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:19 smithi115 bash[45495]: audit 2022-04-22T18:53:19.040152+0000 mon.a (mon.0) 6053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:19 smithi115 bash[46934]: cluster 2022-04-22T18:53:18.398423+0000 mgr.y (mgr.34107) 3017 : cluster [DBG] pgmap v4559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:19 smithi115 bash[46934]: audit 2022-04-22T18:53:19.040152+0000 mon.a (mon.0) 6053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:21 smithi121 bash[36902]: cluster 2022-04-22T18:53:20.399465+0000 mgr.y (mgr.34107) 3018 : cluster [DBG] pgmap v4560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:21 smithi115 bash[45495]: cluster 2022-04-22T18:53:20.399465+0000 mgr.y (mgr.34107) 3018 : cluster [DBG] pgmap v4560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:21 smithi115 bash[46934]: cluster 2022-04-22T18:53:20.399465+0000 mgr.y (mgr.34107) 3018 : cluster [DBG] pgmap v4560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:21.941Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:23 smithi121 bash[36902]: cluster 2022-04-22T18:53:22.400487+0000 mgr.y (mgr.34107) 3019 : cluster [DBG] pgmap v4561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:23 smithi121 bash[36902]: audit 2022-04-22T18:53:22.935103+0000 mon.a (mon.0) 6054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:23 smithi115 bash[45495]: cluster 2022-04-22T18:53:22.400487+0000 mgr.y (mgr.34107) 3019 : cluster [DBG] pgmap v4561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:23 smithi115 bash[45495]: audit 2022-04-22T18:53:22.935103+0000 mon.a (mon.0) 6054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:23 smithi115 bash[46934]: cluster 2022-04-22T18:53:22.400487+0000 mgr.y (mgr.34107) 3019 : cluster [DBG] pgmap v4561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:23 smithi115 bash[46934]: audit 2022-04-22T18:53:22.935103+0000 mon.a (mon.0) 6054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:24.118 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:23.813Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:24 smithi121 bash[36902]: audit 2022-04-22T18:53:24.043443+0000 mon.a (mon.0) 6055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:24 smithi115 bash[45495]: audit 2022-04-22T18:53:24.043443+0000 mon.a (mon.0) 6055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:24.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:24 smithi115 bash[46934]: audit 2022-04-22T18:53:24.043443+0000 mon.a (mon.0) 6055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:25 smithi121 bash[36902]: cluster 2022-04-22T18:53:24.401607+0000 mgr.y (mgr.34107) 3020 : cluster [DBG] pgmap v4562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:25 smithi115 bash[45495]: cluster 2022-04-22T18:53:24.401607+0000 mgr.y (mgr.34107) 3020 : cluster [DBG] pgmap v4562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:25 smithi115 bash[46934]: cluster 2022-04-22T18:53:24.401607+0000 mgr.y (mgr.34107) 3020 : cluster [DBG] pgmap v4562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:27.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:27 smithi121 bash[36902]: cluster 2022-04-22T18:53:26.402697+0000 mgr.y (mgr.34107) 3021 : cluster [DBG] pgmap v4563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:27 smithi115 bash[45495]: cluster 2022-04-22T18:53:26.402697+0000 mgr.y (mgr.34107) 3021 : cluster [DBG] pgmap v4563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:27 smithi115 bash[46934]: cluster 2022-04-22T18:53:26.402697+0000 mgr.y (mgr.34107) 3021 : cluster [DBG] pgmap v4563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:28 smithi121 bash[36902]: audit 2022-04-22T18:53:27.938666+0000 mon.a (mon.0) 6056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:28 smithi115 bash[45495]: audit 2022-04-22T18:53:27.938666+0000 mon.a (mon.0) 6056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:28.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:28 smithi115 bash[46934]: audit 2022-04-22T18:53:27.938666+0000 mon.a (mon.0) 6056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:29 smithi121 bash[36902]: cluster 2022-04-22T18:53:28.403802+0000 mgr.y (mgr.34107) 3022 : cluster [DBG] pgmap v4564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:29 smithi121 bash[36902]: audit 2022-04-22T18:53:29.046766+0000 mon.a (mon.0) 6057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:29 smithi115 bash[45495]: cluster 2022-04-22T18:53:28.403802+0000 mgr.y (mgr.34107) 3022 : cluster [DBG] pgmap v4564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:29 smithi115 bash[45495]: audit 2022-04-22T18:53:29.046766+0000 mon.a (mon.0) 6057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:29 smithi115 bash[46934]: cluster 2022-04-22T18:53:28.403802+0000 mgr.y (mgr.34107) 3022 : cluster [DBG] pgmap v4564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:29 smithi115 bash[46934]: audit 2022-04-22T18:53:29.046766+0000 mon.a (mon.0) 6057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:31 smithi121 bash[36902]: cluster 2022-04-22T18:53:30.404784+0000 mgr.y (mgr.34107) 3023 : cluster [DBG] pgmap v4565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:31 smithi115 bash[45495]: cluster 2022-04-22T18:53:30.404784+0000 mgr.y (mgr.34107) 3023 : cluster [DBG] pgmap v4565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:31 smithi115 bash[46934]: cluster 2022-04-22T18:53:30.404784+0000 mgr.y (mgr.34107) 3023 : cluster [DBG] pgmap v4565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:31.942Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:33 smithi121 bash[36902]: cluster 2022-04-22T18:53:32.405868+0000 mgr.y (mgr.34107) 3024 : cluster [DBG] pgmap v4566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:33 smithi121 bash[36902]: audit 2022-04-22T18:53:32.942273+0000 mon.a (mon.0) 6058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:33 smithi115 bash[45495]: cluster 2022-04-22T18:53:32.405868+0000 mgr.y (mgr.34107) 3024 : cluster [DBG] pgmap v4566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:33 smithi115 bash[45495]: audit 2022-04-22T18:53:32.942273+0000 mon.a (mon.0) 6058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:33 smithi115 bash[46934]: cluster 2022-04-22T18:53:32.405868+0000 mgr.y (mgr.34107) 3024 : cluster [DBG] pgmap v4566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:33 smithi115 bash[46934]: audit 2022-04-22T18:53:32.942273+0000 mon.a (mon.0) 6058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:34.123 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:33.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:34 smithi121 bash[36902]: audit 2022-04-22T18:53:34.050121+0000 mon.a (mon.0) 6059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:34 smithi115 bash[45495]: audit 2022-04-22T18:53:34.050121+0000 mon.a (mon.0) 6059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:34 smithi115 bash[46934]: audit 2022-04-22T18:53:34.050121+0000 mon.a (mon.0) 6059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:35 smithi121 bash[36902]: cluster 2022-04-22T18:53:34.406950+0000 mgr.y (mgr.34107) 3025 : cluster [DBG] pgmap v4567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:35 smithi115 bash[45495]: cluster 2022-04-22T18:53:34.406950+0000 mgr.y (mgr.34107) 3025 : cluster [DBG] pgmap v4567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:35 smithi115 bash[46934]: cluster 2022-04-22T18:53:34.406950+0000 mgr.y (mgr.34107) 3025 : cluster [DBG] pgmap v4567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:37 smithi121 bash[36902]: cluster 2022-04-22T18:53:36.408021+0000 mgr.y (mgr.34107) 3026 : cluster [DBG] pgmap v4568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:37 smithi115 bash[46934]: cluster 2022-04-22T18:53:36.408021+0000 mgr.y (mgr.34107) 3026 : cluster [DBG] pgmap v4568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:37 smithi115 bash[45495]: cluster 2022-04-22T18:53:36.408021+0000 mgr.y (mgr.34107) 3026 : cluster [DBG] pgmap v4568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:38 smithi121 bash[36902]: audit 2022-04-22T18:53:37.946142+0000 mon.a (mon.0) 6060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:38 smithi115 bash[45495]: audit 2022-04-22T18:53:37.946142+0000 mon.a (mon.0) 6060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:38 smithi115 bash[46934]: audit 2022-04-22T18:53:37.946142+0000 mon.a (mon.0) 6060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:39 smithi121 bash[36902]: cluster 2022-04-22T18:53:38.409013+0000 mgr.y (mgr.34107) 3027 : cluster [DBG] pgmap v4569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:39 smithi121 bash[36902]: audit 2022-04-22T18:53:39.053222+0000 mon.a (mon.0) 6061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:39 smithi115 bash[45495]: cluster 2022-04-22T18:53:38.409013+0000 mgr.y (mgr.34107) 3027 : cluster [DBG] pgmap v4569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:39 smithi115 bash[45495]: audit 2022-04-22T18:53:39.053222+0000 mon.a (mon.0) 6061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:39 smithi115 bash[46934]: cluster 2022-04-22T18:53:38.409013+0000 mgr.y (mgr.34107) 3027 : cluster [DBG] pgmap v4569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:39 smithi115 bash[46934]: audit 2022-04-22T18:53:39.053222+0000 mon.a (mon.0) 6061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:41 smithi121 bash[36902]: cluster 2022-04-22T18:53:40.409984+0000 mgr.y (mgr.34107) 3028 : cluster [DBG] pgmap v4570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:41 smithi115 bash[45495]: cluster 2022-04-22T18:53:40.409984+0000 mgr.y (mgr.34107) 3028 : cluster [DBG] pgmap v4570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:41 smithi115 bash[46934]: cluster 2022-04-22T18:53:40.409984+0000 mgr.y (mgr.34107) 3028 : cluster [DBG] pgmap v4570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:41.943Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:43 smithi121 bash[36902]: cluster 2022-04-22T18:53:42.411154+0000 mgr.y (mgr.34107) 3029 : cluster [DBG] pgmap v4571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:43 smithi121 bash[36902]: audit 2022-04-22T18:53:42.949983+0000 mon.a (mon.0) 6062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:43 smithi115 bash[45495]: cluster 2022-04-22T18:53:42.411154+0000 mgr.y (mgr.34107) 3029 : cluster [DBG] pgmap v4571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:43.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:43 smithi115 bash[45495]: audit 2022-04-22T18:53:42.949983+0000 mon.a (mon.0) 6062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:43.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:43 smithi115 bash[46934]: cluster 2022-04-22T18:53:42.411154+0000 mgr.y (mgr.34107) 3029 : cluster [DBG] pgmap v4571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:43.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:43 smithi115 bash[46934]: audit 2022-04-22T18:53:42.949983+0000 mon.a (mon.0) 6062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:44.124 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:43.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:44 smithi121 bash[36902]: audit 2022-04-22T18:53:44.056396+0000 mon.a (mon.0) 6063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:44.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:44 smithi115 bash[45495]: audit 2022-04-22T18:53:44.056396+0000 mon.a (mon.0) 6063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:44.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:44 smithi115 bash[46934]: audit 2022-04-22T18:53:44.056396+0000 mon.a (mon.0) 6063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:45 smithi121 bash[36902]: cluster 2022-04-22T18:53:44.412333+0000 mgr.y (mgr.34107) 3030 : cluster [DBG] pgmap v4572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:45 smithi115 bash[45495]: cluster 2022-04-22T18:53:44.412333+0000 mgr.y (mgr.34107) 3030 : cluster [DBG] pgmap v4572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:45 smithi115 bash[46934]: cluster 2022-04-22T18:53:44.412333+0000 mgr.y (mgr.34107) 3030 : cluster [DBG] pgmap v4572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:47 smithi121 bash[36902]: cluster 2022-04-22T18:53:46.413542+0000 mgr.y (mgr.34107) 3031 : cluster [DBG] pgmap v4573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:47 smithi115 bash[45495]: cluster 2022-04-22T18:53:46.413542+0000 mgr.y (mgr.34107) 3031 : cluster [DBG] pgmap v4573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:47 smithi115 bash[46934]: cluster 2022-04-22T18:53:46.413542+0000 mgr.y (mgr.34107) 3031 : cluster [DBG] pgmap v4573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:48.124 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:53:48.471 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (76m) 4m ago 90m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (90m) 5m ago 90m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (78m) 4m ago 91m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (78m) 5m ago 91m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (86m) 5m ago 95m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (86m) 4m ago 98m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (85m) 4m ago 98m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:53:48.613 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (84m) 5m ago 96m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (85m) 4m ago 96m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (77m) 4m ago 90m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (77m) 5m ago 90m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (83m) 4m ago 95m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (82m) 4m ago 95m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (82m) 4m ago 94m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (82m) 4m ago 94m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (81m) 5m ago 93m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (80m) 5m ago 93m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:53:48.614 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (79m) 5m ago 92m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:53:48.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (79m) 5m ago 92m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:53:48.615 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (77m) 5m ago 91m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:53:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:48 smithi121 bash[36902]: audit 2022-04-22T18:53:47.953976+0000 mon.a (mon.0) 6064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:48 smithi115 bash[45495]: audit 2022-04-22T18:53:47.953976+0000 mon.a (mon.0) 6064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:48 smithi115 bash[46934]: audit 2022-04-22T18:53:47.953976+0000 mon.a (mon.0) 6064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:48.878 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:53:48.878 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:53:48.878 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:53:48.879 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:53:48.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:53:48.880 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:53:48.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:53:48.880 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:53:48.880 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:53:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.098033+0000 mgr.y (mgr.34107) 3032 : audit [DBG] from='client.48526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: cluster 2022-04-22T18:53:48.414678+0000 mgr.y (mgr.34107) 3033 : cluster [DBG] pgmap v4574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.442552+0000 mgr.y (mgr.34107) 3034 : audit [DBG] from='client.38598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.594590+0000 mon.a (mon.0) 6065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.596620+0000 mon.a (mon.0) 6066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.597718+0000 mon.a (mon.0) 6067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.869980+0000 mon.c (mon.1) 164 : audit [DBG] from='client.? 172.21.15.115:0/4063781527' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:48.954226+0000 mon.a (mon.0) 6068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:53:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:49 smithi121 bash[36902]: audit 2022-04-22T18:53:49.059534+0000 mon.a (mon.0) 6069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.098033+0000 mgr.y (mgr.34107) 3032 : audit [DBG] from='client.48526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: cluster 2022-04-22T18:53:48.414678+0000 mgr.y (mgr.34107) 3033 : cluster [DBG] pgmap v4574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.442552+0000 mgr.y (mgr.34107) 3034 : audit [DBG] from='client.38598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.594590+0000 mon.a (mon.0) 6065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:53:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.596620+0000 mon.a (mon.0) 6066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:53:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.597718+0000 mon.a (mon.0) 6067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:53:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.869980+0000 mon.c (mon.1) 164 : audit [DBG] from='client.? 172.21.15.115:0/4063781527' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:48.954226+0000 mon.a (mon.0) 6068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:53:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[45495]: audit 2022-04-22T18:53:49.059534+0000 mon.a (mon.0) 6069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.098033+0000 mgr.y (mgr.34107) 3032 : audit [DBG] from='client.48526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: cluster 2022-04-22T18:53:48.414678+0000 mgr.y (mgr.34107) 3033 : cluster [DBG] pgmap v4574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.442552+0000 mgr.y (mgr.34107) 3034 : audit [DBG] from='client.38598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:53:49.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.594590+0000 mon.a (mon.0) 6065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:53:49.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.596620+0000 mon.a (mon.0) 6066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:53:49.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.597718+0000 mon.a (mon.0) 6067 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:53:49.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.869980+0000 mon.c (mon.1) 164 : audit [DBG] from='client.? 172.21.15.115:0/4063781527' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:53:49.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:48.954226+0000 mon.a (mon.0) 6068 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:53:49.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:49 smithi115 bash[46934]: audit 2022-04-22T18:53:49.059534+0000 mon.a (mon.0) 6069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:51 smithi121 bash[36902]: cluster 2022-04-22T18:53:50.415764+0000 mgr.y (mgr.34107) 3035 : cluster [DBG] pgmap v4575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:51 smithi115 bash[45495]: cluster 2022-04-22T18:53:50.415764+0000 mgr.y (mgr.34107) 3035 : cluster [DBG] pgmap v4575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:51 smithi115 bash[46934]: cluster 2022-04-22T18:53:50.415764+0000 mgr.y (mgr.34107) 3035 : cluster [DBG] pgmap v4575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:51.944Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:53.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:53 smithi121 bash[36902]: cluster 2022-04-22T18:53:52.416829+0000 mgr.y (mgr.34107) 3036 : cluster [DBG] pgmap v4576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:53.530 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:53 smithi121 bash[36902]: audit 2022-04-22T18:53:52.957673+0000 mon.a (mon.0) 6070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:53.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:53 smithi115 bash[45495]: cluster 2022-04-22T18:53:52.416829+0000 mgr.y (mgr.34107) 3036 : cluster [DBG] pgmap v4576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:53.569 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:53 smithi115 bash[45495]: audit 2022-04-22T18:53:52.957673+0000 mon.a (mon.0) 6070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:53.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:53 smithi115 bash[46934]: cluster 2022-04-22T18:53:52.416829+0000 mgr.y (mgr.34107) 3036 : cluster [DBG] pgmap v4576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:53.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:53 smithi115 bash[46934]: audit 2022-04-22T18:53:52.957673+0000 mon.a (mon.0) 6070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:54.127 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:53:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:53:53.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:53:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:54 smithi121 bash[36902]: audit 2022-04-22T18:53:54.062871+0000 mon.a (mon.0) 6071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:54.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:54 smithi115 bash[45495]: audit 2022-04-22T18:53:54.062871+0000 mon.a (mon.0) 6071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:54.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:54 smithi115 bash[46934]: audit 2022-04-22T18:53:54.062871+0000 mon.a (mon.0) 6071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:55 smithi121 bash[36902]: cluster 2022-04-22T18:53:54.418056+0000 mgr.y (mgr.34107) 3037 : cluster [DBG] pgmap v4577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:55 smithi115 bash[45495]: cluster 2022-04-22T18:53:54.418056+0000 mgr.y (mgr.34107) 3037 : cluster [DBG] pgmap v4577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:55 smithi115 bash[46934]: cluster 2022-04-22T18:53:54.418056+0000 mgr.y (mgr.34107) 3037 : cluster [DBG] pgmap v4577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:57 smithi121 bash[36902]: cluster 2022-04-22T18:53:56.419264+0000 mgr.y (mgr.34107) 3038 : cluster [DBG] pgmap v4578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:57 smithi115 bash[45495]: cluster 2022-04-22T18:53:56.419264+0000 mgr.y (mgr.34107) 3038 : cluster [DBG] pgmap v4578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:57 smithi115 bash[46934]: cluster 2022-04-22T18:53:56.419264+0000 mgr.y (mgr.34107) 3038 : cluster [DBG] pgmap v4578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:58 smithi121 bash[36902]: audit 2022-04-22T18:53:57.961444+0000 mon.a (mon.0) 6072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:58 smithi115 bash[45495]: audit 2022-04-22T18:53:57.961444+0000 mon.a (mon.0) 6072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:58 smithi115 bash[46934]: audit 2022-04-22T18:53:57.961444+0000 mon.a (mon.0) 6072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:53:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:59 smithi121 bash[36902]: cluster 2022-04-22T18:53:58.420394+0000 mgr.y (mgr.34107) 3039 : cluster [DBG] pgmap v4579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:53:59 smithi121 bash[36902]: audit 2022-04-22T18:53:59.065712+0000 mon.a (mon.0) 6073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:59 smithi115 bash[45495]: cluster 2022-04-22T18:53:58.420394+0000 mgr.y (mgr.34107) 3039 : cluster [DBG] pgmap v4579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:53:59 smithi115 bash[45495]: audit 2022-04-22T18:53:59.065712+0000 mon.a (mon.0) 6073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:53:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:59 smithi115 bash[46934]: cluster 2022-04-22T18:53:58.420394+0000 mgr.y (mgr.34107) 3039 : cluster [DBG] pgmap v4579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:53:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:53:59 smithi115 bash[46934]: audit 2022-04-22T18:53:59.065712+0000 mon.a (mon.0) 6073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:01 smithi121 bash[36902]: cluster 2022-04-22T18:54:00.421339+0000 mgr.y (mgr.34107) 3040 : cluster [DBG] pgmap v4580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:01 smithi115 bash[45495]: cluster 2022-04-22T18:54:00.421339+0000 mgr.y (mgr.34107) 3040 : cluster [DBG] pgmap v4580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:01.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:01 smithi115 bash[46934]: cluster 2022-04-22T18:54:00.421339+0000 mgr.y (mgr.34107) 3040 : cluster [DBG] pgmap v4580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:01.944Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:03 smithi121 bash[36902]: cluster 2022-04-22T18:54:02.422498+0000 mgr.y (mgr.34107) 3041 : cluster [DBG] pgmap v4581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:03 smithi121 bash[36902]: audit 2022-04-22T18:54:02.965089+0000 mon.a (mon.0) 6074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:03 smithi115 bash[45495]: cluster 2022-04-22T18:54:02.422498+0000 mgr.y (mgr.34107) 3041 : cluster [DBG] pgmap v4581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:03 smithi115 bash[45495]: audit 2022-04-22T18:54:02.965089+0000 mon.a (mon.0) 6074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:03 smithi115 bash[46934]: cluster 2022-04-22T18:54:02.422498+0000 mgr.y (mgr.34107) 3041 : cluster [DBG] pgmap v4581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:03 smithi115 bash[46934]: audit 2022-04-22T18:54:02.965089+0000 mon.a (mon.0) 6074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:04.135 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:03.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:04 smithi121 bash[36902]: audit 2022-04-22T18:54:04.069152+0000 mon.a (mon.0) 6075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:04 smithi115 bash[45495]: audit 2022-04-22T18:54:04.069152+0000 mon.a (mon.0) 6075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:04 smithi115 bash[46934]: audit 2022-04-22T18:54:04.069152+0000 mon.a (mon.0) 6075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:05.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:05 smithi121 bash[36902]: cluster 2022-04-22T18:54:04.423616+0000 mgr.y (mgr.34107) 3042 : cluster [DBG] pgmap v4582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:05 smithi115 bash[45495]: cluster 2022-04-22T18:54:04.423616+0000 mgr.y (mgr.34107) 3042 : cluster [DBG] pgmap v4582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:05 smithi115 bash[46934]: cluster 2022-04-22T18:54:04.423616+0000 mgr.y (mgr.34107) 3042 : cluster [DBG] pgmap v4582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:07 smithi121 bash[36902]: cluster 2022-04-22T18:54:06.424633+0000 mgr.y (mgr.34107) 3043 : cluster [DBG] pgmap v4583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:07 smithi115 bash[45495]: cluster 2022-04-22T18:54:06.424633+0000 mgr.y (mgr.34107) 3043 : cluster [DBG] pgmap v4583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:07 smithi115 bash[46934]: cluster 2022-04-22T18:54:06.424633+0000 mgr.y (mgr.34107) 3043 : cluster [DBG] pgmap v4583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:08 smithi121 bash[36902]: audit 2022-04-22T18:54:07.968556+0000 mon.a (mon.0) 6076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:08 smithi115 bash[45495]: audit 2022-04-22T18:54:07.968556+0000 mon.a (mon.0) 6076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:08 smithi115 bash[46934]: audit 2022-04-22T18:54:07.968556+0000 mon.a (mon.0) 6076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:09 smithi121 bash[36902]: cluster 2022-04-22T18:54:08.425753+0000 mgr.y (mgr.34107) 3044 : cluster [DBG] pgmap v4584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:09 smithi121 bash[36902]: audit 2022-04-22T18:54:09.072283+0000 mon.a (mon.0) 6077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:09 smithi115 bash[45495]: cluster 2022-04-22T18:54:08.425753+0000 mgr.y (mgr.34107) 3044 : cluster [DBG] pgmap v4584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:09 smithi115 bash[45495]: audit 2022-04-22T18:54:09.072283+0000 mon.a (mon.0) 6077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:09 smithi115 bash[46934]: cluster 2022-04-22T18:54:08.425753+0000 mgr.y (mgr.34107) 3044 : cluster [DBG] pgmap v4584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:09 smithi115 bash[46934]: audit 2022-04-22T18:54:09.072283+0000 mon.a (mon.0) 6077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:11.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:11 smithi121 bash[36902]: cluster 2022-04-22T18:54:10.426885+0000 mgr.y (mgr.34107) 3045 : cluster [DBG] pgmap v4585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:11 smithi115 bash[45495]: cluster 2022-04-22T18:54:10.426885+0000 mgr.y (mgr.34107) 3045 : cluster [DBG] pgmap v4585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:11 smithi115 bash[46934]: cluster 2022-04-22T18:54:10.426885+0000 mgr.y (mgr.34107) 3045 : cluster [DBG] pgmap v4585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:11.946Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:13 smithi121 bash[36902]: cluster 2022-04-22T18:54:12.428133+0000 mgr.y (mgr.34107) 3046 : cluster [DBG] pgmap v4586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:13 smithi121 bash[36902]: audit 2022-04-22T18:54:12.972180+0000 mon.a (mon.0) 6078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:13 smithi115 bash[45495]: cluster 2022-04-22T18:54:12.428133+0000 mgr.y (mgr.34107) 3046 : cluster [DBG] pgmap v4586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:13 smithi115 bash[45495]: audit 2022-04-22T18:54:12.972180+0000 mon.a (mon.0) 6078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:13 smithi115 bash[46934]: cluster 2022-04-22T18:54:12.428133+0000 mgr.y (mgr.34107) 3046 : cluster [DBG] pgmap v4586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:13 smithi115 bash[46934]: audit 2022-04-22T18:54:12.972180+0000 mon.a (mon.0) 6078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:14.133 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:13.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:14 smithi121 bash[36902]: audit 2022-04-22T18:54:14.075563+0000 mon.a (mon.0) 6079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:14.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:14 smithi115 bash[45495]: audit 2022-04-22T18:54:14.075563+0000 mon.a (mon.0) 6079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:14.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:14 smithi115 bash[46934]: audit 2022-04-22T18:54:14.075563+0000 mon.a (mon.0) 6079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:15 smithi121 bash[36902]: audit 2022-04-22T18:54:14.238286+0000 mon.a (mon.0) 6080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:54:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:15 smithi121 bash[36902]: audit 2022-04-22T18:54:14.238700+0000 mon.a (mon.0) 6081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:54:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:15 smithi121 bash[36902]: cluster 2022-04-22T18:54:14.429272+0000 mgr.y (mgr.34107) 3047 : cluster [DBG] pgmap v4587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[45495]: audit 2022-04-22T18:54:14.238286+0000 mon.a (mon.0) 6080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:54:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[45495]: audit 2022-04-22T18:54:14.238700+0000 mon.a (mon.0) 6081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:54:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[45495]: cluster 2022-04-22T18:54:14.429272+0000 mgr.y (mgr.34107) 3047 : cluster [DBG] pgmap v4587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[46934]: audit 2022-04-22T18:54:14.238286+0000 mon.a (mon.0) 6080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:54:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[46934]: audit 2022-04-22T18:54:14.238700+0000 mon.a (mon.0) 6081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:54:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:15 smithi115 bash[46934]: cluster 2022-04-22T18:54:14.429272+0000 mgr.y (mgr.34107) 3047 : cluster [DBG] pgmap v4587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:17 smithi121 bash[36902]: cluster 2022-04-22T18:54:16.430321+0000 mgr.y (mgr.34107) 3048 : cluster [DBG] pgmap v4588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:17 smithi115 bash[45495]: cluster 2022-04-22T18:54:16.430321+0000 mgr.y (mgr.34107) 3048 : cluster [DBG] pgmap v4588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:17 smithi115 bash[46934]: cluster 2022-04-22T18:54:16.430321+0000 mgr.y (mgr.34107) 3048 : cluster [DBG] pgmap v4588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:18 smithi121 bash[36902]: audit 2022-04-22T18:54:17.976113+0000 mon.a (mon.0) 6082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:18 smithi115 bash[45495]: audit 2022-04-22T18:54:17.976113+0000 mon.a (mon.0) 6082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:18 smithi115 bash[46934]: audit 2022-04-22T18:54:17.976113+0000 mon.a (mon.0) 6082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:19.245 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:54:19.590 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:54:19.590 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (77m) 4m ago 91m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:54:19.590 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (90m) 5m ago 90m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:54:19.590 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (79m) 4m ago 92m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (78m) 5m ago 92m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (86m) 5m ago 96m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (87m) 4m ago 98m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (86m) 4m ago 98m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (85m) 5m ago 96m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (85m) 4m ago 96m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (78m) 4m ago 91m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (78m) 5m ago 91m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (84m) 4m ago 96m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (83m) 4m ago 95m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:54:19.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (82m) 4m ago 95m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (82m) 4m ago 94m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (81m) 5m ago 94m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (81m) 5m ago 93m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (80m) 5m ago 93m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (79m) 5m ago 92m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:54:19.592 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (78m) 5m ago 91m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:54:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:19 smithi121 bash[36902]: cluster 2022-04-22T18:54:18.431506+0000 mgr.y (mgr.34107) 3049 : cluster [DBG] pgmap v4589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:19.854 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:19 smithi121 bash[36902]: audit 2022-04-22T18:54:19.078324+0000 mon.a (mon.0) 6083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:19 smithi115 bash[45495]: cluster 2022-04-22T18:54:18.431506+0000 mgr.y (mgr.34107) 3049 : cluster [DBG] pgmap v4589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:19 smithi115 bash[45495]: audit 2022-04-22T18:54:19.078324+0000 mon.a (mon.0) 6083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:19.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:19 smithi115 bash[46934]: cluster 2022-04-22T18:54:18.431506+0000 mgr.y (mgr.34107) 3049 : cluster [DBG] pgmap v4589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:19.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:19 smithi115 bash[46934]: audit 2022-04-22T18:54:19.078324+0000 mon.a (mon.0) 6083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:54:19.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:54:19.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:19.993 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:54:19.993 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:54:19.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:54:19.994 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:54:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[45495]: audit 2022-04-22T18:54:19.218024+0000 mgr.y (mgr.34107) 3050 : audit [DBG] from='client.48544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[45495]: audit 2022-04-22T18:54:19.560738+0000 mgr.y (mgr.34107) 3051 : audit [DBG] from='client.38616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[45495]: audit 2022-04-22T18:54:19.983151+0000 mon.c (mon.1) 165 : audit [DBG] from='client.? 172.21.15.115:0/430205858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[46934]: audit 2022-04-22T18:54:19.218024+0000 mgr.y (mgr.34107) 3050 : audit [DBG] from='client.48544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[46934]: audit 2022-04-22T18:54:19.560738+0000 mgr.y (mgr.34107) 3051 : audit [DBG] from='client.38616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:20 smithi115 bash[46934]: audit 2022-04-22T18:54:19.983151+0000 mon.c (mon.1) 165 : audit [DBG] from='client.? 172.21.15.115:0/430205858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:20 smithi121 bash[36902]: audit 2022-04-22T18:54:19.218024+0000 mgr.y (mgr.34107) 3050 : audit [DBG] from='client.48544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:20 smithi121 bash[36902]: audit 2022-04-22T18:54:19.560738+0000 mgr.y (mgr.34107) 3051 : audit [DBG] from='client.38616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:21.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:20 smithi121 bash[36902]: audit 2022-04-22T18:54:19.983151+0000 mon.c (mon.1) 165 : audit [DBG] from='client.? 172.21.15.115:0/430205858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:21 smithi115 bash[45495]: cluster 2022-04-22T18:54:20.432510+0000 mgr.y (mgr.34107) 3052 : cluster [DBG] pgmap v4590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:21 smithi115 bash[46934]: cluster 2022-04-22T18:54:20.432510+0000 mgr.y (mgr.34107) 3052 : cluster [DBG] pgmap v4590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:21 smithi121 bash[36902]: cluster 2022-04-22T18:54:20.432510+0000 mgr.y (mgr.34107) 3052 : cluster [DBG] pgmap v4590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:21.945Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:23 smithi121 bash[36902]: cluster 2022-04-22T18:54:22.433589+0000 mgr.y (mgr.34107) 3053 : cluster [DBG] pgmap v4591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:23 smithi121 bash[36902]: audit 2022-04-22T18:54:22.979733+0000 mon.a (mon.0) 6084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:23 smithi115 bash[45495]: cluster 2022-04-22T18:54:22.433589+0000 mgr.y (mgr.34107) 3053 : cluster [DBG] pgmap v4591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:23 smithi115 bash[45495]: audit 2022-04-22T18:54:22.979733+0000 mon.a (mon.0) 6084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:23 smithi115 bash[46934]: cluster 2022-04-22T18:54:22.433589+0000 mgr.y (mgr.34107) 3053 : cluster [DBG] pgmap v4591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:23 smithi115 bash[46934]: audit 2022-04-22T18:54:22.979733+0000 mon.a (mon.0) 6084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:24.140 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:23.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:24 smithi121 bash[36902]: audit 2022-04-22T18:54:24.081061+0000 mon.a (mon.0) 6085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:24 smithi115 bash[45495]: audit 2022-04-22T18:54:24.081061+0000 mon.a (mon.0) 6085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:24 smithi115 bash[46934]: audit 2022-04-22T18:54:24.081061+0000 mon.a (mon.0) 6085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:25 smithi121 bash[36902]: cluster 2022-04-22T18:54:24.434678+0000 mgr.y (mgr.34107) 3054 : cluster [DBG] pgmap v4592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:25 smithi115 bash[45495]: cluster 2022-04-22T18:54:24.434678+0000 mgr.y (mgr.34107) 3054 : cluster [DBG] pgmap v4592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:25 smithi115 bash[46934]: cluster 2022-04-22T18:54:24.434678+0000 mgr.y (mgr.34107) 3054 : cluster [DBG] pgmap v4592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:27 smithi121 bash[36902]: cluster 2022-04-22T18:54:26.435814+0000 mgr.y (mgr.34107) 3055 : cluster [DBG] pgmap v4593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:27 smithi115 bash[45495]: cluster 2022-04-22T18:54:26.435814+0000 mgr.y (mgr.34107) 3055 : cluster [DBG] pgmap v4593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:27 smithi115 bash[46934]: cluster 2022-04-22T18:54:26.435814+0000 mgr.y (mgr.34107) 3055 : cluster [DBG] pgmap v4593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:28 smithi121 bash[36902]: audit 2022-04-22T18:54:27.983383+0000 mon.a (mon.0) 6086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:28 smithi115 bash[45495]: audit 2022-04-22T18:54:27.983383+0000 mon.a (mon.0) 6086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:28 smithi115 bash[46934]: audit 2022-04-22T18:54:27.983383+0000 mon.a (mon.0) 6086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:29 smithi121 bash[36902]: cluster 2022-04-22T18:54:28.437029+0000 mgr.y (mgr.34107) 3056 : cluster [DBG] pgmap v4594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:29 smithi121 bash[36902]: audit 2022-04-22T18:54:29.084231+0000 mon.a (mon.0) 6087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:29 smithi115 bash[46934]: cluster 2022-04-22T18:54:28.437029+0000 mgr.y (mgr.34107) 3056 : cluster [DBG] pgmap v4594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:29 smithi115 bash[46934]: audit 2022-04-22T18:54:29.084231+0000 mon.a (mon.0) 6087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:29 smithi115 bash[45495]: cluster 2022-04-22T18:54:28.437029+0000 mgr.y (mgr.34107) 3056 : cluster [DBG] pgmap v4594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:29 smithi115 bash[45495]: audit 2022-04-22T18:54:29.084231+0000 mon.a (mon.0) 6087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:31 smithi121 bash[36902]: cluster 2022-04-22T18:54:30.438131+0000 mgr.y (mgr.34107) 3057 : cluster [DBG] pgmap v4595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:31 smithi115 bash[45495]: cluster 2022-04-22T18:54:30.438131+0000 mgr.y (mgr.34107) 3057 : cluster [DBG] pgmap v4595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:31.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:31 smithi115 bash[46934]: cluster 2022-04-22T18:54:30.438131+0000 mgr.y (mgr.34107) 3057 : cluster [DBG] pgmap v4595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:31.948Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:33 smithi121 bash[36902]: cluster 2022-04-22T18:54:32.439238+0000 mgr.y (mgr.34107) 3058 : cluster [DBG] pgmap v4596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:33 smithi121 bash[36902]: audit 2022-04-22T18:54:32.986987+0000 mon.a (mon.0) 6088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:33 smithi115 bash[45495]: cluster 2022-04-22T18:54:32.439238+0000 mgr.y (mgr.34107) 3058 : cluster [DBG] pgmap v4596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:33 smithi115 bash[45495]: audit 2022-04-22T18:54:32.986987+0000 mon.a (mon.0) 6088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:33 smithi115 bash[46934]: cluster 2022-04-22T18:54:32.439238+0000 mgr.y (mgr.34107) 3058 : cluster [DBG] pgmap v4596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:33.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:33 smithi115 bash[46934]: audit 2022-04-22T18:54:32.986987+0000 mon.a (mon.0) 6088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:34.144 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:33.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:34 smithi121 bash[36902]: audit 2022-04-22T18:54:34.087502+0000 mon.a (mon.0) 6089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:34 smithi115 bash[45495]: audit 2022-04-22T18:54:34.087502+0000 mon.a (mon.0) 6089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:34 smithi115 bash[46934]: audit 2022-04-22T18:54:34.087502+0000 mon.a (mon.0) 6089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:35 smithi121 bash[36902]: cluster 2022-04-22T18:54:34.440366+0000 mgr.y (mgr.34107) 3059 : cluster [DBG] pgmap v4597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:35 smithi115 bash[45495]: cluster 2022-04-22T18:54:34.440366+0000 mgr.y (mgr.34107) 3059 : cluster [DBG] pgmap v4597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:35 smithi115 bash[46934]: cluster 2022-04-22T18:54:34.440366+0000 mgr.y (mgr.34107) 3059 : cluster [DBG] pgmap v4597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:37 smithi121 bash[36902]: cluster 2022-04-22T18:54:36.441428+0000 mgr.y (mgr.34107) 3060 : cluster [DBG] pgmap v4598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:37 smithi115 bash[45495]: cluster 2022-04-22T18:54:36.441428+0000 mgr.y (mgr.34107) 3060 : cluster [DBG] pgmap v4598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:37 smithi115 bash[46934]: cluster 2022-04-22T18:54:36.441428+0000 mgr.y (mgr.34107) 3060 : cluster [DBG] pgmap v4598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:38 smithi121 bash[36902]: audit 2022-04-22T18:54:37.990618+0000 mon.a (mon.0) 6090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:38 smithi115 bash[45495]: audit 2022-04-22T18:54:37.990618+0000 mon.a (mon.0) 6090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:38 smithi115 bash[46934]: audit 2022-04-22T18:54:37.990618+0000 mon.a (mon.0) 6090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:39 smithi115 bash[45495]: cluster 2022-04-22T18:54:38.442588+0000 mgr.y (mgr.34107) 3061 : cluster [DBG] pgmap v4599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:39 smithi115 bash[45495]: audit 2022-04-22T18:54:39.090691+0000 mon.a (mon.0) 6091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:39 smithi115 bash[46934]: cluster 2022-04-22T18:54:38.442588+0000 mgr.y (mgr.34107) 3061 : cluster [DBG] pgmap v4599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:39 smithi115 bash[46934]: audit 2022-04-22T18:54:39.090691+0000 mon.a (mon.0) 6091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:39 smithi121 bash[36902]: cluster 2022-04-22T18:54:38.442588+0000 mgr.y (mgr.34107) 3061 : cluster [DBG] pgmap v4599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:39 smithi121 bash[36902]: audit 2022-04-22T18:54:39.090691+0000 mon.a (mon.0) 6091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:41 smithi115 bash[45495]: cluster 2022-04-22T18:54:40.443701+0000 mgr.y (mgr.34107) 3062 : cluster [DBG] pgmap v4600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:41 smithi115 bash[46934]: cluster 2022-04-22T18:54:40.443701+0000 mgr.y (mgr.34107) 3062 : cluster [DBG] pgmap v4600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:41 smithi121 bash[36902]: cluster 2022-04-22T18:54:40.443701+0000 mgr.y (mgr.34107) 3062 : cluster [DBG] pgmap v4600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:41.947Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:43 smithi121 bash[36902]: cluster 2022-04-22T18:54:42.444753+0000 mgr.y (mgr.34107) 3063 : cluster [DBG] pgmap v4601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:43 smithi121 bash[36902]: audit 2022-04-22T18:54:42.993888+0000 mon.a (mon.0) 6092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:43 smithi115 bash[45495]: cluster 2022-04-22T18:54:42.444753+0000 mgr.y (mgr.34107) 3063 : cluster [DBG] pgmap v4601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:43 smithi115 bash[45495]: audit 2022-04-22T18:54:42.993888+0000 mon.a (mon.0) 6092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:43 smithi115 bash[46934]: cluster 2022-04-22T18:54:42.444753+0000 mgr.y (mgr.34107) 3063 : cluster [DBG] pgmap v4601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:43 smithi115 bash[46934]: audit 2022-04-22T18:54:42.993888+0000 mon.a (mon.0) 6092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:44.151 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:43.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:44.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:44 smithi121 bash[36902]: audit 2022-04-22T18:54:44.093765+0000 mon.a (mon.0) 6093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:44 smithi115 bash[45495]: audit 2022-04-22T18:54:44.093765+0000 mon.a (mon.0) 6093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:44 smithi115 bash[46934]: audit 2022-04-22T18:54:44.093765+0000 mon.a (mon.0) 6093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:45 smithi121 bash[36902]: cluster 2022-04-22T18:54:44.445825+0000 mgr.y (mgr.34107) 3064 : cluster [DBG] pgmap v4602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:45 smithi115 bash[45495]: cluster 2022-04-22T18:54:44.445825+0000 mgr.y (mgr.34107) 3064 : cluster [DBG] pgmap v4602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:45 smithi115 bash[46934]: cluster 2022-04-22T18:54:44.445825+0000 mgr.y (mgr.34107) 3064 : cluster [DBG] pgmap v4602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:47 smithi121 bash[36902]: cluster 2022-04-22T18:54:46.446924+0000 mgr.y (mgr.34107) 3065 : cluster [DBG] pgmap v4603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:47 smithi115 bash[45495]: cluster 2022-04-22T18:54:46.446924+0000 mgr.y (mgr.34107) 3065 : cluster [DBG] pgmap v4603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:47 smithi115 bash[46934]: cluster 2022-04-22T18:54:46.446924+0000 mgr.y (mgr.34107) 3065 : cluster [DBG] pgmap v4603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:48 smithi115 bash[45495]: audit 2022-04-22T18:54:47.997800+0000 mon.a (mon.0) 6094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:48 smithi115 bash[46934]: audit 2022-04-22T18:54:47.997800+0000 mon.a (mon.0) 6094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:48.997 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:48 smithi121 bash[36902]: audit 2022-04-22T18:54:47.997800+0000 mon.a (mon.0) 6094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: cluster 2022-04-22T18:54:48.448023+0000 mgr.y (mgr.34107) 3066 : cluster [DBG] pgmap v4604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: audit 2022-04-22T18:54:48.962839+0000 mon.a (mon.0) 6095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:54:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: audit 2022-04-22T18:54:48.965861+0000 mon.a (mon.0) 6096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:54:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: audit 2022-04-22T18:54:48.967576+0000 mon.a (mon.0) 6097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:54:49.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: audit 2022-04-22T18:54:49.096992+0000 mon.a (mon.0) 6098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:49.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:49 smithi121 bash[36902]: audit 2022-04-22T18:54:49.326608+0000 mon.a (mon.0) 6099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:54:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: cluster 2022-04-22T18:54:48.448023+0000 mgr.y (mgr.34107) 3066 : cluster [DBG] pgmap v4604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: audit 2022-04-22T18:54:48.962839+0000 mon.a (mon.0) 6095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:54:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: audit 2022-04-22T18:54:48.965861+0000 mon.a (mon.0) 6096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:54:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: audit 2022-04-22T18:54:48.967576+0000 mon.a (mon.0) 6097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:54:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: audit 2022-04-22T18:54:49.096992+0000 mon.a (mon.0) 6098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[45495]: audit 2022-04-22T18:54:49.326608+0000 mon.a (mon.0) 6099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:54:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: cluster 2022-04-22T18:54:48.448023+0000 mgr.y (mgr.34107) 3066 : cluster [DBG] pgmap v4604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: audit 2022-04-22T18:54:48.962839+0000 mon.a (mon.0) 6095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:54:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: audit 2022-04-22T18:54:48.965861+0000 mon.a (mon.0) 6096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:54:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: audit 2022-04-22T18:54:48.967576+0000 mon.a (mon.0) 6097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:54:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: audit 2022-04-22T18:54:49.096992+0000 mon.a (mon.0) 6098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:49 smithi115 bash[46934]: audit 2022-04-22T18:54:49.326608+0000 mon.a (mon.0) 6099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:54:50.380 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:54:50.725 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:54:50.725 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (77m) 5m ago 91m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:54:50.725 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (91m) 6m ago 91m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:54:50.725 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (79m) 5m ago 92m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (79m) 6m ago 92m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (87m) 6m ago 96m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (87m) 5m ago 99m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (86m) 5m ago 99m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (85m) 6m ago 97m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (86m) 5m ago 97m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (79m) 5m ago 91m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (78m) 6m ago 91m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:54:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (84m) 5m ago 96m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (83m) 5m ago 96m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (83m) 5m ago 95m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (83m) 5m ago 95m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (82m) 6m ago 94m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (81m) 6m ago 94m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (80m) 6m ago 93m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (80m) 6m ago 93m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:54:50.727 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (78m) 6m ago 92m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:54:51.125 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:54:51.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:54:51.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:54:51.365 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:54:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[45495]: audit 2022-04-22T18:54:50.353967+0000 mgr.y (mgr.34107) 3067 : audit [DBG] from='client.48562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[45495]: cluster 2022-04-22T18:54:50.449101+0000 mgr.y (mgr.34107) 3068 : cluster [DBG] pgmap v4605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[45495]: audit 2022-04-22T18:54:50.697731+0000 mgr.y (mgr.34107) 3069 : audit [DBG] from='client.48568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[45495]: audit 2022-04-22T18:54:51.116058+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.115:0/761015321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[46934]: audit 2022-04-22T18:54:50.353967+0000 mgr.y (mgr.34107) 3067 : audit [DBG] from='client.48562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[46934]: cluster 2022-04-22T18:54:50.449101+0000 mgr.y (mgr.34107) 3068 : cluster [DBG] pgmap v4605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[46934]: audit 2022-04-22T18:54:50.697731+0000 mgr.y (mgr.34107) 3069 : audit [DBG] from='client.48568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[46934]: audit 2022-04-22T18:54:51.116058+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.115:0/761015321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:51 smithi121 bash[36902]: audit 2022-04-22T18:54:50.353967+0000 mgr.y (mgr.34107) 3067 : audit [DBG] from='client.48562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:51 smithi121 bash[36902]: cluster 2022-04-22T18:54:50.449101+0000 mgr.y (mgr.34107) 3068 : cluster [DBG] pgmap v4605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:51 smithi121 bash[36902]: audit 2022-04-22T18:54:50.697731+0000 mgr.y (mgr.34107) 3069 : audit [DBG] from='client.48568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:54:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:51 smithi121 bash[36902]: audit 2022-04-22T18:54:51.116058+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.115:0/761015321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:54:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:51.948Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:53 smithi121 bash[36902]: cluster 2022-04-22T18:54:52.450219+0000 mgr.y (mgr.34107) 3070 : cluster [DBG] pgmap v4606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:53 smithi121 bash[36902]: audit 2022-04-22T18:54:53.001516+0000 mon.a (mon.0) 6100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:53 smithi115 bash[45495]: cluster 2022-04-22T18:54:52.450219+0000 mgr.y (mgr.34107) 3070 : cluster [DBG] pgmap v4606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:53 smithi115 bash[45495]: audit 2022-04-22T18:54:53.001516+0000 mon.a (mon.0) 6100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:53 smithi115 bash[46934]: cluster 2022-04-22T18:54:52.450219+0000 mgr.y (mgr.34107) 3070 : cluster [DBG] pgmap v4606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:53 smithi115 bash[46934]: audit 2022-04-22T18:54:53.001516+0000 mon.a (mon.0) 6100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:54.149 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:54:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:54:53.816Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:54:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:54 smithi121 bash[36902]: audit 2022-04-22T18:54:54.099888+0000 mon.a (mon.0) 6101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:54 smithi115 bash[45495]: audit 2022-04-22T18:54:54.099888+0000 mon.a (mon.0) 6101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:54 smithi115 bash[46934]: audit 2022-04-22T18:54:54.099888+0000 mon.a (mon.0) 6101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:55 smithi121 bash[36902]: cluster 2022-04-22T18:54:54.451415+0000 mgr.y (mgr.34107) 3071 : cluster [DBG] pgmap v4607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:55 smithi115 bash[45495]: cluster 2022-04-22T18:54:54.451415+0000 mgr.y (mgr.34107) 3071 : cluster [DBG] pgmap v4607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:55 smithi115 bash[46934]: cluster 2022-04-22T18:54:54.451415+0000 mgr.y (mgr.34107) 3071 : cluster [DBG] pgmap v4607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:57 smithi115 bash[45495]: cluster 2022-04-22T18:54:56.452603+0000 mgr.y (mgr.34107) 3072 : cluster [DBG] pgmap v4608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:57 smithi115 bash[46934]: cluster 2022-04-22T18:54:56.452603+0000 mgr.y (mgr.34107) 3072 : cluster [DBG] pgmap v4608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:57 smithi121 bash[36902]: cluster 2022-04-22T18:54:56.452603+0000 mgr.y (mgr.34107) 3072 : cluster [DBG] pgmap v4608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:58 smithi115 bash[45495]: audit 2022-04-22T18:54:58.005158+0000 mon.a (mon.0) 6102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:58 smithi115 bash[46934]: audit 2022-04-22T18:54:58.005158+0000 mon.a (mon.0) 6102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:58 smithi121 bash[36902]: audit 2022-04-22T18:54:58.005158+0000 mon.a (mon.0) 6102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:54:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:59 smithi115 bash[45495]: cluster 2022-04-22T18:54:58.453914+0000 mgr.y (mgr.34107) 3073 : cluster [DBG] pgmap v4609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:54:59 smithi115 bash[45495]: audit 2022-04-22T18:54:59.102941+0000 mon.a (mon.0) 6103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:54:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:59 smithi115 bash[46934]: cluster 2022-04-22T18:54:58.453914+0000 mgr.y (mgr.34107) 3073 : cluster [DBG] pgmap v4609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:54:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:54:59 smithi115 bash[46934]: audit 2022-04-22T18:54:59.102941+0000 mon.a (mon.0) 6103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:59 smithi121 bash[36902]: cluster 2022-04-22T18:54:58.453914+0000 mgr.y (mgr.34107) 3073 : cluster [DBG] pgmap v4609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:54:59 smithi121 bash[36902]: audit 2022-04-22T18:54:59.102941+0000 mon.a (mon.0) 6103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:01 smithi115 bash[45495]: cluster 2022-04-22T18:55:00.455110+0000 mgr.y (mgr.34107) 3074 : cluster [DBG] pgmap v4610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:01 smithi115 bash[46934]: cluster 2022-04-22T18:55:00.455110+0000 mgr.y (mgr.34107) 3074 : cluster [DBG] pgmap v4610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:02.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:01 smithi121 bash[36902]: cluster 2022-04-22T18:55:00.455110+0000 mgr.y (mgr.34107) 3074 : cluster [DBG] pgmap v4610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:01.949Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:03 smithi121 bash[36902]: cluster 2022-04-22T18:55:02.456258+0000 mgr.y (mgr.34107) 3075 : cluster [DBG] pgmap v4611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:03 smithi121 bash[36902]: audit 2022-04-22T18:55:03.008715+0000 mon.a (mon.0) 6104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:03 smithi115 bash[46934]: cluster 2022-04-22T18:55:02.456258+0000 mgr.y (mgr.34107) 3075 : cluster [DBG] pgmap v4611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:03 smithi115 bash[46934]: audit 2022-04-22T18:55:03.008715+0000 mon.a (mon.0) 6104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:03 smithi115 bash[45495]: cluster 2022-04-22T18:55:02.456258+0000 mgr.y (mgr.34107) 3075 : cluster [DBG] pgmap v4611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:03 smithi115 bash[45495]: audit 2022-04-22T18:55:03.008715+0000 mon.a (mon.0) 6104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:04.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:03.818Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:04 smithi121 bash[36902]: audit 2022-04-22T18:55:04.106383+0000 mon.a (mon.0) 6105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:04 smithi115 bash[45495]: audit 2022-04-22T18:55:04.106383+0000 mon.a (mon.0) 6105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:04 smithi115 bash[46934]: audit 2022-04-22T18:55:04.106383+0000 mon.a (mon.0) 6105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:05 smithi121 bash[36902]: cluster 2022-04-22T18:55:04.457356+0000 mgr.y (mgr.34107) 3076 : cluster [DBG] pgmap v4612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:05 smithi115 bash[45495]: cluster 2022-04-22T18:55:04.457356+0000 mgr.y (mgr.34107) 3076 : cluster [DBG] pgmap v4612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:05 smithi115 bash[46934]: cluster 2022-04-22T18:55:04.457356+0000 mgr.y (mgr.34107) 3076 : cluster [DBG] pgmap v4612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:07 smithi115 bash[45495]: cluster 2022-04-22T18:55:06.458345+0000 mgr.y (mgr.34107) 3077 : cluster [DBG] pgmap v4613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:07 smithi115 bash[46934]: cluster 2022-04-22T18:55:06.458345+0000 mgr.y (mgr.34107) 3077 : cluster [DBG] pgmap v4613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:07 smithi121 bash[36902]: cluster 2022-04-22T18:55:06.458345+0000 mgr.y (mgr.34107) 3077 : cluster [DBG] pgmap v4613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:08 smithi115 bash[45495]: audit 2022-04-22T18:55:08.012291+0000 mon.a (mon.0) 6106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:08 smithi115 bash[46934]: audit 2022-04-22T18:55:08.012291+0000 mon.a (mon.0) 6106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:08 smithi121 bash[36902]: audit 2022-04-22T18:55:08.012291+0000 mon.a (mon.0) 6106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:09 smithi115 bash[46934]: cluster 2022-04-22T18:55:08.459437+0000 mgr.y (mgr.34107) 3078 : cluster [DBG] pgmap v4614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:09 smithi115 bash[46934]: audit 2022-04-22T18:55:09.109633+0000 mon.a (mon.0) 6107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:09 smithi115 bash[45495]: cluster 2022-04-22T18:55:08.459437+0000 mgr.y (mgr.34107) 3078 : cluster [DBG] pgmap v4614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:09 smithi115 bash[45495]: audit 2022-04-22T18:55:09.109633+0000 mon.a (mon.0) 6107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:09 smithi121 bash[36902]: cluster 2022-04-22T18:55:08.459437+0000 mgr.y (mgr.34107) 3078 : cluster [DBG] pgmap v4614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:10.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:09 smithi121 bash[36902]: audit 2022-04-22T18:55:09.109633+0000 mon.a (mon.0) 6107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:11 smithi115 bash[45495]: cluster 2022-04-22T18:55:10.460472+0000 mgr.y (mgr.34107) 3079 : cluster [DBG] pgmap v4615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:11.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:11 smithi115 bash[46934]: cluster 2022-04-22T18:55:10.460472+0000 mgr.y (mgr.34107) 3079 : cluster [DBG] pgmap v4615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:11 smithi121 bash[36902]: cluster 2022-04-22T18:55:10.460472+0000 mgr.y (mgr.34107) 3079 : cluster [DBG] pgmap v4615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:11.949Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:13 smithi121 bash[36902]: cluster 2022-04-22T18:55:12.462005+0000 mgr.y (mgr.34107) 3080 : cluster [DBG] pgmap v4616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:13 smithi121 bash[36902]: audit 2022-04-22T18:55:13.015894+0000 mon.a (mon.0) 6108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:13 smithi115 bash[45495]: cluster 2022-04-22T18:55:12.462005+0000 mgr.y (mgr.34107) 3080 : cluster [DBG] pgmap v4616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:13 smithi115 bash[45495]: audit 2022-04-22T18:55:13.015894+0000 mon.a (mon.0) 6108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:13 smithi115 bash[46934]: cluster 2022-04-22T18:55:12.462005+0000 mgr.y (mgr.34107) 3080 : cluster [DBG] pgmap v4616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:13 smithi115 bash[46934]: audit 2022-04-22T18:55:13.015894+0000 mon.a (mon.0) 6108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:14.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:13.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:14 smithi121 bash[36902]: audit 2022-04-22T18:55:14.113826+0000 mon.a (mon.0) 6109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:14 smithi115 bash[45495]: audit 2022-04-22T18:55:14.113826+0000 mon.a (mon.0) 6109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:14 smithi115 bash[46934]: audit 2022-04-22T18:55:14.113826+0000 mon.a (mon.0) 6109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:15 smithi121 bash[36902]: audit 2022-04-22T18:55:14.252021+0000 mon.a (mon.0) 6110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:55:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:15 smithi121 bash[36902]: audit 2022-04-22T18:55:14.258309+0000 mon.a (mon.0) 6111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:55:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:15 smithi121 bash[36902]: cluster 2022-04-22T18:55:14.463111+0000 mgr.y (mgr.34107) 3081 : cluster [DBG] pgmap v4617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[45495]: audit 2022-04-22T18:55:14.252021+0000 mon.a (mon.0) 6110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:55:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[45495]: audit 2022-04-22T18:55:14.258309+0000 mon.a (mon.0) 6111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:55:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[45495]: cluster 2022-04-22T18:55:14.463111+0000 mgr.y (mgr.34107) 3081 : cluster [DBG] pgmap v4617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[46934]: audit 2022-04-22T18:55:14.252021+0000 mon.a (mon.0) 6110 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:55:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[46934]: audit 2022-04-22T18:55:14.258309+0000 mon.a (mon.0) 6111 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:55:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:15 smithi115 bash[46934]: cluster 2022-04-22T18:55:14.463111+0000 mgr.y (mgr.34107) 3081 : cluster [DBG] pgmap v4617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:17 smithi115 bash[45495]: cluster 2022-04-22T18:55:16.464182+0000 mgr.y (mgr.34107) 3082 : cluster [DBG] pgmap v4618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:17 smithi115 bash[46934]: cluster 2022-04-22T18:55:16.464182+0000 mgr.y (mgr.34107) 3082 : cluster [DBG] pgmap v4618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:17 smithi121 bash[36902]: cluster 2022-04-22T18:55:16.464182+0000 mgr.y (mgr.34107) 3082 : cluster [DBG] pgmap v4618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:18 smithi115 bash[45495]: audit 2022-04-22T18:55:18.019583+0000 mon.a (mon.0) 6112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:18.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:18 smithi115 bash[46934]: audit 2022-04-22T18:55:18.019583+0000 mon.a (mon.0) 6112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:18 smithi121 bash[36902]: audit 2022-04-22T18:55:18.019583+0000 mon.a (mon.0) 6112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:19 smithi115 bash[45495]: cluster 2022-04-22T18:55:18.465319+0000 mgr.y (mgr.34107) 3083 : cluster [DBG] pgmap v4619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:19 smithi115 bash[45495]: audit 2022-04-22T18:55:19.116769+0000 mon.a (mon.0) 6113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:19 smithi115 bash[46934]: cluster 2022-04-22T18:55:18.465319+0000 mgr.y (mgr.34107) 3083 : cluster [DBG] pgmap v4619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:19 smithi115 bash[46934]: audit 2022-04-22T18:55:19.116769+0000 mon.a (mon.0) 6113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:19 smithi121 bash[36902]: cluster 2022-04-22T18:55:18.465319+0000 mgr.y (mgr.34107) 3083 : cluster [DBG] pgmap v4619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:20.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:19 smithi121 bash[36902]: audit 2022-04-22T18:55:19.116769+0000 mon.a (mon.0) 6113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:21.499 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:55:21.851 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:55:21.852 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (78m) 5m ago 92m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:55:21.853 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (91m) 6m ago 91m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:55:21.853 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (80m) 5m ago 93m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:55:21.854 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (79m) 6m ago 93m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:55:21.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (87m) 6m ago 97m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:55:21.857 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (88m) 5m ago 99m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:55:21.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (87m) 5m ago 99m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:55:21.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (86m) 6m ago 97m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:55:21.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (86m) 5m ago 97m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:55:21.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (79m) 5m ago 92m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:55:21.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (79m) 6m ago 92m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:55:21.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (85m) 5m ago 97m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (84m) 5m ago 96m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (84m) 5m ago 96m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (83m) 5m ago 95m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (82m) 6m ago 95m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (82m) 6m ago 94m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (81m) 6m ago 94m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (80m) 6m ago 93m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:55:21.861 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (79m) 6m ago 92m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:55:21.959 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:21 smithi115 bash[45495]: cluster 2022-04-22T18:55:20.466520+0000 mgr.y (mgr.34107) 3084 : cluster [DBG] pgmap v4620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:21.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:21 smithi115 bash[46934]: cluster 2022-04-22T18:55:20.466520+0000 mgr.y (mgr.34107) 3084 : cluster [DBG] pgmap v4620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:21 smithi121 bash[36902]: cluster 2022-04-22T18:55:20.466520+0000 mgr.y (mgr.34107) 3084 : cluster [DBG] pgmap v4620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:22.256 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:22.257 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:55:22.258 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:55:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:21.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:22 smithi121 bash[36902]: audit 2022-04-22T18:55:21.472758+0000 mgr.y (mgr.34107) 3085 : audit [DBG] from='client.38646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:22 smithi121 bash[36902]: audit 2022-04-22T18:55:21.820807+0000 mgr.y (mgr.34107) 3086 : audit [DBG] from='client.38652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:22 smithi121 bash[36902]: audit 2022-04-22T18:55:22.247836+0000 mon.c (mon.1) 167 : audit [DBG] from='client.? 172.21.15.115:0/120691835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:23.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[45495]: audit 2022-04-22T18:55:21.472758+0000 mgr.y (mgr.34107) 3085 : audit [DBG] from='client.38646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[45495]: audit 2022-04-22T18:55:21.820807+0000 mgr.y (mgr.34107) 3086 : audit [DBG] from='client.38652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[45495]: audit 2022-04-22T18:55:22.247836+0000 mon.c (mon.1) 167 : audit [DBG] from='client.? 172.21.15.115:0/120691835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:23.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[46934]: audit 2022-04-22T18:55:21.472758+0000 mgr.y (mgr.34107) 3085 : audit [DBG] from='client.38646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[46934]: audit 2022-04-22T18:55:21.820807+0000 mgr.y (mgr.34107) 3086 : audit [DBG] from='client.38652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:23.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:22 smithi115 bash[46934]: audit 2022-04-22T18:55:22.247836+0000 mon.c (mon.1) 167 : audit [DBG] from='client.? 172.21.15.115:0/120691835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:23.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:23 smithi115 bash[46934]: cluster 2022-04-22T18:55:22.467688+0000 mgr.y (mgr.34107) 3087 : cluster [DBG] pgmap v4621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:23.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:23 smithi115 bash[46934]: audit 2022-04-22T18:55:23.023049+0000 mon.a (mon.0) 6114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:23.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:23 smithi115 bash[45495]: cluster 2022-04-22T18:55:22.467688+0000 mgr.y (mgr.34107) 3087 : cluster [DBG] pgmap v4621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:23.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:23 smithi115 bash[45495]: audit 2022-04-22T18:55:23.023049+0000 mon.a (mon.0) 6114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:23 smithi121 bash[36902]: cluster 2022-04-22T18:55:22.467688+0000 mgr.y (mgr.34107) 3087 : cluster [DBG] pgmap v4621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:23 smithi121 bash[36902]: audit 2022-04-22T18:55:23.023049+0000 mon.a (mon.0) 6114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:23.819Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:24 smithi121 bash[36902]: audit 2022-04-22T18:55:24.120052+0000 mon.a (mon.0) 6115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:24 smithi115 bash[45495]: audit 2022-04-22T18:55:24.120052+0000 mon.a (mon.0) 6115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:24 smithi115 bash[46934]: audit 2022-04-22T18:55:24.120052+0000 mon.a (mon.0) 6115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:25 smithi121 bash[36902]: cluster 2022-04-22T18:55:24.468710+0000 mgr.y (mgr.34107) 3088 : cluster [DBG] pgmap v4622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:25 smithi115 bash[45495]: cluster 2022-04-22T18:55:24.468710+0000 mgr.y (mgr.34107) 3088 : cluster [DBG] pgmap v4622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:25 smithi115 bash[46934]: cluster 2022-04-22T18:55:24.468710+0000 mgr.y (mgr.34107) 3088 : cluster [DBG] pgmap v4622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:27 smithi121 bash[36902]: cluster 2022-04-22T18:55:26.469846+0000 mgr.y (mgr.34107) 3089 : cluster [DBG] pgmap v4623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:27 smithi115 bash[45495]: cluster 2022-04-22T18:55:26.469846+0000 mgr.y (mgr.34107) 3089 : cluster [DBG] pgmap v4623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:27 smithi115 bash[46934]: cluster 2022-04-22T18:55:26.469846+0000 mgr.y (mgr.34107) 3089 : cluster [DBG] pgmap v4623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:28 smithi121 bash[36902]: audit 2022-04-22T18:55:28.026598+0000 mon.a (mon.0) 6116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:28 smithi115 bash[45495]: audit 2022-04-22T18:55:28.026598+0000 mon.a (mon.0) 6116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:28 smithi115 bash[46934]: audit 2022-04-22T18:55:28.026598+0000 mon.a (mon.0) 6116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:29 smithi121 bash[36902]: cluster 2022-04-22T18:55:28.470988+0000 mgr.y (mgr.34107) 3090 : cluster [DBG] pgmap v4624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:29 smithi121 bash[36902]: audit 2022-04-22T18:55:29.123104+0000 mon.a (mon.0) 6117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:29 smithi115 bash[45495]: cluster 2022-04-22T18:55:28.470988+0000 mgr.y (mgr.34107) 3090 : cluster [DBG] pgmap v4624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:30.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:29 smithi115 bash[45495]: audit 2022-04-22T18:55:29.123104+0000 mon.a (mon.0) 6117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:30.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:29 smithi115 bash[46934]: cluster 2022-04-22T18:55:28.470988+0000 mgr.y (mgr.34107) 3090 : cluster [DBG] pgmap v4624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:30.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:29 smithi115 bash[46934]: audit 2022-04-22T18:55:29.123104+0000 mon.a (mon.0) 6117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:31.960 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:31 smithi115 bash[45495]: cluster 2022-04-22T18:55:30.472209+0000 mgr.y (mgr.34107) 3091 : cluster [DBG] pgmap v4625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:31.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:31 smithi115 bash[46934]: cluster 2022-04-22T18:55:30.472209+0000 mgr.y (mgr.34107) 3091 : cluster [DBG] pgmap v4625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:31 smithi121 bash[36902]: cluster 2022-04-22T18:55:30.472209+0000 mgr.y (mgr.34107) 3091 : cluster [DBG] pgmap v4625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:31.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:33 smithi121 bash[36902]: cluster 2022-04-22T18:55:32.473311+0000 mgr.y (mgr.34107) 3092 : cluster [DBG] pgmap v4626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:33 smithi121 bash[36902]: audit 2022-04-22T18:55:33.030022+0000 mon.a (mon.0) 6118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:33 smithi115 bash[45495]: cluster 2022-04-22T18:55:32.473311+0000 mgr.y (mgr.34107) 3092 : cluster [DBG] pgmap v4626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:33 smithi115 bash[45495]: audit 2022-04-22T18:55:33.030022+0000 mon.a (mon.0) 6118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:33 smithi115 bash[46934]: cluster 2022-04-22T18:55:32.473311+0000 mgr.y (mgr.34107) 3092 : cluster [DBG] pgmap v4626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:33 smithi115 bash[46934]: audit 2022-04-22T18:55:33.030022+0000 mon.a (mon.0) 6118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:34.161 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:33.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:34 smithi121 bash[36902]: audit 2022-04-22T18:55:34.126064+0000 mon.a (mon.0) 6119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:34 smithi115 bash[45495]: audit 2022-04-22T18:55:34.126064+0000 mon.a (mon.0) 6119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:34.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:34 smithi115 bash[46934]: audit 2022-04-22T18:55:34.126064+0000 mon.a (mon.0) 6119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:35 smithi121 bash[36902]: cluster 2022-04-22T18:55:34.474402+0000 mgr.y (mgr.34107) 3093 : cluster [DBG] pgmap v4627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:35 smithi115 bash[46934]: cluster 2022-04-22T18:55:34.474402+0000 mgr.y (mgr.34107) 3093 : cluster [DBG] pgmap v4627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:35 smithi115 bash[45495]: cluster 2022-04-22T18:55:34.474402+0000 mgr.y (mgr.34107) 3093 : cluster [DBG] pgmap v4627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:37 smithi115 bash[45495]: cluster 2022-04-22T18:55:36.475561+0000 mgr.y (mgr.34107) 3094 : cluster [DBG] pgmap v4628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:37 smithi115 bash[46934]: cluster 2022-04-22T18:55:36.475561+0000 mgr.y (mgr.34107) 3094 : cluster [DBG] pgmap v4628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:38.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:37 smithi121 bash[36902]: cluster 2022-04-22T18:55:36.475561+0000 mgr.y (mgr.34107) 3094 : cluster [DBG] pgmap v4628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:38 smithi121 bash[36902]: audit 2022-04-22T18:55:38.033516+0000 mon.a (mon.0) 6120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:38 smithi115 bash[45495]: audit 2022-04-22T18:55:38.033516+0000 mon.a (mon.0) 6120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:38 smithi115 bash[46934]: audit 2022-04-22T18:55:38.033516+0000 mon.a (mon.0) 6120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:39 smithi121 bash[36902]: cluster 2022-04-22T18:55:38.476726+0000 mgr.y (mgr.34107) 3095 : cluster [DBG] pgmap v4629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:39 smithi121 bash[36902]: audit 2022-04-22T18:55:39.129351+0000 mon.a (mon.0) 6121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:39 smithi115 bash[46934]: cluster 2022-04-22T18:55:38.476726+0000 mgr.y (mgr.34107) 3095 : cluster [DBG] pgmap v4629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:39 smithi115 bash[46934]: audit 2022-04-22T18:55:39.129351+0000 mon.a (mon.0) 6121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:39 smithi115 bash[45495]: cluster 2022-04-22T18:55:38.476726+0000 mgr.y (mgr.34107) 3095 : cluster [DBG] pgmap v4629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:39 smithi115 bash[45495]: audit 2022-04-22T18:55:39.129351+0000 mon.a (mon.0) 6121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:41.960 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:41 smithi115 bash[45495]: cluster 2022-04-22T18:55:40.477777+0000 mgr.y (mgr.34107) 3096 : cluster [DBG] pgmap v4630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:42.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:41 smithi115 bash[46934]: cluster 2022-04-22T18:55:40.477777+0000 mgr.y (mgr.34107) 3096 : cluster [DBG] pgmap v4630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:42.234 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:41 smithi121 bash[36902]: cluster 2022-04-22T18:55:40.477777+0000 mgr.y (mgr.34107) 3096 : cluster [DBG] pgmap v4630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:41.952Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:43 smithi121 bash[36902]: cluster 2022-04-22T18:55:42.478884+0000 mgr.y (mgr.34107) 3097 : cluster [DBG] pgmap v4631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:43 smithi121 bash[36902]: audit 2022-04-22T18:55:43.037193+0000 mon.a (mon.0) 6122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:43 smithi115 bash[45495]: cluster 2022-04-22T18:55:42.478884+0000 mgr.y (mgr.34107) 3097 : cluster [DBG] pgmap v4631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:43 smithi115 bash[45495]: audit 2022-04-22T18:55:43.037193+0000 mon.a (mon.0) 6122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:43 smithi115 bash[46934]: cluster 2022-04-22T18:55:42.478884+0000 mgr.y (mgr.34107) 3097 : cluster [DBG] pgmap v4631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:43 smithi115 bash[46934]: audit 2022-04-22T18:55:43.037193+0000 mon.a (mon.0) 6122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:44.170 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:43.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:44 smithi121 bash[36902]: audit 2022-04-22T18:55:44.132415+0000 mon.a (mon.0) 6123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:44 smithi115 bash[45495]: audit 2022-04-22T18:55:44.132415+0000 mon.a (mon.0) 6123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:44 smithi115 bash[46934]: audit 2022-04-22T18:55:44.132415+0000 mon.a (mon.0) 6123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:45 smithi121 bash[36902]: cluster 2022-04-22T18:55:44.479990+0000 mgr.y (mgr.34107) 3098 : cluster [DBG] pgmap v4632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:45 smithi115 bash[46934]: cluster 2022-04-22T18:55:44.479990+0000 mgr.y (mgr.34107) 3098 : cluster [DBG] pgmap v4632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:45 smithi115 bash[45495]: cluster 2022-04-22T18:55:44.479990+0000 mgr.y (mgr.34107) 3098 : cluster [DBG] pgmap v4632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:47 smithi121 bash[36902]: cluster 2022-04-22T18:55:46.481081+0000 mgr.y (mgr.34107) 3099 : cluster [DBG] pgmap v4633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:47 smithi115 bash[45495]: cluster 2022-04-22T18:55:46.481081+0000 mgr.y (mgr.34107) 3099 : cluster [DBG] pgmap v4633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:47 smithi115 bash[46934]: cluster 2022-04-22T18:55:46.481081+0000 mgr.y (mgr.34107) 3099 : cluster [DBG] pgmap v4633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:48 smithi121 bash[36902]: audit 2022-04-22T18:55:48.040728+0000 mon.a (mon.0) 6124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:48 smithi115 bash[45495]: audit 2022-04-22T18:55:48.040728+0000 mon.a (mon.0) 6124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:48 smithi115 bash[46934]: audit 2022-04-22T18:55:48.040728+0000 mon.a (mon.0) 6124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:49.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[45495]: cluster 2022-04-22T18:55:48.482252+0000 mgr.y (mgr.34107) 3100 : cluster [DBG] pgmap v4634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:49.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[45495]: audit 2022-04-22T18:55:49.135317+0000 mon.a (mon.0) 6125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:49.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[45495]: audit 2022-04-22T18:55:49.333326+0000 mon.a (mon.0) 6126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:55:49.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[45495]: audit 2022-04-22T18:55:49.336575+0000 mon.a (mon.0) 6127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:55:49.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[45495]: audit 2022-04-22T18:55:49.338301+0000 mon.a (mon.0) 6128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:55:49.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[46934]: cluster 2022-04-22T18:55:48.482252+0000 mgr.y (mgr.34107) 3100 : cluster [DBG] pgmap v4634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:49.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[46934]: audit 2022-04-22T18:55:49.135317+0000 mon.a (mon.0) 6125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:49.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[46934]: audit 2022-04-22T18:55:49.333326+0000 mon.a (mon.0) 6126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:55:49.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[46934]: audit 2022-04-22T18:55:49.336575+0000 mon.a (mon.0) 6127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:55:49.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:49 smithi115 bash[46934]: audit 2022-04-22T18:55:49.338301+0000 mon.a (mon.0) 6128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:49 smithi121 bash[36902]: cluster 2022-04-22T18:55:48.482252+0000 mgr.y (mgr.34107) 3100 : cluster [DBG] pgmap v4634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:49 smithi121 bash[36902]: audit 2022-04-22T18:55:49.135317+0000 mon.a (mon.0) 6125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:49 smithi121 bash[36902]: audit 2022-04-22T18:55:49.333326+0000 mon.a (mon.0) 6126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:49 smithi121 bash[36902]: audit 2022-04-22T18:55:49.336575+0000 mon.a (mon.0) 6127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:49 smithi121 bash[36902]: audit 2022-04-22T18:55:49.338301+0000 mon.a (mon.0) 6128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:55:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:50 smithi121 bash[36902]: audit 2022-04-22T18:55:49.688922+0000 mon.a (mon.0) 6129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:55:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:50 smithi115 bash[45495]: audit 2022-04-22T18:55:49.688922+0000 mon.a (mon.0) 6129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:55:51.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:50 smithi115 bash[46934]: audit 2022-04-22T18:55:49.688922+0000 mon.a (mon.0) 6129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:55:51.961 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:51 smithi115 bash[45495]: cluster 2022-04-22T18:55:50.483435+0000 mgr.y (mgr.34107) 3101 : cluster [DBG] pgmap v4635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:51.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:51 smithi115 bash[46934]: cluster 2022-04-22T18:55:50.483435+0000 mgr.y (mgr.34107) 3101 : cluster [DBG] pgmap v4635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:51 smithi121 bash[36902]: cluster 2022-04-22T18:55:50.483435+0000 mgr.y (mgr.34107) 3101 : cluster [DBG] pgmap v4635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:51.953Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:52.631 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (78m) 6m ago 92m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (92m) 7m ago 92m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (80m) 6m ago 93m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (80m) 7m ago 93m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (88m) 7m ago 97m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:55:52.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (88m) 6m ago 100m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (87m) 6m ago 100m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (86m) 7m ago 98m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (87m) 6m ago 98m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (80m) 6m ago 92m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (79m) 7m ago 92m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (85m) 6m ago 97m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (84m) 6m ago 97m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (84m) 6m ago 96m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (84m) 6m ago 96m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:55:52.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (83m) 7m ago 95m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:55:52.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (82m) 7m ago 95m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:55:52.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (82m) 7m ago 94m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:55:52.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (81m) 7m ago 94m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:55:52.977 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (79m) 7m ago 93m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:55:53.379 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:55:53.379 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:55:53.379 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:55:53.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:53.380 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:55:53.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:55:53.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:53.380 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:55:53.381 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:55:53.382 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:55:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:53 smithi121 bash[36902]: cluster 2022-04-22T18:55:52.484606+0000 mgr.y (mgr.34107) 3102 : cluster [DBG] pgmap v4636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:53.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:53 smithi121 bash[36902]: audit 2022-04-22T18:55:52.606602+0000 mgr.y (mgr.34107) 3103 : audit [DBG] from='client.38664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:53 smithi121 bash[36902]: audit 2022-04-22T18:55:52.945550+0000 mgr.y (mgr.34107) 3104 : audit [DBG] from='client.48604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.521 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:53 smithi121 bash[36902]: audit 2022-04-22T18:55:53.044676+0000 mon.a (mon.0) 6130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[45495]: cluster 2022-04-22T18:55:52.484606+0000 mgr.y (mgr.34107) 3102 : cluster [DBG] pgmap v4636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[45495]: audit 2022-04-22T18:55:52.606602+0000 mgr.y (mgr.34107) 3103 : audit [DBG] from='client.38664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[45495]: audit 2022-04-22T18:55:52.945550+0000 mgr.y (mgr.34107) 3104 : audit [DBG] from='client.48604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[45495]: audit 2022-04-22T18:55:53.044676+0000 mon.a (mon.0) 6130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[46934]: cluster 2022-04-22T18:55:52.484606+0000 mgr.y (mgr.34107) 3102 : cluster [DBG] pgmap v4636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[46934]: audit 2022-04-22T18:55:52.606602+0000 mgr.y (mgr.34107) 3103 : audit [DBG] from='client.38664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[46934]: audit 2022-04-22T18:55:52.945550+0000 mgr.y (mgr.34107) 3104 : audit [DBG] from='client.48604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:55:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[46934]: audit 2022-04-22T18:55:53.044676+0000 mon.a (mon.0) 6130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:54.168 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:55:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:55:53.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:55:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:54 smithi121 bash[36902]: audit 2022-04-22T18:55:53.370264+0000 mon.c (mon.1) 168 : audit [DBG] from='client.? 172.21.15.115:0/899383614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:54 smithi121 bash[36902]: audit 2022-04-22T18:55:54.138468+0000 mon.a (mon.0) 6131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:54 smithi115 bash[45495]: audit 2022-04-22T18:55:53.370264+0000 mon.c (mon.1) 168 : audit [DBG] from='client.? 172.21.15.115:0/899383614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:54 smithi115 bash[45495]: audit 2022-04-22T18:55:54.138468+0000 mon.a (mon.0) 6131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:54 smithi115 bash[46934]: audit 2022-04-22T18:55:53.370264+0000 mon.c (mon.1) 168 : audit [DBG] from='client.? 172.21.15.115:0/899383614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:55:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:54 smithi115 bash[46934]: audit 2022-04-22T18:55:54.138468+0000 mon.a (mon.0) 6131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:55:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:55 smithi121 bash[36902]: cluster 2022-04-22T18:55:54.485713+0000 mgr.y (mgr.34107) 3105 : cluster [DBG] pgmap v4637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:55 smithi115 bash[45495]: cluster 2022-04-22T18:55:54.485713+0000 mgr.y (mgr.34107) 3105 : cluster [DBG] pgmap v4637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:55 smithi115 bash[46934]: cluster 2022-04-22T18:55:54.485713+0000 mgr.y (mgr.34107) 3105 : cluster [DBG] pgmap v4637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:57 smithi121 bash[36902]: cluster 2022-04-22T18:55:56.486809+0000 mgr.y (mgr.34107) 3106 : cluster [DBG] pgmap v4638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:57 smithi115 bash[45495]: cluster 2022-04-22T18:55:56.486809+0000 mgr.y (mgr.34107) 3106 : cluster [DBG] pgmap v4638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:57 smithi115 bash[46934]: cluster 2022-04-22T18:55:56.486809+0000 mgr.y (mgr.34107) 3106 : cluster [DBG] pgmap v4638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:55:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:58 smithi121 bash[36902]: audit 2022-04-22T18:55:58.048078+0000 mon.a (mon.0) 6132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:58 smithi115 bash[45495]: audit 2022-04-22T18:55:58.048078+0000 mon.a (mon.0) 6132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:55:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:58 smithi115 bash[46934]: audit 2022-04-22T18:55:58.048078+0000 mon.a (mon.0) 6132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:59 smithi121 bash[36902]: cluster 2022-04-22T18:55:58.488098+0000 mgr.y (mgr.34107) 3107 : cluster [DBG] pgmap v4639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:55:59 smithi121 bash[36902]: audit 2022-04-22T18:55:59.141686+0000 mon.a (mon.0) 6133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:59 smithi115 bash[45495]: cluster 2022-04-22T18:55:58.488098+0000 mgr.y (mgr.34107) 3107 : cluster [DBG] pgmap v4639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:55:59 smithi115 bash[45495]: audit 2022-04-22T18:55:59.141686+0000 mon.a (mon.0) 6133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:59 smithi115 bash[46934]: cluster 2022-04-22T18:55:58.488098+0000 mgr.y (mgr.34107) 3107 : cluster [DBG] pgmap v4639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:55:59 smithi115 bash[46934]: audit 2022-04-22T18:55:59.141686+0000 mon.a (mon.0) 6133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:01.961 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:01 smithi115 bash[45495]: cluster 2022-04-22T18:56:00.489177+0000 mgr.y (mgr.34107) 3108 : cluster [DBG] pgmap v4640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:01.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:01 smithi115 bash[46934]: cluster 2022-04-22T18:56:00.489177+0000 mgr.y (mgr.34107) 3108 : cluster [DBG] pgmap v4640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:01 smithi121 bash[36902]: cluster 2022-04-22T18:56:00.489177+0000 mgr.y (mgr.34107) 3108 : cluster [DBG] pgmap v4640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:01.953Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:03 smithi121 bash[36902]: cluster 2022-04-22T18:56:02.490229+0000 mgr.y (mgr.34107) 3109 : cluster [DBG] pgmap v4641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:03 smithi121 bash[36902]: audit 2022-04-22T18:56:03.052024+0000 mon.a (mon.0) 6134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:03 smithi115 bash[45495]: cluster 2022-04-22T18:56:02.490229+0000 mgr.y (mgr.34107) 3109 : cluster [DBG] pgmap v4641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:03 smithi115 bash[45495]: audit 2022-04-22T18:56:03.052024+0000 mon.a (mon.0) 6134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:03 smithi115 bash[46934]: cluster 2022-04-22T18:56:02.490229+0000 mgr.y (mgr.34107) 3109 : cluster [DBG] pgmap v4641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:03 smithi115 bash[46934]: audit 2022-04-22T18:56:03.052024+0000 mon.a (mon.0) 6134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:04.172 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:03.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:04 smithi121 bash[36902]: audit 2022-04-22T18:56:04.144741+0000 mon.a (mon.0) 6135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:04.611 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:04 smithi115 bash[45495]: audit 2022-04-22T18:56:04.144741+0000 mon.a (mon.0) 6135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:04 smithi115 bash[46934]: audit 2022-04-22T18:56:04.144741+0000 mon.a (mon.0) 6135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:05 smithi121 bash[36902]: cluster 2022-04-22T18:56:04.491279+0000 mgr.y (mgr.34107) 3110 : cluster [DBG] pgmap v4642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:05 smithi115 bash[45495]: cluster 2022-04-22T18:56:04.491279+0000 mgr.y (mgr.34107) 3110 : cluster [DBG] pgmap v4642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:05 smithi115 bash[46934]: cluster 2022-04-22T18:56:04.491279+0000 mgr.y (mgr.34107) 3110 : cluster [DBG] pgmap v4642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:07 smithi121 bash[36902]: cluster 2022-04-22T18:56:06.492387+0000 mgr.y (mgr.34107) 3111 : cluster [DBG] pgmap v4643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:07 smithi115 bash[46934]: cluster 2022-04-22T18:56:06.492387+0000 mgr.y (mgr.34107) 3111 : cluster [DBG] pgmap v4643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:07 smithi115 bash[45495]: cluster 2022-04-22T18:56:06.492387+0000 mgr.y (mgr.34107) 3111 : cluster [DBG] pgmap v4643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:08 smithi121 bash[36902]: audit 2022-04-22T18:56:08.055771+0000 mon.a (mon.0) 6136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:08 smithi115 bash[45495]: audit 2022-04-22T18:56:08.055771+0000 mon.a (mon.0) 6136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:08 smithi115 bash[46934]: audit 2022-04-22T18:56:08.055771+0000 mon.a (mon.0) 6136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:09 smithi121 bash[36902]: cluster 2022-04-22T18:56:08.493547+0000 mgr.y (mgr.34107) 3112 : cluster [DBG] pgmap v4644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:09 smithi121 bash[36902]: audit 2022-04-22T18:56:09.147574+0000 mon.a (mon.0) 6137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:09 smithi115 bash[45495]: cluster 2022-04-22T18:56:08.493547+0000 mgr.y (mgr.34107) 3112 : cluster [DBG] pgmap v4644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:09 smithi115 bash[45495]: audit 2022-04-22T18:56:09.147574+0000 mon.a (mon.0) 6137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:10.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:09 smithi115 bash[46934]: cluster 2022-04-22T18:56:08.493547+0000 mgr.y (mgr.34107) 3112 : cluster [DBG] pgmap v4644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:10.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:09 smithi115 bash[46934]: audit 2022-04-22T18:56:09.147574+0000 mon.a (mon.0) 6137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:11.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:11 smithi115 bash[46934]: cluster 2022-04-22T18:56:10.494598+0000 mgr.y (mgr.34107) 3113 : cluster [DBG] pgmap v4645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:12.706 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:11 smithi115 bash[45495]: cluster 2022-04-22T18:56:10.494598+0000 mgr.y (mgr.34107) 3113 : cluster [DBG] pgmap v4645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:12.707 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:11.954Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:12.708 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:11 smithi121 bash[36902]: cluster 2022-04-22T18:56:10.494598+0000 mgr.y (mgr.34107) 3113 : cluster [DBG] pgmap v4645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:13 smithi121 bash[36902]: cluster 2022-04-22T18:56:12.495683+0000 mgr.y (mgr.34107) 3114 : cluster [DBG] pgmap v4646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:13 smithi121 bash[36902]: audit 2022-04-22T18:56:13.059445+0000 mon.a (mon.0) 6138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:13 smithi115 bash[45495]: cluster 2022-04-22T18:56:12.495683+0000 mgr.y (mgr.34107) 3114 : cluster [DBG] pgmap v4646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:13 smithi115 bash[45495]: audit 2022-04-22T18:56:13.059445+0000 mon.a (mon.0) 6138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:13 smithi115 bash[46934]: cluster 2022-04-22T18:56:12.495683+0000 mgr.y (mgr.34107) 3114 : cluster [DBG] pgmap v4646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:13 smithi115 bash[46934]: audit 2022-04-22T18:56:13.059445+0000 mon.a (mon.0) 6138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:14.189 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:13.824Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:14 smithi121 bash[36902]: audit 2022-04-22T18:56:14.150624+0000 mon.a (mon.0) 6139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:14 smithi115 bash[45495]: audit 2022-04-22T18:56:14.150624+0000 mon.a (mon.0) 6139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:14 smithi115 bash[46934]: audit 2022-04-22T18:56:14.150624+0000 mon.a (mon.0) 6139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:15 smithi121 bash[36902]: audit 2022-04-22T18:56:14.264338+0000 mon.a (mon.0) 6140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:56:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:15 smithi121 bash[36902]: audit 2022-04-22T18:56:14.272400+0000 mon.a (mon.0) 6141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:56:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:15 smithi121 bash[36902]: cluster 2022-04-22T18:56:14.496777+0000 mgr.y (mgr.34107) 3115 : cluster [DBG] pgmap v4647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[45495]: audit 2022-04-22T18:56:14.264338+0000 mon.a (mon.0) 6140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:56:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[45495]: audit 2022-04-22T18:56:14.272400+0000 mon.a (mon.0) 6141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:56:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[45495]: cluster 2022-04-22T18:56:14.496777+0000 mgr.y (mgr.34107) 3115 : cluster [DBG] pgmap v4647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[46934]: audit 2022-04-22T18:56:14.264338+0000 mon.a (mon.0) 6140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:56:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[46934]: audit 2022-04-22T18:56:14.272400+0000 mon.a (mon.0) 6141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:56:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:15 smithi115 bash[46934]: cluster 2022-04-22T18:56:14.496777+0000 mgr.y (mgr.34107) 3115 : cluster [DBG] pgmap v4647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:17 smithi121 bash[36902]: cluster 2022-04-22T18:56:16.497879+0000 mgr.y (mgr.34107) 3116 : cluster [DBG] pgmap v4648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:17 smithi115 bash[45495]: cluster 2022-04-22T18:56:16.497879+0000 mgr.y (mgr.34107) 3116 : cluster [DBG] pgmap v4648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:17 smithi115 bash[46934]: cluster 2022-04-22T18:56:16.497879+0000 mgr.y (mgr.34107) 3116 : cluster [DBG] pgmap v4648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:18 smithi121 bash[36902]: audit 2022-04-22T18:56:18.063009+0000 mon.a (mon.0) 6142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:18 smithi115 bash[45495]: audit 2022-04-22T18:56:18.063009+0000 mon.a (mon.0) 6142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:18 smithi115 bash[46934]: audit 2022-04-22T18:56:18.063009+0000 mon.a (mon.0) 6142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:19 smithi121 bash[36902]: cluster 2022-04-22T18:56:18.498972+0000 mgr.y (mgr.34107) 3117 : cluster [DBG] pgmap v4649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:19 smithi121 bash[36902]: audit 2022-04-22T18:56:19.153867+0000 mon.a (mon.0) 6143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:19 smithi115 bash[46934]: cluster 2022-04-22T18:56:18.498972+0000 mgr.y (mgr.34107) 3117 : cluster [DBG] pgmap v4649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:19 smithi115 bash[46934]: audit 2022-04-22T18:56:19.153867+0000 mon.a (mon.0) 6143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:19 smithi115 bash[45495]: cluster 2022-04-22T18:56:18.498972+0000 mgr.y (mgr.34107) 3117 : cluster [DBG] pgmap v4649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:19 smithi115 bash[45495]: audit 2022-04-22T18:56:19.153867+0000 mon.a (mon.0) 6143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:21.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:21 smithi115 bash[45495]: cluster 2022-04-22T18:56:20.500213+0000 mgr.y (mgr.34107) 3118 : cluster [DBG] pgmap v4650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:21.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:21 smithi115 bash[46934]: cluster 2022-04-22T18:56:20.500213+0000 mgr.y (mgr.34107) 3118 : cluster [DBG] pgmap v4650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:21 smithi121 bash[36902]: cluster 2022-04-22T18:56:20.500213+0000 mgr.y (mgr.34107) 3118 : cluster [DBG] pgmap v4650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:22.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:21.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:23 smithi121 bash[36902]: cluster 2022-04-22T18:56:22.501361+0000 mgr.y (mgr.34107) 3119 : cluster [DBG] pgmap v4651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:23 smithi121 bash[36902]: audit 2022-04-22T18:56:23.066752+0000 mon.a (mon.0) 6144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:23 smithi115 bash[45495]: cluster 2022-04-22T18:56:22.501361+0000 mgr.y (mgr.34107) 3119 : cluster [DBG] pgmap v4651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:23 smithi115 bash[45495]: audit 2022-04-22T18:56:23.066752+0000 mon.a (mon.0) 6144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:23 smithi115 bash[46934]: cluster 2022-04-22T18:56:22.501361+0000 mgr.y (mgr.34107) 3119 : cluster [DBG] pgmap v4651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:23 smithi115 bash[46934]: audit 2022-04-22T18:56:23.066752+0000 mon.a (mon.0) 6144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:23.756 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:56:24.101 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:56:24.101 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (79m) 6m ago 93m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (92m) 7m ago 92m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (81m) 6m ago 94m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (80m) 7m ago 94m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (88m) 7m ago 98m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (89m) 6m ago 100m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (88m) 6m ago 100m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:56:24.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (87m) 7m ago 98m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:56:24.103 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (87m) 6m ago 98m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:56:24.103 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (80m) 6m ago 93m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:56:24.103 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (80m) 7m ago 93m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:56:24.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (86m) 6m ago 98m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:56:24.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (85m) 6m ago 97m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (85m) 6m ago 97m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (84m) 6m ago 96m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (83m) 7m ago 96m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (83m) 7m ago 95m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (82m) 7m ago 95m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (81m) 7m ago 94m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:56:24.104 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (80m) 7m ago 93m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:56:24.182 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:23.824Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:56:24.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:56:24.504 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:56:24.505 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:56:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:24 smithi121 bash[36902]: audit 2022-04-22T18:56:23.732075+0000 mgr.y (mgr.34107) 3120 : audit [DBG] from='client.38682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:24 smithi121 bash[36902]: audit 2022-04-22T18:56:24.073217+0000 mgr.y (mgr.34107) 3121 : audit [DBG] from='client.48622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:24 smithi121 bash[36902]: audit 2022-04-22T18:56:24.156999+0000 mon.a (mon.0) 6145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[45495]: audit 2022-04-22T18:56:23.732075+0000 mgr.y (mgr.34107) 3120 : audit [DBG] from='client.38682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[45495]: audit 2022-04-22T18:56:24.073217+0000 mgr.y (mgr.34107) 3121 : audit [DBG] from='client.48622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[45495]: audit 2022-04-22T18:56:24.156999+0000 mon.a (mon.0) 6145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[46934]: audit 2022-04-22T18:56:23.732075+0000 mgr.y (mgr.34107) 3120 : audit [DBG] from='client.38682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[46934]: audit 2022-04-22T18:56:24.073217+0000 mgr.y (mgr.34107) 3121 : audit [DBG] from='client.48622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:24 smithi115 bash[46934]: audit 2022-04-22T18:56:24.156999+0000 mon.a (mon.0) 6145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:25 smithi121 bash[36902]: audit 2022-04-22T18:56:24.493694+0000 mon.a (mon.0) 6146 : audit [DBG] from='client.? 172.21.15.115:0/595987662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:25 smithi121 bash[36902]: cluster 2022-04-22T18:56:24.502381+0000 mgr.y (mgr.34107) 3122 : cluster [DBG] pgmap v4652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:25 smithi115 bash[45495]: audit 2022-04-22T18:56:24.493694+0000 mon.a (mon.0) 6146 : audit [DBG] from='client.? 172.21.15.115:0/595987662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:25 smithi115 bash[45495]: cluster 2022-04-22T18:56:24.502381+0000 mgr.y (mgr.34107) 3122 : cluster [DBG] pgmap v4652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:25 smithi115 bash[46934]: audit 2022-04-22T18:56:24.493694+0000 mon.a (mon.0) 6146 : audit [DBG] from='client.? 172.21.15.115:0/595987662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:25 smithi115 bash[46934]: cluster 2022-04-22T18:56:24.502381+0000 mgr.y (mgr.34107) 3122 : cluster [DBG] pgmap v4652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:27 smithi121 bash[36902]: cluster 2022-04-22T18:56:26.503441+0000 mgr.y (mgr.34107) 3123 : cluster [DBG] pgmap v4653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:27 smithi115 bash[45495]: cluster 2022-04-22T18:56:26.503441+0000 mgr.y (mgr.34107) 3123 : cluster [DBG] pgmap v4653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:27 smithi115 bash[46934]: cluster 2022-04-22T18:56:26.503441+0000 mgr.y (mgr.34107) 3123 : cluster [DBG] pgmap v4653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:28 smithi121 bash[36902]: audit 2022-04-22T18:56:28.070488+0000 mon.a (mon.0) 6147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:28 smithi115 bash[45495]: audit 2022-04-22T18:56:28.070488+0000 mon.a (mon.0) 6147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:28 smithi115 bash[46934]: audit 2022-04-22T18:56:28.070488+0000 mon.a (mon.0) 6147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:29 smithi121 bash[36902]: cluster 2022-04-22T18:56:28.504565+0000 mgr.y (mgr.34107) 3124 : cluster [DBG] pgmap v4654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:29 smithi121 bash[36902]: audit 2022-04-22T18:56:29.159675+0000 mon.a (mon.0) 6148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:29 smithi115 bash[45495]: cluster 2022-04-22T18:56:28.504565+0000 mgr.y (mgr.34107) 3124 : cluster [DBG] pgmap v4654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:30.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:29 smithi115 bash[45495]: audit 2022-04-22T18:56:29.159675+0000 mon.a (mon.0) 6148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:30.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:29 smithi115 bash[46934]: cluster 2022-04-22T18:56:28.504565+0000 mgr.y (mgr.34107) 3124 : cluster [DBG] pgmap v4654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:30.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:29 smithi115 bash[46934]: audit 2022-04-22T18:56:29.159675+0000 mon.a (mon.0) 6148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:31.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:31 smithi115 bash[45495]: cluster 2022-04-22T18:56:30.505614+0000 mgr.y (mgr.34107) 3125 : cluster [DBG] pgmap v4655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:31.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:31 smithi115 bash[46934]: cluster 2022-04-22T18:56:30.505614+0000 mgr.y (mgr.34107) 3125 : cluster [DBG] pgmap v4655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:31 smithi121 bash[36902]: cluster 2022-04-22T18:56:30.505614+0000 mgr.y (mgr.34107) 3125 : cluster [DBG] pgmap v4655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:31.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:33 smithi121 bash[36902]: cluster 2022-04-22T18:56:32.506579+0000 mgr.y (mgr.34107) 3126 : cluster [DBG] pgmap v4656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:33 smithi121 bash[36902]: audit 2022-04-22T18:56:33.073871+0000 mon.a (mon.0) 6149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:33 smithi115 bash[45495]: cluster 2022-04-22T18:56:32.506579+0000 mgr.y (mgr.34107) 3126 : cluster [DBG] pgmap v4656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:33 smithi115 bash[45495]: audit 2022-04-22T18:56:33.073871+0000 mon.a (mon.0) 6149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:33 smithi115 bash[46934]: cluster 2022-04-22T18:56:32.506579+0000 mgr.y (mgr.34107) 3126 : cluster [DBG] pgmap v4656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:33 smithi115 bash[46934]: audit 2022-04-22T18:56:33.073871+0000 mon.a (mon.0) 6149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:34.183 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:33.823Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:34 smithi121 bash[36902]: audit 2022-04-22T18:56:34.162416+0000 mon.a (mon.0) 6150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:34 smithi115 bash[45495]: audit 2022-04-22T18:56:34.162416+0000 mon.a (mon.0) 6150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:34 smithi115 bash[46934]: audit 2022-04-22T18:56:34.162416+0000 mon.a (mon.0) 6150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:35 smithi121 bash[36902]: cluster 2022-04-22T18:56:34.507740+0000 mgr.y (mgr.34107) 3127 : cluster [DBG] pgmap v4657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:35 smithi115 bash[45495]: cluster 2022-04-22T18:56:34.507740+0000 mgr.y (mgr.34107) 3127 : cluster [DBG] pgmap v4657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:35 smithi115 bash[46934]: cluster 2022-04-22T18:56:34.507740+0000 mgr.y (mgr.34107) 3127 : cluster [DBG] pgmap v4657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:37 smithi121 bash[36902]: cluster 2022-04-22T18:56:36.508818+0000 mgr.y (mgr.34107) 3128 : cluster [DBG] pgmap v4658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:37 smithi115 bash[45495]: cluster 2022-04-22T18:56:36.508818+0000 mgr.y (mgr.34107) 3128 : cluster [DBG] pgmap v4658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:37 smithi115 bash[46934]: cluster 2022-04-22T18:56:36.508818+0000 mgr.y (mgr.34107) 3128 : cluster [DBG] pgmap v4658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:38 smithi121 bash[36902]: audit 2022-04-22T18:56:38.077216+0000 mon.a (mon.0) 6151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:38 smithi115 bash[45495]: audit 2022-04-22T18:56:38.077216+0000 mon.a (mon.0) 6151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:38 smithi115 bash[46934]: audit 2022-04-22T18:56:38.077216+0000 mon.a (mon.0) 6151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:39 smithi121 bash[36902]: cluster 2022-04-22T18:56:38.509972+0000 mgr.y (mgr.34107) 3129 : cluster [DBG] pgmap v4659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:39 smithi121 bash[36902]: audit 2022-04-22T18:56:39.165599+0000 mon.a (mon.0) 6152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:39 smithi115 bash[46934]: cluster 2022-04-22T18:56:38.509972+0000 mgr.y (mgr.34107) 3129 : cluster [DBG] pgmap v4659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:39 smithi115 bash[46934]: audit 2022-04-22T18:56:39.165599+0000 mon.a (mon.0) 6152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:39 smithi115 bash[45495]: cluster 2022-04-22T18:56:38.509972+0000 mgr.y (mgr.34107) 3129 : cluster [DBG] pgmap v4659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:39 smithi115 bash[45495]: audit 2022-04-22T18:56:39.165599+0000 mon.a (mon.0) 6152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:41.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:41 smithi115 bash[45495]: cluster 2022-04-22T18:56:40.511014+0000 mgr.y (mgr.34107) 3130 : cluster [DBG] pgmap v4660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:41.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:41 smithi115 bash[46934]: cluster 2022-04-22T18:56:40.511014+0000 mgr.y (mgr.34107) 3130 : cluster [DBG] pgmap v4660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:41 smithi121 bash[36902]: cluster 2022-04-22T18:56:40.511014+0000 mgr.y (mgr.34107) 3130 : cluster [DBG] pgmap v4660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:41.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:43 smithi121 bash[36902]: cluster 2022-04-22T18:56:42.512147+0000 mgr.y (mgr.34107) 3131 : cluster [DBG] pgmap v4661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:43 smithi121 bash[36902]: audit 2022-04-22T18:56:43.080729+0000 mon.a (mon.0) 6153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:43 smithi115 bash[45495]: cluster 2022-04-22T18:56:42.512147+0000 mgr.y (mgr.34107) 3131 : cluster [DBG] pgmap v4661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:43 smithi115 bash[45495]: audit 2022-04-22T18:56:43.080729+0000 mon.a (mon.0) 6153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:43 smithi115 bash[46934]: cluster 2022-04-22T18:56:42.512147+0000 mgr.y (mgr.34107) 3131 : cluster [DBG] pgmap v4661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:43 smithi115 bash[46934]: audit 2022-04-22T18:56:43.080729+0000 mon.a (mon.0) 6153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:44.190 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:43.824Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:44 smithi121 bash[36902]: audit 2022-04-22T18:56:44.168604+0000 mon.a (mon.0) 6154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:44 smithi115 bash[46934]: audit 2022-04-22T18:56:44.168604+0000 mon.a (mon.0) 6154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:44 smithi115 bash[45495]: audit 2022-04-22T18:56:44.168604+0000 mon.a (mon.0) 6154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:45.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:45 smithi121 bash[36902]: cluster 2022-04-22T18:56:44.513266+0000 mgr.y (mgr.34107) 3132 : cluster [DBG] pgmap v4662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:45 smithi115 bash[45495]: cluster 2022-04-22T18:56:44.513266+0000 mgr.y (mgr.34107) 3132 : cluster [DBG] pgmap v4662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:45 smithi115 bash[46934]: cluster 2022-04-22T18:56:44.513266+0000 mgr.y (mgr.34107) 3132 : cluster [DBG] pgmap v4662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:48.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:47 smithi121 bash[36902]: cluster 2022-04-22T18:56:46.514388+0000 mgr.y (mgr.34107) 3133 : cluster [DBG] pgmap v4663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:47 smithi115 bash[45495]: cluster 2022-04-22T18:56:46.514388+0000 mgr.y (mgr.34107) 3133 : cluster [DBG] pgmap v4663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:48.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:47 smithi115 bash[46934]: cluster 2022-04-22T18:56:46.514388+0000 mgr.y (mgr.34107) 3133 : cluster [DBG] pgmap v4663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:48 smithi121 bash[36902]: audit 2022-04-22T18:56:48.084351+0000 mon.a (mon.0) 6155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:48 smithi115 bash[45495]: audit 2022-04-22T18:56:48.084351+0000 mon.a (mon.0) 6155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:48 smithi115 bash[46934]: audit 2022-04-22T18:56:48.084351+0000 mon.a (mon.0) 6155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:50.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:49 smithi121 bash[36902]: cluster 2022-04-22T18:56:48.515556+0000 mgr.y (mgr.34107) 3134 : cluster [DBG] pgmap v4664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:50.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:49 smithi121 bash[36902]: audit 2022-04-22T18:56:49.171581+0000 mon.a (mon.0) 6156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:50.024 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:49 smithi115 bash[45495]: cluster 2022-04-22T18:56:48.515556+0000 mgr.y (mgr.34107) 3134 : cluster [DBG] pgmap v4664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:50.024 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:49 smithi115 bash[45495]: audit 2022-04-22T18:56:49.171581+0000 mon.a (mon.0) 6156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:50.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:49 smithi115 bash[46934]: cluster 2022-04-22T18:56:48.515556+0000 mgr.y (mgr.34107) 3134 : cluster [DBG] pgmap v4664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:50.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:49 smithi115 bash[46934]: audit 2022-04-22T18:56:49.171581+0000 mon.a (mon.0) 6156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:50 smithi121 bash[36902]: audit 2022-04-22T18:56:49.697196+0000 mon.a (mon.0) 6157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:50 smithi121 bash[36902]: audit 2022-04-22T18:56:49.700084+0000 mon.a (mon.0) 6158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:50 smithi121 bash[36902]: audit 2022-04-22T18:56:49.701789+0000 mon.a (mon.0) 6159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:50 smithi121 bash[36902]: audit 2022-04-22T18:56:50.063579+0000 mon.a (mon.0) 6160 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:56:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[45495]: audit 2022-04-22T18:56:49.697196+0000 mon.a (mon.0) 6157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[45495]: audit 2022-04-22T18:56:49.700084+0000 mon.a (mon.0) 6158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[45495]: audit 2022-04-22T18:56:49.701789+0000 mon.a (mon.0) 6159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[45495]: audit 2022-04-22T18:56:50.063579+0000 mon.a (mon.0) 6160 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[46934]: audit 2022-04-22T18:56:49.697196+0000 mon.a (mon.0) 6157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[46934]: audit 2022-04-22T18:56:49.700084+0000 mon.a (mon.0) 6158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[46934]: audit 2022-04-22T18:56:49.701789+0000 mon.a (mon.0) 6159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:56:51.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:50 smithi115 bash[46934]: audit 2022-04-22T18:56:50.063579+0000 mon.a (mon.0) 6160 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:56:52.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:52 smithi115 bash[45495]: cluster 2022-04-22T18:56:50.516610+0000 mgr.y (mgr.34107) 3135 : cluster [DBG] pgmap v4665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:51.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:52 smithi115 bash[46934]: cluster 2022-04-22T18:56:50.516610+0000 mgr.y (mgr.34107) 3135 : cluster [DBG] pgmap v4665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:52 smithi121 bash[36902]: cluster 2022-04-22T18:56:50.516610+0000 mgr.y (mgr.34107) 3135 : cluster [DBG] pgmap v4665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:53 smithi121 bash[36902]: cluster 2022-04-22T18:56:52.517798+0000 mgr.y (mgr.34107) 3136 : cluster [DBG] pgmap v4666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:53 smithi121 bash[36902]: audit 2022-04-22T18:56:53.088403+0000 mon.a (mon.0) 6161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:53 smithi115 bash[46934]: cluster 2022-04-22T18:56:52.517798+0000 mgr.y (mgr.34107) 3136 : cluster [DBG] pgmap v4666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:53 smithi115 bash[46934]: audit 2022-04-22T18:56:53.088403+0000 mon.a (mon.0) 6161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:53 smithi115 bash[45495]: cluster 2022-04-22T18:56:52.517798+0000 mgr.y (mgr.34107) 3136 : cluster [DBG] pgmap v4666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:53 smithi115 bash[45495]: audit 2022-04-22T18:56:53.088403+0000 mon.a (mon.0) 6161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:54.195 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:56:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:56:53.825Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:56:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:54 smithi121 bash[36902]: audit 2022-04-22T18:56:54.174594+0000 mon.a (mon.0) 6162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:54 smithi115 bash[45495]: audit 2022-04-22T18:56:54.174594+0000 mon.a (mon.0) 6162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:54 smithi115 bash[46934]: audit 2022-04-22T18:56:54.174594+0000 mon.a (mon.0) 6162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:54.885 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (79m) 7m ago 93m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (93m) 8m ago 93m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (81m) 7m ago 94m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (81m) 8m ago 94m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (89m) 8m ago 99m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:56:55.220 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (89m) 7m ago 101m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (88m) 7m ago 101m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (87m) 8m ago 99m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (88m) 7m ago 99m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (81m) 7m ago 94m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (81m) 8m ago 94m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (86m) 7m ago 98m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (86m) 7m ago 98m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (85m) 7m ago 97m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (85m) 7m ago 97m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (84m) 8m ago 96m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:56:55.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (83m) 8m ago 96m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:56:55.222 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (83m) 8m ago 95m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:56:55.222 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (82m) 8m ago 95m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:56:55.222 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (80m) 8m ago 94m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:56:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:55 smithi121 bash[36902]: cluster 2022-04-22T18:56:54.518917+0000 mgr.y (mgr.34107) 3137 : cluster [DBG] pgmap v4667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:55 smithi121 bash[36902]: audit 2022-04-22T18:56:54.857946+0000 mgr.y (mgr.34107) 3138 : audit [DBG] from='client.48634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:55 smithi115 bash[46934]: cluster 2022-04-22T18:56:54.518917+0000 mgr.y (mgr.34107) 3137 : cluster [DBG] pgmap v4667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:55 smithi115 bash[46934]: audit 2022-04-22T18:56:54.857946+0000 mgr.y (mgr.34107) 3138 : audit [DBG] from='client.48634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:55 smithi115 bash[45495]: cluster 2022-04-22T18:56:54.518917+0000 mgr.y (mgr.34107) 3137 : cluster [DBG] pgmap v4667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:55 smithi115 bash[45495]: audit 2022-04-22T18:56:54.857946+0000 mgr.y (mgr.34107) 3138 : audit [DBG] from='client.48634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:55.621 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:56:55.622 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:56:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:56 smithi121 bash[36902]: audit 2022-04-22T18:56:55.193492+0000 mgr.y (mgr.34107) 3139 : audit [DBG] from='client.38706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:56 smithi121 bash[36902]: audit 2022-04-22T18:56:55.612106+0000 mon.c (mon.1) 169 : audit [DBG] from='client.? 172.21.15.115:0/1727901936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:56 smithi115 bash[45495]: audit 2022-04-22T18:56:55.193492+0000 mgr.y (mgr.34107) 3139 : audit [DBG] from='client.38706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:56 smithi115 bash[45495]: audit 2022-04-22T18:56:55.612106+0000 mon.c (mon.1) 169 : audit [DBG] from='client.? 172.21.15.115:0/1727901936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:56 smithi115 bash[46934]: audit 2022-04-22T18:56:55.193492+0000 mgr.y (mgr.34107) 3139 : audit [DBG] from='client.38706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:56:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:56 smithi115 bash[46934]: audit 2022-04-22T18:56:55.612106+0000 mon.c (mon.1) 169 : audit [DBG] from='client.? 172.21.15.115:0/1727901936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:56:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:57 smithi121 bash[36902]: cluster 2022-04-22T18:56:56.520151+0000 mgr.y (mgr.34107) 3140 : cluster [DBG] pgmap v4668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:57 smithi115 bash[45495]: cluster 2022-04-22T18:56:56.520151+0000 mgr.y (mgr.34107) 3140 : cluster [DBG] pgmap v4668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:57.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:57 smithi115 bash[46934]: cluster 2022-04-22T18:56:56.520151+0000 mgr.y (mgr.34107) 3140 : cluster [DBG] pgmap v4668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:58 smithi121 bash[36902]: audit 2022-04-22T18:56:58.092121+0000 mon.a (mon.0) 6163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:58 smithi115 bash[45495]: audit 2022-04-22T18:56:58.092121+0000 mon.a (mon.0) 6163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:58 smithi115 bash[46934]: audit 2022-04-22T18:56:58.092121+0000 mon.a (mon.0) 6163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:56:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:59 smithi121 bash[36902]: cluster 2022-04-22T18:56:58.521368+0000 mgr.y (mgr.34107) 3141 : cluster [DBG] pgmap v4669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:56:59 smithi121 bash[36902]: audit 2022-04-22T18:56:59.177848+0000 mon.a (mon.0) 6164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:59 smithi115 bash[45495]: cluster 2022-04-22T18:56:58.521368+0000 mgr.y (mgr.34107) 3141 : cluster [DBG] pgmap v4669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:56:59 smithi115 bash[45495]: audit 2022-04-22T18:56:59.177848+0000 mon.a (mon.0) 6164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:56:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:59 smithi115 bash[46934]: cluster 2022-04-22T18:56:58.521368+0000 mgr.y (mgr.34107) 3141 : cluster [DBG] pgmap v4669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:56:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:56:59 smithi115 bash[46934]: audit 2022-04-22T18:56:59.177848+0000 mon.a (mon.0) 6164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:01.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:01 smithi115 bash[46934]: cluster 2022-04-22T18:57:00.522481+0000 mgr.y (mgr.34107) 3142 : cluster [DBG] pgmap v4670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:01.965 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:01 smithi115 bash[45495]: cluster 2022-04-22T18:57:00.522481+0000 mgr.y (mgr.34107) 3142 : cluster [DBG] pgmap v4670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:01 smithi121 bash[36902]: cluster 2022-04-22T18:57:00.522481+0000 mgr.y (mgr.34107) 3142 : cluster [DBG] pgmap v4670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:01.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:03 smithi121 bash[36902]: cluster 2022-04-22T18:57:02.523629+0000 mgr.y (mgr.34107) 3143 : cluster [DBG] pgmap v4671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:03 smithi121 bash[36902]: audit 2022-04-22T18:57:03.095691+0000 mon.a (mon.0) 6165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:03 smithi115 bash[46934]: cluster 2022-04-22T18:57:02.523629+0000 mgr.y (mgr.34107) 3143 : cluster [DBG] pgmap v4671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:03 smithi115 bash[46934]: audit 2022-04-22T18:57:03.095691+0000 mon.a (mon.0) 6165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:03 smithi115 bash[45495]: cluster 2022-04-22T18:57:02.523629+0000 mgr.y (mgr.34107) 3143 : cluster [DBG] pgmap v4671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:03 smithi115 bash[45495]: audit 2022-04-22T18:57:03.095691+0000 mon.a (mon.0) 6165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:04.197 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:03.825Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:04 smithi121 bash[36902]: audit 2022-04-22T18:57:04.180828+0000 mon.a (mon.0) 6166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:04 smithi115 bash[45495]: audit 2022-04-22T18:57:04.180828+0000 mon.a (mon.0) 6166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:04 smithi115 bash[46934]: audit 2022-04-22T18:57:04.180828+0000 mon.a (mon.0) 6166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:05 smithi121 bash[36902]: cluster 2022-04-22T18:57:04.524817+0000 mgr.y (mgr.34107) 3144 : cluster [DBG] pgmap v4672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:05 smithi115 bash[45495]: cluster 2022-04-22T18:57:04.524817+0000 mgr.y (mgr.34107) 3144 : cluster [DBG] pgmap v4672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:05 smithi115 bash[46934]: cluster 2022-04-22T18:57:04.524817+0000 mgr.y (mgr.34107) 3144 : cluster [DBG] pgmap v4672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:07 smithi121 bash[36902]: cluster 2022-04-22T18:57:06.525895+0000 mgr.y (mgr.34107) 3145 : cluster [DBG] pgmap v4673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:07 smithi115 bash[46934]: cluster 2022-04-22T18:57:06.525895+0000 mgr.y (mgr.34107) 3145 : cluster [DBG] pgmap v4673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:07 smithi115 bash[45495]: cluster 2022-04-22T18:57:06.525895+0000 mgr.y (mgr.34107) 3145 : cluster [DBG] pgmap v4673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:09.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:08 smithi121 bash[36902]: audit 2022-04-22T18:57:08.099320+0000 mon.a (mon.0) 6167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:08 smithi115 bash[45495]: audit 2022-04-22T18:57:08.099320+0000 mon.a (mon.0) 6167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:08 smithi115 bash[46934]: audit 2022-04-22T18:57:08.099320+0000 mon.a (mon.0) 6167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:09 smithi121 bash[36902]: cluster 2022-04-22T18:57:08.526937+0000 mgr.y (mgr.34107) 3146 : cluster [DBG] pgmap v4674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:09 smithi121 bash[36902]: audit 2022-04-22T18:57:09.184051+0000 mon.a (mon.0) 6168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:09 smithi115 bash[45495]: cluster 2022-04-22T18:57:08.526937+0000 mgr.y (mgr.34107) 3146 : cluster [DBG] pgmap v4674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:09 smithi115 bash[45495]: audit 2022-04-22T18:57:09.184051+0000 mon.a (mon.0) 6168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:09 smithi115 bash[46934]: cluster 2022-04-22T18:57:08.526937+0000 mgr.y (mgr.34107) 3146 : cluster [DBG] pgmap v4674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:09 smithi115 bash[46934]: audit 2022-04-22T18:57:09.184051+0000 mon.a (mon.0) 6168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:11.965 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:11 smithi115 bash[45495]: cluster 2022-04-22T18:57:10.528027+0000 mgr.y (mgr.34107) 3147 : cluster [DBG] pgmap v4675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:11.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:11 smithi115 bash[46934]: cluster 2022-04-22T18:57:10.528027+0000 mgr.y (mgr.34107) 3147 : cluster [DBG] pgmap v4675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:11 smithi121 bash[36902]: cluster 2022-04-22T18:57:10.528027+0000 mgr.y (mgr.34107) 3147 : cluster [DBG] pgmap v4675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:11.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:13 smithi121 bash[36902]: cluster 2022-04-22T18:57:12.529135+0000 mgr.y (mgr.34107) 3148 : cluster [DBG] pgmap v4676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:13 smithi121 bash[36902]: audit 2022-04-22T18:57:13.103039+0000 mon.a (mon.0) 6169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:13 smithi115 bash[46934]: cluster 2022-04-22T18:57:12.529135+0000 mgr.y (mgr.34107) 3148 : cluster [DBG] pgmap v4676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:13 smithi115 bash[46934]: audit 2022-04-22T18:57:13.103039+0000 mon.a (mon.0) 6169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:13 smithi115 bash[45495]: cluster 2022-04-22T18:57:12.529135+0000 mgr.y (mgr.34107) 3148 : cluster [DBG] pgmap v4676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:13 smithi115 bash[45495]: audit 2022-04-22T18:57:13.103039+0000 mon.a (mon.0) 6169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:14.198 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:13.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:14 smithi121 bash[36902]: audit 2022-04-22T18:57:14.187059+0000 mon.a (mon.0) 6170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:14 smithi115 bash[45495]: audit 2022-04-22T18:57:14.187059+0000 mon.a (mon.0) 6170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:14 smithi115 bash[46934]: audit 2022-04-22T18:57:14.187059+0000 mon.a (mon.0) 6170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:15 smithi121 bash[36902]: audit 2022-04-22T18:57:14.286958+0000 mon.a (mon.0) 6171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:57:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:15 smithi121 bash[36902]: audit 2022-04-22T18:57:14.287240+0000 mon.a (mon.0) 6172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:57:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:15 smithi121 bash[36902]: cluster 2022-04-22T18:57:14.530209+0000 mgr.y (mgr.34107) 3149 : cluster [DBG] pgmap v4677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[45495]: audit 2022-04-22T18:57:14.286958+0000 mon.a (mon.0) 6171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:57:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[45495]: audit 2022-04-22T18:57:14.287240+0000 mon.a (mon.0) 6172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:57:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[45495]: cluster 2022-04-22T18:57:14.530209+0000 mgr.y (mgr.34107) 3149 : cluster [DBG] pgmap v4677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[46934]: audit 2022-04-22T18:57:14.286958+0000 mon.a (mon.0) 6171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:57:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[46934]: audit 2022-04-22T18:57:14.287240+0000 mon.a (mon.0) 6172 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:57:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:15 smithi115 bash[46934]: cluster 2022-04-22T18:57:14.530209+0000 mgr.y (mgr.34107) 3149 : cluster [DBG] pgmap v4677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:17 smithi121 bash[36902]: cluster 2022-04-22T18:57:16.531294+0000 mgr.y (mgr.34107) 3150 : cluster [DBG] pgmap v4678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:17 smithi115 bash[46934]: cluster 2022-04-22T18:57:16.531294+0000 mgr.y (mgr.34107) 3150 : cluster [DBG] pgmap v4678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:17 smithi115 bash[45495]: cluster 2022-04-22T18:57:16.531294+0000 mgr.y (mgr.34107) 3150 : cluster [DBG] pgmap v4678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:18 smithi121 bash[36902]: audit 2022-04-22T18:57:18.106690+0000 mon.a (mon.0) 6173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:18 smithi115 bash[45495]: audit 2022-04-22T18:57:18.106690+0000 mon.a (mon.0) 6173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:18 smithi115 bash[46934]: audit 2022-04-22T18:57:18.106690+0000 mon.a (mon.0) 6173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:19 smithi121 bash[36902]: cluster 2022-04-22T18:57:18.532420+0000 mgr.y (mgr.34107) 3151 : cluster [DBG] pgmap v4679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:19 smithi121 bash[36902]: audit 2022-04-22T18:57:19.190468+0000 mon.a (mon.0) 6174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:19 smithi115 bash[45495]: cluster 2022-04-22T18:57:18.532420+0000 mgr.y (mgr.34107) 3151 : cluster [DBG] pgmap v4679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:19 smithi115 bash[45495]: audit 2022-04-22T18:57:19.190468+0000 mon.a (mon.0) 6174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:19 smithi115 bash[46934]: cluster 2022-04-22T18:57:18.532420+0000 mgr.y (mgr.34107) 3151 : cluster [DBG] pgmap v4679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:19 smithi115 bash[46934]: audit 2022-04-22T18:57:19.190468+0000 mon.a (mon.0) 6174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:21.966 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:21 smithi115 bash[45495]: cluster 2022-04-22T18:57:20.533469+0000 mgr.y (mgr.34107) 3152 : cluster [DBG] pgmap v4680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:21.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:21 smithi115 bash[46934]: cluster 2022-04-22T18:57:20.533469+0000 mgr.y (mgr.34107) 3152 : cluster [DBG] pgmap v4680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:21 smithi121 bash[36902]: cluster 2022-04-22T18:57:20.533469+0000 mgr.y (mgr.34107) 3152 : cluster [DBG] pgmap v4680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:21.957Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:23 smithi121 bash[36902]: cluster 2022-04-22T18:57:22.534661+0000 mgr.y (mgr.34107) 3153 : cluster [DBG] pgmap v4681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:23 smithi121 bash[36902]: audit 2022-04-22T18:57:23.110264+0000 mon.a (mon.0) 6175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:23 smithi115 bash[45495]: cluster 2022-04-22T18:57:22.534661+0000 mgr.y (mgr.34107) 3153 : cluster [DBG] pgmap v4681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:23 smithi115 bash[45495]: audit 2022-04-22T18:57:23.110264+0000 mon.a (mon.0) 6175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:23 smithi115 bash[46934]: cluster 2022-04-22T18:57:22.534661+0000 mgr.y (mgr.34107) 3153 : cluster [DBG] pgmap v4681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:23 smithi115 bash[46934]: audit 2022-04-22T18:57:23.110264+0000 mon.a (mon.0) 6175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:24.254 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:23.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:24 smithi121 bash[36902]: audit 2022-04-22T18:57:24.193638+0000 mon.a (mon.0) 6176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:24 smithi115 bash[46934]: audit 2022-04-22T18:57:24.193638+0000 mon.a (mon.0) 6176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:24 smithi115 bash[45495]: audit 2022-04-22T18:57:24.193638+0000 mon.a (mon.0) 6176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:25 smithi121 bash[36902]: cluster 2022-04-22T18:57:24.535843+0000 mgr.y (mgr.34107) 3154 : cluster [DBG] pgmap v4682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:25 smithi115 bash[45495]: cluster 2022-04-22T18:57:24.535843+0000 mgr.y (mgr.34107) 3154 : cluster [DBG] pgmap v4682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:25 smithi115 bash[46934]: cluster 2022-04-22T18:57:24.535843+0000 mgr.y (mgr.34107) 3154 : cluster [DBG] pgmap v4682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:25.987 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:57:26.343 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (80m) 7m ago 94m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (94m) 8m ago 94m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (82m) 7m ago 95m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (82m) 8m ago 95m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (89m) 8m ago 99m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (90m) 7m ago 101m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (89m) 7m ago 101m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (88m) 8m ago 99m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:57:26.344 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (88m) 7m ago 100m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (81m) 7m ago 94m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (81m) 8m ago 94m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (87m) 7m ago 99m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (86m) 7m ago 98m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (86m) 7m ago 98m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (85m) 7m ago 97m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (84m) 8m ago 97m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (84m) 8m ago 96m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (83m) 8m ago 96m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:57:26.345 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (82m) 8m ago 95m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:57:26.346 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (81m) 8m ago 94m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:57:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:26 smithi115 bash[45495]: audit 2022-04-22T18:57:25.962363+0000 mgr.y (mgr.34107) 3155 : audit [DBG] from='client.48652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:26 smithi115 bash[46934]: audit 2022-04-22T18:57:25.962363+0000 mgr.y (mgr.34107) 3155 : audit [DBG] from='client.48652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:26.739 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:57:26.740 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:57:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:26 smithi121 bash[36902]: audit 2022-04-22T18:57:25.962363+0000 mgr.y (mgr.34107) 3155 : audit [DBG] from='client.48652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[45495]: audit 2022-04-22T18:57:26.314921+0000 mgr.y (mgr.34107) 3156 : audit [DBG] from='client.38724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[45495]: cluster 2022-04-22T18:57:26.536588+0000 mgr.y (mgr.34107) 3157 : cluster [DBG] pgmap v4683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[45495]: audit 2022-04-22T18:57:26.730165+0000 mon.a (mon.0) 6177 : audit [DBG] from='client.? 172.21.15.115:0/3624091563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[46934]: audit 2022-04-22T18:57:26.314921+0000 mgr.y (mgr.34107) 3156 : audit [DBG] from='client.38724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[46934]: cluster 2022-04-22T18:57:26.536588+0000 mgr.y (mgr.34107) 3157 : cluster [DBG] pgmap v4683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:27 smithi115 bash[46934]: audit 2022-04-22T18:57:26.730165+0000 mon.a (mon.0) 6177 : audit [DBG] from='client.? 172.21.15.115:0/3624091563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:27 smithi121 bash[36902]: audit 2022-04-22T18:57:26.314921+0000 mgr.y (mgr.34107) 3156 : audit [DBG] from='client.38724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:27 smithi121 bash[36902]: cluster 2022-04-22T18:57:26.536588+0000 mgr.y (mgr.34107) 3157 : cluster [DBG] pgmap v4683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:27 smithi121 bash[36902]: audit 2022-04-22T18:57:26.730165+0000 mon.a (mon.0) 6177 : audit [DBG] from='client.? 172.21.15.115:0/3624091563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:28 smithi115 bash[45495]: audit 2022-04-22T18:57:28.114029+0000 mon.a (mon.0) 6178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:28 smithi115 bash[46934]: audit 2022-04-22T18:57:28.114029+0000 mon.a (mon.0) 6178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:28 smithi121 bash[36902]: audit 2022-04-22T18:57:28.114029+0000 mon.a (mon.0) 6178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:29 smithi115 bash[46934]: cluster 2022-04-22T18:57:28.537747+0000 mgr.y (mgr.34107) 3158 : cluster [DBG] pgmap v4684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:29 smithi115 bash[46934]: audit 2022-04-22T18:57:29.196704+0000 mon.a (mon.0) 6179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:29 smithi115 bash[45495]: cluster 2022-04-22T18:57:28.537747+0000 mgr.y (mgr.34107) 3158 : cluster [DBG] pgmap v4684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:29 smithi115 bash[45495]: audit 2022-04-22T18:57:29.196704+0000 mon.a (mon.0) 6179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:29 smithi121 bash[36902]: cluster 2022-04-22T18:57:28.537747+0000 mgr.y (mgr.34107) 3158 : cluster [DBG] pgmap v4684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:29.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:29 smithi121 bash[36902]: audit 2022-04-22T18:57:29.196704+0000 mon.a (mon.0) 6179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:31.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:31 smithi115 bash[46934]: cluster 2022-04-22T18:57:30.538971+0000 mgr.y (mgr.34107) 3159 : cluster [DBG] pgmap v4685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:31.966 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:31 smithi115 bash[45495]: cluster 2022-04-22T18:57:30.538971+0000 mgr.y (mgr.34107) 3159 : cluster [DBG] pgmap v4685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:31 smithi121 bash[36902]: cluster 2022-04-22T18:57:30.538971+0000 mgr.y (mgr.34107) 3159 : cluster [DBG] pgmap v4685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:31.957Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:33 smithi121 bash[36902]: cluster 2022-04-22T18:57:32.540165+0000 mgr.y (mgr.34107) 3160 : cluster [DBG] pgmap v4686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:33 smithi121 bash[36902]: audit 2022-04-22T18:57:33.117624+0000 mon.a (mon.0) 6180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:33 smithi115 bash[46934]: cluster 2022-04-22T18:57:32.540165+0000 mgr.y (mgr.34107) 3160 : cluster [DBG] pgmap v4686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:33 smithi115 bash[46934]: audit 2022-04-22T18:57:33.117624+0000 mon.a (mon.0) 6180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:33 smithi115 bash[45495]: cluster 2022-04-22T18:57:32.540165+0000 mgr.y (mgr.34107) 3160 : cluster [DBG] pgmap v4686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:33 smithi115 bash[45495]: audit 2022-04-22T18:57:33.117624+0000 mon.a (mon.0) 6180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:34.264 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:33.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:34 smithi121 bash[36902]: audit 2022-04-22T18:57:34.200100+0000 mon.a (mon.0) 6181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:34 smithi115 bash[45495]: audit 2022-04-22T18:57:34.200100+0000 mon.a (mon.0) 6181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:34 smithi115 bash[46934]: audit 2022-04-22T18:57:34.200100+0000 mon.a (mon.0) 6181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:35 smithi115 bash[45495]: cluster 2022-04-22T18:57:34.541171+0000 mgr.y (mgr.34107) 3161 : cluster [DBG] pgmap v4687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:35 smithi115 bash[46934]: cluster 2022-04-22T18:57:34.541171+0000 mgr.y (mgr.34107) 3161 : cluster [DBG] pgmap v4687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:35 smithi121 bash[36902]: cluster 2022-04-22T18:57:34.541171+0000 mgr.y (mgr.34107) 3161 : cluster [DBG] pgmap v4687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:37 smithi121 bash[36902]: cluster 2022-04-22T18:57:36.542291+0000 mgr.y (mgr.34107) 3162 : cluster [DBG] pgmap v4688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:37 smithi115 bash[45495]: cluster 2022-04-22T18:57:36.542291+0000 mgr.y (mgr.34107) 3162 : cluster [DBG] pgmap v4688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:37 smithi115 bash[46934]: cluster 2022-04-22T18:57:36.542291+0000 mgr.y (mgr.34107) 3162 : cluster [DBG] pgmap v4688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:38 smithi121 bash[36902]: audit 2022-04-22T18:57:38.122271+0000 mon.a (mon.0) 6182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:38 smithi115 bash[46934]: audit 2022-04-22T18:57:38.122271+0000 mon.a (mon.0) 6182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:38 smithi115 bash[45495]: audit 2022-04-22T18:57:38.122271+0000 mon.a (mon.0) 6182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:39 smithi121 bash[36902]: cluster 2022-04-22T18:57:38.543375+0000 mgr.y (mgr.34107) 3163 : cluster [DBG] pgmap v4689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:39 smithi121 bash[36902]: audit 2022-04-22T18:57:39.203361+0000 mon.a (mon.0) 6183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:39 smithi115 bash[46934]: cluster 2022-04-22T18:57:38.543375+0000 mgr.y (mgr.34107) 3163 : cluster [DBG] pgmap v4689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:39 smithi115 bash[46934]: audit 2022-04-22T18:57:39.203361+0000 mon.a (mon.0) 6183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:39 smithi115 bash[45495]: cluster 2022-04-22T18:57:38.543375+0000 mgr.y (mgr.34107) 3163 : cluster [DBG] pgmap v4689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:39 smithi115 bash[45495]: audit 2022-04-22T18:57:39.203361+0000 mon.a (mon.0) 6183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:41.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:41 smithi115 bash[45495]: cluster 2022-04-22T18:57:40.544381+0000 mgr.y (mgr.34107) 3164 : cluster [DBG] pgmap v4690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:41.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:41 smithi115 bash[46934]: cluster 2022-04-22T18:57:40.544381+0000 mgr.y (mgr.34107) 3164 : cluster [DBG] pgmap v4690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:41 smithi121 bash[36902]: cluster 2022-04-22T18:57:40.544381+0000 mgr.y (mgr.34107) 3164 : cluster [DBG] pgmap v4690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:41.958Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:43 smithi121 bash[36902]: cluster 2022-04-22T18:57:42.545560+0000 mgr.y (mgr.34107) 3165 : cluster [DBG] pgmap v4691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:43 smithi121 bash[36902]: audit 2022-04-22T18:57:43.126147+0000 mon.a (mon.0) 6184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:43 smithi115 bash[45495]: cluster 2022-04-22T18:57:42.545560+0000 mgr.y (mgr.34107) 3165 : cluster [DBG] pgmap v4691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:43 smithi115 bash[45495]: audit 2022-04-22T18:57:43.126147+0000 mon.a (mon.0) 6184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:43 smithi115 bash[46934]: cluster 2022-04-22T18:57:42.545560+0000 mgr.y (mgr.34107) 3165 : cluster [DBG] pgmap v4691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:43 smithi115 bash[46934]: audit 2022-04-22T18:57:43.126147+0000 mon.a (mon.0) 6184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:44.267 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:43.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:44 smithi115 bash[45495]: audit 2022-04-22T18:57:44.206507+0000 mon.a (mon.0) 6185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:44 smithi115 bash[46934]: audit 2022-04-22T18:57:44.206507+0000 mon.a (mon.0) 6185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:44 smithi121 bash[36902]: audit 2022-04-22T18:57:44.206507+0000 mon.a (mon.0) 6185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:45 smithi115 bash[45495]: cluster 2022-04-22T18:57:44.546711+0000 mgr.y (mgr.34107) 3166 : cluster [DBG] pgmap v4692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:45 smithi115 bash[46934]: cluster 2022-04-22T18:57:44.546711+0000 mgr.y (mgr.34107) 3166 : cluster [DBG] pgmap v4692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:45 smithi121 bash[36902]: cluster 2022-04-22T18:57:44.546711+0000 mgr.y (mgr.34107) 3166 : cluster [DBG] pgmap v4692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:47 smithi121 bash[36902]: cluster 2022-04-22T18:57:46.547697+0000 mgr.y (mgr.34107) 3167 : cluster [DBG] pgmap v4693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:47 smithi115 bash[45495]: cluster 2022-04-22T18:57:46.547697+0000 mgr.y (mgr.34107) 3167 : cluster [DBG] pgmap v4693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:47 smithi115 bash[46934]: cluster 2022-04-22T18:57:46.547697+0000 mgr.y (mgr.34107) 3167 : cluster [DBG] pgmap v4693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:48 smithi121 bash[36902]: audit 2022-04-22T18:57:48.129941+0000 mon.a (mon.0) 6186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:48 smithi115 bash[45495]: audit 2022-04-22T18:57:48.129941+0000 mon.a (mon.0) 6186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:48 smithi115 bash[46934]: audit 2022-04-22T18:57:48.129941+0000 mon.a (mon.0) 6186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:49 smithi121 bash[36902]: cluster 2022-04-22T18:57:48.549064+0000 mgr.y (mgr.34107) 3168 : cluster [DBG] pgmap v4694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:49 smithi121 bash[36902]: audit 2022-04-22T18:57:49.209721+0000 mon.a (mon.0) 6187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:49 smithi115 bash[45495]: cluster 2022-04-22T18:57:48.549064+0000 mgr.y (mgr.34107) 3168 : cluster [DBG] pgmap v4694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:49 smithi115 bash[45495]: audit 2022-04-22T18:57:49.209721+0000 mon.a (mon.0) 6187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:49 smithi115 bash[46934]: cluster 2022-04-22T18:57:48.549064+0000 mgr.y (mgr.34107) 3168 : cluster [DBG] pgmap v4694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:49 smithi115 bash[46934]: audit 2022-04-22T18:57:49.209721+0000 mon.a (mon.0) 6187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:50 smithi121 bash[36902]: audit 2022-04-22T18:57:50.071807+0000 mon.a (mon.0) 6188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:57:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:50 smithi121 bash[36902]: audit 2022-04-22T18:57:50.074685+0000 mon.a (mon.0) 6189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:57:50.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:50 smithi121 bash[36902]: audit 2022-04-22T18:57:50.076120+0000 mon.a (mon.0) 6190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:57:50.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:50 smithi121 bash[36902]: audit 2022-04-22T18:57:50.425182+0000 mon.a (mon.0) 6191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:57:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[45495]: audit 2022-04-22T18:57:50.071807+0000 mon.a (mon.0) 6188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:57:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[45495]: audit 2022-04-22T18:57:50.074685+0000 mon.a (mon.0) 6189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:57:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[45495]: audit 2022-04-22T18:57:50.076120+0000 mon.a (mon.0) 6190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:57:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[45495]: audit 2022-04-22T18:57:50.425182+0000 mon.a (mon.0) 6191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:57:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[46934]: audit 2022-04-22T18:57:50.071807+0000 mon.a (mon.0) 6188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:57:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[46934]: audit 2022-04-22T18:57:50.074685+0000 mon.a (mon.0) 6189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:57:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[46934]: audit 2022-04-22T18:57:50.076120+0000 mon.a (mon.0) 6190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:57:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:50 smithi115 bash[46934]: audit 2022-04-22T18:57:50.425182+0000 mon.a (mon.0) 6191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:57:51.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:51 smithi115 bash[45495]: cluster 2022-04-22T18:57:50.550098+0000 mgr.y (mgr.34107) 3169 : cluster [DBG] pgmap v4695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:51.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:51 smithi115 bash[46934]: cluster 2022-04-22T18:57:50.550098+0000 mgr.y (mgr.34107) 3169 : cluster [DBG] pgmap v4695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:52.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:51 smithi121 bash[36902]: cluster 2022-04-22T18:57:50.550098+0000 mgr.y (mgr.34107) 3169 : cluster [DBG] pgmap v4695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:51.958Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:53 smithi121 bash[36902]: cluster 2022-04-22T18:57:52.551152+0000 mgr.y (mgr.34107) 3170 : cluster [DBG] pgmap v4696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:53 smithi121 bash[36902]: audit 2022-04-22T18:57:53.133542+0000 mon.a (mon.0) 6192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:53 smithi115 bash[45495]: cluster 2022-04-22T18:57:52.551152+0000 mgr.y (mgr.34107) 3170 : cluster [DBG] pgmap v4696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:53 smithi115 bash[45495]: audit 2022-04-22T18:57:53.133542+0000 mon.a (mon.0) 6192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:53 smithi115 bash[46934]: cluster 2022-04-22T18:57:52.551152+0000 mgr.y (mgr.34107) 3170 : cluster [DBG] pgmap v4696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:53 smithi115 bash[46934]: audit 2022-04-22T18:57:53.133542+0000 mon.a (mon.0) 6192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:54.274 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:57:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:57:53.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:57:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:54 smithi115 bash[45495]: audit 2022-04-22T18:57:54.212731+0000 mon.a (mon.0) 6193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:54.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:54 smithi115 bash[46934]: audit 2022-04-22T18:57:54.212731+0000 mon.a (mon.0) 6193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:54 smithi121 bash[36902]: audit 2022-04-22T18:57:54.212731+0000 mon.a (mon.0) 6193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:57:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:55 smithi115 bash[45495]: cluster 2022-04-22T18:57:54.552236+0000 mgr.y (mgr.34107) 3171 : cluster [DBG] pgmap v4697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:55 smithi115 bash[46934]: cluster 2022-04-22T18:57:54.552236+0000 mgr.y (mgr.34107) 3171 : cluster [DBG] pgmap v4697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:55 smithi121 bash[36902]: cluster 2022-04-22T18:57:54.552236+0000 mgr.y (mgr.34107) 3171 : cluster [DBG] pgmap v4697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:57.120 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:57:57.466 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (81m) 8m ago 94m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (94m) 9m ago 94m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (82m) 8m ago 95m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (82m) 9m ago 95m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (90m) 9m ago 100m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (90m) 8m ago 102m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:57:57.467 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (89m) 8m ago 102m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (88m) 9m ago 100m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (89m) 8m ago 100m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (82m) 8m ago 95m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (82m) 9m ago 95m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (87m) 8m ago 99m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (87m) 8m ago 99m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (86m) 8m ago 98m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (86m) 8m ago 98m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:57:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (85m) 9m ago 97m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:57:57.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (84m) 9m ago 97m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:57:57.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (84m) 9m ago 96m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:57:57.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (83m) 9m ago 96m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:57:57.469 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (81m) 9m ago 95m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:57:57.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:57:57.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:57:57.877 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:57:57.877 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:57:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:57 smithi121 bash[36902]: cluster 2022-04-22T18:57:56.553314+0000 mgr.y (mgr.34107) 3172 : cluster [DBG] pgmap v4698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:57 smithi121 bash[36902]: audit 2022-04-22T18:57:57.092848+0000 mgr.y (mgr.34107) 3173 : audit [DBG] from='client.38736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:57 smithi115 bash[45495]: cluster 2022-04-22T18:57:56.553314+0000 mgr.y (mgr.34107) 3172 : cluster [DBG] pgmap v4698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:57 smithi115 bash[45495]: audit 2022-04-22T18:57:57.092848+0000 mgr.y (mgr.34107) 3173 : audit [DBG] from='client.38736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:57 smithi115 bash[46934]: cluster 2022-04-22T18:57:56.553314+0000 mgr.y (mgr.34107) 3172 : cluster [DBG] pgmap v4698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:57:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:57 smithi115 bash[46934]: audit 2022-04-22T18:57:57.092848+0000 mgr.y (mgr.34107) 3173 : audit [DBG] from='client.38736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:58 smithi121 bash[36902]: audit 2022-04-22T18:57:57.439716+0000 mgr.y (mgr.34107) 3174 : audit [DBG] from='client.48676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:58 smithi121 bash[36902]: audit 2022-04-22T18:57:57.866108+0000 mon.a (mon.0) 6194 : audit [DBG] from='client.? 172.21.15.115:0/4178941901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:58 smithi121 bash[36902]: audit 2022-04-22T18:57:58.137192+0000 mon.a (mon.0) 6195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[45495]: audit 2022-04-22T18:57:57.439716+0000 mgr.y (mgr.34107) 3174 : audit [DBG] from='client.48676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[45495]: audit 2022-04-22T18:57:57.866108+0000 mon.a (mon.0) 6194 : audit [DBG] from='client.? 172.21.15.115:0/4178941901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[45495]: audit 2022-04-22T18:57:58.137192+0000 mon.a (mon.0) 6195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:57:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[46934]: audit 2022-04-22T18:57:57.439716+0000 mgr.y (mgr.34107) 3174 : audit [DBG] from='client.48676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:57:59.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[46934]: audit 2022-04-22T18:57:57.866108+0000 mon.a (mon.0) 6194 : audit [DBG] from='client.? 172.21.15.115:0/4178941901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:57:59.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:58 smithi115 bash[46934]: audit 2022-04-22T18:57:58.137192+0000 mon.a (mon.0) 6195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:59 smithi121 bash[36902]: cluster 2022-04-22T18:57:58.554411+0000 mgr.y (mgr.34107) 3175 : cluster [DBG] pgmap v4699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:57:59 smithi121 bash[36902]: audit 2022-04-22T18:57:59.215548+0000 mon.a (mon.0) 6196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:00.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:59 smithi115 bash[45495]: cluster 2022-04-22T18:57:58.554411+0000 mgr.y (mgr.34107) 3175 : cluster [DBG] pgmap v4699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:00.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:57:59 smithi115 bash[45495]: audit 2022-04-22T18:57:59.215548+0000 mon.a (mon.0) 6196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:00.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:59 smithi115 bash[46934]: cluster 2022-04-22T18:57:58.554411+0000 mgr.y (mgr.34107) 3175 : cluster [DBG] pgmap v4699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:00.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:57:59 smithi115 bash[46934]: audit 2022-04-22T18:57:59.215548+0000 mon.a (mon.0) 6196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:01.969 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:01 smithi115 bash[45495]: cluster 2022-04-22T18:58:00.555533+0000 mgr.y (mgr.34107) 3176 : cluster [DBG] pgmap v4700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:01.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:01 smithi115 bash[46934]: cluster 2022-04-22T18:58:00.555533+0000 mgr.y (mgr.34107) 3176 : cluster [DBG] pgmap v4700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:01 smithi121 bash[36902]: cluster 2022-04-22T18:58:00.555533+0000 mgr.y (mgr.34107) 3176 : cluster [DBG] pgmap v4700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:01.960Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:03 smithi121 bash[36902]: cluster 2022-04-22T18:58:02.556715+0000 mgr.y (mgr.34107) 3177 : cluster [DBG] pgmap v4701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:03 smithi121 bash[36902]: audit 2022-04-22T18:58:03.140604+0000 mon.a (mon.0) 6197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:03 smithi115 bash[45495]: cluster 2022-04-22T18:58:02.556715+0000 mgr.y (mgr.34107) 3177 : cluster [DBG] pgmap v4701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:03 smithi115 bash[45495]: audit 2022-04-22T18:58:03.140604+0000 mon.a (mon.0) 6197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:03 smithi115 bash[46934]: cluster 2022-04-22T18:58:02.556715+0000 mgr.y (mgr.34107) 3177 : cluster [DBG] pgmap v4701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:03 smithi115 bash[46934]: audit 2022-04-22T18:58:03.140604+0000 mon.a (mon.0) 6197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:04.279 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:03.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:04 smithi115 bash[45495]: audit 2022-04-22T18:58:04.218110+0000 mon.a (mon.0) 6198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:04 smithi115 bash[46934]: audit 2022-04-22T18:58:04.218110+0000 mon.a (mon.0) 6198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:04 smithi121 bash[36902]: audit 2022-04-22T18:58:04.218110+0000 mon.a (mon.0) 6198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:05 smithi115 bash[45495]: cluster 2022-04-22T18:58:04.557786+0000 mgr.y (mgr.34107) 3178 : cluster [DBG] pgmap v4702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:05 smithi115 bash[46934]: cluster 2022-04-22T18:58:04.557786+0000 mgr.y (mgr.34107) 3178 : cluster [DBG] pgmap v4702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:05 smithi121 bash[36902]: cluster 2022-04-22T18:58:04.557786+0000 mgr.y (mgr.34107) 3178 : cluster [DBG] pgmap v4702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:07 smithi121 bash[36902]: cluster 2022-04-22T18:58:06.558860+0000 mgr.y (mgr.34107) 3179 : cluster [DBG] pgmap v4703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:07 smithi115 bash[46934]: cluster 2022-04-22T18:58:06.558860+0000 mgr.y (mgr.34107) 3179 : cluster [DBG] pgmap v4703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:07 smithi115 bash[45495]: cluster 2022-04-22T18:58:06.558860+0000 mgr.y (mgr.34107) 3179 : cluster [DBG] pgmap v4703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:08 smithi121 bash[36902]: audit 2022-04-22T18:58:08.144165+0000 mon.a (mon.0) 6199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:08 smithi115 bash[45495]: audit 2022-04-22T18:58:08.144165+0000 mon.a (mon.0) 6199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:08 smithi115 bash[46934]: audit 2022-04-22T18:58:08.144165+0000 mon.a (mon.0) 6199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:09 smithi121 bash[36902]: cluster 2022-04-22T18:58:08.559972+0000 mgr.y (mgr.34107) 3180 : cluster [DBG] pgmap v4704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:09 smithi121 bash[36902]: audit 2022-04-22T18:58:09.221347+0000 mon.a (mon.0) 6200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:09 smithi115 bash[45495]: cluster 2022-04-22T18:58:08.559972+0000 mgr.y (mgr.34107) 3180 : cluster [DBG] pgmap v4704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:10.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:09 smithi115 bash[45495]: audit 2022-04-22T18:58:09.221347+0000 mon.a (mon.0) 6200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:10.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:09 smithi115 bash[46934]: cluster 2022-04-22T18:58:08.559972+0000 mgr.y (mgr.34107) 3180 : cluster [DBG] pgmap v4704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:10.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:09 smithi115 bash[46934]: audit 2022-04-22T18:58:09.221347+0000 mon.a (mon.0) 6200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:11.971 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:11 smithi115 bash[45495]: cluster 2022-04-22T18:58:10.561415+0000 mgr.y (mgr.34107) 3181 : cluster [DBG] pgmap v4705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:11.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:11 smithi115 bash[46934]: cluster 2022-04-22T18:58:10.561415+0000 mgr.y (mgr.34107) 3181 : cluster [DBG] pgmap v4705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:11 smithi121 bash[36902]: cluster 2022-04-22T18:58:10.561415+0000 mgr.y (mgr.34107) 3181 : cluster [DBG] pgmap v4705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:11.962Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:13 smithi121 bash[36902]: cluster 2022-04-22T18:58:12.562581+0000 mgr.y (mgr.34107) 3182 : cluster [DBG] pgmap v4706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:13 smithi121 bash[36902]: audit 2022-04-22T18:58:13.147671+0000 mon.a (mon.0) 6201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:13 smithi115 bash[45495]: cluster 2022-04-22T18:58:12.562581+0000 mgr.y (mgr.34107) 3182 : cluster [DBG] pgmap v4706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:13 smithi115 bash[45495]: audit 2022-04-22T18:58:13.147671+0000 mon.a (mon.0) 6201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:13 smithi115 bash[46934]: cluster 2022-04-22T18:58:12.562581+0000 mgr.y (mgr.34107) 3182 : cluster [DBG] pgmap v4706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:13 smithi115 bash[46934]: audit 2022-04-22T18:58:13.147671+0000 mon.a (mon.0) 6201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:14.286 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:13.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:14 smithi115 bash[45495]: audit 2022-04-22T18:58:14.224282+0000 mon.a (mon.0) 6202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:14 smithi115 bash[46934]: audit 2022-04-22T18:58:14.224282+0000 mon.a (mon.0) 6202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:14 smithi121 bash[36902]: audit 2022-04-22T18:58:14.224282+0000 mon.a (mon.0) 6202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:15 smithi121 bash[36902]: audit 2022-04-22T18:58:14.297262+0000 mon.a (mon.0) 6203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:58:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:15 smithi121 bash[36902]: audit 2022-04-22T18:58:14.297859+0000 mon.a (mon.0) 6204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:58:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:15 smithi121 bash[36902]: cluster 2022-04-22T18:58:14.563762+0000 mgr.y (mgr.34107) 3183 : cluster [DBG] pgmap v4707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[45495]: audit 2022-04-22T18:58:14.297262+0000 mon.a (mon.0) 6203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:58:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[45495]: audit 2022-04-22T18:58:14.297859+0000 mon.a (mon.0) 6204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:58:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[45495]: cluster 2022-04-22T18:58:14.563762+0000 mgr.y (mgr.34107) 3183 : cluster [DBG] pgmap v4707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[46934]: audit 2022-04-22T18:58:14.297262+0000 mon.a (mon.0) 6203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:58:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[46934]: audit 2022-04-22T18:58:14.297859+0000 mon.a (mon.0) 6204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:58:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:15 smithi115 bash[46934]: cluster 2022-04-22T18:58:14.563762+0000 mgr.y (mgr.34107) 3183 : cluster [DBG] pgmap v4707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:17 smithi121 bash[36902]: cluster 2022-04-22T18:58:16.564921+0000 mgr.y (mgr.34107) 3184 : cluster [DBG] pgmap v4708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:17 smithi115 bash[45495]: cluster 2022-04-22T18:58:16.564921+0000 mgr.y (mgr.34107) 3184 : cluster [DBG] pgmap v4708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:17 smithi115 bash[46934]: cluster 2022-04-22T18:58:16.564921+0000 mgr.y (mgr.34107) 3184 : cluster [DBG] pgmap v4708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:18 smithi121 bash[36902]: audit 2022-04-22T18:58:18.150957+0000 mon.a (mon.0) 6205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:18 smithi115 bash[45495]: audit 2022-04-22T18:58:18.150957+0000 mon.a (mon.0) 6205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:18 smithi115 bash[46934]: audit 2022-04-22T18:58:18.150957+0000 mon.a (mon.0) 6205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:19 smithi121 bash[36902]: cluster 2022-04-22T18:58:18.566077+0000 mgr.y (mgr.34107) 3185 : cluster [DBG] pgmap v4709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:19 smithi121 bash[36902]: audit 2022-04-22T18:58:19.227341+0000 mon.a (mon.0) 6206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:19 smithi115 bash[45495]: cluster 2022-04-22T18:58:18.566077+0000 mgr.y (mgr.34107) 3185 : cluster [DBG] pgmap v4709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:19 smithi115 bash[45495]: audit 2022-04-22T18:58:19.227341+0000 mon.a (mon.0) 6206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:19 smithi115 bash[46934]: cluster 2022-04-22T18:58:18.566077+0000 mgr.y (mgr.34107) 3185 : cluster [DBG] pgmap v4709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:19 smithi115 bash[46934]: audit 2022-04-22T18:58:19.227341+0000 mon.a (mon.0) 6206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:21.970 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:21 smithi115 bash[45495]: cluster 2022-04-22T18:58:20.567075+0000 mgr.y (mgr.34107) 3186 : cluster [DBG] pgmap v4710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:21.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:21 smithi115 bash[46934]: cluster 2022-04-22T18:58:20.567075+0000 mgr.y (mgr.34107) 3186 : cluster [DBG] pgmap v4710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:21 smithi121 bash[36902]: cluster 2022-04-22T18:58:20.567075+0000 mgr.y (mgr.34107) 3186 : cluster [DBG] pgmap v4710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:21.961Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:23 smithi121 bash[36902]: cluster 2022-04-22T18:58:22.568252+0000 mgr.y (mgr.34107) 3187 : cluster [DBG] pgmap v4711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:23 smithi121 bash[36902]: audit 2022-04-22T18:58:23.154822+0000 mon.a (mon.0) 6207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:23 smithi115 bash[45495]: cluster 2022-04-22T18:58:22.568252+0000 mgr.y (mgr.34107) 3187 : cluster [DBG] pgmap v4711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:23 smithi115 bash[45495]: audit 2022-04-22T18:58:23.154822+0000 mon.a (mon.0) 6207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:23 smithi115 bash[46934]: cluster 2022-04-22T18:58:22.568252+0000 mgr.y (mgr.34107) 3187 : cluster [DBG] pgmap v4711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:23 smithi115 bash[46934]: audit 2022-04-22T18:58:23.154822+0000 mon.a (mon.0) 6207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:24.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:23.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:24 smithi115 bash[45495]: audit 2022-04-22T18:58:24.230723+0000 mon.a (mon.0) 6208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:24 smithi115 bash[46934]: audit 2022-04-22T18:58:24.230723+0000 mon.a (mon.0) 6208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:24 smithi121 bash[36902]: audit 2022-04-22T18:58:24.230723+0000 mon.a (mon.0) 6208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:25 smithi121 bash[36902]: cluster 2022-04-22T18:58:24.569401+0000 mgr.y (mgr.34107) 3188 : cluster [DBG] pgmap v4712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:25 smithi115 bash[45495]: cluster 2022-04-22T18:58:24.569401+0000 mgr.y (mgr.34107) 3188 : cluster [DBG] pgmap v4712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:25 smithi115 bash[46934]: cluster 2022-04-22T18:58:24.569401+0000 mgr.y (mgr.34107) 3188 : cluster [DBG] pgmap v4712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:27 smithi121 bash[36902]: cluster 2022-04-22T18:58:26.570511+0000 mgr.y (mgr.34107) 3189 : cluster [DBG] pgmap v4713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:27 smithi115 bash[45495]: cluster 2022-04-22T18:58:26.570511+0000 mgr.y (mgr.34107) 3189 : cluster [DBG] pgmap v4713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:27 smithi115 bash[46934]: cluster 2022-04-22T18:58:26.570511+0000 mgr.y (mgr.34107) 3189 : cluster [DBG] pgmap v4713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:28.249 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (81m) 8m ago 95m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (95m) 9m ago 95m 25.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (83m) 8m ago 96m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (83m) 9m ago 96m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (90m) 9m ago 100m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:58:28.591 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (91m) 8m ago 102m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (90m) 8m ago 102m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (89m) 9m ago 100m 158M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (89m) 8m ago 101m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (82m) 8m ago 95m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (82m) 9m ago 95m 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (88m) 8m ago 100m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (87m) 8m ago 99m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (87m) 8m ago 99m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (86m) 8m ago 98m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:58:28.592 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (85m) 9m ago 98m 568M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:58:28.593 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (85m) 9m ago 97m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:58:28.593 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (84m) 9m ago 97m 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:58:28.593 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (83m) 9m ago 96m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:58:28.593 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (82m) 9m ago 95m 63.9M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:58:28.996 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:58:28.996 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:58:28.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:58:28.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:58:28.998 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:58:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:28 smithi121 bash[36902]: audit 2022-04-22T18:58:28.158483+0000 mon.a (mon.0) 6209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:28 smithi115 bash[45495]: audit 2022-04-22T18:58:28.158483+0000 mon.a (mon.0) 6209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:28 smithi115 bash[46934]: audit 2022-04-22T18:58:28.158483+0000 mon.a (mon.0) 6209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:29 smithi121 bash[36902]: audit 2022-04-22T18:58:28.224796+0000 mgr.y (mgr.34107) 3190 : audit [DBG] from='client.48688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:29 smithi121 bash[36902]: audit 2022-04-22T18:58:28.563315+0000 mgr.y (mgr.34107) 3191 : audit [DBG] from='client.38760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:29 smithi121 bash[36902]: cluster 2022-04-22T18:58:28.571153+0000 mgr.y (mgr.34107) 3192 : cluster [DBG] pgmap v4714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:29 smithi121 bash[36902]: audit 2022-04-22T18:58:28.987098+0000 mon.c (mon.1) 170 : audit [DBG] from='client.? 172.21.15.115:0/1716554620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:58:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:29 smithi121 bash[36902]: audit 2022-04-22T18:58:29.235047+0000 mon.a (mon.0) 6210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[46934]: audit 2022-04-22T18:58:28.224796+0000 mgr.y (mgr.34107) 3190 : audit [DBG] from='client.48688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[46934]: audit 2022-04-22T18:58:28.563315+0000 mgr.y (mgr.34107) 3191 : audit [DBG] from='client.38760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[46934]: cluster 2022-04-22T18:58:28.571153+0000 mgr.y (mgr.34107) 3192 : cluster [DBG] pgmap v4714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[46934]: audit 2022-04-22T18:58:28.987098+0000 mon.c (mon.1) 170 : audit [DBG] from='client.? 172.21.15.115:0/1716554620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[46934]: audit 2022-04-22T18:58:29.235047+0000 mon.a (mon.0) 6210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[45495]: audit 2022-04-22T18:58:28.224796+0000 mgr.y (mgr.34107) 3190 : audit [DBG] from='client.48688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[45495]: audit 2022-04-22T18:58:28.563315+0000 mgr.y (mgr.34107) 3191 : audit [DBG] from='client.38760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[45495]: cluster 2022-04-22T18:58:28.571153+0000 mgr.y (mgr.34107) 3192 : cluster [DBG] pgmap v4714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[45495]: audit 2022-04-22T18:58:28.987098+0000 mon.c (mon.1) 170 : audit [DBG] from='client.? 172.21.15.115:0/1716554620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:58:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:29 smithi115 bash[45495]: audit 2022-04-22T18:58:29.235047+0000 mon.a (mon.0) 6210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:31.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:31 smithi115 bash[46934]: cluster 2022-04-22T18:58:30.572173+0000 mgr.y (mgr.34107) 3193 : cluster [DBG] pgmap v4715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:31.972 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:31 smithi115 bash[45495]: cluster 2022-04-22T18:58:30.572173+0000 mgr.y (mgr.34107) 3193 : cluster [DBG] pgmap v4715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:31 smithi121 bash[36902]: cluster 2022-04-22T18:58:30.572173+0000 mgr.y (mgr.34107) 3193 : cluster [DBG] pgmap v4715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:31.963Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:33 smithi121 bash[36902]: cluster 2022-04-22T18:58:32.573293+0000 mgr.y (mgr.34107) 3194 : cluster [DBG] pgmap v4716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:33 smithi121 bash[36902]: audit 2022-04-22T18:58:33.161717+0000 mon.a (mon.0) 6211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:33 smithi115 bash[45495]: cluster 2022-04-22T18:58:32.573293+0000 mgr.y (mgr.34107) 3194 : cluster [DBG] pgmap v4716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:33 smithi115 bash[45495]: audit 2022-04-22T18:58:33.161717+0000 mon.a (mon.0) 6211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:33 smithi115 bash[46934]: cluster 2022-04-22T18:58:32.573293+0000 mgr.y (mgr.34107) 3194 : cluster [DBG] pgmap v4716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:33 smithi115 bash[46934]: audit 2022-04-22T18:58:33.161717+0000 mon.a (mon.0) 6211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:33.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:34 smithi121 bash[36902]: audit 2022-04-22T18:58:34.237992+0000 mon.a (mon.0) 6212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:34 smithi115 bash[45495]: audit 2022-04-22T18:58:34.237992+0000 mon.a (mon.0) 6212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:34 smithi115 bash[46934]: audit 2022-04-22T18:58:34.237992+0000 mon.a (mon.0) 6212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:35 smithi121 bash[36902]: cluster 2022-04-22T18:58:34.574507+0000 mgr.y (mgr.34107) 3195 : cluster [DBG] pgmap v4717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:35 smithi115 bash[45495]: cluster 2022-04-22T18:58:34.574507+0000 mgr.y (mgr.34107) 3195 : cluster [DBG] pgmap v4717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:35 smithi115 bash[46934]: cluster 2022-04-22T18:58:34.574507+0000 mgr.y (mgr.34107) 3195 : cluster [DBG] pgmap v4717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:37 smithi121 bash[36902]: cluster 2022-04-22T18:58:36.575802+0000 mgr.y (mgr.34107) 3196 : cluster [DBG] pgmap v4718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:37 smithi115 bash[45495]: cluster 2022-04-22T18:58:36.575802+0000 mgr.y (mgr.34107) 3196 : cluster [DBG] pgmap v4718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:37 smithi115 bash[46934]: cluster 2022-04-22T18:58:36.575802+0000 mgr.y (mgr.34107) 3196 : cluster [DBG] pgmap v4718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:38 smithi121 bash[36902]: audit 2022-04-22T18:58:38.165395+0000 mon.a (mon.0) 6213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:38 smithi115 bash[46934]: audit 2022-04-22T18:58:38.165395+0000 mon.a (mon.0) 6213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:39.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:38 smithi115 bash[45495]: audit 2022-04-22T18:58:38.165395+0000 mon.a (mon.0) 6213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:39 smithi121 bash[36902]: cluster 2022-04-22T18:58:38.577084+0000 mgr.y (mgr.34107) 3197 : cluster [DBG] pgmap v4719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:39 smithi121 bash[36902]: audit 2022-04-22T18:58:39.241013+0000 mon.a (mon.0) 6214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:39 smithi115 bash[46934]: cluster 2022-04-22T18:58:38.577084+0000 mgr.y (mgr.34107) 3197 : cluster [DBG] pgmap v4719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:39 smithi115 bash[46934]: audit 2022-04-22T18:58:39.241013+0000 mon.a (mon.0) 6214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:39 smithi115 bash[45495]: cluster 2022-04-22T18:58:38.577084+0000 mgr.y (mgr.34107) 3197 : cluster [DBG] pgmap v4719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:39 smithi115 bash[45495]: audit 2022-04-22T18:58:39.241013+0000 mon.a (mon.0) 6214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:41.972 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:41 smithi115 bash[45495]: cluster 2022-04-22T18:58:40.578176+0000 mgr.y (mgr.34107) 3198 : cluster [DBG] pgmap v4720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:41.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:41 smithi115 bash[46934]: cluster 2022-04-22T18:58:40.578176+0000 mgr.y (mgr.34107) 3198 : cluster [DBG] pgmap v4720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:41 smithi121 bash[36902]: cluster 2022-04-22T18:58:40.578176+0000 mgr.y (mgr.34107) 3198 : cluster [DBG] pgmap v4720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:41.963Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:43 smithi121 bash[36902]: cluster 2022-04-22T18:58:42.579298+0000 mgr.y (mgr.34107) 3199 : cluster [DBG] pgmap v4721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:43 smithi121 bash[36902]: audit 2022-04-22T18:58:43.169178+0000 mon.a (mon.0) 6215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:43 smithi115 bash[45495]: cluster 2022-04-22T18:58:42.579298+0000 mgr.y (mgr.34107) 3199 : cluster [DBG] pgmap v4721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:43 smithi115 bash[45495]: audit 2022-04-22T18:58:43.169178+0000 mon.a (mon.0) 6215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:43 smithi115 bash[46934]: cluster 2022-04-22T18:58:42.579298+0000 mgr.y (mgr.34107) 3199 : cluster [DBG] pgmap v4721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:43 smithi115 bash[46934]: audit 2022-04-22T18:58:43.169178+0000 mon.a (mon.0) 6215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:43.834Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:44 smithi121 bash[36902]: audit 2022-04-22T18:58:44.243887+0000 mon.a (mon.0) 6216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:44 smithi115 bash[45495]: audit 2022-04-22T18:58:44.243887+0000 mon.a (mon.0) 6216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:44 smithi115 bash[46934]: audit 2022-04-22T18:58:44.243887+0000 mon.a (mon.0) 6216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:45 smithi121 bash[36902]: cluster 2022-04-22T18:58:44.580429+0000 mgr.y (mgr.34107) 3200 : cluster [DBG] pgmap v4722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:45 smithi115 bash[46934]: cluster 2022-04-22T18:58:44.580429+0000 mgr.y (mgr.34107) 3200 : cluster [DBG] pgmap v4722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:45 smithi115 bash[45495]: cluster 2022-04-22T18:58:44.580429+0000 mgr.y (mgr.34107) 3200 : cluster [DBG] pgmap v4722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:47 smithi121 bash[36902]: cluster 2022-04-22T18:58:46.581575+0000 mgr.y (mgr.34107) 3201 : cluster [DBG] pgmap v4723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:47 smithi115 bash[45495]: cluster 2022-04-22T18:58:46.581575+0000 mgr.y (mgr.34107) 3201 : cluster [DBG] pgmap v4723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:47 smithi115 bash[46934]: cluster 2022-04-22T18:58:46.581575+0000 mgr.y (mgr.34107) 3201 : cluster [DBG] pgmap v4723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:48 smithi121 bash[36902]: audit 2022-04-22T18:58:48.172888+0000 mon.a (mon.0) 6217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:48 smithi115 bash[45495]: audit 2022-04-22T18:58:48.172888+0000 mon.a (mon.0) 6217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:48 smithi115 bash[46934]: audit 2022-04-22T18:58:48.172888+0000 mon.a (mon.0) 6217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:49 smithi121 bash[36902]: cluster 2022-04-22T18:58:48.582687+0000 mgr.y (mgr.34107) 3202 : cluster [DBG] pgmap v4724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:49 smithi121 bash[36902]: audit 2022-04-22T18:58:49.247143+0000 mon.a (mon.0) 6218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:49 smithi115 bash[45495]: cluster 2022-04-22T18:58:48.582687+0000 mgr.y (mgr.34107) 3202 : cluster [DBG] pgmap v4724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:49 smithi115 bash[45495]: audit 2022-04-22T18:58:49.247143+0000 mon.a (mon.0) 6218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:49 smithi115 bash[46934]: cluster 2022-04-22T18:58:48.582687+0000 mgr.y (mgr.34107) 3202 : cluster [DBG] pgmap v4724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:49 smithi115 bash[46934]: audit 2022-04-22T18:58:49.247143+0000 mon.a (mon.0) 6218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:50 smithi121 bash[36902]: audit 2022-04-22T18:58:50.434128+0000 mon.a (mon.0) 6219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:58:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:50 smithi121 bash[36902]: audit 2022-04-22T18:58:50.437253+0000 mon.a (mon.0) 6220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:58:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:50 smithi121 bash[36902]: audit 2022-04-22T18:58:50.438921+0000 mon.a (mon.0) 6221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:58:50.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[46934]: audit 2022-04-22T18:58:50.434128+0000 mon.a (mon.0) 6219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:58:50.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[46934]: audit 2022-04-22T18:58:50.437253+0000 mon.a (mon.0) 6220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:58:50.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[46934]: audit 2022-04-22T18:58:50.438921+0000 mon.a (mon.0) 6221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:58:50.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[45495]: audit 2022-04-22T18:58:50.434128+0000 mon.a (mon.0) 6219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:58:50.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[45495]: audit 2022-04-22T18:58:50.437253+0000 mon.a (mon.0) 6220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:58:50.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:50 smithi115 bash[45495]: audit 2022-04-22T18:58:50.438921+0000 mon.a (mon.0) 6221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:58:51.971 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:51 smithi115 bash[45495]: cluster 2022-04-22T18:58:50.583766+0000 mgr.y (mgr.34107) 3203 : cluster [DBG] pgmap v4725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:51.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:51 smithi115 bash[46934]: cluster 2022-04-22T18:58:50.583766+0000 mgr.y (mgr.34107) 3203 : cluster [DBG] pgmap v4725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:51 smithi121 bash[36902]: cluster 2022-04-22T18:58:50.583766+0000 mgr.y (mgr.34107) 3203 : cluster [DBG] pgmap v4725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:51.963Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:53 smithi121 bash[36902]: cluster 2022-04-22T18:58:52.584940+0000 mgr.y (mgr.34107) 3204 : cluster [DBG] pgmap v4726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:53 smithi121 bash[36902]: audit 2022-04-22T18:58:53.176524+0000 mon.a (mon.0) 6222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:53 smithi115 bash[45495]: cluster 2022-04-22T18:58:52.584940+0000 mgr.y (mgr.34107) 3204 : cluster [DBG] pgmap v4726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:53 smithi115 bash[45495]: audit 2022-04-22T18:58:53.176524+0000 mon.a (mon.0) 6222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:53 smithi115 bash[46934]: cluster 2022-04-22T18:58:52.584940+0000 mgr.y (mgr.34107) 3204 : cluster [DBG] pgmap v4726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:53 smithi115 bash[46934]: audit 2022-04-22T18:58:53.176524+0000 mon.a (mon.0) 6222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:58:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:58:53.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:58:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:54 smithi121 bash[36902]: audit 2022-04-22T18:58:54.250179+0000 mon.a (mon.0) 6223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:54 smithi115 bash[45495]: audit 2022-04-22T18:58:54.250179+0000 mon.a (mon.0) 6223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:54 smithi115 bash[46934]: audit 2022-04-22T18:58:54.250179+0000 mon.a (mon.0) 6223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:58:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:55 smithi121 bash[36902]: cluster 2022-04-22T18:58:54.585996+0000 mgr.y (mgr.34107) 3205 : cluster [DBG] pgmap v4727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:55 smithi121 bash[36902]: audit 2022-04-22T18:58:54.855460+0000 mon.a (mon.0) 6224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:55 smithi121 bash[36902]: audit 2022-04-22T18:58:55.208880+0000 mon.a (mon.0) 6225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[45495]: cluster 2022-04-22T18:58:54.585996+0000 mgr.y (mgr.34107) 3205 : cluster [DBG] pgmap v4727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[45495]: audit 2022-04-22T18:58:54.855460+0000 mon.a (mon.0) 6224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[45495]: audit 2022-04-22T18:58:55.208880+0000 mon.a (mon.0) 6225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[46934]: cluster 2022-04-22T18:58:54.585996+0000 mgr.y (mgr.34107) 3205 : cluster [DBG] pgmap v4727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[46934]: audit 2022-04-22T18:58:54.855460+0000 mon.a (mon.0) 6224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:55 smithi115 bash[46934]: audit 2022-04-22T18:58:55.208880+0000 mon.a (mon.0) 6225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:58:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:57 smithi121 bash[36902]: cluster 2022-04-22T18:58:56.587151+0000 mgr.y (mgr.34107) 3206 : cluster [DBG] pgmap v4728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:57 smithi115 bash[45495]: cluster 2022-04-22T18:58:56.587151+0000 mgr.y (mgr.34107) 3206 : cluster [DBG] pgmap v4728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:57 smithi115 bash[46934]: cluster 2022-04-22T18:58:56.587151+0000 mgr.y (mgr.34107) 3206 : cluster [DBG] pgmap v4728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:58:59.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:58 smithi121 bash[36902]: audit 2022-04-22T18:58:58.180586+0000 mon.a (mon.0) 6226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:58 smithi115 bash[45495]: audit 2022-04-22T18:58:58.180586+0000 mon.a (mon.0) 6226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:58 smithi115 bash[46934]: audit 2022-04-22T18:58:58.180586+0000 mon.a (mon.0) 6226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:58:59.372 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:58:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:58:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (82m) 9m ago 95m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (95m) 4s ago 95m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (84m) 9m ago 96m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (83m) 4s ago 96m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (91m) 4s ago 101m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (91m) 9m ago 103m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (90m) 9m ago 103m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (90m) 4s ago 101m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:58:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (90m) 9m ago 101m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (83m) 9m ago 96m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (83m) 4s ago 96m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (88m) 9m ago 100m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (88m) 9m ago 100m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (87m) 9m ago 99m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (87m) 9m ago 99m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (86m) 4s ago 98m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (85m) 4s ago 98m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (85m) 4s ago 97m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:58:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (84m) 4s ago 97m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:58:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (82m) 4s ago 96m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:59:00.125 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:00.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:59:00.127 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:59:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:59 smithi121 bash[36902]: cluster 2022-04-22T18:58:58.588328+0000 mgr.y (mgr.34107) 3207 : cluster [DBG] pgmap v4729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:58:59 smithi121 bash[36902]: audit 2022-04-22T18:58:59.253118+0000 mon.a (mon.0) 6227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:59 smithi115 bash[45495]: cluster 2022-04-22T18:58:58.588328+0000 mgr.y (mgr.34107) 3207 : cluster [DBG] pgmap v4729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:58:59 smithi115 bash[45495]: audit 2022-04-22T18:58:59.253118+0000 mon.a (mon.0) 6227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:59 smithi115 bash[46934]: cluster 2022-04-22T18:58:58.588328+0000 mgr.y (mgr.34107) 3207 : cluster [DBG] pgmap v4729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:58:59 smithi115 bash[46934]: audit 2022-04-22T18:58:59.253118+0000 mon.a (mon.0) 6227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:00 smithi121 bash[36902]: audit 2022-04-22T18:58:59.348166+0000 mgr.y (mgr.34107) 3208 : audit [DBG] from='client.48706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:00 smithi121 bash[36902]: audit 2022-04-22T18:58:59.689561+0000 mgr.y (mgr.34107) 3209 : audit [DBG] from='client.38778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:00 smithi121 bash[36902]: audit 2022-04-22T18:59:00.116700+0000 mon.a (mon.0) 6228 : audit [DBG] from='client.? 172.21.15.115:0/3532703760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[45495]: audit 2022-04-22T18:58:59.348166+0000 mgr.y (mgr.34107) 3208 : audit [DBG] from='client.48706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[45495]: audit 2022-04-22T18:58:59.689561+0000 mgr.y (mgr.34107) 3209 : audit [DBG] from='client.38778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[45495]: audit 2022-04-22T18:59:00.116700+0000 mon.a (mon.0) 6228 : audit [DBG] from='client.? 172.21.15.115:0/3532703760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[46934]: audit 2022-04-22T18:58:59.348166+0000 mgr.y (mgr.34107) 3208 : audit [DBG] from='client.48706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[46934]: audit 2022-04-22T18:58:59.689561+0000 mgr.y (mgr.34107) 3209 : audit [DBG] from='client.38778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:00 smithi115 bash[46934]: audit 2022-04-22T18:59:00.116700+0000 mon.a (mon.0) 6228 : audit [DBG] from='client.? 172.21.15.115:0/3532703760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:01 smithi121 bash[36902]: cluster 2022-04-22T18:59:00.589422+0000 mgr.y (mgr.34107) 3210 : cluster [DBG] pgmap v4730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:01 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:01.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:02.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:01 smithi115 bash[45495]: cluster 2022-04-22T18:59:00.589422+0000 mgr.y (mgr.34107) 3210 : cluster [DBG] pgmap v4730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:01 smithi115 bash[46934]: cluster 2022-04-22T18:59:00.589422+0000 mgr.y (mgr.34107) 3210 : cluster [DBG] pgmap v4730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:03 smithi121 bash[36902]: cluster 2022-04-22T18:59:02.590619+0000 mgr.y (mgr.34107) 3211 : cluster [DBG] pgmap v4731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:03 smithi121 bash[36902]: audit 2022-04-22T18:59:03.184213+0000 mon.a (mon.0) 6229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:03 smithi115 bash[45495]: cluster 2022-04-22T18:59:02.590619+0000 mgr.y (mgr.34107) 3211 : cluster [DBG] pgmap v4731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:03 smithi115 bash[45495]: audit 2022-04-22T18:59:03.184213+0000 mon.a (mon.0) 6229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:03 smithi115 bash[46934]: cluster 2022-04-22T18:59:02.590619+0000 mgr.y (mgr.34107) 3211 : cluster [DBG] pgmap v4731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:03 smithi115 bash[46934]: audit 2022-04-22T18:59:03.184213+0000 mon.a (mon.0) 6229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:03 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:03.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:04 smithi121 bash[36902]: audit 2022-04-22T18:59:04.256042+0000 mon.a (mon.0) 6230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:04.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:04 smithi115 bash[45495]: audit 2022-04-22T18:59:04.256042+0000 mon.a (mon.0) 6230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:04.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:04 smithi115 bash[46934]: audit 2022-04-22T18:59:04.256042+0000 mon.a (mon.0) 6230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:05 smithi121 bash[36902]: cluster 2022-04-22T18:59:04.591826+0000 mgr.y (mgr.34107) 3212 : cluster [DBG] pgmap v4732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:05.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:05 smithi115 bash[45495]: cluster 2022-04-22T18:59:04.591826+0000 mgr.y (mgr.34107) 3212 : cluster [DBG] pgmap v4732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:05.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:05 smithi115 bash[46934]: cluster 2022-04-22T18:59:04.591826+0000 mgr.y (mgr.34107) 3212 : cluster [DBG] pgmap v4732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:07 smithi121 bash[36902]: cluster 2022-04-22T18:59:06.593004+0000 mgr.y (mgr.34107) 3213 : cluster [DBG] pgmap v4733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:07 smithi115 bash[45495]: cluster 2022-04-22T18:59:06.593004+0000 mgr.y (mgr.34107) 3213 : cluster [DBG] pgmap v4733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:07 smithi115 bash[46934]: cluster 2022-04-22T18:59:06.593004+0000 mgr.y (mgr.34107) 3213 : cluster [DBG] pgmap v4733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:08 smithi121 bash[36902]: audit 2022-04-22T18:59:08.187959+0000 mon.a (mon.0) 6231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:08 smithi115 bash[45495]: audit 2022-04-22T18:59:08.187959+0000 mon.a (mon.0) 6231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:08 smithi115 bash[46934]: audit 2022-04-22T18:59:08.187959+0000 mon.a (mon.0) 6231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:09 smithi121 bash[36902]: cluster 2022-04-22T18:59:08.594176+0000 mgr.y (mgr.34107) 3214 : cluster [DBG] pgmap v4734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:09 smithi121 bash[36902]: audit 2022-04-22T18:59:09.258757+0000 mon.a (mon.0) 6232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:09 smithi115 bash[45495]: cluster 2022-04-22T18:59:08.594176+0000 mgr.y (mgr.34107) 3214 : cluster [DBG] pgmap v4734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:09 smithi115 bash[45495]: audit 2022-04-22T18:59:09.258757+0000 mon.a (mon.0) 6232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:09 smithi115 bash[46934]: cluster 2022-04-22T18:59:08.594176+0000 mgr.y (mgr.34107) 3214 : cluster [DBG] pgmap v4734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:09 smithi115 bash[46934]: audit 2022-04-22T18:59:09.258757+0000 mon.a (mon.0) 6232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:11.974 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:11 smithi115 bash[45495]: cluster 2022-04-22T18:59:10.595234+0000 mgr.y (mgr.34107) 3215 : cluster [DBG] pgmap v4735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:11.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:11 smithi115 bash[46934]: cluster 2022-04-22T18:59:10.595234+0000 mgr.y (mgr.34107) 3215 : cluster [DBG] pgmap v4735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:11 smithi121 bash[36902]: cluster 2022-04-22T18:59:10.595234+0000 mgr.y (mgr.34107) 3215 : cluster [DBG] pgmap v4735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:11 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:11.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:13 smithi121 bash[36902]: cluster 2022-04-22T18:59:12.596352+0000 mgr.y (mgr.34107) 3216 : cluster [DBG] pgmap v4736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:13 smithi121 bash[36902]: audit 2022-04-22T18:59:13.191423+0000 mon.a (mon.0) 6233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:13 smithi115 bash[45495]: cluster 2022-04-22T18:59:12.596352+0000 mgr.y (mgr.34107) 3216 : cluster [DBG] pgmap v4736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:13 smithi115 bash[45495]: audit 2022-04-22T18:59:13.191423+0000 mon.a (mon.0) 6233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:13 smithi115 bash[46934]: cluster 2022-04-22T18:59:12.596352+0000 mgr.y (mgr.34107) 3216 : cluster [DBG] pgmap v4736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:13 smithi115 bash[46934]: audit 2022-04-22T18:59:13.191423+0000 mon.a (mon.0) 6233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:13 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:13.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:14 smithi121 bash[36902]: audit 2022-04-22T18:59:14.261952+0000 mon.a (mon.0) 6234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:14 smithi121 bash[36902]: audit 2022-04-22T18:59:14.306404+0000 mon.a (mon.0) 6235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:59:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:14 smithi121 bash[36902]: audit 2022-04-22T18:59:14.307019+0000 mon.a (mon.0) 6236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:59:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[45495]: audit 2022-04-22T18:59:14.261952+0000 mon.a (mon.0) 6234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[45495]: audit 2022-04-22T18:59:14.306404+0000 mon.a (mon.0) 6235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:59:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[45495]: audit 2022-04-22T18:59:14.307019+0000 mon.a (mon.0) 6236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:59:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[46934]: audit 2022-04-22T18:59:14.261952+0000 mon.a (mon.0) 6234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[46934]: audit 2022-04-22T18:59:14.306404+0000 mon.a (mon.0) 6235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T18:59:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:14 smithi115 bash[46934]: audit 2022-04-22T18:59:14.307019+0000 mon.a (mon.0) 6236 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T18:59:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:15 smithi121 bash[36902]: cluster 2022-04-22T18:59:14.597508+0000 mgr.y (mgr.34107) 3217 : cluster [DBG] pgmap v4737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:15 smithi115 bash[45495]: cluster 2022-04-22T18:59:14.597508+0000 mgr.y (mgr.34107) 3217 : cluster [DBG] pgmap v4737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:15 smithi115 bash[46934]: cluster 2022-04-22T18:59:14.597508+0000 mgr.y (mgr.34107) 3217 : cluster [DBG] pgmap v4737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:17 smithi121 bash[36902]: cluster 2022-04-22T18:59:16.598651+0000 mgr.y (mgr.34107) 3218 : cluster [DBG] pgmap v4738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:17 smithi115 bash[45495]: cluster 2022-04-22T18:59:16.598651+0000 mgr.y (mgr.34107) 3218 : cluster [DBG] pgmap v4738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:17 smithi115 bash[46934]: cluster 2022-04-22T18:59:16.598651+0000 mgr.y (mgr.34107) 3218 : cluster [DBG] pgmap v4738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:18 smithi121 bash[36902]: audit 2022-04-22T18:59:18.195246+0000 mon.a (mon.0) 6237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:18 smithi115 bash[45495]: audit 2022-04-22T18:59:18.195246+0000 mon.a (mon.0) 6237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:18 smithi115 bash[46934]: audit 2022-04-22T18:59:18.195246+0000 mon.a (mon.0) 6237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:19 smithi121 bash[36902]: cluster 2022-04-22T18:59:18.599962+0000 mgr.y (mgr.34107) 3219 : cluster [DBG] pgmap v4739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:19 smithi121 bash[36902]: audit 2022-04-22T18:59:19.264933+0000 mon.a (mon.0) 6238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:19 smithi115 bash[45495]: cluster 2022-04-22T18:59:18.599962+0000 mgr.y (mgr.34107) 3219 : cluster [DBG] pgmap v4739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:19 smithi115 bash[45495]: audit 2022-04-22T18:59:19.264933+0000 mon.a (mon.0) 6238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:19 smithi115 bash[46934]: cluster 2022-04-22T18:59:18.599962+0000 mgr.y (mgr.34107) 3219 : cluster [DBG] pgmap v4739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:19 smithi115 bash[46934]: audit 2022-04-22T18:59:19.264933+0000 mon.a (mon.0) 6238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:21.974 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:21 smithi115 bash[45495]: cluster 2022-04-22T18:59:20.601049+0000 mgr.y (mgr.34107) 3220 : cluster [DBG] pgmap v4740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:22.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:21 smithi115 bash[46934]: cluster 2022-04-22T18:59:20.601049+0000 mgr.y (mgr.34107) 3220 : cluster [DBG] pgmap v4740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:21 smithi121 bash[36902]: cluster 2022-04-22T18:59:20.601049+0000 mgr.y (mgr.34107) 3220 : cluster [DBG] pgmap v4740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:21 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:21.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:23 smithi121 bash[36902]: cluster 2022-04-22T18:59:22.602164+0000 mgr.y (mgr.34107) 3221 : cluster [DBG] pgmap v4741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:23 smithi121 bash[36902]: audit 2022-04-22T18:59:23.198893+0000 mon.a (mon.0) 6239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:23 smithi115 bash[45495]: cluster 2022-04-22T18:59:22.602164+0000 mgr.y (mgr.34107) 3221 : cluster [DBG] pgmap v4741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:23 smithi115 bash[45495]: audit 2022-04-22T18:59:23.198893+0000 mon.a (mon.0) 6239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:23 smithi115 bash[46934]: cluster 2022-04-22T18:59:22.602164+0000 mgr.y (mgr.34107) 3221 : cluster [DBG] pgmap v4741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:23 smithi115 bash[46934]: audit 2022-04-22T18:59:23.198893+0000 mon.a (mon.0) 6239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:23 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:23.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:24 smithi121 bash[36902]: audit 2022-04-22T18:59:24.267890+0000 mon.a (mon.0) 6240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:24.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:24 smithi115 bash[45495]: audit 2022-04-22T18:59:24.267890+0000 mon.a (mon.0) 6240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:24.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:24 smithi115 bash[46934]: audit 2022-04-22T18:59:24.267890+0000 mon.a (mon.0) 6240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:25 smithi121 bash[36902]: cluster 2022-04-22T18:59:24.603295+0000 mgr.y (mgr.34107) 3222 : cluster [DBG] pgmap v4742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:25 smithi115 bash[45495]: cluster 2022-04-22T18:59:24.603295+0000 mgr.y (mgr.34107) 3222 : cluster [DBG] pgmap v4742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:25 smithi115 bash[46934]: cluster 2022-04-22T18:59:24.603295+0000 mgr.y (mgr.34107) 3222 : cluster [DBG] pgmap v4742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:27 smithi121 bash[36902]: cluster 2022-04-22T18:59:26.604448+0000 mgr.y (mgr.34107) 3223 : cluster [DBG] pgmap v4743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:27 smithi115 bash[45495]: cluster 2022-04-22T18:59:26.604448+0000 mgr.y (mgr.34107) 3223 : cluster [DBG] pgmap v4743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:27 smithi115 bash[46934]: cluster 2022-04-22T18:59:26.604448+0000 mgr.y (mgr.34107) 3223 : cluster [DBG] pgmap v4743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:28 smithi121 bash[36902]: audit 2022-04-22T18:59:28.202673+0000 mon.a (mon.0) 6241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:28 smithi115 bash[45495]: audit 2022-04-22T18:59:28.202673+0000 mon.a (mon.0) 6241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:28 smithi115 bash[46934]: audit 2022-04-22T18:59:28.202673+0000 mon.a (mon.0) 6241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:29 smithi121 bash[36902]: cluster 2022-04-22T18:59:28.605548+0000 mgr.y (mgr.34107) 3224 : cluster [DBG] pgmap v4744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:29 smithi121 bash[36902]: audit 2022-04-22T18:59:29.271022+0000 mon.a (mon.0) 6242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:29 smithi115 bash[45495]: cluster 2022-04-22T18:59:28.605548+0000 mgr.y (mgr.34107) 3224 : cluster [DBG] pgmap v4744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:29 smithi115 bash[45495]: audit 2022-04-22T18:59:29.271022+0000 mon.a (mon.0) 6242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:29 smithi115 bash[46934]: cluster 2022-04-22T18:59:28.605548+0000 mgr.y (mgr.34107) 3224 : cluster [DBG] pgmap v4744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:29 smithi115 bash[46934]: audit 2022-04-22T18:59:29.271022+0000 mon.a (mon.0) 6242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:30.501 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (82m) 9m ago 96m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (96m) 35s ago 96m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (84m) 9m ago 97m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (84m) 35s ago 97m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (91m) 35s ago 101m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (92m) 9m ago 103m 476M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T18:59:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (91m) 9m ago 103m 248M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (90m) 35s ago 101m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (90m) 9m ago 102m 160M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (83m) 9m ago 96m 10.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (83m) 35s ago 96m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (89m) 9m ago 101m 698M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (88m) 9m ago 100m 776M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (88m) 9m ago 100m 525M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (87m) 9m ago 99m 479M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (87m) 35s ago 99m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T18:59:30.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (86m) 35s ago 98m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T18:59:30.847 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (85m) 35s ago 98m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T18:59:30.847 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (84m) 35s ago 97m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T18:59:30.847 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (83m) 35s ago 96m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T18:59:31.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T18:59:31.241 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T18:59:31.242 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T18:59:31.242 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T18:59:31.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[46934]: audit 2022-04-22T18:59:30.475980+0000 mgr.y (mgr.34107) 3225 : audit [DBG] from='client.48724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:31.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[46934]: cluster 2022-04-22T18:59:30.606457+0000 mgr.y (mgr.34107) 3226 : cluster [DBG] pgmap v4745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:31.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[46934]: audit 2022-04-22T18:59:30.814753+0000 mgr.y (mgr.34107) 3227 : audit [DBG] from='client.48730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:31.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[46934]: audit 2022-04-22T18:59:31.231130+0000 mon.a (mon.0) 6243 : audit [DBG] from='client.? 172.21.15.115:0/382953748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:31.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[45495]: audit 2022-04-22T18:59:30.475980+0000 mgr.y (mgr.34107) 3225 : audit [DBG] from='client.48724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:31.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[45495]: cluster 2022-04-22T18:59:30.606457+0000 mgr.y (mgr.34107) 3226 : cluster [DBG] pgmap v4745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:31.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[45495]: audit 2022-04-22T18:59:30.814753+0000 mgr.y (mgr.34107) 3227 : audit [DBG] from='client.48730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:31.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[45495]: audit 2022-04-22T18:59:31.231130+0000 mon.a (mon.0) 6243 : audit [DBG] from='client.? 172.21.15.115:0/382953748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:31 smithi121 bash[36902]: audit 2022-04-22T18:59:30.475980+0000 mgr.y (mgr.34107) 3225 : audit [DBG] from='client.48724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:31 smithi121 bash[36902]: cluster 2022-04-22T18:59:30.606457+0000 mgr.y (mgr.34107) 3226 : cluster [DBG] pgmap v4745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:31 smithi121 bash[36902]: audit 2022-04-22T18:59:30.814753+0000 mgr.y (mgr.34107) 3227 : audit [DBG] from='client.48730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T18:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:31 smithi121 bash[36902]: audit 2022-04-22T18:59:31.231130+0000 mon.a (mon.0) 6243 : audit [DBG] from='client.? 172.21.15.115:0/382953748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T18:59:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:31 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:31.967Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:33 smithi121 bash[36902]: cluster 2022-04-22T18:59:32.607614+0000 mgr.y (mgr.34107) 3228 : cluster [DBG] pgmap v4746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:33 smithi121 bash[36902]: audit 2022-04-22T18:59:33.206549+0000 mon.a (mon.0) 6244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:33 smithi115 bash[46934]: cluster 2022-04-22T18:59:32.607614+0000 mgr.y (mgr.34107) 3228 : cluster [DBG] pgmap v4746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:33 smithi115 bash[46934]: audit 2022-04-22T18:59:33.206549+0000 mon.a (mon.0) 6244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:33 smithi115 bash[45495]: cluster 2022-04-22T18:59:32.607614+0000 mgr.y (mgr.34107) 3228 : cluster [DBG] pgmap v4746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:33 smithi115 bash[45495]: audit 2022-04-22T18:59:33.206549+0000 mon.a (mon.0) 6244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:33 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:33.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:34 smithi121 bash[36902]: audit 2022-04-22T18:59:34.274183+0000 mon.a (mon.0) 6245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:34.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:34 smithi115 bash[45495]: audit 2022-04-22T18:59:34.274183+0000 mon.a (mon.0) 6245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:34 smithi115 bash[46934]: audit 2022-04-22T18:59:34.274183+0000 mon.a (mon.0) 6245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:35 smithi121 bash[36902]: cluster 2022-04-22T18:59:34.608724+0000 mgr.y (mgr.34107) 3229 : cluster [DBG] pgmap v4747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:35 smithi115 bash[45495]: cluster 2022-04-22T18:59:34.608724+0000 mgr.y (mgr.34107) 3229 : cluster [DBG] pgmap v4747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:35 smithi115 bash[46934]: cluster 2022-04-22T18:59:34.608724+0000 mgr.y (mgr.34107) 3229 : cluster [DBG] pgmap v4747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:37 smithi121 bash[36902]: cluster 2022-04-22T18:59:36.609768+0000 mgr.y (mgr.34107) 3230 : cluster [DBG] pgmap v4748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:37 smithi115 bash[45495]: cluster 2022-04-22T18:59:36.609768+0000 mgr.y (mgr.34107) 3230 : cluster [DBG] pgmap v4748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:37 smithi115 bash[46934]: cluster 2022-04-22T18:59:36.609768+0000 mgr.y (mgr.34107) 3230 : cluster [DBG] pgmap v4748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:38 smithi121 bash[36902]: audit 2022-04-22T18:59:38.210255+0000 mon.a (mon.0) 6246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:38 smithi115 bash[45495]: audit 2022-04-22T18:59:38.210255+0000 mon.a (mon.0) 6246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:39.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:38 smithi115 bash[46934]: audit 2022-04-22T18:59:38.210255+0000 mon.a (mon.0) 6246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:39 smithi121 bash[36902]: cluster 2022-04-22T18:59:38.610833+0000 mgr.y (mgr.34107) 3231 : cluster [DBG] pgmap v4749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:39 smithi121 bash[36902]: audit 2022-04-22T18:59:39.277352+0000 mon.a (mon.0) 6247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:39 smithi115 bash[45495]: cluster 2022-04-22T18:59:38.610833+0000 mgr.y (mgr.34107) 3231 : cluster [DBG] pgmap v4749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:39 smithi115 bash[45495]: audit 2022-04-22T18:59:39.277352+0000 mon.a (mon.0) 6247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:39 smithi115 bash[46934]: cluster 2022-04-22T18:59:38.610833+0000 mgr.y (mgr.34107) 3231 : cluster [DBG] pgmap v4749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:39 smithi115 bash[46934]: audit 2022-04-22T18:59:39.277352+0000 mon.a (mon.0) 6247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:41.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:41 smithi115 bash[45495]: cluster 2022-04-22T18:59:40.611944+0000 mgr.y (mgr.34107) 3232 : cluster [DBG] pgmap v4750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:41.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:41 smithi115 bash[46934]: cluster 2022-04-22T18:59:40.611944+0000 mgr.y (mgr.34107) 3232 : cluster [DBG] pgmap v4750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:41 smithi121 bash[36902]: cluster 2022-04-22T18:59:40.611944+0000 mgr.y (mgr.34107) 3232 : cluster [DBG] pgmap v4750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:41 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:41.966Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:43 smithi121 bash[36902]: cluster 2022-04-22T18:59:42.613052+0000 mgr.y (mgr.34107) 3233 : cluster [DBG] pgmap v4751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:43 smithi121 bash[36902]: audit 2022-04-22T18:59:43.214040+0000 mon.a (mon.0) 6248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:43 smithi115 bash[45495]: cluster 2022-04-22T18:59:42.613052+0000 mgr.y (mgr.34107) 3233 : cluster [DBG] pgmap v4751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:43 smithi115 bash[45495]: audit 2022-04-22T18:59:43.214040+0000 mon.a (mon.0) 6248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:43 smithi115 bash[46934]: cluster 2022-04-22T18:59:42.613052+0000 mgr.y (mgr.34107) 3233 : cluster [DBG] pgmap v4751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:43 smithi115 bash[46934]: audit 2022-04-22T18:59:43.214040+0000 mon.a (mon.0) 6248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:43 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:43.836Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:44 smithi121 bash[36902]: audit 2022-04-22T18:59:44.280617+0000 mon.a (mon.0) 6249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:44 smithi115 bash[45495]: audit 2022-04-22T18:59:44.280617+0000 mon.a (mon.0) 6249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:44 smithi115 bash[46934]: audit 2022-04-22T18:59:44.280617+0000 mon.a (mon.0) 6249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:45 smithi121 bash[36902]: cluster 2022-04-22T18:59:44.614272+0000 mgr.y (mgr.34107) 3234 : cluster [DBG] pgmap v4752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:45 smithi115 bash[45495]: cluster 2022-04-22T18:59:44.614272+0000 mgr.y (mgr.34107) 3234 : cluster [DBG] pgmap v4752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:45 smithi115 bash[46934]: cluster 2022-04-22T18:59:44.614272+0000 mgr.y (mgr.34107) 3234 : cluster [DBG] pgmap v4752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:47 smithi121 bash[36902]: cluster 2022-04-22T18:59:46.615432+0000 mgr.y (mgr.34107) 3235 : cluster [DBG] pgmap v4753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:47 smithi115 bash[45495]: cluster 2022-04-22T18:59:46.615432+0000 mgr.y (mgr.34107) 3235 : cluster [DBG] pgmap v4753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:47 smithi115 bash[46934]: cluster 2022-04-22T18:59:46.615432+0000 mgr.y (mgr.34107) 3235 : cluster [DBG] pgmap v4753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:48 smithi121 bash[36902]: audit 2022-04-22T18:59:48.218843+0000 mon.a (mon.0) 6250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:48 smithi115 bash[45495]: audit 2022-04-22T18:59:48.218843+0000 mon.a (mon.0) 6250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:48 smithi115 bash[46934]: audit 2022-04-22T18:59:48.218843+0000 mon.a (mon.0) 6250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:49 smithi121 bash[36902]: cluster 2022-04-22T18:59:48.616683+0000 mgr.y (mgr.34107) 3236 : cluster [DBG] pgmap v4754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:49 smithi121 bash[36902]: audit 2022-04-22T18:59:49.283447+0000 mon.a (mon.0) 6251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:49 smithi115 bash[46934]: cluster 2022-04-22T18:59:48.616683+0000 mgr.y (mgr.34107) 3236 : cluster [DBG] pgmap v4754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:49 smithi115 bash[46934]: audit 2022-04-22T18:59:49.283447+0000 mon.a (mon.0) 6251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:49 smithi115 bash[45495]: cluster 2022-04-22T18:59:48.616683+0000 mgr.y (mgr.34107) 3236 : cluster [DBG] pgmap v4754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:49 smithi115 bash[45495]: audit 2022-04-22T18:59:49.283447+0000 mon.a (mon.0) 6251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:51.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:51 smithi115 bash[45495]: cluster 2022-04-22T18:59:50.617699+0000 mgr.y (mgr.34107) 3237 : cluster [DBG] pgmap v4755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:51.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:51 smithi115 bash[46934]: cluster 2022-04-22T18:59:50.617699+0000 mgr.y (mgr.34107) 3237 : cluster [DBG] pgmap v4755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:51 smithi121 bash[36902]: cluster 2022-04-22T18:59:50.617699+0000 mgr.y (mgr.34107) 3237 : cluster [DBG] pgmap v4755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:51 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:51.968Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:53 smithi121 bash[36902]: cluster 2022-04-22T18:59:52.618960+0000 mgr.y (mgr.34107) 3238 : cluster [DBG] pgmap v4756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:53 smithi121 bash[36902]: audit 2022-04-22T18:59:53.222506+0000 mon.a (mon.0) 6252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:53 smithi115 bash[46934]: cluster 2022-04-22T18:59:52.618960+0000 mgr.y (mgr.34107) 3238 : cluster [DBG] pgmap v4756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:53 smithi115 bash[46934]: audit 2022-04-22T18:59:53.222506+0000 mon.a (mon.0) 6252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:53 smithi115 bash[45495]: cluster 2022-04-22T18:59:52.618960+0000 mgr.y (mgr.34107) 3238 : cluster [DBG] pgmap v4756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:53 smithi115 bash[45495]: audit 2022-04-22T18:59:53.222506+0000 mon.a (mon.0) 6252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 18:59:53 smithi115 bash[60466]: level=error ts=2022-04-22T18:59:53.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T18:59:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:54 smithi121 bash[36902]: audit 2022-04-22T18:59:54.286548+0000 mon.a (mon.0) 6253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:54 smithi115 bash[45495]: audit 2022-04-22T18:59:54.286548+0000 mon.a (mon.0) 6253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:54 smithi115 bash[46934]: audit 2022-04-22T18:59:54.286548+0000 mon.a (mon.0) 6253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:55 smithi121 bash[36902]: cluster 2022-04-22T18:59:54.620153+0000 mgr.y (mgr.34107) 3239 : cluster [DBG] pgmap v4757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:55 smithi121 bash[36902]: audit 2022-04-22T18:59:55.220080+0000 mon.a (mon.0) 6254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:59:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:55 smithi121 bash[36902]: audit 2022-04-22T18:59:55.223024+0000 mon.a (mon.0) 6255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:59:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:55 smithi121 bash[36902]: audit 2022-04-22T18:59:55.224560+0000 mon.a (mon.0) 6256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:59:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[46934]: cluster 2022-04-22T18:59:54.620153+0000 mgr.y (mgr.34107) 3239 : cluster [DBG] pgmap v4757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[46934]: audit 2022-04-22T18:59:55.220080+0000 mon.a (mon.0) 6254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:59:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[46934]: audit 2022-04-22T18:59:55.223024+0000 mon.a (mon.0) 6255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:59:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[46934]: audit 2022-04-22T18:59:55.224560+0000 mon.a (mon.0) 6256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:59:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[45495]: cluster 2022-04-22T18:59:54.620153+0000 mgr.y (mgr.34107) 3239 : cluster [DBG] pgmap v4757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[45495]: audit 2022-04-22T18:59:55.220080+0000 mon.a (mon.0) 6254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T18:59:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[45495]: audit 2022-04-22T18:59:55.223024+0000 mon.a (mon.0) 6255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T18:59:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:55 smithi115 bash[45495]: audit 2022-04-22T18:59:55.224560+0000 mon.a (mon.0) 6256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T18:59:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:56 smithi121 bash[36902]: audit 2022-04-22T18:59:55.561263+0000 mon.a (mon.0) 6257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:56 smithi121 bash[36902]: audit 2022-04-22T18:59:55.569201+0000 mon.a (mon.0) 6258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:56 smithi115 bash[45495]: audit 2022-04-22T18:59:55.561263+0000 mon.a (mon.0) 6257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:56 smithi115 bash[45495]: audit 2022-04-22T18:59:55.569201+0000 mon.a (mon.0) 6258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:56 smithi115 bash[46934]: audit 2022-04-22T18:59:55.561263+0000 mon.a (mon.0) 6257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:57.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:56 smithi115 bash[46934]: audit 2022-04-22T18:59:55.569201+0000 mon.a (mon.0) 6258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T18:59:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:57 smithi121 bash[36902]: cluster 2022-04-22T18:59:56.621374+0000 mgr.y (mgr.34107) 3240 : cluster [DBG] pgmap v4758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:57 smithi115 bash[45495]: cluster 2022-04-22T18:59:56.621374+0000 mgr.y (mgr.34107) 3240 : cluster [DBG] pgmap v4758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:57 smithi115 bash[46934]: cluster 2022-04-22T18:59:56.621374+0000 mgr.y (mgr.34107) 3240 : cluster [DBG] pgmap v4758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:58 smithi121 bash[36902]: audit 2022-04-22T18:59:58.226285+0000 mon.a (mon.0) 6259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:58 smithi115 bash[45495]: audit 2022-04-22T18:59:58.226285+0000 mon.a (mon.0) 6259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:58 smithi115 bash[46934]: audit 2022-04-22T18:59:58.226285+0000 mon.a (mon.0) 6259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T18:59:59.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:59 smithi115 bash[45495]: cluster 2022-04-22T18:59:58.622671+0000 mgr.y (mgr.34107) 3241 : cluster [DBG] pgmap v4759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:59.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 18:59:59 smithi115 bash[45495]: audit 2022-04-22T18:59:59.289701+0000 mon.a (mon.0) 6260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T18:59:59.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:59 smithi115 bash[46934]: cluster 2022-04-22T18:59:58.622671+0000 mgr.y (mgr.34107) 3241 : cluster [DBG] pgmap v4759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T18:59:59.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 18:59:59 smithi115 bash[46934]: audit 2022-04-22T18:59:59.289701+0000 mon.a (mon.0) 6260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:59 smithi121 bash[36902]: cluster 2022-04-22T18:59:58.622671+0000 mgr.y (mgr.34107) 3241 : cluster [DBG] pgmap v4759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 18:59:59 smithi121 bash[36902]: audit 2022-04-22T18:59:59.289701+0000 mon.a (mon.0) 6260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[45495]: audit 2022-04-22T18:59:59.758708+0000 mon.a (mon.0) 6261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[45495]: cluster 2022-04-22T19:00:00.000151+0000 mon.a (mon.0) 6262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:00:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[45495]: audit 2022-04-22T19:00:00.097049+0000 mon.a (mon.0) 6263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[46934]: audit 2022-04-22T18:59:59.758708+0000 mon.a (mon.0) 6261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[46934]: cluster 2022-04-22T19:00:00.000151+0000 mon.a (mon.0) 6262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:00:01.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:00 smithi115 bash[46934]: audit 2022-04-22T19:00:00.097049+0000 mon.a (mon.0) 6263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:00 smithi121 bash[36902]: audit 2022-04-22T18:59:59.758708+0000 mon.a (mon.0) 6261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:00 smithi121 bash[36902]: cluster 2022-04-22T19:00:00.000151+0000 mon.a (mon.0) 6262 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:00:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:00 smithi121 bash[36902]: audit 2022-04-22T19:00:00.097049+0000 mon.a (mon.0) 6263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:00:01.618 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:00:01.964 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:00:01.964 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (83m) 2s ago 96m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:00:01.964 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (96m) 67s ago 96m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (85m) 2s ago 97m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (84m) 67s ago 97m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (92m) 67s ago 102m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (92m) 2s ago 104m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (91m) 2s ago 104m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (91m) 67s ago 102m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (91m) 2s ago 102m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (84m) 2s ago 97m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:00:01.965 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (84m) 67s ago 97m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (89m) 2s ago 101m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (89m) 2s ago 101m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (88m) 2s ago 100m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (88m) 2s ago 100m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (87m) 67s ago 99m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (86m) 67s ago 99m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (86m) 67s ago 98m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (85m) 67s ago 98m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:00:01.966 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (83m) 67s ago 97m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:00:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:02 smithi115 bash[46934]: cluster 2022-04-22T19:00:00.623849+0000 mgr.y (mgr.34107) 3242 : cluster [DBG] pgmap v4760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:01.966Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:02 smithi115 bash[45495]: cluster 2022-04-22T19:00:00.623849+0000 mgr.y (mgr.34107) 3242 : cluster [DBG] pgmap v4760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:02.370 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:00:02.370 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:00:02.370 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:00:02.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:02.370 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:00:02.371 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:00:02.372 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:00:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:02 smithi121 bash[36902]: cluster 2022-04-22T19:00:00.623849+0000 mgr.y (mgr.34107) 3242 : cluster [DBG] pgmap v4760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:03 smithi121 bash[36902]: audit 2022-04-22T19:00:01.593571+0000 mgr.y (mgr.34107) 3243 : audit [DBG] from='client.48742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:03 smithi121 bash[36902]: audit 2022-04-22T19:00:01.935411+0000 mgr.y (mgr.34107) 3244 : audit [DBG] from='client.38814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:03 smithi121 bash[36902]: audit 2022-04-22T19:00:02.361137+0000 mon.c (mon.1) 171 : audit [DBG] from='client.? 172.21.15.115:0/2675086425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[45495]: audit 2022-04-22T19:00:01.593571+0000 mgr.y (mgr.34107) 3243 : audit [DBG] from='client.48742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[45495]: audit 2022-04-22T19:00:01.935411+0000 mgr.y (mgr.34107) 3244 : audit [DBG] from='client.38814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[45495]: audit 2022-04-22T19:00:02.361137+0000 mon.c (mon.1) 171 : audit [DBG] from='client.? 172.21.15.115:0/2675086425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[46934]: audit 2022-04-22T19:00:01.593571+0000 mgr.y (mgr.34107) 3243 : audit [DBG] from='client.48742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[46934]: audit 2022-04-22T19:00:01.935411+0000 mgr.y (mgr.34107) 3244 : audit [DBG] from='client.38814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[46934]: audit 2022-04-22T19:00:02.361137+0000 mon.c (mon.1) 171 : audit [DBG] from='client.? 172.21.15.115:0/2675086425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:04.112 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:03.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:04 smithi121 bash[36902]: cluster 2022-04-22T19:00:02.624932+0000 mgr.y (mgr.34107) 3245 : cluster [DBG] pgmap v4761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:04 smithi121 bash[36902]: audit 2022-04-22T19:00:03.229981+0000 mon.a (mon.0) 6264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:04 smithi115 bash[45495]: cluster 2022-04-22T19:00:02.624932+0000 mgr.y (mgr.34107) 3245 : cluster [DBG] pgmap v4761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:04 smithi115 bash[45495]: audit 2022-04-22T19:00:03.229981+0000 mon.a (mon.0) 6264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:04 smithi115 bash[46934]: cluster 2022-04-22T19:00:02.624932+0000 mgr.y (mgr.34107) 3245 : cluster [DBG] pgmap v4761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:04 smithi115 bash[46934]: audit 2022-04-22T19:00:03.229981+0000 mon.a (mon.0) 6264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:05 smithi121 bash[36902]: audit 2022-04-22T19:00:04.292095+0000 mon.a (mon.0) 6265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:05 smithi115 bash[45495]: audit 2022-04-22T19:00:04.292095+0000 mon.a (mon.0) 6265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:05 smithi115 bash[46934]: audit 2022-04-22T19:00:04.292095+0000 mon.a (mon.0) 6265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:06 smithi121 bash[36902]: cluster 2022-04-22T19:00:04.626033+0000 mgr.y (mgr.34107) 3246 : cluster [DBG] pgmap v4762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:06 smithi115 bash[45495]: cluster 2022-04-22T19:00:04.626033+0000 mgr.y (mgr.34107) 3246 : cluster [DBG] pgmap v4762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:06 smithi115 bash[46934]: cluster 2022-04-22T19:00:04.626033+0000 mgr.y (mgr.34107) 3246 : cluster [DBG] pgmap v4762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:08 smithi121 bash[36902]: cluster 2022-04-22T19:00:06.627172+0000 mgr.y (mgr.34107) 3247 : cluster [DBG] pgmap v4763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:08 smithi115 bash[45495]: cluster 2022-04-22T19:00:06.627172+0000 mgr.y (mgr.34107) 3247 : cluster [DBG] pgmap v4763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:08 smithi115 bash[46934]: cluster 2022-04-22T19:00:06.627172+0000 mgr.y (mgr.34107) 3247 : cluster [DBG] pgmap v4763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:09 smithi121 bash[36902]: audit 2022-04-22T19:00:08.233730+0000 mon.a (mon.0) 6266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:09 smithi115 bash[45495]: audit 2022-04-22T19:00:08.233730+0000 mon.a (mon.0) 6266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:09 smithi115 bash[46934]: audit 2022-04-22T19:00:08.233730+0000 mon.a (mon.0) 6266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:10 smithi121 bash[36902]: cluster 2022-04-22T19:00:08.628416+0000 mgr.y (mgr.34107) 3248 : cluster [DBG] pgmap v4764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:10 smithi121 bash[36902]: audit 2022-04-22T19:00:09.295502+0000 mon.a (mon.0) 6267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:10 smithi115 bash[45495]: cluster 2022-04-22T19:00:08.628416+0000 mgr.y (mgr.34107) 3248 : cluster [DBG] pgmap v4764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:10 smithi115 bash[45495]: audit 2022-04-22T19:00:09.295502+0000 mon.a (mon.0) 6267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:10.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:10 smithi115 bash[46934]: cluster 2022-04-22T19:00:08.628416+0000 mgr.y (mgr.34107) 3248 : cluster [DBG] pgmap v4764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:10.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:10 smithi115 bash[46934]: audit 2022-04-22T19:00:09.295502+0000 mon.a (mon.0) 6267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:12.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:12 smithi115 bash[45495]: cluster 2022-04-22T19:00:10.629502+0000 mgr.y (mgr.34107) 3249 : cluster [DBG] pgmap v4765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:11.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:12 smithi115 bash[46934]: cluster 2022-04-22T19:00:10.629502+0000 mgr.y (mgr.34107) 3249 : cluster [DBG] pgmap v4765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:12 smithi121 bash[36902]: cluster 2022-04-22T19:00:10.629502+0000 mgr.y (mgr.34107) 3249 : cluster [DBG] pgmap v4765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:13 smithi121 bash[36902]: cluster 2022-04-22T19:00:12.630587+0000 mgr.y (mgr.34107) 3250 : cluster [DBG] pgmap v4766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:13 smithi121 bash[36902]: audit 2022-04-22T19:00:13.238140+0000 mon.a (mon.0) 6268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:13 smithi115 bash[45495]: cluster 2022-04-22T19:00:12.630587+0000 mgr.y (mgr.34107) 3250 : cluster [DBG] pgmap v4766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:13 smithi115 bash[45495]: audit 2022-04-22T19:00:13.238140+0000 mon.a (mon.0) 6268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:13 smithi115 bash[46934]: cluster 2022-04-22T19:00:12.630587+0000 mgr.y (mgr.34107) 3250 : cluster [DBG] pgmap v4766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:13 smithi115 bash[46934]: audit 2022-04-22T19:00:13.238140+0000 mon.a (mon.0) 6268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:13.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:14 smithi121 bash[36902]: audit 2022-04-22T19:00:14.298674+0000 mon.a (mon.0) 6269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:14.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:14 smithi121 bash[36902]: audit 2022-04-22T19:00:14.319356+0000 mon.a (mon.0) 6270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:00:14.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:14 smithi121 bash[36902]: audit 2022-04-22T19:00:14.325457+0000 mon.a (mon.0) 6271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:00:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[45495]: audit 2022-04-22T19:00:14.298674+0000 mon.a (mon.0) 6269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[45495]: audit 2022-04-22T19:00:14.319356+0000 mon.a (mon.0) 6270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:00:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[45495]: audit 2022-04-22T19:00:14.325457+0000 mon.a (mon.0) 6271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:00:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[46934]: audit 2022-04-22T19:00:14.298674+0000 mon.a (mon.0) 6269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[46934]: audit 2022-04-22T19:00:14.319356+0000 mon.a (mon.0) 6270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:00:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:14 smithi115 bash[46934]: audit 2022-04-22T19:00:14.325457+0000 mon.a (mon.0) 6271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:00:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:15 smithi121 bash[36902]: cluster 2022-04-22T19:00:14.631736+0000 mgr.y (mgr.34107) 3251 : cluster [DBG] pgmap v4767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:15 smithi115 bash[45495]: cluster 2022-04-22T19:00:14.631736+0000 mgr.y (mgr.34107) 3251 : cluster [DBG] pgmap v4767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:15 smithi115 bash[46934]: cluster 2022-04-22T19:00:14.631736+0000 mgr.y (mgr.34107) 3251 : cluster [DBG] pgmap v4767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:17 smithi121 bash[36902]: cluster 2022-04-22T19:00:16.632780+0000 mgr.y (mgr.34107) 3252 : cluster [DBG] pgmap v4768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:17 smithi115 bash[46934]: cluster 2022-04-22T19:00:16.632780+0000 mgr.y (mgr.34107) 3252 : cluster [DBG] pgmap v4768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:17 smithi115 bash[45495]: cluster 2022-04-22T19:00:16.632780+0000 mgr.y (mgr.34107) 3252 : cluster [DBG] pgmap v4768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:18 smithi121 bash[36902]: audit 2022-04-22T19:00:18.241653+0000 mon.a (mon.0) 6272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:18 smithi115 bash[45495]: audit 2022-04-22T19:00:18.241653+0000 mon.a (mon.0) 6272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:18 smithi115 bash[46934]: audit 2022-04-22T19:00:18.241653+0000 mon.a (mon.0) 6272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:19 smithi121 bash[36902]: cluster 2022-04-22T19:00:18.633911+0000 mgr.y (mgr.34107) 3253 : cluster [DBG] pgmap v4769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:19 smithi121 bash[36902]: audit 2022-04-22T19:00:19.301575+0000 mon.a (mon.0) 6273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:19 smithi115 bash[45495]: cluster 2022-04-22T19:00:18.633911+0000 mgr.y (mgr.34107) 3253 : cluster [DBG] pgmap v4769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:19 smithi115 bash[45495]: audit 2022-04-22T19:00:19.301575+0000 mon.a (mon.0) 6273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:19 smithi115 bash[46934]: cluster 2022-04-22T19:00:18.633911+0000 mgr.y (mgr.34107) 3253 : cluster [DBG] pgmap v4769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:19 smithi115 bash[46934]: audit 2022-04-22T19:00:19.301575+0000 mon.a (mon.0) 6273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:21.979 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:21 smithi115 bash[45495]: cluster 2022-04-22T19:00:20.634989+0000 mgr.y (mgr.34107) 3254 : cluster [DBG] pgmap v4770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:21.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:21 smithi115 bash[46934]: cluster 2022-04-22T19:00:20.634989+0000 mgr.y (mgr.34107) 3254 : cluster [DBG] pgmap v4770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:21 smithi121 bash[36902]: cluster 2022-04-22T19:00:20.634989+0000 mgr.y (mgr.34107) 3254 : cluster [DBG] pgmap v4770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:22.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:21.970Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:23 smithi121 bash[36902]: cluster 2022-04-22T19:00:22.636082+0000 mgr.y (mgr.34107) 3255 : cluster [DBG] pgmap v4771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:23 smithi115 bash[45495]: cluster 2022-04-22T19:00:22.636082+0000 mgr.y (mgr.34107) 3255 : cluster [DBG] pgmap v4771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:23 smithi115 bash[46934]: cluster 2022-04-22T19:00:22.636082+0000 mgr.y (mgr.34107) 3255 : cluster [DBG] pgmap v4771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:24.262 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:23.839Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:24 smithi121 bash[36902]: audit 2022-04-22T19:00:23.244949+0000 mon.a (mon.0) 6274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:24 smithi115 bash[45495]: audit 2022-04-22T19:00:23.244949+0000 mon.a (mon.0) 6274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:24 smithi115 bash[46934]: audit 2022-04-22T19:00:23.244949+0000 mon.a (mon.0) 6274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:25 smithi115 bash[45495]: audit 2022-04-22T19:00:24.304897+0000 mon.a (mon.0) 6275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:25.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:25 smithi115 bash[45495]: cluster 2022-04-22T19:00:24.637119+0000 mgr.y (mgr.34107) 3256 : cluster [DBG] pgmap v4772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:25.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:25 smithi115 bash[46934]: audit 2022-04-22T19:00:24.304897+0000 mon.a (mon.0) 6275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:25.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:25 smithi115 bash[46934]: cluster 2022-04-22T19:00:24.637119+0000 mgr.y (mgr.34107) 3256 : cluster [DBG] pgmap v4772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:25.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:25 smithi121 bash[36902]: audit 2022-04-22T19:00:24.304897+0000 mon.a (mon.0) 6275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:25.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:25 smithi121 bash[36902]: cluster 2022-04-22T19:00:24.637119+0000 mgr.y (mgr.34107) 3256 : cluster [DBG] pgmap v4772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:27 smithi121 bash[36902]: cluster 2022-04-22T19:00:26.638144+0000 mgr.y (mgr.34107) 3257 : cluster [DBG] pgmap v4773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:27 smithi115 bash[45495]: cluster 2022-04-22T19:00:26.638144+0000 mgr.y (mgr.34107) 3257 : cluster [DBG] pgmap v4773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:27 smithi115 bash[46934]: cluster 2022-04-22T19:00:26.638144+0000 mgr.y (mgr.34107) 3257 : cluster [DBG] pgmap v4773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:29.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:28 smithi121 bash[36902]: audit 2022-04-22T19:00:28.248011+0000 mon.a (mon.0) 6276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:28 smithi115 bash[45495]: audit 2022-04-22T19:00:28.248011+0000 mon.a (mon.0) 6276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:28 smithi115 bash[46934]: audit 2022-04-22T19:00:28.248011+0000 mon.a (mon.0) 6276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:29 smithi121 bash[36902]: cluster 2022-04-22T19:00:28.639230+0000 mgr.y (mgr.34107) 3258 : cluster [DBG] pgmap v4774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:30.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:29 smithi121 bash[36902]: audit 2022-04-22T19:00:29.308054+0000 mon.a (mon.0) 6277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:29 smithi115 bash[45495]: cluster 2022-04-22T19:00:28.639230+0000 mgr.y (mgr.34107) 3258 : cluster [DBG] pgmap v4774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:29 smithi115 bash[45495]: audit 2022-04-22T19:00:29.308054+0000 mon.a (mon.0) 6277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:29 smithi115 bash[46934]: cluster 2022-04-22T19:00:28.639230+0000 mgr.y (mgr.34107) 3258 : cluster [DBG] pgmap v4774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:30.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:29 smithi115 bash[46934]: audit 2022-04-22T19:00:29.308054+0000 mon.a (mon.0) 6277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:31.979 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:31 smithi115 bash[45495]: cluster 2022-04-22T19:00:30.640307+0000 mgr.y (mgr.34107) 3259 : cluster [DBG] pgmap v4775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:31.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:31 smithi115 bash[46934]: cluster 2022-04-22T19:00:30.640307+0000 mgr.y (mgr.34107) 3259 : cluster [DBG] pgmap v4775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:31 smithi121 bash[36902]: cluster 2022-04-22T19:00:30.640307+0000 mgr.y (mgr.34107) 3259 : cluster [DBG] pgmap v4775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:32.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:31.970Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:32.742 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:00:33.087 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:00:33.087 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (83m) 33s ago 97m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:00:33.087 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (97m) 98s ago 97m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:00:33.087 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (85m) 33s ago 98m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (85m) 98s ago 98m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (92m) 98s ago 102m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (93m) 33s ago 104m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (92m) 33s ago 105m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (91m) 98s ago 103m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (91m) 33s ago 103m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (84m) 33s ago 97m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (84m) 98s ago 97m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (90m) 33s ago 102m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:00:33.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (89m) 33s ago 101m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (89m) 33s ago 101m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (88m) 33s ago 100m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (88m) 98s ago 100m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (87m) 98s ago 99m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (86m) 98s ago 99m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (85m) 98s ago 98m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:00:33.089 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (84m) 98s ago 97m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:00:33.494 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:00:33.494 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:00:33.494 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:00:33.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:33.495 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:00:33.496 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:00:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[45495]: cluster 2022-04-22T19:00:32.641355+0000 mgr.y (mgr.34107) 3260 : cluster [DBG] pgmap v4776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[45495]: audit 2022-04-22T19:00:32.717851+0000 mgr.y (mgr.34107) 3261 : audit [DBG] from='client.38826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[45495]: audit 2022-04-22T19:00:33.058946+0000 mgr.y (mgr.34107) 3262 : audit [DBG] from='client.48766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[46934]: cluster 2022-04-22T19:00:32.641355+0000 mgr.y (mgr.34107) 3260 : cluster [DBG] pgmap v4776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[46934]: audit 2022-04-22T19:00:32.717851+0000 mgr.y (mgr.34107) 3261 : audit [DBG] from='client.38826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[46934]: audit 2022-04-22T19:00:33.058946+0000 mgr.y (mgr.34107) 3262 : audit [DBG] from='client.48766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:33 smithi121 bash[36902]: cluster 2022-04-22T19:00:32.641355+0000 mgr.y (mgr.34107) 3260 : cluster [DBG] pgmap v4776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:33 smithi121 bash[36902]: audit 2022-04-22T19:00:32.717851+0000 mgr.y (mgr.34107) 3261 : audit [DBG] from='client.38826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:33 smithi121 bash[36902]: audit 2022-04-22T19:00:33.058946+0000 mgr.y (mgr.34107) 3262 : audit [DBG] from='client.48766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:00:34.267 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:33.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:34 smithi115 bash[45495]: audit 2022-04-22T19:00:33.255882+0000 mon.a (mon.0) 6278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:34 smithi115 bash[45495]: audit 2022-04-22T19:00:33.484545+0000 mon.c (mon.1) 172 : audit [DBG] from='client.? 172.21.15.115:0/665198930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:34 smithi115 bash[46934]: audit 2022-04-22T19:00:33.255882+0000 mon.a (mon.0) 6278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:34 smithi115 bash[46934]: audit 2022-04-22T19:00:33.484545+0000 mon.c (mon.1) 172 : audit [DBG] from='client.? 172.21.15.115:0/665198930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:34 smithi121 bash[36902]: audit 2022-04-22T19:00:33.255882+0000 mon.a (mon.0) 6278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:00:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:34 smithi121 bash[36902]: audit 2022-04-22T19:00:33.484545+0000 mon.c (mon.1) 172 : audit [DBG] from='client.? 172.21.15.115:0/665198930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:00:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:35 smithi115 bash[45495]: audit 2022-04-22T19:00:34.311807+0000 mon.a (mon.0) 6279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:35 smithi115 bash[45495]: cluster 2022-04-22T19:00:34.642424+0000 mgr.y (mgr.34107) 3263 : cluster [DBG] pgmap v4777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:35 smithi115 bash[46934]: audit 2022-04-22T19:00:34.311807+0000 mon.a (mon.0) 6279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:35 smithi115 bash[46934]: cluster 2022-04-22T19:00:34.642424+0000 mgr.y (mgr.34107) 3263 : cluster [DBG] pgmap v4777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:35 smithi121 bash[36902]: audit 2022-04-22T19:00:34.311807+0000 mon.a (mon.0) 6279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:35 smithi121 bash[36902]: cluster 2022-04-22T19:00:34.642424+0000 mgr.y (mgr.34107) 3263 : cluster [DBG] pgmap v4777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:37 smithi121 bash[36902]: cluster 2022-04-22T19:00:36.643505+0000 mgr.y (mgr.34107) 3264 : cluster [DBG] pgmap v4778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:37 smithi115 bash[45495]: cluster 2022-04-22T19:00:36.643505+0000 mgr.y (mgr.34107) 3264 : cluster [DBG] pgmap v4778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:37 smithi115 bash[46934]: cluster 2022-04-22T19:00:36.643505+0000 mgr.y (mgr.34107) 3264 : cluster [DBG] pgmap v4778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:39 smithi121 bash[36902]: cluster 2022-04-22T19:00:38.644585+0000 mgr.y (mgr.34107) 3265 : cluster [DBG] pgmap v4779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:39 smithi121 bash[36902]: audit 2022-04-22T19:00:39.315160+0000 mon.a (mon.0) 6280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:39 smithi115 bash[45495]: cluster 2022-04-22T19:00:38.644585+0000 mgr.y (mgr.34107) 3265 : cluster [DBG] pgmap v4779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:39 smithi115 bash[45495]: audit 2022-04-22T19:00:39.315160+0000 mon.a (mon.0) 6280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:39 smithi115 bash[46934]: cluster 2022-04-22T19:00:38.644585+0000 mgr.y (mgr.34107) 3265 : cluster [DBG] pgmap v4779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:39 smithi115 bash[46934]: audit 2022-04-22T19:00:39.315160+0000 mon.a (mon.0) 6280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:41.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:41 smithi115 bash[46934]: cluster 2022-04-22T19:00:40.645738+0000 mgr.y (mgr.34107) 3266 : cluster [DBG] pgmap v4780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:41.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:41 smithi115 bash[45495]: cluster 2022-04-22T19:00:40.645738+0000 mgr.y (mgr.34107) 3266 : cluster [DBG] pgmap v4780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:41 smithi121 bash[36902]: cluster 2022-04-22T19:00:40.645738+0000 mgr.y (mgr.34107) 3266 : cluster [DBG] pgmap v4780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:42.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:41.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:43 smithi115 bash[46934]: cluster 2022-04-22T19:00:42.646921+0000 mgr.y (mgr.34107) 3267 : cluster [DBG] pgmap v4781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:43.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:43 smithi115 bash[45495]: cluster 2022-04-22T19:00:42.646921+0000 mgr.y (mgr.34107) 3267 : cluster [DBG] pgmap v4781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:43 smithi121 bash[36902]: cluster 2022-04-22T19:00:42.646921+0000 mgr.y (mgr.34107) 3267 : cluster [DBG] pgmap v4781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:43.841Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:44 smithi121 bash[36902]: audit 2022-04-22T19:00:44.318393+0000 mon.a (mon.0) 6281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:44 smithi115 bash[46934]: audit 2022-04-22T19:00:44.318393+0000 mon.a (mon.0) 6281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:44 smithi115 bash[45495]: audit 2022-04-22T19:00:44.318393+0000 mon.a (mon.0) 6281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:45 smithi121 bash[36902]: cluster 2022-04-22T19:00:44.648051+0000 mgr.y (mgr.34107) 3268 : cluster [DBG] pgmap v4782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:45 smithi115 bash[45495]: cluster 2022-04-22T19:00:44.648051+0000 mgr.y (mgr.34107) 3268 : cluster [DBG] pgmap v4782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:45 smithi115 bash[46934]: cluster 2022-04-22T19:00:44.648051+0000 mgr.y (mgr.34107) 3268 : cluster [DBG] pgmap v4782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:47 smithi121 bash[36902]: cluster 2022-04-22T19:00:46.649111+0000 mgr.y (mgr.34107) 3269 : cluster [DBG] pgmap v4783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:47 smithi115 bash[46934]: cluster 2022-04-22T19:00:46.649111+0000 mgr.y (mgr.34107) 3269 : cluster [DBG] pgmap v4783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:48.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:47 smithi115 bash[45495]: cluster 2022-04-22T19:00:46.649111+0000 mgr.y (mgr.34107) 3269 : cluster [DBG] pgmap v4783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:49 smithi121 bash[36902]: cluster 2022-04-22T19:00:48.650247+0000 mgr.y (mgr.34107) 3270 : cluster [DBG] pgmap v4784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:49 smithi121 bash[36902]: audit 2022-04-22T19:00:49.321393+0000 mon.a (mon.0) 6282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:49 smithi115 bash[46934]: cluster 2022-04-22T19:00:48.650247+0000 mgr.y (mgr.34107) 3270 : cluster [DBG] pgmap v4784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:49 smithi115 bash[46934]: audit 2022-04-22T19:00:49.321393+0000 mon.a (mon.0) 6282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:49 smithi115 bash[45495]: cluster 2022-04-22T19:00:48.650247+0000 mgr.y (mgr.34107) 3270 : cluster [DBG] pgmap v4784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:49 smithi115 bash[45495]: audit 2022-04-22T19:00:49.321393+0000 mon.a (mon.0) 6282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:51.980 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:51 smithi115 bash[45495]: cluster 2022-04-22T19:00:50.651300+0000 mgr.y (mgr.34107) 3271 : cluster [DBG] pgmap v4785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:51.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:51 smithi115 bash[46934]: cluster 2022-04-22T19:00:50.651300+0000 mgr.y (mgr.34107) 3271 : cluster [DBG] pgmap v4785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:51 smithi121 bash[36902]: cluster 2022-04-22T19:00:50.651300+0000 mgr.y (mgr.34107) 3271 : cluster [DBG] pgmap v4785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:51.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:53 smithi115 bash[45495]: cluster 2022-04-22T19:00:52.652423+0000 mgr.y (mgr.34107) 3272 : cluster [DBG] pgmap v4786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:53 smithi115 bash[46934]: cluster 2022-04-22T19:00:52.652423+0000 mgr.y (mgr.34107) 3272 : cluster [DBG] pgmap v4786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:53 smithi121 bash[36902]: cluster 2022-04-22T19:00:52.652423+0000 mgr.y (mgr.34107) 3272 : cluster [DBG] pgmap v4786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:00:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:00:53.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:00:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:54 smithi121 bash[36902]: audit 2022-04-22T19:00:54.324465+0000 mon.a (mon.0) 6283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:54 smithi115 bash[45495]: audit 2022-04-22T19:00:54.324465+0000 mon.a (mon.0) 6283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:54.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:54 smithi115 bash[46934]: audit 2022-04-22T19:00:54.324465+0000 mon.a (mon.0) 6283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:00:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:55 smithi121 bash[36902]: cluster 2022-04-22T19:00:54.653515+0000 mgr.y (mgr.34107) 3273 : cluster [DBG] pgmap v4787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:55 smithi115 bash[45495]: cluster 2022-04-22T19:00:54.653515+0000 mgr.y (mgr.34107) 3273 : cluster [DBG] pgmap v4787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:55 smithi115 bash[46934]: cluster 2022-04-22T19:00:54.653515+0000 mgr.y (mgr.34107) 3273 : cluster [DBG] pgmap v4787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:57 smithi121 bash[36902]: cluster 2022-04-22T19:00:56.654565+0000 mgr.y (mgr.34107) 3274 : cluster [DBG] pgmap v4788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:57 smithi115 bash[45495]: cluster 2022-04-22T19:00:56.654565+0000 mgr.y (mgr.34107) 3274 : cluster [DBG] pgmap v4788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:00:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:57 smithi115 bash[46934]: cluster 2022-04-22T19:00:56.654565+0000 mgr.y (mgr.34107) 3274 : cluster [DBG] pgmap v4788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:59 smithi121 bash[36902]: cluster 2022-04-22T19:00:58.655680+0000 mgr.y (mgr.34107) 3275 : cluster [DBG] pgmap v4789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:00:59 smithi121 bash[36902]: audit 2022-04-22T19:00:59.327838+0000 mon.a (mon.0) 6284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:59 smithi115 bash[45495]: cluster 2022-04-22T19:00:58.655680+0000 mgr.y (mgr.34107) 3275 : cluster [DBG] pgmap v4789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:00:59 smithi115 bash[45495]: audit 2022-04-22T19:00:59.327838+0000 mon.a (mon.0) 6284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:59 smithi115 bash[46934]: cluster 2022-04-22T19:00:58.655680+0000 mgr.y (mgr.34107) 3275 : cluster [DBG] pgmap v4789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:00:59 smithi115 bash[46934]: audit 2022-04-22T19:00:59.327838+0000 mon.a (mon.0) 6284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:00 smithi121 bash[36902]: audit 2022-04-22T19:01:00.105198+0000 mon.a (mon.0) 6285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:01:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:00 smithi121 bash[36902]: audit 2022-04-22T19:01:00.108368+0000 mon.a (mon.0) 6286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:01:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:00 smithi121 bash[36902]: audit 2022-04-22T19:01:00.110271+0000 mon.a (mon.0) 6287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:01:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:00 smithi121 bash[36902]: audit 2022-04-22T19:01:00.468738+0000 mon.a (mon.0) 6288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:01:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[45495]: audit 2022-04-22T19:01:00.105198+0000 mon.a (mon.0) 6285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:01:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[45495]: audit 2022-04-22T19:01:00.108368+0000 mon.a (mon.0) 6286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:01:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[45495]: audit 2022-04-22T19:01:00.110271+0000 mon.a (mon.0) 6287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:01:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[45495]: audit 2022-04-22T19:01:00.468738+0000 mon.a (mon.0) 6288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:01:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[46934]: audit 2022-04-22T19:01:00.105198+0000 mon.a (mon.0) 6285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:01:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[46934]: audit 2022-04-22T19:01:00.108368+0000 mon.a (mon.0) 6286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:01:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[46934]: audit 2022-04-22T19:01:00.110271+0000 mon.a (mon.0) 6287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:01:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:00 smithi115 bash[46934]: audit 2022-04-22T19:01:00.468738+0000 mon.a (mon.0) 6288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:01:01.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:01 smithi115 bash[46934]: cluster 2022-04-22T19:01:00.656751+0000 mgr.y (mgr.34107) 3276 : cluster [DBG] pgmap v4790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:01.982 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:01 smithi115 bash[45495]: cluster 2022-04-22T19:01:00.656751+0000 mgr.y (mgr.34107) 3276 : cluster [DBG] pgmap v4790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:01.983 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:01.972Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:01 smithi121 bash[36902]: cluster 2022-04-22T19:01:00.656751+0000 mgr.y (mgr.34107) 3276 : cluster [DBG] pgmap v4790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:03 smithi115 bash[45495]: cluster 2022-04-22T19:01:02.657855+0000 mgr.y (mgr.34107) 3277 : cluster [DBG] pgmap v4791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:03 smithi115 bash[46934]: cluster 2022-04-22T19:01:02.657855+0000 mgr.y (mgr.34107) 3277 : cluster [DBG] pgmap v4791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:03 smithi121 bash[36902]: cluster 2022-04-22T19:01:02.657855+0000 mgr.y (mgr.34107) 3277 : cluster [DBG] pgmap v4791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:03.866 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:01:04.211 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (84m) 64s ago 97m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (97m) 2m ago 97m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (86m) 64s ago 98m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (85m) 2m ago 98m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (93m) 2m ago 103m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (94m) 64s ago 105m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (93m) 64s ago 105m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (92m) 2m ago 103m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:01:04.212 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (92m) 64s ago 103m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (85m) 64s ago 98m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (85m) 2m ago 98m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (90m) 64s ago 102m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (90m) 64s ago 102m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (89m) 64s ago 101m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (89m) 64s ago 101m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (88m) 2m ago 100m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (87m) 2m ago 100m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (87m) 2m ago 99m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:01:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (86m) 2m ago 99m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:01:04.214 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (84m) 2m ago 98m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:01:04.276 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:03.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:04 smithi115 bash[45495]: audit 2022-04-22T19:01:03.842275+0000 mgr.y (mgr.34107) 3278 : audit [DBG] from='client.48778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:04 smithi115 bash[46934]: audit 2022-04-22T19:01:03.842275+0000 mgr.y (mgr.34107) 3278 : audit [DBG] from='client.48778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:04.621 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:01:04.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:01:04.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:01:04.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:04.622 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:01:04.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:01:04.623 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:01:04.623 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:01:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:04 smithi121 bash[36902]: audit 2022-04-22T19:01:03.842275+0000 mgr.y (mgr.34107) 3278 : audit [DBG] from='client.48778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[45495]: audit 2022-04-22T19:01:04.181545+0000 mgr.y (mgr.34107) 3279 : audit [DBG] from='client.38850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[45495]: audit 2022-04-22T19:01:04.330933+0000 mon.a (mon.0) 6289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[45495]: audit 2022-04-22T19:01:04.612203+0000 mon.c (mon.1) 173 : audit [DBG] from='client.? 172.21.15.115:0/2234728657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[45495]: cluster 2022-04-22T19:01:04.659024+0000 mgr.y (mgr.34107) 3280 : cluster [DBG] pgmap v4792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[46934]: audit 2022-04-22T19:01:04.181545+0000 mgr.y (mgr.34107) 3279 : audit [DBG] from='client.38850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[46934]: audit 2022-04-22T19:01:04.330933+0000 mon.a (mon.0) 6289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[46934]: audit 2022-04-22T19:01:04.612203+0000 mon.c (mon.1) 173 : audit [DBG] from='client.? 172.21.15.115:0/2234728657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:05.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:05 smithi115 bash[46934]: cluster 2022-04-22T19:01:04.659024+0000 mgr.y (mgr.34107) 3280 : cluster [DBG] pgmap v4792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:05 smithi121 bash[36902]: audit 2022-04-22T19:01:04.181545+0000 mgr.y (mgr.34107) 3279 : audit [DBG] from='client.38850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:05 smithi121 bash[36902]: audit 2022-04-22T19:01:04.330933+0000 mon.a (mon.0) 6289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:05 smithi121 bash[36902]: audit 2022-04-22T19:01:04.612203+0000 mon.c (mon.1) 173 : audit [DBG] from='client.? 172.21.15.115:0/2234728657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:05 smithi121 bash[36902]: cluster 2022-04-22T19:01:04.659024+0000 mgr.y (mgr.34107) 3280 : cluster [DBG] pgmap v4792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:07 smithi121 bash[36902]: cluster 2022-04-22T19:01:06.660195+0000 mgr.y (mgr.34107) 3281 : cluster [DBG] pgmap v4793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:07 smithi115 bash[45495]: cluster 2022-04-22T19:01:06.660195+0000 mgr.y (mgr.34107) 3281 : cluster [DBG] pgmap v4793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:07 smithi115 bash[46934]: cluster 2022-04-22T19:01:06.660195+0000 mgr.y (mgr.34107) 3281 : cluster [DBG] pgmap v4793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:09 smithi121 bash[36902]: cluster 2022-04-22T19:01:08.661265+0000 mgr.y (mgr.34107) 3282 : cluster [DBG] pgmap v4794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:09 smithi121 bash[36902]: audit 2022-04-22T19:01:09.333866+0000 mon.a (mon.0) 6290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:09 smithi115 bash[45495]: cluster 2022-04-22T19:01:08.661265+0000 mgr.y (mgr.34107) 3282 : cluster [DBG] pgmap v4794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:09 smithi115 bash[45495]: audit 2022-04-22T19:01:09.333866+0000 mon.a (mon.0) 6290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:09 smithi115 bash[46934]: cluster 2022-04-22T19:01:08.661265+0000 mgr.y (mgr.34107) 3282 : cluster [DBG] pgmap v4794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:09 smithi115 bash[46934]: audit 2022-04-22T19:01:09.333866+0000 mon.a (mon.0) 6290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:11 smithi121 bash[36902]: cluster 2022-04-22T19:01:10.662231+0000 mgr.y (mgr.34107) 3283 : cluster [DBG] pgmap v4795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:11 smithi115 bash[45495]: cluster 2022-04-22T19:01:10.662231+0000 mgr.y (mgr.34107) 3283 : cluster [DBG] pgmap v4795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:12.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:11.974Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:11 smithi115 bash[46934]: cluster 2022-04-22T19:01:10.662231+0000 mgr.y (mgr.34107) 3283 : cluster [DBG] pgmap v4795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:13 smithi115 bash[45495]: cluster 2022-04-22T19:01:12.663335+0000 mgr.y (mgr.34107) 3284 : cluster [DBG] pgmap v4796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:13 smithi115 bash[46934]: cluster 2022-04-22T19:01:12.663335+0000 mgr.y (mgr.34107) 3284 : cluster [DBG] pgmap v4796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:13 smithi121 bash[36902]: cluster 2022-04-22T19:01:12.663335+0000 mgr.y (mgr.34107) 3284 : cluster [DBG] pgmap v4796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:13.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:14 smithi121 bash[36902]: audit 2022-04-22T19:01:14.335185+0000 mon.a (mon.0) 6291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:01:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:14 smithi121 bash[36902]: audit 2022-04-22T19:01:14.335701+0000 mon.a (mon.0) 6292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:01:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:14 smithi121 bash[36902]: audit 2022-04-22T19:01:14.336335+0000 mon.a (mon.0) 6293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[45495]: audit 2022-04-22T19:01:14.335185+0000 mon.a (mon.0) 6291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:01:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[45495]: audit 2022-04-22T19:01:14.335701+0000 mon.a (mon.0) 6292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:01:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[45495]: audit 2022-04-22T19:01:14.336335+0000 mon.a (mon.0) 6293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[46934]: audit 2022-04-22T19:01:14.335185+0000 mon.a (mon.0) 6291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:01:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[46934]: audit 2022-04-22T19:01:14.335701+0000 mon.a (mon.0) 6292 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:01:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:14 smithi115 bash[46934]: audit 2022-04-22T19:01:14.336335+0000 mon.a (mon.0) 6293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:15 smithi121 bash[36902]: cluster 2022-04-22T19:01:14.664396+0000 mgr.y (mgr.34107) 3285 : cluster [DBG] pgmap v4797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:15 smithi115 bash[45495]: cluster 2022-04-22T19:01:14.664396+0000 mgr.y (mgr.34107) 3285 : cluster [DBG] pgmap v4797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:15 smithi115 bash[46934]: cluster 2022-04-22T19:01:14.664396+0000 mgr.y (mgr.34107) 3285 : cluster [DBG] pgmap v4797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:17 smithi121 bash[36902]: cluster 2022-04-22T19:01:16.665492+0000 mgr.y (mgr.34107) 3286 : cluster [DBG] pgmap v4798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:17 smithi115 bash[45495]: cluster 2022-04-22T19:01:16.665492+0000 mgr.y (mgr.34107) 3286 : cluster [DBG] pgmap v4798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:17 smithi115 bash[46934]: cluster 2022-04-22T19:01:16.665492+0000 mgr.y (mgr.34107) 3286 : cluster [DBG] pgmap v4798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:19 smithi121 bash[36902]: cluster 2022-04-22T19:01:18.666611+0000 mgr.y (mgr.34107) 3287 : cluster [DBG] pgmap v4799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:19 smithi121 bash[36902]: audit 2022-04-22T19:01:19.339154+0000 mon.a (mon.0) 6294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:19 smithi115 bash[45495]: cluster 2022-04-22T19:01:18.666611+0000 mgr.y (mgr.34107) 3287 : cluster [DBG] pgmap v4799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:19 smithi115 bash[45495]: audit 2022-04-22T19:01:19.339154+0000 mon.a (mon.0) 6294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:19 smithi115 bash[46934]: cluster 2022-04-22T19:01:18.666611+0000 mgr.y (mgr.34107) 3287 : cluster [DBG] pgmap v4799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:19 smithi115 bash[46934]: audit 2022-04-22T19:01:19.339154+0000 mon.a (mon.0) 6294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:21 smithi121 bash[36902]: cluster 2022-04-22T19:01:20.667704+0000 mgr.y (mgr.34107) 3288 : cluster [DBG] pgmap v4800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:21 smithi115 bash[45495]: cluster 2022-04-22T19:01:20.667704+0000 mgr.y (mgr.34107) 3288 : cluster [DBG] pgmap v4800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:22.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:21.973Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:21 smithi115 bash[46934]: cluster 2022-04-22T19:01:20.667704+0000 mgr.y (mgr.34107) 3288 : cluster [DBG] pgmap v4800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:23 smithi115 bash[45495]: cluster 2022-04-22T19:01:22.668720+0000 mgr.y (mgr.34107) 3289 : cluster [DBG] pgmap v4801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:23 smithi115 bash[46934]: cluster 2022-04-22T19:01:22.668720+0000 mgr.y (mgr.34107) 3289 : cluster [DBG] pgmap v4801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:23 smithi121 bash[36902]: cluster 2022-04-22T19:01:22.668720+0000 mgr.y (mgr.34107) 3289 : cluster [DBG] pgmap v4801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:23.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:24 smithi121 bash[36902]: audit 2022-04-22T19:01:24.342282+0000 mon.a (mon.0) 6295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:24.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:24 smithi115 bash[45495]: audit 2022-04-22T19:01:24.342282+0000 mon.a (mon.0) 6295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:24.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:24 smithi115 bash[46934]: audit 2022-04-22T19:01:24.342282+0000 mon.a (mon.0) 6295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:25 smithi121 bash[36902]: cluster 2022-04-22T19:01:24.669945+0000 mgr.y (mgr.34107) 3290 : cluster [DBG] pgmap v4802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:25 smithi115 bash[45495]: cluster 2022-04-22T19:01:24.669945+0000 mgr.y (mgr.34107) 3290 : cluster [DBG] pgmap v4802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:25 smithi115 bash[46934]: cluster 2022-04-22T19:01:24.669945+0000 mgr.y (mgr.34107) 3290 : cluster [DBG] pgmap v4802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:27 smithi121 bash[36902]: cluster 2022-04-22T19:01:26.670990+0000 mgr.y (mgr.34107) 3291 : cluster [DBG] pgmap v4803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:27 smithi115 bash[45495]: cluster 2022-04-22T19:01:26.670990+0000 mgr.y (mgr.34107) 3291 : cluster [DBG] pgmap v4803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:27 smithi115 bash[46934]: cluster 2022-04-22T19:01:26.670990+0000 mgr.y (mgr.34107) 3291 : cluster [DBG] pgmap v4803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:29 smithi121 bash[36902]: cluster 2022-04-22T19:01:28.672047+0000 mgr.y (mgr.34107) 3292 : cluster [DBG] pgmap v4804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:29 smithi121 bash[36902]: audit 2022-04-22T19:01:29.345287+0000 mon.a (mon.0) 6296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:29 smithi115 bash[45495]: cluster 2022-04-22T19:01:28.672047+0000 mgr.y (mgr.34107) 3292 : cluster [DBG] pgmap v4804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:29 smithi115 bash[45495]: audit 2022-04-22T19:01:29.345287+0000 mon.a (mon.0) 6296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:29 smithi115 bash[46934]: cluster 2022-04-22T19:01:28.672047+0000 mgr.y (mgr.34107) 3292 : cluster [DBG] pgmap v4804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:30.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:29 smithi115 bash[46934]: audit 2022-04-22T19:01:29.345287+0000 mon.a (mon.0) 6296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:31 smithi121 bash[36902]: cluster 2022-04-22T19:01:30.673134+0000 mgr.y (mgr.34107) 3293 : cluster [DBG] pgmap v4805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:31 smithi115 bash[45495]: cluster 2022-04-22T19:01:30.673134+0000 mgr.y (mgr.34107) 3293 : cluster [DBG] pgmap v4805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:32.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:31.973Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:31 smithi115 bash[46934]: cluster 2022-04-22T19:01:30.673134+0000 mgr.y (mgr.34107) 3293 : cluster [DBG] pgmap v4805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:33 smithi115 bash[45495]: cluster 2022-04-22T19:01:32.674241+0000 mgr.y (mgr.34107) 3294 : cluster [DBG] pgmap v4806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:33 smithi115 bash[46934]: cluster 2022-04-22T19:01:32.674241+0000 mgr.y (mgr.34107) 3294 : cluster [DBG] pgmap v4806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:33 smithi121 bash[36902]: cluster 2022-04-22T19:01:32.674241+0000 mgr.y (mgr.34107) 3294 : cluster [DBG] pgmap v4806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:33.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:34 smithi121 bash[36902]: audit 2022-04-22T19:01:34.348539+0000 mon.a (mon.0) 6297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:34 smithi115 bash[45495]: audit 2022-04-22T19:01:34.348539+0000 mon.a (mon.0) 6297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:34.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:34 smithi115 bash[46934]: audit 2022-04-22T19:01:34.348539+0000 mon.a (mon.0) 6297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:35.001 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:01:35.358 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (84m) 95s ago 98m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (98m) 2m ago 98m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (86m) 95s ago 99m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (86m) 2m ago 99m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (93m) 2m ago 103m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (94m) 95s ago 105m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (93m) 95s ago 106m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (92m) 2m ago 104m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (92m) 95s ago 104m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:01:35.359 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (85m) 95s ago 98m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (85m) 2m ago 98m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (91m) 95s ago 103m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (90m) 95s ago 102m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (90m) 95s ago 102m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (89m) 95s ago 101m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (89m) 2m ago 101m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (88m) 2m ago 100m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (87m) 2m ago 100m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (87m) 2m ago 100m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:01:35.360 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (85m) 2m ago 98m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:01:35.762 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:01:35.762 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:01:35.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:01:35.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:01:35.764 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:01:35.764 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:01:35.764 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:01:35.764 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:01:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:35 smithi121 bash[36902]: cluster 2022-04-22T19:01:34.675358+0000 mgr.y (mgr.34107) 3295 : cluster [DBG] pgmap v4807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:35 smithi121 bash[36902]: audit 2022-04-22T19:01:34.972535+0000 mgr.y (mgr.34107) 3296 : audit [DBG] from='client.48796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:35 smithi115 bash[45495]: cluster 2022-04-22T19:01:34.675358+0000 mgr.y (mgr.34107) 3295 : cluster [DBG] pgmap v4807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:35 smithi115 bash[45495]: audit 2022-04-22T19:01:34.972535+0000 mgr.y (mgr.34107) 3296 : audit [DBG] from='client.48796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:35 smithi115 bash[46934]: cluster 2022-04-22T19:01:34.675358+0000 mgr.y (mgr.34107) 3295 : cluster [DBG] pgmap v4807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:35 smithi115 bash[46934]: audit 2022-04-22T19:01:34.972535+0000 mgr.y (mgr.34107) 3296 : audit [DBG] from='client.48796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:36 smithi121 bash[36902]: audit 2022-04-22T19:01:35.329444+0000 mgr.y (mgr.34107) 3297 : audit [DBG] from='client.48802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:36 smithi121 bash[36902]: audit 2022-04-22T19:01:35.753310+0000 mon.c (mon.1) 174 : audit [DBG] from='client.? 172.21.15.115:0/52815283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:36 smithi115 bash[45495]: audit 2022-04-22T19:01:35.329444+0000 mgr.y (mgr.34107) 3297 : audit [DBG] from='client.48802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:36.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:36 smithi115 bash[45495]: audit 2022-04-22T19:01:35.753310+0000 mon.c (mon.1) 174 : audit [DBG] from='client.? 172.21.15.115:0/52815283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:36.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:36 smithi115 bash[46934]: audit 2022-04-22T19:01:35.329444+0000 mgr.y (mgr.34107) 3297 : audit [DBG] from='client.48802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:01:36.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:36 smithi115 bash[46934]: audit 2022-04-22T19:01:35.753310+0000 mon.c (mon.1) 174 : audit [DBG] from='client.? 172.21.15.115:0/52815283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:01:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:37 smithi121 bash[36902]: cluster 2022-04-22T19:01:36.676482+0000 mgr.y (mgr.34107) 3298 : cluster [DBG] pgmap v4808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:37 smithi115 bash[45495]: cluster 2022-04-22T19:01:36.676482+0000 mgr.y (mgr.34107) 3298 : cluster [DBG] pgmap v4808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:37 smithi115 bash[46934]: cluster 2022-04-22T19:01:36.676482+0000 mgr.y (mgr.34107) 3298 : cluster [DBG] pgmap v4808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:39 smithi121 bash[36902]: cluster 2022-04-22T19:01:38.677609+0000 mgr.y (mgr.34107) 3299 : cluster [DBG] pgmap v4809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:39 smithi121 bash[36902]: audit 2022-04-22T19:01:39.351635+0000 mon.a (mon.0) 6298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:39 smithi115 bash[45495]: cluster 2022-04-22T19:01:38.677609+0000 mgr.y (mgr.34107) 3299 : cluster [DBG] pgmap v4809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:39 smithi115 bash[45495]: audit 2022-04-22T19:01:39.351635+0000 mon.a (mon.0) 6298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:39 smithi115 bash[46934]: cluster 2022-04-22T19:01:38.677609+0000 mgr.y (mgr.34107) 3299 : cluster [DBG] pgmap v4809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:40.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:39 smithi115 bash[46934]: audit 2022-04-22T19:01:39.351635+0000 mon.a (mon.0) 6298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:41 smithi121 bash[36902]: cluster 2022-04-22T19:01:40.678673+0000 mgr.y (mgr.34107) 3300 : cluster [DBG] pgmap v4810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:41 smithi115 bash[45495]: cluster 2022-04-22T19:01:40.678673+0000 mgr.y (mgr.34107) 3300 : cluster [DBG] pgmap v4810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:42.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:41.974Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:41 smithi115 bash[46934]: cluster 2022-04-22T19:01:40.678673+0000 mgr.y (mgr.34107) 3300 : cluster [DBG] pgmap v4810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:43 smithi115 bash[45495]: cluster 2022-04-22T19:01:42.679756+0000 mgr.y (mgr.34107) 3301 : cluster [DBG] pgmap v4811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:43.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:43 smithi115 bash[46934]: cluster 2022-04-22T19:01:42.679756+0000 mgr.y (mgr.34107) 3301 : cluster [DBG] pgmap v4811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:43 smithi121 bash[36902]: cluster 2022-04-22T19:01:42.679756+0000 mgr.y (mgr.34107) 3301 : cluster [DBG] pgmap v4811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:43.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:44 smithi121 bash[36902]: audit 2022-04-22T19:01:44.354716+0000 mon.a (mon.0) 6299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:44 smithi115 bash[45495]: audit 2022-04-22T19:01:44.354716+0000 mon.a (mon.0) 6299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:44 smithi115 bash[46934]: audit 2022-04-22T19:01:44.354716+0000 mon.a (mon.0) 6299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:45 smithi121 bash[36902]: cluster 2022-04-22T19:01:44.680744+0000 mgr.y (mgr.34107) 3302 : cluster [DBG] pgmap v4812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:45 smithi115 bash[45495]: cluster 2022-04-22T19:01:44.680744+0000 mgr.y (mgr.34107) 3302 : cluster [DBG] pgmap v4812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:45 smithi115 bash[46934]: cluster 2022-04-22T19:01:44.680744+0000 mgr.y (mgr.34107) 3302 : cluster [DBG] pgmap v4812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:47 smithi121 bash[36902]: cluster 2022-04-22T19:01:46.681829+0000 mgr.y (mgr.34107) 3303 : cluster [DBG] pgmap v4813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:47 smithi115 bash[45495]: cluster 2022-04-22T19:01:46.681829+0000 mgr.y (mgr.34107) 3303 : cluster [DBG] pgmap v4813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:47 smithi115 bash[46934]: cluster 2022-04-22T19:01:46.681829+0000 mgr.y (mgr.34107) 3303 : cluster [DBG] pgmap v4813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:49 smithi121 bash[36902]: cluster 2022-04-22T19:01:48.683025+0000 mgr.y (mgr.34107) 3304 : cluster [DBG] pgmap v4814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:49 smithi121 bash[36902]: audit 2022-04-22T19:01:49.357646+0000 mon.a (mon.0) 6300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:49 smithi115 bash[45495]: cluster 2022-04-22T19:01:48.683025+0000 mgr.y (mgr.34107) 3304 : cluster [DBG] pgmap v4814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:49 smithi115 bash[45495]: audit 2022-04-22T19:01:49.357646+0000 mon.a (mon.0) 6300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:49 smithi115 bash[46934]: cluster 2022-04-22T19:01:48.683025+0000 mgr.y (mgr.34107) 3304 : cluster [DBG] pgmap v4814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:49 smithi115 bash[46934]: audit 2022-04-22T19:01:49.357646+0000 mon.a (mon.0) 6300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:51 smithi121 bash[36902]: cluster 2022-04-22T19:01:50.684153+0000 mgr.y (mgr.34107) 3305 : cluster [DBG] pgmap v4815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:51 smithi115 bash[45495]: cluster 2022-04-22T19:01:50.684153+0000 mgr.y (mgr.34107) 3305 : cluster [DBG] pgmap v4815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:52.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:51.974Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:51 smithi115 bash[46934]: cluster 2022-04-22T19:01:50.684153+0000 mgr.y (mgr.34107) 3305 : cluster [DBG] pgmap v4815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:53 smithi115 bash[45495]: cluster 2022-04-22T19:01:52.685255+0000 mgr.y (mgr.34107) 3306 : cluster [DBG] pgmap v4816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:53 smithi115 bash[46934]: cluster 2022-04-22T19:01:52.685255+0000 mgr.y (mgr.34107) 3306 : cluster [DBG] pgmap v4816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:53 smithi121 bash[36902]: cluster 2022-04-22T19:01:52.685255+0000 mgr.y (mgr.34107) 3306 : cluster [DBG] pgmap v4816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:01:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:01:53.847Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:01:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:54 smithi121 bash[36902]: audit 2022-04-22T19:01:54.360774+0000 mon.a (mon.0) 6301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:54 smithi115 bash[45495]: audit 2022-04-22T19:01:54.360774+0000 mon.a (mon.0) 6301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:54.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:54 smithi115 bash[46934]: audit 2022-04-22T19:01:54.360774+0000 mon.a (mon.0) 6301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:01:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:55 smithi121 bash[36902]: cluster 2022-04-22T19:01:54.686307+0000 mgr.y (mgr.34107) 3307 : cluster [DBG] pgmap v4817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:55 smithi115 bash[45495]: cluster 2022-04-22T19:01:54.686307+0000 mgr.y (mgr.34107) 3307 : cluster [DBG] pgmap v4817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:55 smithi115 bash[46934]: cluster 2022-04-22T19:01:54.686307+0000 mgr.y (mgr.34107) 3307 : cluster [DBG] pgmap v4817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:57 smithi121 bash[36902]: cluster 2022-04-22T19:01:56.687430+0000 mgr.y (mgr.34107) 3308 : cluster [DBG] pgmap v4818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:57 smithi115 bash[45495]: cluster 2022-04-22T19:01:56.687430+0000 mgr.y (mgr.34107) 3308 : cluster [DBG] pgmap v4818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:01:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:57 smithi115 bash[46934]: cluster 2022-04-22T19:01:56.687430+0000 mgr.y (mgr.34107) 3308 : cluster [DBG] pgmap v4818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:00.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:59 smithi121 bash[36902]: cluster 2022-04-22T19:01:58.688573+0000 mgr.y (mgr.34107) 3309 : cluster [DBG] pgmap v4819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:00.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:01:59 smithi121 bash[36902]: audit 2022-04-22T19:01:59.364064+0000 mon.a (mon.0) 6302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:00.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:59 smithi115 bash[45495]: cluster 2022-04-22T19:01:58.688573+0000 mgr.y (mgr.34107) 3309 : cluster [DBG] pgmap v4819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:00.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:01:59 smithi115 bash[45495]: audit 2022-04-22T19:01:59.364064+0000 mon.a (mon.0) 6302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:00.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:59 smithi115 bash[46934]: cluster 2022-04-22T19:01:58.688573+0000 mgr.y (mgr.34107) 3309 : cluster [DBG] pgmap v4819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:00.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:01:59 smithi115 bash[46934]: audit 2022-04-22T19:01:59.364064+0000 mon.a (mon.0) 6302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:00.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[45495]: audit 2022-04-22T19:02:00.478544+0000 mon.a (mon.0) 6303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:02:00.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[45495]: audit 2022-04-22T19:02:00.481674+0000 mon.a (mon.0) 6304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:02:00.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[45495]: audit 2022-04-22T19:02:00.483612+0000 mon.a (mon.0) 6305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:02:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:00 smithi121 bash[36902]: audit 2022-04-22T19:02:00.478544+0000 mon.a (mon.0) 6303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:02:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:00 smithi121 bash[36902]: audit 2022-04-22T19:02:00.481674+0000 mon.a (mon.0) 6304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:02:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:00 smithi121 bash[36902]: audit 2022-04-22T19:02:00.483612+0000 mon.a (mon.0) 6305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:02:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[46934]: audit 2022-04-22T19:02:00.478544+0000 mon.a (mon.0) 6303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:02:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[46934]: audit 2022-04-22T19:02:00.481674+0000 mon.a (mon.0) 6304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:02:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:00 smithi115 bash[46934]: audit 2022-04-22T19:02:00.483612+0000 mon.a (mon.0) 6305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:02:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:01 smithi121 bash[36902]: cluster 2022-04-22T19:02:00.689545+0000 mgr.y (mgr.34107) 3310 : cluster [DBG] pgmap v4820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:01 smithi121 bash[36902]: audit 2022-04-22T19:02:00.843004+0000 mon.a (mon.0) 6306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:02:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:01 smithi115 bash[46934]: cluster 2022-04-22T19:02:00.689545+0000 mgr.y (mgr.34107) 3310 : cluster [DBG] pgmap v4820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:02.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:01 smithi115 bash[46934]: audit 2022-04-22T19:02:00.843004+0000 mon.a (mon.0) 6306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:02:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:01.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:01 smithi115 bash[45495]: cluster 2022-04-22T19:02:00.689545+0000 mgr.y (mgr.34107) 3310 : cluster [DBG] pgmap v4820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:01 smithi115 bash[45495]: audit 2022-04-22T19:02:00.843004+0000 mon.a (mon.0) 6306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:02:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:03 smithi115 bash[45495]: cluster 2022-04-22T19:02:02.690633+0000 mgr.y (mgr.34107) 3311 : cluster [DBG] pgmap v4821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:03 smithi115 bash[46934]: cluster 2022-04-22T19:02:02.690633+0000 mgr.y (mgr.34107) 3311 : cluster [DBG] pgmap v4821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:03 smithi121 bash[36902]: cluster 2022-04-22T19:02:02.690633+0000 mgr.y (mgr.34107) 3311 : cluster [DBG] pgmap v4821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:03.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:04 smithi121 bash[36902]: audit 2022-04-22T19:02:04.367101+0000 mon.a (mon.0) 6307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:04.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:04 smithi115 bash[45495]: audit 2022-04-22T19:02:04.367101+0000 mon.a (mon.0) 6307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:04.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:04 smithi115 bash[46934]: audit 2022-04-22T19:02:04.367101+0000 mon.a (mon.0) 6307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:05 smithi121 bash[36902]: cluster 2022-04-22T19:02:04.691724+0000 mgr.y (mgr.34107) 3312 : cluster [DBG] pgmap v4822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:05.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:05 smithi115 bash[46934]: cluster 2022-04-22T19:02:04.691724+0000 mgr.y (mgr.34107) 3312 : cluster [DBG] pgmap v4822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:05.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:05 smithi115 bash[45495]: cluster 2022-04-22T19:02:04.691724+0000 mgr.y (mgr.34107) 3312 : cluster [DBG] pgmap v4822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:06.159 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:02:06.505 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:02:06.505 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (85m) 2m ago 98m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:02:06.505 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (98m) 3m ago 98m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:02:06.505 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (87m) 2m ago 99m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:02:06.505 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (86m) 3m ago 99m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:02:06.506 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (94m) 3m ago 104m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:02:06.506 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (95m) 2m ago 106m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:02:06.506 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (94m) 2m ago 106m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:02:06.506 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (93m) 3m ago 104m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (93m) 2m ago 104m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (86m) 2m ago 99m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (86m) 3m ago 99m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (91m) 2m ago 103m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (91m) 2m ago 103m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (90m) 2m ago 102m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:02:06.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (90m) 2m ago 102m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:02:06.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (89m) 3m ago 101m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:02:06.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (88m) 3m ago 101m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:02:06.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (88m) 3m ago 101m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:02:06.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (87m) 3m ago 100m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:02:06.508 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (85m) 3m ago 99m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:02:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:06 smithi121 bash[36902]: audit 2022-04-22T19:02:06.133164+0000 mgr.y (mgr.34107) 3313 : audit [DBG] from='client.38880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:06.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:06 smithi115 bash[45495]: audit 2022-04-22T19:02:06.133164+0000 mgr.y (mgr.34107) 3313 : audit [DBG] from='client.38880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:06.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:06 smithi115 bash[46934]: audit 2022-04-22T19:02:06.133164+0000 mgr.y (mgr.34107) 3313 : audit [DBG] from='client.38880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:06.913 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:02:06.914 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:02:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:07 smithi121 bash[36902]: audit 2022-04-22T19:02:06.477572+0000 mgr.y (mgr.34107) 3314 : audit [DBG] from='client.38886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:07 smithi121 bash[36902]: cluster 2022-04-22T19:02:06.692658+0000 mgr.y (mgr.34107) 3315 : cluster [DBG] pgmap v4823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:07 smithi121 bash[36902]: audit 2022-04-22T19:02:06.903685+0000 mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.115:0/1042183229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[45495]: audit 2022-04-22T19:02:06.477572+0000 mgr.y (mgr.34107) 3314 : audit [DBG] from='client.38886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[45495]: cluster 2022-04-22T19:02:06.692658+0000 mgr.y (mgr.34107) 3315 : cluster [DBG] pgmap v4823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[45495]: audit 2022-04-22T19:02:06.903685+0000 mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.115:0/1042183229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[46934]: audit 2022-04-22T19:02:06.477572+0000 mgr.y (mgr.34107) 3314 : audit [DBG] from='client.38886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[46934]: cluster 2022-04-22T19:02:06.692658+0000 mgr.y (mgr.34107) 3315 : cluster [DBG] pgmap v4823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:07 smithi115 bash[46934]: audit 2022-04-22T19:02:06.903685+0000 mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.115:0/1042183229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:09 smithi121 bash[36902]: cluster 2022-04-22T19:02:08.693751+0000 mgr.y (mgr.34107) 3316 : cluster [DBG] pgmap v4824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:09 smithi121 bash[36902]: audit 2022-04-22T19:02:09.370194+0000 mon.a (mon.0) 6308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:09 smithi115 bash[46934]: cluster 2022-04-22T19:02:08.693751+0000 mgr.y (mgr.34107) 3316 : cluster [DBG] pgmap v4824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:09 smithi115 bash[46934]: audit 2022-04-22T19:02:09.370194+0000 mon.a (mon.0) 6308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:09 smithi115 bash[45495]: cluster 2022-04-22T19:02:08.693751+0000 mgr.y (mgr.34107) 3316 : cluster [DBG] pgmap v4824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:09 smithi115 bash[45495]: audit 2022-04-22T19:02:09.370194+0000 mon.a (mon.0) 6308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:12.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:11 smithi121 bash[36902]: cluster 2022-04-22T19:02:10.694788+0000 mgr.y (mgr.34107) 3317 : cluster [DBG] pgmap v4825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:11 smithi115 bash[46934]: cluster 2022-04-22T19:02:10.694788+0000 mgr.y (mgr.34107) 3317 : cluster [DBG] pgmap v4825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:12.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:11.976Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:11 smithi115 bash[45495]: cluster 2022-04-22T19:02:10.694788+0000 mgr.y (mgr.34107) 3317 : cluster [DBG] pgmap v4825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:13 smithi121 bash[36902]: cluster 2022-04-22T19:02:12.695866+0000 mgr.y (mgr.34107) 3318 : cluster [DBG] pgmap v4826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:13.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:13 smithi115 bash[45495]: cluster 2022-04-22T19:02:12.695866+0000 mgr.y (mgr.34107) 3318 : cluster [DBG] pgmap v4826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:13.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:13 smithi115 bash[46934]: cluster 2022-04-22T19:02:12.695866+0000 mgr.y (mgr.34107) 3318 : cluster [DBG] pgmap v4826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:13.847Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:14 smithi121 bash[36902]: audit 2022-04-22T19:02:14.340697+0000 mon.a (mon.0) 6309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:02:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:14 smithi121 bash[36902]: audit 2022-04-22T19:02:14.350777+0000 mon.a (mon.0) 6310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:02:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:14 smithi121 bash[36902]: audit 2022-04-22T19:02:14.373156+0000 mon.a (mon.0) 6311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[46934]: audit 2022-04-22T19:02:14.340697+0000 mon.a (mon.0) 6309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:02:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[46934]: audit 2022-04-22T19:02:14.350777+0000 mon.a (mon.0) 6310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:02:14.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[46934]: audit 2022-04-22T19:02:14.373156+0000 mon.a (mon.0) 6311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[45495]: audit 2022-04-22T19:02:14.340697+0000 mon.a (mon.0) 6309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:02:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[45495]: audit 2022-04-22T19:02:14.350777+0000 mon.a (mon.0) 6310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:02:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:14 smithi115 bash[45495]: audit 2022-04-22T19:02:14.373156+0000 mon.a (mon.0) 6311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:15 smithi121 bash[36902]: cluster 2022-04-22T19:02:14.697068+0000 mgr.y (mgr.34107) 3319 : cluster [DBG] pgmap v4827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:15 smithi115 bash[45495]: cluster 2022-04-22T19:02:14.697068+0000 mgr.y (mgr.34107) 3319 : cluster [DBG] pgmap v4827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:15 smithi115 bash[46934]: cluster 2022-04-22T19:02:14.697068+0000 mgr.y (mgr.34107) 3319 : cluster [DBG] pgmap v4827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:17 smithi115 bash[45495]: cluster 2022-04-22T19:02:16.698346+0000 mgr.y (mgr.34107) 3320 : cluster [DBG] pgmap v4828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:17 smithi115 bash[46934]: cluster 2022-04-22T19:02:16.698346+0000 mgr.y (mgr.34107) 3320 : cluster [DBG] pgmap v4828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:17 smithi121 bash[36902]: cluster 2022-04-22T19:02:16.698346+0000 mgr.y (mgr.34107) 3320 : cluster [DBG] pgmap v4828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:19 smithi115 bash[45495]: cluster 2022-04-22T19:02:18.699634+0000 mgr.y (mgr.34107) 3321 : cluster [DBG] pgmap v4829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:19 smithi115 bash[45495]: audit 2022-04-22T19:02:19.376255+0000 mon.a (mon.0) 6312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:19 smithi115 bash[46934]: cluster 2022-04-22T19:02:18.699634+0000 mgr.y (mgr.34107) 3321 : cluster [DBG] pgmap v4829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:19 smithi115 bash[46934]: audit 2022-04-22T19:02:19.376255+0000 mon.a (mon.0) 6312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:19 smithi121 bash[36902]: cluster 2022-04-22T19:02:18.699634+0000 mgr.y (mgr.34107) 3321 : cluster [DBG] pgmap v4829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:19 smithi121 bash[36902]: audit 2022-04-22T19:02:19.376255+0000 mon.a (mon.0) 6312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:21 smithi115 bash[45495]: cluster 2022-04-22T19:02:20.700874+0000 mgr.y (mgr.34107) 3322 : cluster [DBG] pgmap v4830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:22.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:21.976Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:21 smithi115 bash[46934]: cluster 2022-04-22T19:02:20.700874+0000 mgr.y (mgr.34107) 3322 : cluster [DBG] pgmap v4830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:21 smithi121 bash[36902]: cluster 2022-04-22T19:02:20.700874+0000 mgr.y (mgr.34107) 3322 : cluster [DBG] pgmap v4830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:23 smithi121 bash[36902]: cluster 2022-04-22T19:02:22.702123+0000 mgr.y (mgr.34107) 3323 : cluster [DBG] pgmap v4831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:23 smithi115 bash[45495]: cluster 2022-04-22T19:02:22.702123+0000 mgr.y (mgr.34107) 3323 : cluster [DBG] pgmap v4831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:23 smithi115 bash[46934]: cluster 2022-04-22T19:02:22.702123+0000 mgr.y (mgr.34107) 3323 : cluster [DBG] pgmap v4831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:23.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:24 smithi121 bash[36902]: audit 2022-04-22T19:02:24.379536+0000 mon.a (mon.0) 6313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:24.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:24 smithi115 bash[45495]: audit 2022-04-22T19:02:24.379536+0000 mon.a (mon.0) 6313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:24.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:24 smithi115 bash[46934]: audit 2022-04-22T19:02:24.379536+0000 mon.a (mon.0) 6313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:25 smithi121 bash[36902]: cluster 2022-04-22T19:02:24.703225+0000 mgr.y (mgr.34107) 3324 : cluster [DBG] pgmap v4832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:25 smithi115 bash[45495]: cluster 2022-04-22T19:02:24.703225+0000 mgr.y (mgr.34107) 3324 : cluster [DBG] pgmap v4832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:25 smithi115 bash[46934]: cluster 2022-04-22T19:02:24.703225+0000 mgr.y (mgr.34107) 3324 : cluster [DBG] pgmap v4832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:27 smithi115 bash[45495]: cluster 2022-04-22T19:02:26.704279+0000 mgr.y (mgr.34107) 3325 : cluster [DBG] pgmap v4833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:27 smithi115 bash[46934]: cluster 2022-04-22T19:02:26.704279+0000 mgr.y (mgr.34107) 3325 : cluster [DBG] pgmap v4833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:27 smithi121 bash[36902]: cluster 2022-04-22T19:02:26.704279+0000 mgr.y (mgr.34107) 3325 : cluster [DBG] pgmap v4833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:29 smithi115 bash[45495]: cluster 2022-04-22T19:02:28.705465+0000 mgr.y (mgr.34107) 3326 : cluster [DBG] pgmap v4834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:29 smithi115 bash[45495]: audit 2022-04-22T19:02:29.382598+0000 mon.a (mon.0) 6314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:30.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:29 smithi115 bash[46934]: cluster 2022-04-22T19:02:28.705465+0000 mgr.y (mgr.34107) 3326 : cluster [DBG] pgmap v4834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:29 smithi115 bash[46934]: audit 2022-04-22T19:02:29.382598+0000 mon.a (mon.0) 6314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:29 smithi121 bash[36902]: cluster 2022-04-22T19:02:28.705465+0000 mgr.y (mgr.34107) 3326 : cluster [DBG] pgmap v4834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:29 smithi121 bash[36902]: audit 2022-04-22T19:02:29.382598+0000 mon.a (mon.0) 6314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:31 smithi115 bash[46934]: cluster 2022-04-22T19:02:30.706654+0000 mgr.y (mgr.34107) 3327 : cluster [DBG] pgmap v4835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:32.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:31.977Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:31 smithi115 bash[45495]: cluster 2022-04-22T19:02:30.706654+0000 mgr.y (mgr.34107) 3327 : cluster [DBG] pgmap v4835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:31 smithi121 bash[36902]: cluster 2022-04-22T19:02:30.706654+0000 mgr.y (mgr.34107) 3327 : cluster [DBG] pgmap v4835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:33 smithi121 bash[36902]: cluster 2022-04-22T19:02:32.707853+0000 mgr.y (mgr.34107) 3328 : cluster [DBG] pgmap v4836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:33 smithi115 bash[46934]: cluster 2022-04-22T19:02:32.707853+0000 mgr.y (mgr.34107) 3328 : cluster [DBG] pgmap v4836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:33 smithi115 bash[45495]: cluster 2022-04-22T19:02:32.707853+0000 mgr.y (mgr.34107) 3328 : cluster [DBG] pgmap v4836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:33.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:34 smithi121 bash[36902]: audit 2022-04-22T19:02:34.386067+0000 mon.a (mon.0) 6315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:34.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:34 smithi115 bash[46934]: audit 2022-04-22T19:02:34.386067+0000 mon.a (mon.0) 6315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:34.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:34 smithi115 bash[45495]: audit 2022-04-22T19:02:34.386067+0000 mon.a (mon.0) 6315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:35 smithi121 bash[36902]: cluster 2022-04-22T19:02:34.708964+0000 mgr.y (mgr.34107) 3329 : cluster [DBG] pgmap v4837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:35 smithi115 bash[45495]: cluster 2022-04-22T19:02:34.708964+0000 mgr.y (mgr.34107) 3329 : cluster [DBG] pgmap v4837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:35 smithi115 bash[46934]: cluster 2022-04-22T19:02:34.708964+0000 mgr.y (mgr.34107) 3329 : cluster [DBG] pgmap v4837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:37.283 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:02:37.627 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:02:37.628 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (85m) 2m ago 99m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:02:37.628 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (99m) 3m ago 99m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:02:37.628 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (87m) 2m ago 100m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:02:37.628 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (87m) 3m ago 100m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:02:37.628 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (94m) 3m ago 104m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:02:37.629 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (95m) 2m ago 106m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:02:37.629 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (94m) 2m ago 107m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:02:37.629 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (93m) 3m ago 105m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:02:37.629 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (94m) 2m ago 105m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (86m) 2m ago 99m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (86m) 3m ago 99m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (92m) 2m ago 104m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (91m) 2m ago 103m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (91m) 2m ago 103m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:02:37.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (90m) 2m ago 102m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:02:37.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (90m) 3m ago 102m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:02:37.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (89m) 3m ago 102m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:02:37.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (88m) 3m ago 101m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:02:37.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (88m) 3m ago 101m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:02:37.631 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (86m) 3m ago 99m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:02:38.027 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:02:38.028 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:02:38.029 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:02:38.029 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:38.030 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:02:38.030 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:02:38.030 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:38.031 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:02:38.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:02:38.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:38.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:02:38.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:02:38.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:02:38.032 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:02:38.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:02:38.032 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:02:38.033 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:02:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:37 smithi115 bash[46934]: cluster 2022-04-22T19:02:36.710073+0000 mgr.y (mgr.34107) 3330 : cluster [DBG] pgmap v4838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:37 smithi115 bash[45495]: cluster 2022-04-22T19:02:36.710073+0000 mgr.y (mgr.34107) 3330 : cluster [DBG] pgmap v4838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:37 smithi121 bash[36902]: cluster 2022-04-22T19:02:36.710073+0000 mgr.y (mgr.34107) 3330 : cluster [DBG] pgmap v4838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[45495]: audit 2022-04-22T19:02:37.257995+0000 mgr.y (mgr.34107) 3331 : audit [DBG] from='client.48832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[45495]: audit 2022-04-22T19:02:37.600784+0000 mgr.y (mgr.34107) 3332 : audit [DBG] from='client.48838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[45495]: audit 2022-04-22T19:02:38.017781+0000 mon.a (mon.0) 6316 : audit [DBG] from='client.? 172.21.15.115:0/1068396947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[46934]: audit 2022-04-22T19:02:37.257995+0000 mgr.y (mgr.34107) 3331 : audit [DBG] from='client.48832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[46934]: audit 2022-04-22T19:02:37.600784+0000 mgr.y (mgr.34107) 3332 : audit [DBG] from='client.48838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:38 smithi115 bash[46934]: audit 2022-04-22T19:02:38.017781+0000 mon.a (mon.0) 6316 : audit [DBG] from='client.? 172.21.15.115:0/1068396947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:38 smithi121 bash[36902]: audit 2022-04-22T19:02:37.257995+0000 mgr.y (mgr.34107) 3331 : audit [DBG] from='client.48832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:38 smithi121 bash[36902]: audit 2022-04-22T19:02:37.600784+0000 mgr.y (mgr.34107) 3332 : audit [DBG] from='client.48838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:02:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:38 smithi121 bash[36902]: audit 2022-04-22T19:02:38.017781+0000 mon.a (mon.0) 6316 : audit [DBG] from='client.? 172.21.15.115:0/1068396947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:02:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:39 smithi115 bash[45495]: cluster 2022-04-22T19:02:38.711187+0000 mgr.y (mgr.34107) 3333 : cluster [DBG] pgmap v4839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:39 smithi115 bash[45495]: audit 2022-04-22T19:02:39.389177+0000 mon.a (mon.0) 6317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:39 smithi115 bash[46934]: cluster 2022-04-22T19:02:38.711187+0000 mgr.y (mgr.34107) 3333 : cluster [DBG] pgmap v4839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:39 smithi115 bash[46934]: audit 2022-04-22T19:02:39.389177+0000 mon.a (mon.0) 6317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:39 smithi121 bash[36902]: cluster 2022-04-22T19:02:38.711187+0000 mgr.y (mgr.34107) 3333 : cluster [DBG] pgmap v4839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:39 smithi121 bash[36902]: audit 2022-04-22T19:02:39.389177+0000 mon.a (mon.0) 6317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:41 smithi115 bash[45495]: cluster 2022-04-22T19:02:40.712301+0000 mgr.y (mgr.34107) 3334 : cluster [DBG] pgmap v4840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:42.047 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:41.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:41 smithi115 bash[46934]: cluster 2022-04-22T19:02:40.712301+0000 mgr.y (mgr.34107) 3334 : cluster [DBG] pgmap v4840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:41 smithi121 bash[36902]: cluster 2022-04-22T19:02:40.712301+0000 mgr.y (mgr.34107) 3334 : cluster [DBG] pgmap v4840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:42 smithi115 bash[45495]: audit 2022-04-22T19:02:42.006912+0000 mon.a (mon.0) 6318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:43.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:42 smithi115 bash[46934]: audit 2022-04-22T19:02:42.006912+0000 mon.a (mon.0) 6318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:42 smithi121 bash[36902]: audit 2022-04-22T19:02:42.006912+0000 mon.a (mon.0) 6318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:44.046 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:43.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:43 smithi115 bash[45495]: cluster 2022-04-22T19:02:42.713371+0000 mgr.y (mgr.34107) 3335 : cluster [DBG] pgmap v4841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:43 smithi115 bash[46934]: cluster 2022-04-22T19:02:42.713371+0000 mgr.y (mgr.34107) 3335 : cluster [DBG] pgmap v4841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:43 smithi121 bash[36902]: cluster 2022-04-22T19:02:42.713371+0000 mgr.y (mgr.34107) 3335 : cluster [DBG] pgmap v4841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:44 smithi121 bash[36902]: audit 2022-04-22T19:02:44.392314+0000 mon.a (mon.0) 6319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:44 smithi115 bash[45495]: audit 2022-04-22T19:02:44.392314+0000 mon.a (mon.0) 6319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:44 smithi115 bash[46934]: audit 2022-04-22T19:02:44.392314+0000 mon.a (mon.0) 6319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:45 smithi121 bash[36902]: cluster 2022-04-22T19:02:44.714444+0000 mgr.y (mgr.34107) 3336 : cluster [DBG] pgmap v4842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:45 smithi115 bash[45495]: cluster 2022-04-22T19:02:44.714444+0000 mgr.y (mgr.34107) 3336 : cluster [DBG] pgmap v4842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:45 smithi115 bash[46934]: cluster 2022-04-22T19:02:44.714444+0000 mgr.y (mgr.34107) 3336 : cluster [DBG] pgmap v4842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:47 smithi121 bash[36902]: cluster 2022-04-22T19:02:46.715532+0000 mgr.y (mgr.34107) 3337 : cluster [DBG] pgmap v4843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:47 smithi121 bash[36902]: audit 2022-04-22T19:02:47.010268+0000 mon.a (mon.0) 6320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:47 smithi115 bash[46934]: cluster 2022-04-22T19:02:46.715532+0000 mgr.y (mgr.34107) 3337 : cluster [DBG] pgmap v4843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:47 smithi115 bash[46934]: audit 2022-04-22T19:02:47.010268+0000 mon.a (mon.0) 6320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:47 smithi115 bash[45495]: cluster 2022-04-22T19:02:46.715532+0000 mgr.y (mgr.34107) 3337 : cluster [DBG] pgmap v4843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:47 smithi115 bash[45495]: audit 2022-04-22T19:02:47.010268+0000 mon.a (mon.0) 6320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:49 smithi121 bash[36902]: cluster 2022-04-22T19:02:48.716530+0000 mgr.y (mgr.34107) 3338 : cluster [DBG] pgmap v4844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:49 smithi121 bash[36902]: audit 2022-04-22T19:02:49.395375+0000 mon.a (mon.0) 6321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:49 smithi115 bash[45495]: cluster 2022-04-22T19:02:48.716530+0000 mgr.y (mgr.34107) 3338 : cluster [DBG] pgmap v4844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:49 smithi115 bash[45495]: audit 2022-04-22T19:02:49.395375+0000 mon.a (mon.0) 6321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:49 smithi115 bash[46934]: cluster 2022-04-22T19:02:48.716530+0000 mgr.y (mgr.34107) 3338 : cluster [DBG] pgmap v4844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:49 smithi115 bash[46934]: audit 2022-04-22T19:02:49.395375+0000 mon.a (mon.0) 6321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:52.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:51 smithi121 bash[36902]: cluster 2022-04-22T19:02:50.717627+0000 mgr.y (mgr.34107) 3339 : cluster [DBG] pgmap v4845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:52.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:51 smithi115 bash[45495]: cluster 2022-04-22T19:02:50.717627+0000 mgr.y (mgr.34107) 3339 : cluster [DBG] pgmap v4845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:51.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:51 smithi115 bash[46934]: cluster 2022-04-22T19:02:50.717627+0000 mgr.y (mgr.34107) 3339 : cluster [DBG] pgmap v4845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:52 smithi121 bash[36902]: audit 2022-04-22T19:02:52.013347+0000 mon.a (mon.0) 6322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:52 smithi115 bash[45495]: audit 2022-04-22T19:02:52.013347+0000 mon.a (mon.0) 6322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:52 smithi115 bash[46934]: audit 2022-04-22T19:02:52.013347+0000 mon.a (mon.0) 6322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:53 smithi121 bash[36902]: cluster 2022-04-22T19:02:52.718738+0000 mgr.y (mgr.34107) 3340 : cluster [DBG] pgmap v4846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:02:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:02:53.852Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:02:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:53 smithi115 bash[45495]: cluster 2022-04-22T19:02:52.718738+0000 mgr.y (mgr.34107) 3340 : cluster [DBG] pgmap v4846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:53 smithi115 bash[46934]: cluster 2022-04-22T19:02:52.718738+0000 mgr.y (mgr.34107) 3340 : cluster [DBG] pgmap v4846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:54 smithi121 bash[36902]: audit 2022-04-22T19:02:54.398449+0000 mon.a (mon.0) 6323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:54 smithi115 bash[45495]: audit 2022-04-22T19:02:54.398449+0000 mon.a (mon.0) 6323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:54 smithi115 bash[46934]: audit 2022-04-22T19:02:54.398449+0000 mon.a (mon.0) 6323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:02:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:55 smithi121 bash[36902]: cluster 2022-04-22T19:02:54.719890+0000 mgr.y (mgr.34107) 3341 : cluster [DBG] pgmap v4847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:55 smithi115 bash[45495]: cluster 2022-04-22T19:02:54.719890+0000 mgr.y (mgr.34107) 3341 : cluster [DBG] pgmap v4847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:55 smithi115 bash[46934]: cluster 2022-04-22T19:02:54.719890+0000 mgr.y (mgr.34107) 3341 : cluster [DBG] pgmap v4847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:57 smithi121 bash[36902]: cluster 2022-04-22T19:02:56.721002+0000 mgr.y (mgr.34107) 3342 : cluster [DBG] pgmap v4848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:57 smithi121 bash[36902]: audit 2022-04-22T19:02:57.016566+0000 mon.a (mon.0) 6324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:57 smithi115 bash[45495]: cluster 2022-04-22T19:02:56.721002+0000 mgr.y (mgr.34107) 3342 : cluster [DBG] pgmap v4848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:57 smithi115 bash[45495]: audit 2022-04-22T19:02:57.016566+0000 mon.a (mon.0) 6324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:02:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:57 smithi115 bash[46934]: cluster 2022-04-22T19:02:56.721002+0000 mgr.y (mgr.34107) 3342 : cluster [DBG] pgmap v4848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:02:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:57 smithi115 bash[46934]: audit 2022-04-22T19:02:57.016566+0000 mon.a (mon.0) 6324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:59 smithi121 bash[36902]: cluster 2022-04-22T19:02:58.722144+0000 mgr.y (mgr.34107) 3343 : cluster [DBG] pgmap v4849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:00.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:02:59 smithi121 bash[36902]: audit 2022-04-22T19:02:59.401654+0000 mon.a (mon.0) 6325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:00.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:59 smithi115 bash[45495]: cluster 2022-04-22T19:02:58.722144+0000 mgr.y (mgr.34107) 3343 : cluster [DBG] pgmap v4849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:00.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:02:59 smithi115 bash[45495]: audit 2022-04-22T19:02:59.401654+0000 mon.a (mon.0) 6325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:00.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:59 smithi115 bash[46934]: cluster 2022-04-22T19:02:58.722144+0000 mgr.y (mgr.34107) 3343 : cluster [DBG] pgmap v4849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:00.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:02:59 smithi115 bash[46934]: audit 2022-04-22T19:02:59.401654+0000 mon.a (mon.0) 6325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:01 smithi121 bash[36902]: cluster 2022-04-22T19:03:00.723246+0000 mgr.y (mgr.34107) 3344 : cluster [DBG] pgmap v4850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:01 smithi121 bash[36902]: audit 2022-04-22T19:03:00.851879+0000 mon.a (mon.0) 6326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:03:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:01 smithi121 bash[36902]: audit 2022-04-22T19:03:00.854691+0000 mon.a (mon.0) 6327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:03:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:01 smithi121 bash[36902]: audit 2022-04-22T19:03:00.856351+0000 mon.a (mon.0) 6328 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:03:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:01 smithi121 bash[36902]: audit 2022-04-22T19:03:01.226333+0000 mon.a (mon.0) 6329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:03:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:01.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[45495]: cluster 2022-04-22T19:03:00.723246+0000 mgr.y (mgr.34107) 3344 : cluster [DBG] pgmap v4850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[45495]: audit 2022-04-22T19:03:00.851879+0000 mon.a (mon.0) 6326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:03:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[45495]: audit 2022-04-22T19:03:00.854691+0000 mon.a (mon.0) 6327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:03:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[45495]: audit 2022-04-22T19:03:00.856351+0000 mon.a (mon.0) 6328 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:03:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[45495]: audit 2022-04-22T19:03:01.226333+0000 mon.a (mon.0) 6329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:03:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[46934]: cluster 2022-04-22T19:03:00.723246+0000 mgr.y (mgr.34107) 3344 : cluster [DBG] pgmap v4850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[46934]: audit 2022-04-22T19:03:00.851879+0000 mon.a (mon.0) 6326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:03:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[46934]: audit 2022-04-22T19:03:00.854691+0000 mon.a (mon.0) 6327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:03:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[46934]: audit 2022-04-22T19:03:00.856351+0000 mon.a (mon.0) 6328 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:03:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:01 smithi115 bash[46934]: audit 2022-04-22T19:03:01.226333+0000 mon.a (mon.0) 6329 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:03:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:02 smithi121 bash[36902]: audit 2022-04-22T19:03:02.019627+0000 mon.a (mon.0) 6330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:02 smithi115 bash[45495]: audit 2022-04-22T19:03:02.019627+0000 mon.a (mon.0) 6330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:02 smithi115 bash[46934]: audit 2022-04-22T19:03:02.019627+0000 mon.a (mon.0) 6330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:03 smithi121 bash[36902]: cluster 2022-04-22T19:03:02.724367+0000 mgr.y (mgr.34107) 3345 : cluster [DBG] pgmap v4851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:03 smithi115 bash[46934]: cluster 2022-04-22T19:03:02.724367+0000 mgr.y (mgr.34107) 3345 : cluster [DBG] pgmap v4851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:03.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:04.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:03 smithi115 bash[45495]: cluster 2022-04-22T19:03:02.724367+0000 mgr.y (mgr.34107) 3345 : cluster [DBG] pgmap v4851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:05.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:04 smithi121 bash[36902]: audit 2022-04-22T19:03:04.404682+0000 mon.a (mon.0) 6331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:04 smithi115 bash[45495]: audit 2022-04-22T19:03:04.404682+0000 mon.a (mon.0) 6331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:04 smithi115 bash[46934]: audit 2022-04-22T19:03:04.404682+0000 mon.a (mon.0) 6331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:05 smithi121 bash[36902]: cluster 2022-04-22T19:03:04.725464+0000 mgr.y (mgr.34107) 3346 : cluster [DBG] pgmap v4852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:05 smithi115 bash[45495]: cluster 2022-04-22T19:03:04.725464+0000 mgr.y (mgr.34107) 3346 : cluster [DBG] pgmap v4852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:05 smithi115 bash[46934]: cluster 2022-04-22T19:03:04.725464+0000 mgr.y (mgr.34107) 3346 : cluster [DBG] pgmap v4852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:07 smithi121 bash[36902]: cluster 2022-04-22T19:03:06.726618+0000 mgr.y (mgr.34107) 3347 : cluster [DBG] pgmap v4853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:08.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:07 smithi121 bash[36902]: audit 2022-04-22T19:03:07.022675+0000 mon.a (mon.0) 6332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:07 smithi115 bash[45495]: cluster 2022-04-22T19:03:06.726618+0000 mgr.y (mgr.34107) 3347 : cluster [DBG] pgmap v4853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:07 smithi115 bash[45495]: audit 2022-04-22T19:03:07.022675+0000 mon.a (mon.0) 6332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:08.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:07 smithi115 bash[46934]: cluster 2022-04-22T19:03:06.726618+0000 mgr.y (mgr.34107) 3347 : cluster [DBG] pgmap v4853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:08.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:07 smithi115 bash[46934]: audit 2022-04-22T19:03:07.022675+0000 mon.a (mon.0) 6332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:08.395 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (86m) 3m ago 100m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (99m) 4m ago 99m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (88m) 3m ago 101m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (87m) 4m ago 100m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (95m) 4m ago 105m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:03:08.740 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (96m) 3m ago 107m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (95m) 3m ago 107m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (94m) 4m ago 105m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (94m) 3m ago 105m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (87m) 3m ago 100m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (87m) 4m ago 100m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (92m) 3m ago 104m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (92m) 3m ago 104m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (91m) 3m ago 103m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (91m) 3m ago 103m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (90m) 4m ago 102m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:03:08.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (89m) 4m ago 102m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:03:08.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (89m) 4m ago 102m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:03:08.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (88m) 4m ago 101m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:03:08.742 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (86m) 4m ago 100m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:03:09.146 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:03:09.146 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:03:09.147 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:03:09.148 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:09.148 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:03:09.148 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:03:09.148 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:03:09.148 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:09 smithi121 bash[36902]: audit 2022-04-22T19:03:08.370001+0000 mgr.y (mgr.34107) 3348 : audit [DBG] from='client.48850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:09 smithi121 bash[36902]: audit 2022-04-22T19:03:08.711896+0000 mgr.y (mgr.34107) 3349 : audit [DBG] from='client.48856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:09 smithi121 bash[36902]: cluster 2022-04-22T19:03:08.727553+0000 mgr.y (mgr.34107) 3350 : cluster [DBG] pgmap v4854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:09 smithi121 bash[36902]: audit 2022-04-22T19:03:09.136522+0000 mon.a (mon.0) 6333 : audit [DBG] from='client.? 172.21.15.115:0/1692242365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:09 smithi121 bash[36902]: audit 2022-04-22T19:03:09.408076+0000 mon.a (mon.0) 6334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[45495]: audit 2022-04-22T19:03:08.370001+0000 mgr.y (mgr.34107) 3348 : audit [DBG] from='client.48850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[45495]: audit 2022-04-22T19:03:08.711896+0000 mgr.y (mgr.34107) 3349 : audit [DBG] from='client.48856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[45495]: cluster 2022-04-22T19:03:08.727553+0000 mgr.y (mgr.34107) 3350 : cluster [DBG] pgmap v4854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[45495]: audit 2022-04-22T19:03:09.136522+0000 mon.a (mon.0) 6333 : audit [DBG] from='client.? 172.21.15.115:0/1692242365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[45495]: audit 2022-04-22T19:03:09.408076+0000 mon.a (mon.0) 6334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[46934]: audit 2022-04-22T19:03:08.370001+0000 mgr.y (mgr.34107) 3348 : audit [DBG] from='client.48850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[46934]: audit 2022-04-22T19:03:08.711896+0000 mgr.y (mgr.34107) 3349 : audit [DBG] from='client.48856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[46934]: cluster 2022-04-22T19:03:08.727553+0000 mgr.y (mgr.34107) 3350 : cluster [DBG] pgmap v4854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:10.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[46934]: audit 2022-04-22T19:03:09.136522+0000 mon.a (mon.0) 6333 : audit [DBG] from='client.? 172.21.15.115:0/1692242365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:10.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:09 smithi115 bash[46934]: audit 2022-04-22T19:03:09.408076+0000 mon.a (mon.0) 6334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:11 smithi121 bash[36902]: cluster 2022-04-22T19:03:10.728666+0000 mgr.y (mgr.34107) 3351 : cluster [DBG] pgmap v4855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:12.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:11 smithi115 bash[45495]: cluster 2022-04-22T19:03:10.728666+0000 mgr.y (mgr.34107) 3351 : cluster [DBG] pgmap v4855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:12.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:11 smithi115 bash[46934]: cluster 2022-04-22T19:03:10.728666+0000 mgr.y (mgr.34107) 3351 : cluster [DBG] pgmap v4855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:11.981Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:12 smithi121 bash[36902]: audit 2022-04-22T19:03:12.025834+0000 mon.a (mon.0) 6335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:12 smithi115 bash[45495]: audit 2022-04-22T19:03:12.025834+0000 mon.a (mon.0) 6335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:12 smithi115 bash[46934]: audit 2022-04-22T19:03:12.025834+0000 mon.a (mon.0) 6335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:13 smithi121 bash[36902]: cluster 2022-04-22T19:03:12.729830+0000 mgr.y (mgr.34107) 3352 : cluster [DBG] pgmap v4856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:13 smithi115 bash[45495]: cluster 2022-04-22T19:03:12.729830+0000 mgr.y (mgr.34107) 3352 : cluster [DBG] pgmap v4856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:13.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:13 smithi115 bash[46934]: cluster 2022-04-22T19:03:12.729830+0000 mgr.y (mgr.34107) 3352 : cluster [DBG] pgmap v4856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:14 smithi121 bash[36902]: audit 2022-04-22T19:03:14.361661+0000 mon.a (mon.0) 6336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:03:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:14 smithi121 bash[36902]: audit 2022-04-22T19:03:14.362245+0000 mon.a (mon.0) 6337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:03:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:14 smithi121 bash[36902]: audit 2022-04-22T19:03:14.411072+0000 mon.a (mon.0) 6338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[45495]: audit 2022-04-22T19:03:14.361661+0000 mon.a (mon.0) 6336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:03:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[45495]: audit 2022-04-22T19:03:14.362245+0000 mon.a (mon.0) 6337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:03:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[45495]: audit 2022-04-22T19:03:14.411072+0000 mon.a (mon.0) 6338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[46934]: audit 2022-04-22T19:03:14.361661+0000 mon.a (mon.0) 6336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:03:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[46934]: audit 2022-04-22T19:03:14.362245+0000 mon.a (mon.0) 6337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:03:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:14 smithi115 bash[46934]: audit 2022-04-22T19:03:14.411072+0000 mon.a (mon.0) 6338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:15 smithi121 bash[36902]: cluster 2022-04-22T19:03:14.730902+0000 mgr.y (mgr.34107) 3353 : cluster [DBG] pgmap v4857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:15 smithi115 bash[46934]: cluster 2022-04-22T19:03:14.730902+0000 mgr.y (mgr.34107) 3353 : cluster [DBG] pgmap v4857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:15 smithi115 bash[45495]: cluster 2022-04-22T19:03:14.730902+0000 mgr.y (mgr.34107) 3353 : cluster [DBG] pgmap v4857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:17 smithi121 bash[36902]: cluster 2022-04-22T19:03:16.732066+0000 mgr.y (mgr.34107) 3354 : cluster [DBG] pgmap v4858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:17 smithi121 bash[36902]: audit 2022-04-22T19:03:17.028822+0000 mon.a (mon.0) 6339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:17 smithi115 bash[45495]: cluster 2022-04-22T19:03:16.732066+0000 mgr.y (mgr.34107) 3354 : cluster [DBG] pgmap v4858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:17 smithi115 bash[45495]: audit 2022-04-22T19:03:17.028822+0000 mon.a (mon.0) 6339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:17 smithi115 bash[46934]: cluster 2022-04-22T19:03:16.732066+0000 mgr.y (mgr.34107) 3354 : cluster [DBG] pgmap v4858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:17 smithi115 bash[46934]: audit 2022-04-22T19:03:17.028822+0000 mon.a (mon.0) 6339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:19 smithi121 bash[36902]: cluster 2022-04-22T19:03:18.733236+0000 mgr.y (mgr.34107) 3355 : cluster [DBG] pgmap v4859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:19 smithi121 bash[36902]: audit 2022-04-22T19:03:19.414269+0000 mon.a (mon.0) 6340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:19 smithi115 bash[46934]: cluster 2022-04-22T19:03:18.733236+0000 mgr.y (mgr.34107) 3355 : cluster [DBG] pgmap v4859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:19 smithi115 bash[46934]: audit 2022-04-22T19:03:19.414269+0000 mon.a (mon.0) 6340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:19 smithi115 bash[45495]: cluster 2022-04-22T19:03:18.733236+0000 mgr.y (mgr.34107) 3355 : cluster [DBG] pgmap v4859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:19 smithi115 bash[45495]: audit 2022-04-22T19:03:19.414269+0000 mon.a (mon.0) 6340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:21 smithi121 bash[36902]: cluster 2022-04-22T19:03:20.734331+0000 mgr.y (mgr.34107) 3356 : cluster [DBG] pgmap v4860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:22.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:21 smithi115 bash[46934]: cluster 2022-04-22T19:03:20.734331+0000 mgr.y (mgr.34107) 3356 : cluster [DBG] pgmap v4860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:21.980Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:21 smithi115 bash[45495]: cluster 2022-04-22T19:03:20.734331+0000 mgr.y (mgr.34107) 3356 : cluster [DBG] pgmap v4860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:22 smithi121 bash[36902]: audit 2022-04-22T19:03:22.031707+0000 mon.a (mon.0) 6341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:22 smithi115 bash[45495]: audit 2022-04-22T19:03:22.031707+0000 mon.a (mon.0) 6341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:22 smithi115 bash[46934]: audit 2022-04-22T19:03:22.031707+0000 mon.a (mon.0) 6341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:23 smithi121 bash[36902]: cluster 2022-04-22T19:03:22.735533+0000 mgr.y (mgr.34107) 3357 : cluster [DBG] pgmap v4861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:23 smithi115 bash[45495]: cluster 2022-04-22T19:03:22.735533+0000 mgr.y (mgr.34107) 3357 : cluster [DBG] pgmap v4861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:23.853Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:23 smithi115 bash[46934]: cluster 2022-04-22T19:03:22.735533+0000 mgr.y (mgr.34107) 3357 : cluster [DBG] pgmap v4861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:24 smithi121 bash[36902]: audit 2022-04-22T19:03:24.417578+0000 mon.a (mon.0) 6342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:24 smithi115 bash[45495]: audit 2022-04-22T19:03:24.417578+0000 mon.a (mon.0) 6342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:24 smithi115 bash[46934]: audit 2022-04-22T19:03:24.417578+0000 mon.a (mon.0) 6342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:25 smithi121 bash[36902]: cluster 2022-04-22T19:03:24.736573+0000 mgr.y (mgr.34107) 3358 : cluster [DBG] pgmap v4862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:25 smithi115 bash[45495]: cluster 2022-04-22T19:03:24.736573+0000 mgr.y (mgr.34107) 3358 : cluster [DBG] pgmap v4862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:25 smithi115 bash[46934]: cluster 2022-04-22T19:03:24.736573+0000 mgr.y (mgr.34107) 3358 : cluster [DBG] pgmap v4862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:27 smithi121 bash[36902]: cluster 2022-04-22T19:03:26.737703+0000 mgr.y (mgr.34107) 3359 : cluster [DBG] pgmap v4863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:27 smithi121 bash[36902]: audit 2022-04-22T19:03:27.034904+0000 mon.a (mon.0) 6343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:27 smithi115 bash[45495]: cluster 2022-04-22T19:03:26.737703+0000 mgr.y (mgr.34107) 3359 : cluster [DBG] pgmap v4863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:27 smithi115 bash[45495]: audit 2022-04-22T19:03:27.034904+0000 mon.a (mon.0) 6343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:27 smithi115 bash[46934]: cluster 2022-04-22T19:03:26.737703+0000 mgr.y (mgr.34107) 3359 : cluster [DBG] pgmap v4863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:27 smithi115 bash[46934]: audit 2022-04-22T19:03:27.034904+0000 mon.a (mon.0) 6343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:29 smithi121 bash[36902]: cluster 2022-04-22T19:03:28.738857+0000 mgr.y (mgr.34107) 3360 : cluster [DBG] pgmap v4864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:29 smithi121 bash[36902]: audit 2022-04-22T19:03:29.420581+0000 mon.a (mon.0) 6344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:29 smithi115 bash[45495]: cluster 2022-04-22T19:03:28.738857+0000 mgr.y (mgr.34107) 3360 : cluster [DBG] pgmap v4864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:29 smithi115 bash[45495]: audit 2022-04-22T19:03:29.420581+0000 mon.a (mon.0) 6344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:29 smithi115 bash[46934]: cluster 2022-04-22T19:03:28.738857+0000 mgr.y (mgr.34107) 3360 : cluster [DBG] pgmap v4864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:29 smithi115 bash[46934]: audit 2022-04-22T19:03:29.420581+0000 mon.a (mon.0) 6344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:31 smithi121 bash[36902]: cluster 2022-04-22T19:03:30.739997+0000 mgr.y (mgr.34107) 3361 : cluster [DBG] pgmap v4865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:32.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:31 smithi115 bash[45495]: cluster 2022-04-22T19:03:30.739997+0000 mgr.y (mgr.34107) 3361 : cluster [DBG] pgmap v4865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:31.981Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:31 smithi115 bash[46934]: cluster 2022-04-22T19:03:30.739997+0000 mgr.y (mgr.34107) 3361 : cluster [DBG] pgmap v4865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:32 smithi121 bash[36902]: audit 2022-04-22T19:03:32.038033+0000 mon.a (mon.0) 6345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:32 smithi115 bash[45495]: audit 2022-04-22T19:03:32.038033+0000 mon.a (mon.0) 6345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:32 smithi115 bash[46934]: audit 2022-04-22T19:03:32.038033+0000 mon.a (mon.0) 6345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:33 smithi121 bash[36902]: cluster 2022-04-22T19:03:32.741190+0000 mgr.y (mgr.34107) 3362 : cluster [DBG] pgmap v4866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:33 smithi115 bash[45495]: cluster 2022-04-22T19:03:32.741190+0000 mgr.y (mgr.34107) 3362 : cluster [DBG] pgmap v4866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:33 smithi115 bash[46934]: cluster 2022-04-22T19:03:32.741190+0000 mgr.y (mgr.34107) 3362 : cluster [DBG] pgmap v4866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:33.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:34 smithi121 bash[36902]: audit 2022-04-22T19:03:34.423637+0000 mon.a (mon.0) 6346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:34 smithi115 bash[45495]: audit 2022-04-22T19:03:34.423637+0000 mon.a (mon.0) 6346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:34 smithi115 bash[46934]: audit 2022-04-22T19:03:34.423637+0000 mon.a (mon.0) 6346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:35 smithi121 bash[36902]: cluster 2022-04-22T19:03:34.742388+0000 mgr.y (mgr.34107) 3363 : cluster [DBG] pgmap v4867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:35 smithi115 bash[45495]: cluster 2022-04-22T19:03:34.742388+0000 mgr.y (mgr.34107) 3363 : cluster [DBG] pgmap v4867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:36.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:35 smithi115 bash[46934]: cluster 2022-04-22T19:03:34.742388+0000 mgr.y (mgr.34107) 3363 : cluster [DBG] pgmap v4867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:37 smithi121 bash[36902]: cluster 2022-04-22T19:03:36.743561+0000 mgr.y (mgr.34107) 3364 : cluster [DBG] pgmap v4868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:37 smithi121 bash[36902]: audit 2022-04-22T19:03:37.041190+0000 mon.a (mon.0) 6347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:37 smithi115 bash[45495]: cluster 2022-04-22T19:03:36.743561+0000 mgr.y (mgr.34107) 3364 : cluster [DBG] pgmap v4868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:37 smithi115 bash[45495]: audit 2022-04-22T19:03:37.041190+0000 mon.a (mon.0) 6347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:37 smithi115 bash[46934]: cluster 2022-04-22T19:03:36.743561+0000 mgr.y (mgr.34107) 3364 : cluster [DBG] pgmap v4868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:37 smithi115 bash[46934]: audit 2022-04-22T19:03:37.041190+0000 mon.a (mon.0) 6347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:39.525 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (86m) 3m ago 100m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (100m) 4m ago 100m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (88m) 3m ago 101m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (88m) 4m ago 101m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (95m) 4m ago 105m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:03:39.869 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (96m) 3m ago 107m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (95m) 3m ago 108m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (94m) 4m ago 106m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (95m) 3m ago 106m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (87m) 3m ago 100m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (87m) 4m ago 100m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (93m) 3m ago 105m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:03:39.870 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (92m) 3m ago 104m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (92m) 3m ago 104m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (91m) 3m ago 103m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (91m) 4m ago 103m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (90m) 4m ago 103m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (89m) 4m ago 102m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (89m) 4m ago 102m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:03:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (87m) 4m ago 100m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:03:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:39 smithi121 bash[36902]: cluster 2022-04-22T19:03:38.744681+0000 mgr.y (mgr.34107) 3365 : cluster [DBG] pgmap v4869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:39 smithi121 bash[36902]: audit 2022-04-22T19:03:39.426581+0000 mon.a (mon.0) 6348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:40.267 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:03:40.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:03:40.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:03:40.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:03:40.268 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:03:40.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:03:40.269 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:03:40.269 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:03:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:39 smithi115 bash[45495]: cluster 2022-04-22T19:03:38.744681+0000 mgr.y (mgr.34107) 3365 : cluster [DBG] pgmap v4869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:39 smithi115 bash[45495]: audit 2022-04-22T19:03:39.426581+0000 mon.a (mon.0) 6348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:39 smithi115 bash[46934]: cluster 2022-04-22T19:03:38.744681+0000 mgr.y (mgr.34107) 3365 : cluster [DBG] pgmap v4869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:39 smithi115 bash[46934]: audit 2022-04-22T19:03:39.426581+0000 mon.a (mon.0) 6348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:40 smithi121 bash[36902]: audit 2022-04-22T19:03:39.498891+0000 mgr.y (mgr.34107) 3366 : audit [DBG] from='client.48868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:40 smithi121 bash[36902]: audit 2022-04-22T19:03:39.838910+0000 mgr.y (mgr.34107) 3367 : audit [DBG] from='client.48874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:40 smithi121 bash[36902]: audit 2022-04-22T19:03:40.258164+0000 mon.a (mon.0) 6349 : audit [DBG] from='client.? 172.21.15.115:0/2650485415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[45495]: audit 2022-04-22T19:03:39.498891+0000 mgr.y (mgr.34107) 3366 : audit [DBG] from='client.48868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[45495]: audit 2022-04-22T19:03:39.838910+0000 mgr.y (mgr.34107) 3367 : audit [DBG] from='client.48874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[45495]: audit 2022-04-22T19:03:40.258164+0000 mon.a (mon.0) 6349 : audit [DBG] from='client.? 172.21.15.115:0/2650485415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[46934]: audit 2022-04-22T19:03:39.498891+0000 mgr.y (mgr.34107) 3366 : audit [DBG] from='client.48868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[46934]: audit 2022-04-22T19:03:39.838910+0000 mgr.y (mgr.34107) 3367 : audit [DBG] from='client.48874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:03:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:40 smithi115 bash[46934]: audit 2022-04-22T19:03:40.258164+0000 mon.a (mon.0) 6349 : audit [DBG] from='client.? 172.21.15.115:0/2650485415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:03:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:41 smithi121 bash[36902]: cluster 2022-04-22T19:03:40.745806+0000 mgr.y (mgr.34107) 3368 : cluster [DBG] pgmap v4870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:41 smithi115 bash[45495]: cluster 2022-04-22T19:03:40.745806+0000 mgr.y (mgr.34107) 3368 : cluster [DBG] pgmap v4870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:42.301 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:41.980Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:42.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:41 smithi115 bash[46934]: cluster 2022-04-22T19:03:40.745806+0000 mgr.y (mgr.34107) 3368 : cluster [DBG] pgmap v4870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:42 smithi121 bash[36902]: audit 2022-04-22T19:03:42.044260+0000 mon.a (mon.0) 6350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:43.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:42 smithi115 bash[45495]: audit 2022-04-22T19:03:42.044260+0000 mon.a (mon.0) 6350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:42 smithi115 bash[46934]: audit 2022-04-22T19:03:42.044260+0000 mon.a (mon.0) 6350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:43 smithi121 bash[36902]: cluster 2022-04-22T19:03:42.747074+0000 mgr.y (mgr.34107) 3369 : cluster [DBG] pgmap v4871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:43.856Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:43 smithi115 bash[46934]: cluster 2022-04-22T19:03:42.747074+0000 mgr.y (mgr.34107) 3369 : cluster [DBG] pgmap v4871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:43 smithi115 bash[45495]: cluster 2022-04-22T19:03:42.747074+0000 mgr.y (mgr.34107) 3369 : cluster [DBG] pgmap v4871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:44 smithi121 bash[36902]: audit 2022-04-22T19:03:44.429473+0000 mon.a (mon.0) 6351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:44 smithi115 bash[45495]: audit 2022-04-22T19:03:44.429473+0000 mon.a (mon.0) 6351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:44 smithi115 bash[46934]: audit 2022-04-22T19:03:44.429473+0000 mon.a (mon.0) 6351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:45 smithi121 bash[36902]: cluster 2022-04-22T19:03:44.748217+0000 mgr.y (mgr.34107) 3370 : cluster [DBG] pgmap v4872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:45 smithi115 bash[45495]: cluster 2022-04-22T19:03:44.748217+0000 mgr.y (mgr.34107) 3370 : cluster [DBG] pgmap v4872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:45 smithi115 bash[46934]: cluster 2022-04-22T19:03:44.748217+0000 mgr.y (mgr.34107) 3370 : cluster [DBG] pgmap v4872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:47 smithi121 bash[36902]: cluster 2022-04-22T19:03:46.749351+0000 mgr.y (mgr.34107) 3371 : cluster [DBG] pgmap v4873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:47 smithi121 bash[36902]: audit 2022-04-22T19:03:47.047039+0000 mon.a (mon.0) 6352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:47 smithi115 bash[45495]: cluster 2022-04-22T19:03:46.749351+0000 mgr.y (mgr.34107) 3371 : cluster [DBG] pgmap v4873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:47 smithi115 bash[45495]: audit 2022-04-22T19:03:47.047039+0000 mon.a (mon.0) 6352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:47 smithi115 bash[46934]: cluster 2022-04-22T19:03:46.749351+0000 mgr.y (mgr.34107) 3371 : cluster [DBG] pgmap v4873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:47 smithi115 bash[46934]: audit 2022-04-22T19:03:47.047039+0000 mon.a (mon.0) 6352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:49 smithi121 bash[36902]: cluster 2022-04-22T19:03:48.750524+0000 mgr.y (mgr.34107) 3372 : cluster [DBG] pgmap v4874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:49 smithi121 bash[36902]: audit 2022-04-22T19:03:49.432797+0000 mon.a (mon.0) 6353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:49 smithi115 bash[46934]: cluster 2022-04-22T19:03:48.750524+0000 mgr.y (mgr.34107) 3372 : cluster [DBG] pgmap v4874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:49 smithi115 bash[46934]: audit 2022-04-22T19:03:49.432797+0000 mon.a (mon.0) 6353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:49 smithi115 bash[45495]: cluster 2022-04-22T19:03:48.750524+0000 mgr.y (mgr.34107) 3372 : cluster [DBG] pgmap v4874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:49 smithi115 bash[45495]: audit 2022-04-22T19:03:49.432797+0000 mon.a (mon.0) 6353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:51 smithi121 bash[36902]: cluster 2022-04-22T19:03:50.751611+0000 mgr.y (mgr.34107) 3373 : cluster [DBG] pgmap v4875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:52.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:51 smithi115 bash[45495]: cluster 2022-04-22T19:03:50.751611+0000 mgr.y (mgr.34107) 3373 : cluster [DBG] pgmap v4875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:51.983Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:51 smithi115 bash[46934]: cluster 2022-04-22T19:03:50.751611+0000 mgr.y (mgr.34107) 3373 : cluster [DBG] pgmap v4875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:52 smithi121 bash[36902]: audit 2022-04-22T19:03:52.050222+0000 mon.a (mon.0) 6354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:52 smithi115 bash[45495]: audit 2022-04-22T19:03:52.050222+0000 mon.a (mon.0) 6354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:52 smithi115 bash[46934]: audit 2022-04-22T19:03:52.050222+0000 mon.a (mon.0) 6354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:53 smithi121 bash[36902]: cluster 2022-04-22T19:03:52.752777+0000 mgr.y (mgr.34107) 3374 : cluster [DBG] pgmap v4876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:54.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:53 smithi115 bash[45495]: cluster 2022-04-22T19:03:52.752777+0000 mgr.y (mgr.34107) 3374 : cluster [DBG] pgmap v4876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:03:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:03:53.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:03:54.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:53 smithi115 bash[46934]: cluster 2022-04-22T19:03:52.752777+0000 mgr.y (mgr.34107) 3374 : cluster [DBG] pgmap v4876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:55 smithi121 bash[36902]: audit 2022-04-22T19:03:54.435972+0000 mon.a (mon.0) 6355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:54 smithi115 bash[45495]: audit 2022-04-22T19:03:54.435972+0000 mon.a (mon.0) 6355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:55.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:55 smithi115 bash[46934]: audit 2022-04-22T19:03:54.435972+0000 mon.a (mon.0) 6355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:03:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:56 smithi121 bash[36902]: cluster 2022-04-22T19:03:54.753866+0000 mgr.y (mgr.34107) 3375 : cluster [DBG] pgmap v4877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:56 smithi115 bash[45495]: cluster 2022-04-22T19:03:54.753866+0000 mgr.y (mgr.34107) 3375 : cluster [DBG] pgmap v4877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:56 smithi115 bash[46934]: cluster 2022-04-22T19:03:54.753866+0000 mgr.y (mgr.34107) 3375 : cluster [DBG] pgmap v4877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:58 smithi121 bash[36902]: cluster 2022-04-22T19:03:56.754994+0000 mgr.y (mgr.34107) 3376 : cluster [DBG] pgmap v4878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:03:58 smithi121 bash[36902]: audit 2022-04-22T19:03:57.053162+0000 mon.a (mon.0) 6356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:58 smithi115 bash[45495]: cluster 2022-04-22T19:03:56.754994+0000 mgr.y (mgr.34107) 3376 : cluster [DBG] pgmap v4878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:03:58 smithi115 bash[45495]: audit 2022-04-22T19:03:57.053162+0000 mon.a (mon.0) 6356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:03:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:58 smithi115 bash[46934]: cluster 2022-04-22T19:03:56.754994+0000 mgr.y (mgr.34107) 3376 : cluster [DBG] pgmap v4878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:03:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:03:58 smithi115 bash[46934]: audit 2022-04-22T19:03:57.053162+0000 mon.a (mon.0) 6356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:00 smithi115 bash[45495]: cluster 2022-04-22T19:03:58.756113+0000 mgr.y (mgr.34107) 3377 : cluster [DBG] pgmap v4879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:00 smithi115 bash[45495]: audit 2022-04-22T19:03:59.438969+0000 mon.a (mon.0) 6357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:00 smithi115 bash[46934]: cluster 2022-04-22T19:03:58.756113+0000 mgr.y (mgr.34107) 3377 : cluster [DBG] pgmap v4879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:00 smithi115 bash[46934]: audit 2022-04-22T19:03:59.438969+0000 mon.a (mon.0) 6357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:00 smithi121 bash[36902]: cluster 2022-04-22T19:03:58.756113+0000 mgr.y (mgr.34107) 3377 : cluster [DBG] pgmap v4879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:00 smithi121 bash[36902]: audit 2022-04-22T19:03:59.438969+0000 mon.a (mon.0) 6357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:01.983Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[46934]: cluster 2022-04-22T19:04:00.757364+0000 mgr.y (mgr.34107) 3378 : cluster [DBG] pgmap v4880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[46934]: audit 2022-04-22T19:04:01.233938+0000 mon.a (mon.0) 6358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[46934]: audit 2022-04-22T19:04:01.236804+0000 mon.a (mon.0) 6359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[46934]: audit 2022-04-22T19:04:01.238492+0000 mon.a (mon.0) 6360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[46934]: audit 2022-04-22T19:04:01.594880+0000 mon.a (mon.0) 6361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[45495]: cluster 2022-04-22T19:04:00.757364+0000 mgr.y (mgr.34107) 3378 : cluster [DBG] pgmap v4880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[45495]: audit 2022-04-22T19:04:01.233938+0000 mon.a (mon.0) 6358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:04:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[45495]: audit 2022-04-22T19:04:01.236804+0000 mon.a (mon.0) 6359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:04:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[45495]: audit 2022-04-22T19:04:01.238492+0000 mon.a (mon.0) 6360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:04:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:02 smithi115 bash[45495]: audit 2022-04-22T19:04:01.594880+0000 mon.a (mon.0) 6361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:02 smithi121 bash[36902]: cluster 2022-04-22T19:04:00.757364+0000 mgr.y (mgr.34107) 3378 : cluster [DBG] pgmap v4880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:02 smithi121 bash[36902]: audit 2022-04-22T19:04:01.233938+0000 mon.a (mon.0) 6358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:02 smithi121 bash[36902]: audit 2022-04-22T19:04:01.236804+0000 mon.a (mon.0) 6359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:02 smithi121 bash[36902]: audit 2022-04-22T19:04:01.238492+0000 mon.a (mon.0) 6360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:04:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:02 smithi121 bash[36902]: audit 2022-04-22T19:04:01.594880+0000 mon.a (mon.0) 6361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:04:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:03 smithi115 bash[45495]: audit 2022-04-22T19:04:02.056026+0000 mon.a (mon.0) 6362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:03 smithi115 bash[46934]: audit 2022-04-22T19:04:02.056026+0000 mon.a (mon.0) 6362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:03 smithi121 bash[36902]: audit 2022-04-22T19:04:02.056026+0000 mon.a (mon.0) 6362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:04 smithi115 bash[45495]: cluster 2022-04-22T19:04:02.758571+0000 mgr.y (mgr.34107) 3379 : cluster [DBG] pgmap v4881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:03.855Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:04 smithi115 bash[46934]: cluster 2022-04-22T19:04:02.758571+0000 mgr.y (mgr.34107) 3379 : cluster [DBG] pgmap v4881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:04 smithi121 bash[36902]: cluster 2022-04-22T19:04:02.758571+0000 mgr.y (mgr.34107) 3379 : cluster [DBG] pgmap v4881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:05 smithi115 bash[45495]: audit 2022-04-22T19:04:04.442153+0000 mon.a (mon.0) 6363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:05.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:05 smithi115 bash[46934]: audit 2022-04-22T19:04:04.442153+0000 mon.a (mon.0) 6363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:05 smithi121 bash[36902]: audit 2022-04-22T19:04:04.442153+0000 mon.a (mon.0) 6363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:06 smithi115 bash[45495]: cluster 2022-04-22T19:04:04.759685+0000 mgr.y (mgr.34107) 3380 : cluster [DBG] pgmap v4882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:06 smithi115 bash[46934]: cluster 2022-04-22T19:04:04.759685+0000 mgr.y (mgr.34107) 3380 : cluster [DBG] pgmap v4882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:06 smithi121 bash[36902]: cluster 2022-04-22T19:04:04.759685+0000 mgr.y (mgr.34107) 3380 : cluster [DBG] pgmap v4882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:08 smithi121 bash[36902]: cluster 2022-04-22T19:04:06.760704+0000 mgr.y (mgr.34107) 3381 : cluster [DBG] pgmap v4883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:08 smithi121 bash[36902]: audit 2022-04-22T19:04:07.058837+0000 mon.a (mon.0) 6364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:08 smithi115 bash[45495]: cluster 2022-04-22T19:04:06.760704+0000 mgr.y (mgr.34107) 3381 : cluster [DBG] pgmap v4883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:08 smithi115 bash[45495]: audit 2022-04-22T19:04:07.058837+0000 mon.a (mon.0) 6364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:08 smithi115 bash[46934]: cluster 2022-04-22T19:04:06.760704+0000 mgr.y (mgr.34107) 3381 : cluster [DBG] pgmap v4883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:08 smithi115 bash[46934]: audit 2022-04-22T19:04:07.058837+0000 mon.a (mon.0) 6364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:10 smithi121 bash[36902]: cluster 2022-04-22T19:04:08.761844+0000 mgr.y (mgr.34107) 3382 : cluster [DBG] pgmap v4884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:10 smithi121 bash[36902]: audit 2022-04-22T19:04:09.445403+0000 mon.a (mon.0) 6365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:10 smithi115 bash[45495]: cluster 2022-04-22T19:04:08.761844+0000 mgr.y (mgr.34107) 3382 : cluster [DBG] pgmap v4884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:10 smithi115 bash[45495]: audit 2022-04-22T19:04:09.445403+0000 mon.a (mon.0) 6365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:10.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:10 smithi115 bash[46934]: cluster 2022-04-22T19:04:08.761844+0000 mgr.y (mgr.34107) 3382 : cluster [DBG] pgmap v4884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:10.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:10 smithi115 bash[46934]: audit 2022-04-22T19:04:09.445403+0000 mon.a (mon.0) 6365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:10.635 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:04:10.985 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:04:10.985 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (87m) 4m ago 101m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:04:10.985 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (100m) 5m ago 100m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (89m) 4m ago 102m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (88m) 5m ago 102m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (96m) 5m ago 106m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (97m) 4m ago 108m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (96m) 4m ago 108m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (95m) 5m ago 106m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (95m) 4m ago 106m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (88m) 4m ago 101m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:04:10.986 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (88m) 5m ago 101m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (93m) 4m ago 105m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (93m) 4m ago 105m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (92m) 4m ago 104m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (92m) 4m ago 104m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (91m) 5m ago 104m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (91m) 5m ago 103m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (90m) 5m ago 103m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (89m) 5m ago 102m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:04:10.987 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (87m) 5m ago 101m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:04:11.392 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:04:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:04:11.400 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:04:12.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:11.984Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[46934]: audit 2022-04-22T19:04:10.611141+0000 mgr.y (mgr.34107) 3383 : audit [DBG] from='client.38952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[46934]: cluster 2022-04-22T19:04:10.762876+0000 mgr.y (mgr.34107) 3384 : cluster [DBG] pgmap v4885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[46934]: audit 2022-04-22T19:04:10.955125+0000 mgr.y (mgr.34107) 3385 : audit [DBG] from='client.38958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[46934]: audit 2022-04-22T19:04:11.382868+0000 mon.a (mon.0) 6366 : audit [DBG] from='client.? 172.21.15.115:0/1695020437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[45495]: audit 2022-04-22T19:04:10.611141+0000 mgr.y (mgr.34107) 3383 : audit [DBG] from='client.38952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[45495]: cluster 2022-04-22T19:04:10.762876+0000 mgr.y (mgr.34107) 3384 : cluster [DBG] pgmap v4885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[45495]: audit 2022-04-22T19:04:10.955125+0000 mgr.y (mgr.34107) 3385 : audit [DBG] from='client.38958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:12 smithi115 bash[45495]: audit 2022-04-22T19:04:11.382868+0000 mon.a (mon.0) 6366 : audit [DBG] from='client.? 172.21.15.115:0/1695020437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:12 smithi121 bash[36902]: audit 2022-04-22T19:04:10.611141+0000 mgr.y (mgr.34107) 3383 : audit [DBG] from='client.38952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:12 smithi121 bash[36902]: cluster 2022-04-22T19:04:10.762876+0000 mgr.y (mgr.34107) 3384 : cluster [DBG] pgmap v4885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:12 smithi121 bash[36902]: audit 2022-04-22T19:04:10.955125+0000 mgr.y (mgr.34107) 3385 : audit [DBG] from='client.38958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:12 smithi121 bash[36902]: audit 2022-04-22T19:04:11.382868+0000 mon.a (mon.0) 6366 : audit [DBG] from='client.? 172.21.15.115:0/1695020437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:13 smithi121 bash[36902]: audit 2022-04-22T19:04:12.061654+0000 mon.a (mon.0) 6367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:13 smithi115 bash[45495]: audit 2022-04-22T19:04:12.061654+0000 mon.a (mon.0) 6367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:13 smithi115 bash[46934]: audit 2022-04-22T19:04:12.061654+0000 mon.a (mon.0) 6367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:14 smithi115 bash[45495]: cluster 2022-04-22T19:04:12.764018+0000 mgr.y (mgr.34107) 3386 : cluster [DBG] pgmap v4886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:13.856Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:14 smithi115 bash[46934]: cluster 2022-04-22T19:04:12.764018+0000 mgr.y (mgr.34107) 3386 : cluster [DBG] pgmap v4886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:14 smithi121 bash[36902]: cluster 2022-04-22T19:04:12.764018+0000 mgr.y (mgr.34107) 3386 : cluster [DBG] pgmap v4886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:15 smithi121 bash[36902]: audit 2022-04-22T19:04:14.373730+0000 mon.a (mon.0) 6368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:04:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:15 smithi121 bash[36902]: audit 2022-04-22T19:04:14.374552+0000 mon.a (mon.0) 6369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:04:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:15 smithi121 bash[36902]: audit 2022-04-22T19:04:14.448357+0000 mon.a (mon.0) 6370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[45495]: audit 2022-04-22T19:04:14.373730+0000 mon.a (mon.0) 6368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:04:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[45495]: audit 2022-04-22T19:04:14.374552+0000 mon.a (mon.0) 6369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:04:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[45495]: audit 2022-04-22T19:04:14.448357+0000 mon.a (mon.0) 6370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[46934]: audit 2022-04-22T19:04:14.373730+0000 mon.a (mon.0) 6368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:04:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[46934]: audit 2022-04-22T19:04:14.374552+0000 mon.a (mon.0) 6369 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:04:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:15 smithi115 bash[46934]: audit 2022-04-22T19:04:14.448357+0000 mon.a (mon.0) 6370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:16 smithi121 bash[36902]: cluster 2022-04-22T19:04:14.765092+0000 mgr.y (mgr.34107) 3387 : cluster [DBG] pgmap v4887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:16 smithi115 bash[45495]: cluster 2022-04-22T19:04:14.765092+0000 mgr.y (mgr.34107) 3387 : cluster [DBG] pgmap v4887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:16 smithi115 bash[46934]: cluster 2022-04-22T19:04:14.765092+0000 mgr.y (mgr.34107) 3387 : cluster [DBG] pgmap v4887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:17.546 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Apr 22 19:04:17 smithi115 bash[42914]: debug 2022-04-22T19:04:17.065+0000 7f8eea62c700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-04-22T19:04:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:18 smithi121 bash[36902]: cluster 2022-04-22T19:04:16.766088+0000 mgr.y (mgr.34107) 3388 : cluster [DBG] pgmap v4888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:18 smithi121 bash[36902]: audit 2022-04-22T19:04:17.066810+0000 mon.a (mon.0) 6371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:18 smithi115 bash[45495]: cluster 2022-04-22T19:04:16.766088+0000 mgr.y (mgr.34107) 3388 : cluster [DBG] pgmap v4888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:18 smithi115 bash[45495]: audit 2022-04-22T19:04:17.066810+0000 mon.a (mon.0) 6371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:18 smithi115 bash[46934]: cluster 2022-04-22T19:04:16.766088+0000 mgr.y (mgr.34107) 3388 : cluster [DBG] pgmap v4888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:18 smithi115 bash[46934]: audit 2022-04-22T19:04:17.066810+0000 mon.a (mon.0) 6371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:04:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:20 smithi121 bash[36902]: cluster 2022-04-22T19:04:18.767225+0000 mgr.y (mgr.34107) 3389 : cluster [DBG] pgmap v4889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:20 smithi121 bash[36902]: audit 2022-04-22T19:04:19.451604+0000 mon.a (mon.0) 6372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:20 smithi115 bash[45495]: cluster 2022-04-22T19:04:18.767225+0000 mgr.y (mgr.34107) 3389 : cluster [DBG] pgmap v4889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:20.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:20 smithi115 bash[45495]: audit 2022-04-22T19:04:19.451604+0000 mon.a (mon.0) 6372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:20 smithi115 bash[46934]: cluster 2022-04-22T19:04:18.767225+0000 mgr.y (mgr.34107) 3389 : cluster [DBG] pgmap v4889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:20 smithi115 bash[46934]: audit 2022-04-22T19:04:19.451604+0000 mon.a (mon.0) 6372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:22 smithi115 bash[45495]: cluster 2022-04-22T19:04:20.768314+0000 mgr.y (mgr.34107) 3390 : cluster [DBG] pgmap v4890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:21.985Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:22 smithi115 bash[46934]: cluster 2022-04-22T19:04:20.768314+0000 mgr.y (mgr.34107) 3390 : cluster [DBG] pgmap v4890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:22 smithi121 bash[36902]: cluster 2022-04-22T19:04:20.768314+0000 mgr.y (mgr.34107) 3390 : cluster [DBG] pgmap v4890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:23 smithi121 bash[36902]: cluster 2022-04-22T19:04:22.769361+0000 mgr.y (mgr.34107) 3391 : cluster [DBG] pgmap v4891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:23 smithi115 bash[45495]: cluster 2022-04-22T19:04:22.769361+0000 mgr.y (mgr.34107) 3391 : cluster [DBG] pgmap v4891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:23 smithi115 bash[46934]: cluster 2022-04-22T19:04:22.769361+0000 mgr.y (mgr.34107) 3391 : cluster [DBG] pgmap v4891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:23.857Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:24.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:24 smithi115 bash[45495]: audit 2022-04-22T19:04:24.454961+0000 mon.a (mon.0) 6373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:24.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:24 smithi115 bash[46934]: audit 2022-04-22T19:04:24.454961+0000 mon.a (mon.0) 6373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:24 smithi121 bash[36902]: audit 2022-04-22T19:04:24.454961+0000 mon.a (mon.0) 6373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:25 smithi115 bash[45495]: cluster 2022-04-22T19:04:24.770451+0000 mgr.y (mgr.34107) 3392 : cluster [DBG] pgmap v4892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:25 smithi115 bash[46934]: cluster 2022-04-22T19:04:24.770451+0000 mgr.y (mgr.34107) 3392 : cluster [DBG] pgmap v4892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:25 smithi121 bash[36902]: cluster 2022-04-22T19:04:24.770451+0000 mgr.y (mgr.34107) 3392 : cluster [DBG] pgmap v4892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:27 smithi121 bash[36902]: cluster 2022-04-22T19:04:26.771549+0000 mgr.y (mgr.34107) 3393 : cluster [DBG] pgmap v4893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:27 smithi115 bash[45495]: cluster 2022-04-22T19:04:26.771549+0000 mgr.y (mgr.34107) 3393 : cluster [DBG] pgmap v4893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:27 smithi115 bash[46934]: cluster 2022-04-22T19:04:26.771549+0000 mgr.y (mgr.34107) 3393 : cluster [DBG] pgmap v4893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:29 smithi121 bash[36902]: cluster 2022-04-22T19:04:28.772703+0000 mgr.y (mgr.34107) 3394 : cluster [DBG] pgmap v4894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:29 smithi121 bash[36902]: audit 2022-04-22T19:04:29.458245+0000 mon.a (mon.0) 6374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:29 smithi115 bash[46934]: cluster 2022-04-22T19:04:28.772703+0000 mgr.y (mgr.34107) 3394 : cluster [DBG] pgmap v4894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:29 smithi115 bash[46934]: audit 2022-04-22T19:04:29.458245+0000 mon.a (mon.0) 6374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:29 smithi115 bash[45495]: cluster 2022-04-22T19:04:28.772703+0000 mgr.y (mgr.34107) 3394 : cluster [DBG] pgmap v4894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:29 smithi115 bash[45495]: audit 2022-04-22T19:04:29.458245+0000 mon.a (mon.0) 6374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:31 smithi121 bash[36902]: cluster 2022-04-22T19:04:30.773754+0000 mgr.y (mgr.34107) 3395 : cluster [DBG] pgmap v4895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:32.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:31 smithi115 bash[45495]: cluster 2022-04-22T19:04:30.773754+0000 mgr.y (mgr.34107) 3395 : cluster [DBG] pgmap v4895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:31.986Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:31 smithi115 bash[46934]: cluster 2022-04-22T19:04:30.773754+0000 mgr.y (mgr.34107) 3395 : cluster [DBG] pgmap v4895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:33 smithi121 bash[36902]: cluster 2022-04-22T19:04:32.774811+0000 mgr.y (mgr.34107) 3396 : cluster [DBG] pgmap v4896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:33.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:33 smithi115 bash[45495]: cluster 2022-04-22T19:04:32.774811+0000 mgr.y (mgr.34107) 3396 : cluster [DBG] pgmap v4896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:33.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:33 smithi115 bash[46934]: cluster 2022-04-22T19:04:32.774811+0000 mgr.y (mgr.34107) 3396 : cluster [DBG] pgmap v4896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:33.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:34 smithi115 bash[45495]: audit 2022-04-22T19:04:34.461366+0000 mon.a (mon.0) 6375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:34 smithi115 bash[46934]: audit 2022-04-22T19:04:34.461366+0000 mon.a (mon.0) 6375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:34 smithi121 bash[36902]: audit 2022-04-22T19:04:34.461366+0000 mon.a (mon.0) 6375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:35 smithi115 bash[45495]: cluster 2022-04-22T19:04:34.775906+0000 mgr.y (mgr.34107) 3397 : cluster [DBG] pgmap v4897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:35 smithi115 bash[46934]: cluster 2022-04-22T19:04:34.775906+0000 mgr.y (mgr.34107) 3397 : cluster [DBG] pgmap v4897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:35 smithi121 bash[36902]: cluster 2022-04-22T19:04:34.775906+0000 mgr.y (mgr.34107) 3397 : cluster [DBG] pgmap v4897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:37 smithi121 bash[36902]: cluster 2022-04-22T19:04:36.777011+0000 mgr.y (mgr.34107) 3398 : cluster [DBG] pgmap v4898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:37 smithi115 bash[45495]: cluster 2022-04-22T19:04:36.777011+0000 mgr.y (mgr.34107) 3398 : cluster [DBG] pgmap v4898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:37 smithi115 bash[46934]: cluster 2022-04-22T19:04:36.777011+0000 mgr.y (mgr.34107) 3398 : cluster [DBG] pgmap v4898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:39 smithi121 bash[36902]: cluster 2022-04-22T19:04:38.778076+0000 mgr.y (mgr.34107) 3399 : cluster [DBG] pgmap v4899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:39 smithi121 bash[36902]: audit 2022-04-22T19:04:39.464630+0000 mon.a (mon.0) 6376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:39 smithi115 bash[46934]: cluster 2022-04-22T19:04:38.778076+0000 mgr.y (mgr.34107) 3399 : cluster [DBG] pgmap v4899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:39 smithi115 bash[46934]: audit 2022-04-22T19:04:39.464630+0000 mon.a (mon.0) 6376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:39 smithi115 bash[45495]: cluster 2022-04-22T19:04:38.778076+0000 mgr.y (mgr.34107) 3399 : cluster [DBG] pgmap v4899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:39 smithi115 bash[45495]: audit 2022-04-22T19:04:39.464630+0000 mon.a (mon.0) 6376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:41.764 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:04:42.089 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:04:42.090 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (87m) 4m ago 101m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:04:42.090 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (101m) 5m ago 101m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:04:42.090 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (89m) 4m ago 102m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:04:42.090 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (89m) 5m ago 102m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:04:42.090 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (97m) 5m ago 106m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (97m) 4m ago 108m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (96m) 4m ago 109m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (95m) 5m ago 107m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (96m) 4m ago 107m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (88m) 4m ago 101m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (88m) 5m ago 101m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (94m) 4m ago 106m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (93m) 4m ago 105m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (93m) 4m ago 105m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:04:42.091 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (92m) 4m ago 104m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:04:42.092 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (92m) 5m ago 104m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:04:42.092 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (91m) 5m ago 104m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:04:42.092 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (90m) 5m ago 103m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:04:42.092 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (90m) 5m ago 103m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:04:42.092 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (88m) 5m ago 101m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:04:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:41 smithi121 bash[36902]: cluster 2022-04-22T19:04:40.779153+0000 mgr.y (mgr.34107) 3400 : cluster [DBG] pgmap v4900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:41 smithi115 bash[45495]: cluster 2022-04-22T19:04:40.779153+0000 mgr.y (mgr.34107) 3400 : cluster [DBG] pgmap v4900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:41.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:41 smithi115 bash[46934]: cluster 2022-04-22T19:04:40.779153+0000 mgr.y (mgr.34107) 3400 : cluster [DBG] pgmap v4900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:42.486 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:42.487 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:04:42.488 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:04:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:42 smithi121 bash[36902]: audit 2022-04-22T19:04:41.741117+0000 mgr.y (mgr.34107) 3401 : audit [DBG] from='client.38970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:42 smithi121 bash[36902]: audit 2022-04-22T19:04:42.062314+0000 mgr.y (mgr.34107) 3402 : audit [DBG] from='client.38976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:42 smithi121 bash[36902]: audit 2022-04-22T19:04:42.476821+0000 mon.c (mon.1) 176 : audit [DBG] from='client.? 172.21.15.115:0/3523689797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[45495]: audit 2022-04-22T19:04:41.741117+0000 mgr.y (mgr.34107) 3401 : audit [DBG] from='client.38970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[45495]: audit 2022-04-22T19:04:42.062314+0000 mgr.y (mgr.34107) 3402 : audit [DBG] from='client.38976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[45495]: audit 2022-04-22T19:04:42.476821+0000 mon.c (mon.1) 176 : audit [DBG] from='client.? 172.21.15.115:0/3523689797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[46934]: audit 2022-04-22T19:04:41.741117+0000 mgr.y (mgr.34107) 3401 : audit [DBG] from='client.38970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[46934]: audit 2022-04-22T19:04:42.062314+0000 mgr.y (mgr.34107) 3402 : audit [DBG] from='client.38976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:04:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:42 smithi115 bash[46934]: audit 2022-04-22T19:04:42.476821+0000 mon.c (mon.1) 176 : audit [DBG] from='client.? 172.21.15.115:0/3523689797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:04:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:43 smithi121 bash[36902]: cluster 2022-04-22T19:04:42.780425+0000 mgr.y (mgr.34107) 3403 : cluster [DBG] pgmap v4901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:43 smithi115 bash[45495]: cluster 2022-04-22T19:04:42.780425+0000 mgr.y (mgr.34107) 3403 : cluster [DBG] pgmap v4901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:43.858Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:43 smithi115 bash[46934]: cluster 2022-04-22T19:04:42.780425+0000 mgr.y (mgr.34107) 3403 : cluster [DBG] pgmap v4901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:44 smithi121 bash[36902]: audit 2022-04-22T19:04:44.467681+0000 mon.a (mon.0) 6377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:44 smithi115 bash[45495]: audit 2022-04-22T19:04:44.467681+0000 mon.a (mon.0) 6377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:44 smithi115 bash[46934]: audit 2022-04-22T19:04:44.467681+0000 mon.a (mon.0) 6377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:45 smithi121 bash[36902]: cluster 2022-04-22T19:04:44.781652+0000 mgr.y (mgr.34107) 3404 : cluster [DBG] pgmap v4902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:45 smithi115 bash[45495]: cluster 2022-04-22T19:04:44.781652+0000 mgr.y (mgr.34107) 3404 : cluster [DBG] pgmap v4902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:45 smithi115 bash[46934]: cluster 2022-04-22T19:04:44.781652+0000 mgr.y (mgr.34107) 3404 : cluster [DBG] pgmap v4902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:47 smithi121 bash[36902]: cluster 2022-04-22T19:04:46.782788+0000 mgr.y (mgr.34107) 3405 : cluster [DBG] pgmap v4903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:47 smithi115 bash[45495]: cluster 2022-04-22T19:04:46.782788+0000 mgr.y (mgr.34107) 3405 : cluster [DBG] pgmap v4903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:47 smithi115 bash[46934]: cluster 2022-04-22T19:04:46.782788+0000 mgr.y (mgr.34107) 3405 : cluster [DBG] pgmap v4903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:49 smithi121 bash[36902]: cluster 2022-04-22T19:04:48.783922+0000 mgr.y (mgr.34107) 3406 : cluster [DBG] pgmap v4904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:49 smithi121 bash[36902]: audit 2022-04-22T19:04:49.470681+0000 mon.a (mon.0) 6378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:49 smithi115 bash[45495]: cluster 2022-04-22T19:04:48.783922+0000 mgr.y (mgr.34107) 3406 : cluster [DBG] pgmap v4904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:49 smithi115 bash[45495]: audit 2022-04-22T19:04:49.470681+0000 mon.a (mon.0) 6378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:49 smithi115 bash[46934]: cluster 2022-04-22T19:04:48.783922+0000 mgr.y (mgr.34107) 3406 : cluster [DBG] pgmap v4904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:49 smithi115 bash[46934]: audit 2022-04-22T19:04:49.470681+0000 mon.a (mon.0) 6378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:51 smithi121 bash[36902]: cluster 2022-04-22T19:04:50.784886+0000 mgr.y (mgr.34107) 3407 : cluster [DBG] pgmap v4905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:52.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:51.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:51 smithi115 bash[45495]: cluster 2022-04-22T19:04:50.784886+0000 mgr.y (mgr.34107) 3407 : cluster [DBG] pgmap v4905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:51 smithi115 bash[46934]: cluster 2022-04-22T19:04:50.784886+0000 mgr.y (mgr.34107) 3407 : cluster [DBG] pgmap v4905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:53 smithi121 bash[36902]: cluster 2022-04-22T19:04:52.785971+0000 mgr.y (mgr.34107) 3408 : cluster [DBG] pgmap v4906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:53 smithi115 bash[45495]: cluster 2022-04-22T19:04:52.785971+0000 mgr.y (mgr.34107) 3408 : cluster [DBG] pgmap v4906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:53 smithi115 bash[46934]: cluster 2022-04-22T19:04:52.785971+0000 mgr.y (mgr.34107) 3408 : cluster [DBG] pgmap v4906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:04:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:04:53.860Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:04:54.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:54 smithi115 bash[45495]: audit 2022-04-22T19:04:54.473854+0000 mon.a (mon.0) 6379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:54.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:54 smithi115 bash[46934]: audit 2022-04-22T19:04:54.473854+0000 mon.a (mon.0) 6379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:54 smithi121 bash[36902]: audit 2022-04-22T19:04:54.473854+0000 mon.a (mon.0) 6379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:04:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:55 smithi115 bash[45495]: cluster 2022-04-22T19:04:54.787094+0000 mgr.y (mgr.34107) 3409 : cluster [DBG] pgmap v4907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:55 smithi115 bash[46934]: cluster 2022-04-22T19:04:54.787094+0000 mgr.y (mgr.34107) 3409 : cluster [DBG] pgmap v4907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:55 smithi121 bash[36902]: cluster 2022-04-22T19:04:54.787094+0000 mgr.y (mgr.34107) 3409 : cluster [DBG] pgmap v4907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:57 smithi121 bash[36902]: cluster 2022-04-22T19:04:56.788200+0000 mgr.y (mgr.34107) 3410 : cluster [DBG] pgmap v4908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:57 smithi115 bash[45495]: cluster 2022-04-22T19:04:56.788200+0000 mgr.y (mgr.34107) 3410 : cluster [DBG] pgmap v4908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:04:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:57 smithi115 bash[46934]: cluster 2022-04-22T19:04:56.788200+0000 mgr.y (mgr.34107) 3410 : cluster [DBG] pgmap v4908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:59 smithi121 bash[36902]: cluster 2022-04-22T19:04:58.789338+0000 mgr.y (mgr.34107) 3411 : cluster [DBG] pgmap v4909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:04:59 smithi121 bash[36902]: audit 2022-04-22T19:04:59.476981+0000 mon.a (mon.0) 6380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:59 smithi115 bash[45495]: cluster 2022-04-22T19:04:58.789338+0000 mgr.y (mgr.34107) 3411 : cluster [DBG] pgmap v4909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:04:59 smithi115 bash[45495]: audit 2022-04-22T19:04:59.476981+0000 mon.a (mon.0) 6380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:59 smithi115 bash[46934]: cluster 2022-04-22T19:04:58.789338+0000 mgr.y (mgr.34107) 3411 : cluster [DBG] pgmap v4909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:04:59 smithi115 bash[46934]: audit 2022-04-22T19:04:59.476981+0000 mon.a (mon.0) 6380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:01.982 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[45495]: cluster 2022-04-22T19:05:00.790370+0000 mgr.y (mgr.34107) 3412 : cluster [DBG] pgmap v4910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:01.982 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[45495]: audit 2022-04-22T19:05:01.604529+0000 mon.a (mon.0) 6381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:05:01.982 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[45495]: audit 2022-04-22T19:05:01.607784+0000 mon.a (mon.0) 6382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:05:01.983 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[45495]: audit 2022-04-22T19:05:01.609388+0000 mon.a (mon.0) 6383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:05:01.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[46934]: cluster 2022-04-22T19:05:00.790370+0000 mgr.y (mgr.34107) 3412 : cluster [DBG] pgmap v4910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:01.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[46934]: audit 2022-04-22T19:05:01.604529+0000 mon.a (mon.0) 6381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:05:01.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[46934]: audit 2022-04-22T19:05:01.607784+0000 mon.a (mon.0) 6382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:05:01.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[46934]: audit 2022-04-22T19:05:01.609388+0000 mon.a (mon.0) 6383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:05:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:01 smithi121 bash[36902]: cluster 2022-04-22T19:05:00.790370+0000 mgr.y (mgr.34107) 3412 : cluster [DBG] pgmap v4910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:01 smithi121 bash[36902]: audit 2022-04-22T19:05:01.604529+0000 mon.a (mon.0) 6381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:05:02.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:01 smithi121 bash[36902]: audit 2022-04-22T19:05:01.607784+0000 mon.a (mon.0) 6382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:05:02.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:01 smithi121 bash[36902]: audit 2022-04-22T19:05:01.609388+0000 mon.a (mon.0) 6383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:05:02.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:01.985Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:02 smithi121 bash[36902]: audit 2022-04-22T19:05:01.993166+0000 mon.a (mon.0) 6384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:05:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:02 smithi115 bash[45495]: audit 2022-04-22T19:05:01.993166+0000 mon.a (mon.0) 6384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:05:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:02 smithi115 bash[46934]: audit 2022-04-22T19:05:01.993166+0000 mon.a (mon.0) 6384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:05:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:04 smithi121 bash[36902]: cluster 2022-04-22T19:05:02.791446+0000 mgr.y (mgr.34107) 3413 : cluster [DBG] pgmap v4911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:03 smithi115 bash[45495]: cluster 2022-04-22T19:05:02.791446+0000 mgr.y (mgr.34107) 3413 : cluster [DBG] pgmap v4911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:03 smithi115 bash[46934]: cluster 2022-04-22T19:05:02.791446+0000 mgr.y (mgr.34107) 3413 : cluster [DBG] pgmap v4911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:03.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:05 smithi121 bash[36902]: audit 2022-04-22T19:05:04.480064+0000 mon.a (mon.0) 6385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:05 smithi115 bash[45495]: audit 2022-04-22T19:05:04.480064+0000 mon.a (mon.0) 6385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:05 smithi115 bash[46934]: audit 2022-04-22T19:05:04.480064+0000 mon.a (mon.0) 6385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:06 smithi121 bash[36902]: cluster 2022-04-22T19:05:04.792495+0000 mgr.y (mgr.34107) 3414 : cluster [DBG] pgmap v4912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:06 smithi115 bash[45495]: cluster 2022-04-22T19:05:04.792495+0000 mgr.y (mgr.34107) 3414 : cluster [DBG] pgmap v4912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:06 smithi115 bash[46934]: cluster 2022-04-22T19:05:04.792495+0000 mgr.y (mgr.34107) 3414 : cluster [DBG] pgmap v4912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:08 smithi115 bash[45495]: cluster 2022-04-22T19:05:06.793601+0000 mgr.y (mgr.34107) 3415 : cluster [DBG] pgmap v4913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:08 smithi115 bash[46934]: cluster 2022-04-22T19:05:06.793601+0000 mgr.y (mgr.34107) 3415 : cluster [DBG] pgmap v4913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:08 smithi121 bash[36902]: cluster 2022-04-22T19:05:06.793601+0000 mgr.y (mgr.34107) 3415 : cluster [DBG] pgmap v4913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:10 smithi115 bash[45495]: cluster 2022-04-22T19:05:08.794695+0000 mgr.y (mgr.34107) 3416 : cluster [DBG] pgmap v4914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:10 smithi115 bash[45495]: audit 2022-04-22T19:05:09.483256+0000 mon.a (mon.0) 6386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:10 smithi115 bash[46934]: cluster 2022-04-22T19:05:08.794695+0000 mgr.y (mgr.34107) 3416 : cluster [DBG] pgmap v4914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:10 smithi115 bash[46934]: audit 2022-04-22T19:05:09.483256+0000 mon.a (mon.0) 6386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:10 smithi121 bash[36902]: cluster 2022-04-22T19:05:08.794695+0000 mgr.y (mgr.34107) 3416 : cluster [DBG] pgmap v4914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:10 smithi121 bash[36902]: audit 2022-04-22T19:05:09.483256+0000 mon.a (mon.0) 6386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:12.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:12 smithi115 bash[45495]: cluster 2022-04-22T19:05:10.795847+0000 mgr.y (mgr.34107) 3417 : cluster [DBG] pgmap v4915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:11.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:12 smithi115 bash[46934]: cluster 2022-04-22T19:05:10.795847+0000 mgr.y (mgr.34107) 3417 : cluster [DBG] pgmap v4915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:12 smithi121 bash[36902]: cluster 2022-04-22T19:05:10.795847+0000 mgr.y (mgr.34107) 3417 : cluster [DBG] pgmap v4915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:12.865 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:05:13.212 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (88m) 5m ago 102m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (101m) 6m ago 101m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (90m) 5m ago 103m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (89m) 6m ago 103m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (97m) 6m ago 107m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (98m) 5m ago 109m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (97m) 5m ago 109m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:05:13.213 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (96m) 6m ago 107m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (96m) 5m ago 107m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (89m) 5m ago 102m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (89m) 6m ago 102m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (95m) 5m ago 106m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (94m) 5m ago 106m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (93m) 5m ago 105m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (93m) 5m ago 105m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (92m) 6m ago 105m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:05:13.214 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (92m) 6m ago 104m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:05:13.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (91m) 6m ago 104m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:05:13.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (90m) 6m ago 103m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:05:13.215 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (88m) 6m ago 102m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:13.619 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:05:13.620 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:05:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:13 smithi121 bash[36902]: cluster 2022-04-22T19:05:12.796879+0000 mgr.y (mgr.34107) 3418 : cluster [DBG] pgmap v4916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:13 smithi121 bash[36902]: audit 2022-04-22T19:05:12.839694+0000 mgr.y (mgr.34107) 3419 : audit [DBG] from='client.48922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:13 smithi121 bash[36902]: audit 2022-04-22T19:05:13.182577+0000 mgr.y (mgr.34107) 3420 : audit [DBG] from='client.48928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[45495]: cluster 2022-04-22T19:05:12.796879+0000 mgr.y (mgr.34107) 3418 : cluster [DBG] pgmap v4916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[45495]: audit 2022-04-22T19:05:12.839694+0000 mgr.y (mgr.34107) 3419 : audit [DBG] from='client.48922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[45495]: audit 2022-04-22T19:05:13.182577+0000 mgr.y (mgr.34107) 3420 : audit [DBG] from='client.48928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[46934]: cluster 2022-04-22T19:05:12.796879+0000 mgr.y (mgr.34107) 3418 : cluster [DBG] pgmap v4916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[46934]: audit 2022-04-22T19:05:12.839694+0000 mgr.y (mgr.34107) 3419 : audit [DBG] from='client.48922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[46934]: audit 2022-04-22T19:05:13.182577+0000 mgr.y (mgr.34107) 3420 : audit [DBG] from='client.48928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:13.862Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:14 smithi121 bash[36902]: audit 2022-04-22T19:05:13.609381+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.115:0/293911107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:14 smithi115 bash[45495]: audit 2022-04-22T19:05:13.609381+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.115:0/293911107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:14 smithi115 bash[46934]: audit 2022-04-22T19:05:13.609381+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.115:0/293911107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:15 smithi121 bash[36902]: audit 2022-04-22T19:05:14.387394+0000 mon.a (mon.0) 6387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:05:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:15 smithi121 bash[36902]: audit 2022-04-22T19:05:14.388053+0000 mon.a (mon.0) 6388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:05:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:15 smithi121 bash[36902]: audit 2022-04-22T19:05:14.486500+0000 mon.a (mon.0) 6389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:15 smithi121 bash[36902]: cluster 2022-04-22T19:05:14.797899+0000 mgr.y (mgr.34107) 3421 : cluster [DBG] pgmap v4917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[45495]: audit 2022-04-22T19:05:14.387394+0000 mon.a (mon.0) 6387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[45495]: audit 2022-04-22T19:05:14.388053+0000 mon.a (mon.0) 6388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[45495]: audit 2022-04-22T19:05:14.486500+0000 mon.a (mon.0) 6389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[45495]: cluster 2022-04-22T19:05:14.797899+0000 mgr.y (mgr.34107) 3421 : cluster [DBG] pgmap v4917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[46934]: audit 2022-04-22T19:05:14.387394+0000 mon.a (mon.0) 6387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[46934]: audit 2022-04-22T19:05:14.388053+0000 mon.a (mon.0) 6388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[46934]: audit 2022-04-22T19:05:14.486500+0000 mon.a (mon.0) 6389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:15 smithi115 bash[46934]: cluster 2022-04-22T19:05:14.797899+0000 mgr.y (mgr.34107) 3421 : cluster [DBG] pgmap v4917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:17 smithi121 bash[36902]: cluster 2022-04-22T19:05:16.798972+0000 mgr.y (mgr.34107) 3422 : cluster [DBG] pgmap v4918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:17 smithi115 bash[45495]: cluster 2022-04-22T19:05:16.798972+0000 mgr.y (mgr.34107) 3422 : cluster [DBG] pgmap v4918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:17 smithi115 bash[46934]: cluster 2022-04-22T19:05:16.798972+0000 mgr.y (mgr.34107) 3422 : cluster [DBG] pgmap v4918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:19 smithi121 bash[36902]: cluster 2022-04-22T19:05:18.800214+0000 mgr.y (mgr.34107) 3423 : cluster [DBG] pgmap v4919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:19 smithi121 bash[36902]: audit 2022-04-22T19:05:19.489694+0000 mon.a (mon.0) 6390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:19 smithi115 bash[45495]: cluster 2022-04-22T19:05:18.800214+0000 mgr.y (mgr.34107) 3423 : cluster [DBG] pgmap v4919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:19 smithi115 bash[45495]: audit 2022-04-22T19:05:19.489694+0000 mon.a (mon.0) 6390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:19 smithi115 bash[46934]: cluster 2022-04-22T19:05:18.800214+0000 mgr.y (mgr.34107) 3423 : cluster [DBG] pgmap v4919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:19 smithi115 bash[46934]: audit 2022-04-22T19:05:19.489694+0000 mon.a (mon.0) 6390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:21 smithi121 bash[36902]: cluster 2022-04-22T19:05:20.801253+0000 mgr.y (mgr.34107) 3424 : cluster [DBG] pgmap v4920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:21 smithi115 bash[45495]: cluster 2022-04-22T19:05:20.801253+0000 mgr.y (mgr.34107) 3424 : cluster [DBG] pgmap v4920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:21.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:21 smithi115 bash[46934]: cluster 2022-04-22T19:05:20.801253+0000 mgr.y (mgr.34107) 3424 : cluster [DBG] pgmap v4920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:23 smithi121 bash[36902]: cluster 2022-04-22T19:05:22.802354+0000 mgr.y (mgr.34107) 3425 : cluster [DBG] pgmap v4921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:23 smithi115 bash[45495]: cluster 2022-04-22T19:05:22.802354+0000 mgr.y (mgr.34107) 3425 : cluster [DBG] pgmap v4921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:23 smithi115 bash[46934]: cluster 2022-04-22T19:05:22.802354+0000 mgr.y (mgr.34107) 3425 : cluster [DBG] pgmap v4921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:23.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:24 smithi121 bash[36902]: audit 2022-04-22T19:05:24.492854+0000 mon.a (mon.0) 6391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:24 smithi115 bash[45495]: audit 2022-04-22T19:05:24.492854+0000 mon.a (mon.0) 6391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:24 smithi115 bash[46934]: audit 2022-04-22T19:05:24.492854+0000 mon.a (mon.0) 6391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:25 smithi121 bash[36902]: cluster 2022-04-22T19:05:24.803430+0000 mgr.y (mgr.34107) 3426 : cluster [DBG] pgmap v4922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:25 smithi115 bash[45495]: cluster 2022-04-22T19:05:24.803430+0000 mgr.y (mgr.34107) 3426 : cluster [DBG] pgmap v4922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:25 smithi115 bash[46934]: cluster 2022-04-22T19:05:24.803430+0000 mgr.y (mgr.34107) 3426 : cluster [DBG] pgmap v4922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:27 smithi121 bash[36902]: cluster 2022-04-22T19:05:26.804595+0000 mgr.y (mgr.34107) 3427 : cluster [DBG] pgmap v4923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:27 smithi115 bash[45495]: cluster 2022-04-22T19:05:26.804595+0000 mgr.y (mgr.34107) 3427 : cluster [DBG] pgmap v4923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:28.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:27 smithi115 bash[46934]: cluster 2022-04-22T19:05:26.804595+0000 mgr.y (mgr.34107) 3427 : cluster [DBG] pgmap v4923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:29 smithi121 bash[36902]: cluster 2022-04-22T19:05:28.805749+0000 mgr.y (mgr.34107) 3428 : cluster [DBG] pgmap v4924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:29 smithi121 bash[36902]: audit 2022-04-22T19:05:29.495898+0000 mon.a (mon.0) 6392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:29 smithi115 bash[45495]: cluster 2022-04-22T19:05:28.805749+0000 mgr.y (mgr.34107) 3428 : cluster [DBG] pgmap v4924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:30.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:29 smithi115 bash[45495]: audit 2022-04-22T19:05:29.495898+0000 mon.a (mon.0) 6392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:29 smithi115 bash[46934]: cluster 2022-04-22T19:05:28.805749+0000 mgr.y (mgr.34107) 3428 : cluster [DBG] pgmap v4924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:29 smithi115 bash[46934]: audit 2022-04-22T19:05:29.495898+0000 mon.a (mon.0) 6392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:31 smithi121 bash[36902]: cluster 2022-04-22T19:05:30.806833+0000 mgr.y (mgr.34107) 3429 : cluster [DBG] pgmap v4925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:32.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:31 smithi115 bash[45495]: cluster 2022-04-22T19:05:30.806833+0000 mgr.y (mgr.34107) 3429 : cluster [DBG] pgmap v4925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:31.989Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:31 smithi115 bash[46934]: cluster 2022-04-22T19:05:30.806833+0000 mgr.y (mgr.34107) 3429 : cluster [DBG] pgmap v4925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:33 smithi121 bash[36902]: cluster 2022-04-22T19:05:32.807884+0000 mgr.y (mgr.34107) 3430 : cluster [DBG] pgmap v4926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:33 smithi115 bash[45495]: cluster 2022-04-22T19:05:32.807884+0000 mgr.y (mgr.34107) 3430 : cluster [DBG] pgmap v4926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:33 smithi115 bash[46934]: cluster 2022-04-22T19:05:32.807884+0000 mgr.y (mgr.34107) 3430 : cluster [DBG] pgmap v4926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:33.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:34 smithi121 bash[36902]: audit 2022-04-22T19:05:34.499008+0000 mon.a (mon.0) 6393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:35.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:34 smithi115 bash[45495]: audit 2022-04-22T19:05:34.499008+0000 mon.a (mon.0) 6393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:34 smithi115 bash[46934]: audit 2022-04-22T19:05:34.499008+0000 mon.a (mon.0) 6393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:35 smithi121 bash[36902]: cluster 2022-04-22T19:05:34.808969+0000 mgr.y (mgr.34107) 3431 : cluster [DBG] pgmap v4927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:35 smithi115 bash[45495]: cluster 2022-04-22T19:05:34.808969+0000 mgr.y (mgr.34107) 3431 : cluster [DBG] pgmap v4927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:36.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:35 smithi115 bash[46934]: cluster 2022-04-22T19:05:34.808969+0000 mgr.y (mgr.34107) 3431 : cluster [DBG] pgmap v4927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:37 smithi121 bash[36902]: cluster 2022-04-22T19:05:36.810039+0000 mgr.y (mgr.34107) 3432 : cluster [DBG] pgmap v4928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:37 smithi115 bash[45495]: cluster 2022-04-22T19:05:36.810039+0000 mgr.y (mgr.34107) 3432 : cluster [DBG] pgmap v4928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:37 smithi115 bash[46934]: cluster 2022-04-22T19:05:36.810039+0000 mgr.y (mgr.34107) 3432 : cluster [DBG] pgmap v4928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:39 smithi121 bash[36902]: cluster 2022-04-22T19:05:38.811097+0000 mgr.y (mgr.34107) 3433 : cluster [DBG] pgmap v4929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:39 smithi121 bash[36902]: audit 2022-04-22T19:05:39.502021+0000 mon.a (mon.0) 6394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:39 smithi115 bash[45495]: cluster 2022-04-22T19:05:38.811097+0000 mgr.y (mgr.34107) 3433 : cluster [DBG] pgmap v4929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:39 smithi115 bash[45495]: audit 2022-04-22T19:05:39.502021+0000 mon.a (mon.0) 6394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:39 smithi115 bash[46934]: cluster 2022-04-22T19:05:38.811097+0000 mgr.y (mgr.34107) 3433 : cluster [DBG] pgmap v4929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:39 smithi115 bash[46934]: audit 2022-04-22T19:05:39.502021+0000 mon.a (mon.0) 6394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:41 smithi121 bash[36902]: cluster 2022-04-22T19:05:40.812185+0000 mgr.y (mgr.34107) 3434 : cluster [DBG] pgmap v4930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:41 smithi115 bash[45495]: cluster 2022-04-22T19:05:40.812185+0000 mgr.y (mgr.34107) 3434 : cluster [DBG] pgmap v4930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:41.990Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:41 smithi115 bash[46934]: cluster 2022-04-22T19:05:40.812185+0000 mgr.y (mgr.34107) 3434 : cluster [DBG] pgmap v4930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:43 smithi121 bash[36902]: cluster 2022-04-22T19:05:42.813297+0000 mgr.y (mgr.34107) 3435 : cluster [DBG] pgmap v4931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:43.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:43 smithi115 bash[46934]: cluster 2022-04-22T19:05:42.813297+0000 mgr.y (mgr.34107) 3435 : cluster [DBG] pgmap v4931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:43.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:43 smithi115 bash[45495]: cluster 2022-04-22T19:05:42.813297+0000 mgr.y (mgr.34107) 3435 : cluster [DBG] pgmap v4931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:43.992 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:05:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:43.862Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:44.338 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:05:44.338 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (88m) 5m ago 102m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (102m) 6m ago 102m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (90m) 5m ago 103m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (90m) 6m ago 103m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (98m) 6m ago 107m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (98m) 5m ago 110m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (97m) 5m ago 110m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (96m) 6m ago 108m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (97m) 5m ago 108m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:05:44.339 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (89m) 5m ago 102m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (89m) 6m ago 102m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (95m) 5m ago 107m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (94m) 5m ago 106m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (94m) 5m ago 106m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (93m) 5m ago 105m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (93m) 6m ago 105m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:05:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (92m) 6m ago 105m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:05:44.341 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (91m) 6m ago 104m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:05:44.341 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (91m) 6m ago 104m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:05:44.341 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (89m) 6m ago 102m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:05:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:44 smithi121 bash[36902]: audit 2022-04-22T19:05:43.967518+0000 mgr.y (mgr.34107) 3436 : audit [DBG] from='client.48940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:44.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:44 smithi115 bash[45495]: audit 2022-04-22T19:05:43.967518+0000 mgr.y (mgr.34107) 3436 : audit [DBG] from='client.48940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:44.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:44 smithi115 bash[46934]: audit 2022-04-22T19:05:43.967518+0000 mgr.y (mgr.34107) 3436 : audit [DBG] from='client.48940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:45 smithi121 bash[36902]: audit 2022-04-22T19:05:44.315137+0000 mgr.y (mgr.34107) 3437 : audit [DBG] from='client.39012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:45 smithi121 bash[36902]: audit 2022-04-22T19:05:44.505060+0000 mon.a (mon.0) 6395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:45 smithi121 bash[36902]: audit 2022-04-22T19:05:44.731189+0000 mon.c (mon.1) 178 : audit [DBG] from='client.? 172.21.15.115:0/4043343113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:45 smithi121 bash[36902]: cluster 2022-04-22T19:05:44.814434+0000 mgr.y (mgr.34107) 3438 : cluster [DBG] pgmap v4932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[46934]: audit 2022-04-22T19:05:44.315137+0000 mgr.y (mgr.34107) 3437 : audit [DBG] from='client.39012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[46934]: audit 2022-04-22T19:05:44.505060+0000 mon.a (mon.0) 6395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[46934]: audit 2022-04-22T19:05:44.731189+0000 mon.c (mon.1) 178 : audit [DBG] from='client.? 172.21.15.115:0/4043343113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[46934]: cluster 2022-04-22T19:05:44.814434+0000 mgr.y (mgr.34107) 3438 : cluster [DBG] pgmap v4932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[45495]: audit 2022-04-22T19:05:44.315137+0000 mgr.y (mgr.34107) 3437 : audit [DBG] from='client.39012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:05:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[45495]: audit 2022-04-22T19:05:44.505060+0000 mon.a (mon.0) 6395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[45495]: audit 2022-04-22T19:05:44.731189+0000 mon.c (mon.1) 178 : audit [DBG] from='client.? 172.21.15.115:0/4043343113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:05:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:45 smithi115 bash[45495]: cluster 2022-04-22T19:05:44.814434+0000 mgr.y (mgr.34107) 3438 : cluster [DBG] pgmap v4932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:47 smithi121 bash[36902]: cluster 2022-04-22T19:05:46.815574+0000 mgr.y (mgr.34107) 3439 : cluster [DBG] pgmap v4933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:47 smithi115 bash[45495]: cluster 2022-04-22T19:05:46.815574+0000 mgr.y (mgr.34107) 3439 : cluster [DBG] pgmap v4933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:47 smithi115 bash[46934]: cluster 2022-04-22T19:05:46.815574+0000 mgr.y (mgr.34107) 3439 : cluster [DBG] pgmap v4933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:49 smithi121 bash[36902]: cluster 2022-04-22T19:05:48.816687+0000 mgr.y (mgr.34107) 3440 : cluster [DBG] pgmap v4934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:49 smithi121 bash[36902]: audit 2022-04-22T19:05:49.507918+0000 mon.a (mon.0) 6396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:49 smithi115 bash[45495]: cluster 2022-04-22T19:05:48.816687+0000 mgr.y (mgr.34107) 3440 : cluster [DBG] pgmap v4934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:49 smithi115 bash[45495]: audit 2022-04-22T19:05:49.507918+0000 mon.a (mon.0) 6396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:49 smithi115 bash[46934]: cluster 2022-04-22T19:05:48.816687+0000 mgr.y (mgr.34107) 3440 : cluster [DBG] pgmap v4934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:49 smithi115 bash[46934]: audit 2022-04-22T19:05:49.507918+0000 mon.a (mon.0) 6396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:51 smithi121 bash[36902]: cluster 2022-04-22T19:05:50.817742+0000 mgr.y (mgr.34107) 3441 : cluster [DBG] pgmap v4935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:52.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:51 smithi115 bash[45495]: cluster 2022-04-22T19:05:50.817742+0000 mgr.y (mgr.34107) 3441 : cluster [DBG] pgmap v4935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:51.991Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:51 smithi115 bash[46934]: cluster 2022-04-22T19:05:50.817742+0000 mgr.y (mgr.34107) 3441 : cluster [DBG] pgmap v4935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:53 smithi121 bash[36902]: cluster 2022-04-22T19:05:52.818925+0000 mgr.y (mgr.34107) 3442 : cluster [DBG] pgmap v4936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:53 smithi115 bash[45495]: cluster 2022-04-22T19:05:52.818925+0000 mgr.y (mgr.34107) 3442 : cluster [DBG] pgmap v4936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:53 smithi115 bash[46934]: cluster 2022-04-22T19:05:52.818925+0000 mgr.y (mgr.34107) 3442 : cluster [DBG] pgmap v4936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:05:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:05:53.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:05:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:54 smithi121 bash[36902]: audit 2022-04-22T19:05:54.511150+0000 mon.a (mon.0) 6397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:55.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:54 smithi115 bash[45495]: audit 2022-04-22T19:05:54.511150+0000 mon.a (mon.0) 6397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:54 smithi115 bash[46934]: audit 2022-04-22T19:05:54.511150+0000 mon.a (mon.0) 6397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:05:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:55 smithi121 bash[36902]: cluster 2022-04-22T19:05:54.820067+0000 mgr.y (mgr.34107) 3443 : cluster [DBG] pgmap v4937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:55 smithi115 bash[45495]: cluster 2022-04-22T19:05:54.820067+0000 mgr.y (mgr.34107) 3443 : cluster [DBG] pgmap v4937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:55 smithi115 bash[46934]: cluster 2022-04-22T19:05:54.820067+0000 mgr.y (mgr.34107) 3443 : cluster [DBG] pgmap v4937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:57 smithi121 bash[36902]: cluster 2022-04-22T19:05:56.821145+0000 mgr.y (mgr.34107) 3444 : cluster [DBG] pgmap v4938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:57 smithi115 bash[45495]: cluster 2022-04-22T19:05:56.821145+0000 mgr.y (mgr.34107) 3444 : cluster [DBG] pgmap v4938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:05:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:57 smithi115 bash[46934]: cluster 2022-04-22T19:05:56.821145+0000 mgr.y (mgr.34107) 3444 : cluster [DBG] pgmap v4938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:59 smithi121 bash[36902]: cluster 2022-04-22T19:05:58.822182+0000 mgr.y (mgr.34107) 3445 : cluster [DBG] pgmap v4939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:05:59 smithi121 bash[36902]: audit 2022-04-22T19:05:59.514360+0000 mon.a (mon.0) 6398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:59 smithi115 bash[45495]: cluster 2022-04-22T19:05:58.822182+0000 mgr.y (mgr.34107) 3445 : cluster [DBG] pgmap v4939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:05:59 smithi115 bash[45495]: audit 2022-04-22T19:05:59.514360+0000 mon.a (mon.0) 6398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:59 smithi115 bash[46934]: cluster 2022-04-22T19:05:58.822182+0000 mgr.y (mgr.34107) 3445 : cluster [DBG] pgmap v4939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:05:59 smithi115 bash[46934]: audit 2022-04-22T19:05:59.514360+0000 mon.a (mon.0) 6398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:01 smithi121 bash[36902]: cluster 2022-04-22T19:06:00.823223+0000 mgr.y (mgr.34107) 3446 : cluster [DBG] pgmap v4940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:02.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:01 smithi115 bash[46934]: cluster 2022-04-22T19:06:00.823223+0000 mgr.y (mgr.34107) 3446 : cluster [DBG] pgmap v4940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:02.356 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:01.993Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:02.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:01 smithi115 bash[45495]: cluster 2022-04-22T19:06:00.823223+0000 mgr.y (mgr.34107) 3446 : cluster [DBG] pgmap v4940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:03.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:02 smithi121 bash[36902]: audit 2022-04-22T19:06:02.001544+0000 mon.a (mon.0) 6399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:06:03.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:02 smithi121 bash[36902]: audit 2022-04-22T19:06:02.004188+0000 mon.a (mon.0) 6400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:06:03.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:02 smithi121 bash[36902]: audit 2022-04-22T19:06:02.005843+0000 mon.a (mon.0) 6401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:06:03.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:02 smithi121 bash[36902]: audit 2022-04-22T19:06:02.367965+0000 mon.a (mon.0) 6402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:06:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[45495]: audit 2022-04-22T19:06:02.001544+0000 mon.a (mon.0) 6399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[45495]: audit 2022-04-22T19:06:02.004188+0000 mon.a (mon.0) 6400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[45495]: audit 2022-04-22T19:06:02.005843+0000 mon.a (mon.0) 6401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[45495]: audit 2022-04-22T19:06:02.367965+0000 mon.a (mon.0) 6402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[46934]: audit 2022-04-22T19:06:02.001544+0000 mon.a (mon.0) 6399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[46934]: audit 2022-04-22T19:06:02.004188+0000 mon.a (mon.0) 6400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[46934]: audit 2022-04-22T19:06:02.005843+0000 mon.a (mon.0) 6401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:06:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:02 smithi115 bash[46934]: audit 2022-04-22T19:06:02.367965+0000 mon.a (mon.0) 6402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:06:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:03 smithi121 bash[36902]: cluster 2022-04-22T19:06:02.824350+0000 mgr.y (mgr.34107) 3447 : cluster [DBG] pgmap v4941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:03.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:04.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:03 smithi115 bash[45495]: cluster 2022-04-22T19:06:02.824350+0000 mgr.y (mgr.34107) 3447 : cluster [DBG] pgmap v4941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:03 smithi115 bash[46934]: cluster 2022-04-22T19:06:02.824350+0000 mgr.y (mgr.34107) 3447 : cluster [DBG] pgmap v4941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:04 smithi121 bash[36902]: audit 2022-04-22T19:06:04.517777+0000 mon.a (mon.0) 6403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:04 smithi115 bash[45495]: audit 2022-04-22T19:06:04.517777+0000 mon.a (mon.0) 6403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:05.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:04 smithi115 bash[46934]: audit 2022-04-22T19:06:04.517777+0000 mon.a (mon.0) 6403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:05 smithi121 bash[36902]: cluster 2022-04-22T19:06:04.825415+0000 mgr.y (mgr.34107) 3448 : cluster [DBG] pgmap v4942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:05 smithi115 bash[45495]: cluster 2022-04-22T19:06:04.825415+0000 mgr.y (mgr.34107) 3448 : cluster [DBG] pgmap v4942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:05 smithi115 bash[46934]: cluster 2022-04-22T19:06:04.825415+0000 mgr.y (mgr.34107) 3448 : cluster [DBG] pgmap v4942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:07 smithi121 bash[36902]: cluster 2022-04-22T19:06:06.826499+0000 mgr.y (mgr.34107) 3449 : cluster [DBG] pgmap v4943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:08.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:07 smithi115 bash[45495]: cluster 2022-04-22T19:06:06.826499+0000 mgr.y (mgr.34107) 3449 : cluster [DBG] pgmap v4943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:08.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:07 smithi115 bash[46934]: cluster 2022-04-22T19:06:06.826499+0000 mgr.y (mgr.34107) 3449 : cluster [DBG] pgmap v4943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:09 smithi121 bash[36902]: cluster 2022-04-22T19:06:08.827656+0000 mgr.y (mgr.34107) 3450 : cluster [DBG] pgmap v4944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:09 smithi121 bash[36902]: audit 2022-04-22T19:06:09.520687+0000 mon.a (mon.0) 6404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:09 smithi115 bash[45495]: cluster 2022-04-22T19:06:08.827656+0000 mgr.y (mgr.34107) 3450 : cluster [DBG] pgmap v4944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:09 smithi115 bash[45495]: audit 2022-04-22T19:06:09.520687+0000 mon.a (mon.0) 6404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:09 smithi115 bash[46934]: cluster 2022-04-22T19:06:08.827656+0000 mgr.y (mgr.34107) 3450 : cluster [DBG] pgmap v4944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:09 smithi115 bash[46934]: audit 2022-04-22T19:06:09.520687+0000 mon.a (mon.0) 6404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:11 smithi121 bash[36902]: cluster 2022-04-22T19:06:10.828729+0000 mgr.y (mgr.34107) 3451 : cluster [DBG] pgmap v4945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:12.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:11 smithi115 bash[45495]: cluster 2022-04-22T19:06:10.828729+0000 mgr.y (mgr.34107) 3451 : cluster [DBG] pgmap v4945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:11.993Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:11 smithi115 bash[46934]: cluster 2022-04-22T19:06:10.828729+0000 mgr.y (mgr.34107) 3451 : cluster [DBG] pgmap v4945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:13 smithi121 bash[36902]: cluster 2022-04-22T19:06:12.829932+0000 mgr.y (mgr.34107) 3452 : cluster [DBG] pgmap v4946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:13.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:13 smithi115 bash[45495]: cluster 2022-04-22T19:06:12.829932+0000 mgr.y (mgr.34107) 3452 : cluster [DBG] pgmap v4946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:13 smithi115 bash[46934]: cluster 2022-04-22T19:06:12.829932+0000 mgr.y (mgr.34107) 3452 : cluster [DBG] pgmap v4946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:13.865Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:14 smithi121 bash[36902]: audit 2022-04-22T19:06:14.397554+0000 mon.a (mon.0) 6405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:06:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:14 smithi121 bash[36902]: audit 2022-04-22T19:06:14.397988+0000 mon.a (mon.0) 6406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:06:14.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:14 smithi115 bash[45495]: audit 2022-04-22T19:06:14.397554+0000 mon.a (mon.0) 6405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:06:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:14 smithi115 bash[45495]: audit 2022-04-22T19:06:14.397988+0000 mon.a (mon.0) 6406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:06:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:14 smithi115 bash[46934]: audit 2022-04-22T19:06:14.397554+0000 mon.a (mon.0) 6405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:06:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:14 smithi115 bash[46934]: audit 2022-04-22T19:06:14.397988+0000 mon.a (mon.0) 6406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:06:15.127 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (89m) 6m ago 103m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (102m) 7m ago 102m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (91m) 6m ago 104m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (90m) 7m ago 104m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (98m) 7m ago 108m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:06:15.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (99m) 6m ago 110m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (98m) 6m ago 110m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (97m) 7m ago 108m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (97m) 6m ago 108m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (90m) 6m ago 103m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (90m) 7m ago 103m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (96m) 6m ago 107m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (95m) 6m ago 107m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (94m) 6m ago 107m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (94m) 6m ago 106m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:06:15.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (93m) 7m ago 106m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:06:15.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (93m) 7m ago 105m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:06:15.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (92m) 7m ago 105m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:06:15.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (91m) 7m ago 104m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:06:15.477 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (90m) 7m ago 103m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:06:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:15 smithi121 bash[36902]: audit 2022-04-22T19:06:14.523646+0000 mon.a (mon.0) 6407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:15 smithi121 bash[36902]: cluster 2022-04-22T19:06:14.831067+0000 mgr.y (mgr.34107) 3453 : cluster [DBG] pgmap v4947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:15 smithi121 bash[36902]: audit 2022-04-22T19:06:15.099614+0000 mgr.y (mgr.34107) 3454 : audit [DBG] from='client.39024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[46934]: audit 2022-04-22T19:06:14.523646+0000 mon.a (mon.0) 6407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[46934]: cluster 2022-04-22T19:06:14.831067+0000 mgr.y (mgr.34107) 3453 : cluster [DBG] pgmap v4947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[46934]: audit 2022-04-22T19:06:15.099614+0000 mgr.y (mgr.34107) 3454 : audit [DBG] from='client.39024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[45495]: audit 2022-04-22T19:06:14.523646+0000 mon.a (mon.0) 6407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[45495]: cluster 2022-04-22T19:06:14.831067+0000 mgr.y (mgr.34107) 3453 : cluster [DBG] pgmap v4947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:15 smithi115 bash[45495]: audit 2022-04-22T19:06:15.099614+0000 mgr.y (mgr.34107) 3454 : audit [DBG] from='client.39024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:15.880 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:06:15.880 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:06:15.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:06:15.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:06:15.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:15.882 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:06:15.882 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:06:15.882 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:06:15.882 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:06:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:16 smithi121 bash[36902]: audit 2022-04-22T19:06:15.446350+0000 mgr.y (mgr.34107) 3455 : audit [DBG] from='client.39030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:16.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:16 smithi121 bash[36902]: audit 2022-04-22T19:06:15.870359+0000 mon.c (mon.1) 179 : audit [DBG] from='client.? 172.21.15.115:0/1275707770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:16 smithi115 bash[45495]: audit 2022-04-22T19:06:15.446350+0000 mgr.y (mgr.34107) 3455 : audit [DBG] from='client.39030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:16 smithi115 bash[45495]: audit 2022-04-22T19:06:15.870359+0000 mon.c (mon.1) 179 : audit [DBG] from='client.? 172.21.15.115:0/1275707770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:16 smithi115 bash[46934]: audit 2022-04-22T19:06:15.446350+0000 mgr.y (mgr.34107) 3455 : audit [DBG] from='client.39030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:16 smithi115 bash[46934]: audit 2022-04-22T19:06:15.870359+0000 mon.c (mon.1) 179 : audit [DBG] from='client.? 172.21.15.115:0/1275707770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:17 smithi121 bash[36902]: cluster 2022-04-22T19:06:16.832158+0000 mgr.y (mgr.34107) 3456 : cluster [DBG] pgmap v4948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:17 smithi115 bash[45495]: cluster 2022-04-22T19:06:16.832158+0000 mgr.y (mgr.34107) 3456 : cluster [DBG] pgmap v4948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:17 smithi115 bash[46934]: cluster 2022-04-22T19:06:16.832158+0000 mgr.y (mgr.34107) 3456 : cluster [DBG] pgmap v4948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:19 smithi121 bash[36902]: cluster 2022-04-22T19:06:18.833543+0000 mgr.y (mgr.34107) 3457 : cluster [DBG] pgmap v4949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:19 smithi121 bash[36902]: audit 2022-04-22T19:06:19.526944+0000 mon.a (mon.0) 6408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:19 smithi115 bash[45495]: cluster 2022-04-22T19:06:18.833543+0000 mgr.y (mgr.34107) 3457 : cluster [DBG] pgmap v4949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:19 smithi115 bash[45495]: audit 2022-04-22T19:06:19.526944+0000 mon.a (mon.0) 6408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:19 smithi115 bash[46934]: cluster 2022-04-22T19:06:18.833543+0000 mgr.y (mgr.34107) 3457 : cluster [DBG] pgmap v4949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:19 smithi115 bash[46934]: audit 2022-04-22T19:06:19.526944+0000 mon.a (mon.0) 6408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:21 smithi121 bash[36902]: cluster 2022-04-22T19:06:20.834618+0000 mgr.y (mgr.34107) 3458 : cluster [DBG] pgmap v4950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:22.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:21 smithi115 bash[45495]: cluster 2022-04-22T19:06:20.834618+0000 mgr.y (mgr.34107) 3458 : cluster [DBG] pgmap v4950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:21.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:21 smithi115 bash[46934]: cluster 2022-04-22T19:06:20.834618+0000 mgr.y (mgr.34107) 3458 : cluster [DBG] pgmap v4950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:23 smithi121 bash[36902]: cluster 2022-04-22T19:06:22.835781+0000 mgr.y (mgr.34107) 3459 : cluster [DBG] pgmap v4951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:23 smithi115 bash[45495]: cluster 2022-04-22T19:06:22.835781+0000 mgr.y (mgr.34107) 3459 : cluster [DBG] pgmap v4951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:23 smithi115 bash[46934]: cluster 2022-04-22T19:06:22.835781+0000 mgr.y (mgr.34107) 3459 : cluster [DBG] pgmap v4951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:23.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:24 smithi121 bash[36902]: audit 2022-04-22T19:06:24.529986+0000 mon.a (mon.0) 6409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:24 smithi115 bash[45495]: audit 2022-04-22T19:06:24.529986+0000 mon.a (mon.0) 6409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:24 smithi115 bash[46934]: audit 2022-04-22T19:06:24.529986+0000 mon.a (mon.0) 6409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:25 smithi121 bash[36902]: cluster 2022-04-22T19:06:24.836802+0000 mgr.y (mgr.34107) 3460 : cluster [DBG] pgmap v4952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:25 smithi115 bash[45495]: cluster 2022-04-22T19:06:24.836802+0000 mgr.y (mgr.34107) 3460 : cluster [DBG] pgmap v4952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:25 smithi115 bash[46934]: cluster 2022-04-22T19:06:24.836802+0000 mgr.y (mgr.34107) 3460 : cluster [DBG] pgmap v4952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:27 smithi121 bash[36902]: cluster 2022-04-22T19:06:26.837979+0000 mgr.y (mgr.34107) 3461 : cluster [DBG] pgmap v4953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:28.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:27 smithi115 bash[45495]: cluster 2022-04-22T19:06:26.837979+0000 mgr.y (mgr.34107) 3461 : cluster [DBG] pgmap v4953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:28.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:27 smithi115 bash[46934]: cluster 2022-04-22T19:06:26.837979+0000 mgr.y (mgr.34107) 3461 : cluster [DBG] pgmap v4953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:29 smithi121 bash[36902]: cluster 2022-04-22T19:06:28.839152+0000 mgr.y (mgr.34107) 3462 : cluster [DBG] pgmap v4954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:29 smithi121 bash[36902]: audit 2022-04-22T19:06:29.533195+0000 mon.a (mon.0) 6410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:29 smithi115 bash[45495]: cluster 2022-04-22T19:06:28.839152+0000 mgr.y (mgr.34107) 3462 : cluster [DBG] pgmap v4954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:29 smithi115 bash[45495]: audit 2022-04-22T19:06:29.533195+0000 mon.a (mon.0) 6410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:29 smithi115 bash[46934]: cluster 2022-04-22T19:06:28.839152+0000 mgr.y (mgr.34107) 3462 : cluster [DBG] pgmap v4954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:29 smithi115 bash[46934]: audit 2022-04-22T19:06:29.533195+0000 mon.a (mon.0) 6410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:31 smithi121 bash[36902]: cluster 2022-04-22T19:06:30.840355+0000 mgr.y (mgr.34107) 3463 : cluster [DBG] pgmap v4955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:31 smithi115 bash[45495]: cluster 2022-04-22T19:06:30.840355+0000 mgr.y (mgr.34107) 3463 : cluster [DBG] pgmap v4955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:31 smithi115 bash[46934]: cluster 2022-04-22T19:06:30.840355+0000 mgr.y (mgr.34107) 3463 : cluster [DBG] pgmap v4955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:31.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:33 smithi121 bash[36902]: cluster 2022-04-22T19:06:32.841599+0000 mgr.y (mgr.34107) 3464 : cluster [DBG] pgmap v4956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:33 smithi115 bash[46934]: cluster 2022-04-22T19:06:32.841599+0000 mgr.y (mgr.34107) 3464 : cluster [DBG] pgmap v4956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:33 smithi115 bash[45495]: cluster 2022-04-22T19:06:32.841599+0000 mgr.y (mgr.34107) 3464 : cluster [DBG] pgmap v4956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:33.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:34 smithi121 bash[36902]: audit 2022-04-22T19:06:34.536342+0000 mon.a (mon.0) 6411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:34 smithi115 bash[45495]: audit 2022-04-22T19:06:34.536342+0000 mon.a (mon.0) 6411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:34 smithi115 bash[46934]: audit 2022-04-22T19:06:34.536342+0000 mon.a (mon.0) 6411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:35 smithi121 bash[36902]: cluster 2022-04-22T19:06:34.842716+0000 mgr.y (mgr.34107) 3465 : cluster [DBG] pgmap v4957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:35 smithi115 bash[45495]: cluster 2022-04-22T19:06:34.842716+0000 mgr.y (mgr.34107) 3465 : cluster [DBG] pgmap v4957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:35 smithi115 bash[46934]: cluster 2022-04-22T19:06:34.842716+0000 mgr.y (mgr.34107) 3465 : cluster [DBG] pgmap v4957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:38.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:37 smithi121 bash[36902]: cluster 2022-04-22T19:06:36.843839+0000 mgr.y (mgr.34107) 3466 : cluster [DBG] pgmap v4958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:37 smithi115 bash[45495]: cluster 2022-04-22T19:06:36.843839+0000 mgr.y (mgr.34107) 3466 : cluster [DBG] pgmap v4958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:37 smithi115 bash[46934]: cluster 2022-04-22T19:06:36.843839+0000 mgr.y (mgr.34107) 3466 : cluster [DBG] pgmap v4958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:39 smithi121 bash[36902]: cluster 2022-04-22T19:06:38.844873+0000 mgr.y (mgr.34107) 3467 : cluster [DBG] pgmap v4959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:39 smithi121 bash[36902]: audit 2022-04-22T19:06:39.539578+0000 mon.a (mon.0) 6412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:39 smithi115 bash[45495]: cluster 2022-04-22T19:06:38.844873+0000 mgr.y (mgr.34107) 3467 : cluster [DBG] pgmap v4959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:39 smithi115 bash[45495]: audit 2022-04-22T19:06:39.539578+0000 mon.a (mon.0) 6412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:39 smithi115 bash[46934]: cluster 2022-04-22T19:06:38.844873+0000 mgr.y (mgr.34107) 3467 : cluster [DBG] pgmap v4959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:39 smithi115 bash[46934]: audit 2022-04-22T19:06:39.539578+0000 mon.a (mon.0) 6412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:41 smithi121 bash[36902]: cluster 2022-04-22T19:06:40.845941+0000 mgr.y (mgr.34107) 3468 : cluster [DBG] pgmap v4960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:41 smithi115 bash[45495]: cluster 2022-04-22T19:06:40.845941+0000 mgr.y (mgr.34107) 3468 : cluster [DBG] pgmap v4960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:41.996Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:41 smithi115 bash[46934]: cluster 2022-04-22T19:06:40.845941+0000 mgr.y (mgr.34107) 3468 : cluster [DBG] pgmap v4960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:43 smithi121 bash[36902]: cluster 2022-04-22T19:06:42.847067+0000 mgr.y (mgr.34107) 3469 : cluster [DBG] pgmap v4961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:43.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:43 smithi115 bash[45495]: cluster 2022-04-22T19:06:42.847067+0000 mgr.y (mgr.34107) 3469 : cluster [DBG] pgmap v4961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:43 smithi115 bash[46934]: cluster 2022-04-22T19:06:42.847067+0000 mgr.y (mgr.34107) 3469 : cluster [DBG] pgmap v4961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:43.869Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:44 smithi121 bash[36902]: audit 2022-04-22T19:06:44.542858+0000 mon.a (mon.0) 6413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:44 smithi115 bash[45495]: audit 2022-04-22T19:06:44.542858+0000 mon.a (mon.0) 6413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:44 smithi115 bash[46934]: audit 2022-04-22T19:06:44.542858+0000 mon.a (mon.0) 6413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:45 smithi121 bash[36902]: cluster 2022-04-22T19:06:44.848190+0000 mgr.y (mgr.34107) 3470 : cluster [DBG] pgmap v4962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:45 smithi115 bash[45495]: cluster 2022-04-22T19:06:44.848190+0000 mgr.y (mgr.34107) 3470 : cluster [DBG] pgmap v4962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:45 smithi115 bash[46934]: cluster 2022-04-22T19:06:44.848190+0000 mgr.y (mgr.34107) 3470 : cluster [DBG] pgmap v4962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:46.244 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:06:46.589 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:06:46.589 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (89m) 6m ago 103m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:06:46.589 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (103m) 7m ago 103m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:06:46.589 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (91m) 6m ago 104m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:06:46.589 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (91m) 7m ago 104m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (99m) 7m ago 108m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (99m) 6m ago 111m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (98m) 6m ago 111m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (97m) 7m ago 109m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (98m) 6m ago 109m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (90m) 6m ago 103m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (90m) 7m ago 103m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:06:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (96m) 6m ago 108m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (95m) 6m ago 108m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (95m) 6m ago 107m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (94m) 6m ago 107m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (94m) 7m ago 106m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (93m) 7m ago 106m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (92m) 7m ago 105m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:06:46.591 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (92m) 7m ago 105m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:06:46.592 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (90m) 7m ago 103m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:06:46.997 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:06:46.998 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:06:46.998 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:06:46.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:46.998 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:06:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:06:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:06:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:06:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:06:47.000 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:06:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:47 smithi121 bash[36902]: audit 2022-04-22T19:06:46.219388+0000 mgr.y (mgr.34107) 3471 : audit [DBG] from='client.48976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:47 smithi121 bash[36902]: audit 2022-04-22T19:06:46.560438+0000 mgr.y (mgr.34107) 3472 : audit [DBG] from='client.48982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:47 smithi121 bash[36902]: cluster 2022-04-22T19:06:46.849228+0000 mgr.y (mgr.34107) 3473 : cluster [DBG] pgmap v4963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:47 smithi121 bash[36902]: audit 2022-04-22T19:06:46.987813+0000 mon.a (mon.0) 6414 : audit [DBG] from='client.? 172.21.15.115:0/1827735733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[45495]: audit 2022-04-22T19:06:46.219388+0000 mgr.y (mgr.34107) 3471 : audit [DBG] from='client.48976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[45495]: audit 2022-04-22T19:06:46.560438+0000 mgr.y (mgr.34107) 3472 : audit [DBG] from='client.48982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[45495]: cluster 2022-04-22T19:06:46.849228+0000 mgr.y (mgr.34107) 3473 : cluster [DBG] pgmap v4963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[45495]: audit 2022-04-22T19:06:46.987813+0000 mon.a (mon.0) 6414 : audit [DBG] from='client.? 172.21.15.115:0/1827735733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[46934]: audit 2022-04-22T19:06:46.219388+0000 mgr.y (mgr.34107) 3471 : audit [DBG] from='client.48976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[46934]: audit 2022-04-22T19:06:46.560438+0000 mgr.y (mgr.34107) 3472 : audit [DBG] from='client.48982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:06:48.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[46934]: cluster 2022-04-22T19:06:46.849228+0000 mgr.y (mgr.34107) 3473 : cluster [DBG] pgmap v4963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:48.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:47 smithi115 bash[46934]: audit 2022-04-22T19:06:46.987813+0000 mon.a (mon.0) 6414 : audit [DBG] from='client.? 172.21.15.115:0/1827735733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:06:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:49 smithi121 bash[36902]: cluster 2022-04-22T19:06:48.850355+0000 mgr.y (mgr.34107) 3474 : cluster [DBG] pgmap v4964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:49 smithi121 bash[36902]: audit 2022-04-22T19:06:49.545830+0000 mon.a (mon.0) 6415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:49 smithi115 bash[45495]: cluster 2022-04-22T19:06:48.850355+0000 mgr.y (mgr.34107) 3474 : cluster [DBG] pgmap v4964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:49 smithi115 bash[45495]: audit 2022-04-22T19:06:49.545830+0000 mon.a (mon.0) 6415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:49 smithi115 bash[46934]: cluster 2022-04-22T19:06:48.850355+0000 mgr.y (mgr.34107) 3474 : cluster [DBG] pgmap v4964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:49 smithi115 bash[46934]: audit 2022-04-22T19:06:49.545830+0000 mon.a (mon.0) 6415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:51 smithi121 bash[36902]: cluster 2022-04-22T19:06:50.851344+0000 mgr.y (mgr.34107) 3475 : cluster [DBG] pgmap v4965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:51 smithi115 bash[45495]: cluster 2022-04-22T19:06:50.851344+0000 mgr.y (mgr.34107) 3475 : cluster [DBG] pgmap v4965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:52.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:51 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:51.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:51 smithi115 bash[46934]: cluster 2022-04-22T19:06:50.851344+0000 mgr.y (mgr.34107) 3475 : cluster [DBG] pgmap v4965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:53 smithi121 bash[36902]: cluster 2022-04-22T19:06:52.852454+0000 mgr.y (mgr.34107) 3476 : cluster [DBG] pgmap v4966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:53.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:53 smithi115 bash[45495]: cluster 2022-04-22T19:06:52.852454+0000 mgr.y (mgr.34107) 3476 : cluster [DBG] pgmap v4966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:53.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:53 smithi115 bash[46934]: cluster 2022-04-22T19:06:52.852454+0000 mgr.y (mgr.34107) 3476 : cluster [DBG] pgmap v4966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:06:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:06:53.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:06:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:54 smithi121 bash[36902]: audit 2022-04-22T19:06:54.548440+0000 mon.a (mon.0) 6416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:54 smithi115 bash[46934]: audit 2022-04-22T19:06:54.548440+0000 mon.a (mon.0) 6416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:55.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:54 smithi115 bash[45495]: audit 2022-04-22T19:06:54.548440+0000 mon.a (mon.0) 6416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:06:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:55 smithi121 bash[36902]: cluster 2022-04-22T19:06:54.853683+0000 mgr.y (mgr.34107) 3477 : cluster [DBG] pgmap v4967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:55 smithi115 bash[45495]: cluster 2022-04-22T19:06:54.853683+0000 mgr.y (mgr.34107) 3477 : cluster [DBG] pgmap v4967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:55 smithi115 bash[46934]: cluster 2022-04-22T19:06:54.853683+0000 mgr.y (mgr.34107) 3477 : cluster [DBG] pgmap v4967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:57 smithi121 bash[36902]: cluster 2022-04-22T19:06:56.854906+0000 mgr.y (mgr.34107) 3478 : cluster [DBG] pgmap v4968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:57 smithi115 bash[45495]: cluster 2022-04-22T19:06:56.854906+0000 mgr.y (mgr.34107) 3478 : cluster [DBG] pgmap v4968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:06:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:57 smithi115 bash[46934]: cluster 2022-04-22T19:06:56.854906+0000 mgr.y (mgr.34107) 3478 : cluster [DBG] pgmap v4968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:59 smithi121 bash[36902]: cluster 2022-04-22T19:06:58.856054+0000 mgr.y (mgr.34107) 3479 : cluster [DBG] pgmap v4969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:06:59 smithi121 bash[36902]: audit 2022-04-22T19:06:59.551467+0000 mon.a (mon.0) 6417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:59 smithi115 bash[45495]: cluster 2022-04-22T19:06:58.856054+0000 mgr.y (mgr.34107) 3479 : cluster [DBG] pgmap v4969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:06:59 smithi115 bash[45495]: audit 2022-04-22T19:06:59.551467+0000 mon.a (mon.0) 6417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:59 smithi115 bash[46934]: cluster 2022-04-22T19:06:58.856054+0000 mgr.y (mgr.34107) 3479 : cluster [DBG] pgmap v4969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:06:59 smithi115 bash[46934]: audit 2022-04-22T19:06:59.551467+0000 mon.a (mon.0) 6417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:01 smithi121 bash[36902]: cluster 2022-04-22T19:07:00.857139+0000 mgr.y (mgr.34107) 3480 : cluster [DBG] pgmap v4970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:01.996Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:01 smithi115 bash[45495]: cluster 2022-04-22T19:07:00.857139+0000 mgr.y (mgr.34107) 3480 : cluster [DBG] pgmap v4970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:01 smithi115 bash[46934]: cluster 2022-04-22T19:07:00.857139+0000 mgr.y (mgr.34107) 3480 : cluster [DBG] pgmap v4970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:02 smithi121 bash[36902]: audit 2022-04-22T19:07:02.376680+0000 mon.a (mon.0) 6418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:07:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:02 smithi121 bash[36902]: audit 2022-04-22T19:07:02.380447+0000 mon.a (mon.0) 6419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:07:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:02 smithi121 bash[36902]: audit 2022-04-22T19:07:02.383994+0000 mon.a (mon.0) 6420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:07:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:02 smithi121 bash[36902]: audit 2022-04-22T19:07:02.743606+0000 mon.a (mon.0) 6421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:07:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[45495]: audit 2022-04-22T19:07:02.376680+0000 mon.a (mon.0) 6418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:07:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[45495]: audit 2022-04-22T19:07:02.380447+0000 mon.a (mon.0) 6419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:07:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[45495]: audit 2022-04-22T19:07:02.383994+0000 mon.a (mon.0) 6420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:07:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[45495]: audit 2022-04-22T19:07:02.743606+0000 mon.a (mon.0) 6421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:07:03.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[46934]: audit 2022-04-22T19:07:02.376680+0000 mon.a (mon.0) 6418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:07:03.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[46934]: audit 2022-04-22T19:07:02.380447+0000 mon.a (mon.0) 6419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:07:03.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[46934]: audit 2022-04-22T19:07:02.383994+0000 mon.a (mon.0) 6420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:07:03.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:02 smithi115 bash[46934]: audit 2022-04-22T19:07:02.743606+0000 mon.a (mon.0) 6421 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:07:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:03 smithi121 bash[36902]: cluster 2022-04-22T19:07:02.858228+0000 mgr.y (mgr.34107) 3481 : cluster [DBG] pgmap v4971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:04.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:03 smithi115 bash[45495]: cluster 2022-04-22T19:07:02.858228+0000 mgr.y (mgr.34107) 3481 : cluster [DBG] pgmap v4971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:03.869Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:03 smithi115 bash[46934]: cluster 2022-04-22T19:07:02.858228+0000 mgr.y (mgr.34107) 3481 : cluster [DBG] pgmap v4971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:04 smithi121 bash[36902]: audit 2022-04-22T19:07:04.554783+0000 mon.a (mon.0) 6422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:05.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:04 smithi115 bash[45495]: audit 2022-04-22T19:07:04.554783+0000 mon.a (mon.0) 6422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:05.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:04 smithi115 bash[46934]: audit 2022-04-22T19:07:04.554783+0000 mon.a (mon.0) 6422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:05 smithi121 bash[36902]: cluster 2022-04-22T19:07:04.859279+0000 mgr.y (mgr.34107) 3482 : cluster [DBG] pgmap v4972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:06.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:05 smithi115 bash[45495]: cluster 2022-04-22T19:07:04.859279+0000 mgr.y (mgr.34107) 3482 : cluster [DBG] pgmap v4972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:05 smithi115 bash[46934]: cluster 2022-04-22T19:07:04.859279+0000 mgr.y (mgr.34107) 3482 : cluster [DBG] pgmap v4972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:07 smithi121 bash[36902]: cluster 2022-04-22T19:07:06.860522+0000 mgr.y (mgr.34107) 3483 : cluster [DBG] pgmap v4973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:07 smithi115 bash[45495]: cluster 2022-04-22T19:07:06.860522+0000 mgr.y (mgr.34107) 3483 : cluster [DBG] pgmap v4973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:07 smithi115 bash[46934]: cluster 2022-04-22T19:07:06.860522+0000 mgr.y (mgr.34107) 3483 : cluster [DBG] pgmap v4973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:09 smithi121 bash[36902]: cluster 2022-04-22T19:07:08.861664+0000 mgr.y (mgr.34107) 3484 : cluster [DBG] pgmap v4974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:09 smithi121 bash[36902]: audit 2022-04-22T19:07:09.557965+0000 mon.a (mon.0) 6423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:09 smithi115 bash[45495]: cluster 2022-04-22T19:07:08.861664+0000 mgr.y (mgr.34107) 3484 : cluster [DBG] pgmap v4974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:09 smithi115 bash[45495]: audit 2022-04-22T19:07:09.557965+0000 mon.a (mon.0) 6423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:09 smithi115 bash[46934]: cluster 2022-04-22T19:07:08.861664+0000 mgr.y (mgr.34107) 3484 : cluster [DBG] pgmap v4974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:09 smithi115 bash[46934]: audit 2022-04-22T19:07:09.557965+0000 mon.a (mon.0) 6423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:11 smithi121 bash[36902]: cluster 2022-04-22T19:07:10.862723+0000 mgr.y (mgr.34107) 3485 : cluster [DBG] pgmap v4975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:12.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:11 smithi115 bash[45495]: cluster 2022-04-22T19:07:10.862723+0000 mgr.y (mgr.34107) 3485 : cluster [DBG] pgmap v4975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:12.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:11 smithi115 bash[46934]: cluster 2022-04-22T19:07:10.862723+0000 mgr.y (mgr.34107) 3485 : cluster [DBG] pgmap v4975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:11 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:11.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:13 smithi121 bash[36902]: cluster 2022-04-22T19:07:12.863866+0000 mgr.y (mgr.34107) 3486 : cluster [DBG] pgmap v4976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:13 smithi115 bash[45495]: cluster 2022-04-22T19:07:12.863866+0000 mgr.y (mgr.34107) 3486 : cluster [DBG] pgmap v4976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:13 smithi115 bash[46934]: cluster 2022-04-22T19:07:12.863866+0000 mgr.y (mgr.34107) 3486 : cluster [DBG] pgmap v4976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:13.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:14 smithi121 bash[36902]: audit 2022-04-22T19:07:14.409476+0000 mon.a (mon.0) 6424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:07:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:14 smithi121 bash[36902]: audit 2022-04-22T19:07:14.410018+0000 mon.a (mon.0) 6425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:07:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:14 smithi115 bash[46934]: audit 2022-04-22T19:07:14.409476+0000 mon.a (mon.0) 6424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:07:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:14 smithi115 bash[46934]: audit 2022-04-22T19:07:14.410018+0000 mon.a (mon.0) 6425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:07:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:14 smithi115 bash[45495]: audit 2022-04-22T19:07:14.409476+0000 mon.a (mon.0) 6424 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:07:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:14 smithi115 bash[45495]: audit 2022-04-22T19:07:14.410018+0000 mon.a (mon.0) 6425 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:07:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:15 smithi121 bash[36902]: audit 2022-04-22T19:07:14.560963+0000 mon.a (mon.0) 6426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:15 smithi121 bash[36902]: cluster 2022-04-22T19:07:14.864932+0000 mgr.y (mgr.34107) 3487 : cluster [DBG] pgmap v4977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:15 smithi115 bash[46934]: audit 2022-04-22T19:07:14.560963+0000 mon.a (mon.0) 6426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:15 smithi115 bash[46934]: cluster 2022-04-22T19:07:14.864932+0000 mgr.y (mgr.34107) 3487 : cluster [DBG] pgmap v4977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:15 smithi115 bash[45495]: audit 2022-04-22T19:07:14.560963+0000 mon.a (mon.0) 6426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:15 smithi115 bash[45495]: cluster 2022-04-22T19:07:14.864932+0000 mgr.y (mgr.34107) 3487 : cluster [DBG] pgmap v4977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:17.367 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:07:17.713 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:07:17.713 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (90m) 7m ago 104m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:07:17.713 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (103m) 8m ago 103m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (92m) 7m ago 105m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (91m) 8m ago 105m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (99m) 8m ago 109m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (100m) 7m ago 111m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (99m) 7m ago 111m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (98m) 8m ago 109m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (98m) 7m ago 109m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:07:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (91m) 7m ago 104m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (91m) 8m ago 104m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (97m) 7m ago 109m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (96m) 7m ago 108m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (95m) 7m ago 108m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (95m) 7m ago 107m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (94m) 8m ago 107m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:07:17.715 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (94m) 8m ago 106m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:07:17.716 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (93m) 8m ago 106m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:07:17.716 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (92m) 8m ago 105m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:07:17.716 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (91m) 8m ago 104m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:07:18.121 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:07:18.121 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:07:18.121 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:07:18.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:18.123 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:07:18.123 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:07:18.123 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:07:18.123 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:07:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:17 smithi121 bash[36902]: cluster 2022-04-22T19:07:16.866011+0000 mgr.y (mgr.34107) 3488 : cluster [DBG] pgmap v4978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:17 smithi115 bash[46934]: cluster 2022-04-22T19:07:16.866011+0000 mgr.y (mgr.34107) 3488 : cluster [DBG] pgmap v4978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:17 smithi115 bash[45495]: cluster 2022-04-22T19:07:16.866011+0000 mgr.y (mgr.34107) 3488 : cluster [DBG] pgmap v4978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:18 smithi121 bash[36902]: audit 2022-04-22T19:07:17.342950+0000 mgr.y (mgr.34107) 3489 : audit [DBG] from='client.39060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:18 smithi121 bash[36902]: audit 2022-04-22T19:07:17.685144+0000 mgr.y (mgr.34107) 3490 : audit [DBG] from='client.49000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:18 smithi121 bash[36902]: audit 2022-04-22T19:07:18.111607+0000 mon.a (mon.0) 6427 : audit [DBG] from='client.? 172.21.15.115:0/181881519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[45495]: audit 2022-04-22T19:07:17.342950+0000 mgr.y (mgr.34107) 3489 : audit [DBG] from='client.39060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[45495]: audit 2022-04-22T19:07:17.685144+0000 mgr.y (mgr.34107) 3490 : audit [DBG] from='client.49000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[45495]: audit 2022-04-22T19:07:18.111607+0000 mon.a (mon.0) 6427 : audit [DBG] from='client.? 172.21.15.115:0/181881519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[46934]: audit 2022-04-22T19:07:17.342950+0000 mgr.y (mgr.34107) 3489 : audit [DBG] from='client.39060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[46934]: audit 2022-04-22T19:07:17.685144+0000 mgr.y (mgr.34107) 3490 : audit [DBG] from='client.49000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:19.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:18 smithi115 bash[46934]: audit 2022-04-22T19:07:18.111607+0000 mon.a (mon.0) 6427 : audit [DBG] from='client.? 172.21.15.115:0/181881519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:19 smithi121 bash[36902]: cluster 2022-04-22T19:07:18.867054+0000 mgr.y (mgr.34107) 3491 : cluster [DBG] pgmap v4979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:19 smithi121 bash[36902]: audit 2022-04-22T19:07:19.563987+0000 mon.a (mon.0) 6428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:19 smithi115 bash[45495]: cluster 2022-04-22T19:07:18.867054+0000 mgr.y (mgr.34107) 3491 : cluster [DBG] pgmap v4979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:19 smithi115 bash[45495]: audit 2022-04-22T19:07:19.563987+0000 mon.a (mon.0) 6428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:19 smithi115 bash[46934]: cluster 2022-04-22T19:07:18.867054+0000 mgr.y (mgr.34107) 3491 : cluster [DBG] pgmap v4979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:19 smithi115 bash[46934]: audit 2022-04-22T19:07:19.563987+0000 mon.a (mon.0) 6428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:21 smithi121 bash[36902]: cluster 2022-04-22T19:07:20.868132+0000 mgr.y (mgr.34107) 3492 : cluster [DBG] pgmap v4980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:21 smithi115 bash[45495]: cluster 2022-04-22T19:07:20.868132+0000 mgr.y (mgr.34107) 3492 : cluster [DBG] pgmap v4980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:21 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:21.998Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:21 smithi115 bash[46934]: cluster 2022-04-22T19:07:20.868132+0000 mgr.y (mgr.34107) 3492 : cluster [DBG] pgmap v4980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:23 smithi121 bash[36902]: cluster 2022-04-22T19:07:22.869229+0000 mgr.y (mgr.34107) 3493 : cluster [DBG] pgmap v4981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:23 smithi115 bash[45495]: cluster 2022-04-22T19:07:22.869229+0000 mgr.y (mgr.34107) 3493 : cluster [DBG] pgmap v4981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:23 smithi115 bash[46934]: cluster 2022-04-22T19:07:22.869229+0000 mgr.y (mgr.34107) 3493 : cluster [DBG] pgmap v4981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:23.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:24 smithi121 bash[36902]: audit 2022-04-22T19:07:24.567054+0000 mon.a (mon.0) 6429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:24 smithi115 bash[45495]: audit 2022-04-22T19:07:24.567054+0000 mon.a (mon.0) 6429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:24 smithi115 bash[46934]: audit 2022-04-22T19:07:24.567054+0000 mon.a (mon.0) 6429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:25 smithi121 bash[36902]: cluster 2022-04-22T19:07:24.870355+0000 mgr.y (mgr.34107) 3494 : cluster [DBG] pgmap v4982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:25 smithi115 bash[45495]: cluster 2022-04-22T19:07:24.870355+0000 mgr.y (mgr.34107) 3494 : cluster [DBG] pgmap v4982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:25 smithi115 bash[46934]: cluster 2022-04-22T19:07:24.870355+0000 mgr.y (mgr.34107) 3494 : cluster [DBG] pgmap v4982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:27 smithi121 bash[36902]: cluster 2022-04-22T19:07:26.871560+0000 mgr.y (mgr.34107) 3495 : cluster [DBG] pgmap v4983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:27 smithi115 bash[45495]: cluster 2022-04-22T19:07:26.871560+0000 mgr.y (mgr.34107) 3495 : cluster [DBG] pgmap v4983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:27 smithi115 bash[46934]: cluster 2022-04-22T19:07:26.871560+0000 mgr.y (mgr.34107) 3495 : cluster [DBG] pgmap v4983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:29 smithi121 bash[36902]: cluster 2022-04-22T19:07:28.872547+0000 mgr.y (mgr.34107) 3496 : cluster [DBG] pgmap v4984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:29 smithi121 bash[36902]: audit 2022-04-22T19:07:29.570227+0000 mon.a (mon.0) 6430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:29 smithi115 bash[45495]: cluster 2022-04-22T19:07:28.872547+0000 mgr.y (mgr.34107) 3496 : cluster [DBG] pgmap v4984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:29 smithi115 bash[45495]: audit 2022-04-22T19:07:29.570227+0000 mon.a (mon.0) 6430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:29 smithi115 bash[46934]: cluster 2022-04-22T19:07:28.872547+0000 mgr.y (mgr.34107) 3496 : cluster [DBG] pgmap v4984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:29 smithi115 bash[46934]: audit 2022-04-22T19:07:29.570227+0000 mon.a (mon.0) 6430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:31 smithi121 bash[36902]: cluster 2022-04-22T19:07:30.873571+0000 mgr.y (mgr.34107) 3497 : cluster [DBG] pgmap v4985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:31 smithi115 bash[45495]: cluster 2022-04-22T19:07:30.873571+0000 mgr.y (mgr.34107) 3497 : cluster [DBG] pgmap v4985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:31 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:31.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:31 smithi115 bash[46934]: cluster 2022-04-22T19:07:30.873571+0000 mgr.y (mgr.34107) 3497 : cluster [DBG] pgmap v4985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:33 smithi121 bash[36902]: cluster 2022-04-22T19:07:32.874640+0000 mgr.y (mgr.34107) 3498 : cluster [DBG] pgmap v4986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:33 smithi115 bash[45495]: cluster 2022-04-22T19:07:32.874640+0000 mgr.y (mgr.34107) 3498 : cluster [DBG] pgmap v4986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:33 smithi115 bash[46934]: cluster 2022-04-22T19:07:32.874640+0000 mgr.y (mgr.34107) 3498 : cluster [DBG] pgmap v4986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:33.871Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:34 smithi121 bash[36902]: audit 2022-04-22T19:07:34.573616+0000 mon.a (mon.0) 6431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:34 smithi115 bash[45495]: audit 2022-04-22T19:07:34.573616+0000 mon.a (mon.0) 6431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:34 smithi115 bash[46934]: audit 2022-04-22T19:07:34.573616+0000 mon.a (mon.0) 6431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:35 smithi121 bash[36902]: cluster 2022-04-22T19:07:34.875739+0000 mgr.y (mgr.34107) 3499 : cluster [DBG] pgmap v4987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:35 smithi115 bash[45495]: cluster 2022-04-22T19:07:34.875739+0000 mgr.y (mgr.34107) 3499 : cluster [DBG] pgmap v4987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:35 smithi115 bash[46934]: cluster 2022-04-22T19:07:34.875739+0000 mgr.y (mgr.34107) 3499 : cluster [DBG] pgmap v4987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:37 smithi121 bash[36902]: cluster 2022-04-22T19:07:36.876806+0000 mgr.y (mgr.34107) 3500 : cluster [DBG] pgmap v4988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:37 smithi115 bash[45495]: cluster 2022-04-22T19:07:36.876806+0000 mgr.y (mgr.34107) 3500 : cluster [DBG] pgmap v4988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:37 smithi115 bash[46934]: cluster 2022-04-22T19:07:36.876806+0000 mgr.y (mgr.34107) 3500 : cluster [DBG] pgmap v4988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:39 smithi121 bash[36902]: cluster 2022-04-22T19:07:38.877919+0000 mgr.y (mgr.34107) 3501 : cluster [DBG] pgmap v4989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:39 smithi121 bash[36902]: audit 2022-04-22T19:07:39.576927+0000 mon.a (mon.0) 6432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:39 smithi115 bash[45495]: cluster 2022-04-22T19:07:38.877919+0000 mgr.y (mgr.34107) 3501 : cluster [DBG] pgmap v4989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:39 smithi115 bash[45495]: audit 2022-04-22T19:07:39.576927+0000 mon.a (mon.0) 6432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:39 smithi115 bash[46934]: cluster 2022-04-22T19:07:38.877919+0000 mgr.y (mgr.34107) 3501 : cluster [DBG] pgmap v4989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:39 smithi115 bash[46934]: audit 2022-04-22T19:07:39.576927+0000 mon.a (mon.0) 6432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:41 smithi121 bash[36902]: cluster 2022-04-22T19:07:40.878997+0000 mgr.y (mgr.34107) 3502 : cluster [DBG] pgmap v4990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:42.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:41 smithi115 bash[45495]: cluster 2022-04-22T19:07:40.878997+0000 mgr.y (mgr.34107) 3502 : cluster [DBG] pgmap v4990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:41 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:41.998Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:41 smithi115 bash[46934]: cluster 2022-04-22T19:07:40.878997+0000 mgr.y (mgr.34107) 3502 : cluster [DBG] pgmap v4990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:43 smithi121 bash[36902]: cluster 2022-04-22T19:07:42.880093+0000 mgr.y (mgr.34107) 3503 : cluster [DBG] pgmap v4991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:43 smithi115 bash[45495]: cluster 2022-04-22T19:07:42.880093+0000 mgr.y (mgr.34107) 3503 : cluster [DBG] pgmap v4991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:43 smithi115 bash[46934]: cluster 2022-04-22T19:07:42.880093+0000 mgr.y (mgr.34107) 3503 : cluster [DBG] pgmap v4991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:43.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:44 smithi121 bash[36902]: audit 2022-04-22T19:07:44.580315+0000 mon.a (mon.0) 6433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:44 smithi115 bash[45495]: audit 2022-04-22T19:07:44.580315+0000 mon.a (mon.0) 6433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:44 smithi115 bash[46934]: audit 2022-04-22T19:07:44.580315+0000 mon.a (mon.0) 6433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:45 smithi121 bash[36902]: cluster 2022-04-22T19:07:44.881159+0000 mgr.y (mgr.34107) 3504 : cluster [DBG] pgmap v4992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:45 smithi115 bash[45495]: cluster 2022-04-22T19:07:44.881159+0000 mgr.y (mgr.34107) 3504 : cluster [DBG] pgmap v4992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:45 smithi115 bash[46934]: cluster 2022-04-22T19:07:44.881159+0000 mgr.y (mgr.34107) 3504 : cluster [DBG] pgmap v4992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:47 smithi121 bash[36902]: cluster 2022-04-22T19:07:46.882342+0000 mgr.y (mgr.34107) 3505 : cluster [DBG] pgmap v4993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:47 smithi115 bash[45495]: cluster 2022-04-22T19:07:46.882342+0000 mgr.y (mgr.34107) 3505 : cluster [DBG] pgmap v4993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:47 smithi115 bash[46934]: cluster 2022-04-22T19:07:46.882342+0000 mgr.y (mgr.34107) 3505 : cluster [DBG] pgmap v4993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:48.491 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (90m) 7m ago 104m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (104m) 8m ago 104m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (92m) 7m ago 105m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (92m) 8m ago 105m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (100m) 8m ago 109m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (100m) 7m ago 112m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:07:48.836 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (99m) 7m ago 112m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (98m) 8m ago 110m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (99m) 7m ago 110m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (91m) 7m ago 104m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (91m) 8m ago 104m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (97m) 7m ago 109m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (96m) 7m ago 109m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (96m) 7m ago 108m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (96m) 7m ago 108m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (95m) 8m ago 107m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:07:48.837 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (94m) 8m ago 107m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:07:48.838 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (93m) 8m ago 106m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:07:48.838 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (93m) 8m ago 106m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:07:48.838 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (91m) 8m ago 105m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:07:49.233 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:07:49.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:07:49.235 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:49 smithi121 bash[36902]: audit 2022-04-22T19:07:48.466312+0000 mgr.y (mgr.34107) 3506 : audit [DBG] from='client.39078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:49 smithi121 bash[36902]: audit 2022-04-22T19:07:48.807711+0000 mgr.y (mgr.34107) 3507 : audit [DBG] from='client.39084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:49 smithi121 bash[36902]: cluster 2022-04-22T19:07:48.883376+0000 mgr.y (mgr.34107) 3508 : cluster [DBG] pgmap v4994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:49 smithi121 bash[36902]: audit 2022-04-22T19:07:49.223709+0000 mon.a (mon.0) 6434 : audit [DBG] from='client.? 172.21.15.115:0/4243092937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:49 smithi121 bash[36902]: audit 2022-04-22T19:07:49.583587+0000 mon.a (mon.0) 6435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[45495]: audit 2022-04-22T19:07:48.466312+0000 mgr.y (mgr.34107) 3506 : audit [DBG] from='client.39078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[45495]: audit 2022-04-22T19:07:48.807711+0000 mgr.y (mgr.34107) 3507 : audit [DBG] from='client.39084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[45495]: cluster 2022-04-22T19:07:48.883376+0000 mgr.y (mgr.34107) 3508 : cluster [DBG] pgmap v4994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[45495]: audit 2022-04-22T19:07:49.223709+0000 mon.a (mon.0) 6434 : audit [DBG] from='client.? 172.21.15.115:0/4243092937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[45495]: audit 2022-04-22T19:07:49.583587+0000 mon.a (mon.0) 6435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[46934]: audit 2022-04-22T19:07:48.466312+0000 mgr.y (mgr.34107) 3506 : audit [DBG] from='client.39078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[46934]: audit 2022-04-22T19:07:48.807711+0000 mgr.y (mgr.34107) 3507 : audit [DBG] from='client.39084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:07:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[46934]: cluster 2022-04-22T19:07:48.883376+0000 mgr.y (mgr.34107) 3508 : cluster [DBG] pgmap v4994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[46934]: audit 2022-04-22T19:07:49.223709+0000 mon.a (mon.0) 6434 : audit [DBG] from='client.? 172.21.15.115:0/4243092937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:07:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:49 smithi115 bash[46934]: audit 2022-04-22T19:07:49.583587+0000 mon.a (mon.0) 6435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:51 smithi121 bash[36902]: cluster 2022-04-22T19:07:50.884401+0000 mgr.y (mgr.34107) 3509 : cluster [DBG] pgmap v4995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:51 smithi115 bash[45495]: cluster 2022-04-22T19:07:50.884401+0000 mgr.y (mgr.34107) 3509 : cluster [DBG] pgmap v4995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:51.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:51 smithi115 bash[46934]: cluster 2022-04-22T19:07:50.884401+0000 mgr.y (mgr.34107) 3509 : cluster [DBG] pgmap v4995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:53 smithi121 bash[36902]: cluster 2022-04-22T19:07:52.885464+0000 mgr.y (mgr.34107) 3510 : cluster [DBG] pgmap v4996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:53 smithi115 bash[45495]: cluster 2022-04-22T19:07:52.885464+0000 mgr.y (mgr.34107) 3510 : cluster [DBG] pgmap v4996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:53 smithi115 bash[46934]: cluster 2022-04-22T19:07:52.885464+0000 mgr.y (mgr.34107) 3510 : cluster [DBG] pgmap v4996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:07:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:07:53.871Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:07:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:54 smithi121 bash[36902]: audit 2022-04-22T19:07:54.586859+0000 mon.a (mon.0) 6436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:55.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:54 smithi115 bash[45495]: audit 2022-04-22T19:07:54.586859+0000 mon.a (mon.0) 6436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:54 smithi115 bash[46934]: audit 2022-04-22T19:07:54.586859+0000 mon.a (mon.0) 6436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:07:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:55 smithi121 bash[36902]: cluster 2022-04-22T19:07:54.886720+0000 mgr.y (mgr.34107) 3511 : cluster [DBG] pgmap v4997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:55 smithi115 bash[45495]: cluster 2022-04-22T19:07:54.886720+0000 mgr.y (mgr.34107) 3511 : cluster [DBG] pgmap v4997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:55 smithi115 bash[46934]: cluster 2022-04-22T19:07:54.886720+0000 mgr.y (mgr.34107) 3511 : cluster [DBG] pgmap v4997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:57 smithi121 bash[36902]: cluster 2022-04-22T19:07:56.887903+0000 mgr.y (mgr.34107) 3512 : cluster [DBG] pgmap v4998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:57 smithi115 bash[46934]: cluster 2022-04-22T19:07:56.887903+0000 mgr.y (mgr.34107) 3512 : cluster [DBG] pgmap v4998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:07:58.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:57 smithi115 bash[45495]: cluster 2022-04-22T19:07:56.887903+0000 mgr.y (mgr.34107) 3512 : cluster [DBG] pgmap v4998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:59 smithi121 bash[36902]: cluster 2022-04-22T19:07:58.888989+0000 mgr.y (mgr.34107) 3513 : cluster [DBG] pgmap v4999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:00.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:07:59 smithi121 bash[36902]: audit 2022-04-22T19:07:59.590176+0000 mon.a (mon.0) 6437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:59 smithi115 bash[45495]: cluster 2022-04-22T19:07:58.888989+0000 mgr.y (mgr.34107) 3513 : cluster [DBG] pgmap v4999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:07:59 smithi115 bash[45495]: audit 2022-04-22T19:07:59.590176+0000 mon.a (mon.0) 6437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:59 smithi115 bash[46934]: cluster 2022-04-22T19:07:58.888989+0000 mgr.y (mgr.34107) 3513 : cluster [DBG] pgmap v4999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:07:59 smithi115 bash[46934]: audit 2022-04-22T19:07:59.590176+0000 mon.a (mon.0) 6437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:01 smithi121 bash[36902]: cluster 2022-04-22T19:08:00.890077+0000 mgr.y (mgr.34107) 3514 : cluster [DBG] pgmap v5000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:01 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:01.998Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:01 smithi115 bash[45495]: cluster 2022-04-22T19:08:00.890077+0000 mgr.y (mgr.34107) 3514 : cluster [DBG] pgmap v5000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:01 smithi115 bash[46934]: cluster 2022-04-22T19:08:00.890077+0000 mgr.y (mgr.34107) 3514 : cluster [DBG] pgmap v5000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[46934]: audit 2022-04-22T19:08:02.751940+0000 mon.a (mon.0) 6438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[46934]: audit 2022-04-22T19:08:02.755274+0000 mon.a (mon.0) 6439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[46934]: audit 2022-04-22T19:08:02.757206+0000 mon.a (mon.0) 6440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[46934]: cluster 2022-04-22T19:08:02.891166+0000 mgr.y (mgr.34107) 3515 : cluster [DBG] pgmap v5001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[46934]: audit 2022-04-22T19:08:03.384572+0000 mon.a (mon.0) 6441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:03.424 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[45495]: audit 2022-04-22T19:08:02.751940+0000 mon.a (mon.0) 6438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:08:03.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[45495]: audit 2022-04-22T19:08:02.755274+0000 mon.a (mon.0) 6439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:08:03.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[45495]: audit 2022-04-22T19:08:02.757206+0000 mon.a (mon.0) 6440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:08:03.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[45495]: cluster 2022-04-22T19:08:02.891166+0000 mgr.y (mgr.34107) 3515 : cluster [DBG] pgmap v5001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:03.425 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[45495]: audit 2022-04-22T19:08:03.384572+0000 mon.a (mon.0) 6441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:03 smithi121 bash[36902]: audit 2022-04-22T19:08:02.751940+0000 mon.a (mon.0) 6438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:08:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:03 smithi121 bash[36902]: audit 2022-04-22T19:08:02.755274+0000 mon.a (mon.0) 6439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:08:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:03 smithi121 bash[36902]: audit 2022-04-22T19:08:02.757206+0000 mon.a (mon.0) 6440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:08:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:03 smithi121 bash[36902]: cluster 2022-04-22T19:08:02.891166+0000 mgr.y (mgr.34107) 3515 : cluster [DBG] pgmap v5001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:03.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:03 smithi121 bash[36902]: audit 2022-04-22T19:08:03.384572+0000 mon.a (mon.0) 6441 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:03.873Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:04 smithi121 bash[36902]: audit 2022-04-22T19:08:04.593370+0000 mon.a (mon.0) 6442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:05.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:04 smithi115 bash[45495]: audit 2022-04-22T19:08:04.593370+0000 mon.a (mon.0) 6442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:04 smithi115 bash[46934]: audit 2022-04-22T19:08:04.593370+0000 mon.a (mon.0) 6442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:05 smithi121 bash[36902]: cluster 2022-04-22T19:08:04.892299+0000 mgr.y (mgr.34107) 3516 : cluster [DBG] pgmap v5002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:05 smithi115 bash[45495]: cluster 2022-04-22T19:08:04.892299+0000 mgr.y (mgr.34107) 3516 : cluster [DBG] pgmap v5002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:05 smithi115 bash[46934]: cluster 2022-04-22T19:08:04.892299+0000 mgr.y (mgr.34107) 3516 : cluster [DBG] pgmap v5002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:07 smithi121 bash[36902]: cluster 2022-04-22T19:08:06.893478+0000 mgr.y (mgr.34107) 3517 : cluster [DBG] pgmap v5003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:07 smithi115 bash[45495]: cluster 2022-04-22T19:08:06.893478+0000 mgr.y (mgr.34107) 3517 : cluster [DBG] pgmap v5003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:07 smithi115 bash[46934]: cluster 2022-04-22T19:08:06.893478+0000 mgr.y (mgr.34107) 3517 : cluster [DBG] pgmap v5003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:09 smithi121 bash[36902]: cluster 2022-04-22T19:08:08.894553+0000 mgr.y (mgr.34107) 3518 : cluster [DBG] pgmap v5004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:09 smithi121 bash[36902]: audit 2022-04-22T19:08:09.596278+0000 mon.a (mon.0) 6443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:09 smithi115 bash[45495]: cluster 2022-04-22T19:08:08.894553+0000 mgr.y (mgr.34107) 3518 : cluster [DBG] pgmap v5004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:09 smithi115 bash[45495]: audit 2022-04-22T19:08:09.596278+0000 mon.a (mon.0) 6443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:09 smithi115 bash[46934]: cluster 2022-04-22T19:08:08.894553+0000 mgr.y (mgr.34107) 3518 : cluster [DBG] pgmap v5004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:09 smithi115 bash[46934]: audit 2022-04-22T19:08:09.596278+0000 mon.a (mon.0) 6443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:11 smithi121 bash[36902]: cluster 2022-04-22T19:08:10.895636+0000 mgr.y (mgr.34107) 3519 : cluster [DBG] pgmap v5005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:11 smithi121 bash[36902]: audit 2022-04-22T19:08:11.771839+0000 mon.a (mon.0) 6444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:11 smithi121 bash[36902]: audit 2022-04-22T19:08:11.778205+0000 mon.a (mon.0) 6445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:12.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:12.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[45495]: cluster 2022-04-22T19:08:10.895636+0000 mgr.y (mgr.34107) 3519 : cluster [DBG] pgmap v5005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[45495]: audit 2022-04-22T19:08:11.771839+0000 mon.a (mon.0) 6444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[45495]: audit 2022-04-22T19:08:11.778205+0000 mon.a (mon.0) 6445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[46934]: cluster 2022-04-22T19:08:10.895636+0000 mgr.y (mgr.34107) 3519 : cluster [DBG] pgmap v5005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[46934]: audit 2022-04-22T19:08:11.771839+0000 mon.a (mon.0) 6444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:11 smithi115 bash[46934]: audit 2022-04-22T19:08:11.778205+0000 mon.a (mon.0) 6445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:08:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:12 smithi121 bash[36902]: cephadm 2022-04-22T19:08:11.760104+0000 mgr.y (mgr.34107) 3520 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:08:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:12 smithi115 bash[45495]: cephadm 2022-04-22T19:08:11.760104+0000 mgr.y (mgr.34107) 3520 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:08:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:12 smithi115 bash[46934]: cephadm 2022-04-22T19:08:11.760104+0000 mgr.y (mgr.34107) 3520 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:08:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:13 smithi121 bash[36902]: cluster 2022-04-22T19:08:12.896670+0000 mgr.y (mgr.34107) 3521 : cluster [DBG] pgmap v5006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:13 smithi115 bash[46934]: cluster 2022-04-22T19:08:12.896670+0000 mgr.y (mgr.34107) 3521 : cluster [DBG] pgmap v5006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:13.874Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:14.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:13 smithi115 bash[45495]: cluster 2022-04-22T19:08:12.896670+0000 mgr.y (mgr.34107) 3521 : cluster [DBG] pgmap v5006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:14 smithi121 bash[36902]: audit 2022-04-22T19:08:14.421067+0000 mon.a (mon.0) 6446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:08:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:14 smithi121 bash[36902]: audit 2022-04-22T19:08:14.421656+0000 mon.a (mon.0) 6447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:08:15.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:14 smithi121 bash[36902]: audit 2022-04-22T19:08:14.599301+0000 mon.a (mon.0) 6448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[46934]: audit 2022-04-22T19:08:14.421067+0000 mon.a (mon.0) 6446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:08:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[46934]: audit 2022-04-22T19:08:14.421656+0000 mon.a (mon.0) 6447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:08:15.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[46934]: audit 2022-04-22T19:08:14.599301+0000 mon.a (mon.0) 6448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[45495]: audit 2022-04-22T19:08:14.421067+0000 mon.a (mon.0) 6446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:08:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[45495]: audit 2022-04-22T19:08:14.421656+0000 mon.a (mon.0) 6447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:08:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:14 smithi115 bash[45495]: audit 2022-04-22T19:08:14.599301+0000 mon.a (mon.0) 6448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:15 smithi121 bash[36902]: cluster 2022-04-22T19:08:14.897820+0000 mgr.y (mgr.34107) 3522 : cluster [DBG] pgmap v5007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:15 smithi115 bash[45495]: cluster 2022-04-22T19:08:14.897820+0000 mgr.y (mgr.34107) 3522 : cluster [DBG] pgmap v5007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:15 smithi115 bash[46934]: cluster 2022-04-22T19:08:14.897820+0000 mgr.y (mgr.34107) 3522 : cluster [DBG] pgmap v5007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:17 smithi121 bash[36902]: cluster 2022-04-22T19:08:16.898973+0000 mgr.y (mgr.34107) 3523 : cluster [DBG] pgmap v5008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:17 smithi115 bash[45495]: cluster 2022-04-22T19:08:16.898973+0000 mgr.y (mgr.34107) 3523 : cluster [DBG] pgmap v5008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:17 smithi115 bash[46934]: cluster 2022-04-22T19:08:16.898973+0000 mgr.y (mgr.34107) 3523 : cluster [DBG] pgmap v5008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:19.599 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:08:19.940 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:08:19.945 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (91m) 8m ago 105m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:08:19.945 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (104m) 9m ago 104m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (93m) 8m ago 106m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (92m) 9m ago 106m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (100m) 9m ago 110m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (101m) 8m ago 112m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (100m) 8m ago 112m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (99m) 9m ago 110m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (99m) 8m ago 110m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (92m) 8m ago 105m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:08:19.946 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (92m) 9m ago 105m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (98m) 8m ago 110m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (97m) 8m ago 109m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (96m) 8m ago 109m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (96m) 8m ago 108m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (95m) 9m ago 108m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (95m) 9m ago 107m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (94m) 9m ago 107m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (93m) 9m ago 106m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:08:19.947 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (92m) 9m ago 105m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:08:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:19 smithi121 bash[36902]: cluster 2022-04-22T19:08:18.900062+0000 mgr.y (mgr.34107) 3524 : cluster [DBG] pgmap v5009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:19 smithi121 bash[36902]: audit 2022-04-22T19:08:19.602261+0000 mon.a (mon.0) 6449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:19 smithi115 bash[46934]: cluster 2022-04-22T19:08:18.900062+0000 mgr.y (mgr.34107) 3524 : cluster [DBG] pgmap v5009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:19 smithi115 bash[46934]: audit 2022-04-22T19:08:19.602261+0000 mon.a (mon.0) 6449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:19 smithi115 bash[45495]: cluster 2022-04-22T19:08:18.900062+0000 mgr.y (mgr.34107) 3524 : cluster [DBG] pgmap v5009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:19 smithi115 bash[45495]: audit 2022-04-22T19:08:19.602261+0000 mon.a (mon.0) 6449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:08:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:08:20.348 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:08:20.349 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:08:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:21 smithi121 bash[36902]: audit 2022-04-22T19:08:19.574648+0000 mgr.y (mgr.34107) 3525 : audit [DBG] from='client.49030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:21 smithi121 bash[36902]: audit 2022-04-22T19:08:19.908694+0000 mgr.y (mgr.34107) 3526 : audit [DBG] from='client.49036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:21 smithi121 bash[36902]: audit 2022-04-22T19:08:20.337307+0000 mon.a (mon.0) 6450 : audit [DBG] from='client.? 172.21.15.115:0/2481019500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[45495]: audit 2022-04-22T19:08:19.574648+0000 mgr.y (mgr.34107) 3525 : audit [DBG] from='client.49030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[45495]: audit 2022-04-22T19:08:19.908694+0000 mgr.y (mgr.34107) 3526 : audit [DBG] from='client.49036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[45495]: audit 2022-04-22T19:08:20.337307+0000 mon.a (mon.0) 6450 : audit [DBG] from='client.? 172.21.15.115:0/2481019500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[46934]: audit 2022-04-22T19:08:19.574648+0000 mgr.y (mgr.34107) 3525 : audit [DBG] from='client.49030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[46934]: audit 2022-04-22T19:08:19.908694+0000 mgr.y (mgr.34107) 3526 : audit [DBG] from='client.49036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:20 smithi115 bash[46934]: audit 2022-04-22T19:08:20.337307+0000 mon.a (mon.0) 6450 : audit [DBG] from='client.? 172.21.15.115:0/2481019500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:22 smithi121 bash[36902]: cluster 2022-04-22T19:08:20.901244+0000 mgr.y (mgr.34107) 3527 : cluster [DBG] pgmap v5010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:21 smithi115 bash[45495]: cluster 2022-04-22T19:08:20.901244+0000 mgr.y (mgr.34107) 3527 : cluster [DBG] pgmap v5010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:22.003Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:21 smithi115 bash[46934]: cluster 2022-04-22T19:08:20.901244+0000 mgr.y (mgr.34107) 3527 : cluster [DBG] pgmap v5010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:23 smithi121 bash[36902]: cluster 2022-04-22T19:08:22.902409+0000 mgr.y (mgr.34107) 3528 : cluster [DBG] pgmap v5011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:23 smithi115 bash[45495]: cluster 2022-04-22T19:08:22.902409+0000 mgr.y (mgr.34107) 3528 : cluster [DBG] pgmap v5011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:23 smithi115 bash[46934]: cluster 2022-04-22T19:08:22.902409+0000 mgr.y (mgr.34107) 3528 : cluster [DBG] pgmap v5011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:23.874Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:24 smithi121 bash[36902]: audit 2022-04-22T19:08:24.605090+0000 mon.a (mon.0) 6451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:24 smithi115 bash[45495]: audit 2022-04-22T19:08:24.605090+0000 mon.a (mon.0) 6451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:24 smithi115 bash[46934]: audit 2022-04-22T19:08:24.605090+0000 mon.a (mon.0) 6451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:25 smithi121 bash[36902]: cluster 2022-04-22T19:08:24.903582+0000 mgr.y (mgr.34107) 3529 : cluster [DBG] pgmap v5012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:25 smithi115 bash[45495]: cluster 2022-04-22T19:08:24.903582+0000 mgr.y (mgr.34107) 3529 : cluster [DBG] pgmap v5012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:25 smithi115 bash[46934]: cluster 2022-04-22T19:08:24.903582+0000 mgr.y (mgr.34107) 3529 : cluster [DBG] pgmap v5012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:27 smithi121 bash[36902]: cluster 2022-04-22T19:08:26.904771+0000 mgr.y (mgr.34107) 3530 : cluster [DBG] pgmap v5013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:27 smithi115 bash[45495]: cluster 2022-04-22T19:08:26.904771+0000 mgr.y (mgr.34107) 3530 : cluster [DBG] pgmap v5013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:27 smithi115 bash[46934]: cluster 2022-04-22T19:08:26.904771+0000 mgr.y (mgr.34107) 3530 : cluster [DBG] pgmap v5013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:29 smithi121 bash[36902]: cluster 2022-04-22T19:08:28.905782+0000 mgr.y (mgr.34107) 3531 : cluster [DBG] pgmap v5014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:29 smithi121 bash[36902]: audit 2022-04-22T19:08:29.608345+0000 mon.a (mon.0) 6452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:29 smithi115 bash[45495]: cluster 2022-04-22T19:08:28.905782+0000 mgr.y (mgr.34107) 3531 : cluster [DBG] pgmap v5014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:29 smithi115 bash[45495]: audit 2022-04-22T19:08:29.608345+0000 mon.a (mon.0) 6452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:29 smithi115 bash[46934]: cluster 2022-04-22T19:08:28.905782+0000 mgr.y (mgr.34107) 3531 : cluster [DBG] pgmap v5014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:29 smithi115 bash[46934]: audit 2022-04-22T19:08:29.608345+0000 mon.a (mon.0) 6452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:31 smithi121 bash[36902]: cluster 2022-04-22T19:08:30.906854+0000 mgr.y (mgr.34107) 3532 : cluster [DBG] pgmap v5015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:31 smithi115 bash[45495]: cluster 2022-04-22T19:08:30.906854+0000 mgr.y (mgr.34107) 3532 : cluster [DBG] pgmap v5015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:31 smithi115 bash[46934]: cluster 2022-04-22T19:08:30.906854+0000 mgr.y (mgr.34107) 3532 : cluster [DBG] pgmap v5015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:32.002Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:33 smithi121 bash[36902]: cluster 2022-04-22T19:08:32.907892+0000 mgr.y (mgr.34107) 3533 : cluster [DBG] pgmap v5016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:33 smithi115 bash[45495]: cluster 2022-04-22T19:08:32.907892+0000 mgr.y (mgr.34107) 3533 : cluster [DBG] pgmap v5016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:33.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:33 smithi115 bash[46934]: cluster 2022-04-22T19:08:32.907892+0000 mgr.y (mgr.34107) 3533 : cluster [DBG] pgmap v5016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:33.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:34 smithi121 bash[36902]: audit 2022-04-22T19:08:34.611553+0000 mon.a (mon.0) 6453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:34 smithi115 bash[45495]: audit 2022-04-22T19:08:34.611553+0000 mon.a (mon.0) 6453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:34 smithi115 bash[46934]: audit 2022-04-22T19:08:34.611553+0000 mon.a (mon.0) 6453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:35 smithi121 bash[36902]: cluster 2022-04-22T19:08:34.908988+0000 mgr.y (mgr.34107) 3534 : cluster [DBG] pgmap v5017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:35 smithi115 bash[45495]: cluster 2022-04-22T19:08:34.908988+0000 mgr.y (mgr.34107) 3534 : cluster [DBG] pgmap v5017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:35 smithi115 bash[46934]: cluster 2022-04-22T19:08:34.908988+0000 mgr.y (mgr.34107) 3534 : cluster [DBG] pgmap v5017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:37 smithi121 bash[36902]: cluster 2022-04-22T19:08:36.910232+0000 mgr.y (mgr.34107) 3535 : cluster [DBG] pgmap v5018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:37 smithi115 bash[45495]: cluster 2022-04-22T19:08:36.910232+0000 mgr.y (mgr.34107) 3535 : cluster [DBG] pgmap v5018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:37 smithi115 bash[46934]: cluster 2022-04-22T19:08:36.910232+0000 mgr.y (mgr.34107) 3535 : cluster [DBG] pgmap v5018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:39 smithi121 bash[36902]: cluster 2022-04-22T19:08:38.911304+0000 mgr.y (mgr.34107) 3536 : cluster [DBG] pgmap v5019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:39 smithi121 bash[36902]: audit 2022-04-22T19:08:39.614824+0000 mon.a (mon.0) 6454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:39 smithi115 bash[45495]: cluster 2022-04-22T19:08:38.911304+0000 mgr.y (mgr.34107) 3536 : cluster [DBG] pgmap v5019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:39 smithi115 bash[45495]: audit 2022-04-22T19:08:39.614824+0000 mon.a (mon.0) 6454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:39 smithi115 bash[46934]: cluster 2022-04-22T19:08:38.911304+0000 mgr.y (mgr.34107) 3536 : cluster [DBG] pgmap v5019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:40.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:39 smithi115 bash[46934]: audit 2022-04-22T19:08:39.614824+0000 mon.a (mon.0) 6454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:41 smithi121 bash[36902]: cluster 2022-04-22T19:08:40.912436+0000 mgr.y (mgr.34107) 3537 : cluster [DBG] pgmap v5020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:42.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:41 smithi115 bash[45495]: cluster 2022-04-22T19:08:40.912436+0000 mgr.y (mgr.34107) 3537 : cluster [DBG] pgmap v5020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:42.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:42.002Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:41 smithi115 bash[46934]: cluster 2022-04-22T19:08:40.912436+0000 mgr.y (mgr.34107) 3537 : cluster [DBG] pgmap v5020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:43 smithi121 bash[36902]: cluster 2022-04-22T19:08:42.913427+0000 mgr.y (mgr.34107) 3538 : cluster [DBG] pgmap v5021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:43 smithi115 bash[45495]: cluster 2022-04-22T19:08:42.913427+0000 mgr.y (mgr.34107) 3538 : cluster [DBG] pgmap v5021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:43 smithi115 bash[46934]: cluster 2022-04-22T19:08:42.913427+0000 mgr.y (mgr.34107) 3538 : cluster [DBG] pgmap v5021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:43.874Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:44 smithi121 bash[36902]: audit 2022-04-22T19:08:44.618022+0000 mon.a (mon.0) 6455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:44 smithi115 bash[45495]: audit 2022-04-22T19:08:44.618022+0000 mon.a (mon.0) 6455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:44 smithi115 bash[46934]: audit 2022-04-22T19:08:44.618022+0000 mon.a (mon.0) 6455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:45 smithi121 bash[36902]: cluster 2022-04-22T19:08:44.914541+0000 mgr.y (mgr.34107) 3539 : cluster [DBG] pgmap v5022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:45 smithi115 bash[45495]: cluster 2022-04-22T19:08:44.914541+0000 mgr.y (mgr.34107) 3539 : cluster [DBG] pgmap v5022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:45 smithi115 bash[46934]: cluster 2022-04-22T19:08:44.914541+0000 mgr.y (mgr.34107) 3539 : cluster [DBG] pgmap v5022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:47 smithi121 bash[36902]: cluster 2022-04-22T19:08:46.915661+0000 mgr.y (mgr.34107) 3540 : cluster [DBG] pgmap v5023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:47 smithi115 bash[45495]: cluster 2022-04-22T19:08:46.915661+0000 mgr.y (mgr.34107) 3540 : cluster [DBG] pgmap v5023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:47 smithi115 bash[46934]: cluster 2022-04-22T19:08:46.915661+0000 mgr.y (mgr.34107) 3540 : cluster [DBG] pgmap v5023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:49 smithi121 bash[36902]: cluster 2022-04-22T19:08:48.916663+0000 mgr.y (mgr.34107) 3541 : cluster [DBG] pgmap v5024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:49 smithi121 bash[36902]: audit 2022-04-22T19:08:49.621370+0000 mon.a (mon.0) 6456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:49 smithi115 bash[45495]: cluster 2022-04-22T19:08:48.916663+0000 mgr.y (mgr.34107) 3541 : cluster [DBG] pgmap v5024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:49 smithi115 bash[45495]: audit 2022-04-22T19:08:49.621370+0000 mon.a (mon.0) 6456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:49 smithi115 bash[46934]: cluster 2022-04-22T19:08:48.916663+0000 mgr.y (mgr.34107) 3541 : cluster [DBG] pgmap v5024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:49 smithi115 bash[46934]: audit 2022-04-22T19:08:49.621370+0000 mon.a (mon.0) 6456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:50.726 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (91m) 8m ago 105m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (105m) 9m ago 105m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (93m) 8m ago 106m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (93m) 9m ago 106m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:08:51.074 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (101m) 9m ago 110m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (101m) 8m ago 113m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (100m) 8m ago 113m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (99m) 9m ago 111m 161M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (100m) 8m ago 111m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (93m) 8m ago 105m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (92m) 9m ago 105m 9.97M - 1dbe0e931976 f4b044642a96 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (98m) 8m ago 110m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (97m) 8m ago 110m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (97m) 8m ago 109m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:08:51.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (97m) 8m ago 109m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:08:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (96m) 9m ago 108m 569M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:08:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (95m) 9m ago 108m 700M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:08:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (94m) 9m ago 107m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:08:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (94m) 9m ago 107m 630M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:08:51.076 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (92m) 9m ago 106m 64.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:08:51.492 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:08:51.492 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:08:51.492 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:08:51.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:51.492 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:08:51.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:08:51.494 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:08:51.494 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:08:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:51 smithi121 bash[36902]: audit 2022-04-22T19:08:50.697636+0000 mgr.y (mgr.34107) 3542 : audit [DBG] from='client.49048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:51 smithi121 bash[36902]: cluster 2022-04-22T19:08:50.917752+0000 mgr.y (mgr.34107) 3543 : cluster [DBG] pgmap v5025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:51 smithi121 bash[36902]: audit 2022-04-22T19:08:51.043986+0000 mgr.y (mgr.34107) 3544 : audit [DBG] from='client.39120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:51 smithi121 bash[36902]: audit 2022-04-22T19:08:51.482180+0000 mon.a (mon.0) 6457 : audit [DBG] from='client.? 172.21.15.115:0/1941710706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:52.003Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[45495]: audit 2022-04-22T19:08:50.697636+0000 mgr.y (mgr.34107) 3542 : audit [DBG] from='client.49048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[45495]: cluster 2022-04-22T19:08:50.917752+0000 mgr.y (mgr.34107) 3543 : cluster [DBG] pgmap v5025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[45495]: audit 2022-04-22T19:08:51.043986+0000 mgr.y (mgr.34107) 3544 : audit [DBG] from='client.39120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[45495]: audit 2022-04-22T19:08:51.482180+0000 mon.a (mon.0) 6457 : audit [DBG] from='client.? 172.21.15.115:0/1941710706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[46934]: audit 2022-04-22T19:08:50.697636+0000 mgr.y (mgr.34107) 3542 : audit [DBG] from='client.49048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[46934]: cluster 2022-04-22T19:08:50.917752+0000 mgr.y (mgr.34107) 3543 : cluster [DBG] pgmap v5025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[46934]: audit 2022-04-22T19:08:51.043986+0000 mgr.y (mgr.34107) 3544 : audit [DBG] from='client.39120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:08:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:51 smithi115 bash[46934]: audit 2022-04-22T19:08:51.482180+0000 mon.a (mon.0) 6457 : audit [DBG] from='client.? 172.21.15.115:0/1941710706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:08:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:53 smithi121 bash[36902]: cluster 2022-04-22T19:08:52.918738+0000 mgr.y (mgr.34107) 3545 : cluster [DBG] pgmap v5026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:53 smithi115 bash[45495]: cluster 2022-04-22T19:08:52.918738+0000 mgr.y (mgr.34107) 3545 : cluster [DBG] pgmap v5026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:53 smithi115 bash[46934]: cluster 2022-04-22T19:08:52.918738+0000 mgr.y (mgr.34107) 3545 : cluster [DBG] pgmap v5026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:08:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:08:53.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:08:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:54 smithi121 bash[36902]: audit 2022-04-22T19:08:54.624517+0000 mon.a (mon.0) 6458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:55.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:54 smithi115 bash[45495]: audit 2022-04-22T19:08:54.624517+0000 mon.a (mon.0) 6458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:54 smithi115 bash[46934]: audit 2022-04-22T19:08:54.624517+0000 mon.a (mon.0) 6458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:08:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:55 smithi121 bash[36902]: cluster 2022-04-22T19:08:54.920018+0000 mgr.y (mgr.34107) 3546 : cluster [DBG] pgmap v5027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:55 smithi115 bash[45495]: cluster 2022-04-22T19:08:54.920018+0000 mgr.y (mgr.34107) 3546 : cluster [DBG] pgmap v5027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:55 smithi115 bash[46934]: cluster 2022-04-22T19:08:54.920018+0000 mgr.y (mgr.34107) 3546 : cluster [DBG] pgmap v5027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:57 smithi121 bash[36902]: cluster 2022-04-22T19:08:56.921230+0000 mgr.y (mgr.34107) 3547 : cluster [DBG] pgmap v5028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:57 smithi115 bash[45495]: cluster 2022-04-22T19:08:56.921230+0000 mgr.y (mgr.34107) 3547 : cluster [DBG] pgmap v5028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:08:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:57 smithi115 bash[46934]: cluster 2022-04-22T19:08:56.921230+0000 mgr.y (mgr.34107) 3547 : cluster [DBG] pgmap v5028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:59 smithi121 bash[36902]: cluster 2022-04-22T19:08:58.922376+0000 mgr.y (mgr.34107) 3548 : cluster [DBG] pgmap v5029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:00.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:08:59 smithi121 bash[36902]: audit 2022-04-22T19:08:59.627765+0000 mon.a (mon.0) 6459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:59 smithi115 bash[46934]: cluster 2022-04-22T19:08:58.922376+0000 mgr.y (mgr.34107) 3548 : cluster [DBG] pgmap v5029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:08:59 smithi115 bash[46934]: audit 2022-04-22T19:08:59.627765+0000 mon.a (mon.0) 6459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:59 smithi115 bash[45495]: cluster 2022-04-22T19:08:58.922376+0000 mgr.y (mgr.34107) 3548 : cluster [DBG] pgmap v5029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:08:59 smithi115 bash[45495]: audit 2022-04-22T19:08:59.627765+0000 mon.a (mon.0) 6459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:01 smithi121 bash[36902]: cluster 2022-04-22T19:09:00.923574+0000 mgr.y (mgr.34107) 3549 : cluster [DBG] pgmap v5030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:02.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:01 smithi115 bash[46934]: cluster 2022-04-22T19:09:00.923574+0000 mgr.y (mgr.34107) 3549 : cluster [DBG] pgmap v5030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:02.003Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:01 smithi115 bash[45495]: cluster 2022-04-22T19:09:00.923574+0000 mgr.y (mgr.34107) 3549 : cluster [DBG] pgmap v5030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:03 smithi121 bash[36902]: cluster 2022-04-22T19:09:02.924585+0000 mgr.y (mgr.34107) 3550 : cluster [DBG] pgmap v5031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:03 smithi115 bash[45495]: cluster 2022-04-22T19:09:02.924585+0000 mgr.y (mgr.34107) 3550 : cluster [DBG] pgmap v5031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:03 smithi115 bash[46934]: cluster 2022-04-22T19:09:02.924585+0000 mgr.y (mgr.34107) 3550 : cluster [DBG] pgmap v5031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:03.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:04 smithi121 bash[36902]: audit 2022-04-22T19:09:04.630852+0000 mon.a (mon.0) 6460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:05.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:04 smithi115 bash[45495]: audit 2022-04-22T19:09:04.630852+0000 mon.a (mon.0) 6460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:04 smithi115 bash[46934]: audit 2022-04-22T19:09:04.630852+0000 mon.a (mon.0) 6460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:05 smithi121 bash[36902]: cluster 2022-04-22T19:09:04.925667+0000 mgr.y (mgr.34107) 3551 : cluster [DBG] pgmap v5032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:05 smithi115 bash[46934]: cluster 2022-04-22T19:09:04.925667+0000 mgr.y (mgr.34107) 3551 : cluster [DBG] pgmap v5032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:05 smithi115 bash[45495]: cluster 2022-04-22T19:09:04.925667+0000 mgr.y (mgr.34107) 3551 : cluster [DBG] pgmap v5032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:07 smithi121 bash[36902]: cluster 2022-04-22T19:09:06.926691+0000 mgr.y (mgr.34107) 3552 : cluster [DBG] pgmap v5033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:07 smithi115 bash[46934]: cluster 2022-04-22T19:09:06.926691+0000 mgr.y (mgr.34107) 3552 : cluster [DBG] pgmap v5033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:07 smithi115 bash[45495]: cluster 2022-04-22T19:09:06.926691+0000 mgr.y (mgr.34107) 3552 : cluster [DBG] pgmap v5033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:09 smithi121 bash[36902]: cluster 2022-04-22T19:09:08.927755+0000 mgr.y (mgr.34107) 3553 : cluster [DBG] pgmap v5034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:09 smithi121 bash[36902]: audit 2022-04-22T19:09:09.633929+0000 mon.a (mon.0) 6461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:09 smithi115 bash[45495]: cluster 2022-04-22T19:09:08.927755+0000 mgr.y (mgr.34107) 3553 : cluster [DBG] pgmap v5034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:09 smithi115 bash[45495]: audit 2022-04-22T19:09:09.633929+0000 mon.a (mon.0) 6461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:09 smithi115 bash[46934]: cluster 2022-04-22T19:09:08.927755+0000 mgr.y (mgr.34107) 3553 : cluster [DBG] pgmap v5034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:10.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:09 smithi115 bash[46934]: audit 2022-04-22T19:09:09.633929+0000 mon.a (mon.0) 6461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:12.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[45495]: cluster 2022-04-22T19:09:10.928942+0000 mgr.y (mgr.34107) 3554 : cluster [DBG] pgmap v5035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:12.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[45495]: audit 2022-04-22T19:09:11.788353+0000 mon.a (mon.0) 6462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:09:12.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[45495]: audit 2022-04-22T19:09:11.791523+0000 mon.a (mon.0) 6463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:09:12.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[45495]: audit 2022-04-22T19:09:11.793090+0000 mon.a (mon.0) 6464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:09:12.134 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:12.002Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:12.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[46934]: cluster 2022-04-22T19:09:10.928942+0000 mgr.y (mgr.34107) 3554 : cluster [DBG] pgmap v5035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:12.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[46934]: audit 2022-04-22T19:09:11.788353+0000 mon.a (mon.0) 6462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:09:12.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[46934]: audit 2022-04-22T19:09:11.791523+0000 mon.a (mon.0) 6463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:09:12.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:11 smithi115 bash[46934]: audit 2022-04-22T19:09:11.793090+0000 mon.a (mon.0) 6464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:11 smithi121 bash[36902]: cluster 2022-04-22T19:09:10.928942+0000 mgr.y (mgr.34107) 3554 : cluster [DBG] pgmap v5035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:11 smithi121 bash[36902]: audit 2022-04-22T19:09:11.788353+0000 mon.a (mon.0) 6462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:12 smithi121 bash[36902]: audit 2022-04-22T19:09:11.791523+0000 mon.a (mon.0) 6463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:12 smithi121 bash[36902]: audit 2022-04-22T19:09:11.793090+0000 mon.a (mon.0) 6464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:09:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:13 smithi121 bash[36902]: audit 2022-04-22T19:09:12.388515+0000 mon.a (mon.0) 6465 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:13 smithi121 bash[36902]: cluster 2022-04-22T19:09:12.929935+0000 mgr.y (mgr.34107) 3555 : cluster [DBG] pgmap v5036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:13 smithi115 bash[45495]: audit 2022-04-22T19:09:12.388515+0000 mon.a (mon.0) 6465 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:13 smithi115 bash[45495]: cluster 2022-04-22T19:09:12.929935+0000 mgr.y (mgr.34107) 3555 : cluster [DBG] pgmap v5036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:13 smithi115 bash[46934]: audit 2022-04-22T19:09:12.388515+0000 mon.a (mon.0) 6465 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:13 smithi115 bash[46934]: cluster 2022-04-22T19:09:12.929935+0000 mgr.y (mgr.34107) 3555 : cluster [DBG] pgmap v5036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:13.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:14 smithi121 bash[36902]: audit 2022-04-22T19:09:14.428013+0000 mon.a (mon.0) 6466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:09:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:14 smithi115 bash[46934]: audit 2022-04-22T19:09:14.428013+0000 mon.a (mon.0) 6466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:09:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:14 smithi115 bash[45495]: audit 2022-04-22T19:09:14.428013+0000 mon.a (mon.0) 6466 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:09:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:15 smithi121 bash[36902]: audit 2022-04-22T19:09:14.434040+0000 mon.a (mon.0) 6467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:09:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:15 smithi121 bash[36902]: audit 2022-04-22T19:09:14.637218+0000 mon.a (mon.0) 6468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:15 smithi121 bash[36902]: cluster 2022-04-22T19:09:14.931053+0000 mgr.y (mgr.34107) 3556 : cluster [DBG] pgmap v5037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[45495]: audit 2022-04-22T19:09:14.434040+0000 mon.a (mon.0) 6467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:09:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[45495]: audit 2022-04-22T19:09:14.637218+0000 mon.a (mon.0) 6468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[45495]: cluster 2022-04-22T19:09:14.931053+0000 mgr.y (mgr.34107) 3556 : cluster [DBG] pgmap v5037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[46934]: audit 2022-04-22T19:09:14.434040+0000 mon.a (mon.0) 6467 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:09:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[46934]: audit 2022-04-22T19:09:14.637218+0000 mon.a (mon.0) 6468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:15 smithi115 bash[46934]: cluster 2022-04-22T19:09:14.931053+0000 mgr.y (mgr.34107) 3556 : cluster [DBG] pgmap v5037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:17 smithi121 bash[36902]: audit 2022-04-22T19:09:16.224936+0000 mon.a (mon.0) 6469 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:17 smithi121 bash[36902]: audit 2022-04-22T19:09:16.573429+0000 mon.a (mon.0) 6470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:17 smithi115 bash[45495]: audit 2022-04-22T19:09:16.224936+0000 mon.a (mon.0) 6469 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:17 smithi115 bash[45495]: audit 2022-04-22T19:09:16.573429+0000 mon.a (mon.0) 6470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:17 smithi115 bash[46934]: audit 2022-04-22T19:09:16.224936+0000 mon.a (mon.0) 6469 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:17 smithi115 bash[46934]: audit 2022-04-22T19:09:16.573429+0000 mon.a (mon.0) 6470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:09:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:18 smithi121 bash[36902]: cluster 2022-04-22T19:09:16.932216+0000 mgr.y (mgr.34107) 3557 : cluster [DBG] pgmap v5038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:18 smithi115 bash[45495]: cluster 2022-04-22T19:09:16.932216+0000 mgr.y (mgr.34107) 3557 : cluster [DBG] pgmap v5038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:18 smithi115 bash[46934]: cluster 2022-04-22T19:09:16.932216+0000 mgr.y (mgr.34107) 3557 : cluster [DBG] pgmap v5038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:20 smithi121 bash[36902]: cluster 2022-04-22T19:09:18.933330+0000 mgr.y (mgr.34107) 3558 : cluster [DBG] pgmap v5039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:20 smithi121 bash[36902]: audit 2022-04-22T19:09:19.640409+0000 mon.a (mon.0) 6471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:20 smithi115 bash[45495]: cluster 2022-04-22T19:09:18.933330+0000 mgr.y (mgr.34107) 3558 : cluster [DBG] pgmap v5039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:20 smithi115 bash[45495]: audit 2022-04-22T19:09:19.640409+0000 mon.a (mon.0) 6471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:20 smithi115 bash[46934]: cluster 2022-04-22T19:09:18.933330+0000 mgr.y (mgr.34107) 3558 : cluster [DBG] pgmap v5039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:20 smithi115 bash[46934]: audit 2022-04-22T19:09:19.640409+0000 mon.a (mon.0) 6471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:21.866 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (92m) 9m ago 106m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (105m) 5s ago 105m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (94m) 9m ago 107m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (93m) 5s ago 107m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:09:22.207 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (101m) 5s ago 111m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (102m) 9m ago 113m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (101m) 9m ago 113m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (100m) 5s ago 111m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (100m) 9m ago 111m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (93m) 9m ago 106m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (93m) 5s ago 106m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (99m) 9m ago 111m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (98m) 9m ago 110m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (98m) 9m ago 110m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (97m) 9m ago 109m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:09:22.208 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (96m) 5s ago 109m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:09:22.209 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (96m) 5s ago 108m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:09:22.209 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (95m) 5s ago 108m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:09:22.209 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (94m) 5s ago 107m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:09:22.209 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (93m) 5s ago 106m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:09:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:22 smithi115 bash[45495]: cluster 2022-04-22T19:09:20.934452+0000 mgr.y (mgr.34107) 3559 : cluster [DBG] pgmap v5040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:22.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:22 smithi115 bash[46934]: cluster 2022-04-22T19:09:20.934452+0000 mgr.y (mgr.34107) 3559 : cluster [DBG] pgmap v5040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:22.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:22 smithi121 bash[36902]: cluster 2022-04-22T19:09:20.934452+0000 mgr.y (mgr.34107) 3559 : cluster [DBG] pgmap v5040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:22.612 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:09:22.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:09:22.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:09:22.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:22.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:22.613 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:09:22.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:09:22.614 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:09:22.614 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:09:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:23 smithi121 bash[36902]: audit 2022-04-22T19:09:21.840473+0000 mgr.y (mgr.34107) 3560 : audit [DBG] from='client.39132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:23 smithi121 bash[36902]: audit 2022-04-22T19:09:22.177579+0000 mgr.y (mgr.34107) 3561 : audit [DBG] from='client.39138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:23 smithi121 bash[36902]: audit 2022-04-22T19:09:22.601974+0000 mon.a (mon.0) 6472 : audit [DBG] from='client.? 172.21.15.115:0/324179780' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[45495]: audit 2022-04-22T19:09:21.840473+0000 mgr.y (mgr.34107) 3560 : audit [DBG] from='client.39132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[45495]: audit 2022-04-22T19:09:22.177579+0000 mgr.y (mgr.34107) 3561 : audit [DBG] from='client.39138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[45495]: audit 2022-04-22T19:09:22.601974+0000 mon.a (mon.0) 6472 : audit [DBG] from='client.? 172.21.15.115:0/324179780' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[46934]: audit 2022-04-22T19:09:21.840473+0000 mgr.y (mgr.34107) 3560 : audit [DBG] from='client.39132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[46934]: audit 2022-04-22T19:09:22.177579+0000 mgr.y (mgr.34107) 3561 : audit [DBG] from='client.39138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[46934]: audit 2022-04-22T19:09:22.601974+0000 mon.a (mon.0) 6472 : audit [DBG] from='client.? 172.21.15.115:0/324179780' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:24.249 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:23.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:24 smithi121 bash[36902]: cluster 2022-04-22T19:09:22.935411+0000 mgr.y (mgr.34107) 3562 : cluster [DBG] pgmap v5041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:24.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:24 smithi115 bash[45495]: cluster 2022-04-22T19:09:22.935411+0000 mgr.y (mgr.34107) 3562 : cluster [DBG] pgmap v5041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:24 smithi115 bash[46934]: cluster 2022-04-22T19:09:22.935411+0000 mgr.y (mgr.34107) 3562 : cluster [DBG] pgmap v5041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:25 smithi121 bash[36902]: audit 2022-04-22T19:09:24.643777+0000 mon.a (mon.0) 6473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:25 smithi115 bash[45495]: audit 2022-04-22T19:09:24.643777+0000 mon.a (mon.0) 6473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:25.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:25 smithi115 bash[46934]: audit 2022-04-22T19:09:24.643777+0000 mon.a (mon.0) 6473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:26 smithi121 bash[36902]: cluster 2022-04-22T19:09:24.936537+0000 mgr.y (mgr.34107) 3563 : cluster [DBG] pgmap v5042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:26 smithi115 bash[45495]: cluster 2022-04-22T19:09:24.936537+0000 mgr.y (mgr.34107) 3563 : cluster [DBG] pgmap v5042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:26 smithi115 bash[46934]: cluster 2022-04-22T19:09:24.936537+0000 mgr.y (mgr.34107) 3563 : cluster [DBG] pgmap v5042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:27 smithi115 bash[45495]: cluster 2022-04-22T19:09:26.937557+0000 mgr.y (mgr.34107) 3564 : cluster [DBG] pgmap v5043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:27 smithi115 bash[46934]: cluster 2022-04-22T19:09:26.937557+0000 mgr.y (mgr.34107) 3564 : cluster [DBG] pgmap v5043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:27 smithi121 bash[36902]: cluster 2022-04-22T19:09:26.937557+0000 mgr.y (mgr.34107) 3564 : cluster [DBG] pgmap v5043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:30 smithi121 bash[36902]: cluster 2022-04-22T19:09:28.938689+0000 mgr.y (mgr.34107) 3565 : cluster [DBG] pgmap v5044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:30 smithi121 bash[36902]: audit 2022-04-22T19:09:29.646803+0000 mon.a (mon.0) 6474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:29 smithi115 bash[45495]: cluster 2022-04-22T19:09:28.938689+0000 mgr.y (mgr.34107) 3565 : cluster [DBG] pgmap v5044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:29 smithi115 bash[45495]: audit 2022-04-22T19:09:29.646803+0000 mon.a (mon.0) 6474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:29 smithi115 bash[46934]: cluster 2022-04-22T19:09:28.938689+0000 mgr.y (mgr.34107) 3565 : cluster [DBG] pgmap v5044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:29 smithi115 bash[46934]: audit 2022-04-22T19:09:29.646803+0000 mon.a (mon.0) 6474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:32 smithi121 bash[36902]: cluster 2022-04-22T19:09:30.939858+0000 mgr.y (mgr.34107) 3566 : cluster [DBG] pgmap v5045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:31 smithi115 bash[45495]: cluster 2022-04-22T19:09:30.939858+0000 mgr.y (mgr.34107) 3566 : cluster [DBG] pgmap v5045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:32.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:32.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:32 smithi115 bash[46934]: cluster 2022-04-22T19:09:30.939858+0000 mgr.y (mgr.34107) 3566 : cluster [DBG] pgmap v5045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:33 smithi121 bash[36902]: cluster 2022-04-22T19:09:32.940941+0000 mgr.y (mgr.34107) 3567 : cluster [DBG] pgmap v5046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:33 smithi115 bash[45495]: cluster 2022-04-22T19:09:32.940941+0000 mgr.y (mgr.34107) 3567 : cluster [DBG] pgmap v5046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:33 smithi115 bash[46934]: cluster 2022-04-22T19:09:32.940941+0000 mgr.y (mgr.34107) 3567 : cluster [DBG] pgmap v5046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:33.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:34 smithi121 bash[36902]: audit 2022-04-22T19:09:34.650178+0000 mon.a (mon.0) 6475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:34 smithi115 bash[45495]: audit 2022-04-22T19:09:34.650178+0000 mon.a (mon.0) 6475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:34 smithi115 bash[46934]: audit 2022-04-22T19:09:34.650178+0000 mon.a (mon.0) 6475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:35 smithi121 bash[36902]: cluster 2022-04-22T19:09:34.941927+0000 mgr.y (mgr.34107) 3568 : cluster [DBG] pgmap v5047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:35 smithi115 bash[45495]: cluster 2022-04-22T19:09:34.941927+0000 mgr.y (mgr.34107) 3568 : cluster [DBG] pgmap v5047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:35 smithi115 bash[46934]: cluster 2022-04-22T19:09:34.941927+0000 mgr.y (mgr.34107) 3568 : cluster [DBG] pgmap v5047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:38 smithi121 bash[36902]: cluster 2022-04-22T19:09:36.942965+0000 mgr.y (mgr.34107) 3569 : cluster [DBG] pgmap v5048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:37 smithi115 bash[45495]: cluster 2022-04-22T19:09:36.942965+0000 mgr.y (mgr.34107) 3569 : cluster [DBG] pgmap v5048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:38 smithi115 bash[46934]: cluster 2022-04-22T19:09:36.942965+0000 mgr.y (mgr.34107) 3569 : cluster [DBG] pgmap v5048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:40 smithi121 bash[36902]: cluster 2022-04-22T19:09:38.944050+0000 mgr.y (mgr.34107) 3570 : cluster [DBG] pgmap v5049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:40 smithi121 bash[36902]: audit 2022-04-22T19:09:39.653474+0000 mon.a (mon.0) 6476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:40 smithi115 bash[45495]: cluster 2022-04-22T19:09:38.944050+0000 mgr.y (mgr.34107) 3570 : cluster [DBG] pgmap v5049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:40 smithi115 bash[45495]: audit 2022-04-22T19:09:39.653474+0000 mon.a (mon.0) 6476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:40.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:40 smithi115 bash[46934]: cluster 2022-04-22T19:09:38.944050+0000 mgr.y (mgr.34107) 3570 : cluster [DBG] pgmap v5049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:40.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:40 smithi115 bash[46934]: audit 2022-04-22T19:09:39.653474+0000 mon.a (mon.0) 6476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:42.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:42 smithi115 bash[45495]: cluster 2022-04-22T19:09:40.945134+0000 mgr.y (mgr.34107) 3571 : cluster [DBG] pgmap v5050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:42.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:42.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:42 smithi115 bash[46934]: cluster 2022-04-22T19:09:40.945134+0000 mgr.y (mgr.34107) 3571 : cluster [DBG] pgmap v5050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:42 smithi121 bash[36902]: cluster 2022-04-22T19:09:40.945134+0000 mgr.y (mgr.34107) 3571 : cluster [DBG] pgmap v5050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:43 smithi121 bash[36902]: cluster 2022-04-22T19:09:42.946150+0000 mgr.y (mgr.34107) 3572 : cluster [DBG] pgmap v5051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:43 smithi115 bash[45495]: cluster 2022-04-22T19:09:42.946150+0000 mgr.y (mgr.34107) 3572 : cluster [DBG] pgmap v5051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:43 smithi115 bash[46934]: cluster 2022-04-22T19:09:42.946150+0000 mgr.y (mgr.34107) 3572 : cluster [DBG] pgmap v5051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:43.880Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:44 smithi121 bash[36902]: audit 2022-04-22T19:09:44.656892+0000 mon.a (mon.0) 6477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:44 smithi115 bash[45495]: audit 2022-04-22T19:09:44.656892+0000 mon.a (mon.0) 6477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:44 smithi115 bash[46934]: audit 2022-04-22T19:09:44.656892+0000 mon.a (mon.0) 6477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:45 smithi121 bash[36902]: cluster 2022-04-22T19:09:44.947218+0000 mgr.y (mgr.34107) 3573 : cluster [DBG] pgmap v5052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:45 smithi115 bash[45495]: cluster 2022-04-22T19:09:44.947218+0000 mgr.y (mgr.34107) 3573 : cluster [DBG] pgmap v5052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:45 smithi115 bash[46934]: cluster 2022-04-22T19:09:44.947218+0000 mgr.y (mgr.34107) 3573 : cluster [DBG] pgmap v5052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:48 smithi115 bash[45495]: cluster 2022-04-22T19:09:46.948308+0000 mgr.y (mgr.34107) 3574 : cluster [DBG] pgmap v5053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:48 smithi115 bash[46934]: cluster 2022-04-22T19:09:46.948308+0000 mgr.y (mgr.34107) 3574 : cluster [DBG] pgmap v5053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:48 smithi121 bash[36902]: cluster 2022-04-22T19:09:46.948308+0000 mgr.y (mgr.34107) 3574 : cluster [DBG] pgmap v5053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:50 smithi115 bash[45495]: cluster 2022-04-22T19:09:48.949328+0000 mgr.y (mgr.34107) 3575 : cluster [DBG] pgmap v5054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:50.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:50 smithi115 bash[45495]: audit 2022-04-22T19:09:49.659744+0000 mon.a (mon.0) 6478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:50 smithi115 bash[46934]: cluster 2022-04-22T19:09:48.949328+0000 mgr.y (mgr.34107) 3575 : cluster [DBG] pgmap v5054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:50.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:50 smithi115 bash[46934]: audit 2022-04-22T19:09:49.659744+0000 mon.a (mon.0) 6478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:50 smithi121 bash[36902]: cluster 2022-04-22T19:09:48.949328+0000 mgr.y (mgr.34107) 3575 : cluster [DBG] pgmap v5054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:50.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:50 smithi121 bash[36902]: audit 2022-04-22T19:09:49.659744+0000 mon.a (mon.0) 6478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:52.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:52 smithi115 bash[46934]: cluster 2022-04-22T19:09:50.950434+0000 mgr.y (mgr.34107) 3576 : cluster [DBG] pgmap v5055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:52.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:52.007Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:52 smithi115 bash[45495]: cluster 2022-04-22T19:09:50.950434+0000 mgr.y (mgr.34107) 3576 : cluster [DBG] pgmap v5055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:52 smithi121 bash[36902]: cluster 2022-04-22T19:09:50.950434+0000 mgr.y (mgr.34107) 3576 : cluster [DBG] pgmap v5055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:52.987 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:09:53.328 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:09:53.328 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (92m) 9m ago 106m 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:09:53.329 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (106m) 37s ago 106m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:09:53.329 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (94m) 9m ago 107m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (94m) 37s ago 107m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (102m) 37s ago 112m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (102m) 9m ago 114m 477M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (101m) 9m ago 114m 255M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (100m) 37s ago 112m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (101m) 9m ago 112m 164M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (94m) 9m ago 106m 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (93m) 37s ago 106m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:09:53.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (99m) 9m ago 111m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (98m) 9m ago 111m 777M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (98m) 9m ago 110m 526M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (98m) 9m ago 110m 480M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (97m) 37s ago 109m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (96m) 37s ago 109m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (96m) 37s ago 108m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (95m) 37s ago 108m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:09:53.331 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (93m) 37s ago 107m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:09:53.734 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:53.735 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:09:53.736 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:09:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:53 smithi121 bash[36902]: cluster 2022-04-22T19:09:52.951358+0000 mgr.y (mgr.34107) 3577 : cluster [DBG] pgmap v5056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:53 smithi121 bash[36902]: audit 2022-04-22T19:09:52.961831+0000 mgr.y (mgr.34107) 3578 : audit [DBG] from='client.39150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:53 smithi115 bash[45495]: cluster 2022-04-22T19:09:52.951358+0000 mgr.y (mgr.34107) 3577 : cluster [DBG] pgmap v5056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:53 smithi115 bash[45495]: audit 2022-04-22T19:09:52.961831+0000 mgr.y (mgr.34107) 3578 : audit [DBG] from='client.39150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:53 smithi115 bash[46934]: cluster 2022-04-22T19:09:52.951358+0000 mgr.y (mgr.34107) 3577 : cluster [DBG] pgmap v5056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:53 smithi115 bash[46934]: audit 2022-04-22T19:09:52.961831+0000 mgr.y (mgr.34107) 3578 : audit [DBG] from='client.39150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:09:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:09:53.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:09:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:54 smithi121 bash[36902]: audit 2022-04-22T19:09:53.300119+0000 mgr.y (mgr.34107) 3579 : audit [DBG] from='client.39156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:54 smithi121 bash[36902]: audit 2022-04-22T19:09:53.724330+0000 mon.c (mon.1) 180 : audit [DBG] from='client.? 172.21.15.115:0/1079974272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:54 smithi115 bash[45495]: audit 2022-04-22T19:09:53.300119+0000 mgr.y (mgr.34107) 3579 : audit [DBG] from='client.39156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:54 smithi115 bash[45495]: audit 2022-04-22T19:09:53.724330+0000 mon.c (mon.1) 180 : audit [DBG] from='client.? 172.21.15.115:0/1079974272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:54.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:54 smithi115 bash[46934]: audit 2022-04-22T19:09:53.300119+0000 mgr.y (mgr.34107) 3579 : audit [DBG] from='client.39156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:09:54.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:54 smithi115 bash[46934]: audit 2022-04-22T19:09:53.724330+0000 mon.c (mon.1) 180 : audit [DBG] from='client.? 172.21.15.115:0/1079974272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:09:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:55 smithi121 bash[36902]: audit 2022-04-22T19:09:54.662816+0000 mon.a (mon.0) 6479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:55 smithi121 bash[36902]: cluster 2022-04-22T19:09:54.952379+0000 mgr.y (mgr.34107) 3580 : cluster [DBG] pgmap v5057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:55 smithi115 bash[45495]: audit 2022-04-22T19:09:54.662816+0000 mon.a (mon.0) 6479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:55 smithi115 bash[45495]: cluster 2022-04-22T19:09:54.952379+0000 mgr.y (mgr.34107) 3580 : cluster [DBG] pgmap v5057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:55 smithi115 bash[46934]: audit 2022-04-22T19:09:54.662816+0000 mon.a (mon.0) 6479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:09:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:55 smithi115 bash[46934]: cluster 2022-04-22T19:09:54.952379+0000 mgr.y (mgr.34107) 3580 : cluster [DBG] pgmap v5057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:09:58 smithi115 bash[45495]: cluster 2022-04-22T19:09:56.953418+0000 mgr.y (mgr.34107) 3581 : cluster [DBG] pgmap v5058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:58.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:09:58 smithi115 bash[46934]: cluster 2022-04-22T19:09:56.953418+0000 mgr.y (mgr.34107) 3581 : cluster [DBG] pgmap v5058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:09:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:09:58 smithi121 bash[36902]: cluster 2022-04-22T19:09:56.953418+0000 mgr.y (mgr.34107) 3581 : cluster [DBG] pgmap v5058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[45495]: cluster 2022-04-22T19:09:58.954485+0000 mgr.y (mgr.34107) 3582 : cluster [DBG] pgmap v5059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[45495]: audit 2022-04-22T19:09:59.665744+0000 mon.a (mon.0) 6480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[45495]: cluster 2022-04-22T19:10:00.000173+0000 mon.a (mon.0) 6481 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:10:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[46934]: cluster 2022-04-22T19:09:58.954485+0000 mgr.y (mgr.34107) 3582 : cluster [DBG] pgmap v5059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[46934]: audit 2022-04-22T19:09:59.665744+0000 mon.a (mon.0) 6480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:00 smithi115 bash[46934]: cluster 2022-04-22T19:10:00.000173+0000 mon.a (mon.0) 6481 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:10:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:00 smithi121 bash[36902]: cluster 2022-04-22T19:09:58.954485+0000 mgr.y (mgr.34107) 3582 : cluster [DBG] pgmap v5059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:00 smithi121 bash[36902]: audit 2022-04-22T19:09:59.665744+0000 mon.a (mon.0) 6480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:00 smithi121 bash[36902]: cluster 2022-04-22T19:10:00.000173+0000 mon.a (mon.0) 6481 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:10:02.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:02.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:02 smithi115 bash[45495]: cluster 2022-04-22T19:10:00.955726+0000 mgr.y (mgr.34107) 3583 : cluster [DBG] pgmap v5060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:02 smithi115 bash[46934]: cluster 2022-04-22T19:10:00.955726+0000 mgr.y (mgr.34107) 3583 : cluster [DBG] pgmap v5060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:02 smithi121 bash[36902]: cluster 2022-04-22T19:10:00.955726+0000 mgr.y (mgr.34107) 3583 : cluster [DBG] pgmap v5060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:03 smithi121 bash[36902]: cluster 2022-04-22T19:10:02.956630+0000 mgr.y (mgr.34107) 3584 : cluster [DBG] pgmap v5061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:03 smithi115 bash[46934]: cluster 2022-04-22T19:10:02.956630+0000 mgr.y (mgr.34107) 3584 : cluster [DBG] pgmap v5061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:03.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:03 smithi115 bash[45495]: cluster 2022-04-22T19:10:02.956630+0000 mgr.y (mgr.34107) 3584 : cluster [DBG] pgmap v5061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:03.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:04 smithi121 bash[36902]: audit 2022-04-22T19:10:04.668579+0000 mon.a (mon.0) 6482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:05.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:04 smithi115 bash[45495]: audit 2022-04-22T19:10:04.668579+0000 mon.a (mon.0) 6482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:04 smithi115 bash[46934]: audit 2022-04-22T19:10:04.668579+0000 mon.a (mon.0) 6482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:05 smithi121 bash[36902]: cluster 2022-04-22T19:10:04.957676+0000 mgr.y (mgr.34107) 3585 : cluster [DBG] pgmap v5062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:05 smithi115 bash[46934]: cluster 2022-04-22T19:10:04.957676+0000 mgr.y (mgr.34107) 3585 : cluster [DBG] pgmap v5062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:05 smithi115 bash[45495]: cluster 2022-04-22T19:10:04.957676+0000 mgr.y (mgr.34107) 3585 : cluster [DBG] pgmap v5062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:08 smithi115 bash[46934]: cluster 2022-04-22T19:10:06.958718+0000 mgr.y (mgr.34107) 3586 : cluster [DBG] pgmap v5063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:08 smithi115 bash[45495]: cluster 2022-04-22T19:10:06.958718+0000 mgr.y (mgr.34107) 3586 : cluster [DBG] pgmap v5063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:08 smithi121 bash[36902]: cluster 2022-04-22T19:10:06.958718+0000 mgr.y (mgr.34107) 3586 : cluster [DBG] pgmap v5063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:10 smithi115 bash[46934]: cluster 2022-04-22T19:10:08.959822+0000 mgr.y (mgr.34107) 3587 : cluster [DBG] pgmap v5064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:10.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:10 smithi115 bash[46934]: audit 2022-04-22T19:10:09.671835+0000 mon.a (mon.0) 6483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:10.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:10 smithi115 bash[45495]: cluster 2022-04-22T19:10:08.959822+0000 mgr.y (mgr.34107) 3587 : cluster [DBG] pgmap v5064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:10.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:10 smithi115 bash[45495]: audit 2022-04-22T19:10:09.671835+0000 mon.a (mon.0) 6483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:10 smithi121 bash[36902]: cluster 2022-04-22T19:10:08.959822+0000 mgr.y (mgr.34107) 3587 : cluster [DBG] pgmap v5064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:10 smithi121 bash[36902]: audit 2022-04-22T19:10:09.671835+0000 mon.a (mon.0) 6483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:12.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:12 smithi115 bash[45495]: cluster 2022-04-22T19:10:10.960921+0000 mgr.y (mgr.34107) 3588 : cluster [DBG] pgmap v5065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:12.010Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:12 smithi115 bash[46934]: cluster 2022-04-22T19:10:10.960921+0000 mgr.y (mgr.34107) 3588 : cluster [DBG] pgmap v5065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:12 smithi121 bash[36902]: cluster 2022-04-22T19:10:10.960921+0000 mgr.y (mgr.34107) 3588 : cluster [DBG] pgmap v5065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:13 smithi121 bash[36902]: cluster 2022-04-22T19:10:12.961924+0000 mgr.y (mgr.34107) 3589 : cluster [DBG] pgmap v5066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:13 smithi115 bash[45495]: cluster 2022-04-22T19:10:12.961924+0000 mgr.y (mgr.34107) 3589 : cluster [DBG] pgmap v5066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:13 smithi115 bash[46934]: cluster 2022-04-22T19:10:12.961924+0000 mgr.y (mgr.34107) 3589 : cluster [DBG] pgmap v5066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:13.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:14 smithi121 bash[36902]: audit 2022-04-22T19:10:14.448752+0000 mon.a (mon.0) 6484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:10:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:14 smithi121 bash[36902]: audit 2022-04-22T19:10:14.449288+0000 mon.a (mon.0) 6485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:10:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:14 smithi115 bash[45495]: audit 2022-04-22T19:10:14.448752+0000 mon.a (mon.0) 6484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:10:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:14 smithi115 bash[45495]: audit 2022-04-22T19:10:14.449288+0000 mon.a (mon.0) 6485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:10:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:14 smithi115 bash[46934]: audit 2022-04-22T19:10:14.448752+0000 mon.a (mon.0) 6484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:10:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:14 smithi115 bash[46934]: audit 2022-04-22T19:10:14.449288+0000 mon.a (mon.0) 6485 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:10:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:15 smithi121 bash[36902]: audit 2022-04-22T19:10:14.675005+0000 mon.a (mon.0) 6486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:15 smithi121 bash[36902]: cluster 2022-04-22T19:10:14.963019+0000 mgr.y (mgr.34107) 3590 : cluster [DBG] pgmap v5067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:15 smithi115 bash[45495]: audit 2022-04-22T19:10:14.675005+0000 mon.a (mon.0) 6486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:15 smithi115 bash[45495]: cluster 2022-04-22T19:10:14.963019+0000 mgr.y (mgr.34107) 3590 : cluster [DBG] pgmap v5067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:15 smithi115 bash[46934]: audit 2022-04-22T19:10:14.675005+0000 mon.a (mon.0) 6486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:15 smithi115 bash[46934]: cluster 2022-04-22T19:10:14.963019+0000 mgr.y (mgr.34107) 3590 : cluster [DBG] pgmap v5067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:16.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[46934]: audit 2022-04-22T19:10:16.581814+0000 mon.a (mon.0) 6487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:10:16.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[46934]: audit 2022-04-22T19:10:16.584612+0000 mon.a (mon.0) 6488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:10:16.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[46934]: audit 2022-04-22T19:10:16.586221+0000 mon.a (mon.0) 6489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:10:16.897 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[45495]: audit 2022-04-22T19:10:16.581814+0000 mon.a (mon.0) 6487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:10:16.897 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[45495]: audit 2022-04-22T19:10:16.584612+0000 mon.a (mon.0) 6488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:10:16.897 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:16 smithi115 bash[45495]: audit 2022-04-22T19:10:16.586221+0000 mon.a (mon.0) 6489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:10:16.910 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:16 smithi121 bash[36902]: audit 2022-04-22T19:10:16.581814+0000 mon.a (mon.0) 6487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:10:16.910 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:16 smithi121 bash[36902]: audit 2022-04-22T19:10:16.584612+0000 mon.a (mon.0) 6488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:10:16.910 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:16 smithi121 bash[36902]: audit 2022-04-22T19:10:16.586221+0000 mon.a (mon.0) 6489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:10:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:17 smithi121 bash[36902]: audit 2022-04-22T19:10:16.902571+0000 mon.a (mon.0) 6490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:17 smithi121 bash[36902]: audit 2022-04-22T19:10:16.916869+0000 mon.a (mon.0) 6491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:17 smithi121 bash[36902]: cluster 2022-04-22T19:10:16.963945+0000 mgr.y (mgr.34107) 3591 : cluster [DBG] pgmap v5068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[46934]: audit 2022-04-22T19:10:16.902571+0000 mon.a (mon.0) 6490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[46934]: audit 2022-04-22T19:10:16.916869+0000 mon.a (mon.0) 6491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[46934]: cluster 2022-04-22T19:10:16.963945+0000 mgr.y (mgr.34107) 3591 : cluster [DBG] pgmap v5068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[45495]: audit 2022-04-22T19:10:16.902571+0000 mon.a (mon.0) 6490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[45495]: audit 2022-04-22T19:10:16.916869+0000 mon.a (mon.0) 6491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:18.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:17 smithi115 bash[45495]: cluster 2022-04-22T19:10:16.963945+0000 mgr.y (mgr.34107) 3591 : cluster [DBG] pgmap v5068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:20 smithi115 bash[45495]: cluster 2022-04-22T19:10:18.965066+0000 mgr.y (mgr.34107) 3592 : cluster [DBG] pgmap v5069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:20 smithi115 bash[45495]: audit 2022-04-22T19:10:19.678131+0000 mon.a (mon.0) 6492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:20 smithi115 bash[46934]: cluster 2022-04-22T19:10:18.965066+0000 mgr.y (mgr.34107) 3592 : cluster [DBG] pgmap v5069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:20 smithi115 bash[46934]: audit 2022-04-22T19:10:19.678131+0000 mon.a (mon.0) 6492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:20 smithi121 bash[36902]: cluster 2022-04-22T19:10:18.965066+0000 mgr.y (mgr.34107) 3592 : cluster [DBG] pgmap v5069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:20 smithi121 bash[36902]: audit 2022-04-22T19:10:19.678131+0000 mon.a (mon.0) 6492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:22 smithi115 bash[45495]: cluster 2022-04-22T19:10:20.966017+0000 mgr.y (mgr.34107) 3593 : cluster [DBG] pgmap v5070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:22 smithi115 bash[45495]: audit 2022-04-22T19:10:21.102314+0000 mon.a (mon.0) 6493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:22.009Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:22 smithi115 bash[46934]: cluster 2022-04-22T19:10:20.966017+0000 mgr.y (mgr.34107) 3593 : cluster [DBG] pgmap v5070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:22 smithi115 bash[46934]: audit 2022-04-22T19:10:21.102314+0000 mon.a (mon.0) 6493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:22 smithi121 bash[36902]: cluster 2022-04-22T19:10:20.966017+0000 mgr.y (mgr.34107) 3593 : cluster [DBG] pgmap v5070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:22.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:22 smithi121 bash[36902]: audit 2022-04-22T19:10:21.102314+0000 mon.a (mon.0) 6493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:23 smithi121 bash[36902]: cluster 2022-04-22T19:10:22.967152+0000 mgr.y (mgr.34107) 3594 : cluster [DBG] pgmap v5071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:23 smithi115 bash[45495]: cluster 2022-04-22T19:10:22.967152+0000 mgr.y (mgr.34107) 3594 : cluster [DBG] pgmap v5071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:23 smithi115 bash[46934]: cluster 2022-04-22T19:10:22.967152+0000 mgr.y (mgr.34107) 3594 : cluster [DBG] pgmap v5071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:24.115 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:10:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:23.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:10:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (93m) 3s ago 107m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:10:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (106m) 68s ago 106m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:10:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (95m) 3s ago 108m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:10:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (94m) 68s ago 108m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (102m) 68s ago 112m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (103m) 3s ago 114m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (102m) 3s ago 114m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (101m) 68s ago 112m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (101m) 3s ago 112m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (94m) 3s ago 107m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (94m) 68s ago 107m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (100m) 3s ago 112m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:10:24.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (99m) 3s ago 111m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:10:24.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (99m) 3s ago 111m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:10:24.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (98m) 3s ago 110m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:10:24.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (97m) 68s ago 110m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:10:24.467 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (97m) 68s ago 109m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:10:24.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (96m) 68s ago 109m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:10:24.468 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (95m) 68s ago 108m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:10:24.468 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (94m) 68s ago 107m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:10:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:24 smithi121 bash[36902]: audit 2022-04-22T19:10:24.088345+0000 mgr.y (mgr.34107) 3595 : audit [DBG] from='client.39168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:24.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:24 smithi115 bash[45495]: audit 2022-04-22T19:10:24.088345+0000 mgr.y (mgr.34107) 3595 : audit [DBG] from='client.39168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:24.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:24 smithi115 bash[46934]: audit 2022-04-22T19:10:24.088345+0000 mgr.y (mgr.34107) 3595 : audit [DBG] from='client.39168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:24.855 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:10:24.855 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:10:24.857 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:10:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:25 smithi121 bash[36902]: audit 2022-04-22T19:10:24.436974+0000 mgr.y (mgr.34107) 3596 : audit [DBG] from='client.39174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:25 smithi121 bash[36902]: audit 2022-04-22T19:10:24.680948+0000 mon.a (mon.0) 6494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:25 smithi121 bash[36902]: audit 2022-04-22T19:10:24.845726+0000 mon.c (mon.1) 181 : audit [DBG] from='client.? 172.21.15.115:0/2995591701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:25 smithi121 bash[36902]: cluster 2022-04-22T19:10:24.968344+0000 mgr.y (mgr.34107) 3597 : cluster [DBG] pgmap v5072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:25.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[45495]: audit 2022-04-22T19:10:24.436974+0000 mgr.y (mgr.34107) 3596 : audit [DBG] from='client.39174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:25.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[45495]: audit 2022-04-22T19:10:24.680948+0000 mon.a (mon.0) 6494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:25.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[45495]: audit 2022-04-22T19:10:24.845726+0000 mon.c (mon.1) 181 : audit [DBG] from='client.? 172.21.15.115:0/2995591701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:25.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[45495]: cluster 2022-04-22T19:10:24.968344+0000 mgr.y (mgr.34107) 3597 : cluster [DBG] pgmap v5072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:25.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[46934]: audit 2022-04-22T19:10:24.436974+0000 mgr.y (mgr.34107) 3596 : audit [DBG] from='client.39174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:25.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[46934]: audit 2022-04-22T19:10:24.680948+0000 mon.a (mon.0) 6494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:25.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[46934]: audit 2022-04-22T19:10:24.845726+0000 mon.c (mon.1) 181 : audit [DBG] from='client.? 172.21.15.115:0/2995591701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:25.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:25 smithi115 bash[46934]: cluster 2022-04-22T19:10:24.968344+0000 mgr.y (mgr.34107) 3597 : cluster [DBG] pgmap v5072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:28 smithi115 bash[45495]: cluster 2022-04-22T19:10:26.969414+0000 mgr.y (mgr.34107) 3598 : cluster [DBG] pgmap v5073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:28 smithi115 bash[46934]: cluster 2022-04-22T19:10:26.969414+0000 mgr.y (mgr.34107) 3598 : cluster [DBG] pgmap v5073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:28 smithi121 bash[36902]: cluster 2022-04-22T19:10:26.969414+0000 mgr.y (mgr.34107) 3598 : cluster [DBG] pgmap v5073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:30.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[45495]: cluster 2022-04-22T19:10:28.970233+0000 mgr.y (mgr.34107) 3599 : cluster [DBG] pgmap v5074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[45495]: audit 2022-04-22T19:10:29.683379+0000 mon.a (mon.0) 6495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[45495]: audit 2022-04-22T19:10:29.894794+0000 mon.a (mon.0) 6496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[45495]: audit 2022-04-22T19:10:29.903888+0000 mon.a (mon.0) 6497 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[46934]: cluster 2022-04-22T19:10:28.970233+0000 mgr.y (mgr.34107) 3599 : cluster [DBG] pgmap v5074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[46934]: audit 2022-04-22T19:10:29.683379+0000 mon.a (mon.0) 6495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:30.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[46934]: audit 2022-04-22T19:10:29.894794+0000 mon.a (mon.0) 6496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:30.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:30 smithi115 bash[46934]: audit 2022-04-22T19:10:29.903888+0000 mon.a (mon.0) 6497 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:30 smithi121 bash[36902]: cluster 2022-04-22T19:10:28.970233+0000 mgr.y (mgr.34107) 3599 : cluster [DBG] pgmap v5074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:30 smithi121 bash[36902]: audit 2022-04-22T19:10:29.683379+0000 mon.a (mon.0) 6495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:30 smithi121 bash[36902]: audit 2022-04-22T19:10:29.894794+0000 mon.a (mon.0) 6496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:30 smithi121 bash[36902]: audit 2022-04-22T19:10:29.903888+0000 mon.a (mon.0) 6497 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:10:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:31 smithi121 bash[36902]: cephadm 2022-04-22T19:10:29.884149+0000 mgr.y (mgr.34107) 3600 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:10:31.573 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:31 smithi115 bash[45495]: cephadm 2022-04-22T19:10:29.884149+0000 mgr.y (mgr.34107) 3600 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:10:31.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:31 smithi115 bash[46934]: cephadm 2022-04-22T19:10:29.884149+0000 mgr.y (mgr.34107) 3600 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:10:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:32 smithi115 bash[45495]: cluster 2022-04-22T19:10:30.971209+0000 mgr.y (mgr.34107) 3601 : cluster [DBG] pgmap v5075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:32.011Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:32.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:32 smithi115 bash[46934]: cluster 2022-04-22T19:10:30.971209+0000 mgr.y (mgr.34107) 3601 : cluster [DBG] pgmap v5075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:32 smithi121 bash[36902]: cluster 2022-04-22T19:10:30.971209+0000 mgr.y (mgr.34107) 3601 : cluster [DBG] pgmap v5075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:33 smithi121 bash[36902]: cluster 2022-04-22T19:10:32.972264+0000 mgr.y (mgr.34107) 3602 : cluster [DBG] pgmap v5076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:33 smithi115 bash[45495]: cluster 2022-04-22T19:10:32.972264+0000 mgr.y (mgr.34107) 3602 : cluster [DBG] pgmap v5076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:33 smithi115 bash[46934]: cluster 2022-04-22T19:10:32.972264+0000 mgr.y (mgr.34107) 3602 : cluster [DBG] pgmap v5076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:33.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:34 smithi121 bash[36902]: audit 2022-04-22T19:10:34.686481+0000 mon.a (mon.0) 6498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:34 smithi115 bash[45495]: audit 2022-04-22T19:10:34.686481+0000 mon.a (mon.0) 6498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:34 smithi115 bash[46934]: audit 2022-04-22T19:10:34.686481+0000 mon.a (mon.0) 6498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:35 smithi121 bash[36902]: cluster 2022-04-22T19:10:34.973088+0000 mgr.y (mgr.34107) 3603 : cluster [DBG] pgmap v5077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:35 smithi115 bash[45495]: cluster 2022-04-22T19:10:34.973088+0000 mgr.y (mgr.34107) 3603 : cluster [DBG] pgmap v5077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:35 smithi115 bash[46934]: cluster 2022-04-22T19:10:34.973088+0000 mgr.y (mgr.34107) 3603 : cluster [DBG] pgmap v5077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:38 smithi115 bash[45495]: cluster 2022-04-22T19:10:36.974117+0000 mgr.y (mgr.34107) 3604 : cluster [DBG] pgmap v5078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:38.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:38 smithi115 bash[46934]: cluster 2022-04-22T19:10:36.974117+0000 mgr.y (mgr.34107) 3604 : cluster [DBG] pgmap v5078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:38 smithi121 bash[36902]: cluster 2022-04-22T19:10:36.974117+0000 mgr.y (mgr.34107) 3604 : cluster [DBG] pgmap v5078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:40 smithi115 bash[45495]: cluster 2022-04-22T19:10:38.975220+0000 mgr.y (mgr.34107) 3605 : cluster [DBG] pgmap v5079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:40.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:40 smithi115 bash[45495]: audit 2022-04-22T19:10:39.689702+0000 mon.a (mon.0) 6499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:40.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:40 smithi115 bash[46934]: cluster 2022-04-22T19:10:38.975220+0000 mgr.y (mgr.34107) 3605 : cluster [DBG] pgmap v5079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:40.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:40 smithi115 bash[46934]: audit 2022-04-22T19:10:39.689702+0000 mon.a (mon.0) 6499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:40 smithi121 bash[36902]: cluster 2022-04-22T19:10:38.975220+0000 mgr.y (mgr.34107) 3605 : cluster [DBG] pgmap v5079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:40 smithi121 bash[36902]: audit 2022-04-22T19:10:39.689702+0000 mon.a (mon.0) 6499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:42.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:42 smithi115 bash[45495]: cluster 2022-04-22T19:10:40.976338+0000 mgr.y (mgr.34107) 3606 : cluster [DBG] pgmap v5080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:42.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:42 smithi115 bash[46934]: cluster 2022-04-22T19:10:40.976338+0000 mgr.y (mgr.34107) 3606 : cluster [DBG] pgmap v5080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:42.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:42.012Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:42 smithi121 bash[36902]: cluster 2022-04-22T19:10:40.976338+0000 mgr.y (mgr.34107) 3606 : cluster [DBG] pgmap v5080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:43 smithi121 bash[36902]: cluster 2022-04-22T19:10:42.977361+0000 mgr.y (mgr.34107) 3607 : cluster [DBG] pgmap v5081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:43 smithi115 bash[45495]: cluster 2022-04-22T19:10:42.977361+0000 mgr.y (mgr.34107) 3607 : cluster [DBG] pgmap v5081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:43 smithi115 bash[46934]: cluster 2022-04-22T19:10:42.977361+0000 mgr.y (mgr.34107) 3607 : cluster [DBG] pgmap v5081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:43.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:44 smithi121 bash[36902]: audit 2022-04-22T19:10:44.692868+0000 mon.a (mon.0) 6500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:44 smithi115 bash[45495]: audit 2022-04-22T19:10:44.692868+0000 mon.a (mon.0) 6500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:44 smithi115 bash[46934]: audit 2022-04-22T19:10:44.692868+0000 mon.a (mon.0) 6500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:45 smithi115 bash[45495]: cluster 2022-04-22T19:10:44.978597+0000 mgr.y (mgr.34107) 3608 : cluster [DBG] pgmap v5082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:45 smithi115 bash[46934]: cluster 2022-04-22T19:10:44.978597+0000 mgr.y (mgr.34107) 3608 : cluster [DBG] pgmap v5082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:45 smithi121 bash[36902]: cluster 2022-04-22T19:10:44.978597+0000 mgr.y (mgr.34107) 3608 : cluster [DBG] pgmap v5082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:48 smithi115 bash[45495]: cluster 2022-04-22T19:10:46.979777+0000 mgr.y (mgr.34107) 3609 : cluster [DBG] pgmap v5083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:48 smithi115 bash[46934]: cluster 2022-04-22T19:10:46.979777+0000 mgr.y (mgr.34107) 3609 : cluster [DBG] pgmap v5083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:48 smithi121 bash[36902]: cluster 2022-04-22T19:10:46.979777+0000 mgr.y (mgr.34107) 3609 : cluster [DBG] pgmap v5083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:50 smithi121 bash[36902]: cluster 2022-04-22T19:10:48.980869+0000 mgr.y (mgr.34107) 3610 : cluster [DBG] pgmap v5084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:50.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:50 smithi121 bash[36902]: audit 2022-04-22T19:10:49.696092+0000 mon.a (mon.0) 6501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:50.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:50 smithi115 bash[45495]: cluster 2022-04-22T19:10:48.980869+0000 mgr.y (mgr.34107) 3610 : cluster [DBG] pgmap v5084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:50 smithi115 bash[45495]: audit 2022-04-22T19:10:49.696092+0000 mon.a (mon.0) 6501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:50 smithi115 bash[46934]: cluster 2022-04-22T19:10:48.980869+0000 mgr.y (mgr.34107) 3610 : cluster [DBG] pgmap v5084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:50 smithi115 bash[46934]: audit 2022-04-22T19:10:49.696092+0000 mon.a (mon.0) 6501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:52 smithi115 bash[45495]: cluster 2022-04-22T19:10:50.981956+0000 mgr.y (mgr.34107) 3611 : cluster [DBG] pgmap v5085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:52.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:52.013Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:52 smithi115 bash[46934]: cluster 2022-04-22T19:10:50.981956+0000 mgr.y (mgr.34107) 3611 : cluster [DBG] pgmap v5085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:52 smithi121 bash[36902]: cluster 2022-04-22T19:10:50.981956+0000 mgr.y (mgr.34107) 3611 : cluster [DBG] pgmap v5085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:53 smithi121 bash[36902]: cluster 2022-04-22T19:10:52.982963+0000 mgr.y (mgr.34107) 3612 : cluster [DBG] pgmap v5086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:53 smithi115 bash[46934]: cluster 2022-04-22T19:10:52.982963+0000 mgr.y (mgr.34107) 3612 : cluster [DBG] pgmap v5086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:53 smithi115 bash[45495]: cluster 2022-04-22T19:10:52.982963+0000 mgr.y (mgr.34107) 3612 : cluster [DBG] pgmap v5086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:10:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:10:53.885Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:10:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:54 smithi121 bash[36902]: audit 2022-04-22T19:10:54.699089+0000 mon.a (mon.0) 6502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:54 smithi115 bash[46934]: audit 2022-04-22T19:10:54.699089+0000 mon.a (mon.0) 6502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:55.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:54 smithi115 bash[45495]: audit 2022-04-22T19:10:54.699089+0000 mon.a (mon.0) 6502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:10:55.226 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:10:55.574 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:10:55.574 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (93m) 34s ago 107m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:10:55.574 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (107m) 99s ago 107m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (95m) 34s ago 108m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (95m) 99s ago 108m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (103m) 99s ago 113m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (103m) 34s ago 115m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (102m) 34s ago 115m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (101m) 99s ago 113m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (102m) 34s ago 113m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (95m) 34s ago 108m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (95m) 99s ago 108m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:10:55.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (100m) 34s ago 112m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (100m) 34s ago 112m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (99m) 34s ago 111m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (99m) 34s ago 111m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (98m) 99s ago 110m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (97m) 99s ago 110m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (97m) 99s ago 109m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (96m) 99s ago 109m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:10:55.576 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (94m) 99s ago 108m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:10:55.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:10:55.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:10:55.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:10:55.981 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:10:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:55 smithi115 bash[45495]: cluster 2022-04-22T19:10:54.983613+0000 mgr.y (mgr.34107) 3613 : cluster [DBG] pgmap v5087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:56.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:55 smithi115 bash[45495]: audit 2022-04-22T19:10:55.197638+0000 mgr.y (mgr.34107) 3614 : audit [DBG] from='client.49120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:55 smithi115 bash[46934]: cluster 2022-04-22T19:10:54.983613+0000 mgr.y (mgr.34107) 3613 : cluster [DBG] pgmap v5087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:55 smithi115 bash[46934]: audit 2022-04-22T19:10:55.197638+0000 mgr.y (mgr.34107) 3614 : audit [DBG] from='client.49120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:55 smithi121 bash[36902]: cluster 2022-04-22T19:10:54.983613+0000 mgr.y (mgr.34107) 3613 : cluster [DBG] pgmap v5087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:55 smithi121 bash[36902]: audit 2022-04-22T19:10:55.197638+0000 mgr.y (mgr.34107) 3614 : audit [DBG] from='client.49120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:57.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:56 smithi115 bash[45495]: audit 2022-04-22T19:10:55.544548+0000 mgr.y (mgr.34107) 3615 : audit [DBG] from='client.49126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:57.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:56 smithi115 bash[45495]: audit 2022-04-22T19:10:55.968735+0000 mon.a (mon.0) 6503 : audit [DBG] from='client.? 172.21.15.115:0/1544906481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:57.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:56 smithi115 bash[46934]: audit 2022-04-22T19:10:55.544548+0000 mgr.y (mgr.34107) 3615 : audit [DBG] from='client.49126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:57.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:56 smithi115 bash[46934]: audit 2022-04-22T19:10:55.968735+0000 mon.a (mon.0) 6503 : audit [DBG] from='client.? 172.21.15.115:0/1544906481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:56 smithi121 bash[36902]: audit 2022-04-22T19:10:55.544548+0000 mgr.y (mgr.34107) 3615 : audit [DBG] from='client.49126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:10:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:56 smithi121 bash[36902]: audit 2022-04-22T19:10:55.968735+0000 mon.a (mon.0) 6503 : audit [DBG] from='client.? 172.21.15.115:0/1544906481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:10:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:10:57 smithi115 bash[45495]: cluster 2022-04-22T19:10:56.984555+0000 mgr.y (mgr.34107) 3616 : cluster [DBG] pgmap v5088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:10:57 smithi115 bash[46934]: cluster 2022-04-22T19:10:56.984555+0000 mgr.y (mgr.34107) 3616 : cluster [DBG] pgmap v5088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:10:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:10:57 smithi121 bash[36902]: cluster 2022-04-22T19:10:56.984555+0000 mgr.y (mgr.34107) 3616 : cluster [DBG] pgmap v5088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:00 smithi121 bash[36902]: cluster 2022-04-22T19:10:58.985618+0000 mgr.y (mgr.34107) 3617 : cluster [DBG] pgmap v5089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:00 smithi121 bash[36902]: audit 2022-04-22T19:10:59.702365+0000 mon.a (mon.0) 6504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:00 smithi115 bash[45495]: cluster 2022-04-22T19:10:58.985618+0000 mgr.y (mgr.34107) 3617 : cluster [DBG] pgmap v5089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:00 smithi115 bash[45495]: audit 2022-04-22T19:10:59.702365+0000 mon.a (mon.0) 6504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:00 smithi115 bash[46934]: cluster 2022-04-22T19:10:58.985618+0000 mgr.y (mgr.34107) 3617 : cluster [DBG] pgmap v5089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:00 smithi115 bash[46934]: audit 2022-04-22T19:10:59.702365+0000 mon.a (mon.0) 6504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:02 smithi115 bash[45495]: cluster 2022-04-22T19:11:00.986790+0000 mgr.y (mgr.34107) 3618 : cluster [DBG] pgmap v5090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:02.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:02.013Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:02 smithi115 bash[46934]: cluster 2022-04-22T19:11:00.986790+0000 mgr.y (mgr.34107) 3618 : cluster [DBG] pgmap v5090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:02 smithi121 bash[36902]: cluster 2022-04-22T19:11:00.986790+0000 mgr.y (mgr.34107) 3618 : cluster [DBG] pgmap v5090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:03 smithi121 bash[36902]: cluster 2022-04-22T19:11:02.987781+0000 mgr.y (mgr.34107) 3619 : cluster [DBG] pgmap v5091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:03 smithi115 bash[45495]: cluster 2022-04-22T19:11:02.987781+0000 mgr.y (mgr.34107) 3619 : cluster [DBG] pgmap v5091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:03 smithi115 bash[46934]: cluster 2022-04-22T19:11:02.987781+0000 mgr.y (mgr.34107) 3619 : cluster [DBG] pgmap v5091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:03.885Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:05.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:04 smithi115 bash[45495]: audit 2022-04-22T19:11:04.705644+0000 mon.a (mon.0) 6505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:05.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:04 smithi115 bash[46934]: audit 2022-04-22T19:11:04.705644+0000 mon.a (mon.0) 6505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:04 smithi121 bash[36902]: audit 2022-04-22T19:11:04.705644+0000 mon.a (mon.0) 6505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:05 smithi115 bash[45495]: cluster 2022-04-22T19:11:04.988915+0000 mgr.y (mgr.34107) 3620 : cluster [DBG] pgmap v5092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:05 smithi115 bash[46934]: cluster 2022-04-22T19:11:04.988915+0000 mgr.y (mgr.34107) 3620 : cluster [DBG] pgmap v5092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:05 smithi121 bash[36902]: cluster 2022-04-22T19:11:04.988915+0000 mgr.y (mgr.34107) 3620 : cluster [DBG] pgmap v5092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:08 smithi121 bash[36902]: cluster 2022-04-22T19:11:06.990021+0000 mgr.y (mgr.34107) 3621 : cluster [DBG] pgmap v5093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:08 smithi115 bash[45495]: cluster 2022-04-22T19:11:06.990021+0000 mgr.y (mgr.34107) 3621 : cluster [DBG] pgmap v5093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:08 smithi115 bash[46934]: cluster 2022-04-22T19:11:06.990021+0000 mgr.y (mgr.34107) 3621 : cluster [DBG] pgmap v5093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:10 smithi121 bash[36902]: cluster 2022-04-22T19:11:08.991096+0000 mgr.y (mgr.34107) 3622 : cluster [DBG] pgmap v5094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:10 smithi121 bash[36902]: audit 2022-04-22T19:11:09.708688+0000 mon.a (mon.0) 6506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:10 smithi115 bash[45495]: cluster 2022-04-22T19:11:08.991096+0000 mgr.y (mgr.34107) 3622 : cluster [DBG] pgmap v5094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:10.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:10 smithi115 bash[45495]: audit 2022-04-22T19:11:09.708688+0000 mon.a (mon.0) 6506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:10.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:10 smithi115 bash[46934]: cluster 2022-04-22T19:11:08.991096+0000 mgr.y (mgr.34107) 3622 : cluster [DBG] pgmap v5094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:10.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:10 smithi115 bash[46934]: audit 2022-04-22T19:11:09.708688+0000 mon.a (mon.0) 6506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:12.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:12 smithi115 bash[45495]: cluster 2022-04-22T19:11:10.992258+0000 mgr.y (mgr.34107) 3623 : cluster [DBG] pgmap v5095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:12.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:12.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:12 smithi115 bash[46934]: cluster 2022-04-22T19:11:10.992258+0000 mgr.y (mgr.34107) 3623 : cluster [DBG] pgmap v5095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:12 smithi121 bash[36902]: cluster 2022-04-22T19:11:10.992258+0000 mgr.y (mgr.34107) 3623 : cluster [DBG] pgmap v5095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:13 smithi121 bash[36902]: cluster 2022-04-22T19:11:12.993228+0000 mgr.y (mgr.34107) 3624 : cluster [DBG] pgmap v5096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:13 smithi115 bash[45495]: cluster 2022-04-22T19:11:12.993228+0000 mgr.y (mgr.34107) 3624 : cluster [DBG] pgmap v5096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:13 smithi115 bash[46934]: cluster 2022-04-22T19:11:12.993228+0000 mgr.y (mgr.34107) 3624 : cluster [DBG] pgmap v5096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:13.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:14 smithi121 bash[36902]: audit 2022-04-22T19:11:14.461369+0000 mon.a (mon.0) 6507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:11:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:14 smithi121 bash[36902]: audit 2022-04-22T19:11:14.461884+0000 mon.a (mon.0) 6508 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:11:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:14 smithi115 bash[45495]: audit 2022-04-22T19:11:14.461369+0000 mon.a (mon.0) 6507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:11:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:14 smithi115 bash[45495]: audit 2022-04-22T19:11:14.461884+0000 mon.a (mon.0) 6508 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:11:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:14 smithi115 bash[46934]: audit 2022-04-22T19:11:14.461369+0000 mon.a (mon.0) 6507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:11:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:14 smithi115 bash[46934]: audit 2022-04-22T19:11:14.461884+0000 mon.a (mon.0) 6508 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:11:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:15 smithi121 bash[36902]: audit 2022-04-22T19:11:14.711797+0000 mon.a (mon.0) 6509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:15 smithi121 bash[36902]: cluster 2022-04-22T19:11:14.994295+0000 mgr.y (mgr.34107) 3625 : cluster [DBG] pgmap v5097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:15 smithi115 bash[45495]: audit 2022-04-22T19:11:14.711797+0000 mon.a (mon.0) 6509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:15 smithi115 bash[45495]: cluster 2022-04-22T19:11:14.994295+0000 mgr.y (mgr.34107) 3625 : cluster [DBG] pgmap v5097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:15 smithi115 bash[46934]: audit 2022-04-22T19:11:14.711797+0000 mon.a (mon.0) 6509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:15 smithi115 bash[46934]: cluster 2022-04-22T19:11:14.994295+0000 mgr.y (mgr.34107) 3625 : cluster [DBG] pgmap v5097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:18 smithi121 bash[36902]: cluster 2022-04-22T19:11:16.995372+0000 mgr.y (mgr.34107) 3626 : cluster [DBG] pgmap v5098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:18 smithi115 bash[45495]: cluster 2022-04-22T19:11:16.995372+0000 mgr.y (mgr.34107) 3626 : cluster [DBG] pgmap v5098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:18 smithi115 bash[46934]: cluster 2022-04-22T19:11:16.995372+0000 mgr.y (mgr.34107) 3626 : cluster [DBG] pgmap v5098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:20 smithi121 bash[36902]: cluster 2022-04-22T19:11:18.996488+0000 mgr.y (mgr.34107) 3627 : cluster [DBG] pgmap v5099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:20 smithi121 bash[36902]: audit 2022-04-22T19:11:19.714856+0000 mon.a (mon.0) 6510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:20 smithi115 bash[45495]: cluster 2022-04-22T19:11:18.996488+0000 mgr.y (mgr.34107) 3627 : cluster [DBG] pgmap v5099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:20 smithi115 bash[45495]: audit 2022-04-22T19:11:19.714856+0000 mon.a (mon.0) 6510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:20 smithi115 bash[46934]: cluster 2022-04-22T19:11:18.996488+0000 mgr.y (mgr.34107) 3627 : cluster [DBG] pgmap v5099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:20 smithi115 bash[46934]: audit 2022-04-22T19:11:19.714856+0000 mon.a (mon.0) 6510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:22.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:22 smithi115 bash[45495]: cluster 2022-04-22T19:11:20.997625+0000 mgr.y (mgr.34107) 3628 : cluster [DBG] pgmap v5100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:22 smithi115 bash[46934]: cluster 2022-04-22T19:11:20.997625+0000 mgr.y (mgr.34107) 3628 : cluster [DBG] pgmap v5100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:22.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:22.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:22 smithi121 bash[36902]: cluster 2022-04-22T19:11:20.997625+0000 mgr.y (mgr.34107) 3628 : cluster [DBG] pgmap v5100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:23 smithi121 bash[36902]: cluster 2022-04-22T19:11:22.998707+0000 mgr.y (mgr.34107) 3629 : cluster [DBG] pgmap v5101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:23 smithi115 bash[46934]: cluster 2022-04-22T19:11:22.998707+0000 mgr.y (mgr.34107) 3629 : cluster [DBG] pgmap v5101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:23 smithi115 bash[45495]: cluster 2022-04-22T19:11:22.998707+0000 mgr.y (mgr.34107) 3629 : cluster [DBG] pgmap v5101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:23.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:24 smithi115 bash[46934]: audit 2022-04-22T19:11:24.718088+0000 mon.a (mon.0) 6511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:25.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:24 smithi115 bash[45495]: audit 2022-04-22T19:11:24.718088+0000 mon.a (mon.0) 6511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:24 smithi121 bash[36902]: audit 2022-04-22T19:11:24.718088+0000 mon.a (mon.0) 6511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:25 smithi115 bash[45495]: cluster 2022-04-22T19:11:24.999889+0000 mgr.y (mgr.34107) 3630 : cluster [DBG] pgmap v5102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:25 smithi115 bash[46934]: cluster 2022-04-22T19:11:24.999889+0000 mgr.y (mgr.34107) 3630 : cluster [DBG] pgmap v5102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:25 smithi121 bash[36902]: cluster 2022-04-22T19:11:24.999889+0000 mgr.y (mgr.34107) 3630 : cluster [DBG] pgmap v5102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:26.355 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:11:26.701 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (94m) 65s ago 108m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (108m) 2m ago 108m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (96m) 65s ago 109m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (96m) 2m ago 109m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (103m) 2m ago 113m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (104m) 65s ago 115m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:11:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (103m) 65s ago 115m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (102m) 2m ago 113m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (102m) 65s ago 114m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (95m) 65s ago 108m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (95m) 2m ago 108m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (101m) 65s ago 113m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (100m) 65s ago 112m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (100m) 65s ago 112m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (99m) 65s ago 111m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (98m) 2m ago 111m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:11:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (98m) 2m ago 110m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:11:26.704 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (97m) 2m ago 110m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:11:26.704 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (96m) 2m ago 109m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:11:26.704 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (95m) 2m ago 108m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:11:27.110 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:27.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:11:27.112 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:11:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:28 smithi121 bash[36902]: audit 2022-04-22T19:11:26.329358+0000 mgr.y (mgr.34107) 3631 : audit [DBG] from='client.49138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:28 smithi121 bash[36902]: audit 2022-04-22T19:11:26.669879+0000 mgr.y (mgr.34107) 3632 : audit [DBG] from='client.39210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:28 smithi121 bash[36902]: cluster 2022-04-22T19:11:27.000893+0000 mgr.y (mgr.34107) 3633 : cluster [DBG] pgmap v5103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:28 smithi121 bash[36902]: audit 2022-04-22T19:11:27.100290+0000 mon.a (mon.0) 6512 : audit [DBG] from='client.? 172.21.15.115:0/1170270647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:11:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[46934]: audit 2022-04-22T19:11:26.329358+0000 mgr.y (mgr.34107) 3631 : audit [DBG] from='client.49138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[46934]: audit 2022-04-22T19:11:26.669879+0000 mgr.y (mgr.34107) 3632 : audit [DBG] from='client.39210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[46934]: cluster 2022-04-22T19:11:27.000893+0000 mgr.y (mgr.34107) 3633 : cluster [DBG] pgmap v5103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[46934]: audit 2022-04-22T19:11:27.100290+0000 mon.a (mon.0) 6512 : audit [DBG] from='client.? 172.21.15.115:0/1170270647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[45495]: audit 2022-04-22T19:11:26.329358+0000 mgr.y (mgr.34107) 3631 : audit [DBG] from='client.49138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[45495]: audit 2022-04-22T19:11:26.669879+0000 mgr.y (mgr.34107) 3632 : audit [DBG] from='client.39210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[45495]: cluster 2022-04-22T19:11:27.000893+0000 mgr.y (mgr.34107) 3633 : cluster [DBG] pgmap v5103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:28 smithi115 bash[45495]: audit 2022-04-22T19:11:27.100290+0000 mon.a (mon.0) 6512 : audit [DBG] from='client.? 172.21.15.115:0/1170270647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:11:30.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:30 smithi121 bash[36902]: cluster 2022-04-22T19:11:29.001834+0000 mgr.y (mgr.34107) 3634 : cluster [DBG] pgmap v5104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:30.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:30 smithi121 bash[36902]: audit 2022-04-22T19:11:29.721087+0000 mon.a (mon.0) 6513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:30.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:30 smithi121 bash[36902]: audit 2022-04-22T19:11:29.912057+0000 mon.a (mon.0) 6514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:11:30.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:30 smithi121 bash[36902]: audit 2022-04-22T19:11:29.915069+0000 mon.a (mon.0) 6515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:11:30.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:30 smithi121 bash[36902]: audit 2022-04-22T19:11:29.916918+0000 mon.a (mon.0) 6516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:11:30.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[45495]: cluster 2022-04-22T19:11:29.001834+0000 mgr.y (mgr.34107) 3634 : cluster [DBG] pgmap v5104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:30.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[45495]: audit 2022-04-22T19:11:29.721087+0000 mon.a (mon.0) 6513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:30.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[45495]: audit 2022-04-22T19:11:29.912057+0000 mon.a (mon.0) 6514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:11:30.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[45495]: audit 2022-04-22T19:11:29.915069+0000 mon.a (mon.0) 6515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[45495]: audit 2022-04-22T19:11:29.916918+0000 mon.a (mon.0) 6516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[46934]: cluster 2022-04-22T19:11:29.001834+0000 mgr.y (mgr.34107) 3634 : cluster [DBG] pgmap v5104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[46934]: audit 2022-04-22T19:11:29.721087+0000 mon.a (mon.0) 6513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[46934]: audit 2022-04-22T19:11:29.912057+0000 mon.a (mon.0) 6514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[46934]: audit 2022-04-22T19:11:29.915069+0000 mon.a (mon.0) 6515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:11:30.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:30 smithi115 bash[46934]: audit 2022-04-22T19:11:29.916918+0000 mon.a (mon.0) 6516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:11:31.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:31 smithi115 bash[46934]: audit 2022-04-22T19:11:30.279006+0000 mon.a (mon.0) 6517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:11:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:31 smithi115 bash[46934]: cluster 2022-04-22T19:11:31.002975+0000 mgr.y (mgr.34107) 3635 : cluster [DBG] pgmap v5105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:31 smithi115 bash[45495]: audit 2022-04-22T19:11:30.279006+0000 mon.a (mon.0) 6517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:11:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:31 smithi115 bash[45495]: cluster 2022-04-22T19:11:31.002975+0000 mgr.y (mgr.34107) 3635 : cluster [DBG] pgmap v5105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:31 smithi121 bash[36902]: audit 2022-04-22T19:11:30.279006+0000 mon.a (mon.0) 6517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:11:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:31 smithi121 bash[36902]: cluster 2022-04-22T19:11:31.002975+0000 mgr.y (mgr.34107) 3635 : cluster [DBG] pgmap v5105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:32.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:32.015Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:33 smithi121 bash[36902]: cluster 2022-04-22T19:11:33.004100+0000 mgr.y (mgr.34107) 3636 : cluster [DBG] pgmap v5106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:33 smithi115 bash[45495]: cluster 2022-04-22T19:11:33.004100+0000 mgr.y (mgr.34107) 3636 : cluster [DBG] pgmap v5106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:33 smithi115 bash[46934]: cluster 2022-04-22T19:11:33.004100+0000 mgr.y (mgr.34107) 3636 : cluster [DBG] pgmap v5106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:33.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:35.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:34 smithi115 bash[45495]: audit 2022-04-22T19:11:34.724040+0000 mon.a (mon.0) 6518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:34 smithi115 bash[46934]: audit 2022-04-22T19:11:34.724040+0000 mon.a (mon.0) 6518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:34 smithi121 bash[36902]: audit 2022-04-22T19:11:34.724040+0000 mon.a (mon.0) 6518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:35 smithi115 bash[45495]: cluster 2022-04-22T19:11:35.005262+0000 mgr.y (mgr.34107) 3637 : cluster [DBG] pgmap v5107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:35 smithi115 bash[46934]: cluster 2022-04-22T19:11:35.005262+0000 mgr.y (mgr.34107) 3637 : cluster [DBG] pgmap v5107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:35 smithi121 bash[36902]: cluster 2022-04-22T19:11:35.005262+0000 mgr.y (mgr.34107) 3637 : cluster [DBG] pgmap v5107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:38 smithi121 bash[36902]: cluster 2022-04-22T19:11:37.006422+0000 mgr.y (mgr.34107) 3638 : cluster [DBG] pgmap v5108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:38 smithi115 bash[45495]: cluster 2022-04-22T19:11:37.006422+0000 mgr.y (mgr.34107) 3638 : cluster [DBG] pgmap v5108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:38 smithi115 bash[46934]: cluster 2022-04-22T19:11:37.006422+0000 mgr.y (mgr.34107) 3638 : cluster [DBG] pgmap v5108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:40 smithi121 bash[36902]: cluster 2022-04-22T19:11:39.007400+0000 mgr.y (mgr.34107) 3639 : cluster [DBG] pgmap v5109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:40 smithi121 bash[36902]: audit 2022-04-22T19:11:39.727067+0000 mon.a (mon.0) 6519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:40 smithi115 bash[45495]: cluster 2022-04-22T19:11:39.007400+0000 mgr.y (mgr.34107) 3639 : cluster [DBG] pgmap v5109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:40 smithi115 bash[45495]: audit 2022-04-22T19:11:39.727067+0000 mon.a (mon.0) 6519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:40 smithi115 bash[46934]: cluster 2022-04-22T19:11:39.007400+0000 mgr.y (mgr.34107) 3639 : cluster [DBG] pgmap v5109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:40 smithi115 bash[46934]: audit 2022-04-22T19:11:39.727067+0000 mon.a (mon.0) 6519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:42 smithi115 bash[45495]: cluster 2022-04-22T19:11:41.008552+0000 mgr.y (mgr.34107) 3640 : cluster [DBG] pgmap v5110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:42.015Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:42 smithi115 bash[46934]: cluster 2022-04-22T19:11:41.008552+0000 mgr.y (mgr.34107) 3640 : cluster [DBG] pgmap v5110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:42 smithi121 bash[36902]: cluster 2022-04-22T19:11:41.008552+0000 mgr.y (mgr.34107) 3640 : cluster [DBG] pgmap v5110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:43 smithi121 bash[36902]: cluster 2022-04-22T19:11:43.009591+0000 mgr.y (mgr.34107) 3641 : cluster [DBG] pgmap v5111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:43 smithi115 bash[45495]: cluster 2022-04-22T19:11:43.009591+0000 mgr.y (mgr.34107) 3641 : cluster [DBG] pgmap v5111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:43 smithi115 bash[46934]: cluster 2022-04-22T19:11:43.009591+0000 mgr.y (mgr.34107) 3641 : cluster [DBG] pgmap v5111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:43.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:45.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:44 smithi115 bash[45495]: audit 2022-04-22T19:11:44.730170+0000 mon.a (mon.0) 6520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:45.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:44 smithi115 bash[46934]: audit 2022-04-22T19:11:44.730170+0000 mon.a (mon.0) 6520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:44 smithi121 bash[36902]: audit 2022-04-22T19:11:44.730170+0000 mon.a (mon.0) 6520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:45 smithi121 bash[36902]: cluster 2022-04-22T19:11:45.010710+0000 mgr.y (mgr.34107) 3642 : cluster [DBG] pgmap v5112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:45 smithi115 bash[45495]: cluster 2022-04-22T19:11:45.010710+0000 mgr.y (mgr.34107) 3642 : cluster [DBG] pgmap v5112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:45 smithi115 bash[46934]: cluster 2022-04-22T19:11:45.010710+0000 mgr.y (mgr.34107) 3642 : cluster [DBG] pgmap v5112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:48 smithi121 bash[36902]: cluster 2022-04-22T19:11:47.011888+0000 mgr.y (mgr.34107) 3643 : cluster [DBG] pgmap v5113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:48 smithi115 bash[45495]: cluster 2022-04-22T19:11:47.011888+0000 mgr.y (mgr.34107) 3643 : cluster [DBG] pgmap v5113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:48 smithi115 bash[46934]: cluster 2022-04-22T19:11:47.011888+0000 mgr.y (mgr.34107) 3643 : cluster [DBG] pgmap v5113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:50 smithi121 bash[36902]: cluster 2022-04-22T19:11:49.012886+0000 mgr.y (mgr.34107) 3644 : cluster [DBG] pgmap v5114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:50 smithi121 bash[36902]: audit 2022-04-22T19:11:49.733402+0000 mon.a (mon.0) 6521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:50.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:50 smithi115 bash[45495]: cluster 2022-04-22T19:11:49.012886+0000 mgr.y (mgr.34107) 3644 : cluster [DBG] pgmap v5114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:50 smithi115 bash[45495]: audit 2022-04-22T19:11:49.733402+0000 mon.a (mon.0) 6521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:50 smithi115 bash[46934]: cluster 2022-04-22T19:11:49.012886+0000 mgr.y (mgr.34107) 3644 : cluster [DBG] pgmap v5114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:50 smithi115 bash[46934]: audit 2022-04-22T19:11:49.733402+0000 mon.a (mon.0) 6521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:52 smithi115 bash[45495]: cluster 2022-04-22T19:11:51.013973+0000 mgr.y (mgr.34107) 3645 : cluster [DBG] pgmap v5115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:52.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:52 smithi115 bash[46934]: cluster 2022-04-22T19:11:51.013973+0000 mgr.y (mgr.34107) 3645 : cluster [DBG] pgmap v5115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:52.320 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:52.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:52 smithi121 bash[36902]: cluster 2022-04-22T19:11:51.013973+0000 mgr.y (mgr.34107) 3645 : cluster [DBG] pgmap v5115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:53 smithi121 bash[36902]: cluster 2022-04-22T19:11:53.015053+0000 mgr.y (mgr.34107) 3646 : cluster [DBG] pgmap v5116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:53.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:53 smithi115 bash[45495]: cluster 2022-04-22T19:11:53.015053+0000 mgr.y (mgr.34107) 3646 : cluster [DBG] pgmap v5116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:53.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:53 smithi115 bash[46934]: cluster 2022-04-22T19:11:53.015053+0000 mgr.y (mgr.34107) 3646 : cluster [DBG] pgmap v5116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:11:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:11:53.890Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:11:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:54 smithi121 bash[36902]: audit 2022-04-22T19:11:54.736468+0000 mon.a (mon.0) 6522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:54 smithi115 bash[45495]: audit 2022-04-22T19:11:54.736468+0000 mon.a (mon.0) 6522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:54 smithi115 bash[46934]: audit 2022-04-22T19:11:54.736468+0000 mon.a (mon.0) 6522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:11:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:55 smithi121 bash[36902]: cluster 2022-04-22T19:11:55.016229+0000 mgr.y (mgr.34107) 3647 : cluster [DBG] pgmap v5117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:55 smithi115 bash[46934]: cluster 2022-04-22T19:11:55.016229+0000 mgr.y (mgr.34107) 3647 : cluster [DBG] pgmap v5117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:55 smithi115 bash[45495]: cluster 2022-04-22T19:11:55.016229+0000 mgr.y (mgr.34107) 3647 : cluster [DBG] pgmap v5117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:57.482 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:11:57.826 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:11:57.826 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (95m) 96s ago 108m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:11:57.826 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (108m) 2m ago 108m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:11:57.826 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (97m) 96s ago 109m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (96m) 2m ago 109m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (104m) 2m ago 114m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (104m) 96s ago 116m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (103m) 96s ago 116m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (102m) 2m ago 114m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (103m) 96s ago 114m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (96m) 96s ago 109m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (96m) 2m ago 109m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (101m) 96s ago 113m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (101m) 96s ago 113m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:11:57.827 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (100m) 96s ago 112m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (100m) 96s ago 112m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (99m) 2m ago 111m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (98m) 2m ago 111m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (98m) 2m ago 110m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (97m) 2m ago 110m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:11:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (95m) 2m ago 109m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:11:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:11:58.232 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:11:58.232 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:11:58.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:11:58.234 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:11:58.234 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:11:58.234 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:11:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:11:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:58 smithi121 bash[36902]: cluster 2022-04-22T19:11:57.017507+0000 mgr.y (mgr.34107) 3648 : cluster [DBG] pgmap v5118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:58 smithi115 bash[45495]: cluster 2022-04-22T19:11:57.017507+0000 mgr.y (mgr.34107) 3648 : cluster [DBG] pgmap v5118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:58 smithi115 bash[46934]: cluster 2022-04-22T19:11:57.017507+0000 mgr.y (mgr.34107) 3648 : cluster [DBG] pgmap v5118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:11:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:59 smithi121 bash[36902]: audit 2022-04-22T19:11:57.456278+0000 mgr.y (mgr.34107) 3649 : audit [DBG] from='client.39222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:59 smithi121 bash[36902]: audit 2022-04-22T19:11:57.796834+0000 mgr.y (mgr.34107) 3650 : audit [DBG] from='client.49162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:11:59 smithi121 bash[36902]: audit 2022-04-22T19:11:58.222126+0000 mon.a (mon.0) 6523 : audit [DBG] from='client.? 172.21.15.115:0/2146246519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:11:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[45495]: audit 2022-04-22T19:11:57.456278+0000 mgr.y (mgr.34107) 3649 : audit [DBG] from='client.39222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[45495]: audit 2022-04-22T19:11:57.796834+0000 mgr.y (mgr.34107) 3650 : audit [DBG] from='client.49162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[45495]: audit 2022-04-22T19:11:58.222126+0000 mon.a (mon.0) 6523 : audit [DBG] from='client.? 172.21.15.115:0/2146246519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:11:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[46934]: audit 2022-04-22T19:11:57.456278+0000 mgr.y (mgr.34107) 3649 : audit [DBG] from='client.39222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[46934]: audit 2022-04-22T19:11:57.796834+0000 mgr.y (mgr.34107) 3650 : audit [DBG] from='client.49162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:11:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:11:59 smithi115 bash[46934]: audit 2022-04-22T19:11:58.222126+0000 mon.a (mon.0) 6523 : audit [DBG] from='client.? 172.21.15.115:0/2146246519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:12:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:00 smithi121 bash[36902]: cluster 2022-04-22T19:11:59.018460+0000 mgr.y (mgr.34107) 3651 : cluster [DBG] pgmap v5119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:00 smithi121 bash[36902]: audit 2022-04-22T19:11:59.739452+0000 mon.a (mon.0) 6524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:00 smithi115 bash[45495]: cluster 2022-04-22T19:11:59.018460+0000 mgr.y (mgr.34107) 3651 : cluster [DBG] pgmap v5119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:00 smithi115 bash[45495]: audit 2022-04-22T19:11:59.739452+0000 mon.a (mon.0) 6524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:00 smithi115 bash[46934]: cluster 2022-04-22T19:11:59.018460+0000 mgr.y (mgr.34107) 3651 : cluster [DBG] pgmap v5119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:00 smithi115 bash[46934]: audit 2022-04-22T19:11:59.739452+0000 mon.a (mon.0) 6524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:02 smithi115 bash[45495]: cluster 2022-04-22T19:12:01.019637+0000 mgr.y (mgr.34107) 3652 : cluster [DBG] pgmap v5120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:02.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:02.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:02 smithi115 bash[46934]: cluster 2022-04-22T19:12:01.019637+0000 mgr.y (mgr.34107) 3652 : cluster [DBG] pgmap v5120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:02 smithi121 bash[36902]: cluster 2022-04-22T19:12:01.019637+0000 mgr.y (mgr.34107) 3652 : cluster [DBG] pgmap v5120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:03 smithi121 bash[36902]: cluster 2022-04-22T19:12:03.020646+0000 mgr.y (mgr.34107) 3653 : cluster [DBG] pgmap v5121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:03 smithi115 bash[45495]: cluster 2022-04-22T19:12:03.020646+0000 mgr.y (mgr.34107) 3653 : cluster [DBG] pgmap v5121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:03 smithi115 bash[46934]: cluster 2022-04-22T19:12:03.020646+0000 mgr.y (mgr.34107) 3653 : cluster [DBG] pgmap v5121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:03.890Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:04 smithi121 bash[36902]: audit 2022-04-22T19:12:04.742590+0000 mon.a (mon.0) 6525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:05.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:04 smithi115 bash[45495]: audit 2022-04-22T19:12:04.742590+0000 mon.a (mon.0) 6525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:04 smithi115 bash[46934]: audit 2022-04-22T19:12:04.742590+0000 mon.a (mon.0) 6525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:05 smithi121 bash[36902]: cluster 2022-04-22T19:12:05.021756+0000 mgr.y (mgr.34107) 3654 : cluster [DBG] pgmap v5122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:06.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:05 smithi115 bash[45495]: cluster 2022-04-22T19:12:05.021756+0000 mgr.y (mgr.34107) 3654 : cluster [DBG] pgmap v5122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:06.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:05 smithi115 bash[46934]: cluster 2022-04-22T19:12:05.021756+0000 mgr.y (mgr.34107) 3654 : cluster [DBG] pgmap v5122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:08 smithi121 bash[36902]: cluster 2022-04-22T19:12:07.022959+0000 mgr.y (mgr.34107) 3655 : cluster [DBG] pgmap v5123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:08 smithi115 bash[45495]: cluster 2022-04-22T19:12:07.022959+0000 mgr.y (mgr.34107) 3655 : cluster [DBG] pgmap v5123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:08 smithi115 bash[46934]: cluster 2022-04-22T19:12:07.022959+0000 mgr.y (mgr.34107) 3655 : cluster [DBG] pgmap v5123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:10 smithi121 bash[36902]: cluster 2022-04-22T19:12:09.023963+0000 mgr.y (mgr.34107) 3656 : cluster [DBG] pgmap v5124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:10 smithi121 bash[36902]: audit 2022-04-22T19:12:09.745634+0000 mon.a (mon.0) 6526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:10 smithi115 bash[45495]: cluster 2022-04-22T19:12:09.023963+0000 mgr.y (mgr.34107) 3656 : cluster [DBG] pgmap v5124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:10 smithi115 bash[45495]: audit 2022-04-22T19:12:09.745634+0000 mon.a (mon.0) 6526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:10 smithi115 bash[46934]: cluster 2022-04-22T19:12:09.023963+0000 mgr.y (mgr.34107) 3656 : cluster [DBG] pgmap v5124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:10 smithi115 bash[46934]: audit 2022-04-22T19:12:09.745634+0000 mon.a (mon.0) 6526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:12 smithi115 bash[45495]: cluster 2022-04-22T19:12:11.025056+0000 mgr.y (mgr.34107) 3657 : cluster [DBG] pgmap v5125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:12 smithi115 bash[46934]: cluster 2022-04-22T19:12:11.025056+0000 mgr.y (mgr.34107) 3657 : cluster [DBG] pgmap v5125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:12.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:12.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:12 smithi121 bash[36902]: cluster 2022-04-22T19:12:11.025056+0000 mgr.y (mgr.34107) 3657 : cluster [DBG] pgmap v5125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:13 smithi121 bash[36902]: cluster 2022-04-22T19:12:13.026080+0000 mgr.y (mgr.34107) 3658 : cluster [DBG] pgmap v5126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:13 smithi115 bash[45495]: cluster 2022-04-22T19:12:13.026080+0000 mgr.y (mgr.34107) 3658 : cluster [DBG] pgmap v5126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:13 smithi115 bash[46934]: cluster 2022-04-22T19:12:13.026080+0000 mgr.y (mgr.34107) 3658 : cluster [DBG] pgmap v5126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:13.890Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:14 smithi115 bash[45495]: audit 2022-04-22T19:12:14.472152+0000 mon.a (mon.0) 6527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:12:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:14 smithi115 bash[45495]: audit 2022-04-22T19:12:14.475423+0000 mon.a (mon.0) 6528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:12:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:14 smithi115 bash[46934]: audit 2022-04-22T19:12:14.472152+0000 mon.a (mon.0) 6527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:12:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:14 smithi115 bash[46934]: audit 2022-04-22T19:12:14.475423+0000 mon.a (mon.0) 6528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:12:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:14 smithi121 bash[36902]: audit 2022-04-22T19:12:14.472152+0000 mon.a (mon.0) 6527 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:12:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:14 smithi121 bash[36902]: audit 2022-04-22T19:12:14.475423+0000 mon.a (mon.0) 6528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:12:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:15 smithi115 bash[45495]: audit 2022-04-22T19:12:14.748619+0000 mon.a (mon.0) 6529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:15 smithi115 bash[45495]: cluster 2022-04-22T19:12:15.027191+0000 mgr.y (mgr.34107) 3659 : cluster [DBG] pgmap v5127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:15 smithi115 bash[46934]: audit 2022-04-22T19:12:14.748619+0000 mon.a (mon.0) 6529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:15 smithi115 bash[46934]: cluster 2022-04-22T19:12:15.027191+0000 mgr.y (mgr.34107) 3659 : cluster [DBG] pgmap v5127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:15 smithi121 bash[36902]: audit 2022-04-22T19:12:14.748619+0000 mon.a (mon.0) 6529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:16.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:15 smithi121 bash[36902]: cluster 2022-04-22T19:12:15.027191+0000 mgr.y (mgr.34107) 3659 : cluster [DBG] pgmap v5127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:18 smithi121 bash[36902]: cluster 2022-04-22T19:12:17.028403+0000 mgr.y (mgr.34107) 3660 : cluster [DBG] pgmap v5128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:18 smithi115 bash[45495]: cluster 2022-04-22T19:12:17.028403+0000 mgr.y (mgr.34107) 3660 : cluster [DBG] pgmap v5128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:18 smithi115 bash[46934]: cluster 2022-04-22T19:12:17.028403+0000 mgr.y (mgr.34107) 3660 : cluster [DBG] pgmap v5128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:20 smithi121 bash[36902]: cluster 2022-04-22T19:12:19.029349+0000 mgr.y (mgr.34107) 3661 : cluster [DBG] pgmap v5129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:20 smithi121 bash[36902]: audit 2022-04-22T19:12:19.751822+0000 mon.a (mon.0) 6530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:20 smithi115 bash[45495]: cluster 2022-04-22T19:12:19.029349+0000 mgr.y (mgr.34107) 3661 : cluster [DBG] pgmap v5129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:20.559 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:20 smithi115 bash[45495]: audit 2022-04-22T19:12:19.751822+0000 mon.a (mon.0) 6530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:20.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:20 smithi115 bash[46934]: cluster 2022-04-22T19:12:19.029349+0000 mgr.y (mgr.34107) 3661 : cluster [DBG] pgmap v5129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:20.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:20 smithi115 bash[46934]: audit 2022-04-22T19:12:19.751822+0000 mon.a (mon.0) 6530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:22.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:22 smithi115 bash[45495]: cluster 2022-04-22T19:12:21.030505+0000 mgr.y (mgr.34107) 3662 : cluster [DBG] pgmap v5130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:22.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:22.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:22.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:22 smithi115 bash[46934]: cluster 2022-04-22T19:12:21.030505+0000 mgr.y (mgr.34107) 3662 : cluster [DBG] pgmap v5130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:22 smithi121 bash[36902]: cluster 2022-04-22T19:12:21.030505+0000 mgr.y (mgr.34107) 3662 : cluster [DBG] pgmap v5130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:23 smithi115 bash[45495]: cluster 2022-04-22T19:12:23.031549+0000 mgr.y (mgr.34107) 3663 : cluster [DBG] pgmap v5131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:23.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:23 smithi115 bash[46934]: cluster 2022-04-22T19:12:23.031549+0000 mgr.y (mgr.34107) 3663 : cluster [DBG] pgmap v5131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:23 smithi121 bash[36902]: cluster 2022-04-22T19:12:23.031549+0000 mgr.y (mgr.34107) 3663 : cluster [DBG] pgmap v5131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:23.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:24 smithi121 bash[36902]: audit 2022-04-22T19:12:24.755034+0000 mon.a (mon.0) 6531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:25.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:24 smithi115 bash[45495]: audit 2022-04-22T19:12:24.755034+0000 mon.a (mon.0) 6531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:24 smithi115 bash[46934]: audit 2022-04-22T19:12:24.755034+0000 mon.a (mon.0) 6531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:25 smithi121 bash[36902]: cluster 2022-04-22T19:12:25.032669+0000 mgr.y (mgr.34107) 3664 : cluster [DBG] pgmap v5132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:25 smithi115 bash[45495]: cluster 2022-04-22T19:12:25.032669+0000 mgr.y (mgr.34107) 3664 : cluster [DBG] pgmap v5132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:25 smithi115 bash[46934]: cluster 2022-04-22T19:12:25.032669+0000 mgr.y (mgr.34107) 3664 : cluster [DBG] pgmap v5132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:28 smithi121 bash[36902]: cluster 2022-04-22T19:12:27.033707+0000 mgr.y (mgr.34107) 3665 : cluster [DBG] pgmap v5133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:28 smithi115 bash[46934]: cluster 2022-04-22T19:12:27.033707+0000 mgr.y (mgr.34107) 3665 : cluster [DBG] pgmap v5133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:28 smithi115 bash[45495]: cluster 2022-04-22T19:12:27.033707+0000 mgr.y (mgr.34107) 3665 : cluster [DBG] pgmap v5133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:28.607 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:12:28.945 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (95m) 2m ago 109m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (109m) 3m ago 109m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (97m) 2m ago 110m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (97m) 3m ago 110m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (104m) 3m ago 114m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (105m) 2m ago 116m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:12:28.946 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (104m) 2m ago 116m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (103m) 3m ago 114m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (103m) 2m ago 115m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (96m) 2m ago 109m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (96m) 3m ago 109m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (102m) 2m ago 114m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (101m) 2m ago 113m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (101m) 2m ago 113m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (100m) 2m ago 112m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (99m) 3m ago 112m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:12:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (99m) 3m ago 111m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:12:28.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (98m) 3m ago 111m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:12:28.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (97m) 3m ago 110m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:12:28.948 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (96m) 3m ago 109m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:12:29.349 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:12:29.349 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:12:29.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:12:29.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:12:29.350 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:12:29.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:12:29.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:12:29.351 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:12:29.351 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:12:29.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:12:29.351 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:12:29.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:12:29.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:12:29.352 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:12:29.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:12:29.352 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:12:29.353 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:12:30.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[46934]: audit 2022-04-22T19:12:28.582525+0000 mgr.y (mgr.34107) 3666 : audit [DBG] from='client.39240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[46934]: audit 2022-04-22T19:12:28.914499+0000 mgr.y (mgr.34107) 3667 : audit [DBG] from='client.39246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[46934]: cluster 2022-04-22T19:12:29.034602+0000 mgr.y (mgr.34107) 3668 : cluster [DBG] pgmap v5134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[46934]: audit 2022-04-22T19:12:29.338652+0000 mon.a (mon.0) 6532 : audit [DBG] from='client.? 172.21.15.115:0/3538604306' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[46934]: audit 2022-04-22T19:12:29.758207+0000 mon.a (mon.0) 6533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[45495]: audit 2022-04-22T19:12:28.582525+0000 mgr.y (mgr.34107) 3666 : audit [DBG] from='client.39240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[45495]: audit 2022-04-22T19:12:28.914499+0000 mgr.y (mgr.34107) 3667 : audit [DBG] from='client.39246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[45495]: cluster 2022-04-22T19:12:29.034602+0000 mgr.y (mgr.34107) 3668 : cluster [DBG] pgmap v5134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[45495]: audit 2022-04-22T19:12:29.338652+0000 mon.a (mon.0) 6532 : audit [DBG] from='client.? 172.21.15.115:0/3538604306' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:12:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:30 smithi115 bash[45495]: audit 2022-04-22T19:12:29.758207+0000 mon.a (mon.0) 6533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:30.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:30 smithi121 bash[36902]: audit 2022-04-22T19:12:28.582525+0000 mgr.y (mgr.34107) 3666 : audit [DBG] from='client.39240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:30 smithi121 bash[36902]: audit 2022-04-22T19:12:28.914499+0000 mgr.y (mgr.34107) 3667 : audit [DBG] from='client.39246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:12:30.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:30 smithi121 bash[36902]: cluster 2022-04-22T19:12:29.034602+0000 mgr.y (mgr.34107) 3668 : cluster [DBG] pgmap v5134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:30.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:30 smithi121 bash[36902]: audit 2022-04-22T19:12:29.338652+0000 mon.a (mon.0) 6532 : audit [DBG] from='client.? 172.21.15.115:0/3538604306' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:12:30.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:30 smithi121 bash[36902]: audit 2022-04-22T19:12:29.758207+0000 mon.a (mon.0) 6533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:31 smithi121 bash[36902]: audit 2022-04-22T19:12:30.286497+0000 mon.a (mon.0) 6534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:12:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:31 smithi121 bash[36902]: audit 2022-04-22T19:12:30.290235+0000 mon.a (mon.0) 6535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:12:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:31 smithi121 bash[36902]: audit 2022-04-22T19:12:30.291911+0000 mon.a (mon.0) 6536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:12:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:31 smithi121 bash[36902]: audit 2022-04-22T19:12:30.628678+0000 mon.a (mon.0) 6537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:12:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[45495]: audit 2022-04-22T19:12:30.286497+0000 mon.a (mon.0) 6534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[45495]: audit 2022-04-22T19:12:30.290235+0000 mon.a (mon.0) 6535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[45495]: audit 2022-04-22T19:12:30.291911+0000 mon.a (mon.0) 6536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[45495]: audit 2022-04-22T19:12:30.628678+0000 mon.a (mon.0) 6537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[46934]: audit 2022-04-22T19:12:30.286497+0000 mon.a (mon.0) 6534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[46934]: audit 2022-04-22T19:12:30.290235+0000 mon.a (mon.0) 6535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[46934]: audit 2022-04-22T19:12:30.291911+0000 mon.a (mon.0) 6536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:12:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:31 smithi115 bash[46934]: audit 2022-04-22T19:12:30.628678+0000 mon.a (mon.0) 6537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:12:32.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:32 smithi115 bash[45495]: cluster 2022-04-22T19:12:31.035802+0000 mgr.y (mgr.34107) 3669 : cluster [DBG] pgmap v5135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:32.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:32 smithi115 bash[46934]: cluster 2022-04-22T19:12:31.035802+0000 mgr.y (mgr.34107) 3669 : cluster [DBG] pgmap v5135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:32.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:32.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:32 smithi121 bash[36902]: cluster 2022-04-22T19:12:31.035802+0000 mgr.y (mgr.34107) 3669 : cluster [DBG] pgmap v5135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:33 smithi115 bash[45495]: cluster 2022-04-22T19:12:33.036833+0000 mgr.y (mgr.34107) 3670 : cluster [DBG] pgmap v5136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:33.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:33 smithi115 bash[46934]: cluster 2022-04-22T19:12:33.036833+0000 mgr.y (mgr.34107) 3670 : cluster [DBG] pgmap v5136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:33 smithi121 bash[36902]: cluster 2022-04-22T19:12:33.036833+0000 mgr.y (mgr.34107) 3670 : cluster [DBG] pgmap v5136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:33.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:34 smithi121 bash[36902]: audit 2022-04-22T19:12:34.761317+0000 mon.a (mon.0) 6538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:35.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:34 smithi115 bash[45495]: audit 2022-04-22T19:12:34.761317+0000 mon.a (mon.0) 6538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:34 smithi115 bash[46934]: audit 2022-04-22T19:12:34.761317+0000 mon.a (mon.0) 6538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:35 smithi121 bash[36902]: cluster 2022-04-22T19:12:35.037976+0000 mgr.y (mgr.34107) 3671 : cluster [DBG] pgmap v5137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:35 smithi115 bash[45495]: cluster 2022-04-22T19:12:35.037976+0000 mgr.y (mgr.34107) 3671 : cluster [DBG] pgmap v5137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:35 smithi115 bash[46934]: cluster 2022-04-22T19:12:35.037976+0000 mgr.y (mgr.34107) 3671 : cluster [DBG] pgmap v5137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:38 smithi121 bash[36902]: cluster 2022-04-22T19:12:37.039232+0000 mgr.y (mgr.34107) 3672 : cluster [DBG] pgmap v5138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:38 smithi115 bash[45495]: cluster 2022-04-22T19:12:37.039232+0000 mgr.y (mgr.34107) 3672 : cluster [DBG] pgmap v5138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:38 smithi115 bash[46934]: cluster 2022-04-22T19:12:37.039232+0000 mgr.y (mgr.34107) 3672 : cluster [DBG] pgmap v5138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:40 smithi121 bash[36902]: cluster 2022-04-22T19:12:39.040248+0000 mgr.y (mgr.34107) 3673 : cluster [DBG] pgmap v5139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:40 smithi121 bash[36902]: audit 2022-04-22T19:12:39.764528+0000 mon.a (mon.0) 6539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:40.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:40 smithi115 bash[46934]: cluster 2022-04-22T19:12:39.040248+0000 mgr.y (mgr.34107) 3673 : cluster [DBG] pgmap v5139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:40 smithi115 bash[46934]: audit 2022-04-22T19:12:39.764528+0000 mon.a (mon.0) 6539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:40 smithi115 bash[45495]: cluster 2022-04-22T19:12:39.040248+0000 mgr.y (mgr.34107) 3673 : cluster [DBG] pgmap v5139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:40 smithi115 bash[45495]: audit 2022-04-22T19:12:39.764528+0000 mon.a (mon.0) 6539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:42 smithi115 bash[45495]: cluster 2022-04-22T19:12:41.041408+0000 mgr.y (mgr.34107) 3674 : cluster [DBG] pgmap v5140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:42.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:42 smithi115 bash[46934]: cluster 2022-04-22T19:12:41.041408+0000 mgr.y (mgr.34107) 3674 : cluster [DBG] pgmap v5140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:42.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:42.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:42 smithi121 bash[36902]: cluster 2022-04-22T19:12:41.041408+0000 mgr.y (mgr.34107) 3674 : cluster [DBG] pgmap v5140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:43.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:43 smithi115 bash[45495]: cluster 2022-04-22T19:12:43.042436+0000 mgr.y (mgr.34107) 3675 : cluster [DBG] pgmap v5141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:43 smithi115 bash[46934]: cluster 2022-04-22T19:12:43.042436+0000 mgr.y (mgr.34107) 3675 : cluster [DBG] pgmap v5141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:43 smithi121 bash[36902]: cluster 2022-04-22T19:12:43.042436+0000 mgr.y (mgr.34107) 3675 : cluster [DBG] pgmap v5141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:43.893Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:44 smithi121 bash[36902]: audit 2022-04-22T19:12:44.767607+0000 mon.a (mon.0) 6540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:45.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:44 smithi115 bash[45495]: audit 2022-04-22T19:12:44.767607+0000 mon.a (mon.0) 6540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:45.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:44 smithi115 bash[46934]: audit 2022-04-22T19:12:44.767607+0000 mon.a (mon.0) 6540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:45 smithi121 bash[36902]: cluster 2022-04-22T19:12:45.043599+0000 mgr.y (mgr.34107) 3676 : cluster [DBG] pgmap v5142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:45 smithi115 bash[45495]: cluster 2022-04-22T19:12:45.043599+0000 mgr.y (mgr.34107) 3676 : cluster [DBG] pgmap v5142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:45 smithi115 bash[46934]: cluster 2022-04-22T19:12:45.043599+0000 mgr.y (mgr.34107) 3676 : cluster [DBG] pgmap v5142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:48 smithi121 bash[36902]: cluster 2022-04-22T19:12:47.044800+0000 mgr.y (mgr.34107) 3677 : cluster [DBG] pgmap v5143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:48 smithi115 bash[45495]: cluster 2022-04-22T19:12:47.044800+0000 mgr.y (mgr.34107) 3677 : cluster [DBG] pgmap v5143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:48 smithi115 bash[46934]: cluster 2022-04-22T19:12:47.044800+0000 mgr.y (mgr.34107) 3677 : cluster [DBG] pgmap v5143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:50 smithi121 bash[36902]: cluster 2022-04-22T19:12:49.045733+0000 mgr.y (mgr.34107) 3678 : cluster [DBG] pgmap v5144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:50 smithi121 bash[36902]: audit 2022-04-22T19:12:49.770520+0000 mon.a (mon.0) 6541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:50 smithi115 bash[45495]: cluster 2022-04-22T19:12:49.045733+0000 mgr.y (mgr.34107) 3678 : cluster [DBG] pgmap v5144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:50 smithi115 bash[45495]: audit 2022-04-22T19:12:49.770520+0000 mon.a (mon.0) 6541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:50 smithi115 bash[46934]: cluster 2022-04-22T19:12:49.045733+0000 mgr.y (mgr.34107) 3678 : cluster [DBG] pgmap v5144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:50 smithi115 bash[46934]: audit 2022-04-22T19:12:49.770520+0000 mon.a (mon.0) 6541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:52.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:52 smithi115 bash[45495]: cluster 2022-04-22T19:12:51.046916+0000 mgr.y (mgr.34107) 3679 : cluster [DBG] pgmap v5145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:52.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:52.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:52 smithi115 bash[46934]: cluster 2022-04-22T19:12:51.046916+0000 mgr.y (mgr.34107) 3679 : cluster [DBG] pgmap v5145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:52 smithi121 bash[36902]: cluster 2022-04-22T19:12:51.046916+0000 mgr.y (mgr.34107) 3679 : cluster [DBG] pgmap v5145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:53 smithi115 bash[45495]: cluster 2022-04-22T19:12:53.048035+0000 mgr.y (mgr.34107) 3680 : cluster [DBG] pgmap v5146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:53 smithi115 bash[46934]: cluster 2022-04-22T19:12:53.048035+0000 mgr.y (mgr.34107) 3680 : cluster [DBG] pgmap v5146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:53 smithi121 bash[36902]: cluster 2022-04-22T19:12:53.048035+0000 mgr.y (mgr.34107) 3680 : cluster [DBG] pgmap v5146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:12:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:12:53.894Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:12:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:54 smithi121 bash[36902]: audit 2022-04-22T19:12:54.773692+0000 mon.a (mon.0) 6542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:54 smithi115 bash[46934]: audit 2022-04-22T19:12:54.773692+0000 mon.a (mon.0) 6542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:54 smithi115 bash[45495]: audit 2022-04-22T19:12:54.773692+0000 mon.a (mon.0) 6542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:12:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:55 smithi121 bash[36902]: cluster 2022-04-22T19:12:55.048949+0000 mgr.y (mgr.34107) 3681 : cluster [DBG] pgmap v5147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:55 smithi115 bash[45495]: cluster 2022-04-22T19:12:55.048949+0000 mgr.y (mgr.34107) 3681 : cluster [DBG] pgmap v5147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:55 smithi115 bash[46934]: cluster 2022-04-22T19:12:55.048949+0000 mgr.y (mgr.34107) 3681 : cluster [DBG] pgmap v5147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:12:58 smithi121 bash[36902]: cluster 2022-04-22T19:12:57.050036+0000 mgr.y (mgr.34107) 3682 : cluster [DBG] pgmap v5148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:12:58 smithi115 bash[45495]: cluster 2022-04-22T19:12:57.050036+0000 mgr.y (mgr.34107) 3682 : cluster [DBG] pgmap v5148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:12:58 smithi115 bash[46934]: cluster 2022-04-22T19:12:57.050036+0000 mgr.y (mgr.34107) 3682 : cluster [DBG] pgmap v5148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:12:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (96m) 2m ago 109m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (109m) 3m ago 109m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (98m) 2m ago 110m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (97m) 3m ago 110m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:13:00.069 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (105m) 3m ago 115m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (105m) 2m ago 117m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (104m) 2m ago 117m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (104m) 3m ago 115m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (104m) 2m ago 115m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (97m) 2m ago 110m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (97m) 3m ago 110m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (102m) 2m ago 114m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (102m) 2m ago 114m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (101m) 2m ago 113m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (101m) 2m ago 113m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:13:00.070 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (100m) 3m ago 112m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:13:00.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (99m) 3m ago 112m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:13:00.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (99m) 3m ago 111m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:13:00.071 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (98m) 3m ago 111m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:13:00.071 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (96m) 3m ago 110m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:13:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:13:00.487 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:13:00.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:13:00.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:13:00.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:13:00.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:00.489 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:13:00.489 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:13:00.489 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:13:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:13:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:00 smithi121 bash[36902]: cluster 2022-04-22T19:12:59.051024+0000 mgr.y (mgr.34107) 3683 : cluster [DBG] pgmap v5149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:00 smithi121 bash[36902]: audit 2022-04-22T19:12:59.776720+0000 mon.a (mon.0) 6543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:00 smithi115 bash[45495]: cluster 2022-04-22T19:12:59.051024+0000 mgr.y (mgr.34107) 3683 : cluster [DBG] pgmap v5149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:00 smithi115 bash[45495]: audit 2022-04-22T19:12:59.776720+0000 mon.a (mon.0) 6543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:00 smithi115 bash[46934]: cluster 2022-04-22T19:12:59.051024+0000 mgr.y (mgr.34107) 3683 : cluster [DBG] pgmap v5149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:00 smithi115 bash[46934]: audit 2022-04-22T19:12:59.776720+0000 mon.a (mon.0) 6543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:01 smithi121 bash[36902]: audit 2022-04-22T19:12:59.694072+0000 mgr.y (mgr.34107) 3684 : audit [DBG] from='client.49192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:01 smithi121 bash[36902]: audit 2022-04-22T19:13:00.041906+0000 mgr.y (mgr.34107) 3685 : audit [DBG] from='client.49198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:01 smithi121 bash[36902]: audit 2022-04-22T19:13:00.476457+0000 mon.c (mon.1) 182 : audit [DBG] from='client.? 172.21.15.115:0/962709594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[45495]: audit 2022-04-22T19:12:59.694072+0000 mgr.y (mgr.34107) 3684 : audit [DBG] from='client.49192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[45495]: audit 2022-04-22T19:13:00.041906+0000 mgr.y (mgr.34107) 3685 : audit [DBG] from='client.49198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[45495]: audit 2022-04-22T19:13:00.476457+0000 mon.c (mon.1) 182 : audit [DBG] from='client.? 172.21.15.115:0/962709594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[46934]: audit 2022-04-22T19:12:59.694072+0000 mgr.y (mgr.34107) 3684 : audit [DBG] from='client.49192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[46934]: audit 2022-04-22T19:13:00.041906+0000 mgr.y (mgr.34107) 3685 : audit [DBG] from='client.49198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:01 smithi115 bash[46934]: audit 2022-04-22T19:13:00.476457+0000 mon.c (mon.1) 182 : audit [DBG] from='client.? 172.21.15.115:0/962709594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:02.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:02 smithi115 bash[45495]: cluster 2022-04-22T19:13:01.052324+0000 mgr.y (mgr.34107) 3686 : cluster [DBG] pgmap v5150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:02.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:02.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:02 smithi115 bash[46934]: cluster 2022-04-22T19:13:01.052324+0000 mgr.y (mgr.34107) 3686 : cluster [DBG] pgmap v5150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:02 smithi121 bash[36902]: cluster 2022-04-22T19:13:01.052324+0000 mgr.y (mgr.34107) 3686 : cluster [DBG] pgmap v5150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:03.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:03 smithi115 bash[45495]: cluster 2022-04-22T19:13:03.053376+0000 mgr.y (mgr.34107) 3687 : cluster [DBG] pgmap v5151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:03 smithi115 bash[46934]: cluster 2022-04-22T19:13:03.053376+0000 mgr.y (mgr.34107) 3687 : cluster [DBG] pgmap v5151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:03 smithi121 bash[36902]: cluster 2022-04-22T19:13:03.053376+0000 mgr.y (mgr.34107) 3687 : cluster [DBG] pgmap v5151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:03.892Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:04 smithi121 bash[36902]: audit 2022-04-22T19:13:04.779736+0000 mon.a (mon.0) 6544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:04 smithi115 bash[45495]: audit 2022-04-22T19:13:04.779736+0000 mon.a (mon.0) 6544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:04 smithi115 bash[46934]: audit 2022-04-22T19:13:04.779736+0000 mon.a (mon.0) 6544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:05 smithi121 bash[36902]: cluster 2022-04-22T19:13:05.054447+0000 mgr.y (mgr.34107) 3688 : cluster [DBG] pgmap v5152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:06.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:05 smithi115 bash[45495]: cluster 2022-04-22T19:13:05.054447+0000 mgr.y (mgr.34107) 3688 : cluster [DBG] pgmap v5152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:05 smithi115 bash[46934]: cluster 2022-04-22T19:13:05.054447+0000 mgr.y (mgr.34107) 3688 : cluster [DBG] pgmap v5152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:08 smithi121 bash[36902]: cluster 2022-04-22T19:13:07.055661+0000 mgr.y (mgr.34107) 3689 : cluster [DBG] pgmap v5153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:08 smithi115 bash[45495]: cluster 2022-04-22T19:13:07.055661+0000 mgr.y (mgr.34107) 3689 : cluster [DBG] pgmap v5153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:08 smithi115 bash[46934]: cluster 2022-04-22T19:13:07.055661+0000 mgr.y (mgr.34107) 3689 : cluster [DBG] pgmap v5153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:10 smithi121 bash[36902]: cluster 2022-04-22T19:13:09.056560+0000 mgr.y (mgr.34107) 3690 : cluster [DBG] pgmap v5154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:10 smithi121 bash[36902]: audit 2022-04-22T19:13:09.782918+0000 mon.a (mon.0) 6545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:10 smithi115 bash[46934]: cluster 2022-04-22T19:13:09.056560+0000 mgr.y (mgr.34107) 3690 : cluster [DBG] pgmap v5154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:10 smithi115 bash[46934]: audit 2022-04-22T19:13:09.782918+0000 mon.a (mon.0) 6545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:10 smithi115 bash[45495]: cluster 2022-04-22T19:13:09.056560+0000 mgr.y (mgr.34107) 3690 : cluster [DBG] pgmap v5154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:10 smithi115 bash[45495]: audit 2022-04-22T19:13:09.782918+0000 mon.a (mon.0) 6545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:12 smithi115 bash[46934]: cluster 2022-04-22T19:13:11.057674+0000 mgr.y (mgr.34107) 3691 : cluster [DBG] pgmap v5155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:12.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:12 smithi115 bash[45495]: cluster 2022-04-22T19:13:11.057674+0000 mgr.y (mgr.34107) 3691 : cluster [DBG] pgmap v5155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:12.301 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:12.021Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:12 smithi121 bash[36902]: cluster 2022-04-22T19:13:11.057674+0000 mgr.y (mgr.34107) 3691 : cluster [DBG] pgmap v5155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:13 smithi115 bash[46934]: cluster 2022-04-22T19:13:13.058717+0000 mgr.y (mgr.34107) 3692 : cluster [DBG] pgmap v5156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:13.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:13 smithi115 bash[45495]: cluster 2022-04-22T19:13:13.058717+0000 mgr.y (mgr.34107) 3692 : cluster [DBG] pgmap v5156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:13 smithi121 bash[36902]: cluster 2022-04-22T19:13:13.058717+0000 mgr.y (mgr.34107) 3692 : cluster [DBG] pgmap v5156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:13.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:14 smithi115 bash[45495]: audit 2022-04-22T19:13:14.483848+0000 mon.a (mon.0) 6546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:13:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:14 smithi115 bash[45495]: audit 2022-04-22T19:13:14.484374+0000 mon.a (mon.0) 6547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:13:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:14 smithi115 bash[46934]: audit 2022-04-22T19:13:14.483848+0000 mon.a (mon.0) 6546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:13:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:14 smithi115 bash[46934]: audit 2022-04-22T19:13:14.484374+0000 mon.a (mon.0) 6547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:13:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:14 smithi121 bash[36902]: audit 2022-04-22T19:13:14.483848+0000 mon.a (mon.0) 6546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:13:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:14 smithi121 bash[36902]: audit 2022-04-22T19:13:14.484374+0000 mon.a (mon.0) 6547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:13:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:15 smithi115 bash[46934]: audit 2022-04-22T19:13:14.786272+0000 mon.a (mon.0) 6548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:15 smithi115 bash[46934]: cluster 2022-04-22T19:13:15.059784+0000 mgr.y (mgr.34107) 3693 : cluster [DBG] pgmap v5157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:15 smithi115 bash[45495]: audit 2022-04-22T19:13:14.786272+0000 mon.a (mon.0) 6548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:15 smithi115 bash[45495]: cluster 2022-04-22T19:13:15.059784+0000 mgr.y (mgr.34107) 3693 : cluster [DBG] pgmap v5157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:15 smithi121 bash[36902]: audit 2022-04-22T19:13:14.786272+0000 mon.a (mon.0) 6548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:15 smithi121 bash[36902]: cluster 2022-04-22T19:13:15.059784+0000 mgr.y (mgr.34107) 3693 : cluster [DBG] pgmap v5157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:18 smithi121 bash[36902]: cluster 2022-04-22T19:13:17.060901+0000 mgr.y (mgr.34107) 3694 : cluster [DBG] pgmap v5158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:18 smithi115 bash[45495]: cluster 2022-04-22T19:13:17.060901+0000 mgr.y (mgr.34107) 3694 : cluster [DBG] pgmap v5158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:18 smithi115 bash[46934]: cluster 2022-04-22T19:13:17.060901+0000 mgr.y (mgr.34107) 3694 : cluster [DBG] pgmap v5158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:20.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:20 smithi121 bash[36902]: cluster 2022-04-22T19:13:19.061962+0000 mgr.y (mgr.34107) 3695 : cluster [DBG] pgmap v5159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:20.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:20 smithi121 bash[36902]: audit 2022-04-22T19:13:19.789611+0000 mon.a (mon.0) 6549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:20 smithi115 bash[45495]: cluster 2022-04-22T19:13:19.061962+0000 mgr.y (mgr.34107) 3695 : cluster [DBG] pgmap v5159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:20 smithi115 bash[45495]: audit 2022-04-22T19:13:19.789611+0000 mon.a (mon.0) 6549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:20 smithi115 bash[46934]: cluster 2022-04-22T19:13:19.061962+0000 mgr.y (mgr.34107) 3695 : cluster [DBG] pgmap v5159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:20 smithi115 bash[46934]: audit 2022-04-22T19:13:19.789611+0000 mon.a (mon.0) 6549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:22.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:22 smithi115 bash[45495]: cluster 2022-04-22T19:13:21.062999+0000 mgr.y (mgr.34107) 3696 : cluster [DBG] pgmap v5160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:22.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:22.022Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:22.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:22 smithi115 bash[46934]: cluster 2022-04-22T19:13:21.062999+0000 mgr.y (mgr.34107) 3696 : cluster [DBG] pgmap v5160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:22 smithi121 bash[36902]: cluster 2022-04-22T19:13:21.062999+0000 mgr.y (mgr.34107) 3696 : cluster [DBG] pgmap v5160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:23 smithi115 bash[45495]: cluster 2022-04-22T19:13:23.064078+0000 mgr.y (mgr.34107) 3697 : cluster [DBG] pgmap v5161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:23 smithi115 bash[46934]: cluster 2022-04-22T19:13:23.064078+0000 mgr.y (mgr.34107) 3697 : cluster [DBG] pgmap v5161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:23 smithi121 bash[36902]: cluster 2022-04-22T19:13:23.064078+0000 mgr.y (mgr.34107) 3697 : cluster [DBG] pgmap v5161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:23.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:24 smithi121 bash[36902]: audit 2022-04-22T19:13:24.792862+0000 mon.a (mon.0) 6550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:24 smithi115 bash[46934]: audit 2022-04-22T19:13:24.792862+0000 mon.a (mon.0) 6550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:24 smithi115 bash[45495]: audit 2022-04-22T19:13:24.792862+0000 mon.a (mon.0) 6550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:25 smithi121 bash[36902]: cluster 2022-04-22T19:13:25.065106+0000 mgr.y (mgr.34107) 3698 : cluster [DBG] pgmap v5162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:25 smithi115 bash[45495]: cluster 2022-04-22T19:13:25.065106+0000 mgr.y (mgr.34107) 3698 : cluster [DBG] pgmap v5162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:25 smithi115 bash[46934]: cluster 2022-04-22T19:13:25.065106+0000 mgr.y (mgr.34107) 3698 : cluster [DBG] pgmap v5162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:28 smithi121 bash[36902]: cluster 2022-04-22T19:13:27.066455+0000 mgr.y (mgr.34107) 3699 : cluster [DBG] pgmap v5163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:28 smithi115 bash[45495]: cluster 2022-04-22T19:13:27.066455+0000 mgr.y (mgr.34107) 3699 : cluster [DBG] pgmap v5163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:28 smithi115 bash[46934]: cluster 2022-04-22T19:13:27.066455+0000 mgr.y (mgr.34107) 3699 : cluster [DBG] pgmap v5163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:30 smithi121 bash[36902]: cluster 2022-04-22T19:13:29.067492+0000 mgr.y (mgr.34107) 3700 : cluster [DBG] pgmap v5164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:30 smithi121 bash[36902]: audit 2022-04-22T19:13:29.795788+0000 mon.a (mon.0) 6551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:30 smithi115 bash[45495]: cluster 2022-04-22T19:13:29.067492+0000 mgr.y (mgr.34107) 3700 : cluster [DBG] pgmap v5164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:30 smithi115 bash[45495]: audit 2022-04-22T19:13:29.795788+0000 mon.a (mon.0) 6551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:30 smithi115 bash[46934]: cluster 2022-04-22T19:13:29.067492+0000 mgr.y (mgr.34107) 3700 : cluster [DBG] pgmap v5164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:30 smithi115 bash[46934]: audit 2022-04-22T19:13:29.795788+0000 mon.a (mon.0) 6551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:30.866 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:13:31.211 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:13:31.212 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (96m) 3m ago 110m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:13:31.212 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (110m) 4m ago 110m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:13:31.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (98m) 3m ago 111m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:13:31.213 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (98m) 4m ago 111m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:13:31.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (105m) 4m ago 115m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:13:31.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (106m) 3m ago 117m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:13:31.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (105m) 3m ago 117m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:13:31.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (104m) 4m ago 115m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (104m) 3m ago 116m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (97m) 3m ago 110m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (97m) 4m ago 110m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (103m) 3m ago 115m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (102m) 3m ago 114m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (102m) 3m ago 114m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (101m) 3m ago 113m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:13:31.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (101m) 4m ago 113m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:13:31.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (100m) 4m ago 112m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:13:31.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (99m) 4m ago 112m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:13:31.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (98m) 4m ago 111m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:13:31.216 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (97m) 4m ago 110m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:13:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:31 smithi121 bash[36902]: audit 2022-04-22T19:13:30.637274+0000 mon.a (mon.0) 6552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:13:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:31 smithi121 bash[36902]: audit 2022-04-22T19:13:30.639674+0000 mon.a (mon.0) 6553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:13:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:31 smithi121 bash[36902]: audit 2022-04-22T19:13:30.641013+0000 mon.a (mon.0) 6554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:13:31.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:31 smithi121 bash[36902]: audit 2022-04-22T19:13:30.996235+0000 mon.a (mon.0) 6555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:13:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[46934]: audit 2022-04-22T19:13:30.637274+0000 mon.a (mon.0) 6552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:13:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[46934]: audit 2022-04-22T19:13:30.639674+0000 mon.a (mon.0) 6553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:13:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[46934]: audit 2022-04-22T19:13:30.641013+0000 mon.a (mon.0) 6554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:13:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[46934]: audit 2022-04-22T19:13:30.996235+0000 mon.a (mon.0) 6555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:13:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[45495]: audit 2022-04-22T19:13:30.637274+0000 mon.a (mon.0) 6552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:13:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[45495]: audit 2022-04-22T19:13:30.639674+0000 mon.a (mon.0) 6553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:13:31.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[45495]: audit 2022-04-22T19:13:30.641013+0000 mon.a (mon.0) 6554 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:13:31.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:31 smithi115 bash[45495]: audit 2022-04-22T19:13:30.996235+0000 mon.a (mon.0) 6555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:13:31.621 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:13:31.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:13:31.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:13:31.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:31.623 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:13:31.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:13:31.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:31.623 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:13:31.623 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:13:31.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:31.624 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:13:31.624 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:13:31.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:13:31.624 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:13:31.625 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:13:31.625 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:13:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:13:32.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:32.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[45495]: audit 2022-04-22T19:13:30.840885+0000 mgr.y (mgr.34107) 3701 : audit [DBG] from='client.49210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[45495]: cluster 2022-04-22T19:13:31.068515+0000 mgr.y (mgr.34107) 3702 : cluster [DBG] pgmap v5165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[45495]: audit 2022-04-22T19:13:31.179872+0000 mgr.y (mgr.34107) 3703 : audit [DBG] from='client.39282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[45495]: audit 2022-04-22T19:13:31.610422+0000 mon.a (mon.0) 6556 : audit [DBG] from='client.? 172.21.15.115:0/686422881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[46934]: audit 2022-04-22T19:13:30.840885+0000 mgr.y (mgr.34107) 3701 : audit [DBG] from='client.49210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[46934]: cluster 2022-04-22T19:13:31.068515+0000 mgr.y (mgr.34107) 3702 : cluster [DBG] pgmap v5165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[46934]: audit 2022-04-22T19:13:31.179872+0000 mgr.y (mgr.34107) 3703 : audit [DBG] from='client.39282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:32 smithi115 bash[46934]: audit 2022-04-22T19:13:31.610422+0000 mon.a (mon.0) 6556 : audit [DBG] from='client.? 172.21.15.115:0/686422881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:32 smithi121 bash[36902]: audit 2022-04-22T19:13:30.840885+0000 mgr.y (mgr.34107) 3701 : audit [DBG] from='client.49210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:32 smithi121 bash[36902]: cluster 2022-04-22T19:13:31.068515+0000 mgr.y (mgr.34107) 3702 : cluster [DBG] pgmap v5165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:32 smithi121 bash[36902]: audit 2022-04-22T19:13:31.179872+0000 mgr.y (mgr.34107) 3703 : audit [DBG] from='client.39282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:32 smithi121 bash[36902]: audit 2022-04-22T19:13:31.610422+0000 mon.a (mon.0) 6556 : audit [DBG] from='client.? 172.21.15.115:0/686422881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:13:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:33 smithi115 bash[45495]: cluster 2022-04-22T19:13:33.069613+0000 mgr.y (mgr.34107) 3704 : cluster [DBG] pgmap v5166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:33 smithi115 bash[46934]: cluster 2022-04-22T19:13:33.069613+0000 mgr.y (mgr.34107) 3704 : cluster [DBG] pgmap v5166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:33 smithi121 bash[36902]: cluster 2022-04-22T19:13:33.069613+0000 mgr.y (mgr.34107) 3704 : cluster [DBG] pgmap v5166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:33.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:34 smithi121 bash[36902]: audit 2022-04-22T19:13:34.798868+0000 mon.a (mon.0) 6557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:35.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:34 smithi115 bash[45495]: audit 2022-04-22T19:13:34.798868+0000 mon.a (mon.0) 6557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:35.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:34 smithi115 bash[46934]: audit 2022-04-22T19:13:34.798868+0000 mon.a (mon.0) 6557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:35 smithi121 bash[36902]: cluster 2022-04-22T19:13:35.070612+0000 mgr.y (mgr.34107) 3705 : cluster [DBG] pgmap v5167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:35 smithi115 bash[45495]: cluster 2022-04-22T19:13:35.070612+0000 mgr.y (mgr.34107) 3705 : cluster [DBG] pgmap v5167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:35 smithi115 bash[46934]: cluster 2022-04-22T19:13:35.070612+0000 mgr.y (mgr.34107) 3705 : cluster [DBG] pgmap v5167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:38 smithi121 bash[36902]: cluster 2022-04-22T19:13:37.071777+0000 mgr.y (mgr.34107) 3706 : cluster [DBG] pgmap v5168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:38 smithi115 bash[46934]: cluster 2022-04-22T19:13:37.071777+0000 mgr.y (mgr.34107) 3706 : cluster [DBG] pgmap v5168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:38 smithi115 bash[45495]: cluster 2022-04-22T19:13:37.071777+0000 mgr.y (mgr.34107) 3706 : cluster [DBG] pgmap v5168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:40 smithi121 bash[36902]: cluster 2022-04-22T19:13:39.072789+0000 mgr.y (mgr.34107) 3707 : cluster [DBG] pgmap v5169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:40 smithi121 bash[36902]: audit 2022-04-22T19:13:39.802079+0000 mon.a (mon.0) 6558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:40 smithi115 bash[45495]: cluster 2022-04-22T19:13:39.072789+0000 mgr.y (mgr.34107) 3707 : cluster [DBG] pgmap v5169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:40 smithi115 bash[45495]: audit 2022-04-22T19:13:39.802079+0000 mon.a (mon.0) 6558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:40 smithi115 bash[46934]: cluster 2022-04-22T19:13:39.072789+0000 mgr.y (mgr.34107) 3707 : cluster [DBG] pgmap v5169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:40 smithi115 bash[46934]: audit 2022-04-22T19:13:39.802079+0000 mon.a (mon.0) 6558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:42 smithi115 bash[45495]: cluster 2022-04-22T19:13:41.073824+0000 mgr.y (mgr.34107) 3708 : cluster [DBG] pgmap v5170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:42 smithi115 bash[46934]: cluster 2022-04-22T19:13:41.073824+0000 mgr.y (mgr.34107) 3708 : cluster [DBG] pgmap v5170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:42.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:42.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:42 smithi121 bash[36902]: cluster 2022-04-22T19:13:41.073824+0000 mgr.y (mgr.34107) 3708 : cluster [DBG] pgmap v5170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:43.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:43 smithi115 bash[45495]: cluster 2022-04-22T19:13:43.074875+0000 mgr.y (mgr.34107) 3709 : cluster [DBG] pgmap v5171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:43 smithi115 bash[46934]: cluster 2022-04-22T19:13:43.074875+0000 mgr.y (mgr.34107) 3709 : cluster [DBG] pgmap v5171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:43 smithi121 bash[36902]: cluster 2022-04-22T19:13:43.074875+0000 mgr.y (mgr.34107) 3709 : cluster [DBG] pgmap v5171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:43.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:44 smithi121 bash[36902]: audit 2022-04-22T19:13:44.805459+0000 mon.a (mon.0) 6559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:45.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:44 smithi115 bash[46934]: audit 2022-04-22T19:13:44.805459+0000 mon.a (mon.0) 6559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:45.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:44 smithi115 bash[45495]: audit 2022-04-22T19:13:44.805459+0000 mon.a (mon.0) 6559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:45 smithi121 bash[36902]: cluster 2022-04-22T19:13:45.075955+0000 mgr.y (mgr.34107) 3710 : cluster [DBG] pgmap v5172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:45 smithi115 bash[46934]: cluster 2022-04-22T19:13:45.075955+0000 mgr.y (mgr.34107) 3710 : cluster [DBG] pgmap v5172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:45 smithi115 bash[45495]: cluster 2022-04-22T19:13:45.075955+0000 mgr.y (mgr.34107) 3710 : cluster [DBG] pgmap v5172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:48 smithi121 bash[36902]: cluster 2022-04-22T19:13:47.077129+0000 mgr.y (mgr.34107) 3711 : cluster [DBG] pgmap v5173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:48 smithi115 bash[46934]: cluster 2022-04-22T19:13:47.077129+0000 mgr.y (mgr.34107) 3711 : cluster [DBG] pgmap v5173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:48 smithi115 bash[45495]: cluster 2022-04-22T19:13:47.077129+0000 mgr.y (mgr.34107) 3711 : cluster [DBG] pgmap v5173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:50 smithi121 bash[36902]: cluster 2022-04-22T19:13:49.078157+0000 mgr.y (mgr.34107) 3712 : cluster [DBG] pgmap v5174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:50 smithi121 bash[36902]: audit 2022-04-22T19:13:49.808863+0000 mon.a (mon.0) 6560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:50.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:50 smithi115 bash[45495]: cluster 2022-04-22T19:13:49.078157+0000 mgr.y (mgr.34107) 3712 : cluster [DBG] pgmap v5174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:50.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:50 smithi115 bash[45495]: audit 2022-04-22T19:13:49.808863+0000 mon.a (mon.0) 6560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:50.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:50 smithi115 bash[46934]: cluster 2022-04-22T19:13:49.078157+0000 mgr.y (mgr.34107) 3712 : cluster [DBG] pgmap v5174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:50.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:50 smithi115 bash[46934]: audit 2022-04-22T19:13:49.808863+0000 mon.a (mon.0) 6560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:52 smithi115 bash[45495]: cluster 2022-04-22T19:13:51.079192+0000 mgr.y (mgr.34107) 3713 : cluster [DBG] pgmap v5175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:52 smithi115 bash[46934]: cluster 2022-04-22T19:13:51.079192+0000 mgr.y (mgr.34107) 3713 : cluster [DBG] pgmap v5175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:52.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:52.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:52 smithi121 bash[36902]: cluster 2022-04-22T19:13:51.079192+0000 mgr.y (mgr.34107) 3713 : cluster [DBG] pgmap v5175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:53.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:53 smithi115 bash[46934]: cluster 2022-04-22T19:13:53.080276+0000 mgr.y (mgr.34107) 3714 : cluster [DBG] pgmap v5176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:53.908 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:53 smithi115 bash[45495]: cluster 2022-04-22T19:13:53.080276+0000 mgr.y (mgr.34107) 3714 : cluster [DBG] pgmap v5176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:53 smithi121 bash[36902]: cluster 2022-04-22T19:13:53.080276+0000 mgr.y (mgr.34107) 3714 : cluster [DBG] pgmap v5176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:13:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:13:53.897Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:13:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:54 smithi121 bash[36902]: audit 2022-04-22T19:13:54.812005+0000 mon.a (mon.0) 6561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:54 smithi115 bash[45495]: audit 2022-04-22T19:13:54.812005+0000 mon.a (mon.0) 6561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:54 smithi115 bash[46934]: audit 2022-04-22T19:13:54.812005+0000 mon.a (mon.0) 6561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:13:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:55 smithi121 bash[36902]: cluster 2022-04-22T19:13:55.081298+0000 mgr.y (mgr.34107) 3715 : cluster [DBG] pgmap v5177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:55 smithi115 bash[45495]: cluster 2022-04-22T19:13:55.081298+0000 mgr.y (mgr.34107) 3715 : cluster [DBG] pgmap v5177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:55 smithi115 bash[46934]: cluster 2022-04-22T19:13:55.081298+0000 mgr.y (mgr.34107) 3715 : cluster [DBG] pgmap v5177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:13:58 smithi121 bash[36902]: cluster 2022-04-22T19:13:57.082582+0000 mgr.y (mgr.34107) 3716 : cluster [DBG] pgmap v5178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:13:58 smithi115 bash[45495]: cluster 2022-04-22T19:13:57.082582+0000 mgr.y (mgr.34107) 3716 : cluster [DBG] pgmap v5178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:13:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:13:58 smithi115 bash[46934]: cluster 2022-04-22T19:13:57.082582+0000 mgr.y (mgr.34107) 3716 : cluster [DBG] pgmap v5178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:00 smithi121 bash[36902]: cluster 2022-04-22T19:13:59.083681+0000 mgr.y (mgr.34107) 3717 : cluster [DBG] pgmap v5179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:00.541 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:00 smithi121 bash[36902]: audit 2022-04-22T19:13:59.815175+0000 mon.a (mon.0) 6562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:00 smithi115 bash[45495]: cluster 2022-04-22T19:13:59.083681+0000 mgr.y (mgr.34107) 3717 : cluster [DBG] pgmap v5179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:00 smithi115 bash[45495]: audit 2022-04-22T19:13:59.815175+0000 mon.a (mon.0) 6562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:00 smithi115 bash[46934]: cluster 2022-04-22T19:13:59.083681+0000 mgr.y (mgr.34107) 3717 : cluster [DBG] pgmap v5179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:00 smithi115 bash[46934]: audit 2022-04-22T19:13:59.815175+0000 mon.a (mon.0) 6562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:01.996 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:14:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:02 smithi115 bash[46934]: cluster 2022-04-22T19:14:01.084774+0000 mgr.y (mgr.34107) 3718 : cluster [DBG] pgmap v5180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:02.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:02 smithi115 bash[45495]: cluster 2022-04-22T19:14:01.084774+0000 mgr.y (mgr.34107) 3718 : cluster [DBG] pgmap v5180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:02.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:02.024Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:02.346 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:14:02.346 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (97m) 3m ago 110m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (110m) 4m ago 110m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (99m) 3m ago 111m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (98m) 4m ago 111m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (106m) 4m ago 116m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (106m) 3m ago 118m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (105m) 3m ago 118m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (105m) 4m ago 116m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:14:02.347 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (105m) 3m ago 116m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (98m) 3m ago 111m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (98m) 4m ago 111m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (103m) 3m ago 115m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (103m) 3m ago 115m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (102m) 3m ago 114m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (102m) 3m ago 114m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (101m) 4m ago 113m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (100m) 4m ago 113m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (100m) 4m ago 112m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:14:02.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (99m) 4m ago 112m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:14:02.349 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (97m) 4m ago 111m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:14:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:02 smithi121 bash[36902]: cluster 2022-04-22T19:14:01.084774+0000 mgr.y (mgr.34107) 3718 : cluster [DBG] pgmap v5180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:14:02.745 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:14:02.745 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:14:02.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:02.745 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:14:02.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:14:02.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:14:02.747 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:14:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:03 smithi121 bash[36902]: audit 2022-04-22T19:14:01.969954+0000 mgr.y (mgr.34107) 3719 : audit [DBG] from='client.49228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:03.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:03 smithi121 bash[36902]: audit 2022-04-22T19:14:02.734317+0000 mon.c (mon.1) 183 : audit [DBG] from='client.? 172.21.15.115:0/3453060496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:03 smithi115 bash[45495]: audit 2022-04-22T19:14:01.969954+0000 mgr.y (mgr.34107) 3719 : audit [DBG] from='client.49228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:03 smithi115 bash[45495]: audit 2022-04-22T19:14:02.734317+0000 mon.c (mon.1) 183 : audit [DBG] from='client.? 172.21.15.115:0/3453060496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:03 smithi115 bash[46934]: audit 2022-04-22T19:14:01.969954+0000 mgr.y (mgr.34107) 3719 : audit [DBG] from='client.49228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:03 smithi115 bash[46934]: audit 2022-04-22T19:14:02.734317+0000 mon.c (mon.1) 183 : audit [DBG] from='client.? 172.21.15.115:0/3453060496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:04.161 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:03.898Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:04 smithi121 bash[36902]: audit 2022-04-22T19:14:02.315379+0000 mgr.y (mgr.34107) 3720 : audit [DBG] from='client.39300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:04 smithi121 bash[36902]: cluster 2022-04-22T19:14:03.085860+0000 mgr.y (mgr.34107) 3721 : cluster [DBG] pgmap v5181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:04.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:04 smithi115 bash[45495]: audit 2022-04-22T19:14:02.315379+0000 mgr.y (mgr.34107) 3720 : audit [DBG] from='client.39300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:04.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:04 smithi115 bash[45495]: cluster 2022-04-22T19:14:03.085860+0000 mgr.y (mgr.34107) 3721 : cluster [DBG] pgmap v5181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:04 smithi115 bash[46934]: audit 2022-04-22T19:14:02.315379+0000 mgr.y (mgr.34107) 3720 : audit [DBG] from='client.39300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:04 smithi115 bash[46934]: cluster 2022-04-22T19:14:03.085860+0000 mgr.y (mgr.34107) 3721 : cluster [DBG] pgmap v5181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:05 smithi121 bash[36902]: audit 2022-04-22T19:14:04.818224+0000 mon.a (mon.0) 6563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:05 smithi115 bash[45495]: audit 2022-04-22T19:14:04.818224+0000 mon.a (mon.0) 6563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:05 smithi115 bash[46934]: audit 2022-04-22T19:14:04.818224+0000 mon.a (mon.0) 6563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:06 smithi121 bash[36902]: cluster 2022-04-22T19:14:05.086926+0000 mgr.y (mgr.34107) 3722 : cluster [DBG] pgmap v5182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:06.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:06 smithi115 bash[45495]: cluster 2022-04-22T19:14:05.086926+0000 mgr.y (mgr.34107) 3722 : cluster [DBG] pgmap v5182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:06 smithi115 bash[46934]: cluster 2022-04-22T19:14:05.086926+0000 mgr.y (mgr.34107) 3722 : cluster [DBG] pgmap v5182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:08 smithi121 bash[36902]: cluster 2022-04-22T19:14:07.088109+0000 mgr.y (mgr.34107) 3723 : cluster [DBG] pgmap v5183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:08 smithi115 bash[45495]: cluster 2022-04-22T19:14:07.088109+0000 mgr.y (mgr.34107) 3723 : cluster [DBG] pgmap v5183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:08 smithi115 bash[46934]: cluster 2022-04-22T19:14:07.088109+0000 mgr.y (mgr.34107) 3723 : cluster [DBG] pgmap v5183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:10 smithi121 bash[36902]: cluster 2022-04-22T19:14:09.089089+0000 mgr.y (mgr.34107) 3724 : cluster [DBG] pgmap v5184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:10 smithi121 bash[36902]: audit 2022-04-22T19:14:09.821310+0000 mon.a (mon.0) 6564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:10 smithi115 bash[45495]: cluster 2022-04-22T19:14:09.089089+0000 mgr.y (mgr.34107) 3724 : cluster [DBG] pgmap v5184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:10 smithi115 bash[45495]: audit 2022-04-22T19:14:09.821310+0000 mon.a (mon.0) 6564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:10 smithi115 bash[46934]: cluster 2022-04-22T19:14:09.089089+0000 mgr.y (mgr.34107) 3724 : cluster [DBG] pgmap v5184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:10 smithi115 bash[46934]: audit 2022-04-22T19:14:09.821310+0000 mon.a (mon.0) 6564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:12 smithi115 bash[45495]: cluster 2022-04-22T19:14:11.090077+0000 mgr.y (mgr.34107) 3725 : cluster [DBG] pgmap v5185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:12 smithi115 bash[46934]: cluster 2022-04-22T19:14:11.090077+0000 mgr.y (mgr.34107) 3725 : cluster [DBG] pgmap v5185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:12.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:12.025Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:12 smithi121 bash[36902]: cluster 2022-04-22T19:14:11.090077+0000 mgr.y (mgr.34107) 3725 : cluster [DBG] pgmap v5185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:13.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:13 smithi115 bash[45495]: cluster 2022-04-22T19:14:13.091171+0000 mgr.y (mgr.34107) 3726 : cluster [DBG] pgmap v5186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:13 smithi115 bash[46934]: cluster 2022-04-22T19:14:13.091171+0000 mgr.y (mgr.34107) 3726 : cluster [DBG] pgmap v5186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:13 smithi121 bash[36902]: cluster 2022-04-22T19:14:13.091171+0000 mgr.y (mgr.34107) 3726 : cluster [DBG] pgmap v5186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:13.899Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:14 smithi121 bash[36902]: audit 2022-04-22T19:14:14.495101+0000 mon.a (mon.0) 6565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:14:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:14 smithi121 bash[36902]: audit 2022-04-22T19:14:14.495426+0000 mon.a (mon.0) 6566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:14:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:14 smithi115 bash[45495]: audit 2022-04-22T19:14:14.495101+0000 mon.a (mon.0) 6565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:14:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:14 smithi115 bash[45495]: audit 2022-04-22T19:14:14.495426+0000 mon.a (mon.0) 6566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:14:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:14 smithi115 bash[46934]: audit 2022-04-22T19:14:14.495101+0000 mon.a (mon.0) 6565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:14:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:14 smithi115 bash[46934]: audit 2022-04-22T19:14:14.495426+0000 mon.a (mon.0) 6566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:14:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:15 smithi121 bash[36902]: audit 2022-04-22T19:14:14.824133+0000 mon.a (mon.0) 6567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:15 smithi121 bash[36902]: cluster 2022-04-22T19:14:15.092308+0000 mgr.y (mgr.34107) 3727 : cluster [DBG] pgmap v5187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:15 smithi115 bash[45495]: audit 2022-04-22T19:14:14.824133+0000 mon.a (mon.0) 6567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:15 smithi115 bash[45495]: cluster 2022-04-22T19:14:15.092308+0000 mgr.y (mgr.34107) 3727 : cluster [DBG] pgmap v5187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:15 smithi115 bash[46934]: audit 2022-04-22T19:14:14.824133+0000 mon.a (mon.0) 6567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:15 smithi115 bash[46934]: cluster 2022-04-22T19:14:15.092308+0000 mgr.y (mgr.34107) 3727 : cluster [DBG] pgmap v5187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:18 smithi121 bash[36902]: cluster 2022-04-22T19:14:17.093433+0000 mgr.y (mgr.34107) 3728 : cluster [DBG] pgmap v5188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:18 smithi115 bash[45495]: cluster 2022-04-22T19:14:17.093433+0000 mgr.y (mgr.34107) 3728 : cluster [DBG] pgmap v5188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:18.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:18 smithi115 bash[46934]: cluster 2022-04-22T19:14:17.093433+0000 mgr.y (mgr.34107) 3728 : cluster [DBG] pgmap v5188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:20 smithi121 bash[36902]: cluster 2022-04-22T19:14:19.094444+0000 mgr.y (mgr.34107) 3729 : cluster [DBG] pgmap v5189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:20 smithi121 bash[36902]: audit 2022-04-22T19:14:19.827522+0000 mon.a (mon.0) 6568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:20 smithi115 bash[46934]: cluster 2022-04-22T19:14:19.094444+0000 mgr.y (mgr.34107) 3729 : cluster [DBG] pgmap v5189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:20.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:20 smithi115 bash[46934]: audit 2022-04-22T19:14:19.827522+0000 mon.a (mon.0) 6568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:20 smithi115 bash[45495]: cluster 2022-04-22T19:14:19.094444+0000 mgr.y (mgr.34107) 3729 : cluster [DBG] pgmap v5189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:20 smithi115 bash[45495]: audit 2022-04-22T19:14:19.827522+0000 mon.a (mon.0) 6568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:22.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:22 smithi115 bash[45495]: cluster 2022-04-22T19:14:21.095512+0000 mgr.y (mgr.34107) 3730 : cluster [DBG] pgmap v5190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:22.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:22 smithi115 bash[46934]: cluster 2022-04-22T19:14:21.095512+0000 mgr.y (mgr.34107) 3730 : cluster [DBG] pgmap v5190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:22.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:22.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:22 smithi121 bash[36902]: cluster 2022-04-22T19:14:21.095512+0000 mgr.y (mgr.34107) 3730 : cluster [DBG] pgmap v5190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:23.913 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:23.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:23.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:23 smithi115 bash[45495]: cluster 2022-04-22T19:14:23.096543+0000 mgr.y (mgr.34107) 3731 : cluster [DBG] pgmap v5191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:23.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:23 smithi115 bash[46934]: cluster 2022-04-22T19:14:23.096543+0000 mgr.y (mgr.34107) 3731 : cluster [DBG] pgmap v5191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:23 smithi121 bash[36902]: cluster 2022-04-22T19:14:23.096543+0000 mgr.y (mgr.34107) 3731 : cluster [DBG] pgmap v5191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:24 smithi121 bash[36902]: audit 2022-04-22T19:14:24.830476+0000 mon.a (mon.0) 6569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:24 smithi115 bash[46934]: audit 2022-04-22T19:14:24.830476+0000 mon.a (mon.0) 6569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:25.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:24 smithi115 bash[45495]: audit 2022-04-22T19:14:24.830476+0000 mon.a (mon.0) 6569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:25 smithi121 bash[36902]: cluster 2022-04-22T19:14:25.097733+0000 mgr.y (mgr.34107) 3732 : cluster [DBG] pgmap v5192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:25 smithi115 bash[45495]: cluster 2022-04-22T19:14:25.097733+0000 mgr.y (mgr.34107) 3732 : cluster [DBG] pgmap v5192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:25 smithi115 bash[46934]: cluster 2022-04-22T19:14:25.097733+0000 mgr.y (mgr.34107) 3732 : cluster [DBG] pgmap v5192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:28 smithi121 bash[36902]: cluster 2022-04-22T19:14:27.098944+0000 mgr.y (mgr.34107) 3733 : cluster [DBG] pgmap v5193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:28 smithi115 bash[46934]: cluster 2022-04-22T19:14:27.098944+0000 mgr.y (mgr.34107) 3733 : cluster [DBG] pgmap v5193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:28 smithi115 bash[45495]: cluster 2022-04-22T19:14:27.098944+0000 mgr.y (mgr.34107) 3733 : cluster [DBG] pgmap v5193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:30 smithi121 bash[36902]: cluster 2022-04-22T19:14:29.100034+0000 mgr.y (mgr.34107) 3734 : cluster [DBG] pgmap v5194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:30 smithi121 bash[36902]: audit 2022-04-22T19:14:29.833460+0000 mon.a (mon.0) 6570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:30 smithi115 bash[45495]: cluster 2022-04-22T19:14:29.100034+0000 mgr.y (mgr.34107) 3734 : cluster [DBG] pgmap v5194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:30 smithi115 bash[45495]: audit 2022-04-22T19:14:29.833460+0000 mon.a (mon.0) 6570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:30 smithi115 bash[46934]: cluster 2022-04-22T19:14:29.100034+0000 mgr.y (mgr.34107) 3734 : cluster [DBG] pgmap v5194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:30 smithi115 bash[46934]: audit 2022-04-22T19:14:29.833460+0000 mon.a (mon.0) 6570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[45495]: audit 2022-04-22T19:14:31.005891+0000 mon.a (mon.0) 6571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:14:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[45495]: audit 2022-04-22T19:14:31.009010+0000 mon.a (mon.0) 6572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:14:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[45495]: audit 2022-04-22T19:14:31.010893+0000 mon.a (mon.0) 6573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:14:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[46934]: audit 2022-04-22T19:14:31.005891+0000 mon.a (mon.0) 6571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:14:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[46934]: audit 2022-04-22T19:14:31.009010+0000 mon.a (mon.0) 6572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:14:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:31 smithi115 bash[46934]: audit 2022-04-22T19:14:31.010893+0000 mon.a (mon.0) 6573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:14:31.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:31 smithi121 bash[36902]: audit 2022-04-22T19:14:31.005891+0000 mon.a (mon.0) 6571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:14:31.392 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:31 smithi121 bash[36902]: audit 2022-04-22T19:14:31.009010+0000 mon.a (mon.0) 6572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:14:31.392 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:31 smithi121 bash[36902]: audit 2022-04-22T19:14:31.010893+0000 mon.a (mon.0) 6573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:14:32.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:32.028Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:32 smithi115 bash[45495]: cluster 2022-04-22T19:14:31.101108+0000 mgr.y (mgr.34107) 3735 : cluster [DBG] pgmap v5195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:32.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:32 smithi115 bash[45495]: audit 2022-04-22T19:14:31.403419+0000 mon.a (mon.0) 6574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:14:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:32 smithi115 bash[46934]: cluster 2022-04-22T19:14:31.101108+0000 mgr.y (mgr.34107) 3735 : cluster [DBG] pgmap v5195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:32 smithi115 bash[46934]: audit 2022-04-22T19:14:31.403419+0000 mon.a (mon.0) 6574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:14:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:32 smithi121 bash[36902]: cluster 2022-04-22T19:14:31.101108+0000 mgr.y (mgr.34107) 3735 : cluster [DBG] pgmap v5195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:32 smithi121 bash[36902]: audit 2022-04-22T19:14:31.403419+0000 mon.a (mon.0) 6574 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:14:33.123 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:14:33.463 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (97m) 4m ago 111m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (111m) 5m ago 111m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (99m) 4m ago 112m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (99m) 5m ago 112m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (106m) 5m ago 116m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (107m) 4m ago 118m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (106m) 4m ago 119m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:14:33.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (105m) 5m ago 117m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (105m) 4m ago 117m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (98m) 4m ago 111m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (98m) 5m ago 111m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (104m) 4m ago 116m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (103m) 4m ago 115m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (103m) 4m ago 115m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (102m) 4m ago 114m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (102m) 5m ago 114m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (101m) 5m ago 113m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:14:33.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (100m) 5m ago 113m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:14:33.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (100m) 5m ago 112m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:14:33.466 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (98m) 5m ago 111m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:14:33.859 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:33.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:14:33.861 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:14:33.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:33 smithi115 bash[46934]: audit 2022-04-22T19:14:33.097340+0000 mgr.y (mgr.34107) 3736 : audit [DBG] from='client.39312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:33.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:33 smithi115 bash[46934]: cluster 2022-04-22T19:14:33.101991+0000 mgr.y (mgr.34107) 3737 : cluster [DBG] pgmap v5196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:33.912 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:33 smithi115 bash[45495]: audit 2022-04-22T19:14:33.097340+0000 mgr.y (mgr.34107) 3736 : audit [DBG] from='client.39312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:33.912 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:33 smithi115 bash[45495]: cluster 2022-04-22T19:14:33.101991+0000 mgr.y (mgr.34107) 3737 : cluster [DBG] pgmap v5196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:33 smithi121 bash[36902]: audit 2022-04-22T19:14:33.097340+0000 mgr.y (mgr.34107) 3736 : audit [DBG] from='client.39312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:33 smithi121 bash[36902]: cluster 2022-04-22T19:14:33.101991+0000 mgr.y (mgr.34107) 3737 : cluster [DBG] pgmap v5196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:33.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:34 smithi121 bash[36902]: audit 2022-04-22T19:14:33.433588+0000 mgr.y (mgr.34107) 3738 : audit [DBG] from='client.49252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:34 smithi121 bash[36902]: audit 2022-04-22T19:14:33.848669+0000 mon.a (mon.0) 6575 : audit [DBG] from='client.? 172.21.15.115:0/1900664566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:34 smithi115 bash[46934]: audit 2022-04-22T19:14:33.433588+0000 mgr.y (mgr.34107) 3738 : audit [DBG] from='client.49252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:34 smithi115 bash[46934]: audit 2022-04-22T19:14:33.848669+0000 mon.a (mon.0) 6575 : audit [DBG] from='client.? 172.21.15.115:0/1900664566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:35.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:34 smithi115 bash[45495]: audit 2022-04-22T19:14:33.433588+0000 mgr.y (mgr.34107) 3738 : audit [DBG] from='client.49252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:14:35.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:34 smithi115 bash[45495]: audit 2022-04-22T19:14:33.848669+0000 mon.a (mon.0) 6575 : audit [DBG] from='client.? 172.21.15.115:0/1900664566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:14:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:35 smithi121 bash[36902]: audit 2022-04-22T19:14:34.836574+0000 mon.a (mon.0) 6576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:35 smithi121 bash[36902]: cluster 2022-04-22T19:14:35.103013+0000 mgr.y (mgr.34107) 3739 : cluster [DBG] pgmap v5197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:35 smithi115 bash[45495]: audit 2022-04-22T19:14:34.836574+0000 mon.a (mon.0) 6576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:35 smithi115 bash[45495]: cluster 2022-04-22T19:14:35.103013+0000 mgr.y (mgr.34107) 3739 : cluster [DBG] pgmap v5197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:35 smithi115 bash[46934]: audit 2022-04-22T19:14:34.836574+0000 mon.a (mon.0) 6576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:35 smithi115 bash[46934]: cluster 2022-04-22T19:14:35.103013+0000 mgr.y (mgr.34107) 3739 : cluster [DBG] pgmap v5197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:38 smithi121 bash[36902]: cluster 2022-04-22T19:14:37.104201+0000 mgr.y (mgr.34107) 3740 : cluster [DBG] pgmap v5198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:38 smithi115 bash[45495]: cluster 2022-04-22T19:14:37.104201+0000 mgr.y (mgr.34107) 3740 : cluster [DBG] pgmap v5198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:38 smithi115 bash[46934]: cluster 2022-04-22T19:14:37.104201+0000 mgr.y (mgr.34107) 3740 : cluster [DBG] pgmap v5198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:40 smithi121 bash[36902]: cluster 2022-04-22T19:14:39.105241+0000 mgr.y (mgr.34107) 3741 : cluster [DBG] pgmap v5199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:40 smithi121 bash[36902]: audit 2022-04-22T19:14:39.839712+0000 mon.a (mon.0) 6577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:40 smithi115 bash[45495]: cluster 2022-04-22T19:14:39.105241+0000 mgr.y (mgr.34107) 3741 : cluster [DBG] pgmap v5199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:40 smithi115 bash[45495]: audit 2022-04-22T19:14:39.839712+0000 mon.a (mon.0) 6577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:40 smithi115 bash[46934]: cluster 2022-04-22T19:14:39.105241+0000 mgr.y (mgr.34107) 3741 : cluster [DBG] pgmap v5199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:40 smithi115 bash[46934]: audit 2022-04-22T19:14:39.839712+0000 mon.a (mon.0) 6577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:42 smithi115 bash[45495]: cluster 2022-04-22T19:14:41.106290+0000 mgr.y (mgr.34107) 3742 : cluster [DBG] pgmap v5200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:42.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:42 smithi115 bash[46934]: cluster 2022-04-22T19:14:41.106290+0000 mgr.y (mgr.34107) 3742 : cluster [DBG] pgmap v5200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:42 smithi121 bash[36902]: cluster 2022-04-22T19:14:41.106290+0000 mgr.y (mgr.34107) 3742 : cluster [DBG] pgmap v5200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:43.911 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:43 smithi115 bash[45495]: cluster 2022-04-22T19:14:43.107297+0000 mgr.y (mgr.34107) 3743 : cluster [DBG] pgmap v5201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:43.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:43 smithi115 bash[46934]: cluster 2022-04-22T19:14:43.107297+0000 mgr.y (mgr.34107) 3743 : cluster [DBG] pgmap v5201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:43 smithi121 bash[36902]: cluster 2022-04-22T19:14:43.107297+0000 mgr.y (mgr.34107) 3743 : cluster [DBG] pgmap v5201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:43.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:44 smithi121 bash[36902]: audit 2022-04-22T19:14:44.843017+0000 mon.a (mon.0) 6578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:45.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:44 smithi115 bash[45495]: audit 2022-04-22T19:14:44.843017+0000 mon.a (mon.0) 6578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:45.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:44 smithi115 bash[46934]: audit 2022-04-22T19:14:44.843017+0000 mon.a (mon.0) 6578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:45 smithi121 bash[36902]: cluster 2022-04-22T19:14:45.108458+0000 mgr.y (mgr.34107) 3744 : cluster [DBG] pgmap v5202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:45 smithi115 bash[45495]: cluster 2022-04-22T19:14:45.108458+0000 mgr.y (mgr.34107) 3744 : cluster [DBG] pgmap v5202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:45 smithi115 bash[46934]: cluster 2022-04-22T19:14:45.108458+0000 mgr.y (mgr.34107) 3744 : cluster [DBG] pgmap v5202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:48 smithi121 bash[36902]: cluster 2022-04-22T19:14:47.109727+0000 mgr.y (mgr.34107) 3745 : cluster [DBG] pgmap v5203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:48 smithi115 bash[45495]: cluster 2022-04-22T19:14:47.109727+0000 mgr.y (mgr.34107) 3745 : cluster [DBG] pgmap v5203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:48.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:48 smithi115 bash[46934]: cluster 2022-04-22T19:14:47.109727+0000 mgr.y (mgr.34107) 3745 : cluster [DBG] pgmap v5203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:50 smithi121 bash[36902]: cluster 2022-04-22T19:14:49.110846+0000 mgr.y (mgr.34107) 3746 : cluster [DBG] pgmap v5204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:50 smithi121 bash[36902]: audit 2022-04-22T19:14:49.846326+0000 mon.a (mon.0) 6579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:50 smithi115 bash[45495]: cluster 2022-04-22T19:14:49.110846+0000 mgr.y (mgr.34107) 3746 : cluster [DBG] pgmap v5204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:50 smithi115 bash[45495]: audit 2022-04-22T19:14:49.846326+0000 mon.a (mon.0) 6579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:50 smithi115 bash[46934]: cluster 2022-04-22T19:14:49.110846+0000 mgr.y (mgr.34107) 3746 : cluster [DBG] pgmap v5204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:50 smithi115 bash[46934]: audit 2022-04-22T19:14:49.846326+0000 mon.a (mon.0) 6579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:52 smithi115 bash[45495]: cluster 2022-04-22T19:14:51.111874+0000 mgr.y (mgr.34107) 3747 : cluster [DBG] pgmap v5205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:52.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:52.029Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:52.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:52 smithi115 bash[46934]: cluster 2022-04-22T19:14:51.111874+0000 mgr.y (mgr.34107) 3747 : cluster [DBG] pgmap v5205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:52 smithi121 bash[36902]: cluster 2022-04-22T19:14:51.111874+0000 mgr.y (mgr.34107) 3747 : cluster [DBG] pgmap v5205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:53.912 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:53 smithi115 bash[45495]: cluster 2022-04-22T19:14:53.112850+0000 mgr.y (mgr.34107) 3748 : cluster [DBG] pgmap v5206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:53.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:53 smithi115 bash[46934]: cluster 2022-04-22T19:14:53.112850+0000 mgr.y (mgr.34107) 3748 : cluster [DBG] pgmap v5206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:53 smithi121 bash[36902]: cluster 2022-04-22T19:14:53.112850+0000 mgr.y (mgr.34107) 3748 : cluster [DBG] pgmap v5206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:54.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:14:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:14:53.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:14:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:54 smithi121 bash[36902]: audit 2022-04-22T19:14:54.849534+0000 mon.a (mon.0) 6580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:54 smithi115 bash[45495]: audit 2022-04-22T19:14:54.849534+0000 mon.a (mon.0) 6580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:54 smithi115 bash[46934]: audit 2022-04-22T19:14:54.849534+0000 mon.a (mon.0) 6580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:14:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:55 smithi121 bash[36902]: cluster 2022-04-22T19:14:55.113911+0000 mgr.y (mgr.34107) 3749 : cluster [DBG] pgmap v5207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:55 smithi115 bash[46934]: cluster 2022-04-22T19:14:55.113911+0000 mgr.y (mgr.34107) 3749 : cluster [DBG] pgmap v5207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:55 smithi115 bash[45495]: cluster 2022-04-22T19:14:55.113911+0000 mgr.y (mgr.34107) 3749 : cluster [DBG] pgmap v5207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:14:58 smithi121 bash[36902]: cluster 2022-04-22T19:14:57.115102+0000 mgr.y (mgr.34107) 3750 : cluster [DBG] pgmap v5208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:14:58 smithi115 bash[45495]: cluster 2022-04-22T19:14:57.115102+0000 mgr.y (mgr.34107) 3750 : cluster [DBG] pgmap v5208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:14:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:14:58 smithi115 bash[46934]: cluster 2022-04-22T19:14:57.115102+0000 mgr.y (mgr.34107) 3750 : cluster [DBG] pgmap v5208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:00 smithi121 bash[36902]: cluster 2022-04-22T19:14:59.116149+0000 mgr.y (mgr.34107) 3751 : cluster [DBG] pgmap v5209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:00 smithi121 bash[36902]: audit 2022-04-22T19:14:59.852530+0000 mon.a (mon.0) 6581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:00 smithi115 bash[45495]: cluster 2022-04-22T19:14:59.116149+0000 mgr.y (mgr.34107) 3751 : cluster [DBG] pgmap v5209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:00 smithi115 bash[45495]: audit 2022-04-22T19:14:59.852530+0000 mon.a (mon.0) 6581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:00 smithi115 bash[46934]: cluster 2022-04-22T19:14:59.116149+0000 mgr.y (mgr.34107) 3751 : cluster [DBG] pgmap v5209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:00 smithi115 bash[46934]: audit 2022-04-22T19:14:59.852530+0000 mon.a (mon.0) 6581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:02.191 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:02.029Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:02 smithi121 bash[36902]: cluster 2022-04-22T19:15:01.117302+0000 mgr.y (mgr.34107) 3752 : cluster [DBG] pgmap v5210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:02.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:02 smithi115 bash[45495]: cluster 2022-04-22T19:15:01.117302+0000 mgr.y (mgr.34107) 3752 : cluster [DBG] pgmap v5210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:02.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:02 smithi115 bash[46934]: cluster 2022-04-22T19:15:01.117302+0000 mgr.y (mgr.34107) 3752 : cluster [DBG] pgmap v5210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:03.911 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:03 smithi115 bash[45495]: cluster 2022-04-22T19:15:03.118501+0000 mgr.y (mgr.34107) 3753 : cluster [DBG] pgmap v5211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:03.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:03 smithi115 bash[46934]: cluster 2022-04-22T19:15:03.118501+0000 mgr.y (mgr.34107) 3753 : cluster [DBG] pgmap v5211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:03 smithi121 bash[36902]: cluster 2022-04-22T19:15:03.118501+0000 mgr.y (mgr.34107) 3753 : cluster [DBG] pgmap v5211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:04.234 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:15:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:03.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:04.580 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:15:04.580 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (98m) 4m ago 111m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:15:04.580 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (111m) 5m ago 111m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:15:04.580 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (100m) 4m ago 112m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:15:04.580 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (99m) 5m ago 112m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (107m) 5m ago 117m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (108m) 4m ago 119m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (107m) 4m ago 119m 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (106m) 5m ago 117m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (106m) 4m ago 117m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (99m) 4m ago 112m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (99m) 5m ago 112m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (104m) 4m ago 116m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:15:04.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (104m) 4m ago 116m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (103m) 4m ago 115m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (103m) 4m ago 115m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (102m) 5m ago 114m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (101m) 5m ago 114m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (101m) 5m ago 113m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (100m) 5m ago 113m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:15:04.582 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (98m) 5m ago 112m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:15:04.982 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:15:04.982 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:15:04.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:15:04.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:15:04.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:15:04.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:04.984 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:15:04.984 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:15:04.984 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:15:04.984 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:15:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:04 smithi121 bash[36902]: audit 2022-04-22T19:15:04.208123+0000 mgr.y (mgr.34107) 3754 : audit [DBG] from='client.49264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:05.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:04 smithi115 bash[45495]: audit 2022-04-22T19:15:04.208123+0000 mgr.y (mgr.34107) 3754 : audit [DBG] from='client.49264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:05.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:04 smithi115 bash[46934]: audit 2022-04-22T19:15:04.208123+0000 mgr.y (mgr.34107) 3754 : audit [DBG] from='client.49264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:05 smithi121 bash[36902]: audit 2022-04-22T19:15:04.548672+0000 mgr.y (mgr.34107) 3755 : audit [DBG] from='client.49270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:06.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:05 smithi121 bash[36902]: audit 2022-04-22T19:15:04.855702+0000 mon.a (mon.0) 6582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:06.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:05 smithi121 bash[36902]: audit 2022-04-22T19:15:04.971004+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.115:0/937732413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:06.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:05 smithi121 bash[36902]: cluster 2022-04-22T19:15:05.119630+0000 mgr.y (mgr.34107) 3756 : cluster [DBG] pgmap v5212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:06.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[45495]: audit 2022-04-22T19:15:04.548672+0000 mgr.y (mgr.34107) 3755 : audit [DBG] from='client.49270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:06.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[45495]: audit 2022-04-22T19:15:04.855702+0000 mon.a (mon.0) 6582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[45495]: audit 2022-04-22T19:15:04.971004+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.115:0/937732413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[45495]: cluster 2022-04-22T19:15:05.119630+0000 mgr.y (mgr.34107) 3756 : cluster [DBG] pgmap v5212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[46934]: audit 2022-04-22T19:15:04.548672+0000 mgr.y (mgr.34107) 3755 : audit [DBG] from='client.49270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[46934]: audit 2022-04-22T19:15:04.855702+0000 mon.a (mon.0) 6582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[46934]: audit 2022-04-22T19:15:04.971004+0000 mon.c (mon.1) 184 : audit [DBG] from='client.? 172.21.15.115:0/937732413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:06.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:05 smithi115 bash[46934]: cluster 2022-04-22T19:15:05.119630+0000 mgr.y (mgr.34107) 3756 : cluster [DBG] pgmap v5212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:08 smithi121 bash[36902]: cluster 2022-04-22T19:15:07.120849+0000 mgr.y (mgr.34107) 3757 : cluster [DBG] pgmap v5213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:08 smithi115 bash[45495]: cluster 2022-04-22T19:15:07.120849+0000 mgr.y (mgr.34107) 3757 : cluster [DBG] pgmap v5213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:08 smithi115 bash[46934]: cluster 2022-04-22T19:15:07.120849+0000 mgr.y (mgr.34107) 3757 : cluster [DBG] pgmap v5213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:10 smithi121 bash[36902]: cluster 2022-04-22T19:15:09.122038+0000 mgr.y (mgr.34107) 3758 : cluster [DBG] pgmap v5214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:10 smithi121 bash[36902]: audit 2022-04-22T19:15:09.858772+0000 mon.a (mon.0) 6583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:10 smithi115 bash[45495]: cluster 2022-04-22T19:15:09.122038+0000 mgr.y (mgr.34107) 3758 : cluster [DBG] pgmap v5214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:10 smithi115 bash[45495]: audit 2022-04-22T19:15:09.858772+0000 mon.a (mon.0) 6583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:10 smithi115 bash[46934]: cluster 2022-04-22T19:15:09.122038+0000 mgr.y (mgr.34107) 3758 : cluster [DBG] pgmap v5214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:10 smithi115 bash[46934]: audit 2022-04-22T19:15:09.858772+0000 mon.a (mon.0) 6583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:12 smithi115 bash[46934]: cluster 2022-04-22T19:15:11.123068+0000 mgr.y (mgr.34107) 3759 : cluster [DBG] pgmap v5215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:12.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:12 smithi115 bash[45495]: cluster 2022-04-22T19:15:11.123068+0000 mgr.y (mgr.34107) 3759 : cluster [DBG] pgmap v5215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:12.304 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:12.030Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:12 smithi121 bash[36902]: cluster 2022-04-22T19:15:11.123068+0000 mgr.y (mgr.34107) 3759 : cluster [DBG] pgmap v5215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:13.914 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:13 smithi115 bash[45495]: cluster 2022-04-22T19:15:13.124069+0000 mgr.y (mgr.34107) 3760 : cluster [DBG] pgmap v5216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:13.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:13 smithi115 bash[46934]: cluster 2022-04-22T19:15:13.124069+0000 mgr.y (mgr.34107) 3760 : cluster [DBG] pgmap v5216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:13 smithi121 bash[36902]: cluster 2022-04-22T19:15:13.124069+0000 mgr.y (mgr.34107) 3760 : cluster [DBG] pgmap v5216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:13.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:14 smithi121 bash[36902]: audit 2022-04-22T19:15:14.508850+0000 mon.a (mon.0) 6584 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:15:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:14 smithi121 bash[36902]: audit 2022-04-22T19:15:14.509275+0000 mon.a (mon.0) 6585 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:15:15.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:14 smithi115 bash[45495]: audit 2022-04-22T19:15:14.508850+0000 mon.a (mon.0) 6584 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:15:15.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:14 smithi115 bash[45495]: audit 2022-04-22T19:15:14.509275+0000 mon.a (mon.0) 6585 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:15:15.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:14 smithi115 bash[46934]: audit 2022-04-22T19:15:14.508850+0000 mon.a (mon.0) 6584 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:15:15.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:14 smithi115 bash[46934]: audit 2022-04-22T19:15:14.509275+0000 mon.a (mon.0) 6585 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:15:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:15 smithi121 bash[36902]: audit 2022-04-22T19:15:14.862074+0000 mon.a (mon.0) 6586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:15 smithi121 bash[36902]: cluster 2022-04-22T19:15:15.125165+0000 mgr.y (mgr.34107) 3761 : cluster [DBG] pgmap v5217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:15 smithi115 bash[45495]: audit 2022-04-22T19:15:14.862074+0000 mon.a (mon.0) 6586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:15 smithi115 bash[45495]: cluster 2022-04-22T19:15:15.125165+0000 mgr.y (mgr.34107) 3761 : cluster [DBG] pgmap v5217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:15 smithi115 bash[46934]: audit 2022-04-22T19:15:14.862074+0000 mon.a (mon.0) 6586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:15 smithi115 bash[46934]: cluster 2022-04-22T19:15:15.125165+0000 mgr.y (mgr.34107) 3761 : cluster [DBG] pgmap v5217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:18 smithi121 bash[36902]: cluster 2022-04-22T19:15:17.126195+0000 mgr.y (mgr.34107) 3762 : cluster [DBG] pgmap v5218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:18 smithi115 bash[46934]: cluster 2022-04-22T19:15:17.126195+0000 mgr.y (mgr.34107) 3762 : cluster [DBG] pgmap v5218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:18 smithi115 bash[45495]: cluster 2022-04-22T19:15:17.126195+0000 mgr.y (mgr.34107) 3762 : cluster [DBG] pgmap v5218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:20 smithi121 bash[36902]: cluster 2022-04-22T19:15:19.127240+0000 mgr.y (mgr.34107) 3763 : cluster [DBG] pgmap v5219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:20 smithi121 bash[36902]: audit 2022-04-22T19:15:19.865199+0000 mon.a (mon.0) 6587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:20 smithi115 bash[45495]: cluster 2022-04-22T19:15:19.127240+0000 mgr.y (mgr.34107) 3763 : cluster [DBG] pgmap v5219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:20 smithi115 bash[45495]: audit 2022-04-22T19:15:19.865199+0000 mon.a (mon.0) 6587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:20 smithi115 bash[46934]: cluster 2022-04-22T19:15:19.127240+0000 mgr.y (mgr.34107) 3763 : cluster [DBG] pgmap v5219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:20 smithi115 bash[46934]: audit 2022-04-22T19:15:19.865199+0000 mon.a (mon.0) 6587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:22.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:22 smithi115 bash[45495]: cluster 2022-04-22T19:15:21.128304+0000 mgr.y (mgr.34107) 3764 : cluster [DBG] pgmap v5220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:22.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:22.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:22.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:22 smithi115 bash[46934]: cluster 2022-04-22T19:15:21.128304+0000 mgr.y (mgr.34107) 3764 : cluster [DBG] pgmap v5220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:22 smithi121 bash[36902]: cluster 2022-04-22T19:15:21.128304+0000 mgr.y (mgr.34107) 3764 : cluster [DBG] pgmap v5220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:23.915 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:23 smithi115 bash[45495]: cluster 2022-04-22T19:15:23.129364+0000 mgr.y (mgr.34107) 3765 : cluster [DBG] pgmap v5221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:23.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:23 smithi115 bash[46934]: cluster 2022-04-22T19:15:23.129364+0000 mgr.y (mgr.34107) 3765 : cluster [DBG] pgmap v5221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:23 smithi121 bash[36902]: cluster 2022-04-22T19:15:23.129364+0000 mgr.y (mgr.34107) 3765 : cluster [DBG] pgmap v5221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:23.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:24 smithi121 bash[36902]: audit 2022-04-22T19:15:24.868325+0000 mon.a (mon.0) 6588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:24 smithi115 bash[45495]: audit 2022-04-22T19:15:24.868325+0000 mon.a (mon.0) 6588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:24 smithi115 bash[46934]: audit 2022-04-22T19:15:24.868325+0000 mon.a (mon.0) 6588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:25 smithi121 bash[36902]: cluster 2022-04-22T19:15:25.130465+0000 mgr.y (mgr.34107) 3766 : cluster [DBG] pgmap v5222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:25 smithi115 bash[45495]: cluster 2022-04-22T19:15:25.130465+0000 mgr.y (mgr.34107) 3766 : cluster [DBG] pgmap v5222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:25 smithi115 bash[46934]: cluster 2022-04-22T19:15:25.130465+0000 mgr.y (mgr.34107) 3766 : cluster [DBG] pgmap v5222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:28 smithi121 bash[36902]: cluster 2022-04-22T19:15:27.131510+0000 mgr.y (mgr.34107) 3767 : cluster [DBG] pgmap v5223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:28 smithi115 bash[46934]: cluster 2022-04-22T19:15:27.131510+0000 mgr.y (mgr.34107) 3767 : cluster [DBG] pgmap v5223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:28 smithi115 bash[45495]: cluster 2022-04-22T19:15:27.131510+0000 mgr.y (mgr.34107) 3767 : cluster [DBG] pgmap v5223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:30 smithi121 bash[36902]: cluster 2022-04-22T19:15:29.132496+0000 mgr.y (mgr.34107) 3768 : cluster [DBG] pgmap v5224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:30 smithi121 bash[36902]: audit 2022-04-22T19:15:29.871510+0000 mon.a (mon.0) 6589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:30 smithi115 bash[45495]: cluster 2022-04-22T19:15:29.132496+0000 mgr.y (mgr.34107) 3768 : cluster [DBG] pgmap v5224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:30 smithi115 bash[45495]: audit 2022-04-22T19:15:29.871510+0000 mon.a (mon.0) 6589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:30 smithi115 bash[46934]: cluster 2022-04-22T19:15:29.132496+0000 mgr.y (mgr.34107) 3768 : cluster [DBG] pgmap v5224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:30 smithi115 bash[46934]: audit 2022-04-22T19:15:29.871510+0000 mon.a (mon.0) 6589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:32.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:32.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:32.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[45495]: cluster 2022-04-22T19:15:31.133550+0000 mgr.y (mgr.34107) 3769 : cluster [DBG] pgmap v5225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:32.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[45495]: audit 2022-04-22T19:15:31.412728+0000 mon.a (mon.0) 6590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:15:32.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[45495]: audit 2022-04-22T19:15:31.415714+0000 mon.a (mon.0) 6591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:15:32.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[45495]: audit 2022-04-22T19:15:31.417504+0000 mon.a (mon.0) 6592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:15:32.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[45495]: audit 2022-04-22T19:15:31.777553+0000 mon.a (mon.0) 6593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:15:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[46934]: cluster 2022-04-22T19:15:31.133550+0000 mgr.y (mgr.34107) 3769 : cluster [DBG] pgmap v5225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[46934]: audit 2022-04-22T19:15:31.412728+0000 mon.a (mon.0) 6590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:15:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[46934]: audit 2022-04-22T19:15:31.415714+0000 mon.a (mon.0) 6591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:15:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[46934]: audit 2022-04-22T19:15:31.417504+0000 mon.a (mon.0) 6592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:15:32.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:32 smithi115 bash[46934]: audit 2022-04-22T19:15:31.777553+0000 mon.a (mon.0) 6593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:32 smithi121 bash[36902]: cluster 2022-04-22T19:15:31.133550+0000 mgr.y (mgr.34107) 3769 : cluster [DBG] pgmap v5225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:32 smithi121 bash[36902]: audit 2022-04-22T19:15:31.412728+0000 mon.a (mon.0) 6590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:32 smithi121 bash[36902]: audit 2022-04-22T19:15:31.415714+0000 mon.a (mon.0) 6591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:32 smithi121 bash[36902]: audit 2022-04-22T19:15:31.417504+0000 mon.a (mon.0) 6592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:32 smithi121 bash[36902]: audit 2022-04-22T19:15:31.777553+0000 mon.a (mon.0) 6593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:15:33.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:33 smithi115 bash[46934]: cluster 2022-04-22T19:15:33.134599+0000 mgr.y (mgr.34107) 3770 : cluster [DBG] pgmap v5226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:33.915 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:33 smithi115 bash[45495]: cluster 2022-04-22T19:15:33.134599+0000 mgr.y (mgr.34107) 3770 : cluster [DBG] pgmap v5226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:33 smithi121 bash[36902]: cluster 2022-04-22T19:15:33.134599+0000 mgr.y (mgr.34107) 3770 : cluster [DBG] pgmap v5226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:33.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:34 smithi121 bash[36902]: audit 2022-04-22T19:15:34.874784+0000 mon.a (mon.0) 6594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:35.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:34 smithi115 bash[46934]: audit 2022-04-22T19:15:34.874784+0000 mon.a (mon.0) 6594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:35.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:34 smithi115 bash[45495]: audit 2022-04-22T19:15:34.874784+0000 mon.a (mon.0) 6594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:35.362 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (98m) 5m ago 112m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (112m) 6m ago 112m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (100m) 5m ago 113m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (100m) 6m ago 113m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (107m) 6m ago 117m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (108m) 5m ago 119m 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:15:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (107m) 5m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (106m) 6m ago 118m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (107m) 5m ago 118m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (99m) 5m ago 112m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (99m) 6m ago 112m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (105m) 5m ago 117m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (104m) 5m ago 116m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (104m) 5m ago 116m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (103m) 5m ago 115m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (103m) 6m ago 115m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:15:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (102m) 6m ago 114m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:15:35.711 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (101m) 6m ago 114m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:15:35.711 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (101m) 6m ago 114m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:15:35.711 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (99m) 6m ago 112m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:15:36.126 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:15:36.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:15:36.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:15:36.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:36.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:36.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:15:36.128 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:15:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:35 smithi121 bash[36902]: cluster 2022-04-22T19:15:35.135635+0000 mgr.y (mgr.34107) 3771 : cluster [DBG] pgmap v5227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:35 smithi115 bash[46934]: cluster 2022-04-22T19:15:35.135635+0000 mgr.y (mgr.34107) 3771 : cluster [DBG] pgmap v5227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:36.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:35 smithi115 bash[45495]: cluster 2022-04-22T19:15:35.135635+0000 mgr.y (mgr.34107) 3771 : cluster [DBG] pgmap v5227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:36 smithi121 bash[36902]: audit 2022-04-22T19:15:35.335402+0000 mgr.y (mgr.34107) 3772 : audit [DBG] from='client.39348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:36 smithi121 bash[36902]: audit 2022-04-22T19:15:35.676992+0000 mgr.y (mgr.34107) 3773 : audit [DBG] from='client.49288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:36 smithi121 bash[36902]: audit 2022-04-22T19:15:36.114374+0000 mon.a (mon.0) 6595 : audit [DBG] from='client.? 172.21.15.115:0/2396710746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[45495]: audit 2022-04-22T19:15:35.335402+0000 mgr.y (mgr.34107) 3772 : audit [DBG] from='client.39348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[45495]: audit 2022-04-22T19:15:35.676992+0000 mgr.y (mgr.34107) 3773 : audit [DBG] from='client.49288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[45495]: audit 2022-04-22T19:15:36.114374+0000 mon.a (mon.0) 6595 : audit [DBG] from='client.? 172.21.15.115:0/2396710746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[46934]: audit 2022-04-22T19:15:35.335402+0000 mgr.y (mgr.34107) 3772 : audit [DBG] from='client.39348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[46934]: audit 2022-04-22T19:15:35.676992+0000 mgr.y (mgr.34107) 3773 : audit [DBG] from='client.49288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:15:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:36 smithi115 bash[46934]: audit 2022-04-22T19:15:36.114374+0000 mon.a (mon.0) 6595 : audit [DBG] from='client.? 172.21.15.115:0/2396710746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:15:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:37 smithi121 bash[36902]: cluster 2022-04-22T19:15:37.136669+0000 mgr.y (mgr.34107) 3774 : cluster [DBG] pgmap v5228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:38.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:37 smithi115 bash[45495]: cluster 2022-04-22T19:15:37.136669+0000 mgr.y (mgr.34107) 3774 : cluster [DBG] pgmap v5228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:38.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:37 smithi115 bash[46934]: cluster 2022-04-22T19:15:37.136669+0000 mgr.y (mgr.34107) 3774 : cluster [DBG] pgmap v5228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:38 smithi121 bash[36902]: audit 2022-04-22T19:15:38.253968+0000 mon.a (mon.0) 6596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:39.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:38 smithi115 bash[45495]: audit 2022-04-22T19:15:38.253968+0000 mon.a (mon.0) 6596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:38 smithi115 bash[46934]: audit 2022-04-22T19:15:38.253968+0000 mon.a (mon.0) 6596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:39 smithi121 bash[36902]: cluster 2022-04-22T19:15:39.137738+0000 mgr.y (mgr.34107) 3775 : cluster [DBG] pgmap v5229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:39 smithi121 bash[36902]: audit 2022-04-22T19:15:39.878167+0000 mon.a (mon.0) 6597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:40.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:39 smithi115 bash[45495]: cluster 2022-04-22T19:15:39.137738+0000 mgr.y (mgr.34107) 3775 : cluster [DBG] pgmap v5229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:40.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:39 smithi115 bash[45495]: audit 2022-04-22T19:15:39.878167+0000 mon.a (mon.0) 6597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:40.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:39 smithi115 bash[46934]: cluster 2022-04-22T19:15:39.137738+0000 mgr.y (mgr.34107) 3775 : cluster [DBG] pgmap v5229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:40.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:39 smithi115 bash[46934]: audit 2022-04-22T19:15:39.878167+0000 mon.a (mon.0) 6597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:42 smithi115 bash[45495]: cluster 2022-04-22T19:15:41.138787+0000 mgr.y (mgr.34107) 3776 : cluster [DBG] pgmap v5230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:42.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:42.030Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:42 smithi115 bash[46934]: cluster 2022-04-22T19:15:41.138787+0000 mgr.y (mgr.34107) 3776 : cluster [DBG] pgmap v5230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:42 smithi121 bash[36902]: cluster 2022-04-22T19:15:41.138787+0000 mgr.y (mgr.34107) 3776 : cluster [DBG] pgmap v5230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:43.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:43 smithi115 bash[45495]: cluster 2022-04-22T19:15:43.139840+0000 mgr.y (mgr.34107) 3777 : cluster [DBG] pgmap v5231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:43.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:43 smithi115 bash[45495]: audit 2022-04-22T19:15:43.257669+0000 mon.a (mon.0) 6598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:43.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:43 smithi115 bash[46934]: cluster 2022-04-22T19:15:43.139840+0000 mgr.y (mgr.34107) 3777 : cluster [DBG] pgmap v5231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:43.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:43 smithi115 bash[46934]: audit 2022-04-22T19:15:43.257669+0000 mon.a (mon.0) 6598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:43 smithi121 bash[36902]: cluster 2022-04-22T19:15:43.139840+0000 mgr.y (mgr.34107) 3777 : cluster [DBG] pgmap v5231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:43 smithi121 bash[36902]: audit 2022-04-22T19:15:43.257669+0000 mon.a (mon.0) 6598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:43.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:44 smithi121 bash[36902]: audit 2022-04-22T19:15:44.881468+0000 mon.a (mon.0) 6599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:45.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:44 smithi115 bash[45495]: audit 2022-04-22T19:15:44.881468+0000 mon.a (mon.0) 6599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:45.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:44 smithi115 bash[46934]: audit 2022-04-22T19:15:44.881468+0000 mon.a (mon.0) 6599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:45 smithi121 bash[36902]: cluster 2022-04-22T19:15:45.140918+0000 mgr.y (mgr.34107) 3778 : cluster [DBG] pgmap v5232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:45 smithi115 bash[45495]: cluster 2022-04-22T19:15:45.140918+0000 mgr.y (mgr.34107) 3778 : cluster [DBG] pgmap v5232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:45 smithi115 bash[46934]: cluster 2022-04-22T19:15:45.140918+0000 mgr.y (mgr.34107) 3778 : cluster [DBG] pgmap v5232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:48 smithi121 bash[36902]: cluster 2022-04-22T19:15:47.141879+0000 mgr.y (mgr.34107) 3779 : cluster [DBG] pgmap v5233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:48 smithi115 bash[45495]: cluster 2022-04-22T19:15:47.141879+0000 mgr.y (mgr.34107) 3779 : cluster [DBG] pgmap v5233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:48 smithi115 bash[46934]: cluster 2022-04-22T19:15:47.141879+0000 mgr.y (mgr.34107) 3779 : cluster [DBG] pgmap v5233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:49 smithi121 bash[36902]: audit 2022-04-22T19:15:48.261493+0000 mon.a (mon.0) 6600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:49 smithi115 bash[45495]: audit 2022-04-22T19:15:48.261493+0000 mon.a (mon.0) 6600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:49 smithi115 bash[46934]: audit 2022-04-22T19:15:48.261493+0000 mon.a (mon.0) 6600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:50 smithi121 bash[36902]: cluster 2022-04-22T19:15:49.143008+0000 mgr.y (mgr.34107) 3780 : cluster [DBG] pgmap v5234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:50 smithi121 bash[36902]: audit 2022-04-22T19:15:49.884713+0000 mon.a (mon.0) 6601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:50 smithi115 bash[45495]: cluster 2022-04-22T19:15:49.143008+0000 mgr.y (mgr.34107) 3780 : cluster [DBG] pgmap v5234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:50.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:50 smithi115 bash[45495]: audit 2022-04-22T19:15:49.884713+0000 mon.a (mon.0) 6601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:50 smithi115 bash[46934]: cluster 2022-04-22T19:15:49.143008+0000 mgr.y (mgr.34107) 3780 : cluster [DBG] pgmap v5234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:50 smithi115 bash[46934]: audit 2022-04-22T19:15:49.884713+0000 mon.a (mon.0) 6601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:52 smithi115 bash[45495]: cluster 2022-04-22T19:15:51.144112+0000 mgr.y (mgr.34107) 3781 : cluster [DBG] pgmap v5235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:52.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:52.030Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:52.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:52 smithi115 bash[46934]: cluster 2022-04-22T19:15:51.144112+0000 mgr.y (mgr.34107) 3781 : cluster [DBG] pgmap v5235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:52 smithi121 bash[36902]: cluster 2022-04-22T19:15:51.144112+0000 mgr.y (mgr.34107) 3781 : cluster [DBG] pgmap v5235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:53.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:53 smithi115 bash[46934]: cluster 2022-04-22T19:15:53.145154+0000 mgr.y (mgr.34107) 3782 : cluster [DBG] pgmap v5236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:53.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:53 smithi115 bash[46934]: audit 2022-04-22T19:15:53.264953+0000 mon.a (mon.0) 6602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:53.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:53 smithi115 bash[45495]: cluster 2022-04-22T19:15:53.145154+0000 mgr.y (mgr.34107) 3782 : cluster [DBG] pgmap v5236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:53.917 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:53 smithi115 bash[45495]: audit 2022-04-22T19:15:53.264953+0000 mon.a (mon.0) 6602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:53 smithi121 bash[36902]: cluster 2022-04-22T19:15:53.145154+0000 mgr.y (mgr.34107) 3782 : cluster [DBG] pgmap v5236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:53 smithi121 bash[36902]: audit 2022-04-22T19:15:53.264953+0000 mon.a (mon.0) 6602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:15:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:15:53.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:15:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:54 smithi121 bash[36902]: audit 2022-04-22T19:15:54.888148+0000 mon.a (mon.0) 6603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:54 smithi115 bash[46934]: audit 2022-04-22T19:15:54.888148+0000 mon.a (mon.0) 6603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:54 smithi115 bash[45495]: audit 2022-04-22T19:15:54.888148+0000 mon.a (mon.0) 6603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:15:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:55 smithi121 bash[36902]: cluster 2022-04-22T19:15:55.146285+0000 mgr.y (mgr.34107) 3783 : cluster [DBG] pgmap v5237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:55 smithi115 bash[45495]: cluster 2022-04-22T19:15:55.146285+0000 mgr.y (mgr.34107) 3783 : cluster [DBG] pgmap v5237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:56.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:55 smithi115 bash[46934]: cluster 2022-04-22T19:15:55.146285+0000 mgr.y (mgr.34107) 3783 : cluster [DBG] pgmap v5237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:58 smithi121 bash[36902]: cluster 2022-04-22T19:15:57.147370+0000 mgr.y (mgr.34107) 3784 : cluster [DBG] pgmap v5238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:58 smithi115 bash[45495]: cluster 2022-04-22T19:15:57.147370+0000 mgr.y (mgr.34107) 3784 : cluster [DBG] pgmap v5238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:58 smithi115 bash[46934]: cluster 2022-04-22T19:15:57.147370+0000 mgr.y (mgr.34107) 3784 : cluster [DBG] pgmap v5238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:15:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:15:59 smithi121 bash[36902]: audit 2022-04-22T19:15:58.268856+0000 mon.a (mon.0) 6604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:15:59 smithi115 bash[45495]: audit 2022-04-22T19:15:58.268856+0000 mon.a (mon.0) 6604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:15:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:15:59 smithi115 bash[46934]: audit 2022-04-22T19:15:58.268856+0000 mon.a (mon.0) 6604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:00 smithi121 bash[36902]: cluster 2022-04-22T19:15:59.148383+0000 mgr.y (mgr.34107) 3785 : cluster [DBG] pgmap v5239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:00 smithi121 bash[36902]: audit 2022-04-22T19:15:59.891400+0000 mon.a (mon.0) 6605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:00 smithi115 bash[45495]: cluster 2022-04-22T19:15:59.148383+0000 mgr.y (mgr.34107) 3785 : cluster [DBG] pgmap v5239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:00 smithi115 bash[45495]: audit 2022-04-22T19:15:59.891400+0000 mon.a (mon.0) 6605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:00 smithi115 bash[46934]: cluster 2022-04-22T19:15:59.148383+0000 mgr.y (mgr.34107) 3785 : cluster [DBG] pgmap v5239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:00 smithi115 bash[46934]: audit 2022-04-22T19:15:59.891400+0000 mon.a (mon.0) 6605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:02.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:02 smithi115 bash[45495]: cluster 2022-04-22T19:16:01.149461+0000 mgr.y (mgr.34107) 3786 : cluster [DBG] pgmap v5240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:02.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:02.032Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:02 smithi115 bash[46934]: cluster 2022-04-22T19:16:01.149461+0000 mgr.y (mgr.34107) 3786 : cluster [DBG] pgmap v5240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:02 smithi121 bash[36902]: cluster 2022-04-22T19:16:01.149461+0000 mgr.y (mgr.34107) 3786 : cluster [DBG] pgmap v5240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:03.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:03 smithi115 bash[46934]: cluster 2022-04-22T19:16:03.150817+0000 mgr.y (mgr.34107) 3787 : cluster [DBG] pgmap v5241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:03.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:03 smithi115 bash[46934]: audit 2022-04-22T19:16:03.272379+0000 mon.a (mon.0) 6606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:03.916 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:03 smithi115 bash[45495]: cluster 2022-04-22T19:16:03.150817+0000 mgr.y (mgr.34107) 3787 : cluster [DBG] pgmap v5241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:03.916 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:03 smithi115 bash[45495]: audit 2022-04-22T19:16:03.272379+0000 mon.a (mon.0) 6606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:03 smithi121 bash[36902]: cluster 2022-04-22T19:16:03.150817+0000 mgr.y (mgr.34107) 3787 : cluster [DBG] pgmap v5241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:03 smithi121 bash[36902]: audit 2022-04-22T19:16:03.272379+0000 mon.a (mon.0) 6606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:03.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:04 smithi121 bash[36902]: audit 2022-04-22T19:16:04.894619+0000 mon.a (mon.0) 6607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:04 smithi115 bash[45495]: audit 2022-04-22T19:16:04.894619+0000 mon.a (mon.0) 6607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:04 smithi115 bash[46934]: audit 2022-04-22T19:16:04.894619+0000 mon.a (mon.0) 6607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:05 smithi121 bash[36902]: cluster 2022-04-22T19:16:05.152006+0000 mgr.y (mgr.34107) 3788 : cluster [DBG] pgmap v5242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:05 smithi115 bash[46934]: cluster 2022-04-22T19:16:05.152006+0000 mgr.y (mgr.34107) 3788 : cluster [DBG] pgmap v5242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:05 smithi115 bash[45495]: cluster 2022-04-22T19:16:05.152006+0000 mgr.y (mgr.34107) 3788 : cluster [DBG] pgmap v5242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:06.509 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:16:06.855 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (99m) 5m ago 112m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (112m) 6m ago 112m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (101m) 5m ago 113m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (100m) 6m ago 113m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (108m) 6m ago 118m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (109m) 5m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:16:06.856 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (108m) 5m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (107m) 6m ago 118m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (107m) 5m ago 118m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (100m) 5m ago 113m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (100m) 6m ago 113m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (105m) 5m ago 117m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (105m) 5m ago 117m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (104m) 5m ago 116m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (104m) 5m ago 116m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (103m) 6m ago 115m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:16:06.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (102m) 6m ago 115m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:16:06.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (102m) 6m ago 115m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:16:06.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (101m) 6m ago 114m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:16:06.858 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (99m) 6m ago 113m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:16:07.267 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:07.268 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:16:07.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:16:07.270 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:16:07.270 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:16:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:08 smithi121 bash[36902]: audit 2022-04-22T19:16:06.483436+0000 mgr.y (mgr.34107) 3789 : audit [DBG] from='client.39366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:08 smithi121 bash[36902]: audit 2022-04-22T19:16:06.824383+0000 mgr.y (mgr.34107) 3790 : audit [DBG] from='client.39372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:08 smithi121 bash[36902]: cluster 2022-04-22T19:16:07.153038+0000 mgr.y (mgr.34107) 3791 : cluster [DBG] pgmap v5243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:08 smithi121 bash[36902]: audit 2022-04-22T19:16:07.256877+0000 mon.c (mon.1) 185 : audit [DBG] from='client.? 172.21.15.115:0/1149857344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[45495]: audit 2022-04-22T19:16:06.483436+0000 mgr.y (mgr.34107) 3789 : audit [DBG] from='client.39366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[45495]: audit 2022-04-22T19:16:06.824383+0000 mgr.y (mgr.34107) 3790 : audit [DBG] from='client.39372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[45495]: cluster 2022-04-22T19:16:07.153038+0000 mgr.y (mgr.34107) 3791 : cluster [DBG] pgmap v5243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[45495]: audit 2022-04-22T19:16:07.256877+0000 mon.c (mon.1) 185 : audit [DBG] from='client.? 172.21.15.115:0/1149857344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[46934]: audit 2022-04-22T19:16:06.483436+0000 mgr.y (mgr.34107) 3789 : audit [DBG] from='client.39366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[46934]: audit 2022-04-22T19:16:06.824383+0000 mgr.y (mgr.34107) 3790 : audit [DBG] from='client.39372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[46934]: cluster 2022-04-22T19:16:07.153038+0000 mgr.y (mgr.34107) 3791 : cluster [DBG] pgmap v5243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:08 smithi115 bash[46934]: audit 2022-04-22T19:16:07.256877+0000 mon.c (mon.1) 185 : audit [DBG] from='client.? 172.21.15.115:0/1149857344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:09 smithi121 bash[36902]: audit 2022-04-22T19:16:08.276113+0000 mon.a (mon.0) 6608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:09 smithi115 bash[45495]: audit 2022-04-22T19:16:08.276113+0000 mon.a (mon.0) 6608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:09 smithi115 bash[46934]: audit 2022-04-22T19:16:08.276113+0000 mon.a (mon.0) 6608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:10 smithi121 bash[36902]: cluster 2022-04-22T19:16:09.154130+0000 mgr.y (mgr.34107) 3792 : cluster [DBG] pgmap v5244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:10 smithi121 bash[36902]: audit 2022-04-22T19:16:09.897967+0000 mon.a (mon.0) 6609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:10 smithi115 bash[45495]: cluster 2022-04-22T19:16:09.154130+0000 mgr.y (mgr.34107) 3792 : cluster [DBG] pgmap v5244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:10 smithi115 bash[45495]: audit 2022-04-22T19:16:09.897967+0000 mon.a (mon.0) 6609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:10 smithi115 bash[46934]: cluster 2022-04-22T19:16:09.154130+0000 mgr.y (mgr.34107) 3792 : cluster [DBG] pgmap v5244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:10 smithi115 bash[46934]: audit 2022-04-22T19:16:09.897967+0000 mon.a (mon.0) 6609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:12 smithi115 bash[45495]: cluster 2022-04-22T19:16:11.155157+0000 mgr.y (mgr.34107) 3793 : cluster [DBG] pgmap v5245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:12.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:12.034Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:12 smithi115 bash[46934]: cluster 2022-04-22T19:16:11.155157+0000 mgr.y (mgr.34107) 3793 : cluster [DBG] pgmap v5245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:12 smithi121 bash[36902]: cluster 2022-04-22T19:16:11.155157+0000 mgr.y (mgr.34107) 3793 : cluster [DBG] pgmap v5245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:13.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:13 smithi115 bash[46934]: cluster 2022-04-22T19:16:13.156051+0000 mgr.y (mgr.34107) 3794 : cluster [DBG] pgmap v5246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:13.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:13 smithi115 bash[46934]: audit 2022-04-22T19:16:13.279876+0000 mon.a (mon.0) 6610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:13.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:13 smithi115 bash[45495]: cluster 2022-04-22T19:16:13.156051+0000 mgr.y (mgr.34107) 3794 : cluster [DBG] pgmap v5246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:13.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:13 smithi115 bash[45495]: audit 2022-04-22T19:16:13.279876+0000 mon.a (mon.0) 6610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:13 smithi121 bash[36902]: cluster 2022-04-22T19:16:13.156051+0000 mgr.y (mgr.34107) 3794 : cluster [DBG] pgmap v5246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:13 smithi121 bash[36902]: audit 2022-04-22T19:16:13.279876+0000 mon.a (mon.0) 6610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:13.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:14 smithi121 bash[36902]: audit 2022-04-22T19:16:14.518988+0000 mon.a (mon.0) 6611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:16:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:14 smithi121 bash[36902]: audit 2022-04-22T19:16:14.520652+0000 mon.a (mon.0) 6612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:16:15.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:14 smithi115 bash[45495]: audit 2022-04-22T19:16:14.518988+0000 mon.a (mon.0) 6611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:16:15.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:14 smithi115 bash[45495]: audit 2022-04-22T19:16:14.520652+0000 mon.a (mon.0) 6612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:16:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:14 smithi115 bash[46934]: audit 2022-04-22T19:16:14.518988+0000 mon.a (mon.0) 6611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:16:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:14 smithi115 bash[46934]: audit 2022-04-22T19:16:14.520652+0000 mon.a (mon.0) 6612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:16:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:15 smithi121 bash[36902]: audit 2022-04-22T19:16:14.901053+0000 mon.a (mon.0) 6613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:15 smithi121 bash[36902]: cluster 2022-04-22T19:16:15.157153+0000 mgr.y (mgr.34107) 3795 : cluster [DBG] pgmap v5247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:15 smithi115 bash[46934]: audit 2022-04-22T19:16:14.901053+0000 mon.a (mon.0) 6613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:15 smithi115 bash[46934]: cluster 2022-04-22T19:16:15.157153+0000 mgr.y (mgr.34107) 3795 : cluster [DBG] pgmap v5247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:15 smithi115 bash[45495]: audit 2022-04-22T19:16:14.901053+0000 mon.a (mon.0) 6613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:15 smithi115 bash[45495]: cluster 2022-04-22T19:16:15.157153+0000 mgr.y (mgr.34107) 3795 : cluster [DBG] pgmap v5247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:18 smithi121 bash[36902]: cluster 2022-04-22T19:16:17.158194+0000 mgr.y (mgr.34107) 3796 : cluster [DBG] pgmap v5248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:18 smithi115 bash[46934]: cluster 2022-04-22T19:16:17.158194+0000 mgr.y (mgr.34107) 3796 : cluster [DBG] pgmap v5248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:18 smithi115 bash[45495]: cluster 2022-04-22T19:16:17.158194+0000 mgr.y (mgr.34107) 3796 : cluster [DBG] pgmap v5248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:19 smithi121 bash[36902]: audit 2022-04-22T19:16:18.283118+0000 mon.a (mon.0) 6614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:19 smithi115 bash[46934]: audit 2022-04-22T19:16:18.283118+0000 mon.a (mon.0) 6614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:19 smithi115 bash[45495]: audit 2022-04-22T19:16:18.283118+0000 mon.a (mon.0) 6614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:20 smithi121 bash[36902]: cluster 2022-04-22T19:16:19.159278+0000 mgr.y (mgr.34107) 3797 : cluster [DBG] pgmap v5249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:20 smithi121 bash[36902]: audit 2022-04-22T19:16:19.904274+0000 mon.a (mon.0) 6615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:20 smithi115 bash[45495]: cluster 2022-04-22T19:16:19.159278+0000 mgr.y (mgr.34107) 3797 : cluster [DBG] pgmap v5249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:20.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:20 smithi115 bash[45495]: audit 2022-04-22T19:16:19.904274+0000 mon.a (mon.0) 6615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:20 smithi115 bash[46934]: cluster 2022-04-22T19:16:19.159278+0000 mgr.y (mgr.34107) 3797 : cluster [DBG] pgmap v5249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:20 smithi115 bash[46934]: audit 2022-04-22T19:16:19.904274+0000 mon.a (mon.0) 6615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:22.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:22 smithi115 bash[45495]: cluster 2022-04-22T19:16:21.160355+0000 mgr.y (mgr.34107) 3798 : cluster [DBG] pgmap v5250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:22.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:22.034Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:22.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:22 smithi115 bash[46934]: cluster 2022-04-22T19:16:21.160355+0000 mgr.y (mgr.34107) 3798 : cluster [DBG] pgmap v5250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:22 smithi121 bash[36902]: cluster 2022-04-22T19:16:21.160355+0000 mgr.y (mgr.34107) 3798 : cluster [DBG] pgmap v5250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:23.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:23 smithi115 bash[45495]: cluster 2022-04-22T19:16:23.161418+0000 mgr.y (mgr.34107) 3799 : cluster [DBG] pgmap v5251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:23.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:23 smithi115 bash[45495]: audit 2022-04-22T19:16:23.286608+0000 mon.a (mon.0) 6616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:23.923 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:23.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:23.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:23 smithi115 bash[46934]: cluster 2022-04-22T19:16:23.161418+0000 mgr.y (mgr.34107) 3799 : cluster [DBG] pgmap v5251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:23.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:23 smithi115 bash[46934]: audit 2022-04-22T19:16:23.286608+0000 mon.a (mon.0) 6616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:23 smithi121 bash[36902]: cluster 2022-04-22T19:16:23.161418+0000 mgr.y (mgr.34107) 3799 : cluster [DBG] pgmap v5251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:23 smithi121 bash[36902]: audit 2022-04-22T19:16:23.286608+0000 mon.a (mon.0) 6616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:24 smithi121 bash[36902]: audit 2022-04-22T19:16:24.907463+0000 mon.a (mon.0) 6617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:24 smithi115 bash[45495]: audit 2022-04-22T19:16:24.907463+0000 mon.a (mon.0) 6617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:25.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:24 smithi115 bash[46934]: audit 2022-04-22T19:16:24.907463+0000 mon.a (mon.0) 6617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:25 smithi121 bash[36902]: cluster 2022-04-22T19:16:25.162581+0000 mgr.y (mgr.34107) 3800 : cluster [DBG] pgmap v5252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:25 smithi115 bash[45495]: cluster 2022-04-22T19:16:25.162581+0000 mgr.y (mgr.34107) 3800 : cluster [DBG] pgmap v5252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:25 smithi115 bash[46934]: cluster 2022-04-22T19:16:25.162581+0000 mgr.y (mgr.34107) 3800 : cluster [DBG] pgmap v5252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:28 smithi121 bash[36902]: cluster 2022-04-22T19:16:27.163774+0000 mgr.y (mgr.34107) 3801 : cluster [DBG] pgmap v5253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:28 smithi115 bash[46934]: cluster 2022-04-22T19:16:27.163774+0000 mgr.y (mgr.34107) 3801 : cluster [DBG] pgmap v5253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:28 smithi115 bash[45495]: cluster 2022-04-22T19:16:27.163774+0000 mgr.y (mgr.34107) 3801 : cluster [DBG] pgmap v5253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:29 smithi121 bash[36902]: audit 2022-04-22T19:16:28.289954+0000 mon.a (mon.0) 6618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:29 smithi115 bash[46934]: audit 2022-04-22T19:16:28.289954+0000 mon.a (mon.0) 6618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:29 smithi115 bash[45495]: audit 2022-04-22T19:16:28.289954+0000 mon.a (mon.0) 6618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:30 smithi121 bash[36902]: cluster 2022-04-22T19:16:29.164825+0000 mgr.y (mgr.34107) 3802 : cluster [DBG] pgmap v5254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:30 smithi121 bash[36902]: audit 2022-04-22T19:16:29.910721+0000 mon.a (mon.0) 6619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:30 smithi115 bash[45495]: cluster 2022-04-22T19:16:29.164825+0000 mgr.y (mgr.34107) 3802 : cluster [DBG] pgmap v5254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:30 smithi115 bash[45495]: audit 2022-04-22T19:16:29.910721+0000 mon.a (mon.0) 6619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:30 smithi115 bash[46934]: cluster 2022-04-22T19:16:29.164825+0000 mgr.y (mgr.34107) 3802 : cluster [DBG] pgmap v5254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:30 smithi115 bash[46934]: audit 2022-04-22T19:16:29.910721+0000 mon.a (mon.0) 6619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:32.161 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:32.035Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:32 smithi121 bash[36902]: cluster 2022-04-22T19:16:31.165915+0000 mgr.y (mgr.34107) 3803 : cluster [DBG] pgmap v5255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:32 smithi121 bash[36902]: audit 2022-04-22T19:16:31.786603+0000 mon.a (mon.0) 6620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:16:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:32 smithi121 bash[36902]: audit 2022-04-22T19:16:31.789589+0000 mon.a (mon.0) 6621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:16:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:32 smithi121 bash[36902]: audit 2022-04-22T19:16:31.791354+0000 mon.a (mon.0) 6622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:16:32.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:32 smithi121 bash[36902]: audit 2022-04-22T19:16:32.160796+0000 mon.a (mon.0) 6623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:16:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[45495]: cluster 2022-04-22T19:16:31.165915+0000 mgr.y (mgr.34107) 3803 : cluster [DBG] pgmap v5255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[45495]: audit 2022-04-22T19:16:31.786603+0000 mon.a (mon.0) 6620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:16:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[45495]: audit 2022-04-22T19:16:31.789589+0000 mon.a (mon.0) 6621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[45495]: audit 2022-04-22T19:16:31.791354+0000 mon.a (mon.0) 6622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[45495]: audit 2022-04-22T19:16:32.160796+0000 mon.a (mon.0) 6623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[46934]: cluster 2022-04-22T19:16:31.165915+0000 mgr.y (mgr.34107) 3803 : cluster [DBG] pgmap v5255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[46934]: audit 2022-04-22T19:16:31.786603+0000 mon.a (mon.0) 6620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[46934]: audit 2022-04-22T19:16:31.789589+0000 mon.a (mon.0) 6621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[46934]: audit 2022-04-22T19:16:31.791354+0000 mon.a (mon.0) 6622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:16:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:32 smithi115 bash[46934]: audit 2022-04-22T19:16:32.160796+0000 mon.a (mon.0) 6623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:16:33.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:33 smithi115 bash[45495]: cluster 2022-04-22T19:16:33.166973+0000 mgr.y (mgr.34107) 3804 : cluster [DBG] pgmap v5256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:33.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:33 smithi115 bash[45495]: audit 2022-04-22T19:16:33.293553+0000 mon.a (mon.0) 6624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:33.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:33 smithi115 bash[46934]: cluster 2022-04-22T19:16:33.166973+0000 mgr.y (mgr.34107) 3804 : cluster [DBG] pgmap v5256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:33.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:33 smithi115 bash[46934]: audit 2022-04-22T19:16:33.293553+0000 mon.a (mon.0) 6624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:33 smithi121 bash[36902]: cluster 2022-04-22T19:16:33.166973+0000 mgr.y (mgr.34107) 3804 : cluster [DBG] pgmap v5256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:33 smithi121 bash[36902]: audit 2022-04-22T19:16:33.293553+0000 mon.a (mon.0) 6624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:33.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:34 smithi121 bash[36902]: audit 2022-04-22T19:16:34.913756+0000 mon.a (mon.0) 6625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:35.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:34 smithi115 bash[45495]: audit 2022-04-22T19:16:34.913756+0000 mon.a (mon.0) 6625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:35.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:34 smithi115 bash[46934]: audit 2022-04-22T19:16:34.913756+0000 mon.a (mon.0) 6625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:35 smithi121 bash[36902]: cluster 2022-04-22T19:16:35.168171+0000 mgr.y (mgr.34107) 3805 : cluster [DBG] pgmap v5257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:35 smithi115 bash[46934]: cluster 2022-04-22T19:16:35.168171+0000 mgr.y (mgr.34107) 3805 : cluster [DBG] pgmap v5257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:36.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:35 smithi115 bash[45495]: cluster 2022-04-22T19:16:35.168171+0000 mgr.y (mgr.34107) 3805 : cluster [DBG] pgmap v5257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:37.639 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (99m) 6m ago 113m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (113m) 7m ago 113m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (101m) 6m ago 114m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (101m) 7m ago 114m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:16:37.978 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (108m) 7m ago 118m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (109m) 6m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (108m) 6m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (107m) 7m ago 119m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (108m) 6m ago 119m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (100m) 6m ago 113m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (100m) 7m ago 113m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (106m) 6m ago 118m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (105m) 6m ago 117m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (105m) 6m ago 117m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:16:37.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (104m) 6m ago 116m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:16:37.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (104m) 7m ago 116m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:16:37.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (103m) 7m ago 116m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:16:37.988 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (102m) 7m ago 115m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:16:37.988 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (102m) 7m ago 115m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:16:37.988 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (100m) 7m ago 113m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:16:38.377 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:16:38.377 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:16:38.377 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:16:38.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:38.377 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:16:38.378 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:16:38.379 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:16:38.379 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:16:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:38 smithi121 bash[36902]: cluster 2022-04-22T19:16:37.169252+0000 mgr.y (mgr.34107) 3806 : cluster [DBG] pgmap v5258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:38 smithi115 bash[46934]: cluster 2022-04-22T19:16:37.169252+0000 mgr.y (mgr.34107) 3806 : cluster [DBG] pgmap v5258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:38.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:38 smithi115 bash[45495]: cluster 2022-04-22T19:16:37.169252+0000 mgr.y (mgr.34107) 3806 : cluster [DBG] pgmap v5258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:39 smithi121 bash[36902]: audit 2022-04-22T19:16:37.613352+0000 mgr.y (mgr.34107) 3807 : audit [DBG] from='client.39384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:39 smithi121 bash[36902]: audit 2022-04-22T19:16:37.951869+0000 mgr.y (mgr.34107) 3808 : audit [DBG] from='client.39390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:39 smithi121 bash[36902]: audit 2022-04-22T19:16:38.297020+0000 mon.a (mon.0) 6626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:39 smithi121 bash[36902]: audit 2022-04-22T19:16:38.365854+0000 mon.a (mon.0) 6627 : audit [DBG] from='client.? 172.21.15.115:0/3297244247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[46934]: audit 2022-04-22T19:16:37.613352+0000 mgr.y (mgr.34107) 3807 : audit [DBG] from='client.39384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[46934]: audit 2022-04-22T19:16:37.951869+0000 mgr.y (mgr.34107) 3808 : audit [DBG] from='client.39390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[46934]: audit 2022-04-22T19:16:38.297020+0000 mon.a (mon.0) 6626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[46934]: audit 2022-04-22T19:16:38.365854+0000 mon.a (mon.0) 6627 : audit [DBG] from='client.? 172.21.15.115:0/3297244247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[45495]: audit 2022-04-22T19:16:37.613352+0000 mgr.y (mgr.34107) 3807 : audit [DBG] from='client.39384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[45495]: audit 2022-04-22T19:16:37.951869+0000 mgr.y (mgr.34107) 3808 : audit [DBG] from='client.39390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[45495]: audit 2022-04-22T19:16:38.297020+0000 mon.a (mon.0) 6626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:39 smithi115 bash[45495]: audit 2022-04-22T19:16:38.365854+0000 mon.a (mon.0) 6627 : audit [DBG] from='client.? 172.21.15.115:0/3297244247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:16:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:40 smithi121 bash[36902]: cluster 2022-04-22T19:16:39.170351+0000 mgr.y (mgr.34107) 3809 : cluster [DBG] pgmap v5259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:40 smithi121 bash[36902]: audit 2022-04-22T19:16:39.916983+0000 mon.a (mon.0) 6628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:40 smithi115 bash[45495]: cluster 2022-04-22T19:16:39.170351+0000 mgr.y (mgr.34107) 3809 : cluster [DBG] pgmap v5259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:40.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:40 smithi115 bash[45495]: audit 2022-04-22T19:16:39.916983+0000 mon.a (mon.0) 6628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:40 smithi115 bash[46934]: cluster 2022-04-22T19:16:39.170351+0000 mgr.y (mgr.34107) 3809 : cluster [DBG] pgmap v5259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:40 smithi115 bash[46934]: audit 2022-04-22T19:16:39.916983+0000 mon.a (mon.0) 6628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:42.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:42 smithi115 bash[45495]: cluster 2022-04-22T19:16:41.171424+0000 mgr.y (mgr.34107) 3810 : cluster [DBG] pgmap v5260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:42.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:42.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:42.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:42 smithi115 bash[46934]: cluster 2022-04-22T19:16:41.171424+0000 mgr.y (mgr.34107) 3810 : cluster [DBG] pgmap v5260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:42 smithi121 bash[36902]: cluster 2022-04-22T19:16:41.171424+0000 mgr.y (mgr.34107) 3810 : cluster [DBG] pgmap v5260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:43.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:43 smithi115 bash[45495]: cluster 2022-04-22T19:16:43.172471+0000 mgr.y (mgr.34107) 3811 : cluster [DBG] pgmap v5261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:43.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:43 smithi115 bash[45495]: audit 2022-04-22T19:16:43.300189+0000 mon.a (mon.0) 6629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:43.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:43 smithi115 bash[46934]: cluster 2022-04-22T19:16:43.172471+0000 mgr.y (mgr.34107) 3811 : cluster [DBG] pgmap v5261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:43.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:43 smithi115 bash[46934]: audit 2022-04-22T19:16:43.300189+0000 mon.a (mon.0) 6629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:43 smithi121 bash[36902]: cluster 2022-04-22T19:16:43.172471+0000 mgr.y (mgr.34107) 3811 : cluster [DBG] pgmap v5261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:43 smithi121 bash[36902]: audit 2022-04-22T19:16:43.300189+0000 mon.a (mon.0) 6629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:43.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:44 smithi121 bash[36902]: audit 2022-04-22T19:16:44.920234+0000 mon.a (mon.0) 6630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:45.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:44 smithi115 bash[46934]: audit 2022-04-22T19:16:44.920234+0000 mon.a (mon.0) 6630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:44 smithi115 bash[45495]: audit 2022-04-22T19:16:44.920234+0000 mon.a (mon.0) 6630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:45 smithi121 bash[36902]: cluster 2022-04-22T19:16:45.173633+0000 mgr.y (mgr.34107) 3812 : cluster [DBG] pgmap v5262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:45 smithi115 bash[45495]: cluster 2022-04-22T19:16:45.173633+0000 mgr.y (mgr.34107) 3812 : cluster [DBG] pgmap v5262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:45 smithi115 bash[46934]: cluster 2022-04-22T19:16:45.173633+0000 mgr.y (mgr.34107) 3812 : cluster [DBG] pgmap v5262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:48 smithi121 bash[36902]: cluster 2022-04-22T19:16:47.174590+0000 mgr.y (mgr.34107) 3813 : cluster [DBG] pgmap v5263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:48 smithi115 bash[45495]: cluster 2022-04-22T19:16:47.174590+0000 mgr.y (mgr.34107) 3813 : cluster [DBG] pgmap v5263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:48 smithi115 bash[46934]: cluster 2022-04-22T19:16:47.174590+0000 mgr.y (mgr.34107) 3813 : cluster [DBG] pgmap v5263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:49 smithi121 bash[36902]: audit 2022-04-22T19:16:48.303490+0000 mon.a (mon.0) 6631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:49 smithi115 bash[45495]: audit 2022-04-22T19:16:48.303490+0000 mon.a (mon.0) 6631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:49 smithi115 bash[46934]: audit 2022-04-22T19:16:48.303490+0000 mon.a (mon.0) 6631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:50 smithi121 bash[36902]: cluster 2022-04-22T19:16:49.175753+0000 mgr.y (mgr.34107) 3814 : cluster [DBG] pgmap v5264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:50 smithi121 bash[36902]: audit 2022-04-22T19:16:49.923332+0000 mon.a (mon.0) 6632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:50 smithi115 bash[45495]: cluster 2022-04-22T19:16:49.175753+0000 mgr.y (mgr.34107) 3814 : cluster [DBG] pgmap v5264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:50 smithi115 bash[45495]: audit 2022-04-22T19:16:49.923332+0000 mon.a (mon.0) 6632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:50 smithi115 bash[46934]: cluster 2022-04-22T19:16:49.175753+0000 mgr.y (mgr.34107) 3814 : cluster [DBG] pgmap v5264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:50 smithi115 bash[46934]: audit 2022-04-22T19:16:49.923332+0000 mon.a (mon.0) 6632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:52.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:52 smithi115 bash[46934]: cluster 2022-04-22T19:16:51.176847+0000 mgr.y (mgr.34107) 3815 : cluster [DBG] pgmap v5265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:52 smithi115 bash[45495]: cluster 2022-04-22T19:16:51.176847+0000 mgr.y (mgr.34107) 3815 : cluster [DBG] pgmap v5265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:52.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:52.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:52 smithi121 bash[36902]: cluster 2022-04-22T19:16:51.176847+0000 mgr.y (mgr.34107) 3815 : cluster [DBG] pgmap v5265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:53.918 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:53 smithi115 bash[45495]: cluster 2022-04-22T19:16:53.177911+0000 mgr.y (mgr.34107) 3816 : cluster [DBG] pgmap v5266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:53.918 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:53 smithi115 bash[45495]: audit 2022-04-22T19:16:53.306403+0000 mon.a (mon.0) 6633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:53.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:53 smithi115 bash[46934]: cluster 2022-04-22T19:16:53.177911+0000 mgr.y (mgr.34107) 3816 : cluster [DBG] pgmap v5266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:53.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:53 smithi115 bash[46934]: audit 2022-04-22T19:16:53.306403+0000 mon.a (mon.0) 6633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:53 smithi121 bash[36902]: cluster 2022-04-22T19:16:53.177911+0000 mgr.y (mgr.34107) 3816 : cluster [DBG] pgmap v5266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:53 smithi121 bash[36902]: audit 2022-04-22T19:16:53.306403+0000 mon.a (mon.0) 6633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:16:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:16:53.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:16:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:54 smithi121 bash[36902]: audit 2022-04-22T19:16:54.926693+0000 mon.a (mon.0) 6634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:54 smithi115 bash[45495]: audit 2022-04-22T19:16:54.926693+0000 mon.a (mon.0) 6634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:55.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:54 smithi115 bash[46934]: audit 2022-04-22T19:16:54.926693+0000 mon.a (mon.0) 6634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:16:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:56 smithi121 bash[36902]: cluster 2022-04-22T19:16:55.179078+0000 mgr.y (mgr.34107) 3817 : cluster [DBG] pgmap v5267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:56.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:55 smithi115 bash[45495]: cluster 2022-04-22T19:16:55.179078+0000 mgr.y (mgr.34107) 3817 : cluster [DBG] pgmap v5267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:56.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:55 smithi115 bash[46934]: cluster 2022-04-22T19:16:55.179078+0000 mgr.y (mgr.34107) 3817 : cluster [DBG] pgmap v5267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:58 smithi121 bash[36902]: cluster 2022-04-22T19:16:57.180245+0000 mgr.y (mgr.34107) 3818 : cluster [DBG] pgmap v5268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:58 smithi115 bash[45495]: cluster 2022-04-22T19:16:57.180245+0000 mgr.y (mgr.34107) 3818 : cluster [DBG] pgmap v5268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:58.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:58 smithi115 bash[46934]: cluster 2022-04-22T19:16:57.180245+0000 mgr.y (mgr.34107) 3818 : cluster [DBG] pgmap v5268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:16:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:16:59 smithi121 bash[36902]: audit 2022-04-22T19:16:58.310060+0000 mon.a (mon.0) 6635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:16:59 smithi115 bash[45495]: audit 2022-04-22T19:16:58.310060+0000 mon.a (mon.0) 6635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:16:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:16:59 smithi115 bash[46934]: audit 2022-04-22T19:16:58.310060+0000 mon.a (mon.0) 6635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:00 smithi121 bash[36902]: cluster 2022-04-22T19:16:59.181255+0000 mgr.y (mgr.34107) 3819 : cluster [DBG] pgmap v5269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:00 smithi121 bash[36902]: audit 2022-04-22T19:16:59.929875+0000 mon.a (mon.0) 6636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:00 smithi115 bash[45495]: cluster 2022-04-22T19:16:59.181255+0000 mgr.y (mgr.34107) 3819 : cluster [DBG] pgmap v5269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:00 smithi115 bash[45495]: audit 2022-04-22T19:16:59.929875+0000 mon.a (mon.0) 6636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:00 smithi115 bash[46934]: cluster 2022-04-22T19:16:59.181255+0000 mgr.y (mgr.34107) 3819 : cluster [DBG] pgmap v5269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:00 smithi115 bash[46934]: audit 2022-04-22T19:16:59.929875+0000 mon.a (mon.0) 6636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:02.266 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:02.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:02.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:02 smithi115 bash[45495]: cluster 2022-04-22T19:17:01.182548+0000 mgr.y (mgr.34107) 3820 : cluster [DBG] pgmap v5270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:02.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:02 smithi115 bash[46934]: cluster 2022-04-22T19:17:01.182548+0000 mgr.y (mgr.34107) 3820 : cluster [DBG] pgmap v5270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:02 smithi121 bash[36902]: cluster 2022-04-22T19:17:01.182548+0000 mgr.y (mgr.34107) 3820 : cluster [DBG] pgmap v5270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:03.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:03 smithi115 bash[45495]: cluster 2022-04-22T19:17:03.183666+0000 mgr.y (mgr.34107) 3821 : cluster [DBG] pgmap v5271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:03.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:03 smithi115 bash[45495]: audit 2022-04-22T19:17:03.313593+0000 mon.a (mon.0) 6637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:03.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:03 smithi115 bash[46934]: cluster 2022-04-22T19:17:03.183666+0000 mgr.y (mgr.34107) 3821 : cluster [DBG] pgmap v5271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:03.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:03 smithi115 bash[46934]: audit 2022-04-22T19:17:03.313593+0000 mon.a (mon.0) 6637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:03 smithi121 bash[36902]: cluster 2022-04-22T19:17:03.183666+0000 mgr.y (mgr.34107) 3821 : cluster [DBG] pgmap v5271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:03 smithi121 bash[36902]: audit 2022-04-22T19:17:03.313593+0000 mon.a (mon.0) 6637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:03.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:04 smithi121 bash[36902]: audit 2022-04-22T19:17:04.933416+0000 mon.a (mon.0) 6638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:04 smithi115 bash[46934]: audit 2022-04-22T19:17:04.933416+0000 mon.a (mon.0) 6638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:05.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:04 smithi115 bash[45495]: audit 2022-04-22T19:17:04.933416+0000 mon.a (mon.0) 6638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:06 smithi121 bash[36902]: cluster 2022-04-22T19:17:05.184761+0000 mgr.y (mgr.34107) 3822 : cluster [DBG] pgmap v5272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:05 smithi115 bash[45495]: cluster 2022-04-22T19:17:05.184761+0000 mgr.y (mgr.34107) 3822 : cluster [DBG] pgmap v5272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:06.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:06 smithi115 bash[46934]: cluster 2022-04-22T19:17:05.184761+0000 mgr.y (mgr.34107) 3822 : cluster [DBG] pgmap v5272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:08 smithi121 bash[36902]: cluster 2022-04-22T19:17:07.185769+0000 mgr.y (mgr.34107) 3823 : cluster [DBG] pgmap v5273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:08 smithi115 bash[45495]: cluster 2022-04-22T19:17:07.185769+0000 mgr.y (mgr.34107) 3823 : cluster [DBG] pgmap v5273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:08 smithi115 bash[46934]: cluster 2022-04-22T19:17:07.185769+0000 mgr.y (mgr.34107) 3823 : cluster [DBG] pgmap v5273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:08.747 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:17:09.093 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:17:09.093 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (100m) 6m ago 114m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:17:09.093 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (113m) 7m ago 113m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:17:09.093 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (102m) 6m ago 115m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:17:09.094 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (101m) 7m ago 114m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:17:09.094 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (109m) 7m ago 119m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:17:09.094 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (110m) 6m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:17:09.095 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (109m) 6m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:17:09.095 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (108m) 7m ago 119m 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:17:09.095 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (108m) 6m ago 119m 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:17:09.095 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (101m) 6m ago 114m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:17:09.095 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (101m) 7m ago 114m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:17:09.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (106m) 6m ago 118m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:17:09.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (106m) 6m ago 118m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:17:09.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (105m) 6m ago 117m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:17:09.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (105m) 6m ago 117m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:17:09.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (104m) 7m ago 116m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:17:09.097 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (103m) 7m ago 116m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:17:09.097 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (103m) 7m ago 116m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:17:09.097 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (102m) 7m ago 115m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:17:09.097 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (100m) 7m ago 114m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:17:09.495 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:17:09.496 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:17:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:09 smithi121 bash[36902]: audit 2022-04-22T19:17:08.317167+0000 mon.a (mon.0) 6639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:09 smithi115 bash[45495]: audit 2022-04-22T19:17:08.317167+0000 mon.a (mon.0) 6639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:09 smithi115 bash[46934]: audit 2022-04-22T19:17:08.317167+0000 mon.a (mon.0) 6639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:10 smithi121 bash[36902]: audit 2022-04-22T19:17:08.719456+0000 mgr.y (mgr.34107) 3824 : audit [DBG] from='client.49336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:10 smithi121 bash[36902]: audit 2022-04-22T19:17:09.062563+0000 mgr.y (mgr.34107) 3825 : audit [DBG] from='client.49342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:10 smithi121 bash[36902]: cluster 2022-04-22T19:17:09.186714+0000 mgr.y (mgr.34107) 3826 : cluster [DBG] pgmap v5274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:10 smithi121 bash[36902]: audit 2022-04-22T19:17:09.483565+0000 mon.c (mon.1) 186 : audit [DBG] from='client.? 172.21.15.115:0/1120628499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:10 smithi121 bash[36902]: audit 2022-04-22T19:17:09.936501+0000 mon.a (mon.0) 6640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[45495]: audit 2022-04-22T19:17:08.719456+0000 mgr.y (mgr.34107) 3824 : audit [DBG] from='client.49336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[45495]: audit 2022-04-22T19:17:09.062563+0000 mgr.y (mgr.34107) 3825 : audit [DBG] from='client.49342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[45495]: cluster 2022-04-22T19:17:09.186714+0000 mgr.y (mgr.34107) 3826 : cluster [DBG] pgmap v5274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[45495]: audit 2022-04-22T19:17:09.483565+0000 mon.c (mon.1) 186 : audit [DBG] from='client.? 172.21.15.115:0/1120628499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[45495]: audit 2022-04-22T19:17:09.936501+0000 mon.a (mon.0) 6640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[46934]: audit 2022-04-22T19:17:08.719456+0000 mgr.y (mgr.34107) 3824 : audit [DBG] from='client.49336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[46934]: audit 2022-04-22T19:17:09.062563+0000 mgr.y (mgr.34107) 3825 : audit [DBG] from='client.49342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[46934]: cluster 2022-04-22T19:17:09.186714+0000 mgr.y (mgr.34107) 3826 : cluster [DBG] pgmap v5274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[46934]: audit 2022-04-22T19:17:09.483565+0000 mon.c (mon.1) 186 : audit [DBG] from='client.? 172.21.15.115:0/1120628499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:10 smithi115 bash[46934]: audit 2022-04-22T19:17:09.936501+0000 mon.a (mon.0) 6640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:12.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:12 smithi115 bash[45495]: cluster 2022-04-22T19:17:11.187777+0000 mgr.y (mgr.34107) 3827 : cluster [DBG] pgmap v5275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:12.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:12.038Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:12.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:12 smithi115 bash[46934]: cluster 2022-04-22T19:17:11.187777+0000 mgr.y (mgr.34107) 3827 : cluster [DBG] pgmap v5275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:12 smithi121 bash[36902]: cluster 2022-04-22T19:17:11.187777+0000 mgr.y (mgr.34107) 3827 : cluster [DBG] pgmap v5275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:13.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:13 smithi115 bash[46934]: cluster 2022-04-22T19:17:13.188728+0000 mgr.y (mgr.34107) 3828 : cluster [DBG] pgmap v5276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:13.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:13 smithi115 bash[46934]: audit 2022-04-22T19:17:13.320898+0000 mon.a (mon.0) 6641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:13.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:13 smithi115 bash[45495]: cluster 2022-04-22T19:17:13.188728+0000 mgr.y (mgr.34107) 3828 : cluster [DBG] pgmap v5276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:13.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:13 smithi115 bash[45495]: audit 2022-04-22T19:17:13.320898+0000 mon.a (mon.0) 6641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:13 smithi121 bash[36902]: cluster 2022-04-22T19:17:13.188728+0000 mgr.y (mgr.34107) 3828 : cluster [DBG] pgmap v5276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:13 smithi121 bash[36902]: audit 2022-04-22T19:17:13.320898+0000 mon.a (mon.0) 6641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:13.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:14 smithi121 bash[36902]: audit 2022-04-22T19:17:14.528647+0000 mon.a (mon.0) 6642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:17:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:14 smithi121 bash[36902]: audit 2022-04-22T19:17:14.529263+0000 mon.a (mon.0) 6643 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:17:15.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:14 smithi115 bash[45495]: audit 2022-04-22T19:17:14.528647+0000 mon.a (mon.0) 6642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:17:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:14 smithi115 bash[45495]: audit 2022-04-22T19:17:14.529263+0000 mon.a (mon.0) 6643 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:17:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:14 smithi115 bash[46934]: audit 2022-04-22T19:17:14.528647+0000 mon.a (mon.0) 6642 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:17:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:14 smithi115 bash[46934]: audit 2022-04-22T19:17:14.529263+0000 mon.a (mon.0) 6643 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:17:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:15 smithi121 bash[36902]: audit 2022-04-22T19:17:14.939499+0000 mon.a (mon.0) 6644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:15 smithi121 bash[36902]: cluster 2022-04-22T19:17:15.189728+0000 mgr.y (mgr.34107) 3829 : cluster [DBG] pgmap v5277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:15 smithi115 bash[45495]: audit 2022-04-22T19:17:14.939499+0000 mon.a (mon.0) 6644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:15 smithi115 bash[45495]: cluster 2022-04-22T19:17:15.189728+0000 mgr.y (mgr.34107) 3829 : cluster [DBG] pgmap v5277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:15 smithi115 bash[46934]: audit 2022-04-22T19:17:14.939499+0000 mon.a (mon.0) 6644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:15 smithi115 bash[46934]: cluster 2022-04-22T19:17:15.189728+0000 mgr.y (mgr.34107) 3829 : cluster [DBG] pgmap v5277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:18 smithi121 bash[36902]: cluster 2022-04-22T19:17:17.190417+0000 mgr.y (mgr.34107) 3830 : cluster [DBG] pgmap v5278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:18 smithi115 bash[45495]: cluster 2022-04-22T19:17:17.190417+0000 mgr.y (mgr.34107) 3830 : cluster [DBG] pgmap v5278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:18 smithi115 bash[46934]: cluster 2022-04-22T19:17:17.190417+0000 mgr.y (mgr.34107) 3830 : cluster [DBG] pgmap v5278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:19 smithi115 bash[45495]: audit 2022-04-22T19:17:18.324304+0000 mon.a (mon.0) 6645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:19.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:19 smithi115 bash[46934]: audit 2022-04-22T19:17:18.324304+0000 mon.a (mon.0) 6645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:19 smithi121 bash[36902]: audit 2022-04-22T19:17:18.324304+0000 mon.a (mon.0) 6645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:20.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:20 smithi115 bash[45495]: cluster 2022-04-22T19:17:19.191324+0000 mgr.y (mgr.34107) 3831 : cluster [DBG] pgmap v5279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:20.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:20 smithi115 bash[45495]: audit 2022-04-22T19:17:19.942568+0000 mon.a (mon.0) 6646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:20 smithi115 bash[46934]: cluster 2022-04-22T19:17:19.191324+0000 mgr.y (mgr.34107) 3831 : cluster [DBG] pgmap v5279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:20.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:20 smithi115 bash[46934]: audit 2022-04-22T19:17:19.942568+0000 mon.a (mon.0) 6646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:20 smithi121 bash[36902]: cluster 2022-04-22T19:17:19.191324+0000 mgr.y (mgr.34107) 3831 : cluster [DBG] pgmap v5279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:20 smithi121 bash[36902]: audit 2022-04-22T19:17:19.942568+0000 mon.a (mon.0) 6646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:22.040Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:22.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:22 smithi115 bash[45495]: cluster 2022-04-22T19:17:21.192306+0000 mgr.y (mgr.34107) 3832 : cluster [DBG] pgmap v5280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:22.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:22 smithi115 bash[46934]: cluster 2022-04-22T19:17:21.192306+0000 mgr.y (mgr.34107) 3832 : cluster [DBG] pgmap v5280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:22 smithi121 bash[36902]: cluster 2022-04-22T19:17:21.192306+0000 mgr.y (mgr.34107) 3832 : cluster [DBG] pgmap v5280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:23.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:23 smithi115 bash[46934]: cluster 2022-04-22T19:17:23.193416+0000 mgr.y (mgr.34107) 3833 : cluster [DBG] pgmap v5281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:23.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:23 smithi115 bash[46934]: audit 2022-04-22T19:17:23.327868+0000 mon.a (mon.0) 6647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:23.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:23 smithi115 bash[45495]: cluster 2022-04-22T19:17:23.193416+0000 mgr.y (mgr.34107) 3833 : cluster [DBG] pgmap v5281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:23.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:23 smithi115 bash[45495]: audit 2022-04-22T19:17:23.327868+0000 mon.a (mon.0) 6647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:23 smithi121 bash[36902]: cluster 2022-04-22T19:17:23.193416+0000 mgr.y (mgr.34107) 3833 : cluster [DBG] pgmap v5281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:23 smithi121 bash[36902]: audit 2022-04-22T19:17:23.327868+0000 mon.a (mon.0) 6647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:23.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:25 smithi121 bash[36902]: audit 2022-04-22T19:17:24.945809+0000 mon.a (mon.0) 6648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:24 smithi115 bash[45495]: audit 2022-04-22T19:17:24.945809+0000 mon.a (mon.0) 6648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:25.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:24 smithi115 bash[46934]: audit 2022-04-22T19:17:24.945809+0000 mon.a (mon.0) 6648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:26 smithi115 bash[45495]: cluster 2022-04-22T19:17:25.194190+0000 mgr.y (mgr.34107) 3834 : cluster [DBG] pgmap v5282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:26.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:26 smithi115 bash[46934]: cluster 2022-04-22T19:17:25.194190+0000 mgr.y (mgr.34107) 3834 : cluster [DBG] pgmap v5282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:26 smithi121 bash[36902]: cluster 2022-04-22T19:17:25.194190+0000 mgr.y (mgr.34107) 3834 : cluster [DBG] pgmap v5282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:28 smithi121 bash[36902]: cluster 2022-04-22T19:17:27.195235+0000 mgr.y (mgr.34107) 3835 : cluster [DBG] pgmap v5283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:28 smithi115 bash[45495]: cluster 2022-04-22T19:17:27.195235+0000 mgr.y (mgr.34107) 3835 : cluster [DBG] pgmap v5283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:28 smithi115 bash[46934]: cluster 2022-04-22T19:17:27.195235+0000 mgr.y (mgr.34107) 3835 : cluster [DBG] pgmap v5283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:29 smithi115 bash[45495]: audit 2022-04-22T19:17:28.331581+0000 mon.a (mon.0) 6649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:29 smithi115 bash[46934]: audit 2022-04-22T19:17:28.331581+0000 mon.a (mon.0) 6649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:29 smithi121 bash[36902]: audit 2022-04-22T19:17:28.331581+0000 mon.a (mon.0) 6649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:30 smithi115 bash[45495]: cluster 2022-04-22T19:17:29.196315+0000 mgr.y (mgr.34107) 3836 : cluster [DBG] pgmap v5284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:30.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:30 smithi115 bash[45495]: audit 2022-04-22T19:17:29.948914+0000 mon.a (mon.0) 6650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:30.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:30 smithi115 bash[46934]: cluster 2022-04-22T19:17:29.196315+0000 mgr.y (mgr.34107) 3836 : cluster [DBG] pgmap v5284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:30.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:30 smithi115 bash[46934]: audit 2022-04-22T19:17:29.948914+0000 mon.a (mon.0) 6650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:30 smithi121 bash[36902]: cluster 2022-04-22T19:17:29.196315+0000 mgr.y (mgr.34107) 3836 : cluster [DBG] pgmap v5284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:30 smithi121 bash[36902]: audit 2022-04-22T19:17:29.948914+0000 mon.a (mon.0) 6650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:32 smithi121 bash[36902]: cluster 2022-04-22T19:17:31.197423+0000 mgr.y (mgr.34107) 3837 : cluster [DBG] pgmap v5285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:32 smithi121 bash[36902]: audit 2022-04-22T19:17:32.168647+0000 mon.a (mon.0) 6651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:32 smithi121 bash[36902]: audit 2022-04-22T19:17:32.171682+0000 mon.a (mon.0) 6652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:32 smithi121 bash[36902]: audit 2022-04-22T19:17:32.173549+0000 mon.a (mon.0) 6653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:17:32.532 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:32.040Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:32.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[46934]: cluster 2022-04-22T19:17:31.197423+0000 mgr.y (mgr.34107) 3837 : cluster [DBG] pgmap v5285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:32.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[46934]: audit 2022-04-22T19:17:32.168647+0000 mon.a (mon.0) 6651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:17:32.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[46934]: audit 2022-04-22T19:17:32.171682+0000 mon.a (mon.0) 6652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:17:32.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[46934]: audit 2022-04-22T19:17:32.173549+0000 mon.a (mon.0) 6653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:17:32.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[45495]: cluster 2022-04-22T19:17:31.197423+0000 mgr.y (mgr.34107) 3837 : cluster [DBG] pgmap v5285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:32.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[45495]: audit 2022-04-22T19:17:32.168647+0000 mon.a (mon.0) 6651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:17:32.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[45495]: audit 2022-04-22T19:17:32.171682+0000 mon.a (mon.0) 6652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:17:32.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:32 smithi115 bash[45495]: audit 2022-04-22T19:17:32.173549+0000 mon.a (mon.0) 6653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:17:33.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[45495]: audit 2022-04-22T19:17:32.537407+0000 mon.a (mon.0) 6654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:17:33.920 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[45495]: cluster 2022-04-22T19:17:33.198354+0000 mgr.y (mgr.34107) 3838 : cluster [DBG] pgmap v5286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:33.921 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[45495]: audit 2022-04-22T19:17:33.335091+0000 mon.a (mon.0) 6655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:33.921 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[46934]: audit 2022-04-22T19:17:32.537407+0000 mon.a (mon.0) 6654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:17:33.921 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[46934]: cluster 2022-04-22T19:17:33.198354+0000 mgr.y (mgr.34107) 3838 : cluster [DBG] pgmap v5286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:33.921 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[46934]: audit 2022-04-22T19:17:33.335091+0000 mon.a (mon.0) 6655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:33 smithi121 bash[36902]: audit 2022-04-22T19:17:32.537407+0000 mon.a (mon.0) 6654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:17:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:33 smithi121 bash[36902]: cluster 2022-04-22T19:17:33.198354+0000 mgr.y (mgr.34107) 3838 : cluster [DBG] pgmap v5286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:34.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:33 smithi121 bash[36902]: audit 2022-04-22T19:17:33.335091+0000 mon.a (mon.0) 6655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:33.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:35.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:35 smithi115 bash[45495]: audit 2022-04-22T19:17:34.952188+0000 mon.a (mon.0) 6656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:35.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:35 smithi115 bash[46934]: audit 2022-04-22T19:17:34.952188+0000 mon.a (mon.0) 6656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:35 smithi121 bash[36902]: audit 2022-04-22T19:17:34.952188+0000 mon.a (mon.0) 6656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:36.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:36 smithi115 bash[45495]: cluster 2022-04-22T19:17:35.199516+0000 mgr.y (mgr.34107) 3839 : cluster [DBG] pgmap v5287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:36 smithi115 bash[46934]: cluster 2022-04-22T19:17:35.199516+0000 mgr.y (mgr.34107) 3839 : cluster [DBG] pgmap v5287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:36 smithi121 bash[36902]: cluster 2022-04-22T19:17:35.199516+0000 mgr.y (mgr.34107) 3839 : cluster [DBG] pgmap v5287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:38.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:38 smithi115 bash[45495]: cluster 2022-04-22T19:17:37.200519+0000 mgr.y (mgr.34107) 3840 : cluster [DBG] pgmap v5288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:38.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:38 smithi115 bash[46934]: cluster 2022-04-22T19:17:37.200519+0000 mgr.y (mgr.34107) 3840 : cluster [DBG] pgmap v5288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:38 smithi121 bash[36902]: cluster 2022-04-22T19:17:37.200519+0000 mgr.y (mgr.34107) 3840 : cluster [DBG] pgmap v5288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:39 smithi115 bash[45495]: audit 2022-04-22T19:17:38.338671+0000 mon.a (mon.0) 6657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:39 smithi115 bash[46934]: audit 2022-04-22T19:17:38.338671+0000 mon.a (mon.0) 6657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:39 smithi121 bash[36902]: audit 2022-04-22T19:17:38.338671+0000 mon.a (mon.0) 6657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:39.871 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:17:40.213 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:17:40.213 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (100m) 7m ago 114m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:17:40.213 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (114m) 8m ago 114m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (102m) 7m ago 115m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (102m) 8m ago 115m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (109m) 8m ago 119m 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (110m) 7m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (109m) 7m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:17:40.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (108m) 8m ago 2h 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (109m) 7m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (101m) 7m ago 114m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (101m) 8m ago 114m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (107m) 7m ago 119m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (106m) 7m ago 118m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:17:40.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (106m) 7m ago 118m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (105m) 7m ago 117m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (105m) 8m ago 117m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (104m) 8m ago 117m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (103m) 8m ago 116m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (103m) 8m ago 116m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:17:40.216 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (101m) 8m ago 114m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:17:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:40 smithi115 bash[46934]: cluster 2022-04-22T19:17:39.201545+0000 mgr.y (mgr.34107) 3841 : cluster [DBG] pgmap v5289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:40 smithi115 bash[46934]: audit 2022-04-22T19:17:39.954985+0000 mon.a (mon.0) 6658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:40 smithi115 bash[45495]: cluster 2022-04-22T19:17:39.201545+0000 mgr.y (mgr.34107) 3841 : cluster [DBG] pgmap v5289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:40 smithi115 bash[45495]: audit 2022-04-22T19:17:39.954985+0000 mon.a (mon.0) 6658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:17:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:40 smithi121 bash[36902]: cluster 2022-04-22T19:17:39.201545+0000 mgr.y (mgr.34107) 3841 : cluster [DBG] pgmap v5289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:40 smithi121 bash[36902]: audit 2022-04-22T19:17:39.954985+0000 mon.a (mon.0) 6658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[45495]: audit 2022-04-22T19:17:39.844248+0000 mgr.y (mgr.34107) 3842 : audit [DBG] from='client.39420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[45495]: audit 2022-04-22T19:17:40.182804+0000 mgr.y (mgr.34107) 3843 : audit [DBG] from='client.49360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[45495]: audit 2022-04-22T19:17:40.602105+0000 mon.a (mon.0) 6659 : audit [DBG] from='client.? 172.21.15.115:0/1645047034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[46934]: audit 2022-04-22T19:17:39.844248+0000 mgr.y (mgr.34107) 3842 : audit [DBG] from='client.39420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[46934]: audit 2022-04-22T19:17:40.182804+0000 mgr.y (mgr.34107) 3843 : audit [DBG] from='client.49360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:41 smithi115 bash[46934]: audit 2022-04-22T19:17:40.602105+0000 mon.a (mon.0) 6659 : audit [DBG] from='client.? 172.21.15.115:0/1645047034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:41 smithi121 bash[36902]: audit 2022-04-22T19:17:39.844248+0000 mgr.y (mgr.34107) 3842 : audit [DBG] from='client.39420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:41 smithi121 bash[36902]: audit 2022-04-22T19:17:40.182804+0000 mgr.y (mgr.34107) 3843 : audit [DBG] from='client.49360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:17:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:41 smithi121 bash[36902]: audit 2022-04-22T19:17:40.602105+0000 mon.a (mon.0) 6659 : audit [DBG] from='client.? 172.21.15.115:0/1645047034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:17:42.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:42 smithi115 bash[45495]: cluster 2022-04-22T19:17:41.202395+0000 mgr.y (mgr.34107) 3844 : cluster [DBG] pgmap v5290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:42.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:42 smithi115 bash[46934]: cluster 2022-04-22T19:17:41.202395+0000 mgr.y (mgr.34107) 3844 : cluster [DBG] pgmap v5290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:42.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:42 smithi121 bash[36902]: cluster 2022-04-22T19:17:41.202395+0000 mgr.y (mgr.34107) 3844 : cluster [DBG] pgmap v5290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:43.922 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:43 smithi115 bash[46934]: cluster 2022-04-22T19:17:43.203579+0000 mgr.y (mgr.34107) 3845 : cluster [DBG] pgmap v5291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:43.922 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:43 smithi115 bash[46934]: audit 2022-04-22T19:17:43.342325+0000 mon.a (mon.0) 6660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:43.922 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:43 smithi115 bash[45495]: cluster 2022-04-22T19:17:43.203579+0000 mgr.y (mgr.34107) 3845 : cluster [DBG] pgmap v5291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:43.923 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:43 smithi115 bash[45495]: audit 2022-04-22T19:17:43.342325+0000 mon.a (mon.0) 6660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:43 smithi121 bash[36902]: cluster 2022-04-22T19:17:43.203579+0000 mgr.y (mgr.34107) 3845 : cluster [DBG] pgmap v5291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:43 smithi121 bash[36902]: audit 2022-04-22T19:17:43.342325+0000 mon.a (mon.0) 6660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:43.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:45 smithi115 bash[45495]: audit 2022-04-22T19:17:44.957947+0000 mon.a (mon.0) 6661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:45.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:45 smithi115 bash[46934]: audit 2022-04-22T19:17:44.957947+0000 mon.a (mon.0) 6661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:45 smithi121 bash[36902]: audit 2022-04-22T19:17:44.957947+0000 mon.a (mon.0) 6661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:46 smithi115 bash[45495]: cluster 2022-04-22T19:17:45.204688+0000 mgr.y (mgr.34107) 3846 : cluster [DBG] pgmap v5292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:46 smithi115 bash[46934]: cluster 2022-04-22T19:17:45.204688+0000 mgr.y (mgr.34107) 3846 : cluster [DBG] pgmap v5292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:46 smithi121 bash[36902]: cluster 2022-04-22T19:17:45.204688+0000 mgr.y (mgr.34107) 3846 : cluster [DBG] pgmap v5292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:48 smithi115 bash[46934]: cluster 2022-04-22T19:17:47.205742+0000 mgr.y (mgr.34107) 3847 : cluster [DBG] pgmap v5293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:48.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:48 smithi115 bash[45495]: cluster 2022-04-22T19:17:47.205742+0000 mgr.y (mgr.34107) 3847 : cluster [DBG] pgmap v5293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:48 smithi121 bash[36902]: cluster 2022-04-22T19:17:47.205742+0000 mgr.y (mgr.34107) 3847 : cluster [DBG] pgmap v5293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:49 smithi115 bash[46934]: audit 2022-04-22T19:17:48.346334+0000 mon.a (mon.0) 6662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:49 smithi115 bash[45495]: audit 2022-04-22T19:17:48.346334+0000 mon.a (mon.0) 6662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:49 smithi121 bash[36902]: audit 2022-04-22T19:17:48.346334+0000 mon.a (mon.0) 6662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:50 smithi115 bash[45495]: cluster 2022-04-22T19:17:49.206801+0000 mgr.y (mgr.34107) 3848 : cluster [DBG] pgmap v5294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:50.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:50 smithi115 bash[45495]: audit 2022-04-22T19:17:49.961059+0000 mon.a (mon.0) 6663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:50 smithi115 bash[46934]: cluster 2022-04-22T19:17:49.206801+0000 mgr.y (mgr.34107) 3848 : cluster [DBG] pgmap v5294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:50 smithi115 bash[46934]: audit 2022-04-22T19:17:49.961059+0000 mon.a (mon.0) 6663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:50 smithi121 bash[36902]: cluster 2022-04-22T19:17:49.206801+0000 mgr.y (mgr.34107) 3848 : cluster [DBG] pgmap v5294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:50 smithi121 bash[36902]: audit 2022-04-22T19:17:49.961059+0000 mon.a (mon.0) 6663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:52.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:52 smithi115 bash[45495]: cluster 2022-04-22T19:17:51.207876+0000 mgr.y (mgr.34107) 3849 : cluster [DBG] pgmap v5295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:52.042Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:52.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:52 smithi115 bash[46934]: cluster 2022-04-22T19:17:51.207876+0000 mgr.y (mgr.34107) 3849 : cluster [DBG] pgmap v5295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:52 smithi121 bash[36902]: cluster 2022-04-22T19:17:51.207876+0000 mgr.y (mgr.34107) 3849 : cluster [DBG] pgmap v5295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:53.922 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:53 smithi115 bash[46934]: cluster 2022-04-22T19:17:53.208926+0000 mgr.y (mgr.34107) 3850 : cluster [DBG] pgmap v5296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:53.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:53 smithi115 bash[46934]: audit 2022-04-22T19:17:53.349917+0000 mon.a (mon.0) 6664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:53.923 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:53 smithi115 bash[45495]: cluster 2022-04-22T19:17:53.208926+0000 mgr.y (mgr.34107) 3850 : cluster [DBG] pgmap v5296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:53.923 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:53 smithi115 bash[45495]: audit 2022-04-22T19:17:53.349917+0000 mon.a (mon.0) 6664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:53 smithi121 bash[36902]: cluster 2022-04-22T19:17:53.208926+0000 mgr.y (mgr.34107) 3850 : cluster [DBG] pgmap v5296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:53 smithi121 bash[36902]: audit 2022-04-22T19:17:53.349917+0000 mon.a (mon.0) 6664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:17:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:17:53.911Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:17:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:55 smithi115 bash[45495]: audit 2022-04-22T19:17:54.964056+0000 mon.a (mon.0) 6665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:55.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:55 smithi115 bash[46934]: audit 2022-04-22T19:17:54.964056+0000 mon.a (mon.0) 6665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:55 smithi121 bash[36902]: audit 2022-04-22T19:17:54.964056+0000 mon.a (mon.0) 6665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:17:56.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:56 smithi115 bash[45495]: cluster 2022-04-22T19:17:55.210026+0000 mgr.y (mgr.34107) 3851 : cluster [DBG] pgmap v5297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:56.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:56 smithi115 bash[46934]: cluster 2022-04-22T19:17:55.210026+0000 mgr.y (mgr.34107) 3851 : cluster [DBG] pgmap v5297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:56 smithi121 bash[36902]: cluster 2022-04-22T19:17:55.210026+0000 mgr.y (mgr.34107) 3851 : cluster [DBG] pgmap v5297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:58 smithi115 bash[46934]: cluster 2022-04-22T19:17:57.211142+0000 mgr.y (mgr.34107) 3852 : cluster [DBG] pgmap v5298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:58.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:58 smithi115 bash[45495]: cluster 2022-04-22T19:17:57.211142+0000 mgr.y (mgr.34107) 3852 : cluster [DBG] pgmap v5298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:58 smithi121 bash[36902]: cluster 2022-04-22T19:17:57.211142+0000 mgr.y (mgr.34107) 3852 : cluster [DBG] pgmap v5298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:17:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:17:59 smithi115 bash[46934]: audit 2022-04-22T19:17:58.353650+0000 mon.a (mon.0) 6666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:59.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:17:59 smithi115 bash[45495]: audit 2022-04-22T19:17:58.353650+0000 mon.a (mon.0) 6666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:17:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:17:59 smithi121 bash[36902]: audit 2022-04-22T19:17:58.353650+0000 mon.a (mon.0) 6666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:00.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:00 smithi115 bash[45495]: cluster 2022-04-22T19:17:59.212152+0000 mgr.y (mgr.34107) 3853 : cluster [DBG] pgmap v5299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:00.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:00 smithi115 bash[45495]: audit 2022-04-22T19:17:59.967073+0000 mon.a (mon.0) 6667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:00 smithi115 bash[46934]: cluster 2022-04-22T19:17:59.212152+0000 mgr.y (mgr.34107) 3853 : cluster [DBG] pgmap v5299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:00.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:00 smithi115 bash[46934]: audit 2022-04-22T19:17:59.967073+0000 mon.a (mon.0) 6667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:00 smithi121 bash[36902]: cluster 2022-04-22T19:17:59.212152+0000 mgr.y (mgr.34107) 3853 : cluster [DBG] pgmap v5299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:00 smithi121 bash[36902]: audit 2022-04-22T19:17:59.967073+0000 mon.a (mon.0) 6667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:02.330 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:02.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:02.330 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:02 smithi115 bash[45495]: cluster 2022-04-22T19:18:01.213109+0000 mgr.y (mgr.34107) 3854 : cluster [DBG] pgmap v5300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:02 smithi121 bash[36902]: cluster 2022-04-22T19:18:01.213109+0000 mgr.y (mgr.34107) 3854 : cluster [DBG] pgmap v5300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:02.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:02 smithi115 bash[46934]: cluster 2022-04-22T19:18:01.213109+0000 mgr.y (mgr.34107) 3854 : cluster [DBG] pgmap v5300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:03.922 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:03 smithi115 bash[45495]: cluster 2022-04-22T19:18:03.214225+0000 mgr.y (mgr.34107) 3855 : cluster [DBG] pgmap v5301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:03.922 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:03 smithi115 bash[45495]: audit 2022-04-22T19:18:03.357334+0000 mon.a (mon.0) 6668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:03.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:03 smithi115 bash[46934]: cluster 2022-04-22T19:18:03.214225+0000 mgr.y (mgr.34107) 3855 : cluster [DBG] pgmap v5301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:03.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:03 smithi115 bash[46934]: audit 2022-04-22T19:18:03.357334+0000 mon.a (mon.0) 6668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:03 smithi121 bash[36902]: cluster 2022-04-22T19:18:03.214225+0000 mgr.y (mgr.34107) 3855 : cluster [DBG] pgmap v5301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:03 smithi121 bash[36902]: audit 2022-04-22T19:18:03.357334+0000 mon.a (mon.0) 6668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:03.911Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:05 smithi115 bash[45495]: audit 2022-04-22T19:18:04.970147+0000 mon.a (mon.0) 6669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:05 smithi115 bash[46934]: audit 2022-04-22T19:18:04.970147+0000 mon.a (mon.0) 6669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:05 smithi121 bash[36902]: audit 2022-04-22T19:18:04.970147+0000 mon.a (mon.0) 6669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:06 smithi115 bash[46934]: cluster 2022-04-22T19:18:05.215271+0000 mgr.y (mgr.34107) 3856 : cluster [DBG] pgmap v5302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:06.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:06 smithi115 bash[45495]: cluster 2022-04-22T19:18:05.215271+0000 mgr.y (mgr.34107) 3856 : cluster [DBG] pgmap v5302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:06 smithi121 bash[36902]: cluster 2022-04-22T19:18:05.215271+0000 mgr.y (mgr.34107) 3856 : cluster [DBG] pgmap v5302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:08.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:08 smithi115 bash[45495]: cluster 2022-04-22T19:18:07.216322+0000 mgr.y (mgr.34107) 3857 : cluster [DBG] pgmap v5303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:08 smithi115 bash[46934]: cluster 2022-04-22T19:18:07.216322+0000 mgr.y (mgr.34107) 3857 : cluster [DBG] pgmap v5303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:08 smithi121 bash[36902]: cluster 2022-04-22T19:18:07.216322+0000 mgr.y (mgr.34107) 3857 : cluster [DBG] pgmap v5303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:09 smithi115 bash[45495]: audit 2022-04-22T19:18:08.360968+0000 mon.a (mon.0) 6670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:09 smithi115 bash[46934]: audit 2022-04-22T19:18:08.360968+0000 mon.a (mon.0) 6670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:09 smithi121 bash[36902]: audit 2022-04-22T19:18:08.360968+0000 mon.a (mon.0) 6670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:10 smithi115 bash[45495]: cluster 2022-04-22T19:18:09.217368+0000 mgr.y (mgr.34107) 3858 : cluster [DBG] pgmap v5304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:10.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:10 smithi115 bash[45495]: audit 2022-04-22T19:18:09.973424+0000 mon.a (mon.0) 6671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:10 smithi115 bash[46934]: cluster 2022-04-22T19:18:09.217368+0000 mgr.y (mgr.34107) 3858 : cluster [DBG] pgmap v5304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:10 smithi115 bash[46934]: audit 2022-04-22T19:18:09.973424+0000 mon.a (mon.0) 6671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:10 smithi121 bash[36902]: cluster 2022-04-22T19:18:09.217368+0000 mgr.y (mgr.34107) 3858 : cluster [DBG] pgmap v5304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:10 smithi121 bash[36902]: audit 2022-04-22T19:18:09.973424+0000 mon.a (mon.0) 6671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:10.990 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (101m) 7m ago 115m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (114m) 8m ago 114m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (103m) 7m ago 116m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (102m) 8m ago 116m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (110m) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:18:11.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (111m) 7m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (110m) 7m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (109m) 8m ago 2h 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (109m) 7m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (102m) 7m ago 115m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (102m) 8m ago 115m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (107m) 7m ago 119m 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (107m) 7m ago 119m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:18:11.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (106m) 7m ago 118m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:18:11.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (106m) 7m ago 118m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:18:11.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (105m) 8m ago 118m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:18:11.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (105m) 8m ago 117m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:18:11.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (104m) 8m ago 117m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:18:11.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (103m) 8m ago 116m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:18:11.339 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (101m) 8m ago 115m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:18:11.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:18:11.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:11.737 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:18:11.737 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:18:11.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:18:11.739 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:18:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:12.042Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[45495]: audit 2022-04-22T19:18:10.962990+0000 mgr.y (mgr.34107) 3859 : audit [DBG] from='client.49372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[45495]: cluster 2022-04-22T19:18:11.218414+0000 mgr.y (mgr.34107) 3860 : cluster [DBG] pgmap v5305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[45495]: audit 2022-04-22T19:18:11.303496+0000 mgr.y (mgr.34107) 3861 : audit [DBG] from='client.39444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[45495]: audit 2022-04-22T19:18:11.724222+0000 mon.a (mon.0) 6672 : audit [DBG] from='client.? 172.21.15.115:0/1086481880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[46934]: audit 2022-04-22T19:18:10.962990+0000 mgr.y (mgr.34107) 3859 : audit [DBG] from='client.49372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[46934]: cluster 2022-04-22T19:18:11.218414+0000 mgr.y (mgr.34107) 3860 : cluster [DBG] pgmap v5305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[46934]: audit 2022-04-22T19:18:11.303496+0000 mgr.y (mgr.34107) 3861 : audit [DBG] from='client.39444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:12 smithi115 bash[46934]: audit 2022-04-22T19:18:11.724222+0000 mon.a (mon.0) 6672 : audit [DBG] from='client.? 172.21.15.115:0/1086481880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:12 smithi121 bash[36902]: audit 2022-04-22T19:18:10.962990+0000 mgr.y (mgr.34107) 3859 : audit [DBG] from='client.49372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:12 smithi121 bash[36902]: cluster 2022-04-22T19:18:11.218414+0000 mgr.y (mgr.34107) 3860 : cluster [DBG] pgmap v5305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:12 smithi121 bash[36902]: audit 2022-04-22T19:18:11.303496+0000 mgr.y (mgr.34107) 3861 : audit [DBG] from='client.39444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:12 smithi121 bash[36902]: audit 2022-04-22T19:18:11.724222+0000 mon.a (mon.0) 6672 : audit [DBG] from='client.? 172.21.15.115:0/1086481880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:13.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:13 smithi115 bash[45495]: cluster 2022-04-22T19:18:13.219550+0000 mgr.y (mgr.34107) 3862 : cluster [DBG] pgmap v5306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:13.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:13 smithi115 bash[45495]: audit 2022-04-22T19:18:13.364631+0000 mon.a (mon.0) 6673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:13.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:13 smithi115 bash[46934]: cluster 2022-04-22T19:18:13.219550+0000 mgr.y (mgr.34107) 3862 : cluster [DBG] pgmap v5306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:13.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:13 smithi115 bash[46934]: audit 2022-04-22T19:18:13.364631+0000 mon.a (mon.0) 6673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:13 smithi121 bash[36902]: cluster 2022-04-22T19:18:13.219550+0000 mgr.y (mgr.34107) 3862 : cluster [DBG] pgmap v5306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:13 smithi121 bash[36902]: audit 2022-04-22T19:18:13.364631+0000 mon.a (mon.0) 6673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:13.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:14 smithi121 bash[36902]: audit 2022-04-22T19:18:14.545725+0000 mon.a (mon.0) 6674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:18:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:14 smithi121 bash[36902]: audit 2022-04-22T19:18:14.546259+0000 mon.a (mon.0) 6675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:18:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:14 smithi115 bash[45495]: audit 2022-04-22T19:18:14.545725+0000 mon.a (mon.0) 6674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:18:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:14 smithi115 bash[45495]: audit 2022-04-22T19:18:14.546259+0000 mon.a (mon.0) 6675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:18:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:14 smithi115 bash[46934]: audit 2022-04-22T19:18:14.545725+0000 mon.a (mon.0) 6674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:18:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:14 smithi115 bash[46934]: audit 2022-04-22T19:18:14.546259+0000 mon.a (mon.0) 6675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:18:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:15 smithi121 bash[36902]: audit 2022-04-22T19:18:14.976648+0000 mon.a (mon.0) 6676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:15 smithi121 bash[36902]: cluster 2022-04-22T19:18:15.220518+0000 mgr.y (mgr.34107) 3863 : cluster [DBG] pgmap v5307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:15 smithi115 bash[45495]: audit 2022-04-22T19:18:14.976648+0000 mon.a (mon.0) 6676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:15 smithi115 bash[45495]: cluster 2022-04-22T19:18:15.220518+0000 mgr.y (mgr.34107) 3863 : cluster [DBG] pgmap v5307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:15 smithi115 bash[46934]: audit 2022-04-22T19:18:14.976648+0000 mon.a (mon.0) 6676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:15 smithi115 bash[46934]: cluster 2022-04-22T19:18:15.220518+0000 mgr.y (mgr.34107) 3863 : cluster [DBG] pgmap v5307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:18.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:18 smithi115 bash[45495]: cluster 2022-04-22T19:18:17.221538+0000 mgr.y (mgr.34107) 3864 : cluster [DBG] pgmap v5308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:18.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:18 smithi115 bash[46934]: cluster 2022-04-22T19:18:17.221538+0000 mgr.y (mgr.34107) 3864 : cluster [DBG] pgmap v5308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:18 smithi121 bash[36902]: cluster 2022-04-22T19:18:17.221538+0000 mgr.y (mgr.34107) 3864 : cluster [DBG] pgmap v5308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:19.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:19 smithi115 bash[45495]: audit 2022-04-22T19:18:18.368752+0000 mon.a (mon.0) 6677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:19.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:19 smithi115 bash[46934]: audit 2022-04-22T19:18:18.368752+0000 mon.a (mon.0) 6677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:19.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:19 smithi121 bash[36902]: audit 2022-04-22T19:18:18.368752+0000 mon.a (mon.0) 6677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:20 smithi121 bash[36902]: cluster 2022-04-22T19:18:19.222598+0000 mgr.y (mgr.34107) 3865 : cluster [DBG] pgmap v5309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:20.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:20 smithi121 bash[36902]: audit 2022-04-22T19:18:19.979923+0000 mon.a (mon.0) 6678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:20.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:20 smithi115 bash[45495]: cluster 2022-04-22T19:18:19.222598+0000 mgr.y (mgr.34107) 3865 : cluster [DBG] pgmap v5309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:20.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:20 smithi115 bash[45495]: audit 2022-04-22T19:18:19.979923+0000 mon.a (mon.0) 6678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:20 smithi115 bash[46934]: cluster 2022-04-22T19:18:19.222598+0000 mgr.y (mgr.34107) 3865 : cluster [DBG] pgmap v5309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:20 smithi115 bash[46934]: audit 2022-04-22T19:18:19.979923+0000 mon.a (mon.0) 6678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:22.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:22 smithi115 bash[45495]: cluster 2022-04-22T19:18:21.223657+0000 mgr.y (mgr.34107) 3866 : cluster [DBG] pgmap v5310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:22.307 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:22.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:22 smithi121 bash[36902]: cluster 2022-04-22T19:18:21.223657+0000 mgr.y (mgr.34107) 3866 : cluster [DBG] pgmap v5310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:22.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:22 smithi115 bash[46934]: cluster 2022-04-22T19:18:21.223657+0000 mgr.y (mgr.34107) 3866 : cluster [DBG] pgmap v5310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:23.923 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:23 smithi115 bash[45495]: cluster 2022-04-22T19:18:23.224711+0000 mgr.y (mgr.34107) 3867 : cluster [DBG] pgmap v5311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:23.924 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:23 smithi115 bash[45495]: audit 2022-04-22T19:18:23.374609+0000 mon.a (mon.0) 6679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:23.924 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:23 smithi115 bash[46934]: cluster 2022-04-22T19:18:23.224711+0000 mgr.y (mgr.34107) 3867 : cluster [DBG] pgmap v5311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:23.924 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:23 smithi115 bash[46934]: audit 2022-04-22T19:18:23.374609+0000 mon.a (mon.0) 6679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:23 smithi121 bash[36902]: cluster 2022-04-22T19:18:23.224711+0000 mgr.y (mgr.34107) 3867 : cluster [DBG] pgmap v5311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:24.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:23 smithi121 bash[36902]: audit 2022-04-22T19:18:23.374609+0000 mon.a (mon.0) 6679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:23.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:25 smithi121 bash[36902]: audit 2022-04-22T19:18:24.983334+0000 mon.a (mon.0) 6680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:25 smithi115 bash[45495]: audit 2022-04-22T19:18:24.983334+0000 mon.a (mon.0) 6680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:25 smithi115 bash[46934]: audit 2022-04-22T19:18:24.983334+0000 mon.a (mon.0) 6680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:26 smithi121 bash[36902]: cluster 2022-04-22T19:18:25.225819+0000 mgr.y (mgr.34107) 3868 : cluster [DBG] pgmap v5312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:26 smithi115 bash[45495]: cluster 2022-04-22T19:18:25.225819+0000 mgr.y (mgr.34107) 3868 : cluster [DBG] pgmap v5312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:26.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:26 smithi115 bash[46934]: cluster 2022-04-22T19:18:25.225819+0000 mgr.y (mgr.34107) 3868 : cluster [DBG] pgmap v5312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:28 smithi115 bash[45495]: cluster 2022-04-22T19:18:27.226884+0000 mgr.y (mgr.34107) 3869 : cluster [DBG] pgmap v5313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:28.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:28 smithi115 bash[46934]: cluster 2022-04-22T19:18:27.226884+0000 mgr.y (mgr.34107) 3869 : cluster [DBG] pgmap v5313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:28 smithi121 bash[36902]: cluster 2022-04-22T19:18:27.226884+0000 mgr.y (mgr.34107) 3869 : cluster [DBG] pgmap v5313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:29 smithi121 bash[36902]: audit 2022-04-22T19:18:28.378249+0000 mon.a (mon.0) 6681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:29 smithi115 bash[45495]: audit 2022-04-22T19:18:28.378249+0000 mon.a (mon.0) 6681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:29 smithi115 bash[46934]: audit 2022-04-22T19:18:28.378249+0000 mon.a (mon.0) 6681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:30 smithi121 bash[36902]: cluster 2022-04-22T19:18:29.228008+0000 mgr.y (mgr.34107) 3870 : cluster [DBG] pgmap v5314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:30 smithi121 bash[36902]: audit 2022-04-22T19:18:29.986581+0000 mon.a (mon.0) 6682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:30.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:30 smithi115 bash[45495]: cluster 2022-04-22T19:18:29.228008+0000 mgr.y (mgr.34107) 3870 : cluster [DBG] pgmap v5314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:30 smithi115 bash[45495]: audit 2022-04-22T19:18:29.986581+0000 mon.a (mon.0) 6682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:30 smithi115 bash[46934]: cluster 2022-04-22T19:18:29.228008+0000 mgr.y (mgr.34107) 3870 : cluster [DBG] pgmap v5314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:30 smithi115 bash[46934]: audit 2022-04-22T19:18:29.986581+0000 mon.a (mon.0) 6682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:32.308 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:32.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:32.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:32 smithi115 bash[46934]: cluster 2022-04-22T19:18:31.229100+0000 mgr.y (mgr.34107) 3871 : cluster [DBG] pgmap v5315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:32.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:32 smithi115 bash[45495]: cluster 2022-04-22T19:18:31.229100+0000 mgr.y (mgr.34107) 3871 : cluster [DBG] pgmap v5315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:32.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:32 smithi121 bash[36902]: cluster 2022-04-22T19:18:31.229100+0000 mgr.y (mgr.34107) 3871 : cluster [DBG] pgmap v5315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:33 smithi121 bash[36902]: audit 2022-04-22T19:18:32.543995+0000 mon.a (mon.0) 6683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:18:33.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:33 smithi121 bash[36902]: audit 2022-04-22T19:18:32.546968+0000 mon.a (mon.0) 6684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:18:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:33 smithi121 bash[36902]: audit 2022-04-22T19:18:32.548801+0000 mon.a (mon.0) 6685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:18:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:33 smithi121 bash[36902]: audit 2022-04-22T19:18:32.924505+0000 mon.a (mon.0) 6686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:18:33.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[45495]: audit 2022-04-22T19:18:32.543995+0000 mon.a (mon.0) 6683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:18:33.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[45495]: audit 2022-04-22T19:18:32.546968+0000 mon.a (mon.0) 6684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:18:33.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[45495]: audit 2022-04-22T19:18:32.548801+0000 mon.a (mon.0) 6685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:18:33.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[45495]: audit 2022-04-22T19:18:32.924505+0000 mon.a (mon.0) 6686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:18:33.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[46934]: audit 2022-04-22T19:18:32.543995+0000 mon.a (mon.0) 6683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:18:33.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[46934]: audit 2022-04-22T19:18:32.546968+0000 mon.a (mon.0) 6684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:18:33.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[46934]: audit 2022-04-22T19:18:32.548801+0000 mon.a (mon.0) 6685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:18:33.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[46934]: audit 2022-04-22T19:18:32.924505+0000 mon.a (mon.0) 6686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:18:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:33.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:34 smithi121 bash[36902]: cluster 2022-04-22T19:18:33.230256+0000 mgr.y (mgr.34107) 3872 : cluster [DBG] pgmap v5316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:34 smithi121 bash[36902]: audit 2022-04-22T19:18:33.381851+0000 mon.a (mon.0) 6687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:34.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:34 smithi115 bash[45495]: cluster 2022-04-22T19:18:33.230256+0000 mgr.y (mgr.34107) 3872 : cluster [DBG] pgmap v5316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:34.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:34 smithi115 bash[45495]: audit 2022-04-22T19:18:33.381851+0000 mon.a (mon.0) 6687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:34.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:34 smithi115 bash[46934]: cluster 2022-04-22T19:18:33.230256+0000 mgr.y (mgr.34107) 3872 : cluster [DBG] pgmap v5316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:34.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:34 smithi115 bash[46934]: audit 2022-04-22T19:18:33.381851+0000 mon.a (mon.0) 6687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:35 smithi121 bash[36902]: audit 2022-04-22T19:18:34.989871+0000 mon.a (mon.0) 6688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:35 smithi115 bash[45495]: audit 2022-04-22T19:18:34.989871+0000 mon.a (mon.0) 6688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:35 smithi115 bash[46934]: audit 2022-04-22T19:18:34.989871+0000 mon.a (mon.0) 6688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:36.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:36 smithi121 bash[36902]: cluster 2022-04-22T19:18:35.231316+0000 mgr.y (mgr.34107) 3873 : cluster [DBG] pgmap v5317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:36.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:36 smithi115 bash[45495]: cluster 2022-04-22T19:18:35.231316+0000 mgr.y (mgr.34107) 3873 : cluster [DBG] pgmap v5317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:36.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:36 smithi115 bash[46934]: cluster 2022-04-22T19:18:35.231316+0000 mgr.y (mgr.34107) 3873 : cluster [DBG] pgmap v5317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:38 smithi121 bash[36902]: cluster 2022-04-22T19:18:37.232381+0000 mgr.y (mgr.34107) 3874 : cluster [DBG] pgmap v5318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:38 smithi115 bash[45495]: cluster 2022-04-22T19:18:37.232381+0000 mgr.y (mgr.34107) 3874 : cluster [DBG] pgmap v5318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:38.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:38 smithi115 bash[46934]: cluster 2022-04-22T19:18:37.232381+0000 mgr.y (mgr.34107) 3874 : cluster [DBG] pgmap v5318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:39 smithi121 bash[36902]: audit 2022-04-22T19:18:38.385594+0000 mon.a (mon.0) 6689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:39 smithi115 bash[45495]: audit 2022-04-22T19:18:38.385594+0000 mon.a (mon.0) 6689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:39 smithi115 bash[46934]: audit 2022-04-22T19:18:38.385594+0000 mon.a (mon.0) 6689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:40 smithi121 bash[36902]: cluster 2022-04-22T19:18:39.233425+0000 mgr.y (mgr.34107) 3875 : cluster [DBG] pgmap v5319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:40 smithi121 bash[36902]: audit 2022-04-22T19:18:39.992950+0000 mon.a (mon.0) 6690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:40 smithi115 bash[45495]: cluster 2022-04-22T19:18:39.233425+0000 mgr.y (mgr.34107) 3875 : cluster [DBG] pgmap v5319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:40 smithi115 bash[45495]: audit 2022-04-22T19:18:39.992950+0000 mon.a (mon.0) 6690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:40 smithi115 bash[46934]: cluster 2022-04-22T19:18:39.233425+0000 mgr.y (mgr.34107) 3875 : cluster [DBG] pgmap v5319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:40 smithi115 bash[46934]: audit 2022-04-22T19:18:39.992950+0000 mon.a (mon.0) 6690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:42.115 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:18:42.342 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:42.044Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:42.461 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:18:42.461 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (101m) 8m ago 115m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:18:42.461 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (115m) 9m ago 115m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:18:42.461 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (103m) 8m ago 116m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (103m) 9m ago 116m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (111m) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (111m) 8m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (110m) 8m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (109m) 9m ago 2h 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (110m) 8m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (102m) 8m ago 115m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (102m) 9m ago 115m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (108m) 8m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:18:42.462 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (107m) 8m ago 119m 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (107m) 8m ago 119m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (106m) 8m ago 118m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (106m) 9m ago 118m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (105m) 9m ago 118m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (104m) 9m ago 117m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (104m) 9m ago 117m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:18:42.463 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (102m) 9m ago 115m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:18:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:42 smithi121 bash[36902]: cluster 2022-04-22T19:18:41.234498+0000 mgr.y (mgr.34107) 3876 : cluster [DBG] pgmap v5320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:42.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:42 smithi115 bash[45495]: cluster 2022-04-22T19:18:41.234498+0000 mgr.y (mgr.34107) 3876 : cluster [DBG] pgmap v5320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:42.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:42 smithi115 bash[46934]: cluster 2022-04-22T19:18:41.234498+0000 mgr.y (mgr.34107) 3876 : cluster [DBG] pgmap v5320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:18:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:18:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:18:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:18:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:18:42.877 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:18:42.877 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:18:42.877 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:18:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:43 smithi121 bash[36902]: audit 2022-04-22T19:18:42.088021+0000 mgr.y (mgr.34107) 3877 : audit [DBG] from='client.39456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:43 smithi121 bash[36902]: audit 2022-04-22T19:18:42.863119+0000 mon.a (mon.0) 6691 : audit [DBG] from='client.? 172.21.15.115:0/622870920' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:43 smithi115 bash[46934]: audit 2022-04-22T19:18:42.088021+0000 mgr.y (mgr.34107) 3877 : audit [DBG] from='client.39456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:43 smithi115 bash[46934]: audit 2022-04-22T19:18:42.863119+0000 mon.a (mon.0) 6691 : audit [DBG] from='client.? 172.21.15.115:0/622870920' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:43.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:43 smithi115 bash[45495]: audit 2022-04-22T19:18:42.088021+0000 mgr.y (mgr.34107) 3877 : audit [DBG] from='client.39456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:43.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:43 smithi115 bash[45495]: audit 2022-04-22T19:18:42.863119+0000 mon.a (mon.0) 6691 : audit [DBG] from='client.? 172.21.15.115:0/622870920' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:18:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:43.913Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:44 smithi121 bash[36902]: audit 2022-04-22T19:18:42.430445+0000 mgr.y (mgr.34107) 3878 : audit [DBG] from='client.49396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:44 smithi121 bash[36902]: cluster 2022-04-22T19:18:43.235620+0000 mgr.y (mgr.34107) 3879 : cluster [DBG] pgmap v5321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:44 smithi121 bash[36902]: audit 2022-04-22T19:18:43.389382+0000 mon.a (mon.0) 6692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[45495]: audit 2022-04-22T19:18:42.430445+0000 mgr.y (mgr.34107) 3878 : audit [DBG] from='client.49396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[45495]: cluster 2022-04-22T19:18:43.235620+0000 mgr.y (mgr.34107) 3879 : cluster [DBG] pgmap v5321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[45495]: audit 2022-04-22T19:18:43.389382+0000 mon.a (mon.0) 6692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[46934]: audit 2022-04-22T19:18:42.430445+0000 mgr.y (mgr.34107) 3878 : audit [DBG] from='client.49396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[46934]: cluster 2022-04-22T19:18:43.235620+0000 mgr.y (mgr.34107) 3879 : cluster [DBG] pgmap v5321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:44 smithi115 bash[46934]: audit 2022-04-22T19:18:43.389382+0000 mon.a (mon.0) 6692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:45 smithi121 bash[36902]: audit 2022-04-22T19:18:44.996177+0000 mon.a (mon.0) 6693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:45 smithi115 bash[45495]: audit 2022-04-22T19:18:44.996177+0000 mon.a (mon.0) 6693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:45 smithi115 bash[46934]: audit 2022-04-22T19:18:44.996177+0000 mon.a (mon.0) 6693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:46 smithi121 bash[36902]: cluster 2022-04-22T19:18:45.236643+0000 mgr.y (mgr.34107) 3880 : cluster [DBG] pgmap v5322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:46.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:46 smithi115 bash[45495]: cluster 2022-04-22T19:18:45.236643+0000 mgr.y (mgr.34107) 3880 : cluster [DBG] pgmap v5322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:46.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:46 smithi115 bash[46934]: cluster 2022-04-22T19:18:45.236643+0000 mgr.y (mgr.34107) 3880 : cluster [DBG] pgmap v5322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:48 smithi121 bash[36902]: cluster 2022-04-22T19:18:47.237703+0000 mgr.y (mgr.34107) 3881 : cluster [DBG] pgmap v5323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:48 smithi115 bash[45495]: cluster 2022-04-22T19:18:47.237703+0000 mgr.y (mgr.34107) 3881 : cluster [DBG] pgmap v5323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:48 smithi115 bash[46934]: cluster 2022-04-22T19:18:47.237703+0000 mgr.y (mgr.34107) 3881 : cluster [DBG] pgmap v5323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:49 smithi121 bash[36902]: audit 2022-04-22T19:18:48.393236+0000 mon.a (mon.0) 6694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:49 smithi115 bash[45495]: audit 2022-04-22T19:18:48.393236+0000 mon.a (mon.0) 6694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:49 smithi115 bash[46934]: audit 2022-04-22T19:18:48.393236+0000 mon.a (mon.0) 6694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:50 smithi121 bash[36902]: cluster 2022-04-22T19:18:49.238778+0000 mgr.y (mgr.34107) 3882 : cluster [DBG] pgmap v5324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:50 smithi121 bash[36902]: audit 2022-04-22T19:18:49.999217+0000 mon.a (mon.0) 6695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:50 smithi115 bash[45495]: cluster 2022-04-22T19:18:49.238778+0000 mgr.y (mgr.34107) 3882 : cluster [DBG] pgmap v5324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:50 smithi115 bash[45495]: audit 2022-04-22T19:18:49.999217+0000 mon.a (mon.0) 6695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:50.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:50 smithi115 bash[46934]: cluster 2022-04-22T19:18:49.238778+0000 mgr.y (mgr.34107) 3882 : cluster [DBG] pgmap v5324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:50.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:50 smithi115 bash[46934]: audit 2022-04-22T19:18:49.999217+0000 mon.a (mon.0) 6695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:51 smithi121 bash[36902]: cluster 2022-04-22T19:18:51.239914+0000 mgr.y (mgr.34107) 3883 : cluster [DBG] pgmap v5325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:51.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:51 smithi115 bash[45495]: cluster 2022-04-22T19:18:51.239914+0000 mgr.y (mgr.34107) 3883 : cluster [DBG] pgmap v5325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:51.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:51 smithi115 bash[46934]: cluster 2022-04-22T19:18:51.239914+0000 mgr.y (mgr.34107) 3883 : cluster [DBG] pgmap v5325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:52.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:52.045Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:53.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:53 smithi115 bash[45495]: cluster 2022-04-22T19:18:53.241007+0000 mgr.y (mgr.34107) 3884 : cluster [DBG] pgmap v5326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:53.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:53 smithi115 bash[45495]: audit 2022-04-22T19:18:53.396940+0000 mon.a (mon.0) 6696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:53.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:53 smithi115 bash[46934]: cluster 2022-04-22T19:18:53.241007+0000 mgr.y (mgr.34107) 3884 : cluster [DBG] pgmap v5326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:53.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:53 smithi115 bash[46934]: audit 2022-04-22T19:18:53.396940+0000 mon.a (mon.0) 6696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:53 smithi121 bash[36902]: cluster 2022-04-22T19:18:53.241007+0000 mgr.y (mgr.34107) 3884 : cluster [DBG] pgmap v5326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:53 smithi121 bash[36902]: audit 2022-04-22T19:18:53.396940+0000 mon.a (mon.0) 6696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:54.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:18:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:18:53.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:18:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:55 smithi121 bash[36902]: audit 2022-04-22T19:18:55.002446+0000 mon.a (mon.0) 6697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:55 smithi115 bash[46934]: audit 2022-04-22T19:18:55.002446+0000 mon.a (mon.0) 6697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:55 smithi115 bash[45495]: audit 2022-04-22T19:18:55.002446+0000 mon.a (mon.0) 6697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:18:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:56 smithi121 bash[36902]: cluster 2022-04-22T19:18:55.242050+0000 mgr.y (mgr.34107) 3885 : cluster [DBG] pgmap v5327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:56 smithi115 bash[46934]: cluster 2022-04-22T19:18:55.242050+0000 mgr.y (mgr.34107) 3885 : cluster [DBG] pgmap v5327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:56.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:56 smithi115 bash[45495]: cluster 2022-04-22T19:18:55.242050+0000 mgr.y (mgr.34107) 3885 : cluster [DBG] pgmap v5327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:58 smithi121 bash[36902]: cluster 2022-04-22T19:18:57.243054+0000 mgr.y (mgr.34107) 3886 : cluster [DBG] pgmap v5328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:58.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:58 smithi115 bash[46934]: cluster 2022-04-22T19:18:57.243054+0000 mgr.y (mgr.34107) 3886 : cluster [DBG] pgmap v5328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:58 smithi115 bash[45495]: cluster 2022-04-22T19:18:57.243054+0000 mgr.y (mgr.34107) 3886 : cluster [DBG] pgmap v5328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:18:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:18:59 smithi121 bash[36902]: audit 2022-04-22T19:18:58.400553+0000 mon.a (mon.0) 6698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:18:59 smithi115 bash[45495]: audit 2022-04-22T19:18:58.400553+0000 mon.a (mon.0) 6698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:18:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:18:59 smithi115 bash[46934]: audit 2022-04-22T19:18:58.400553+0000 mon.a (mon.0) 6698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:00 smithi121 bash[36902]: cluster 2022-04-22T19:18:59.244073+0000 mgr.y (mgr.34107) 3887 : cluster [DBG] pgmap v5329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:00.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:00 smithi121 bash[36902]: audit 2022-04-22T19:19:00.005860+0000 mon.a (mon.0) 6699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:00 smithi115 bash[45495]: cluster 2022-04-22T19:18:59.244073+0000 mgr.y (mgr.34107) 3887 : cluster [DBG] pgmap v5329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:00 smithi115 bash[45495]: audit 2022-04-22T19:19:00.005860+0000 mon.a (mon.0) 6699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:00 smithi115 bash[46934]: cluster 2022-04-22T19:18:59.244073+0000 mgr.y (mgr.34107) 3887 : cluster [DBG] pgmap v5329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:00 smithi115 bash[46934]: audit 2022-04-22T19:19:00.005860+0000 mon.a (mon.0) 6699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:02.325 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:02.046Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:02 smithi121 bash[36902]: cluster 2022-04-22T19:19:01.245172+0000 mgr.y (mgr.34107) 3888 : cluster [DBG] pgmap v5330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:02.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:02 smithi115 bash[45495]: cluster 2022-04-22T19:19:01.245172+0000 mgr.y (mgr.34107) 3888 : cluster [DBG] pgmap v5330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:02.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:02 smithi115 bash[46934]: cluster 2022-04-22T19:19:01.245172+0000 mgr.y (mgr.34107) 3888 : cluster [DBG] pgmap v5330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:03.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:03 smithi115 bash[45495]: cluster 2022-04-22T19:19:03.246318+0000 mgr.y (mgr.34107) 3889 : cluster [DBG] pgmap v5331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:03.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:03 smithi115 bash[45495]: audit 2022-04-22T19:19:03.404041+0000 mon.a (mon.0) 6700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:03.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:03 smithi115 bash[46934]: cluster 2022-04-22T19:19:03.246318+0000 mgr.y (mgr.34107) 3889 : cluster [DBG] pgmap v5331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:03.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:03 smithi115 bash[46934]: audit 2022-04-22T19:19:03.404041+0000 mon.a (mon.0) 6700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:03 smithi121 bash[36902]: cluster 2022-04-22T19:19:03.246318+0000 mgr.y (mgr.34107) 3889 : cluster [DBG] pgmap v5331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:04.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:03 smithi121 bash[36902]: audit 2022-04-22T19:19:03.404041+0000 mon.a (mon.0) 6700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:03.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:05 smithi121 bash[36902]: audit 2022-04-22T19:19:05.009334+0000 mon.a (mon.0) 6701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:05 smithi115 bash[45495]: audit 2022-04-22T19:19:05.009334+0000 mon.a (mon.0) 6701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:05 smithi115 bash[46934]: audit 2022-04-22T19:19:05.009334+0000 mon.a (mon.0) 6701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:06 smithi121 bash[36902]: cluster 2022-04-22T19:19:05.247452+0000 mgr.y (mgr.34107) 3890 : cluster [DBG] pgmap v5332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:06 smithi115 bash[45495]: cluster 2022-04-22T19:19:05.247452+0000 mgr.y (mgr.34107) 3890 : cluster [DBG] pgmap v5332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:06.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:06 smithi115 bash[46934]: cluster 2022-04-22T19:19:05.247452+0000 mgr.y (mgr.34107) 3890 : cluster [DBG] pgmap v5332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:08 smithi121 bash[36902]: cluster 2022-04-22T19:19:07.248484+0000 mgr.y (mgr.34107) 3891 : cluster [DBG] pgmap v5333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:08 smithi115 bash[45495]: cluster 2022-04-22T19:19:07.248484+0000 mgr.y (mgr.34107) 3891 : cluster [DBG] pgmap v5333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:08 smithi115 bash[46934]: cluster 2022-04-22T19:19:07.248484+0000 mgr.y (mgr.34107) 3891 : cluster [DBG] pgmap v5333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:09 smithi121 bash[36902]: audit 2022-04-22T19:19:08.407785+0000 mon.a (mon.0) 6702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:09 smithi115 bash[45495]: audit 2022-04-22T19:19:08.407785+0000 mon.a (mon.0) 6702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:09 smithi115 bash[46934]: audit 2022-04-22T19:19:08.407785+0000 mon.a (mon.0) 6702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:10 smithi121 bash[36902]: cluster 2022-04-22T19:19:09.249523+0000 mgr.y (mgr.34107) 3892 : cluster [DBG] pgmap v5334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:10 smithi121 bash[36902]: audit 2022-04-22T19:19:10.012468+0000 mon.a (mon.0) 6703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:10 smithi115 bash[45495]: cluster 2022-04-22T19:19:09.249523+0000 mgr.y (mgr.34107) 3892 : cluster [DBG] pgmap v5334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:10 smithi115 bash[45495]: audit 2022-04-22T19:19:10.012468+0000 mon.a (mon.0) 6703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:10 smithi115 bash[46934]: cluster 2022-04-22T19:19:09.249523+0000 mgr.y (mgr.34107) 3892 : cluster [DBG] pgmap v5334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:10 smithi115 bash[46934]: audit 2022-04-22T19:19:10.012468+0000 mon.a (mon.0) 6703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:12.331 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:12.046Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:12 smithi121 bash[36902]: cluster 2022-04-22T19:19:11.250612+0000 mgr.y (mgr.34107) 3893 : cluster [DBG] pgmap v5335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:12.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:12 smithi115 bash[46934]: cluster 2022-04-22T19:19:11.250612+0000 mgr.y (mgr.34107) 3893 : cluster [DBG] pgmap v5335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:12 smithi115 bash[45495]: cluster 2022-04-22T19:19:11.250612+0000 mgr.y (mgr.34107) 3893 : cluster [DBG] pgmap v5335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:13.247 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (102m) 8m ago 116m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (115m) 9m ago 115m 25.5M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (104m) 8m ago 117m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (103m) 9m ago 117m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (111m) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (112m) 8m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (111m) 8m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:19:13.593 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (110m) 9m ago 2h 166M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (110m) 8m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (103m) 8m ago 116m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (103m) 9m ago 116m 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (109m) 8m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (108m) 8m ago 2h 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (107m) 8m ago 119m 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (107m) 8m ago 119m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (106m) 9m ago 119m 571M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (106m) 9m ago 118m 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:19:13.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (105m) 9m ago 118m 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:19:13.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (104m) 9m ago 117m 631M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:19:13.595 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (102m) 9m ago 116m 65.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:19:13.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[45495]: audit 2022-04-22T19:19:13.219356+0000 mgr.y (mgr.34107) 3894 : audit [DBG] from='client.39474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:13.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[45495]: cluster 2022-04-22T19:19:13.251467+0000 mgr.y (mgr.34107) 3895 : cluster [DBG] pgmap v5336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:13.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[45495]: audit 2022-04-22T19:19:13.411365+0000 mon.a (mon.0) 6704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:13.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[46934]: audit 2022-04-22T19:19:13.219356+0000 mgr.y (mgr.34107) 3894 : audit [DBG] from='client.39474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:13.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[46934]: cluster 2022-04-22T19:19:13.251467+0000 mgr.y (mgr.34107) 3895 : cluster [DBG] pgmap v5336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:13.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[46934]: audit 2022-04-22T19:19:13.411365+0000 mon.a (mon.0) 6704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:14.005 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:19:14.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:19:14.007 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:19:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:13 smithi121 bash[36902]: audit 2022-04-22T19:19:13.219356+0000 mgr.y (mgr.34107) 3894 : audit [DBG] from='client.39474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:13 smithi121 bash[36902]: cluster 2022-04-22T19:19:13.251467+0000 mgr.y (mgr.34107) 3895 : cluster [DBG] pgmap v5336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:13 smithi121 bash[36902]: audit 2022-04-22T19:19:13.411365+0000 mon.a (mon.0) 6704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:13.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:14 smithi121 bash[36902]: audit 2022-04-22T19:19:13.561902+0000 mgr.y (mgr.34107) 3896 : audit [DBG] from='client.49414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:14 smithi121 bash[36902]: audit 2022-04-22T19:19:13.993755+0000 mon.a (mon.0) 6705 : audit [DBG] from='client.? 172.21.15.115:0/1437794475' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:14 smithi121 bash[36902]: audit 2022-04-22T19:19:14.557227+0000 mon.a (mon.0) 6706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:19:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:14 smithi121 bash[36902]: audit 2022-04-22T19:19:14.558740+0000 mon.a (mon.0) 6707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:19:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[46934]: audit 2022-04-22T19:19:13.561902+0000 mgr.y (mgr.34107) 3896 : audit [DBG] from='client.49414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[46934]: audit 2022-04-22T19:19:13.993755+0000 mon.a (mon.0) 6705 : audit [DBG] from='client.? 172.21.15.115:0/1437794475' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[46934]: audit 2022-04-22T19:19:14.557227+0000 mon.a (mon.0) 6706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:19:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[46934]: audit 2022-04-22T19:19:14.558740+0000 mon.a (mon.0) 6707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:19:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[45495]: audit 2022-04-22T19:19:13.561902+0000 mgr.y (mgr.34107) 3896 : audit [DBG] from='client.49414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[45495]: audit 2022-04-22T19:19:13.993755+0000 mon.a (mon.0) 6705 : audit [DBG] from='client.? 172.21.15.115:0/1437794475' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[45495]: audit 2022-04-22T19:19:14.557227+0000 mon.a (mon.0) 6706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:19:15.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:14 smithi115 bash[45495]: audit 2022-04-22T19:19:14.558740+0000 mon.a (mon.0) 6707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:19:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:15 smithi121 bash[36902]: audit 2022-04-22T19:19:15.015749+0000 mon.a (mon.0) 6708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:15 smithi121 bash[36902]: cluster 2022-04-22T19:19:15.252509+0000 mgr.y (mgr.34107) 3897 : cluster [DBG] pgmap v5337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:15 smithi115 bash[46934]: audit 2022-04-22T19:19:15.015749+0000 mon.a (mon.0) 6708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:15 smithi115 bash[46934]: cluster 2022-04-22T19:19:15.252509+0000 mgr.y (mgr.34107) 3897 : cluster [DBG] pgmap v5337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:15 smithi115 bash[45495]: audit 2022-04-22T19:19:15.015749+0000 mon.a (mon.0) 6708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:15 smithi115 bash[45495]: cluster 2022-04-22T19:19:15.252509+0000 mgr.y (mgr.34107) 3897 : cluster [DBG] pgmap v5337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:18 smithi121 bash[36902]: cluster 2022-04-22T19:19:17.253672+0000 mgr.y (mgr.34107) 3898 : cluster [DBG] pgmap v5338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:18.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:18 smithi115 bash[45495]: cluster 2022-04-22T19:19:17.253672+0000 mgr.y (mgr.34107) 3898 : cluster [DBG] pgmap v5338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:18 smithi115 bash[46934]: cluster 2022-04-22T19:19:17.253672+0000 mgr.y (mgr.34107) 3898 : cluster [DBG] pgmap v5338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:19 smithi121 bash[36902]: audit 2022-04-22T19:19:18.415089+0000 mon.a (mon.0) 6709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:19 smithi115 bash[46934]: audit 2022-04-22T19:19:18.415089+0000 mon.a (mon.0) 6709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:19.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:19 smithi115 bash[45495]: audit 2022-04-22T19:19:18.415089+0000 mon.a (mon.0) 6709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:20 smithi121 bash[36902]: cluster 2022-04-22T19:19:19.254721+0000 mgr.y (mgr.34107) 3899 : cluster [DBG] pgmap v5339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:20 smithi121 bash[36902]: audit 2022-04-22T19:19:20.018892+0000 mon.a (mon.0) 6710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:20 smithi115 bash[46934]: cluster 2022-04-22T19:19:19.254721+0000 mgr.y (mgr.34107) 3899 : cluster [DBG] pgmap v5339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:20 smithi115 bash[46934]: audit 2022-04-22T19:19:20.018892+0000 mon.a (mon.0) 6710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:20 smithi115 bash[45495]: cluster 2022-04-22T19:19:19.254721+0000 mgr.y (mgr.34107) 3899 : cluster [DBG] pgmap v5339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:20 smithi115 bash[45495]: audit 2022-04-22T19:19:20.018892+0000 mon.a (mon.0) 6710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:22.335 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:22.047Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:22 smithi121 bash[36902]: cluster 2022-04-22T19:19:21.255830+0000 mgr.y (mgr.34107) 3900 : cluster [DBG] pgmap v5340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:22 smithi115 bash[45495]: cluster 2022-04-22T19:19:21.255830+0000 mgr.y (mgr.34107) 3900 : cluster [DBG] pgmap v5340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:22 smithi115 bash[46934]: cluster 2022-04-22T19:19:21.255830+0000 mgr.y (mgr.34107) 3900 : cluster [DBG] pgmap v5340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:23.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:23 smithi115 bash[45495]: cluster 2022-04-22T19:19:23.256877+0000 mgr.y (mgr.34107) 3901 : cluster [DBG] pgmap v5341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:23.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:23 smithi115 bash[45495]: audit 2022-04-22T19:19:23.418690+0000 mon.a (mon.0) 6711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:23.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:23 smithi115 bash[46934]: cluster 2022-04-22T19:19:23.256877+0000 mgr.y (mgr.34107) 3901 : cluster [DBG] pgmap v5341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:23.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:23 smithi115 bash[46934]: audit 2022-04-22T19:19:23.418690+0000 mon.a (mon.0) 6711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:23 smithi121 bash[36902]: cluster 2022-04-22T19:19:23.256877+0000 mgr.y (mgr.34107) 3901 : cluster [DBG] pgmap v5341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:23 smithi121 bash[36902]: audit 2022-04-22T19:19:23.418690+0000 mon.a (mon.0) 6711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:24.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:23.916Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:25 smithi121 bash[36902]: audit 2022-04-22T19:19:25.022022+0000 mon.a (mon.0) 6712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:25 smithi115 bash[45495]: audit 2022-04-22T19:19:25.022022+0000 mon.a (mon.0) 6712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:25 smithi115 bash[46934]: audit 2022-04-22T19:19:25.022022+0000 mon.a (mon.0) 6712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:26 smithi121 bash[36902]: cluster 2022-04-22T19:19:25.257866+0000 mgr.y (mgr.34107) 3902 : cluster [DBG] pgmap v5342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:26.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:26 smithi115 bash[45495]: cluster 2022-04-22T19:19:25.257866+0000 mgr.y (mgr.34107) 3902 : cluster [DBG] pgmap v5342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:26.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:26 smithi115 bash[46934]: cluster 2022-04-22T19:19:25.257866+0000 mgr.y (mgr.34107) 3902 : cluster [DBG] pgmap v5342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:28 smithi121 bash[36902]: cluster 2022-04-22T19:19:27.258877+0000 mgr.y (mgr.34107) 3903 : cluster [DBG] pgmap v5343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:28 smithi115 bash[45495]: cluster 2022-04-22T19:19:27.258877+0000 mgr.y (mgr.34107) 3903 : cluster [DBG] pgmap v5343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:28.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:28 smithi115 bash[46934]: cluster 2022-04-22T19:19:27.258877+0000 mgr.y (mgr.34107) 3903 : cluster [DBG] pgmap v5343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:29 smithi121 bash[36902]: audit 2022-04-22T19:19:28.422587+0000 mon.a (mon.0) 6713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:29.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:29 smithi115 bash[45495]: audit 2022-04-22T19:19:28.422587+0000 mon.a (mon.0) 6713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:29 smithi115 bash[46934]: audit 2022-04-22T19:19:28.422587+0000 mon.a (mon.0) 6713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:30 smithi121 bash[36902]: cluster 2022-04-22T19:19:29.259941+0000 mgr.y (mgr.34107) 3904 : cluster [DBG] pgmap v5344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:30 smithi121 bash[36902]: audit 2022-04-22T19:19:30.024798+0000 mon.a (mon.0) 6714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:30 smithi115 bash[45495]: cluster 2022-04-22T19:19:29.259941+0000 mgr.y (mgr.34107) 3904 : cluster [DBG] pgmap v5344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:30 smithi115 bash[45495]: audit 2022-04-22T19:19:30.024798+0000 mon.a (mon.0) 6714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:30 smithi115 bash[46934]: cluster 2022-04-22T19:19:29.259941+0000 mgr.y (mgr.34107) 3904 : cluster [DBG] pgmap v5344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:30 smithi115 bash[46934]: audit 2022-04-22T19:19:30.024798+0000 mon.a (mon.0) 6714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:32.339 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:32.047Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:32 smithi121 bash[36902]: cluster 2022-04-22T19:19:31.261050+0000 mgr.y (mgr.34107) 3905 : cluster [DBG] pgmap v5345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:32.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:32 smithi115 bash[45495]: cluster 2022-04-22T19:19:31.261050+0000 mgr.y (mgr.34107) 3905 : cluster [DBG] pgmap v5345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:32 smithi115 bash[46934]: cluster 2022-04-22T19:19:31.261050+0000 mgr.y (mgr.34107) 3905 : cluster [DBG] pgmap v5345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:33.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[46934]: audit 2022-04-22T19:19:32.931546+0000 mon.a (mon.0) 6715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:19:33.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[46934]: audit 2022-04-22T19:19:32.934649+0000 mon.a (mon.0) 6716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:19:33.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[46934]: audit 2022-04-22T19:19:32.936613+0000 mon.a (mon.0) 6717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:19:33.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[46934]: cluster 2022-04-22T19:19:33.261861+0000 mgr.y (mgr.34107) 3906 : cluster [DBG] pgmap v5346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:33.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[46934]: audit 2022-04-22T19:19:33.426128+0000 mon.a (mon.0) 6718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:33.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[45495]: audit 2022-04-22T19:19:32.931546+0000 mon.a (mon.0) 6715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:19:33.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[45495]: audit 2022-04-22T19:19:32.934649+0000 mon.a (mon.0) 6716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:19:33.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[45495]: audit 2022-04-22T19:19:32.936613+0000 mon.a (mon.0) 6717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:19:33.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[45495]: cluster 2022-04-22T19:19:33.261861+0000 mgr.y (mgr.34107) 3906 : cluster [DBG] pgmap v5346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:33.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[45495]: audit 2022-04-22T19:19:33.426128+0000 mon.a (mon.0) 6718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:33 smithi121 bash[36902]: audit 2022-04-22T19:19:32.931546+0000 mon.a (mon.0) 6715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:33 smithi121 bash[36902]: audit 2022-04-22T19:19:32.934649+0000 mon.a (mon.0) 6716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:33 smithi121 bash[36902]: audit 2022-04-22T19:19:32.936613+0000 mon.a (mon.0) 6717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:33 smithi121 bash[36902]: cluster 2022-04-22T19:19:33.261861+0000 mgr.y (mgr.34107) 3906 : cluster [DBG] pgmap v5346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:33 smithi121 bash[36902]: audit 2022-04-22T19:19:33.426128+0000 mon.a (mon.0) 6718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:33.917Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:35 smithi121 bash[36902]: audit 2022-04-22T19:19:35.027837+0000 mon.a (mon.0) 6719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:35 smithi115 bash[45495]: audit 2022-04-22T19:19:35.027837+0000 mon.a (mon.0) 6719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:35 smithi115 bash[46934]: audit 2022-04-22T19:19:35.027837+0000 mon.a (mon.0) 6719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:36 smithi121 bash[36902]: cluster 2022-04-22T19:19:35.262815+0000 mgr.y (mgr.34107) 3907 : cluster [DBG] pgmap v5347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:36 smithi115 bash[45495]: cluster 2022-04-22T19:19:35.262815+0000 mgr.y (mgr.34107) 3907 : cluster [DBG] pgmap v5347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:36 smithi115 bash[46934]: cluster 2022-04-22T19:19:35.262815+0000 mgr.y (mgr.34107) 3907 : cluster [DBG] pgmap v5347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:38 smithi121 bash[36902]: cluster 2022-04-22T19:19:37.264021+0000 mgr.y (mgr.34107) 3908 : cluster [DBG] pgmap v5348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:38 smithi121 bash[36902]: audit 2022-04-22T19:19:37.327080+0000 mon.a (mon.0) 6720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:38 smithi121 bash[36902]: audit 2022-04-22T19:19:37.690396+0000 mon.a (mon.0) 6721 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[45495]: cluster 2022-04-22T19:19:37.264021+0000 mgr.y (mgr.34107) 3908 : cluster [DBG] pgmap v5348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[45495]: audit 2022-04-22T19:19:37.327080+0000 mon.a (mon.0) 6720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[45495]: audit 2022-04-22T19:19:37.690396+0000 mon.a (mon.0) 6721 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[46934]: cluster 2022-04-22T19:19:37.264021+0000 mgr.y (mgr.34107) 3908 : cluster [DBG] pgmap v5348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[46934]: audit 2022-04-22T19:19:37.327080+0000 mon.a (mon.0) 6720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:38 smithi115 bash[46934]: audit 2022-04-22T19:19:37.690396+0000 mon.a (mon.0) 6721 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:19:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:39 smithi121 bash[36902]: audit 2022-04-22T19:19:38.429969+0000 mon.a (mon.0) 6722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:39.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:39 smithi115 bash[45495]: audit 2022-04-22T19:19:38.429969+0000 mon.a (mon.0) 6722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:39.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:39 smithi115 bash[46934]: audit 2022-04-22T19:19:38.429969+0000 mon.a (mon.0) 6722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:40.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:40 smithi121 bash[36902]: cluster 2022-04-22T19:19:39.265241+0000 mgr.y (mgr.34107) 3909 : cluster [DBG] pgmap v5349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:40.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:40 smithi121 bash[36902]: audit 2022-04-22T19:19:40.031031+0000 mon.a (mon.0) 6723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:40 smithi115 bash[45495]: cluster 2022-04-22T19:19:39.265241+0000 mgr.y (mgr.34107) 3909 : cluster [DBG] pgmap v5349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:40 smithi115 bash[45495]: audit 2022-04-22T19:19:40.031031+0000 mon.a (mon.0) 6723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:40.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:40 smithi115 bash[46934]: cluster 2022-04-22T19:19:39.265241+0000 mgr.y (mgr.34107) 3909 : cluster [DBG] pgmap v5349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:40.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:40 smithi115 bash[46934]: audit 2022-04-22T19:19:40.031031+0000 mon.a (mon.0) 6723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:42.352 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:42.047Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:42.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:42 smithi121 bash[36902]: cluster 2022-04-22T19:19:41.266385+0000 mgr.y (mgr.34107) 3910 : cluster [DBG] pgmap v5350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:42.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:42 smithi115 bash[45495]: cluster 2022-04-22T19:19:41.266385+0000 mgr.y (mgr.34107) 3910 : cluster [DBG] pgmap v5350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:42.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:42 smithi115 bash[46934]: cluster 2022-04-22T19:19:41.266385+0000 mgr.y (mgr.34107) 3910 : cluster [DBG] pgmap v5350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:43.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:43 smithi115 bash[45495]: cluster 2022-04-22T19:19:43.267555+0000 mgr.y (mgr.34107) 3911 : cluster [DBG] pgmap v5351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:43.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:43 smithi115 bash[45495]: audit 2022-04-22T19:19:43.433485+0000 mon.a (mon.0) 6724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:43.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:43 smithi115 bash[46934]: cluster 2022-04-22T19:19:43.267555+0000 mgr.y (mgr.34107) 3911 : cluster [DBG] pgmap v5351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:43.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:43 smithi115 bash[46934]: audit 2022-04-22T19:19:43.433485+0000 mon.a (mon.0) 6724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:43 smithi121 bash[36902]: cluster 2022-04-22T19:19:43.267555+0000 mgr.y (mgr.34107) 3911 : cluster [DBG] pgmap v5351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:43 smithi121 bash[36902]: audit 2022-04-22T19:19:43.433485+0000 mon.a (mon.0) 6724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:43.916Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:44.385 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (102m) 9m ago 116m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (116m) 7s ago 116m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (104m) 9m ago 117m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (104m) 7s ago 117m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (112m) 7s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (112m) 9m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (111m) 9m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:19:44.732 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (110m) 7s ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (111m) 9m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (103m) 9m ago 116m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (103m) 7s ago 116m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (109m) 9m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (108m) 9m ago 2h 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (108m) 9m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (107m) 9m ago 119m 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (107m) 7s ago 119m 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (106m) 7s ago 119m 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:19:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (105m) 7s ago 118m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:19:44.734 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (105m) 7s ago 118m 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:19:44.734 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (103m) 7s ago 116m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:19:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:44 smithi121 bash[36902]: audit 2022-04-22T19:19:44.356033+0000 mgr.y (mgr.34107) 3912 : audit [DBG] from='client.39492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:45.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:44 smithi115 bash[46934]: audit 2022-04-22T19:19:44.356033+0000 mgr.y (mgr.34107) 3912 : audit [DBG] from='client.39492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:45.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:44 smithi115 bash[45495]: audit 2022-04-22T19:19:44.356033+0000 mgr.y (mgr.34107) 3912 : audit [DBG] from='client.39492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:45.136 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:19:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:19:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:19:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:19:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:19:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:19:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:19:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:19:45.138 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:19:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:45 smithi121 bash[36902]: audit 2022-04-22T19:19:44.701332+0000 mgr.y (mgr.34107) 3913 : audit [DBG] from='client.49432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:45 smithi121 bash[36902]: audit 2022-04-22T19:19:45.034192+0000 mon.a (mon.0) 6725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:45 smithi121 bash[36902]: audit 2022-04-22T19:19:45.124408+0000 mon.a (mon.0) 6726 : audit [DBG] from='client.? 172.21.15.115:0/4183553381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:45 smithi121 bash[36902]: cluster 2022-04-22T19:19:45.268613+0000 mgr.y (mgr.34107) 3914 : cluster [DBG] pgmap v5352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[45495]: audit 2022-04-22T19:19:44.701332+0000 mgr.y (mgr.34107) 3913 : audit [DBG] from='client.49432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[45495]: audit 2022-04-22T19:19:45.034192+0000 mon.a (mon.0) 6725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[45495]: audit 2022-04-22T19:19:45.124408+0000 mon.a (mon.0) 6726 : audit [DBG] from='client.? 172.21.15.115:0/4183553381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[45495]: cluster 2022-04-22T19:19:45.268613+0000 mgr.y (mgr.34107) 3914 : cluster [DBG] pgmap v5352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:46.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[46934]: audit 2022-04-22T19:19:44.701332+0000 mgr.y (mgr.34107) 3913 : audit [DBG] from='client.49432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:19:46.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[46934]: audit 2022-04-22T19:19:45.034192+0000 mon.a (mon.0) 6725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:46.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[46934]: audit 2022-04-22T19:19:45.124408+0000 mon.a (mon.0) 6726 : audit [DBG] from='client.? 172.21.15.115:0/4183553381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:19:46.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:45 smithi115 bash[46934]: cluster 2022-04-22T19:19:45.268613+0000 mgr.y (mgr.34107) 3914 : cluster [DBG] pgmap v5352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:48 smithi121 bash[36902]: cluster 2022-04-22T19:19:47.269740+0000 mgr.y (mgr.34107) 3915 : cluster [DBG] pgmap v5353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:48 smithi115 bash[45495]: cluster 2022-04-22T19:19:47.269740+0000 mgr.y (mgr.34107) 3915 : cluster [DBG] pgmap v5353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:48 smithi115 bash[46934]: cluster 2022-04-22T19:19:47.269740+0000 mgr.y (mgr.34107) 3915 : cluster [DBG] pgmap v5353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:49 smithi121 bash[36902]: audit 2022-04-22T19:19:48.437592+0000 mon.a (mon.0) 6727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:49 smithi115 bash[45495]: audit 2022-04-22T19:19:48.437592+0000 mon.a (mon.0) 6727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:49 smithi115 bash[46934]: audit 2022-04-22T19:19:48.437592+0000 mon.a (mon.0) 6727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:50 smithi121 bash[36902]: cluster 2022-04-22T19:19:49.270757+0000 mgr.y (mgr.34107) 3916 : cluster [DBG] pgmap v5354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:50 smithi121 bash[36902]: audit 2022-04-22T19:19:50.037129+0000 mon.a (mon.0) 6728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:50 smithi115 bash[46934]: cluster 2022-04-22T19:19:49.270757+0000 mgr.y (mgr.34107) 3916 : cluster [DBG] pgmap v5354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:50 smithi115 bash[46934]: audit 2022-04-22T19:19:50.037129+0000 mon.a (mon.0) 6728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:50 smithi115 bash[45495]: cluster 2022-04-22T19:19:49.270757+0000 mgr.y (mgr.34107) 3916 : cluster [DBG] pgmap v5354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:50 smithi115 bash[45495]: audit 2022-04-22T19:19:50.037129+0000 mon.a (mon.0) 6728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:52.350 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:52.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:52 smithi121 bash[36902]: cluster 2022-04-22T19:19:51.271868+0000 mgr.y (mgr.34107) 3917 : cluster [DBG] pgmap v5355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:52.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:52 smithi115 bash[45495]: cluster 2022-04-22T19:19:51.271868+0000 mgr.y (mgr.34107) 3917 : cluster [DBG] pgmap v5355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:52 smithi115 bash[46934]: cluster 2022-04-22T19:19:51.271868+0000 mgr.y (mgr.34107) 3917 : cluster [DBG] pgmap v5355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:53.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:53 smithi115 bash[45495]: cluster 2022-04-22T19:19:53.273008+0000 mgr.y (mgr.34107) 3918 : cluster [DBG] pgmap v5356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:53.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:53 smithi115 bash[45495]: audit 2022-04-22T19:19:53.441284+0000 mon.a (mon.0) 6729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:53.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:53 smithi115 bash[46934]: cluster 2022-04-22T19:19:53.273008+0000 mgr.y (mgr.34107) 3918 : cluster [DBG] pgmap v5356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:53.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:53 smithi115 bash[46934]: audit 2022-04-22T19:19:53.441284+0000 mon.a (mon.0) 6729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:53 smithi121 bash[36902]: cluster 2022-04-22T19:19:53.273008+0000 mgr.y (mgr.34107) 3918 : cluster [DBG] pgmap v5356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:53 smithi121 bash[36902]: audit 2022-04-22T19:19:53.441284+0000 mon.a (mon.0) 6729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:54.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:19:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:19:53.918Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:19:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:55 smithi121 bash[36902]: audit 2022-04-22T19:19:55.040282+0000 mon.a (mon.0) 6730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:55 smithi115 bash[45495]: audit 2022-04-22T19:19:55.040282+0000 mon.a (mon.0) 6730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:55 smithi115 bash[46934]: audit 2022-04-22T19:19:55.040282+0000 mon.a (mon.0) 6730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:19:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:56 smithi121 bash[36902]: cluster 2022-04-22T19:19:55.274107+0000 mgr.y (mgr.34107) 3919 : cluster [DBG] pgmap v5357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:56 smithi115 bash[45495]: cluster 2022-04-22T19:19:55.274107+0000 mgr.y (mgr.34107) 3919 : cluster [DBG] pgmap v5357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:56 smithi115 bash[46934]: cluster 2022-04-22T19:19:55.274107+0000 mgr.y (mgr.34107) 3919 : cluster [DBG] pgmap v5357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:58 smithi121 bash[36902]: cluster 2022-04-22T19:19:57.275161+0000 mgr.y (mgr.34107) 3920 : cluster [DBG] pgmap v5358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:58 smithi115 bash[45495]: cluster 2022-04-22T19:19:57.275161+0000 mgr.y (mgr.34107) 3920 : cluster [DBG] pgmap v5358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:58.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:58 smithi115 bash[46934]: cluster 2022-04-22T19:19:57.275161+0000 mgr.y (mgr.34107) 3920 : cluster [DBG] pgmap v5358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:19:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:19:59 smithi121 bash[36902]: audit 2022-04-22T19:19:58.444736+0000 mon.a (mon.0) 6731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:19:59 smithi115 bash[45495]: audit 2022-04-22T19:19:58.444736+0000 mon.a (mon.0) 6731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:19:59.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:19:59 smithi115 bash[46934]: audit 2022-04-22T19:19:58.444736+0000 mon.a (mon.0) 6731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:00 smithi121 bash[36902]: cluster 2022-04-22T19:19:59.276252+0000 mgr.y (mgr.34107) 3921 : cluster [DBG] pgmap v5359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:00 smithi121 bash[36902]: cluster 2022-04-22T19:20:00.000119+0000 mon.a (mon.0) 6732 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:20:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:00 smithi121 bash[36902]: audit 2022-04-22T19:20:00.043523+0000 mon.a (mon.0) 6733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[45495]: cluster 2022-04-22T19:19:59.276252+0000 mgr.y (mgr.34107) 3921 : cluster [DBG] pgmap v5359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[45495]: cluster 2022-04-22T19:20:00.000119+0000 mon.a (mon.0) 6732 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:20:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[45495]: audit 2022-04-22T19:20:00.043523+0000 mon.a (mon.0) 6733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[46934]: cluster 2022-04-22T19:19:59.276252+0000 mgr.y (mgr.34107) 3921 : cluster [DBG] pgmap v5359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:00.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[46934]: cluster 2022-04-22T19:20:00.000119+0000 mon.a (mon.0) 6732 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:20:00.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:00 smithi115 bash[46934]: audit 2022-04-22T19:20:00.043523+0000 mon.a (mon.0) 6733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:02.357 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:02.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:02 smithi121 bash[36902]: cluster 2022-04-22T19:20:01.277302+0000 mgr.y (mgr.34107) 3922 : cluster [DBG] pgmap v5360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:02.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:02 smithi115 bash[45495]: cluster 2022-04-22T19:20:01.277302+0000 mgr.y (mgr.34107) 3922 : cluster [DBG] pgmap v5360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:02.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:02 smithi115 bash[46934]: cluster 2022-04-22T19:20:01.277302+0000 mgr.y (mgr.34107) 3922 : cluster [DBG] pgmap v5360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:03.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:03 smithi115 bash[45495]: cluster 2022-04-22T19:20:03.278411+0000 mgr.y (mgr.34107) 3923 : cluster [DBG] pgmap v5361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:03.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:03 smithi115 bash[45495]: audit 2022-04-22T19:20:03.448266+0000 mon.a (mon.0) 6734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:03.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:03 smithi115 bash[46934]: cluster 2022-04-22T19:20:03.278411+0000 mgr.y (mgr.34107) 3923 : cluster [DBG] pgmap v5361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:03.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:03 smithi115 bash[46934]: audit 2022-04-22T19:20:03.448266+0000 mon.a (mon.0) 6734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:03 smithi121 bash[36902]: cluster 2022-04-22T19:20:03.278411+0000 mgr.y (mgr.34107) 3923 : cluster [DBG] pgmap v5361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:03 smithi121 bash[36902]: audit 2022-04-22T19:20:03.448266+0000 mon.a (mon.0) 6734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:03.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:05 smithi121 bash[36902]: audit 2022-04-22T19:20:05.046688+0000 mon.a (mon.0) 6735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:05 smithi115 bash[45495]: audit 2022-04-22T19:20:05.046688+0000 mon.a (mon.0) 6735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:05 smithi115 bash[46934]: audit 2022-04-22T19:20:05.046688+0000 mon.a (mon.0) 6735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:06 smithi121 bash[36902]: cluster 2022-04-22T19:20:05.279494+0000 mgr.y (mgr.34107) 3924 : cluster [DBG] pgmap v5362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:06 smithi115 bash[45495]: cluster 2022-04-22T19:20:05.279494+0000 mgr.y (mgr.34107) 3924 : cluster [DBG] pgmap v5362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:06.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:06 smithi115 bash[46934]: cluster 2022-04-22T19:20:05.279494+0000 mgr.y (mgr.34107) 3924 : cluster [DBG] pgmap v5362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:08 smithi121 bash[36902]: cluster 2022-04-22T19:20:07.280465+0000 mgr.y (mgr.34107) 3925 : cluster [DBG] pgmap v5363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:08 smithi115 bash[45495]: cluster 2022-04-22T19:20:07.280465+0000 mgr.y (mgr.34107) 3925 : cluster [DBG] pgmap v5363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:08 smithi115 bash[46934]: cluster 2022-04-22T19:20:07.280465+0000 mgr.y (mgr.34107) 3925 : cluster [DBG] pgmap v5363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:09 smithi121 bash[36902]: audit 2022-04-22T19:20:08.451845+0000 mon.a (mon.0) 6736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:09 smithi115 bash[46934]: audit 2022-04-22T19:20:08.451845+0000 mon.a (mon.0) 6736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:09 smithi115 bash[45495]: audit 2022-04-22T19:20:08.451845+0000 mon.a (mon.0) 6736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:10 smithi121 bash[36902]: cluster 2022-04-22T19:20:09.281491+0000 mgr.y (mgr.34107) 3926 : cluster [DBG] pgmap v5364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:10 smithi121 bash[36902]: audit 2022-04-22T19:20:10.049797+0000 mon.a (mon.0) 6737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:10 smithi115 bash[45495]: cluster 2022-04-22T19:20:09.281491+0000 mgr.y (mgr.34107) 3926 : cluster [DBG] pgmap v5364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:10 smithi115 bash[45495]: audit 2022-04-22T19:20:10.049797+0000 mon.a (mon.0) 6737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:10 smithi115 bash[46934]: cluster 2022-04-22T19:20:09.281491+0000 mgr.y (mgr.34107) 3926 : cluster [DBG] pgmap v5364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:10.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:10 smithi115 bash[46934]: audit 2022-04-22T19:20:10.049797+0000 mon.a (mon.0) 6737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:12.363 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:12.050Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:12 smithi121 bash[36902]: cluster 2022-04-22T19:20:11.282617+0000 mgr.y (mgr.34107) 3927 : cluster [DBG] pgmap v5365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:12 smithi115 bash[45495]: cluster 2022-04-22T19:20:11.282617+0000 mgr.y (mgr.34107) 3927 : cluster [DBG] pgmap v5365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:12.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:12 smithi115 bash[46934]: cluster 2022-04-22T19:20:11.282617+0000 mgr.y (mgr.34107) 3927 : cluster [DBG] pgmap v5365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:13.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:13 smithi115 bash[45495]: cluster 2022-04-22T19:20:13.283797+0000 mgr.y (mgr.34107) 3928 : cluster [DBG] pgmap v5366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:13.932 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:13 smithi115 bash[45495]: audit 2022-04-22T19:20:13.455533+0000 mon.a (mon.0) 6738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:13.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:13 smithi115 bash[46934]: cluster 2022-04-22T19:20:13.283797+0000 mgr.y (mgr.34107) 3928 : cluster [DBG] pgmap v5366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:13.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:13 smithi115 bash[46934]: audit 2022-04-22T19:20:13.455533+0000 mon.a (mon.0) 6738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:14.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:13 smithi121 bash[36902]: cluster 2022-04-22T19:20:13.283797+0000 mgr.y (mgr.34107) 3928 : cluster [DBG] pgmap v5366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:14.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:13 smithi121 bash[36902]: audit 2022-04-22T19:20:13.455533+0000 mon.a (mon.0) 6738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:13.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:14 smithi121 bash[36902]: audit 2022-04-22T19:20:14.571005+0000 mon.a (mon.0) 6739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:20:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:14 smithi121 bash[36902]: audit 2022-04-22T19:20:14.571452+0000 mon.a (mon.0) 6740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:20:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:14 smithi115 bash[46934]: audit 2022-04-22T19:20:14.571005+0000 mon.a (mon.0) 6739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:20:15.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:14 smithi115 bash[46934]: audit 2022-04-22T19:20:14.571452+0000 mon.a (mon.0) 6740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:20:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:14 smithi115 bash[45495]: audit 2022-04-22T19:20:14.571005+0000 mon.a (mon.0) 6739 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:20:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:14 smithi115 bash[45495]: audit 2022-04-22T19:20:14.571452+0000 mon.a (mon.0) 6740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:20:15.510 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:20:15.852 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (103m) 9m ago 117m 15.3M - ba2b418f427c 9255ba08f72f 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (116m) 38s ago 116m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (105m) 9m ago 118m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (104m) 38s ago 118m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (112m) 38s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (113m) 9m ago 2h 478M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:20:15.853 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (112m) 9m ago 2h 266M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (111m) 38s ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (111m) 9m ago 2h 168M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (104m) 9m ago 117m 10.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (104m) 38s ago 117m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (110m) 9m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (109m) 9m ago 2h 778M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (108m) 9m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (108m) 9m ago 2h 481M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:20:15.854 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (107m) 38s ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:20:15.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (107m) 38s ago 119m 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:20:15.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (106m) 38s ago 119m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:20:15.855 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (105m) 38s ago 118m 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:20:15.855 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (104m) 38s ago 117m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:20:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:15 smithi121 bash[36902]: audit 2022-04-22T19:20:15.052863+0000 mon.a (mon.0) 6741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:15 smithi121 bash[36902]: cluster 2022-04-22T19:20:15.284638+0000 mgr.y (mgr.34107) 3929 : cluster [DBG] pgmap v5367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:15 smithi115 bash[46934]: audit 2022-04-22T19:20:15.052863+0000 mon.a (mon.0) 6741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:15 smithi115 bash[46934]: cluster 2022-04-22T19:20:15.284638+0000 mgr.y (mgr.34107) 3929 : cluster [DBG] pgmap v5367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:15 smithi115 bash[45495]: audit 2022-04-22T19:20:15.052863+0000 mon.a (mon.0) 6741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:15 smithi115 bash[45495]: cluster 2022-04-22T19:20:15.284638+0000 mgr.y (mgr.34107) 3929 : cluster [DBG] pgmap v5367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:16.264 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:20:16.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:20:16.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:20:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:20:16.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:16.266 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:20:16.266 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:20:16.266 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:20:16.266 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:20:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:16 smithi121 bash[36902]: audit 2022-04-22T19:20:15.483233+0000 mgr.y (mgr.34107) 3930 : audit [DBG] from='client.49444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:16 smithi121 bash[36902]: audit 2022-04-22T19:20:15.826338+0000 mgr.y (mgr.34107) 3931 : audit [DBG] from='client.49450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:16 smithi121 bash[36902]: audit 2022-04-22T19:20:16.252791+0000 mon.a (mon.0) 6742 : audit [DBG] from='client.? 172.21.15.115:0/347812277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:17.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[45495]: audit 2022-04-22T19:20:15.483233+0000 mgr.y (mgr.34107) 3930 : audit [DBG] from='client.49444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[45495]: audit 2022-04-22T19:20:15.826338+0000 mgr.y (mgr.34107) 3931 : audit [DBG] from='client.49450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[45495]: audit 2022-04-22T19:20:16.252791+0000 mon.a (mon.0) 6742 : audit [DBG] from='client.? 172.21.15.115:0/347812277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:17.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[46934]: audit 2022-04-22T19:20:15.483233+0000 mgr.y (mgr.34107) 3930 : audit [DBG] from='client.49444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[46934]: audit 2022-04-22T19:20:15.826338+0000 mgr.y (mgr.34107) 3931 : audit [DBG] from='client.49450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:17.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:16 smithi115 bash[46934]: audit 2022-04-22T19:20:16.252791+0000 mon.a (mon.0) 6742 : audit [DBG] from='client.? 172.21.15.115:0/347812277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:17 smithi121 bash[36902]: cluster 2022-04-22T19:20:17.285803+0000 mgr.y (mgr.34107) 3932 : cluster [DBG] pgmap v5368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:18.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:17 smithi115 bash[46934]: cluster 2022-04-22T19:20:17.285803+0000 mgr.y (mgr.34107) 3932 : cluster [DBG] pgmap v5368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:18.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:17 smithi115 bash[45495]: cluster 2022-04-22T19:20:17.285803+0000 mgr.y (mgr.34107) 3932 : cluster [DBG] pgmap v5368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:18 smithi121 bash[36902]: audit 2022-04-22T19:20:18.459094+0000 mon.a (mon.0) 6743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:19.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:18 smithi115 bash[46934]: audit 2022-04-22T19:20:18.459094+0000 mon.a (mon.0) 6743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:19.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:18 smithi115 bash[45495]: audit 2022-04-22T19:20:18.459094+0000 mon.a (mon.0) 6743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:19 smithi121 bash[36902]: cluster 2022-04-22T19:20:19.286933+0000 mgr.y (mgr.34107) 3933 : cluster [DBG] pgmap v5369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:20.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:19 smithi115 bash[45495]: cluster 2022-04-22T19:20:19.286933+0000 mgr.y (mgr.34107) 3933 : cluster [DBG] pgmap v5369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:20.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:19 smithi115 bash[46934]: cluster 2022-04-22T19:20:19.286933+0000 mgr.y (mgr.34107) 3933 : cluster [DBG] pgmap v5369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:20 smithi121 bash[36902]: audit 2022-04-22T19:20:20.055646+0000 mon.a (mon.0) 6744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:20 smithi115 bash[45495]: audit 2022-04-22T19:20:20.055646+0000 mon.a (mon.0) 6744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:21.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:20 smithi115 bash[46934]: audit 2022-04-22T19:20:20.055646+0000 mon.a (mon.0) 6744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:21 smithi121 bash[36902]: cluster 2022-04-22T19:20:21.288066+0000 mgr.y (mgr.34107) 3934 : cluster [DBG] pgmap v5370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:21 smithi115 bash[45495]: cluster 2022-04-22T19:20:21.288066+0000 mgr.y (mgr.34107) 3934 : cluster [DBG] pgmap v5370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:21 smithi115 bash[46934]: cluster 2022-04-22T19:20:21.288066+0000 mgr.y (mgr.34107) 3934 : cluster [DBG] pgmap v5370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:22.049Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:23 smithi121 bash[36902]: cluster 2022-04-22T19:20:23.288916+0000 mgr.y (mgr.34107) 3935 : cluster [DBG] pgmap v5371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:23 smithi121 bash[36902]: audit 2022-04-22T19:20:23.462537+0000 mon.a (mon.0) 6745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:23.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:24.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:23 smithi115 bash[45495]: cluster 2022-04-22T19:20:23.288916+0000 mgr.y (mgr.34107) 3935 : cluster [DBG] pgmap v5371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:24.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:23 smithi115 bash[45495]: audit 2022-04-22T19:20:23.462537+0000 mon.a (mon.0) 6745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:23 smithi115 bash[46934]: cluster 2022-04-22T19:20:23.288916+0000 mgr.y (mgr.34107) 3935 : cluster [DBG] pgmap v5371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:23 smithi115 bash[46934]: audit 2022-04-22T19:20:23.462537+0000 mon.a (mon.0) 6745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:25 smithi121 bash[36902]: audit 2022-04-22T19:20:25.058822+0000 mon.a (mon.0) 6746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:25 smithi115 bash[45495]: audit 2022-04-22T19:20:25.058822+0000 mon.a (mon.0) 6746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:25 smithi115 bash[46934]: audit 2022-04-22T19:20:25.058822+0000 mon.a (mon.0) 6746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:26 smithi121 bash[36902]: cluster 2022-04-22T19:20:25.289977+0000 mgr.y (mgr.34107) 3936 : cluster [DBG] pgmap v5372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:26 smithi115 bash[45495]: cluster 2022-04-22T19:20:25.289977+0000 mgr.y (mgr.34107) 3936 : cluster [DBG] pgmap v5372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:26.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:26 smithi115 bash[46934]: cluster 2022-04-22T19:20:25.289977+0000 mgr.y (mgr.34107) 3936 : cluster [DBG] pgmap v5372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:28 smithi121 bash[36902]: cluster 2022-04-22T19:20:27.291054+0000 mgr.y (mgr.34107) 3937 : cluster [DBG] pgmap v5373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:28 smithi115 bash[45495]: cluster 2022-04-22T19:20:27.291054+0000 mgr.y (mgr.34107) 3937 : cluster [DBG] pgmap v5373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:28.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:28 smithi115 bash[46934]: cluster 2022-04-22T19:20:27.291054+0000 mgr.y (mgr.34107) 3937 : cluster [DBG] pgmap v5373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:29.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:29 smithi121 bash[36902]: audit 2022-04-22T19:20:28.466253+0000 mon.a (mon.0) 6747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:29 smithi115 bash[46934]: audit 2022-04-22T19:20:28.466253+0000 mon.a (mon.0) 6747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:29.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:29 smithi115 bash[45495]: audit 2022-04-22T19:20:28.466253+0000 mon.a (mon.0) 6747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:30 smithi121 bash[36902]: cluster 2022-04-22T19:20:29.292157+0000 mgr.y (mgr.34107) 3938 : cluster [DBG] pgmap v5374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:30 smithi121 bash[36902]: audit 2022-04-22T19:20:30.061953+0000 mon.a (mon.0) 6748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:30 smithi115 bash[45495]: cluster 2022-04-22T19:20:29.292157+0000 mgr.y (mgr.34107) 3938 : cluster [DBG] pgmap v5374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:30 smithi115 bash[45495]: audit 2022-04-22T19:20:30.061953+0000 mon.a (mon.0) 6748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:30 smithi115 bash[46934]: cluster 2022-04-22T19:20:29.292157+0000 mgr.y (mgr.34107) 3938 : cluster [DBG] pgmap v5374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:30 smithi115 bash[46934]: audit 2022-04-22T19:20:30.061953+0000 mon.a (mon.0) 6748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:32.372 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:32.052Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:32 smithi121 bash[36902]: cluster 2022-04-22T19:20:31.292975+0000 mgr.y (mgr.34107) 3939 : cluster [DBG] pgmap v5375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:32.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:32 smithi115 bash[45495]: cluster 2022-04-22T19:20:31.292975+0000 mgr.y (mgr.34107) 3939 : cluster [DBG] pgmap v5375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:32 smithi115 bash[46934]: cluster 2022-04-22T19:20:31.292975+0000 mgr.y (mgr.34107) 3939 : cluster [DBG] pgmap v5375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:33 smithi121 bash[36902]: cluster 2022-04-22T19:20:33.293985+0000 mgr.y (mgr.34107) 3940 : cluster [DBG] pgmap v5376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:33 smithi121 bash[36902]: audit 2022-04-22T19:20:33.469916+0000 mon.a (mon.0) 6749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:34.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:33.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:33 smithi115 bash[45495]: cluster 2022-04-22T19:20:33.293985+0000 mgr.y (mgr.34107) 3940 : cluster [DBG] pgmap v5376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:33 smithi115 bash[45495]: audit 2022-04-22T19:20:33.469916+0000 mon.a (mon.0) 6749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:33 smithi115 bash[46934]: cluster 2022-04-22T19:20:33.293985+0000 mgr.y (mgr.34107) 3940 : cluster [DBG] pgmap v5376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:33 smithi115 bash[46934]: audit 2022-04-22T19:20:33.469916+0000 mon.a (mon.0) 6749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:35 smithi121 bash[36902]: audit 2022-04-22T19:20:35.065070+0000 mon.a (mon.0) 6750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:35 smithi115 bash[45495]: audit 2022-04-22T19:20:35.065070+0000 mon.a (mon.0) 6750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:35 smithi115 bash[46934]: audit 2022-04-22T19:20:35.065070+0000 mon.a (mon.0) 6750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:36 smithi121 bash[36902]: cluster 2022-04-22T19:20:35.295077+0000 mgr.y (mgr.34107) 3941 : cluster [DBG] pgmap v5377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:36 smithi115 bash[45495]: cluster 2022-04-22T19:20:35.295077+0000 mgr.y (mgr.34107) 3941 : cluster [DBG] pgmap v5377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:36 smithi115 bash[46934]: cluster 2022-04-22T19:20:35.295077+0000 mgr.y (mgr.34107) 3941 : cluster [DBG] pgmap v5377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: cluster 2022-04-22T19:20:37.296147+0000 mgr.y (mgr.34107) 3942 : cluster [DBG] pgmap v5378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: audit 2022-04-22T19:20:37.700170+0000 mon.a (mon.0) 6751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: audit 2022-04-22T19:20:37.702982+0000 mon.a (mon.0) 6752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: audit 2022-04-22T19:20:37.704705+0000 mon.a (mon.0) 6753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: audit 2022-04-22T19:20:38.024098+0000 mon.a (mon.0) 6754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:38 smithi121 bash[36902]: audit 2022-04-22T19:20:38.039610+0000 mon.a (mon.0) 6755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: cluster 2022-04-22T19:20:37.296147+0000 mgr.y (mgr.34107) 3942 : cluster [DBG] pgmap v5378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: audit 2022-04-22T19:20:37.700170+0000 mon.a (mon.0) 6751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:20:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: audit 2022-04-22T19:20:37.702982+0000 mon.a (mon.0) 6752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:20:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: audit 2022-04-22T19:20:37.704705+0000 mon.a (mon.0) 6753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:20:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: audit 2022-04-22T19:20:38.024098+0000 mon.a (mon.0) 6754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[45495]: audit 2022-04-22T19:20:38.039610+0000 mon.a (mon.0) 6755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: cluster 2022-04-22T19:20:37.296147+0000 mgr.y (mgr.34107) 3942 : cluster [DBG] pgmap v5378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: audit 2022-04-22T19:20:37.700170+0000 mon.a (mon.0) 6751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: audit 2022-04-22T19:20:37.702982+0000 mon.a (mon.0) 6752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: audit 2022-04-22T19:20:37.704705+0000 mon.a (mon.0) 6753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: audit 2022-04-22T19:20:38.024098+0000 mon.a (mon.0) 6754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:38 smithi115 bash[46934]: audit 2022-04-22T19:20:38.039610+0000 mon.a (mon.0) 6755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:39 smithi121 bash[36902]: audit 2022-04-22T19:20:38.473532+0000 mon.a (mon.0) 6756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:39.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:39 smithi115 bash[45495]: audit 2022-04-22T19:20:38.473532+0000 mon.a (mon.0) 6756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:39 smithi115 bash[46934]: audit 2022-04-22T19:20:38.473532+0000 mon.a (mon.0) 6756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:40 smithi121 bash[36902]: cluster 2022-04-22T19:20:39.297122+0000 mgr.y (mgr.34107) 3943 : cluster [DBG] pgmap v5379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:40 smithi121 bash[36902]: audit 2022-04-22T19:20:40.068395+0000 mon.a (mon.0) 6757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:40 smithi115 bash[45495]: cluster 2022-04-22T19:20:39.297122+0000 mgr.y (mgr.34107) 3943 : cluster [DBG] pgmap v5379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:40.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:40 smithi115 bash[45495]: audit 2022-04-22T19:20:40.068395+0000 mon.a (mon.0) 6757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:40 smithi115 bash[46934]: cluster 2022-04-22T19:20:39.297122+0000 mgr.y (mgr.34107) 3943 : cluster [DBG] pgmap v5379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:40.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:40 smithi115 bash[46934]: audit 2022-04-22T19:20:40.068395+0000 mon.a (mon.0) 6757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:42.380 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:42.052Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:42 smithi121 bash[36902]: cluster 2022-04-22T19:20:41.298150+0000 mgr.y (mgr.34107) 3944 : cluster [DBG] pgmap v5380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:42 smithi121 bash[36902]: audit 2022-04-22T19:20:42.210768+0000 mon.a (mon.0) 6758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:42.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:42 smithi115 bash[45495]: cluster 2022-04-22T19:20:41.298150+0000 mgr.y (mgr.34107) 3944 : cluster [DBG] pgmap v5380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:42.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:42 smithi115 bash[45495]: audit 2022-04-22T19:20:42.210768+0000 mon.a (mon.0) 6758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:42.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:42 smithi115 bash[46934]: cluster 2022-04-22T19:20:41.298150+0000 mgr.y (mgr.34107) 3944 : cluster [DBG] pgmap v5380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:42.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:42 smithi115 bash[46934]: audit 2022-04-22T19:20:42.210768+0000 mon.a (mon.0) 6758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:43.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[45495]: audit 2022-04-22T19:20:42.560588+0000 mon.a (mon.0) 6759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:43.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[45495]: cluster 2022-04-22T19:20:42.679613+0000 mon.a (mon.0) 6760 : cluster [DBG] osdmap e2222: 8 total, 8 up, 8 in 2022-04-22T19:20:43.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[45495]: cluster 2022-04-22T19:20:43.299232+0000 mgr.y (mgr.34107) 3945 : cluster [DBG] pgmap v5382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:43.934 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[45495]: audit 2022-04-22T19:20:43.477069+0000 mon.a (mon.0) 6761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:43.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[46934]: audit 2022-04-22T19:20:42.560588+0000 mon.a (mon.0) 6759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:43.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[46934]: cluster 2022-04-22T19:20:42.679613+0000 mon.a (mon.0) 6760 : cluster [DBG] osdmap e2222: 8 total, 8 up, 8 in 2022-04-22T19:20:43.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[46934]: cluster 2022-04-22T19:20:43.299232+0000 mgr.y (mgr.34107) 3945 : cluster [DBG] pgmap v5382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:43.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[46934]: audit 2022-04-22T19:20:43.477069+0000 mon.a (mon.0) 6761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:43 smithi121 bash[36902]: audit 2022-04-22T19:20:42.560588+0000 mon.a (mon.0) 6759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:43 smithi121 bash[36902]: cluster 2022-04-22T19:20:42.679613+0000 mon.a (mon.0) 6760 : cluster [DBG] osdmap e2222: 8 total, 8 up, 8 in 2022-04-22T19:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:43 smithi121 bash[36902]: cluster 2022-04-22T19:20:43.299232+0000 mgr.y (mgr.34107) 3945 : cluster [DBG] pgmap v5382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:43 smithi121 bash[36902]: audit 2022-04-22T19:20:43.477069+0000 mon.a (mon.0) 6761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:43.921Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:45 smithi121 bash[36902]: audit 2022-04-22T19:20:45.071437+0000 mon.a (mon.0) 6762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:45 smithi115 bash[45495]: audit 2022-04-22T19:20:45.071437+0000 mon.a (mon.0) 6762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:45.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:45 smithi115 bash[46934]: audit 2022-04-22T19:20:45.071437+0000 mon.a (mon.0) 6762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:46 smithi121 bash[36902]: cluster 2022-04-22T19:20:45.300486+0000 mgr.y (mgr.34107) 3946 : cluster [DBG] pgmap v5383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:46 smithi115 bash[46934]: cluster 2022-04-22T19:20:45.300486+0000 mgr.y (mgr.34107) 3946 : cluster [DBG] pgmap v5383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:46.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:46 smithi115 bash[45495]: cluster 2022-04-22T19:20:45.300486+0000 mgr.y (mgr.34107) 3946 : cluster [DBG] pgmap v5383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:46.640 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:20:46.988 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:20:46.988 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (103m) 4s ago 117m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:20:46.988 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (117m) 69s ago 117m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (105m) 4s ago 118m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (105m) 69s ago 118m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (113m) 69s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (113m) 4s ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (112m) 4s ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (111m) 69s ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (112m) 4s ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:20:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (104m) 4s ago 117m 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (104m) 69s ago 117m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (110m) 4s ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (109m) 4s ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (109m) 4s ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (108m) 4s ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (108m) 69s ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (107m) 69s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (106m) 69s ago 119m 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (106m) 69s ago 119m 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:20:46.990 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (104m) 69s ago 117m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:20:47.405 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:20:47.405 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:20:47.405 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:20:47.406 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:20:47.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:20:47.407 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:20:47.407 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:20:47.407 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:20:47.407 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:20:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:48 smithi121 bash[36902]: audit 2022-04-22T19:20:46.611413+0000 mgr.y (mgr.34107) 3947 : audit [DBG] from='client.39528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:48 smithi121 bash[36902]: audit 2022-04-22T19:20:46.955904+0000 mgr.y (mgr.34107) 3948 : audit [DBG] from='client.49468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:48 smithi121 bash[36902]: cluster 2022-04-22T19:20:47.301251+0000 mgr.y (mgr.34107) 3949 : cluster [DBG] pgmap v5384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:48 smithi121 bash[36902]: audit 2022-04-22T19:20:47.393162+0000 mon.a (mon.0) 6763 : audit [DBG] from='client.? 172.21.15.115:0/3779497626' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[45495]: audit 2022-04-22T19:20:46.611413+0000 mgr.y (mgr.34107) 3947 : audit [DBG] from='client.39528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[45495]: audit 2022-04-22T19:20:46.955904+0000 mgr.y (mgr.34107) 3948 : audit [DBG] from='client.49468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[45495]: cluster 2022-04-22T19:20:47.301251+0000 mgr.y (mgr.34107) 3949 : cluster [DBG] pgmap v5384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[45495]: audit 2022-04-22T19:20:47.393162+0000 mon.a (mon.0) 6763 : audit [DBG] from='client.? 172.21.15.115:0/3779497626' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:48.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[46934]: audit 2022-04-22T19:20:46.611413+0000 mgr.y (mgr.34107) 3947 : audit [DBG] from='client.39528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[46934]: audit 2022-04-22T19:20:46.955904+0000 mgr.y (mgr.34107) 3948 : audit [DBG] from='client.49468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:20:48.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[46934]: cluster 2022-04-22T19:20:47.301251+0000 mgr.y (mgr.34107) 3949 : cluster [DBG] pgmap v5384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:48.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:48 smithi115 bash[46934]: audit 2022-04-22T19:20:47.393162+0000 mon.a (mon.0) 6763 : audit [DBG] from='client.? 172.21.15.115:0/3779497626' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:20:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:49 smithi121 bash[36902]: audit 2022-04-22T19:20:48.480740+0000 mon.a (mon.0) 6764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:49 smithi115 bash[45495]: audit 2022-04-22T19:20:48.480740+0000 mon.a (mon.0) 6764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:49 smithi115 bash[46934]: audit 2022-04-22T19:20:48.480740+0000 mon.a (mon.0) 6764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:50 smithi121 bash[36902]: cluster 2022-04-22T19:20:49.302251+0000 mgr.y (mgr.34107) 3950 : cluster [DBG] pgmap v5385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:50 smithi121 bash[36902]: audit 2022-04-22T19:20:50.074524+0000 mon.a (mon.0) 6765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:50 smithi115 bash[45495]: cluster 2022-04-22T19:20:49.302251+0000 mgr.y (mgr.34107) 3950 : cluster [DBG] pgmap v5385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:50 smithi115 bash[45495]: audit 2022-04-22T19:20:50.074524+0000 mon.a (mon.0) 6765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:50 smithi115 bash[46934]: cluster 2022-04-22T19:20:49.302251+0000 mgr.y (mgr.34107) 3950 : cluster [DBG] pgmap v5385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:50.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:50 smithi115 bash[46934]: audit 2022-04-22T19:20:50.074524+0000 mon.a (mon.0) 6765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:52.383 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:52.053Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:52 smithi121 bash[36902]: cluster 2022-04-22T19:20:51.303257+0000 mgr.y (mgr.34107) 3951 : cluster [DBG] pgmap v5386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:52.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:52 smithi115 bash[45495]: cluster 2022-04-22T19:20:51.303257+0000 mgr.y (mgr.34107) 3951 : cluster [DBG] pgmap v5386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:52 smithi115 bash[46934]: cluster 2022-04-22T19:20:51.303257+0000 mgr.y (mgr.34107) 3951 : cluster [DBG] pgmap v5386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:53 smithi121 bash[36902]: cluster 2022-04-22T19:20:53.304361+0000 mgr.y (mgr.34107) 3952 : cluster [DBG] pgmap v5387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:53 smithi121 bash[36902]: audit 2022-04-22T19:20:53.484623+0000 mon.a (mon.0) 6766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:54.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:20:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:20:53.922Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:20:54.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:53 smithi115 bash[45495]: cluster 2022-04-22T19:20:53.304361+0000 mgr.y (mgr.34107) 3952 : cluster [DBG] pgmap v5387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:54.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:53 smithi115 bash[45495]: audit 2022-04-22T19:20:53.484623+0000 mon.a (mon.0) 6766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:53 smithi115 bash[46934]: cluster 2022-04-22T19:20:53.304361+0000 mgr.y (mgr.34107) 3952 : cluster [DBG] pgmap v5387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:53 smithi115 bash[46934]: audit 2022-04-22T19:20:53.484623+0000 mon.a (mon.0) 6766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:55 smithi121 bash[36902]: audit 2022-04-22T19:20:55.077322+0000 mon.a (mon.0) 6767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:55 smithi115 bash[45495]: audit 2022-04-22T19:20:55.077322+0000 mon.a (mon.0) 6767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:55 smithi115 bash[46934]: audit 2022-04-22T19:20:55.077322+0000 mon.a (mon.0) 6767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:20:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:56 smithi121 bash[36902]: cluster 2022-04-22T19:20:55.305323+0000 mgr.y (mgr.34107) 3953 : cluster [DBG] pgmap v5388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:56 smithi115 bash[45495]: cluster 2022-04-22T19:20:55.305323+0000 mgr.y (mgr.34107) 3953 : cluster [DBG] pgmap v5388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:56 smithi115 bash[46934]: cluster 2022-04-22T19:20:55.305323+0000 mgr.y (mgr.34107) 3953 : cluster [DBG] pgmap v5388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:58 smithi121 bash[36902]: cluster 2022-04-22T19:20:57.306374+0000 mgr.y (mgr.34107) 3954 : cluster [DBG] pgmap v5389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:58 smithi115 bash[45495]: cluster 2022-04-22T19:20:57.306374+0000 mgr.y (mgr.34107) 3954 : cluster [DBG] pgmap v5389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:58.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:58 smithi115 bash[46934]: cluster 2022-04-22T19:20:57.306374+0000 mgr.y (mgr.34107) 3954 : cluster [DBG] pgmap v5389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:20:59 smithi121 bash[36902]: audit 2022-04-22T19:20:58.488437+0000 mon.a (mon.0) 6768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:20:59 smithi115 bash[45495]: audit 2022-04-22T19:20:58.488437+0000 mon.a (mon.0) 6768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:20:59.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:20:59 smithi115 bash[46934]: audit 2022-04-22T19:20:58.488437+0000 mon.a (mon.0) 6768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:00 smithi121 bash[36902]: cluster 2022-04-22T19:20:59.307559+0000 mgr.y (mgr.34107) 3955 : cluster [DBG] pgmap v5390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:00 smithi121 bash[36902]: audit 2022-04-22T19:21:00.080469+0000 mon.a (mon.0) 6769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:00 smithi115 bash[45495]: cluster 2022-04-22T19:20:59.307559+0000 mgr.y (mgr.34107) 3955 : cluster [DBG] pgmap v5390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:00 smithi115 bash[45495]: audit 2022-04-22T19:21:00.080469+0000 mon.a (mon.0) 6769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:00 smithi115 bash[46934]: cluster 2022-04-22T19:20:59.307559+0000 mgr.y (mgr.34107) 3955 : cluster [DBG] pgmap v5390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:00.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:00 smithi115 bash[46934]: audit 2022-04-22T19:21:00.080469+0000 mon.a (mon.0) 6769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:02.387 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:02.054Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:02 smithi121 bash[36902]: cluster 2022-04-22T19:21:01.308667+0000 mgr.y (mgr.34107) 3956 : cluster [DBG] pgmap v5391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:02.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:02 smithi115 bash[45495]: cluster 2022-04-22T19:21:01.308667+0000 mgr.y (mgr.34107) 3956 : cluster [DBG] pgmap v5391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:02.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:02 smithi115 bash[46934]: cluster 2022-04-22T19:21:01.308667+0000 mgr.y (mgr.34107) 3956 : cluster [DBG] pgmap v5391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:03 smithi121 bash[36902]: cluster 2022-04-22T19:21:03.309791+0000 mgr.y (mgr.34107) 3957 : cluster [DBG] pgmap v5392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:03 smithi121 bash[36902]: audit 2022-04-22T19:21:03.492106+0000 mon.a (mon.0) 6770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:04.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:03.923Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:04.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:03 smithi115 bash[45495]: cluster 2022-04-22T19:21:03.309791+0000 mgr.y (mgr.34107) 3957 : cluster [DBG] pgmap v5392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:04.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:03 smithi115 bash[45495]: audit 2022-04-22T19:21:03.492106+0000 mon.a (mon.0) 6770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:03 smithi115 bash[46934]: cluster 2022-04-22T19:21:03.309791+0000 mgr.y (mgr.34107) 3957 : cluster [DBG] pgmap v5392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:03 smithi115 bash[46934]: audit 2022-04-22T19:21:03.492106+0000 mon.a (mon.0) 6770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:05 smithi121 bash[36902]: audit 2022-04-22T19:21:05.083626+0000 mon.a (mon.0) 6771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:05 smithi115 bash[45495]: audit 2022-04-22T19:21:05.083626+0000 mon.a (mon.0) 6771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:05 smithi115 bash[46934]: audit 2022-04-22T19:21:05.083626+0000 mon.a (mon.0) 6771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:06 smithi121 bash[36902]: cluster 2022-04-22T19:21:05.310893+0000 mgr.y (mgr.34107) 3958 : cluster [DBG] pgmap v5393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:06 smithi115 bash[45495]: cluster 2022-04-22T19:21:05.310893+0000 mgr.y (mgr.34107) 3958 : cluster [DBG] pgmap v5393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:06.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:06 smithi115 bash[46934]: cluster 2022-04-22T19:21:05.310893+0000 mgr.y (mgr.34107) 3958 : cluster [DBG] pgmap v5393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:08 smithi121 bash[36902]: cluster 2022-04-22T19:21:07.311984+0000 mgr.y (mgr.34107) 3959 : cluster [DBG] pgmap v5394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:08 smithi115 bash[45495]: cluster 2022-04-22T19:21:07.311984+0000 mgr.y (mgr.34107) 3959 : cluster [DBG] pgmap v5394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:08 smithi115 bash[46934]: cluster 2022-04-22T19:21:07.311984+0000 mgr.y (mgr.34107) 3959 : cluster [DBG] pgmap v5394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:09 smithi121 bash[36902]: audit 2022-04-22T19:21:08.496179+0000 mon.a (mon.0) 6772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:09 smithi115 bash[45495]: audit 2022-04-22T19:21:08.496179+0000 mon.a (mon.0) 6772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:09 smithi115 bash[46934]: audit 2022-04-22T19:21:08.496179+0000 mon.a (mon.0) 6772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:10 smithi121 bash[36902]: cluster 2022-04-22T19:21:09.313011+0000 mgr.y (mgr.34107) 3960 : cluster [DBG] pgmap v5395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:10 smithi121 bash[36902]: audit 2022-04-22T19:21:10.086716+0000 mon.a (mon.0) 6773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:10.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:10 smithi115 bash[45495]: cluster 2022-04-22T19:21:09.313011+0000 mgr.y (mgr.34107) 3960 : cluster [DBG] pgmap v5395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:10.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:10 smithi115 bash[45495]: audit 2022-04-22T19:21:10.086716+0000 mon.a (mon.0) 6773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:10.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:10 smithi115 bash[46934]: cluster 2022-04-22T19:21:09.313011+0000 mgr.y (mgr.34107) 3960 : cluster [DBG] pgmap v5395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:10.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:10 smithi115 bash[46934]: audit 2022-04-22T19:21:10.086716+0000 mon.a (mon.0) 6773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:12.401 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:12.054Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:12 smithi121 bash[36902]: cluster 2022-04-22T19:21:11.314057+0000 mgr.y (mgr.34107) 3961 : cluster [DBG] pgmap v5396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:12.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:12 smithi115 bash[45495]: cluster 2022-04-22T19:21:11.314057+0000 mgr.y (mgr.34107) 3961 : cluster [DBG] pgmap v5396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:12.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:12 smithi115 bash[46934]: cluster 2022-04-22T19:21:11.314057+0000 mgr.y (mgr.34107) 3961 : cluster [DBG] pgmap v5396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:13 smithi121 bash[36902]: cluster 2022-04-22T19:21:13.315148+0000 mgr.y (mgr.34107) 3962 : cluster [DBG] pgmap v5397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:13 smithi121 bash[36902]: audit 2022-04-22T19:21:13.499609+0000 mon.a (mon.0) 6774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:14.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:13 smithi115 bash[46934]: cluster 2022-04-22T19:21:13.315148+0000 mgr.y (mgr.34107) 3962 : cluster [DBG] pgmap v5397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:14.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:13 smithi115 bash[46934]: audit 2022-04-22T19:21:13.499609+0000 mon.a (mon.0) 6774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:14.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:13.923Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:13 smithi115 bash[45495]: cluster 2022-04-22T19:21:13.315148+0000 mgr.y (mgr.34107) 3962 : cluster [DBG] pgmap v5397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:13 smithi115 bash[45495]: audit 2022-04-22T19:21:13.499609+0000 mon.a (mon.0) 6774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:14 smithi121 bash[36902]: audit 2022-04-22T19:21:14.582775+0000 mon.a (mon.0) 6775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:21:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:14 smithi121 bash[36902]: audit 2022-04-22T19:21:14.582998+0000 mon.a (mon.0) 6776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:21:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:14 smithi115 bash[45495]: audit 2022-04-22T19:21:14.582775+0000 mon.a (mon.0) 6775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:21:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:14 smithi115 bash[45495]: audit 2022-04-22T19:21:14.582998+0000 mon.a (mon.0) 6776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:21:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:14 smithi115 bash[46934]: audit 2022-04-22T19:21:14.582775+0000 mon.a (mon.0) 6775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:21:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:14 smithi115 bash[46934]: audit 2022-04-22T19:21:14.582998+0000 mon.a (mon.0) 6776 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:21:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:15 smithi121 bash[36902]: audit 2022-04-22T19:21:15.090026+0000 mon.a (mon.0) 6777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:15 smithi121 bash[36902]: cluster 2022-04-22T19:21:15.316231+0000 mgr.y (mgr.34107) 3963 : cluster [DBG] pgmap v5398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:15 smithi115 bash[46934]: audit 2022-04-22T19:21:15.090026+0000 mon.a (mon.0) 6777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:15 smithi115 bash[46934]: cluster 2022-04-22T19:21:15.316231+0000 mgr.y (mgr.34107) 3963 : cluster [DBG] pgmap v5398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:15 smithi115 bash[45495]: audit 2022-04-22T19:21:15.090026+0000 mon.a (mon.0) 6777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:15 smithi115 bash[45495]: cluster 2022-04-22T19:21:15.316231+0000 mgr.y (mgr.34107) 3963 : cluster [DBG] pgmap v5398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:17.785 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:21:18.132 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (104m) 35s ago 118m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (117m) 100s ago 117m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (106m) 35s ago 119m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (105m) 100s ago 119m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (113m) 100s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (114m) 35s ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:21:18.164 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (113m) 35s ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:21:18.165 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (112m) 100s ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:21:18.165 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (112m) 35s ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:21:18.165 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (105m) 35s ago 118m 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:21:18.165 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (105m) 100s ago 118m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (111m) 35s ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (110m) 35s ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (109m) 35s ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (109m) 35s ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (108m) 100s ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (108m) 100s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (107m) 100s ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (106m) 100s ago 119m 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:21:18.173 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (105m) 100s ago 118m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:21:18.537 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:21:18.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:21:18.539 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:21:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:18 smithi121 bash[36902]: cluster 2022-04-22T19:21:17.317389+0000 mgr.y (mgr.34107) 3964 : cluster [DBG] pgmap v5399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:18.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:18 smithi115 bash[46934]: cluster 2022-04-22T19:21:17.317389+0000 mgr.y (mgr.34107) 3964 : cluster [DBG] pgmap v5399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:18.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:18 smithi115 bash[45495]: cluster 2022-04-22T19:21:17.317389+0000 mgr.y (mgr.34107) 3964 : cluster [DBG] pgmap v5399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:19 smithi121 bash[36902]: audit 2022-04-22T19:21:17.757170+0000 mgr.y (mgr.34107) 3965 : audit [DBG] from='client.39546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:19 smithi121 bash[36902]: audit 2022-04-22T19:21:18.101366+0000 mgr.y (mgr.34107) 3966 : audit [DBG] from='client.49486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:19 smithi121 bash[36902]: audit 2022-04-22T19:21:18.502978+0000 mon.a (mon.0) 6778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:19.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:19 smithi121 bash[36902]: audit 2022-04-22T19:21:18.525374+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.115:0/433551923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[46934]: audit 2022-04-22T19:21:17.757170+0000 mgr.y (mgr.34107) 3965 : audit [DBG] from='client.39546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[46934]: audit 2022-04-22T19:21:18.101366+0000 mgr.y (mgr.34107) 3966 : audit [DBG] from='client.49486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[46934]: audit 2022-04-22T19:21:18.502978+0000 mon.a (mon.0) 6778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:19.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[46934]: audit 2022-04-22T19:21:18.525374+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.115:0/433551923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:19.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[45495]: audit 2022-04-22T19:21:17.757170+0000 mgr.y (mgr.34107) 3965 : audit [DBG] from='client.39546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[45495]: audit 2022-04-22T19:21:18.101366+0000 mgr.y (mgr.34107) 3966 : audit [DBG] from='client.49486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:19.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[45495]: audit 2022-04-22T19:21:18.502978+0000 mon.a (mon.0) 6778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:19.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:19 smithi115 bash[45495]: audit 2022-04-22T19:21:18.525374+0000 mon.c (mon.1) 187 : audit [DBG] from='client.? 172.21.15.115:0/433551923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:20 smithi121 bash[36902]: cluster 2022-04-22T19:21:19.318589+0000 mgr.y (mgr.34107) 3967 : cluster [DBG] pgmap v5400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:20 smithi121 bash[36902]: audit 2022-04-22T19:21:20.093113+0000 mon.a (mon.0) 6779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:20 smithi115 bash[46934]: cluster 2022-04-22T19:21:19.318589+0000 mgr.y (mgr.34107) 3967 : cluster [DBG] pgmap v5400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:20 smithi115 bash[46934]: audit 2022-04-22T19:21:20.093113+0000 mon.a (mon.0) 6779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:20.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:20 smithi115 bash[45495]: cluster 2022-04-22T19:21:19.318589+0000 mgr.y (mgr.34107) 3967 : cluster [DBG] pgmap v5400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:20.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:20 smithi115 bash[45495]: audit 2022-04-22T19:21:20.093113+0000 mon.a (mon.0) 6779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:22.401 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:22.053Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:22 smithi121 bash[36902]: cluster 2022-04-22T19:21:21.319721+0000 mgr.y (mgr.34107) 3968 : cluster [DBG] pgmap v5401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:22 smithi115 bash[45495]: cluster 2022-04-22T19:21:21.319721+0000 mgr.y (mgr.34107) 3968 : cluster [DBG] pgmap v5401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:22.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:22 smithi115 bash[46934]: cluster 2022-04-22T19:21:21.319721+0000 mgr.y (mgr.34107) 3968 : cluster [DBG] pgmap v5401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:23 smithi121 bash[36902]: cluster 2022-04-22T19:21:23.320922+0000 mgr.y (mgr.34107) 3969 : cluster [DBG] pgmap v5402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:23 smithi121 bash[36902]: audit 2022-04-22T19:21:23.506395+0000 mon.a (mon.0) 6780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:23.924Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:24.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:23 smithi115 bash[45495]: cluster 2022-04-22T19:21:23.320922+0000 mgr.y (mgr.34107) 3969 : cluster [DBG] pgmap v5402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:24.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:23 smithi115 bash[45495]: audit 2022-04-22T19:21:23.506395+0000 mon.a (mon.0) 6780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:23 smithi115 bash[46934]: cluster 2022-04-22T19:21:23.320922+0000 mgr.y (mgr.34107) 3969 : cluster [DBG] pgmap v5402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:24.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:23 smithi115 bash[46934]: audit 2022-04-22T19:21:23.506395+0000 mon.a (mon.0) 6780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:25 smithi121 bash[36902]: audit 2022-04-22T19:21:25.096549+0000 mon.a (mon.0) 6781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:25 smithi115 bash[45495]: audit 2022-04-22T19:21:25.096549+0000 mon.a (mon.0) 6781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:25 smithi115 bash[46934]: audit 2022-04-22T19:21:25.096549+0000 mon.a (mon.0) 6781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:26 smithi121 bash[36902]: cluster 2022-04-22T19:21:25.322039+0000 mgr.y (mgr.34107) 3970 : cluster [DBG] pgmap v5403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:26 smithi115 bash[45495]: cluster 2022-04-22T19:21:25.322039+0000 mgr.y (mgr.34107) 3970 : cluster [DBG] pgmap v5403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:26 smithi115 bash[46934]: cluster 2022-04-22T19:21:25.322039+0000 mgr.y (mgr.34107) 3970 : cluster [DBG] pgmap v5403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:28 smithi121 bash[36902]: cluster 2022-04-22T19:21:27.322829+0000 mgr.y (mgr.34107) 3971 : cluster [DBG] pgmap v5404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:28.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:28 smithi115 bash[46934]: cluster 2022-04-22T19:21:27.322829+0000 mgr.y (mgr.34107) 3971 : cluster [DBG] pgmap v5404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:28.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:28 smithi115 bash[45495]: cluster 2022-04-22T19:21:27.322829+0000 mgr.y (mgr.34107) 3971 : cluster [DBG] pgmap v5404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:29 smithi121 bash[36902]: audit 2022-04-22T19:21:28.509801+0000 mon.a (mon.0) 6782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:29 smithi115 bash[46934]: audit 2022-04-22T19:21:28.509801+0000 mon.a (mon.0) 6782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:29.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:29 smithi115 bash[45495]: audit 2022-04-22T19:21:28.509801+0000 mon.a (mon.0) 6782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:30 smithi121 bash[36902]: cluster 2022-04-22T19:21:29.323851+0000 mgr.y (mgr.34107) 3972 : cluster [DBG] pgmap v5405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:30 smithi121 bash[36902]: audit 2022-04-22T19:21:30.099682+0000 mon.a (mon.0) 6783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:30 smithi115 bash[46934]: cluster 2022-04-22T19:21:29.323851+0000 mgr.y (mgr.34107) 3972 : cluster [DBG] pgmap v5405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:30.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:30 smithi115 bash[46934]: audit 2022-04-22T19:21:30.099682+0000 mon.a (mon.0) 6783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:30 smithi115 bash[45495]: cluster 2022-04-22T19:21:29.323851+0000 mgr.y (mgr.34107) 3972 : cluster [DBG] pgmap v5405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:30 smithi115 bash[45495]: audit 2022-04-22T19:21:30.099682+0000 mon.a (mon.0) 6783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:32.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:32.054Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:32 smithi121 bash[36902]: cluster 2022-04-22T19:21:31.324905+0000 mgr.y (mgr.34107) 3973 : cluster [DBG] pgmap v5406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:32.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:32 smithi115 bash[45495]: cluster 2022-04-22T19:21:31.324905+0000 mgr.y (mgr.34107) 3973 : cluster [DBG] pgmap v5406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:32 smithi115 bash[46934]: cluster 2022-04-22T19:21:31.324905+0000 mgr.y (mgr.34107) 3973 : cluster [DBG] pgmap v5406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:33 smithi121 bash[36902]: cluster 2022-04-22T19:21:33.326105+0000 mgr.y (mgr.34107) 3974 : cluster [DBG] pgmap v5407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:33 smithi121 bash[36902]: audit 2022-04-22T19:21:33.513765+0000 mon.a (mon.0) 6784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:34.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:33.925Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:33 smithi115 bash[45495]: cluster 2022-04-22T19:21:33.326105+0000 mgr.y (mgr.34107) 3974 : cluster [DBG] pgmap v5407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:34.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:33 smithi115 bash[45495]: audit 2022-04-22T19:21:33.513765+0000 mon.a (mon.0) 6784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:33 smithi115 bash[46934]: cluster 2022-04-22T19:21:33.326105+0000 mgr.y (mgr.34107) 3974 : cluster [DBG] pgmap v5407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:33 smithi115 bash[46934]: audit 2022-04-22T19:21:33.513765+0000 mon.a (mon.0) 6784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:35 smithi121 bash[36902]: audit 2022-04-22T19:21:35.102616+0000 mon.a (mon.0) 6785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:35 smithi115 bash[45495]: audit 2022-04-22T19:21:35.102616+0000 mon.a (mon.0) 6785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:35 smithi115 bash[46934]: audit 2022-04-22T19:21:35.102616+0000 mon.a (mon.0) 6785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:36 smithi121 bash[36902]: cluster 2022-04-22T19:21:35.327311+0000 mgr.y (mgr.34107) 3975 : cluster [DBG] pgmap v5408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:36 smithi115 bash[46934]: cluster 2022-04-22T19:21:35.327311+0000 mgr.y (mgr.34107) 3975 : cluster [DBG] pgmap v5408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:36 smithi115 bash[45495]: cluster 2022-04-22T19:21:35.327311+0000 mgr.y (mgr.34107) 3975 : cluster [DBG] pgmap v5408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:38 smithi121 bash[36902]: cluster 2022-04-22T19:21:37.328368+0000 mgr.y (mgr.34107) 3976 : cluster [DBG] pgmap v5409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:38.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:38 smithi115 bash[45495]: cluster 2022-04-22T19:21:37.328368+0000 mgr.y (mgr.34107) 3976 : cluster [DBG] pgmap v5409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:38.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:38 smithi115 bash[46934]: cluster 2022-04-22T19:21:37.328368+0000 mgr.y (mgr.34107) 3976 : cluster [DBG] pgmap v5409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:39 smithi121 bash[36902]: audit 2022-04-22T19:21:38.517374+0000 mon.a (mon.0) 6786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:39.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:39 smithi115 bash[46934]: audit 2022-04-22T19:21:38.517374+0000 mon.a (mon.0) 6786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:39.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:39 smithi115 bash[45495]: audit 2022-04-22T19:21:38.517374+0000 mon.a (mon.0) 6786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:40 smithi121 bash[36902]: cluster 2022-04-22T19:21:39.329377+0000 mgr.y (mgr.34107) 3977 : cluster [DBG] pgmap v5410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:40 smithi121 bash[36902]: audit 2022-04-22T19:21:40.105749+0000 mon.a (mon.0) 6787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:40 smithi115 bash[46934]: cluster 2022-04-22T19:21:39.329377+0000 mgr.y (mgr.34107) 3977 : cluster [DBG] pgmap v5410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:40 smithi115 bash[46934]: audit 2022-04-22T19:21:40.105749+0000 mon.a (mon.0) 6787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:40.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:40 smithi115 bash[45495]: cluster 2022-04-22T19:21:39.329377+0000 mgr.y (mgr.34107) 3977 : cluster [DBG] pgmap v5410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:40.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:40 smithi115 bash[45495]: audit 2022-04-22T19:21:40.105749+0000 mon.a (mon.0) 6787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:41 smithi121 bash[36902]: cluster 2022-04-22T19:21:41.330512+0000 mgr.y (mgr.34107) 3978 : cluster [DBG] pgmap v5411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:41 smithi115 bash[46934]: cluster 2022-04-22T19:21:41.330512+0000 mgr.y (mgr.34107) 3978 : cluster [DBG] pgmap v5411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:41.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:41 smithi115 bash[45495]: cluster 2022-04-22T19:21:41.330512+0000 mgr.y (mgr.34107) 3978 : cluster [DBG] pgmap v5411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:42.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:42.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[45495]: audit 2022-04-22T19:21:42.568109+0000 mon.a (mon.0) 6788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:21:42.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[45495]: audit 2022-04-22T19:21:42.570848+0000 mon.a (mon.0) 6789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:21:42.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[45495]: audit 2022-04-22T19:21:42.572564+0000 mon.a (mon.0) 6790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:21:42.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[46934]: audit 2022-04-22T19:21:42.568109+0000 mon.a (mon.0) 6788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:21:42.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[46934]: audit 2022-04-22T19:21:42.570848+0000 mon.a (mon.0) 6789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:21:42.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:42 smithi115 bash[46934]: audit 2022-04-22T19:21:42.572564+0000 mon.a (mon.0) 6790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:21:42.933 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:42 smithi121 bash[36902]: audit 2022-04-22T19:21:42.568109+0000 mon.a (mon.0) 6788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:21:42.933 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:42 smithi121 bash[36902]: audit 2022-04-22T19:21:42.570848+0000 mon.a (mon.0) 6789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:21:42.933 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:42 smithi121 bash[36902]: audit 2022-04-22T19:21:42.572564+0000 mon.a (mon.0) 6790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:21:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:43 smithi121 bash[36902]: audit 2022-04-22T19:21:42.935459+0000 mon.a (mon.0) 6791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:21:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:43 smithi121 bash[36902]: cluster 2022-04-22T19:21:43.331461+0000 mgr.y (mgr.34107) 3979 : cluster [DBG] pgmap v5412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:44.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:43 smithi121 bash[36902]: audit 2022-04-22T19:21:43.521112+0000 mon.a (mon.0) 6792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:43.925Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:44.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[45495]: audit 2022-04-22T19:21:42.935459+0000 mon.a (mon.0) 6791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:21:44.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[45495]: cluster 2022-04-22T19:21:43.331461+0000 mgr.y (mgr.34107) 3979 : cluster [DBG] pgmap v5412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:44.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[45495]: audit 2022-04-22T19:21:43.521112+0000 mon.a (mon.0) 6792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:44.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[46934]: audit 2022-04-22T19:21:42.935459+0000 mon.a (mon.0) 6791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:21:44.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[46934]: cluster 2022-04-22T19:21:43.331461+0000 mgr.y (mgr.34107) 3979 : cluster [DBG] pgmap v5412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:44.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:43 smithi115 bash[46934]: audit 2022-04-22T19:21:43.521112+0000 mon.a (mon.0) 6792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:45 smithi121 bash[36902]: audit 2022-04-22T19:21:45.108871+0000 mon.a (mon.0) 6793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:45 smithi115 bash[45495]: audit 2022-04-22T19:21:45.108871+0000 mon.a (mon.0) 6793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:45 smithi115 bash[46934]: audit 2022-04-22T19:21:45.108871+0000 mon.a (mon.0) 6793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:46 smithi121 bash[36902]: cluster 2022-04-22T19:21:45.332513+0000 mgr.y (mgr.34107) 3980 : cluster [DBG] pgmap v5413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:46.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:46 smithi115 bash[45495]: cluster 2022-04-22T19:21:45.332513+0000 mgr.y (mgr.34107) 3980 : cluster [DBG] pgmap v5413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:46.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:46 smithi115 bash[46934]: cluster 2022-04-22T19:21:45.332513+0000 mgr.y (mgr.34107) 3980 : cluster [DBG] pgmap v5413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:48 smithi121 bash[36902]: cluster 2022-04-22T19:21:47.333578+0000 mgr.y (mgr.34107) 3981 : cluster [DBG] pgmap v5414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:48.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:48 smithi115 bash[45495]: cluster 2022-04-22T19:21:47.333578+0000 mgr.y (mgr.34107) 3981 : cluster [DBG] pgmap v5414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:48.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:48 smithi115 bash[46934]: cluster 2022-04-22T19:21:47.333578+0000 mgr.y (mgr.34107) 3981 : cluster [DBG] pgmap v5414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:48.906 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:21:49.253 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:21:49.253 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (104m) 67s ago 118m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:21:49.253 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (118m) 2m ago 118m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:21:49.253 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (106m) 67s ago 119m 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:21:49.253 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (106m) 2m ago 119m 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (114m) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (114m) 67s ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (113m) 67s ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (112m) 2m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (113m) 67s ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (105m) 67s ago 118m 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (105m) 2m ago 118m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (111m) 67s ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (110m) 67s ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:21:49.254 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (110m) 67s ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (110m) 67s ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (109m) 2m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (108m) 2m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (107m) 2m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (107m) 2m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:21:49.255 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (105m) 2m ago 119m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:21:49.656 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:21:49.656 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:21:49.656 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:21:49.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:21:49.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:21:49.658 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:21:49.658 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:21:49.658 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:21:49.658 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:21:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:49 smithi121 bash[36902]: audit 2022-04-22T19:21:48.524494+0000 mon.a (mon.0) 6794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:49.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:49 smithi115 bash[46934]: audit 2022-04-22T19:21:48.524494+0000 mon.a (mon.0) 6794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:49.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:49 smithi115 bash[45495]: audit 2022-04-22T19:21:48.524494+0000 mon.a (mon.0) 6794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:50 smithi121 bash[36902]: audit 2022-04-22T19:21:48.879427+0000 mgr.y (mgr.34107) 3982 : audit [DBG] from='client.49498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:50 smithi121 bash[36902]: audit 2022-04-22T19:21:49.221471+0000 mgr.y (mgr.34107) 3983 : audit [DBG] from='client.49504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:50 smithi121 bash[36902]: cluster 2022-04-22T19:21:49.334606+0000 mgr.y (mgr.34107) 3984 : cluster [DBG] pgmap v5415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:50 smithi121 bash[36902]: audit 2022-04-22T19:21:49.643856+0000 mon.a (mon.0) 6795 : audit [DBG] from='client.? 172.21.15.115:0/1793231202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:50 smithi121 bash[36902]: audit 2022-04-22T19:21:50.111915+0000 mon.a (mon.0) 6796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[45495]: audit 2022-04-22T19:21:48.879427+0000 mgr.y (mgr.34107) 3982 : audit [DBG] from='client.49498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[45495]: audit 2022-04-22T19:21:49.221471+0000 mgr.y (mgr.34107) 3983 : audit [DBG] from='client.49504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[45495]: cluster 2022-04-22T19:21:49.334606+0000 mgr.y (mgr.34107) 3984 : cluster [DBG] pgmap v5415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[45495]: audit 2022-04-22T19:21:49.643856+0000 mon.a (mon.0) 6795 : audit [DBG] from='client.? 172.21.15.115:0/1793231202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:50.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[45495]: audit 2022-04-22T19:21:50.111915+0000 mon.a (mon.0) 6796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:50.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[46934]: audit 2022-04-22T19:21:48.879427+0000 mgr.y (mgr.34107) 3982 : audit [DBG] from='client.49498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[46934]: audit 2022-04-22T19:21:49.221471+0000 mgr.y (mgr.34107) 3983 : audit [DBG] from='client.49504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:21:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[46934]: cluster 2022-04-22T19:21:49.334606+0000 mgr.y (mgr.34107) 3984 : cluster [DBG] pgmap v5415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[46934]: audit 2022-04-22T19:21:49.643856+0000 mon.a (mon.0) 6795 : audit [DBG] from='client.? 172.21.15.115:0/1793231202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:21:50.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:50 smithi115 bash[46934]: audit 2022-04-22T19:21:50.111915+0000 mon.a (mon.0) 6796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:51 smithi121 bash[36902]: cluster 2022-04-22T19:21:51.335730+0000 mgr.y (mgr.34107) 3985 : cluster [DBG] pgmap v5416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:51.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:51 smithi115 bash[45495]: cluster 2022-04-22T19:21:51.335730+0000 mgr.y (mgr.34107) 3985 : cluster [DBG] pgmap v5416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:51.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:51 smithi115 bash[46934]: cluster 2022-04-22T19:21:51.335730+0000 mgr.y (mgr.34107) 3985 : cluster [DBG] pgmap v5416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:52.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:53 smithi121 bash[36902]: cluster 2022-04-22T19:21:53.336796+0000 mgr.y (mgr.34107) 3986 : cluster [DBG] pgmap v5417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:53 smithi121 bash[36902]: audit 2022-04-22T19:21:53.528097+0000 mon.a (mon.0) 6797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:54.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:53 smithi115 bash[45495]: cluster 2022-04-22T19:21:53.336796+0000 mgr.y (mgr.34107) 3986 : cluster [DBG] pgmap v5417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:54.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:53 smithi115 bash[45495]: audit 2022-04-22T19:21:53.528097+0000 mon.a (mon.0) 6797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:54.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:21:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:21:53.927Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:21:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:53 smithi115 bash[46934]: cluster 2022-04-22T19:21:53.336796+0000 mgr.y (mgr.34107) 3986 : cluster [DBG] pgmap v5417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:53 smithi115 bash[46934]: audit 2022-04-22T19:21:53.528097+0000 mon.a (mon.0) 6797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:55 smithi121 bash[36902]: audit 2022-04-22T19:21:55.115200+0000 mon.a (mon.0) 6798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:55.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:55 smithi115 bash[45495]: audit 2022-04-22T19:21:55.115200+0000 mon.a (mon.0) 6798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:55 smithi115 bash[46934]: audit 2022-04-22T19:21:55.115200+0000 mon.a (mon.0) 6798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:21:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:56 smithi121 bash[36902]: cluster 2022-04-22T19:21:55.337978+0000 mgr.y (mgr.34107) 3987 : cluster [DBG] pgmap v5418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:56 smithi115 bash[45495]: cluster 2022-04-22T19:21:55.337978+0000 mgr.y (mgr.34107) 3987 : cluster [DBG] pgmap v5418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:56.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:56 smithi115 bash[46934]: cluster 2022-04-22T19:21:55.337978+0000 mgr.y (mgr.34107) 3987 : cluster [DBG] pgmap v5418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:58 smithi121 bash[36902]: cluster 2022-04-22T19:21:57.339062+0000 mgr.y (mgr.34107) 3988 : cluster [DBG] pgmap v5419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:58 smithi115 bash[45495]: cluster 2022-04-22T19:21:57.339062+0000 mgr.y (mgr.34107) 3988 : cluster [DBG] pgmap v5419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:58.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:58 smithi115 bash[46934]: cluster 2022-04-22T19:21:57.339062+0000 mgr.y (mgr.34107) 3988 : cluster [DBG] pgmap v5419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:21:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:21:59 smithi121 bash[36902]: audit 2022-04-22T19:21:58.531228+0000 mon.a (mon.0) 6799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:21:59 smithi115 bash[45495]: audit 2022-04-22T19:21:58.531228+0000 mon.a (mon.0) 6799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:21:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:21:59 smithi115 bash[46934]: audit 2022-04-22T19:21:58.531228+0000 mon.a (mon.0) 6799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:00 smithi121 bash[36902]: cluster 2022-04-22T19:21:59.340119+0000 mgr.y (mgr.34107) 3989 : cluster [DBG] pgmap v5420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:00 smithi121 bash[36902]: audit 2022-04-22T19:22:00.118130+0000 mon.a (mon.0) 6800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:00 smithi115 bash[46934]: cluster 2022-04-22T19:21:59.340119+0000 mgr.y (mgr.34107) 3989 : cluster [DBG] pgmap v5420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:00 smithi115 bash[46934]: audit 2022-04-22T19:22:00.118130+0000 mon.a (mon.0) 6800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:00.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:00 smithi115 bash[45495]: cluster 2022-04-22T19:21:59.340119+0000 mgr.y (mgr.34107) 3989 : cluster [DBG] pgmap v5420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:00.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:00 smithi115 bash[45495]: audit 2022-04-22T19:22:00.118130+0000 mon.a (mon.0) 6800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:01 smithi121 bash[36902]: cluster 2022-04-22T19:22:01.341304+0000 mgr.y (mgr.34107) 3990 : cluster [DBG] pgmap v5421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:01.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:01 smithi115 bash[45495]: cluster 2022-04-22T19:22:01.341304+0000 mgr.y (mgr.34107) 3990 : cluster [DBG] pgmap v5421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:01.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:01 smithi115 bash[46934]: cluster 2022-04-22T19:22:01.341304+0000 mgr.y (mgr.34107) 3990 : cluster [DBG] pgmap v5421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:02.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:02.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:03 smithi121 bash[36902]: cluster 2022-04-22T19:22:03.342528+0000 mgr.y (mgr.34107) 3991 : cluster [DBG] pgmap v5422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:03 smithi121 bash[36902]: audit 2022-04-22T19:22:03.534986+0000 mon.a (mon.0) 6801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:04.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:03.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:04.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:03 smithi115 bash[45495]: cluster 2022-04-22T19:22:03.342528+0000 mgr.y (mgr.34107) 3991 : cluster [DBG] pgmap v5422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:04.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:03 smithi115 bash[45495]: audit 2022-04-22T19:22:03.534986+0000 mon.a (mon.0) 6801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:03 smithi115 bash[46934]: cluster 2022-04-22T19:22:03.342528+0000 mgr.y (mgr.34107) 3991 : cluster [DBG] pgmap v5422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:03 smithi115 bash[46934]: audit 2022-04-22T19:22:03.534986+0000 mon.a (mon.0) 6801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:05 smithi121 bash[36902]: audit 2022-04-22T19:22:05.121367+0000 mon.a (mon.0) 6802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:05 smithi115 bash[45495]: audit 2022-04-22T19:22:05.121367+0000 mon.a (mon.0) 6802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:05 smithi115 bash[46934]: audit 2022-04-22T19:22:05.121367+0000 mon.a (mon.0) 6802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:06 smithi121 bash[36902]: cluster 2022-04-22T19:22:05.343627+0000 mgr.y (mgr.34107) 3992 : cluster [DBG] pgmap v5423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:06 smithi115 bash[45495]: cluster 2022-04-22T19:22:05.343627+0000 mgr.y (mgr.34107) 3992 : cluster [DBG] pgmap v5423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:06 smithi115 bash[46934]: cluster 2022-04-22T19:22:05.343627+0000 mgr.y (mgr.34107) 3992 : cluster [DBG] pgmap v5423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:08 smithi121 bash[36902]: cluster 2022-04-22T19:22:07.344425+0000 mgr.y (mgr.34107) 3993 : cluster [DBG] pgmap v5424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:08.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:08 smithi115 bash[45495]: cluster 2022-04-22T19:22:07.344425+0000 mgr.y (mgr.34107) 3993 : cluster [DBG] pgmap v5424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:08.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:08 smithi115 bash[46934]: cluster 2022-04-22T19:22:07.344425+0000 mgr.y (mgr.34107) 3993 : cluster [DBG] pgmap v5424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:09 smithi121 bash[36902]: audit 2022-04-22T19:22:08.539114+0000 mon.a (mon.0) 6803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:09 smithi121 bash[36902]: cluster 2022-04-22T19:22:09.345425+0000 mgr.y (mgr.34107) 3994 : cluster [DBG] pgmap v5425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:09 smithi115 bash[45495]: audit 2022-04-22T19:22:08.539114+0000 mon.a (mon.0) 6803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:09 smithi115 bash[45495]: cluster 2022-04-22T19:22:09.345425+0000 mgr.y (mgr.34107) 3994 : cluster [DBG] pgmap v5425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:09.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:09 smithi115 bash[46934]: audit 2022-04-22T19:22:08.539114+0000 mon.a (mon.0) 6803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:09.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:09 smithi115 bash[46934]: cluster 2022-04-22T19:22:09.345425+0000 mgr.y (mgr.34107) 3994 : cluster [DBG] pgmap v5425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:10 smithi121 bash[36902]: audit 2022-04-22T19:22:10.124801+0000 mon.a (mon.0) 6804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:10.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:10 smithi115 bash[45495]: audit 2022-04-22T19:22:10.124801+0000 mon.a (mon.0) 6804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:10 smithi115 bash[46934]: audit 2022-04-22T19:22:10.124801+0000 mon.a (mon.0) 6804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:11 smithi121 bash[36902]: cluster 2022-04-22T19:22:11.346544+0000 mgr.y (mgr.34107) 3995 : cluster [DBG] pgmap v5426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:11 smithi115 bash[45495]: cluster 2022-04-22T19:22:11.346544+0000 mgr.y (mgr.34107) 3995 : cluster [DBG] pgmap v5426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:11 smithi115 bash[46934]: cluster 2022-04-22T19:22:11.346544+0000 mgr.y (mgr.34107) 3995 : cluster [DBG] pgmap v5426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:12.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:13 smithi121 bash[36902]: cluster 2022-04-22T19:22:13.347676+0000 mgr.y (mgr.34107) 3996 : cluster [DBG] pgmap v5427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:13 smithi121 bash[36902]: audit 2022-04-22T19:22:13.542751+0000 mon.a (mon.0) 6805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:13.927Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:13 smithi115 bash[45495]: cluster 2022-04-22T19:22:13.347676+0000 mgr.y (mgr.34107) 3996 : cluster [DBG] pgmap v5427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:13 smithi115 bash[45495]: audit 2022-04-22T19:22:13.542751+0000 mon.a (mon.0) 6805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:13 smithi115 bash[46934]: cluster 2022-04-22T19:22:13.347676+0000 mgr.y (mgr.34107) 3996 : cluster [DBG] pgmap v5427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:14.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:13 smithi115 bash[46934]: audit 2022-04-22T19:22:13.542751+0000 mon.a (mon.0) 6805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:14 smithi121 bash[36902]: audit 2022-04-22T19:22:14.594236+0000 mon.a (mon.0) 6806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:22:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:14 smithi121 bash[36902]: audit 2022-04-22T19:22:14.594923+0000 mon.a (mon.0) 6807 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:22:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:14 smithi115 bash[45495]: audit 2022-04-22T19:22:14.594236+0000 mon.a (mon.0) 6806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:22:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:14 smithi115 bash[45495]: audit 2022-04-22T19:22:14.594923+0000 mon.a (mon.0) 6807 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:22:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:14 smithi115 bash[46934]: audit 2022-04-22T19:22:14.594236+0000 mon.a (mon.0) 6806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:22:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:14 smithi115 bash[46934]: audit 2022-04-22T19:22:14.594923+0000 mon.a (mon.0) 6807 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:22:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:15 smithi121 bash[36902]: audit 2022-04-22T19:22:15.127970+0000 mon.a (mon.0) 6808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:15 smithi121 bash[36902]: cluster 2022-04-22T19:22:15.348728+0000 mgr.y (mgr.34107) 3997 : cluster [DBG] pgmap v5428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:15 smithi115 bash[45495]: audit 2022-04-22T19:22:15.127970+0000 mon.a (mon.0) 6808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:15 smithi115 bash[45495]: cluster 2022-04-22T19:22:15.348728+0000 mgr.y (mgr.34107) 3997 : cluster [DBG] pgmap v5428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:15 smithi115 bash[46934]: audit 2022-04-22T19:22:15.127970+0000 mon.a (mon.0) 6808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:15 smithi115 bash[46934]: cluster 2022-04-22T19:22:15.348728+0000 mgr.y (mgr.34107) 3997 : cluster [DBG] pgmap v5428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:17 smithi121 bash[36902]: cluster 2022-04-22T19:22:17.349795+0000 mgr.y (mgr.34107) 3998 : cluster [DBG] pgmap v5429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:17.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:17 smithi115 bash[45495]: cluster 2022-04-22T19:22:17.349795+0000 mgr.y (mgr.34107) 3998 : cluster [DBG] pgmap v5429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:17.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:17 smithi115 bash[46934]: cluster 2022-04-22T19:22:17.349795+0000 mgr.y (mgr.34107) 3998 : cluster [DBG] pgmap v5429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:18 smithi121 bash[36902]: audit 2022-04-22T19:22:18.546393+0000 mon.a (mon.0) 6809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:19.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:18 smithi115 bash[45495]: audit 2022-04-22T19:22:18.546393+0000 mon.a (mon.0) 6809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:18 smithi115 bash[46934]: audit 2022-04-22T19:22:18.546393+0000 mon.a (mon.0) 6809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:19 smithi121 bash[36902]: cluster 2022-04-22T19:22:19.350869+0000 mgr.y (mgr.34107) 3999 : cluster [DBG] pgmap v5430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:20.036 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:22:20.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:19 smithi115 bash[45495]: cluster 2022-04-22T19:22:19.350869+0000 mgr.y (mgr.34107) 3999 : cluster [DBG] pgmap v5430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:20.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:19 smithi115 bash[46934]: cluster 2022-04-22T19:22:19.350869+0000 mgr.y (mgr.34107) 3999 : cluster [DBG] pgmap v5430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:20.381 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:22:20.382 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (105m) 98s ago 119m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:22:20.382 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (118m) 2m ago 118m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:22:20.382 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (107m) 98s ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:22:20.382 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (106m) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (114m) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (115m) 98s ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (114m) 98s ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (113m) 2m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (113m) 98s ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:22:20.383 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (106m) 98s ago 119m 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (106m) 2m ago 119m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (112m) 98s ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (111m) 98s ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (110m) 98s ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (110m) 98s ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:22:20.384 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (109m) 2m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:22:20.385 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (109m) 2m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:22:20.385 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (108m) 2m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:22:20.385 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (107m) 2m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:22:20.385 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (106m) 2m ago 119m 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:22:20.790 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:22:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:20 smithi121 bash[36902]: audit 2022-04-22T19:22:20.008542+0000 mgr.y (mgr.34107) 4000 : audit [DBG] from='client.49516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:20 smithi121 bash[36902]: audit 2022-04-22T19:22:20.130859+0000 mon.a (mon.0) 6810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:20 smithi121 bash[36902]: audit 2022-04-22T19:22:20.350003+0000 mgr.y (mgr.34107) 4001 : audit [DBG] from='client.39588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[45495]: audit 2022-04-22T19:22:20.008542+0000 mgr.y (mgr.34107) 4000 : audit [DBG] from='client.49516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[45495]: audit 2022-04-22T19:22:20.130859+0000 mon.a (mon.0) 6810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:21.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[45495]: audit 2022-04-22T19:22:20.350003+0000 mgr.y (mgr.34107) 4001 : audit [DBG] from='client.39588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:21.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[46934]: audit 2022-04-22T19:22:20.008542+0000 mgr.y (mgr.34107) 4000 : audit [DBG] from='client.49516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:21.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[46934]: audit 2022-04-22T19:22:20.130859+0000 mon.a (mon.0) 6810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:21.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:20 smithi115 bash[46934]: audit 2022-04-22T19:22:20.350003+0000 mgr.y (mgr.34107) 4001 : audit [DBG] from='client.39588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:21 smithi121 bash[36902]: audit 2022-04-22T19:22:20.776655+0000 mon.a (mon.0) 6811 : audit [DBG] from='client.? 172.21.15.115:0/1501919655' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:21 smithi121 bash[36902]: cluster 2022-04-22T19:22:21.352091+0000 mgr.y (mgr.34107) 4002 : cluster [DBG] pgmap v5431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:21 smithi115 bash[45495]: audit 2022-04-22T19:22:20.776655+0000 mon.a (mon.0) 6811 : audit [DBG] from='client.? 172.21.15.115:0/1501919655' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:21 smithi115 bash[45495]: cluster 2022-04-22T19:22:21.352091+0000 mgr.y (mgr.34107) 4002 : cluster [DBG] pgmap v5431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:22.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:21 smithi115 bash[46934]: audit 2022-04-22T19:22:20.776655+0000 mon.a (mon.0) 6811 : audit [DBG] from='client.? 172.21.15.115:0/1501919655' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:22.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:21 smithi115 bash[46934]: cluster 2022-04-22T19:22:21.352091+0000 mgr.y (mgr.34107) 4002 : cluster [DBG] pgmap v5431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:22.058Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:23 smithi121 bash[36902]: cluster 2022-04-22T19:22:23.353274+0000 mgr.y (mgr.34107) 4003 : cluster [DBG] pgmap v5432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:23 smithi121 bash[36902]: audit 2022-04-22T19:22:23.550131+0000 mon.a (mon.0) 6812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:23.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:24.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:23 smithi115 bash[45495]: cluster 2022-04-22T19:22:23.353274+0000 mgr.y (mgr.34107) 4003 : cluster [DBG] pgmap v5432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:24.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:23 smithi115 bash[45495]: audit 2022-04-22T19:22:23.550131+0000 mon.a (mon.0) 6812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:23 smithi115 bash[46934]: cluster 2022-04-22T19:22:23.353274+0000 mgr.y (mgr.34107) 4003 : cluster [DBG] pgmap v5432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:24.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:23 smithi115 bash[46934]: audit 2022-04-22T19:22:23.550131+0000 mon.a (mon.0) 6812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:25 smithi121 bash[36902]: audit 2022-04-22T19:22:25.133517+0000 mon.a (mon.0) 6813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:25 smithi115 bash[45495]: audit 2022-04-22T19:22:25.133517+0000 mon.a (mon.0) 6813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:25 smithi115 bash[46934]: audit 2022-04-22T19:22:25.133517+0000 mon.a (mon.0) 6813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:26 smithi121 bash[36902]: cluster 2022-04-22T19:22:25.354355+0000 mgr.y (mgr.34107) 4004 : cluster [DBG] pgmap v5433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:26 smithi115 bash[45495]: cluster 2022-04-22T19:22:25.354355+0000 mgr.y (mgr.34107) 4004 : cluster [DBG] pgmap v5433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:26 smithi115 bash[46934]: cluster 2022-04-22T19:22:25.354355+0000 mgr.y (mgr.34107) 4004 : cluster [DBG] pgmap v5433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:27 smithi121 bash[36902]: cluster 2022-04-22T19:22:27.355539+0000 mgr.y (mgr.34107) 4005 : cluster [DBG] pgmap v5434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:27 smithi115 bash[45495]: cluster 2022-04-22T19:22:27.355539+0000 mgr.y (mgr.34107) 4005 : cluster [DBG] pgmap v5434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:27 smithi115 bash[46934]: cluster 2022-04-22T19:22:27.355539+0000 mgr.y (mgr.34107) 4005 : cluster [DBG] pgmap v5434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:28 smithi121 bash[36902]: audit 2022-04-22T19:22:28.553703+0000 mon.a (mon.0) 6814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:29.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:28 smithi115 bash[45495]: audit 2022-04-22T19:22:28.553703+0000 mon.a (mon.0) 6814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:28 smithi115 bash[46934]: audit 2022-04-22T19:22:28.553703+0000 mon.a (mon.0) 6814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:29 smithi121 bash[36902]: cluster 2022-04-22T19:22:29.356602+0000 mgr.y (mgr.34107) 4006 : cluster [DBG] pgmap v5435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:30.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:29 smithi115 bash[45495]: cluster 2022-04-22T19:22:29.356602+0000 mgr.y (mgr.34107) 4006 : cluster [DBG] pgmap v5435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:30.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:29 smithi115 bash[46934]: cluster 2022-04-22T19:22:29.356602+0000 mgr.y (mgr.34107) 4006 : cluster [DBG] pgmap v5435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:30 smithi121 bash[36902]: audit 2022-04-22T19:22:30.136428+0000 mon.a (mon.0) 6815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:31.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:30 smithi115 bash[45495]: audit 2022-04-22T19:22:30.136428+0000 mon.a (mon.0) 6815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:31.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:30 smithi115 bash[46934]: audit 2022-04-22T19:22:30.136428+0000 mon.a (mon.0) 6815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:31 smithi121 bash[36902]: cluster 2022-04-22T19:22:31.357747+0000 mgr.y (mgr.34107) 4007 : cluster [DBG] pgmap v5436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:31 smithi115 bash[45495]: cluster 2022-04-22T19:22:31.357747+0000 mgr.y (mgr.34107) 4007 : cluster [DBG] pgmap v5436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:32.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:31 smithi115 bash[46934]: cluster 2022-04-22T19:22:31.357747+0000 mgr.y (mgr.34107) 4007 : cluster [DBG] pgmap v5436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:32.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:32.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:33 smithi121 bash[36902]: cluster 2022-04-22T19:22:33.358896+0000 mgr.y (mgr.34107) 4008 : cluster [DBG] pgmap v5437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:33 smithi121 bash[36902]: audit 2022-04-22T19:22:33.557602+0000 mon.a (mon.0) 6816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:33 smithi115 bash[45495]: cluster 2022-04-22T19:22:33.358896+0000 mgr.y (mgr.34107) 4008 : cluster [DBG] pgmap v5437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:33 smithi115 bash[45495]: audit 2022-04-22T19:22:33.557602+0000 mon.a (mon.0) 6816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:34.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:33.930Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:33 smithi115 bash[46934]: cluster 2022-04-22T19:22:33.358896+0000 mgr.y (mgr.34107) 4008 : cluster [DBG] pgmap v5437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:33 smithi115 bash[46934]: audit 2022-04-22T19:22:33.557602+0000 mon.a (mon.0) 6816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:35 smithi121 bash[36902]: audit 2022-04-22T19:22:35.139636+0000 mon.a (mon.0) 6817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:35 smithi115 bash[45495]: audit 2022-04-22T19:22:35.139636+0000 mon.a (mon.0) 6817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:35 smithi115 bash[46934]: audit 2022-04-22T19:22:35.139636+0000 mon.a (mon.0) 6817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:36 smithi121 bash[36902]: cluster 2022-04-22T19:22:35.359862+0000 mgr.y (mgr.34107) 4009 : cluster [DBG] pgmap v5438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:36.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:36 smithi115 bash[45495]: cluster 2022-04-22T19:22:35.359862+0000 mgr.y (mgr.34107) 4009 : cluster [DBG] pgmap v5438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:36.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:36 smithi115 bash[46934]: cluster 2022-04-22T19:22:35.359862+0000 mgr.y (mgr.34107) 4009 : cluster [DBG] pgmap v5438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:37 smithi121 bash[36902]: cluster 2022-04-22T19:22:37.360935+0000 mgr.y (mgr.34107) 4010 : cluster [DBG] pgmap v5439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:37.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:37 smithi115 bash[45495]: cluster 2022-04-22T19:22:37.360935+0000 mgr.y (mgr.34107) 4010 : cluster [DBG] pgmap v5439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:37.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:37 smithi115 bash[46934]: cluster 2022-04-22T19:22:37.360935+0000 mgr.y (mgr.34107) 4010 : cluster [DBG] pgmap v5439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:38 smithi121 bash[36902]: audit 2022-04-22T19:22:38.561146+0000 mon.a (mon.0) 6818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:39.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:38 smithi115 bash[45495]: audit 2022-04-22T19:22:38.561146+0000 mon.a (mon.0) 6818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:39.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:38 smithi115 bash[46934]: audit 2022-04-22T19:22:38.561146+0000 mon.a (mon.0) 6818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:39 smithi121 bash[36902]: cluster 2022-04-22T19:22:39.362039+0000 mgr.y (mgr.34107) 4011 : cluster [DBG] pgmap v5440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:39 smithi115 bash[45495]: cluster 2022-04-22T19:22:39.362039+0000 mgr.y (mgr.34107) 4011 : cluster [DBG] pgmap v5440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:39 smithi115 bash[46934]: cluster 2022-04-22T19:22:39.362039+0000 mgr.y (mgr.34107) 4011 : cluster [DBG] pgmap v5440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:40 smithi121 bash[36902]: audit 2022-04-22T19:22:40.142656+0000 mon.a (mon.0) 6819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:41.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:40 smithi115 bash[45495]: audit 2022-04-22T19:22:40.142656+0000 mon.a (mon.0) 6819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:41.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:40 smithi115 bash[46934]: audit 2022-04-22T19:22:40.142656+0000 mon.a (mon.0) 6819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:41 smithi121 bash[36902]: cluster 2022-04-22T19:22:41.363116+0000 mgr.y (mgr.34107) 4012 : cluster [DBG] pgmap v5441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:42.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:41 smithi115 bash[45495]: cluster 2022-04-22T19:22:41.363116+0000 mgr.y (mgr.34107) 4012 : cluster [DBG] pgmap v5441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:42.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:41 smithi115 bash[46934]: cluster 2022-04-22T19:22:41.363116+0000 mgr.y (mgr.34107) 4012 : cluster [DBG] pgmap v5441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:42.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: audit 2022-04-22T19:22:42.943838+0000 mon.a (mon.0) 6820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: audit 2022-04-22T19:22:42.946731+0000 mon.a (mon.0) 6821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: audit 2022-04-22T19:22:42.948227+0000 mon.a (mon.0) 6822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: audit 2022-04-22T19:22:43.313064+0000 mon.a (mon.0) 6823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: cluster 2022-04-22T19:22:43.364294+0000 mgr.y (mgr.34107) 4013 : cluster [DBG] pgmap v5442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:43 smithi121 bash[36902]: audit 2022-04-22T19:22:43.564658+0000 mon.a (mon.0) 6824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:44.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:43.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: audit 2022-04-22T19:22:42.943838+0000 mon.a (mon.0) 6820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: audit 2022-04-22T19:22:42.946731+0000 mon.a (mon.0) 6821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: audit 2022-04-22T19:22:42.948227+0000 mon.a (mon.0) 6822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: audit 2022-04-22T19:22:43.313064+0000 mon.a (mon.0) 6823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: cluster 2022-04-22T19:22:43.364294+0000 mgr.y (mgr.34107) 4013 : cluster [DBG] pgmap v5442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[45495]: audit 2022-04-22T19:22:43.564658+0000 mon.a (mon.0) 6824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: audit 2022-04-22T19:22:42.943838+0000 mon.a (mon.0) 6820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: audit 2022-04-22T19:22:42.946731+0000 mon.a (mon.0) 6821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: audit 2022-04-22T19:22:42.948227+0000 mon.a (mon.0) 6822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: audit 2022-04-22T19:22:43.313064+0000 mon.a (mon.0) 6823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: cluster 2022-04-22T19:22:43.364294+0000 mgr.y (mgr.34107) 4013 : cluster [DBG] pgmap v5442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:44.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:43 smithi115 bash[46934]: audit 2022-04-22T19:22:43.564658+0000 mon.a (mon.0) 6824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:45 smithi121 bash[36902]: audit 2022-04-22T19:22:45.146174+0000 mon.a (mon.0) 6825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:45 smithi115 bash[45495]: audit 2022-04-22T19:22:45.146174+0000 mon.a (mon.0) 6825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:45.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:45 smithi115 bash[46934]: audit 2022-04-22T19:22:45.146174+0000 mon.a (mon.0) 6825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:46 smithi121 bash[36902]: cluster 2022-04-22T19:22:45.365378+0000 mgr.y (mgr.34107) 4014 : cluster [DBG] pgmap v5443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:46.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:46 smithi115 bash[45495]: cluster 2022-04-22T19:22:45.365378+0000 mgr.y (mgr.34107) 4014 : cluster [DBG] pgmap v5443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:46.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:46 smithi115 bash[46934]: cluster 2022-04-22T19:22:45.365378+0000 mgr.y (mgr.34107) 4014 : cluster [DBG] pgmap v5443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:47 smithi121 bash[36902]: cluster 2022-04-22T19:22:47.366465+0000 mgr.y (mgr.34107) 4015 : cluster [DBG] pgmap v5444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:47.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:47 smithi115 bash[45495]: cluster 2022-04-22T19:22:47.366465+0000 mgr.y (mgr.34107) 4015 : cluster [DBG] pgmap v5444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:47.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:47 smithi115 bash[46934]: cluster 2022-04-22T19:22:47.366465+0000 mgr.y (mgr.34107) 4015 : cluster [DBG] pgmap v5444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:48 smithi121 bash[36902]: audit 2022-04-22T19:22:48.568421+0000 mon.a (mon.0) 6826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:49.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:48 smithi115 bash[45495]: audit 2022-04-22T19:22:48.568421+0000 mon.a (mon.0) 6826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:49.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:48 smithi115 bash[46934]: audit 2022-04-22T19:22:48.568421+0000 mon.a (mon.0) 6826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:49 smithi121 bash[36902]: cluster 2022-04-22T19:22:49.367526+0000 mgr.y (mgr.34107) 4016 : cluster [DBG] pgmap v5445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:50.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:49 smithi115 bash[45495]: cluster 2022-04-22T19:22:49.367526+0000 mgr.y (mgr.34107) 4016 : cluster [DBG] pgmap v5445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:50.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:49 smithi115 bash[46934]: cluster 2022-04-22T19:22:49.367526+0000 mgr.y (mgr.34107) 4016 : cluster [DBG] pgmap v5445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:50 smithi121 bash[36902]: audit 2022-04-22T19:22:50.149295+0000 mon.a (mon.0) 6827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:51.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:50 smithi115 bash[46934]: audit 2022-04-22T19:22:50.149295+0000 mon.a (mon.0) 6827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:51.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:50 smithi115 bash[45495]: audit 2022-04-22T19:22:50.149295+0000 mon.a (mon.0) 6827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:51.155 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:22:51.506 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:22:51.506 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (105m) 2m ago 119m 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:22:51.506 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (119m) 3m ago 119m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:22:51.506 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (107m) 2m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:22:51.506 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (107m) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (115m) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (115m) 2m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (114m) 2m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (113m) 3m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (114m) 2m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (107m) 2m ago 119m 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (106m) 3m ago 119m 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (112m) 2m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (111m) 2m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:22:51.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (111m) 2m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (111m) 2m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (110m) 3m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (109m) 3m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (108m) 3m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (108m) 3m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:22:51.508 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (106m) 3m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:22:51.912 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:22:51.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:22:51.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:22:51.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:22:51.913 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:22:51.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:22:51.914 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:22:51.914 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:22:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:51 smithi121 bash[36902]: audit 2022-04-22T19:22:51.128441+0000 mgr.y (mgr.34107) 4017 : audit [DBG] from='client.39600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:51 smithi121 bash[36902]: cluster 2022-04-22T19:22:51.368552+0000 mgr.y (mgr.34107) 4018 : cluster [DBG] pgmap v5446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:51 smithi115 bash[45495]: audit 2022-04-22T19:22:51.128441+0000 mgr.y (mgr.34107) 4017 : audit [DBG] from='client.39600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:52.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:51 smithi115 bash[45495]: cluster 2022-04-22T19:22:51.368552+0000 mgr.y (mgr.34107) 4018 : cluster [DBG] pgmap v5446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:51 smithi115 bash[46934]: audit 2022-04-22T19:22:51.128441+0000 mgr.y (mgr.34107) 4017 : audit [DBG] from='client.39600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:51 smithi115 bash[46934]: cluster 2022-04-22T19:22:51.368552+0000 mgr.y (mgr.34107) 4018 : cluster [DBG] pgmap v5446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:52.058Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:52 smithi121 bash[36902]: audit 2022-04-22T19:22:51.474390+0000 mgr.y (mgr.34107) 4019 : audit [DBG] from='client.39606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:52 smithi121 bash[36902]: audit 2022-04-22T19:22:51.899694+0000 mon.c (mon.1) 188 : audit [DBG] from='client.? 172.21.15.115:0/3893065855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:53.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:52 smithi115 bash[45495]: audit 2022-04-22T19:22:51.474390+0000 mgr.y (mgr.34107) 4019 : audit [DBG] from='client.39606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:53.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:52 smithi115 bash[45495]: audit 2022-04-22T19:22:51.899694+0000 mon.c (mon.1) 188 : audit [DBG] from='client.? 172.21.15.115:0/3893065855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:53.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:52 smithi115 bash[46934]: audit 2022-04-22T19:22:51.474390+0000 mgr.y (mgr.34107) 4019 : audit [DBG] from='client.39606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:22:53.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:52 smithi115 bash[46934]: audit 2022-04-22T19:22:51.899694+0000 mon.c (mon.1) 188 : audit [DBG] from='client.? 172.21.15.115:0/3893065855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:22:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:53 smithi121 bash[36902]: cluster 2022-04-22T19:22:53.369697+0000 mgr.y (mgr.34107) 4020 : cluster [DBG] pgmap v5447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:53 smithi121 bash[36902]: audit 2022-04-22T19:22:53.572035+0000 mon.a (mon.0) 6828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:54.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:22:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:22:53.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:22:54.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:53 smithi115 bash[45495]: cluster 2022-04-22T19:22:53.369697+0000 mgr.y (mgr.34107) 4020 : cluster [DBG] pgmap v5447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:54.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:53 smithi115 bash[45495]: audit 2022-04-22T19:22:53.572035+0000 mon.a (mon.0) 6828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:54.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:53 smithi115 bash[46934]: cluster 2022-04-22T19:22:53.369697+0000 mgr.y (mgr.34107) 4020 : cluster [DBG] pgmap v5447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:54.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:53 smithi115 bash[46934]: audit 2022-04-22T19:22:53.572035+0000 mon.a (mon.0) 6828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:55 smithi121 bash[36902]: audit 2022-04-22T19:22:55.152110+0000 mon.a (mon.0) 6829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:55 smithi115 bash[45495]: audit 2022-04-22T19:22:55.152110+0000 mon.a (mon.0) 6829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:55 smithi115 bash[46934]: audit 2022-04-22T19:22:55.152110+0000 mon.a (mon.0) 6829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:22:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:56 smithi121 bash[36902]: cluster 2022-04-22T19:22:55.370724+0000 mgr.y (mgr.34107) 4021 : cluster [DBG] pgmap v5448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:56 smithi115 bash[46934]: cluster 2022-04-22T19:22:55.370724+0000 mgr.y (mgr.34107) 4021 : cluster [DBG] pgmap v5448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:56.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:56 smithi115 bash[45495]: cluster 2022-04-22T19:22:55.370724+0000 mgr.y (mgr.34107) 4021 : cluster [DBG] pgmap v5448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:57 smithi121 bash[36902]: cluster 2022-04-22T19:22:57.371803+0000 mgr.y (mgr.34107) 4022 : cluster [DBG] pgmap v5449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:57.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:57 smithi115 bash[45495]: cluster 2022-04-22T19:22:57.371803+0000 mgr.y (mgr.34107) 4022 : cluster [DBG] pgmap v5449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:57.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:57 smithi115 bash[46934]: cluster 2022-04-22T19:22:57.371803+0000 mgr.y (mgr.34107) 4022 : cluster [DBG] pgmap v5449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:22:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:58 smithi121 bash[36902]: audit 2022-04-22T19:22:58.575634+0000 mon.a (mon.0) 6830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:59.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:58 smithi115 bash[45495]: audit 2022-04-22T19:22:58.575634+0000 mon.a (mon.0) 6830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:22:59.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:58 smithi115 bash[46934]: audit 2022-04-22T19:22:58.575634+0000 mon.a (mon.0) 6830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:00.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:22:59 smithi121 bash[36902]: cluster 2022-04-22T19:22:59.372868+0000 mgr.y (mgr.34107) 4023 : cluster [DBG] pgmap v5450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:00.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:22:59 smithi115 bash[45495]: cluster 2022-04-22T19:22:59.372868+0000 mgr.y (mgr.34107) 4023 : cluster [DBG] pgmap v5450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:00.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:22:59 smithi115 bash[46934]: cluster 2022-04-22T19:22:59.372868+0000 mgr.y (mgr.34107) 4023 : cluster [DBG] pgmap v5450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:00 smithi121 bash[36902]: audit 2022-04-22T19:23:00.155416+0000 mon.a (mon.0) 6831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:01.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:00 smithi115 bash[45495]: audit 2022-04-22T19:23:00.155416+0000 mon.a (mon.0) 6831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:01.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:00 smithi115 bash[46934]: audit 2022-04-22T19:23:00.155416+0000 mon.a (mon.0) 6831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:01 smithi121 bash[36902]: cluster 2022-04-22T19:23:01.373960+0000 mgr.y (mgr.34107) 4024 : cluster [DBG] pgmap v5451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:02.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:01 smithi115 bash[45495]: cluster 2022-04-22T19:23:01.373960+0000 mgr.y (mgr.34107) 4024 : cluster [DBG] pgmap v5451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:02.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:01 smithi115 bash[46934]: cluster 2022-04-22T19:23:01.373960+0000 mgr.y (mgr.34107) 4024 : cluster [DBG] pgmap v5451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:02.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:02.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:03 smithi121 bash[36902]: cluster 2022-04-22T19:23:03.375167+0000 mgr.y (mgr.34107) 4025 : cluster [DBG] pgmap v5452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:03 smithi121 bash[36902]: audit 2022-04-22T19:23:03.579540+0000 mon.a (mon.0) 6832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:04.054 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:03.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:04.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:03 smithi115 bash[45495]: cluster 2022-04-22T19:23:03.375167+0000 mgr.y (mgr.34107) 4025 : cluster [DBG] pgmap v5452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:04.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:03 smithi115 bash[45495]: audit 2022-04-22T19:23:03.579540+0000 mon.a (mon.0) 6832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:04.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:03 smithi115 bash[46934]: cluster 2022-04-22T19:23:03.375167+0000 mgr.y (mgr.34107) 4025 : cluster [DBG] pgmap v5452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:04.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:03 smithi115 bash[46934]: audit 2022-04-22T19:23:03.579540+0000 mon.a (mon.0) 6832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:05 smithi121 bash[36902]: audit 2022-04-22T19:23:05.158736+0000 mon.a (mon.0) 6833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:05 smithi115 bash[45495]: audit 2022-04-22T19:23:05.158736+0000 mon.a (mon.0) 6833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:05 smithi115 bash[46934]: audit 2022-04-22T19:23:05.158736+0000 mon.a (mon.0) 6833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:06 smithi121 bash[36902]: cluster 2022-04-22T19:23:05.376282+0000 mgr.y (mgr.34107) 4026 : cluster [DBG] pgmap v5453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:06 smithi115 bash[45495]: cluster 2022-04-22T19:23:05.376282+0000 mgr.y (mgr.34107) 4026 : cluster [DBG] pgmap v5453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:06.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:06 smithi115 bash[46934]: cluster 2022-04-22T19:23:05.376282+0000 mgr.y (mgr.34107) 4026 : cluster [DBG] pgmap v5453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:07 smithi121 bash[36902]: cluster 2022-04-22T19:23:07.377310+0000 mgr.y (mgr.34107) 4027 : cluster [DBG] pgmap v5454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:07 smithi115 bash[45495]: cluster 2022-04-22T19:23:07.377310+0000 mgr.y (mgr.34107) 4027 : cluster [DBG] pgmap v5454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:07.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:07 smithi115 bash[46934]: cluster 2022-04-22T19:23:07.377310+0000 mgr.y (mgr.34107) 4027 : cluster [DBG] pgmap v5454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:08 smithi121 bash[36902]: audit 2022-04-22T19:23:08.583440+0000 mon.a (mon.0) 6834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:09.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:08 smithi115 bash[45495]: audit 2022-04-22T19:23:08.583440+0000 mon.a (mon.0) 6834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:08 smithi115 bash[46934]: audit 2022-04-22T19:23:08.583440+0000 mon.a (mon.0) 6834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:09 smithi121 bash[36902]: cluster 2022-04-22T19:23:09.378368+0000 mgr.y (mgr.34107) 4028 : cluster [DBG] pgmap v5455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:10.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:09 smithi115 bash[45495]: cluster 2022-04-22T19:23:09.378368+0000 mgr.y (mgr.34107) 4028 : cluster [DBG] pgmap v5455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:10.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:09 smithi115 bash[46934]: cluster 2022-04-22T19:23:09.378368+0000 mgr.y (mgr.34107) 4028 : cluster [DBG] pgmap v5455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:10 smithi121 bash[36902]: audit 2022-04-22T19:23:10.161862+0000 mon.a (mon.0) 6835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:11.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:10 smithi115 bash[45495]: audit 2022-04-22T19:23:10.161862+0000 mon.a (mon.0) 6835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:11.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:10 smithi115 bash[46934]: audit 2022-04-22T19:23:10.161862+0000 mon.a (mon.0) 6835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:11 smithi121 bash[36902]: cluster 2022-04-22T19:23:11.379523+0000 mgr.y (mgr.34107) 4029 : cluster [DBG] pgmap v5456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:12.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:11 smithi115 bash[45495]: cluster 2022-04-22T19:23:11.379523+0000 mgr.y (mgr.34107) 4029 : cluster [DBG] pgmap v5456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:12.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:11 smithi115 bash[46934]: cluster 2022-04-22T19:23:11.379523+0000 mgr.y (mgr.34107) 4029 : cluster [DBG] pgmap v5456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:12.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:12.060Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:13 smithi121 bash[36902]: cluster 2022-04-22T19:23:13.380697+0000 mgr.y (mgr.34107) 4030 : cluster [DBG] pgmap v5457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:13 smithi121 bash[36902]: audit 2022-04-22T19:23:13.587106+0000 mon.a (mon.0) 6836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:14.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:13.933Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:13 smithi115 bash[45495]: cluster 2022-04-22T19:23:13.380697+0000 mgr.y (mgr.34107) 4030 : cluster [DBG] pgmap v5457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:13 smithi115 bash[45495]: audit 2022-04-22T19:23:13.587106+0000 mon.a (mon.0) 6836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:13 smithi115 bash[46934]: cluster 2022-04-22T19:23:13.380697+0000 mgr.y (mgr.34107) 4030 : cluster [DBG] pgmap v5457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:13 smithi115 bash[46934]: audit 2022-04-22T19:23:13.587106+0000 mon.a (mon.0) 6836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:14 smithi121 bash[36902]: audit 2022-04-22T19:23:14.604509+0000 mon.a (mon.0) 6837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:23:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:14 smithi121 bash[36902]: audit 2022-04-22T19:23:14.605023+0000 mon.a (mon.0) 6838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:23:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:14 smithi115 bash[45495]: audit 2022-04-22T19:23:14.604509+0000 mon.a (mon.0) 6837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:23:15.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:14 smithi115 bash[45495]: audit 2022-04-22T19:23:14.605023+0000 mon.a (mon.0) 6838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:23:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:14 smithi115 bash[46934]: audit 2022-04-22T19:23:14.604509+0000 mon.a (mon.0) 6837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:23:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:14 smithi115 bash[46934]: audit 2022-04-22T19:23:14.605023+0000 mon.a (mon.0) 6838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:23:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:15 smithi121 bash[36902]: audit 2022-04-22T19:23:15.165141+0000 mon.a (mon.0) 6839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:15 smithi121 bash[36902]: cluster 2022-04-22T19:23:15.381746+0000 mgr.y (mgr.34107) 4031 : cluster [DBG] pgmap v5458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:16.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:15 smithi115 bash[45495]: audit 2022-04-22T19:23:15.165141+0000 mon.a (mon.0) 6839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:15 smithi115 bash[45495]: cluster 2022-04-22T19:23:15.381746+0000 mgr.y (mgr.34107) 4031 : cluster [DBG] pgmap v5458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:15 smithi115 bash[46934]: audit 2022-04-22T19:23:15.165141+0000 mon.a (mon.0) 6839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:15 smithi115 bash[46934]: cluster 2022-04-22T19:23:15.381746+0000 mgr.y (mgr.34107) 4031 : cluster [DBG] pgmap v5458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:17 smithi121 bash[36902]: cluster 2022-04-22T19:23:17.382773+0000 mgr.y (mgr.34107) 4032 : cluster [DBG] pgmap v5459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:17 smithi115 bash[46934]: cluster 2022-04-22T19:23:17.382773+0000 mgr.y (mgr.34107) 4032 : cluster [DBG] pgmap v5459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:17.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:17 smithi115 bash[45495]: cluster 2022-04-22T19:23:17.382773+0000 mgr.y (mgr.34107) 4032 : cluster [DBG] pgmap v5459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:18 smithi121 bash[36902]: audit 2022-04-22T19:23:18.590875+0000 mon.a (mon.0) 6840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:19.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:18 smithi115 bash[45495]: audit 2022-04-22T19:23:18.590875+0000 mon.a (mon.0) 6840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:18 smithi115 bash[46934]: audit 2022-04-22T19:23:18.590875+0000 mon.a (mon.0) 6840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:19 smithi121 bash[36902]: cluster 2022-04-22T19:23:19.383855+0000 mgr.y (mgr.34107) 4033 : cluster [DBG] pgmap v5460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:20.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:19 smithi115 bash[45495]: cluster 2022-04-22T19:23:19.383855+0000 mgr.y (mgr.34107) 4033 : cluster [DBG] pgmap v5460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:20.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:19 smithi115 bash[46934]: cluster 2022-04-22T19:23:19.383855+0000 mgr.y (mgr.34107) 4033 : cluster [DBG] pgmap v5460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:20 smithi121 bash[36902]: audit 2022-04-22T19:23:20.168267+0000 mon.a (mon.0) 6841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:20 smithi115 bash[45495]: audit 2022-04-22T19:23:20.168267+0000 mon.a (mon.0) 6841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:21.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:20 smithi115 bash[46934]: audit 2022-04-22T19:23:20.168267+0000 mon.a (mon.0) 6841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:21 smithi115 bash[45495]: cluster 2022-04-22T19:23:21.385026+0000 mgr.y (mgr.34107) 4034 : cluster [DBG] pgmap v5461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:22.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:21 smithi115 bash[46934]: cluster 2022-04-22T19:23:21.385026+0000 mgr.y (mgr.34107) 4034 : cluster [DBG] pgmap v5461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:21 smithi121 bash[36902]: cluster 2022-04-22T19:23:21.385026+0000 mgr.y (mgr.34107) 4034 : cluster [DBG] pgmap v5461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:22.288 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:23:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:22.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:22.628 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:23:22.628 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (106m) 2m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:23:22.628 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (119m) 3m ago 119m 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (108m) 2m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (107m) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (115m) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (116m) 2m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (115m) 2m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:23:22.629 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (114m) 3m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:23:22.630 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (114m) 2m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:23:22.630 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (107m) 2m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:23:22.630 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (107m) 3m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:23:22.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (113m) 2m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:23:22.630 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (112m) 2m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:23:22.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (112m) 2m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:23:22.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (111m) 2m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:23:22.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (110m) 3m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:23:22.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (110m) 3m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:23:22.631 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (109m) 3m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:23:22.632 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (108m) 3m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:23:22.632 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (107m) 3m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:23:23.030 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:23:23.030 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:23:23.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:23:23.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:23.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:23:23.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:23:23.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:23.032 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:23:23.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:23:23.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:23.032 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:23:23.032 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:23:23.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:23.033 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:23:23.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:23:23.033 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:23:23.033 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:23:23.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:22 smithi115 bash[45495]: audit 2022-04-22T19:23:22.259757+0000 mgr.y (mgr.34107) 4035 : audit [DBG] from='client.39618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:23.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:22 smithi115 bash[46934]: audit 2022-04-22T19:23:22.259757+0000 mgr.y (mgr.34107) 4035 : audit [DBG] from='client.39618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:22 smithi121 bash[36902]: audit 2022-04-22T19:23:22.259757+0000 mgr.y (mgr.34107) 4035 : audit [DBG] from='client.39618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:23.933Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:24.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[45495]: audit 2022-04-22T19:23:22.595168+0000 mgr.y (mgr.34107) 4036 : audit [DBG] from='client.39624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:24.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[45495]: audit 2022-04-22T19:23:23.017776+0000 mon.a (mon.0) 6842 : audit [DBG] from='client.? 172.21.15.115:0/3051544348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:24.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[45495]: cluster 2022-04-22T19:23:23.386218+0000 mgr.y (mgr.34107) 4037 : cluster [DBG] pgmap v5462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:24.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[45495]: audit 2022-04-22T19:23:23.594330+0000 mon.a (mon.0) 6843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:24.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[46934]: audit 2022-04-22T19:23:22.595168+0000 mgr.y (mgr.34107) 4036 : audit [DBG] from='client.39624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:24.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[46934]: audit 2022-04-22T19:23:23.017776+0000 mon.a (mon.0) 6842 : audit [DBG] from='client.? 172.21.15.115:0/3051544348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:24.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[46934]: cluster 2022-04-22T19:23:23.386218+0000 mgr.y (mgr.34107) 4037 : cluster [DBG] pgmap v5462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:24.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:23 smithi115 bash[46934]: audit 2022-04-22T19:23:23.594330+0000 mon.a (mon.0) 6843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:23 smithi121 bash[36902]: audit 2022-04-22T19:23:22.595168+0000 mgr.y (mgr.34107) 4036 : audit [DBG] from='client.39624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:23 smithi121 bash[36902]: audit 2022-04-22T19:23:23.017776+0000 mon.a (mon.0) 6842 : audit [DBG] from='client.? 172.21.15.115:0/3051544348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:23 smithi121 bash[36902]: cluster 2022-04-22T19:23:23.386218+0000 mgr.y (mgr.34107) 4037 : cluster [DBG] pgmap v5462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:23 smithi121 bash[36902]: audit 2022-04-22T19:23:23.594330+0000 mon.a (mon.0) 6843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:25 smithi121 bash[36902]: audit 2022-04-22T19:23:25.171305+0000 mon.a (mon.0) 6844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:25 smithi115 bash[45495]: audit 2022-04-22T19:23:25.171305+0000 mon.a (mon.0) 6844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:25 smithi115 bash[46934]: audit 2022-04-22T19:23:25.171305+0000 mon.a (mon.0) 6844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:26 smithi121 bash[36902]: cluster 2022-04-22T19:23:25.387084+0000 mgr.y (mgr.34107) 4038 : cluster [DBG] pgmap v5463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:26 smithi115 bash[45495]: cluster 2022-04-22T19:23:25.387084+0000 mgr.y (mgr.34107) 4038 : cluster [DBG] pgmap v5463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:26 smithi115 bash[46934]: cluster 2022-04-22T19:23:25.387084+0000 mgr.y (mgr.34107) 4038 : cluster [DBG] pgmap v5463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:27 smithi121 bash[36902]: cluster 2022-04-22T19:23:27.388158+0000 mgr.y (mgr.34107) 4039 : cluster [DBG] pgmap v5464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:27 smithi115 bash[45495]: cluster 2022-04-22T19:23:27.388158+0000 mgr.y (mgr.34107) 4039 : cluster [DBG] pgmap v5464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:27.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:27 smithi115 bash[46934]: cluster 2022-04-22T19:23:27.388158+0000 mgr.y (mgr.34107) 4039 : cluster [DBG] pgmap v5464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:28 smithi121 bash[36902]: audit 2022-04-22T19:23:28.597966+0000 mon.a (mon.0) 6845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:29.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:28 smithi115 bash[45495]: audit 2022-04-22T19:23:28.597966+0000 mon.a (mon.0) 6845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:28 smithi115 bash[46934]: audit 2022-04-22T19:23:28.597966+0000 mon.a (mon.0) 6845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:29 smithi121 bash[36902]: cluster 2022-04-22T19:23:29.389220+0000 mgr.y (mgr.34107) 4040 : cluster [DBG] pgmap v5465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:30.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:29 smithi115 bash[45495]: cluster 2022-04-22T19:23:29.389220+0000 mgr.y (mgr.34107) 4040 : cluster [DBG] pgmap v5465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:30.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:29 smithi115 bash[46934]: cluster 2022-04-22T19:23:29.389220+0000 mgr.y (mgr.34107) 4040 : cluster [DBG] pgmap v5465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:30 smithi121 bash[36902]: audit 2022-04-22T19:23:30.174392+0000 mon.a (mon.0) 6846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:31.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:30 smithi115 bash[45495]: audit 2022-04-22T19:23:30.174392+0000 mon.a (mon.0) 6846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:31.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:30 smithi115 bash[46934]: audit 2022-04-22T19:23:30.174392+0000 mon.a (mon.0) 6846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:31 smithi115 bash[45495]: cluster 2022-04-22T19:23:31.390301+0000 mgr.y (mgr.34107) 4041 : cluster [DBG] pgmap v5466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:32.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:31 smithi115 bash[46934]: cluster 2022-04-22T19:23:31.390301+0000 mgr.y (mgr.34107) 4041 : cluster [DBG] pgmap v5466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:31 smithi121 bash[36902]: cluster 2022-04-22T19:23:31.390301+0000 mgr.y (mgr.34107) 4041 : cluster [DBG] pgmap v5466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:32.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:32.061Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:33 smithi121 bash[36902]: cluster 2022-04-22T19:23:33.391545+0000 mgr.y (mgr.34107) 4042 : cluster [DBG] pgmap v5467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:33 smithi121 bash[36902]: audit 2022-04-22T19:23:33.601970+0000 mon.a (mon.0) 6847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:33 smithi115 bash[45495]: cluster 2022-04-22T19:23:33.391545+0000 mgr.y (mgr.34107) 4042 : cluster [DBG] pgmap v5467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:33 smithi115 bash[45495]: audit 2022-04-22T19:23:33.601970+0000 mon.a (mon.0) 6847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:34.050 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:33.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:33 smithi115 bash[46934]: cluster 2022-04-22T19:23:33.391545+0000 mgr.y (mgr.34107) 4042 : cluster [DBG] pgmap v5467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:33 smithi115 bash[46934]: audit 2022-04-22T19:23:33.601970+0000 mon.a (mon.0) 6847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:35 smithi121 bash[36902]: audit 2022-04-22T19:23:35.177485+0000 mon.a (mon.0) 6848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:35 smithi115 bash[45495]: audit 2022-04-22T19:23:35.177485+0000 mon.a (mon.0) 6848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:35.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:35 smithi115 bash[46934]: audit 2022-04-22T19:23:35.177485+0000 mon.a (mon.0) 6848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:36 smithi121 bash[36902]: cluster 2022-04-22T19:23:35.392614+0000 mgr.y (mgr.34107) 4043 : cluster [DBG] pgmap v5468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:36 smithi115 bash[45495]: cluster 2022-04-22T19:23:35.392614+0000 mgr.y (mgr.34107) 4043 : cluster [DBG] pgmap v5468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:36.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:36 smithi115 bash[46934]: cluster 2022-04-22T19:23:35.392614+0000 mgr.y (mgr.34107) 4043 : cluster [DBG] pgmap v5468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:37 smithi121 bash[36902]: cluster 2022-04-22T19:23:37.393691+0000 mgr.y (mgr.34107) 4044 : cluster [DBG] pgmap v5469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:37.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:37 smithi115 bash[45495]: cluster 2022-04-22T19:23:37.393691+0000 mgr.y (mgr.34107) 4044 : cluster [DBG] pgmap v5469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:37.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:37 smithi115 bash[46934]: cluster 2022-04-22T19:23:37.393691+0000 mgr.y (mgr.34107) 4044 : cluster [DBG] pgmap v5469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:38 smithi121 bash[36902]: audit 2022-04-22T19:23:38.605584+0000 mon.a (mon.0) 6849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:39.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:38 smithi115 bash[45495]: audit 2022-04-22T19:23:38.605584+0000 mon.a (mon.0) 6849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:39.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:38 smithi115 bash[46934]: audit 2022-04-22T19:23:38.605584+0000 mon.a (mon.0) 6849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:39 smithi121 bash[36902]: cluster 2022-04-22T19:23:39.394869+0000 mgr.y (mgr.34107) 4045 : cluster [DBG] pgmap v5470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:39 smithi115 bash[45495]: cluster 2022-04-22T19:23:39.394869+0000 mgr.y (mgr.34107) 4045 : cluster [DBG] pgmap v5470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:39 smithi115 bash[46934]: cluster 2022-04-22T19:23:39.394869+0000 mgr.y (mgr.34107) 4045 : cluster [DBG] pgmap v5470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:40 smithi121 bash[36902]: audit 2022-04-22T19:23:40.180737+0000 mon.a (mon.0) 6850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:41.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:40 smithi115 bash[45495]: audit 2022-04-22T19:23:40.180737+0000 mon.a (mon.0) 6850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:41.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:40 smithi115 bash[46934]: audit 2022-04-22T19:23:40.180737+0000 mon.a (mon.0) 6850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:42.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:41 smithi115 bash[45495]: cluster 2022-04-22T19:23:41.396186+0000 mgr.y (mgr.34107) 4046 : cluster [DBG] pgmap v5471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:42.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:41 smithi115 bash[46934]: cluster 2022-04-22T19:23:41.396186+0000 mgr.y (mgr.34107) 4046 : cluster [DBG] pgmap v5471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:41 smithi121 bash[36902]: cluster 2022-04-22T19:23:41.396186+0000 mgr.y (mgr.34107) 4046 : cluster [DBG] pgmap v5471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:42.062Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:43.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: audit 2022-04-22T19:23:43.322255+0000 mon.a (mon.0) 6851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:23:43.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: audit 2022-04-22T19:23:43.325145+0000 mon.a (mon.0) 6852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:23:43.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: audit 2022-04-22T19:23:43.326611+0000 mon.a (mon.0) 6853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:23:43.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: cluster 2022-04-22T19:23:43.396979+0000 mgr.y (mgr.34107) 4047 : cluster [DBG] pgmap v5472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:43.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: audit 2022-04-22T19:23:43.609154+0000 mon.a (mon.0) 6854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:43.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[45495]: audit 2022-04-22T19:23:43.691217+0000 mon.a (mon.0) 6855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: audit 2022-04-22T19:23:43.322255+0000 mon.a (mon.0) 6851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: audit 2022-04-22T19:23:43.325145+0000 mon.a (mon.0) 6852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: audit 2022-04-22T19:23:43.326611+0000 mon.a (mon.0) 6853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: cluster 2022-04-22T19:23:43.396979+0000 mgr.y (mgr.34107) 4047 : cluster [DBG] pgmap v5472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: audit 2022-04-22T19:23:43.609154+0000 mon.a (mon.0) 6854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:43.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[46934]: audit 2022-04-22T19:23:43.691217+0000 mon.a (mon.0) 6855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:23:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: audit 2022-04-22T19:23:43.322255+0000 mon.a (mon.0) 6851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:23:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: audit 2022-04-22T19:23:43.325145+0000 mon.a (mon.0) 6852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:23:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: audit 2022-04-22T19:23:43.326611+0000 mon.a (mon.0) 6853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:23:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: cluster 2022-04-22T19:23:43.396979+0000 mgr.y (mgr.34107) 4047 : cluster [DBG] pgmap v5472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: audit 2022-04-22T19:23:43.609154+0000 mon.a (mon.0) 6854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:43 smithi121 bash[36902]: audit 2022-04-22T19:23:43.691217+0000 mon.a (mon.0) 6855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:23:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:43.936Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:45 smithi121 bash[36902]: audit 2022-04-22T19:23:45.183977+0000 mon.a (mon.0) 6856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:45 smithi115 bash[45495]: audit 2022-04-22T19:23:45.183977+0000 mon.a (mon.0) 6856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:45 smithi115 bash[46934]: audit 2022-04-22T19:23:45.183977+0000 mon.a (mon.0) 6856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:46 smithi121 bash[36902]: cluster 2022-04-22T19:23:45.398009+0000 mgr.y (mgr.34107) 4048 : cluster [DBG] pgmap v5473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:46 smithi115 bash[45495]: cluster 2022-04-22T19:23:45.398009+0000 mgr.y (mgr.34107) 4048 : cluster [DBG] pgmap v5473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:46 smithi115 bash[46934]: cluster 2022-04-22T19:23:45.398009+0000 mgr.y (mgr.34107) 4048 : cluster [DBG] pgmap v5473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:47 smithi121 bash[36902]: cluster 2022-04-22T19:23:47.399207+0000 mgr.y (mgr.34107) 4049 : cluster [DBG] pgmap v5474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:47.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:47 smithi115 bash[45495]: cluster 2022-04-22T19:23:47.399207+0000 mgr.y (mgr.34107) 4049 : cluster [DBG] pgmap v5474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:47.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:47 smithi115 bash[46934]: cluster 2022-04-22T19:23:47.399207+0000 mgr.y (mgr.34107) 4049 : cluster [DBG] pgmap v5474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:48 smithi121 bash[36902]: audit 2022-04-22T19:23:48.612335+0000 mon.a (mon.0) 6857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:49.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:48 smithi115 bash[45495]: audit 2022-04-22T19:23:48.612335+0000 mon.a (mon.0) 6857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:49.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:48 smithi115 bash[46934]: audit 2022-04-22T19:23:48.612335+0000 mon.a (mon.0) 6857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:50.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:49 smithi115 bash[45495]: cluster 2022-04-22T19:23:49.400314+0000 mgr.y (mgr.34107) 4050 : cluster [DBG] pgmap v5475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:50.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:49 smithi115 bash[46934]: cluster 2022-04-22T19:23:49.400314+0000 mgr.y (mgr.34107) 4050 : cluster [DBG] pgmap v5475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:49 smithi121 bash[36902]: cluster 2022-04-22T19:23:49.400314+0000 mgr.y (mgr.34107) 4050 : cluster [DBG] pgmap v5475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:51.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:50 smithi115 bash[45495]: audit 2022-04-22T19:23:50.187080+0000 mon.a (mon.0) 6858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:51.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:50 smithi115 bash[46934]: audit 2022-04-22T19:23:50.187080+0000 mon.a (mon.0) 6858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:50 smithi121 bash[36902]: audit 2022-04-22T19:23:50.187080+0000 mon.a (mon.0) 6858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:51 smithi115 bash[45495]: cluster 2022-04-22T19:23:51.401561+0000 mgr.y (mgr.34107) 4051 : cluster [DBG] pgmap v5476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:52.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:51 smithi115 bash[46934]: cluster 2022-04-22T19:23:51.401561+0000 mgr.y (mgr.34107) 4051 : cluster [DBG] pgmap v5476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:51 smithi121 bash[36902]: cluster 2022-04-22T19:23:51.401561+0000 mgr.y (mgr.34107) 4051 : cluster [DBG] pgmap v5476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:52.065Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:53.403 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:23:53.749 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (106m) 3m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (108m) 3m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (108m) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (116m) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (116m) 3m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (115m) 3m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:23:53.750 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (114m) 4m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (115m) 3m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (108m) 3m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (107m) 4m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (113m) 3m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (112m) 3m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (112m) 3m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (112m) 3m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (111m) 4m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (110m) 4m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:23:53.751 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (110m) 4m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:23:53.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (109m) 4m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:23:53.752 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (107m) 4m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:23:54.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:23:53.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[46934]: audit 2022-04-22T19:23:53.375245+0000 mgr.y (mgr.34107) 4052 : audit [DBG] from='client.39636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[46934]: cluster 2022-04-22T19:23:53.402647+0000 mgr.y (mgr.34107) 4053 : cluster [DBG] pgmap v5477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[46934]: audit 2022-04-22T19:23:53.615798+0000 mon.a (mon.0) 6859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[45495]: audit 2022-04-22T19:23:53.375245+0000 mgr.y (mgr.34107) 4052 : audit [DBG] from='client.39636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[45495]: cluster 2022-04-22T19:23:53.402647+0000 mgr.y (mgr.34107) 4053 : cluster [DBG] pgmap v5477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:54.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:53 smithi115 bash[45495]: audit 2022-04-22T19:23:53.615798+0000 mon.a (mon.0) 6859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:54.155 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:23:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:23:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:23:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:23:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:23:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:23:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:23:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:23:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:23:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:23:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:23:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:23:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:23:54.159 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:23:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:53 smithi121 bash[36902]: audit 2022-04-22T19:23:53.375245+0000 mgr.y (mgr.34107) 4052 : audit [DBG] from='client.39636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:53 smithi121 bash[36902]: cluster 2022-04-22T19:23:53.402647+0000 mgr.y (mgr.34107) 4053 : cluster [DBG] pgmap v5477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:53 smithi121 bash[36902]: audit 2022-04-22T19:23:53.615798+0000 mon.a (mon.0) 6859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:55.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:54 smithi115 bash[45495]: audit 2022-04-22T19:23:53.717562+0000 mgr.y (mgr.34107) 4054 : audit [DBG] from='client.39642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:55.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:54 smithi115 bash[45495]: audit 2022-04-22T19:23:54.142613+0000 mon.a (mon.0) 6860 : audit [DBG] from='client.? 172.21.15.115:0/1424704954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:55.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:54 smithi115 bash[46934]: audit 2022-04-22T19:23:53.717562+0000 mgr.y (mgr.34107) 4054 : audit [DBG] from='client.39642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:55.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:54 smithi115 bash[46934]: audit 2022-04-22T19:23:54.142613+0000 mon.a (mon.0) 6860 : audit [DBG] from='client.? 172.21.15.115:0/1424704954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:54 smithi121 bash[36902]: audit 2022-04-22T19:23:53.717562+0000 mgr.y (mgr.34107) 4054 : audit [DBG] from='client.39642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:23:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:54 smithi121 bash[36902]: audit 2022-04-22T19:23:54.142613+0000 mon.a (mon.0) 6860 : audit [DBG] from='client.? 172.21.15.115:0/1424704954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:23:56.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:55 smithi115 bash[45495]: audit 2022-04-22T19:23:55.189690+0000 mon.a (mon.0) 6861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:56.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:55 smithi115 bash[45495]: cluster 2022-04-22T19:23:55.403731+0000 mgr.y (mgr.34107) 4055 : cluster [DBG] pgmap v5478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:56.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:55 smithi115 bash[46934]: audit 2022-04-22T19:23:55.189690+0000 mon.a (mon.0) 6861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:56.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:55 smithi115 bash[46934]: cluster 2022-04-22T19:23:55.403731+0000 mgr.y (mgr.34107) 4055 : cluster [DBG] pgmap v5478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:55 smithi121 bash[36902]: audit 2022-04-22T19:23:55.189690+0000 mon.a (mon.0) 6861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:23:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:55 smithi121 bash[36902]: cluster 2022-04-22T19:23:55.403731+0000 mgr.y (mgr.34107) 4055 : cluster [DBG] pgmap v5478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:57 smithi121 bash[36902]: cluster 2022-04-22T19:23:57.404816+0000 mgr.y (mgr.34107) 4056 : cluster [DBG] pgmap v5479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:57.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:57 smithi115 bash[45495]: cluster 2022-04-22T19:23:57.404816+0000 mgr.y (mgr.34107) 4056 : cluster [DBG] pgmap v5479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:57.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:57 smithi115 bash[46934]: cluster 2022-04-22T19:23:57.404816+0000 mgr.y (mgr.34107) 4056 : cluster [DBG] pgmap v5479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:23:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:58 smithi121 bash[36902]: audit 2022-04-22T19:23:58.619426+0000 mon.a (mon.0) 6862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:59.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:58 smithi115 bash[45495]: audit 2022-04-22T19:23:58.619426+0000 mon.a (mon.0) 6862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:23:59.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:58 smithi115 bash[46934]: audit 2022-04-22T19:23:58.619426+0000 mon.a (mon.0) 6862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:00.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:23:59 smithi115 bash[45495]: cluster 2022-04-22T19:23:59.405985+0000 mgr.y (mgr.34107) 4057 : cluster [DBG] pgmap v5480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:00.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:23:59 smithi115 bash[46934]: cluster 2022-04-22T19:23:59.405985+0000 mgr.y (mgr.34107) 4057 : cluster [DBG] pgmap v5480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:23:59 smithi121 bash[36902]: cluster 2022-04-22T19:23:59.405985+0000 mgr.y (mgr.34107) 4057 : cluster [DBG] pgmap v5480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:01.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:00 smithi115 bash[46934]: audit 2022-04-22T19:24:00.192837+0000 mon.a (mon.0) 6863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:01.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:00 smithi115 bash[45495]: audit 2022-04-22T19:24:00.192837+0000 mon.a (mon.0) 6863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:00 smithi121 bash[36902]: audit 2022-04-22T19:24:00.192837+0000 mon.a (mon.0) 6863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:02.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:01 smithi115 bash[46934]: cluster 2022-04-22T19:24:01.407202+0000 mgr.y (mgr.34107) 4058 : cluster [DBG] pgmap v5481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:02.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:01 smithi115 bash[45495]: cluster 2022-04-22T19:24:01.407202+0000 mgr.y (mgr.34107) 4058 : cluster [DBG] pgmap v5481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:01 smithi121 bash[36902]: cluster 2022-04-22T19:24:01.407202+0000 mgr.y (mgr.34107) 4058 : cluster [DBG] pgmap v5481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:02.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:02.065Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:04.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:03.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:04.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:03 smithi115 bash[45495]: cluster 2022-04-22T19:24:03.408473+0000 mgr.y (mgr.34107) 4059 : cluster [DBG] pgmap v5482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:04.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:03 smithi115 bash[45495]: audit 2022-04-22T19:24:03.623298+0000 mon.a (mon.0) 6864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:03 smithi115 bash[46934]: cluster 2022-04-22T19:24:03.408473+0000 mgr.y (mgr.34107) 4059 : cluster [DBG] pgmap v5482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:03 smithi115 bash[46934]: audit 2022-04-22T19:24:03.623298+0000 mon.a (mon.0) 6864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:03 smithi121 bash[36902]: cluster 2022-04-22T19:24:03.408473+0000 mgr.y (mgr.34107) 4059 : cluster [DBG] pgmap v5482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:03 smithi121 bash[36902]: audit 2022-04-22T19:24:03.623298+0000 mon.a (mon.0) 6864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:05 smithi121 bash[36902]: audit 2022-04-22T19:24:05.195882+0000 mon.a (mon.0) 6865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:05.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:05 smithi115 bash[45495]: audit 2022-04-22T19:24:05.195882+0000 mon.a (mon.0) 6865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:05 smithi115 bash[46934]: audit 2022-04-22T19:24:05.195882+0000 mon.a (mon.0) 6865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:06 smithi115 bash[45495]: cluster 2022-04-22T19:24:05.409579+0000 mgr.y (mgr.34107) 4060 : cluster [DBG] pgmap v5483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:06 smithi115 bash[46934]: cluster 2022-04-22T19:24:05.409579+0000 mgr.y (mgr.34107) 4060 : cluster [DBG] pgmap v5483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:06 smithi121 bash[36902]: cluster 2022-04-22T19:24:05.409579+0000 mgr.y (mgr.34107) 4060 : cluster [DBG] pgmap v5483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:07 smithi121 bash[36902]: cluster 2022-04-22T19:24:07.410636+0000 mgr.y (mgr.34107) 4061 : cluster [DBG] pgmap v5484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:07 smithi115 bash[45495]: cluster 2022-04-22T19:24:07.410636+0000 mgr.y (mgr.34107) 4061 : cluster [DBG] pgmap v5484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:07.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:07 smithi115 bash[46934]: cluster 2022-04-22T19:24:07.410636+0000 mgr.y (mgr.34107) 4061 : cluster [DBG] pgmap v5484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:08 smithi115 bash[46934]: audit 2022-04-22T19:24:08.627021+0000 mon.a (mon.0) 6866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:09.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:08 smithi115 bash[45495]: audit 2022-04-22T19:24:08.627021+0000 mon.a (mon.0) 6866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:08 smithi121 bash[36902]: audit 2022-04-22T19:24:08.627021+0000 mon.a (mon.0) 6866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:10.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:09 smithi115 bash[45495]: cluster 2022-04-22T19:24:09.411739+0000 mgr.y (mgr.34107) 4062 : cluster [DBG] pgmap v5485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:10.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:09 smithi115 bash[46934]: cluster 2022-04-22T19:24:09.411739+0000 mgr.y (mgr.34107) 4062 : cluster [DBG] pgmap v5485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:09 smithi121 bash[36902]: cluster 2022-04-22T19:24:09.411739+0000 mgr.y (mgr.34107) 4062 : cluster [DBG] pgmap v5485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:11.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:10 smithi115 bash[45495]: audit 2022-04-22T19:24:10.199134+0000 mon.a (mon.0) 6867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:11.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:10 smithi115 bash[46934]: audit 2022-04-22T19:24:10.199134+0000 mon.a (mon.0) 6867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:10 smithi121 bash[36902]: audit 2022-04-22T19:24:10.199134+0000 mon.a (mon.0) 6867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:12.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:11 smithi115 bash[45495]: cluster 2022-04-22T19:24:11.412900+0000 mgr.y (mgr.34107) 4063 : cluster [DBG] pgmap v5486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:12.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:11 smithi115 bash[46934]: cluster 2022-04-22T19:24:11.412900+0000 mgr.y (mgr.34107) 4063 : cluster [DBG] pgmap v5486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:11 smithi121 bash[36902]: cluster 2022-04-22T19:24:11.412900+0000 mgr.y (mgr.34107) 4063 : cluster [DBG] pgmap v5486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:12.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:12.064Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:14.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:13.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:14.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:13 smithi115 bash[45495]: cluster 2022-04-22T19:24:13.414021+0000 mgr.y (mgr.34107) 4064 : cluster [DBG] pgmap v5487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:14.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:13 smithi115 bash[45495]: audit 2022-04-22T19:24:13.630781+0000 mon.a (mon.0) 6868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:13 smithi115 bash[46934]: cluster 2022-04-22T19:24:13.414021+0000 mgr.y (mgr.34107) 4064 : cluster [DBG] pgmap v5487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:14.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:13 smithi115 bash[46934]: audit 2022-04-22T19:24:13.630781+0000 mon.a (mon.0) 6868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:13 smithi121 bash[36902]: cluster 2022-04-22T19:24:13.414021+0000 mgr.y (mgr.34107) 4064 : cluster [DBG] pgmap v5487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:13 smithi121 bash[36902]: audit 2022-04-22T19:24:13.630781+0000 mon.a (mon.0) 6868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:14 smithi115 bash[45495]: audit 2022-04-22T19:24:14.612254+0000 mon.a (mon.0) 6869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:24:15.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:14 smithi115 bash[45495]: audit 2022-04-22T19:24:14.620376+0000 mon.a (mon.0) 6870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:24:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:14 smithi115 bash[46934]: audit 2022-04-22T19:24:14.612254+0000 mon.a (mon.0) 6869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:24:15.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:14 smithi115 bash[46934]: audit 2022-04-22T19:24:14.620376+0000 mon.a (mon.0) 6870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:24:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:14 smithi121 bash[36902]: audit 2022-04-22T19:24:14.612254+0000 mon.a (mon.0) 6869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:24:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:14 smithi121 bash[36902]: audit 2022-04-22T19:24:14.620376+0000 mon.a (mon.0) 6870 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:24:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:15 smithi115 bash[45495]: audit 2022-04-22T19:24:15.202357+0000 mon.a (mon.0) 6871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:16.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:15 smithi115 bash[45495]: cluster 2022-04-22T19:24:15.415043+0000 mgr.y (mgr.34107) 4065 : cluster [DBG] pgmap v5488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:16.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:15 smithi115 bash[46934]: audit 2022-04-22T19:24:15.202357+0000 mon.a (mon.0) 6871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:16.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:15 smithi115 bash[46934]: cluster 2022-04-22T19:24:15.415043+0000 mgr.y (mgr.34107) 4065 : cluster [DBG] pgmap v5488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:15 smithi121 bash[36902]: audit 2022-04-22T19:24:15.202357+0000 mon.a (mon.0) 6871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:15 smithi121 bash[36902]: cluster 2022-04-22T19:24:15.415043+0000 mgr.y (mgr.34107) 4065 : cluster [DBG] pgmap v5488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:17 smithi121 bash[36902]: cluster 2022-04-22T19:24:17.416146+0000 mgr.y (mgr.34107) 4066 : cluster [DBG] pgmap v5489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:17.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:17 smithi115 bash[45495]: cluster 2022-04-22T19:24:17.416146+0000 mgr.y (mgr.34107) 4066 : cluster [DBG] pgmap v5489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:17 smithi115 bash[46934]: cluster 2022-04-22T19:24:17.416146+0000 mgr.y (mgr.34107) 4066 : cluster [DBG] pgmap v5489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:18 smithi115 bash[46934]: audit 2022-04-22T19:24:18.634578+0000 mon.a (mon.0) 6872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:19.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:18 smithi115 bash[45495]: audit 2022-04-22T19:24:18.634578+0000 mon.a (mon.0) 6872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:18 smithi121 bash[36902]: audit 2022-04-22T19:24:18.634578+0000 mon.a (mon.0) 6872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:20.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:19 smithi115 bash[45495]: cluster 2022-04-22T19:24:19.417174+0000 mgr.y (mgr.34107) 4067 : cluster [DBG] pgmap v5490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:20.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:19 smithi115 bash[46934]: cluster 2022-04-22T19:24:19.417174+0000 mgr.y (mgr.34107) 4067 : cluster [DBG] pgmap v5490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:19 smithi121 bash[36902]: cluster 2022-04-22T19:24:19.417174+0000 mgr.y (mgr.34107) 4067 : cluster [DBG] pgmap v5490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:21.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:20 smithi115 bash[45495]: audit 2022-04-22T19:24:20.205605+0000 mon.a (mon.0) 6873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:21.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:20 smithi115 bash[46934]: audit 2022-04-22T19:24:20.205605+0000 mon.a (mon.0) 6873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:20 smithi121 bash[36902]: audit 2022-04-22T19:24:20.205605+0000 mon.a (mon.0) 6873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:21 smithi115 bash[45495]: cluster 2022-04-22T19:24:21.418280+0000 mgr.y (mgr.34107) 4068 : cluster [DBG] pgmap v5491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:22.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:21 smithi115 bash[46934]: cluster 2022-04-22T19:24:21.418280+0000 mgr.y (mgr.34107) 4068 : cluster [DBG] pgmap v5491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:21 smithi121 bash[36902]: cluster 2022-04-22T19:24:21.418280+0000 mgr.y (mgr.34107) 4068 : cluster [DBG] pgmap v5491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:22.064Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:24.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:23.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:24.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:23 smithi115 bash[45495]: cluster 2022-04-22T19:24:23.419513+0000 mgr.y (mgr.34107) 4069 : cluster [DBG] pgmap v5492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:24.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:23 smithi115 bash[45495]: audit 2022-04-22T19:24:23.638175+0000 mon.a (mon.0) 6874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:23 smithi115 bash[46934]: cluster 2022-04-22T19:24:23.419513+0000 mgr.y (mgr.34107) 4069 : cluster [DBG] pgmap v5492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:24.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:23 smithi115 bash[46934]: audit 2022-04-22T19:24:23.638175+0000 mon.a (mon.0) 6874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:23 smithi121 bash[36902]: cluster 2022-04-22T19:24:23.419513+0000 mgr.y (mgr.34107) 4069 : cluster [DBG] pgmap v5492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:23 smithi121 bash[36902]: audit 2022-04-22T19:24:23.638175+0000 mon.a (mon.0) 6874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:24.533 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:24:24.879 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:24:24.879 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (107m) 3m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:24:24.879 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (109m) 3m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (108m) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (116m) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (117m) 3m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (116m) 3m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (115m) 4m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (115m) 3m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:24:24.880 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (108m) 3m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (108m) 4m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (114m) 3m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (113m) 3m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (113m) 3m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (112m) 3m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (111m) 4m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (111m) 4m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (110m) 4m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (109m) 4m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:24:24.881 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (108m) 4m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:24:25.285 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:24:25.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:24:25.287 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:24:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:25 smithi115 bash[45495]: audit 2022-04-22T19:24:25.208366+0000 mon.a (mon.0) 6875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:25.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:25 smithi115 bash[46934]: audit 2022-04-22T19:24:25.208366+0000 mon.a (mon.0) 6875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:25 smithi121 bash[36902]: audit 2022-04-22T19:24:25.208366+0000 mon.a (mon.0) 6875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[45495]: audit 2022-04-22T19:24:24.503140+0000 mgr.y (mgr.34107) 4070 : audit [DBG] from='client.49588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[45495]: audit 2022-04-22T19:24:24.849188+0000 mgr.y (mgr.34107) 4071 : audit [DBG] from='client.39660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[45495]: audit 2022-04-22T19:24:25.274313+0000 mon.c (mon.1) 189 : audit [DBG] from='client.? 172.21.15.115:0/3623181950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:26.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[45495]: cluster 2022-04-22T19:24:25.420569+0000 mgr.y (mgr.34107) 4072 : cluster [DBG] pgmap v5493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[46934]: audit 2022-04-22T19:24:24.503140+0000 mgr.y (mgr.34107) 4070 : audit [DBG] from='client.49588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[46934]: audit 2022-04-22T19:24:24.849188+0000 mgr.y (mgr.34107) 4071 : audit [DBG] from='client.39660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[46934]: audit 2022-04-22T19:24:25.274313+0000 mon.c (mon.1) 189 : audit [DBG] from='client.? 172.21.15.115:0/3623181950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:26 smithi115 bash[46934]: cluster 2022-04-22T19:24:25.420569+0000 mgr.y (mgr.34107) 4072 : cluster [DBG] pgmap v5493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:26 smithi121 bash[36902]: audit 2022-04-22T19:24:24.503140+0000 mgr.y (mgr.34107) 4070 : audit [DBG] from='client.49588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:26 smithi121 bash[36902]: audit 2022-04-22T19:24:24.849188+0000 mgr.y (mgr.34107) 4071 : audit [DBG] from='client.39660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:26 smithi121 bash[36902]: audit 2022-04-22T19:24:25.274313+0000 mon.c (mon.1) 189 : audit [DBG] from='client.? 172.21.15.115:0/3623181950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:26 smithi121 bash[36902]: cluster 2022-04-22T19:24:25.420569+0000 mgr.y (mgr.34107) 4072 : cluster [DBG] pgmap v5493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:27 smithi121 bash[36902]: cluster 2022-04-22T19:24:27.421666+0000 mgr.y (mgr.34107) 4073 : cluster [DBG] pgmap v5494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:27 smithi115 bash[45495]: cluster 2022-04-22T19:24:27.421666+0000 mgr.y (mgr.34107) 4073 : cluster [DBG] pgmap v5494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:27 smithi115 bash[46934]: cluster 2022-04-22T19:24:27.421666+0000 mgr.y (mgr.34107) 4073 : cluster [DBG] pgmap v5494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:29.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:28 smithi115 bash[45495]: audit 2022-04-22T19:24:28.642100+0000 mon.a (mon.0) 6876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:29.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:28 smithi115 bash[46934]: audit 2022-04-22T19:24:28.642100+0000 mon.a (mon.0) 6876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:28 smithi121 bash[36902]: audit 2022-04-22T19:24:28.642100+0000 mon.a (mon.0) 6876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:30.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:29 smithi115 bash[45495]: cluster 2022-04-22T19:24:29.422756+0000 mgr.y (mgr.34107) 4074 : cluster [DBG] pgmap v5495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:30.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:29 smithi115 bash[46934]: cluster 2022-04-22T19:24:29.422756+0000 mgr.y (mgr.34107) 4074 : cluster [DBG] pgmap v5495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:29 smithi121 bash[36902]: cluster 2022-04-22T19:24:29.422756+0000 mgr.y (mgr.34107) 4074 : cluster [DBG] pgmap v5495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:31.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:30 smithi115 bash[45495]: audit 2022-04-22T19:24:30.211130+0000 mon.a (mon.0) 6877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:31.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:30 smithi115 bash[46934]: audit 2022-04-22T19:24:30.211130+0000 mon.a (mon.0) 6877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:30 smithi121 bash[36902]: audit 2022-04-22T19:24:30.211130+0000 mon.a (mon.0) 6877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:31 smithi115 bash[45495]: cluster 2022-04-22T19:24:31.423881+0000 mgr.y (mgr.34107) 4075 : cluster [DBG] pgmap v5496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:32.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:31 smithi115 bash[46934]: cluster 2022-04-22T19:24:31.423881+0000 mgr.y (mgr.34107) 4075 : cluster [DBG] pgmap v5496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:31 smithi121 bash[36902]: cluster 2022-04-22T19:24:31.423881+0000 mgr.y (mgr.34107) 4075 : cluster [DBG] pgmap v5496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:32.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:32.067Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:34.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:33.939Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:33 smithi115 bash[45495]: cluster 2022-04-22T19:24:33.425012+0000 mgr.y (mgr.34107) 4076 : cluster [DBG] pgmap v5497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:34.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:33 smithi115 bash[45495]: audit 2022-04-22T19:24:33.645899+0000 mon.a (mon.0) 6878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:33 smithi115 bash[46934]: cluster 2022-04-22T19:24:33.425012+0000 mgr.y (mgr.34107) 4076 : cluster [DBG] pgmap v5497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:33 smithi115 bash[46934]: audit 2022-04-22T19:24:33.645899+0000 mon.a (mon.0) 6878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:33 smithi121 bash[36902]: cluster 2022-04-22T19:24:33.425012+0000 mgr.y (mgr.34107) 4076 : cluster [DBG] pgmap v5497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:33 smithi121 bash[36902]: audit 2022-04-22T19:24:33.645899+0000 mon.a (mon.0) 6878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:35 smithi115 bash[45495]: audit 2022-04-22T19:24:35.214282+0000 mon.a (mon.0) 6879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:35 smithi115 bash[46934]: audit 2022-04-22T19:24:35.214282+0000 mon.a (mon.0) 6879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:35 smithi121 bash[36902]: audit 2022-04-22T19:24:35.214282+0000 mon.a (mon.0) 6879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:36 smithi115 bash[45495]: cluster 2022-04-22T19:24:35.426118+0000 mgr.y (mgr.34107) 4077 : cluster [DBG] pgmap v5498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:36 smithi115 bash[46934]: cluster 2022-04-22T19:24:35.426118+0000 mgr.y (mgr.34107) 4077 : cluster [DBG] pgmap v5498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:36 smithi121 bash[36902]: cluster 2022-04-22T19:24:35.426118+0000 mgr.y (mgr.34107) 4077 : cluster [DBG] pgmap v5498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:37 smithi121 bash[36902]: cluster 2022-04-22T19:24:37.427197+0000 mgr.y (mgr.34107) 4078 : cluster [DBG] pgmap v5499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:37 smithi115 bash[46934]: cluster 2022-04-22T19:24:37.427197+0000 mgr.y (mgr.34107) 4078 : cluster [DBG] pgmap v5499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:37.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:37 smithi115 bash[45495]: cluster 2022-04-22T19:24:37.427197+0000 mgr.y (mgr.34107) 4078 : cluster [DBG] pgmap v5499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:39.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:38 smithi115 bash[45495]: audit 2022-04-22T19:24:38.649431+0000 mon.a (mon.0) 6880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:39.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:38 smithi115 bash[46934]: audit 2022-04-22T19:24:38.649431+0000 mon.a (mon.0) 6880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:38 smithi121 bash[36902]: audit 2022-04-22T19:24:38.649431+0000 mon.a (mon.0) 6880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:40.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:39 smithi115 bash[45495]: cluster 2022-04-22T19:24:39.428310+0000 mgr.y (mgr.34107) 4079 : cluster [DBG] pgmap v5500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:39 smithi115 bash[46934]: cluster 2022-04-22T19:24:39.428310+0000 mgr.y (mgr.34107) 4079 : cluster [DBG] pgmap v5500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:39 smithi121 bash[36902]: cluster 2022-04-22T19:24:39.428310+0000 mgr.y (mgr.34107) 4079 : cluster [DBG] pgmap v5500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:41.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:40 smithi115 bash[45495]: audit 2022-04-22T19:24:40.217464+0000 mon.a (mon.0) 6881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:41.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:40 smithi115 bash[46934]: audit 2022-04-22T19:24:40.217464+0000 mon.a (mon.0) 6881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:40 smithi121 bash[36902]: audit 2022-04-22T19:24:40.217464+0000 mon.a (mon.0) 6881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:42.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:42.067Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:43.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:42 smithi115 bash[45495]: cluster 2022-04-22T19:24:41.429369+0000 mgr.y (mgr.34107) 4080 : cluster [DBG] pgmap v5501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:43.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:42 smithi115 bash[46934]: cluster 2022-04-22T19:24:41.429369+0000 mgr.y (mgr.34107) 4080 : cluster [DBG] pgmap v5501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:42 smithi121 bash[36902]: cluster 2022-04-22T19:24:41.429369+0000 mgr.y (mgr.34107) 4080 : cluster [DBG] pgmap v5501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:43 smithi121 bash[36902]: audit 2022-04-22T19:24:43.653522+0000 mon.a (mon.0) 6882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:43 smithi121 bash[36902]: audit 2022-04-22T19:24:43.699017+0000 mon.a (mon.0) 6883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:43 smithi121 bash[36902]: audit 2022-04-22T19:24:43.701945+0000 mon.a (mon.0) 6884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:43 smithi121 bash[36902]: audit 2022-04-22T19:24:43.703816+0000 mon.a (mon.0) 6885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:24:44.049 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:43.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:44.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[45495]: audit 2022-04-22T19:24:43.653522+0000 mon.a (mon.0) 6882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[45495]: audit 2022-04-22T19:24:43.699017+0000 mon.a (mon.0) 6883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:24:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[45495]: audit 2022-04-22T19:24:43.701945+0000 mon.a (mon.0) 6884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:24:44.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[45495]: audit 2022-04-22T19:24:43.703816+0000 mon.a (mon.0) 6885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:24:44.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[46934]: audit 2022-04-22T19:24:43.653522+0000 mon.a (mon.0) 6882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:44.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[46934]: audit 2022-04-22T19:24:43.699017+0000 mon.a (mon.0) 6883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:24:44.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[46934]: audit 2022-04-22T19:24:43.701945+0000 mon.a (mon.0) 6884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:24:44.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:43 smithi115 bash[46934]: audit 2022-04-22T19:24:43.703816+0000 mon.a (mon.0) 6885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:24:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:45 smithi121 bash[36902]: cluster 2022-04-22T19:24:43.430539+0000 mgr.y (mgr.34107) 4081 : cluster [DBG] pgmap v5502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:45 smithi121 bash[36902]: audit 2022-04-22T19:24:44.072231+0000 mon.a (mon.0) 6886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:24:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:45 smithi115 bash[45495]: cluster 2022-04-22T19:24:43.430539+0000 mgr.y (mgr.34107) 4081 : cluster [DBG] pgmap v5502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:45 smithi115 bash[45495]: audit 2022-04-22T19:24:44.072231+0000 mon.a (mon.0) 6886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:24:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:45 smithi115 bash[46934]: cluster 2022-04-22T19:24:43.430539+0000 mgr.y (mgr.34107) 4081 : cluster [DBG] pgmap v5502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:45 smithi115 bash[46934]: audit 2022-04-22T19:24:44.072231+0000 mon.a (mon.0) 6886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:24:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:46 smithi121 bash[36902]: audit 2022-04-22T19:24:45.220741+0000 mon.a (mon.0) 6887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:46 smithi115 bash[45495]: audit 2022-04-22T19:24:45.220741+0000 mon.a (mon.0) 6887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:46 smithi115 bash[46934]: audit 2022-04-22T19:24:45.220741+0000 mon.a (mon.0) 6887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:47 smithi121 bash[36902]: cluster 2022-04-22T19:24:45.431572+0000 mgr.y (mgr.34107) 4082 : cluster [DBG] pgmap v5503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:47.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:47 smithi115 bash[45495]: cluster 2022-04-22T19:24:45.431572+0000 mgr.y (mgr.34107) 4082 : cluster [DBG] pgmap v5503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:47.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:47 smithi115 bash[46934]: cluster 2022-04-22T19:24:45.431572+0000 mgr.y (mgr.34107) 4082 : cluster [DBG] pgmap v5503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:49 smithi121 bash[36902]: cluster 2022-04-22T19:24:47.432626+0000 mgr.y (mgr.34107) 4083 : cluster [DBG] pgmap v5504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:49 smithi121 bash[36902]: audit 2022-04-22T19:24:48.657003+0000 mon.a (mon.0) 6888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:49 smithi115 bash[45495]: cluster 2022-04-22T19:24:47.432626+0000 mgr.y (mgr.34107) 4083 : cluster [DBG] pgmap v5504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:49 smithi115 bash[45495]: audit 2022-04-22T19:24:48.657003+0000 mon.a (mon.0) 6888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:49 smithi115 bash[46934]: cluster 2022-04-22T19:24:47.432626+0000 mgr.y (mgr.34107) 4083 : cluster [DBG] pgmap v5504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:49.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:49 smithi115 bash[46934]: audit 2022-04-22T19:24:48.657003+0000 mon.a (mon.0) 6888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:51 smithi121 bash[36902]: cluster 2022-04-22T19:24:49.433692+0000 mgr.y (mgr.34107) 4084 : cluster [DBG] pgmap v5505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:51 smithi121 bash[36902]: audit 2022-04-22T19:24:50.223655+0000 mon.a (mon.0) 6889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:51 smithi115 bash[46934]: cluster 2022-04-22T19:24:49.433692+0000 mgr.y (mgr.34107) 4084 : cluster [DBG] pgmap v5505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:51 smithi115 bash[46934]: audit 2022-04-22T19:24:50.223655+0000 mon.a (mon.0) 6889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:51 smithi115 bash[45495]: cluster 2022-04-22T19:24:49.433692+0000 mgr.y (mgr.34107) 4084 : cluster [DBG] pgmap v5505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:51 smithi115 bash[45495]: audit 2022-04-22T19:24:50.223655+0000 mon.a (mon.0) 6889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:52.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:53 smithi121 bash[36902]: cluster 2022-04-22T19:24:51.434802+0000 mgr.y (mgr.34107) 4085 : cluster [DBG] pgmap v5506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:53.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:53 smithi115 bash[45495]: cluster 2022-04-22T19:24:51.434802+0000 mgr.y (mgr.34107) 4085 : cluster [DBG] pgmap v5506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:53.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:53 smithi115 bash[46934]: cluster 2022-04-22T19:24:51.434802+0000 mgr.y (mgr.34107) 4085 : cluster [DBG] pgmap v5506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:54.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:54 smithi115 bash[45495]: audit 2022-04-22T19:24:53.660710+0000 mon.a (mon.0) 6890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:54.301 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:24:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:24:53.940Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:24:54.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:54 smithi115 bash[46934]: audit 2022-04-22T19:24:53.660710+0000 mon.a (mon.0) 6890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:54 smithi121 bash[36902]: audit 2022-04-22T19:24:53.660710+0000 mon.a (mon.0) 6890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:55 smithi121 bash[36902]: cluster 2022-04-22T19:24:53.435967+0000 mgr.y (mgr.34107) 4086 : cluster [DBG] pgmap v5507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:55 smithi115 bash[45495]: cluster 2022-04-22T19:24:53.435967+0000 mgr.y (mgr.34107) 4086 : cluster [DBG] pgmap v5507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:55.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:55 smithi115 bash[46934]: cluster 2022-04-22T19:24:53.435967+0000 mgr.y (mgr.34107) 4086 : cluster [DBG] pgmap v5507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:55.664 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:24:56.013 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:24:56.013 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (108m) 4m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:24:56.013 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:24:56.013 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (109m) 4m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (109m) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (117m) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (117m) 4m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (116m) 4m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (115m) 5m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (116m) 4m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (109m) 4m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:24:56.014 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (109m) 5m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (114m) 4m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (114m) 4m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (113m) 4m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (113m) 4m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (112m) 5m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (111m) 5m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (111m) 5m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (110m) 5m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:24:56.015 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (108m) 5m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:24:56.411 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:24:56.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:24:56.411 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:24:56.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:56.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:24:56.412 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:24:56.413 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:24:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:56 smithi121 bash[36902]: audit 2022-04-22T19:24:55.226734+0000 mon.a (mon.0) 6891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:56 smithi115 bash[46934]: audit 2022-04-22T19:24:55.226734+0000 mon.a (mon.0) 6891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:56.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:56 smithi115 bash[45495]: audit 2022-04-22T19:24:55.226734+0000 mon.a (mon.0) 6891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:24:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:57 smithi121 bash[36902]: cluster 2022-04-22T19:24:55.437015+0000 mgr.y (mgr.34107) 4087 : cluster [DBG] pgmap v5508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:57 smithi121 bash[36902]: audit 2022-04-22T19:24:55.632940+0000 mgr.y (mgr.34107) 4088 : audit [DBG] from='client.49606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:57 smithi121 bash[36902]: audit 2022-04-22T19:24:55.976952+0000 mgr.y (mgr.34107) 4089 : audit [DBG] from='client.49612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:57 smithi121 bash[36902]: audit 2022-04-22T19:24:56.398995+0000 mon.a (mon.0) 6892 : audit [DBG] from='client.? 172.21.15.115:0/974309534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:57.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[45495]: cluster 2022-04-22T19:24:55.437015+0000 mgr.y (mgr.34107) 4087 : cluster [DBG] pgmap v5508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[45495]: audit 2022-04-22T19:24:55.632940+0000 mgr.y (mgr.34107) 4088 : audit [DBG] from='client.49606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[45495]: audit 2022-04-22T19:24:55.976952+0000 mgr.y (mgr.34107) 4089 : audit [DBG] from='client.49612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[45495]: audit 2022-04-22T19:24:56.398995+0000 mon.a (mon.0) 6892 : audit [DBG] from='client.? 172.21.15.115:0/974309534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[46934]: cluster 2022-04-22T19:24:55.437015+0000 mgr.y (mgr.34107) 4087 : cluster [DBG] pgmap v5508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[46934]: audit 2022-04-22T19:24:55.632940+0000 mgr.y (mgr.34107) 4088 : audit [DBG] from='client.49606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[46934]: audit 2022-04-22T19:24:55.976952+0000 mgr.y (mgr.34107) 4089 : audit [DBG] from='client.49612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:24:57.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:57 smithi115 bash[46934]: audit 2022-04-22T19:24:56.398995+0000 mon.a (mon.0) 6892 : audit [DBG] from='client.? 172.21.15.115:0/974309534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:24:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:59 smithi121 bash[36902]: cluster 2022-04-22T19:24:57.438093+0000 mgr.y (mgr.34107) 4090 : cluster [DBG] pgmap v5509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:24:59 smithi121 bash[36902]: audit 2022-04-22T19:24:58.664266+0000 mon.a (mon.0) 6893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:59 smithi115 bash[45495]: cluster 2022-04-22T19:24:57.438093+0000 mgr.y (mgr.34107) 4090 : cluster [DBG] pgmap v5509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:24:59 smithi115 bash[45495]: audit 2022-04-22T19:24:58.664266+0000 mon.a (mon.0) 6893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:24:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:59 smithi115 bash[46934]: cluster 2022-04-22T19:24:57.438093+0000 mgr.y (mgr.34107) 4090 : cluster [DBG] pgmap v5509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:24:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:24:59 smithi115 bash[46934]: audit 2022-04-22T19:24:58.664266+0000 mon.a (mon.0) 6893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:01 smithi121 bash[36902]: cluster 2022-04-22T19:24:59.439188+0000 mgr.y (mgr.34107) 4091 : cluster [DBG] pgmap v5510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:01 smithi121 bash[36902]: audit 2022-04-22T19:25:00.229982+0000 mon.a (mon.0) 6894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:01 smithi115 bash[45495]: cluster 2022-04-22T19:24:59.439188+0000 mgr.y (mgr.34107) 4091 : cluster [DBG] pgmap v5510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:01 smithi115 bash[45495]: audit 2022-04-22T19:25:00.229982+0000 mon.a (mon.0) 6894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:01 smithi115 bash[46934]: cluster 2022-04-22T19:24:59.439188+0000 mgr.y (mgr.34107) 4091 : cluster [DBG] pgmap v5510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:01 smithi115 bash[46934]: audit 2022-04-22T19:25:00.229982+0000 mon.a (mon.0) 6894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:02.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:02.069Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:03 smithi121 bash[36902]: cluster 2022-04-22T19:25:01.440334+0000 mgr.y (mgr.34107) 4092 : cluster [DBG] pgmap v5511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:03.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:03 smithi115 bash[46934]: cluster 2022-04-22T19:25:01.440334+0000 mgr.y (mgr.34107) 4092 : cluster [DBG] pgmap v5511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:03.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:03 smithi115 bash[45495]: cluster 2022-04-22T19:25:01.440334+0000 mgr.y (mgr.34107) 4092 : cluster [DBG] pgmap v5511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:04.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:04 smithi115 bash[46934]: audit 2022-04-22T19:25:03.668026+0000 mon.a (mon.0) 6895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:04.300 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:03.940Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:04.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:04 smithi115 bash[45495]: audit 2022-04-22T19:25:03.668026+0000 mon.a (mon.0) 6895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:04 smithi121 bash[36902]: audit 2022-04-22T19:25:03.668026+0000 mon.a (mon.0) 6895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:05 smithi121 bash[36902]: cluster 2022-04-22T19:25:03.441497+0000 mgr.y (mgr.34107) 4093 : cluster [DBG] pgmap v5512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:05 smithi115 bash[46934]: cluster 2022-04-22T19:25:03.441497+0000 mgr.y (mgr.34107) 4093 : cluster [DBG] pgmap v5512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:05 smithi115 bash[45495]: cluster 2022-04-22T19:25:03.441497+0000 mgr.y (mgr.34107) 4093 : cluster [DBG] pgmap v5512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:06 smithi121 bash[36902]: audit 2022-04-22T19:25:05.232920+0000 mon.a (mon.0) 6896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:06.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:06 smithi115 bash[45495]: audit 2022-04-22T19:25:05.232920+0000 mon.a (mon.0) 6896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:06.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:06 smithi115 bash[46934]: audit 2022-04-22T19:25:05.232920+0000 mon.a (mon.0) 6896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:07 smithi121 bash[36902]: cluster 2022-04-22T19:25:05.442684+0000 mgr.y (mgr.34107) 4094 : cluster [DBG] pgmap v5513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:07.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:07 smithi115 bash[45495]: cluster 2022-04-22T19:25:05.442684+0000 mgr.y (mgr.34107) 4094 : cluster [DBG] pgmap v5513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:07.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:07 smithi115 bash[46934]: cluster 2022-04-22T19:25:05.442684+0000 mgr.y (mgr.34107) 4094 : cluster [DBG] pgmap v5513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:09 smithi121 bash[36902]: cluster 2022-04-22T19:25:07.443811+0000 mgr.y (mgr.34107) 4095 : cluster [DBG] pgmap v5514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:09 smithi121 bash[36902]: audit 2022-04-22T19:25:08.671714+0000 mon.a (mon.0) 6897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:09 smithi115 bash[45495]: cluster 2022-04-22T19:25:07.443811+0000 mgr.y (mgr.34107) 4095 : cluster [DBG] pgmap v5514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:09 smithi115 bash[45495]: audit 2022-04-22T19:25:08.671714+0000 mon.a (mon.0) 6897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:09 smithi115 bash[46934]: cluster 2022-04-22T19:25:07.443811+0000 mgr.y (mgr.34107) 4095 : cluster [DBG] pgmap v5514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:09.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:09 smithi115 bash[46934]: audit 2022-04-22T19:25:08.671714+0000 mon.a (mon.0) 6897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:11 smithi121 bash[36902]: cluster 2022-04-22T19:25:09.444866+0000 mgr.y (mgr.34107) 4096 : cluster [DBG] pgmap v5515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:11 smithi121 bash[36902]: audit 2022-04-22T19:25:10.235934+0000 mon.a (mon.0) 6898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:11.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:11 smithi115 bash[45495]: cluster 2022-04-22T19:25:09.444866+0000 mgr.y (mgr.34107) 4096 : cluster [DBG] pgmap v5515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:11.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:11 smithi115 bash[45495]: audit 2022-04-22T19:25:10.235934+0000 mon.a (mon.0) 6898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:11.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:11 smithi115 bash[46934]: cluster 2022-04-22T19:25:09.444866+0000 mgr.y (mgr.34107) 4096 : cluster [DBG] pgmap v5515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:11.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:11 smithi115 bash[46934]: audit 2022-04-22T19:25:10.235934+0000 mon.a (mon.0) 6898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:12.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:12.069Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:13 smithi121 bash[36902]: cluster 2022-04-22T19:25:11.445908+0000 mgr.y (mgr.34107) 4097 : cluster [DBG] pgmap v5516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:13.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:13 smithi115 bash[45495]: cluster 2022-04-22T19:25:11.445908+0000 mgr.y (mgr.34107) 4097 : cluster [DBG] pgmap v5516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:13.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:13 smithi115 bash[46934]: cluster 2022-04-22T19:25:11.445908+0000 mgr.y (mgr.34107) 4097 : cluster [DBG] pgmap v5516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:14 smithi115 bash[45495]: audit 2022-04-22T19:25:13.675125+0000 mon.a (mon.0) 6899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:13.942Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:14.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:14 smithi115 bash[46934]: audit 2022-04-22T19:25:13.675125+0000 mon.a (mon.0) 6899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:14 smithi121 bash[36902]: audit 2022-04-22T19:25:13.675125+0000 mon.a (mon.0) 6899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:15 smithi121 bash[36902]: cluster 2022-04-22T19:25:13.447173+0000 mgr.y (mgr.34107) 4098 : cluster [DBG] pgmap v5517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:15 smithi121 bash[36902]: audit 2022-04-22T19:25:14.627797+0000 mon.a (mon.0) 6900 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:25:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:15 smithi121 bash[36902]: audit 2022-04-22T19:25:14.632069+0000 mon.a (mon.0) 6901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:25:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[46934]: cluster 2022-04-22T19:25:13.447173+0000 mgr.y (mgr.34107) 4098 : cluster [DBG] pgmap v5517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[46934]: audit 2022-04-22T19:25:14.627797+0000 mon.a (mon.0) 6900 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:25:15.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[46934]: audit 2022-04-22T19:25:14.632069+0000 mon.a (mon.0) 6901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:25:15.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[45495]: cluster 2022-04-22T19:25:13.447173+0000 mgr.y (mgr.34107) 4098 : cluster [DBG] pgmap v5517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[45495]: audit 2022-04-22T19:25:14.627797+0000 mon.a (mon.0) 6900 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:25:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:15 smithi115 bash[45495]: audit 2022-04-22T19:25:14.632069+0000 mon.a (mon.0) 6901 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:25:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:16 smithi121 bash[36902]: audit 2022-04-22T19:25:15.239236+0000 mon.a (mon.0) 6902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:16.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:16 smithi115 bash[46934]: audit 2022-04-22T19:25:15.239236+0000 mon.a (mon.0) 6902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:16.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:16 smithi115 bash[45495]: audit 2022-04-22T19:25:15.239236+0000 mon.a (mon.0) 6902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:17 smithi121 bash[36902]: cluster 2022-04-22T19:25:15.448253+0000 mgr.y (mgr.34107) 4099 : cluster [DBG] pgmap v5518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:17.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:17 smithi115 bash[45495]: cluster 2022-04-22T19:25:15.448253+0000 mgr.y (mgr.34107) 4099 : cluster [DBG] pgmap v5518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:17.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:17 smithi115 bash[46934]: cluster 2022-04-22T19:25:15.448253+0000 mgr.y (mgr.34107) 4099 : cluster [DBG] pgmap v5518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:19 smithi121 bash[36902]: cluster 2022-04-22T19:25:17.449369+0000 mgr.y (mgr.34107) 4100 : cluster [DBG] pgmap v5519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:19 smithi121 bash[36902]: audit 2022-04-22T19:25:18.678667+0000 mon.a (mon.0) 6903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:19.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:19 smithi115 bash[45495]: cluster 2022-04-22T19:25:17.449369+0000 mgr.y (mgr.34107) 4100 : cluster [DBG] pgmap v5519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:19.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:19 smithi115 bash[45495]: audit 2022-04-22T19:25:18.678667+0000 mon.a (mon.0) 6903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:19.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:19 smithi115 bash[46934]: cluster 2022-04-22T19:25:17.449369+0000 mgr.y (mgr.34107) 4100 : cluster [DBG] pgmap v5519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:19.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:19 smithi115 bash[46934]: audit 2022-04-22T19:25:18.678667+0000 mon.a (mon.0) 6903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:21 smithi121 bash[36902]: cluster 2022-04-22T19:25:19.450570+0000 mgr.y (mgr.34107) 4101 : cluster [DBG] pgmap v5520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:21 smithi121 bash[36902]: audit 2022-04-22T19:25:20.242290+0000 mon.a (mon.0) 6904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:21 smithi115 bash[46934]: cluster 2022-04-22T19:25:19.450570+0000 mgr.y (mgr.34107) 4101 : cluster [DBG] pgmap v5520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:21 smithi115 bash[46934]: audit 2022-04-22T19:25:20.242290+0000 mon.a (mon.0) 6904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:21.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:21 smithi115 bash[45495]: cluster 2022-04-22T19:25:19.450570+0000 mgr.y (mgr.34107) 4101 : cluster [DBG] pgmap v5520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:21.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:21 smithi115 bash[45495]: audit 2022-04-22T19:25:20.242290+0000 mon.a (mon.0) 6904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:22.070Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:23 smithi121 bash[36902]: cluster 2022-04-22T19:25:21.451729+0000 mgr.y (mgr.34107) 4102 : cluster [DBG] pgmap v5521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:23.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:23 smithi115 bash[45495]: cluster 2022-04-22T19:25:21.451729+0000 mgr.y (mgr.34107) 4102 : cluster [DBG] pgmap v5521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:23.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:23 smithi115 bash[46934]: cluster 2022-04-22T19:25:21.451729+0000 mgr.y (mgr.34107) 4102 : cluster [DBG] pgmap v5521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:24.222 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:23.943Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:24.224 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:24 smithi115 bash[45495]: audit 2022-04-22T19:25:23.682279+0000 mon.a (mon.0) 6905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:24 smithi121 bash[36902]: audit 2022-04-22T19:25:23.682279+0000 mon.a (mon.0) 6905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:24.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:24 smithi115 bash[46934]: audit 2022-04-22T19:25:23.682279+0000 mon.a (mon.0) 6905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:25 smithi121 bash[36902]: cluster 2022-04-22T19:25:23.453008+0000 mgr.y (mgr.34107) 4103 : cluster [DBG] pgmap v5522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:25.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:25 smithi115 bash[45495]: cluster 2022-04-22T19:25:23.453008+0000 mgr.y (mgr.34107) 4103 : cluster [DBG] pgmap v5522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:25.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:25 smithi115 bash[46934]: cluster 2022-04-22T19:25:23.453008+0000 mgr.y (mgr.34107) 4103 : cluster [DBG] pgmap v5522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:26 smithi121 bash[36902]: audit 2022-04-22T19:25:25.245281+0000 mon.a (mon.0) 6906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:26.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:26 smithi115 bash[46934]: audit 2022-04-22T19:25:25.245281+0000 mon.a (mon.0) 6906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:26.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:26 smithi115 bash[45495]: audit 2022-04-22T19:25:25.245281+0000 mon.a (mon.0) 6906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:26.780 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:25:27.120 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:25:27.120 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (108m) 4m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:25:27.120 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:25:27.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (110m) 4m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (110m) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (117m) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (118m) 4m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (117m) 4m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (116m) 5m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (116m) 4m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (109m) 4m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (109m) 5m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:25:27.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (115m) 4m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (114m) 4m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (114m) 4m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (113m) 4m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (112m) 5m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (112m) 5m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (111m) 5m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (110m) 5m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:25:27.122 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (109m) 5m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:25:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:27 smithi121 bash[36902]: cluster 2022-04-22T19:25:25.454099+0000 mgr.y (mgr.34107) 4104 : cluster [DBG] pgmap v5523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:27.517 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:25:27.517 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:25:27.518 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:25:27.519 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:25:27.520 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:25:27.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:27 smithi115 bash[45495]: cluster 2022-04-22T19:25:25.454099+0000 mgr.y (mgr.34107) 4104 : cluster [DBG] pgmap v5523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:27.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:27 smithi115 bash[46934]: cluster 2022-04-22T19:25:25.454099+0000 mgr.y (mgr.34107) 4104 : cluster [DBG] pgmap v5523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:28 smithi121 bash[36902]: audit 2022-04-22T19:25:26.751800+0000 mgr.y (mgr.34107) 4105 : audit [DBG] from='client.49624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:28 smithi121 bash[36902]: audit 2022-04-22T19:25:27.094360+0000 mgr.y (mgr.34107) 4106 : audit [DBG] from='client.49630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:28 smithi121 bash[36902]: audit 2022-04-22T19:25:27.505904+0000 mon.a (mon.0) 6907 : audit [DBG] from='client.? 172.21.15.115:0/3704210491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[45495]: audit 2022-04-22T19:25:26.751800+0000 mgr.y (mgr.34107) 4105 : audit [DBG] from='client.49624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[45495]: audit 2022-04-22T19:25:27.094360+0000 mgr.y (mgr.34107) 4106 : audit [DBG] from='client.49630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[45495]: audit 2022-04-22T19:25:27.505904+0000 mon.a (mon.0) 6907 : audit [DBG] from='client.? 172.21.15.115:0/3704210491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[46934]: audit 2022-04-22T19:25:26.751800+0000 mgr.y (mgr.34107) 4105 : audit [DBG] from='client.49624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[46934]: audit 2022-04-22T19:25:27.094360+0000 mgr.y (mgr.34107) 4106 : audit [DBG] from='client.49630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:28 smithi115 bash[46934]: audit 2022-04-22T19:25:27.505904+0000 mon.a (mon.0) 6907 : audit [DBG] from='client.? 172.21.15.115:0/3704210491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:29 smithi121 bash[36902]: cluster 2022-04-22T19:25:27.454789+0000 mgr.y (mgr.34107) 4107 : cluster [DBG] pgmap v5524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:29 smithi121 bash[36902]: audit 2022-04-22T19:25:28.685894+0000 mon.a (mon.0) 6908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:29 smithi115 bash[45495]: cluster 2022-04-22T19:25:27.454789+0000 mgr.y (mgr.34107) 4107 : cluster [DBG] pgmap v5524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:29.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:29 smithi115 bash[45495]: audit 2022-04-22T19:25:28.685894+0000 mon.a (mon.0) 6908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:29 smithi115 bash[46934]: cluster 2022-04-22T19:25:27.454789+0000 mgr.y (mgr.34107) 4107 : cluster [DBG] pgmap v5524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:29 smithi115 bash[46934]: audit 2022-04-22T19:25:28.685894+0000 mon.a (mon.0) 6908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:31 smithi121 bash[36902]: cluster 2022-04-22T19:25:29.455854+0000 mgr.y (mgr.34107) 4108 : cluster [DBG] pgmap v5525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:31 smithi121 bash[36902]: audit 2022-04-22T19:25:30.248107+0000 mon.a (mon.0) 6909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:31.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:31 smithi115 bash[45495]: cluster 2022-04-22T19:25:29.455854+0000 mgr.y (mgr.34107) 4108 : cluster [DBG] pgmap v5525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:31.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:31 smithi115 bash[45495]: audit 2022-04-22T19:25:30.248107+0000 mon.a (mon.0) 6909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:31.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:31 smithi115 bash[46934]: cluster 2022-04-22T19:25:29.455854+0000 mgr.y (mgr.34107) 4108 : cluster [DBG] pgmap v5525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:31.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:31 smithi115 bash[46934]: audit 2022-04-22T19:25:30.248107+0000 mon.a (mon.0) 6909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:32.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:32.070Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:33 smithi121 bash[36902]: cluster 2022-04-22T19:25:31.456952+0000 mgr.y (mgr.34107) 4109 : cluster [DBG] pgmap v5526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:33.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:33 smithi115 bash[45495]: cluster 2022-04-22T19:25:31.456952+0000 mgr.y (mgr.34107) 4109 : cluster [DBG] pgmap v5526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:33 smithi115 bash[46934]: cluster 2022-04-22T19:25:31.456952+0000 mgr.y (mgr.34107) 4109 : cluster [DBG] pgmap v5526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:34.261 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:33.944Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:34.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:34 smithi115 bash[46934]: audit 2022-04-22T19:25:33.689772+0000 mon.a (mon.0) 6910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:34.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:34 smithi115 bash[45495]: audit 2022-04-22T19:25:33.689772+0000 mon.a (mon.0) 6910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:34 smithi121 bash[36902]: audit 2022-04-22T19:25:33.689772+0000 mon.a (mon.0) 6910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:35 smithi121 bash[36902]: cluster 2022-04-22T19:25:33.458058+0000 mgr.y (mgr.34107) 4110 : cluster [DBG] pgmap v5527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:35 smithi115 bash[46934]: cluster 2022-04-22T19:25:33.458058+0000 mgr.y (mgr.34107) 4110 : cluster [DBG] pgmap v5527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:35.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:35 smithi115 bash[45495]: cluster 2022-04-22T19:25:33.458058+0000 mgr.y (mgr.34107) 4110 : cluster [DBG] pgmap v5527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:36.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:36 smithi115 bash[45495]: audit 2022-04-22T19:25:35.250713+0000 mon.a (mon.0) 6911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:36 smithi115 bash[46934]: audit 2022-04-22T19:25:35.250713+0000 mon.a (mon.0) 6911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:36 smithi121 bash[36902]: audit 2022-04-22T19:25:35.250713+0000 mon.a (mon.0) 6911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:37.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:37 smithi115 bash[45495]: cluster 2022-04-22T19:25:35.459137+0000 mgr.y (mgr.34107) 4111 : cluster [DBG] pgmap v5528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:37.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:37 smithi115 bash[46934]: cluster 2022-04-22T19:25:35.459137+0000 mgr.y (mgr.34107) 4111 : cluster [DBG] pgmap v5528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:37 smithi121 bash[36902]: cluster 2022-04-22T19:25:35.459137+0000 mgr.y (mgr.34107) 4111 : cluster [DBG] pgmap v5528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:39 smithi115 bash[45495]: cluster 2022-04-22T19:25:37.460358+0000 mgr.y (mgr.34107) 4112 : cluster [DBG] pgmap v5529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:39 smithi115 bash[45495]: audit 2022-04-22T19:25:38.693335+0000 mon.a (mon.0) 6912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:39 smithi115 bash[46934]: cluster 2022-04-22T19:25:37.460358+0000 mgr.y (mgr.34107) 4112 : cluster [DBG] pgmap v5529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:39 smithi115 bash[46934]: audit 2022-04-22T19:25:38.693335+0000 mon.a (mon.0) 6912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:39 smithi121 bash[36902]: cluster 2022-04-22T19:25:37.460358+0000 mgr.y (mgr.34107) 4112 : cluster [DBG] pgmap v5529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:39 smithi121 bash[36902]: audit 2022-04-22T19:25:38.693335+0000 mon.a (mon.0) 6912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:40.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:40 smithi115 bash[45495]: audit 2022-04-22T19:25:40.254653+0000 mon.a (mon.0) 6913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:40.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:40 smithi115 bash[46934]: audit 2022-04-22T19:25:40.254653+0000 mon.a (mon.0) 6913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:40 smithi121 bash[36902]: audit 2022-04-22T19:25:40.254653+0000 mon.a (mon.0) 6913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:41 smithi115 bash[46934]: cluster 2022-04-22T19:25:39.461381+0000 mgr.y (mgr.34107) 4113 : cluster [DBG] pgmap v5530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:41 smithi115 bash[45495]: cluster 2022-04-22T19:25:39.461381+0000 mgr.y (mgr.34107) 4113 : cluster [DBG] pgmap v5530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:41 smithi121 bash[36902]: cluster 2022-04-22T19:25:39.461381+0000 mgr.y (mgr.34107) 4113 : cluster [DBG] pgmap v5530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:42.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:43 smithi121 bash[36902]: cluster 2022-04-22T19:25:41.462406+0000 mgr.y (mgr.34107) 4114 : cluster [DBG] pgmap v5531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:43.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:43 smithi115 bash[46934]: cluster 2022-04-22T19:25:41.462406+0000 mgr.y (mgr.34107) 4114 : cluster [DBG] pgmap v5531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:43.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:43 smithi115 bash[45495]: cluster 2022-04-22T19:25:41.462406+0000 mgr.y (mgr.34107) 4114 : cluster [DBG] pgmap v5531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:44.301 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:43.944Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:44.455 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:44 smithi121 bash[36902]: audit 2022-04-22T19:25:43.696933+0000 mon.a (mon.0) 6914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:44.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:44 smithi121 bash[36902]: audit 2022-04-22T19:25:44.082430+0000 mon.a (mon.0) 6915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:25:44.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:44 smithi121 bash[36902]: audit 2022-04-22T19:25:44.085422+0000 mon.a (mon.0) 6916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:25:44.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:44 smithi121 bash[36902]: audit 2022-04-22T19:25:44.087246+0000 mon.a (mon.0) 6917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:25:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[45495]: audit 2022-04-22T19:25:43.696933+0000 mon.a (mon.0) 6914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[45495]: audit 2022-04-22T19:25:44.082430+0000 mon.a (mon.0) 6915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:25:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[45495]: audit 2022-04-22T19:25:44.085422+0000 mon.a (mon.0) 6916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:25:44.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[45495]: audit 2022-04-22T19:25:44.087246+0000 mon.a (mon.0) 6917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:25:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[46934]: audit 2022-04-22T19:25:43.696933+0000 mon.a (mon.0) 6914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[46934]: audit 2022-04-22T19:25:44.082430+0000 mon.a (mon.0) 6915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:25:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[46934]: audit 2022-04-22T19:25:44.085422+0000 mon.a (mon.0) 6916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:25:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:44 smithi115 bash[46934]: audit 2022-04-22T19:25:44.087246+0000 mon.a (mon.0) 6917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:25:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:45 smithi121 bash[36902]: cluster 2022-04-22T19:25:43.463535+0000 mgr.y (mgr.34107) 4115 : cluster [DBG] pgmap v5532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:45 smithi121 bash[36902]: audit 2022-04-22T19:25:44.459031+0000 mon.a (mon.0) 6918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:25:45.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:45 smithi121 bash[36902]: audit 2022-04-22T19:25:45.257547+0000 mon.a (mon.0) 6919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[45495]: cluster 2022-04-22T19:25:43.463535+0000 mgr.y (mgr.34107) 4115 : cluster [DBG] pgmap v5532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[45495]: audit 2022-04-22T19:25:44.459031+0000 mon.a (mon.0) 6918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:25:45.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[45495]: audit 2022-04-22T19:25:45.257547+0000 mon.a (mon.0) 6919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:45.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[46934]: cluster 2022-04-22T19:25:43.463535+0000 mgr.y (mgr.34107) 4115 : cluster [DBG] pgmap v5532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:45.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[46934]: audit 2022-04-22T19:25:44.459031+0000 mon.a (mon.0) 6918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:25:45.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:45 smithi115 bash[46934]: audit 2022-04-22T19:25:45.257547+0000 mon.a (mon.0) 6919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:47 smithi121 bash[36902]: cluster 2022-04-22T19:25:45.464570+0000 mgr.y (mgr.34107) 4116 : cluster [DBG] pgmap v5533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:47.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:47 smithi115 bash[45495]: cluster 2022-04-22T19:25:45.464570+0000 mgr.y (mgr.34107) 4116 : cluster [DBG] pgmap v5533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:47.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:47 smithi115 bash[46934]: cluster 2022-04-22T19:25:45.464570+0000 mgr.y (mgr.34107) 4116 : cluster [DBG] pgmap v5533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:49 smithi121 bash[36902]: cluster 2022-04-22T19:25:47.465721+0000 mgr.y (mgr.34107) 4117 : cluster [DBG] pgmap v5534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:49 smithi121 bash[36902]: audit 2022-04-22T19:25:48.700670+0000 mon.a (mon.0) 6920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:49 smithi115 bash[45495]: cluster 2022-04-22T19:25:47.465721+0000 mgr.y (mgr.34107) 4117 : cluster [DBG] pgmap v5534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:49.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:49 smithi115 bash[45495]: audit 2022-04-22T19:25:48.700670+0000 mon.a (mon.0) 6920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:49.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:49 smithi115 bash[46934]: cluster 2022-04-22T19:25:47.465721+0000 mgr.y (mgr.34107) 4117 : cluster [DBG] pgmap v5534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:49.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:49 smithi115 bash[46934]: audit 2022-04-22T19:25:48.700670+0000 mon.a (mon.0) 6920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:50 smithi121 bash[36902]: audit 2022-04-22T19:25:50.260619+0000 mon.a (mon.0) 6921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:50.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:50 smithi115 bash[45495]: audit 2022-04-22T19:25:50.260619+0000 mon.a (mon.0) 6921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:50.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:50 smithi115 bash[46934]: audit 2022-04-22T19:25:50.260619+0000 mon.a (mon.0) 6921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:51 smithi121 bash[36902]: cluster 2022-04-22T19:25:49.466792+0000 mgr.y (mgr.34107) 4118 : cluster [DBG] pgmap v5535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:51.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:51 smithi115 bash[45495]: cluster 2022-04-22T19:25:49.466792+0000 mgr.y (mgr.34107) 4118 : cluster [DBG] pgmap v5535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:51.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:51 smithi115 bash[46934]: cluster 2022-04-22T19:25:49.466792+0000 mgr.y (mgr.34107) 4118 : cluster [DBG] pgmap v5535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:52.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:53 smithi121 bash[36902]: cluster 2022-04-22T19:25:51.467901+0000 mgr.y (mgr.34107) 4119 : cluster [DBG] pgmap v5536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:53.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:53 smithi115 bash[45495]: cluster 2022-04-22T19:25:51.467901+0000 mgr.y (mgr.34107) 4119 : cluster [DBG] pgmap v5536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:53.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:53 smithi115 bash[46934]: cluster 2022-04-22T19:25:51.467901+0000 mgr.y (mgr.34107) 4119 : cluster [DBG] pgmap v5536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:54.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:25:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:25:53.945Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:25:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:54 smithi121 bash[36902]: audit 2022-04-22T19:25:53.704389+0000 mon.a (mon.0) 6922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:54.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:54 smithi115 bash[45495]: audit 2022-04-22T19:25:53.704389+0000 mon.a (mon.0) 6922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:54.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:54 smithi115 bash[46934]: audit 2022-04-22T19:25:53.704389+0000 mon.a (mon.0) 6922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:55 smithi121 bash[36902]: cluster 2022-04-22T19:25:53.469016+0000 mgr.y (mgr.34107) 4120 : cluster [DBG] pgmap v5537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:55.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:55 smithi121 bash[36902]: audit 2022-04-22T19:25:55.263832+0000 mon.a (mon.0) 6923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:55 smithi115 bash[45495]: cluster 2022-04-22T19:25:53.469016+0000 mgr.y (mgr.34107) 4120 : cluster [DBG] pgmap v5537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:55 smithi115 bash[45495]: audit 2022-04-22T19:25:55.263832+0000 mon.a (mon.0) 6923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:55.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:55 smithi115 bash[46934]: cluster 2022-04-22T19:25:53.469016+0000 mgr.y (mgr.34107) 4120 : cluster [DBG] pgmap v5537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:55.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:55 smithi115 bash[46934]: audit 2022-04-22T19:25:55.263832+0000 mon.a (mon.0) 6923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:25:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:57 smithi121 bash[36902]: cluster 2022-04-22T19:25:55.470070+0000 mgr.y (mgr.34107) 4121 : cluster [DBG] pgmap v5538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:57.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:57 smithi115 bash[45495]: cluster 2022-04-22T19:25:55.470070+0000 mgr.y (mgr.34107) 4121 : cluster [DBG] pgmap v5538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:57 smithi115 bash[46934]: cluster 2022-04-22T19:25:55.470070+0000 mgr.y (mgr.34107) 4121 : cluster [DBG] pgmap v5538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:57.891 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (109m) 5m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (111m) 5m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (110m) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:25:58.232 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (118m) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:25:58.233 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (118m) 5m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:25:58.233 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (117m) 5m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:25:58.233 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (116m) 6m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (117m) 5m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (110m) 5m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (110m) 6m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (115m) 5m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (115m) 5m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:25:58.234 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (114m) 5m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (114m) 5m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (113m) 6m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (112m) 6m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (112m) 6m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (111m) 6m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:25:58.235 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (109m) 6m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:25:58.639 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:25:58.639 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:25:58.639 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:58.640 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:25:58.641 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:25:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:59 smithi121 bash[36902]: cluster 2022-04-22T19:25:57.471287+0000 mgr.y (mgr.34107) 4122 : cluster [DBG] pgmap v5539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:59 smithi121 bash[36902]: audit 2022-04-22T19:25:57.863461+0000 mgr.y (mgr.34107) 4123 : audit [DBG] from='client.39708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:59 smithi121 bash[36902]: audit 2022-04-22T19:25:58.200539+0000 mgr.y (mgr.34107) 4124 : audit [DBG] from='client.39714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:59 smithi121 bash[36902]: audit 2022-04-22T19:25:58.627549+0000 mon.a (mon.0) 6924 : audit [DBG] from='client.? 172.21.15.115:0/3948847120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:25:59 smithi121 bash[36902]: audit 2022-04-22T19:25:58.708278+0000 mon.a (mon.0) 6925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[45495]: cluster 2022-04-22T19:25:57.471287+0000 mgr.y (mgr.34107) 4122 : cluster [DBG] pgmap v5539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[45495]: audit 2022-04-22T19:25:57.863461+0000 mgr.y (mgr.34107) 4123 : audit [DBG] from='client.39708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[45495]: audit 2022-04-22T19:25:58.200539+0000 mgr.y (mgr.34107) 4124 : audit [DBG] from='client.39714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[45495]: audit 2022-04-22T19:25:58.627549+0000 mon.a (mon.0) 6924 : audit [DBG] from='client.? 172.21.15.115:0/3948847120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:59.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[45495]: audit 2022-04-22T19:25:58.708278+0000 mon.a (mon.0) 6925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:25:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[46934]: cluster 2022-04-22T19:25:57.471287+0000 mgr.y (mgr.34107) 4122 : cluster [DBG] pgmap v5539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:25:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[46934]: audit 2022-04-22T19:25:57.863461+0000 mgr.y (mgr.34107) 4123 : audit [DBG] from='client.39708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[46934]: audit 2022-04-22T19:25:58.200539+0000 mgr.y (mgr.34107) 4124 : audit [DBG] from='client.39714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:25:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[46934]: audit 2022-04-22T19:25:58.627549+0000 mon.a (mon.0) 6924 : audit [DBG] from='client.? 172.21.15.115:0/3948847120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:25:59.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:25:59 smithi115 bash[46934]: audit 2022-04-22T19:25:58.708278+0000 mon.a (mon.0) 6925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:00 smithi121 bash[36902]: audit 2022-04-22T19:26:00.266940+0000 mon.a (mon.0) 6926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:00 smithi115 bash[45495]: audit 2022-04-22T19:26:00.266940+0000 mon.a (mon.0) 6926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:00 smithi115 bash[46934]: audit 2022-04-22T19:26:00.266940+0000 mon.a (mon.0) 6926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:01 smithi121 bash[36902]: cluster 2022-04-22T19:25:59.472378+0000 mgr.y (mgr.34107) 4125 : cluster [DBG] pgmap v5540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:01.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:01 smithi115 bash[45495]: cluster 2022-04-22T19:25:59.472378+0000 mgr.y (mgr.34107) 4125 : cluster [DBG] pgmap v5540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:01.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:01 smithi115 bash[46934]: cluster 2022-04-22T19:25:59.472378+0000 mgr.y (mgr.34107) 4125 : cluster [DBG] pgmap v5540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:02.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:02.072Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:03 smithi121 bash[36902]: cluster 2022-04-22T19:26:01.473462+0000 mgr.y (mgr.34107) 4126 : cluster [DBG] pgmap v5541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:03.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:03 smithi115 bash[45495]: cluster 2022-04-22T19:26:01.473462+0000 mgr.y (mgr.34107) 4126 : cluster [DBG] pgmap v5541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:03.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:03 smithi115 bash[46934]: cluster 2022-04-22T19:26:01.473462+0000 mgr.y (mgr.34107) 4126 : cluster [DBG] pgmap v5541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:03.945Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:04 smithi121 bash[36902]: audit 2022-04-22T19:26:03.711769+0000 mon.a (mon.0) 6927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:04.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:04 smithi115 bash[45495]: audit 2022-04-22T19:26:03.711769+0000 mon.a (mon.0) 6927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:04.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:04 smithi115 bash[46934]: audit 2022-04-22T19:26:03.711769+0000 mon.a (mon.0) 6927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:05 smithi121 bash[36902]: cluster 2022-04-22T19:26:03.474546+0000 mgr.y (mgr.34107) 4127 : cluster [DBG] pgmap v5542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:05 smithi121 bash[36902]: audit 2022-04-22T19:26:05.269981+0000 mon.a (mon.0) 6928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:05.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:05 smithi115 bash[45495]: cluster 2022-04-22T19:26:03.474546+0000 mgr.y (mgr.34107) 4127 : cluster [DBG] pgmap v5542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:05.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:05 smithi115 bash[45495]: audit 2022-04-22T19:26:05.269981+0000 mon.a (mon.0) 6928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:05.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:05 smithi115 bash[46934]: cluster 2022-04-22T19:26:03.474546+0000 mgr.y (mgr.34107) 4127 : cluster [DBG] pgmap v5542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:05.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:05 smithi115 bash[46934]: audit 2022-04-22T19:26:05.269981+0000 mon.a (mon.0) 6928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:07 smithi121 bash[36902]: cluster 2022-04-22T19:26:05.475592+0000 mgr.y (mgr.34107) 4128 : cluster [DBG] pgmap v5543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:07 smithi115 bash[45495]: cluster 2022-04-22T19:26:05.475592+0000 mgr.y (mgr.34107) 4128 : cluster [DBG] pgmap v5543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:07.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:07 smithi115 bash[46934]: cluster 2022-04-22T19:26:05.475592+0000 mgr.y (mgr.34107) 4128 : cluster [DBG] pgmap v5543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:09 smithi121 bash[36902]: cluster 2022-04-22T19:26:07.476583+0000 mgr.y (mgr.34107) 4129 : cluster [DBG] pgmap v5544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:09 smithi121 bash[36902]: audit 2022-04-22T19:26:08.715445+0000 mon.a (mon.0) 6929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:09 smithi115 bash[45495]: cluster 2022-04-22T19:26:07.476583+0000 mgr.y (mgr.34107) 4129 : cluster [DBG] pgmap v5544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:09 smithi115 bash[45495]: audit 2022-04-22T19:26:08.715445+0000 mon.a (mon.0) 6929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:09 smithi115 bash[46934]: cluster 2022-04-22T19:26:07.476583+0000 mgr.y (mgr.34107) 4129 : cluster [DBG] pgmap v5544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:09 smithi115 bash[46934]: audit 2022-04-22T19:26:08.715445+0000 mon.a (mon.0) 6929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:10 smithi121 bash[36902]: audit 2022-04-22T19:26:10.273532+0000 mon.a (mon.0) 6930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:10.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:10 smithi115 bash[45495]: audit 2022-04-22T19:26:10.273532+0000 mon.a (mon.0) 6930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:10.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:10 smithi115 bash[46934]: audit 2022-04-22T19:26:10.273532+0000 mon.a (mon.0) 6930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:11 smithi121 bash[36902]: cluster 2022-04-22T19:26:09.477657+0000 mgr.y (mgr.34107) 4130 : cluster [DBG] pgmap v5545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:11 smithi115 bash[45495]: cluster 2022-04-22T19:26:09.477657+0000 mgr.y (mgr.34107) 4130 : cluster [DBG] pgmap v5545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:11 smithi115 bash[46934]: cluster 2022-04-22T19:26:09.477657+0000 mgr.y (mgr.34107) 4130 : cluster [DBG] pgmap v5545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:12.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:12.072Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:13 smithi121 bash[36902]: cluster 2022-04-22T19:26:11.478755+0000 mgr.y (mgr.34107) 4131 : cluster [DBG] pgmap v5546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:13.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:13 smithi115 bash[45495]: cluster 2022-04-22T19:26:11.478755+0000 mgr.y (mgr.34107) 4131 : cluster [DBG] pgmap v5546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:13.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:13 smithi115 bash[46934]: cluster 2022-04-22T19:26:11.478755+0000 mgr.y (mgr.34107) 4131 : cluster [DBG] pgmap v5546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:13.946Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:14 smithi121 bash[36902]: audit 2022-04-22T19:26:13.719176+0000 mon.a (mon.0) 6931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:14.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:14 smithi115 bash[46934]: audit 2022-04-22T19:26:13.719176+0000 mon.a (mon.0) 6931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:14.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:14 smithi115 bash[45495]: audit 2022-04-22T19:26:13.719176+0000 mon.a (mon.0) 6931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:15 smithi121 bash[36902]: cluster 2022-04-22T19:26:13.479863+0000 mgr.y (mgr.34107) 4132 : cluster [DBG] pgmap v5547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:15 smithi121 bash[36902]: audit 2022-04-22T19:26:14.642891+0000 mon.a (mon.0) 6932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:26:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:15 smithi121 bash[36902]: audit 2022-04-22T19:26:14.643330+0000 mon.a (mon.0) 6933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:26:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:15 smithi121 bash[36902]: audit 2022-04-22T19:26:15.276658+0000 mon.a (mon.0) 6934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[45495]: cluster 2022-04-22T19:26:13.479863+0000 mgr.y (mgr.34107) 4132 : cluster [DBG] pgmap v5547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[45495]: audit 2022-04-22T19:26:14.642891+0000 mon.a (mon.0) 6932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:26:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[45495]: audit 2022-04-22T19:26:14.643330+0000 mon.a (mon.0) 6933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:26:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[45495]: audit 2022-04-22T19:26:15.276658+0000 mon.a (mon.0) 6934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[46934]: cluster 2022-04-22T19:26:13.479863+0000 mgr.y (mgr.34107) 4132 : cluster [DBG] pgmap v5547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[46934]: audit 2022-04-22T19:26:14.642891+0000 mon.a (mon.0) 6932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:26:15.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[46934]: audit 2022-04-22T19:26:14.643330+0000 mon.a (mon.0) 6933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:26:15.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:15 smithi115 bash[46934]: audit 2022-04-22T19:26:15.276658+0000 mon.a (mon.0) 6934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:17 smithi121 bash[36902]: cluster 2022-04-22T19:26:15.480998+0000 mgr.y (mgr.34107) 4133 : cluster [DBG] pgmap v5548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:17 smithi115 bash[46934]: cluster 2022-04-22T19:26:15.480998+0000 mgr.y (mgr.34107) 4133 : cluster [DBG] pgmap v5548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:17.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:17 smithi115 bash[45495]: cluster 2022-04-22T19:26:15.480998+0000 mgr.y (mgr.34107) 4133 : cluster [DBG] pgmap v5548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:19 smithi121 bash[36902]: cluster 2022-04-22T19:26:17.482065+0000 mgr.y (mgr.34107) 4134 : cluster [DBG] pgmap v5549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:19 smithi121 bash[36902]: audit 2022-04-22T19:26:18.723093+0000 mon.a (mon.0) 6935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:19.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:19 smithi115 bash[45495]: cluster 2022-04-22T19:26:17.482065+0000 mgr.y (mgr.34107) 4134 : cluster [DBG] pgmap v5549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:19.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:19 smithi115 bash[45495]: audit 2022-04-22T19:26:18.723093+0000 mon.a (mon.0) 6935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:19 smithi115 bash[46934]: cluster 2022-04-22T19:26:17.482065+0000 mgr.y (mgr.34107) 4134 : cluster [DBG] pgmap v5549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:19 smithi115 bash[46934]: audit 2022-04-22T19:26:18.723093+0000 mon.a (mon.0) 6935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:20 smithi121 bash[36902]: audit 2022-04-22T19:26:20.279831+0000 mon.a (mon.0) 6936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:20.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:20 smithi115 bash[45495]: audit 2022-04-22T19:26:20.279831+0000 mon.a (mon.0) 6936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:20.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:20 smithi115 bash[46934]: audit 2022-04-22T19:26:20.279831+0000 mon.a (mon.0) 6936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:21 smithi121 bash[36902]: cluster 2022-04-22T19:26:19.483143+0000 mgr.y (mgr.34107) 4135 : cluster [DBG] pgmap v5550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:21 smithi115 bash[45495]: cluster 2022-04-22T19:26:19.483143+0000 mgr.y (mgr.34107) 4135 : cluster [DBG] pgmap v5550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:21.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:21 smithi115 bash[46934]: cluster 2022-04-22T19:26:19.483143+0000 mgr.y (mgr.34107) 4135 : cluster [DBG] pgmap v5550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:22.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:23 smithi121 bash[36902]: cluster 2022-04-22T19:26:21.484256+0000 mgr.y (mgr.34107) 4136 : cluster [DBG] pgmap v5551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:23.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:23 smithi115 bash[45495]: cluster 2022-04-22T19:26:21.484256+0000 mgr.y (mgr.34107) 4136 : cluster [DBG] pgmap v5551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:23.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:23 smithi115 bash[46934]: cluster 2022-04-22T19:26:21.484256+0000 mgr.y (mgr.34107) 4136 : cluster [DBG] pgmap v5551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:24.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:23.947Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:24 smithi121 bash[36902]: audit 2022-04-22T19:26:23.727098+0000 mon.a (mon.0) 6937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:24.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:24 smithi115 bash[45495]: audit 2022-04-22T19:26:23.727098+0000 mon.a (mon.0) 6937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:24.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:24 smithi115 bash[46934]: audit 2022-04-22T19:26:23.727098+0000 mon.a (mon.0) 6937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:25 smithi121 bash[36902]: cluster 2022-04-22T19:26:23.485417+0000 mgr.y (mgr.34107) 4137 : cluster [DBG] pgmap v5552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:25 smithi121 bash[36902]: audit 2022-04-22T19:26:25.283148+0000 mon.a (mon.0) 6938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:25.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:25 smithi115 bash[45495]: cluster 2022-04-22T19:26:23.485417+0000 mgr.y (mgr.34107) 4137 : cluster [DBG] pgmap v5552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:25.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:25 smithi115 bash[45495]: audit 2022-04-22T19:26:25.283148+0000 mon.a (mon.0) 6938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:25 smithi115 bash[46934]: cluster 2022-04-22T19:26:23.485417+0000 mgr.y (mgr.34107) 4137 : cluster [DBG] pgmap v5552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:25.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:25 smithi115 bash[46934]: audit 2022-04-22T19:26:25.283148+0000 mon.a (mon.0) 6938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:27 smithi121 bash[36902]: cluster 2022-04-22T19:26:25.486551+0000 mgr.y (mgr.34107) 4138 : cluster [DBG] pgmap v5553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:27.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:27 smithi115 bash[45495]: cluster 2022-04-22T19:26:25.486551+0000 mgr.y (mgr.34107) 4138 : cluster [DBG] pgmap v5553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:27 smithi115 bash[46934]: cluster 2022-04-22T19:26:25.486551+0000 mgr.y (mgr.34107) 4138 : cluster [DBG] pgmap v5553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:29.019 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:26:29.371 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:26:29.371 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (109m) 5m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:26:29.371 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (111m) 5m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (111m) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (118m) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (119m) 5m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (118m) 5m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (117m) 6m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (117m) 5m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (110m) 5m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (110m) 6m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:26:29.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (116m) 5m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (115m) 5m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (115m) 5m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (114m) 5m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (114m) 6m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (113m) 6m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (112m) 6m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (111m) 6m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:26:29.373 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (110m) 6m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:26:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:29 smithi121 bash[36902]: cluster 2022-04-22T19:26:27.487703+0000 mgr.y (mgr.34107) 4139 : cluster [DBG] pgmap v5554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:29 smithi121 bash[36902]: audit 2022-04-22T19:26:28.730778+0000 mon.a (mon.0) 6939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:29.780 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:26:29.780 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:26:29.780 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:26:29.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:26:29.780 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:26:29.781 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:26:29.782 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:26:29.782 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:26:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:29 smithi115 bash[45495]: cluster 2022-04-22T19:26:27.487703+0000 mgr.y (mgr.34107) 4139 : cluster [DBG] pgmap v5554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:29.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:29 smithi115 bash[45495]: audit 2022-04-22T19:26:28.730778+0000 mon.a (mon.0) 6939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:29 smithi115 bash[46934]: cluster 2022-04-22T19:26:27.487703+0000 mgr.y (mgr.34107) 4139 : cluster [DBG] pgmap v5554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:29 smithi115 bash[46934]: audit 2022-04-22T19:26:28.730778+0000 mon.a (mon.0) 6939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:30 smithi121 bash[36902]: audit 2022-04-22T19:26:28.990627+0000 mgr.y (mgr.34107) 4140 : audit [DBG] from='client.49660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:30 smithi121 bash[36902]: audit 2022-04-22T19:26:29.339453+0000 mgr.y (mgr.34107) 4141 : audit [DBG] from='client.39732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:30 smithi121 bash[36902]: audit 2022-04-22T19:26:29.768643+0000 mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.115:0/3203373238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:26:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:30 smithi121 bash[36902]: audit 2022-04-22T19:26:30.286342+0000 mon.a (mon.0) 6940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:30.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[45495]: audit 2022-04-22T19:26:28.990627+0000 mgr.y (mgr.34107) 4140 : audit [DBG] from='client.49660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[45495]: audit 2022-04-22T19:26:29.339453+0000 mgr.y (mgr.34107) 4141 : audit [DBG] from='client.39732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[45495]: audit 2022-04-22T19:26:29.768643+0000 mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.115:0/3203373238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[45495]: audit 2022-04-22T19:26:30.286342+0000 mon.a (mon.0) 6940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[46934]: audit 2022-04-22T19:26:28.990627+0000 mgr.y (mgr.34107) 4140 : audit [DBG] from='client.49660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[46934]: audit 2022-04-22T19:26:29.339453+0000 mgr.y (mgr.34107) 4141 : audit [DBG] from='client.39732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:26:30.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[46934]: audit 2022-04-22T19:26:29.768643+0000 mon.c (mon.1) 190 : audit [DBG] from='client.? 172.21.15.115:0/3203373238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:26:30.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:30 smithi115 bash[46934]: audit 2022-04-22T19:26:30.286342+0000 mon.a (mon.0) 6940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:31 smithi121 bash[36902]: cluster 2022-04-22T19:26:29.488429+0000 mgr.y (mgr.34107) 4142 : cluster [DBG] pgmap v5555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:31 smithi115 bash[45495]: cluster 2022-04-22T19:26:29.488429+0000 mgr.y (mgr.34107) 4142 : cluster [DBG] pgmap v5555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:31.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:31 smithi115 bash[46934]: cluster 2022-04-22T19:26:29.488429+0000 mgr.y (mgr.34107) 4142 : cluster [DBG] pgmap v5555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:32.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:33 smithi121 bash[36902]: cluster 2022-04-22T19:26:31.489513+0000 mgr.y (mgr.34107) 4143 : cluster [DBG] pgmap v5556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:33.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:33 smithi115 bash[45495]: cluster 2022-04-22T19:26:31.489513+0000 mgr.y (mgr.34107) 4143 : cluster [DBG] pgmap v5556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:33.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:33 smithi115 bash[46934]: cluster 2022-04-22T19:26:31.489513+0000 mgr.y (mgr.34107) 4143 : cluster [DBG] pgmap v5556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:33.947Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:34 smithi121 bash[36902]: audit 2022-04-22T19:26:33.734356+0000 mon.a (mon.0) 6941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:34.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:34 smithi115 bash[45495]: audit 2022-04-22T19:26:33.734356+0000 mon.a (mon.0) 6941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:34.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:34 smithi115 bash[46934]: audit 2022-04-22T19:26:33.734356+0000 mon.a (mon.0) 6941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:35 smithi121 bash[36902]: cluster 2022-04-22T19:26:33.490636+0000 mgr.y (mgr.34107) 4144 : cluster [DBG] pgmap v5557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:35 smithi121 bash[36902]: audit 2022-04-22T19:26:35.289553+0000 mon.a (mon.0) 6942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:35 smithi115 bash[45495]: cluster 2022-04-22T19:26:33.490636+0000 mgr.y (mgr.34107) 4144 : cluster [DBG] pgmap v5557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:35 smithi115 bash[45495]: audit 2022-04-22T19:26:35.289553+0000 mon.a (mon.0) 6942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:35 smithi115 bash[46934]: cluster 2022-04-22T19:26:33.490636+0000 mgr.y (mgr.34107) 4144 : cluster [DBG] pgmap v5557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:35 smithi115 bash[46934]: audit 2022-04-22T19:26:35.289553+0000 mon.a (mon.0) 6942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:36 smithi121 bash[36902]: cluster 2022-04-22T19:26:35.491718+0000 mgr.y (mgr.34107) 4145 : cluster [DBG] pgmap v5558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:36.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:36 smithi115 bash[45495]: cluster 2022-04-22T19:26:35.491718+0000 mgr.y (mgr.34107) 4145 : cluster [DBG] pgmap v5558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:36.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:36 smithi115 bash[46934]: cluster 2022-04-22T19:26:35.491718+0000 mgr.y (mgr.34107) 4145 : cluster [DBG] pgmap v5558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:38 smithi121 bash[36902]: cluster 2022-04-22T19:26:37.492708+0000 mgr.y (mgr.34107) 4146 : cluster [DBG] pgmap v5559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:38 smithi121 bash[36902]: audit 2022-04-22T19:26:38.738029+0000 mon.a (mon.0) 6943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:38 smithi115 bash[45495]: cluster 2022-04-22T19:26:37.492708+0000 mgr.y (mgr.34107) 4146 : cluster [DBG] pgmap v5559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:38 smithi115 bash[45495]: audit 2022-04-22T19:26:38.738029+0000 mon.a (mon.0) 6943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:39.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:38 smithi115 bash[46934]: cluster 2022-04-22T19:26:37.492708+0000 mgr.y (mgr.34107) 4146 : cluster [DBG] pgmap v5559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:39.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:38 smithi115 bash[46934]: audit 2022-04-22T19:26:38.738029+0000 mon.a (mon.0) 6943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:40 smithi121 bash[36902]: cluster 2022-04-22T19:26:39.493692+0000 mgr.y (mgr.34107) 4147 : cluster [DBG] pgmap v5560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:40 smithi121 bash[36902]: audit 2022-04-22T19:26:40.292685+0000 mon.a (mon.0) 6944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:40 smithi115 bash[45495]: cluster 2022-04-22T19:26:39.493692+0000 mgr.y (mgr.34107) 4147 : cluster [DBG] pgmap v5560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:40 smithi115 bash[45495]: audit 2022-04-22T19:26:40.292685+0000 mon.a (mon.0) 6944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:40 smithi115 bash[46934]: cluster 2022-04-22T19:26:39.493692+0000 mgr.y (mgr.34107) 4147 : cluster [DBG] pgmap v5560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:40 smithi115 bash[46934]: audit 2022-04-22T19:26:40.292685+0000 mon.a (mon.0) 6944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:42.075Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:42 smithi121 bash[36902]: cluster 2022-04-22T19:26:41.494736+0000 mgr.y (mgr.34107) 4148 : cluster [DBG] pgmap v5561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:43.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:42 smithi115 bash[46934]: cluster 2022-04-22T19:26:41.494736+0000 mgr.y (mgr.34107) 4148 : cluster [DBG] pgmap v5561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:43.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:42 smithi115 bash[45495]: cluster 2022-04-22T19:26:41.494736+0000 mgr.y (mgr.34107) 4148 : cluster [DBG] pgmap v5561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:43 smithi121 bash[36902]: audit 2022-04-22T19:26:43.741724+0000 mon.a (mon.0) 6945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:43.947Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:44.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:43 smithi115 bash[45495]: audit 2022-04-22T19:26:43.741724+0000 mon.a (mon.0) 6945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:44.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:43 smithi115 bash[46934]: audit 2022-04-22T19:26:43.741724+0000 mon.a (mon.0) 6945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:44 smithi121 bash[36902]: cluster 2022-04-22T19:26:43.495835+0000 mgr.y (mgr.34107) 4149 : cluster [DBG] pgmap v5562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:44 smithi121 bash[36902]: audit 2022-04-22T19:26:44.468893+0000 mon.a (mon.0) 6946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:26:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:44 smithi121 bash[36902]: audit 2022-04-22T19:26:44.471834+0000 mon.a (mon.0) 6947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:26:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:44 smithi121 bash[36902]: audit 2022-04-22T19:26:44.473568+0000 mon.a (mon.0) 6948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:26:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[45495]: cluster 2022-04-22T19:26:43.495835+0000 mgr.y (mgr.34107) 4149 : cluster [DBG] pgmap v5562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[45495]: audit 2022-04-22T19:26:44.468893+0000 mon.a (mon.0) 6946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:26:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[45495]: audit 2022-04-22T19:26:44.471834+0000 mon.a (mon.0) 6947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:26:45.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[45495]: audit 2022-04-22T19:26:44.473568+0000 mon.a (mon.0) 6948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:26:45.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[46934]: cluster 2022-04-22T19:26:43.495835+0000 mgr.y (mgr.34107) 4149 : cluster [DBG] pgmap v5562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:45.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[46934]: audit 2022-04-22T19:26:44.468893+0000 mon.a (mon.0) 6946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:26:45.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[46934]: audit 2022-04-22T19:26:44.471834+0000 mon.a (mon.0) 6947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:26:45.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:44 smithi115 bash[46934]: audit 2022-04-22T19:26:44.473568+0000 mon.a (mon.0) 6948 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:26:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:45 smithi121 bash[36902]: audit 2022-04-22T19:26:44.841815+0000 mon.a (mon.0) 6949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:26:46.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:45 smithi121 bash[36902]: audit 2022-04-22T19:26:45.295966+0000 mon.a (mon.0) 6950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:45 smithi115 bash[45495]: audit 2022-04-22T19:26:44.841815+0000 mon.a (mon.0) 6949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:26:46.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:45 smithi115 bash[45495]: audit 2022-04-22T19:26:45.295966+0000 mon.a (mon.0) 6950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:45 smithi115 bash[46934]: audit 2022-04-22T19:26:44.841815+0000 mon.a (mon.0) 6949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:26:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:45 smithi115 bash[46934]: audit 2022-04-22T19:26:45.295966+0000 mon.a (mon.0) 6950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:46 smithi121 bash[36902]: cluster 2022-04-22T19:26:45.496899+0000 mgr.y (mgr.34107) 4150 : cluster [DBG] pgmap v5563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:47.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:46 smithi115 bash[45495]: cluster 2022-04-22T19:26:45.496899+0000 mgr.y (mgr.34107) 4150 : cluster [DBG] pgmap v5563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:47.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:46 smithi115 bash[46934]: cluster 2022-04-22T19:26:45.496899+0000 mgr.y (mgr.34107) 4150 : cluster [DBG] pgmap v5563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:48 smithi121 bash[36902]: cluster 2022-04-22T19:26:47.498005+0000 mgr.y (mgr.34107) 4151 : cluster [DBG] pgmap v5564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:48 smithi121 bash[36902]: audit 2022-04-22T19:26:48.745321+0000 mon.a (mon.0) 6951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:49.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:48 smithi115 bash[45495]: cluster 2022-04-22T19:26:47.498005+0000 mgr.y (mgr.34107) 4151 : cluster [DBG] pgmap v5564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:49.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:48 smithi115 bash[45495]: audit 2022-04-22T19:26:48.745321+0000 mon.a (mon.0) 6951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:48 smithi115 bash[46934]: cluster 2022-04-22T19:26:47.498005+0000 mgr.y (mgr.34107) 4151 : cluster [DBG] pgmap v5564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:48 smithi115 bash[46934]: audit 2022-04-22T19:26:48.745321+0000 mon.a (mon.0) 6951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:50 smithi121 bash[36902]: cluster 2022-04-22T19:26:49.499079+0000 mgr.y (mgr.34107) 4152 : cluster [DBG] pgmap v5565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:50 smithi121 bash[36902]: audit 2022-04-22T19:26:50.299086+0000 mon.a (mon.0) 6952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:50 smithi115 bash[45495]: cluster 2022-04-22T19:26:49.499079+0000 mgr.y (mgr.34107) 4152 : cluster [DBG] pgmap v5565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:50 smithi115 bash[45495]: audit 2022-04-22T19:26:50.299086+0000 mon.a (mon.0) 6952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:50 smithi115 bash[46934]: cluster 2022-04-22T19:26:49.499079+0000 mgr.y (mgr.34107) 4152 : cluster [DBG] pgmap v5565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:50 smithi115 bash[46934]: audit 2022-04-22T19:26:50.299086+0000 mon.a (mon.0) 6952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:52.075Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:52 smithi121 bash[36902]: cluster 2022-04-22T19:26:51.500122+0000 mgr.y (mgr.34107) 4153 : cluster [DBG] pgmap v5566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:52 smithi115 bash[45495]: cluster 2022-04-22T19:26:51.500122+0000 mgr.y (mgr.34107) 4153 : cluster [DBG] pgmap v5566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:52 smithi115 bash[46934]: cluster 2022-04-22T19:26:51.500122+0000 mgr.y (mgr.34107) 4153 : cluster [DBG] pgmap v5566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:53 smithi121 bash[36902]: audit 2022-04-22T19:26:53.749294+0000 mon.a (mon.0) 6953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:26:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:26:53.947Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:26:54.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:53 smithi115 bash[45495]: audit 2022-04-22T19:26:53.749294+0000 mon.a (mon.0) 6953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:54.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:53 smithi115 bash[46934]: audit 2022-04-22T19:26:53.749294+0000 mon.a (mon.0) 6953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:54 smithi121 bash[36902]: cluster 2022-04-22T19:26:53.501255+0000 mgr.y (mgr.34107) 4154 : cluster [DBG] pgmap v5567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:54 smithi115 bash[45495]: cluster 2022-04-22T19:26:53.501255+0000 mgr.y (mgr.34107) 4154 : cluster [DBG] pgmap v5567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:55.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:54 smithi115 bash[46934]: cluster 2022-04-22T19:26:53.501255+0000 mgr.y (mgr.34107) 4154 : cluster [DBG] pgmap v5567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:55 smithi121 bash[36902]: audit 2022-04-22T19:26:55.302231+0000 mon.a (mon.0) 6954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:55 smithi115 bash[45495]: audit 2022-04-22T19:26:55.302231+0000 mon.a (mon.0) 6954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:56.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:55 smithi115 bash[46934]: audit 2022-04-22T19:26:55.302231+0000 mon.a (mon.0) 6954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:26:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:56 smithi121 bash[36902]: cluster 2022-04-22T19:26:55.502410+0000 mgr.y (mgr.34107) 4155 : cluster [DBG] pgmap v5568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:57.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:56 smithi115 bash[45495]: cluster 2022-04-22T19:26:55.502410+0000 mgr.y (mgr.34107) 4155 : cluster [DBG] pgmap v5568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:57.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:56 smithi115 bash[46934]: cluster 2022-04-22T19:26:55.502410+0000 mgr.y (mgr.34107) 4155 : cluster [DBG] pgmap v5568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:58 smithi121 bash[36902]: cluster 2022-04-22T19:26:57.503556+0000 mgr.y (mgr.34107) 4156 : cluster [DBG] pgmap v5569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:26:58 smithi121 bash[36902]: audit 2022-04-22T19:26:58.753089+0000 mon.a (mon.0) 6955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:59.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:58 smithi115 bash[45495]: cluster 2022-04-22T19:26:57.503556+0000 mgr.y (mgr.34107) 4156 : cluster [DBG] pgmap v5569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:59.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:26:58 smithi115 bash[45495]: audit 2022-04-22T19:26:58.753089+0000 mon.a (mon.0) 6955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:26:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:58 smithi115 bash[46934]: cluster 2022-04-22T19:26:57.503556+0000 mgr.y (mgr.34107) 4156 : cluster [DBG] pgmap v5569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:26:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:26:58 smithi115 bash[46934]: audit 2022-04-22T19:26:58.753089+0000 mon.a (mon.0) 6955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:00.164 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:27:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (110m) 6m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (112m) 6m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (111m) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (119m) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (119m) 6m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (118m) 6m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:27:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (118m) 7m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (118m) 6m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (111m) 6m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (111m) 7m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (116m) 6m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (116m) 6m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (115m) 6m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (115m) 6m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (114m) 7m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:27:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (113m) 7m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:27:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (113m) 7m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:27:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (112m) 7m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:27:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (110m) 7m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:27:00.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:00.914 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:27:00.915 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:27:00.915 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:27:00.915 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:00 smithi121 bash[36902]: cluster 2022-04-22T19:26:59.504656+0000 mgr.y (mgr.34107) 4157 : cluster [DBG] pgmap v5570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:00 smithi121 bash[36902]: audit 2022-04-22T19:27:00.136662+0000 mgr.y (mgr.34107) 4158 : audit [DBG] from='client.49678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:00 smithi121 bash[36902]: audit 2022-04-22T19:27:00.305196+0000 mon.a (mon.0) 6956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:00 smithi121 bash[36902]: audit 2022-04-22T19:27:00.901535+0000 mon.c (mon.1) 191 : audit [DBG] from='client.? 172.21.15.115:0/452698713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[45495]: cluster 2022-04-22T19:26:59.504656+0000 mgr.y (mgr.34107) 4157 : cluster [DBG] pgmap v5570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[45495]: audit 2022-04-22T19:27:00.136662+0000 mgr.y (mgr.34107) 4158 : audit [DBG] from='client.49678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[45495]: audit 2022-04-22T19:27:00.305196+0000 mon.a (mon.0) 6956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[45495]: audit 2022-04-22T19:27:00.901535+0000 mon.c (mon.1) 191 : audit [DBG] from='client.? 172.21.15.115:0/452698713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[46934]: cluster 2022-04-22T19:26:59.504656+0000 mgr.y (mgr.34107) 4157 : cluster [DBG] pgmap v5570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[46934]: audit 2022-04-22T19:27:00.136662+0000 mgr.y (mgr.34107) 4158 : audit [DBG] from='client.49678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[46934]: audit 2022-04-22T19:27:00.305196+0000 mon.a (mon.0) 6956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:01.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:00 smithi115 bash[46934]: audit 2022-04-22T19:27:00.901535+0000 mon.c (mon.1) 191 : audit [DBG] from='client.? 172.21.15.115:0/452698713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:01 smithi121 bash[36902]: audit 2022-04-22T19:27:00.479173+0000 mgr.y (mgr.34107) 4159 : audit [DBG] from='client.49684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:01 smithi115 bash[46934]: audit 2022-04-22T19:27:00.479173+0000 mgr.y (mgr.34107) 4159 : audit [DBG] from='client.49684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:02.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:02.076Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:02.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:01 smithi115 bash[45495]: audit 2022-04-22T19:27:00.479173+0000 mgr.y (mgr.34107) 4159 : audit [DBG] from='client.49684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:02 smithi121 bash[36902]: cluster 2022-04-22T19:27:01.505711+0000 mgr.y (mgr.34107) 4160 : cluster [DBG] pgmap v5571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:03.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:02 smithi115 bash[45495]: cluster 2022-04-22T19:27:01.505711+0000 mgr.y (mgr.34107) 4160 : cluster [DBG] pgmap v5571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:03.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:02 smithi115 bash[46934]: cluster 2022-04-22T19:27:01.505711+0000 mgr.y (mgr.34107) 4160 : cluster [DBG] pgmap v5571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:03 smithi121 bash[36902]: audit 2022-04-22T19:27:03.756309+0000 mon.a (mon.0) 6957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:03 smithi115 bash[45495]: audit 2022-04-22T19:27:03.756309+0000 mon.a (mon.0) 6957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:04.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:03 smithi115 bash[46934]: audit 2022-04-22T19:27:03.756309+0000 mon.a (mon.0) 6957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:03.948Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:04 smithi121 bash[36902]: cluster 2022-04-22T19:27:03.506823+0000 mgr.y (mgr.34107) 4161 : cluster [DBG] pgmap v5572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:04 smithi115 bash[45495]: cluster 2022-04-22T19:27:03.506823+0000 mgr.y (mgr.34107) 4161 : cluster [DBG] pgmap v5572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:05.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:04 smithi115 bash[46934]: cluster 2022-04-22T19:27:03.506823+0000 mgr.y (mgr.34107) 4161 : cluster [DBG] pgmap v5572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:05 smithi121 bash[36902]: audit 2022-04-22T19:27:05.308029+0000 mon.a (mon.0) 6958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:05 smithi115 bash[45495]: audit 2022-04-22T19:27:05.308029+0000 mon.a (mon.0) 6958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:05 smithi115 bash[46934]: audit 2022-04-22T19:27:05.308029+0000 mon.a (mon.0) 6958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:06 smithi121 bash[36902]: cluster 2022-04-22T19:27:05.507948+0000 mgr.y (mgr.34107) 4162 : cluster [DBG] pgmap v5573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:07.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:06 smithi115 bash[45495]: cluster 2022-04-22T19:27:05.507948+0000 mgr.y (mgr.34107) 4162 : cluster [DBG] pgmap v5573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:07.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:06 smithi115 bash[46934]: cluster 2022-04-22T19:27:05.507948+0000 mgr.y (mgr.34107) 4162 : cluster [DBG] pgmap v5573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:08 smithi121 bash[36902]: cluster 2022-04-22T19:27:07.508999+0000 mgr.y (mgr.34107) 4163 : cluster [DBG] pgmap v5574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:08 smithi121 bash[36902]: audit 2022-04-22T19:27:08.759798+0000 mon.a (mon.0) 6959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:08 smithi115 bash[45495]: cluster 2022-04-22T19:27:07.508999+0000 mgr.y (mgr.34107) 4163 : cluster [DBG] pgmap v5574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:08 smithi115 bash[45495]: audit 2022-04-22T19:27:08.759798+0000 mon.a (mon.0) 6959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:09.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:08 smithi115 bash[46934]: cluster 2022-04-22T19:27:07.508999+0000 mgr.y (mgr.34107) 4163 : cluster [DBG] pgmap v5574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:09.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:08 smithi115 bash[46934]: audit 2022-04-22T19:27:08.759798+0000 mon.a (mon.0) 6959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:10 smithi121 bash[36902]: cluster 2022-04-22T19:27:09.510193+0000 mgr.y (mgr.34107) 4164 : cluster [DBG] pgmap v5575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:10 smithi121 bash[36902]: audit 2022-04-22T19:27:10.311350+0000 mon.a (mon.0) 6960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:11.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:10 smithi115 bash[45495]: cluster 2022-04-22T19:27:09.510193+0000 mgr.y (mgr.34107) 4164 : cluster [DBG] pgmap v5575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:11.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:10 smithi115 bash[45495]: audit 2022-04-22T19:27:10.311350+0000 mon.a (mon.0) 6960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:11.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:10 smithi115 bash[46934]: cluster 2022-04-22T19:27:09.510193+0000 mgr.y (mgr.34107) 4164 : cluster [DBG] pgmap v5575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:11.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:10 smithi115 bash[46934]: audit 2022-04-22T19:27:10.311350+0000 mon.a (mon.0) 6960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:12.077Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:12 smithi121 bash[36902]: cluster 2022-04-22T19:27:11.511318+0000 mgr.y (mgr.34107) 4165 : cluster [DBG] pgmap v5576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:12 smithi115 bash[45495]: cluster 2022-04-22T19:27:11.511318+0000 mgr.y (mgr.34107) 4165 : cluster [DBG] pgmap v5576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:12 smithi115 bash[46934]: cluster 2022-04-22T19:27:11.511318+0000 mgr.y (mgr.34107) 4165 : cluster [DBG] pgmap v5576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:13 smithi121 bash[36902]: audit 2022-04-22T19:27:13.763578+0000 mon.a (mon.0) 6961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:13.949Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:13 smithi115 bash[45495]: audit 2022-04-22T19:27:13.763578+0000 mon.a (mon.0) 6961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:14.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:13 smithi115 bash[46934]: audit 2022-04-22T19:27:13.763578+0000 mon.a (mon.0) 6961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:14 smithi121 bash[36902]: cluster 2022-04-22T19:27:13.512458+0000 mgr.y (mgr.34107) 4166 : cluster [DBG] pgmap v5577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:14 smithi121 bash[36902]: audit 2022-04-22T19:27:14.654425+0000 mon.a (mon.0) 6962 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:27:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:14 smithi121 bash[36902]: audit 2022-04-22T19:27:14.654875+0000 mon.a (mon.0) 6963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:27:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[45495]: cluster 2022-04-22T19:27:13.512458+0000 mgr.y (mgr.34107) 4166 : cluster [DBG] pgmap v5577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:15.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[45495]: audit 2022-04-22T19:27:14.654425+0000 mon.a (mon.0) 6962 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:27:15.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[45495]: audit 2022-04-22T19:27:14.654875+0000 mon.a (mon.0) 6963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:27:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[46934]: cluster 2022-04-22T19:27:13.512458+0000 mgr.y (mgr.34107) 4166 : cluster [DBG] pgmap v5577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[46934]: audit 2022-04-22T19:27:14.654425+0000 mon.a (mon.0) 6962 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:27:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:14 smithi115 bash[46934]: audit 2022-04-22T19:27:14.654875+0000 mon.a (mon.0) 6963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:27:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:15 smithi121 bash[36902]: audit 2022-04-22T19:27:15.314487+0000 mon.a (mon.0) 6964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:16.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:15 smithi115 bash[45495]: audit 2022-04-22T19:27:15.314487+0000 mon.a (mon.0) 6964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:16.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:15 smithi115 bash[46934]: audit 2022-04-22T19:27:15.314487+0000 mon.a (mon.0) 6964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:16 smithi121 bash[36902]: cluster 2022-04-22T19:27:15.513477+0000 mgr.y (mgr.34107) 4167 : cluster [DBG] pgmap v5578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:17.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:16 smithi115 bash[46934]: cluster 2022-04-22T19:27:15.513477+0000 mgr.y (mgr.34107) 4167 : cluster [DBG] pgmap v5578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:17.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:16 smithi115 bash[45495]: cluster 2022-04-22T19:27:15.513477+0000 mgr.y (mgr.34107) 4167 : cluster [DBG] pgmap v5578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:18 smithi121 bash[36902]: cluster 2022-04-22T19:27:17.514644+0000 mgr.y (mgr.34107) 4168 : cluster [DBG] pgmap v5579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:18 smithi121 bash[36902]: audit 2022-04-22T19:27:18.767377+0000 mon.a (mon.0) 6965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:19.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:18 smithi115 bash[45495]: cluster 2022-04-22T19:27:17.514644+0000 mgr.y (mgr.34107) 4168 : cluster [DBG] pgmap v5579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:19.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:18 smithi115 bash[45495]: audit 2022-04-22T19:27:18.767377+0000 mon.a (mon.0) 6965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:19.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:18 smithi115 bash[46934]: cluster 2022-04-22T19:27:17.514644+0000 mgr.y (mgr.34107) 4168 : cluster [DBG] pgmap v5579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:19.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:18 smithi115 bash[46934]: audit 2022-04-22T19:27:18.767377+0000 mon.a (mon.0) 6965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:20 smithi121 bash[36902]: cluster 2022-04-22T19:27:19.515848+0000 mgr.y (mgr.34107) 4169 : cluster [DBG] pgmap v5580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:20 smithi121 bash[36902]: audit 2022-04-22T19:27:20.317728+0000 mon.a (mon.0) 6966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:20 smithi115 bash[46934]: cluster 2022-04-22T19:27:19.515848+0000 mgr.y (mgr.34107) 4169 : cluster [DBG] pgmap v5580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:20 smithi115 bash[46934]: audit 2022-04-22T19:27:20.317728+0000 mon.a (mon.0) 6966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:21.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:20 smithi115 bash[45495]: cluster 2022-04-22T19:27:19.515848+0000 mgr.y (mgr.34107) 4169 : cluster [DBG] pgmap v5580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:21.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:20 smithi115 bash[45495]: audit 2022-04-22T19:27:20.317728+0000 mon.a (mon.0) 6966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:22.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:22 smithi121 bash[36902]: cluster 2022-04-22T19:27:21.516931+0000 mgr.y (mgr.34107) 4170 : cluster [DBG] pgmap v5581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:22 smithi115 bash[46934]: cluster 2022-04-22T19:27:21.516931+0000 mgr.y (mgr.34107) 4170 : cluster [DBG] pgmap v5581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:23.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:22 smithi115 bash[45495]: cluster 2022-04-22T19:27:21.516931+0000 mgr.y (mgr.34107) 4170 : cluster [DBG] pgmap v5581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:24 smithi121 bash[36902]: audit 2022-04-22T19:27:23.770845+0000 mon.a (mon.0) 6967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:23 smithi115 bash[45495]: audit 2022-04-22T19:27:23.770845+0000 mon.a (mon.0) 6967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:24.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:23.950Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:24.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:23 smithi115 bash[46934]: audit 2022-04-22T19:27:23.770845+0000 mon.a (mon.0) 6967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:25 smithi115 bash[45495]: cluster 2022-04-22T19:27:23.518049+0000 mgr.y (mgr.34107) 4171 : cluster [DBG] pgmap v5582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:25.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:25 smithi115 bash[46934]: cluster 2022-04-22T19:27:23.518049+0000 mgr.y (mgr.34107) 4171 : cluster [DBG] pgmap v5582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:25 smithi121 bash[36902]: cluster 2022-04-22T19:27:23.518049+0000 mgr.y (mgr.34107) 4171 : cluster [DBG] pgmap v5582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:26 smithi115 bash[45495]: audit 2022-04-22T19:27:25.321019+0000 mon.a (mon.0) 6968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:26.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:26 smithi115 bash[46934]: audit 2022-04-22T19:27:25.321019+0000 mon.a (mon.0) 6968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:26 smithi121 bash[36902]: audit 2022-04-22T19:27:25.321019+0000 mon.a (mon.0) 6968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:27.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:27 smithi115 bash[45495]: cluster 2022-04-22T19:27:25.519039+0000 mgr.y (mgr.34107) 4172 : cluster [DBG] pgmap v5583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:27.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:27 smithi115 bash[46934]: cluster 2022-04-22T19:27:25.519039+0000 mgr.y (mgr.34107) 4172 : cluster [DBG] pgmap v5583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:27 smithi121 bash[36902]: cluster 2022-04-22T19:27:25.519039+0000 mgr.y (mgr.34107) 4172 : cluster [DBG] pgmap v5583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:29.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:29 smithi115 bash[45495]: cluster 2022-04-22T19:27:27.520139+0000 mgr.y (mgr.34107) 4173 : cluster [DBG] pgmap v5584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:29.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:29 smithi115 bash[45495]: audit 2022-04-22T19:27:28.774531+0000 mon.a (mon.0) 6969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:29 smithi115 bash[46934]: cluster 2022-04-22T19:27:27.520139+0000 mgr.y (mgr.34107) 4173 : cluster [DBG] pgmap v5584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:29 smithi115 bash[46934]: audit 2022-04-22T19:27:28.774531+0000 mon.a (mon.0) 6969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:29 smithi121 bash[36902]: cluster 2022-04-22T19:27:27.520139+0000 mgr.y (mgr.34107) 4173 : cluster [DBG] pgmap v5584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:29 smithi121 bash[36902]: audit 2022-04-22T19:27:28.774531+0000 mon.a (mon.0) 6969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:31.284 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:27:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:31 smithi115 bash[45495]: cluster 2022-04-22T19:27:29.521191+0000 mgr.y (mgr.34107) 4174 : cluster [DBG] pgmap v5585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:31 smithi115 bash[45495]: audit 2022-04-22T19:27:30.324065+0000 mon.a (mon.0) 6970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:31 smithi115 bash[46934]: cluster 2022-04-22T19:27:29.521191+0000 mgr.y (mgr.34107) 4174 : cluster [DBG] pgmap v5585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:31 smithi115 bash[46934]: audit 2022-04-22T19:27:30.324065+0000 mon.a (mon.0) 6970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:31 smithi121 bash[36902]: cluster 2022-04-22T19:27:29.521191+0000 mgr.y (mgr.34107) 4174 : cluster [DBG] pgmap v5585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:31 smithi121 bash[36902]: audit 2022-04-22T19:27:30.324065+0000 mon.a (mon.0) 6970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:31.624 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:27:31.624 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (110m) 6m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:27:31.624 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:27:31.624 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (112m) 6m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:27:31.624 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (112m) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (119m) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (119m) 6m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (118m) 7m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (118m) 6m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (111m) 6m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (111m) 7m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (117m) 6m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:27:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (116m) 6m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (116m) 6m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (115m) 6m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (115m) 7m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (114m) 7m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (113m) 7m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (112m) 7m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:27:31.626 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (111m) 7m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:27:32.021 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:27:32.022 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:27:32.023 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:27:32.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:32.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:32.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:32 smithi115 bash[45495]: audit 2022-04-22T19:27:31.256839+0000 mgr.y (mgr.34107) 4175 : audit [DBG] from='client.39762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:32.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:32 smithi115 bash[45495]: audit 2022-04-22T19:27:32.010244+0000 mon.a (mon.0) 6971 : audit [DBG] from='client.? 172.21.15.115:0/1779889037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:32 smithi115 bash[46934]: audit 2022-04-22T19:27:31.256839+0000 mgr.y (mgr.34107) 4175 : audit [DBG] from='client.39762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:32.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:32 smithi115 bash[46934]: audit 2022-04-22T19:27:32.010244+0000 mon.a (mon.0) 6971 : audit [DBG] from='client.? 172.21.15.115:0/1779889037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:32 smithi121 bash[36902]: audit 2022-04-22T19:27:31.256839+0000 mgr.y (mgr.34107) 4175 : audit [DBG] from='client.39762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:32 smithi121 bash[36902]: audit 2022-04-22T19:27:32.010244+0000 mon.a (mon.0) 6971 : audit [DBG] from='client.? 172.21.15.115:0/1779889037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:27:33.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:33 smithi115 bash[45495]: cluster 2022-04-22T19:27:31.522231+0000 mgr.y (mgr.34107) 4176 : cluster [DBG] pgmap v5586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:33.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:33 smithi115 bash[45495]: audit 2022-04-22T19:27:31.594393+0000 mgr.y (mgr.34107) 4177 : audit [DBG] from='client.49702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:33.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:33 smithi115 bash[46934]: cluster 2022-04-22T19:27:31.522231+0000 mgr.y (mgr.34107) 4176 : cluster [DBG] pgmap v5586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:33.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:33 smithi115 bash[46934]: audit 2022-04-22T19:27:31.594393+0000 mgr.y (mgr.34107) 4177 : audit [DBG] from='client.49702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:33 smithi121 bash[36902]: cluster 2022-04-22T19:27:31.522231+0000 mgr.y (mgr.34107) 4176 : cluster [DBG] pgmap v5586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:33 smithi121 bash[36902]: audit 2022-04-22T19:27:31.594393+0000 mgr.y (mgr.34107) 4177 : audit [DBG] from='client.49702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:27:34.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:34 smithi115 bash[45495]: audit 2022-04-22T19:27:33.778161+0000 mon.a (mon.0) 6972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:33.950Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:34 smithi115 bash[46934]: audit 2022-04-22T19:27:33.778161+0000 mon.a (mon.0) 6972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:34 smithi121 bash[36902]: audit 2022-04-22T19:27:33.778161+0000 mon.a (mon.0) 6972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:35 smithi121 bash[36902]: cluster 2022-04-22T19:27:33.523319+0000 mgr.y (mgr.34107) 4178 : cluster [DBG] pgmap v5587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:35 smithi115 bash[45495]: cluster 2022-04-22T19:27:33.523319+0000 mgr.y (mgr.34107) 4178 : cluster [DBG] pgmap v5587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:35 smithi115 bash[46934]: cluster 2022-04-22T19:27:33.523319+0000 mgr.y (mgr.34107) 4178 : cluster [DBG] pgmap v5587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:36 smithi121 bash[36902]: audit 2022-04-22T19:27:35.326973+0000 mon.a (mon.0) 6973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:36 smithi115 bash[45495]: audit 2022-04-22T19:27:35.326973+0000 mon.a (mon.0) 6973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:36 smithi115 bash[46934]: audit 2022-04-22T19:27:35.326973+0000 mon.a (mon.0) 6973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:37 smithi121 bash[36902]: cluster 2022-04-22T19:27:35.524367+0000 mgr.y (mgr.34107) 4179 : cluster [DBG] pgmap v5588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:37 smithi115 bash[45495]: cluster 2022-04-22T19:27:35.524367+0000 mgr.y (mgr.34107) 4179 : cluster [DBG] pgmap v5588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:37.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:37 smithi115 bash[46934]: cluster 2022-04-22T19:27:35.524367+0000 mgr.y (mgr.34107) 4179 : cluster [DBG] pgmap v5588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:39 smithi121 bash[36902]: cluster 2022-04-22T19:27:37.525474+0000 mgr.y (mgr.34107) 4180 : cluster [DBG] pgmap v5589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:39 smithi121 bash[36902]: audit 2022-04-22T19:27:38.781948+0000 mon.a (mon.0) 6974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:39 smithi115 bash[45495]: cluster 2022-04-22T19:27:37.525474+0000 mgr.y (mgr.34107) 4180 : cluster [DBG] pgmap v5589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:39 smithi115 bash[45495]: audit 2022-04-22T19:27:38.781948+0000 mon.a (mon.0) 6974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:39 smithi115 bash[46934]: cluster 2022-04-22T19:27:37.525474+0000 mgr.y (mgr.34107) 4180 : cluster [DBG] pgmap v5589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:39 smithi115 bash[46934]: audit 2022-04-22T19:27:38.781948+0000 mon.a (mon.0) 6974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:41 smithi121 bash[36902]: cluster 2022-04-22T19:27:39.526591+0000 mgr.y (mgr.34107) 4181 : cluster [DBG] pgmap v5590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:41 smithi121 bash[36902]: audit 2022-04-22T19:27:40.329729+0000 mon.a (mon.0) 6975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:41.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:41 smithi115 bash[45495]: cluster 2022-04-22T19:27:39.526591+0000 mgr.y (mgr.34107) 4181 : cluster [DBG] pgmap v5590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:41.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:41 smithi115 bash[45495]: audit 2022-04-22T19:27:40.329729+0000 mon.a (mon.0) 6975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:41 smithi115 bash[46934]: cluster 2022-04-22T19:27:39.526591+0000 mgr.y (mgr.34107) 4181 : cluster [DBG] pgmap v5590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:41 smithi115 bash[46934]: audit 2022-04-22T19:27:40.329729+0000 mon.a (mon.0) 6975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:42.076Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:43 smithi121 bash[36902]: cluster 2022-04-22T19:27:41.527692+0000 mgr.y (mgr.34107) 4182 : cluster [DBG] pgmap v5591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:43 smithi115 bash[45495]: cluster 2022-04-22T19:27:41.527692+0000 mgr.y (mgr.34107) 4182 : cluster [DBG] pgmap v5591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:43.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:43 smithi115 bash[46934]: cluster 2022-04-22T19:27:41.527692+0000 mgr.y (mgr.34107) 4182 : cluster [DBG] pgmap v5591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:43.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:44.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:44 smithi115 bash[45495]: audit 2022-04-22T19:27:43.785628+0000 mon.a (mon.0) 6976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:44.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:44 smithi115 bash[46934]: audit 2022-04-22T19:27:43.785628+0000 mon.a (mon.0) 6976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:44 smithi121 bash[36902]: audit 2022-04-22T19:27:43.785628+0000 mon.a (mon.0) 6976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[46934]: cluster 2022-04-22T19:27:43.528791+0000 mgr.y (mgr.34107) 4183 : cluster [DBG] pgmap v5592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[46934]: audit 2022-04-22T19:27:44.851025+0000 mon.a (mon.0) 6977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[46934]: audit 2022-04-22T19:27:44.853904+0000 mon.a (mon.0) 6978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[46934]: audit 2022-04-22T19:27:44.855537+0000 mon.a (mon.0) 6979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[45495]: cluster 2022-04-22T19:27:43.528791+0000 mgr.y (mgr.34107) 4183 : cluster [DBG] pgmap v5592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[45495]: audit 2022-04-22T19:27:44.851025+0000 mon.a (mon.0) 6977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:27:45.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[45495]: audit 2022-04-22T19:27:44.853904+0000 mon.a (mon.0) 6978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:27:45.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:45 smithi115 bash[45495]: audit 2022-04-22T19:27:44.855537+0000 mon.a (mon.0) 6979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:27:45.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:45 smithi121 bash[36902]: cluster 2022-04-22T19:27:43.528791+0000 mgr.y (mgr.34107) 4183 : cluster [DBG] pgmap v5592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:45.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:45 smithi121 bash[36902]: audit 2022-04-22T19:27:44.851025+0000 mon.a (mon.0) 6977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:27:45.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:45 smithi121 bash[36902]: audit 2022-04-22T19:27:44.853904+0000 mon.a (mon.0) 6978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:27:45.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:45 smithi121 bash[36902]: audit 2022-04-22T19:27:44.855537+0000 mon.a (mon.0) 6979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:27:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:46 smithi121 bash[36902]: audit 2022-04-22T19:27:45.214499+0000 mon.a (mon.0) 6980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:27:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:46 smithi121 bash[36902]: audit 2022-04-22T19:27:45.332961+0000 mon.a (mon.0) 6981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:46 smithi115 bash[45495]: audit 2022-04-22T19:27:45.214499+0000 mon.a (mon.0) 6980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:27:46.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:46 smithi115 bash[45495]: audit 2022-04-22T19:27:45.332961+0000 mon.a (mon.0) 6981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:46 smithi115 bash[46934]: audit 2022-04-22T19:27:45.214499+0000 mon.a (mon.0) 6980 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:27:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:46 smithi115 bash[46934]: audit 2022-04-22T19:27:45.332961+0000 mon.a (mon.0) 6981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:47 smithi121 bash[36902]: cluster 2022-04-22T19:27:45.529846+0000 mgr.y (mgr.34107) 4184 : cluster [DBG] pgmap v5593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:47 smithi115 bash[45495]: cluster 2022-04-22T19:27:45.529846+0000 mgr.y (mgr.34107) 4184 : cluster [DBG] pgmap v5593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:47.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:47 smithi115 bash[46934]: cluster 2022-04-22T19:27:45.529846+0000 mgr.y (mgr.34107) 4184 : cluster [DBG] pgmap v5593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:49 smithi121 bash[36902]: cluster 2022-04-22T19:27:47.530892+0000 mgr.y (mgr.34107) 4185 : cluster [DBG] pgmap v5594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:49 smithi121 bash[36902]: audit 2022-04-22T19:27:48.789295+0000 mon.a (mon.0) 6982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:49 smithi115 bash[45495]: cluster 2022-04-22T19:27:47.530892+0000 mgr.y (mgr.34107) 4185 : cluster [DBG] pgmap v5594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:49 smithi115 bash[45495]: audit 2022-04-22T19:27:48.789295+0000 mon.a (mon.0) 6982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:49 smithi115 bash[46934]: cluster 2022-04-22T19:27:47.530892+0000 mgr.y (mgr.34107) 4185 : cluster [DBG] pgmap v5594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:49 smithi115 bash[46934]: audit 2022-04-22T19:27:48.789295+0000 mon.a (mon.0) 6982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:51 smithi121 bash[36902]: cluster 2022-04-22T19:27:49.532063+0000 mgr.y (mgr.34107) 4186 : cluster [DBG] pgmap v5595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:51 smithi121 bash[36902]: audit 2022-04-22T19:27:50.336077+0000 mon.a (mon.0) 6983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:51 smithi115 bash[45495]: cluster 2022-04-22T19:27:49.532063+0000 mgr.y (mgr.34107) 4186 : cluster [DBG] pgmap v5595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:51 smithi115 bash[45495]: audit 2022-04-22T19:27:50.336077+0000 mon.a (mon.0) 6983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:51 smithi115 bash[46934]: cluster 2022-04-22T19:27:49.532063+0000 mgr.y (mgr.34107) 4186 : cluster [DBG] pgmap v5595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:51 smithi115 bash[46934]: audit 2022-04-22T19:27:50.336077+0000 mon.a (mon.0) 6983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:52.079Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:53 smithi121 bash[36902]: cluster 2022-04-22T19:27:51.533171+0000 mgr.y (mgr.34107) 4187 : cluster [DBG] pgmap v5596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:53 smithi115 bash[45495]: cluster 2022-04-22T19:27:51.533171+0000 mgr.y (mgr.34107) 4187 : cluster [DBG] pgmap v5596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:53.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:53 smithi115 bash[46934]: cluster 2022-04-22T19:27:51.533171+0000 mgr.y (mgr.34107) 4187 : cluster [DBG] pgmap v5596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:54.254 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:27:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:27:53.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:27:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:54 smithi121 bash[36902]: audit 2022-04-22T19:27:53.792938+0000 mon.a (mon.0) 6984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:54.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:54 smithi115 bash[45495]: audit 2022-04-22T19:27:53.792938+0000 mon.a (mon.0) 6984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:54.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:54 smithi115 bash[46934]: audit 2022-04-22T19:27:53.792938+0000 mon.a (mon.0) 6984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:55 smithi121 bash[36902]: cluster 2022-04-22T19:27:53.534330+0000 mgr.y (mgr.34107) 4188 : cluster [DBG] pgmap v5597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:55 smithi115 bash[45495]: cluster 2022-04-22T19:27:53.534330+0000 mgr.y (mgr.34107) 4188 : cluster [DBG] pgmap v5597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:55 smithi115 bash[46934]: cluster 2022-04-22T19:27:53.534330+0000 mgr.y (mgr.34107) 4188 : cluster [DBG] pgmap v5597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:56 smithi115 bash[45495]: audit 2022-04-22T19:27:55.338849+0000 mon.a (mon.0) 6985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:56 smithi115 bash[46934]: audit 2022-04-22T19:27:55.338849+0000 mon.a (mon.0) 6985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:56 smithi121 bash[36902]: audit 2022-04-22T19:27:55.338849+0000 mon.a (mon.0) 6985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:27:57.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:57 smithi115 bash[46934]: cluster 2022-04-22T19:27:55.535410+0000 mgr.y (mgr.34107) 4189 : cluster [DBG] pgmap v5598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:57.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:57 smithi115 bash[45495]: cluster 2022-04-22T19:27:55.535410+0000 mgr.y (mgr.34107) 4189 : cluster [DBG] pgmap v5598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:57 smithi121 bash[36902]: cluster 2022-04-22T19:27:55.535410+0000 mgr.y (mgr.34107) 4189 : cluster [DBG] pgmap v5598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:59 smithi115 bash[45495]: cluster 2022-04-22T19:27:57.536443+0000 mgr.y (mgr.34107) 4190 : cluster [DBG] pgmap v5599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:27:59 smithi115 bash[45495]: audit 2022-04-22T19:27:58.796753+0000 mon.a (mon.0) 6986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:59 smithi115 bash[46934]: cluster 2022-04-22T19:27:57.536443+0000 mgr.y (mgr.34107) 4190 : cluster [DBG] pgmap v5599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:27:59 smithi115 bash[46934]: audit 2022-04-22T19:27:58.796753+0000 mon.a (mon.0) 6986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:27:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:59 smithi121 bash[36902]: cluster 2022-04-22T19:27:57.536443+0000 mgr.y (mgr.34107) 4190 : cluster [DBG] pgmap v5599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:27:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:27:59 smithi121 bash[36902]: audit 2022-04-22T19:27:58.796753+0000 mon.a (mon.0) 6986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:01 smithi115 bash[45495]: cluster 2022-04-22T19:27:59.537499+0000 mgr.y (mgr.34107) 4191 : cluster [DBG] pgmap v5600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:01.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:01 smithi115 bash[45495]: audit 2022-04-22T19:28:00.341655+0000 mon.a (mon.0) 6987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:01 smithi115 bash[46934]: cluster 2022-04-22T19:27:59.537499+0000 mgr.y (mgr.34107) 4191 : cluster [DBG] pgmap v5600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:01 smithi115 bash[46934]: audit 2022-04-22T19:28:00.341655+0000 mon.a (mon.0) 6987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:01 smithi121 bash[36902]: cluster 2022-04-22T19:27:59.537499+0000 mgr.y (mgr.34107) 4191 : cluster [DBG] pgmap v5600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:01 smithi121 bash[36902]: audit 2022-04-22T19:28:00.341655+0000 mon.a (mon.0) 6987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:02.398 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:28:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:02.081Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:28:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (111m) 7m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:28:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:28:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (113m) 7m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:28:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (112m) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (119m) 7m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (119m) 8m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (119m) 7m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (112m) 7m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (112m) 8m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (117m) 7m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (117m) 7m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (116m) 7m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:28:02.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (116m) 7m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:28:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (115m) 8m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:28:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (114m) 8m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:28:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (114m) 8m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:28:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (113m) 8m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:28:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (111m) 8m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:28:03.140 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:28:03.140 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:28:03.140 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:28:03.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:03.140 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:28:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:28:03.142 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:28:03.142 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:28:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[45495]: cluster 2022-04-22T19:28:01.538661+0000 mgr.y (mgr.34107) 4192 : cluster [DBG] pgmap v5601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[45495]: audit 2022-04-22T19:28:02.371122+0000 mgr.y (mgr.34107) 4193 : audit [DBG] from='client.49714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:03.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[45495]: audit 2022-04-22T19:28:03.128719+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.115:0/2435590214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[46934]: cluster 2022-04-22T19:28:01.538661+0000 mgr.y (mgr.34107) 4192 : cluster [DBG] pgmap v5601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[46934]: audit 2022-04-22T19:28:02.371122+0000 mgr.y (mgr.34107) 4193 : audit [DBG] from='client.49714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[46934]: audit 2022-04-22T19:28:03.128719+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.115:0/2435590214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:03 smithi121 bash[36902]: cluster 2022-04-22T19:28:01.538661+0000 mgr.y (mgr.34107) 4192 : cluster [DBG] pgmap v5601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:03 smithi121 bash[36902]: audit 2022-04-22T19:28:02.371122+0000 mgr.y (mgr.34107) 4193 : audit [DBG] from='client.49714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:03 smithi121 bash[36902]: audit 2022-04-22T19:28:03.128719+0000 mon.c (mon.1) 192 : audit [DBG] from='client.? 172.21.15.115:0/2435590214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:04.287 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:03.952Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:04 smithi115 bash[46934]: audit 2022-04-22T19:28:02.712899+0000 mgr.y (mgr.34107) 4194 : audit [DBG] from='client.49720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:04.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:04 smithi115 bash[46934]: audit 2022-04-22T19:28:03.800599+0000 mon.a (mon.0) 6988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:04.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:04 smithi115 bash[45495]: audit 2022-04-22T19:28:02.712899+0000 mgr.y (mgr.34107) 4194 : audit [DBG] from='client.49720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:04.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:04 smithi115 bash[45495]: audit 2022-04-22T19:28:03.800599+0000 mon.a (mon.0) 6988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:04 smithi121 bash[36902]: audit 2022-04-22T19:28:02.712899+0000 mgr.y (mgr.34107) 4194 : audit [DBG] from='client.49720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:04 smithi121 bash[36902]: audit 2022-04-22T19:28:03.800599+0000 mon.a (mon.0) 6988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:05 smithi115 bash[45495]: cluster 2022-04-22T19:28:03.539891+0000 mgr.y (mgr.34107) 4195 : cluster [DBG] pgmap v5602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:05 smithi115 bash[46934]: cluster 2022-04-22T19:28:03.539891+0000 mgr.y (mgr.34107) 4195 : cluster [DBG] pgmap v5602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:05 smithi121 bash[36902]: cluster 2022-04-22T19:28:03.539891+0000 mgr.y (mgr.34107) 4195 : cluster [DBG] pgmap v5602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:06 smithi121 bash[36902]: audit 2022-04-22T19:28:05.344262+0000 mon.a (mon.0) 6989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:06 smithi115 bash[45495]: audit 2022-04-22T19:28:05.344262+0000 mon.a (mon.0) 6989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:06.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:06 smithi115 bash[46934]: audit 2022-04-22T19:28:05.344262+0000 mon.a (mon.0) 6989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:07 smithi121 bash[36902]: cluster 2022-04-22T19:28:05.540856+0000 mgr.y (mgr.34107) 4196 : cluster [DBG] pgmap v5603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:07 smithi115 bash[45495]: cluster 2022-04-22T19:28:05.540856+0000 mgr.y (mgr.34107) 4196 : cluster [DBG] pgmap v5603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:07.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:07 smithi115 bash[46934]: cluster 2022-04-22T19:28:05.540856+0000 mgr.y (mgr.34107) 4196 : cluster [DBG] pgmap v5603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:09 smithi121 bash[36902]: cluster 2022-04-22T19:28:07.541938+0000 mgr.y (mgr.34107) 4197 : cluster [DBG] pgmap v5604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:09 smithi121 bash[36902]: audit 2022-04-22T19:28:08.804460+0000 mon.a (mon.0) 6990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:09 smithi115 bash[46934]: cluster 2022-04-22T19:28:07.541938+0000 mgr.y (mgr.34107) 4197 : cluster [DBG] pgmap v5604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:09 smithi115 bash[46934]: audit 2022-04-22T19:28:08.804460+0000 mon.a (mon.0) 6990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:09 smithi115 bash[45495]: cluster 2022-04-22T19:28:07.541938+0000 mgr.y (mgr.34107) 4197 : cluster [DBG] pgmap v5604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:09.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:09 smithi115 bash[45495]: audit 2022-04-22T19:28:08.804460+0000 mon.a (mon.0) 6990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:11 smithi121 bash[36902]: cluster 2022-04-22T19:28:09.543039+0000 mgr.y (mgr.34107) 4198 : cluster [DBG] pgmap v5605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:11 smithi121 bash[36902]: audit 2022-04-22T19:28:10.347283+0000 mon.a (mon.0) 6991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:11 smithi115 bash[46934]: cluster 2022-04-22T19:28:09.543039+0000 mgr.y (mgr.34107) 4198 : cluster [DBG] pgmap v5605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:11 smithi115 bash[46934]: audit 2022-04-22T19:28:10.347283+0000 mon.a (mon.0) 6991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:11 smithi115 bash[45495]: cluster 2022-04-22T19:28:09.543039+0000 mgr.y (mgr.34107) 4198 : cluster [DBG] pgmap v5605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:11.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:11 smithi115 bash[45495]: audit 2022-04-22T19:28:10.347283+0000 mon.a (mon.0) 6991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:12.081Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:13 smithi121 bash[36902]: cluster 2022-04-22T19:28:11.544187+0000 mgr.y (mgr.34107) 4199 : cluster [DBG] pgmap v5606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:13 smithi115 bash[46934]: cluster 2022-04-22T19:28:11.544187+0000 mgr.y (mgr.34107) 4199 : cluster [DBG] pgmap v5606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:13.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:13 smithi115 bash[45495]: cluster 2022-04-22T19:28:11.544187+0000 mgr.y (mgr.34107) 4199 : cluster [DBG] pgmap v5606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:13.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:14 smithi121 bash[36902]: audit 2022-04-22T19:28:13.808152+0000 mon.a (mon.0) 6992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:14 smithi115 bash[45495]: audit 2022-04-22T19:28:13.808152+0000 mon.a (mon.0) 6992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:14.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:14 smithi115 bash[46934]: audit 2022-04-22T19:28:13.808152+0000 mon.a (mon.0) 6992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:15 smithi121 bash[36902]: cluster 2022-04-22T19:28:13.545298+0000 mgr.y (mgr.34107) 4200 : cluster [DBG] pgmap v5607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:15 smithi121 bash[36902]: audit 2022-04-22T19:28:14.662439+0000 mon.a (mon.0) 6993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:15 smithi121 bash[36902]: audit 2022-04-22T19:28:14.662931+0000 mon.a (mon.0) 6994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:28:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[45495]: cluster 2022-04-22T19:28:13.545298+0000 mgr.y (mgr.34107) 4200 : cluster [DBG] pgmap v5607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[45495]: audit 2022-04-22T19:28:14.662439+0000 mon.a (mon.0) 6993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:28:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[45495]: audit 2022-04-22T19:28:14.662931+0000 mon.a (mon.0) 6994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:28:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[46934]: cluster 2022-04-22T19:28:13.545298+0000 mgr.y (mgr.34107) 4200 : cluster [DBG] pgmap v5607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[46934]: audit 2022-04-22T19:28:14.662439+0000 mon.a (mon.0) 6993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:28:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:15 smithi115 bash[46934]: audit 2022-04-22T19:28:14.662931+0000 mon.a (mon.0) 6994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:28:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:16 smithi121 bash[36902]: audit 2022-04-22T19:28:15.350344+0000 mon.a (mon.0) 6995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:16 smithi115 bash[45495]: audit 2022-04-22T19:28:15.350344+0000 mon.a (mon.0) 6995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:16.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:16 smithi115 bash[46934]: audit 2022-04-22T19:28:15.350344+0000 mon.a (mon.0) 6995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:17 smithi121 bash[36902]: cluster 2022-04-22T19:28:15.546475+0000 mgr.y (mgr.34107) 4201 : cluster [DBG] pgmap v5608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:17 smithi115 bash[46934]: cluster 2022-04-22T19:28:15.546475+0000 mgr.y (mgr.34107) 4201 : cluster [DBG] pgmap v5608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:17.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:17 smithi115 bash[45495]: cluster 2022-04-22T19:28:15.546475+0000 mgr.y (mgr.34107) 4201 : cluster [DBG] pgmap v5608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:19 smithi121 bash[36902]: cluster 2022-04-22T19:28:17.547696+0000 mgr.y (mgr.34107) 4202 : cluster [DBG] pgmap v5609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:19 smithi121 bash[36902]: audit 2022-04-22T19:28:18.811819+0000 mon.a (mon.0) 6996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:19 smithi115 bash[45495]: cluster 2022-04-22T19:28:17.547696+0000 mgr.y (mgr.34107) 4202 : cluster [DBG] pgmap v5609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:19.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:19 smithi115 bash[45495]: audit 2022-04-22T19:28:18.811819+0000 mon.a (mon.0) 6996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:19 smithi115 bash[46934]: cluster 2022-04-22T19:28:17.547696+0000 mgr.y (mgr.34107) 4202 : cluster [DBG] pgmap v5609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:19.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:19 smithi115 bash[46934]: audit 2022-04-22T19:28:18.811819+0000 mon.a (mon.0) 6996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:21 smithi121 bash[36902]: cluster 2022-04-22T19:28:19.548751+0000 mgr.y (mgr.34107) 4203 : cluster [DBG] pgmap v5610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:21 smithi121 bash[36902]: audit 2022-04-22T19:28:20.353345+0000 mon.a (mon.0) 6997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:21 smithi115 bash[46934]: cluster 2022-04-22T19:28:19.548751+0000 mgr.y (mgr.34107) 4203 : cluster [DBG] pgmap v5610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:21 smithi115 bash[46934]: audit 2022-04-22T19:28:20.353345+0000 mon.a (mon.0) 6997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:21 smithi115 bash[45495]: cluster 2022-04-22T19:28:19.548751+0000 mgr.y (mgr.34107) 4203 : cluster [DBG] pgmap v5610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:21.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:21 smithi115 bash[45495]: audit 2022-04-22T19:28:20.353345+0000 mon.a (mon.0) 6997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:22.083Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:23 smithi121 bash[36902]: cluster 2022-04-22T19:28:21.549863+0000 mgr.y (mgr.34107) 4204 : cluster [DBG] pgmap v5611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:23 smithi115 bash[45495]: cluster 2022-04-22T19:28:21.549863+0000 mgr.y (mgr.34107) 4204 : cluster [DBG] pgmap v5611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:23.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:23 smithi115 bash[46934]: cluster 2022-04-22T19:28:21.549863+0000 mgr.y (mgr.34107) 4204 : cluster [DBG] pgmap v5611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:24.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:23.953Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:24 smithi121 bash[36902]: audit 2022-04-22T19:28:23.815526+0000 mon.a (mon.0) 6998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:24.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:24 smithi115 bash[45495]: audit 2022-04-22T19:28:23.815526+0000 mon.a (mon.0) 6998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:24.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:24 smithi115 bash[46934]: audit 2022-04-22T19:28:23.815526+0000 mon.a (mon.0) 6998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:25 smithi121 bash[36902]: cluster 2022-04-22T19:28:23.551090+0000 mgr.y (mgr.34107) 4205 : cluster [DBG] pgmap v5612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:25 smithi115 bash[46934]: cluster 2022-04-22T19:28:23.551090+0000 mgr.y (mgr.34107) 4205 : cluster [DBG] pgmap v5612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:25.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:25 smithi115 bash[45495]: cluster 2022-04-22T19:28:23.551090+0000 mgr.y (mgr.34107) 4205 : cluster [DBG] pgmap v5612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:26 smithi121 bash[36902]: audit 2022-04-22T19:28:25.355731+0000 mon.a (mon.0) 6999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:26 smithi115 bash[46934]: audit 2022-04-22T19:28:25.355731+0000 mon.a (mon.0) 6999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:26.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:26 smithi115 bash[45495]: audit 2022-04-22T19:28:25.355731+0000 mon.a (mon.0) 6999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:27 smithi121 bash[36902]: cluster 2022-04-22T19:28:25.552221+0000 mgr.y (mgr.34107) 4206 : cluster [DBG] pgmap v5613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:27 smithi115 bash[45495]: cluster 2022-04-22T19:28:25.552221+0000 mgr.y (mgr.34107) 4206 : cluster [DBG] pgmap v5613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:27.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:27 smithi115 bash[46934]: cluster 2022-04-22T19:28:25.552221+0000 mgr.y (mgr.34107) 4206 : cluster [DBG] pgmap v5613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:29 smithi121 bash[36902]: cluster 2022-04-22T19:28:27.553355+0000 mgr.y (mgr.34107) 4207 : cluster [DBG] pgmap v5614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:29 smithi121 bash[36902]: audit 2022-04-22T19:28:28.819128+0000 mon.a (mon.0) 7000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:29 smithi115 bash[45495]: cluster 2022-04-22T19:28:27.553355+0000 mgr.y (mgr.34107) 4207 : cluster [DBG] pgmap v5614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:29 smithi115 bash[45495]: audit 2022-04-22T19:28:28.819128+0000 mon.a (mon.0) 7000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:29 smithi115 bash[46934]: cluster 2022-04-22T19:28:27.553355+0000 mgr.y (mgr.34107) 4207 : cluster [DBG] pgmap v5614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:29.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:29 smithi115 bash[46934]: audit 2022-04-22T19:28:28.819128+0000 mon.a (mon.0) 7000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:30 smithi121 bash[36902]: audit 2022-04-22T19:28:30.358389+0000 mon.a (mon.0) 7001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:30.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:30 smithi115 bash[45495]: audit 2022-04-22T19:28:30.358389+0000 mon.a (mon.0) 7001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:30 smithi115 bash[46934]: audit 2022-04-22T19:28:30.358389+0000 mon.a (mon.0) 7001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:31 smithi121 bash[36902]: cluster 2022-04-22T19:28:29.554544+0000 mgr.y (mgr.34107) 4208 : cluster [DBG] pgmap v5615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:31 smithi115 bash[45495]: cluster 2022-04-22T19:28:29.554544+0000 mgr.y (mgr.34107) 4208 : cluster [DBG] pgmap v5615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:31.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:31 smithi115 bash[46934]: cluster 2022-04-22T19:28:29.554544+0000 mgr.y (mgr.34107) 4208 : cluster [DBG] pgmap v5615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:32.082Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:33.520 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:28:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:33 smithi121 bash[36902]: cluster 2022-04-22T19:28:31.555746+0000 mgr.y (mgr.34107) 4209 : cluster [DBG] pgmap v5616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:33.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:33 smithi115 bash[46934]: cluster 2022-04-22T19:28:31.555746+0000 mgr.y (mgr.34107) 4209 : cluster [DBG] pgmap v5616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:33.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:33 smithi115 bash[45495]: cluster 2022-04-22T19:28:31.555746+0000 mgr.y (mgr.34107) 4209 : cluster [DBG] pgmap v5616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:33.863 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:28:33.863 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (111m) 7m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:28:33.863 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (113m) 7m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (113m) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (119m) 8m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (119m) 7m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (112m) 7m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (112m) 8m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:28:33.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (118m) 7m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (117m) 7m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (117m) 7m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (116m) 7m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (116m) 8m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (115m) 8m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (114m) 8m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (114m) 8m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:28:33.865 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (112m) 8m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:28:34.268 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:28:34.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:28:34.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:28:34.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:34.269 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:28:34.269 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:28:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:28:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:28:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:28:34.271 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:28:34.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:28:34.271 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:28:34.271 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:28:34.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:28:34.272 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:28:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:33.953Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:34 smithi121 bash[36902]: audit 2022-04-22T19:28:33.822967+0000 mon.a (mon.0) 7002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:34 smithi121 bash[36902]: audit 2022-04-22T19:28:34.256904+0000 mon.a (mon.0) 7003 : audit [DBG] from='client.? 172.21.15.115:0/1708291499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:34 smithi115 bash[46934]: audit 2022-04-22T19:28:33.822967+0000 mon.a (mon.0) 7002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:34 smithi115 bash[46934]: audit 2022-04-22T19:28:34.256904+0000 mon.a (mon.0) 7003 : audit [DBG] from='client.? 172.21.15.115:0/1708291499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:34.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:34 smithi115 bash[45495]: audit 2022-04-22T19:28:33.822967+0000 mon.a (mon.0) 7002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:34.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:34 smithi115 bash[45495]: audit 2022-04-22T19:28:34.256904+0000 mon.a (mon.0) 7003 : audit [DBG] from='client.? 172.21.15.115:0/1708291499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:28:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:35 smithi121 bash[36902]: audit 2022-04-22T19:28:33.493450+0000 mgr.y (mgr.34107) 4210 : audit [DBG] from='client.39798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:35 smithi121 bash[36902]: cluster 2022-04-22T19:28:33.556835+0000 mgr.y (mgr.34107) 4211 : cluster [DBG] pgmap v5617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:35 smithi121 bash[36902]: audit 2022-04-22T19:28:33.833419+0000 mgr.y (mgr.34107) 4212 : audit [DBG] from='client.39804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:35 smithi121 bash[36902]: audit 2022-04-22T19:28:35.361426+0000 mon.a (mon.0) 7004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[45495]: audit 2022-04-22T19:28:33.493450+0000 mgr.y (mgr.34107) 4210 : audit [DBG] from='client.39798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[45495]: cluster 2022-04-22T19:28:33.556835+0000 mgr.y (mgr.34107) 4211 : cluster [DBG] pgmap v5617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[45495]: audit 2022-04-22T19:28:33.833419+0000 mgr.y (mgr.34107) 4212 : audit [DBG] from='client.39804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[45495]: audit 2022-04-22T19:28:35.361426+0000 mon.a (mon.0) 7004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[46934]: audit 2022-04-22T19:28:33.493450+0000 mgr.y (mgr.34107) 4210 : audit [DBG] from='client.39798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[46934]: cluster 2022-04-22T19:28:33.556835+0000 mgr.y (mgr.34107) 4211 : cluster [DBG] pgmap v5617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:35.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[46934]: audit 2022-04-22T19:28:33.833419+0000 mgr.y (mgr.34107) 4212 : audit [DBG] from='client.39804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:28:35.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:35 smithi115 bash[46934]: audit 2022-04-22T19:28:35.361426+0000 mon.a (mon.0) 7004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:37 smithi121 bash[36902]: cluster 2022-04-22T19:28:35.557900+0000 mgr.y (mgr.34107) 4213 : cluster [DBG] pgmap v5618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:37 smithi115 bash[45495]: cluster 2022-04-22T19:28:35.557900+0000 mgr.y (mgr.34107) 4213 : cluster [DBG] pgmap v5618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:37 smithi115 bash[46934]: cluster 2022-04-22T19:28:35.557900+0000 mgr.y (mgr.34107) 4213 : cluster [DBG] pgmap v5618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:39 smithi121 bash[36902]: cluster 2022-04-22T19:28:37.558906+0000 mgr.y (mgr.34107) 4214 : cluster [DBG] pgmap v5619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:39 smithi121 bash[36902]: audit 2022-04-22T19:28:38.826691+0000 mon.a (mon.0) 7005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:39 smithi115 bash[45495]: cluster 2022-04-22T19:28:37.558906+0000 mgr.y (mgr.34107) 4214 : cluster [DBG] pgmap v5619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:39.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:39 smithi115 bash[45495]: audit 2022-04-22T19:28:38.826691+0000 mon.a (mon.0) 7005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:39 smithi115 bash[46934]: cluster 2022-04-22T19:28:37.558906+0000 mgr.y (mgr.34107) 4214 : cluster [DBG] pgmap v5619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:39.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:39 smithi115 bash[46934]: audit 2022-04-22T19:28:38.826691+0000 mon.a (mon.0) 7005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:40 smithi121 bash[36902]: audit 2022-04-22T19:28:40.364612+0000 mon.a (mon.0) 7006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:40.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:40 smithi115 bash[45495]: audit 2022-04-22T19:28:40.364612+0000 mon.a (mon.0) 7006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:40.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:40 smithi115 bash[46934]: audit 2022-04-22T19:28:40.364612+0000 mon.a (mon.0) 7006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:41 smithi121 bash[36902]: cluster 2022-04-22T19:28:39.559934+0000 mgr.y (mgr.34107) 4215 : cluster [DBG] pgmap v5620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:41 smithi115 bash[45495]: cluster 2022-04-22T19:28:39.559934+0000 mgr.y (mgr.34107) 4215 : cluster [DBG] pgmap v5620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:41 smithi115 bash[46934]: cluster 2022-04-22T19:28:39.559934+0000 mgr.y (mgr.34107) 4215 : cluster [DBG] pgmap v5620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:42.082Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:43 smithi121 bash[36902]: cluster 2022-04-22T19:28:41.560947+0000 mgr.y (mgr.34107) 4216 : cluster [DBG] pgmap v5621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:43.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:43 smithi115 bash[45495]: cluster 2022-04-22T19:28:41.560947+0000 mgr.y (mgr.34107) 4216 : cluster [DBG] pgmap v5621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:43 smithi115 bash[46934]: cluster 2022-04-22T19:28:41.560947+0000 mgr.y (mgr.34107) 4216 : cluster [DBG] pgmap v5621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:43.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:44 smithi121 bash[36902]: audit 2022-04-22T19:28:43.830174+0000 mon.a (mon.0) 7007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:44 smithi115 bash[45495]: audit 2022-04-22T19:28:43.830174+0000 mon.a (mon.0) 7007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:44 smithi115 bash[46934]: audit 2022-04-22T19:28:43.830174+0000 mon.a (mon.0) 7007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:45 smithi121 bash[36902]: cluster 2022-04-22T19:28:43.562142+0000 mgr.y (mgr.34107) 4217 : cluster [DBG] pgmap v5622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:45 smithi121 bash[36902]: audit 2022-04-22T19:28:45.222898+0000 mon.a (mon.0) 7008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:28:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:45 smithi121 bash[36902]: audit 2022-04-22T19:28:45.225347+0000 mon.a (mon.0) 7009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:28:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:45 smithi121 bash[36902]: audit 2022-04-22T19:28:45.226774+0000 mon.a (mon.0) 7010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:28:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:45 smithi121 bash[36902]: audit 2022-04-22T19:28:45.367814+0000 mon.a (mon.0) 7011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[46934]: cluster 2022-04-22T19:28:43.562142+0000 mgr.y (mgr.34107) 4217 : cluster [DBG] pgmap v5622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[46934]: audit 2022-04-22T19:28:45.222898+0000 mon.a (mon.0) 7008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:28:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[46934]: audit 2022-04-22T19:28:45.225347+0000 mon.a (mon.0) 7009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:28:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[46934]: audit 2022-04-22T19:28:45.226774+0000 mon.a (mon.0) 7010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:28:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[46934]: audit 2022-04-22T19:28:45.367814+0000 mon.a (mon.0) 7011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[45495]: cluster 2022-04-22T19:28:43.562142+0000 mgr.y (mgr.34107) 4217 : cluster [DBG] pgmap v5622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[45495]: audit 2022-04-22T19:28:45.222898+0000 mon.a (mon.0) 7008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[45495]: audit 2022-04-22T19:28:45.225347+0000 mon.a (mon.0) 7009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[45495]: audit 2022-04-22T19:28:45.226774+0000 mon.a (mon.0) 7010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:45 smithi115 bash[45495]: audit 2022-04-22T19:28:45.367814+0000 mon.a (mon.0) 7011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:46 smithi121 bash[36902]: cluster 2022-04-22T19:28:45.563211+0000 mgr.y (mgr.34107) 4218 : cluster [DBG] pgmap v5623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:46 smithi121 bash[36902]: audit 2022-04-22T19:28:45.592188+0000 mon.a (mon.0) 7012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:28:47.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:46 smithi115 bash[45495]: cluster 2022-04-22T19:28:45.563211+0000 mgr.y (mgr.34107) 4218 : cluster [DBG] pgmap v5623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:47.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:46 smithi115 bash[45495]: audit 2022-04-22T19:28:45.592188+0000 mon.a (mon.0) 7012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:28:47.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:46 smithi115 bash[46934]: cluster 2022-04-22T19:28:45.563211+0000 mgr.y (mgr.34107) 4218 : cluster [DBG] pgmap v5623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:47.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:46 smithi115 bash[46934]: audit 2022-04-22T19:28:45.592188+0000 mon.a (mon.0) 7012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:28:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:48 smithi121 bash[36902]: cluster 2022-04-22T19:28:47.564470+0000 mgr.y (mgr.34107) 4219 : cluster [DBG] pgmap v5624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:48 smithi121 bash[36902]: audit 2022-04-22T19:28:48.833768+0000 mon.a (mon.0) 7013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:49.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:48 smithi115 bash[45495]: cluster 2022-04-22T19:28:47.564470+0000 mgr.y (mgr.34107) 4219 : cluster [DBG] pgmap v5624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:49.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:48 smithi115 bash[45495]: audit 2022-04-22T19:28:48.833768+0000 mon.a (mon.0) 7013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:48 smithi115 bash[46934]: cluster 2022-04-22T19:28:47.564470+0000 mgr.y (mgr.34107) 4219 : cluster [DBG] pgmap v5624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:48 smithi115 bash[46934]: audit 2022-04-22T19:28:48.833768+0000 mon.a (mon.0) 7013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:50 smithi121 bash[36902]: cluster 2022-04-22T19:28:49.565668+0000 mgr.y (mgr.34107) 4220 : cluster [DBG] pgmap v5625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:50 smithi121 bash[36902]: audit 2022-04-22T19:28:50.370754+0000 mon.a (mon.0) 7014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:51.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:50 smithi115 bash[45495]: cluster 2022-04-22T19:28:49.565668+0000 mgr.y (mgr.34107) 4220 : cluster [DBG] pgmap v5625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:51.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:50 smithi115 bash[45495]: audit 2022-04-22T19:28:50.370754+0000 mon.a (mon.0) 7014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:50 smithi115 bash[46934]: cluster 2022-04-22T19:28:49.565668+0000 mgr.y (mgr.34107) 4220 : cluster [DBG] pgmap v5625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:50 smithi115 bash[46934]: audit 2022-04-22T19:28:50.370754+0000 mon.a (mon.0) 7014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:52.084Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:52 smithi121 bash[36902]: cluster 2022-04-22T19:28:51.566800+0000 mgr.y (mgr.34107) 4221 : cluster [DBG] pgmap v5626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:53.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:52 smithi115 bash[45495]: cluster 2022-04-22T19:28:51.566800+0000 mgr.y (mgr.34107) 4221 : cluster [DBG] pgmap v5626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:53.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:52 smithi115 bash[46934]: cluster 2022-04-22T19:28:51.566800+0000 mgr.y (mgr.34107) 4221 : cluster [DBG] pgmap v5626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:53 smithi121 bash[36902]: audit 2022-04-22T19:28:53.837508+0000 mon.a (mon.0) 7015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:54.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:53 smithi115 bash[45495]: audit 2022-04-22T19:28:53.837508+0000 mon.a (mon.0) 7015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:28:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:28:53.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:28:54.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:53 smithi115 bash[46934]: audit 2022-04-22T19:28:53.837508+0000 mon.a (mon.0) 7015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:54 smithi121 bash[36902]: cluster 2022-04-22T19:28:53.567952+0000 mgr.y (mgr.34107) 4222 : cluster [DBG] pgmap v5627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:54 smithi115 bash[45495]: cluster 2022-04-22T19:28:53.567952+0000 mgr.y (mgr.34107) 4222 : cluster [DBG] pgmap v5627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:54 smithi115 bash[46934]: cluster 2022-04-22T19:28:53.567952+0000 mgr.y (mgr.34107) 4222 : cluster [DBG] pgmap v5627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:55 smithi121 bash[36902]: audit 2022-04-22T19:28:55.374170+0000 mon.a (mon.0) 7016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:55 smithi115 bash[46934]: audit 2022-04-22T19:28:55.374170+0000 mon.a (mon.0) 7016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:55 smithi115 bash[45495]: audit 2022-04-22T19:28:55.374170+0000 mon.a (mon.0) 7016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:28:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:56 smithi121 bash[36902]: cluster 2022-04-22T19:28:55.569029+0000 mgr.y (mgr.34107) 4223 : cluster [DBG] pgmap v5628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:57.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:56 smithi115 bash[45495]: cluster 2022-04-22T19:28:55.569029+0000 mgr.y (mgr.34107) 4223 : cluster [DBG] pgmap v5628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:57.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:56 smithi115 bash[46934]: cluster 2022-04-22T19:28:55.569029+0000 mgr.y (mgr.34107) 4223 : cluster [DBG] pgmap v5628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:58 smithi121 bash[36902]: cluster 2022-04-22T19:28:57.570099+0000 mgr.y (mgr.34107) 4224 : cluster [DBG] pgmap v5629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:28:58 smithi121 bash[36902]: audit 2022-04-22T19:28:58.841156+0000 mon.a (mon.0) 7017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:58 smithi115 bash[45495]: cluster 2022-04-22T19:28:57.570099+0000 mgr.y (mgr.34107) 4224 : cluster [DBG] pgmap v5629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:59.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:28:58 smithi115 bash[45495]: audit 2022-04-22T19:28:58.841156+0000 mon.a (mon.0) 7017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:28:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:58 smithi115 bash[46934]: cluster 2022-04-22T19:28:57.570099+0000 mgr.y (mgr.34107) 4224 : cluster [DBG] pgmap v5629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:28:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:28:58 smithi115 bash[46934]: audit 2022-04-22T19:28:58.841156+0000 mon.a (mon.0) 7017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:00 smithi121 bash[36902]: cluster 2022-04-22T19:28:59.571182+0000 mgr.y (mgr.34107) 4225 : cluster [DBG] pgmap v5630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:00 smithi121 bash[36902]: audit 2022-04-22T19:29:00.377539+0000 mon.a (mon.0) 7018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:00 smithi115 bash[45495]: cluster 2022-04-22T19:28:59.571182+0000 mgr.y (mgr.34107) 4225 : cluster [DBG] pgmap v5630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:00 smithi115 bash[45495]: audit 2022-04-22T19:29:00.377539+0000 mon.a (mon.0) 7018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:00 smithi115 bash[46934]: cluster 2022-04-22T19:28:59.571182+0000 mgr.y (mgr.34107) 4225 : cluster [DBG] pgmap v5630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:00 smithi115 bash[46934]: audit 2022-04-22T19:29:00.377539+0000 mon.a (mon.0) 7018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:02.085Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:02 smithi121 bash[36902]: cluster 2022-04-22T19:29:01.572311+0000 mgr.y (mgr.34107) 4226 : cluster [DBG] pgmap v5631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:03.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:02 smithi115 bash[45495]: cluster 2022-04-22T19:29:01.572311+0000 mgr.y (mgr.34107) 4226 : cluster [DBG] pgmap v5631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:03.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:02 smithi115 bash[46934]: cluster 2022-04-22T19:29:01.572311+0000 mgr.y (mgr.34107) 4226 : cluster [DBG] pgmap v5631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:03 smithi121 bash[36902]: audit 2022-04-22T19:29:03.844838+0000 mon.a (mon.0) 7019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:03 smithi115 bash[45495]: audit 2022-04-22T19:29:03.844838+0000 mon.a (mon.0) 7019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:03.957Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:04.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:03 smithi115 bash[46934]: audit 2022-04-22T19:29:03.844838+0000 mon.a (mon.0) 7019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:04.650 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (112m) 8m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (114m) 8m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (113m) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:29:04.993 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (113m) 8m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (113m) 9m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (118m) 8m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (118m) 8m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:29:04.994 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (117m) 8m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (117m) 8m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (116m) 9m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (115m) 9m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (115m) 9m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (114m) 9m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:29:04.996 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (112m) 9m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:29:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:04 smithi121 bash[36902]: cluster 2022-04-22T19:29:03.573431+0000 mgr.y (mgr.34107) 4227 : cluster [DBG] pgmap v5632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:04 smithi115 bash[46934]: cluster 2022-04-22T19:29:03.573431+0000 mgr.y (mgr.34107) 4227 : cluster [DBG] pgmap v5632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:04 smithi115 bash[45495]: cluster 2022-04-22T19:29:03.573431+0000 mgr.y (mgr.34107) 4227 : cluster [DBG] pgmap v5632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:05.399 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:29:05.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:29:05.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:29:05.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:05.401 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:29:05.401 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:29:05.401 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:29:05.401 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:29:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:05 smithi121 bash[36902]: audit 2022-04-22T19:29:04.624093+0000 mgr.y (mgr.34107) 4228 : audit [DBG] from='client.39816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:05 smithi121 bash[36902]: audit 2022-04-22T19:29:04.962639+0000 mgr.y (mgr.34107) 4229 : audit [DBG] from='client.39822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:05 smithi121 bash[36902]: audit 2022-04-22T19:29:05.380992+0000 mon.a (mon.0) 7020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:05 smithi121 bash[36902]: audit 2022-04-22T19:29:05.388337+0000 mon.c (mon.1) 193 : audit [DBG] from='client.? 172.21.15.115:0/2185342557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[45495]: audit 2022-04-22T19:29:04.624093+0000 mgr.y (mgr.34107) 4228 : audit [DBG] from='client.39816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[45495]: audit 2022-04-22T19:29:04.962639+0000 mgr.y (mgr.34107) 4229 : audit [DBG] from='client.39822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[45495]: audit 2022-04-22T19:29:05.380992+0000 mon.a (mon.0) 7020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:06.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[45495]: audit 2022-04-22T19:29:05.388337+0000 mon.c (mon.1) 193 : audit [DBG] from='client.? 172.21.15.115:0/2185342557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[46934]: audit 2022-04-22T19:29:04.624093+0000 mgr.y (mgr.34107) 4228 : audit [DBG] from='client.39816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[46934]: audit 2022-04-22T19:29:04.962639+0000 mgr.y (mgr.34107) 4229 : audit [DBG] from='client.39822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[46934]: audit 2022-04-22T19:29:05.380992+0000 mon.a (mon.0) 7020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:06.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:05 smithi115 bash[46934]: audit 2022-04-22T19:29:05.388337+0000 mon.c (mon.1) 193 : audit [DBG] from='client.? 172.21.15.115:0/2185342557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:06 smithi121 bash[36902]: cluster 2022-04-22T19:29:05.574510+0000 mgr.y (mgr.34107) 4230 : cluster [DBG] pgmap v5633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:07.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:06 smithi115 bash[45495]: cluster 2022-04-22T19:29:05.574510+0000 mgr.y (mgr.34107) 4230 : cluster [DBG] pgmap v5633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:07.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:06 smithi115 bash[46934]: cluster 2022-04-22T19:29:05.574510+0000 mgr.y (mgr.34107) 4230 : cluster [DBG] pgmap v5633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:08 smithi121 bash[36902]: cluster 2022-04-22T19:29:07.575657+0000 mgr.y (mgr.34107) 4231 : cluster [DBG] pgmap v5634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:08 smithi121 bash[36902]: audit 2022-04-22T19:29:08.848578+0000 mon.a (mon.0) 7021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:08 smithi115 bash[46934]: cluster 2022-04-22T19:29:07.575657+0000 mgr.y (mgr.34107) 4231 : cluster [DBG] pgmap v5634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:08 smithi115 bash[46934]: audit 2022-04-22T19:29:08.848578+0000 mon.a (mon.0) 7021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:08 smithi115 bash[45495]: cluster 2022-04-22T19:29:07.575657+0000 mgr.y (mgr.34107) 4231 : cluster [DBG] pgmap v5634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:09.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:08 smithi115 bash[45495]: audit 2022-04-22T19:29:08.848578+0000 mon.a (mon.0) 7021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:10 smithi121 bash[36902]: cluster 2022-04-22T19:29:09.576625+0000 mgr.y (mgr.34107) 4232 : cluster [DBG] pgmap v5635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:10 smithi121 bash[36902]: audit 2022-04-22T19:29:10.383933+0000 mon.a (mon.0) 7022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:11.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:10 smithi115 bash[45495]: cluster 2022-04-22T19:29:09.576625+0000 mgr.y (mgr.34107) 4232 : cluster [DBG] pgmap v5635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:11.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:10 smithi115 bash[45495]: audit 2022-04-22T19:29:10.383933+0000 mon.a (mon.0) 7022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:11.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:10 smithi115 bash[46934]: cluster 2022-04-22T19:29:09.576625+0000 mgr.y (mgr.34107) 4232 : cluster [DBG] pgmap v5635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:11.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:10 smithi115 bash[46934]: audit 2022-04-22T19:29:10.383933+0000 mon.a (mon.0) 7022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:12.085Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:12 smithi121 bash[36902]: cluster 2022-04-22T19:29:11.577715+0000 mgr.y (mgr.34107) 4233 : cluster [DBG] pgmap v5636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:12 smithi115 bash[45495]: cluster 2022-04-22T19:29:11.577715+0000 mgr.y (mgr.34107) 4233 : cluster [DBG] pgmap v5636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:12 smithi115 bash[46934]: cluster 2022-04-22T19:29:11.577715+0000 mgr.y (mgr.34107) 4233 : cluster [DBG] pgmap v5636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:13 smithi121 bash[36902]: audit 2022-04-22T19:29:13.852205+0000 mon.a (mon.0) 7023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:14.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:13 smithi115 bash[45495]: audit 2022-04-22T19:29:13.852205+0000 mon.a (mon.0) 7023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:14.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:13 smithi115 bash[46934]: audit 2022-04-22T19:29:13.852205+0000 mon.a (mon.0) 7023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:14.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:13.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:14 smithi121 bash[36902]: cluster 2022-04-22T19:29:13.578952+0000 mgr.y (mgr.34107) 4234 : cluster [DBG] pgmap v5637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:14 smithi121 bash[36902]: audit 2022-04-22T19:29:14.678209+0000 mon.a (mon.0) 7024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:29:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:14 smithi121 bash[36902]: audit 2022-04-22T19:29:14.678452+0000 mon.a (mon.0) 7025 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:29:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[45495]: cluster 2022-04-22T19:29:13.578952+0000 mgr.y (mgr.34107) 4234 : cluster [DBG] pgmap v5637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:15.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[45495]: audit 2022-04-22T19:29:14.678209+0000 mon.a (mon.0) 7024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:29:15.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[45495]: audit 2022-04-22T19:29:14.678452+0000 mon.a (mon.0) 7025 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:29:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[46934]: cluster 2022-04-22T19:29:13.578952+0000 mgr.y (mgr.34107) 4234 : cluster [DBG] pgmap v5637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[46934]: audit 2022-04-22T19:29:14.678209+0000 mon.a (mon.0) 7024 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:29:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:14 smithi115 bash[46934]: audit 2022-04-22T19:29:14.678452+0000 mon.a (mon.0) 7025 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:29:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:15 smithi121 bash[36902]: audit 2022-04-22T19:29:15.387075+0000 mon.a (mon.0) 7026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:16.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:15 smithi115 bash[45495]: audit 2022-04-22T19:29:15.387075+0000 mon.a (mon.0) 7026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:16.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:15 smithi115 bash[46934]: audit 2022-04-22T19:29:15.387075+0000 mon.a (mon.0) 7026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:16 smithi121 bash[36902]: cluster 2022-04-22T19:29:15.580188+0000 mgr.y (mgr.34107) 4235 : cluster [DBG] pgmap v5638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:17.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:16 smithi115 bash[45495]: cluster 2022-04-22T19:29:15.580188+0000 mgr.y (mgr.34107) 4235 : cluster [DBG] pgmap v5638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:16 smithi115 bash[46934]: cluster 2022-04-22T19:29:15.580188+0000 mgr.y (mgr.34107) 4235 : cluster [DBG] pgmap v5638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:18 smithi121 bash[36902]: cluster 2022-04-22T19:29:17.581302+0000 mgr.y (mgr.34107) 4236 : cluster [DBG] pgmap v5639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:18 smithi121 bash[36902]: audit 2022-04-22T19:29:18.855948+0000 mon.a (mon.0) 7027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:19.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:18 smithi115 bash[45495]: cluster 2022-04-22T19:29:17.581302+0000 mgr.y (mgr.34107) 4236 : cluster [DBG] pgmap v5639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:19.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:18 smithi115 bash[45495]: audit 2022-04-22T19:29:18.855948+0000 mon.a (mon.0) 7027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:18 smithi115 bash[46934]: cluster 2022-04-22T19:29:17.581302+0000 mgr.y (mgr.34107) 4236 : cluster [DBG] pgmap v5639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:18 smithi115 bash[46934]: audit 2022-04-22T19:29:18.855948+0000 mon.a (mon.0) 7027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:20 smithi121 bash[36902]: cluster 2022-04-22T19:29:19.582367+0000 mgr.y (mgr.34107) 4237 : cluster [DBG] pgmap v5640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:20 smithi121 bash[36902]: audit 2022-04-22T19:29:20.390222+0000 mon.a (mon.0) 7028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:20 smithi115 bash[45495]: cluster 2022-04-22T19:29:19.582367+0000 mgr.y (mgr.34107) 4237 : cluster [DBG] pgmap v5640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:20 smithi115 bash[45495]: audit 2022-04-22T19:29:20.390222+0000 mon.a (mon.0) 7028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:20 smithi115 bash[46934]: cluster 2022-04-22T19:29:19.582367+0000 mgr.y (mgr.34107) 4237 : cluster [DBG] pgmap v5640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:20 smithi115 bash[46934]: audit 2022-04-22T19:29:20.390222+0000 mon.a (mon.0) 7028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:22.087Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:22 smithi121 bash[36902]: cluster 2022-04-22T19:29:21.583444+0000 mgr.y (mgr.34107) 4238 : cluster [DBG] pgmap v5641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:22 smithi115 bash[45495]: cluster 2022-04-22T19:29:21.583444+0000 mgr.y (mgr.34107) 4238 : cluster [DBG] pgmap v5641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:22 smithi115 bash[46934]: cluster 2022-04-22T19:29:21.583444+0000 mgr.y (mgr.34107) 4238 : cluster [DBG] pgmap v5641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:23 smithi121 bash[36902]: audit 2022-04-22T19:29:23.859677+0000 mon.a (mon.0) 7029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:23 smithi115 bash[45495]: audit 2022-04-22T19:29:23.859677+0000 mon.a (mon.0) 7029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:24.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:23 smithi115 bash[46934]: audit 2022-04-22T19:29:23.859677+0000 mon.a (mon.0) 7029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:24.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:23.958Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:24 smithi121 bash[36902]: cluster 2022-04-22T19:29:23.584474+0000 mgr.y (mgr.34107) 4239 : cluster [DBG] pgmap v5642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:24 smithi115 bash[45495]: cluster 2022-04-22T19:29:23.584474+0000 mgr.y (mgr.34107) 4239 : cluster [DBG] pgmap v5642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:25.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:24 smithi115 bash[46934]: cluster 2022-04-22T19:29:23.584474+0000 mgr.y (mgr.34107) 4239 : cluster [DBG] pgmap v5642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:25 smithi121 bash[36902]: audit 2022-04-22T19:29:25.393480+0000 mon.a (mon.0) 7030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:26.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:25 smithi115 bash[45495]: audit 2022-04-22T19:29:25.393480+0000 mon.a (mon.0) 7030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:25 smithi115 bash[46934]: audit 2022-04-22T19:29:25.393480+0000 mon.a (mon.0) 7030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:26 smithi121 bash[36902]: cluster 2022-04-22T19:29:25.585551+0000 mgr.y (mgr.34107) 4240 : cluster [DBG] pgmap v5643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:27.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:26 smithi115 bash[45495]: cluster 2022-04-22T19:29:25.585551+0000 mgr.y (mgr.34107) 4240 : cluster [DBG] pgmap v5643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:27.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:26 smithi115 bash[46934]: cluster 2022-04-22T19:29:25.585551+0000 mgr.y (mgr.34107) 4240 : cluster [DBG] pgmap v5643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:29 smithi121 bash[36902]: cluster 2022-04-22T19:29:27.586649+0000 mgr.y (mgr.34107) 4241 : cluster [DBG] pgmap v5644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:29 smithi121 bash[36902]: audit 2022-04-22T19:29:28.863389+0000 mon.a (mon.0) 7031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:29.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:28 smithi115 bash[45495]: cluster 2022-04-22T19:29:27.586649+0000 mgr.y (mgr.34107) 4241 : cluster [DBG] pgmap v5644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:29.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:28 smithi115 bash[45495]: audit 2022-04-22T19:29:28.863389+0000 mon.a (mon.0) 7031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:28 smithi115 bash[46934]: cluster 2022-04-22T19:29:27.586649+0000 mgr.y (mgr.34107) 4241 : cluster [DBG] pgmap v5644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:28 smithi115 bash[46934]: audit 2022-04-22T19:29:28.863389+0000 mon.a (mon.0) 7031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:31 smithi121 bash[36902]: cluster 2022-04-22T19:29:29.587789+0000 mgr.y (mgr.34107) 4242 : cluster [DBG] pgmap v5645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:31 smithi121 bash[36902]: audit 2022-04-22T19:29:30.396713+0000 mon.a (mon.0) 7032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:30 smithi115 bash[45495]: cluster 2022-04-22T19:29:29.587789+0000 mgr.y (mgr.34107) 4242 : cluster [DBG] pgmap v5645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:31.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:30 smithi115 bash[45495]: audit 2022-04-22T19:29:30.396713+0000 mon.a (mon.0) 7032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:30 smithi115 bash[46934]: cluster 2022-04-22T19:29:29.587789+0000 mgr.y (mgr.34107) 4242 : cluster [DBG] pgmap v5645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:31.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:30 smithi115 bash[46934]: audit 2022-04-22T19:29:30.396713+0000 mon.a (mon.0) 7032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:32.088Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:33 smithi121 bash[36902]: cluster 2022-04-22T19:29:31.588790+0000 mgr.y (mgr.34107) 4243 : cluster [DBG] pgmap v5646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:33.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:33 smithi115 bash[45495]: cluster 2022-04-22T19:29:31.588790+0000 mgr.y (mgr.34107) 4243 : cluster [DBG] pgmap v5646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:33.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:33 smithi115 bash[46934]: cluster 2022-04-22T19:29:31.588790+0000 mgr.y (mgr.34107) 4243 : cluster [DBG] pgmap v5646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:33.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:34.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:34 smithi115 bash[45495]: audit 2022-04-22T19:29:33.867060+0000 mon.a (mon.0) 7033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:34.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:34 smithi115 bash[46934]: audit 2022-04-22T19:29:33.867060+0000 mon.a (mon.0) 7033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:34 smithi121 bash[36902]: audit 2022-04-22T19:29:33.867060+0000 mon.a (mon.0) 7033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:35.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:35 smithi115 bash[45495]: cluster 2022-04-22T19:29:33.589963+0000 mgr.y (mgr.34107) 4244 : cluster [DBG] pgmap v5647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:35.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:35 smithi115 bash[46934]: cluster 2022-04-22T19:29:33.589963+0000 mgr.y (mgr.34107) 4244 : cluster [DBG] pgmap v5647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:35 smithi121 bash[36902]: cluster 2022-04-22T19:29:33.589963+0000 mgr.y (mgr.34107) 4244 : cluster [DBG] pgmap v5647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:35.777 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:29:36.130 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:29:36.130 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (112m) 8m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:29:36.130 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.6M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:29:36.130 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (114m) 8m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:29:36.131 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (114m) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:29:36.131 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:29:36.131 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:29:36.131 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:29:36.133 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 173M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:29:36.133 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:29:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (113m) 8m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:29:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (113m) 9m ago 2h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T19:29:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (119m) 8m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:29:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (118m) 8m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:29:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (118m) 8m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:29:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (117m) 8m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:29:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (117m) 9m ago 2h 572M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:29:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (116m) 9m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:29:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (115m) 9m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:29:36.136 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (115m) 9m ago 2h 632M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:29:36.136 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (113m) 9m ago 2h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:29:36.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:36 smithi115 bash[45495]: audit 2022-04-22T19:29:35.399764+0000 mon.a (mon.0) 7034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:36 smithi115 bash[46934]: audit 2022-04-22T19:29:35.399764+0000 mon.a (mon.0) 7034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:36 smithi121 bash[36902]: audit 2022-04-22T19:29:35.399764+0000 mon.a (mon.0) 7034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:36.533 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:29:36.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:36.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:29:36.535 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:29:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[45495]: cluster 2022-04-22T19:29:35.590722+0000 mgr.y (mgr.34107) 4245 : cluster [DBG] pgmap v5648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[45495]: audit 2022-04-22T19:29:35.749978+0000 mgr.y (mgr.34107) 4246 : audit [DBG] from='client.49768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[45495]: audit 2022-04-22T19:29:36.100288+0000 mgr.y (mgr.34107) 4247 : audit [DBG] from='client.39840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[45495]: audit 2022-04-22T19:29:36.522506+0000 mon.a (mon.0) 7035 : audit [DBG] from='client.? 172.21.15.115:0/152759514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[46934]: cluster 2022-04-22T19:29:35.590722+0000 mgr.y (mgr.34107) 4245 : cluster [DBG] pgmap v5648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[46934]: audit 2022-04-22T19:29:35.749978+0000 mgr.y (mgr.34107) 4246 : audit [DBG] from='client.49768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[46934]: audit 2022-04-22T19:29:36.100288+0000 mgr.y (mgr.34107) 4247 : audit [DBG] from='client.39840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:37 smithi115 bash[46934]: audit 2022-04-22T19:29:36.522506+0000 mon.a (mon.0) 7035 : audit [DBG] from='client.? 172.21.15.115:0/152759514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:37 smithi121 bash[36902]: cluster 2022-04-22T19:29:35.590722+0000 mgr.y (mgr.34107) 4245 : cluster [DBG] pgmap v5648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:37 smithi121 bash[36902]: audit 2022-04-22T19:29:35.749978+0000 mgr.y (mgr.34107) 4246 : audit [DBG] from='client.49768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:37 smithi121 bash[36902]: audit 2022-04-22T19:29:36.100288+0000 mgr.y (mgr.34107) 4247 : audit [DBG] from='client.39840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:29:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:37 smithi121 bash[36902]: audit 2022-04-22T19:29:36.522506+0000 mon.a (mon.0) 7035 : audit [DBG] from='client.? 172.21.15.115:0/152759514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:29:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:39 smithi115 bash[46934]: cluster 2022-04-22T19:29:37.591774+0000 mgr.y (mgr.34107) 4248 : cluster [DBG] pgmap v5649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:39 smithi115 bash[46934]: audit 2022-04-22T19:29:38.870766+0000 mon.a (mon.0) 7036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:39 smithi115 bash[45495]: cluster 2022-04-22T19:29:37.591774+0000 mgr.y (mgr.34107) 4248 : cluster [DBG] pgmap v5649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:39.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:39 smithi115 bash[45495]: audit 2022-04-22T19:29:38.870766+0000 mon.a (mon.0) 7036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:39 smithi121 bash[36902]: cluster 2022-04-22T19:29:37.591774+0000 mgr.y (mgr.34107) 4248 : cluster [DBG] pgmap v5649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:39 smithi121 bash[36902]: audit 2022-04-22T19:29:38.870766+0000 mon.a (mon.0) 7036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:41.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:41 smithi115 bash[45495]: cluster 2022-04-22T19:29:39.592752+0000 mgr.y (mgr.34107) 4249 : cluster [DBG] pgmap v5650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:41.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:41 smithi115 bash[45495]: audit 2022-04-22T19:29:40.402901+0000 mon.a (mon.0) 7037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:41 smithi115 bash[46934]: cluster 2022-04-22T19:29:39.592752+0000 mgr.y (mgr.34107) 4249 : cluster [DBG] pgmap v5650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:41.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:41 smithi115 bash[46934]: audit 2022-04-22T19:29:40.402901+0000 mon.a (mon.0) 7037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:41 smithi121 bash[36902]: cluster 2022-04-22T19:29:39.592752+0000 mgr.y (mgr.34107) 4249 : cluster [DBG] pgmap v5650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:41 smithi121 bash[36902]: audit 2022-04-22T19:29:40.402901+0000 mon.a (mon.0) 7037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:42.086Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:43 smithi121 bash[36902]: cluster 2022-04-22T19:29:41.593783+0000 mgr.y (mgr.34107) 4250 : cluster [DBG] pgmap v5651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:43 smithi115 bash[45495]: cluster 2022-04-22T19:29:41.593783+0000 mgr.y (mgr.34107) 4250 : cluster [DBG] pgmap v5651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:43 smithi115 bash[46934]: cluster 2022-04-22T19:29:41.593783+0000 mgr.y (mgr.34107) 4250 : cluster [DBG] pgmap v5651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:44.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:44 smithi115 bash[45495]: audit 2022-04-22T19:29:43.874453+0000 mon.a (mon.0) 7038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:43.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:44.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:44 smithi115 bash[46934]: audit 2022-04-22T19:29:43.874453+0000 mon.a (mon.0) 7038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:44 smithi121 bash[36902]: audit 2022-04-22T19:29:43.874453+0000 mon.a (mon.0) 7038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:45 smithi121 bash[36902]: cluster 2022-04-22T19:29:43.594947+0000 mgr.y (mgr.34107) 4251 : cluster [DBG] pgmap v5652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:45 smithi115 bash[45495]: cluster 2022-04-22T19:29:43.594947+0000 mgr.y (mgr.34107) 4251 : cluster [DBG] pgmap v5652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:45 smithi115 bash[46934]: cluster 2022-04-22T19:29:43.594947+0000 mgr.y (mgr.34107) 4251 : cluster [DBG] pgmap v5652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[45495]: audit 2022-04-22T19:29:45.406106+0000 mon.a (mon.0) 7039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[45495]: audit 2022-04-22T19:29:45.600625+0000 mon.a (mon.0) 7040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:29:46.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[45495]: audit 2022-04-22T19:29:45.603401+0000 mon.a (mon.0) 7041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:29:46.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[45495]: audit 2022-04-22T19:29:45.604822+0000 mon.a (mon.0) 7042 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:29:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[46934]: audit 2022-04-22T19:29:45.406106+0000 mon.a (mon.0) 7039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[46934]: audit 2022-04-22T19:29:45.600625+0000 mon.a (mon.0) 7040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:29:46.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[46934]: audit 2022-04-22T19:29:45.603401+0000 mon.a (mon.0) 7041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:29:46.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:46 smithi115 bash[46934]: audit 2022-04-22T19:29:45.604822+0000 mon.a (mon.0) 7042 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:29:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:46 smithi121 bash[36902]: audit 2022-04-22T19:29:45.406106+0000 mon.a (mon.0) 7039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:46 smithi121 bash[36902]: audit 2022-04-22T19:29:45.600625+0000 mon.a (mon.0) 7040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:29:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:46 smithi121 bash[36902]: audit 2022-04-22T19:29:45.603401+0000 mon.a (mon.0) 7041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:29:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:46 smithi121 bash[36902]: audit 2022-04-22T19:29:45.604822+0000 mon.a (mon.0) 7042 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:29:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:47 smithi121 bash[36902]: cluster 2022-04-22T19:29:45.596056+0000 mgr.y (mgr.34107) 4252 : cluster [DBG] pgmap v5653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:47 smithi115 bash[45495]: cluster 2022-04-22T19:29:45.596056+0000 mgr.y (mgr.34107) 4252 : cluster [DBG] pgmap v5653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:47 smithi115 bash[46934]: cluster 2022-04-22T19:29:45.596056+0000 mgr.y (mgr.34107) 4252 : cluster [DBG] pgmap v5653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:49 smithi121 bash[36902]: cluster 2022-04-22T19:29:47.597125+0000 mgr.y (mgr.34107) 4253 : cluster [DBG] pgmap v5654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:49 smithi121 bash[36902]: audit 2022-04-22T19:29:48.878318+0000 mon.a (mon.0) 7043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:49 smithi115 bash[45495]: cluster 2022-04-22T19:29:47.597125+0000 mgr.y (mgr.34107) 4253 : cluster [DBG] pgmap v5654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:49 smithi115 bash[45495]: audit 2022-04-22T19:29:48.878318+0000 mon.a (mon.0) 7043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:49 smithi115 bash[46934]: cluster 2022-04-22T19:29:47.597125+0000 mgr.y (mgr.34107) 4253 : cluster [DBG] pgmap v5654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:49.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:49 smithi115 bash[46934]: audit 2022-04-22T19:29:48.878318+0000 mon.a (mon.0) 7043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:50 smithi121 bash[36902]: cluster 2022-04-22T19:29:49.598196+0000 mgr.y (mgr.34107) 4254 : cluster [DBG] pgmap v5655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:50 smithi121 bash[36902]: audit 2022-04-22T19:29:49.964976+0000 mon.a (mon.0) 7044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:50 smithi121 bash[36902]: audit 2022-04-22T19:29:50.277086+0000 mon.a (mon.0) 7045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:50 smithi121 bash[36902]: audit 2022-04-22T19:29:50.409365+0000 mon.a (mon.0) 7046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:51.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[46934]: cluster 2022-04-22T19:29:49.598196+0000 mgr.y (mgr.34107) 4254 : cluster [DBG] pgmap v5655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:51.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[46934]: audit 2022-04-22T19:29:49.964976+0000 mon.a (mon.0) 7044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[46934]: audit 2022-04-22T19:29:50.277086+0000 mon.a (mon.0) 7045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[46934]: audit 2022-04-22T19:29:50.409365+0000 mon.a (mon.0) 7046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[45495]: cluster 2022-04-22T19:29:49.598196+0000 mgr.y (mgr.34107) 4254 : cluster [DBG] pgmap v5655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[45495]: audit 2022-04-22T19:29:49.964976+0000 mon.a (mon.0) 7044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[45495]: audit 2022-04-22T19:29:50.277086+0000 mon.a (mon.0) 7045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:29:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:50 smithi115 bash[45495]: audit 2022-04-22T19:29:50.409365+0000 mon.a (mon.0) 7046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:52.090Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:52 smithi121 bash[36902]: cluster 2022-04-22T19:29:51.599297+0000 mgr.y (mgr.34107) 4255 : cluster [DBG] pgmap v5656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:53.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:52 smithi115 bash[45495]: cluster 2022-04-22T19:29:51.599297+0000 mgr.y (mgr.34107) 4255 : cluster [DBG] pgmap v5656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:53.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:52 smithi115 bash[46934]: cluster 2022-04-22T19:29:51.599297+0000 mgr.y (mgr.34107) 4255 : cluster [DBG] pgmap v5656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:53 smithi121 bash[36902]: audit 2022-04-22T19:29:53.881160+0000 mon.a (mon.0) 7047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:54.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:53 smithi115 bash[45495]: audit 2022-04-22T19:29:53.881160+0000 mon.a (mon.0) 7047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:54.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:53 smithi115 bash[46934]: audit 2022-04-22T19:29:53.881160+0000 mon.a (mon.0) 7047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:54.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:29:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:29:53.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:29:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:54 smithi121 bash[36902]: cluster 2022-04-22T19:29:53.600404+0000 mgr.y (mgr.34107) 4256 : cluster [DBG] pgmap v5657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:55.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:54 smithi115 bash[45495]: cluster 2022-04-22T19:29:53.600404+0000 mgr.y (mgr.34107) 4256 : cluster [DBG] pgmap v5657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:55.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:54 smithi115 bash[46934]: cluster 2022-04-22T19:29:53.600404+0000 mgr.y (mgr.34107) 4256 : cluster [DBG] pgmap v5657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:55 smithi121 bash[36902]: audit 2022-04-22T19:29:55.412447+0000 mon.a (mon.0) 7048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:56.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:55 smithi115 bash[45495]: audit 2022-04-22T19:29:55.412447+0000 mon.a (mon.0) 7048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:55 smithi115 bash[46934]: audit 2022-04-22T19:29:55.412447+0000 mon.a (mon.0) 7048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:29:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:57 smithi121 bash[36902]: cluster 2022-04-22T19:29:55.601463+0000 mgr.y (mgr.34107) 4257 : cluster [DBG] pgmap v5658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:57.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:56 smithi115 bash[45495]: cluster 2022-04-22T19:29:55.601463+0000 mgr.y (mgr.34107) 4257 : cluster [DBG] pgmap v5658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:57.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:56 smithi115 bash[46934]: cluster 2022-04-22T19:29:55.601463+0000 mgr.y (mgr.34107) 4257 : cluster [DBG] pgmap v5658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:59 smithi121 bash[36902]: cluster 2022-04-22T19:29:57.602529+0000 mgr.y (mgr.34107) 4258 : cluster [DBG] pgmap v5659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:29:59 smithi121 bash[36902]: audit 2022-04-22T19:29:58.884827+0000 mon.a (mon.0) 7049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:58 smithi115 bash[45495]: cluster 2022-04-22T19:29:57.602529+0000 mgr.y (mgr.34107) 4258 : cluster [DBG] pgmap v5659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:29:58 smithi115 bash[45495]: audit 2022-04-22T19:29:58.884827+0000 mon.a (mon.0) 7049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:29:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:58 smithi115 bash[46934]: cluster 2022-04-22T19:29:57.602529+0000 mgr.y (mgr.34107) 4258 : cluster [DBG] pgmap v5659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:29:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:29:58 smithi115 bash[46934]: audit 2022-04-22T19:29:58.884827+0000 mon.a (mon.0) 7049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:01 smithi121 bash[36902]: cluster 2022-04-22T19:29:59.603595+0000 mgr.y (mgr.34107) 4259 : cluster [DBG] pgmap v5660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:01 smithi121 bash[36902]: cluster 2022-04-22T19:30:00.001471+0000 mon.a (mon.0) 7050 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:01 smithi121 bash[36902]: audit 2022-04-22T19:30:00.415608+0000 mon.a (mon.0) 7051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:00 smithi115 bash[45495]: cluster 2022-04-22T19:29:59.603595+0000 mgr.y (mgr.34107) 4259 : cluster [DBG] pgmap v5660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:01 smithi115 bash[45495]: cluster 2022-04-22T19:30:00.001471+0000 mon.a (mon.0) 7050 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:30:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:01 smithi115 bash[45495]: audit 2022-04-22T19:30:00.415608+0000 mon.a (mon.0) 7051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:01 smithi115 bash[46934]: cluster 2022-04-22T19:29:59.603595+0000 mgr.y (mgr.34107) 4259 : cluster [DBG] pgmap v5660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:01 smithi115 bash[46934]: cluster 2022-04-22T19:30:00.001471+0000 mon.a (mon.0) 7050 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:30:01.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:01 smithi115 bash[46934]: audit 2022-04-22T19:30:00.415608+0000 mon.a (mon.0) 7051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:02.089Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:03.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:03 smithi115 bash[45495]: cluster 2022-04-22T19:30:01.604661+0000 mgr.y (mgr.34107) 4260 : cluster [DBG] pgmap v5661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:03.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:03 smithi115 bash[46934]: cluster 2022-04-22T19:30:01.604661+0000 mgr.y (mgr.34107) 4260 : cluster [DBG] pgmap v5661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:03 smithi121 bash[36902]: cluster 2022-04-22T19:30:01.604661+0000 mgr.y (mgr.34107) 4260 : cluster [DBG] pgmap v5661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:04 smithi115 bash[45495]: audit 2022-04-22T19:30:03.888525+0000 mon.a (mon.0) 7052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:04 smithi115 bash[46934]: audit 2022-04-22T19:30:03.888525+0000 mon.a (mon.0) 7052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:03.960Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:04 smithi121 bash[36902]: audit 2022-04-22T19:30:03.888525+0000 mon.a (mon.0) 7052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:05 smithi115 bash[45495]: cluster 2022-04-22T19:30:03.605790+0000 mgr.y (mgr.34107) 4261 : cluster [DBG] pgmap v5662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:05 smithi115 bash[46934]: cluster 2022-04-22T19:30:03.605790+0000 mgr.y (mgr.34107) 4261 : cluster [DBG] pgmap v5662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:05 smithi121 bash[36902]: cluster 2022-04-22T19:30:03.605790+0000 mgr.y (mgr.34107) 4261 : cluster [DBG] pgmap v5662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:06.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:06 smithi115 bash[45495]: audit 2022-04-22T19:30:05.418681+0000 mon.a (mon.0) 7053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:06 smithi115 bash[46934]: audit 2022-04-22T19:30:05.418681+0000 mon.a (mon.0) 7053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:06 smithi121 bash[36902]: audit 2022-04-22T19:30:05.418681+0000 mon.a (mon.0) 7053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:06.900 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:30:07.254 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (113m) 9m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 17s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (115m) 9m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (114m) 17s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 17s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:30:07.255 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 17s ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (114m) 9m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (114m) 17s ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (119m) 9m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (119m) 9m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (118m) 9m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:30:07.256 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (118m) 9m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:30:07.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (117m) 17s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:30:07.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (116m) 17s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:30:07.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (116m) 17s ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:30:07.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (115m) 17s ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:30:07.257 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (113m) 17s ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:30:07.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:07 smithi115 bash[45495]: cluster 2022-04-22T19:30:05.606802+0000 mgr.y (mgr.34107) 4262 : cluster [DBG] pgmap v5663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:07.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:07 smithi115 bash[46934]: cluster 2022-04-22T19:30:05.606802+0000 mgr.y (mgr.34107) 4262 : cluster [DBG] pgmap v5663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:07 smithi121 bash[36902]: cluster 2022-04-22T19:30:05.606802+0000 mgr.y (mgr.34107) 4262 : cluster [DBG] pgmap v5663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:07.655 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:30:07.655 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:30:07.656 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:30:07.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:07.657 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:30:07.657 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:30:07.657 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:30:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:30:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:08 smithi121 bash[36902]: audit 2022-04-22T19:30:06.873339+0000 mgr.y (mgr.34107) 4263 : audit [DBG] from='client.39852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:08 smithi121 bash[36902]: audit 2022-04-22T19:30:07.221818+0000 mgr.y (mgr.34107) 4264 : audit [DBG] from='client.39858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:08 smithi121 bash[36902]: audit 2022-04-22T19:30:07.644231+0000 mon.c (mon.1) 194 : audit [DBG] from='client.? 172.21.15.115:0/3625712288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[45495]: audit 2022-04-22T19:30:06.873339+0000 mgr.y (mgr.34107) 4263 : audit [DBG] from='client.39852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[45495]: audit 2022-04-22T19:30:07.221818+0000 mgr.y (mgr.34107) 4264 : audit [DBG] from='client.39858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[45495]: audit 2022-04-22T19:30:07.644231+0000 mon.c (mon.1) 194 : audit [DBG] from='client.? 172.21.15.115:0/3625712288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[46934]: audit 2022-04-22T19:30:06.873339+0000 mgr.y (mgr.34107) 4263 : audit [DBG] from='client.39852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[46934]: audit 2022-04-22T19:30:07.221818+0000 mgr.y (mgr.34107) 4264 : audit [DBG] from='client.39858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:08 smithi115 bash[46934]: audit 2022-04-22T19:30:07.644231+0000 mon.c (mon.1) 194 : audit [DBG] from='client.? 172.21.15.115:0/3625712288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:09 smithi121 bash[36902]: cluster 2022-04-22T19:30:07.607807+0000 mgr.y (mgr.34107) 4265 : cluster [DBG] pgmap v5664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:09 smithi121 bash[36902]: audit 2022-04-22T19:30:08.891975+0000 mon.a (mon.0) 7054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:09 smithi115 bash[45495]: cluster 2022-04-22T19:30:07.607807+0000 mgr.y (mgr.34107) 4265 : cluster [DBG] pgmap v5664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:09 smithi115 bash[45495]: audit 2022-04-22T19:30:08.891975+0000 mon.a (mon.0) 7054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:09 smithi115 bash[46934]: cluster 2022-04-22T19:30:07.607807+0000 mgr.y (mgr.34107) 4265 : cluster [DBG] pgmap v5664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:09 smithi115 bash[46934]: audit 2022-04-22T19:30:08.891975+0000 mon.a (mon.0) 7054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:11 smithi121 bash[36902]: cluster 2022-04-22T19:30:09.608771+0000 mgr.y (mgr.34107) 4266 : cluster [DBG] pgmap v5665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:11 smithi121 bash[36902]: audit 2022-04-22T19:30:10.421738+0000 mon.a (mon.0) 7055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:11 smithi115 bash[45495]: cluster 2022-04-22T19:30:09.608771+0000 mgr.y (mgr.34107) 4266 : cluster [DBG] pgmap v5665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:11 smithi115 bash[45495]: audit 2022-04-22T19:30:10.421738+0000 mon.a (mon.0) 7055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:11 smithi115 bash[46934]: cluster 2022-04-22T19:30:09.608771+0000 mgr.y (mgr.34107) 4266 : cluster [DBG] pgmap v5665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:11.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:11 smithi115 bash[46934]: audit 2022-04-22T19:30:10.421738+0000 mon.a (mon.0) 7055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:12.091Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:13 smithi121 bash[36902]: cluster 2022-04-22T19:30:11.609882+0000 mgr.y (mgr.34107) 4267 : cluster [DBG] pgmap v5666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:13 smithi115 bash[45495]: cluster 2022-04-22T19:30:11.609882+0000 mgr.y (mgr.34107) 4267 : cluster [DBG] pgmap v5666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:13.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:13 smithi115 bash[46934]: cluster 2022-04-22T19:30:11.609882+0000 mgr.y (mgr.34107) 4267 : cluster [DBG] pgmap v5666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:14.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:14 smithi115 bash[46934]: audit 2022-04-22T19:30:13.895561+0000 mon.a (mon.0) 7056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:13.960Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:14 smithi115 bash[45495]: audit 2022-04-22T19:30:13.895561+0000 mon.a (mon.0) 7056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:14 smithi121 bash[36902]: audit 2022-04-22T19:30:13.895561+0000 mon.a (mon.0) 7056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:15 smithi121 bash[36902]: cluster 2022-04-22T19:30:13.610988+0000 mgr.y (mgr.34107) 4268 : cluster [DBG] pgmap v5667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:15 smithi121 bash[36902]: audit 2022-04-22T19:30:14.690937+0000 mon.a (mon.0) 7057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:30:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:15 smithi121 bash[36902]: audit 2022-04-22T19:30:14.691140+0000 mon.a (mon.0) 7058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:30:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[45495]: cluster 2022-04-22T19:30:13.610988+0000 mgr.y (mgr.34107) 4268 : cluster [DBG] pgmap v5667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[45495]: audit 2022-04-22T19:30:14.690937+0000 mon.a (mon.0) 7057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:30:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[45495]: audit 2022-04-22T19:30:14.691140+0000 mon.a (mon.0) 7058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:30:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[46934]: cluster 2022-04-22T19:30:13.610988+0000 mgr.y (mgr.34107) 4268 : cluster [DBG] pgmap v5667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[46934]: audit 2022-04-22T19:30:14.690937+0000 mon.a (mon.0) 7057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:30:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:15 smithi115 bash[46934]: audit 2022-04-22T19:30:14.691140+0000 mon.a (mon.0) 7058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:30:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:16 smithi121 bash[36902]: audit 2022-04-22T19:30:15.424191+0000 mon.a (mon.0) 7059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:16.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:16 smithi115 bash[45495]: audit 2022-04-22T19:30:15.424191+0000 mon.a (mon.0) 7059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:16 smithi115 bash[46934]: audit 2022-04-22T19:30:15.424191+0000 mon.a (mon.0) 7059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:17 smithi121 bash[36902]: cluster 2022-04-22T19:30:15.612100+0000 mgr.y (mgr.34107) 4269 : cluster [DBG] pgmap v5668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:17.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:17 smithi115 bash[45495]: cluster 2022-04-22T19:30:15.612100+0000 mgr.y (mgr.34107) 4269 : cluster [DBG] pgmap v5668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:17 smithi115 bash[46934]: cluster 2022-04-22T19:30:15.612100+0000 mgr.y (mgr.34107) 4269 : cluster [DBG] pgmap v5668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:19 smithi121 bash[36902]: cluster 2022-04-22T19:30:17.613182+0000 mgr.y (mgr.34107) 4270 : cluster [DBG] pgmap v5669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:19 smithi121 bash[36902]: audit 2022-04-22T19:30:18.899421+0000 mon.a (mon.0) 7060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:19 smithi115 bash[45495]: cluster 2022-04-22T19:30:17.613182+0000 mgr.y (mgr.34107) 4270 : cluster [DBG] pgmap v5669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:19 smithi115 bash[45495]: audit 2022-04-22T19:30:18.899421+0000 mon.a (mon.0) 7060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:19 smithi115 bash[46934]: cluster 2022-04-22T19:30:17.613182+0000 mgr.y (mgr.34107) 4270 : cluster [DBG] pgmap v5669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:19 smithi115 bash[46934]: audit 2022-04-22T19:30:18.899421+0000 mon.a (mon.0) 7060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:21.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:21 smithi121 bash[36902]: cluster 2022-04-22T19:30:19.614275+0000 mgr.y (mgr.34107) 4271 : cluster [DBG] pgmap v5670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:21.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:21 smithi121 bash[36902]: audit 2022-04-22T19:30:20.427180+0000 mon.a (mon.0) 7061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:21 smithi115 bash[45495]: cluster 2022-04-22T19:30:19.614275+0000 mgr.y (mgr.34107) 4271 : cluster [DBG] pgmap v5670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:21 smithi115 bash[45495]: audit 2022-04-22T19:30:20.427180+0000 mon.a (mon.0) 7061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:21 smithi115 bash[46934]: cluster 2022-04-22T19:30:19.614275+0000 mgr.y (mgr.34107) 4271 : cluster [DBG] pgmap v5670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:21 smithi115 bash[46934]: audit 2022-04-22T19:30:20.427180+0000 mon.a (mon.0) 7061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:22.091Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:23 smithi121 bash[36902]: cluster 2022-04-22T19:30:21.615366+0000 mgr.y (mgr.34107) 4272 : cluster [DBG] pgmap v5671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:23 smithi115 bash[45495]: cluster 2022-04-22T19:30:21.615366+0000 mgr.y (mgr.34107) 4272 : cluster [DBG] pgmap v5671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:23 smithi115 bash[46934]: cluster 2022-04-22T19:30:21.615366+0000 mgr.y (mgr.34107) 4272 : cluster [DBG] pgmap v5671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:24 smithi115 bash[45495]: audit 2022-04-22T19:30:23.902825+0000 mon.a (mon.0) 7062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:23.960Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:24.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:24 smithi115 bash[46934]: audit 2022-04-22T19:30:23.902825+0000 mon.a (mon.0) 7062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:24 smithi121 bash[36902]: audit 2022-04-22T19:30:23.902825+0000 mon.a (mon.0) 7062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:25 smithi121 bash[36902]: cluster 2022-04-22T19:30:23.616486+0000 mgr.y (mgr.34107) 4273 : cluster [DBG] pgmap v5672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:25.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:25 smithi115 bash[45495]: cluster 2022-04-22T19:30:23.616486+0000 mgr.y (mgr.34107) 4273 : cluster [DBG] pgmap v5672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:25.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:25 smithi115 bash[46934]: cluster 2022-04-22T19:30:23.616486+0000 mgr.y (mgr.34107) 4273 : cluster [DBG] pgmap v5672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:26 smithi121 bash[36902]: audit 2022-04-22T19:30:25.430320+0000 mon.a (mon.0) 7063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:26.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:26 smithi115 bash[45495]: audit 2022-04-22T19:30:25.430320+0000 mon.a (mon.0) 7063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:26 smithi115 bash[46934]: audit 2022-04-22T19:30:25.430320+0000 mon.a (mon.0) 7063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:27 smithi121 bash[36902]: cluster 2022-04-22T19:30:25.617562+0000 mgr.y (mgr.34107) 4274 : cluster [DBG] pgmap v5673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:27 smithi115 bash[45495]: cluster 2022-04-22T19:30:25.617562+0000 mgr.y (mgr.34107) 4274 : cluster [DBG] pgmap v5673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:27 smithi115 bash[46934]: cluster 2022-04-22T19:30:25.617562+0000 mgr.y (mgr.34107) 4274 : cluster [DBG] pgmap v5673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:29 smithi121 bash[36902]: cluster 2022-04-22T19:30:27.618651+0000 mgr.y (mgr.34107) 4275 : cluster [DBG] pgmap v5674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:29 smithi121 bash[36902]: audit 2022-04-22T19:30:28.906734+0000 mon.a (mon.0) 7064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:29 smithi115 bash[45495]: cluster 2022-04-22T19:30:27.618651+0000 mgr.y (mgr.34107) 4275 : cluster [DBG] pgmap v5674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:29 smithi115 bash[45495]: audit 2022-04-22T19:30:28.906734+0000 mon.a (mon.0) 7064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:29 smithi115 bash[46934]: cluster 2022-04-22T19:30:27.618651+0000 mgr.y (mgr.34107) 4275 : cluster [DBG] pgmap v5674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:29 smithi115 bash[46934]: audit 2022-04-22T19:30:28.906734+0000 mon.a (mon.0) 7064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:31 smithi121 bash[36902]: cluster 2022-04-22T19:30:29.619751+0000 mgr.y (mgr.34107) 4276 : cluster [DBG] pgmap v5675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:31 smithi121 bash[36902]: audit 2022-04-22T19:30:30.433469+0000 mon.a (mon.0) 7065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:31 smithi115 bash[45495]: cluster 2022-04-22T19:30:29.619751+0000 mgr.y (mgr.34107) 4276 : cluster [DBG] pgmap v5675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:31 smithi115 bash[45495]: audit 2022-04-22T19:30:30.433469+0000 mon.a (mon.0) 7065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:31.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:31 smithi115 bash[46934]: cluster 2022-04-22T19:30:29.619751+0000 mgr.y (mgr.34107) 4276 : cluster [DBG] pgmap v5675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:31.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:31 smithi115 bash[46934]: audit 2022-04-22T19:30:30.433469+0000 mon.a (mon.0) 7065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:32.092Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:33.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:33 smithi115 bash[46934]: cluster 2022-04-22T19:30:31.620771+0000 mgr.y (mgr.34107) 4277 : cluster [DBG] pgmap v5676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:33.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:33 smithi115 bash[45495]: cluster 2022-04-22T19:30:31.620771+0000 mgr.y (mgr.34107) 4277 : cluster [DBG] pgmap v5676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:33 smithi121 bash[36902]: cluster 2022-04-22T19:30:31.620771+0000 mgr.y (mgr.34107) 4277 : cluster [DBG] pgmap v5676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:34 smithi115 bash[45495]: audit 2022-04-22T19:30:33.910606+0000 mon.a (mon.0) 7066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:34.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:33.962Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:34.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:34 smithi115 bash[46934]: audit 2022-04-22T19:30:33.910606+0000 mon.a (mon.0) 7066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:34 smithi121 bash[36902]: audit 2022-04-22T19:30:33.910606+0000 mon.a (mon.0) 7066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:35 smithi121 bash[36902]: cluster 2022-04-22T19:30:33.621900+0000 mgr.y (mgr.34107) 4278 : cluster [DBG] pgmap v5677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:35 smithi115 bash[45495]: cluster 2022-04-22T19:30:33.621900+0000 mgr.y (mgr.34107) 4278 : cluster [DBG] pgmap v5677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:35 smithi115 bash[46934]: cluster 2022-04-22T19:30:33.621900+0000 mgr.y (mgr.34107) 4278 : cluster [DBG] pgmap v5677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:36 smithi121 bash[36902]: audit 2022-04-22T19:30:35.436619+0000 mon.a (mon.0) 7067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:36 smithi115 bash[45495]: audit 2022-04-22T19:30:35.436619+0000 mon.a (mon.0) 7067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:36.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:36 smithi115 bash[46934]: audit 2022-04-22T19:30:35.436619+0000 mon.a (mon.0) 7067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:37 smithi121 bash[36902]: cluster 2022-04-22T19:30:35.622951+0000 mgr.y (mgr.34107) 4279 : cluster [DBG] pgmap v5678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:37 smithi115 bash[45495]: cluster 2022-04-22T19:30:35.622951+0000 mgr.y (mgr.34107) 4279 : cluster [DBG] pgmap v5678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:37 smithi115 bash[46934]: cluster 2022-04-22T19:30:35.622951+0000 mgr.y (mgr.34107) 4279 : cluster [DBG] pgmap v5678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:38.031 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (113m) 9m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 48s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (115m) 9m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (115m) 48s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 48s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:30:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 479M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 274M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 48s ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 175M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (114m) 9m ago 2h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (114m) 48s ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 699M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (119m) 9m ago 2h 779M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:30:38.369 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (119m) 9m ago 2h 528M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (118m) 9m ago 2h 483M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (118m) 48s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (117m) 48s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (116m) 48s ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (116m) 48s ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:30:38.370 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (114m) 48s ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:30:38.768 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:30:38.768 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:38.769 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:30:38.770 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:30:38.771 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:30:38.771 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:39 smithi121 bash[36902]: cluster 2022-04-22T19:30:37.624054+0000 mgr.y (mgr.34107) 4280 : cluster [DBG] pgmap v5679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:39 smithi121 bash[36902]: audit 2022-04-22T19:30:38.004561+0000 mgr.y (mgr.34107) 4281 : audit [DBG] from='client.39870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:39 smithi121 bash[36902]: audit 2022-04-22T19:30:38.342661+0000 mgr.y (mgr.34107) 4282 : audit [DBG] from='client.39876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:39 smithi121 bash[36902]: audit 2022-04-22T19:30:38.757447+0000 mon.c (mon.1) 195 : audit [DBG] from='client.? 172.21.15.115:0/1419994887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:39 smithi121 bash[36902]: audit 2022-04-22T19:30:38.914276+0000 mon.a (mon.0) 7068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[45495]: cluster 2022-04-22T19:30:37.624054+0000 mgr.y (mgr.34107) 4280 : cluster [DBG] pgmap v5679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[45495]: audit 2022-04-22T19:30:38.004561+0000 mgr.y (mgr.34107) 4281 : audit [DBG] from='client.39870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[45495]: audit 2022-04-22T19:30:38.342661+0000 mgr.y (mgr.34107) 4282 : audit [DBG] from='client.39876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[45495]: audit 2022-04-22T19:30:38.757447+0000 mon.c (mon.1) 195 : audit [DBG] from='client.? 172.21.15.115:0/1419994887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[45495]: audit 2022-04-22T19:30:38.914276+0000 mon.a (mon.0) 7068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[46934]: cluster 2022-04-22T19:30:37.624054+0000 mgr.y (mgr.34107) 4280 : cluster [DBG] pgmap v5679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[46934]: audit 2022-04-22T19:30:38.004561+0000 mgr.y (mgr.34107) 4281 : audit [DBG] from='client.39870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[46934]: audit 2022-04-22T19:30:38.342661+0000 mgr.y (mgr.34107) 4282 : audit [DBG] from='client.39876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:30:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[46934]: audit 2022-04-22T19:30:38.757447+0000 mon.c (mon.1) 195 : audit [DBG] from='client.? 172.21.15.115:0/1419994887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:30:39.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:39 smithi115 bash[46934]: audit 2022-04-22T19:30:38.914276+0000 mon.a (mon.0) 7068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:41 smithi121 bash[36902]: cluster 2022-04-22T19:30:39.625124+0000 mgr.y (mgr.34107) 4283 : cluster [DBG] pgmap v5680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:41 smithi121 bash[36902]: audit 2022-04-22T19:30:40.439723+0000 mon.a (mon.0) 7069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:41 smithi115 bash[45495]: cluster 2022-04-22T19:30:39.625124+0000 mgr.y (mgr.34107) 4283 : cluster [DBG] pgmap v5680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:41 smithi115 bash[45495]: audit 2022-04-22T19:30:40.439723+0000 mon.a (mon.0) 7069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:41 smithi115 bash[46934]: cluster 2022-04-22T19:30:39.625124+0000 mgr.y (mgr.34107) 4283 : cluster [DBG] pgmap v5680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:41 smithi115 bash[46934]: audit 2022-04-22T19:30:40.439723+0000 mon.a (mon.0) 7069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:42.092Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:43 smithi121 bash[36902]: cluster 2022-04-22T19:30:41.626202+0000 mgr.y (mgr.34107) 4284 : cluster [DBG] pgmap v5681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:43 smithi115 bash[45495]: cluster 2022-04-22T19:30:41.626202+0000 mgr.y (mgr.34107) 4284 : cluster [DBG] pgmap v5681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:43 smithi115 bash[46934]: cluster 2022-04-22T19:30:41.626202+0000 mgr.y (mgr.34107) 4284 : cluster [DBG] pgmap v5681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:44.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:44 smithi115 bash[45495]: audit 2022-04-22T19:30:43.917730+0000 mon.a (mon.0) 7070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:43.964Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:44.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:44 smithi115 bash[46934]: audit 2022-04-22T19:30:43.917730+0000 mon.a (mon.0) 7070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:44 smithi121 bash[36902]: audit 2022-04-22T19:30:43.917730+0000 mon.a (mon.0) 7070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:45 smithi121 bash[36902]: cluster 2022-04-22T19:30:43.627353+0000 mgr.y (mgr.34107) 4285 : cluster [DBG] pgmap v5682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:45 smithi115 bash[45495]: cluster 2022-04-22T19:30:43.627353+0000 mgr.y (mgr.34107) 4285 : cluster [DBG] pgmap v5682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:45 smithi115 bash[46934]: cluster 2022-04-22T19:30:43.627353+0000 mgr.y (mgr.34107) 4285 : cluster [DBG] pgmap v5682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:46 smithi121 bash[36902]: audit 2022-04-22T19:30:45.442958+0000 mon.a (mon.0) 7071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:46 smithi115 bash[46934]: audit 2022-04-22T19:30:45.442958+0000 mon.a (mon.0) 7071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:46 smithi115 bash[45495]: audit 2022-04-22T19:30:45.442958+0000 mon.a (mon.0) 7071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:47 smithi121 bash[36902]: cluster 2022-04-22T19:30:45.628470+0000 mgr.y (mgr.34107) 4286 : cluster [DBG] pgmap v5683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:47 smithi115 bash[46934]: cluster 2022-04-22T19:30:45.628470+0000 mgr.y (mgr.34107) 4286 : cluster [DBG] pgmap v5683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:47 smithi115 bash[45495]: cluster 2022-04-22T19:30:45.628470+0000 mgr.y (mgr.34107) 4286 : cluster [DBG] pgmap v5683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:49 smithi121 bash[36902]: cluster 2022-04-22T19:30:47.629545+0000 mgr.y (mgr.34107) 4287 : cluster [DBG] pgmap v5684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:49 smithi121 bash[36902]: audit 2022-04-22T19:30:48.921391+0000 mon.a (mon.0) 7072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:49 smithi115 bash[46934]: cluster 2022-04-22T19:30:47.629545+0000 mgr.y (mgr.34107) 4287 : cluster [DBG] pgmap v5684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:49 smithi115 bash[46934]: audit 2022-04-22T19:30:48.921391+0000 mon.a (mon.0) 7072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:49 smithi115 bash[45495]: cluster 2022-04-22T19:30:47.629545+0000 mgr.y (mgr.34107) 4287 : cluster [DBG] pgmap v5684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:49 smithi115 bash[45495]: audit 2022-04-22T19:30:48.921391+0000 mon.a (mon.0) 7072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: cluster 2022-04-22T19:30:49.630591+0000 mgr.y (mgr.34107) 4288 : cluster [DBG] pgmap v5685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.286510+0000 mon.a (mon.0) 7073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.289543+0000 mon.a (mon.0) 7074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.291328+0000 mon.a (mon.0) 7075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.446109+0000 mon.a (mon.0) 7076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.608137+0000 mon.a (mon.0) 7077 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:51.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:51 smithi121 bash[36902]: audit 2022-04-22T19:30:50.621496+0000 mon.a (mon.0) 7078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: cluster 2022-04-22T19:30:49.630591+0000 mgr.y (mgr.34107) 4288 : cluster [DBG] pgmap v5685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.286510+0000 mon.a (mon.0) 7073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:30:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.289543+0000 mon.a (mon.0) 7074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:30:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.291328+0000 mon.a (mon.0) 7075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.446109+0000 mon.a (mon.0) 7076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.608137+0000 mon.a (mon.0) 7077 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[46934]: audit 2022-04-22T19:30:50.621496+0000 mon.a (mon.0) 7078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: cluster 2022-04-22T19:30:49.630591+0000 mgr.y (mgr.34107) 4288 : cluster [DBG] pgmap v5685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.286510+0000 mon.a (mon.0) 7073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.289543+0000 mon.a (mon.0) 7074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:30:51.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.291328+0000 mon.a (mon.0) 7075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:30:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.446109+0000 mon.a (mon.0) 7076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.608137+0000 mon.a (mon.0) 7077 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:51.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:51 smithi115 bash[45495]: audit 2022-04-22T19:30:50.621496+0000 mon.a (mon.0) 7078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:52.094Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:53 smithi121 bash[36902]: cluster 2022-04-22T19:30:51.631830+0000 mgr.y (mgr.34107) 4289 : cluster [DBG] pgmap v5686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:53 smithi115 bash[45495]: cluster 2022-04-22T19:30:51.631830+0000 mgr.y (mgr.34107) 4289 : cluster [DBG] pgmap v5686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:53 smithi115 bash[46934]: cluster 2022-04-22T19:30:51.631830+0000 mgr.y (mgr.34107) 4289 : cluster [DBG] pgmap v5686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:30:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:30:53.962Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:30:54.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:54 smithi115 bash[45495]: audit 2022-04-22T19:30:53.924899+0000 mon.a (mon.0) 7079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:54.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:54 smithi115 bash[46934]: audit 2022-04-22T19:30:53.924899+0000 mon.a (mon.0) 7079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:54 smithi121 bash[36902]: audit 2022-04-22T19:30:53.924899+0000 mon.a (mon.0) 7079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:55 smithi121 bash[36902]: cluster 2022-04-22T19:30:53.632849+0000 mgr.y (mgr.34107) 4290 : cluster [DBG] pgmap v5687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:55 smithi121 bash[36902]: audit 2022-04-22T19:30:54.755084+0000 mon.a (mon.0) 7080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:55 smithi121 bash[36902]: audit 2022-04-22T19:30:55.100901+0000 mon.a (mon.0) 7081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[45495]: cluster 2022-04-22T19:30:53.632849+0000 mgr.y (mgr.34107) 4290 : cluster [DBG] pgmap v5687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[45495]: audit 2022-04-22T19:30:54.755084+0000 mon.a (mon.0) 7080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[45495]: audit 2022-04-22T19:30:55.100901+0000 mon.a (mon.0) 7081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[46934]: cluster 2022-04-22T19:30:53.632849+0000 mgr.y (mgr.34107) 4290 : cluster [DBG] pgmap v5687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[46934]: audit 2022-04-22T19:30:54.755084+0000 mon.a (mon.0) 7080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:55 smithi115 bash[46934]: audit 2022-04-22T19:30:55.100901+0000 mon.a (mon.0) 7081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:30:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:56 smithi121 bash[36902]: audit 2022-04-22T19:30:55.449391+0000 mon.a (mon.0) 7082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:56 smithi115 bash[45495]: audit 2022-04-22T19:30:55.449391+0000 mon.a (mon.0) 7082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:56.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:56 smithi115 bash[46934]: audit 2022-04-22T19:30:55.449391+0000 mon.a (mon.0) 7082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:30:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:57 smithi121 bash[36902]: cluster 2022-04-22T19:30:55.633894+0000 mgr.y (mgr.34107) 4291 : cluster [DBG] pgmap v5688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:57 smithi115 bash[45495]: cluster 2022-04-22T19:30:55.633894+0000 mgr.y (mgr.34107) 4291 : cluster [DBG] pgmap v5688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:57.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:57 smithi115 bash[46934]: cluster 2022-04-22T19:30:55.633894+0000 mgr.y (mgr.34107) 4291 : cluster [DBG] pgmap v5688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:59 smithi121 bash[36902]: cluster 2022-04-22T19:30:57.635000+0000 mgr.y (mgr.34107) 4292 : cluster [DBG] pgmap v5689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:30:59 smithi121 bash[36902]: audit 2022-04-22T19:30:58.928481+0000 mon.a (mon.0) 7083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:59 smithi115 bash[45495]: cluster 2022-04-22T19:30:57.635000+0000 mgr.y (mgr.34107) 4292 : cluster [DBG] pgmap v5689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:30:59 smithi115 bash[45495]: audit 2022-04-22T19:30:58.928481+0000 mon.a (mon.0) 7083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:30:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:59 smithi115 bash[46934]: cluster 2022-04-22T19:30:57.635000+0000 mgr.y (mgr.34107) 4292 : cluster [DBG] pgmap v5689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:30:59.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:30:59 smithi115 bash[46934]: audit 2022-04-22T19:30:58.928481+0000 mon.a (mon.0) 7083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:01 smithi121 bash[36902]: cluster 2022-04-22T19:30:59.636059+0000 mgr.y (mgr.34107) 4293 : cluster [DBG] pgmap v5690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:01 smithi121 bash[36902]: audit 2022-04-22T19:31:00.452517+0000 mon.a (mon.0) 7084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:01 smithi115 bash[45495]: cluster 2022-04-22T19:30:59.636059+0000 mgr.y (mgr.34107) 4293 : cluster [DBG] pgmap v5690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:01 smithi115 bash[45495]: audit 2022-04-22T19:31:00.452517+0000 mon.a (mon.0) 7084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:01 smithi115 bash[46934]: cluster 2022-04-22T19:30:59.636059+0000 mgr.y (mgr.34107) 4293 : cluster [DBG] pgmap v5690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:01 smithi115 bash[46934]: audit 2022-04-22T19:31:00.452517+0000 mon.a (mon.0) 7084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:02.095Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:03 smithi121 bash[36902]: cluster 2022-04-22T19:31:01.637181+0000 mgr.y (mgr.34107) 4294 : cluster [DBG] pgmap v5691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:03 smithi115 bash[45495]: cluster 2022-04-22T19:31:01.637181+0000 mgr.y (mgr.34107) 4294 : cluster [DBG] pgmap v5691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:03.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:03 smithi115 bash[46934]: cluster 2022-04-22T19:31:01.637181+0000 mgr.y (mgr.34107) 4294 : cluster [DBG] pgmap v5691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:04.249 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:03.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:04 smithi121 bash[36902]: audit 2022-04-22T19:31:03.932135+0000 mon.a (mon.0) 7085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:04 smithi115 bash[46934]: audit 2022-04-22T19:31:03.932135+0000 mon.a (mon.0) 7085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:04.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:04 smithi115 bash[45495]: audit 2022-04-22T19:31:03.932135+0000 mon.a (mon.0) 7085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:05 smithi121 bash[36902]: cluster 2022-04-22T19:31:03.638349+0000 mgr.y (mgr.34107) 4295 : cluster [DBG] pgmap v5692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:05 smithi115 bash[45495]: cluster 2022-04-22T19:31:03.638349+0000 mgr.y (mgr.34107) 4295 : cluster [DBG] pgmap v5692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:05 smithi115 bash[46934]: cluster 2022-04-22T19:31:03.638349+0000 mgr.y (mgr.34107) 4295 : cluster [DBG] pgmap v5692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:06 smithi121 bash[36902]: audit 2022-04-22T19:31:05.455634+0000 mon.a (mon.0) 7086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:06.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:06 smithi115 bash[45495]: audit 2022-04-22T19:31:05.455634+0000 mon.a (mon.0) 7086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:06.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:06 smithi115 bash[46934]: audit 2022-04-22T19:31:05.455634+0000 mon.a (mon.0) 7086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:07 smithi121 bash[36902]: cluster 2022-04-22T19:31:05.639361+0000 mgr.y (mgr.34107) 4296 : cluster [DBG] pgmap v5693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:07.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:07 smithi115 bash[45495]: cluster 2022-04-22T19:31:05.639361+0000 mgr.y (mgr.34107) 4296 : cluster [DBG] pgmap v5693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:07.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:07 smithi115 bash[46934]: cluster 2022-04-22T19:31:05.639361+0000 mgr.y (mgr.34107) 4296 : cluster [DBG] pgmap v5693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:09.144 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:31:09.489 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:31:09.489 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (114m) 14s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:31:09.489 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 79s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:31:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (116m) 14s ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:31:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (115m) 79s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:31:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 79s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:31:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 14s ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:31:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 14s ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:31:09.491 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 79s ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:31:09.491 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 14s ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:31:09.491 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (115m) 14s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:31:09.491 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (115m) 79s ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:31:09.491 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 14s ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:31:09.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 14s ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:31:09.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (119m) 14s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:31:09.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (119m) 14s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:31:09.492 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (118m) 79s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:31:09.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (117m) 79s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:31:09.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (117m) 79s ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:31:09.493 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (116m) 79s ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:31:09.493 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (114m) 79s ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:31:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:09 smithi115 bash[46934]: cluster 2022-04-22T19:31:07.640395+0000 mgr.y (mgr.34107) 4297 : cluster [DBG] pgmap v5694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:09 smithi115 bash[46934]: audit 2022-04-22T19:31:08.935786+0000 mon.a (mon.0) 7087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:09 smithi115 bash[45495]: cluster 2022-04-22T19:31:07.640395+0000 mgr.y (mgr.34107) 4297 : cluster [DBG] pgmap v5694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:09.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:09 smithi115 bash[45495]: audit 2022-04-22T19:31:08.935786+0000 mon.a (mon.0) 7087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:09 smithi121 bash[36902]: cluster 2022-04-22T19:31:07.640395+0000 mgr.y (mgr.34107) 4297 : cluster [DBG] pgmap v5694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:09 smithi121 bash[36902]: audit 2022-04-22T19:31:08.935786+0000 mon.a (mon.0) 7087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:09.893 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:31:09.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:31:09.895 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:31:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[45495]: audit 2022-04-22T19:31:09.119381+0000 mgr.y (mgr.34107) 4298 : audit [DBG] from='client.49822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[45495]: audit 2022-04-22T19:31:09.458550+0000 mgr.y (mgr.34107) 4299 : audit [DBG] from='client.39894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[45495]: audit 2022-04-22T19:31:09.882574+0000 mon.a (mon.0) 7088 : audit [DBG] from='client.? 172.21.15.115:0/1394449229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[46934]: audit 2022-04-22T19:31:09.119381+0000 mgr.y (mgr.34107) 4298 : audit [DBG] from='client.49822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[46934]: audit 2022-04-22T19:31:09.458550+0000 mgr.y (mgr.34107) 4299 : audit [DBG] from='client.39894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:10 smithi115 bash[46934]: audit 2022-04-22T19:31:09.882574+0000 mon.a (mon.0) 7088 : audit [DBG] from='client.? 172.21.15.115:0/1394449229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:10 smithi121 bash[36902]: audit 2022-04-22T19:31:09.119381+0000 mgr.y (mgr.34107) 4298 : audit [DBG] from='client.49822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:10 smithi121 bash[36902]: audit 2022-04-22T19:31:09.458550+0000 mgr.y (mgr.34107) 4299 : audit [DBG] from='client.39894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:10 smithi121 bash[36902]: audit 2022-04-22T19:31:09.882574+0000 mon.a (mon.0) 7088 : audit [DBG] from='client.? 172.21.15.115:0/1394449229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:11 smithi115 bash[45495]: cluster 2022-04-22T19:31:09.641111+0000 mgr.y (mgr.34107) 4300 : cluster [DBG] pgmap v5695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:11 smithi115 bash[45495]: audit 2022-04-22T19:31:10.458896+0000 mon.a (mon.0) 7089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:11 smithi115 bash[46934]: cluster 2022-04-22T19:31:09.641111+0000 mgr.y (mgr.34107) 4300 : cluster [DBG] pgmap v5695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:11 smithi115 bash[46934]: audit 2022-04-22T19:31:10.458896+0000 mon.a (mon.0) 7089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:11 smithi121 bash[36902]: cluster 2022-04-22T19:31:09.641111+0000 mgr.y (mgr.34107) 4300 : cluster [DBG] pgmap v5695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:11 smithi121 bash[36902]: audit 2022-04-22T19:31:10.458896+0000 mon.a (mon.0) 7089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:12.095Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:13 smithi115 bash[45495]: cluster 2022-04-22T19:31:11.642301+0000 mgr.y (mgr.34107) 4301 : cluster [DBG] pgmap v5696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:13 smithi115 bash[46934]: cluster 2022-04-22T19:31:11.642301+0000 mgr.y (mgr.34107) 4301 : cluster [DBG] pgmap v5696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:13 smithi121 bash[36902]: cluster 2022-04-22T19:31:11.642301+0000 mgr.y (mgr.34107) 4301 : cluster [DBG] pgmap v5696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:13.966Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:14.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:14 smithi115 bash[45495]: audit 2022-04-22T19:31:13.939257+0000 mon.a (mon.0) 7090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:14.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:14 smithi115 bash[46934]: audit 2022-04-22T19:31:13.939257+0000 mon.a (mon.0) 7090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:14 smithi121 bash[36902]: audit 2022-04-22T19:31:13.939257+0000 mon.a (mon.0) 7090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:15 smithi121 bash[36902]: cluster 2022-04-22T19:31:13.643628+0000 mgr.y (mgr.34107) 4302 : cluster [DBG] pgmap v5697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:15 smithi121 bash[36902]: audit 2022-04-22T19:31:14.701319+0000 mon.a (mon.0) 7091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:31:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:15 smithi121 bash[36902]: audit 2022-04-22T19:31:14.701979+0000 mon.a (mon.0) 7092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:31:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[46934]: cluster 2022-04-22T19:31:13.643628+0000 mgr.y (mgr.34107) 4302 : cluster [DBG] pgmap v5697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[46934]: audit 2022-04-22T19:31:14.701319+0000 mon.a (mon.0) 7091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:31:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[46934]: audit 2022-04-22T19:31:14.701979+0000 mon.a (mon.0) 7092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:31:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[45495]: cluster 2022-04-22T19:31:13.643628+0000 mgr.y (mgr.34107) 4302 : cluster [DBG] pgmap v5697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[45495]: audit 2022-04-22T19:31:14.701319+0000 mon.a (mon.0) 7091 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:31:15.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:15 smithi115 bash[45495]: audit 2022-04-22T19:31:14.701979+0000 mon.a (mon.0) 7092 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:31:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:16 smithi121 bash[36902]: audit 2022-04-22T19:31:15.462068+0000 mon.a (mon.0) 7093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:16.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:16 smithi115 bash[45495]: audit 2022-04-22T19:31:15.462068+0000 mon.a (mon.0) 7093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:16 smithi115 bash[46934]: audit 2022-04-22T19:31:15.462068+0000 mon.a (mon.0) 7093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:17 smithi121 bash[36902]: cluster 2022-04-22T19:31:15.644683+0000 mgr.y (mgr.34107) 4303 : cluster [DBG] pgmap v5698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:17 smithi115 bash[45495]: cluster 2022-04-22T19:31:15.644683+0000 mgr.y (mgr.34107) 4303 : cluster [DBG] pgmap v5698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:17 smithi115 bash[46934]: cluster 2022-04-22T19:31:15.644683+0000 mgr.y (mgr.34107) 4303 : cluster [DBG] pgmap v5698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:19 smithi121 bash[36902]: cluster 2022-04-22T19:31:17.645757+0000 mgr.y (mgr.34107) 4304 : cluster [DBG] pgmap v5699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:19 smithi121 bash[36902]: audit 2022-04-22T19:31:18.942790+0000 mon.a (mon.0) 7094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:19 smithi115 bash[45495]: cluster 2022-04-22T19:31:17.645757+0000 mgr.y (mgr.34107) 4304 : cluster [DBG] pgmap v5699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:19 smithi115 bash[45495]: audit 2022-04-22T19:31:18.942790+0000 mon.a (mon.0) 7094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:19 smithi115 bash[46934]: cluster 2022-04-22T19:31:17.645757+0000 mgr.y (mgr.34107) 4304 : cluster [DBG] pgmap v5699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:19 smithi115 bash[46934]: audit 2022-04-22T19:31:18.942790+0000 mon.a (mon.0) 7094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:21 smithi121 bash[36902]: cluster 2022-04-22T19:31:19.646844+0000 mgr.y (mgr.34107) 4305 : cluster [DBG] pgmap v5700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:21 smithi121 bash[36902]: audit 2022-04-22T19:31:20.465619+0000 mon.a (mon.0) 7095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:21 smithi115 bash[45495]: cluster 2022-04-22T19:31:19.646844+0000 mgr.y (mgr.34107) 4305 : cluster [DBG] pgmap v5700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:21 smithi115 bash[45495]: audit 2022-04-22T19:31:20.465619+0000 mon.a (mon.0) 7095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:21 smithi115 bash[46934]: cluster 2022-04-22T19:31:19.646844+0000 mgr.y (mgr.34107) 4305 : cluster [DBG] pgmap v5700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:21 smithi115 bash[46934]: audit 2022-04-22T19:31:20.465619+0000 mon.a (mon.0) 7095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:22.096Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:23 smithi121 bash[36902]: cluster 2022-04-22T19:31:21.647945+0000 mgr.y (mgr.34107) 4306 : cluster [DBG] pgmap v5701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:23 smithi115 bash[45495]: cluster 2022-04-22T19:31:21.647945+0000 mgr.y (mgr.34107) 4306 : cluster [DBG] pgmap v5701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:23 smithi115 bash[46934]: cluster 2022-04-22T19:31:21.647945+0000 mgr.y (mgr.34107) 4306 : cluster [DBG] pgmap v5701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:23.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:24 smithi121 bash[36902]: audit 2022-04-22T19:31:23.946444+0000 mon.a (mon.0) 7096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:24.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:24 smithi115 bash[45495]: audit 2022-04-22T19:31:23.946444+0000 mon.a (mon.0) 7096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:24.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:24 smithi115 bash[46934]: audit 2022-04-22T19:31:23.946444+0000 mon.a (mon.0) 7096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:25 smithi121 bash[36902]: cluster 2022-04-22T19:31:23.649119+0000 mgr.y (mgr.34107) 4307 : cluster [DBG] pgmap v5702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:25.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:25 smithi115 bash[45495]: cluster 2022-04-22T19:31:23.649119+0000 mgr.y (mgr.34107) 4307 : cluster [DBG] pgmap v5702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:25 smithi115 bash[46934]: cluster 2022-04-22T19:31:23.649119+0000 mgr.y (mgr.34107) 4307 : cluster [DBG] pgmap v5702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:26 smithi121 bash[36902]: audit 2022-04-22T19:31:25.468609+0000 mon.a (mon.0) 7097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:26 smithi115 bash[45495]: audit 2022-04-22T19:31:25.468609+0000 mon.a (mon.0) 7097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:26 smithi115 bash[46934]: audit 2022-04-22T19:31:25.468609+0000 mon.a (mon.0) 7097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:27 smithi121 bash[36902]: cluster 2022-04-22T19:31:25.650299+0000 mgr.y (mgr.34107) 4308 : cluster [DBG] pgmap v5703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:27 smithi115 bash[45495]: cluster 2022-04-22T19:31:25.650299+0000 mgr.y (mgr.34107) 4308 : cluster [DBG] pgmap v5703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:27 smithi115 bash[46934]: cluster 2022-04-22T19:31:25.650299+0000 mgr.y (mgr.34107) 4308 : cluster [DBG] pgmap v5703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:29 smithi121 bash[36902]: cluster 2022-04-22T19:31:27.651568+0000 mgr.y (mgr.34107) 4309 : cluster [DBG] pgmap v5704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:29 smithi121 bash[36902]: audit 2022-04-22T19:31:28.950391+0000 mon.a (mon.0) 7098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:29 smithi115 bash[45495]: cluster 2022-04-22T19:31:27.651568+0000 mgr.y (mgr.34107) 4309 : cluster [DBG] pgmap v5704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:29 smithi115 bash[45495]: audit 2022-04-22T19:31:28.950391+0000 mon.a (mon.0) 7098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:29 smithi115 bash[46934]: cluster 2022-04-22T19:31:27.651568+0000 mgr.y (mgr.34107) 4309 : cluster [DBG] pgmap v5704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:29 smithi115 bash[46934]: audit 2022-04-22T19:31:28.950391+0000 mon.a (mon.0) 7098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:31 smithi121 bash[36902]: cluster 2022-04-22T19:31:29.652716+0000 mgr.y (mgr.34107) 4310 : cluster [DBG] pgmap v5705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:31 smithi121 bash[36902]: audit 2022-04-22T19:31:30.472094+0000 mon.a (mon.0) 7099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:31 smithi115 bash[45495]: cluster 2022-04-22T19:31:29.652716+0000 mgr.y (mgr.34107) 4310 : cluster [DBG] pgmap v5705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:31 smithi115 bash[45495]: audit 2022-04-22T19:31:30.472094+0000 mon.a (mon.0) 7099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:31 smithi115 bash[46934]: cluster 2022-04-22T19:31:29.652716+0000 mgr.y (mgr.34107) 4310 : cluster [DBG] pgmap v5705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:31 smithi115 bash[46934]: audit 2022-04-22T19:31:30.472094+0000 mon.a (mon.0) 7099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:32.096Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:33 smithi121 bash[36902]: cluster 2022-04-22T19:31:31.653934+0000 mgr.y (mgr.34107) 4311 : cluster [DBG] pgmap v5706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:33 smithi115 bash[45495]: cluster 2022-04-22T19:31:31.653934+0000 mgr.y (mgr.34107) 4311 : cluster [DBG] pgmap v5706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:33.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:33 smithi115 bash[46934]: cluster 2022-04-22T19:31:31.653934+0000 mgr.y (mgr.34107) 4311 : cluster [DBG] pgmap v5706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:33.966Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:34 smithi121 bash[36902]: audit 2022-04-22T19:31:33.954308+0000 mon.a (mon.0) 7100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:34.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:34 smithi115 bash[45495]: audit 2022-04-22T19:31:33.954308+0000 mon.a (mon.0) 7100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:34 smithi115 bash[46934]: audit 2022-04-22T19:31:33.954308+0000 mon.a (mon.0) 7100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:35 smithi121 bash[36902]: cluster 2022-04-22T19:31:33.655121+0000 mgr.y (mgr.34107) 4312 : cluster [DBG] pgmap v5707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:35 smithi115 bash[45495]: cluster 2022-04-22T19:31:33.655121+0000 mgr.y (mgr.34107) 4312 : cluster [DBG] pgmap v5707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:35 smithi115 bash[46934]: cluster 2022-04-22T19:31:33.655121+0000 mgr.y (mgr.34107) 4312 : cluster [DBG] pgmap v5707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:36 smithi121 bash[36902]: audit 2022-04-22T19:31:35.475249+0000 mon.a (mon.0) 7101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:36.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:36 smithi115 bash[45495]: audit 2022-04-22T19:31:35.475249+0000 mon.a (mon.0) 7101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:36.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:36 smithi115 bash[46934]: audit 2022-04-22T19:31:35.475249+0000 mon.a (mon.0) 7101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:37 smithi121 bash[36902]: cluster 2022-04-22T19:31:35.656227+0000 mgr.y (mgr.34107) 4313 : cluster [DBG] pgmap v5708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:37 smithi115 bash[45495]: cluster 2022-04-22T19:31:35.656227+0000 mgr.y (mgr.34107) 4313 : cluster [DBG] pgmap v5708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:37 smithi115 bash[46934]: cluster 2022-04-22T19:31:35.656227+0000 mgr.y (mgr.34107) 4313 : cluster [DBG] pgmap v5708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:39 smithi121 bash[36902]: cluster 2022-04-22T19:31:37.657364+0000 mgr.y (mgr.34107) 4314 : cluster [DBG] pgmap v5709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:39 smithi121 bash[36902]: audit 2022-04-22T19:31:38.958062+0000 mon.a (mon.0) 7102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:39 smithi115 bash[45495]: cluster 2022-04-22T19:31:37.657364+0000 mgr.y (mgr.34107) 4314 : cluster [DBG] pgmap v5709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:39 smithi115 bash[45495]: audit 2022-04-22T19:31:38.958062+0000 mon.a (mon.0) 7102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:39 smithi115 bash[46934]: cluster 2022-04-22T19:31:37.657364+0000 mgr.y (mgr.34107) 4314 : cluster [DBG] pgmap v5709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:39 smithi115 bash[46934]: audit 2022-04-22T19:31:38.958062+0000 mon.a (mon.0) 7102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:40.270 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:31:40.613 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (114m) 45s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 110s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (116m) 45s ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (116m) 110s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 110s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 45s ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 45s ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 110s ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:31:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 45s ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (115m) 45s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (115m) 110s ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 45s ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 45s ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 45s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (119m) 45s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (119m) 110s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (118m) 110s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (117m) 110s ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (117m) 110s ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:31:40.615 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (115m) 110s ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:31:41.024 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:31:41.024 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:31:41.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:31:41.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:41.025 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:31:41.028 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:31:41.028 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:41.028 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:31:41.028 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:31:41.028 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:41.029 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:31:41.029 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:31:41.029 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:31:41.030 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:31:41.030 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:31:41.030 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:31:41.030 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:41 smithi121 bash[36902]: cluster 2022-04-22T19:31:39.658610+0000 mgr.y (mgr.34107) 4315 : cluster [DBG] pgmap v5710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:41 smithi121 bash[36902]: audit 2022-04-22T19:31:40.242537+0000 mgr.y (mgr.34107) 4316 : audit [DBG] from='client.39906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:41 smithi121 bash[36902]: audit 2022-04-22T19:31:40.478303+0000 mon.a (mon.0) 7103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:41 smithi121 bash[36902]: audit 2022-04-22T19:31:41.011946+0000 mon.a (mon.0) 7104 : audit [DBG] from='client.? 172.21.15.115:0/253101657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[45495]: cluster 2022-04-22T19:31:39.658610+0000 mgr.y (mgr.34107) 4315 : cluster [DBG] pgmap v5710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[45495]: audit 2022-04-22T19:31:40.242537+0000 mgr.y (mgr.34107) 4316 : audit [DBG] from='client.39906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[45495]: audit 2022-04-22T19:31:40.478303+0000 mon.a (mon.0) 7103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:41.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[45495]: audit 2022-04-22T19:31:41.011946+0000 mon.a (mon.0) 7104 : audit [DBG] from='client.? 172.21.15.115:0/253101657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[46934]: cluster 2022-04-22T19:31:39.658610+0000 mgr.y (mgr.34107) 4315 : cluster [DBG] pgmap v5710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[46934]: audit 2022-04-22T19:31:40.242537+0000 mgr.y (mgr.34107) 4316 : audit [DBG] from='client.39906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[46934]: audit 2022-04-22T19:31:40.478303+0000 mon.a (mon.0) 7103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:41 smithi115 bash[46934]: audit 2022-04-22T19:31:41.011946+0000 mon.a (mon.0) 7104 : audit [DBG] from='client.? 172.21.15.115:0/253101657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:31:42.391 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:42.097Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:42 smithi121 bash[36902]: audit 2022-04-22T19:31:40.584879+0000 mgr.y (mgr.34107) 4317 : audit [DBG] from='client.49846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:42.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:42 smithi115 bash[45495]: audit 2022-04-22T19:31:40.584879+0000 mgr.y (mgr.34107) 4317 : audit [DBG] from='client.49846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:42 smithi115 bash[46934]: audit 2022-04-22T19:31:40.584879+0000 mgr.y (mgr.34107) 4317 : audit [DBG] from='client.49846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:31:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:43 smithi121 bash[36902]: cluster 2022-04-22T19:31:41.659999+0000 mgr.y (mgr.34107) 4318 : cluster [DBG] pgmap v5711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:43.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:43 smithi115 bash[45495]: cluster 2022-04-22T19:31:41.659999+0000 mgr.y (mgr.34107) 4318 : cluster [DBG] pgmap v5711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:43.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:43 smithi115 bash[46934]: cluster 2022-04-22T19:31:41.659999+0000 mgr.y (mgr.34107) 4318 : cluster [DBG] pgmap v5711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:43.967Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:44 smithi121 bash[36902]: audit 2022-04-22T19:31:43.961638+0000 mon.a (mon.0) 7105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:44.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:44 smithi115 bash[46934]: audit 2022-04-22T19:31:43.961638+0000 mon.a (mon.0) 7105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:44 smithi115 bash[45495]: audit 2022-04-22T19:31:43.961638+0000 mon.a (mon.0) 7105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:45 smithi121 bash[36902]: cluster 2022-04-22T19:31:43.661191+0000 mgr.y (mgr.34107) 4319 : cluster [DBG] pgmap v5712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:45 smithi115 bash[45495]: cluster 2022-04-22T19:31:43.661191+0000 mgr.y (mgr.34107) 4319 : cluster [DBG] pgmap v5712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:45 smithi115 bash[46934]: cluster 2022-04-22T19:31:43.661191+0000 mgr.y (mgr.34107) 4319 : cluster [DBG] pgmap v5712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:46 smithi121 bash[36902]: audit 2022-04-22T19:31:45.481162+0000 mon.a (mon.0) 7106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:46 smithi115 bash[46934]: audit 2022-04-22T19:31:45.481162+0000 mon.a (mon.0) 7106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:46.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:46 smithi115 bash[45495]: audit 2022-04-22T19:31:45.481162+0000 mon.a (mon.0) 7106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:47 smithi121 bash[36902]: cluster 2022-04-22T19:31:45.662423+0000 mgr.y (mgr.34107) 4320 : cluster [DBG] pgmap v5713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:47 smithi115 bash[45495]: cluster 2022-04-22T19:31:45.662423+0000 mgr.y (mgr.34107) 4320 : cluster [DBG] pgmap v5713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:47 smithi115 bash[46934]: cluster 2022-04-22T19:31:45.662423+0000 mgr.y (mgr.34107) 4320 : cluster [DBG] pgmap v5713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:49 smithi121 bash[36902]: cluster 2022-04-22T19:31:47.663681+0000 mgr.y (mgr.34107) 4321 : cluster [DBG] pgmap v5714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:49 smithi121 bash[36902]: audit 2022-04-22T19:31:48.965249+0000 mon.a (mon.0) 7107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:49 smithi115 bash[45495]: cluster 2022-04-22T19:31:47.663681+0000 mgr.y (mgr.34107) 4321 : cluster [DBG] pgmap v5714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:49 smithi115 bash[45495]: audit 2022-04-22T19:31:48.965249+0000 mon.a (mon.0) 7107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:49 smithi115 bash[46934]: cluster 2022-04-22T19:31:47.663681+0000 mgr.y (mgr.34107) 4321 : cluster [DBG] pgmap v5714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:49 smithi115 bash[46934]: audit 2022-04-22T19:31:48.965249+0000 mon.a (mon.0) 7107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:51 smithi121 bash[36902]: cluster 2022-04-22T19:31:49.664778+0000 mgr.y (mgr.34107) 4322 : cluster [DBG] pgmap v5715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:51 smithi121 bash[36902]: audit 2022-04-22T19:31:50.484146+0000 mon.a (mon.0) 7108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:51 smithi115 bash[45495]: cluster 2022-04-22T19:31:49.664778+0000 mgr.y (mgr.34107) 4322 : cluster [DBG] pgmap v5715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:51 smithi115 bash[45495]: audit 2022-04-22T19:31:50.484146+0000 mon.a (mon.0) 7108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:51 smithi115 bash[46934]: cluster 2022-04-22T19:31:49.664778+0000 mgr.y (mgr.34107) 4322 : cluster [DBG] pgmap v5715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:51 smithi115 bash[46934]: audit 2022-04-22T19:31:50.484146+0000 mon.a (mon.0) 7108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:52.098Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:53 smithi121 bash[36902]: cluster 2022-04-22T19:31:51.665881+0000 mgr.y (mgr.34107) 4323 : cluster [DBG] pgmap v5716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:53.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:53 smithi115 bash[45495]: cluster 2022-04-22T19:31:51.665881+0000 mgr.y (mgr.34107) 4323 : cluster [DBG] pgmap v5716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:53 smithi115 bash[46934]: cluster 2022-04-22T19:31:51.665881+0000 mgr.y (mgr.34107) 4323 : cluster [DBG] pgmap v5716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:31:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:31:53.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:31:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:54 smithi121 bash[36902]: audit 2022-04-22T19:31:53.969111+0000 mon.a (mon.0) 7109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:54.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:54 smithi115 bash[45495]: audit 2022-04-22T19:31:53.969111+0000 mon.a (mon.0) 7109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:54 smithi115 bash[46934]: audit 2022-04-22T19:31:53.969111+0000 mon.a (mon.0) 7109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:55 smithi121 bash[36902]: cluster 2022-04-22T19:31:53.667019+0000 mgr.y (mgr.34107) 4324 : cluster [DBG] pgmap v5717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:55 smithi121 bash[36902]: audit 2022-04-22T19:31:55.109581+0000 mon.a (mon.0) 7110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:31:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:55 smithi121 bash[36902]: audit 2022-04-22T19:31:55.113485+0000 mon.a (mon.0) 7111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:31:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:55 smithi121 bash[36902]: audit 2022-04-22T19:31:55.115376+0000 mon.a (mon.0) 7112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:31:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[46934]: cluster 2022-04-22T19:31:53.667019+0000 mgr.y (mgr.34107) 4324 : cluster [DBG] pgmap v5717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[46934]: audit 2022-04-22T19:31:55.109581+0000 mon.a (mon.0) 7110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:31:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[46934]: audit 2022-04-22T19:31:55.113485+0000 mon.a (mon.0) 7111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:31:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[46934]: audit 2022-04-22T19:31:55.115376+0000 mon.a (mon.0) 7112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:31:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[45495]: cluster 2022-04-22T19:31:53.667019+0000 mgr.y (mgr.34107) 4324 : cluster [DBG] pgmap v5717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[45495]: audit 2022-04-22T19:31:55.109581+0000 mon.a (mon.0) 7110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:31:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[45495]: audit 2022-04-22T19:31:55.113485+0000 mon.a (mon.0) 7111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:31:55.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:55 smithi115 bash[45495]: audit 2022-04-22T19:31:55.115376+0000 mon.a (mon.0) 7112 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:31:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:56 smithi121 bash[36902]: audit 2022-04-22T19:31:55.478305+0000 mon.a (mon.0) 7113 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:31:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:56 smithi121 bash[36902]: audit 2022-04-22T19:31:55.486631+0000 mon.a (mon.0) 7114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:56 smithi115 bash[45495]: audit 2022-04-22T19:31:55.478305+0000 mon.a (mon.0) 7113 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:31:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:56 smithi115 bash[45495]: audit 2022-04-22T19:31:55.486631+0000 mon.a (mon.0) 7114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:56.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:56 smithi115 bash[46934]: audit 2022-04-22T19:31:55.478305+0000 mon.a (mon.0) 7113 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:31:56.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:56 smithi115 bash[46934]: audit 2022-04-22T19:31:55.486631+0000 mon.a (mon.0) 7114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:31:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:57 smithi121 bash[36902]: cluster 2022-04-22T19:31:55.668100+0000 mgr.y (mgr.34107) 4325 : cluster [DBG] pgmap v5718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:57 smithi115 bash[45495]: cluster 2022-04-22T19:31:55.668100+0000 mgr.y (mgr.34107) 4325 : cluster [DBG] pgmap v5718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:57 smithi115 bash[46934]: cluster 2022-04-22T19:31:55.668100+0000 mgr.y (mgr.34107) 4325 : cluster [DBG] pgmap v5718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:59 smithi121 bash[36902]: cluster 2022-04-22T19:31:57.669206+0000 mgr.y (mgr.34107) 4326 : cluster [DBG] pgmap v5719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:31:59 smithi121 bash[36902]: audit 2022-04-22T19:31:58.972792+0000 mon.a (mon.0) 7115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:59 smithi115 bash[45495]: cluster 2022-04-22T19:31:57.669206+0000 mgr.y (mgr.34107) 4326 : cluster [DBG] pgmap v5719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:31:59 smithi115 bash[45495]: audit 2022-04-22T19:31:58.972792+0000 mon.a (mon.0) 7115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:31:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:59 smithi115 bash[46934]: cluster 2022-04-22T19:31:57.669206+0000 mgr.y (mgr.34107) 4326 : cluster [DBG] pgmap v5719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:31:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:31:59 smithi115 bash[46934]: audit 2022-04-22T19:31:58.972792+0000 mon.a (mon.0) 7115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:00 smithi121 bash[36902]: audit 2022-04-22T19:32:00.488987+0000 mon.a (mon.0) 7116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:00 smithi115 bash[45495]: audit 2022-04-22T19:32:00.488987+0000 mon.a (mon.0) 7116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:00.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:00 smithi115 bash[46934]: audit 2022-04-22T19:32:00.488987+0000 mon.a (mon.0) 7116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:01 smithi121 bash[36902]: cluster 2022-04-22T19:31:59.670280+0000 mgr.y (mgr.34107) 4327 : cluster [DBG] pgmap v5720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:01 smithi115 bash[45495]: cluster 2022-04-22T19:31:59.670280+0000 mgr.y (mgr.34107) 4327 : cluster [DBG] pgmap v5720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:01 smithi115 bash[46934]: cluster 2022-04-22T19:31:59.670280+0000 mgr.y (mgr.34107) 4327 : cluster [DBG] pgmap v5720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:02.097Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:03.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:03 smithi115 bash[45495]: cluster 2022-04-22T19:32:01.671410+0000 mgr.y (mgr.34107) 4328 : cluster [DBG] pgmap v5721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:03 smithi115 bash[46934]: cluster 2022-04-22T19:32:01.671410+0000 mgr.y (mgr.34107) 4328 : cluster [DBG] pgmap v5721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:03 smithi121 bash[36902]: cluster 2022-04-22T19:32:01.671410+0000 mgr.y (mgr.34107) 4328 : cluster [DBG] pgmap v5721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:03.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:04.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:04 smithi115 bash[45495]: audit 2022-04-22T19:32:03.976177+0000 mon.a (mon.0) 7117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:04 smithi115 bash[46934]: audit 2022-04-22T19:32:03.976177+0000 mon.a (mon.0) 7117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:04 smithi121 bash[36902]: audit 2022-04-22T19:32:03.976177+0000 mon.a (mon.0) 7117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:05.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:05 smithi115 bash[45495]: cluster 2022-04-22T19:32:03.672592+0000 mgr.y (mgr.34107) 4329 : cluster [DBG] pgmap v5722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:05.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:05 smithi115 bash[45495]: audit 2022-04-22T19:32:05.492298+0000 mon.a (mon.0) 7118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:05 smithi115 bash[46934]: cluster 2022-04-22T19:32:03.672592+0000 mgr.y (mgr.34107) 4329 : cluster [DBG] pgmap v5722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:05 smithi115 bash[46934]: audit 2022-04-22T19:32:05.492298+0000 mon.a (mon.0) 7118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:06.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:05 smithi121 bash[36902]: cluster 2022-04-22T19:32:03.672592+0000 mgr.y (mgr.34107) 4329 : cluster [DBG] pgmap v5722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:06.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:05 smithi121 bash[36902]: audit 2022-04-22T19:32:05.492298+0000 mon.a (mon.0) 7118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:06 smithi121 bash[36902]: cluster 2022-04-22T19:32:05.673710+0000 mgr.y (mgr.34107) 4330 : cluster [DBG] pgmap v5723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:07.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:06 smithi115 bash[45495]: cluster 2022-04-22T19:32:05.673710+0000 mgr.y (mgr.34107) 4330 : cluster [DBG] pgmap v5723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:06 smithi115 bash[46934]: cluster 2022-04-22T19:32:05.673710+0000 mgr.y (mgr.34107) 4330 : cluster [DBG] pgmap v5723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:08 smithi121 bash[36902]: cluster 2022-04-22T19:32:07.674828+0000 mgr.y (mgr.34107) 4331 : cluster [DBG] pgmap v5724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:09.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:08 smithi115 bash[45495]: cluster 2022-04-22T19:32:07.674828+0000 mgr.y (mgr.34107) 4331 : cluster [DBG] pgmap v5724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:08 smithi115 bash[46934]: cluster 2022-04-22T19:32:07.674828+0000 mgr.y (mgr.34107) 4331 : cluster [DBG] pgmap v5724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:09 smithi121 bash[36902]: audit 2022-04-22T19:32:08.979929+0000 mon.a (mon.0) 7119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:10.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:09 smithi115 bash[45495]: audit 2022-04-22T19:32:08.979929+0000 mon.a (mon.0) 7119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:10.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:09 smithi115 bash[46934]: audit 2022-04-22T19:32:08.979929+0000 mon.a (mon.0) 7119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:10 smithi121 bash[36902]: cluster 2022-04-22T19:32:09.675919+0000 mgr.y (mgr.34107) 4332 : cluster [DBG] pgmap v5725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:10 smithi121 bash[36902]: audit 2022-04-22T19:32:10.495383+0000 mon.a (mon.0) 7120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:11.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:10 smithi115 bash[45495]: cluster 2022-04-22T19:32:09.675919+0000 mgr.y (mgr.34107) 4332 : cluster [DBG] pgmap v5725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:11.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:10 smithi115 bash[45495]: audit 2022-04-22T19:32:10.495383+0000 mon.a (mon.0) 7120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:11.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:10 smithi115 bash[46934]: cluster 2022-04-22T19:32:09.675919+0000 mgr.y (mgr.34107) 4332 : cluster [DBG] pgmap v5725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:11.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:10 smithi115 bash[46934]: audit 2022-04-22T19:32:10.495383+0000 mon.a (mon.0) 7120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:11.395 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:32:11.731 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:32:11.732 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (115m) 76s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:32:11.732 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:32:11.732 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (117m) 76s ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:32:11.732 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (116m) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:32:11.733 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:32:11.733 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 76s ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:32:11.733 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 76s ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:32:11.733 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:32:11.734 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 76s ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:32:11.734 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (116m) 76s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:32:11.734 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (116m) 2m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:32:11.734 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 76s ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:32:11.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 76s ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:32:11.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 76s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:32:11.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 76s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:32:11.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (119m) 2m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:32:11.735 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (119m) 2m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:32:11.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (118m) 2m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:32:11.736 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (117m) 2m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:32:11.736 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (115m) 2m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:32:12.131 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:32:12.131 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:32:12.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:32:12.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:12.133 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:32:12.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:32:12.133 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:32:12.133 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:32:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:11 smithi121 bash[36902]: audit 2022-04-22T19:32:11.368375+0000 mgr.y (mgr.34107) 4333 : audit [DBG] from='client.39924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:12.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:11 smithi115 bash[46934]: audit 2022-04-22T19:32:11.368375+0000 mgr.y (mgr.34107) 4333 : audit [DBG] from='client.39924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:12.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:11 smithi115 bash[45495]: audit 2022-04-22T19:32:11.368375+0000 mgr.y (mgr.34107) 4333 : audit [DBG] from='client.39924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:12.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:12.097Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:12 smithi121 bash[36902]: cluster 2022-04-22T19:32:11.676925+0000 mgr.y (mgr.34107) 4334 : cluster [DBG] pgmap v5726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:12 smithi121 bash[36902]: audit 2022-04-22T19:32:11.706497+0000 mgr.y (mgr.34107) 4335 : audit [DBG] from='client.49864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:12 smithi121 bash[36902]: audit 2022-04-22T19:32:12.120748+0000 mon.a (mon.0) 7121 : audit [DBG] from='client.? 172.21.15.115:0/2602917619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[45495]: cluster 2022-04-22T19:32:11.676925+0000 mgr.y (mgr.34107) 4334 : cluster [DBG] pgmap v5726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[45495]: audit 2022-04-22T19:32:11.706497+0000 mgr.y (mgr.34107) 4335 : audit [DBG] from='client.49864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[45495]: audit 2022-04-22T19:32:12.120748+0000 mon.a (mon.0) 7121 : audit [DBG] from='client.? 172.21.15.115:0/2602917619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[46934]: cluster 2022-04-22T19:32:11.676925+0000 mgr.y (mgr.34107) 4334 : cluster [DBG] pgmap v5726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[46934]: audit 2022-04-22T19:32:11.706497+0000 mgr.y (mgr.34107) 4335 : audit [DBG] from='client.49864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:12 smithi115 bash[46934]: audit 2022-04-22T19:32:12.120748+0000 mon.a (mon.0) 7121 : audit [DBG] from='client.? 172.21.15.115:0/2602917619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:13.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:14 smithi121 bash[36902]: cluster 2022-04-22T19:32:13.678075+0000 mgr.y (mgr.34107) 4336 : cluster [DBG] pgmap v5727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:14 smithi121 bash[36902]: audit 2022-04-22T19:32:13.983684+0000 mon.a (mon.0) 7122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:14 smithi121 bash[36902]: audit 2022-04-22T19:32:14.712822+0000 mon.a (mon.0) 7123 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:32:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:14 smithi121 bash[36902]: audit 2022-04-22T19:32:14.714576+0000 mon.a (mon.0) 7124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[45495]: cluster 2022-04-22T19:32:13.678075+0000 mgr.y (mgr.34107) 4336 : cluster [DBG] pgmap v5727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[45495]: audit 2022-04-22T19:32:13.983684+0000 mon.a (mon.0) 7122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[45495]: audit 2022-04-22T19:32:14.712822+0000 mon.a (mon.0) 7123 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[45495]: audit 2022-04-22T19:32:14.714576+0000 mon.a (mon.0) 7124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[46934]: cluster 2022-04-22T19:32:13.678075+0000 mgr.y (mgr.34107) 4336 : cluster [DBG] pgmap v5727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[46934]: audit 2022-04-22T19:32:13.983684+0000 mon.a (mon.0) 7122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[46934]: audit 2022-04-22T19:32:14.712822+0000 mon.a (mon.0) 7123 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:32:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:14 smithi115 bash[46934]: audit 2022-04-22T19:32:14.714576+0000 mon.a (mon.0) 7124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:32:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:15 smithi121 bash[36902]: audit 2022-04-22T19:32:15.498465+0000 mon.a (mon.0) 7125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:16.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:15 smithi115 bash[45495]: audit 2022-04-22T19:32:15.498465+0000 mon.a (mon.0) 7125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:16.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:15 smithi115 bash[46934]: audit 2022-04-22T19:32:15.498465+0000 mon.a (mon.0) 7125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:16 smithi121 bash[36902]: cluster 2022-04-22T19:32:15.679147+0000 mgr.y (mgr.34107) 4337 : cluster [DBG] pgmap v5728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:17.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:16 smithi115 bash[45495]: cluster 2022-04-22T19:32:15.679147+0000 mgr.y (mgr.34107) 4337 : cluster [DBG] pgmap v5728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:16 smithi115 bash[46934]: cluster 2022-04-22T19:32:15.679147+0000 mgr.y (mgr.34107) 4337 : cluster [DBG] pgmap v5728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:18 smithi121 bash[36902]: cluster 2022-04-22T19:32:17.680197+0000 mgr.y (mgr.34107) 4338 : cluster [DBG] pgmap v5729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:18 smithi115 bash[45495]: cluster 2022-04-22T19:32:17.680197+0000 mgr.y (mgr.34107) 4338 : cluster [DBG] pgmap v5729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:18 smithi115 bash[46934]: cluster 2022-04-22T19:32:17.680197+0000 mgr.y (mgr.34107) 4338 : cluster [DBG] pgmap v5729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:19 smithi121 bash[36902]: audit 2022-04-22T19:32:18.987274+0000 mon.a (mon.0) 7126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:19 smithi115 bash[45495]: audit 2022-04-22T19:32:18.987274+0000 mon.a (mon.0) 7126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:19 smithi115 bash[46934]: audit 2022-04-22T19:32:18.987274+0000 mon.a (mon.0) 7126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:20 smithi121 bash[36902]: cluster 2022-04-22T19:32:19.681284+0000 mgr.y (mgr.34107) 4339 : cluster [DBG] pgmap v5730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:20 smithi121 bash[36902]: audit 2022-04-22T19:32:20.501607+0000 mon.a (mon.0) 7127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:20 smithi115 bash[45495]: cluster 2022-04-22T19:32:19.681284+0000 mgr.y (mgr.34107) 4339 : cluster [DBG] pgmap v5730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:20 smithi115 bash[45495]: audit 2022-04-22T19:32:20.501607+0000 mon.a (mon.0) 7127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:20 smithi115 bash[46934]: cluster 2022-04-22T19:32:19.681284+0000 mgr.y (mgr.34107) 4339 : cluster [DBG] pgmap v5730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:20 smithi115 bash[46934]: audit 2022-04-22T19:32:20.501607+0000 mon.a (mon.0) 7127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:22.099Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:22 smithi121 bash[36902]: cluster 2022-04-22T19:32:21.682350+0000 mgr.y (mgr.34107) 4340 : cluster [DBG] pgmap v5731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:22 smithi115 bash[45495]: cluster 2022-04-22T19:32:21.682350+0000 mgr.y (mgr.34107) 4340 : cluster [DBG] pgmap v5731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:22 smithi115 bash[46934]: cluster 2022-04-22T19:32:21.682350+0000 mgr.y (mgr.34107) 4340 : cluster [DBG] pgmap v5731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:23.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:24 smithi121 bash[36902]: cluster 2022-04-22T19:32:23.683507+0000 mgr.y (mgr.34107) 4341 : cluster [DBG] pgmap v5732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:24 smithi121 bash[36902]: audit 2022-04-22T19:32:23.990940+0000 mon.a (mon.0) 7128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:25.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:24 smithi115 bash[45495]: cluster 2022-04-22T19:32:23.683507+0000 mgr.y (mgr.34107) 4341 : cluster [DBG] pgmap v5732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:25.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:24 smithi115 bash[45495]: audit 2022-04-22T19:32:23.990940+0000 mon.a (mon.0) 7128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:24 smithi115 bash[46934]: cluster 2022-04-22T19:32:23.683507+0000 mgr.y (mgr.34107) 4341 : cluster [DBG] pgmap v5732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:25.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:24 smithi115 bash[46934]: audit 2022-04-22T19:32:23.990940+0000 mon.a (mon.0) 7128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:25 smithi121 bash[36902]: audit 2022-04-22T19:32:25.504805+0000 mon.a (mon.0) 7129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:25 smithi115 bash[45495]: audit 2022-04-22T19:32:25.504805+0000 mon.a (mon.0) 7129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:26.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:25 smithi115 bash[46934]: audit 2022-04-22T19:32:25.504805+0000 mon.a (mon.0) 7129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:26 smithi121 bash[36902]: cluster 2022-04-22T19:32:25.684555+0000 mgr.y (mgr.34107) 4342 : cluster [DBG] pgmap v5733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:27.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:26 smithi115 bash[45495]: cluster 2022-04-22T19:32:25.684555+0000 mgr.y (mgr.34107) 4342 : cluster [DBG] pgmap v5733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:27.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:26 smithi115 bash[46934]: cluster 2022-04-22T19:32:25.684555+0000 mgr.y (mgr.34107) 4342 : cluster [DBG] pgmap v5733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:29 smithi121 bash[36902]: cluster 2022-04-22T19:32:27.685669+0000 mgr.y (mgr.34107) 4343 : cluster [DBG] pgmap v5734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:29.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:28 smithi115 bash[45495]: cluster 2022-04-22T19:32:27.685669+0000 mgr.y (mgr.34107) 4343 : cluster [DBG] pgmap v5734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:29.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:28 smithi115 bash[46934]: cluster 2022-04-22T19:32:27.685669+0000 mgr.y (mgr.34107) 4343 : cluster [DBG] pgmap v5734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:30 smithi121 bash[36902]: audit 2022-04-22T19:32:28.997380+0000 mon.a (mon.0) 7130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:30 smithi115 bash[45495]: audit 2022-04-22T19:32:28.997380+0000 mon.a (mon.0) 7130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:30 smithi115 bash[46934]: audit 2022-04-22T19:32:28.997380+0000 mon.a (mon.0) 7130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:31 smithi115 bash[46934]: cluster 2022-04-22T19:32:29.686762+0000 mgr.y (mgr.34107) 4344 : cluster [DBG] pgmap v5735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:31.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:31 smithi115 bash[46934]: audit 2022-04-22T19:32:30.507870+0000 mon.a (mon.0) 7131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:31 smithi115 bash[45495]: cluster 2022-04-22T19:32:29.686762+0000 mgr.y (mgr.34107) 4344 : cluster [DBG] pgmap v5735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:31.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:31 smithi115 bash[45495]: audit 2022-04-22T19:32:30.507870+0000 mon.a (mon.0) 7131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:31 smithi121 bash[36902]: cluster 2022-04-22T19:32:29.686762+0000 mgr.y (mgr.34107) 4344 : cluster [DBG] pgmap v5735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:31 smithi121 bash[36902]: audit 2022-04-22T19:32:30.507870+0000 mon.a (mon.0) 7131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:32.100Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:33.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:33 smithi115 bash[45495]: cluster 2022-04-22T19:32:31.687902+0000 mgr.y (mgr.34107) 4345 : cluster [DBG] pgmap v5736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:33.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:33 smithi115 bash[46934]: cluster 2022-04-22T19:32:31.687902+0000 mgr.y (mgr.34107) 4345 : cluster [DBG] pgmap v5736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:33 smithi121 bash[36902]: cluster 2022-04-22T19:32:31.687902+0000 mgr.y (mgr.34107) 4345 : cluster [DBG] pgmap v5736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:34 smithi115 bash[45495]: audit 2022-04-22T19:32:33.998692+0000 mon.a (mon.0) 7132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:33.970Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:34.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:34 smithi115 bash[46934]: audit 2022-04-22T19:32:33.998692+0000 mon.a (mon.0) 7132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:34 smithi121 bash[36902]: audit 2022-04-22T19:32:33.998692+0000 mon.a (mon.0) 7132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:35.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:35 smithi115 bash[45495]: cluster 2022-04-22T19:32:33.689039+0000 mgr.y (mgr.34107) 4346 : cluster [DBG] pgmap v5737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:35.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:35 smithi115 bash[46934]: cluster 2022-04-22T19:32:33.689039+0000 mgr.y (mgr.34107) 4346 : cluster [DBG] pgmap v5737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:35 smithi121 bash[36902]: cluster 2022-04-22T19:32:33.689039+0000 mgr.y (mgr.34107) 4346 : cluster [DBG] pgmap v5737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:36.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:36 smithi115 bash[45495]: audit 2022-04-22T19:32:35.511180+0000 mon.a (mon.0) 7133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:36 smithi115 bash[46934]: audit 2022-04-22T19:32:35.511180+0000 mon.a (mon.0) 7133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:36 smithi121 bash[36902]: audit 2022-04-22T19:32:35.511180+0000 mon.a (mon.0) 7133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:37.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:37 smithi115 bash[45495]: cluster 2022-04-22T19:32:35.690099+0000 mgr.y (mgr.34107) 4347 : cluster [DBG] pgmap v5738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:37.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:37 smithi115 bash[46934]: cluster 2022-04-22T19:32:35.690099+0000 mgr.y (mgr.34107) 4347 : cluster [DBG] pgmap v5738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:37 smithi121 bash[36902]: cluster 2022-04-22T19:32:35.690099+0000 mgr.y (mgr.34107) 4347 : cluster [DBG] pgmap v5738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:39 smithi121 bash[36902]: cluster 2022-04-22T19:32:37.691159+0000 mgr.y (mgr.34107) 4348 : cluster [DBG] pgmap v5739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:39 smithi121 bash[36902]: audit 2022-04-22T19:32:39.002200+0000 mon.a (mon.0) 7134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:39 smithi115 bash[45495]: cluster 2022-04-22T19:32:37.691159+0000 mgr.y (mgr.34107) 4348 : cluster [DBG] pgmap v5739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:39 smithi115 bash[45495]: audit 2022-04-22T19:32:39.002200+0000 mon.a (mon.0) 7134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:39 smithi115 bash[46934]: cluster 2022-04-22T19:32:37.691159+0000 mgr.y (mgr.34107) 4348 : cluster [DBG] pgmap v5739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:39 smithi115 bash[46934]: audit 2022-04-22T19:32:39.002200+0000 mon.a (mon.0) 7134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:41 smithi121 bash[36902]: cluster 2022-04-22T19:32:39.692263+0000 mgr.y (mgr.34107) 4349 : cluster [DBG] pgmap v5740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:41 smithi121 bash[36902]: audit 2022-04-22T19:32:40.514399+0000 mon.a (mon.0) 7135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:41 smithi115 bash[45495]: cluster 2022-04-22T19:32:39.692263+0000 mgr.y (mgr.34107) 4349 : cluster [DBG] pgmap v5740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:41 smithi115 bash[45495]: audit 2022-04-22T19:32:40.514399+0000 mon.a (mon.0) 7135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:41 smithi115 bash[46934]: cluster 2022-04-22T19:32:39.692263+0000 mgr.y (mgr.34107) 4349 : cluster [DBG] pgmap v5740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:41 smithi115 bash[46934]: audit 2022-04-22T19:32:40.514399+0000 mon.a (mon.0) 7135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:42.504 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:32:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:42.101Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:42.842 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (115m) 108s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (117m) 108s ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (117m) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 108s ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 108s ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 108s ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:32:42.843 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (116m) 108s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (116m) 2m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 108s ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 108s ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 108s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 108s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (119m) 2m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (118m) 2m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (118m) 2m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:32:42.844 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (116m) 2m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:32:43.246 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:32:43.246 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:32:43.246 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:32:43.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:32:43.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:32:43.248 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:32:43.248 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:32:43.248 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:32:43.248 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:32:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:43 smithi121 bash[36902]: cluster 2022-04-22T19:32:41.693386+0000 mgr.y (mgr.34107) 4350 : cluster [DBG] pgmap v5741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:43 smithi121 bash[36902]: audit 2022-04-22T19:32:42.477983+0000 mgr.y (mgr.34107) 4351 : audit [DBG] from='client.49876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:43 smithi115 bash[45495]: cluster 2022-04-22T19:32:41.693386+0000 mgr.y (mgr.34107) 4350 : cluster [DBG] pgmap v5741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:43 smithi115 bash[45495]: audit 2022-04-22T19:32:42.477983+0000 mgr.y (mgr.34107) 4351 : audit [DBG] from='client.49876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:43 smithi115 bash[46934]: cluster 2022-04-22T19:32:41.693386+0000 mgr.y (mgr.34107) 4350 : cluster [DBG] pgmap v5741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:43 smithi115 bash[46934]: audit 2022-04-22T19:32:42.477983+0000 mgr.y (mgr.34107) 4351 : audit [DBG] from='client.49876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:43.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[45495]: audit 2022-04-22T19:32:42.813252+0000 mgr.y (mgr.34107) 4352 : audit [DBG] from='client.49882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[45495]: audit 2022-04-22T19:32:43.235294+0000 mon.c (mon.1) 196 : audit [DBG] from='client.? 172.21.15.115:0/3273270971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[45495]: audit 2022-04-22T19:32:44.006251+0000 mon.a (mon.0) 7136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[46934]: audit 2022-04-22T19:32:42.813252+0000 mgr.y (mgr.34107) 4352 : audit [DBG] from='client.49882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[46934]: audit 2022-04-22T19:32:43.235294+0000 mon.c (mon.1) 196 : audit [DBG] from='client.? 172.21.15.115:0/3273270971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:44.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:44 smithi115 bash[46934]: audit 2022-04-22T19:32:44.006251+0000 mon.a (mon.0) 7136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:44 smithi121 bash[36902]: audit 2022-04-22T19:32:42.813252+0000 mgr.y (mgr.34107) 4352 : audit [DBG] from='client.49882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:32:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:44 smithi121 bash[36902]: audit 2022-04-22T19:32:43.235294+0000 mon.c (mon.1) 196 : audit [DBG] from='client.? 172.21.15.115:0/3273270971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:32:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:44 smithi121 bash[36902]: audit 2022-04-22T19:32:44.006251+0000 mon.a (mon.0) 7136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:45 smithi121 bash[36902]: cluster 2022-04-22T19:32:43.694453+0000 mgr.y (mgr.34107) 4353 : cluster [DBG] pgmap v5742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:45 smithi115 bash[45495]: cluster 2022-04-22T19:32:43.694453+0000 mgr.y (mgr.34107) 4353 : cluster [DBG] pgmap v5742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:45 smithi115 bash[46934]: cluster 2022-04-22T19:32:43.694453+0000 mgr.y (mgr.34107) 4353 : cluster [DBG] pgmap v5742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:46 smithi121 bash[36902]: audit 2022-04-22T19:32:45.517711+0000 mon.a (mon.0) 7137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:46 smithi115 bash[46934]: audit 2022-04-22T19:32:45.517711+0000 mon.a (mon.0) 7137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:46 smithi115 bash[45495]: audit 2022-04-22T19:32:45.517711+0000 mon.a (mon.0) 7137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:47 smithi121 bash[36902]: cluster 2022-04-22T19:32:45.695540+0000 mgr.y (mgr.34107) 4354 : cluster [DBG] pgmap v5743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:47 smithi115 bash[46934]: cluster 2022-04-22T19:32:45.695540+0000 mgr.y (mgr.34107) 4354 : cluster [DBG] pgmap v5743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:47 smithi115 bash[45495]: cluster 2022-04-22T19:32:45.695540+0000 mgr.y (mgr.34107) 4354 : cluster [DBG] pgmap v5743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:49 smithi121 bash[36902]: cluster 2022-04-22T19:32:47.696570+0000 mgr.y (mgr.34107) 4355 : cluster [DBG] pgmap v5744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:49 smithi121 bash[36902]: audit 2022-04-22T19:32:49.010005+0000 mon.a (mon.0) 7138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:49 smithi115 bash[46934]: cluster 2022-04-22T19:32:47.696570+0000 mgr.y (mgr.34107) 4355 : cluster [DBG] pgmap v5744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:49 smithi115 bash[46934]: audit 2022-04-22T19:32:49.010005+0000 mon.a (mon.0) 7138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:49 smithi115 bash[45495]: cluster 2022-04-22T19:32:47.696570+0000 mgr.y (mgr.34107) 4355 : cluster [DBG] pgmap v5744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:49.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:49 smithi115 bash[45495]: audit 2022-04-22T19:32:49.010005+0000 mon.a (mon.0) 7138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:51 smithi121 bash[36902]: cluster 2022-04-22T19:32:49.697655+0000 mgr.y (mgr.34107) 4356 : cluster [DBG] pgmap v5745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:51 smithi121 bash[36902]: audit 2022-04-22T19:32:50.521003+0000 mon.a (mon.0) 7139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:51 smithi115 bash[46934]: cluster 2022-04-22T19:32:49.697655+0000 mgr.y (mgr.34107) 4356 : cluster [DBG] pgmap v5745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:51 smithi115 bash[46934]: audit 2022-04-22T19:32:50.521003+0000 mon.a (mon.0) 7139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:51 smithi115 bash[45495]: cluster 2022-04-22T19:32:49.697655+0000 mgr.y (mgr.34107) 4356 : cluster [DBG] pgmap v5745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:51 smithi115 bash[45495]: audit 2022-04-22T19:32:50.521003+0000 mon.a (mon.0) 7139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:52.100Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:53 smithi121 bash[36902]: cluster 2022-04-22T19:32:51.698799+0000 mgr.y (mgr.34107) 4357 : cluster [DBG] pgmap v5746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:53 smithi115 bash[45495]: cluster 2022-04-22T19:32:51.698799+0000 mgr.y (mgr.34107) 4357 : cluster [DBG] pgmap v5746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:53 smithi115 bash[46934]: cluster 2022-04-22T19:32:51.698799+0000 mgr.y (mgr.34107) 4357 : cluster [DBG] pgmap v5746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:54.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:54 smithi115 bash[45495]: audit 2022-04-22T19:32:54.013668+0000 mon.a (mon.0) 7140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:32:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:32:53.986Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:32:54.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:54 smithi115 bash[46934]: audit 2022-04-22T19:32:54.013668+0000 mon.a (mon.0) 7140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:54 smithi121 bash[36902]: audit 2022-04-22T19:32:54.013668+0000 mon.a (mon.0) 7140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:55 smithi121 bash[36902]: cluster 2022-04-22T19:32:53.700012+0000 mgr.y (mgr.34107) 4358 : cluster [DBG] pgmap v5747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:55.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:55 smithi115 bash[45495]: cluster 2022-04-22T19:32:53.700012+0000 mgr.y (mgr.34107) 4358 : cluster [DBG] pgmap v5747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:55.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:55 smithi115 bash[46934]: cluster 2022-04-22T19:32:53.700012+0000 mgr.y (mgr.34107) 4358 : cluster [DBG] pgmap v5747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:56 smithi121 bash[36902]: audit 2022-04-22T19:32:55.484876+0000 mon.a (mon.0) 7141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:32:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:56 smithi121 bash[36902]: audit 2022-04-22T19:32:55.487726+0000 mon.a (mon.0) 7142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:32:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:56 smithi121 bash[36902]: audit 2022-04-22T19:32:55.489349+0000 mon.a (mon.0) 7143 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:32:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:56 smithi121 bash[36902]: audit 2022-04-22T19:32:55.524112+0000 mon.a (mon.0) 7144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:56 smithi121 bash[36902]: audit 2022-04-22T19:32:55.864325+0000 mon.a (mon.0) 7145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:32:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[45495]: audit 2022-04-22T19:32:55.484876+0000 mon.a (mon.0) 7141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:32:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[45495]: audit 2022-04-22T19:32:55.487726+0000 mon.a (mon.0) 7142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:32:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[45495]: audit 2022-04-22T19:32:55.489349+0000 mon.a (mon.0) 7143 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:32:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[45495]: audit 2022-04-22T19:32:55.524112+0000 mon.a (mon.0) 7144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[45495]: audit 2022-04-22T19:32:55.864325+0000 mon.a (mon.0) 7145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:32:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[46934]: audit 2022-04-22T19:32:55.484876+0000 mon.a (mon.0) 7141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:32:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[46934]: audit 2022-04-22T19:32:55.487726+0000 mon.a (mon.0) 7142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:32:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[46934]: audit 2022-04-22T19:32:55.489349+0000 mon.a (mon.0) 7143 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:32:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[46934]: audit 2022-04-22T19:32:55.524112+0000 mon.a (mon.0) 7144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:32:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:56 smithi115 bash[46934]: audit 2022-04-22T19:32:55.864325+0000 mon.a (mon.0) 7145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:32:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:57 smithi121 bash[36902]: cluster 2022-04-22T19:32:55.701160+0000 mgr.y (mgr.34107) 4359 : cluster [DBG] pgmap v5748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:57 smithi115 bash[45495]: cluster 2022-04-22T19:32:55.701160+0000 mgr.y (mgr.34107) 4359 : cluster [DBG] pgmap v5748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:57.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:57 smithi115 bash[46934]: cluster 2022-04-22T19:32:55.701160+0000 mgr.y (mgr.34107) 4359 : cluster [DBG] pgmap v5748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:59 smithi121 bash[36902]: cluster 2022-04-22T19:32:57.702277+0000 mgr.y (mgr.34107) 4360 : cluster [DBG] pgmap v5749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:32:59 smithi121 bash[36902]: audit 2022-04-22T19:32:59.017327+0000 mon.a (mon.0) 7146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:59 smithi115 bash[45495]: cluster 2022-04-22T19:32:57.702277+0000 mgr.y (mgr.34107) 4360 : cluster [DBG] pgmap v5749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:32:59 smithi115 bash[45495]: audit 2022-04-22T19:32:59.017327+0000 mon.a (mon.0) 7146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:32:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:59 smithi115 bash[46934]: cluster 2022-04-22T19:32:57.702277+0000 mgr.y (mgr.34107) 4360 : cluster [DBG] pgmap v5749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:32:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:32:59 smithi115 bash[46934]: audit 2022-04-22T19:32:59.017327+0000 mon.a (mon.0) 7146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:01 smithi121 bash[36902]: cluster 2022-04-22T19:32:59.703331+0000 mgr.y (mgr.34107) 4361 : cluster [DBG] pgmap v5750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:01 smithi121 bash[36902]: audit 2022-04-22T19:33:00.527316+0000 mon.a (mon.0) 7147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:01 smithi115 bash[45495]: cluster 2022-04-22T19:32:59.703331+0000 mgr.y (mgr.34107) 4361 : cluster [DBG] pgmap v5750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:01 smithi115 bash[45495]: audit 2022-04-22T19:33:00.527316+0000 mon.a (mon.0) 7147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:01 smithi115 bash[46934]: cluster 2022-04-22T19:32:59.703331+0000 mgr.y (mgr.34107) 4361 : cluster [DBG] pgmap v5750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:01 smithi115 bash[46934]: audit 2022-04-22T19:33:00.527316+0000 mon.a (mon.0) 7147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:02.103Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:03 smithi121 bash[36902]: cluster 2022-04-22T19:33:01.704435+0000 mgr.y (mgr.34107) 4362 : cluster [DBG] pgmap v5751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:03 smithi115 bash[45495]: cluster 2022-04-22T19:33:01.704435+0000 mgr.y (mgr.34107) 4362 : cluster [DBG] pgmap v5751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:03 smithi115 bash[46934]: cluster 2022-04-22T19:33:01.704435+0000 mgr.y (mgr.34107) 4362 : cluster [DBG] pgmap v5751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:04 smithi115 bash[45495]: audit 2022-04-22T19:33:04.021070+0000 mon.a (mon.0) 7148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:03.972Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:04 smithi115 bash[46934]: audit 2022-04-22T19:33:04.021070+0000 mon.a (mon.0) 7148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:04 smithi121 bash[36902]: audit 2022-04-22T19:33:04.021070+0000 mon.a (mon.0) 7148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:05 smithi121 bash[36902]: cluster 2022-04-22T19:33:03.705568+0000 mgr.y (mgr.34107) 4363 : cluster [DBG] pgmap v5752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:05 smithi115 bash[45495]: cluster 2022-04-22T19:33:03.705568+0000 mgr.y (mgr.34107) 4363 : cluster [DBG] pgmap v5752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:05 smithi115 bash[46934]: cluster 2022-04-22T19:33:03.705568+0000 mgr.y (mgr.34107) 4363 : cluster [DBG] pgmap v5752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:06 smithi121 bash[36902]: audit 2022-04-22T19:33:05.530487+0000 mon.a (mon.0) 7149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:06 smithi115 bash[45495]: audit 2022-04-22T19:33:05.530487+0000 mon.a (mon.0) 7149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:06 smithi115 bash[46934]: audit 2022-04-22T19:33:05.530487+0000 mon.a (mon.0) 7149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:07 smithi121 bash[36902]: cluster 2022-04-22T19:33:05.706754+0000 mgr.y (mgr.34107) 4364 : cluster [DBG] pgmap v5753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:07.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:07 smithi115 bash[45495]: cluster 2022-04-22T19:33:05.706754+0000 mgr.y (mgr.34107) 4364 : cluster [DBG] pgmap v5753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:07.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:07 smithi115 bash[46934]: cluster 2022-04-22T19:33:05.706754+0000 mgr.y (mgr.34107) 4364 : cluster [DBG] pgmap v5753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:09 smithi121 bash[36902]: cluster 2022-04-22T19:33:07.707902+0000 mgr.y (mgr.34107) 4365 : cluster [DBG] pgmap v5754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:09 smithi121 bash[36902]: audit 2022-04-22T19:33:09.024763+0000 mon.a (mon.0) 7150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:09 smithi115 bash[45495]: cluster 2022-04-22T19:33:07.707902+0000 mgr.y (mgr.34107) 4365 : cluster [DBG] pgmap v5754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:09 smithi115 bash[45495]: audit 2022-04-22T19:33:09.024763+0000 mon.a (mon.0) 7150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:09 smithi115 bash[46934]: cluster 2022-04-22T19:33:07.707902+0000 mgr.y (mgr.34107) 4365 : cluster [DBG] pgmap v5754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:09.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:09 smithi115 bash[46934]: audit 2022-04-22T19:33:09.024763+0000 mon.a (mon.0) 7150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:11 smithi121 bash[36902]: cluster 2022-04-22T19:33:09.708995+0000 mgr.y (mgr.34107) 4366 : cluster [DBG] pgmap v5755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:11 smithi121 bash[36902]: audit 2022-04-22T19:33:10.533598+0000 mon.a (mon.0) 7151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:11 smithi115 bash[45495]: cluster 2022-04-22T19:33:09.708995+0000 mgr.y (mgr.34107) 4366 : cluster [DBG] pgmap v5755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:11 smithi115 bash[45495]: audit 2022-04-22T19:33:10.533598+0000 mon.a (mon.0) 7151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:11 smithi115 bash[46934]: cluster 2022-04-22T19:33:09.708995+0000 mgr.y (mgr.34107) 4366 : cluster [DBG] pgmap v5755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:11 smithi115 bash[46934]: audit 2022-04-22T19:33:10.533598+0000 mon.a (mon.0) 7151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:12.102Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:13 smithi121 bash[36902]: cluster 2022-04-22T19:33:11.710163+0000 mgr.y (mgr.34107) 4367 : cluster [DBG] pgmap v5756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:13 smithi115 bash[45495]: cluster 2022-04-22T19:33:11.710163+0000 mgr.y (mgr.34107) 4367 : cluster [DBG] pgmap v5756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:13 smithi115 bash[46934]: cluster 2022-04-22T19:33:11.710163+0000 mgr.y (mgr.34107) 4367 : cluster [DBG] pgmap v5756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:13.615 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (116m) 2m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (118m) 2m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (117m) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:33:13.960 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (117m) 2m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (117m) 3m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:33:13.961 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:33:13.962 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:33:13.962 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:33:13.962 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (119m) 3m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:33:13.962 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (118m) 3m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:33:13.962 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (116m) 3m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:33:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:13.973Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:14.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:14 smithi115 bash[45495]: audit 2022-04-22T19:33:14.028280+0000 mon.a (mon.0) 7152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:14.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:14 smithi115 bash[46934]: audit 2022-04-22T19:33:14.028280+0000 mon.a (mon.0) 7152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:33:14.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:33:14.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:33:14.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:33:14.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:14.364 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:33:14.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:33:14.364 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:33:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:33:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:14 smithi121 bash[36902]: audit 2022-04-22T19:33:14.028280+0000 mon.a (mon.0) 7152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: audit 2022-04-22T19:33:13.589944+0000 mgr.y (mgr.34107) 4368 : audit [DBG] from='client.39960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: cluster 2022-04-22T19:33:13.710871+0000 mgr.y (mgr.34107) 4369 : cluster [DBG] pgmap v5757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: audit 2022-04-22T19:33:13.928290+0000 mgr.y (mgr.34107) 4370 : audit [DBG] from='client.39966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: audit 2022-04-22T19:33:14.351457+0000 mon.a (mon.0) 7153 : audit [DBG] from='client.? 172.21.15.115:0/2463324550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: audit 2022-04-22T19:33:14.726298+0000 mon.a (mon.0) 7154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:33:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:15 smithi121 bash[36902]: audit 2022-04-22T19:33:14.726755+0000 mon.a (mon.0) 7155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: audit 2022-04-22T19:33:13.589944+0000 mgr.y (mgr.34107) 4368 : audit [DBG] from='client.39960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: cluster 2022-04-22T19:33:13.710871+0000 mgr.y (mgr.34107) 4369 : cluster [DBG] pgmap v5757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: audit 2022-04-22T19:33:13.928290+0000 mgr.y (mgr.34107) 4370 : audit [DBG] from='client.39966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: audit 2022-04-22T19:33:14.351457+0000 mon.a (mon.0) 7153 : audit [DBG] from='client.? 172.21.15.115:0/2463324550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: audit 2022-04-22T19:33:14.726298+0000 mon.a (mon.0) 7154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:33:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[46934]: audit 2022-04-22T19:33:14.726755+0000 mon.a (mon.0) 7155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: audit 2022-04-22T19:33:13.589944+0000 mgr.y (mgr.34107) 4368 : audit [DBG] from='client.39960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: cluster 2022-04-22T19:33:13.710871+0000 mgr.y (mgr.34107) 4369 : cluster [DBG] pgmap v5757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: audit 2022-04-22T19:33:13.928290+0000 mgr.y (mgr.34107) 4370 : audit [DBG] from='client.39966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: audit 2022-04-22T19:33:14.351457+0000 mon.a (mon.0) 7153 : audit [DBG] from='client.? 172.21.15.115:0/2463324550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: audit 2022-04-22T19:33:14.726298+0000 mon.a (mon.0) 7154 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:33:15.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:15 smithi115 bash[45495]: audit 2022-04-22T19:33:14.726755+0000 mon.a (mon.0) 7155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:33:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:16 smithi121 bash[36902]: audit 2022-04-22T19:33:15.536867+0000 mon.a (mon.0) 7156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:16.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:16 smithi115 bash[45495]: audit 2022-04-22T19:33:15.536867+0000 mon.a (mon.0) 7156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:16 smithi115 bash[46934]: audit 2022-04-22T19:33:15.536867+0000 mon.a (mon.0) 7156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:17 smithi121 bash[36902]: cluster 2022-04-22T19:33:15.712017+0000 mgr.y (mgr.34107) 4371 : cluster [DBG] pgmap v5758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:17.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:17 smithi115 bash[45495]: cluster 2022-04-22T19:33:15.712017+0000 mgr.y (mgr.34107) 4371 : cluster [DBG] pgmap v5758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:17 smithi115 bash[46934]: cluster 2022-04-22T19:33:15.712017+0000 mgr.y (mgr.34107) 4371 : cluster [DBG] pgmap v5758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:19 smithi121 bash[36902]: cluster 2022-04-22T19:33:17.713070+0000 mgr.y (mgr.34107) 4372 : cluster [DBG] pgmap v5759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:19 smithi121 bash[36902]: audit 2022-04-22T19:33:19.031789+0000 mon.a (mon.0) 7157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:19 smithi115 bash[45495]: cluster 2022-04-22T19:33:17.713070+0000 mgr.y (mgr.34107) 4372 : cluster [DBG] pgmap v5759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:19 smithi115 bash[45495]: audit 2022-04-22T19:33:19.031789+0000 mon.a (mon.0) 7157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:19 smithi115 bash[46934]: cluster 2022-04-22T19:33:17.713070+0000 mgr.y (mgr.34107) 4372 : cluster [DBG] pgmap v5759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:19 smithi115 bash[46934]: audit 2022-04-22T19:33:19.031789+0000 mon.a (mon.0) 7157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:21 smithi121 bash[36902]: cluster 2022-04-22T19:33:19.714156+0000 mgr.y (mgr.34107) 4373 : cluster [DBG] pgmap v5760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:21 smithi121 bash[36902]: audit 2022-04-22T19:33:20.540379+0000 mon.a (mon.0) 7158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:21 smithi115 bash[45495]: cluster 2022-04-22T19:33:19.714156+0000 mgr.y (mgr.34107) 4373 : cluster [DBG] pgmap v5760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:21 smithi115 bash[45495]: audit 2022-04-22T19:33:20.540379+0000 mon.a (mon.0) 7158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:21 smithi115 bash[46934]: cluster 2022-04-22T19:33:19.714156+0000 mgr.y (mgr.34107) 4373 : cluster [DBG] pgmap v5760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:21 smithi115 bash[46934]: audit 2022-04-22T19:33:20.540379+0000 mon.a (mon.0) 7158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:22.102Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:23 smithi121 bash[36902]: cluster 2022-04-22T19:33:21.715204+0000 mgr.y (mgr.34107) 4374 : cluster [DBG] pgmap v5761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:23 smithi115 bash[45495]: cluster 2022-04-22T19:33:21.715204+0000 mgr.y (mgr.34107) 4374 : cluster [DBG] pgmap v5761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:23 smithi115 bash[46934]: cluster 2022-04-22T19:33:21.715204+0000 mgr.y (mgr.34107) 4374 : cluster [DBG] pgmap v5761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:24.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:24 smithi115 bash[45495]: audit 2022-04-22T19:33:24.035565+0000 mon.a (mon.0) 7159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:23.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:24.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:24 smithi115 bash[46934]: audit 2022-04-22T19:33:24.035565+0000 mon.a (mon.0) 7159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:24 smithi121 bash[36902]: audit 2022-04-22T19:33:24.035565+0000 mon.a (mon.0) 7159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:25 smithi121 bash[36902]: cluster 2022-04-22T19:33:23.716342+0000 mgr.y (mgr.34107) 4375 : cluster [DBG] pgmap v5762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:25 smithi115 bash[45495]: cluster 2022-04-22T19:33:23.716342+0000 mgr.y (mgr.34107) 4375 : cluster [DBG] pgmap v5762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:25 smithi115 bash[46934]: cluster 2022-04-22T19:33:23.716342+0000 mgr.y (mgr.34107) 4375 : cluster [DBG] pgmap v5762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:26 smithi121 bash[36902]: audit 2022-04-22T19:33:25.543420+0000 mon.a (mon.0) 7160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:26.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:26 smithi115 bash[45495]: audit 2022-04-22T19:33:25.543420+0000 mon.a (mon.0) 7160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:26 smithi115 bash[46934]: audit 2022-04-22T19:33:25.543420+0000 mon.a (mon.0) 7160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:27 smithi121 bash[36902]: cluster 2022-04-22T19:33:25.717396+0000 mgr.y (mgr.34107) 4376 : cluster [DBG] pgmap v5763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:27 smithi115 bash[45495]: cluster 2022-04-22T19:33:25.717396+0000 mgr.y (mgr.34107) 4376 : cluster [DBG] pgmap v5763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:27.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:27 smithi115 bash[46934]: cluster 2022-04-22T19:33:25.717396+0000 mgr.y (mgr.34107) 4376 : cluster [DBG] pgmap v5763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:29 smithi121 bash[36902]: cluster 2022-04-22T19:33:27.718457+0000 mgr.y (mgr.34107) 4377 : cluster [DBG] pgmap v5764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:29 smithi121 bash[36902]: audit 2022-04-22T19:33:29.039165+0000 mon.a (mon.0) 7161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:29 smithi115 bash[45495]: cluster 2022-04-22T19:33:27.718457+0000 mgr.y (mgr.34107) 4377 : cluster [DBG] pgmap v5764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:29 smithi115 bash[45495]: audit 2022-04-22T19:33:29.039165+0000 mon.a (mon.0) 7161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:29 smithi115 bash[46934]: cluster 2022-04-22T19:33:27.718457+0000 mgr.y (mgr.34107) 4377 : cluster [DBG] pgmap v5764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:29 smithi115 bash[46934]: audit 2022-04-22T19:33:29.039165+0000 mon.a (mon.0) 7161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:31 smithi121 bash[36902]: cluster 2022-04-22T19:33:29.719467+0000 mgr.y (mgr.34107) 4378 : cluster [DBG] pgmap v5765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:31 smithi121 bash[36902]: audit 2022-04-22T19:33:30.546657+0000 mon.a (mon.0) 7162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:31 smithi115 bash[45495]: cluster 2022-04-22T19:33:29.719467+0000 mgr.y (mgr.34107) 4378 : cluster [DBG] pgmap v5765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:31 smithi115 bash[45495]: audit 2022-04-22T19:33:30.546657+0000 mon.a (mon.0) 7162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:31 smithi115 bash[46934]: cluster 2022-04-22T19:33:29.719467+0000 mgr.y (mgr.34107) 4378 : cluster [DBG] pgmap v5765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:31 smithi115 bash[46934]: audit 2022-04-22T19:33:30.546657+0000 mon.a (mon.0) 7162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:32.104Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:33 smithi121 bash[36902]: cluster 2022-04-22T19:33:31.720623+0000 mgr.y (mgr.34107) 4379 : cluster [DBG] pgmap v5766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:33.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:33 smithi115 bash[45495]: cluster 2022-04-22T19:33:31.720623+0000 mgr.y (mgr.34107) 4379 : cluster [DBG] pgmap v5766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:33 smithi115 bash[46934]: cluster 2022-04-22T19:33:31.720623+0000 mgr.y (mgr.34107) 4379 : cluster [DBG] pgmap v5766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:34.248 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:33.974Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:34 smithi121 bash[36902]: audit 2022-04-22T19:33:34.042959+0000 mon.a (mon.0) 7163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:34.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:34 smithi115 bash[45495]: audit 2022-04-22T19:33:34.042959+0000 mon.a (mon.0) 7163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:34.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:34 smithi115 bash[46934]: audit 2022-04-22T19:33:34.042959+0000 mon.a (mon.0) 7163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:35 smithi121 bash[36902]: cluster 2022-04-22T19:33:33.721754+0000 mgr.y (mgr.34107) 4380 : cluster [DBG] pgmap v5767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:35 smithi115 bash[45495]: cluster 2022-04-22T19:33:33.721754+0000 mgr.y (mgr.34107) 4380 : cluster [DBG] pgmap v5767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:35 smithi115 bash[46934]: cluster 2022-04-22T19:33:33.721754+0000 mgr.y (mgr.34107) 4380 : cluster [DBG] pgmap v5767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:36 smithi121 bash[36902]: audit 2022-04-22T19:33:35.549712+0000 mon.a (mon.0) 7164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:36.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:36 smithi115 bash[45495]: audit 2022-04-22T19:33:35.549712+0000 mon.a (mon.0) 7164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:36.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:36 smithi115 bash[46934]: audit 2022-04-22T19:33:35.549712+0000 mon.a (mon.0) 7164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:37 smithi121 bash[36902]: cluster 2022-04-22T19:33:35.722850+0000 mgr.y (mgr.34107) 4381 : cluster [DBG] pgmap v5768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:37 smithi115 bash[45495]: cluster 2022-04-22T19:33:35.722850+0000 mgr.y (mgr.34107) 4381 : cluster [DBG] pgmap v5768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:37 smithi115 bash[46934]: cluster 2022-04-22T19:33:35.722850+0000 mgr.y (mgr.34107) 4381 : cluster [DBG] pgmap v5768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:39 smithi115 bash[45495]: cluster 2022-04-22T19:33:37.723933+0000 mgr.y (mgr.34107) 4382 : cluster [DBG] pgmap v5769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:39.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:39 smithi115 bash[45495]: audit 2022-04-22T19:33:39.046761+0000 mon.a (mon.0) 7165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:39 smithi115 bash[46934]: cluster 2022-04-22T19:33:37.723933+0000 mgr.y (mgr.34107) 4382 : cluster [DBG] pgmap v5769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:39 smithi115 bash[46934]: audit 2022-04-22T19:33:39.046761+0000 mon.a (mon.0) 7165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:39 smithi121 bash[36902]: cluster 2022-04-22T19:33:37.723933+0000 mgr.y (mgr.34107) 4382 : cluster [DBG] pgmap v5769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:39 smithi121 bash[36902]: audit 2022-04-22T19:33:39.046761+0000 mon.a (mon.0) 7165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:41 smithi115 bash[45495]: cluster 2022-04-22T19:33:39.724934+0000 mgr.y (mgr.34107) 4383 : cluster [DBG] pgmap v5770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:41 smithi115 bash[45495]: audit 2022-04-22T19:33:40.552797+0000 mon.a (mon.0) 7166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:41 smithi115 bash[46934]: cluster 2022-04-22T19:33:39.724934+0000 mgr.y (mgr.34107) 4383 : cluster [DBG] pgmap v5770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:41 smithi115 bash[46934]: audit 2022-04-22T19:33:40.552797+0000 mon.a (mon.0) 7166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:41 smithi121 bash[36902]: cluster 2022-04-22T19:33:39.724934+0000 mgr.y (mgr.34107) 4383 : cluster [DBG] pgmap v5770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:41 smithi121 bash[36902]: audit 2022-04-22T19:33:40.552797+0000 mon.a (mon.0) 7166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:42.106Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:43 smithi115 bash[45495]: cluster 2022-04-22T19:33:41.725978+0000 mgr.y (mgr.34107) 4384 : cluster [DBG] pgmap v5771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:43.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:43 smithi115 bash[46934]: cluster 2022-04-22T19:33:41.725978+0000 mgr.y (mgr.34107) 4384 : cluster [DBG] pgmap v5771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:43 smithi121 bash[36902]: cluster 2022-04-22T19:33:41.725978+0000 mgr.y (mgr.34107) 4384 : cluster [DBG] pgmap v5771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:44.285 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:43.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:44.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:44 smithi115 bash[46934]: audit 2022-04-22T19:33:44.050698+0000 mon.a (mon.0) 7167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:44.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:44 smithi115 bash[45495]: audit 2022-04-22T19:33:44.050698+0000 mon.a (mon.0) 7167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:44.729 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:33:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:44 smithi121 bash[36902]: audit 2022-04-22T19:33:44.050698+0000 mon.a (mon.0) 7167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (116m) 2m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (118m) 2m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (118m) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:33:45.080 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (117m) 2m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (117m) 3m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:33:45.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:33:45.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (119m) 3m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:33:45.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (119m) 3m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:33:45.082 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (117m) 3m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:33:45.490 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:33:45.490 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:45.491 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:33:45.492 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:33:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:45 smithi115 bash[45495]: cluster 2022-04-22T19:33:43.727150+0000 mgr.y (mgr.34107) 4385 : cluster [DBG] pgmap v5772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:45 smithi115 bash[46934]: cluster 2022-04-22T19:33:43.727150+0000 mgr.y (mgr.34107) 4385 : cluster [DBG] pgmap v5772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:45 smithi121 bash[36902]: cluster 2022-04-22T19:33:43.727150+0000 mgr.y (mgr.34107) 4385 : cluster [DBG] pgmap v5772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[45495]: audit 2022-04-22T19:33:44.703519+0000 mgr.y (mgr.34107) 4386 : audit [DBG] from='client.49912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[45495]: audit 2022-04-22T19:33:45.043591+0000 mgr.y (mgr.34107) 4387 : audit [DBG] from='client.39984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[45495]: audit 2022-04-22T19:33:45.479744+0000 mon.c (mon.1) 197 : audit [DBG] from='client.? 172.21.15.115:0/1827615929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[45495]: audit 2022-04-22T19:33:45.556013+0000 mon.a (mon.0) 7168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[46934]: audit 2022-04-22T19:33:44.703519+0000 mgr.y (mgr.34107) 4386 : audit [DBG] from='client.49912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[46934]: audit 2022-04-22T19:33:45.043591+0000 mgr.y (mgr.34107) 4387 : audit [DBG] from='client.39984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[46934]: audit 2022-04-22T19:33:45.479744+0000 mon.c (mon.1) 197 : audit [DBG] from='client.? 172.21.15.115:0/1827615929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:46 smithi115 bash[46934]: audit 2022-04-22T19:33:45.556013+0000 mon.a (mon.0) 7168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:46 smithi121 bash[36902]: audit 2022-04-22T19:33:44.703519+0000 mgr.y (mgr.34107) 4386 : audit [DBG] from='client.49912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:46 smithi121 bash[36902]: audit 2022-04-22T19:33:45.043591+0000 mgr.y (mgr.34107) 4387 : audit [DBG] from='client.39984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:33:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:46 smithi121 bash[36902]: audit 2022-04-22T19:33:45.479744+0000 mon.c (mon.1) 197 : audit [DBG] from='client.? 172.21.15.115:0/1827615929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:33:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:46 smithi121 bash[36902]: audit 2022-04-22T19:33:45.556013+0000 mon.a (mon.0) 7168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:47 smithi115 bash[45495]: cluster 2022-04-22T19:33:45.728229+0000 mgr.y (mgr.34107) 4388 : cluster [DBG] pgmap v5773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:47 smithi115 bash[46934]: cluster 2022-04-22T19:33:45.728229+0000 mgr.y (mgr.34107) 4388 : cluster [DBG] pgmap v5773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:47 smithi121 bash[36902]: cluster 2022-04-22T19:33:45.728229+0000 mgr.y (mgr.34107) 4388 : cluster [DBG] pgmap v5773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:49 smithi121 bash[36902]: cluster 2022-04-22T19:33:47.729278+0000 mgr.y (mgr.34107) 4389 : cluster [DBG] pgmap v5774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:49 smithi121 bash[36902]: audit 2022-04-22T19:33:49.054263+0000 mon.a (mon.0) 7169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:49 smithi115 bash[45495]: cluster 2022-04-22T19:33:47.729278+0000 mgr.y (mgr.34107) 4389 : cluster [DBG] pgmap v5774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:49 smithi115 bash[45495]: audit 2022-04-22T19:33:49.054263+0000 mon.a (mon.0) 7169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:49 smithi115 bash[46934]: cluster 2022-04-22T19:33:47.729278+0000 mgr.y (mgr.34107) 4389 : cluster [DBG] pgmap v5774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:49 smithi115 bash[46934]: audit 2022-04-22T19:33:49.054263+0000 mon.a (mon.0) 7169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:51 smithi121 bash[36902]: cluster 2022-04-22T19:33:49.730337+0000 mgr.y (mgr.34107) 4390 : cluster [DBG] pgmap v5775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:51 smithi121 bash[36902]: audit 2022-04-22T19:33:50.559236+0000 mon.a (mon.0) 7170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:51 smithi115 bash[45495]: cluster 2022-04-22T19:33:49.730337+0000 mgr.y (mgr.34107) 4390 : cluster [DBG] pgmap v5775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:51 smithi115 bash[45495]: audit 2022-04-22T19:33:50.559236+0000 mon.a (mon.0) 7170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:51 smithi115 bash[46934]: cluster 2022-04-22T19:33:49.730337+0000 mgr.y (mgr.34107) 4390 : cluster [DBG] pgmap v5775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:51 smithi115 bash[46934]: audit 2022-04-22T19:33:50.559236+0000 mon.a (mon.0) 7170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:52.105Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:53 smithi121 bash[36902]: cluster 2022-04-22T19:33:51.731580+0000 mgr.y (mgr.34107) 4391 : cluster [DBG] pgmap v5776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:53 smithi115 bash[45495]: cluster 2022-04-22T19:33:51.731580+0000 mgr.y (mgr.34107) 4391 : cluster [DBG] pgmap v5776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:53 smithi115 bash[46934]: cluster 2022-04-22T19:33:51.731580+0000 mgr.y (mgr.34107) 4391 : cluster [DBG] pgmap v5776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:33:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:33:53.977Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:33:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:54 smithi121 bash[36902]: audit 2022-04-22T19:33:54.057769+0000 mon.a (mon.0) 7171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:54 smithi115 bash[45495]: audit 2022-04-22T19:33:54.057769+0000 mon.a (mon.0) 7171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:54 smithi115 bash[46934]: audit 2022-04-22T19:33:54.057769+0000 mon.a (mon.0) 7171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:55 smithi121 bash[36902]: cluster 2022-04-22T19:33:53.732692+0000 mgr.y (mgr.34107) 4392 : cluster [DBG] pgmap v5777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:55 smithi115 bash[45495]: cluster 2022-04-22T19:33:53.732692+0000 mgr.y (mgr.34107) 4392 : cluster [DBG] pgmap v5777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:55 smithi115 bash[46934]: cluster 2022-04-22T19:33:53.732692+0000 mgr.y (mgr.34107) 4392 : cluster [DBG] pgmap v5777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:56 smithi121 bash[36902]: audit 2022-04-22T19:33:55.562293+0000 mon.a (mon.0) 7172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:56 smithi121 bash[36902]: audit 2022-04-22T19:33:55.871898+0000 mon.a (mon.0) 7173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:56 smithi121 bash[36902]: audit 2022-04-22T19:33:55.875726+0000 mon.a (mon.0) 7174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:56 smithi121 bash[36902]: audit 2022-04-22T19:33:55.877532+0000 mon.a (mon.0) 7175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:56 smithi121 bash[36902]: audit 2022-04-22T19:33:56.234048+0000 mon.a (mon.0) 7176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[45495]: audit 2022-04-22T19:33:55.562293+0000 mon.a (mon.0) 7172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[45495]: audit 2022-04-22T19:33:55.871898+0000 mon.a (mon.0) 7173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[45495]: audit 2022-04-22T19:33:55.875726+0000 mon.a (mon.0) 7174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[45495]: audit 2022-04-22T19:33:55.877532+0000 mon.a (mon.0) 7175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[45495]: audit 2022-04-22T19:33:56.234048+0000 mon.a (mon.0) 7176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:33:56.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[46934]: audit 2022-04-22T19:33:55.562293+0000 mon.a (mon.0) 7172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:33:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[46934]: audit 2022-04-22T19:33:55.871898+0000 mon.a (mon.0) 7173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:33:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[46934]: audit 2022-04-22T19:33:55.875726+0000 mon.a (mon.0) 7174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:33:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[46934]: audit 2022-04-22T19:33:55.877532+0000 mon.a (mon.0) 7175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:33:56.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:56 smithi115 bash[46934]: audit 2022-04-22T19:33:56.234048+0000 mon.a (mon.0) 7176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:33:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:57 smithi121 bash[36902]: cluster 2022-04-22T19:33:55.733739+0000 mgr.y (mgr.34107) 4393 : cluster [DBG] pgmap v5778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:57.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:57 smithi115 bash[45495]: cluster 2022-04-22T19:33:55.733739+0000 mgr.y (mgr.34107) 4393 : cluster [DBG] pgmap v5778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:57.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:57 smithi115 bash[46934]: cluster 2022-04-22T19:33:55.733739+0000 mgr.y (mgr.34107) 4393 : cluster [DBG] pgmap v5778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:59 smithi121 bash[36902]: cluster 2022-04-22T19:33:57.735160+0000 mgr.y (mgr.34107) 4394 : cluster [DBG] pgmap v5779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:33:59 smithi121 bash[36902]: audit 2022-04-22T19:33:59.061495+0000 mon.a (mon.0) 7177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:59 smithi115 bash[46934]: cluster 2022-04-22T19:33:57.735160+0000 mgr.y (mgr.34107) 4394 : cluster [DBG] pgmap v5779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:33:59 smithi115 bash[46934]: audit 2022-04-22T19:33:59.061495+0000 mon.a (mon.0) 7177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:33:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:59 smithi115 bash[45495]: cluster 2022-04-22T19:33:57.735160+0000 mgr.y (mgr.34107) 4394 : cluster [DBG] pgmap v5779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:33:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:33:59 smithi115 bash[45495]: audit 2022-04-22T19:33:59.061495+0000 mon.a (mon.0) 7177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:01 smithi121 bash[36902]: cluster 2022-04-22T19:33:59.736211+0000 mgr.y (mgr.34107) 4395 : cluster [DBG] pgmap v5780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:01 smithi121 bash[36902]: audit 2022-04-22T19:34:00.565475+0000 mon.a (mon.0) 7178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:01 smithi115 bash[45495]: cluster 2022-04-22T19:33:59.736211+0000 mgr.y (mgr.34107) 4395 : cluster [DBG] pgmap v5780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:01 smithi115 bash[45495]: audit 2022-04-22T19:34:00.565475+0000 mon.a (mon.0) 7178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:01 smithi115 bash[46934]: cluster 2022-04-22T19:33:59.736211+0000 mgr.y (mgr.34107) 4395 : cluster [DBG] pgmap v5780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:01 smithi115 bash[46934]: audit 2022-04-22T19:34:00.565475+0000 mon.a (mon.0) 7178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:02.107Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:03 smithi121 bash[36902]: cluster 2022-04-22T19:34:01.737366+0000 mgr.y (mgr.34107) 4396 : cluster [DBG] pgmap v5781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:03 smithi115 bash[45495]: cluster 2022-04-22T19:34:01.737366+0000 mgr.y (mgr.34107) 4396 : cluster [DBG] pgmap v5781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:03 smithi115 bash[46934]: cluster 2022-04-22T19:34:01.737366+0000 mgr.y (mgr.34107) 4396 : cluster [DBG] pgmap v5781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:03.977Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:04 smithi121 bash[36902]: audit 2022-04-22T19:34:04.065415+0000 mon.a (mon.0) 7179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:04 smithi115 bash[45495]: audit 2022-04-22T19:34:04.065415+0000 mon.a (mon.0) 7179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:04 smithi115 bash[46934]: audit 2022-04-22T19:34:04.065415+0000 mon.a (mon.0) 7179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:05 smithi121 bash[36902]: cluster 2022-04-22T19:34:03.738602+0000 mgr.y (mgr.34107) 4397 : cluster [DBG] pgmap v5782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:05 smithi115 bash[45495]: cluster 2022-04-22T19:34:03.738602+0000 mgr.y (mgr.34107) 4397 : cluster [DBG] pgmap v5782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:05 smithi115 bash[46934]: cluster 2022-04-22T19:34:03.738602+0000 mgr.y (mgr.34107) 4397 : cluster [DBG] pgmap v5782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:06 smithi121 bash[36902]: audit 2022-04-22T19:34:05.568612+0000 mon.a (mon.0) 7180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:06 smithi115 bash[45495]: audit 2022-04-22T19:34:05.568612+0000 mon.a (mon.0) 7180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:06 smithi115 bash[46934]: audit 2022-04-22T19:34:05.568612+0000 mon.a (mon.0) 7180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:07 smithi121 bash[36902]: cluster 2022-04-22T19:34:05.739871+0000 mgr.y (mgr.34107) 4398 : cluster [DBG] pgmap v5783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:07 smithi115 bash[45495]: cluster 2022-04-22T19:34:05.739871+0000 mgr.y (mgr.34107) 4398 : cluster [DBG] pgmap v5783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:07 smithi115 bash[46934]: cluster 2022-04-22T19:34:05.739871+0000 mgr.y (mgr.34107) 4398 : cluster [DBG] pgmap v5783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:09 smithi121 bash[36902]: cluster 2022-04-22T19:34:07.741073+0000 mgr.y (mgr.34107) 4399 : cluster [DBG] pgmap v5784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:09 smithi121 bash[36902]: audit 2022-04-22T19:34:09.069054+0000 mon.a (mon.0) 7181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:09 smithi115 bash[45495]: cluster 2022-04-22T19:34:07.741073+0000 mgr.y (mgr.34107) 4399 : cluster [DBG] pgmap v5784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:09 smithi115 bash[45495]: audit 2022-04-22T19:34:09.069054+0000 mon.a (mon.0) 7181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:09 smithi115 bash[46934]: cluster 2022-04-22T19:34:07.741073+0000 mgr.y (mgr.34107) 4399 : cluster [DBG] pgmap v5784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:09 smithi115 bash[46934]: audit 2022-04-22T19:34:09.069054+0000 mon.a (mon.0) 7181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:11 smithi121 bash[36902]: cluster 2022-04-22T19:34:09.742072+0000 mgr.y (mgr.34107) 4400 : cluster [DBG] pgmap v5785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:11 smithi121 bash[36902]: audit 2022-04-22T19:34:10.571753+0000 mon.a (mon.0) 7182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:11 smithi115 bash[45495]: cluster 2022-04-22T19:34:09.742072+0000 mgr.y (mgr.34107) 4400 : cluster [DBG] pgmap v5785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:11 smithi115 bash[45495]: audit 2022-04-22T19:34:10.571753+0000 mon.a (mon.0) 7182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:11 smithi115 bash[46934]: cluster 2022-04-22T19:34:09.742072+0000 mgr.y (mgr.34107) 4400 : cluster [DBG] pgmap v5785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:11 smithi115 bash[46934]: audit 2022-04-22T19:34:10.571753+0000 mon.a (mon.0) 7182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:12.107Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:13 smithi121 bash[36902]: cluster 2022-04-22T19:34:11.743163+0000 mgr.y (mgr.34107) 4401 : cluster [DBG] pgmap v5786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:13 smithi115 bash[45495]: cluster 2022-04-22T19:34:11.743163+0000 mgr.y (mgr.34107) 4401 : cluster [DBG] pgmap v5786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:13 smithi115 bash[46934]: cluster 2022-04-22T19:34:11.743163+0000 mgr.y (mgr.34107) 4401 : cluster [DBG] pgmap v5786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:13.979Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:14.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:14 smithi121 bash[36902]: audit 2022-04-22T19:34:14.072941+0000 mon.a (mon.0) 7183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:14 smithi115 bash[45495]: audit 2022-04-22T19:34:14.072941+0000 mon.a (mon.0) 7183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:14.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:14 smithi115 bash[46934]: audit 2022-04-22T19:34:14.072941+0000 mon.a (mon.0) 7183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:15 smithi121 bash[36902]: cluster 2022-04-22T19:34:13.744280+0000 mgr.y (mgr.34107) 4402 : cluster [DBG] pgmap v5787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:15 smithi121 bash[36902]: audit 2022-04-22T19:34:14.735425+0000 mon.a (mon.0) 7184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:34:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:15 smithi121 bash[36902]: audit 2022-04-22T19:34:14.739878+0000 mon.a (mon.0) 7185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:34:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[46934]: cluster 2022-04-22T19:34:13.744280+0000 mgr.y (mgr.34107) 4402 : cluster [DBG] pgmap v5787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[46934]: audit 2022-04-22T19:34:14.735425+0000 mon.a (mon.0) 7184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:34:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[46934]: audit 2022-04-22T19:34:14.739878+0000 mon.a (mon.0) 7185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:34:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[45495]: cluster 2022-04-22T19:34:13.744280+0000 mgr.y (mgr.34107) 4402 : cluster [DBG] pgmap v5787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[45495]: audit 2022-04-22T19:34:14.735425+0000 mon.a (mon.0) 7184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:34:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:15 smithi115 bash[45495]: audit 2022-04-22T19:34:14.739878+0000 mon.a (mon.0) 7185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:34:15.869 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:34:16.215 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:34:16.215 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (117m) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:34:16.215 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (119m) 3m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (118m) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:34:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (118m) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (118m) 4m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:34:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:34:16.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:34:16.218 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (119m) 4m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:34:16.218 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (118m) 4m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:34:16.620 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:34:16.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:34:16.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:34:16.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:16.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:34:16.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:34:16.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:34:16.622 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:34:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:16 smithi121 bash[36902]: audit 2022-04-22T19:34:15.574903+0000 mon.a (mon.0) 7186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:16 smithi115 bash[45495]: audit 2022-04-22T19:34:15.574903+0000 mon.a (mon.0) 7186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:16 smithi115 bash[46934]: audit 2022-04-22T19:34:15.574903+0000 mon.a (mon.0) 7186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:17 smithi121 bash[36902]: cluster 2022-04-22T19:34:15.745315+0000 mgr.y (mgr.34107) 4403 : cluster [DBG] pgmap v5788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:17 smithi121 bash[36902]: audit 2022-04-22T19:34:15.840911+0000 mgr.y (mgr.34107) 4404 : audit [DBG] from='client.49930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:17 smithi121 bash[36902]: audit 2022-04-22T19:34:16.185312+0000 mgr.y (mgr.34107) 4405 : audit [DBG] from='client.40002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:17 smithi121 bash[36902]: audit 2022-04-22T19:34:16.609691+0000 mon.c (mon.1) 198 : audit [DBG] from='client.? 172.21.15.115:0/2319253579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[45495]: cluster 2022-04-22T19:34:15.745315+0000 mgr.y (mgr.34107) 4403 : cluster [DBG] pgmap v5788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[45495]: audit 2022-04-22T19:34:15.840911+0000 mgr.y (mgr.34107) 4404 : audit [DBG] from='client.49930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[45495]: audit 2022-04-22T19:34:16.185312+0000 mgr.y (mgr.34107) 4405 : audit [DBG] from='client.40002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[45495]: audit 2022-04-22T19:34:16.609691+0000 mon.c (mon.1) 198 : audit [DBG] from='client.? 172.21.15.115:0/2319253579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[46934]: cluster 2022-04-22T19:34:15.745315+0000 mgr.y (mgr.34107) 4403 : cluster [DBG] pgmap v5788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[46934]: audit 2022-04-22T19:34:15.840911+0000 mgr.y (mgr.34107) 4404 : audit [DBG] from='client.49930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[46934]: audit 2022-04-22T19:34:16.185312+0000 mgr.y (mgr.34107) 4405 : audit [DBG] from='client.40002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:17.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:17 smithi115 bash[46934]: audit 2022-04-22T19:34:16.609691+0000 mon.c (mon.1) 198 : audit [DBG] from='client.? 172.21.15.115:0/2319253579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:19 smithi121 bash[36902]: cluster 2022-04-22T19:34:17.746507+0000 mgr.y (mgr.34107) 4406 : cluster [DBG] pgmap v5789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:19 smithi121 bash[36902]: audit 2022-04-22T19:34:19.076571+0000 mon.a (mon.0) 7187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:19 smithi115 bash[45495]: cluster 2022-04-22T19:34:17.746507+0000 mgr.y (mgr.34107) 4406 : cluster [DBG] pgmap v5789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:19 smithi115 bash[45495]: audit 2022-04-22T19:34:19.076571+0000 mon.a (mon.0) 7187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:19 smithi115 bash[46934]: cluster 2022-04-22T19:34:17.746507+0000 mgr.y (mgr.34107) 4406 : cluster [DBG] pgmap v5789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:19 smithi115 bash[46934]: audit 2022-04-22T19:34:19.076571+0000 mon.a (mon.0) 7187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:21 smithi121 bash[36902]: cluster 2022-04-22T19:34:19.747567+0000 mgr.y (mgr.34107) 4407 : cluster [DBG] pgmap v5790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:21 smithi121 bash[36902]: audit 2022-04-22T19:34:20.577934+0000 mon.a (mon.0) 7188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:21 smithi115 bash[45495]: cluster 2022-04-22T19:34:19.747567+0000 mgr.y (mgr.34107) 4407 : cluster [DBG] pgmap v5790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:21 smithi115 bash[45495]: audit 2022-04-22T19:34:20.577934+0000 mon.a (mon.0) 7188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:21 smithi115 bash[46934]: cluster 2022-04-22T19:34:19.747567+0000 mgr.y (mgr.34107) 4407 : cluster [DBG] pgmap v5790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:21 smithi115 bash[46934]: audit 2022-04-22T19:34:20.577934+0000 mon.a (mon.0) 7188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:22.419 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:22.107Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:22.419 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:22 smithi115 bash[46934]: audit 2022-04-22T19:34:22.066722+0000 mon.a (mon.0) 7189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:22.420 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:22 smithi115 bash[45495]: audit 2022-04-22T19:34:22.066722+0000 mon.a (mon.0) 7189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:22 smithi121 bash[36902]: audit 2022-04-22T19:34:22.066722+0000 mon.a (mon.0) 7189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:23 smithi121 bash[36902]: cluster 2022-04-22T19:34:21.748597+0000 mgr.y (mgr.34107) 4408 : cluster [DBG] pgmap v5791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:23 smithi115 bash[45495]: cluster 2022-04-22T19:34:21.748597+0000 mgr.y (mgr.34107) 4408 : cluster [DBG] pgmap v5791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:23 smithi115 bash[46934]: cluster 2022-04-22T19:34:21.748597+0000 mgr.y (mgr.34107) 4408 : cluster [DBG] pgmap v5791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:23.979Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:24 smithi121 bash[36902]: audit 2022-04-22T19:34:24.080219+0000 mon.a (mon.0) 7190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:24.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:24 smithi115 bash[45495]: audit 2022-04-22T19:34:24.080219+0000 mon.a (mon.0) 7190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:24.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:24 smithi115 bash[46934]: audit 2022-04-22T19:34:24.080219+0000 mon.a (mon.0) 7190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:25 smithi121 bash[36902]: cluster 2022-04-22T19:34:23.749739+0000 mgr.y (mgr.34107) 4409 : cluster [DBG] pgmap v5792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:25 smithi115 bash[46934]: cluster 2022-04-22T19:34:23.749739+0000 mgr.y (mgr.34107) 4409 : cluster [DBG] pgmap v5792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:25.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:25 smithi115 bash[45495]: cluster 2022-04-22T19:34:23.749739+0000 mgr.y (mgr.34107) 4409 : cluster [DBG] pgmap v5792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:26 smithi121 bash[36902]: audit 2022-04-22T19:34:25.581057+0000 mon.a (mon.0) 7191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:26 smithi115 bash[45495]: audit 2022-04-22T19:34:25.581057+0000 mon.a (mon.0) 7191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:26 smithi115 bash[46934]: audit 2022-04-22T19:34:25.581057+0000 mon.a (mon.0) 7191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:27 smithi121 bash[36902]: cluster 2022-04-22T19:34:25.750816+0000 mgr.y (mgr.34107) 4410 : cluster [DBG] pgmap v5793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:27 smithi121 bash[36902]: audit 2022-04-22T19:34:27.069734+0000 mon.a (mon.0) 7192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:27 smithi115 bash[45495]: cluster 2022-04-22T19:34:25.750816+0000 mgr.y (mgr.34107) 4410 : cluster [DBG] pgmap v5793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:27 smithi115 bash[45495]: audit 2022-04-22T19:34:27.069734+0000 mon.a (mon.0) 7192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:27 smithi115 bash[46934]: cluster 2022-04-22T19:34:25.750816+0000 mgr.y (mgr.34107) 4410 : cluster [DBG] pgmap v5793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:27 smithi115 bash[46934]: audit 2022-04-22T19:34:27.069734+0000 mon.a (mon.0) 7192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:29 smithi121 bash[36902]: cluster 2022-04-22T19:34:27.751924+0000 mgr.y (mgr.34107) 4411 : cluster [DBG] pgmap v5794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:29 smithi121 bash[36902]: audit 2022-04-22T19:34:29.083856+0000 mon.a (mon.0) 7193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:29 smithi115 bash[45495]: cluster 2022-04-22T19:34:27.751924+0000 mgr.y (mgr.34107) 4411 : cluster [DBG] pgmap v5794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:29 smithi115 bash[45495]: audit 2022-04-22T19:34:29.083856+0000 mon.a (mon.0) 7193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:29 smithi115 bash[46934]: cluster 2022-04-22T19:34:27.751924+0000 mgr.y (mgr.34107) 4411 : cluster [DBG] pgmap v5794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:29 smithi115 bash[46934]: audit 2022-04-22T19:34:29.083856+0000 mon.a (mon.0) 7193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:31 smithi121 bash[36902]: cluster 2022-04-22T19:34:29.752979+0000 mgr.y (mgr.34107) 4412 : cluster [DBG] pgmap v5795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:31 smithi121 bash[36902]: audit 2022-04-22T19:34:30.584132+0000 mon.a (mon.0) 7194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:31 smithi115 bash[46934]: cluster 2022-04-22T19:34:29.752979+0000 mgr.y (mgr.34107) 4412 : cluster [DBG] pgmap v5795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:31 smithi115 bash[46934]: audit 2022-04-22T19:34:30.584132+0000 mon.a (mon.0) 7194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:31 smithi115 bash[45495]: cluster 2022-04-22T19:34:29.752979+0000 mgr.y (mgr.34107) 4412 : cluster [DBG] pgmap v5795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:31 smithi115 bash[45495]: audit 2022-04-22T19:34:30.584132+0000 mon.a (mon.0) 7194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:32.463 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:32.110Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:32.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:32 smithi115 bash[46934]: audit 2022-04-22T19:34:32.072838+0000 mon.a (mon.0) 7195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:32 smithi121 bash[36902]: audit 2022-04-22T19:34:32.072838+0000 mon.a (mon.0) 7195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:32.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:32 smithi115 bash[45495]: audit 2022-04-22T19:34:32.072838+0000 mon.a (mon.0) 7195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:33 smithi121 bash[36902]: cluster 2022-04-22T19:34:31.754100+0000 mgr.y (mgr.34107) 4413 : cluster [DBG] pgmap v5796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:33 smithi115 bash[45495]: cluster 2022-04-22T19:34:31.754100+0000 mgr.y (mgr.34107) 4413 : cluster [DBG] pgmap v5796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:33 smithi115 bash[46934]: cluster 2022-04-22T19:34:31.754100+0000 mgr.y (mgr.34107) 4413 : cluster [DBG] pgmap v5796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:33.979Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:34 smithi121 bash[36902]: audit 2022-04-22T19:34:34.087417+0000 mon.a (mon.0) 7196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:34.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:34 smithi115 bash[45495]: audit 2022-04-22T19:34:34.087417+0000 mon.a (mon.0) 7196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:34 smithi115 bash[46934]: audit 2022-04-22T19:34:34.087417+0000 mon.a (mon.0) 7196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:35 smithi121 bash[36902]: cluster 2022-04-22T19:34:33.755248+0000 mgr.y (mgr.34107) 4414 : cluster [DBG] pgmap v5797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:35 smithi115 bash[45495]: cluster 2022-04-22T19:34:33.755248+0000 mgr.y (mgr.34107) 4414 : cluster [DBG] pgmap v5797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:35 smithi115 bash[46934]: cluster 2022-04-22T19:34:33.755248+0000 mgr.y (mgr.34107) 4414 : cluster [DBG] pgmap v5797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:36 smithi121 bash[36902]: audit 2022-04-22T19:34:35.587287+0000 mon.a (mon.0) 7197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:36 smithi115 bash[46934]: audit 2022-04-22T19:34:35.587287+0000 mon.a (mon.0) 7197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:36.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:36 smithi115 bash[45495]: audit 2022-04-22T19:34:35.587287+0000 mon.a (mon.0) 7197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:37 smithi121 bash[36902]: cluster 2022-04-22T19:34:35.756344+0000 mgr.y (mgr.34107) 4415 : cluster [DBG] pgmap v5798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:37 smithi121 bash[36902]: audit 2022-04-22T19:34:37.075878+0000 mon.a (mon.0) 7198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:37 smithi115 bash[45495]: cluster 2022-04-22T19:34:35.756344+0000 mgr.y (mgr.34107) 4415 : cluster [DBG] pgmap v5798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:37 smithi115 bash[45495]: audit 2022-04-22T19:34:37.075878+0000 mon.a (mon.0) 7198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:37 smithi115 bash[46934]: cluster 2022-04-22T19:34:35.756344+0000 mgr.y (mgr.34107) 4415 : cluster [DBG] pgmap v5798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:37 smithi115 bash[46934]: audit 2022-04-22T19:34:37.075878+0000 mon.a (mon.0) 7198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:39 smithi121 bash[36902]: cluster 2022-04-22T19:34:37.757418+0000 mgr.y (mgr.34107) 4416 : cluster [DBG] pgmap v5799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:39 smithi121 bash[36902]: audit 2022-04-22T19:34:39.090979+0000 mon.a (mon.0) 7199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:39 smithi115 bash[45495]: cluster 2022-04-22T19:34:37.757418+0000 mgr.y (mgr.34107) 4416 : cluster [DBG] pgmap v5799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:39 smithi115 bash[45495]: audit 2022-04-22T19:34:39.090979+0000 mon.a (mon.0) 7199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:39 smithi115 bash[46934]: cluster 2022-04-22T19:34:37.757418+0000 mgr.y (mgr.34107) 4416 : cluster [DBG] pgmap v5799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:39 smithi115 bash[46934]: audit 2022-04-22T19:34:39.090979+0000 mon.a (mon.0) 7199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:41 smithi121 bash[36902]: cluster 2022-04-22T19:34:39.758510+0000 mgr.y (mgr.34107) 4417 : cluster [DBG] pgmap v5800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:41 smithi121 bash[36902]: audit 2022-04-22T19:34:40.590497+0000 mon.a (mon.0) 7200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:41 smithi115 bash[45495]: cluster 2022-04-22T19:34:39.758510+0000 mgr.y (mgr.34107) 4417 : cluster [DBG] pgmap v5800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:41 smithi115 bash[45495]: audit 2022-04-22T19:34:40.590497+0000 mon.a (mon.0) 7200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:41 smithi115 bash[46934]: cluster 2022-04-22T19:34:39.758510+0000 mgr.y (mgr.34107) 4417 : cluster [DBG] pgmap v5800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:41 smithi115 bash[46934]: audit 2022-04-22T19:34:40.590497+0000 mon.a (mon.0) 7200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:42.502 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:42.109Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:42 smithi121 bash[36902]: audit 2022-04-22T19:34:42.078831+0000 mon.a (mon.0) 7201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:42.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:42 smithi115 bash[45495]: audit 2022-04-22T19:34:42.078831+0000 mon.a (mon.0) 7201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:42 smithi115 bash[46934]: audit 2022-04-22T19:34:42.078831+0000 mon.a (mon.0) 7201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:43 smithi121 bash[36902]: cluster 2022-04-22T19:34:41.759570+0000 mgr.y (mgr.34107) 4418 : cluster [DBG] pgmap v5801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:43 smithi115 bash[45495]: cluster 2022-04-22T19:34:41.759570+0000 mgr.y (mgr.34107) 4418 : cluster [DBG] pgmap v5801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:43 smithi115 bash[46934]: cluster 2022-04-22T19:34:41.759570+0000 mgr.y (mgr.34107) 4418 : cluster [DBG] pgmap v5801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:43.981Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:44 smithi121 bash[36902]: audit 2022-04-22T19:34:44.094808+0000 mon.a (mon.0) 7202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:44.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:44 smithi115 bash[45495]: audit 2022-04-22T19:34:44.094808+0000 mon.a (mon.0) 7202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:44.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:44 smithi115 bash[46934]: audit 2022-04-22T19:34:44.094808+0000 mon.a (mon.0) 7202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:45 smithi115 bash[45495]: cluster 2022-04-22T19:34:43.760672+0000 mgr.y (mgr.34107) 4419 : cluster [DBG] pgmap v5802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:45 smithi115 bash[46934]: cluster 2022-04-22T19:34:43.760672+0000 mgr.y (mgr.34107) 4419 : cluster [DBG] pgmap v5802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:45 smithi121 bash[36902]: cluster 2022-04-22T19:34:43.760672+0000 mgr.y (mgr.34107) 4419 : cluster [DBG] pgmap v5802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:46.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:46 smithi115 bash[45495]: audit 2022-04-22T19:34:45.593490+0000 mon.a (mon.0) 7203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:46 smithi115 bash[46934]: audit 2022-04-22T19:34:45.593490+0000 mon.a (mon.0) 7203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:46.992 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:34:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:46 smithi121 bash[36902]: audit 2022-04-22T19:34:45.593490+0000 mon.a (mon.0) 7203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:47.337 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:34:47.337 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (117m) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:34:47.337 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (119m) 3m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (119m) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (118m) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:34:47.338 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (118m) 4m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:34:47.339 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (118m) 4m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:34:47.733 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:34:47.733 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:34:47.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:34:47.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:34:47.734 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:34:47.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:34:47.735 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:34:47.735 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:34:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:47 smithi115 bash[45495]: cluster 2022-04-22T19:34:45.761699+0000 mgr.y (mgr.34107) 4420 : cluster [DBG] pgmap v5803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:47 smithi115 bash[45495]: audit 2022-04-22T19:34:47.081792+0000 mon.a (mon.0) 7204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:47 smithi115 bash[46934]: cluster 2022-04-22T19:34:45.761699+0000 mgr.y (mgr.34107) 4420 : cluster [DBG] pgmap v5803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:47 smithi115 bash[46934]: audit 2022-04-22T19:34:47.081792+0000 mon.a (mon.0) 7204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:47 smithi121 bash[36902]: cluster 2022-04-22T19:34:45.761699+0000 mgr.y (mgr.34107) 4420 : cluster [DBG] pgmap v5803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:47 smithi121 bash[36902]: audit 2022-04-22T19:34:47.081792+0000 mon.a (mon.0) 7204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[46934]: audit 2022-04-22T19:34:46.965575+0000 mgr.y (mgr.34107) 4421 : audit [DBG] from='client.40014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[46934]: audit 2022-04-22T19:34:47.308631+0000 mgr.y (mgr.34107) 4422 : audit [DBG] from='client.40020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[46934]: audit 2022-04-22T19:34:47.722725+0000 mon.a (mon.0) 7205 : audit [DBG] from='client.? 172.21.15.115:0/1624822441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[45495]: audit 2022-04-22T19:34:46.965575+0000 mgr.y (mgr.34107) 4421 : audit [DBG] from='client.40014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[45495]: audit 2022-04-22T19:34:47.308631+0000 mgr.y (mgr.34107) 4422 : audit [DBG] from='client.40020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:48.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:48 smithi115 bash[45495]: audit 2022-04-22T19:34:47.722725+0000 mon.a (mon.0) 7205 : audit [DBG] from='client.? 172.21.15.115:0/1624822441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:48 smithi121 bash[36902]: audit 2022-04-22T19:34:46.965575+0000 mgr.y (mgr.34107) 4421 : audit [DBG] from='client.40014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:48 smithi121 bash[36902]: audit 2022-04-22T19:34:47.308631+0000 mgr.y (mgr.34107) 4422 : audit [DBG] from='client.40020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:34:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:48 smithi121 bash[36902]: audit 2022-04-22T19:34:47.722725+0000 mon.a (mon.0) 7205 : audit [DBG] from='client.? 172.21.15.115:0/1624822441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:34:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:49 smithi115 bash[45495]: cluster 2022-04-22T19:34:47.762761+0000 mgr.y (mgr.34107) 4423 : cluster [DBG] pgmap v5804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:49 smithi115 bash[45495]: audit 2022-04-22T19:34:49.098333+0000 mon.a (mon.0) 7206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:49 smithi115 bash[46934]: cluster 2022-04-22T19:34:47.762761+0000 mgr.y (mgr.34107) 4423 : cluster [DBG] pgmap v5804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:49 smithi115 bash[46934]: audit 2022-04-22T19:34:49.098333+0000 mon.a (mon.0) 7206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:49 smithi121 bash[36902]: cluster 2022-04-22T19:34:47.762761+0000 mgr.y (mgr.34107) 4423 : cluster [DBG] pgmap v5804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:49 smithi121 bash[36902]: audit 2022-04-22T19:34:49.098333+0000 mon.a (mon.0) 7206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:51 smithi115 bash[45495]: cluster 2022-04-22T19:34:49.763860+0000 mgr.y (mgr.34107) 4424 : cluster [DBG] pgmap v5805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:51.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:51 smithi115 bash[45495]: audit 2022-04-22T19:34:50.596985+0000 mon.a (mon.0) 7207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:51.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:51 smithi115 bash[46934]: cluster 2022-04-22T19:34:49.763860+0000 mgr.y (mgr.34107) 4424 : cluster [DBG] pgmap v5805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:51.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:51 smithi115 bash[46934]: audit 2022-04-22T19:34:50.596985+0000 mon.a (mon.0) 7207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:51 smithi121 bash[36902]: cluster 2022-04-22T19:34:49.763860+0000 mgr.y (mgr.34107) 4424 : cluster [DBG] pgmap v5805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:51 smithi121 bash[36902]: audit 2022-04-22T19:34:50.596985+0000 mon.a (mon.0) 7207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:52.109Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:52 smithi121 bash[36902]: audit 2022-04-22T19:34:52.083843+0000 mon.a (mon.0) 7208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:53.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:52 smithi115 bash[45495]: audit 2022-04-22T19:34:52.083843+0000 mon.a (mon.0) 7208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:53.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:52 smithi115 bash[46934]: audit 2022-04-22T19:34:52.083843+0000 mon.a (mon.0) 7208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:53.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:53 smithi115 bash[45495]: cluster 2022-04-22T19:34:51.764922+0000 mgr.y (mgr.34107) 4425 : cluster [DBG] pgmap v5806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:53.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:53 smithi115 bash[46934]: cluster 2022-04-22T19:34:51.764922+0000 mgr.y (mgr.34107) 4425 : cluster [DBG] pgmap v5806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:54.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:53 smithi121 bash[36902]: cluster 2022-04-22T19:34:51.764922+0000 mgr.y (mgr.34107) 4425 : cluster [DBG] pgmap v5806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:54.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:34:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:34:53.982Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:34:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:54 smithi121 bash[36902]: audit 2022-04-22T19:34:54.102192+0000 mon.a (mon.0) 7209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:54 smithi115 bash[46934]: audit 2022-04-22T19:34:54.102192+0000 mon.a (mon.0) 7209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:55.121 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:54 smithi115 bash[45495]: audit 2022-04-22T19:34:54.102192+0000 mon.a (mon.0) 7209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:34:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:55 smithi121 bash[36902]: cluster 2022-04-22T19:34:53.766079+0000 mgr.y (mgr.34107) 4426 : cluster [DBG] pgmap v5807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:55 smithi115 bash[45495]: cluster 2022-04-22T19:34:53.766079+0000 mgr.y (mgr.34107) 4426 : cluster [DBG] pgmap v5807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:55 smithi115 bash[46934]: cluster 2022-04-22T19:34:53.766079+0000 mgr.y (mgr.34107) 4426 : cluster [DBG] pgmap v5807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:56 smithi121 bash[36902]: audit 2022-04-22T19:34:55.600228+0000 mon.a (mon.0) 7210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:57.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:56 smithi121 bash[36902]: cluster 2022-04-22T19:34:55.767179+0000 mgr.y (mgr.34107) 4427 : cluster [DBG] pgmap v5808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:56 smithi121 bash[36902]: audit 2022-04-22T19:34:56.242645+0000 mon.a (mon.0) 7211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:34:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:56 smithi121 bash[36902]: audit 2022-04-22T19:34:56.245568+0000 mon.a (mon.0) 7212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:34:57.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:56 smithi121 bash[36902]: audit 2022-04-22T19:34:56.247375+0000 mon.a (mon.0) 7213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[45495]: audit 2022-04-22T19:34:55.600228+0000 mon.a (mon.0) 7210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[45495]: cluster 2022-04-22T19:34:55.767179+0000 mgr.y (mgr.34107) 4427 : cluster [DBG] pgmap v5808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[45495]: audit 2022-04-22T19:34:56.242645+0000 mon.a (mon.0) 7211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[45495]: audit 2022-04-22T19:34:56.245568+0000 mon.a (mon.0) 7212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[45495]: audit 2022-04-22T19:34:56.247375+0000 mon.a (mon.0) 7213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:34:57.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[46934]: audit 2022-04-22T19:34:55.600228+0000 mon.a (mon.0) 7210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:34:57.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[46934]: cluster 2022-04-22T19:34:55.767179+0000 mgr.y (mgr.34107) 4427 : cluster [DBG] pgmap v5808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:57.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[46934]: audit 2022-04-22T19:34:56.242645+0000 mon.a (mon.0) 7211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:34:57.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[46934]: audit 2022-04-22T19:34:56.245568+0000 mon.a (mon.0) 7212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:34:57.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:56 smithi115 bash[46934]: audit 2022-04-22T19:34:56.247375+0000 mon.a (mon.0) 7213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:34:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:57 smithi121 bash[36902]: audit 2022-04-22T19:34:56.623988+0000 mon.a (mon.0) 7214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:34:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:57 smithi121 bash[36902]: audit 2022-04-22T19:34:57.086780+0000 mon.a (mon.0) 7215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:57 smithi115 bash[45495]: audit 2022-04-22T19:34:56.623988+0000 mon.a (mon.0) 7214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:34:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:57 smithi115 bash[45495]: audit 2022-04-22T19:34:57.086780+0000 mon.a (mon.0) 7215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:57 smithi115 bash[46934]: audit 2022-04-22T19:34:56.623988+0000 mon.a (mon.0) 7214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:34:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:57 smithi115 bash[46934]: audit 2022-04-22T19:34:57.086780+0000 mon.a (mon.0) 7215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:34:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:58 smithi121 bash[36902]: cluster 2022-04-22T19:34:57.768245+0000 mgr.y (mgr.34107) 4428 : cluster [DBG] pgmap v5809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:58 smithi115 bash[45495]: cluster 2022-04-22T19:34:57.768245+0000 mgr.y (mgr.34107) 4428 : cluster [DBG] pgmap v5809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:34:59.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:58 smithi115 bash[46934]: cluster 2022-04-22T19:34:57.768245+0000 mgr.y (mgr.34107) 4428 : cluster [DBG] pgmap v5809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:34:59 smithi121 bash[36902]: audit 2022-04-22T19:34:59.105969+0000 mon.a (mon.0) 7216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:00.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:34:59 smithi115 bash[45495]: audit 2022-04-22T19:34:59.105969+0000 mon.a (mon.0) 7216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:00.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:34:59 smithi115 bash[46934]: audit 2022-04-22T19:34:59.105969+0000 mon.a (mon.0) 7216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:00 smithi121 bash[36902]: cluster 2022-04-22T19:34:59.769261+0000 mgr.y (mgr.34107) 4429 : cluster [DBG] pgmap v5810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:00 smithi121 bash[36902]: audit 2022-04-22T19:35:00.603413+0000 mon.a (mon.0) 7217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:00 smithi115 bash[45495]: cluster 2022-04-22T19:34:59.769261+0000 mgr.y (mgr.34107) 4429 : cluster [DBG] pgmap v5810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:00 smithi115 bash[45495]: audit 2022-04-22T19:35:00.603413+0000 mon.a (mon.0) 7217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:01.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:00 smithi115 bash[46934]: cluster 2022-04-22T19:34:59.769261+0000 mgr.y (mgr.34107) 4429 : cluster [DBG] pgmap v5810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:01.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:00 smithi115 bash[46934]: audit 2022-04-22T19:35:00.603413+0000 mon.a (mon.0) 7217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:02.110Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:03 smithi121 bash[36902]: cluster 2022-04-22T19:35:01.770383+0000 mgr.y (mgr.34107) 4430 : cluster [DBG] pgmap v5811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:03 smithi121 bash[36902]: audit 2022-04-22T19:35:02.089825+0000 mon.a (mon.0) 7218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:03.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:02 smithi115 bash[45495]: cluster 2022-04-22T19:35:01.770383+0000 mgr.y (mgr.34107) 4430 : cluster [DBG] pgmap v5811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:03.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:02 smithi115 bash[45495]: audit 2022-04-22T19:35:02.089825+0000 mon.a (mon.0) 7218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:03.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:02 smithi115 bash[46934]: cluster 2022-04-22T19:35:01.770383+0000 mgr.y (mgr.34107) 4430 : cluster [DBG] pgmap v5811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:03.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:02 smithi115 bash[46934]: audit 2022-04-22T19:35:02.089825+0000 mon.a (mon.0) 7218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:04.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:03.983Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:05 smithi121 bash[36902]: cluster 2022-04-22T19:35:03.771540+0000 mgr.y (mgr.34107) 4431 : cluster [DBG] pgmap v5812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:05 smithi121 bash[36902]: audit 2022-04-22T19:35:04.109506+0000 mon.a (mon.0) 7219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:05.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:04 smithi115 bash[46934]: cluster 2022-04-22T19:35:03.771540+0000 mgr.y (mgr.34107) 4431 : cluster [DBG] pgmap v5812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:05.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:04 smithi115 bash[46934]: audit 2022-04-22T19:35:04.109506+0000 mon.a (mon.0) 7219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:04 smithi115 bash[45495]: cluster 2022-04-22T19:35:03.771540+0000 mgr.y (mgr.34107) 4431 : cluster [DBG] pgmap v5812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:05.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:04 smithi115 bash[45495]: audit 2022-04-22T19:35:04.109506+0000 mon.a (mon.0) 7219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:06 smithi121 bash[36902]: audit 2022-04-22T19:35:05.606482+0000 mon.a (mon.0) 7220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:06.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:06 smithi115 bash[45495]: audit 2022-04-22T19:35:05.606482+0000 mon.a (mon.0) 7220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:06.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:06 smithi115 bash[46934]: audit 2022-04-22T19:35:05.606482+0000 mon.a (mon.0) 7220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:07.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:07 smithi115 bash[45495]: cluster 2022-04-22T19:35:05.772520+0000 mgr.y (mgr.34107) 4432 : cluster [DBG] pgmap v5813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:07.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:07 smithi115 bash[46934]: cluster 2022-04-22T19:35:05.772520+0000 mgr.y (mgr.34107) 4432 : cluster [DBG] pgmap v5813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:07 smithi121 bash[36902]: cluster 2022-04-22T19:35:05.772520+0000 mgr.y (mgr.34107) 4432 : cluster [DBG] pgmap v5813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:08.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:08 smithi115 bash[45495]: audit 2022-04-22T19:35:07.092537+0000 mon.a (mon.0) 7221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:08.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:08 smithi115 bash[46934]: audit 2022-04-22T19:35:07.092537+0000 mon.a (mon.0) 7221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:08 smithi121 bash[36902]: audit 2022-04-22T19:35:07.092537+0000 mon.a (mon.0) 7221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:09.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:09 smithi115 bash[45495]: cluster 2022-04-22T19:35:07.773580+0000 mgr.y (mgr.34107) 4433 : cluster [DBG] pgmap v5814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:09 smithi115 bash[46934]: cluster 2022-04-22T19:35:07.773580+0000 mgr.y (mgr.34107) 4433 : cluster [DBG] pgmap v5814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:09 smithi121 bash[36902]: cluster 2022-04-22T19:35:07.773580+0000 mgr.y (mgr.34107) 4433 : cluster [DBG] pgmap v5814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:10.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:10 smithi115 bash[45495]: audit 2022-04-22T19:35:09.113354+0000 mon.a (mon.0) 7222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:10.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:10 smithi115 bash[46934]: audit 2022-04-22T19:35:09.113354+0000 mon.a (mon.0) 7222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:10 smithi121 bash[36902]: audit 2022-04-22T19:35:09.113354+0000 mon.a (mon.0) 7222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:11 smithi115 bash[45495]: cluster 2022-04-22T19:35:09.774695+0000 mgr.y (mgr.34107) 4434 : cluster [DBG] pgmap v5815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:11.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:11 smithi115 bash[45495]: audit 2022-04-22T19:35:10.609547+0000 mon.a (mon.0) 7223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:11.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:11 smithi115 bash[46934]: cluster 2022-04-22T19:35:09.774695+0000 mgr.y (mgr.34107) 4434 : cluster [DBG] pgmap v5815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:11.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:11 smithi115 bash[46934]: audit 2022-04-22T19:35:10.609547+0000 mon.a (mon.0) 7223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:11 smithi121 bash[36902]: cluster 2022-04-22T19:35:09.774695+0000 mgr.y (mgr.34107) 4434 : cluster [DBG] pgmap v5815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:11 smithi121 bash[36902]: audit 2022-04-22T19:35:10.609547+0000 mon.a (mon.0) 7223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:12.111Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:13 smithi115 bash[45495]: cluster 2022-04-22T19:35:11.775861+0000 mgr.y (mgr.34107) 4435 : cluster [DBG] pgmap v5816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:13 smithi115 bash[45495]: audit 2022-04-22T19:35:12.095537+0000 mon.a (mon.0) 7224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:13 smithi115 bash[46934]: cluster 2022-04-22T19:35:11.775861+0000 mgr.y (mgr.34107) 4435 : cluster [DBG] pgmap v5816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:13 smithi115 bash[46934]: audit 2022-04-22T19:35:12.095537+0000 mon.a (mon.0) 7224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:13 smithi121 bash[36902]: cluster 2022-04-22T19:35:11.775861+0000 mgr.y (mgr.34107) 4435 : cluster [DBG] pgmap v5816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:13 smithi121 bash[36902]: audit 2022-04-22T19:35:12.095537+0000 mon.a (mon.0) 7224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:13.984Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:15.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[45495]: cluster 2022-04-22T19:35:13.776984+0000 mgr.y (mgr.34107) 4436 : cluster [DBG] pgmap v5817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[45495]: audit 2022-04-22T19:35:14.116875+0000 mon.a (mon.0) 7225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[45495]: audit 2022-04-22T19:35:14.746071+0000 mon.a (mon.0) 7226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:35:15.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[45495]: audit 2022-04-22T19:35:14.751111+0000 mon.a (mon.0) 7227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:35:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[46934]: cluster 2022-04-22T19:35:13.776984+0000 mgr.y (mgr.34107) 4436 : cluster [DBG] pgmap v5817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:15.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[46934]: audit 2022-04-22T19:35:14.116875+0000 mon.a (mon.0) 7225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[46934]: audit 2022-04-22T19:35:14.746071+0000 mon.a (mon.0) 7226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:35:15.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:15 smithi115 bash[46934]: audit 2022-04-22T19:35:14.751111+0000 mon.a (mon.0) 7227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:35:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:15 smithi121 bash[36902]: cluster 2022-04-22T19:35:13.776984+0000 mgr.y (mgr.34107) 4436 : cluster [DBG] pgmap v5817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:15 smithi121 bash[36902]: audit 2022-04-22T19:35:14.116875+0000 mon.a (mon.0) 7225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:15 smithi121 bash[36902]: audit 2022-04-22T19:35:14.746071+0000 mon.a (mon.0) 7226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:35:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:15 smithi121 bash[36902]: audit 2022-04-22T19:35:14.751111+0000 mon.a (mon.0) 7227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:35:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:16 smithi121 bash[36902]: audit 2022-04-22T19:35:15.612756+0000 mon.a (mon.0) 7228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:16 smithi115 bash[45495]: audit 2022-04-22T19:35:15.612756+0000 mon.a (mon.0) 7228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:16 smithi115 bash[46934]: audit 2022-04-22T19:35:15.612756+0000 mon.a (mon.0) 7228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:17 smithi121 bash[36902]: cluster 2022-04-22T19:35:15.778065+0000 mgr.y (mgr.34107) 4437 : cluster [DBG] pgmap v5818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:17 smithi115 bash[46934]: cluster 2022-04-22T19:35:15.778065+0000 mgr.y (mgr.34107) 4437 : cluster [DBG] pgmap v5818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:17.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:17 smithi115 bash[45495]: cluster 2022-04-22T19:35:15.778065+0000 mgr.y (mgr.34107) 4437 : cluster [DBG] pgmap v5818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:18.110 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (118m) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (119m) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:35:18.455 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (119m) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (119m) 5m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:35:18.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:35:18.457 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:35:18.457 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:35:18.457 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:35:18.457 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (119m) 5m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:35:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:18 smithi121 bash[36902]: audit 2022-04-22T19:35:17.098612+0000 mon.a (mon.0) 7229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:18 smithi115 bash[46934]: audit 2022-04-22T19:35:17.098612+0000 mon.a (mon.0) 7229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:18.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:18 smithi115 bash[45495]: audit 2022-04-22T19:35:17.098612+0000 mon.a (mon.0) 7229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:18.860 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:35:18.861 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:19 smithi121 bash[36902]: cluster 2022-04-22T19:35:17.779386+0000 mgr.y (mgr.34107) 4438 : cluster [DBG] pgmap v5819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:19 smithi121 bash[36902]: audit 2022-04-22T19:35:18.084010+0000 mgr.y (mgr.34107) 4439 : audit [DBG] from='client.49966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:19 smithi121 bash[36902]: audit 2022-04-22T19:35:18.424119+0000 mgr.y (mgr.34107) 4440 : audit [DBG] from='client.40038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:19 smithi121 bash[36902]: audit 2022-04-22T19:35:18.849399+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.115:0/1613251949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[45495]: cluster 2022-04-22T19:35:17.779386+0000 mgr.y (mgr.34107) 4438 : cluster [DBG] pgmap v5819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[45495]: audit 2022-04-22T19:35:18.084010+0000 mgr.y (mgr.34107) 4439 : audit [DBG] from='client.49966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[45495]: audit 2022-04-22T19:35:18.424119+0000 mgr.y (mgr.34107) 4440 : audit [DBG] from='client.40038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[45495]: audit 2022-04-22T19:35:18.849399+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.115:0/1613251949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[46934]: cluster 2022-04-22T19:35:17.779386+0000 mgr.y (mgr.34107) 4438 : cluster [DBG] pgmap v5819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[46934]: audit 2022-04-22T19:35:18.084010+0000 mgr.y (mgr.34107) 4439 : audit [DBG] from='client.49966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[46934]: audit 2022-04-22T19:35:18.424119+0000 mgr.y (mgr.34107) 4440 : audit [DBG] from='client.40038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:19.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:19 smithi115 bash[46934]: audit 2022-04-22T19:35:18.849399+0000 mon.c (mon.1) 199 : audit [DBG] from='client.? 172.21.15.115:0/1613251949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:20 smithi121 bash[36902]: audit 2022-04-22T19:35:19.120322+0000 mon.a (mon.0) 7230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:20.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:20 smithi115 bash[45495]: audit 2022-04-22T19:35:19.120322+0000 mon.a (mon.0) 7230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:20.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:20 smithi115 bash[46934]: audit 2022-04-22T19:35:19.120322+0000 mon.a (mon.0) 7230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:21 smithi121 bash[36902]: cluster 2022-04-22T19:35:19.780524+0000 mgr.y (mgr.34107) 4441 : cluster [DBG] pgmap v5820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:21 smithi121 bash[36902]: audit 2022-04-22T19:35:20.616006+0000 mon.a (mon.0) 7231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:21 smithi115 bash[45495]: cluster 2022-04-22T19:35:19.780524+0000 mgr.y (mgr.34107) 4441 : cluster [DBG] pgmap v5820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:21 smithi115 bash[45495]: audit 2022-04-22T19:35:20.616006+0000 mon.a (mon.0) 7231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:21 smithi115 bash[46934]: cluster 2022-04-22T19:35:19.780524+0000 mgr.y (mgr.34107) 4441 : cluster [DBG] pgmap v5820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:21.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:21 smithi115 bash[46934]: audit 2022-04-22T19:35:20.616006+0000 mon.a (mon.0) 7231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:22.112Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:23 smithi121 bash[36902]: cluster 2022-04-22T19:35:21.781639+0000 mgr.y (mgr.34107) 4442 : cluster [DBG] pgmap v5821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:23 smithi121 bash[36902]: audit 2022-04-22T19:35:22.101482+0000 mon.a (mon.0) 7232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:23 smithi115 bash[45495]: cluster 2022-04-22T19:35:21.781639+0000 mgr.y (mgr.34107) 4442 : cluster [DBG] pgmap v5821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:23 smithi115 bash[45495]: audit 2022-04-22T19:35:22.101482+0000 mon.a (mon.0) 7232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:23 smithi115 bash[46934]: cluster 2022-04-22T19:35:21.781639+0000 mgr.y (mgr.34107) 4442 : cluster [DBG] pgmap v5821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:23.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:23 smithi115 bash[46934]: audit 2022-04-22T19:35:22.101482+0000 mon.a (mon.0) 7232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:23.983Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:25 smithi121 bash[36902]: cluster 2022-04-22T19:35:23.782785+0000 mgr.y (mgr.34107) 4443 : cluster [DBG] pgmap v5822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:25 smithi121 bash[36902]: audit 2022-04-22T19:35:24.123791+0000 mon.a (mon.0) 7233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:25 smithi115 bash[45495]: cluster 2022-04-22T19:35:23.782785+0000 mgr.y (mgr.34107) 4443 : cluster [DBG] pgmap v5822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:25 smithi115 bash[45495]: audit 2022-04-22T19:35:24.123791+0000 mon.a (mon.0) 7233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:25 smithi115 bash[46934]: cluster 2022-04-22T19:35:23.782785+0000 mgr.y (mgr.34107) 4443 : cluster [DBG] pgmap v5822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:25 smithi115 bash[46934]: audit 2022-04-22T19:35:24.123791+0000 mon.a (mon.0) 7233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:26 smithi121 bash[36902]: audit 2022-04-22T19:35:25.619248+0000 mon.a (mon.0) 7234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:26 smithi115 bash[45495]: audit 2022-04-22T19:35:25.619248+0000 mon.a (mon.0) 7234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:26 smithi115 bash[46934]: audit 2022-04-22T19:35:25.619248+0000 mon.a (mon.0) 7234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:27 smithi121 bash[36902]: cluster 2022-04-22T19:35:25.783927+0000 mgr.y (mgr.34107) 4444 : cluster [DBG] pgmap v5823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:27 smithi115 bash[45495]: cluster 2022-04-22T19:35:25.783927+0000 mgr.y (mgr.34107) 4444 : cluster [DBG] pgmap v5823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:27 smithi115 bash[46934]: cluster 2022-04-22T19:35:25.783927+0000 mgr.y (mgr.34107) 4444 : cluster [DBG] pgmap v5823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:28 smithi121 bash[36902]: audit 2022-04-22T19:35:27.106222+0000 mon.a (mon.0) 7235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:28 smithi115 bash[45495]: audit 2022-04-22T19:35:27.106222+0000 mon.a (mon.0) 7235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:28 smithi115 bash[46934]: audit 2022-04-22T19:35:27.106222+0000 mon.a (mon.0) 7235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:29 smithi121 bash[36902]: cluster 2022-04-22T19:35:27.785018+0000 mgr.y (mgr.34107) 4445 : cluster [DBG] pgmap v5824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:29 smithi115 bash[45495]: cluster 2022-04-22T19:35:27.785018+0000 mgr.y (mgr.34107) 4445 : cluster [DBG] pgmap v5824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:29 smithi115 bash[46934]: cluster 2022-04-22T19:35:27.785018+0000 mgr.y (mgr.34107) 4445 : cluster [DBG] pgmap v5824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:30 smithi121 bash[36902]: audit 2022-04-22T19:35:29.127251+0000 mon.a (mon.0) 7236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:30 smithi115 bash[45495]: audit 2022-04-22T19:35:29.127251+0000 mon.a (mon.0) 7236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:30 smithi115 bash[46934]: audit 2022-04-22T19:35:29.127251+0000 mon.a (mon.0) 7236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:31 smithi121 bash[36902]: cluster 2022-04-22T19:35:29.786154+0000 mgr.y (mgr.34107) 4446 : cluster [DBG] pgmap v5825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:31 smithi121 bash[36902]: audit 2022-04-22T19:35:30.622456+0000 mon.a (mon.0) 7237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:31 smithi115 bash[45495]: cluster 2022-04-22T19:35:29.786154+0000 mgr.y (mgr.34107) 4446 : cluster [DBG] pgmap v5825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:31 smithi115 bash[45495]: audit 2022-04-22T19:35:30.622456+0000 mon.a (mon.0) 7237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:31 smithi115 bash[46934]: cluster 2022-04-22T19:35:29.786154+0000 mgr.y (mgr.34107) 4446 : cluster [DBG] pgmap v5825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:31 smithi115 bash[46934]: audit 2022-04-22T19:35:30.622456+0000 mon.a (mon.0) 7237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:32 smithi121 bash[36902]: audit 2022-04-22T19:35:32.108736+0000 mon.a (mon.0) 7238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:32.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:32 smithi115 bash[45495]: audit 2022-04-22T19:35:32.108736+0000 mon.a (mon.0) 7238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:32.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:32 smithi115 bash[46934]: audit 2022-04-22T19:35:32.108736+0000 mon.a (mon.0) 7238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:32.114Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:33 smithi121 bash[36902]: cluster 2022-04-22T19:35:31.787326+0000 mgr.y (mgr.34107) 4447 : cluster [DBG] pgmap v5826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:33 smithi115 bash[45495]: cluster 2022-04-22T19:35:31.787326+0000 mgr.y (mgr.34107) 4447 : cluster [DBG] pgmap v5826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:33 smithi115 bash[46934]: cluster 2022-04-22T19:35:31.787326+0000 mgr.y (mgr.34107) 4447 : cluster [DBG] pgmap v5826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:34 smithi115 bash[46934]: audit 2022-04-22T19:35:34.130856+0000 mon.a (mon.0) 7239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:34.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:34 smithi115 bash[45495]: audit 2022-04-22T19:35:34.130856+0000 mon.a (mon.0) 7239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:34.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:33.984Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:34 smithi121 bash[36902]: audit 2022-04-22T19:35:34.130856+0000 mon.a (mon.0) 7239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:35 smithi121 bash[36902]: cluster 2022-04-22T19:35:33.788542+0000 mgr.y (mgr.34107) 4448 : cluster [DBG] pgmap v5827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:35 smithi115 bash[45495]: cluster 2022-04-22T19:35:33.788542+0000 mgr.y (mgr.34107) 4448 : cluster [DBG] pgmap v5827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:35 smithi115 bash[46934]: cluster 2022-04-22T19:35:33.788542+0000 mgr.y (mgr.34107) 4448 : cluster [DBG] pgmap v5827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:36 smithi121 bash[36902]: audit 2022-04-22T19:35:35.625724+0000 mon.a (mon.0) 7240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:36.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:36 smithi115 bash[46934]: audit 2022-04-22T19:35:35.625724+0000 mon.a (mon.0) 7240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:36 smithi115 bash[45495]: audit 2022-04-22T19:35:35.625724+0000 mon.a (mon.0) 7240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:37 smithi121 bash[36902]: cluster 2022-04-22T19:35:35.789552+0000 mgr.y (mgr.34107) 4449 : cluster [DBG] pgmap v5828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:37 smithi121 bash[36902]: audit 2022-04-22T19:35:37.111752+0000 mon.a (mon.0) 7241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:37 smithi115 bash[45495]: cluster 2022-04-22T19:35:35.789552+0000 mgr.y (mgr.34107) 4449 : cluster [DBG] pgmap v5828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:37.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:37 smithi115 bash[45495]: audit 2022-04-22T19:35:37.111752+0000 mon.a (mon.0) 7241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:37 smithi115 bash[46934]: cluster 2022-04-22T19:35:35.789552+0000 mgr.y (mgr.34107) 4449 : cluster [DBG] pgmap v5828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:37 smithi115 bash[46934]: audit 2022-04-22T19:35:37.111752+0000 mon.a (mon.0) 7241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:39 smithi121 bash[36902]: cluster 2022-04-22T19:35:37.790689+0000 mgr.y (mgr.34107) 4450 : cluster [DBG] pgmap v5829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:39 smithi121 bash[36902]: audit 2022-04-22T19:35:39.134417+0000 mon.a (mon.0) 7242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:39 smithi115 bash[46934]: cluster 2022-04-22T19:35:37.790689+0000 mgr.y (mgr.34107) 4450 : cluster [DBG] pgmap v5829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:39 smithi115 bash[46934]: audit 2022-04-22T19:35:39.134417+0000 mon.a (mon.0) 7242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:39 smithi115 bash[45495]: cluster 2022-04-22T19:35:37.790689+0000 mgr.y (mgr.34107) 4450 : cluster [DBG] pgmap v5829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:39.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:39 smithi115 bash[45495]: audit 2022-04-22T19:35:39.134417+0000 mon.a (mon.0) 7242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:41 smithi121 bash[36902]: cluster 2022-04-22T19:35:39.791795+0000 mgr.y (mgr.34107) 4451 : cluster [DBG] pgmap v5830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:41 smithi121 bash[36902]: audit 2022-04-22T19:35:40.629085+0000 mon.a (mon.0) 7243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:41.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:41 smithi115 bash[46934]: cluster 2022-04-22T19:35:39.791795+0000 mgr.y (mgr.34107) 4451 : cluster [DBG] pgmap v5830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:41.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:41 smithi115 bash[46934]: audit 2022-04-22T19:35:40.629085+0000 mon.a (mon.0) 7243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:41 smithi115 bash[45495]: cluster 2022-04-22T19:35:39.791795+0000 mgr.y (mgr.34107) 4451 : cluster [DBG] pgmap v5830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:41 smithi115 bash[45495]: audit 2022-04-22T19:35:40.629085+0000 mon.a (mon.0) 7243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:42 smithi121 bash[36902]: audit 2022-04-22T19:35:42.115101+0000 mon.a (mon.0) 7244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:42.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:42 smithi115 bash[46934]: audit 2022-04-22T19:35:42.115101+0000 mon.a (mon.0) 7244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:42.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:42.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:42 smithi115 bash[45495]: audit 2022-04-22T19:35:42.115101+0000 mon.a (mon.0) 7244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:43 smithi121 bash[36902]: cluster 2022-04-22T19:35:41.792971+0000 mgr.y (mgr.34107) 4452 : cluster [DBG] pgmap v5831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:43 smithi115 bash[45495]: cluster 2022-04-22T19:35:41.792971+0000 mgr.y (mgr.34107) 4452 : cluster [DBG] pgmap v5831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:43 smithi115 bash[46934]: cluster 2022-04-22T19:35:41.792971+0000 mgr.y (mgr.34107) 4452 : cluster [DBG] pgmap v5831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:44 smithi115 bash[45495]: audit 2022-04-22T19:35:44.138333+0000 mon.a (mon.0) 7245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:44.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:44 smithi115 bash[46934]: audit 2022-04-22T19:35:44.138333+0000 mon.a (mon.0) 7245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:43.985Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:44 smithi121 bash[36902]: audit 2022-04-22T19:35:44.138333+0000 mon.a (mon.0) 7245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:45 smithi121 bash[36902]: cluster 2022-04-22T19:35:43.794176+0000 mgr.y (mgr.34107) 4453 : cluster [DBG] pgmap v5832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:45 smithi115 bash[46934]: cluster 2022-04-22T19:35:43.794176+0000 mgr.y (mgr.34107) 4453 : cluster [DBG] pgmap v5832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:45 smithi115 bash[45495]: cluster 2022-04-22T19:35:43.794176+0000 mgr.y (mgr.34107) 4453 : cluster [DBG] pgmap v5832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:46 smithi121 bash[36902]: audit 2022-04-22T19:35:45.632195+0000 mon.a (mon.0) 7246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:46 smithi115 bash[45495]: audit 2022-04-22T19:35:45.632195+0000 mon.a (mon.0) 7246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:46 smithi115 bash[46934]: audit 2022-04-22T19:35:45.632195+0000 mon.a (mon.0) 7246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:47 smithi121 bash[36902]: cluster 2022-04-22T19:35:45.795352+0000 mgr.y (mgr.34107) 4454 : cluster [DBG] pgmap v5833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:47 smithi121 bash[36902]: audit 2022-04-22T19:35:47.118237+0000 mon.a (mon.0) 7247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:47 smithi115 bash[45495]: cluster 2022-04-22T19:35:45.795352+0000 mgr.y (mgr.34107) 4454 : cluster [DBG] pgmap v5833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:47.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:47 smithi115 bash[45495]: audit 2022-04-22T19:35:47.118237+0000 mon.a (mon.0) 7247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:47 smithi115 bash[46934]: cluster 2022-04-22T19:35:45.795352+0000 mgr.y (mgr.34107) 4454 : cluster [DBG] pgmap v5833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:47 smithi115 bash[46934]: audit 2022-04-22T19:35:47.118237+0000 mon.a (mon.0) 7247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:49.234 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:35:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:49 smithi121 bash[36902]: cluster 2022-04-22T19:35:47.796618+0000 mgr.y (mgr.34107) 4455 : cluster [DBG] pgmap v5834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:49 smithi121 bash[36902]: audit 2022-04-22T19:35:49.142162+0000 mon.a (mon.0) 7248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:49 smithi115 bash[45495]: cluster 2022-04-22T19:35:47.796618+0000 mgr.y (mgr.34107) 4455 : cluster [DBG] pgmap v5834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:49 smithi115 bash[45495]: audit 2022-04-22T19:35:49.142162+0000 mon.a (mon.0) 7248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:49 smithi115 bash[46934]: cluster 2022-04-22T19:35:47.796618+0000 mgr.y (mgr.34107) 4455 : cluster [DBG] pgmap v5834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:49 smithi115 bash[46934]: audit 2022-04-22T19:35:49.142162+0000 mon.a (mon.0) 7248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:49.579 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:35:49.579 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (118m) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:35:49.579 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:35:49.579 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (119m) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (119m) 5m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:35:49.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:35:49.581 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (119m) 5m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:49.978 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:35:49.979 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:35:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:50 smithi121 bash[36902]: audit 2022-04-22T19:35:49.207386+0000 mgr.y (mgr.34107) 4456 : audit [DBG] from='client.40050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:50 smithi121 bash[36902]: audit 2022-04-22T19:35:49.548381+0000 mgr.y (mgr.34107) 4457 : audit [DBG] from='client.40056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:50 smithi121 bash[36902]: audit 2022-04-22T19:35:49.967184+0000 mon.a (mon.0) 7249 : audit [DBG] from='client.? 172.21.15.115:0/357874616' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[45495]: audit 2022-04-22T19:35:49.207386+0000 mgr.y (mgr.34107) 4456 : audit [DBG] from='client.40050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[45495]: audit 2022-04-22T19:35:49.548381+0000 mgr.y (mgr.34107) 4457 : audit [DBG] from='client.40056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[45495]: audit 2022-04-22T19:35:49.967184+0000 mon.a (mon.0) 7249 : audit [DBG] from='client.? 172.21.15.115:0/357874616' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[46934]: audit 2022-04-22T19:35:49.207386+0000 mgr.y (mgr.34107) 4456 : audit [DBG] from='client.40050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[46934]: audit 2022-04-22T19:35:49.548381+0000 mgr.y (mgr.34107) 4457 : audit [DBG] from='client.40056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:35:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:50 smithi115 bash[46934]: audit 2022-04-22T19:35:49.967184+0000 mon.a (mon.0) 7249 : audit [DBG] from='client.? 172.21.15.115:0/357874616' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:35:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:51 smithi121 bash[36902]: cluster 2022-04-22T19:35:49.797707+0000 mgr.y (mgr.34107) 4458 : cluster [DBG] pgmap v5835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:51 smithi121 bash[36902]: audit 2022-04-22T19:35:50.635129+0000 mon.a (mon.0) 7250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:51 smithi115 bash[45495]: cluster 2022-04-22T19:35:49.797707+0000 mgr.y (mgr.34107) 4458 : cluster [DBG] pgmap v5835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:51 smithi115 bash[45495]: audit 2022-04-22T19:35:50.635129+0000 mon.a (mon.0) 7250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:51 smithi115 bash[46934]: cluster 2022-04-22T19:35:49.797707+0000 mgr.y (mgr.34107) 4458 : cluster [DBG] pgmap v5835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:51 smithi115 bash[46934]: audit 2022-04-22T19:35:50.635129+0000 mon.a (mon.0) 7250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:52 smithi121 bash[36902]: audit 2022-04-22T19:35:52.121286+0000 mon.a (mon.0) 7251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:52 smithi115 bash[46934]: audit 2022-04-22T19:35:52.121286+0000 mon.a (mon.0) 7251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:52.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:52.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:52 smithi115 bash[45495]: audit 2022-04-22T19:35:52.121286+0000 mon.a (mon.0) 7251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:53 smithi121 bash[36902]: cluster 2022-04-22T19:35:51.798882+0000 mgr.y (mgr.34107) 4459 : cluster [DBG] pgmap v5836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:53 smithi115 bash[45495]: cluster 2022-04-22T19:35:51.798882+0000 mgr.y (mgr.34107) 4459 : cluster [DBG] pgmap v5836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:53 smithi115 bash[46934]: cluster 2022-04-22T19:35:51.798882+0000 mgr.y (mgr.34107) 4459 : cluster [DBG] pgmap v5836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:54.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:54 smithi115 bash[45495]: audit 2022-04-22T19:35:54.145868+0000 mon.a (mon.0) 7252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:54.334 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:35:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:35:53.986Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:35:54.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:54 smithi115 bash[46934]: audit 2022-04-22T19:35:54.145868+0000 mon.a (mon.0) 7252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:54 smithi121 bash[36902]: audit 2022-04-22T19:35:54.145868+0000 mon.a (mon.0) 7252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:55 smithi121 bash[36902]: cluster 2022-04-22T19:35:53.800217+0000 mgr.y (mgr.34107) 4460 : cluster [DBG] pgmap v5837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:55 smithi115 bash[45495]: cluster 2022-04-22T19:35:53.800217+0000 mgr.y (mgr.34107) 4460 : cluster [DBG] pgmap v5837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:55 smithi115 bash[46934]: cluster 2022-04-22T19:35:53.800217+0000 mgr.y (mgr.34107) 4460 : cluster [DBG] pgmap v5837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:56 smithi121 bash[36902]: audit 2022-04-22T19:35:55.638442+0000 mon.a (mon.0) 7253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:56.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:56 smithi115 bash[45495]: audit 2022-04-22T19:35:55.638442+0000 mon.a (mon.0) 7253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:56.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:56 smithi115 bash[46934]: audit 2022-04-22T19:35:55.638442+0000 mon.a (mon.0) 7253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:35:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: cluster 2022-04-22T19:35:55.801349+0000 mgr.y (mgr.34107) 4461 : cluster [DBG] pgmap v5838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:57.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: audit 2022-04-22T19:35:56.634301+0000 mon.a (mon.0) 7254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:35:57.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: audit 2022-04-22T19:35:56.637108+0000 mon.a (mon.0) 7255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:35:57.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: audit 2022-04-22T19:35:56.638714+0000 mon.a (mon.0) 7256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:35:57.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: audit 2022-04-22T19:35:56.994637+0000 mon.a (mon.0) 7257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:35:57.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:57 smithi121 bash[36902]: audit 2022-04-22T19:35:57.123911+0000 mon.a (mon.0) 7258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: cluster 2022-04-22T19:35:55.801349+0000 mgr.y (mgr.34107) 4461 : cluster [DBG] pgmap v5838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: audit 2022-04-22T19:35:56.634301+0000 mon.a (mon.0) 7254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: audit 2022-04-22T19:35:56.637108+0000 mon.a (mon.0) 7255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: audit 2022-04-22T19:35:56.638714+0000 mon.a (mon.0) 7256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: audit 2022-04-22T19:35:56.994637+0000 mon.a (mon.0) 7257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:35:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[45495]: audit 2022-04-22T19:35:57.123911+0000 mon.a (mon.0) 7258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: cluster 2022-04-22T19:35:55.801349+0000 mgr.y (mgr.34107) 4461 : cluster [DBG] pgmap v5838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: audit 2022-04-22T19:35:56.634301+0000 mon.a (mon.0) 7254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: audit 2022-04-22T19:35:56.637108+0000 mon.a (mon.0) 7255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: audit 2022-04-22T19:35:56.638714+0000 mon.a (mon.0) 7256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: audit 2022-04-22T19:35:56.994637+0000 mon.a (mon.0) 7257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:35:57.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:57 smithi115 bash[46934]: audit 2022-04-22T19:35:57.123911+0000 mon.a (mon.0) 7258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:35:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:59 smithi121 bash[36902]: cluster 2022-04-22T19:35:57.802447+0000 mgr.y (mgr.34107) 4462 : cluster [DBG] pgmap v5839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:35:59 smithi121 bash[36902]: audit 2022-04-22T19:35:59.149618+0000 mon.a (mon.0) 7259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:59 smithi115 bash[45495]: cluster 2022-04-22T19:35:57.802447+0000 mgr.y (mgr.34107) 4462 : cluster [DBG] pgmap v5839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:35:59 smithi115 bash[45495]: audit 2022-04-22T19:35:59.149618+0000 mon.a (mon.0) 7259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:35:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:59 smithi115 bash[46934]: cluster 2022-04-22T19:35:57.802447+0000 mgr.y (mgr.34107) 4462 : cluster [DBG] pgmap v5839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:35:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:35:59 smithi115 bash[46934]: audit 2022-04-22T19:35:59.149618+0000 mon.a (mon.0) 7259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:01 smithi115 bash[45495]: cluster 2022-04-22T19:35:59.803571+0000 mgr.y (mgr.34107) 4463 : cluster [DBG] pgmap v5840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:01 smithi115 bash[45495]: audit 2022-04-22T19:36:00.641450+0000 mon.a (mon.0) 7260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:01 smithi115 bash[46934]: cluster 2022-04-22T19:35:59.803571+0000 mgr.y (mgr.34107) 4463 : cluster [DBG] pgmap v5840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:01.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:01 smithi115 bash[46934]: audit 2022-04-22T19:36:00.641450+0000 mon.a (mon.0) 7260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:01 smithi121 bash[36902]: cluster 2022-04-22T19:35:59.803571+0000 mgr.y (mgr.34107) 4463 : cluster [DBG] pgmap v5840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:01 smithi121 bash[36902]: audit 2022-04-22T19:36:00.641450+0000 mon.a (mon.0) 7260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:02 smithi115 bash[45495]: audit 2022-04-22T19:36:02.127001+0000 mon.a (mon.0) 7261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:02.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:02.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:02 smithi115 bash[46934]: audit 2022-04-22T19:36:02.127001+0000 mon.a (mon.0) 7261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:02 smithi121 bash[36902]: audit 2022-04-22T19:36:02.127001+0000 mon.a (mon.0) 7261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:03 smithi115 bash[45495]: cluster 2022-04-22T19:36:01.804627+0000 mgr.y (mgr.34107) 4464 : cluster [DBG] pgmap v5841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:03 smithi115 bash[46934]: cluster 2022-04-22T19:36:01.804627+0000 mgr.y (mgr.34107) 4464 : cluster [DBG] pgmap v5841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:03 smithi121 bash[36902]: cluster 2022-04-22T19:36:01.804627+0000 mgr.y (mgr.34107) 4464 : cluster [DBG] pgmap v5841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:04.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:03.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:04.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:04 smithi115 bash[45495]: audit 2022-04-22T19:36:04.153506+0000 mon.a (mon.0) 7262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:04 smithi115 bash[46934]: audit 2022-04-22T19:36:04.153506+0000 mon.a (mon.0) 7262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:04 smithi121 bash[36902]: audit 2022-04-22T19:36:04.153506+0000 mon.a (mon.0) 7262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:05 smithi115 bash[45495]: cluster 2022-04-22T19:36:03.805865+0000 mgr.y (mgr.34107) 4465 : cluster [DBG] pgmap v5842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:05 smithi115 bash[46934]: cluster 2022-04-22T19:36:03.805865+0000 mgr.y (mgr.34107) 4465 : cluster [DBG] pgmap v5842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:05.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:05 smithi121 bash[36902]: cluster 2022-04-22T19:36:03.805865+0000 mgr.y (mgr.34107) 4465 : cluster [DBG] pgmap v5842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:06 smithi121 bash[36902]: audit 2022-04-22T19:36:05.644963+0000 mon.a (mon.0) 7263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:06 smithi115 bash[45495]: audit 2022-04-22T19:36:05.644963+0000 mon.a (mon.0) 7263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:06 smithi115 bash[46934]: audit 2022-04-22T19:36:05.644963+0000 mon.a (mon.0) 7263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:07 smithi121 bash[36902]: cluster 2022-04-22T19:36:05.806990+0000 mgr.y (mgr.34107) 4466 : cluster [DBG] pgmap v5843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:07 smithi121 bash[36902]: audit 2022-04-22T19:36:07.129756+0000 mon.a (mon.0) 7264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:07 smithi115 bash[45495]: cluster 2022-04-22T19:36:05.806990+0000 mgr.y (mgr.34107) 4466 : cluster [DBG] pgmap v5843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:07 smithi115 bash[45495]: audit 2022-04-22T19:36:07.129756+0000 mon.a (mon.0) 7264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:07.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:07 smithi115 bash[46934]: cluster 2022-04-22T19:36:05.806990+0000 mgr.y (mgr.34107) 4466 : cluster [DBG] pgmap v5843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:07.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:07 smithi115 bash[46934]: audit 2022-04-22T19:36:07.129756+0000 mon.a (mon.0) 7264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:09 smithi121 bash[36902]: cluster 2022-04-22T19:36:07.808096+0000 mgr.y (mgr.34107) 4467 : cluster [DBG] pgmap v5844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:09 smithi121 bash[36902]: audit 2022-04-22T19:36:09.157293+0000 mon.a (mon.0) 7265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:09 smithi115 bash[45495]: cluster 2022-04-22T19:36:07.808096+0000 mgr.y (mgr.34107) 4467 : cluster [DBG] pgmap v5844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:09 smithi115 bash[45495]: audit 2022-04-22T19:36:09.157293+0000 mon.a (mon.0) 7265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:09 smithi115 bash[46934]: cluster 2022-04-22T19:36:07.808096+0000 mgr.y (mgr.34107) 4467 : cluster [DBG] pgmap v5844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:09.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:09 smithi115 bash[46934]: audit 2022-04-22T19:36:09.157293+0000 mon.a (mon.0) 7265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:11 smithi121 bash[36902]: cluster 2022-04-22T19:36:09.809181+0000 mgr.y (mgr.34107) 4468 : cluster [DBG] pgmap v5845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:11 smithi121 bash[36902]: audit 2022-04-22T19:36:10.647922+0000 mon.a (mon.0) 7266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:11 smithi115 bash[45495]: cluster 2022-04-22T19:36:09.809181+0000 mgr.y (mgr.34107) 4468 : cluster [DBG] pgmap v5845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:11 smithi115 bash[45495]: audit 2022-04-22T19:36:10.647922+0000 mon.a (mon.0) 7266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:11 smithi115 bash[46934]: cluster 2022-04-22T19:36:09.809181+0000 mgr.y (mgr.34107) 4468 : cluster [DBG] pgmap v5845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:11 smithi115 bash[46934]: audit 2022-04-22T19:36:10.647922+0000 mon.a (mon.0) 7266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:12.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:12 smithi115 bash[45495]: audit 2022-04-22T19:36:12.132775+0000 mon.a (mon.0) 7267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:12.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:12.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:12 smithi115 bash[46934]: audit 2022-04-22T19:36:12.132775+0000 mon.a (mon.0) 7267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:12 smithi121 bash[36902]: audit 2022-04-22T19:36:12.132775+0000 mon.a (mon.0) 7267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:13 smithi121 bash[36902]: cluster 2022-04-22T19:36:11.810296+0000 mgr.y (mgr.34107) 4469 : cluster [DBG] pgmap v5846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:13 smithi115 bash[45495]: cluster 2022-04-22T19:36:11.810296+0000 mgr.y (mgr.34107) 4469 : cluster [DBG] pgmap v5846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:13 smithi115 bash[46934]: cluster 2022-04-22T19:36:11.810296+0000 mgr.y (mgr.34107) 4469 : cluster [DBG] pgmap v5846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:13.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:14 smithi121 bash[36902]: audit 2022-04-22T19:36:14.161113+0000 mon.a (mon.0) 7268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:14.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:14 smithi115 bash[45495]: audit 2022-04-22T19:36:14.161113+0000 mon.a (mon.0) 7268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:14 smithi115 bash[46934]: audit 2022-04-22T19:36:14.161113+0000 mon.a (mon.0) 7268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:15 smithi121 bash[36902]: cluster 2022-04-22T19:36:13.811436+0000 mgr.y (mgr.34107) 4470 : cluster [DBG] pgmap v5847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:15 smithi121 bash[36902]: audit 2022-04-22T19:36:14.761553+0000 mon.a (mon.0) 7269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:36:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:15 smithi121 bash[36902]: audit 2022-04-22T19:36:14.761816+0000 mon.a (mon.0) 7270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:36:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[46934]: cluster 2022-04-22T19:36:13.811436+0000 mgr.y (mgr.34107) 4470 : cluster [DBG] pgmap v5847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[46934]: audit 2022-04-22T19:36:14.761553+0000 mon.a (mon.0) 7269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:36:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[46934]: audit 2022-04-22T19:36:14.761816+0000 mon.a (mon.0) 7270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:36:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[45495]: cluster 2022-04-22T19:36:13.811436+0000 mgr.y (mgr.34107) 4470 : cluster [DBG] pgmap v5847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[45495]: audit 2022-04-22T19:36:14.761553+0000 mon.a (mon.0) 7269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:36:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:15 smithi115 bash[45495]: audit 2022-04-22T19:36:14.761816+0000 mon.a (mon.0) 7270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:36:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:16 smithi121 bash[36902]: audit 2022-04-22T19:36:15.650906+0000 mon.a (mon.0) 7271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:16 smithi115 bash[45495]: audit 2022-04-22T19:36:15.650906+0000 mon.a (mon.0) 7271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:16.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:16 smithi115 bash[46934]: audit 2022-04-22T19:36:15.650906+0000 mon.a (mon.0) 7271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:17 smithi121 bash[36902]: cluster 2022-04-22T19:36:15.812410+0000 mgr.y (mgr.34107) 4471 : cluster [DBG] pgmap v5848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:17 smithi121 bash[36902]: audit 2022-04-22T19:36:17.135489+0000 mon.a (mon.0) 7272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:17.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:17 smithi115 bash[45495]: cluster 2022-04-22T19:36:15.812410+0000 mgr.y (mgr.34107) 4471 : cluster [DBG] pgmap v5848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:17 smithi115 bash[45495]: audit 2022-04-22T19:36:17.135489+0000 mon.a (mon.0) 7272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:17 smithi115 bash[46934]: cluster 2022-04-22T19:36:15.812410+0000 mgr.y (mgr.34107) 4471 : cluster [DBG] pgmap v5848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:17 smithi115 bash[46934]: audit 2022-04-22T19:36:17.135489+0000 mon.a (mon.0) 7272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:19 smithi121 bash[36902]: cluster 2022-04-22T19:36:17.813523+0000 mgr.y (mgr.34107) 4472 : cluster [DBG] pgmap v5849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:19 smithi121 bash[36902]: audit 2022-04-22T19:36:19.164780+0000 mon.a (mon.0) 7273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:19 smithi115 bash[45495]: cluster 2022-04-22T19:36:17.813523+0000 mgr.y (mgr.34107) 4472 : cluster [DBG] pgmap v5849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:19.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:19 smithi115 bash[45495]: audit 2022-04-22T19:36:19.164780+0000 mon.a (mon.0) 7273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:19 smithi115 bash[46934]: cluster 2022-04-22T19:36:17.813523+0000 mgr.y (mgr.34107) 4472 : cluster [DBG] pgmap v5849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:19 smithi115 bash[46934]: audit 2022-04-22T19:36:19.164780+0000 mon.a (mon.0) 7273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:20.365 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (119m) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:36:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:36:20.711 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:36:20.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:36:20.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:36:20.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:36:20.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:36:20.713 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:36:20.713 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:36:20.713 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:36:20.713 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:36:20.713 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:36:21.114 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:36:21.114 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:36:21.114 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:36:21.114 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:21.114 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:21.115 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:36:21.116 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:36:21.116 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:36:21.116 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:21 smithi121 bash[36902]: cluster 2022-04-22T19:36:19.814666+0000 mgr.y (mgr.34107) 4473 : cluster [DBG] pgmap v5850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:21 smithi121 bash[36902]: audit 2022-04-22T19:36:20.337383+0000 mgr.y (mgr.34107) 4474 : audit [DBG] from='client.40068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:21 smithi121 bash[36902]: audit 2022-04-22T19:36:20.653964+0000 mon.a (mon.0) 7274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:21 smithi121 bash[36902]: audit 2022-04-22T19:36:21.103749+0000 mon.c (mon.1) 200 : audit [DBG] from='client.? 172.21.15.115:0/1774570168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[45495]: cluster 2022-04-22T19:36:19.814666+0000 mgr.y (mgr.34107) 4473 : cluster [DBG] pgmap v5850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[45495]: audit 2022-04-22T19:36:20.337383+0000 mgr.y (mgr.34107) 4474 : audit [DBG] from='client.40068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[45495]: audit 2022-04-22T19:36:20.653964+0000 mon.a (mon.0) 7274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[45495]: audit 2022-04-22T19:36:21.103749+0000 mon.c (mon.1) 200 : audit [DBG] from='client.? 172.21.15.115:0/1774570168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[46934]: cluster 2022-04-22T19:36:19.814666+0000 mgr.y (mgr.34107) 4473 : cluster [DBG] pgmap v5850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[46934]: audit 2022-04-22T19:36:20.337383+0000 mgr.y (mgr.34107) 4474 : audit [DBG] from='client.40068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[46934]: audit 2022-04-22T19:36:20.653964+0000 mon.a (mon.0) 7274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:21.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:21 smithi115 bash[46934]: audit 2022-04-22T19:36:21.103749+0000 mon.c (mon.1) 200 : audit [DBG] from='client.? 172.21.15.115:0/1774570168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:22 smithi115 bash[45495]: audit 2022-04-22T19:36:20.681352+0000 mgr.y (mgr.34107) 4475 : audit [DBG] from='client.50008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:22.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:22 smithi115 bash[45495]: audit 2022-04-22T19:36:22.138554+0000 mon.a (mon.0) 7275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:22.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:22 smithi115 bash[46934]: audit 2022-04-22T19:36:20.681352+0000 mgr.y (mgr.34107) 4475 : audit [DBG] from='client.50008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:22 smithi115 bash[46934]: audit 2022-04-22T19:36:22.138554+0000 mon.a (mon.0) 7275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:22 smithi121 bash[36902]: audit 2022-04-22T19:36:20.681352+0000 mgr.y (mgr.34107) 4475 : audit [DBG] from='client.50008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:22 smithi121 bash[36902]: audit 2022-04-22T19:36:22.138554+0000 mon.a (mon.0) 7275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:23 smithi121 bash[36902]: cluster 2022-04-22T19:36:21.815825+0000 mgr.y (mgr.34107) 4476 : cluster [DBG] pgmap v5851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:23 smithi115 bash[45495]: cluster 2022-04-22T19:36:21.815825+0000 mgr.y (mgr.34107) 4476 : cluster [DBG] pgmap v5851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:23 smithi115 bash[46934]: cluster 2022-04-22T19:36:21.815825+0000 mgr.y (mgr.34107) 4476 : cluster [DBG] pgmap v5851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:23.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:24 smithi121 bash[36902]: audit 2022-04-22T19:36:24.168694+0000 mon.a (mon.0) 7276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:24.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:24 smithi115 bash[45495]: audit 2022-04-22T19:36:24.168694+0000 mon.a (mon.0) 7276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:24.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:24 smithi115 bash[46934]: audit 2022-04-22T19:36:24.168694+0000 mon.a (mon.0) 7276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:25.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:25 smithi121 bash[36902]: cluster 2022-04-22T19:36:23.816936+0000 mgr.y (mgr.34107) 4477 : cluster [DBG] pgmap v5852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:25 smithi115 bash[45495]: cluster 2022-04-22T19:36:23.816936+0000 mgr.y (mgr.34107) 4477 : cluster [DBG] pgmap v5852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:25 smithi115 bash[46934]: cluster 2022-04-22T19:36:23.816936+0000 mgr.y (mgr.34107) 4477 : cluster [DBG] pgmap v5852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:26 smithi121 bash[36902]: audit 2022-04-22T19:36:25.656321+0000 mon.a (mon.0) 7277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:26 smithi115 bash[45495]: audit 2022-04-22T19:36:25.656321+0000 mon.a (mon.0) 7277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:26 smithi115 bash[46934]: audit 2022-04-22T19:36:25.656321+0000 mon.a (mon.0) 7277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:27 smithi121 bash[36902]: cluster 2022-04-22T19:36:25.818002+0000 mgr.y (mgr.34107) 4478 : cluster [DBG] pgmap v5853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:27 smithi121 bash[36902]: audit 2022-04-22T19:36:27.141451+0000 mon.a (mon.0) 7278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:27 smithi115 bash[45495]: cluster 2022-04-22T19:36:25.818002+0000 mgr.y (mgr.34107) 4478 : cluster [DBG] pgmap v5853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:27 smithi115 bash[45495]: audit 2022-04-22T19:36:27.141451+0000 mon.a (mon.0) 7278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:27 smithi115 bash[46934]: cluster 2022-04-22T19:36:25.818002+0000 mgr.y (mgr.34107) 4478 : cluster [DBG] pgmap v5853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:27 smithi115 bash[46934]: audit 2022-04-22T19:36:27.141451+0000 mon.a (mon.0) 7278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:29 smithi121 bash[36902]: cluster 2022-04-22T19:36:27.819103+0000 mgr.y (mgr.34107) 4479 : cluster [DBG] pgmap v5854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:29 smithi121 bash[36902]: audit 2022-04-22T19:36:29.172335+0000 mon.a (mon.0) 7279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:29 smithi115 bash[45495]: cluster 2022-04-22T19:36:27.819103+0000 mgr.y (mgr.34107) 4479 : cluster [DBG] pgmap v5854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:29.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:29 smithi115 bash[45495]: audit 2022-04-22T19:36:29.172335+0000 mon.a (mon.0) 7279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:29 smithi115 bash[46934]: cluster 2022-04-22T19:36:27.819103+0000 mgr.y (mgr.34107) 4479 : cluster [DBG] pgmap v5854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:29 smithi115 bash[46934]: audit 2022-04-22T19:36:29.172335+0000 mon.a (mon.0) 7279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:31 smithi121 bash[36902]: cluster 2022-04-22T19:36:29.820281+0000 mgr.y (mgr.34107) 4480 : cluster [DBG] pgmap v5855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:31 smithi121 bash[36902]: audit 2022-04-22T19:36:30.659293+0000 mon.a (mon.0) 7280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:31 smithi115 bash[45495]: cluster 2022-04-22T19:36:29.820281+0000 mgr.y (mgr.34107) 4480 : cluster [DBG] pgmap v5855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:31 smithi115 bash[45495]: audit 2022-04-22T19:36:30.659293+0000 mon.a (mon.0) 7280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:31 smithi115 bash[46934]: cluster 2022-04-22T19:36:29.820281+0000 mgr.y (mgr.34107) 4480 : cluster [DBG] pgmap v5855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:31 smithi115 bash[46934]: audit 2022-04-22T19:36:30.659293+0000 mon.a (mon.0) 7280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:32.415 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:32.118Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:32 smithi121 bash[36902]: audit 2022-04-22T19:36:32.144621+0000 mon.a (mon.0) 7281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:32.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:32 smithi115 bash[45495]: audit 2022-04-22T19:36:32.144621+0000 mon.a (mon.0) 7281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:32.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:32 smithi115 bash[46934]: audit 2022-04-22T19:36:32.144621+0000 mon.a (mon.0) 7281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:33.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:33 smithi121 bash[36902]: cluster 2022-04-22T19:36:31.821382+0000 mgr.y (mgr.34107) 4481 : cluster [DBG] pgmap v5856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:33.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:33 smithi115 bash[45495]: cluster 2022-04-22T19:36:31.821382+0000 mgr.y (mgr.34107) 4481 : cluster [DBG] pgmap v5856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:33.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:33 smithi115 bash[46934]: cluster 2022-04-22T19:36:31.821382+0000 mgr.y (mgr.34107) 4481 : cluster [DBG] pgmap v5856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:33.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:34 smithi121 bash[36902]: audit 2022-04-22T19:36:34.176090+0000 mon.a (mon.0) 7282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:34.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:34 smithi115 bash[45495]: audit 2022-04-22T19:36:34.176090+0000 mon.a (mon.0) 7282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:34 smithi115 bash[46934]: audit 2022-04-22T19:36:34.176090+0000 mon.a (mon.0) 7282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:35 smithi121 bash[36902]: cluster 2022-04-22T19:36:33.822555+0000 mgr.y (mgr.34107) 4482 : cluster [DBG] pgmap v5857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:35 smithi115 bash[45495]: cluster 2022-04-22T19:36:33.822555+0000 mgr.y (mgr.34107) 4482 : cluster [DBG] pgmap v5857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:35 smithi115 bash[46934]: cluster 2022-04-22T19:36:33.822555+0000 mgr.y (mgr.34107) 4482 : cluster [DBG] pgmap v5857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:36 smithi121 bash[36902]: audit 2022-04-22T19:36:35.662655+0000 mon.a (mon.0) 7283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:36 smithi115 bash[45495]: audit 2022-04-22T19:36:35.662655+0000 mon.a (mon.0) 7283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:36.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:36 smithi115 bash[46934]: audit 2022-04-22T19:36:35.662655+0000 mon.a (mon.0) 7283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:37 smithi121 bash[36902]: cluster 2022-04-22T19:36:35.823664+0000 mgr.y (mgr.34107) 4483 : cluster [DBG] pgmap v5858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:37 smithi121 bash[36902]: audit 2022-04-22T19:36:37.147566+0000 mon.a (mon.0) 7284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:37 smithi115 bash[45495]: cluster 2022-04-22T19:36:35.823664+0000 mgr.y (mgr.34107) 4483 : cluster [DBG] pgmap v5858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:37 smithi115 bash[45495]: audit 2022-04-22T19:36:37.147566+0000 mon.a (mon.0) 7284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:37 smithi115 bash[46934]: cluster 2022-04-22T19:36:35.823664+0000 mgr.y (mgr.34107) 4483 : cluster [DBG] pgmap v5858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:37.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:37 smithi115 bash[46934]: audit 2022-04-22T19:36:37.147566+0000 mon.a (mon.0) 7284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:39 smithi121 bash[36902]: cluster 2022-04-22T19:36:37.824699+0000 mgr.y (mgr.34107) 4484 : cluster [DBG] pgmap v5859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:39 smithi121 bash[36902]: audit 2022-04-22T19:36:39.179804+0000 mon.a (mon.0) 7285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:39 smithi115 bash[45495]: cluster 2022-04-22T19:36:37.824699+0000 mgr.y (mgr.34107) 4484 : cluster [DBG] pgmap v5859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:39 smithi115 bash[45495]: audit 2022-04-22T19:36:39.179804+0000 mon.a (mon.0) 7285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:39 smithi115 bash[46934]: cluster 2022-04-22T19:36:37.824699+0000 mgr.y (mgr.34107) 4484 : cluster [DBG] pgmap v5859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:39 smithi115 bash[46934]: audit 2022-04-22T19:36:39.179804+0000 mon.a (mon.0) 7285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:41 smithi121 bash[36902]: cluster 2022-04-22T19:36:39.825803+0000 mgr.y (mgr.34107) 4485 : cluster [DBG] pgmap v5860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:41 smithi121 bash[36902]: audit 2022-04-22T19:36:40.665513+0000 mon.a (mon.0) 7286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:41 smithi115 bash[45495]: cluster 2022-04-22T19:36:39.825803+0000 mgr.y (mgr.34107) 4485 : cluster [DBG] pgmap v5860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:41 smithi115 bash[45495]: audit 2022-04-22T19:36:40.665513+0000 mon.a (mon.0) 7286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:41 smithi115 bash[46934]: cluster 2022-04-22T19:36:39.825803+0000 mgr.y (mgr.34107) 4485 : cluster [DBG] pgmap v5860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:41 smithi115 bash[46934]: audit 2022-04-22T19:36:40.665513+0000 mon.a (mon.0) 7286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:42.455 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:42.118Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:42 smithi115 bash[45495]: audit 2022-04-22T19:36:42.150656+0000 mon.a (mon.0) 7287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:42.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:42 smithi115 bash[46934]: audit 2022-04-22T19:36:42.150656+0000 mon.a (mon.0) 7287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:42 smithi121 bash[36902]: audit 2022-04-22T19:36:42.150656+0000 mon.a (mon.0) 7287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:43 smithi121 bash[36902]: cluster 2022-04-22T19:36:41.826962+0000 mgr.y (mgr.34107) 4486 : cluster [DBG] pgmap v5861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:43 smithi115 bash[45495]: cluster 2022-04-22T19:36:41.826962+0000 mgr.y (mgr.34107) 4486 : cluster [DBG] pgmap v5861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:43 smithi115 bash[46934]: cluster 2022-04-22T19:36:41.826962+0000 mgr.y (mgr.34107) 4486 : cluster [DBG] pgmap v5861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:44.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:43.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:44 smithi121 bash[36902]: audit 2022-04-22T19:36:44.183600+0000 mon.a (mon.0) 7288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:44.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:44 smithi115 bash[45495]: audit 2022-04-22T19:36:44.183600+0000 mon.a (mon.0) 7288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:44.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:44 smithi115 bash[46934]: audit 2022-04-22T19:36:44.183600+0000 mon.a (mon.0) 7288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:45 smithi121 bash[36902]: cluster 2022-04-22T19:36:43.828128+0000 mgr.y (mgr.34107) 4487 : cluster [DBG] pgmap v5862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:45 smithi115 bash[45495]: cluster 2022-04-22T19:36:43.828128+0000 mgr.y (mgr.34107) 4487 : cluster [DBG] pgmap v5862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:45 smithi115 bash[46934]: cluster 2022-04-22T19:36:43.828128+0000 mgr.y (mgr.34107) 4487 : cluster [DBG] pgmap v5862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:46 smithi121 bash[36902]: audit 2022-04-22T19:36:45.668866+0000 mon.a (mon.0) 7289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:46 smithi115 bash[45495]: audit 2022-04-22T19:36:45.668866+0000 mon.a (mon.0) 7289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:46 smithi115 bash[46934]: audit 2022-04-22T19:36:45.668866+0000 mon.a (mon.0) 7289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:47 smithi121 bash[36902]: cluster 2022-04-22T19:36:45.829157+0000 mgr.y (mgr.34107) 4488 : cluster [DBG] pgmap v5863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:47 smithi121 bash[36902]: audit 2022-04-22T19:36:47.153676+0000 mon.a (mon.0) 7290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:47 smithi115 bash[45495]: cluster 2022-04-22T19:36:45.829157+0000 mgr.y (mgr.34107) 4488 : cluster [DBG] pgmap v5863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:47 smithi115 bash[45495]: audit 2022-04-22T19:36:47.153676+0000 mon.a (mon.0) 7290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:47 smithi115 bash[46934]: cluster 2022-04-22T19:36:45.829157+0000 mgr.y (mgr.34107) 4488 : cluster [DBG] pgmap v5863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:47 smithi115 bash[46934]: audit 2022-04-22T19:36:47.153676+0000 mon.a (mon.0) 7290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:49 smithi121 bash[36902]: cluster 2022-04-22T19:36:47.830240+0000 mgr.y (mgr.34107) 4489 : cluster [DBG] pgmap v5864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:49 smithi121 bash[36902]: audit 2022-04-22T19:36:49.187434+0000 mon.a (mon.0) 7291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:49 smithi115 bash[45495]: cluster 2022-04-22T19:36:47.830240+0000 mgr.y (mgr.34107) 4489 : cluster [DBG] pgmap v5864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:49 smithi115 bash[45495]: audit 2022-04-22T19:36:49.187434+0000 mon.a (mon.0) 7291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:49 smithi115 bash[46934]: cluster 2022-04-22T19:36:47.830240+0000 mgr.y (mgr.34107) 4489 : cluster [DBG] pgmap v5864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:49 smithi115 bash[46934]: audit 2022-04-22T19:36:49.187434+0000 mon.a (mon.0) 7291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:51.484 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:36:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:51 smithi121 bash[36902]: cluster 2022-04-22T19:36:49.831294+0000 mgr.y (mgr.34107) 4490 : cluster [DBG] pgmap v5865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:51 smithi121 bash[36902]: audit 2022-04-22T19:36:50.672066+0000 mon.a (mon.0) 7292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:51 smithi115 bash[45495]: cluster 2022-04-22T19:36:49.831294+0000 mgr.y (mgr.34107) 4490 : cluster [DBG] pgmap v5865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:51 smithi115 bash[45495]: audit 2022-04-22T19:36:50.672066+0000 mon.a (mon.0) 7292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:51 smithi115 bash[46934]: cluster 2022-04-22T19:36:49.831294+0000 mgr.y (mgr.34107) 4490 : cluster [DBG] pgmap v5865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:51 smithi115 bash[46934]: audit 2022-04-22T19:36:50.672066+0000 mon.a (mon.0) 7292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:51.831 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (119m) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:36:51.832 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:36:51.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:36:51.834 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:36:51.834 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:36:52.234 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:36:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:36:52.236 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:36:52.493 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:52.116Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:52.493 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[45495]: audit 2022-04-22T19:36:51.458300+0000 mgr.y (mgr.34107) 4491 : audit [DBG] from='client.50020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:52.493 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[45495]: audit 2022-04-22T19:36:52.156743+0000 mon.a (mon.0) 7293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:52.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[45495]: audit 2022-04-22T19:36:52.224289+0000 mon.a (mon.0) 7294 : audit [DBG] from='client.? 172.21.15.115:0/4293973585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:52 smithi121 bash[36902]: audit 2022-04-22T19:36:51.458300+0000 mgr.y (mgr.34107) 4491 : audit [DBG] from='client.50020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:52 smithi121 bash[36902]: audit 2022-04-22T19:36:52.156743+0000 mon.a (mon.0) 7293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:52 smithi121 bash[36902]: audit 2022-04-22T19:36:52.224289+0000 mon.a (mon.0) 7294 : audit [DBG] from='client.? 172.21.15.115:0/4293973585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:52.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[46934]: audit 2022-04-22T19:36:51.458300+0000 mgr.y (mgr.34107) 4491 : audit [DBG] from='client.50020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:52.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[46934]: audit 2022-04-22T19:36:52.156743+0000 mon.a (mon.0) 7293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:52.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:52 smithi115 bash[46934]: audit 2022-04-22T19:36:52.224289+0000 mon.a (mon.0) 7294 : audit [DBG] from='client.? 172.21.15.115:0/4293973585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:36:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:53 smithi121 bash[36902]: audit 2022-04-22T19:36:51.798107+0000 mgr.y (mgr.34107) 4492 : audit [DBG] from='client.50026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:53 smithi121 bash[36902]: cluster 2022-04-22T19:36:51.832313+0000 mgr.y (mgr.34107) 4493 : cluster [DBG] pgmap v5866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:53 smithi115 bash[45495]: audit 2022-04-22T19:36:51.798107+0000 mgr.y (mgr.34107) 4492 : audit [DBG] from='client.50026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:53.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:53 smithi115 bash[45495]: cluster 2022-04-22T19:36:51.832313+0000 mgr.y (mgr.34107) 4493 : cluster [DBG] pgmap v5866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:53 smithi115 bash[46934]: audit 2022-04-22T19:36:51.798107+0000 mgr.y (mgr.34107) 4492 : audit [DBG] from='client.50026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:36:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:53 smithi115 bash[46934]: cluster 2022-04-22T19:36:51.832313+0000 mgr.y (mgr.34107) 4493 : cluster [DBG] pgmap v5866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:36:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:36:53.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:36:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:54 smithi121 bash[36902]: audit 2022-04-22T19:36:54.191073+0000 mon.a (mon.0) 7295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:54 smithi115 bash[45495]: audit 2022-04-22T19:36:54.191073+0000 mon.a (mon.0) 7295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:54.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:54 smithi115 bash[46934]: audit 2022-04-22T19:36:54.191073+0000 mon.a (mon.0) 7295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:55 smithi121 bash[36902]: cluster 2022-04-22T19:36:53.833511+0000 mgr.y (mgr.34107) 4494 : cluster [DBG] pgmap v5867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:55 smithi115 bash[45495]: cluster 2022-04-22T19:36:53.833511+0000 mgr.y (mgr.34107) 4494 : cluster [DBG] pgmap v5867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:55 smithi115 bash[46934]: cluster 2022-04-22T19:36:53.833511+0000 mgr.y (mgr.34107) 4494 : cluster [DBG] pgmap v5867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:56 smithi121 bash[36902]: audit 2022-04-22T19:36:55.674996+0000 mon.a (mon.0) 7296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:56 smithi115 bash[46934]: audit 2022-04-22T19:36:55.674996+0000 mon.a (mon.0) 7296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:56.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:56 smithi115 bash[45495]: audit 2022-04-22T19:36:55.674996+0000 mon.a (mon.0) 7296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:36:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: cluster 2022-04-22T19:36:55.834561+0000 mgr.y (mgr.34107) 4495 : cluster [DBG] pgmap v5868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: audit 2022-04-22T19:36:57.003447+0000 mon.a (mon.0) 7297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:36:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: audit 2022-04-22T19:36:57.006634+0000 mon.a (mon.0) 7298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:36:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: audit 2022-04-22T19:36:57.008453+0000 mon.a (mon.0) 7299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:36:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: audit 2022-04-22T19:36:57.159744+0000 mon.a (mon.0) 7300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:57.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:57 smithi121 bash[36902]: audit 2022-04-22T19:36:57.387821+0000 mon.a (mon.0) 7301 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: cluster 2022-04-22T19:36:55.834561+0000 mgr.y (mgr.34107) 4495 : cluster [DBG] pgmap v5868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: audit 2022-04-22T19:36:57.003447+0000 mon.a (mon.0) 7297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: audit 2022-04-22T19:36:57.006634+0000 mon.a (mon.0) 7298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: audit 2022-04-22T19:36:57.008453+0000 mon.a (mon.0) 7299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: audit 2022-04-22T19:36:57.159744+0000 mon.a (mon.0) 7300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[45495]: audit 2022-04-22T19:36:57.387821+0000 mon.a (mon.0) 7301 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: cluster 2022-04-22T19:36:55.834561+0000 mgr.y (mgr.34107) 4495 : cluster [DBG] pgmap v5868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: audit 2022-04-22T19:36:57.003447+0000 mon.a (mon.0) 7297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: audit 2022-04-22T19:36:57.006634+0000 mon.a (mon.0) 7298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: audit 2022-04-22T19:36:57.008453+0000 mon.a (mon.0) 7299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: audit 2022-04-22T19:36:57.159744+0000 mon.a (mon.0) 7300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:36:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:57 smithi115 bash[46934]: audit 2022-04-22T19:36:57.387821+0000 mon.a (mon.0) 7301 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:36:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:59 smithi115 bash[45495]: cluster 2022-04-22T19:36:57.835674+0000 mgr.y (mgr.34107) 4496 : cluster [DBG] pgmap v5869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:36:59 smithi115 bash[45495]: audit 2022-04-22T19:36:59.194557+0000 mon.a (mon.0) 7302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:36:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:59 smithi115 bash[46934]: cluster 2022-04-22T19:36:57.835674+0000 mgr.y (mgr.34107) 4496 : cluster [DBG] pgmap v5869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:36:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:36:59 smithi115 bash[46934]: audit 2022-04-22T19:36:59.194557+0000 mon.a (mon.0) 7302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:59 smithi121 bash[36902]: cluster 2022-04-22T19:36:57.835674+0000 mgr.y (mgr.34107) 4496 : cluster [DBG] pgmap v5869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:36:59 smithi121 bash[36902]: audit 2022-04-22T19:36:59.194557+0000 mon.a (mon.0) 7302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:01 smithi115 bash[45495]: cluster 2022-04-22T19:36:59.836891+0000 mgr.y (mgr.34107) 4497 : cluster [DBG] pgmap v5870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:01 smithi115 bash[45495]: audit 2022-04-22T19:37:00.678027+0000 mon.a (mon.0) 7303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:01 smithi115 bash[46934]: cluster 2022-04-22T19:36:59.836891+0000 mgr.y (mgr.34107) 4497 : cluster [DBG] pgmap v5870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:01 smithi115 bash[46934]: audit 2022-04-22T19:37:00.678027+0000 mon.a (mon.0) 7303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:01 smithi121 bash[36902]: cluster 2022-04-22T19:36:59.836891+0000 mgr.y (mgr.34107) 4497 : cluster [DBG] pgmap v5870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:01 smithi121 bash[36902]: audit 2022-04-22T19:37:00.678027+0000 mon.a (mon.0) 7303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:02.535 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:02.119Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:02.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:02 smithi115 bash[45495]: audit 2022-04-22T19:37:02.162560+0000 mon.a (mon.0) 7304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:02.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:02 smithi115 bash[46934]: audit 2022-04-22T19:37:02.162560+0000 mon.a (mon.0) 7304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:02 smithi121 bash[36902]: audit 2022-04-22T19:37:02.162560+0000 mon.a (mon.0) 7304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:03 smithi115 bash[45495]: cluster 2022-04-22T19:37:01.838142+0000 mgr.y (mgr.34107) 4498 : cluster [DBG] pgmap v5871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:03 smithi115 bash[46934]: cluster 2022-04-22T19:37:01.838142+0000 mgr.y (mgr.34107) 4498 : cluster [DBG] pgmap v5871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:03 smithi121 bash[36902]: cluster 2022-04-22T19:37:01.838142+0000 mgr.y (mgr.34107) 4498 : cluster [DBG] pgmap v5871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:03.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:04 smithi115 bash[45495]: audit 2022-04-22T19:37:04.198166+0000 mon.a (mon.0) 7305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:04 smithi115 bash[46934]: audit 2022-04-22T19:37:04.198166+0000 mon.a (mon.0) 7305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:04 smithi121 bash[36902]: audit 2022-04-22T19:37:04.198166+0000 mon.a (mon.0) 7305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:05.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:05 smithi115 bash[45495]: cluster 2022-04-22T19:37:03.839334+0000 mgr.y (mgr.34107) 4499 : cluster [DBG] pgmap v5872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:05 smithi115 bash[46934]: cluster 2022-04-22T19:37:03.839334+0000 mgr.y (mgr.34107) 4499 : cluster [DBG] pgmap v5872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:05 smithi121 bash[36902]: cluster 2022-04-22T19:37:03.839334+0000 mgr.y (mgr.34107) 4499 : cluster [DBG] pgmap v5872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:06 smithi121 bash[36902]: audit 2022-04-22T19:37:05.680958+0000 mon.a (mon.0) 7306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:06 smithi115 bash[45495]: audit 2022-04-22T19:37:05.680958+0000 mon.a (mon.0) 7306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:07.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:06 smithi115 bash[46934]: audit 2022-04-22T19:37:05.680958+0000 mon.a (mon.0) 7306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:07 smithi121 bash[36902]: cluster 2022-04-22T19:37:05.840398+0000 mgr.y (mgr.34107) 4500 : cluster [DBG] pgmap v5873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:07 smithi121 bash[36902]: audit 2022-04-22T19:37:07.165540+0000 mon.a (mon.0) 7307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:07 smithi115 bash[45495]: cluster 2022-04-22T19:37:05.840398+0000 mgr.y (mgr.34107) 4500 : cluster [DBG] pgmap v5873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:08.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:07 smithi115 bash[45495]: audit 2022-04-22T19:37:07.165540+0000 mon.a (mon.0) 7307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:08.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:07 smithi115 bash[46934]: cluster 2022-04-22T19:37:05.840398+0000 mgr.y (mgr.34107) 4500 : cluster [DBG] pgmap v5873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:08.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:07 smithi115 bash[46934]: audit 2022-04-22T19:37:07.165540+0000 mon.a (mon.0) 7307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:09 smithi121 bash[36902]: cluster 2022-04-22T19:37:07.841507+0000 mgr.y (mgr.34107) 4501 : cluster [DBG] pgmap v5874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:09 smithi121 bash[36902]: audit 2022-04-22T19:37:09.201921+0000 mon.a (mon.0) 7308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:10.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:09 smithi115 bash[45495]: cluster 2022-04-22T19:37:07.841507+0000 mgr.y (mgr.34107) 4501 : cluster [DBG] pgmap v5874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:10.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:09 smithi115 bash[45495]: audit 2022-04-22T19:37:09.201921+0000 mon.a (mon.0) 7308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:10.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:09 smithi115 bash[46934]: cluster 2022-04-22T19:37:07.841507+0000 mgr.y (mgr.34107) 4501 : cluster [DBG] pgmap v5874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:10.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:09 smithi115 bash[46934]: audit 2022-04-22T19:37:09.201921+0000 mon.a (mon.0) 7308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:11 smithi121 bash[36902]: cluster 2022-04-22T19:37:09.842613+0000 mgr.y (mgr.34107) 4502 : cluster [DBG] pgmap v5875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:11 smithi121 bash[36902]: audit 2022-04-22T19:37:10.683902+0000 mon.a (mon.0) 7309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:11 smithi115 bash[45495]: cluster 2022-04-22T19:37:09.842613+0000 mgr.y (mgr.34107) 4502 : cluster [DBG] pgmap v5875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:12.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:11 smithi115 bash[45495]: audit 2022-04-22T19:37:10.683902+0000 mon.a (mon.0) 7309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:11 smithi115 bash[46934]: cluster 2022-04-22T19:37:09.842613+0000 mgr.y (mgr.34107) 4502 : cluster [DBG] pgmap v5875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:12.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:11 smithi115 bash[46934]: audit 2022-04-22T19:37:10.683902+0000 mon.a (mon.0) 7309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:12.119Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:12 smithi121 bash[36902]: audit 2022-04-22T19:37:12.168790+0000 mon.a (mon.0) 7310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:13.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:12 smithi115 bash[45495]: audit 2022-04-22T19:37:12.168790+0000 mon.a (mon.0) 7310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:13.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:12 smithi115 bash[46934]: audit 2022-04-22T19:37:12.168790+0000 mon.a (mon.0) 7310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:14.002 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:13 smithi115 bash[45495]: cluster 2022-04-22T19:37:11.843809+0000 mgr.y (mgr.34107) 4503 : cluster [DBG] pgmap v5876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:14.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:13 smithi115 bash[46934]: cluster 2022-04-22T19:37:11.843809+0000 mgr.y (mgr.34107) 4503 : cluster [DBG] pgmap v5876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:13 smithi121 bash[36902]: cluster 2022-04-22T19:37:11.843809+0000 mgr.y (mgr.34107) 4503 : cluster [DBG] pgmap v5876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:13.991Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:14 smithi121 bash[36902]: audit 2022-04-22T19:37:14.205436+0000 mon.a (mon.0) 7311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:15.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:14 smithi115 bash[45495]: audit 2022-04-22T19:37:14.205436+0000 mon.a (mon.0) 7311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:15.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:14 smithi115 bash[46934]: audit 2022-04-22T19:37:14.205436+0000 mon.a (mon.0) 7311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:15 smithi121 bash[36902]: cluster 2022-04-22T19:37:13.844914+0000 mgr.y (mgr.34107) 4504 : cluster [DBG] pgmap v5877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:15 smithi121 bash[36902]: audit 2022-04-22T19:37:14.769694+0000 mon.a (mon.0) 7312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:37:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:15 smithi121 bash[36902]: audit 2022-04-22T19:37:14.772101+0000 mon.a (mon.0) 7313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:37:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[46934]: cluster 2022-04-22T19:37:13.844914+0000 mgr.y (mgr.34107) 4504 : cluster [DBG] pgmap v5877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:16.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[46934]: audit 2022-04-22T19:37:14.769694+0000 mon.a (mon.0) 7312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:37:16.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[46934]: audit 2022-04-22T19:37:14.772101+0000 mon.a (mon.0) 7313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:37:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[45495]: cluster 2022-04-22T19:37:13.844914+0000 mgr.y (mgr.34107) 4504 : cluster [DBG] pgmap v5877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[45495]: audit 2022-04-22T19:37:14.769694+0000 mon.a (mon.0) 7312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:37:16.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:15 smithi115 bash[45495]: audit 2022-04-22T19:37:14.772101+0000 mon.a (mon.0) 7313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:37:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:16 smithi121 bash[36902]: audit 2022-04-22T19:37:15.687091+0000 mon.a (mon.0) 7314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:17.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:16 smithi115 bash[45495]: audit 2022-04-22T19:37:15.687091+0000 mon.a (mon.0) 7314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:17.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:16 smithi115 bash[46934]: audit 2022-04-22T19:37:15.687091+0000 mon.a (mon.0) 7314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:17 smithi121 bash[36902]: cluster 2022-04-22T19:37:15.845988+0000 mgr.y (mgr.34107) 4505 : cluster [DBG] pgmap v5878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:18.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:17 smithi121 bash[36902]: audit 2022-04-22T19:37:17.171946+0000 mon.a (mon.0) 7315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:17 smithi115 bash[45495]: cluster 2022-04-22T19:37:15.845988+0000 mgr.y (mgr.34107) 4505 : cluster [DBG] pgmap v5878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:18.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:17 smithi115 bash[45495]: audit 2022-04-22T19:37:17.171946+0000 mon.a (mon.0) 7315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:17 smithi115 bash[46934]: cluster 2022-04-22T19:37:15.845988+0000 mgr.y (mgr.34107) 4505 : cluster [DBG] pgmap v5878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:17 smithi115 bash[46934]: audit 2022-04-22T19:37:17.171946+0000 mon.a (mon.0) 7315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:19 smithi115 bash[45495]: cluster 2022-04-22T19:37:17.847117+0000 mgr.y (mgr.34107) 4506 : cluster [DBG] pgmap v5879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:19.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:19 smithi115 bash[46934]: cluster 2022-04-22T19:37:17.847117+0000 mgr.y (mgr.34107) 4506 : cluster [DBG] pgmap v5879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:19 smithi121 bash[36902]: cluster 2022-04-22T19:37:17.847117+0000 mgr.y (mgr.34107) 4506 : cluster [DBG] pgmap v5879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:20.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:20 smithi115 bash[45495]: audit 2022-04-22T19:37:19.209224+0000 mon.a (mon.0) 7316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:20.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:20 smithi115 bash[46934]: audit 2022-04-22T19:37:19.209224+0000 mon.a (mon.0) 7316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:20 smithi121 bash[36902]: audit 2022-04-22T19:37:19.209224+0000 mon.a (mon.0) 7316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:21 smithi115 bash[45495]: cluster 2022-04-22T19:37:19.848118+0000 mgr.y (mgr.34107) 4507 : cluster [DBG] pgmap v5880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:21 smithi115 bash[45495]: audit 2022-04-22T19:37:20.690353+0000 mon.a (mon.0) 7317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:21 smithi115 bash[46934]: cluster 2022-04-22T19:37:19.848118+0000 mgr.y (mgr.34107) 4507 : cluster [DBG] pgmap v5880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:21.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:21 smithi115 bash[46934]: audit 2022-04-22T19:37:20.690353+0000 mon.a (mon.0) 7317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:21 smithi121 bash[36902]: cluster 2022-04-22T19:37:19.848118+0000 mgr.y (mgr.34107) 4507 : cluster [DBG] pgmap v5880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:21.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:21 smithi121 bash[36902]: audit 2022-04-22T19:37:20.690353+0000 mon.a (mon.0) 7317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:22.119Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:22.610 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:37:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:37:22.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:37:22.957 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:37:22.957 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:37:22.957 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:37:22.957 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:37:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:23 smithi115 bash[45495]: cluster 2022-04-22T19:37:21.849138+0000 mgr.y (mgr.34107) 4508 : cluster [DBG] pgmap v5881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:23 smithi115 bash[45495]: audit 2022-04-22T19:37:22.175285+0000 mon.a (mon.0) 7318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:23 smithi115 bash[46934]: cluster 2022-04-22T19:37:21.849138+0000 mgr.y (mgr.34107) 4508 : cluster [DBG] pgmap v5881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:23 smithi115 bash[46934]: audit 2022-04-22T19:37:22.175285+0000 mon.a (mon.0) 7318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:23.362 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:37:23.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:23.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:37:23.364 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:37:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:23 smithi121 bash[36902]: cluster 2022-04-22T19:37:21.849138+0000 mgr.y (mgr.34107) 4508 : cluster [DBG] pgmap v5881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:23.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:23 smithi121 bash[36902]: audit 2022-04-22T19:37:22.175285+0000 mon.a (mon.0) 7318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:23.990Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[45495]: audit 2022-04-22T19:37:22.583901+0000 mgr.y (mgr.34107) 4509 : audit [DBG] from='client.50038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[45495]: audit 2022-04-22T19:37:22.925649+0000 mgr.y (mgr.34107) 4510 : audit [DBG] from='client.40110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[45495]: audit 2022-04-22T19:37:23.352073+0000 mon.a (mon.0) 7319 : audit [DBG] from='client.? 172.21.15.115:0/2181666948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[46934]: audit 2022-04-22T19:37:22.583901+0000 mgr.y (mgr.34107) 4509 : audit [DBG] from='client.50038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[46934]: audit 2022-04-22T19:37:22.925649+0000 mgr.y (mgr.34107) 4510 : audit [DBG] from='client.40110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:24 smithi115 bash[46934]: audit 2022-04-22T19:37:23.352073+0000 mon.a (mon.0) 7319 : audit [DBG] from='client.? 172.21.15.115:0/2181666948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:24 smithi121 bash[36902]: audit 2022-04-22T19:37:22.583901+0000 mgr.y (mgr.34107) 4509 : audit [DBG] from='client.50038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:24 smithi121 bash[36902]: audit 2022-04-22T19:37:22.925649+0000 mgr.y (mgr.34107) 4510 : audit [DBG] from='client.40110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:24 smithi121 bash[36902]: audit 2022-04-22T19:37:23.352073+0000 mon.a (mon.0) 7319 : audit [DBG] from='client.? 172.21.15.115:0/2181666948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:25 smithi121 bash[36902]: cluster 2022-04-22T19:37:23.850303+0000 mgr.y (mgr.34107) 4511 : cluster [DBG] pgmap v5882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:25 smithi121 bash[36902]: audit 2022-04-22T19:37:24.212697+0000 mon.a (mon.0) 7320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:25 smithi115 bash[45495]: cluster 2022-04-22T19:37:23.850303+0000 mgr.y (mgr.34107) 4511 : cluster [DBG] pgmap v5882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:25.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:25 smithi115 bash[45495]: audit 2022-04-22T19:37:24.212697+0000 mon.a (mon.0) 7320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:25 smithi115 bash[46934]: cluster 2022-04-22T19:37:23.850303+0000 mgr.y (mgr.34107) 4511 : cluster [DBG] pgmap v5882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:25 smithi115 bash[46934]: audit 2022-04-22T19:37:24.212697+0000 mon.a (mon.0) 7320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:26 smithi121 bash[36902]: audit 2022-04-22T19:37:25.693792+0000 mon.a (mon.0) 7321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:26 smithi115 bash[45495]: audit 2022-04-22T19:37:25.693792+0000 mon.a (mon.0) 7321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:26 smithi115 bash[46934]: audit 2022-04-22T19:37:25.693792+0000 mon.a (mon.0) 7321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:27 smithi121 bash[36902]: cluster 2022-04-22T19:37:25.851396+0000 mgr.y (mgr.34107) 4512 : cluster [DBG] pgmap v5883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:27.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:27 smithi115 bash[45495]: cluster 2022-04-22T19:37:25.851396+0000 mgr.y (mgr.34107) 4512 : cluster [DBG] pgmap v5883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:27 smithi115 bash[46934]: cluster 2022-04-22T19:37:25.851396+0000 mgr.y (mgr.34107) 4512 : cluster [DBG] pgmap v5883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:28 smithi121 bash[36902]: audit 2022-04-22T19:37:27.178338+0000 mon.a (mon.0) 7322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:28 smithi115 bash[45495]: audit 2022-04-22T19:37:27.178338+0000 mon.a (mon.0) 7322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:28 smithi115 bash[46934]: audit 2022-04-22T19:37:27.178338+0000 mon.a (mon.0) 7322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:29 smithi121 bash[36902]: cluster 2022-04-22T19:37:27.852462+0000 mgr.y (mgr.34107) 4513 : cluster [DBG] pgmap v5884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:29 smithi115 bash[45495]: cluster 2022-04-22T19:37:27.852462+0000 mgr.y (mgr.34107) 4513 : cluster [DBG] pgmap v5884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:29 smithi115 bash[46934]: cluster 2022-04-22T19:37:27.852462+0000 mgr.y (mgr.34107) 4513 : cluster [DBG] pgmap v5884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:30 smithi121 bash[36902]: audit 2022-04-22T19:37:29.216391+0000 mon.a (mon.0) 7323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:30.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:30 smithi115 bash[45495]: audit 2022-04-22T19:37:29.216391+0000 mon.a (mon.0) 7323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:30.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:30 smithi115 bash[46934]: audit 2022-04-22T19:37:29.216391+0000 mon.a (mon.0) 7323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:31 smithi121 bash[36902]: cluster 2022-04-22T19:37:29.853562+0000 mgr.y (mgr.34107) 4514 : cluster [DBG] pgmap v5885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:31 smithi121 bash[36902]: audit 2022-04-22T19:37:30.697039+0000 mon.a (mon.0) 7324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:31 smithi115 bash[45495]: cluster 2022-04-22T19:37:29.853562+0000 mgr.y (mgr.34107) 4514 : cluster [DBG] pgmap v5885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:31 smithi115 bash[45495]: audit 2022-04-22T19:37:30.697039+0000 mon.a (mon.0) 7324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:31 smithi115 bash[46934]: cluster 2022-04-22T19:37:29.853562+0000 mgr.y (mgr.34107) 4514 : cluster [DBG] pgmap v5885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:31 smithi115 bash[46934]: audit 2022-04-22T19:37:30.697039+0000 mon.a (mon.0) 7324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:32.120Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:33 smithi121 bash[36902]: cluster 2022-04-22T19:37:31.854651+0000 mgr.y (mgr.34107) 4515 : cluster [DBG] pgmap v5886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:33 smithi121 bash[36902]: audit 2022-04-22T19:37:32.181478+0000 mon.a (mon.0) 7325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:33 smithi115 bash[45495]: cluster 2022-04-22T19:37:31.854651+0000 mgr.y (mgr.34107) 4515 : cluster [DBG] pgmap v5886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:33.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:33 smithi115 bash[45495]: audit 2022-04-22T19:37:32.181478+0000 mon.a (mon.0) 7325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:33 smithi115 bash[46934]: cluster 2022-04-22T19:37:31.854651+0000 mgr.y (mgr.34107) 4515 : cluster [DBG] pgmap v5886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:33 smithi115 bash[46934]: audit 2022-04-22T19:37:32.181478+0000 mon.a (mon.0) 7325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:33.991Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:35 smithi121 bash[36902]: cluster 2022-04-22T19:37:33.855803+0000 mgr.y (mgr.34107) 4516 : cluster [DBG] pgmap v5887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:35 smithi121 bash[36902]: audit 2022-04-22T19:37:34.220237+0000 mon.a (mon.0) 7326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:35 smithi115 bash[45495]: cluster 2022-04-22T19:37:33.855803+0000 mgr.y (mgr.34107) 4516 : cluster [DBG] pgmap v5887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:35 smithi115 bash[45495]: audit 2022-04-22T19:37:34.220237+0000 mon.a (mon.0) 7326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:35 smithi115 bash[46934]: cluster 2022-04-22T19:37:33.855803+0000 mgr.y (mgr.34107) 4516 : cluster [DBG] pgmap v5887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:35 smithi115 bash[46934]: audit 2022-04-22T19:37:34.220237+0000 mon.a (mon.0) 7326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:36 smithi121 bash[36902]: audit 2022-04-22T19:37:35.699957+0000 mon.a (mon.0) 7327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:36 smithi115 bash[45495]: audit 2022-04-22T19:37:35.699957+0000 mon.a (mon.0) 7327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:36.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:36 smithi115 bash[46934]: audit 2022-04-22T19:37:35.699957+0000 mon.a (mon.0) 7327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:37 smithi121 bash[36902]: cluster 2022-04-22T19:37:35.856900+0000 mgr.y (mgr.34107) 4517 : cluster [DBG] pgmap v5888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:37 smithi115 bash[45495]: cluster 2022-04-22T19:37:35.856900+0000 mgr.y (mgr.34107) 4517 : cluster [DBG] pgmap v5888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:37 smithi115 bash[46934]: cluster 2022-04-22T19:37:35.856900+0000 mgr.y (mgr.34107) 4517 : cluster [DBG] pgmap v5888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:38 smithi121 bash[36902]: audit 2022-04-22T19:37:37.184611+0000 mon.a (mon.0) 7328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:38 smithi115 bash[45495]: audit 2022-04-22T19:37:37.184611+0000 mon.a (mon.0) 7328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:38 smithi115 bash[46934]: audit 2022-04-22T19:37:37.184611+0000 mon.a (mon.0) 7328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:39.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:39 smithi121 bash[36902]: cluster 2022-04-22T19:37:37.857995+0000 mgr.y (mgr.34107) 4518 : cluster [DBG] pgmap v5889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:39 smithi115 bash[45495]: cluster 2022-04-22T19:37:37.857995+0000 mgr.y (mgr.34107) 4518 : cluster [DBG] pgmap v5889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:39 smithi115 bash[46934]: cluster 2022-04-22T19:37:37.857995+0000 mgr.y (mgr.34107) 4518 : cluster [DBG] pgmap v5889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:40 smithi121 bash[36902]: audit 2022-04-22T19:37:39.223962+0000 mon.a (mon.0) 7329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:40 smithi115 bash[45495]: audit 2022-04-22T19:37:39.223962+0000 mon.a (mon.0) 7329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:40 smithi115 bash[46934]: audit 2022-04-22T19:37:39.223962+0000 mon.a (mon.0) 7329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:41 smithi121 bash[36902]: cluster 2022-04-22T19:37:39.859182+0000 mgr.y (mgr.34107) 4519 : cluster [DBG] pgmap v5890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:41 smithi121 bash[36902]: audit 2022-04-22T19:37:40.703343+0000 mon.a (mon.0) 7330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:41 smithi115 bash[45495]: cluster 2022-04-22T19:37:39.859182+0000 mgr.y (mgr.34107) 4519 : cluster [DBG] pgmap v5890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:41 smithi115 bash[45495]: audit 2022-04-22T19:37:40.703343+0000 mon.a (mon.0) 7330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:41.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:41 smithi115 bash[46934]: cluster 2022-04-22T19:37:39.859182+0000 mgr.y (mgr.34107) 4519 : cluster [DBG] pgmap v5890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:41 smithi115 bash[46934]: audit 2022-04-22T19:37:40.703343+0000 mon.a (mon.0) 7330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:42.122Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:43 smithi121 bash[36902]: cluster 2022-04-22T19:37:41.860246+0000 mgr.y (mgr.34107) 4520 : cluster [DBG] pgmap v5891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:43 smithi121 bash[36902]: audit 2022-04-22T19:37:42.187656+0000 mon.a (mon.0) 7331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:43 smithi115 bash[45495]: cluster 2022-04-22T19:37:41.860246+0000 mgr.y (mgr.34107) 4520 : cluster [DBG] pgmap v5891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:43 smithi115 bash[45495]: audit 2022-04-22T19:37:42.187656+0000 mon.a (mon.0) 7331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:43 smithi115 bash[46934]: cluster 2022-04-22T19:37:41.860246+0000 mgr.y (mgr.34107) 4520 : cluster [DBG] pgmap v5891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:43 smithi115 bash[46934]: audit 2022-04-22T19:37:42.187656+0000 mon.a (mon.0) 7331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:43.992Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:45 smithi121 bash[36902]: cluster 2022-04-22T19:37:43.861422+0000 mgr.y (mgr.34107) 4521 : cluster [DBG] pgmap v5892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:45.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:45 smithi121 bash[36902]: audit 2022-04-22T19:37:44.227515+0000 mon.a (mon.0) 7332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:45 smithi115 bash[45495]: cluster 2022-04-22T19:37:43.861422+0000 mgr.y (mgr.34107) 4521 : cluster [DBG] pgmap v5892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:45.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:45 smithi115 bash[45495]: audit 2022-04-22T19:37:44.227515+0000 mon.a (mon.0) 7332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:45.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:45 smithi115 bash[46934]: cluster 2022-04-22T19:37:43.861422+0000 mgr.y (mgr.34107) 4521 : cluster [DBG] pgmap v5892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:45 smithi115 bash[46934]: audit 2022-04-22T19:37:44.227515+0000 mon.a (mon.0) 7332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:46 smithi121 bash[36902]: audit 2022-04-22T19:37:45.706681+0000 mon.a (mon.0) 7333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:46.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:46 smithi115 bash[45495]: audit 2022-04-22T19:37:45.706681+0000 mon.a (mon.0) 7333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:46 smithi115 bash[46934]: audit 2022-04-22T19:37:45.706681+0000 mon.a (mon.0) 7333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:47 smithi121 bash[36902]: cluster 2022-04-22T19:37:45.862460+0000 mgr.y (mgr.34107) 4522 : cluster [DBG] pgmap v5893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:47.572 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:47 smithi115 bash[45495]: cluster 2022-04-22T19:37:45.862460+0000 mgr.y (mgr.34107) 4522 : cluster [DBG] pgmap v5893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:47.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:47 smithi115 bash[46934]: cluster 2022-04-22T19:37:45.862460+0000 mgr.y (mgr.34107) 4522 : cluster [DBG] pgmap v5893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:48 smithi121 bash[36902]: audit 2022-04-22T19:37:47.190582+0000 mon.a (mon.0) 7334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:48 smithi115 bash[45495]: audit 2022-04-22T19:37:47.190582+0000 mon.a (mon.0) 7334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:48.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:48 smithi115 bash[46934]: audit 2022-04-22T19:37:47.190582+0000 mon.a (mon.0) 7334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:49 smithi121 bash[36902]: cluster 2022-04-22T19:37:47.863598+0000 mgr.y (mgr.34107) 4523 : cluster [DBG] pgmap v5894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:49 smithi115 bash[45495]: cluster 2022-04-22T19:37:47.863598+0000 mgr.y (mgr.34107) 4523 : cluster [DBG] pgmap v5894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:49 smithi115 bash[46934]: cluster 2022-04-22T19:37:47.863598+0000 mgr.y (mgr.34107) 4523 : cluster [DBG] pgmap v5894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:50 smithi121 bash[36902]: audit 2022-04-22T19:37:49.230990+0000 mon.a (mon.0) 7335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:50 smithi115 bash[46934]: audit 2022-04-22T19:37:49.230990+0000 mon.a (mon.0) 7335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:50 smithi115 bash[45495]: audit 2022-04-22T19:37:49.230990+0000 mon.a (mon.0) 7335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:51 smithi121 bash[36902]: cluster 2022-04-22T19:37:49.865007+0000 mgr.y (mgr.34107) 4524 : cluster [DBG] pgmap v5895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:51 smithi121 bash[36902]: audit 2022-04-22T19:37:50.709803+0000 mon.a (mon.0) 7336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:51 smithi115 bash[45495]: cluster 2022-04-22T19:37:49.865007+0000 mgr.y (mgr.34107) 4524 : cluster [DBG] pgmap v5895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:51 smithi115 bash[45495]: audit 2022-04-22T19:37:50.709803+0000 mon.a (mon.0) 7336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:51 smithi115 bash[46934]: cluster 2022-04-22T19:37:49.865007+0000 mgr.y (mgr.34107) 4524 : cluster [DBG] pgmap v5895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:51 smithi115 bash[46934]: audit 2022-04-22T19:37:50.709803+0000 mon.a (mon.0) 7336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:52.122Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:53 smithi121 bash[36902]: cluster 2022-04-22T19:37:51.866100+0000 mgr.y (mgr.34107) 4525 : cluster [DBG] pgmap v5896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:53 smithi121 bash[36902]: audit 2022-04-22T19:37:52.193602+0000 mon.a (mon.0) 7337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:53 smithi115 bash[45495]: cluster 2022-04-22T19:37:51.866100+0000 mgr.y (mgr.34107) 4525 : cluster [DBG] pgmap v5896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:53 smithi115 bash[45495]: audit 2022-04-22T19:37:52.193602+0000 mon.a (mon.0) 7337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:53 smithi115 bash[46934]: cluster 2022-04-22T19:37:51.866100+0000 mgr.y (mgr.34107) 4525 : cluster [DBG] pgmap v5896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:53 smithi115 bash[46934]: audit 2022-04-22T19:37:52.193602+0000 mon.a (mon.0) 7337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:53.730 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:37:54.074 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:37:54.074 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:37:54.074 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:37:54.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:37:54.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:37:54.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:37:54.076 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:37:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:37:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:37:53.990Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:37:54.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:37:54.476 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:55 smithi121 bash[36902]: audit 2022-04-22T19:37:53.704651+0000 mgr.y (mgr.34107) 4526 : audit [DBG] from='client.40122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:55 smithi121 bash[36902]: cluster 2022-04-22T19:37:53.867186+0000 mgr.y (mgr.34107) 4527 : cluster [DBG] pgmap v5897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:55 smithi121 bash[36902]: audit 2022-04-22T19:37:54.046071+0000 mgr.y (mgr.34107) 4528 : audit [DBG] from='client.50062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:55 smithi121 bash[36902]: audit 2022-04-22T19:37:54.234503+0000 mon.a (mon.0) 7338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:55 smithi121 bash[36902]: audit 2022-04-22T19:37:54.464787+0000 mon.c (mon.1) 201 : audit [DBG] from='client.? 172.21.15.115:0/4088507317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[45495]: audit 2022-04-22T19:37:53.704651+0000 mgr.y (mgr.34107) 4526 : audit [DBG] from='client.40122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[45495]: cluster 2022-04-22T19:37:53.867186+0000 mgr.y (mgr.34107) 4527 : cluster [DBG] pgmap v5897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[45495]: audit 2022-04-22T19:37:54.046071+0000 mgr.y (mgr.34107) 4528 : audit [DBG] from='client.50062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[45495]: audit 2022-04-22T19:37:54.234503+0000 mon.a (mon.0) 7338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[45495]: audit 2022-04-22T19:37:54.464787+0000 mon.c (mon.1) 201 : audit [DBG] from='client.? 172.21.15.115:0/4088507317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[46934]: audit 2022-04-22T19:37:53.704651+0000 mgr.y (mgr.34107) 4526 : audit [DBG] from='client.40122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[46934]: cluster 2022-04-22T19:37:53.867186+0000 mgr.y (mgr.34107) 4527 : cluster [DBG] pgmap v5897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[46934]: audit 2022-04-22T19:37:54.046071+0000 mgr.y (mgr.34107) 4528 : audit [DBG] from='client.50062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:37:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[46934]: audit 2022-04-22T19:37:54.234503+0000 mon.a (mon.0) 7338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:37:55.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:55 smithi115 bash[46934]: audit 2022-04-22T19:37:54.464787+0000 mon.c (mon.1) 201 : audit [DBG] from='client.? 172.21.15.115:0/4088507317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:37:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:56 smithi121 bash[36902]: audit 2022-04-22T19:37:55.713024+0000 mon.a (mon.0) 7339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:56.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:56 smithi115 bash[46934]: audit 2022-04-22T19:37:55.713024+0000 mon.a (mon.0) 7339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:56 smithi115 bash[45495]: audit 2022-04-22T19:37:55.713024+0000 mon.a (mon.0) 7339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:37:57.451 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:57 smithi115 bash[45495]: cluster 2022-04-22T19:37:55.868252+0000 mgr.y (mgr.34107) 4529 : cluster [DBG] pgmap v5898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:57.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:57 smithi115 bash[46934]: cluster 2022-04-22T19:37:55.868252+0000 mgr.y (mgr.34107) 4529 : cluster [DBG] pgmap v5898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:57.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:57 smithi121 bash[36902]: cluster 2022-04-22T19:37:55.868252+0000 mgr.y (mgr.34107) 4529 : cluster [DBG] pgmap v5898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:58 smithi121 bash[36902]: audit 2022-04-22T19:37:57.196742+0000 mon.a (mon.0) 7340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:58 smithi121 bash[36902]: audit 2022-04-22T19:37:57.395836+0000 mon.a (mon.0) 7341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:37:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:58 smithi121 bash[36902]: audit 2022-04-22T19:37:57.398916+0000 mon.a (mon.0) 7342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:37:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:58 smithi121 bash[36902]: audit 2022-04-22T19:37:57.400784+0000 mon.a (mon.0) 7343 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:37:58.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:58 smithi121 bash[36902]: audit 2022-04-22T19:37:57.768925+0000 mon.a (mon.0) 7344 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:37:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[45495]: audit 2022-04-22T19:37:57.196742+0000 mon.a (mon.0) 7340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[45495]: audit 2022-04-22T19:37:57.395836+0000 mon.a (mon.0) 7341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[45495]: audit 2022-04-22T19:37:57.398916+0000 mon.a (mon.0) 7342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[45495]: audit 2022-04-22T19:37:57.400784+0000 mon.a (mon.0) 7343 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[45495]: audit 2022-04-22T19:37:57.768925+0000 mon.a (mon.0) 7344 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[46934]: audit 2022-04-22T19:37:57.196742+0000 mon.a (mon.0) 7340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[46934]: audit 2022-04-22T19:37:57.395836+0000 mon.a (mon.0) 7341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:37:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[46934]: audit 2022-04-22T19:37:57.398916+0000 mon.a (mon.0) 7342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:37:58.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[46934]: audit 2022-04-22T19:37:57.400784+0000 mon.a (mon.0) 7343 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:37:58.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:58 smithi115 bash[46934]: audit 2022-04-22T19:37:57.768925+0000 mon.a (mon.0) 7344 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:37:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:37:59 smithi121 bash[36902]: cluster 2022-04-22T19:37:57.869441+0000 mgr.y (mgr.34107) 4530 : cluster [DBG] pgmap v5899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:37:59 smithi115 bash[45495]: cluster 2022-04-22T19:37:57.869441+0000 mgr.y (mgr.34107) 4530 : cluster [DBG] pgmap v5899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:37:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:37:59 smithi115 bash[46934]: cluster 2022-04-22T19:37:57.869441+0000 mgr.y (mgr.34107) 4530 : cluster [DBG] pgmap v5899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:00 smithi121 bash[36902]: audit 2022-04-22T19:37:59.238029+0000 mon.a (mon.0) 7345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:00 smithi115 bash[45495]: audit 2022-04-22T19:37:59.238029+0000 mon.a (mon.0) 7345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:00 smithi115 bash[46934]: audit 2022-04-22T19:37:59.238029+0000 mon.a (mon.0) 7345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:01 smithi121 bash[36902]: cluster 2022-04-22T19:37:59.870517+0000 mgr.y (mgr.34107) 4531 : cluster [DBG] pgmap v5900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:01 smithi121 bash[36902]: audit 2022-04-22T19:38:00.716176+0000 mon.a (mon.0) 7346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:01 smithi115 bash[45495]: cluster 2022-04-22T19:37:59.870517+0000 mgr.y (mgr.34107) 4531 : cluster [DBG] pgmap v5900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:01.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:01 smithi115 bash[45495]: audit 2022-04-22T19:38:00.716176+0000 mon.a (mon.0) 7346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:01 smithi115 bash[46934]: cluster 2022-04-22T19:37:59.870517+0000 mgr.y (mgr.34107) 4531 : cluster [DBG] pgmap v5900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:01 smithi115 bash[46934]: audit 2022-04-22T19:38:00.716176+0000 mon.a (mon.0) 7346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:02 smithi121 bash[36902]: audit 2022-04-22T19:38:02.199128+0000 mon.a (mon.0) 7347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:02 smithi115 bash[45495]: audit 2022-04-22T19:38:02.199128+0000 mon.a (mon.0) 7347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:02.123Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:02.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:02 smithi115 bash[46934]: audit 2022-04-22T19:38:02.199128+0000 mon.a (mon.0) 7347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:03 smithi121 bash[36902]: cluster 2022-04-22T19:38:01.871572+0000 mgr.y (mgr.34107) 4532 : cluster [DBG] pgmap v5901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:03 smithi115 bash[45495]: cluster 2022-04-22T19:38:01.871572+0000 mgr.y (mgr.34107) 4532 : cluster [DBG] pgmap v5901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:03 smithi115 bash[46934]: cluster 2022-04-22T19:38:01.871572+0000 mgr.y (mgr.34107) 4532 : cluster [DBG] pgmap v5901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:04.299 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:03.992Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:05 smithi121 bash[36902]: cluster 2022-04-22T19:38:03.872695+0000 mgr.y (mgr.34107) 4533 : cluster [DBG] pgmap v5902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:05 smithi121 bash[36902]: audit 2022-04-22T19:38:04.241613+0000 mon.a (mon.0) 7348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:05 smithi115 bash[46934]: cluster 2022-04-22T19:38:03.872695+0000 mgr.y (mgr.34107) 4533 : cluster [DBG] pgmap v5902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:05.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:05 smithi115 bash[46934]: audit 2022-04-22T19:38:04.241613+0000 mon.a (mon.0) 7348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:05.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:05 smithi115 bash[45495]: cluster 2022-04-22T19:38:03.872695+0000 mgr.y (mgr.34107) 4533 : cluster [DBG] pgmap v5902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:05.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:05 smithi115 bash[45495]: audit 2022-04-22T19:38:04.241613+0000 mon.a (mon.0) 7348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:06 smithi121 bash[36902]: audit 2022-04-22T19:38:05.719540+0000 mon.a (mon.0) 7349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:06 smithi115 bash[45495]: audit 2022-04-22T19:38:05.719540+0000 mon.a (mon.0) 7349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:06.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:06 smithi115 bash[46934]: audit 2022-04-22T19:38:05.719540+0000 mon.a (mon.0) 7349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:07 smithi121 bash[36902]: cluster 2022-04-22T19:38:05.873673+0000 mgr.y (mgr.34107) 4534 : cluster [DBG] pgmap v5903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:07 smithi121 bash[36902]: audit 2022-04-22T19:38:07.202051+0000 mon.a (mon.0) 7350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:07 smithi115 bash[45495]: cluster 2022-04-22T19:38:05.873673+0000 mgr.y (mgr.34107) 4534 : cluster [DBG] pgmap v5903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:07.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:07 smithi115 bash[45495]: audit 2022-04-22T19:38:07.202051+0000 mon.a (mon.0) 7350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:07.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:07 smithi115 bash[46934]: cluster 2022-04-22T19:38:05.873673+0000 mgr.y (mgr.34107) 4534 : cluster [DBG] pgmap v5903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:07.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:07 smithi115 bash[46934]: audit 2022-04-22T19:38:07.202051+0000 mon.a (mon.0) 7350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:09 smithi121 bash[36902]: cluster 2022-04-22T19:38:07.874870+0000 mgr.y (mgr.34107) 4535 : cluster [DBG] pgmap v5904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:09 smithi115 bash[45495]: cluster 2022-04-22T19:38:07.874870+0000 mgr.y (mgr.34107) 4535 : cluster [DBG] pgmap v5904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:09 smithi115 bash[46934]: cluster 2022-04-22T19:38:07.874870+0000 mgr.y (mgr.34107) 4535 : cluster [DBG] pgmap v5904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:10 smithi121 bash[36902]: audit 2022-04-22T19:38:09.250850+0000 mon.a (mon.0) 7351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:10 smithi115 bash[45495]: audit 2022-04-22T19:38:09.250850+0000 mon.a (mon.0) 7351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:10 smithi115 bash[46934]: audit 2022-04-22T19:38:09.250850+0000 mon.a (mon.0) 7351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:11 smithi121 bash[36902]: cluster 2022-04-22T19:38:09.875987+0000 mgr.y (mgr.34107) 4536 : cluster [DBG] pgmap v5905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:11 smithi121 bash[36902]: audit 2022-04-22T19:38:10.722395+0000 mon.a (mon.0) 7352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:11 smithi115 bash[45495]: cluster 2022-04-22T19:38:09.875987+0000 mgr.y (mgr.34107) 4536 : cluster [DBG] pgmap v5905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:11.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:11 smithi115 bash[45495]: audit 2022-04-22T19:38:10.722395+0000 mon.a (mon.0) 7352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:11.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:11 smithi115 bash[46934]: cluster 2022-04-22T19:38:09.875987+0000 mgr.y (mgr.34107) 4536 : cluster [DBG] pgmap v5905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:11.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:11 smithi115 bash[46934]: audit 2022-04-22T19:38:10.722395+0000 mon.a (mon.0) 7352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:12.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:12 smithi115 bash[45495]: audit 2022-04-22T19:38:12.205654+0000 mon.a (mon.0) 7353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:12.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:12.123Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:12.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:12 smithi115 bash[46934]: audit 2022-04-22T19:38:12.205654+0000 mon.a (mon.0) 7353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:12 smithi121 bash[36902]: audit 2022-04-22T19:38:12.205654+0000 mon.a (mon.0) 7353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:13 smithi115 bash[45495]: cluster 2022-04-22T19:38:11.877193+0000 mgr.y (mgr.34107) 4537 : cluster [DBG] pgmap v5906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:13 smithi115 bash[46934]: cluster 2022-04-22T19:38:11.877193+0000 mgr.y (mgr.34107) 4537 : cluster [DBG] pgmap v5906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:13 smithi121 bash[36902]: cluster 2022-04-22T19:38:11.877193+0000 mgr.y (mgr.34107) 4537 : cluster [DBG] pgmap v5906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:14.283 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:13.993Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:14.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:14 smithi115 bash[46934]: audit 2022-04-22T19:38:14.249974+0000 mon.a (mon.0) 7354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:14.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:14 smithi115 bash[45495]: audit 2022-04-22T19:38:14.249974+0000 mon.a (mon.0) 7354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:14 smithi121 bash[36902]: audit 2022-04-22T19:38:14.249974+0000 mon.a (mon.0) 7354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[45495]: cluster 2022-04-22T19:38:13.878369+0000 mgr.y (mgr.34107) 4538 : cluster [DBG] pgmap v5907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[45495]: audit 2022-04-22T19:38:14.780592+0000 mon.a (mon.0) 7355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:38:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[45495]: audit 2022-04-22T19:38:14.785327+0000 mon.a (mon.0) 7356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:38:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[46934]: cluster 2022-04-22T19:38:13.878369+0000 mgr.y (mgr.34107) 4538 : cluster [DBG] pgmap v5907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[46934]: audit 2022-04-22T19:38:14.780592+0000 mon.a (mon.0) 7355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:38:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:15 smithi115 bash[46934]: audit 2022-04-22T19:38:14.785327+0000 mon.a (mon.0) 7356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:38:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:15 smithi121 bash[36902]: cluster 2022-04-22T19:38:13.878369+0000 mgr.y (mgr.34107) 4538 : cluster [DBG] pgmap v5907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:15 smithi121 bash[36902]: audit 2022-04-22T19:38:14.780592+0000 mon.a (mon.0) 7355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:38:15.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:15 smithi121 bash[36902]: audit 2022-04-22T19:38:14.785327+0000 mon.a (mon.0) 7356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:38:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:16 smithi115 bash[45495]: audit 2022-04-22T19:38:15.725413+0000 mon.a (mon.0) 7357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:16 smithi115 bash[46934]: audit 2022-04-22T19:38:15.725413+0000 mon.a (mon.0) 7357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:16 smithi121 bash[36902]: audit 2022-04-22T19:38:15.725413+0000 mon.a (mon.0) 7357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:17 smithi121 bash[36902]: cluster 2022-04-22T19:38:15.879422+0000 mgr.y (mgr.34107) 4539 : cluster [DBG] pgmap v5908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:17 smithi121 bash[36902]: audit 2022-04-22T19:38:17.208741+0000 mon.a (mon.0) 7358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:17.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:17 smithi115 bash[45495]: cluster 2022-04-22T19:38:15.879422+0000 mgr.y (mgr.34107) 4539 : cluster [DBG] pgmap v5908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:17.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:17 smithi115 bash[45495]: audit 2022-04-22T19:38:17.208741+0000 mon.a (mon.0) 7358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:17 smithi115 bash[46934]: cluster 2022-04-22T19:38:15.879422+0000 mgr.y (mgr.34107) 4539 : cluster [DBG] pgmap v5908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:17 smithi115 bash[46934]: audit 2022-04-22T19:38:17.208741+0000 mon.a (mon.0) 7358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:19 smithi121 bash[36902]: cluster 2022-04-22T19:38:17.880803+0000 mgr.y (mgr.34107) 4540 : cluster [DBG] pgmap v5909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:19 smithi121 bash[36902]: audit 2022-04-22T19:38:19.253851+0000 mon.a (mon.0) 7359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:19 smithi115 bash[45495]: cluster 2022-04-22T19:38:17.880803+0000 mgr.y (mgr.34107) 4540 : cluster [DBG] pgmap v5909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:19 smithi115 bash[45495]: audit 2022-04-22T19:38:19.253851+0000 mon.a (mon.0) 7359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:19 smithi115 bash[46934]: cluster 2022-04-22T19:38:17.880803+0000 mgr.y (mgr.34107) 4540 : cluster [DBG] pgmap v5909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:19 smithi115 bash[46934]: audit 2022-04-22T19:38:19.253851+0000 mon.a (mon.0) 7359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:21 smithi121 bash[36902]: cluster 2022-04-22T19:38:19.881967+0000 mgr.y (mgr.34107) 4541 : cluster [DBG] pgmap v5910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:21 smithi121 bash[36902]: audit 2022-04-22T19:38:20.728584+0000 mon.a (mon.0) 7360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:21 smithi115 bash[45495]: cluster 2022-04-22T19:38:19.881967+0000 mgr.y (mgr.34107) 4541 : cluster [DBG] pgmap v5910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:21 smithi115 bash[45495]: audit 2022-04-22T19:38:20.728584+0000 mon.a (mon.0) 7360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:21 smithi115 bash[46934]: cluster 2022-04-22T19:38:19.881967+0000 mgr.y (mgr.34107) 4541 : cluster [DBG] pgmap v5910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:21 smithi115 bash[46934]: audit 2022-04-22T19:38:20.728584+0000 mon.a (mon.0) 7360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:22.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:22 smithi115 bash[45495]: audit 2022-04-22T19:38:22.211763+0000 mon.a (mon.0) 7361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:22.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:22 smithi115 bash[46934]: audit 2022-04-22T19:38:22.211763+0000 mon.a (mon.0) 7361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:22.589 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:22.125Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:22 smithi121 bash[36902]: audit 2022-04-22T19:38:22.211763+0000 mon.a (mon.0) 7361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:23 smithi121 bash[36902]: cluster 2022-04-22T19:38:21.883114+0000 mgr.y (mgr.34107) 4542 : cluster [DBG] pgmap v5911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:23 smithi115 bash[45495]: cluster 2022-04-22T19:38:21.883114+0000 mgr.y (mgr.34107) 4542 : cluster [DBG] pgmap v5911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:23 smithi115 bash[46934]: cluster 2022-04-22T19:38:21.883114+0000 mgr.y (mgr.34107) 4542 : cluster [DBG] pgmap v5911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:23 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:23.993Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:24 smithi121 bash[36902]: audit 2022-04-22T19:38:24.257520+0000 mon.a (mon.0) 7362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:24.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:24 smithi115 bash[45495]: audit 2022-04-22T19:38:24.257520+0000 mon.a (mon.0) 7362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:24.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:24 smithi115 bash[46934]: audit 2022-04-22T19:38:24.257520+0000 mon.a (mon.0) 7362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:24.849 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:38:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:38:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:38:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:38:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:38:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:38:25.195 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:38:25.195 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:38:25.195 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:38:25.195 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:38:25.195 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:38:25.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:38:25.197 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:38:25.197 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:38:25.197 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:38:25.197 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:38:25.602 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:38:25.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:38:25.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:38:25.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:25.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:38:25.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:38:25.604 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:38:25.604 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:38:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:25 smithi121 bash[36902]: cluster 2022-04-22T19:38:23.884308+0000 mgr.y (mgr.34107) 4543 : cluster [DBG] pgmap v5912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:25 smithi115 bash[45495]: cluster 2022-04-22T19:38:23.884308+0000 mgr.y (mgr.34107) 4543 : cluster [DBG] pgmap v5912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:25 smithi115 bash[46934]: cluster 2022-04-22T19:38:23.884308+0000 mgr.y (mgr.34107) 4543 : cluster [DBG] pgmap v5912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:26 smithi121 bash[36902]: audit 2022-04-22T19:38:24.822788+0000 mgr.y (mgr.34107) 4544 : audit [DBG] from='client.40140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:26 smithi121 bash[36902]: audit 2022-04-22T19:38:25.164037+0000 mgr.y (mgr.34107) 4545 : audit [DBG] from='client.40146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:26 smithi121 bash[36902]: audit 2022-04-22T19:38:25.591628+0000 mon.a (mon.0) 7363 : audit [DBG] from='client.? 172.21.15.115:0/4008190750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:26 smithi121 bash[36902]: audit 2022-04-22T19:38:25.731592+0000 mon.a (mon.0) 7364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[45495]: audit 2022-04-22T19:38:24.822788+0000 mgr.y (mgr.34107) 4544 : audit [DBG] from='client.40140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[45495]: audit 2022-04-22T19:38:25.164037+0000 mgr.y (mgr.34107) 4545 : audit [DBG] from='client.40146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[45495]: audit 2022-04-22T19:38:25.591628+0000 mon.a (mon.0) 7363 : audit [DBG] from='client.? 172.21.15.115:0/4008190750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[45495]: audit 2022-04-22T19:38:25.731592+0000 mon.a (mon.0) 7364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[46934]: audit 2022-04-22T19:38:24.822788+0000 mgr.y (mgr.34107) 4544 : audit [DBG] from='client.40140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[46934]: audit 2022-04-22T19:38:25.164037+0000 mgr.y (mgr.34107) 4545 : audit [DBG] from='client.40146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[46934]: audit 2022-04-22T19:38:25.591628+0000 mon.a (mon.0) 7363 : audit [DBG] from='client.? 172.21.15.115:0/4008190750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:26 smithi115 bash[46934]: audit 2022-04-22T19:38:25.731592+0000 mon.a (mon.0) 7364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:27 smithi121 bash[36902]: cluster 2022-04-22T19:38:25.885510+0000 mgr.y (mgr.34107) 4546 : cluster [DBG] pgmap v5913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:27 smithi121 bash[36902]: audit 2022-04-22T19:38:27.214862+0000 mon.a (mon.0) 7365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:27 smithi115 bash[45495]: cluster 2022-04-22T19:38:25.885510+0000 mgr.y (mgr.34107) 4546 : cluster [DBG] pgmap v5913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:27 smithi115 bash[45495]: audit 2022-04-22T19:38:27.214862+0000 mon.a (mon.0) 7365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:27 smithi115 bash[46934]: cluster 2022-04-22T19:38:25.885510+0000 mgr.y (mgr.34107) 4546 : cluster [DBG] pgmap v5913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:27 smithi115 bash[46934]: audit 2022-04-22T19:38:27.214862+0000 mon.a (mon.0) 7365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:29 smithi121 bash[36902]: cluster 2022-04-22T19:38:27.886654+0000 mgr.y (mgr.34107) 4547 : cluster [DBG] pgmap v5914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:29 smithi121 bash[36902]: audit 2022-04-22T19:38:29.261066+0000 mon.a (mon.0) 7366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:29 smithi115 bash[45495]: cluster 2022-04-22T19:38:27.886654+0000 mgr.y (mgr.34107) 4547 : cluster [DBG] pgmap v5914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:29 smithi115 bash[45495]: audit 2022-04-22T19:38:29.261066+0000 mon.a (mon.0) 7366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:29 smithi115 bash[46934]: cluster 2022-04-22T19:38:27.886654+0000 mgr.y (mgr.34107) 4547 : cluster [DBG] pgmap v5914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:29.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:29 smithi115 bash[46934]: audit 2022-04-22T19:38:29.261066+0000 mon.a (mon.0) 7366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:31 smithi121 bash[36902]: cluster 2022-04-22T19:38:29.887748+0000 mgr.y (mgr.34107) 4548 : cluster [DBG] pgmap v5915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:31 smithi121 bash[36902]: audit 2022-04-22T19:38:30.734693+0000 mon.a (mon.0) 7367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:31 smithi115 bash[45495]: cluster 2022-04-22T19:38:29.887748+0000 mgr.y (mgr.34107) 4548 : cluster [DBG] pgmap v5915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:31 smithi115 bash[45495]: audit 2022-04-22T19:38:30.734693+0000 mon.a (mon.0) 7367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:31 smithi115 bash[46934]: cluster 2022-04-22T19:38:29.887748+0000 mgr.y (mgr.34107) 4548 : cluster [DBG] pgmap v5915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:31 smithi115 bash[46934]: audit 2022-04-22T19:38:30.734693+0000 mon.a (mon.0) 7367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:32.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:32 smithi115 bash[45495]: audit 2022-04-22T19:38:32.218011+0000 mon.a (mon.0) 7368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:32.125Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:32.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:32 smithi115 bash[46934]: audit 2022-04-22T19:38:32.218011+0000 mon.a (mon.0) 7368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:32 smithi121 bash[36902]: audit 2022-04-22T19:38:32.218011+0000 mon.a (mon.0) 7368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:33 smithi121 bash[36902]: cluster 2022-04-22T19:38:31.888852+0000 mgr.y (mgr.34107) 4549 : cluster [DBG] pgmap v5916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:33 smithi115 bash[45495]: cluster 2022-04-22T19:38:31.888852+0000 mgr.y (mgr.34107) 4549 : cluster [DBG] pgmap v5916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:33 smithi115 bash[46934]: cluster 2022-04-22T19:38:31.888852+0000 mgr.y (mgr.34107) 4549 : cluster [DBG] pgmap v5916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:33.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:34 smithi121 bash[36902]: audit 2022-04-22T19:38:34.264313+0000 mon.a (mon.0) 7369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:34 smithi115 bash[46934]: audit 2022-04-22T19:38:34.264313+0000 mon.a (mon.0) 7369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:34.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:34 smithi115 bash[45495]: audit 2022-04-22T19:38:34.264313+0000 mon.a (mon.0) 7369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:35 smithi121 bash[36902]: cluster 2022-04-22T19:38:33.889968+0000 mgr.y (mgr.34107) 4550 : cluster [DBG] pgmap v5917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:35 smithi115 bash[45495]: cluster 2022-04-22T19:38:33.889968+0000 mgr.y (mgr.34107) 4550 : cluster [DBG] pgmap v5917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:35 smithi115 bash[46934]: cluster 2022-04-22T19:38:33.889968+0000 mgr.y (mgr.34107) 4550 : cluster [DBG] pgmap v5917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:36 smithi121 bash[36902]: audit 2022-04-22T19:38:35.737666+0000 mon.a (mon.0) 7370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:36 smithi115 bash[45495]: audit 2022-04-22T19:38:35.737666+0000 mon.a (mon.0) 7370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:36.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:36 smithi115 bash[46934]: audit 2022-04-22T19:38:35.737666+0000 mon.a (mon.0) 7370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:37 smithi121 bash[36902]: cluster 2022-04-22T19:38:35.891011+0000 mgr.y (mgr.34107) 4551 : cluster [DBG] pgmap v5918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:37 smithi121 bash[36902]: audit 2022-04-22T19:38:37.221152+0000 mon.a (mon.0) 7371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:37 smithi115 bash[46934]: cluster 2022-04-22T19:38:35.891011+0000 mgr.y (mgr.34107) 4551 : cluster [DBG] pgmap v5918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:37 smithi115 bash[46934]: audit 2022-04-22T19:38:37.221152+0000 mon.a (mon.0) 7371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:37 smithi115 bash[45495]: cluster 2022-04-22T19:38:35.891011+0000 mgr.y (mgr.34107) 4551 : cluster [DBG] pgmap v5918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:37 smithi115 bash[45495]: audit 2022-04-22T19:38:37.221152+0000 mon.a (mon.0) 7371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:39 smithi121 bash[36902]: cluster 2022-04-22T19:38:37.892162+0000 mgr.y (mgr.34107) 4552 : cluster [DBG] pgmap v5919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:39 smithi121 bash[36902]: audit 2022-04-22T19:38:39.268050+0000 mon.a (mon.0) 7372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:39 smithi115 bash[45495]: cluster 2022-04-22T19:38:37.892162+0000 mgr.y (mgr.34107) 4552 : cluster [DBG] pgmap v5919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:39.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:39 smithi115 bash[45495]: audit 2022-04-22T19:38:39.268050+0000 mon.a (mon.0) 7372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:39 smithi115 bash[46934]: cluster 2022-04-22T19:38:37.892162+0000 mgr.y (mgr.34107) 4552 : cluster [DBG] pgmap v5919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:39 smithi115 bash[46934]: audit 2022-04-22T19:38:39.268050+0000 mon.a (mon.0) 7372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:41 smithi121 bash[36902]: cluster 2022-04-22T19:38:39.893224+0000 mgr.y (mgr.34107) 4553 : cluster [DBG] pgmap v5920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:41 smithi121 bash[36902]: audit 2022-04-22T19:38:40.740805+0000 mon.a (mon.0) 7373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:41.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:41 smithi115 bash[45495]: cluster 2022-04-22T19:38:39.893224+0000 mgr.y (mgr.34107) 4553 : cluster [DBG] pgmap v5920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:41.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:41 smithi115 bash[45495]: audit 2022-04-22T19:38:40.740805+0000 mon.a (mon.0) 7373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:41 smithi115 bash[46934]: cluster 2022-04-22T19:38:39.893224+0000 mgr.y (mgr.34107) 4553 : cluster [DBG] pgmap v5920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:41.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:41 smithi115 bash[46934]: audit 2022-04-22T19:38:40.740805+0000 mon.a (mon.0) 7373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:42.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:42.126Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:42 smithi121 bash[36902]: audit 2022-04-22T19:38:42.224387+0000 mon.a (mon.0) 7374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:42.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:42 smithi115 bash[45495]: audit 2022-04-22T19:38:42.224387+0000 mon.a (mon.0) 7374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:42.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:42 smithi115 bash[46934]: audit 2022-04-22T19:38:42.224387+0000 mon.a (mon.0) 7374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:43 smithi121 bash[36902]: cluster 2022-04-22T19:38:41.894349+0000 mgr.y (mgr.34107) 4554 : cluster [DBG] pgmap v5921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:43 smithi115 bash[45495]: cluster 2022-04-22T19:38:41.894349+0000 mgr.y (mgr.34107) 4554 : cluster [DBG] pgmap v5921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:43 smithi115 bash[46934]: cluster 2022-04-22T19:38:41.894349+0000 mgr.y (mgr.34107) 4554 : cluster [DBG] pgmap v5921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:43 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:43.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:44 smithi121 bash[36902]: audit 2022-04-22T19:38:44.271730+0000 mon.a (mon.0) 7375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:44.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:44 smithi115 bash[45495]: audit 2022-04-22T19:38:44.271730+0000 mon.a (mon.0) 7375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:44.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:44 smithi115 bash[46934]: audit 2022-04-22T19:38:44.271730+0000 mon.a (mon.0) 7375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:45 smithi121 bash[36902]: cluster 2022-04-22T19:38:43.895405+0000 mgr.y (mgr.34107) 4555 : cluster [DBG] pgmap v5922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:45 smithi115 bash[45495]: cluster 2022-04-22T19:38:43.895405+0000 mgr.y (mgr.34107) 4555 : cluster [DBG] pgmap v5922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:45 smithi115 bash[46934]: cluster 2022-04-22T19:38:43.895405+0000 mgr.y (mgr.34107) 4555 : cluster [DBG] pgmap v5922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:46 smithi121 bash[36902]: audit 2022-04-22T19:38:45.743895+0000 mon.a (mon.0) 7376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:46 smithi115 bash[45495]: audit 2022-04-22T19:38:45.743895+0000 mon.a (mon.0) 7376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:46 smithi115 bash[46934]: audit 2022-04-22T19:38:45.743895+0000 mon.a (mon.0) 7376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:47 smithi121 bash[36902]: cluster 2022-04-22T19:38:45.896429+0000 mgr.y (mgr.34107) 4556 : cluster [DBG] pgmap v5923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:47 smithi121 bash[36902]: audit 2022-04-22T19:38:47.227389+0000 mon.a (mon.0) 7377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:47 smithi115 bash[45495]: cluster 2022-04-22T19:38:45.896429+0000 mgr.y (mgr.34107) 4556 : cluster [DBG] pgmap v5923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:47 smithi115 bash[45495]: audit 2022-04-22T19:38:47.227389+0000 mon.a (mon.0) 7377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:47 smithi115 bash[46934]: cluster 2022-04-22T19:38:45.896429+0000 mgr.y (mgr.34107) 4556 : cluster [DBG] pgmap v5923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:47 smithi115 bash[46934]: audit 2022-04-22T19:38:47.227389+0000 mon.a (mon.0) 7377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:49 smithi121 bash[36902]: cluster 2022-04-22T19:38:47.897688+0000 mgr.y (mgr.34107) 4557 : cluster [DBG] pgmap v5924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:49 smithi121 bash[36902]: audit 2022-04-22T19:38:49.275497+0000 mon.a (mon.0) 7378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:49 smithi115 bash[45495]: cluster 2022-04-22T19:38:47.897688+0000 mgr.y (mgr.34107) 4557 : cluster [DBG] pgmap v5924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:49 smithi115 bash[45495]: audit 2022-04-22T19:38:49.275497+0000 mon.a (mon.0) 7378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:49 smithi115 bash[46934]: cluster 2022-04-22T19:38:47.897688+0000 mgr.y (mgr.34107) 4557 : cluster [DBG] pgmap v5924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:49 smithi115 bash[46934]: audit 2022-04-22T19:38:49.275497+0000 mon.a (mon.0) 7378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:51 smithi121 bash[36902]: cluster 2022-04-22T19:38:49.898920+0000 mgr.y (mgr.34107) 4558 : cluster [DBG] pgmap v5925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:51 smithi121 bash[36902]: audit 2022-04-22T19:38:50.747095+0000 mon.a (mon.0) 7379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:51 smithi115 bash[45495]: cluster 2022-04-22T19:38:49.898920+0000 mgr.y (mgr.34107) 4558 : cluster [DBG] pgmap v5925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:51 smithi115 bash[45495]: audit 2022-04-22T19:38:50.747095+0000 mon.a (mon.0) 7379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:51 smithi115 bash[46934]: cluster 2022-04-22T19:38:49.898920+0000 mgr.y (mgr.34107) 4558 : cluster [DBG] pgmap v5925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:51 smithi115 bash[46934]: audit 2022-04-22T19:38:50.747095+0000 mon.a (mon.0) 7379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:52.438 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:52.127Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:52 smithi121 bash[36902]: audit 2022-04-22T19:38:52.230246+0000 mon.a (mon.0) 7380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:52.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:52 smithi115 bash[45495]: audit 2022-04-22T19:38:52.230246+0000 mon.a (mon.0) 7380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:52.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:52 smithi115 bash[46934]: audit 2022-04-22T19:38:52.230246+0000 mon.a (mon.0) 7380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:53 smithi121 bash[36902]: cluster 2022-04-22T19:38:51.900212+0000 mgr.y (mgr.34107) 4559 : cluster [DBG] pgmap v5926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:53 smithi115 bash[45495]: cluster 2022-04-22T19:38:51.900212+0000 mgr.y (mgr.34107) 4559 : cluster [DBG] pgmap v5926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:53 smithi115 bash[46934]: cluster 2022-04-22T19:38:51.900212+0000 mgr.y (mgr.34107) 4559 : cluster [DBG] pgmap v5926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:38:53 smithi115 bash[60466]: level=error ts=2022-04-22T19:38:53.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:38:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:54 smithi121 bash[36902]: audit 2022-04-22T19:38:54.279272+0000 mon.a (mon.0) 7381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:54.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:54 smithi115 bash[45495]: audit 2022-04-22T19:38:54.279272+0000 mon.a (mon.0) 7381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:54.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:54 smithi115 bash[46934]: audit 2022-04-22T19:38:54.279272+0000 mon.a (mon.0) 7381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:55 smithi121 bash[36902]: cluster 2022-04-22T19:38:53.901320+0000 mgr.y (mgr.34107) 4560 : cluster [DBG] pgmap v5927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:55 smithi115 bash[46934]: cluster 2022-04-22T19:38:53.901320+0000 mgr.y (mgr.34107) 4560 : cluster [DBG] pgmap v5927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:55 smithi115 bash[45495]: cluster 2022-04-22T19:38:53.901320+0000 mgr.y (mgr.34107) 4560 : cluster [DBG] pgmap v5927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:55.975 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:38:56.330 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:38:56.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:38:56.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:38:56.333 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:38:56.333 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:38:56.736 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:38:56.737 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:38:56.738 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:38:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:56 smithi121 bash[36902]: audit 2022-04-22T19:38:55.749958+0000 mon.a (mon.0) 7382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:56 smithi115 bash[45495]: audit 2022-04-22T19:38:55.749958+0000 mon.a (mon.0) 7382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:56 smithi115 bash[46934]: audit 2022-04-22T19:38:55.749958+0000 mon.a (mon.0) 7382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:38:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:57 smithi121 bash[36902]: cluster 2022-04-22T19:38:55.902052+0000 mgr.y (mgr.34107) 4561 : cluster [DBG] pgmap v5928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:57 smithi121 bash[36902]: audit 2022-04-22T19:38:55.948747+0000 mgr.y (mgr.34107) 4562 : audit [DBG] from='client.40158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:57 smithi121 bash[36902]: audit 2022-04-22T19:38:56.301704+0000 mgr.y (mgr.34107) 4563 : audit [DBG] from='client.40164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:57 smithi121 bash[36902]: audit 2022-04-22T19:38:56.725806+0000 mon.a (mon.0) 7383 : audit [DBG] from='client.? 172.21.15.115:0/3014931812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:57 smithi121 bash[36902]: audit 2022-04-22T19:38:57.232907+0000 mon.a (mon.0) 7384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[45495]: cluster 2022-04-22T19:38:55.902052+0000 mgr.y (mgr.34107) 4561 : cluster [DBG] pgmap v5928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[45495]: audit 2022-04-22T19:38:55.948747+0000 mgr.y (mgr.34107) 4562 : audit [DBG] from='client.40158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[45495]: audit 2022-04-22T19:38:56.301704+0000 mgr.y (mgr.34107) 4563 : audit [DBG] from='client.40164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[45495]: audit 2022-04-22T19:38:56.725806+0000 mon.a (mon.0) 7383 : audit [DBG] from='client.? 172.21.15.115:0/3014931812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[45495]: audit 2022-04-22T19:38:57.232907+0000 mon.a (mon.0) 7384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[46934]: cluster 2022-04-22T19:38:55.902052+0000 mgr.y (mgr.34107) 4561 : cluster [DBG] pgmap v5928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[46934]: audit 2022-04-22T19:38:55.948747+0000 mgr.y (mgr.34107) 4562 : audit [DBG] from='client.40158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[46934]: audit 2022-04-22T19:38:56.301704+0000 mgr.y (mgr.34107) 4563 : audit [DBG] from='client.40164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:38:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[46934]: audit 2022-04-22T19:38:56.725806+0000 mon.a (mon.0) 7383 : audit [DBG] from='client.? 172.21.15.115:0/3014931812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:38:57.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:57 smithi115 bash[46934]: audit 2022-04-22T19:38:57.232907+0000 mon.a (mon.0) 7384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:38:58.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[45495]: audit 2022-04-22T19:38:57.803333+0000 mon.a (mon.0) 7385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:38:58.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[45495]: audit 2022-04-22T19:38:57.806262+0000 mon.a (mon.0) 7386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:38:58.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[45495]: audit 2022-04-22T19:38:57.808147+0000 mon.a (mon.0) 7387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:38:58.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[45495]: audit 2022-04-22T19:38:58.405117+0000 mon.a (mon.0) 7388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:38:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:58 smithi121 bash[36902]: audit 2022-04-22T19:38:57.803333+0000 mon.a (mon.0) 7385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:38:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:58 smithi121 bash[36902]: audit 2022-04-22T19:38:57.806262+0000 mon.a (mon.0) 7386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:38:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:58 smithi121 bash[36902]: audit 2022-04-22T19:38:57.808147+0000 mon.a (mon.0) 7387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:38:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:58 smithi121 bash[36902]: audit 2022-04-22T19:38:58.405117+0000 mon.a (mon.0) 7388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:38:58.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[46934]: audit 2022-04-22T19:38:57.803333+0000 mon.a (mon.0) 7385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:38:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[46934]: audit 2022-04-22T19:38:57.806262+0000 mon.a (mon.0) 7386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:38:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[46934]: audit 2022-04-22T19:38:57.808147+0000 mon.a (mon.0) 7387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:38:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:58 smithi115 bash[46934]: audit 2022-04-22T19:38:58.405117+0000 mon.a (mon.0) 7388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:38:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:59 smithi121 bash[36902]: cluster 2022-04-22T19:38:57.903006+0000 mgr.y (mgr.34107) 4564 : cluster [DBG] pgmap v5929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:38:59 smithi121 bash[36902]: audit 2022-04-22T19:38:59.283012+0000 mon.a (mon.0) 7389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:59 smithi115 bash[45495]: cluster 2022-04-22T19:38:57.903006+0000 mgr.y (mgr.34107) 4564 : cluster [DBG] pgmap v5929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:38:59 smithi115 bash[45495]: audit 2022-04-22T19:38:59.283012+0000 mon.a (mon.0) 7389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:38:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:59 smithi115 bash[46934]: cluster 2022-04-22T19:38:57.903006+0000 mgr.y (mgr.34107) 4564 : cluster [DBG] pgmap v5929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:38:59.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:38:59 smithi115 bash[46934]: audit 2022-04-22T19:38:59.283012+0000 mon.a (mon.0) 7389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:01 smithi121 bash[36902]: cluster 2022-04-22T19:38:59.904085+0000 mgr.y (mgr.34107) 4565 : cluster [DBG] pgmap v5930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:01 smithi121 bash[36902]: audit 2022-04-22T19:39:00.752628+0000 mon.a (mon.0) 7390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:01 smithi115 bash[45495]: cluster 2022-04-22T19:38:59.904085+0000 mgr.y (mgr.34107) 4565 : cluster [DBG] pgmap v5930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:01 smithi115 bash[45495]: audit 2022-04-22T19:39:00.752628+0000 mon.a (mon.0) 7390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:01 smithi115 bash[46934]: cluster 2022-04-22T19:38:59.904085+0000 mgr.y (mgr.34107) 4565 : cluster [DBG] pgmap v5930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:01.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:01 smithi115 bash[46934]: audit 2022-04-22T19:39:00.752628+0000 mon.a (mon.0) 7390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:02.488 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:02.127Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:02.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:02 smithi121 bash[36902]: audit 2022-04-22T19:39:02.236297+0000 mon.a (mon.0) 7391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:02.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:02 smithi115 bash[45495]: audit 2022-04-22T19:39:02.236297+0000 mon.a (mon.0) 7391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:02.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:02 smithi115 bash[46934]: audit 2022-04-22T19:39:02.236297+0000 mon.a (mon.0) 7391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:03 smithi121 bash[36902]: cluster 2022-04-22T19:39:01.905205+0000 mgr.y (mgr.34107) 4566 : cluster [DBG] pgmap v5931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:03 smithi115 bash[46934]: cluster 2022-04-22T19:39:01.905205+0000 mgr.y (mgr.34107) 4566 : cluster [DBG] pgmap v5931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:03.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:03 smithi115 bash[45495]: cluster 2022-04-22T19:39:01.905205+0000 mgr.y (mgr.34107) 4566 : cluster [DBG] pgmap v5931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:03 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:03.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:04 smithi121 bash[36902]: audit 2022-04-22T19:39:04.286761+0000 mon.a (mon.0) 7392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:04 smithi115 bash[45495]: audit 2022-04-22T19:39:04.286761+0000 mon.a (mon.0) 7392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:04 smithi115 bash[46934]: audit 2022-04-22T19:39:04.286761+0000 mon.a (mon.0) 7392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:05 smithi121 bash[36902]: cluster 2022-04-22T19:39:03.906434+0000 mgr.y (mgr.34107) 4567 : cluster [DBG] pgmap v5932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:05 smithi115 bash[45495]: cluster 2022-04-22T19:39:03.906434+0000 mgr.y (mgr.34107) 4567 : cluster [DBG] pgmap v5932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:05 smithi115 bash[46934]: cluster 2022-04-22T19:39:03.906434+0000 mgr.y (mgr.34107) 4567 : cluster [DBG] pgmap v5932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:06 smithi121 bash[36902]: audit 2022-04-22T19:39:05.755713+0000 mon.a (mon.0) 7393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:06 smithi115 bash[45495]: audit 2022-04-22T19:39:05.755713+0000 mon.a (mon.0) 7393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:06 smithi115 bash[46934]: audit 2022-04-22T19:39:05.755713+0000 mon.a (mon.0) 7393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:07 smithi121 bash[36902]: cluster 2022-04-22T19:39:05.907562+0000 mgr.y (mgr.34107) 4568 : cluster [DBG] pgmap v5933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:07 smithi121 bash[36902]: audit 2022-04-22T19:39:06.905326+0000 mon.a (mon.0) 7394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:07 smithi121 bash[36902]: audit 2022-04-22T19:39:06.912706+0000 mon.a (mon.0) 7395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:07 smithi121 bash[36902]: audit 2022-04-22T19:39:07.239163+0000 mon.a (mon.0) 7396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[45495]: cluster 2022-04-22T19:39:05.907562+0000 mgr.y (mgr.34107) 4568 : cluster [DBG] pgmap v5933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[45495]: audit 2022-04-22T19:39:06.905326+0000 mon.a (mon.0) 7394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[45495]: audit 2022-04-22T19:39:06.912706+0000 mon.a (mon.0) 7395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[45495]: audit 2022-04-22T19:39:07.239163+0000 mon.a (mon.0) 7396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[46934]: cluster 2022-04-22T19:39:05.907562+0000 mgr.y (mgr.34107) 4568 : cluster [DBG] pgmap v5933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[46934]: audit 2022-04-22T19:39:06.905326+0000 mon.a (mon.0) 7394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[46934]: audit 2022-04-22T19:39:06.912706+0000 mon.a (mon.0) 7395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:39:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:07 smithi115 bash[46934]: audit 2022-04-22T19:39:07.239163+0000 mon.a (mon.0) 7396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:08.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:08 smithi115 bash[45495]: cephadm 2022-04-22T19:39:06.893537+0000 mgr.y (mgr.34107) 4569 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:39:08.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:08 smithi115 bash[46934]: cephadm 2022-04-22T19:39:06.893537+0000 mgr.y (mgr.34107) 4569 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:39:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:08 smithi121 bash[36902]: cephadm 2022-04-22T19:39:06.893537+0000 mgr.y (mgr.34107) 4569 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T19:39:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:09 smithi115 bash[45495]: cluster 2022-04-22T19:39:07.908620+0000 mgr.y (mgr.34107) 4570 : cluster [DBG] pgmap v5934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:09 smithi115 bash[45495]: audit 2022-04-22T19:39:09.290583+0000 mon.a (mon.0) 7397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:09 smithi115 bash[46934]: cluster 2022-04-22T19:39:07.908620+0000 mgr.y (mgr.34107) 4570 : cluster [DBG] pgmap v5934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:09 smithi115 bash[46934]: audit 2022-04-22T19:39:09.290583+0000 mon.a (mon.0) 7397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:09 smithi121 bash[36902]: cluster 2022-04-22T19:39:07.908620+0000 mgr.y (mgr.34107) 4570 : cluster [DBG] pgmap v5934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:09 smithi121 bash[36902]: audit 2022-04-22T19:39:09.290583+0000 mon.a (mon.0) 7397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:11 smithi115 bash[45495]: cluster 2022-04-22T19:39:09.909736+0000 mgr.y (mgr.34107) 4571 : cluster [DBG] pgmap v5935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:11 smithi115 bash[45495]: audit 2022-04-22T19:39:10.759216+0000 mon.a (mon.0) 7398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:11 smithi115 bash[46934]: cluster 2022-04-22T19:39:09.909736+0000 mgr.y (mgr.34107) 4571 : cluster [DBG] pgmap v5935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:11.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:11 smithi115 bash[46934]: audit 2022-04-22T19:39:10.759216+0000 mon.a (mon.0) 7398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:11 smithi121 bash[36902]: cluster 2022-04-22T19:39:09.909736+0000 mgr.y (mgr.34107) 4571 : cluster [DBG] pgmap v5935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:11 smithi121 bash[36902]: audit 2022-04-22T19:39:10.759216+0000 mon.a (mon.0) 7398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:12.533 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:12.127Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:12 smithi115 bash[45495]: audit 2022-04-22T19:39:12.242427+0000 mon.a (mon.0) 7399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:12 smithi115 bash[46934]: audit 2022-04-22T19:39:12.242427+0000 mon.a (mon.0) 7399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:12 smithi121 bash[36902]: audit 2022-04-22T19:39:12.242427+0000 mon.a (mon.0) 7399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:13 smithi115 bash[45495]: cluster 2022-04-22T19:39:11.911010+0000 mgr.y (mgr.34107) 4572 : cluster [DBG] pgmap v5936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:13 smithi115 bash[46934]: cluster 2022-04-22T19:39:11.911010+0000 mgr.y (mgr.34107) 4572 : cluster [DBG] pgmap v5936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:13 smithi121 bash[36902]: cluster 2022-04-22T19:39:11.911010+0000 mgr.y (mgr.34107) 4572 : cluster [DBG] pgmap v5936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:13 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:13.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:14 smithi115 bash[45495]: audit 2022-04-22T19:39:14.294226+0000 mon.a (mon.0) 7400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:14.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:14 smithi115 bash[46934]: audit 2022-04-22T19:39:14.294226+0000 mon.a (mon.0) 7400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:14 smithi121 bash[36902]: audit 2022-04-22T19:39:14.294226+0000 mon.a (mon.0) 7400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[45495]: cluster 2022-04-22T19:39:13.912100+0000 mgr.y (mgr.34107) 4573 : cluster [DBG] pgmap v5937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[45495]: audit 2022-04-22T19:39:14.795346+0000 mon.a (mon.0) 7401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:39:15.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[45495]: audit 2022-04-22T19:39:14.795843+0000 mon.a (mon.0) 7402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:39:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[46934]: cluster 2022-04-22T19:39:13.912100+0000 mgr.y (mgr.34107) 4573 : cluster [DBG] pgmap v5937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[46934]: audit 2022-04-22T19:39:14.795346+0000 mon.a (mon.0) 7401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:39:15.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:15 smithi115 bash[46934]: audit 2022-04-22T19:39:14.795843+0000 mon.a (mon.0) 7402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:39:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:15 smithi121 bash[36902]: cluster 2022-04-22T19:39:13.912100+0000 mgr.y (mgr.34107) 4573 : cluster [DBG] pgmap v5937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:15 smithi121 bash[36902]: audit 2022-04-22T19:39:14.795346+0000 mon.a (mon.0) 7401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:39:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:15 smithi121 bash[36902]: audit 2022-04-22T19:39:14.795843+0000 mon.a (mon.0) 7402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:39:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:16 smithi121 bash[36902]: audit 2022-04-22T19:39:15.762278+0000 mon.a (mon.0) 7403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:16 smithi115 bash[45495]: audit 2022-04-22T19:39:15.762278+0000 mon.a (mon.0) 7403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:17.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:16 smithi115 bash[46934]: audit 2022-04-22T19:39:15.762278+0000 mon.a (mon.0) 7403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:17 smithi121 bash[36902]: cluster 2022-04-22T19:39:15.913047+0000 mgr.y (mgr.34107) 4574 : cluster [DBG] pgmap v5938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:17 smithi121 bash[36902]: audit 2022-04-22T19:39:17.245244+0000 mon.a (mon.0) 7404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:17 smithi115 bash[46934]: cluster 2022-04-22T19:39:15.913047+0000 mgr.y (mgr.34107) 4574 : cluster [DBG] pgmap v5938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:17 smithi115 bash[46934]: audit 2022-04-22T19:39:17.245244+0000 mon.a (mon.0) 7404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:18.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:17 smithi115 bash[45495]: cluster 2022-04-22T19:39:15.913047+0000 mgr.y (mgr.34107) 4574 : cluster [DBG] pgmap v5938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:18.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:17 smithi115 bash[45495]: audit 2022-04-22T19:39:17.245244+0000 mon.a (mon.0) 7404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:19 smithi121 bash[36902]: cluster 2022-04-22T19:39:17.914261+0000 mgr.y (mgr.34107) 4575 : cluster [DBG] pgmap v5939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:19 smithi121 bash[36902]: audit 2022-04-22T19:39:19.298102+0000 mon.a (mon.0) 7405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:19 smithi115 bash[45495]: cluster 2022-04-22T19:39:17.914261+0000 mgr.y (mgr.34107) 4575 : cluster [DBG] pgmap v5939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:20.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:19 smithi115 bash[45495]: audit 2022-04-22T19:39:19.298102+0000 mon.a (mon.0) 7405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:19 smithi115 bash[46934]: cluster 2022-04-22T19:39:17.914261+0000 mgr.y (mgr.34107) 4575 : cluster [DBG] pgmap v5939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:19 smithi115 bash[46934]: audit 2022-04-22T19:39:19.298102+0000 mon.a (mon.0) 7405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:21 smithi121 bash[36902]: cluster 2022-04-22T19:39:19.915534+0000 mgr.y (mgr.34107) 4576 : cluster [DBG] pgmap v5940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:21 smithi121 bash[36902]: audit 2022-04-22T19:39:20.765350+0000 mon.a (mon.0) 7406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:21 smithi115 bash[46934]: cluster 2022-04-22T19:39:19.915534+0000 mgr.y (mgr.34107) 4576 : cluster [DBG] pgmap v5940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:22.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:21 smithi115 bash[46934]: audit 2022-04-22T19:39:20.765350+0000 mon.a (mon.0) 7406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:22.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:21 smithi115 bash[45495]: cluster 2022-04-22T19:39:19.915534+0000 mgr.y (mgr.34107) 4576 : cluster [DBG] pgmap v5940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:22.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:21 smithi115 bash[45495]: audit 2022-04-22T19:39:20.765350+0000 mon.a (mon.0) 7406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:22.129Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:22 smithi121 bash[36902]: audit 2022-04-22T19:39:22.248432+0000 mon.a (mon.0) 7407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:22 smithi115 bash[45495]: audit 2022-04-22T19:39:22.248432+0000 mon.a (mon.0) 7407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:22 smithi115 bash[46934]: audit 2022-04-22T19:39:22.248432+0000 mon.a (mon.0) 7407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:24.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:23 smithi115 bash[46934]: cluster 2022-04-22T19:39:21.916835+0000 mgr.y (mgr.34107) 4577 : cluster [DBG] pgmap v5941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:24.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:23 smithi115 bash[45495]: cluster 2022-04-22T19:39:21.916835+0000 mgr.y (mgr.34107) 4577 : cluster [DBG] pgmap v5941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:24.038 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:23 smithi121 bash[36902]: cluster 2022-04-22T19:39:21.916835+0000 mgr.y (mgr.34107) 4577 : cluster [DBG] pgmap v5941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:24.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:24 smithi121 bash[36902]: audit 2022-04-22T19:39:24.301800+0000 mon.a (mon.0) 7408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:24 smithi115 bash[45495]: audit 2022-04-22T19:39:24.301800+0000 mon.a (mon.0) 7408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:25.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:24 smithi115 bash[46934]: audit 2022-04-22T19:39:24.301800+0000 mon.a (mon.0) 7408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:25 smithi121 bash[36902]: cluster 2022-04-22T19:39:23.918085+0000 mgr.y (mgr.34107) 4578 : cluster [DBG] pgmap v5942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:25 smithi115 bash[45495]: cluster 2022-04-22T19:39:23.918085+0000 mgr.y (mgr.34107) 4578 : cluster [DBG] pgmap v5942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:26.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:25 smithi115 bash[46934]: cluster 2022-04-22T19:39:23.918085+0000 mgr.y (mgr.34107) 4578 : cluster [DBG] pgmap v5942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:26 smithi121 bash[36902]: audit 2022-04-22T19:39:25.768503+0000 mon.a (mon.0) 7409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:26 smithi115 bash[45495]: audit 2022-04-22T19:39:25.768503+0000 mon.a (mon.0) 7409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:27.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:26 smithi115 bash[46934]: audit 2022-04-22T19:39:25.768503+0000 mon.a (mon.0) 7409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:27.107 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:39:27.454 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:39:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:39:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:39:27.457 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:39:27.457 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:27.859 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:39:27.860 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:39:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:27 smithi121 bash[36902]: cluster 2022-04-22T19:39:25.919044+0000 mgr.y (mgr.34107) 4579 : cluster [DBG] pgmap v5943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:27 smithi121 bash[36902]: audit 2022-04-22T19:39:27.251573+0000 mon.a (mon.0) 7410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:27 smithi115 bash[45495]: cluster 2022-04-22T19:39:25.919044+0000 mgr.y (mgr.34107) 4579 : cluster [DBG] pgmap v5943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:27 smithi115 bash[45495]: audit 2022-04-22T19:39:27.251573+0000 mon.a (mon.0) 7410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:27 smithi115 bash[46934]: cluster 2022-04-22T19:39:25.919044+0000 mgr.y (mgr.34107) 4579 : cluster [DBG] pgmap v5943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:27 smithi115 bash[46934]: audit 2022-04-22T19:39:27.251573+0000 mon.a (mon.0) 7410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:28 smithi121 bash[36902]: audit 2022-04-22T19:39:27.080446+0000 mgr.y (mgr.34107) 4580 : audit [DBG] from='client.40176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:28 smithi121 bash[36902]: audit 2022-04-22T19:39:27.425345+0000 mgr.y (mgr.34107) 4581 : audit [DBG] from='client.40182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:28 smithi121 bash[36902]: audit 2022-04-22T19:39:27.848278+0000 mon.a (mon.0) 7411 : audit [DBG] from='client.? 172.21.15.115:0/787971925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:39:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[45495]: audit 2022-04-22T19:39:27.080446+0000 mgr.y (mgr.34107) 4580 : audit [DBG] from='client.40176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[45495]: audit 2022-04-22T19:39:27.425345+0000 mgr.y (mgr.34107) 4581 : audit [DBG] from='client.40182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[45495]: audit 2022-04-22T19:39:27.848278+0000 mon.a (mon.0) 7411 : audit [DBG] from='client.? 172.21.15.115:0/787971925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:39:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[46934]: audit 2022-04-22T19:39:27.080446+0000 mgr.y (mgr.34107) 4580 : audit [DBG] from='client.40176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[46934]: audit 2022-04-22T19:39:27.425345+0000 mgr.y (mgr.34107) 4581 : audit [DBG] from='client.40182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:28 smithi115 bash[46934]: audit 2022-04-22T19:39:27.848278+0000 mon.a (mon.0) 7411 : audit [DBG] from='client.? 172.21.15.115:0/787971925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:39:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:29 smithi121 bash[36902]: cluster 2022-04-22T19:39:27.920142+0000 mgr.y (mgr.34107) 4582 : cluster [DBG] pgmap v5944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:29 smithi121 bash[36902]: audit 2022-04-22T19:39:29.305465+0000 mon.a (mon.0) 7412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:29 smithi115 bash[45495]: cluster 2022-04-22T19:39:27.920142+0000 mgr.y (mgr.34107) 4582 : cluster [DBG] pgmap v5944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:30.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:29 smithi115 bash[45495]: audit 2022-04-22T19:39:29.305465+0000 mon.a (mon.0) 7412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:29 smithi115 bash[46934]: cluster 2022-04-22T19:39:27.920142+0000 mgr.y (mgr.34107) 4582 : cluster [DBG] pgmap v5944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:29 smithi115 bash[46934]: audit 2022-04-22T19:39:29.305465+0000 mon.a (mon.0) 7412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:31 smithi121 bash[36902]: cluster 2022-04-22T19:39:29.921360+0000 mgr.y (mgr.34107) 4583 : cluster [DBG] pgmap v5945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:31 smithi121 bash[36902]: audit 2022-04-22T19:39:30.771512+0000 mon.a (mon.0) 7413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:31 smithi115 bash[45495]: cluster 2022-04-22T19:39:29.921360+0000 mgr.y (mgr.34107) 4583 : cluster [DBG] pgmap v5945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:32.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:31 smithi115 bash[45495]: audit 2022-04-22T19:39:30.771512+0000 mon.a (mon.0) 7413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:31 smithi115 bash[46934]: cluster 2022-04-22T19:39:29.921360+0000 mgr.y (mgr.34107) 4583 : cluster [DBG] pgmap v5945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:31 smithi115 bash[46934]: audit 2022-04-22T19:39:30.771512+0000 mon.a (mon.0) 7413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:32.130Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:32 smithi121 bash[36902]: cluster 2022-04-22T19:39:31.922639+0000 mgr.y (mgr.34107) 4584 : cluster [DBG] pgmap v5946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:32 smithi121 bash[36902]: audit 2022-04-22T19:39:32.254341+0000 mon.a (mon.0) 7414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:32 smithi115 bash[46934]: cluster 2022-04-22T19:39:31.922639+0000 mgr.y (mgr.34107) 4584 : cluster [DBG] pgmap v5946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:32 smithi115 bash[46934]: audit 2022-04-22T19:39:32.254341+0000 mon.a (mon.0) 7414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:33.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:32 smithi115 bash[45495]: cluster 2022-04-22T19:39:31.922639+0000 mgr.y (mgr.34107) 4584 : cluster [DBG] pgmap v5946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:33.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:32 smithi115 bash[45495]: audit 2022-04-22T19:39:32.254341+0000 mon.a (mon.0) 7414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:33 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:33.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:35 smithi121 bash[36902]: cluster 2022-04-22T19:39:33.923838+0000 mgr.y (mgr.34107) 4585 : cluster [DBG] pgmap v5947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:35.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:35 smithi121 bash[36902]: audit 2022-04-22T19:39:34.309113+0000 mon.a (mon.0) 7415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:35 smithi115 bash[46934]: cluster 2022-04-22T19:39:33.923838+0000 mgr.y (mgr.34107) 4585 : cluster [DBG] pgmap v5947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:35 smithi115 bash[46934]: audit 2022-04-22T19:39:34.309113+0000 mon.a (mon.0) 7415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:35 smithi115 bash[45495]: cluster 2022-04-22T19:39:33.923838+0000 mgr.y (mgr.34107) 4585 : cluster [DBG] pgmap v5947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:35.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:35 smithi115 bash[45495]: audit 2022-04-22T19:39:34.309113+0000 mon.a (mon.0) 7415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:36 smithi121 bash[36902]: audit 2022-04-22T19:39:35.774774+0000 mon.a (mon.0) 7416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:36 smithi115 bash[46934]: audit 2022-04-22T19:39:35.774774+0000 mon.a (mon.0) 7416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:36.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:36 smithi115 bash[45495]: audit 2022-04-22T19:39:35.774774+0000 mon.a (mon.0) 7416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:37 smithi121 bash[36902]: cluster 2022-04-22T19:39:35.924747+0000 mgr.y (mgr.34107) 4586 : cluster [DBG] pgmap v5948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:37 smithi115 bash[45495]: cluster 2022-04-22T19:39:35.924747+0000 mgr.y (mgr.34107) 4586 : cluster [DBG] pgmap v5948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:37 smithi115 bash[46934]: cluster 2022-04-22T19:39:35.924747+0000 mgr.y (mgr.34107) 4586 : cluster [DBG] pgmap v5948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:38 smithi121 bash[36902]: audit 2022-04-22T19:39:37.256776+0000 mon.a (mon.0) 7417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:38 smithi115 bash[45495]: audit 2022-04-22T19:39:37.256776+0000 mon.a (mon.0) 7417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:38 smithi115 bash[46934]: audit 2022-04-22T19:39:37.256776+0000 mon.a (mon.0) 7417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:39 smithi121 bash[36902]: cluster 2022-04-22T19:39:37.925811+0000 mgr.y (mgr.34107) 4587 : cluster [DBG] pgmap v5949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:39 smithi115 bash[45495]: cluster 2022-04-22T19:39:37.925811+0000 mgr.y (mgr.34107) 4587 : cluster [DBG] pgmap v5949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:39 smithi115 bash[46934]: cluster 2022-04-22T19:39:37.925811+0000 mgr.y (mgr.34107) 4587 : cluster [DBG] pgmap v5949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:40 smithi121 bash[36902]: audit 2022-04-22T19:39:39.312858+0000 mon.a (mon.0) 7418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:40.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:40 smithi115 bash[46934]: audit 2022-04-22T19:39:39.312858+0000 mon.a (mon.0) 7418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:40 smithi115 bash[45495]: audit 2022-04-22T19:39:39.312858+0000 mon.a (mon.0) 7418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:41 smithi121 bash[36902]: cluster 2022-04-22T19:39:39.926920+0000 mgr.y (mgr.34107) 4588 : cluster [DBG] pgmap v5950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:41 smithi121 bash[36902]: audit 2022-04-22T19:39:40.777867+0000 mon.a (mon.0) 7419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:41.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:41 smithi115 bash[46934]: cluster 2022-04-22T19:39:39.926920+0000 mgr.y (mgr.34107) 4588 : cluster [DBG] pgmap v5950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:41.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:41 smithi115 bash[46934]: audit 2022-04-22T19:39:40.777867+0000 mon.a (mon.0) 7419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:41 smithi115 bash[45495]: cluster 2022-04-22T19:39:39.926920+0000 mgr.y (mgr.34107) 4588 : cluster [DBG] pgmap v5950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:41 smithi115 bash[45495]: audit 2022-04-22T19:39:40.777867+0000 mon.a (mon.0) 7419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:42.130Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:43 smithi121 bash[36902]: cluster 2022-04-22T19:39:41.928183+0000 mgr.y (mgr.34107) 4589 : cluster [DBG] pgmap v5951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:43 smithi121 bash[36902]: audit 2022-04-22T19:39:42.259900+0000 mon.a (mon.0) 7420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:43 smithi115 bash[45495]: cluster 2022-04-22T19:39:41.928183+0000 mgr.y (mgr.34107) 4589 : cluster [DBG] pgmap v5951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:43 smithi115 bash[45495]: audit 2022-04-22T19:39:42.259900+0000 mon.a (mon.0) 7420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:43 smithi115 bash[46934]: cluster 2022-04-22T19:39:41.928183+0000 mgr.y (mgr.34107) 4589 : cluster [DBG] pgmap v5951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:43 smithi115 bash[46934]: audit 2022-04-22T19:39:42.259900+0000 mon.a (mon.0) 7420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:44.001Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:45 smithi121 bash[36902]: cluster 2022-04-22T19:39:43.929404+0000 mgr.y (mgr.34107) 4590 : cluster [DBG] pgmap v5952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:45 smithi121 bash[36902]: audit 2022-04-22T19:39:44.316494+0000 mon.a (mon.0) 7421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:45 smithi115 bash[46934]: cluster 2022-04-22T19:39:43.929404+0000 mgr.y (mgr.34107) 4590 : cluster [DBG] pgmap v5952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:45 smithi115 bash[46934]: audit 2022-04-22T19:39:44.316494+0000 mon.a (mon.0) 7421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:45 smithi115 bash[45495]: cluster 2022-04-22T19:39:43.929404+0000 mgr.y (mgr.34107) 4590 : cluster [DBG] pgmap v5952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:45 smithi115 bash[45495]: audit 2022-04-22T19:39:44.316494+0000 mon.a (mon.0) 7421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:46 smithi121 bash[36902]: audit 2022-04-22T19:39:45.781148+0000 mon.a (mon.0) 7422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:46 smithi115 bash[45495]: audit 2022-04-22T19:39:45.781148+0000 mon.a (mon.0) 7422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:46 smithi115 bash[46934]: audit 2022-04-22T19:39:45.781148+0000 mon.a (mon.0) 7422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:47 smithi121 bash[36902]: cluster 2022-04-22T19:39:45.930495+0000 mgr.y (mgr.34107) 4591 : cluster [DBG] pgmap v5953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:47.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:47 smithi115 bash[45495]: cluster 2022-04-22T19:39:45.930495+0000 mgr.y (mgr.34107) 4591 : cluster [DBG] pgmap v5953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:47 smithi115 bash[46934]: cluster 2022-04-22T19:39:45.930495+0000 mgr.y (mgr.34107) 4591 : cluster [DBG] pgmap v5953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:48 smithi121 bash[36902]: audit 2022-04-22T19:39:47.262773+0000 mon.a (mon.0) 7423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:48.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:48 smithi115 bash[45495]: audit 2022-04-22T19:39:47.262773+0000 mon.a (mon.0) 7423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:48.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:48 smithi115 bash[46934]: audit 2022-04-22T19:39:47.262773+0000 mon.a (mon.0) 7423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:49 smithi121 bash[36902]: cluster 2022-04-22T19:39:47.931645+0000 mgr.y (mgr.34107) 4592 : cluster [DBG] pgmap v5954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:49 smithi115 bash[45495]: cluster 2022-04-22T19:39:47.931645+0000 mgr.y (mgr.34107) 4592 : cluster [DBG] pgmap v5954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:49.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:49 smithi115 bash[46934]: cluster 2022-04-22T19:39:47.931645+0000 mgr.y (mgr.34107) 4592 : cluster [DBG] pgmap v5954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:50 smithi121 bash[36902]: audit 2022-04-22T19:39:49.320171+0000 mon.a (mon.0) 7424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:50 smithi115 bash[45495]: audit 2022-04-22T19:39:49.320171+0000 mon.a (mon.0) 7424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:50 smithi115 bash[46934]: audit 2022-04-22T19:39:49.320171+0000 mon.a (mon.0) 7424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:51 smithi121 bash[36902]: cluster 2022-04-22T19:39:49.932716+0000 mgr.y (mgr.34107) 4593 : cluster [DBG] pgmap v5955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:51 smithi121 bash[36902]: audit 2022-04-22T19:39:50.784247+0000 mon.a (mon.0) 7425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:51 smithi115 bash[45495]: cluster 2022-04-22T19:39:49.932716+0000 mgr.y (mgr.34107) 4593 : cluster [DBG] pgmap v5955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:51 smithi115 bash[45495]: audit 2022-04-22T19:39:50.784247+0000 mon.a (mon.0) 7425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:51 smithi115 bash[46934]: cluster 2022-04-22T19:39:49.932716+0000 mgr.y (mgr.34107) 4593 : cluster [DBG] pgmap v5955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:51 smithi115 bash[46934]: audit 2022-04-22T19:39:50.784247+0000 mon.a (mon.0) 7425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:52.129Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:53 smithi121 bash[36902]: cluster 2022-04-22T19:39:51.933793+0000 mgr.y (mgr.34107) 4594 : cluster [DBG] pgmap v5956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:53 smithi121 bash[36902]: audit 2022-04-22T19:39:52.265591+0000 mon.a (mon.0) 7426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:53 smithi115 bash[45495]: cluster 2022-04-22T19:39:51.933793+0000 mgr.y (mgr.34107) 4594 : cluster [DBG] pgmap v5956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:53 smithi115 bash[45495]: audit 2022-04-22T19:39:52.265591+0000 mon.a (mon.0) 7426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:53 smithi115 bash[46934]: cluster 2022-04-22T19:39:51.933793+0000 mgr.y (mgr.34107) 4594 : cluster [DBG] pgmap v5956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:53 smithi115 bash[46934]: audit 2022-04-22T19:39:52.265591+0000 mon.a (mon.0) 7426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:39:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:39:54.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:39:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:55 smithi121 bash[36902]: cluster 2022-04-22T19:39:53.934984+0000 mgr.y (mgr.34107) 4595 : cluster [DBG] pgmap v5957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:55 smithi121 bash[36902]: audit 2022-04-22T19:39:54.323468+0000 mon.a (mon.0) 7427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:55.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:55 smithi115 bash[45495]: cluster 2022-04-22T19:39:53.934984+0000 mgr.y (mgr.34107) 4595 : cluster [DBG] pgmap v5957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:55.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:55 smithi115 bash[45495]: audit 2022-04-22T19:39:54.323468+0000 mon.a (mon.0) 7427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:55 smithi115 bash[46934]: cluster 2022-04-22T19:39:53.934984+0000 mgr.y (mgr.34107) 4595 : cluster [DBG] pgmap v5957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:55.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:55 smithi115 bash[46934]: audit 2022-04-22T19:39:54.323468+0000 mon.a (mon.0) 7427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:39:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:56 smithi121 bash[36902]: audit 2022-04-22T19:39:55.787554+0000 mon.a (mon.0) 7428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:56.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:56 smithi115 bash[45495]: audit 2022-04-22T19:39:55.787554+0000 mon.a (mon.0) 7428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:56.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:56 smithi115 bash[46934]: audit 2022-04-22T19:39:55.787554+0000 mon.a (mon.0) 7428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:39:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:57 smithi121 bash[36902]: cluster 2022-04-22T19:39:55.935991+0000 mgr.y (mgr.34107) 4596 : cluster [DBG] pgmap v5958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:57.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:57 smithi115 bash[45495]: cluster 2022-04-22T19:39:55.935991+0000 mgr.y (mgr.34107) 4596 : cluster [DBG] pgmap v5958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:57.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:57 smithi115 bash[46934]: cluster 2022-04-22T19:39:55.935991+0000 mgr.y (mgr.34107) 4596 : cluster [DBG] pgmap v5958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:58.230 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:39:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:58 smithi121 bash[36902]: audit 2022-04-22T19:39:57.268321+0000 mon.a (mon.0) 7429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:58 smithi115 bash[45495]: audit 2022-04-22T19:39:57.268321+0000 mon.a (mon.0) 7429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:58 smithi115 bash[46934]: audit 2022-04-22T19:39:57.268321+0000 mon.a (mon.0) 7429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:39:58.579 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:39:58.579 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:39:58.579 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 10m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 10m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 10m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 10m ago 2h 179M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:39:58.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 10m ago 2h 12.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 10m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 10m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 10m ago 2h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 10m ago 2h 634M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:39:58.581 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 10m ago 2h 67.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:39:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:39:58.976 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:39:58.976 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:39:58.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:39:58.977 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:39:58.978 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:39:58.978 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:39:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:39:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:59 smithi121 bash[36902]: cluster 2022-04-22T19:39:57.937079+0000 mgr.y (mgr.34107) 4597 : cluster [DBG] pgmap v5959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:59 smithi121 bash[36902]: audit 2022-04-22T19:39:58.201985+0000 mgr.y (mgr.34107) 4598 : audit [DBG] from='client.50128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:59 smithi121 bash[36902]: audit 2022-04-22T19:39:58.550926+0000 mgr.y (mgr.34107) 4599 : audit [DBG] from='client.50134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:39:59 smithi121 bash[36902]: audit 2022-04-22T19:39:58.965793+0000 mon.c (mon.1) 202 : audit [DBG] from='client.? 172.21.15.115:0/2032189570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:39:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[45495]: cluster 2022-04-22T19:39:57.937079+0000 mgr.y (mgr.34107) 4597 : cluster [DBG] pgmap v5959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[45495]: audit 2022-04-22T19:39:58.201985+0000 mgr.y (mgr.34107) 4598 : audit [DBG] from='client.50128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[45495]: audit 2022-04-22T19:39:58.550926+0000 mgr.y (mgr.34107) 4599 : audit [DBG] from='client.50134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[45495]: audit 2022-04-22T19:39:58.965793+0000 mon.c (mon.1) 202 : audit [DBG] from='client.? 172.21.15.115:0/2032189570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[46934]: cluster 2022-04-22T19:39:57.937079+0000 mgr.y (mgr.34107) 4597 : cluster [DBG] pgmap v5959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[46934]: audit 2022-04-22T19:39:58.201985+0000 mgr.y (mgr.34107) 4598 : audit [DBG] from='client.50128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[46934]: audit 2022-04-22T19:39:58.550926+0000 mgr.y (mgr.34107) 4599 : audit [DBG] from='client.50134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:39:59.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:39:59 smithi115 bash[46934]: audit 2022-04-22T19:39:58.965793+0000 mon.c (mon.1) 202 : audit [DBG] from='client.? 172.21.15.115:0/2032189570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:40:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:00 smithi121 bash[36902]: audit 2022-04-22T19:39:59.327096+0000 mon.a (mon.0) 7430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:00 smithi121 bash[36902]: cluster 2022-04-22T19:40:00.000164+0000 mon.a (mon.0) 7431 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:40:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:00 smithi115 bash[45495]: audit 2022-04-22T19:39:59.327096+0000 mon.a (mon.0) 7430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:00 smithi115 bash[45495]: cluster 2022-04-22T19:40:00.000164+0000 mon.a (mon.0) 7431 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:40:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:00 smithi115 bash[46934]: audit 2022-04-22T19:39:59.327096+0000 mon.a (mon.0) 7430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:00 smithi115 bash[46934]: cluster 2022-04-22T19:40:00.000164+0000 mon.a (mon.0) 7431 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:40:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:01 smithi121 bash[36902]: cluster 2022-04-22T19:39:59.938148+0000 mgr.y (mgr.34107) 4600 : cluster [DBG] pgmap v5960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:01 smithi121 bash[36902]: audit 2022-04-22T19:40:00.790732+0000 mon.a (mon.0) 7432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:01 smithi115 bash[45495]: cluster 2022-04-22T19:39:59.938148+0000 mgr.y (mgr.34107) 4600 : cluster [DBG] pgmap v5960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:01 smithi115 bash[45495]: audit 2022-04-22T19:40:00.790732+0000 mon.a (mon.0) 7432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:01 smithi115 bash[46934]: cluster 2022-04-22T19:39:59.938148+0000 mgr.y (mgr.34107) 4600 : cluster [DBG] pgmap v5960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:01 smithi115 bash[46934]: audit 2022-04-22T19:40:00.790732+0000 mon.a (mon.0) 7432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:02.131Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:03 smithi121 bash[36902]: cluster 2022-04-22T19:40:01.939204+0000 mgr.y (mgr.34107) 4601 : cluster [DBG] pgmap v5961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:03 smithi121 bash[36902]: audit 2022-04-22T19:40:02.270877+0000 mon.a (mon.0) 7433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:03 smithi115 bash[45495]: cluster 2022-04-22T19:40:01.939204+0000 mgr.y (mgr.34107) 4601 : cluster [DBG] pgmap v5961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:03 smithi115 bash[45495]: audit 2022-04-22T19:40:02.270877+0000 mon.a (mon.0) 7433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:03 smithi115 bash[46934]: cluster 2022-04-22T19:40:01.939204+0000 mgr.y (mgr.34107) 4601 : cluster [DBG] pgmap v5961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:03 smithi115 bash[46934]: audit 2022-04-22T19:40:02.270877+0000 mon.a (mon.0) 7433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:04.001Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:05 smithi121 bash[36902]: cluster 2022-04-22T19:40:03.940426+0000 mgr.y (mgr.34107) 4602 : cluster [DBG] pgmap v5962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:05 smithi121 bash[36902]: audit 2022-04-22T19:40:04.330885+0000 mon.a (mon.0) 7434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:05 smithi115 bash[45495]: cluster 2022-04-22T19:40:03.940426+0000 mgr.y (mgr.34107) 4602 : cluster [DBG] pgmap v5962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:05 smithi115 bash[45495]: audit 2022-04-22T19:40:04.330885+0000 mon.a (mon.0) 7434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:05 smithi115 bash[46934]: cluster 2022-04-22T19:40:03.940426+0000 mgr.y (mgr.34107) 4602 : cluster [DBG] pgmap v5962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:05 smithi115 bash[46934]: audit 2022-04-22T19:40:04.330885+0000 mon.a (mon.0) 7434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:06 smithi121 bash[36902]: audit 2022-04-22T19:40:05.793748+0000 mon.a (mon.0) 7435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:06.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:06 smithi115 bash[45495]: audit 2022-04-22T19:40:05.793748+0000 mon.a (mon.0) 7435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:06 smithi115 bash[46934]: audit 2022-04-22T19:40:05.793748+0000 mon.a (mon.0) 7435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:07.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[46934]: cluster 2022-04-22T19:40:05.941367+0000 mgr.y (mgr.34107) 4603 : cluster [DBG] pgmap v5963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:07.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[46934]: audit 2022-04-22T19:40:06.921825+0000 mon.a (mon.0) 7436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:40:07.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[46934]: audit 2022-04-22T19:40:06.924921+0000 mon.a (mon.0) 7437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:40:07.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[46934]: audit 2022-04-22T19:40:06.926611+0000 mon.a (mon.0) 7438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:40:07.510 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[45495]: cluster 2022-04-22T19:40:05.941367+0000 mgr.y (mgr.34107) 4603 : cluster [DBG] pgmap v5963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:07.511 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[45495]: audit 2022-04-22T19:40:06.921825+0000 mon.a (mon.0) 7436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:40:07.511 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[45495]: audit 2022-04-22T19:40:06.924921+0000 mon.a (mon.0) 7437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:40:07.511 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:07 smithi115 bash[45495]: audit 2022-04-22T19:40:06.926611+0000 mon.a (mon.0) 7438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:40:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:07 smithi121 bash[36902]: cluster 2022-04-22T19:40:05.941367+0000 mgr.y (mgr.34107) 4603 : cluster [DBG] pgmap v5963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:07 smithi121 bash[36902]: audit 2022-04-22T19:40:06.921825+0000 mon.a (mon.0) 7436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:40:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:07 smithi121 bash[36902]: audit 2022-04-22T19:40:06.924921+0000 mon.a (mon.0) 7437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:40:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:07 smithi121 bash[36902]: audit 2022-04-22T19:40:06.926611+0000 mon.a (mon.0) 7438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:40:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:08 smithi121 bash[36902]: audit 2022-04-22T19:40:07.273811+0000 mon.a (mon.0) 7439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:08 smithi121 bash[36902]: audit 2022-04-22T19:40:07.515968+0000 mon.a (mon.0) 7440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:08 smithi115 bash[45495]: audit 2022-04-22T19:40:07.273811+0000 mon.a (mon.0) 7439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:08 smithi115 bash[45495]: audit 2022-04-22T19:40:07.515968+0000 mon.a (mon.0) 7440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:08 smithi115 bash[46934]: audit 2022-04-22T19:40:07.273811+0000 mon.a (mon.0) 7439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:08 smithi115 bash[46934]: audit 2022-04-22T19:40:07.515968+0000 mon.a (mon.0) 7440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:09 smithi115 bash[45495]: cluster 2022-04-22T19:40:07.942532+0000 mgr.y (mgr.34107) 4604 : cluster [DBG] pgmap v5964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:09 smithi115 bash[45495]: audit 2022-04-22T19:40:09.334636+0000 mon.a (mon.0) 7441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:09 smithi115 bash[46934]: cluster 2022-04-22T19:40:07.942532+0000 mgr.y (mgr.34107) 4604 : cluster [DBG] pgmap v5964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:09 smithi115 bash[46934]: audit 2022-04-22T19:40:09.334636+0000 mon.a (mon.0) 7441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:09 smithi121 bash[36902]: cluster 2022-04-22T19:40:07.942532+0000 mgr.y (mgr.34107) 4604 : cluster [DBG] pgmap v5964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:09 smithi121 bash[36902]: audit 2022-04-22T19:40:09.334636+0000 mon.a (mon.0) 7441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:11.732 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:11 smithi121 bash[36902]: cluster 2022-04-22T19:40:09.943665+0000 mgr.y (mgr.34107) 4605 : cluster [DBG] pgmap v5965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:11.732 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:11 smithi121 bash[36902]: audit 2022-04-22T19:40:10.796950+0000 mon.a (mon.0) 7442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:11.733 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:11 smithi121 bash[36902]: audit 2022-04-22T19:40:11.385794+0000 mon.a (mon.0) 7443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[45495]: cluster 2022-04-22T19:40:09.943665+0000 mgr.y (mgr.34107) 4605 : cluster [DBG] pgmap v5965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[45495]: audit 2022-04-22T19:40:10.796950+0000 mon.a (mon.0) 7442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:11.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[45495]: audit 2022-04-22T19:40:11.385794+0000 mon.a (mon.0) 7443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:11.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[46934]: cluster 2022-04-22T19:40:09.943665+0000 mgr.y (mgr.34107) 4605 : cluster [DBG] pgmap v5965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:11.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[46934]: audit 2022-04-22T19:40:10.796950+0000 mon.a (mon.0) 7442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:11.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:11 smithi115 bash[46934]: audit 2022-04-22T19:40:11.385794+0000 mon.a (mon.0) 7443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:12.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:12.130Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:12 smithi121 bash[36902]: audit 2022-04-22T19:40:11.735685+0000 mon.a (mon.0) 7444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:12 smithi121 bash[36902]: cluster 2022-04-22T19:40:11.944663+0000 mgr.y (mgr.34107) 4606 : cluster [DBG] pgmap v5966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:12 smithi121 bash[36902]: audit 2022-04-22T19:40:12.276357+0000 mon.a (mon.0) 7445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:13.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[45495]: audit 2022-04-22T19:40:11.735685+0000 mon.a (mon.0) 7444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:13.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[45495]: cluster 2022-04-22T19:40:11.944663+0000 mgr.y (mgr.34107) 4606 : cluster [DBG] pgmap v5966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:13.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[45495]: audit 2022-04-22T19:40:12.276357+0000 mon.a (mon.0) 7445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:13.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[46934]: audit 2022-04-22T19:40:11.735685+0000 mon.a (mon.0) 7444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:40:13.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[46934]: cluster 2022-04-22T19:40:11.944663+0000 mgr.y (mgr.34107) 4606 : cluster [DBG] pgmap v5966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:13.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:12 smithi115 bash[46934]: audit 2022-04-22T19:40:12.276357+0000 mon.a (mon.0) 7445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:14.002Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:15 smithi121 bash[36902]: cluster 2022-04-22T19:40:13.945810+0000 mgr.y (mgr.34107) 4607 : cluster [DBG] pgmap v5967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:15 smithi121 bash[36902]: audit 2022-04-22T19:40:14.338242+0000 mon.a (mon.0) 7446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:15 smithi121 bash[36902]: audit 2022-04-22T19:40:14.802556+0000 mon.a (mon.0) 7447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:40:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:15 smithi121 bash[36902]: audit 2022-04-22T19:40:14.809576+0000 mon.a (mon.0) 7448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:40:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[45495]: cluster 2022-04-22T19:40:13.945810+0000 mgr.y (mgr.34107) 4607 : cluster [DBG] pgmap v5967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[45495]: audit 2022-04-22T19:40:14.338242+0000 mon.a (mon.0) 7446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[45495]: audit 2022-04-22T19:40:14.802556+0000 mon.a (mon.0) 7447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:40:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[45495]: audit 2022-04-22T19:40:14.809576+0000 mon.a (mon.0) 7448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:40:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[46934]: cluster 2022-04-22T19:40:13.945810+0000 mgr.y (mgr.34107) 4607 : cluster [DBG] pgmap v5967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[46934]: audit 2022-04-22T19:40:14.338242+0000 mon.a (mon.0) 7446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[46934]: audit 2022-04-22T19:40:14.802556+0000 mon.a (mon.0) 7447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:40:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:15 smithi115 bash[46934]: audit 2022-04-22T19:40:14.809576+0000 mon.a (mon.0) 7448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:40:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:16 smithi121 bash[36902]: audit 2022-04-22T19:40:15.800325+0000 mon.a (mon.0) 7449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:16 smithi115 bash[46934]: audit 2022-04-22T19:40:15.800325+0000 mon.a (mon.0) 7449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:16.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:16 smithi115 bash[45495]: audit 2022-04-22T19:40:15.800325+0000 mon.a (mon.0) 7449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:17 smithi121 bash[36902]: cluster 2022-04-22T19:40:15.946770+0000 mgr.y (mgr.34107) 4608 : cluster [DBG] pgmap v5968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:17 smithi115 bash[45495]: cluster 2022-04-22T19:40:15.946770+0000 mgr.y (mgr.34107) 4608 : cluster [DBG] pgmap v5968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:17 smithi115 bash[46934]: cluster 2022-04-22T19:40:15.946770+0000 mgr.y (mgr.34107) 4608 : cluster [DBG] pgmap v5968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:18 smithi121 bash[36902]: audit 2022-04-22T19:40:17.279528+0000 mon.a (mon.0) 7450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:18 smithi115 bash[45495]: audit 2022-04-22T19:40:17.279528+0000 mon.a (mon.0) 7450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:18 smithi115 bash[46934]: audit 2022-04-22T19:40:17.279528+0000 mon.a (mon.0) 7450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:19 smithi121 bash[36902]: cluster 2022-04-22T19:40:17.947863+0000 mgr.y (mgr.34107) 4609 : cluster [DBG] pgmap v5969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:19 smithi115 bash[45495]: cluster 2022-04-22T19:40:17.947863+0000 mgr.y (mgr.34107) 4609 : cluster [DBG] pgmap v5969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:19 smithi115 bash[46934]: cluster 2022-04-22T19:40:17.947863+0000 mgr.y (mgr.34107) 4609 : cluster [DBG] pgmap v5969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:20 smithi121 bash[36902]: audit 2022-04-22T19:40:19.342012+0000 mon.a (mon.0) 7451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:20 smithi115 bash[45495]: audit 2022-04-22T19:40:19.342012+0000 mon.a (mon.0) 7451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:20.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:20 smithi115 bash[46934]: audit 2022-04-22T19:40:19.342012+0000 mon.a (mon.0) 7451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:21 smithi121 bash[36902]: cluster 2022-04-22T19:40:19.948931+0000 mgr.y (mgr.34107) 4610 : cluster [DBG] pgmap v5970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:21 smithi121 bash[36902]: audit 2022-04-22T19:40:20.803624+0000 mon.a (mon.0) 7452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:21.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:21 smithi115 bash[45495]: cluster 2022-04-22T19:40:19.948931+0000 mgr.y (mgr.34107) 4610 : cluster [DBG] pgmap v5970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:21 smithi115 bash[45495]: audit 2022-04-22T19:40:20.803624+0000 mon.a (mon.0) 7452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:21 smithi115 bash[46934]: cluster 2022-04-22T19:40:19.948931+0000 mgr.y (mgr.34107) 4610 : cluster [DBG] pgmap v5970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:21 smithi115 bash[46934]: audit 2022-04-22T19:40:20.803624+0000 mon.a (mon.0) 7452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:22.133Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:23 smithi121 bash[36902]: cluster 2022-04-22T19:40:21.949978+0000 mgr.y (mgr.34107) 4611 : cluster [DBG] pgmap v5971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:23 smithi121 bash[36902]: audit 2022-04-22T19:40:22.282545+0000 mon.a (mon.0) 7453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:23 smithi115 bash[45495]: cluster 2022-04-22T19:40:21.949978+0000 mgr.y (mgr.34107) 4611 : cluster [DBG] pgmap v5971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:23 smithi115 bash[45495]: audit 2022-04-22T19:40:22.282545+0000 mon.a (mon.0) 7453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:23 smithi115 bash[46934]: cluster 2022-04-22T19:40:21.949978+0000 mgr.y (mgr.34107) 4611 : cluster [DBG] pgmap v5971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:23 smithi115 bash[46934]: audit 2022-04-22T19:40:22.282545+0000 mon.a (mon.0) 7453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:24.003Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:25 smithi121 bash[36902]: cluster 2022-04-22T19:40:23.951197+0000 mgr.y (mgr.34107) 4612 : cluster [DBG] pgmap v5972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:25.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:25 smithi121 bash[36902]: audit 2022-04-22T19:40:24.345838+0000 mon.a (mon.0) 7454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:25 smithi115 bash[45495]: cluster 2022-04-22T19:40:23.951197+0000 mgr.y (mgr.34107) 4612 : cluster [DBG] pgmap v5972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:25.561 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:25 smithi115 bash[45495]: audit 2022-04-22T19:40:24.345838+0000 mon.a (mon.0) 7454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:25.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:25 smithi115 bash[46934]: cluster 2022-04-22T19:40:23.951197+0000 mgr.y (mgr.34107) 4612 : cluster [DBG] pgmap v5972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:25 smithi115 bash[46934]: audit 2022-04-22T19:40:24.345838+0000 mon.a (mon.0) 7454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:26 smithi121 bash[36902]: audit 2022-04-22T19:40:25.806606+0000 mon.a (mon.0) 7455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:26 smithi115 bash[45495]: audit 2022-04-22T19:40:25.806606+0000 mon.a (mon.0) 7455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:26.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:26 smithi115 bash[46934]: audit 2022-04-22T19:40:25.806606+0000 mon.a (mon.0) 7455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:27 smithi121 bash[36902]: cluster 2022-04-22T19:40:25.952190+0000 mgr.y (mgr.34107) 4613 : cluster [DBG] pgmap v5973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:27 smithi115 bash[45495]: cluster 2022-04-22T19:40:25.952190+0000 mgr.y (mgr.34107) 4613 : cluster [DBG] pgmap v5973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:27 smithi115 bash[46934]: cluster 2022-04-22T19:40:25.952190+0000 mgr.y (mgr.34107) 4613 : cluster [DBG] pgmap v5973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:28 smithi121 bash[36902]: audit 2022-04-22T19:40:27.285650+0000 mon.a (mon.0) 7456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:28 smithi115 bash[45495]: audit 2022-04-22T19:40:27.285650+0000 mon.a (mon.0) 7456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:28 smithi115 bash[46934]: audit 2022-04-22T19:40:27.285650+0000 mon.a (mon.0) 7456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:29.350 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:40:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:29 smithi121 bash[36902]: cluster 2022-04-22T19:40:27.953326+0000 mgr.y (mgr.34107) 4614 : cluster [DBG] pgmap v5974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:29 smithi115 bash[46934]: cluster 2022-04-22T19:40:27.953326+0000 mgr.y (mgr.34107) 4614 : cluster [DBG] pgmap v5974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:29 smithi115 bash[45495]: cluster 2022-04-22T19:40:27.953326+0000 mgr.y (mgr.34107) 4614 : cluster [DBG] pgmap v5974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:29.694 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:40:29.694 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:40:29.694 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 18s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:40:29.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:40:29.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 18s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:40:29.695 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 18s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:40:29.695 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:40:29.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:40:29.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 18s ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:40:29.696 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 18s ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:40:29.697 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:40:29.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 18s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:40:29.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 18s ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:40:29.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 18s ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:40:29.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 18s ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:40:29.698 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 18s ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:40:30.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:40:30.093 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:40:30.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:40:30.094 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:40:30.094 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:40:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:30 smithi121 bash[36902]: audit 2022-04-22T19:40:29.324811+0000 mgr.y (mgr.34107) 4615 : audit [DBG] from='client.40212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:30 smithi121 bash[36902]: audit 2022-04-22T19:40:29.349369+0000 mon.a (mon.0) 7457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:30 smithi121 bash[36902]: audit 2022-04-22T19:40:30.081531+0000 mon.c (mon.1) 203 : audit [DBG] from='client.? 172.21.15.115:0/2305664208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:40:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[45495]: audit 2022-04-22T19:40:29.324811+0000 mgr.y (mgr.34107) 4615 : audit [DBG] from='client.40212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:30.628 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[45495]: audit 2022-04-22T19:40:29.349369+0000 mon.a (mon.0) 7457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:30.639 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[45495]: audit 2022-04-22T19:40:30.081531+0000 mon.c (mon.1) 203 : audit [DBG] from='client.? 172.21.15.115:0/2305664208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:40:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[46934]: audit 2022-04-22T19:40:29.324811+0000 mgr.y (mgr.34107) 4615 : audit [DBG] from='client.40212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[46934]: audit 2022-04-22T19:40:29.349369+0000 mon.a (mon.0) 7457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:30 smithi115 bash[46934]: audit 2022-04-22T19:40:30.081531+0000 mon.c (mon.1) 203 : audit [DBG] from='client.? 172.21.15.115:0/2305664208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:40:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:31 smithi121 bash[36902]: audit 2022-04-22T19:40:29.664405+0000 mgr.y (mgr.34107) 4616 : audit [DBG] from='client.40218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:31 smithi121 bash[36902]: cluster 2022-04-22T19:40:29.954333+0000 mgr.y (mgr.34107) 4617 : cluster [DBG] pgmap v5975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:31 smithi121 bash[36902]: audit 2022-04-22T19:40:30.809852+0000 mon.a (mon.0) 7458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[45495]: audit 2022-04-22T19:40:29.664405+0000 mgr.y (mgr.34107) 4616 : audit [DBG] from='client.40218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[45495]: cluster 2022-04-22T19:40:29.954333+0000 mgr.y (mgr.34107) 4617 : cluster [DBG] pgmap v5975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:31.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[45495]: audit 2022-04-22T19:40:30.809852+0000 mon.a (mon.0) 7458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[46934]: audit 2022-04-22T19:40:29.664405+0000 mgr.y (mgr.34107) 4616 : audit [DBG] from='client.40218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:40:31.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[46934]: cluster 2022-04-22T19:40:29.954333+0000 mgr.y (mgr.34107) 4617 : cluster [DBG] pgmap v5975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:31.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:31 smithi115 bash[46934]: audit 2022-04-22T19:40:30.809852+0000 mon.a (mon.0) 7458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:32.133Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:33 smithi121 bash[36902]: cluster 2022-04-22T19:40:31.955413+0000 mgr.y (mgr.34107) 4618 : cluster [DBG] pgmap v5976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:33 smithi121 bash[36902]: audit 2022-04-22T19:40:32.288608+0000 mon.a (mon.0) 7459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:33 smithi115 bash[45495]: cluster 2022-04-22T19:40:31.955413+0000 mgr.y (mgr.34107) 4618 : cluster [DBG] pgmap v5976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:33 smithi115 bash[45495]: audit 2022-04-22T19:40:32.288608+0000 mon.a (mon.0) 7459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:33 smithi115 bash[46934]: cluster 2022-04-22T19:40:31.955413+0000 mgr.y (mgr.34107) 4618 : cluster [DBG] pgmap v5976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:33 smithi115 bash[46934]: audit 2022-04-22T19:40:32.288608+0000 mon.a (mon.0) 7459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:34.004Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:35 smithi121 bash[36902]: cluster 2022-04-22T19:40:33.956543+0000 mgr.y (mgr.34107) 4619 : cluster [DBG] pgmap v5977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:35 smithi121 bash[36902]: audit 2022-04-22T19:40:34.352935+0000 mon.a (mon.0) 7460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:35 smithi115 bash[45495]: cluster 2022-04-22T19:40:33.956543+0000 mgr.y (mgr.34107) 4619 : cluster [DBG] pgmap v5977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:35 smithi115 bash[45495]: audit 2022-04-22T19:40:34.352935+0000 mon.a (mon.0) 7460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:35 smithi115 bash[46934]: cluster 2022-04-22T19:40:33.956543+0000 mgr.y (mgr.34107) 4619 : cluster [DBG] pgmap v5977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:35 smithi115 bash[46934]: audit 2022-04-22T19:40:34.352935+0000 mon.a (mon.0) 7460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:36 smithi121 bash[36902]: audit 2022-04-22T19:40:35.813134+0000 mon.a (mon.0) 7461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:36.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:36 smithi115 bash[45495]: audit 2022-04-22T19:40:35.813134+0000 mon.a (mon.0) 7461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:36.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:36 smithi115 bash[46934]: audit 2022-04-22T19:40:35.813134+0000 mon.a (mon.0) 7461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:37 smithi121 bash[36902]: cluster 2022-04-22T19:40:35.957452+0000 mgr.y (mgr.34107) 4620 : cluster [DBG] pgmap v5978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:37 smithi115 bash[45495]: cluster 2022-04-22T19:40:35.957452+0000 mgr.y (mgr.34107) 4620 : cluster [DBG] pgmap v5978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:37.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:37 smithi115 bash[46934]: cluster 2022-04-22T19:40:35.957452+0000 mgr.y (mgr.34107) 4620 : cluster [DBG] pgmap v5978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:38 smithi121 bash[36902]: audit 2022-04-22T19:40:37.291788+0000 mon.a (mon.0) 7462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:38 smithi115 bash[45495]: audit 2022-04-22T19:40:37.291788+0000 mon.a (mon.0) 7462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:38.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:38 smithi115 bash[46934]: audit 2022-04-22T19:40:37.291788+0000 mon.a (mon.0) 7462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:39 smithi121 bash[36902]: cluster 2022-04-22T19:40:37.958638+0000 mgr.y (mgr.34107) 4621 : cluster [DBG] pgmap v5979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:39 smithi115 bash[45495]: cluster 2022-04-22T19:40:37.958638+0000 mgr.y (mgr.34107) 4621 : cluster [DBG] pgmap v5979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:39.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:39 smithi115 bash[46934]: cluster 2022-04-22T19:40:37.958638+0000 mgr.y (mgr.34107) 4621 : cluster [DBG] pgmap v5979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:40 smithi121 bash[36902]: audit 2022-04-22T19:40:39.356677+0000 mon.a (mon.0) 7463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:40 smithi115 bash[45495]: audit 2022-04-22T19:40:39.356677+0000 mon.a (mon.0) 7463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:40 smithi115 bash[46934]: audit 2022-04-22T19:40:39.356677+0000 mon.a (mon.0) 7463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:41 smithi121 bash[36902]: cluster 2022-04-22T19:40:39.959797+0000 mgr.y (mgr.34107) 4622 : cluster [DBG] pgmap v5980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:41 smithi121 bash[36902]: audit 2022-04-22T19:40:40.816107+0000 mon.a (mon.0) 7464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:41 smithi115 bash[45495]: cluster 2022-04-22T19:40:39.959797+0000 mgr.y (mgr.34107) 4622 : cluster [DBG] pgmap v5980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:41.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:41 smithi115 bash[45495]: audit 2022-04-22T19:40:40.816107+0000 mon.a (mon.0) 7464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:41.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:41 smithi115 bash[46934]: cluster 2022-04-22T19:40:39.959797+0000 mgr.y (mgr.34107) 4622 : cluster [DBG] pgmap v5980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:41 smithi115 bash[46934]: audit 2022-04-22T19:40:40.816107+0000 mon.a (mon.0) 7464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:42.133Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:43 smithi121 bash[36902]: cluster 2022-04-22T19:40:41.960805+0000 mgr.y (mgr.34107) 4623 : cluster [DBG] pgmap v5981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:43 smithi121 bash[36902]: audit 2022-04-22T19:40:42.294420+0000 mon.a (mon.0) 7465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:43 smithi115 bash[45495]: cluster 2022-04-22T19:40:41.960805+0000 mgr.y (mgr.34107) 4623 : cluster [DBG] pgmap v5981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:43.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:43 smithi115 bash[45495]: audit 2022-04-22T19:40:42.294420+0000 mon.a (mon.0) 7465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:43 smithi115 bash[46934]: cluster 2022-04-22T19:40:41.960805+0000 mgr.y (mgr.34107) 4623 : cluster [DBG] pgmap v5981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:43 smithi115 bash[46934]: audit 2022-04-22T19:40:42.294420+0000 mon.a (mon.0) 7465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:44.005Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:45 smithi121 bash[36902]: cluster 2022-04-22T19:40:43.961967+0000 mgr.y (mgr.34107) 4624 : cluster [DBG] pgmap v5982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:45 smithi121 bash[36902]: audit 2022-04-22T19:40:44.360117+0000 mon.a (mon.0) 7466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:45.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:45 smithi115 bash[46934]: cluster 2022-04-22T19:40:43.961967+0000 mgr.y (mgr.34107) 4624 : cluster [DBG] pgmap v5982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:45 smithi115 bash[46934]: audit 2022-04-22T19:40:44.360117+0000 mon.a (mon.0) 7466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:45 smithi115 bash[45495]: cluster 2022-04-22T19:40:43.961967+0000 mgr.y (mgr.34107) 4624 : cluster [DBG] pgmap v5982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:45 smithi115 bash[45495]: audit 2022-04-22T19:40:44.360117+0000 mon.a (mon.0) 7466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:46 smithi121 bash[36902]: audit 2022-04-22T19:40:45.819352+0000 mon.a (mon.0) 7467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:46 smithi115 bash[45495]: audit 2022-04-22T19:40:45.819352+0000 mon.a (mon.0) 7467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:46.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:46 smithi115 bash[46934]: audit 2022-04-22T19:40:45.819352+0000 mon.a (mon.0) 7467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:47 smithi121 bash[36902]: cluster 2022-04-22T19:40:45.962964+0000 mgr.y (mgr.34107) 4625 : cluster [DBG] pgmap v5983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:47.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:47 smithi115 bash[45495]: cluster 2022-04-22T19:40:45.962964+0000 mgr.y (mgr.34107) 4625 : cluster [DBG] pgmap v5983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:47 smithi115 bash[46934]: cluster 2022-04-22T19:40:45.962964+0000 mgr.y (mgr.34107) 4625 : cluster [DBG] pgmap v5983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:48 smithi121 bash[36902]: audit 2022-04-22T19:40:47.296849+0000 mon.a (mon.0) 7468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:48.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:48 smithi115 bash[45495]: audit 2022-04-22T19:40:47.296849+0000 mon.a (mon.0) 7468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:48 smithi115 bash[46934]: audit 2022-04-22T19:40:47.296849+0000 mon.a (mon.0) 7468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:49 smithi121 bash[36902]: cluster 2022-04-22T19:40:47.964061+0000 mgr.y (mgr.34107) 4626 : cluster [DBG] pgmap v5984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:49.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:49 smithi115 bash[45495]: cluster 2022-04-22T19:40:47.964061+0000 mgr.y (mgr.34107) 4626 : cluster [DBG] pgmap v5984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:49 smithi115 bash[46934]: cluster 2022-04-22T19:40:47.964061+0000 mgr.y (mgr.34107) 4626 : cluster [DBG] pgmap v5984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:50.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:50 smithi115 bash[46934]: audit 2022-04-22T19:40:49.363992+0000 mon.a (mon.0) 7469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:50 smithi115 bash[45495]: audit 2022-04-22T19:40:49.363992+0000 mon.a (mon.0) 7469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:50 smithi121 bash[36902]: audit 2022-04-22T19:40:49.363992+0000 mon.a (mon.0) 7469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:51.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:51 smithi115 bash[46934]: cluster 2022-04-22T19:40:49.965142+0000 mgr.y (mgr.34107) 4627 : cluster [DBG] pgmap v5985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:51.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:51 smithi115 bash[46934]: audit 2022-04-22T19:40:50.822560+0000 mon.a (mon.0) 7470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:51 smithi115 bash[45495]: cluster 2022-04-22T19:40:49.965142+0000 mgr.y (mgr.34107) 4627 : cluster [DBG] pgmap v5985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:51.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:51 smithi115 bash[45495]: audit 2022-04-22T19:40:50.822560+0000 mon.a (mon.0) 7470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:51 smithi121 bash[36902]: cluster 2022-04-22T19:40:49.965142+0000 mgr.y (mgr.34107) 4627 : cluster [DBG] pgmap v5985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:51 smithi121 bash[36902]: audit 2022-04-22T19:40:50.822560+0000 mon.a (mon.0) 7470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:52.135Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:53 smithi115 bash[45495]: cluster 2022-04-22T19:40:51.966238+0000 mgr.y (mgr.34107) 4628 : cluster [DBG] pgmap v5986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:53 smithi115 bash[45495]: audit 2022-04-22T19:40:52.299552+0000 mon.a (mon.0) 7471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:53 smithi115 bash[46934]: cluster 2022-04-22T19:40:51.966238+0000 mgr.y (mgr.34107) 4628 : cluster [DBG] pgmap v5986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:53 smithi115 bash[46934]: audit 2022-04-22T19:40:52.299552+0000 mon.a (mon.0) 7471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:53 smithi121 bash[36902]: cluster 2022-04-22T19:40:51.966238+0000 mgr.y (mgr.34107) 4628 : cluster [DBG] pgmap v5986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:53 smithi121 bash[36902]: audit 2022-04-22T19:40:52.299552+0000 mon.a (mon.0) 7471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:40:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:40:54.004Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:40:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:55 smithi115 bash[45495]: cluster 2022-04-22T19:40:53.967342+0000 mgr.y (mgr.34107) 4629 : cluster [DBG] pgmap v5987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:55.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:55 smithi115 bash[45495]: audit 2022-04-22T19:40:54.367742+0000 mon.a (mon.0) 7472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:55 smithi115 bash[46934]: cluster 2022-04-22T19:40:53.967342+0000 mgr.y (mgr.34107) 4629 : cluster [DBG] pgmap v5987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:55 smithi115 bash[46934]: audit 2022-04-22T19:40:54.367742+0000 mon.a (mon.0) 7472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:55 smithi121 bash[36902]: cluster 2022-04-22T19:40:53.967342+0000 mgr.y (mgr.34107) 4629 : cluster [DBG] pgmap v5987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:55 smithi121 bash[36902]: audit 2022-04-22T19:40:54.367742+0000 mon.a (mon.0) 7472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:40:56.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:56 smithi115 bash[45495]: audit 2022-04-22T19:40:55.825693+0000 mon.a (mon.0) 7473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:56.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:56 smithi115 bash[46934]: audit 2022-04-22T19:40:55.825693+0000 mon.a (mon.0) 7473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:56 smithi121 bash[36902]: audit 2022-04-22T19:40:55.825693+0000 mon.a (mon.0) 7473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:40:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:57 smithi115 bash[46934]: cluster 2022-04-22T19:40:55.968319+0000 mgr.y (mgr.34107) 4630 : cluster [DBG] pgmap v5988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:57.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:57 smithi115 bash[45495]: cluster 2022-04-22T19:40:55.968319+0000 mgr.y (mgr.34107) 4630 : cluster [DBG] pgmap v5988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:57 smithi121 bash[36902]: cluster 2022-04-22T19:40:55.968319+0000 mgr.y (mgr.34107) 4630 : cluster [DBG] pgmap v5988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:58 smithi115 bash[45495]: audit 2022-04-22T19:40:57.302316+0000 mon.a (mon.0) 7474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:58 smithi115 bash[46934]: audit 2022-04-22T19:40:57.302316+0000 mon.a (mon.0) 7474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:58 smithi121 bash[36902]: audit 2022-04-22T19:40:57.302316+0000 mon.a (mon.0) 7474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:40:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:40:59 smithi121 bash[36902]: cluster 2022-04-22T19:40:57.969364+0000 mgr.y (mgr.34107) 4631 : cluster [DBG] pgmap v5989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:40:59 smithi115 bash[46934]: cluster 2022-04-22T19:40:57.969364+0000 mgr.y (mgr.34107) 4631 : cluster [DBG] pgmap v5989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:40:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:40:59 smithi115 bash[45495]: cluster 2022-04-22T19:40:57.969364+0000 mgr.y (mgr.34107) 4631 : cluster [DBG] pgmap v5989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:41:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:00 smithi121 bash[36902]: audit 2022-04-22T19:40:59.371418+0000 mon.a (mon.0) 7475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:00 smithi115 bash[46934]: audit 2022-04-22T19:40:59.371418+0000 mon.a (mon.0) 7475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:00 smithi115 bash[45495]: audit 2022-04-22T19:40:59.371418+0000 mon.a (mon.0) 7475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:00.821 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:41:00.821 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 10m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:41:00.821 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 49s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 10m ago 2h 23.4M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 49s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 49s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 10m ago 2h 480M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 10m ago 2h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 49s ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 10m ago 2h 181M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 10m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:41:00.822 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 49s ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 10m ago 2h 701M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 10m ago 2h 780M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 10m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 10m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 49s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 49s ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 49s ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:41:00.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 49s ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:41:00.824 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 49s ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:41:01.224 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:41:01.225 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:41:01.226 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:41:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:01 smithi121 bash[36902]: cluster 2022-04-22T19:40:59.970406+0000 mgr.y (mgr.34107) 4632 : cluster [DBG] pgmap v5990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:01 smithi121 bash[36902]: audit 2022-04-22T19:41:00.444812+0000 mgr.y (mgr.34107) 4633 : audit [DBG] from='client.50164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:01 smithi121 bash[36902]: audit 2022-04-22T19:41:00.828933+0000 mon.a (mon.0) 7476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:01 smithi121 bash[36902]: audit 2022-04-22T19:41:01.214433+0000 mon.a (mon.0) 7477 : audit [DBG] from='client.? 172.21.15.115:0/229973707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[45495]: cluster 2022-04-22T19:40:59.970406+0000 mgr.y (mgr.34107) 4632 : cluster [DBG] pgmap v5990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[45495]: audit 2022-04-22T19:41:00.444812+0000 mgr.y (mgr.34107) 4633 : audit [DBG] from='client.50164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[45495]: audit 2022-04-22T19:41:00.828933+0000 mon.a (mon.0) 7476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[45495]: audit 2022-04-22T19:41:01.214433+0000 mon.a (mon.0) 7477 : audit [DBG] from='client.? 172.21.15.115:0/229973707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[46934]: cluster 2022-04-22T19:40:59.970406+0000 mgr.y (mgr.34107) 4632 : cluster [DBG] pgmap v5990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:01.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[46934]: audit 2022-04-22T19:41:00.444812+0000 mgr.y (mgr.34107) 4633 : audit [DBG] from='client.50164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:01.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[46934]: audit 2022-04-22T19:41:00.828933+0000 mon.a (mon.0) 7476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:01.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:01 smithi115 bash[46934]: audit 2022-04-22T19:41:01.214433+0000 mon.a (mon.0) 7477 : audit [DBG] from='client.? 172.21.15.115:0/229973707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:02 smithi115 bash[45495]: audit 2022-04-22T19:41:00.790463+0000 mgr.y (mgr.34107) 4634 : audit [DBG] from='client.40236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:02.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:02.135Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:02.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:02 smithi115 bash[46934]: audit 2022-04-22T19:41:00.790463+0000 mgr.y (mgr.34107) 4634 : audit [DBG] from='client.40236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:02 smithi121 bash[36902]: audit 2022-04-22T19:41:00.790463+0000 mgr.y (mgr.34107) 4634 : audit [DBG] from='client.40236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:03 smithi121 bash[36902]: cluster 2022-04-22T19:41:01.971392+0000 mgr.y (mgr.34107) 4635 : cluster [DBG] pgmap v5991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:03 smithi121 bash[36902]: audit 2022-04-22T19:41:02.306800+0000 mon.a (mon.0) 7478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:03 smithi115 bash[45495]: cluster 2022-04-22T19:41:01.971392+0000 mgr.y (mgr.34107) 4635 : cluster [DBG] pgmap v5991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:03 smithi115 bash[45495]: audit 2022-04-22T19:41:02.306800+0000 mon.a (mon.0) 7478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:03 smithi115 bash[46934]: cluster 2022-04-22T19:41:01.971392+0000 mgr.y (mgr.34107) 4635 : cluster [DBG] pgmap v5991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:03 smithi115 bash[46934]: audit 2022-04-22T19:41:02.306800+0000 mon.a (mon.0) 7478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:04.005Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:05 smithi121 bash[36902]: cluster 2022-04-22T19:41:03.972446+0000 mgr.y (mgr.34107) 4636 : cluster [DBG] pgmap v5992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:05 smithi121 bash[36902]: audit 2022-04-22T19:41:04.375158+0000 mon.a (mon.0) 7479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:05 smithi115 bash[45495]: cluster 2022-04-22T19:41:03.972446+0000 mgr.y (mgr.34107) 4636 : cluster [DBG] pgmap v5992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:05 smithi115 bash[45495]: audit 2022-04-22T19:41:04.375158+0000 mon.a (mon.0) 7479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:05 smithi115 bash[46934]: cluster 2022-04-22T19:41:03.972446+0000 mgr.y (mgr.34107) 4636 : cluster [DBG] pgmap v5992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:05 smithi115 bash[46934]: audit 2022-04-22T19:41:04.375158+0000 mon.a (mon.0) 7479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:06 smithi121 bash[36902]: audit 2022-04-22T19:41:05.832032+0000 mon.a (mon.0) 7480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:06 smithi115 bash[45495]: audit 2022-04-22T19:41:05.832032+0000 mon.a (mon.0) 7480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:06 smithi115 bash[46934]: audit 2022-04-22T19:41:05.832032+0000 mon.a (mon.0) 7480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:07 smithi121 bash[36902]: cluster 2022-04-22T19:41:05.973347+0000 mgr.y (mgr.34107) 4637 : cluster [DBG] pgmap v5993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:07.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:07 smithi121 bash[36902]: audit 2022-04-22T19:41:07.309439+0000 mon.a (mon.0) 7481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:07 smithi115 bash[45495]: cluster 2022-04-22T19:41:05.973347+0000 mgr.y (mgr.34107) 4637 : cluster [DBG] pgmap v5993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:07 smithi115 bash[45495]: audit 2022-04-22T19:41:07.309439+0000 mon.a (mon.0) 7481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:07 smithi115 bash[46934]: cluster 2022-04-22T19:41:05.973347+0000 mgr.y (mgr.34107) 4637 : cluster [DBG] pgmap v5993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:07 smithi115 bash[46934]: audit 2022-04-22T19:41:07.309439+0000 mon.a (mon.0) 7481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:09 smithi121 bash[36902]: cluster 2022-04-22T19:41:07.974408+0000 mgr.y (mgr.34107) 4638 : cluster [DBG] pgmap v5994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:09 smithi115 bash[45495]: cluster 2022-04-22T19:41:07.974408+0000 mgr.y (mgr.34107) 4638 : cluster [DBG] pgmap v5994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:09 smithi115 bash[46934]: cluster 2022-04-22T19:41:07.974408+0000 mgr.y (mgr.34107) 4638 : cluster [DBG] pgmap v5994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:10 smithi121 bash[36902]: audit 2022-04-22T19:41:09.381357+0000 mon.a (mon.0) 7482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:10.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:10 smithi115 bash[46934]: audit 2022-04-22T19:41:09.381357+0000 mon.a (mon.0) 7482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:10 smithi115 bash[45495]: audit 2022-04-22T19:41:09.381357+0000 mon.a (mon.0) 7482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:11 smithi121 bash[36902]: cluster 2022-04-22T19:41:09.975441+0000 mgr.y (mgr.34107) 4639 : cluster [DBG] pgmap v5995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:11 smithi121 bash[36902]: audit 2022-04-22T19:41:10.835304+0000 mon.a (mon.0) 7483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:11.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:11 smithi115 bash[45495]: cluster 2022-04-22T19:41:09.975441+0000 mgr.y (mgr.34107) 4639 : cluster [DBG] pgmap v5995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:11.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:11 smithi115 bash[45495]: audit 2022-04-22T19:41:10.835304+0000 mon.a (mon.0) 7483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:11.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:11 smithi115 bash[46934]: cluster 2022-04-22T19:41:09.975441+0000 mgr.y (mgr.34107) 4639 : cluster [DBG] pgmap v5995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:11.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:11 smithi115 bash[46934]: audit 2022-04-22T19:41:10.835304+0000 mon.a (mon.0) 7483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:12.359 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:12.133Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:11.744658+0000 mon.a (mon.0) 7484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:41:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:11.747730+0000 mon.a (mon.0) 7485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:41:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:11.749427+0000 mon.a (mon.0) 7486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:41:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:12.094073+0000 mon.a (mon.0) 7487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:12.098834+0000 mon.a (mon.0) 7488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[46934]: audit 2022-04-22T19:41:12.312372+0000 mon.a (mon.0) 7489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:12.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:11.744658+0000 mon.a (mon.0) 7484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:41:12.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:11.747730+0000 mon.a (mon.0) 7485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:41:12.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:11.749427+0000 mon.a (mon.0) 7486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:41:12.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:12.094073+0000 mon.a (mon.0) 7487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:12.098834+0000 mon.a (mon.0) 7488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:12 smithi115 bash[45495]: audit 2022-04-22T19:41:12.312372+0000 mon.a (mon.0) 7489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:11.744658+0000 mon.a (mon.0) 7484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:11.747730+0000 mon.a (mon.0) 7485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:11.749427+0000 mon.a (mon.0) 7486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:12.094073+0000 mon.a (mon.0) 7487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:12.098834+0000 mon.a (mon.0) 7488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:12 smithi121 bash[36902]: audit 2022-04-22T19:41:12.312372+0000 mon.a (mon.0) 7489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:13 smithi121 bash[36902]: cluster 2022-04-22T19:41:11.976482+0000 mgr.y (mgr.34107) 4640 : cluster [DBG] pgmap v5996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:13 smithi115 bash[46934]: cluster 2022-04-22T19:41:11.976482+0000 mgr.y (mgr.34107) 4640 : cluster [DBG] pgmap v5996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:13.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:13 smithi115 bash[45495]: cluster 2022-04-22T19:41:11.976482+0000 mgr.y (mgr.34107) 4640 : cluster [DBG] pgmap v5996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:14.005Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:15 smithi121 bash[36902]: cluster 2022-04-22T19:41:13.977548+0000 mgr.y (mgr.34107) 4641 : cluster [DBG] pgmap v5997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:15 smithi121 bash[36902]: audit 2022-04-22T19:41:14.385008+0000 mon.a (mon.0) 7490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:15 smithi121 bash[36902]: audit 2022-04-22T19:41:14.817850+0000 mon.a (mon.0) 7491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:15 smithi121 bash[36902]: audit 2022-04-22T19:41:14.823032+0000 mon.a (mon.0) 7492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:41:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[45495]: cluster 2022-04-22T19:41:13.977548+0000 mgr.y (mgr.34107) 4641 : cluster [DBG] pgmap v5997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[45495]: audit 2022-04-22T19:41:14.385008+0000 mon.a (mon.0) 7490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[45495]: audit 2022-04-22T19:41:14.817850+0000 mon.a (mon.0) 7491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:41:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[45495]: audit 2022-04-22T19:41:14.823032+0000 mon.a (mon.0) 7492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:41:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[46934]: cluster 2022-04-22T19:41:13.977548+0000 mgr.y (mgr.34107) 4641 : cluster [DBG] pgmap v5997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[46934]: audit 2022-04-22T19:41:14.385008+0000 mon.a (mon.0) 7490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[46934]: audit 2022-04-22T19:41:14.817850+0000 mon.a (mon.0) 7491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:41:15.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:15 smithi115 bash[46934]: audit 2022-04-22T19:41:14.823032+0000 mon.a (mon.0) 7492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:41:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:16 smithi115 bash[45495]: audit 2022-04-22T19:41:15.838455+0000 mon.a (mon.0) 7493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:16 smithi115 bash[45495]: audit 2022-04-22T19:41:16.243156+0000 mon.a (mon.0) 7494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:16 smithi115 bash[46934]: audit 2022-04-22T19:41:15.838455+0000 mon.a (mon.0) 7493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:16 smithi115 bash[46934]: audit 2022-04-22T19:41:16.243156+0000 mon.a (mon.0) 7494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:16 smithi121 bash[36902]: audit 2022-04-22T19:41:15.838455+0000 mon.a (mon.0) 7493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:16 smithi121 bash[36902]: audit 2022-04-22T19:41:16.243156+0000 mon.a (mon.0) 7494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:17 smithi121 bash[36902]: cluster 2022-04-22T19:41:15.978371+0000 mgr.y (mgr.34107) 4642 : cluster [DBG] pgmap v5998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:17 smithi121 bash[36902]: audit 2022-04-22T19:41:17.314869+0000 mon.a (mon.0) 7495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:17 smithi115 bash[46934]: cluster 2022-04-22T19:41:15.978371+0000 mgr.y (mgr.34107) 4642 : cluster [DBG] pgmap v5998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:17.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:17 smithi115 bash[46934]: audit 2022-04-22T19:41:17.314869+0000 mon.a (mon.0) 7495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:17 smithi115 bash[45495]: cluster 2022-04-22T19:41:15.978371+0000 mgr.y (mgr.34107) 4642 : cluster [DBG] pgmap v5998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:17 smithi115 bash[45495]: audit 2022-04-22T19:41:17.314869+0000 mon.a (mon.0) 7495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:19.638 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:19 smithi115 bash[45495]: cluster 2022-04-22T19:41:17.979383+0000 mgr.y (mgr.34107) 4643 : cluster [DBG] pgmap v5999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:19.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:19 smithi115 bash[46934]: cluster 2022-04-22T19:41:17.979383+0000 mgr.y (mgr.34107) 4643 : cluster [DBG] pgmap v5999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:19 smithi121 bash[36902]: cluster 2022-04-22T19:41:17.979383+0000 mgr.y (mgr.34107) 4643 : cluster [DBG] pgmap v5999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:20 smithi121 bash[36902]: audit 2022-04-22T19:41:19.388690+0000 mon.a (mon.0) 7496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:20 smithi115 bash[45495]: audit 2022-04-22T19:41:19.388690+0000 mon.a (mon.0) 7496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:20 smithi115 bash[46934]: audit 2022-04-22T19:41:19.388690+0000 mon.a (mon.0) 7496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:21.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:21 smithi115 bash[45495]: cluster 2022-04-22T19:41:19.980434+0000 mgr.y (mgr.34107) 4644 : cluster [DBG] pgmap v6000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:21 smithi115 bash[45495]: audit 2022-04-22T19:41:20.841298+0000 mon.a (mon.0) 7497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:21 smithi115 bash[46934]: cluster 2022-04-22T19:41:19.980434+0000 mgr.y (mgr.34107) 4644 : cluster [DBG] pgmap v6000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:21 smithi115 bash[46934]: audit 2022-04-22T19:41:20.841298+0000 mon.a (mon.0) 7497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:21 smithi121 bash[36902]: cluster 2022-04-22T19:41:19.980434+0000 mgr.y (mgr.34107) 4644 : cluster [DBG] pgmap v6000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:21 smithi121 bash[36902]: audit 2022-04-22T19:41:20.841298+0000 mon.a (mon.0) 7497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:22.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:22.136Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:22 smithi121 bash[36902]: audit 2022-04-22T19:41:22.317893+0000 mon.a (mon.0) 7498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:22.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:22 smithi115 bash[45495]: audit 2022-04-22T19:41:22.317893+0000 mon.a (mon.0) 7498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:22.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:22 smithi115 bash[46934]: audit 2022-04-22T19:41:22.317893+0000 mon.a (mon.0) 7498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:23 smithi115 bash[46934]: cluster 2022-04-22T19:41:21.981509+0000 mgr.y (mgr.34107) 4645 : cluster [DBG] pgmap v6001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:23 smithi115 bash[45495]: cluster 2022-04-22T19:41:21.981509+0000 mgr.y (mgr.34107) 4645 : cluster [DBG] pgmap v6001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:23 smithi121 bash[36902]: cluster 2022-04-22T19:41:21.981509+0000 mgr.y (mgr.34107) 4645 : cluster [DBG] pgmap v6001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:24.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:24.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:24 smithi121 bash[36902]: audit 2022-04-22T19:41:24.392171+0000 mon.a (mon.0) 7499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:24.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:24 smithi115 bash[45495]: audit 2022-04-22T19:41:24.392171+0000 mon.a (mon.0) 7499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:24.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:24 smithi115 bash[46934]: audit 2022-04-22T19:41:24.392171+0000 mon.a (mon.0) 7499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:25 smithi121 bash[36902]: cluster 2022-04-22T19:41:23.982591+0000 mgr.y (mgr.34107) 4646 : cluster [DBG] pgmap v6002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:25 smithi115 bash[45495]: cluster 2022-04-22T19:41:23.982591+0000 mgr.y (mgr.34107) 4646 : cluster [DBG] pgmap v6002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:25 smithi115 bash[46934]: cluster 2022-04-22T19:41:23.982591+0000 mgr.y (mgr.34107) 4646 : cluster [DBG] pgmap v6002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:26 smithi121 bash[36902]: cephadm 2022-04-22T19:41:25.596013+0000 mgr.y (mgr.34107) 4647 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:41:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:26 smithi121 bash[36902]: audit 2022-04-22T19:41:25.606345+0000 mon.a (mon.0) 7500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:26 smithi121 bash[36902]: audit 2022-04-22T19:41:25.615313+0000 mon.a (mon.0) 7501 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:26 smithi121 bash[36902]: audit 2022-04-22T19:41:25.844145+0000 mon.a (mon.0) 7502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[45495]: cephadm 2022-04-22T19:41:25.596013+0000 mgr.y (mgr.34107) 4647 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[45495]: audit 2022-04-22T19:41:25.606345+0000 mon.a (mon.0) 7500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[45495]: audit 2022-04-22T19:41:25.615313+0000 mon.a (mon.0) 7501 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[45495]: audit 2022-04-22T19:41:25.844145+0000 mon.a (mon.0) 7502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[46934]: cephadm 2022-04-22T19:41:25.596013+0000 mgr.y (mgr.34107) 4647 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[46934]: audit 2022-04-22T19:41:25.606345+0000 mon.a (mon.0) 7500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[46934]: audit 2022-04-22T19:41:25.615313+0000 mon.a (mon.0) 7501 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:41:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:26 smithi115 bash[46934]: audit 2022-04-22T19:41:25.844145+0000 mon.a (mon.0) 7502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:27 smithi121 bash[36902]: cluster 2022-04-22T19:41:25.983453+0000 mgr.y (mgr.34107) 4648 : cluster [DBG] pgmap v6003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:27 smithi121 bash[36902]: audit 2022-04-22T19:41:27.320155+0000 mon.a (mon.0) 7503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:27 smithi115 bash[46934]: cluster 2022-04-22T19:41:25.983453+0000 mgr.y (mgr.34107) 4648 : cluster [DBG] pgmap v6003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:27 smithi115 bash[46934]: audit 2022-04-22T19:41:27.320155+0000 mon.a (mon.0) 7503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:27 smithi115 bash[45495]: cluster 2022-04-22T19:41:25.983453+0000 mgr.y (mgr.34107) 4648 : cluster [DBG] pgmap v6003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:27 smithi115 bash[45495]: audit 2022-04-22T19:41:27.320155+0000 mon.a (mon.0) 7503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:29 smithi121 bash[36902]: cluster 2022-04-22T19:41:27.984451+0000 mgr.y (mgr.34107) 4649 : cluster [DBG] pgmap v6004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:29 smithi121 bash[36902]: audit 2022-04-22T19:41:29.395905+0000 mon.a (mon.0) 7504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:29 smithi115 bash[45495]: cluster 2022-04-22T19:41:27.984451+0000 mgr.y (mgr.34107) 4649 : cluster [DBG] pgmap v6004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:29 smithi115 bash[45495]: audit 2022-04-22T19:41:29.395905+0000 mon.a (mon.0) 7504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:29 smithi115 bash[46934]: cluster 2022-04-22T19:41:27.984451+0000 mgr.y (mgr.34107) 4649 : cluster [DBG] pgmap v6004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:29 smithi115 bash[46934]: audit 2022-04-22T19:41:29.395905+0000 mon.a (mon.0) 7504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:31.593 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:41:31.934 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 15s ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 80s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 15s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 80s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 80s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 15s ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 15s ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 80s ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:41:31.935 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 15s ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 15s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 80s ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 15s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 15s ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 15s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 15s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 80s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 80s ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 80s ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:41:31.936 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 80s ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:41:31.937 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 80s ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:41:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:31 smithi121 bash[36902]: cluster 2022-04-22T19:41:29.985655+0000 mgr.y (mgr.34107) 4650 : cluster [DBG] pgmap v6005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:31 smithi121 bash[36902]: audit 2022-04-22T19:41:30.847285+0000 mon.a (mon.0) 7505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:31 smithi115 bash[46934]: cluster 2022-04-22T19:41:29.985655+0000 mgr.y (mgr.34107) 4650 : cluster [DBG] pgmap v6005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:32.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:31 smithi115 bash[46934]: audit 2022-04-22T19:41:30.847285+0000 mon.a (mon.0) 7505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:31 smithi115 bash[45495]: cluster 2022-04-22T19:41:29.985655+0000 mgr.y (mgr.34107) 4650 : cluster [DBG] pgmap v6005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:32.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:31 smithi115 bash[45495]: audit 2022-04-22T19:41:30.847285+0000 mon.a (mon.0) 7505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:32.349 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:41:32.349 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:41:32.349 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:41:32.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:32.349 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:41:32.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:41:32.351 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:41:32.351 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:41:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:32.136Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:32 smithi121 bash[36902]: audit 2022-04-22T19:41:31.566393+0000 mgr.y (mgr.34107) 4651 : audit [DBG] from='client.40248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:32 smithi121 bash[36902]: audit 2022-04-22T19:41:32.322852+0000 mon.a (mon.0) 7506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:32 smithi121 bash[36902]: audit 2022-04-22T19:41:32.338909+0000 mon.c (mon.1) 204 : audit [DBG] from='client.? 172.21.15.115:0/2679689104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[45495]: audit 2022-04-22T19:41:31.566393+0000 mgr.y (mgr.34107) 4651 : audit [DBG] from='client.40248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:33.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[45495]: audit 2022-04-22T19:41:32.322852+0000 mon.a (mon.0) 7506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:33.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[45495]: audit 2022-04-22T19:41:32.338909+0000 mon.c (mon.1) 204 : audit [DBG] from='client.? 172.21.15.115:0/2679689104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[46934]: audit 2022-04-22T19:41:31.566393+0000 mgr.y (mgr.34107) 4651 : audit [DBG] from='client.40248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[46934]: audit 2022-04-22T19:41:32.322852+0000 mon.a (mon.0) 7506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:32 smithi115 bash[46934]: audit 2022-04-22T19:41:32.338909+0000 mon.c (mon.1) 204 : audit [DBG] from='client.? 172.21.15.115:0/2679689104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:41:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:33 smithi121 bash[36902]: audit 2022-04-22T19:41:31.903906+0000 mgr.y (mgr.34107) 4652 : audit [DBG] from='client.40254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:33 smithi121 bash[36902]: cluster 2022-04-22T19:41:31.986429+0000 mgr.y (mgr.34107) 4653 : cluster [DBG] pgmap v6006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:34.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:33 smithi115 bash[45495]: audit 2022-04-22T19:41:31.903906+0000 mgr.y (mgr.34107) 4652 : audit [DBG] from='client.40254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:34.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:33 smithi115 bash[45495]: cluster 2022-04-22T19:41:31.986429+0000 mgr.y (mgr.34107) 4653 : cluster [DBG] pgmap v6006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:34.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:33 smithi115 bash[46934]: audit 2022-04-22T19:41:31.903906+0000 mgr.y (mgr.34107) 4652 : audit [DBG] from='client.40254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:41:34.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:33 smithi115 bash[46934]: cluster 2022-04-22T19:41:31.986429+0000 mgr.y (mgr.34107) 4653 : cluster [DBG] pgmap v6006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:34.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:34 smithi121 bash[36902]: cluster 2022-04-22T19:41:33.987519+0000 mgr.y (mgr.34107) 4654 : cluster [DBG] pgmap v6007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:34 smithi121 bash[36902]: audit 2022-04-22T19:41:34.399432+0000 mon.a (mon.0) 7507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:35.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:34 smithi115 bash[45495]: cluster 2022-04-22T19:41:33.987519+0000 mgr.y (mgr.34107) 4654 : cluster [DBG] pgmap v6007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:35.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:34 smithi115 bash[45495]: audit 2022-04-22T19:41:34.399432+0000 mon.a (mon.0) 7507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:34 smithi115 bash[46934]: cluster 2022-04-22T19:41:33.987519+0000 mgr.y (mgr.34107) 4654 : cluster [DBG] pgmap v6007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:34 smithi115 bash[46934]: audit 2022-04-22T19:41:34.399432+0000 mon.a (mon.0) 7507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:35 smithi121 bash[36902]: audit 2022-04-22T19:41:35.850426+0000 mon.a (mon.0) 7508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:35 smithi115 bash[45495]: audit 2022-04-22T19:41:35.850426+0000 mon.a (mon.0) 7508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:36.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:35 smithi115 bash[46934]: audit 2022-04-22T19:41:35.850426+0000 mon.a (mon.0) 7508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:36 smithi121 bash[36902]: cluster 2022-04-22T19:41:35.988440+0000 mgr.y (mgr.34107) 4655 : cluster [DBG] pgmap v6008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:37.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:36 smithi115 bash[45495]: cluster 2022-04-22T19:41:35.988440+0000 mgr.y (mgr.34107) 4655 : cluster [DBG] pgmap v6008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:36 smithi115 bash[46934]: cluster 2022-04-22T19:41:35.988440+0000 mgr.y (mgr.34107) 4655 : cluster [DBG] pgmap v6008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:37 smithi121 bash[36902]: audit 2022-04-22T19:41:37.325859+0000 mon.a (mon.0) 7509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:38.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:37 smithi115 bash[45495]: audit 2022-04-22T19:41:37.325859+0000 mon.a (mon.0) 7509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:38.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:37 smithi115 bash[46934]: audit 2022-04-22T19:41:37.325859+0000 mon.a (mon.0) 7509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:39 smithi121 bash[36902]: cluster 2022-04-22T19:41:37.989627+0000 mgr.y (mgr.34107) 4656 : cluster [DBG] pgmap v6009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:39 smithi115 bash[45495]: cluster 2022-04-22T19:41:37.989627+0000 mgr.y (mgr.34107) 4656 : cluster [DBG] pgmap v6009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:39.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:39 smithi115 bash[46934]: cluster 2022-04-22T19:41:37.989627+0000 mgr.y (mgr.34107) 4656 : cluster [DBG] pgmap v6009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:40 smithi121 bash[36902]: audit 2022-04-22T19:41:39.403227+0000 mon.a (mon.0) 7510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:40.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:40 smithi115 bash[45495]: audit 2022-04-22T19:41:39.403227+0000 mon.a (mon.0) 7510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:40.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:40 smithi115 bash[46934]: audit 2022-04-22T19:41:39.403227+0000 mon.a (mon.0) 7510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:41 smithi121 bash[36902]: cluster 2022-04-22T19:41:39.990751+0000 mgr.y (mgr.34107) 4657 : cluster [DBG] pgmap v6010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:41 smithi121 bash[36902]: audit 2022-04-22T19:41:40.853771+0000 mon.a (mon.0) 7511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:41.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:41 smithi115 bash[45495]: cluster 2022-04-22T19:41:39.990751+0000 mgr.y (mgr.34107) 4657 : cluster [DBG] pgmap v6010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:41.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:41 smithi115 bash[45495]: audit 2022-04-22T19:41:40.853771+0000 mon.a (mon.0) 7511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:41 smithi115 bash[46934]: cluster 2022-04-22T19:41:39.990751+0000 mgr.y (mgr.34107) 4657 : cluster [DBG] pgmap v6010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:41.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:41 smithi115 bash[46934]: audit 2022-04-22T19:41:40.853771+0000 mon.a (mon.0) 7511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:42.136Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:43 smithi121 bash[36902]: cluster 2022-04-22T19:41:41.991817+0000 mgr.y (mgr.34107) 4658 : cluster [DBG] pgmap v6011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:43 smithi121 bash[36902]: audit 2022-04-22T19:41:42.328787+0000 mon.a (mon.0) 7512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:43 smithi115 bash[45495]: cluster 2022-04-22T19:41:41.991817+0000 mgr.y (mgr.34107) 4658 : cluster [DBG] pgmap v6011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:43.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:43 smithi115 bash[45495]: audit 2022-04-22T19:41:42.328787+0000 mon.a (mon.0) 7512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:43 smithi115 bash[46934]: cluster 2022-04-22T19:41:41.991817+0000 mgr.y (mgr.34107) 4658 : cluster [DBG] pgmap v6011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:43 smithi115 bash[46934]: audit 2022-04-22T19:41:42.328787+0000 mon.a (mon.0) 7512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:44.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:45 smithi121 bash[36902]: cluster 2022-04-22T19:41:43.992831+0000 mgr.y (mgr.34107) 4659 : cluster [DBG] pgmap v6012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:45 smithi121 bash[36902]: audit 2022-04-22T19:41:44.407004+0000 mon.a (mon.0) 7513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:45 smithi115 bash[45495]: cluster 2022-04-22T19:41:43.992831+0000 mgr.y (mgr.34107) 4659 : cluster [DBG] pgmap v6012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:45 smithi115 bash[45495]: audit 2022-04-22T19:41:44.407004+0000 mon.a (mon.0) 7513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:45 smithi115 bash[46934]: cluster 2022-04-22T19:41:43.992831+0000 mgr.y (mgr.34107) 4659 : cluster [DBG] pgmap v6012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:45 smithi115 bash[46934]: audit 2022-04-22T19:41:44.407004+0000 mon.a (mon.0) 7513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:46 smithi121 bash[36902]: audit 2022-04-22T19:41:45.856830+0000 mon.a (mon.0) 7514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:46.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:46 smithi115 bash[45495]: audit 2022-04-22T19:41:45.856830+0000 mon.a (mon.0) 7514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:46.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:46 smithi115 bash[46934]: audit 2022-04-22T19:41:45.856830+0000 mon.a (mon.0) 7514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:47 smithi121 bash[36902]: cluster 2022-04-22T19:41:45.993827+0000 mgr.y (mgr.34107) 4660 : cluster [DBG] pgmap v6013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:47.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:47 smithi115 bash[45495]: cluster 2022-04-22T19:41:45.993827+0000 mgr.y (mgr.34107) 4660 : cluster [DBG] pgmap v6013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:47.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:47 smithi115 bash[46934]: cluster 2022-04-22T19:41:45.993827+0000 mgr.y (mgr.34107) 4660 : cluster [DBG] pgmap v6013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:48 smithi121 bash[36902]: audit 2022-04-22T19:41:47.331795+0000 mon.a (mon.0) 7515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:48.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:48 smithi115 bash[45495]: audit 2022-04-22T19:41:47.331795+0000 mon.a (mon.0) 7515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:48.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:48 smithi115 bash[46934]: audit 2022-04-22T19:41:47.331795+0000 mon.a (mon.0) 7515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:49 smithi121 bash[36902]: cluster 2022-04-22T19:41:47.995049+0000 mgr.y (mgr.34107) 4661 : cluster [DBG] pgmap v6014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:49 smithi115 bash[45495]: cluster 2022-04-22T19:41:47.995049+0000 mgr.y (mgr.34107) 4661 : cluster [DBG] pgmap v6014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:49 smithi115 bash[46934]: cluster 2022-04-22T19:41:47.995049+0000 mgr.y (mgr.34107) 4661 : cluster [DBG] pgmap v6014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:50 smithi121 bash[36902]: audit 2022-04-22T19:41:49.410865+0000 mon.a (mon.0) 7516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:50.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:50 smithi115 bash[45495]: audit 2022-04-22T19:41:49.410865+0000 mon.a (mon.0) 7516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:50.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:50 smithi115 bash[46934]: audit 2022-04-22T19:41:49.410865+0000 mon.a (mon.0) 7516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:51 smithi121 bash[36902]: cluster 2022-04-22T19:41:49.996126+0000 mgr.y (mgr.34107) 4662 : cluster [DBG] pgmap v6015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:51 smithi121 bash[36902]: audit 2022-04-22T19:41:50.859751+0000 mon.a (mon.0) 7517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:51 smithi115 bash[45495]: cluster 2022-04-22T19:41:49.996126+0000 mgr.y (mgr.34107) 4662 : cluster [DBG] pgmap v6015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:51 smithi115 bash[45495]: audit 2022-04-22T19:41:50.859751+0000 mon.a (mon.0) 7517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:51 smithi115 bash[46934]: cluster 2022-04-22T19:41:49.996126+0000 mgr.y (mgr.34107) 4662 : cluster [DBG] pgmap v6015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:51.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:51 smithi115 bash[46934]: audit 2022-04-22T19:41:50.859751+0000 mon.a (mon.0) 7517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:52.139Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:53 smithi121 bash[36902]: cluster 2022-04-22T19:41:51.997190+0000 mgr.y (mgr.34107) 4663 : cluster [DBG] pgmap v6016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:53 smithi121 bash[36902]: audit 2022-04-22T19:41:52.334861+0000 mon.a (mon.0) 7518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:53 smithi115 bash[45495]: cluster 2022-04-22T19:41:51.997190+0000 mgr.y (mgr.34107) 4663 : cluster [DBG] pgmap v6016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:53.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:53 smithi115 bash[45495]: audit 2022-04-22T19:41:52.334861+0000 mon.a (mon.0) 7518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:53 smithi115 bash[46934]: cluster 2022-04-22T19:41:51.997190+0000 mgr.y (mgr.34107) 4663 : cluster [DBG] pgmap v6016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:53 smithi115 bash[46934]: audit 2022-04-22T19:41:52.334861+0000 mon.a (mon.0) 7518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:41:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:41:54.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:41:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:55 smithi121 bash[36902]: cluster 2022-04-22T19:41:53.998323+0000 mgr.y (mgr.34107) 4664 : cluster [DBG] pgmap v6017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:55 smithi121 bash[36902]: audit 2022-04-22T19:41:54.414631+0000 mon.a (mon.0) 7519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:55 smithi115 bash[45495]: cluster 2022-04-22T19:41:53.998323+0000 mgr.y (mgr.34107) 4664 : cluster [DBG] pgmap v6017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:55 smithi115 bash[45495]: audit 2022-04-22T19:41:54.414631+0000 mon.a (mon.0) 7519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:55 smithi115 bash[46934]: cluster 2022-04-22T19:41:53.998323+0000 mgr.y (mgr.34107) 4664 : cluster [DBG] pgmap v6017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:55 smithi115 bash[46934]: audit 2022-04-22T19:41:54.414631+0000 mon.a (mon.0) 7519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:41:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:56 smithi121 bash[36902]: audit 2022-04-22T19:41:55.863019+0000 mon.a (mon.0) 7520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:56.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:56 smithi115 bash[45495]: audit 2022-04-22T19:41:55.863019+0000 mon.a (mon.0) 7520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:56.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:56 smithi115 bash[46934]: audit 2022-04-22T19:41:55.863019+0000 mon.a (mon.0) 7520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:41:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:57 smithi121 bash[36902]: cluster 2022-04-22T19:41:55.998935+0000 mgr.y (mgr.34107) 4665 : cluster [DBG] pgmap v6018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:57.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:57 smithi115 bash[45495]: cluster 2022-04-22T19:41:55.998935+0000 mgr.y (mgr.34107) 4665 : cluster [DBG] pgmap v6018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:57 smithi115 bash[46934]: cluster 2022-04-22T19:41:55.998935+0000 mgr.y (mgr.34107) 4665 : cluster [DBG] pgmap v6018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:58 smithi121 bash[36902]: audit 2022-04-22T19:41:57.337990+0000 mon.a (mon.0) 7521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:58 smithi115 bash[45495]: audit 2022-04-22T19:41:57.337990+0000 mon.a (mon.0) 7521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:58.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:58 smithi115 bash[46934]: audit 2022-04-22T19:41:57.337990+0000 mon.a (mon.0) 7521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:41:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:41:59 smithi121 bash[36902]: cluster 2022-04-22T19:41:58.000028+0000 mgr.y (mgr.34107) 4666 : cluster [DBG] pgmap v6019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:41:59 smithi115 bash[45495]: cluster 2022-04-22T19:41:58.000028+0000 mgr.y (mgr.34107) 4666 : cluster [DBG] pgmap v6019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:41:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:41:59 smithi115 bash[46934]: cluster 2022-04-22T19:41:58.000028+0000 mgr.y (mgr.34107) 4666 : cluster [DBG] pgmap v6019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:00 smithi121 bash[36902]: audit 2022-04-22T19:41:59.418379+0000 mon.a (mon.0) 7522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:00 smithi115 bash[45495]: audit 2022-04-22T19:41:59.418379+0000 mon.a (mon.0) 7522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:00.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:00 smithi115 bash[46934]: audit 2022-04-22T19:41:59.418379+0000 mon.a (mon.0) 7522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:01 smithi121 bash[36902]: cluster 2022-04-22T19:42:00.001056+0000 mgr.y (mgr.34107) 4667 : cluster [DBG] pgmap v6020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:01 smithi121 bash[36902]: audit 2022-04-22T19:42:00.866358+0000 mon.a (mon.0) 7523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:01 smithi115 bash[45495]: cluster 2022-04-22T19:42:00.001056+0000 mgr.y (mgr.34107) 4667 : cluster [DBG] pgmap v6020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:01 smithi115 bash[45495]: audit 2022-04-22T19:42:00.866358+0000 mon.a (mon.0) 7523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:01 smithi115 bash[46934]: cluster 2022-04-22T19:42:00.001056+0000 mgr.y (mgr.34107) 4667 : cluster [DBG] pgmap v6020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:01 smithi115 bash[46934]: audit 2022-04-22T19:42:00.866358+0000 mon.a (mon.0) 7523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:02.140Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:02.726 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:42:03.067 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 46s ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 111s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 46s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 111s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 111s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 46s ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 46s ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:42:03.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 111s ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 46s ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 46s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 111s ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 46s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 46s ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 46s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 46s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 111s ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 111s ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 111s ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:42:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 111s ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:42:03.070 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 111s ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:42:03.458 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:42:03.458 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:42:03.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:42:03.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:03.459 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:42:03.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:42:03.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:42:03.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:03.461 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:42:03.461 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:42:03.461 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:42:03.461 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:42:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:03 smithi121 bash[36902]: cluster 2022-04-22T19:42:02.002144+0000 mgr.y (mgr.34107) 4668 : cluster [DBG] pgmap v6021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:03 smithi121 bash[36902]: audit 2022-04-22T19:42:02.340965+0000 mon.a (mon.0) 7524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:03 smithi115 bash[45495]: cluster 2022-04-22T19:42:02.002144+0000 mgr.y (mgr.34107) 4668 : cluster [DBG] pgmap v6021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:03 smithi115 bash[45495]: audit 2022-04-22T19:42:02.340965+0000 mon.a (mon.0) 7524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:03 smithi115 bash[46934]: cluster 2022-04-22T19:42:02.002144+0000 mgr.y (mgr.34107) 4668 : cluster [DBG] pgmap v6021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:03 smithi115 bash[46934]: audit 2022-04-22T19:42:02.340965+0000 mon.a (mon.0) 7524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:04.009Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[45495]: audit 2022-04-22T19:42:02.700127+0000 mgr.y (mgr.34107) 4669 : audit [DBG] from='client.40266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[45495]: audit 2022-04-22T19:42:03.043607+0000 mgr.y (mgr.34107) 4670 : audit [DBG] from='client.40272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[45495]: audit 2022-04-22T19:42:03.448418+0000 mon.a (mon.0) 7525 : audit [DBG] from='client.? 172.21.15.115:0/2241251860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[46934]: audit 2022-04-22T19:42:02.700127+0000 mgr.y (mgr.34107) 4669 : audit [DBG] from='client.40266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[46934]: audit 2022-04-22T19:42:03.043607+0000 mgr.y (mgr.34107) 4670 : audit [DBG] from='client.40272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:04 smithi115 bash[46934]: audit 2022-04-22T19:42:03.448418+0000 mon.a (mon.0) 7525 : audit [DBG] from='client.? 172.21.15.115:0/2241251860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:04 smithi121 bash[36902]: audit 2022-04-22T19:42:02.700127+0000 mgr.y (mgr.34107) 4669 : audit [DBG] from='client.40266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:04 smithi121 bash[36902]: audit 2022-04-22T19:42:03.043607+0000 mgr.y (mgr.34107) 4670 : audit [DBG] from='client.40272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:04 smithi121 bash[36902]: audit 2022-04-22T19:42:03.448418+0000 mon.a (mon.0) 7525 : audit [DBG] from='client.? 172.21.15.115:0/2241251860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:05 smithi121 bash[36902]: cluster 2022-04-22T19:42:04.003317+0000 mgr.y (mgr.34107) 4671 : cluster [DBG] pgmap v6022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:05 smithi121 bash[36902]: audit 2022-04-22T19:42:04.421764+0000 mon.a (mon.0) 7526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:05 smithi115 bash[45495]: cluster 2022-04-22T19:42:04.003317+0000 mgr.y (mgr.34107) 4671 : cluster [DBG] pgmap v6022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:05.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:05 smithi115 bash[45495]: audit 2022-04-22T19:42:04.421764+0000 mon.a (mon.0) 7526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:05 smithi115 bash[46934]: cluster 2022-04-22T19:42:04.003317+0000 mgr.y (mgr.34107) 4671 : cluster [DBG] pgmap v6022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:05 smithi115 bash[46934]: audit 2022-04-22T19:42:04.421764+0000 mon.a (mon.0) 7526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:06 smithi121 bash[36902]: audit 2022-04-22T19:42:05.869495+0000 mon.a (mon.0) 7527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:06 smithi115 bash[45495]: audit 2022-04-22T19:42:05.869495+0000 mon.a (mon.0) 7527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:06.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:06 smithi115 bash[46934]: audit 2022-04-22T19:42:05.869495+0000 mon.a (mon.0) 7527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:07 smithi121 bash[36902]: cluster 2022-04-22T19:42:06.004116+0000 mgr.y (mgr.34107) 4672 : cluster [DBG] pgmap v6023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:07 smithi115 bash[45495]: cluster 2022-04-22T19:42:06.004116+0000 mgr.y (mgr.34107) 4672 : cluster [DBG] pgmap v6023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:07.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:07 smithi115 bash[46934]: cluster 2022-04-22T19:42:06.004116+0000 mgr.y (mgr.34107) 4672 : cluster [DBG] pgmap v6023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:08 smithi121 bash[36902]: audit 2022-04-22T19:42:07.344128+0000 mon.a (mon.0) 7528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:08 smithi115 bash[45495]: audit 2022-04-22T19:42:07.344128+0000 mon.a (mon.0) 7528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:08 smithi115 bash[46934]: audit 2022-04-22T19:42:07.344128+0000 mon.a (mon.0) 7528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:09 smithi121 bash[36902]: cluster 2022-04-22T19:42:08.005250+0000 mgr.y (mgr.34107) 4673 : cluster [DBG] pgmap v6024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:09 smithi115 bash[45495]: cluster 2022-04-22T19:42:08.005250+0000 mgr.y (mgr.34107) 4673 : cluster [DBG] pgmap v6024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:09.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:09 smithi115 bash[46934]: cluster 2022-04-22T19:42:08.005250+0000 mgr.y (mgr.34107) 4673 : cluster [DBG] pgmap v6024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:10 smithi121 bash[36902]: audit 2022-04-22T19:42:09.425532+0000 mon.a (mon.0) 7529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:10 smithi115 bash[45495]: audit 2022-04-22T19:42:09.425532+0000 mon.a (mon.0) 7529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:10 smithi115 bash[46934]: audit 2022-04-22T19:42:09.425532+0000 mon.a (mon.0) 7529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:11 smithi115 bash[45495]: cluster 2022-04-22T19:42:10.006648+0000 mgr.y (mgr.34107) 4674 : cluster [DBG] pgmap v6025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:11.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:11 smithi115 bash[45495]: audit 2022-04-22T19:42:10.872660+0000 mon.a (mon.0) 7530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:11 smithi115 bash[46934]: cluster 2022-04-22T19:42:10.006648+0000 mgr.y (mgr.34107) 4674 : cluster [DBG] pgmap v6025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:11 smithi115 bash[46934]: audit 2022-04-22T19:42:10.872660+0000 mon.a (mon.0) 7530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:11 smithi121 bash[36902]: cluster 2022-04-22T19:42:10.006648+0000 mgr.y (mgr.34107) 4674 : cluster [DBG] pgmap v6025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:11 smithi121 bash[36902]: audit 2022-04-22T19:42:10.872660+0000 mon.a (mon.0) 7530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:12.141Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:13 smithi115 bash[46934]: cluster 2022-04-22T19:42:12.007828+0000 mgr.y (mgr.34107) 4675 : cluster [DBG] pgmap v6026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:13 smithi115 bash[46934]: audit 2022-04-22T19:42:12.347074+0000 mon.a (mon.0) 7531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:13 smithi115 bash[45495]: cluster 2022-04-22T19:42:12.007828+0000 mgr.y (mgr.34107) 4675 : cluster [DBG] pgmap v6026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:13 smithi115 bash[45495]: audit 2022-04-22T19:42:12.347074+0000 mon.a (mon.0) 7531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:13 smithi121 bash[36902]: cluster 2022-04-22T19:42:12.007828+0000 mgr.y (mgr.34107) 4675 : cluster [DBG] pgmap v6026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:13 smithi121 bash[36902]: audit 2022-04-22T19:42:12.347074+0000 mon.a (mon.0) 7531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:14.010Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[45495]: cluster 2022-04-22T19:42:14.008987+0000 mgr.y (mgr.34107) 4676 : cluster [DBG] pgmap v6027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[45495]: audit 2022-04-22T19:42:14.429006+0000 mon.a (mon.0) 7532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[45495]: audit 2022-04-22T19:42:14.830419+0000 mon.a (mon.0) 7533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[45495]: audit 2022-04-22T19:42:14.838397+0000 mon.a (mon.0) 7534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[46934]: cluster 2022-04-22T19:42:14.008987+0000 mgr.y (mgr.34107) 4676 : cluster [DBG] pgmap v6027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[46934]: audit 2022-04-22T19:42:14.429006+0000 mon.a (mon.0) 7532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[46934]: audit 2022-04-22T19:42:14.830419+0000 mon.a (mon.0) 7533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:42:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:15 smithi115 bash[46934]: audit 2022-04-22T19:42:14.838397+0000 mon.a (mon.0) 7534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:42:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:15 smithi121 bash[36902]: cluster 2022-04-22T19:42:14.008987+0000 mgr.y (mgr.34107) 4676 : cluster [DBG] pgmap v6027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:15 smithi121 bash[36902]: audit 2022-04-22T19:42:14.429006+0000 mon.a (mon.0) 7532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:15 smithi121 bash[36902]: audit 2022-04-22T19:42:14.830419+0000 mon.a (mon.0) 7533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:42:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:15 smithi121 bash[36902]: audit 2022-04-22T19:42:14.838397+0000 mon.a (mon.0) 7534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:42:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:16 smithi115 bash[46934]: audit 2022-04-22T19:42:15.875738+0000 mon.a (mon.0) 7535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:16.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:16 smithi115 bash[45495]: audit 2022-04-22T19:42:15.875738+0000 mon.a (mon.0) 7535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:16 smithi121 bash[36902]: audit 2022-04-22T19:42:15.875738+0000 mon.a (mon.0) 7535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:17 smithi115 bash[45495]: cluster 2022-04-22T19:42:16.009952+0000 mgr.y (mgr.34107) 4677 : cluster [DBG] pgmap v6028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:17 smithi115 bash[46934]: cluster 2022-04-22T19:42:16.009952+0000 mgr.y (mgr.34107) 4677 : cluster [DBG] pgmap v6028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:17 smithi121 bash[36902]: cluster 2022-04-22T19:42:16.009952+0000 mgr.y (mgr.34107) 4677 : cluster [DBG] pgmap v6028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:18 smithi115 bash[45495]: audit 2022-04-22T19:42:17.349994+0000 mon.a (mon.0) 7536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:18 smithi115 bash[46934]: audit 2022-04-22T19:42:17.349994+0000 mon.a (mon.0) 7536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:18 smithi121 bash[36902]: audit 2022-04-22T19:42:17.349994+0000 mon.a (mon.0) 7536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:19 smithi115 bash[45495]: cluster 2022-04-22T19:42:18.011187+0000 mgr.y (mgr.34107) 4678 : cluster [DBG] pgmap v6029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:19.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:19 smithi115 bash[46934]: cluster 2022-04-22T19:42:18.011187+0000 mgr.y (mgr.34107) 4678 : cluster [DBG] pgmap v6029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:19 smithi121 bash[36902]: cluster 2022-04-22T19:42:18.011187+0000 mgr.y (mgr.34107) 4678 : cluster [DBG] pgmap v6029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:20 smithi121 bash[36902]: audit 2022-04-22T19:42:19.432618+0000 mon.a (mon.0) 7537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:20 smithi115 bash[45495]: audit 2022-04-22T19:42:19.432618+0000 mon.a (mon.0) 7537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:20 smithi115 bash[46934]: audit 2022-04-22T19:42:19.432618+0000 mon.a (mon.0) 7537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:21 smithi121 bash[36902]: cluster 2022-04-22T19:42:20.012153+0000 mgr.y (mgr.34107) 4679 : cluster [DBG] pgmap v6030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:21 smithi121 bash[36902]: audit 2022-04-22T19:42:20.879084+0000 mon.a (mon.0) 7538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:21 smithi115 bash[45495]: cluster 2022-04-22T19:42:20.012153+0000 mgr.y (mgr.34107) 4679 : cluster [DBG] pgmap v6030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:21 smithi115 bash[45495]: audit 2022-04-22T19:42:20.879084+0000 mon.a (mon.0) 7538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:21 smithi115 bash[46934]: cluster 2022-04-22T19:42:20.012153+0000 mgr.y (mgr.34107) 4679 : cluster [DBG] pgmap v6030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:21 smithi115 bash[46934]: audit 2022-04-22T19:42:20.879084+0000 mon.a (mon.0) 7538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:22.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:23 smithi121 bash[36902]: cluster 2022-04-22T19:42:22.013212+0000 mgr.y (mgr.34107) 4680 : cluster [DBG] pgmap v6031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:23 smithi121 bash[36902]: audit 2022-04-22T19:42:22.353018+0000 mon.a (mon.0) 7539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:23 smithi115 bash[45495]: cluster 2022-04-22T19:42:22.013212+0000 mgr.y (mgr.34107) 4680 : cluster [DBG] pgmap v6031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:23 smithi115 bash[45495]: audit 2022-04-22T19:42:22.353018+0000 mon.a (mon.0) 7539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:23 smithi115 bash[46934]: cluster 2022-04-22T19:42:22.013212+0000 mgr.y (mgr.34107) 4680 : cluster [DBG] pgmap v6031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:23 smithi115 bash[46934]: audit 2022-04-22T19:42:22.353018+0000 mon.a (mon.0) 7539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:24.011Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:25.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:25 smithi115 bash[46934]: cluster 2022-04-22T19:42:24.014297+0000 mgr.y (mgr.34107) 4681 : cluster [DBG] pgmap v6032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:25.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:25 smithi115 bash[46934]: audit 2022-04-22T19:42:24.436319+0000 mon.a (mon.0) 7540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:25.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:25 smithi115 bash[45495]: cluster 2022-04-22T19:42:24.014297+0000 mgr.y (mgr.34107) 4681 : cluster [DBG] pgmap v6032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:25.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:25 smithi115 bash[45495]: audit 2022-04-22T19:42:24.436319+0000 mon.a (mon.0) 7540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:25.660 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:25 smithi121 bash[36902]: cluster 2022-04-22T19:42:24.014297+0000 mgr.y (mgr.34107) 4681 : cluster [DBG] pgmap v6032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:25.660 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:25 smithi121 bash[36902]: audit 2022-04-22T19:42:24.436319+0000 mon.a (mon.0) 7540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:26 smithi121 bash[36902]: audit 2022-04-22T19:42:25.623912+0000 mon.a (mon.0) 7541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:42:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:26 smithi121 bash[36902]: audit 2022-04-22T19:42:25.627038+0000 mon.a (mon.0) 7542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:42:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:26 smithi121 bash[36902]: audit 2022-04-22T19:42:25.628671+0000 mon.a (mon.0) 7543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:42:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:26 smithi121 bash[36902]: audit 2022-04-22T19:42:25.882535+0000 mon.a (mon.0) 7544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:26 smithi121 bash[36902]: audit 2022-04-22T19:42:25.998144+0000 mon.a (mon.0) 7545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:42:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[45495]: audit 2022-04-22T19:42:25.623912+0000 mon.a (mon.0) 7541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:42:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[45495]: audit 2022-04-22T19:42:25.627038+0000 mon.a (mon.0) 7542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[45495]: audit 2022-04-22T19:42:25.628671+0000 mon.a (mon.0) 7543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[45495]: audit 2022-04-22T19:42:25.882535+0000 mon.a (mon.0) 7544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[45495]: audit 2022-04-22T19:42:25.998144+0000 mon.a (mon.0) 7545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[46934]: audit 2022-04-22T19:42:25.623912+0000 mon.a (mon.0) 7541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[46934]: audit 2022-04-22T19:42:25.627038+0000 mon.a (mon.0) 7542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[46934]: audit 2022-04-22T19:42:25.628671+0000 mon.a (mon.0) 7543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:42:26.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[46934]: audit 2022-04-22T19:42:25.882535+0000 mon.a (mon.0) 7544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:26.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:26 smithi115 bash[46934]: audit 2022-04-22T19:42:25.998144+0000 mon.a (mon.0) 7545 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:42:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:27 smithi121 bash[36902]: cluster 2022-04-22T19:42:26.015088+0000 mgr.y (mgr.34107) 4682 : cluster [DBG] pgmap v6033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:27 smithi115 bash[45495]: cluster 2022-04-22T19:42:26.015088+0000 mgr.y (mgr.34107) 4682 : cluster [DBG] pgmap v6033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:27 smithi115 bash[46934]: cluster 2022-04-22T19:42:26.015088+0000 mgr.y (mgr.34107) 4682 : cluster [DBG] pgmap v6033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:28 smithi121 bash[36902]: audit 2022-04-22T19:42:27.355586+0000 mon.a (mon.0) 7546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:28 smithi115 bash[45495]: audit 2022-04-22T19:42:27.355586+0000 mon.a (mon.0) 7546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:28 smithi115 bash[46934]: audit 2022-04-22T19:42:27.355586+0000 mon.a (mon.0) 7546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:29 smithi121 bash[36902]: cluster 2022-04-22T19:42:28.016140+0000 mgr.y (mgr.34107) 4683 : cluster [DBG] pgmap v6034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:29 smithi115 bash[45495]: cluster 2022-04-22T19:42:28.016140+0000 mgr.y (mgr.34107) 4683 : cluster [DBG] pgmap v6034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:29 smithi115 bash[46934]: cluster 2022-04-22T19:42:28.016140+0000 mgr.y (mgr.34107) 4683 : cluster [DBG] pgmap v6034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:30 smithi121 bash[36902]: audit 2022-04-22T19:42:29.439923+0000 mon.a (mon.0) 7547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:30 smithi115 bash[46934]: audit 2022-04-22T19:42:29.439923+0000 mon.a (mon.0) 7547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:30 smithi115 bash[45495]: audit 2022-04-22T19:42:29.439923+0000 mon.a (mon.0) 7547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:31 smithi121 bash[36902]: cluster 2022-04-22T19:42:30.017367+0000 mgr.y (mgr.34107) 4684 : cluster [DBG] pgmap v6035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:31 smithi121 bash[36902]: audit 2022-04-22T19:42:30.885687+0000 mon.a (mon.0) 7548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:31 smithi115 bash[45495]: cluster 2022-04-22T19:42:30.017367+0000 mgr.y (mgr.34107) 4684 : cluster [DBG] pgmap v6035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:31 smithi115 bash[45495]: audit 2022-04-22T19:42:30.885687+0000 mon.a (mon.0) 7548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:31 smithi115 bash[46934]: cluster 2022-04-22T19:42:30.017367+0000 mgr.y (mgr.34107) 4684 : cluster [DBG] pgmap v6035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:31 smithi115 bash[46934]: audit 2022-04-22T19:42:30.885687+0000 mon.a (mon.0) 7548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:32.140Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:33 smithi121 bash[36902]: cluster 2022-04-22T19:42:32.018723+0000 mgr.y (mgr.34107) 4685 : cluster [DBG] pgmap v6036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:33 smithi121 bash[36902]: audit 2022-04-22T19:42:32.358556+0000 mon.a (mon.0) 7549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:33 smithi115 bash[45495]: cluster 2022-04-22T19:42:32.018723+0000 mgr.y (mgr.34107) 4685 : cluster [DBG] pgmap v6036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:33 smithi115 bash[45495]: audit 2022-04-22T19:42:32.358556+0000 mon.a (mon.0) 7549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:33 smithi115 bash[46934]: cluster 2022-04-22T19:42:32.018723+0000 mgr.y (mgr.34107) 4685 : cluster [DBG] pgmap v6036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:33 smithi115 bash[46934]: audit 2022-04-22T19:42:32.358556+0000 mon.a (mon.0) 7549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:33.834 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 77s ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 77s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:42:34.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 77s ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 77s ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 77s ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 77s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 77s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 77s ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 77s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 77s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:42:34.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:42:34.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:42:34.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:42:34.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:42:34.186 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:42:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:34.011Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:34.591 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:42:34.591 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:42:34.593 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:42:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:35 smithi121 bash[36902]: audit 2022-04-22T19:42:33.808435+0000 mgr.y (mgr.34107) 4686 : audit [DBG] from='client.40284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:35 smithi121 bash[36902]: cluster 2022-04-22T19:42:34.019787+0000 mgr.y (mgr.34107) 4687 : cluster [DBG] pgmap v6037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:35 smithi121 bash[36902]: audit 2022-04-22T19:42:34.153825+0000 mgr.y (mgr.34107) 4688 : audit [DBG] from='client.40290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:35 smithi121 bash[36902]: audit 2022-04-22T19:42:34.443754+0000 mon.a (mon.0) 7550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:35 smithi121 bash[36902]: audit 2022-04-22T19:42:34.581291+0000 mon.a (mon.0) 7551 : audit [DBG] from='client.? 172.21.15.115:0/4064039354' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[46934]: audit 2022-04-22T19:42:33.808435+0000 mgr.y (mgr.34107) 4686 : audit [DBG] from='client.40284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[46934]: cluster 2022-04-22T19:42:34.019787+0000 mgr.y (mgr.34107) 4687 : cluster [DBG] pgmap v6037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[46934]: audit 2022-04-22T19:42:34.153825+0000 mgr.y (mgr.34107) 4688 : audit [DBG] from='client.40290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[46934]: audit 2022-04-22T19:42:34.443754+0000 mon.a (mon.0) 7550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[46934]: audit 2022-04-22T19:42:34.581291+0000 mon.a (mon.0) 7551 : audit [DBG] from='client.? 172.21.15.115:0/4064039354' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[45495]: audit 2022-04-22T19:42:33.808435+0000 mgr.y (mgr.34107) 4686 : audit [DBG] from='client.40284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[45495]: cluster 2022-04-22T19:42:34.019787+0000 mgr.y (mgr.34107) 4687 : cluster [DBG] pgmap v6037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[45495]: audit 2022-04-22T19:42:34.153825+0000 mgr.y (mgr.34107) 4688 : audit [DBG] from='client.40290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:42:35.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[45495]: audit 2022-04-22T19:42:34.443754+0000 mon.a (mon.0) 7550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:35.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:35 smithi115 bash[45495]: audit 2022-04-22T19:42:34.581291+0000 mon.a (mon.0) 7551 : audit [DBG] from='client.? 172.21.15.115:0/4064039354' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:42:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:36 smithi121 bash[36902]: audit 2022-04-22T19:42:35.888696+0000 mon.a (mon.0) 7552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:36 smithi115 bash[46934]: audit 2022-04-22T19:42:35.888696+0000 mon.a (mon.0) 7552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:36.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:36 smithi115 bash[45495]: audit 2022-04-22T19:42:35.888696+0000 mon.a (mon.0) 7552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:37 smithi121 bash[36902]: cluster 2022-04-22T19:42:36.020780+0000 mgr.y (mgr.34107) 4689 : cluster [DBG] pgmap v6038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:37 smithi115 bash[46934]: cluster 2022-04-22T19:42:36.020780+0000 mgr.y (mgr.34107) 4689 : cluster [DBG] pgmap v6038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:37 smithi115 bash[45495]: cluster 2022-04-22T19:42:36.020780+0000 mgr.y (mgr.34107) 4689 : cluster [DBG] pgmap v6038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:38 smithi121 bash[36902]: audit 2022-04-22T19:42:37.361481+0000 mon.a (mon.0) 7553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:38 smithi115 bash[45495]: audit 2022-04-22T19:42:37.361481+0000 mon.a (mon.0) 7553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:38 smithi115 bash[46934]: audit 2022-04-22T19:42:37.361481+0000 mon.a (mon.0) 7553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:39 smithi121 bash[36902]: cluster 2022-04-22T19:42:38.022037+0000 mgr.y (mgr.34107) 4690 : cluster [DBG] pgmap v6039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:39 smithi115 bash[46934]: cluster 2022-04-22T19:42:38.022037+0000 mgr.y (mgr.34107) 4690 : cluster [DBG] pgmap v6039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:39 smithi115 bash[45495]: cluster 2022-04-22T19:42:38.022037+0000 mgr.y (mgr.34107) 4690 : cluster [DBG] pgmap v6039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:40 smithi121 bash[36902]: audit 2022-04-22T19:42:39.447736+0000 mon.a (mon.0) 7554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:40 smithi115 bash[46934]: audit 2022-04-22T19:42:39.447736+0000 mon.a (mon.0) 7554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:40 smithi115 bash[45495]: audit 2022-04-22T19:42:39.447736+0000 mon.a (mon.0) 7554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:41 smithi121 bash[36902]: cluster 2022-04-22T19:42:40.023144+0000 mgr.y (mgr.34107) 4691 : cluster [DBG] pgmap v6040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:41 smithi121 bash[36902]: audit 2022-04-22T19:42:40.891750+0000 mon.a (mon.0) 7555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:41 smithi115 bash[45495]: cluster 2022-04-22T19:42:40.023144+0000 mgr.y (mgr.34107) 4691 : cluster [DBG] pgmap v6040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:41 smithi115 bash[45495]: audit 2022-04-22T19:42:40.891750+0000 mon.a (mon.0) 7555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:41 smithi115 bash[46934]: cluster 2022-04-22T19:42:40.023144+0000 mgr.y (mgr.34107) 4691 : cluster [DBG] pgmap v6040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:41 smithi115 bash[46934]: audit 2022-04-22T19:42:40.891750+0000 mon.a (mon.0) 7555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:42.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:42 smithi115 bash[45495]: audit 2022-04-22T19:42:42.364004+0000 mon.a (mon.0) 7556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:42.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:42.142Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:42.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:42 smithi115 bash[46934]: audit 2022-04-22T19:42:42.364004+0000 mon.a (mon.0) 7556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:42 smithi121 bash[36902]: audit 2022-04-22T19:42:42.364004+0000 mon.a (mon.0) 7556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:43 smithi121 bash[36902]: cluster 2022-04-22T19:42:42.024312+0000 mgr.y (mgr.34107) 4692 : cluster [DBG] pgmap v6041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:43.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:43 smithi115 bash[45495]: cluster 2022-04-22T19:42:42.024312+0000 mgr.y (mgr.34107) 4692 : cluster [DBG] pgmap v6041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:43 smithi115 bash[46934]: cluster 2022-04-22T19:42:42.024312+0000 mgr.y (mgr.34107) 4692 : cluster [DBG] pgmap v6041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:44.010Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:45 smithi121 bash[36902]: cluster 2022-04-22T19:42:44.025381+0000 mgr.y (mgr.34107) 4693 : cluster [DBG] pgmap v6042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:45 smithi121 bash[36902]: audit 2022-04-22T19:42:44.451237+0000 mon.a (mon.0) 7557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:45 smithi115 bash[45495]: cluster 2022-04-22T19:42:44.025381+0000 mgr.y (mgr.34107) 4693 : cluster [DBG] pgmap v6042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:45 smithi115 bash[45495]: audit 2022-04-22T19:42:44.451237+0000 mon.a (mon.0) 7557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:45 smithi115 bash[46934]: cluster 2022-04-22T19:42:44.025381+0000 mgr.y (mgr.34107) 4693 : cluster [DBG] pgmap v6042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:45 smithi115 bash[46934]: audit 2022-04-22T19:42:44.451237+0000 mon.a (mon.0) 7557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:46 smithi121 bash[36902]: audit 2022-04-22T19:42:45.895025+0000 mon.a (mon.0) 7558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:46 smithi115 bash[46934]: audit 2022-04-22T19:42:45.895025+0000 mon.a (mon.0) 7558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:46 smithi115 bash[45495]: audit 2022-04-22T19:42:45.895025+0000 mon.a (mon.0) 7558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:47 smithi121 bash[36902]: cluster 2022-04-22T19:42:46.026313+0000 mgr.y (mgr.34107) 4694 : cluster [DBG] pgmap v6043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:47 smithi121 bash[36902]: audit 2022-04-22T19:42:47.366915+0000 mon.a (mon.0) 7559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:47 smithi115 bash[45495]: cluster 2022-04-22T19:42:46.026313+0000 mgr.y (mgr.34107) 4694 : cluster [DBG] pgmap v6043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:47 smithi115 bash[45495]: audit 2022-04-22T19:42:47.366915+0000 mon.a (mon.0) 7559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:47 smithi115 bash[46934]: cluster 2022-04-22T19:42:46.026313+0000 mgr.y (mgr.34107) 4694 : cluster [DBG] pgmap v6043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:47 smithi115 bash[46934]: audit 2022-04-22T19:42:47.366915+0000 mon.a (mon.0) 7559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:49 smithi121 bash[36902]: cluster 2022-04-22T19:42:48.027407+0000 mgr.y (mgr.34107) 4695 : cluster [DBG] pgmap v6044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:49 smithi115 bash[46934]: cluster 2022-04-22T19:42:48.027407+0000 mgr.y (mgr.34107) 4695 : cluster [DBG] pgmap v6044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:49 smithi115 bash[45495]: cluster 2022-04-22T19:42:48.027407+0000 mgr.y (mgr.34107) 4695 : cluster [DBG] pgmap v6044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:50 smithi121 bash[36902]: audit 2022-04-22T19:42:49.454791+0000 mon.a (mon.0) 7560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:50 smithi115 bash[46934]: audit 2022-04-22T19:42:49.454791+0000 mon.a (mon.0) 7560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:50.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:50 smithi115 bash[45495]: audit 2022-04-22T19:42:49.454791+0000 mon.a (mon.0) 7560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:51 smithi121 bash[36902]: cluster 2022-04-22T19:42:50.028474+0000 mgr.y (mgr.34107) 4696 : cluster [DBG] pgmap v6045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:51 smithi121 bash[36902]: audit 2022-04-22T19:42:50.898263+0000 mon.a (mon.0) 7561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:51 smithi115 bash[46934]: cluster 2022-04-22T19:42:50.028474+0000 mgr.y (mgr.34107) 4696 : cluster [DBG] pgmap v6045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:51 smithi115 bash[46934]: audit 2022-04-22T19:42:50.898263+0000 mon.a (mon.0) 7561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:51 smithi115 bash[45495]: cluster 2022-04-22T19:42:50.028474+0000 mgr.y (mgr.34107) 4696 : cluster [DBG] pgmap v6045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:51.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:51 smithi115 bash[45495]: audit 2022-04-22T19:42:50.898263+0000 mon.a (mon.0) 7561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:52.431 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:52.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:52 smithi121 bash[36902]: audit 2022-04-22T19:42:52.369585+0000 mon.a (mon.0) 7562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:52.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:52 smithi115 bash[46934]: audit 2022-04-22T19:42:52.369585+0000 mon.a (mon.0) 7562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:52.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:52 smithi115 bash[45495]: audit 2022-04-22T19:42:52.369585+0000 mon.a (mon.0) 7562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:53 smithi121 bash[36902]: cluster 2022-04-22T19:42:52.029675+0000 mgr.y (mgr.34107) 4697 : cluster [DBG] pgmap v6046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:53 smithi115 bash[45495]: cluster 2022-04-22T19:42:52.029675+0000 mgr.y (mgr.34107) 4697 : cluster [DBG] pgmap v6046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:53 smithi115 bash[46934]: cluster 2022-04-22T19:42:52.029675+0000 mgr.y (mgr.34107) 4697 : cluster [DBG] pgmap v6046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:42:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:42:54.012Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:42:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:55 smithi121 bash[36902]: cluster 2022-04-22T19:42:54.030801+0000 mgr.y (mgr.34107) 4698 : cluster [DBG] pgmap v6047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:55 smithi121 bash[36902]: audit 2022-04-22T19:42:54.458308+0000 mon.a (mon.0) 7563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:55 smithi115 bash[45495]: cluster 2022-04-22T19:42:54.030801+0000 mgr.y (mgr.34107) 4698 : cluster [DBG] pgmap v6047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:55 smithi115 bash[45495]: audit 2022-04-22T19:42:54.458308+0000 mon.a (mon.0) 7563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:55 smithi115 bash[46934]: cluster 2022-04-22T19:42:54.030801+0000 mgr.y (mgr.34107) 4698 : cluster [DBG] pgmap v6047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:55 smithi115 bash[46934]: audit 2022-04-22T19:42:54.458308+0000 mon.a (mon.0) 7563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:42:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:56 smithi121 bash[36902]: audit 2022-04-22T19:42:55.901219+0000 mon.a (mon.0) 7564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:56 smithi115 bash[45495]: audit 2022-04-22T19:42:55.901219+0000 mon.a (mon.0) 7564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:56 smithi115 bash[46934]: audit 2022-04-22T19:42:55.901219+0000 mon.a (mon.0) 7564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:42:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:57 smithi121 bash[36902]: cluster 2022-04-22T19:42:56.031835+0000 mgr.y (mgr.34107) 4699 : cluster [DBG] pgmap v6048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:57 smithi121 bash[36902]: audit 2022-04-22T19:42:57.372526+0000 mon.a (mon.0) 7565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:57 smithi115 bash[45495]: cluster 2022-04-22T19:42:56.031835+0000 mgr.y (mgr.34107) 4699 : cluster [DBG] pgmap v6048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:57 smithi115 bash[45495]: audit 2022-04-22T19:42:57.372526+0000 mon.a (mon.0) 7565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:57 smithi115 bash[46934]: cluster 2022-04-22T19:42:56.031835+0000 mgr.y (mgr.34107) 4699 : cluster [DBG] pgmap v6048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:57 smithi115 bash[46934]: audit 2022-04-22T19:42:57.372526+0000 mon.a (mon.0) 7565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:42:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:42:59 smithi121 bash[36902]: cluster 2022-04-22T19:42:58.032891+0000 mgr.y (mgr.34107) 4700 : cluster [DBG] pgmap v6049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:42:59 smithi115 bash[46934]: cluster 2022-04-22T19:42:58.032891+0000 mgr.y (mgr.34107) 4700 : cluster [DBG] pgmap v6049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:42:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:42:59 smithi115 bash[45495]: cluster 2022-04-22T19:42:58.032891+0000 mgr.y (mgr.34107) 4700 : cluster [DBG] pgmap v6049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:00 smithi121 bash[36902]: audit 2022-04-22T19:42:59.461728+0000 mon.a (mon.0) 7566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:00 smithi115 bash[45495]: audit 2022-04-22T19:42:59.461728+0000 mon.a (mon.0) 7566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:00 smithi115 bash[46934]: audit 2022-04-22T19:42:59.461728+0000 mon.a (mon.0) 7566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:01 smithi121 bash[36902]: cluster 2022-04-22T19:43:00.033956+0000 mgr.y (mgr.34107) 4701 : cluster [DBG] pgmap v6050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:01 smithi121 bash[36902]: audit 2022-04-22T19:43:00.904445+0000 mon.a (mon.0) 7567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:01 smithi115 bash[46934]: cluster 2022-04-22T19:43:00.033956+0000 mgr.y (mgr.34107) 4701 : cluster [DBG] pgmap v6050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:01 smithi115 bash[46934]: audit 2022-04-22T19:43:00.904445+0000 mon.a (mon.0) 7567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:01 smithi115 bash[45495]: cluster 2022-04-22T19:43:00.033956+0000 mgr.y (mgr.34107) 4701 : cluster [DBG] pgmap v6050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:01 smithi115 bash[45495]: audit 2022-04-22T19:43:00.904445+0000 mon.a (mon.0) 7567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:02.479 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:02.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:02 smithi121 bash[36902]: audit 2022-04-22T19:43:02.375465+0000 mon.a (mon.0) 7568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:02.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:02 smithi115 bash[45495]: audit 2022-04-22T19:43:02.375465+0000 mon.a (mon.0) 7568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:02.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:02 smithi115 bash[46934]: audit 2022-04-22T19:43:02.375465+0000 mon.a (mon.0) 7568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:03 smithi121 bash[36902]: cluster 2022-04-22T19:43:02.035087+0000 mgr.y (mgr.34107) 4702 : cluster [DBG] pgmap v6051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:03 smithi115 bash[45495]: cluster 2022-04-22T19:43:02.035087+0000 mgr.y (mgr.34107) 4702 : cluster [DBG] pgmap v6051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:03 smithi115 bash[46934]: cluster 2022-04-22T19:43:02.035087+0000 mgr.y (mgr.34107) 4702 : cluster [DBG] pgmap v6051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:04.012Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:04 smithi121 bash[36902]: audit 2022-04-22T19:43:04.465158+0000 mon.a (mon.0) 7569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:04.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:04 smithi115 bash[46934]: audit 2022-04-22T19:43:04.465158+0000 mon.a (mon.0) 7569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:04.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:04 smithi115 bash[45495]: audit 2022-04-22T19:43:04.465158+0000 mon.a (mon.0) 7569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:04.967 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:43:05.313 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:43:05.313 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 109s ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:43:05.313 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 109s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 109s ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 109s ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 109s ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 109s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:43:05.314 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 109s ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 109s ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 109s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 109s ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:43:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:43:05.721 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:43:05.721 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:43:05.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:43:05.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:05.722 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:43:05.722 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:43:05.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:05.723 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:43:05.723 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:43:05.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:05.723 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:43:05.724 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:43:05.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:05.724 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:43:05.724 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:43:05.724 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:43:05.725 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:43:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:05 smithi121 bash[36902]: cluster 2022-04-22T19:43:04.036194+0000 mgr.y (mgr.34107) 4703 : cluster [DBG] pgmap v6052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:05 smithi115 bash[45495]: cluster 2022-04-22T19:43:04.036194+0000 mgr.y (mgr.34107) 4703 : cluster [DBG] pgmap v6052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:05 smithi115 bash[46934]: cluster 2022-04-22T19:43:04.036194+0000 mgr.y (mgr.34107) 4703 : cluster [DBG] pgmap v6052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:06 smithi121 bash[36902]: audit 2022-04-22T19:43:04.939721+0000 mgr.y (mgr.34107) 4704 : audit [DBG] from='client.40302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:06 smithi121 bash[36902]: audit 2022-04-22T19:43:05.284189+0000 mgr.y (mgr.34107) 4705 : audit [DBG] from='client.40308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:06 smithi121 bash[36902]: audit 2022-04-22T19:43:05.710717+0000 mon.a (mon.0) 7570 : audit [DBG] from='client.? 172.21.15.115:0/3226776641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:06 smithi121 bash[36902]: audit 2022-04-22T19:43:05.907749+0000 mon.a (mon.0) 7571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[45495]: audit 2022-04-22T19:43:04.939721+0000 mgr.y (mgr.34107) 4704 : audit [DBG] from='client.40302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[45495]: audit 2022-04-22T19:43:05.284189+0000 mgr.y (mgr.34107) 4705 : audit [DBG] from='client.40308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[45495]: audit 2022-04-22T19:43:05.710717+0000 mon.a (mon.0) 7570 : audit [DBG] from='client.? 172.21.15.115:0/3226776641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[45495]: audit 2022-04-22T19:43:05.907749+0000 mon.a (mon.0) 7571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[46934]: audit 2022-04-22T19:43:04.939721+0000 mgr.y (mgr.34107) 4704 : audit [DBG] from='client.40302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[46934]: audit 2022-04-22T19:43:05.284189+0000 mgr.y (mgr.34107) 4705 : audit [DBG] from='client.40308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[46934]: audit 2022-04-22T19:43:05.710717+0000 mon.a (mon.0) 7570 : audit [DBG] from='client.? 172.21.15.115:0/3226776641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:06 smithi115 bash[46934]: audit 2022-04-22T19:43:05.907749+0000 mon.a (mon.0) 7571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:07 smithi121 bash[36902]: cluster 2022-04-22T19:43:06.037204+0000 mgr.y (mgr.34107) 4706 : cluster [DBG] pgmap v6053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:07 smithi121 bash[36902]: audit 2022-04-22T19:43:07.378505+0000 mon.a (mon.0) 7572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:07 smithi115 bash[45495]: cluster 2022-04-22T19:43:06.037204+0000 mgr.y (mgr.34107) 4706 : cluster [DBG] pgmap v6053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:07 smithi115 bash[45495]: audit 2022-04-22T19:43:07.378505+0000 mon.a (mon.0) 7572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:07.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:07 smithi115 bash[46934]: cluster 2022-04-22T19:43:06.037204+0000 mgr.y (mgr.34107) 4706 : cluster [DBG] pgmap v6053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:07.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:07 smithi115 bash[46934]: audit 2022-04-22T19:43:07.378505+0000 mon.a (mon.0) 7572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:09 smithi121 bash[36902]: cluster 2022-04-22T19:43:08.038288+0000 mgr.y (mgr.34107) 4707 : cluster [DBG] pgmap v6054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:09 smithi121 bash[36902]: audit 2022-04-22T19:43:09.468702+0000 mon.a (mon.0) 7573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:09 smithi115 bash[45495]: cluster 2022-04-22T19:43:08.038288+0000 mgr.y (mgr.34107) 4707 : cluster [DBG] pgmap v6054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:09.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:09 smithi115 bash[45495]: audit 2022-04-22T19:43:09.468702+0000 mon.a (mon.0) 7573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:09 smithi115 bash[46934]: cluster 2022-04-22T19:43:08.038288+0000 mgr.y (mgr.34107) 4707 : cluster [DBG] pgmap v6054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:09.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:09 smithi115 bash[46934]: audit 2022-04-22T19:43:09.468702+0000 mon.a (mon.0) 7573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:11 smithi115 bash[45495]: cluster 2022-04-22T19:43:10.039367+0000 mgr.y (mgr.34107) 4708 : cluster [DBG] pgmap v6055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:11 smithi115 bash[45495]: audit 2022-04-22T19:43:10.910876+0000 mon.a (mon.0) 7574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:11 smithi115 bash[46934]: cluster 2022-04-22T19:43:10.039367+0000 mgr.y (mgr.34107) 4708 : cluster [DBG] pgmap v6055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:11 smithi115 bash[46934]: audit 2022-04-22T19:43:10.910876+0000 mon.a (mon.0) 7574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:11 smithi121 bash[36902]: cluster 2022-04-22T19:43:10.039367+0000 mgr.y (mgr.34107) 4708 : cluster [DBG] pgmap v6055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:11 smithi121 bash[36902]: audit 2022-04-22T19:43:10.910876+0000 mon.a (mon.0) 7574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:12.525 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:12.145Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:12 smithi115 bash[45495]: audit 2022-04-22T19:43:12.381748+0000 mon.a (mon.0) 7575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:12.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:12 smithi115 bash[46934]: audit 2022-04-22T19:43:12.381748+0000 mon.a (mon.0) 7575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:12 smithi121 bash[36902]: audit 2022-04-22T19:43:12.381748+0000 mon.a (mon.0) 7575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:13 smithi115 bash[45495]: cluster 2022-04-22T19:43:12.040482+0000 mgr.y (mgr.34107) 4709 : cluster [DBG] pgmap v6056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:13 smithi115 bash[46934]: cluster 2022-04-22T19:43:12.040482+0000 mgr.y (mgr.34107) 4709 : cluster [DBG] pgmap v6056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:13 smithi121 bash[36902]: cluster 2022-04-22T19:43:12.040482+0000 mgr.y (mgr.34107) 4709 : cluster [DBG] pgmap v6056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:14.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:14.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:14 smithi115 bash[45495]: audit 2022-04-22T19:43:14.472275+0000 mon.a (mon.0) 7576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:14.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:14 smithi115 bash[46934]: audit 2022-04-22T19:43:14.472275+0000 mon.a (mon.0) 7576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:14 smithi121 bash[36902]: audit 2022-04-22T19:43:14.472275+0000 mon.a (mon.0) 7576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[45495]: cluster 2022-04-22T19:43:14.041581+0000 mgr.y (mgr.34107) 4710 : cluster [DBG] pgmap v6057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[45495]: audit 2022-04-22T19:43:14.848527+0000 mon.a (mon.0) 7577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:43:15.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[45495]: audit 2022-04-22T19:43:14.849400+0000 mon.a (mon.0) 7578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:43:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[46934]: cluster 2022-04-22T19:43:14.041581+0000 mgr.y (mgr.34107) 4710 : cluster [DBG] pgmap v6057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[46934]: audit 2022-04-22T19:43:14.848527+0000 mon.a (mon.0) 7577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:43:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:15 smithi115 bash[46934]: audit 2022-04-22T19:43:14.849400+0000 mon.a (mon.0) 7578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:43:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:15 smithi121 bash[36902]: cluster 2022-04-22T19:43:14.041581+0000 mgr.y (mgr.34107) 4710 : cluster [DBG] pgmap v6057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:15 smithi121 bash[36902]: audit 2022-04-22T19:43:14.848527+0000 mon.a (mon.0) 7577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:43:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:15 smithi121 bash[36902]: audit 2022-04-22T19:43:14.849400+0000 mon.a (mon.0) 7578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:43:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:16 smithi121 bash[36902]: audit 2022-04-22T19:43:15.914078+0000 mon.a (mon.0) 7579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:16 smithi115 bash[45495]: audit 2022-04-22T19:43:15.914078+0000 mon.a (mon.0) 7579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:17.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:16 smithi115 bash[46934]: audit 2022-04-22T19:43:15.914078+0000 mon.a (mon.0) 7579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:17 smithi121 bash[36902]: cluster 2022-04-22T19:43:16.042635+0000 mgr.y (mgr.34107) 4711 : cluster [DBG] pgmap v6058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:17 smithi121 bash[36902]: audit 2022-04-22T19:43:17.386458+0000 mon.a (mon.0) 7580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:17 smithi115 bash[45495]: cluster 2022-04-22T19:43:16.042635+0000 mgr.y (mgr.34107) 4711 : cluster [DBG] pgmap v6058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:17 smithi115 bash[45495]: audit 2022-04-22T19:43:17.386458+0000 mon.a (mon.0) 7580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:17 smithi115 bash[46934]: cluster 2022-04-22T19:43:16.042635+0000 mgr.y (mgr.34107) 4711 : cluster [DBG] pgmap v6058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:17 smithi115 bash[46934]: audit 2022-04-22T19:43:17.386458+0000 mon.a (mon.0) 7580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:19 smithi121 bash[36902]: cluster 2022-04-22T19:43:18.043776+0000 mgr.y (mgr.34107) 4712 : cluster [DBG] pgmap v6059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:19 smithi121 bash[36902]: audit 2022-04-22T19:43:19.474917+0000 mon.a (mon.0) 7581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:19 smithi115 bash[45495]: cluster 2022-04-22T19:43:18.043776+0000 mgr.y (mgr.34107) 4712 : cluster [DBG] pgmap v6059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:20.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:19 smithi115 bash[45495]: audit 2022-04-22T19:43:19.474917+0000 mon.a (mon.0) 7581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:19 smithi115 bash[46934]: cluster 2022-04-22T19:43:18.043776+0000 mgr.y (mgr.34107) 4712 : cluster [DBG] pgmap v6059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:19 smithi115 bash[46934]: audit 2022-04-22T19:43:19.474917+0000 mon.a (mon.0) 7581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:21 smithi121 bash[36902]: cluster 2022-04-22T19:43:20.044799+0000 mgr.y (mgr.34107) 4713 : cluster [DBG] pgmap v6060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:21 smithi121 bash[36902]: audit 2022-04-22T19:43:20.917030+0000 mon.a (mon.0) 7582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:21 smithi115 bash[45495]: cluster 2022-04-22T19:43:20.044799+0000 mgr.y (mgr.34107) 4713 : cluster [DBG] pgmap v6060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:21 smithi115 bash[45495]: audit 2022-04-22T19:43:20.917030+0000 mon.a (mon.0) 7582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:21 smithi115 bash[46934]: cluster 2022-04-22T19:43:20.044799+0000 mgr.y (mgr.34107) 4713 : cluster [DBG] pgmap v6060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:21 smithi115 bash[46934]: audit 2022-04-22T19:43:20.917030+0000 mon.a (mon.0) 7582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:22.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:22 smithi121 bash[36902]: audit 2022-04-22T19:43:22.389635+0000 mon.a (mon.0) 7583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:22 smithi115 bash[45495]: audit 2022-04-22T19:43:22.389635+0000 mon.a (mon.0) 7583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:22 smithi115 bash[46934]: audit 2022-04-22T19:43:22.389635+0000 mon.a (mon.0) 7583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:23 smithi121 bash[36902]: cluster 2022-04-22T19:43:22.045958+0000 mgr.y (mgr.34107) 4714 : cluster [DBG] pgmap v6061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:24.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:23 smithi115 bash[45495]: cluster 2022-04-22T19:43:22.045958+0000 mgr.y (mgr.34107) 4714 : cluster [DBG] pgmap v6061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:24.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:23 smithi115 bash[46934]: cluster 2022-04-22T19:43:22.045958+0000 mgr.y (mgr.34107) 4714 : cluster [DBG] pgmap v6061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:24.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:24 smithi121 bash[36902]: audit 2022-04-22T19:43:24.478777+0000 mon.a (mon.0) 7584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:24 smithi115 bash[45495]: audit 2022-04-22T19:43:24.478777+0000 mon.a (mon.0) 7584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:24 smithi115 bash[46934]: audit 2022-04-22T19:43:24.478777+0000 mon.a (mon.0) 7584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:25 smithi121 bash[36902]: cluster 2022-04-22T19:43:24.047126+0000 mgr.y (mgr.34107) 4715 : cluster [DBG] pgmap v6062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:26.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:25 smithi115 bash[45495]: cluster 2022-04-22T19:43:24.047126+0000 mgr.y (mgr.34107) 4715 : cluster [DBG] pgmap v6062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:26.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:25 smithi115 bash[46934]: cluster 2022-04-22T19:43:24.047126+0000 mgr.y (mgr.34107) 4715 : cluster [DBG] pgmap v6062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:26 smithi121 bash[36902]: audit 2022-04-22T19:43:25.920262+0000 mon.a (mon.0) 7585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:26 smithi121 bash[36902]: audit 2022-04-22T19:43:26.004870+0000 mon.a (mon.0) 7586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:43:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:26 smithi121 bash[36902]: audit 2022-04-22T19:43:26.006539+0000 mon.a (mon.0) 7587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:43:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:26 smithi121 bash[36902]: audit 2022-04-22T19:43:26.007700+0000 mon.a (mon.0) 7588 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:43:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:26 smithi121 bash[36902]: audit 2022-04-22T19:43:26.336965+0000 mon.a (mon.0) 7589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:43:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[45495]: audit 2022-04-22T19:43:25.920262+0000 mon.a (mon.0) 7585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[45495]: audit 2022-04-22T19:43:26.004870+0000 mon.a (mon.0) 7586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:43:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[45495]: audit 2022-04-22T19:43:26.006539+0000 mon.a (mon.0) 7587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[45495]: audit 2022-04-22T19:43:26.007700+0000 mon.a (mon.0) 7588 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[45495]: audit 2022-04-22T19:43:26.336965+0000 mon.a (mon.0) 7589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[46934]: audit 2022-04-22T19:43:25.920262+0000 mon.a (mon.0) 7585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[46934]: audit 2022-04-22T19:43:26.004870+0000 mon.a (mon.0) 7586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[46934]: audit 2022-04-22T19:43:26.006539+0000 mon.a (mon.0) 7587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[46934]: audit 2022-04-22T19:43:26.007700+0000 mon.a (mon.0) 7588 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:43:27.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:26 smithi115 bash[46934]: audit 2022-04-22T19:43:26.336965+0000 mon.a (mon.0) 7589 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:43:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:27 smithi121 bash[36902]: cluster 2022-04-22T19:43:26.048080+0000 mgr.y (mgr.34107) 4716 : cluster [DBG] pgmap v6063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:27 smithi121 bash[36902]: audit 2022-04-22T19:43:27.392563+0000 mon.a (mon.0) 7590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:27 smithi115 bash[45495]: cluster 2022-04-22T19:43:26.048080+0000 mgr.y (mgr.34107) 4716 : cluster [DBG] pgmap v6063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:27 smithi115 bash[45495]: audit 2022-04-22T19:43:27.392563+0000 mon.a (mon.0) 7590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:27 smithi115 bash[46934]: cluster 2022-04-22T19:43:26.048080+0000 mgr.y (mgr.34107) 4716 : cluster [DBG] pgmap v6063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:27 smithi115 bash[46934]: audit 2022-04-22T19:43:27.392563+0000 mon.a (mon.0) 7590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:29 smithi121 bash[36902]: cluster 2022-04-22T19:43:28.049245+0000 mgr.y (mgr.34107) 4717 : cluster [DBG] pgmap v6064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:29 smithi121 bash[36902]: audit 2022-04-22T19:43:29.482562+0000 mon.a (mon.0) 7591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:29 smithi115 bash[45495]: cluster 2022-04-22T19:43:28.049245+0000 mgr.y (mgr.34107) 4717 : cluster [DBG] pgmap v6064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:29 smithi115 bash[45495]: audit 2022-04-22T19:43:29.482562+0000 mon.a (mon.0) 7591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:29 smithi115 bash[46934]: cluster 2022-04-22T19:43:28.049245+0000 mgr.y (mgr.34107) 4717 : cluster [DBG] pgmap v6064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:29 smithi115 bash[46934]: audit 2022-04-22T19:43:29.482562+0000 mon.a (mon.0) 7591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:31 smithi121 bash[36902]: cluster 2022-04-22T19:43:30.050364+0000 mgr.y (mgr.34107) 4718 : cluster [DBG] pgmap v6065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:31 smithi121 bash[36902]: audit 2022-04-22T19:43:30.923443+0000 mon.a (mon.0) 7592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:31 smithi115 bash[45495]: cluster 2022-04-22T19:43:30.050364+0000 mgr.y (mgr.34107) 4718 : cluster [DBG] pgmap v6065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:32.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:31 smithi115 bash[45495]: audit 2022-04-22T19:43:30.923443+0000 mon.a (mon.0) 7592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:31 smithi115 bash[46934]: cluster 2022-04-22T19:43:30.050364+0000 mgr.y (mgr.34107) 4718 : cluster [DBG] pgmap v6065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:32.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:31 smithi115 bash[46934]: audit 2022-04-22T19:43:30.923443+0000 mon.a (mon.0) 7592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:32.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:32 smithi121 bash[36902]: audit 2022-04-22T19:43:32.395737+0000 mon.a (mon.0) 7593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:32 smithi115 bash[45495]: audit 2022-04-22T19:43:32.395737+0000 mon.a (mon.0) 7593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:32 smithi115 bash[46934]: audit 2022-04-22T19:43:32.395737+0000 mon.a (mon.0) 7593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:33 smithi121 bash[36902]: cluster 2022-04-22T19:43:32.051532+0000 mgr.y (mgr.34107) 4719 : cluster [DBG] pgmap v6066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:34.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:33 smithi115 bash[45495]: cluster 2022-04-22T19:43:32.051532+0000 mgr.y (mgr.34107) 4719 : cluster [DBG] pgmap v6066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:34.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:33 smithi115 bash[46934]: cluster 2022-04-22T19:43:32.051532+0000 mgr.y (mgr.34107) 4719 : cluster [DBG] pgmap v6066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:34.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:34 smithi121 bash[36902]: audit 2022-04-22T19:43:34.486197+0000 mon.a (mon.0) 7594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:34 smithi115 bash[45495]: audit 2022-04-22T19:43:34.486197+0000 mon.a (mon.0) 7594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:34 smithi115 bash[46934]: audit 2022-04-22T19:43:34.486197+0000 mon.a (mon.0) 7594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:35 smithi121 bash[36902]: cluster 2022-04-22T19:43:34.052591+0000 mgr.y (mgr.34107) 4720 : cluster [DBG] pgmap v6067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:35 smithi115 bash[45495]: cluster 2022-04-22T19:43:34.052591+0000 mgr.y (mgr.34107) 4720 : cluster [DBG] pgmap v6067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:35 smithi115 bash[46934]: cluster 2022-04-22T19:43:34.052591+0000 mgr.y (mgr.34107) 4720 : cluster [DBG] pgmap v6067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:36.101 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:43:36.452 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:43:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:43:36.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:43:36.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:43:36.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:43:36.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:43:36.455 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:43:36.860 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:43:36.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:43:36.860 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:43:36.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:43:36.862 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:43:36.862 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:43:36.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:43:36.862 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:43:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:36 smithi121 bash[36902]: audit 2022-04-22T19:43:35.926525+0000 mon.a (mon.0) 7595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:36 smithi115 bash[45495]: audit 2022-04-22T19:43:35.926525+0000 mon.a (mon.0) 7595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:36 smithi115 bash[46934]: audit 2022-04-22T19:43:35.926525+0000 mon.a (mon.0) 7595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:37 smithi121 bash[36902]: cluster 2022-04-22T19:43:36.053492+0000 mgr.y (mgr.34107) 4721 : cluster [DBG] pgmap v6068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:37 smithi121 bash[36902]: audit 2022-04-22T19:43:36.075069+0000 mgr.y (mgr.34107) 4722 : audit [DBG] from='client.40320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:37 smithi121 bash[36902]: audit 2022-04-22T19:43:36.423753+0000 mgr.y (mgr.34107) 4723 : audit [DBG] from='client.50260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:37 smithi121 bash[36902]: audit 2022-04-22T19:43:36.850242+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.115:0/4287205525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:38.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:37 smithi121 bash[36902]: audit 2022-04-22T19:43:37.398977+0000 mon.a (mon.0) 7596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[45495]: cluster 2022-04-22T19:43:36.053492+0000 mgr.y (mgr.34107) 4721 : cluster [DBG] pgmap v6068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[45495]: audit 2022-04-22T19:43:36.075069+0000 mgr.y (mgr.34107) 4722 : audit [DBG] from='client.40320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[45495]: audit 2022-04-22T19:43:36.423753+0000 mgr.y (mgr.34107) 4723 : audit [DBG] from='client.50260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[45495]: audit 2022-04-22T19:43:36.850242+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.115:0/4287205525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[45495]: audit 2022-04-22T19:43:37.398977+0000 mon.a (mon.0) 7596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[46934]: cluster 2022-04-22T19:43:36.053492+0000 mgr.y (mgr.34107) 4721 : cluster [DBG] pgmap v6068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[46934]: audit 2022-04-22T19:43:36.075069+0000 mgr.y (mgr.34107) 4722 : audit [DBG] from='client.40320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[46934]: audit 2022-04-22T19:43:36.423753+0000 mgr.y (mgr.34107) 4723 : audit [DBG] from='client.50260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:43:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[46934]: audit 2022-04-22T19:43:36.850242+0000 mon.c (mon.1) 205 : audit [DBG] from='client.? 172.21.15.115:0/4287205525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:43:38.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:37 smithi115 bash[46934]: audit 2022-04-22T19:43:37.398977+0000 mon.a (mon.0) 7596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:39 smithi121 bash[36902]: cluster 2022-04-22T19:43:38.054644+0000 mgr.y (mgr.34107) 4724 : cluster [DBG] pgmap v6069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:40.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:39 smithi121 bash[36902]: audit 2022-04-22T19:43:39.489919+0000 mon.a (mon.0) 7597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:39 smithi115 bash[45495]: cluster 2022-04-22T19:43:38.054644+0000 mgr.y (mgr.34107) 4724 : cluster [DBG] pgmap v6069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:39 smithi115 bash[45495]: audit 2022-04-22T19:43:39.489919+0000 mon.a (mon.0) 7597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:40.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:39 smithi115 bash[46934]: cluster 2022-04-22T19:43:38.054644+0000 mgr.y (mgr.34107) 4724 : cluster [DBG] pgmap v6069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:40.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:39 smithi115 bash[46934]: audit 2022-04-22T19:43:39.489919+0000 mon.a (mon.0) 7597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:41 smithi121 bash[36902]: cluster 2022-04-22T19:43:40.055832+0000 mgr.y (mgr.34107) 4725 : cluster [DBG] pgmap v6070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:41 smithi121 bash[36902]: audit 2022-04-22T19:43:40.929238+0000 mon.a (mon.0) 7598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:41 smithi115 bash[45495]: cluster 2022-04-22T19:43:40.055832+0000 mgr.y (mgr.34107) 4725 : cluster [DBG] pgmap v6070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:41 smithi115 bash[45495]: audit 2022-04-22T19:43:40.929238+0000 mon.a (mon.0) 7598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:41 smithi115 bash[46934]: cluster 2022-04-22T19:43:40.055832+0000 mgr.y (mgr.34107) 4725 : cluster [DBG] pgmap v6070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:41 smithi115 bash[46934]: audit 2022-04-22T19:43:40.929238+0000 mon.a (mon.0) 7598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:42.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:42 smithi121 bash[36902]: audit 2022-04-22T19:43:42.401946+0000 mon.a (mon.0) 7599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:42 smithi115 bash[45495]: audit 2022-04-22T19:43:42.401946+0000 mon.a (mon.0) 7599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:42 smithi115 bash[46934]: audit 2022-04-22T19:43:42.401946+0000 mon.a (mon.0) 7599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:43 smithi121 bash[36902]: cluster 2022-04-22T19:43:42.056898+0000 mgr.y (mgr.34107) 4726 : cluster [DBG] pgmap v6071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:44.026 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:43 smithi115 bash[45495]: cluster 2022-04-22T19:43:42.056898+0000 mgr.y (mgr.34107) 4726 : cluster [DBG] pgmap v6071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:44.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:43 smithi115 bash[46934]: cluster 2022-04-22T19:43:42.056898+0000 mgr.y (mgr.34107) 4726 : cluster [DBG] pgmap v6071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:44.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:44 smithi121 bash[36902]: audit 2022-04-22T19:43:44.493744+0000 mon.a (mon.0) 7600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:44 smithi115 bash[45495]: audit 2022-04-22T19:43:44.493744+0000 mon.a (mon.0) 7600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:44 smithi115 bash[46934]: audit 2022-04-22T19:43:44.493744+0000 mon.a (mon.0) 7600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:45 smithi121 bash[36902]: cluster 2022-04-22T19:43:44.057964+0000 mgr.y (mgr.34107) 4727 : cluster [DBG] pgmap v6072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:45 smithi115 bash[45495]: cluster 2022-04-22T19:43:44.057964+0000 mgr.y (mgr.34107) 4727 : cluster [DBG] pgmap v6072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:45 smithi115 bash[46934]: cluster 2022-04-22T19:43:44.057964+0000 mgr.y (mgr.34107) 4727 : cluster [DBG] pgmap v6072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:46 smithi121 bash[36902]: audit 2022-04-22T19:43:45.931991+0000 mon.a (mon.0) 7601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:46 smithi121 bash[36902]: cluster 2022-04-22T19:43:46.058982+0000 mgr.y (mgr.34107) 4728 : cluster [DBG] pgmap v6073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:47.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:46 smithi115 bash[45495]: audit 2022-04-22T19:43:45.931991+0000 mon.a (mon.0) 7601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:47.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:46 smithi115 bash[45495]: cluster 2022-04-22T19:43:46.058982+0000 mgr.y (mgr.34107) 4728 : cluster [DBG] pgmap v6073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:46 smithi115 bash[46934]: audit 2022-04-22T19:43:45.931991+0000 mon.a (mon.0) 7601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:47.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:46 smithi115 bash[46934]: cluster 2022-04-22T19:43:46.058982+0000 mgr.y (mgr.34107) 4728 : cluster [DBG] pgmap v6073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:47 smithi121 bash[36902]: audit 2022-04-22T19:43:47.405023+0000 mon.a (mon.0) 7602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:47 smithi115 bash[45495]: audit 2022-04-22T19:43:47.405023+0000 mon.a (mon.0) 7602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:48.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:47 smithi115 bash[46934]: audit 2022-04-22T19:43:47.405023+0000 mon.a (mon.0) 7602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:49.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:49 smithi121 bash[36902]: cluster 2022-04-22T19:43:48.060148+0000 mgr.y (mgr.34107) 4729 : cluster [DBG] pgmap v6074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:49.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:49 smithi115 bash[45495]: cluster 2022-04-22T19:43:48.060148+0000 mgr.y (mgr.34107) 4729 : cluster [DBG] pgmap v6074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:49.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:49 smithi115 bash[46934]: cluster 2022-04-22T19:43:48.060148+0000 mgr.y (mgr.34107) 4729 : cluster [DBG] pgmap v6074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:50 smithi121 bash[36902]: audit 2022-04-22T19:43:49.497421+0000 mon.a (mon.0) 7603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:50.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:50 smithi115 bash[45495]: audit 2022-04-22T19:43:49.497421+0000 mon.a (mon.0) 7603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:50.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:50 smithi115 bash[46934]: audit 2022-04-22T19:43:49.497421+0000 mon.a (mon.0) 7603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:51 smithi121 bash[36902]: cluster 2022-04-22T19:43:50.061225+0000 mgr.y (mgr.34107) 4730 : cluster [DBG] pgmap v6075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:51 smithi121 bash[36902]: audit 2022-04-22T19:43:50.934913+0000 mon.a (mon.0) 7604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:51 smithi115 bash[45495]: cluster 2022-04-22T19:43:50.061225+0000 mgr.y (mgr.34107) 4730 : cluster [DBG] pgmap v6075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:51.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:51 smithi115 bash[45495]: audit 2022-04-22T19:43:50.934913+0000 mon.a (mon.0) 7604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:51 smithi115 bash[46934]: cluster 2022-04-22T19:43:50.061225+0000 mgr.y (mgr.34107) 4730 : cluster [DBG] pgmap v6075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:51.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:51 smithi115 bash[46934]: audit 2022-04-22T19:43:50.934913+0000 mon.a (mon.0) 7604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:52.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:52.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:53 smithi121 bash[36902]: cluster 2022-04-22T19:43:52.062392+0000 mgr.y (mgr.34107) 4731 : cluster [DBG] pgmap v6076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:53 smithi121 bash[36902]: audit 2022-04-22T19:43:52.408110+0000 mon.a (mon.0) 7605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:53 smithi115 bash[45495]: cluster 2022-04-22T19:43:52.062392+0000 mgr.y (mgr.34107) 4731 : cluster [DBG] pgmap v6076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:53.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:53 smithi115 bash[45495]: audit 2022-04-22T19:43:52.408110+0000 mon.a (mon.0) 7605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:53 smithi115 bash[46934]: cluster 2022-04-22T19:43:52.062392+0000 mgr.y (mgr.34107) 4731 : cluster [DBG] pgmap v6076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:53.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:53 smithi115 bash[46934]: audit 2022-04-22T19:43:52.408110+0000 mon.a (mon.0) 7605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:43:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:43:54.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:43:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:55 smithi121 bash[36902]: cluster 2022-04-22T19:43:54.063517+0000 mgr.y (mgr.34107) 4732 : cluster [DBG] pgmap v6077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:55 smithi121 bash[36902]: audit 2022-04-22T19:43:54.500888+0000 mon.a (mon.0) 7606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:55 smithi115 bash[45495]: cluster 2022-04-22T19:43:54.063517+0000 mgr.y (mgr.34107) 4732 : cluster [DBG] pgmap v6077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:55 smithi115 bash[45495]: audit 2022-04-22T19:43:54.500888+0000 mon.a (mon.0) 7606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:55 smithi115 bash[46934]: cluster 2022-04-22T19:43:54.063517+0000 mgr.y (mgr.34107) 4732 : cluster [DBG] pgmap v6077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:55.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:55 smithi115 bash[46934]: audit 2022-04-22T19:43:54.500888+0000 mon.a (mon.0) 7606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:43:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:56 smithi121 bash[36902]: audit 2022-04-22T19:43:55.938082+0000 mon.a (mon.0) 7607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:56.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:56 smithi115 bash[45495]: audit 2022-04-22T19:43:55.938082+0000 mon.a (mon.0) 7607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:56.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:56 smithi115 bash[46934]: audit 2022-04-22T19:43:55.938082+0000 mon.a (mon.0) 7607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:43:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:57 smithi121 bash[36902]: cluster 2022-04-22T19:43:56.064505+0000 mgr.y (mgr.34107) 4733 : cluster [DBG] pgmap v6078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:57.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:57 smithi115 bash[45495]: cluster 2022-04-22T19:43:56.064505+0000 mgr.y (mgr.34107) 4733 : cluster [DBG] pgmap v6078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:57.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:57 smithi115 bash[46934]: cluster 2022-04-22T19:43:56.064505+0000 mgr.y (mgr.34107) 4733 : cluster [DBG] pgmap v6078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:58 smithi121 bash[36902]: audit 2022-04-22T19:43:57.410713+0000 mon.a (mon.0) 7608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:58.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:58 smithi115 bash[45495]: audit 2022-04-22T19:43:57.410713+0000 mon.a (mon.0) 7608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:58.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:58 smithi115 bash[46934]: audit 2022-04-22T19:43:57.410713+0000 mon.a (mon.0) 7608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:43:59 smithi121 bash[36902]: cluster 2022-04-22T19:43:58.065638+0000 mgr.y (mgr.34107) 4734 : cluster [DBG] pgmap v6079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:43:59 smithi115 bash[45495]: cluster 2022-04-22T19:43:58.065638+0000 mgr.y (mgr.34107) 4734 : cluster [DBG] pgmap v6079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:43:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:43:59 smithi115 bash[46934]: cluster 2022-04-22T19:43:58.065638+0000 mgr.y (mgr.34107) 4734 : cluster [DBG] pgmap v6079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:00 smithi121 bash[36902]: audit 2022-04-22T19:43:59.504466+0000 mon.a (mon.0) 7609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:00.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:00 smithi115 bash[46934]: audit 2022-04-22T19:43:59.504466+0000 mon.a (mon.0) 7609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:00 smithi115 bash[45495]: audit 2022-04-22T19:43:59.504466+0000 mon.a (mon.0) 7609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:01 smithi121 bash[36902]: cluster 2022-04-22T19:44:00.066807+0000 mgr.y (mgr.34107) 4735 : cluster [DBG] pgmap v6080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:01 smithi121 bash[36902]: audit 2022-04-22T19:44:00.941165+0000 mon.a (mon.0) 7610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:01 smithi115 bash[45495]: cluster 2022-04-22T19:44:00.066807+0000 mgr.y (mgr.34107) 4735 : cluster [DBG] pgmap v6080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:01 smithi115 bash[45495]: audit 2022-04-22T19:44:00.941165+0000 mon.a (mon.0) 7610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:01 smithi115 bash[46934]: cluster 2022-04-22T19:44:00.066807+0000 mgr.y (mgr.34107) 4735 : cluster [DBG] pgmap v6080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:01 smithi115 bash[46934]: audit 2022-04-22T19:44:00.941165+0000 mon.a (mon.0) 7610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:02.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:03 smithi121 bash[36902]: cluster 2022-04-22T19:44:02.067934+0000 mgr.y (mgr.34107) 4736 : cluster [DBG] pgmap v6081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:03 smithi121 bash[36902]: audit 2022-04-22T19:44:02.413969+0000 mon.a (mon.0) 7611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:03 smithi115 bash[45495]: cluster 2022-04-22T19:44:02.067934+0000 mgr.y (mgr.34107) 4736 : cluster [DBG] pgmap v6081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:03 smithi115 bash[45495]: audit 2022-04-22T19:44:02.413969+0000 mon.a (mon.0) 7611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:03 smithi115 bash[46934]: cluster 2022-04-22T19:44:02.067934+0000 mgr.y (mgr.34107) 4736 : cluster [DBG] pgmap v6081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:03 smithi115 bash[46934]: audit 2022-04-22T19:44:02.413969+0000 mon.a (mon.0) 7611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:04.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:05 smithi121 bash[36902]: cluster 2022-04-22T19:44:04.069166+0000 mgr.y (mgr.34107) 4737 : cluster [DBG] pgmap v6082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:05 smithi121 bash[36902]: audit 2022-04-22T19:44:04.508301+0000 mon.a (mon.0) 7612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:05 smithi115 bash[45495]: cluster 2022-04-22T19:44:04.069166+0000 mgr.y (mgr.34107) 4737 : cluster [DBG] pgmap v6082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:05.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:05 smithi115 bash[45495]: audit 2022-04-22T19:44:04.508301+0000 mon.a (mon.0) 7612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:05.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:05 smithi115 bash[46934]: cluster 2022-04-22T19:44:04.069166+0000 mgr.y (mgr.34107) 4737 : cluster [DBG] pgmap v6082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:05.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:05 smithi115 bash[46934]: audit 2022-04-22T19:44:04.508301+0000 mon.a (mon.0) 7612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:06 smithi121 bash[36902]: audit 2022-04-22T19:44:05.944355+0000 mon.a (mon.0) 7613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:06 smithi115 bash[45495]: audit 2022-04-22T19:44:05.944355+0000 mon.a (mon.0) 7613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:06 smithi115 bash[46934]: audit 2022-04-22T19:44:05.944355+0000 mon.a (mon.0) 7613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:07.233 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:44:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:07 smithi121 bash[36902]: cluster 2022-04-22T19:44:06.070304+0000 mgr.y (mgr.34107) 4738 : cluster [DBG] pgmap v6083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:07 smithi115 bash[45495]: cluster 2022-04-22T19:44:06.070304+0000 mgr.y (mgr.34107) 4738 : cluster [DBG] pgmap v6083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:07.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:07 smithi115 bash[46934]: cluster 2022-04-22T19:44:06.070304+0000 mgr.y (mgr.34107) 4738 : cluster [DBG] pgmap v6083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:44:07.991 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:07.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:44:07.993 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:44:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:08 smithi121 bash[36902]: audit 2022-04-22T19:44:07.208016+0000 mgr.y (mgr.34107) 4739 : audit [DBG] from='client.40338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:08 smithi121 bash[36902]: audit 2022-04-22T19:44:07.417036+0000 mon.a (mon.0) 7614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:08 smithi121 bash[36902]: audit 2022-04-22T19:44:07.556107+0000 mgr.y (mgr.34107) 4740 : audit [DBG] from='client.40344 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:08 smithi121 bash[36902]: audit 2022-04-22T19:44:07.981232+0000 mon.a (mon.0) 7615 : audit [DBG] from='client.? 172.21.15.115:0/2975918998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[45495]: audit 2022-04-22T19:44:07.208016+0000 mgr.y (mgr.34107) 4739 : audit [DBG] from='client.40338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[45495]: audit 2022-04-22T19:44:07.417036+0000 mon.a (mon.0) 7614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[45495]: audit 2022-04-22T19:44:07.556107+0000 mgr.y (mgr.34107) 4740 : audit [DBG] from='client.40344 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[45495]: audit 2022-04-22T19:44:07.981232+0000 mon.a (mon.0) 7615 : audit [DBG] from='client.? 172.21.15.115:0/2975918998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:08.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[46934]: audit 2022-04-22T19:44:07.208016+0000 mgr.y (mgr.34107) 4739 : audit [DBG] from='client.40338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[46934]: audit 2022-04-22T19:44:07.417036+0000 mon.a (mon.0) 7614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[46934]: audit 2022-04-22T19:44:07.556107+0000 mgr.y (mgr.34107) 4740 : audit [DBG] from='client.40344 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:08.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:08 smithi115 bash[46934]: audit 2022-04-22T19:44:07.981232+0000 mon.a (mon.0) 7615 : audit [DBG] from='client.? 172.21.15.115:0/2975918998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:09 smithi121 bash[36902]: cluster 2022-04-22T19:44:08.071503+0000 mgr.y (mgr.34107) 4741 : cluster [DBG] pgmap v6084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:09 smithi115 bash[45495]: cluster 2022-04-22T19:44:08.071503+0000 mgr.y (mgr.34107) 4741 : cluster [DBG] pgmap v6084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:09 smithi115 bash[46934]: cluster 2022-04-22T19:44:08.071503+0000 mgr.y (mgr.34107) 4741 : cluster [DBG] pgmap v6084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:10 smithi121 bash[36902]: audit 2022-04-22T19:44:09.511942+0000 mon.a (mon.0) 7616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:10 smithi115 bash[45495]: audit 2022-04-22T19:44:09.511942+0000 mon.a (mon.0) 7616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:10.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:10 smithi115 bash[46934]: audit 2022-04-22T19:44:09.511942+0000 mon.a (mon.0) 7616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:11 smithi115 bash[45495]: cluster 2022-04-22T19:44:10.072675+0000 mgr.y (mgr.34107) 4742 : cluster [DBG] pgmap v6085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:11 smithi115 bash[45495]: audit 2022-04-22T19:44:10.947636+0000 mon.a (mon.0) 7617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:11 smithi115 bash[46934]: cluster 2022-04-22T19:44:10.072675+0000 mgr.y (mgr.34107) 4742 : cluster [DBG] pgmap v6085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:11 smithi115 bash[46934]: audit 2022-04-22T19:44:10.947636+0000 mon.a (mon.0) 7617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:11 smithi121 bash[36902]: cluster 2022-04-22T19:44:10.072675+0000 mgr.y (mgr.34107) 4742 : cluster [DBG] pgmap v6085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:11 smithi121 bash[36902]: audit 2022-04-22T19:44:10.947636+0000 mon.a (mon.0) 7617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:12.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:13 smithi115 bash[46934]: cluster 2022-04-22T19:44:12.074009+0000 mgr.y (mgr.34107) 4743 : cluster [DBG] pgmap v6086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:13 smithi115 bash[46934]: audit 2022-04-22T19:44:12.419647+0000 mon.a (mon.0) 7618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:13 smithi115 bash[45495]: cluster 2022-04-22T19:44:12.074009+0000 mgr.y (mgr.34107) 4743 : cluster [DBG] pgmap v6086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:13 smithi115 bash[45495]: audit 2022-04-22T19:44:12.419647+0000 mon.a (mon.0) 7618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:13 smithi121 bash[36902]: cluster 2022-04-22T19:44:12.074009+0000 mgr.y (mgr.34107) 4743 : cluster [DBG] pgmap v6086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:13 smithi121 bash[36902]: audit 2022-04-22T19:44:12.419647+0000 mon.a (mon.0) 7618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:14.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:15.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[45495]: cluster 2022-04-22T19:44:14.075138+0000 mgr.y (mgr.34107) 4744 : cluster [DBG] pgmap v6087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[45495]: audit 2022-04-22T19:44:14.515791+0000 mon.a (mon.0) 7619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[45495]: audit 2022-04-22T19:44:14.857470+0000 mon.a (mon.0) 7620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[45495]: audit 2022-04-22T19:44:14.862051+0000 mon.a (mon.0) 7621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[46934]: cluster 2022-04-22T19:44:14.075138+0000 mgr.y (mgr.34107) 4744 : cluster [DBG] pgmap v6087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[46934]: audit 2022-04-22T19:44:14.515791+0000 mon.a (mon.0) 7619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[46934]: audit 2022-04-22T19:44:14.857470+0000 mon.a (mon.0) 7620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:15 smithi115 bash[46934]: audit 2022-04-22T19:44:14.862051+0000 mon.a (mon.0) 7621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:44:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:15 smithi121 bash[36902]: cluster 2022-04-22T19:44:14.075138+0000 mgr.y (mgr.34107) 4744 : cluster [DBG] pgmap v6087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:15 smithi121 bash[36902]: audit 2022-04-22T19:44:14.515791+0000 mon.a (mon.0) 7619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:15 smithi121 bash[36902]: audit 2022-04-22T19:44:14.857470+0000 mon.a (mon.0) 7620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:44:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:15 smithi121 bash[36902]: audit 2022-04-22T19:44:14.862051+0000 mon.a (mon.0) 7621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:44:16.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:16 smithi115 bash[46934]: audit 2022-04-22T19:44:15.950575+0000 mon.a (mon.0) 7622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:16.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:16 smithi115 bash[45495]: audit 2022-04-22T19:44:15.950575+0000 mon.a (mon.0) 7622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:16 smithi121 bash[36902]: audit 2022-04-22T19:44:15.950575+0000 mon.a (mon.0) 7622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:17 smithi115 bash[45495]: cluster 2022-04-22T19:44:16.076247+0000 mgr.y (mgr.34107) 4745 : cluster [DBG] pgmap v6088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:17 smithi115 bash[46934]: cluster 2022-04-22T19:44:16.076247+0000 mgr.y (mgr.34107) 4745 : cluster [DBG] pgmap v6088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:17 smithi121 bash[36902]: cluster 2022-04-22T19:44:16.076247+0000 mgr.y (mgr.34107) 4745 : cluster [DBG] pgmap v6088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:18 smithi115 bash[45495]: audit 2022-04-22T19:44:17.422706+0000 mon.a (mon.0) 7623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:18.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:18 smithi115 bash[46934]: audit 2022-04-22T19:44:17.422706+0000 mon.a (mon.0) 7623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:18 smithi121 bash[36902]: audit 2022-04-22T19:44:17.422706+0000 mon.a (mon.0) 7623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:19 smithi121 bash[36902]: cluster 2022-04-22T19:44:18.077273+0000 mgr.y (mgr.34107) 4746 : cluster [DBG] pgmap v6089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:19 smithi115 bash[45495]: cluster 2022-04-22T19:44:18.077273+0000 mgr.y (mgr.34107) 4746 : cluster [DBG] pgmap v6089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:19.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:19 smithi115 bash[46934]: cluster 2022-04-22T19:44:18.077273+0000 mgr.y (mgr.34107) 4746 : cluster [DBG] pgmap v6089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:20 smithi121 bash[36902]: audit 2022-04-22T19:44:19.519353+0000 mon.a (mon.0) 7624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:20 smithi115 bash[45495]: audit 2022-04-22T19:44:19.519353+0000 mon.a (mon.0) 7624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:20 smithi115 bash[46934]: audit 2022-04-22T19:44:19.519353+0000 mon.a (mon.0) 7624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:21 smithi121 bash[36902]: cluster 2022-04-22T19:44:20.078310+0000 mgr.y (mgr.34107) 4747 : cluster [DBG] pgmap v6090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:21 smithi121 bash[36902]: audit 2022-04-22T19:44:20.953815+0000 mon.a (mon.0) 7625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:21 smithi115 bash[45495]: cluster 2022-04-22T19:44:20.078310+0000 mgr.y (mgr.34107) 4747 : cluster [DBG] pgmap v6090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:21 smithi115 bash[45495]: audit 2022-04-22T19:44:20.953815+0000 mon.a (mon.0) 7625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:21 smithi115 bash[46934]: cluster 2022-04-22T19:44:20.078310+0000 mgr.y (mgr.34107) 4747 : cluster [DBG] pgmap v6090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:21 smithi115 bash[46934]: audit 2022-04-22T19:44:20.953815+0000 mon.a (mon.0) 7625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:22.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:23 smithi121 bash[36902]: cluster 2022-04-22T19:44:22.079445+0000 mgr.y (mgr.34107) 4748 : cluster [DBG] pgmap v6091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:23 smithi121 bash[36902]: audit 2022-04-22T19:44:22.425562+0000 mon.a (mon.0) 7626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:23 smithi115 bash[45495]: cluster 2022-04-22T19:44:22.079445+0000 mgr.y (mgr.34107) 4748 : cluster [DBG] pgmap v6091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:23.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:23 smithi115 bash[45495]: audit 2022-04-22T19:44:22.425562+0000 mon.a (mon.0) 7626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:23.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:23 smithi115 bash[46934]: cluster 2022-04-22T19:44:22.079445+0000 mgr.y (mgr.34107) 4748 : cluster [DBG] pgmap v6091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:23.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:23 smithi115 bash[46934]: audit 2022-04-22T19:44:22.425562+0000 mon.a (mon.0) 7626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:24.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:25 smithi121 bash[36902]: cluster 2022-04-22T19:44:24.080484+0000 mgr.y (mgr.34107) 4749 : cluster [DBG] pgmap v6092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:25.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:25 smithi121 bash[36902]: audit 2022-04-22T19:44:24.522881+0000 mon.a (mon.0) 7627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:25 smithi115 bash[45495]: cluster 2022-04-22T19:44:24.080484+0000 mgr.y (mgr.34107) 4749 : cluster [DBG] pgmap v6092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:25 smithi115 bash[45495]: audit 2022-04-22T19:44:24.522881+0000 mon.a (mon.0) 7627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:25 smithi115 bash[46934]: cluster 2022-04-22T19:44:24.080484+0000 mgr.y (mgr.34107) 4749 : cluster [DBG] pgmap v6092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:25 smithi115 bash[46934]: audit 2022-04-22T19:44:24.522881+0000 mon.a (mon.0) 7627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:26.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:26 smithi115 bash[45495]: audit 2022-04-22T19:44:25.957180+0000 mon.a (mon.0) 7628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:26.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:26 smithi115 bash[46934]: audit 2022-04-22T19:44:25.957180+0000 mon.a (mon.0) 7628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:26.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:26 smithi121 bash[36902]: audit 2022-04-22T19:44:25.957180+0000 mon.a (mon.0) 7628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:27 smithi121 bash[36902]: cluster 2022-04-22T19:44:26.081570+0000 mgr.y (mgr.34107) 4750 : cluster [DBG] pgmap v6093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:27 smithi121 bash[36902]: audit 2022-04-22T19:44:26.346156+0000 mon.a (mon.0) 7629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:44:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:27 smithi121 bash[36902]: audit 2022-04-22T19:44:26.348363+0000 mon.a (mon.0) 7630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:44:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:27 smithi121 bash[36902]: audit 2022-04-22T19:44:26.349743+0000 mon.a (mon.0) 7631 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:44:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:27 smithi121 bash[36902]: audit 2022-04-22T19:44:26.712889+0000 mon.a (mon.0) 7632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:44:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[45495]: cluster 2022-04-22T19:44:26.081570+0000 mgr.y (mgr.34107) 4750 : cluster [DBG] pgmap v6093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[45495]: audit 2022-04-22T19:44:26.346156+0000 mon.a (mon.0) 7629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:44:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[45495]: audit 2022-04-22T19:44:26.348363+0000 mon.a (mon.0) 7630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:44:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[45495]: audit 2022-04-22T19:44:26.349743+0000 mon.a (mon.0) 7631 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[45495]: audit 2022-04-22T19:44:26.712889+0000 mon.a (mon.0) 7632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[46934]: cluster 2022-04-22T19:44:26.081570+0000 mgr.y (mgr.34107) 4750 : cluster [DBG] pgmap v6093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[46934]: audit 2022-04-22T19:44:26.346156+0000 mon.a (mon.0) 7629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[46934]: audit 2022-04-22T19:44:26.348363+0000 mon.a (mon.0) 7630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[46934]: audit 2022-04-22T19:44:26.349743+0000 mon.a (mon.0) 7631 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:44:27.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:27 smithi115 bash[46934]: audit 2022-04-22T19:44:26.712889+0000 mon.a (mon.0) 7632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:44:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:28 smithi121 bash[36902]: audit 2022-04-22T19:44:27.428426+0000 mon.a (mon.0) 7633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:28 smithi115 bash[45495]: audit 2022-04-22T19:44:27.428426+0000 mon.a (mon.0) 7633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:28 smithi115 bash[46934]: audit 2022-04-22T19:44:27.428426+0000 mon.a (mon.0) 7633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:29 smithi121 bash[36902]: cluster 2022-04-22T19:44:28.082599+0000 mgr.y (mgr.34107) 4751 : cluster [DBG] pgmap v6094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:29 smithi115 bash[45495]: cluster 2022-04-22T19:44:28.082599+0000 mgr.y (mgr.34107) 4751 : cluster [DBG] pgmap v6094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:29 smithi115 bash[46934]: cluster 2022-04-22T19:44:28.082599+0000 mgr.y (mgr.34107) 4751 : cluster [DBG] pgmap v6094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:30 smithi121 bash[36902]: audit 2022-04-22T19:44:29.526620+0000 mon.a (mon.0) 7634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:30 smithi115 bash[45495]: audit 2022-04-22T19:44:29.526620+0000 mon.a (mon.0) 7634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:30.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:30 smithi115 bash[46934]: audit 2022-04-22T19:44:29.526620+0000 mon.a (mon.0) 7634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:31 smithi121 bash[36902]: cluster 2022-04-22T19:44:30.083748+0000 mgr.y (mgr.34107) 4752 : cluster [DBG] pgmap v6095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:31 smithi121 bash[36902]: audit 2022-04-22T19:44:30.960254+0000 mon.a (mon.0) 7635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:31 smithi115 bash[45495]: cluster 2022-04-22T19:44:30.083748+0000 mgr.y (mgr.34107) 4752 : cluster [DBG] pgmap v6095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:31 smithi115 bash[45495]: audit 2022-04-22T19:44:30.960254+0000 mon.a (mon.0) 7635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:31 smithi115 bash[46934]: cluster 2022-04-22T19:44:30.083748+0000 mgr.y (mgr.34107) 4752 : cluster [DBG] pgmap v6095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:31 smithi115 bash[46934]: audit 2022-04-22T19:44:30.960254+0000 mon.a (mon.0) 7635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:32.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:33 smithi121 bash[36902]: cluster 2022-04-22T19:44:32.084867+0000 mgr.y (mgr.34107) 4753 : cluster [DBG] pgmap v6096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:33 smithi121 bash[36902]: audit 2022-04-22T19:44:32.431342+0000 mon.a (mon.0) 7636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:33 smithi115 bash[45495]: cluster 2022-04-22T19:44:32.084867+0000 mgr.y (mgr.34107) 4753 : cluster [DBG] pgmap v6096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:33 smithi115 bash[45495]: audit 2022-04-22T19:44:32.431342+0000 mon.a (mon.0) 7636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:33 smithi115 bash[46934]: cluster 2022-04-22T19:44:32.084867+0000 mgr.y (mgr.34107) 4753 : cluster [DBG] pgmap v6096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:33 smithi115 bash[46934]: audit 2022-04-22T19:44:32.431342+0000 mon.a (mon.0) 7636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:34.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:35 smithi121 bash[36902]: cluster 2022-04-22T19:44:34.085963+0000 mgr.y (mgr.34107) 4754 : cluster [DBG] pgmap v6097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:35 smithi121 bash[36902]: audit 2022-04-22T19:44:34.530224+0000 mon.a (mon.0) 7637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:35 smithi115 bash[45495]: cluster 2022-04-22T19:44:34.085963+0000 mgr.y (mgr.34107) 4754 : cluster [DBG] pgmap v6097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:35 smithi115 bash[45495]: audit 2022-04-22T19:44:34.530224+0000 mon.a (mon.0) 7637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:35 smithi115 bash[46934]: cluster 2022-04-22T19:44:34.085963+0000 mgr.y (mgr.34107) 4754 : cluster [DBG] pgmap v6097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:35.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:35 smithi115 bash[46934]: audit 2022-04-22T19:44:34.530224+0000 mon.a (mon.0) 7637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:36 smithi121 bash[36902]: audit 2022-04-22T19:44:35.963624+0000 mon.a (mon.0) 7638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:36 smithi115 bash[45495]: audit 2022-04-22T19:44:35.963624+0000 mon.a (mon.0) 7638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:36 smithi115 bash[46934]: audit 2022-04-22T19:44:35.963624+0000 mon.a (mon.0) 7638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:37 smithi121 bash[36902]: cluster 2022-04-22T19:44:36.086982+0000 mgr.y (mgr.34107) 4755 : cluster [DBG] pgmap v6098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:37.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:37 smithi115 bash[45495]: cluster 2022-04-22T19:44:36.086982+0000 mgr.y (mgr.34107) 4755 : cluster [DBG] pgmap v6098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:37.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:37 smithi115 bash[46934]: cluster 2022-04-22T19:44:36.086982+0000 mgr.y (mgr.34107) 4755 : cluster [DBG] pgmap v6098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:38.367 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:44:38.708 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:44:38.708 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:44:38.709 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:44:38.710 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:44:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:38 smithi121 bash[36902]: audit 2022-04-22T19:44:37.434227+0000 mon.a (mon.0) 7639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:38 smithi115 bash[45495]: audit 2022-04-22T19:44:37.434227+0000 mon.a (mon.0) 7639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:38 smithi115 bash[46934]: audit 2022-04-22T19:44:37.434227+0000 mon.a (mon.0) 7639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:39.111 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:44:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:44:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:39 smithi121 bash[36902]: cluster 2022-04-22T19:44:38.087980+0000 mgr.y (mgr.34107) 4756 : cluster [DBG] pgmap v6099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:39 smithi121 bash[36902]: audit 2022-04-22T19:44:38.341440+0000 mgr.y (mgr.34107) 4757 : audit [DBG] from='client.40356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:39 smithi121 bash[36902]: audit 2022-04-22T19:44:39.100370+0000 mon.a (mon.0) 7640 : audit [DBG] from='client.? 172.21.15.115:0/1129490351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[45495]: cluster 2022-04-22T19:44:38.087980+0000 mgr.y (mgr.34107) 4756 : cluster [DBG] pgmap v6099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[45495]: audit 2022-04-22T19:44:38.341440+0000 mgr.y (mgr.34107) 4757 : audit [DBG] from='client.40356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[45495]: audit 2022-04-22T19:44:39.100370+0000 mon.a (mon.0) 7640 : audit [DBG] from='client.? 172.21.15.115:0/1129490351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[46934]: cluster 2022-04-22T19:44:38.087980+0000 mgr.y (mgr.34107) 4756 : cluster [DBG] pgmap v6099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[46934]: audit 2022-04-22T19:44:38.341440+0000 mgr.y (mgr.34107) 4757 : audit [DBG] from='client.40356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:39 smithi115 bash[46934]: audit 2022-04-22T19:44:39.100370+0000 mon.a (mon.0) 7640 : audit [DBG] from='client.? 172.21.15.115:0/1129490351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:44:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:40 smithi121 bash[36902]: audit 2022-04-22T19:44:38.676836+0000 mgr.y (mgr.34107) 4758 : audit [DBG] from='client.40362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:40 smithi121 bash[36902]: audit 2022-04-22T19:44:39.533948+0000 mon.a (mon.0) 7641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:40 smithi115 bash[45495]: audit 2022-04-22T19:44:38.676836+0000 mgr.y (mgr.34107) 4758 : audit [DBG] from='client.40362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:40 smithi115 bash[45495]: audit 2022-04-22T19:44:39.533948+0000 mon.a (mon.0) 7641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:40.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:40 smithi115 bash[46934]: audit 2022-04-22T19:44:38.676836+0000 mgr.y (mgr.34107) 4758 : audit [DBG] from='client.40362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:44:40.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:40 smithi115 bash[46934]: audit 2022-04-22T19:44:39.533948+0000 mon.a (mon.0) 7641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:41 smithi121 bash[36902]: cluster 2022-04-22T19:44:40.089238+0000 mgr.y (mgr.34107) 4759 : cluster [DBG] pgmap v6100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:41 smithi121 bash[36902]: audit 2022-04-22T19:44:40.966737+0000 mon.a (mon.0) 7642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:41 smithi115 bash[45495]: cluster 2022-04-22T19:44:40.089238+0000 mgr.y (mgr.34107) 4759 : cluster [DBG] pgmap v6100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:41 smithi115 bash[45495]: audit 2022-04-22T19:44:40.966737+0000 mon.a (mon.0) 7642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:41 smithi115 bash[46934]: cluster 2022-04-22T19:44:40.089238+0000 mgr.y (mgr.34107) 4759 : cluster [DBG] pgmap v6100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:41 smithi115 bash[46934]: audit 2022-04-22T19:44:40.966737+0000 mon.a (mon.0) 7642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:42.150Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:43 smithi121 bash[36902]: cluster 2022-04-22T19:44:42.090414+0000 mgr.y (mgr.34107) 4760 : cluster [DBG] pgmap v6101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:43 smithi121 bash[36902]: audit 2022-04-22T19:44:42.437326+0000 mon.a (mon.0) 7643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:43 smithi115 bash[45495]: cluster 2022-04-22T19:44:42.090414+0000 mgr.y (mgr.34107) 4760 : cluster [DBG] pgmap v6101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:43 smithi115 bash[45495]: audit 2022-04-22T19:44:42.437326+0000 mon.a (mon.0) 7643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:43 smithi115 bash[46934]: cluster 2022-04-22T19:44:42.090414+0000 mgr.y (mgr.34107) 4760 : cluster [DBG] pgmap v6101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:43 smithi115 bash[46934]: audit 2022-04-22T19:44:42.437326+0000 mon.a (mon.0) 7643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:44.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:45 smithi121 bash[36902]: cluster 2022-04-22T19:44:44.091528+0000 mgr.y (mgr.34107) 4761 : cluster [DBG] pgmap v6102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:45 smithi121 bash[36902]: audit 2022-04-22T19:44:44.537751+0000 mon.a (mon.0) 7644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:45 smithi115 bash[45495]: cluster 2022-04-22T19:44:44.091528+0000 mgr.y (mgr.34107) 4761 : cluster [DBG] pgmap v6102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:45 smithi115 bash[45495]: audit 2022-04-22T19:44:44.537751+0000 mon.a (mon.0) 7644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:45 smithi115 bash[46934]: cluster 2022-04-22T19:44:44.091528+0000 mgr.y (mgr.34107) 4761 : cluster [DBG] pgmap v6102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:45 smithi115 bash[46934]: audit 2022-04-22T19:44:44.537751+0000 mon.a (mon.0) 7644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:46 smithi121 bash[36902]: audit 2022-04-22T19:44:45.969817+0000 mon.a (mon.0) 7645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:46 smithi115 bash[45495]: audit 2022-04-22T19:44:45.969817+0000 mon.a (mon.0) 7645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:46 smithi115 bash[46934]: audit 2022-04-22T19:44:45.969817+0000 mon.a (mon.0) 7645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:47 smithi121 bash[36902]: cluster 2022-04-22T19:44:46.092512+0000 mgr.y (mgr.34107) 4762 : cluster [DBG] pgmap v6103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:47 smithi115 bash[45495]: cluster 2022-04-22T19:44:46.092512+0000 mgr.y (mgr.34107) 4762 : cluster [DBG] pgmap v6103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:47 smithi115 bash[46934]: cluster 2022-04-22T19:44:46.092512+0000 mgr.y (mgr.34107) 4762 : cluster [DBG] pgmap v6103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:48 smithi121 bash[36902]: audit 2022-04-22T19:44:47.440286+0000 mon.a (mon.0) 7646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:48.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:48 smithi115 bash[45495]: audit 2022-04-22T19:44:47.440286+0000 mon.a (mon.0) 7646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:48.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:48 smithi115 bash[46934]: audit 2022-04-22T19:44:47.440286+0000 mon.a (mon.0) 7646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:49 smithi121 bash[36902]: cluster 2022-04-22T19:44:48.093535+0000 mgr.y (mgr.34107) 4763 : cluster [DBG] pgmap v6104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:49 smithi115 bash[45495]: cluster 2022-04-22T19:44:48.093535+0000 mgr.y (mgr.34107) 4763 : cluster [DBG] pgmap v6104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:49 smithi115 bash[46934]: cluster 2022-04-22T19:44:48.093535+0000 mgr.y (mgr.34107) 4763 : cluster [DBG] pgmap v6104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:50 smithi121 bash[36902]: audit 2022-04-22T19:44:49.541480+0000 mon.a (mon.0) 7647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:50.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:50 smithi115 bash[45495]: audit 2022-04-22T19:44:49.541480+0000 mon.a (mon.0) 7647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:50 smithi115 bash[46934]: audit 2022-04-22T19:44:49.541480+0000 mon.a (mon.0) 7647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:51 smithi121 bash[36902]: cluster 2022-04-22T19:44:50.094615+0000 mgr.y (mgr.34107) 4764 : cluster [DBG] pgmap v6105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:51 smithi121 bash[36902]: audit 2022-04-22T19:44:50.972886+0000 mon.a (mon.0) 7648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:51 smithi115 bash[45495]: cluster 2022-04-22T19:44:50.094615+0000 mgr.y (mgr.34107) 4764 : cluster [DBG] pgmap v6105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:51 smithi115 bash[45495]: audit 2022-04-22T19:44:50.972886+0000 mon.a (mon.0) 7648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:51 smithi115 bash[46934]: cluster 2022-04-22T19:44:50.094615+0000 mgr.y (mgr.34107) 4764 : cluster [DBG] pgmap v6105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:51 smithi115 bash[46934]: audit 2022-04-22T19:44:50.972886+0000 mon.a (mon.0) 7648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:52.152Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:53 smithi121 bash[36902]: cluster 2022-04-22T19:44:52.095783+0000 mgr.y (mgr.34107) 4765 : cluster [DBG] pgmap v6106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:53 smithi121 bash[36902]: audit 2022-04-22T19:44:52.444266+0000 mon.a (mon.0) 7649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:53 smithi115 bash[45495]: cluster 2022-04-22T19:44:52.095783+0000 mgr.y (mgr.34107) 4765 : cluster [DBG] pgmap v6106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:53 smithi115 bash[45495]: audit 2022-04-22T19:44:52.444266+0000 mon.a (mon.0) 7649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:53 smithi115 bash[46934]: cluster 2022-04-22T19:44:52.095783+0000 mgr.y (mgr.34107) 4765 : cluster [DBG] pgmap v6106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:53 smithi115 bash[46934]: audit 2022-04-22T19:44:52.444266+0000 mon.a (mon.0) 7649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:44:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:44:54.019Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:44:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:55 smithi121 bash[36902]: cluster 2022-04-22T19:44:54.096829+0000 mgr.y (mgr.34107) 4766 : cluster [DBG] pgmap v6107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:55 smithi121 bash[36902]: audit 2022-04-22T19:44:54.545354+0000 mon.a (mon.0) 7650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:55 smithi115 bash[45495]: cluster 2022-04-22T19:44:54.096829+0000 mgr.y (mgr.34107) 4766 : cluster [DBG] pgmap v6107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:55 smithi115 bash[45495]: audit 2022-04-22T19:44:54.545354+0000 mon.a (mon.0) 7650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:55 smithi115 bash[46934]: cluster 2022-04-22T19:44:54.096829+0000 mgr.y (mgr.34107) 4766 : cluster [DBG] pgmap v6107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:55 smithi115 bash[46934]: audit 2022-04-22T19:44:54.545354+0000 mon.a (mon.0) 7650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:44:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:56 smithi121 bash[36902]: audit 2022-04-22T19:44:55.976125+0000 mon.a (mon.0) 7651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:56 smithi115 bash[45495]: audit 2022-04-22T19:44:55.976125+0000 mon.a (mon.0) 7651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:56 smithi115 bash[46934]: audit 2022-04-22T19:44:55.976125+0000 mon.a (mon.0) 7651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:44:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:57 smithi121 bash[36902]: cluster 2022-04-22T19:44:56.097808+0000 mgr.y (mgr.34107) 4767 : cluster [DBG] pgmap v6108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:57 smithi121 bash[36902]: audit 2022-04-22T19:44:57.446630+0000 mon.a (mon.0) 7652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:57 smithi115 bash[45495]: cluster 2022-04-22T19:44:56.097808+0000 mgr.y (mgr.34107) 4767 : cluster [DBG] pgmap v6108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:57 smithi115 bash[45495]: audit 2022-04-22T19:44:57.446630+0000 mon.a (mon.0) 7652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:57 smithi115 bash[46934]: cluster 2022-04-22T19:44:56.097808+0000 mgr.y (mgr.34107) 4767 : cluster [DBG] pgmap v6108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:57.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:57 smithi115 bash[46934]: audit 2022-04-22T19:44:57.446630+0000 mon.a (mon.0) 7652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:44:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:44:59 smithi121 bash[36902]: cluster 2022-04-22T19:44:58.098819+0000 mgr.y (mgr.34107) 4768 : cluster [DBG] pgmap v6109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:44:59 smithi115 bash[46934]: cluster 2022-04-22T19:44:58.098819+0000 mgr.y (mgr.34107) 4768 : cluster [DBG] pgmap v6109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:44:59.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:44:59 smithi115 bash[45495]: cluster 2022-04-22T19:44:58.098819+0000 mgr.y (mgr.34107) 4768 : cluster [DBG] pgmap v6109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:00 smithi121 bash[36902]: audit 2022-04-22T19:44:59.549199+0000 mon.a (mon.0) 7653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:00 smithi115 bash[45495]: audit 2022-04-22T19:44:59.549199+0000 mon.a (mon.0) 7653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:00 smithi115 bash[46934]: audit 2022-04-22T19:44:59.549199+0000 mon.a (mon.0) 7653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:01 smithi121 bash[36902]: cluster 2022-04-22T19:45:00.100043+0000 mgr.y (mgr.34107) 4769 : cluster [DBG] pgmap v6110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:01 smithi121 bash[36902]: audit 2022-04-22T19:45:00.979281+0000 mon.a (mon.0) 7654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:01 smithi115 bash[46934]: cluster 2022-04-22T19:45:00.100043+0000 mgr.y (mgr.34107) 4769 : cluster [DBG] pgmap v6110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:01 smithi115 bash[46934]: audit 2022-04-22T19:45:00.979281+0000 mon.a (mon.0) 7654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:01 smithi115 bash[45495]: cluster 2022-04-22T19:45:00.100043+0000 mgr.y (mgr.34107) 4769 : cluster [DBG] pgmap v6110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:01.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:01 smithi115 bash[45495]: audit 2022-04-22T19:45:00.979281+0000 mon.a (mon.0) 7654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:02.488 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:02.152Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:02.489 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:02 smithi115 bash[45495]: audit 2022-04-22T19:45:02.449916+0000 mon.a (mon.0) 7655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:02 smithi121 bash[36902]: audit 2022-04-22T19:45:02.449916+0000 mon.a (mon.0) 7655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:02.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:02 smithi115 bash[46934]: audit 2022-04-22T19:45:02.449916+0000 mon.a (mon.0) 7655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:03 smithi121 bash[36902]: cluster 2022-04-22T19:45:02.101249+0000 mgr.y (mgr.34107) 4770 : cluster [DBG] pgmap v6111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:03 smithi115 bash[45495]: cluster 2022-04-22T19:45:02.101249+0000 mgr.y (mgr.34107) 4770 : cluster [DBG] pgmap v6111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:03 smithi115 bash[46934]: cluster 2022-04-22T19:45:02.101249+0000 mgr.y (mgr.34107) 4770 : cluster [DBG] pgmap v6111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:04.021Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:05 smithi121 bash[36902]: cluster 2022-04-22T19:45:04.102332+0000 mgr.y (mgr.34107) 4771 : cluster [DBG] pgmap v6112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:05 smithi121 bash[36902]: audit 2022-04-22T19:45:04.552872+0000 mon.a (mon.0) 7656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:05 smithi115 bash[45495]: cluster 2022-04-22T19:45:04.102332+0000 mgr.y (mgr.34107) 4771 : cluster [DBG] pgmap v6112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:05 smithi115 bash[45495]: audit 2022-04-22T19:45:04.552872+0000 mon.a (mon.0) 7656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:05 smithi115 bash[46934]: cluster 2022-04-22T19:45:04.102332+0000 mgr.y (mgr.34107) 4771 : cluster [DBG] pgmap v6112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:05 smithi115 bash[46934]: audit 2022-04-22T19:45:04.552872+0000 mon.a (mon.0) 7656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:06 smithi121 bash[36902]: audit 2022-04-22T19:45:05.982533+0000 mon.a (mon.0) 7657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:06 smithi115 bash[45495]: audit 2022-04-22T19:45:05.982533+0000 mon.a (mon.0) 7657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:06.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:06 smithi115 bash[46934]: audit 2022-04-22T19:45:05.982533+0000 mon.a (mon.0) 7657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:07 smithi121 bash[36902]: cluster 2022-04-22T19:45:06.103360+0000 mgr.y (mgr.34107) 4772 : cluster [DBG] pgmap v6113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:07 smithi121 bash[36902]: audit 2022-04-22T19:45:07.453026+0000 mon.a (mon.0) 7658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:07 smithi115 bash[45495]: cluster 2022-04-22T19:45:06.103360+0000 mgr.y (mgr.34107) 4772 : cluster [DBG] pgmap v6113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:07 smithi115 bash[45495]: audit 2022-04-22T19:45:07.453026+0000 mon.a (mon.0) 7658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:07 smithi115 bash[46934]: cluster 2022-04-22T19:45:06.103360+0000 mgr.y (mgr.34107) 4772 : cluster [DBG] pgmap v6113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:07.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:07 smithi115 bash[46934]: audit 2022-04-22T19:45:07.453026+0000 mon.a (mon.0) 7658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:09.490 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:45:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:09 smithi121 bash[36902]: cluster 2022-04-22T19:45:08.104462+0000 mgr.y (mgr.34107) 4773 : cluster [DBG] pgmap v6114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:09 smithi115 bash[46934]: cluster 2022-04-22T19:45:08.104462+0000 mgr.y (mgr.34107) 4773 : cluster [DBG] pgmap v6114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:09 smithi115 bash[45495]: cluster 2022-04-22T19:45:08.104462+0000 mgr.y (mgr.34107) 4773 : cluster [DBG] pgmap v6114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:09.834 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:45:09.834 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:45:09.835 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:45:09.836 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:45:10.237 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:45:10.237 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:45:10.237 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:45:10.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:45:10.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:10.239 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:45:10.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:45:10.239 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:45:10.239 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:45:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[45495]: audit 2022-04-22T19:45:09.461612+0000 mgr.y (mgr.34107) 4774 : audit [DBG] from='client.50308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[45495]: audit 2022-04-22T19:45:09.556399+0000 mon.a (mon.0) 7659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:10.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[45495]: audit 2022-04-22T19:45:10.226964+0000 mon.c (mon.1) 206 : audit [DBG] from='client.? 172.21.15.115:0/4018660635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[46934]: audit 2022-04-22T19:45:09.461612+0000 mgr.y (mgr.34107) 4774 : audit [DBG] from='client.50308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[46934]: audit 2022-04-22T19:45:09.556399+0000 mon.a (mon.0) 7659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:10 smithi115 bash[46934]: audit 2022-04-22T19:45:10.226964+0000 mon.c (mon.1) 206 : audit [DBG] from='client.? 172.21.15.115:0/4018660635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:10 smithi121 bash[36902]: audit 2022-04-22T19:45:09.461612+0000 mgr.y (mgr.34107) 4774 : audit [DBG] from='client.50308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:10 smithi121 bash[36902]: audit 2022-04-22T19:45:09.556399+0000 mon.a (mon.0) 7659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:10 smithi121 bash[36902]: audit 2022-04-22T19:45:10.226964+0000 mon.c (mon.1) 206 : audit [DBG] from='client.? 172.21.15.115:0/4018660635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[45495]: audit 2022-04-22T19:45:09.805891+0000 mgr.y (mgr.34107) 4775 : audit [DBG] from='client.50314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[45495]: cluster 2022-04-22T19:45:10.105562+0000 mgr.y (mgr.34107) 4776 : cluster [DBG] pgmap v6115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:11.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[45495]: audit 2022-04-22T19:45:10.985928+0000 mon.a (mon.0) 7660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[46934]: audit 2022-04-22T19:45:09.805891+0000 mgr.y (mgr.34107) 4775 : audit [DBG] from='client.50314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[46934]: cluster 2022-04-22T19:45:10.105562+0000 mgr.y (mgr.34107) 4776 : cluster [DBG] pgmap v6115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:11.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:11 smithi115 bash[46934]: audit 2022-04-22T19:45:10.985928+0000 mon.a (mon.0) 7660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:11 smithi121 bash[36902]: audit 2022-04-22T19:45:09.805891+0000 mgr.y (mgr.34107) 4775 : audit [DBG] from='client.50314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:11 smithi121 bash[36902]: cluster 2022-04-22T19:45:10.105562+0000 mgr.y (mgr.34107) 4776 : cluster [DBG] pgmap v6115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:11 smithi121 bash[36902]: audit 2022-04-22T19:45:10.985928+0000 mon.a (mon.0) 7660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:12.529 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:12.152Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:12.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:12 smithi115 bash[45495]: audit 2022-04-22T19:45:12.456156+0000 mon.a (mon.0) 7661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:12.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:12 smithi115 bash[46934]: audit 2022-04-22T19:45:12.456156+0000 mon.a (mon.0) 7661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:12 smithi121 bash[36902]: audit 2022-04-22T19:45:12.456156+0000 mon.a (mon.0) 7661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:13 smithi115 bash[45495]: cluster 2022-04-22T19:45:12.106767+0000 mgr.y (mgr.34107) 4777 : cluster [DBG] pgmap v6116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:13 smithi115 bash[46934]: cluster 2022-04-22T19:45:12.106767+0000 mgr.y (mgr.34107) 4777 : cluster [DBG] pgmap v6116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:13 smithi121 bash[36902]: cluster 2022-04-22T19:45:12.106767+0000 mgr.y (mgr.34107) 4777 : cluster [DBG] pgmap v6116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:14.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[45495]: cluster 2022-04-22T19:45:14.107894+0000 mgr.y (mgr.34107) 4778 : cluster [DBG] pgmap v6117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:15.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[45495]: audit 2022-04-22T19:45:14.559704+0000 mon.a (mon.0) 7662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[45495]: audit 2022-04-22T19:45:14.872798+0000 mon.a (mon.0) 7663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[45495]: audit 2022-04-22T19:45:14.873018+0000 mon.a (mon.0) 7664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[46934]: cluster 2022-04-22T19:45:14.107894+0000 mgr.y (mgr.34107) 4778 : cluster [DBG] pgmap v6117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[46934]: audit 2022-04-22T19:45:14.559704+0000 mon.a (mon.0) 7662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[46934]: audit 2022-04-22T19:45:14.872798+0000 mon.a (mon.0) 7663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:45:15.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:15 smithi115 bash[46934]: audit 2022-04-22T19:45:14.873018+0000 mon.a (mon.0) 7664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:45:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:15 smithi121 bash[36902]: cluster 2022-04-22T19:45:14.107894+0000 mgr.y (mgr.34107) 4778 : cluster [DBG] pgmap v6117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:16.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:15 smithi121 bash[36902]: audit 2022-04-22T19:45:14.559704+0000 mon.a (mon.0) 7662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:16.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:15 smithi121 bash[36902]: audit 2022-04-22T19:45:14.872798+0000 mon.a (mon.0) 7663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:45:16.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:15 smithi121 bash[36902]: audit 2022-04-22T19:45:14.873018+0000 mon.a (mon.0) 7664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:45:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:16 smithi115 bash[45495]: audit 2022-04-22T19:45:15.990112+0000 mon.a (mon.0) 7665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:16 smithi115 bash[46934]: audit 2022-04-22T19:45:15.990112+0000 mon.a (mon.0) 7665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:16 smithi121 bash[36902]: audit 2022-04-22T19:45:15.990112+0000 mon.a (mon.0) 7665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:17 smithi121 bash[36902]: cluster 2022-04-22T19:45:16.108973+0000 mgr.y (mgr.34107) 4779 : cluster [DBG] pgmap v6118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:17 smithi121 bash[36902]: audit 2022-04-22T19:45:17.459320+0000 mon.a (mon.0) 7666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:17 smithi115 bash[45495]: cluster 2022-04-22T19:45:16.108973+0000 mgr.y (mgr.34107) 4779 : cluster [DBG] pgmap v6118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:17 smithi115 bash[45495]: audit 2022-04-22T19:45:17.459320+0000 mon.a (mon.0) 7666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:17 smithi115 bash[46934]: cluster 2022-04-22T19:45:16.108973+0000 mgr.y (mgr.34107) 4779 : cluster [DBG] pgmap v6118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:18.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:17 smithi115 bash[46934]: audit 2022-04-22T19:45:17.459320+0000 mon.a (mon.0) 7666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:19 smithi121 bash[36902]: cluster 2022-04-22T19:45:18.110378+0000 mgr.y (mgr.34107) 4780 : cluster [DBG] pgmap v6119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:19 smithi115 bash[45495]: cluster 2022-04-22T19:45:18.110378+0000 mgr.y (mgr.34107) 4780 : cluster [DBG] pgmap v6119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:20.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:19 smithi115 bash[46934]: cluster 2022-04-22T19:45:18.110378+0000 mgr.y (mgr.34107) 4780 : cluster [DBG] pgmap v6119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:20 smithi121 bash[36902]: audit 2022-04-22T19:45:19.563190+0000 mon.a (mon.0) 7667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:20 smithi115 bash[45495]: audit 2022-04-22T19:45:19.563190+0000 mon.a (mon.0) 7667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:20 smithi115 bash[46934]: audit 2022-04-22T19:45:19.563190+0000 mon.a (mon.0) 7667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:21 smithi121 bash[36902]: cluster 2022-04-22T19:45:20.111429+0000 mgr.y (mgr.34107) 4781 : cluster [DBG] pgmap v6120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:21 smithi121 bash[36902]: audit 2022-04-22T19:45:20.995448+0000 mon.a (mon.0) 7668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:21 smithi115 bash[45495]: cluster 2022-04-22T19:45:20.111429+0000 mgr.y (mgr.34107) 4781 : cluster [DBG] pgmap v6120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:22.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:21 smithi115 bash[45495]: audit 2022-04-22T19:45:20.995448+0000 mon.a (mon.0) 7668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:21 smithi115 bash[46934]: cluster 2022-04-22T19:45:20.111429+0000 mgr.y (mgr.34107) 4781 : cluster [DBG] pgmap v6120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:21 smithi115 bash[46934]: audit 2022-04-22T19:45:20.995448+0000 mon.a (mon.0) 7668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:22.154Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:22 smithi121 bash[36902]: audit 2022-04-22T19:45:22.463239+0000 mon.a (mon.0) 7669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:22 smithi115 bash[45495]: audit 2022-04-22T19:45:22.463239+0000 mon.a (mon.0) 7669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:22 smithi115 bash[46934]: audit 2022-04-22T19:45:22.463239+0000 mon.a (mon.0) 7669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:23 smithi121 bash[36902]: cluster 2022-04-22T19:45:22.112555+0000 mgr.y (mgr.34107) 4782 : cluster [DBG] pgmap v6121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:24.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:23 smithi115 bash[45495]: cluster 2022-04-22T19:45:22.112555+0000 mgr.y (mgr.34107) 4782 : cluster [DBG] pgmap v6121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:24.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:23 smithi115 bash[46934]: cluster 2022-04-22T19:45:22.112555+0000 mgr.y (mgr.34107) 4782 : cluster [DBG] pgmap v6121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:24.022Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:25 smithi121 bash[36902]: cluster 2022-04-22T19:45:24.113929+0000 mgr.y (mgr.34107) 4783 : cluster [DBG] pgmap v6122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:25 smithi121 bash[36902]: audit 2022-04-22T19:45:24.569978+0000 mon.a (mon.0) 7670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:25 smithi115 bash[45495]: cluster 2022-04-22T19:45:24.113929+0000 mgr.y (mgr.34107) 4783 : cluster [DBG] pgmap v6122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:25 smithi115 bash[45495]: audit 2022-04-22T19:45:24.569978+0000 mon.a (mon.0) 7670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:26.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:25 smithi115 bash[46934]: cluster 2022-04-22T19:45:24.113929+0000 mgr.y (mgr.34107) 4783 : cluster [DBG] pgmap v6122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:26.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:25 smithi115 bash[46934]: audit 2022-04-22T19:45:24.569978+0000 mon.a (mon.0) 7670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:26 smithi121 bash[36902]: audit 2022-04-22T19:45:25.998148+0000 mon.a (mon.0) 7671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:26 smithi115 bash[45495]: audit 2022-04-22T19:45:25.998148+0000 mon.a (mon.0) 7671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:26 smithi115 bash[46934]: audit 2022-04-22T19:45:25.998148+0000 mon.a (mon.0) 7671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: cluster 2022-04-22T19:45:26.115052+0000 mgr.y (mgr.34107) 4784 : cluster [DBG] pgmap v6123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: audit 2022-04-22T19:45:26.721578+0000 mon.a (mon.0) 7672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: audit 2022-04-22T19:45:26.724530+0000 mon.a (mon.0) 7673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: audit 2022-04-22T19:45:26.726239+0000 mon.a (mon.0) 7674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: audit 2022-04-22T19:45:27.083801+0000 mon.a (mon.0) 7675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:45:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:27 smithi121 bash[36902]: audit 2022-04-22T19:45:27.468456+0000 mon.a (mon.0) 7676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: cluster 2022-04-22T19:45:26.115052+0000 mgr.y (mgr.34107) 4784 : cluster [DBG] pgmap v6123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: audit 2022-04-22T19:45:26.721578+0000 mon.a (mon.0) 7672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: audit 2022-04-22T19:45:26.724530+0000 mon.a (mon.0) 7673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: audit 2022-04-22T19:45:26.726239+0000 mon.a (mon.0) 7674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: audit 2022-04-22T19:45:27.083801+0000 mon.a (mon.0) 7675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[45495]: audit 2022-04-22T19:45:27.468456+0000 mon.a (mon.0) 7676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: cluster 2022-04-22T19:45:26.115052+0000 mgr.y (mgr.34107) 4784 : cluster [DBG] pgmap v6123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:28.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: audit 2022-04-22T19:45:26.721578+0000 mon.a (mon.0) 7672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:45:28.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: audit 2022-04-22T19:45:26.724530+0000 mon.a (mon.0) 7673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:45:28.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: audit 2022-04-22T19:45:26.726239+0000 mon.a (mon.0) 7674 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:45:28.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: audit 2022-04-22T19:45:27.083801+0000 mon.a (mon.0) 7675 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:45:28.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:27 smithi115 bash[46934]: audit 2022-04-22T19:45:27.468456+0000 mon.a (mon.0) 7676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:29 smithi121 bash[36902]: cluster 2022-04-22T19:45:28.116079+0000 mgr.y (mgr.34107) 4785 : cluster [DBG] pgmap v6124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:29 smithi121 bash[36902]: audit 2022-04-22T19:45:29.572057+0000 mon.a (mon.0) 7677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:29 smithi115 bash[45495]: cluster 2022-04-22T19:45:28.116079+0000 mgr.y (mgr.34107) 4785 : cluster [DBG] pgmap v6124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:29 smithi115 bash[45495]: audit 2022-04-22T19:45:29.572057+0000 mon.a (mon.0) 7677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:29 smithi115 bash[46934]: cluster 2022-04-22T19:45:28.116079+0000 mgr.y (mgr.34107) 4785 : cluster [DBG] pgmap v6124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:29 smithi115 bash[46934]: audit 2022-04-22T19:45:29.572057+0000 mon.a (mon.0) 7677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:31 smithi121 bash[36902]: cluster 2022-04-22T19:45:30.117105+0000 mgr.y (mgr.34107) 4786 : cluster [DBG] pgmap v6125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:31 smithi121 bash[36902]: audit 2022-04-22T19:45:31.000387+0000 mon.a (mon.0) 7678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:31 smithi115 bash[46934]: cluster 2022-04-22T19:45:30.117105+0000 mgr.y (mgr.34107) 4786 : cluster [DBG] pgmap v6125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:31 smithi115 bash[46934]: audit 2022-04-22T19:45:31.000387+0000 mon.a (mon.0) 7678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:32.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:31 smithi115 bash[45495]: cluster 2022-04-22T19:45:30.117105+0000 mgr.y (mgr.34107) 4786 : cluster [DBG] pgmap v6125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:32.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:31 smithi115 bash[45495]: audit 2022-04-22T19:45:31.000387+0000 mon.a (mon.0) 7678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:32.154Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:32 smithi121 bash[36902]: audit 2022-04-22T19:45:32.471683+0000 mon.a (mon.0) 7679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:32 smithi115 bash[45495]: audit 2022-04-22T19:45:32.471683+0000 mon.a (mon.0) 7679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:32 smithi115 bash[46934]: audit 2022-04-22T19:45:32.471683+0000 mon.a (mon.0) 7679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:33 smithi121 bash[36902]: cluster 2022-04-22T19:45:32.118292+0000 mgr.y (mgr.34107) 4787 : cluster [DBG] pgmap v6126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:34.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:33 smithi115 bash[46934]: cluster 2022-04-22T19:45:32.118292+0000 mgr.y (mgr.34107) 4787 : cluster [DBG] pgmap v6126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:34.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:33 smithi115 bash[45495]: cluster 2022-04-22T19:45:32.118292+0000 mgr.y (mgr.34107) 4787 : cluster [DBG] pgmap v6126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:34.024Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:34 smithi121 bash[36902]: audit 2022-04-22T19:45:34.575759+0000 mon.a (mon.0) 7680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:34 smithi115 bash[45495]: audit 2022-04-22T19:45:34.575759+0000 mon.a (mon.0) 7680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:35.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:34 smithi115 bash[46934]: audit 2022-04-22T19:45:34.575759+0000 mon.a (mon.0) 7680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:35 smithi121 bash[36902]: cluster 2022-04-22T19:45:34.119383+0000 mgr.y (mgr.34107) 4788 : cluster [DBG] pgmap v6127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:35 smithi115 bash[45495]: cluster 2022-04-22T19:45:34.119383+0000 mgr.y (mgr.34107) 4788 : cluster [DBG] pgmap v6127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:35 smithi115 bash[46934]: cluster 2022-04-22T19:45:34.119383+0000 mgr.y (mgr.34107) 4788 : cluster [DBG] pgmap v6127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:36 smithi121 bash[36902]: audit 2022-04-22T19:45:36.002784+0000 mon.a (mon.0) 7681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:36 smithi115 bash[46934]: audit 2022-04-22T19:45:36.002784+0000 mon.a (mon.0) 7681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:36 smithi115 bash[45495]: audit 2022-04-22T19:45:36.002784+0000 mon.a (mon.0) 7681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:37 smithi121 bash[36902]: cluster 2022-04-22T19:45:36.120431+0000 mgr.y (mgr.34107) 4789 : cluster [DBG] pgmap v6128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:37 smithi121 bash[36902]: audit 2022-04-22T19:45:37.474582+0000 mon.a (mon.0) 7682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:37 smithi115 bash[45495]: cluster 2022-04-22T19:45:36.120431+0000 mgr.y (mgr.34107) 4789 : cluster [DBG] pgmap v6128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:37 smithi115 bash[45495]: audit 2022-04-22T19:45:37.474582+0000 mon.a (mon.0) 7682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:37 smithi115 bash[46934]: cluster 2022-04-22T19:45:36.120431+0000 mgr.y (mgr.34107) 4789 : cluster [DBG] pgmap v6128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:37 smithi115 bash[46934]: audit 2022-04-22T19:45:37.474582+0000 mon.a (mon.0) 7682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:39 smithi121 bash[36902]: cluster 2022-04-22T19:45:38.121600+0000 mgr.y (mgr.34107) 4790 : cluster [DBG] pgmap v6129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:39 smithi121 bash[36902]: audit 2022-04-22T19:45:39.579418+0000 mon.a (mon.0) 7683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:39 smithi115 bash[45495]: cluster 2022-04-22T19:45:38.121600+0000 mgr.y (mgr.34107) 4790 : cluster [DBG] pgmap v6129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:39 smithi115 bash[45495]: audit 2022-04-22T19:45:39.579418+0000 mon.a (mon.0) 7683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:39 smithi115 bash[46934]: cluster 2022-04-22T19:45:38.121600+0000 mgr.y (mgr.34107) 4790 : cluster [DBG] pgmap v6129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:39 smithi115 bash[46934]: audit 2022-04-22T19:45:39.579418+0000 mon.a (mon.0) 7683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:40.602 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:45:40.947 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:45:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:45:40.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:45:40.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:45:40.950 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:41.352 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:45:41.353 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:45:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:41 smithi121 bash[36902]: cluster 2022-04-22T19:45:40.122692+0000 mgr.y (mgr.34107) 4791 : cluster [DBG] pgmap v6130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:41 smithi121 bash[36902]: audit 2022-04-22T19:45:40.577308+0000 mgr.y (mgr.34107) 4792 : audit [DBG] from='client.40392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:41 smithi121 bash[36902]: audit 2022-04-22T19:45:41.004993+0000 mon.a (mon.0) 7684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:41 smithi121 bash[36902]: audit 2022-04-22T19:45:41.341024+0000 mon.a (mon.0) 7685 : audit [DBG] from='client.? 172.21.15.115:0/2846385389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[45495]: cluster 2022-04-22T19:45:40.122692+0000 mgr.y (mgr.34107) 4791 : cluster [DBG] pgmap v6130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[45495]: audit 2022-04-22T19:45:40.577308+0000 mgr.y (mgr.34107) 4792 : audit [DBG] from='client.40392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[45495]: audit 2022-04-22T19:45:41.004993+0000 mon.a (mon.0) 7684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[45495]: audit 2022-04-22T19:45:41.341024+0000 mon.a (mon.0) 7685 : audit [DBG] from='client.? 172.21.15.115:0/2846385389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[46934]: cluster 2022-04-22T19:45:40.122692+0000 mgr.y (mgr.34107) 4791 : cluster [DBG] pgmap v6130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[46934]: audit 2022-04-22T19:45:40.577308+0000 mgr.y (mgr.34107) 4792 : audit [DBG] from='client.40392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[46934]: audit 2022-04-22T19:45:41.004993+0000 mon.a (mon.0) 7684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:41 smithi115 bash[46934]: audit 2022-04-22T19:45:41.341024+0000 mon.a (mon.0) 7685 : audit [DBG] from='client.? 172.21.15.115:0/2846385389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:45:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:42.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:42 smithi121 bash[36902]: audit 2022-04-22T19:45:40.918119+0000 mgr.y (mgr.34107) 4793 : audit [DBG] from='client.40398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:42 smithi121 bash[36902]: audit 2022-04-22T19:45:42.477506+0000 mon.a (mon.0) 7686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:42 smithi115 bash[45495]: audit 2022-04-22T19:45:40.918119+0000 mgr.y (mgr.34107) 4793 : audit [DBG] from='client.40398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:42 smithi115 bash[45495]: audit 2022-04-22T19:45:42.477506+0000 mon.a (mon.0) 7686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:43.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:42 smithi115 bash[46934]: audit 2022-04-22T19:45:40.918119+0000 mgr.y (mgr.34107) 4793 : audit [DBG] from='client.40398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:45:43.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:42 smithi115 bash[46934]: audit 2022-04-22T19:45:42.477506+0000 mon.a (mon.0) 7686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:43 smithi121 bash[36902]: cluster 2022-04-22T19:45:42.123882+0000 mgr.y (mgr.34107) 4794 : cluster [DBG] pgmap v6131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:44.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:43 smithi115 bash[45495]: cluster 2022-04-22T19:45:42.123882+0000 mgr.y (mgr.34107) 4794 : cluster [DBG] pgmap v6131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:44.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:43 smithi115 bash[46934]: cluster 2022-04-22T19:45:42.123882+0000 mgr.y (mgr.34107) 4794 : cluster [DBG] pgmap v6131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:44.024Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:44 smithi121 bash[36902]: audit 2022-04-22T19:45:44.583075+0000 mon.a (mon.0) 7687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:44 smithi115 bash[46934]: audit 2022-04-22T19:45:44.583075+0000 mon.a (mon.0) 7687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:45.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:44 smithi115 bash[45495]: audit 2022-04-22T19:45:44.583075+0000 mon.a (mon.0) 7687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:45 smithi121 bash[36902]: cluster 2022-04-22T19:45:44.124955+0000 mgr.y (mgr.34107) 4795 : cluster [DBG] pgmap v6132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:45 smithi115 bash[45495]: cluster 2022-04-22T19:45:44.124955+0000 mgr.y (mgr.34107) 4795 : cluster [DBG] pgmap v6132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:45 smithi115 bash[46934]: cluster 2022-04-22T19:45:44.124955+0000 mgr.y (mgr.34107) 4795 : cluster [DBG] pgmap v6132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:46 smithi121 bash[36902]: audit 2022-04-22T19:45:46.007987+0000 mon.a (mon.0) 7688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:47.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:46 smithi115 bash[45495]: audit 2022-04-22T19:45:46.007987+0000 mon.a (mon.0) 7688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:47.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:46 smithi115 bash[46934]: audit 2022-04-22T19:45:46.007987+0000 mon.a (mon.0) 7688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:47 smithi121 bash[36902]: cluster 2022-04-22T19:45:46.125980+0000 mgr.y (mgr.34107) 4796 : cluster [DBG] pgmap v6133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:47 smithi121 bash[36902]: audit 2022-04-22T19:45:47.480433+0000 mon.a (mon.0) 7689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:47 smithi115 bash[45495]: cluster 2022-04-22T19:45:46.125980+0000 mgr.y (mgr.34107) 4796 : cluster [DBG] pgmap v6133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:47 smithi115 bash[45495]: audit 2022-04-22T19:45:47.480433+0000 mon.a (mon.0) 7689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:48.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:47 smithi115 bash[46934]: cluster 2022-04-22T19:45:46.125980+0000 mgr.y (mgr.34107) 4796 : cluster [DBG] pgmap v6133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:48.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:47 smithi115 bash[46934]: audit 2022-04-22T19:45:47.480433+0000 mon.a (mon.0) 7689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:49 smithi121 bash[36902]: cluster 2022-04-22T19:45:48.127068+0000 mgr.y (mgr.34107) 4797 : cluster [DBG] pgmap v6134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:49 smithi121 bash[36902]: audit 2022-04-22T19:45:49.586793+0000 mon.a (mon.0) 7690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:49 smithi115 bash[45495]: cluster 2022-04-22T19:45:48.127068+0000 mgr.y (mgr.34107) 4797 : cluster [DBG] pgmap v6134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:49 smithi115 bash[45495]: audit 2022-04-22T19:45:49.586793+0000 mon.a (mon.0) 7690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:50.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:49 smithi115 bash[46934]: cluster 2022-04-22T19:45:48.127068+0000 mgr.y (mgr.34107) 4797 : cluster [DBG] pgmap v6134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:50.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:49 smithi115 bash[46934]: audit 2022-04-22T19:45:49.586793+0000 mon.a (mon.0) 7690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:51 smithi121 bash[36902]: cluster 2022-04-22T19:45:50.128150+0000 mgr.y (mgr.34107) 4798 : cluster [DBG] pgmap v6135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:51 smithi121 bash[36902]: audit 2022-04-22T19:45:51.011180+0000 mon.a (mon.0) 7691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:51 smithi115 bash[45495]: cluster 2022-04-22T19:45:50.128150+0000 mgr.y (mgr.34107) 4798 : cluster [DBG] pgmap v6135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:51 smithi115 bash[45495]: audit 2022-04-22T19:45:51.011180+0000 mon.a (mon.0) 7691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:51 smithi115 bash[46934]: cluster 2022-04-22T19:45:50.128150+0000 mgr.y (mgr.34107) 4798 : cluster [DBG] pgmap v6135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:51 smithi115 bash[46934]: audit 2022-04-22T19:45:51.011180+0000 mon.a (mon.0) 7691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:52.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:52 smithi121 bash[36902]: audit 2022-04-22T19:45:52.483451+0000 mon.a (mon.0) 7692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:53.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:52 smithi115 bash[45495]: audit 2022-04-22T19:45:52.483451+0000 mon.a (mon.0) 7692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:52 smithi115 bash[46934]: audit 2022-04-22T19:45:52.483451+0000 mon.a (mon.0) 7692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:53 smithi121 bash[36902]: cluster 2022-04-22T19:45:52.129238+0000 mgr.y (mgr.34107) 4799 : cluster [DBG] pgmap v6136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:54.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:53 smithi115 bash[45495]: cluster 2022-04-22T19:45:52.129238+0000 mgr.y (mgr.34107) 4799 : cluster [DBG] pgmap v6136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:54.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:53 smithi115 bash[46934]: cluster 2022-04-22T19:45:52.129238+0000 mgr.y (mgr.34107) 4799 : cluster [DBG] pgmap v6136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:45:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:45:54.024Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:45:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:54 smithi121 bash[36902]: audit 2022-04-22T19:45:54.590425+0000 mon.a (mon.0) 7693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:55.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:54 smithi115 bash[45495]: audit 2022-04-22T19:45:54.590425+0000 mon.a (mon.0) 7693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:55.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:54 smithi115 bash[46934]: audit 2022-04-22T19:45:54.590425+0000 mon.a (mon.0) 7693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:45:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:55 smithi121 bash[36902]: cluster 2022-04-22T19:45:54.130254+0000 mgr.y (mgr.34107) 4800 : cluster [DBG] pgmap v6137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:55 smithi115 bash[45495]: cluster 2022-04-22T19:45:54.130254+0000 mgr.y (mgr.34107) 4800 : cluster [DBG] pgmap v6137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:55 smithi115 bash[46934]: cluster 2022-04-22T19:45:54.130254+0000 mgr.y (mgr.34107) 4800 : cluster [DBG] pgmap v6137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:56 smithi121 bash[36902]: audit 2022-04-22T19:45:56.014262+0000 mon.a (mon.0) 7694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:56 smithi121 bash[36902]: cluster 2022-04-22T19:45:56.131269+0000 mgr.y (mgr.34107) 4801 : cluster [DBG] pgmap v6138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:57.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:56 smithi115 bash[45495]: audit 2022-04-22T19:45:56.014262+0000 mon.a (mon.0) 7694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:57.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:56 smithi115 bash[45495]: cluster 2022-04-22T19:45:56.131269+0000 mgr.y (mgr.34107) 4801 : cluster [DBG] pgmap v6138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:57.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:56 smithi115 bash[46934]: audit 2022-04-22T19:45:56.014262+0000 mon.a (mon.0) 7694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:45:57.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:56 smithi115 bash[46934]: cluster 2022-04-22T19:45:56.131269+0000 mgr.y (mgr.34107) 4801 : cluster [DBG] pgmap v6138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:57 smithi121 bash[36902]: audit 2022-04-22T19:45:57.486586+0000 mon.a (mon.0) 7695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:57 smithi115 bash[45495]: audit 2022-04-22T19:45:57.486586+0000 mon.a (mon.0) 7695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:58.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:57 smithi115 bash[46934]: audit 2022-04-22T19:45:57.486586+0000 mon.a (mon.0) 7695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:45:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:45:59 smithi121 bash[36902]: cluster 2022-04-22T19:45:58.132464+0000 mgr.y (mgr.34107) 4802 : cluster [DBG] pgmap v6139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:59.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:45:59 smithi115 bash[45495]: cluster 2022-04-22T19:45:58.132464+0000 mgr.y (mgr.34107) 4802 : cluster [DBG] pgmap v6139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:45:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:45:59 smithi115 bash[46934]: cluster 2022-04-22T19:45:58.132464+0000 mgr.y (mgr.34107) 4802 : cluster [DBG] pgmap v6139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:00 smithi121 bash[36902]: audit 2022-04-22T19:45:59.594080+0000 mon.a (mon.0) 7696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:00 smithi115 bash[45495]: audit 2022-04-22T19:45:59.594080+0000 mon.a (mon.0) 7696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:00.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:00 smithi115 bash[46934]: audit 2022-04-22T19:45:59.594080+0000 mon.a (mon.0) 7696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:01 smithi121 bash[36902]: cluster 2022-04-22T19:46:00.133628+0000 mgr.y (mgr.34107) 4803 : cluster [DBG] pgmap v6140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:01 smithi121 bash[36902]: audit 2022-04-22T19:46:01.018043+0000 mon.a (mon.0) 7697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:01 smithi115 bash[45495]: cluster 2022-04-22T19:46:00.133628+0000 mgr.y (mgr.34107) 4803 : cluster [DBG] pgmap v6140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:01 smithi115 bash[45495]: audit 2022-04-22T19:46:01.018043+0000 mon.a (mon.0) 7697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:01 smithi115 bash[46934]: cluster 2022-04-22T19:46:00.133628+0000 mgr.y (mgr.34107) 4803 : cluster [DBG] pgmap v6140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:01.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:01 smithi115 bash[46934]: audit 2022-04-22T19:46:01.018043+0000 mon.a (mon.0) 7697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:02.154Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:03 smithi121 bash[36902]: cluster 2022-04-22T19:46:02.134955+0000 mgr.y (mgr.34107) 4804 : cluster [DBG] pgmap v6141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:03 smithi121 bash[36902]: audit 2022-04-22T19:46:02.489465+0000 mon.a (mon.0) 7698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:03 smithi115 bash[45495]: cluster 2022-04-22T19:46:02.134955+0000 mgr.y (mgr.34107) 4804 : cluster [DBG] pgmap v6141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:03 smithi115 bash[45495]: audit 2022-04-22T19:46:02.489465+0000 mon.a (mon.0) 7698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:03 smithi115 bash[46934]: cluster 2022-04-22T19:46:02.134955+0000 mgr.y (mgr.34107) 4804 : cluster [DBG] pgmap v6141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:03 smithi115 bash[46934]: audit 2022-04-22T19:46:02.489465+0000 mon.a (mon.0) 7698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:04.026Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:05 smithi121 bash[36902]: cluster 2022-04-22T19:46:04.136066+0000 mgr.y (mgr.34107) 4805 : cluster [DBG] pgmap v6142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:05 smithi121 bash[36902]: audit 2022-04-22T19:46:04.597796+0000 mon.a (mon.0) 7699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:05 smithi115 bash[45495]: cluster 2022-04-22T19:46:04.136066+0000 mgr.y (mgr.34107) 4805 : cluster [DBG] pgmap v6142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:05.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:05 smithi115 bash[45495]: audit 2022-04-22T19:46:04.597796+0000 mon.a (mon.0) 7699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:05 smithi115 bash[46934]: cluster 2022-04-22T19:46:04.136066+0000 mgr.y (mgr.34107) 4805 : cluster [DBG] pgmap v6142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:05.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:05 smithi115 bash[46934]: audit 2022-04-22T19:46:04.597796+0000 mon.a (mon.0) 7699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:06 smithi121 bash[36902]: audit 2022-04-22T19:46:06.021037+0000 mon.a (mon.0) 7700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:06 smithi115 bash[45495]: audit 2022-04-22T19:46:06.021037+0000 mon.a (mon.0) 7700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:06.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:06 smithi115 bash[46934]: audit 2022-04-22T19:46:06.021037+0000 mon.a (mon.0) 7700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:07 smithi121 bash[36902]: cluster 2022-04-22T19:46:06.137058+0000 mgr.y (mgr.34107) 4806 : cluster [DBG] pgmap v6143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:07 smithi115 bash[45495]: cluster 2022-04-22T19:46:06.137058+0000 mgr.y (mgr.34107) 4806 : cluster [DBG] pgmap v6143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:07 smithi115 bash[46934]: cluster 2022-04-22T19:46:06.137058+0000 mgr.y (mgr.34107) 4806 : cluster [DBG] pgmap v6143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:08 smithi121 bash[36902]: audit 2022-04-22T19:46:07.492476+0000 mon.a (mon.0) 7701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:08 smithi115 bash[45495]: audit 2022-04-22T19:46:07.492476+0000 mon.a (mon.0) 7701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:08 smithi115 bash[46934]: audit 2022-04-22T19:46:07.492476+0000 mon.a (mon.0) 7701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:09.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:09 smithi115 bash[46934]: cluster 2022-04-22T19:46:08.138059+0000 mgr.y (mgr.34107) 4807 : cluster [DBG] pgmap v6144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:09 smithi115 bash[45495]: cluster 2022-04-22T19:46:08.138059+0000 mgr.y (mgr.34107) 4807 : cluster [DBG] pgmap v6144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:09 smithi121 bash[36902]: cluster 2022-04-22T19:46:08.138059+0000 mgr.y (mgr.34107) 4807 : cluster [DBG] pgmap v6144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:10.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:10 smithi115 bash[45495]: audit 2022-04-22T19:46:09.601560+0000 mon.a (mon.0) 7702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:10.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:10 smithi115 bash[46934]: audit 2022-04-22T19:46:09.601560+0000 mon.a (mon.0) 7702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:10 smithi121 bash[36902]: audit 2022-04-22T19:46:09.601560+0000 mon.a (mon.0) 7702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:11 smithi115 bash[45495]: cluster 2022-04-22T19:46:10.139219+0000 mgr.y (mgr.34107) 4808 : cluster [DBG] pgmap v6145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:11.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:11 smithi115 bash[45495]: audit 2022-04-22T19:46:11.023984+0000 mon.a (mon.0) 7703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:11 smithi115 bash[46934]: cluster 2022-04-22T19:46:10.139219+0000 mgr.y (mgr.34107) 4808 : cluster [DBG] pgmap v6145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:11.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:11 smithi115 bash[46934]: audit 2022-04-22T19:46:11.023984+0000 mon.a (mon.0) 7703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:11.720 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:46:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:11 smithi121 bash[36902]: cluster 2022-04-22T19:46:10.139219+0000 mgr.y (mgr.34107) 4808 : cluster [DBG] pgmap v6145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:11 smithi121 bash[36902]: audit 2022-04-22T19:46:11.023984+0000 mon.a (mon.0) 7703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:12.061 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:46:12.061 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:46:12.061 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:46:12.061 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:46:12.061 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:46:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:46:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:46:12.064 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:46:12.064 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:46:12.064 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:46:12.064 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:46:12.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:46:12.470 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:46:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:12.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[45495]: audit 2022-04-22T19:46:11.694850+0000 mgr.y (mgr.34107) 4809 : audit [DBG] from='client.40410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[45495]: audit 2022-04-22T19:46:12.036608+0000 mgr.y (mgr.34107) 4810 : audit [DBG] from='client.50350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[45495]: cluster 2022-04-22T19:46:12.140308+0000 mgr.y (mgr.34107) 4811 : cluster [DBG] pgmap v6146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[45495]: audit 2022-04-22T19:46:12.458721+0000 mon.c (mon.1) 207 : audit [DBG] from='client.? 172.21.15.115:0/2357801197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[45495]: audit 2022-04-22T19:46:12.495513+0000 mon.a (mon.0) 7704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[46934]: audit 2022-04-22T19:46:11.694850+0000 mgr.y (mgr.34107) 4809 : audit [DBG] from='client.40410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[46934]: audit 2022-04-22T19:46:12.036608+0000 mgr.y (mgr.34107) 4810 : audit [DBG] from='client.50350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[46934]: cluster 2022-04-22T19:46:12.140308+0000 mgr.y (mgr.34107) 4811 : cluster [DBG] pgmap v6146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:13.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[46934]: audit 2022-04-22T19:46:12.458721+0000 mon.c (mon.1) 207 : audit [DBG] from='client.? 172.21.15.115:0/2357801197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:13.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:13 smithi115 bash[46934]: audit 2022-04-22T19:46:12.495513+0000 mon.a (mon.0) 7704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:13 smithi121 bash[36902]: audit 2022-04-22T19:46:11.694850+0000 mgr.y (mgr.34107) 4809 : audit [DBG] from='client.40410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:13 smithi121 bash[36902]: audit 2022-04-22T19:46:12.036608+0000 mgr.y (mgr.34107) 4810 : audit [DBG] from='client.50350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:13.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:13 smithi121 bash[36902]: cluster 2022-04-22T19:46:12.140308+0000 mgr.y (mgr.34107) 4811 : cluster [DBG] pgmap v6146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:13.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:13 smithi121 bash[36902]: audit 2022-04-22T19:46:12.458721+0000 mon.c (mon.1) 207 : audit [DBG] from='client.? 172.21.15.115:0/2357801197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:13.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:13 smithi121 bash[36902]: audit 2022-04-22T19:46:12.495513+0000 mon.a (mon.0) 7704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:14.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[46934]: cluster 2022-04-22T19:46:14.141371+0000 mgr.y (mgr.34107) 4812 : cluster [DBG] pgmap v6147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:15.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[46934]: audit 2022-04-22T19:46:14.605521+0000 mon.a (mon.0) 7705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[46934]: audit 2022-04-22T19:46:14.887257+0000 mon.a (mon.0) 7706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[46934]: audit 2022-04-22T19:46:14.887722+0000 mon.a (mon.0) 7707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[45495]: cluster 2022-04-22T19:46:14.141371+0000 mgr.y (mgr.34107) 4812 : cluster [DBG] pgmap v6147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[45495]: audit 2022-04-22T19:46:14.605521+0000 mon.a (mon.0) 7705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[45495]: audit 2022-04-22T19:46:14.887257+0000 mon.a (mon.0) 7706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:46:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:15 smithi115 bash[45495]: audit 2022-04-22T19:46:14.887722+0000 mon.a (mon.0) 7707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:46:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:15 smithi121 bash[36902]: cluster 2022-04-22T19:46:14.141371+0000 mgr.y (mgr.34107) 4812 : cluster [DBG] pgmap v6147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:15 smithi121 bash[36902]: audit 2022-04-22T19:46:14.605521+0000 mon.a (mon.0) 7705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:15 smithi121 bash[36902]: audit 2022-04-22T19:46:14.887257+0000 mon.a (mon.0) 7706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:46:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:15 smithi121 bash[36902]: audit 2022-04-22T19:46:14.887722+0000 mon.a (mon.0) 7707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:46:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:16 smithi115 bash[45495]: audit 2022-04-22T19:46:16.026972+0000 mon.a (mon.0) 7708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:16.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:16 smithi115 bash[46934]: audit 2022-04-22T19:46:16.026972+0000 mon.a (mon.0) 7708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:16 smithi121 bash[36902]: audit 2022-04-22T19:46:16.026972+0000 mon.a (mon.0) 7708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:17.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:17 smithi115 bash[45495]: cluster 2022-04-22T19:46:16.142423+0000 mgr.y (mgr.34107) 4813 : cluster [DBG] pgmap v6148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:17 smithi115 bash[46934]: cluster 2022-04-22T19:46:16.142423+0000 mgr.y (mgr.34107) 4813 : cluster [DBG] pgmap v6148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:17 smithi121 bash[36902]: cluster 2022-04-22T19:46:16.142423+0000 mgr.y (mgr.34107) 4813 : cluster [DBG] pgmap v6148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:18 smithi121 bash[36902]: audit 2022-04-22T19:46:17.498125+0000 mon.a (mon.0) 7709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:18 smithi115 bash[46934]: audit 2022-04-22T19:46:17.498125+0000 mon.a (mon.0) 7709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:18 smithi115 bash[45495]: audit 2022-04-22T19:46:17.498125+0000 mon.a (mon.0) 7709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:19 smithi121 bash[36902]: cluster 2022-04-22T19:46:18.143590+0000 mgr.y (mgr.34107) 4814 : cluster [DBG] pgmap v6149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:19 smithi115 bash[45495]: cluster 2022-04-22T19:46:18.143590+0000 mgr.y (mgr.34107) 4814 : cluster [DBG] pgmap v6149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:19 smithi115 bash[46934]: cluster 2022-04-22T19:46:18.143590+0000 mgr.y (mgr.34107) 4814 : cluster [DBG] pgmap v6149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:20 smithi121 bash[36902]: audit 2022-04-22T19:46:19.609115+0000 mon.a (mon.0) 7710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:20 smithi115 bash[46934]: audit 2022-04-22T19:46:19.609115+0000 mon.a (mon.0) 7710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:20 smithi115 bash[45495]: audit 2022-04-22T19:46:19.609115+0000 mon.a (mon.0) 7710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:21 smithi121 bash[36902]: cluster 2022-04-22T19:46:20.144878+0000 mgr.y (mgr.34107) 4815 : cluster [DBG] pgmap v6150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:21 smithi121 bash[36902]: audit 2022-04-22T19:46:21.029850+0000 mon.a (mon.0) 7711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:21 smithi115 bash[45495]: cluster 2022-04-22T19:46:20.144878+0000 mgr.y (mgr.34107) 4815 : cluster [DBG] pgmap v6150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:21 smithi115 bash[45495]: audit 2022-04-22T19:46:21.029850+0000 mon.a (mon.0) 7711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:21 smithi115 bash[46934]: cluster 2022-04-22T19:46:20.144878+0000 mgr.y (mgr.34107) 4815 : cluster [DBG] pgmap v6150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:21 smithi115 bash[46934]: audit 2022-04-22T19:46:21.029850+0000 mon.a (mon.0) 7711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:22.158Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:23 smithi121 bash[36902]: cluster 2022-04-22T19:46:22.146093+0000 mgr.y (mgr.34107) 4816 : cluster [DBG] pgmap v6151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:23 smithi121 bash[36902]: audit 2022-04-22T19:46:22.501010+0000 mon.a (mon.0) 7712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:23 smithi115 bash[45495]: cluster 2022-04-22T19:46:22.146093+0000 mgr.y (mgr.34107) 4816 : cluster [DBG] pgmap v6151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:23 smithi115 bash[45495]: audit 2022-04-22T19:46:22.501010+0000 mon.a (mon.0) 7712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:23 smithi115 bash[46934]: cluster 2022-04-22T19:46:22.146093+0000 mgr.y (mgr.34107) 4816 : cluster [DBG] pgmap v6151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:23 smithi115 bash[46934]: audit 2022-04-22T19:46:22.501010+0000 mon.a (mon.0) 7712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:24.028Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:25 smithi121 bash[36902]: cluster 2022-04-22T19:46:24.147160+0000 mgr.y (mgr.34107) 4817 : cluster [DBG] pgmap v6152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:25 smithi121 bash[36902]: audit 2022-04-22T19:46:24.612809+0000 mon.a (mon.0) 7713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:25 smithi115 bash[45495]: cluster 2022-04-22T19:46:24.147160+0000 mgr.y (mgr.34107) 4817 : cluster [DBG] pgmap v6152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:25 smithi115 bash[45495]: audit 2022-04-22T19:46:24.612809+0000 mon.a (mon.0) 7713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:25 smithi115 bash[46934]: cluster 2022-04-22T19:46:24.147160+0000 mgr.y (mgr.34107) 4817 : cluster [DBG] pgmap v6152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:25 smithi115 bash[46934]: audit 2022-04-22T19:46:24.612809+0000 mon.a (mon.0) 7713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:26 smithi121 bash[36902]: audit 2022-04-22T19:46:26.032855+0000 mon.a (mon.0) 7714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:26 smithi115 bash[45495]: audit 2022-04-22T19:46:26.032855+0000 mon.a (mon.0) 7714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:26 smithi115 bash[46934]: audit 2022-04-22T19:46:26.032855+0000 mon.a (mon.0) 7714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:27.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[46934]: cluster 2022-04-22T19:46:26.148200+0000 mgr.y (mgr.34107) 4818 : cluster [DBG] pgmap v6153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:27.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[46934]: audit 2022-04-22T19:46:27.091006+0000 mon.a (mon.0) 7715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:46:27.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[46934]: audit 2022-04-22T19:46:27.094307+0000 mon.a (mon.0) 7716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:46:27.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[46934]: audit 2022-04-22T19:46:27.096350+0000 mon.a (mon.0) 7717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:46:27.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[45495]: cluster 2022-04-22T19:46:26.148200+0000 mgr.y (mgr.34107) 4818 : cluster [DBG] pgmap v6153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:27.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[45495]: audit 2022-04-22T19:46:27.091006+0000 mon.a (mon.0) 7715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:46:27.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[45495]: audit 2022-04-22T19:46:27.094307+0000 mon.a (mon.0) 7716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:46:27.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:27 smithi115 bash[45495]: audit 2022-04-22T19:46:27.096350+0000 mon.a (mon.0) 7717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:46:27.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:27 smithi121 bash[36902]: cluster 2022-04-22T19:46:26.148200+0000 mgr.y (mgr.34107) 4818 : cluster [DBG] pgmap v6153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:27.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:27 smithi121 bash[36902]: audit 2022-04-22T19:46:27.091006+0000 mon.a (mon.0) 7715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:46:27.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:27 smithi121 bash[36902]: audit 2022-04-22T19:46:27.094307+0000 mon.a (mon.0) 7716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:46:27.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:27 smithi121 bash[36902]: audit 2022-04-22T19:46:27.096350+0000 mon.a (mon.0) 7717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:46:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:28 smithi121 bash[36902]: audit 2022-04-22T19:46:27.472642+0000 mon.a (mon.0) 7718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:46:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:28 smithi121 bash[36902]: audit 2022-04-22T19:46:27.504036+0000 mon.a (mon.0) 7719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:28 smithi115 bash[45495]: audit 2022-04-22T19:46:27.472642+0000 mon.a (mon.0) 7718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:46:28.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:28 smithi115 bash[45495]: audit 2022-04-22T19:46:27.504036+0000 mon.a (mon.0) 7719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:28 smithi115 bash[46934]: audit 2022-04-22T19:46:27.472642+0000 mon.a (mon.0) 7718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:46:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:28 smithi115 bash[46934]: audit 2022-04-22T19:46:27.504036+0000 mon.a (mon.0) 7719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:29 smithi121 bash[36902]: cluster 2022-04-22T19:46:28.149270+0000 mgr.y (mgr.34107) 4819 : cluster [DBG] pgmap v6154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:29 smithi115 bash[45495]: cluster 2022-04-22T19:46:28.149270+0000 mgr.y (mgr.34107) 4819 : cluster [DBG] pgmap v6154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:29 smithi115 bash[46934]: cluster 2022-04-22T19:46:28.149270+0000 mgr.y (mgr.34107) 4819 : cluster [DBG] pgmap v6154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:30 smithi121 bash[36902]: audit 2022-04-22T19:46:29.616366+0000 mon.a (mon.0) 7720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:30 smithi115 bash[45495]: audit 2022-04-22T19:46:29.616366+0000 mon.a (mon.0) 7720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:30 smithi115 bash[46934]: audit 2022-04-22T19:46:29.616366+0000 mon.a (mon.0) 7720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:31 smithi121 bash[36902]: cluster 2022-04-22T19:46:30.150389+0000 mgr.y (mgr.34107) 4820 : cluster [DBG] pgmap v6155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:31 smithi121 bash[36902]: audit 2022-04-22T19:46:31.035953+0000 mon.a (mon.0) 7721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:31 smithi115 bash[45495]: cluster 2022-04-22T19:46:30.150389+0000 mgr.y (mgr.34107) 4820 : cluster [DBG] pgmap v6155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:31 smithi115 bash[45495]: audit 2022-04-22T19:46:31.035953+0000 mon.a (mon.0) 7721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:31 smithi115 bash[46934]: cluster 2022-04-22T19:46:30.150389+0000 mgr.y (mgr.34107) 4820 : cluster [DBG] pgmap v6155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:31 smithi115 bash[46934]: audit 2022-04-22T19:46:31.035953+0000 mon.a (mon.0) 7721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:32.158Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:33 smithi121 bash[36902]: cluster 2022-04-22T19:46:32.151461+0000 mgr.y (mgr.34107) 4821 : cluster [DBG] pgmap v6156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:33 smithi121 bash[36902]: audit 2022-04-22T19:46:32.506742+0000 mon.a (mon.0) 7722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:33 smithi115 bash[45495]: cluster 2022-04-22T19:46:32.151461+0000 mgr.y (mgr.34107) 4821 : cluster [DBG] pgmap v6156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:33 smithi115 bash[45495]: audit 2022-04-22T19:46:32.506742+0000 mon.a (mon.0) 7722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:33 smithi115 bash[46934]: cluster 2022-04-22T19:46:32.151461+0000 mgr.y (mgr.34107) 4821 : cluster [DBG] pgmap v6156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:33 smithi115 bash[46934]: audit 2022-04-22T19:46:32.506742+0000 mon.a (mon.0) 7722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:34.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:35 smithi121 bash[36902]: cluster 2022-04-22T19:46:34.152530+0000 mgr.y (mgr.34107) 4822 : cluster [DBG] pgmap v6157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:35 smithi121 bash[36902]: audit 2022-04-22T19:46:34.620040+0000 mon.a (mon.0) 7723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:35 smithi115 bash[45495]: cluster 2022-04-22T19:46:34.152530+0000 mgr.y (mgr.34107) 4822 : cluster [DBG] pgmap v6157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:35 smithi115 bash[45495]: audit 2022-04-22T19:46:34.620040+0000 mon.a (mon.0) 7723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:35 smithi115 bash[46934]: cluster 2022-04-22T19:46:34.152530+0000 mgr.y (mgr.34107) 4822 : cluster [DBG] pgmap v6157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:35 smithi115 bash[46934]: audit 2022-04-22T19:46:34.620040+0000 mon.a (mon.0) 7723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:36 smithi115 bash[45495]: audit 2022-04-22T19:46:36.038792+0000 mon.a (mon.0) 7724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:36.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:36 smithi115 bash[46934]: audit 2022-04-22T19:46:36.038792+0000 mon.a (mon.0) 7724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:36 smithi121 bash[36902]: audit 2022-04-22T19:46:36.038792+0000 mon.a (mon.0) 7724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:37 smithi115 bash[45495]: cluster 2022-04-22T19:46:36.153624+0000 mgr.y (mgr.34107) 4823 : cluster [DBG] pgmap v6158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:37.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:37 smithi115 bash[45495]: audit 2022-04-22T19:46:37.509727+0000 mon.a (mon.0) 7725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:37 smithi115 bash[46934]: cluster 2022-04-22T19:46:36.153624+0000 mgr.y (mgr.34107) 4823 : cluster [DBG] pgmap v6158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:37.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:37 smithi115 bash[46934]: audit 2022-04-22T19:46:37.509727+0000 mon.a (mon.0) 7725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:37 smithi121 bash[36902]: cluster 2022-04-22T19:46:36.153624+0000 mgr.y (mgr.34107) 4823 : cluster [DBG] pgmap v6158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:37 smithi121 bash[36902]: audit 2022-04-22T19:46:37.509727+0000 mon.a (mon.0) 7725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:39 smithi115 bash[45495]: cluster 2022-04-22T19:46:38.154679+0000 mgr.y (mgr.34107) 4824 : cluster [DBG] pgmap v6159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:39 smithi115 bash[46934]: cluster 2022-04-22T19:46:38.154679+0000 mgr.y (mgr.34107) 4824 : cluster [DBG] pgmap v6159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:39 smithi121 bash[36902]: cluster 2022-04-22T19:46:38.154679+0000 mgr.y (mgr.34107) 4824 : cluster [DBG] pgmap v6159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:40 smithi115 bash[45495]: audit 2022-04-22T19:46:39.623896+0000 mon.a (mon.0) 7726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:40 smithi115 bash[46934]: audit 2022-04-22T19:46:39.623896+0000 mon.a (mon.0) 7726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:40 smithi121 bash[36902]: audit 2022-04-22T19:46:39.623896+0000 mon.a (mon.0) 7726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:41 smithi115 bash[45495]: cluster 2022-04-22T19:46:40.155910+0000 mgr.y (mgr.34107) 4825 : cluster [DBG] pgmap v6160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:41 smithi115 bash[45495]: audit 2022-04-22T19:46:41.041906+0000 mon.a (mon.0) 7727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:41 smithi115 bash[46934]: cluster 2022-04-22T19:46:40.155910+0000 mgr.y (mgr.34107) 4825 : cluster [DBG] pgmap v6160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:41.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:41 smithi115 bash[46934]: audit 2022-04-22T19:46:41.041906+0000 mon.a (mon.0) 7727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:41 smithi121 bash[36902]: cluster 2022-04-22T19:46:40.155910+0000 mgr.y (mgr.34107) 4825 : cluster [DBG] pgmap v6160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:41 smithi121 bash[36902]: audit 2022-04-22T19:46:41.041906+0000 mon.a (mon.0) 7727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:42.159Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:42.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:42 smithi115 bash[46934]: audit 2022-04-22T19:46:42.512751+0000 mon.a (mon.0) 7728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:42.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:42 smithi115 bash[45495]: audit 2022-04-22T19:46:42.512751+0000 mon.a (mon.0) 7728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:42.838 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:46:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:42 smithi121 bash[36902]: audit 2022-04-22T19:46:42.512751+0000 mon.a (mon.0) 7728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:43.176 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:46:43.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:46:43.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:46:43.179 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:46:43.587 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:46:43.587 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:46:43.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:43.588 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:46:43.589 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:46:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:43 smithi121 bash[36902]: cluster 2022-04-22T19:46:42.156940+0000 mgr.y (mgr.34107) 4826 : cluster [DBG] pgmap v6161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:44.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:43 smithi115 bash[45495]: cluster 2022-04-22T19:46:42.156940+0000 mgr.y (mgr.34107) 4826 : cluster [DBG] pgmap v6161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:44.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:43 smithi115 bash[46934]: cluster 2022-04-22T19:46:42.156940+0000 mgr.y (mgr.34107) 4826 : cluster [DBG] pgmap v6161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:44.028Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:44 smithi121 bash[36902]: audit 2022-04-22T19:46:42.813386+0000 mgr.y (mgr.34107) 4827 : audit [DBG] from='client.50362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:44 smithi121 bash[36902]: audit 2022-04-22T19:46:43.152261+0000 mgr.y (mgr.34107) 4828 : audit [DBG] from='client.40434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:44 smithi121 bash[36902]: audit 2022-04-22T19:46:43.577463+0000 mon.c (mon.1) 208 : audit [DBG] from='client.? 172.21.15.115:0/3390532012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[45495]: audit 2022-04-22T19:46:42.813386+0000 mgr.y (mgr.34107) 4827 : audit [DBG] from='client.50362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[45495]: audit 2022-04-22T19:46:43.152261+0000 mgr.y (mgr.34107) 4828 : audit [DBG] from='client.40434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[45495]: audit 2022-04-22T19:46:43.577463+0000 mon.c (mon.1) 208 : audit [DBG] from='client.? 172.21.15.115:0/3390532012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:45.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[46934]: audit 2022-04-22T19:46:42.813386+0000 mgr.y (mgr.34107) 4827 : audit [DBG] from='client.50362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[46934]: audit 2022-04-22T19:46:43.152261+0000 mgr.y (mgr.34107) 4828 : audit [DBG] from='client.40434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:46:45.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:44 smithi115 bash[46934]: audit 2022-04-22T19:46:43.577463+0000 mon.c (mon.1) 208 : audit [DBG] from='client.? 172.21.15.115:0/3390532012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:46:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:45 smithi121 bash[36902]: cluster 2022-04-22T19:46:44.157960+0000 mgr.y (mgr.34107) 4829 : cluster [DBG] pgmap v6162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:45 smithi121 bash[36902]: audit 2022-04-22T19:46:44.627455+0000 mon.a (mon.0) 7729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:45 smithi115 bash[45495]: cluster 2022-04-22T19:46:44.157960+0000 mgr.y (mgr.34107) 4829 : cluster [DBG] pgmap v6162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:45 smithi115 bash[45495]: audit 2022-04-22T19:46:44.627455+0000 mon.a (mon.0) 7729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:45 smithi115 bash[46934]: cluster 2022-04-22T19:46:44.157960+0000 mgr.y (mgr.34107) 4829 : cluster [DBG] pgmap v6162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:45 smithi115 bash[46934]: audit 2022-04-22T19:46:44.627455+0000 mon.a (mon.0) 7729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:46 smithi121 bash[36902]: audit 2022-04-22T19:46:46.045010+0000 mon.a (mon.0) 7730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:47.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:46 smithi115 bash[45495]: audit 2022-04-22T19:46:46.045010+0000 mon.a (mon.0) 7730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:46 smithi115 bash[46934]: audit 2022-04-22T19:46:46.045010+0000 mon.a (mon.0) 7730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:47 smithi121 bash[36902]: cluster 2022-04-22T19:46:46.159036+0000 mgr.y (mgr.34107) 4830 : cluster [DBG] pgmap v6163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:47 smithi121 bash[36902]: audit 2022-04-22T19:46:47.515697+0000 mon.a (mon.0) 7731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:47 smithi115 bash[46934]: cluster 2022-04-22T19:46:46.159036+0000 mgr.y (mgr.34107) 4830 : cluster [DBG] pgmap v6163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:47 smithi115 bash[46934]: audit 2022-04-22T19:46:47.515697+0000 mon.a (mon.0) 7731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:47 smithi115 bash[45495]: cluster 2022-04-22T19:46:46.159036+0000 mgr.y (mgr.34107) 4830 : cluster [DBG] pgmap v6163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:48.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:47 smithi115 bash[45495]: audit 2022-04-22T19:46:47.515697+0000 mon.a (mon.0) 7731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:49 smithi121 bash[36902]: cluster 2022-04-22T19:46:48.160234+0000 mgr.y (mgr.34107) 4831 : cluster [DBG] pgmap v6164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:49 smithi115 bash[46934]: cluster 2022-04-22T19:46:48.160234+0000 mgr.y (mgr.34107) 4831 : cluster [DBG] pgmap v6164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:49 smithi115 bash[45495]: cluster 2022-04-22T19:46:48.160234+0000 mgr.y (mgr.34107) 4831 : cluster [DBG] pgmap v6164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:50 smithi121 bash[36902]: audit 2022-04-22T19:46:49.631232+0000 mon.a (mon.0) 7732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:50 smithi115 bash[45495]: audit 2022-04-22T19:46:49.631232+0000 mon.a (mon.0) 7732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:50 smithi115 bash[46934]: audit 2022-04-22T19:46:49.631232+0000 mon.a (mon.0) 7732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:51 smithi121 bash[36902]: cluster 2022-04-22T19:46:50.161236+0000 mgr.y (mgr.34107) 4832 : cluster [DBG] pgmap v6165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:52.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:51 smithi121 bash[36902]: audit 2022-04-22T19:46:51.048166+0000 mon.a (mon.0) 7733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:51 smithi115 bash[45495]: cluster 2022-04-22T19:46:50.161236+0000 mgr.y (mgr.34107) 4832 : cluster [DBG] pgmap v6165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:51 smithi115 bash[45495]: audit 2022-04-22T19:46:51.048166+0000 mon.a (mon.0) 7733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:51 smithi115 bash[46934]: cluster 2022-04-22T19:46:50.161236+0000 mgr.y (mgr.34107) 4832 : cluster [DBG] pgmap v6165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:51 smithi115 bash[46934]: audit 2022-04-22T19:46:51.048166+0000 mon.a (mon.0) 7733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:52.158Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:52 smithi121 bash[36902]: audit 2022-04-22T19:46:52.518725+0000 mon.a (mon.0) 7734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:53.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:52 smithi115 bash[45495]: audit 2022-04-22T19:46:52.518725+0000 mon.a (mon.0) 7734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:52 smithi115 bash[46934]: audit 2022-04-22T19:46:52.518725+0000 mon.a (mon.0) 7734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:53 smithi121 bash[36902]: cluster 2022-04-22T19:46:52.162234+0000 mgr.y (mgr.34107) 4833 : cluster [DBG] pgmap v6166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:54.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:53 smithi115 bash[45495]: cluster 2022-04-22T19:46:52.162234+0000 mgr.y (mgr.34107) 4833 : cluster [DBG] pgmap v6166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:54.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:53 smithi115 bash[46934]: cluster 2022-04-22T19:46:52.162234+0000 mgr.y (mgr.34107) 4833 : cluster [DBG] pgmap v6166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:46:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:46:54.028Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:46:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:55 smithi121 bash[36902]: cluster 2022-04-22T19:46:54.163304+0000 mgr.y (mgr.34107) 4834 : cluster [DBG] pgmap v6167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:55 smithi121 bash[36902]: audit 2022-04-22T19:46:54.634957+0000 mon.a (mon.0) 7735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:55 smithi115 bash[45495]: cluster 2022-04-22T19:46:54.163304+0000 mgr.y (mgr.34107) 4834 : cluster [DBG] pgmap v6167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:55 smithi115 bash[45495]: audit 2022-04-22T19:46:54.634957+0000 mon.a (mon.0) 7735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:55 smithi115 bash[46934]: cluster 2022-04-22T19:46:54.163304+0000 mgr.y (mgr.34107) 4834 : cluster [DBG] pgmap v6167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:55 smithi115 bash[46934]: audit 2022-04-22T19:46:54.634957+0000 mon.a (mon.0) 7735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:46:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:56 smithi121 bash[36902]: audit 2022-04-22T19:46:56.051097+0000 mon.a (mon.0) 7736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:57.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:56 smithi115 bash[45495]: audit 2022-04-22T19:46:56.051097+0000 mon.a (mon.0) 7736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:57.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:56 smithi115 bash[46934]: audit 2022-04-22T19:46:56.051097+0000 mon.a (mon.0) 7736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:46:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:57 smithi121 bash[36902]: cluster 2022-04-22T19:46:56.164324+0000 mgr.y (mgr.34107) 4835 : cluster [DBG] pgmap v6168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:57 smithi121 bash[36902]: audit 2022-04-22T19:46:57.521689+0000 mon.a (mon.0) 7737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:57 smithi115 bash[45495]: cluster 2022-04-22T19:46:56.164324+0000 mgr.y (mgr.34107) 4835 : cluster [DBG] pgmap v6168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:57 smithi115 bash[45495]: audit 2022-04-22T19:46:57.521689+0000 mon.a (mon.0) 7737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:46:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:57 smithi115 bash[46934]: cluster 2022-04-22T19:46:56.164324+0000 mgr.y (mgr.34107) 4835 : cluster [DBG] pgmap v6168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:46:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:57 smithi115 bash[46934]: audit 2022-04-22T19:46:57.521689+0000 mon.a (mon.0) 7737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:46:59 smithi121 bash[36902]: cluster 2022-04-22T19:46:58.165335+0000 mgr.y (mgr.34107) 4836 : cluster [DBG] pgmap v6169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:00.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:46:59 smithi115 bash[45495]: cluster 2022-04-22T19:46:58.165335+0000 mgr.y (mgr.34107) 4836 : cluster [DBG] pgmap v6169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:00.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:46:59 smithi115 bash[46934]: cluster 2022-04-22T19:46:58.165335+0000 mgr.y (mgr.34107) 4836 : cluster [DBG] pgmap v6169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:00 smithi121 bash[36902]: audit 2022-04-22T19:46:59.638650+0000 mon.a (mon.0) 7738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:01.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:00 smithi115 bash[45495]: audit 2022-04-22T19:46:59.638650+0000 mon.a (mon.0) 7738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:01.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:00 smithi115 bash[46934]: audit 2022-04-22T19:46:59.638650+0000 mon.a (mon.0) 7738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:01 smithi121 bash[36902]: cluster 2022-04-22T19:47:00.166468+0000 mgr.y (mgr.34107) 4837 : cluster [DBG] pgmap v6170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:01 smithi121 bash[36902]: audit 2022-04-22T19:47:01.054280+0000 mon.a (mon.0) 7739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:01 smithi115 bash[45495]: cluster 2022-04-22T19:47:00.166468+0000 mgr.y (mgr.34107) 4837 : cluster [DBG] pgmap v6170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:01 smithi115 bash[45495]: audit 2022-04-22T19:47:01.054280+0000 mon.a (mon.0) 7739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:01 smithi115 bash[46934]: cluster 2022-04-22T19:47:00.166468+0000 mgr.y (mgr.34107) 4837 : cluster [DBG] pgmap v6170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:02.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:01 smithi115 bash[46934]: audit 2022-04-22T19:47:01.054280+0000 mon.a (mon.0) 7739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:02.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:02 smithi121 bash[36902]: audit 2022-04-22T19:47:02.525060+0000 mon.a (mon.0) 7740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:02 smithi115 bash[45495]: audit 2022-04-22T19:47:02.525060+0000 mon.a (mon.0) 7740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:02 smithi115 bash[46934]: audit 2022-04-22T19:47:02.525060+0000 mon.a (mon.0) 7740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:03 smithi121 bash[36902]: cluster 2022-04-22T19:47:02.167511+0000 mgr.y (mgr.34107) 4838 : cluster [DBG] pgmap v6171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:04.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:03 smithi115 bash[45495]: cluster 2022-04-22T19:47:02.167511+0000 mgr.y (mgr.34107) 4838 : cluster [DBG] pgmap v6171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:04.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:03 smithi115 bash[46934]: cluster 2022-04-22T19:47:02.167511+0000 mgr.y (mgr.34107) 4838 : cluster [DBG] pgmap v6171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:04.029Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:05 smithi121 bash[36902]: cluster 2022-04-22T19:47:04.168533+0000 mgr.y (mgr.34107) 4839 : cluster [DBG] pgmap v6172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:05 smithi121 bash[36902]: audit 2022-04-22T19:47:04.641960+0000 mon.a (mon.0) 7741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:05 smithi115 bash[45495]: cluster 2022-04-22T19:47:04.168533+0000 mgr.y (mgr.34107) 4839 : cluster [DBG] pgmap v6172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:06.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:05 smithi115 bash[45495]: audit 2022-04-22T19:47:04.641960+0000 mon.a (mon.0) 7741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:05 smithi115 bash[46934]: cluster 2022-04-22T19:47:04.168533+0000 mgr.y (mgr.34107) 4839 : cluster [DBG] pgmap v6172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:06.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:05 smithi115 bash[46934]: audit 2022-04-22T19:47:04.641960+0000 mon.a (mon.0) 7741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:06 smithi121 bash[36902]: audit 2022-04-22T19:47:06.057419+0000 mon.a (mon.0) 7742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:06 smithi115 bash[46934]: audit 2022-04-22T19:47:06.057419+0000 mon.a (mon.0) 7742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:06 smithi115 bash[45495]: audit 2022-04-22T19:47:06.057419+0000 mon.a (mon.0) 7742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:07 smithi121 bash[36902]: cluster 2022-04-22T19:47:06.169622+0000 mgr.y (mgr.34107) 4840 : cluster [DBG] pgmap v6173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:07 smithi121 bash[36902]: audit 2022-04-22T19:47:07.528095+0000 mon.a (mon.0) 7743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:07 smithi115 bash[45495]: cluster 2022-04-22T19:47:06.169622+0000 mgr.y (mgr.34107) 4840 : cluster [DBG] pgmap v6173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:07 smithi115 bash[45495]: audit 2022-04-22T19:47:07.528095+0000 mon.a (mon.0) 7743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:08.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:07 smithi115 bash[46934]: cluster 2022-04-22T19:47:06.169622+0000 mgr.y (mgr.34107) 4840 : cluster [DBG] pgmap v6173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:08.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:07 smithi115 bash[46934]: audit 2022-04-22T19:47:07.528095+0000 mon.a (mon.0) 7743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:09 smithi121 bash[36902]: cluster 2022-04-22T19:47:08.170820+0000 mgr.y (mgr.34107) 4841 : cluster [DBG] pgmap v6174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:09 smithi121 bash[36902]: audit 2022-04-22T19:47:09.645530+0000 mon.a (mon.0) 7744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:09 smithi115 bash[45495]: cluster 2022-04-22T19:47:08.170820+0000 mgr.y (mgr.34107) 4841 : cluster [DBG] pgmap v6174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:10.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:09 smithi115 bash[45495]: audit 2022-04-22T19:47:09.645530+0000 mon.a (mon.0) 7744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:10.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:09 smithi115 bash[46934]: cluster 2022-04-22T19:47:08.170820+0000 mgr.y (mgr.34107) 4841 : cluster [DBG] pgmap v6174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:10.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:09 smithi115 bash[46934]: audit 2022-04-22T19:47:09.645530+0000 mon.a (mon.0) 7744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:11 smithi121 bash[36902]: cluster 2022-04-22T19:47:10.172034+0000 mgr.y (mgr.34107) 4842 : cluster [DBG] pgmap v6175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:11 smithi121 bash[36902]: audit 2022-04-22T19:47:11.060564+0000 mon.a (mon.0) 7745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:11 smithi115 bash[45495]: cluster 2022-04-22T19:47:10.172034+0000 mgr.y (mgr.34107) 4842 : cluster [DBG] pgmap v6175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:11 smithi115 bash[45495]: audit 2022-04-22T19:47:11.060564+0000 mon.a (mon.0) 7745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:11 smithi115 bash[46934]: cluster 2022-04-22T19:47:10.172034+0000 mgr.y (mgr.34107) 4842 : cluster [DBG] pgmap v6175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:12.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:11 smithi115 bash[46934]: audit 2022-04-22T19:47:11.060564+0000 mon.a (mon.0) 7745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:12.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:12 smithi121 bash[36902]: audit 2022-04-22T19:47:12.530943+0000 mon.a (mon.0) 7746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:12 smithi115 bash[46934]: audit 2022-04-22T19:47:12.530943+0000 mon.a (mon.0) 7746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:13.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:12 smithi115 bash[45495]: audit 2022-04-22T19:47:12.530943+0000 mon.a (mon.0) 7746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:13.966 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:47:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:13 smithi121 bash[36902]: cluster 2022-04-22T19:47:12.173095+0000 mgr.y (mgr.34107) 4843 : cluster [DBG] pgmap v6176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:14.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:13 smithi115 bash[46934]: cluster 2022-04-22T19:47:12.173095+0000 mgr.y (mgr.34107) 4843 : cluster [DBG] pgmap v6176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:14.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:13 smithi115 bash[45495]: cluster 2022-04-22T19:47:12.173095+0000 mgr.y (mgr.34107) 4843 : cluster [DBG] pgmap v6176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:14.029Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:47:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:47:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:47:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:47:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:47:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:47:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:47:14.731 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:47:14.731 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:47:14.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:47:14.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:47:14.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:47:14.733 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:47:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:14 smithi121 bash[36902]: audit 2022-04-22T19:47:14.649212+0000 mon.a (mon.0) 7747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:15.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:14 smithi115 bash[45495]: audit 2022-04-22T19:47:14.649212+0000 mon.a (mon.0) 7747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:14 smithi115 bash[46934]: audit 2022-04-22T19:47:14.649212+0000 mon.a (mon.0) 7747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: audit 2022-04-22T19:47:13.940373+0000 mgr.y (mgr.34107) 4844 : audit [DBG] from='client.50380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: cluster 2022-04-22T19:47:14.173796+0000 mgr.y (mgr.34107) 4845 : cluster [DBG] pgmap v6177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: audit 2022-04-22T19:47:14.283986+0000 mgr.y (mgr.34107) 4846 : audit [DBG] from='client.40452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: audit 2022-04-22T19:47:14.721282+0000 mon.a (mon.0) 7748 : audit [DBG] from='client.? 172.21.15.115:0/3153122377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: audit 2022-04-22T19:47:14.897600+0000 mon.a (mon.0) 7749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:47:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:15 smithi121 bash[36902]: audit 2022-04-22T19:47:14.899271+0000 mon.a (mon.0) 7750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:47:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: audit 2022-04-22T19:47:13.940373+0000 mgr.y (mgr.34107) 4844 : audit [DBG] from='client.50380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: cluster 2022-04-22T19:47:14.173796+0000 mgr.y (mgr.34107) 4845 : cluster [DBG] pgmap v6177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: audit 2022-04-22T19:47:14.283986+0000 mgr.y (mgr.34107) 4846 : audit [DBG] from='client.40452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: audit 2022-04-22T19:47:14.721282+0000 mon.a (mon.0) 7748 : audit [DBG] from='client.? 172.21.15.115:0/3153122377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: audit 2022-04-22T19:47:14.897600+0000 mon.a (mon.0) 7749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[45495]: audit 2022-04-22T19:47:14.899271+0000 mon.a (mon.0) 7750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: audit 2022-04-22T19:47:13.940373+0000 mgr.y (mgr.34107) 4844 : audit [DBG] from='client.50380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: cluster 2022-04-22T19:47:14.173796+0000 mgr.y (mgr.34107) 4845 : cluster [DBG] pgmap v6177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: audit 2022-04-22T19:47:14.283986+0000 mgr.y (mgr.34107) 4846 : audit [DBG] from='client.40452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: audit 2022-04-22T19:47:14.721282+0000 mon.a (mon.0) 7748 : audit [DBG] from='client.? 172.21.15.115:0/3153122377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: audit 2022-04-22T19:47:14.897600+0000 mon.a (mon.0) 7749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:47:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:15 smithi115 bash[46934]: audit 2022-04-22T19:47:14.899271+0000 mon.a (mon.0) 7750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:47:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:16 smithi121 bash[36902]: audit 2022-04-22T19:47:16.063655+0000 mon.a (mon.0) 7751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:16 smithi115 bash[45495]: audit 2022-04-22T19:47:16.063655+0000 mon.a (mon.0) 7751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:16 smithi115 bash[46934]: audit 2022-04-22T19:47:16.063655+0000 mon.a (mon.0) 7751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:17 smithi121 bash[36902]: cluster 2022-04-22T19:47:16.174822+0000 mgr.y (mgr.34107) 4847 : cluster [DBG] pgmap v6178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:17 smithi121 bash[36902]: audit 2022-04-22T19:47:17.534202+0000 mon.a (mon.0) 7752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:17 smithi115 bash[46934]: cluster 2022-04-22T19:47:16.174822+0000 mgr.y (mgr.34107) 4847 : cluster [DBG] pgmap v6178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:17 smithi115 bash[46934]: audit 2022-04-22T19:47:17.534202+0000 mon.a (mon.0) 7752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:17 smithi115 bash[45495]: cluster 2022-04-22T19:47:16.174822+0000 mgr.y (mgr.34107) 4847 : cluster [DBG] pgmap v6178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:17 smithi115 bash[45495]: audit 2022-04-22T19:47:17.534202+0000 mon.a (mon.0) 7752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:19 smithi121 bash[36902]: cluster 2022-04-22T19:47:18.175983+0000 mgr.y (mgr.34107) 4848 : cluster [DBG] pgmap v6179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:19 smithi121 bash[36902]: audit 2022-04-22T19:47:19.652883+0000 mon.a (mon.0) 7753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:19 smithi115 bash[46934]: cluster 2022-04-22T19:47:18.175983+0000 mgr.y (mgr.34107) 4848 : cluster [DBG] pgmap v6179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:19 smithi115 bash[46934]: audit 2022-04-22T19:47:19.652883+0000 mon.a (mon.0) 7753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:19 smithi115 bash[45495]: cluster 2022-04-22T19:47:18.175983+0000 mgr.y (mgr.34107) 4848 : cluster [DBG] pgmap v6179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:19 smithi115 bash[45495]: audit 2022-04-22T19:47:19.652883+0000 mon.a (mon.0) 7753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:20 smithi115 bash[45495]: cluster 2022-04-22T19:47:20.177192+0000 mgr.y (mgr.34107) 4849 : cluster [DBG] pgmap v6180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:20 smithi115 bash[46934]: cluster 2022-04-22T19:47:20.177192+0000 mgr.y (mgr.34107) 4849 : cluster [DBG] pgmap v6180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:20 smithi121 bash[36902]: cluster 2022-04-22T19:47:20.177192+0000 mgr.y (mgr.34107) 4849 : cluster [DBG] pgmap v6180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:21 smithi115 bash[45495]: audit 2022-04-22T19:47:21.066813+0000 mon.a (mon.0) 7754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:21 smithi115 bash[46934]: audit 2022-04-22T19:47:21.066813+0000 mon.a (mon.0) 7754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:21 smithi121 bash[36902]: audit 2022-04-22T19:47:21.066813+0000 mon.a (mon.0) 7754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:22.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:22 smithi115 bash[45495]: cluster 2022-04-22T19:47:22.178132+0000 mgr.y (mgr.34107) 4850 : cluster [DBG] pgmap v6181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:22 smithi115 bash[45495]: audit 2022-04-22T19:47:22.537205+0000 mon.a (mon.0) 7755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:22 smithi115 bash[46934]: cluster 2022-04-22T19:47:22.178132+0000 mgr.y (mgr.34107) 4850 : cluster [DBG] pgmap v6181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:23.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:22 smithi115 bash[46934]: audit 2022-04-22T19:47:22.537205+0000 mon.a (mon.0) 7755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:22 smithi121 bash[36902]: cluster 2022-04-22T19:47:22.178132+0000 mgr.y (mgr.34107) 4850 : cluster [DBG] pgmap v6181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:22 smithi121 bash[36902]: audit 2022-04-22T19:47:22.537205+0000 mon.a (mon.0) 7755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:24.030Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:25 smithi121 bash[36902]: cluster 2022-04-22T19:47:24.179084+0000 mgr.y (mgr.34107) 4851 : cluster [DBG] pgmap v6182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:25 smithi121 bash[36902]: audit 2022-04-22T19:47:24.656525+0000 mon.a (mon.0) 7756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:25 smithi115 bash[45495]: cluster 2022-04-22T19:47:24.179084+0000 mgr.y (mgr.34107) 4851 : cluster [DBG] pgmap v6182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:25.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:25 smithi115 bash[45495]: audit 2022-04-22T19:47:24.656525+0000 mon.a (mon.0) 7756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:25 smithi115 bash[46934]: cluster 2022-04-22T19:47:24.179084+0000 mgr.y (mgr.34107) 4851 : cluster [DBG] pgmap v6182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:25 smithi115 bash[46934]: audit 2022-04-22T19:47:24.656525+0000 mon.a (mon.0) 7756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:26 smithi121 bash[36902]: audit 2022-04-22T19:47:26.070223+0000 mon.a (mon.0) 7757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:26 smithi115 bash[45495]: audit 2022-04-22T19:47:26.070223+0000 mon.a (mon.0) 7757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:26.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:26 smithi115 bash[46934]: audit 2022-04-22T19:47:26.070223+0000 mon.a (mon.0) 7757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:27 smithi121 bash[36902]: cluster 2022-04-22T19:47:26.180145+0000 mgr.y (mgr.34107) 4852 : cluster [DBG] pgmap v6183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:27.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:27 smithi115 bash[46934]: cluster 2022-04-22T19:47:26.180145+0000 mgr.y (mgr.34107) 4852 : cluster [DBG] pgmap v6183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:27.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:27 smithi115 bash[45495]: cluster 2022-04-22T19:47:26.180145+0000 mgr.y (mgr.34107) 4852 : cluster [DBG] pgmap v6183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:28 smithi121 bash[36902]: audit 2022-04-22T19:47:27.482963+0000 mon.a (mon.0) 7758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:47:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:28 smithi121 bash[36902]: audit 2022-04-22T19:47:27.485909+0000 mon.a (mon.0) 7759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:47:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:28 smithi121 bash[36902]: audit 2022-04-22T19:47:27.487653+0000 mon.a (mon.0) 7760 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:47:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:28 smithi121 bash[36902]: audit 2022-04-22T19:47:27.539993+0000 mon.a (mon.0) 7761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:28 smithi121 bash[36902]: audit 2022-04-22T19:47:27.829209+0000 mon.a (mon.0) 7762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:47:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[45495]: audit 2022-04-22T19:47:27.482963+0000 mon.a (mon.0) 7758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:47:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[45495]: audit 2022-04-22T19:47:27.485909+0000 mon.a (mon.0) 7759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:47:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[45495]: audit 2022-04-22T19:47:27.487653+0000 mon.a (mon.0) 7760 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:47:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[45495]: audit 2022-04-22T19:47:27.539993+0000 mon.a (mon.0) 7761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[45495]: audit 2022-04-22T19:47:27.829209+0000 mon.a (mon.0) 7762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[46934]: audit 2022-04-22T19:47:27.482963+0000 mon.a (mon.0) 7758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[46934]: audit 2022-04-22T19:47:27.485909+0000 mon.a (mon.0) 7759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[46934]: audit 2022-04-22T19:47:27.487653+0000 mon.a (mon.0) 7760 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[46934]: audit 2022-04-22T19:47:27.539993+0000 mon.a (mon.0) 7761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:28 smithi115 bash[46934]: audit 2022-04-22T19:47:27.829209+0000 mon.a (mon.0) 7762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:47:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:29 smithi121 bash[36902]: cluster 2022-04-22T19:47:28.181199+0000 mgr.y (mgr.34107) 4853 : cluster [DBG] pgmap v6184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:29 smithi115 bash[45495]: cluster 2022-04-22T19:47:28.181199+0000 mgr.y (mgr.34107) 4853 : cluster [DBG] pgmap v6184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:29 smithi115 bash[46934]: cluster 2022-04-22T19:47:28.181199+0000 mgr.y (mgr.34107) 4853 : cluster [DBG] pgmap v6184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:30 smithi121 bash[36902]: audit 2022-04-22T19:47:29.660254+0000 mon.a (mon.0) 7763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:30 smithi115 bash[45495]: audit 2022-04-22T19:47:29.660254+0000 mon.a (mon.0) 7763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:30.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:30 smithi115 bash[46934]: audit 2022-04-22T19:47:29.660254+0000 mon.a (mon.0) 7763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:31 smithi121 bash[36902]: cluster 2022-04-22T19:47:30.182378+0000 mgr.y (mgr.34107) 4854 : cluster [DBG] pgmap v6185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:31 smithi121 bash[36902]: audit 2022-04-22T19:47:31.073201+0000 mon.a (mon.0) 7764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:31 smithi115 bash[45495]: cluster 2022-04-22T19:47:30.182378+0000 mgr.y (mgr.34107) 4854 : cluster [DBG] pgmap v6185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:31.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:31 smithi115 bash[45495]: audit 2022-04-22T19:47:31.073201+0000 mon.a (mon.0) 7764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:31.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:31 smithi115 bash[46934]: cluster 2022-04-22T19:47:30.182378+0000 mgr.y (mgr.34107) 4854 : cluster [DBG] pgmap v6185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:31.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:31 smithi115 bash[46934]: audit 2022-04-22T19:47:31.073201+0000 mon.a (mon.0) 7764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:32.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:33 smithi115 bash[45495]: cluster 2022-04-22T19:47:32.183435+0000 mgr.y (mgr.34107) 4855 : cluster [DBG] pgmap v6186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:33.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:33 smithi115 bash[45495]: audit 2022-04-22T19:47:32.542657+0000 mon.a (mon.0) 7765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:33 smithi115 bash[46934]: cluster 2022-04-22T19:47:32.183435+0000 mgr.y (mgr.34107) 4855 : cluster [DBG] pgmap v6186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:33.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:33 smithi115 bash[46934]: audit 2022-04-22T19:47:32.542657+0000 mon.a (mon.0) 7765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:33 smithi121 bash[36902]: cluster 2022-04-22T19:47:32.183435+0000 mgr.y (mgr.34107) 4855 : cluster [DBG] pgmap v6186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:33 smithi121 bash[36902]: audit 2022-04-22T19:47:32.542657+0000 mon.a (mon.0) 7765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:34.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:34.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:35 smithi115 bash[46934]: cluster 2022-04-22T19:47:34.184453+0000 mgr.y (mgr.34107) 4856 : cluster [DBG] pgmap v6187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:35.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:35 smithi115 bash[46934]: audit 2022-04-22T19:47:34.663880+0000 mon.a (mon.0) 7766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:35.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:35 smithi115 bash[45495]: cluster 2022-04-22T19:47:34.184453+0000 mgr.y (mgr.34107) 4856 : cluster [DBG] pgmap v6187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:35.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:35 smithi115 bash[45495]: audit 2022-04-22T19:47:34.663880+0000 mon.a (mon.0) 7766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:35 smithi121 bash[36902]: cluster 2022-04-22T19:47:34.184453+0000 mgr.y (mgr.34107) 4856 : cluster [DBG] pgmap v6187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:35 smithi121 bash[36902]: audit 2022-04-22T19:47:34.663880+0000 mon.a (mon.0) 7766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:36 smithi115 bash[46934]: audit 2022-04-22T19:47:36.076331+0000 mon.a (mon.0) 7767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:36.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:36 smithi115 bash[45495]: audit 2022-04-22T19:47:36.076331+0000 mon.a (mon.0) 7767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:36 smithi121 bash[36902]: audit 2022-04-22T19:47:36.076331+0000 mon.a (mon.0) 7767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:37.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:37 smithi115 bash[45495]: cluster 2022-04-22T19:47:36.185482+0000 mgr.y (mgr.34107) 4857 : cluster [DBG] pgmap v6188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:37 smithi115 bash[46934]: cluster 2022-04-22T19:47:36.185482+0000 mgr.y (mgr.34107) 4857 : cluster [DBG] pgmap v6188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:37 smithi121 bash[36902]: cluster 2022-04-22T19:47:36.185482+0000 mgr.y (mgr.34107) 4857 : cluster [DBG] pgmap v6188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:38 smithi115 bash[45495]: audit 2022-04-22T19:47:37.545492+0000 mon.a (mon.0) 7768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:38.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:38 smithi115 bash[46934]: audit 2022-04-22T19:47:37.545492+0000 mon.a (mon.0) 7768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:38 smithi121 bash[36902]: audit 2022-04-22T19:47:37.545492+0000 mon.a (mon.0) 7768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:39.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:39 smithi115 bash[45495]: cluster 2022-04-22T19:47:38.186540+0000 mgr.y (mgr.34107) 4858 : cluster [DBG] pgmap v6189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:39.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:39 smithi115 bash[46934]: cluster 2022-04-22T19:47:38.186540+0000 mgr.y (mgr.34107) 4858 : cluster [DBG] pgmap v6189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:39 smithi121 bash[36902]: cluster 2022-04-22T19:47:38.186540+0000 mgr.y (mgr.34107) 4858 : cluster [DBG] pgmap v6189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:40 smithi121 bash[36902]: audit 2022-04-22T19:47:39.667434+0000 mon.a (mon.0) 7769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:40 smithi115 bash[45495]: audit 2022-04-22T19:47:39.667434+0000 mon.a (mon.0) 7769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:40.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:40 smithi115 bash[46934]: audit 2022-04-22T19:47:39.667434+0000 mon.a (mon.0) 7769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:41 smithi121 bash[36902]: cluster 2022-04-22T19:47:40.187676+0000 mgr.y (mgr.34107) 4859 : cluster [DBG] pgmap v6190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:41 smithi121 bash[36902]: audit 2022-04-22T19:47:41.079447+0000 mon.a (mon.0) 7770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:41 smithi115 bash[45495]: cluster 2022-04-22T19:47:40.187676+0000 mgr.y (mgr.34107) 4859 : cluster [DBG] pgmap v6190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:41 smithi115 bash[45495]: audit 2022-04-22T19:47:41.079447+0000 mon.a (mon.0) 7770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:41 smithi115 bash[46934]: cluster 2022-04-22T19:47:40.187676+0000 mgr.y (mgr.34107) 4859 : cluster [DBG] pgmap v6190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:41 smithi115 bash[46934]: audit 2022-04-22T19:47:41.079447+0000 mon.a (mon.0) 7770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:42.162Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:43 smithi121 bash[36902]: cluster 2022-04-22T19:47:42.188695+0000 mgr.y (mgr.34107) 4860 : cluster [DBG] pgmap v6191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:43 smithi121 bash[36902]: audit 2022-04-22T19:47:42.548342+0000 mon.a (mon.0) 7771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:43 smithi115 bash[45495]: cluster 2022-04-22T19:47:42.188695+0000 mgr.y (mgr.34107) 4860 : cluster [DBG] pgmap v6191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:43 smithi115 bash[45495]: audit 2022-04-22T19:47:42.548342+0000 mon.a (mon.0) 7771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:43 smithi115 bash[46934]: cluster 2022-04-22T19:47:42.188695+0000 mgr.y (mgr.34107) 4860 : cluster [DBG] pgmap v6191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:43 smithi115 bash[46934]: audit 2022-04-22T19:47:42.548342+0000 mon.a (mon.0) 7771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:44.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:44.033Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:47:45.445 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:47:45.445 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:47:45.445 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:47:45.445 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:47:45.445 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:47:45.446 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:47:45.447 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:47:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:45 smithi121 bash[36902]: cluster 2022-04-22T19:47:44.189746+0000 mgr.y (mgr.34107) 4861 : cluster [DBG] pgmap v6192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:45 smithi121 bash[36902]: audit 2022-04-22T19:47:44.671089+0000 mon.a (mon.0) 7772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:45 smithi115 bash[46934]: cluster 2022-04-22T19:47:44.189746+0000 mgr.y (mgr.34107) 4861 : cluster [DBG] pgmap v6192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:45 smithi115 bash[46934]: audit 2022-04-22T19:47:44.671089+0000 mon.a (mon.0) 7772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:45 smithi115 bash[45495]: cluster 2022-04-22T19:47:44.189746+0000 mgr.y (mgr.34107) 4861 : cluster [DBG] pgmap v6192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:45.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:45 smithi115 bash[45495]: audit 2022-04-22T19:47:44.671089+0000 mon.a (mon.0) 7772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:45.845 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:47:45.847 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:47:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:46 smithi121 bash[36902]: audit 2022-04-22T19:47:45.077150+0000 mgr.y (mgr.34107) 4862 : audit [DBG] from='client.40464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:46 smithi121 bash[36902]: audit 2022-04-22T19:47:45.414242+0000 mgr.y (mgr.34107) 4863 : audit [DBG] from='client.40470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:46 smithi121 bash[36902]: audit 2022-04-22T19:47:45.835073+0000 mon.c (mon.1) 209 : audit [DBG] from='client.? 172.21.15.115:0/2462228258' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:46 smithi121 bash[36902]: audit 2022-04-22T19:47:46.082570+0000 mon.a (mon.0) 7773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[45495]: audit 2022-04-22T19:47:45.077150+0000 mgr.y (mgr.34107) 4862 : audit [DBG] from='client.40464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[45495]: audit 2022-04-22T19:47:45.414242+0000 mgr.y (mgr.34107) 4863 : audit [DBG] from='client.40470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[45495]: audit 2022-04-22T19:47:45.835073+0000 mon.c (mon.1) 209 : audit [DBG] from='client.? 172.21.15.115:0/2462228258' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[45495]: audit 2022-04-22T19:47:46.082570+0000 mon.a (mon.0) 7773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[46934]: audit 2022-04-22T19:47:45.077150+0000 mgr.y (mgr.34107) 4862 : audit [DBG] from='client.40464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[46934]: audit 2022-04-22T19:47:45.414242+0000 mgr.y (mgr.34107) 4863 : audit [DBG] from='client.40470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:47:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[46934]: audit 2022-04-22T19:47:45.835073+0000 mon.c (mon.1) 209 : audit [DBG] from='client.? 172.21.15.115:0/2462228258' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:47:46.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:46 smithi115 bash[46934]: audit 2022-04-22T19:47:46.082570+0000 mon.a (mon.0) 7773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:47 smithi121 bash[36902]: cluster 2022-04-22T19:47:46.190941+0000 mgr.y (mgr.34107) 4864 : cluster [DBG] pgmap v6193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:47.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:47 smithi115 bash[45495]: cluster 2022-04-22T19:47:46.190941+0000 mgr.y (mgr.34107) 4864 : cluster [DBG] pgmap v6193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:47.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:47 smithi115 bash[46934]: cluster 2022-04-22T19:47:46.190941+0000 mgr.y (mgr.34107) 4864 : cluster [DBG] pgmap v6193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:48 smithi121 bash[36902]: audit 2022-04-22T19:47:47.551318+0000 mon.a (mon.0) 7774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:48.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:48 smithi115 bash[45495]: audit 2022-04-22T19:47:47.551318+0000 mon.a (mon.0) 7774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:48 smithi115 bash[46934]: audit 2022-04-22T19:47:47.551318+0000 mon.a (mon.0) 7774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:49 smithi121 bash[36902]: cluster 2022-04-22T19:47:48.192080+0000 mgr.y (mgr.34107) 4865 : cluster [DBG] pgmap v6194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:49.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:49 smithi115 bash[45495]: cluster 2022-04-22T19:47:48.192080+0000 mgr.y (mgr.34107) 4865 : cluster [DBG] pgmap v6194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:49 smithi115 bash[46934]: cluster 2022-04-22T19:47:48.192080+0000 mgr.y (mgr.34107) 4865 : cluster [DBG] pgmap v6194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:50 smithi121 bash[36902]: audit 2022-04-22T19:47:49.674890+0000 mon.a (mon.0) 7775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:50.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:50 smithi115 bash[45495]: audit 2022-04-22T19:47:49.674890+0000 mon.a (mon.0) 7775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:50 smithi115 bash[46934]: audit 2022-04-22T19:47:49.674890+0000 mon.a (mon.0) 7775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:51 smithi121 bash[36902]: cluster 2022-04-22T19:47:50.193222+0000 mgr.y (mgr.34107) 4866 : cluster [DBG] pgmap v6195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:51 smithi121 bash[36902]: audit 2022-04-22T19:47:51.085591+0000 mon.a (mon.0) 7776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:51 smithi115 bash[45495]: cluster 2022-04-22T19:47:50.193222+0000 mgr.y (mgr.34107) 4866 : cluster [DBG] pgmap v6195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:51.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:51 smithi115 bash[45495]: audit 2022-04-22T19:47:51.085591+0000 mon.a (mon.0) 7776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:51 smithi115 bash[46934]: cluster 2022-04-22T19:47:50.193222+0000 mgr.y (mgr.34107) 4866 : cluster [DBG] pgmap v6195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:51.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:51 smithi115 bash[46934]: audit 2022-04-22T19:47:51.085591+0000 mon.a (mon.0) 7776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:52.161Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:53 smithi121 bash[36902]: cluster 2022-04-22T19:47:52.194100+0000 mgr.y (mgr.34107) 4867 : cluster [DBG] pgmap v6196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:53 smithi121 bash[36902]: audit 2022-04-22T19:47:52.554380+0000 mon.a (mon.0) 7777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:53 smithi115 bash[45495]: cluster 2022-04-22T19:47:52.194100+0000 mgr.y (mgr.34107) 4867 : cluster [DBG] pgmap v6196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:53 smithi115 bash[45495]: audit 2022-04-22T19:47:52.554380+0000 mon.a (mon.0) 7777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:53 smithi115 bash[46934]: cluster 2022-04-22T19:47:52.194100+0000 mgr.y (mgr.34107) 4867 : cluster [DBG] pgmap v6196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:53 smithi115 bash[46934]: audit 2022-04-22T19:47:52.554380+0000 mon.a (mon.0) 7777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:47:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:47:54.034Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:47:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:55 smithi121 bash[36902]: cluster 2022-04-22T19:47:54.195491+0000 mgr.y (mgr.34107) 4868 : cluster [DBG] pgmap v6197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:55 smithi121 bash[36902]: audit 2022-04-22T19:47:54.678761+0000 mon.a (mon.0) 7778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:55 smithi115 bash[45495]: cluster 2022-04-22T19:47:54.195491+0000 mgr.y (mgr.34107) 4868 : cluster [DBG] pgmap v6197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:55 smithi115 bash[45495]: audit 2022-04-22T19:47:54.678761+0000 mon.a (mon.0) 7778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:55 smithi115 bash[46934]: cluster 2022-04-22T19:47:54.195491+0000 mgr.y (mgr.34107) 4868 : cluster [DBG] pgmap v6197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:55.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:55 smithi115 bash[46934]: audit 2022-04-22T19:47:54.678761+0000 mon.a (mon.0) 7778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:47:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:56 smithi121 bash[36902]: audit 2022-04-22T19:47:56.088662+0000 mon.a (mon.0) 7779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:56 smithi115 bash[46934]: audit 2022-04-22T19:47:56.088662+0000 mon.a (mon.0) 7779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:56 smithi115 bash[45495]: audit 2022-04-22T19:47:56.088662+0000 mon.a (mon.0) 7779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:47:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:57 smithi121 bash[36902]: cluster 2022-04-22T19:47:56.196604+0000 mgr.y (mgr.34107) 4869 : cluster [DBG] pgmap v6198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:57 smithi115 bash[46934]: cluster 2022-04-22T19:47:56.196604+0000 mgr.y (mgr.34107) 4869 : cluster [DBG] pgmap v6198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:57 smithi115 bash[45495]: cluster 2022-04-22T19:47:56.196604+0000 mgr.y (mgr.34107) 4869 : cluster [DBG] pgmap v6198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:58 smithi121 bash[36902]: audit 2022-04-22T19:47:57.557486+0000 mon.a (mon.0) 7780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:58.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:58 smithi115 bash[45495]: audit 2022-04-22T19:47:57.557486+0000 mon.a (mon.0) 7780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:58.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:58 smithi115 bash[46934]: audit 2022-04-22T19:47:57.557486+0000 mon.a (mon.0) 7780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:47:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:47:59 smithi121 bash[36902]: cluster 2022-04-22T19:47:58.197683+0000 mgr.y (mgr.34107) 4870 : cluster [DBG] pgmap v6199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:47:59 smithi115 bash[45495]: cluster 2022-04-22T19:47:58.197683+0000 mgr.y (mgr.34107) 4870 : cluster [DBG] pgmap v6199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:47:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:47:59 smithi115 bash[46934]: cluster 2022-04-22T19:47:58.197683+0000 mgr.y (mgr.34107) 4870 : cluster [DBG] pgmap v6199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:00 smithi121 bash[36902]: audit 2022-04-22T19:47:59.682543+0000 mon.a (mon.0) 7781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:00 smithi115 bash[46934]: audit 2022-04-22T19:47:59.682543+0000 mon.a (mon.0) 7781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:00 smithi115 bash[45495]: audit 2022-04-22T19:47:59.682543+0000 mon.a (mon.0) 7781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:01 smithi121 bash[36902]: cluster 2022-04-22T19:48:00.198840+0000 mgr.y (mgr.34107) 4871 : cluster [DBG] pgmap v6200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:01 smithi121 bash[36902]: audit 2022-04-22T19:48:01.092079+0000 mon.a (mon.0) 7782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:01 smithi115 bash[45495]: cluster 2022-04-22T19:48:00.198840+0000 mgr.y (mgr.34107) 4871 : cluster [DBG] pgmap v6200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:01 smithi115 bash[45495]: audit 2022-04-22T19:48:01.092079+0000 mon.a (mon.0) 7782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:01 smithi115 bash[46934]: cluster 2022-04-22T19:48:00.198840+0000 mgr.y (mgr.34107) 4871 : cluster [DBG] pgmap v6200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:01 smithi115 bash[46934]: audit 2022-04-22T19:48:01.092079+0000 mon.a (mon.0) 7782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:02.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:03 smithi121 bash[36902]: cluster 2022-04-22T19:48:02.199956+0000 mgr.y (mgr.34107) 4872 : cluster [DBG] pgmap v6201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:03 smithi121 bash[36902]: audit 2022-04-22T19:48:02.560412+0000 mon.a (mon.0) 7783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:03 smithi115 bash[45495]: cluster 2022-04-22T19:48:02.199956+0000 mgr.y (mgr.34107) 4872 : cluster [DBG] pgmap v6201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:03 smithi115 bash[45495]: audit 2022-04-22T19:48:02.560412+0000 mon.a (mon.0) 7783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:03 smithi115 bash[46934]: cluster 2022-04-22T19:48:02.199956+0000 mgr.y (mgr.34107) 4872 : cluster [DBG] pgmap v6201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:03.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:03 smithi115 bash[46934]: audit 2022-04-22T19:48:02.560412+0000 mon.a (mon.0) 7783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:04.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:04.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:05 smithi121 bash[36902]: cluster 2022-04-22T19:48:04.201113+0000 mgr.y (mgr.34107) 4873 : cluster [DBG] pgmap v6202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:05 smithi121 bash[36902]: audit 2022-04-22T19:48:04.686321+0000 mon.a (mon.0) 7784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:05 smithi115 bash[45495]: cluster 2022-04-22T19:48:04.201113+0000 mgr.y (mgr.34107) 4873 : cluster [DBG] pgmap v6202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:05 smithi115 bash[45495]: audit 2022-04-22T19:48:04.686321+0000 mon.a (mon.0) 7784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:05 smithi115 bash[46934]: cluster 2022-04-22T19:48:04.201113+0000 mgr.y (mgr.34107) 4873 : cluster [DBG] pgmap v6202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:05.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:05 smithi115 bash[46934]: audit 2022-04-22T19:48:04.686321+0000 mon.a (mon.0) 7784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:06 smithi121 bash[36902]: audit 2022-04-22T19:48:06.095325+0000 mon.a (mon.0) 7785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:06 smithi115 bash[45495]: audit 2022-04-22T19:48:06.095325+0000 mon.a (mon.0) 7785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:06 smithi115 bash[46934]: audit 2022-04-22T19:48:06.095325+0000 mon.a (mon.0) 7785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:07.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:07 smithi121 bash[36902]: cluster 2022-04-22T19:48:06.202051+0000 mgr.y (mgr.34107) 4874 : cluster [DBG] pgmap v6203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:07 smithi115 bash[45495]: cluster 2022-04-22T19:48:06.202051+0000 mgr.y (mgr.34107) 4874 : cluster [DBG] pgmap v6203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:07 smithi115 bash[46934]: cluster 2022-04-22T19:48:06.202051+0000 mgr.y (mgr.34107) 4874 : cluster [DBG] pgmap v6203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:08 smithi121 bash[36902]: audit 2022-04-22T19:48:07.563561+0000 mon.a (mon.0) 7786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:08.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:08 smithi115 bash[45495]: audit 2022-04-22T19:48:07.563561+0000 mon.a (mon.0) 7786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:08.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:08 smithi115 bash[46934]: audit 2022-04-22T19:48:07.563561+0000 mon.a (mon.0) 7786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:09 smithi121 bash[36902]: cluster 2022-04-22T19:48:08.203114+0000 mgr.y (mgr.34107) 4875 : cluster [DBG] pgmap v6204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:09.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:09 smithi115 bash[45495]: cluster 2022-04-22T19:48:08.203114+0000 mgr.y (mgr.34107) 4875 : cluster [DBG] pgmap v6204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:09.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:09 smithi115 bash[46934]: cluster 2022-04-22T19:48:08.203114+0000 mgr.y (mgr.34107) 4875 : cluster [DBG] pgmap v6204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:10 smithi121 bash[36902]: audit 2022-04-22T19:48:09.690057+0000 mon.a (mon.0) 7787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:10 smithi115 bash[45495]: audit 2022-04-22T19:48:09.690057+0000 mon.a (mon.0) 7787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:10.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:10 smithi115 bash[46934]: audit 2022-04-22T19:48:09.690057+0000 mon.a (mon.0) 7787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:11 smithi121 bash[36902]: cluster 2022-04-22T19:48:10.204232+0000 mgr.y (mgr.34107) 4876 : cluster [DBG] pgmap v6205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:11 smithi121 bash[36902]: audit 2022-04-22T19:48:11.098311+0000 mon.a (mon.0) 7788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:11 smithi115 bash[45495]: cluster 2022-04-22T19:48:10.204232+0000 mgr.y (mgr.34107) 4876 : cluster [DBG] pgmap v6205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:11.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:11 smithi115 bash[45495]: audit 2022-04-22T19:48:11.098311+0000 mon.a (mon.0) 7788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:11.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:11 smithi115 bash[46934]: cluster 2022-04-22T19:48:10.204232+0000 mgr.y (mgr.34107) 4876 : cluster [DBG] pgmap v6205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:11.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:11 smithi115 bash[46934]: audit 2022-04-22T19:48:11.098311+0000 mon.a (mon.0) 7788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:12.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:13 smithi121 bash[36902]: cluster 2022-04-22T19:48:12.205338+0000 mgr.y (mgr.34107) 4877 : cluster [DBG] pgmap v6206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:13 smithi121 bash[36902]: audit 2022-04-22T19:48:12.566594+0000 mon.a (mon.0) 7789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:13 smithi115 bash[45495]: cluster 2022-04-22T19:48:12.205338+0000 mgr.y (mgr.34107) 4877 : cluster [DBG] pgmap v6206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:13 smithi115 bash[45495]: audit 2022-04-22T19:48:12.566594+0000 mon.a (mon.0) 7789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:13 smithi115 bash[46934]: cluster 2022-04-22T19:48:12.205338+0000 mgr.y (mgr.34107) 4877 : cluster [DBG] pgmap v6206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:13.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:13 smithi115 bash[46934]: audit 2022-04-22T19:48:12.566594+0000 mon.a (mon.0) 7789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:14.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:14.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:15 smithi121 bash[36902]: cluster 2022-04-22T19:48:14.206433+0000 mgr.y (mgr.34107) 4878 : cluster [DBG] pgmap v6207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:15 smithi121 bash[36902]: audit 2022-04-22T19:48:14.693771+0000 mon.a (mon.0) 7790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:15 smithi121 bash[36902]: audit 2022-04-22T19:48:14.910630+0000 mon.a (mon.0) 7791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:48:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:15 smithi121 bash[36902]: audit 2022-04-22T19:48:14.910933+0000 mon.a (mon.0) 7792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:48:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[46934]: cluster 2022-04-22T19:48:14.206433+0000 mgr.y (mgr.34107) 4878 : cluster [DBG] pgmap v6207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:15.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[46934]: audit 2022-04-22T19:48:14.693771+0000 mon.a (mon.0) 7790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:15.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[46934]: audit 2022-04-22T19:48:14.910630+0000 mon.a (mon.0) 7791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:48:15.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[46934]: audit 2022-04-22T19:48:14.910933+0000 mon.a (mon.0) 7792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:48:15.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[45495]: cluster 2022-04-22T19:48:14.206433+0000 mgr.y (mgr.34107) 4878 : cluster [DBG] pgmap v6207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:15.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[45495]: audit 2022-04-22T19:48:14.693771+0000 mon.a (mon.0) 7790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:15.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[45495]: audit 2022-04-22T19:48:14.910630+0000 mon.a (mon.0) 7791 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:48:15.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:15 smithi115 bash[45495]: audit 2022-04-22T19:48:14.910933+0000 mon.a (mon.0) 7792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:48:16.219 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:48:16.568 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:48:16.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:48:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:48:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:48:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:48:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:48:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:48:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:16 smithi121 bash[36902]: audit 2022-04-22T19:48:16.101893+0000 mon.a (mon.0) 7793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:16 smithi115 bash[45495]: audit 2022-04-22T19:48:16.101893+0000 mon.a (mon.0) 7793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:16 smithi115 bash[46934]: audit 2022-04-22T19:48:16.101893+0000 mon.a (mon.0) 7793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:16.966 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:48:16.966 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:48:16.966 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:48:16.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:16.966 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:16.967 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:48:16.968 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:48:16.968 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:48:16.968 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:48:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:17 smithi121 bash[36902]: audit 2022-04-22T19:48:16.193936+0000 mgr.y (mgr.34107) 4879 : audit [DBG] from='client.50416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:17 smithi121 bash[36902]: cluster 2022-04-22T19:48:16.207400+0000 mgr.y (mgr.34107) 4880 : cluster [DBG] pgmap v6208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:17 smithi121 bash[36902]: audit 2022-04-22T19:48:16.538186+0000 mgr.y (mgr.34107) 4881 : audit [DBG] from='client.40488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:17 smithi121 bash[36902]: audit 2022-04-22T19:48:16.955983+0000 mon.a (mon.0) 7794 : audit [DBG] from='client.? 172.21.15.115:0/1075264059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[46934]: audit 2022-04-22T19:48:16.193936+0000 mgr.y (mgr.34107) 4879 : audit [DBG] from='client.50416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[46934]: cluster 2022-04-22T19:48:16.207400+0000 mgr.y (mgr.34107) 4880 : cluster [DBG] pgmap v6208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[46934]: audit 2022-04-22T19:48:16.538186+0000 mgr.y (mgr.34107) 4881 : audit [DBG] from='client.40488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[46934]: audit 2022-04-22T19:48:16.955983+0000 mon.a (mon.0) 7794 : audit [DBG] from='client.? 172.21.15.115:0/1075264059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[45495]: audit 2022-04-22T19:48:16.193936+0000 mgr.y (mgr.34107) 4879 : audit [DBG] from='client.50416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[45495]: cluster 2022-04-22T19:48:16.207400+0000 mgr.y (mgr.34107) 4880 : cluster [DBG] pgmap v6208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[45495]: audit 2022-04-22T19:48:16.538186+0000 mgr.y (mgr.34107) 4881 : audit [DBG] from='client.40488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:17.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:17 smithi115 bash[45495]: audit 2022-04-22T19:48:16.955983+0000 mon.a (mon.0) 7794 : audit [DBG] from='client.? 172.21.15.115:0/1075264059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:18 smithi121 bash[36902]: audit 2022-04-22T19:48:17.569660+0000 mon.a (mon.0) 7795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:18.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:18 smithi115 bash[45495]: audit 2022-04-22T19:48:17.569660+0000 mon.a (mon.0) 7795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:18 smithi115 bash[46934]: audit 2022-04-22T19:48:17.569660+0000 mon.a (mon.0) 7795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:19 smithi121 bash[36902]: cluster 2022-04-22T19:48:18.208508+0000 mgr.y (mgr.34107) 4882 : cluster [DBG] pgmap v6209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:19 smithi115 bash[45495]: cluster 2022-04-22T19:48:18.208508+0000 mgr.y (mgr.34107) 4882 : cluster [DBG] pgmap v6209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:19 smithi115 bash[46934]: cluster 2022-04-22T19:48:18.208508+0000 mgr.y (mgr.34107) 4882 : cluster [DBG] pgmap v6209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:20 smithi121 bash[36902]: audit 2022-04-22T19:48:19.697302+0000 mon.a (mon.0) 7796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:20.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:20 smithi115 bash[45495]: audit 2022-04-22T19:48:19.697302+0000 mon.a (mon.0) 7796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:20 smithi115 bash[46934]: audit 2022-04-22T19:48:19.697302+0000 mon.a (mon.0) 7796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:21 smithi121 bash[36902]: cluster 2022-04-22T19:48:20.209555+0000 mgr.y (mgr.34107) 4883 : cluster [DBG] pgmap v6210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:21 smithi121 bash[36902]: audit 2022-04-22T19:48:21.104594+0000 mon.a (mon.0) 7797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:21 smithi115 bash[45495]: cluster 2022-04-22T19:48:20.209555+0000 mgr.y (mgr.34107) 4883 : cluster [DBG] pgmap v6210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:21 smithi115 bash[45495]: audit 2022-04-22T19:48:21.104594+0000 mon.a (mon.0) 7797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:21 smithi115 bash[46934]: cluster 2022-04-22T19:48:20.209555+0000 mgr.y (mgr.34107) 4883 : cluster [DBG] pgmap v6210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:21.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:21 smithi115 bash[46934]: audit 2022-04-22T19:48:21.104594+0000 mon.a (mon.0) 7797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:22.164Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:23 smithi121 bash[36902]: cluster 2022-04-22T19:48:22.210698+0000 mgr.y (mgr.34107) 4884 : cluster [DBG] pgmap v6211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:23 smithi121 bash[36902]: audit 2022-04-22T19:48:22.572586+0000 mon.a (mon.0) 7798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:23 smithi115 bash[46934]: cluster 2022-04-22T19:48:22.210698+0000 mgr.y (mgr.34107) 4884 : cluster [DBG] pgmap v6211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:23 smithi115 bash[46934]: audit 2022-04-22T19:48:22.572586+0000 mon.a (mon.0) 7798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:23 smithi115 bash[45495]: cluster 2022-04-22T19:48:22.210698+0000 mgr.y (mgr.34107) 4884 : cluster [DBG] pgmap v6211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:23 smithi115 bash[45495]: audit 2022-04-22T19:48:22.572586+0000 mon.a (mon.0) 7798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:24.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:24.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:25 smithi121 bash[36902]: cluster 2022-04-22T19:48:24.211886+0000 mgr.y (mgr.34107) 4885 : cluster [DBG] pgmap v6212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:25 smithi121 bash[36902]: audit 2022-04-22T19:48:24.700986+0000 mon.a (mon.0) 7799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:25 smithi115 bash[45495]: cluster 2022-04-22T19:48:24.211886+0000 mgr.y (mgr.34107) 4885 : cluster [DBG] pgmap v6212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:25 smithi115 bash[45495]: audit 2022-04-22T19:48:24.700986+0000 mon.a (mon.0) 7799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:25 smithi115 bash[46934]: cluster 2022-04-22T19:48:24.211886+0000 mgr.y (mgr.34107) 4885 : cluster [DBG] pgmap v6212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:25.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:25 smithi115 bash[46934]: audit 2022-04-22T19:48:24.700986+0000 mon.a (mon.0) 7799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:26 smithi121 bash[36902]: audit 2022-04-22T19:48:26.108058+0000 mon.a (mon.0) 7800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:26 smithi115 bash[45495]: audit 2022-04-22T19:48:26.108058+0000 mon.a (mon.0) 7800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:26 smithi115 bash[46934]: audit 2022-04-22T19:48:26.108058+0000 mon.a (mon.0) 7800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:27 smithi121 bash[36902]: cluster 2022-04-22T19:48:26.213072+0000 mgr.y (mgr.34107) 4886 : cluster [DBG] pgmap v6213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:27 smithi115 bash[45495]: cluster 2022-04-22T19:48:26.213072+0000 mgr.y (mgr.34107) 4886 : cluster [DBG] pgmap v6213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:27 smithi115 bash[46934]: cluster 2022-04-22T19:48:26.213072+0000 mgr.y (mgr.34107) 4886 : cluster [DBG] pgmap v6213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:28 smithi121 bash[36902]: audit 2022-04-22T19:48:27.575794+0000 mon.a (mon.0) 7801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:28 smithi121 bash[36902]: audit 2022-04-22T19:48:27.837667+0000 mon.a (mon.0) 7802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:28 smithi121 bash[36902]: audit 2022-04-22T19:48:27.840825+0000 mon.a (mon.0) 7803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:28 smithi121 bash[36902]: audit 2022-04-22T19:48:27.842595+0000 mon.a (mon.0) 7804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:48:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:28 smithi121 bash[36902]: audit 2022-04-22T19:48:28.206555+0000 mon.a (mon.0) 7805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:48:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[46934]: audit 2022-04-22T19:48:27.575794+0000 mon.a (mon.0) 7801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[46934]: audit 2022-04-22T19:48:27.837667+0000 mon.a (mon.0) 7802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:48:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[46934]: audit 2022-04-22T19:48:27.840825+0000 mon.a (mon.0) 7803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:48:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[46934]: audit 2022-04-22T19:48:27.842595+0000 mon.a (mon.0) 7804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:48:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[46934]: audit 2022-04-22T19:48:28.206555+0000 mon.a (mon.0) 7805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:48:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[45495]: audit 2022-04-22T19:48:27.575794+0000 mon.a (mon.0) 7801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[45495]: audit 2022-04-22T19:48:27.837667+0000 mon.a (mon.0) 7802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:48:28.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[45495]: audit 2022-04-22T19:48:27.840825+0000 mon.a (mon.0) 7803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:48:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[45495]: audit 2022-04-22T19:48:27.842595+0000 mon.a (mon.0) 7804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:48:28.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:28 smithi115 bash[45495]: audit 2022-04-22T19:48:28.206555+0000 mon.a (mon.0) 7805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:48:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:29 smithi121 bash[36902]: cluster 2022-04-22T19:48:28.214088+0000 mgr.y (mgr.34107) 4887 : cluster [DBG] pgmap v6214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:29 smithi115 bash[46934]: cluster 2022-04-22T19:48:28.214088+0000 mgr.y (mgr.34107) 4887 : cluster [DBG] pgmap v6214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:29 smithi115 bash[45495]: cluster 2022-04-22T19:48:28.214088+0000 mgr.y (mgr.34107) 4887 : cluster [DBG] pgmap v6214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:30 smithi121 bash[36902]: audit 2022-04-22T19:48:29.704646+0000 mon.a (mon.0) 7806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:30.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:30 smithi115 bash[45495]: audit 2022-04-22T19:48:29.704646+0000 mon.a (mon.0) 7806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:30 smithi115 bash[46934]: audit 2022-04-22T19:48:29.704646+0000 mon.a (mon.0) 7806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:31 smithi115 bash[45495]: cluster 2022-04-22T19:48:30.215074+0000 mgr.y (mgr.34107) 4888 : cluster [DBG] pgmap v6215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:31 smithi115 bash[45495]: audit 2022-04-22T19:48:31.111202+0000 mon.a (mon.0) 7807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:31 smithi115 bash[46934]: cluster 2022-04-22T19:48:30.215074+0000 mgr.y (mgr.34107) 4888 : cluster [DBG] pgmap v6215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:31 smithi115 bash[46934]: audit 2022-04-22T19:48:31.111202+0000 mon.a (mon.0) 7807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:31 smithi121 bash[36902]: cluster 2022-04-22T19:48:30.215074+0000 mgr.y (mgr.34107) 4888 : cluster [DBG] pgmap v6215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:31 smithi121 bash[36902]: audit 2022-04-22T19:48:31.111202+0000 mon.a (mon.0) 7807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:32.165Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:33 smithi115 bash[45495]: cluster 2022-04-22T19:48:32.216132+0000 mgr.y (mgr.34107) 4889 : cluster [DBG] pgmap v6216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:33 smithi115 bash[45495]: audit 2022-04-22T19:48:32.578565+0000 mon.a (mon.0) 7808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:33 smithi115 bash[46934]: cluster 2022-04-22T19:48:32.216132+0000 mgr.y (mgr.34107) 4889 : cluster [DBG] pgmap v6216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:33.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:33 smithi115 bash[46934]: audit 2022-04-22T19:48:32.578565+0000 mon.a (mon.0) 7808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:33 smithi121 bash[36902]: cluster 2022-04-22T19:48:32.216132+0000 mgr.y (mgr.34107) 4889 : cluster [DBG] pgmap v6216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:33 smithi121 bash[36902]: audit 2022-04-22T19:48:32.578565+0000 mon.a (mon.0) 7808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:34.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:34.038Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:35 smithi115 bash[45495]: cluster 2022-04-22T19:48:34.217207+0000 mgr.y (mgr.34107) 4890 : cluster [DBG] pgmap v6217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:35 smithi115 bash[45495]: audit 2022-04-22T19:48:34.708319+0000 mon.a (mon.0) 7809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:35 smithi115 bash[46934]: cluster 2022-04-22T19:48:34.217207+0000 mgr.y (mgr.34107) 4890 : cluster [DBG] pgmap v6217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:35.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:35 smithi115 bash[46934]: audit 2022-04-22T19:48:34.708319+0000 mon.a (mon.0) 7809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:35 smithi121 bash[36902]: cluster 2022-04-22T19:48:34.217207+0000 mgr.y (mgr.34107) 4890 : cluster [DBG] pgmap v6217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:35 smithi121 bash[36902]: audit 2022-04-22T19:48:34.708319+0000 mon.a (mon.0) 7809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:36.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:36 smithi115 bash[45495]: audit 2022-04-22T19:48:36.114390+0000 mon.a (mon.0) 7810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:36 smithi115 bash[46934]: audit 2022-04-22T19:48:36.114390+0000 mon.a (mon.0) 7810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:36 smithi121 bash[36902]: audit 2022-04-22T19:48:36.114390+0000 mon.a (mon.0) 7810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:37.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:37 smithi115 bash[45495]: cluster 2022-04-22T19:48:36.218389+0000 mgr.y (mgr.34107) 4891 : cluster [DBG] pgmap v6218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:37 smithi115 bash[46934]: cluster 2022-04-22T19:48:36.218389+0000 mgr.y (mgr.34107) 4891 : cluster [DBG] pgmap v6218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:37 smithi121 bash[36902]: cluster 2022-04-22T19:48:36.218389+0000 mgr.y (mgr.34107) 4891 : cluster [DBG] pgmap v6218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:38 smithi115 bash[45495]: audit 2022-04-22T19:48:37.581408+0000 mon.a (mon.0) 7811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:38 smithi115 bash[46934]: audit 2022-04-22T19:48:37.581408+0000 mon.a (mon.0) 7811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:38 smithi121 bash[36902]: audit 2022-04-22T19:48:37.581408+0000 mon.a (mon.0) 7811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:39 smithi121 bash[36902]: cluster 2022-04-22T19:48:38.219581+0000 mgr.y (mgr.34107) 4892 : cluster [DBG] pgmap v6219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:40.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:39 smithi115 bash[45495]: cluster 2022-04-22T19:48:38.219581+0000 mgr.y (mgr.34107) 4892 : cluster [DBG] pgmap v6219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:39 smithi115 bash[46934]: cluster 2022-04-22T19:48:38.219581+0000 mgr.y (mgr.34107) 4892 : cluster [DBG] pgmap v6219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:40 smithi121 bash[36902]: audit 2022-04-22T19:48:39.711927+0000 mon.a (mon.0) 7812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:41.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:40 smithi115 bash[45495]: audit 2022-04-22T19:48:39.711927+0000 mon.a (mon.0) 7812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:40 smithi115 bash[46934]: audit 2022-04-22T19:48:39.711927+0000 mon.a (mon.0) 7812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:41 smithi121 bash[36902]: cluster 2022-04-22T19:48:40.220577+0000 mgr.y (mgr.34107) 4893 : cluster [DBG] pgmap v6220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:41 smithi121 bash[36902]: audit 2022-04-22T19:48:41.117415+0000 mon.a (mon.0) 7813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:41 smithi115 bash[45495]: cluster 2022-04-22T19:48:40.220577+0000 mgr.y (mgr.34107) 4893 : cluster [DBG] pgmap v6220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:41 smithi115 bash[45495]: audit 2022-04-22T19:48:41.117415+0000 mon.a (mon.0) 7813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:41 smithi115 bash[46934]: cluster 2022-04-22T19:48:40.220577+0000 mgr.y (mgr.34107) 4893 : cluster [DBG] pgmap v6220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:41 smithi115 bash[46934]: audit 2022-04-22T19:48:41.117415+0000 mon.a (mon.0) 7813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:42.165Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:43 smithi121 bash[36902]: cluster 2022-04-22T19:48:42.221680+0000 mgr.y (mgr.34107) 4894 : cluster [DBG] pgmap v6221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:43 smithi121 bash[36902]: audit 2022-04-22T19:48:42.584543+0000 mon.a (mon.0) 7814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:43 smithi115 bash[46934]: cluster 2022-04-22T19:48:42.221680+0000 mgr.y (mgr.34107) 4894 : cluster [DBG] pgmap v6221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:43 smithi115 bash[46934]: audit 2022-04-22T19:48:42.584543+0000 mon.a (mon.0) 7814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:43 smithi115 bash[45495]: cluster 2022-04-22T19:48:42.221680+0000 mgr.y (mgr.34107) 4894 : cluster [DBG] pgmap v6221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:44.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:43 smithi115 bash[45495]: audit 2022-04-22T19:48:42.584543+0000 mon.a (mon.0) 7814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:44.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:44.037Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:45 smithi121 bash[36902]: cluster 2022-04-22T19:48:44.222744+0000 mgr.y (mgr.34107) 4895 : cluster [DBG] pgmap v6222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:45 smithi121 bash[36902]: audit 2022-04-22T19:48:44.715605+0000 mon.a (mon.0) 7815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:45 smithi115 bash[45495]: cluster 2022-04-22T19:48:44.222744+0000 mgr.y (mgr.34107) 4895 : cluster [DBG] pgmap v6222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:45 smithi115 bash[45495]: audit 2022-04-22T19:48:44.715605+0000 mon.a (mon.0) 7815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:45 smithi115 bash[46934]: cluster 2022-04-22T19:48:44.222744+0000 mgr.y (mgr.34107) 4895 : cluster [DBG] pgmap v6222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:45 smithi115 bash[46934]: audit 2022-04-22T19:48:44.715605+0000 mon.a (mon.0) 7815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:46 smithi121 bash[36902]: audit 2022-04-22T19:48:46.120595+0000 mon.a (mon.0) 7816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:47.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:46 smithi115 bash[45495]: audit 2022-04-22T19:48:46.120595+0000 mon.a (mon.0) 7816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:46 smithi115 bash[46934]: audit 2022-04-22T19:48:46.120595+0000 mon.a (mon.0) 7816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:47.345 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:48:47.697 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:48:47.697 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:48:47.697 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:48:47.697 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:48:47.698 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:48:47.699 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:48:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:47 smithi121 bash[36902]: cluster 2022-04-22T19:48:46.223822+0000 mgr.y (mgr.34107) 4896 : cluster [DBG] pgmap v6223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:47 smithi115 bash[45495]: cluster 2022-04-22T19:48:46.223822+0000 mgr.y (mgr.34107) 4896 : cluster [DBG] pgmap v6223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:47 smithi115 bash[46934]: cluster 2022-04-22T19:48:46.223822+0000 mgr.y (mgr.34107) 4896 : cluster [DBG] pgmap v6223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:48.095 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:48:48.095 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:48:48.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:48:48.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:48:48.097 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:48:48.097 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:48:48.097 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:48:48.097 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:48 smithi121 bash[36902]: audit 2022-04-22T19:48:47.318108+0000 mgr.y (mgr.34107) 4897 : audit [DBG] from='client.40500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:48 smithi121 bash[36902]: audit 2022-04-22T19:48:47.587314+0000 mon.a (mon.0) 7817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:48 smithi121 bash[36902]: audit 2022-04-22T19:48:47.669439+0000 mgr.y (mgr.34107) 4898 : audit [DBG] from='client.50440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:48 smithi121 bash[36902]: audit 2022-04-22T19:48:48.085646+0000 mon.c (mon.1) 210 : audit [DBG] from='client.? 172.21.15.115:0/4085177666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:49.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[45495]: audit 2022-04-22T19:48:47.318108+0000 mgr.y (mgr.34107) 4897 : audit [DBG] from='client.40500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[45495]: audit 2022-04-22T19:48:47.587314+0000 mon.a (mon.0) 7817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:49.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[45495]: audit 2022-04-22T19:48:47.669439+0000 mgr.y (mgr.34107) 4898 : audit [DBG] from='client.50440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[45495]: audit 2022-04-22T19:48:48.085646+0000 mon.c (mon.1) 210 : audit [DBG] from='client.? 172.21.15.115:0/4085177666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:49.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[46934]: audit 2022-04-22T19:48:47.318108+0000 mgr.y (mgr.34107) 4897 : audit [DBG] from='client.40500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[46934]: audit 2022-04-22T19:48:47.587314+0000 mon.a (mon.0) 7817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:49.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[46934]: audit 2022-04-22T19:48:47.669439+0000 mgr.y (mgr.34107) 4898 : audit [DBG] from='client.50440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:48:49.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:48 smithi115 bash[46934]: audit 2022-04-22T19:48:48.085646+0000 mon.c (mon.1) 210 : audit [DBG] from='client.? 172.21.15.115:0/4085177666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:48:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:49 smithi121 bash[36902]: cluster 2022-04-22T19:48:48.224877+0000 mgr.y (mgr.34107) 4899 : cluster [DBG] pgmap v6224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:49 smithi115 bash[45495]: cluster 2022-04-22T19:48:48.224877+0000 mgr.y (mgr.34107) 4899 : cluster [DBG] pgmap v6224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:49 smithi115 bash[46934]: cluster 2022-04-22T19:48:48.224877+0000 mgr.y (mgr.34107) 4899 : cluster [DBG] pgmap v6224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:50 smithi121 bash[36902]: audit 2022-04-22T19:48:49.719157+0000 mon.a (mon.0) 7818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:50 smithi115 bash[45495]: audit 2022-04-22T19:48:49.719157+0000 mon.a (mon.0) 7818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:50 smithi115 bash[46934]: audit 2022-04-22T19:48:49.719157+0000 mon.a (mon.0) 7818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:51 smithi121 bash[36902]: cluster 2022-04-22T19:48:50.225982+0000 mgr.y (mgr.34107) 4900 : cluster [DBG] pgmap v6225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:51 smithi121 bash[36902]: audit 2022-04-22T19:48:51.123731+0000 mon.a (mon.0) 7819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:51 smithi115 bash[45495]: cluster 2022-04-22T19:48:50.225982+0000 mgr.y (mgr.34107) 4900 : cluster [DBG] pgmap v6225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:52.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:51 smithi115 bash[45495]: audit 2022-04-22T19:48:51.123731+0000 mon.a (mon.0) 7819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:51 smithi115 bash[46934]: cluster 2022-04-22T19:48:50.225982+0000 mgr.y (mgr.34107) 4900 : cluster [DBG] pgmap v6225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:52.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:51 smithi115 bash[46934]: audit 2022-04-22T19:48:51.123731+0000 mon.a (mon.0) 7819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:52.166Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:52 smithi121 bash[36902]: audit 2022-04-22T19:48:52.590332+0000 mon.a (mon.0) 7820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:53.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:52 smithi115 bash[45495]: audit 2022-04-22T19:48:52.590332+0000 mon.a (mon.0) 7820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:53.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:52 smithi115 bash[46934]: audit 2022-04-22T19:48:52.590332+0000 mon.a (mon.0) 7820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:53 smithi121 bash[36902]: cluster 2022-04-22T19:48:52.227113+0000 mgr.y (mgr.34107) 4901 : cluster [DBG] pgmap v6226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:53 smithi115 bash[46934]: cluster 2022-04-22T19:48:52.227113+0000 mgr.y (mgr.34107) 4901 : cluster [DBG] pgmap v6226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:54.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:53 smithi115 bash[45495]: cluster 2022-04-22T19:48:52.227113+0000 mgr.y (mgr.34107) 4901 : cluster [DBG] pgmap v6226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:54.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:48:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:48:54.039Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:48:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:55 smithi121 bash[36902]: cluster 2022-04-22T19:48:54.228237+0000 mgr.y (mgr.34107) 4902 : cluster [DBG] pgmap v6227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:55 smithi121 bash[36902]: audit 2022-04-22T19:48:54.722747+0000 mon.a (mon.0) 7821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:55 smithi115 bash[45495]: cluster 2022-04-22T19:48:54.228237+0000 mgr.y (mgr.34107) 4902 : cluster [DBG] pgmap v6227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:55 smithi115 bash[45495]: audit 2022-04-22T19:48:54.722747+0000 mon.a (mon.0) 7821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:55 smithi115 bash[46934]: cluster 2022-04-22T19:48:54.228237+0000 mgr.y (mgr.34107) 4902 : cluster [DBG] pgmap v6227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:55 smithi115 bash[46934]: audit 2022-04-22T19:48:54.722747+0000 mon.a (mon.0) 7821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:48:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:56 smithi121 bash[36902]: audit 2022-04-22T19:48:56.127284+0000 mon.a (mon.0) 7822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:57.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:56 smithi115 bash[45495]: audit 2022-04-22T19:48:56.127284+0000 mon.a (mon.0) 7822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:57.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:56 smithi115 bash[46934]: audit 2022-04-22T19:48:56.127284+0000 mon.a (mon.0) 7822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:48:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:57 smithi121 bash[36902]: cluster 2022-04-22T19:48:56.229306+0000 mgr.y (mgr.34107) 4903 : cluster [DBG] pgmap v6228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:57 smithi121 bash[36902]: audit 2022-04-22T19:48:57.593380+0000 mon.a (mon.0) 7823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:57 smithi115 bash[45495]: cluster 2022-04-22T19:48:56.229306+0000 mgr.y (mgr.34107) 4903 : cluster [DBG] pgmap v6228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:57 smithi115 bash[45495]: audit 2022-04-22T19:48:57.593380+0000 mon.a (mon.0) 7823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:48:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:57 smithi115 bash[46934]: cluster 2022-04-22T19:48:56.229306+0000 mgr.y (mgr.34107) 4903 : cluster [DBG] pgmap v6228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:48:58.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:57 smithi115 bash[46934]: audit 2022-04-22T19:48:57.593380+0000 mon.a (mon.0) 7823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:48:59 smithi121 bash[36902]: cluster 2022-04-22T19:48:58.230468+0000 mgr.y (mgr.34107) 4904 : cluster [DBG] pgmap v6229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:00.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:48:59 smithi115 bash[45495]: cluster 2022-04-22T19:48:58.230468+0000 mgr.y (mgr.34107) 4904 : cluster [DBG] pgmap v6229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:48:59 smithi115 bash[46934]: cluster 2022-04-22T19:48:58.230468+0000 mgr.y (mgr.34107) 4904 : cluster [DBG] pgmap v6229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:00 smithi121 bash[36902]: audit 2022-04-22T19:48:59.726506+0000 mon.a (mon.0) 7824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:01.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:00 smithi115 bash[45495]: audit 2022-04-22T19:48:59.726506+0000 mon.a (mon.0) 7824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:00 smithi115 bash[46934]: audit 2022-04-22T19:48:59.726506+0000 mon.a (mon.0) 7824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:01 smithi121 bash[36902]: cluster 2022-04-22T19:49:00.231557+0000 mgr.y (mgr.34107) 4905 : cluster [DBG] pgmap v6230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:01 smithi121 bash[36902]: audit 2022-04-22T19:49:01.130454+0000 mon.a (mon.0) 7825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:01 smithi115 bash[45495]: cluster 2022-04-22T19:49:00.231557+0000 mgr.y (mgr.34107) 4905 : cluster [DBG] pgmap v6230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:01 smithi115 bash[45495]: audit 2022-04-22T19:49:01.130454+0000 mon.a (mon.0) 7825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:01 smithi115 bash[46934]: cluster 2022-04-22T19:49:00.231557+0000 mgr.y (mgr.34107) 4905 : cluster [DBG] pgmap v6230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:01 smithi115 bash[46934]: audit 2022-04-22T19:49:01.130454+0000 mon.a (mon.0) 7825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:02.167Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:02 smithi121 bash[36902]: audit 2022-04-22T19:49:02.596383+0000 mon.a (mon.0) 7826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:02 smithi115 bash[46934]: audit 2022-04-22T19:49:02.596383+0000 mon.a (mon.0) 7826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:02 smithi115 bash[45495]: audit 2022-04-22T19:49:02.596383+0000 mon.a (mon.0) 7826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:03 smithi121 bash[36902]: cluster 2022-04-22T19:49:02.232614+0000 mgr.y (mgr.34107) 4906 : cluster [DBG] pgmap v6231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:03 smithi115 bash[46934]: cluster 2022-04-22T19:49:02.232614+0000 mgr.y (mgr.34107) 4906 : cluster [DBG] pgmap v6231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:03 smithi115 bash[45495]: cluster 2022-04-22T19:49:02.232614+0000 mgr.y (mgr.34107) 4906 : cluster [DBG] pgmap v6231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:04.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:04.039Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:05 smithi121 bash[36902]: cluster 2022-04-22T19:49:04.233699+0000 mgr.y (mgr.34107) 4907 : cluster [DBG] pgmap v6232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:05 smithi121 bash[36902]: audit 2022-04-22T19:49:04.730121+0000 mon.a (mon.0) 7827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:05 smithi115 bash[45495]: cluster 2022-04-22T19:49:04.233699+0000 mgr.y (mgr.34107) 4907 : cluster [DBG] pgmap v6232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:05 smithi115 bash[45495]: audit 2022-04-22T19:49:04.730121+0000 mon.a (mon.0) 7827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:05 smithi115 bash[46934]: cluster 2022-04-22T19:49:04.233699+0000 mgr.y (mgr.34107) 4907 : cluster [DBG] pgmap v6232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:05 smithi115 bash[46934]: audit 2022-04-22T19:49:04.730121+0000 mon.a (mon.0) 7827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:06 smithi121 bash[36902]: audit 2022-04-22T19:49:06.133743+0000 mon.a (mon.0) 7828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:06 smithi115 bash[45495]: audit 2022-04-22T19:49:06.133743+0000 mon.a (mon.0) 7828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:06 smithi115 bash[46934]: audit 2022-04-22T19:49:06.133743+0000 mon.a (mon.0) 7828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:07 smithi121 bash[36902]: cluster 2022-04-22T19:49:06.234631+0000 mgr.y (mgr.34107) 4908 : cluster [DBG] pgmap v6233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:07 smithi121 bash[36902]: audit 2022-04-22T19:49:07.599254+0000 mon.a (mon.0) 7829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:07 smithi115 bash[45495]: cluster 2022-04-22T19:49:06.234631+0000 mgr.y (mgr.34107) 4908 : cluster [DBG] pgmap v6233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:07 smithi115 bash[45495]: audit 2022-04-22T19:49:07.599254+0000 mon.a (mon.0) 7829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:07 smithi115 bash[46934]: cluster 2022-04-22T19:49:06.234631+0000 mgr.y (mgr.34107) 4908 : cluster [DBG] pgmap v6233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:07 smithi115 bash[46934]: audit 2022-04-22T19:49:07.599254+0000 mon.a (mon.0) 7829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:09 smithi121 bash[36902]: cluster 2022-04-22T19:49:08.235817+0000 mgr.y (mgr.34107) 4909 : cluster [DBG] pgmap v6234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:09 smithi115 bash[45495]: cluster 2022-04-22T19:49:08.235817+0000 mgr.y (mgr.34107) 4909 : cluster [DBG] pgmap v6234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:10.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:09 smithi115 bash[46934]: cluster 2022-04-22T19:49:08.235817+0000 mgr.y (mgr.34107) 4909 : cluster [DBG] pgmap v6234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:10 smithi121 bash[36902]: audit 2022-04-22T19:49:09.733810+0000 mon.a (mon.0) 7830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:10 smithi115 bash[45495]: audit 2022-04-22T19:49:09.733810+0000 mon.a (mon.0) 7830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:10 smithi115 bash[46934]: audit 2022-04-22T19:49:09.733810+0000 mon.a (mon.0) 7830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:11 smithi121 bash[36902]: cluster 2022-04-22T19:49:10.236838+0000 mgr.y (mgr.34107) 4910 : cluster [DBG] pgmap v6235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:11 smithi121 bash[36902]: audit 2022-04-22T19:49:11.136827+0000 mon.a (mon.0) 7831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:11 smithi115 bash[45495]: cluster 2022-04-22T19:49:10.236838+0000 mgr.y (mgr.34107) 4910 : cluster [DBG] pgmap v6235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:11 smithi115 bash[45495]: audit 2022-04-22T19:49:11.136827+0000 mon.a (mon.0) 7831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:11 smithi115 bash[46934]: cluster 2022-04-22T19:49:10.236838+0000 mgr.y (mgr.34107) 4910 : cluster [DBG] pgmap v6235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:11 smithi115 bash[46934]: audit 2022-04-22T19:49:11.136827+0000 mon.a (mon.0) 7831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:12.168Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:12 smithi121 bash[36902]: audit 2022-04-22T19:49:12.602480+0000 mon.a (mon.0) 7832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:13.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:12 smithi115 bash[45495]: audit 2022-04-22T19:49:12.602480+0000 mon.a (mon.0) 7832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:12 smithi115 bash[46934]: audit 2022-04-22T19:49:12.602480+0000 mon.a (mon.0) 7832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:13 smithi121 bash[36902]: cluster 2022-04-22T19:49:12.237871+0000 mgr.y (mgr.34107) 4911 : cluster [DBG] pgmap v6236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:13 smithi115 bash[45495]: cluster 2022-04-22T19:49:12.237871+0000 mgr.y (mgr.34107) 4911 : cluster [DBG] pgmap v6236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:13 smithi115 bash[46934]: cluster 2022-04-22T19:49:12.237871+0000 mgr.y (mgr.34107) 4911 : cluster [DBG] pgmap v6236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:14.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:14.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:14 smithi121 bash[36902]: cluster 2022-04-22T19:49:14.238943+0000 mgr.y (mgr.34107) 4912 : cluster [DBG] pgmap v6237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:15.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:14 smithi115 bash[45495]: cluster 2022-04-22T19:49:14.238943+0000 mgr.y (mgr.34107) 4912 : cluster [DBG] pgmap v6237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:14 smithi115 bash[46934]: cluster 2022-04-22T19:49:14.238943+0000 mgr.y (mgr.34107) 4912 : cluster [DBG] pgmap v6237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:15 smithi121 bash[36902]: audit 2022-04-22T19:49:14.739063+0000 mon.a (mon.0) 7833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:15 smithi121 bash[36902]: audit 2022-04-22T19:49:14.920330+0000 mon.a (mon.0) 7834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:49:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:15 smithi121 bash[36902]: audit 2022-04-22T19:49:14.920548+0000 mon.a (mon.0) 7835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:49:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[46934]: audit 2022-04-22T19:49:14.739063+0000 mon.a (mon.0) 7833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[46934]: audit 2022-04-22T19:49:14.920330+0000 mon.a (mon.0) 7834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:49:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[46934]: audit 2022-04-22T19:49:14.920548+0000 mon.a (mon.0) 7835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:49:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[45495]: audit 2022-04-22T19:49:14.739063+0000 mon.a (mon.0) 7833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[45495]: audit 2022-04-22T19:49:14.920330+0000 mon.a (mon.0) 7834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:49:16.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:15 smithi115 bash[45495]: audit 2022-04-22T19:49:14.920548+0000 mon.a (mon.0) 7835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:49:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:16 smithi121 bash[36902]: audit 2022-04-22T19:49:16.140036+0000 mon.a (mon.0) 7836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:16 smithi121 bash[36902]: cluster 2022-04-22T19:49:16.239977+0000 mgr.y (mgr.34107) 4913 : cluster [DBG] pgmap v6238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:16 smithi115 bash[45495]: audit 2022-04-22T19:49:16.140036+0000 mon.a (mon.0) 7836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:16 smithi115 bash[45495]: cluster 2022-04-22T19:49:16.239977+0000 mgr.y (mgr.34107) 4913 : cluster [DBG] pgmap v6238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:16 smithi115 bash[46934]: audit 2022-04-22T19:49:16.140036+0000 mon.a (mon.0) 7836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:16 smithi115 bash[46934]: cluster 2022-04-22T19:49:16.239977+0000 mgr.y (mgr.34107) 4913 : cluster [DBG] pgmap v6238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:17 smithi121 bash[36902]: audit 2022-04-22T19:49:17.605467+0000 mon.a (mon.0) 7837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:17 smithi115 bash[45495]: audit 2022-04-22T19:49:17.605467+0000 mon.a (mon.0) 7837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:17 smithi115 bash[46934]: audit 2022-04-22T19:49:17.605467+0000 mon.a (mon.0) 7837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:18.462 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:49:18.806 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:49:18.809 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:49:18.809 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:49:18.810 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:49:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:49:19.205 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:49:19.303 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:49:19.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:49:19.305 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[45495]: cluster 2022-04-22T19:49:18.240958+0000 mgr.y (mgr.34107) 4914 : cluster [DBG] pgmap v6239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[45495]: audit 2022-04-22T19:49:18.437539+0000 mgr.y (mgr.34107) 4915 : audit [DBG] from='client.40518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[45495]: audit 2022-04-22T19:49:19.194507+0000 mon.c (mon.1) 211 : audit [DBG] from='client.? 172.21.15.115:0/425031169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[46934]: cluster 2022-04-22T19:49:18.240958+0000 mgr.y (mgr.34107) 4914 : cluster [DBG] pgmap v6239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[46934]: audit 2022-04-22T19:49:18.437539+0000 mgr.y (mgr.34107) 4915 : audit [DBG] from='client.40518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:19 smithi115 bash[46934]: audit 2022-04-22T19:49:19.194507+0000 mon.c (mon.1) 211 : audit [DBG] from='client.? 172.21.15.115:0/425031169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:19 smithi121 bash[36902]: cluster 2022-04-22T19:49:18.240958+0000 mgr.y (mgr.34107) 4914 : cluster [DBG] pgmap v6239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:19 smithi121 bash[36902]: audit 2022-04-22T19:49:18.437539+0000 mgr.y (mgr.34107) 4915 : audit [DBG] from='client.40518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:19 smithi121 bash[36902]: audit 2022-04-22T19:49:19.194507+0000 mon.c (mon.1) 211 : audit [DBG] from='client.? 172.21.15.115:0/425031169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:20 smithi115 bash[45495]: audit 2022-04-22T19:49:18.781568+0000 mgr.y (mgr.34107) 4916 : audit [DBG] from='client.50458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:20.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:20 smithi115 bash[45495]: audit 2022-04-22T19:49:19.740960+0000 mon.a (mon.0) 7838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:20 smithi115 bash[46934]: audit 2022-04-22T19:49:18.781568+0000 mgr.y (mgr.34107) 4916 : audit [DBG] from='client.50458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:20.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:20 smithi115 bash[46934]: audit 2022-04-22T19:49:19.740960+0000 mon.a (mon.0) 7838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:20 smithi121 bash[36902]: audit 2022-04-22T19:49:18.781568+0000 mgr.y (mgr.34107) 4916 : audit [DBG] from='client.50458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:20 smithi121 bash[36902]: audit 2022-04-22T19:49:19.740960+0000 mon.a (mon.0) 7838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:21.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:21 smithi115 bash[45495]: cluster 2022-04-22T19:49:20.241951+0000 mgr.y (mgr.34107) 4917 : cluster [DBG] pgmap v6240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:21.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:21 smithi115 bash[45495]: audit 2022-04-22T19:49:21.143100+0000 mon.a (mon.0) 7839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:21.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:21 smithi115 bash[46934]: cluster 2022-04-22T19:49:20.241951+0000 mgr.y (mgr.34107) 4917 : cluster [DBG] pgmap v6240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:21.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:21 smithi115 bash[46934]: audit 2022-04-22T19:49:21.143100+0000 mon.a (mon.0) 7839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:21 smithi121 bash[36902]: cluster 2022-04-22T19:49:20.241951+0000 mgr.y (mgr.34107) 4917 : cluster [DBG] pgmap v6240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:21 smithi121 bash[36902]: audit 2022-04-22T19:49:21.143100+0000 mon.a (mon.0) 7839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:22.168Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:23 smithi115 bash[45495]: cluster 2022-04-22T19:49:22.243035+0000 mgr.y (mgr.34107) 4918 : cluster [DBG] pgmap v6241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:23 smithi115 bash[45495]: audit 2022-04-22T19:49:22.608311+0000 mon.a (mon.0) 7840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:23 smithi115 bash[46934]: cluster 2022-04-22T19:49:22.243035+0000 mgr.y (mgr.34107) 4918 : cluster [DBG] pgmap v6241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:23 smithi115 bash[46934]: audit 2022-04-22T19:49:22.608311+0000 mon.a (mon.0) 7840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:23 smithi121 bash[36902]: cluster 2022-04-22T19:49:22.243035+0000 mgr.y (mgr.34107) 4918 : cluster [DBG] pgmap v6241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:23 smithi121 bash[36902]: audit 2022-04-22T19:49:22.608311+0000 mon.a (mon.0) 7840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:24.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:24.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:25 smithi121 bash[36902]: cluster 2022-04-22T19:49:24.244176+0000 mgr.y (mgr.34107) 4919 : cluster [DBG] pgmap v6242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:25 smithi121 bash[36902]: audit 2022-04-22T19:49:24.744352+0000 mon.a (mon.0) 7841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:25 smithi115 bash[45495]: cluster 2022-04-22T19:49:24.244176+0000 mgr.y (mgr.34107) 4919 : cluster [DBG] pgmap v6242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:25 smithi115 bash[45495]: audit 2022-04-22T19:49:24.744352+0000 mon.a (mon.0) 7841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:25 smithi115 bash[46934]: cluster 2022-04-22T19:49:24.244176+0000 mgr.y (mgr.34107) 4919 : cluster [DBG] pgmap v6242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:25 smithi115 bash[46934]: audit 2022-04-22T19:49:24.744352+0000 mon.a (mon.0) 7841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:26 smithi121 bash[36902]: audit 2022-04-22T19:49:26.146503+0000 mon.a (mon.0) 7842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:26 smithi115 bash[45495]: audit 2022-04-22T19:49:26.146503+0000 mon.a (mon.0) 7842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:26 smithi115 bash[46934]: audit 2022-04-22T19:49:26.146503+0000 mon.a (mon.0) 7842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:27 smithi121 bash[36902]: cluster 2022-04-22T19:49:26.245151+0000 mgr.y (mgr.34107) 4920 : cluster [DBG] pgmap v6243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:27.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:27 smithi115 bash[45495]: cluster 2022-04-22T19:49:26.245151+0000 mgr.y (mgr.34107) 4920 : cluster [DBG] pgmap v6243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:27 smithi115 bash[46934]: cluster 2022-04-22T19:49:26.245151+0000 mgr.y (mgr.34107) 4920 : cluster [DBG] pgmap v6243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:28 smithi121 bash[36902]: audit 2022-04-22T19:49:27.611325+0000 mon.a (mon.0) 7843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:28 smithi121 bash[36902]: audit 2022-04-22T19:49:28.214367+0000 mon.a (mon.0) 7844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:49:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:28 smithi121 bash[36902]: audit 2022-04-22T19:49:28.217197+0000 mon.a (mon.0) 7845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:49:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:28 smithi121 bash[36902]: audit 2022-04-22T19:49:28.218760+0000 mon.a (mon.0) 7846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:49:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[45495]: audit 2022-04-22T19:49:27.611325+0000 mon.a (mon.0) 7843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[45495]: audit 2022-04-22T19:49:28.214367+0000 mon.a (mon.0) 7844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[45495]: audit 2022-04-22T19:49:28.217197+0000 mon.a (mon.0) 7845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[45495]: audit 2022-04-22T19:49:28.218760+0000 mon.a (mon.0) 7846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[46934]: audit 2022-04-22T19:49:27.611325+0000 mon.a (mon.0) 7843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[46934]: audit 2022-04-22T19:49:28.214367+0000 mon.a (mon.0) 7844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[46934]: audit 2022-04-22T19:49:28.217197+0000 mon.a (mon.0) 7845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:49:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:28 smithi115 bash[46934]: audit 2022-04-22T19:49:28.218760+0000 mon.a (mon.0) 7846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:49:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:29 smithi121 bash[36902]: cluster 2022-04-22T19:49:28.246251+0000 mgr.y (mgr.34107) 4921 : cluster [DBG] pgmap v6244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:29 smithi121 bash[36902]: audit 2022-04-22T19:49:28.583944+0000 mon.a (mon.0) 7847 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:49:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:29 smithi115 bash[45495]: cluster 2022-04-22T19:49:28.246251+0000 mgr.y (mgr.34107) 4921 : cluster [DBG] pgmap v6244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:29 smithi115 bash[45495]: audit 2022-04-22T19:49:28.583944+0000 mon.a (mon.0) 7847 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:49:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:29 smithi115 bash[46934]: cluster 2022-04-22T19:49:28.246251+0000 mgr.y (mgr.34107) 4921 : cluster [DBG] pgmap v6244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:30.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:29 smithi115 bash[46934]: audit 2022-04-22T19:49:28.583944+0000 mon.a (mon.0) 7847 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:49:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:30 smithi121 bash[36902]: audit 2022-04-22T19:49:29.748037+0000 mon.a (mon.0) 7848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:31.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:30 smithi115 bash[45495]: audit 2022-04-22T19:49:29.748037+0000 mon.a (mon.0) 7848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:30 smithi115 bash[46934]: audit 2022-04-22T19:49:29.748037+0000 mon.a (mon.0) 7848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:31 smithi121 bash[36902]: cluster 2022-04-22T19:49:30.247249+0000 mgr.y (mgr.34107) 4922 : cluster [DBG] pgmap v6245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:31 smithi121 bash[36902]: audit 2022-04-22T19:49:31.149678+0000 mon.a (mon.0) 7849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:31 smithi115 bash[45495]: cluster 2022-04-22T19:49:30.247249+0000 mgr.y (mgr.34107) 4922 : cluster [DBG] pgmap v6245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:31 smithi115 bash[45495]: audit 2022-04-22T19:49:31.149678+0000 mon.a (mon.0) 7849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:31 smithi115 bash[46934]: cluster 2022-04-22T19:49:30.247249+0000 mgr.y (mgr.34107) 4922 : cluster [DBG] pgmap v6245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:31 smithi115 bash[46934]: audit 2022-04-22T19:49:31.149678+0000 mon.a (mon.0) 7849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:32.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:32.168Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:33 smithi121 bash[36902]: cluster 2022-04-22T19:49:32.247989+0000 mgr.y (mgr.34107) 4923 : cluster [DBG] pgmap v6246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:33 smithi121 bash[36902]: audit 2022-04-22T19:49:32.614419+0000 mon.a (mon.0) 7850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:33 smithi115 bash[46934]: cluster 2022-04-22T19:49:32.247989+0000 mgr.y (mgr.34107) 4923 : cluster [DBG] pgmap v6246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:33 smithi115 bash[46934]: audit 2022-04-22T19:49:32.614419+0000 mon.a (mon.0) 7850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:33 smithi115 bash[45495]: cluster 2022-04-22T19:49:32.247989+0000 mgr.y (mgr.34107) 4923 : cluster [DBG] pgmap v6246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:33 smithi115 bash[45495]: audit 2022-04-22T19:49:32.614419+0000 mon.a (mon.0) 7850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:34.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:34.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:35 smithi121 bash[36902]: cluster 2022-04-22T19:49:34.248977+0000 mgr.y (mgr.34107) 4924 : cluster [DBG] pgmap v6247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:35 smithi121 bash[36902]: audit 2022-04-22T19:49:34.751540+0000 mon.a (mon.0) 7851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:35 smithi115 bash[45495]: cluster 2022-04-22T19:49:34.248977+0000 mgr.y (mgr.34107) 4924 : cluster [DBG] pgmap v6247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:35 smithi115 bash[45495]: audit 2022-04-22T19:49:34.751540+0000 mon.a (mon.0) 7851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:35 smithi115 bash[46934]: cluster 2022-04-22T19:49:34.248977+0000 mgr.y (mgr.34107) 4924 : cluster [DBG] pgmap v6247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:35 smithi115 bash[46934]: audit 2022-04-22T19:49:34.751540+0000 mon.a (mon.0) 7851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:37.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:36 smithi121 bash[36902]: audit 2022-04-22T19:49:36.152926+0000 mon.a (mon.0) 7852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:36 smithi115 bash[45495]: audit 2022-04-22T19:49:36.152926+0000 mon.a (mon.0) 7852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:36 smithi115 bash[46934]: audit 2022-04-22T19:49:36.152926+0000 mon.a (mon.0) 7852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:37 smithi121 bash[36902]: cluster 2022-04-22T19:49:36.250210+0000 mgr.y (mgr.34107) 4925 : cluster [DBG] pgmap v6248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:37 smithi115 bash[45495]: cluster 2022-04-22T19:49:36.250210+0000 mgr.y (mgr.34107) 4925 : cluster [DBG] pgmap v6248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:37 smithi115 bash[46934]: cluster 2022-04-22T19:49:36.250210+0000 mgr.y (mgr.34107) 4925 : cluster [DBG] pgmap v6248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:38 smithi121 bash[36902]: audit 2022-04-22T19:49:37.618321+0000 mon.a (mon.0) 7853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:38 smithi115 bash[45495]: audit 2022-04-22T19:49:37.618321+0000 mon.a (mon.0) 7853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:38 smithi115 bash[46934]: audit 2022-04-22T19:49:37.618321+0000 mon.a (mon.0) 7853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:39 smithi121 bash[36902]: cluster 2022-04-22T19:49:38.251422+0000 mgr.y (mgr.34107) 4926 : cluster [DBG] pgmap v6249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:39 smithi115 bash[46934]: cluster 2022-04-22T19:49:38.251422+0000 mgr.y (mgr.34107) 4926 : cluster [DBG] pgmap v6249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:40.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:39 smithi115 bash[45495]: cluster 2022-04-22T19:49:38.251422+0000 mgr.y (mgr.34107) 4926 : cluster [DBG] pgmap v6249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:40 smithi121 bash[36902]: audit 2022-04-22T19:49:39.755372+0000 mon.a (mon.0) 7854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:41.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:40 smithi115 bash[45495]: audit 2022-04-22T19:49:39.755372+0000 mon.a (mon.0) 7854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:40 smithi115 bash[46934]: audit 2022-04-22T19:49:39.755372+0000 mon.a (mon.0) 7854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:41 smithi121 bash[36902]: cluster 2022-04-22T19:49:40.252485+0000 mgr.y (mgr.34107) 4927 : cluster [DBG] pgmap v6250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:41 smithi121 bash[36902]: audit 2022-04-22T19:49:41.156199+0000 mon.a (mon.0) 7855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:41 smithi115 bash[45495]: cluster 2022-04-22T19:49:40.252485+0000 mgr.y (mgr.34107) 4927 : cluster [DBG] pgmap v6250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:41 smithi115 bash[45495]: audit 2022-04-22T19:49:41.156199+0000 mon.a (mon.0) 7855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:41 smithi115 bash[46934]: cluster 2022-04-22T19:49:40.252485+0000 mgr.y (mgr.34107) 4927 : cluster [DBG] pgmap v6250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:41 smithi115 bash[46934]: audit 2022-04-22T19:49:41.156199+0000 mon.a (mon.0) 7855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:42.169Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:42 smithi121 bash[36902]: audit 2022-04-22T19:49:42.620670+0000 mon.a (mon.0) 7856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:42 smithi115 bash[45495]: audit 2022-04-22T19:49:42.620670+0000 mon.a (mon.0) 7856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:42 smithi115 bash[46934]: audit 2022-04-22T19:49:42.620670+0000 mon.a (mon.0) 7856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:43 smithi121 bash[36902]: cluster 2022-04-22T19:49:42.253554+0000 mgr.y (mgr.34107) 4928 : cluster [DBG] pgmap v6251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:43 smithi115 bash[46934]: cluster 2022-04-22T19:49:42.253554+0000 mgr.y (mgr.34107) 4928 : cluster [DBG] pgmap v6251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:43 smithi115 bash[45495]: cluster 2022-04-22T19:49:42.253554+0000 mgr.y (mgr.34107) 4928 : cluster [DBG] pgmap v6251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:44.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:44.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:45 smithi121 bash[36902]: cluster 2022-04-22T19:49:44.254677+0000 mgr.y (mgr.34107) 4929 : cluster [DBG] pgmap v6252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:45 smithi121 bash[36902]: audit 2022-04-22T19:49:44.758997+0000 mon.a (mon.0) 7857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:45 smithi115 bash[45495]: cluster 2022-04-22T19:49:44.254677+0000 mgr.y (mgr.34107) 4929 : cluster [DBG] pgmap v6252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:45 smithi115 bash[45495]: audit 2022-04-22T19:49:44.758997+0000 mon.a (mon.0) 7857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:45 smithi115 bash[46934]: cluster 2022-04-22T19:49:44.254677+0000 mgr.y (mgr.34107) 4929 : cluster [DBG] pgmap v6252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:45 smithi115 bash[46934]: audit 2022-04-22T19:49:44.758997+0000 mon.a (mon.0) 7857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:46 smithi121 bash[36902]: audit 2022-04-22T19:49:46.159446+0000 mon.a (mon.0) 7858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:46 smithi115 bash[45495]: audit 2022-04-22T19:49:46.159446+0000 mon.a (mon.0) 7858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:46 smithi115 bash[46934]: audit 2022-04-22T19:49:46.159446+0000 mon.a (mon.0) 7858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:47 smithi121 bash[36902]: cluster 2022-04-22T19:49:46.255711+0000 mgr.y (mgr.34107) 4930 : cluster [DBG] pgmap v6253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:47 smithi121 bash[36902]: audit 2022-04-22T19:49:47.623610+0000 mon.a (mon.0) 7859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:47 smithi115 bash[45495]: cluster 2022-04-22T19:49:46.255711+0000 mgr.y (mgr.34107) 4930 : cluster [DBG] pgmap v6253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:48.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:47 smithi115 bash[45495]: audit 2022-04-22T19:49:47.623610+0000 mon.a (mon.0) 7859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:47 smithi115 bash[46934]: cluster 2022-04-22T19:49:46.255711+0000 mgr.y (mgr.34107) 4930 : cluster [DBG] pgmap v6253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:47 smithi115 bash[46934]: audit 2022-04-22T19:49:47.623610+0000 mon.a (mon.0) 7859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:49.582 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:49:49.920 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:49:49.920 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:49:49.921 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:49:49.922 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:49:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:49 smithi121 bash[36902]: cluster 2022-04-22T19:49:48.256891+0000 mgr.y (mgr.34107) 4931 : cluster [DBG] pgmap v6254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:49 smithi115 bash[46934]: cluster 2022-04-22T19:49:48.256891+0000 mgr.y (mgr.34107) 4931 : cluster [DBG] pgmap v6254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:49 smithi115 bash[45495]: cluster 2022-04-22T19:49:48.256891+0000 mgr.y (mgr.34107) 4931 : cluster [DBG] pgmap v6254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:50.318 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:49:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:49:50.320 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:49:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:50 smithi121 bash[36902]: audit 2022-04-22T19:49:49.556058+0000 mgr.y (mgr.34107) 4932 : audit [DBG] from='client.50470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:50 smithi121 bash[36902]: audit 2022-04-22T19:49:49.762506+0000 mon.a (mon.0) 7860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:50 smithi121 bash[36902]: audit 2022-04-22T19:49:50.308981+0000 mon.c (mon.1) 212 : audit [DBG] from='client.? 172.21.15.115:0/33811400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[45495]: audit 2022-04-22T19:49:49.556058+0000 mgr.y (mgr.34107) 4932 : audit [DBG] from='client.50470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[45495]: audit 2022-04-22T19:49:49.762506+0000 mon.a (mon.0) 7860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[45495]: audit 2022-04-22T19:49:50.308981+0000 mon.c (mon.1) 212 : audit [DBG] from='client.? 172.21.15.115:0/33811400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[46934]: audit 2022-04-22T19:49:49.556058+0000 mgr.y (mgr.34107) 4932 : audit [DBG] from='client.50470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[46934]: audit 2022-04-22T19:49:49.762506+0000 mon.a (mon.0) 7860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:50 smithi115 bash[46934]: audit 2022-04-22T19:49:50.308981+0000 mon.c (mon.1) 212 : audit [DBG] from='client.? 172.21.15.115:0/33811400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:49:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:51 smithi121 bash[36902]: audit 2022-04-22T19:49:49.893057+0000 mgr.y (mgr.34107) 4933 : audit [DBG] from='client.40542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:52.025 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:51 smithi121 bash[36902]: cluster 2022-04-22T19:49:50.257617+0000 mgr.y (mgr.34107) 4934 : cluster [DBG] pgmap v6255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:52.026 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:51 smithi121 bash[36902]: audit 2022-04-22T19:49:51.162565+0000 mon.a (mon.0) 7861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[45495]: audit 2022-04-22T19:49:49.893057+0000 mgr.y (mgr.34107) 4933 : audit [DBG] from='client.40542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[45495]: cluster 2022-04-22T19:49:50.257617+0000 mgr.y (mgr.34107) 4934 : cluster [DBG] pgmap v6255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[45495]: audit 2022-04-22T19:49:51.162565+0000 mon.a (mon.0) 7861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[46934]: audit 2022-04-22T19:49:49.893057+0000 mgr.y (mgr.34107) 4933 : audit [DBG] from='client.40542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[46934]: cluster 2022-04-22T19:49:50.257617+0000 mgr.y (mgr.34107) 4934 : cluster [DBG] pgmap v6255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:51 smithi115 bash[46934]: audit 2022-04-22T19:49:51.162565+0000 mon.a (mon.0) 7861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:52.170Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:52 smithi121 bash[36902]: audit 2022-04-22T19:49:52.626672+0000 mon.a (mon.0) 7862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:52 smithi115 bash[45495]: audit 2022-04-22T19:49:52.626672+0000 mon.a (mon.0) 7862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:52 smithi115 bash[46934]: audit 2022-04-22T19:49:52.626672+0000 mon.a (mon.0) 7862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:53 smithi121 bash[36902]: cluster 2022-04-22T19:49:52.258672+0000 mgr.y (mgr.34107) 4935 : cluster [DBG] pgmap v6256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:53 smithi115 bash[45495]: cluster 2022-04-22T19:49:52.258672+0000 mgr.y (mgr.34107) 4935 : cluster [DBG] pgmap v6256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:53 smithi115 bash[46934]: cluster 2022-04-22T19:49:52.258672+0000 mgr.y (mgr.34107) 4935 : cluster [DBG] pgmap v6256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:54.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:49:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:49:54.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:49:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:55 smithi121 bash[36902]: cluster 2022-04-22T19:49:54.259734+0000 mgr.y (mgr.34107) 4936 : cluster [DBG] pgmap v6257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:55 smithi121 bash[36902]: audit 2022-04-22T19:49:54.766208+0000 mon.a (mon.0) 7863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:55 smithi115 bash[45495]: cluster 2022-04-22T19:49:54.259734+0000 mgr.y (mgr.34107) 4936 : cluster [DBG] pgmap v6257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:55 smithi115 bash[45495]: audit 2022-04-22T19:49:54.766208+0000 mon.a (mon.0) 7863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:55 smithi115 bash[46934]: cluster 2022-04-22T19:49:54.259734+0000 mgr.y (mgr.34107) 4936 : cluster [DBG] pgmap v6257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:55 smithi115 bash[46934]: audit 2022-04-22T19:49:54.766208+0000 mon.a (mon.0) 7863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:49:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:56 smithi121 bash[36902]: audit 2022-04-22T19:49:56.166074+0000 mon.a (mon.0) 7864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:56 smithi115 bash[45495]: audit 2022-04-22T19:49:56.166074+0000 mon.a (mon.0) 7864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:57.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:56 smithi115 bash[46934]: audit 2022-04-22T19:49:56.166074+0000 mon.a (mon.0) 7864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:49:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:57 smithi121 bash[36902]: cluster 2022-04-22T19:49:56.260816+0000 mgr.y (mgr.34107) 4937 : cluster [DBG] pgmap v6258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:57 smithi121 bash[36902]: audit 2022-04-22T19:49:57.629937+0000 mon.a (mon.0) 7865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:57 smithi115 bash[45495]: cluster 2022-04-22T19:49:56.260816+0000 mgr.y (mgr.34107) 4937 : cluster [DBG] pgmap v6258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:57 smithi115 bash[45495]: audit 2022-04-22T19:49:57.629937+0000 mon.a (mon.0) 7865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:49:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:57 smithi115 bash[46934]: cluster 2022-04-22T19:49:56.260816+0000 mgr.y (mgr.34107) 4937 : cluster [DBG] pgmap v6258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:49:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:57 smithi115 bash[46934]: audit 2022-04-22T19:49:57.629937+0000 mon.a (mon.0) 7865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:49:59 smithi121 bash[36902]: cluster 2022-04-22T19:49:58.261977+0000 mgr.y (mgr.34107) 4938 : cluster [DBG] pgmap v6259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:00.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:49:59 smithi115 bash[45495]: cluster 2022-04-22T19:49:58.261977+0000 mgr.y (mgr.34107) 4938 : cluster [DBG] pgmap v6259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:00.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:49:59 smithi115 bash[46934]: cluster 2022-04-22T19:49:58.261977+0000 mgr.y (mgr.34107) 4938 : cluster [DBG] pgmap v6259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:00 smithi121 bash[36902]: audit 2022-04-22T19:49:59.770027+0000 mon.a (mon.0) 7866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:00 smithi121 bash[36902]: cluster 2022-04-22T19:50:00.000181+0000 mon.a (mon.0) 7867 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:50:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:00 smithi115 bash[45495]: audit 2022-04-22T19:49:59.770027+0000 mon.a (mon.0) 7866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:01.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:00 smithi115 bash[45495]: cluster 2022-04-22T19:50:00.000181+0000 mon.a (mon.0) 7867 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:50:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:00 smithi115 bash[46934]: audit 2022-04-22T19:49:59.770027+0000 mon.a (mon.0) 7866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:00 smithi115 bash[46934]: cluster 2022-04-22T19:50:00.000181+0000 mon.a (mon.0) 7867 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T19:50:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:01 smithi121 bash[36902]: cluster 2022-04-22T19:50:00.263084+0000 mgr.y (mgr.34107) 4939 : cluster [DBG] pgmap v6260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:01 smithi121 bash[36902]: audit 2022-04-22T19:50:01.169016+0000 mon.a (mon.0) 7868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:01 smithi115 bash[45495]: cluster 2022-04-22T19:50:00.263084+0000 mgr.y (mgr.34107) 4939 : cluster [DBG] pgmap v6260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:01 smithi115 bash[45495]: audit 2022-04-22T19:50:01.169016+0000 mon.a (mon.0) 7868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:01 smithi115 bash[46934]: cluster 2022-04-22T19:50:00.263084+0000 mgr.y (mgr.34107) 4939 : cluster [DBG] pgmap v6260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:01 smithi115 bash[46934]: audit 2022-04-22T19:50:01.169016+0000 mon.a (mon.0) 7868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:02.171Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:02 smithi121 bash[36902]: audit 2022-04-22T19:50:02.632914+0000 mon.a (mon.0) 7869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:02 smithi115 bash[45495]: audit 2022-04-22T19:50:02.632914+0000 mon.a (mon.0) 7869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:02 smithi115 bash[46934]: audit 2022-04-22T19:50:02.632914+0000 mon.a (mon.0) 7869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:03 smithi121 bash[36902]: cluster 2022-04-22T19:50:02.264221+0000 mgr.y (mgr.34107) 4940 : cluster [DBG] pgmap v6261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:03 smithi115 bash[45495]: cluster 2022-04-22T19:50:02.264221+0000 mgr.y (mgr.34107) 4940 : cluster [DBG] pgmap v6261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:03 smithi115 bash[46934]: cluster 2022-04-22T19:50:02.264221+0000 mgr.y (mgr.34107) 4940 : cluster [DBG] pgmap v6261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:04.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:05 smithi121 bash[36902]: cluster 2022-04-22T19:50:04.265315+0000 mgr.y (mgr.34107) 4941 : cluster [DBG] pgmap v6262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:05 smithi121 bash[36902]: audit 2022-04-22T19:50:04.773526+0000 mon.a (mon.0) 7870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:05 smithi115 bash[45495]: cluster 2022-04-22T19:50:04.265315+0000 mgr.y (mgr.34107) 4941 : cluster [DBG] pgmap v6262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:05 smithi115 bash[45495]: audit 2022-04-22T19:50:04.773526+0000 mon.a (mon.0) 7870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:05 smithi115 bash[46934]: cluster 2022-04-22T19:50:04.265315+0000 mgr.y (mgr.34107) 4941 : cluster [DBG] pgmap v6262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:06.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:05 smithi115 bash[46934]: audit 2022-04-22T19:50:04.773526+0000 mon.a (mon.0) 7870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:06 smithi121 bash[36902]: audit 2022-04-22T19:50:06.172495+0000 mon.a (mon.0) 7871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:06 smithi115 bash[45495]: audit 2022-04-22T19:50:06.172495+0000 mon.a (mon.0) 7871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:06 smithi115 bash[46934]: audit 2022-04-22T19:50:06.172495+0000 mon.a (mon.0) 7871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:07 smithi121 bash[36902]: cluster 2022-04-22T19:50:06.266404+0000 mgr.y (mgr.34107) 4942 : cluster [DBG] pgmap v6263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:07 smithi121 bash[36902]: audit 2022-04-22T19:50:07.636176+0000 mon.a (mon.0) 7872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:07 smithi115 bash[45495]: cluster 2022-04-22T19:50:06.266404+0000 mgr.y (mgr.34107) 4942 : cluster [DBG] pgmap v6263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:07 smithi115 bash[45495]: audit 2022-04-22T19:50:07.636176+0000 mon.a (mon.0) 7872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:07 smithi115 bash[46934]: cluster 2022-04-22T19:50:06.266404+0000 mgr.y (mgr.34107) 4942 : cluster [DBG] pgmap v6263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:07 smithi115 bash[46934]: audit 2022-04-22T19:50:07.636176+0000 mon.a (mon.0) 7872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:09 smithi115 bash[45495]: cluster 2022-04-22T19:50:08.267532+0000 mgr.y (mgr.34107) 4943 : cluster [DBG] pgmap v6264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:09.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:09 smithi115 bash[46934]: cluster 2022-04-22T19:50:08.267532+0000 mgr.y (mgr.34107) 4943 : cluster [DBG] pgmap v6264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:09 smithi121 bash[36902]: cluster 2022-04-22T19:50:08.267532+0000 mgr.y (mgr.34107) 4943 : cluster [DBG] pgmap v6264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:10 smithi121 bash[36902]: audit 2022-04-22T19:50:09.777360+0000 mon.a (mon.0) 7873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:10 smithi115 bash[45495]: audit 2022-04-22T19:50:09.777360+0000 mon.a (mon.0) 7873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:10 smithi115 bash[46934]: audit 2022-04-22T19:50:09.777360+0000 mon.a (mon.0) 7873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:11 smithi121 bash[36902]: cluster 2022-04-22T19:50:10.268553+0000 mgr.y (mgr.34107) 4944 : cluster [DBG] pgmap v6265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:11 smithi121 bash[36902]: audit 2022-04-22T19:50:11.175608+0000 mon.a (mon.0) 7874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:11 smithi115 bash[45495]: cluster 2022-04-22T19:50:10.268553+0000 mgr.y (mgr.34107) 4944 : cluster [DBG] pgmap v6265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:11 smithi115 bash[45495]: audit 2022-04-22T19:50:11.175608+0000 mon.a (mon.0) 7874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:11 smithi115 bash[46934]: cluster 2022-04-22T19:50:10.268553+0000 mgr.y (mgr.34107) 4944 : cluster [DBG] pgmap v6265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:11 smithi115 bash[46934]: audit 2022-04-22T19:50:11.175608+0000 mon.a (mon.0) 7874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:12.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:12.172Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:13 smithi121 bash[36902]: cluster 2022-04-22T19:50:12.269662+0000 mgr.y (mgr.34107) 4945 : cluster [DBG] pgmap v6266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:13 smithi121 bash[36902]: audit 2022-04-22T19:50:12.639077+0000 mon.a (mon.0) 7875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:13 smithi115 bash[45495]: cluster 2022-04-22T19:50:12.269662+0000 mgr.y (mgr.34107) 4945 : cluster [DBG] pgmap v6266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:13 smithi115 bash[45495]: audit 2022-04-22T19:50:12.639077+0000 mon.a (mon.0) 7875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:13 smithi115 bash[46934]: cluster 2022-04-22T19:50:12.269662+0000 mgr.y (mgr.34107) 4945 : cluster [DBG] pgmap v6266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:13 smithi115 bash[46934]: audit 2022-04-22T19:50:12.639077+0000 mon.a (mon.0) 7875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:14.044Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:15 smithi121 bash[36902]: cluster 2022-04-22T19:50:14.270783+0000 mgr.y (mgr.34107) 4946 : cluster [DBG] pgmap v6267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:15 smithi121 bash[36902]: audit 2022-04-22T19:50:14.781084+0000 mon.a (mon.0) 7876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:15 smithi121 bash[36902]: audit 2022-04-22T19:50:14.929790+0000 mon.a (mon.0) 7877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:50:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:15 smithi121 bash[36902]: audit 2022-04-22T19:50:14.932245+0000 mon.a (mon.0) 7878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:50:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[45495]: cluster 2022-04-22T19:50:14.270783+0000 mgr.y (mgr.34107) 4946 : cluster [DBG] pgmap v6267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:15.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[45495]: audit 2022-04-22T19:50:14.781084+0000 mon.a (mon.0) 7876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:15.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[45495]: audit 2022-04-22T19:50:14.929790+0000 mon.a (mon.0) 7877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:50:15.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[45495]: audit 2022-04-22T19:50:14.932245+0000 mon.a (mon.0) 7878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:50:15.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[46934]: cluster 2022-04-22T19:50:14.270783+0000 mgr.y (mgr.34107) 4946 : cluster [DBG] pgmap v6267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:15.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[46934]: audit 2022-04-22T19:50:14.781084+0000 mon.a (mon.0) 7876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:15.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[46934]: audit 2022-04-22T19:50:14.929790+0000 mon.a (mon.0) 7877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:50:15.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:15 smithi115 bash[46934]: audit 2022-04-22T19:50:14.932245+0000 mon.a (mon.0) 7878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:50:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:16 smithi121 bash[36902]: audit 2022-04-22T19:50:16.178782+0000 mon.a (mon.0) 7879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:16.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:16 smithi115 bash[45495]: audit 2022-04-22T19:50:16.178782+0000 mon.a (mon.0) 7879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:16 smithi115 bash[46934]: audit 2022-04-22T19:50:16.178782+0000 mon.a (mon.0) 7879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:17 smithi121 bash[36902]: cluster 2022-04-22T19:50:16.271841+0000 mgr.y (mgr.34107) 4947 : cluster [DBG] pgmap v6268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:17 smithi115 bash[45495]: cluster 2022-04-22T19:50:16.271841+0000 mgr.y (mgr.34107) 4947 : cluster [DBG] pgmap v6268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:17 smithi115 bash[46934]: cluster 2022-04-22T19:50:16.271841+0000 mgr.y (mgr.34107) 4947 : cluster [DBG] pgmap v6268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:18 smithi121 bash[36902]: audit 2022-04-22T19:50:17.642018+0000 mon.a (mon.0) 7880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:18 smithi115 bash[45495]: audit 2022-04-22T19:50:17.642018+0000 mon.a (mon.0) 7880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:18.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:18 smithi115 bash[46934]: audit 2022-04-22T19:50:17.642018+0000 mon.a (mon.0) 7880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:19 smithi121 bash[36902]: cluster 2022-04-22T19:50:18.273107+0000 mgr.y (mgr.34107) 4948 : cluster [DBG] pgmap v6269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:19 smithi115 bash[45495]: cluster 2022-04-22T19:50:18.273107+0000 mgr.y (mgr.34107) 4948 : cluster [DBG] pgmap v6269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:19 smithi115 bash[46934]: cluster 2022-04-22T19:50:18.273107+0000 mgr.y (mgr.34107) 4948 : cluster [DBG] pgmap v6269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:20.697 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:50:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:20 smithi121 bash[36902]: audit 2022-04-22T19:50:19.784809+0000 mon.a (mon.0) 7881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:20 smithi115 bash[46934]: audit 2022-04-22T19:50:19.784809+0000 mon.a (mon.0) 7881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:20 smithi115 bash[45495]: audit 2022-04-22T19:50:19.784809+0000 mon.a (mon.0) 7881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:21.034 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:50:21.034 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:50:21.034 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 10m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:50:21.034 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:50:21.034 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 10m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 10m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 10m ago 2h 184M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 10m ago 2h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:50:21.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:50:21.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 10m ago 2h 573M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:50:21.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 10m ago 2h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:50:21.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 10m ago 2h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:50:21.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 10m ago 2h 635M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:50:21.036 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 10m ago 2h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:50:21.442 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:50:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:50:21.444 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:50:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:21 smithi121 bash[36902]: cluster 2022-04-22T19:50:20.274282+0000 mgr.y (mgr.34107) 4949 : cluster [DBG] pgmap v6270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:21 smithi121 bash[36902]: audit 2022-04-22T19:50:20.670313+0000 mgr.y (mgr.34107) 4950 : audit [DBG] from='client.40554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:21 smithi121 bash[36902]: audit 2022-04-22T19:50:21.182045+0000 mon.a (mon.0) 7882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[45495]: cluster 2022-04-22T19:50:20.274282+0000 mgr.y (mgr.34107) 4949 : cluster [DBG] pgmap v6270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[45495]: audit 2022-04-22T19:50:20.670313+0000 mgr.y (mgr.34107) 4950 : audit [DBG] from='client.40554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[45495]: audit 2022-04-22T19:50:21.182045+0000 mon.a (mon.0) 7882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[46934]: cluster 2022-04-22T19:50:20.274282+0000 mgr.y (mgr.34107) 4949 : cluster [DBG] pgmap v6270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[46934]: audit 2022-04-22T19:50:20.670313+0000 mgr.y (mgr.34107) 4950 : audit [DBG] from='client.40554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:21 smithi115 bash[46934]: audit 2022-04-22T19:50:21.182045+0000 mon.a (mon.0) 7882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:22.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:22.173Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:22.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:22 smithi115 bash[45495]: audit 2022-04-22T19:50:21.008812+0000 mgr.y (mgr.34107) 4951 : audit [DBG] from='client.50494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:22.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:22 smithi115 bash[45495]: audit 2022-04-22T19:50:21.432932+0000 mon.c (mon.1) 213 : audit [DBG] from='client.? 172.21.15.115:0/4138673619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:22 smithi115 bash[46934]: audit 2022-04-22T19:50:21.008812+0000 mgr.y (mgr.34107) 4951 : audit [DBG] from='client.50494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:22.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:22 smithi115 bash[46934]: audit 2022-04-22T19:50:21.432932+0000 mon.c (mon.1) 213 : audit [DBG] from='client.? 172.21.15.115:0/4138673619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:22 smithi121 bash[36902]: audit 2022-04-22T19:50:21.008812+0000 mgr.y (mgr.34107) 4951 : audit [DBG] from='client.50494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:22 smithi121 bash[36902]: audit 2022-04-22T19:50:21.432932+0000 mon.c (mon.1) 213 : audit [DBG] from='client.? 172.21.15.115:0/4138673619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:23 smithi121 bash[36902]: cluster 2022-04-22T19:50:22.275433+0000 mgr.y (mgr.34107) 4952 : cluster [DBG] pgmap v6271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:23 smithi121 bash[36902]: audit 2022-04-22T19:50:22.644957+0000 mon.a (mon.0) 7883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:23 smithi115 bash[45495]: cluster 2022-04-22T19:50:22.275433+0000 mgr.y (mgr.34107) 4952 : cluster [DBG] pgmap v6271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:23 smithi115 bash[45495]: audit 2022-04-22T19:50:22.644957+0000 mon.a (mon.0) 7883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:23 smithi115 bash[46934]: cluster 2022-04-22T19:50:22.275433+0000 mgr.y (mgr.34107) 4952 : cluster [DBG] pgmap v6271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:23 smithi115 bash[46934]: audit 2022-04-22T19:50:22.644957+0000 mon.a (mon.0) 7883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:24.045Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:25 smithi121 bash[36902]: cluster 2022-04-22T19:50:24.276457+0000 mgr.y (mgr.34107) 4953 : cluster [DBG] pgmap v6272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:25 smithi121 bash[36902]: audit 2022-04-22T19:50:24.788270+0000 mon.a (mon.0) 7884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:25 smithi115 bash[46934]: cluster 2022-04-22T19:50:24.276457+0000 mgr.y (mgr.34107) 4953 : cluster [DBG] pgmap v6272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:25 smithi115 bash[46934]: audit 2022-04-22T19:50:24.788270+0000 mon.a (mon.0) 7884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:25 smithi115 bash[45495]: cluster 2022-04-22T19:50:24.276457+0000 mgr.y (mgr.34107) 4953 : cluster [DBG] pgmap v6272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:25 smithi115 bash[45495]: audit 2022-04-22T19:50:24.788270+0000 mon.a (mon.0) 7884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:26 smithi121 bash[36902]: audit 2022-04-22T19:50:26.184952+0000 mon.a (mon.0) 7885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:26 smithi115 bash[45495]: audit 2022-04-22T19:50:26.184952+0000 mon.a (mon.0) 7885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:26 smithi115 bash[46934]: audit 2022-04-22T19:50:26.184952+0000 mon.a (mon.0) 7885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:27 smithi121 bash[36902]: cluster 2022-04-22T19:50:26.277530+0000 mgr.y (mgr.34107) 4954 : cluster [DBG] pgmap v6273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:27 smithi115 bash[45495]: cluster 2022-04-22T19:50:26.277530+0000 mgr.y (mgr.34107) 4954 : cluster [DBG] pgmap v6273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:27 smithi115 bash[46934]: cluster 2022-04-22T19:50:26.277530+0000 mgr.y (mgr.34107) 4954 : cluster [DBG] pgmap v6273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:28.649 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:28 smithi121 bash[36902]: audit 2022-04-22T19:50:27.647920+0000 mon.a (mon.0) 7886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:28.650 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:28 smithi115 bash[45495]: audit 2022-04-22T19:50:27.647920+0000 mon.a (mon.0) 7886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:28.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:28 smithi115 bash[46934]: audit 2022-04-22T19:50:27.647920+0000 mon.a (mon.0) 7886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:29 smithi121 bash[36902]: cluster 2022-04-22T19:50:28.278649+0000 mgr.y (mgr.34107) 4955 : cluster [DBG] pgmap v6274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:29 smithi121 bash[36902]: audit 2022-04-22T19:50:28.592878+0000 mon.a (mon.0) 7887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:50:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:29 smithi121 bash[36902]: audit 2022-04-22T19:50:28.595873+0000 mon.a (mon.0) 7888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:50:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:29 smithi121 bash[36902]: audit 2022-04-22T19:50:28.597604+0000 mon.a (mon.0) 7889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:50:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[45495]: cluster 2022-04-22T19:50:28.278649+0000 mgr.y (mgr.34107) 4955 : cluster [DBG] pgmap v6274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[45495]: audit 2022-04-22T19:50:28.592878+0000 mon.a (mon.0) 7887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[45495]: audit 2022-04-22T19:50:28.595873+0000 mon.a (mon.0) 7888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[45495]: audit 2022-04-22T19:50:28.597604+0000 mon.a (mon.0) 7889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[46934]: cluster 2022-04-22T19:50:28.278649+0000 mgr.y (mgr.34107) 4955 : cluster [DBG] pgmap v6274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[46934]: audit 2022-04-22T19:50:28.592878+0000 mon.a (mon.0) 7887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[46934]: audit 2022-04-22T19:50:28.595873+0000 mon.a (mon.0) 7888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:50:29.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:29 smithi115 bash[46934]: audit 2022-04-22T19:50:28.597604+0000 mon.a (mon.0) 7889 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:50:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:30 smithi121 bash[36902]: audit 2022-04-22T19:50:29.792031+0000 mon.a (mon.0) 7890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:30.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:30 smithi115 bash[45495]: audit 2022-04-22T19:50:29.792031+0000 mon.a (mon.0) 7890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:30 smithi115 bash[46934]: audit 2022-04-22T19:50:29.792031+0000 mon.a (mon.0) 7890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:31 smithi121 bash[36902]: cluster 2022-04-22T19:50:30.279747+0000 mgr.y (mgr.34107) 4956 : cluster [DBG] pgmap v6275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:31 smithi121 bash[36902]: audit 2022-04-22T19:50:31.188051+0000 mon.a (mon.0) 7891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:31 smithi115 bash[46934]: cluster 2022-04-22T19:50:30.279747+0000 mgr.y (mgr.34107) 4956 : cluster [DBG] pgmap v6275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:31 smithi115 bash[46934]: audit 2022-04-22T19:50:31.188051+0000 mon.a (mon.0) 7891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:31 smithi115 bash[45495]: cluster 2022-04-22T19:50:30.279747+0000 mgr.y (mgr.34107) 4956 : cluster [DBG] pgmap v6275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:31.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:31 smithi115 bash[45495]: audit 2022-04-22T19:50:31.188051+0000 mon.a (mon.0) 7891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:32.174Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:33 smithi121 bash[36902]: cluster 2022-04-22T19:50:32.280461+0000 mgr.y (mgr.34107) 4957 : cluster [DBG] pgmap v6276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:33 smithi121 bash[36902]: audit 2022-04-22T19:50:32.650824+0000 mon.a (mon.0) 7892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:33 smithi121 bash[36902]: audit 2022-04-22T19:50:33.064127+0000 mon.a (mon.0) 7893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[45495]: cluster 2022-04-22T19:50:32.280461+0000 mgr.y (mgr.34107) 4957 : cluster [DBG] pgmap v6276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[45495]: audit 2022-04-22T19:50:32.650824+0000 mon.a (mon.0) 7892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[45495]: audit 2022-04-22T19:50:33.064127+0000 mon.a (mon.0) 7893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[46934]: cluster 2022-04-22T19:50:32.280461+0000 mgr.y (mgr.34107) 4957 : cluster [DBG] pgmap v6276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[46934]: audit 2022-04-22T19:50:32.650824+0000 mon.a (mon.0) 7892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:33 smithi115 bash[46934]: audit 2022-04-22T19:50:33.064127+0000 mon.a (mon.0) 7893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:34.435 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:34.046Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:34 smithi121 bash[36902]: audit 2022-04-22T19:50:33.418419+0000 mon.a (mon.0) 7894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:34 smithi115 bash[45495]: audit 2022-04-22T19:50:33.418419+0000 mon.a (mon.0) 7894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:34 smithi115 bash[46934]: audit 2022-04-22T19:50:33.418419+0000 mon.a (mon.0) 7894 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:50:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:35 smithi121 bash[36902]: cluster 2022-04-22T19:50:34.281455+0000 mgr.y (mgr.34107) 4958 : cluster [DBG] pgmap v6277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:35 smithi121 bash[36902]: audit 2022-04-22T19:50:34.795701+0000 mon.a (mon.0) 7895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:35 smithi115 bash[45495]: cluster 2022-04-22T19:50:34.281455+0000 mgr.y (mgr.34107) 4958 : cluster [DBG] pgmap v6277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:35.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:35 smithi115 bash[45495]: audit 2022-04-22T19:50:34.795701+0000 mon.a (mon.0) 7895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:35.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:35 smithi115 bash[46934]: cluster 2022-04-22T19:50:34.281455+0000 mgr.y (mgr.34107) 4958 : cluster [DBG] pgmap v6277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:35.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:35 smithi115 bash[46934]: audit 2022-04-22T19:50:34.795701+0000 mon.a (mon.0) 7895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:36 smithi121 bash[36902]: audit 2022-04-22T19:50:36.191055+0000 mon.a (mon.0) 7896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:36 smithi115 bash[45495]: audit 2022-04-22T19:50:36.191055+0000 mon.a (mon.0) 7896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:36 smithi115 bash[46934]: audit 2022-04-22T19:50:36.191055+0000 mon.a (mon.0) 7896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:37 smithi121 bash[36902]: cluster 2022-04-22T19:50:36.282564+0000 mgr.y (mgr.34107) 4959 : cluster [DBG] pgmap v6278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:37 smithi115 bash[45495]: cluster 2022-04-22T19:50:36.282564+0000 mgr.y (mgr.34107) 4959 : cluster [DBG] pgmap v6278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:37 smithi115 bash[46934]: cluster 2022-04-22T19:50:36.282564+0000 mgr.y (mgr.34107) 4959 : cluster [DBG] pgmap v6278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:38 smithi121 bash[36902]: audit 2022-04-22T19:50:37.653967+0000 mon.a (mon.0) 7897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:38 smithi115 bash[45495]: audit 2022-04-22T19:50:37.653967+0000 mon.a (mon.0) 7897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:38 smithi115 bash[46934]: audit 2022-04-22T19:50:37.653967+0000 mon.a (mon.0) 7897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:39 smithi121 bash[36902]: cluster 2022-04-22T19:50:38.283657+0000 mgr.y (mgr.34107) 4960 : cluster [DBG] pgmap v6279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:39 smithi115 bash[45495]: cluster 2022-04-22T19:50:38.283657+0000 mgr.y (mgr.34107) 4960 : cluster [DBG] pgmap v6279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:39 smithi115 bash[46934]: cluster 2022-04-22T19:50:38.283657+0000 mgr.y (mgr.34107) 4960 : cluster [DBG] pgmap v6279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:40 smithi121 bash[36902]: audit 2022-04-22T19:50:39.799510+0000 mon.a (mon.0) 7898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:40 smithi115 bash[45495]: audit 2022-04-22T19:50:39.799510+0000 mon.a (mon.0) 7898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:40 smithi115 bash[46934]: audit 2022-04-22T19:50:39.799510+0000 mon.a (mon.0) 7898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:41 smithi121 bash[36902]: cluster 2022-04-22T19:50:40.284613+0000 mgr.y (mgr.34107) 4961 : cluster [DBG] pgmap v6280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:41 smithi121 bash[36902]: audit 2022-04-22T19:50:41.193901+0000 mon.a (mon.0) 7899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:41 smithi115 bash[46934]: cluster 2022-04-22T19:50:40.284613+0000 mgr.y (mgr.34107) 4961 : cluster [DBG] pgmap v6280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:41 smithi115 bash[46934]: audit 2022-04-22T19:50:41.193901+0000 mon.a (mon.0) 7899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:41 smithi115 bash[45495]: cluster 2022-04-22T19:50:40.284613+0000 mgr.y (mgr.34107) 4961 : cluster [DBG] pgmap v6280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:41 smithi115 bash[45495]: audit 2022-04-22T19:50:41.193901+0000 mon.a (mon.0) 7899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:42.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:42.173Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:43 smithi121 bash[36902]: cluster 2022-04-22T19:50:42.285727+0000 mgr.y (mgr.34107) 4962 : cluster [DBG] pgmap v6281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:43 smithi121 bash[36902]: audit 2022-04-22T19:50:42.657100+0000 mon.a (mon.0) 7900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:43 smithi115 bash[45495]: cluster 2022-04-22T19:50:42.285727+0000 mgr.y (mgr.34107) 4962 : cluster [DBG] pgmap v6281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:43 smithi115 bash[45495]: audit 2022-04-22T19:50:42.657100+0000 mon.a (mon.0) 7900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:43 smithi115 bash[46934]: cluster 2022-04-22T19:50:42.285727+0000 mgr.y (mgr.34107) 4962 : cluster [DBG] pgmap v6281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:43 smithi115 bash[46934]: audit 2022-04-22T19:50:42.657100+0000 mon.a (mon.0) 7900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:44.049Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:45 smithi121 bash[36902]: cluster 2022-04-22T19:50:44.286812+0000 mgr.y (mgr.34107) 4963 : cluster [DBG] pgmap v6282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:45 smithi121 bash[36902]: audit 2022-04-22T19:50:44.803133+0000 mon.a (mon.0) 7901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:45 smithi115 bash[45495]: cluster 2022-04-22T19:50:44.286812+0000 mgr.y (mgr.34107) 4963 : cluster [DBG] pgmap v6282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:45 smithi115 bash[45495]: audit 2022-04-22T19:50:44.803133+0000 mon.a (mon.0) 7901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:45 smithi115 bash[46934]: cluster 2022-04-22T19:50:44.286812+0000 mgr.y (mgr.34107) 4963 : cluster [DBG] pgmap v6282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:45 smithi115 bash[46934]: audit 2022-04-22T19:50:44.803133+0000 mon.a (mon.0) 7901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:46 smithi121 bash[36902]: audit 2022-04-22T19:50:46.196826+0000 mon.a (mon.0) 7902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:46.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:46 smithi115 bash[45495]: audit 2022-04-22T19:50:46.196826+0000 mon.a (mon.0) 7902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:46 smithi115 bash[46934]: audit 2022-04-22T19:50:46.196826+0000 mon.a (mon.0) 7902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:47 smithi121 bash[36902]: cluster 2022-04-22T19:50:46.287902+0000 mgr.y (mgr.34107) 4964 : cluster [DBG] pgmap v6283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:47 smithi115 bash[45495]: cluster 2022-04-22T19:50:46.287902+0000 mgr.y (mgr.34107) 4964 : cluster [DBG] pgmap v6283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:47 smithi115 bash[46934]: cluster 2022-04-22T19:50:46.287902+0000 mgr.y (mgr.34107) 4964 : cluster [DBG] pgmap v6283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:48 smithi121 bash[36902]: audit 2022-04-22T19:50:47.660092+0000 mon.a (mon.0) 7903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:48 smithi115 bash[45495]: audit 2022-04-22T19:50:47.660092+0000 mon.a (mon.0) 7903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:48.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:48 smithi115 bash[46934]: audit 2022-04-22T19:50:47.660092+0000 mon.a (mon.0) 7903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:49 smithi121 bash[36902]: cluster 2022-04-22T19:50:48.289042+0000 mgr.y (mgr.34107) 4965 : cluster [DBG] pgmap v6284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:49 smithi115 bash[45495]: cluster 2022-04-22T19:50:48.289042+0000 mgr.y (mgr.34107) 4965 : cluster [DBG] pgmap v6284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:49 smithi115 bash[46934]: cluster 2022-04-22T19:50:48.289042+0000 mgr.y (mgr.34107) 4965 : cluster [DBG] pgmap v6284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:50 smithi121 bash[36902]: audit 2022-04-22T19:50:49.806835+0000 mon.a (mon.0) 7904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:50 smithi115 bash[45495]: audit 2022-04-22T19:50:49.806835+0000 mon.a (mon.0) 7904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:50 smithi115 bash[46934]: audit 2022-04-22T19:50:49.806835+0000 mon.a (mon.0) 7904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:51 smithi121 bash[36902]: cluster 2022-04-22T19:50:50.290105+0000 mgr.y (mgr.34107) 4966 : cluster [DBG] pgmap v6285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:51 smithi121 bash[36902]: audit 2022-04-22T19:50:51.200032+0000 mon.a (mon.0) 7905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:51 smithi115 bash[45495]: cluster 2022-04-22T19:50:50.290105+0000 mgr.y (mgr.34107) 4966 : cluster [DBG] pgmap v6285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:51.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:51 smithi115 bash[45495]: audit 2022-04-22T19:50:51.200032+0000 mon.a (mon.0) 7905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:51 smithi115 bash[46934]: cluster 2022-04-22T19:50:50.290105+0000 mgr.y (mgr.34107) 4966 : cluster [DBG] pgmap v6285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:51.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:51 smithi115 bash[46934]: audit 2022-04-22T19:50:51.200032+0000 mon.a (mon.0) 7905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:51.821 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:50:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:50:52.310 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 19s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 19s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 19s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 19s ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:50:52.311 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 19s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 19s ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 19s ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:50:52.312 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 19s ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:50:52.313 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 19s ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:50:52.313 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 19s ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:50:52.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:52.173Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:52.570 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:50:52.570 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:52.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:50:52.572 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:50:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:53 smithi121 bash[36902]: audit 2022-04-22T19:50:51.797802+0000 mgr.y (mgr.34107) 4967 : audit [DBG] from='client.40572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:53 smithi121 bash[36902]: audit 2022-04-22T19:50:52.138827+0000 mgr.y (mgr.34107) 4968 : audit [DBG] from='client.40578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:53 smithi121 bash[36902]: cluster 2022-04-22T19:50:52.291053+0000 mgr.y (mgr.34107) 4969 : cluster [DBG] pgmap v6286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:53 smithi121 bash[36902]: audit 2022-04-22T19:50:52.560977+0000 mon.c (mon.1) 214 : audit [DBG] from='client.? 172.21.15.115:0/1674751457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:53 smithi121 bash[36902]: audit 2022-04-22T19:50:52.663105+0000 mon.a (mon.0) 7906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[45495]: audit 2022-04-22T19:50:51.797802+0000 mgr.y (mgr.34107) 4967 : audit [DBG] from='client.40572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[45495]: audit 2022-04-22T19:50:52.138827+0000 mgr.y (mgr.34107) 4968 : audit [DBG] from='client.40578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[45495]: cluster 2022-04-22T19:50:52.291053+0000 mgr.y (mgr.34107) 4969 : cluster [DBG] pgmap v6286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[45495]: audit 2022-04-22T19:50:52.560977+0000 mon.c (mon.1) 214 : audit [DBG] from='client.? 172.21.15.115:0/1674751457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[45495]: audit 2022-04-22T19:50:52.663105+0000 mon.a (mon.0) 7906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[46934]: audit 2022-04-22T19:50:51.797802+0000 mgr.y (mgr.34107) 4967 : audit [DBG] from='client.40572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[46934]: audit 2022-04-22T19:50:52.138827+0000 mgr.y (mgr.34107) 4968 : audit [DBG] from='client.40578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:50:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[46934]: cluster 2022-04-22T19:50:52.291053+0000 mgr.y (mgr.34107) 4969 : cluster [DBG] pgmap v6286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[46934]: audit 2022-04-22T19:50:52.560977+0000 mon.c (mon.1) 214 : audit [DBG] from='client.? 172.21.15.115:0/1674751457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:50:53.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:53 smithi115 bash[46934]: audit 2022-04-22T19:50:52.663105+0000 mon.a (mon.0) 7906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:50:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:50:54.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:50:55.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:55 smithi115 bash[45495]: cluster 2022-04-22T19:50:54.292184+0000 mgr.y (mgr.34107) 4970 : cluster [DBG] pgmap v6287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:55 smithi115 bash[45495]: audit 2022-04-22T19:50:54.810348+0000 mon.a (mon.0) 7907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:55 smithi115 bash[46934]: cluster 2022-04-22T19:50:54.292184+0000 mgr.y (mgr.34107) 4970 : cluster [DBG] pgmap v6287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:55.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:55 smithi115 bash[46934]: audit 2022-04-22T19:50:54.810348+0000 mon.a (mon.0) 7907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:55 smithi121 bash[36902]: cluster 2022-04-22T19:50:54.292184+0000 mgr.y (mgr.34107) 4970 : cluster [DBG] pgmap v6287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:55 smithi121 bash[36902]: audit 2022-04-22T19:50:54.810348+0000 mon.a (mon.0) 7907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:50:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:56 smithi115 bash[45495]: audit 2022-04-22T19:50:56.203409+0000 mon.a (mon.0) 7908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:56 smithi115 bash[46934]: audit 2022-04-22T19:50:56.203409+0000 mon.a (mon.0) 7908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:56 smithi121 bash[36902]: audit 2022-04-22T19:50:56.203409+0000 mon.a (mon.0) 7908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:50:57.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:57 smithi115 bash[45495]: cluster 2022-04-22T19:50:56.293265+0000 mgr.y (mgr.34107) 4971 : cluster [DBG] pgmap v6288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:57.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:57 smithi115 bash[46934]: cluster 2022-04-22T19:50:56.293265+0000 mgr.y (mgr.34107) 4971 : cluster [DBG] pgmap v6288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:57 smithi121 bash[36902]: cluster 2022-04-22T19:50:56.293265+0000 mgr.y (mgr.34107) 4971 : cluster [DBG] pgmap v6288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:58 smithi115 bash[46934]: audit 2022-04-22T19:50:57.666234+0000 mon.a (mon.0) 7909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:58 smithi115 bash[45495]: audit 2022-04-22T19:50:57.666234+0000 mon.a (mon.0) 7909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:58 smithi121 bash[36902]: audit 2022-04-22T19:50:57.666234+0000 mon.a (mon.0) 7909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:50:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:50:59 smithi115 bash[45495]: cluster 2022-04-22T19:50:58.294329+0000 mgr.y (mgr.34107) 4972 : cluster [DBG] pgmap v6289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:50:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:50:59 smithi115 bash[46934]: cluster 2022-04-22T19:50:58.294329+0000 mgr.y (mgr.34107) 4972 : cluster [DBG] pgmap v6289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:50:59 smithi121 bash[36902]: cluster 2022-04-22T19:50:58.294329+0000 mgr.y (mgr.34107) 4972 : cluster [DBG] pgmap v6289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:00 smithi115 bash[45495]: audit 2022-04-22T19:50:59.813899+0000 mon.a (mon.0) 7910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:00 smithi115 bash[46934]: audit 2022-04-22T19:50:59.813899+0000 mon.a (mon.0) 7910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:00 smithi121 bash[36902]: audit 2022-04-22T19:50:59.813899+0000 mon.a (mon.0) 7910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:01 smithi121 bash[36902]: cluster 2022-04-22T19:51:00.295439+0000 mgr.y (mgr.34107) 4973 : cluster [DBG] pgmap v6290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:01 smithi121 bash[36902]: audit 2022-04-22T19:51:01.206617+0000 mon.a (mon.0) 7911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:01 smithi115 bash[45495]: cluster 2022-04-22T19:51:00.295439+0000 mgr.y (mgr.34107) 4973 : cluster [DBG] pgmap v6290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:01 smithi115 bash[45495]: audit 2022-04-22T19:51:01.206617+0000 mon.a (mon.0) 7911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:01 smithi115 bash[46934]: cluster 2022-04-22T19:51:00.295439+0000 mgr.y (mgr.34107) 4973 : cluster [DBG] pgmap v6290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:01 smithi115 bash[46934]: audit 2022-04-22T19:51:01.206617+0000 mon.a (mon.0) 7911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:02.175Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:03 smithi121 bash[36902]: cluster 2022-04-22T19:51:02.296605+0000 mgr.y (mgr.34107) 4974 : cluster [DBG] pgmap v6291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:03 smithi121 bash[36902]: audit 2022-04-22T19:51:02.669126+0000 mon.a (mon.0) 7912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:03 smithi115 bash[45495]: cluster 2022-04-22T19:51:02.296605+0000 mgr.y (mgr.34107) 4974 : cluster [DBG] pgmap v6291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:03 smithi115 bash[45495]: audit 2022-04-22T19:51:02.669126+0000 mon.a (mon.0) 7912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:03 smithi115 bash[46934]: cluster 2022-04-22T19:51:02.296605+0000 mgr.y (mgr.34107) 4974 : cluster [DBG] pgmap v6291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:03 smithi115 bash[46934]: audit 2022-04-22T19:51:02.669126+0000 mon.a (mon.0) 7912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:04.050Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:05 smithi121 bash[36902]: cluster 2022-04-22T19:51:04.297710+0000 mgr.y (mgr.34107) 4975 : cluster [DBG] pgmap v6292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:05 smithi121 bash[36902]: audit 2022-04-22T19:51:04.817705+0000 mon.a (mon.0) 7913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:05 smithi115 bash[45495]: cluster 2022-04-22T19:51:04.297710+0000 mgr.y (mgr.34107) 4975 : cluster [DBG] pgmap v6292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:05 smithi115 bash[45495]: audit 2022-04-22T19:51:04.817705+0000 mon.a (mon.0) 7913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:05 smithi115 bash[46934]: cluster 2022-04-22T19:51:04.297710+0000 mgr.y (mgr.34107) 4975 : cluster [DBG] pgmap v6292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:05 smithi115 bash[46934]: audit 2022-04-22T19:51:04.817705+0000 mon.a (mon.0) 7913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:06 smithi121 bash[36902]: audit 2022-04-22T19:51:06.209901+0000 mon.a (mon.0) 7914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:07.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:06 smithi115 bash[45495]: audit 2022-04-22T19:51:06.209901+0000 mon.a (mon.0) 7914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:06 smithi115 bash[46934]: audit 2022-04-22T19:51:06.209901+0000 mon.a (mon.0) 7914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:07 smithi121 bash[36902]: cluster 2022-04-22T19:51:06.298732+0000 mgr.y (mgr.34107) 4976 : cluster [DBG] pgmap v6293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:07 smithi115 bash[46934]: cluster 2022-04-22T19:51:06.298732+0000 mgr.y (mgr.34107) 4976 : cluster [DBG] pgmap v6293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:07 smithi115 bash[45495]: cluster 2022-04-22T19:51:06.298732+0000 mgr.y (mgr.34107) 4976 : cluster [DBG] pgmap v6293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:08 smithi121 bash[36902]: audit 2022-04-22T19:51:07.672157+0000 mon.a (mon.0) 7915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:08 smithi115 bash[45495]: audit 2022-04-22T19:51:07.672157+0000 mon.a (mon.0) 7915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:08 smithi115 bash[46934]: audit 2022-04-22T19:51:07.672157+0000 mon.a (mon.0) 7915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:09 smithi121 bash[36902]: cluster 2022-04-22T19:51:08.299904+0000 mgr.y (mgr.34107) 4977 : cluster [DBG] pgmap v6294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:09 smithi115 bash[45495]: cluster 2022-04-22T19:51:08.299904+0000 mgr.y (mgr.34107) 4977 : cluster [DBG] pgmap v6294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:09 smithi115 bash[46934]: cluster 2022-04-22T19:51:08.299904+0000 mgr.y (mgr.34107) 4977 : cluster [DBG] pgmap v6294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:10 smithi121 bash[36902]: audit 2022-04-22T19:51:09.821341+0000 mon.a (mon.0) 7916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:10 smithi115 bash[45495]: audit 2022-04-22T19:51:09.821341+0000 mon.a (mon.0) 7916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:10 smithi115 bash[46934]: audit 2022-04-22T19:51:09.821341+0000 mon.a (mon.0) 7916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:11 smithi121 bash[36902]: cluster 2022-04-22T19:51:10.300986+0000 mgr.y (mgr.34107) 4978 : cluster [DBG] pgmap v6295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:11 smithi121 bash[36902]: audit 2022-04-22T19:51:11.212920+0000 mon.a (mon.0) 7917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:11 smithi115 bash[45495]: cluster 2022-04-22T19:51:10.300986+0000 mgr.y (mgr.34107) 4978 : cluster [DBG] pgmap v6295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:12.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:11 smithi115 bash[45495]: audit 2022-04-22T19:51:11.212920+0000 mon.a (mon.0) 7917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:12.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:11 smithi115 bash[46934]: cluster 2022-04-22T19:51:10.300986+0000 mgr.y (mgr.34107) 4978 : cluster [DBG] pgmap v6295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:12.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:11 smithi115 bash[46934]: audit 2022-04-22T19:51:11.212920+0000 mon.a (mon.0) 7917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:12.176Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:13 smithi121 bash[36902]: cluster 2022-04-22T19:51:12.302041+0000 mgr.y (mgr.34107) 4979 : cluster [DBG] pgmap v6296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:14.027 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:13 smithi121 bash[36902]: audit 2022-04-22T19:51:12.675355+0000 mon.a (mon.0) 7918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:13 smithi115 bash[45495]: cluster 2022-04-22T19:51:12.302041+0000 mgr.y (mgr.34107) 4979 : cluster [DBG] pgmap v6296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:13 smithi115 bash[45495]: audit 2022-04-22T19:51:12.675355+0000 mon.a (mon.0) 7918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:13 smithi115 bash[46934]: cluster 2022-04-22T19:51:12.302041+0000 mgr.y (mgr.34107) 4979 : cluster [DBG] pgmap v6296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:14.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:13 smithi115 bash[46934]: audit 2022-04-22T19:51:12.675355+0000 mon.a (mon.0) 7918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:14.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:14.050Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:15 smithi121 bash[36902]: cluster 2022-04-22T19:51:14.303100+0000 mgr.y (mgr.34107) 4980 : cluster [DBG] pgmap v6297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:15 smithi121 bash[36902]: audit 2022-04-22T19:51:14.825177+0000 mon.a (mon.0) 7919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:15 smithi121 bash[36902]: audit 2022-04-22T19:51:14.942395+0000 mon.a (mon.0) 7920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:51:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:15 smithi121 bash[36902]: audit 2022-04-22T19:51:14.942942+0000 mon.a (mon.0) 7921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:51:16.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[46934]: cluster 2022-04-22T19:51:14.303100+0000 mgr.y (mgr.34107) 4980 : cluster [DBG] pgmap v6297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[46934]: audit 2022-04-22T19:51:14.825177+0000 mon.a (mon.0) 7919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[46934]: audit 2022-04-22T19:51:14.942395+0000 mon.a (mon.0) 7920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[46934]: audit 2022-04-22T19:51:14.942942+0000 mon.a (mon.0) 7921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[45495]: cluster 2022-04-22T19:51:14.303100+0000 mgr.y (mgr.34107) 4980 : cluster [DBG] pgmap v6297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[45495]: audit 2022-04-22T19:51:14.825177+0000 mon.a (mon.0) 7919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[45495]: audit 2022-04-22T19:51:14.942395+0000 mon.a (mon.0) 7920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:51:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:15 smithi115 bash[45495]: audit 2022-04-22T19:51:14.942942+0000 mon.a (mon.0) 7921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:51:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:16 smithi121 bash[36902]: audit 2022-04-22T19:51:16.216364+0000 mon.a (mon.0) 7922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:16 smithi115 bash[46934]: audit 2022-04-22T19:51:16.216364+0000 mon.a (mon.0) 7922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:16 smithi115 bash[45495]: audit 2022-04-22T19:51:16.216364+0000 mon.a (mon.0) 7922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:17 smithi121 bash[36902]: cluster 2022-04-22T19:51:16.304172+0000 mgr.y (mgr.34107) 4981 : cluster [DBG] pgmap v6298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:17 smithi115 bash[45495]: cluster 2022-04-22T19:51:16.304172+0000 mgr.y (mgr.34107) 4981 : cluster [DBG] pgmap v6298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:17 smithi115 bash[46934]: cluster 2022-04-22T19:51:16.304172+0000 mgr.y (mgr.34107) 4981 : cluster [DBG] pgmap v6298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:18 smithi121 bash[36902]: audit 2022-04-22T19:51:17.678638+0000 mon.a (mon.0) 7923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:18 smithi115 bash[45495]: audit 2022-04-22T19:51:17.678638+0000 mon.a (mon.0) 7923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:18 smithi115 bash[46934]: audit 2022-04-22T19:51:17.678638+0000 mon.a (mon.0) 7923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:19 smithi121 bash[36902]: cluster 2022-04-22T19:51:18.305312+0000 mgr.y (mgr.34107) 4982 : cluster [DBG] pgmap v6299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:19 smithi115 bash[45495]: cluster 2022-04-22T19:51:18.305312+0000 mgr.y (mgr.34107) 4982 : cluster [DBG] pgmap v6299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:19 smithi115 bash[46934]: cluster 2022-04-22T19:51:18.305312+0000 mgr.y (mgr.34107) 4982 : cluster [DBG] pgmap v6299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:20 smithi121 bash[36902]: audit 2022-04-22T19:51:19.828662+0000 mon.a (mon.0) 7924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:20 smithi115 bash[45495]: audit 2022-04-22T19:51:19.828662+0000 mon.a (mon.0) 7924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:20 smithi115 bash[46934]: audit 2022-04-22T19:51:19.828662+0000 mon.a (mon.0) 7924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:21 smithi121 bash[36902]: cluster 2022-04-22T19:51:20.306287+0000 mgr.y (mgr.34107) 4983 : cluster [DBG] pgmap v6300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:21 smithi121 bash[36902]: audit 2022-04-22T19:51:21.219375+0000 mon.a (mon.0) 7925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:21 smithi115 bash[45495]: cluster 2022-04-22T19:51:20.306287+0000 mgr.y (mgr.34107) 4983 : cluster [DBG] pgmap v6300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:21 smithi115 bash[45495]: audit 2022-04-22T19:51:21.219375+0000 mon.a (mon.0) 7925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:21 smithi115 bash[46934]: cluster 2022-04-22T19:51:20.306287+0000 mgr.y (mgr.34107) 4983 : cluster [DBG] pgmap v6300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:21 smithi115 bash[46934]: audit 2022-04-22T19:51:21.219375+0000 mon.a (mon.0) 7925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:22.176Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:22.938 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 10m ago 2h 15.2M - ba2b418f427c 9255ba08f72f 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 50s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 10m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 50s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 50s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 10m ago 2h 481M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:51:23.283 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 10m ago 2h 292M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 50s ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 10m ago 2h 186M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 10m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 50s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 10m ago 2h 702M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 10m ago 2h 781M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 10m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 10m ago 2h 484M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 50s ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:51:23.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 50s ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:51:23.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 50s ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:51:23.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 50s ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:51:23.285 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 50s ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:51:23.690 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:51:23.690 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:51:23.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:51:23.692 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:51:23.693 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:51:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:23 smithi121 bash[36902]: cluster 2022-04-22T19:51:22.307371+0000 mgr.y (mgr.34107) 4984 : cluster [DBG] pgmap v6301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:23 smithi121 bash[36902]: audit 2022-04-22T19:51:22.681617+0000 mon.a (mon.0) 7926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:23 smithi115 bash[45495]: cluster 2022-04-22T19:51:22.307371+0000 mgr.y (mgr.34107) 4984 : cluster [DBG] pgmap v6301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:23 smithi115 bash[45495]: audit 2022-04-22T19:51:22.681617+0000 mon.a (mon.0) 7926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:23 smithi115 bash[46934]: cluster 2022-04-22T19:51:22.307371+0000 mgr.y (mgr.34107) 4984 : cluster [DBG] pgmap v6301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:23 smithi115 bash[46934]: audit 2022-04-22T19:51:22.681617+0000 mon.a (mon.0) 7926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:24.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:24.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:24 smithi121 bash[36902]: audit 2022-04-22T19:51:22.914678+0000 mgr.y (mgr.34107) 4985 : audit [DBG] from='client.50524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:24 smithi121 bash[36902]: audit 2022-04-22T19:51:23.253546+0000 mgr.y (mgr.34107) 4986 : audit [DBG] from='client.50530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:24 smithi121 bash[36902]: audit 2022-04-22T19:51:23.681185+0000 mon.a (mon.0) 7927 : audit [DBG] from='client.? 172.21.15.115:0/3163042122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[45495]: audit 2022-04-22T19:51:22.914678+0000 mgr.y (mgr.34107) 4985 : audit [DBG] from='client.50524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[45495]: audit 2022-04-22T19:51:23.253546+0000 mgr.y (mgr.34107) 4986 : audit [DBG] from='client.50530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[45495]: audit 2022-04-22T19:51:23.681185+0000 mon.a (mon.0) 7927 : audit [DBG] from='client.? 172.21.15.115:0/3163042122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[46934]: audit 2022-04-22T19:51:22.914678+0000 mgr.y (mgr.34107) 4985 : audit [DBG] from='client.50524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[46934]: audit 2022-04-22T19:51:23.253546+0000 mgr.y (mgr.34107) 4986 : audit [DBG] from='client.50530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:25.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:24 smithi115 bash[46934]: audit 2022-04-22T19:51:23.681185+0000 mon.a (mon.0) 7927 : audit [DBG] from='client.? 172.21.15.115:0/3163042122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:25 smithi121 bash[36902]: cluster 2022-04-22T19:51:24.308409+0000 mgr.y (mgr.34107) 4987 : cluster [DBG] pgmap v6302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:25 smithi121 bash[36902]: audit 2022-04-22T19:51:24.832456+0000 mon.a (mon.0) 7928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:25 smithi115 bash[45495]: cluster 2022-04-22T19:51:24.308409+0000 mgr.y (mgr.34107) 4987 : cluster [DBG] pgmap v6302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:25 smithi115 bash[45495]: audit 2022-04-22T19:51:24.832456+0000 mon.a (mon.0) 7928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:25 smithi115 bash[46934]: cluster 2022-04-22T19:51:24.308409+0000 mgr.y (mgr.34107) 4987 : cluster [DBG] pgmap v6302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:25 smithi115 bash[46934]: audit 2022-04-22T19:51:24.832456+0000 mon.a (mon.0) 7928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:26 smithi121 bash[36902]: audit 2022-04-22T19:51:26.222353+0000 mon.a (mon.0) 7929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:26 smithi115 bash[45495]: audit 2022-04-22T19:51:26.222353+0000 mon.a (mon.0) 7929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:27.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:26 smithi115 bash[46934]: audit 2022-04-22T19:51:26.222353+0000 mon.a (mon.0) 7929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:27 smithi121 bash[36902]: cluster 2022-04-22T19:51:26.309639+0000 mgr.y (mgr.34107) 4988 : cluster [DBG] pgmap v6303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:27 smithi115 bash[45495]: cluster 2022-04-22T19:51:26.309639+0000 mgr.y (mgr.34107) 4988 : cluster [DBG] pgmap v6303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:27 smithi115 bash[46934]: cluster 2022-04-22T19:51:26.309639+0000 mgr.y (mgr.34107) 4988 : cluster [DBG] pgmap v6303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:28 smithi121 bash[36902]: audit 2022-04-22T19:51:27.683870+0000 mon.a (mon.0) 7930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:28 smithi115 bash[45495]: audit 2022-04-22T19:51:27.683870+0000 mon.a (mon.0) 7930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:28 smithi115 bash[46934]: audit 2022-04-22T19:51:27.683870+0000 mon.a (mon.0) 7930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:29 smithi121 bash[36902]: cluster 2022-04-22T19:51:28.310868+0000 mgr.y (mgr.34107) 4989 : cluster [DBG] pgmap v6304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:29 smithi115 bash[45495]: cluster 2022-04-22T19:51:28.310868+0000 mgr.y (mgr.34107) 4989 : cluster [DBG] pgmap v6304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:30.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:29 smithi115 bash[46934]: cluster 2022-04-22T19:51:28.310868+0000 mgr.y (mgr.34107) 4989 : cluster [DBG] pgmap v6304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:30 smithi121 bash[36902]: audit 2022-04-22T19:51:29.836036+0000 mon.a (mon.0) 7931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:30 smithi115 bash[45495]: audit 2022-04-22T19:51:29.836036+0000 mon.a (mon.0) 7931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:30 smithi115 bash[46934]: audit 2022-04-22T19:51:29.836036+0000 mon.a (mon.0) 7931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:31 smithi121 bash[36902]: cluster 2022-04-22T19:51:30.312084+0000 mgr.y (mgr.34107) 4990 : cluster [DBG] pgmap v6305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:31 smithi121 bash[36902]: audit 2022-04-22T19:51:31.225330+0000 mon.a (mon.0) 7932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:31 smithi115 bash[45495]: cluster 2022-04-22T19:51:30.312084+0000 mgr.y (mgr.34107) 4990 : cluster [DBG] pgmap v6305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:31 smithi115 bash[45495]: audit 2022-04-22T19:51:31.225330+0000 mon.a (mon.0) 7932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:31 smithi115 bash[46934]: cluster 2022-04-22T19:51:30.312084+0000 mgr.y (mgr.34107) 4990 : cluster [DBG] pgmap v6305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:31 smithi115 bash[46934]: audit 2022-04-22T19:51:31.225330+0000 mon.a (mon.0) 7932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:32.176Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:33.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[45495]: cluster 2022-04-22T19:51:32.313251+0000 mgr.y (mgr.34107) 4991 : cluster [DBG] pgmap v6306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:33.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[45495]: audit 2022-04-22T19:51:32.686515+0000 mon.a (mon.0) 7933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[45495]: audit 2022-04-22T19:51:33.426302+0000 mon.a (mon.0) 7934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[45495]: audit 2022-04-22T19:51:33.429289+0000 mon.a (mon.0) 7935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[45495]: audit 2022-04-22T19:51:33.431169+0000 mon.a (mon.0) 7936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[46934]: cluster 2022-04-22T19:51:32.313251+0000 mgr.y (mgr.34107) 4991 : cluster [DBG] pgmap v6306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[46934]: audit 2022-04-22T19:51:32.686515+0000 mon.a (mon.0) 7933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[46934]: audit 2022-04-22T19:51:33.426302+0000 mon.a (mon.0) 7934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[46934]: audit 2022-04-22T19:51:33.429289+0000 mon.a (mon.0) 7935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:51:33.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:33 smithi115 bash[46934]: audit 2022-04-22T19:51:33.431169+0000 mon.a (mon.0) 7936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:51:33.759 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:33 smithi121 bash[36902]: cluster 2022-04-22T19:51:32.313251+0000 mgr.y (mgr.34107) 4991 : cluster [DBG] pgmap v6306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:33.759 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:33 smithi121 bash[36902]: audit 2022-04-22T19:51:32.686515+0000 mon.a (mon.0) 7933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:33.759 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:33 smithi121 bash[36902]: audit 2022-04-22T19:51:33.426302+0000 mon.a (mon.0) 7934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:51:33.759 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:33 smithi121 bash[36902]: audit 2022-04-22T19:51:33.429289+0000 mon.a (mon.0) 7935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:51:33.759 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:33 smithi121 bash[36902]: audit 2022-04-22T19:51:33.431169+0000 mon.a (mon.0) 7936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:51:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:34.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:35.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[46934]: audit 2022-04-22T19:51:33.770104+0000 mon.a (mon.0) 7937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[46934]: audit 2022-04-22T19:51:33.775123+0000 mon.a (mon.0) 7938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[46934]: cluster 2022-04-22T19:51:34.314442+0000 mgr.y (mgr.34107) 4992 : cluster [DBG] pgmap v6307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[45495]: audit 2022-04-22T19:51:33.770104+0000 mon.a (mon.0) 7937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[45495]: audit 2022-04-22T19:51:33.775123+0000 mon.a (mon.0) 7938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:34 smithi115 bash[45495]: cluster 2022-04-22T19:51:34.314442+0000 mgr.y (mgr.34107) 4992 : cluster [DBG] pgmap v6307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:34 smithi121 bash[36902]: audit 2022-04-22T19:51:33.770104+0000 mon.a (mon.0) 7937 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:34 smithi121 bash[36902]: audit 2022-04-22T19:51:33.775123+0000 mon.a (mon.0) 7938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:34 smithi121 bash[36902]: cluster 2022-04-22T19:51:34.314442+0000 mgr.y (mgr.34107) 4992 : cluster [DBG] pgmap v6307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:35 smithi115 bash[45495]: audit 2022-04-22T19:51:34.839514+0000 mon.a (mon.0) 7939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:35 smithi115 bash[46934]: audit 2022-04-22T19:51:34.839514+0000 mon.a (mon.0) 7939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:35 smithi121 bash[36902]: audit 2022-04-22T19:51:34.839514+0000 mon.a (mon.0) 7939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:36 smithi115 bash[45495]: audit 2022-04-22T19:51:36.228514+0000 mon.a (mon.0) 7940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:36 smithi115 bash[45495]: cluster 2022-04-22T19:51:36.315591+0000 mgr.y (mgr.34107) 4993 : cluster [DBG] pgmap v6308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:36 smithi115 bash[46934]: audit 2022-04-22T19:51:36.228514+0000 mon.a (mon.0) 7940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:37.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:36 smithi115 bash[46934]: cluster 2022-04-22T19:51:36.315591+0000 mgr.y (mgr.34107) 4993 : cluster [DBG] pgmap v6308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:36 smithi121 bash[36902]: audit 2022-04-22T19:51:36.228514+0000 mon.a (mon.0) 7940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:37.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:36 smithi121 bash[36902]: cluster 2022-04-22T19:51:36.315591+0000 mgr.y (mgr.34107) 4993 : cluster [DBG] pgmap v6308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:37 smithi115 bash[45495]: audit 2022-04-22T19:51:37.689506+0000 mon.a (mon.0) 7941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:37 smithi115 bash[46934]: audit 2022-04-22T19:51:37.689506+0000 mon.a (mon.0) 7941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:37 smithi121 bash[36902]: audit 2022-04-22T19:51:37.689506+0000 mon.a (mon.0) 7941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:38 smithi121 bash[36902]: audit 2022-04-22T19:51:37.838284+0000 mon.a (mon.0) 7942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:38 smithi121 bash[36902]: audit 2022-04-22T19:51:38.190832+0000 mon.a (mon.0) 7943 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:38 smithi121 bash[36902]: cluster 2022-04-22T19:51:38.316498+0000 mgr.y (mgr.34107) 4994 : cluster [DBG] pgmap v6309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[45495]: audit 2022-04-22T19:51:37.838284+0000 mon.a (mon.0) 7942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[45495]: audit 2022-04-22T19:51:38.190832+0000 mon.a (mon.0) 7943 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[45495]: cluster 2022-04-22T19:51:38.316498+0000 mgr.y (mgr.34107) 4994 : cluster [DBG] pgmap v6309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[46934]: audit 2022-04-22T19:51:37.838284+0000 mon.a (mon.0) 7942 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[46934]: audit 2022-04-22T19:51:38.190832+0000 mon.a (mon.0) 7943 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:51:39.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:38 smithi115 bash[46934]: cluster 2022-04-22T19:51:38.316498+0000 mgr.y (mgr.34107) 4994 : cluster [DBG] pgmap v6309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:40 smithi121 bash[36902]: audit 2022-04-22T19:51:39.843286+0000 mon.a (mon.0) 7944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:40 smithi115 bash[45495]: audit 2022-04-22T19:51:39.843286+0000 mon.a (mon.0) 7944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:40 smithi115 bash[46934]: audit 2022-04-22T19:51:39.843286+0000 mon.a (mon.0) 7944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:41 smithi121 bash[36902]: cluster 2022-04-22T19:51:40.317785+0000 mgr.y (mgr.34107) 4995 : cluster [DBG] pgmap v6310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:41 smithi121 bash[36902]: audit 2022-04-22T19:51:41.231458+0000 mon.a (mon.0) 7945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:41 smithi115 bash[45495]: cluster 2022-04-22T19:51:40.317785+0000 mgr.y (mgr.34107) 4995 : cluster [DBG] pgmap v6310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:41 smithi115 bash[45495]: audit 2022-04-22T19:51:41.231458+0000 mon.a (mon.0) 7945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:41 smithi115 bash[46934]: cluster 2022-04-22T19:51:40.317785+0000 mgr.y (mgr.34107) 4995 : cluster [DBG] pgmap v6310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:41 smithi115 bash[46934]: audit 2022-04-22T19:51:41.231458+0000 mon.a (mon.0) 7945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:42.177Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:43 smithi121 bash[36902]: cluster 2022-04-22T19:51:42.318973+0000 mgr.y (mgr.34107) 4996 : cluster [DBG] pgmap v6311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:43 smithi121 bash[36902]: audit 2022-04-22T19:51:42.692531+0000 mon.a (mon.0) 7946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:43 smithi115 bash[45495]: cluster 2022-04-22T19:51:42.318973+0000 mgr.y (mgr.34107) 4996 : cluster [DBG] pgmap v6311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:43.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:43 smithi115 bash[45495]: audit 2022-04-22T19:51:42.692531+0000 mon.a (mon.0) 7946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:43.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:43 smithi115 bash[46934]: cluster 2022-04-22T19:51:42.318973+0000 mgr.y (mgr.34107) 4996 : cluster [DBG] pgmap v6311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:43.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:43 smithi115 bash[46934]: audit 2022-04-22T19:51:42.692531+0000 mon.a (mon.0) 7946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:44.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:45 smithi121 bash[36902]: cluster 2022-04-22T19:51:44.320070+0000 mgr.y (mgr.34107) 4997 : cluster [DBG] pgmap v6312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:45 smithi121 bash[36902]: audit 2022-04-22T19:51:44.846868+0000 mon.a (mon.0) 7947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:45 smithi115 bash[45495]: cluster 2022-04-22T19:51:44.320070+0000 mgr.y (mgr.34107) 4997 : cluster [DBG] pgmap v6312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:45.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:45 smithi115 bash[45495]: audit 2022-04-22T19:51:44.846868+0000 mon.a (mon.0) 7947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:45 smithi115 bash[46934]: cluster 2022-04-22T19:51:44.320070+0000 mgr.y (mgr.34107) 4997 : cluster [DBG] pgmap v6312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:45.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:45 smithi115 bash[46934]: audit 2022-04-22T19:51:44.846868+0000 mon.a (mon.0) 7947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:46 smithi121 bash[36902]: audit 2022-04-22T19:51:46.234480+0000 mon.a (mon.0) 7948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:46.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:46 smithi115 bash[45495]: audit 2022-04-22T19:51:46.234480+0000 mon.a (mon.0) 7948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:46.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:46 smithi115 bash[46934]: audit 2022-04-22T19:51:46.234480+0000 mon.a (mon.0) 7948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:47 smithi121 bash[36902]: cluster 2022-04-22T19:51:46.321137+0000 mgr.y (mgr.34107) 4998 : cluster [DBG] pgmap v6313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:47 smithi115 bash[45495]: cluster 2022-04-22T19:51:46.321137+0000 mgr.y (mgr.34107) 4998 : cluster [DBG] pgmap v6313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:47 smithi115 bash[46934]: cluster 2022-04-22T19:51:46.321137+0000 mgr.y (mgr.34107) 4998 : cluster [DBG] pgmap v6313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:48 smithi121 bash[36902]: audit 2022-04-22T19:51:47.695376+0000 mon.a (mon.0) 7949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:48 smithi115 bash[45495]: audit 2022-04-22T19:51:47.695376+0000 mon.a (mon.0) 7949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:48.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:48 smithi115 bash[46934]: audit 2022-04-22T19:51:47.695376+0000 mon.a (mon.0) 7949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:49 smithi121 bash[36902]: cluster 2022-04-22T19:51:48.322038+0000 mgr.y (mgr.34107) 4999 : cluster [DBG] pgmap v6314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:49 smithi115 bash[45495]: cluster 2022-04-22T19:51:48.322038+0000 mgr.y (mgr.34107) 4999 : cluster [DBG] pgmap v6314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:49 smithi115 bash[46934]: cluster 2022-04-22T19:51:48.322038+0000 mgr.y (mgr.34107) 4999 : cluster [DBG] pgmap v6314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:50 smithi121 bash[36902]: audit 2022-04-22T19:51:49.850479+0000 mon.a (mon.0) 7950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:50 smithi115 bash[45495]: audit 2022-04-22T19:51:49.850479+0000 mon.a (mon.0) 7950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:50 smithi115 bash[46934]: audit 2022-04-22T19:51:49.850479+0000 mon.a (mon.0) 7950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:51 smithi121 bash[36902]: cluster 2022-04-22T19:51:50.323111+0000 mgr.y (mgr.34107) 5000 : cluster [DBG] pgmap v6315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:51 smithi121 bash[36902]: audit 2022-04-22T19:51:51.237625+0000 mon.a (mon.0) 7951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:51.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:51 smithi115 bash[45495]: cluster 2022-04-22T19:51:50.323111+0000 mgr.y (mgr.34107) 5000 : cluster [DBG] pgmap v6315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:51.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:51 smithi115 bash[45495]: audit 2022-04-22T19:51:51.237625+0000 mon.a (mon.0) 7951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:51.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:51 smithi115 bash[46934]: cluster 2022-04-22T19:51:50.323111+0000 mgr.y (mgr.34107) 5000 : cluster [DBG] pgmap v6315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:51.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:51 smithi115 bash[46934]: audit 2022-04-22T19:51:51.237625+0000 mon.a (mon.0) 7951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:52.177Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:53 smithi121 bash[36902]: cluster 2022-04-22T19:51:52.324251+0000 mgr.y (mgr.34107) 5001 : cluster [DBG] pgmap v6316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:53 smithi121 bash[36902]: audit 2022-04-22T19:51:52.698540+0000 mon.a (mon.0) 7952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:53.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:53 smithi115 bash[46934]: cluster 2022-04-22T19:51:52.324251+0000 mgr.y (mgr.34107) 5001 : cluster [DBG] pgmap v6316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:53 smithi115 bash[46934]: audit 2022-04-22T19:51:52.698540+0000 mon.a (mon.0) 7952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:53 smithi115 bash[45495]: cluster 2022-04-22T19:51:52.324251+0000 mgr.y (mgr.34107) 5001 : cluster [DBG] pgmap v6316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:53.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:53 smithi115 bash[45495]: audit 2022-04-22T19:51:52.698540+0000 mon.a (mon.0) 7952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:54.066 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:51:54.414 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:51:54.414 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 16s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:51:54.414 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 81s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:51:54.415 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 16s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:51:54.415 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 81s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:51:54.415 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 81s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:51:54.416 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 16s ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:51:54.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 16s ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:51:54.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 81s ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:51:54.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 16s ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 16s ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 81s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 16s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 16s ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 16s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 16s ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 81s ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 81s ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 81s ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:51:54.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 81s ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:51:54.418 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 81s ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:51:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:51:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:51:54.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:51:54.816 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:51:54.817 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:51:54.818 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:51:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:55 smithi121 bash[36902]: audit 2022-04-22T19:51:54.039687+0000 mgr.y (mgr.34107) 5002 : audit [DBG] from='client.40608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:55 smithi121 bash[36902]: cluster 2022-04-22T19:51:54.325275+0000 mgr.y (mgr.34107) 5003 : cluster [DBG] pgmap v6317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:55 smithi121 bash[36902]: audit 2022-04-22T19:51:54.380779+0000 mgr.y (mgr.34107) 5004 : audit [DBG] from='client.50548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:55 smithi121 bash[36902]: audit 2022-04-22T19:51:54.806590+0000 mon.a (mon.0) 7953 : audit [DBG] from='client.? 172.21.15.115:0/3908188662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:55 smithi121 bash[36902]: audit 2022-04-22T19:51:54.854250+0000 mon.a (mon.0) 7954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[45495]: audit 2022-04-22T19:51:54.039687+0000 mgr.y (mgr.34107) 5002 : audit [DBG] from='client.40608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[45495]: cluster 2022-04-22T19:51:54.325275+0000 mgr.y (mgr.34107) 5003 : cluster [DBG] pgmap v6317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[45495]: audit 2022-04-22T19:51:54.380779+0000 mgr.y (mgr.34107) 5004 : audit [DBG] from='client.50548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[45495]: audit 2022-04-22T19:51:54.806590+0000 mon.a (mon.0) 7953 : audit [DBG] from='client.? 172.21.15.115:0/3908188662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:55.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[45495]: audit 2022-04-22T19:51:54.854250+0000 mon.a (mon.0) 7954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:55.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[46934]: audit 2022-04-22T19:51:54.039687+0000 mgr.y (mgr.34107) 5002 : audit [DBG] from='client.40608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[46934]: cluster 2022-04-22T19:51:54.325275+0000 mgr.y (mgr.34107) 5003 : cluster [DBG] pgmap v6317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:55.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[46934]: audit 2022-04-22T19:51:54.380779+0000 mgr.y (mgr.34107) 5004 : audit [DBG] from='client.50548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:51:55.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[46934]: audit 2022-04-22T19:51:54.806590+0000 mon.a (mon.0) 7953 : audit [DBG] from='client.? 172.21.15.115:0/3908188662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:51:55.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:55 smithi115 bash[46934]: audit 2022-04-22T19:51:54.854250+0000 mon.a (mon.0) 7954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:51:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:56 smithi121 bash[36902]: audit 2022-04-22T19:51:56.240910+0000 mon.a (mon.0) 7955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:56.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:56 smithi115 bash[45495]: audit 2022-04-22T19:51:56.240910+0000 mon.a (mon.0) 7955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:56.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:56 smithi115 bash[46934]: audit 2022-04-22T19:51:56.240910+0000 mon.a (mon.0) 7955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:51:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:57 smithi121 bash[36902]: cluster 2022-04-22T19:51:56.326303+0000 mgr.y (mgr.34107) 5005 : cluster [DBG] pgmap v6318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:57 smithi115 bash[46934]: cluster 2022-04-22T19:51:56.326303+0000 mgr.y (mgr.34107) 5005 : cluster [DBG] pgmap v6318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:57 smithi115 bash[45495]: cluster 2022-04-22T19:51:56.326303+0000 mgr.y (mgr.34107) 5005 : cluster [DBG] pgmap v6318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:58 smithi121 bash[36902]: audit 2022-04-22T19:51:57.701543+0000 mon.a (mon.0) 7956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:58.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:58 smithi115 bash[46934]: audit 2022-04-22T19:51:57.701543+0000 mon.a (mon.0) 7956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:58 smithi115 bash[45495]: audit 2022-04-22T19:51:57.701543+0000 mon.a (mon.0) 7956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:51:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:51:59 smithi121 bash[36902]: cluster 2022-04-22T19:51:58.327361+0000 mgr.y (mgr.34107) 5006 : cluster [DBG] pgmap v6319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:51:59 smithi115 bash[46934]: cluster 2022-04-22T19:51:58.327361+0000 mgr.y (mgr.34107) 5006 : cluster [DBG] pgmap v6319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:51:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:51:59 smithi115 bash[45495]: cluster 2022-04-22T19:51:58.327361+0000 mgr.y (mgr.34107) 5006 : cluster [DBG] pgmap v6319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:00 smithi121 bash[36902]: audit 2022-04-22T19:51:59.857972+0000 mon.a (mon.0) 7957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:00 smithi115 bash[46934]: audit 2022-04-22T19:51:59.857972+0000 mon.a (mon.0) 7957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:00.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:00 smithi115 bash[45495]: audit 2022-04-22T19:51:59.857972+0000 mon.a (mon.0) 7957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:01 smithi121 bash[36902]: cluster 2022-04-22T19:52:00.328393+0000 mgr.y (mgr.34107) 5007 : cluster [DBG] pgmap v6320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:01 smithi121 bash[36902]: audit 2022-04-22T19:52:01.244042+0000 mon.a (mon.0) 7958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:01 smithi115 bash[45495]: cluster 2022-04-22T19:52:00.328393+0000 mgr.y (mgr.34107) 5007 : cluster [DBG] pgmap v6320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:01.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:01 smithi115 bash[45495]: audit 2022-04-22T19:52:01.244042+0000 mon.a (mon.0) 7958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:01 smithi115 bash[46934]: cluster 2022-04-22T19:52:00.328393+0000 mgr.y (mgr.34107) 5007 : cluster [DBG] pgmap v6320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:01.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:01 smithi115 bash[46934]: audit 2022-04-22T19:52:01.244042+0000 mon.a (mon.0) 7958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:02.179Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:03 smithi121 bash[36902]: cluster 2022-04-22T19:52:02.329493+0000 mgr.y (mgr.34107) 5008 : cluster [DBG] pgmap v6321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:03 smithi121 bash[36902]: audit 2022-04-22T19:52:02.704789+0000 mon.a (mon.0) 7959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:03 smithi115 bash[45495]: cluster 2022-04-22T19:52:02.329493+0000 mgr.y (mgr.34107) 5008 : cluster [DBG] pgmap v6321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:03 smithi115 bash[45495]: audit 2022-04-22T19:52:02.704789+0000 mon.a (mon.0) 7959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:03 smithi115 bash[46934]: cluster 2022-04-22T19:52:02.329493+0000 mgr.y (mgr.34107) 5008 : cluster [DBG] pgmap v6321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:03 smithi115 bash[46934]: audit 2022-04-22T19:52:02.704789+0000 mon.a (mon.0) 7959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:04.054Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:05 smithi121 bash[36902]: cluster 2022-04-22T19:52:04.330867+0000 mgr.y (mgr.34107) 5009 : cluster [DBG] pgmap v6322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:05 smithi121 bash[36902]: audit 2022-04-22T19:52:04.861749+0000 mon.a (mon.0) 7960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:05.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:05 smithi115 bash[45495]: cluster 2022-04-22T19:52:04.330867+0000 mgr.y (mgr.34107) 5009 : cluster [DBG] pgmap v6322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:05 smithi115 bash[45495]: audit 2022-04-22T19:52:04.861749+0000 mon.a (mon.0) 7960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:05 smithi115 bash[46934]: cluster 2022-04-22T19:52:04.330867+0000 mgr.y (mgr.34107) 5009 : cluster [DBG] pgmap v6322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:05 smithi115 bash[46934]: audit 2022-04-22T19:52:04.861749+0000 mon.a (mon.0) 7960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:06 smithi121 bash[36902]: audit 2022-04-22T19:52:06.247053+0000 mon.a (mon.0) 7961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:06.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:06 smithi115 bash[45495]: audit 2022-04-22T19:52:06.247053+0000 mon.a (mon.0) 7961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:06.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:06 smithi115 bash[46934]: audit 2022-04-22T19:52:06.247053+0000 mon.a (mon.0) 7961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:07 smithi121 bash[36902]: cluster 2022-04-22T19:52:06.331957+0000 mgr.y (mgr.34107) 5010 : cluster [DBG] pgmap v6323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:07 smithi115 bash[45495]: cluster 2022-04-22T19:52:06.331957+0000 mgr.y (mgr.34107) 5010 : cluster [DBG] pgmap v6323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:07 smithi115 bash[46934]: cluster 2022-04-22T19:52:06.331957+0000 mgr.y (mgr.34107) 5010 : cluster [DBG] pgmap v6323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:08 smithi121 bash[36902]: audit 2022-04-22T19:52:07.707885+0000 mon.a (mon.0) 7962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:08 smithi115 bash[45495]: audit 2022-04-22T19:52:07.707885+0000 mon.a (mon.0) 7962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:08.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:08 smithi115 bash[46934]: audit 2022-04-22T19:52:07.707885+0000 mon.a (mon.0) 7962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:09 smithi121 bash[36902]: cluster 2022-04-22T19:52:08.332993+0000 mgr.y (mgr.34107) 5011 : cluster [DBG] pgmap v6324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:09 smithi115 bash[45495]: cluster 2022-04-22T19:52:08.332993+0000 mgr.y (mgr.34107) 5011 : cluster [DBG] pgmap v6324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:09 smithi115 bash[46934]: cluster 2022-04-22T19:52:08.332993+0000 mgr.y (mgr.34107) 5011 : cluster [DBG] pgmap v6324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:10 smithi121 bash[36902]: audit 2022-04-22T19:52:09.865481+0000 mon.a (mon.0) 7963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:10.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:10 smithi115 bash[45495]: audit 2022-04-22T19:52:09.865481+0000 mon.a (mon.0) 7963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:10.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:10 smithi115 bash[46934]: audit 2022-04-22T19:52:09.865481+0000 mon.a (mon.0) 7963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:11 smithi121 bash[36902]: cluster 2022-04-22T19:52:10.334060+0000 mgr.y (mgr.34107) 5012 : cluster [DBG] pgmap v6325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:11 smithi121 bash[36902]: audit 2022-04-22T19:52:11.250071+0000 mon.a (mon.0) 7964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:11.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:11 smithi115 bash[45495]: cluster 2022-04-22T19:52:10.334060+0000 mgr.y (mgr.34107) 5012 : cluster [DBG] pgmap v6325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:11.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:11 smithi115 bash[45495]: audit 2022-04-22T19:52:11.250071+0000 mon.a (mon.0) 7964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:11 smithi115 bash[46934]: cluster 2022-04-22T19:52:10.334060+0000 mgr.y (mgr.34107) 5012 : cluster [DBG] pgmap v6325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:11.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:11 smithi115 bash[46934]: audit 2022-04-22T19:52:11.250071+0000 mon.a (mon.0) 7964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:12.180Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:13 smithi121 bash[36902]: cluster 2022-04-22T19:52:12.334922+0000 mgr.y (mgr.34107) 5013 : cluster [DBG] pgmap v6326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:13 smithi121 bash[36902]: audit 2022-04-22T19:52:12.710728+0000 mon.a (mon.0) 7965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:13.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:13 smithi115 bash[45495]: cluster 2022-04-22T19:52:12.334922+0000 mgr.y (mgr.34107) 5013 : cluster [DBG] pgmap v6326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:13.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:13 smithi115 bash[45495]: audit 2022-04-22T19:52:12.710728+0000 mon.a (mon.0) 7965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:13 smithi115 bash[46934]: cluster 2022-04-22T19:52:12.334922+0000 mgr.y (mgr.34107) 5013 : cluster [DBG] pgmap v6326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:13.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:13 smithi115 bash[46934]: audit 2022-04-22T19:52:12.710728+0000 mon.a (mon.0) 7965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:14.055Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:15 smithi121 bash[36902]: cluster 2022-04-22T19:52:14.335985+0000 mgr.y (mgr.34107) 5014 : cluster [DBG] pgmap v6327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:15 smithi121 bash[36902]: audit 2022-04-22T19:52:14.868992+0000 mon.a (mon.0) 7966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:15 smithi121 bash[36902]: audit 2022-04-22T19:52:14.954866+0000 mon.a (mon.0) 7967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:52:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:15 smithi121 bash[36902]: audit 2022-04-22T19:52:14.955366+0000 mon.a (mon.0) 7968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:52:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[45495]: cluster 2022-04-22T19:52:14.335985+0000 mgr.y (mgr.34107) 5014 : cluster [DBG] pgmap v6327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[45495]: audit 2022-04-22T19:52:14.868992+0000 mon.a (mon.0) 7966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[45495]: audit 2022-04-22T19:52:14.954866+0000 mon.a (mon.0) 7967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:52:15.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[45495]: audit 2022-04-22T19:52:14.955366+0000 mon.a (mon.0) 7968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:52:15.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[46934]: cluster 2022-04-22T19:52:14.335985+0000 mgr.y (mgr.34107) 5014 : cluster [DBG] pgmap v6327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[46934]: audit 2022-04-22T19:52:14.868992+0000 mon.a (mon.0) 7966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[46934]: audit 2022-04-22T19:52:14.954866+0000 mon.a (mon.0) 7967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:52:15.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:15 smithi115 bash[46934]: audit 2022-04-22T19:52:14.955366+0000 mon.a (mon.0) 7968 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:52:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:16 smithi121 bash[36902]: audit 2022-04-22T19:52:16.253124+0000 mon.a (mon.0) 7969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:16.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:16 smithi115 bash[45495]: audit 2022-04-22T19:52:16.253124+0000 mon.a (mon.0) 7969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:16 smithi115 bash[46934]: audit 2022-04-22T19:52:16.253124+0000 mon.a (mon.0) 7969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:17 smithi121 bash[36902]: cluster 2022-04-22T19:52:16.336939+0000 mgr.y (mgr.34107) 5015 : cluster [DBG] pgmap v6328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:17 smithi115 bash[45495]: cluster 2022-04-22T19:52:16.336939+0000 mgr.y (mgr.34107) 5015 : cluster [DBG] pgmap v6328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:17 smithi115 bash[46934]: cluster 2022-04-22T19:52:16.336939+0000 mgr.y (mgr.34107) 5015 : cluster [DBG] pgmap v6328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:18 smithi121 bash[36902]: audit 2022-04-22T19:52:17.714025+0000 mon.a (mon.0) 7970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:18 smithi115 bash[45495]: audit 2022-04-22T19:52:17.714025+0000 mon.a (mon.0) 7970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:18 smithi115 bash[46934]: audit 2022-04-22T19:52:17.714025+0000 mon.a (mon.0) 7970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:19 smithi121 bash[36902]: cluster 2022-04-22T19:52:18.337929+0000 mgr.y (mgr.34107) 5016 : cluster [DBG] pgmap v6329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:19 smithi115 bash[45495]: cluster 2022-04-22T19:52:18.337929+0000 mgr.y (mgr.34107) 5016 : cluster [DBG] pgmap v6329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:19.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:19 smithi115 bash[46934]: cluster 2022-04-22T19:52:18.337929+0000 mgr.y (mgr.34107) 5016 : cluster [DBG] pgmap v6329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:20 smithi121 bash[36902]: audit 2022-04-22T19:52:19.872573+0000 mon.a (mon.0) 7971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:20 smithi115 bash[45495]: audit 2022-04-22T19:52:19.872573+0000 mon.a (mon.0) 7971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:20 smithi115 bash[46934]: audit 2022-04-22T19:52:19.872573+0000 mon.a (mon.0) 7971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:21 smithi121 bash[36902]: cluster 2022-04-22T19:52:20.338857+0000 mgr.y (mgr.34107) 5017 : cluster [DBG] pgmap v6330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:21 smithi121 bash[36902]: audit 2022-04-22T19:52:21.256493+0000 mon.a (mon.0) 7972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:21 smithi115 bash[45495]: cluster 2022-04-22T19:52:20.338857+0000 mgr.y (mgr.34107) 5017 : cluster [DBG] pgmap v6330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:21 smithi115 bash[45495]: audit 2022-04-22T19:52:21.256493+0000 mon.a (mon.0) 7972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:21.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:21 smithi115 bash[46934]: cluster 2022-04-22T19:52:20.338857+0000 mgr.y (mgr.34107) 5017 : cluster [DBG] pgmap v6330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:21 smithi115 bash[46934]: audit 2022-04-22T19:52:21.256493+0000 mon.a (mon.0) 7972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:22.179Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:23 smithi121 bash[36902]: cluster 2022-04-22T19:52:22.339962+0000 mgr.y (mgr.34107) 5018 : cluster [DBG] pgmap v6331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:23 smithi121 bash[36902]: audit 2022-04-22T19:52:22.716928+0000 mon.a (mon.0) 7973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:23 smithi115 bash[46934]: cluster 2022-04-22T19:52:22.339962+0000 mgr.y (mgr.34107) 5018 : cluster [DBG] pgmap v6331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:23 smithi115 bash[46934]: audit 2022-04-22T19:52:22.716928+0000 mon.a (mon.0) 7973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:23 smithi115 bash[45495]: cluster 2022-04-22T19:52:22.339962+0000 mgr.y (mgr.34107) 5018 : cluster [DBG] pgmap v6331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:23 smithi115 bash[45495]: audit 2022-04-22T19:52:22.716928+0000 mon.a (mon.0) 7973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:24.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:25.199 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:52:25.541 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:52:25.542 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 47s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 112s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 47s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 112s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 112s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 47s ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 47s ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 112s ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 47s ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:52:25.543 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 47s ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 112s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 47s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 47s ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 47s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 47s ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 112s ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 112s ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 112s ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 112s ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:52:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 112s ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:52:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:25 smithi121 bash[36902]: cluster 2022-04-22T19:52:24.341079+0000 mgr.y (mgr.34107) 5019 : cluster [DBG] pgmap v6332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:25 smithi121 bash[36902]: audit 2022-04-22T19:52:24.876477+0000 mon.a (mon.0) 7974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:25 smithi115 bash[46934]: cluster 2022-04-22T19:52:24.341079+0000 mgr.y (mgr.34107) 5019 : cluster [DBG] pgmap v6332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:25 smithi115 bash[46934]: audit 2022-04-22T19:52:24.876477+0000 mon.a (mon.0) 7974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:25 smithi115 bash[45495]: cluster 2022-04-22T19:52:24.341079+0000 mgr.y (mgr.34107) 5019 : cluster [DBG] pgmap v6332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:25.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:25 smithi115 bash[45495]: audit 2022-04-22T19:52:24.876477+0000 mon.a (mon.0) 7974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:52:25.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:52:25.944 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:52:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:26 smithi121 bash[36902]: audit 2022-04-22T19:52:25.173779+0000 mgr.y (mgr.34107) 5020 : audit [DBG] from='client.40626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:26 smithi121 bash[36902]: audit 2022-04-22T19:52:25.512228+0000 mgr.y (mgr.34107) 5021 : audit [DBG] from='client.40632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:26 smithi121 bash[36902]: audit 2022-04-22T19:52:25.933504+0000 mon.a (mon.0) 7975 : audit [DBG] from='client.? 172.21.15.115:0/942753679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:26 smithi121 bash[36902]: audit 2022-04-22T19:52:26.259527+0000 mon.a (mon.0) 7976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[45495]: audit 2022-04-22T19:52:25.173779+0000 mgr.y (mgr.34107) 5020 : audit [DBG] from='client.40626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[45495]: audit 2022-04-22T19:52:25.512228+0000 mgr.y (mgr.34107) 5021 : audit [DBG] from='client.40632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[45495]: audit 2022-04-22T19:52:25.933504+0000 mon.a (mon.0) 7975 : audit [DBG] from='client.? 172.21.15.115:0/942753679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:26.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[45495]: audit 2022-04-22T19:52:26.259527+0000 mon.a (mon.0) 7976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[46934]: audit 2022-04-22T19:52:25.173779+0000 mgr.y (mgr.34107) 5020 : audit [DBG] from='client.40626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[46934]: audit 2022-04-22T19:52:25.512228+0000 mgr.y (mgr.34107) 5021 : audit [DBG] from='client.40632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[46934]: audit 2022-04-22T19:52:25.933504+0000 mon.a (mon.0) 7975 : audit [DBG] from='client.? 172.21.15.115:0/942753679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:26.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:26 smithi115 bash[46934]: audit 2022-04-22T19:52:26.259527+0000 mon.a (mon.0) 7976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:27 smithi121 bash[36902]: cluster 2022-04-22T19:52:26.342148+0000 mgr.y (mgr.34107) 5022 : cluster [DBG] pgmap v6333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:27 smithi115 bash[45495]: cluster 2022-04-22T19:52:26.342148+0000 mgr.y (mgr.34107) 5022 : cluster [DBG] pgmap v6333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:27.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:27 smithi115 bash[46934]: cluster 2022-04-22T19:52:26.342148+0000 mgr.y (mgr.34107) 5022 : cluster [DBG] pgmap v6333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:28 smithi121 bash[36902]: audit 2022-04-22T19:52:27.720058+0000 mon.a (mon.0) 7977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:28 smithi115 bash[45495]: audit 2022-04-22T19:52:27.720058+0000 mon.a (mon.0) 7977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:28 smithi115 bash[46934]: audit 2022-04-22T19:52:27.720058+0000 mon.a (mon.0) 7977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:29 smithi121 bash[36902]: cluster 2022-04-22T19:52:28.343035+0000 mgr.y (mgr.34107) 5023 : cluster [DBG] pgmap v6334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:29.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:29 smithi115 bash[45495]: cluster 2022-04-22T19:52:28.343035+0000 mgr.y (mgr.34107) 5023 : cluster [DBG] pgmap v6334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:29 smithi115 bash[46934]: cluster 2022-04-22T19:52:28.343035+0000 mgr.y (mgr.34107) 5023 : cluster [DBG] pgmap v6334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:30 smithi121 bash[36902]: audit 2022-04-22T19:52:29.879813+0000 mon.a (mon.0) 7978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:30.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:30 smithi115 bash[45495]: audit 2022-04-22T19:52:29.879813+0000 mon.a (mon.0) 7978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:30 smithi115 bash[46934]: audit 2022-04-22T19:52:29.879813+0000 mon.a (mon.0) 7978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:31 smithi121 bash[36902]: cluster 2022-04-22T19:52:30.344094+0000 mgr.y (mgr.34107) 5024 : cluster [DBG] pgmap v6335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:31 smithi121 bash[36902]: audit 2022-04-22T19:52:31.262632+0000 mon.a (mon.0) 7979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:31 smithi115 bash[45495]: cluster 2022-04-22T19:52:30.344094+0000 mgr.y (mgr.34107) 5024 : cluster [DBG] pgmap v6335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:31.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:31 smithi115 bash[45495]: audit 2022-04-22T19:52:31.262632+0000 mon.a (mon.0) 7979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:31 smithi115 bash[46934]: cluster 2022-04-22T19:52:30.344094+0000 mgr.y (mgr.34107) 5024 : cluster [DBG] pgmap v6335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:31 smithi115 bash[46934]: audit 2022-04-22T19:52:31.262632+0000 mon.a (mon.0) 7979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:32.180Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:33 smithi121 bash[36902]: cluster 2022-04-22T19:52:32.345196+0000 mgr.y (mgr.34107) 5025 : cluster [DBG] pgmap v6336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:33 smithi121 bash[36902]: audit 2022-04-22T19:52:32.723118+0000 mon.a (mon.0) 7980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:33 smithi115 bash[45495]: cluster 2022-04-22T19:52:32.345196+0000 mgr.y (mgr.34107) 5025 : cluster [DBG] pgmap v6336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:33.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:33 smithi115 bash[45495]: audit 2022-04-22T19:52:32.723118+0000 mon.a (mon.0) 7980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:33 smithi115 bash[46934]: cluster 2022-04-22T19:52:32.345196+0000 mgr.y (mgr.34107) 5025 : cluster [DBG] pgmap v6336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:33 smithi115 bash[46934]: audit 2022-04-22T19:52:32.723118+0000 mon.a (mon.0) 7980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:34.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:35 smithi121 bash[36902]: cluster 2022-04-22T19:52:34.346306+0000 mgr.y (mgr.34107) 5026 : cluster [DBG] pgmap v6337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:35 smithi121 bash[36902]: audit 2022-04-22T19:52:34.883414+0000 mon.a (mon.0) 7981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:35 smithi115 bash[46934]: cluster 2022-04-22T19:52:34.346306+0000 mgr.y (mgr.34107) 5026 : cluster [DBG] pgmap v6337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:35 smithi115 bash[46934]: audit 2022-04-22T19:52:34.883414+0000 mon.a (mon.0) 7981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:35 smithi115 bash[45495]: cluster 2022-04-22T19:52:34.346306+0000 mgr.y (mgr.34107) 5026 : cluster [DBG] pgmap v6337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:35 smithi115 bash[45495]: audit 2022-04-22T19:52:34.883414+0000 mon.a (mon.0) 7981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:36 smithi115 bash[45495]: audit 2022-04-22T19:52:36.265965+0000 mon.a (mon.0) 7982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:36 smithi115 bash[46934]: audit 2022-04-22T19:52:36.265965+0000 mon.a (mon.0) 7982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:36 smithi121 bash[36902]: audit 2022-04-22T19:52:36.265965+0000 mon.a (mon.0) 7982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:37 smithi115 bash[45495]: cluster 2022-04-22T19:52:36.347289+0000 mgr.y (mgr.34107) 5027 : cluster [DBG] pgmap v6338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:37 smithi115 bash[46934]: cluster 2022-04-22T19:52:36.347289+0000 mgr.y (mgr.34107) 5027 : cluster [DBG] pgmap v6338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:37 smithi121 bash[36902]: cluster 2022-04-22T19:52:36.347289+0000 mgr.y (mgr.34107) 5027 : cluster [DBG] pgmap v6338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[45495]: audit 2022-04-22T19:52:37.726233+0000 mon.a (mon.0) 7983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[45495]: audit 2022-04-22T19:52:38.199241+0000 mon.a (mon.0) 7984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[45495]: audit 2022-04-22T19:52:38.202350+0000 mon.a (mon.0) 7985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[45495]: audit 2022-04-22T19:52:38.204195+0000 mon.a (mon.0) 7986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[46934]: audit 2022-04-22T19:52:37.726233+0000 mon.a (mon.0) 7983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[46934]: audit 2022-04-22T19:52:38.199241+0000 mon.a (mon.0) 7984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:52:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[46934]: audit 2022-04-22T19:52:38.202350+0000 mon.a (mon.0) 7985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:52:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:38 smithi115 bash[46934]: audit 2022-04-22T19:52:38.204195+0000 mon.a (mon.0) 7986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:52:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:38 smithi121 bash[36902]: audit 2022-04-22T19:52:37.726233+0000 mon.a (mon.0) 7983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:39.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:38 smithi121 bash[36902]: audit 2022-04-22T19:52:38.199241+0000 mon.a (mon.0) 7984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:52:39.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:38 smithi121 bash[36902]: audit 2022-04-22T19:52:38.202350+0000 mon.a (mon.0) 7985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:52:39.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:38 smithi121 bash[36902]: audit 2022-04-22T19:52:38.204195+0000 mon.a (mon.0) 7986 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:52:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:39 smithi121 bash[36902]: cluster 2022-04-22T19:52:38.348370+0000 mgr.y (mgr.34107) 5028 : cluster [DBG] pgmap v6339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:39 smithi121 bash[36902]: audit 2022-04-22T19:52:38.571126+0000 mon.a (mon.0) 7987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:52:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:39 smithi115 bash[45495]: cluster 2022-04-22T19:52:38.348370+0000 mgr.y (mgr.34107) 5028 : cluster [DBG] pgmap v6339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:39 smithi115 bash[45495]: audit 2022-04-22T19:52:38.571126+0000 mon.a (mon.0) 7987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:52:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:39 smithi115 bash[46934]: cluster 2022-04-22T19:52:38.348370+0000 mgr.y (mgr.34107) 5028 : cluster [DBG] pgmap v6339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:39 smithi115 bash[46934]: audit 2022-04-22T19:52:38.571126+0000 mon.a (mon.0) 7987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:52:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:40 smithi121 bash[36902]: audit 2022-04-22T19:52:39.887165+0000 mon.a (mon.0) 7988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:40 smithi115 bash[45495]: audit 2022-04-22T19:52:39.887165+0000 mon.a (mon.0) 7988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:41.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:40 smithi115 bash[46934]: audit 2022-04-22T19:52:39.887165+0000 mon.a (mon.0) 7988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:41 smithi121 bash[36902]: cluster 2022-04-22T19:52:40.349408+0000 mgr.y (mgr.34107) 5029 : cluster [DBG] pgmap v6340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:41 smithi121 bash[36902]: audit 2022-04-22T19:52:41.269142+0000 mon.a (mon.0) 7989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:41 smithi115 bash[45495]: cluster 2022-04-22T19:52:40.349408+0000 mgr.y (mgr.34107) 5029 : cluster [DBG] pgmap v6340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:42.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:41 smithi115 bash[45495]: audit 2022-04-22T19:52:41.269142+0000 mon.a (mon.0) 7989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:41 smithi115 bash[46934]: cluster 2022-04-22T19:52:40.349408+0000 mgr.y (mgr.34107) 5029 : cluster [DBG] pgmap v6340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:41 smithi115 bash[46934]: audit 2022-04-22T19:52:41.269142+0000 mon.a (mon.0) 7989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:42.181Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:43 smithi121 bash[36902]: cluster 2022-04-22T19:52:42.350462+0000 mgr.y (mgr.34107) 5030 : cluster [DBG] pgmap v6341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:43 smithi121 bash[36902]: audit 2022-04-22T19:52:42.729545+0000 mon.a (mon.0) 7990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:43 smithi115 bash[46934]: cluster 2022-04-22T19:52:42.350462+0000 mgr.y (mgr.34107) 5030 : cluster [DBG] pgmap v6341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:43 smithi115 bash[46934]: audit 2022-04-22T19:52:42.729545+0000 mon.a (mon.0) 7990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:43 smithi115 bash[45495]: cluster 2022-04-22T19:52:42.350462+0000 mgr.y (mgr.34107) 5030 : cluster [DBG] pgmap v6341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:44.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:43 smithi115 bash[45495]: audit 2022-04-22T19:52:42.729545+0000 mon.a (mon.0) 7990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:44.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:45 smithi121 bash[36902]: cluster 2022-04-22T19:52:44.351431+0000 mgr.y (mgr.34107) 5031 : cluster [DBG] pgmap v6342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:45 smithi121 bash[36902]: audit 2022-04-22T19:52:44.890660+0000 mon.a (mon.0) 7991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:45 smithi115 bash[45495]: cluster 2022-04-22T19:52:44.351431+0000 mgr.y (mgr.34107) 5031 : cluster [DBG] pgmap v6342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:45 smithi115 bash[45495]: audit 2022-04-22T19:52:44.890660+0000 mon.a (mon.0) 7991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:45 smithi115 bash[46934]: cluster 2022-04-22T19:52:44.351431+0000 mgr.y (mgr.34107) 5031 : cluster [DBG] pgmap v6342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:45 smithi115 bash[46934]: audit 2022-04-22T19:52:44.890660+0000 mon.a (mon.0) 7991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:46 smithi121 bash[36902]: audit 2022-04-22T19:52:46.272223+0000 mon.a (mon.0) 7992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:46 smithi115 bash[45495]: audit 2022-04-22T19:52:46.272223+0000 mon.a (mon.0) 7992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:46 smithi115 bash[46934]: audit 2022-04-22T19:52:46.272223+0000 mon.a (mon.0) 7992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:47 smithi121 bash[36902]: cluster 2022-04-22T19:52:46.352360+0000 mgr.y (mgr.34107) 5032 : cluster [DBG] pgmap v6343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:47 smithi115 bash[45495]: cluster 2022-04-22T19:52:46.352360+0000 mgr.y (mgr.34107) 5032 : cluster [DBG] pgmap v6343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:47 smithi115 bash[46934]: cluster 2022-04-22T19:52:46.352360+0000 mgr.y (mgr.34107) 5032 : cluster [DBG] pgmap v6343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:48 smithi121 bash[36902]: audit 2022-04-22T19:52:47.732728+0000 mon.a (mon.0) 7993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:48 smithi115 bash[45495]: audit 2022-04-22T19:52:47.732728+0000 mon.a (mon.0) 7993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:48 smithi115 bash[46934]: audit 2022-04-22T19:52:47.732728+0000 mon.a (mon.0) 7993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:49 smithi121 bash[36902]: cluster 2022-04-22T19:52:48.353587+0000 mgr.y (mgr.34107) 5033 : cluster [DBG] pgmap v6344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:49 smithi115 bash[46934]: cluster 2022-04-22T19:52:48.353587+0000 mgr.y (mgr.34107) 5033 : cluster [DBG] pgmap v6344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:49 smithi115 bash[45495]: cluster 2022-04-22T19:52:48.353587+0000 mgr.y (mgr.34107) 5033 : cluster [DBG] pgmap v6344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:50 smithi121 bash[36902]: audit 2022-04-22T19:52:49.894329+0000 mon.a (mon.0) 7994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:50 smithi115 bash[46934]: audit 2022-04-22T19:52:49.894329+0000 mon.a (mon.0) 7994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:51.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:50 smithi115 bash[45495]: audit 2022-04-22T19:52:49.894329+0000 mon.a (mon.0) 7994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:51 smithi121 bash[36902]: cluster 2022-04-22T19:52:50.354811+0000 mgr.y (mgr.34107) 5034 : cluster [DBG] pgmap v6345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:51 smithi121 bash[36902]: audit 2022-04-22T19:52:51.275245+0000 mon.a (mon.0) 7995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:51 smithi115 bash[46934]: cluster 2022-04-22T19:52:50.354811+0000 mgr.y (mgr.34107) 5034 : cluster [DBG] pgmap v6345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:51 smithi115 bash[46934]: audit 2022-04-22T19:52:51.275245+0000 mon.a (mon.0) 7995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:51 smithi115 bash[45495]: cluster 2022-04-22T19:52:50.354811+0000 mgr.y (mgr.34107) 5034 : cluster [DBG] pgmap v6345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:51 smithi115 bash[45495]: audit 2022-04-22T19:52:51.275245+0000 mon.a (mon.0) 7995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:52.182Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:53 smithi121 bash[36902]: cluster 2022-04-22T19:52:52.355959+0000 mgr.y (mgr.34107) 5035 : cluster [DBG] pgmap v6346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:53 smithi121 bash[36902]: audit 2022-04-22T19:52:52.735813+0000 mon.a (mon.0) 7996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:53 smithi115 bash[45495]: cluster 2022-04-22T19:52:52.355959+0000 mgr.y (mgr.34107) 5035 : cluster [DBG] pgmap v6346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:54.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:53 smithi115 bash[45495]: audit 2022-04-22T19:52:52.735813+0000 mon.a (mon.0) 7996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:53 smithi115 bash[46934]: cluster 2022-04-22T19:52:52.355959+0000 mgr.y (mgr.34107) 5035 : cluster [DBG] pgmap v6346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:53 smithi115 bash[46934]: audit 2022-04-22T19:52:52.735813+0000 mon.a (mon.0) 7996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:52:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:52:54.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:52:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:55 smithi121 bash[36902]: cluster 2022-04-22T19:52:54.357069+0000 mgr.y (mgr.34107) 5036 : cluster [DBG] pgmap v6347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:55 smithi121 bash[36902]: audit 2022-04-22T19:52:54.897892+0000 mon.a (mon.0) 7997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:55 smithi115 bash[45495]: cluster 2022-04-22T19:52:54.357069+0000 mgr.y (mgr.34107) 5036 : cluster [DBG] pgmap v6347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:55 smithi115 bash[45495]: audit 2022-04-22T19:52:54.897892+0000 mon.a (mon.0) 7997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:55 smithi115 bash[46934]: cluster 2022-04-22T19:52:54.357069+0000 mgr.y (mgr.34107) 5036 : cluster [DBG] pgmap v6347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:55 smithi115 bash[46934]: audit 2022-04-22T19:52:54.897892+0000 mon.a (mon.0) 7997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:52:56.322 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:52:56.663 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:52:56.663 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 78s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 78s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 78s ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 78s ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 78s ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:52:56.664 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 78s ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 78s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 78s ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 78s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 78s ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:52:56.665 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:52:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:56 smithi121 bash[36902]: audit 2022-04-22T19:52:56.278026+0000 mon.a (mon.0) 7998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:56 smithi115 bash[46934]: audit 2022-04-22T19:52:56.278026+0000 mon.a (mon.0) 7998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:56 smithi115 bash[45495]: audit 2022-04-22T19:52:56.278026+0000 mon.a (mon.0) 7998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:52:57.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:52:57.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:52:57.062 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:52:57.062 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:52:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:57 smithi121 bash[36902]: audit 2022-04-22T19:52:56.297170+0000 mgr.y (mgr.34107) 5037 : audit [DBG] from='client.40644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:57 smithi121 bash[36902]: cluster 2022-04-22T19:52:56.358047+0000 mgr.y (mgr.34107) 5038 : cluster [DBG] pgmap v6348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:57 smithi121 bash[36902]: audit 2022-04-22T19:52:56.636678+0000 mgr.y (mgr.34107) 5039 : audit [DBG] from='client.40650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:57 smithi121 bash[36902]: audit 2022-04-22T19:52:57.051067+0000 mon.a (mon.0) 7999 : audit [DBG] from='client.? 172.21.15.115:0/2544563086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:58.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[45495]: audit 2022-04-22T19:52:56.297170+0000 mgr.y (mgr.34107) 5037 : audit [DBG] from='client.40644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[45495]: cluster 2022-04-22T19:52:56.358047+0000 mgr.y (mgr.34107) 5038 : cluster [DBG] pgmap v6348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[45495]: audit 2022-04-22T19:52:56.636678+0000 mgr.y (mgr.34107) 5039 : audit [DBG] from='client.40650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[45495]: audit 2022-04-22T19:52:57.051067+0000 mon.a (mon.0) 7999 : audit [DBG] from='client.? 172.21.15.115:0/2544563086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[46934]: audit 2022-04-22T19:52:56.297170+0000 mgr.y (mgr.34107) 5037 : audit [DBG] from='client.40644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[46934]: cluster 2022-04-22T19:52:56.358047+0000 mgr.y (mgr.34107) 5038 : cluster [DBG] pgmap v6348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[46934]: audit 2022-04-22T19:52:56.636678+0000 mgr.y (mgr.34107) 5039 : audit [DBG] from='client.40650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:52:58.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:57 smithi115 bash[46934]: audit 2022-04-22T19:52:57.051067+0000 mon.a (mon.0) 7999 : audit [DBG] from='client.? 172.21.15.115:0/2544563086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:52:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:58 smithi121 bash[36902]: audit 2022-04-22T19:52:57.738826+0000 mon.a (mon.0) 8000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:58 smithi115 bash[46934]: audit 2022-04-22T19:52:57.738826+0000 mon.a (mon.0) 8000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:52:59.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:58 smithi115 bash[45495]: audit 2022-04-22T19:52:57.738826+0000 mon.a (mon.0) 8000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:52:59 smithi121 bash[36902]: cluster 2022-04-22T19:52:58.359229+0000 mgr.y (mgr.34107) 5040 : cluster [DBG] pgmap v6349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:52:59 smithi115 bash[46934]: cluster 2022-04-22T19:52:58.359229+0000 mgr.y (mgr.34107) 5040 : cluster [DBG] pgmap v6349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:00.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:52:59 smithi115 bash[45495]: cluster 2022-04-22T19:52:58.359229+0000 mgr.y (mgr.34107) 5040 : cluster [DBG] pgmap v6349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:00 smithi121 bash[36902]: audit 2022-04-22T19:52:59.901638+0000 mon.a (mon.0) 8001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:00 smithi115 bash[45495]: audit 2022-04-22T19:52:59.901638+0000 mon.a (mon.0) 8001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:00 smithi115 bash[46934]: audit 2022-04-22T19:52:59.901638+0000 mon.a (mon.0) 8001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:01 smithi121 bash[36902]: cluster 2022-04-22T19:53:00.360288+0000 mgr.y (mgr.34107) 5041 : cluster [DBG] pgmap v6350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:01 smithi121 bash[36902]: audit 2022-04-22T19:53:01.281011+0000 mon.a (mon.0) 8002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:01 smithi115 bash[45495]: cluster 2022-04-22T19:53:00.360288+0000 mgr.y (mgr.34107) 5041 : cluster [DBG] pgmap v6350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:01 smithi115 bash[45495]: audit 2022-04-22T19:53:01.281011+0000 mon.a (mon.0) 8002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:01 smithi115 bash[46934]: cluster 2022-04-22T19:53:00.360288+0000 mgr.y (mgr.34107) 5041 : cluster [DBG] pgmap v6350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:01 smithi115 bash[46934]: audit 2022-04-22T19:53:01.281011+0000 mon.a (mon.0) 8002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:02.182Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:03 smithi121 bash[36902]: cluster 2022-04-22T19:53:02.361372+0000 mgr.y (mgr.34107) 5042 : cluster [DBG] pgmap v6351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:04.077 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:03 smithi121 bash[36902]: audit 2022-04-22T19:53:02.741878+0000 mon.a (mon.0) 8003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:04.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:03 smithi115 bash[45495]: cluster 2022-04-22T19:53:02.361372+0000 mgr.y (mgr.34107) 5042 : cluster [DBG] pgmap v6351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:04.078 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:03 smithi115 bash[45495]: audit 2022-04-22T19:53:02.741878+0000 mon.a (mon.0) 8003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:04.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:03 smithi115 bash[46934]: cluster 2022-04-22T19:53:02.361372+0000 mgr.y (mgr.34107) 5042 : cluster [DBG] pgmap v6351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:04.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:03 smithi115 bash[46934]: audit 2022-04-22T19:53:02.741878+0000 mon.a (mon.0) 8003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:04.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:05 smithi121 bash[36902]: cluster 2022-04-22T19:53:04.362574+0000 mgr.y (mgr.34107) 5043 : cluster [DBG] pgmap v6352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:05 smithi121 bash[36902]: audit 2022-04-22T19:53:04.905301+0000 mon.a (mon.0) 8004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:05 smithi115 bash[45495]: cluster 2022-04-22T19:53:04.362574+0000 mgr.y (mgr.34107) 5043 : cluster [DBG] pgmap v6352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:05 smithi115 bash[45495]: audit 2022-04-22T19:53:04.905301+0000 mon.a (mon.0) 8004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:05 smithi115 bash[46934]: cluster 2022-04-22T19:53:04.362574+0000 mgr.y (mgr.34107) 5043 : cluster [DBG] pgmap v6352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:05 smithi115 bash[46934]: audit 2022-04-22T19:53:04.905301+0000 mon.a (mon.0) 8004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:06 smithi121 bash[36902]: audit 2022-04-22T19:53:06.284117+0000 mon.a (mon.0) 8005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:06 smithi115 bash[45495]: audit 2022-04-22T19:53:06.284117+0000 mon.a (mon.0) 8005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:06 smithi115 bash[46934]: audit 2022-04-22T19:53:06.284117+0000 mon.a (mon.0) 8005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:07 smithi121 bash[36902]: cluster 2022-04-22T19:53:06.363653+0000 mgr.y (mgr.34107) 5044 : cluster [DBG] pgmap v6353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:08.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:07 smithi115 bash[45495]: cluster 2022-04-22T19:53:06.363653+0000 mgr.y (mgr.34107) 5044 : cluster [DBG] pgmap v6353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:08.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:07 smithi115 bash[46934]: cluster 2022-04-22T19:53:06.363653+0000 mgr.y (mgr.34107) 5044 : cluster [DBG] pgmap v6353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:08 smithi121 bash[36902]: audit 2022-04-22T19:53:07.744517+0000 mon.a (mon.0) 8006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:08 smithi115 bash[45495]: audit 2022-04-22T19:53:07.744517+0000 mon.a (mon.0) 8006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:08 smithi115 bash[46934]: audit 2022-04-22T19:53:07.744517+0000 mon.a (mon.0) 8006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:09 smithi121 bash[36902]: cluster 2022-04-22T19:53:08.364962+0000 mgr.y (mgr.34107) 5045 : cluster [DBG] pgmap v6354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:09 smithi115 bash[45495]: cluster 2022-04-22T19:53:08.364962+0000 mgr.y (mgr.34107) 5045 : cluster [DBG] pgmap v6354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:09 smithi115 bash[46934]: cluster 2022-04-22T19:53:08.364962+0000 mgr.y (mgr.34107) 5045 : cluster [DBG] pgmap v6354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:10 smithi121 bash[36902]: audit 2022-04-22T19:53:09.909132+0000 mon.a (mon.0) 8007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:10 smithi115 bash[45495]: audit 2022-04-22T19:53:09.909132+0000 mon.a (mon.0) 8007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:10 smithi115 bash[46934]: audit 2022-04-22T19:53:09.909132+0000 mon.a (mon.0) 8007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:11 smithi121 bash[36902]: cluster 2022-04-22T19:53:10.365990+0000 mgr.y (mgr.34107) 5046 : cluster [DBG] pgmap v6355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:11 smithi121 bash[36902]: audit 2022-04-22T19:53:11.287310+0000 mon.a (mon.0) 8008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:11 smithi115 bash[45495]: cluster 2022-04-22T19:53:10.365990+0000 mgr.y (mgr.34107) 5046 : cluster [DBG] pgmap v6355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:11 smithi115 bash[45495]: audit 2022-04-22T19:53:11.287310+0000 mon.a (mon.0) 8008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:11 smithi115 bash[46934]: cluster 2022-04-22T19:53:10.365990+0000 mgr.y (mgr.34107) 5046 : cluster [DBG] pgmap v6355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:11 smithi115 bash[46934]: audit 2022-04-22T19:53:11.287310+0000 mon.a (mon.0) 8008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:12.183Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:13 smithi121 bash[36902]: cluster 2022-04-22T19:53:12.367092+0000 mgr.y (mgr.34107) 5047 : cluster [DBG] pgmap v6356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:13 smithi121 bash[36902]: audit 2022-04-22T19:53:12.747458+0000 mon.a (mon.0) 8009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:13 smithi115 bash[45495]: cluster 2022-04-22T19:53:12.367092+0000 mgr.y (mgr.34107) 5047 : cluster [DBG] pgmap v6356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:13 smithi115 bash[45495]: audit 2022-04-22T19:53:12.747458+0000 mon.a (mon.0) 8009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:13 smithi115 bash[46934]: cluster 2022-04-22T19:53:12.367092+0000 mgr.y (mgr.34107) 5047 : cluster [DBG] pgmap v6356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:13 smithi115 bash[46934]: audit 2022-04-22T19:53:12.747458+0000 mon.a (mon.0) 8009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:14.060Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:15 smithi121 bash[36902]: cluster 2022-04-22T19:53:14.368134+0000 mgr.y (mgr.34107) 5048 : cluster [DBG] pgmap v6357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:15 smithi121 bash[36902]: audit 2022-04-22T19:53:14.912843+0000 mon.a (mon.0) 8010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:15 smithi121 bash[36902]: audit 2022-04-22T19:53:14.961369+0000 mon.a (mon.0) 8011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:53:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:15 smithi121 bash[36902]: audit 2022-04-22T19:53:14.964480+0000 mon.a (mon.0) 8012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:53:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[45495]: cluster 2022-04-22T19:53:14.368134+0000 mgr.y (mgr.34107) 5048 : cluster [DBG] pgmap v6357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[45495]: audit 2022-04-22T19:53:14.912843+0000 mon.a (mon.0) 8010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[45495]: audit 2022-04-22T19:53:14.961369+0000 mon.a (mon.0) 8011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[45495]: audit 2022-04-22T19:53:14.964480+0000 mon.a (mon.0) 8012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[46934]: cluster 2022-04-22T19:53:14.368134+0000 mgr.y (mgr.34107) 5048 : cluster [DBG] pgmap v6357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[46934]: audit 2022-04-22T19:53:14.912843+0000 mon.a (mon.0) 8010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[46934]: audit 2022-04-22T19:53:14.961369+0000 mon.a (mon.0) 8011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:53:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:15 smithi115 bash[46934]: audit 2022-04-22T19:53:14.964480+0000 mon.a (mon.0) 8012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:53:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:16 smithi121 bash[36902]: audit 2022-04-22T19:53:16.290348+0000 mon.a (mon.0) 8013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:16 smithi115 bash[45495]: audit 2022-04-22T19:53:16.290348+0000 mon.a (mon.0) 8013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:17.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:16 smithi115 bash[46934]: audit 2022-04-22T19:53:16.290348+0000 mon.a (mon.0) 8013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:17 smithi121 bash[36902]: cluster 2022-04-22T19:53:16.369167+0000 mgr.y (mgr.34107) 5049 : cluster [DBG] pgmap v6358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:18.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:17 smithi115 bash[46934]: cluster 2022-04-22T19:53:16.369167+0000 mgr.y (mgr.34107) 5049 : cluster [DBG] pgmap v6358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:18.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:17 smithi115 bash[45495]: cluster 2022-04-22T19:53:16.369167+0000 mgr.y (mgr.34107) 5049 : cluster [DBG] pgmap v6358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:18 smithi121 bash[36902]: audit 2022-04-22T19:53:17.750364+0000 mon.a (mon.0) 8014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:18 smithi115 bash[45495]: audit 2022-04-22T19:53:17.750364+0000 mon.a (mon.0) 8014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:18 smithi115 bash[46934]: audit 2022-04-22T19:53:17.750364+0000 mon.a (mon.0) 8014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:19 smithi121 bash[36902]: cluster 2022-04-22T19:53:18.370267+0000 mgr.y (mgr.34107) 5050 : cluster [DBG] pgmap v6359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:19 smithi115 bash[45495]: cluster 2022-04-22T19:53:18.370267+0000 mgr.y (mgr.34107) 5050 : cluster [DBG] pgmap v6359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:20.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:19 smithi115 bash[46934]: cluster 2022-04-22T19:53:18.370267+0000 mgr.y (mgr.34107) 5050 : cluster [DBG] pgmap v6359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:20 smithi121 bash[36902]: audit 2022-04-22T19:53:19.916601+0000 mon.a (mon.0) 8015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:20 smithi115 bash[45495]: audit 2022-04-22T19:53:19.916601+0000 mon.a (mon.0) 8015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:20 smithi115 bash[46934]: audit 2022-04-22T19:53:19.916601+0000 mon.a (mon.0) 8015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:21 smithi121 bash[36902]: cluster 2022-04-22T19:53:20.371312+0000 mgr.y (mgr.34107) 5051 : cluster [DBG] pgmap v6360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:21 smithi121 bash[36902]: audit 2022-04-22T19:53:21.293550+0000 mon.a (mon.0) 8016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:21 smithi115 bash[45495]: cluster 2022-04-22T19:53:20.371312+0000 mgr.y (mgr.34107) 5051 : cluster [DBG] pgmap v6360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:21 smithi115 bash[45495]: audit 2022-04-22T19:53:21.293550+0000 mon.a (mon.0) 8016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:21 smithi115 bash[46934]: cluster 2022-04-22T19:53:20.371312+0000 mgr.y (mgr.34107) 5051 : cluster [DBG] pgmap v6360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:21 smithi115 bash[46934]: audit 2022-04-22T19:53:21.293550+0000 mon.a (mon.0) 8016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:22.183Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:23 smithi121 bash[36902]: cluster 2022-04-22T19:53:22.372375+0000 mgr.y (mgr.34107) 5052 : cluster [DBG] pgmap v6361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:23 smithi121 bash[36902]: audit 2022-04-22T19:53:22.753131+0000 mon.a (mon.0) 8017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:23 smithi115 bash[45495]: cluster 2022-04-22T19:53:22.372375+0000 mgr.y (mgr.34107) 5052 : cluster [DBG] pgmap v6361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:23 smithi115 bash[45495]: audit 2022-04-22T19:53:22.753131+0000 mon.a (mon.0) 8017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:23 smithi115 bash[46934]: cluster 2022-04-22T19:53:22.372375+0000 mgr.y (mgr.34107) 5052 : cluster [DBG] pgmap v6361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:23 smithi115 bash[46934]: audit 2022-04-22T19:53:22.753131+0000 mon.a (mon.0) 8017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:24.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:25 smithi121 bash[36902]: cluster 2022-04-22T19:53:24.373450+0000 mgr.y (mgr.34107) 5053 : cluster [DBG] pgmap v6362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:25 smithi121 bash[36902]: audit 2022-04-22T19:53:24.920199+0000 mon.a (mon.0) 8018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:25 smithi115 bash[45495]: cluster 2022-04-22T19:53:24.373450+0000 mgr.y (mgr.34107) 5053 : cluster [DBG] pgmap v6362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:26.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:25 smithi115 bash[45495]: audit 2022-04-22T19:53:24.920199+0000 mon.a (mon.0) 8018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:26.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:25 smithi115 bash[46934]: cluster 2022-04-22T19:53:24.373450+0000 mgr.y (mgr.34107) 5053 : cluster [DBG] pgmap v6362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:26.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:25 smithi115 bash[46934]: audit 2022-04-22T19:53:24.920199+0000 mon.a (mon.0) 8018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:26 smithi121 bash[36902]: audit 2022-04-22T19:53:26.296474+0000 mon.a (mon.0) 8019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:26 smithi115 bash[46934]: audit 2022-04-22T19:53:26.296474+0000 mon.a (mon.0) 8019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:27.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:26 smithi115 bash[45495]: audit 2022-04-22T19:53:26.296474+0000 mon.a (mon.0) 8019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:27.434 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 109s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 109s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:53:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 109s ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 109s ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 109s ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 109s ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 109s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 109s ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 109s ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:53:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 109s ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:53:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:53:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:53:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:53:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:53:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:53:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:27 smithi121 bash[36902]: cluster 2022-04-22T19:53:26.374560+0000 mgr.y (mgr.34107) 5054 : cluster [DBG] pgmap v6363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:27 smithi115 bash[46934]: cluster 2022-04-22T19:53:26.374560+0000 mgr.y (mgr.34107) 5054 : cluster [DBG] pgmap v6363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:27 smithi115 bash[45495]: cluster 2022-04-22T19:53:26.374560+0000 mgr.y (mgr.34107) 5054 : cluster [DBG] pgmap v6363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:53:28.161 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:53:28.162 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:53:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:28 smithi121 bash[36902]: audit 2022-04-22T19:53:27.411166+0000 mgr.y (mgr.34107) 5055 : audit [DBG] from='client.40662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:28 smithi121 bash[36902]: audit 2022-04-22T19:53:27.744295+0000 mgr.y (mgr.34107) 5056 : audit [DBG] from='client.50602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:28 smithi121 bash[36902]: audit 2022-04-22T19:53:27.755660+0000 mon.a (mon.0) 8020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:29.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:28 smithi121 bash[36902]: audit 2022-04-22T19:53:28.151559+0000 mon.a (mon.0) 8021 : audit [DBG] from='client.? 172.21.15.115:0/1884463784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:53:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[45495]: audit 2022-04-22T19:53:27.411166+0000 mgr.y (mgr.34107) 5055 : audit [DBG] from='client.40662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[45495]: audit 2022-04-22T19:53:27.744295+0000 mgr.y (mgr.34107) 5056 : audit [DBG] from='client.50602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[45495]: audit 2022-04-22T19:53:27.755660+0000 mon.a (mon.0) 8020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:29.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[45495]: audit 2022-04-22T19:53:28.151559+0000 mon.a (mon.0) 8021 : audit [DBG] from='client.? 172.21.15.115:0/1884463784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:53:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[46934]: audit 2022-04-22T19:53:27.411166+0000 mgr.y (mgr.34107) 5055 : audit [DBG] from='client.40662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[46934]: audit 2022-04-22T19:53:27.744295+0000 mgr.y (mgr.34107) 5056 : audit [DBG] from='client.50602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[46934]: audit 2022-04-22T19:53:27.755660+0000 mon.a (mon.0) 8020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:29.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:28 smithi115 bash[46934]: audit 2022-04-22T19:53:28.151559+0000 mon.a (mon.0) 8021 : audit [DBG] from='client.? 172.21.15.115:0/1884463784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:53:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:29 smithi121 bash[36902]: cluster 2022-04-22T19:53:28.375755+0000 mgr.y (mgr.34107) 5057 : cluster [DBG] pgmap v6364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:29 smithi115 bash[46934]: cluster 2022-04-22T19:53:28.375755+0000 mgr.y (mgr.34107) 5057 : cluster [DBG] pgmap v6364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:29 smithi115 bash[45495]: cluster 2022-04-22T19:53:28.375755+0000 mgr.y (mgr.34107) 5057 : cluster [DBG] pgmap v6364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:30 smithi115 bash[45495]: audit 2022-04-22T19:53:29.923677+0000 mon.a (mon.0) 8022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:30 smithi115 bash[46934]: audit 2022-04-22T19:53:29.923677+0000 mon.a (mon.0) 8022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:30 smithi121 bash[36902]: audit 2022-04-22T19:53:29.923677+0000 mon.a (mon.0) 8022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:31 smithi115 bash[45495]: cluster 2022-04-22T19:53:30.376749+0000 mgr.y (mgr.34107) 5058 : cluster [DBG] pgmap v6365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:31 smithi115 bash[45495]: audit 2022-04-22T19:53:31.299572+0000 mon.a (mon.0) 8023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:31 smithi115 bash[46934]: cluster 2022-04-22T19:53:30.376749+0000 mgr.y (mgr.34107) 5058 : cluster [DBG] pgmap v6365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:31 smithi115 bash[46934]: audit 2022-04-22T19:53:31.299572+0000 mon.a (mon.0) 8023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:31 smithi121 bash[36902]: cluster 2022-04-22T19:53:30.376749+0000 mgr.y (mgr.34107) 5058 : cluster [DBG] pgmap v6365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:31 smithi121 bash[36902]: audit 2022-04-22T19:53:31.299572+0000 mon.a (mon.0) 8023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:32.185Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:32 smithi115 bash[45495]: audit 2022-04-22T19:53:32.757867+0000 mon.a (mon.0) 8024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:32 smithi115 bash[46934]: audit 2022-04-22T19:53:32.757867+0000 mon.a (mon.0) 8024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:32 smithi121 bash[36902]: audit 2022-04-22T19:53:32.757867+0000 mon.a (mon.0) 8024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:33 smithi115 bash[45495]: cluster 2022-04-22T19:53:32.377825+0000 mgr.y (mgr.34107) 5059 : cluster [DBG] pgmap v6366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:33 smithi115 bash[46934]: cluster 2022-04-22T19:53:32.377825+0000 mgr.y (mgr.34107) 5059 : cluster [DBG] pgmap v6366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:33 smithi121 bash[36902]: cluster 2022-04-22T19:53:32.377825+0000 mgr.y (mgr.34107) 5059 : cluster [DBG] pgmap v6366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:34.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:34.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:36.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:35 smithi115 bash[45495]: cluster 2022-04-22T19:53:34.379018+0000 mgr.y (mgr.34107) 5060 : cluster [DBG] pgmap v6367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:35 smithi115 bash[45495]: audit 2022-04-22T19:53:34.927229+0000 mon.a (mon.0) 8025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:35 smithi115 bash[46934]: cluster 2022-04-22T19:53:34.379018+0000 mgr.y (mgr.34107) 5060 : cluster [DBG] pgmap v6367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:35 smithi115 bash[46934]: audit 2022-04-22T19:53:34.927229+0000 mon.a (mon.0) 8025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:35 smithi121 bash[36902]: cluster 2022-04-22T19:53:34.379018+0000 mgr.y (mgr.34107) 5060 : cluster [DBG] pgmap v6367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:35 smithi121 bash[36902]: audit 2022-04-22T19:53:34.927229+0000 mon.a (mon.0) 8025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:36 smithi115 bash[46934]: audit 2022-04-22T19:53:36.302475+0000 mon.a (mon.0) 8026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:37.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:36 smithi115 bash[45495]: audit 2022-04-22T19:53:36.302475+0000 mon.a (mon.0) 8026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:36 smithi121 bash[36902]: audit 2022-04-22T19:53:36.302475+0000 mon.a (mon.0) 8026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:37 smithi115 bash[46934]: cluster 2022-04-22T19:53:36.380140+0000 mgr.y (mgr.34107) 5061 : cluster [DBG] pgmap v6368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:37 smithi115 bash[46934]: audit 2022-04-22T19:53:37.760931+0000 mon.a (mon.0) 8027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:37 smithi115 bash[45495]: cluster 2022-04-22T19:53:36.380140+0000 mgr.y (mgr.34107) 5061 : cluster [DBG] pgmap v6368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:37 smithi115 bash[45495]: audit 2022-04-22T19:53:37.760931+0000 mon.a (mon.0) 8027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:37 smithi121 bash[36902]: cluster 2022-04-22T19:53:36.380140+0000 mgr.y (mgr.34107) 5061 : cluster [DBG] pgmap v6368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:37 smithi121 bash[36902]: audit 2022-04-22T19:53:37.760931+0000 mon.a (mon.0) 8027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:39 smithi121 bash[36902]: cluster 2022-04-22T19:53:38.381235+0000 mgr.y (mgr.34107) 5062 : cluster [DBG] pgmap v6369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:39 smithi121 bash[36902]: audit 2022-04-22T19:53:38.579576+0000 mon.a (mon.0) 8028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:39 smithi121 bash[36902]: audit 2022-04-22T19:53:38.582657+0000 mon.a (mon.0) 8029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:39 smithi121 bash[36902]: audit 2022-04-22T19:53:38.584678+0000 mon.a (mon.0) 8030 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:53:39.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:39 smithi121 bash[36902]: audit 2022-04-22T19:53:38.940255+0000 mon.a (mon.0) 8031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[45495]: cluster 2022-04-22T19:53:38.381235+0000 mgr.y (mgr.34107) 5062 : cluster [DBG] pgmap v6369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[45495]: audit 2022-04-22T19:53:38.579576+0000 mon.a (mon.0) 8028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[45495]: audit 2022-04-22T19:53:38.582657+0000 mon.a (mon.0) 8029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[45495]: audit 2022-04-22T19:53:38.584678+0000 mon.a (mon.0) 8030 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[45495]: audit 2022-04-22T19:53:38.940255+0000 mon.a (mon.0) 8031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[46934]: cluster 2022-04-22T19:53:38.381235+0000 mgr.y (mgr.34107) 5062 : cluster [DBG] pgmap v6369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[46934]: audit 2022-04-22T19:53:38.579576+0000 mon.a (mon.0) 8028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:53:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[46934]: audit 2022-04-22T19:53:38.582657+0000 mon.a (mon.0) 8029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:53:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[46934]: audit 2022-04-22T19:53:38.584678+0000 mon.a (mon.0) 8030 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:53:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:39 smithi115 bash[46934]: audit 2022-04-22T19:53:38.940255+0000 mon.a (mon.0) 8031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:53:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:40 smithi121 bash[36902]: audit 2022-04-22T19:53:39.930896+0000 mon.a (mon.0) 8032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:40.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:40 smithi115 bash[45495]: audit 2022-04-22T19:53:39.930896+0000 mon.a (mon.0) 8032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:40 smithi115 bash[46934]: audit 2022-04-22T19:53:39.930896+0000 mon.a (mon.0) 8032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:41 smithi121 bash[36902]: cluster 2022-04-22T19:53:40.382375+0000 mgr.y (mgr.34107) 5063 : cluster [DBG] pgmap v6370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:41 smithi121 bash[36902]: audit 2022-04-22T19:53:41.305856+0000 mon.a (mon.0) 8033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:41.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:41 smithi115 bash[45495]: cluster 2022-04-22T19:53:40.382375+0000 mgr.y (mgr.34107) 5063 : cluster [DBG] pgmap v6370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:41.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:41 smithi115 bash[45495]: audit 2022-04-22T19:53:41.305856+0000 mon.a (mon.0) 8033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:41.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:41 smithi115 bash[46934]: cluster 2022-04-22T19:53:40.382375+0000 mgr.y (mgr.34107) 5063 : cluster [DBG] pgmap v6370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:41.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:41 smithi115 bash[46934]: audit 2022-04-22T19:53:41.305856+0000 mon.a (mon.0) 8033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:42.186Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:43 smithi121 bash[36902]: cluster 2022-04-22T19:53:42.383599+0000 mgr.y (mgr.34107) 5064 : cluster [DBG] pgmap v6371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:43 smithi121 bash[36902]: audit 2022-04-22T19:53:42.764187+0000 mon.a (mon.0) 8034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:43.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:43 smithi115 bash[45495]: cluster 2022-04-22T19:53:42.383599+0000 mgr.y (mgr.34107) 5064 : cluster [DBG] pgmap v6371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:43.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:43 smithi115 bash[45495]: audit 2022-04-22T19:53:42.764187+0000 mon.a (mon.0) 8034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:43.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:43 smithi115 bash[46934]: cluster 2022-04-22T19:53:42.383599+0000 mgr.y (mgr.34107) 5064 : cluster [DBG] pgmap v6371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:43.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:43 smithi115 bash[46934]: audit 2022-04-22T19:53:42.764187+0000 mon.a (mon.0) 8034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:44.060Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:45 smithi121 bash[36902]: cluster 2022-04-22T19:53:44.384683+0000 mgr.y (mgr.34107) 5065 : cluster [DBG] pgmap v6372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:45 smithi121 bash[36902]: audit 2022-04-22T19:53:44.934569+0000 mon.a (mon.0) 8035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:45 smithi115 bash[45495]: cluster 2022-04-22T19:53:44.384683+0000 mgr.y (mgr.34107) 5065 : cluster [DBG] pgmap v6372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:45.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:45 smithi115 bash[45495]: audit 2022-04-22T19:53:44.934569+0000 mon.a (mon.0) 8035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:45 smithi115 bash[46934]: cluster 2022-04-22T19:53:44.384683+0000 mgr.y (mgr.34107) 5065 : cluster [DBG] pgmap v6372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:45.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:45 smithi115 bash[46934]: audit 2022-04-22T19:53:44.934569+0000 mon.a (mon.0) 8035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:46 smithi121 bash[36902]: audit 2022-04-22T19:53:46.309333+0000 mon.a (mon.0) 8036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:46.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:46 smithi115 bash[46934]: audit 2022-04-22T19:53:46.309333+0000 mon.a (mon.0) 8036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:46.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:46 smithi115 bash[45495]: audit 2022-04-22T19:53:46.309333+0000 mon.a (mon.0) 8036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:47 smithi121 bash[36902]: cluster 2022-04-22T19:53:46.385729+0000 mgr.y (mgr.34107) 5066 : cluster [DBG] pgmap v6373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:47 smithi115 bash[45495]: cluster 2022-04-22T19:53:46.385729+0000 mgr.y (mgr.34107) 5066 : cluster [DBG] pgmap v6373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:47 smithi115 bash[46934]: cluster 2022-04-22T19:53:46.385729+0000 mgr.y (mgr.34107) 5066 : cluster [DBG] pgmap v6373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:48 smithi121 bash[36902]: audit 2022-04-22T19:53:47.767374+0000 mon.a (mon.0) 8037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:48 smithi115 bash[45495]: audit 2022-04-22T19:53:47.767374+0000 mon.a (mon.0) 8037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:48 smithi115 bash[46934]: audit 2022-04-22T19:53:47.767374+0000 mon.a (mon.0) 8037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:49 smithi121 bash[36902]: cluster 2022-04-22T19:53:48.386883+0000 mgr.y (mgr.34107) 5067 : cluster [DBG] pgmap v6374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:49 smithi115 bash[45495]: cluster 2022-04-22T19:53:48.386883+0000 mgr.y (mgr.34107) 5067 : cluster [DBG] pgmap v6374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:49.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:49 smithi115 bash[46934]: cluster 2022-04-22T19:53:48.386883+0000 mgr.y (mgr.34107) 5067 : cluster [DBG] pgmap v6374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:50 smithi121 bash[36902]: audit 2022-04-22T19:53:49.937911+0000 mon.a (mon.0) 8038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:50 smithi115 bash[45495]: audit 2022-04-22T19:53:49.937911+0000 mon.a (mon.0) 8038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:50 smithi115 bash[46934]: audit 2022-04-22T19:53:49.937911+0000 mon.a (mon.0) 8038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:51 smithi121 bash[36902]: cluster 2022-04-22T19:53:50.387966+0000 mgr.y (mgr.34107) 5068 : cluster [DBG] pgmap v6375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:51 smithi121 bash[36902]: audit 2022-04-22T19:53:51.312546+0000 mon.a (mon.0) 8039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:51 smithi115 bash[45495]: cluster 2022-04-22T19:53:50.387966+0000 mgr.y (mgr.34107) 5068 : cluster [DBG] pgmap v6375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:51 smithi115 bash[45495]: audit 2022-04-22T19:53:51.312546+0000 mon.a (mon.0) 8039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:51 smithi115 bash[46934]: cluster 2022-04-22T19:53:50.387966+0000 mgr.y (mgr.34107) 5068 : cluster [DBG] pgmap v6375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:51.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:51 smithi115 bash[46934]: audit 2022-04-22T19:53:51.312546+0000 mon.a (mon.0) 8039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:52.186Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:53 smithi121 bash[36902]: cluster 2022-04-22T19:53:52.389017+0000 mgr.y (mgr.34107) 5069 : cluster [DBG] pgmap v6376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:53 smithi121 bash[36902]: audit 2022-04-22T19:53:52.770575+0000 mon.a (mon.0) 8040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:53 smithi115 bash[45495]: cluster 2022-04-22T19:53:52.389017+0000 mgr.y (mgr.34107) 5069 : cluster [DBG] pgmap v6376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:53 smithi115 bash[45495]: audit 2022-04-22T19:53:52.770575+0000 mon.a (mon.0) 8040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:53 smithi115 bash[46934]: cluster 2022-04-22T19:53:52.389017+0000 mgr.y (mgr.34107) 5069 : cluster [DBG] pgmap v6376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:53 smithi115 bash[46934]: audit 2022-04-22T19:53:52.770575+0000 mon.a (mon.0) 8040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:53:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:53:54.061Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:53:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:55 smithi121 bash[36902]: cluster 2022-04-22T19:53:54.390114+0000 mgr.y (mgr.34107) 5070 : cluster [DBG] pgmap v6377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:55 smithi121 bash[36902]: audit 2022-04-22T19:53:54.941589+0000 mon.a (mon.0) 8041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:55 smithi115 bash[46934]: cluster 2022-04-22T19:53:54.390114+0000 mgr.y (mgr.34107) 5070 : cluster [DBG] pgmap v6377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:55.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:55 smithi115 bash[46934]: audit 2022-04-22T19:53:54.941589+0000 mon.a (mon.0) 8041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:55 smithi115 bash[45495]: cluster 2022-04-22T19:53:54.390114+0000 mgr.y (mgr.34107) 5070 : cluster [DBG] pgmap v6377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:55.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:55 smithi115 bash[45495]: audit 2022-04-22T19:53:54.941589+0000 mon.a (mon.0) 8041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:53:56.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:56 smithi121 bash[36902]: audit 2022-04-22T19:53:56.315425+0000 mon.a (mon.0) 8042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:56.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:56 smithi115 bash[46934]: audit 2022-04-22T19:53:56.315425+0000 mon.a (mon.0) 8042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:56.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:56 smithi115 bash[45495]: audit 2022-04-22T19:53:56.315425+0000 mon.a (mon.0) 8042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:53:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:57 smithi121 bash[36902]: cluster 2022-04-22T19:53:56.391266+0000 mgr.y (mgr.34107) 5071 : cluster [DBG] pgmap v6378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:57.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:57 smithi115 bash[46934]: cluster 2022-04-22T19:53:56.391266+0000 mgr.y (mgr.34107) 5071 : cluster [DBG] pgmap v6378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:57.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:57 smithi115 bash[45495]: cluster 2022-04-22T19:53:56.391266+0000 mgr.y (mgr.34107) 5071 : cluster [DBG] pgmap v6378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:58.537 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:53:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:58 smithi121 bash[36902]: audit 2022-04-22T19:53:57.773902+0000 mon.a (mon.0) 8043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:58 smithi115 bash[46934]: audit 2022-04-22T19:53:57.773902+0000 mon.a (mon.0) 8043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:58 smithi115 bash[45495]: audit 2022-04-22T19:53:57.773902+0000 mon.a (mon.0) 8043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:53:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:53:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:53:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:53:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:53:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:53:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:53:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:59.278 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:53:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:53:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:59 smithi121 bash[36902]: cluster 2022-04-22T19:53:58.392380+0000 mgr.y (mgr.34107) 5072 : cluster [DBG] pgmap v6379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:59 smithi121 bash[36902]: audit 2022-04-22T19:53:58.511137+0000 mgr.y (mgr.34107) 5073 : audit [DBG] from='client.40680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:53:59 smithi121 bash[36902]: audit 2022-04-22T19:53:59.268485+0000 mon.c (mon.1) 215 : audit [DBG] from='client.? 172.21.15.115:0/3937223612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:53:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[45495]: cluster 2022-04-22T19:53:58.392380+0000 mgr.y (mgr.34107) 5072 : cluster [DBG] pgmap v6379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[45495]: audit 2022-04-22T19:53:58.511137+0000 mgr.y (mgr.34107) 5073 : audit [DBG] from='client.40680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[45495]: audit 2022-04-22T19:53:59.268485+0000 mon.c (mon.1) 215 : audit [DBG] from='client.? 172.21.15.115:0/3937223612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:53:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[46934]: cluster 2022-04-22T19:53:58.392380+0000 mgr.y (mgr.34107) 5072 : cluster [DBG] pgmap v6379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:53:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[46934]: audit 2022-04-22T19:53:58.511137+0000 mgr.y (mgr.34107) 5073 : audit [DBG] from='client.40680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:53:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:53:59 smithi115 bash[46934]: audit 2022-04-22T19:53:59.268485+0000 mon.c (mon.1) 215 : audit [DBG] from='client.? 172.21.15.115:0/3937223612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:54:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:00 smithi121 bash[36902]: audit 2022-04-22T19:53:58.850466+0000 mgr.y (mgr.34107) 5074 : audit [DBG] from='client.40686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:00 smithi121 bash[36902]: audit 2022-04-22T19:53:59.945297+0000 mon.a (mon.0) 8044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:00 smithi115 bash[45495]: audit 2022-04-22T19:53:58.850466+0000 mgr.y (mgr.34107) 5074 : audit [DBG] from='client.40686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:00 smithi115 bash[45495]: audit 2022-04-22T19:53:59.945297+0000 mon.a (mon.0) 8044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:00 smithi115 bash[46934]: audit 2022-04-22T19:53:58.850466+0000 mgr.y (mgr.34107) 5074 : audit [DBG] from='client.40686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:00.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:00 smithi115 bash[46934]: audit 2022-04-22T19:53:59.945297+0000 mon.a (mon.0) 8044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:01 smithi121 bash[36902]: cluster 2022-04-22T19:54:00.393412+0000 mgr.y (mgr.34107) 5075 : cluster [DBG] pgmap v6380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:01 smithi121 bash[36902]: audit 2022-04-22T19:54:01.318343+0000 mon.a (mon.0) 8045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:01 smithi115 bash[45495]: cluster 2022-04-22T19:54:00.393412+0000 mgr.y (mgr.34107) 5075 : cluster [DBG] pgmap v6380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:01.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:01 smithi115 bash[45495]: audit 2022-04-22T19:54:01.318343+0000 mon.a (mon.0) 8045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:01.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:01 smithi115 bash[46934]: cluster 2022-04-22T19:54:00.393412+0000 mgr.y (mgr.34107) 5075 : cluster [DBG] pgmap v6380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:01.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:01 smithi115 bash[46934]: audit 2022-04-22T19:54:01.318343+0000 mon.a (mon.0) 8045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:02.186Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:03.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:03 smithi115 bash[46934]: cluster 2022-04-22T19:54:02.394481+0000 mgr.y (mgr.34107) 5076 : cluster [DBG] pgmap v6381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:03 smithi115 bash[46934]: audit 2022-04-22T19:54:02.776627+0000 mon.a (mon.0) 8046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:03.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:03 smithi115 bash[45495]: cluster 2022-04-22T19:54:02.394481+0000 mgr.y (mgr.34107) 5076 : cluster [DBG] pgmap v6381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:03.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:03 smithi115 bash[45495]: audit 2022-04-22T19:54:02.776627+0000 mon.a (mon.0) 8046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:03 smithi121 bash[36902]: cluster 2022-04-22T19:54:02.394481+0000 mgr.y (mgr.34107) 5076 : cluster [DBG] pgmap v6381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:03 smithi121 bash[36902]: audit 2022-04-22T19:54:02.776627+0000 mon.a (mon.0) 8046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:04.061Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:05.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:05 smithi115 bash[45495]: cluster 2022-04-22T19:54:04.395585+0000 mgr.y (mgr.34107) 5077 : cluster [DBG] pgmap v6382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:05.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:05 smithi115 bash[45495]: audit 2022-04-22T19:54:04.948888+0000 mon.a (mon.0) 8047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:05 smithi115 bash[46934]: cluster 2022-04-22T19:54:04.395585+0000 mgr.y (mgr.34107) 5077 : cluster [DBG] pgmap v6382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:05.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:05 smithi115 bash[46934]: audit 2022-04-22T19:54:04.948888+0000 mon.a (mon.0) 8047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:05 smithi121 bash[36902]: cluster 2022-04-22T19:54:04.395585+0000 mgr.y (mgr.34107) 5077 : cluster [DBG] pgmap v6382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:05 smithi121 bash[36902]: audit 2022-04-22T19:54:04.948888+0000 mon.a (mon.0) 8047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:06.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:06 smithi115 bash[45495]: audit 2022-04-22T19:54:06.321281+0000 mon.a (mon.0) 8048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:06.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:06 smithi115 bash[46934]: audit 2022-04-22T19:54:06.321281+0000 mon.a (mon.0) 8048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:06 smithi121 bash[36902]: audit 2022-04-22T19:54:06.321281+0000 mon.a (mon.0) 8048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:07 smithi115 bash[45495]: cluster 2022-04-22T19:54:06.396576+0000 mgr.y (mgr.34107) 5078 : cluster [DBG] pgmap v6383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:07.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:07 smithi115 bash[46934]: cluster 2022-04-22T19:54:06.396576+0000 mgr.y (mgr.34107) 5078 : cluster [DBG] pgmap v6383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:07 smithi121 bash[36902]: cluster 2022-04-22T19:54:06.396576+0000 mgr.y (mgr.34107) 5078 : cluster [DBG] pgmap v6383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:08 smithi115 bash[46934]: audit 2022-04-22T19:54:07.779711+0000 mon.a (mon.0) 8049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:08.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:08 smithi115 bash[45495]: audit 2022-04-22T19:54:07.779711+0000 mon.a (mon.0) 8049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:08 smithi121 bash[36902]: audit 2022-04-22T19:54:07.779711+0000 mon.a (mon.0) 8049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:09 smithi115 bash[46934]: cluster 2022-04-22T19:54:08.397642+0000 mgr.y (mgr.34107) 5079 : cluster [DBG] pgmap v6384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:09.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:09 smithi115 bash[45495]: cluster 2022-04-22T19:54:08.397642+0000 mgr.y (mgr.34107) 5079 : cluster [DBG] pgmap v6384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:09 smithi121 bash[36902]: cluster 2022-04-22T19:54:08.397642+0000 mgr.y (mgr.34107) 5079 : cluster [DBG] pgmap v6384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:10.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:10 smithi115 bash[46934]: audit 2022-04-22T19:54:09.952629+0000 mon.a (mon.0) 8050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:10.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:10 smithi115 bash[45495]: audit 2022-04-22T19:54:09.952629+0000 mon.a (mon.0) 8050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:10 smithi121 bash[36902]: audit 2022-04-22T19:54:09.952629+0000 mon.a (mon.0) 8050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:11 smithi121 bash[36902]: cluster 2022-04-22T19:54:10.398633+0000 mgr.y (mgr.34107) 5080 : cluster [DBG] pgmap v6385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:11 smithi121 bash[36902]: audit 2022-04-22T19:54:11.324362+0000 mon.a (mon.0) 8051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:11 smithi115 bash[46934]: cluster 2022-04-22T19:54:10.398633+0000 mgr.y (mgr.34107) 5080 : cluster [DBG] pgmap v6385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:11 smithi115 bash[46934]: audit 2022-04-22T19:54:11.324362+0000 mon.a (mon.0) 8051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:11 smithi115 bash[45495]: cluster 2022-04-22T19:54:10.398633+0000 mgr.y (mgr.34107) 5080 : cluster [DBG] pgmap v6385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:11 smithi115 bash[45495]: audit 2022-04-22T19:54:11.324362+0000 mon.a (mon.0) 8051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:12.188Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:13 smithi121 bash[36902]: cluster 2022-04-22T19:54:12.399699+0000 mgr.y (mgr.34107) 5081 : cluster [DBG] pgmap v6386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:13 smithi121 bash[36902]: audit 2022-04-22T19:54:12.782788+0000 mon.a (mon.0) 8052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:13 smithi115 bash[45495]: cluster 2022-04-22T19:54:12.399699+0000 mgr.y (mgr.34107) 5081 : cluster [DBG] pgmap v6386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:13 smithi115 bash[45495]: audit 2022-04-22T19:54:12.782788+0000 mon.a (mon.0) 8052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:13 smithi115 bash[46934]: cluster 2022-04-22T19:54:12.399699+0000 mgr.y (mgr.34107) 5081 : cluster [DBG] pgmap v6386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:13 smithi115 bash[46934]: audit 2022-04-22T19:54:12.782788+0000 mon.a (mon.0) 8052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:14.062Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:15 smithi121 bash[36902]: cluster 2022-04-22T19:54:14.400858+0000 mgr.y (mgr.34107) 5082 : cluster [DBG] pgmap v6387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:15 smithi121 bash[36902]: audit 2022-04-22T19:54:14.956295+0000 mon.a (mon.0) 8053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:15 smithi121 bash[36902]: audit 2022-04-22T19:54:14.977461+0000 mon.a (mon.0) 8054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:54:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:15 smithi121 bash[36902]: audit 2022-04-22T19:54:14.978250+0000 mon.a (mon.0) 8055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:54:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[45495]: cluster 2022-04-22T19:54:14.400858+0000 mgr.y (mgr.34107) 5082 : cluster [DBG] pgmap v6387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[45495]: audit 2022-04-22T19:54:14.956295+0000 mon.a (mon.0) 8053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[45495]: audit 2022-04-22T19:54:14.977461+0000 mon.a (mon.0) 8054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:54:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[45495]: audit 2022-04-22T19:54:14.978250+0000 mon.a (mon.0) 8055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:54:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[46934]: cluster 2022-04-22T19:54:14.400858+0000 mgr.y (mgr.34107) 5082 : cluster [DBG] pgmap v6387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[46934]: audit 2022-04-22T19:54:14.956295+0000 mon.a (mon.0) 8053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[46934]: audit 2022-04-22T19:54:14.977461+0000 mon.a (mon.0) 8054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:54:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:15 smithi115 bash[46934]: audit 2022-04-22T19:54:14.978250+0000 mon.a (mon.0) 8055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:54:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:16 smithi121 bash[36902]: audit 2022-04-22T19:54:16.327413+0000 mon.a (mon.0) 8056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:16 smithi115 bash[45495]: audit 2022-04-22T19:54:16.327413+0000 mon.a (mon.0) 8056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:16 smithi115 bash[46934]: audit 2022-04-22T19:54:16.327413+0000 mon.a (mon.0) 8056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:17 smithi121 bash[36902]: cluster 2022-04-22T19:54:16.401951+0000 mgr.y (mgr.34107) 5083 : cluster [DBG] pgmap v6388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:17 smithi115 bash[45495]: cluster 2022-04-22T19:54:16.401951+0000 mgr.y (mgr.34107) 5083 : cluster [DBG] pgmap v6388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:17 smithi115 bash[46934]: cluster 2022-04-22T19:54:16.401951+0000 mgr.y (mgr.34107) 5083 : cluster [DBG] pgmap v6388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:18 smithi121 bash[36902]: audit 2022-04-22T19:54:17.785776+0000 mon.a (mon.0) 8057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:18 smithi115 bash[45495]: audit 2022-04-22T19:54:17.785776+0000 mon.a (mon.0) 8057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:18 smithi115 bash[46934]: audit 2022-04-22T19:54:17.785776+0000 mon.a (mon.0) 8057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:19 smithi121 bash[36902]: cluster 2022-04-22T19:54:18.403201+0000 mgr.y (mgr.34107) 5084 : cluster [DBG] pgmap v6389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:20.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:19 smithi115 bash[45495]: cluster 2022-04-22T19:54:18.403201+0000 mgr.y (mgr.34107) 5084 : cluster [DBG] pgmap v6389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:20.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:19 smithi115 bash[46934]: cluster 2022-04-22T19:54:18.403201+0000 mgr.y (mgr.34107) 5084 : cluster [DBG] pgmap v6389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:20 smithi121 bash[36902]: audit 2022-04-22T19:54:19.959746+0000 mon.a (mon.0) 8058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:20 smithi115 bash[45495]: audit 2022-04-22T19:54:19.959746+0000 mon.a (mon.0) 8058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:20 smithi115 bash[46934]: audit 2022-04-22T19:54:19.959746+0000 mon.a (mon.0) 8058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:21 smithi121 bash[36902]: cluster 2022-04-22T19:54:20.404312+0000 mgr.y (mgr.34107) 5085 : cluster [DBG] pgmap v6390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:21 smithi121 bash[36902]: audit 2022-04-22T19:54:21.330657+0000 mon.a (mon.0) 8059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:21 smithi115 bash[45495]: cluster 2022-04-22T19:54:20.404312+0000 mgr.y (mgr.34107) 5085 : cluster [DBG] pgmap v6390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:21 smithi115 bash[45495]: audit 2022-04-22T19:54:21.330657+0000 mon.a (mon.0) 8059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:21 smithi115 bash[46934]: cluster 2022-04-22T19:54:20.404312+0000 mgr.y (mgr.34107) 5085 : cluster [DBG] pgmap v6390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:21 smithi115 bash[46934]: audit 2022-04-22T19:54:21.330657+0000 mon.a (mon.0) 8059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:22.189Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:23 smithi121 bash[36902]: cluster 2022-04-22T19:54:22.405397+0000 mgr.y (mgr.34107) 5086 : cluster [DBG] pgmap v6391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:23 smithi121 bash[36902]: audit 2022-04-22T19:54:22.789000+0000 mon.a (mon.0) 8060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:23 smithi115 bash[46934]: cluster 2022-04-22T19:54:22.405397+0000 mgr.y (mgr.34107) 5086 : cluster [DBG] pgmap v6391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:23 smithi115 bash[46934]: audit 2022-04-22T19:54:22.789000+0000 mon.a (mon.0) 8060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:23 smithi115 bash[45495]: cluster 2022-04-22T19:54:22.405397+0000 mgr.y (mgr.34107) 5086 : cluster [DBG] pgmap v6391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:23 smithi115 bash[45495]: audit 2022-04-22T19:54:22.789000+0000 mon.a (mon.0) 8060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:24.062Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:25 smithi121 bash[36902]: cluster 2022-04-22T19:54:24.406546+0000 mgr.y (mgr.34107) 5087 : cluster [DBG] pgmap v6392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:25 smithi121 bash[36902]: audit 2022-04-22T19:54:24.964570+0000 mon.a (mon.0) 8061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:25 smithi115 bash[46934]: cluster 2022-04-22T19:54:24.406546+0000 mgr.y (mgr.34107) 5087 : cluster [DBG] pgmap v6392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:25 smithi115 bash[46934]: audit 2022-04-22T19:54:24.964570+0000 mon.a (mon.0) 8061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:25 smithi115 bash[45495]: cluster 2022-04-22T19:54:24.406546+0000 mgr.y (mgr.34107) 5087 : cluster [DBG] pgmap v6392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:25 smithi115 bash[45495]: audit 2022-04-22T19:54:24.964570+0000 mon.a (mon.0) 8061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:26 smithi121 bash[36902]: audit 2022-04-22T19:54:26.333785+0000 mon.a (mon.0) 8062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:26 smithi115 bash[46934]: audit 2022-04-22T19:54:26.333785+0000 mon.a (mon.0) 8062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:26 smithi115 bash[45495]: audit 2022-04-22T19:54:26.333785+0000 mon.a (mon.0) 8062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:27 smithi121 bash[36902]: cluster 2022-04-22T19:54:26.407494+0000 mgr.y (mgr.34107) 5088 : cluster [DBG] pgmap v6393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:27 smithi115 bash[46934]: cluster 2022-04-22T19:54:26.407494+0000 mgr.y (mgr.34107) 5088 : cluster [DBG] pgmap v6393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:27 smithi115 bash[45495]: cluster 2022-04-22T19:54:26.407494+0000 mgr.y (mgr.34107) 5088 : cluster [DBG] pgmap v6393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:28 smithi121 bash[36902]: audit 2022-04-22T19:54:27.792132+0000 mon.a (mon.0) 8063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:28 smithi115 bash[45495]: audit 2022-04-22T19:54:27.792132+0000 mon.a (mon.0) 8063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:28 smithi115 bash[46934]: audit 2022-04-22T19:54:27.792132+0000 mon.a (mon.0) 8063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:29.661 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:54:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:54:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:54:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:54:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:54:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:54:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:54:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:54:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:29 smithi121 bash[36902]: cluster 2022-04-22T19:54:28.408651+0000 mgr.y (mgr.34107) 5089 : cluster [DBG] pgmap v6394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:29 smithi115 bash[46934]: cluster 2022-04-22T19:54:28.408651+0000 mgr.y (mgr.34107) 5089 : cluster [DBG] pgmap v6394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:29 smithi115 bash[45495]: cluster 2022-04-22T19:54:28.408651+0000 mgr.y (mgr.34107) 5089 : cluster [DBG] pgmap v6394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:30.409 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:54:30.409 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:54:30.410 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:54:30.411 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:54:30.412 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:54:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:30 smithi121 bash[36902]: audit 2022-04-22T19:54:29.636078+0000 mgr.y (mgr.34107) 5090 : audit [DBG] from='client.50632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:30 smithi121 bash[36902]: audit 2022-04-22T19:54:29.968159+0000 mon.a (mon.0) 8064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:30 smithi121 bash[36902]: audit 2022-04-22T19:54:30.399636+0000 mon.c (mon.1) 216 : audit [DBG] from='client.? 172.21.15.115:0/3572470522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:54:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[45495]: audit 2022-04-22T19:54:29.636078+0000 mgr.y (mgr.34107) 5090 : audit [DBG] from='client.50632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[45495]: audit 2022-04-22T19:54:29.968159+0000 mon.a (mon.0) 8064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:31.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[45495]: audit 2022-04-22T19:54:30.399636+0000 mon.c (mon.1) 216 : audit [DBG] from='client.? 172.21.15.115:0/3572470522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:54:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[46934]: audit 2022-04-22T19:54:29.636078+0000 mgr.y (mgr.34107) 5090 : audit [DBG] from='client.50632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[46934]: audit 2022-04-22T19:54:29.968159+0000 mon.a (mon.0) 8064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:30 smithi115 bash[46934]: audit 2022-04-22T19:54:30.399636+0000 mon.c (mon.1) 216 : audit [DBG] from='client.? 172.21.15.115:0/3572470522' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:54:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:31 smithi121 bash[36902]: audit 2022-04-22T19:54:29.973069+0000 mgr.y (mgr.34107) 5091 : audit [DBG] from='client.50638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:31 smithi121 bash[36902]: cluster 2022-04-22T19:54:30.409618+0000 mgr.y (mgr.34107) 5092 : cluster [DBG] pgmap v6395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:31 smithi121 bash[36902]: audit 2022-04-22T19:54:31.336887+0000 mon.a (mon.0) 8065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[45495]: audit 2022-04-22T19:54:29.973069+0000 mgr.y (mgr.34107) 5091 : audit [DBG] from='client.50638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[45495]: cluster 2022-04-22T19:54:30.409618+0000 mgr.y (mgr.34107) 5092 : cluster [DBG] pgmap v6395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[45495]: audit 2022-04-22T19:54:31.336887+0000 mon.a (mon.0) 8065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[46934]: audit 2022-04-22T19:54:29.973069+0000 mgr.y (mgr.34107) 5091 : audit [DBG] from='client.50638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:54:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[46934]: cluster 2022-04-22T19:54:30.409618+0000 mgr.y (mgr.34107) 5092 : cluster [DBG] pgmap v6395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:31 smithi115 bash[46934]: audit 2022-04-22T19:54:31.336887+0000 mon.a (mon.0) 8065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:32.191Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:33 smithi121 bash[36902]: cluster 2022-04-22T19:54:32.410663+0000 mgr.y (mgr.34107) 5093 : cluster [DBG] pgmap v6396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:33 smithi121 bash[36902]: audit 2022-04-22T19:54:32.795204+0000 mon.a (mon.0) 8066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:33 smithi115 bash[46934]: cluster 2022-04-22T19:54:32.410663+0000 mgr.y (mgr.34107) 5093 : cluster [DBG] pgmap v6396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:33 smithi115 bash[46934]: audit 2022-04-22T19:54:32.795204+0000 mon.a (mon.0) 8066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:33 smithi115 bash[45495]: cluster 2022-04-22T19:54:32.410663+0000 mgr.y (mgr.34107) 5093 : cluster [DBG] pgmap v6396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:33 smithi115 bash[45495]: audit 2022-04-22T19:54:32.795204+0000 mon.a (mon.0) 8066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:34.063Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:35 smithi121 bash[36902]: cluster 2022-04-22T19:54:34.411812+0000 mgr.y (mgr.34107) 5094 : cluster [DBG] pgmap v6397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:35 smithi121 bash[36902]: audit 2022-04-22T19:54:34.971495+0000 mon.a (mon.0) 8067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:36.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:35 smithi115 bash[45495]: cluster 2022-04-22T19:54:34.411812+0000 mgr.y (mgr.34107) 5094 : cluster [DBG] pgmap v6397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:36.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:35 smithi115 bash[45495]: audit 2022-04-22T19:54:34.971495+0000 mon.a (mon.0) 8067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:35 smithi115 bash[46934]: cluster 2022-04-22T19:54:34.411812+0000 mgr.y (mgr.34107) 5094 : cluster [DBG] pgmap v6397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:35 smithi115 bash[46934]: audit 2022-04-22T19:54:34.971495+0000 mon.a (mon.0) 8067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:36 smithi121 bash[36902]: audit 2022-04-22T19:54:36.339905+0000 mon.a (mon.0) 8068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:36 smithi115 bash[45495]: audit 2022-04-22T19:54:36.339905+0000 mon.a (mon.0) 8068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:37.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:36 smithi115 bash[46934]: audit 2022-04-22T19:54:36.339905+0000 mon.a (mon.0) 8068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:37 smithi121 bash[36902]: cluster 2022-04-22T19:54:36.412774+0000 mgr.y (mgr.34107) 5095 : cluster [DBG] pgmap v6398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:37 smithi115 bash[45495]: cluster 2022-04-22T19:54:36.412774+0000 mgr.y (mgr.34107) 5095 : cluster [DBG] pgmap v6398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:37 smithi115 bash[46934]: cluster 2022-04-22T19:54:36.412774+0000 mgr.y (mgr.34107) 5095 : cluster [DBG] pgmap v6398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:38.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:38 smithi115 bash[45495]: audit 2022-04-22T19:54:37.798404+0000 mon.a (mon.0) 8069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:38.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:38 smithi115 bash[46934]: audit 2022-04-22T19:54:37.798404+0000 mon.a (mon.0) 8069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:38.984 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:38 smithi121 bash[36902]: audit 2022-04-22T19:54:37.798404+0000 mon.a (mon.0) 8069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:39 smithi121 bash[36902]: cluster 2022-04-22T19:54:38.413932+0000 mgr.y (mgr.34107) 5096 : cluster [DBG] pgmap v6399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:40.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:39 smithi121 bash[36902]: audit 2022-04-22T19:54:38.950126+0000 mon.a (mon.0) 8070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:54:40.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:39 smithi121 bash[36902]: audit 2022-04-22T19:54:38.952966+0000 mon.a (mon.0) 8071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:54:40.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:39 smithi121 bash[36902]: audit 2022-04-22T19:54:38.954773+0000 mon.a (mon.0) 8072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:54:40.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:39 smithi121 bash[36902]: audit 2022-04-22T19:54:39.316672+0000 mon.a (mon.0) 8073 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:54:40.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[45495]: cluster 2022-04-22T19:54:38.413932+0000 mgr.y (mgr.34107) 5096 : cluster [DBG] pgmap v6399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:40.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[45495]: audit 2022-04-22T19:54:38.950126+0000 mon.a (mon.0) 8070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:54:40.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[45495]: audit 2022-04-22T19:54:38.952966+0000 mon.a (mon.0) 8071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:54:40.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[45495]: audit 2022-04-22T19:54:38.954773+0000 mon.a (mon.0) 8072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:54:40.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[45495]: audit 2022-04-22T19:54:39.316672+0000 mon.a (mon.0) 8073 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:54:40.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[46934]: cluster 2022-04-22T19:54:38.413932+0000 mgr.y (mgr.34107) 5096 : cluster [DBG] pgmap v6399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:40.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[46934]: audit 2022-04-22T19:54:38.950126+0000 mon.a (mon.0) 8070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:54:40.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[46934]: audit 2022-04-22T19:54:38.952966+0000 mon.a (mon.0) 8071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:54:40.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[46934]: audit 2022-04-22T19:54:38.954773+0000 mon.a (mon.0) 8072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:54:40.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:39 smithi115 bash[46934]: audit 2022-04-22T19:54:39.316672+0000 mon.a (mon.0) 8073 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:54:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:40 smithi121 bash[36902]: audit 2022-04-22T19:54:39.975157+0000 mon.a (mon.0) 8074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:40 smithi115 bash[45495]: audit 2022-04-22T19:54:39.975157+0000 mon.a (mon.0) 8074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:40 smithi115 bash[46934]: audit 2022-04-22T19:54:39.975157+0000 mon.a (mon.0) 8074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:41 smithi121 bash[36902]: cluster 2022-04-22T19:54:40.414979+0000 mgr.y (mgr.34107) 5097 : cluster [DBG] pgmap v6400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:41 smithi121 bash[36902]: audit 2022-04-22T19:54:41.342652+0000 mon.a (mon.0) 8075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:41 smithi115 bash[45495]: cluster 2022-04-22T19:54:40.414979+0000 mgr.y (mgr.34107) 5097 : cluster [DBG] pgmap v6400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:41 smithi115 bash[45495]: audit 2022-04-22T19:54:41.342652+0000 mon.a (mon.0) 8075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:41 smithi115 bash[46934]: cluster 2022-04-22T19:54:40.414979+0000 mgr.y (mgr.34107) 5097 : cluster [DBG] pgmap v6400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:41 smithi115 bash[46934]: audit 2022-04-22T19:54:41.342652+0000 mon.a (mon.0) 8075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:42.191Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:43 smithi121 bash[36902]: cluster 2022-04-22T19:54:42.416083+0000 mgr.y (mgr.34107) 5098 : cluster [DBG] pgmap v6401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:43 smithi121 bash[36902]: audit 2022-04-22T19:54:42.801447+0000 mon.a (mon.0) 8076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:43 smithi115 bash[45495]: cluster 2022-04-22T19:54:42.416083+0000 mgr.y (mgr.34107) 5098 : cluster [DBG] pgmap v6401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:43 smithi115 bash[45495]: audit 2022-04-22T19:54:42.801447+0000 mon.a (mon.0) 8076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:43 smithi115 bash[46934]: cluster 2022-04-22T19:54:42.416083+0000 mgr.y (mgr.34107) 5098 : cluster [DBG] pgmap v6401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:43 smithi115 bash[46934]: audit 2022-04-22T19:54:42.801447+0000 mon.a (mon.0) 8076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:44.064Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:45 smithi121 bash[36902]: cluster 2022-04-22T19:54:44.417200+0000 mgr.y (mgr.34107) 5099 : cluster [DBG] pgmap v6402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:45 smithi121 bash[36902]: audit 2022-04-22T19:54:44.978785+0000 mon.a (mon.0) 8077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:45 smithi115 bash[45495]: cluster 2022-04-22T19:54:44.417200+0000 mgr.y (mgr.34107) 5099 : cluster [DBG] pgmap v6402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:45 smithi115 bash[45495]: audit 2022-04-22T19:54:44.978785+0000 mon.a (mon.0) 8077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:46.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:45 smithi115 bash[46934]: cluster 2022-04-22T19:54:44.417200+0000 mgr.y (mgr.34107) 5099 : cluster [DBG] pgmap v6402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:45 smithi115 bash[46934]: audit 2022-04-22T19:54:44.978785+0000 mon.a (mon.0) 8077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:46 smithi121 bash[36902]: audit 2022-04-22T19:54:46.345745+0000 mon.a (mon.0) 8078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:46 smithi115 bash[45495]: audit 2022-04-22T19:54:46.345745+0000 mon.a (mon.0) 8078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:46 smithi115 bash[46934]: audit 2022-04-22T19:54:46.345745+0000 mon.a (mon.0) 8078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:47 smithi121 bash[36902]: cluster 2022-04-22T19:54:46.418259+0000 mgr.y (mgr.34107) 5100 : cluster [DBG] pgmap v6403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:47 smithi115 bash[45495]: cluster 2022-04-22T19:54:46.418259+0000 mgr.y (mgr.34107) 5100 : cluster [DBG] pgmap v6403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:47 smithi115 bash[46934]: cluster 2022-04-22T19:54:46.418259+0000 mgr.y (mgr.34107) 5100 : cluster [DBG] pgmap v6403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:48 smithi121 bash[36902]: audit 2022-04-22T19:54:47.804504+0000 mon.a (mon.0) 8079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:48 smithi115 bash[45495]: audit 2022-04-22T19:54:47.804504+0000 mon.a (mon.0) 8079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:48 smithi115 bash[46934]: audit 2022-04-22T19:54:47.804504+0000 mon.a (mon.0) 8079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:49 smithi121 bash[36902]: cluster 2022-04-22T19:54:48.419469+0000 mgr.y (mgr.34107) 5101 : cluster [DBG] pgmap v6404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:49 smithi115 bash[45495]: cluster 2022-04-22T19:54:48.419469+0000 mgr.y (mgr.34107) 5101 : cluster [DBG] pgmap v6404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:50.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:49 smithi115 bash[46934]: cluster 2022-04-22T19:54:48.419469+0000 mgr.y (mgr.34107) 5101 : cluster [DBG] pgmap v6404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:50 smithi121 bash[36902]: audit 2022-04-22T19:54:49.982395+0000 mon.a (mon.0) 8080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:50 smithi115 bash[45495]: audit 2022-04-22T19:54:49.982395+0000 mon.a (mon.0) 8080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:51.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:50 smithi115 bash[46934]: audit 2022-04-22T19:54:49.982395+0000 mon.a (mon.0) 8080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:51 smithi121 bash[36902]: cluster 2022-04-22T19:54:50.420549+0000 mgr.y (mgr.34107) 5102 : cluster [DBG] pgmap v6405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:51 smithi121 bash[36902]: audit 2022-04-22T19:54:51.348695+0000 mon.a (mon.0) 8081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:51 smithi115 bash[46934]: cluster 2022-04-22T19:54:50.420549+0000 mgr.y (mgr.34107) 5102 : cluster [DBG] pgmap v6405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:51 smithi115 bash[46934]: audit 2022-04-22T19:54:51.348695+0000 mon.a (mon.0) 8081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:51 smithi115 bash[45495]: cluster 2022-04-22T19:54:50.420549+0000 mgr.y (mgr.34107) 5102 : cluster [DBG] pgmap v6405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:51 smithi115 bash[45495]: audit 2022-04-22T19:54:51.348695+0000 mon.a (mon.0) 8081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:52.193Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:53 smithi121 bash[36902]: cluster 2022-04-22T19:54:52.421637+0000 mgr.y (mgr.34107) 5103 : cluster [DBG] pgmap v6406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:53 smithi121 bash[36902]: audit 2022-04-22T19:54:52.807383+0000 mon.a (mon.0) 8082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:53 smithi115 bash[45495]: cluster 2022-04-22T19:54:52.421637+0000 mgr.y (mgr.34107) 5103 : cluster [DBG] pgmap v6406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:54.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:53 smithi115 bash[45495]: audit 2022-04-22T19:54:52.807383+0000 mon.a (mon.0) 8082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:53 smithi115 bash[46934]: cluster 2022-04-22T19:54:52.421637+0000 mgr.y (mgr.34107) 5103 : cluster [DBG] pgmap v6406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:53 smithi115 bash[46934]: audit 2022-04-22T19:54:52.807383+0000 mon.a (mon.0) 8082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:54:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:54:54.066Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:54:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:55 smithi121 bash[36902]: cluster 2022-04-22T19:54:54.422806+0000 mgr.y (mgr.34107) 5104 : cluster [DBG] pgmap v6407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:55 smithi121 bash[36902]: audit 2022-04-22T19:54:54.986014+0000 mon.a (mon.0) 8083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:55 smithi115 bash[45495]: cluster 2022-04-22T19:54:54.422806+0000 mgr.y (mgr.34107) 5104 : cluster [DBG] pgmap v6407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:55 smithi115 bash[45495]: audit 2022-04-22T19:54:54.986014+0000 mon.a (mon.0) 8083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:55 smithi115 bash[46934]: cluster 2022-04-22T19:54:54.422806+0000 mgr.y (mgr.34107) 5104 : cluster [DBG] pgmap v6407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:55 smithi115 bash[46934]: audit 2022-04-22T19:54:54.986014+0000 mon.a (mon.0) 8083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:54:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:56 smithi121 bash[36902]: audit 2022-04-22T19:54:56.351877+0000 mon.a (mon.0) 8084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:56 smithi115 bash[45495]: audit 2022-04-22T19:54:56.351877+0000 mon.a (mon.0) 8084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:56 smithi115 bash[46934]: audit 2022-04-22T19:54:56.351877+0000 mon.a (mon.0) 8084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:54:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:57 smithi121 bash[36902]: cluster 2022-04-22T19:54:56.424280+0000 mgr.y (mgr.34107) 5105 : cluster [DBG] pgmap v6408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:57 smithi115 bash[45495]: cluster 2022-04-22T19:54:56.424280+0000 mgr.y (mgr.34107) 5105 : cluster [DBG] pgmap v6408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:57 smithi115 bash[46934]: cluster 2022-04-22T19:54:56.424280+0000 mgr.y (mgr.34107) 5105 : cluster [DBG] pgmap v6408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:54:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:58 smithi121 bash[36902]: audit 2022-04-22T19:54:57.810420+0000 mon.a (mon.0) 8085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:58 smithi115 bash[45495]: audit 2022-04-22T19:54:57.810420+0000 mon.a (mon.0) 8085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:54:59.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:58 smithi115 bash[46934]: audit 2022-04-22T19:54:57.810420+0000 mon.a (mon.0) 8085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:54:59 smithi121 bash[36902]: cluster 2022-04-22T19:54:58.425531+0000 mgr.y (mgr.34107) 5106 : cluster [DBG] pgmap v6409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:54:59 smithi115 bash[45495]: cluster 2022-04-22T19:54:58.425531+0000 mgr.y (mgr.34107) 5106 : cluster [DBG] pgmap v6409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:54:59 smithi115 bash[46934]: cluster 2022-04-22T19:54:58.425531+0000 mgr.y (mgr.34107) 5106 : cluster [DBG] pgmap v6409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:00.786 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:55:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:00 smithi121 bash[36902]: audit 2022-04-22T19:54:59.989876+0000 mon.a (mon.0) 8086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:00 smithi115 bash[45495]: audit 2022-04-22T19:54:59.989876+0000 mon.a (mon.0) 8086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:00 smithi115 bash[46934]: audit 2022-04-22T19:54:59.989876+0000 mon.a (mon.0) 8086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:01.131 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:55:01.132 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:55:01.132 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:55:01.132 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:55:01.132 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:55:01.133 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:55:01.133 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:55:01.133 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:55:01.133 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:55:01.134 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:55:01.134 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:55:01.134 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:55:01.134 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:55:01.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:55:01.136 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:55:01.136 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:55:01.536 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:55:01.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:55:01.536 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:55:01.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:01.537 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:55:01.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:55:01.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:01.538 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:55:01.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:55:01.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:01.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:55:01.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:55:01.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:01.539 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:55:01.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:55:01.539 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:55:01.539 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:55:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:01 smithi121 bash[36902]: cluster 2022-04-22T19:55:00.426659+0000 mgr.y (mgr.34107) 5107 : cluster [DBG] pgmap v6410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:01 smithi121 bash[36902]: audit 2022-04-22T19:55:00.762900+0000 mgr.y (mgr.34107) 5108 : audit [DBG] from='client.40716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:01 smithi121 bash[36902]: audit 2022-04-22T19:55:01.355032+0000 mon.a (mon.0) 8087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:01 smithi121 bash[36902]: audit 2022-04-22T19:55:01.526817+0000 mon.a (mon.0) 8088 : audit [DBG] from='client.? 172.21.15.115:0/3838640337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[45495]: cluster 2022-04-22T19:55:00.426659+0000 mgr.y (mgr.34107) 5107 : cluster [DBG] pgmap v6410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[45495]: audit 2022-04-22T19:55:00.762900+0000 mgr.y (mgr.34107) 5108 : audit [DBG] from='client.40716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[45495]: audit 2022-04-22T19:55:01.355032+0000 mon.a (mon.0) 8087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[45495]: audit 2022-04-22T19:55:01.526817+0000 mon.a (mon.0) 8088 : audit [DBG] from='client.? 172.21.15.115:0/3838640337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[46934]: cluster 2022-04-22T19:55:00.426659+0000 mgr.y (mgr.34107) 5107 : cluster [DBG] pgmap v6410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[46934]: audit 2022-04-22T19:55:00.762900+0000 mgr.y (mgr.34107) 5108 : audit [DBG] from='client.40716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[46934]: audit 2022-04-22T19:55:01.355032+0000 mon.a (mon.0) 8087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:01 smithi115 bash[46934]: audit 2022-04-22T19:55:01.526817+0000 mon.a (mon.0) 8088 : audit [DBG] from='client.? 172.21.15.115:0/3838640337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:02.192Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:02 smithi121 bash[36902]: audit 2022-04-22T19:55:01.103253+0000 mgr.y (mgr.34107) 5109 : audit [DBG] from='client.40722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:02 smithi115 bash[45495]: audit 2022-04-22T19:55:01.103253+0000 mgr.y (mgr.34107) 5109 : audit [DBG] from='client.40722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:02 smithi115 bash[46934]: audit 2022-04-22T19:55:01.103253+0000 mgr.y (mgr.34107) 5109 : audit [DBG] from='client.40722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:03 smithi121 bash[36902]: cluster 2022-04-22T19:55:02.427754+0000 mgr.y (mgr.34107) 5110 : cluster [DBG] pgmap v6411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:03 smithi121 bash[36902]: audit 2022-04-22T19:55:02.813763+0000 mon.a (mon.0) 8089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:03 smithi115 bash[45495]: cluster 2022-04-22T19:55:02.427754+0000 mgr.y (mgr.34107) 5110 : cluster [DBG] pgmap v6411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:03 smithi115 bash[45495]: audit 2022-04-22T19:55:02.813763+0000 mon.a (mon.0) 8089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:03 smithi115 bash[46934]: cluster 2022-04-22T19:55:02.427754+0000 mgr.y (mgr.34107) 5110 : cluster [DBG] pgmap v6411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:03 smithi115 bash[46934]: audit 2022-04-22T19:55:02.813763+0000 mon.a (mon.0) 8089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:04.067Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:05 smithi121 bash[36902]: cluster 2022-04-22T19:55:04.428857+0000 mgr.y (mgr.34107) 5111 : cluster [DBG] pgmap v6412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:05 smithi121 bash[36902]: audit 2022-04-22T19:55:04.993591+0000 mon.a (mon.0) 8090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:05 smithi115 bash[45495]: cluster 2022-04-22T19:55:04.428857+0000 mgr.y (mgr.34107) 5111 : cluster [DBG] pgmap v6412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:05 smithi115 bash[45495]: audit 2022-04-22T19:55:04.993591+0000 mon.a (mon.0) 8090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:05 smithi115 bash[46934]: cluster 2022-04-22T19:55:04.428857+0000 mgr.y (mgr.34107) 5111 : cluster [DBG] pgmap v6412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:05 smithi115 bash[46934]: audit 2022-04-22T19:55:04.993591+0000 mon.a (mon.0) 8090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:06 smithi121 bash[36902]: audit 2022-04-22T19:55:06.358144+0000 mon.a (mon.0) 8091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:06 smithi115 bash[45495]: audit 2022-04-22T19:55:06.358144+0000 mon.a (mon.0) 8091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:06 smithi115 bash[46934]: audit 2022-04-22T19:55:06.358144+0000 mon.a (mon.0) 8091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:07 smithi121 bash[36902]: cluster 2022-04-22T19:55:06.429892+0000 mgr.y (mgr.34107) 5112 : cluster [DBG] pgmap v6413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:07 smithi115 bash[45495]: cluster 2022-04-22T19:55:06.429892+0000 mgr.y (mgr.34107) 5112 : cluster [DBG] pgmap v6413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:07 smithi115 bash[46934]: cluster 2022-04-22T19:55:06.429892+0000 mgr.y (mgr.34107) 5112 : cluster [DBG] pgmap v6413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:08 smithi115 bash[46934]: audit 2022-04-22T19:55:07.816890+0000 mon.a (mon.0) 8092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:09.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:08 smithi115 bash[45495]: audit 2022-04-22T19:55:07.816890+0000 mon.a (mon.0) 8092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:08 smithi121 bash[36902]: audit 2022-04-22T19:55:07.816890+0000 mon.a (mon.0) 8092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:09 smithi115 bash[45495]: cluster 2022-04-22T19:55:08.431016+0000 mgr.y (mgr.34107) 5113 : cluster [DBG] pgmap v6414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:09 smithi115 bash[46934]: cluster 2022-04-22T19:55:08.431016+0000 mgr.y (mgr.34107) 5113 : cluster [DBG] pgmap v6414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:09 smithi121 bash[36902]: cluster 2022-04-22T19:55:08.431016+0000 mgr.y (mgr.34107) 5113 : cluster [DBG] pgmap v6414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:10 smithi115 bash[45495]: audit 2022-04-22T19:55:09.997060+0000 mon.a (mon.0) 8093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:10 smithi115 bash[46934]: audit 2022-04-22T19:55:09.997060+0000 mon.a (mon.0) 8093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:10 smithi121 bash[36902]: audit 2022-04-22T19:55:09.997060+0000 mon.a (mon.0) 8093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:11 smithi115 bash[45495]: cluster 2022-04-22T19:55:10.432070+0000 mgr.y (mgr.34107) 5114 : cluster [DBG] pgmap v6415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:11 smithi115 bash[45495]: audit 2022-04-22T19:55:11.361139+0000 mon.a (mon.0) 8094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:11 smithi115 bash[46934]: cluster 2022-04-22T19:55:10.432070+0000 mgr.y (mgr.34107) 5114 : cluster [DBG] pgmap v6415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:11 smithi115 bash[46934]: audit 2022-04-22T19:55:11.361139+0000 mon.a (mon.0) 8094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:11 smithi121 bash[36902]: cluster 2022-04-22T19:55:10.432070+0000 mgr.y (mgr.34107) 5114 : cluster [DBG] pgmap v6415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:11 smithi121 bash[36902]: audit 2022-04-22T19:55:11.361139+0000 mon.a (mon.0) 8094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:12.193Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:13 smithi115 bash[46934]: cluster 2022-04-22T19:55:12.432893+0000 mgr.y (mgr.34107) 5115 : cluster [DBG] pgmap v6416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:13 smithi115 bash[46934]: audit 2022-04-22T19:55:12.819812+0000 mon.a (mon.0) 8095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:13 smithi115 bash[45495]: cluster 2022-04-22T19:55:12.432893+0000 mgr.y (mgr.34107) 5115 : cluster [DBG] pgmap v6416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:13 smithi115 bash[45495]: audit 2022-04-22T19:55:12.819812+0000 mon.a (mon.0) 8095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:13 smithi121 bash[36902]: cluster 2022-04-22T19:55:12.432893+0000 mgr.y (mgr.34107) 5115 : cluster [DBG] pgmap v6416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:13 smithi121 bash[36902]: audit 2022-04-22T19:55:12.819812+0000 mon.a (mon.0) 8095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:14.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[45495]: cluster 2022-04-22T19:55:14.433965+0000 mgr.y (mgr.34107) 5116 : cluster [DBG] pgmap v6417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[45495]: audit 2022-04-22T19:55:14.991323+0000 mon.a (mon.0) 8096 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[45495]: audit 2022-04-22T19:55:14.991864+0000 mon.a (mon.0) 8097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[45495]: audit 2022-04-22T19:55:15.000491+0000 mon.a (mon.0) 8098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[46934]: cluster 2022-04-22T19:55:14.433965+0000 mgr.y (mgr.34107) 5116 : cluster [DBG] pgmap v6417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[46934]: audit 2022-04-22T19:55:14.991323+0000 mon.a (mon.0) 8096 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[46934]: audit 2022-04-22T19:55:14.991864+0000 mon.a (mon.0) 8097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:55:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:15 smithi115 bash[46934]: audit 2022-04-22T19:55:15.000491+0000 mon.a (mon.0) 8098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:15 smithi121 bash[36902]: cluster 2022-04-22T19:55:14.433965+0000 mgr.y (mgr.34107) 5116 : cluster [DBG] pgmap v6417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:15 smithi121 bash[36902]: audit 2022-04-22T19:55:14.991323+0000 mon.a (mon.0) 8096 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:55:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:15 smithi121 bash[36902]: audit 2022-04-22T19:55:14.991864+0000 mon.a (mon.0) 8097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:55:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:15 smithi121 bash[36902]: audit 2022-04-22T19:55:15.000491+0000 mon.a (mon.0) 8098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:16 smithi115 bash[45495]: audit 2022-04-22T19:55:16.364166+0000 mon.a (mon.0) 8099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:17.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:16 smithi115 bash[46934]: audit 2022-04-22T19:55:16.364166+0000 mon.a (mon.0) 8099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:16 smithi121 bash[36902]: audit 2022-04-22T19:55:16.364166+0000 mon.a (mon.0) 8099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:17 smithi115 bash[45495]: cluster 2022-04-22T19:55:16.435176+0000 mgr.y (mgr.34107) 5117 : cluster [DBG] pgmap v6418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:17 smithi115 bash[46934]: cluster 2022-04-22T19:55:16.435176+0000 mgr.y (mgr.34107) 5117 : cluster [DBG] pgmap v6418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:17 smithi121 bash[36902]: cluster 2022-04-22T19:55:16.435176+0000 mgr.y (mgr.34107) 5117 : cluster [DBG] pgmap v6418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:18 smithi121 bash[36902]: audit 2022-04-22T19:55:17.822631+0000 mon.a (mon.0) 8100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:18 smithi115 bash[45495]: audit 2022-04-22T19:55:17.822631+0000 mon.a (mon.0) 8100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:19.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:18 smithi115 bash[46934]: audit 2022-04-22T19:55:17.822631+0000 mon.a (mon.0) 8100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:19 smithi121 bash[36902]: cluster 2022-04-22T19:55:18.436380+0000 mgr.y (mgr.34107) 5118 : cluster [DBG] pgmap v6419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:19 smithi115 bash[45495]: cluster 2022-04-22T19:55:18.436380+0000 mgr.y (mgr.34107) 5118 : cluster [DBG] pgmap v6419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:19 smithi115 bash[46934]: cluster 2022-04-22T19:55:18.436380+0000 mgr.y (mgr.34107) 5118 : cluster [DBG] pgmap v6419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:20 smithi121 bash[36902]: audit 2022-04-22T19:55:20.004330+0000 mon.a (mon.0) 8101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:20 smithi115 bash[45495]: audit 2022-04-22T19:55:20.004330+0000 mon.a (mon.0) 8101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:20 smithi115 bash[46934]: audit 2022-04-22T19:55:20.004330+0000 mon.a (mon.0) 8101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:22.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:21 smithi115 bash[45495]: cluster 2022-04-22T19:55:20.437178+0000 mgr.y (mgr.34107) 5119 : cluster [DBG] pgmap v6420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:22.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:21 smithi115 bash[45495]: audit 2022-04-22T19:55:21.367012+0000 mon.a (mon.0) 8102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:22.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:21 smithi115 bash[46934]: cluster 2022-04-22T19:55:20.437178+0000 mgr.y (mgr.34107) 5119 : cluster [DBG] pgmap v6420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:22.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:21 smithi115 bash[46934]: audit 2022-04-22T19:55:21.367012+0000 mon.a (mon.0) 8102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:21 smithi121 bash[36902]: cluster 2022-04-22T19:55:20.437178+0000 mgr.y (mgr.34107) 5119 : cluster [DBG] pgmap v6420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:21 smithi121 bash[36902]: audit 2022-04-22T19:55:21.367012+0000 mon.a (mon.0) 8102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:22.193Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:22 smithi121 bash[36902]: cluster 2022-04-22T19:55:22.438212+0000 mgr.y (mgr.34107) 5120 : cluster [DBG] pgmap v6421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:22 smithi115 bash[45495]: cluster 2022-04-22T19:55:22.438212+0000 mgr.y (mgr.34107) 5120 : cluster [DBG] pgmap v6421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:22 smithi115 bash[46934]: cluster 2022-04-22T19:55:22.438212+0000 mgr.y (mgr.34107) 5120 : cluster [DBG] pgmap v6421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:24.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:23 smithi115 bash[45495]: audit 2022-04-22T19:55:22.824755+0000 mon.a (mon.0) 8103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:24.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:23 smithi115 bash[46934]: audit 2022-04-22T19:55:22.824755+0000 mon.a (mon.0) 8103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:23 smithi121 bash[36902]: audit 2022-04-22T19:55:22.824755+0000 mon.a (mon.0) 8103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:24.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:24.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:24 smithi121 bash[36902]: cluster 2022-04-22T19:55:24.439303+0000 mgr.y (mgr.34107) 5121 : cluster [DBG] pgmap v6422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:24 smithi115 bash[45495]: cluster 2022-04-22T19:55:24.439303+0000 mgr.y (mgr.34107) 5121 : cluster [DBG] pgmap v6422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:24 smithi115 bash[46934]: cluster 2022-04-22T19:55:24.439303+0000 mgr.y (mgr.34107) 5121 : cluster [DBG] pgmap v6422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:25 smithi121 bash[36902]: audit 2022-04-22T19:55:25.008241+0000 mon.a (mon.0) 8104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:25 smithi115 bash[45495]: audit 2022-04-22T19:55:25.008241+0000 mon.a (mon.0) 8104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:25 smithi115 bash[46934]: audit 2022-04-22T19:55:25.008241+0000 mon.a (mon.0) 8104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:26 smithi121 bash[36902]: audit 2022-04-22T19:55:26.370009+0000 mon.a (mon.0) 8105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:26 smithi121 bash[36902]: cluster 2022-04-22T19:55:26.440412+0000 mgr.y (mgr.34107) 5122 : cluster [DBG] pgmap v6423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:26 smithi115 bash[45495]: audit 2022-04-22T19:55:26.370009+0000 mon.a (mon.0) 8105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:27.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:26 smithi115 bash[45495]: cluster 2022-04-22T19:55:26.440412+0000 mgr.y (mgr.34107) 5122 : cluster [DBG] pgmap v6423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:27.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:26 smithi115 bash[46934]: audit 2022-04-22T19:55:26.370009+0000 mon.a (mon.0) 8105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:27.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:26 smithi115 bash[46934]: cluster 2022-04-22T19:55:26.440412+0000 mgr.y (mgr.34107) 5122 : cluster [DBG] pgmap v6423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:27 smithi121 bash[36902]: audit 2022-04-22T19:55:27.827379+0000 mon.a (mon.0) 8106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:27 smithi115 bash[45495]: audit 2022-04-22T19:55:27.827379+0000 mon.a (mon.0) 8106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:27 smithi115 bash[46934]: audit 2022-04-22T19:55:27.827379+0000 mon.a (mon.0) 8106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:29 smithi121 bash[36902]: cluster 2022-04-22T19:55:28.441484+0000 mgr.y (mgr.34107) 5123 : cluster [DBG] pgmap v6424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:29.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:29 smithi115 bash[46934]: cluster 2022-04-22T19:55:28.441484+0000 mgr.y (mgr.34107) 5123 : cluster [DBG] pgmap v6424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:29.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:29 smithi115 bash[45495]: cluster 2022-04-22T19:55:28.441484+0000 mgr.y (mgr.34107) 5123 : cluster [DBG] pgmap v6424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:30 smithi121 bash[36902]: audit 2022-04-22T19:55:30.011659+0000 mon.a (mon.0) 8107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:30.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:30 smithi115 bash[45495]: audit 2022-04-22T19:55:30.011659+0000 mon.a (mon.0) 8107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:30.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:30 smithi115 bash[46934]: audit 2022-04-22T19:55:30.011659+0000 mon.a (mon.0) 8107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:31 smithi121 bash[36902]: cluster 2022-04-22T19:55:30.442558+0000 mgr.y (mgr.34107) 5124 : cluster [DBG] pgmap v6425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:31 smithi121 bash[36902]: audit 2022-04-22T19:55:31.372988+0000 mon.a (mon.0) 8108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:31 smithi115 bash[45495]: cluster 2022-04-22T19:55:30.442558+0000 mgr.y (mgr.34107) 5124 : cluster [DBG] pgmap v6425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:31.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:31 smithi115 bash[45495]: audit 2022-04-22T19:55:31.372988+0000 mon.a (mon.0) 8108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:31 smithi115 bash[46934]: cluster 2022-04-22T19:55:30.442558+0000 mgr.y (mgr.34107) 5124 : cluster [DBG] pgmap v6425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:31.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:31 smithi115 bash[46934]: audit 2022-04-22T19:55:31.372988+0000 mon.a (mon.0) 8108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:31.913 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:55:32.258 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:55:32.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:55:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:55:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:55:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:55:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:55:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:55:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:32.194Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:55:32.661 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:55:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:33 smithi121 bash[36902]: audit 2022-04-22T19:55:31.886376+0000 mgr.y (mgr.34107) 5125 : audit [DBG] from='client.50668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:33 smithi121 bash[36902]: audit 2022-04-22T19:55:32.230096+0000 mgr.y (mgr.34107) 5126 : audit [DBG] from='client.50674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:33 smithi121 bash[36902]: cluster 2022-04-22T19:55:32.443645+0000 mgr.y (mgr.34107) 5127 : cluster [DBG] pgmap v6426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:33 smithi121 bash[36902]: audit 2022-04-22T19:55:32.650529+0000 mon.c (mon.1) 217 : audit [DBG] from='client.? 172.21.15.115:0/3034989947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:33 smithi121 bash[36902]: audit 2022-04-22T19:55:32.830267+0000 mon.a (mon.0) 8109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[46934]: audit 2022-04-22T19:55:31.886376+0000 mgr.y (mgr.34107) 5125 : audit [DBG] from='client.50668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[46934]: audit 2022-04-22T19:55:32.230096+0000 mgr.y (mgr.34107) 5126 : audit [DBG] from='client.50674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[46934]: cluster 2022-04-22T19:55:32.443645+0000 mgr.y (mgr.34107) 5127 : cluster [DBG] pgmap v6426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[46934]: audit 2022-04-22T19:55:32.650529+0000 mon.c (mon.1) 217 : audit [DBG] from='client.? 172.21.15.115:0/3034989947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[46934]: audit 2022-04-22T19:55:32.830267+0000 mon.a (mon.0) 8109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[45495]: audit 2022-04-22T19:55:31.886376+0000 mgr.y (mgr.34107) 5125 : audit [DBG] from='client.50668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[45495]: audit 2022-04-22T19:55:32.230096+0000 mgr.y (mgr.34107) 5126 : audit [DBG] from='client.50674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[45495]: cluster 2022-04-22T19:55:32.443645+0000 mgr.y (mgr.34107) 5127 : cluster [DBG] pgmap v6426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:33.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[45495]: audit 2022-04-22T19:55:32.650529+0000 mon.c (mon.1) 217 : audit [DBG] from='client.? 172.21.15.115:0/3034989947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:55:33.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:33 smithi115 bash[45495]: audit 2022-04-22T19:55:32.830267+0000 mon.a (mon.0) 8109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:34.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:35 smithi115 bash[46934]: cluster 2022-04-22T19:55:34.444681+0000 mgr.y (mgr.34107) 5128 : cluster [DBG] pgmap v6427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:35.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:35 smithi115 bash[46934]: audit 2022-04-22T19:55:35.015444+0000 mon.a (mon.0) 8110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:35 smithi115 bash[45495]: cluster 2022-04-22T19:55:34.444681+0000 mgr.y (mgr.34107) 5128 : cluster [DBG] pgmap v6427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:35.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:35 smithi115 bash[45495]: audit 2022-04-22T19:55:35.015444+0000 mon.a (mon.0) 8110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:35 smithi121 bash[36902]: cluster 2022-04-22T19:55:34.444681+0000 mgr.y (mgr.34107) 5128 : cluster [DBG] pgmap v6427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:35 smithi121 bash[36902]: audit 2022-04-22T19:55:35.015444+0000 mon.a (mon.0) 8110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:36 smithi115 bash[45495]: audit 2022-04-22T19:55:36.376177+0000 mon.a (mon.0) 8111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:36 smithi115 bash[46934]: audit 2022-04-22T19:55:36.376177+0000 mon.a (mon.0) 8111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:36 smithi121 bash[36902]: audit 2022-04-22T19:55:36.376177+0000 mon.a (mon.0) 8111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:37 smithi115 bash[45495]: cluster 2022-04-22T19:55:36.445772+0000 mgr.y (mgr.34107) 5129 : cluster [DBG] pgmap v6428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:37.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:37 smithi115 bash[46934]: cluster 2022-04-22T19:55:36.445772+0000 mgr.y (mgr.34107) 5129 : cluster [DBG] pgmap v6428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:37 smithi121 bash[36902]: cluster 2022-04-22T19:55:36.445772+0000 mgr.y (mgr.34107) 5129 : cluster [DBG] pgmap v6428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:38 smithi115 bash[46934]: audit 2022-04-22T19:55:37.833519+0000 mon.a (mon.0) 8112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:38.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:38 smithi115 bash[45495]: audit 2022-04-22T19:55:37.833519+0000 mon.a (mon.0) 8112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:38 smithi121 bash[36902]: audit 2022-04-22T19:55:37.833519+0000 mon.a (mon.0) 8112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:39.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[45495]: cluster 2022-04-22T19:55:38.446909+0000 mgr.y (mgr.34107) 5130 : cluster [DBG] pgmap v6429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:39.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[45495]: audit 2022-04-22T19:55:39.323216+0000 mon.a (mon.0) 8113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:55:39.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[45495]: audit 2022-04-22T19:55:39.326173+0000 mon.a (mon.0) 8114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:55:39.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[45495]: audit 2022-04-22T19:55:39.327917+0000 mon.a (mon.0) 8115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:55:39.668 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[46934]: cluster 2022-04-22T19:55:38.446909+0000 mgr.y (mgr.34107) 5130 : cluster [DBG] pgmap v6429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:39.668 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[46934]: audit 2022-04-22T19:55:39.323216+0000 mon.a (mon.0) 8113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:55:39.668 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[46934]: audit 2022-04-22T19:55:39.326173+0000 mon.a (mon.0) 8114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:55:39.668 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:39 smithi115 bash[46934]: audit 2022-04-22T19:55:39.327917+0000 mon.a (mon.0) 8115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:55:39.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:39 smithi121 bash[36902]: cluster 2022-04-22T19:55:38.446909+0000 mgr.y (mgr.34107) 5130 : cluster [DBG] pgmap v6429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:39.694 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:39 smithi121 bash[36902]: audit 2022-04-22T19:55:39.323216+0000 mon.a (mon.0) 8113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:55:39.694 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:39 smithi121 bash[36902]: audit 2022-04-22T19:55:39.326173+0000 mon.a (mon.0) 8114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:55:39.694 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:39 smithi121 bash[36902]: audit 2022-04-22T19:55:39.327917+0000 mon.a (mon.0) 8115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:55:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:40 smithi121 bash[36902]: audit 2022-04-22T19:55:39.700932+0000 mon.a (mon.0) 8116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:55:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:40 smithi121 bash[36902]: audit 2022-04-22T19:55:40.019182+0000 mon.a (mon.0) 8117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:40 smithi115 bash[45495]: audit 2022-04-22T19:55:39.700932+0000 mon.a (mon.0) 8116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:55:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:40 smithi115 bash[45495]: audit 2022-04-22T19:55:40.019182+0000 mon.a (mon.0) 8117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:40 smithi115 bash[46934]: audit 2022-04-22T19:55:39.700932+0000 mon.a (mon.0) 8116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:55:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:40 smithi115 bash[46934]: audit 2022-04-22T19:55:40.019182+0000 mon.a (mon.0) 8117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:41 smithi121 bash[36902]: cluster 2022-04-22T19:55:40.447884+0000 mgr.y (mgr.34107) 5131 : cluster [DBG] pgmap v6430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:41 smithi121 bash[36902]: audit 2022-04-22T19:55:41.379306+0000 mon.a (mon.0) 8118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:41 smithi115 bash[45495]: cluster 2022-04-22T19:55:40.447884+0000 mgr.y (mgr.34107) 5131 : cluster [DBG] pgmap v6430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:41 smithi115 bash[45495]: audit 2022-04-22T19:55:41.379306+0000 mon.a (mon.0) 8118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:41 smithi115 bash[46934]: cluster 2022-04-22T19:55:40.447884+0000 mgr.y (mgr.34107) 5131 : cluster [DBG] pgmap v6430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:41 smithi115 bash[46934]: audit 2022-04-22T19:55:41.379306+0000 mon.a (mon.0) 8118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:42.196Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:43 smithi121 bash[36902]: cluster 2022-04-22T19:55:42.448947+0000 mgr.y (mgr.34107) 5132 : cluster [DBG] pgmap v6431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:43 smithi121 bash[36902]: audit 2022-04-22T19:55:42.836539+0000 mon.a (mon.0) 8119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:43 smithi115 bash[45495]: cluster 2022-04-22T19:55:42.448947+0000 mgr.y (mgr.34107) 5132 : cluster [DBG] pgmap v6431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:43 smithi115 bash[45495]: audit 2022-04-22T19:55:42.836539+0000 mon.a (mon.0) 8119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:43 smithi115 bash[46934]: cluster 2022-04-22T19:55:42.448947+0000 mgr.y (mgr.34107) 5132 : cluster [DBG] pgmap v6431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:43 smithi115 bash[46934]: audit 2022-04-22T19:55:42.836539+0000 mon.a (mon.0) 8119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:44.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:45 smithi121 bash[36902]: cluster 2022-04-22T19:55:44.449845+0000 mgr.y (mgr.34107) 5133 : cluster [DBG] pgmap v6432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:45 smithi121 bash[36902]: audit 2022-04-22T19:55:45.022828+0000 mon.a (mon.0) 8120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:45 smithi115 bash[45495]: cluster 2022-04-22T19:55:44.449845+0000 mgr.y (mgr.34107) 5133 : cluster [DBG] pgmap v6432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:45 smithi115 bash[45495]: audit 2022-04-22T19:55:45.022828+0000 mon.a (mon.0) 8120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:45 smithi115 bash[46934]: cluster 2022-04-22T19:55:44.449845+0000 mgr.y (mgr.34107) 5133 : cluster [DBG] pgmap v6432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:45 smithi115 bash[46934]: audit 2022-04-22T19:55:45.022828+0000 mon.a (mon.0) 8120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:46 smithi121 bash[36902]: audit 2022-04-22T19:55:46.382372+0000 mon.a (mon.0) 8121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:46 smithi115 bash[45495]: audit 2022-04-22T19:55:46.382372+0000 mon.a (mon.0) 8121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:46 smithi115 bash[46934]: audit 2022-04-22T19:55:46.382372+0000 mon.a (mon.0) 8121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:47 smithi121 bash[36902]: cluster 2022-04-22T19:55:46.450871+0000 mgr.y (mgr.34107) 5134 : cluster [DBG] pgmap v6433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:47 smithi115 bash[45495]: cluster 2022-04-22T19:55:46.450871+0000 mgr.y (mgr.34107) 5134 : cluster [DBG] pgmap v6433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:48.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:47 smithi115 bash[46934]: cluster 2022-04-22T19:55:46.450871+0000 mgr.y (mgr.34107) 5134 : cluster [DBG] pgmap v6433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:48 smithi121 bash[36902]: audit 2022-04-22T19:55:47.839614+0000 mon.a (mon.0) 8122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:48 smithi115 bash[45495]: audit 2022-04-22T19:55:47.839614+0000 mon.a (mon.0) 8122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:48 smithi115 bash[46934]: audit 2022-04-22T19:55:47.839614+0000 mon.a (mon.0) 8122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:49 smithi121 bash[36902]: cluster 2022-04-22T19:55:48.452020+0000 mgr.y (mgr.34107) 5135 : cluster [DBG] pgmap v6434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:49 smithi115 bash[45495]: cluster 2022-04-22T19:55:48.452020+0000 mgr.y (mgr.34107) 5135 : cluster [DBG] pgmap v6434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:49 smithi115 bash[46934]: cluster 2022-04-22T19:55:48.452020+0000 mgr.y (mgr.34107) 5135 : cluster [DBG] pgmap v6434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:50 smithi121 bash[36902]: audit 2022-04-22T19:55:50.026245+0000 mon.a (mon.0) 8123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:50 smithi115 bash[45495]: audit 2022-04-22T19:55:50.026245+0000 mon.a (mon.0) 8123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:50 smithi115 bash[46934]: audit 2022-04-22T19:55:50.026245+0000 mon.a (mon.0) 8123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:51 smithi121 bash[36902]: cluster 2022-04-22T19:55:50.453088+0000 mgr.y (mgr.34107) 5136 : cluster [DBG] pgmap v6435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:51 smithi121 bash[36902]: audit 2022-04-22T19:55:51.385648+0000 mon.a (mon.0) 8124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:51 smithi115 bash[45495]: cluster 2022-04-22T19:55:50.453088+0000 mgr.y (mgr.34107) 5136 : cluster [DBG] pgmap v6435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:52.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:51 smithi115 bash[45495]: audit 2022-04-22T19:55:51.385648+0000 mon.a (mon.0) 8124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:52.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:51 smithi115 bash[46934]: cluster 2022-04-22T19:55:50.453088+0000 mgr.y (mgr.34107) 5136 : cluster [DBG] pgmap v6435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:52.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:51 smithi115 bash[46934]: audit 2022-04-22T19:55:51.385648+0000 mon.a (mon.0) 8124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:52.195Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:53 smithi121 bash[36902]: cluster 2022-04-22T19:55:52.454204+0000 mgr.y (mgr.34107) 5137 : cluster [DBG] pgmap v6436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:53 smithi121 bash[36902]: audit 2022-04-22T19:55:52.842678+0000 mon.a (mon.0) 8125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:53 smithi115 bash[45495]: cluster 2022-04-22T19:55:52.454204+0000 mgr.y (mgr.34107) 5137 : cluster [DBG] pgmap v6436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:53 smithi115 bash[45495]: audit 2022-04-22T19:55:52.842678+0000 mon.a (mon.0) 8125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:53 smithi115 bash[46934]: cluster 2022-04-22T19:55:52.454204+0000 mgr.y (mgr.34107) 5137 : cluster [DBG] pgmap v6436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:53 smithi115 bash[46934]: audit 2022-04-22T19:55:52.842678+0000 mon.a (mon.0) 8125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:55:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:55:54.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:55:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:55 smithi121 bash[36902]: cluster 2022-04-22T19:55:54.455429+0000 mgr.y (mgr.34107) 5138 : cluster [DBG] pgmap v6437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:55 smithi121 bash[36902]: audit 2022-04-22T19:55:55.029608+0000 mon.a (mon.0) 8126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:55 smithi115 bash[45495]: cluster 2022-04-22T19:55:54.455429+0000 mgr.y (mgr.34107) 5138 : cluster [DBG] pgmap v6437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:55 smithi115 bash[45495]: audit 2022-04-22T19:55:55.029608+0000 mon.a (mon.0) 8126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:56.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:55 smithi115 bash[46934]: cluster 2022-04-22T19:55:54.455429+0000 mgr.y (mgr.34107) 5138 : cluster [DBG] pgmap v6437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:55 smithi115 bash[46934]: audit 2022-04-22T19:55:55.029608+0000 mon.a (mon.0) 8126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:55:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:56 smithi115 bash[45495]: audit 2022-04-22T19:55:56.388675+0000 mon.a (mon.0) 8127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:57.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:56 smithi115 bash[46934]: audit 2022-04-22T19:55:56.388675+0000 mon.a (mon.0) 8127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:56 smithi121 bash[36902]: audit 2022-04-22T19:55:56.388675+0000 mon.a (mon.0) 8127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:55:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:57 smithi115 bash[46934]: cluster 2022-04-22T19:55:56.456553+0000 mgr.y (mgr.34107) 5139 : cluster [DBG] pgmap v6438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:58.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:57 smithi115 bash[45495]: cluster 2022-04-22T19:55:56.456553+0000 mgr.y (mgr.34107) 5139 : cluster [DBG] pgmap v6438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:57 smithi121 bash[36902]: cluster 2022-04-22T19:55:56.456553+0000 mgr.y (mgr.34107) 5139 : cluster [DBG] pgmap v6438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:55:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:58 smithi115 bash[45495]: audit 2022-04-22T19:55:57.845897+0000 mon.a (mon.0) 8128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:58 smithi115 bash[46934]: audit 2022-04-22T19:55:57.845897+0000 mon.a (mon.0) 8128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:55:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:58 smithi121 bash[36902]: audit 2022-04-22T19:55:57.845897+0000 mon.a (mon.0) 8128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:55:59 smithi115 bash[45495]: cluster 2022-04-22T19:55:58.457739+0000 mgr.y (mgr.34107) 5140 : cluster [DBG] pgmap v6439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:55:59 smithi115 bash[46934]: cluster 2022-04-22T19:55:58.457739+0000 mgr.y (mgr.34107) 5140 : cluster [DBG] pgmap v6439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:55:59 smithi121 bash[36902]: cluster 2022-04-22T19:55:58.457739+0000 mgr.y (mgr.34107) 5140 : cluster [DBG] pgmap v6439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:00 smithi115 bash[45495]: audit 2022-04-22T19:56:00.033238+0000 mon.a (mon.0) 8129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:00 smithi115 bash[46934]: audit 2022-04-22T19:56:00.033238+0000 mon.a (mon.0) 8129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:00 smithi121 bash[36902]: audit 2022-04-22T19:56:00.033238+0000 mon.a (mon.0) 8129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:01 smithi115 bash[45495]: cluster 2022-04-22T19:56:00.458801+0000 mgr.y (mgr.34107) 5141 : cluster [DBG] pgmap v6440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:02.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:01 smithi115 bash[45495]: audit 2022-04-22T19:56:01.391644+0000 mon.a (mon.0) 8130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:01 smithi115 bash[46934]: cluster 2022-04-22T19:56:00.458801+0000 mgr.y (mgr.34107) 5141 : cluster [DBG] pgmap v6440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:01 smithi115 bash[46934]: audit 2022-04-22T19:56:01.391644+0000 mon.a (mon.0) 8130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:01 smithi121 bash[36902]: cluster 2022-04-22T19:56:00.458801+0000 mgr.y (mgr.34107) 5141 : cluster [DBG] pgmap v6440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:01 smithi121 bash[36902]: audit 2022-04-22T19:56:01.391644+0000 mon.a (mon.0) 8130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:02.196Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:03.028 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:56:03.369 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:56:03.369 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:56:03.370 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:56:03.371 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:56:03.781 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:56:03.781 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:56:03.781 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:56:03.782 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:56:03.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:03.783 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:56:03.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:56:03.783 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:56:03.783 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:56:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[45495]: cluster 2022-04-22T19:56:02.459950+0000 mgr.y (mgr.34107) 5142 : cluster [DBG] pgmap v6441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[45495]: audit 2022-04-22T19:56:02.848697+0000 mon.a (mon.0) 8131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[45495]: audit 2022-04-22T19:56:03.772187+0000 mon.a (mon.0) 8132 : audit [DBG] from='client.? 172.21.15.115:0/4256614629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[46934]: cluster 2022-04-22T19:56:02.459950+0000 mgr.y (mgr.34107) 5142 : cluster [DBG] pgmap v6441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[46934]: audit 2022-04-22T19:56:02.848697+0000 mon.a (mon.0) 8131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:03 smithi115 bash[46934]: audit 2022-04-22T19:56:03.772187+0000 mon.a (mon.0) 8132 : audit [DBG] from='client.? 172.21.15.115:0/4256614629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:03 smithi121 bash[36902]: cluster 2022-04-22T19:56:02.459950+0000 mgr.y (mgr.34107) 5142 : cluster [DBG] pgmap v6441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:03 smithi121 bash[36902]: audit 2022-04-22T19:56:02.848697+0000 mon.a (mon.0) 8131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:03 smithi121 bash[36902]: audit 2022-04-22T19:56:03.772187+0000 mon.a (mon.0) 8132 : audit [DBG] from='client.? 172.21.15.115:0/4256614629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:04.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:04 smithi115 bash[45495]: audit 2022-04-22T19:56:03.002312+0000 mgr.y (mgr.34107) 5143 : audit [DBG] from='client.50686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:04 smithi115 bash[45495]: audit 2022-04-22T19:56:03.341395+0000 mgr.y (mgr.34107) 5144 : audit [DBG] from='client.50692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:04 smithi115 bash[46934]: audit 2022-04-22T19:56:03.002312+0000 mgr.y (mgr.34107) 5143 : audit [DBG] from='client.50686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:04 smithi115 bash[46934]: audit 2022-04-22T19:56:03.341395+0000 mgr.y (mgr.34107) 5144 : audit [DBG] from='client.50692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:04 smithi121 bash[36902]: audit 2022-04-22T19:56:03.002312+0000 mgr.y (mgr.34107) 5143 : audit [DBG] from='client.50686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:04 smithi121 bash[36902]: audit 2022-04-22T19:56:03.341395+0000 mgr.y (mgr.34107) 5144 : audit [DBG] from='client.50692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:05 smithi115 bash[45495]: cluster 2022-04-22T19:56:04.461066+0000 mgr.y (mgr.34107) 5145 : cluster [DBG] pgmap v6442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:05 smithi115 bash[45495]: audit 2022-04-22T19:56:05.037271+0000 mon.a (mon.0) 8133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:05 smithi115 bash[46934]: cluster 2022-04-22T19:56:04.461066+0000 mgr.y (mgr.34107) 5145 : cluster [DBG] pgmap v6442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:05 smithi115 bash[46934]: audit 2022-04-22T19:56:05.037271+0000 mon.a (mon.0) 8133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:05 smithi121 bash[36902]: cluster 2022-04-22T19:56:04.461066+0000 mgr.y (mgr.34107) 5145 : cluster [DBG] pgmap v6442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:05 smithi121 bash[36902]: audit 2022-04-22T19:56:05.037271+0000 mon.a (mon.0) 8133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:06 smithi121 bash[36902]: audit 2022-04-22T19:56:06.394787+0000 mon.a (mon.0) 8134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:06 smithi115 bash[45495]: audit 2022-04-22T19:56:06.394787+0000 mon.a (mon.0) 8134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:06 smithi115 bash[46934]: audit 2022-04-22T19:56:06.394787+0000 mon.a (mon.0) 8134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:07 smithi121 bash[36902]: cluster 2022-04-22T19:56:06.462209+0000 mgr.y (mgr.34107) 5146 : cluster [DBG] pgmap v6443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:07 smithi115 bash[45495]: cluster 2022-04-22T19:56:06.462209+0000 mgr.y (mgr.34107) 5146 : cluster [DBG] pgmap v6443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:07 smithi115 bash[46934]: cluster 2022-04-22T19:56:06.462209+0000 mgr.y (mgr.34107) 5146 : cluster [DBG] pgmap v6443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:08 smithi121 bash[36902]: audit 2022-04-22T19:56:07.851501+0000 mon.a (mon.0) 8135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:08 smithi115 bash[45495]: audit 2022-04-22T19:56:07.851501+0000 mon.a (mon.0) 8135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:08 smithi115 bash[46934]: audit 2022-04-22T19:56:07.851501+0000 mon.a (mon.0) 8135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:09 smithi121 bash[36902]: cluster 2022-04-22T19:56:08.463406+0000 mgr.y (mgr.34107) 5147 : cluster [DBG] pgmap v6444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:09 smithi115 bash[45495]: cluster 2022-04-22T19:56:08.463406+0000 mgr.y (mgr.34107) 5147 : cluster [DBG] pgmap v6444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:09 smithi115 bash[46934]: cluster 2022-04-22T19:56:08.463406+0000 mgr.y (mgr.34107) 5147 : cluster [DBG] pgmap v6444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:10 smithi121 bash[36902]: audit 2022-04-22T19:56:10.041271+0000 mon.a (mon.0) 8136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:10 smithi115 bash[45495]: audit 2022-04-22T19:56:10.041271+0000 mon.a (mon.0) 8136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:10 smithi115 bash[46934]: audit 2022-04-22T19:56:10.041271+0000 mon.a (mon.0) 8136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:12.206 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:11 smithi115 bash[45495]: cluster 2022-04-22T19:56:10.464441+0000 mgr.y (mgr.34107) 5148 : cluster [DBG] pgmap v6445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:12.207 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:11 smithi115 bash[45495]: audit 2022-04-22T19:56:11.397930+0000 mon.a (mon.0) 8137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:12.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:11 smithi115 bash[46934]: cluster 2022-04-22T19:56:10.464441+0000 mgr.y (mgr.34107) 5148 : cluster [DBG] pgmap v6445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:12.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:11 smithi115 bash[46934]: audit 2022-04-22T19:56:11.397930+0000 mon.a (mon.0) 8137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:11 smithi121 bash[36902]: cluster 2022-04-22T19:56:10.464441+0000 mgr.y (mgr.34107) 5148 : cluster [DBG] pgmap v6445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:11 smithi121 bash[36902]: audit 2022-04-22T19:56:11.397930+0000 mon.a (mon.0) 8137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:12.197Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:12 smithi121 bash[36902]: cluster 2022-04-22T19:56:12.465538+0000 mgr.y (mgr.34107) 5149 : cluster [DBG] pgmap v6446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:12 smithi115 bash[45495]: cluster 2022-04-22T19:56:12.465538+0000 mgr.y (mgr.34107) 5149 : cluster [DBG] pgmap v6446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:12 smithi115 bash[46934]: cluster 2022-04-22T19:56:12.465538+0000 mgr.y (mgr.34107) 5149 : cluster [DBG] pgmap v6446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:13 smithi121 bash[36902]: audit 2022-04-22T19:56:12.854300+0000 mon.a (mon.0) 8138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:13 smithi115 bash[45495]: audit 2022-04-22T19:56:12.854300+0000 mon.a (mon.0) 8138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:14.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:13 smithi115 bash[46934]: audit 2022-04-22T19:56:12.854300+0000 mon.a (mon.0) 8138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:14 smithi121 bash[36902]: cluster 2022-04-22T19:56:14.466622+0000 mgr.y (mgr.34107) 5150 : cluster [DBG] pgmap v6447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:14 smithi115 bash[45495]: cluster 2022-04-22T19:56:14.466622+0000 mgr.y (mgr.34107) 5150 : cluster [DBG] pgmap v6447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:14 smithi115 bash[46934]: cluster 2022-04-22T19:56:14.466622+0000 mgr.y (mgr.34107) 5150 : cluster [DBG] pgmap v6447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:15 smithi121 bash[36902]: audit 2022-04-22T19:56:15.002114+0000 mon.a (mon.0) 8139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:56:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:15 smithi121 bash[36902]: audit 2022-04-22T19:56:15.003301+0000 mon.a (mon.0) 8140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:56:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:15 smithi121 bash[36902]: audit 2022-04-22T19:56:15.044792+0000 mon.a (mon.0) 8141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[45495]: audit 2022-04-22T19:56:15.002114+0000 mon.a (mon.0) 8139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:56:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[45495]: audit 2022-04-22T19:56:15.003301+0000 mon.a (mon.0) 8140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:56:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[45495]: audit 2022-04-22T19:56:15.044792+0000 mon.a (mon.0) 8141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[46934]: audit 2022-04-22T19:56:15.002114+0000 mon.a (mon.0) 8139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:56:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[46934]: audit 2022-04-22T19:56:15.003301+0000 mon.a (mon.0) 8140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:56:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:15 smithi115 bash[46934]: audit 2022-04-22T19:56:15.044792+0000 mon.a (mon.0) 8141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:16 smithi121 bash[36902]: audit 2022-04-22T19:56:16.401148+0000 mon.a (mon.0) 8142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:16 smithi121 bash[36902]: cluster 2022-04-22T19:56:16.467737+0000 mgr.y (mgr.34107) 5151 : cluster [DBG] pgmap v6448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:16 smithi115 bash[45495]: audit 2022-04-22T19:56:16.401148+0000 mon.a (mon.0) 8142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:16 smithi115 bash[45495]: cluster 2022-04-22T19:56:16.467737+0000 mgr.y (mgr.34107) 5151 : cluster [DBG] pgmap v6448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:16 smithi115 bash[46934]: audit 2022-04-22T19:56:16.401148+0000 mon.a (mon.0) 8142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:17.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:16 smithi115 bash[46934]: cluster 2022-04-22T19:56:16.467737+0000 mgr.y (mgr.34107) 5151 : cluster [DBG] pgmap v6448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:17 smithi121 bash[36902]: audit 2022-04-22T19:56:17.857513+0000 mon.a (mon.0) 8143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:17 smithi115 bash[46934]: audit 2022-04-22T19:56:17.857513+0000 mon.a (mon.0) 8143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:17 smithi115 bash[45495]: audit 2022-04-22T19:56:17.857513+0000 mon.a (mon.0) 8143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:19 smithi121 bash[36902]: cluster 2022-04-22T19:56:18.468844+0000 mgr.y (mgr.34107) 5152 : cluster [DBG] pgmap v6449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:19 smithi115 bash[45495]: cluster 2022-04-22T19:56:18.468844+0000 mgr.y (mgr.34107) 5152 : cluster [DBG] pgmap v6449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:19 smithi115 bash[46934]: cluster 2022-04-22T19:56:18.468844+0000 mgr.y (mgr.34107) 5152 : cluster [DBG] pgmap v6449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:20 smithi121 bash[36902]: audit 2022-04-22T19:56:20.048282+0000 mon.a (mon.0) 8144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:20 smithi115 bash[45495]: audit 2022-04-22T19:56:20.048282+0000 mon.a (mon.0) 8144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:20 smithi115 bash[46934]: audit 2022-04-22T19:56:20.048282+0000 mon.a (mon.0) 8144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:21 smithi121 bash[36902]: cluster 2022-04-22T19:56:20.469945+0000 mgr.y (mgr.34107) 5153 : cluster [DBG] pgmap v6450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:21 smithi121 bash[36902]: audit 2022-04-22T19:56:21.404238+0000 mon.a (mon.0) 8145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:21 smithi115 bash[45495]: cluster 2022-04-22T19:56:20.469945+0000 mgr.y (mgr.34107) 5153 : cluster [DBG] pgmap v6450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:21 smithi115 bash[45495]: audit 2022-04-22T19:56:21.404238+0000 mon.a (mon.0) 8145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:21 smithi115 bash[46934]: cluster 2022-04-22T19:56:20.469945+0000 mgr.y (mgr.34107) 5153 : cluster [DBG] pgmap v6450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:21.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:21 smithi115 bash[46934]: audit 2022-04-22T19:56:21.404238+0000 mon.a (mon.0) 8145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:22.197Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:23.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:23 smithi115 bash[45495]: cluster 2022-04-22T19:56:22.471094+0000 mgr.y (mgr.34107) 5154 : cluster [DBG] pgmap v6451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:23.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:23 smithi115 bash[45495]: audit 2022-04-22T19:56:22.860546+0000 mon.a (mon.0) 8146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:23 smithi115 bash[46934]: cluster 2022-04-22T19:56:22.471094+0000 mgr.y (mgr.34107) 5154 : cluster [DBG] pgmap v6451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:23 smithi115 bash[46934]: audit 2022-04-22T19:56:22.860546+0000 mon.a (mon.0) 8146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:23 smithi121 bash[36902]: cluster 2022-04-22T19:56:22.471094+0000 mgr.y (mgr.34107) 5154 : cluster [DBG] pgmap v6451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:23 smithi121 bash[36902]: audit 2022-04-22T19:56:22.860546+0000 mon.a (mon.0) 8146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:24.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:25 smithi115 bash[45495]: cluster 2022-04-22T19:56:24.472182+0000 mgr.y (mgr.34107) 5155 : cluster [DBG] pgmap v6452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:25.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:25 smithi115 bash[45495]: audit 2022-04-22T19:56:25.051936+0000 mon.a (mon.0) 8147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:25 smithi115 bash[46934]: cluster 2022-04-22T19:56:24.472182+0000 mgr.y (mgr.34107) 5155 : cluster [DBG] pgmap v6452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:25.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:25 smithi115 bash[46934]: audit 2022-04-22T19:56:25.051936+0000 mon.a (mon.0) 8147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:25 smithi121 bash[36902]: cluster 2022-04-22T19:56:24.472182+0000 mgr.y (mgr.34107) 5155 : cluster [DBG] pgmap v6452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:25 smithi121 bash[36902]: audit 2022-04-22T19:56:25.051936+0000 mon.a (mon.0) 8147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:26 smithi121 bash[36902]: audit 2022-04-22T19:56:26.407297+0000 mon.a (mon.0) 8148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:26 smithi115 bash[45495]: audit 2022-04-22T19:56:26.407297+0000 mon.a (mon.0) 8148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:26 smithi115 bash[46934]: audit 2022-04-22T19:56:26.407297+0000 mon.a (mon.0) 8148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:27 smithi121 bash[36902]: cluster 2022-04-22T19:56:26.473359+0000 mgr.y (mgr.34107) 5156 : cluster [DBG] pgmap v6453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:27 smithi115 bash[45495]: cluster 2022-04-22T19:56:26.473359+0000 mgr.y (mgr.34107) 5156 : cluster [DBG] pgmap v6453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:27 smithi115 bash[46934]: cluster 2022-04-22T19:56:26.473359+0000 mgr.y (mgr.34107) 5156 : cluster [DBG] pgmap v6453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:28 smithi121 bash[36902]: audit 2022-04-22T19:56:27.863758+0000 mon.a (mon.0) 8149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:28 smithi115 bash[45495]: audit 2022-04-22T19:56:27.863758+0000 mon.a (mon.0) 8149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:28 smithi115 bash[46934]: audit 2022-04-22T19:56:27.863758+0000 mon.a (mon.0) 8149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:29 smithi121 bash[36902]: cluster 2022-04-22T19:56:28.474585+0000 mgr.y (mgr.34107) 5157 : cluster [DBG] pgmap v6454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:29 smithi115 bash[45495]: cluster 2022-04-22T19:56:28.474585+0000 mgr.y (mgr.34107) 5157 : cluster [DBG] pgmap v6454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:29 smithi115 bash[46934]: cluster 2022-04-22T19:56:28.474585+0000 mgr.y (mgr.34107) 5157 : cluster [DBG] pgmap v6454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:30 smithi121 bash[36902]: audit 2022-04-22T19:56:30.055621+0000 mon.a (mon.0) 8150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:30 smithi115 bash[45495]: audit 2022-04-22T19:56:30.055621+0000 mon.a (mon.0) 8150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:30 smithi115 bash[46934]: audit 2022-04-22T19:56:30.055621+0000 mon.a (mon.0) 8150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:32.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:31 smithi121 bash[36902]: cluster 2022-04-22T19:56:30.475706+0000 mgr.y (mgr.34107) 5158 : cluster [DBG] pgmap v6455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:32.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:31 smithi121 bash[36902]: audit 2022-04-22T19:56:31.410922+0000 mon.a (mon.0) 8151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:31 smithi115 bash[45495]: cluster 2022-04-22T19:56:30.475706+0000 mgr.y (mgr.34107) 5158 : cluster [DBG] pgmap v6455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:31 smithi115 bash[45495]: audit 2022-04-22T19:56:31.410922+0000 mon.a (mon.0) 8151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:31 smithi115 bash[46934]: cluster 2022-04-22T19:56:30.475706+0000 mgr.y (mgr.34107) 5158 : cluster [DBG] pgmap v6455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:31 smithi115 bash[46934]: audit 2022-04-22T19:56:31.410922+0000 mon.a (mon.0) 8151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:32.197Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:33 smithi121 bash[36902]: cluster 2022-04-22T19:56:32.476731+0000 mgr.y (mgr.34107) 5159 : cluster [DBG] pgmap v6456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:33 smithi121 bash[36902]: audit 2022-04-22T19:56:32.866802+0000 mon.a (mon.0) 8152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:33 smithi115 bash[46934]: cluster 2022-04-22T19:56:32.476731+0000 mgr.y (mgr.34107) 5159 : cluster [DBG] pgmap v6456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:33 smithi115 bash[46934]: audit 2022-04-22T19:56:32.866802+0000 mon.a (mon.0) 8152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:33 smithi115 bash[45495]: cluster 2022-04-22T19:56:32.476731+0000 mgr.y (mgr.34107) 5159 : cluster [DBG] pgmap v6456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:33 smithi115 bash[45495]: audit 2022-04-22T19:56:32.866802+0000 mon.a (mon.0) 8152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:34.162 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:56:34.505 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:56:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:56:34.507 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:56:34.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:56:34.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:56:34.508 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:56:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:34.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:56:34.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:56:34.908 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:56:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:35 smithi121 bash[36902]: audit 2022-04-22T19:56:34.135410+0000 mgr.y (mgr.34107) 5160 : audit [DBG] from='client.40770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:35 smithi121 bash[36902]: audit 2022-04-22T19:56:34.474439+0000 mgr.y (mgr.34107) 5161 : audit [DBG] from='client.50710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:35 smithi121 bash[36902]: cluster 2022-04-22T19:56:34.478064+0000 mgr.y (mgr.34107) 5162 : cluster [DBG] pgmap v6457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:35 smithi121 bash[36902]: audit 2022-04-22T19:56:34.897630+0000 mon.c (mon.1) 218 : audit [DBG] from='client.? 172.21.15.115:0/3797402968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:35 smithi121 bash[36902]: audit 2022-04-22T19:56:35.059572+0000 mon.a (mon.0) 8153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[45495]: audit 2022-04-22T19:56:34.135410+0000 mgr.y (mgr.34107) 5160 : audit [DBG] from='client.40770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[45495]: audit 2022-04-22T19:56:34.474439+0000 mgr.y (mgr.34107) 5161 : audit [DBG] from='client.50710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[45495]: cluster 2022-04-22T19:56:34.478064+0000 mgr.y (mgr.34107) 5162 : cluster [DBG] pgmap v6457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[45495]: audit 2022-04-22T19:56:34.897630+0000 mon.c (mon.1) 218 : audit [DBG] from='client.? 172.21.15.115:0/3797402968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[45495]: audit 2022-04-22T19:56:35.059572+0000 mon.a (mon.0) 8153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[46934]: audit 2022-04-22T19:56:34.135410+0000 mgr.y (mgr.34107) 5160 : audit [DBG] from='client.40770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[46934]: audit 2022-04-22T19:56:34.474439+0000 mgr.y (mgr.34107) 5161 : audit [DBG] from='client.50710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:56:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[46934]: cluster 2022-04-22T19:56:34.478064+0000 mgr.y (mgr.34107) 5162 : cluster [DBG] pgmap v6457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[46934]: audit 2022-04-22T19:56:34.897630+0000 mon.c (mon.1) 218 : audit [DBG] from='client.? 172.21.15.115:0/3797402968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:56:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:35 smithi115 bash[46934]: audit 2022-04-22T19:56:35.059572+0000 mon.a (mon.0) 8153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:36 smithi121 bash[36902]: audit 2022-04-22T19:56:36.414350+0000 mon.a (mon.0) 8154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:36 smithi115 bash[45495]: audit 2022-04-22T19:56:36.414350+0000 mon.a (mon.0) 8154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:36 smithi115 bash[46934]: audit 2022-04-22T19:56:36.414350+0000 mon.a (mon.0) 8154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:37 smithi121 bash[36902]: cluster 2022-04-22T19:56:36.479124+0000 mgr.y (mgr.34107) 5163 : cluster [DBG] pgmap v6458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:37 smithi115 bash[45495]: cluster 2022-04-22T19:56:36.479124+0000 mgr.y (mgr.34107) 5163 : cluster [DBG] pgmap v6458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:37 smithi115 bash[46934]: cluster 2022-04-22T19:56:36.479124+0000 mgr.y (mgr.34107) 5163 : cluster [DBG] pgmap v6458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:38 smithi121 bash[36902]: audit 2022-04-22T19:56:37.869906+0000 mon.a (mon.0) 8155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:38 smithi115 bash[46934]: audit 2022-04-22T19:56:37.869906+0000 mon.a (mon.0) 8155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:38 smithi115 bash[45495]: audit 2022-04-22T19:56:37.869906+0000 mon.a (mon.0) 8155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:39 smithi121 bash[36902]: cluster 2022-04-22T19:56:38.480308+0000 mgr.y (mgr.34107) 5164 : cluster [DBG] pgmap v6459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:39 smithi115 bash[46934]: cluster 2022-04-22T19:56:38.480308+0000 mgr.y (mgr.34107) 5164 : cluster [DBG] pgmap v6459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:39 smithi115 bash[45495]: cluster 2022-04-22T19:56:38.480308+0000 mgr.y (mgr.34107) 5164 : cluster [DBG] pgmap v6459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:40 smithi121 bash[36902]: audit 2022-04-22T19:56:39.708142+0000 mon.a (mon.0) 8156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:56:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:40 smithi121 bash[36902]: audit 2022-04-22T19:56:39.711228+0000 mon.a (mon.0) 8157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:56:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:40 smithi121 bash[36902]: audit 2022-04-22T19:56:39.712928+0000 mon.a (mon.0) 8158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:56:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:40 smithi121 bash[36902]: audit 2022-04-22T19:56:40.064513+0000 mon.a (mon.0) 8159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:56:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:40 smithi121 bash[36902]: audit 2022-04-22T19:56:40.064763+0000 mon.a (mon.0) 8160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[45495]: audit 2022-04-22T19:56:39.708142+0000 mon.a (mon.0) 8156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:56:41.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[45495]: audit 2022-04-22T19:56:39.711228+0000 mon.a (mon.0) 8157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:56:41.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[45495]: audit 2022-04-22T19:56:39.712928+0000 mon.a (mon.0) 8158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:56:41.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[45495]: audit 2022-04-22T19:56:40.064513+0000 mon.a (mon.0) 8159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:56:41.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[45495]: audit 2022-04-22T19:56:40.064763+0000 mon.a (mon.0) 8160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:41.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[46934]: audit 2022-04-22T19:56:39.708142+0000 mon.a (mon.0) 8156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:56:41.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[46934]: audit 2022-04-22T19:56:39.711228+0000 mon.a (mon.0) 8157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:56:41.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[46934]: audit 2022-04-22T19:56:39.712928+0000 mon.a (mon.0) 8158 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:56:41.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[46934]: audit 2022-04-22T19:56:40.064513+0000 mon.a (mon.0) 8159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:56:41.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:40 smithi115 bash[46934]: audit 2022-04-22T19:56:40.064763+0000 mon.a (mon.0) 8160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:41 smithi121 bash[36902]: cluster 2022-04-22T19:56:40.481393+0000 mgr.y (mgr.34107) 5165 : cluster [DBG] pgmap v6460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:41 smithi121 bash[36902]: audit 2022-04-22T19:56:41.417383+0000 mon.a (mon.0) 8161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:41 smithi115 bash[45495]: cluster 2022-04-22T19:56:40.481393+0000 mgr.y (mgr.34107) 5165 : cluster [DBG] pgmap v6460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:41 smithi115 bash[45495]: audit 2022-04-22T19:56:41.417383+0000 mon.a (mon.0) 8161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:41 smithi115 bash[46934]: cluster 2022-04-22T19:56:40.481393+0000 mgr.y (mgr.34107) 5165 : cluster [DBG] pgmap v6460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:41 smithi115 bash[46934]: audit 2022-04-22T19:56:41.417383+0000 mon.a (mon.0) 8161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:42.198Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:43 smithi121 bash[36902]: cluster 2022-04-22T19:56:42.482501+0000 mgr.y (mgr.34107) 5166 : cluster [DBG] pgmap v6461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:43 smithi121 bash[36902]: audit 2022-04-22T19:56:42.872965+0000 mon.a (mon.0) 8162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:43 smithi115 bash[45495]: cluster 2022-04-22T19:56:42.482501+0000 mgr.y (mgr.34107) 5166 : cluster [DBG] pgmap v6461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:43 smithi115 bash[45495]: audit 2022-04-22T19:56:42.872965+0000 mon.a (mon.0) 8162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:43 smithi115 bash[46934]: cluster 2022-04-22T19:56:42.482501+0000 mgr.y (mgr.34107) 5166 : cluster [DBG] pgmap v6461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:43 smithi115 bash[46934]: audit 2022-04-22T19:56:42.872965+0000 mon.a (mon.0) 8162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:44.073Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:45 smithi121 bash[36902]: cluster 2022-04-22T19:56:44.483629+0000 mgr.y (mgr.34107) 5167 : cluster [DBG] pgmap v6462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:46.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:45 smithi121 bash[36902]: audit 2022-04-22T19:56:45.067189+0000 mon.a (mon.0) 8163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:45 smithi115 bash[45495]: cluster 2022-04-22T19:56:44.483629+0000 mgr.y (mgr.34107) 5167 : cluster [DBG] pgmap v6462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:45 smithi115 bash[45495]: audit 2022-04-22T19:56:45.067189+0000 mon.a (mon.0) 8163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:45 smithi115 bash[46934]: cluster 2022-04-22T19:56:44.483629+0000 mgr.y (mgr.34107) 5167 : cluster [DBG] pgmap v6462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:45 smithi115 bash[46934]: audit 2022-04-22T19:56:45.067189+0000 mon.a (mon.0) 8163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:46 smithi121 bash[36902]: audit 2022-04-22T19:56:46.420469+0000 mon.a (mon.0) 8164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:46 smithi115 bash[45495]: audit 2022-04-22T19:56:46.420469+0000 mon.a (mon.0) 8164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:46 smithi115 bash[46934]: audit 2022-04-22T19:56:46.420469+0000 mon.a (mon.0) 8164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:47 smithi121 bash[36902]: cluster 2022-04-22T19:56:46.484636+0000 mgr.y (mgr.34107) 5168 : cluster [DBG] pgmap v6463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:47 smithi115 bash[45495]: cluster 2022-04-22T19:56:46.484636+0000 mgr.y (mgr.34107) 5168 : cluster [DBG] pgmap v6463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:47 smithi115 bash[46934]: cluster 2022-04-22T19:56:46.484636+0000 mgr.y (mgr.34107) 5168 : cluster [DBG] pgmap v6463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:48 smithi121 bash[36902]: audit 2022-04-22T19:56:47.876078+0000 mon.a (mon.0) 8165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:48 smithi115 bash[46934]: audit 2022-04-22T19:56:47.876078+0000 mon.a (mon.0) 8165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:48 smithi115 bash[45495]: audit 2022-04-22T19:56:47.876078+0000 mon.a (mon.0) 8165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:49 smithi115 bash[46934]: cluster 2022-04-22T19:56:48.485831+0000 mgr.y (mgr.34107) 5169 : cluster [DBG] pgmap v6464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:49 smithi115 bash[45495]: cluster 2022-04-22T19:56:48.485831+0000 mgr.y (mgr.34107) 5169 : cluster [DBG] pgmap v6464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:49 smithi121 bash[36902]: cluster 2022-04-22T19:56:48.485831+0000 mgr.y (mgr.34107) 5169 : cluster [DBG] pgmap v6464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:50 smithi121 bash[36902]: audit 2022-04-22T19:56:50.070855+0000 mon.a (mon.0) 8166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:50 smithi115 bash[46934]: audit 2022-04-22T19:56:50.070855+0000 mon.a (mon.0) 8166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:50 smithi115 bash[45495]: audit 2022-04-22T19:56:50.070855+0000 mon.a (mon.0) 8166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:51 smithi121 bash[36902]: cluster 2022-04-22T19:56:50.486932+0000 mgr.y (mgr.34107) 5170 : cluster [DBG] pgmap v6465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:51 smithi121 bash[36902]: audit 2022-04-22T19:56:51.423621+0000 mon.a (mon.0) 8167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:51 smithi115 bash[46934]: cluster 2022-04-22T19:56:50.486932+0000 mgr.y (mgr.34107) 5170 : cluster [DBG] pgmap v6465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:51 smithi115 bash[46934]: audit 2022-04-22T19:56:51.423621+0000 mon.a (mon.0) 8167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:51 smithi115 bash[45495]: cluster 2022-04-22T19:56:50.486932+0000 mgr.y (mgr.34107) 5170 : cluster [DBG] pgmap v6465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:51 smithi115 bash[45495]: audit 2022-04-22T19:56:51.423621+0000 mon.a (mon.0) 8167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:52.198Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:53 smithi121 bash[36902]: cluster 2022-04-22T19:56:52.488051+0000 mgr.y (mgr.34107) 5171 : cluster [DBG] pgmap v6466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:53 smithi121 bash[36902]: audit 2022-04-22T19:56:52.879073+0000 mon.a (mon.0) 8168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:53 smithi115 bash[45495]: cluster 2022-04-22T19:56:52.488051+0000 mgr.y (mgr.34107) 5171 : cluster [DBG] pgmap v6466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:54.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:53 smithi115 bash[45495]: audit 2022-04-22T19:56:52.879073+0000 mon.a (mon.0) 8168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:54.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:53 smithi115 bash[46934]: cluster 2022-04-22T19:56:52.488051+0000 mgr.y (mgr.34107) 5171 : cluster [DBG] pgmap v6466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:54.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:53 smithi115 bash[46934]: audit 2022-04-22T19:56:52.879073+0000 mon.a (mon.0) 8168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:56:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:56:54.076Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:56:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:55 smithi121 bash[36902]: cluster 2022-04-22T19:56:54.489160+0000 mgr.y (mgr.34107) 5172 : cluster [DBG] pgmap v6467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:55 smithi121 bash[36902]: audit 2022-04-22T19:56:55.074268+0000 mon.a (mon.0) 8169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:55 smithi115 bash[45495]: cluster 2022-04-22T19:56:54.489160+0000 mgr.y (mgr.34107) 5172 : cluster [DBG] pgmap v6467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:55 smithi115 bash[45495]: audit 2022-04-22T19:56:55.074268+0000 mon.a (mon.0) 8169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:55 smithi115 bash[46934]: cluster 2022-04-22T19:56:54.489160+0000 mgr.y (mgr.34107) 5172 : cluster [DBG] pgmap v6467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:55 smithi115 bash[46934]: audit 2022-04-22T19:56:55.074268+0000 mon.a (mon.0) 8169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:56:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:56 smithi121 bash[36902]: audit 2022-04-22T19:56:56.426600+0000 mon.a (mon.0) 8170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:56 smithi115 bash[45495]: audit 2022-04-22T19:56:56.426600+0000 mon.a (mon.0) 8170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:57.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:56 smithi115 bash[46934]: audit 2022-04-22T19:56:56.426600+0000 mon.a (mon.0) 8170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:56:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:57 smithi121 bash[36902]: cluster 2022-04-22T19:56:56.490127+0000 mgr.y (mgr.34107) 5173 : cluster [DBG] pgmap v6468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:57 smithi115 bash[45495]: cluster 2022-04-22T19:56:56.490127+0000 mgr.y (mgr.34107) 5173 : cluster [DBG] pgmap v6468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:57 smithi115 bash[46934]: cluster 2022-04-22T19:56:56.490127+0000 mgr.y (mgr.34107) 5173 : cluster [DBG] pgmap v6468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:56:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:58 smithi121 bash[36902]: audit 2022-04-22T19:56:57.882082+0000 mon.a (mon.0) 8171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:58 smithi115 bash[45495]: audit 2022-04-22T19:56:57.882082+0000 mon.a (mon.0) 8171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:56:59.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:58 smithi115 bash[46934]: audit 2022-04-22T19:56:57.882082+0000 mon.a (mon.0) 8171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:56:59 smithi121 bash[36902]: cluster 2022-04-22T19:56:58.491314+0000 mgr.y (mgr.34107) 5174 : cluster [DBG] pgmap v6469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:56:59 smithi115 bash[45495]: cluster 2022-04-22T19:56:58.491314+0000 mgr.y (mgr.34107) 5174 : cluster [DBG] pgmap v6469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:56:59 smithi115 bash[46934]: cluster 2022-04-22T19:56:58.491314+0000 mgr.y (mgr.34107) 5174 : cluster [DBG] pgmap v6469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:00 smithi121 bash[36902]: audit 2022-04-22T19:57:00.078266+0000 mon.a (mon.0) 8172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:00 smithi115 bash[45495]: audit 2022-04-22T19:57:00.078266+0000 mon.a (mon.0) 8172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:00 smithi115 bash[46934]: audit 2022-04-22T19:57:00.078266+0000 mon.a (mon.0) 8172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:01 smithi121 bash[36902]: cluster 2022-04-22T19:57:00.492372+0000 mgr.y (mgr.34107) 5175 : cluster [DBG] pgmap v6470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:01 smithi121 bash[36902]: audit 2022-04-22T19:57:01.429875+0000 mon.a (mon.0) 8173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:01 smithi115 bash[45495]: cluster 2022-04-22T19:57:00.492372+0000 mgr.y (mgr.34107) 5175 : cluster [DBG] pgmap v6470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:02.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:01 smithi115 bash[45495]: audit 2022-04-22T19:57:01.429875+0000 mon.a (mon.0) 8173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:01 smithi115 bash[46934]: cluster 2022-04-22T19:57:00.492372+0000 mgr.y (mgr.34107) 5175 : cluster [DBG] pgmap v6470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:01 smithi115 bash[46934]: audit 2022-04-22T19:57:01.429875+0000 mon.a (mon.0) 8173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:02.200Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:03 smithi121 bash[36902]: cluster 2022-04-22T19:57:02.493527+0000 mgr.y (mgr.34107) 5176 : cluster [DBG] pgmap v6471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:03 smithi121 bash[36902]: audit 2022-04-22T19:57:02.884993+0000 mon.a (mon.0) 8174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:03 smithi115 bash[45495]: cluster 2022-04-22T19:57:02.493527+0000 mgr.y (mgr.34107) 5176 : cluster [DBG] pgmap v6471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:03 smithi115 bash[45495]: audit 2022-04-22T19:57:02.884993+0000 mon.a (mon.0) 8174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:03 smithi115 bash[46934]: cluster 2022-04-22T19:57:02.493527+0000 mgr.y (mgr.34107) 5176 : cluster [DBG] pgmap v6471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:03 smithi115 bash[46934]: audit 2022-04-22T19:57:02.884993+0000 mon.a (mon.0) 8174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:04.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:05.282 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:57:05.627 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:57:05.627 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:57:05.627 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:57:05.627 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:57:05.627 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:57:05.628 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:57:05.629 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:57:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:05 smithi121 bash[36902]: cluster 2022-04-22T19:57:04.494587+0000 mgr.y (mgr.34107) 5177 : cluster [DBG] pgmap v6472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:05 smithi121 bash[36902]: audit 2022-04-22T19:57:05.081681+0000 mon.a (mon.0) 8175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:06.037 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:57:06.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:57:06.039 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:57:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:05 smithi115 bash[45495]: cluster 2022-04-22T19:57:04.494587+0000 mgr.y (mgr.34107) 5177 : cluster [DBG] pgmap v6472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:05 smithi115 bash[45495]: audit 2022-04-22T19:57:05.081681+0000 mon.a (mon.0) 8175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:05 smithi115 bash[46934]: cluster 2022-04-22T19:57:04.494587+0000 mgr.y (mgr.34107) 5177 : cluster [DBG] pgmap v6472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:05 smithi115 bash[46934]: audit 2022-04-22T19:57:05.081681+0000 mon.a (mon.0) 8175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:06 smithi121 bash[36902]: audit 2022-04-22T19:57:05.258819+0000 mgr.y (mgr.34107) 5178 : audit [DBG] from='client.50722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:06 smithi121 bash[36902]: audit 2022-04-22T19:57:05.600245+0000 mgr.y (mgr.34107) 5179 : audit [DBG] from='client.50728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:06 smithi121 bash[36902]: audit 2022-04-22T19:57:06.028877+0000 mon.c (mon.1) 219 : audit [DBG] from='client.? 172.21.15.115:0/3100004160' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:06 smithi121 bash[36902]: audit 2022-04-22T19:57:06.432980+0000 mon.a (mon.0) 8176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[45495]: audit 2022-04-22T19:57:05.258819+0000 mgr.y (mgr.34107) 5178 : audit [DBG] from='client.50722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[45495]: audit 2022-04-22T19:57:05.600245+0000 mgr.y (mgr.34107) 5179 : audit [DBG] from='client.50728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[45495]: audit 2022-04-22T19:57:06.028877+0000 mon.c (mon.1) 219 : audit [DBG] from='client.? 172.21.15.115:0/3100004160' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[45495]: audit 2022-04-22T19:57:06.432980+0000 mon.a (mon.0) 8176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[46934]: audit 2022-04-22T19:57:05.258819+0000 mgr.y (mgr.34107) 5178 : audit [DBG] from='client.50722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[46934]: audit 2022-04-22T19:57:05.600245+0000 mgr.y (mgr.34107) 5179 : audit [DBG] from='client.50728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[46934]: audit 2022-04-22T19:57:06.028877+0000 mon.c (mon.1) 219 : audit [DBG] from='client.? 172.21.15.115:0/3100004160' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:06 smithi115 bash[46934]: audit 2022-04-22T19:57:06.432980+0000 mon.a (mon.0) 8176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:07 smithi121 bash[36902]: cluster 2022-04-22T19:57:06.495737+0000 mgr.y (mgr.34107) 5180 : cluster [DBG] pgmap v6473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:07 smithi115 bash[45495]: cluster 2022-04-22T19:57:06.495737+0000 mgr.y (mgr.34107) 5180 : cluster [DBG] pgmap v6473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:07 smithi115 bash[46934]: cluster 2022-04-22T19:57:06.495737+0000 mgr.y (mgr.34107) 5180 : cluster [DBG] pgmap v6473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:08 smithi121 bash[36902]: audit 2022-04-22T19:57:07.888021+0000 mon.a (mon.0) 8177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:08 smithi115 bash[45495]: audit 2022-04-22T19:57:07.888021+0000 mon.a (mon.0) 8177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:08 smithi115 bash[46934]: audit 2022-04-22T19:57:07.888021+0000 mon.a (mon.0) 8177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:09 smithi121 bash[36902]: cluster 2022-04-22T19:57:08.496909+0000 mgr.y (mgr.34107) 5181 : cluster [DBG] pgmap v6474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:09 smithi115 bash[45495]: cluster 2022-04-22T19:57:08.496909+0000 mgr.y (mgr.34107) 5181 : cluster [DBG] pgmap v6474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:09 smithi115 bash[46934]: cluster 2022-04-22T19:57:08.496909+0000 mgr.y (mgr.34107) 5181 : cluster [DBG] pgmap v6474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:10 smithi121 bash[36902]: audit 2022-04-22T19:57:10.085308+0000 mon.a (mon.0) 8178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:10 smithi115 bash[45495]: audit 2022-04-22T19:57:10.085308+0000 mon.a (mon.0) 8178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:10 smithi115 bash[46934]: audit 2022-04-22T19:57:10.085308+0000 mon.a (mon.0) 8178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:11 smithi121 bash[36902]: cluster 2022-04-22T19:57:10.498022+0000 mgr.y (mgr.34107) 5182 : cluster [DBG] pgmap v6475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:11 smithi121 bash[36902]: audit 2022-04-22T19:57:11.435824+0000 mon.a (mon.0) 8179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:11 smithi115 bash[45495]: cluster 2022-04-22T19:57:10.498022+0000 mgr.y (mgr.34107) 5182 : cluster [DBG] pgmap v6475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:11 smithi115 bash[45495]: audit 2022-04-22T19:57:11.435824+0000 mon.a (mon.0) 8179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:11 smithi115 bash[46934]: cluster 2022-04-22T19:57:10.498022+0000 mgr.y (mgr.34107) 5182 : cluster [DBG] pgmap v6475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:11 smithi115 bash[46934]: audit 2022-04-22T19:57:11.435824+0000 mon.a (mon.0) 8179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:12.198Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:13 smithi121 bash[36902]: cluster 2022-04-22T19:57:12.499080+0000 mgr.y (mgr.34107) 5183 : cluster [DBG] pgmap v6476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:13 smithi121 bash[36902]: audit 2022-04-22T19:57:12.891016+0000 mon.a (mon.0) 8180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:13 smithi115 bash[45495]: cluster 2022-04-22T19:57:12.499080+0000 mgr.y (mgr.34107) 5183 : cluster [DBG] pgmap v6476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:13 smithi115 bash[45495]: audit 2022-04-22T19:57:12.891016+0000 mon.a (mon.0) 8180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:13 smithi115 bash[46934]: cluster 2022-04-22T19:57:12.499080+0000 mgr.y (mgr.34107) 5183 : cluster [DBG] pgmap v6476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:13 smithi115 bash[46934]: audit 2022-04-22T19:57:12.891016+0000 mon.a (mon.0) 8180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:14.075Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:15 smithi121 bash[36902]: cluster 2022-04-22T19:57:14.500138+0000 mgr.y (mgr.34107) 5184 : cluster [DBG] pgmap v6477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:15 smithi121 bash[36902]: audit 2022-04-22T19:57:15.013015+0000 mon.a (mon.0) 8181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:57:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:15 smithi121 bash[36902]: audit 2022-04-22T19:57:15.013891+0000 mon.a (mon.0) 8182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:57:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:15 smithi121 bash[36902]: audit 2022-04-22T19:57:15.089060+0000 mon.a (mon.0) 8183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[45495]: cluster 2022-04-22T19:57:14.500138+0000 mgr.y (mgr.34107) 5184 : cluster [DBG] pgmap v6477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[45495]: audit 2022-04-22T19:57:15.013015+0000 mon.a (mon.0) 8181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:57:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[45495]: audit 2022-04-22T19:57:15.013891+0000 mon.a (mon.0) 8182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:57:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[45495]: audit 2022-04-22T19:57:15.089060+0000 mon.a (mon.0) 8183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[46934]: cluster 2022-04-22T19:57:14.500138+0000 mgr.y (mgr.34107) 5184 : cluster [DBG] pgmap v6477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[46934]: audit 2022-04-22T19:57:15.013015+0000 mon.a (mon.0) 8181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:57:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[46934]: audit 2022-04-22T19:57:15.013891+0000 mon.a (mon.0) 8182 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:57:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:15 smithi115 bash[46934]: audit 2022-04-22T19:57:15.089060+0000 mon.a (mon.0) 8183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:16 smithi121 bash[36902]: audit 2022-04-22T19:57:16.439019+0000 mon.a (mon.0) 8184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:16 smithi115 bash[45495]: audit 2022-04-22T19:57:16.439019+0000 mon.a (mon.0) 8184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:17.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:16 smithi115 bash[46934]: audit 2022-04-22T19:57:16.439019+0000 mon.a (mon.0) 8184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:17 smithi121 bash[36902]: cluster 2022-04-22T19:57:16.501097+0000 mgr.y (mgr.34107) 5185 : cluster [DBG] pgmap v6478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:17 smithi115 bash[45495]: cluster 2022-04-22T19:57:16.501097+0000 mgr.y (mgr.34107) 5185 : cluster [DBG] pgmap v6478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:17 smithi115 bash[46934]: cluster 2022-04-22T19:57:16.501097+0000 mgr.y (mgr.34107) 5185 : cluster [DBG] pgmap v6478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:18 smithi121 bash[36902]: audit 2022-04-22T19:57:17.894156+0000 mon.a (mon.0) 8185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:18 smithi115 bash[45495]: audit 2022-04-22T19:57:17.894156+0000 mon.a (mon.0) 8185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:18 smithi115 bash[46934]: audit 2022-04-22T19:57:17.894156+0000 mon.a (mon.0) 8185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:19 smithi121 bash[36902]: cluster 2022-04-22T19:57:18.502312+0000 mgr.y (mgr.34107) 5186 : cluster [DBG] pgmap v6479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:19 smithi115 bash[45495]: cluster 2022-04-22T19:57:18.502312+0000 mgr.y (mgr.34107) 5186 : cluster [DBG] pgmap v6479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:19 smithi115 bash[46934]: cluster 2022-04-22T19:57:18.502312+0000 mgr.y (mgr.34107) 5186 : cluster [DBG] pgmap v6479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:20 smithi121 bash[36902]: audit 2022-04-22T19:57:20.092521+0000 mon.a (mon.0) 8186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:20 smithi115 bash[45495]: audit 2022-04-22T19:57:20.092521+0000 mon.a (mon.0) 8186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:20 smithi115 bash[46934]: audit 2022-04-22T19:57:20.092521+0000 mon.a (mon.0) 8186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:21 smithi121 bash[36902]: cluster 2022-04-22T19:57:20.503360+0000 mgr.y (mgr.34107) 5187 : cluster [DBG] pgmap v6480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:21 smithi121 bash[36902]: audit 2022-04-22T19:57:21.442364+0000 mon.a (mon.0) 8187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:21 smithi115 bash[46934]: cluster 2022-04-22T19:57:20.503360+0000 mgr.y (mgr.34107) 5187 : cluster [DBG] pgmap v6480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:21 smithi115 bash[46934]: audit 2022-04-22T19:57:21.442364+0000 mon.a (mon.0) 8187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:21 smithi115 bash[45495]: cluster 2022-04-22T19:57:20.503360+0000 mgr.y (mgr.34107) 5187 : cluster [DBG] pgmap v6480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:21 smithi115 bash[45495]: audit 2022-04-22T19:57:21.442364+0000 mon.a (mon.0) 8187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:22.200Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:23 smithi121 bash[36902]: cluster 2022-04-22T19:57:22.504482+0000 mgr.y (mgr.34107) 5188 : cluster [DBG] pgmap v6481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:23 smithi121 bash[36902]: audit 2022-04-22T19:57:22.897275+0000 mon.a (mon.0) 8188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:23 smithi115 bash[45495]: cluster 2022-04-22T19:57:22.504482+0000 mgr.y (mgr.34107) 5188 : cluster [DBG] pgmap v6481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:23 smithi115 bash[45495]: audit 2022-04-22T19:57:22.897275+0000 mon.a (mon.0) 8188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:23 smithi115 bash[46934]: cluster 2022-04-22T19:57:22.504482+0000 mgr.y (mgr.34107) 5188 : cluster [DBG] pgmap v6481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:23 smithi115 bash[46934]: audit 2022-04-22T19:57:22.897275+0000 mon.a (mon.0) 8188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:24.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:25 smithi121 bash[36902]: cluster 2022-04-22T19:57:24.505651+0000 mgr.y (mgr.34107) 5189 : cluster [DBG] pgmap v6482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:25 smithi121 bash[36902]: audit 2022-04-22T19:57:25.096128+0000 mon.a (mon.0) 8189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:25 smithi115 bash[45495]: cluster 2022-04-22T19:57:24.505651+0000 mgr.y (mgr.34107) 5189 : cluster [DBG] pgmap v6482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:25 smithi115 bash[45495]: audit 2022-04-22T19:57:25.096128+0000 mon.a (mon.0) 8189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:25 smithi115 bash[46934]: cluster 2022-04-22T19:57:24.505651+0000 mgr.y (mgr.34107) 5189 : cluster [DBG] pgmap v6482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:25 smithi115 bash[46934]: audit 2022-04-22T19:57:25.096128+0000 mon.a (mon.0) 8189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:26 smithi121 bash[36902]: audit 2022-04-22T19:57:26.445496+0000 mon.a (mon.0) 8190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:26 smithi115 bash[45495]: audit 2022-04-22T19:57:26.445496+0000 mon.a (mon.0) 8190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:26 smithi115 bash[46934]: audit 2022-04-22T19:57:26.445496+0000 mon.a (mon.0) 8190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:27 smithi121 bash[36902]: cluster 2022-04-22T19:57:26.506781+0000 mgr.y (mgr.34107) 5190 : cluster [DBG] pgmap v6483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:27 smithi115 bash[45495]: cluster 2022-04-22T19:57:26.506781+0000 mgr.y (mgr.34107) 5190 : cluster [DBG] pgmap v6483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:27 smithi115 bash[46934]: cluster 2022-04-22T19:57:26.506781+0000 mgr.y (mgr.34107) 5190 : cluster [DBG] pgmap v6483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:28 smithi121 bash[36902]: audit 2022-04-22T19:57:27.900194+0000 mon.a (mon.0) 8191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:28 smithi115 bash[45495]: audit 2022-04-22T19:57:27.900194+0000 mon.a (mon.0) 8191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:28 smithi115 bash[46934]: audit 2022-04-22T19:57:27.900194+0000 mon.a (mon.0) 8191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:29 smithi115 bash[45495]: cluster 2022-04-22T19:57:28.507948+0000 mgr.y (mgr.34107) 5191 : cluster [DBG] pgmap v6484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:29 smithi115 bash[46934]: cluster 2022-04-22T19:57:28.507948+0000 mgr.y (mgr.34107) 5191 : cluster [DBG] pgmap v6484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:29 smithi121 bash[36902]: cluster 2022-04-22T19:57:28.507948+0000 mgr.y (mgr.34107) 5191 : cluster [DBG] pgmap v6484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:30 smithi115 bash[45495]: audit 2022-04-22T19:57:30.100091+0000 mon.a (mon.0) 8192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:30 smithi115 bash[46934]: audit 2022-04-22T19:57:30.100091+0000 mon.a (mon.0) 8192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:30 smithi121 bash[36902]: audit 2022-04-22T19:57:30.100091+0000 mon.a (mon.0) 8192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:31 smithi115 bash[45495]: cluster 2022-04-22T19:57:30.509810+0000 mgr.y (mgr.34107) 5192 : cluster [DBG] pgmap v6485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:31 smithi115 bash[45495]: audit 2022-04-22T19:57:31.448645+0000 mon.a (mon.0) 8193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:31 smithi115 bash[46934]: cluster 2022-04-22T19:57:30.509810+0000 mgr.y (mgr.34107) 5192 : cluster [DBG] pgmap v6485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:31 smithi115 bash[46934]: audit 2022-04-22T19:57:31.448645+0000 mon.a (mon.0) 8193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:31 smithi121 bash[36902]: cluster 2022-04-22T19:57:30.509810+0000 mgr.y (mgr.34107) 5192 : cluster [DBG] pgmap v6485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:31 smithi121 bash[36902]: audit 2022-04-22T19:57:31.448645+0000 mon.a (mon.0) 8193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:32.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:33 smithi115 bash[46934]: cluster 2022-04-22T19:57:32.511007+0000 mgr.y (mgr.34107) 5193 : cluster [DBG] pgmap v6486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:33 smithi115 bash[46934]: audit 2022-04-22T19:57:32.903281+0000 mon.a (mon.0) 8194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:33 smithi115 bash[45495]: cluster 2022-04-22T19:57:32.511007+0000 mgr.y (mgr.34107) 5193 : cluster [DBG] pgmap v6486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:34.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:33 smithi115 bash[45495]: audit 2022-04-22T19:57:32.903281+0000 mon.a (mon.0) 8194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:33 smithi121 bash[36902]: cluster 2022-04-22T19:57:32.511007+0000 mgr.y (mgr.34107) 5193 : cluster [DBG] pgmap v6486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:33 smithi121 bash[36902]: audit 2022-04-22T19:57:32.903281+0000 mon.a (mon.0) 8194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:34.076Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:36.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:35 smithi115 bash[45495]: cluster 2022-04-22T19:57:34.512088+0000 mgr.y (mgr.34107) 5194 : cluster [DBG] pgmap v6487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:36.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:35 smithi115 bash[45495]: audit 2022-04-22T19:57:35.103830+0000 mon.a (mon.0) 8195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:35 smithi115 bash[46934]: cluster 2022-04-22T19:57:34.512088+0000 mgr.y (mgr.34107) 5194 : cluster [DBG] pgmap v6487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:36.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:35 smithi115 bash[46934]: audit 2022-04-22T19:57:35.103830+0000 mon.a (mon.0) 8195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:35 smithi121 bash[36902]: cluster 2022-04-22T19:57:34.512088+0000 mgr.y (mgr.34107) 5194 : cluster [DBG] pgmap v6487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:35 smithi121 bash[36902]: audit 2022-04-22T19:57:35.103830+0000 mon.a (mon.0) 8195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:36.426 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:57:36.776 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:57:36.777 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:57:36.777 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:57:36.777 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:57:36.777 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:57:36.777 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:57:36.778 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:57:36.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:57:36.780 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:57:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:36 smithi115 bash[45495]: audit 2022-04-22T19:57:36.451583+0000 mon.a (mon.0) 8196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:36 smithi115 bash[46934]: audit 2022-04-22T19:57:36.451583+0000 mon.a (mon.0) 8196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:37.175 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:57:37.176 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:57:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:36 smithi121 bash[36902]: audit 2022-04-22T19:57:36.451583+0000 mon.a (mon.0) 8196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[45495]: audit 2022-04-22T19:57:36.402628+0000 mgr.y (mgr.34107) 5195 : audit [DBG] from='client.40806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[45495]: cluster 2022-04-22T19:57:36.513150+0000 mgr.y (mgr.34107) 5196 : cluster [DBG] pgmap v6488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[45495]: audit 2022-04-22T19:57:36.746937+0000 mgr.y (mgr.34107) 5197 : audit [DBG] from='client.50746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[45495]: audit 2022-04-22T19:57:37.165800+0000 mon.a (mon.0) 8197 : audit [DBG] from='client.? 172.21.15.115:0/1115107128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[46934]: audit 2022-04-22T19:57:36.402628+0000 mgr.y (mgr.34107) 5195 : audit [DBG] from='client.40806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[46934]: cluster 2022-04-22T19:57:36.513150+0000 mgr.y (mgr.34107) 5196 : cluster [DBG] pgmap v6488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[46934]: audit 2022-04-22T19:57:36.746937+0000 mgr.y (mgr.34107) 5197 : audit [DBG] from='client.50746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:37 smithi115 bash[46934]: audit 2022-04-22T19:57:37.165800+0000 mon.a (mon.0) 8197 : audit [DBG] from='client.? 172.21.15.115:0/1115107128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:37 smithi121 bash[36902]: audit 2022-04-22T19:57:36.402628+0000 mgr.y (mgr.34107) 5195 : audit [DBG] from='client.40806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:37 smithi121 bash[36902]: cluster 2022-04-22T19:57:36.513150+0000 mgr.y (mgr.34107) 5196 : cluster [DBG] pgmap v6488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:37 smithi121 bash[36902]: audit 2022-04-22T19:57:36.746937+0000 mgr.y (mgr.34107) 5197 : audit [DBG] from='client.50746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:57:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:37 smithi121 bash[36902]: audit 2022-04-22T19:57:37.165800+0000 mon.a (mon.0) 8197 : audit [DBG] from='client.? 172.21.15.115:0/1115107128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:57:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:38 smithi121 bash[36902]: audit 2022-04-22T19:57:37.906320+0000 mon.a (mon.0) 8198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:38 smithi115 bash[45495]: audit 2022-04-22T19:57:37.906320+0000 mon.a (mon.0) 8198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:38 smithi115 bash[46934]: audit 2022-04-22T19:57:37.906320+0000 mon.a (mon.0) 8198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:40.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:39 smithi115 bash[45495]: cluster 2022-04-22T19:57:38.514048+0000 mgr.y (mgr.34107) 5198 : cluster [DBG] pgmap v6489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:40.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:39 smithi115 bash[46934]: cluster 2022-04-22T19:57:38.514048+0000 mgr.y (mgr.34107) 5198 : cluster [DBG] pgmap v6489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:40.107 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:39 smithi121 bash[36902]: cluster 2022-04-22T19:57:38.514048+0000 mgr.y (mgr.34107) 5198 : cluster [DBG] pgmap v6489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:40 smithi121 bash[36902]: audit 2022-04-22T19:57:40.073042+0000 mon.a (mon.0) 8199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:57:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:40 smithi121 bash[36902]: audit 2022-04-22T19:57:40.076040+0000 mon.a (mon.0) 8200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:57:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:40 smithi121 bash[36902]: audit 2022-04-22T19:57:40.077784+0000 mon.a (mon.0) 8201 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:57:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:40 smithi121 bash[36902]: audit 2022-04-22T19:57:40.107440+0000 mon.a (mon.0) 8202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:40 smithi121 bash[36902]: audit 2022-04-22T19:57:40.433368+0000 mon.a (mon.0) 8203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:57:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[46934]: audit 2022-04-22T19:57:40.073042+0000 mon.a (mon.0) 8199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:57:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[46934]: audit 2022-04-22T19:57:40.076040+0000 mon.a (mon.0) 8200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:57:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[46934]: audit 2022-04-22T19:57:40.077784+0000 mon.a (mon.0) 8201 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[46934]: audit 2022-04-22T19:57:40.107440+0000 mon.a (mon.0) 8202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[46934]: audit 2022-04-22T19:57:40.433368+0000 mon.a (mon.0) 8203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[45495]: audit 2022-04-22T19:57:40.073042+0000 mon.a (mon.0) 8199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[45495]: audit 2022-04-22T19:57:40.076040+0000 mon.a (mon.0) 8200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[45495]: audit 2022-04-22T19:57:40.077784+0000 mon.a (mon.0) 8201 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[45495]: audit 2022-04-22T19:57:40.107440+0000 mon.a (mon.0) 8202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:41.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:40 smithi115 bash[45495]: audit 2022-04-22T19:57:40.433368+0000 mon.a (mon.0) 8203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:57:42.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:41 smithi115 bash[45495]: cluster 2022-04-22T19:57:40.515161+0000 mgr.y (mgr.34107) 5199 : cluster [DBG] pgmap v6490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:42.212 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:41 smithi115 bash[45495]: audit 2022-04-22T19:57:41.454414+0000 mon.a (mon.0) 8204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:42.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:41 smithi115 bash[46934]: cluster 2022-04-22T19:57:40.515161+0000 mgr.y (mgr.34107) 5199 : cluster [DBG] pgmap v6490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:42.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:41 smithi115 bash[46934]: audit 2022-04-22T19:57:41.454414+0000 mon.a (mon.0) 8204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:41 smithi121 bash[36902]: cluster 2022-04-22T19:57:40.515161+0000 mgr.y (mgr.34107) 5199 : cluster [DBG] pgmap v6490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:41 smithi121 bash[36902]: audit 2022-04-22T19:57:41.454414+0000 mon.a (mon.0) 8204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:42.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:44.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:43 smithi115 bash[45495]: cluster 2022-04-22T19:57:42.516326+0000 mgr.y (mgr.34107) 5200 : cluster [DBG] pgmap v6491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:44.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:43 smithi115 bash[45495]: audit 2022-04-22T19:57:42.909486+0000 mon.a (mon.0) 8205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:44.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:43 smithi115 bash[46934]: cluster 2022-04-22T19:57:42.516326+0000 mgr.y (mgr.34107) 5200 : cluster [DBG] pgmap v6491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:44.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:43 smithi115 bash[46934]: audit 2022-04-22T19:57:42.909486+0000 mon.a (mon.0) 8205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:43 smithi121 bash[36902]: cluster 2022-04-22T19:57:42.516326+0000 mgr.y (mgr.34107) 5200 : cluster [DBG] pgmap v6491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:43 smithi121 bash[36902]: audit 2022-04-22T19:57:42.909486+0000 mon.a (mon.0) 8205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:44.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:45 smithi121 bash[36902]: cluster 2022-04-22T19:57:44.517391+0000 mgr.y (mgr.34107) 5201 : cluster [DBG] pgmap v6492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:45 smithi121 bash[36902]: audit 2022-04-22T19:57:45.110951+0000 mon.a (mon.0) 8206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:45 smithi115 bash[46934]: cluster 2022-04-22T19:57:44.517391+0000 mgr.y (mgr.34107) 5201 : cluster [DBG] pgmap v6492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:45 smithi115 bash[46934]: audit 2022-04-22T19:57:45.110951+0000 mon.a (mon.0) 8206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:45 smithi115 bash[45495]: cluster 2022-04-22T19:57:44.517391+0000 mgr.y (mgr.34107) 5201 : cluster [DBG] pgmap v6492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:45 smithi115 bash[45495]: audit 2022-04-22T19:57:45.110951+0000 mon.a (mon.0) 8206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:46 smithi121 bash[36902]: audit 2022-04-22T19:57:46.457620+0000 mon.a (mon.0) 8207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:46 smithi115 bash[46934]: audit 2022-04-22T19:57:46.457620+0000 mon.a (mon.0) 8207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:46 smithi115 bash[45495]: audit 2022-04-22T19:57:46.457620+0000 mon.a (mon.0) 8207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:47 smithi121 bash[36902]: cluster 2022-04-22T19:57:46.518062+0000 mgr.y (mgr.34107) 5202 : cluster [DBG] pgmap v6493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:47 smithi115 bash[45495]: cluster 2022-04-22T19:57:46.518062+0000 mgr.y (mgr.34107) 5202 : cluster [DBG] pgmap v6493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:47 smithi115 bash[46934]: cluster 2022-04-22T19:57:46.518062+0000 mgr.y (mgr.34107) 5202 : cluster [DBG] pgmap v6493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:48 smithi121 bash[36902]: audit 2022-04-22T19:57:47.912540+0000 mon.a (mon.0) 8208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:48 smithi115 bash[45495]: audit 2022-04-22T19:57:47.912540+0000 mon.a (mon.0) 8208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:48 smithi115 bash[46934]: audit 2022-04-22T19:57:47.912540+0000 mon.a (mon.0) 8208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:49 smithi121 bash[36902]: cluster 2022-04-22T19:57:48.519153+0000 mgr.y (mgr.34107) 5203 : cluster [DBG] pgmap v6494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:49 smithi115 bash[45495]: cluster 2022-04-22T19:57:48.519153+0000 mgr.y (mgr.34107) 5203 : cluster [DBG] pgmap v6494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:49 smithi115 bash[46934]: cluster 2022-04-22T19:57:48.519153+0000 mgr.y (mgr.34107) 5203 : cluster [DBG] pgmap v6494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:50 smithi121 bash[36902]: audit 2022-04-22T19:57:50.114914+0000 mon.a (mon.0) 8209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:50 smithi115 bash[45495]: audit 2022-04-22T19:57:50.114914+0000 mon.a (mon.0) 8209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:50 smithi115 bash[46934]: audit 2022-04-22T19:57:50.114914+0000 mon.a (mon.0) 8209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:52.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:51 smithi115 bash[45495]: cluster 2022-04-22T19:57:50.520245+0000 mgr.y (mgr.34107) 5204 : cluster [DBG] pgmap v6495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:52.211 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:51 smithi115 bash[45495]: audit 2022-04-22T19:57:51.460818+0000 mon.a (mon.0) 8210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:52.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:51 smithi115 bash[46934]: cluster 2022-04-22T19:57:50.520245+0000 mgr.y (mgr.34107) 5204 : cluster [DBG] pgmap v6495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:52.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:51 smithi115 bash[46934]: audit 2022-04-22T19:57:51.460818+0000 mon.a (mon.0) 8210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:51 smithi121 bash[36902]: cluster 2022-04-22T19:57:50.520245+0000 mgr.y (mgr.34107) 5204 : cluster [DBG] pgmap v6495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:51 smithi121 bash[36902]: audit 2022-04-22T19:57:51.460818+0000 mon.a (mon.0) 8210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:52.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:52 smithi121 bash[36902]: cluster 2022-04-22T19:57:52.521335+0000 mgr.y (mgr.34107) 5205 : cluster [DBG] pgmap v6496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:53.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:52 smithi115 bash[45495]: cluster 2022-04-22T19:57:52.521335+0000 mgr.y (mgr.34107) 5205 : cluster [DBG] pgmap v6496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:53.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:52 smithi115 bash[46934]: cluster 2022-04-22T19:57:52.521335+0000 mgr.y (mgr.34107) 5205 : cluster [DBG] pgmap v6496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:53 smithi121 bash[36902]: audit 2022-04-22T19:57:52.915358+0000 mon.a (mon.0) 8211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:53 smithi115 bash[45495]: audit 2022-04-22T19:57:52.915358+0000 mon.a (mon.0) 8211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:57:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:57:54.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:57:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:53 smithi115 bash[46934]: audit 2022-04-22T19:57:52.915358+0000 mon.a (mon.0) 8211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:54 smithi121 bash[36902]: cluster 2022-04-22T19:57:54.522087+0000 mgr.y (mgr.34107) 5206 : cluster [DBG] pgmap v6497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:54 smithi115 bash[45495]: cluster 2022-04-22T19:57:54.522087+0000 mgr.y (mgr.34107) 5206 : cluster [DBG] pgmap v6497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:54 smithi115 bash[46934]: cluster 2022-04-22T19:57:54.522087+0000 mgr.y (mgr.34107) 5206 : cluster [DBG] pgmap v6497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:55 smithi121 bash[36902]: audit 2022-04-22T19:57:55.118506+0000 mon.a (mon.0) 8212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:56.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:55 smithi115 bash[45495]: audit 2022-04-22T19:57:55.118506+0000 mon.a (mon.0) 8212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:56.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:55 smithi115 bash[46934]: audit 2022-04-22T19:57:55.118506+0000 mon.a (mon.0) 8212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:57:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:56 smithi121 bash[36902]: audit 2022-04-22T19:57:56.463378+0000 mon.a (mon.0) 8213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:56 smithi121 bash[36902]: cluster 2022-04-22T19:57:56.523098+0000 mgr.y (mgr.34107) 5207 : cluster [DBG] pgmap v6498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:56 smithi115 bash[45495]: audit 2022-04-22T19:57:56.463378+0000 mon.a (mon.0) 8213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:56 smithi115 bash[45495]: cluster 2022-04-22T19:57:56.523098+0000 mgr.y (mgr.34107) 5207 : cluster [DBG] pgmap v6498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:56 smithi115 bash[46934]: audit 2022-04-22T19:57:56.463378+0000 mon.a (mon.0) 8213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:57:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:56 smithi115 bash[46934]: cluster 2022-04-22T19:57:56.523098+0000 mgr.y (mgr.34107) 5207 : cluster [DBG] pgmap v6498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:57 smithi121 bash[36902]: audit 2022-04-22T19:57:57.918341+0000 mon.a (mon.0) 8214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:58.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:57 smithi115 bash[45495]: audit 2022-04-22T19:57:57.918341+0000 mon.a (mon.0) 8214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:57 smithi115 bash[46934]: audit 2022-04-22T19:57:57.918341+0000 mon.a (mon.0) 8214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:57:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:57:59 smithi121 bash[36902]: cluster 2022-04-22T19:57:58.524260+0000 mgr.y (mgr.34107) 5208 : cluster [DBG] pgmap v6499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:57:59 smithi115 bash[46934]: cluster 2022-04-22T19:57:58.524260+0000 mgr.y (mgr.34107) 5208 : cluster [DBG] pgmap v6499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:57:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:57:59 smithi115 bash[45495]: cluster 2022-04-22T19:57:58.524260+0000 mgr.y (mgr.34107) 5208 : cluster [DBG] pgmap v6499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:00 smithi121 bash[36902]: audit 2022-04-22T19:58:00.122262+0000 mon.a (mon.0) 8215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:00 smithi115 bash[45495]: audit 2022-04-22T19:58:00.122262+0000 mon.a (mon.0) 8215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:00.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:00 smithi115 bash[46934]: audit 2022-04-22T19:58:00.122262+0000 mon.a (mon.0) 8215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:01 smithi121 bash[36902]: cluster 2022-04-22T19:58:00.525375+0000 mgr.y (mgr.34107) 5209 : cluster [DBG] pgmap v6500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:01 smithi115 bash[45495]: cluster 2022-04-22T19:58:00.525375+0000 mgr.y (mgr.34107) 5209 : cluster [DBG] pgmap v6500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:01 smithi115 bash[46934]: cluster 2022-04-22T19:58:00.525375+0000 mgr.y (mgr.34107) 5209 : cluster [DBG] pgmap v6500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:02.466 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:02.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:02 smithi121 bash[36902]: audit 2022-04-22T19:58:01.465862+0000 mon.a (mon.0) 8216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:02.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:02 smithi115 bash[45495]: audit 2022-04-22T19:58:01.465862+0000 mon.a (mon.0) 8216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:02.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:02 smithi115 bash[46934]: audit 2022-04-22T19:58:01.465862+0000 mon.a (mon.0) 8216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:03 smithi121 bash[36902]: cluster 2022-04-22T19:58:02.526057+0000 mgr.y (mgr.34107) 5210 : cluster [DBG] pgmap v6501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:03 smithi121 bash[36902]: audit 2022-04-22T19:58:02.921478+0000 mon.a (mon.0) 8217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:03.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:03 smithi115 bash[46934]: cluster 2022-04-22T19:58:02.526057+0000 mgr.y (mgr.34107) 5210 : cluster [DBG] pgmap v6501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:03.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:03 smithi115 bash[46934]: audit 2022-04-22T19:58:02.921478+0000 mon.a (mon.0) 8217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:03 smithi115 bash[45495]: cluster 2022-04-22T19:58:02.526057+0000 mgr.y (mgr.34107) 5210 : cluster [DBG] pgmap v6501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:03 smithi115 bash[45495]: audit 2022-04-22T19:58:02.921478+0000 mon.a (mon.0) 8217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:04.078Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:05 smithi121 bash[36902]: cluster 2022-04-22T19:58:04.527159+0000 mgr.y (mgr.34107) 5211 : cluster [DBG] pgmap v6502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:05 smithi121 bash[36902]: audit 2022-04-22T19:58:05.125974+0000 mon.a (mon.0) 8218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:05 smithi115 bash[45495]: cluster 2022-04-22T19:58:04.527159+0000 mgr.y (mgr.34107) 5211 : cluster [DBG] pgmap v6502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:05 smithi115 bash[45495]: audit 2022-04-22T19:58:05.125974+0000 mon.a (mon.0) 8218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:05 smithi115 bash[46934]: cluster 2022-04-22T19:58:04.527159+0000 mgr.y (mgr.34107) 5211 : cluster [DBG] pgmap v6502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:05 smithi115 bash[46934]: audit 2022-04-22T19:58:05.125974+0000 mon.a (mon.0) 8218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:06 smithi121 bash[36902]: audit 2022-04-22T19:58:06.468742+0000 mon.a (mon.0) 8219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:06 smithi115 bash[45495]: audit 2022-04-22T19:58:06.468742+0000 mon.a (mon.0) 8219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:06 smithi115 bash[46934]: audit 2022-04-22T19:58:06.468742+0000 mon.a (mon.0) 8219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:07.561 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:58:07.902 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:58:07.903 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:58:07.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:58:07.904 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:58:07.904 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:58:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:07 smithi121 bash[36902]: cluster 2022-04-22T19:58:06.528215+0000 mgr.y (mgr.34107) 5212 : cluster [DBG] pgmap v6503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:07 smithi115 bash[45495]: cluster 2022-04-22T19:58:06.528215+0000 mgr.y (mgr.34107) 5212 : cluster [DBG] pgmap v6503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:07 smithi115 bash[46934]: cluster 2022-04-22T19:58:06.528215+0000 mgr.y (mgr.34107) 5212 : cluster [DBG] pgmap v6503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:08.300 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:58:08.301 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:58:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:08 smithi121 bash[36902]: audit 2022-04-22T19:58:07.532715+0000 mgr.y (mgr.34107) 5213 : audit [DBG] from='client.40824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:08 smithi121 bash[36902]: audit 2022-04-22T19:58:07.924292+0000 mon.a (mon.0) 8220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:08 smithi121 bash[36902]: audit 2022-04-22T19:58:08.290779+0000 mon.a (mon.0) 8221 : audit [DBG] from='client.? 172.21.15.115:0/786279650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[45495]: audit 2022-04-22T19:58:07.532715+0000 mgr.y (mgr.34107) 5213 : audit [DBG] from='client.40824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[45495]: audit 2022-04-22T19:58:07.924292+0000 mon.a (mon.0) 8220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:09.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[45495]: audit 2022-04-22T19:58:08.290779+0000 mon.a (mon.0) 8221 : audit [DBG] from='client.? 172.21.15.115:0/786279650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[46934]: audit 2022-04-22T19:58:07.532715+0000 mgr.y (mgr.34107) 5213 : audit [DBG] from='client.40824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[46934]: audit 2022-04-22T19:58:07.924292+0000 mon.a (mon.0) 8220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:08 smithi115 bash[46934]: audit 2022-04-22T19:58:08.290779+0000 mon.a (mon.0) 8221 : audit [DBG] from='client.? 172.21.15.115:0/786279650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:09 smithi121 bash[36902]: audit 2022-04-22T19:58:07.878740+0000 mgr.y (mgr.34107) 5214 : audit [DBG] from='client.40830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:10.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:09 smithi121 bash[36902]: cluster 2022-04-22T19:58:08.529322+0000 mgr.y (mgr.34107) 5215 : cluster [DBG] pgmap v6504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:09 smithi115 bash[45495]: audit 2022-04-22T19:58:07.878740+0000 mgr.y (mgr.34107) 5214 : audit [DBG] from='client.40830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:09 smithi115 bash[45495]: cluster 2022-04-22T19:58:08.529322+0000 mgr.y (mgr.34107) 5215 : cluster [DBG] pgmap v6504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:09 smithi115 bash[46934]: audit 2022-04-22T19:58:07.878740+0000 mgr.y (mgr.34107) 5214 : audit [DBG] from='client.40830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:09 smithi115 bash[46934]: cluster 2022-04-22T19:58:08.529322+0000 mgr.y (mgr.34107) 5215 : cluster [DBG] pgmap v6504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:10 smithi121 bash[36902]: audit 2022-04-22T19:58:10.129583+0000 mon.a (mon.0) 8222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:10 smithi115 bash[45495]: audit 2022-04-22T19:58:10.129583+0000 mon.a (mon.0) 8222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:11.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:10 smithi115 bash[46934]: audit 2022-04-22T19:58:10.129583+0000 mon.a (mon.0) 8222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:11 smithi121 bash[36902]: cluster 2022-04-22T19:58:10.530413+0000 mgr.y (mgr.34107) 5216 : cluster [DBG] pgmap v6505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:11 smithi121 bash[36902]: audit 2022-04-22T19:58:11.471815+0000 mon.a (mon.0) 8223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:11 smithi115 bash[45495]: cluster 2022-04-22T19:58:10.530413+0000 mgr.y (mgr.34107) 5216 : cluster [DBG] pgmap v6505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:11 smithi115 bash[45495]: audit 2022-04-22T19:58:11.471815+0000 mon.a (mon.0) 8223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:11 smithi115 bash[46934]: cluster 2022-04-22T19:58:10.530413+0000 mgr.y (mgr.34107) 5216 : cluster [DBG] pgmap v6505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:11 smithi115 bash[46934]: audit 2022-04-22T19:58:11.471815+0000 mon.a (mon.0) 8223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:12.205Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:13 smithi121 bash[36902]: cluster 2022-04-22T19:58:12.531580+0000 mgr.y (mgr.34107) 5217 : cluster [DBG] pgmap v6506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:13 smithi121 bash[36902]: audit 2022-04-22T19:58:12.927014+0000 mon.a (mon.0) 8224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:13 smithi115 bash[45495]: cluster 2022-04-22T19:58:12.531580+0000 mgr.y (mgr.34107) 5217 : cluster [DBG] pgmap v6506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:14.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:13 smithi115 bash[45495]: audit 2022-04-22T19:58:12.927014+0000 mon.a (mon.0) 8224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:13 smithi115 bash[46934]: cluster 2022-04-22T19:58:12.531580+0000 mgr.y (mgr.34107) 5217 : cluster [DBG] pgmap v6506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:14.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:13 smithi115 bash[46934]: audit 2022-04-22T19:58:12.927014+0000 mon.a (mon.0) 8224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:14.079Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:15 smithi121 bash[36902]: cluster 2022-04-22T19:58:14.532655+0000 mgr.y (mgr.34107) 5218 : cluster [DBG] pgmap v6507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:15 smithi121 bash[36902]: audit 2022-04-22T19:58:15.024039+0000 mon.a (mon.0) 8225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:58:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:15 smithi121 bash[36902]: audit 2022-04-22T19:58:15.024499+0000 mon.a (mon.0) 8226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:58:16.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:15 smithi121 bash[36902]: audit 2022-04-22T19:58:15.133244+0000 mon.a (mon.0) 8227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[45495]: cluster 2022-04-22T19:58:14.532655+0000 mgr.y (mgr.34107) 5218 : cluster [DBG] pgmap v6507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[45495]: audit 2022-04-22T19:58:15.024039+0000 mon.a (mon.0) 8225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[45495]: audit 2022-04-22T19:58:15.024499+0000 mon.a (mon.0) 8226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[45495]: audit 2022-04-22T19:58:15.133244+0000 mon.a (mon.0) 8227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[46934]: cluster 2022-04-22T19:58:14.532655+0000 mgr.y (mgr.34107) 5218 : cluster [DBG] pgmap v6507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[46934]: audit 2022-04-22T19:58:15.024039+0000 mon.a (mon.0) 8225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[46934]: audit 2022-04-22T19:58:15.024499+0000 mon.a (mon.0) 8226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:58:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:15 smithi115 bash[46934]: audit 2022-04-22T19:58:15.133244+0000 mon.a (mon.0) 8227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:16 smithi121 bash[36902]: audit 2022-04-22T19:58:16.474975+0000 mon.a (mon.0) 8228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:16 smithi115 bash[45495]: audit 2022-04-22T19:58:16.474975+0000 mon.a (mon.0) 8228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:17.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:16 smithi115 bash[46934]: audit 2022-04-22T19:58:16.474975+0000 mon.a (mon.0) 8228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:17 smithi121 bash[36902]: cluster 2022-04-22T19:58:16.533699+0000 mgr.y (mgr.34107) 5219 : cluster [DBG] pgmap v6508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:17 smithi115 bash[46934]: cluster 2022-04-22T19:58:16.533699+0000 mgr.y (mgr.34107) 5219 : cluster [DBG] pgmap v6508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:17 smithi115 bash[45495]: cluster 2022-04-22T19:58:16.533699+0000 mgr.y (mgr.34107) 5219 : cluster [DBG] pgmap v6508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:18 smithi121 bash[36902]: audit 2022-04-22T19:58:17.930214+0000 mon.a (mon.0) 8229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:18 smithi115 bash[45495]: audit 2022-04-22T19:58:17.930214+0000 mon.a (mon.0) 8229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:18 smithi115 bash[46934]: audit 2022-04-22T19:58:17.930214+0000 mon.a (mon.0) 8229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:19 smithi121 bash[36902]: cluster 2022-04-22T19:58:18.534849+0000 mgr.y (mgr.34107) 5220 : cluster [DBG] pgmap v6509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:19 smithi115 bash[45495]: cluster 2022-04-22T19:58:18.534849+0000 mgr.y (mgr.34107) 5220 : cluster [DBG] pgmap v6509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:19 smithi115 bash[46934]: cluster 2022-04-22T19:58:18.534849+0000 mgr.y (mgr.34107) 5220 : cluster [DBG] pgmap v6509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:20 smithi121 bash[36902]: audit 2022-04-22T19:58:20.137079+0000 mon.a (mon.0) 8230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:20 smithi115 bash[45495]: audit 2022-04-22T19:58:20.137079+0000 mon.a (mon.0) 8230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:21.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:20 smithi115 bash[46934]: audit 2022-04-22T19:58:20.137079+0000 mon.a (mon.0) 8230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:21 smithi121 bash[36902]: cluster 2022-04-22T19:58:20.535965+0000 mgr.y (mgr.34107) 5221 : cluster [DBG] pgmap v6510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:21 smithi121 bash[36902]: audit 2022-04-22T19:58:21.477992+0000 mon.a (mon.0) 8231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:21 smithi115 bash[45495]: cluster 2022-04-22T19:58:20.535965+0000 mgr.y (mgr.34107) 5221 : cluster [DBG] pgmap v6510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:21 smithi115 bash[45495]: audit 2022-04-22T19:58:21.477992+0000 mon.a (mon.0) 8231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:21 smithi115 bash[46934]: cluster 2022-04-22T19:58:20.535965+0000 mgr.y (mgr.34107) 5221 : cluster [DBG] pgmap v6510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:21 smithi115 bash[46934]: audit 2022-04-22T19:58:21.477992+0000 mon.a (mon.0) 8231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:22.205Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:23 smithi121 bash[36902]: cluster 2022-04-22T19:58:22.536948+0000 mgr.y (mgr.34107) 5222 : cluster [DBG] pgmap v6511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:23 smithi121 bash[36902]: audit 2022-04-22T19:58:22.933151+0000 mon.a (mon.0) 8232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:23 smithi115 bash[45495]: cluster 2022-04-22T19:58:22.536948+0000 mgr.y (mgr.34107) 5222 : cluster [DBG] pgmap v6511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:23 smithi115 bash[45495]: audit 2022-04-22T19:58:22.933151+0000 mon.a (mon.0) 8232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:23 smithi115 bash[46934]: cluster 2022-04-22T19:58:22.536948+0000 mgr.y (mgr.34107) 5222 : cluster [DBG] pgmap v6511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:23 smithi115 bash[46934]: audit 2022-04-22T19:58:22.933151+0000 mon.a (mon.0) 8232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:24.079Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:25 smithi121 bash[36902]: cluster 2022-04-22T19:58:24.538121+0000 mgr.y (mgr.34107) 5223 : cluster [DBG] pgmap v6512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:25 smithi121 bash[36902]: audit 2022-04-22T19:58:25.140856+0000 mon.a (mon.0) 8233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:25 smithi115 bash[45495]: cluster 2022-04-22T19:58:24.538121+0000 mgr.y (mgr.34107) 5223 : cluster [DBG] pgmap v6512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:25 smithi115 bash[45495]: audit 2022-04-22T19:58:25.140856+0000 mon.a (mon.0) 8233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:25 smithi115 bash[46934]: cluster 2022-04-22T19:58:24.538121+0000 mgr.y (mgr.34107) 5223 : cluster [DBG] pgmap v6512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:25 smithi115 bash[46934]: audit 2022-04-22T19:58:25.140856+0000 mon.a (mon.0) 8233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:26 smithi121 bash[36902]: audit 2022-04-22T19:58:26.481142+0000 mon.a (mon.0) 8234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:27.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:26 smithi115 bash[45495]: audit 2022-04-22T19:58:26.481142+0000 mon.a (mon.0) 8234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:27.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:26 smithi115 bash[46934]: audit 2022-04-22T19:58:26.481142+0000 mon.a (mon.0) 8234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:27 smithi121 bash[36902]: cluster 2022-04-22T19:58:26.538942+0000 mgr.y (mgr.34107) 5224 : cluster [DBG] pgmap v6513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:28.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:27 smithi115 bash[45495]: cluster 2022-04-22T19:58:26.538942+0000 mgr.y (mgr.34107) 5224 : cluster [DBG] pgmap v6513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:27 smithi115 bash[46934]: cluster 2022-04-22T19:58:26.538942+0000 mgr.y (mgr.34107) 5224 : cluster [DBG] pgmap v6513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:28 smithi121 bash[36902]: audit 2022-04-22T19:58:27.936010+0000 mon.a (mon.0) 8235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:28 smithi115 bash[45495]: audit 2022-04-22T19:58:27.936010+0000 mon.a (mon.0) 8235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:29.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:28 smithi115 bash[46934]: audit 2022-04-22T19:58:27.936010+0000 mon.a (mon.0) 8235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:29 smithi121 bash[36902]: cluster 2022-04-22T19:58:28.540094+0000 mgr.y (mgr.34107) 5225 : cluster [DBG] pgmap v6514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:29 smithi115 bash[45495]: cluster 2022-04-22T19:58:28.540094+0000 mgr.y (mgr.34107) 5225 : cluster [DBG] pgmap v6514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:29 smithi115 bash[46934]: cluster 2022-04-22T19:58:28.540094+0000 mgr.y (mgr.34107) 5225 : cluster [DBG] pgmap v6514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:30 smithi121 bash[36902]: audit 2022-04-22T19:58:30.144605+0000 mon.a (mon.0) 8236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:30 smithi115 bash[45495]: audit 2022-04-22T19:58:30.144605+0000 mon.a (mon.0) 8236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:30 smithi115 bash[46934]: audit 2022-04-22T19:58:30.144605+0000 mon.a (mon.0) 8236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:32.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:31 smithi121 bash[36902]: cluster 2022-04-22T19:58:30.541146+0000 mgr.y (mgr.34107) 5226 : cluster [DBG] pgmap v6515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:32.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:31 smithi121 bash[36902]: audit 2022-04-22T19:58:31.484390+0000 mon.a (mon.0) 8237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:31 smithi115 bash[45495]: cluster 2022-04-22T19:58:30.541146+0000 mgr.y (mgr.34107) 5226 : cluster [DBG] pgmap v6515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:31 smithi115 bash[45495]: audit 2022-04-22T19:58:31.484390+0000 mon.a (mon.0) 8237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:31 smithi115 bash[46934]: cluster 2022-04-22T19:58:30.541146+0000 mgr.y (mgr.34107) 5226 : cluster [DBG] pgmap v6515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:31 smithi115 bash[46934]: audit 2022-04-22T19:58:31.484390+0000 mon.a (mon.0) 8237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:32.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:32.206Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:33 smithi121 bash[36902]: cluster 2022-04-22T19:58:32.542230+0000 mgr.y (mgr.34107) 5227 : cluster [DBG] pgmap v6516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:33 smithi121 bash[36902]: audit 2022-04-22T19:58:32.938717+0000 mon.a (mon.0) 8238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:33 smithi115 bash[45495]: cluster 2022-04-22T19:58:32.542230+0000 mgr.y (mgr.34107) 5227 : cluster [DBG] pgmap v6516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:33 smithi115 bash[45495]: audit 2022-04-22T19:58:32.938717+0000 mon.a (mon.0) 8238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:33 smithi115 bash[46934]: cluster 2022-04-22T19:58:32.542230+0000 mgr.y (mgr.34107) 5227 : cluster [DBG] pgmap v6516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:33 smithi115 bash[46934]: audit 2022-04-22T19:58:32.938717+0000 mon.a (mon.0) 8238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:34.081Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:35 smithi121 bash[36902]: cluster 2022-04-22T19:58:34.543353+0000 mgr.y (mgr.34107) 5228 : cluster [DBG] pgmap v6517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:36.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:35 smithi121 bash[36902]: audit 2022-04-22T19:58:35.148258+0000 mon.a (mon.0) 8239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:35 smithi115 bash[45495]: cluster 2022-04-22T19:58:34.543353+0000 mgr.y (mgr.34107) 5228 : cluster [DBG] pgmap v6517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:36.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:35 smithi115 bash[45495]: audit 2022-04-22T19:58:35.148258+0000 mon.a (mon.0) 8239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:35 smithi115 bash[46934]: cluster 2022-04-22T19:58:34.543353+0000 mgr.y (mgr.34107) 5228 : cluster [DBG] pgmap v6517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:36.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:35 smithi115 bash[46934]: audit 2022-04-22T19:58:35.148258+0000 mon.a (mon.0) 8239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:36 smithi121 bash[36902]: audit 2022-04-22T19:58:36.487353+0000 mon.a (mon.0) 8240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:36 smithi115 bash[46934]: audit 2022-04-22T19:58:36.487353+0000 mon.a (mon.0) 8240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:36 smithi115 bash[45495]: audit 2022-04-22T19:58:36.487353+0000 mon.a (mon.0) 8240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:37 smithi121 bash[36902]: cluster 2022-04-22T19:58:36.544357+0000 mgr.y (mgr.34107) 5229 : cluster [DBG] pgmap v6518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:37 smithi115 bash[45495]: cluster 2022-04-22T19:58:36.544357+0000 mgr.y (mgr.34107) 5229 : cluster [DBG] pgmap v6518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:37 smithi115 bash[46934]: cluster 2022-04-22T19:58:36.544357+0000 mgr.y (mgr.34107) 5229 : cluster [DBG] pgmap v6518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:38.677 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:58:39.011 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:58:39.011 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:58:39.012 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:58:39.013 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:58:39.013 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:58:39.013 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:58:39.013 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:58:39.013 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:58:39.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:58:39.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:58:39.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:58:39.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:58:39.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:58:39.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:58:39.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:58:39.015 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:58:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:38 smithi121 bash[36902]: audit 2022-04-22T19:58:37.941303+0000 mon.a (mon.0) 8241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:38 smithi115 bash[46934]: audit 2022-04-22T19:58:37.941303+0000 mon.a (mon.0) 8241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:38 smithi115 bash[45495]: audit 2022-04-22T19:58:37.941303+0000 mon.a (mon.0) 8241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:39.416 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:58:39.416 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:58:39.416 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:58:39.417 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:58:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:58:39.419 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:58:39.419 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:58:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:39 smithi121 bash[36902]: cluster 2022-04-22T19:58:38.545386+0000 mgr.y (mgr.34107) 5230 : cluster [DBG] pgmap v6519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:39 smithi121 bash[36902]: audit 2022-04-22T19:58:38.648558+0000 mgr.y (mgr.34107) 5231 : audit [DBG] from='client.50776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:39 smithi121 bash[36902]: audit 2022-04-22T19:58:39.406987+0000 mon.a (mon.0) 8242 : audit [DBG] from='client.? 172.21.15.115:0/647815012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[45495]: cluster 2022-04-22T19:58:38.545386+0000 mgr.y (mgr.34107) 5230 : cluster [DBG] pgmap v6519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[45495]: audit 2022-04-22T19:58:38.648558+0000 mgr.y (mgr.34107) 5231 : audit [DBG] from='client.50776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[45495]: audit 2022-04-22T19:58:39.406987+0000 mon.a (mon.0) 8242 : audit [DBG] from='client.? 172.21.15.115:0/647815012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[46934]: cluster 2022-04-22T19:58:38.545386+0000 mgr.y (mgr.34107) 5230 : cluster [DBG] pgmap v6519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[46934]: audit 2022-04-22T19:58:38.648558+0000 mgr.y (mgr.34107) 5231 : audit [DBG] from='client.50776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:40.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:39 smithi115 bash[46934]: audit 2022-04-22T19:58:39.406987+0000 mon.a (mon.0) 8242 : audit [DBG] from='client.? 172.21.15.115:0/647815012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:58:40.721 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[45495]: audit 2022-04-22T19:58:38.988335+0000 mgr.y (mgr.34107) 5232 : audit [DBG] from='client.50782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:40.722 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[45495]: audit 2022-04-22T19:58:40.151801+0000 mon.a (mon.0) 8243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:40.722 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[45495]: audit 2022-04-22T19:58:40.440436+0000 mon.a (mon.0) 8244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:58:40.722 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[45495]: audit 2022-04-22T19:58:40.443139+0000 mon.a (mon.0) 8245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:58:40.722 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[45495]: audit 2022-04-22T19:58:40.444705+0000 mon.a (mon.0) 8246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:58:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:40 smithi121 bash[36902]: audit 2022-04-22T19:58:38.988335+0000 mgr.y (mgr.34107) 5232 : audit [DBG] from='client.50782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:40 smithi121 bash[36902]: audit 2022-04-22T19:58:40.151801+0000 mon.a (mon.0) 8243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:40 smithi121 bash[36902]: audit 2022-04-22T19:58:40.440436+0000 mon.a (mon.0) 8244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:58:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:40 smithi121 bash[36902]: audit 2022-04-22T19:58:40.443139+0000 mon.a (mon.0) 8245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:58:41.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:40 smithi121 bash[36902]: audit 2022-04-22T19:58:40.444705+0000 mon.a (mon.0) 8246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:58:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[46934]: audit 2022-04-22T19:58:38.988335+0000 mgr.y (mgr.34107) 5232 : audit [DBG] from='client.50782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:58:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[46934]: audit 2022-04-22T19:58:40.151801+0000 mon.a (mon.0) 8243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[46934]: audit 2022-04-22T19:58:40.440436+0000 mon.a (mon.0) 8244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:58:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[46934]: audit 2022-04-22T19:58:40.443139+0000 mon.a (mon.0) 8245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:58:41.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:40 smithi115 bash[46934]: audit 2022-04-22T19:58:40.444705+0000 mon.a (mon.0) 8246 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:58:42.215 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[45495]: cluster 2022-04-22T19:58:40.546493+0000 mgr.y (mgr.34107) 5233 : cluster [DBG] pgmap v6520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:42.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[45495]: audit 2022-04-22T19:58:40.810964+0000 mon.a (mon.0) 8247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:58:42.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[45495]: audit 2022-04-22T19:58:41.490586+0000 mon.a (mon.0) 8248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:42.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[46934]: cluster 2022-04-22T19:58:40.546493+0000 mgr.y (mgr.34107) 5233 : cluster [DBG] pgmap v6520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:42.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[46934]: audit 2022-04-22T19:58:40.810964+0000 mon.a (mon.0) 8247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:58:42.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:41 smithi115 bash[46934]: audit 2022-04-22T19:58:41.490586+0000 mon.a (mon.0) 8248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:41 smithi121 bash[36902]: cluster 2022-04-22T19:58:40.546493+0000 mgr.y (mgr.34107) 5233 : cluster [DBG] pgmap v6520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:41 smithi121 bash[36902]: audit 2022-04-22T19:58:40.810964+0000 mon.a (mon.0) 8247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:58:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:41 smithi121 bash[36902]: audit 2022-04-22T19:58:41.490586+0000 mon.a (mon.0) 8248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:42.206Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:44.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:43 smithi115 bash[45495]: cluster 2022-04-22T19:58:42.547557+0000 mgr.y (mgr.34107) 5234 : cluster [DBG] pgmap v6521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:44.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:43 smithi115 bash[45495]: audit 2022-04-22T19:58:42.943411+0000 mon.a (mon.0) 8249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:44.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:43 smithi115 bash[46934]: cluster 2022-04-22T19:58:42.547557+0000 mgr.y (mgr.34107) 5234 : cluster [DBG] pgmap v6521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:44.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:43 smithi115 bash[46934]: audit 2022-04-22T19:58:42.943411+0000 mon.a (mon.0) 8249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:43 smithi121 bash[36902]: cluster 2022-04-22T19:58:42.547557+0000 mgr.y (mgr.34107) 5234 : cluster [DBG] pgmap v6521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:43 smithi121 bash[36902]: audit 2022-04-22T19:58:42.943411+0000 mon.a (mon.0) 8249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:44.080Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:45 smithi121 bash[36902]: cluster 2022-04-22T19:58:44.548545+0000 mgr.y (mgr.34107) 5235 : cluster [DBG] pgmap v6522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:45 smithi121 bash[36902]: audit 2022-04-22T19:58:45.155281+0000 mon.a (mon.0) 8250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:45 smithi115 bash[45495]: cluster 2022-04-22T19:58:44.548545+0000 mgr.y (mgr.34107) 5235 : cluster [DBG] pgmap v6522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:45 smithi115 bash[45495]: audit 2022-04-22T19:58:45.155281+0000 mon.a (mon.0) 8250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:45 smithi115 bash[46934]: cluster 2022-04-22T19:58:44.548545+0000 mgr.y (mgr.34107) 5235 : cluster [DBG] pgmap v6522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:45 smithi115 bash[46934]: audit 2022-04-22T19:58:45.155281+0000 mon.a (mon.0) 8250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:46 smithi121 bash[36902]: audit 2022-04-22T19:58:46.493796+0000 mon.a (mon.0) 8251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:46 smithi115 bash[45495]: audit 2022-04-22T19:58:46.493796+0000 mon.a (mon.0) 8251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:46 smithi115 bash[46934]: audit 2022-04-22T19:58:46.493796+0000 mon.a (mon.0) 8251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:47 smithi121 bash[36902]: cluster 2022-04-22T19:58:46.549595+0000 mgr.y (mgr.34107) 5236 : cluster [DBG] pgmap v6523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:47 smithi115 bash[45495]: cluster 2022-04-22T19:58:46.549595+0000 mgr.y (mgr.34107) 5236 : cluster [DBG] pgmap v6523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:47 smithi115 bash[46934]: cluster 2022-04-22T19:58:46.549595+0000 mgr.y (mgr.34107) 5236 : cluster [DBG] pgmap v6523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:48 smithi121 bash[36902]: audit 2022-04-22T19:58:47.946325+0000 mon.a (mon.0) 8252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:48 smithi115 bash[45495]: audit 2022-04-22T19:58:47.946325+0000 mon.a (mon.0) 8252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:49.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:48 smithi115 bash[46934]: audit 2022-04-22T19:58:47.946325+0000 mon.a (mon.0) 8252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:49 smithi121 bash[36902]: cluster 2022-04-22T19:58:48.550720+0000 mgr.y (mgr.34107) 5237 : cluster [DBG] pgmap v6524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:49 smithi115 bash[45495]: cluster 2022-04-22T19:58:48.550720+0000 mgr.y (mgr.34107) 5237 : cluster [DBG] pgmap v6524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:49 smithi115 bash[46934]: cluster 2022-04-22T19:58:48.550720+0000 mgr.y (mgr.34107) 5237 : cluster [DBG] pgmap v6524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:50 smithi121 bash[36902]: audit 2022-04-22T19:58:50.158978+0000 mon.a (mon.0) 8253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:50 smithi115 bash[45495]: audit 2022-04-22T19:58:50.158978+0000 mon.a (mon.0) 8253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:50 smithi115 bash[46934]: audit 2022-04-22T19:58:50.158978+0000 mon.a (mon.0) 8253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:52.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:51 smithi115 bash[46934]: cluster 2022-04-22T19:58:50.551814+0000 mgr.y (mgr.34107) 5238 : cluster [DBG] pgmap v6525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:52.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:51 smithi115 bash[46934]: audit 2022-04-22T19:58:51.496835+0000 mon.a (mon.0) 8254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:52.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:51 smithi115 bash[45495]: cluster 2022-04-22T19:58:50.551814+0000 mgr.y (mgr.34107) 5238 : cluster [DBG] pgmap v6525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:52.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:51 smithi115 bash[45495]: audit 2022-04-22T19:58:51.496835+0000 mon.a (mon.0) 8254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:51 smithi121 bash[36902]: cluster 2022-04-22T19:58:50.551814+0000 mgr.y (mgr.34107) 5238 : cluster [DBG] pgmap v6525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:51 smithi121 bash[36902]: audit 2022-04-22T19:58:51.496835+0000 mon.a (mon.0) 8254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:52.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:52 smithi121 bash[36902]: cluster 2022-04-22T19:58:52.552890+0000 mgr.y (mgr.34107) 5239 : cluster [DBG] pgmap v6526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:52 smithi115 bash[45495]: cluster 2022-04-22T19:58:52.552890+0000 mgr.y (mgr.34107) 5239 : cluster [DBG] pgmap v6526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:52 smithi115 bash[46934]: cluster 2022-04-22T19:58:52.552890+0000 mgr.y (mgr.34107) 5239 : cluster [DBG] pgmap v6526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:53 smithi121 bash[36902]: audit 2022-04-22T19:58:52.948918+0000 mon.a (mon.0) 8255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:53 smithi115 bash[45495]: audit 2022-04-22T19:58:52.948918+0000 mon.a (mon.0) 8255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:58:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:58:54.082Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:58:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:53 smithi115 bash[46934]: audit 2022-04-22T19:58:52.948918+0000 mon.a (mon.0) 8255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:55.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:54 smithi121 bash[36902]: cluster 2022-04-22T19:58:54.554078+0000 mgr.y (mgr.34107) 5240 : cluster [DBG] pgmap v6527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:54 smithi115 bash[45495]: cluster 2022-04-22T19:58:54.554078+0000 mgr.y (mgr.34107) 5240 : cluster [DBG] pgmap v6527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:54 smithi115 bash[46934]: cluster 2022-04-22T19:58:54.554078+0000 mgr.y (mgr.34107) 5240 : cluster [DBG] pgmap v6527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:55 smithi121 bash[36902]: audit 2022-04-22T19:58:55.162619+0000 mon.a (mon.0) 8256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:55 smithi115 bash[45495]: audit 2022-04-22T19:58:55.162619+0000 mon.a (mon.0) 8256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:55 smithi115 bash[46934]: audit 2022-04-22T19:58:55.162619+0000 mon.a (mon.0) 8256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:58:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:56 smithi121 bash[36902]: audit 2022-04-22T19:58:56.500030+0000 mon.a (mon.0) 8257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:56 smithi121 bash[36902]: cluster 2022-04-22T19:58:56.555176+0000 mgr.y (mgr.34107) 5241 : cluster [DBG] pgmap v6528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:57.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:56 smithi115 bash[45495]: audit 2022-04-22T19:58:56.500030+0000 mon.a (mon.0) 8257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:57.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:56 smithi115 bash[45495]: cluster 2022-04-22T19:58:56.555176+0000 mgr.y (mgr.34107) 5241 : cluster [DBG] pgmap v6528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:56 smithi115 bash[46934]: audit 2022-04-22T19:58:56.500030+0000 mon.a (mon.0) 8257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:58:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:56 smithi115 bash[46934]: cluster 2022-04-22T19:58:56.555176+0000 mgr.y (mgr.34107) 5241 : cluster [DBG] pgmap v6528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:58 smithi121 bash[36902]: audit 2022-04-22T19:58:57.951799+0000 mon.a (mon.0) 8258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:58 smithi115 bash[45495]: audit 2022-04-22T19:58:57.951799+0000 mon.a (mon.0) 8258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:58 smithi115 bash[46934]: audit 2022-04-22T19:58:57.951799+0000 mon.a (mon.0) 8258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:58:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:58:59 smithi121 bash[36902]: cluster 2022-04-22T19:58:58.556051+0000 mgr.y (mgr.34107) 5242 : cluster [DBG] pgmap v6529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:58:59 smithi115 bash[45495]: cluster 2022-04-22T19:58:58.556051+0000 mgr.y (mgr.34107) 5242 : cluster [DBG] pgmap v6529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:58:59.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:58:59 smithi115 bash[46934]: cluster 2022-04-22T19:58:58.556051+0000 mgr.y (mgr.34107) 5242 : cluster [DBG] pgmap v6529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:00 smithi121 bash[36902]: audit 2022-04-22T19:59:00.166610+0000 mon.a (mon.0) 8259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:00 smithi115 bash[45495]: audit 2022-04-22T19:59:00.166610+0000 mon.a (mon.0) 8259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:00 smithi115 bash[46934]: audit 2022-04-22T19:59:00.166610+0000 mon.a (mon.0) 8259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:01 smithi121 bash[36902]: cluster 2022-04-22T19:59:00.557104+0000 mgr.y (mgr.34107) 5243 : cluster [DBG] pgmap v6530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:01.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:01 smithi115 bash[45495]: cluster 2022-04-22T19:59:00.557104+0000 mgr.y (mgr.34107) 5243 : cluster [DBG] pgmap v6530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:01.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:01 smithi115 bash[46934]: cluster 2022-04-22T19:59:00.557104+0000 mgr.y (mgr.34107) 5243 : cluster [DBG] pgmap v6530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:02.492 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:02 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:02.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:02 smithi121 bash[36902]: audit 2022-04-22T19:59:01.503397+0000 mon.a (mon.0) 8260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:02.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:02 smithi115 bash[45495]: audit 2022-04-22T19:59:01.503397+0000 mon.a (mon.0) 8260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:02.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:02 smithi115 bash[46934]: audit 2022-04-22T19:59:01.503397+0000 mon.a (mon.0) 8260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:03 smithi121 bash[36902]: cluster 2022-04-22T19:59:02.558224+0000 mgr.y (mgr.34107) 5244 : cluster [DBG] pgmap v6531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:03 smithi121 bash[36902]: audit 2022-04-22T19:59:02.954392+0000 mon.a (mon.0) 8261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:03 smithi115 bash[45495]: cluster 2022-04-22T19:59:02.558224+0000 mgr.y (mgr.34107) 5244 : cluster [DBG] pgmap v6531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:03 smithi115 bash[45495]: audit 2022-04-22T19:59:02.954392+0000 mon.a (mon.0) 8261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:03.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:03 smithi115 bash[46934]: cluster 2022-04-22T19:59:02.558224+0000 mgr.y (mgr.34107) 5244 : cluster [DBG] pgmap v6531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:03.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:03 smithi115 bash[46934]: audit 2022-04-22T19:59:02.954392+0000 mon.a (mon.0) 8261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:04 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:04.081Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:05 smithi121 bash[36902]: cluster 2022-04-22T19:59:04.559313+0000 mgr.y (mgr.34107) 5245 : cluster [DBG] pgmap v6532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:05 smithi121 bash[36902]: audit 2022-04-22T19:59:05.170326+0000 mon.a (mon.0) 8262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:05 smithi115 bash[45495]: cluster 2022-04-22T19:59:04.559313+0000 mgr.y (mgr.34107) 5245 : cluster [DBG] pgmap v6532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:05 smithi115 bash[45495]: audit 2022-04-22T19:59:05.170326+0000 mon.a (mon.0) 8262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:05 smithi115 bash[46934]: cluster 2022-04-22T19:59:04.559313+0000 mgr.y (mgr.34107) 5245 : cluster [DBG] pgmap v6532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:05 smithi115 bash[46934]: audit 2022-04-22T19:59:05.170326+0000 mon.a (mon.0) 8262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:06 smithi121 bash[36902]: audit 2022-04-22T19:59:06.506500+0000 mon.a (mon.0) 8263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:06 smithi115 bash[45495]: audit 2022-04-22T19:59:06.506500+0000 mon.a (mon.0) 8263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:06 smithi115 bash[46934]: audit 2022-04-22T19:59:06.506500+0000 mon.a (mon.0) 8263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:07 smithi121 bash[36902]: cluster 2022-04-22T19:59:06.560181+0000 mgr.y (mgr.34107) 5246 : cluster [DBG] pgmap v6533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:08.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:07 smithi115 bash[45495]: cluster 2022-04-22T19:59:06.560181+0000 mgr.y (mgr.34107) 5246 : cluster [DBG] pgmap v6533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:08.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:07 smithi115 bash[46934]: cluster 2022-04-22T19:59:06.560181+0000 mgr.y (mgr.34107) 5246 : cluster [DBG] pgmap v6533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:08 smithi121 bash[36902]: audit 2022-04-22T19:59:07.956927+0000 mon.a (mon.0) 8264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:08 smithi115 bash[45495]: audit 2022-04-22T19:59:07.956927+0000 mon.a (mon.0) 8264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:09.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:08 smithi115 bash[46934]: audit 2022-04-22T19:59:07.956927+0000 mon.a (mon.0) 8264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:09.799 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:59:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:09 smithi121 bash[36902]: cluster 2022-04-22T19:59:08.561328+0000 mgr.y (mgr.34107) 5247 : cluster [DBG] pgmap v6534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:09 smithi115 bash[45495]: cluster 2022-04-22T19:59:08.561328+0000 mgr.y (mgr.34107) 5247 : cluster [DBG] pgmap v6534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:10.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:09 smithi115 bash[46934]: cluster 2022-04-22T19:59:08.561328+0000 mgr.y (mgr.34107) 5247 : cluster [DBG] pgmap v6534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:10.137 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:59:10.138 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:59:10.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:59:10.140 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:59:10.140 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:59:10.140 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:59:10.536 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:59:10.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:59:10.538 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:59:10.538 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:59:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:10 smithi121 bash[36902]: audit 2022-04-22T19:59:09.773336+0000 mgr.y (mgr.34107) 5248 : audit [DBG] from='client.50794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:10 smithi121 bash[36902]: audit 2022-04-22T19:59:10.174269+0000 mon.a (mon.0) 8265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:10 smithi121 bash[36902]: audit 2022-04-22T19:59:10.527006+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.115:0/3086159482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[46934]: audit 2022-04-22T19:59:09.773336+0000 mgr.y (mgr.34107) 5248 : audit [DBG] from='client.50794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[46934]: audit 2022-04-22T19:59:10.174269+0000 mon.a (mon.0) 8265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[46934]: audit 2022-04-22T19:59:10.527006+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.115:0/3086159482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[45495]: audit 2022-04-22T19:59:09.773336+0000 mgr.y (mgr.34107) 5248 : audit [DBG] from='client.50794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[45495]: audit 2022-04-22T19:59:10.174269+0000 mon.a (mon.0) 8265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:11.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:10 smithi115 bash[45495]: audit 2022-04-22T19:59:10.527006+0000 mon.c (mon.1) 220 : audit [DBG] from='client.? 172.21.15.115:0/3086159482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:11 smithi121 bash[36902]: audit 2022-04-22T19:59:10.110386+0000 mgr.y (mgr.34107) 5249 : audit [DBG] from='client.50800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:11 smithi121 bash[36902]: cluster 2022-04-22T19:59:10.562541+0000 mgr.y (mgr.34107) 5250 : cluster [DBG] pgmap v6535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:11 smithi121 bash[36902]: audit 2022-04-22T19:59:11.509497+0000 mon.a (mon.0) 8266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[45495]: audit 2022-04-22T19:59:10.110386+0000 mgr.y (mgr.34107) 5249 : audit [DBG] from='client.50800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[45495]: cluster 2022-04-22T19:59:10.562541+0000 mgr.y (mgr.34107) 5250 : cluster [DBG] pgmap v6535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[45495]: audit 2022-04-22T19:59:11.509497+0000 mon.a (mon.0) 8266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[46934]: audit 2022-04-22T19:59:10.110386+0000 mgr.y (mgr.34107) 5249 : audit [DBG] from='client.50800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[46934]: cluster 2022-04-22T19:59:10.562541+0000 mgr.y (mgr.34107) 5250 : cluster [DBG] pgmap v6535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:11 smithi115 bash[46934]: audit 2022-04-22T19:59:11.509497+0000 mon.a (mon.0) 8266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:12 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:12.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:13 smithi121 bash[36902]: cluster 2022-04-22T19:59:12.563740+0000 mgr.y (mgr.34107) 5251 : cluster [DBG] pgmap v6536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:13 smithi121 bash[36902]: audit 2022-04-22T19:59:12.959559+0000 mon.a (mon.0) 8267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:13 smithi115 bash[45495]: cluster 2022-04-22T19:59:12.563740+0000 mgr.y (mgr.34107) 5251 : cluster [DBG] pgmap v6536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:13 smithi115 bash[45495]: audit 2022-04-22T19:59:12.959559+0000 mon.a (mon.0) 8267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:13 smithi115 bash[46934]: cluster 2022-04-22T19:59:12.563740+0000 mgr.y (mgr.34107) 5251 : cluster [DBG] pgmap v6536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:13 smithi115 bash[46934]: audit 2022-04-22T19:59:12.959559+0000 mon.a (mon.0) 8267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:14 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:14.081Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:15 smithi121 bash[36902]: cluster 2022-04-22T19:59:14.564810+0000 mgr.y (mgr.34107) 5252 : cluster [DBG] pgmap v6537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:15 smithi121 bash[36902]: audit 2022-04-22T19:59:15.033147+0000 mon.a (mon.0) 8268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:59:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:15 smithi121 bash[36902]: audit 2022-04-22T19:59:15.033788+0000 mon.a (mon.0) 8269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:59:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:15 smithi121 bash[36902]: audit 2022-04-22T19:59:15.178624+0000 mon.a (mon.0) 8270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[45495]: cluster 2022-04-22T19:59:14.564810+0000 mgr.y (mgr.34107) 5252 : cluster [DBG] pgmap v6537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:16.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[45495]: audit 2022-04-22T19:59:15.033147+0000 mon.a (mon.0) 8268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:59:16.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[45495]: audit 2022-04-22T19:59:15.033788+0000 mon.a (mon.0) 8269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:59:16.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[45495]: audit 2022-04-22T19:59:15.178624+0000 mon.a (mon.0) 8270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:16.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[46934]: cluster 2022-04-22T19:59:14.564810+0000 mgr.y (mgr.34107) 5252 : cluster [DBG] pgmap v6537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:16.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[46934]: audit 2022-04-22T19:59:15.033147+0000 mon.a (mon.0) 8268 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T19:59:16.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[46934]: audit 2022-04-22T19:59:15.033788+0000 mon.a (mon.0) 8269 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T19:59:16.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:15 smithi115 bash[46934]: audit 2022-04-22T19:59:15.178624+0000 mon.a (mon.0) 8270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:16 smithi121 bash[36902]: audit 2022-04-22T19:59:16.512590+0000 mon.a (mon.0) 8271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:16 smithi115 bash[46934]: audit 2022-04-22T19:59:16.512590+0000 mon.a (mon.0) 8271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:17.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:16 smithi115 bash[45495]: audit 2022-04-22T19:59:16.512590+0000 mon.a (mon.0) 8271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:17 smithi121 bash[36902]: cluster 2022-04-22T19:59:16.565772+0000 mgr.y (mgr.34107) 5253 : cluster [DBG] pgmap v6538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:17 smithi115 bash[45495]: cluster 2022-04-22T19:59:16.565772+0000 mgr.y (mgr.34107) 5253 : cluster [DBG] pgmap v6538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:17 smithi115 bash[46934]: cluster 2022-04-22T19:59:16.565772+0000 mgr.y (mgr.34107) 5253 : cluster [DBG] pgmap v6538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:18 smithi121 bash[36902]: audit 2022-04-22T19:59:17.962225+0000 mon.a (mon.0) 8272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:18 smithi115 bash[45495]: audit 2022-04-22T19:59:17.962225+0000 mon.a (mon.0) 8272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:18 smithi115 bash[46934]: audit 2022-04-22T19:59:17.962225+0000 mon.a (mon.0) 8272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:19 smithi121 bash[36902]: cluster 2022-04-22T19:59:18.566911+0000 mgr.y (mgr.34107) 5254 : cluster [DBG] pgmap v6539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:19 smithi115 bash[45495]: cluster 2022-04-22T19:59:18.566911+0000 mgr.y (mgr.34107) 5254 : cluster [DBG] pgmap v6539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:19 smithi115 bash[46934]: cluster 2022-04-22T19:59:18.566911+0000 mgr.y (mgr.34107) 5254 : cluster [DBG] pgmap v6539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:20 smithi121 bash[36902]: audit 2022-04-22T19:59:20.182147+0000 mon.a (mon.0) 8273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:20 smithi115 bash[45495]: audit 2022-04-22T19:59:20.182147+0000 mon.a (mon.0) 8273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:20 smithi115 bash[46934]: audit 2022-04-22T19:59:20.182147+0000 mon.a (mon.0) 8273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:21 smithi121 bash[36902]: cluster 2022-04-22T19:59:20.568015+0000 mgr.y (mgr.34107) 5255 : cluster [DBG] pgmap v6540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:21 smithi121 bash[36902]: audit 2022-04-22T19:59:21.515693+0000 mon.a (mon.0) 8274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:21 smithi115 bash[45495]: cluster 2022-04-22T19:59:20.568015+0000 mgr.y (mgr.34107) 5255 : cluster [DBG] pgmap v6540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:21 smithi115 bash[45495]: audit 2022-04-22T19:59:21.515693+0000 mon.a (mon.0) 8274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:21 smithi115 bash[46934]: cluster 2022-04-22T19:59:20.568015+0000 mgr.y (mgr.34107) 5255 : cluster [DBG] pgmap v6540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:21 smithi115 bash[46934]: audit 2022-04-22T19:59:21.515693+0000 mon.a (mon.0) 8274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:22.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:22 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:22.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:23 smithi121 bash[36902]: cluster 2022-04-22T19:59:22.569082+0000 mgr.y (mgr.34107) 5256 : cluster [DBG] pgmap v6541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:23 smithi121 bash[36902]: audit 2022-04-22T19:59:22.964981+0000 mon.a (mon.0) 8275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:23 smithi115 bash[45495]: cluster 2022-04-22T19:59:22.569082+0000 mgr.y (mgr.34107) 5256 : cluster [DBG] pgmap v6541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:23 smithi115 bash[45495]: audit 2022-04-22T19:59:22.964981+0000 mon.a (mon.0) 8275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:23 smithi115 bash[46934]: cluster 2022-04-22T19:59:22.569082+0000 mgr.y (mgr.34107) 5256 : cluster [DBG] pgmap v6541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:23 smithi115 bash[46934]: audit 2022-04-22T19:59:22.964981+0000 mon.a (mon.0) 8275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:24 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:24.085Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:25 smithi121 bash[36902]: cluster 2022-04-22T19:59:24.570206+0000 mgr.y (mgr.34107) 5257 : cluster [DBG] pgmap v6542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:25 smithi121 bash[36902]: audit 2022-04-22T19:59:25.185840+0000 mon.a (mon.0) 8276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:25 smithi115 bash[46934]: cluster 2022-04-22T19:59:24.570206+0000 mgr.y (mgr.34107) 5257 : cluster [DBG] pgmap v6542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:25 smithi115 bash[46934]: audit 2022-04-22T19:59:25.185840+0000 mon.a (mon.0) 8276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:25 smithi115 bash[45495]: cluster 2022-04-22T19:59:24.570206+0000 mgr.y (mgr.34107) 5257 : cluster [DBG] pgmap v6542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:26.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:25 smithi115 bash[45495]: audit 2022-04-22T19:59:25.185840+0000 mon.a (mon.0) 8276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:26 smithi121 bash[36902]: audit 2022-04-22T19:59:26.519188+0000 mon.a (mon.0) 8277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:26 smithi115 bash[45495]: audit 2022-04-22T19:59:26.519188+0000 mon.a (mon.0) 8277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:26 smithi115 bash[46934]: audit 2022-04-22T19:59:26.519188+0000 mon.a (mon.0) 8277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:27 smithi121 bash[36902]: cluster 2022-04-22T19:59:26.571318+0000 mgr.y (mgr.34107) 5258 : cluster [DBG] pgmap v6543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:27 smithi115 bash[45495]: cluster 2022-04-22T19:59:26.571318+0000 mgr.y (mgr.34107) 5258 : cluster [DBG] pgmap v6543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:27 smithi115 bash[46934]: cluster 2022-04-22T19:59:26.571318+0000 mgr.y (mgr.34107) 5258 : cluster [DBG] pgmap v6543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:28 smithi121 bash[36902]: audit 2022-04-22T19:59:27.967728+0000 mon.a (mon.0) 8278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:29.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:28 smithi115 bash[45495]: audit 2022-04-22T19:59:27.967728+0000 mon.a (mon.0) 8278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:29.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:28 smithi115 bash[46934]: audit 2022-04-22T19:59:27.967728+0000 mon.a (mon.0) 8278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:29 smithi121 bash[36902]: cluster 2022-04-22T19:59:28.572534+0000 mgr.y (mgr.34107) 5259 : cluster [DBG] pgmap v6544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:29 smithi115 bash[46934]: cluster 2022-04-22T19:59:28.572534+0000 mgr.y (mgr.34107) 5259 : cluster [DBG] pgmap v6544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:30.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:29 smithi115 bash[45495]: cluster 2022-04-22T19:59:28.572534+0000 mgr.y (mgr.34107) 5259 : cluster [DBG] pgmap v6544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:30 smithi121 bash[36902]: audit 2022-04-22T19:59:30.189648+0000 mon.a (mon.0) 8279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:31.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:30 smithi115 bash[45495]: audit 2022-04-22T19:59:30.189648+0000 mon.a (mon.0) 8279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:31.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:30 smithi115 bash[46934]: audit 2022-04-22T19:59:30.189648+0000 mon.a (mon.0) 8279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:31 smithi121 bash[36902]: cluster 2022-04-22T19:59:30.573515+0000 mgr.y (mgr.34107) 5260 : cluster [DBG] pgmap v6545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:31 smithi121 bash[36902]: audit 2022-04-22T19:59:31.522498+0000 mon.a (mon.0) 8280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:31 smithi115 bash[46934]: cluster 2022-04-22T19:59:30.573515+0000 mgr.y (mgr.34107) 5260 : cluster [DBG] pgmap v6545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:31 smithi115 bash[46934]: audit 2022-04-22T19:59:31.522498+0000 mon.a (mon.0) 8280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:31 smithi115 bash[45495]: cluster 2022-04-22T19:59:30.573515+0000 mgr.y (mgr.34107) 5260 : cluster [DBG] pgmap v6545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:32.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:31 smithi115 bash[45495]: audit 2022-04-22T19:59:31.522498+0000 mon.a (mon.0) 8280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:32 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:32.208Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:33 smithi121 bash[36902]: cluster 2022-04-22T19:59:32.574549+0000 mgr.y (mgr.34107) 5261 : cluster [DBG] pgmap v6546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:33 smithi121 bash[36902]: audit 2022-04-22T19:59:32.970283+0000 mon.a (mon.0) 8281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:33 smithi115 bash[45495]: cluster 2022-04-22T19:59:32.574549+0000 mgr.y (mgr.34107) 5261 : cluster [DBG] pgmap v6546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:33 smithi115 bash[45495]: audit 2022-04-22T19:59:32.970283+0000 mon.a (mon.0) 8281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:33 smithi115 bash[46934]: cluster 2022-04-22T19:59:32.574549+0000 mgr.y (mgr.34107) 5261 : cluster [DBG] pgmap v6546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:33 smithi115 bash[46934]: audit 2022-04-22T19:59:32.970283+0000 mon.a (mon.0) 8281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:34.551 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:34 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:34.085Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:35 smithi121 bash[36902]: cluster 2022-04-22T19:59:34.575645+0000 mgr.y (mgr.34107) 5262 : cluster [DBG] pgmap v6547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:36.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:35 smithi121 bash[36902]: audit 2022-04-22T19:59:35.193221+0000 mon.a (mon.0) 8282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:35 smithi115 bash[45495]: cluster 2022-04-22T19:59:34.575645+0000 mgr.y (mgr.34107) 5262 : cluster [DBG] pgmap v6547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:35 smithi115 bash[45495]: audit 2022-04-22T19:59:35.193221+0000 mon.a (mon.0) 8282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:35 smithi115 bash[46934]: cluster 2022-04-22T19:59:34.575645+0000 mgr.y (mgr.34107) 5262 : cluster [DBG] pgmap v6547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:35 smithi115 bash[46934]: audit 2022-04-22T19:59:35.193221+0000 mon.a (mon.0) 8282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:36 smithi121 bash[36902]: audit 2022-04-22T19:59:36.525332+0000 mon.a (mon.0) 8283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:36 smithi115 bash[45495]: audit 2022-04-22T19:59:36.525332+0000 mon.a (mon.0) 8283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:36 smithi115 bash[46934]: audit 2022-04-22T19:59:36.525332+0000 mon.a (mon.0) 8283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:37 smithi121 bash[36902]: cluster 2022-04-22T19:59:36.576664+0000 mgr.y (mgr.34107) 5263 : cluster [DBG] pgmap v6548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:38.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:37 smithi115 bash[45495]: cluster 2022-04-22T19:59:36.576664+0000 mgr.y (mgr.34107) 5263 : cluster [DBG] pgmap v6548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:38.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:37 smithi115 bash[46934]: cluster 2022-04-22T19:59:36.576664+0000 mgr.y (mgr.34107) 5263 : cluster [DBG] pgmap v6548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:38 smithi121 bash[36902]: audit 2022-04-22T19:59:37.973339+0000 mon.a (mon.0) 8284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:39.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:38 smithi115 bash[45495]: audit 2022-04-22T19:59:37.973339+0000 mon.a (mon.0) 8284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:39.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:38 smithi115 bash[46934]: audit 2022-04-22T19:59:37.973339+0000 mon.a (mon.0) 8284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:39 smithi121 bash[36902]: cluster 2022-04-22T19:59:38.577838+0000 mgr.y (mgr.34107) 5264 : cluster [DBG] pgmap v6549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:39 smithi115 bash[45495]: cluster 2022-04-22T19:59:38.577838+0000 mgr.y (mgr.34107) 5264 : cluster [DBG] pgmap v6549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:39 smithi115 bash[46934]: cluster 2022-04-22T19:59:38.577838+0000 mgr.y (mgr.34107) 5264 : cluster [DBG] pgmap v6549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:40.922 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T19:59:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:40 smithi121 bash[36902]: audit 2022-04-22T19:59:40.196717+0000 mon.a (mon.0) 8285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:40 smithi115 bash[46934]: audit 2022-04-22T19:59:40.196717+0000 mon.a (mon.0) 8285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:40 smithi115 bash[45495]: audit 2022-04-22T19:59:40.196717+0000 mon.a (mon.0) 8285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:41.279 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T19:59:41.279 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T19:59:41.279 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T19:59:41.279 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T19:59:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T19:59:41.281 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T19:59:41.685 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T19:59:41.685 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T19:59:41.686 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T19:59:41.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:41.686 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T19:59:41.686 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:59:41.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T19:59:41.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T19:59:41.688 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T19:59:41.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T19:59:41.688 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T19:59:41.688 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T19:59:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: cluster 2022-04-22T19:59:40.579021+0000 mgr.y (mgr.34107) 5265 : cluster [DBG] pgmap v6550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:40.817905+0000 mon.a (mon.0) 8286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:40.820636+0000 mon.a (mon.0) 8287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:40.822366+0000 mon.a (mon.0) 8288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:41.162679+0000 mon.a (mon.0) 8289 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:41.528851+0000 mon.a (mon.0) 8290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[45495]: audit 2022-04-22T19:59:41.675715+0000 mon.c (mon.1) 221 : audit [DBG] from='client.? 172.21.15.115:0/368480382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: cluster 2022-04-22T19:59:40.579021+0000 mgr.y (mgr.34107) 5265 : cluster [DBG] pgmap v6550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:40.817905+0000 mon.a (mon.0) 8286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:40.820636+0000 mon.a (mon.0) 8287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:40.822366+0000 mon.a (mon.0) 8288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:41.162679+0000 mon.a (mon.0) 8289 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:41.528851+0000 mon.a (mon.0) 8290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:42.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:41 smithi115 bash[46934]: audit 2022-04-22T19:59:41.675715+0000 mon.c (mon.1) 221 : audit [DBG] from='client.? 172.21.15.115:0/368480382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: cluster 2022-04-22T19:59:40.579021+0000 mgr.y (mgr.34107) 5265 : cluster [DBG] pgmap v6550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:40.817905+0000 mon.a (mon.0) 8286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:40.820636+0000 mon.a (mon.0) 8287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:40.822366+0000 mon.a (mon.0) 8288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:41.162679+0000 mon.a (mon.0) 8289 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:41.528851+0000 mon.a (mon.0) 8290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:41 smithi121 bash[36902]: audit 2022-04-22T19:59:41.675715+0000 mon.c (mon.1) 221 : audit [DBG] from='client.? 172.21.15.115:0/368480382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T19:59:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:42 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:42.209Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:42 smithi115 bash[45495]: audit 2022-04-22T19:59:40.899305+0000 mgr.y (mgr.34107) 5266 : audit [DBG] from='client.40878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:42 smithi115 bash[45495]: audit 2022-04-22T19:59:41.246299+0000 mgr.y (mgr.34107) 5267 : audit [DBG] from='client.50818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:42 smithi115 bash[46934]: audit 2022-04-22T19:59:40.899305+0000 mgr.y (mgr.34107) 5266 : audit [DBG] from='client.40878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:42 smithi115 bash[46934]: audit 2022-04-22T19:59:41.246299+0000 mgr.y (mgr.34107) 5267 : audit [DBG] from='client.50818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:42 smithi121 bash[36902]: audit 2022-04-22T19:59:40.899305+0000 mgr.y (mgr.34107) 5266 : audit [DBG] from='client.40878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:42 smithi121 bash[36902]: audit 2022-04-22T19:59:41.246299+0000 mgr.y (mgr.34107) 5267 : audit [DBG] from='client.50818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T19:59:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:43 smithi115 bash[45495]: cluster 2022-04-22T19:59:42.580137+0000 mgr.y (mgr.34107) 5268 : cluster [DBG] pgmap v6551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:43 smithi115 bash[45495]: audit 2022-04-22T19:59:42.976381+0000 mon.a (mon.0) 8291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:43 smithi115 bash[46934]: cluster 2022-04-22T19:59:42.580137+0000 mgr.y (mgr.34107) 5268 : cluster [DBG] pgmap v6551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:43 smithi115 bash[46934]: audit 2022-04-22T19:59:42.976381+0000 mon.a (mon.0) 8291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:43 smithi121 bash[36902]: cluster 2022-04-22T19:59:42.580137+0000 mgr.y (mgr.34107) 5268 : cluster [DBG] pgmap v6551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:44.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:43 smithi121 bash[36902]: audit 2022-04-22T19:59:42.976381+0000 mon.a (mon.0) 8291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:44 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:44.086Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:45 smithi115 bash[45495]: cluster 2022-04-22T19:59:44.581253+0000 mgr.y (mgr.34107) 5269 : cluster [DBG] pgmap v6552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:45 smithi115 bash[45495]: audit 2022-04-22T19:59:45.200299+0000 mon.a (mon.0) 8292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:45 smithi115 bash[46934]: cluster 2022-04-22T19:59:44.581253+0000 mgr.y (mgr.34107) 5269 : cluster [DBG] pgmap v6552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:45 smithi115 bash[46934]: audit 2022-04-22T19:59:45.200299+0000 mon.a (mon.0) 8292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:45 smithi121 bash[36902]: cluster 2022-04-22T19:59:44.581253+0000 mgr.y (mgr.34107) 5269 : cluster [DBG] pgmap v6552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:45 smithi121 bash[36902]: audit 2022-04-22T19:59:45.200299+0000 mon.a (mon.0) 8292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:46 smithi115 bash[45495]: audit 2022-04-22T19:59:46.531424+0000 mon.a (mon.0) 8293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:47.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:46 smithi115 bash[46934]: audit 2022-04-22T19:59:46.531424+0000 mon.a (mon.0) 8293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:46 smithi121 bash[36902]: audit 2022-04-22T19:59:46.531424+0000 mon.a (mon.0) 8293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:47 smithi115 bash[46934]: cluster 2022-04-22T19:59:46.582344+0000 mgr.y (mgr.34107) 5270 : cluster [DBG] pgmap v6553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:47 smithi115 bash[45495]: cluster 2022-04-22T19:59:46.582344+0000 mgr.y (mgr.34107) 5270 : cluster [DBG] pgmap v6553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:47 smithi121 bash[36902]: cluster 2022-04-22T19:59:46.582344+0000 mgr.y (mgr.34107) 5270 : cluster [DBG] pgmap v6553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:48 smithi115 bash[45495]: audit 2022-04-22T19:59:47.979421+0000 mon.a (mon.0) 8294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:48 smithi115 bash[46934]: audit 2022-04-22T19:59:47.979421+0000 mon.a (mon.0) 8294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:49.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:48 smithi121 bash[36902]: audit 2022-04-22T19:59:47.979421+0000 mon.a (mon.0) 8294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:49 smithi121 bash[36902]: cluster 2022-04-22T19:59:48.583529+0000 mgr.y (mgr.34107) 5271 : cluster [DBG] pgmap v6554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:49 smithi115 bash[45495]: cluster 2022-04-22T19:59:48.583529+0000 mgr.y (mgr.34107) 5271 : cluster [DBG] pgmap v6554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:50.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:49 smithi115 bash[46934]: cluster 2022-04-22T19:59:48.583529+0000 mgr.y (mgr.34107) 5271 : cluster [DBG] pgmap v6554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:50 smithi121 bash[36902]: audit 2022-04-22T19:59:50.204126+0000 mon.a (mon.0) 8295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:51.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:50 smithi115 bash[45495]: audit 2022-04-22T19:59:50.204126+0000 mon.a (mon.0) 8295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:51.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:50 smithi115 bash[46934]: audit 2022-04-22T19:59:50.204126+0000 mon.a (mon.0) 8295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:52.218 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:51 smithi115 bash[45495]: cluster 2022-04-22T19:59:50.584610+0000 mgr.y (mgr.34107) 5272 : cluster [DBG] pgmap v6555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:52.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:51 smithi115 bash[45495]: audit 2022-04-22T19:59:51.534047+0000 mon.a (mon.0) 8296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:52.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:51 smithi115 bash[46934]: cluster 2022-04-22T19:59:50.584610+0000 mgr.y (mgr.34107) 5272 : cluster [DBG] pgmap v6555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:52.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:51 smithi115 bash[46934]: audit 2022-04-22T19:59:51.534047+0000 mon.a (mon.0) 8296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:51 smithi121 bash[36902]: cluster 2022-04-22T19:59:50.584610+0000 mgr.y (mgr.34107) 5272 : cluster [DBG] pgmap v6555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:51 smithi121 bash[36902]: audit 2022-04-22T19:59:51.534047+0000 mon.a (mon.0) 8296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:52 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:52.209Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:54.094 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:53 smithi115 bash[45495]: cluster 2022-04-22T19:59:52.585732+0000 mgr.y (mgr.34107) 5273 : cluster [DBG] pgmap v6556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:54.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:53 smithi115 bash[45495]: audit 2022-04-22T19:59:52.982465+0000 mon.a (mon.0) 8297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:54.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:53 smithi115 bash[46934]: cluster 2022-04-22T19:59:52.585732+0000 mgr.y (mgr.34107) 5273 : cluster [DBG] pgmap v6556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:54.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:53 smithi115 bash[46934]: audit 2022-04-22T19:59:52.982465+0000 mon.a (mon.0) 8297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:53 smithi121 bash[36902]: cluster 2022-04-22T19:59:52.585732+0000 mgr.y (mgr.34107) 5273 : cluster [DBG] pgmap v6556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:53 smithi121 bash[36902]: audit 2022-04-22T19:59:52.982465+0000 mon.a (mon.0) 8297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 19:59:54 smithi115 bash[60466]: level=error ts=2022-04-22T19:59:54.085Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T19:59:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:55 smithi121 bash[36902]: cluster 2022-04-22T19:59:54.586762+0000 mgr.y (mgr.34107) 5274 : cluster [DBG] pgmap v6557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:55 smithi121 bash[36902]: audit 2022-04-22T19:59:55.207756+0000 mon.a (mon.0) 8298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:55 smithi115 bash[45495]: cluster 2022-04-22T19:59:54.586762+0000 mgr.y (mgr.34107) 5274 : cluster [DBG] pgmap v6557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:55 smithi115 bash[45495]: audit 2022-04-22T19:59:55.207756+0000 mon.a (mon.0) 8298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:55 smithi115 bash[46934]: cluster 2022-04-22T19:59:54.586762+0000 mgr.y (mgr.34107) 5274 : cluster [DBG] pgmap v6557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:55 smithi115 bash[46934]: audit 2022-04-22T19:59:55.207756+0000 mon.a (mon.0) 8298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T19:59:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:56 smithi121 bash[36902]: audit 2022-04-22T19:59:56.536900+0000 mon.a (mon.0) 8299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:56 smithi115 bash[45495]: audit 2022-04-22T19:59:56.536900+0000 mon.a (mon.0) 8299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:56 smithi115 bash[46934]: audit 2022-04-22T19:59:56.536900+0000 mon.a (mon.0) 8299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T19:59:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:57 smithi121 bash[36902]: cluster 2022-04-22T19:59:56.587963+0000 mgr.y (mgr.34107) 5275 : cluster [DBG] pgmap v6558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:57 smithi115 bash[45495]: cluster 2022-04-22T19:59:56.587963+0000 mgr.y (mgr.34107) 5275 : cluster [DBG] pgmap v6558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:57 smithi115 bash[46934]: cluster 2022-04-22T19:59:56.587963+0000 mgr.y (mgr.34107) 5275 : cluster [DBG] pgmap v6558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T19:59:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:58 smithi121 bash[36902]: audit 2022-04-22T19:59:57.985461+0000 mon.a (mon.0) 8300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:58 smithi115 bash[45495]: audit 2022-04-22T19:59:57.985461+0000 mon.a (mon.0) 8300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T19:59:59.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:58 smithi115 bash[46934]: audit 2022-04-22T19:59:57.985461+0000 mon.a (mon.0) 8300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 19:59:59 smithi121 bash[36902]: cluster 2022-04-22T19:59:58.589175+0000 mgr.y (mgr.34107) 5276 : cluster [DBG] pgmap v6559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 19:59:59 smithi115 bash[45495]: cluster 2022-04-22T19:59:58.589175+0000 mgr.y (mgr.34107) 5276 : cluster [DBG] pgmap v6559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 19:59:59 smithi115 bash[46934]: cluster 2022-04-22T19:59:58.589175+0000 mgr.y (mgr.34107) 5276 : cluster [DBG] pgmap v6559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:00 smithi121 bash[36902]: cluster 2022-04-22T20:00:00.000123+0000 mon.a (mon.0) 8301 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:00:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:00 smithi121 bash[36902]: audit 2022-04-22T20:00:00.211136+0000 mon.a (mon.0) 8302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:00 smithi115 bash[45495]: cluster 2022-04-22T20:00:00.000123+0000 mon.a (mon.0) 8301 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:00:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:00 smithi115 bash[45495]: audit 2022-04-22T20:00:00.211136+0000 mon.a (mon.0) 8302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:00 smithi115 bash[46934]: cluster 2022-04-22T20:00:00.000123+0000 mon.a (mon.0) 8301 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:00:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:00 smithi115 bash[46934]: audit 2022-04-22T20:00:00.211136+0000 mon.a (mon.0) 8302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:02.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:01 smithi115 bash[45495]: cluster 2022-04-22T20:00:00.590273+0000 mgr.y (mgr.34107) 5277 : cluster [DBG] pgmap v6560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:02.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:01 smithi115 bash[45495]: audit 2022-04-22T20:00:01.540164+0000 mon.a (mon.0) 8303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:02.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:01 smithi115 bash[46934]: cluster 2022-04-22T20:00:00.590273+0000 mgr.y (mgr.34107) 5277 : cluster [DBG] pgmap v6560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:02.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:01 smithi115 bash[46934]: audit 2022-04-22T20:00:01.540164+0000 mon.a (mon.0) 8303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:01 smithi121 bash[36902]: cluster 2022-04-22T20:00:00.590273+0000 mgr.y (mgr.34107) 5277 : cluster [DBG] pgmap v6560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:01 smithi121 bash[36902]: audit 2022-04-22T20:00:01.540164+0000 mon.a (mon.0) 8303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:02.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:04.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:03 smithi115 bash[45495]: cluster 2022-04-22T20:00:02.591396+0000 mgr.y (mgr.34107) 5278 : cluster [DBG] pgmap v6561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:04.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:03 smithi115 bash[45495]: audit 2022-04-22T20:00:02.988432+0000 mon.a (mon.0) 8304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:04.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:03 smithi115 bash[46934]: cluster 2022-04-22T20:00:02.591396+0000 mgr.y (mgr.34107) 5278 : cluster [DBG] pgmap v6561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:04.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:03 smithi115 bash[46934]: audit 2022-04-22T20:00:02.988432+0000 mon.a (mon.0) 8304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:04.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:03 smithi121 bash[36902]: cluster 2022-04-22T20:00:02.591396+0000 mgr.y (mgr.34107) 5278 : cluster [DBG] pgmap v6561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:03 smithi121 bash[36902]: audit 2022-04-22T20:00:02.988432+0000 mon.a (mon.0) 8304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:04.088Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:05 smithi121 bash[36902]: cluster 2022-04-22T20:00:04.592512+0000 mgr.y (mgr.34107) 5279 : cluster [DBG] pgmap v6562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:05 smithi121 bash[36902]: audit 2022-04-22T20:00:05.214783+0000 mon.a (mon.0) 8305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:05 smithi115 bash[45495]: cluster 2022-04-22T20:00:04.592512+0000 mgr.y (mgr.34107) 5279 : cluster [DBG] pgmap v6562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:05 smithi115 bash[45495]: audit 2022-04-22T20:00:05.214783+0000 mon.a (mon.0) 8305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:05 smithi115 bash[46934]: cluster 2022-04-22T20:00:04.592512+0000 mgr.y (mgr.34107) 5279 : cluster [DBG] pgmap v6562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:05 smithi115 bash[46934]: audit 2022-04-22T20:00:05.214783+0000 mon.a (mon.0) 8305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:07.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:06 smithi121 bash[36902]: audit 2022-04-22T20:00:06.543225+0000 mon.a (mon.0) 8306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:06 smithi115 bash[45495]: audit 2022-04-22T20:00:06.543225+0000 mon.a (mon.0) 8306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:06 smithi115 bash[46934]: audit 2022-04-22T20:00:06.543225+0000 mon.a (mon.0) 8306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:07 smithi121 bash[36902]: cluster 2022-04-22T20:00:06.593586+0000 mgr.y (mgr.34107) 5280 : cluster [DBG] pgmap v6563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:07 smithi115 bash[45495]: cluster 2022-04-22T20:00:06.593586+0000 mgr.y (mgr.34107) 5280 : cluster [DBG] pgmap v6563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:07 smithi115 bash[46934]: cluster 2022-04-22T20:00:06.593586+0000 mgr.y (mgr.34107) 5280 : cluster [DBG] pgmap v6563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:08 smithi121 bash[36902]: audit 2022-04-22T20:00:07.991607+0000 mon.a (mon.0) 8307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:08 smithi115 bash[45495]: audit 2022-04-22T20:00:07.991607+0000 mon.a (mon.0) 8307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:08 smithi115 bash[46934]: audit 2022-04-22T20:00:07.991607+0000 mon.a (mon.0) 8307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:09 smithi121 bash[36902]: cluster 2022-04-22T20:00:08.594693+0000 mgr.y (mgr.34107) 5281 : cluster [DBG] pgmap v6564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:09 smithi115 bash[45495]: cluster 2022-04-22T20:00:08.594693+0000 mgr.y (mgr.34107) 5281 : cluster [DBG] pgmap v6564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:09 smithi115 bash[46934]: cluster 2022-04-22T20:00:08.594693+0000 mgr.y (mgr.34107) 5281 : cluster [DBG] pgmap v6564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:10 smithi121 bash[36902]: audit 2022-04-22T20:00:10.218605+0000 mon.a (mon.0) 8308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:10 smithi121 bash[36902]: cluster 2022-04-22T20:00:10.595881+0000 mgr.y (mgr.34107) 5282 : cluster [DBG] pgmap v6565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:10 smithi115 bash[45495]: audit 2022-04-22T20:00:10.218605+0000 mon.a (mon.0) 8308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:11.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:10 smithi115 bash[45495]: cluster 2022-04-22T20:00:10.595881+0000 mgr.y (mgr.34107) 5282 : cluster [DBG] pgmap v6565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:10 smithi115 bash[46934]: audit 2022-04-22T20:00:10.218605+0000 mon.a (mon.0) 8308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:10 smithi115 bash[46934]: cluster 2022-04-22T20:00:10.595881+0000 mgr.y (mgr.34107) 5282 : cluster [DBG] pgmap v6565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:12.073 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:00:12.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:11 smithi115 bash[46934]: audit 2022-04-22T20:00:11.546470+0000 mon.a (mon.0) 8309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:12.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:11 smithi115 bash[45495]: audit 2022-04-22T20:00:11.546470+0000 mon.a (mon.0) 8309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:11 smithi121 bash[36902]: audit 2022-04-22T20:00:11.546470+0000 mon.a (mon.0) 8309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:12.411 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:00:12.424 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:00:12.425 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:00:12.426 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:00:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:12.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:12.817 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:00:12.817 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:00:12.817 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:00:12.818 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:00:12.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:12.819 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:00:12.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:00:12.819 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:00:12.819 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:00:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:12 smithi121 bash[36902]: audit 2022-04-22T20:00:12.044901+0000 mgr.y (mgr.34107) 5283 : audit [DBG] from='client.40896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:12 smithi121 bash[36902]: audit 2022-04-22T20:00:12.384171+0000 mgr.y (mgr.34107) 5284 : audit [DBG] from='client.50836 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:12 smithi121 bash[36902]: cluster 2022-04-22T20:00:12.596998+0000 mgr.y (mgr.34107) 5285 : cluster [DBG] pgmap v6566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:12 smithi121 bash[36902]: audit 2022-04-22T20:00:12.808286+0000 mon.a (mon.0) 8310 : audit [DBG] from='client.? 172.21.15.115:0/1272578589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[45495]: audit 2022-04-22T20:00:12.044901+0000 mgr.y (mgr.34107) 5283 : audit [DBG] from='client.40896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[45495]: audit 2022-04-22T20:00:12.384171+0000 mgr.y (mgr.34107) 5284 : audit [DBG] from='client.50836 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[45495]: cluster 2022-04-22T20:00:12.596998+0000 mgr.y (mgr.34107) 5285 : cluster [DBG] pgmap v6566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[45495]: audit 2022-04-22T20:00:12.808286+0000 mon.a (mon.0) 8310 : audit [DBG] from='client.? 172.21.15.115:0/1272578589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[46934]: audit 2022-04-22T20:00:12.044901+0000 mgr.y (mgr.34107) 5283 : audit [DBG] from='client.40896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[46934]: audit 2022-04-22T20:00:12.384171+0000 mgr.y (mgr.34107) 5284 : audit [DBG] from='client.50836 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[46934]: cluster 2022-04-22T20:00:12.596998+0000 mgr.y (mgr.34107) 5285 : cluster [DBG] pgmap v6566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:12 smithi115 bash[46934]: audit 2022-04-22T20:00:12.808286+0000 mon.a (mon.0) 8310 : audit [DBG] from='client.? 172.21.15.115:0/1272578589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:14.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:13 smithi121 bash[36902]: audit 2022-04-22T20:00:12.994675+0000 mon.a (mon.0) 8311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:14.089Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:13 smithi115 bash[45495]: audit 2022-04-22T20:00:12.994675+0000 mon.a (mon.0) 8311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:13 smithi115 bash[46934]: audit 2022-04-22T20:00:12.994675+0000 mon.a (mon.0) 8311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:14 smithi121 bash[36902]: cluster 2022-04-22T20:00:14.598033+0000 mgr.y (mgr.34107) 5286 : cluster [DBG] pgmap v6567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:14 smithi115 bash[45495]: cluster 2022-04-22T20:00:14.598033+0000 mgr.y (mgr.34107) 5286 : cluster [DBG] pgmap v6567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:14 smithi115 bash[46934]: cluster 2022-04-22T20:00:14.598033+0000 mgr.y (mgr.34107) 5286 : cluster [DBG] pgmap v6567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:15 smithi121 bash[36902]: audit 2022-04-22T20:00:15.052863+0000 mon.a (mon.0) 8312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:00:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:15 smithi121 bash[36902]: audit 2022-04-22T20:00:15.053205+0000 mon.a (mon.0) 8313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:00:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:15 smithi121 bash[36902]: audit 2022-04-22T20:00:15.222441+0000 mon.a (mon.0) 8314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[45495]: audit 2022-04-22T20:00:15.052863+0000 mon.a (mon.0) 8312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:00:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[45495]: audit 2022-04-22T20:00:15.053205+0000 mon.a (mon.0) 8313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:00:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[45495]: audit 2022-04-22T20:00:15.222441+0000 mon.a (mon.0) 8314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[46934]: audit 2022-04-22T20:00:15.052863+0000 mon.a (mon.0) 8312 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:00:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[46934]: audit 2022-04-22T20:00:15.053205+0000 mon.a (mon.0) 8313 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:00:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:15 smithi115 bash[46934]: audit 2022-04-22T20:00:15.222441+0000 mon.a (mon.0) 8314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:16 smithi121 bash[36902]: audit 2022-04-22T20:00:16.549655+0000 mon.a (mon.0) 8315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:17.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:16 smithi121 bash[36902]: cluster 2022-04-22T20:00:16.599064+0000 mgr.y (mgr.34107) 5287 : cluster [DBG] pgmap v6568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:17.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:16 smithi115 bash[45495]: audit 2022-04-22T20:00:16.549655+0000 mon.a (mon.0) 8315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:17.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:16 smithi115 bash[45495]: cluster 2022-04-22T20:00:16.599064+0000 mgr.y (mgr.34107) 5287 : cluster [DBG] pgmap v6568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:16 smithi115 bash[46934]: audit 2022-04-22T20:00:16.549655+0000 mon.a (mon.0) 8315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:17.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:16 smithi115 bash[46934]: cluster 2022-04-22T20:00:16.599064+0000 mgr.y (mgr.34107) 5287 : cluster [DBG] pgmap v6568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:18 smithi121 bash[36902]: audit 2022-04-22T20:00:17.997771+0000 mon.a (mon.0) 8316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:18 smithi115 bash[45495]: audit 2022-04-22T20:00:17.997771+0000 mon.a (mon.0) 8316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:18 smithi115 bash[46934]: audit 2022-04-22T20:00:17.997771+0000 mon.a (mon.0) 8316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:19 smithi121 bash[36902]: cluster 2022-04-22T20:00:18.600299+0000 mgr.y (mgr.34107) 5288 : cluster [DBG] pgmap v6569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:19 smithi115 bash[45495]: cluster 2022-04-22T20:00:18.600299+0000 mgr.y (mgr.34107) 5288 : cluster [DBG] pgmap v6569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:19 smithi115 bash[46934]: cluster 2022-04-22T20:00:18.600299+0000 mgr.y (mgr.34107) 5288 : cluster [DBG] pgmap v6569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:20 smithi121 bash[36902]: audit 2022-04-22T20:00:20.226294+0000 mon.a (mon.0) 8317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:20 smithi115 bash[46934]: audit 2022-04-22T20:00:20.226294+0000 mon.a (mon.0) 8317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:20 smithi115 bash[45495]: audit 2022-04-22T20:00:20.226294+0000 mon.a (mon.0) 8317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:21.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:21 smithi115 bash[45495]: cluster 2022-04-22T20:00:20.601334+0000 mgr.y (mgr.34107) 5289 : cluster [DBG] pgmap v6570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:21.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:21 smithi115 bash[46934]: cluster 2022-04-22T20:00:20.601334+0000 mgr.y (mgr.34107) 5289 : cluster [DBG] pgmap v6570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:21 smithi121 bash[36902]: cluster 2022-04-22T20:00:20.601334+0000 mgr.y (mgr.34107) 5289 : cluster [DBG] pgmap v6570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:22.519 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:22.211Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:22 smithi115 bash[45495]: audit 2022-04-22T20:00:21.552727+0000 mon.a (mon.0) 8318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:22 smithi115 bash[46934]: audit 2022-04-22T20:00:21.552727+0000 mon.a (mon.0) 8318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:22 smithi121 bash[36902]: audit 2022-04-22T20:00:21.552727+0000 mon.a (mon.0) 8318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:23.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:23 smithi115 bash[45495]: cluster 2022-04-22T20:00:22.602338+0000 mgr.y (mgr.34107) 5290 : cluster [DBG] pgmap v6571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:23.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:23 smithi115 bash[45495]: audit 2022-04-22T20:00:23.000651+0000 mon.a (mon.0) 8319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:23.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:23 smithi115 bash[46934]: cluster 2022-04-22T20:00:22.602338+0000 mgr.y (mgr.34107) 5290 : cluster [DBG] pgmap v6571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:23.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:23 smithi115 bash[46934]: audit 2022-04-22T20:00:23.000651+0000 mon.a (mon.0) 8319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:23 smithi121 bash[36902]: cluster 2022-04-22T20:00:22.602338+0000 mgr.y (mgr.34107) 5290 : cluster [DBG] pgmap v6571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:23 smithi121 bash[36902]: audit 2022-04-22T20:00:23.000651+0000 mon.a (mon.0) 8319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:24.089Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:25 smithi121 bash[36902]: cluster 2022-04-22T20:00:24.603409+0000 mgr.y (mgr.34107) 5291 : cluster [DBG] pgmap v6572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:25 smithi121 bash[36902]: audit 2022-04-22T20:00:25.230279+0000 mon.a (mon.0) 8320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:25 smithi115 bash[45495]: cluster 2022-04-22T20:00:24.603409+0000 mgr.y (mgr.34107) 5291 : cluster [DBG] pgmap v6572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:26.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:25 smithi115 bash[45495]: audit 2022-04-22T20:00:25.230279+0000 mon.a (mon.0) 8320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:26.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:25 smithi115 bash[46934]: cluster 2022-04-22T20:00:24.603409+0000 mgr.y (mgr.34107) 5291 : cluster [DBG] pgmap v6572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:26.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:25 smithi115 bash[46934]: audit 2022-04-22T20:00:25.230279+0000 mon.a (mon.0) 8320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:26 smithi121 bash[36902]: audit 2022-04-22T20:00:26.555891+0000 mon.a (mon.0) 8321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:26 smithi115 bash[46934]: audit 2022-04-22T20:00:26.555891+0000 mon.a (mon.0) 8321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:26 smithi115 bash[45495]: audit 2022-04-22T20:00:26.555891+0000 mon.a (mon.0) 8321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:27 smithi121 bash[36902]: cluster 2022-04-22T20:00:26.604487+0000 mgr.y (mgr.34107) 5292 : cluster [DBG] pgmap v6573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:27 smithi115 bash[45495]: cluster 2022-04-22T20:00:26.604487+0000 mgr.y (mgr.34107) 5292 : cluster [DBG] pgmap v6573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:28.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:27 smithi115 bash[46934]: cluster 2022-04-22T20:00:26.604487+0000 mgr.y (mgr.34107) 5292 : cluster [DBG] pgmap v6573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:28 smithi121 bash[36902]: audit 2022-04-22T20:00:28.003711+0000 mon.a (mon.0) 8322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:28 smithi115 bash[45495]: audit 2022-04-22T20:00:28.003711+0000 mon.a (mon.0) 8322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:28 smithi115 bash[46934]: audit 2022-04-22T20:00:28.003711+0000 mon.a (mon.0) 8322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:29 smithi121 bash[36902]: cluster 2022-04-22T20:00:28.605623+0000 mgr.y (mgr.34107) 5293 : cluster [DBG] pgmap v6574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:29 smithi115 bash[46934]: cluster 2022-04-22T20:00:28.605623+0000 mgr.y (mgr.34107) 5293 : cluster [DBG] pgmap v6574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:30.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:29 smithi115 bash[45495]: cluster 2022-04-22T20:00:28.605623+0000 mgr.y (mgr.34107) 5293 : cluster [DBG] pgmap v6574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:30 smithi121 bash[36902]: audit 2022-04-22T20:00:30.234031+0000 mon.a (mon.0) 8323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:30 smithi115 bash[45495]: audit 2022-04-22T20:00:30.234031+0000 mon.a (mon.0) 8323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:30 smithi115 bash[46934]: audit 2022-04-22T20:00:30.234031+0000 mon.a (mon.0) 8323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:31 smithi121 bash[36902]: cluster 2022-04-22T20:00:30.606747+0000 mgr.y (mgr.34107) 5294 : cluster [DBG] pgmap v6575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:31 smithi121 bash[36902]: audit 2022-04-22T20:00:31.558999+0000 mon.a (mon.0) 8324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:31 smithi115 bash[45495]: cluster 2022-04-22T20:00:30.606747+0000 mgr.y (mgr.34107) 5294 : cluster [DBG] pgmap v6575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:31 smithi115 bash[45495]: audit 2022-04-22T20:00:31.558999+0000 mon.a (mon.0) 8324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:31 smithi115 bash[46934]: cluster 2022-04-22T20:00:30.606747+0000 mgr.y (mgr.34107) 5294 : cluster [DBG] pgmap v6575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:31 smithi115 bash[46934]: audit 2022-04-22T20:00:31.558999+0000 mon.a (mon.0) 8324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:32.212Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:33 smithi121 bash[36902]: cluster 2022-04-22T20:00:32.607923+0000 mgr.y (mgr.34107) 5295 : cluster [DBG] pgmap v6576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:33 smithi121 bash[36902]: audit 2022-04-22T20:00:33.006728+0000 mon.a (mon.0) 8325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:33 smithi115 bash[45495]: cluster 2022-04-22T20:00:32.607923+0000 mgr.y (mgr.34107) 5295 : cluster [DBG] pgmap v6576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:33 smithi115 bash[45495]: audit 2022-04-22T20:00:33.006728+0000 mon.a (mon.0) 8325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:33 smithi115 bash[46934]: cluster 2022-04-22T20:00:32.607923+0000 mgr.y (mgr.34107) 5295 : cluster [DBG] pgmap v6576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:34.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:33 smithi115 bash[46934]: audit 2022-04-22T20:00:33.006728+0000 mon.a (mon.0) 8325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:34.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:34.089Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:35 smithi121 bash[36902]: cluster 2022-04-22T20:00:34.609031+0000 mgr.y (mgr.34107) 5296 : cluster [DBG] pgmap v6577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:36.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:35 smithi121 bash[36902]: audit 2022-04-22T20:00:35.237812+0000 mon.a (mon.0) 8326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:35 smithi115 bash[45495]: cluster 2022-04-22T20:00:34.609031+0000 mgr.y (mgr.34107) 5296 : cluster [DBG] pgmap v6577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:36.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:35 smithi115 bash[45495]: audit 2022-04-22T20:00:35.237812+0000 mon.a (mon.0) 8326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:35 smithi115 bash[46934]: cluster 2022-04-22T20:00:34.609031+0000 mgr.y (mgr.34107) 5296 : cluster [DBG] pgmap v6577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:36.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:35 smithi115 bash[46934]: audit 2022-04-22T20:00:35.237812+0000 mon.a (mon.0) 8326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:37.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:36 smithi121 bash[36902]: audit 2022-04-22T20:00:36.562213+0000 mon.a (mon.0) 8327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:36 smithi115 bash[45495]: audit 2022-04-22T20:00:36.562213+0000 mon.a (mon.0) 8327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:36 smithi115 bash[46934]: audit 2022-04-22T20:00:36.562213+0000 mon.a (mon.0) 8327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:37 smithi121 bash[36902]: cluster 2022-04-22T20:00:36.610093+0000 mgr.y (mgr.34107) 5297 : cluster [DBG] pgmap v6578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:38.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:37 smithi115 bash[46934]: cluster 2022-04-22T20:00:36.610093+0000 mgr.y (mgr.34107) 5297 : cluster [DBG] pgmap v6578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:37 smithi115 bash[45495]: cluster 2022-04-22T20:00:36.610093+0000 mgr.y (mgr.34107) 5297 : cluster [DBG] pgmap v6578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:39.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:38 smithi121 bash[36902]: audit 2022-04-22T20:00:38.009666+0000 mon.a (mon.0) 8328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:39.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:38 smithi115 bash[46934]: audit 2022-04-22T20:00:38.009666+0000 mon.a (mon.0) 8328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:39.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:38 smithi115 bash[45495]: audit 2022-04-22T20:00:38.009666+0000 mon.a (mon.0) 8328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:40.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:39 smithi121 bash[36902]: cluster 2022-04-22T20:00:38.611289+0000 mgr.y (mgr.34107) 5298 : cluster [DBG] pgmap v6579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:39 smithi115 bash[46934]: cluster 2022-04-22T20:00:38.611289+0000 mgr.y (mgr.34107) 5298 : cluster [DBG] pgmap v6579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:39 smithi115 bash[45495]: cluster 2022-04-22T20:00:38.611289+0000 mgr.y (mgr.34107) 5298 : cluster [DBG] pgmap v6579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:40 smithi121 bash[36902]: audit 2022-04-22T20:00:40.241511+0000 mon.a (mon.0) 8329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:40 smithi115 bash[46934]: audit 2022-04-22T20:00:40.241511+0000 mon.a (mon.0) 8329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:40 smithi115 bash[45495]: audit 2022-04-22T20:00:40.241511+0000 mon.a (mon.0) 8329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[46934]: cluster 2022-04-22T20:00:40.612409+0000 mgr.y (mgr.34107) 5299 : cluster [DBG] pgmap v6580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[46934]: audit 2022-04-22T20:00:41.172413+0000 mon.a (mon.0) 8330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:00:41.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[46934]: audit 2022-04-22T20:00:41.175550+0000 mon.a (mon.0) 8331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[46934]: audit 2022-04-22T20:00:41.177170+0000 mon.a (mon.0) 8332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[46934]: audit 2022-04-22T20:00:41.565632+0000 mon.a (mon.0) 8333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[45495]: cluster 2022-04-22T20:00:40.612409+0000 mgr.y (mgr.34107) 5299 : cluster [DBG] pgmap v6580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[45495]: audit 2022-04-22T20:00:41.172413+0000 mon.a (mon.0) 8330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[45495]: audit 2022-04-22T20:00:41.175550+0000 mon.a (mon.0) 8331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[45495]: audit 2022-04-22T20:00:41.177170+0000 mon.a (mon.0) 8332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:00:41.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:41 smithi115 bash[45495]: audit 2022-04-22T20:00:41.565632+0000 mon.a (mon.0) 8333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:41 smithi121 bash[36902]: cluster 2022-04-22T20:00:40.612409+0000 mgr.y (mgr.34107) 5299 : cluster [DBG] pgmap v6580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:41 smithi121 bash[36902]: audit 2022-04-22T20:00:41.172413+0000 mon.a (mon.0) 8330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:41 smithi121 bash[36902]: audit 2022-04-22T20:00:41.175550+0000 mon.a (mon.0) 8331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:41 smithi121 bash[36902]: audit 2022-04-22T20:00:41.177170+0000 mon.a (mon.0) 8332 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:41 smithi121 bash[36902]: audit 2022-04-22T20:00:41.565632+0000 mon.a (mon.0) 8333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:42.213Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:43.191 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 10m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 10m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 10m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:00:43.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:00:43.532 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:00:43.532 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 10m ago 2h 189M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:00:43.532 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:00:43.532 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 10m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:00:43.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 10m ago 2h 574M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:00:43.534 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 10m ago 2h 706M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:00:43.534 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 10m ago 2h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:00:43.534 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 10m ago 2h 636M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:00:43.534 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 10m ago 2h 68.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:00:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:00:43.938 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:00:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:00:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:43 smithi121 bash[36902]: cluster 2022-04-22T20:00:42.613527+0000 mgr.y (mgr.34107) 5300 : cluster [DBG] pgmap v6581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:43 smithi121 bash[36902]: audit 2022-04-22T20:00:43.012646+0000 mon.a (mon.0) 8334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:43 smithi115 bash[45495]: cluster 2022-04-22T20:00:42.613527+0000 mgr.y (mgr.34107) 5300 : cluster [DBG] pgmap v6581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:43 smithi115 bash[45495]: audit 2022-04-22T20:00:43.012646+0000 mon.a (mon.0) 8334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:43 smithi115 bash[46934]: cluster 2022-04-22T20:00:42.613527+0000 mgr.y (mgr.34107) 5300 : cluster [DBG] pgmap v6581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:43 smithi115 bash[46934]: audit 2022-04-22T20:00:43.012646+0000 mon.a (mon.0) 8334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:44.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:44.093Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:44 smithi121 bash[36902]: audit 2022-04-22T20:00:43.167688+0000 mgr.y (mgr.34107) 5301 : audit [DBG] from='client.40914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:44 smithi121 bash[36902]: audit 2022-04-22T20:00:43.505951+0000 mgr.y (mgr.34107) 5302 : audit [DBG] from='client.40920 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:44 smithi121 bash[36902]: audit 2022-04-22T20:00:43.928031+0000 mon.a (mon.0) 8335 : audit [DBG] from='client.? 172.21.15.115:0/67391932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:45.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[45495]: audit 2022-04-22T20:00:43.167688+0000 mgr.y (mgr.34107) 5301 : audit [DBG] from='client.40914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[45495]: audit 2022-04-22T20:00:43.505951+0000 mgr.y (mgr.34107) 5302 : audit [DBG] from='client.40920 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[45495]: audit 2022-04-22T20:00:43.928031+0000 mon.a (mon.0) 8335 : audit [DBG] from='client.? 172.21.15.115:0/67391932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[46934]: audit 2022-04-22T20:00:43.167688+0000 mgr.y (mgr.34107) 5301 : audit [DBG] from='client.40914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[46934]: audit 2022-04-22T20:00:43.505951+0000 mgr.y (mgr.34107) 5302 : audit [DBG] from='client.40920 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:00:45.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:44 smithi115 bash[46934]: audit 2022-04-22T20:00:43.928031+0000 mon.a (mon.0) 8335 : audit [DBG] from='client.? 172.21.15.115:0/67391932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:00:45.985 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:45 smithi121 bash[36902]: cluster 2022-04-22T20:00:44.614616+0000 mgr.y (mgr.34107) 5303 : cluster [DBG] pgmap v6582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:45.985 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:45 smithi121 bash[36902]: audit 2022-04-22T20:00:45.245337+0000 mon.a (mon.0) 8336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:45.985 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:45 smithi121 bash[36902]: audit 2022-04-22T20:00:45.670822+0000 mon.a (mon.0) 8337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[45495]: cluster 2022-04-22T20:00:44.614616+0000 mgr.y (mgr.34107) 5303 : cluster [DBG] pgmap v6582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:46.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[45495]: audit 2022-04-22T20:00:45.245337+0000 mon.a (mon.0) 8336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[45495]: audit 2022-04-22T20:00:45.670822+0000 mon.a (mon.0) 8337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[46934]: cluster 2022-04-22T20:00:44.614616+0000 mgr.y (mgr.34107) 5303 : cluster [DBG] pgmap v6582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[46934]: audit 2022-04-22T20:00:45.245337+0000 mon.a (mon.0) 8336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:46.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:45 smithi115 bash[46934]: audit 2022-04-22T20:00:45.670822+0000 mon.a (mon.0) 8337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:47 smithi121 bash[36902]: audit 2022-04-22T20:00:45.988597+0000 mon.a (mon.0) 8338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:47 smithi121 bash[36902]: audit 2022-04-22T20:00:46.568545+0000 mon.a (mon.0) 8339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:47 smithi121 bash[36902]: cluster 2022-04-22T20:00:46.615674+0000 mgr.y (mgr.34107) 5304 : cluster [DBG] pgmap v6583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[45495]: audit 2022-04-22T20:00:45.988597+0000 mon.a (mon.0) 8338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[45495]: audit 2022-04-22T20:00:46.568545+0000 mon.a (mon.0) 8339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[45495]: cluster 2022-04-22T20:00:46.615674+0000 mgr.y (mgr.34107) 5304 : cluster [DBG] pgmap v6583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[46934]: audit 2022-04-22T20:00:45.988597+0000 mon.a (mon.0) 8338 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:00:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[46934]: audit 2022-04-22T20:00:46.568545+0000 mon.a (mon.0) 8339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:46 smithi115 bash[46934]: cluster 2022-04-22T20:00:46.615674+0000 mgr.y (mgr.34107) 5304 : cluster [DBG] pgmap v6583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:48 smithi121 bash[36902]: audit 2022-04-22T20:00:48.015347+0000 mon.a (mon.0) 8340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:48 smithi115 bash[45495]: audit 2022-04-22T20:00:48.015347+0000 mon.a (mon.0) 8340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:48 smithi115 bash[46934]: audit 2022-04-22T20:00:48.015347+0000 mon.a (mon.0) 8340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:49.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:49 smithi115 bash[45495]: cluster 2022-04-22T20:00:48.616831+0000 mgr.y (mgr.34107) 5305 : cluster [DBG] pgmap v6584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:49.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:49 smithi115 bash[46934]: cluster 2022-04-22T20:00:48.616831+0000 mgr.y (mgr.34107) 5305 : cluster [DBG] pgmap v6584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:49 smithi121 bash[36902]: cluster 2022-04-22T20:00:48.616831+0000 mgr.y (mgr.34107) 5305 : cluster [DBG] pgmap v6584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:50.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:50 smithi115 bash[45495]: audit 2022-04-22T20:00:50.249012+0000 mon.a (mon.0) 8341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:50.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:50 smithi115 bash[46934]: audit 2022-04-22T20:00:50.249012+0000 mon.a (mon.0) 8341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:50 smithi121 bash[36902]: audit 2022-04-22T20:00:50.249012+0000 mon.a (mon.0) 8341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:51.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:51 smithi115 bash[45495]: cluster 2022-04-22T20:00:50.617960+0000 mgr.y (mgr.34107) 5306 : cluster [DBG] pgmap v6585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:51.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:51 smithi115 bash[46934]: cluster 2022-04-22T20:00:50.617960+0000 mgr.y (mgr.34107) 5306 : cluster [DBG] pgmap v6585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:51 smithi121 bash[36902]: cluster 2022-04-22T20:00:50.617960+0000 mgr.y (mgr.34107) 5306 : cluster [DBG] pgmap v6585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:52.529 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:52.215Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:52.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:52 smithi115 bash[45495]: audit 2022-04-22T20:00:51.571665+0000 mon.a (mon.0) 8342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:52.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:52 smithi115 bash[46934]: audit 2022-04-22T20:00:51.571665+0000 mon.a (mon.0) 8342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:52 smithi121 bash[36902]: audit 2022-04-22T20:00:51.571665+0000 mon.a (mon.0) 8342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:53 smithi115 bash[45495]: cluster 2022-04-22T20:00:52.619067+0000 mgr.y (mgr.34107) 5307 : cluster [DBG] pgmap v6586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:53 smithi115 bash[45495]: audit 2022-04-22T20:00:53.018456+0000 mon.a (mon.0) 8343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:53.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:53 smithi115 bash[46934]: cluster 2022-04-22T20:00:52.619067+0000 mgr.y (mgr.34107) 5307 : cluster [DBG] pgmap v6586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:53 smithi115 bash[46934]: audit 2022-04-22T20:00:53.018456+0000 mon.a (mon.0) 8343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:53 smithi121 bash[36902]: cluster 2022-04-22T20:00:52.619067+0000 mgr.y (mgr.34107) 5307 : cluster [DBG] pgmap v6586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:53 smithi121 bash[36902]: audit 2022-04-22T20:00:53.018456+0000 mon.a (mon.0) 8343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:00:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:00:54.092Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:00:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:55 smithi121 bash[36902]: cluster 2022-04-22T20:00:54.620113+0000 mgr.y (mgr.34107) 5308 : cluster [DBG] pgmap v6587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:55 smithi121 bash[36902]: audit 2022-04-22T20:00:55.252816+0000 mon.a (mon.0) 8344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:55 smithi115 bash[45495]: cluster 2022-04-22T20:00:54.620113+0000 mgr.y (mgr.34107) 5308 : cluster [DBG] pgmap v6587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:56.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:55 smithi115 bash[45495]: audit 2022-04-22T20:00:55.252816+0000 mon.a (mon.0) 8344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:55 smithi115 bash[46934]: cluster 2022-04-22T20:00:54.620113+0000 mgr.y (mgr.34107) 5308 : cluster [DBG] pgmap v6587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:56.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:55 smithi115 bash[46934]: audit 2022-04-22T20:00:55.252816+0000 mon.a (mon.0) 8344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:00:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:56 smithi121 bash[36902]: audit 2022-04-22T20:00:56.574733+0000 mon.a (mon.0) 8345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:56 smithi115 bash[46934]: audit 2022-04-22T20:00:56.574733+0000 mon.a (mon.0) 8345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:57.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:56 smithi115 bash[45495]: audit 2022-04-22T20:00:56.574733+0000 mon.a (mon.0) 8345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:00:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:57 smithi121 bash[36902]: cluster 2022-04-22T20:00:56.621189+0000 mgr.y (mgr.34107) 5309 : cluster [DBG] pgmap v6588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:57 smithi115 bash[45495]: cluster 2022-04-22T20:00:56.621189+0000 mgr.y (mgr.34107) 5309 : cluster [DBG] pgmap v6588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:58.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:57 smithi115 bash[46934]: cluster 2022-04-22T20:00:56.621189+0000 mgr.y (mgr.34107) 5309 : cluster [DBG] pgmap v6588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:00:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:58 smithi121 bash[36902]: audit 2022-04-22T20:00:58.021553+0000 mon.a (mon.0) 8346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:59.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:58 smithi115 bash[46934]: audit 2022-04-22T20:00:58.021553+0000 mon.a (mon.0) 8346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:00:59.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:58 smithi115 bash[45495]: audit 2022-04-22T20:00:58.021553+0000 mon.a (mon.0) 8346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:00:59 smithi121 bash[36902]: cluster 2022-04-22T20:00:58.622366+0000 mgr.y (mgr.34107) 5310 : cluster [DBG] pgmap v6589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:00:59 smithi115 bash[45495]: cluster 2022-04-22T20:00:58.622366+0000 mgr.y (mgr.34107) 5310 : cluster [DBG] pgmap v6589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:00:59 smithi115 bash[46934]: cluster 2022-04-22T20:00:58.622366+0000 mgr.y (mgr.34107) 5310 : cluster [DBG] pgmap v6589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:00 smithi121 bash[36902]: audit 2022-04-22T20:01:00.256253+0000 mon.a (mon.0) 8347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:00 smithi115 bash[45495]: audit 2022-04-22T20:01:00.256253+0000 mon.a (mon.0) 8347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:00 smithi115 bash[46934]: audit 2022-04-22T20:01:00.256253+0000 mon.a (mon.0) 8347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:01 smithi121 bash[36902]: cluster 2022-04-22T20:01:00.623298+0000 mgr.y (mgr.34107) 5311 : cluster [DBG] pgmap v6590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:01 smithi121 bash[36902]: audit 2022-04-22T20:01:01.577879+0000 mon.a (mon.0) 8348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:01 smithi115 bash[45495]: cluster 2022-04-22T20:01:00.623298+0000 mgr.y (mgr.34107) 5311 : cluster [DBG] pgmap v6590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:01 smithi115 bash[45495]: audit 2022-04-22T20:01:01.577879+0000 mon.a (mon.0) 8348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:01 smithi115 bash[46934]: cluster 2022-04-22T20:01:00.623298+0000 mgr.y (mgr.34107) 5311 : cluster [DBG] pgmap v6590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:01 smithi115 bash[46934]: audit 2022-04-22T20:01:01.577879+0000 mon.a (mon.0) 8348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:02.215Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:03 smithi121 bash[36902]: cluster 2022-04-22T20:01:02.624364+0000 mgr.y (mgr.34107) 5312 : cluster [DBG] pgmap v6591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:03 smithi121 bash[36902]: audit 2022-04-22T20:01:03.024675+0000 mon.a (mon.0) 8349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:03 smithi115 bash[46934]: cluster 2022-04-22T20:01:02.624364+0000 mgr.y (mgr.34107) 5312 : cluster [DBG] pgmap v6591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:03 smithi115 bash[46934]: audit 2022-04-22T20:01:03.024675+0000 mon.a (mon.0) 8349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:03 smithi115 bash[45495]: cluster 2022-04-22T20:01:02.624364+0000 mgr.y (mgr.34107) 5312 : cluster [DBG] pgmap v6591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:03 smithi115 bash[45495]: audit 2022-04-22T20:01:03.024675+0000 mon.a (mon.0) 8349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:04.095Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:05 smithi121 bash[36902]: cluster 2022-04-22T20:01:04.625478+0000 mgr.y (mgr.34107) 5313 : cluster [DBG] pgmap v6592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:05 smithi121 bash[36902]: audit 2022-04-22T20:01:05.259798+0000 mon.a (mon.0) 8350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:05 smithi115 bash[45495]: cluster 2022-04-22T20:01:04.625478+0000 mgr.y (mgr.34107) 5313 : cluster [DBG] pgmap v6592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:05 smithi115 bash[45495]: audit 2022-04-22T20:01:05.259798+0000 mon.a (mon.0) 8350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:05 smithi115 bash[46934]: cluster 2022-04-22T20:01:04.625478+0000 mgr.y (mgr.34107) 5313 : cluster [DBG] pgmap v6592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:05 smithi115 bash[46934]: audit 2022-04-22T20:01:05.259798+0000 mon.a (mon.0) 8350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:06 smithi121 bash[36902]: audit 2022-04-22T20:01:06.580998+0000 mon.a (mon.0) 8351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:06 smithi115 bash[45495]: audit 2022-04-22T20:01:06.580998+0000 mon.a (mon.0) 8351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:06 smithi115 bash[46934]: audit 2022-04-22T20:01:06.580998+0000 mon.a (mon.0) 8351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:07 smithi121 bash[36902]: cluster 2022-04-22T20:01:06.626678+0000 mgr.y (mgr.34107) 5314 : cluster [DBG] pgmap v6593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:07 smithi115 bash[45495]: cluster 2022-04-22T20:01:06.626678+0000 mgr.y (mgr.34107) 5314 : cluster [DBG] pgmap v6593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:07 smithi115 bash[46934]: cluster 2022-04-22T20:01:06.626678+0000 mgr.y (mgr.34107) 5314 : cluster [DBG] pgmap v6593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:09.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:08 smithi121 bash[36902]: audit 2022-04-22T20:01:08.027785+0000 mon.a (mon.0) 8352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:08 smithi115 bash[45495]: audit 2022-04-22T20:01:08.027785+0000 mon.a (mon.0) 8352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:08 smithi115 bash[46934]: audit 2022-04-22T20:01:08.027785+0000 mon.a (mon.0) 8352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:09 smithi121 bash[36902]: cluster 2022-04-22T20:01:08.627896+0000 mgr.y (mgr.34107) 5315 : cluster [DBG] pgmap v6594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:10.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:09 smithi115 bash[45495]: cluster 2022-04-22T20:01:08.627896+0000 mgr.y (mgr.34107) 5315 : cluster [DBG] pgmap v6594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:10.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:09 smithi115 bash[46934]: cluster 2022-04-22T20:01:08.627896+0000 mgr.y (mgr.34107) 5315 : cluster [DBG] pgmap v6594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:10 smithi121 bash[36902]: audit 2022-04-22T20:01:10.263862+0000 mon.a (mon.0) 8353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:10 smithi115 bash[45495]: audit 2022-04-22T20:01:10.263862+0000 mon.a (mon.0) 8353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:10 smithi115 bash[46934]: audit 2022-04-22T20:01:10.263862+0000 mon.a (mon.0) 8353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:11 smithi121 bash[36902]: cluster 2022-04-22T20:01:10.628995+0000 mgr.y (mgr.34107) 5316 : cluster [DBG] pgmap v6595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:11 smithi121 bash[36902]: audit 2022-04-22T20:01:11.584142+0000 mon.a (mon.0) 8354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:11 smithi115 bash[45495]: cluster 2022-04-22T20:01:10.628995+0000 mgr.y (mgr.34107) 5316 : cluster [DBG] pgmap v6595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:11 smithi115 bash[45495]: audit 2022-04-22T20:01:11.584142+0000 mon.a (mon.0) 8354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:11 smithi115 bash[46934]: cluster 2022-04-22T20:01:10.628995+0000 mgr.y (mgr.34107) 5316 : cluster [DBG] pgmap v6595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:11 smithi115 bash[46934]: audit 2022-04-22T20:01:11.584142+0000 mon.a (mon.0) 8354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:12.214Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:13 smithi121 bash[36902]: cluster 2022-04-22T20:01:12.630111+0000 mgr.y (mgr.34107) 5317 : cluster [DBG] pgmap v6596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:13 smithi121 bash[36902]: audit 2022-04-22T20:01:13.031057+0000 mon.a (mon.0) 8355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:13 smithi115 bash[46934]: cluster 2022-04-22T20:01:12.630111+0000 mgr.y (mgr.34107) 5317 : cluster [DBG] pgmap v6596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:13 smithi115 bash[46934]: audit 2022-04-22T20:01:13.031057+0000 mon.a (mon.0) 8355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:13 smithi115 bash[45495]: cluster 2022-04-22T20:01:12.630111+0000 mgr.y (mgr.34107) 5317 : cluster [DBG] pgmap v6596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:13 smithi115 bash[45495]: audit 2022-04-22T20:01:13.031057+0000 mon.a (mon.0) 8355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:01:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:14.090Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:14.660 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:01:14.660 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:01:14.660 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 28s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:01:14.660 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 28s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 28s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 28s ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 28s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:01:14.661 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 28s ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 28s ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 28s ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 28s ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:01:14.662 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 28s ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:01:15.063 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:01:15.063 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:01:15.063 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:01:15.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:15.063 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:01:15.064 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:01:15.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:15.064 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:01:15.064 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:01:15.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:01:15.065 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:01:15.066 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:01:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:14.286997+0000 mgr.y (mgr.34107) 5318 : audit [DBG] from='client.40932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: cluster 2022-04-22T20:01:14.631141+0000 mgr.y (mgr.34107) 5319 : cluster [DBG] pgmap v6597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:14.632686+0000 mgr.y (mgr.34107) 5320 : audit [DBG] from='client.40938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:15.053755+0000 mon.a (mon.0) 8356 : audit [DBG] from='client.? 172.21.15.115:0/3149413127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:15.062947+0000 mon.a (mon.0) 8357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:15.063324+0000 mon.a (mon.0) 8358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:01:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:15 smithi121 bash[36902]: audit 2022-04-22T20:01:15.267383+0000 mon.a (mon.0) 8359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:14.286997+0000 mgr.y (mgr.34107) 5318 : audit [DBG] from='client.40932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: cluster 2022-04-22T20:01:14.631141+0000 mgr.y (mgr.34107) 5319 : cluster [DBG] pgmap v6597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:14.632686+0000 mgr.y (mgr.34107) 5320 : audit [DBG] from='client.40938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:15.053755+0000 mon.a (mon.0) 8356 : audit [DBG] from='client.? 172.21.15.115:0/3149413127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:15.062947+0000 mon.a (mon.0) 8357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:15.063324+0000 mon.a (mon.0) 8358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[45495]: audit 2022-04-22T20:01:15.267383+0000 mon.a (mon.0) 8359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:14.286997+0000 mgr.y (mgr.34107) 5318 : audit [DBG] from='client.40932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: cluster 2022-04-22T20:01:14.631141+0000 mgr.y (mgr.34107) 5319 : cluster [DBG] pgmap v6597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:14.632686+0000 mgr.y (mgr.34107) 5320 : audit [DBG] from='client.40938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:15.053755+0000 mon.a (mon.0) 8356 : audit [DBG] from='client.? 172.21.15.115:0/3149413127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:15.062947+0000 mon.a (mon.0) 8357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:01:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:15.063324+0000 mon.a (mon.0) 8358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:01:16.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:15 smithi115 bash[46934]: audit 2022-04-22T20:01:15.267383+0000 mon.a (mon.0) 8359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:17.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:16 smithi115 bash[46934]: audit 2022-04-22T20:01:16.587415+0000 mon.a (mon.0) 8360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:17.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:16 smithi115 bash[45495]: audit 2022-04-22T20:01:16.587415+0000 mon.a (mon.0) 8360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:16 smithi121 bash[36902]: audit 2022-04-22T20:01:16.587415+0000 mon.a (mon.0) 8360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:17 smithi115 bash[45495]: cluster 2022-04-22T20:01:16.632148+0000 mgr.y (mgr.34107) 5321 : cluster [DBG] pgmap v6598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:17 smithi115 bash[46934]: cluster 2022-04-22T20:01:16.632148+0000 mgr.y (mgr.34107) 5321 : cluster [DBG] pgmap v6598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:17 smithi121 bash[36902]: cluster 2022-04-22T20:01:16.632148+0000 mgr.y (mgr.34107) 5321 : cluster [DBG] pgmap v6598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:18 smithi115 bash[45495]: audit 2022-04-22T20:01:18.034235+0000 mon.a (mon.0) 8361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:18 smithi115 bash[46934]: audit 2022-04-22T20:01:18.034235+0000 mon.a (mon.0) 8361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:18 smithi121 bash[36902]: audit 2022-04-22T20:01:18.034235+0000 mon.a (mon.0) 8361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:19 smithi115 bash[45495]: cluster 2022-04-22T20:01:18.633313+0000 mgr.y (mgr.34107) 5322 : cluster [DBG] pgmap v6599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:19 smithi115 bash[46934]: cluster 2022-04-22T20:01:18.633313+0000 mgr.y (mgr.34107) 5322 : cluster [DBG] pgmap v6599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:19 smithi121 bash[36902]: cluster 2022-04-22T20:01:18.633313+0000 mgr.y (mgr.34107) 5322 : cluster [DBG] pgmap v6599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:20 smithi115 bash[45495]: audit 2022-04-22T20:01:20.271069+0000 mon.a (mon.0) 8362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:21.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:20 smithi115 bash[46934]: audit 2022-04-22T20:01:20.271069+0000 mon.a (mon.0) 8362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:20 smithi121 bash[36902]: audit 2022-04-22T20:01:20.271069+0000 mon.a (mon.0) 8362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:21 smithi115 bash[46934]: cluster 2022-04-22T20:01:20.634429+0000 mgr.y (mgr.34107) 5323 : cluster [DBG] pgmap v6600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:21 smithi115 bash[46934]: audit 2022-04-22T20:01:21.590652+0000 mon.a (mon.0) 8363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:21 smithi115 bash[45495]: cluster 2022-04-22T20:01:20.634429+0000 mgr.y (mgr.34107) 5323 : cluster [DBG] pgmap v6600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:21 smithi115 bash[45495]: audit 2022-04-22T20:01:21.590652+0000 mon.a (mon.0) 8363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:21 smithi121 bash[36902]: cluster 2022-04-22T20:01:20.634429+0000 mgr.y (mgr.34107) 5323 : cluster [DBG] pgmap v6600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:21 smithi121 bash[36902]: audit 2022-04-22T20:01:21.590652+0000 mon.a (mon.0) 8363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:22.216Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:23 smithi115 bash[45495]: cluster 2022-04-22T20:01:22.635652+0000 mgr.y (mgr.34107) 5324 : cluster [DBG] pgmap v6601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:23 smithi115 bash[45495]: audit 2022-04-22T20:01:23.037215+0000 mon.a (mon.0) 8364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:23 smithi115 bash[46934]: cluster 2022-04-22T20:01:22.635652+0000 mgr.y (mgr.34107) 5324 : cluster [DBG] pgmap v6601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:24.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:23 smithi115 bash[46934]: audit 2022-04-22T20:01:23.037215+0000 mon.a (mon.0) 8364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:23 smithi121 bash[36902]: cluster 2022-04-22T20:01:22.635652+0000 mgr.y (mgr.34107) 5324 : cluster [DBG] pgmap v6601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:23 smithi121 bash[36902]: audit 2022-04-22T20:01:23.037215+0000 mon.a (mon.0) 8364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:24.093Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:26.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:25 smithi121 bash[36902]: cluster 2022-04-22T20:01:24.636801+0000 mgr.y (mgr.34107) 5325 : cluster [DBG] pgmap v6602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:26.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:25 smithi121 bash[36902]: audit 2022-04-22T20:01:25.274572+0000 mon.a (mon.0) 8365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:25 smithi115 bash[45495]: cluster 2022-04-22T20:01:24.636801+0000 mgr.y (mgr.34107) 5325 : cluster [DBG] pgmap v6602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:25 smithi115 bash[45495]: audit 2022-04-22T20:01:25.274572+0000 mon.a (mon.0) 8365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:25 smithi115 bash[46934]: cluster 2022-04-22T20:01:24.636801+0000 mgr.y (mgr.34107) 5325 : cluster [DBG] pgmap v6602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:25 smithi115 bash[46934]: audit 2022-04-22T20:01:25.274572+0000 mon.a (mon.0) 8365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:26 smithi121 bash[36902]: audit 2022-04-22T20:01:26.593858+0000 mon.a (mon.0) 8366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:26 smithi115 bash[45495]: audit 2022-04-22T20:01:26.593858+0000 mon.a (mon.0) 8366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:26 smithi115 bash[46934]: audit 2022-04-22T20:01:26.593858+0000 mon.a (mon.0) 8366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:27 smithi121 bash[36902]: cluster 2022-04-22T20:01:26.637868+0000 mgr.y (mgr.34107) 5326 : cluster [DBG] pgmap v6603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:27 smithi115 bash[46934]: cluster 2022-04-22T20:01:26.637868+0000 mgr.y (mgr.34107) 5326 : cluster [DBG] pgmap v6603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:27 smithi115 bash[45495]: cluster 2022-04-22T20:01:26.637868+0000 mgr.y (mgr.34107) 5326 : cluster [DBG] pgmap v6603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:28 smithi121 bash[36902]: audit 2022-04-22T20:01:28.040262+0000 mon.a (mon.0) 8367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:28 smithi115 bash[46934]: audit 2022-04-22T20:01:28.040262+0000 mon.a (mon.0) 8367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:28 smithi115 bash[45495]: audit 2022-04-22T20:01:28.040262+0000 mon.a (mon.0) 8367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:29 smithi121 bash[36902]: cluster 2022-04-22T20:01:28.638953+0000 mgr.y (mgr.34107) 5327 : cluster [DBG] pgmap v6604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:29 smithi115 bash[45495]: cluster 2022-04-22T20:01:28.638953+0000 mgr.y (mgr.34107) 5327 : cluster [DBG] pgmap v6604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:29 smithi115 bash[46934]: cluster 2022-04-22T20:01:28.638953+0000 mgr.y (mgr.34107) 5327 : cluster [DBG] pgmap v6604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:30 smithi121 bash[36902]: audit 2022-04-22T20:01:30.278231+0000 mon.a (mon.0) 8368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:30 smithi115 bash[45495]: audit 2022-04-22T20:01:30.278231+0000 mon.a (mon.0) 8368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:30 smithi115 bash[46934]: audit 2022-04-22T20:01:30.278231+0000 mon.a (mon.0) 8368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:32.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:31 smithi115 bash[46934]: cluster 2022-04-22T20:01:30.640130+0000 mgr.y (mgr.34107) 5328 : cluster [DBG] pgmap v6605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:32.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:31 smithi115 bash[46934]: audit 2022-04-22T20:01:31.596911+0000 mon.a (mon.0) 8369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:32.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:31 smithi115 bash[45495]: cluster 2022-04-22T20:01:30.640130+0000 mgr.y (mgr.34107) 5328 : cluster [DBG] pgmap v6605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:32.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:31 smithi115 bash[45495]: audit 2022-04-22T20:01:31.596911+0000 mon.a (mon.0) 8369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:31 smithi121 bash[36902]: cluster 2022-04-22T20:01:30.640130+0000 mgr.y (mgr.34107) 5328 : cluster [DBG] pgmap v6605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:31 smithi121 bash[36902]: audit 2022-04-22T20:01:31.596911+0000 mon.a (mon.0) 8369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:32.217Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:34.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:33 smithi115 bash[45495]: cluster 2022-04-22T20:01:32.641274+0000 mgr.y (mgr.34107) 5329 : cluster [DBG] pgmap v6606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:34.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:33 smithi115 bash[45495]: audit 2022-04-22T20:01:33.043044+0000 mon.a (mon.0) 8370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:34.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:33 smithi115 bash[46934]: cluster 2022-04-22T20:01:32.641274+0000 mgr.y (mgr.34107) 5329 : cluster [DBG] pgmap v6606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:34.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:33 smithi115 bash[46934]: audit 2022-04-22T20:01:33.043044+0000 mon.a (mon.0) 8370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:33 smithi121 bash[36902]: cluster 2022-04-22T20:01:32.641274+0000 mgr.y (mgr.34107) 5329 : cluster [DBG] pgmap v6606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:34.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:33 smithi121 bash[36902]: audit 2022-04-22T20:01:33.043044+0000 mon.a (mon.0) 8370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:34.093Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:35 smithi121 bash[36902]: cluster 2022-04-22T20:01:34.642329+0000 mgr.y (mgr.34107) 5330 : cluster [DBG] pgmap v6607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:35 smithi121 bash[36902]: audit 2022-04-22T20:01:35.281979+0000 mon.a (mon.0) 8371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:35 smithi115 bash[45495]: cluster 2022-04-22T20:01:34.642329+0000 mgr.y (mgr.34107) 5330 : cluster [DBG] pgmap v6607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:35 smithi115 bash[45495]: audit 2022-04-22T20:01:35.281979+0000 mon.a (mon.0) 8371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:35 smithi115 bash[46934]: cluster 2022-04-22T20:01:34.642329+0000 mgr.y (mgr.34107) 5330 : cluster [DBG] pgmap v6607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:35 smithi115 bash[46934]: audit 2022-04-22T20:01:35.281979+0000 mon.a (mon.0) 8371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:36 smithi121 bash[36902]: audit 2022-04-22T20:01:36.599930+0000 mon.a (mon.0) 8372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:36 smithi115 bash[46934]: audit 2022-04-22T20:01:36.599930+0000 mon.a (mon.0) 8372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:36 smithi115 bash[45495]: audit 2022-04-22T20:01:36.599930+0000 mon.a (mon.0) 8372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:37 smithi121 bash[36902]: cluster 2022-04-22T20:01:36.643379+0000 mgr.y (mgr.34107) 5331 : cluster [DBG] pgmap v6608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:37 smithi115 bash[46934]: cluster 2022-04-22T20:01:36.643379+0000 mgr.y (mgr.34107) 5331 : cluster [DBG] pgmap v6608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:37 smithi115 bash[45495]: cluster 2022-04-22T20:01:36.643379+0000 mgr.y (mgr.34107) 5331 : cluster [DBG] pgmap v6608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:38 smithi121 bash[36902]: audit 2022-04-22T20:01:38.046201+0000 mon.a (mon.0) 8373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:38 smithi115 bash[45495]: audit 2022-04-22T20:01:38.046201+0000 mon.a (mon.0) 8373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:38 smithi115 bash[46934]: audit 2022-04-22T20:01:38.046201+0000 mon.a (mon.0) 8373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:39 smithi121 bash[36902]: cluster 2022-04-22T20:01:38.644477+0000 mgr.y (mgr.34107) 5332 : cluster [DBG] pgmap v6609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:39 smithi115 bash[46934]: cluster 2022-04-22T20:01:38.644477+0000 mgr.y (mgr.34107) 5332 : cluster [DBG] pgmap v6609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:39 smithi115 bash[45495]: cluster 2022-04-22T20:01:38.644477+0000 mgr.y (mgr.34107) 5332 : cluster [DBG] pgmap v6609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:40 smithi121 bash[36902]: audit 2022-04-22T20:01:40.285767+0000 mon.a (mon.0) 8374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:40 smithi115 bash[46934]: audit 2022-04-22T20:01:40.285767+0000 mon.a (mon.0) 8374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:40 smithi115 bash[45495]: audit 2022-04-22T20:01:40.285767+0000 mon.a (mon.0) 8374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:42.227 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:41 smithi115 bash[45495]: cluster 2022-04-22T20:01:40.645520+0000 mgr.y (mgr.34107) 5333 : cluster [DBG] pgmap v6610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:42.227 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:41 smithi115 bash[45495]: audit 2022-04-22T20:01:41.603066+0000 mon.a (mon.0) 8375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:42.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:41 smithi115 bash[46934]: cluster 2022-04-22T20:01:40.645520+0000 mgr.y (mgr.34107) 5333 : cluster [DBG] pgmap v6610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:42.228 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:41 smithi115 bash[46934]: audit 2022-04-22T20:01:41.603066+0000 mon.a (mon.0) 8375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:41 smithi121 bash[36902]: cluster 2022-04-22T20:01:40.645520+0000 mgr.y (mgr.34107) 5333 : cluster [DBG] pgmap v6610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:41 smithi121 bash[36902]: audit 2022-04-22T20:01:41.603066+0000 mon.a (mon.0) 8375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:42.218Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:43 smithi121 bash[36902]: cluster 2022-04-22T20:01:42.646583+0000 mgr.y (mgr.34107) 5334 : cluster [DBG] pgmap v6611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:43 smithi121 bash[36902]: audit 2022-04-22T20:01:43.049218+0000 mon.a (mon.0) 8376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:44.094Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:43 smithi115 bash[46934]: cluster 2022-04-22T20:01:42.646583+0000 mgr.y (mgr.34107) 5334 : cluster [DBG] pgmap v6611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:43 smithi115 bash[46934]: audit 2022-04-22T20:01:43.049218+0000 mon.a (mon.0) 8376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:43 smithi115 bash[45495]: cluster 2022-04-22T20:01:42.646583+0000 mgr.y (mgr.34107) 5334 : cluster [DBG] pgmap v6611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:43 smithi115 bash[45495]: audit 2022-04-22T20:01:43.049218+0000 mon.a (mon.0) 8376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:45.439 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:01:45.779 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:01:45.779 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 10m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:01:45.779 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 60s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 10m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 60s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 60s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 10m ago 2h 482M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 10m ago 2h 302M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 60s ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 10m ago 2h 192M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 10m ago 2h 11.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 60s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:01:45.780 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 10m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 10m ago 2h 782M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 10m ago 2h 529M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 10m ago 2h 485M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 60s ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 60s ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 60s ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 60s ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:01:45.781 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 60s ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:01:46.204 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:01:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:01:46.206 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:01:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:45 smithi121 bash[36902]: cluster 2022-04-22T20:01:44.647661+0000 mgr.y (mgr.34107) 5335 : cluster [DBG] pgmap v6612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:45 smithi121 bash[36902]: audit 2022-04-22T20:01:45.289444+0000 mon.a (mon.0) 8377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:45 smithi115 bash[45495]: cluster 2022-04-22T20:01:44.647661+0000 mgr.y (mgr.34107) 5335 : cluster [DBG] pgmap v6612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:45 smithi115 bash[45495]: audit 2022-04-22T20:01:45.289444+0000 mon.a (mon.0) 8377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:45 smithi115 bash[46934]: cluster 2022-04-22T20:01:44.647661+0000 mgr.y (mgr.34107) 5335 : cluster [DBG] pgmap v6612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:45 smithi115 bash[46934]: audit 2022-04-22T20:01:45.289444+0000 mon.a (mon.0) 8377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:45.412770+0000 mgr.y (mgr.34107) 5336 : audit [DBG] from='client.50884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:45.752266+0000 mgr.y (mgr.34107) 5337 : audit [DBG] from='client.40956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:45.999669+0000 mon.a (mon.0) 8378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.001019+0000 mon.a (mon.0) 8379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.001732+0000 mon.a (mon.0) 8380 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.195763+0000 mon.a (mon.0) 8381 : audit [DBG] from='client.? 172.21.15.115:0/3035275321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.306216+0000 mon.a (mon.0) 8382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.341517+0000 mon.a (mon.0) 8383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:46 smithi121 bash[36902]: audit 2022-04-22T20:01:46.606146+0000 mon.a (mon.0) 8384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:45.412770+0000 mgr.y (mgr.34107) 5336 : audit [DBG] from='client.50884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:45.752266+0000 mgr.y (mgr.34107) 5337 : audit [DBG] from='client.40956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:45.999669+0000 mon.a (mon.0) 8378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:01:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.001019+0000 mon.a (mon.0) 8379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.001732+0000 mon.a (mon.0) 8380 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.195763+0000 mon.a (mon.0) 8381 : audit [DBG] from='client.? 172.21.15.115:0/3035275321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.306216+0000 mon.a (mon.0) 8382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.341517+0000 mon.a (mon.0) 8383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[45495]: audit 2022-04-22T20:01:46.606146+0000 mon.a (mon.0) 8384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:45.412770+0000 mgr.y (mgr.34107) 5336 : audit [DBG] from='client.50884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:45.752266+0000 mgr.y (mgr.34107) 5337 : audit [DBG] from='client.40956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:45.999669+0000 mon.a (mon.0) 8378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.001019+0000 mon.a (mon.0) 8379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.001732+0000 mon.a (mon.0) 8380 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.195763+0000 mon.a (mon.0) 8381 : audit [DBG] from='client.? 172.21.15.115:0/3035275321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.306216+0000 mon.a (mon.0) 8382 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.341517+0000 mon.a (mon.0) 8383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:46 smithi115 bash[46934]: audit 2022-04-22T20:01:46.606146+0000 mon.a (mon.0) 8384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:47 smithi121 bash[36902]: cluster 2022-04-22T20:01:46.648645+0000 mgr.y (mgr.34107) 5338 : cluster [DBG] pgmap v6613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:47 smithi115 bash[45495]: cluster 2022-04-22T20:01:46.648645+0000 mgr.y (mgr.34107) 5338 : cluster [DBG] pgmap v6613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:47 smithi115 bash[46934]: cluster 2022-04-22T20:01:46.648645+0000 mgr.y (mgr.34107) 5338 : cluster [DBG] pgmap v6613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:48 smithi121 bash[36902]: audit 2022-04-22T20:01:48.052399+0000 mon.a (mon.0) 8385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:48 smithi115 bash[45495]: audit 2022-04-22T20:01:48.052399+0000 mon.a (mon.0) 8385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:48 smithi115 bash[46934]: audit 2022-04-22T20:01:48.052399+0000 mon.a (mon.0) 8385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:49 smithi121 bash[36902]: cluster 2022-04-22T20:01:48.649718+0000 mgr.y (mgr.34107) 5339 : cluster [DBG] pgmap v6614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:50.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:49 smithi115 bash[45495]: cluster 2022-04-22T20:01:48.649718+0000 mgr.y (mgr.34107) 5339 : cluster [DBG] pgmap v6614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:49 smithi115 bash[46934]: cluster 2022-04-22T20:01:48.649718+0000 mgr.y (mgr.34107) 5339 : cluster [DBG] pgmap v6614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:50 smithi121 bash[36902]: audit 2022-04-22T20:01:50.292904+0000 mon.a (mon.0) 8386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:50 smithi121 bash[36902]: audit 2022-04-22T20:01:50.497072+0000 mon.a (mon.0) 8387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:50 smithi121 bash[36902]: audit 2022-04-22T20:01:50.829768+0000 mon.a (mon.0) 8388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[45495]: audit 2022-04-22T20:01:50.292904+0000 mon.a (mon.0) 8386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[45495]: audit 2022-04-22T20:01:50.497072+0000 mon.a (mon.0) 8387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[45495]: audit 2022-04-22T20:01:50.829768+0000 mon.a (mon.0) 8388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[46934]: audit 2022-04-22T20:01:50.292904+0000 mon.a (mon.0) 8386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[46934]: audit 2022-04-22T20:01:50.497072+0000 mon.a (mon.0) 8387 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:50 smithi115 bash[46934]: audit 2022-04-22T20:01:50.829768+0000 mon.a (mon.0) 8388 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:01:52.227 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:51 smithi115 bash[45495]: cluster 2022-04-22T20:01:50.650694+0000 mgr.y (mgr.34107) 5340 : cluster [DBG] pgmap v6615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:52.227 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:51 smithi115 bash[45495]: audit 2022-04-22T20:01:51.609033+0000 mon.a (mon.0) 8389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:52.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:51 smithi115 bash[46934]: cluster 2022-04-22T20:01:50.650694+0000 mgr.y (mgr.34107) 5340 : cluster [DBG] pgmap v6615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:52.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:51 smithi115 bash[46934]: audit 2022-04-22T20:01:51.609033+0000 mon.a (mon.0) 8389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:51 smithi121 bash[36902]: cluster 2022-04-22T20:01:50.650694+0000 mgr.y (mgr.34107) 5340 : cluster [DBG] pgmap v6615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:51 smithi121 bash[36902]: audit 2022-04-22T20:01:51.609033+0000 mon.a (mon.0) 8389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:52.218Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:53 smithi121 bash[36902]: cluster 2022-04-22T20:01:52.651855+0000 mgr.y (mgr.34107) 5341 : cluster [DBG] pgmap v6616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:53 smithi121 bash[36902]: audit 2022-04-22T20:01:53.055339+0000 mon.a (mon.0) 8390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:01:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:01:54.097Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:01:54.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:53 smithi115 bash[45495]: cluster 2022-04-22T20:01:52.651855+0000 mgr.y (mgr.34107) 5341 : cluster [DBG] pgmap v6616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:54.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:53 smithi115 bash[45495]: audit 2022-04-22T20:01:53.055339+0000 mon.a (mon.0) 8390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:53 smithi115 bash[46934]: cluster 2022-04-22T20:01:52.651855+0000 mgr.y (mgr.34107) 5341 : cluster [DBG] pgmap v6616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:54.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:53 smithi115 bash[46934]: audit 2022-04-22T20:01:53.055339+0000 mon.a (mon.0) 8390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:54 smithi121 bash[36902]: cluster 2022-04-22T20:01:54.652971+0000 mgr.y (mgr.34107) 5342 : cluster [DBG] pgmap v6617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:54 smithi115 bash[45495]: cluster 2022-04-22T20:01:54.652971+0000 mgr.y (mgr.34107) 5342 : cluster [DBG] pgmap v6617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:54 smithi115 bash[46934]: cluster 2022-04-22T20:01:54.652971+0000 mgr.y (mgr.34107) 5342 : cluster [DBG] pgmap v6617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:55 smithi121 bash[36902]: audit 2022-04-22T20:01:55.296634+0000 mon.a (mon.0) 8391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:55 smithi115 bash[45495]: audit 2022-04-22T20:01:55.296634+0000 mon.a (mon.0) 8391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:55 smithi115 bash[46934]: audit 2022-04-22T20:01:55.296634+0000 mon.a (mon.0) 8391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:01:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:56 smithi121 bash[36902]: audit 2022-04-22T20:01:56.612212+0000 mon.a (mon.0) 8392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:56 smithi121 bash[36902]: cluster 2022-04-22T20:01:56.654038+0000 mgr.y (mgr.34107) 5343 : cluster [DBG] pgmap v6618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:56 smithi115 bash[45495]: audit 2022-04-22T20:01:56.612212+0000 mon.a (mon.0) 8392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:56 smithi115 bash[45495]: cluster 2022-04-22T20:01:56.654038+0000 mgr.y (mgr.34107) 5343 : cluster [DBG] pgmap v6618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:56 smithi115 bash[46934]: audit 2022-04-22T20:01:56.612212+0000 mon.a (mon.0) 8392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:01:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:56 smithi115 bash[46934]: cluster 2022-04-22T20:01:56.654038+0000 mgr.y (mgr.34107) 5343 : cluster [DBG] pgmap v6618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:58 smithi121 bash[36902]: audit 2022-04-22T20:01:58.058485+0000 mon.a (mon.0) 8393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:58 smithi115 bash[45495]: audit 2022-04-22T20:01:58.058485+0000 mon.a (mon.0) 8393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:58 smithi115 bash[46934]: audit 2022-04-22T20:01:58.058485+0000 mon.a (mon.0) 8393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:01:59.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:01:59 smithi115 bash[45495]: cluster 2022-04-22T20:01:58.655190+0000 mgr.y (mgr.34107) 5344 : cluster [DBG] pgmap v6619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:01:59.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:01:59 smithi115 bash[46934]: cluster 2022-04-22T20:01:58.655190+0000 mgr.y (mgr.34107) 5344 : cluster [DBG] pgmap v6619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:01:59 smithi121 bash[36902]: cluster 2022-04-22T20:01:58.655190+0000 mgr.y (mgr.34107) 5344 : cluster [DBG] pgmap v6619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:00.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:00 smithi115 bash[46934]: audit 2022-04-22T20:02:00.300270+0000 mon.a (mon.0) 8394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:00.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:00 smithi115 bash[45495]: audit 2022-04-22T20:02:00.300270+0000 mon.a (mon.0) 8394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:00 smithi121 bash[36902]: audit 2022-04-22T20:02:00.300270+0000 mon.a (mon.0) 8394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:01 smithi121 bash[36902]: cluster 2022-04-22T20:02:00.656301+0000 mgr.y (mgr.34107) 5345 : cluster [DBG] pgmap v6620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:01 smithi115 bash[46934]: cluster 2022-04-22T20:02:00.656301+0000 mgr.y (mgr.34107) 5345 : cluster [DBG] pgmap v6620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:01 smithi115 bash[45495]: cluster 2022-04-22T20:02:00.656301+0000 mgr.y (mgr.34107) 5345 : cluster [DBG] pgmap v6620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:02.220Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:02 smithi121 bash[36902]: audit 2022-04-22T20:02:01.615330+0000 mon.a (mon.0) 8395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:02 smithi115 bash[45495]: audit 2022-04-22T20:02:01.615330+0000 mon.a (mon.0) 8395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:02 smithi115 bash[46934]: audit 2022-04-22T20:02:01.615330+0000 mon.a (mon.0) 8395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:03 smithi121 bash[36902]: cluster 2022-04-22T20:02:02.657423+0000 mgr.y (mgr.34107) 5346 : cluster [DBG] pgmap v6621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:03 smithi121 bash[36902]: audit 2022-04-22T20:02:03.061770+0000 mon.a (mon.0) 8396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:03 smithi115 bash[45495]: cluster 2022-04-22T20:02:02.657423+0000 mgr.y (mgr.34107) 5346 : cluster [DBG] pgmap v6621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:03 smithi115 bash[45495]: audit 2022-04-22T20:02:03.061770+0000 mon.a (mon.0) 8396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:03 smithi115 bash[46934]: cluster 2022-04-22T20:02:02.657423+0000 mgr.y (mgr.34107) 5346 : cluster [DBG] pgmap v6621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:03 smithi115 bash[46934]: audit 2022-04-22T20:02:03.061770+0000 mon.a (mon.0) 8396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:04.096Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:05 smithi121 bash[36902]: cluster 2022-04-22T20:02:04.658520+0000 mgr.y (mgr.34107) 5347 : cluster [DBG] pgmap v6622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:05 smithi121 bash[36902]: audit 2022-04-22T20:02:05.304157+0000 mon.a (mon.0) 8397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:05 smithi115 bash[45495]: cluster 2022-04-22T20:02:04.658520+0000 mgr.y (mgr.34107) 5347 : cluster [DBG] pgmap v6622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:05 smithi115 bash[45495]: audit 2022-04-22T20:02:05.304157+0000 mon.a (mon.0) 8397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:05 smithi115 bash[46934]: cluster 2022-04-22T20:02:04.658520+0000 mgr.y (mgr.34107) 5347 : cluster [DBG] pgmap v6622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:05 smithi115 bash[46934]: audit 2022-04-22T20:02:05.304157+0000 mon.a (mon.0) 8397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:07.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:06 smithi121 bash[36902]: audit 2022-04-22T20:02:06.619276+0000 mon.a (mon.0) 8398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:07.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:06 smithi115 bash[45495]: audit 2022-04-22T20:02:06.619276+0000 mon.a (mon.0) 8398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:06 smithi115 bash[46934]: audit 2022-04-22T20:02:06.619276+0000 mon.a (mon.0) 8398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:07 smithi121 bash[36902]: cluster 2022-04-22T20:02:06.659597+0000 mgr.y (mgr.34107) 5348 : cluster [DBG] pgmap v6623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:08.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:07 smithi115 bash[45495]: cluster 2022-04-22T20:02:06.659597+0000 mgr.y (mgr.34107) 5348 : cluster [DBG] pgmap v6623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:08.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:07 smithi115 bash[46934]: cluster 2022-04-22T20:02:06.659597+0000 mgr.y (mgr.34107) 5348 : cluster [DBG] pgmap v6623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:08 smithi121 bash[36902]: audit 2022-04-22T20:02:08.064693+0000 mon.a (mon.0) 8399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:09.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:08 smithi115 bash[45495]: audit 2022-04-22T20:02:08.064693+0000 mon.a (mon.0) 8399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:08 smithi115 bash[46934]: audit 2022-04-22T20:02:08.064693+0000 mon.a (mon.0) 8399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:09 smithi121 bash[36902]: cluster 2022-04-22T20:02:08.660632+0000 mgr.y (mgr.34107) 5349 : cluster [DBG] pgmap v6624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:09 smithi115 bash[45495]: cluster 2022-04-22T20:02:08.660632+0000 mgr.y (mgr.34107) 5349 : cluster [DBG] pgmap v6624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:09 smithi115 bash[46934]: cluster 2022-04-22T20:02:08.660632+0000 mgr.y (mgr.34107) 5349 : cluster [DBG] pgmap v6624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:10 smithi121 bash[36902]: audit 2022-04-22T20:02:10.307942+0000 mon.a (mon.0) 8400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:10 smithi115 bash[45495]: audit 2022-04-22T20:02:10.307942+0000 mon.a (mon.0) 8400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:10 smithi115 bash[46934]: audit 2022-04-22T20:02:10.307942+0000 mon.a (mon.0) 8400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:11 smithi121 bash[36902]: cluster 2022-04-22T20:02:10.661691+0000 mgr.y (mgr.34107) 5350 : cluster [DBG] pgmap v6625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:11 smithi121 bash[36902]: audit 2022-04-22T20:02:11.622475+0000 mon.a (mon.0) 8401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:11 smithi115 bash[46934]: cluster 2022-04-22T20:02:10.661691+0000 mgr.y (mgr.34107) 5350 : cluster [DBG] pgmap v6625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:12.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:11 smithi115 bash[46934]: audit 2022-04-22T20:02:11.622475+0000 mon.a (mon.0) 8401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:11 smithi115 bash[45495]: cluster 2022-04-22T20:02:10.661691+0000 mgr.y (mgr.34107) 5350 : cluster [DBG] pgmap v6625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:12.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:11 smithi115 bash[45495]: audit 2022-04-22T20:02:11.622475+0000 mon.a (mon.0) 8401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:12.220Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:13 smithi121 bash[36902]: cluster 2022-04-22T20:02:12.662803+0000 mgr.y (mgr.34107) 5351 : cluster [DBG] pgmap v6626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:14.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:13 smithi121 bash[36902]: audit 2022-04-22T20:02:13.067835+0000 mon.a (mon.0) 8402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:13 smithi115 bash[45495]: cluster 2022-04-22T20:02:12.662803+0000 mgr.y (mgr.34107) 5351 : cluster [DBG] pgmap v6626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:13 smithi115 bash[45495]: audit 2022-04-22T20:02:13.067835+0000 mon.a (mon.0) 8402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:13 smithi115 bash[46934]: cluster 2022-04-22T20:02:12.662803+0000 mgr.y (mgr.34107) 5351 : cluster [DBG] pgmap v6626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:13 smithi115 bash[46934]: audit 2022-04-22T20:02:13.067835+0000 mon.a (mon.0) 8402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:14.097Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[45495]: cluster 2022-04-22T20:02:14.663934+0000 mgr.y (mgr.34107) 5352 : cluster [DBG] pgmap v6627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[45495]: audit 2022-04-22T20:02:15.073899+0000 mon.a (mon.0) 8403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:02:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[45495]: audit 2022-04-22T20:02:15.074613+0000 mon.a (mon.0) 8404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:02:16.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[45495]: audit 2022-04-22T20:02:15.311372+0000 mon.a (mon.0) 8405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[46934]: cluster 2022-04-22T20:02:14.663934+0000 mgr.y (mgr.34107) 5352 : cluster [DBG] pgmap v6627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[46934]: audit 2022-04-22T20:02:15.073899+0000 mon.a (mon.0) 8403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:02:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[46934]: audit 2022-04-22T20:02:15.074613+0000 mon.a (mon.0) 8404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:02:16.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:15 smithi115 bash[46934]: audit 2022-04-22T20:02:15.311372+0000 mon.a (mon.0) 8405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:15 smithi121 bash[36902]: cluster 2022-04-22T20:02:14.663934+0000 mgr.y (mgr.34107) 5352 : cluster [DBG] pgmap v6627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:15 smithi121 bash[36902]: audit 2022-04-22T20:02:15.073899+0000 mon.a (mon.0) 8403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:02:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:15 smithi121 bash[36902]: audit 2022-04-22T20:02:15.074613+0000 mon.a (mon.0) 8404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:02:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:15 smithi121 bash[36902]: audit 2022-04-22T20:02:15.311372+0000 mon.a (mon.0) 8405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:16.582 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:02:16.935 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 26s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 91s ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 26s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 91s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 91s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 26s ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 26s ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 91s ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 26s ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:02:16.936 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 26s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 91s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 26s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 26s ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 26s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 26s ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 91s ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 91s ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 91s ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:02:16.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 91s ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:02:16.938 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 91s ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:02:17.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:16 smithi115 bash[45495]: audit 2022-04-22T20:02:16.625540+0000 mon.a (mon.0) 8406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:17.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:16 smithi115 bash[46934]: audit 2022-04-22T20:02:16.625540+0000 mon.a (mon.0) 8406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:16 smithi121 bash[36902]: audit 2022-04-22T20:02:16.625540+0000 mon.a (mon.0) 8406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:17.342 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:02:17.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:02:17.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:02:17.345 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:02:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[45495]: audit 2022-04-22T20:02:16.558268+0000 mgr.y (mgr.34107) 5353 : audit [DBG] from='client.40968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:18.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[45495]: cluster 2022-04-22T20:02:16.665036+0000 mgr.y (mgr.34107) 5354 : cluster [DBG] pgmap v6628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:18.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[45495]: audit 2022-04-22T20:02:17.333730+0000 mon.c (mon.1) 222 : audit [DBG] from='client.? 172.21.15.115:0/1721036391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:18.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[46934]: audit 2022-04-22T20:02:16.558268+0000 mgr.y (mgr.34107) 5353 : audit [DBG] from='client.40968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:18.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[46934]: cluster 2022-04-22T20:02:16.665036+0000 mgr.y (mgr.34107) 5354 : cluster [DBG] pgmap v6628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:18.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:17 smithi115 bash[46934]: audit 2022-04-22T20:02:17.333730+0000 mon.c (mon.1) 222 : audit [DBG] from='client.? 172.21.15.115:0/1721036391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:17 smithi121 bash[36902]: audit 2022-04-22T20:02:16.558268+0000 mgr.y (mgr.34107) 5353 : audit [DBG] from='client.40968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:17 smithi121 bash[36902]: cluster 2022-04-22T20:02:16.665036+0000 mgr.y (mgr.34107) 5354 : cluster [DBG] pgmap v6628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:17 smithi121 bash[36902]: audit 2022-04-22T20:02:17.333730+0000 mon.c (mon.1) 222 : audit [DBG] from='client.? 172.21.15.115:0/1721036391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:18 smithi115 bash[45495]: audit 2022-04-22T20:02:16.907616+0000 mgr.y (mgr.34107) 5355 : audit [DBG] from='client.40974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:19.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:18 smithi115 bash[45495]: audit 2022-04-22T20:02:18.070994+0000 mon.a (mon.0) 8407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:18 smithi115 bash[46934]: audit 2022-04-22T20:02:16.907616+0000 mgr.y (mgr.34107) 5355 : audit [DBG] from='client.40974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:18 smithi115 bash[46934]: audit 2022-04-22T20:02:18.070994+0000 mon.a (mon.0) 8407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:18 smithi121 bash[36902]: audit 2022-04-22T20:02:16.907616+0000 mgr.y (mgr.34107) 5355 : audit [DBG] from='client.40974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:18 smithi121 bash[36902]: audit 2022-04-22T20:02:18.070994+0000 mon.a (mon.0) 8407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:19 smithi115 bash[45495]: cluster 2022-04-22T20:02:18.666197+0000 mgr.y (mgr.34107) 5356 : cluster [DBG] pgmap v6629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:19 smithi115 bash[46934]: cluster 2022-04-22T20:02:18.666197+0000 mgr.y (mgr.34107) 5356 : cluster [DBG] pgmap v6629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:19 smithi121 bash[36902]: cluster 2022-04-22T20:02:18.666197+0000 mgr.y (mgr.34107) 5356 : cluster [DBG] pgmap v6629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:20 smithi121 bash[36902]: audit 2022-04-22T20:02:20.314906+0000 mon.a (mon.0) 8408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:20 smithi115 bash[45495]: audit 2022-04-22T20:02:20.314906+0000 mon.a (mon.0) 8408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:20 smithi115 bash[46934]: audit 2022-04-22T20:02:20.314906+0000 mon.a (mon.0) 8408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:22.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:21 smithi115 bash[46934]: cluster 2022-04-22T20:02:20.667304+0000 mgr.y (mgr.34107) 5357 : cluster [DBG] pgmap v6630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:22.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:21 smithi115 bash[46934]: audit 2022-04-22T20:02:21.628539+0000 mon.a (mon.0) 8409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:22.229 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:21 smithi115 bash[45495]: cluster 2022-04-22T20:02:20.667304+0000 mgr.y (mgr.34107) 5357 : cluster [DBG] pgmap v6630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:22.229 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:21 smithi115 bash[45495]: audit 2022-04-22T20:02:21.628539+0000 mon.a (mon.0) 8409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:21 smithi121 bash[36902]: cluster 2022-04-22T20:02:20.667304+0000 mgr.y (mgr.34107) 5357 : cluster [DBG] pgmap v6630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:21 smithi121 bash[36902]: audit 2022-04-22T20:02:21.628539+0000 mon.a (mon.0) 8409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:22.220Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:24.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:23 smithi115 bash[45495]: cluster 2022-04-22T20:02:22.668407+0000 mgr.y (mgr.34107) 5358 : cluster [DBG] pgmap v6631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:24.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:23 smithi115 bash[45495]: audit 2022-04-22T20:02:23.074055+0000 mon.a (mon.0) 8410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:23 smithi115 bash[46934]: cluster 2022-04-22T20:02:22.668407+0000 mgr.y (mgr.34107) 5358 : cluster [DBG] pgmap v6631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:23 smithi115 bash[46934]: audit 2022-04-22T20:02:23.074055+0000 mon.a (mon.0) 8410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:23 smithi121 bash[36902]: cluster 2022-04-22T20:02:22.668407+0000 mgr.y (mgr.34107) 5358 : cluster [DBG] pgmap v6631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:23 smithi121 bash[36902]: audit 2022-04-22T20:02:23.074055+0000 mon.a (mon.0) 8410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:24.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:24.098Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:25 smithi121 bash[36902]: cluster 2022-04-22T20:02:24.669446+0000 mgr.y (mgr.34107) 5359 : cluster [DBG] pgmap v6632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:25 smithi121 bash[36902]: audit 2022-04-22T20:02:25.318704+0000 mon.a (mon.0) 8411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:25 smithi115 bash[45495]: cluster 2022-04-22T20:02:24.669446+0000 mgr.y (mgr.34107) 5359 : cluster [DBG] pgmap v6632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:25 smithi115 bash[45495]: audit 2022-04-22T20:02:25.318704+0000 mon.a (mon.0) 8411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:25 smithi115 bash[46934]: cluster 2022-04-22T20:02:24.669446+0000 mgr.y (mgr.34107) 5359 : cluster [DBG] pgmap v6632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:25 smithi115 bash[46934]: audit 2022-04-22T20:02:25.318704+0000 mon.a (mon.0) 8411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:26 smithi121 bash[36902]: audit 2022-04-22T20:02:26.631555+0000 mon.a (mon.0) 8412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:26 smithi115 bash[45495]: audit 2022-04-22T20:02:26.631555+0000 mon.a (mon.0) 8412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:26 smithi115 bash[46934]: audit 2022-04-22T20:02:26.631555+0000 mon.a (mon.0) 8412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:27 smithi121 bash[36902]: cluster 2022-04-22T20:02:26.670534+0000 mgr.y (mgr.34107) 5360 : cluster [DBG] pgmap v6633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:27 smithi115 bash[45495]: cluster 2022-04-22T20:02:26.670534+0000 mgr.y (mgr.34107) 5360 : cluster [DBG] pgmap v6633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:27 smithi115 bash[46934]: cluster 2022-04-22T20:02:26.670534+0000 mgr.y (mgr.34107) 5360 : cluster [DBG] pgmap v6633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:28 smithi121 bash[36902]: audit 2022-04-22T20:02:28.077042+0000 mon.a (mon.0) 8413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:28 smithi115 bash[45495]: audit 2022-04-22T20:02:28.077042+0000 mon.a (mon.0) 8413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:28 smithi115 bash[46934]: audit 2022-04-22T20:02:28.077042+0000 mon.a (mon.0) 8413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:29 smithi121 bash[36902]: cluster 2022-04-22T20:02:28.671882+0000 mgr.y (mgr.34107) 5361 : cluster [DBG] pgmap v6634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:29 smithi115 bash[45495]: cluster 2022-04-22T20:02:28.671882+0000 mgr.y (mgr.34107) 5361 : cluster [DBG] pgmap v6634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:29 smithi115 bash[46934]: cluster 2022-04-22T20:02:28.671882+0000 mgr.y (mgr.34107) 5361 : cluster [DBG] pgmap v6634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:30 smithi121 bash[36902]: audit 2022-04-22T20:02:30.322508+0000 mon.a (mon.0) 8414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:30 smithi115 bash[45495]: audit 2022-04-22T20:02:30.322508+0000 mon.a (mon.0) 8414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:30 smithi115 bash[46934]: audit 2022-04-22T20:02:30.322508+0000 mon.a (mon.0) 8414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:32.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:31 smithi115 bash[45495]: cluster 2022-04-22T20:02:30.673077+0000 mgr.y (mgr.34107) 5362 : cluster [DBG] pgmap v6635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:32.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:31 smithi115 bash[45495]: audit 2022-04-22T20:02:31.635007+0000 mon.a (mon.0) 8415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:32.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:31 smithi115 bash[46934]: cluster 2022-04-22T20:02:30.673077+0000 mgr.y (mgr.34107) 5362 : cluster [DBG] pgmap v6635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:32.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:31 smithi115 bash[46934]: audit 2022-04-22T20:02:31.635007+0000 mon.a (mon.0) 8415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:31 smithi121 bash[36902]: cluster 2022-04-22T20:02:30.673077+0000 mgr.y (mgr.34107) 5362 : cluster [DBG] pgmap v6635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:31 smithi121 bash[36902]: audit 2022-04-22T20:02:31.635007+0000 mon.a (mon.0) 8415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:32.220Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:34.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:33 smithi115 bash[45495]: cluster 2022-04-22T20:02:32.674323+0000 mgr.y (mgr.34107) 5363 : cluster [DBG] pgmap v6636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:34.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:33 smithi115 bash[45495]: audit 2022-04-22T20:02:33.080146+0000 mon.a (mon.0) 8416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:33 smithi115 bash[46934]: cluster 2022-04-22T20:02:32.674323+0000 mgr.y (mgr.34107) 5363 : cluster [DBG] pgmap v6636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:33 smithi115 bash[46934]: audit 2022-04-22T20:02:33.080146+0000 mon.a (mon.0) 8416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:33 smithi121 bash[36902]: cluster 2022-04-22T20:02:32.674323+0000 mgr.y (mgr.34107) 5363 : cluster [DBG] pgmap v6636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:33 smithi121 bash[36902]: audit 2022-04-22T20:02:33.080146+0000 mon.a (mon.0) 8416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:34.099Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:35 smithi121 bash[36902]: cluster 2022-04-22T20:02:34.675494+0000 mgr.y (mgr.34107) 5364 : cluster [DBG] pgmap v6637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:35 smithi121 bash[36902]: audit 2022-04-22T20:02:35.326446+0000 mon.a (mon.0) 8417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:35 smithi115 bash[45495]: cluster 2022-04-22T20:02:34.675494+0000 mgr.y (mgr.34107) 5364 : cluster [DBG] pgmap v6637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:35 smithi115 bash[45495]: audit 2022-04-22T20:02:35.326446+0000 mon.a (mon.0) 8417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:35 smithi115 bash[46934]: cluster 2022-04-22T20:02:34.675494+0000 mgr.y (mgr.34107) 5364 : cluster [DBG] pgmap v6637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:35 smithi115 bash[46934]: audit 2022-04-22T20:02:35.326446+0000 mon.a (mon.0) 8417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:36 smithi121 bash[36902]: audit 2022-04-22T20:02:36.638121+0000 mon.a (mon.0) 8418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:36 smithi115 bash[45495]: audit 2022-04-22T20:02:36.638121+0000 mon.a (mon.0) 8418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:36 smithi115 bash[46934]: audit 2022-04-22T20:02:36.638121+0000 mon.a (mon.0) 8418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:37 smithi121 bash[36902]: cluster 2022-04-22T20:02:36.676513+0000 mgr.y (mgr.34107) 5365 : cluster [DBG] pgmap v6638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:37 smithi115 bash[45495]: cluster 2022-04-22T20:02:36.676513+0000 mgr.y (mgr.34107) 5365 : cluster [DBG] pgmap v6638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:38.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:37 smithi115 bash[46934]: cluster 2022-04-22T20:02:36.676513+0000 mgr.y (mgr.34107) 5365 : cluster [DBG] pgmap v6638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:38 smithi121 bash[36902]: audit 2022-04-22T20:02:38.084168+0000 mon.a (mon.0) 8419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:38 smithi115 bash[45495]: audit 2022-04-22T20:02:38.084168+0000 mon.a (mon.0) 8419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:39.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:38 smithi115 bash[46934]: audit 2022-04-22T20:02:38.084168+0000 mon.a (mon.0) 8419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:39 smithi121 bash[36902]: cluster 2022-04-22T20:02:38.677833+0000 mgr.y (mgr.34107) 5366 : cluster [DBG] pgmap v6639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:39 smithi115 bash[45495]: cluster 2022-04-22T20:02:38.677833+0000 mgr.y (mgr.34107) 5366 : cluster [DBG] pgmap v6639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:39 smithi115 bash[46934]: cluster 2022-04-22T20:02:38.677833+0000 mgr.y (mgr.34107) 5366 : cluster [DBG] pgmap v6639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:40 smithi121 bash[36902]: audit 2022-04-22T20:02:40.330141+0000 mon.a (mon.0) 8420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:40 smithi115 bash[45495]: audit 2022-04-22T20:02:40.330141+0000 mon.a (mon.0) 8420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:40 smithi115 bash[46934]: audit 2022-04-22T20:02:40.330141+0000 mon.a (mon.0) 8420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:42.229 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:41 smithi115 bash[45495]: cluster 2022-04-22T20:02:40.679006+0000 mgr.y (mgr.34107) 5367 : cluster [DBG] pgmap v6640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:42.229 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:41 smithi115 bash[45495]: audit 2022-04-22T20:02:41.640967+0000 mon.a (mon.0) 8421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:42.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:41 smithi115 bash[46934]: cluster 2022-04-22T20:02:40.679006+0000 mgr.y (mgr.34107) 5367 : cluster [DBG] pgmap v6640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:42.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:41 smithi115 bash[46934]: audit 2022-04-22T20:02:41.640967+0000 mon.a (mon.0) 8421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:41 smithi121 bash[36902]: cluster 2022-04-22T20:02:40.679006+0000 mgr.y (mgr.34107) 5367 : cluster [DBG] pgmap v6640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:41 smithi121 bash[36902]: audit 2022-04-22T20:02:41.640967+0000 mon.a (mon.0) 8421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:42.220Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:43 smithi121 bash[36902]: cluster 2022-04-22T20:02:42.680244+0000 mgr.y (mgr.34107) 5368 : cluster [DBG] pgmap v6641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:43 smithi121 bash[36902]: audit 2022-04-22T20:02:43.087080+0000 mon.a (mon.0) 8422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:44.100Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:43 smithi115 bash[45495]: cluster 2022-04-22T20:02:42.680244+0000 mgr.y (mgr.34107) 5368 : cluster [DBG] pgmap v6641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:43 smithi115 bash[45495]: audit 2022-04-22T20:02:43.087080+0000 mon.a (mon.0) 8422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:43 smithi115 bash[46934]: cluster 2022-04-22T20:02:42.680244+0000 mgr.y (mgr.34107) 5368 : cluster [DBG] pgmap v6641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:43 smithi115 bash[46934]: audit 2022-04-22T20:02:43.087080+0000 mon.a (mon.0) 8422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:45 smithi121 bash[36902]: cluster 2022-04-22T20:02:44.681495+0000 mgr.y (mgr.34107) 5369 : cluster [DBG] pgmap v6642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:45 smithi121 bash[36902]: audit 2022-04-22T20:02:45.334161+0000 mon.a (mon.0) 8423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:45 smithi115 bash[45495]: cluster 2022-04-22T20:02:44.681495+0000 mgr.y (mgr.34107) 5369 : cluster [DBG] pgmap v6642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:45 smithi115 bash[45495]: audit 2022-04-22T20:02:45.334161+0000 mon.a (mon.0) 8423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:45 smithi115 bash[46934]: cluster 2022-04-22T20:02:44.681495+0000 mgr.y (mgr.34107) 5369 : cluster [DBG] pgmap v6642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:45 smithi115 bash[46934]: audit 2022-04-22T20:02:45.334161+0000 mon.a (mon.0) 8423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:46 smithi121 bash[36902]: audit 2022-04-22T20:02:46.644145+0000 mon.a (mon.0) 8424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:47.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:46 smithi115 bash[45495]: audit 2022-04-22T20:02:46.644145+0000 mon.a (mon.0) 8424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:46 smithi115 bash[46934]: audit 2022-04-22T20:02:46.644145+0000 mon.a (mon.0) 8424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:47.716 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:02:48.063 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:02:48.063 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 57s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:02:48.063 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:02:48.063 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 57s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 57s ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 57s ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 57s ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 57s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:02:48.064 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 57s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 57s ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 57s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 57s ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:02:48.065 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:02:48.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:47 smithi121 bash[36902]: cluster 2022-04-22T20:02:46.682506+0000 mgr.y (mgr.34107) 5370 : cluster [DBG] pgmap v6643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:47 smithi115 bash[46934]: cluster 2022-04-22T20:02:46.682506+0000 mgr.y (mgr.34107) 5370 : cluster [DBG] pgmap v6643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:47 smithi115 bash[45495]: cluster 2022-04-22T20:02:46.682506+0000 mgr.y (mgr.34107) 5370 : cluster [DBG] pgmap v6643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:48.476 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:02:48.476 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:02:48.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:02:48.478 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:02:48.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:02:48.478 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:02:48.478 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:02:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:48 smithi121 bash[36902]: audit 2022-04-22T20:02:47.691321+0000 mgr.y (mgr.34107) 5371 : audit [DBG] from='client.40986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:48 smithi121 bash[36902]: audit 2022-04-22T20:02:48.033878+0000 mgr.y (mgr.34107) 5372 : audit [DBG] from='client.50926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:48 smithi121 bash[36902]: audit 2022-04-22T20:02:48.089944+0000 mon.a (mon.0) 8425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:48 smithi121 bash[36902]: audit 2022-04-22T20:02:48.465467+0000 mon.c (mon.1) 223 : audit [DBG] from='client.? 172.21.15.115:0/3106731453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:48 smithi121 bash[36902]: cluster 2022-04-22T20:02:48.683683+0000 mgr.y (mgr.34107) 5373 : cluster [DBG] pgmap v6644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[45495]: audit 2022-04-22T20:02:47.691321+0000 mgr.y (mgr.34107) 5371 : audit [DBG] from='client.40986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[45495]: audit 2022-04-22T20:02:48.033878+0000 mgr.y (mgr.34107) 5372 : audit [DBG] from='client.50926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[45495]: audit 2022-04-22T20:02:48.089944+0000 mon.a (mon.0) 8425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:49.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[45495]: audit 2022-04-22T20:02:48.465467+0000 mon.c (mon.1) 223 : audit [DBG] from='client.? 172.21.15.115:0/3106731453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:49.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[45495]: cluster 2022-04-22T20:02:48.683683+0000 mgr.y (mgr.34107) 5373 : cluster [DBG] pgmap v6644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:49.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[46934]: audit 2022-04-22T20:02:47.691321+0000 mgr.y (mgr.34107) 5371 : audit [DBG] from='client.40986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[46934]: audit 2022-04-22T20:02:48.033878+0000 mgr.y (mgr.34107) 5372 : audit [DBG] from='client.50926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:02:49.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[46934]: audit 2022-04-22T20:02:48.089944+0000 mon.a (mon.0) 8425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:49.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[46934]: audit 2022-04-22T20:02:48.465467+0000 mon.c (mon.1) 223 : audit [DBG] from='client.? 172.21.15.115:0/3106731453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:02:49.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:48 smithi115 bash[46934]: cluster 2022-04-22T20:02:48.683683+0000 mgr.y (mgr.34107) 5373 : cluster [DBG] pgmap v6644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:50 smithi115 bash[46934]: audit 2022-04-22T20:02:50.338468+0000 mon.a (mon.0) 8426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:50.869 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:50 smithi115 bash[45495]: audit 2022-04-22T20:02:50.338468+0000 mon.a (mon.0) 8426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:50.872 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:50 smithi121 bash[36902]: audit 2022-04-22T20:02:50.338468+0000 mon.a (mon.0) 8426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:51 smithi121 bash[36902]: cluster 2022-04-22T20:02:50.684816+0000 mgr.y (mgr.34107) 5374 : cluster [DBG] pgmap v6645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:51 smithi121 bash[36902]: audit 2022-04-22T20:02:50.838498+0000 mon.a (mon.0) 8427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:02:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:51 smithi121 bash[36902]: audit 2022-04-22T20:02:50.841422+0000 mon.a (mon.0) 8428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:02:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:51 smithi121 bash[36902]: audit 2022-04-22T20:02:50.843179+0000 mon.a (mon.0) 8429 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:02:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:51 smithi121 bash[36902]: audit 2022-04-22T20:02:51.192550+0000 mon.a (mon.0) 8430 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[46934]: cluster 2022-04-22T20:02:50.684816+0000 mgr.y (mgr.34107) 5374 : cluster [DBG] pgmap v6645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[46934]: audit 2022-04-22T20:02:50.838498+0000 mon.a (mon.0) 8427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[46934]: audit 2022-04-22T20:02:50.841422+0000 mon.a (mon.0) 8428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[46934]: audit 2022-04-22T20:02:50.843179+0000 mon.a (mon.0) 8429 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[46934]: audit 2022-04-22T20:02:51.192550+0000 mon.a (mon.0) 8430 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:02:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[45495]: cluster 2022-04-22T20:02:50.684816+0000 mgr.y (mgr.34107) 5374 : cluster [DBG] pgmap v6645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[45495]: audit 2022-04-22T20:02:50.838498+0000 mon.a (mon.0) 8427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:02:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[45495]: audit 2022-04-22T20:02:50.841422+0000 mon.a (mon.0) 8428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:02:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[45495]: audit 2022-04-22T20:02:50.843179+0000 mon.a (mon.0) 8429 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:02:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:51 smithi115 bash[45495]: audit 2022-04-22T20:02:51.192550+0000 mon.a (mon.0) 8430 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:02:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:52.221Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:52 smithi121 bash[36902]: audit 2022-04-22T20:02:51.647028+0000 mon.a (mon.0) 8431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:52 smithi115 bash[45495]: audit 2022-04-22T20:02:51.647028+0000 mon.a (mon.0) 8431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:52 smithi115 bash[46934]: audit 2022-04-22T20:02:51.647028+0000 mon.a (mon.0) 8431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:53 smithi121 bash[36902]: cluster 2022-04-22T20:02:52.685934+0000 mgr.y (mgr.34107) 5375 : cluster [DBG] pgmap v6646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:53 smithi121 bash[36902]: audit 2022-04-22T20:02:53.092903+0000 mon.a (mon.0) 8432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:53 smithi115 bash[45495]: cluster 2022-04-22T20:02:52.685934+0000 mgr.y (mgr.34107) 5375 : cluster [DBG] pgmap v6646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:53 smithi115 bash[45495]: audit 2022-04-22T20:02:53.092903+0000 mon.a (mon.0) 8432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:53 smithi115 bash[46934]: cluster 2022-04-22T20:02:52.685934+0000 mgr.y (mgr.34107) 5375 : cluster [DBG] pgmap v6646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:54.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:53 smithi115 bash[46934]: audit 2022-04-22T20:02:53.092903+0000 mon.a (mon.0) 8432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:02:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:02:54.099Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:02:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:55 smithi121 bash[36902]: cluster 2022-04-22T20:02:54.687093+0000 mgr.y (mgr.34107) 5376 : cluster [DBG] pgmap v6647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:55 smithi121 bash[36902]: audit 2022-04-22T20:02:55.342192+0000 mon.a (mon.0) 8433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:55 smithi115 bash[45495]: cluster 2022-04-22T20:02:54.687093+0000 mgr.y (mgr.34107) 5376 : cluster [DBG] pgmap v6647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:56.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:55 smithi115 bash[45495]: audit 2022-04-22T20:02:55.342192+0000 mon.a (mon.0) 8433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:55 smithi115 bash[46934]: cluster 2022-04-22T20:02:54.687093+0000 mgr.y (mgr.34107) 5376 : cluster [DBG] pgmap v6647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:56.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:55 smithi115 bash[46934]: audit 2022-04-22T20:02:55.342192+0000 mon.a (mon.0) 8433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:02:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:56 smithi121 bash[36902]: audit 2022-04-22T20:02:56.650139+0000 mon.a (mon.0) 8434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:57.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:56 smithi115 bash[45495]: audit 2022-04-22T20:02:56.650139+0000 mon.a (mon.0) 8434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:57.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:56 smithi115 bash[46934]: audit 2022-04-22T20:02:56.650139+0000 mon.a (mon.0) 8434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:02:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:57 smithi121 bash[36902]: cluster 2022-04-22T20:02:56.688203+0000 mgr.y (mgr.34107) 5377 : cluster [DBG] pgmap v6648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:58.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:57 smithi115 bash[45495]: cluster 2022-04-22T20:02:56.688203+0000 mgr.y (mgr.34107) 5377 : cluster [DBG] pgmap v6648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:58.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:57 smithi115 bash[46934]: cluster 2022-04-22T20:02:56.688203+0000 mgr.y (mgr.34107) 5377 : cluster [DBG] pgmap v6648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:02:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:58 smithi121 bash[36902]: audit 2022-04-22T20:02:58.095737+0000 mon.a (mon.0) 8435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:59.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:58 smithi115 bash[45495]: audit 2022-04-22T20:02:58.095737+0000 mon.a (mon.0) 8435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:02:59.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:58 smithi115 bash[46934]: audit 2022-04-22T20:02:58.095737+0000 mon.a (mon.0) 8435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:02:59 smithi115 bash[45495]: cluster 2022-04-22T20:02:58.689341+0000 mgr.y (mgr.34107) 5378 : cluster [DBG] pgmap v6649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:02:59 smithi115 bash[46934]: cluster 2022-04-22T20:02:58.689341+0000 mgr.y (mgr.34107) 5378 : cluster [DBG] pgmap v6649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:02:59 smithi121 bash[36902]: cluster 2022-04-22T20:02:58.689341+0000 mgr.y (mgr.34107) 5378 : cluster [DBG] pgmap v6649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:00 smithi115 bash[45495]: audit 2022-04-22T20:03:00.345615+0000 mon.a (mon.0) 8436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:01.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:00 smithi115 bash[46934]: audit 2022-04-22T20:03:00.345615+0000 mon.a (mon.0) 8436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:00 smithi121 bash[36902]: audit 2022-04-22T20:03:00.345615+0000 mon.a (mon.0) 8436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:01 smithi115 bash[45495]: cluster 2022-04-22T20:03:00.690416+0000 mgr.y (mgr.34107) 5379 : cluster [DBG] pgmap v6650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:01 smithi115 bash[45495]: audit 2022-04-22T20:03:01.653150+0000 mon.a (mon.0) 8437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:01 smithi115 bash[46934]: cluster 2022-04-22T20:03:00.690416+0000 mgr.y (mgr.34107) 5379 : cluster [DBG] pgmap v6650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:01 smithi115 bash[46934]: audit 2022-04-22T20:03:01.653150+0000 mon.a (mon.0) 8437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:01 smithi121 bash[36902]: cluster 2022-04-22T20:03:00.690416+0000 mgr.y (mgr.34107) 5379 : cluster [DBG] pgmap v6650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:01 smithi121 bash[36902]: audit 2022-04-22T20:03:01.653150+0000 mon.a (mon.0) 8437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:02.222Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:03 smithi115 bash[45495]: cluster 2022-04-22T20:03:02.691533+0000 mgr.y (mgr.34107) 5380 : cluster [DBG] pgmap v6651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:03 smithi115 bash[45495]: audit 2022-04-22T20:03:03.098611+0000 mon.a (mon.0) 8438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:03 smithi115 bash[46934]: cluster 2022-04-22T20:03:02.691533+0000 mgr.y (mgr.34107) 5380 : cluster [DBG] pgmap v6651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:04.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:03 smithi115 bash[46934]: audit 2022-04-22T20:03:03.098611+0000 mon.a (mon.0) 8438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:03 smithi121 bash[36902]: cluster 2022-04-22T20:03:02.691533+0000 mgr.y (mgr.34107) 5380 : cluster [DBG] pgmap v6651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:03 smithi121 bash[36902]: audit 2022-04-22T20:03:03.098611+0000 mon.a (mon.0) 8438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:04.099Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:05 smithi115 bash[45495]: cluster 2022-04-22T20:03:04.692610+0000 mgr.y (mgr.34107) 5381 : cluster [DBG] pgmap v6652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:05 smithi115 bash[45495]: audit 2022-04-22T20:03:05.348941+0000 mon.a (mon.0) 8439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:05 smithi115 bash[46934]: cluster 2022-04-22T20:03:04.692610+0000 mgr.y (mgr.34107) 5381 : cluster [DBG] pgmap v6652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:05 smithi115 bash[46934]: audit 2022-04-22T20:03:05.348941+0000 mon.a (mon.0) 8439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:05 smithi121 bash[36902]: cluster 2022-04-22T20:03:04.692610+0000 mgr.y (mgr.34107) 5381 : cluster [DBG] pgmap v6652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:05 smithi121 bash[36902]: audit 2022-04-22T20:03:05.348941+0000 mon.a (mon.0) 8439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:06 smithi121 bash[36902]: audit 2022-04-22T20:03:06.656430+0000 mon.a (mon.0) 8440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:06 smithi115 bash[46934]: audit 2022-04-22T20:03:06.656430+0000 mon.a (mon.0) 8440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:06 smithi115 bash[45495]: audit 2022-04-22T20:03:06.656430+0000 mon.a (mon.0) 8440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:07 smithi121 bash[36902]: cluster 2022-04-22T20:03:06.693681+0000 mgr.y (mgr.34107) 5382 : cluster [DBG] pgmap v6653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:07 smithi115 bash[45495]: cluster 2022-04-22T20:03:06.693681+0000 mgr.y (mgr.34107) 5382 : cluster [DBG] pgmap v6653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:07 smithi115 bash[46934]: cluster 2022-04-22T20:03:06.693681+0000 mgr.y (mgr.34107) 5382 : cluster [DBG] pgmap v6653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:08 smithi121 bash[36902]: audit 2022-04-22T20:03:08.101691+0000 mon.a (mon.0) 8441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:08 smithi115 bash[45495]: audit 2022-04-22T20:03:08.101691+0000 mon.a (mon.0) 8441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:08 smithi115 bash[46934]: audit 2022-04-22T20:03:08.101691+0000 mon.a (mon.0) 8441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:09 smithi121 bash[36902]: cluster 2022-04-22T20:03:08.694891+0000 mgr.y (mgr.34107) 5383 : cluster [DBG] pgmap v6654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:09 smithi115 bash[45495]: cluster 2022-04-22T20:03:08.694891+0000 mgr.y (mgr.34107) 5383 : cluster [DBG] pgmap v6654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:09 smithi115 bash[46934]: cluster 2022-04-22T20:03:08.694891+0000 mgr.y (mgr.34107) 5383 : cluster [DBG] pgmap v6654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:10 smithi121 bash[36902]: audit 2022-04-22T20:03:10.352182+0000 mon.a (mon.0) 8442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:10 smithi115 bash[45495]: audit 2022-04-22T20:03:10.352182+0000 mon.a (mon.0) 8442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:11.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:10 smithi115 bash[46934]: audit 2022-04-22T20:03:10.352182+0000 mon.a (mon.0) 8442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:12.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:11 smithi115 bash[45495]: cluster 2022-04-22T20:03:10.696140+0000 mgr.y (mgr.34107) 5384 : cluster [DBG] pgmap v6655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:12.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:11 smithi115 bash[45495]: audit 2022-04-22T20:03:11.659732+0000 mon.a (mon.0) 8443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:12.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:11 smithi115 bash[46934]: cluster 2022-04-22T20:03:10.696140+0000 mgr.y (mgr.34107) 5384 : cluster [DBG] pgmap v6655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:12.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:11 smithi115 bash[46934]: audit 2022-04-22T20:03:11.659732+0000 mon.a (mon.0) 8443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:11 smithi121 bash[36902]: cluster 2022-04-22T20:03:10.696140+0000 mgr.y (mgr.34107) 5384 : cluster [DBG] pgmap v6655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:11 smithi121 bash[36902]: audit 2022-04-22T20:03:11.659732+0000 mon.a (mon.0) 8443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:12.221Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:13 smithi115 bash[46934]: cluster 2022-04-22T20:03:12.697237+0000 mgr.y (mgr.34107) 5385 : cluster [DBG] pgmap v6656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:13 smithi115 bash[46934]: audit 2022-04-22T20:03:13.104763+0000 mon.a (mon.0) 8444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:14.110 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:13 smithi115 bash[45495]: cluster 2022-04-22T20:03:12.697237+0000 mgr.y (mgr.34107) 5385 : cluster [DBG] pgmap v6656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:14.110 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:13 smithi115 bash[45495]: audit 2022-04-22T20:03:13.104763+0000 mon.a (mon.0) 8444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:13 smithi121 bash[36902]: cluster 2022-04-22T20:03:12.697237+0000 mgr.y (mgr.34107) 5385 : cluster [DBG] pgmap v6656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:13 smithi121 bash[36902]: audit 2022-04-22T20:03:13.104763+0000 mon.a (mon.0) 8444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:14.100Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:15 smithi121 bash[36902]: cluster 2022-04-22T20:03:14.698301+0000 mgr.y (mgr.34107) 5386 : cluster [DBG] pgmap v6657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:15 smithi121 bash[36902]: audit 2022-04-22T20:03:15.085850+0000 mon.a (mon.0) 8445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:03:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:15 smithi121 bash[36902]: audit 2022-04-22T20:03:15.086549+0000 mon.a (mon.0) 8446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:03:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:15 smithi121 bash[36902]: audit 2022-04-22T20:03:15.355895+0000 mon.a (mon.0) 8447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[46934]: cluster 2022-04-22T20:03:14.698301+0000 mgr.y (mgr.34107) 5386 : cluster [DBG] pgmap v6657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[46934]: audit 2022-04-22T20:03:15.085850+0000 mon.a (mon.0) 8445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[46934]: audit 2022-04-22T20:03:15.086549+0000 mon.a (mon.0) 8446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:03:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[46934]: audit 2022-04-22T20:03:15.355895+0000 mon.a (mon.0) 8447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[45495]: cluster 2022-04-22T20:03:14.698301+0000 mgr.y (mgr.34107) 5386 : cluster [DBG] pgmap v6657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[45495]: audit 2022-04-22T20:03:15.085850+0000 mon.a (mon.0) 8445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:03:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[45495]: audit 2022-04-22T20:03:15.086549+0000 mon.a (mon.0) 8446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:03:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:15 smithi115 bash[45495]: audit 2022-04-22T20:03:15.355895+0000 mon.a (mon.0) 8447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:16 smithi121 bash[36902]: audit 2022-04-22T20:03:16.662818+0000 mon.a (mon.0) 8448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:17.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:16 smithi115 bash[45495]: audit 2022-04-22T20:03:16.662818+0000 mon.a (mon.0) 8448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:17.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:16 smithi115 bash[46934]: audit 2022-04-22T20:03:16.662818+0000 mon.a (mon.0) 8448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:17 smithi121 bash[36902]: cluster 2022-04-22T20:03:16.699355+0000 mgr.y (mgr.34107) 5387 : cluster [DBG] pgmap v6658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:17 smithi115 bash[46934]: cluster 2022-04-22T20:03:16.699355+0000 mgr.y (mgr.34107) 5387 : cluster [DBG] pgmap v6658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:17 smithi115 bash[45495]: cluster 2022-04-22T20:03:16.699355+0000 mgr.y (mgr.34107) 5387 : cluster [DBG] pgmap v6658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:18.845 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:03:19.200 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:03:19.200 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 88s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 88s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 88s ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 88s ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 88s ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:03:19.201 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 88s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 88s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 88s ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 88s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 88s ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:03:19.202 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:03:19.203 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:03:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:18 smithi121 bash[36902]: audit 2022-04-22T20:03:18.107617+0000 mon.a (mon.0) 8449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:18 smithi115 bash[46934]: audit 2022-04-22T20:03:18.107617+0000 mon.a (mon.0) 8449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:18 smithi115 bash[45495]: audit 2022-04-22T20:03:18.107617+0000 mon.a (mon.0) 8449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:19.613 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:03:19.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:03:19.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:03:19.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:03:19.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:03:19.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:03:19.615 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:03:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:19 smithi121 bash[36902]: cluster 2022-04-22T20:03:18.700422+0000 mgr.y (mgr.34107) 5388 : cluster [DBG] pgmap v6659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:19 smithi121 bash[36902]: audit 2022-04-22T20:03:18.820720+0000 mgr.y (mgr.34107) 5389 : audit [DBG] from='client.50938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:19 smithi121 bash[36902]: audit 2022-04-22T20:03:19.604322+0000 mon.a (mon.0) 8450 : audit [DBG] from='client.? 172.21.15.115:0/229706509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[45495]: cluster 2022-04-22T20:03:18.700422+0000 mgr.y (mgr.34107) 5388 : cluster [DBG] pgmap v6659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[45495]: audit 2022-04-22T20:03:18.820720+0000 mgr.y (mgr.34107) 5389 : audit [DBG] from='client.50938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[45495]: audit 2022-04-22T20:03:19.604322+0000 mon.a (mon.0) 8450 : audit [DBG] from='client.? 172.21.15.115:0/229706509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[46934]: cluster 2022-04-22T20:03:18.700422+0000 mgr.y (mgr.34107) 5388 : cluster [DBG] pgmap v6659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[46934]: audit 2022-04-22T20:03:18.820720+0000 mgr.y (mgr.34107) 5389 : audit [DBG] from='client.50938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:20.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:19 smithi115 bash[46934]: audit 2022-04-22T20:03:19.604322+0000 mon.a (mon.0) 8450 : audit [DBG] from='client.? 172.21.15.115:0/229706509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:20 smithi121 bash[36902]: audit 2022-04-22T20:03:19.167485+0000 mgr.y (mgr.34107) 5390 : audit [DBG] from='client.41010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:20 smithi121 bash[36902]: audit 2022-04-22T20:03:20.359589+0000 mon.a (mon.0) 8451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:20 smithi115 bash[45495]: audit 2022-04-22T20:03:19.167485+0000 mgr.y (mgr.34107) 5390 : audit [DBG] from='client.41010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:20 smithi115 bash[45495]: audit 2022-04-22T20:03:20.359589+0000 mon.a (mon.0) 8451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:20 smithi115 bash[46934]: audit 2022-04-22T20:03:19.167485+0000 mgr.y (mgr.34107) 5390 : audit [DBG] from='client.41010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:20 smithi115 bash[46934]: audit 2022-04-22T20:03:20.359589+0000 mon.a (mon.0) 8451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:22.237 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:22.223Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:22.237 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:21 smithi115 bash[45495]: cluster 2022-04-22T20:03:20.701510+0000 mgr.y (mgr.34107) 5391 : cluster [DBG] pgmap v6660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:22.238 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:21 smithi115 bash[45495]: audit 2022-04-22T20:03:21.666132+0000 mon.a (mon.0) 8452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:22.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:21 smithi115 bash[46934]: cluster 2022-04-22T20:03:20.701510+0000 mgr.y (mgr.34107) 5391 : cluster [DBG] pgmap v6660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:22.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:21 smithi115 bash[46934]: audit 2022-04-22T20:03:21.666132+0000 mon.a (mon.0) 8452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:21 smithi121 bash[36902]: cluster 2022-04-22T20:03:20.701510+0000 mgr.y (mgr.34107) 5391 : cluster [DBG] pgmap v6660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:21 smithi121 bash[36902]: audit 2022-04-22T20:03:21.666132+0000 mon.a (mon.0) 8452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:23 smithi121 bash[36902]: cluster 2022-04-22T20:03:22.702645+0000 mgr.y (mgr.34107) 5392 : cluster [DBG] pgmap v6661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:23 smithi121 bash[36902]: audit 2022-04-22T20:03:23.110732+0000 mon.a (mon.0) 8453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:24.100Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:23 smithi115 bash[45495]: cluster 2022-04-22T20:03:22.702645+0000 mgr.y (mgr.34107) 5392 : cluster [DBG] pgmap v6661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:23 smithi115 bash[45495]: audit 2022-04-22T20:03:23.110732+0000 mon.a (mon.0) 8453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:23 smithi115 bash[46934]: cluster 2022-04-22T20:03:22.702645+0000 mgr.y (mgr.34107) 5392 : cluster [DBG] pgmap v6661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:23 smithi115 bash[46934]: audit 2022-04-22T20:03:23.110732+0000 mon.a (mon.0) 8453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:25 smithi121 bash[36902]: cluster 2022-04-22T20:03:24.703759+0000 mgr.y (mgr.34107) 5393 : cluster [DBG] pgmap v6662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:25 smithi121 bash[36902]: audit 2022-04-22T20:03:25.363335+0000 mon.a (mon.0) 8454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:25 smithi115 bash[45495]: cluster 2022-04-22T20:03:24.703759+0000 mgr.y (mgr.34107) 5393 : cluster [DBG] pgmap v6662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:25 smithi115 bash[45495]: audit 2022-04-22T20:03:25.363335+0000 mon.a (mon.0) 8454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:25 smithi115 bash[46934]: cluster 2022-04-22T20:03:24.703759+0000 mgr.y (mgr.34107) 5393 : cluster [DBG] pgmap v6662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:25 smithi115 bash[46934]: audit 2022-04-22T20:03:25.363335+0000 mon.a (mon.0) 8454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:26 smithi121 bash[36902]: audit 2022-04-22T20:03:26.669542+0000 mon.a (mon.0) 8455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:26 smithi115 bash[45495]: audit 2022-04-22T20:03:26.669542+0000 mon.a (mon.0) 8455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:27.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:26 smithi115 bash[46934]: audit 2022-04-22T20:03:26.669542+0000 mon.a (mon.0) 8455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:27 smithi121 bash[36902]: cluster 2022-04-22T20:03:26.704987+0000 mgr.y (mgr.34107) 5394 : cluster [DBG] pgmap v6663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:27 smithi115 bash[45495]: cluster 2022-04-22T20:03:26.704987+0000 mgr.y (mgr.34107) 5394 : cluster [DBG] pgmap v6663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:27 smithi115 bash[46934]: cluster 2022-04-22T20:03:26.704987+0000 mgr.y (mgr.34107) 5394 : cluster [DBG] pgmap v6663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:28 smithi121 bash[36902]: audit 2022-04-22T20:03:28.114141+0000 mon.a (mon.0) 8456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:28 smithi115 bash[45495]: audit 2022-04-22T20:03:28.114141+0000 mon.a (mon.0) 8456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:28 smithi115 bash[46934]: audit 2022-04-22T20:03:28.114141+0000 mon.a (mon.0) 8456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:29 smithi121 bash[36902]: cluster 2022-04-22T20:03:28.706197+0000 mgr.y (mgr.34107) 5395 : cluster [DBG] pgmap v6664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:29 smithi115 bash[45495]: cluster 2022-04-22T20:03:28.706197+0000 mgr.y (mgr.34107) 5395 : cluster [DBG] pgmap v6664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:29 smithi115 bash[46934]: cluster 2022-04-22T20:03:28.706197+0000 mgr.y (mgr.34107) 5395 : cluster [DBG] pgmap v6664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:30 smithi121 bash[36902]: audit 2022-04-22T20:03:30.366760+0000 mon.a (mon.0) 8457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:30 smithi115 bash[45495]: audit 2022-04-22T20:03:30.366760+0000 mon.a (mon.0) 8457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:30 smithi115 bash[46934]: audit 2022-04-22T20:03:30.366760+0000 mon.a (mon.0) 8457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:32.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:31 smithi115 bash[46934]: cluster 2022-04-22T20:03:30.707397+0000 mgr.y (mgr.34107) 5396 : cluster [DBG] pgmap v6665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:32.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:31 smithi115 bash[46934]: audit 2022-04-22T20:03:31.672822+0000 mon.a (mon.0) 8458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:32.233 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:31 smithi115 bash[45495]: cluster 2022-04-22T20:03:30.707397+0000 mgr.y (mgr.34107) 5396 : cluster [DBG] pgmap v6665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:32.233 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:31 smithi115 bash[45495]: audit 2022-04-22T20:03:31.672822+0000 mon.a (mon.0) 8458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:31 smithi121 bash[36902]: cluster 2022-04-22T20:03:30.707397+0000 mgr.y (mgr.34107) 5396 : cluster [DBG] pgmap v6665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:31 smithi121 bash[36902]: audit 2022-04-22T20:03:31.672822+0000 mon.a (mon.0) 8458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:32.223Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:33 smithi121 bash[36902]: cluster 2022-04-22T20:03:32.708649+0000 mgr.y (mgr.34107) 5397 : cluster [DBG] pgmap v6666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:33 smithi121 bash[36902]: audit 2022-04-22T20:03:33.117372+0000 mon.a (mon.0) 8459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:33 smithi115 bash[45495]: cluster 2022-04-22T20:03:32.708649+0000 mgr.y (mgr.34107) 5397 : cluster [DBG] pgmap v6666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:33 smithi115 bash[45495]: audit 2022-04-22T20:03:33.117372+0000 mon.a (mon.0) 8459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:34.101Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:33 smithi115 bash[46934]: cluster 2022-04-22T20:03:32.708649+0000 mgr.y (mgr.34107) 5397 : cluster [DBG] pgmap v6666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:33 smithi115 bash[46934]: audit 2022-04-22T20:03:33.117372+0000 mon.a (mon.0) 8459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:35 smithi121 bash[36902]: cluster 2022-04-22T20:03:34.709916+0000 mgr.y (mgr.34107) 5398 : cluster [DBG] pgmap v6667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:35 smithi121 bash[36902]: audit 2022-04-22T20:03:35.370570+0000 mon.a (mon.0) 8460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:35 smithi115 bash[45495]: cluster 2022-04-22T20:03:34.709916+0000 mgr.y (mgr.34107) 5398 : cluster [DBG] pgmap v6667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:35 smithi115 bash[45495]: audit 2022-04-22T20:03:35.370570+0000 mon.a (mon.0) 8460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:35 smithi115 bash[46934]: cluster 2022-04-22T20:03:34.709916+0000 mgr.y (mgr.34107) 5398 : cluster [DBG] pgmap v6667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:35 smithi115 bash[46934]: audit 2022-04-22T20:03:35.370570+0000 mon.a (mon.0) 8460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:36 smithi121 bash[36902]: audit 2022-04-22T20:03:36.675910+0000 mon.a (mon.0) 8461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:36 smithi115 bash[45495]: audit 2022-04-22T20:03:36.675910+0000 mon.a (mon.0) 8461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:36 smithi115 bash[46934]: audit 2022-04-22T20:03:36.675910+0000 mon.a (mon.0) 8461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:37 smithi121 bash[36902]: cluster 2022-04-22T20:03:36.710911+0000 mgr.y (mgr.34107) 5399 : cluster [DBG] pgmap v6668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:37 smithi115 bash[45495]: cluster 2022-04-22T20:03:36.710911+0000 mgr.y (mgr.34107) 5399 : cluster [DBG] pgmap v6668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:37 smithi115 bash[46934]: cluster 2022-04-22T20:03:36.710911+0000 mgr.y (mgr.34107) 5399 : cluster [DBG] pgmap v6668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:38 smithi121 bash[36902]: audit 2022-04-22T20:03:38.120574+0000 mon.a (mon.0) 8462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:38 smithi121 bash[36902]: cluster 2022-04-22T20:03:38.712113+0000 mgr.y (mgr.34107) 5400 : cluster [DBG] pgmap v6669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:38 smithi115 bash[45495]: audit 2022-04-22T20:03:38.120574+0000 mon.a (mon.0) 8462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:38 smithi115 bash[45495]: cluster 2022-04-22T20:03:38.712113+0000 mgr.y (mgr.34107) 5400 : cluster [DBG] pgmap v6669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:38 smithi115 bash[46934]: audit 2022-04-22T20:03:38.120574+0000 mon.a (mon.0) 8462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:38 smithi115 bash[46934]: cluster 2022-04-22T20:03:38.712113+0000 mgr.y (mgr.34107) 5400 : cluster [DBG] pgmap v6669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:40 smithi121 bash[36902]: audit 2022-04-22T20:03:40.374280+0000 mon.a (mon.0) 8463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:40 smithi115 bash[45495]: audit 2022-04-22T20:03:40.374280+0000 mon.a (mon.0) 8463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:40 smithi115 bash[46934]: audit 2022-04-22T20:03:40.374280+0000 mon.a (mon.0) 8463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:41 smithi121 bash[36902]: cluster 2022-04-22T20:03:40.713225+0000 mgr.y (mgr.34107) 5401 : cluster [DBG] pgmap v6670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:41 smithi115 bash[46934]: cluster 2022-04-22T20:03:40.713225+0000 mgr.y (mgr.34107) 5401 : cluster [DBG] pgmap v6670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:41 smithi115 bash[45495]: cluster 2022-04-22T20:03:40.713225+0000 mgr.y (mgr.34107) 5401 : cluster [DBG] pgmap v6670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:42.223Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:42 smithi121 bash[36902]: audit 2022-04-22T20:03:41.679303+0000 mon.a (mon.0) 8464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:42 smithi115 bash[45495]: audit 2022-04-22T20:03:41.679303+0000 mon.a (mon.0) 8464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:42 smithi115 bash[46934]: audit 2022-04-22T20:03:41.679303+0000 mon.a (mon.0) 8464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:43 smithi121 bash[36902]: cluster 2022-04-22T20:03:42.714389+0000 mgr.y (mgr.34107) 5402 : cluster [DBG] pgmap v6671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:43 smithi121 bash[36902]: audit 2022-04-22T20:03:43.123492+0000 mon.a (mon.0) 8465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:43 smithi115 bash[45495]: cluster 2022-04-22T20:03:42.714389+0000 mgr.y (mgr.34107) 5402 : cluster [DBG] pgmap v6671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:43 smithi115 bash[45495]: audit 2022-04-22T20:03:43.123492+0000 mon.a (mon.0) 8465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:43 smithi115 bash[46934]: cluster 2022-04-22T20:03:42.714389+0000 mgr.y (mgr.34107) 5402 : cluster [DBG] pgmap v6671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:43 smithi115 bash[46934]: audit 2022-04-22T20:03:43.123492+0000 mon.a (mon.0) 8465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:44.102Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:45 smithi115 bash[45495]: cluster 2022-04-22T20:03:44.715508+0000 mgr.y (mgr.34107) 5403 : cluster [DBG] pgmap v6672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:46.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:45 smithi115 bash[45495]: audit 2022-04-22T20:03:45.377743+0000 mon.a (mon.0) 8466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:46.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:45 smithi115 bash[46934]: cluster 2022-04-22T20:03:44.715508+0000 mgr.y (mgr.34107) 5403 : cluster [DBG] pgmap v6672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:46.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:45 smithi115 bash[46934]: audit 2022-04-22T20:03:45.377743+0000 mon.a (mon.0) 8466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:45 smithi121 bash[36902]: cluster 2022-04-22T20:03:44.715508+0000 mgr.y (mgr.34107) 5403 : cluster [DBG] pgmap v6672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:45 smithi121 bash[36902]: audit 2022-04-22T20:03:45.377743+0000 mon.a (mon.0) 8466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:47.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:46 smithi115 bash[45495]: audit 2022-04-22T20:03:46.682522+0000 mon.a (mon.0) 8467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:47.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:46 smithi115 bash[46934]: audit 2022-04-22T20:03:46.682522+0000 mon.a (mon.0) 8467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:46 smithi121 bash[36902]: audit 2022-04-22T20:03:46.682522+0000 mon.a (mon.0) 8467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:48.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:47 smithi115 bash[45495]: cluster 2022-04-22T20:03:46.716594+0000 mgr.y (mgr.34107) 5404 : cluster [DBG] pgmap v6673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:48.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:47 smithi115 bash[46934]: cluster 2022-04-22T20:03:46.716594+0000 mgr.y (mgr.34107) 5404 : cluster [DBG] pgmap v6673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:47 smithi121 bash[36902]: cluster 2022-04-22T20:03:46.716594+0000 mgr.y (mgr.34107) 5404 : cluster [DBG] pgmap v6673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:49.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:48 smithi115 bash[45495]: audit 2022-04-22T20:03:48.126199+0000 mon.a (mon.0) 8468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:49.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:48 smithi115 bash[46934]: audit 2022-04-22T20:03:48.126199+0000 mon.a (mon.0) 8468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:48 smithi121 bash[36902]: audit 2022-04-22T20:03:48.126199+0000 mon.a (mon.0) 8468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:49.993 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:03:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:49 smithi115 bash[45495]: cluster 2022-04-22T20:03:48.717778+0000 mgr.y (mgr.34107) 5405 : cluster [DBG] pgmap v6674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:49 smithi115 bash[46934]: cluster 2022-04-22T20:03:48.717778+0000 mgr.y (mgr.34107) 5405 : cluster [DBG] pgmap v6674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:49 smithi121 bash[36902]: cluster 2022-04-22T20:03:48.717778+0000 mgr.y (mgr.34107) 5405 : cluster [DBG] pgmap v6674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 119s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 119s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 119s ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:03:50.336 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 119s ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 119s ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 119s ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 119s ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 119s ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 119s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 119s ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:03:50.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:03:50.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:03:50.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:03:50.338 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:03:50.743 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:03:50.743 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:03:50.743 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:03:50.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:50.743 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:03:50.744 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:03:50.745 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:03:51.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:50 smithi115 bash[45495]: audit 2022-04-22T20:03:50.381401+0000 mon.a (mon.0) 8469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:51.231 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:50 smithi115 bash[45495]: audit 2022-04-22T20:03:50.732736+0000 mon.c (mon.1) 224 : audit [DBG] from='client.? 172.21.15.115:0/1536680278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:51.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:50 smithi115 bash[46934]: audit 2022-04-22T20:03:50.381401+0000 mon.a (mon.0) 8469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:51.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:50 smithi115 bash[46934]: audit 2022-04-22T20:03:50.732736+0000 mon.c (mon.1) 224 : audit [DBG] from='client.? 172.21.15.115:0/1536680278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:51.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:50 smithi121 bash[36902]: audit 2022-04-22T20:03:50.381401+0000 mon.a (mon.0) 8469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:51.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:50 smithi121 bash[36902]: audit 2022-04-22T20:03:50.732736+0000 mon.c (mon.1) 224 : audit [DBG] from='client.? 172.21.15.115:0/1536680278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:03:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:49.969800+0000 mgr.y (mgr.34107) 5406 : audit [DBG] from='client.50956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:50.306593+0000 mgr.y (mgr.34107) 5407 : audit [DBG] from='client.41028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: cluster 2022-04-22T20:03:50.718846+0000 mgr.y (mgr.34107) 5408 : cluster [DBG] pgmap v6675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:51.200114+0000 mon.a (mon.0) 8470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:51.203031+0000 mon.a (mon.0) 8471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:51.204736+0000 mon.a (mon.0) 8472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:51.579696+0000 mon.a (mon.0) 8473 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:51 smithi121 bash[36902]: audit 2022-04-22T20:03:51.685602+0000 mon.a (mon.0) 8474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:49.969800+0000 mgr.y (mgr.34107) 5406 : audit [DBG] from='client.50956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:50.306593+0000 mgr.y (mgr.34107) 5407 : audit [DBG] from='client.41028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: cluster 2022-04-22T20:03:50.718846+0000 mgr.y (mgr.34107) 5408 : cluster [DBG] pgmap v6675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:51.200114+0000 mon.a (mon.0) 8470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:51.203031+0000 mon.a (mon.0) 8471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:51.204736+0000 mon.a (mon.0) 8472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:51.579696+0000 mon.a (mon.0) 8473 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[46934]: audit 2022-04-22T20:03:51.685602+0000 mon.a (mon.0) 8474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:49.969800+0000 mgr.y (mgr.34107) 5406 : audit [DBG] from='client.50956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:50.306593+0000 mgr.y (mgr.34107) 5407 : audit [DBG] from='client.41028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:03:52.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: cluster 2022-04-22T20:03:50.718846+0000 mgr.y (mgr.34107) 5408 : cluster [DBG] pgmap v6675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:51.200114+0000 mon.a (mon.0) 8470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:03:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:51.203031+0000 mon.a (mon.0) 8471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:03:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:51.204736+0000 mon.a (mon.0) 8472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:03:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:51.579696+0000 mon.a (mon.0) 8473 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:03:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:51 smithi115 bash[45495]: audit 2022-04-22T20:03:51.685602+0000 mon.a (mon.0) 8474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:52.223Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:54.111 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:53 smithi115 bash[45495]: cluster 2022-04-22T20:03:52.719939+0000 mgr.y (mgr.34107) 5409 : cluster [DBG] pgmap v6676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:54.112 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:53 smithi115 bash[45495]: audit 2022-04-22T20:03:53.129330+0000 mon.a (mon.0) 8475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:53 smithi115 bash[46934]: cluster 2022-04-22T20:03:52.719939+0000 mgr.y (mgr.34107) 5409 : cluster [DBG] pgmap v6676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:53 smithi115 bash[46934]: audit 2022-04-22T20:03:53.129330+0000 mon.a (mon.0) 8475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:53 smithi121 bash[36902]: cluster 2022-04-22T20:03:52.719939+0000 mgr.y (mgr.34107) 5409 : cluster [DBG] pgmap v6676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:53 smithi121 bash[36902]: audit 2022-04-22T20:03:53.129330+0000 mon.a (mon.0) 8475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:03:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:03:54.102Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:03:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:55 smithi121 bash[36902]: cluster 2022-04-22T20:03:54.721049+0000 mgr.y (mgr.34107) 5410 : cluster [DBG] pgmap v6677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:55 smithi121 bash[36902]: audit 2022-04-22T20:03:55.384713+0000 mon.a (mon.0) 8476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:55 smithi115 bash[45495]: cluster 2022-04-22T20:03:54.721049+0000 mgr.y (mgr.34107) 5410 : cluster [DBG] pgmap v6677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:55 smithi115 bash[45495]: audit 2022-04-22T20:03:55.384713+0000 mon.a (mon.0) 8476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:55 smithi115 bash[46934]: cluster 2022-04-22T20:03:54.721049+0000 mgr.y (mgr.34107) 5410 : cluster [DBG] pgmap v6677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:55 smithi115 bash[46934]: audit 2022-04-22T20:03:55.384713+0000 mon.a (mon.0) 8476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:03:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:56 smithi121 bash[36902]: audit 2022-04-22T20:03:56.688758+0000 mon.a (mon.0) 8477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:56 smithi115 bash[45495]: audit 2022-04-22T20:03:56.688758+0000 mon.a (mon.0) 8477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:56 smithi115 bash[46934]: audit 2022-04-22T20:03:56.688758+0000 mon.a (mon.0) 8477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:03:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:57 smithi121 bash[36902]: cluster 2022-04-22T20:03:56.722078+0000 mgr.y (mgr.34107) 5411 : cluster [DBG] pgmap v6678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:57 smithi115 bash[45495]: cluster 2022-04-22T20:03:56.722078+0000 mgr.y (mgr.34107) 5411 : cluster [DBG] pgmap v6678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:58.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:57 smithi115 bash[46934]: cluster 2022-04-22T20:03:56.722078+0000 mgr.y (mgr.34107) 5411 : cluster [DBG] pgmap v6678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:03:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:58 smithi121 bash[36902]: audit 2022-04-22T20:03:58.132385+0000 mon.a (mon.0) 8478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:58 smithi115 bash[45495]: audit 2022-04-22T20:03:58.132385+0000 mon.a (mon.0) 8478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:03:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:58 smithi115 bash[46934]: audit 2022-04-22T20:03:58.132385+0000 mon.a (mon.0) 8478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:03:59 smithi121 bash[36902]: cluster 2022-04-22T20:03:58.723198+0000 mgr.y (mgr.34107) 5412 : cluster [DBG] pgmap v6679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:03:59 smithi115 bash[45495]: cluster 2022-04-22T20:03:58.723198+0000 mgr.y (mgr.34107) 5412 : cluster [DBG] pgmap v6679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:03:59 smithi115 bash[46934]: cluster 2022-04-22T20:03:58.723198+0000 mgr.y (mgr.34107) 5412 : cluster [DBG] pgmap v6679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:00 smithi121 bash[36902]: audit 2022-04-22T20:04:00.388238+0000 mon.a (mon.0) 8479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:00 smithi115 bash[45495]: audit 2022-04-22T20:04:00.388238+0000 mon.a (mon.0) 8479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:01.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:00 smithi115 bash[46934]: audit 2022-04-22T20:04:00.388238+0000 mon.a (mon.0) 8479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:02.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:01 smithi115 bash[45495]: cluster 2022-04-22T20:04:00.724325+0000 mgr.y (mgr.34107) 5413 : cluster [DBG] pgmap v6680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:02.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:01 smithi115 bash[45495]: audit 2022-04-22T20:04:01.691962+0000 mon.a (mon.0) 8480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:02.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:01 smithi115 bash[46934]: cluster 2022-04-22T20:04:00.724325+0000 mgr.y (mgr.34107) 5413 : cluster [DBG] pgmap v6680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:02.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:01 smithi115 bash[46934]: audit 2022-04-22T20:04:01.691962+0000 mon.a (mon.0) 8480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:01 smithi121 bash[36902]: cluster 2022-04-22T20:04:00.724325+0000 mgr.y (mgr.34107) 5413 : cluster [DBG] pgmap v6680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:01 smithi121 bash[36902]: audit 2022-04-22T20:04:01.691962+0000 mon.a (mon.0) 8480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:02.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:02.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:04.113 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:03 smithi115 bash[45495]: cluster 2022-04-22T20:04:02.725561+0000 mgr.y (mgr.34107) 5414 : cluster [DBG] pgmap v6681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:04.124 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:03 smithi115 bash[45495]: audit 2022-04-22T20:04:03.135620+0000 mon.a (mon.0) 8481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:04.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:03 smithi115 bash[46934]: cluster 2022-04-22T20:04:02.725561+0000 mgr.y (mgr.34107) 5414 : cluster [DBG] pgmap v6681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:04.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:03 smithi115 bash[46934]: audit 2022-04-22T20:04:03.135620+0000 mon.a (mon.0) 8481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:03 smithi121 bash[36902]: cluster 2022-04-22T20:04:02.725561+0000 mgr.y (mgr.34107) 5414 : cluster [DBG] pgmap v6681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:03 smithi121 bash[36902]: audit 2022-04-22T20:04:03.135620+0000 mon.a (mon.0) 8481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:04.105Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:05 smithi121 bash[36902]: cluster 2022-04-22T20:04:04.726669+0000 mgr.y (mgr.34107) 5415 : cluster [DBG] pgmap v6682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:05 smithi121 bash[36902]: audit 2022-04-22T20:04:05.391924+0000 mon.a (mon.0) 8482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:05 smithi115 bash[45495]: cluster 2022-04-22T20:04:04.726669+0000 mgr.y (mgr.34107) 5415 : cluster [DBG] pgmap v6682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:05 smithi115 bash[45495]: audit 2022-04-22T20:04:05.391924+0000 mon.a (mon.0) 8482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:05 smithi115 bash[46934]: cluster 2022-04-22T20:04:04.726669+0000 mgr.y (mgr.34107) 5415 : cluster [DBG] pgmap v6682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:05 smithi115 bash[46934]: audit 2022-04-22T20:04:05.391924+0000 mon.a (mon.0) 8482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:06 smithi121 bash[36902]: audit 2022-04-22T20:04:06.695233+0000 mon.a (mon.0) 8483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:06 smithi115 bash[46934]: audit 2022-04-22T20:04:06.695233+0000 mon.a (mon.0) 8483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:06 smithi115 bash[45495]: audit 2022-04-22T20:04:06.695233+0000 mon.a (mon.0) 8483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:07 smithi121 bash[36902]: cluster 2022-04-22T20:04:06.727921+0000 mgr.y (mgr.34107) 5416 : cluster [DBG] pgmap v6683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:07 smithi115 bash[46934]: cluster 2022-04-22T20:04:06.727921+0000 mgr.y (mgr.34107) 5416 : cluster [DBG] pgmap v6683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:07 smithi115 bash[45495]: cluster 2022-04-22T20:04:06.727921+0000 mgr.y (mgr.34107) 5416 : cluster [DBG] pgmap v6683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:08 smithi121 bash[36902]: audit 2022-04-22T20:04:08.138699+0000 mon.a (mon.0) 8484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:08 smithi115 bash[46934]: audit 2022-04-22T20:04:08.138699+0000 mon.a (mon.0) 8484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:08 smithi115 bash[45495]: audit 2022-04-22T20:04:08.138699+0000 mon.a (mon.0) 8484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:09 smithi121 bash[36902]: cluster 2022-04-22T20:04:08.729123+0000 mgr.y (mgr.34107) 5417 : cluster [DBG] pgmap v6684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:09 smithi115 bash[45495]: cluster 2022-04-22T20:04:08.729123+0000 mgr.y (mgr.34107) 5417 : cluster [DBG] pgmap v6684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:09 smithi115 bash[46934]: cluster 2022-04-22T20:04:08.729123+0000 mgr.y (mgr.34107) 5417 : cluster [DBG] pgmap v6684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:10 smithi121 bash[36902]: audit 2022-04-22T20:04:10.395237+0000 mon.a (mon.0) 8485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:10 smithi115 bash[45495]: audit 2022-04-22T20:04:10.395237+0000 mon.a (mon.0) 8485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:11.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:10 smithi115 bash[46934]: audit 2022-04-22T20:04:10.395237+0000 mon.a (mon.0) 8485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:12.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:11 smithi115 bash[45495]: cluster 2022-04-22T20:04:10.730223+0000 mgr.y (mgr.34107) 5418 : cluster [DBG] pgmap v6685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:12.233 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:11 smithi115 bash[45495]: audit 2022-04-22T20:04:11.698640+0000 mon.a (mon.0) 8486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:12.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:11 smithi115 bash[46934]: cluster 2022-04-22T20:04:10.730223+0000 mgr.y (mgr.34107) 5418 : cluster [DBG] pgmap v6685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:12.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:11 smithi115 bash[46934]: audit 2022-04-22T20:04:11.698640+0000 mon.a (mon.0) 8486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:11 smithi121 bash[36902]: cluster 2022-04-22T20:04:10.730223+0000 mgr.y (mgr.34107) 5418 : cluster [DBG] pgmap v6685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:11 smithi121 bash[36902]: audit 2022-04-22T20:04:11.698640+0000 mon.a (mon.0) 8486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:12.224Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:13 smithi121 bash[36902]: cluster 2022-04-22T20:04:12.731314+0000 mgr.y (mgr.34107) 5419 : cluster [DBG] pgmap v6686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:13 smithi121 bash[36902]: audit 2022-04-22T20:04:13.141682+0000 mon.a (mon.0) 8487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:14.103Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:13 smithi115 bash[45495]: cluster 2022-04-22T20:04:12.731314+0000 mgr.y (mgr.34107) 5419 : cluster [DBG] pgmap v6686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:13 smithi115 bash[45495]: audit 2022-04-22T20:04:13.141682+0000 mon.a (mon.0) 8487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:13 smithi115 bash[46934]: cluster 2022-04-22T20:04:12.731314+0000 mgr.y (mgr.34107) 5419 : cluster [DBG] pgmap v6686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:13 smithi115 bash[46934]: audit 2022-04-22T20:04:13.141682+0000 mon.a (mon.0) 8487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:15 smithi121 bash[36902]: cluster 2022-04-22T20:04:14.732413+0000 mgr.y (mgr.34107) 5420 : cluster [DBG] pgmap v6687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:15 smithi121 bash[36902]: audit 2022-04-22T20:04:15.098461+0000 mon.a (mon.0) 8488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:04:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:15 smithi121 bash[36902]: audit 2022-04-22T20:04:15.098921+0000 mon.a (mon.0) 8489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:04:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:15 smithi121 bash[36902]: audit 2022-04-22T20:04:15.398852+0000 mon.a (mon.0) 8490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[45495]: cluster 2022-04-22T20:04:14.732413+0000 mgr.y (mgr.34107) 5420 : cluster [DBG] pgmap v6687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[45495]: audit 2022-04-22T20:04:15.098461+0000 mon.a (mon.0) 8488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[45495]: audit 2022-04-22T20:04:15.098921+0000 mon.a (mon.0) 8489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[45495]: audit 2022-04-22T20:04:15.398852+0000 mon.a (mon.0) 8490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[46934]: cluster 2022-04-22T20:04:14.732413+0000 mgr.y (mgr.34107) 5420 : cluster [DBG] pgmap v6687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[46934]: audit 2022-04-22T20:04:15.098461+0000 mon.a (mon.0) 8488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[46934]: audit 2022-04-22T20:04:15.098921+0000 mon.a (mon.0) 8489 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:04:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:15 smithi115 bash[46934]: audit 2022-04-22T20:04:15.398852+0000 mon.a (mon.0) 8490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:16 smithi121 bash[36902]: audit 2022-04-22T20:04:16.702100+0000 mon.a (mon.0) 8491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:16 smithi115 bash[45495]: audit 2022-04-22T20:04:16.702100+0000 mon.a (mon.0) 8491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:16 smithi115 bash[46934]: audit 2022-04-22T20:04:16.702100+0000 mon.a (mon.0) 8491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:17 smithi121 bash[36902]: cluster 2022-04-22T20:04:16.733586+0000 mgr.y (mgr.34107) 5421 : cluster [DBG] pgmap v6688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:17 smithi115 bash[45495]: cluster 2022-04-22T20:04:16.733586+0000 mgr.y (mgr.34107) 5421 : cluster [DBG] pgmap v6688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:17 smithi115 bash[46934]: cluster 2022-04-22T20:04:16.733586+0000 mgr.y (mgr.34107) 5421 : cluster [DBG] pgmap v6688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:18 smithi121 bash[36902]: audit 2022-04-22T20:04:18.144634+0000 mon.a (mon.0) 8492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:18 smithi115 bash[45495]: audit 2022-04-22T20:04:18.144634+0000 mon.a (mon.0) 8492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:18 smithi115 bash[46934]: audit 2022-04-22T20:04:18.144634+0000 mon.a (mon.0) 8492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:19 smithi121 bash[36902]: cluster 2022-04-22T20:04:18.734811+0000 mgr.y (mgr.34107) 5422 : cluster [DBG] pgmap v6689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:19 smithi115 bash[45495]: cluster 2022-04-22T20:04:18.734811+0000 mgr.y (mgr.34107) 5422 : cluster [DBG] pgmap v6689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:19 smithi115 bash[46934]: cluster 2022-04-22T20:04:18.734811+0000 mgr.y (mgr.34107) 5422 : cluster [DBG] pgmap v6689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:21.116 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:04:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:20 smithi121 bash[36902]: audit 2022-04-22T20:04:20.402472+0000 mon.a (mon.0) 8493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:20 smithi115 bash[45495]: audit 2022-04-22T20:04:20.402472+0000 mon.a (mon.0) 8493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:20 smithi115 bash[46934]: audit 2022-04-22T20:04:20.402472+0000 mon.a (mon.0) 8493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:21.464 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:04:21.464 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:04:21.464 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:04:21.464 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:04:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:04:21.466 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:04:21.867 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:04:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:04:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:04:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:04:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:04:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:04:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:04:21.869 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:04:22.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[45495]: cluster 2022-04-22T20:04:20.735887+0000 mgr.y (mgr.34107) 5423 : cluster [DBG] pgmap v6690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:22.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[45495]: audit 2022-04-22T20:04:21.705003+0000 mon.a (mon.0) 8494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:22.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[45495]: audit 2022-04-22T20:04:21.858294+0000 mon.a (mon.0) 8495 : audit [DBG] from='client.? 172.21.15.115:0/142993248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:22.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[46934]: cluster 2022-04-22T20:04:20.735887+0000 mgr.y (mgr.34107) 5423 : cluster [DBG] pgmap v6690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:22.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[46934]: audit 2022-04-22T20:04:21.705003+0000 mon.a (mon.0) 8494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:22.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:21 smithi115 bash[46934]: audit 2022-04-22T20:04:21.858294+0000 mon.a (mon.0) 8495 : audit [DBG] from='client.? 172.21.15.115:0/142993248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:21 smithi121 bash[36902]: cluster 2022-04-22T20:04:20.735887+0000 mgr.y (mgr.34107) 5423 : cluster [DBG] pgmap v6690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:21 smithi121 bash[36902]: audit 2022-04-22T20:04:21.705003+0000 mon.a (mon.0) 8494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:21 smithi121 bash[36902]: audit 2022-04-22T20:04:21.858294+0000 mon.a (mon.0) 8495 : audit [DBG] from='client.? 172.21.15.115:0/142993248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:22.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:22 smithi121 bash[36902]: audit 2022-04-22T20:04:21.092096+0000 mgr.y (mgr.34107) 5424 : audit [DBG] from='client.50974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:22 smithi121 bash[36902]: audit 2022-04-22T20:04:21.437995+0000 mgr.y (mgr.34107) 5425 : audit [DBG] from='client.50980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:22 smithi115 bash[45495]: audit 2022-04-22T20:04:21.092096+0000 mgr.y (mgr.34107) 5424 : audit [DBG] from='client.50974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:22 smithi115 bash[45495]: audit 2022-04-22T20:04:21.437995+0000 mgr.y (mgr.34107) 5425 : audit [DBG] from='client.50980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:22 smithi115 bash[46934]: audit 2022-04-22T20:04:21.092096+0000 mgr.y (mgr.34107) 5424 : audit [DBG] from='client.50974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:22 smithi115 bash[46934]: audit 2022-04-22T20:04:21.437995+0000 mgr.y (mgr.34107) 5425 : audit [DBG] from='client.50980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:23 smithi121 bash[36902]: cluster 2022-04-22T20:04:22.736927+0000 mgr.y (mgr.34107) 5426 : cluster [DBG] pgmap v6691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:23 smithi121 bash[36902]: audit 2022-04-22T20:04:23.147555+0000 mon.a (mon.0) 8496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:23 smithi115 bash[46934]: cluster 2022-04-22T20:04:22.736927+0000 mgr.y (mgr.34107) 5426 : cluster [DBG] pgmap v6691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:23 smithi115 bash[46934]: audit 2022-04-22T20:04:23.147555+0000 mon.a (mon.0) 8496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:24.102Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:23 smithi115 bash[45495]: cluster 2022-04-22T20:04:22.736927+0000 mgr.y (mgr.34107) 5426 : cluster [DBG] pgmap v6691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:24.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:23 smithi115 bash[45495]: audit 2022-04-22T20:04:23.147555+0000 mon.a (mon.0) 8496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:24 smithi121 bash[36902]: cluster 2022-04-22T20:04:24.738156+0000 mgr.y (mgr.34107) 5427 : cluster [DBG] pgmap v6692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:24 smithi115 bash[46934]: cluster 2022-04-22T20:04:24.738156+0000 mgr.y (mgr.34107) 5427 : cluster [DBG] pgmap v6692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:24 smithi115 bash[45495]: cluster 2022-04-22T20:04:24.738156+0000 mgr.y (mgr.34107) 5427 : cluster [DBG] pgmap v6692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:25 smithi121 bash[36902]: audit 2022-04-22T20:04:25.406111+0000 mon.a (mon.0) 8497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:25 smithi115 bash[45495]: audit 2022-04-22T20:04:25.406111+0000 mon.a (mon.0) 8497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:25 smithi115 bash[46934]: audit 2022-04-22T20:04:25.406111+0000 mon.a (mon.0) 8497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:26 smithi121 bash[36902]: audit 2022-04-22T20:04:26.708153+0000 mon.a (mon.0) 8498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:26 smithi121 bash[36902]: cluster 2022-04-22T20:04:26.739398+0000 mgr.y (mgr.34107) 5428 : cluster [DBG] pgmap v6693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:26 smithi115 bash[45495]: audit 2022-04-22T20:04:26.708153+0000 mon.a (mon.0) 8498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:26 smithi115 bash[45495]: cluster 2022-04-22T20:04:26.739398+0000 mgr.y (mgr.34107) 5428 : cluster [DBG] pgmap v6693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:26 smithi115 bash[46934]: audit 2022-04-22T20:04:26.708153+0000 mon.a (mon.0) 8498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:26 smithi115 bash[46934]: cluster 2022-04-22T20:04:26.739398+0000 mgr.y (mgr.34107) 5428 : cluster [DBG] pgmap v6693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:28 smithi121 bash[36902]: audit 2022-04-22T20:04:28.150467+0000 mon.a (mon.0) 8499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:28 smithi115 bash[45495]: audit 2022-04-22T20:04:28.150467+0000 mon.a (mon.0) 8499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:28 smithi115 bash[46934]: audit 2022-04-22T20:04:28.150467+0000 mon.a (mon.0) 8499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:29 smithi121 bash[36902]: cluster 2022-04-22T20:04:28.740592+0000 mgr.y (mgr.34107) 5429 : cluster [DBG] pgmap v6694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:29 smithi115 bash[45495]: cluster 2022-04-22T20:04:28.740592+0000 mgr.y (mgr.34107) 5429 : cluster [DBG] pgmap v6694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:29 smithi115 bash[46934]: cluster 2022-04-22T20:04:28.740592+0000 mgr.y (mgr.34107) 5429 : cluster [DBG] pgmap v6694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:30 smithi121 bash[36902]: audit 2022-04-22T20:04:30.409822+0000 mon.a (mon.0) 8500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:30 smithi115 bash[45495]: audit 2022-04-22T20:04:30.409822+0000 mon.a (mon.0) 8500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:30 smithi115 bash[46934]: audit 2022-04-22T20:04:30.409822+0000 mon.a (mon.0) 8500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:31 smithi121 bash[36902]: cluster 2022-04-22T20:04:30.741820+0000 mgr.y (mgr.34107) 5430 : cluster [DBG] pgmap v6695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:31 smithi115 bash[45495]: cluster 2022-04-22T20:04:30.741820+0000 mgr.y (mgr.34107) 5430 : cluster [DBG] pgmap v6695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:31 smithi115 bash[46934]: cluster 2022-04-22T20:04:30.741820+0000 mgr.y (mgr.34107) 5430 : cluster [DBG] pgmap v6695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:32.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:32 smithi121 bash[36902]: audit 2022-04-22T20:04:31.711382+0000 mon.a (mon.0) 8501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:32 smithi115 bash[45495]: audit 2022-04-22T20:04:31.711382+0000 mon.a (mon.0) 8501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:32 smithi115 bash[46934]: audit 2022-04-22T20:04:31.711382+0000 mon.a (mon.0) 8501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:33 smithi121 bash[36902]: cluster 2022-04-22T20:04:32.742952+0000 mgr.y (mgr.34107) 5431 : cluster [DBG] pgmap v6696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:33 smithi121 bash[36902]: audit 2022-04-22T20:04:33.153697+0000 mon.a (mon.0) 8502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:33 smithi115 bash[45495]: cluster 2022-04-22T20:04:32.742952+0000 mgr.y (mgr.34107) 5431 : cluster [DBG] pgmap v6696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:33 smithi115 bash[45495]: audit 2022-04-22T20:04:33.153697+0000 mon.a (mon.0) 8502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:33 smithi115 bash[46934]: cluster 2022-04-22T20:04:32.742952+0000 mgr.y (mgr.34107) 5431 : cluster [DBG] pgmap v6696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:33 smithi115 bash[46934]: audit 2022-04-22T20:04:33.153697+0000 mon.a (mon.0) 8502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:34.105Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:35 smithi121 bash[36902]: cluster 2022-04-22T20:04:34.744044+0000 mgr.y (mgr.34107) 5432 : cluster [DBG] pgmap v6697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:35 smithi121 bash[36902]: audit 2022-04-22T20:04:35.413550+0000 mon.a (mon.0) 8503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:35 smithi115 bash[45495]: cluster 2022-04-22T20:04:34.744044+0000 mgr.y (mgr.34107) 5432 : cluster [DBG] pgmap v6697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:35 smithi115 bash[45495]: audit 2022-04-22T20:04:35.413550+0000 mon.a (mon.0) 8503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:35 smithi115 bash[46934]: cluster 2022-04-22T20:04:34.744044+0000 mgr.y (mgr.34107) 5432 : cluster [DBG] pgmap v6697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:35 smithi115 bash[46934]: audit 2022-04-22T20:04:35.413550+0000 mon.a (mon.0) 8503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:36 smithi121 bash[36902]: audit 2022-04-22T20:04:36.714776+0000 mon.a (mon.0) 8504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:36 smithi115 bash[45495]: audit 2022-04-22T20:04:36.714776+0000 mon.a (mon.0) 8504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:36 smithi115 bash[46934]: audit 2022-04-22T20:04:36.714776+0000 mon.a (mon.0) 8504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:37 smithi121 bash[36902]: cluster 2022-04-22T20:04:36.745108+0000 mgr.y (mgr.34107) 5433 : cluster [DBG] pgmap v6698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:37 smithi115 bash[46934]: cluster 2022-04-22T20:04:36.745108+0000 mgr.y (mgr.34107) 5433 : cluster [DBG] pgmap v6698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:37 smithi115 bash[45495]: cluster 2022-04-22T20:04:36.745108+0000 mgr.y (mgr.34107) 5433 : cluster [DBG] pgmap v6698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:38 smithi121 bash[36902]: audit 2022-04-22T20:04:38.156773+0000 mon.a (mon.0) 8505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:38 smithi115 bash[46934]: audit 2022-04-22T20:04:38.156773+0000 mon.a (mon.0) 8505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:38 smithi115 bash[45495]: audit 2022-04-22T20:04:38.156773+0000 mon.a (mon.0) 8505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:39 smithi121 bash[36902]: cluster 2022-04-22T20:04:38.746336+0000 mgr.y (mgr.34107) 5434 : cluster [DBG] pgmap v6699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:39 smithi115 bash[45495]: cluster 2022-04-22T20:04:38.746336+0000 mgr.y (mgr.34107) 5434 : cluster [DBG] pgmap v6699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:39 smithi115 bash[46934]: cluster 2022-04-22T20:04:38.746336+0000 mgr.y (mgr.34107) 5434 : cluster [DBG] pgmap v6699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:40 smithi121 bash[36902]: audit 2022-04-22T20:04:40.417149+0000 mon.a (mon.0) 8506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:40 smithi115 bash[46934]: audit 2022-04-22T20:04:40.417149+0000 mon.a (mon.0) 8506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:40 smithi115 bash[45495]: audit 2022-04-22T20:04:40.417149+0000 mon.a (mon.0) 8506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:42.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:41 smithi115 bash[45495]: cluster 2022-04-22T20:04:40.747510+0000 mgr.y (mgr.34107) 5435 : cluster [DBG] pgmap v6700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:42.263 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:41 smithi115 bash[45495]: audit 2022-04-22T20:04:41.718098+0000 mon.a (mon.0) 8507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:42.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:41 smithi115 bash[46934]: cluster 2022-04-22T20:04:40.747510+0000 mgr.y (mgr.34107) 5435 : cluster [DBG] pgmap v6700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:42.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:41 smithi115 bash[46934]: audit 2022-04-22T20:04:41.718098+0000 mon.a (mon.0) 8507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:41 smithi121 bash[36902]: cluster 2022-04-22T20:04:40.747510+0000 mgr.y (mgr.34107) 5435 : cluster [DBG] pgmap v6700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:41 smithi121 bash[36902]: audit 2022-04-22T20:04:41.718098+0000 mon.a (mon.0) 8507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:42.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:44.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:43 smithi115 bash[45495]: cluster 2022-04-22T20:04:42.748557+0000 mgr.y (mgr.34107) 5436 : cluster [DBG] pgmap v6701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:44.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:43 smithi115 bash[45495]: audit 2022-04-22T20:04:43.159800+0000 mon.a (mon.0) 8508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:44.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:43 smithi115 bash[46934]: cluster 2022-04-22T20:04:42.748557+0000 mgr.y (mgr.34107) 5436 : cluster [DBG] pgmap v6701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:44.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:43 smithi115 bash[46934]: audit 2022-04-22T20:04:43.159800+0000 mon.a (mon.0) 8508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:43 smithi121 bash[36902]: cluster 2022-04-22T20:04:42.748557+0000 mgr.y (mgr.34107) 5436 : cluster [DBG] pgmap v6701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:43 smithi121 bash[36902]: audit 2022-04-22T20:04:43.159800+0000 mon.a (mon.0) 8508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:44.106Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:45 smithi121 bash[36902]: cluster 2022-04-22T20:04:44.749652+0000 mgr.y (mgr.34107) 5437 : cluster [DBG] pgmap v6702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:45 smithi121 bash[36902]: audit 2022-04-22T20:04:45.420753+0000 mon.a (mon.0) 8509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:45 smithi115 bash[45495]: cluster 2022-04-22T20:04:44.749652+0000 mgr.y (mgr.34107) 5437 : cluster [DBG] pgmap v6702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:45 smithi115 bash[45495]: audit 2022-04-22T20:04:45.420753+0000 mon.a (mon.0) 8509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:45 smithi115 bash[46934]: cluster 2022-04-22T20:04:44.749652+0000 mgr.y (mgr.34107) 5437 : cluster [DBG] pgmap v6702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:45 smithi115 bash[46934]: audit 2022-04-22T20:04:45.420753+0000 mon.a (mon.0) 8509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:46 smithi121 bash[36902]: audit 2022-04-22T20:04:46.721341+0000 mon.a (mon.0) 8510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:46 smithi115 bash[46934]: audit 2022-04-22T20:04:46.721341+0000 mon.a (mon.0) 8510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:46 smithi115 bash[45495]: audit 2022-04-22T20:04:46.721341+0000 mon.a (mon.0) 8510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:47 smithi121 bash[36902]: cluster 2022-04-22T20:04:46.750649+0000 mgr.y (mgr.34107) 5438 : cluster [DBG] pgmap v6703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:47 smithi115 bash[45495]: cluster 2022-04-22T20:04:46.750649+0000 mgr.y (mgr.34107) 5438 : cluster [DBG] pgmap v6703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:47 smithi115 bash[46934]: cluster 2022-04-22T20:04:46.750649+0000 mgr.y (mgr.34107) 5438 : cluster [DBG] pgmap v6703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:48 smithi121 bash[36902]: audit 2022-04-22T20:04:48.162799+0000 mon.a (mon.0) 8511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:48 smithi115 bash[45495]: audit 2022-04-22T20:04:48.162799+0000 mon.a (mon.0) 8511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:48 smithi115 bash[46934]: audit 2022-04-22T20:04:48.162799+0000 mon.a (mon.0) 8511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:49 smithi121 bash[36902]: cluster 2022-04-22T20:04:48.751948+0000 mgr.y (mgr.34107) 5439 : cluster [DBG] pgmap v6704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:49 smithi115 bash[45495]: cluster 2022-04-22T20:04:48.751948+0000 mgr.y (mgr.34107) 5439 : cluster [DBG] pgmap v6704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:49 smithi115 bash[46934]: cluster 2022-04-22T20:04:48.751948+0000 mgr.y (mgr.34107) 5439 : cluster [DBG] pgmap v6704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:50 smithi121 bash[36902]: audit 2022-04-22T20:04:50.424395+0000 mon.a (mon.0) 8512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:50 smithi115 bash[45495]: audit 2022-04-22T20:04:50.424395+0000 mon.a (mon.0) 8512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:50 smithi115 bash[46934]: audit 2022-04-22T20:04:50.424395+0000 mon.a (mon.0) 8512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:51.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[46934]: cluster 2022-04-22T20:04:50.752955+0000 mgr.y (mgr.34107) 5440 : cluster [DBG] pgmap v6705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:51.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[46934]: audit 2022-04-22T20:04:51.597989+0000 mon.a (mon.0) 8513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:04:51.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[46934]: audit 2022-04-22T20:04:51.601114+0000 mon.a (mon.0) 8514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:04:51.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[46934]: audit 2022-04-22T20:04:51.603226+0000 mon.a (mon.0) 8515 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:04:51.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[46934]: audit 2022-04-22T20:04:51.724403+0000 mon.a (mon.0) 8516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:51.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[45495]: cluster 2022-04-22T20:04:50.752955+0000 mgr.y (mgr.34107) 5440 : cluster [DBG] pgmap v6705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:51.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[45495]: audit 2022-04-22T20:04:51.597989+0000 mon.a (mon.0) 8513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:04:51.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[45495]: audit 2022-04-22T20:04:51.601114+0000 mon.a (mon.0) 8514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:04:51.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[45495]: audit 2022-04-22T20:04:51.603226+0000 mon.a (mon.0) 8515 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:04:51.950 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:51 smithi115 bash[45495]: audit 2022-04-22T20:04:51.724403+0000 mon.a (mon.0) 8516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:51.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:51 smithi121 bash[36902]: cluster 2022-04-22T20:04:50.752955+0000 mgr.y (mgr.34107) 5440 : cluster [DBG] pgmap v6705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:51.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:51 smithi121 bash[36902]: audit 2022-04-22T20:04:51.597989+0000 mon.a (mon.0) 8513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:04:51.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:51 smithi121 bash[36902]: audit 2022-04-22T20:04:51.601114+0000 mon.a (mon.0) 8514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:04:51.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:51 smithi121 bash[36902]: audit 2022-04-22T20:04:51.603226+0000 mon.a (mon.0) 8515 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:04:51.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:51 smithi121 bash[36902]: audit 2022-04-22T20:04:51.724403+0000 mon.a (mon.0) 8516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:52.251 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:04:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:52.227Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:52.592 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:04:52.592 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:04:52.592 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:04:52.593 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:04:52.594 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:53.004 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:04:53.005 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:04:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:52 smithi121 bash[36902]: audit 2022-04-22T20:04:51.957726+0000 mon.a (mon.0) 8517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:04:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:52 smithi121 bash[36902]: audit 2022-04-22T20:04:52.221285+0000 mgr.y (mgr.34107) 5441 : audit [DBG] from='client.41058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:52 smithi121 bash[36902]: audit 2022-04-22T20:04:52.564472+0000 mgr.y (mgr.34107) 5442 : audit [DBG] from='client.50998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:52 smithi121 bash[36902]: cluster 2022-04-22T20:04:52.753990+0000 mgr.y (mgr.34107) 5443 : cluster [DBG] pgmap v6706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[45495]: audit 2022-04-22T20:04:51.957726+0000 mon.a (mon.0) 8517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:04:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[45495]: audit 2022-04-22T20:04:52.221285+0000 mgr.y (mgr.34107) 5441 : audit [DBG] from='client.41058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[45495]: audit 2022-04-22T20:04:52.564472+0000 mgr.y (mgr.34107) 5442 : audit [DBG] from='client.50998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[45495]: cluster 2022-04-22T20:04:52.753990+0000 mgr.y (mgr.34107) 5443 : cluster [DBG] pgmap v6706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[46934]: audit 2022-04-22T20:04:51.957726+0000 mon.a (mon.0) 8517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[46934]: audit 2022-04-22T20:04:52.221285+0000 mgr.y (mgr.34107) 5441 : audit [DBG] from='client.41058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[46934]: audit 2022-04-22T20:04:52.564472+0000 mgr.y (mgr.34107) 5442 : audit [DBG] from='client.50998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:04:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:52 smithi115 bash[46934]: cluster 2022-04-22T20:04:52.753990+0000 mgr.y (mgr.34107) 5443 : cluster [DBG] pgmap v6706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:53 smithi121 bash[36902]: audit 2022-04-22T20:04:52.995307+0000 mon.c (mon.1) 225 : audit [DBG] from='client.? 172.21.15.115:0/1260523227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:53 smithi121 bash[36902]: audit 2022-04-22T20:04:53.165848+0000 mon.a (mon.0) 8518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:04:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:04:54.108Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:04:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:53 smithi115 bash[45495]: audit 2022-04-22T20:04:52.995307+0000 mon.c (mon.1) 225 : audit [DBG] from='client.? 172.21.15.115:0/1260523227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:53 smithi115 bash[45495]: audit 2022-04-22T20:04:53.165848+0000 mon.a (mon.0) 8518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:53 smithi115 bash[46934]: audit 2022-04-22T20:04:52.995307+0000 mon.c (mon.1) 225 : audit [DBG] from='client.? 172.21.15.115:0/1260523227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:04:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:53 smithi115 bash[46934]: audit 2022-04-22T20:04:53.165848+0000 mon.a (mon.0) 8518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:54 smithi121 bash[36902]: cluster 2022-04-22T20:04:54.755025+0000 mgr.y (mgr.34107) 5444 : cluster [DBG] pgmap v6707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:54 smithi115 bash[46934]: cluster 2022-04-22T20:04:54.755025+0000 mgr.y (mgr.34107) 5444 : cluster [DBG] pgmap v6707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:54 smithi115 bash[45495]: cluster 2022-04-22T20:04:54.755025+0000 mgr.y (mgr.34107) 5444 : cluster [DBG] pgmap v6707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:55 smithi121 bash[36902]: audit 2022-04-22T20:04:55.428142+0000 mon.a (mon.0) 8519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:55 smithi115 bash[45495]: audit 2022-04-22T20:04:55.428142+0000 mon.a (mon.0) 8519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:55 smithi115 bash[46934]: audit 2022-04-22T20:04:55.428142+0000 mon.a (mon.0) 8519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:04:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:57 smithi121 bash[36902]: audit 2022-04-22T20:04:56.727296+0000 mon.a (mon.0) 8520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:57 smithi121 bash[36902]: cluster 2022-04-22T20:04:56.756114+0000 mgr.y (mgr.34107) 5445 : cluster [DBG] pgmap v6708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:56 smithi115 bash[45495]: audit 2022-04-22T20:04:56.727296+0000 mon.a (mon.0) 8520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:56 smithi115 bash[45495]: cluster 2022-04-22T20:04:56.756114+0000 mgr.y (mgr.34107) 5445 : cluster [DBG] pgmap v6708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:56 smithi115 bash[46934]: audit 2022-04-22T20:04:56.727296+0000 mon.a (mon.0) 8520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:04:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:56 smithi115 bash[46934]: cluster 2022-04-22T20:04:56.756114+0000 mgr.y (mgr.34107) 5445 : cluster [DBG] pgmap v6708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:04:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:58 smithi121 bash[36902]: audit 2022-04-22T20:04:58.169069+0000 mon.a (mon.0) 8521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:58 smithi115 bash[46934]: audit 2022-04-22T20:04:58.169069+0000 mon.a (mon.0) 8521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:04:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:58 smithi115 bash[45495]: audit 2022-04-22T20:04:58.169069+0000 mon.a (mon.0) 8521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:04:59 smithi121 bash[36902]: cluster 2022-04-22T20:04:58.757285+0000 mgr.y (mgr.34107) 5446 : cluster [DBG] pgmap v6709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:04:59 smithi115 bash[45495]: cluster 2022-04-22T20:04:58.757285+0000 mgr.y (mgr.34107) 5446 : cluster [DBG] pgmap v6709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:00.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:04:59 smithi115 bash[46934]: cluster 2022-04-22T20:04:58.757285+0000 mgr.y (mgr.34107) 5446 : cluster [DBG] pgmap v6709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:00 smithi121 bash[36902]: audit 2022-04-22T20:05:00.431614+0000 mon.a (mon.0) 8522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:00 smithi115 bash[45495]: audit 2022-04-22T20:05:00.431614+0000 mon.a (mon.0) 8522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:00 smithi115 bash[46934]: audit 2022-04-22T20:05:00.431614+0000 mon.a (mon.0) 8522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:01.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:01 smithi115 bash[46934]: cluster 2022-04-22T20:05:00.758271+0000 mgr.y (mgr.34107) 5447 : cluster [DBG] pgmap v6710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:01.894 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:01 smithi115 bash[45495]: cluster 2022-04-22T20:05:00.758271+0000 mgr.y (mgr.34107) 5447 : cluster [DBG] pgmap v6710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:01 smithi121 bash[36902]: cluster 2022-04-22T20:05:00.758271+0000 mgr.y (mgr.34107) 5447 : cluster [DBG] pgmap v6710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:02.235 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:02.226Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:02 smithi121 bash[36902]: audit 2022-04-22T20:05:01.730330+0000 mon.a (mon.0) 8523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:02 smithi115 bash[45495]: audit 2022-04-22T20:05:01.730330+0000 mon.a (mon.0) 8523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:02 smithi115 bash[46934]: audit 2022-04-22T20:05:01.730330+0000 mon.a (mon.0) 8523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:03 smithi121 bash[36902]: cluster 2022-04-22T20:05:02.759408+0000 mgr.y (mgr.34107) 5448 : cluster [DBG] pgmap v6711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:03 smithi121 bash[36902]: audit 2022-04-22T20:05:03.172306+0000 mon.a (mon.0) 8524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:03 smithi115 bash[45495]: cluster 2022-04-22T20:05:02.759408+0000 mgr.y (mgr.34107) 5448 : cluster [DBG] pgmap v6711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:03 smithi115 bash[45495]: audit 2022-04-22T20:05:03.172306+0000 mon.a (mon.0) 8524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:03 smithi115 bash[46934]: cluster 2022-04-22T20:05:02.759408+0000 mgr.y (mgr.34107) 5448 : cluster [DBG] pgmap v6711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:03 smithi115 bash[46934]: audit 2022-04-22T20:05:03.172306+0000 mon.a (mon.0) 8524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:04.108Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:05 smithi121 bash[36902]: cluster 2022-04-22T20:05:04.760531+0000 mgr.y (mgr.34107) 5449 : cluster [DBG] pgmap v6712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:05 smithi121 bash[36902]: audit 2022-04-22T20:05:05.435276+0000 mon.a (mon.0) 8525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:05 smithi115 bash[45495]: cluster 2022-04-22T20:05:04.760531+0000 mgr.y (mgr.34107) 5449 : cluster [DBG] pgmap v6712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:05 smithi115 bash[45495]: audit 2022-04-22T20:05:05.435276+0000 mon.a (mon.0) 8525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:05 smithi115 bash[46934]: cluster 2022-04-22T20:05:04.760531+0000 mgr.y (mgr.34107) 5449 : cluster [DBG] pgmap v6712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:05 smithi115 bash[46934]: audit 2022-04-22T20:05:05.435276+0000 mon.a (mon.0) 8525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:06 smithi121 bash[36902]: audit 2022-04-22T20:05:06.733476+0000 mon.a (mon.0) 8526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:06 smithi115 bash[45495]: audit 2022-04-22T20:05:06.733476+0000 mon.a (mon.0) 8526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:06 smithi115 bash[46934]: audit 2022-04-22T20:05:06.733476+0000 mon.a (mon.0) 8526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:07 smithi121 bash[36902]: cluster 2022-04-22T20:05:06.761661+0000 mgr.y (mgr.34107) 5450 : cluster [DBG] pgmap v6713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:07 smithi115 bash[45495]: cluster 2022-04-22T20:05:06.761661+0000 mgr.y (mgr.34107) 5450 : cluster [DBG] pgmap v6713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:07 smithi115 bash[46934]: cluster 2022-04-22T20:05:06.761661+0000 mgr.y (mgr.34107) 5450 : cluster [DBG] pgmap v6713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:08 smithi121 bash[36902]: audit 2022-04-22T20:05:08.175356+0000 mon.a (mon.0) 8527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:08 smithi115 bash[45495]: audit 2022-04-22T20:05:08.175356+0000 mon.a (mon.0) 8527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:08 smithi115 bash[46934]: audit 2022-04-22T20:05:08.175356+0000 mon.a (mon.0) 8527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:09 smithi121 bash[36902]: cluster 2022-04-22T20:05:08.762757+0000 mgr.y (mgr.34107) 5451 : cluster [DBG] pgmap v6714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:09 smithi115 bash[45495]: cluster 2022-04-22T20:05:08.762757+0000 mgr.y (mgr.34107) 5451 : cluster [DBG] pgmap v6714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:09 smithi115 bash[46934]: cluster 2022-04-22T20:05:08.762757+0000 mgr.y (mgr.34107) 5451 : cluster [DBG] pgmap v6714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:10 smithi121 bash[36902]: audit 2022-04-22T20:05:10.438997+0000 mon.a (mon.0) 8528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:10 smithi115 bash[45495]: audit 2022-04-22T20:05:10.438997+0000 mon.a (mon.0) 8528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:10 smithi115 bash[46934]: audit 2022-04-22T20:05:10.438997+0000 mon.a (mon.0) 8528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:12.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:11 smithi115 bash[46934]: cluster 2022-04-22T20:05:10.763841+0000 mgr.y (mgr.34107) 5452 : cluster [DBG] pgmap v6715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:12.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:11 smithi115 bash[46934]: audit 2022-04-22T20:05:11.736668+0000 mon.a (mon.0) 8529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:12.236 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:11 smithi115 bash[45495]: cluster 2022-04-22T20:05:10.763841+0000 mgr.y (mgr.34107) 5452 : cluster [DBG] pgmap v6715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:12.236 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:11 smithi115 bash[45495]: audit 2022-04-22T20:05:11.736668+0000 mon.a (mon.0) 8529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:11 smithi121 bash[36902]: cluster 2022-04-22T20:05:10.763841+0000 mgr.y (mgr.34107) 5452 : cluster [DBG] pgmap v6715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:11 smithi121 bash[36902]: audit 2022-04-22T20:05:11.736668+0000 mon.a (mon.0) 8529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:12.227Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:14.117 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:13 smithi115 bash[45495]: cluster 2022-04-22T20:05:12.764919+0000 mgr.y (mgr.34107) 5453 : cluster [DBG] pgmap v6716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:14.117 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:13 smithi115 bash[45495]: audit 2022-04-22T20:05:13.178504+0000 mon.a (mon.0) 8530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:13 smithi115 bash[46934]: cluster 2022-04-22T20:05:12.764919+0000 mgr.y (mgr.34107) 5453 : cluster [DBG] pgmap v6716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:13 smithi115 bash[46934]: audit 2022-04-22T20:05:13.178504+0000 mon.a (mon.0) 8530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:13 smithi121 bash[36902]: cluster 2022-04-22T20:05:12.764919+0000 mgr.y (mgr.34107) 5453 : cluster [DBG] pgmap v6716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:13 smithi121 bash[36902]: audit 2022-04-22T20:05:13.178504+0000 mon.a (mon.0) 8530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:14.108Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:15 smithi121 bash[36902]: cluster 2022-04-22T20:05:14.766019+0000 mgr.y (mgr.34107) 5454 : cluster [DBG] pgmap v6717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:15 smithi121 bash[36902]: audit 2022-04-22T20:05:15.109639+0000 mon.a (mon.0) 8531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:05:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:15 smithi121 bash[36902]: audit 2022-04-22T20:05:15.111573+0000 mon.a (mon.0) 8532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:05:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:15 smithi121 bash[36902]: audit 2022-04-22T20:05:15.442632+0000 mon.a (mon.0) 8533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[45495]: cluster 2022-04-22T20:05:14.766019+0000 mgr.y (mgr.34107) 5454 : cluster [DBG] pgmap v6717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[45495]: audit 2022-04-22T20:05:15.109639+0000 mon.a (mon.0) 8531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:05:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[45495]: audit 2022-04-22T20:05:15.111573+0000 mon.a (mon.0) 8532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:05:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[45495]: audit 2022-04-22T20:05:15.442632+0000 mon.a (mon.0) 8533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[46934]: cluster 2022-04-22T20:05:14.766019+0000 mgr.y (mgr.34107) 5454 : cluster [DBG] pgmap v6717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[46934]: audit 2022-04-22T20:05:15.109639+0000 mon.a (mon.0) 8531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:05:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[46934]: audit 2022-04-22T20:05:15.111573+0000 mon.a (mon.0) 8532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:05:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:15 smithi115 bash[46934]: audit 2022-04-22T20:05:15.442632+0000 mon.a (mon.0) 8533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:16 smithi121 bash[36902]: audit 2022-04-22T20:05:16.739816+0000 mon.a (mon.0) 8534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:17.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:16 smithi115 bash[45495]: audit 2022-04-22T20:05:16.739816+0000 mon.a (mon.0) 8534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:16 smithi115 bash[46934]: audit 2022-04-22T20:05:16.739816+0000 mon.a (mon.0) 8534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:17 smithi121 bash[36902]: cluster 2022-04-22T20:05:16.767081+0000 mgr.y (mgr.34107) 5455 : cluster [DBG] pgmap v6718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:17 smithi115 bash[46934]: cluster 2022-04-22T20:05:16.767081+0000 mgr.y (mgr.34107) 5455 : cluster [DBG] pgmap v6718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:17 smithi115 bash[45495]: cluster 2022-04-22T20:05:16.767081+0000 mgr.y (mgr.34107) 5455 : cluster [DBG] pgmap v6718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:18 smithi121 bash[36902]: audit 2022-04-22T20:05:18.181437+0000 mon.a (mon.0) 8535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:18 smithi115 bash[45495]: audit 2022-04-22T20:05:18.181437+0000 mon.a (mon.0) 8535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:18 smithi115 bash[46934]: audit 2022-04-22T20:05:18.181437+0000 mon.a (mon.0) 8535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:19 smithi121 bash[36902]: cluster 2022-04-22T20:05:18.768248+0000 mgr.y (mgr.34107) 5456 : cluster [DBG] pgmap v6719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:19 smithi115 bash[45495]: cluster 2022-04-22T20:05:18.768248+0000 mgr.y (mgr.34107) 5456 : cluster [DBG] pgmap v6719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:19 smithi115 bash[46934]: cluster 2022-04-22T20:05:18.768248+0000 mgr.y (mgr.34107) 5456 : cluster [DBG] pgmap v6719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:20 smithi121 bash[36902]: audit 2022-04-22T20:05:20.446237+0000 mon.a (mon.0) 8536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:20 smithi115 bash[46934]: audit 2022-04-22T20:05:20.446237+0000 mon.a (mon.0) 8536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:20 smithi115 bash[45495]: audit 2022-04-22T20:05:20.446237+0000 mon.a (mon.0) 8536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:22.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:21 smithi115 bash[46934]: cluster 2022-04-22T20:05:20.769361+0000 mgr.y (mgr.34107) 5457 : cluster [DBG] pgmap v6720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:22.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:21 smithi115 bash[46934]: audit 2022-04-22T20:05:21.742849+0000 mon.a (mon.0) 8537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:22.244 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:21 smithi115 bash[45495]: cluster 2022-04-22T20:05:20.769361+0000 mgr.y (mgr.34107) 5457 : cluster [DBG] pgmap v6720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:22.244 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:21 smithi115 bash[45495]: audit 2022-04-22T20:05:21.742849+0000 mon.a (mon.0) 8537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:21 smithi121 bash[36902]: cluster 2022-04-22T20:05:20.769361+0000 mgr.y (mgr.34107) 5457 : cluster [DBG] pgmap v6720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:21 smithi121 bash[36902]: audit 2022-04-22T20:05:21.742849+0000 mon.a (mon.0) 8537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:22.229Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:23.383 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:05:23.722 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:05:23.723 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:05:23.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:05:23.724 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:05:24.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:05:24.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:24.139 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:05:24.139 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:05:24.139 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:05:24.140 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:05:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:23 smithi121 bash[36902]: cluster 2022-04-22T20:05:22.770451+0000 mgr.y (mgr.34107) 5458 : cluster [DBG] pgmap v6721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:23 smithi121 bash[36902]: audit 2022-04-22T20:05:23.184508+0000 mon.a (mon.0) 8538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:23 smithi115 bash[45495]: cluster 2022-04-22T20:05:22.770451+0000 mgr.y (mgr.34107) 5458 : cluster [DBG] pgmap v6721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:23 smithi115 bash[45495]: audit 2022-04-22T20:05:23.184508+0000 mon.a (mon.0) 8538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:24.109Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:23 smithi115 bash[46934]: cluster 2022-04-22T20:05:22.770451+0000 mgr.y (mgr.34107) 5458 : cluster [DBG] pgmap v6721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:23 smithi115 bash[46934]: audit 2022-04-22T20:05:23.184508+0000 mon.a (mon.0) 8538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:24 smithi121 bash[36902]: audit 2022-04-22T20:05:23.358445+0000 mgr.y (mgr.34107) 5459 : audit [DBG] from='client.41076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:24 smithi121 bash[36902]: audit 2022-04-22T20:05:23.697980+0000 mgr.y (mgr.34107) 5460 : audit [DBG] from='client.51016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:24 smithi121 bash[36902]: audit 2022-04-22T20:05:24.128202+0000 mon.a (mon.0) 8539 : audit [DBG] from='client.? 172.21.15.115:0/719143722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[46934]: audit 2022-04-22T20:05:23.358445+0000 mgr.y (mgr.34107) 5459 : audit [DBG] from='client.41076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[46934]: audit 2022-04-22T20:05:23.697980+0000 mgr.y (mgr.34107) 5460 : audit [DBG] from='client.51016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[46934]: audit 2022-04-22T20:05:24.128202+0000 mon.a (mon.0) 8539 : audit [DBG] from='client.? 172.21.15.115:0/719143722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[45495]: audit 2022-04-22T20:05:23.358445+0000 mgr.y (mgr.34107) 5459 : audit [DBG] from='client.41076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[45495]: audit 2022-04-22T20:05:23.697980+0000 mgr.y (mgr.34107) 5460 : audit [DBG] from='client.51016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:25.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:24 smithi115 bash[45495]: audit 2022-04-22T20:05:24.128202+0000 mon.a (mon.0) 8539 : audit [DBG] from='client.? 172.21.15.115:0/719143722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:25 smithi121 bash[36902]: cluster 2022-04-22T20:05:24.771552+0000 mgr.y (mgr.34107) 5461 : cluster [DBG] pgmap v6722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:25 smithi121 bash[36902]: audit 2022-04-22T20:05:25.449940+0000 mon.a (mon.0) 8540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:25 smithi115 bash[45495]: cluster 2022-04-22T20:05:24.771552+0000 mgr.y (mgr.34107) 5461 : cluster [DBG] pgmap v6722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:25 smithi115 bash[45495]: audit 2022-04-22T20:05:25.449940+0000 mon.a (mon.0) 8540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:25 smithi115 bash[46934]: cluster 2022-04-22T20:05:24.771552+0000 mgr.y (mgr.34107) 5461 : cluster [DBG] pgmap v6722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:25 smithi115 bash[46934]: audit 2022-04-22T20:05:25.449940+0000 mon.a (mon.0) 8540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:26 smithi121 bash[36902]: audit 2022-04-22T20:05:26.746214+0000 mon.a (mon.0) 8541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:26 smithi115 bash[45495]: audit 2022-04-22T20:05:26.746214+0000 mon.a (mon.0) 8541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:26 smithi115 bash[46934]: audit 2022-04-22T20:05:26.746214+0000 mon.a (mon.0) 8541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:28.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:27 smithi121 bash[36902]: cluster 2022-04-22T20:05:26.772629+0000 mgr.y (mgr.34107) 5462 : cluster [DBG] pgmap v6723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:27 smithi115 bash[45495]: cluster 2022-04-22T20:05:26.772629+0000 mgr.y (mgr.34107) 5462 : cluster [DBG] pgmap v6723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:27 smithi115 bash[46934]: cluster 2022-04-22T20:05:26.772629+0000 mgr.y (mgr.34107) 5462 : cluster [DBG] pgmap v6723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:28 smithi121 bash[36902]: audit 2022-04-22T20:05:28.187138+0000 mon.a (mon.0) 8542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:28 smithi115 bash[45495]: audit 2022-04-22T20:05:28.187138+0000 mon.a (mon.0) 8542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:28 smithi115 bash[46934]: audit 2022-04-22T20:05:28.187138+0000 mon.a (mon.0) 8542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:29 smithi121 bash[36902]: cluster 2022-04-22T20:05:28.773707+0000 mgr.y (mgr.34107) 5463 : cluster [DBG] pgmap v6724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:29 smithi115 bash[45495]: cluster 2022-04-22T20:05:28.773707+0000 mgr.y (mgr.34107) 5463 : cluster [DBG] pgmap v6724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:30.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:29 smithi115 bash[46934]: cluster 2022-04-22T20:05:28.773707+0000 mgr.y (mgr.34107) 5463 : cluster [DBG] pgmap v6724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:30 smithi121 bash[36902]: audit 2022-04-22T20:05:30.453696+0000 mon.a (mon.0) 8543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:30 smithi115 bash[45495]: audit 2022-04-22T20:05:30.453696+0000 mon.a (mon.0) 8543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:30 smithi115 bash[46934]: audit 2022-04-22T20:05:30.453696+0000 mon.a (mon.0) 8543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:32.239 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:31 smithi115 bash[45495]: cluster 2022-04-22T20:05:30.774780+0000 mgr.y (mgr.34107) 5464 : cluster [DBG] pgmap v6725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:32.239 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:31 smithi115 bash[45495]: audit 2022-04-22T20:05:31.749396+0000 mon.a (mon.0) 8544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:32.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:31 smithi115 bash[46934]: cluster 2022-04-22T20:05:30.774780+0000 mgr.y (mgr.34107) 5464 : cluster [DBG] pgmap v6725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:32.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:31 smithi115 bash[46934]: audit 2022-04-22T20:05:31.749396+0000 mon.a (mon.0) 8544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:31 smithi121 bash[36902]: cluster 2022-04-22T20:05:30.774780+0000 mgr.y (mgr.34107) 5464 : cluster [DBG] pgmap v6725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:31 smithi121 bash[36902]: audit 2022-04-22T20:05:31.749396+0000 mon.a (mon.0) 8544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:32.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:32.230Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:32 smithi121 bash[36902]: cluster 2022-04-22T20:05:32.775889+0000 mgr.y (mgr.34107) 5465 : cluster [DBG] pgmap v6726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:32 smithi115 bash[45495]: cluster 2022-04-22T20:05:32.775889+0000 mgr.y (mgr.34107) 5465 : cluster [DBG] pgmap v6726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:32 smithi115 bash[46934]: cluster 2022-04-22T20:05:32.775889+0000 mgr.y (mgr.34107) 5465 : cluster [DBG] pgmap v6726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:33 smithi121 bash[36902]: audit 2022-04-22T20:05:33.190177+0000 mon.a (mon.0) 8545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:33 smithi115 bash[45495]: audit 2022-04-22T20:05:33.190177+0000 mon.a (mon.0) 8545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:34.110Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:33 smithi115 bash[46934]: audit 2022-04-22T20:05:33.190177+0000 mon.a (mon.0) 8545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:34 smithi121 bash[36902]: cluster 2022-04-22T20:05:34.776980+0000 mgr.y (mgr.34107) 5466 : cluster [DBG] pgmap v6727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:34 smithi115 bash[45495]: cluster 2022-04-22T20:05:34.776980+0000 mgr.y (mgr.34107) 5466 : cluster [DBG] pgmap v6727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:35.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:34 smithi115 bash[46934]: cluster 2022-04-22T20:05:34.776980+0000 mgr.y (mgr.34107) 5466 : cluster [DBG] pgmap v6727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:35 smithi121 bash[36902]: audit 2022-04-22T20:05:35.457207+0000 mon.a (mon.0) 8546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:35 smithi115 bash[45495]: audit 2022-04-22T20:05:35.457207+0000 mon.a (mon.0) 8546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:35 smithi115 bash[46934]: audit 2022-04-22T20:05:35.457207+0000 mon.a (mon.0) 8546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:36 smithi121 bash[36902]: audit 2022-04-22T20:05:36.752653+0000 mon.a (mon.0) 8547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:36 smithi121 bash[36902]: cluster 2022-04-22T20:05:36.778094+0000 mgr.y (mgr.34107) 5467 : cluster [DBG] pgmap v6728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:36 smithi115 bash[46934]: audit 2022-04-22T20:05:36.752653+0000 mon.a (mon.0) 8547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:36 smithi115 bash[46934]: cluster 2022-04-22T20:05:36.778094+0000 mgr.y (mgr.34107) 5467 : cluster [DBG] pgmap v6728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:36 smithi115 bash[45495]: audit 2022-04-22T20:05:36.752653+0000 mon.a (mon.0) 8547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:36 smithi115 bash[45495]: cluster 2022-04-22T20:05:36.778094+0000 mgr.y (mgr.34107) 5467 : cluster [DBG] pgmap v6728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:38 smithi121 bash[36902]: audit 2022-04-22T20:05:38.193223+0000 mon.a (mon.0) 8548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:38 smithi115 bash[46934]: audit 2022-04-22T20:05:38.193223+0000 mon.a (mon.0) 8548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:38 smithi115 bash[45495]: audit 2022-04-22T20:05:38.193223+0000 mon.a (mon.0) 8548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:39 smithi121 bash[36902]: cluster 2022-04-22T20:05:38.779337+0000 mgr.y (mgr.34107) 5468 : cluster [DBG] pgmap v6729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:39 smithi115 bash[45495]: cluster 2022-04-22T20:05:38.779337+0000 mgr.y (mgr.34107) 5468 : cluster [DBG] pgmap v6729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:40.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:39 smithi115 bash[46934]: cluster 2022-04-22T20:05:38.779337+0000 mgr.y (mgr.34107) 5468 : cluster [DBG] pgmap v6729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:40 smithi121 bash[36902]: audit 2022-04-22T20:05:40.460742+0000 mon.a (mon.0) 8549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:40 smithi115 bash[45495]: audit 2022-04-22T20:05:40.460742+0000 mon.a (mon.0) 8549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:40 smithi115 bash[46934]: audit 2022-04-22T20:05:40.460742+0000 mon.a (mon.0) 8549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:41 smithi121 bash[36902]: cluster 2022-04-22T20:05:40.780407+0000 mgr.y (mgr.34107) 5469 : cluster [DBG] pgmap v6730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:41 smithi115 bash[45495]: cluster 2022-04-22T20:05:40.780407+0000 mgr.y (mgr.34107) 5469 : cluster [DBG] pgmap v6730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:41 smithi115 bash[46934]: cluster 2022-04-22T20:05:40.780407+0000 mgr.y (mgr.34107) 5469 : cluster [DBG] pgmap v6730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:42.231Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:42 smithi121 bash[36902]: audit 2022-04-22T20:05:41.755720+0000 mon.a (mon.0) 8550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:42 smithi115 bash[45495]: audit 2022-04-22T20:05:41.755720+0000 mon.a (mon.0) 8550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:42 smithi115 bash[46934]: audit 2022-04-22T20:05:41.755720+0000 mon.a (mon.0) 8550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:43 smithi121 bash[36902]: cluster 2022-04-22T20:05:42.781594+0000 mgr.y (mgr.34107) 5470 : cluster [DBG] pgmap v6731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:43 smithi121 bash[36902]: audit 2022-04-22T20:05:43.196338+0000 mon.a (mon.0) 8551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:43 smithi115 bash[45495]: cluster 2022-04-22T20:05:42.781594+0000 mgr.y (mgr.34107) 5470 : cluster [DBG] pgmap v6731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:43 smithi115 bash[45495]: audit 2022-04-22T20:05:43.196338+0000 mon.a (mon.0) 8551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:43 smithi115 bash[46934]: cluster 2022-04-22T20:05:42.781594+0000 mgr.y (mgr.34107) 5470 : cluster [DBG] pgmap v6731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:44.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:43 smithi115 bash[46934]: audit 2022-04-22T20:05:43.196338+0000 mon.a (mon.0) 8551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:44.111Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:45 smithi121 bash[36902]: cluster 2022-04-22T20:05:44.782746+0000 mgr.y (mgr.34107) 5471 : cluster [DBG] pgmap v6732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:45 smithi121 bash[36902]: audit 2022-04-22T20:05:45.464281+0000 mon.a (mon.0) 8552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:45 smithi115 bash[45495]: cluster 2022-04-22T20:05:44.782746+0000 mgr.y (mgr.34107) 5471 : cluster [DBG] pgmap v6732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:45 smithi115 bash[45495]: audit 2022-04-22T20:05:45.464281+0000 mon.a (mon.0) 8552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:45 smithi115 bash[46934]: cluster 2022-04-22T20:05:44.782746+0000 mgr.y (mgr.34107) 5471 : cluster [DBG] pgmap v6732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:45 smithi115 bash[46934]: audit 2022-04-22T20:05:45.464281+0000 mon.a (mon.0) 8552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:47.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:46 smithi121 bash[36902]: audit 2022-04-22T20:05:46.759142+0000 mon.a (mon.0) 8553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:46 smithi115 bash[45495]: audit 2022-04-22T20:05:46.759142+0000 mon.a (mon.0) 8553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:46 smithi115 bash[46934]: audit 2022-04-22T20:05:46.759142+0000 mon.a (mon.0) 8553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:47 smithi121 bash[36902]: cluster 2022-04-22T20:05:46.783755+0000 mgr.y (mgr.34107) 5472 : cluster [DBG] pgmap v6733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:47 smithi115 bash[45495]: cluster 2022-04-22T20:05:46.783755+0000 mgr.y (mgr.34107) 5472 : cluster [DBG] pgmap v6733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:47 smithi115 bash[46934]: cluster 2022-04-22T20:05:46.783755+0000 mgr.y (mgr.34107) 5472 : cluster [DBG] pgmap v6733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:48 smithi121 bash[36902]: audit 2022-04-22T20:05:48.199336+0000 mon.a (mon.0) 8554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:48 smithi115 bash[45495]: audit 2022-04-22T20:05:48.199336+0000 mon.a (mon.0) 8554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:48 smithi115 bash[46934]: audit 2022-04-22T20:05:48.199336+0000 mon.a (mon.0) 8554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:49 smithi121 bash[36902]: cluster 2022-04-22T20:05:48.784842+0000 mgr.y (mgr.34107) 5473 : cluster [DBG] pgmap v6734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:49 smithi115 bash[45495]: cluster 2022-04-22T20:05:48.784842+0000 mgr.y (mgr.34107) 5473 : cluster [DBG] pgmap v6734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:49 smithi115 bash[46934]: cluster 2022-04-22T20:05:48.784842+0000 mgr.y (mgr.34107) 5473 : cluster [DBG] pgmap v6734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:50 smithi121 bash[36902]: audit 2022-04-22T20:05:50.467789+0000 mon.a (mon.0) 8555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:50 smithi115 bash[45495]: audit 2022-04-22T20:05:50.467789+0000 mon.a (mon.0) 8555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:50 smithi115 bash[46934]: audit 2022-04-22T20:05:50.467789+0000 mon.a (mon.0) 8555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:52.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:51 smithi115 bash[46934]: cluster 2022-04-22T20:05:50.785892+0000 mgr.y (mgr.34107) 5474 : cluster [DBG] pgmap v6735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:51 smithi115 bash[46934]: audit 2022-04-22T20:05:51.762235+0000 mon.a (mon.0) 8556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:52.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:51 smithi115 bash[45495]: cluster 2022-04-22T20:05:50.785892+0000 mgr.y (mgr.34107) 5474 : cluster [DBG] pgmap v6735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:52.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:51 smithi115 bash[45495]: audit 2022-04-22T20:05:51.762235+0000 mon.a (mon.0) 8556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:52.229Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:52.548 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:51 smithi121 bash[36902]: cluster 2022-04-22T20:05:50.785892+0000 mgr.y (mgr.34107) 5474 : cluster [DBG] pgmap v6735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:52.549 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:51 smithi121 bash[36902]: audit 2022-04-22T20:05:51.762235+0000 mon.a (mon.0) 8556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:52 smithi121 bash[36902]: audit 2022-04-22T20:05:51.966118+0000 mon.a (mon.0) 8557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:05:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:52 smithi121 bash[36902]: audit 2022-04-22T20:05:51.969261+0000 mon.a (mon.0) 8558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:05:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:52 smithi121 bash[36902]: audit 2022-04-22T20:05:51.971123+0000 mon.a (mon.0) 8559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:05:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:52 smithi121 bash[36902]: audit 2022-04-22T20:05:52.323739+0000 mon.a (mon.0) 8560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:05:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[45495]: audit 2022-04-22T20:05:51.966118+0000 mon.a (mon.0) 8557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:05:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[45495]: audit 2022-04-22T20:05:51.969261+0000 mon.a (mon.0) 8558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:05:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[45495]: audit 2022-04-22T20:05:51.971123+0000 mon.a (mon.0) 8559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:05:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[45495]: audit 2022-04-22T20:05:52.323739+0000 mon.a (mon.0) 8560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:05:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[46934]: audit 2022-04-22T20:05:51.966118+0000 mon.a (mon.0) 8557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:05:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[46934]: audit 2022-04-22T20:05:51.969261+0000 mon.a (mon.0) 8558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:05:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[46934]: audit 2022-04-22T20:05:51.971123+0000 mon.a (mon.0) 8559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:05:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:52 smithi115 bash[46934]: audit 2022-04-22T20:05:52.323739+0000 mon.a (mon.0) 8560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:05:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:53 smithi121 bash[36902]: cluster 2022-04-22T20:05:52.787092+0000 mgr.y (mgr.34107) 5475 : cluster [DBG] pgmap v6736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:53 smithi121 bash[36902]: audit 2022-04-22T20:05:53.202508+0000 mon.a (mon.0) 8561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:54.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:05:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:05:54.110Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:05:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:53 smithi115 bash[46934]: cluster 2022-04-22T20:05:52.787092+0000 mgr.y (mgr.34107) 5475 : cluster [DBG] pgmap v6736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:53 smithi115 bash[46934]: audit 2022-04-22T20:05:53.202508+0000 mon.a (mon.0) 8561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:53 smithi115 bash[45495]: cluster 2022-04-22T20:05:52.787092+0000 mgr.y (mgr.34107) 5475 : cluster [DBG] pgmap v6736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:53 smithi115 bash[45495]: audit 2022-04-22T20:05:53.202508+0000 mon.a (mon.0) 8561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:54.508 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:05:54.860 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:05:54.861 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:05:54.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:05:54.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:05:54.863 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:05:55.259 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:05:55.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:05:55.261 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:05:55.261 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:05:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:55 smithi121 bash[36902]: audit 2022-04-22T20:05:54.482629+0000 mgr.y (mgr.34107) 5476 : audit [DBG] from='client.51028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:55 smithi121 bash[36902]: cluster 2022-04-22T20:05:54.788207+0000 mgr.y (mgr.34107) 5477 : cluster [DBG] pgmap v6737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:55 smithi121 bash[36902]: audit 2022-04-22T20:05:54.830102+0000 mgr.y (mgr.34107) 5478 : audit [DBG] from='client.51034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:55 smithi121 bash[36902]: audit 2022-04-22T20:05:55.249880+0000 mon.a (mon.0) 8562 : audit [DBG] from='client.? 172.21.15.115:0/3574209190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:56.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:55 smithi121 bash[36902]: audit 2022-04-22T20:05:55.471563+0000 mon.a (mon.0) 8563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[45495]: audit 2022-04-22T20:05:54.482629+0000 mgr.y (mgr.34107) 5476 : audit [DBG] from='client.51028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[45495]: cluster 2022-04-22T20:05:54.788207+0000 mgr.y (mgr.34107) 5477 : cluster [DBG] pgmap v6737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[45495]: audit 2022-04-22T20:05:54.830102+0000 mgr.y (mgr.34107) 5478 : audit [DBG] from='client.51034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[45495]: audit 2022-04-22T20:05:55.249880+0000 mon.a (mon.0) 8562 : audit [DBG] from='client.? 172.21.15.115:0/3574209190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[45495]: audit 2022-04-22T20:05:55.471563+0000 mon.a (mon.0) 8563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[46934]: audit 2022-04-22T20:05:54.482629+0000 mgr.y (mgr.34107) 5476 : audit [DBG] from='client.51028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[46934]: cluster 2022-04-22T20:05:54.788207+0000 mgr.y (mgr.34107) 5477 : cluster [DBG] pgmap v6737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[46934]: audit 2022-04-22T20:05:54.830102+0000 mgr.y (mgr.34107) 5478 : audit [DBG] from='client.51034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:05:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[46934]: audit 2022-04-22T20:05:55.249880+0000 mon.a (mon.0) 8562 : audit [DBG] from='client.? 172.21.15.115:0/3574209190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:05:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:55 smithi115 bash[46934]: audit 2022-04-22T20:05:55.471563+0000 mon.a (mon.0) 8563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:05:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:56 smithi121 bash[36902]: audit 2022-04-22T20:05:56.765187+0000 mon.a (mon.0) 8564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:56 smithi115 bash[45495]: audit 2022-04-22T20:05:56.765187+0000 mon.a (mon.0) 8564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:56 smithi115 bash[46934]: audit 2022-04-22T20:05:56.765187+0000 mon.a (mon.0) 8564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:05:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:57 smithi121 bash[36902]: cluster 2022-04-22T20:05:56.789256+0000 mgr.y (mgr.34107) 5479 : cluster [DBG] pgmap v6738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:57 smithi115 bash[45495]: cluster 2022-04-22T20:05:56.789256+0000 mgr.y (mgr.34107) 5479 : cluster [DBG] pgmap v6738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:57 smithi115 bash[46934]: cluster 2022-04-22T20:05:56.789256+0000 mgr.y (mgr.34107) 5479 : cluster [DBG] pgmap v6738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:05:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:58 smithi121 bash[36902]: audit 2022-04-22T20:05:58.205448+0000 mon.a (mon.0) 8565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:58 smithi115 bash[45495]: audit 2022-04-22T20:05:58.205448+0000 mon.a (mon.0) 8565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:05:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:58 smithi115 bash[46934]: audit 2022-04-22T20:05:58.205448+0000 mon.a (mon.0) 8565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:05:59 smithi121 bash[36902]: cluster 2022-04-22T20:05:58.790393+0000 mgr.y (mgr.34107) 5480 : cluster [DBG] pgmap v6739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:05:59 smithi115 bash[45495]: cluster 2022-04-22T20:05:58.790393+0000 mgr.y (mgr.34107) 5480 : cluster [DBG] pgmap v6739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:05:59 smithi115 bash[46934]: cluster 2022-04-22T20:05:58.790393+0000 mgr.y (mgr.34107) 5480 : cluster [DBG] pgmap v6739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:00 smithi121 bash[36902]: audit 2022-04-22T20:06:00.475112+0000 mon.a (mon.0) 8566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:00 smithi115 bash[45495]: audit 2022-04-22T20:06:00.475112+0000 mon.a (mon.0) 8566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:01.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:00 smithi115 bash[46934]: audit 2022-04-22T20:06:00.475112+0000 mon.a (mon.0) 8566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:02.241 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:01 smithi115 bash[45495]: cluster 2022-04-22T20:06:00.791453+0000 mgr.y (mgr.34107) 5481 : cluster [DBG] pgmap v6740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:02.241 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:01 smithi115 bash[45495]: audit 2022-04-22T20:06:01.768247+0000 mon.a (mon.0) 8567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:02.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:01 smithi115 bash[46934]: cluster 2022-04-22T20:06:00.791453+0000 mgr.y (mgr.34107) 5481 : cluster [DBG] pgmap v6740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:02.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:01 smithi115 bash[46934]: audit 2022-04-22T20:06:01.768247+0000 mon.a (mon.0) 8567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:01 smithi121 bash[36902]: cluster 2022-04-22T20:06:00.791453+0000 mgr.y (mgr.34107) 5481 : cluster [DBG] pgmap v6740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:01 smithi121 bash[36902]: audit 2022-04-22T20:06:01.768247+0000 mon.a (mon.0) 8567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:02.232Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:03 smithi121 bash[36902]: cluster 2022-04-22T20:06:02.792582+0000 mgr.y (mgr.34107) 5482 : cluster [DBG] pgmap v6741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:03 smithi121 bash[36902]: audit 2022-04-22T20:06:03.208600+0000 mon.a (mon.0) 8568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:04.113Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:03 smithi115 bash[45495]: cluster 2022-04-22T20:06:02.792582+0000 mgr.y (mgr.34107) 5482 : cluster [DBG] pgmap v6741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:03 smithi115 bash[45495]: audit 2022-04-22T20:06:03.208600+0000 mon.a (mon.0) 8568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:03 smithi115 bash[46934]: cluster 2022-04-22T20:06:02.792582+0000 mgr.y (mgr.34107) 5482 : cluster [DBG] pgmap v6741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:03 smithi115 bash[46934]: audit 2022-04-22T20:06:03.208600+0000 mon.a (mon.0) 8568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:05 smithi121 bash[36902]: cluster 2022-04-22T20:06:04.793683+0000 mgr.y (mgr.34107) 5483 : cluster [DBG] pgmap v6742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:05 smithi121 bash[36902]: audit 2022-04-22T20:06:05.477920+0000 mon.a (mon.0) 8569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:05 smithi115 bash[45495]: cluster 2022-04-22T20:06:04.793683+0000 mgr.y (mgr.34107) 5483 : cluster [DBG] pgmap v6742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:05 smithi115 bash[45495]: audit 2022-04-22T20:06:05.477920+0000 mon.a (mon.0) 8569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:05 smithi115 bash[46934]: cluster 2022-04-22T20:06:04.793683+0000 mgr.y (mgr.34107) 5483 : cluster [DBG] pgmap v6742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:05 smithi115 bash[46934]: audit 2022-04-22T20:06:05.477920+0000 mon.a (mon.0) 8569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:06 smithi121 bash[36902]: audit 2022-04-22T20:06:06.771348+0000 mon.a (mon.0) 8570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:06 smithi115 bash[45495]: audit 2022-04-22T20:06:06.771348+0000 mon.a (mon.0) 8570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:06 smithi115 bash[46934]: audit 2022-04-22T20:06:06.771348+0000 mon.a (mon.0) 8570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:07 smithi121 bash[36902]: cluster 2022-04-22T20:06:06.794798+0000 mgr.y (mgr.34107) 5484 : cluster [DBG] pgmap v6743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:07 smithi115 bash[45495]: cluster 2022-04-22T20:06:06.794798+0000 mgr.y (mgr.34107) 5484 : cluster [DBG] pgmap v6743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:07 smithi115 bash[46934]: cluster 2022-04-22T20:06:06.794798+0000 mgr.y (mgr.34107) 5484 : cluster [DBG] pgmap v6743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:08 smithi121 bash[36902]: audit 2022-04-22T20:06:08.211610+0000 mon.a (mon.0) 8571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:08 smithi115 bash[45495]: audit 2022-04-22T20:06:08.211610+0000 mon.a (mon.0) 8571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:08 smithi115 bash[46934]: audit 2022-04-22T20:06:08.211610+0000 mon.a (mon.0) 8571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:09 smithi121 bash[36902]: cluster 2022-04-22T20:06:08.795985+0000 mgr.y (mgr.34107) 5485 : cluster [DBG] pgmap v6744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:09 smithi115 bash[45495]: cluster 2022-04-22T20:06:08.795985+0000 mgr.y (mgr.34107) 5485 : cluster [DBG] pgmap v6744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:09 smithi115 bash[46934]: cluster 2022-04-22T20:06:08.795985+0000 mgr.y (mgr.34107) 5485 : cluster [DBG] pgmap v6744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:10 smithi121 bash[36902]: audit 2022-04-22T20:06:10.481698+0000 mon.a (mon.0) 8572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:10 smithi115 bash[45495]: audit 2022-04-22T20:06:10.481698+0000 mon.a (mon.0) 8572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:10 smithi115 bash[46934]: audit 2022-04-22T20:06:10.481698+0000 mon.a (mon.0) 8572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:12.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:11 smithi115 bash[45495]: cluster 2022-04-22T20:06:10.797047+0000 mgr.y (mgr.34107) 5486 : cluster [DBG] pgmap v6745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:12.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:11 smithi115 bash[45495]: audit 2022-04-22T20:06:11.774601+0000 mon.a (mon.0) 8573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:12.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:11 smithi115 bash[46934]: cluster 2022-04-22T20:06:10.797047+0000 mgr.y (mgr.34107) 5486 : cluster [DBG] pgmap v6745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:12.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:11 smithi115 bash[46934]: audit 2022-04-22T20:06:11.774601+0000 mon.a (mon.0) 8573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:11 smithi121 bash[36902]: cluster 2022-04-22T20:06:10.797047+0000 mgr.y (mgr.34107) 5486 : cluster [DBG] pgmap v6745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:11 smithi121 bash[36902]: audit 2022-04-22T20:06:11.774601+0000 mon.a (mon.0) 8573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:12.231Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:12 smithi121 bash[36902]: cluster 2022-04-22T20:06:12.798169+0000 mgr.y (mgr.34107) 5487 : cluster [DBG] pgmap v6746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:12 smithi115 bash[45495]: cluster 2022-04-22T20:06:12.798169+0000 mgr.y (mgr.34107) 5487 : cluster [DBG] pgmap v6746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:12 smithi115 bash[46934]: cluster 2022-04-22T20:06:12.798169+0000 mgr.y (mgr.34107) 5487 : cluster [DBG] pgmap v6746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:13 smithi121 bash[36902]: audit 2022-04-22T20:06:13.214565+0000 mon.a (mon.0) 8574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:13 smithi115 bash[45495]: audit 2022-04-22T20:06:13.214565+0000 mon.a (mon.0) 8574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:14.112Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:13 smithi115 bash[46934]: audit 2022-04-22T20:06:13.214565+0000 mon.a (mon.0) 8574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:14 smithi121 bash[36902]: cluster 2022-04-22T20:06:14.799355+0000 mgr.y (mgr.34107) 5488 : cluster [DBG] pgmap v6747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:14 smithi115 bash[45495]: cluster 2022-04-22T20:06:14.799355+0000 mgr.y (mgr.34107) 5488 : cluster [DBG] pgmap v6747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:14 smithi115 bash[46934]: cluster 2022-04-22T20:06:14.799355+0000 mgr.y (mgr.34107) 5488 : cluster [DBG] pgmap v6747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:15 smithi121 bash[36902]: audit 2022-04-22T20:06:15.122264+0000 mon.a (mon.0) 8575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:06:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:15 smithi121 bash[36902]: audit 2022-04-22T20:06:15.122878+0000 mon.a (mon.0) 8576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:06:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:15 smithi121 bash[36902]: audit 2022-04-22T20:06:15.485553+0000 mon.a (mon.0) 8577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[45495]: audit 2022-04-22T20:06:15.122264+0000 mon.a (mon.0) 8575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:06:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[45495]: audit 2022-04-22T20:06:15.122878+0000 mon.a (mon.0) 8576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:06:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[45495]: audit 2022-04-22T20:06:15.485553+0000 mon.a (mon.0) 8577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[46934]: audit 2022-04-22T20:06:15.122264+0000 mon.a (mon.0) 8575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:06:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[46934]: audit 2022-04-22T20:06:15.122878+0000 mon.a (mon.0) 8576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:06:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:15 smithi115 bash[46934]: audit 2022-04-22T20:06:15.485553+0000 mon.a (mon.0) 8577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:16 smithi121 bash[36902]: audit 2022-04-22T20:06:16.777948+0000 mon.a (mon.0) 8578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:16 smithi121 bash[36902]: cluster 2022-04-22T20:06:16.800518+0000 mgr.y (mgr.34107) 5489 : cluster [DBG] pgmap v6748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:16 smithi115 bash[45495]: audit 2022-04-22T20:06:16.777948+0000 mon.a (mon.0) 8578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:16 smithi115 bash[45495]: cluster 2022-04-22T20:06:16.800518+0000 mgr.y (mgr.34107) 5489 : cluster [DBG] pgmap v6748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:16 smithi115 bash[46934]: audit 2022-04-22T20:06:16.777948+0000 mon.a (mon.0) 8578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:16 smithi115 bash[46934]: cluster 2022-04-22T20:06:16.800518+0000 mgr.y (mgr.34107) 5489 : cluster [DBG] pgmap v6748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:18 smithi115 bash[45495]: audit 2022-04-22T20:06:18.217642+0000 mon.a (mon.0) 8579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:18 smithi115 bash[46934]: audit 2022-04-22T20:06:18.217642+0000 mon.a (mon.0) 8579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:18 smithi121 bash[36902]: audit 2022-04-22T20:06:18.217642+0000 mon.a (mon.0) 8579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:19 smithi121 bash[36902]: cluster 2022-04-22T20:06:18.801701+0000 mgr.y (mgr.34107) 5490 : cluster [DBG] pgmap v6749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:19 smithi115 bash[45495]: cluster 2022-04-22T20:06:18.801701+0000 mgr.y (mgr.34107) 5490 : cluster [DBG] pgmap v6749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:19 smithi115 bash[46934]: cluster 2022-04-22T20:06:18.801701+0000 mgr.y (mgr.34107) 5490 : cluster [DBG] pgmap v6749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:20 smithi121 bash[36902]: audit 2022-04-22T20:06:20.489068+0000 mon.a (mon.0) 8580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:20 smithi115 bash[45495]: audit 2022-04-22T20:06:20.489068+0000 mon.a (mon.0) 8580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:21.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:20 smithi115 bash[46934]: audit 2022-04-22T20:06:20.489068+0000 mon.a (mon.0) 8580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:21 smithi121 bash[36902]: cluster 2022-04-22T20:06:20.802804+0000 mgr.y (mgr.34107) 5491 : cluster [DBG] pgmap v6750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:21 smithi115 bash[45495]: cluster 2022-04-22T20:06:20.802804+0000 mgr.y (mgr.34107) 5491 : cluster [DBG] pgmap v6750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:21 smithi115 bash[46934]: cluster 2022-04-22T20:06:20.802804+0000 mgr.y (mgr.34107) 5491 : cluster [DBG] pgmap v6750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:22.233Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:22 smithi121 bash[36902]: audit 2022-04-22T20:06:21.781169+0000 mon.a (mon.0) 8581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:22 smithi115 bash[45495]: audit 2022-04-22T20:06:21.781169+0000 mon.a (mon.0) 8581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:22 smithi115 bash[46934]: audit 2022-04-22T20:06:21.781169+0000 mon.a (mon.0) 8581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:23 smithi121 bash[36902]: cluster 2022-04-22T20:06:22.803929+0000 mgr.y (mgr.34107) 5492 : cluster [DBG] pgmap v6751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:23 smithi121 bash[36902]: audit 2022-04-22T20:06:23.220618+0000 mon.a (mon.0) 8582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:23 smithi115 bash[45495]: cluster 2022-04-22T20:06:22.803929+0000 mgr.y (mgr.34107) 5492 : cluster [DBG] pgmap v6751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:23 smithi115 bash[45495]: audit 2022-04-22T20:06:23.220618+0000 mon.a (mon.0) 8582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:23 smithi115 bash[46934]: cluster 2022-04-22T20:06:22.803929+0000 mgr.y (mgr.34107) 5492 : cluster [DBG] pgmap v6751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:23 smithi115 bash[46934]: audit 2022-04-22T20:06:23.220618+0000 mon.a (mon.0) 8582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:24.112Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:25.634 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:06:25.978 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:06:25.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:06:25.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:06:25.981 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:06:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:25 smithi121 bash[36902]: cluster 2022-04-22T20:06:24.805064+0000 mgr.y (mgr.34107) 5493 : cluster [DBG] pgmap v6752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:25 smithi121 bash[36902]: audit 2022-04-22T20:06:25.492594+0000 mon.a (mon.0) 8583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:25 smithi115 bash[46934]: cluster 2022-04-22T20:06:24.805064+0000 mgr.y (mgr.34107) 5493 : cluster [DBG] pgmap v6752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:25 smithi115 bash[46934]: audit 2022-04-22T20:06:25.492594+0000 mon.a (mon.0) 8583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:25 smithi115 bash[45495]: cluster 2022-04-22T20:06:24.805064+0000 mgr.y (mgr.34107) 5493 : cluster [DBG] pgmap v6752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:26.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:25 smithi115 bash[45495]: audit 2022-04-22T20:06:25.492594+0000 mon.a (mon.0) 8583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:26.383 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:06:26.383 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:06:26.383 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:06:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:26.385 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:06:26.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:06:26.385 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:06:26.385 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:06:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:26 smithi121 bash[36902]: audit 2022-04-22T20:06:25.610578+0000 mgr.y (mgr.34107) 5494 : audit [DBG] from='client.41112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:26 smithi121 bash[36902]: audit 2022-04-22T20:06:26.374505+0000 mon.a (mon.0) 8584 : audit [DBG] from='client.? 172.21.15.115:0/2871893621' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:26 smithi121 bash[36902]: audit 2022-04-22T20:06:26.784239+0000 mon.a (mon.0) 8585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[45495]: audit 2022-04-22T20:06:25.610578+0000 mgr.y (mgr.34107) 5494 : audit [DBG] from='client.41112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[45495]: audit 2022-04-22T20:06:26.374505+0000 mon.a (mon.0) 8584 : audit [DBG] from='client.? 172.21.15.115:0/2871893621' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[45495]: audit 2022-04-22T20:06:26.784239+0000 mon.a (mon.0) 8585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[46934]: audit 2022-04-22T20:06:25.610578+0000 mgr.y (mgr.34107) 5494 : audit [DBG] from='client.41112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[46934]: audit 2022-04-22T20:06:26.374505+0000 mon.a (mon.0) 8584 : audit [DBG] from='client.? 172.21.15.115:0/2871893621' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:26 smithi115 bash[46934]: audit 2022-04-22T20:06:26.784239+0000 mon.a (mon.0) 8585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:27 smithi121 bash[36902]: audit 2022-04-22T20:06:25.951131+0000 mgr.y (mgr.34107) 5495 : audit [DBG] from='client.41118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:27 smithi121 bash[36902]: cluster 2022-04-22T20:06:26.806112+0000 mgr.y (mgr.34107) 5496 : cluster [DBG] pgmap v6753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:27 smithi115 bash[45495]: audit 2022-04-22T20:06:25.951131+0000 mgr.y (mgr.34107) 5495 : audit [DBG] from='client.41118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:27 smithi115 bash[45495]: cluster 2022-04-22T20:06:26.806112+0000 mgr.y (mgr.34107) 5496 : cluster [DBG] pgmap v6753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:27 smithi115 bash[46934]: audit 2022-04-22T20:06:25.951131+0000 mgr.y (mgr.34107) 5495 : audit [DBG] from='client.41118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:27 smithi115 bash[46934]: cluster 2022-04-22T20:06:26.806112+0000 mgr.y (mgr.34107) 5496 : cluster [DBG] pgmap v6753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:28 smithi121 bash[36902]: audit 2022-04-22T20:06:28.223779+0000 mon.a (mon.0) 8586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:28 smithi115 bash[45495]: audit 2022-04-22T20:06:28.223779+0000 mon.a (mon.0) 8586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:28 smithi115 bash[46934]: audit 2022-04-22T20:06:28.223779+0000 mon.a (mon.0) 8586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:29 smithi121 bash[36902]: cluster 2022-04-22T20:06:28.807277+0000 mgr.y (mgr.34107) 5497 : cluster [DBG] pgmap v6754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:29 smithi115 bash[46934]: cluster 2022-04-22T20:06:28.807277+0000 mgr.y (mgr.34107) 5497 : cluster [DBG] pgmap v6754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:30.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:29 smithi115 bash[45495]: cluster 2022-04-22T20:06:28.807277+0000 mgr.y (mgr.34107) 5497 : cluster [DBG] pgmap v6754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:30 smithi121 bash[36902]: audit 2022-04-22T20:06:30.496165+0000 mon.a (mon.0) 8587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:30 smithi115 bash[45495]: audit 2022-04-22T20:06:30.496165+0000 mon.a (mon.0) 8587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:30 smithi115 bash[46934]: audit 2022-04-22T20:06:30.496165+0000 mon.a (mon.0) 8587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:32.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:31 smithi115 bash[45495]: cluster 2022-04-22T20:06:30.808388+0000 mgr.y (mgr.34107) 5498 : cluster [DBG] pgmap v6755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:32.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:31 smithi115 bash[45495]: audit 2022-04-22T20:06:31.787540+0000 mon.a (mon.0) 8588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:32.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:31 smithi115 bash[46934]: cluster 2022-04-22T20:06:30.808388+0000 mgr.y (mgr.34107) 5498 : cluster [DBG] pgmap v6755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:32.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:31 smithi115 bash[46934]: audit 2022-04-22T20:06:31.787540+0000 mon.a (mon.0) 8588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:31 smithi121 bash[36902]: cluster 2022-04-22T20:06:30.808388+0000 mgr.y (mgr.34107) 5498 : cluster [DBG] pgmap v6755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:31 smithi121 bash[36902]: audit 2022-04-22T20:06:31.787540+0000 mon.a (mon.0) 8588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:32.233Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:33 smithi121 bash[36902]: cluster 2022-04-22T20:06:32.809473+0000 mgr.y (mgr.34107) 5499 : cluster [DBG] pgmap v6756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:33 smithi121 bash[36902]: audit 2022-04-22T20:06:33.226811+0000 mon.a (mon.0) 8589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:34.114Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:33 smithi115 bash[45495]: cluster 2022-04-22T20:06:32.809473+0000 mgr.y (mgr.34107) 5499 : cluster [DBG] pgmap v6756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:33 smithi115 bash[45495]: audit 2022-04-22T20:06:33.226811+0000 mon.a (mon.0) 8589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:33 smithi115 bash[46934]: cluster 2022-04-22T20:06:32.809473+0000 mgr.y (mgr.34107) 5499 : cluster [DBG] pgmap v6756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:33 smithi115 bash[46934]: audit 2022-04-22T20:06:33.226811+0000 mon.a (mon.0) 8589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:35 smithi121 bash[36902]: cluster 2022-04-22T20:06:34.810560+0000 mgr.y (mgr.34107) 5500 : cluster [DBG] pgmap v6757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:35 smithi121 bash[36902]: audit 2022-04-22T20:06:35.499807+0000 mon.a (mon.0) 8590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:35 smithi115 bash[45495]: cluster 2022-04-22T20:06:34.810560+0000 mgr.y (mgr.34107) 5500 : cluster [DBG] pgmap v6757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:35 smithi115 bash[45495]: audit 2022-04-22T20:06:35.499807+0000 mon.a (mon.0) 8590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:35 smithi115 bash[46934]: cluster 2022-04-22T20:06:34.810560+0000 mgr.y (mgr.34107) 5500 : cluster [DBG] pgmap v6757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:35 smithi115 bash[46934]: audit 2022-04-22T20:06:35.499807+0000 mon.a (mon.0) 8590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:36 smithi121 bash[36902]: audit 2022-04-22T20:06:36.790472+0000 mon.a (mon.0) 8591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:36 smithi115 bash[45495]: audit 2022-04-22T20:06:36.790472+0000 mon.a (mon.0) 8591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:37.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:36 smithi115 bash[46934]: audit 2022-04-22T20:06:36.790472+0000 mon.a (mon.0) 8591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:37 smithi121 bash[36902]: cluster 2022-04-22T20:06:36.811579+0000 mgr.y (mgr.34107) 5501 : cluster [DBG] pgmap v6758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:37 smithi115 bash[45495]: cluster 2022-04-22T20:06:36.811579+0000 mgr.y (mgr.34107) 5501 : cluster [DBG] pgmap v6758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:37 smithi115 bash[46934]: cluster 2022-04-22T20:06:36.811579+0000 mgr.y (mgr.34107) 5501 : cluster [DBG] pgmap v6758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:38 smithi121 bash[36902]: audit 2022-04-22T20:06:38.229819+0000 mon.a (mon.0) 8592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:38 smithi115 bash[45495]: audit 2022-04-22T20:06:38.229819+0000 mon.a (mon.0) 8592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:38 smithi115 bash[46934]: audit 2022-04-22T20:06:38.229819+0000 mon.a (mon.0) 8592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:39 smithi121 bash[36902]: cluster 2022-04-22T20:06:38.812680+0000 mgr.y (mgr.34107) 5502 : cluster [DBG] pgmap v6759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:39 smithi115 bash[45495]: cluster 2022-04-22T20:06:38.812680+0000 mgr.y (mgr.34107) 5502 : cluster [DBG] pgmap v6759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:39 smithi115 bash[46934]: cluster 2022-04-22T20:06:38.812680+0000 mgr.y (mgr.34107) 5502 : cluster [DBG] pgmap v6759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:40 smithi121 bash[36902]: audit 2022-04-22T20:06:40.503622+0000 mon.a (mon.0) 8593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:40 smithi115 bash[45495]: audit 2022-04-22T20:06:40.503622+0000 mon.a (mon.0) 8593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:40 smithi115 bash[46934]: audit 2022-04-22T20:06:40.503622+0000 mon.a (mon.0) 8593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:42.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:41 smithi115 bash[45495]: cluster 2022-04-22T20:06:40.813699+0000 mgr.y (mgr.34107) 5503 : cluster [DBG] pgmap v6760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:42.243 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:41 smithi115 bash[45495]: audit 2022-04-22T20:06:41.793833+0000 mon.a (mon.0) 8594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:42.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:41 smithi115 bash[46934]: cluster 2022-04-22T20:06:40.813699+0000 mgr.y (mgr.34107) 5503 : cluster [DBG] pgmap v6760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:42.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:41 smithi115 bash[46934]: audit 2022-04-22T20:06:41.793833+0000 mon.a (mon.0) 8594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:41 smithi121 bash[36902]: cluster 2022-04-22T20:06:40.813699+0000 mgr.y (mgr.34107) 5503 : cluster [DBG] pgmap v6760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:41 smithi121 bash[36902]: audit 2022-04-22T20:06:41.793833+0000 mon.a (mon.0) 8594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:42.233Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:43 smithi121 bash[36902]: cluster 2022-04-22T20:06:42.814799+0000 mgr.y (mgr.34107) 5504 : cluster [DBG] pgmap v6761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:43 smithi121 bash[36902]: audit 2022-04-22T20:06:43.232892+0000 mon.a (mon.0) 8595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:43 smithi115 bash[45495]: cluster 2022-04-22T20:06:42.814799+0000 mgr.y (mgr.34107) 5504 : cluster [DBG] pgmap v6761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:43 smithi115 bash[45495]: audit 2022-04-22T20:06:43.232892+0000 mon.a (mon.0) 8595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:44.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:43 smithi115 bash[46934]: cluster 2022-04-22T20:06:42.814799+0000 mgr.y (mgr.34107) 5504 : cluster [DBG] pgmap v6761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:44.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:43 smithi115 bash[46934]: audit 2022-04-22T20:06:43.232892+0000 mon.a (mon.0) 8595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:45 smithi121 bash[36902]: cluster 2022-04-22T20:06:44.815905+0000 mgr.y (mgr.34107) 5505 : cluster [DBG] pgmap v6762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:45 smithi121 bash[36902]: audit 2022-04-22T20:06:45.506534+0000 mon.a (mon.0) 8596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:45 smithi115 bash[46934]: cluster 2022-04-22T20:06:44.815905+0000 mgr.y (mgr.34107) 5505 : cluster [DBG] pgmap v6762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:45 smithi115 bash[46934]: audit 2022-04-22T20:06:45.506534+0000 mon.a (mon.0) 8596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:45 smithi115 bash[45495]: cluster 2022-04-22T20:06:44.815905+0000 mgr.y (mgr.34107) 5505 : cluster [DBG] pgmap v6762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:45 smithi115 bash[45495]: audit 2022-04-22T20:06:45.506534+0000 mon.a (mon.0) 8596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:46 smithi121 bash[36902]: audit 2022-04-22T20:06:46.796753+0000 mon.a (mon.0) 8597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:46 smithi115 bash[45495]: audit 2022-04-22T20:06:46.796753+0000 mon.a (mon.0) 8597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:46 smithi115 bash[46934]: audit 2022-04-22T20:06:46.796753+0000 mon.a (mon.0) 8597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:47 smithi121 bash[36902]: cluster 2022-04-22T20:06:46.816959+0000 mgr.y (mgr.34107) 5506 : cluster [DBG] pgmap v6763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:47 smithi115 bash[45495]: cluster 2022-04-22T20:06:46.816959+0000 mgr.y (mgr.34107) 5506 : cluster [DBG] pgmap v6763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:47 smithi115 bash[46934]: cluster 2022-04-22T20:06:46.816959+0000 mgr.y (mgr.34107) 5506 : cluster [DBG] pgmap v6763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:48 smithi121 bash[36902]: audit 2022-04-22T20:06:48.235923+0000 mon.a (mon.0) 8598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:48 smithi121 bash[36902]: cluster 2022-04-22T20:06:48.818179+0000 mgr.y (mgr.34107) 5507 : cluster [DBG] pgmap v6764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:48 smithi115 bash[45495]: audit 2022-04-22T20:06:48.235923+0000 mon.a (mon.0) 8598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:48 smithi115 bash[45495]: cluster 2022-04-22T20:06:48.818179+0000 mgr.y (mgr.34107) 5507 : cluster [DBG] pgmap v6764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:48 smithi115 bash[46934]: audit 2022-04-22T20:06:48.235923+0000 mon.a (mon.0) 8598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:48 smithi115 bash[46934]: cluster 2022-04-22T20:06:48.818179+0000 mgr.y (mgr.34107) 5507 : cluster [DBG] pgmap v6764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:50 smithi121 bash[36902]: audit 2022-04-22T20:06:50.510234+0000 mon.a (mon.0) 8599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:50 smithi115 bash[45495]: audit 2022-04-22T20:06:50.510234+0000 mon.a (mon.0) 8599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:50 smithi115 bash[46934]: audit 2022-04-22T20:06:50.510234+0000 mon.a (mon.0) 8599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:51 smithi121 bash[36902]: cluster 2022-04-22T20:06:50.819301+0000 mgr.y (mgr.34107) 5508 : cluster [DBG] pgmap v6765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:51 smithi115 bash[45495]: cluster 2022-04-22T20:06:50.819301+0000 mgr.y (mgr.34107) 5508 : cluster [DBG] pgmap v6765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:51 smithi115 bash[46934]: cluster 2022-04-22T20:06:50.819301+0000 mgr.y (mgr.34107) 5508 : cluster [DBG] pgmap v6765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:52.233Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:52 smithi121 bash[36902]: audit 2022-04-22T20:06:51.799894+0000 mon.a (mon.0) 8600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:52 smithi121 bash[36902]: audit 2022-04-22T20:06:52.332952+0000 mon.a (mon.0) 8601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:06:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:52 smithi121 bash[36902]: audit 2022-04-22T20:06:52.336019+0000 mon.a (mon.0) 8602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:06:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:52 smithi121 bash[36902]: audit 2022-04-22T20:06:52.337716+0000 mon.a (mon.0) 8603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:06:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[45495]: audit 2022-04-22T20:06:51.799894+0000 mon.a (mon.0) 8600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[45495]: audit 2022-04-22T20:06:52.332952+0000 mon.a (mon.0) 8601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:06:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[45495]: audit 2022-04-22T20:06:52.336019+0000 mon.a (mon.0) 8602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:06:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[45495]: audit 2022-04-22T20:06:52.337716+0000 mon.a (mon.0) 8603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:06:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[46934]: audit 2022-04-22T20:06:51.799894+0000 mon.a (mon.0) 8600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[46934]: audit 2022-04-22T20:06:52.332952+0000 mon.a (mon.0) 8601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:06:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[46934]: audit 2022-04-22T20:06:52.336019+0000 mon.a (mon.0) 8602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:06:53.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:52 smithi115 bash[46934]: audit 2022-04-22T20:06:52.337716+0000 mon.a (mon.0) 8603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:06:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:53 smithi121 bash[36902]: audit 2022-04-22T20:06:52.702135+0000 mon.a (mon.0) 8604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:06:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:53 smithi121 bash[36902]: cluster 2022-04-22T20:06:52.820375+0000 mgr.y (mgr.34107) 5509 : cluster [DBG] pgmap v6766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:53 smithi121 bash[36902]: audit 2022-04-22T20:06:53.239040+0000 mon.a (mon.0) 8605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[46934]: audit 2022-04-22T20:06:52.702135+0000 mon.a (mon.0) 8604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[46934]: cluster 2022-04-22T20:06:52.820375+0000 mgr.y (mgr.34107) 5509 : cluster [DBG] pgmap v6766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[46934]: audit 2022-04-22T20:06:53.239040+0000 mon.a (mon.0) 8605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[45495]: audit 2022-04-22T20:06:52.702135+0000 mon.a (mon.0) 8604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[45495]: cluster 2022-04-22T20:06:52.820375+0000 mgr.y (mgr.34107) 5509 : cluster [DBG] pgmap v6766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:53 smithi115 bash[45495]: audit 2022-04-22T20:06:53.239040+0000 mon.a (mon.0) 8605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:54.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:06:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:06:54.115Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:06:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:55 smithi121 bash[36902]: cluster 2022-04-22T20:06:54.821490+0000 mgr.y (mgr.34107) 5510 : cluster [DBG] pgmap v6767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:55 smithi121 bash[36902]: audit 2022-04-22T20:06:55.513982+0000 mon.a (mon.0) 8606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:55 smithi115 bash[45495]: cluster 2022-04-22T20:06:54.821490+0000 mgr.y (mgr.34107) 5510 : cluster [DBG] pgmap v6767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:55 smithi115 bash[45495]: audit 2022-04-22T20:06:55.513982+0000 mon.a (mon.0) 8606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:55 smithi115 bash[46934]: cluster 2022-04-22T20:06:54.821490+0000 mgr.y (mgr.34107) 5510 : cluster [DBG] pgmap v6767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:55 smithi115 bash[46934]: audit 2022-04-22T20:06:55.513982+0000 mon.a (mon.0) 8606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:06:56.753 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:06:57.100 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:06:57.101 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:06:57.102 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:06:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:56 smithi121 bash[36902]: audit 2022-04-22T20:06:56.803086+0000 mon.a (mon.0) 8607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:56 smithi115 bash[45495]: audit 2022-04-22T20:06:56.803086+0000 mon.a (mon.0) 8607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:57.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:56 smithi115 bash[46934]: audit 2022-04-22T20:06:56.803086+0000 mon.a (mon.0) 8607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:06:57.506 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:06:57.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:06:57.508 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:06:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:57 smithi121 bash[36902]: audit 2022-04-22T20:06:56.729511+0000 mgr.y (mgr.34107) 5511 : audit [DBG] from='client.51064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:57 smithi121 bash[36902]: cluster 2022-04-22T20:06:56.822442+0000 mgr.y (mgr.34107) 5512 : cluster [DBG] pgmap v6768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:57 smithi121 bash[36902]: audit 2022-04-22T20:06:57.497224+0000 mon.a (mon.0) 8608 : audit [DBG] from='client.? 172.21.15.115:0/1773388214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[45495]: audit 2022-04-22T20:06:56.729511+0000 mgr.y (mgr.34107) 5511 : audit [DBG] from='client.51064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[45495]: cluster 2022-04-22T20:06:56.822442+0000 mgr.y (mgr.34107) 5512 : cluster [DBG] pgmap v6768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[45495]: audit 2022-04-22T20:06:57.497224+0000 mon.a (mon.0) 8608 : audit [DBG] from='client.? 172.21.15.115:0/1773388214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[46934]: audit 2022-04-22T20:06:56.729511+0000 mgr.y (mgr.34107) 5511 : audit [DBG] from='client.51064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[46934]: cluster 2022-04-22T20:06:56.822442+0000 mgr.y (mgr.34107) 5512 : cluster [DBG] pgmap v6768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:06:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:57 smithi115 bash[46934]: audit 2022-04-22T20:06:57.497224+0000 mon.a (mon.0) 8608 : audit [DBG] from='client.? 172.21.15.115:0/1773388214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:06:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:58 smithi121 bash[36902]: audit 2022-04-22T20:06:57.072527+0000 mgr.y (mgr.34107) 5513 : audit [DBG] from='client.51070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:58 smithi121 bash[36902]: audit 2022-04-22T20:06:58.242136+0000 mon.a (mon.0) 8609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:58 smithi115 bash[45495]: audit 2022-04-22T20:06:57.072527+0000 mgr.y (mgr.34107) 5513 : audit [DBG] from='client.51070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:58 smithi115 bash[45495]: audit 2022-04-22T20:06:58.242136+0000 mon.a (mon.0) 8609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:06:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:58 smithi115 bash[46934]: audit 2022-04-22T20:06:57.072527+0000 mgr.y (mgr.34107) 5513 : audit [DBG] from='client.51070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:06:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:58 smithi115 bash[46934]: audit 2022-04-22T20:06:58.242136+0000 mon.a (mon.0) 8609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:06:59 smithi121 bash[36902]: cluster 2022-04-22T20:06:58.823608+0000 mgr.y (mgr.34107) 5514 : cluster [DBG] pgmap v6769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:06:59 smithi115 bash[45495]: cluster 2022-04-22T20:06:58.823608+0000 mgr.y (mgr.34107) 5514 : cluster [DBG] pgmap v6769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:06:59 smithi115 bash[46934]: cluster 2022-04-22T20:06:58.823608+0000 mgr.y (mgr.34107) 5514 : cluster [DBG] pgmap v6769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:00 smithi121 bash[36902]: audit 2022-04-22T20:07:00.517537+0000 mon.a (mon.0) 8610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:00 smithi115 bash[45495]: audit 2022-04-22T20:07:00.517537+0000 mon.a (mon.0) 8610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:00 smithi115 bash[46934]: audit 2022-04-22T20:07:00.517537+0000 mon.a (mon.0) 8610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:02.243 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:01 smithi115 bash[45495]: cluster 2022-04-22T20:07:00.824597+0000 mgr.y (mgr.34107) 5515 : cluster [DBG] pgmap v6770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:02.243 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:01 smithi115 bash[45495]: audit 2022-04-22T20:07:01.805675+0000 mon.a (mon.0) 8611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:02.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:01 smithi115 bash[46934]: cluster 2022-04-22T20:07:00.824597+0000 mgr.y (mgr.34107) 5515 : cluster [DBG] pgmap v6770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:02.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:01 smithi115 bash[46934]: audit 2022-04-22T20:07:01.805675+0000 mon.a (mon.0) 8611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:01 smithi121 bash[36902]: cluster 2022-04-22T20:07:00.824597+0000 mgr.y (mgr.34107) 5515 : cluster [DBG] pgmap v6770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:01 smithi121 bash[36902]: audit 2022-04-22T20:07:01.805675+0000 mon.a (mon.0) 8611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:02.235Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:03 smithi121 bash[36902]: cluster 2022-04-22T20:07:02.825710+0000 mgr.y (mgr.34107) 5516 : cluster [DBG] pgmap v6771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:04.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:03 smithi121 bash[36902]: audit 2022-04-22T20:07:03.245649+0000 mon.a (mon.0) 8612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:04.117Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:03 smithi115 bash[45495]: cluster 2022-04-22T20:07:02.825710+0000 mgr.y (mgr.34107) 5516 : cluster [DBG] pgmap v6771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:03 smithi115 bash[45495]: audit 2022-04-22T20:07:03.245649+0000 mon.a (mon.0) 8612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:03 smithi115 bash[46934]: cluster 2022-04-22T20:07:02.825710+0000 mgr.y (mgr.34107) 5516 : cluster [DBG] pgmap v6771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:04.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:03 smithi115 bash[46934]: audit 2022-04-22T20:07:03.245649+0000 mon.a (mon.0) 8612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:05 smithi121 bash[36902]: cluster 2022-04-22T20:07:04.826812+0000 mgr.y (mgr.34107) 5517 : cluster [DBG] pgmap v6772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:06.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:05 smithi121 bash[36902]: audit 2022-04-22T20:07:05.521289+0000 mon.a (mon.0) 8613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:05 smithi115 bash[45495]: cluster 2022-04-22T20:07:04.826812+0000 mgr.y (mgr.34107) 5517 : cluster [DBG] pgmap v6772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:05 smithi115 bash[45495]: audit 2022-04-22T20:07:05.521289+0000 mon.a (mon.0) 8613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:05 smithi115 bash[46934]: cluster 2022-04-22T20:07:04.826812+0000 mgr.y (mgr.34107) 5517 : cluster [DBG] pgmap v6772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:05 smithi115 bash[46934]: audit 2022-04-22T20:07:05.521289+0000 mon.a (mon.0) 8613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:06 smithi121 bash[36902]: audit 2022-04-22T20:07:06.808773+0000 mon.a (mon.0) 8614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:06 smithi115 bash[45495]: audit 2022-04-22T20:07:06.808773+0000 mon.a (mon.0) 8614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:07.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:06 smithi115 bash[46934]: audit 2022-04-22T20:07:06.808773+0000 mon.a (mon.0) 8614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:07 smithi121 bash[36902]: cluster 2022-04-22T20:07:06.827940+0000 mgr.y (mgr.34107) 5518 : cluster [DBG] pgmap v6773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:07 smithi115 bash[45495]: cluster 2022-04-22T20:07:06.827940+0000 mgr.y (mgr.34107) 5518 : cluster [DBG] pgmap v6773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:07 smithi115 bash[46934]: cluster 2022-04-22T20:07:06.827940+0000 mgr.y (mgr.34107) 5518 : cluster [DBG] pgmap v6773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:08 smithi121 bash[36902]: audit 2022-04-22T20:07:08.248557+0000 mon.a (mon.0) 8615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:08 smithi115 bash[45495]: audit 2022-04-22T20:07:08.248557+0000 mon.a (mon.0) 8615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:08 smithi115 bash[46934]: audit 2022-04-22T20:07:08.248557+0000 mon.a (mon.0) 8615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:09 smithi121 bash[36902]: cluster 2022-04-22T20:07:08.829155+0000 mgr.y (mgr.34107) 5519 : cluster [DBG] pgmap v6774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:10.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:09 smithi115 bash[45495]: cluster 2022-04-22T20:07:08.829155+0000 mgr.y (mgr.34107) 5519 : cluster [DBG] pgmap v6774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:10.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:09 smithi115 bash[46934]: cluster 2022-04-22T20:07:08.829155+0000 mgr.y (mgr.34107) 5519 : cluster [DBG] pgmap v6774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:10 smithi121 bash[36902]: audit 2022-04-22T20:07:10.525146+0000 mon.a (mon.0) 8616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:10 smithi115 bash[45495]: audit 2022-04-22T20:07:10.525146+0000 mon.a (mon.0) 8616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:10 smithi115 bash[46934]: audit 2022-04-22T20:07:10.525146+0000 mon.a (mon.0) 8616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:12.245 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:11 smithi115 bash[45495]: cluster 2022-04-22T20:07:10.830227+0000 mgr.y (mgr.34107) 5520 : cluster [DBG] pgmap v6775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:12.245 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:11 smithi115 bash[45495]: audit 2022-04-22T20:07:11.811905+0000 mon.a (mon.0) 8617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:12.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:11 smithi115 bash[46934]: cluster 2022-04-22T20:07:10.830227+0000 mgr.y (mgr.34107) 5520 : cluster [DBG] pgmap v6775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:12.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:11 smithi115 bash[46934]: audit 2022-04-22T20:07:11.811905+0000 mon.a (mon.0) 8617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:11 smithi121 bash[36902]: cluster 2022-04-22T20:07:10.830227+0000 mgr.y (mgr.34107) 5520 : cluster [DBG] pgmap v6775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:11 smithi121 bash[36902]: audit 2022-04-22T20:07:11.811905+0000 mon.a (mon.0) 8617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:12.236Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:12 smithi121 bash[36902]: cluster 2022-04-22T20:07:12.831358+0000 mgr.y (mgr.34107) 5521 : cluster [DBG] pgmap v6776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:12 smithi115 bash[45495]: cluster 2022-04-22T20:07:12.831358+0000 mgr.y (mgr.34107) 5521 : cluster [DBG] pgmap v6776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:12 smithi115 bash[46934]: cluster 2022-04-22T20:07:12.831358+0000 mgr.y (mgr.34107) 5521 : cluster [DBG] pgmap v6776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:13 smithi121 bash[36902]: audit 2022-04-22T20:07:13.252153+0000 mon.a (mon.0) 8618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:13 smithi115 bash[45495]: audit 2022-04-22T20:07:13.252153+0000 mon.a (mon.0) 8618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:14.118Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:13 smithi115 bash[46934]: audit 2022-04-22T20:07:13.252153+0000 mon.a (mon.0) 8618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:14 smithi121 bash[36902]: cluster 2022-04-22T20:07:14.832398+0000 mgr.y (mgr.34107) 5522 : cluster [DBG] pgmap v6777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:14 smithi115 bash[45495]: cluster 2022-04-22T20:07:14.832398+0000 mgr.y (mgr.34107) 5522 : cluster [DBG] pgmap v6777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:14 smithi115 bash[46934]: cluster 2022-04-22T20:07:14.832398+0000 mgr.y (mgr.34107) 5522 : cluster [DBG] pgmap v6777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:15 smithi121 bash[36902]: audit 2022-04-22T20:07:15.133524+0000 mon.a (mon.0) 8619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:07:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:15 smithi121 bash[36902]: audit 2022-04-22T20:07:15.136491+0000 mon.a (mon.0) 8620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:07:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:15 smithi121 bash[36902]: audit 2022-04-22T20:07:15.528987+0000 mon.a (mon.0) 8621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[45495]: audit 2022-04-22T20:07:15.133524+0000 mon.a (mon.0) 8619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:07:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[45495]: audit 2022-04-22T20:07:15.136491+0000 mon.a (mon.0) 8620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:07:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[45495]: audit 2022-04-22T20:07:15.528987+0000 mon.a (mon.0) 8621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[46934]: audit 2022-04-22T20:07:15.133524+0000 mon.a (mon.0) 8619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:07:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[46934]: audit 2022-04-22T20:07:15.136491+0000 mon.a (mon.0) 8620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:07:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:15 smithi115 bash[46934]: audit 2022-04-22T20:07:15.528987+0000 mon.a (mon.0) 8621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:16 smithi121 bash[36902]: audit 2022-04-22T20:07:16.814999+0000 mon.a (mon.0) 8622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:16 smithi121 bash[36902]: cluster 2022-04-22T20:07:16.833456+0000 mgr.y (mgr.34107) 5523 : cluster [DBG] pgmap v6778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:16 smithi115 bash[45495]: audit 2022-04-22T20:07:16.814999+0000 mon.a (mon.0) 8622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:16 smithi115 bash[45495]: cluster 2022-04-22T20:07:16.833456+0000 mgr.y (mgr.34107) 5523 : cluster [DBG] pgmap v6778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:16 smithi115 bash[46934]: audit 2022-04-22T20:07:16.814999+0000 mon.a (mon.0) 8622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:16 smithi115 bash[46934]: cluster 2022-04-22T20:07:16.833456+0000 mgr.y (mgr.34107) 5523 : cluster [DBG] pgmap v6778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:18 smithi121 bash[36902]: audit 2022-04-22T20:07:18.255489+0000 mon.a (mon.0) 8623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:18 smithi115 bash[45495]: audit 2022-04-22T20:07:18.255489+0000 mon.a (mon.0) 8623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:18 smithi115 bash[46934]: audit 2022-04-22T20:07:18.255489+0000 mon.a (mon.0) 8623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:19 smithi121 bash[36902]: cluster 2022-04-22T20:07:18.834622+0000 mgr.y (mgr.34107) 5524 : cluster [DBG] pgmap v6779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:19 smithi115 bash[45495]: cluster 2022-04-22T20:07:18.834622+0000 mgr.y (mgr.34107) 5524 : cluster [DBG] pgmap v6779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:19 smithi115 bash[46934]: cluster 2022-04-22T20:07:18.834622+0000 mgr.y (mgr.34107) 5524 : cluster [DBG] pgmap v6779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:20 smithi121 bash[36902]: audit 2022-04-22T20:07:20.532673+0000 mon.a (mon.0) 8624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:21.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:20 smithi115 bash[45495]: audit 2022-04-22T20:07:20.532673+0000 mon.a (mon.0) 8624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:20 smithi115 bash[46934]: audit 2022-04-22T20:07:20.532673+0000 mon.a (mon.0) 8624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:21 smithi121 bash[36902]: cluster 2022-04-22T20:07:20.835865+0000 mgr.y (mgr.34107) 5525 : cluster [DBG] pgmap v6780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:21 smithi115 bash[45495]: cluster 2022-04-22T20:07:20.835865+0000 mgr.y (mgr.34107) 5525 : cluster [DBG] pgmap v6780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:21 smithi115 bash[46934]: cluster 2022-04-22T20:07:20.835865+0000 mgr.y (mgr.34107) 5525 : cluster [DBG] pgmap v6780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:22.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:22 smithi121 bash[36902]: audit 2022-04-22T20:07:21.818138+0000 mon.a (mon.0) 8625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:22 smithi115 bash[45495]: audit 2022-04-22T20:07:21.818138+0000 mon.a (mon.0) 8625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:22 smithi115 bash[46934]: audit 2022-04-22T20:07:21.818138+0000 mon.a (mon.0) 8625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:23 smithi121 bash[36902]: cluster 2022-04-22T20:07:22.837019+0000 mgr.y (mgr.34107) 5526 : cluster [DBG] pgmap v6781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:23 smithi121 bash[36902]: audit 2022-04-22T20:07:23.258534+0000 mon.a (mon.0) 8626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:23 smithi115 bash[45495]: cluster 2022-04-22T20:07:22.837019+0000 mgr.y (mgr.34107) 5526 : cluster [DBG] pgmap v6781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:23 smithi115 bash[45495]: audit 2022-04-22T20:07:23.258534+0000 mon.a (mon.0) 8626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:23 smithi115 bash[46934]: cluster 2022-04-22T20:07:22.837019+0000 mgr.y (mgr.34107) 5526 : cluster [DBG] pgmap v6781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:23 smithi115 bash[46934]: audit 2022-04-22T20:07:23.258534+0000 mon.a (mon.0) 8626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:24.118Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:25 smithi121 bash[36902]: cluster 2022-04-22T20:07:24.838182+0000 mgr.y (mgr.34107) 5527 : cluster [DBG] pgmap v6782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:25 smithi121 bash[36902]: audit 2022-04-22T20:07:25.536211+0000 mon.a (mon.0) 8627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:26.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:25 smithi115 bash[46934]: cluster 2022-04-22T20:07:24.838182+0000 mgr.y (mgr.34107) 5527 : cluster [DBG] pgmap v6782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:25 smithi115 bash[46934]: audit 2022-04-22T20:07:25.536211+0000 mon.a (mon.0) 8627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:25 smithi115 bash[45495]: cluster 2022-04-22T20:07:24.838182+0000 mgr.y (mgr.34107) 5527 : cluster [DBG] pgmap v6782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:25 smithi115 bash[45495]: audit 2022-04-22T20:07:25.536211+0000 mon.a (mon.0) 8627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:26 smithi121 bash[36902]: audit 2022-04-22T20:07:26.821324+0000 mon.a (mon.0) 8628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:26 smithi115 bash[45495]: audit 2022-04-22T20:07:26.821324+0000 mon.a (mon.0) 8628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:26 smithi115 bash[46934]: audit 2022-04-22T20:07:26.821324+0000 mon.a (mon.0) 8628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:27.878 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:07:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:07:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:07:28.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:07:28.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:07:28.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:07:28.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:07:28.220 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:07:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:27 smithi121 bash[36902]: cluster 2022-04-22T20:07:26.839236+0000 mgr.y (mgr.34107) 5528 : cluster [DBG] pgmap v6783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:27 smithi115 bash[46934]: cluster 2022-04-22T20:07:26.839236+0000 mgr.y (mgr.34107) 5528 : cluster [DBG] pgmap v6783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:27 smithi115 bash[45495]: cluster 2022-04-22T20:07:26.839236+0000 mgr.y (mgr.34107) 5528 : cluster [DBG] pgmap v6783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:28.627 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:07:28.627 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:07:28.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:07:28.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:28.627 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:07:28.628 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:07:28.629 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:07:28.629 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:07:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:28 smithi121 bash[36902]: audit 2022-04-22T20:07:27.853992+0000 mgr.y (mgr.34107) 5529 : audit [DBG] from='client.51082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:28 smithi121 bash[36902]: audit 2022-04-22T20:07:28.261578+0000 mon.a (mon.0) 8629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:28 smithi121 bash[36902]: audit 2022-04-22T20:07:28.618342+0000 mon.a (mon.0) 8630 : audit [DBG] from='client.? 172.21.15.115:0/1606075135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:07:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[45495]: audit 2022-04-22T20:07:27.853992+0000 mgr.y (mgr.34107) 5529 : audit [DBG] from='client.51082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[45495]: audit 2022-04-22T20:07:28.261578+0000 mon.a (mon.0) 8629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[45495]: audit 2022-04-22T20:07:28.618342+0000 mon.a (mon.0) 8630 : audit [DBG] from='client.? 172.21.15.115:0/1606075135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:07:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[46934]: audit 2022-04-22T20:07:27.853992+0000 mgr.y (mgr.34107) 5529 : audit [DBG] from='client.51082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[46934]: audit 2022-04-22T20:07:28.261578+0000 mon.a (mon.0) 8629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:28 smithi115 bash[46934]: audit 2022-04-22T20:07:28.618342+0000 mon.a (mon.0) 8630 : audit [DBG] from='client.? 172.21.15.115:0/1606075135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:07:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:29 smithi121 bash[36902]: audit 2022-04-22T20:07:28.195081+0000 mgr.y (mgr.34107) 5530 : audit [DBG] from='client.51088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:29 smithi121 bash[36902]: cluster 2022-04-22T20:07:28.840452+0000 mgr.y (mgr.34107) 5531 : cluster [DBG] pgmap v6784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:30.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:29 smithi115 bash[45495]: audit 2022-04-22T20:07:28.195081+0000 mgr.y (mgr.34107) 5530 : audit [DBG] from='client.51088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:30.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:29 smithi115 bash[45495]: cluster 2022-04-22T20:07:28.840452+0000 mgr.y (mgr.34107) 5531 : cluster [DBG] pgmap v6784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:29 smithi115 bash[46934]: audit 2022-04-22T20:07:28.195081+0000 mgr.y (mgr.34107) 5530 : audit [DBG] from='client.51088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:07:30.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:29 smithi115 bash[46934]: cluster 2022-04-22T20:07:28.840452+0000 mgr.y (mgr.34107) 5531 : cluster [DBG] pgmap v6784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:30 smithi121 bash[36902]: audit 2022-04-22T20:07:30.539875+0000 mon.a (mon.0) 8631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:30 smithi115 bash[45495]: audit 2022-04-22T20:07:30.539875+0000 mon.a (mon.0) 8631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:30 smithi115 bash[46934]: audit 2022-04-22T20:07:30.539875+0000 mon.a (mon.0) 8631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:32.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:31 smithi115 bash[45495]: cluster 2022-04-22T20:07:30.841574+0000 mgr.y (mgr.34107) 5532 : cluster [DBG] pgmap v6785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:32.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:31 smithi115 bash[45495]: audit 2022-04-22T20:07:31.824335+0000 mon.a (mon.0) 8632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:32.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:31 smithi115 bash[46934]: cluster 2022-04-22T20:07:30.841574+0000 mgr.y (mgr.34107) 5532 : cluster [DBG] pgmap v6785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:32.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:31 smithi115 bash[46934]: audit 2022-04-22T20:07:31.824335+0000 mon.a (mon.0) 8632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:31 smithi121 bash[36902]: cluster 2022-04-22T20:07:30.841574+0000 mgr.y (mgr.34107) 5532 : cluster [DBG] pgmap v6785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:31 smithi121 bash[36902]: audit 2022-04-22T20:07:31.824335+0000 mon.a (mon.0) 8632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:32.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:33 smithi121 bash[36902]: cluster 2022-04-22T20:07:32.842710+0000 mgr.y (mgr.34107) 5533 : cluster [DBG] pgmap v6786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:33 smithi121 bash[36902]: audit 2022-04-22T20:07:33.264275+0000 mon.a (mon.0) 8633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:34.118Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:33 smithi115 bash[45495]: cluster 2022-04-22T20:07:32.842710+0000 mgr.y (mgr.34107) 5533 : cluster [DBG] pgmap v6786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:33 smithi115 bash[45495]: audit 2022-04-22T20:07:33.264275+0000 mon.a (mon.0) 8633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:33 smithi115 bash[46934]: cluster 2022-04-22T20:07:32.842710+0000 mgr.y (mgr.34107) 5533 : cluster [DBG] pgmap v6786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:33 smithi115 bash[46934]: audit 2022-04-22T20:07:33.264275+0000 mon.a (mon.0) 8633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:35 smithi121 bash[36902]: cluster 2022-04-22T20:07:34.843792+0000 mgr.y (mgr.34107) 5534 : cluster [DBG] pgmap v6787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:35 smithi121 bash[36902]: audit 2022-04-22T20:07:35.545532+0000 mon.a (mon.0) 8634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:35 smithi115 bash[45495]: cluster 2022-04-22T20:07:34.843792+0000 mgr.y (mgr.34107) 5534 : cluster [DBG] pgmap v6787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:35 smithi115 bash[45495]: audit 2022-04-22T20:07:35.545532+0000 mon.a (mon.0) 8634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:35 smithi115 bash[46934]: cluster 2022-04-22T20:07:34.843792+0000 mgr.y (mgr.34107) 5534 : cluster [DBG] pgmap v6787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:35 smithi115 bash[46934]: audit 2022-04-22T20:07:35.545532+0000 mon.a (mon.0) 8634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:36 smithi121 bash[36902]: audit 2022-04-22T20:07:36.827566+0000 mon.a (mon.0) 8635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:36 smithi115 bash[45495]: audit 2022-04-22T20:07:36.827566+0000 mon.a (mon.0) 8635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:36 smithi115 bash[46934]: audit 2022-04-22T20:07:36.827566+0000 mon.a (mon.0) 8635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:37 smithi121 bash[36902]: cluster 2022-04-22T20:07:36.844793+0000 mgr.y (mgr.34107) 5535 : cluster [DBG] pgmap v6788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:37 smithi115 bash[45495]: cluster 2022-04-22T20:07:36.844793+0000 mgr.y (mgr.34107) 5535 : cluster [DBG] pgmap v6788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:37 smithi115 bash[46934]: cluster 2022-04-22T20:07:36.844793+0000 mgr.y (mgr.34107) 5535 : cluster [DBG] pgmap v6788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:38 smithi121 bash[36902]: audit 2022-04-22T20:07:38.267337+0000 mon.a (mon.0) 8636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:38 smithi121 bash[36902]: cluster 2022-04-22T20:07:38.845947+0000 mgr.y (mgr.34107) 5536 : cluster [DBG] pgmap v6789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:38 smithi115 bash[46934]: audit 2022-04-22T20:07:38.267337+0000 mon.a (mon.0) 8636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:38 smithi115 bash[46934]: cluster 2022-04-22T20:07:38.845947+0000 mgr.y (mgr.34107) 5536 : cluster [DBG] pgmap v6789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:38 smithi115 bash[45495]: audit 2022-04-22T20:07:38.267337+0000 mon.a (mon.0) 8636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:39.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:38 smithi115 bash[45495]: cluster 2022-04-22T20:07:38.845947+0000 mgr.y (mgr.34107) 5536 : cluster [DBG] pgmap v6789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:40 smithi121 bash[36902]: audit 2022-04-22T20:07:40.549348+0000 mon.a (mon.0) 8637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:40 smithi115 bash[45495]: audit 2022-04-22T20:07:40.549348+0000 mon.a (mon.0) 8637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:40 smithi115 bash[46934]: audit 2022-04-22T20:07:40.549348+0000 mon.a (mon.0) 8637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:41 smithi121 bash[36902]: cluster 2022-04-22T20:07:40.847033+0000 mgr.y (mgr.34107) 5537 : cluster [DBG] pgmap v6790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:41 smithi115 bash[45495]: cluster 2022-04-22T20:07:40.847033+0000 mgr.y (mgr.34107) 5537 : cluster [DBG] pgmap v6790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:41 smithi115 bash[46934]: cluster 2022-04-22T20:07:40.847033+0000 mgr.y (mgr.34107) 5537 : cluster [DBG] pgmap v6790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:42.237Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:42 smithi121 bash[36902]: audit 2022-04-22T20:07:41.830702+0000 mon.a (mon.0) 8638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:42 smithi115 bash[46934]: audit 2022-04-22T20:07:41.830702+0000 mon.a (mon.0) 8638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:43.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:42 smithi115 bash[45495]: audit 2022-04-22T20:07:41.830702+0000 mon.a (mon.0) 8638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:43 smithi121 bash[36902]: cluster 2022-04-22T20:07:42.848194+0000 mgr.y (mgr.34107) 5538 : cluster [DBG] pgmap v6791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:43 smithi121 bash[36902]: audit 2022-04-22T20:07:43.270141+0000 mon.a (mon.0) 8639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:43 smithi115 bash[46934]: cluster 2022-04-22T20:07:42.848194+0000 mgr.y (mgr.34107) 5538 : cluster [DBG] pgmap v6791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:43 smithi115 bash[46934]: audit 2022-04-22T20:07:43.270141+0000 mon.a (mon.0) 8639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:43 smithi115 bash[45495]: cluster 2022-04-22T20:07:42.848194+0000 mgr.y (mgr.34107) 5538 : cluster [DBG] pgmap v6791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:43 smithi115 bash[45495]: audit 2022-04-22T20:07:43.270141+0000 mon.a (mon.0) 8639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:44.119Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:45 smithi121 bash[36902]: cluster 2022-04-22T20:07:44.849302+0000 mgr.y (mgr.34107) 5539 : cluster [DBG] pgmap v6792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:45 smithi121 bash[36902]: audit 2022-04-22T20:07:45.553180+0000 mon.a (mon.0) 8640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:46.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:45 smithi115 bash[45495]: cluster 2022-04-22T20:07:44.849302+0000 mgr.y (mgr.34107) 5539 : cluster [DBG] pgmap v6792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:45 smithi115 bash[45495]: audit 2022-04-22T20:07:45.553180+0000 mon.a (mon.0) 8640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:45 smithi115 bash[46934]: cluster 2022-04-22T20:07:44.849302+0000 mgr.y (mgr.34107) 5539 : cluster [DBG] pgmap v6792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:45 smithi115 bash[46934]: audit 2022-04-22T20:07:45.553180+0000 mon.a (mon.0) 8640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:46 smithi121 bash[36902]: audit 2022-04-22T20:07:46.834087+0000 mon.a (mon.0) 8641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:46 smithi115 bash[45495]: audit 2022-04-22T20:07:46.834087+0000 mon.a (mon.0) 8641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:46 smithi115 bash[46934]: audit 2022-04-22T20:07:46.834087+0000 mon.a (mon.0) 8641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:47 smithi121 bash[36902]: cluster 2022-04-22T20:07:46.850320+0000 mgr.y (mgr.34107) 5540 : cluster [DBG] pgmap v6793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:47 smithi115 bash[45495]: cluster 2022-04-22T20:07:46.850320+0000 mgr.y (mgr.34107) 5540 : cluster [DBG] pgmap v6793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:47 smithi115 bash[46934]: cluster 2022-04-22T20:07:46.850320+0000 mgr.y (mgr.34107) 5540 : cluster [DBG] pgmap v6793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:48 smithi121 bash[36902]: audit 2022-04-22T20:07:48.273286+0000 mon.a (mon.0) 8642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:48 smithi115 bash[45495]: audit 2022-04-22T20:07:48.273286+0000 mon.a (mon.0) 8642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:48 smithi115 bash[46934]: audit 2022-04-22T20:07:48.273286+0000 mon.a (mon.0) 8642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:49 smithi121 bash[36902]: cluster 2022-04-22T20:07:48.851630+0000 mgr.y (mgr.34107) 5541 : cluster [DBG] pgmap v6794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:49 smithi115 bash[45495]: cluster 2022-04-22T20:07:48.851630+0000 mgr.y (mgr.34107) 5541 : cluster [DBG] pgmap v6794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:50.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:49 smithi115 bash[46934]: cluster 2022-04-22T20:07:48.851630+0000 mgr.y (mgr.34107) 5541 : cluster [DBG] pgmap v6794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:50 smithi121 bash[36902]: audit 2022-04-22T20:07:50.556850+0000 mon.a (mon.0) 8643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:50 smithi115 bash[45495]: audit 2022-04-22T20:07:50.556850+0000 mon.a (mon.0) 8643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:50 smithi115 bash[46934]: audit 2022-04-22T20:07:50.556850+0000 mon.a (mon.0) 8643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:52.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:51 smithi115 bash[45495]: cluster 2022-04-22T20:07:50.852681+0000 mgr.y (mgr.34107) 5542 : cluster [DBG] pgmap v6795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:52.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:51 smithi115 bash[45495]: audit 2022-04-22T20:07:51.837215+0000 mon.a (mon.0) 8644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:52.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:51 smithi115 bash[46934]: cluster 2022-04-22T20:07:50.852681+0000 mgr.y (mgr.34107) 5542 : cluster [DBG] pgmap v6795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:52.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:51 smithi115 bash[46934]: audit 2022-04-22T20:07:51.837215+0000 mon.a (mon.0) 8644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:51 smithi121 bash[36902]: cluster 2022-04-22T20:07:50.852681+0000 mgr.y (mgr.34107) 5542 : cluster [DBG] pgmap v6795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:52.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:51 smithi121 bash[36902]: audit 2022-04-22T20:07:51.837215+0000 mon.a (mon.0) 8644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:52.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:52 smithi121 bash[36902]: audit 2022-04-22T20:07:52.712539+0000 mon.a (mon.0) 8645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:07:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:52 smithi121 bash[36902]: audit 2022-04-22T20:07:52.715657+0000 mon.a (mon.0) 8646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:07:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:52 smithi121 bash[36902]: audit 2022-04-22T20:07:52.717423+0000 mon.a (mon.0) 8647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:07:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[45495]: audit 2022-04-22T20:07:52.712539+0000 mon.a (mon.0) 8645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:07:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[45495]: audit 2022-04-22T20:07:52.715657+0000 mon.a (mon.0) 8646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:07:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[45495]: audit 2022-04-22T20:07:52.717423+0000 mon.a (mon.0) 8647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:07:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[46934]: audit 2022-04-22T20:07:52.712539+0000 mon.a (mon.0) 8645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:07:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[46934]: audit 2022-04-22T20:07:52.715657+0000 mon.a (mon.0) 8646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:07:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:52 smithi115 bash[46934]: audit 2022-04-22T20:07:52.717423+0000 mon.a (mon.0) 8647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:07:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:53 smithi121 bash[36902]: cluster 2022-04-22T20:07:52.853869+0000 mgr.y (mgr.34107) 5543 : cluster [DBG] pgmap v6796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:54.279 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:53 smithi121 bash[36902]: audit 2022-04-22T20:07:53.080646+0000 mon.a (mon.0) 8648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:07:54.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:53 smithi121 bash[36902]: audit 2022-04-22T20:07:53.276486+0000 mon.a (mon.0) 8649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:07:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:07:54.120Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:07:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[45495]: cluster 2022-04-22T20:07:52.853869+0000 mgr.y (mgr.34107) 5543 : cluster [DBG] pgmap v6796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[45495]: audit 2022-04-22T20:07:53.080646+0000 mon.a (mon.0) 8648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:07:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[45495]: audit 2022-04-22T20:07:53.276486+0000 mon.a (mon.0) 8649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[46934]: cluster 2022-04-22T20:07:52.853869+0000 mgr.y (mgr.34107) 5543 : cluster [DBG] pgmap v6796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[46934]: audit 2022-04-22T20:07:53.080646+0000 mon.a (mon.0) 8648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:07:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:53 smithi115 bash[46934]: audit 2022-04-22T20:07:53.276486+0000 mon.a (mon.0) 8649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:55 smithi121 bash[36902]: cluster 2022-04-22T20:07:54.855006+0000 mgr.y (mgr.34107) 5544 : cluster [DBG] pgmap v6797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:55 smithi121 bash[36902]: audit 2022-04-22T20:07:55.560892+0000 mon.a (mon.0) 8650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:55 smithi115 bash[45495]: cluster 2022-04-22T20:07:54.855006+0000 mgr.y (mgr.34107) 5544 : cluster [DBG] pgmap v6797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:55 smithi115 bash[45495]: audit 2022-04-22T20:07:55.560892+0000 mon.a (mon.0) 8650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:55 smithi115 bash[46934]: cluster 2022-04-22T20:07:54.855006+0000 mgr.y (mgr.34107) 5544 : cluster [DBG] pgmap v6797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:55 smithi115 bash[46934]: audit 2022-04-22T20:07:55.560892+0000 mon.a (mon.0) 8650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:07:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:56 smithi121 bash[36902]: audit 2022-04-22T20:07:56.840305+0000 mon.a (mon.0) 8651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:56 smithi115 bash[45495]: audit 2022-04-22T20:07:56.840305+0000 mon.a (mon.0) 8651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:56 smithi115 bash[46934]: audit 2022-04-22T20:07:56.840305+0000 mon.a (mon.0) 8651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:07:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:57 smithi121 bash[36902]: cluster 2022-04-22T20:07:56.856081+0000 mgr.y (mgr.34107) 5545 : cluster [DBG] pgmap v6798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:57 smithi115 bash[45495]: cluster 2022-04-22T20:07:56.856081+0000 mgr.y (mgr.34107) 5545 : cluster [DBG] pgmap v6798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:57 smithi115 bash[46934]: cluster 2022-04-22T20:07:56.856081+0000 mgr.y (mgr.34107) 5545 : cluster [DBG] pgmap v6798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:07:58.998 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:07:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:58 smithi121 bash[36902]: audit 2022-04-22T20:07:58.279460+0000 mon.a (mon.0) 8652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:58 smithi115 bash[46934]: audit 2022-04-22T20:07:58.279460+0000 mon.a (mon.0) 8652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:58 smithi115 bash[45495]: audit 2022-04-22T20:07:58.279460+0000 mon.a (mon.0) 8652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:07:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:07:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:07:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:07:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:07:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:07:59.744 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:07:59.744 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:07:59.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:59.745 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:07:59.745 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:07:59.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:59.745 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:07:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:07:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:07:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:07:59.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:07:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:07:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:07:59.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:07:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:08:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:59 smithi121 bash[36902]: cluster 2022-04-22T20:07:58.857181+0000 mgr.y (mgr.34107) 5546 : cluster [DBG] pgmap v6799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:07:59 smithi121 bash[36902]: audit 2022-04-22T20:07:59.734862+0000 mon.a (mon.0) 8653 : audit [DBG] from='client.? 172.21.15.115:0/3255186057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:59 smithi115 bash[45495]: cluster 2022-04-22T20:07:58.857181+0000 mgr.y (mgr.34107) 5546 : cluster [DBG] pgmap v6799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:00.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:07:59 smithi115 bash[45495]: audit 2022-04-22T20:07:59.734862+0000 mon.a (mon.0) 8653 : audit [DBG] from='client.? 172.21.15.115:0/3255186057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:00.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:59 smithi115 bash[46934]: cluster 2022-04-22T20:07:58.857181+0000 mgr.y (mgr.34107) 5546 : cluster [DBG] pgmap v6799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:00.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:07:59 smithi115 bash[46934]: audit 2022-04-22T20:07:59.734862+0000 mon.a (mon.0) 8653 : audit [DBG] from='client.? 172.21.15.115:0/3255186057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:00 smithi121 bash[36902]: audit 2022-04-22T20:07:58.975177+0000 mgr.y (mgr.34107) 5547 : audit [DBG] from='client.41166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:00 smithi121 bash[36902]: audit 2022-04-22T20:07:59.316376+0000 mgr.y (mgr.34107) 5548 : audit [DBG] from='client.51106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:00 smithi121 bash[36902]: audit 2022-04-22T20:08:00.564514+0000 mon.a (mon.0) 8654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:00 smithi121 bash[36902]: cluster 2022-04-22T20:08:00.858274+0000 mgr.y (mgr.34107) 5549 : cluster [DBG] pgmap v6800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[46934]: audit 2022-04-22T20:07:58.975177+0000 mgr.y (mgr.34107) 5547 : audit [DBG] from='client.41166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[46934]: audit 2022-04-22T20:07:59.316376+0000 mgr.y (mgr.34107) 5548 : audit [DBG] from='client.51106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[46934]: audit 2022-04-22T20:08:00.564514+0000 mon.a (mon.0) 8654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[46934]: cluster 2022-04-22T20:08:00.858274+0000 mgr.y (mgr.34107) 5549 : cluster [DBG] pgmap v6800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:01.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[45495]: audit 2022-04-22T20:07:58.975177+0000 mgr.y (mgr.34107) 5547 : audit [DBG] from='client.41166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[45495]: audit 2022-04-22T20:07:59.316376+0000 mgr.y (mgr.34107) 5548 : audit [DBG] from='client.51106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:01.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[45495]: audit 2022-04-22T20:08:00.564514+0000 mon.a (mon.0) 8654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:01.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:00 smithi115 bash[45495]: cluster 2022-04-22T20:08:00.858274+0000 mgr.y (mgr.34107) 5549 : cluster [DBG] pgmap v6800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:02.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:01 smithi115 bash[46934]: audit 2022-04-22T20:08:01.843412+0000 mon.a (mon.0) 8655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:02.249 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:01 smithi115 bash[45495]: audit 2022-04-22T20:08:01.843412+0000 mon.a (mon.0) 8655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:01 smithi121 bash[36902]: audit 2022-04-22T20:08:01.843412+0000 mon.a (mon.0) 8655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:02.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:02 smithi121 bash[36902]: cluster 2022-04-22T20:08:02.859417+0000 mgr.y (mgr.34107) 5550 : cluster [DBG] pgmap v6801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:02 smithi115 bash[46934]: cluster 2022-04-22T20:08:02.859417+0000 mgr.y (mgr.34107) 5550 : cluster [DBG] pgmap v6801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:02 smithi115 bash[45495]: cluster 2022-04-22T20:08:02.859417+0000 mgr.y (mgr.34107) 5550 : cluster [DBG] pgmap v6801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:03 smithi121 bash[36902]: audit 2022-04-22T20:08:03.282537+0000 mon.a (mon.0) 8656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:03 smithi115 bash[46934]: audit 2022-04-22T20:08:03.282537+0000 mon.a (mon.0) 8656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:04.120Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:03 smithi115 bash[45495]: audit 2022-04-22T20:08:03.282537+0000 mon.a (mon.0) 8656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:04 smithi121 bash[36902]: cluster 2022-04-22T20:08:04.860572+0000 mgr.y (mgr.34107) 5551 : cluster [DBG] pgmap v6802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:04 smithi115 bash[45495]: cluster 2022-04-22T20:08:04.860572+0000 mgr.y (mgr.34107) 5551 : cluster [DBG] pgmap v6802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:04 smithi115 bash[46934]: cluster 2022-04-22T20:08:04.860572+0000 mgr.y (mgr.34107) 5551 : cluster [DBG] pgmap v6802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:05 smithi121 bash[36902]: audit 2022-04-22T20:08:05.568249+0000 mon.a (mon.0) 8657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:05 smithi115 bash[45495]: audit 2022-04-22T20:08:05.568249+0000 mon.a (mon.0) 8657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:05 smithi115 bash[46934]: audit 2022-04-22T20:08:05.568249+0000 mon.a (mon.0) 8657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:06 smithi121 bash[36902]: audit 2022-04-22T20:08:06.846756+0000 mon.a (mon.0) 8658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:06 smithi121 bash[36902]: cluster 2022-04-22T20:08:06.861647+0000 mgr.y (mgr.34107) 5552 : cluster [DBG] pgmap v6803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:06 smithi115 bash[46934]: audit 2022-04-22T20:08:06.846756+0000 mon.a (mon.0) 8658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:06 smithi115 bash[46934]: cluster 2022-04-22T20:08:06.861647+0000 mgr.y (mgr.34107) 5552 : cluster [DBG] pgmap v6803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:06 smithi115 bash[45495]: audit 2022-04-22T20:08:06.846756+0000 mon.a (mon.0) 8658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:06 smithi115 bash[45495]: cluster 2022-04-22T20:08:06.861647+0000 mgr.y (mgr.34107) 5552 : cluster [DBG] pgmap v6803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:08 smithi121 bash[36902]: audit 2022-04-22T20:08:08.285740+0000 mon.a (mon.0) 8659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:08 smithi115 bash[45495]: audit 2022-04-22T20:08:08.285740+0000 mon.a (mon.0) 8659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:08 smithi115 bash[46934]: audit 2022-04-22T20:08:08.285740+0000 mon.a (mon.0) 8659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:09 smithi121 bash[36902]: cluster 2022-04-22T20:08:08.862830+0000 mgr.y (mgr.34107) 5553 : cluster [DBG] pgmap v6804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:09 smithi115 bash[45495]: cluster 2022-04-22T20:08:08.862830+0000 mgr.y (mgr.34107) 5553 : cluster [DBG] pgmap v6804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:09 smithi115 bash[46934]: cluster 2022-04-22T20:08:08.862830+0000 mgr.y (mgr.34107) 5553 : cluster [DBG] pgmap v6804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:10 smithi121 bash[36902]: audit 2022-04-22T20:08:10.571998+0000 mon.a (mon.0) 8660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:10 smithi115 bash[45495]: audit 2022-04-22T20:08:10.571998+0000 mon.a (mon.0) 8660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:10 smithi115 bash[46934]: audit 2022-04-22T20:08:10.571998+0000 mon.a (mon.0) 8660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:11 smithi121 bash[36902]: cluster 2022-04-22T20:08:10.863936+0000 mgr.y (mgr.34107) 5554 : cluster [DBG] pgmap v6805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:11 smithi115 bash[45495]: cluster 2022-04-22T20:08:10.863936+0000 mgr.y (mgr.34107) 5554 : cluster [DBG] pgmap v6805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:11 smithi115 bash[46934]: cluster 2022-04-22T20:08:10.863936+0000 mgr.y (mgr.34107) 5554 : cluster [DBG] pgmap v6805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:12.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:12 smithi121 bash[36902]: audit 2022-04-22T20:08:11.849939+0000 mon.a (mon.0) 8661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:13.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:12 smithi115 bash[46934]: audit 2022-04-22T20:08:11.849939+0000 mon.a (mon.0) 8661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:12 smithi115 bash[45495]: audit 2022-04-22T20:08:11.849939+0000 mon.a (mon.0) 8661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:13 smithi121 bash[36902]: cluster 2022-04-22T20:08:12.864980+0000 mgr.y (mgr.34107) 5555 : cluster [DBG] pgmap v6806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:13 smithi121 bash[36902]: audit 2022-04-22T20:08:13.288865+0000 mon.a (mon.0) 8662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:13 smithi115 bash[45495]: cluster 2022-04-22T20:08:12.864980+0000 mgr.y (mgr.34107) 5555 : cluster [DBG] pgmap v6806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:13 smithi115 bash[45495]: audit 2022-04-22T20:08:13.288865+0000 mon.a (mon.0) 8662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:13 smithi115 bash[46934]: cluster 2022-04-22T20:08:12.864980+0000 mgr.y (mgr.34107) 5555 : cluster [DBG] pgmap v6806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:13 smithi115 bash[46934]: audit 2022-04-22T20:08:13.288865+0000 mon.a (mon.0) 8662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:14.120Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:15 smithi121 bash[36902]: cluster 2022-04-22T20:08:14.866096+0000 mgr.y (mgr.34107) 5556 : cluster [DBG] pgmap v6807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:15 smithi121 bash[36902]: audit 2022-04-22T20:08:15.145394+0000 mon.a (mon.0) 8663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:08:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:15 smithi121 bash[36902]: audit 2022-04-22T20:08:15.145775+0000 mon.a (mon.0) 8664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:08:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:15 smithi121 bash[36902]: audit 2022-04-22T20:08:15.575935+0000 mon.a (mon.0) 8665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[45495]: cluster 2022-04-22T20:08:14.866096+0000 mgr.y (mgr.34107) 5556 : cluster [DBG] pgmap v6807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[45495]: audit 2022-04-22T20:08:15.145394+0000 mon.a (mon.0) 8663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[45495]: audit 2022-04-22T20:08:15.145775+0000 mon.a (mon.0) 8664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[45495]: audit 2022-04-22T20:08:15.575935+0000 mon.a (mon.0) 8665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[46934]: cluster 2022-04-22T20:08:14.866096+0000 mgr.y (mgr.34107) 5556 : cluster [DBG] pgmap v6807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[46934]: audit 2022-04-22T20:08:15.145394+0000 mon.a (mon.0) 8663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:08:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[46934]: audit 2022-04-22T20:08:15.145775+0000 mon.a (mon.0) 8664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:08:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:15 smithi115 bash[46934]: audit 2022-04-22T20:08:15.575935+0000 mon.a (mon.0) 8665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:16 smithi121 bash[36902]: audit 2022-04-22T20:08:16.853050+0000 mon.a (mon.0) 8666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:17.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:16 smithi115 bash[45495]: audit 2022-04-22T20:08:16.853050+0000 mon.a (mon.0) 8666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:16 smithi115 bash[46934]: audit 2022-04-22T20:08:16.853050+0000 mon.a (mon.0) 8666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:17 smithi121 bash[36902]: cluster 2022-04-22T20:08:16.867150+0000 mgr.y (mgr.34107) 5557 : cluster [DBG] pgmap v6808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:17 smithi115 bash[45495]: cluster 2022-04-22T20:08:16.867150+0000 mgr.y (mgr.34107) 5557 : cluster [DBG] pgmap v6808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:17 smithi115 bash[46934]: cluster 2022-04-22T20:08:16.867150+0000 mgr.y (mgr.34107) 5557 : cluster [DBG] pgmap v6808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:18 smithi121 bash[36902]: audit 2022-04-22T20:08:18.292005+0000 mon.a (mon.0) 8667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:19.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:18 smithi115 bash[45495]: audit 2022-04-22T20:08:18.292005+0000 mon.a (mon.0) 8667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:19.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:18 smithi115 bash[46934]: audit 2022-04-22T20:08:18.292005+0000 mon.a (mon.0) 8667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:19 smithi121 bash[36902]: cluster 2022-04-22T20:08:18.868367+0000 mgr.y (mgr.34107) 5558 : cluster [DBG] pgmap v6809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:20.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:19 smithi115 bash[45495]: cluster 2022-04-22T20:08:18.868367+0000 mgr.y (mgr.34107) 5558 : cluster [DBG] pgmap v6809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:19 smithi115 bash[46934]: cluster 2022-04-22T20:08:18.868367+0000 mgr.y (mgr.34107) 5558 : cluster [DBG] pgmap v6809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:20 smithi121 bash[36902]: audit 2022-04-22T20:08:20.579612+0000 mon.a (mon.0) 8668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:20 smithi115 bash[45495]: audit 2022-04-22T20:08:20.579612+0000 mon.a (mon.0) 8668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:20 smithi115 bash[46934]: audit 2022-04-22T20:08:20.579612+0000 mon.a (mon.0) 8668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:22.249 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:21 smithi115 bash[45495]: cluster 2022-04-22T20:08:20.869492+0000 mgr.y (mgr.34107) 5559 : cluster [DBG] pgmap v6810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:22.249 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:21 smithi115 bash[45495]: audit 2022-04-22T20:08:21.856109+0000 mon.a (mon.0) 8669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:22.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:21 smithi115 bash[46934]: cluster 2022-04-22T20:08:20.869492+0000 mgr.y (mgr.34107) 5559 : cluster [DBG] pgmap v6810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:22.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:21 smithi115 bash[46934]: audit 2022-04-22T20:08:21.856109+0000 mon.a (mon.0) 8669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:22.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:21 smithi121 bash[36902]: cluster 2022-04-22T20:08:20.869492+0000 mgr.y (mgr.34107) 5559 : cluster [DBG] pgmap v6810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:22.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:21 smithi121 bash[36902]: audit 2022-04-22T20:08:21.856109+0000 mon.a (mon.0) 8669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:22.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:22.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:23 smithi121 bash[36902]: cluster 2022-04-22T20:08:22.870573+0000 mgr.y (mgr.34107) 5560 : cluster [DBG] pgmap v6811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:24.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:23 smithi121 bash[36902]: audit 2022-04-22T20:08:23.295060+0000 mon.a (mon.0) 8670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:24.121Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:23 smithi115 bash[45495]: cluster 2022-04-22T20:08:22.870573+0000 mgr.y (mgr.34107) 5560 : cluster [DBG] pgmap v6811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:23 smithi115 bash[45495]: audit 2022-04-22T20:08:23.295060+0000 mon.a (mon.0) 8670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:23 smithi115 bash[46934]: cluster 2022-04-22T20:08:22.870573+0000 mgr.y (mgr.34107) 5560 : cluster [DBG] pgmap v6811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:23 smithi115 bash[46934]: audit 2022-04-22T20:08:23.295060+0000 mon.a (mon.0) 8670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:24 smithi121 bash[36902]: cluster 2022-04-22T20:08:24.871708+0000 mgr.y (mgr.34107) 5561 : cluster [DBG] pgmap v6812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:25.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:24 smithi115 bash[45495]: cluster 2022-04-22T20:08:24.871708+0000 mgr.y (mgr.34107) 5561 : cluster [DBG] pgmap v6812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:25.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:24 smithi115 bash[46934]: cluster 2022-04-22T20:08:24.871708+0000 mgr.y (mgr.34107) 5561 : cluster [DBG] pgmap v6812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:25 smithi121 bash[36902]: audit 2022-04-22T20:08:25.583097+0000 mon.a (mon.0) 8671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:25 smithi115 bash[45495]: audit 2022-04-22T20:08:25.583097+0000 mon.a (mon.0) 8671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:25 smithi115 bash[46934]: audit 2022-04-22T20:08:25.583097+0000 mon.a (mon.0) 8671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:26 smithi121 bash[36902]: audit 2022-04-22T20:08:26.859323+0000 mon.a (mon.0) 8672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:26 smithi121 bash[36902]: cluster 2022-04-22T20:08:26.872812+0000 mgr.y (mgr.34107) 5562 : cluster [DBG] pgmap v6813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:26 smithi115 bash[45495]: audit 2022-04-22T20:08:26.859323+0000 mon.a (mon.0) 8672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:26 smithi115 bash[45495]: cluster 2022-04-22T20:08:26.872812+0000 mgr.y (mgr.34107) 5562 : cluster [DBG] pgmap v6813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:27.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:26 smithi115 bash[46934]: audit 2022-04-22T20:08:26.859323+0000 mon.a (mon.0) 8672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:27.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:26 smithi115 bash[46934]: cluster 2022-04-22T20:08:26.872812+0000 mgr.y (mgr.34107) 5562 : cluster [DBG] pgmap v6813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:28 smithi121 bash[36902]: audit 2022-04-22T20:08:28.298202+0000 mon.a (mon.0) 8673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:28 smithi115 bash[45495]: audit 2022-04-22T20:08:28.298202+0000 mon.a (mon.0) 8673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:28 smithi115 bash[46934]: audit 2022-04-22T20:08:28.298202+0000 mon.a (mon.0) 8673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:29 smithi121 bash[36902]: cluster 2022-04-22T20:08:28.874025+0000 mgr.y (mgr.34107) 5563 : cluster [DBG] pgmap v6814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:29 smithi115 bash[45495]: cluster 2022-04-22T20:08:28.874025+0000 mgr.y (mgr.34107) 5563 : cluster [DBG] pgmap v6814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:29 smithi115 bash[46934]: cluster 2022-04-22T20:08:28.874025+0000 mgr.y (mgr.34107) 5563 : cluster [DBG] pgmap v6814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:30.125 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:08:30.471 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:08:30.471 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:08:30.471 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:08:30.471 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:08:30.471 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:08:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:08:30.473 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:08:30.876 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:08:30.877 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:08:30.878 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:08:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:30 smithi121 bash[36902]: audit 2022-04-22T20:08:30.586632+0000 mon.a (mon.0) 8674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:30 smithi115 bash[45495]: audit 2022-04-22T20:08:30.586632+0000 mon.a (mon.0) 8674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:30 smithi115 bash[46934]: audit 2022-04-22T20:08:30.586632+0000 mon.a (mon.0) 8674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:31 smithi121 bash[36902]: audit 2022-04-22T20:08:30.101685+0000 mgr.y (mgr.34107) 5564 : audit [DBG] from='client.51118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:31 smithi121 bash[36902]: audit 2022-04-22T20:08:30.443738+0000 mgr.y (mgr.34107) 5565 : audit [DBG] from='client.51124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:31 smithi121 bash[36902]: audit 2022-04-22T20:08:30.868068+0000 mon.a (mon.0) 8675 : audit [DBG] from='client.? 172.21.15.115:0/3286473316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:31 smithi121 bash[36902]: cluster 2022-04-22T20:08:30.874752+0000 mgr.y (mgr.34107) 5566 : cluster [DBG] pgmap v6815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[45495]: audit 2022-04-22T20:08:30.101685+0000 mgr.y (mgr.34107) 5564 : audit [DBG] from='client.51118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[45495]: audit 2022-04-22T20:08:30.443738+0000 mgr.y (mgr.34107) 5565 : audit [DBG] from='client.51124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[45495]: audit 2022-04-22T20:08:30.868068+0000 mon.a (mon.0) 8675 : audit [DBG] from='client.? 172.21.15.115:0/3286473316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[45495]: cluster 2022-04-22T20:08:30.874752+0000 mgr.y (mgr.34107) 5566 : cluster [DBG] pgmap v6815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[46934]: audit 2022-04-22T20:08:30.101685+0000 mgr.y (mgr.34107) 5564 : audit [DBG] from='client.51118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[46934]: audit 2022-04-22T20:08:30.443738+0000 mgr.y (mgr.34107) 5565 : audit [DBG] from='client.51124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:08:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[46934]: audit 2022-04-22T20:08:30.868068+0000 mon.a (mon.0) 8675 : audit [DBG] from='client.? 172.21.15.115:0/3286473316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:08:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:31 smithi115 bash[46934]: cluster 2022-04-22T20:08:30.874752+0000 mgr.y (mgr.34107) 5566 : cluster [DBG] pgmap v6815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:32.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:32 smithi121 bash[36902]: audit 2022-04-22T20:08:31.862380+0000 mon.a (mon.0) 8676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:32 smithi115 bash[45495]: audit 2022-04-22T20:08:31.862380+0000 mon.a (mon.0) 8676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:32 smithi115 bash[46934]: audit 2022-04-22T20:08:31.862380+0000 mon.a (mon.0) 8676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:33 smithi121 bash[36902]: cluster 2022-04-22T20:08:32.875869+0000 mgr.y (mgr.34107) 5567 : cluster [DBG] pgmap v6816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:33 smithi121 bash[36902]: audit 2022-04-22T20:08:33.301239+0000 mon.a (mon.0) 8677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:33 smithi115 bash[45495]: cluster 2022-04-22T20:08:32.875869+0000 mgr.y (mgr.34107) 5567 : cluster [DBG] pgmap v6816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:33 smithi115 bash[45495]: audit 2022-04-22T20:08:33.301239+0000 mon.a (mon.0) 8677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:33 smithi115 bash[46934]: cluster 2022-04-22T20:08:32.875869+0000 mgr.y (mgr.34107) 5567 : cluster [DBG] pgmap v6816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:33 smithi115 bash[46934]: audit 2022-04-22T20:08:33.301239+0000 mon.a (mon.0) 8677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:34.122Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:35 smithi121 bash[36902]: cluster 2022-04-22T20:08:34.876944+0000 mgr.y (mgr.34107) 5568 : cluster [DBG] pgmap v6817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:35 smithi121 bash[36902]: audit 2022-04-22T20:08:35.590280+0000 mon.a (mon.0) 8678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:35 smithi115 bash[45495]: cluster 2022-04-22T20:08:34.876944+0000 mgr.y (mgr.34107) 5568 : cluster [DBG] pgmap v6817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:35 smithi115 bash[45495]: audit 2022-04-22T20:08:35.590280+0000 mon.a (mon.0) 8678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:35 smithi115 bash[46934]: cluster 2022-04-22T20:08:34.876944+0000 mgr.y (mgr.34107) 5568 : cluster [DBG] pgmap v6817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:35 smithi115 bash[46934]: audit 2022-04-22T20:08:35.590280+0000 mon.a (mon.0) 8678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:36 smithi121 bash[36902]: audit 2022-04-22T20:08:36.865576+0000 mon.a (mon.0) 8679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:36 smithi115 bash[45495]: audit 2022-04-22T20:08:36.865576+0000 mon.a (mon.0) 8679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:36 smithi115 bash[46934]: audit 2022-04-22T20:08:36.865576+0000 mon.a (mon.0) 8679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:37 smithi121 bash[36902]: cluster 2022-04-22T20:08:36.877947+0000 mgr.y (mgr.34107) 5569 : cluster [DBG] pgmap v6818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:37 smithi115 bash[45495]: cluster 2022-04-22T20:08:36.877947+0000 mgr.y (mgr.34107) 5569 : cluster [DBG] pgmap v6818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:37 smithi115 bash[46934]: cluster 2022-04-22T20:08:36.877947+0000 mgr.y (mgr.34107) 5569 : cluster [DBG] pgmap v6818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:38 smithi121 bash[36902]: audit 2022-04-22T20:08:38.304263+0000 mon.a (mon.0) 8680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:38 smithi115 bash[45495]: audit 2022-04-22T20:08:38.304263+0000 mon.a (mon.0) 8680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:38 smithi115 bash[46934]: audit 2022-04-22T20:08:38.304263+0000 mon.a (mon.0) 8680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:39 smithi121 bash[36902]: cluster 2022-04-22T20:08:38.879095+0000 mgr.y (mgr.34107) 5570 : cluster [DBG] pgmap v6819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:40.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:39 smithi115 bash[45495]: cluster 2022-04-22T20:08:38.879095+0000 mgr.y (mgr.34107) 5570 : cluster [DBG] pgmap v6819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:40.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:39 smithi115 bash[46934]: cluster 2022-04-22T20:08:38.879095+0000 mgr.y (mgr.34107) 5570 : cluster [DBG] pgmap v6819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:40 smithi121 bash[36902]: audit 2022-04-22T20:08:40.593846+0000 mon.a (mon.0) 8681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:40 smithi115 bash[45495]: audit 2022-04-22T20:08:40.593846+0000 mon.a (mon.0) 8681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:40 smithi115 bash[46934]: audit 2022-04-22T20:08:40.593846+0000 mon.a (mon.0) 8681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:42.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:41 smithi115 bash[45495]: cluster 2022-04-22T20:08:40.880186+0000 mgr.y (mgr.34107) 5571 : cluster [DBG] pgmap v6820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:42.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:41 smithi115 bash[45495]: audit 2022-04-22T20:08:41.868802+0000 mon.a (mon.0) 8682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:42.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:41 smithi115 bash[46934]: cluster 2022-04-22T20:08:40.880186+0000 mgr.y (mgr.34107) 5571 : cluster [DBG] pgmap v6820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:42.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:41 smithi115 bash[46934]: audit 2022-04-22T20:08:41.868802+0000 mon.a (mon.0) 8682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:41 smithi121 bash[36902]: cluster 2022-04-22T20:08:40.880186+0000 mgr.y (mgr.34107) 5571 : cluster [DBG] pgmap v6820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:41 smithi121 bash[36902]: audit 2022-04-22T20:08:41.868802+0000 mon.a (mon.0) 8682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:42.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:43 smithi115 bash[45495]: cluster 2022-04-22T20:08:42.881302+0000 mgr.y (mgr.34107) 5572 : cluster [DBG] pgmap v6821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:43 smithi115 bash[46934]: cluster 2022-04-22T20:08:42.881302+0000 mgr.y (mgr.34107) 5572 : cluster [DBG] pgmap v6821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:43.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:43 smithi121 bash[36902]: cluster 2022-04-22T20:08:42.881302+0000 mgr.y (mgr.34107) 5572 : cluster [DBG] pgmap v6821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:44 smithi115 bash[45495]: audit 2022-04-22T20:08:43.307310+0000 mon.a (mon.0) 8683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:44 smithi115 bash[46934]: audit 2022-04-22T20:08:43.307310+0000 mon.a (mon.0) 8683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:44.123Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:44 smithi121 bash[36902]: audit 2022-04-22T20:08:43.307310+0000 mon.a (mon.0) 8683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:45 smithi115 bash[45495]: cluster 2022-04-22T20:08:44.882507+0000 mgr.y (mgr.34107) 5573 : cluster [DBG] pgmap v6822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:45 smithi115 bash[46934]: cluster 2022-04-22T20:08:44.882507+0000 mgr.y (mgr.34107) 5573 : cluster [DBG] pgmap v6822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:45 smithi121 bash[36902]: cluster 2022-04-22T20:08:44.882507+0000 mgr.y (mgr.34107) 5573 : cluster [DBG] pgmap v6822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:46 smithi115 bash[45495]: audit 2022-04-22T20:08:45.597522+0000 mon.a (mon.0) 8684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:46 smithi115 bash[46934]: audit 2022-04-22T20:08:45.597522+0000 mon.a (mon.0) 8684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:46 smithi121 bash[36902]: audit 2022-04-22T20:08:45.597522+0000 mon.a (mon.0) 8684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:47 smithi115 bash[46934]: audit 2022-04-22T20:08:46.872149+0000 mon.a (mon.0) 8685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:47 smithi115 bash[46934]: cluster 2022-04-22T20:08:46.883638+0000 mgr.y (mgr.34107) 5574 : cluster [DBG] pgmap v6823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:47 smithi115 bash[45495]: audit 2022-04-22T20:08:46.872149+0000 mon.a (mon.0) 8685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:47 smithi115 bash[45495]: cluster 2022-04-22T20:08:46.883638+0000 mgr.y (mgr.34107) 5574 : cluster [DBG] pgmap v6823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:47 smithi121 bash[36902]: audit 2022-04-22T20:08:46.872149+0000 mon.a (mon.0) 8685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:47 smithi121 bash[36902]: cluster 2022-04-22T20:08:46.883638+0000 mgr.y (mgr.34107) 5574 : cluster [DBG] pgmap v6823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:48 smithi121 bash[36902]: audit 2022-04-22T20:08:48.310222+0000 mon.a (mon.0) 8686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:48 smithi115 bash[45495]: audit 2022-04-22T20:08:48.310222+0000 mon.a (mon.0) 8686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:48 smithi115 bash[46934]: audit 2022-04-22T20:08:48.310222+0000 mon.a (mon.0) 8686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:49 smithi121 bash[36902]: cluster 2022-04-22T20:08:48.884749+0000 mgr.y (mgr.34107) 5575 : cluster [DBG] pgmap v6824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:50.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:49 smithi115 bash[46934]: cluster 2022-04-22T20:08:48.884749+0000 mgr.y (mgr.34107) 5575 : cluster [DBG] pgmap v6824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:49 smithi115 bash[45495]: cluster 2022-04-22T20:08:48.884749+0000 mgr.y (mgr.34107) 5575 : cluster [DBG] pgmap v6824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:50 smithi121 bash[36902]: audit 2022-04-22T20:08:50.601279+0000 mon.a (mon.0) 8687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:50 smithi115 bash[46934]: audit 2022-04-22T20:08:50.601279+0000 mon.a (mon.0) 8687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:50 smithi115 bash[45495]: audit 2022-04-22T20:08:50.601279+0000 mon.a (mon.0) 8687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:51 smithi121 bash[36902]: cluster 2022-04-22T20:08:50.885831+0000 mgr.y (mgr.34107) 5576 : cluster [DBG] pgmap v6825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:51 smithi115 bash[46934]: cluster 2022-04-22T20:08:50.885831+0000 mgr.y (mgr.34107) 5576 : cluster [DBG] pgmap v6825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:51 smithi115 bash[45495]: cluster 2022-04-22T20:08:50.885831+0000 mgr.y (mgr.34107) 5576 : cluster [DBG] pgmap v6825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:52.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:52 smithi121 bash[36902]: audit 2022-04-22T20:08:51.875651+0000 mon.a (mon.0) 8688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:52 smithi115 bash[45495]: audit 2022-04-22T20:08:51.875651+0000 mon.a (mon.0) 8688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:52 smithi115 bash[46934]: audit 2022-04-22T20:08:51.875651+0000 mon.a (mon.0) 8688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: cluster 2022-04-22T20:08:52.886943+0000 mgr.y (mgr.34107) 5577 : cluster [DBG] pgmap v6826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: audit 2022-04-22T20:08:53.088908+0000 mon.a (mon.0) 8689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:08:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: audit 2022-04-22T20:08:53.091829+0000 mon.a (mon.0) 8690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:08:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: audit 2022-04-22T20:08:53.093540+0000 mon.a (mon.0) 8691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:08:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: audit 2022-04-22T20:08:53.313212+0000 mon.a (mon.0) 8692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:53 smithi121 bash[36902]: audit 2022-04-22T20:08:53.449819+0000 mon.a (mon.0) 8693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:08:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: cluster 2022-04-22T20:08:52.886943+0000 mgr.y (mgr.34107) 5577 : cluster [DBG] pgmap v6826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:54.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: audit 2022-04-22T20:08:53.088908+0000 mon.a (mon.0) 8689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: audit 2022-04-22T20:08:53.091829+0000 mon.a (mon.0) 8690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: audit 2022-04-22T20:08:53.093540+0000 mon.a (mon.0) 8691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: audit 2022-04-22T20:08:53.313212+0000 mon.a (mon.0) 8692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[45495]: audit 2022-04-22T20:08:53.449819+0000 mon.a (mon.0) 8693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: cluster 2022-04-22T20:08:52.886943+0000 mgr.y (mgr.34107) 5577 : cluster [DBG] pgmap v6826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:54.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: audit 2022-04-22T20:08:53.088908+0000 mon.a (mon.0) 8689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:08:54.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: audit 2022-04-22T20:08:53.091829+0000 mon.a (mon.0) 8690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:08:54.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: audit 2022-04-22T20:08:53.093540+0000 mon.a (mon.0) 8691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:08:54.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: audit 2022-04-22T20:08:53.313212+0000 mon.a (mon.0) 8692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:54.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:53 smithi115 bash[46934]: audit 2022-04-22T20:08:53.449819+0000 mon.a (mon.0) 8693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:08:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:08:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:08:54.122Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:08:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:55 smithi121 bash[36902]: cluster 2022-04-22T20:08:54.888023+0000 mgr.y (mgr.34107) 5578 : cluster [DBG] pgmap v6827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:55 smithi121 bash[36902]: audit 2022-04-22T20:08:55.604507+0000 mon.a (mon.0) 8694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:55 smithi115 bash[45495]: cluster 2022-04-22T20:08:54.888023+0000 mgr.y (mgr.34107) 5578 : cluster [DBG] pgmap v6827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:56.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:55 smithi115 bash[45495]: audit 2022-04-22T20:08:55.604507+0000 mon.a (mon.0) 8694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:56.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:55 smithi115 bash[46934]: cluster 2022-04-22T20:08:54.888023+0000 mgr.y (mgr.34107) 5578 : cluster [DBG] pgmap v6827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:56.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:55 smithi115 bash[46934]: audit 2022-04-22T20:08:55.604507+0000 mon.a (mon.0) 8694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:08:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:56 smithi121 bash[36902]: audit 2022-04-22T20:08:56.878848+0000 mon.a (mon.0) 8695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:56 smithi115 bash[46934]: audit 2022-04-22T20:08:56.878848+0000 mon.a (mon.0) 8695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:57.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:56 smithi115 bash[45495]: audit 2022-04-22T20:08:56.878848+0000 mon.a (mon.0) 8695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:08:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:57 smithi121 bash[36902]: cluster 2022-04-22T20:08:56.888893+0000 mgr.y (mgr.34107) 5579 : cluster [DBG] pgmap v6828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:58.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:57 smithi115 bash[45495]: cluster 2022-04-22T20:08:56.888893+0000 mgr.y (mgr.34107) 5579 : cluster [DBG] pgmap v6828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:58.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:57 smithi115 bash[46934]: cluster 2022-04-22T20:08:56.888893+0000 mgr.y (mgr.34107) 5579 : cluster [DBG] pgmap v6828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:08:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:58 smithi121 bash[36902]: audit 2022-04-22T20:08:58.316061+0000 mon.a (mon.0) 8696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:58 smithi115 bash[45495]: audit 2022-04-22T20:08:58.316061+0000 mon.a (mon.0) 8696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:08:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:58 smithi115 bash[46934]: audit 2022-04-22T20:08:58.316061+0000 mon.a (mon.0) 8696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:08:59 smithi121 bash[36902]: cluster 2022-04-22T20:08:58.890042+0000 mgr.y (mgr.34107) 5580 : cluster [DBG] pgmap v6829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:08:59 smithi115 bash[45495]: cluster 2022-04-22T20:08:58.890042+0000 mgr.y (mgr.34107) 5580 : cluster [DBG] pgmap v6829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:00.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:08:59 smithi115 bash[46934]: cluster 2022-04-22T20:08:58.890042+0000 mgr.y (mgr.34107) 5580 : cluster [DBG] pgmap v6829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:01.256 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:09:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:00 smithi121 bash[36902]: audit 2022-04-22T20:09:00.608342+0000 mon.a (mon.0) 8697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:01.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:00 smithi115 bash[46934]: audit 2022-04-22T20:09:00.608342+0000 mon.a (mon.0) 8697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:00 smithi115 bash[45495]: audit 2022-04-22T20:09:00.608342+0000 mon.a (mon.0) 8697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:09:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:09:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:09:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:09:01.599 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:09:01.600 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:09:01.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:09:01.602 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:09:02.004 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:09:02.004 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:09:02.004 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:09:02.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:09:02.005 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:09:02.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:02.006 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:09:02.006 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:09:02.006 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:09:02.006 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:09:02.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[46934]: cluster 2022-04-22T20:09:00.891063+0000 mgr.y (mgr.34107) 5581 : cluster [DBG] pgmap v6830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:02.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[46934]: audit 2022-04-22T20:09:01.228538+0000 mgr.y (mgr.34107) 5582 : audit [DBG] from='client.51136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[46934]: audit 2022-04-22T20:09:01.570257+0000 mgr.y (mgr.34107) 5583 : audit [DBG] from='client.51142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[46934]: audit 2022-04-22T20:09:01.881989+0000 mon.a (mon.0) 8698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:02.255 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[45495]: cluster 2022-04-22T20:09:00.891063+0000 mgr.y (mgr.34107) 5581 : cluster [DBG] pgmap v6830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:02.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[45495]: audit 2022-04-22T20:09:01.228538+0000 mgr.y (mgr.34107) 5582 : audit [DBG] from='client.51136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[45495]: audit 2022-04-22T20:09:01.570257+0000 mgr.y (mgr.34107) 5583 : audit [DBG] from='client.51142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:01 smithi115 bash[45495]: audit 2022-04-22T20:09:01.881989+0000 mon.a (mon.0) 8698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:01 smithi121 bash[36902]: cluster 2022-04-22T20:09:00.891063+0000 mgr.y (mgr.34107) 5581 : cluster [DBG] pgmap v6830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:01 smithi121 bash[36902]: audit 2022-04-22T20:09:01.228538+0000 mgr.y (mgr.34107) 5582 : audit [DBG] from='client.51136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:01 smithi121 bash[36902]: audit 2022-04-22T20:09:01.570257+0000 mgr.y (mgr.34107) 5583 : audit [DBG] from='client.51142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:02.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:01 smithi121 bash[36902]: audit 2022-04-22T20:09:01.881989+0000 mon.a (mon.0) 8698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:02.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:02.245Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:02 smithi121 bash[36902]: audit 2022-04-22T20:09:01.995944+0000 mon.a (mon.0) 8699 : audit [DBG] from='client.? 172.21.15.115:0/71834717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:02 smithi121 bash[36902]: cluster 2022-04-22T20:09:02.892330+0000 mgr.y (mgr.34107) 5584 : cluster [DBG] pgmap v6831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:02 smithi115 bash[45495]: audit 2022-04-22T20:09:01.995944+0000 mon.a (mon.0) 8699 : audit [DBG] from='client.? 172.21.15.115:0/71834717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:02 smithi115 bash[45495]: cluster 2022-04-22T20:09:02.892330+0000 mgr.y (mgr.34107) 5584 : cluster [DBG] pgmap v6831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:02 smithi115 bash[46934]: audit 2022-04-22T20:09:01.995944+0000 mon.a (mon.0) 8699 : audit [DBG] from='client.? 172.21.15.115:0/71834717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:02 smithi115 bash[46934]: cluster 2022-04-22T20:09:02.892330+0000 mgr.y (mgr.34107) 5584 : cluster [DBG] pgmap v6831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:03 smithi121 bash[36902]: audit 2022-04-22T20:09:03.319100+0000 mon.a (mon.0) 8700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:03 smithi115 bash[45495]: audit 2022-04-22T20:09:03.319100+0000 mon.a (mon.0) 8700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:04.124Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:03 smithi115 bash[46934]: audit 2022-04-22T20:09:03.319100+0000 mon.a (mon.0) 8700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:04 smithi121 bash[36902]: cluster 2022-04-22T20:09:04.893607+0000 mgr.y (mgr.34107) 5585 : cluster [DBG] pgmap v6832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:04 smithi115 bash[45495]: cluster 2022-04-22T20:09:04.893607+0000 mgr.y (mgr.34107) 5585 : cluster [DBG] pgmap v6832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:04 smithi115 bash[46934]: cluster 2022-04-22T20:09:04.893607+0000 mgr.y (mgr.34107) 5585 : cluster [DBG] pgmap v6832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:06 smithi121 bash[36902]: audit 2022-04-22T20:09:05.611846+0000 mon.a (mon.0) 8701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:06.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:05 smithi115 bash[45495]: audit 2022-04-22T20:09:05.611846+0000 mon.a (mon.0) 8701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:06.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:05 smithi115 bash[46934]: audit 2022-04-22T20:09:05.611846+0000 mon.a (mon.0) 8701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:07 smithi121 bash[36902]: audit 2022-04-22T20:09:06.885074+0000 mon.a (mon.0) 8702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:07 smithi121 bash[36902]: cluster 2022-04-22T20:09:06.894727+0000 mgr.y (mgr.34107) 5586 : cluster [DBG] pgmap v6833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:07 smithi115 bash[46934]: audit 2022-04-22T20:09:06.885074+0000 mon.a (mon.0) 8702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:07 smithi115 bash[46934]: cluster 2022-04-22T20:09:06.894727+0000 mgr.y (mgr.34107) 5586 : cluster [DBG] pgmap v6833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:07 smithi115 bash[45495]: audit 2022-04-22T20:09:06.885074+0000 mon.a (mon.0) 8702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:07 smithi115 bash[45495]: cluster 2022-04-22T20:09:06.894727+0000 mgr.y (mgr.34107) 5586 : cluster [DBG] pgmap v6833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:08 smithi121 bash[36902]: audit 2022-04-22T20:09:08.322014+0000 mon.a (mon.0) 8703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:08 smithi115 bash[45495]: audit 2022-04-22T20:09:08.322014+0000 mon.a (mon.0) 8703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:08 smithi115 bash[46934]: audit 2022-04-22T20:09:08.322014+0000 mon.a (mon.0) 8703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:09 smithi121 bash[36902]: cluster 2022-04-22T20:09:08.896004+0000 mgr.y (mgr.34107) 5587 : cluster [DBG] pgmap v6834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:10.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:09 smithi115 bash[45495]: cluster 2022-04-22T20:09:08.896004+0000 mgr.y (mgr.34107) 5587 : cluster [DBG] pgmap v6834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:10.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:09 smithi115 bash[46934]: cluster 2022-04-22T20:09:08.896004+0000 mgr.y (mgr.34107) 5587 : cluster [DBG] pgmap v6834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:11.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:10 smithi121 bash[36902]: audit 2022-04-22T20:09:10.615268+0000 mon.a (mon.0) 8704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:10 smithi115 bash[45495]: audit 2022-04-22T20:09:10.615268+0000 mon.a (mon.0) 8704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:10 smithi115 bash[46934]: audit 2022-04-22T20:09:10.615268+0000 mon.a (mon.0) 8704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:11 smithi121 bash[36902]: cluster 2022-04-22T20:09:10.897279+0000 mgr.y (mgr.34107) 5588 : cluster [DBG] pgmap v6835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:11 smithi115 bash[45495]: cluster 2022-04-22T20:09:10.897279+0000 mgr.y (mgr.34107) 5588 : cluster [DBG] pgmap v6835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:11 smithi115 bash[46934]: cluster 2022-04-22T20:09:10.897279+0000 mgr.y (mgr.34107) 5588 : cluster [DBG] pgmap v6835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:12.245Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:12 smithi121 bash[36902]: audit 2022-04-22T20:09:11.888372+0000 mon.a (mon.0) 8705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:12 smithi115 bash[45495]: audit 2022-04-22T20:09:11.888372+0000 mon.a (mon.0) 8705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:12 smithi115 bash[46934]: audit 2022-04-22T20:09:11.888372+0000 mon.a (mon.0) 8705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:13 smithi121 bash[36902]: cluster 2022-04-22T20:09:12.898435+0000 mgr.y (mgr.34107) 5589 : cluster [DBG] pgmap v6836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:13 smithi121 bash[36902]: audit 2022-04-22T20:09:13.325256+0000 mon.a (mon.0) 8706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:13 smithi115 bash[45495]: cluster 2022-04-22T20:09:12.898435+0000 mgr.y (mgr.34107) 5589 : cluster [DBG] pgmap v6836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:13 smithi115 bash[45495]: audit 2022-04-22T20:09:13.325256+0000 mon.a (mon.0) 8706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:13 smithi115 bash[46934]: cluster 2022-04-22T20:09:12.898435+0000 mgr.y (mgr.34107) 5589 : cluster [DBG] pgmap v6836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:13 smithi115 bash[46934]: audit 2022-04-22T20:09:13.325256+0000 mon.a (mon.0) 8706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:14.126Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:15 smithi121 bash[36902]: cluster 2022-04-22T20:09:14.899570+0000 mgr.y (mgr.34107) 5590 : cluster [DBG] pgmap v6837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:15 smithi121 bash[36902]: audit 2022-04-22T20:09:15.156436+0000 mon.a (mon.0) 8707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:09:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:15 smithi121 bash[36902]: audit 2022-04-22T20:09:15.156939+0000 mon.a (mon.0) 8708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:09:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:15 smithi121 bash[36902]: audit 2022-04-22T20:09:15.618726+0000 mon.a (mon.0) 8709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[45495]: cluster 2022-04-22T20:09:14.899570+0000 mgr.y (mgr.34107) 5590 : cluster [DBG] pgmap v6837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[45495]: audit 2022-04-22T20:09:15.156436+0000 mon.a (mon.0) 8707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:09:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[45495]: audit 2022-04-22T20:09:15.156939+0000 mon.a (mon.0) 8708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:09:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[45495]: audit 2022-04-22T20:09:15.618726+0000 mon.a (mon.0) 8709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[46934]: cluster 2022-04-22T20:09:14.899570+0000 mgr.y (mgr.34107) 5590 : cluster [DBG] pgmap v6837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[46934]: audit 2022-04-22T20:09:15.156436+0000 mon.a (mon.0) 8707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:09:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[46934]: audit 2022-04-22T20:09:15.156939+0000 mon.a (mon.0) 8708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:09:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:15 smithi115 bash[46934]: audit 2022-04-22T20:09:15.618726+0000 mon.a (mon.0) 8709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:16 smithi121 bash[36902]: audit 2022-04-22T20:09:16.891580+0000 mon.a (mon.0) 8710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:16 smithi115 bash[45495]: audit 2022-04-22T20:09:16.891580+0000 mon.a (mon.0) 8710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:16 smithi115 bash[46934]: audit 2022-04-22T20:09:16.891580+0000 mon.a (mon.0) 8710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:17 smithi121 bash[36902]: cluster 2022-04-22T20:09:16.900539+0000 mgr.y (mgr.34107) 5591 : cluster [DBG] pgmap v6838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:17 smithi115 bash[46934]: cluster 2022-04-22T20:09:16.900539+0000 mgr.y (mgr.34107) 5591 : cluster [DBG] pgmap v6838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:17 smithi115 bash[45495]: cluster 2022-04-22T20:09:16.900539+0000 mgr.y (mgr.34107) 5591 : cluster [DBG] pgmap v6838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:18 smithi121 bash[36902]: audit 2022-04-22T20:09:18.328288+0000 mon.a (mon.0) 8711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:18 smithi121 bash[36902]: cluster 2022-04-22T20:09:18.901683+0000 mgr.y (mgr.34107) 5592 : cluster [DBG] pgmap v6839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:18 smithi115 bash[45495]: audit 2022-04-22T20:09:18.328288+0000 mon.a (mon.0) 8711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:18 smithi115 bash[45495]: cluster 2022-04-22T20:09:18.901683+0000 mgr.y (mgr.34107) 5592 : cluster [DBG] pgmap v6839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:18 smithi115 bash[46934]: audit 2022-04-22T20:09:18.328288+0000 mon.a (mon.0) 8711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:18 smithi115 bash[46934]: cluster 2022-04-22T20:09:18.901683+0000 mgr.y (mgr.34107) 5592 : cluster [DBG] pgmap v6839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:20 smithi121 bash[36902]: audit 2022-04-22T20:09:20.622353+0000 mon.a (mon.0) 8712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:20 smithi115 bash[45495]: audit 2022-04-22T20:09:20.622353+0000 mon.a (mon.0) 8712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:20 smithi115 bash[46934]: audit 2022-04-22T20:09:20.622353+0000 mon.a (mon.0) 8712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:21 smithi121 bash[36902]: cluster 2022-04-22T20:09:20.902830+0000 mgr.y (mgr.34107) 5593 : cluster [DBG] pgmap v6840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:21 smithi115 bash[46934]: cluster 2022-04-22T20:09:20.902830+0000 mgr.y (mgr.34107) 5593 : cluster [DBG] pgmap v6840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:21 smithi115 bash[45495]: cluster 2022-04-22T20:09:20.902830+0000 mgr.y (mgr.34107) 5593 : cluster [DBG] pgmap v6840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:22.247Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:22 smithi121 bash[36902]: audit 2022-04-22T20:09:21.894759+0000 mon.a (mon.0) 8713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:23.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:22 smithi115 bash[45495]: audit 2022-04-22T20:09:21.894759+0000 mon.a (mon.0) 8713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:22 smithi115 bash[46934]: audit 2022-04-22T20:09:21.894759+0000 mon.a (mon.0) 8713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:23 smithi121 bash[36902]: cluster 2022-04-22T20:09:22.903955+0000 mgr.y (mgr.34107) 5594 : cluster [DBG] pgmap v6841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:23 smithi121 bash[36902]: audit 2022-04-22T20:09:23.331292+0000 mon.a (mon.0) 8714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:23 smithi115 bash[45495]: cluster 2022-04-22T20:09:22.903955+0000 mgr.y (mgr.34107) 5594 : cluster [DBG] pgmap v6841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:23 smithi115 bash[45495]: audit 2022-04-22T20:09:23.331292+0000 mon.a (mon.0) 8714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:23 smithi115 bash[46934]: cluster 2022-04-22T20:09:22.903955+0000 mgr.y (mgr.34107) 5594 : cluster [DBG] pgmap v6841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:23 smithi115 bash[46934]: audit 2022-04-22T20:09:23.331292+0000 mon.a (mon.0) 8714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:24.125Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:25 smithi121 bash[36902]: cluster 2022-04-22T20:09:24.905004+0000 mgr.y (mgr.34107) 5595 : cluster [DBG] pgmap v6842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:25 smithi121 bash[36902]: audit 2022-04-22T20:09:25.626260+0000 mon.a (mon.0) 8715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:25 smithi115 bash[45495]: cluster 2022-04-22T20:09:24.905004+0000 mgr.y (mgr.34107) 5595 : cluster [DBG] pgmap v6842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:25 smithi115 bash[45495]: audit 2022-04-22T20:09:25.626260+0000 mon.a (mon.0) 8715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:25 smithi115 bash[46934]: cluster 2022-04-22T20:09:24.905004+0000 mgr.y (mgr.34107) 5595 : cluster [DBG] pgmap v6842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:25 smithi115 bash[46934]: audit 2022-04-22T20:09:25.626260+0000 mon.a (mon.0) 8715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:26 smithi121 bash[36902]: audit 2022-04-22T20:09:26.898120+0000 mon.a (mon.0) 8716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:26 smithi115 bash[45495]: audit 2022-04-22T20:09:26.898120+0000 mon.a (mon.0) 8716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:26 smithi115 bash[46934]: audit 2022-04-22T20:09:26.898120+0000 mon.a (mon.0) 8716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:27 smithi121 bash[36902]: cluster 2022-04-22T20:09:26.906154+0000 mgr.y (mgr.34107) 5596 : cluster [DBG] pgmap v6843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:28.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:27 smithi115 bash[45495]: cluster 2022-04-22T20:09:26.906154+0000 mgr.y (mgr.34107) 5596 : cluster [DBG] pgmap v6843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:27 smithi115 bash[46934]: cluster 2022-04-22T20:09:26.906154+0000 mgr.y (mgr.34107) 5596 : cluster [DBG] pgmap v6843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:28 smithi121 bash[36902]: audit 2022-04-22T20:09:28.334550+0000 mon.a (mon.0) 8717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:28 smithi121 bash[36902]: cluster 2022-04-22T20:09:28.907326+0000 mgr.y (mgr.34107) 5597 : cluster [DBG] pgmap v6844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:29.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:28 smithi115 bash[45495]: audit 2022-04-22T20:09:28.334550+0000 mon.a (mon.0) 8717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:29.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:28 smithi115 bash[45495]: cluster 2022-04-22T20:09:28.907326+0000 mgr.y (mgr.34107) 5597 : cluster [DBG] pgmap v6844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:29.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:28 smithi115 bash[46934]: audit 2022-04-22T20:09:28.334550+0000 mon.a (mon.0) 8717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:29.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:28 smithi115 bash[46934]: cluster 2022-04-22T20:09:28.907326+0000 mgr.y (mgr.34107) 5597 : cluster [DBG] pgmap v6844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:30 smithi121 bash[36902]: audit 2022-04-22T20:09:30.629886+0000 mon.a (mon.0) 8718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:30 smithi115 bash[45495]: audit 2022-04-22T20:09:30.629886+0000 mon.a (mon.0) 8718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:30 smithi115 bash[46934]: audit 2022-04-22T20:09:30.629886+0000 mon.a (mon.0) 8718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:31 smithi121 bash[36902]: cluster 2022-04-22T20:09:30.908395+0000 mgr.y (mgr.34107) 5598 : cluster [DBG] pgmap v6845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:31 smithi115 bash[45495]: cluster 2022-04-22T20:09:30.908395+0000 mgr.y (mgr.34107) 5598 : cluster [DBG] pgmap v6845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:32.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:31 smithi115 bash[46934]: cluster 2022-04-22T20:09:30.908395+0000 mgr.y (mgr.34107) 5598 : cluster [DBG] pgmap v6845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:32.385 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:09:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:32.245Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:09:32.727 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:09:32.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:09:32.731 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:09:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:32 smithi121 bash[36902]: audit 2022-04-22T20:09:31.901324+0000 mon.a (mon.0) 8719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:32 smithi115 bash[46934]: audit 2022-04-22T20:09:31.901324+0000 mon.a (mon.0) 8719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:32 smithi115 bash[45495]: audit 2022-04-22T20:09:31.901324+0000 mon.a (mon.0) 8719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:33.137 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:09:33.138 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:09:33.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:33.139 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:09:33.141 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:09:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:33 smithi121 bash[36902]: audit 2022-04-22T20:09:32.360947+0000 mgr.y (mgr.34107) 5599 : audit [DBG] from='client.51154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:33 smithi121 bash[36902]: audit 2022-04-22T20:09:32.704553+0000 mgr.y (mgr.34107) 5600 : audit [DBG] from='client.51160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:33 smithi121 bash[36902]: cluster 2022-04-22T20:09:32.909443+0000 mgr.y (mgr.34107) 5601 : cluster [DBG] pgmap v6846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:33 smithi121 bash[36902]: audit 2022-04-22T20:09:33.129094+0000 mon.a (mon.0) 8720 : audit [DBG] from='client.? 172.21.15.115:0/322424798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:33 smithi121 bash[36902]: audit 2022-04-22T20:09:33.337801+0000 mon.a (mon.0) 8721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[45495]: audit 2022-04-22T20:09:32.360947+0000 mgr.y (mgr.34107) 5599 : audit [DBG] from='client.51154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[45495]: audit 2022-04-22T20:09:32.704553+0000 mgr.y (mgr.34107) 5600 : audit [DBG] from='client.51160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[45495]: cluster 2022-04-22T20:09:32.909443+0000 mgr.y (mgr.34107) 5601 : cluster [DBG] pgmap v6846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[45495]: audit 2022-04-22T20:09:33.129094+0000 mon.a (mon.0) 8720 : audit [DBG] from='client.? 172.21.15.115:0/322424798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:34.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[45495]: audit 2022-04-22T20:09:33.337801+0000 mon.a (mon.0) 8721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:34.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[46934]: audit 2022-04-22T20:09:32.360947+0000 mgr.y (mgr.34107) 5599 : audit [DBG] from='client.51154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[46934]: audit 2022-04-22T20:09:32.704553+0000 mgr.y (mgr.34107) 5600 : audit [DBG] from='client.51160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:09:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[46934]: cluster 2022-04-22T20:09:32.909443+0000 mgr.y (mgr.34107) 5601 : cluster [DBG] pgmap v6846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[46934]: audit 2022-04-22T20:09:33.129094+0000 mon.a (mon.0) 8720 : audit [DBG] from='client.? 172.21.15.115:0/322424798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:09:34.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:33 smithi115 bash[46934]: audit 2022-04-22T20:09:33.337801+0000 mon.a (mon.0) 8721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:34.126Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:35 smithi121 bash[36902]: cluster 2022-04-22T20:09:34.910526+0000 mgr.y (mgr.34107) 5602 : cluster [DBG] pgmap v6847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:36.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:35 smithi121 bash[36902]: audit 2022-04-22T20:09:35.633515+0000 mon.a (mon.0) 8722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:35 smithi115 bash[45495]: cluster 2022-04-22T20:09:34.910526+0000 mgr.y (mgr.34107) 5602 : cluster [DBG] pgmap v6847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:35 smithi115 bash[45495]: audit 2022-04-22T20:09:35.633515+0000 mon.a (mon.0) 8722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:35 smithi115 bash[46934]: cluster 2022-04-22T20:09:34.910526+0000 mgr.y (mgr.34107) 5602 : cluster [DBG] pgmap v6847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:35 smithi115 bash[46934]: audit 2022-04-22T20:09:35.633515+0000 mon.a (mon.0) 8722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:36 smithi121 bash[36902]: audit 2022-04-22T20:09:36.904666+0000 mon.a (mon.0) 8723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:36 smithi115 bash[45495]: audit 2022-04-22T20:09:36.904666+0000 mon.a (mon.0) 8723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:36 smithi115 bash[46934]: audit 2022-04-22T20:09:36.904666+0000 mon.a (mon.0) 8723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:37 smithi121 bash[36902]: cluster 2022-04-22T20:09:36.911541+0000 mgr.y (mgr.34107) 5603 : cluster [DBG] pgmap v6848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:37 smithi115 bash[45495]: cluster 2022-04-22T20:09:36.911541+0000 mgr.y (mgr.34107) 5603 : cluster [DBG] pgmap v6848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:37 smithi115 bash[46934]: cluster 2022-04-22T20:09:36.911541+0000 mgr.y (mgr.34107) 5603 : cluster [DBG] pgmap v6848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:38 smithi121 bash[36902]: audit 2022-04-22T20:09:38.340977+0000 mon.a (mon.0) 8724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:38 smithi121 bash[36902]: cluster 2022-04-22T20:09:38.912578+0000 mgr.y (mgr.34107) 5604 : cluster [DBG] pgmap v6849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:38 smithi115 bash[45495]: audit 2022-04-22T20:09:38.340977+0000 mon.a (mon.0) 8724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:38 smithi115 bash[45495]: cluster 2022-04-22T20:09:38.912578+0000 mgr.y (mgr.34107) 5604 : cluster [DBG] pgmap v6849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:38 smithi115 bash[46934]: audit 2022-04-22T20:09:38.340977+0000 mon.a (mon.0) 8724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:39.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:38 smithi115 bash[46934]: cluster 2022-04-22T20:09:38.912578+0000 mgr.y (mgr.34107) 5604 : cluster [DBG] pgmap v6849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:40 smithi121 bash[36902]: audit 2022-04-22T20:09:40.637586+0000 mon.a (mon.0) 8725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:40 smithi115 bash[45495]: audit 2022-04-22T20:09:40.637586+0000 mon.a (mon.0) 8725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:40 smithi115 bash[46934]: audit 2022-04-22T20:09:40.637586+0000 mon.a (mon.0) 8725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:41 smithi121 bash[36902]: cluster 2022-04-22T20:09:40.913566+0000 mgr.y (mgr.34107) 5605 : cluster [DBG] pgmap v6850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:41 smithi115 bash[45495]: cluster 2022-04-22T20:09:40.913566+0000 mgr.y (mgr.34107) 5605 : cluster [DBG] pgmap v6850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:41 smithi115 bash[46934]: cluster 2022-04-22T20:09:40.913566+0000 mgr.y (mgr.34107) 5605 : cluster [DBG] pgmap v6850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:42.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:42.249Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:42 smithi121 bash[36902]: audit 2022-04-22T20:09:41.907686+0000 mon.a (mon.0) 8726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:42 smithi115 bash[45495]: audit 2022-04-22T20:09:41.907686+0000 mon.a (mon.0) 8726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:42 smithi115 bash[46934]: audit 2022-04-22T20:09:41.907686+0000 mon.a (mon.0) 8726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:43 smithi121 bash[36902]: cluster 2022-04-22T20:09:42.914641+0000 mgr.y (mgr.34107) 5606 : cluster [DBG] pgmap v6851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:43 smithi121 bash[36902]: audit 2022-04-22T20:09:43.344119+0000 mon.a (mon.0) 8727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:43 smithi115 bash[45495]: cluster 2022-04-22T20:09:42.914641+0000 mgr.y (mgr.34107) 5606 : cluster [DBG] pgmap v6851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:43 smithi115 bash[45495]: audit 2022-04-22T20:09:43.344119+0000 mon.a (mon.0) 8727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:43 smithi115 bash[46934]: cluster 2022-04-22T20:09:42.914641+0000 mgr.y (mgr.34107) 5606 : cluster [DBG] pgmap v6851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:43 smithi115 bash[46934]: audit 2022-04-22T20:09:43.344119+0000 mon.a (mon.0) 8727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:44.125Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:45 smithi121 bash[36902]: cluster 2022-04-22T20:09:44.915747+0000 mgr.y (mgr.34107) 5607 : cluster [DBG] pgmap v6852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:45 smithi121 bash[36902]: audit 2022-04-22T20:09:45.641471+0000 mon.a (mon.0) 8728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:45 smithi115 bash[45495]: cluster 2022-04-22T20:09:44.915747+0000 mgr.y (mgr.34107) 5607 : cluster [DBG] pgmap v6852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:45 smithi115 bash[45495]: audit 2022-04-22T20:09:45.641471+0000 mon.a (mon.0) 8728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:45 smithi115 bash[46934]: cluster 2022-04-22T20:09:44.915747+0000 mgr.y (mgr.34107) 5607 : cluster [DBG] pgmap v6852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:46.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:45 smithi115 bash[46934]: audit 2022-04-22T20:09:45.641471+0000 mon.a (mon.0) 8728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:46 smithi121 bash[36902]: audit 2022-04-22T20:09:46.910828+0000 mon.a (mon.0) 8729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:46 smithi121 bash[36902]: cluster 2022-04-22T20:09:46.916889+0000 mgr.y (mgr.34107) 5608 : cluster [DBG] pgmap v6853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:46 smithi115 bash[45495]: audit 2022-04-22T20:09:46.910828+0000 mon.a (mon.0) 8729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:47.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:46 smithi115 bash[45495]: cluster 2022-04-22T20:09:46.916889+0000 mgr.y (mgr.34107) 5608 : cluster [DBG] pgmap v6853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:47.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:46 smithi115 bash[46934]: audit 2022-04-22T20:09:46.910828+0000 mon.a (mon.0) 8729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:47.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:46 smithi115 bash[46934]: cluster 2022-04-22T20:09:46.916889+0000 mgr.y (mgr.34107) 5608 : cluster [DBG] pgmap v6853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:48 smithi121 bash[36902]: audit 2022-04-22T20:09:48.346925+0000 mon.a (mon.0) 8730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:48 smithi115 bash[45495]: audit 2022-04-22T20:09:48.346925+0000 mon.a (mon.0) 8730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:48 smithi115 bash[46934]: audit 2022-04-22T20:09:48.346925+0000 mon.a (mon.0) 8730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:49 smithi121 bash[36902]: cluster 2022-04-22T20:09:48.917964+0000 mgr.y (mgr.34107) 5609 : cluster [DBG] pgmap v6854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:49 smithi115 bash[46934]: cluster 2022-04-22T20:09:48.917964+0000 mgr.y (mgr.34107) 5609 : cluster [DBG] pgmap v6854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:49 smithi115 bash[45495]: cluster 2022-04-22T20:09:48.917964+0000 mgr.y (mgr.34107) 5609 : cluster [DBG] pgmap v6854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:50 smithi121 bash[36902]: audit 2022-04-22T20:09:50.644959+0000 mon.a (mon.0) 8731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:50 smithi115 bash[45495]: audit 2022-04-22T20:09:50.644959+0000 mon.a (mon.0) 8731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:50 smithi115 bash[46934]: audit 2022-04-22T20:09:50.644959+0000 mon.a (mon.0) 8731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:51 smithi121 bash[36902]: cluster 2022-04-22T20:09:50.919047+0000 mgr.y (mgr.34107) 5610 : cluster [DBG] pgmap v6855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:52.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:51 smithi115 bash[45495]: cluster 2022-04-22T20:09:50.919047+0000 mgr.y (mgr.34107) 5610 : cluster [DBG] pgmap v6855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:52.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:51 smithi115 bash[46934]: cluster 2022-04-22T20:09:50.919047+0000 mgr.y (mgr.34107) 5610 : cluster [DBG] pgmap v6855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:52.249Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:52 smithi121 bash[36902]: audit 2022-04-22T20:09:51.914115+0000 mon.a (mon.0) 8732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:52 smithi115 bash[45495]: audit 2022-04-22T20:09:51.914115+0000 mon.a (mon.0) 8732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:52 smithi115 bash[46934]: audit 2022-04-22T20:09:51.914115+0000 mon.a (mon.0) 8732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:53 smithi121 bash[36902]: cluster 2022-04-22T20:09:52.920178+0000 mgr.y (mgr.34107) 5611 : cluster [DBG] pgmap v6856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:53 smithi121 bash[36902]: audit 2022-04-22T20:09:53.349985+0000 mon.a (mon.0) 8733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:53 smithi121 bash[36902]: audit 2022-04-22T20:09:53.458316+0000 mon.a (mon.0) 8734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:09:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:53 smithi121 bash[36902]: audit 2022-04-22T20:09:53.461167+0000 mon.a (mon.0) 8735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:09:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:53 smithi121 bash[36902]: audit 2022-04-22T20:09:53.462679+0000 mon.a (mon.0) 8736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[45495]: cluster 2022-04-22T20:09:52.920178+0000 mgr.y (mgr.34107) 5611 : cluster [DBG] pgmap v6856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[45495]: audit 2022-04-22T20:09:53.349985+0000 mon.a (mon.0) 8733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[45495]: audit 2022-04-22T20:09:53.458316+0000 mon.a (mon.0) 8734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[45495]: audit 2022-04-22T20:09:53.461167+0000 mon.a (mon.0) 8735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[45495]: audit 2022-04-22T20:09:53.462679+0000 mon.a (mon.0) 8736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:09:53.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[46934]: cluster 2022-04-22T20:09:52.920178+0000 mgr.y (mgr.34107) 5611 : cluster [DBG] pgmap v6856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[46934]: audit 2022-04-22T20:09:53.349985+0000 mon.a (mon.0) 8733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[46934]: audit 2022-04-22T20:09:53.458316+0000 mon.a (mon.0) 8734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:09:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[46934]: audit 2022-04-22T20:09:53.461167+0000 mon.a (mon.0) 8735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:09:53.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:53 smithi115 bash[46934]: audit 2022-04-22T20:09:53.462679+0000 mon.a (mon.0) 8736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:09:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:09:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:09:54.129Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:09:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:55 smithi121 bash[36902]: audit 2022-04-22T20:09:54.081527+0000 mon.a (mon.0) 8737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:09:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:55 smithi121 bash[36902]: cluster 2022-04-22T20:09:54.921258+0000 mgr.y (mgr.34107) 5612 : cluster [DBG] pgmap v6857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:55 smithi115 bash[45495]: audit 2022-04-22T20:09:54.081527+0000 mon.a (mon.0) 8737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:09:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:55 smithi115 bash[45495]: cluster 2022-04-22T20:09:54.921258+0000 mgr.y (mgr.34107) 5612 : cluster [DBG] pgmap v6857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:55 smithi115 bash[46934]: audit 2022-04-22T20:09:54.081527+0000 mon.a (mon.0) 8737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:09:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:55 smithi115 bash[46934]: cluster 2022-04-22T20:09:54.921258+0000 mgr.y (mgr.34107) 5612 : cluster [DBG] pgmap v6857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:56 smithi121 bash[36902]: audit 2022-04-22T20:09:55.648406+0000 mon.a (mon.0) 8738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:56 smithi115 bash[45495]: audit 2022-04-22T20:09:55.648406+0000 mon.a (mon.0) 8738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:56 smithi115 bash[46934]: audit 2022-04-22T20:09:55.648406+0000 mon.a (mon.0) 8738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:09:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:57 smithi121 bash[36902]: audit 2022-04-22T20:09:56.917074+0000 mon.a (mon.0) 8739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:57 smithi121 bash[36902]: cluster 2022-04-22T20:09:56.922372+0000 mgr.y (mgr.34107) 5613 : cluster [DBG] pgmap v6858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:57 smithi115 bash[45495]: audit 2022-04-22T20:09:56.917074+0000 mon.a (mon.0) 8739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:57.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:57 smithi115 bash[45495]: cluster 2022-04-22T20:09:56.922372+0000 mgr.y (mgr.34107) 5613 : cluster [DBG] pgmap v6858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:57.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:57 smithi115 bash[46934]: audit 2022-04-22T20:09:56.917074+0000 mon.a (mon.0) 8739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:09:57.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:57 smithi115 bash[46934]: cluster 2022-04-22T20:09:56.922372+0000 mgr.y (mgr.34107) 5613 : cluster [DBG] pgmap v6858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:09:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:58 smithi121 bash[36902]: audit 2022-04-22T20:09:58.353350+0000 mon.a (mon.0) 8740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:58 smithi115 bash[45495]: audit 2022-04-22T20:09:58.353350+0000 mon.a (mon.0) 8740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:09:58.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:58 smithi115 bash[46934]: audit 2022-04-22T20:09:58.353350+0000 mon.a (mon.0) 8740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:09:59 smithi121 bash[36902]: cluster 2022-04-22T20:09:58.923432+0000 mgr.y (mgr.34107) 5614 : cluster [DBG] pgmap v6859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:09:59 smithi115 bash[45495]: cluster 2022-04-22T20:09:58.923432+0000 mgr.y (mgr.34107) 5614 : cluster [DBG] pgmap v6859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:09:59 smithi115 bash[46934]: cluster 2022-04-22T20:09:58.923432+0000 mgr.y (mgr.34107) 5614 : cluster [DBG] pgmap v6859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:00 smithi121 bash[36902]: cluster 2022-04-22T20:10:00.000150+0000 mon.a (mon.0) 8741 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:10:01.161 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:00 smithi121 bash[36902]: audit 2022-04-22T20:10:00.652501+0000 mon.a (mon.0) 8742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:01.162 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:00 smithi115 bash[45495]: cluster 2022-04-22T20:10:00.000150+0000 mon.a (mon.0) 8741 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:10:01.162 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:00 smithi115 bash[45495]: audit 2022-04-22T20:10:00.652501+0000 mon.a (mon.0) 8742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:01.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:00 smithi115 bash[46934]: cluster 2022-04-22T20:10:00.000150+0000 mon.a (mon.0) 8741 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:10:01.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:00 smithi115 bash[46934]: audit 2022-04-22T20:10:00.652501+0000 mon.a (mon.0) 8742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:01.991 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:01 smithi121 bash[36902]: cluster 2022-04-22T20:10:00.924875+0000 mgr.y (mgr.34107) 5615 : cluster [DBG] pgmap v6860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:02.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:01 smithi115 bash[45495]: cluster 2022-04-22T20:10:00.924875+0000 mgr.y (mgr.34107) 5615 : cluster [DBG] pgmap v6860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:01 smithi115 bash[46934]: cluster 2022-04-22T20:10:00.924875+0000 mgr.y (mgr.34107) 5615 : cluster [DBG] pgmap v6860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:02.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:02.248Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:02 smithi121 bash[36902]: audit 2022-04-22T20:10:01.920406+0000 mon.a (mon.0) 8743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:02 smithi121 bash[36902]: audit 2022-04-22T20:10:02.671686+0000 mon.a (mon.0) 8744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:02 smithi121 bash[36902]: audit 2022-04-22T20:10:02.678010+0000 mon.a (mon.0) 8745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[45495]: audit 2022-04-22T20:10:01.920406+0000 mon.a (mon.0) 8743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[45495]: audit 2022-04-22T20:10:02.671686+0000 mon.a (mon.0) 8744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[45495]: audit 2022-04-22T20:10:02.678010+0000 mon.a (mon.0) 8745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[46934]: audit 2022-04-22T20:10:01.920406+0000 mon.a (mon.0) 8743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[46934]: audit 2022-04-22T20:10:02.671686+0000 mon.a (mon.0) 8744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:02 smithi115 bash[46934]: audit 2022-04-22T20:10:02.678010+0000 mon.a (mon.0) 8745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:10:03.512 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:10:03.857 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:10:03.857 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:10:03.857 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:10:03.857 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:10:03.857 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:10:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:10:03.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:10:03.860 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:10:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:03 smithi121 bash[36902]: cephadm 2022-04-22T20:10:02.660461+0000 mgr.y (mgr.34107) 5616 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:10:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:03 smithi121 bash[36902]: cluster 2022-04-22T20:10:02.925995+0000 mgr.y (mgr.34107) 5617 : cluster [DBG] pgmap v6861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:03 smithi121 bash[36902]: audit 2022-04-22T20:10:03.356160+0000 mon.a (mon.0) 8746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[46934]: cephadm 2022-04-22T20:10:02.660461+0000 mgr.y (mgr.34107) 5616 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:10:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[46934]: cluster 2022-04-22T20:10:02.925995+0000 mgr.y (mgr.34107) 5617 : cluster [DBG] pgmap v6861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[46934]: audit 2022-04-22T20:10:03.356160+0000 mon.a (mon.0) 8746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[45495]: cephadm 2022-04-22T20:10:02.660461+0000 mgr.y (mgr.34107) 5616 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:10:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[45495]: cluster 2022-04-22T20:10:02.925995+0000 mgr.y (mgr.34107) 5617 : cluster [DBG] pgmap v6861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:03 smithi115 bash[45495]: audit 2022-04-22T20:10:03.356160+0000 mon.a (mon.0) 8746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:04.266 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:10:04.268 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:10:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:04.125Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:04 smithi121 bash[36902]: audit 2022-04-22T20:10:03.487979+0000 mgr.y (mgr.34107) 5618 : audit [DBG] from='client.51172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:04 smithi121 bash[36902]: audit 2022-04-22T20:10:03.828872+0000 mgr.y (mgr.34107) 5619 : audit [DBG] from='client.51178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:04 smithi121 bash[36902]: audit 2022-04-22T20:10:04.257984+0000 mon.c (mon.1) 226 : audit [DBG] from='client.? 172.21.15.115:0/3637589264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[45495]: audit 2022-04-22T20:10:03.487979+0000 mgr.y (mgr.34107) 5618 : audit [DBG] from='client.51172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[45495]: audit 2022-04-22T20:10:03.828872+0000 mgr.y (mgr.34107) 5619 : audit [DBG] from='client.51178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[45495]: audit 2022-04-22T20:10:04.257984+0000 mon.c (mon.1) 226 : audit [DBG] from='client.? 172.21.15.115:0/3637589264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[46934]: audit 2022-04-22T20:10:03.487979+0000 mgr.y (mgr.34107) 5618 : audit [DBG] from='client.51172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[46934]: audit 2022-04-22T20:10:03.828872+0000 mgr.y (mgr.34107) 5619 : audit [DBG] from='client.51178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:05.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:04 smithi115 bash[46934]: audit 2022-04-22T20:10:04.257984+0000 mon.c (mon.1) 226 : audit [DBG] from='client.? 172.21.15.115:0/3637589264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:05 smithi121 bash[36902]: cluster 2022-04-22T20:10:04.927091+0000 mgr.y (mgr.34107) 5620 : cluster [DBG] pgmap v6862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:05 smithi121 bash[36902]: audit 2022-04-22T20:10:05.656148+0000 mon.a (mon.0) 8747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:06.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:05 smithi115 bash[45495]: cluster 2022-04-22T20:10:04.927091+0000 mgr.y (mgr.34107) 5620 : cluster [DBG] pgmap v6862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:06.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:05 smithi115 bash[45495]: audit 2022-04-22T20:10:05.656148+0000 mon.a (mon.0) 8747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:05 smithi115 bash[46934]: cluster 2022-04-22T20:10:04.927091+0000 mgr.y (mgr.34107) 5620 : cluster [DBG] pgmap v6862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:06.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:05 smithi115 bash[46934]: audit 2022-04-22T20:10:05.656148+0000 mon.a (mon.0) 8747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:06 smithi121 bash[36902]: audit 2022-04-22T20:10:06.923455+0000 mon.a (mon.0) 8748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:06 smithi115 bash[45495]: audit 2022-04-22T20:10:06.923455+0000 mon.a (mon.0) 8748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:06 smithi115 bash[46934]: audit 2022-04-22T20:10:06.923455+0000 mon.a (mon.0) 8748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:07 smithi121 bash[36902]: cluster 2022-04-22T20:10:06.928260+0000 mgr.y (mgr.34107) 5621 : cluster [DBG] pgmap v6863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:07 smithi115 bash[45495]: cluster 2022-04-22T20:10:06.928260+0000 mgr.y (mgr.34107) 5621 : cluster [DBG] pgmap v6863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:07 smithi115 bash[46934]: cluster 2022-04-22T20:10:06.928260+0000 mgr.y (mgr.34107) 5621 : cluster [DBG] pgmap v6863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:08 smithi121 bash[36902]: audit 2022-04-22T20:10:08.358930+0000 mon.a (mon.0) 8749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:08 smithi121 bash[36902]: cluster 2022-04-22T20:10:08.929361+0000 mgr.y (mgr.34107) 5622 : cluster [DBG] pgmap v6864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:08 smithi115 bash[45495]: audit 2022-04-22T20:10:08.358930+0000 mon.a (mon.0) 8749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:08 smithi115 bash[45495]: cluster 2022-04-22T20:10:08.929361+0000 mgr.y (mgr.34107) 5622 : cluster [DBG] pgmap v6864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:08 smithi115 bash[46934]: audit 2022-04-22T20:10:08.358930+0000 mon.a (mon.0) 8749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:08 smithi115 bash[46934]: cluster 2022-04-22T20:10:08.929361+0000 mgr.y (mgr.34107) 5622 : cluster [DBG] pgmap v6864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:10 smithi121 bash[36902]: audit 2022-04-22T20:10:10.659968+0000 mon.a (mon.0) 8750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:10 smithi115 bash[45495]: audit 2022-04-22T20:10:10.659968+0000 mon.a (mon.0) 8750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:10 smithi115 bash[46934]: audit 2022-04-22T20:10:10.659968+0000 mon.a (mon.0) 8750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:11 smithi121 bash[36902]: cluster 2022-04-22T20:10:10.930415+0000 mgr.y (mgr.34107) 5623 : cluster [DBG] pgmap v6865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:11 smithi115 bash[45495]: cluster 2022-04-22T20:10:10.930415+0000 mgr.y (mgr.34107) 5623 : cluster [DBG] pgmap v6865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:11 smithi115 bash[46934]: cluster 2022-04-22T20:10:10.930415+0000 mgr.y (mgr.34107) 5623 : cluster [DBG] pgmap v6865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:12.249Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:12 smithi121 bash[36902]: audit 2022-04-22T20:10:11.926678+0000 mon.a (mon.0) 8751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:12 smithi115 bash[45495]: audit 2022-04-22T20:10:11.926678+0000 mon.a (mon.0) 8751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:13.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:12 smithi115 bash[46934]: audit 2022-04-22T20:10:11.926678+0000 mon.a (mon.0) 8751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:13 smithi121 bash[36902]: cluster 2022-04-22T20:10:12.931335+0000 mgr.y (mgr.34107) 5624 : cluster [DBG] pgmap v6866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:13 smithi121 bash[36902]: audit 2022-04-22T20:10:13.363240+0000 mon.a (mon.0) 8752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:13 smithi115 bash[45495]: cluster 2022-04-22T20:10:12.931335+0000 mgr.y (mgr.34107) 5624 : cluster [DBG] pgmap v6866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:13 smithi115 bash[45495]: audit 2022-04-22T20:10:13.363240+0000 mon.a (mon.0) 8752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:13 smithi115 bash[46934]: cluster 2022-04-22T20:10:12.931335+0000 mgr.y (mgr.34107) 5624 : cluster [DBG] pgmap v6866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:14.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:13 smithi115 bash[46934]: audit 2022-04-22T20:10:13.363240+0000 mon.a (mon.0) 8752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:14.129Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:15 smithi115 bash[45495]: cluster 2022-04-22T20:10:14.932375+0000 mgr.y (mgr.34107) 5625 : cluster [DBG] pgmap v6867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:15 smithi115 bash[46934]: cluster 2022-04-22T20:10:14.932375+0000 mgr.y (mgr.34107) 5625 : cluster [DBG] pgmap v6867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:15 smithi121 bash[36902]: cluster 2022-04-22T20:10:14.932375+0000 mgr.y (mgr.34107) 5625 : cluster [DBG] pgmap v6867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[45495]: audit 2022-04-22T20:10:15.166999+0000 mon.a (mon.0) 8753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:10:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[45495]: audit 2022-04-22T20:10:15.167539+0000 mon.a (mon.0) 8754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:10:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[45495]: audit 2022-04-22T20:10:15.663662+0000 mon.a (mon.0) 8755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[46934]: audit 2022-04-22T20:10:15.166999+0000 mon.a (mon.0) 8753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:10:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[46934]: audit 2022-04-22T20:10:15.167539+0000 mon.a (mon.0) 8754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:10:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:16 smithi115 bash[46934]: audit 2022-04-22T20:10:15.663662+0000 mon.a (mon.0) 8755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:16 smithi121 bash[36902]: audit 2022-04-22T20:10:15.166999+0000 mon.a (mon.0) 8753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:10:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:16 smithi121 bash[36902]: audit 2022-04-22T20:10:15.167539+0000 mon.a (mon.0) 8754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:10:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:16 smithi121 bash[36902]: audit 2022-04-22T20:10:15.663662+0000 mon.a (mon.0) 8755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:17 smithi121 bash[36902]: audit 2022-04-22T20:10:16.930211+0000 mon.a (mon.0) 8756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:17 smithi121 bash[36902]: cluster 2022-04-22T20:10:16.933875+0000 mgr.y (mgr.34107) 5626 : cluster [DBG] pgmap v6868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:17 smithi115 bash[45495]: audit 2022-04-22T20:10:16.930211+0000 mon.a (mon.0) 8756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:17 smithi115 bash[45495]: cluster 2022-04-22T20:10:16.933875+0000 mgr.y (mgr.34107) 5626 : cluster [DBG] pgmap v6868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:17 smithi115 bash[46934]: audit 2022-04-22T20:10:16.930211+0000 mon.a (mon.0) 8756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:17 smithi115 bash[46934]: cluster 2022-04-22T20:10:16.933875+0000 mgr.y (mgr.34107) 5626 : cluster [DBG] pgmap v6868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:18 smithi121 bash[36902]: audit 2022-04-22T20:10:18.366703+0000 mon.a (mon.0) 8757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:18 smithi115 bash[45495]: audit 2022-04-22T20:10:18.366703+0000 mon.a (mon.0) 8757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:18 smithi115 bash[46934]: audit 2022-04-22T20:10:18.366703+0000 mon.a (mon.0) 8757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:19 smithi121 bash[36902]: cluster 2022-04-22T20:10:18.935049+0000 mgr.y (mgr.34107) 5627 : cluster [DBG] pgmap v6869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:19 smithi115 bash[45495]: cluster 2022-04-22T20:10:18.935049+0000 mgr.y (mgr.34107) 5627 : cluster [DBG] pgmap v6869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:19 smithi115 bash[46934]: cluster 2022-04-22T20:10:18.935049+0000 mgr.y (mgr.34107) 5627 : cluster [DBG] pgmap v6869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:20 smithi121 bash[36902]: audit 2022-04-22T20:10:20.667525+0000 mon.a (mon.0) 8758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:20 smithi115 bash[45495]: audit 2022-04-22T20:10:20.667525+0000 mon.a (mon.0) 8758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:20 smithi115 bash[46934]: audit 2022-04-22T20:10:20.667525+0000 mon.a (mon.0) 8758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:21 smithi121 bash[36902]: cluster 2022-04-22T20:10:20.936128+0000 mgr.y (mgr.34107) 5628 : cluster [DBG] pgmap v6870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:21 smithi115 bash[45495]: cluster 2022-04-22T20:10:20.936128+0000 mgr.y (mgr.34107) 5628 : cluster [DBG] pgmap v6870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:21 smithi115 bash[46934]: cluster 2022-04-22T20:10:20.936128+0000 mgr.y (mgr.34107) 5628 : cluster [DBG] pgmap v6870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:22.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:22 smithi121 bash[36902]: audit 2022-04-22T20:10:21.933403+0000 mon.a (mon.0) 8759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:22 smithi115 bash[45495]: audit 2022-04-22T20:10:21.933403+0000 mon.a (mon.0) 8759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:22 smithi115 bash[46934]: audit 2022-04-22T20:10:21.933403+0000 mon.a (mon.0) 8759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:23 smithi121 bash[36902]: cluster 2022-04-22T20:10:22.937112+0000 mgr.y (mgr.34107) 5629 : cluster [DBG] pgmap v6871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:23 smithi121 bash[36902]: audit 2022-04-22T20:10:23.369760+0000 mon.a (mon.0) 8760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:23 smithi115 bash[45495]: cluster 2022-04-22T20:10:22.937112+0000 mgr.y (mgr.34107) 5629 : cluster [DBG] pgmap v6871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:23 smithi115 bash[45495]: audit 2022-04-22T20:10:23.369760+0000 mon.a (mon.0) 8760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:23 smithi115 bash[46934]: cluster 2022-04-22T20:10:22.937112+0000 mgr.y (mgr.34107) 5629 : cluster [DBG] pgmap v6871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:23 smithi115 bash[46934]: audit 2022-04-22T20:10:23.369760+0000 mon.a (mon.0) 8760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:24.128Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:25 smithi115 bash[45495]: cluster 2022-04-22T20:10:24.938234+0000 mgr.y (mgr.34107) 5630 : cluster [DBG] pgmap v6872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:25 smithi115 bash[46934]: cluster 2022-04-22T20:10:24.938234+0000 mgr.y (mgr.34107) 5630 : cluster [DBG] pgmap v6872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:25 smithi121 bash[36902]: cluster 2022-04-22T20:10:24.938234+0000 mgr.y (mgr.34107) 5630 : cluster [DBG] pgmap v6872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:26 smithi115 bash[45495]: audit 2022-04-22T20:10:25.671398+0000 mon.a (mon.0) 8761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:26 smithi115 bash[46934]: audit 2022-04-22T20:10:25.671398+0000 mon.a (mon.0) 8761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:26 smithi121 bash[36902]: audit 2022-04-22T20:10:25.671398+0000 mon.a (mon.0) 8761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:27 smithi121 bash[36902]: audit 2022-04-22T20:10:26.936660+0000 mon.a (mon.0) 8762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:27 smithi121 bash[36902]: cluster 2022-04-22T20:10:26.939274+0000 mgr.y (mgr.34107) 5631 : cluster [DBG] pgmap v6873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:27 smithi115 bash[46934]: audit 2022-04-22T20:10:26.936660+0000 mon.a (mon.0) 8762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:27 smithi115 bash[46934]: cluster 2022-04-22T20:10:26.939274+0000 mgr.y (mgr.34107) 5631 : cluster [DBG] pgmap v6873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:27 smithi115 bash[45495]: audit 2022-04-22T20:10:26.936660+0000 mon.a (mon.0) 8762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:27 smithi115 bash[45495]: cluster 2022-04-22T20:10:26.939274+0000 mgr.y (mgr.34107) 5631 : cluster [DBG] pgmap v6873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:28 smithi121 bash[36902]: audit 2022-04-22T20:10:28.372823+0000 mon.a (mon.0) 8763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:28 smithi115 bash[45495]: audit 2022-04-22T20:10:28.372823+0000 mon.a (mon.0) 8763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:28 smithi115 bash[46934]: audit 2022-04-22T20:10:28.372823+0000 mon.a (mon.0) 8763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:29 smithi121 bash[36902]: cluster 2022-04-22T20:10:28.940458+0000 mgr.y (mgr.34107) 5632 : cluster [DBG] pgmap v6874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:30.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:29 smithi115 bash[45495]: cluster 2022-04-22T20:10:28.940458+0000 mgr.y (mgr.34107) 5632 : cluster [DBG] pgmap v6874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:30.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:29 smithi115 bash[46934]: cluster 2022-04-22T20:10:28.940458+0000 mgr.y (mgr.34107) 5632 : cluster [DBG] pgmap v6874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:30 smithi121 bash[36902]: audit 2022-04-22T20:10:30.674754+0000 mon.a (mon.0) 8764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:30 smithi115 bash[45495]: audit 2022-04-22T20:10:30.674754+0000 mon.a (mon.0) 8764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:30 smithi115 bash[46934]: audit 2022-04-22T20:10:30.674754+0000 mon.a (mon.0) 8764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:31 smithi121 bash[36902]: cluster 2022-04-22T20:10:30.941579+0000 mgr.y (mgr.34107) 5633 : cluster [DBG] pgmap v6875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:31 smithi115 bash[45495]: cluster 2022-04-22T20:10:30.941579+0000 mgr.y (mgr.34107) 5633 : cluster [DBG] pgmap v6875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:31 smithi115 bash[46934]: cluster 2022-04-22T20:10:30.941579+0000 mgr.y (mgr.34107) 5633 : cluster [DBG] pgmap v6875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:32.251Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:32 smithi121 bash[36902]: audit 2022-04-22T20:10:31.939829+0000 mon.a (mon.0) 8765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:33.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:32 smithi115 bash[45495]: audit 2022-04-22T20:10:31.939829+0000 mon.a (mon.0) 8765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:32 smithi115 bash[46934]: audit 2022-04-22T20:10:31.939829+0000 mon.a (mon.0) 8765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:33 smithi121 bash[36902]: cluster 2022-04-22T20:10:32.942642+0000 mgr.y (mgr.34107) 5634 : cluster [DBG] pgmap v6876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:33 smithi121 bash[36902]: audit 2022-04-22T20:10:33.375673+0000 mon.a (mon.0) 8766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:34.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:33 smithi115 bash[45495]: cluster 2022-04-22T20:10:32.942642+0000 mgr.y (mgr.34107) 5634 : cluster [DBG] pgmap v6876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:33 smithi115 bash[45495]: audit 2022-04-22T20:10:33.375673+0000 mon.a (mon.0) 8766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:33 smithi115 bash[46934]: cluster 2022-04-22T20:10:32.942642+0000 mgr.y (mgr.34107) 5634 : cluster [DBG] pgmap v6876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:33 smithi115 bash[46934]: audit 2022-04-22T20:10:33.375673+0000 mon.a (mon.0) 8766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:34.131Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:34.641 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:10:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:10:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:10:34.984 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:10:34.986 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:10:34.986 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:10:34.986 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:10:34.986 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:10:34.986 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:10:34.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:10:34.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:10:34.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:10:34.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:10:34.987 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:10:34.988 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:10:34.988 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:10:34.988 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:10:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[45495]: audit 2022-04-22T20:10:34.616718+0000 mgr.y (mgr.34107) 5635 : audit [DBG] from='client.51190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[45495]: cluster 2022-04-22T20:10:34.943391+0000 mgr.y (mgr.34107) 5636 : cluster [DBG] pgmap v6877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[45495]: audit 2022-04-22T20:10:34.956659+0000 mgr.y (mgr.34107) 5637 : audit [DBG] from='client.51196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[46934]: audit 2022-04-22T20:10:34.616718+0000 mgr.y (mgr.34107) 5635 : audit [DBG] from='client.51190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:35.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[46934]: cluster 2022-04-22T20:10:34.943391+0000 mgr.y (mgr.34107) 5636 : cluster [DBG] pgmap v6877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:35.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:35 smithi115 bash[46934]: audit 2022-04-22T20:10:34.956659+0000 mgr.y (mgr.34107) 5637 : audit [DBG] from='client.51196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:35.387 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:10:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:10:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:10:35.417 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:10:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:35 smithi121 bash[36902]: audit 2022-04-22T20:10:34.616718+0000 mgr.y (mgr.34107) 5635 : audit [DBG] from='client.51190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:35 smithi121 bash[36902]: cluster 2022-04-22T20:10:34.943391+0000 mgr.y (mgr.34107) 5636 : cluster [DBG] pgmap v6877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:35 smithi121 bash[36902]: audit 2022-04-22T20:10:34.956659+0000 mgr.y (mgr.34107) 5637 : audit [DBG] from='client.51196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:10:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:36 smithi115 bash[45495]: audit 2022-04-22T20:10:35.378663+0000 mon.c (mon.1) 227 : audit [DBG] from='client.? 172.21.15.115:0/2496930528' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:36 smithi115 bash[45495]: audit 2022-04-22T20:10:35.678669+0000 mon.a (mon.0) 8767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:36 smithi115 bash[46934]: audit 2022-04-22T20:10:35.378663+0000 mon.c (mon.1) 227 : audit [DBG] from='client.? 172.21.15.115:0/2496930528' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:36 smithi115 bash[46934]: audit 2022-04-22T20:10:35.678669+0000 mon.a (mon.0) 8767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:36 smithi121 bash[36902]: audit 2022-04-22T20:10:35.378663+0000 mon.c (mon.1) 227 : audit [DBG] from='client.? 172.21.15.115:0/2496930528' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:36 smithi121 bash[36902]: audit 2022-04-22T20:10:35.678669+0000 mon.a (mon.0) 8767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:37 smithi121 bash[36902]: audit 2022-04-22T20:10:36.942926+0000 mon.a (mon.0) 8768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:37 smithi121 bash[36902]: cluster 2022-04-22T20:10:36.944496+0000 mgr.y (mgr.34107) 5638 : cluster [DBG] pgmap v6878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:37 smithi115 bash[45495]: audit 2022-04-22T20:10:36.942926+0000 mon.a (mon.0) 8768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:37 smithi115 bash[45495]: cluster 2022-04-22T20:10:36.944496+0000 mgr.y (mgr.34107) 5638 : cluster [DBG] pgmap v6878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:37 smithi115 bash[46934]: audit 2022-04-22T20:10:36.942926+0000 mon.a (mon.0) 8768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:37 smithi115 bash[46934]: cluster 2022-04-22T20:10:36.944496+0000 mgr.y (mgr.34107) 5638 : cluster [DBG] pgmap v6878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:38 smithi121 bash[36902]: audit 2022-04-22T20:10:38.378520+0000 mon.a (mon.0) 8769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:38 smithi115 bash[45495]: audit 2022-04-22T20:10:38.378520+0000 mon.a (mon.0) 8769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:38 smithi115 bash[46934]: audit 2022-04-22T20:10:38.378520+0000 mon.a (mon.0) 8769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:39 smithi121 bash[36902]: cluster 2022-04-22T20:10:38.945699+0000 mgr.y (mgr.34107) 5639 : cluster [DBG] pgmap v6879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:40.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:39 smithi115 bash[45495]: cluster 2022-04-22T20:10:38.945699+0000 mgr.y (mgr.34107) 5639 : cluster [DBG] pgmap v6879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:39 smithi115 bash[46934]: cluster 2022-04-22T20:10:38.945699+0000 mgr.y (mgr.34107) 5639 : cluster [DBG] pgmap v6879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:40 smithi121 bash[36902]: audit 2022-04-22T20:10:40.682388+0000 mon.a (mon.0) 8770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:41.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:40 smithi115 bash[45495]: audit 2022-04-22T20:10:40.682388+0000 mon.a (mon.0) 8770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:40 smithi115 bash[46934]: audit 2022-04-22T20:10:40.682388+0000 mon.a (mon.0) 8770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:41 smithi121 bash[36902]: cluster 2022-04-22T20:10:40.946710+0000 mgr.y (mgr.34107) 5640 : cluster [DBG] pgmap v6880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:41 smithi115 bash[45495]: cluster 2022-04-22T20:10:40.946710+0000 mgr.y (mgr.34107) 5640 : cluster [DBG] pgmap v6880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:42.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:41 smithi115 bash[46934]: cluster 2022-04-22T20:10:40.946710+0000 mgr.y (mgr.34107) 5640 : cluster [DBG] pgmap v6880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:42.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:42 smithi121 bash[36902]: audit 2022-04-22T20:10:41.946292+0000 mon.a (mon.0) 8771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:43.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:42 smithi115 bash[45495]: audit 2022-04-22T20:10:41.946292+0000 mon.a (mon.0) 8771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:42 smithi115 bash[46934]: audit 2022-04-22T20:10:41.946292+0000 mon.a (mon.0) 8771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:43 smithi121 bash[36902]: cluster 2022-04-22T20:10:42.947714+0000 mgr.y (mgr.34107) 5641 : cluster [DBG] pgmap v6881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:43 smithi121 bash[36902]: audit 2022-04-22T20:10:43.381393+0000 mon.a (mon.0) 8772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:43 smithi115 bash[45495]: cluster 2022-04-22T20:10:42.947714+0000 mgr.y (mgr.34107) 5641 : cluster [DBG] pgmap v6881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:43 smithi115 bash[45495]: audit 2022-04-22T20:10:43.381393+0000 mon.a (mon.0) 8772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:43 smithi115 bash[46934]: cluster 2022-04-22T20:10:42.947714+0000 mgr.y (mgr.34107) 5641 : cluster [DBG] pgmap v6881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:43 smithi115 bash[46934]: audit 2022-04-22T20:10:43.381393+0000 mon.a (mon.0) 8772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:44.131Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:45 smithi115 bash[45495]: cluster 2022-04-22T20:10:44.948751+0000 mgr.y (mgr.34107) 5642 : cluster [DBG] pgmap v6882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:45 smithi115 bash[46934]: cluster 2022-04-22T20:10:44.948751+0000 mgr.y (mgr.34107) 5642 : cluster [DBG] pgmap v6882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:45 smithi121 bash[36902]: cluster 2022-04-22T20:10:44.948751+0000 mgr.y (mgr.34107) 5642 : cluster [DBG] pgmap v6882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:46.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:46 smithi115 bash[46934]: audit 2022-04-22T20:10:45.686335+0000 mon.a (mon.0) 8773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:46.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:46 smithi115 bash[45495]: audit 2022-04-22T20:10:45.686335+0000 mon.a (mon.0) 8773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:46 smithi121 bash[36902]: audit 2022-04-22T20:10:45.686335+0000 mon.a (mon.0) 8773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:47 smithi121 bash[36902]: audit 2022-04-22T20:10:46.949325+0000 mon.a (mon.0) 8774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:47 smithi121 bash[36902]: cluster 2022-04-22T20:10:46.949823+0000 mgr.y (mgr.34107) 5643 : cluster [DBG] pgmap v6883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:47 smithi115 bash[46934]: audit 2022-04-22T20:10:46.949325+0000 mon.a (mon.0) 8774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:47 smithi115 bash[46934]: cluster 2022-04-22T20:10:46.949823+0000 mgr.y (mgr.34107) 5643 : cluster [DBG] pgmap v6883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:47 smithi115 bash[45495]: audit 2022-04-22T20:10:46.949325+0000 mon.a (mon.0) 8774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:47.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:47 smithi115 bash[45495]: cluster 2022-04-22T20:10:46.949823+0000 mgr.y (mgr.34107) 5643 : cluster [DBG] pgmap v6883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:48 smithi121 bash[36902]: audit 2022-04-22T20:10:48.384431+0000 mon.a (mon.0) 8775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:48 smithi115 bash[45495]: audit 2022-04-22T20:10:48.384431+0000 mon.a (mon.0) 8775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:48 smithi115 bash[46934]: audit 2022-04-22T20:10:48.384431+0000 mon.a (mon.0) 8775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:49 smithi121 bash[36902]: cluster 2022-04-22T20:10:48.951022+0000 mgr.y (mgr.34107) 5644 : cluster [DBG] pgmap v6884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:50.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:49 smithi115 bash[46934]: cluster 2022-04-22T20:10:48.951022+0000 mgr.y (mgr.34107) 5644 : cluster [DBG] pgmap v6884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:50.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:49 smithi115 bash[45495]: cluster 2022-04-22T20:10:48.951022+0000 mgr.y (mgr.34107) 5644 : cluster [DBG] pgmap v6884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:50 smithi121 bash[36902]: audit 2022-04-22T20:10:50.690052+0000 mon.a (mon.0) 8776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:50 smithi115 bash[45495]: audit 2022-04-22T20:10:50.690052+0000 mon.a (mon.0) 8776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:50 smithi115 bash[46934]: audit 2022-04-22T20:10:50.690052+0000 mon.a (mon.0) 8776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:51 smithi121 bash[36902]: cluster 2022-04-22T20:10:50.952104+0000 mgr.y (mgr.34107) 5645 : cluster [DBG] pgmap v6885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:52.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:51 smithi115 bash[45495]: cluster 2022-04-22T20:10:50.952104+0000 mgr.y (mgr.34107) 5645 : cluster [DBG] pgmap v6885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:52.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:51 smithi115 bash[46934]: cluster 2022-04-22T20:10:50.952104+0000 mgr.y (mgr.34107) 5645 : cluster [DBG] pgmap v6885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:52.251Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:52 smithi121 bash[36902]: audit 2022-04-22T20:10:51.952586+0000 mon.a (mon.0) 8777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:53.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:52 smithi115 bash[45495]: audit 2022-04-22T20:10:51.952586+0000 mon.a (mon.0) 8777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:53.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:52 smithi115 bash[46934]: audit 2022-04-22T20:10:51.952586+0000 mon.a (mon.0) 8777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:53 smithi121 bash[36902]: cluster 2022-04-22T20:10:52.953168+0000 mgr.y (mgr.34107) 5646 : cluster [DBG] pgmap v6886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:53 smithi121 bash[36902]: audit 2022-04-22T20:10:53.387432+0000 mon.a (mon.0) 8778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:53 smithi115 bash[45495]: cluster 2022-04-22T20:10:52.953168+0000 mgr.y (mgr.34107) 5646 : cluster [DBG] pgmap v6886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:54.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:53 smithi115 bash[45495]: audit 2022-04-22T20:10:53.387432+0000 mon.a (mon.0) 8778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:53 smithi115 bash[46934]: cluster 2022-04-22T20:10:52.953168+0000 mgr.y (mgr.34107) 5646 : cluster [DBG] pgmap v6886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:54.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:53 smithi115 bash[46934]: audit 2022-04-22T20:10:53.387432+0000 mon.a (mon.0) 8778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:10:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:10:54.130Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:10:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:55 smithi115 bash[45495]: cluster 2022-04-22T20:10:54.954303+0000 mgr.y (mgr.34107) 5647 : cluster [DBG] pgmap v6887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:55 smithi115 bash[46934]: cluster 2022-04-22T20:10:54.954303+0000 mgr.y (mgr.34107) 5647 : cluster [DBG] pgmap v6887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:55 smithi121 bash[36902]: cluster 2022-04-22T20:10:54.954303+0000 mgr.y (mgr.34107) 5647 : cluster [DBG] pgmap v6887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:56 smithi121 bash[36902]: audit 2022-04-22T20:10:55.693696+0000 mon.a (mon.0) 8779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:56 smithi115 bash[45495]: audit 2022-04-22T20:10:55.693696+0000 mon.a (mon.0) 8779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:56 smithi115 bash[46934]: audit 2022-04-22T20:10:55.693696+0000 mon.a (mon.0) 8779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:10:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:57 smithi121 bash[36902]: cluster 2022-04-22T20:10:56.955603+0000 mgr.y (mgr.34107) 5648 : cluster [DBG] pgmap v6888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:57 smithi121 bash[36902]: audit 2022-04-22T20:10:56.957288+0000 mon.a (mon.0) 8780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:57 smithi115 bash[46934]: cluster 2022-04-22T20:10:56.955603+0000 mgr.y (mgr.34107) 5648 : cluster [DBG] pgmap v6888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:57 smithi115 bash[46934]: audit 2022-04-22T20:10:56.957288+0000 mon.a (mon.0) 8780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:57 smithi115 bash[45495]: cluster 2022-04-22T20:10:56.955603+0000 mgr.y (mgr.34107) 5648 : cluster [DBG] pgmap v6888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:10:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:57 smithi115 bash[45495]: audit 2022-04-22T20:10:56.957288+0000 mon.a (mon.0) 8780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:10:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:58 smithi121 bash[36902]: audit 2022-04-22T20:10:58.390228+0000 mon.a (mon.0) 8781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:58 smithi115 bash[45495]: audit 2022-04-22T20:10:58.390228+0000 mon.a (mon.0) 8781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:10:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:58 smithi115 bash[46934]: audit 2022-04-22T20:10:58.390228+0000 mon.a (mon.0) 8781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:10:59 smithi121 bash[36902]: cluster 2022-04-22T20:10:58.956791+0000 mgr.y (mgr.34107) 5649 : cluster [DBG] pgmap v6889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:10:59 smithi115 bash[45495]: cluster 2022-04-22T20:10:58.956791+0000 mgr.y (mgr.34107) 5649 : cluster [DBG] pgmap v6889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:10:59 smithi115 bash[46934]: cluster 2022-04-22T20:10:58.956791+0000 mgr.y (mgr.34107) 5649 : cluster [DBG] pgmap v6889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:00 smithi121 bash[36902]: audit 2022-04-22T20:11:00.697398+0000 mon.a (mon.0) 8782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:01.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:00 smithi115 bash[45495]: audit 2022-04-22T20:11:00.697398+0000 mon.a (mon.0) 8782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:01.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:00 smithi115 bash[46934]: audit 2022-04-22T20:11:00.697398+0000 mon.a (mon.0) 8782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:01 smithi121 bash[36902]: cluster 2022-04-22T20:11:00.957865+0000 mgr.y (mgr.34107) 5650 : cluster [DBG] pgmap v6890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:02.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:01 smithi115 bash[45495]: cluster 2022-04-22T20:11:00.957865+0000 mgr.y (mgr.34107) 5650 : cluster [DBG] pgmap v6890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:02.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:01 smithi115 bash[46934]: cluster 2022-04-22T20:11:00.957865+0000 mgr.y (mgr.34107) 5650 : cluster [DBG] pgmap v6890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:02.251Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:02 smithi121 bash[36902]: audit 2022-04-22T20:11:01.959316+0000 mon.a (mon.0) 8783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:02 smithi121 bash[36902]: audit 2022-04-22T20:11:02.687575+0000 mon.a (mon.0) 8784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:11:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:02 smithi121 bash[36902]: audit 2022-04-22T20:11:02.690817+0000 mon.a (mon.0) 8785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:11:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:02 smithi121 bash[36902]: audit 2022-04-22T20:11:02.692659+0000 mon.a (mon.0) 8786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:11:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[46934]: audit 2022-04-22T20:11:01.959316+0000 mon.a (mon.0) 8783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[46934]: audit 2022-04-22T20:11:02.687575+0000 mon.a (mon.0) 8784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:11:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[46934]: audit 2022-04-22T20:11:02.690817+0000 mon.a (mon.0) 8785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:11:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[46934]: audit 2022-04-22T20:11:02.692659+0000 mon.a (mon.0) 8786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:11:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[45495]: audit 2022-04-22T20:11:01.959316+0000 mon.a (mon.0) 8783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[45495]: audit 2022-04-22T20:11:02.687575+0000 mon.a (mon.0) 8784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:11:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[45495]: audit 2022-04-22T20:11:02.690817+0000 mon.a (mon.0) 8785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:11:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:02 smithi115 bash[45495]: audit 2022-04-22T20:11:02.692659+0000 mon.a (mon.0) 8786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:11:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:03 smithi121 bash[36902]: cluster 2022-04-22T20:11:02.958431+0000 mgr.y (mgr.34107) 5651 : cluster [DBG] pgmap v6891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:04.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:03 smithi121 bash[36902]: audit 2022-04-22T20:11:03.305654+0000 mon.a (mon.0) 8787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:04.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:03 smithi121 bash[36902]: audit 2022-04-22T20:11:03.393231+0000 mon.a (mon.0) 8788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:04.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[45495]: cluster 2022-04-22T20:11:02.958431+0000 mgr.y (mgr.34107) 5651 : cluster [DBG] pgmap v6891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:04.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[45495]: audit 2022-04-22T20:11:03.305654+0000 mon.a (mon.0) 8787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:04.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[45495]: audit 2022-04-22T20:11:03.393231+0000 mon.a (mon.0) 8788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:04.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[46934]: cluster 2022-04-22T20:11:02.958431+0000 mgr.y (mgr.34107) 5651 : cluster [DBG] pgmap v6891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:04.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[46934]: audit 2022-04-22T20:11:03.305654+0000 mon.a (mon.0) 8787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:04.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:03 smithi115 bash[46934]: audit 2022-04-22T20:11:03.393231+0000 mon.a (mon.0) 8788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:04.132Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:05 smithi121 bash[36902]: cluster 2022-04-22T20:11:04.959448+0000 mgr.y (mgr.34107) 5652 : cluster [DBG] pgmap v6892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:05 smithi115 bash[45495]: cluster 2022-04-22T20:11:04.959448+0000 mgr.y (mgr.34107) 5652 : cluster [DBG] pgmap v6892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:05 smithi115 bash[46934]: cluster 2022-04-22T20:11:04.959448+0000 mgr.y (mgr.34107) 5652 : cluster [DBG] pgmap v6892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 10m ago 2h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 10m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 10m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:11:06.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 10m ago 2h 197M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 10m ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 10m ago 2h 575M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:11:06.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 10m ago 2h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:11:06.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 10m ago 2h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:11:06.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 10m ago 2h 637M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:11:06.122 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 10m ago 2h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:11:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:06 smithi121 bash[36902]: audit 2022-04-22T20:11:05.701223+0000 mon.a (mon.0) 8789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:06 smithi121 bash[36902]: audit 2022-04-22T20:11:05.739935+0000 mgr.y (mgr.34107) 5653 : audit [DBG] from='client.41274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:06.525 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:11:06.525 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:11:06.525 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:11:06.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:06.527 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:11:06.527 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:11:06.527 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:11:06.527 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:11:06.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:06 smithi115 bash[45495]: audit 2022-04-22T20:11:05.701223+0000 mon.a (mon.0) 8789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:06 smithi115 bash[45495]: audit 2022-04-22T20:11:05.739935+0000 mgr.y (mgr.34107) 5653 : audit [DBG] from='client.41274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:06 smithi115 bash[46934]: audit 2022-04-22T20:11:05.701223+0000 mon.a (mon.0) 8789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:06 smithi115 bash[46934]: audit 2022-04-22T20:11:05.739935+0000 mgr.y (mgr.34107) 5653 : audit [DBG] from='client.41274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:07.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:07 smithi121 bash[36902]: audit 2022-04-22T20:11:06.092354+0000 mgr.y (mgr.34107) 5654 : audit [DBG] from='client.51214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:07.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:07 smithi121 bash[36902]: audit 2022-04-22T20:11:06.516938+0000 mon.a (mon.0) 8790 : audit [DBG] from='client.? 172.21.15.115:0/2234787997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:07.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:07 smithi121 bash[36902]: cluster 2022-04-22T20:11:06.960616+0000 mgr.y (mgr.34107) 5655 : cluster [DBG] pgmap v6893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:07.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:07 smithi121 bash[36902]: audit 2022-04-22T20:11:06.962377+0000 mon.a (mon.0) 8791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[46934]: audit 2022-04-22T20:11:06.092354+0000 mgr.y (mgr.34107) 5654 : audit [DBG] from='client.51214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[46934]: audit 2022-04-22T20:11:06.516938+0000 mon.a (mon.0) 8790 : audit [DBG] from='client.? 172.21.15.115:0/2234787997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[46934]: cluster 2022-04-22T20:11:06.960616+0000 mgr.y (mgr.34107) 5655 : cluster [DBG] pgmap v6893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[46934]: audit 2022-04-22T20:11:06.962377+0000 mon.a (mon.0) 8791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[45495]: audit 2022-04-22T20:11:06.092354+0000 mgr.y (mgr.34107) 5654 : audit [DBG] from='client.51214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[45495]: audit 2022-04-22T20:11:06.516938+0000 mon.a (mon.0) 8790 : audit [DBG] from='client.? 172.21.15.115:0/2234787997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[45495]: cluster 2022-04-22T20:11:06.960616+0000 mgr.y (mgr.34107) 5655 : cluster [DBG] pgmap v6893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:07.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:07 smithi115 bash[45495]: audit 2022-04-22T20:11:06.962377+0000 mon.a (mon.0) 8791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:08 smithi121 bash[36902]: audit 2022-04-22T20:11:07.139884+0000 mon.a (mon.0) 8792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:08 smithi121 bash[36902]: audit 2022-04-22T20:11:07.484891+0000 mon.a (mon.0) 8793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:08 smithi115 bash[45495]: audit 2022-04-22T20:11:07.139884+0000 mon.a (mon.0) 8792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:08 smithi115 bash[45495]: audit 2022-04-22T20:11:07.484891+0000 mon.a (mon.0) 8793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:08 smithi115 bash[46934]: audit 2022-04-22T20:11:07.139884+0000 mon.a (mon.0) 8792 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:08 smithi115 bash[46934]: audit 2022-04-22T20:11:07.484891+0000 mon.a (mon.0) 8793 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:11:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:09 smithi121 bash[36902]: audit 2022-04-22T20:11:08.396642+0000 mon.a (mon.0) 8794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:09 smithi121 bash[36902]: cluster 2022-04-22T20:11:08.961833+0000 mgr.y (mgr.34107) 5656 : cluster [DBG] pgmap v6894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:09 smithi115 bash[45495]: audit 2022-04-22T20:11:08.396642+0000 mon.a (mon.0) 8794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:09 smithi115 bash[45495]: cluster 2022-04-22T20:11:08.961833+0000 mgr.y (mgr.34107) 5656 : cluster [DBG] pgmap v6894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:09 smithi115 bash[46934]: audit 2022-04-22T20:11:08.396642+0000 mon.a (mon.0) 8794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:09 smithi115 bash[46934]: cluster 2022-04-22T20:11:08.961833+0000 mgr.y (mgr.34107) 5656 : cluster [DBG] pgmap v6894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:10 smithi121 bash[36902]: audit 2022-04-22T20:11:10.704714+0000 mon.a (mon.0) 8795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:11.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:10 smithi115 bash[45495]: audit 2022-04-22T20:11:10.704714+0000 mon.a (mon.0) 8795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:10 smithi115 bash[46934]: audit 2022-04-22T20:11:10.704714+0000 mon.a (mon.0) 8795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:11 smithi121 bash[36902]: cluster 2022-04-22T20:11:10.962954+0000 mgr.y (mgr.34107) 5657 : cluster [DBG] pgmap v6895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:12.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:11 smithi115 bash[46934]: cluster 2022-04-22T20:11:10.962954+0000 mgr.y (mgr.34107) 5657 : cluster [DBG] pgmap v6895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:12.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:11 smithi115 bash[45495]: cluster 2022-04-22T20:11:10.962954+0000 mgr.y (mgr.34107) 5657 : cluster [DBG] pgmap v6895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:12.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:12.253Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:12 smithi121 bash[36902]: audit 2022-04-22T20:11:11.965453+0000 mon.a (mon.0) 8796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:12 smithi115 bash[45495]: audit 2022-04-22T20:11:11.965453+0000 mon.a (mon.0) 8796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:13.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:12 smithi115 bash[46934]: audit 2022-04-22T20:11:11.965453+0000 mon.a (mon.0) 8796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:13 smithi121 bash[36902]: cluster 2022-04-22T20:11:12.964044+0000 mgr.y (mgr.34107) 5658 : cluster [DBG] pgmap v6896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:13 smithi121 bash[36902]: audit 2022-04-22T20:11:13.399866+0000 mon.a (mon.0) 8797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:13 smithi115 bash[45495]: cluster 2022-04-22T20:11:12.964044+0000 mgr.y (mgr.34107) 5658 : cluster [DBG] pgmap v6896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:14.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:13 smithi115 bash[45495]: audit 2022-04-22T20:11:13.399866+0000 mon.a (mon.0) 8797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:13 smithi115 bash[46934]: cluster 2022-04-22T20:11:12.964044+0000 mgr.y (mgr.34107) 5658 : cluster [DBG] pgmap v6896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:14.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:13 smithi115 bash[46934]: audit 2022-04-22T20:11:13.399866+0000 mon.a (mon.0) 8797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:14.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:14.133Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:15 smithi121 bash[36902]: cluster 2022-04-22T20:11:14.965196+0000 mgr.y (mgr.34107) 5659 : cluster [DBG] pgmap v6897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:15 smithi115 bash[45495]: cluster 2022-04-22T20:11:14.965196+0000 mgr.y (mgr.34107) 5659 : cluster [DBG] pgmap v6897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:15 smithi115 bash[46934]: cluster 2022-04-22T20:11:14.965196+0000 mgr.y (mgr.34107) 5659 : cluster [DBG] pgmap v6897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:16 smithi121 bash[36902]: audit 2022-04-22T20:11:15.178096+0000 mon.a (mon.0) 8798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:11:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:16 smithi121 bash[36902]: audit 2022-04-22T20:11:15.178579+0000 mon.a (mon.0) 8799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:11:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:16 smithi121 bash[36902]: audit 2022-04-22T20:11:15.708472+0000 mon.a (mon.0) 8800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[46934]: audit 2022-04-22T20:11:15.178096+0000 mon.a (mon.0) 8798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:11:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[46934]: audit 2022-04-22T20:11:15.178579+0000 mon.a (mon.0) 8799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:11:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[46934]: audit 2022-04-22T20:11:15.708472+0000 mon.a (mon.0) 8800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[45495]: audit 2022-04-22T20:11:15.178096+0000 mon.a (mon.0) 8798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:11:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[45495]: audit 2022-04-22T20:11:15.178579+0000 mon.a (mon.0) 8799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:11:16.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:16 smithi115 bash[45495]: audit 2022-04-22T20:11:15.708472+0000 mon.a (mon.0) 8800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:17 smithi121 bash[36902]: cluster 2022-04-22T20:11:16.966415+0000 mgr.y (mgr.34107) 5660 : cluster [DBG] pgmap v6898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:17 smithi121 bash[36902]: audit 2022-04-22T20:11:16.968512+0000 mon.a (mon.0) 8801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:17 smithi115 bash[45495]: cluster 2022-04-22T20:11:16.966415+0000 mgr.y (mgr.34107) 5660 : cluster [DBG] pgmap v6898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:17 smithi115 bash[45495]: audit 2022-04-22T20:11:16.968512+0000 mon.a (mon.0) 8801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:17 smithi115 bash[46934]: cluster 2022-04-22T20:11:16.966415+0000 mgr.y (mgr.34107) 5660 : cluster [DBG] pgmap v6898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:17 smithi115 bash[46934]: audit 2022-04-22T20:11:16.968512+0000 mon.a (mon.0) 8801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:18 smithi121 bash[36902]: audit 2022-04-22T20:11:18.402722+0000 mon.a (mon.0) 8802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:18 smithi115 bash[45495]: audit 2022-04-22T20:11:18.402722+0000 mon.a (mon.0) 8802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:18 smithi115 bash[46934]: audit 2022-04-22T20:11:18.402722+0000 mon.a (mon.0) 8802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:19 smithi121 bash[36902]: cluster 2022-04-22T20:11:18.967678+0000 mgr.y (mgr.34107) 5661 : cluster [DBG] pgmap v6899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:20.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:19 smithi115 bash[45495]: cluster 2022-04-22T20:11:18.967678+0000 mgr.y (mgr.34107) 5661 : cluster [DBG] pgmap v6899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:20.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:19 smithi115 bash[46934]: cluster 2022-04-22T20:11:18.967678+0000 mgr.y (mgr.34107) 5661 : cluster [DBG] pgmap v6899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:20 smithi121 bash[36902]: audit 2022-04-22T20:11:20.712078+0000 mon.a (mon.0) 8803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:21.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:20 smithi115 bash[45495]: audit 2022-04-22T20:11:20.712078+0000 mon.a (mon.0) 8803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:20 smithi115 bash[46934]: audit 2022-04-22T20:11:20.712078+0000 mon.a (mon.0) 8803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:21 smithi121 bash[36902]: cluster 2022-04-22T20:11:20.968735+0000 mgr.y (mgr.34107) 5662 : cluster [DBG] pgmap v6900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:21 smithi115 bash[45495]: cluster 2022-04-22T20:11:20.968735+0000 mgr.y (mgr.34107) 5662 : cluster [DBG] pgmap v6900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:21 smithi115 bash[46934]: cluster 2022-04-22T20:11:20.968735+0000 mgr.y (mgr.34107) 5662 : cluster [DBG] pgmap v6900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:22.253Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:22 smithi121 bash[36902]: audit 2022-04-22T20:11:21.971778+0000 mon.a (mon.0) 8804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:22 smithi115 bash[45495]: audit 2022-04-22T20:11:21.971778+0000 mon.a (mon.0) 8804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:22 smithi115 bash[46934]: audit 2022-04-22T20:11:21.971778+0000 mon.a (mon.0) 8804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:23 smithi121 bash[36902]: cluster 2022-04-22T20:11:22.969889+0000 mgr.y (mgr.34107) 5663 : cluster [DBG] pgmap v6901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:23 smithi121 bash[36902]: audit 2022-04-22T20:11:23.405733+0000 mon.a (mon.0) 8805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:23 smithi115 bash[45495]: cluster 2022-04-22T20:11:22.969889+0000 mgr.y (mgr.34107) 5663 : cluster [DBG] pgmap v6901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:23 smithi115 bash[45495]: audit 2022-04-22T20:11:23.405733+0000 mon.a (mon.0) 8805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:24.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:23 smithi115 bash[46934]: cluster 2022-04-22T20:11:22.969889+0000 mgr.y (mgr.34107) 5663 : cluster [DBG] pgmap v6901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:24.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:23 smithi115 bash[46934]: audit 2022-04-22T20:11:23.405733+0000 mon.a (mon.0) 8805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:24.134Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:25 smithi121 bash[36902]: cluster 2022-04-22T20:11:24.970935+0000 mgr.y (mgr.34107) 5664 : cluster [DBG] pgmap v6902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:25 smithi115 bash[45495]: cluster 2022-04-22T20:11:24.970935+0000 mgr.y (mgr.34107) 5664 : cluster [DBG] pgmap v6902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:25 smithi115 bash[46934]: cluster 2022-04-22T20:11:24.970935+0000 mgr.y (mgr.34107) 5664 : cluster [DBG] pgmap v6902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:26 smithi121 bash[36902]: audit 2022-04-22T20:11:25.715983+0000 mon.a (mon.0) 8806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:26 smithi115 bash[45495]: audit 2022-04-22T20:11:25.715983+0000 mon.a (mon.0) 8806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:26 smithi115 bash[46934]: audit 2022-04-22T20:11:25.715983+0000 mon.a (mon.0) 8806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:27 smithi121 bash[36902]: cluster 2022-04-22T20:11:26.972072+0000 mgr.y (mgr.34107) 5665 : cluster [DBG] pgmap v6903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:27 smithi121 bash[36902]: audit 2022-04-22T20:11:26.974786+0000 mon.a (mon.0) 8807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:27 smithi115 bash[45495]: cluster 2022-04-22T20:11:26.972072+0000 mgr.y (mgr.34107) 5665 : cluster [DBG] pgmap v6903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:27 smithi115 bash[45495]: audit 2022-04-22T20:11:26.974786+0000 mon.a (mon.0) 8807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:27 smithi115 bash[46934]: cluster 2022-04-22T20:11:26.972072+0000 mgr.y (mgr.34107) 5665 : cluster [DBG] pgmap v6903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:27 smithi115 bash[46934]: audit 2022-04-22T20:11:26.974786+0000 mon.a (mon.0) 8807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:28 smithi121 bash[36902]: audit 2022-04-22T20:11:28.408735+0000 mon.a (mon.0) 8808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:28.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:28 smithi115 bash[45495]: audit 2022-04-22T20:11:28.408735+0000 mon.a (mon.0) 8808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:28.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:28 smithi115 bash[46934]: audit 2022-04-22T20:11:28.408735+0000 mon.a (mon.0) 8808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:29 smithi121 bash[36902]: cluster 2022-04-22T20:11:28.973256+0000 mgr.y (mgr.34107) 5666 : cluster [DBG] pgmap v6904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:30.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:29 smithi115 bash[45495]: cluster 2022-04-22T20:11:28.973256+0000 mgr.y (mgr.34107) 5666 : cluster [DBG] pgmap v6904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:30.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:29 smithi115 bash[46934]: cluster 2022-04-22T20:11:28.973256+0000 mgr.y (mgr.34107) 5666 : cluster [DBG] pgmap v6904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:30 smithi121 bash[36902]: audit 2022-04-22T20:11:30.719760+0000 mon.a (mon.0) 8809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:30 smithi115 bash[45495]: audit 2022-04-22T20:11:30.719760+0000 mon.a (mon.0) 8809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:30 smithi115 bash[46934]: audit 2022-04-22T20:11:30.719760+0000 mon.a (mon.0) 8809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:31 smithi121 bash[36902]: cluster 2022-04-22T20:11:30.974260+0000 mgr.y (mgr.34107) 5667 : cluster [DBG] pgmap v6905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:31 smithi115 bash[45495]: cluster 2022-04-22T20:11:30.974260+0000 mgr.y (mgr.34107) 5667 : cluster [DBG] pgmap v6905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:31 smithi115 bash[46934]: cluster 2022-04-22T20:11:30.974260+0000 mgr.y (mgr.34107) 5667 : cluster [DBG] pgmap v6905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:32.254Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:33.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:32 smithi121 bash[36902]: audit 2022-04-22T20:11:31.977906+0000 mon.a (mon.0) 8810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:32 smithi115 bash[45495]: audit 2022-04-22T20:11:31.977906+0000 mon.a (mon.0) 8810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:32 smithi115 bash[46934]: audit 2022-04-22T20:11:31.977906+0000 mon.a (mon.0) 8810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:33 smithi121 bash[36902]: cluster 2022-04-22T20:11:32.975336+0000 mgr.y (mgr.34107) 5668 : cluster [DBG] pgmap v6906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:33 smithi121 bash[36902]: audit 2022-04-22T20:11:33.411749+0000 mon.a (mon.0) 8811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:33 smithi115 bash[45495]: cluster 2022-04-22T20:11:32.975336+0000 mgr.y (mgr.34107) 5668 : cluster [DBG] pgmap v6906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:33 smithi115 bash[45495]: audit 2022-04-22T20:11:33.411749+0000 mon.a (mon.0) 8811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:33 smithi115 bash[46934]: cluster 2022-04-22T20:11:32.975336+0000 mgr.y (mgr.34107) 5668 : cluster [DBG] pgmap v6906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:33 smithi115 bash[46934]: audit 2022-04-22T20:11:33.411749+0000 mon.a (mon.0) 8811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:34.135Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:35 smithi121 bash[36902]: cluster 2022-04-22T20:11:34.976438+0000 mgr.y (mgr.34107) 5669 : cluster [DBG] pgmap v6907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:35 smithi115 bash[45495]: cluster 2022-04-22T20:11:34.976438+0000 mgr.y (mgr.34107) 5669 : cluster [DBG] pgmap v6907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:35 smithi115 bash[46934]: cluster 2022-04-22T20:11:34.976438+0000 mgr.y (mgr.34107) 5669 : cluster [DBG] pgmap v6907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:36 smithi121 bash[36902]: audit 2022-04-22T20:11:35.723506+0000 mon.a (mon.0) 8812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:36.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:36 smithi115 bash[45495]: audit 2022-04-22T20:11:35.723506+0000 mon.a (mon.0) 8812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:36.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:36 smithi115 bash[46934]: audit 2022-04-22T20:11:35.723506+0000 mon.a (mon.0) 8812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:36.901 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:11:37.251 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:11:37.251 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:11:37.251 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 30s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:11:37.251 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 30s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 30s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 30s ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 30s ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:11:37.252 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 30s ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 30s ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 30s ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 30s ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:11:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 30s ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:11:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:37 smithi121 bash[36902]: audit 2022-04-22T20:11:36.878198+0000 mgr.y (mgr.34107) 5670 : audit [DBG] from='client.41292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:37 smithi121 bash[36902]: cluster 2022-04-22T20:11:36.977575+0000 mgr.y (mgr.34107) 5671 : cluster [DBG] pgmap v6908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:37 smithi121 bash[36902]: audit 2022-04-22T20:11:36.980802+0000 mon.a (mon.0) 8813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[45495]: audit 2022-04-22T20:11:36.878198+0000 mgr.y (mgr.34107) 5670 : audit [DBG] from='client.41292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[45495]: cluster 2022-04-22T20:11:36.977575+0000 mgr.y (mgr.34107) 5671 : cluster [DBG] pgmap v6908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[45495]: audit 2022-04-22T20:11:36.980802+0000 mon.a (mon.0) 8813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[46934]: audit 2022-04-22T20:11:36.878198+0000 mgr.y (mgr.34107) 5670 : audit [DBG] from='client.41292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[46934]: cluster 2022-04-22T20:11:36.977575+0000 mgr.y (mgr.34107) 5671 : cluster [DBG] pgmap v6908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:37.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:37 smithi115 bash[46934]: audit 2022-04-22T20:11:36.980802+0000 mon.a (mon.0) 8813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:37.666 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:11:37.666 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:11:37.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:11:37.668 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:11:37.668 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:11:37.668 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:11:37.668 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:11:37.668 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:11:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:38 smithi121 bash[36902]: audit 2022-04-22T20:11:37.220289+0000 mgr.y (mgr.34107) 5672 : audit [DBG] from='client.41298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:38 smithi121 bash[36902]: audit 2022-04-22T20:11:37.654079+0000 mon.a (mon.0) 8814 : audit [DBG] from='client.? 172.21.15.115:0/2499260292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:38 smithi115 bash[45495]: audit 2022-04-22T20:11:37.220289+0000 mgr.y (mgr.34107) 5672 : audit [DBG] from='client.41298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:38 smithi115 bash[45495]: audit 2022-04-22T20:11:37.654079+0000 mon.a (mon.0) 8814 : audit [DBG] from='client.? 172.21.15.115:0/2499260292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:38 smithi115 bash[46934]: audit 2022-04-22T20:11:37.220289+0000 mgr.y (mgr.34107) 5672 : audit [DBG] from='client.41298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:11:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:38 smithi115 bash[46934]: audit 2022-04-22T20:11:37.654079+0000 mon.a (mon.0) 8814 : audit [DBG] from='client.? 172.21.15.115:0/2499260292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:11:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:39 smithi121 bash[36902]: audit 2022-04-22T20:11:38.414918+0000 mon.a (mon.0) 8815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:39 smithi121 bash[36902]: cluster 2022-04-22T20:11:38.978779+0000 mgr.y (mgr.34107) 5673 : cluster [DBG] pgmap v6909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:39 smithi115 bash[45495]: audit 2022-04-22T20:11:38.414918+0000 mon.a (mon.0) 8815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:39.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:39 smithi115 bash[45495]: cluster 2022-04-22T20:11:38.978779+0000 mgr.y (mgr.34107) 5673 : cluster [DBG] pgmap v6909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:39 smithi115 bash[46934]: audit 2022-04-22T20:11:38.414918+0000 mon.a (mon.0) 8815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:39 smithi115 bash[46934]: cluster 2022-04-22T20:11:38.978779+0000 mgr.y (mgr.34107) 5673 : cluster [DBG] pgmap v6909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:40 smithi121 bash[36902]: audit 2022-04-22T20:11:40.727067+0000 mon.a (mon.0) 8816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:40 smithi115 bash[45495]: audit 2022-04-22T20:11:40.727067+0000 mon.a (mon.0) 8816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:41.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:40 smithi115 bash[46934]: audit 2022-04-22T20:11:40.727067+0000 mon.a (mon.0) 8816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:41 smithi121 bash[36902]: cluster 2022-04-22T20:11:40.979951+0000 mgr.y (mgr.34107) 5674 : cluster [DBG] pgmap v6910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:42.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:41 smithi115 bash[45495]: cluster 2022-04-22T20:11:40.979951+0000 mgr.y (mgr.34107) 5674 : cluster [DBG] pgmap v6910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:42.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:41 smithi115 bash[46934]: cluster 2022-04-22T20:11:40.979951+0000 mgr.y (mgr.34107) 5674 : cluster [DBG] pgmap v6910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:42.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:42.254Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:42 smithi121 bash[36902]: audit 2022-04-22T20:11:41.983663+0000 mon.a (mon.0) 8817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:43.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:42 smithi115 bash[45495]: audit 2022-04-22T20:11:41.983663+0000 mon.a (mon.0) 8817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:42 smithi115 bash[46934]: audit 2022-04-22T20:11:41.983663+0000 mon.a (mon.0) 8817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:43 smithi121 bash[36902]: cluster 2022-04-22T20:11:42.980968+0000 mgr.y (mgr.34107) 5675 : cluster [DBG] pgmap v6911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:43 smithi121 bash[36902]: audit 2022-04-22T20:11:43.418035+0000 mon.a (mon.0) 8818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:43 smithi115 bash[45495]: cluster 2022-04-22T20:11:42.980968+0000 mgr.y (mgr.34107) 5675 : cluster [DBG] pgmap v6911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:44.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:43 smithi115 bash[45495]: audit 2022-04-22T20:11:43.418035+0000 mon.a (mon.0) 8818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:43 smithi115 bash[46934]: cluster 2022-04-22T20:11:42.980968+0000 mgr.y (mgr.34107) 5675 : cluster [DBG] pgmap v6911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:44.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:43 smithi115 bash[46934]: audit 2022-04-22T20:11:43.418035+0000 mon.a (mon.0) 8818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:44.134Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:45 smithi121 bash[36902]: cluster 2022-04-22T20:11:44.982087+0000 mgr.y (mgr.34107) 5676 : cluster [DBG] pgmap v6912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:45 smithi115 bash[45495]: cluster 2022-04-22T20:11:44.982087+0000 mgr.y (mgr.34107) 5676 : cluster [DBG] pgmap v6912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:45 smithi115 bash[46934]: cluster 2022-04-22T20:11:44.982087+0000 mgr.y (mgr.34107) 5676 : cluster [DBG] pgmap v6912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:46 smithi121 bash[36902]: audit 2022-04-22T20:11:45.730889+0000 mon.a (mon.0) 8819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:46.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:46 smithi115 bash[45495]: audit 2022-04-22T20:11:45.730889+0000 mon.a (mon.0) 8819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:46 smithi115 bash[46934]: audit 2022-04-22T20:11:45.730889+0000 mon.a (mon.0) 8819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:47 smithi121 bash[36902]: cluster 2022-04-22T20:11:46.983189+0000 mgr.y (mgr.34107) 5677 : cluster [DBG] pgmap v6913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:47 smithi121 bash[36902]: audit 2022-04-22T20:11:46.986572+0000 mon.a (mon.0) 8820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:47 smithi115 bash[45495]: cluster 2022-04-22T20:11:46.983189+0000 mgr.y (mgr.34107) 5677 : cluster [DBG] pgmap v6913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:47 smithi115 bash[45495]: audit 2022-04-22T20:11:46.986572+0000 mon.a (mon.0) 8820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:47 smithi115 bash[46934]: cluster 2022-04-22T20:11:46.983189+0000 mgr.y (mgr.34107) 5677 : cluster [DBG] pgmap v6913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:47 smithi115 bash[46934]: audit 2022-04-22T20:11:46.986572+0000 mon.a (mon.0) 8820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:48 smithi121 bash[36902]: audit 2022-04-22T20:11:48.420942+0000 mon.a (mon.0) 8821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:48.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:48 smithi115 bash[45495]: audit 2022-04-22T20:11:48.420942+0000 mon.a (mon.0) 8821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:48 smithi115 bash[46934]: audit 2022-04-22T20:11:48.420942+0000 mon.a (mon.0) 8821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:49 smithi121 bash[36902]: cluster 2022-04-22T20:11:48.984425+0000 mgr.y (mgr.34107) 5678 : cluster [DBG] pgmap v6914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:50.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:49 smithi115 bash[45495]: cluster 2022-04-22T20:11:48.984425+0000 mgr.y (mgr.34107) 5678 : cluster [DBG] pgmap v6914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:50.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:49 smithi115 bash[46934]: cluster 2022-04-22T20:11:48.984425+0000 mgr.y (mgr.34107) 5678 : cluster [DBG] pgmap v6914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:51.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:50 smithi115 bash[45495]: audit 2022-04-22T20:11:50.734827+0000 mon.a (mon.0) 8822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:51.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:50 smithi115 bash[46934]: audit 2022-04-22T20:11:50.734827+0000 mon.a (mon.0) 8822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:50 smithi121 bash[36902]: audit 2022-04-22T20:11:50.734827+0000 mon.a (mon.0) 8822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:51 smithi121 bash[36902]: cluster 2022-04-22T20:11:50.985572+0000 mgr.y (mgr.34107) 5679 : cluster [DBG] pgmap v6915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:52.265 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:51 smithi115 bash[46934]: cluster 2022-04-22T20:11:50.985572+0000 mgr.y (mgr.34107) 5679 : cluster [DBG] pgmap v6915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:52.265 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:51 smithi115 bash[45495]: cluster 2022-04-22T20:11:50.985572+0000 mgr.y (mgr.34107) 5679 : cluster [DBG] pgmap v6915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:52.256Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:52 smithi121 bash[36902]: audit 2022-04-22T20:11:51.989644+0000 mon.a (mon.0) 8823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:52 smithi115 bash[45495]: audit 2022-04-22T20:11:51.989644+0000 mon.a (mon.0) 8823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:52 smithi115 bash[46934]: audit 2022-04-22T20:11:51.989644+0000 mon.a (mon.0) 8823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:54.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:53 smithi115 bash[45495]: cluster 2022-04-22T20:11:52.986623+0000 mgr.y (mgr.34107) 5680 : cluster [DBG] pgmap v6916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:54.145 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:53 smithi115 bash[45495]: audit 2022-04-22T20:11:53.423794+0000 mon.a (mon.0) 8824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:54.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:53 smithi115 bash[46934]: cluster 2022-04-22T20:11:52.986623+0000 mgr.y (mgr.34107) 5680 : cluster [DBG] pgmap v6916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:54.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:53 smithi115 bash[46934]: audit 2022-04-22T20:11:53.423794+0000 mon.a (mon.0) 8824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:53 smithi121 bash[36902]: cluster 2022-04-22T20:11:52.986623+0000 mgr.y (mgr.34107) 5680 : cluster [DBG] pgmap v6916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:53 smithi121 bash[36902]: audit 2022-04-22T20:11:53.423794+0000 mon.a (mon.0) 8824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:11:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:11:54.135Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:11:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:55 smithi121 bash[36902]: cluster 2022-04-22T20:11:54.987750+0000 mgr.y (mgr.34107) 5681 : cluster [DBG] pgmap v6917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:55 smithi115 bash[45495]: cluster 2022-04-22T20:11:54.987750+0000 mgr.y (mgr.34107) 5681 : cluster [DBG] pgmap v6917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:55.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:55 smithi115 bash[46934]: cluster 2022-04-22T20:11:54.987750+0000 mgr.y (mgr.34107) 5681 : cluster [DBG] pgmap v6917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:56 smithi121 bash[36902]: audit 2022-04-22T20:11:55.738756+0000 mon.a (mon.0) 8825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:56.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:56 smithi115 bash[45495]: audit 2022-04-22T20:11:55.738756+0000 mon.a (mon.0) 8825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:56.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:56 smithi115 bash[46934]: audit 2022-04-22T20:11:55.738756+0000 mon.a (mon.0) 8825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:11:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:57 smithi121 bash[36902]: cluster 2022-04-22T20:11:56.988906+0000 mgr.y (mgr.34107) 5682 : cluster [DBG] pgmap v6918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:57 smithi121 bash[36902]: audit 2022-04-22T20:11:56.993121+0000 mon.a (mon.0) 8826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:57 smithi115 bash[45495]: cluster 2022-04-22T20:11:56.988906+0000 mgr.y (mgr.34107) 5682 : cluster [DBG] pgmap v6918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:57 smithi115 bash[45495]: audit 2022-04-22T20:11:56.993121+0000 mon.a (mon.0) 8826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:57 smithi115 bash[46934]: cluster 2022-04-22T20:11:56.988906+0000 mgr.y (mgr.34107) 5682 : cluster [DBG] pgmap v6918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:11:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:57 smithi115 bash[46934]: audit 2022-04-22T20:11:56.993121+0000 mon.a (mon.0) 8826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:11:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:58 smithi121 bash[36902]: audit 2022-04-22T20:11:58.427215+0000 mon.a (mon.0) 8827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:58 smithi115 bash[45495]: audit 2022-04-22T20:11:58.427215+0000 mon.a (mon.0) 8827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:11:58.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:58 smithi115 bash[46934]: audit 2022-04-22T20:11:58.427215+0000 mon.a (mon.0) 8827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:11:59 smithi121 bash[36902]: cluster 2022-04-22T20:11:58.990084+0000 mgr.y (mgr.34107) 5683 : cluster [DBG] pgmap v6919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:00.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:11:59 smithi115 bash[46934]: cluster 2022-04-22T20:11:58.990084+0000 mgr.y (mgr.34107) 5683 : cluster [DBG] pgmap v6919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:00.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:11:59 smithi115 bash[45495]: cluster 2022-04-22T20:11:58.990084+0000 mgr.y (mgr.34107) 5683 : cluster [DBG] pgmap v6919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:00 smithi121 bash[36902]: audit 2022-04-22T20:12:00.742506+0000 mon.a (mon.0) 8828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:01.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:00 smithi115 bash[45495]: audit 2022-04-22T20:12:00.742506+0000 mon.a (mon.0) 8828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:01.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:00 smithi115 bash[46934]: audit 2022-04-22T20:12:00.742506+0000 mon.a (mon.0) 8828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:01 smithi121 bash[36902]: cluster 2022-04-22T20:12:00.991150+0000 mgr.y (mgr.34107) 5684 : cluster [DBG] pgmap v6920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:02.266 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:01 smithi115 bash[45495]: cluster 2022-04-22T20:12:00.991150+0000 mgr.y (mgr.34107) 5684 : cluster [DBG] pgmap v6920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:02.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:01 smithi115 bash[46934]: cluster 2022-04-22T20:12:00.991150+0000 mgr.y (mgr.34107) 5684 : cluster [DBG] pgmap v6920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:02.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:02.258Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:02 smithi121 bash[36902]: audit 2022-04-22T20:12:01.996251+0000 mon.a (mon.0) 8829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:02 smithi115 bash[45495]: audit 2022-04-22T20:12:01.996251+0000 mon.a (mon.0) 8829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:02 smithi115 bash[46934]: audit 2022-04-22T20:12:01.996251+0000 mon.a (mon.0) 8829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:04.145 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:03 smithi115 bash[45495]: cluster 2022-04-22T20:12:02.992308+0000 mgr.y (mgr.34107) 5685 : cluster [DBG] pgmap v6921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:04.145 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:03 smithi115 bash[45495]: audit 2022-04-22T20:12:03.430272+0000 mon.a (mon.0) 8830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:04.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:03 smithi115 bash[46934]: cluster 2022-04-22T20:12:02.992308+0000 mgr.y (mgr.34107) 5685 : cluster [DBG] pgmap v6921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:04.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:03 smithi115 bash[46934]: audit 2022-04-22T20:12:03.430272+0000 mon.a (mon.0) 8830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:03 smithi121 bash[36902]: cluster 2022-04-22T20:12:02.992308+0000 mgr.y (mgr.34107) 5685 : cluster [DBG] pgmap v6921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:03 smithi121 bash[36902]: audit 2022-04-22T20:12:03.430272+0000 mon.a (mon.0) 8830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:04.136Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:05 smithi121 bash[36902]: cluster 2022-04-22T20:12:04.993453+0000 mgr.y (mgr.34107) 5686 : cluster [DBG] pgmap v6922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:05 smithi115 bash[45495]: cluster 2022-04-22T20:12:04.993453+0000 mgr.y (mgr.34107) 5686 : cluster [DBG] pgmap v6922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:05 smithi115 bash[46934]: cluster 2022-04-22T20:12:04.993453+0000 mgr.y (mgr.34107) 5686 : cluster [DBG] pgmap v6922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:06 smithi121 bash[36902]: audit 2022-04-22T20:12:05.746331+0000 mon.a (mon.0) 8831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:06 smithi115 bash[45495]: audit 2022-04-22T20:12:05.746331+0000 mon.a (mon.0) 8831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:06.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:06 smithi115 bash[46934]: audit 2022-04-22T20:12:05.746331+0000 mon.a (mon.0) 8831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:07 smithi121 bash[36902]: cluster 2022-04-22T20:12:06.994557+0000 mgr.y (mgr.34107) 5687 : cluster [DBG] pgmap v6923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:07 smithi121 bash[36902]: audit 2022-04-22T20:12:06.999349+0000 mon.a (mon.0) 8832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:07.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:07 smithi115 bash[45495]: cluster 2022-04-22T20:12:06.994557+0000 mgr.y (mgr.34107) 5687 : cluster [DBG] pgmap v6923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:07.517 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:07 smithi115 bash[45495]: audit 2022-04-22T20:12:06.999349+0000 mon.a (mon.0) 8832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:07.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:07 smithi115 bash[46934]: cluster 2022-04-22T20:12:06.994557+0000 mgr.y (mgr.34107) 5687 : cluster [DBG] pgmap v6923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:07.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:07 smithi115 bash[46934]: audit 2022-04-22T20:12:06.999349+0000 mon.a (mon.0) 8832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:08.061 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 10m ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 61s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 10m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 61s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:12:08.404 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 61s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 10m ago 2h 483M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 10m ago 2h 311M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 61s ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 10m ago 2h 200M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 10m ago 2h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 61s ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 10m ago 2h 703M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 10m ago 2h 783M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 10m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:12:08.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 10m ago 2h 487M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:12:08.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 61s ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:12:08.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 61s ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:12:08.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 61s ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:12:08.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 61s ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:12:08.407 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 61s ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:12:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:08 smithi121 bash[36902]: audit 2022-04-22T20:12:07.491743+0000 mon.a (mon.0) 8833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:12:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:08 smithi121 bash[36902]: audit 2022-04-22T20:12:07.494112+0000 mon.a (mon.0) 8834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:12:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:08 smithi121 bash[36902]: audit 2022-04-22T20:12:07.495334+0000 mon.a (mon.0) 8835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:12:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:08 smithi121 bash[36902]: audit 2022-04-22T20:12:07.794825+0000 mon.a (mon.0) 8836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:08 smithi121 bash[36902]: audit 2022-04-22T20:12:07.812240+0000 mon.a (mon.0) 8837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[45495]: audit 2022-04-22T20:12:07.491743+0000 mon.a (mon.0) 8833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[45495]: audit 2022-04-22T20:12:07.494112+0000 mon.a (mon.0) 8834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[45495]: audit 2022-04-22T20:12:07.495334+0000 mon.a (mon.0) 8835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[45495]: audit 2022-04-22T20:12:07.794825+0000 mon.a (mon.0) 8836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[45495]: audit 2022-04-22T20:12:07.812240+0000 mon.a (mon.0) 8837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[46934]: audit 2022-04-22T20:12:07.491743+0000 mon.a (mon.0) 8833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:12:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[46934]: audit 2022-04-22T20:12:07.494112+0000 mon.a (mon.0) 8834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[46934]: audit 2022-04-22T20:12:07.495334+0000 mon.a (mon.0) 8835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[46934]: audit 2022-04-22T20:12:07.794825+0000 mon.a (mon.0) 8836 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:08 smithi115 bash[46934]: audit 2022-04-22T20:12:07.812240+0000 mon.a (mon.0) 8837 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:08.805 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:08.806 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:08.807 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:12:08.808 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:12:08.808 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:12:08.808 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:12:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:09 smithi121 bash[36902]: audit 2022-04-22T20:12:08.037276+0000 mgr.y (mgr.34107) 5688 : audit [DBG] from='client.41310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:09 smithi121 bash[36902]: audit 2022-04-22T20:12:08.379276+0000 mgr.y (mgr.34107) 5689 : audit [DBG] from='client.51250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:09 smithi121 bash[36902]: audit 2022-04-22T20:12:08.433494+0000 mon.a (mon.0) 8838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:09 smithi121 bash[36902]: audit 2022-04-22T20:12:08.796459+0000 mon.a (mon.0) 8839 : audit [DBG] from='client.? 172.21.15.115:0/3380960489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:09.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:09 smithi121 bash[36902]: cluster 2022-04-22T20:12:08.995771+0000 mgr.y (mgr.34107) 5690 : cluster [DBG] pgmap v6924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:09.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[45495]: audit 2022-04-22T20:12:08.037276+0000 mgr.y (mgr.34107) 5688 : audit [DBG] from='client.41310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[45495]: audit 2022-04-22T20:12:08.379276+0000 mgr.y (mgr.34107) 5689 : audit [DBG] from='client.51250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[45495]: audit 2022-04-22T20:12:08.433494+0000 mon.a (mon.0) 8838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[45495]: audit 2022-04-22T20:12:08.796459+0000 mon.a (mon.0) 8839 : audit [DBG] from='client.? 172.21.15.115:0/3380960489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[45495]: cluster 2022-04-22T20:12:08.995771+0000 mgr.y (mgr.34107) 5690 : cluster [DBG] pgmap v6924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[46934]: audit 2022-04-22T20:12:08.037276+0000 mgr.y (mgr.34107) 5688 : audit [DBG] from='client.41310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[46934]: audit 2022-04-22T20:12:08.379276+0000 mgr.y (mgr.34107) 5689 : audit [DBG] from='client.51250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[46934]: audit 2022-04-22T20:12:08.433494+0000 mon.a (mon.0) 8838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[46934]: audit 2022-04-22T20:12:08.796459+0000 mon.a (mon.0) 8839 : audit [DBG] from='client.? 172.21.15.115:0/3380960489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:09.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:09 smithi115 bash[46934]: cluster 2022-04-22T20:12:08.995771+0000 mgr.y (mgr.34107) 5690 : cluster [DBG] pgmap v6924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:10 smithi121 bash[36902]: audit 2022-04-22T20:12:10.749791+0000 mon.a (mon.0) 8840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:10 smithi115 bash[45495]: audit 2022-04-22T20:12:10.749791+0000 mon.a (mon.0) 8840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:10 smithi115 bash[46934]: audit 2022-04-22T20:12:10.749791+0000 mon.a (mon.0) 8840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:11 smithi121 bash[36902]: cluster 2022-04-22T20:12:10.996467+0000 mgr.y (mgr.34107) 5691 : cluster [DBG] pgmap v6925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:12.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:11 smithi115 bash[46934]: cluster 2022-04-22T20:12:10.996467+0000 mgr.y (mgr.34107) 5691 : cluster [DBG] pgmap v6925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:12.272 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:11 smithi115 bash[45495]: cluster 2022-04-22T20:12:10.996467+0000 mgr.y (mgr.34107) 5691 : cluster [DBG] pgmap v6925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:12.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:12 smithi121 bash[36902]: audit 2022-04-22T20:12:11.957182+0000 mon.a (mon.0) 8841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:12 smithi121 bash[36902]: audit 2022-04-22T20:12:12.002449+0000 mon.a (mon.0) 8842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:12 smithi115 bash[45495]: audit 2022-04-22T20:12:11.957182+0000 mon.a (mon.0) 8841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:12 smithi115 bash[45495]: audit 2022-04-22T20:12:12.002449+0000 mon.a (mon.0) 8842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:12 smithi115 bash[46934]: audit 2022-04-22T20:12:11.957182+0000 mon.a (mon.0) 8841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:12 smithi115 bash[46934]: audit 2022-04-22T20:12:12.002449+0000 mon.a (mon.0) 8842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:13 smithi121 bash[36902]: cluster 2022-04-22T20:12:12.997359+0000 mgr.y (mgr.34107) 5692 : cluster [DBG] pgmap v6926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:13 smithi121 bash[36902]: audit 2022-04-22T20:12:13.436484+0000 mon.a (mon.0) 8843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:14.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:14.137Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:13 smithi115 bash[45495]: cluster 2022-04-22T20:12:12.997359+0000 mgr.y (mgr.34107) 5692 : cluster [DBG] pgmap v6926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:13 smithi115 bash[45495]: audit 2022-04-22T20:12:13.436484+0000 mon.a (mon.0) 8843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:13 smithi115 bash[46934]: cluster 2022-04-22T20:12:12.997359+0000 mgr.y (mgr.34107) 5692 : cluster [DBG] pgmap v6926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:13 smithi115 bash[46934]: audit 2022-04-22T20:12:13.436484+0000 mon.a (mon.0) 8843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:15.367 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:15 smithi115 bash[45495]: cluster 2022-04-22T20:12:14.998411+0000 mgr.y (mgr.34107) 5693 : cluster [DBG] pgmap v6927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:15 smithi115 bash[46934]: cluster 2022-04-22T20:12:14.998411+0000 mgr.y (mgr.34107) 5693 : cluster [DBG] pgmap v6927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:15 smithi121 bash[36902]: cluster 2022-04-22T20:12:14.998411+0000 mgr.y (mgr.34107) 5693 : cluster [DBG] pgmap v6927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:16 smithi121 bash[36902]: audit 2022-04-22T20:12:15.185486+0000 mon.a (mon.0) 8844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:12:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:16 smithi121 bash[36902]: audit 2022-04-22T20:12:15.186064+0000 mon.a (mon.0) 8845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:12:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:16 smithi121 bash[36902]: audit 2022-04-22T20:12:15.753267+0000 mon.a (mon.0) 8846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[45495]: audit 2022-04-22T20:12:15.185486+0000 mon.a (mon.0) 8844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[45495]: audit 2022-04-22T20:12:15.186064+0000 mon.a (mon.0) 8845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[45495]: audit 2022-04-22T20:12:15.753267+0000 mon.a (mon.0) 8846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[46934]: audit 2022-04-22T20:12:15.185486+0000 mon.a (mon.0) 8844 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[46934]: audit 2022-04-22T20:12:15.186064+0000 mon.a (mon.0) 8845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:12:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:16 smithi115 bash[46934]: audit 2022-04-22T20:12:15.753267+0000 mon.a (mon.0) 8846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:17 smithi121 bash[36902]: cluster 2022-04-22T20:12:16.999607+0000 mgr.y (mgr.34107) 5694 : cluster [DBG] pgmap v6928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:17 smithi121 bash[36902]: audit 2022-04-22T20:12:17.004960+0000 mon.a (mon.0) 8847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:17 smithi115 bash[45495]: cluster 2022-04-22T20:12:16.999607+0000 mgr.y (mgr.34107) 5694 : cluster [DBG] pgmap v6928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:17 smithi115 bash[45495]: audit 2022-04-22T20:12:17.004960+0000 mon.a (mon.0) 8847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:17 smithi115 bash[46934]: cluster 2022-04-22T20:12:16.999607+0000 mgr.y (mgr.34107) 5694 : cluster [DBG] pgmap v6928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:17 smithi115 bash[46934]: audit 2022-04-22T20:12:17.004960+0000 mon.a (mon.0) 8847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:18.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:18 smithi121 bash[36902]: audit 2022-04-22T20:12:18.439463+0000 mon.a (mon.0) 8848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:18.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:18 smithi115 bash[45495]: audit 2022-04-22T20:12:18.439463+0000 mon.a (mon.0) 8848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:18.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:18 smithi115 bash[46934]: audit 2022-04-22T20:12:18.439463+0000 mon.a (mon.0) 8848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:20 smithi121 bash[36902]: cluster 2022-04-22T20:12:19.000779+0000 mgr.y (mgr.34107) 5695 : cluster [DBG] pgmap v6929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:20 smithi115 bash[46934]: cluster 2022-04-22T20:12:19.000779+0000 mgr.y (mgr.34107) 5695 : cluster [DBG] pgmap v6929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:21.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:20 smithi115 bash[45495]: cluster 2022-04-22T20:12:19.000779+0000 mgr.y (mgr.34107) 5695 : cluster [DBG] pgmap v6929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:21 smithi121 bash[36902]: audit 2022-04-22T20:12:20.756873+0000 mon.a (mon.0) 8849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:21 smithi121 bash[36902]: audit 2022-04-22T20:12:21.097365+0000 mon.a (mon.0) 8850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:21 smithi121 bash[36902]: audit 2022-04-22T20:12:21.104891+0000 mon.a (mon.0) 8851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[45495]: audit 2022-04-22T20:12:20.756873+0000 mon.a (mon.0) 8849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[45495]: audit 2022-04-22T20:12:21.097365+0000 mon.a (mon.0) 8850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[45495]: audit 2022-04-22T20:12:21.104891+0000 mon.a (mon.0) 8851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[46934]: audit 2022-04-22T20:12:20.756873+0000 mon.a (mon.0) 8849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:22.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[46934]: audit 2022-04-22T20:12:21.097365+0000 mon.a (mon.0) 8850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:21 smithi115 bash[46934]: audit 2022-04-22T20:12:21.104891+0000 mon.a (mon.0) 8851 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:12:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:22.257Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:22 smithi121 bash[36902]: cluster 2022-04-22T20:12:21.002131+0000 mgr.y (mgr.34107) 5696 : cluster [DBG] pgmap v6930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:22 smithi121 bash[36902]: cephadm 2022-04-22T20:12:21.087162+0000 mgr.y (mgr.34107) 5697 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:12:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:22 smithi121 bash[36902]: audit 2022-04-22T20:12:22.007734+0000 mon.a (mon.0) 8852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[45495]: cluster 2022-04-22T20:12:21.002131+0000 mgr.y (mgr.34107) 5696 : cluster [DBG] pgmap v6930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[45495]: cephadm 2022-04-22T20:12:21.087162+0000 mgr.y (mgr.34107) 5697 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:12:23.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[45495]: audit 2022-04-22T20:12:22.007734+0000 mon.a (mon.0) 8852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[46934]: cluster 2022-04-22T20:12:21.002131+0000 mgr.y (mgr.34107) 5696 : cluster [DBG] pgmap v6930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:23.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[46934]: cephadm 2022-04-22T20:12:21.087162+0000 mgr.y (mgr.34107) 5697 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:12:23.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:22 smithi115 bash[46934]: audit 2022-04-22T20:12:22.007734+0000 mon.a (mon.0) 8852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:23 smithi121 bash[36902]: audit 2022-04-22T20:12:23.442366+0000 mon.a (mon.0) 8853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:24.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:23 smithi115 bash[45495]: audit 2022-04-22T20:12:23.442366+0000 mon.a (mon.0) 8853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:23 smithi115 bash[46934]: audit 2022-04-22T20:12:23.442366+0000 mon.a (mon.0) 8853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:24.139Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:24 smithi115 bash[45495]: cluster 2022-04-22T20:12:23.003082+0000 mgr.y (mgr.34107) 5698 : cluster [DBG] pgmap v6931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:24 smithi115 bash[46934]: cluster 2022-04-22T20:12:23.003082+0000 mgr.y (mgr.34107) 5698 : cluster [DBG] pgmap v6931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:24 smithi121 bash[36902]: cluster 2022-04-22T20:12:23.003082+0000 mgr.y (mgr.34107) 5698 : cluster [DBG] pgmap v6931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:26 smithi115 bash[45495]: cluster 2022-04-22T20:12:25.004200+0000 mgr.y (mgr.34107) 5699 : cluster [DBG] pgmap v6932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:27.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:26 smithi115 bash[45495]: audit 2022-04-22T20:12:25.763986+0000 mon.a (mon.0) 8854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:26 smithi115 bash[46934]: cluster 2022-04-22T20:12:25.004200+0000 mgr.y (mgr.34107) 5699 : cluster [DBG] pgmap v6932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:27.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:26 smithi115 bash[46934]: audit 2022-04-22T20:12:25.763986+0000 mon.a (mon.0) 8854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:26 smithi121 bash[36902]: cluster 2022-04-22T20:12:25.004200+0000 mgr.y (mgr.34107) 5699 : cluster [DBG] pgmap v6932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:26 smithi121 bash[36902]: audit 2022-04-22T20:12:25.763986+0000 mon.a (mon.0) 8854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:28.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:27 smithi115 bash[45495]: audit 2022-04-22T20:12:27.010932+0000 mon.a (mon.0) 8855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:27 smithi115 bash[46934]: audit 2022-04-22T20:12:27.010932+0000 mon.a (mon.0) 8855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:27 smithi121 bash[36902]: audit 2022-04-22T20:12:27.010932+0000 mon.a (mon.0) 8855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:28 smithi115 bash[45495]: cluster 2022-04-22T20:12:27.005304+0000 mgr.y (mgr.34107) 5700 : cluster [DBG] pgmap v6933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:29.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:28 smithi115 bash[45495]: audit 2022-04-22T20:12:28.445474+0000 mon.a (mon.0) 8856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:28 smithi115 bash[46934]: cluster 2022-04-22T20:12:27.005304+0000 mgr.y (mgr.34107) 5700 : cluster [DBG] pgmap v6933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:29.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:28 smithi115 bash[46934]: audit 2022-04-22T20:12:28.445474+0000 mon.a (mon.0) 8856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:28 smithi121 bash[36902]: cluster 2022-04-22T20:12:27.005304+0000 mgr.y (mgr.34107) 5700 : cluster [DBG] pgmap v6933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:28 smithi121 bash[36902]: audit 2022-04-22T20:12:28.445474+0000 mon.a (mon.0) 8856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:30 smithi115 bash[45495]: cluster 2022-04-22T20:12:29.006466+0000 mgr.y (mgr.34107) 5701 : cluster [DBG] pgmap v6934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:30 smithi115 bash[46934]: cluster 2022-04-22T20:12:29.006466+0000 mgr.y (mgr.34107) 5701 : cluster [DBG] pgmap v6934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:30 smithi121 bash[36902]: cluster 2022-04-22T20:12:29.006466+0000 mgr.y (mgr.34107) 5701 : cluster [DBG] pgmap v6934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:32.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:31 smithi115 bash[45495]: audit 2022-04-22T20:12:31.764809+0000 mon.a (mon.0) 8857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:32.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:31 smithi115 bash[46934]: audit 2022-04-22T20:12:31.764809+0000 mon.a (mon.0) 8857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:31 smithi121 bash[36902]: audit 2022-04-22T20:12:31.764809+0000 mon.a (mon.0) 8857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:32.258Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:32 smithi115 bash[45495]: cluster 2022-04-22T20:12:31.007632+0000 mgr.y (mgr.34107) 5702 : cluster [DBG] pgmap v6935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:32 smithi115 bash[45495]: audit 2022-04-22T20:12:32.014219+0000 mon.a (mon.0) 8858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:32 smithi115 bash[46934]: cluster 2022-04-22T20:12:31.007632+0000 mgr.y (mgr.34107) 5702 : cluster [DBG] pgmap v6935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:33.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:32 smithi115 bash[46934]: audit 2022-04-22T20:12:32.014219+0000 mon.a (mon.0) 8858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:32 smithi121 bash[36902]: cluster 2022-04-22T20:12:31.007632+0000 mgr.y (mgr.34107) 5702 : cluster [DBG] pgmap v6935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:32 smithi121 bash[36902]: audit 2022-04-22T20:12:32.014219+0000 mon.a (mon.0) 8858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:34.147 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:33 smithi115 bash[45495]: audit 2022-04-22T20:12:33.448504+0000 mon.a (mon.0) 8859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:34.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:33 smithi115 bash[46934]: audit 2022-04-22T20:12:33.448504+0000 mon.a (mon.0) 8859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:33 smithi121 bash[36902]: audit 2022-04-22T20:12:33.448504+0000 mon.a (mon.0) 8859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:34.139Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:34 smithi121 bash[36902]: cluster 2022-04-22T20:12:33.008799+0000 mgr.y (mgr.34107) 5703 : cluster [DBG] pgmap v6936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:34 smithi115 bash[45495]: cluster 2022-04-22T20:12:33.008799+0000 mgr.y (mgr.34107) 5703 : cluster [DBG] pgmap v6936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:34 smithi115 bash[46934]: cluster 2022-04-22T20:12:33.008799+0000 mgr.y (mgr.34107) 5703 : cluster [DBG] pgmap v6936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:36 smithi121 bash[36902]: cluster 2022-04-22T20:12:35.010025+0000 mgr.y (mgr.34107) 5704 : cluster [DBG] pgmap v6937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:36 smithi121 bash[36902]: audit 2022-04-22T20:12:36.768402+0000 mon.a (mon.0) 8860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:36 smithi115 bash[45495]: cluster 2022-04-22T20:12:35.010025+0000 mgr.y (mgr.34107) 5704 : cluster [DBG] pgmap v6937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:36 smithi115 bash[45495]: audit 2022-04-22T20:12:36.768402+0000 mon.a (mon.0) 8860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:36 smithi115 bash[46934]: cluster 2022-04-22T20:12:35.010025+0000 mgr.y (mgr.34107) 5704 : cluster [DBG] pgmap v6937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:36 smithi115 bash[46934]: audit 2022-04-22T20:12:36.768402+0000 mon.a (mon.0) 8860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:37 smithi121 bash[36902]: audit 2022-04-22T20:12:37.017043+0000 mon.a (mon.0) 8861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:37 smithi115 bash[45495]: audit 2022-04-22T20:12:37.017043+0000 mon.a (mon.0) 8861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:37 smithi115 bash[46934]: audit 2022-04-22T20:12:37.017043+0000 mon.a (mon.0) 8861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:39.178 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:12:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:38 smithi121 bash[36902]: cluster 2022-04-22T20:12:37.011189+0000 mgr.y (mgr.34107) 5705 : cluster [DBG] pgmap v6938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:38 smithi121 bash[36902]: audit 2022-04-22T20:12:38.451696+0000 mon.a (mon.0) 8862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:38 smithi115 bash[46934]: cluster 2022-04-22T20:12:37.011189+0000 mgr.y (mgr.34107) 5705 : cluster [DBG] pgmap v6938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:38 smithi115 bash[46934]: audit 2022-04-22T20:12:38.451696+0000 mon.a (mon.0) 8862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:38 smithi115 bash[45495]: cluster 2022-04-22T20:12:37.011189+0000 mgr.y (mgr.34107) 5705 : cluster [DBG] pgmap v6938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:38 smithi115 bash[45495]: audit 2022-04-22T20:12:38.451696+0000 mon.a (mon.0) 8862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:39.521 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 27s ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 92s ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 27s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 92s ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 92s ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 27s ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 27s ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 92s ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 27s ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:12:39.522 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 27s ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 92s ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 27s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 27s ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 27s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 27s ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 92s ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 92s ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:12:39.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 92s ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:12:39.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 92s ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:12:39.524 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 92s ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:12:39.920 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:39.926 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:12:39.927 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:12:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:40 smithi121 bash[36902]: cluster 2022-04-22T20:12:39.012323+0000 mgr.y (mgr.34107) 5706 : cluster [DBG] pgmap v6939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:40 smithi121 bash[36902]: audit 2022-04-22T20:12:39.153612+0000 mgr.y (mgr.34107) 5707 : audit [DBG] from='client.41328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:40 smithi121 bash[36902]: audit 2022-04-22T20:12:39.498496+0000 mgr.y (mgr.34107) 5708 : audit [DBG] from='client.41334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:40 smithi121 bash[36902]: audit 2022-04-22T20:12:39.911007+0000 mon.c (mon.1) 228 : audit [DBG] from='client.? 172.21.15.115:0/1240744978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[45495]: cluster 2022-04-22T20:12:39.012323+0000 mgr.y (mgr.34107) 5706 : cluster [DBG] pgmap v6939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[45495]: audit 2022-04-22T20:12:39.153612+0000 mgr.y (mgr.34107) 5707 : audit [DBG] from='client.41328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[45495]: audit 2022-04-22T20:12:39.498496+0000 mgr.y (mgr.34107) 5708 : audit [DBG] from='client.41334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[45495]: audit 2022-04-22T20:12:39.911007+0000 mon.c (mon.1) 228 : audit [DBG] from='client.? 172.21.15.115:0/1240744978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[46934]: cluster 2022-04-22T20:12:39.012323+0000 mgr.y (mgr.34107) 5706 : cluster [DBG] pgmap v6939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[46934]: audit 2022-04-22T20:12:39.153612+0000 mgr.y (mgr.34107) 5707 : audit [DBG] from='client.41328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[46934]: audit 2022-04-22T20:12:39.498496+0000 mgr.y (mgr.34107) 5708 : audit [DBG] from='client.41334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:12:41.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:40 smithi115 bash[46934]: audit 2022-04-22T20:12:39.911007+0000 mon.c (mon.1) 228 : audit [DBG] from='client.? 172.21.15.115:0/1240744978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:12:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:41 smithi121 bash[36902]: audit 2022-04-22T20:12:41.772247+0000 mon.a (mon.0) 8863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:42.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:41 smithi115 bash[45495]: audit 2022-04-22T20:12:41.772247+0000 mon.a (mon.0) 8863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:42.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:41 smithi115 bash[46934]: audit 2022-04-22T20:12:41.772247+0000 mon.a (mon.0) 8863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:42.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:42.260Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:42 smithi121 bash[36902]: cluster 2022-04-22T20:12:41.013421+0000 mgr.y (mgr.34107) 5709 : cluster [DBG] pgmap v6940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:42 smithi121 bash[36902]: audit 2022-04-22T20:12:42.020161+0000 mon.a (mon.0) 8864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:42 smithi115 bash[45495]: cluster 2022-04-22T20:12:41.013421+0000 mgr.y (mgr.34107) 5709 : cluster [DBG] pgmap v6940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:42 smithi115 bash[45495]: audit 2022-04-22T20:12:42.020161+0000 mon.a (mon.0) 8864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:42 smithi115 bash[46934]: cluster 2022-04-22T20:12:41.013421+0000 mgr.y (mgr.34107) 5709 : cluster [DBG] pgmap v6940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:42 smithi115 bash[46934]: audit 2022-04-22T20:12:42.020161+0000 mon.a (mon.0) 8864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:44.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:43 smithi115 bash[45495]: audit 2022-04-22T20:12:43.454628+0000 mon.a (mon.0) 8865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:44.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:43 smithi115 bash[46934]: audit 2022-04-22T20:12:43.454628+0000 mon.a (mon.0) 8865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:43 smithi121 bash[36902]: audit 2022-04-22T20:12:43.454628+0000 mon.a (mon.0) 8865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:44.140Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:44 smithi121 bash[36902]: cluster 2022-04-22T20:12:43.014416+0000 mgr.y (mgr.34107) 5710 : cluster [DBG] pgmap v6941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:45.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:44 smithi115 bash[46934]: cluster 2022-04-22T20:12:43.014416+0000 mgr.y (mgr.34107) 5710 : cluster [DBG] pgmap v6941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:44 smithi115 bash[45495]: cluster 2022-04-22T20:12:43.014416+0000 mgr.y (mgr.34107) 5710 : cluster [DBG] pgmap v6941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:46 smithi121 bash[36902]: cluster 2022-04-22T20:12:45.015543+0000 mgr.y (mgr.34107) 5711 : cluster [DBG] pgmap v6942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:47.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:46 smithi121 bash[36902]: audit 2022-04-22T20:12:46.776065+0000 mon.a (mon.0) 8866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:46 smithi115 bash[45495]: cluster 2022-04-22T20:12:45.015543+0000 mgr.y (mgr.34107) 5711 : cluster [DBG] pgmap v6942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:47.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:46 smithi115 bash[45495]: audit 2022-04-22T20:12:46.776065+0000 mon.a (mon.0) 8866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:47.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:46 smithi115 bash[46934]: cluster 2022-04-22T20:12:45.015543+0000 mgr.y (mgr.34107) 5711 : cluster [DBG] pgmap v6942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:47.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:46 smithi115 bash[46934]: audit 2022-04-22T20:12:46.776065+0000 mon.a (mon.0) 8866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:48.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:47 smithi121 bash[36902]: audit 2022-04-22T20:12:47.023167+0000 mon.a (mon.0) 8867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:47 smithi115 bash[45495]: audit 2022-04-22T20:12:47.023167+0000 mon.a (mon.0) 8867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:47 smithi115 bash[46934]: audit 2022-04-22T20:12:47.023167+0000 mon.a (mon.0) 8867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:48 smithi121 bash[36902]: cluster 2022-04-22T20:12:47.016628+0000 mgr.y (mgr.34107) 5712 : cluster [DBG] pgmap v6943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:48 smithi121 bash[36902]: audit 2022-04-22T20:12:48.457706+0000 mon.a (mon.0) 8868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:48 smithi115 bash[45495]: cluster 2022-04-22T20:12:47.016628+0000 mgr.y (mgr.34107) 5712 : cluster [DBG] pgmap v6943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:48 smithi115 bash[45495]: audit 2022-04-22T20:12:48.457706+0000 mon.a (mon.0) 8868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:48 smithi115 bash[46934]: cluster 2022-04-22T20:12:47.016628+0000 mgr.y (mgr.34107) 5712 : cluster [DBG] pgmap v6943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:48 smithi115 bash[46934]: audit 2022-04-22T20:12:48.457706+0000 mon.a (mon.0) 8868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:50 smithi121 bash[36902]: cluster 2022-04-22T20:12:49.017678+0000 mgr.y (mgr.34107) 5713 : cluster [DBG] pgmap v6944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:50 smithi115 bash[45495]: cluster 2022-04-22T20:12:49.017678+0000 mgr.y (mgr.34107) 5713 : cluster [DBG] pgmap v6944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:51.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:50 smithi115 bash[46934]: cluster 2022-04-22T20:12:49.017678+0000 mgr.y (mgr.34107) 5713 : cluster [DBG] pgmap v6944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:51 smithi121 bash[36902]: audit 2022-04-22T20:12:51.779921+0000 mon.a (mon.0) 8869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:52.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:51 smithi115 bash[45495]: audit 2022-04-22T20:12:51.779921+0000 mon.a (mon.0) 8869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:52.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:51 smithi115 bash[46934]: audit 2022-04-22T20:12:51.779921+0000 mon.a (mon.0) 8869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:52.259Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:52 smithi121 bash[36902]: cluster 2022-04-22T20:12:51.018756+0000 mgr.y (mgr.34107) 5714 : cluster [DBG] pgmap v6945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:52 smithi121 bash[36902]: audit 2022-04-22T20:12:52.026326+0000 mon.a (mon.0) 8870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:52 smithi115 bash[45495]: cluster 2022-04-22T20:12:51.018756+0000 mgr.y (mgr.34107) 5714 : cluster [DBG] pgmap v6945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:52 smithi115 bash[45495]: audit 2022-04-22T20:12:52.026326+0000 mon.a (mon.0) 8870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:52 smithi115 bash[46934]: cluster 2022-04-22T20:12:51.018756+0000 mgr.y (mgr.34107) 5714 : cluster [DBG] pgmap v6945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:52 smithi115 bash[46934]: audit 2022-04-22T20:12:52.026326+0000 mon.a (mon.0) 8870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:54.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:53 smithi115 bash[46934]: audit 2022-04-22T20:12:53.460697+0000 mon.a (mon.0) 8871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:54.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:53 smithi115 bash[45495]: audit 2022-04-22T20:12:53.460697+0000 mon.a (mon.0) 8871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:53 smithi121 bash[36902]: audit 2022-04-22T20:12:53.460697+0000 mon.a (mon.0) 8871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:12:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:12:54.140Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:12:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:54 smithi121 bash[36902]: cluster 2022-04-22T20:12:53.019812+0000 mgr.y (mgr.34107) 5715 : cluster [DBG] pgmap v6946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:54 smithi115 bash[45495]: cluster 2022-04-22T20:12:53.019812+0000 mgr.y (mgr.34107) 5715 : cluster [DBG] pgmap v6946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:55.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:54 smithi115 bash[46934]: cluster 2022-04-22T20:12:53.019812+0000 mgr.y (mgr.34107) 5715 : cluster [DBG] pgmap v6946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:56 smithi121 bash[36902]: cluster 2022-04-22T20:12:55.020917+0000 mgr.y (mgr.34107) 5716 : cluster [DBG] pgmap v6947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:56 smithi121 bash[36902]: audit 2022-04-22T20:12:56.783656+0000 mon.a (mon.0) 8872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:56 smithi115 bash[45495]: cluster 2022-04-22T20:12:55.020917+0000 mgr.y (mgr.34107) 5716 : cluster [DBG] pgmap v6947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:56 smithi115 bash[45495]: audit 2022-04-22T20:12:56.783656+0000 mon.a (mon.0) 8872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:56 smithi115 bash[46934]: cluster 2022-04-22T20:12:55.020917+0000 mgr.y (mgr.34107) 5716 : cluster [DBG] pgmap v6947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:56 smithi115 bash[46934]: audit 2022-04-22T20:12:56.783656+0000 mon.a (mon.0) 8872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:12:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:57 smithi121 bash[36902]: audit 2022-04-22T20:12:57.029589+0000 mon.a (mon.0) 8873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:57 smithi115 bash[45495]: audit 2022-04-22T20:12:57.029589+0000 mon.a (mon.0) 8873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:57 smithi115 bash[46934]: audit 2022-04-22T20:12:57.029589+0000 mon.a (mon.0) 8873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:12:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:58 smithi121 bash[36902]: cluster 2022-04-22T20:12:57.022118+0000 mgr.y (mgr.34107) 5717 : cluster [DBG] pgmap v6948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:12:58 smithi121 bash[36902]: audit 2022-04-22T20:12:58.463798+0000 mon.a (mon.0) 8874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:58 smithi115 bash[45495]: cluster 2022-04-22T20:12:57.022118+0000 mgr.y (mgr.34107) 5717 : cluster [DBG] pgmap v6948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:12:58 smithi115 bash[45495]: audit 2022-04-22T20:12:58.463798+0000 mon.a (mon.0) 8874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:12:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:58 smithi115 bash[46934]: cluster 2022-04-22T20:12:57.022118+0000 mgr.y (mgr.34107) 5717 : cluster [DBG] pgmap v6948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:12:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:12:58 smithi115 bash[46934]: audit 2022-04-22T20:12:58.463798+0000 mon.a (mon.0) 8874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:00 smithi121 bash[36902]: cluster 2022-04-22T20:12:59.023424+0000 mgr.y (mgr.34107) 5718 : cluster [DBG] pgmap v6949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:00 smithi115 bash[45495]: cluster 2022-04-22T20:12:59.023424+0000 mgr.y (mgr.34107) 5718 : cluster [DBG] pgmap v6949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:00 smithi115 bash[46934]: cluster 2022-04-22T20:12:59.023424+0000 mgr.y (mgr.34107) 5718 : cluster [DBG] pgmap v6949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:01 smithi121 bash[36902]: audit 2022-04-22T20:13:01.787331+0000 mon.a (mon.0) 8875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:02.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:01 smithi115 bash[45495]: audit 2022-04-22T20:13:01.787331+0000 mon.a (mon.0) 8875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:02.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:01 smithi115 bash[46934]: audit 2022-04-22T20:13:01.787331+0000 mon.a (mon.0) 8875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:02.259Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:02 smithi121 bash[36902]: cluster 2022-04-22T20:13:01.024614+0000 mgr.y (mgr.34107) 5719 : cluster [DBG] pgmap v6950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:02 smithi121 bash[36902]: audit 2022-04-22T20:13:02.032585+0000 mon.a (mon.0) 8876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:02 smithi115 bash[45495]: cluster 2022-04-22T20:13:01.024614+0000 mgr.y (mgr.34107) 5719 : cluster [DBG] pgmap v6950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:02 smithi115 bash[45495]: audit 2022-04-22T20:13:02.032585+0000 mon.a (mon.0) 8876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:02 smithi115 bash[46934]: cluster 2022-04-22T20:13:01.024614+0000 mgr.y (mgr.34107) 5719 : cluster [DBG] pgmap v6950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:02 smithi115 bash[46934]: audit 2022-04-22T20:13:02.032585+0000 mon.a (mon.0) 8876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:03 smithi121 bash[36902]: audit 2022-04-22T20:13:03.467580+0000 mon.a (mon.0) 8877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:03 smithi115 bash[46934]: audit 2022-04-22T20:13:03.467580+0000 mon.a (mon.0) 8877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:04.142Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:03 smithi115 bash[45495]: audit 2022-04-22T20:13:03.467580+0000 mon.a (mon.0) 8877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:04 smithi121 bash[36902]: cluster 2022-04-22T20:13:03.025750+0000 mgr.y (mgr.34107) 5720 : cluster [DBG] pgmap v6951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:04 smithi115 bash[46934]: cluster 2022-04-22T20:13:03.025750+0000 mgr.y (mgr.34107) 5720 : cluster [DBG] pgmap v6951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:04 smithi115 bash[45495]: cluster 2022-04-22T20:13:03.025750+0000 mgr.y (mgr.34107) 5720 : cluster [DBG] pgmap v6951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:06 smithi121 bash[36902]: cluster 2022-04-22T20:13:05.026913+0000 mgr.y (mgr.34107) 5721 : cluster [DBG] pgmap v6952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:06 smithi121 bash[36902]: audit 2022-04-22T20:13:06.791330+0000 mon.a (mon.0) 8878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:06 smithi115 bash[46934]: cluster 2022-04-22T20:13:05.026913+0000 mgr.y (mgr.34107) 5721 : cluster [DBG] pgmap v6952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:06 smithi115 bash[46934]: audit 2022-04-22T20:13:06.791330+0000 mon.a (mon.0) 8878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:06 smithi115 bash[45495]: cluster 2022-04-22T20:13:05.026913+0000 mgr.y (mgr.34107) 5721 : cluster [DBG] pgmap v6952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:06 smithi115 bash[45495]: audit 2022-04-22T20:13:06.791330+0000 mon.a (mon.0) 8878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:07 smithi121 bash[36902]: audit 2022-04-22T20:13:07.035609+0000 mon.a (mon.0) 8879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:07 smithi115 bash[46934]: audit 2022-04-22T20:13:07.035609+0000 mon.a (mon.0) 8879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:07 smithi115 bash[45495]: audit 2022-04-22T20:13:07.035609+0000 mon.a (mon.0) 8879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:08 smithi121 bash[36902]: cluster 2022-04-22T20:13:07.028144+0000 mgr.y (mgr.34107) 5722 : cluster [DBG] pgmap v6953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:08 smithi121 bash[36902]: audit 2022-04-22T20:13:08.471826+0000 mon.a (mon.0) 8880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:09.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:08 smithi115 bash[45495]: cluster 2022-04-22T20:13:07.028144+0000 mgr.y (mgr.34107) 5722 : cluster [DBG] pgmap v6953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:09.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:08 smithi115 bash[45495]: audit 2022-04-22T20:13:08.471826+0000 mon.a (mon.0) 8880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:08 smithi115 bash[46934]: cluster 2022-04-22T20:13:07.028144+0000 mgr.y (mgr.34107) 5722 : cluster [DBG] pgmap v6953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:08 smithi115 bash[46934]: audit 2022-04-22T20:13:08.471826+0000 mon.a (mon.0) 8880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:10.300 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:13:10.646 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:13:10.646 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 58s ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:13:10.646 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:13:10.646 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 58s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 58s ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 58s ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 58s ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:13:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 58s ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 58s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 58s ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 58s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 58s ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:13:10.648 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:13:10.649 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:13:10.649 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:11.048 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:13:11.049 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:13:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:10 smithi121 bash[36902]: cluster 2022-04-22T20:13:09.029167+0000 mgr.y (mgr.34107) 5723 : cluster [DBG] pgmap v6954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:10 smithi115 bash[45495]: cluster 2022-04-22T20:13:09.029167+0000 mgr.y (mgr.34107) 5723 : cluster [DBG] pgmap v6954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:10 smithi115 bash[46934]: cluster 2022-04-22T20:13:09.029167+0000 mgr.y (mgr.34107) 5723 : cluster [DBG] pgmap v6954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:11 smithi121 bash[36902]: audit 2022-04-22T20:13:10.276203+0000 mgr.y (mgr.34107) 5724 : audit [DBG] from='client.41346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:11 smithi121 bash[36902]: audit 2022-04-22T20:13:10.617106+0000 mgr.y (mgr.34107) 5725 : audit [DBG] from='client.51286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:11 smithi121 bash[36902]: audit 2022-04-22T20:13:11.039195+0000 mon.a (mon.0) 8881 : audit [DBG] from='client.? 172.21.15.115:0/3219616815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:11 smithi121 bash[36902]: audit 2022-04-22T20:13:11.795010+0000 mon.a (mon.0) 8882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:12.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[45495]: audit 2022-04-22T20:13:10.276203+0000 mgr.y (mgr.34107) 5724 : audit [DBG] from='client.41346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[45495]: audit 2022-04-22T20:13:10.617106+0000 mgr.y (mgr.34107) 5725 : audit [DBG] from='client.51286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[45495]: audit 2022-04-22T20:13:11.039195+0000 mon.a (mon.0) 8881 : audit [DBG] from='client.? 172.21.15.115:0/3219616815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[45495]: audit 2022-04-22T20:13:11.795010+0000 mon.a (mon.0) 8882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[46934]: audit 2022-04-22T20:13:10.276203+0000 mgr.y (mgr.34107) 5724 : audit [DBG] from='client.41346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[46934]: audit 2022-04-22T20:13:10.617106+0000 mgr.y (mgr.34107) 5725 : audit [DBG] from='client.51286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[46934]: audit 2022-04-22T20:13:11.039195+0000 mon.a (mon.0) 8881 : audit [DBG] from='client.? 172.21.15.115:0/3219616815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:12.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:11 smithi115 bash[46934]: audit 2022-04-22T20:13:11.795010+0000 mon.a (mon.0) 8882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:12.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:12.259Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:12 smithi121 bash[36902]: cluster 2022-04-22T20:13:11.030284+0000 mgr.y (mgr.34107) 5726 : cluster [DBG] pgmap v6955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:12 smithi121 bash[36902]: audit 2022-04-22T20:13:12.038636+0000 mon.a (mon.0) 8883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:12 smithi115 bash[45495]: cluster 2022-04-22T20:13:11.030284+0000 mgr.y (mgr.34107) 5726 : cluster [DBG] pgmap v6955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:12 smithi115 bash[45495]: audit 2022-04-22T20:13:12.038636+0000 mon.a (mon.0) 8883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:12 smithi115 bash[46934]: cluster 2022-04-22T20:13:11.030284+0000 mgr.y (mgr.34107) 5726 : cluster [DBG] pgmap v6955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:12 smithi115 bash[46934]: audit 2022-04-22T20:13:12.038636+0000 mon.a (mon.0) 8883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:13 smithi121 bash[36902]: audit 2022-04-22T20:13:13.474851+0000 mon.a (mon.0) 8884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:13 smithi115 bash[46934]: audit 2022-04-22T20:13:13.474851+0000 mon.a (mon.0) 8884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:14.142Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:14.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:13 smithi115 bash[45495]: audit 2022-04-22T20:13:13.474851+0000 mon.a (mon.0) 8884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:14 smithi121 bash[36902]: cluster 2022-04-22T20:13:13.031332+0000 mgr.y (mgr.34107) 5727 : cluster [DBG] pgmap v6956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:14 smithi115 bash[45495]: cluster 2022-04-22T20:13:13.031332+0000 mgr.y (mgr.34107) 5727 : cluster [DBG] pgmap v6956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:14 smithi115 bash[46934]: cluster 2022-04-22T20:13:13.031332+0000 mgr.y (mgr.34107) 5727 : cluster [DBG] pgmap v6956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:15 smithi121 bash[36902]: audit 2022-04-22T20:13:15.198115+0000 mon.a (mon.0) 8885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:13:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:15 smithi121 bash[36902]: audit 2022-04-22T20:13:15.198467+0000 mon.a (mon.0) 8886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:13:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:15 smithi115 bash[45495]: audit 2022-04-22T20:13:15.198115+0000 mon.a (mon.0) 8885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:13:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:15 smithi115 bash[45495]: audit 2022-04-22T20:13:15.198467+0000 mon.a (mon.0) 8886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:13:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:15 smithi115 bash[46934]: audit 2022-04-22T20:13:15.198115+0000 mon.a (mon.0) 8885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:13:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:15 smithi115 bash[46934]: audit 2022-04-22T20:13:15.198467+0000 mon.a (mon.0) 8886 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:13:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:16 smithi121 bash[36902]: cluster 2022-04-22T20:13:15.032437+0000 mgr.y (mgr.34107) 5728 : cluster [DBG] pgmap v6957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:16 smithi121 bash[36902]: audit 2022-04-22T20:13:16.798607+0000 mon.a (mon.0) 8887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:16 smithi115 bash[46934]: cluster 2022-04-22T20:13:15.032437+0000 mgr.y (mgr.34107) 5728 : cluster [DBG] pgmap v6957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:16 smithi115 bash[46934]: audit 2022-04-22T20:13:16.798607+0000 mon.a (mon.0) 8887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:16 smithi115 bash[45495]: cluster 2022-04-22T20:13:15.032437+0000 mgr.y (mgr.34107) 5728 : cluster [DBG] pgmap v6957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:16 smithi115 bash[45495]: audit 2022-04-22T20:13:16.798607+0000 mon.a (mon.0) 8887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:17 smithi121 bash[36902]: audit 2022-04-22T20:13:17.041602+0000 mon.a (mon.0) 8888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:17 smithi115 bash[45495]: audit 2022-04-22T20:13:17.041602+0000 mon.a (mon.0) 8888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:17 smithi115 bash[46934]: audit 2022-04-22T20:13:17.041602+0000 mon.a (mon.0) 8888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:19 smithi121 bash[36902]: cluster 2022-04-22T20:13:17.033645+0000 mgr.y (mgr.34107) 5729 : cluster [DBG] pgmap v6958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:19 smithi121 bash[36902]: audit 2022-04-22T20:13:18.477689+0000 mon.a (mon.0) 8889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:18 smithi115 bash[45495]: cluster 2022-04-22T20:13:17.033645+0000 mgr.y (mgr.34107) 5729 : cluster [DBG] pgmap v6958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:18 smithi115 bash[45495]: audit 2022-04-22T20:13:18.477689+0000 mon.a (mon.0) 8889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:18 smithi115 bash[46934]: cluster 2022-04-22T20:13:17.033645+0000 mgr.y (mgr.34107) 5729 : cluster [DBG] pgmap v6958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:18 smithi115 bash[46934]: audit 2022-04-22T20:13:18.477689+0000 mon.a (mon.0) 8889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:21 smithi121 bash[36902]: cluster 2022-04-22T20:13:19.034690+0000 mgr.y (mgr.34107) 5730 : cluster [DBG] pgmap v6959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:20 smithi115 bash[45495]: cluster 2022-04-22T20:13:19.034690+0000 mgr.y (mgr.34107) 5730 : cluster [DBG] pgmap v6959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:21 smithi115 bash[46934]: cluster 2022-04-22T20:13:19.034690+0000 mgr.y (mgr.34107) 5730 : cluster [DBG] pgmap v6959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:22.270 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[45495]: audit 2022-04-22T20:13:21.111903+0000 mon.a (mon.0) 8890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:13:22.270 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[45495]: audit 2022-04-22T20:13:21.114931+0000 mon.a (mon.0) 8891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:13:22.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[45495]: audit 2022-04-22T20:13:21.116387+0000 mon.a (mon.0) 8892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:13:22.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[45495]: audit 2022-04-22T20:13:21.481869+0000 mon.a (mon.0) 8893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:13:22.271 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[45495]: audit 2022-04-22T20:13:21.802433+0000 mon.a (mon.0) 8894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:22.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[46934]: audit 2022-04-22T20:13:21.111903+0000 mon.a (mon.0) 8890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:13:22.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[46934]: audit 2022-04-22T20:13:21.114931+0000 mon.a (mon.0) 8891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:13:22.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[46934]: audit 2022-04-22T20:13:21.116387+0000 mon.a (mon.0) 8892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:13:22.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[46934]: audit 2022-04-22T20:13:21.481869+0000 mon.a (mon.0) 8893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:13:22.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[46934]: audit 2022-04-22T20:13:21.802433+0000 mon.a (mon.0) 8894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:22 smithi121 bash[36902]: audit 2022-04-22T20:13:21.111903+0000 mon.a (mon.0) 8890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:22 smithi121 bash[36902]: audit 2022-04-22T20:13:21.114931+0000 mon.a (mon.0) 8891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:22 smithi121 bash[36902]: audit 2022-04-22T20:13:21.116387+0000 mon.a (mon.0) 8892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:22 smithi121 bash[36902]: audit 2022-04-22T20:13:21.481869+0000 mon.a (mon.0) 8893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:22 smithi121 bash[36902]: audit 2022-04-22T20:13:21.802433+0000 mon.a (mon.0) 8894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:22.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:22.261Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:23 smithi115 bash[45495]: cluster 2022-04-22T20:13:21.035778+0000 mgr.y (mgr.34107) 5731 : cluster [DBG] pgmap v6960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:23 smithi115 bash[45495]: audit 2022-04-22T20:13:22.044486+0000 mon.a (mon.0) 8895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:23 smithi115 bash[46934]: cluster 2022-04-22T20:13:21.035778+0000 mgr.y (mgr.34107) 5731 : cluster [DBG] pgmap v6960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:23 smithi115 bash[46934]: audit 2022-04-22T20:13:22.044486+0000 mon.a (mon.0) 8895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:23 smithi121 bash[36902]: cluster 2022-04-22T20:13:21.035778+0000 mgr.y (mgr.34107) 5731 : cluster [DBG] pgmap v6960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:23 smithi121 bash[36902]: audit 2022-04-22T20:13:22.044486+0000 mon.a (mon.0) 8895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:24.142Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:24 smithi115 bash[45495]: audit 2022-04-22T20:13:23.480665+0000 mon.a (mon.0) 8896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:24 smithi115 bash[46934]: audit 2022-04-22T20:13:23.480665+0000 mon.a (mon.0) 8896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:24 smithi121 bash[36902]: audit 2022-04-22T20:13:23.480665+0000 mon.a (mon.0) 8896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:25 smithi115 bash[46934]: cluster 2022-04-22T20:13:23.036720+0000 mgr.y (mgr.34107) 5732 : cluster [DBG] pgmap v6961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:25.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:25 smithi115 bash[45495]: cluster 2022-04-22T20:13:23.036720+0000 mgr.y (mgr.34107) 5732 : cluster [DBG] pgmap v6961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:25 smithi121 bash[36902]: cluster 2022-04-22T20:13:23.036720+0000 mgr.y (mgr.34107) 5732 : cluster [DBG] pgmap v6961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:26.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:26 smithi115 bash[45495]: cluster 2022-04-22T20:13:25.037687+0000 mgr.y (mgr.34107) 5733 : cluster [DBG] pgmap v6962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:26.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:26 smithi115 bash[46934]: cluster 2022-04-22T20:13:25.037687+0000 mgr.y (mgr.34107) 5733 : cluster [DBG] pgmap v6962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:26 smithi121 bash[36902]: cluster 2022-04-22T20:13:25.037687+0000 mgr.y (mgr.34107) 5733 : cluster [DBG] pgmap v6962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:27 smithi115 bash[45495]: audit 2022-04-22T20:13:26.806079+0000 mon.a (mon.0) 8897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:27 smithi115 bash[46934]: audit 2022-04-22T20:13:26.806079+0000 mon.a (mon.0) 8897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:27 smithi121 bash[36902]: audit 2022-04-22T20:13:26.806079+0000 mon.a (mon.0) 8897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:28 smithi121 bash[36902]: cluster 2022-04-22T20:13:27.038783+0000 mgr.y (mgr.34107) 5734 : cluster [DBG] pgmap v6963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:28 smithi121 bash[36902]: audit 2022-04-22T20:13:27.047014+0000 mon.a (mon.0) 8898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:28 smithi115 bash[45495]: cluster 2022-04-22T20:13:27.038783+0000 mgr.y (mgr.34107) 5734 : cluster [DBG] pgmap v6963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:28 smithi115 bash[45495]: audit 2022-04-22T20:13:27.047014+0000 mon.a (mon.0) 8898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:28 smithi115 bash[46934]: cluster 2022-04-22T20:13:27.038783+0000 mgr.y (mgr.34107) 5734 : cluster [DBG] pgmap v6963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:28 smithi115 bash[46934]: audit 2022-04-22T20:13:27.047014+0000 mon.a (mon.0) 8898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:29 smithi121 bash[36902]: audit 2022-04-22T20:13:28.483974+0000 mon.a (mon.0) 8899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:29 smithi115 bash[45495]: audit 2022-04-22T20:13:28.483974+0000 mon.a (mon.0) 8899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:29 smithi115 bash[46934]: audit 2022-04-22T20:13:28.483974+0000 mon.a (mon.0) 8899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:31.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:30 smithi115 bash[45495]: cluster 2022-04-22T20:13:29.039879+0000 mgr.y (mgr.34107) 5735 : cluster [DBG] pgmap v6964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:31.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:30 smithi115 bash[46934]: cluster 2022-04-22T20:13:29.039879+0000 mgr.y (mgr.34107) 5735 : cluster [DBG] pgmap v6964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:30 smithi121 bash[36902]: cluster 2022-04-22T20:13:29.039879+0000 mgr.y (mgr.34107) 5735 : cluster [DBG] pgmap v6964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:32.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:32.262Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:33.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[45495]: cluster 2022-04-22T20:13:31.040862+0000 mgr.y (mgr.34107) 5736 : cluster [DBG] pgmap v6965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[45495]: audit 2022-04-22T20:13:31.809570+0000 mon.a (mon.0) 8900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:33.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[45495]: audit 2022-04-22T20:13:32.049625+0000 mon.a (mon.0) 8901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[46934]: cluster 2022-04-22T20:13:31.040862+0000 mgr.y (mgr.34107) 5736 : cluster [DBG] pgmap v6965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:33.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[46934]: audit 2022-04-22T20:13:31.809570+0000 mon.a (mon.0) 8900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:33.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:32 smithi115 bash[46934]: audit 2022-04-22T20:13:32.049625+0000 mon.a (mon.0) 8901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:32 smithi121 bash[36902]: cluster 2022-04-22T20:13:31.040862+0000 mgr.y (mgr.34107) 5736 : cluster [DBG] pgmap v6965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:32 smithi121 bash[36902]: audit 2022-04-22T20:13:31.809570+0000 mon.a (mon.0) 8900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:32 smithi121 bash[36902]: audit 2022-04-22T20:13:32.049625+0000 mon.a (mon.0) 8901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:34.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:33 smithi115 bash[46934]: audit 2022-04-22T20:13:33.486903+0000 mon.a (mon.0) 8902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:34.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:33 smithi115 bash[45495]: audit 2022-04-22T20:13:33.486903+0000 mon.a (mon.0) 8902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:33 smithi121 bash[36902]: audit 2022-04-22T20:13:33.486903+0000 mon.a (mon.0) 8902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:34.142Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:35.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:34 smithi115 bash[45495]: cluster 2022-04-22T20:13:33.041841+0000 mgr.y (mgr.34107) 5737 : cluster [DBG] pgmap v6966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:35.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:34 smithi115 bash[46934]: cluster 2022-04-22T20:13:33.041841+0000 mgr.y (mgr.34107) 5737 : cluster [DBG] pgmap v6966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:34 smithi121 bash[36902]: cluster 2022-04-22T20:13:33.041841+0000 mgr.y (mgr.34107) 5737 : cluster [DBG] pgmap v6966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:37.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:36 smithi115 bash[45495]: cluster 2022-04-22T20:13:35.043051+0000 mgr.y (mgr.34107) 5738 : cluster [DBG] pgmap v6967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:37.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:36 smithi115 bash[46934]: cluster 2022-04-22T20:13:35.043051+0000 mgr.y (mgr.34107) 5738 : cluster [DBG] pgmap v6967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:36 smithi121 bash[36902]: cluster 2022-04-22T20:13:35.043051+0000 mgr.y (mgr.34107) 5738 : cluster [DBG] pgmap v6967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:37 smithi121 bash[36902]: audit 2022-04-22T20:13:36.813430+0000 mon.a (mon.0) 8903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:37 smithi121 bash[36902]: audit 2022-04-22T20:13:37.052679+0000 mon.a (mon.0) 8904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:37 smithi115 bash[45495]: audit 2022-04-22T20:13:36.813430+0000 mon.a (mon.0) 8903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:37 smithi115 bash[45495]: audit 2022-04-22T20:13:37.052679+0000 mon.a (mon.0) 8904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:37 smithi115 bash[46934]: audit 2022-04-22T20:13:36.813430+0000 mon.a (mon.0) 8903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:37 smithi115 bash[46934]: audit 2022-04-22T20:13:37.052679+0000 mon.a (mon.0) 8904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:38 smithi121 bash[36902]: cluster 2022-04-22T20:13:37.044250+0000 mgr.y (mgr.34107) 5739 : cluster [DBG] pgmap v6968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:39.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:38 smithi121 bash[36902]: audit 2022-04-22T20:13:38.490195+0000 mon.a (mon.0) 8905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:38 smithi115 bash[45495]: cluster 2022-04-22T20:13:37.044250+0000 mgr.y (mgr.34107) 5739 : cluster [DBG] pgmap v6968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:38 smithi115 bash[45495]: audit 2022-04-22T20:13:38.490195+0000 mon.a (mon.0) 8905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:38 smithi115 bash[46934]: cluster 2022-04-22T20:13:37.044250+0000 mgr.y (mgr.34107) 5739 : cluster [DBG] pgmap v6968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:38 smithi115 bash[46934]: audit 2022-04-22T20:13:38.490195+0000 mon.a (mon.0) 8905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:40 smithi121 bash[36902]: cluster 2022-04-22T20:13:39.045318+0000 mgr.y (mgr.34107) 5740 : cluster [DBG] pgmap v6969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:40 smithi115 bash[45495]: cluster 2022-04-22T20:13:39.045318+0000 mgr.y (mgr.34107) 5740 : cluster [DBG] pgmap v6969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:40 smithi115 bash[46934]: cluster 2022-04-22T20:13:39.045318+0000 mgr.y (mgr.34107) 5740 : cluster [DBG] pgmap v6969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:41.424 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:13:41.777 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 89s ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 2m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 89s ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 89s ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 89s ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:13:41.778 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 89s ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 89s ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 89s ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 89s ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 89s ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 89s ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:13:41.779 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:13:41.780 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:13:41.780 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:13:41.780 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:13:41.780 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:13:42.178 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:13:42.178 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:13:42.179 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:13:42.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:42.179 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:13:42.179 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:13:42.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:42.180 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:13:42.180 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:13:42.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:42.181 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:13:42.182 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:13:42.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:13:42.182 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:13:42.182 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:13:42.182 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:13:42.183 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:13:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:42.263Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: cluster 2022-04-22T20:13:41.046403+0000 mgr.y (mgr.34107) 5741 : cluster [DBG] pgmap v6970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:43.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: audit 2022-04-22T20:13:41.401144+0000 mgr.y (mgr.34107) 5742 : audit [DBG] from='client.51298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: audit 2022-04-22T20:13:41.750274+0000 mgr.y (mgr.34107) 5743 : audit [DBG] from='client.51304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: audit 2022-04-22T20:13:41.816716+0000 mon.a (mon.0) 8906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:43.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: audit 2022-04-22T20:13:42.055702+0000 mon.a (mon.0) 8907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:43.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:42 smithi121 bash[36902]: audit 2022-04-22T20:13:42.169261+0000 mon.c (mon.1) 229 : audit [DBG] from='client.? 172.21.15.115:0/1340362162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: cluster 2022-04-22T20:13:41.046403+0000 mgr.y (mgr.34107) 5741 : cluster [DBG] pgmap v6970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: audit 2022-04-22T20:13:41.401144+0000 mgr.y (mgr.34107) 5742 : audit [DBG] from='client.51298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: audit 2022-04-22T20:13:41.750274+0000 mgr.y (mgr.34107) 5743 : audit [DBG] from='client.51304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: audit 2022-04-22T20:13:41.816716+0000 mon.a (mon.0) 8906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: audit 2022-04-22T20:13:42.055702+0000 mon.a (mon.0) 8907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[45495]: audit 2022-04-22T20:13:42.169261+0000 mon.c (mon.1) 229 : audit [DBG] from='client.? 172.21.15.115:0/1340362162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: cluster 2022-04-22T20:13:41.046403+0000 mgr.y (mgr.34107) 5741 : cluster [DBG] pgmap v6970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:43.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: audit 2022-04-22T20:13:41.401144+0000 mgr.y (mgr.34107) 5742 : audit [DBG] from='client.51298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: audit 2022-04-22T20:13:41.750274+0000 mgr.y (mgr.34107) 5743 : audit [DBG] from='client.51304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:13:43.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: audit 2022-04-22T20:13:41.816716+0000 mon.a (mon.0) 8906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:43.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: audit 2022-04-22T20:13:42.055702+0000 mon.a (mon.0) 8907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:43.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:42 smithi115 bash[46934]: audit 2022-04-22T20:13:42.169261+0000 mon.c (mon.1) 229 : audit [DBG] from='client.? 172.21.15.115:0/1340362162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:13:44.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:43 smithi115 bash[45495]: audit 2022-04-22T20:13:43.493090+0000 mon.a (mon.0) 8908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:44.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:43 smithi115 bash[46934]: audit 2022-04-22T20:13:43.493090+0000 mon.a (mon.0) 8908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:43 smithi121 bash[36902]: audit 2022-04-22T20:13:43.493090+0000 mon.a (mon.0) 8908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:44.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:44.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:44 smithi121 bash[36902]: cluster 2022-04-22T20:13:43.047515+0000 mgr.y (mgr.34107) 5744 : cluster [DBG] pgmap v6971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:44 smithi115 bash[45495]: cluster 2022-04-22T20:13:43.047515+0000 mgr.y (mgr.34107) 5744 : cluster [DBG] pgmap v6971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:44 smithi115 bash[46934]: cluster 2022-04-22T20:13:43.047515+0000 mgr.y (mgr.34107) 5744 : cluster [DBG] pgmap v6971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:46 smithi121 bash[36902]: cluster 2022-04-22T20:13:45.048646+0000 mgr.y (mgr.34107) 5745 : cluster [DBG] pgmap v6972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:46 smithi121 bash[36902]: audit 2022-04-22T20:13:46.820323+0000 mon.a (mon.0) 8909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:46 smithi115 bash[45495]: cluster 2022-04-22T20:13:45.048646+0000 mgr.y (mgr.34107) 5745 : cluster [DBG] pgmap v6972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:46 smithi115 bash[45495]: audit 2022-04-22T20:13:46.820323+0000 mon.a (mon.0) 8909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:46 smithi115 bash[46934]: cluster 2022-04-22T20:13:45.048646+0000 mgr.y (mgr.34107) 5745 : cluster [DBG] pgmap v6972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:46 smithi115 bash[46934]: audit 2022-04-22T20:13:46.820323+0000 mon.a (mon.0) 8909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:47 smithi121 bash[36902]: audit 2022-04-22T20:13:47.058542+0000 mon.a (mon.0) 8910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:47 smithi115 bash[46934]: audit 2022-04-22T20:13:47.058542+0000 mon.a (mon.0) 8910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:47 smithi115 bash[45495]: audit 2022-04-22T20:13:47.058542+0000 mon.a (mon.0) 8910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:48 smithi121 bash[36902]: cluster 2022-04-22T20:13:47.049859+0000 mgr.y (mgr.34107) 5746 : cluster [DBG] pgmap v6973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:48 smithi121 bash[36902]: audit 2022-04-22T20:13:48.496055+0000 mon.a (mon.0) 8911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:48 smithi115 bash[45495]: cluster 2022-04-22T20:13:47.049859+0000 mgr.y (mgr.34107) 5746 : cluster [DBG] pgmap v6973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:48 smithi115 bash[45495]: audit 2022-04-22T20:13:48.496055+0000 mon.a (mon.0) 8911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:48 smithi115 bash[46934]: cluster 2022-04-22T20:13:47.049859+0000 mgr.y (mgr.34107) 5746 : cluster [DBG] pgmap v6973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:48 smithi115 bash[46934]: audit 2022-04-22T20:13:48.496055+0000 mon.a (mon.0) 8911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:50 smithi121 bash[36902]: cluster 2022-04-22T20:13:49.050932+0000 mgr.y (mgr.34107) 5747 : cluster [DBG] pgmap v6974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:50 smithi115 bash[45495]: cluster 2022-04-22T20:13:49.050932+0000 mgr.y (mgr.34107) 5747 : cluster [DBG] pgmap v6974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:50 smithi115 bash[46934]: cluster 2022-04-22T20:13:49.050932+0000 mgr.y (mgr.34107) 5747 : cluster [DBG] pgmap v6974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:51 smithi121 bash[36902]: audit 2022-04-22T20:13:51.824104+0000 mon.a (mon.0) 8912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:52.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:51 smithi115 bash[46934]: audit 2022-04-22T20:13:51.824104+0000 mon.a (mon.0) 8912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:52.270 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:51 smithi115 bash[45495]: audit 2022-04-22T20:13:51.824104+0000 mon.a (mon.0) 8912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:52.262Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:52 smithi121 bash[36902]: cluster 2022-04-22T20:13:51.052049+0000 mgr.y (mgr.34107) 5748 : cluster [DBG] pgmap v6975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:52 smithi121 bash[36902]: audit 2022-04-22T20:13:52.061398+0000 mon.a (mon.0) 8913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:52 smithi115 bash[45495]: cluster 2022-04-22T20:13:51.052049+0000 mgr.y (mgr.34107) 5748 : cluster [DBG] pgmap v6975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:52 smithi115 bash[45495]: audit 2022-04-22T20:13:52.061398+0000 mon.a (mon.0) 8913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:52 smithi115 bash[46934]: cluster 2022-04-22T20:13:51.052049+0000 mgr.y (mgr.34107) 5748 : cluster [DBG] pgmap v6975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:52 smithi115 bash[46934]: audit 2022-04-22T20:13:52.061398+0000 mon.a (mon.0) 8913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:54.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:53 smithi115 bash[45495]: audit 2022-04-22T20:13:53.499393+0000 mon.a (mon.0) 8914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:53 smithi115 bash[46934]: audit 2022-04-22T20:13:53.499393+0000 mon.a (mon.0) 8914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:53 smithi121 bash[36902]: audit 2022-04-22T20:13:53.499393+0000 mon.a (mon.0) 8914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:13:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:13:54.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:13:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:54 smithi121 bash[36902]: cluster 2022-04-22T20:13:53.053051+0000 mgr.y (mgr.34107) 5749 : cluster [DBG] pgmap v6976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:54 smithi115 bash[45495]: cluster 2022-04-22T20:13:53.053051+0000 mgr.y (mgr.34107) 5749 : cluster [DBG] pgmap v6976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:54 smithi115 bash[46934]: cluster 2022-04-22T20:13:53.053051+0000 mgr.y (mgr.34107) 5749 : cluster [DBG] pgmap v6976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:56 smithi121 bash[36902]: cluster 2022-04-22T20:13:55.054187+0000 mgr.y (mgr.34107) 5750 : cluster [DBG] pgmap v6977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:56 smithi121 bash[36902]: audit 2022-04-22T20:13:56.827767+0000 mon.a (mon.0) 8915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:56 smithi115 bash[45495]: cluster 2022-04-22T20:13:55.054187+0000 mgr.y (mgr.34107) 5750 : cluster [DBG] pgmap v6977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:57.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:56 smithi115 bash[45495]: audit 2022-04-22T20:13:56.827767+0000 mon.a (mon.0) 8915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:57.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:56 smithi115 bash[46934]: cluster 2022-04-22T20:13:55.054187+0000 mgr.y (mgr.34107) 5750 : cluster [DBG] pgmap v6977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:57.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:56 smithi115 bash[46934]: audit 2022-04-22T20:13:56.827767+0000 mon.a (mon.0) 8915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:13:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:57 smithi121 bash[36902]: audit 2022-04-22T20:13:57.064550+0000 mon.a (mon.0) 8916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:57 smithi115 bash[45495]: audit 2022-04-22T20:13:57.064550+0000 mon.a (mon.0) 8916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:57 smithi115 bash[46934]: audit 2022-04-22T20:13:57.064550+0000 mon.a (mon.0) 8916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:13:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:58 smithi121 bash[36902]: cluster 2022-04-22T20:13:57.055375+0000 mgr.y (mgr.34107) 5751 : cluster [DBG] pgmap v6978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:13:58 smithi121 bash[36902]: audit 2022-04-22T20:13:58.502643+0000 mon.a (mon.0) 8917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:58 smithi115 bash[46934]: cluster 2022-04-22T20:13:57.055375+0000 mgr.y (mgr.34107) 5751 : cluster [DBG] pgmap v6978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:13:58 smithi115 bash[46934]: audit 2022-04-22T20:13:58.502643+0000 mon.a (mon.0) 8917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:13:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:58 smithi115 bash[45495]: cluster 2022-04-22T20:13:57.055375+0000 mgr.y (mgr.34107) 5751 : cluster [DBG] pgmap v6978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:13:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:13:58 smithi115 bash[45495]: audit 2022-04-22T20:13:58.502643+0000 mon.a (mon.0) 8917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:00 smithi121 bash[36902]: cluster 2022-04-22T20:13:59.056497+0000 mgr.y (mgr.34107) 5752 : cluster [DBG] pgmap v6979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:00 smithi115 bash[45495]: cluster 2022-04-22T20:13:59.056497+0000 mgr.y (mgr.34107) 5752 : cluster [DBG] pgmap v6979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:00 smithi115 bash[46934]: cluster 2022-04-22T20:13:59.056497+0000 mgr.y (mgr.34107) 5752 : cluster [DBG] pgmap v6979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:01 smithi121 bash[36902]: audit 2022-04-22T20:14:01.831221+0000 mon.a (mon.0) 8918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:02.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:01 smithi115 bash[46934]: audit 2022-04-22T20:14:01.831221+0000 mon.a (mon.0) 8918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:02.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:01 smithi115 bash[45495]: audit 2022-04-22T20:14:01.831221+0000 mon.a (mon.0) 8918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:02.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:02 smithi121 bash[36902]: cluster 2022-04-22T20:14:01.057509+0000 mgr.y (mgr.34107) 5753 : cluster [DBG] pgmap v6980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:02 smithi121 bash[36902]: audit 2022-04-22T20:14:02.067451+0000 mon.a (mon.0) 8919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:02 smithi115 bash[45495]: cluster 2022-04-22T20:14:01.057509+0000 mgr.y (mgr.34107) 5753 : cluster [DBG] pgmap v6980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:02 smithi115 bash[45495]: audit 2022-04-22T20:14:02.067451+0000 mon.a (mon.0) 8919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:03.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:02 smithi115 bash[46934]: cluster 2022-04-22T20:14:01.057509+0000 mgr.y (mgr.34107) 5753 : cluster [DBG] pgmap v6980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:03.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:02 smithi115 bash[46934]: audit 2022-04-22T20:14:02.067451+0000 mon.a (mon.0) 8919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:03 smithi121 bash[36902]: audit 2022-04-22T20:14:03.505879+0000 mon.a (mon.0) 8920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:03 smithi115 bash[45495]: audit 2022-04-22T20:14:03.505879+0000 mon.a (mon.0) 8920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:04.145Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:03 smithi115 bash[46934]: audit 2022-04-22T20:14:03.505879+0000 mon.a (mon.0) 8920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:04 smithi121 bash[36902]: cluster 2022-04-22T20:14:03.058566+0000 mgr.y (mgr.34107) 5754 : cluster [DBG] pgmap v6981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:04 smithi115 bash[45495]: cluster 2022-04-22T20:14:03.058566+0000 mgr.y (mgr.34107) 5754 : cluster [DBG] pgmap v6981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:04 smithi115 bash[46934]: cluster 2022-04-22T20:14:03.058566+0000 mgr.y (mgr.34107) 5754 : cluster [DBG] pgmap v6981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:06 smithi121 bash[36902]: cluster 2022-04-22T20:14:05.059576+0000 mgr.y (mgr.34107) 5755 : cluster [DBG] pgmap v6982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:07.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:06 smithi121 bash[36902]: audit 2022-04-22T20:14:06.835006+0000 mon.a (mon.0) 8921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:07.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:06 smithi115 bash[45495]: cluster 2022-04-22T20:14:05.059576+0000 mgr.y (mgr.34107) 5755 : cluster [DBG] pgmap v6982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:07.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:06 smithi115 bash[45495]: audit 2022-04-22T20:14:06.835006+0000 mon.a (mon.0) 8921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:06 smithi115 bash[46934]: cluster 2022-04-22T20:14:05.059576+0000 mgr.y (mgr.34107) 5755 : cluster [DBG] pgmap v6982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:07.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:06 smithi115 bash[46934]: audit 2022-04-22T20:14:06.835006+0000 mon.a (mon.0) 8921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:07 smithi121 bash[36902]: audit 2022-04-22T20:14:07.070403+0000 mon.a (mon.0) 8922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:07 smithi115 bash[45495]: audit 2022-04-22T20:14:07.070403+0000 mon.a (mon.0) 8922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:07 smithi115 bash[46934]: audit 2022-04-22T20:14:07.070403+0000 mon.a (mon.0) 8922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:08 smithi121 bash[36902]: cluster 2022-04-22T20:14:07.060767+0000 mgr.y (mgr.34107) 5756 : cluster [DBG] pgmap v6983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:09.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:08 smithi121 bash[36902]: audit 2022-04-22T20:14:08.508933+0000 mon.a (mon.0) 8923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:08 smithi115 bash[45495]: cluster 2022-04-22T20:14:07.060767+0000 mgr.y (mgr.34107) 5756 : cluster [DBG] pgmap v6983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:09.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:08 smithi115 bash[45495]: audit 2022-04-22T20:14:08.508933+0000 mon.a (mon.0) 8923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:08 smithi115 bash[46934]: cluster 2022-04-22T20:14:07.060767+0000 mgr.y (mgr.34107) 5756 : cluster [DBG] pgmap v6983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:08 smithi115 bash[46934]: audit 2022-04-22T20:14:08.508933+0000 mon.a (mon.0) 8923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:10 smithi121 bash[36902]: cluster 2022-04-22T20:14:09.061812+0000 mgr.y (mgr.34107) 5757 : cluster [DBG] pgmap v6984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:10 smithi115 bash[45495]: cluster 2022-04-22T20:14:09.061812+0000 mgr.y (mgr.34107) 5757 : cluster [DBG] pgmap v6984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:10 smithi115 bash[46934]: cluster 2022-04-22T20:14:09.061812+0000 mgr.y (mgr.34107) 5757 : cluster [DBG] pgmap v6984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:11 smithi121 bash[36902]: audit 2022-04-22T20:14:11.838834+0000 mon.a (mon.0) 8924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:12.272 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:11 smithi115 bash[45495]: audit 2022-04-22T20:14:11.838834+0000 mon.a (mon.0) 8924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:12.273 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:11 smithi115 bash[46934]: audit 2022-04-22T20:14:11.838834+0000 mon.a (mon.0) 8924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:12.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:12.552 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:14:12.898 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:14:12.899 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:14:12.900 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:14:12.900 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:14:12.900 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:14:12.900 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:14:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:12 smithi121 bash[36902]: cluster 2022-04-22T20:14:11.062929+0000 mgr.y (mgr.34107) 5758 : cluster [DBG] pgmap v6985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:12 smithi121 bash[36902]: audit 2022-04-22T20:14:12.073372+0000 mon.a (mon.0) 8925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:12 smithi115 bash[45495]: cluster 2022-04-22T20:14:11.062929+0000 mgr.y (mgr.34107) 5758 : cluster [DBG] pgmap v6985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:12 smithi115 bash[45495]: audit 2022-04-22T20:14:12.073372+0000 mon.a (mon.0) 8925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:12 smithi115 bash[46934]: cluster 2022-04-22T20:14:11.062929+0000 mgr.y (mgr.34107) 5758 : cluster [DBG] pgmap v6985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:12 smithi115 bash[46934]: audit 2022-04-22T20:14:12.073372+0000 mon.a (mon.0) 8925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:13.304 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:14:13.304 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:14:13.305 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:14:13.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:13.305 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:14:13.305 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:14:13.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:13.306 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:14:13.306 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:14:13.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:13.307 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:14:13.308 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:14:13.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:13.308 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:14:13.309 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:14:13.309 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:14:13.309 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:14:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:13 smithi121 bash[36902]: audit 2022-04-22T20:14:12.529041+0000 mgr.y (mgr.34107) 5759 : audit [DBG] from='client.51316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:13 smithi121 bash[36902]: audit 2022-04-22T20:14:12.870007+0000 mgr.y (mgr.34107) 5760 : audit [DBG] from='client.41388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:13 smithi121 bash[36902]: audit 2022-04-22T20:14:13.295589+0000 mon.c (mon.1) 230 : audit [DBG] from='client.? 172.21.15.115:0/2557151512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:13 smithi121 bash[36902]: audit 2022-04-22T20:14:13.511595+0000 mon.a (mon.0) 8926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:14.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:14.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[45495]: audit 2022-04-22T20:14:12.529041+0000 mgr.y (mgr.34107) 5759 : audit [DBG] from='client.51316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[45495]: audit 2022-04-22T20:14:12.870007+0000 mgr.y (mgr.34107) 5760 : audit [DBG] from='client.41388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[45495]: audit 2022-04-22T20:14:13.295589+0000 mon.c (mon.1) 230 : audit [DBG] from='client.? 172.21.15.115:0/2557151512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:14.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[45495]: audit 2022-04-22T20:14:13.511595+0000 mon.a (mon.0) 8926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:14.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[46934]: audit 2022-04-22T20:14:12.529041+0000 mgr.y (mgr.34107) 5759 : audit [DBG] from='client.51316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[46934]: audit 2022-04-22T20:14:12.870007+0000 mgr.y (mgr.34107) 5760 : audit [DBG] from='client.41388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:14.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[46934]: audit 2022-04-22T20:14:13.295589+0000 mon.c (mon.1) 230 : audit [DBG] from='client.? 172.21.15.115:0/2557151512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:14.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:13 smithi115 bash[46934]: audit 2022-04-22T20:14:13.511595+0000 mon.a (mon.0) 8926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:14 smithi121 bash[36902]: cluster 2022-04-22T20:14:13.063981+0000 mgr.y (mgr.34107) 5761 : cluster [DBG] pgmap v6986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:15.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:14 smithi115 bash[45495]: cluster 2022-04-22T20:14:13.063981+0000 mgr.y (mgr.34107) 5761 : cluster [DBG] pgmap v6986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:14 smithi115 bash[46934]: cluster 2022-04-22T20:14:13.063981+0000 mgr.y (mgr.34107) 5761 : cluster [DBG] pgmap v6986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:15 smithi121 bash[36902]: audit 2022-04-22T20:14:15.205634+0000 mon.a (mon.0) 8927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:14:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:15 smithi121 bash[36902]: audit 2022-04-22T20:14:15.212793+0000 mon.a (mon.0) 8928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:14:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:15 smithi115 bash[46934]: audit 2022-04-22T20:14:15.205634+0000 mon.a (mon.0) 8927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:14:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:15 smithi115 bash[46934]: audit 2022-04-22T20:14:15.212793+0000 mon.a (mon.0) 8928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:14:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:15 smithi115 bash[45495]: audit 2022-04-22T20:14:15.205634+0000 mon.a (mon.0) 8927 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:14:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:15 smithi115 bash[45495]: audit 2022-04-22T20:14:15.212793+0000 mon.a (mon.0) 8928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:14:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:16 smithi121 bash[36902]: cluster 2022-04-22T20:14:15.065007+0000 mgr.y (mgr.34107) 5762 : cluster [DBG] pgmap v6987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:16 smithi121 bash[36902]: audit 2022-04-22T20:14:16.842409+0000 mon.a (mon.0) 8929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:16 smithi115 bash[46934]: cluster 2022-04-22T20:14:15.065007+0000 mgr.y (mgr.34107) 5762 : cluster [DBG] pgmap v6987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:16 smithi115 bash[46934]: audit 2022-04-22T20:14:16.842409+0000 mon.a (mon.0) 8929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:16 smithi115 bash[45495]: cluster 2022-04-22T20:14:15.065007+0000 mgr.y (mgr.34107) 5762 : cluster [DBG] pgmap v6987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:16 smithi115 bash[45495]: audit 2022-04-22T20:14:16.842409+0000 mon.a (mon.0) 8929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:17 smithi121 bash[36902]: audit 2022-04-22T20:14:17.076140+0000 mon.a (mon.0) 8930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:17 smithi115 bash[45495]: audit 2022-04-22T20:14:17.076140+0000 mon.a (mon.0) 8930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:17 smithi115 bash[46934]: audit 2022-04-22T20:14:17.076140+0000 mon.a (mon.0) 8930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:18 smithi121 bash[36902]: cluster 2022-04-22T20:14:17.066140+0000 mgr.y (mgr.34107) 5763 : cluster [DBG] pgmap v6988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:18 smithi121 bash[36902]: audit 2022-04-22T20:14:18.514476+0000 mon.a (mon.0) 8931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:18 smithi115 bash[45495]: cluster 2022-04-22T20:14:17.066140+0000 mgr.y (mgr.34107) 5763 : cluster [DBG] pgmap v6988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:18 smithi115 bash[45495]: audit 2022-04-22T20:14:18.514476+0000 mon.a (mon.0) 8931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:18 smithi115 bash[46934]: cluster 2022-04-22T20:14:17.066140+0000 mgr.y (mgr.34107) 5763 : cluster [DBG] pgmap v6988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:18 smithi115 bash[46934]: audit 2022-04-22T20:14:18.514476+0000 mon.a (mon.0) 8931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:20 smithi121 bash[36902]: cluster 2022-04-22T20:14:19.067249+0000 mgr.y (mgr.34107) 5764 : cluster [DBG] pgmap v6989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:20 smithi115 bash[45495]: cluster 2022-04-22T20:14:19.067249+0000 mgr.y (mgr.34107) 5764 : cluster [DBG] pgmap v6989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:20 smithi115 bash[46934]: cluster 2022-04-22T20:14:19.067249+0000 mgr.y (mgr.34107) 5764 : cluster [DBG] pgmap v6989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:22 smithi121 bash[36902]: audit 2022-04-22T20:14:21.491510+0000 mon.a (mon.0) 8932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:14:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:22 smithi121 bash[36902]: audit 2022-04-22T20:14:21.494563+0000 mon.a (mon.0) 8933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:14:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:22 smithi121 bash[36902]: audit 2022-04-22T20:14:21.497342+0000 mon.a (mon.0) 8934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:14:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:22 smithi121 bash[36902]: audit 2022-04-22T20:14:21.846038+0000 mon.a (mon.0) 8935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:22 smithi121 bash[36902]: audit 2022-04-22T20:14:21.868595+0000 mon.a (mon.0) 8936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:14:22.273 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[46934]: audit 2022-04-22T20:14:21.491510+0000 mon.a (mon.0) 8932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[46934]: audit 2022-04-22T20:14:21.494563+0000 mon.a (mon.0) 8933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[46934]: audit 2022-04-22T20:14:21.497342+0000 mon.a (mon.0) 8934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[46934]: audit 2022-04-22T20:14:21.846038+0000 mon.a (mon.0) 8935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[46934]: audit 2022-04-22T20:14:21.868595+0000 mon.a (mon.0) 8936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[45495]: audit 2022-04-22T20:14:21.491510+0000 mon.a (mon.0) 8932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[45495]: audit 2022-04-22T20:14:21.494563+0000 mon.a (mon.0) 8933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:14:22.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[45495]: audit 2022-04-22T20:14:21.497342+0000 mon.a (mon.0) 8934 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:14:22.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[45495]: audit 2022-04-22T20:14:21.846038+0000 mon.a (mon.0) 8935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:22.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:21 smithi115 bash[45495]: audit 2022-04-22T20:14:21.868595+0000 mon.a (mon.0) 8936 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:14:22.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:22.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:23.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:23 smithi121 bash[36902]: cluster 2022-04-22T20:14:21.068333+0000 mgr.y (mgr.34107) 5765 : cluster [DBG] pgmap v6990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:23.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:23 smithi121 bash[36902]: audit 2022-04-22T20:14:22.079115+0000 mon.a (mon.0) 8937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:22 smithi115 bash[45495]: cluster 2022-04-22T20:14:21.068333+0000 mgr.y (mgr.34107) 5765 : cluster [DBG] pgmap v6990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:22 smithi115 bash[45495]: audit 2022-04-22T20:14:22.079115+0000 mon.a (mon.0) 8937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:22 smithi115 bash[46934]: cluster 2022-04-22T20:14:21.068333+0000 mgr.y (mgr.34107) 5765 : cluster [DBG] pgmap v6990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:22 smithi115 bash[46934]: audit 2022-04-22T20:14:22.079115+0000 mon.a (mon.0) 8937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:24 smithi121 bash[36902]: audit 2022-04-22T20:14:23.517602+0000 mon.a (mon.0) 8938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:24.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:24 smithi115 bash[45495]: audit 2022-04-22T20:14:23.517602+0000 mon.a (mon.0) 8938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:24 smithi115 bash[46934]: audit 2022-04-22T20:14:23.517602+0000 mon.a (mon.0) 8938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:25 smithi121 bash[36902]: cluster 2022-04-22T20:14:23.069380+0000 mgr.y (mgr.34107) 5766 : cluster [DBG] pgmap v6991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:25 smithi115 bash[45495]: cluster 2022-04-22T20:14:23.069380+0000 mgr.y (mgr.34107) 5766 : cluster [DBG] pgmap v6991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:25 smithi115 bash[46934]: cluster 2022-04-22T20:14:23.069380+0000 mgr.y (mgr.34107) 5766 : cluster [DBG] pgmap v6991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:27 smithi115 bash[45495]: cluster 2022-04-22T20:14:25.070401+0000 mgr.y (mgr.34107) 5767 : cluster [DBG] pgmap v6992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:27 smithi115 bash[45495]: audit 2022-04-22T20:14:26.849618+0000 mon.a (mon.0) 8939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:27 smithi115 bash[46934]: cluster 2022-04-22T20:14:25.070401+0000 mgr.y (mgr.34107) 5767 : cluster [DBG] pgmap v6992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:27 smithi115 bash[46934]: audit 2022-04-22T20:14:26.849618+0000 mon.a (mon.0) 8939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:27 smithi121 bash[36902]: cluster 2022-04-22T20:14:25.070401+0000 mgr.y (mgr.34107) 5767 : cluster [DBG] pgmap v6992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:27 smithi121 bash[36902]: audit 2022-04-22T20:14:26.849618+0000 mon.a (mon.0) 8939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:28 smithi115 bash[46934]: audit 2022-04-22T20:14:27.082357+0000 mon.a (mon.0) 8940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:28 smithi115 bash[45495]: audit 2022-04-22T20:14:27.082357+0000 mon.a (mon.0) 8940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:28 smithi121 bash[36902]: audit 2022-04-22T20:14:27.082357+0000 mon.a (mon.0) 8940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:29 smithi115 bash[45495]: cluster 2022-04-22T20:14:27.071582+0000 mgr.y (mgr.34107) 5768 : cluster [DBG] pgmap v6993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:29 smithi115 bash[45495]: audit 2022-04-22T20:14:28.520550+0000 mon.a (mon.0) 8941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:29 smithi115 bash[46934]: cluster 2022-04-22T20:14:27.071582+0000 mgr.y (mgr.34107) 5768 : cluster [DBG] pgmap v6993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:29 smithi115 bash[46934]: audit 2022-04-22T20:14:28.520550+0000 mon.a (mon.0) 8941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:29 smithi121 bash[36902]: cluster 2022-04-22T20:14:27.071582+0000 mgr.y (mgr.34107) 5768 : cluster [DBG] pgmap v6993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:29 smithi121 bash[36902]: audit 2022-04-22T20:14:28.520550+0000 mon.a (mon.0) 8941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:31 smithi115 bash[45495]: cluster 2022-04-22T20:14:29.072626+0000 mgr.y (mgr.34107) 5769 : cluster [DBG] pgmap v6994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:31 smithi115 bash[46934]: cluster 2022-04-22T20:14:29.072626+0000 mgr.y (mgr.34107) 5769 : cluster [DBG] pgmap v6994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:31 smithi121 bash[36902]: cluster 2022-04-22T20:14:29.072626+0000 mgr.y (mgr.34107) 5769 : cluster [DBG] pgmap v6994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:32 smithi121 bash[36902]: audit 2022-04-22T20:14:31.853341+0000 mon.a (mon.0) 8942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:32 smithi115 bash[45495]: audit 2022-04-22T20:14:31.853341+0000 mon.a (mon.0) 8942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:32 smithi115 bash[46934]: audit 2022-04-22T20:14:31.853341+0000 mon.a (mon.0) 8942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:32.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:32.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:33 smithi115 bash[46934]: cluster 2022-04-22T20:14:31.073711+0000 mgr.y (mgr.34107) 5770 : cluster [DBG] pgmap v6995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:33 smithi115 bash[46934]: audit 2022-04-22T20:14:32.085143+0000 mon.a (mon.0) 8943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:33 smithi115 bash[45495]: cluster 2022-04-22T20:14:31.073711+0000 mgr.y (mgr.34107) 5770 : cluster [DBG] pgmap v6995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:33 smithi115 bash[45495]: audit 2022-04-22T20:14:32.085143+0000 mon.a (mon.0) 8943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:33 smithi121 bash[36902]: cluster 2022-04-22T20:14:31.073711+0000 mgr.y (mgr.34107) 5770 : cluster [DBG] pgmap v6995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:33 smithi121 bash[36902]: audit 2022-04-22T20:14:32.085143+0000 mon.a (mon.0) 8943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:34 smithi121 bash[36902]: cluster 2022-04-22T20:14:33.074863+0000 mgr.y (mgr.34107) 5771 : cluster [DBG] pgmap v6996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:34 smithi121 bash[36902]: audit 2022-04-22T20:14:33.523527+0000 mon.a (mon.0) 8944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:34.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:34 smithi115 bash[45495]: cluster 2022-04-22T20:14:33.074863+0000 mgr.y (mgr.34107) 5771 : cluster [DBG] pgmap v6996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:34.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:34 smithi115 bash[45495]: audit 2022-04-22T20:14:33.523527+0000 mon.a (mon.0) 8944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:34.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:34.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:34.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:34 smithi115 bash[46934]: cluster 2022-04-22T20:14:33.074863+0000 mgr.y (mgr.34107) 5771 : cluster [DBG] pgmap v6996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:34.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:34 smithi115 bash[46934]: audit 2022-04-22T20:14:33.523527+0000 mon.a (mon.0) 8944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:36 smithi121 bash[36902]: cluster 2022-04-22T20:14:35.075797+0000 mgr.y (mgr.34107) 5772 : cluster [DBG] pgmap v6997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:36.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:36 smithi115 bash[45495]: cluster 2022-04-22T20:14:35.075797+0000 mgr.y (mgr.34107) 5772 : cluster [DBG] pgmap v6997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:36.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:36 smithi115 bash[46934]: cluster 2022-04-22T20:14:35.075797+0000 mgr.y (mgr.34107) 5772 : cluster [DBG] pgmap v6997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:37 smithi121 bash[36902]: audit 2022-04-22T20:14:36.857004+0000 mon.a (mon.0) 8945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:37 smithi121 bash[36902]: audit 2022-04-22T20:14:37.088006+0000 mon.a (mon.0) 8946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:37 smithi115 bash[45495]: audit 2022-04-22T20:14:36.857004+0000 mon.a (mon.0) 8945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:37 smithi115 bash[45495]: audit 2022-04-22T20:14:37.088006+0000 mon.a (mon.0) 8946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:37 smithi115 bash[46934]: audit 2022-04-22T20:14:36.857004+0000 mon.a (mon.0) 8945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:37 smithi115 bash[46934]: audit 2022-04-22T20:14:37.088006+0000 mon.a (mon.0) 8946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:38 smithi121 bash[36902]: cluster 2022-04-22T20:14:37.076924+0000 mgr.y (mgr.34107) 5773 : cluster [DBG] pgmap v6998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:38 smithi115 bash[45495]: cluster 2022-04-22T20:14:37.076924+0000 mgr.y (mgr.34107) 5773 : cluster [DBG] pgmap v6998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:38 smithi115 bash[46934]: cluster 2022-04-22T20:14:37.076924+0000 mgr.y (mgr.34107) 5773 : cluster [DBG] pgmap v6998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:39 smithi121 bash[36902]: audit 2022-04-22T20:14:38.526521+0000 mon.a (mon.0) 8947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:39.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:39 smithi115 bash[45495]: audit 2022-04-22T20:14:38.526521+0000 mon.a (mon.0) 8947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:39 smithi115 bash[46934]: audit 2022-04-22T20:14:38.526521+0000 mon.a (mon.0) 8947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:40 smithi121 bash[36902]: cluster 2022-04-22T20:14:39.078037+0000 mgr.y (mgr.34107) 5774 : cluster [DBG] pgmap v6999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:40 smithi115 bash[45495]: cluster 2022-04-22T20:14:39.078037+0000 mgr.y (mgr.34107) 5774 : cluster [DBG] pgmap v6999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:40 smithi115 bash[46934]: cluster 2022-04-22T20:14:39.078037+0000 mgr.y (mgr.34107) 5774 : cluster [DBG] pgmap v6999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:42.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:42 smithi121 bash[36902]: cluster 2022-04-22T20:14:41.079130+0000 mgr.y (mgr.34107) 5775 : cluster [DBG] pgmap v7000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:42 smithi121 bash[36902]: audit 2022-04-22T20:14:41.860376+0000 mon.a (mon.0) 8948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:42 smithi121 bash[36902]: audit 2022-04-22T20:14:42.091080+0000 mon.a (mon.0) 8949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[45495]: cluster 2022-04-22T20:14:41.079130+0000 mgr.y (mgr.34107) 5775 : cluster [DBG] pgmap v7000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[45495]: audit 2022-04-22T20:14:41.860376+0000 mon.a (mon.0) 8948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[45495]: audit 2022-04-22T20:14:42.091080+0000 mon.a (mon.0) 8949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[46934]: cluster 2022-04-22T20:14:41.079130+0000 mgr.y (mgr.34107) 5775 : cluster [DBG] pgmap v7000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[46934]: audit 2022-04-22T20:14:41.860376+0000 mon.a (mon.0) 8948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:42 smithi115 bash[46934]: audit 2022-04-22T20:14:42.091080+0000 mon.a (mon.0) 8949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:43.687 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:14:44.036 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:14:44.037 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:14:44.037 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 3m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:14:44.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:14:44.037 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:14:44.037 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:14:44.038 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:14:44.039 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:14:44.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:14:44.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:14:44.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:14:44.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:14:44.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:14:44.041 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:14:44.153 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:43 smithi115 bash[45495]: audit 2022-04-22T20:14:43.529655+0000 mon.a (mon.0) 8950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:44.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:43 smithi115 bash[46934]: audit 2022-04-22T20:14:43.529655+0000 mon.a (mon.0) 8950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:43 smithi121 bash[36902]: audit 2022-04-22T20:14:43.529655+0000 mon.a (mon.0) 8950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:44.443 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:14:44.444 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:14:44.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:44.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:44 smithi121 bash[36902]: cluster 2022-04-22T20:14:43.080260+0000 mgr.y (mgr.34107) 5776 : cluster [DBG] pgmap v7001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:44 smithi121 bash[36902]: audit 2022-04-22T20:14:43.661608+0000 mgr.y (mgr.34107) 5777 : audit [DBG] from='client.41400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:44 smithi121 bash[36902]: audit 2022-04-22T20:14:44.006987+0000 mgr.y (mgr.34107) 5778 : audit [DBG] from='client.51340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:44 smithi121 bash[36902]: audit 2022-04-22T20:14:44.433964+0000 mon.c (mon.1) 231 : audit [DBG] from='client.? 172.21.15.115:0/1611693043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[45495]: cluster 2022-04-22T20:14:43.080260+0000 mgr.y (mgr.34107) 5776 : cluster [DBG] pgmap v7001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[45495]: audit 2022-04-22T20:14:43.661608+0000 mgr.y (mgr.34107) 5777 : audit [DBG] from='client.41400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[45495]: audit 2022-04-22T20:14:44.006987+0000 mgr.y (mgr.34107) 5778 : audit [DBG] from='client.51340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[45495]: audit 2022-04-22T20:14:44.433964+0000 mon.c (mon.1) 231 : audit [DBG] from='client.? 172.21.15.115:0/1611693043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[46934]: cluster 2022-04-22T20:14:43.080260+0000 mgr.y (mgr.34107) 5776 : cluster [DBG] pgmap v7001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[46934]: audit 2022-04-22T20:14:43.661608+0000 mgr.y (mgr.34107) 5777 : audit [DBG] from='client.41400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[46934]: audit 2022-04-22T20:14:44.006987+0000 mgr.y (mgr.34107) 5778 : audit [DBG] from='client.51340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:14:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:44 smithi115 bash[46934]: audit 2022-04-22T20:14:44.433964+0000 mon.c (mon.1) 231 : audit [DBG] from='client.? 172.21.15.115:0/1611693043' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:14:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:46 smithi121 bash[36902]: cluster 2022-04-22T20:14:45.081124+0000 mgr.y (mgr.34107) 5779 : cluster [DBG] pgmap v7002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:46 smithi115 bash[45495]: cluster 2022-04-22T20:14:45.081124+0000 mgr.y (mgr.34107) 5779 : cluster [DBG] pgmap v7002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:46 smithi115 bash[46934]: cluster 2022-04-22T20:14:45.081124+0000 mgr.y (mgr.34107) 5779 : cluster [DBG] pgmap v7002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:47 smithi121 bash[36902]: audit 2022-04-22T20:14:46.863948+0000 mon.a (mon.0) 8951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:47 smithi121 bash[36902]: audit 2022-04-22T20:14:47.094029+0000 mon.a (mon.0) 8952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:47 smithi115 bash[45495]: audit 2022-04-22T20:14:46.863948+0000 mon.a (mon.0) 8951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:47 smithi115 bash[45495]: audit 2022-04-22T20:14:47.094029+0000 mon.a (mon.0) 8952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:47 smithi115 bash[46934]: audit 2022-04-22T20:14:46.863948+0000 mon.a (mon.0) 8951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:47 smithi115 bash[46934]: audit 2022-04-22T20:14:47.094029+0000 mon.a (mon.0) 8952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:48 smithi121 bash[36902]: cluster 2022-04-22T20:14:47.082313+0000 mgr.y (mgr.34107) 5780 : cluster [DBG] pgmap v7003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:48 smithi121 bash[36902]: audit 2022-04-22T20:14:48.532285+0000 mon.a (mon.0) 8953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:48 smithi115 bash[45495]: cluster 2022-04-22T20:14:47.082313+0000 mgr.y (mgr.34107) 5780 : cluster [DBG] pgmap v7003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:48 smithi115 bash[45495]: audit 2022-04-22T20:14:48.532285+0000 mon.a (mon.0) 8953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:48 smithi115 bash[46934]: cluster 2022-04-22T20:14:47.082313+0000 mgr.y (mgr.34107) 5780 : cluster [DBG] pgmap v7003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:48 smithi115 bash[46934]: audit 2022-04-22T20:14:48.532285+0000 mon.a (mon.0) 8953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:50 smithi121 bash[36902]: cluster 2022-04-22T20:14:49.083379+0000 mgr.y (mgr.34107) 5781 : cluster [DBG] pgmap v7004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:50 smithi115 bash[45495]: cluster 2022-04-22T20:14:49.083379+0000 mgr.y (mgr.34107) 5781 : cluster [DBG] pgmap v7004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:50 smithi115 bash[46934]: cluster 2022-04-22T20:14:49.083379+0000 mgr.y (mgr.34107) 5781 : cluster [DBG] pgmap v7004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:52.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:52 smithi121 bash[36902]: cluster 2022-04-22T20:14:51.084360+0000 mgr.y (mgr.34107) 5782 : cluster [DBG] pgmap v7005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:52 smithi121 bash[36902]: audit 2022-04-22T20:14:51.867697+0000 mon.a (mon.0) 8954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:52 smithi121 bash[36902]: audit 2022-04-22T20:14:52.096896+0000 mon.a (mon.0) 8955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[45495]: cluster 2022-04-22T20:14:51.084360+0000 mgr.y (mgr.34107) 5782 : cluster [DBG] pgmap v7005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[45495]: audit 2022-04-22T20:14:51.867697+0000 mon.a (mon.0) 8954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[45495]: audit 2022-04-22T20:14:52.096896+0000 mon.a (mon.0) 8955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[46934]: cluster 2022-04-22T20:14:51.084360+0000 mgr.y (mgr.34107) 5782 : cluster [DBG] pgmap v7005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[46934]: audit 2022-04-22T20:14:51.867697+0000 mon.a (mon.0) 8954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:52 smithi115 bash[46934]: audit 2022-04-22T20:14:52.096896+0000 mon.a (mon.0) 8955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:54.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:53 smithi115 bash[45495]: audit 2022-04-22T20:14:53.535216+0000 mon.a (mon.0) 8956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:54.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:53 smithi115 bash[46934]: audit 2022-04-22T20:14:53.535216+0000 mon.a (mon.0) 8956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:53 smithi121 bash[36902]: audit 2022-04-22T20:14:53.535216+0000 mon.a (mon.0) 8956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:14:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:14:54.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:14:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:54 smithi121 bash[36902]: cluster 2022-04-22T20:14:53.085439+0000 mgr.y (mgr.34107) 5783 : cluster [DBG] pgmap v7006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:54 smithi115 bash[45495]: cluster 2022-04-22T20:14:53.085439+0000 mgr.y (mgr.34107) 5783 : cluster [DBG] pgmap v7006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:54 smithi115 bash[46934]: cluster 2022-04-22T20:14:53.085439+0000 mgr.y (mgr.34107) 5783 : cluster [DBG] pgmap v7006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:56 smithi121 bash[36902]: cluster 2022-04-22T20:14:55.086367+0000 mgr.y (mgr.34107) 5784 : cluster [DBG] pgmap v7007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:56 smithi115 bash[45495]: cluster 2022-04-22T20:14:55.086367+0000 mgr.y (mgr.34107) 5784 : cluster [DBG] pgmap v7007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:56 smithi115 bash[46934]: cluster 2022-04-22T20:14:55.086367+0000 mgr.y (mgr.34107) 5784 : cluster [DBG] pgmap v7007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:58.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:57 smithi121 bash[36902]: audit 2022-04-22T20:14:56.871332+0000 mon.a (mon.0) 8957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:58.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:57 smithi121 bash[36902]: audit 2022-04-22T20:14:57.100114+0000 mon.a (mon.0) 8958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:57 smithi115 bash[46934]: audit 2022-04-22T20:14:56.871332+0000 mon.a (mon.0) 8957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:57 smithi115 bash[46934]: audit 2022-04-22T20:14:57.100114+0000 mon.a (mon.0) 8958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:57 smithi115 bash[45495]: audit 2022-04-22T20:14:56.871332+0000 mon.a (mon.0) 8957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:14:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:57 smithi115 bash[45495]: audit 2022-04-22T20:14:57.100114+0000 mon.a (mon.0) 8958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:14:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:58 smithi121 bash[36902]: cluster 2022-04-22T20:14:57.087563+0000 mgr.y (mgr.34107) 5785 : cluster [DBG] pgmap v7008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:14:58 smithi121 bash[36902]: audit 2022-04-22T20:14:58.538244+0000 mon.a (mon.0) 8959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:58 smithi115 bash[46934]: cluster 2022-04-22T20:14:57.087563+0000 mgr.y (mgr.34107) 5785 : cluster [DBG] pgmap v7008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:14:58 smithi115 bash[46934]: audit 2022-04-22T20:14:58.538244+0000 mon.a (mon.0) 8959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:14:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:58 smithi115 bash[45495]: cluster 2022-04-22T20:14:57.087563+0000 mgr.y (mgr.34107) 5785 : cluster [DBG] pgmap v7008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:14:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:14:58 smithi115 bash[45495]: audit 2022-04-22T20:14:58.538244+0000 mon.a (mon.0) 8959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:00 smithi121 bash[36902]: cluster 2022-04-22T20:14:59.088576+0000 mgr.y (mgr.34107) 5786 : cluster [DBG] pgmap v7009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:00 smithi115 bash[45495]: cluster 2022-04-22T20:14:59.088576+0000 mgr.y (mgr.34107) 5786 : cluster [DBG] pgmap v7009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:00 smithi115 bash[46934]: cluster 2022-04-22T20:14:59.088576+0000 mgr.y (mgr.34107) 5786 : cluster [DBG] pgmap v7009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:02.269Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:02 smithi121 bash[36902]: cluster 2022-04-22T20:15:01.089642+0000 mgr.y (mgr.34107) 5787 : cluster [DBG] pgmap v7010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:02 smithi121 bash[36902]: audit 2022-04-22T20:15:01.874644+0000 mon.a (mon.0) 8960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:02 smithi121 bash[36902]: audit 2022-04-22T20:15:02.103139+0000 mon.a (mon.0) 8961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[45495]: cluster 2022-04-22T20:15:01.089642+0000 mgr.y (mgr.34107) 5787 : cluster [DBG] pgmap v7010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[45495]: audit 2022-04-22T20:15:01.874644+0000 mon.a (mon.0) 8960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[45495]: audit 2022-04-22T20:15:02.103139+0000 mon.a (mon.0) 8961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[46934]: cluster 2022-04-22T20:15:01.089642+0000 mgr.y (mgr.34107) 5787 : cluster [DBG] pgmap v7010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[46934]: audit 2022-04-22T20:15:01.874644+0000 mon.a (mon.0) 8960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:02 smithi115 bash[46934]: audit 2022-04-22T20:15:02.103139+0000 mon.a (mon.0) 8961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:04.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:03 smithi115 bash[45495]: audit 2022-04-22T20:15:03.541357+0000 mon.a (mon.0) 8962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:04.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:03 smithi115 bash[46934]: audit 2022-04-22T20:15:03.541357+0000 mon.a (mon.0) 8962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:03 smithi121 bash[36902]: audit 2022-04-22T20:15:03.541357+0000 mon.a (mon.0) 8962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:04.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:04 smithi121 bash[36902]: cluster 2022-04-22T20:15:03.090742+0000 mgr.y (mgr.34107) 5788 : cluster [DBG] pgmap v7011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:04 smithi115 bash[46934]: cluster 2022-04-22T20:15:03.090742+0000 mgr.y (mgr.34107) 5788 : cluster [DBG] pgmap v7011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:04 smithi115 bash[45495]: cluster 2022-04-22T20:15:03.090742+0000 mgr.y (mgr.34107) 5788 : cluster [DBG] pgmap v7011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:06 smithi121 bash[36902]: cluster 2022-04-22T20:15:05.091709+0000 mgr.y (mgr.34107) 5789 : cluster [DBG] pgmap v7012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:06 smithi115 bash[45495]: cluster 2022-04-22T20:15:05.091709+0000 mgr.y (mgr.34107) 5789 : cluster [DBG] pgmap v7012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:06 smithi115 bash[46934]: cluster 2022-04-22T20:15:05.091709+0000 mgr.y (mgr.34107) 5789 : cluster [DBG] pgmap v7012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:07 smithi121 bash[36902]: audit 2022-04-22T20:15:06.878727+0000 mon.a (mon.0) 8963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:07 smithi121 bash[36902]: audit 2022-04-22T20:15:07.106395+0000 mon.a (mon.0) 8964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:08.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:07 smithi115 bash[45495]: audit 2022-04-22T20:15:06.878727+0000 mon.a (mon.0) 8963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:07 smithi115 bash[45495]: audit 2022-04-22T20:15:07.106395+0000 mon.a (mon.0) 8964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:07 smithi115 bash[46934]: audit 2022-04-22T20:15:06.878727+0000 mon.a (mon.0) 8963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:07 smithi115 bash[46934]: audit 2022-04-22T20:15:07.106395+0000 mon.a (mon.0) 8964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:08 smithi121 bash[36902]: cluster 2022-04-22T20:15:07.092933+0000 mgr.y (mgr.34107) 5790 : cluster [DBG] pgmap v7013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:08 smithi121 bash[36902]: audit 2022-04-22T20:15:08.544392+0000 mon.a (mon.0) 8965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:08 smithi115 bash[45495]: cluster 2022-04-22T20:15:07.092933+0000 mgr.y (mgr.34107) 5790 : cluster [DBG] pgmap v7013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:08 smithi115 bash[45495]: audit 2022-04-22T20:15:08.544392+0000 mon.a (mon.0) 8965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:08 smithi115 bash[46934]: cluster 2022-04-22T20:15:07.092933+0000 mgr.y (mgr.34107) 5790 : cluster [DBG] pgmap v7013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:08 smithi115 bash[46934]: audit 2022-04-22T20:15:08.544392+0000 mon.a (mon.0) 8965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:10 smithi121 bash[36902]: cluster 2022-04-22T20:15:09.094005+0000 mgr.y (mgr.34107) 5791 : cluster [DBG] pgmap v7014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:10 smithi115 bash[45495]: cluster 2022-04-22T20:15:09.094005+0000 mgr.y (mgr.34107) 5791 : cluster [DBG] pgmap v7014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:10 smithi115 bash[46934]: cluster 2022-04-22T20:15:09.094005+0000 mgr.y (mgr.34107) 5791 : cluster [DBG] pgmap v7014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:11 smithi121 bash[36902]: audit 2022-04-22T20:15:11.882122+0000 mon.a (mon.0) 8966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:12.277 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:11 smithi115 bash[45495]: audit 2022-04-22T20:15:11.882122+0000 mon.a (mon.0) 8966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:12.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:11 smithi115 bash[46934]: audit 2022-04-22T20:15:11.882122+0000 mon.a (mon.0) 8966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:12.269Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:12 smithi121 bash[36902]: cluster 2022-04-22T20:15:11.095097+0000 mgr.y (mgr.34107) 5792 : cluster [DBG] pgmap v7015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:12 smithi121 bash[36902]: audit 2022-04-22T20:15:12.109219+0000 mon.a (mon.0) 8967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:12 smithi115 bash[45495]: cluster 2022-04-22T20:15:11.095097+0000 mgr.y (mgr.34107) 5792 : cluster [DBG] pgmap v7015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:12 smithi115 bash[45495]: audit 2022-04-22T20:15:12.109219+0000 mon.a (mon.0) 8967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:12 smithi115 bash[46934]: cluster 2022-04-22T20:15:11.095097+0000 mgr.y (mgr.34107) 5792 : cluster [DBG] pgmap v7015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:12 smithi115 bash[46934]: audit 2022-04-22T20:15:12.109219+0000 mon.a (mon.0) 8967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:13 smithi121 bash[36902]: audit 2022-04-22T20:15:13.547341+0000 mon.a (mon.0) 8968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:14.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:13 smithi115 bash[46934]: audit 2022-04-22T20:15:13.547341+0000 mon.a (mon.0) 8968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:14.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:14.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:14.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:13 smithi115 bash[45495]: audit 2022-04-22T20:15:13.547341+0000 mon.a (mon.0) 8968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:14.808 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:15:15.154 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:15:15.154 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:15:15.154 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:15:15.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:15:15.155 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:15:15.155 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:15:15.155 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 2h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:15:15.155 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 2h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:15:15.155 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:15:15.156 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:15:15.156 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:15:15.156 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:15:15.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:15:15.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:15:15.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:15:15.157 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:15:15.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:15:15.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:15:15.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:15:15.159 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:15:15.159 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:15:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:14 smithi121 bash[36902]: cluster 2022-04-22T20:15:13.096273+0000 mgr.y (mgr.34107) 5793 : cluster [DBG] pgmap v7016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:14 smithi115 bash[45495]: cluster 2022-04-22T20:15:13.096273+0000 mgr.y (mgr.34107) 5793 : cluster [DBG] pgmap v7016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:14 smithi115 bash[46934]: cluster 2022-04-22T20:15:13.096273+0000 mgr.y (mgr.34107) 5793 : cluster [DBG] pgmap v7016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:15.551 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:15:15.551 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:15:15.551 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:15:15.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:15.551 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:15.552 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:15:15.553 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:15:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:15 smithi121 bash[36902]: audit 2022-04-22T20:15:14.784580+0000 mgr.y (mgr.34107) 5794 : audit [DBG] from='client.41418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:15 smithi121 bash[36902]: audit 2022-04-22T20:15:15.223162+0000 mon.a (mon.0) 8969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:15:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:15 smithi121 bash[36902]: audit 2022-04-22T20:15:15.223511+0000 mon.a (mon.0) 8970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:15:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:15 smithi121 bash[36902]: audit 2022-04-22T20:15:15.542124+0000 mon.c (mon.1) 232 : audit [DBG] from='client.? 172.21.15.115:0/2444487034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[45495]: audit 2022-04-22T20:15:14.784580+0000 mgr.y (mgr.34107) 5794 : audit [DBG] from='client.41418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[45495]: audit 2022-04-22T20:15:15.223162+0000 mon.a (mon.0) 8969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:15:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[45495]: audit 2022-04-22T20:15:15.223511+0000 mon.a (mon.0) 8970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:15:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[45495]: audit 2022-04-22T20:15:15.542124+0000 mon.c (mon.1) 232 : audit [DBG] from='client.? 172.21.15.115:0/2444487034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[46934]: audit 2022-04-22T20:15:14.784580+0000 mgr.y (mgr.34107) 5794 : audit [DBG] from='client.41418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[46934]: audit 2022-04-22T20:15:15.223162+0000 mon.a (mon.0) 8969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:15:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[46934]: audit 2022-04-22T20:15:15.223511+0000 mon.a (mon.0) 8970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:15:16.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:15 smithi115 bash[46934]: audit 2022-04-22T20:15:15.542124+0000 mon.c (mon.1) 232 : audit [DBG] from='client.? 172.21.15.115:0/2444487034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:16 smithi121 bash[36902]: cluster 2022-04-22T20:15:15.097123+0000 mgr.y (mgr.34107) 5795 : cluster [DBG] pgmap v7017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:16 smithi121 bash[36902]: audit 2022-04-22T20:15:15.125264+0000 mgr.y (mgr.34107) 5796 : audit [DBG] from='client.41424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:16 smithi121 bash[36902]: audit 2022-04-22T20:15:16.885933+0000 mon.a (mon.0) 8971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[45495]: cluster 2022-04-22T20:15:15.097123+0000 mgr.y (mgr.34107) 5795 : cluster [DBG] pgmap v7017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:17.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[45495]: audit 2022-04-22T20:15:15.125264+0000 mgr.y (mgr.34107) 5796 : audit [DBG] from='client.41424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:17.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[45495]: audit 2022-04-22T20:15:16.885933+0000 mon.a (mon.0) 8971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:17.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[46934]: cluster 2022-04-22T20:15:15.097123+0000 mgr.y (mgr.34107) 5795 : cluster [DBG] pgmap v7017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:17.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[46934]: audit 2022-04-22T20:15:15.125264+0000 mgr.y (mgr.34107) 5796 : audit [DBG] from='client.41424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:17.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:16 smithi115 bash[46934]: audit 2022-04-22T20:15:16.885933+0000 mon.a (mon.0) 8971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:17 smithi121 bash[36902]: audit 2022-04-22T20:15:17.112044+0000 mon.a (mon.0) 8972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:17 smithi115 bash[46934]: audit 2022-04-22T20:15:17.112044+0000 mon.a (mon.0) 8972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:17 smithi115 bash[45495]: audit 2022-04-22T20:15:17.112044+0000 mon.a (mon.0) 8972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:18 smithi121 bash[36902]: cluster 2022-04-22T20:15:17.098191+0000 mgr.y (mgr.34107) 5797 : cluster [DBG] pgmap v7018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:18 smithi121 bash[36902]: audit 2022-04-22T20:15:18.550121+0000 mon.a (mon.0) 8973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:18 smithi115 bash[45495]: cluster 2022-04-22T20:15:17.098191+0000 mgr.y (mgr.34107) 5797 : cluster [DBG] pgmap v7018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:18 smithi115 bash[45495]: audit 2022-04-22T20:15:18.550121+0000 mon.a (mon.0) 8973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:18 smithi115 bash[46934]: cluster 2022-04-22T20:15:17.098191+0000 mgr.y (mgr.34107) 5797 : cluster [DBG] pgmap v7018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:18 smithi115 bash[46934]: audit 2022-04-22T20:15:18.550121+0000 mon.a (mon.0) 8973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:20 smithi121 bash[36902]: cluster 2022-04-22T20:15:19.099217+0000 mgr.y (mgr.34107) 5798 : cluster [DBG] pgmap v7019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:20 smithi115 bash[45495]: cluster 2022-04-22T20:15:19.099217+0000 mgr.y (mgr.34107) 5798 : cluster [DBG] pgmap v7019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:20 smithi115 bash[46934]: cluster 2022-04-22T20:15:19.099217+0000 mgr.y (mgr.34107) 5798 : cluster [DBG] pgmap v7019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:22.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[45495]: audit 2022-04-22T20:15:21.876793+0000 mon.a (mon.0) 8974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:15:22.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[45495]: audit 2022-04-22T20:15:21.879749+0000 mon.a (mon.0) 8975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:15:22.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[45495]: audit 2022-04-22T20:15:21.881496+0000 mon.a (mon.0) 8976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:15:22.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[45495]: audit 2022-04-22T20:15:21.889655+0000 mon.a (mon.0) 8977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:22.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[46934]: audit 2022-04-22T20:15:21.876793+0000 mon.a (mon.0) 8974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:15:22.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[46934]: audit 2022-04-22T20:15:21.879749+0000 mon.a (mon.0) 8975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:15:22.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[46934]: audit 2022-04-22T20:15:21.881496+0000 mon.a (mon.0) 8976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:15:22.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:21 smithi115 bash[46934]: audit 2022-04-22T20:15:21.889655+0000 mon.a (mon.0) 8977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:22.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:21 smithi121 bash[36902]: audit 2022-04-22T20:15:21.876793+0000 mon.a (mon.0) 8974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:15:22.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:21 smithi121 bash[36902]: audit 2022-04-22T20:15:21.879749+0000 mon.a (mon.0) 8975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:15:22.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:21 smithi121 bash[36902]: audit 2022-04-22T20:15:21.881496+0000 mon.a (mon.0) 8976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:15:22.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:21 smithi121 bash[36902]: audit 2022-04-22T20:15:21.889655+0000 mon.a (mon.0) 8977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:22.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:22 smithi121 bash[36902]: cluster 2022-04-22T20:15:21.100288+0000 mgr.y (mgr.34107) 5799 : cluster [DBG] pgmap v7020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:22 smithi121 bash[36902]: audit 2022-04-22T20:15:22.115021+0000 mon.a (mon.0) 8978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:22 smithi121 bash[36902]: audit 2022-04-22T20:15:22.251843+0000 mon.a (mon.0) 8979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:15:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[46934]: cluster 2022-04-22T20:15:21.100288+0000 mgr.y (mgr.34107) 5799 : cluster [DBG] pgmap v7020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[46934]: audit 2022-04-22T20:15:22.115021+0000 mon.a (mon.0) 8978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[46934]: audit 2022-04-22T20:15:22.251843+0000 mon.a (mon.0) 8979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:15:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[45495]: cluster 2022-04-22T20:15:21.100288+0000 mgr.y (mgr.34107) 5799 : cluster [DBG] pgmap v7020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[45495]: audit 2022-04-22T20:15:22.115021+0000 mon.a (mon.0) 8978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:22 smithi115 bash[45495]: audit 2022-04-22T20:15:22.251843+0000 mon.a (mon.0) 8979 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:15:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:23 smithi121 bash[36902]: audit 2022-04-22T20:15:23.553041+0000 mon.a (mon.0) 8980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:23 smithi115 bash[45495]: audit 2022-04-22T20:15:23.553041+0000 mon.a (mon.0) 8980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:24.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:24.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:23 smithi115 bash[46934]: audit 2022-04-22T20:15:23.553041+0000 mon.a (mon.0) 8980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:24 smithi121 bash[36902]: cluster 2022-04-22T20:15:23.101478+0000 mgr.y (mgr.34107) 5800 : cluster [DBG] pgmap v7021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:24 smithi115 bash[45495]: cluster 2022-04-22T20:15:23.101478+0000 mgr.y (mgr.34107) 5800 : cluster [DBG] pgmap v7021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:24 smithi115 bash[46934]: cluster 2022-04-22T20:15:23.101478+0000 mgr.y (mgr.34107) 5800 : cluster [DBG] pgmap v7021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:26 smithi121 bash[36902]: cluster 2022-04-22T20:15:25.102482+0000 mgr.y (mgr.34107) 5801 : cluster [DBG] pgmap v7022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:26 smithi121 bash[36902]: audit 2022-04-22T20:15:26.893114+0000 mon.a (mon.0) 8981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:26 smithi115 bash[45495]: cluster 2022-04-22T20:15:25.102482+0000 mgr.y (mgr.34107) 5801 : cluster [DBG] pgmap v7022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:26 smithi115 bash[45495]: audit 2022-04-22T20:15:26.893114+0000 mon.a (mon.0) 8981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:26 smithi115 bash[46934]: cluster 2022-04-22T20:15:25.102482+0000 mgr.y (mgr.34107) 5801 : cluster [DBG] pgmap v7022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:26 smithi115 bash[46934]: audit 2022-04-22T20:15:26.893114+0000 mon.a (mon.0) 8981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:27 smithi121 bash[36902]: audit 2022-04-22T20:15:27.117984+0000 mon.a (mon.0) 8982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:27 smithi115 bash[45495]: audit 2022-04-22T20:15:27.117984+0000 mon.a (mon.0) 8982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:27 smithi115 bash[46934]: audit 2022-04-22T20:15:27.117984+0000 mon.a (mon.0) 8982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:28 smithi121 bash[36902]: cluster 2022-04-22T20:15:27.103653+0000 mgr.y (mgr.34107) 5802 : cluster [DBG] pgmap v7023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:28 smithi121 bash[36902]: audit 2022-04-22T20:15:28.556064+0000 mon.a (mon.0) 8983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:28 smithi115 bash[45495]: cluster 2022-04-22T20:15:27.103653+0000 mgr.y (mgr.34107) 5802 : cluster [DBG] pgmap v7023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:28 smithi115 bash[45495]: audit 2022-04-22T20:15:28.556064+0000 mon.a (mon.0) 8983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:28 smithi115 bash[46934]: cluster 2022-04-22T20:15:27.103653+0000 mgr.y (mgr.34107) 5802 : cluster [DBG] pgmap v7023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:28 smithi115 bash[46934]: audit 2022-04-22T20:15:28.556064+0000 mon.a (mon.0) 8983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:30 smithi121 bash[36902]: cluster 2022-04-22T20:15:29.104893+0000 mgr.y (mgr.34107) 5803 : cluster [DBG] pgmap v7024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:30 smithi115 bash[45495]: cluster 2022-04-22T20:15:29.104893+0000 mgr.y (mgr.34107) 5803 : cluster [DBG] pgmap v7024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:30 smithi115 bash[46934]: cluster 2022-04-22T20:15:29.104893+0000 mgr.y (mgr.34107) 5803 : cluster [DBG] pgmap v7024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:31 smithi121 bash[36902]: audit 2022-04-22T20:15:31.896893+0000 mon.a (mon.0) 8984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:32.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:31 smithi115 bash[45495]: audit 2022-04-22T20:15:31.896893+0000 mon.a (mon.0) 8984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:32.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:31 smithi115 bash[46934]: audit 2022-04-22T20:15:31.896893+0000 mon.a (mon.0) 8984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:32.270Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:33 smithi121 bash[36902]: cluster 2022-04-22T20:15:31.105944+0000 mgr.y (mgr.34107) 5804 : cluster [DBG] pgmap v7025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:33 smithi121 bash[36902]: audit 2022-04-22T20:15:32.121034+0000 mon.a (mon.0) 8985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:32 smithi115 bash[45495]: cluster 2022-04-22T20:15:31.105944+0000 mgr.y (mgr.34107) 5804 : cluster [DBG] pgmap v7025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:32 smithi115 bash[45495]: audit 2022-04-22T20:15:32.121034+0000 mon.a (mon.0) 8985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:32 smithi115 bash[46934]: cluster 2022-04-22T20:15:31.105944+0000 mgr.y (mgr.34107) 5804 : cluster [DBG] pgmap v7025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:32 smithi115 bash[46934]: audit 2022-04-22T20:15:32.121034+0000 mon.a (mon.0) 8985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:34 smithi121 bash[36902]: audit 2022-04-22T20:15:33.558960+0000 mon.a (mon.0) 8986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:34 smithi115 bash[45495]: audit 2022-04-22T20:15:33.558960+0000 mon.a (mon.0) 8986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:34.150Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:34 smithi115 bash[46934]: audit 2022-04-22T20:15:33.558960+0000 mon.a (mon.0) 8986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:35 smithi121 bash[36902]: cluster 2022-04-22T20:15:33.107201+0000 mgr.y (mgr.34107) 5805 : cluster [DBG] pgmap v7026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:35 smithi115 bash[45495]: cluster 2022-04-22T20:15:33.107201+0000 mgr.y (mgr.34107) 5805 : cluster [DBG] pgmap v7026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:35 smithi115 bash[46934]: cluster 2022-04-22T20:15:33.107201+0000 mgr.y (mgr.34107) 5805 : cluster [DBG] pgmap v7026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:37 smithi115 bash[45495]: cluster 2022-04-22T20:15:35.108332+0000 mgr.y (mgr.34107) 5806 : cluster [DBG] pgmap v7027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:37 smithi115 bash[45495]: audit 2022-04-22T20:15:36.900810+0000 mon.a (mon.0) 8987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:37 smithi115 bash[46934]: cluster 2022-04-22T20:15:35.108332+0000 mgr.y (mgr.34107) 5806 : cluster [DBG] pgmap v7027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:37 smithi115 bash[46934]: audit 2022-04-22T20:15:36.900810+0000 mon.a (mon.0) 8987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:37 smithi121 bash[36902]: cluster 2022-04-22T20:15:35.108332+0000 mgr.y (mgr.34107) 5806 : cluster [DBG] pgmap v7027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:37 smithi121 bash[36902]: audit 2022-04-22T20:15:36.900810+0000 mon.a (mon.0) 8987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:38 smithi115 bash[46934]: audit 2022-04-22T20:15:37.124379+0000 mon.a (mon.0) 8988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:38 smithi115 bash[45495]: audit 2022-04-22T20:15:37.124379+0000 mon.a (mon.0) 8988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:38 smithi121 bash[36902]: audit 2022-04-22T20:15:37.124379+0000 mon.a (mon.0) 8988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:39 smithi115 bash[46934]: cluster 2022-04-22T20:15:37.109602+0000 mgr.y (mgr.34107) 5807 : cluster [DBG] pgmap v7028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:39 smithi115 bash[46934]: audit 2022-04-22T20:15:38.561882+0000 mon.a (mon.0) 8989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:39 smithi115 bash[45495]: cluster 2022-04-22T20:15:37.109602+0000 mgr.y (mgr.34107) 5807 : cluster [DBG] pgmap v7028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:39 smithi115 bash[45495]: audit 2022-04-22T20:15:38.561882+0000 mon.a (mon.0) 8989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:39 smithi121 bash[36902]: cluster 2022-04-22T20:15:37.109602+0000 mgr.y (mgr.34107) 5807 : cluster [DBG] pgmap v7028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:39 smithi121 bash[36902]: audit 2022-04-22T20:15:38.561882+0000 mon.a (mon.0) 8989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:41 smithi115 bash[45495]: cluster 2022-04-22T20:15:39.110764+0000 mgr.y (mgr.34107) 5808 : cluster [DBG] pgmap v7029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:41 smithi115 bash[46934]: cluster 2022-04-22T20:15:39.110764+0000 mgr.y (mgr.34107) 5808 : cluster [DBG] pgmap v7029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:41 smithi121 bash[36902]: cluster 2022-04-22T20:15:39.110764+0000 mgr.y (mgr.34107) 5808 : cluster [DBG] pgmap v7029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:42.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:42 smithi115 bash[45495]: audit 2022-04-22T20:15:41.904533+0000 mon.a (mon.0) 8990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:42.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:42 smithi115 bash[46934]: audit 2022-04-22T20:15:41.904533+0000 mon.a (mon.0) 8990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:42.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:42.270Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:42 smithi121 bash[36902]: audit 2022-04-22T20:15:41.904533+0000 mon.a (mon.0) 8990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:43 smithi115 bash[45495]: cluster 2022-04-22T20:15:41.111910+0000 mgr.y (mgr.34107) 5809 : cluster [DBG] pgmap v7030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:43 smithi115 bash[45495]: audit 2022-04-22T20:15:42.127348+0000 mon.a (mon.0) 8991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:43 smithi115 bash[46934]: cluster 2022-04-22T20:15:41.111910+0000 mgr.y (mgr.34107) 5809 : cluster [DBG] pgmap v7030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:43 smithi115 bash[46934]: audit 2022-04-22T20:15:42.127348+0000 mon.a (mon.0) 8991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:43 smithi121 bash[36902]: cluster 2022-04-22T20:15:41.111910+0000 mgr.y (mgr.34107) 5809 : cluster [DBG] pgmap v7030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:43 smithi121 bash[36902]: audit 2022-04-22T20:15:42.127348+0000 mon.a (mon.0) 8991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:44 smithi121 bash[36902]: audit 2022-04-22T20:15:43.564975+0000 mon.a (mon.0) 8992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:44 smithi115 bash[45495]: audit 2022-04-22T20:15:43.564975+0000 mon.a (mon.0) 8992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:44 smithi115 bash[46934]: audit 2022-04-22T20:15:43.564975+0000 mon.a (mon.0) 8992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:44.150Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:45 smithi121 bash[36902]: cluster 2022-04-22T20:15:43.113034+0000 mgr.y (mgr.34107) 5810 : cluster [DBG] pgmap v7031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:45 smithi115 bash[45495]: cluster 2022-04-22T20:15:43.113034+0000 mgr.y (mgr.34107) 5810 : cluster [DBG] pgmap v7031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:45 smithi115 bash[46934]: cluster 2022-04-22T20:15:43.113034+0000 mgr.y (mgr.34107) 5810 : cluster [DBG] pgmap v7031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:45.923 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:15:46.274 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:15:46.274 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:15:46.274 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 4m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:15:46.274 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:15:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:15:46.276 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:15:46.277 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:15:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:46 smithi121 bash[36902]: cluster 2022-04-22T20:15:45.114018+0000 mgr.y (mgr.34107) 5811 : cluster [DBG] pgmap v7032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:46 smithi121 bash[36902]: audit 2022-04-22T20:15:45.899065+0000 mgr.y (mgr.34107) 5812 : audit [DBG] from='client.41436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:46 smithi115 bash[46934]: cluster 2022-04-22T20:15:45.114018+0000 mgr.y (mgr.34107) 5811 : cluster [DBG] pgmap v7032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:46 smithi115 bash[46934]: audit 2022-04-22T20:15:45.899065+0000 mgr.y (mgr.34107) 5812 : audit [DBG] from='client.41436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:46.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:46 smithi115 bash[45495]: cluster 2022-04-22T20:15:45.114018+0000 mgr.y (mgr.34107) 5811 : cluster [DBG] pgmap v7032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:46.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:46 smithi115 bash[45495]: audit 2022-04-22T20:15:45.899065+0000 mgr.y (mgr.34107) 5812 : audit [DBG] from='client.41436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:15:46.686 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:15:46.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:15:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:15:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:15:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:15:46.688 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:15:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:47 smithi121 bash[36902]: audit 2022-04-22T20:15:46.250090+0000 mgr.y (mgr.34107) 5813 : audit [DBG] from='client.41442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:47 smithi121 bash[36902]: audit 2022-04-22T20:15:46.677345+0000 mon.a (mon.0) 8993 : audit [DBG] from='client.? 172.21.15.115:0/3683061771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:47 smithi121 bash[36902]: audit 2022-04-22T20:15:46.908596+0000 mon.a (mon.0) 8994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[45495]: audit 2022-04-22T20:15:46.250090+0000 mgr.y (mgr.34107) 5813 : audit [DBG] from='client.41442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[45495]: audit 2022-04-22T20:15:46.677345+0000 mon.a (mon.0) 8993 : audit [DBG] from='client.? 172.21.15.115:0/3683061771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[45495]: audit 2022-04-22T20:15:46.908596+0000 mon.a (mon.0) 8994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[46934]: audit 2022-04-22T20:15:46.250090+0000 mgr.y (mgr.34107) 5813 : audit [DBG] from='client.41442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:15:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[46934]: audit 2022-04-22T20:15:46.677345+0000 mon.a (mon.0) 8993 : audit [DBG] from='client.? 172.21.15.115:0/3683061771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:15:47.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:47 smithi115 bash[46934]: audit 2022-04-22T20:15:46.908596+0000 mon.a (mon.0) 8994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:48 smithi121 bash[36902]: cluster 2022-04-22T20:15:47.114955+0000 mgr.y (mgr.34107) 5814 : cluster [DBG] pgmap v7033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:48 smithi121 bash[36902]: audit 2022-04-22T20:15:47.130521+0000 mon.a (mon.0) 8995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:48 smithi115 bash[45495]: cluster 2022-04-22T20:15:47.114955+0000 mgr.y (mgr.34107) 5814 : cluster [DBG] pgmap v7033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:48 smithi115 bash[45495]: audit 2022-04-22T20:15:47.130521+0000 mon.a (mon.0) 8995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:48 smithi115 bash[46934]: cluster 2022-04-22T20:15:47.114955+0000 mgr.y (mgr.34107) 5814 : cluster [DBG] pgmap v7033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:48 smithi115 bash[46934]: audit 2022-04-22T20:15:47.130521+0000 mon.a (mon.0) 8995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:49 smithi121 bash[36902]: audit 2022-04-22T20:15:48.567758+0000 mon.a (mon.0) 8996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:49 smithi115 bash[45495]: audit 2022-04-22T20:15:48.567758+0000 mon.a (mon.0) 8996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:49 smithi115 bash[46934]: audit 2022-04-22T20:15:48.567758+0000 mon.a (mon.0) 8996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:50 smithi121 bash[36902]: cluster 2022-04-22T20:15:49.115882+0000 mgr.y (mgr.34107) 5815 : cluster [DBG] pgmap v7034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:50.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:50 smithi115 bash[45495]: cluster 2022-04-22T20:15:49.115882+0000 mgr.y (mgr.34107) 5815 : cluster [DBG] pgmap v7034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:50 smithi115 bash[46934]: cluster 2022-04-22T20:15:49.115882+0000 mgr.y (mgr.34107) 5815 : cluster [DBG] pgmap v7034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:52 smithi121 bash[36902]: cluster 2022-04-22T20:15:51.116911+0000 mgr.y (mgr.34107) 5816 : cluster [DBG] pgmap v7035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:52 smithi121 bash[36902]: audit 2022-04-22T20:15:51.912346+0000 mon.a (mon.0) 8997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:52 smithi121 bash[36902]: audit 2022-04-22T20:15:52.133608+0000 mon.a (mon.0) 8998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[46934]: cluster 2022-04-22T20:15:51.116911+0000 mgr.y (mgr.34107) 5816 : cluster [DBG] pgmap v7035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[46934]: audit 2022-04-22T20:15:51.912346+0000 mon.a (mon.0) 8997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[46934]: audit 2022-04-22T20:15:52.133608+0000 mon.a (mon.0) 8998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:52.271Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[45495]: cluster 2022-04-22T20:15:51.116911+0000 mgr.y (mgr.34107) 5816 : cluster [DBG] pgmap v7035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[45495]: audit 2022-04-22T20:15:51.912346+0000 mon.a (mon.0) 8997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:52.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:52 smithi115 bash[45495]: audit 2022-04-22T20:15:52.133608+0000 mon.a (mon.0) 8998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:54 smithi121 bash[36902]: cluster 2022-04-22T20:15:53.118025+0000 mgr.y (mgr.34107) 5817 : cluster [DBG] pgmap v7036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:54 smithi121 bash[36902]: audit 2022-04-22T20:15:53.570558+0000 mon.a (mon.0) 8999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:15:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:15:54.150Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:15:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:54 smithi115 bash[46934]: cluster 2022-04-22T20:15:53.118025+0000 mgr.y (mgr.34107) 5817 : cluster [DBG] pgmap v7036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:54 smithi115 bash[46934]: audit 2022-04-22T20:15:53.570558+0000 mon.a (mon.0) 8999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:54 smithi115 bash[45495]: cluster 2022-04-22T20:15:53.118025+0000 mgr.y (mgr.34107) 5817 : cluster [DBG] pgmap v7036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:54 smithi115 bash[45495]: audit 2022-04-22T20:15:53.570558+0000 mon.a (mon.0) 8999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:56 smithi121 bash[36902]: cluster 2022-04-22T20:15:55.118994+0000 mgr.y (mgr.34107) 5818 : cluster [DBG] pgmap v7037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:56 smithi115 bash[45495]: cluster 2022-04-22T20:15:55.118994+0000 mgr.y (mgr.34107) 5818 : cluster [DBG] pgmap v7037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:56 smithi115 bash[46934]: cluster 2022-04-22T20:15:55.118994+0000 mgr.y (mgr.34107) 5818 : cluster [DBG] pgmap v7037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:57 smithi121 bash[36902]: audit 2022-04-22T20:15:56.916053+0000 mon.a (mon.0) 9000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:57 smithi121 bash[36902]: audit 2022-04-22T20:15:57.136873+0000 mon.a (mon.0) 9001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:57 smithi115 bash[45495]: audit 2022-04-22T20:15:56.916053+0000 mon.a (mon.0) 9000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:57 smithi115 bash[45495]: audit 2022-04-22T20:15:57.136873+0000 mon.a (mon.0) 9001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:57 smithi115 bash[46934]: audit 2022-04-22T20:15:56.916053+0000 mon.a (mon.0) 9000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:15:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:57 smithi115 bash[46934]: audit 2022-04-22T20:15:57.136873+0000 mon.a (mon.0) 9001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:15:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:58 smithi121 bash[36902]: cluster 2022-04-22T20:15:57.120160+0000 mgr.y (mgr.34107) 5819 : cluster [DBG] pgmap v7038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:58 smithi115 bash[45495]: cluster 2022-04-22T20:15:57.120160+0000 mgr.y (mgr.34107) 5819 : cluster [DBG] pgmap v7038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:58 smithi115 bash[46934]: cluster 2022-04-22T20:15:57.120160+0000 mgr.y (mgr.34107) 5819 : cluster [DBG] pgmap v7038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:15:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:15:59 smithi121 bash[36902]: audit 2022-04-22T20:15:58.573676+0000 mon.a (mon.0) 9002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:15:59 smithi115 bash[45495]: audit 2022-04-22T20:15:58.573676+0000 mon.a (mon.0) 9002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:15:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:15:59 smithi115 bash[46934]: audit 2022-04-22T20:15:58.573676+0000 mon.a (mon.0) 9002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:00 smithi121 bash[36902]: cluster 2022-04-22T20:15:59.121197+0000 mgr.y (mgr.34107) 5820 : cluster [DBG] pgmap v7039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:00 smithi115 bash[45495]: cluster 2022-04-22T20:15:59.121197+0000 mgr.y (mgr.34107) 5820 : cluster [DBG] pgmap v7039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:01.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:00 smithi115 bash[46934]: cluster 2022-04-22T20:15:59.121197+0000 mgr.y (mgr.34107) 5820 : cluster [DBG] pgmap v7039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:02.272Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:02 smithi121 bash[36902]: cluster 2022-04-22T20:16:01.122239+0000 mgr.y (mgr.34107) 5821 : cluster [DBG] pgmap v7040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:02 smithi121 bash[36902]: audit 2022-04-22T20:16:01.919657+0000 mon.a (mon.0) 9003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:02 smithi121 bash[36902]: audit 2022-04-22T20:16:02.139977+0000 mon.a (mon.0) 9004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[45495]: cluster 2022-04-22T20:16:01.122239+0000 mgr.y (mgr.34107) 5821 : cluster [DBG] pgmap v7040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:03.412 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[45495]: audit 2022-04-22T20:16:01.919657+0000 mon.a (mon.0) 9003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:03.412 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[45495]: audit 2022-04-22T20:16:02.139977+0000 mon.a (mon.0) 9004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:03.413 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[46934]: cluster 2022-04-22T20:16:01.122239+0000 mgr.y (mgr.34107) 5821 : cluster [DBG] pgmap v7040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:03.413 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[46934]: audit 2022-04-22T20:16:01.919657+0000 mon.a (mon.0) 9003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:03.413 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:02 smithi115 bash[46934]: audit 2022-04-22T20:16:02.139977+0000 mon.a (mon.0) 9004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:04.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:03 smithi115 bash[45495]: audit 2022-04-22T20:16:03.576846+0000 mon.a (mon.0) 9005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:04.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:03 smithi115 bash[46934]: audit 2022-04-22T20:16:03.576846+0000 mon.a (mon.0) 9005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:03 smithi121 bash[36902]: audit 2022-04-22T20:16:03.576846+0000 mon.a (mon.0) 9005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:04.151Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:04 smithi121 bash[36902]: cluster 2022-04-22T20:16:03.123290+0000 mgr.y (mgr.34107) 5822 : cluster [DBG] pgmap v7041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:04 smithi115 bash[45495]: cluster 2022-04-22T20:16:03.123290+0000 mgr.y (mgr.34107) 5822 : cluster [DBG] pgmap v7041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:04 smithi115 bash[46934]: cluster 2022-04-22T20:16:03.123290+0000 mgr.y (mgr.34107) 5822 : cluster [DBG] pgmap v7041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:06 smithi121 bash[36902]: cluster 2022-04-22T20:16:05.124538+0000 mgr.y (mgr.34107) 5823 : cluster [DBG] pgmap v7042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:06 smithi115 bash[45495]: cluster 2022-04-22T20:16:05.124538+0000 mgr.y (mgr.34107) 5823 : cluster [DBG] pgmap v7042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:06 smithi115 bash[46934]: cluster 2022-04-22T20:16:05.124538+0000 mgr.y (mgr.34107) 5823 : cluster [DBG] pgmap v7042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:07 smithi121 bash[36902]: audit 2022-04-22T20:16:06.923234+0000 mon.a (mon.0) 9006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:07 smithi121 bash[36902]: audit 2022-04-22T20:16:07.143155+0000 mon.a (mon.0) 9007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:07 smithi115 bash[45495]: audit 2022-04-22T20:16:06.923234+0000 mon.a (mon.0) 9006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:07 smithi115 bash[45495]: audit 2022-04-22T20:16:07.143155+0000 mon.a (mon.0) 9007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:07 smithi115 bash[46934]: audit 2022-04-22T20:16:06.923234+0000 mon.a (mon.0) 9006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:07 smithi115 bash[46934]: audit 2022-04-22T20:16:07.143155+0000 mon.a (mon.0) 9007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:08 smithi121 bash[36902]: cluster 2022-04-22T20:16:07.125703+0000 mgr.y (mgr.34107) 5824 : cluster [DBG] pgmap v7043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:08 smithi121 bash[36902]: audit 2022-04-22T20:16:08.579948+0000 mon.a (mon.0) 9008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:08 smithi115 bash[45495]: cluster 2022-04-22T20:16:07.125703+0000 mgr.y (mgr.34107) 5824 : cluster [DBG] pgmap v7043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:08 smithi115 bash[45495]: audit 2022-04-22T20:16:08.579948+0000 mon.a (mon.0) 9008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:08 smithi115 bash[46934]: cluster 2022-04-22T20:16:07.125703+0000 mgr.y (mgr.34107) 5824 : cluster [DBG] pgmap v7043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:08 smithi115 bash[46934]: audit 2022-04-22T20:16:08.579948+0000 mon.a (mon.0) 9008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:10 smithi121 bash[36902]: cluster 2022-04-22T20:16:09.126918+0000 mgr.y (mgr.34107) 5825 : cluster [DBG] pgmap v7044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:10 smithi115 bash[45495]: cluster 2022-04-22T20:16:09.126918+0000 mgr.y (mgr.34107) 5825 : cluster [DBG] pgmap v7044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:10 smithi115 bash[46934]: cluster 2022-04-22T20:16:09.126918+0000 mgr.y (mgr.34107) 5825 : cluster [DBG] pgmap v7044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:12.272Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:12 smithi121 bash[36902]: cluster 2022-04-22T20:16:11.128217+0000 mgr.y (mgr.34107) 5826 : cluster [DBG] pgmap v7045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:12 smithi121 bash[36902]: audit 2022-04-22T20:16:11.926948+0000 mon.a (mon.0) 9009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:12 smithi121 bash[36902]: audit 2022-04-22T20:16:12.146324+0000 mon.a (mon.0) 9010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[45495]: cluster 2022-04-22T20:16:11.128217+0000 mgr.y (mgr.34107) 5826 : cluster [DBG] pgmap v7045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[45495]: audit 2022-04-22T20:16:11.926948+0000 mon.a (mon.0) 9009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[45495]: audit 2022-04-22T20:16:12.146324+0000 mon.a (mon.0) 9010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[46934]: cluster 2022-04-22T20:16:11.128217+0000 mgr.y (mgr.34107) 5826 : cluster [DBG] pgmap v7045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[46934]: audit 2022-04-22T20:16:11.926948+0000 mon.a (mon.0) 9009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:12 smithi115 bash[46934]: audit 2022-04-22T20:16:12.146324+0000 mon.a (mon.0) 9010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:14.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:13 smithi115 bash[45495]: audit 2022-04-22T20:16:13.583089+0000 mon.a (mon.0) 9011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:14.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:13 smithi115 bash[46934]: audit 2022-04-22T20:16:13.583089+0000 mon.a (mon.0) 9011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:13 smithi121 bash[36902]: audit 2022-04-22T20:16:13.583089+0000 mon.a (mon.0) 9011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:14.152Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:14 smithi121 bash[36902]: cluster 2022-04-22T20:16:13.129205+0000 mgr.y (mgr.34107) 5827 : cluster [DBG] pgmap v7046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:14 smithi115 bash[45495]: cluster 2022-04-22T20:16:13.129205+0000 mgr.y (mgr.34107) 5827 : cluster [DBG] pgmap v7046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:14 smithi115 bash[46934]: cluster 2022-04-22T20:16:13.129205+0000 mgr.y (mgr.34107) 5827 : cluster [DBG] pgmap v7046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:15 smithi121 bash[36902]: audit 2022-04-22T20:16:15.233627+0000 mon.a (mon.0) 9012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:16:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:15 smithi121 bash[36902]: audit 2022-04-22T20:16:15.234123+0000 mon.a (mon.0) 9013 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:16:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:15 smithi115 bash[45495]: audit 2022-04-22T20:16:15.233627+0000 mon.a (mon.0) 9012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:16:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:15 smithi115 bash[45495]: audit 2022-04-22T20:16:15.234123+0000 mon.a (mon.0) 9013 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:16:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:15 smithi115 bash[46934]: audit 2022-04-22T20:16:15.233627+0000 mon.a (mon.0) 9012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:16:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:15 smithi115 bash[46934]: audit 2022-04-22T20:16:15.234123+0000 mon.a (mon.0) 9013 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:16:17.060 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:16:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:16 smithi121 bash[36902]: cluster 2022-04-22T20:16:15.130125+0000 mgr.y (mgr.34107) 5828 : cluster [DBG] pgmap v7047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:16 smithi115 bash[45495]: cluster 2022-04-22T20:16:15.130125+0000 mgr.y (mgr.34107) 5828 : cluster [DBG] pgmap v7047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:16 smithi115 bash[46934]: cluster 2022-04-22T20:16:15.130125+0000 mgr.y (mgr.34107) 5828 : cluster [DBG] pgmap v7047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:17.404 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:16:17.415 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:16:17.415 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:16:17.415 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:16:17.416 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:16:17.417 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:16:17.812 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:16:17.812 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:16:17.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:16:17.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:17.813 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:16:17.813 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:16:17.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:17.813 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:16:17.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:16:17.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:17.814 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:16:17.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:16:17.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:17.815 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:16:17.815 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:16:17.815 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:16:17.815 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:16:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:17 smithi121 bash[36902]: audit 2022-04-22T20:16:16.930601+0000 mon.a (mon.0) 9014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:17 smithi121 bash[36902]: audit 2022-04-22T20:16:17.035218+0000 mgr.y (mgr.34107) 5829 : audit [DBG] from='client.41454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:17 smithi121 bash[36902]: audit 2022-04-22T20:16:17.149440+0000 mon.a (mon.0) 9015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:17 smithi121 bash[36902]: audit 2022-04-22T20:16:17.802857+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.115:0/3651130443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[46934]: audit 2022-04-22T20:16:16.930601+0000 mon.a (mon.0) 9014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[46934]: audit 2022-04-22T20:16:17.035218+0000 mgr.y (mgr.34107) 5829 : audit [DBG] from='client.41454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[46934]: audit 2022-04-22T20:16:17.149440+0000 mon.a (mon.0) 9015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[46934]: audit 2022-04-22T20:16:17.802857+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.115:0/3651130443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[45495]: audit 2022-04-22T20:16:16.930601+0000 mon.a (mon.0) 9014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[45495]: audit 2022-04-22T20:16:17.035218+0000 mgr.y (mgr.34107) 5829 : audit [DBG] from='client.41454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[45495]: audit 2022-04-22T20:16:17.149440+0000 mon.a (mon.0) 9015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:17 smithi115 bash[45495]: audit 2022-04-22T20:16:17.802857+0000 mon.c (mon.1) 233 : audit [DBG] from='client.? 172.21.15.115:0/3651130443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:18 smithi121 bash[36902]: cluster 2022-04-22T20:16:17.131247+0000 mgr.y (mgr.34107) 5830 : cluster [DBG] pgmap v7048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:18 smithi121 bash[36902]: audit 2022-04-22T20:16:17.374088+0000 mgr.y (mgr.34107) 5831 : audit [DBG] from='client.41460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:18 smithi121 bash[36902]: audit 2022-04-22T20:16:18.585780+0000 mon.a (mon.0) 9016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[46934]: cluster 2022-04-22T20:16:17.131247+0000 mgr.y (mgr.34107) 5830 : cluster [DBG] pgmap v7048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[46934]: audit 2022-04-22T20:16:17.374088+0000 mgr.y (mgr.34107) 5831 : audit [DBG] from='client.41460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[46934]: audit 2022-04-22T20:16:18.585780+0000 mon.a (mon.0) 9016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[45495]: cluster 2022-04-22T20:16:17.131247+0000 mgr.y (mgr.34107) 5830 : cluster [DBG] pgmap v7048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[45495]: audit 2022-04-22T20:16:17.374088+0000 mgr.y (mgr.34107) 5831 : audit [DBG] from='client.41460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:19.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:18 smithi115 bash[45495]: audit 2022-04-22T20:16:18.585780+0000 mon.a (mon.0) 9016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:20 smithi121 bash[36902]: cluster 2022-04-22T20:16:19.132365+0000 mgr.y (mgr.34107) 5832 : cluster [DBG] pgmap v7049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:20 smithi115 bash[46934]: cluster 2022-04-22T20:16:19.132365+0000 mgr.y (mgr.34107) 5832 : cluster [DBG] pgmap v7049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:20 smithi115 bash[45495]: cluster 2022-04-22T20:16:19.132365+0000 mgr.y (mgr.34107) 5832 : cluster [DBG] pgmap v7049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:22.549 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:22.275Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: cluster 2022-04-22T20:16:21.133526+0000 mgr.y (mgr.34107) 5833 : cluster [DBG] pgmap v7050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:21.934219+0000 mon.a (mon.0) 9017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:22.152236+0000 mon.a (mon.0) 9018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:22.259988+0000 mon.a (mon.0) 9019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:16:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:22.262662+0000 mon.a (mon.0) 9020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:16:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:22.264339+0000 mon.a (mon.0) 9021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:16:23.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:22 smithi121 bash[36902]: audit 2022-04-22T20:16:22.614046+0000 mon.a (mon.0) 9022 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:16:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: cluster 2022-04-22T20:16:21.133526+0000 mgr.y (mgr.34107) 5833 : cluster [DBG] pgmap v7050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:21.934219+0000 mon.a (mon.0) 9017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:22.152236+0000 mon.a (mon.0) 9018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:22.259988+0000 mon.a (mon.0) 9019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:16:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:22.262662+0000 mon.a (mon.0) 9020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:16:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:22.264339+0000 mon.a (mon.0) 9021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:16:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[45495]: audit 2022-04-22T20:16:22.614046+0000 mon.a (mon.0) 9022 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:16:23.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: cluster 2022-04-22T20:16:21.133526+0000 mgr.y (mgr.34107) 5833 : cluster [DBG] pgmap v7050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:23.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:21.934219+0000 mon.a (mon.0) 9017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:23.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:22.152236+0000 mon.a (mon.0) 9018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:23.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:22.259988+0000 mon.a (mon.0) 9019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:16:23.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:22.262662+0000 mon.a (mon.0) 9020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:16:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:22.264339+0000 mon.a (mon.0) 9021 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:16:23.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:22 smithi115 bash[46934]: audit 2022-04-22T20:16:22.614046+0000 mon.a (mon.0) 9022 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:16:24.162 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:23 smithi115 bash[45495]: audit 2022-04-22T20:16:23.588163+0000 mon.a (mon.0) 9023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:24.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:23 smithi115 bash[46934]: audit 2022-04-22T20:16:23.588163+0000 mon.a (mon.0) 9023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:23 smithi121 bash[36902]: audit 2022-04-22T20:16:23.588163+0000 mon.a (mon.0) 9023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:24.153Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:25.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:24 smithi121 bash[36902]: cluster 2022-04-22T20:16:23.134659+0000 mgr.y (mgr.34107) 5834 : cluster [DBG] pgmap v7051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:24 smithi115 bash[45495]: cluster 2022-04-22T20:16:23.134659+0000 mgr.y (mgr.34107) 5834 : cluster [DBG] pgmap v7051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:24 smithi115 bash[46934]: cluster 2022-04-22T20:16:23.134659+0000 mgr.y (mgr.34107) 5834 : cluster [DBG] pgmap v7051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:26 smithi121 bash[36902]: cluster 2022-04-22T20:16:25.135654+0000 mgr.y (mgr.34107) 5835 : cluster [DBG] pgmap v7052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:27.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:26 smithi115 bash[45495]: cluster 2022-04-22T20:16:25.135654+0000 mgr.y (mgr.34107) 5835 : cluster [DBG] pgmap v7052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:27.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:26 smithi115 bash[46934]: cluster 2022-04-22T20:16:25.135654+0000 mgr.y (mgr.34107) 5835 : cluster [DBG] pgmap v7052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:27 smithi121 bash[36902]: audit 2022-04-22T20:16:26.937889+0000 mon.a (mon.0) 9024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:27 smithi121 bash[36902]: audit 2022-04-22T20:16:27.155327+0000 mon.a (mon.0) 9025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:27 smithi115 bash[45495]: audit 2022-04-22T20:16:26.937889+0000 mon.a (mon.0) 9024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:27 smithi115 bash[45495]: audit 2022-04-22T20:16:27.155327+0000 mon.a (mon.0) 9025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:27 smithi115 bash[46934]: audit 2022-04-22T20:16:26.937889+0000 mon.a (mon.0) 9024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:27 smithi115 bash[46934]: audit 2022-04-22T20:16:27.155327+0000 mon.a (mon.0) 9025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:28 smithi121 bash[36902]: cluster 2022-04-22T20:16:27.136732+0000 mgr.y (mgr.34107) 5836 : cluster [DBG] pgmap v7053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:28 smithi121 bash[36902]: audit 2022-04-22T20:16:28.591219+0000 mon.a (mon.0) 9026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:28 smithi115 bash[45495]: cluster 2022-04-22T20:16:27.136732+0000 mgr.y (mgr.34107) 5836 : cluster [DBG] pgmap v7053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:28 smithi115 bash[45495]: audit 2022-04-22T20:16:28.591219+0000 mon.a (mon.0) 9026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:28 smithi115 bash[46934]: cluster 2022-04-22T20:16:27.136732+0000 mgr.y (mgr.34107) 5836 : cluster [DBG] pgmap v7053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:28 smithi115 bash[46934]: audit 2022-04-22T20:16:28.591219+0000 mon.a (mon.0) 9026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:30 smithi121 bash[36902]: cluster 2022-04-22T20:16:29.137825+0000 mgr.y (mgr.34107) 5837 : cluster [DBG] pgmap v7054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:31.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:30 smithi115 bash[45495]: cluster 2022-04-22T20:16:29.137825+0000 mgr.y (mgr.34107) 5837 : cluster [DBG] pgmap v7054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:31.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:30 smithi115 bash[46934]: cluster 2022-04-22T20:16:29.137825+0000 mgr.y (mgr.34107) 5837 : cluster [DBG] pgmap v7054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:32.275Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:32 smithi121 bash[36902]: cluster 2022-04-22T20:16:31.138876+0000 mgr.y (mgr.34107) 5838 : cluster [DBG] pgmap v7055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:32 smithi121 bash[36902]: audit 2022-04-22T20:16:31.941518+0000 mon.a (mon.0) 9027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:32 smithi121 bash[36902]: audit 2022-04-22T20:16:32.158349+0000 mon.a (mon.0) 9028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[45495]: cluster 2022-04-22T20:16:31.138876+0000 mgr.y (mgr.34107) 5838 : cluster [DBG] pgmap v7055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[45495]: audit 2022-04-22T20:16:31.941518+0000 mon.a (mon.0) 9027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[45495]: audit 2022-04-22T20:16:32.158349+0000 mon.a (mon.0) 9028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[46934]: cluster 2022-04-22T20:16:31.138876+0000 mgr.y (mgr.34107) 5838 : cluster [DBG] pgmap v7055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[46934]: audit 2022-04-22T20:16:31.941518+0000 mon.a (mon.0) 9027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:32 smithi115 bash[46934]: audit 2022-04-22T20:16:32.158349+0000 mon.a (mon.0) 9028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:33 smithi121 bash[36902]: audit 2022-04-22T20:16:33.594497+0000 mon.a (mon.0) 9029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:33 smithi115 bash[46934]: audit 2022-04-22T20:16:33.594497+0000 mon.a (mon.0) 9029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:34.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:34.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:33 smithi115 bash[45495]: audit 2022-04-22T20:16:33.594497+0000 mon.a (mon.0) 9029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:34 smithi121 bash[36902]: cluster 2022-04-22T20:16:33.140070+0000 mgr.y (mgr.34107) 5839 : cluster [DBG] pgmap v7056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:34 smithi115 bash[46934]: cluster 2022-04-22T20:16:33.140070+0000 mgr.y (mgr.34107) 5839 : cluster [DBG] pgmap v7056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:34 smithi115 bash[45495]: cluster 2022-04-22T20:16:33.140070+0000 mgr.y (mgr.34107) 5839 : cluster [DBG] pgmap v7056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:36 smithi121 bash[36902]: cluster 2022-04-22T20:16:35.140975+0000 mgr.y (mgr.34107) 5840 : cluster [DBG] pgmap v7057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:36 smithi115 bash[45495]: cluster 2022-04-22T20:16:35.140975+0000 mgr.y (mgr.34107) 5840 : cluster [DBG] pgmap v7057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:36 smithi115 bash[46934]: cluster 2022-04-22T20:16:35.140975+0000 mgr.y (mgr.34107) 5840 : cluster [DBG] pgmap v7057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:37 smithi121 bash[36902]: audit 2022-04-22T20:16:36.944933+0000 mon.a (mon.0) 9030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:37 smithi121 bash[36902]: audit 2022-04-22T20:16:37.161373+0000 mon.a (mon.0) 9031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:37 smithi115 bash[46934]: audit 2022-04-22T20:16:36.944933+0000 mon.a (mon.0) 9030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:37 smithi115 bash[46934]: audit 2022-04-22T20:16:37.161373+0000 mon.a (mon.0) 9031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:37 smithi115 bash[45495]: audit 2022-04-22T20:16:36.944933+0000 mon.a (mon.0) 9030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:37 smithi115 bash[45495]: audit 2022-04-22T20:16:37.161373+0000 mon.a (mon.0) 9031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:38 smithi121 bash[36902]: cluster 2022-04-22T20:16:37.142110+0000 mgr.y (mgr.34107) 5841 : cluster [DBG] pgmap v7058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:38 smithi121 bash[36902]: audit 2022-04-22T20:16:38.597519+0000 mon.a (mon.0) 9032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:38 smithi115 bash[45495]: cluster 2022-04-22T20:16:37.142110+0000 mgr.y (mgr.34107) 5841 : cluster [DBG] pgmap v7058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:38 smithi115 bash[45495]: audit 2022-04-22T20:16:38.597519+0000 mon.a (mon.0) 9032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:38 smithi115 bash[46934]: cluster 2022-04-22T20:16:37.142110+0000 mgr.y (mgr.34107) 5841 : cluster [DBG] pgmap v7058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:38 smithi115 bash[46934]: audit 2022-04-22T20:16:38.597519+0000 mon.a (mon.0) 9032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:40 smithi121 bash[36902]: cluster 2022-04-22T20:16:39.143157+0000 mgr.y (mgr.34107) 5842 : cluster [DBG] pgmap v7059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:40 smithi115 bash[45495]: cluster 2022-04-22T20:16:39.143157+0000 mgr.y (mgr.34107) 5842 : cluster [DBG] pgmap v7059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:40 smithi115 bash[46934]: cluster 2022-04-22T20:16:39.143157+0000 mgr.y (mgr.34107) 5842 : cluster [DBG] pgmap v7059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:41 smithi121 bash[36902]: audit 2022-04-22T20:16:41.948175+0000 mon.a (mon.0) 9033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:42.284 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:41 smithi115 bash[45495]: audit 2022-04-22T20:16:41.948175+0000 mon.a (mon.0) 9033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:42.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:41 smithi115 bash[46934]: audit 2022-04-22T20:16:41.948175+0000 mon.a (mon.0) 9033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:42.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:42.276Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:42 smithi121 bash[36902]: cluster 2022-04-22T20:16:41.144376+0000 mgr.y (mgr.34107) 5843 : cluster [DBG] pgmap v7060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:42 smithi121 bash[36902]: audit 2022-04-22T20:16:42.164203+0000 mon.a (mon.0) 9034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:42 smithi115 bash[45495]: cluster 2022-04-22T20:16:41.144376+0000 mgr.y (mgr.34107) 5843 : cluster [DBG] pgmap v7060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:42 smithi115 bash[45495]: audit 2022-04-22T20:16:42.164203+0000 mon.a (mon.0) 9034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:42 smithi115 bash[46934]: cluster 2022-04-22T20:16:41.144376+0000 mgr.y (mgr.34107) 5843 : cluster [DBG] pgmap v7060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:42 smithi115 bash[46934]: audit 2022-04-22T20:16:42.164203+0000 mon.a (mon.0) 9034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:43 smithi121 bash[36902]: audit 2022-04-22T20:16:43.600684+0000 mon.a (mon.0) 9035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:43 smithi115 bash[45495]: audit 2022-04-22T20:16:43.600684+0000 mon.a (mon.0) 9035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:44.155Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:43 smithi115 bash[46934]: audit 2022-04-22T20:16:43.600684+0000 mon.a (mon.0) 9035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:44 smithi121 bash[36902]: cluster 2022-04-22T20:16:43.145577+0000 mgr.y (mgr.34107) 5844 : cluster [DBG] pgmap v7061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:44 smithi115 bash[46934]: cluster 2022-04-22T20:16:43.145577+0000 mgr.y (mgr.34107) 5844 : cluster [DBG] pgmap v7061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:44 smithi115 bash[45495]: cluster 2022-04-22T20:16:43.145577+0000 mgr.y (mgr.34107) 5844 : cluster [DBG] pgmap v7061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:46 smithi121 bash[36902]: cluster 2022-04-22T20:16:45.146592+0000 mgr.y (mgr.34107) 5845 : cluster [DBG] pgmap v7062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:46 smithi121 bash[36902]: audit 2022-04-22T20:16:46.951874+0000 mon.a (mon.0) 9036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:46 smithi115 bash[45495]: cluster 2022-04-22T20:16:45.146592+0000 mgr.y (mgr.34107) 5845 : cluster [DBG] pgmap v7062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:46 smithi115 bash[45495]: audit 2022-04-22T20:16:46.951874+0000 mon.a (mon.0) 9036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:46 smithi115 bash[46934]: cluster 2022-04-22T20:16:45.146592+0000 mgr.y (mgr.34107) 5845 : cluster [DBG] pgmap v7062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:46 smithi115 bash[46934]: audit 2022-04-22T20:16:46.951874+0000 mon.a (mon.0) 9036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:48.182 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:16:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:48 smithi121 bash[36902]: audit 2022-04-22T20:16:47.167296+0000 mon.a (mon.0) 9037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:47 smithi115 bash[45495]: audit 2022-04-22T20:16:47.167296+0000 mon.a (mon.0) 9037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:47 smithi115 bash[46934]: audit 2022-04-22T20:16:47.167296+0000 mon.a (mon.0) 9037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:48.537 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:16:48.537 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:16:48.537 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 5m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:16:48.537 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:16:48.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:16:48.539 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:16:48.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:16:48.943 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:16:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:49 smithi121 bash[36902]: cluster 2022-04-22T20:16:47.147829+0000 mgr.y (mgr.34107) 5846 : cluster [DBG] pgmap v7063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:49 smithi121 bash[36902]: audit 2022-04-22T20:16:48.603526+0000 mon.a (mon.0) 9038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:49 smithi121 bash[36902]: audit 2022-04-22T20:16:48.932751+0000 mon.a (mon.0) 9039 : audit [DBG] from='client.? 172.21.15.115:0/2751176420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[45495]: cluster 2022-04-22T20:16:47.147829+0000 mgr.y (mgr.34107) 5846 : cluster [DBG] pgmap v7063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[45495]: audit 2022-04-22T20:16:48.603526+0000 mon.a (mon.0) 9038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[45495]: audit 2022-04-22T20:16:48.932751+0000 mon.a (mon.0) 9039 : audit [DBG] from='client.? 172.21.15.115:0/2751176420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[46934]: cluster 2022-04-22T20:16:47.147829+0000 mgr.y (mgr.34107) 5846 : cluster [DBG] pgmap v7063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[46934]: audit 2022-04-22T20:16:48.603526+0000 mon.a (mon.0) 9038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:48 smithi115 bash[46934]: audit 2022-04-22T20:16:48.932751+0000 mon.a (mon.0) 9039 : audit [DBG] from='client.? 172.21.15.115:0/2751176420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:16:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:50 smithi121 bash[36902]: audit 2022-04-22T20:16:48.157376+0000 mgr.y (mgr.34107) 5847 : audit [DBG] from='client.41472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:50 smithi121 bash[36902]: audit 2022-04-22T20:16:48.507833+0000 mgr.y (mgr.34107) 5848 : audit [DBG] from='client.41478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:50.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:50 smithi115 bash[45495]: audit 2022-04-22T20:16:48.157376+0000 mgr.y (mgr.34107) 5847 : audit [DBG] from='client.41472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:50.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:50 smithi115 bash[45495]: audit 2022-04-22T20:16:48.507833+0000 mgr.y (mgr.34107) 5848 : audit [DBG] from='client.41478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:50.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:50 smithi115 bash[46934]: audit 2022-04-22T20:16:48.157376+0000 mgr.y (mgr.34107) 5847 : audit [DBG] from='client.41472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:50.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:50 smithi115 bash[46934]: audit 2022-04-22T20:16:48.507833+0000 mgr.y (mgr.34107) 5848 : audit [DBG] from='client.41478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:16:51.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:51 smithi115 bash[45495]: cluster 2022-04-22T20:16:49.149188+0000 mgr.y (mgr.34107) 5849 : cluster [DBG] pgmap v7064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:51.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:51 smithi115 bash[46934]: cluster 2022-04-22T20:16:49.149188+0000 mgr.y (mgr.34107) 5849 : cluster [DBG] pgmap v7064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:51 smithi121 bash[36902]: cluster 2022-04-22T20:16:49.149188+0000 mgr.y (mgr.34107) 5849 : cluster [DBG] pgmap v7064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:52.285 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:52 smithi115 bash[45495]: audit 2022-04-22T20:16:51.955529+0000 mon.a (mon.0) 9040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:52.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:52 smithi115 bash[46934]: audit 2022-04-22T20:16:51.955529+0000 mon.a (mon.0) 9040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:52.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:52 smithi121 bash[36902]: audit 2022-04-22T20:16:51.955529+0000 mon.a (mon.0) 9040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:52.276Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:53 smithi115 bash[45495]: cluster 2022-04-22T20:16:51.150327+0000 mgr.y (mgr.34107) 5850 : cluster [DBG] pgmap v7065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:53 smithi115 bash[45495]: audit 2022-04-22T20:16:52.170316+0000 mon.a (mon.0) 9041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:53 smithi115 bash[46934]: cluster 2022-04-22T20:16:51.150327+0000 mgr.y (mgr.34107) 5850 : cluster [DBG] pgmap v7065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:53 smithi115 bash[46934]: audit 2022-04-22T20:16:52.170316+0000 mon.a (mon.0) 9041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:53 smithi121 bash[36902]: cluster 2022-04-22T20:16:51.150327+0000 mgr.y (mgr.34107) 5850 : cluster [DBG] pgmap v7065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:53 smithi121 bash[36902]: audit 2022-04-22T20:16:52.170316+0000 mon.a (mon.0) 9041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:54 smithi115 bash[45495]: audit 2022-04-22T20:16:53.606317+0000 mon.a (mon.0) 9042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:16:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:16:54.156Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:16:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:54 smithi115 bash[46934]: audit 2022-04-22T20:16:53.606317+0000 mon.a (mon.0) 9042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:54 smithi121 bash[36902]: audit 2022-04-22T20:16:53.606317+0000 mon.a (mon.0) 9042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:55 smithi115 bash[45495]: cluster 2022-04-22T20:16:53.151429+0000 mgr.y (mgr.34107) 5851 : cluster [DBG] pgmap v7066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:55 smithi115 bash[46934]: cluster 2022-04-22T20:16:53.151429+0000 mgr.y (mgr.34107) 5851 : cluster [DBG] pgmap v7066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:55 smithi121 bash[36902]: cluster 2022-04-22T20:16:53.151429+0000 mgr.y (mgr.34107) 5851 : cluster [DBG] pgmap v7066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:57 smithi115 bash[46934]: cluster 2022-04-22T20:16:55.152460+0000 mgr.y (mgr.34107) 5852 : cluster [DBG] pgmap v7067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:57 smithi115 bash[46934]: audit 2022-04-22T20:16:56.959146+0000 mon.a (mon.0) 9043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:57 smithi115 bash[45495]: cluster 2022-04-22T20:16:55.152460+0000 mgr.y (mgr.34107) 5852 : cluster [DBG] pgmap v7067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:57 smithi115 bash[45495]: audit 2022-04-22T20:16:56.959146+0000 mon.a (mon.0) 9043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:57 smithi121 bash[36902]: cluster 2022-04-22T20:16:55.152460+0000 mgr.y (mgr.34107) 5852 : cluster [DBG] pgmap v7067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:57 smithi121 bash[36902]: audit 2022-04-22T20:16:56.959146+0000 mon.a (mon.0) 9043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:16:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:58 smithi121 bash[36902]: audit 2022-04-22T20:16:57.173129+0000 mon.a (mon.0) 9044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:58 smithi115 bash[46934]: audit 2022-04-22T20:16:57.173129+0000 mon.a (mon.0) 9044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:58 smithi115 bash[45495]: audit 2022-04-22T20:16:57.173129+0000 mon.a (mon.0) 9044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:16:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:59 smithi121 bash[36902]: cluster 2022-04-22T20:16:57.153553+0000 mgr.y (mgr.34107) 5853 : cluster [DBG] pgmap v7068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:16:59 smithi121 bash[36902]: audit 2022-04-22T20:16:58.609224+0000 mon.a (mon.0) 9045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:59 smithi115 bash[45495]: cluster 2022-04-22T20:16:57.153553+0000 mgr.y (mgr.34107) 5853 : cluster [DBG] pgmap v7068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:16:59 smithi115 bash[45495]: audit 2022-04-22T20:16:58.609224+0000 mon.a (mon.0) 9045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:16:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:59 smithi115 bash[46934]: cluster 2022-04-22T20:16:57.153553+0000 mgr.y (mgr.34107) 5853 : cluster [DBG] pgmap v7068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:16:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:16:59 smithi115 bash[46934]: audit 2022-04-22T20:16:58.609224+0000 mon.a (mon.0) 9045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:01 smithi121 bash[36902]: cluster 2022-04-22T20:16:59.154656+0000 mgr.y (mgr.34107) 5854 : cluster [DBG] pgmap v7069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:01.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:01 smithi115 bash[45495]: cluster 2022-04-22T20:16:59.154656+0000 mgr.y (mgr.34107) 5854 : cluster [DBG] pgmap v7069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:01 smithi115 bash[46934]: cluster 2022-04-22T20:16:59.154656+0000 mgr.y (mgr.34107) 5854 : cluster [DBG] pgmap v7069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:02.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:02 smithi115 bash[45495]: audit 2022-04-22T20:17:01.962923+0000 mon.a (mon.0) 9046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:02.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:02 smithi115 bash[46934]: audit 2022-04-22T20:17:01.962923+0000 mon.a (mon.0) 9046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:02 smithi121 bash[36902]: audit 2022-04-22T20:17:01.962923+0000 mon.a (mon.0) 9046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:02.277Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:03 smithi121 bash[36902]: cluster 2022-04-22T20:17:01.155818+0000 mgr.y (mgr.34107) 5855 : cluster [DBG] pgmap v7070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:03 smithi121 bash[36902]: audit 2022-04-22T20:17:02.175994+0000 mon.a (mon.0) 9047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:03 smithi115 bash[45495]: cluster 2022-04-22T20:17:01.155818+0000 mgr.y (mgr.34107) 5855 : cluster [DBG] pgmap v7070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:03 smithi115 bash[45495]: audit 2022-04-22T20:17:02.175994+0000 mon.a (mon.0) 9047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:03 smithi115 bash[46934]: cluster 2022-04-22T20:17:01.155818+0000 mgr.y (mgr.34107) 5855 : cluster [DBG] pgmap v7070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:03 smithi115 bash[46934]: audit 2022-04-22T20:17:02.175994+0000 mon.a (mon.0) 9047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:04 smithi121 bash[36902]: cluster 2022-04-22T20:17:03.156909+0000 mgr.y (mgr.34107) 5856 : cluster [DBG] pgmap v7071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:04 smithi121 bash[36902]: audit 2022-04-22T20:17:03.612096+0000 mon.a (mon.0) 9048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:04.157Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:04 smithi115 bash[45495]: cluster 2022-04-22T20:17:03.156909+0000 mgr.y (mgr.34107) 5856 : cluster [DBG] pgmap v7071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:04 smithi115 bash[45495]: audit 2022-04-22T20:17:03.612096+0000 mon.a (mon.0) 9048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:04 smithi115 bash[46934]: cluster 2022-04-22T20:17:03.156909+0000 mgr.y (mgr.34107) 5856 : cluster [DBG] pgmap v7071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:04 smithi115 bash[46934]: audit 2022-04-22T20:17:03.612096+0000 mon.a (mon.0) 9048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:06 smithi121 bash[36902]: cluster 2022-04-22T20:17:05.157974+0000 mgr.y (mgr.34107) 5857 : cluster [DBG] pgmap v7072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:06 smithi115 bash[45495]: cluster 2022-04-22T20:17:05.157974+0000 mgr.y (mgr.34107) 5857 : cluster [DBG] pgmap v7072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:06 smithi115 bash[46934]: cluster 2022-04-22T20:17:05.157974+0000 mgr.y (mgr.34107) 5857 : cluster [DBG] pgmap v7072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:07 smithi121 bash[36902]: audit 2022-04-22T20:17:06.966465+0000 mon.a (mon.0) 9049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:07 smithi121 bash[36902]: audit 2022-04-22T20:17:07.178952+0000 mon.a (mon.0) 9050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:07 smithi115 bash[46934]: audit 2022-04-22T20:17:06.966465+0000 mon.a (mon.0) 9049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:07 smithi115 bash[46934]: audit 2022-04-22T20:17:07.178952+0000 mon.a (mon.0) 9050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:07 smithi115 bash[45495]: audit 2022-04-22T20:17:06.966465+0000 mon.a (mon.0) 9049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:07 smithi115 bash[45495]: audit 2022-04-22T20:17:07.178952+0000 mon.a (mon.0) 9050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:08 smithi121 bash[36902]: cluster 2022-04-22T20:17:07.159123+0000 mgr.y (mgr.34107) 5858 : cluster [DBG] pgmap v7073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:08 smithi115 bash[46934]: cluster 2022-04-22T20:17:07.159123+0000 mgr.y (mgr.34107) 5858 : cluster [DBG] pgmap v7073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:08 smithi115 bash[45495]: cluster 2022-04-22T20:17:07.159123+0000 mgr.y (mgr.34107) 5858 : cluster [DBG] pgmap v7073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:09 smithi121 bash[36902]: audit 2022-04-22T20:17:08.614981+0000 mon.a (mon.0) 9051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:09.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:09 smithi115 bash[46934]: audit 2022-04-22T20:17:08.614981+0000 mon.a (mon.0) 9051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:09.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:09 smithi115 bash[45495]: audit 2022-04-22T20:17:08.614981+0000 mon.a (mon.0) 9051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:10 smithi121 bash[36902]: cluster 2022-04-22T20:17:09.160238+0000 mgr.y (mgr.34107) 5859 : cluster [DBG] pgmap v7074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:10 smithi115 bash[46934]: cluster 2022-04-22T20:17:09.160238+0000 mgr.y (mgr.34107) 5859 : cluster [DBG] pgmap v7074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:10 smithi115 bash[45495]: cluster 2022-04-22T20:17:09.160238+0000 mgr.y (mgr.34107) 5859 : cluster [DBG] pgmap v7074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:12.278Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:12 smithi121 bash[36902]: cluster 2022-04-22T20:17:11.161268+0000 mgr.y (mgr.34107) 5860 : cluster [DBG] pgmap v7075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:12 smithi121 bash[36902]: audit 2022-04-22T20:17:11.970160+0000 mon.a (mon.0) 9052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:12 smithi121 bash[36902]: audit 2022-04-22T20:17:12.181953+0000 mon.a (mon.0) 9053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[46934]: cluster 2022-04-22T20:17:11.161268+0000 mgr.y (mgr.34107) 5860 : cluster [DBG] pgmap v7075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[46934]: audit 2022-04-22T20:17:11.970160+0000 mon.a (mon.0) 9052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[46934]: audit 2022-04-22T20:17:12.181953+0000 mon.a (mon.0) 9053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[45495]: cluster 2022-04-22T20:17:11.161268+0000 mgr.y (mgr.34107) 5860 : cluster [DBG] pgmap v7075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[45495]: audit 2022-04-22T20:17:11.970160+0000 mon.a (mon.0) 9052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:12 smithi115 bash[45495]: audit 2022-04-22T20:17:12.181953+0000 mon.a (mon.0) 9053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:14.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:13 smithi115 bash[45495]: audit 2022-04-22T20:17:13.617987+0000 mon.a (mon.0) 9054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:14.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:13 smithi115 bash[46934]: audit 2022-04-22T20:17:13.617987+0000 mon.a (mon.0) 9054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:13 smithi121 bash[36902]: audit 2022-04-22T20:17:13.617987+0000 mon.a (mon.0) 9054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:14.157Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:14 smithi121 bash[36902]: cluster 2022-04-22T20:17:13.162403+0000 mgr.y (mgr.34107) 5861 : cluster [DBG] pgmap v7076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:14 smithi115 bash[45495]: cluster 2022-04-22T20:17:13.162403+0000 mgr.y (mgr.34107) 5861 : cluster [DBG] pgmap v7076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:14 smithi115 bash[46934]: cluster 2022-04-22T20:17:13.162403+0000 mgr.y (mgr.34107) 5861 : cluster [DBG] pgmap v7076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:15 smithi121 bash[36902]: audit 2022-04-22T20:17:15.244015+0000 mon.a (mon.0) 9055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:17:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:15 smithi121 bash[36902]: audit 2022-04-22T20:17:15.244578+0000 mon.a (mon.0) 9056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:17:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:15 smithi115 bash[45495]: audit 2022-04-22T20:17:15.244015+0000 mon.a (mon.0) 9055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:17:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:15 smithi115 bash[45495]: audit 2022-04-22T20:17:15.244578+0000 mon.a (mon.0) 9056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:17:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:15 smithi115 bash[46934]: audit 2022-04-22T20:17:15.244015+0000 mon.a (mon.0) 9055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:17:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:15 smithi115 bash[46934]: audit 2022-04-22T20:17:15.244578+0000 mon.a (mon.0) 9056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:17:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:16 smithi121 bash[36902]: cluster 2022-04-22T20:17:15.163517+0000 mgr.y (mgr.34107) 5862 : cluster [DBG] pgmap v7077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:16 smithi115 bash[45495]: cluster 2022-04-22T20:17:15.163517+0000 mgr.y (mgr.34107) 5862 : cluster [DBG] pgmap v7077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:17.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:16 smithi115 bash[46934]: cluster 2022-04-22T20:17:15.163517+0000 mgr.y (mgr.34107) 5862 : cluster [DBG] pgmap v7077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:17 smithi121 bash[36902]: audit 2022-04-22T20:17:16.974743+0000 mon.a (mon.0) 9057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:17 smithi121 bash[36902]: audit 2022-04-22T20:17:17.184904+0000 mon.a (mon.0) 9058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:17 smithi115 bash[45495]: audit 2022-04-22T20:17:16.974743+0000 mon.a (mon.0) 9057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:18.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:17 smithi115 bash[45495]: audit 2022-04-22T20:17:17.184904+0000 mon.a (mon.0) 9058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:17 smithi115 bash[46934]: audit 2022-04-22T20:17:16.974743+0000 mon.a (mon.0) 9057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:18.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:17 smithi115 bash[46934]: audit 2022-04-22T20:17:17.184904+0000 mon.a (mon.0) 9058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:18 smithi121 bash[36902]: cluster 2022-04-22T20:17:17.164620+0000 mgr.y (mgr.34107) 5863 : cluster [DBG] pgmap v7078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:18 smithi121 bash[36902]: audit 2022-04-22T20:17:18.621075+0000 mon.a (mon.0) 9059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:18 smithi115 bash[45495]: cluster 2022-04-22T20:17:17.164620+0000 mgr.y (mgr.34107) 5863 : cluster [DBG] pgmap v7078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:18 smithi115 bash[45495]: audit 2022-04-22T20:17:18.621075+0000 mon.a (mon.0) 9059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:18 smithi115 bash[46934]: cluster 2022-04-22T20:17:17.164620+0000 mgr.y (mgr.34107) 5863 : cluster [DBG] pgmap v7078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:18 smithi115 bash[46934]: audit 2022-04-22T20:17:18.621075+0000 mon.a (mon.0) 9059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:19.323 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:17:19.674 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 2h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 2h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:17:19.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:17:19.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:17:19.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:17:19.676 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:17:20.077 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:17:20.077 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:17:20.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:17:20.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:20.078 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:17:20.079 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:17:20.079 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:17:20.079 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:17:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:20 smithi121 bash[36902]: cluster 2022-04-22T20:17:19.165622+0000 mgr.y (mgr.34107) 5864 : cluster [DBG] pgmap v7079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:20 smithi121 bash[36902]: audit 2022-04-22T20:17:19.297752+0000 mgr.y (mgr.34107) 5865 : audit [DBG] from='client.41490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:20 smithi121 bash[36902]: audit 2022-04-22T20:17:19.640636+0000 mgr.y (mgr.34107) 5866 : audit [DBG] from='client.41496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:20 smithi121 bash[36902]: audit 2022-04-22T20:17:20.068508+0000 mon.a (mon.0) 9060 : audit [DBG] from='client.? 172.21.15.115:0/4128122265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[45495]: cluster 2022-04-22T20:17:19.165622+0000 mgr.y (mgr.34107) 5864 : cluster [DBG] pgmap v7079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[45495]: audit 2022-04-22T20:17:19.297752+0000 mgr.y (mgr.34107) 5865 : audit [DBG] from='client.41490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[45495]: audit 2022-04-22T20:17:19.640636+0000 mgr.y (mgr.34107) 5866 : audit [DBG] from='client.41496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[45495]: audit 2022-04-22T20:17:20.068508+0000 mon.a (mon.0) 9060 : audit [DBG] from='client.? 172.21.15.115:0/4128122265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[46934]: cluster 2022-04-22T20:17:19.165622+0000 mgr.y (mgr.34107) 5864 : cluster [DBG] pgmap v7079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[46934]: audit 2022-04-22T20:17:19.297752+0000 mgr.y (mgr.34107) 5865 : audit [DBG] from='client.41490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[46934]: audit 2022-04-22T20:17:19.640636+0000 mgr.y (mgr.34107) 5866 : audit [DBG] from='client.41496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:21.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:20 smithi115 bash[46934]: audit 2022-04-22T20:17:20.068508+0000 mon.a (mon.0) 9060 : audit [DBG] from='client.? 172.21.15.115:0/4128122265' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:22.277Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:22.936 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: cluster 2022-04-22T20:17:21.166665+0000 mgr.y (mgr.34107) 5867 : cluster [DBG] pgmap v7080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: audit 2022-04-22T20:17:21.978073+0000 mon.a (mon.0) 9061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: audit 2022-04-22T20:17:22.187983+0000 mon.a (mon.0) 9062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: audit 2022-04-22T20:17:22.622625+0000 mon.a (mon.0) 9063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: audit 2022-04-22T20:17:22.625517+0000 mon.a (mon.0) 9064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[45495]: audit 2022-04-22T20:17:22.627217+0000 mon.a (mon.0) 9065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: cluster 2022-04-22T20:17:21.166665+0000 mgr.y (mgr.34107) 5867 : cluster [DBG] pgmap v7080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:22.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: audit 2022-04-22T20:17:21.978073+0000 mon.a (mon.0) 9061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:22.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: audit 2022-04-22T20:17:22.187983+0000 mon.a (mon.0) 9062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:22.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: audit 2022-04-22T20:17:22.622625+0000 mon.a (mon.0) 9063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:17:22.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: audit 2022-04-22T20:17:22.625517+0000 mon.a (mon.0) 9064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:17:22.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:22 smithi115 bash[46934]: audit 2022-04-22T20:17:22.627217+0000 mon.a (mon.0) 9065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:17:22.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: cluster 2022-04-22T20:17:21.166665+0000 mgr.y (mgr.34107) 5867 : cluster [DBG] pgmap v7080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:22.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: audit 2022-04-22T20:17:21.978073+0000 mon.a (mon.0) 9061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:22.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: audit 2022-04-22T20:17:22.187983+0000 mon.a (mon.0) 9062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:22.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: audit 2022-04-22T20:17:22.622625+0000 mon.a (mon.0) 9063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:17:22.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: audit 2022-04-22T20:17:22.625517+0000 mon.a (mon.0) 9064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:17:22.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:22 smithi121 bash[36902]: audit 2022-04-22T20:17:22.627217+0000 mon.a (mon.0) 9065 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:17:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:23 smithi121 bash[36902]: audit 2022-04-22T20:17:22.982603+0000 mon.a (mon.0) 9066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:17:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:23 smithi121 bash[36902]: audit 2022-04-22T20:17:23.624188+0000 mon.a (mon.0) 9067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:23 smithi115 bash[45495]: audit 2022-04-22T20:17:22.982603+0000 mon.a (mon.0) 9066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:17:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:23 smithi115 bash[45495]: audit 2022-04-22T20:17:23.624188+0000 mon.a (mon.0) 9067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:24.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:24.159Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:23 smithi115 bash[46934]: audit 2022-04-22T20:17:22.982603+0000 mon.a (mon.0) 9066 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:17:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:23 smithi115 bash[46934]: audit 2022-04-22T20:17:23.624188+0000 mon.a (mon.0) 9067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:25 smithi121 bash[36902]: cluster 2022-04-22T20:17:23.167793+0000 mgr.y (mgr.34107) 5868 : cluster [DBG] pgmap v7081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:24 smithi115 bash[45495]: cluster 2022-04-22T20:17:23.167793+0000 mgr.y (mgr.34107) 5868 : cluster [DBG] pgmap v7081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:24 smithi115 bash[46934]: cluster 2022-04-22T20:17:23.167793+0000 mgr.y (mgr.34107) 5868 : cluster [DBG] pgmap v7081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:27 smithi121 bash[36902]: cluster 2022-04-22T20:17:25.168764+0000 mgr.y (mgr.34107) 5869 : cluster [DBG] pgmap v7082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:27 smithi121 bash[36902]: audit 2022-04-22T20:17:26.981925+0000 mon.a (mon.0) 9068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:26 smithi115 bash[45495]: cluster 2022-04-22T20:17:25.168764+0000 mgr.y (mgr.34107) 5869 : cluster [DBG] pgmap v7082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:27.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:26 smithi115 bash[45495]: audit 2022-04-22T20:17:26.981925+0000 mon.a (mon.0) 9068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:27.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:26 smithi115 bash[46934]: cluster 2022-04-22T20:17:25.168764+0000 mgr.y (mgr.34107) 5869 : cluster [DBG] pgmap v7082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:27.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:26 smithi115 bash[46934]: audit 2022-04-22T20:17:26.981925+0000 mon.a (mon.0) 9068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:28 smithi115 bash[45495]: audit 2022-04-22T20:17:27.191157+0000 mon.a (mon.0) 9069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:28 smithi115 bash[46934]: audit 2022-04-22T20:17:27.191157+0000 mon.a (mon.0) 9069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:28 smithi121 bash[36902]: audit 2022-04-22T20:17:27.191157+0000 mon.a (mon.0) 9069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:29 smithi115 bash[45495]: cluster 2022-04-22T20:17:27.169773+0000 mgr.y (mgr.34107) 5870 : cluster [DBG] pgmap v7083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:29 smithi115 bash[45495]: audit 2022-04-22T20:17:28.627395+0000 mon.a (mon.0) 9070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:29 smithi115 bash[46934]: cluster 2022-04-22T20:17:27.169773+0000 mgr.y (mgr.34107) 5870 : cluster [DBG] pgmap v7083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:29.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:29 smithi115 bash[46934]: audit 2022-04-22T20:17:28.627395+0000 mon.a (mon.0) 9070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:29 smithi121 bash[36902]: cluster 2022-04-22T20:17:27.169773+0000 mgr.y (mgr.34107) 5870 : cluster [DBG] pgmap v7083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:29 smithi121 bash[36902]: audit 2022-04-22T20:17:28.627395+0000 mon.a (mon.0) 9070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:31 smithi115 bash[45495]: cluster 2022-04-22T20:17:29.170825+0000 mgr.y (mgr.34107) 5871 : cluster [DBG] pgmap v7084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:31 smithi115 bash[46934]: cluster 2022-04-22T20:17:29.170825+0000 mgr.y (mgr.34107) 5871 : cluster [DBG] pgmap v7084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:31 smithi121 bash[36902]: cluster 2022-04-22T20:17:29.170825+0000 mgr.y (mgr.34107) 5871 : cluster [DBG] pgmap v7084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:32.287 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:32 smithi115 bash[45495]: audit 2022-04-22T20:17:31.985688+0000 mon.a (mon.0) 9071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:32.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:32 smithi115 bash[46934]: audit 2022-04-22T20:17:31.985688+0000 mon.a (mon.0) 9071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:32 smithi121 bash[36902]: audit 2022-04-22T20:17:31.985688+0000 mon.a (mon.0) 9071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:32.277Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:33 smithi115 bash[45495]: cluster 2022-04-22T20:17:31.171885+0000 mgr.y (mgr.34107) 5872 : cluster [DBG] pgmap v7085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:33 smithi115 bash[45495]: audit 2022-04-22T20:17:32.194504+0000 mon.a (mon.0) 9072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:33 smithi115 bash[46934]: cluster 2022-04-22T20:17:31.171885+0000 mgr.y (mgr.34107) 5872 : cluster [DBG] pgmap v7085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:33 smithi115 bash[46934]: audit 2022-04-22T20:17:32.194504+0000 mon.a (mon.0) 9072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:33 smithi121 bash[36902]: cluster 2022-04-22T20:17:31.171885+0000 mgr.y (mgr.34107) 5872 : cluster [DBG] pgmap v7085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:33 smithi121 bash[36902]: audit 2022-04-22T20:17:32.194504+0000 mon.a (mon.0) 9072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:34 smithi115 bash[45495]: audit 2022-04-22T20:17:33.630554+0000 mon.a (mon.0) 9073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:34.159Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:34.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:34 smithi115 bash[46934]: audit 2022-04-22T20:17:33.630554+0000 mon.a (mon.0) 9073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:34 smithi121 bash[36902]: audit 2022-04-22T20:17:33.630554+0000 mon.a (mon.0) 9073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:35 smithi115 bash[45495]: cluster 2022-04-22T20:17:33.173011+0000 mgr.y (mgr.34107) 5873 : cluster [DBG] pgmap v7086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:35 smithi115 bash[46934]: cluster 2022-04-22T20:17:33.173011+0000 mgr.y (mgr.34107) 5873 : cluster [DBG] pgmap v7086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:35 smithi121 bash[36902]: cluster 2022-04-22T20:17:33.173011+0000 mgr.y (mgr.34107) 5873 : cluster [DBG] pgmap v7086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:37 smithi121 bash[36902]: cluster 2022-04-22T20:17:35.174037+0000 mgr.y (mgr.34107) 5874 : cluster [DBG] pgmap v7087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:37 smithi121 bash[36902]: audit 2022-04-22T20:17:36.989480+0000 mon.a (mon.0) 9074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:37 smithi115 bash[45495]: cluster 2022-04-22T20:17:35.174037+0000 mgr.y (mgr.34107) 5874 : cluster [DBG] pgmap v7087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:37 smithi115 bash[45495]: audit 2022-04-22T20:17:36.989480+0000 mon.a (mon.0) 9074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:37 smithi115 bash[46934]: cluster 2022-04-22T20:17:35.174037+0000 mgr.y (mgr.34107) 5874 : cluster [DBG] pgmap v7087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:37 smithi115 bash[46934]: audit 2022-04-22T20:17:36.989480+0000 mon.a (mon.0) 9074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:38 smithi121 bash[36902]: audit 2022-04-22T20:17:37.197466+0000 mon.a (mon.0) 9075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:38 smithi115 bash[45495]: audit 2022-04-22T20:17:37.197466+0000 mon.a (mon.0) 9075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:38 smithi115 bash[46934]: audit 2022-04-22T20:17:37.197466+0000 mon.a (mon.0) 9075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:39 smithi121 bash[36902]: cluster 2022-04-22T20:17:37.175014+0000 mgr.y (mgr.34107) 5875 : cluster [DBG] pgmap v7088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:39 smithi121 bash[36902]: audit 2022-04-22T20:17:38.633524+0000 mon.a (mon.0) 9076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:39.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:39 smithi115 bash[45495]: cluster 2022-04-22T20:17:37.175014+0000 mgr.y (mgr.34107) 5875 : cluster [DBG] pgmap v7088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:39.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:39 smithi115 bash[45495]: audit 2022-04-22T20:17:38.633524+0000 mon.a (mon.0) 9076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:39 smithi115 bash[46934]: cluster 2022-04-22T20:17:37.175014+0000 mgr.y (mgr.34107) 5875 : cluster [DBG] pgmap v7088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:39.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:39 smithi115 bash[46934]: audit 2022-04-22T20:17:38.633524+0000 mon.a (mon.0) 9076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:41 smithi121 bash[36902]: cluster 2022-04-22T20:17:39.176126+0000 mgr.y (mgr.34107) 5876 : cluster [DBG] pgmap v7089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:41 smithi115 bash[45495]: cluster 2022-04-22T20:17:39.176126+0000 mgr.y (mgr.34107) 5876 : cluster [DBG] pgmap v7089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:41 smithi115 bash[46934]: cluster 2022-04-22T20:17:39.176126+0000 mgr.y (mgr.34107) 5876 : cluster [DBG] pgmap v7089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:42 smithi121 bash[36902]: audit 2022-04-22T20:17:41.993255+0000 mon.a (mon.0) 9077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:42.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:42 smithi115 bash[46934]: audit 2022-04-22T20:17:41.993255+0000 mon.a (mon.0) 9077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:42.279Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:42 smithi115 bash[45495]: audit 2022-04-22T20:17:41.993255+0000 mon.a (mon.0) 9077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:43 smithi121 bash[36902]: cluster 2022-04-22T20:17:41.177232+0000 mgr.y (mgr.34107) 5877 : cluster [DBG] pgmap v7090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:43 smithi121 bash[36902]: audit 2022-04-22T20:17:42.200464+0000 mon.a (mon.0) 9078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:43 smithi115 bash[45495]: cluster 2022-04-22T20:17:41.177232+0000 mgr.y (mgr.34107) 5877 : cluster [DBG] pgmap v7090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:43 smithi115 bash[45495]: audit 2022-04-22T20:17:42.200464+0000 mon.a (mon.0) 9078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:43 smithi115 bash[46934]: cluster 2022-04-22T20:17:41.177232+0000 mgr.y (mgr.34107) 5877 : cluster [DBG] pgmap v7090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:43 smithi115 bash[46934]: audit 2022-04-22T20:17:42.200464+0000 mon.a (mon.0) 9078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:44 smithi121 bash[36902]: audit 2022-04-22T20:17:43.636741+0000 mon.a (mon.0) 9079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:44 smithi115 bash[45495]: audit 2022-04-22T20:17:43.636741+0000 mon.a (mon.0) 9079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:44.159Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:44 smithi115 bash[46934]: audit 2022-04-22T20:17:43.636741+0000 mon.a (mon.0) 9079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:45 smithi121 bash[36902]: cluster 2022-04-22T20:17:43.178367+0000 mgr.y (mgr.34107) 5878 : cluster [DBG] pgmap v7091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:45 smithi115 bash[45495]: cluster 2022-04-22T20:17:43.178367+0000 mgr.y (mgr.34107) 5878 : cluster [DBG] pgmap v7091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:45 smithi115 bash[46934]: cluster 2022-04-22T20:17:43.178367+0000 mgr.y (mgr.34107) 5878 : cluster [DBG] pgmap v7091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:46 smithi121 bash[36902]: cluster 2022-04-22T20:17:45.179460+0000 mgr.y (mgr.34107) 5879 : cluster [DBG] pgmap v7092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:46.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:46 smithi115 bash[45495]: cluster 2022-04-22T20:17:45.179460+0000 mgr.y (mgr.34107) 5879 : cluster [DBG] pgmap v7092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:46 smithi115 bash[46934]: cluster 2022-04-22T20:17:45.179460+0000 mgr.y (mgr.34107) 5879 : cluster [DBG] pgmap v7092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:47 smithi121 bash[36902]: audit 2022-04-22T20:17:46.997075+0000 mon.a (mon.0) 9080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:47 smithi115 bash[45495]: audit 2022-04-22T20:17:46.997075+0000 mon.a (mon.0) 9080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:47 smithi115 bash[46934]: audit 2022-04-22T20:17:46.997075+0000 mon.a (mon.0) 9080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:48 smithi121 bash[36902]: cluster 2022-04-22T20:17:47.180571+0000 mgr.y (mgr.34107) 5880 : cluster [DBG] pgmap v7093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:48 smithi121 bash[36902]: audit 2022-04-22T20:17:47.203529+0000 mon.a (mon.0) 9081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:48 smithi115 bash[45495]: cluster 2022-04-22T20:17:47.180571+0000 mgr.y (mgr.34107) 5880 : cluster [DBG] pgmap v7093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:48 smithi115 bash[45495]: audit 2022-04-22T20:17:47.203529+0000 mon.a (mon.0) 9081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:48 smithi115 bash[46934]: cluster 2022-04-22T20:17:47.180571+0000 mgr.y (mgr.34107) 5880 : cluster [DBG] pgmap v7093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:48 smithi115 bash[46934]: audit 2022-04-22T20:17:47.203529+0000 mon.a (mon.0) 9081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:49 smithi121 bash[36902]: audit 2022-04-22T20:17:48.639892+0000 mon.a (mon.0) 9082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:49 smithi115 bash[45495]: audit 2022-04-22T20:17:48.639892+0000 mon.a (mon.0) 9082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:49 smithi115 bash[46934]: audit 2022-04-22T20:17:48.639892+0000 mon.a (mon.0) 9082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:50.446 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:17:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:50 smithi121 bash[36902]: cluster 2022-04-22T20:17:49.181655+0000 mgr.y (mgr.34107) 5881 : cluster [DBG] pgmap v7094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:50.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:50 smithi115 bash[45495]: cluster 2022-04-22T20:17:49.181655+0000 mgr.y (mgr.34107) 5881 : cluster [DBG] pgmap v7094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:50 smithi115 bash[46934]: cluster 2022-04-22T20:17:49.181655+0000 mgr.y (mgr.34107) 5881 : cluster [DBG] pgmap v7094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:50.787 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:17:50.787 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:17:50.787 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 6m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:17:50.787 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 6m ago 2h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 5m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 2h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:17:50.788 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:17:50.789 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:17:51.189 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:17:51.189 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:17:51.189 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:17:51.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:51.189 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:17:51.190 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:17:51.191 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:17:51.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:17:51.191 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:17:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:51 smithi121 bash[36902]: audit 2022-04-22T20:17:50.420859+0000 mgr.y (mgr.34107) 5882 : audit [DBG] from='client.41508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:51 smithi121 bash[36902]: audit 2022-04-22T20:17:50.758758+0000 mgr.y (mgr.34107) 5883 : audit [DBG] from='client.41514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:51.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:51 smithi115 bash[45495]: audit 2022-04-22T20:17:50.420859+0000 mgr.y (mgr.34107) 5882 : audit [DBG] from='client.41508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:51.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:51 smithi115 bash[45495]: audit 2022-04-22T20:17:50.758758+0000 mgr.y (mgr.34107) 5883 : audit [DBG] from='client.41514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:51.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:51 smithi115 bash[46934]: audit 2022-04-22T20:17:50.420859+0000 mgr.y (mgr.34107) 5882 : audit [DBG] from='client.41508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:51.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:51 smithi115 bash[46934]: audit 2022-04-22T20:17:50.758758+0000 mgr.y (mgr.34107) 5883 : audit [DBG] from='client.41514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:17:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:52 smithi121 bash[36902]: audit 2022-04-22T20:17:51.180498+0000 mon.c (mon.1) 234 : audit [DBG] from='client.? 172.21.15.115:0/2551316653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:52 smithi121 bash[36902]: cluster 2022-04-22T20:17:51.182715+0000 mgr.y (mgr.34107) 5884 : cluster [DBG] pgmap v7095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:52 smithi121 bash[36902]: audit 2022-04-22T20:17:52.000760+0000 mon.a (mon.0) 9083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:52.279Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[45495]: audit 2022-04-22T20:17:51.180498+0000 mon.c (mon.1) 234 : audit [DBG] from='client.? 172.21.15.115:0/2551316653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[45495]: cluster 2022-04-22T20:17:51.182715+0000 mgr.y (mgr.34107) 5884 : cluster [DBG] pgmap v7095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[45495]: audit 2022-04-22T20:17:52.000760+0000 mon.a (mon.0) 9083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[46934]: audit 2022-04-22T20:17:51.180498+0000 mon.c (mon.1) 234 : audit [DBG] from='client.? 172.21.15.115:0/2551316653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:17:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[46934]: cluster 2022-04-22T20:17:51.182715+0000 mgr.y (mgr.34107) 5884 : cluster [DBG] pgmap v7095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:52 smithi115 bash[46934]: audit 2022-04-22T20:17:52.000760+0000 mon.a (mon.0) 9083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:53 smithi121 bash[36902]: audit 2022-04-22T20:17:52.206651+0000 mon.a (mon.0) 9084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:53 smithi115 bash[45495]: audit 2022-04-22T20:17:52.206651+0000 mon.a (mon.0) 9084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:53 smithi115 bash[46934]: audit 2022-04-22T20:17:52.206651+0000 mon.a (mon.0) 9084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:54 smithi121 bash[36902]: cluster 2022-04-22T20:17:53.183913+0000 mgr.y (mgr.34107) 5885 : cluster [DBG] pgmap v7096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:54 smithi121 bash[36902]: audit 2022-04-22T20:17:53.642981+0000 mon.a (mon.0) 9085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:17:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:17:54.158Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:17:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:54 smithi115 bash[45495]: cluster 2022-04-22T20:17:53.183913+0000 mgr.y (mgr.34107) 5885 : cluster [DBG] pgmap v7096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:54 smithi115 bash[45495]: audit 2022-04-22T20:17:53.642981+0000 mon.a (mon.0) 9085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:54 smithi115 bash[46934]: cluster 2022-04-22T20:17:53.183913+0000 mgr.y (mgr.34107) 5885 : cluster [DBG] pgmap v7096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:54 smithi115 bash[46934]: audit 2022-04-22T20:17:53.642981+0000 mon.a (mon.0) 9085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:56 smithi121 bash[36902]: cluster 2022-04-22T20:17:55.184974+0000 mgr.y (mgr.34107) 5886 : cluster [DBG] pgmap v7097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:56 smithi115 bash[46934]: cluster 2022-04-22T20:17:55.184974+0000 mgr.y (mgr.34107) 5886 : cluster [DBG] pgmap v7097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:56 smithi115 bash[45495]: cluster 2022-04-22T20:17:55.184974+0000 mgr.y (mgr.34107) 5886 : cluster [DBG] pgmap v7097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:57 smithi121 bash[36902]: audit 2022-04-22T20:17:57.004389+0000 mon.a (mon.0) 9086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:57 smithi121 bash[36902]: audit 2022-04-22T20:17:57.209817+0000 mon.a (mon.0) 9087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:57 smithi115 bash[46934]: audit 2022-04-22T20:17:57.004389+0000 mon.a (mon.0) 9086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:57 smithi115 bash[46934]: audit 2022-04-22T20:17:57.209817+0000 mon.a (mon.0) 9087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:57.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:57 smithi115 bash[45495]: audit 2022-04-22T20:17:57.004389+0000 mon.a (mon.0) 9086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:17:57.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:57 smithi115 bash[45495]: audit 2022-04-22T20:17:57.209817+0000 mon.a (mon.0) 9087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:17:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:58 smithi121 bash[36902]: cluster 2022-04-22T20:17:57.185975+0000 mgr.y (mgr.34107) 5887 : cluster [DBG] pgmap v7098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:58 smithi115 bash[46934]: cluster 2022-04-22T20:17:57.185975+0000 mgr.y (mgr.34107) 5887 : cluster [DBG] pgmap v7098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:58 smithi115 bash[45495]: cluster 2022-04-22T20:17:57.185975+0000 mgr.y (mgr.34107) 5887 : cluster [DBG] pgmap v7098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:17:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:17:59 smithi121 bash[36902]: audit 2022-04-22T20:17:58.646202+0000 mon.a (mon.0) 9088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:17:59 smithi115 bash[45495]: audit 2022-04-22T20:17:58.646202+0000 mon.a (mon.0) 9088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:17:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:17:59 smithi115 bash[46934]: audit 2022-04-22T20:17:58.646202+0000 mon.a (mon.0) 9088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:00 smithi121 bash[36902]: cluster 2022-04-22T20:17:59.187097+0000 mgr.y (mgr.34107) 5888 : cluster [DBG] pgmap v7099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:00 smithi115 bash[45495]: cluster 2022-04-22T20:17:59.187097+0000 mgr.y (mgr.34107) 5888 : cluster [DBG] pgmap v7099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:00 smithi115 bash[46934]: cluster 2022-04-22T20:17:59.187097+0000 mgr.y (mgr.34107) 5888 : cluster [DBG] pgmap v7099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:02.279Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:02 smithi121 bash[36902]: cluster 2022-04-22T20:18:01.188241+0000 mgr.y (mgr.34107) 5889 : cluster [DBG] pgmap v7100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:02 smithi121 bash[36902]: audit 2022-04-22T20:18:02.008256+0000 mon.a (mon.0) 9089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:02 smithi121 bash[36902]: audit 2022-04-22T20:18:02.212686+0000 mon.a (mon.0) 9090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[45495]: cluster 2022-04-22T20:18:01.188241+0000 mgr.y (mgr.34107) 5889 : cluster [DBG] pgmap v7100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[45495]: audit 2022-04-22T20:18:02.008256+0000 mon.a (mon.0) 9089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[45495]: audit 2022-04-22T20:18:02.212686+0000 mon.a (mon.0) 9090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[46934]: cluster 2022-04-22T20:18:01.188241+0000 mgr.y (mgr.34107) 5889 : cluster [DBG] pgmap v7100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[46934]: audit 2022-04-22T20:18:02.008256+0000 mon.a (mon.0) 9089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:02 smithi115 bash[46934]: audit 2022-04-22T20:18:02.212686+0000 mon.a (mon.0) 9090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:04.168 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:03 smithi115 bash[45495]: audit 2022-04-22T20:18:03.649432+0000 mon.a (mon.0) 9091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:04.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:03 smithi115 bash[46934]: audit 2022-04-22T20:18:03.649432+0000 mon.a (mon.0) 9091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:04.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:03 smithi121 bash[36902]: audit 2022-04-22T20:18:03.649432+0000 mon.a (mon.0) 9091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:04.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:04 smithi121 bash[36902]: cluster 2022-04-22T20:18:03.189307+0000 mgr.y (mgr.34107) 5890 : cluster [DBG] pgmap v7101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:04 smithi115 bash[45495]: cluster 2022-04-22T20:18:03.189307+0000 mgr.y (mgr.34107) 5890 : cluster [DBG] pgmap v7101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:04 smithi115 bash[46934]: cluster 2022-04-22T20:18:03.189307+0000 mgr.y (mgr.34107) 5890 : cluster [DBG] pgmap v7101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:06 smithi121 bash[36902]: cluster 2022-04-22T20:18:05.190350+0000 mgr.y (mgr.34107) 5891 : cluster [DBG] pgmap v7102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:06 smithi115 bash[45495]: cluster 2022-04-22T20:18:05.190350+0000 mgr.y (mgr.34107) 5891 : cluster [DBG] pgmap v7102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:06 smithi115 bash[46934]: cluster 2022-04-22T20:18:05.190350+0000 mgr.y (mgr.34107) 5891 : cluster [DBG] pgmap v7102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:07 smithi121 bash[36902]: audit 2022-04-22T20:18:07.012097+0000 mon.a (mon.0) 9092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:07 smithi121 bash[36902]: audit 2022-04-22T20:18:07.215858+0000 mon.a (mon.0) 9093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:07 smithi115 bash[45495]: audit 2022-04-22T20:18:07.012097+0000 mon.a (mon.0) 9092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:07 smithi115 bash[45495]: audit 2022-04-22T20:18:07.215858+0000 mon.a (mon.0) 9093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:07 smithi115 bash[46934]: audit 2022-04-22T20:18:07.012097+0000 mon.a (mon.0) 9092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:08.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:07 smithi115 bash[46934]: audit 2022-04-22T20:18:07.215858+0000 mon.a (mon.0) 9093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:08 smithi121 bash[36902]: cluster 2022-04-22T20:18:07.191358+0000 mgr.y (mgr.34107) 5892 : cluster [DBG] pgmap v7103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:09.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:08 smithi121 bash[36902]: audit 2022-04-22T20:18:08.652603+0000 mon.a (mon.0) 9094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:09.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:08 smithi115 bash[45495]: cluster 2022-04-22T20:18:07.191358+0000 mgr.y (mgr.34107) 5892 : cluster [DBG] pgmap v7103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:09.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:08 smithi115 bash[45495]: audit 2022-04-22T20:18:08.652603+0000 mon.a (mon.0) 9094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:09.434 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:08 smithi115 bash[46934]: cluster 2022-04-22T20:18:07.191358+0000 mgr.y (mgr.34107) 5892 : cluster [DBG] pgmap v7103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:09.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:08 smithi115 bash[46934]: audit 2022-04-22T20:18:08.652603+0000 mon.a (mon.0) 9094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:10 smithi121 bash[36902]: cluster 2022-04-22T20:18:09.192424+0000 mgr.y (mgr.34107) 5893 : cluster [DBG] pgmap v7104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:10 smithi115 bash[45495]: cluster 2022-04-22T20:18:09.192424+0000 mgr.y (mgr.34107) 5893 : cluster [DBG] pgmap v7104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:10 smithi115 bash[46934]: cluster 2022-04-22T20:18:09.192424+0000 mgr.y (mgr.34107) 5893 : cluster [DBG] pgmap v7104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:12.280Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:12 smithi121 bash[36902]: cluster 2022-04-22T20:18:11.193592+0000 mgr.y (mgr.34107) 5894 : cluster [DBG] pgmap v7105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:12 smithi121 bash[36902]: audit 2022-04-22T20:18:12.015742+0000 mon.a (mon.0) 9095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:13.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:12 smithi121 bash[36902]: audit 2022-04-22T20:18:12.218727+0000 mon.a (mon.0) 9096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[45495]: cluster 2022-04-22T20:18:11.193592+0000 mgr.y (mgr.34107) 5894 : cluster [DBG] pgmap v7105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[45495]: audit 2022-04-22T20:18:12.015742+0000 mon.a (mon.0) 9095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[45495]: audit 2022-04-22T20:18:12.218727+0000 mon.a (mon.0) 9096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[46934]: cluster 2022-04-22T20:18:11.193592+0000 mgr.y (mgr.34107) 5894 : cluster [DBG] pgmap v7105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:13.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[46934]: audit 2022-04-22T20:18:12.015742+0000 mon.a (mon.0) 9095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:13.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:12 smithi115 bash[46934]: audit 2022-04-22T20:18:12.218727+0000 mon.a (mon.0) 9096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:13 smithi121 bash[36902]: audit 2022-04-22T20:18:13.655742+0000 mon.a (mon.0) 9097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:14.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:13 smithi115 bash[45495]: audit 2022-04-22T20:18:13.655742+0000 mon.a (mon.0) 9097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:14.160Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:14.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:13 smithi115 bash[46934]: audit 2022-04-22T20:18:13.655742+0000 mon.a (mon.0) 9097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:14 smithi121 bash[36902]: cluster 2022-04-22T20:18:13.194594+0000 mgr.y (mgr.34107) 5895 : cluster [DBG] pgmap v7106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:14 smithi115 bash[45495]: cluster 2022-04-22T20:18:13.194594+0000 mgr.y (mgr.34107) 5895 : cluster [DBG] pgmap v7106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:15.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:14 smithi115 bash[46934]: cluster 2022-04-22T20:18:13.194594+0000 mgr.y (mgr.34107) 5895 : cluster [DBG] pgmap v7106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:15 smithi121 bash[36902]: audit 2022-04-22T20:18:15.254717+0000 mon.a (mon.0) 9098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:18:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:15 smithi121 bash[36902]: audit 2022-04-22T20:18:15.254948+0000 mon.a (mon.0) 9099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:18:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:15 smithi115 bash[45495]: audit 2022-04-22T20:18:15.254717+0000 mon.a (mon.0) 9098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:18:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:15 smithi115 bash[45495]: audit 2022-04-22T20:18:15.254948+0000 mon.a (mon.0) 9099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:18:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:15 smithi115 bash[46934]: audit 2022-04-22T20:18:15.254717+0000 mon.a (mon.0) 9098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:18:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:15 smithi115 bash[46934]: audit 2022-04-22T20:18:15.254948+0000 mon.a (mon.0) 9099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:18:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:16 smithi121 bash[36902]: cluster 2022-04-22T20:18:15.195699+0000 mgr.y (mgr.34107) 5896 : cluster [DBG] pgmap v7107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:16 smithi115 bash[45495]: cluster 2022-04-22T20:18:15.195699+0000 mgr.y (mgr.34107) 5896 : cluster [DBG] pgmap v7107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:16 smithi115 bash[46934]: cluster 2022-04-22T20:18:15.195699+0000 mgr.y (mgr.34107) 5896 : cluster [DBG] pgmap v7107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:17 smithi121 bash[36902]: audit 2022-04-22T20:18:17.019284+0000 mon.a (mon.0) 9100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:17 smithi121 bash[36902]: audit 2022-04-22T20:18:17.221917+0000 mon.a (mon.0) 9101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:17 smithi115 bash[45495]: audit 2022-04-22T20:18:17.019284+0000 mon.a (mon.0) 9100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:17 smithi115 bash[45495]: audit 2022-04-22T20:18:17.221917+0000 mon.a (mon.0) 9101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:17 smithi115 bash[46934]: audit 2022-04-22T20:18:17.019284+0000 mon.a (mon.0) 9100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:17 smithi115 bash[46934]: audit 2022-04-22T20:18:17.221917+0000 mon.a (mon.0) 9101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:18 smithi121 bash[36902]: cluster 2022-04-22T20:18:17.196743+0000 mgr.y (mgr.34107) 5897 : cluster [DBG] pgmap v7108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:18 smithi121 bash[36902]: audit 2022-04-22T20:18:18.658756+0000 mon.a (mon.0) 9102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:18 smithi115 bash[45495]: cluster 2022-04-22T20:18:17.196743+0000 mgr.y (mgr.34107) 5897 : cluster [DBG] pgmap v7108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:18 smithi115 bash[45495]: audit 2022-04-22T20:18:18.658756+0000 mon.a (mon.0) 9102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:18 smithi115 bash[46934]: cluster 2022-04-22T20:18:17.196743+0000 mgr.y (mgr.34107) 5897 : cluster [DBG] pgmap v7108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:18 smithi115 bash[46934]: audit 2022-04-22T20:18:18.658756+0000 mon.a (mon.0) 9102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:20 smithi121 bash[36902]: cluster 2022-04-22T20:18:19.197863+0000 mgr.y (mgr.34107) 5898 : cluster [DBG] pgmap v7109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:20 smithi115 bash[45495]: cluster 2022-04-22T20:18:19.197863+0000 mgr.y (mgr.34107) 5898 : cluster [DBG] pgmap v7109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:20 smithi115 bash[46934]: cluster 2022-04-22T20:18:19.197863+0000 mgr.y (mgr.34107) 5898 : cluster [DBG] pgmap v7109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:21.568 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:18:21.914 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:18:21.915 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 2h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:18:21.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:18:21.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:18:21.917 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:18:22.301 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:18:22.302 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:18:22.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:18:22.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:22.302 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:18:22.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:18:22.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:22.304 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:18:22.304 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:18:22.304 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:18:22.304 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:18:22.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:22.279Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: cluster 2022-04-22T20:18:21.199155+0000 mgr.y (mgr.34107) 5899 : cluster [DBG] pgmap v7110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: audit 2022-04-22T20:18:21.544052+0000 mgr.y (mgr.34107) 5900 : audit [DBG] from='client.41526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: audit 2022-04-22T20:18:21.889186+0000 mgr.y (mgr.34107) 5901 : audit [DBG] from='client.51466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: audit 2022-04-22T20:18:22.022848+0000 mon.a (mon.0) 9103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: audit 2022-04-22T20:18:22.224745+0000 mon.a (mon.0) 9104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:22 smithi121 bash[36902]: audit 2022-04-22T20:18:22.293170+0000 mon.c (mon.1) 235 : audit [DBG] from='client.? 172.21.15.115:0/2716625093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: cluster 2022-04-22T20:18:21.199155+0000 mgr.y (mgr.34107) 5899 : cluster [DBG] pgmap v7110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: audit 2022-04-22T20:18:21.544052+0000 mgr.y (mgr.34107) 5900 : audit [DBG] from='client.41526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: audit 2022-04-22T20:18:21.889186+0000 mgr.y (mgr.34107) 5901 : audit [DBG] from='client.51466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: audit 2022-04-22T20:18:22.022848+0000 mon.a (mon.0) 9103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: audit 2022-04-22T20:18:22.224745+0000 mon.a (mon.0) 9104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[46934]: audit 2022-04-22T20:18:22.293170+0000 mon.c (mon.1) 235 : audit [DBG] from='client.? 172.21.15.115:0/2716625093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: cluster 2022-04-22T20:18:21.199155+0000 mgr.y (mgr.34107) 5899 : cluster [DBG] pgmap v7110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: audit 2022-04-22T20:18:21.544052+0000 mgr.y (mgr.34107) 5900 : audit [DBG] from='client.41526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: audit 2022-04-22T20:18:21.889186+0000 mgr.y (mgr.34107) 5901 : audit [DBG] from='client.51466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: audit 2022-04-22T20:18:22.022848+0000 mon.a (mon.0) 9103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: audit 2022-04-22T20:18:22.224745+0000 mon.a (mon.0) 9104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:22 smithi115 bash[45495]: audit 2022-04-22T20:18:22.293170+0000 mon.c (mon.1) 235 : audit [DBG] from='client.? 172.21.15.115:0/2716625093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:23 smithi121 bash[36902]: audit 2022-04-22T20:18:22.991569+0000 mon.a (mon.0) 9105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:18:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:23 smithi121 bash[36902]: audit 2022-04-22T20:18:22.994236+0000 mon.a (mon.0) 9106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:18:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:23 smithi121 bash[36902]: audit 2022-04-22T20:18:22.995905+0000 mon.a (mon.0) 9107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:18:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:23 smithi121 bash[36902]: audit 2022-04-22T20:18:23.352047+0000 mon.a (mon.0) 9108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:18:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:23 smithi121 bash[36902]: audit 2022-04-22T20:18:23.661855+0000 mon.a (mon.0) 9109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:24.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:24.161Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[45495]: audit 2022-04-22T20:18:22.991569+0000 mon.a (mon.0) 9105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:18:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[45495]: audit 2022-04-22T20:18:22.994236+0000 mon.a (mon.0) 9106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:18:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[45495]: audit 2022-04-22T20:18:22.995905+0000 mon.a (mon.0) 9107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:18:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[45495]: audit 2022-04-22T20:18:23.352047+0000 mon.a (mon.0) 9108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:18:24.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[45495]: audit 2022-04-22T20:18:23.661855+0000 mon.a (mon.0) 9109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[46934]: audit 2022-04-22T20:18:22.991569+0000 mon.a (mon.0) 9105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:18:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[46934]: audit 2022-04-22T20:18:22.994236+0000 mon.a (mon.0) 9106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:18:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[46934]: audit 2022-04-22T20:18:22.995905+0000 mon.a (mon.0) 9107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:18:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[46934]: audit 2022-04-22T20:18:23.352047+0000 mon.a (mon.0) 9108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:18:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:23 smithi115 bash[46934]: audit 2022-04-22T20:18:23.661855+0000 mon.a (mon.0) 9109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:24 smithi121 bash[36902]: cluster 2022-04-22T20:18:23.200206+0000 mgr.y (mgr.34107) 5902 : cluster [DBG] pgmap v7111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:25.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:24 smithi115 bash[45495]: cluster 2022-04-22T20:18:23.200206+0000 mgr.y (mgr.34107) 5902 : cluster [DBG] pgmap v7111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:24 smithi115 bash[46934]: cluster 2022-04-22T20:18:23.200206+0000 mgr.y (mgr.34107) 5902 : cluster [DBG] pgmap v7111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:26 smithi121 bash[36902]: cluster 2022-04-22T20:18:25.201273+0000 mgr.y (mgr.34107) 5903 : cluster [DBG] pgmap v7112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:26 smithi115 bash[45495]: cluster 2022-04-22T20:18:25.201273+0000 mgr.y (mgr.34107) 5903 : cluster [DBG] pgmap v7112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:26 smithi115 bash[46934]: cluster 2022-04-22T20:18:25.201273+0000 mgr.y (mgr.34107) 5903 : cluster [DBG] pgmap v7112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:27 smithi121 bash[36902]: audit 2022-04-22T20:18:27.026159+0000 mon.a (mon.0) 9110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:28.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:27 smithi121 bash[36902]: audit 2022-04-22T20:18:27.227602+0000 mon.a (mon.0) 9111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:27 smithi115 bash[45495]: audit 2022-04-22T20:18:27.026159+0000 mon.a (mon.0) 9110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:27 smithi115 bash[45495]: audit 2022-04-22T20:18:27.227602+0000 mon.a (mon.0) 9111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:27 smithi115 bash[46934]: audit 2022-04-22T20:18:27.026159+0000 mon.a (mon.0) 9110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:28.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:27 smithi115 bash[46934]: audit 2022-04-22T20:18:27.227602+0000 mon.a (mon.0) 9111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:28 smithi121 bash[36902]: cluster 2022-04-22T20:18:27.202339+0000 mgr.y (mgr.34107) 5904 : cluster [DBG] pgmap v7113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:28 smithi121 bash[36902]: audit 2022-04-22T20:18:28.665204+0000 mon.a (mon.0) 9112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:28 smithi115 bash[45495]: cluster 2022-04-22T20:18:27.202339+0000 mgr.y (mgr.34107) 5904 : cluster [DBG] pgmap v7113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:29.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:28 smithi115 bash[45495]: audit 2022-04-22T20:18:28.665204+0000 mon.a (mon.0) 9112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:29.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:28 smithi115 bash[46934]: cluster 2022-04-22T20:18:27.202339+0000 mgr.y (mgr.34107) 5904 : cluster [DBG] pgmap v7113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:29.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:28 smithi115 bash[46934]: audit 2022-04-22T20:18:28.665204+0000 mon.a (mon.0) 9112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:30 smithi121 bash[36902]: cluster 2022-04-22T20:18:29.203469+0000 mgr.y (mgr.34107) 5905 : cluster [DBG] pgmap v7114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:30 smithi115 bash[45495]: cluster 2022-04-22T20:18:29.203469+0000 mgr.y (mgr.34107) 5905 : cluster [DBG] pgmap v7114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:30 smithi115 bash[46934]: cluster 2022-04-22T20:18:29.203469+0000 mgr.y (mgr.34107) 5905 : cluster [DBG] pgmap v7114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:32.282Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:33 smithi121 bash[36902]: cluster 2022-04-22T20:18:31.204575+0000 mgr.y (mgr.34107) 5906 : cluster [DBG] pgmap v7115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:33 smithi121 bash[36902]: audit 2022-04-22T20:18:32.029541+0000 mon.a (mon.0) 9113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:33 smithi121 bash[36902]: audit 2022-04-22T20:18:32.230527+0000 mon.a (mon.0) 9114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[45495]: cluster 2022-04-22T20:18:31.204575+0000 mgr.y (mgr.34107) 5906 : cluster [DBG] pgmap v7115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[45495]: audit 2022-04-22T20:18:32.029541+0000 mon.a (mon.0) 9113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[45495]: audit 2022-04-22T20:18:32.230527+0000 mon.a (mon.0) 9114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[46934]: cluster 2022-04-22T20:18:31.204575+0000 mgr.y (mgr.34107) 5906 : cluster [DBG] pgmap v7115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[46934]: audit 2022-04-22T20:18:32.029541+0000 mon.a (mon.0) 9113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:32 smithi115 bash[46934]: audit 2022-04-22T20:18:32.230527+0000 mon.a (mon.0) 9114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:34 smithi121 bash[36902]: audit 2022-04-22T20:18:33.668323+0000 mon.a (mon.0) 9115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:34 smithi115 bash[45495]: audit 2022-04-22T20:18:33.668323+0000 mon.a (mon.0) 9115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:34 smithi115 bash[46934]: audit 2022-04-22T20:18:33.668323+0000 mon.a (mon.0) 9115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:34.162Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:35 smithi121 bash[36902]: cluster 2022-04-22T20:18:33.205729+0000 mgr.y (mgr.34107) 5907 : cluster [DBG] pgmap v7116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:35 smithi115 bash[45495]: cluster 2022-04-22T20:18:33.205729+0000 mgr.y (mgr.34107) 5907 : cluster [DBG] pgmap v7116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:35 smithi115 bash[46934]: cluster 2022-04-22T20:18:33.205729+0000 mgr.y (mgr.34107) 5907 : cluster [DBG] pgmap v7116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:37 smithi115 bash[45495]: cluster 2022-04-22T20:18:35.206855+0000 mgr.y (mgr.34107) 5908 : cluster [DBG] pgmap v7117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:37 smithi115 bash[46934]: cluster 2022-04-22T20:18:35.206855+0000 mgr.y (mgr.34107) 5908 : cluster [DBG] pgmap v7117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:37 smithi121 bash[36902]: cluster 2022-04-22T20:18:35.206855+0000 mgr.y (mgr.34107) 5908 : cluster [DBG] pgmap v7117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:38 smithi115 bash[45495]: audit 2022-04-22T20:18:37.033182+0000 mon.a (mon.0) 9116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:38 smithi115 bash[45495]: audit 2022-04-22T20:18:37.233607+0000 mon.a (mon.0) 9117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:38 smithi115 bash[46934]: audit 2022-04-22T20:18:37.033182+0000 mon.a (mon.0) 9116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:38 smithi115 bash[46934]: audit 2022-04-22T20:18:37.233607+0000 mon.a (mon.0) 9117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:38 smithi121 bash[36902]: audit 2022-04-22T20:18:37.033182+0000 mon.a (mon.0) 9116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:38 smithi121 bash[36902]: audit 2022-04-22T20:18:37.233607+0000 mon.a (mon.0) 9117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:39 smithi115 bash[45495]: cluster 2022-04-22T20:18:37.207959+0000 mgr.y (mgr.34107) 5909 : cluster [DBG] pgmap v7118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:39 smithi115 bash[45495]: audit 2022-04-22T20:18:38.671359+0000 mon.a (mon.0) 9118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:39 smithi115 bash[46934]: cluster 2022-04-22T20:18:37.207959+0000 mgr.y (mgr.34107) 5909 : cluster [DBG] pgmap v7118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:39 smithi115 bash[46934]: audit 2022-04-22T20:18:38.671359+0000 mon.a (mon.0) 9118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:39 smithi121 bash[36902]: cluster 2022-04-22T20:18:37.207959+0000 mgr.y (mgr.34107) 5909 : cluster [DBG] pgmap v7118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:39 smithi121 bash[36902]: audit 2022-04-22T20:18:38.671359+0000 mon.a (mon.0) 9118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:41 smithi115 bash[45495]: cluster 2022-04-22T20:18:39.208963+0000 mgr.y (mgr.34107) 5910 : cluster [DBG] pgmap v7119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:41 smithi115 bash[46934]: cluster 2022-04-22T20:18:39.208963+0000 mgr.y (mgr.34107) 5910 : cluster [DBG] pgmap v7119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:41 smithi121 bash[36902]: cluster 2022-04-22T20:18:39.208963+0000 mgr.y (mgr.34107) 5910 : cluster [DBG] pgmap v7119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:42.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:42.283Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[46934]: cluster 2022-04-22T20:18:41.210064+0000 mgr.y (mgr.34107) 5911 : cluster [DBG] pgmap v7120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[46934]: audit 2022-04-22T20:18:42.036144+0000 mon.a (mon.0) 9119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[46934]: audit 2022-04-22T20:18:42.236672+0000 mon.a (mon.0) 9120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[45495]: cluster 2022-04-22T20:18:41.210064+0000 mgr.y (mgr.34107) 5911 : cluster [DBG] pgmap v7120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[45495]: audit 2022-04-22T20:18:42.036144+0000 mon.a (mon.0) 9119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:43 smithi115 bash[45495]: audit 2022-04-22T20:18:42.236672+0000 mon.a (mon.0) 9120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:43 smithi121 bash[36902]: cluster 2022-04-22T20:18:41.210064+0000 mgr.y (mgr.34107) 5911 : cluster [DBG] pgmap v7120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:43 smithi121 bash[36902]: audit 2022-04-22T20:18:42.036144+0000 mon.a (mon.0) 9119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:43 smithi121 bash[36902]: audit 2022-04-22T20:18:42.236672+0000 mon.a (mon.0) 9120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:44 smithi115 bash[46934]: audit 2022-04-22T20:18:43.674881+0000 mon.a (mon.0) 9121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:44.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:44.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:44 smithi115 bash[45495]: audit 2022-04-22T20:18:43.674881+0000 mon.a (mon.0) 9121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:44 smithi121 bash[36902]: audit 2022-04-22T20:18:43.674881+0000 mon.a (mon.0) 9121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:45 smithi121 bash[36902]: cluster 2022-04-22T20:18:43.211190+0000 mgr.y (mgr.34107) 5912 : cluster [DBG] pgmap v7121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:45 smithi115 bash[46934]: cluster 2022-04-22T20:18:43.211190+0000 mgr.y (mgr.34107) 5912 : cluster [DBG] pgmap v7121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:45.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:45 smithi115 bash[45495]: cluster 2022-04-22T20:18:43.211190+0000 mgr.y (mgr.34107) 5912 : cluster [DBG] pgmap v7121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:47 smithi121 bash[36902]: cluster 2022-04-22T20:18:45.212304+0000 mgr.y (mgr.34107) 5913 : cluster [DBG] pgmap v7122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:47 smithi121 bash[36902]: audit 2022-04-22T20:18:47.039729+0000 mon.a (mon.0) 9122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:47 smithi115 bash[45495]: cluster 2022-04-22T20:18:45.212304+0000 mgr.y (mgr.34107) 5913 : cluster [DBG] pgmap v7122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:47 smithi115 bash[45495]: audit 2022-04-22T20:18:47.039729+0000 mon.a (mon.0) 9122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:47 smithi115 bash[46934]: cluster 2022-04-22T20:18:45.212304+0000 mgr.y (mgr.34107) 5913 : cluster [DBG] pgmap v7122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:47 smithi115 bash[46934]: audit 2022-04-22T20:18:47.039729+0000 mon.a (mon.0) 9122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:48 smithi121 bash[36902]: audit 2022-04-22T20:18:47.239832+0000 mon.a (mon.0) 9123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:48.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:48 smithi115 bash[45495]: audit 2022-04-22T20:18:47.239832+0000 mon.a (mon.0) 9123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:48 smithi115 bash[46934]: audit 2022-04-22T20:18:47.239832+0000 mon.a (mon.0) 9123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:49 smithi121 bash[36902]: cluster 2022-04-22T20:18:47.213386+0000 mgr.y (mgr.34107) 5914 : cluster [DBG] pgmap v7123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:49 smithi121 bash[36902]: audit 2022-04-22T20:18:48.677847+0000 mon.a (mon.0) 9124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:49 smithi115 bash[46934]: cluster 2022-04-22T20:18:47.213386+0000 mgr.y (mgr.34107) 5914 : cluster [DBG] pgmap v7123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:49 smithi115 bash[46934]: audit 2022-04-22T20:18:48.677847+0000 mon.a (mon.0) 9124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:49 smithi115 bash[45495]: cluster 2022-04-22T20:18:47.213386+0000 mgr.y (mgr.34107) 5914 : cluster [DBG] pgmap v7123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:49 smithi115 bash[45495]: audit 2022-04-22T20:18:48.677847+0000 mon.a (mon.0) 9124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:51 smithi121 bash[36902]: cluster 2022-04-22T20:18:49.214524+0000 mgr.y (mgr.34107) 5915 : cluster [DBG] pgmap v7124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:51.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:51 smithi115 bash[45495]: cluster 2022-04-22T20:18:49.214524+0000 mgr.y (mgr.34107) 5915 : cluster [DBG] pgmap v7124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:51.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:51 smithi115 bash[46934]: cluster 2022-04-22T20:18:49.214524+0000 mgr.y (mgr.34107) 5915 : cluster [DBG] pgmap v7124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:52 smithi121 bash[36902]: audit 2022-04-22T20:18:52.043329+0000 mon.a (mon.0) 9125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:52 smithi115 bash[45495]: audit 2022-04-22T20:18:52.043329+0000 mon.a (mon.0) 9125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:52 smithi115 bash[46934]: audit 2022-04-22T20:18:52.043329+0000 mon.a (mon.0) 9125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:52.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:52.282Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:52.676 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:18:53.027 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:18:53.027 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:18:53.027 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 7m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:18:53.028 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:18:53.028 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:18:53.028 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:18:53.028 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 6m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:18:53.028 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 6m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:18:53.029 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:18:53.029 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 6m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:18:53.029 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:18:53.029 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:18:53.030 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:18:53.030 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:18:53.030 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 2h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:18:53.030 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:18:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:18:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:18:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:18:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:18:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:18:53.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:18:53.442 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:18:53.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:18:53.443 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:18:53.443 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:18:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:53 smithi121 bash[36902]: cluster 2022-04-22T20:18:51.215563+0000 mgr.y (mgr.34107) 5916 : cluster [DBG] pgmap v7125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:53 smithi121 bash[36902]: audit 2022-04-22T20:18:52.243290+0000 mon.a (mon.0) 9126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:53 smithi115 bash[45495]: cluster 2022-04-22T20:18:51.215563+0000 mgr.y (mgr.34107) 5916 : cluster [DBG] pgmap v7125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:53 smithi115 bash[45495]: audit 2022-04-22T20:18:52.243290+0000 mon.a (mon.0) 9126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:53 smithi115 bash[46934]: cluster 2022-04-22T20:18:51.215563+0000 mgr.y (mgr.34107) 5916 : cluster [DBG] pgmap v7125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:53 smithi115 bash[46934]: audit 2022-04-22T20:18:52.243290+0000 mon.a (mon.0) 9126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:54 smithi121 bash[36902]: audit 2022-04-22T20:18:52.648956+0000 mgr.y (mgr.34107) 5917 : audit [DBG] from='client.51478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:54 smithi121 bash[36902]: audit 2022-04-22T20:18:52.991486+0000 mgr.y (mgr.34107) 5918 : audit [DBG] from='client.41550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:54 smithi121 bash[36902]: audit 2022-04-22T20:18:53.432125+0000 mon.c (mon.1) 236 : audit [DBG] from='client.? 172.21.15.115:0/1434188486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:54 smithi121 bash[36902]: audit 2022-04-22T20:18:53.680971+0000 mon.a (mon.0) 9127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:18:54.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:18:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[45495]: audit 2022-04-22T20:18:52.648956+0000 mgr.y (mgr.34107) 5917 : audit [DBG] from='client.51478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[45495]: audit 2022-04-22T20:18:52.991486+0000 mgr.y (mgr.34107) 5918 : audit [DBG] from='client.41550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[45495]: audit 2022-04-22T20:18:53.432125+0000 mon.c (mon.1) 236 : audit [DBG] from='client.? 172.21.15.115:0/1434188486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[45495]: audit 2022-04-22T20:18:53.680971+0000 mon.a (mon.0) 9127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[46934]: audit 2022-04-22T20:18:52.648956+0000 mgr.y (mgr.34107) 5917 : audit [DBG] from='client.51478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[46934]: audit 2022-04-22T20:18:52.991486+0000 mgr.y (mgr.34107) 5918 : audit [DBG] from='client.41550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:18:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[46934]: audit 2022-04-22T20:18:53.432125+0000 mon.c (mon.1) 236 : audit [DBG] from='client.? 172.21.15.115:0/1434188486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:18:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:54 smithi115 bash[46934]: audit 2022-04-22T20:18:53.680971+0000 mon.a (mon.0) 9127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:55 smithi121 bash[36902]: cluster 2022-04-22T20:18:53.216337+0000 mgr.y (mgr.34107) 5919 : cluster [DBG] pgmap v7126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:55 smithi115 bash[45495]: cluster 2022-04-22T20:18:53.216337+0000 mgr.y (mgr.34107) 5919 : cluster [DBG] pgmap v7126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:55 smithi115 bash[46934]: cluster 2022-04-22T20:18:53.216337+0000 mgr.y (mgr.34107) 5919 : cluster [DBG] pgmap v7126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:57 smithi121 bash[36902]: cluster 2022-04-22T20:18:55.217379+0000 mgr.y (mgr.34107) 5920 : cluster [DBG] pgmap v7127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:57 smithi121 bash[36902]: audit 2022-04-22T20:18:57.047103+0000 mon.a (mon.0) 9128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:57 smithi115 bash[45495]: cluster 2022-04-22T20:18:55.217379+0000 mgr.y (mgr.34107) 5920 : cluster [DBG] pgmap v7127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:57 smithi115 bash[45495]: audit 2022-04-22T20:18:57.047103+0000 mon.a (mon.0) 9128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:57 smithi115 bash[46934]: cluster 2022-04-22T20:18:55.217379+0000 mgr.y (mgr.34107) 5920 : cluster [DBG] pgmap v7127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:57 smithi115 bash[46934]: audit 2022-04-22T20:18:57.047103+0000 mon.a (mon.0) 9128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:18:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:58 smithi121 bash[36902]: audit 2022-04-22T20:18:57.246143+0000 mon.a (mon.0) 9129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:58 smithi115 bash[45495]: audit 2022-04-22T20:18:57.246143+0000 mon.a (mon.0) 9129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:58.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:58 smithi115 bash[46934]: audit 2022-04-22T20:18:57.246143+0000 mon.a (mon.0) 9129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:18:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:59 smithi121 bash[36902]: cluster 2022-04-22T20:18:57.218482+0000 mgr.y (mgr.34107) 5921 : cluster [DBG] pgmap v7128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:18:59 smithi121 bash[36902]: audit 2022-04-22T20:18:58.684214+0000 mon.a (mon.0) 9130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:59 smithi115 bash[45495]: cluster 2022-04-22T20:18:57.218482+0000 mgr.y (mgr.34107) 5921 : cluster [DBG] pgmap v7128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:18:59 smithi115 bash[45495]: audit 2022-04-22T20:18:58.684214+0000 mon.a (mon.0) 9130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:18:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:59 smithi115 bash[46934]: cluster 2022-04-22T20:18:57.218482+0000 mgr.y (mgr.34107) 5921 : cluster [DBG] pgmap v7128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:18:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:18:59 smithi115 bash[46934]: audit 2022-04-22T20:18:58.684214+0000 mon.a (mon.0) 9130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:00 smithi121 bash[36902]: cluster 2022-04-22T20:18:59.219563+0000 mgr.y (mgr.34107) 5922 : cluster [DBG] pgmap v7129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:00.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:00 smithi115 bash[45495]: cluster 2022-04-22T20:18:59.219563+0000 mgr.y (mgr.34107) 5922 : cluster [DBG] pgmap v7129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:00.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:00 smithi115 bash[46934]: cluster 2022-04-22T20:18:59.219563+0000 mgr.y (mgr.34107) 5922 : cluster [DBG] pgmap v7129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:02.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:02.284Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[45495]: cluster 2022-04-22T20:19:01.220573+0000 mgr.y (mgr.34107) 5923 : cluster [DBG] pgmap v7130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[45495]: audit 2022-04-22T20:19:02.050635+0000 mon.a (mon.0) 9131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:02.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[45495]: audit 2022-04-22T20:19:02.249274+0000 mon.a (mon.0) 9132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[46934]: cluster 2022-04-22T20:19:01.220573+0000 mgr.y (mgr.34107) 5923 : cluster [DBG] pgmap v7130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[46934]: audit 2022-04-22T20:19:02.050635+0000 mon.a (mon.0) 9131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:02 smithi115 bash[46934]: audit 2022-04-22T20:19:02.249274+0000 mon.a (mon.0) 9132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:02 smithi121 bash[36902]: cluster 2022-04-22T20:19:01.220573+0000 mgr.y (mgr.34107) 5923 : cluster [DBG] pgmap v7130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:02 smithi121 bash[36902]: audit 2022-04-22T20:19:02.050635+0000 mon.a (mon.0) 9131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:02 smithi121 bash[36902]: audit 2022-04-22T20:19:02.249274+0000 mon.a (mon.0) 9132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:04.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:04 smithi115 bash[45495]: cluster 2022-04-22T20:19:03.221687+0000 mgr.y (mgr.34107) 5924 : cluster [DBG] pgmap v7131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:04.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:04 smithi115 bash[45495]: audit 2022-04-22T20:19:03.687345+0000 mon.a (mon.0) 9133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:04.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:04.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:04 smithi115 bash[46934]: cluster 2022-04-22T20:19:03.221687+0000 mgr.y (mgr.34107) 5924 : cluster [DBG] pgmap v7131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:04.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:04 smithi115 bash[46934]: audit 2022-04-22T20:19:03.687345+0000 mon.a (mon.0) 9133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:04 smithi121 bash[36902]: cluster 2022-04-22T20:19:03.221687+0000 mgr.y (mgr.34107) 5924 : cluster [DBG] pgmap v7131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:04 smithi121 bash[36902]: audit 2022-04-22T20:19:03.687345+0000 mon.a (mon.0) 9133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:06 smithi115 bash[46934]: cluster 2022-04-22T20:19:05.222856+0000 mgr.y (mgr.34107) 5925 : cluster [DBG] pgmap v7132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:06 smithi115 bash[45495]: cluster 2022-04-22T20:19:05.222856+0000 mgr.y (mgr.34107) 5925 : cluster [DBG] pgmap v7132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:06 smithi121 bash[36902]: cluster 2022-04-22T20:19:05.222856+0000 mgr.y (mgr.34107) 5925 : cluster [DBG] pgmap v7132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:07 smithi121 bash[36902]: audit 2022-04-22T20:19:07.054241+0000 mon.a (mon.0) 9134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:07 smithi121 bash[36902]: audit 2022-04-22T20:19:07.252090+0000 mon.a (mon.0) 9135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:07 smithi115 bash[45495]: audit 2022-04-22T20:19:07.054241+0000 mon.a (mon.0) 9134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:07 smithi115 bash[45495]: audit 2022-04-22T20:19:07.252090+0000 mon.a (mon.0) 9135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:07 smithi115 bash[46934]: audit 2022-04-22T20:19:07.054241+0000 mon.a (mon.0) 9134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:07.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:07 smithi115 bash[46934]: audit 2022-04-22T20:19:07.252090+0000 mon.a (mon.0) 9135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:08 smithi121 bash[36902]: cluster 2022-04-22T20:19:07.223976+0000 mgr.y (mgr.34107) 5926 : cluster [DBG] pgmap v7133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:08.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:08 smithi115 bash[45495]: cluster 2022-04-22T20:19:07.223976+0000 mgr.y (mgr.34107) 5926 : cluster [DBG] pgmap v7133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:08.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:08 smithi115 bash[46934]: cluster 2022-04-22T20:19:07.223976+0000 mgr.y (mgr.34107) 5926 : cluster [DBG] pgmap v7133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:09 smithi121 bash[36902]: audit 2022-04-22T20:19:08.690408+0000 mon.a (mon.0) 9136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:09 smithi115 bash[45495]: audit 2022-04-22T20:19:08.690408+0000 mon.a (mon.0) 9136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:09.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:09 smithi115 bash[46934]: audit 2022-04-22T20:19:08.690408+0000 mon.a (mon.0) 9136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:10 smithi121 bash[36902]: cluster 2022-04-22T20:19:09.225049+0000 mgr.y (mgr.34107) 5927 : cluster [DBG] pgmap v7134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:10 smithi115 bash[45495]: cluster 2022-04-22T20:19:09.225049+0000 mgr.y (mgr.34107) 5927 : cluster [DBG] pgmap v7134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:10 smithi115 bash[46934]: cluster 2022-04-22T20:19:09.225049+0000 mgr.y (mgr.34107) 5927 : cluster [DBG] pgmap v7134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:12.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:12.284Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:12 smithi121 bash[36902]: cluster 2022-04-22T20:19:11.226080+0000 mgr.y (mgr.34107) 5928 : cluster [DBG] pgmap v7135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:12 smithi121 bash[36902]: audit 2022-04-22T20:19:12.057850+0000 mon.a (mon.0) 9137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:12 smithi121 bash[36902]: audit 2022-04-22T20:19:12.255225+0000 mon.a (mon.0) 9138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[46934]: cluster 2022-04-22T20:19:11.226080+0000 mgr.y (mgr.34107) 5928 : cluster [DBG] pgmap v7135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[46934]: audit 2022-04-22T20:19:12.057850+0000 mon.a (mon.0) 9137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[46934]: audit 2022-04-22T20:19:12.255225+0000 mon.a (mon.0) 9138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[45495]: cluster 2022-04-22T20:19:11.226080+0000 mgr.y (mgr.34107) 5928 : cluster [DBG] pgmap v7135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[45495]: audit 2022-04-22T20:19:12.057850+0000 mon.a (mon.0) 9137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:13.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:12 smithi115 bash[45495]: audit 2022-04-22T20:19:12.255225+0000 mon.a (mon.0) 9138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:14.175 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:13 smithi115 bash[45495]: audit 2022-04-22T20:19:13.693651+0000 mon.a (mon.0) 9139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:14.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:13 smithi115 bash[46934]: audit 2022-04-22T20:19:13.693651+0000 mon.a (mon.0) 9139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:13 smithi121 bash[36902]: audit 2022-04-22T20:19:13.693651+0000 mon.a (mon.0) 9139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:14.166Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:14 smithi121 bash[36902]: cluster 2022-04-22T20:19:13.227176+0000 mgr.y (mgr.34107) 5929 : cluster [DBG] pgmap v7136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:15.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:14 smithi115 bash[45495]: cluster 2022-04-22T20:19:13.227176+0000 mgr.y (mgr.34107) 5929 : cluster [DBG] pgmap v7136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:15.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:14 smithi115 bash[46934]: cluster 2022-04-22T20:19:13.227176+0000 mgr.y (mgr.34107) 5929 : cluster [DBG] pgmap v7136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:15 smithi121 bash[36902]: audit 2022-04-22T20:19:15.266357+0000 mon.a (mon.0) 9140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:19:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:15 smithi121 bash[36902]: audit 2022-04-22T20:19:15.266944+0000 mon.a (mon.0) 9141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:19:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:15 smithi115 bash[45495]: audit 2022-04-22T20:19:15.266357+0000 mon.a (mon.0) 9140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:19:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:15 smithi115 bash[45495]: audit 2022-04-22T20:19:15.266944+0000 mon.a (mon.0) 9141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:19:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:15 smithi115 bash[46934]: audit 2022-04-22T20:19:15.266357+0000 mon.a (mon.0) 9140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:19:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:15 smithi115 bash[46934]: audit 2022-04-22T20:19:15.266944+0000 mon.a (mon.0) 9141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:19:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:16 smithi121 bash[36902]: cluster 2022-04-22T20:19:15.228244+0000 mgr.y (mgr.34107) 5930 : cluster [DBG] pgmap v7137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:16 smithi115 bash[45495]: cluster 2022-04-22T20:19:15.228244+0000 mgr.y (mgr.34107) 5930 : cluster [DBG] pgmap v7137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:16 smithi115 bash[46934]: cluster 2022-04-22T20:19:15.228244+0000 mgr.y (mgr.34107) 5930 : cluster [DBG] pgmap v7137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:17 smithi121 bash[36902]: audit 2022-04-22T20:19:17.061745+0000 mon.a (mon.0) 9142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:17 smithi121 bash[36902]: audit 2022-04-22T20:19:17.258372+0000 mon.a (mon.0) 9143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:17 smithi115 bash[46934]: audit 2022-04-22T20:19:17.061745+0000 mon.a (mon.0) 9142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:18.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:17 smithi115 bash[46934]: audit 2022-04-22T20:19:17.258372+0000 mon.a (mon.0) 9143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:17 smithi115 bash[45495]: audit 2022-04-22T20:19:17.061745+0000 mon.a (mon.0) 9142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:18.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:17 smithi115 bash[45495]: audit 2022-04-22T20:19:17.258372+0000 mon.a (mon.0) 9143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:18 smithi121 bash[36902]: cluster 2022-04-22T20:19:17.229287+0000 mgr.y (mgr.34107) 5931 : cluster [DBG] pgmap v7138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:18 smithi121 bash[36902]: audit 2022-04-22T20:19:18.696985+0000 mon.a (mon.0) 9144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:18 smithi115 bash[46934]: cluster 2022-04-22T20:19:17.229287+0000 mgr.y (mgr.34107) 5931 : cluster [DBG] pgmap v7138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:19.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:18 smithi115 bash[46934]: audit 2022-04-22T20:19:18.696985+0000 mon.a (mon.0) 9144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:18 smithi115 bash[45495]: cluster 2022-04-22T20:19:17.229287+0000 mgr.y (mgr.34107) 5931 : cluster [DBG] pgmap v7138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:19.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:18 smithi115 bash[45495]: audit 2022-04-22T20:19:18.696985+0000 mon.a (mon.0) 9144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:20 smithi121 bash[36902]: cluster 2022-04-22T20:19:19.230403+0000 mgr.y (mgr.34107) 5932 : cluster [DBG] pgmap v7139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:21.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:20 smithi115 bash[45495]: cluster 2022-04-22T20:19:19.230403+0000 mgr.y (mgr.34107) 5932 : cluster [DBG] pgmap v7139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:21.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:20 smithi115 bash[46934]: cluster 2022-04-22T20:19:19.230403+0000 mgr.y (mgr.34107) 5932 : cluster [DBG] pgmap v7139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:22.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:22.285Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:22 smithi121 bash[36902]: cluster 2022-04-22T20:19:21.231501+0000 mgr.y (mgr.34107) 5933 : cluster [DBG] pgmap v7140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:22 smithi121 bash[36902]: audit 2022-04-22T20:19:22.065355+0000 mon.a (mon.0) 9145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:22 smithi121 bash[36902]: audit 2022-04-22T20:19:22.261189+0000 mon.a (mon.0) 9146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[45495]: cluster 2022-04-22T20:19:21.231501+0000 mgr.y (mgr.34107) 5933 : cluster [DBG] pgmap v7140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[45495]: audit 2022-04-22T20:19:22.065355+0000 mon.a (mon.0) 9145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:23.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[45495]: audit 2022-04-22T20:19:22.261189+0000 mon.a (mon.0) 9146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[46934]: cluster 2022-04-22T20:19:21.231501+0000 mgr.y (mgr.34107) 5933 : cluster [DBG] pgmap v7140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:23.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[46934]: audit 2022-04-22T20:19:22.065355+0000 mon.a (mon.0) 9145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:22 smithi115 bash[46934]: audit 2022-04-22T20:19:22.261189+0000 mon.a (mon.0) 9146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:23.818 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:19:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:23 smithi121 bash[36902]: audit 2022-04-22T20:19:23.361895+0000 mon.a (mon.0) 9147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:23 smithi121 bash[36902]: audit 2022-04-22T20:19:23.364866+0000 mon.a (mon.0) 9148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:23 smithi121 bash[36902]: audit 2022-04-22T20:19:23.366588+0000 mon.a (mon.0) 9149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:23 smithi121 bash[36902]: audit 2022-04-22T20:19:23.700305+0000 mon.a (mon.0) 9150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:23 smithi121 bash[36902]: audit 2022-04-22T20:19:23.721737+0000 mon.a (mon.0) 9151 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:19:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[46934]: audit 2022-04-22T20:19:23.361895+0000 mon.a (mon.0) 9147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:19:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[46934]: audit 2022-04-22T20:19:23.364866+0000 mon.a (mon.0) 9148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:19:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[46934]: audit 2022-04-22T20:19:23.366588+0000 mon.a (mon.0) 9149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:19:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[46934]: audit 2022-04-22T20:19:23.700305+0000 mon.a (mon.0) 9150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[46934]: audit 2022-04-22T20:19:23.721737+0000 mon.a (mon.0) 9151 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:19:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[45495]: audit 2022-04-22T20:19:23.361895+0000 mon.a (mon.0) 9147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:19:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[45495]: audit 2022-04-22T20:19:23.364866+0000 mon.a (mon.0) 9148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:19:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[45495]: audit 2022-04-22T20:19:23.366588+0000 mon.a (mon.0) 9149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:19:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[45495]: audit 2022-04-22T20:19:23.700305+0000 mon.a (mon.0) 9150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:23 smithi115 bash[45495]: audit 2022-04-22T20:19:23.721737+0000 mon.a (mon.0) 9151 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:19:24.153 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:19:24.153 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:19:24.153 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:19:24.153 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:19:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 2h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:19:24.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:19:24.156 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:19:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:24.163Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:24.553 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:19:24.553 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:19:24.553 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:19:24.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:19:24.555 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:19:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:24 smithi121 bash[36902]: cluster 2022-04-22T20:19:23.232660+0000 mgr.y (mgr.34107) 5934 : cluster [DBG] pgmap v7141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:24 smithi121 bash[36902]: audit 2022-04-22T20:19:23.793322+0000 mgr.y (mgr.34107) 5935 : audit [DBG] from='client.51496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:24 smithi121 bash[36902]: audit 2022-04-22T20:19:24.544256+0000 mon.a (mon.0) 9152 : audit [DBG] from='client.? 172.21.15.115:0/1332288784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[45495]: cluster 2022-04-22T20:19:23.232660+0000 mgr.y (mgr.34107) 5934 : cluster [DBG] pgmap v7141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[45495]: audit 2022-04-22T20:19:23.793322+0000 mgr.y (mgr.34107) 5935 : audit [DBG] from='client.51496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:25.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[45495]: audit 2022-04-22T20:19:24.544256+0000 mon.a (mon.0) 9152 : audit [DBG] from='client.? 172.21.15.115:0/1332288784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[46934]: cluster 2022-04-22T20:19:23.232660+0000 mgr.y (mgr.34107) 5934 : cluster [DBG] pgmap v7141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:25.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[46934]: audit 2022-04-22T20:19:23.793322+0000 mgr.y (mgr.34107) 5935 : audit [DBG] from='client.51496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:25.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:24 smithi115 bash[46934]: audit 2022-04-22T20:19:24.544256+0000 mon.a (mon.0) 9152 : audit [DBG] from='client.? 172.21.15.115:0/1332288784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:25 smithi121 bash[36902]: audit 2022-04-22T20:19:24.128477+0000 mgr.y (mgr.34107) 5936 : audit [DBG] from='client.41568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:26.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:25 smithi115 bash[45495]: audit 2022-04-22T20:19:24.128477+0000 mgr.y (mgr.34107) 5936 : audit [DBG] from='client.41568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:26.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:25 smithi115 bash[46934]: audit 2022-04-22T20:19:24.128477+0000 mgr.y (mgr.34107) 5936 : audit [DBG] from='client.41568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:26 smithi121 bash[36902]: cluster 2022-04-22T20:19:25.233716+0000 mgr.y (mgr.34107) 5937 : cluster [DBG] pgmap v7142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:27.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:26 smithi115 bash[45495]: cluster 2022-04-22T20:19:25.233716+0000 mgr.y (mgr.34107) 5937 : cluster [DBG] pgmap v7142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:27.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:26 smithi115 bash[46934]: cluster 2022-04-22T20:19:25.233716+0000 mgr.y (mgr.34107) 5937 : cluster [DBG] pgmap v7142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:27 smithi121 bash[36902]: audit 2022-04-22T20:19:27.069136+0000 mon.a (mon.0) 9153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:27 smithi121 bash[36902]: audit 2022-04-22T20:19:27.264431+0000 mon.a (mon.0) 9154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:28.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:27 smithi115 bash[45495]: audit 2022-04-22T20:19:27.069136+0000 mon.a (mon.0) 9153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:28.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:27 smithi115 bash[45495]: audit 2022-04-22T20:19:27.264431+0000 mon.a (mon.0) 9154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:27 smithi115 bash[46934]: audit 2022-04-22T20:19:27.069136+0000 mon.a (mon.0) 9153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:28.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:27 smithi115 bash[46934]: audit 2022-04-22T20:19:27.264431+0000 mon.a (mon.0) 9154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:28 smithi121 bash[36902]: cluster 2022-04-22T20:19:27.234765+0000 mgr.y (mgr.34107) 5938 : cluster [DBG] pgmap v7143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:28 smithi121 bash[36902]: audit 2022-04-22T20:19:28.702302+0000 mon.a (mon.0) 9155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:28 smithi115 bash[45495]: cluster 2022-04-22T20:19:27.234765+0000 mgr.y (mgr.34107) 5938 : cluster [DBG] pgmap v7143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:29.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:28 smithi115 bash[45495]: audit 2022-04-22T20:19:28.702302+0000 mon.a (mon.0) 9155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:28 smithi115 bash[46934]: cluster 2022-04-22T20:19:27.234765+0000 mgr.y (mgr.34107) 5938 : cluster [DBG] pgmap v7143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:29.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:28 smithi115 bash[46934]: audit 2022-04-22T20:19:28.702302+0000 mon.a (mon.0) 9155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:30 smithi121 bash[36902]: cluster 2022-04-22T20:19:29.235912+0000 mgr.y (mgr.34107) 5939 : cluster [DBG] pgmap v7144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:31.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:30 smithi115 bash[45495]: cluster 2022-04-22T20:19:29.235912+0000 mgr.y (mgr.34107) 5939 : cluster [DBG] pgmap v7144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:31.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:30 smithi115 bash[46934]: cluster 2022-04-22T20:19:29.235912+0000 mgr.y (mgr.34107) 5939 : cluster [DBG] pgmap v7144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:32.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:32.285Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:32 smithi121 bash[36902]: cluster 2022-04-22T20:19:31.236991+0000 mgr.y (mgr.34107) 5940 : cluster [DBG] pgmap v7145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:32 smithi121 bash[36902]: audit 2022-04-22T20:19:32.072973+0000 mon.a (mon.0) 9156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:32 smithi121 bash[36902]: audit 2022-04-22T20:19:32.267605+0000 mon.a (mon.0) 9157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[45495]: cluster 2022-04-22T20:19:31.236991+0000 mgr.y (mgr.34107) 5940 : cluster [DBG] pgmap v7145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[45495]: audit 2022-04-22T20:19:32.072973+0000 mon.a (mon.0) 9156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:33.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[45495]: audit 2022-04-22T20:19:32.267605+0000 mon.a (mon.0) 9157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:33.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[46934]: cluster 2022-04-22T20:19:31.236991+0000 mgr.y (mgr.34107) 5940 : cluster [DBG] pgmap v7145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[46934]: audit 2022-04-22T20:19:32.072973+0000 mon.a (mon.0) 9156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:33.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:32 smithi115 bash[46934]: audit 2022-04-22T20:19:32.267605+0000 mon.a (mon.0) 9157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:33 smithi121 bash[36902]: audit 2022-04-22T20:19:33.705378+0000 mon.a (mon.0) 9158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:34.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:33 smithi115 bash[45495]: audit 2022-04-22T20:19:33.705378+0000 mon.a (mon.0) 9158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:34.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:34.165Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:33 smithi115 bash[46934]: audit 2022-04-22T20:19:33.705378+0000 mon.a (mon.0) 9158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:34 smithi121 bash[36902]: cluster 2022-04-22T20:19:33.238082+0000 mgr.y (mgr.34107) 5941 : cluster [DBG] pgmap v7146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:35.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:34 smithi115 bash[45495]: cluster 2022-04-22T20:19:33.238082+0000 mgr.y (mgr.34107) 5941 : cluster [DBG] pgmap v7146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:35.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:34 smithi115 bash[46934]: cluster 2022-04-22T20:19:33.238082+0000 mgr.y (mgr.34107) 5941 : cluster [DBG] pgmap v7146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:36 smithi121 bash[36902]: cluster 2022-04-22T20:19:35.239056+0000 mgr.y (mgr.34107) 5942 : cluster [DBG] pgmap v7147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:37.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:36 smithi115 bash[45495]: cluster 2022-04-22T20:19:35.239056+0000 mgr.y (mgr.34107) 5942 : cluster [DBG] pgmap v7147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:36 smithi115 bash[46934]: cluster 2022-04-22T20:19:35.239056+0000 mgr.y (mgr.34107) 5942 : cluster [DBG] pgmap v7147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:37 smithi121 bash[36902]: audit 2022-04-22T20:19:37.076493+0000 mon.a (mon.0) 9159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:37 smithi121 bash[36902]: audit 2022-04-22T20:19:37.270719+0000 mon.a (mon.0) 9160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:37 smithi115 bash[45495]: audit 2022-04-22T20:19:37.076493+0000 mon.a (mon.0) 9159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:38.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:37 smithi115 bash[45495]: audit 2022-04-22T20:19:37.270719+0000 mon.a (mon.0) 9160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:37 smithi115 bash[46934]: audit 2022-04-22T20:19:37.076493+0000 mon.a (mon.0) 9159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:37 smithi115 bash[46934]: audit 2022-04-22T20:19:37.270719+0000 mon.a (mon.0) 9160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:39 smithi121 bash[36902]: cluster 2022-04-22T20:19:37.239999+0000 mgr.y (mgr.34107) 5943 : cluster [DBG] pgmap v7148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:39 smithi121 bash[36902]: audit 2022-04-22T20:19:38.708420+0000 mon.a (mon.0) 9161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:38 smithi115 bash[45495]: cluster 2022-04-22T20:19:37.239999+0000 mgr.y (mgr.34107) 5943 : cluster [DBG] pgmap v7148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:39.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:38 smithi115 bash[45495]: audit 2022-04-22T20:19:38.708420+0000 mon.a (mon.0) 9161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:39.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:38 smithi115 bash[46934]: cluster 2022-04-22T20:19:37.239999+0000 mgr.y (mgr.34107) 5943 : cluster [DBG] pgmap v7148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:39.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:38 smithi115 bash[46934]: audit 2022-04-22T20:19:38.708420+0000 mon.a (mon.0) 9161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:41 smithi121 bash[36902]: cluster 2022-04-22T20:19:39.241096+0000 mgr.y (mgr.34107) 5944 : cluster [DBG] pgmap v7149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:40 smithi115 bash[45495]: cluster 2022-04-22T20:19:39.241096+0000 mgr.y (mgr.34107) 5944 : cluster [DBG] pgmap v7149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:40 smithi115 bash[46934]: cluster 2022-04-22T20:19:39.241096+0000 mgr.y (mgr.34107) 5944 : cluster [DBG] pgmap v7149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:42.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:42.286Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:43 smithi121 bash[36902]: cluster 2022-04-22T20:19:41.242062+0000 mgr.y (mgr.34107) 5945 : cluster [DBG] pgmap v7150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:43 smithi121 bash[36902]: audit 2022-04-22T20:19:42.080179+0000 mon.a (mon.0) 9162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:43 smithi121 bash[36902]: audit 2022-04-22T20:19:42.273551+0000 mon.a (mon.0) 9163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[45495]: cluster 2022-04-22T20:19:41.242062+0000 mgr.y (mgr.34107) 5945 : cluster [DBG] pgmap v7150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[45495]: audit 2022-04-22T20:19:42.080179+0000 mon.a (mon.0) 9162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[45495]: audit 2022-04-22T20:19:42.273551+0000 mon.a (mon.0) 9163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[46934]: cluster 2022-04-22T20:19:41.242062+0000 mgr.y (mgr.34107) 5945 : cluster [DBG] pgmap v7150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[46934]: audit 2022-04-22T20:19:42.080179+0000 mon.a (mon.0) 9162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:43 smithi115 bash[46934]: audit 2022-04-22T20:19:42.273551+0000 mon.a (mon.0) 9163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:44 smithi115 bash[45495]: audit 2022-04-22T20:19:43.711496+0000 mon.a (mon.0) 9164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:44.167Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:44 smithi115 bash[46934]: audit 2022-04-22T20:19:43.711496+0000 mon.a (mon.0) 9164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:44 smithi121 bash[36902]: audit 2022-04-22T20:19:43.711496+0000 mon.a (mon.0) 9164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:45 smithi115 bash[45495]: cluster 2022-04-22T20:19:43.243207+0000 mgr.y (mgr.34107) 5946 : cluster [DBG] pgmap v7151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:45.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:45 smithi115 bash[46934]: cluster 2022-04-22T20:19:43.243207+0000 mgr.y (mgr.34107) 5946 : cluster [DBG] pgmap v7151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:45 smithi121 bash[36902]: cluster 2022-04-22T20:19:43.243207+0000 mgr.y (mgr.34107) 5946 : cluster [DBG] pgmap v7151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:47 smithi115 bash[45495]: cluster 2022-04-22T20:19:45.244422+0000 mgr.y (mgr.34107) 5947 : cluster [DBG] pgmap v7152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:47 smithi115 bash[46934]: cluster 2022-04-22T20:19:45.244422+0000 mgr.y (mgr.34107) 5947 : cluster [DBG] pgmap v7152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:47 smithi121 bash[36902]: cluster 2022-04-22T20:19:45.244422+0000 mgr.y (mgr.34107) 5947 : cluster [DBG] pgmap v7152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:48 smithi115 bash[45495]: audit 2022-04-22T20:19:47.084078+0000 mon.a (mon.0) 9165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:48 smithi115 bash[45495]: audit 2022-04-22T20:19:47.276532+0000 mon.a (mon.0) 9166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:48 smithi115 bash[46934]: audit 2022-04-22T20:19:47.084078+0000 mon.a (mon.0) 9165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:48 smithi115 bash[46934]: audit 2022-04-22T20:19:47.276532+0000 mon.a (mon.0) 9166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:48 smithi121 bash[36902]: audit 2022-04-22T20:19:47.084078+0000 mon.a (mon.0) 9165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:48 smithi121 bash[36902]: audit 2022-04-22T20:19:47.276532+0000 mon.a (mon.0) 9166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:49.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:49 smithi115 bash[45495]: cluster 2022-04-22T20:19:47.245505+0000 mgr.y (mgr.34107) 5948 : cluster [DBG] pgmap v7153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:49.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:49 smithi115 bash[45495]: audit 2022-04-22T20:19:48.714676+0000 mon.a (mon.0) 9167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:49 smithi115 bash[46934]: cluster 2022-04-22T20:19:47.245505+0000 mgr.y (mgr.34107) 5948 : cluster [DBG] pgmap v7153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:49.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:49 smithi115 bash[46934]: audit 2022-04-22T20:19:48.714676+0000 mon.a (mon.0) 9167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:49 smithi121 bash[36902]: cluster 2022-04-22T20:19:47.245505+0000 mgr.y (mgr.34107) 5948 : cluster [DBG] pgmap v7153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:49 smithi121 bash[36902]: audit 2022-04-22T20:19:48.714676+0000 mon.a (mon.0) 9167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:51 smithi115 bash[45495]: cluster 2022-04-22T20:19:49.246704+0000 mgr.y (mgr.34107) 5949 : cluster [DBG] pgmap v7154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:51 smithi115 bash[46934]: cluster 2022-04-22T20:19:49.246704+0000 mgr.y (mgr.34107) 5949 : cluster [DBG] pgmap v7154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:51 smithi121 bash[36902]: cluster 2022-04-22T20:19:49.246704+0000 mgr.y (mgr.34107) 5949 : cluster [DBG] pgmap v7154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:52.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:52.286Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:53 smithi121 bash[36902]: cluster 2022-04-22T20:19:51.247860+0000 mgr.y (mgr.34107) 5950 : cluster [DBG] pgmap v7155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:53 smithi121 bash[36902]: audit 2022-04-22T20:19:52.087883+0000 mon.a (mon.0) 9168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:53 smithi121 bash[36902]: audit 2022-04-22T20:19:52.279580+0000 mon.a (mon.0) 9169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[45495]: cluster 2022-04-22T20:19:51.247860+0000 mgr.y (mgr.34107) 5950 : cluster [DBG] pgmap v7155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[45495]: audit 2022-04-22T20:19:52.087883+0000 mon.a (mon.0) 9168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[45495]: audit 2022-04-22T20:19:52.279580+0000 mon.a (mon.0) 9169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[46934]: cluster 2022-04-22T20:19:51.247860+0000 mgr.y (mgr.34107) 5950 : cluster [DBG] pgmap v7155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[46934]: audit 2022-04-22T20:19:52.087883+0000 mon.a (mon.0) 9168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:53.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:53 smithi115 bash[46934]: audit 2022-04-22T20:19:52.279580+0000 mon.a (mon.0) 9169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:54 smithi121 bash[36902]: audit 2022-04-22T20:19:53.717832+0000 mon.a (mon.0) 9170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:54.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:54 smithi115 bash[45495]: audit 2022-04-22T20:19:53.717832+0000 mon.a (mon.0) 9170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:19:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:19:54.166Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:19:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:54 smithi115 bash[46934]: audit 2022-04-22T20:19:53.717832+0000 mon.a (mon.0) 9170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:54.935 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:19:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:19:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:19:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 8m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:19:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 7m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 7m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 8m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:19:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 7m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 2h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:19:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:19:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:55 smithi121 bash[36902]: cluster 2022-04-22T20:19:53.248940+0000 mgr.y (mgr.34107) 5951 : cluster [DBG] pgmap v7156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:55 smithi115 bash[46934]: cluster 2022-04-22T20:19:53.248940+0000 mgr.y (mgr.34107) 5951 : cluster [DBG] pgmap v7156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:55 smithi115 bash[45495]: cluster 2022-04-22T20:19:53.248940+0000 mgr.y (mgr.34107) 5951 : cluster [DBG] pgmap v7156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:55.676 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:19:55.676 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:19:55.676 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:19:55.676 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:55.676 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:19:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:19:55.678 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:19:55.678 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:19:55.678 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:19:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:56 smithi121 bash[36902]: audit 2022-04-22T20:19:54.909366+0000 mgr.y (mgr.34107) 5952 : audit [DBG] from='client.51514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:56 smithi121 bash[36902]: audit 2022-04-22T20:19:55.667107+0000 mon.a (mon.0) 9171 : audit [DBG] from='client.? 172.21.15.115:0/1485486309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:56 smithi115 bash[45495]: audit 2022-04-22T20:19:54.909366+0000 mgr.y (mgr.34107) 5952 : audit [DBG] from='client.51514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:56.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:56 smithi115 bash[45495]: audit 2022-04-22T20:19:55.667107+0000 mon.a (mon.0) 9171 : audit [DBG] from='client.? 172.21.15.115:0/1485486309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:56.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:56 smithi115 bash[46934]: audit 2022-04-22T20:19:54.909366+0000 mgr.y (mgr.34107) 5952 : audit [DBG] from='client.51514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:56.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:56 smithi115 bash[46934]: audit 2022-04-22T20:19:55.667107+0000 mon.a (mon.0) 9171 : audit [DBG] from='client.? 172.21.15.115:0/1485486309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:19:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:57 smithi121 bash[36902]: audit 2022-04-22T20:19:55.244761+0000 mgr.y (mgr.34107) 5953 : audit [DBG] from='client.41586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:57 smithi121 bash[36902]: cluster 2022-04-22T20:19:55.249651+0000 mgr.y (mgr.34107) 5954 : cluster [DBG] pgmap v7157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:57 smithi115 bash[45495]: audit 2022-04-22T20:19:55.244761+0000 mgr.y (mgr.34107) 5953 : audit [DBG] from='client.41586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:57.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:57 smithi115 bash[45495]: cluster 2022-04-22T20:19:55.249651+0000 mgr.y (mgr.34107) 5954 : cluster [DBG] pgmap v7157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:57 smithi115 bash[46934]: audit 2022-04-22T20:19:55.244761+0000 mgr.y (mgr.34107) 5953 : audit [DBG] from='client.41586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:19:57.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:57 smithi115 bash[46934]: cluster 2022-04-22T20:19:55.249651+0000 mgr.y (mgr.34107) 5954 : cluster [DBG] pgmap v7157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:58 smithi121 bash[36902]: audit 2022-04-22T20:19:57.091423+0000 mon.a (mon.0) 9172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:58 smithi121 bash[36902]: audit 2022-04-22T20:19:57.282719+0000 mon.a (mon.0) 9173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:58 smithi115 bash[45495]: audit 2022-04-22T20:19:57.091423+0000 mon.a (mon.0) 9172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:58 smithi115 bash[45495]: audit 2022-04-22T20:19:57.282719+0000 mon.a (mon.0) 9173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:58 smithi115 bash[46934]: audit 2022-04-22T20:19:57.091423+0000 mon.a (mon.0) 9172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:19:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:58 smithi115 bash[46934]: audit 2022-04-22T20:19:57.282719+0000 mon.a (mon.0) 9173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:19:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:59 smithi121 bash[36902]: cluster 2022-04-22T20:19:57.250757+0000 mgr.y (mgr.34107) 5955 : cluster [DBG] pgmap v7158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:19:59 smithi121 bash[36902]: audit 2022-04-22T20:19:58.720889+0000 mon.a (mon.0) 9174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:59 smithi115 bash[45495]: cluster 2022-04-22T20:19:57.250757+0000 mgr.y (mgr.34107) 5955 : cluster [DBG] pgmap v7158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:19:59 smithi115 bash[45495]: audit 2022-04-22T20:19:58.720889+0000 mon.a (mon.0) 9174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:19:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:59 smithi115 bash[46934]: cluster 2022-04-22T20:19:57.250757+0000 mgr.y (mgr.34107) 5955 : cluster [DBG] pgmap v7158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:19:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:19:59 smithi115 bash[46934]: audit 2022-04-22T20:19:58.720889+0000 mon.a (mon.0) 9174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:00 smithi121 bash[36902]: cluster 2022-04-22T20:20:00.000119+0000 mon.a (mon.0) 9175 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:20:00.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:00 smithi115 bash[45495]: cluster 2022-04-22T20:20:00.000119+0000 mon.a (mon.0) 9175 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:20:00.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:00 smithi115 bash[46934]: cluster 2022-04-22T20:20:00.000119+0000 mon.a (mon.0) 9175 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:20:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:01 smithi121 bash[36902]: cluster 2022-04-22T20:19:59.251942+0000 mgr.y (mgr.34107) 5956 : cluster [DBG] pgmap v7159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:01 smithi115 bash[46934]: cluster 2022-04-22T20:19:59.251942+0000 mgr.y (mgr.34107) 5956 : cluster [DBG] pgmap v7159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:01.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:01 smithi115 bash[45495]: cluster 2022-04-22T20:19:59.251942+0000 mgr.y (mgr.34107) 5956 : cluster [DBG] pgmap v7159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:02.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:03 smithi121 bash[36902]: cluster 2022-04-22T20:20:01.253042+0000 mgr.y (mgr.34107) 5957 : cluster [DBG] pgmap v7160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:03 smithi121 bash[36902]: audit 2022-04-22T20:20:02.095139+0000 mon.a (mon.0) 9176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:03 smithi121 bash[36902]: audit 2022-04-22T20:20:02.285705+0000 mon.a (mon.0) 9177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[45495]: cluster 2022-04-22T20:20:01.253042+0000 mgr.y (mgr.34107) 5957 : cluster [DBG] pgmap v7160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[45495]: audit 2022-04-22T20:20:02.095139+0000 mon.a (mon.0) 9176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:03.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[45495]: audit 2022-04-22T20:20:02.285705+0000 mon.a (mon.0) 9177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[46934]: cluster 2022-04-22T20:20:01.253042+0000 mgr.y (mgr.34107) 5957 : cluster [DBG] pgmap v7160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[46934]: audit 2022-04-22T20:20:02.095139+0000 mon.a (mon.0) 9176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:03 smithi115 bash[46934]: audit 2022-04-22T20:20:02.285705+0000 mon.a (mon.0) 9177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:04 smithi121 bash[36902]: audit 2022-04-22T20:20:03.723996+0000 mon.a (mon.0) 9178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:04 smithi115 bash[45495]: audit 2022-04-22T20:20:03.723996+0000 mon.a (mon.0) 9178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:04.168Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:04 smithi115 bash[46934]: audit 2022-04-22T20:20:03.723996+0000 mon.a (mon.0) 9178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:05 smithi121 bash[36902]: cluster 2022-04-22T20:20:03.254190+0000 mgr.y (mgr.34107) 5958 : cluster [DBG] pgmap v7161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:05 smithi115 bash[45495]: cluster 2022-04-22T20:20:03.254190+0000 mgr.y (mgr.34107) 5958 : cluster [DBG] pgmap v7161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:05 smithi115 bash[46934]: cluster 2022-04-22T20:20:03.254190+0000 mgr.y (mgr.34107) 5958 : cluster [DBG] pgmap v7161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:06 smithi121 bash[36902]: cluster 2022-04-22T20:20:05.255311+0000 mgr.y (mgr.34107) 5959 : cluster [DBG] pgmap v7162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:06 smithi115 bash[45495]: cluster 2022-04-22T20:20:05.255311+0000 mgr.y (mgr.34107) 5959 : cluster [DBG] pgmap v7162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:06 smithi115 bash[46934]: cluster 2022-04-22T20:20:05.255311+0000 mgr.y (mgr.34107) 5959 : cluster [DBG] pgmap v7162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:07 smithi121 bash[36902]: audit 2022-04-22T20:20:07.098394+0000 mon.a (mon.0) 9179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:07.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:07 smithi115 bash[45495]: audit 2022-04-22T20:20:07.098394+0000 mon.a (mon.0) 9179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:07.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:07 smithi115 bash[46934]: audit 2022-04-22T20:20:07.098394+0000 mon.a (mon.0) 9179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:08 smithi121 bash[36902]: cluster 2022-04-22T20:20:07.256448+0000 mgr.y (mgr.34107) 5960 : cluster [DBG] pgmap v7163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:08 smithi121 bash[36902]: audit 2022-04-22T20:20:07.288680+0000 mon.a (mon.0) 9180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:08 smithi115 bash[45495]: cluster 2022-04-22T20:20:07.256448+0000 mgr.y (mgr.34107) 5960 : cluster [DBG] pgmap v7163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:08 smithi115 bash[45495]: audit 2022-04-22T20:20:07.288680+0000 mon.a (mon.0) 9180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:08 smithi115 bash[46934]: cluster 2022-04-22T20:20:07.256448+0000 mgr.y (mgr.34107) 5960 : cluster [DBG] pgmap v7163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:08 smithi115 bash[46934]: audit 2022-04-22T20:20:07.288680+0000 mon.a (mon.0) 9180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:09 smithi121 bash[36902]: audit 2022-04-22T20:20:08.726920+0000 mon.a (mon.0) 9181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:09.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:09 smithi115 bash[45495]: audit 2022-04-22T20:20:08.726920+0000 mon.a (mon.0) 9181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:09.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:09 smithi115 bash[46934]: audit 2022-04-22T20:20:08.726920+0000 mon.a (mon.0) 9181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:10 smithi121 bash[36902]: cluster 2022-04-22T20:20:09.257525+0000 mgr.y (mgr.34107) 5961 : cluster [DBG] pgmap v7164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:10 smithi115 bash[45495]: cluster 2022-04-22T20:20:09.257525+0000 mgr.y (mgr.34107) 5961 : cluster [DBG] pgmap v7164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:10.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:10 smithi115 bash[46934]: cluster 2022-04-22T20:20:09.257525+0000 mgr.y (mgr.34107) 5961 : cluster [DBG] pgmap v7164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:12 smithi121 bash[36902]: cluster 2022-04-22T20:20:11.258636+0000 mgr.y (mgr.34107) 5962 : cluster [DBG] pgmap v7165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:12.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:12 smithi121 bash[36902]: audit 2022-04-22T20:20:12.102191+0000 mon.a (mon.0) 9182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:12.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:12 smithi121 bash[36902]: audit 2022-04-22T20:20:12.291629+0000 mon.a (mon.0) 9183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:12.811 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:12.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:12.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[45495]: cluster 2022-04-22T20:20:11.258636+0000 mgr.y (mgr.34107) 5962 : cluster [DBG] pgmap v7165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:12.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[45495]: audit 2022-04-22T20:20:12.102191+0000 mon.a (mon.0) 9182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:12.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[45495]: audit 2022-04-22T20:20:12.291629+0000 mon.a (mon.0) 9183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:12.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[46934]: cluster 2022-04-22T20:20:11.258636+0000 mgr.y (mgr.34107) 5962 : cluster [DBG] pgmap v7165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:12.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[46934]: audit 2022-04-22T20:20:12.102191+0000 mon.a (mon.0) 9182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:12.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:12 smithi115 bash[46934]: audit 2022-04-22T20:20:12.291629+0000 mon.a (mon.0) 9183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:14.169Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:14 smithi115 bash[45495]: cluster 2022-04-22T20:20:13.259806+0000 mgr.y (mgr.34107) 5963 : cluster [DBG] pgmap v7166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:14 smithi115 bash[45495]: audit 2022-04-22T20:20:13.730069+0000 mon.a (mon.0) 9184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:14 smithi115 bash[46934]: cluster 2022-04-22T20:20:13.259806+0000 mgr.y (mgr.34107) 5963 : cluster [DBG] pgmap v7166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:14 smithi115 bash[46934]: audit 2022-04-22T20:20:13.730069+0000 mon.a (mon.0) 9184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:14 smithi121 bash[36902]: cluster 2022-04-22T20:20:13.259806+0000 mgr.y (mgr.34107) 5963 : cluster [DBG] pgmap v7166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:14 smithi121 bash[36902]: audit 2022-04-22T20:20:13.730069+0000 mon.a (mon.0) 9184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:15 smithi121 bash[36902]: audit 2022-04-22T20:20:15.275907+0000 mon.a (mon.0) 9185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:20:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:15 smithi121 bash[36902]: audit 2022-04-22T20:20:15.276663+0000 mon.a (mon.0) 9186 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:20:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:15 smithi115 bash[45495]: audit 2022-04-22T20:20:15.275907+0000 mon.a (mon.0) 9185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:20:15.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:15 smithi115 bash[45495]: audit 2022-04-22T20:20:15.276663+0000 mon.a (mon.0) 9186 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:20:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:15 smithi115 bash[46934]: audit 2022-04-22T20:20:15.275907+0000 mon.a (mon.0) 9185 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:20:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:15 smithi115 bash[46934]: audit 2022-04-22T20:20:15.276663+0000 mon.a (mon.0) 9186 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:20:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:16 smithi121 bash[36902]: cluster 2022-04-22T20:20:15.260831+0000 mgr.y (mgr.34107) 5964 : cluster [DBG] pgmap v7167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:16.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:16 smithi115 bash[45495]: cluster 2022-04-22T20:20:15.260831+0000 mgr.y (mgr.34107) 5964 : cluster [DBG] pgmap v7167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:16.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:16 smithi115 bash[46934]: cluster 2022-04-22T20:20:15.260831+0000 mgr.y (mgr.34107) 5964 : cluster [DBG] pgmap v7167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:17 smithi121 bash[36902]: audit 2022-04-22T20:20:17.106040+0000 mon.a (mon.0) 9187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:17 smithi121 bash[36902]: audit 2022-04-22T20:20:17.294768+0000 mon.a (mon.0) 9188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:17 smithi115 bash[45495]: audit 2022-04-22T20:20:17.106040+0000 mon.a (mon.0) 9187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:17.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:17 smithi115 bash[45495]: audit 2022-04-22T20:20:17.294768+0000 mon.a (mon.0) 9188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:17 smithi115 bash[46934]: audit 2022-04-22T20:20:17.106040+0000 mon.a (mon.0) 9187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:17.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:17 smithi115 bash[46934]: audit 2022-04-22T20:20:17.294768+0000 mon.a (mon.0) 9188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:18 smithi121 bash[36902]: cluster 2022-04-22T20:20:17.261871+0000 mgr.y (mgr.34107) 5965 : cluster [DBG] pgmap v7168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:18.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:18 smithi115 bash[45495]: cluster 2022-04-22T20:20:17.261871+0000 mgr.y (mgr.34107) 5965 : cluster [DBG] pgmap v7168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:18.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:18 smithi115 bash[46934]: cluster 2022-04-22T20:20:17.261871+0000 mgr.y (mgr.34107) 5965 : cluster [DBG] pgmap v7168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:19 smithi121 bash[36902]: audit 2022-04-22T20:20:18.733207+0000 mon.a (mon.0) 9189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:19 smithi115 bash[45495]: audit 2022-04-22T20:20:18.733207+0000 mon.a (mon.0) 9189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:19 smithi115 bash[46934]: audit 2022-04-22T20:20:18.733207+0000 mon.a (mon.0) 9189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:20 smithi121 bash[36902]: cluster 2022-04-22T20:20:19.262964+0000 mgr.y (mgr.34107) 5966 : cluster [DBG] pgmap v7169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:21.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:20 smithi115 bash[45495]: cluster 2022-04-22T20:20:19.262964+0000 mgr.y (mgr.34107) 5966 : cluster [DBG] pgmap v7169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:21.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:20 smithi115 bash[46934]: cluster 2022-04-22T20:20:19.262964+0000 mgr.y (mgr.34107) 5966 : cluster [DBG] pgmap v7169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:22.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:22.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:22 smithi121 bash[36902]: cluster 2022-04-22T20:20:21.264174+0000 mgr.y (mgr.34107) 5967 : cluster [DBG] pgmap v7170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:22 smithi121 bash[36902]: audit 2022-04-22T20:20:22.109631+0000 mon.a (mon.0) 9190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:22 smithi121 bash[36902]: audit 2022-04-22T20:20:22.297992+0000 mon.a (mon.0) 9191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[45495]: cluster 2022-04-22T20:20:21.264174+0000 mgr.y (mgr.34107) 5967 : cluster [DBG] pgmap v7170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[45495]: audit 2022-04-22T20:20:22.109631+0000 mon.a (mon.0) 9190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[45495]: audit 2022-04-22T20:20:22.297992+0000 mon.a (mon.0) 9191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[46934]: cluster 2022-04-22T20:20:21.264174+0000 mgr.y (mgr.34107) 5967 : cluster [DBG] pgmap v7170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:23.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[46934]: audit 2022-04-22T20:20:22.109631+0000 mon.a (mon.0) 9190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:22 smithi115 bash[46934]: audit 2022-04-22T20:20:22.297992+0000 mon.a (mon.0) 9191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:23 smithi121 bash[36902]: audit 2022-04-22T20:20:23.729414+0000 mon.a (mon.0) 9192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:20:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:23 smithi121 bash[36902]: audit 2022-04-22T20:20:23.732659+0000 mon.a (mon.0) 9193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:20:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:23 smithi121 bash[36902]: audit 2022-04-22T20:20:23.734491+0000 mon.a (mon.0) 9194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:20:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:23 smithi121 bash[36902]: audit 2022-04-22T20:20:23.736320+0000 mon.a (mon.0) 9195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[46934]: audit 2022-04-22T20:20:23.729414+0000 mon.a (mon.0) 9192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:20:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[46934]: audit 2022-04-22T20:20:23.732659+0000 mon.a (mon.0) 9193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:20:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[46934]: audit 2022-04-22T20:20:23.734491+0000 mon.a (mon.0) 9194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:20:24.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[46934]: audit 2022-04-22T20:20:23.736320+0000 mon.a (mon.0) 9195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[45495]: audit 2022-04-22T20:20:23.729414+0000 mon.a (mon.0) 9192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:20:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[45495]: audit 2022-04-22T20:20:23.732659+0000 mon.a (mon.0) 9193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:20:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[45495]: audit 2022-04-22T20:20:23.734491+0000 mon.a (mon.0) 9194 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:20:24.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:23 smithi115 bash[45495]: audit 2022-04-22T20:20:23.736320+0000 mon.a (mon.0) 9195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:24.169Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:25 smithi121 bash[36902]: cluster 2022-04-22T20:20:23.265350+0000 mgr.y (mgr.34107) 5968 : cluster [DBG] pgmap v7171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:25 smithi121 bash[36902]: audit 2022-04-22T20:20:24.103032+0000 mon.a (mon.0) 9196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:20:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:25 smithi115 bash[45495]: cluster 2022-04-22T20:20:23.265350+0000 mgr.y (mgr.34107) 5968 : cluster [DBG] pgmap v7171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:25 smithi115 bash[45495]: audit 2022-04-22T20:20:24.103032+0000 mon.a (mon.0) 9196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:20:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:25 smithi115 bash[46934]: cluster 2022-04-22T20:20:23.265350+0000 mgr.y (mgr.34107) 5968 : cluster [DBG] pgmap v7171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:25 smithi115 bash[46934]: audit 2022-04-22T20:20:24.103032+0000 mon.a (mon.0) 9196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:20:26.056 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:20:26.406 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:20:26.406 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:20:26.406 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:20:26.406 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:20:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 3h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 2h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:20:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:20:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:26 smithi121 bash[36902]: cluster 2022-04-22T20:20:25.266481+0000 mgr.y (mgr.34107) 5969 : cluster [DBG] pgmap v7172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:26 smithi121 bash[36902]: audit 2022-04-22T20:20:26.031023+0000 mgr.y (mgr.34107) 5970 : audit [DBG] from='client.41598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:26 smithi115 bash[45495]: cluster 2022-04-22T20:20:25.266481+0000 mgr.y (mgr.34107) 5969 : cluster [DBG] pgmap v7172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:26 smithi115 bash[45495]: audit 2022-04-22T20:20:26.031023+0000 mgr.y (mgr.34107) 5970 : audit [DBG] from='client.41598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:26 smithi115 bash[46934]: cluster 2022-04-22T20:20:25.266481+0000 mgr.y (mgr.34107) 5969 : cluster [DBG] pgmap v7172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:26.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:26 smithi115 bash[46934]: audit 2022-04-22T20:20:26.031023+0000 mgr.y (mgr.34107) 5970 : audit [DBG] from='client.41598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:26.813 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:20:26.813 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:20:26.813 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:20:26.814 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:20:26.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:26.815 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:20:26.815 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:20:26.815 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:20:26.815 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:20:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:27 smithi121 bash[36902]: audit 2022-04-22T20:20:26.377931+0000 mgr.y (mgr.34107) 5971 : audit [DBG] from='client.41604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:27 smithi121 bash[36902]: audit 2022-04-22T20:20:26.804498+0000 mon.c (mon.1) 237 : audit [DBG] from='client.? 172.21.15.115:0/1527078735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:27 smithi115 bash[45495]: audit 2022-04-22T20:20:26.377931+0000 mgr.y (mgr.34107) 5971 : audit [DBG] from='client.41604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:27 smithi115 bash[45495]: audit 2022-04-22T20:20:26.804498+0000 mon.c (mon.1) 237 : audit [DBG] from='client.? 172.21.15.115:0/1527078735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:27 smithi115 bash[46934]: audit 2022-04-22T20:20:26.377931+0000 mgr.y (mgr.34107) 5971 : audit [DBG] from='client.41604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:27.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:27 smithi115 bash[46934]: audit 2022-04-22T20:20:26.804498+0000 mon.c (mon.1) 237 : audit [DBG] from='client.? 172.21.15.115:0/1527078735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:28 smithi121 bash[36902]: audit 2022-04-22T20:20:27.116642+0000 mon.a (mon.0) 9197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:28 smithi121 bash[36902]: cluster 2022-04-22T20:20:27.267583+0000 mgr.y (mgr.34107) 5972 : cluster [DBG] pgmap v7173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:28 smithi121 bash[36902]: audit 2022-04-22T20:20:27.300748+0000 mon.a (mon.0) 9198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[46934]: audit 2022-04-22T20:20:27.116642+0000 mon.a (mon.0) 9197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:28.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[46934]: cluster 2022-04-22T20:20:27.267583+0000 mgr.y (mgr.34107) 5972 : cluster [DBG] pgmap v7173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:28.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[46934]: audit 2022-04-22T20:20:27.300748+0000 mon.a (mon.0) 9198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[45495]: audit 2022-04-22T20:20:27.116642+0000 mon.a (mon.0) 9197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:28.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[45495]: cluster 2022-04-22T20:20:27.267583+0000 mgr.y (mgr.34107) 5972 : cluster [DBG] pgmap v7173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:28.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:28 smithi115 bash[45495]: audit 2022-04-22T20:20:27.300748+0000 mon.a (mon.0) 9198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:29 smithi121 bash[36902]: audit 2022-04-22T20:20:28.739672+0000 mon.a (mon.0) 9199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:29 smithi115 bash[45495]: audit 2022-04-22T20:20:28.739672+0000 mon.a (mon.0) 9199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:29 smithi115 bash[46934]: audit 2022-04-22T20:20:28.739672+0000 mon.a (mon.0) 9199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:30 smithi121 bash[36902]: cluster 2022-04-22T20:20:29.268624+0000 mgr.y (mgr.34107) 5973 : cluster [DBG] pgmap v7174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:30 smithi115 bash[46934]: cluster 2022-04-22T20:20:29.268624+0000 mgr.y (mgr.34107) 5973 : cluster [DBG] pgmap v7174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:30.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:30 smithi115 bash[45495]: cluster 2022-04-22T20:20:29.268624+0000 mgr.y (mgr.34107) 5973 : cluster [DBG] pgmap v7174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:32 smithi121 bash[36902]: cluster 2022-04-22T20:20:31.269694+0000 mgr.y (mgr.34107) 5974 : cluster [DBG] pgmap v7175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:32 smithi121 bash[36902]: audit 2022-04-22T20:20:32.116610+0000 mon.a (mon.0) 9200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:32 smithi121 bash[36902]: audit 2022-04-22T20:20:32.303919+0000 mon.a (mon.0) 9201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:32.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:32.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[45495]: cluster 2022-04-22T20:20:31.269694+0000 mgr.y (mgr.34107) 5974 : cluster [DBG] pgmap v7175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[45495]: audit 2022-04-22T20:20:32.116610+0000 mon.a (mon.0) 9200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[45495]: audit 2022-04-22T20:20:32.303919+0000 mon.a (mon.0) 9201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:32.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[46934]: cluster 2022-04-22T20:20:31.269694+0000 mgr.y (mgr.34107) 5974 : cluster [DBG] pgmap v7175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:32.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[46934]: audit 2022-04-22T20:20:32.116610+0000 mon.a (mon.0) 9200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:32.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:32 smithi115 bash[46934]: audit 2022-04-22T20:20:32.303919+0000 mon.a (mon.0) 9201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:34.169Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:34 smithi115 bash[45495]: cluster 2022-04-22T20:20:33.270814+0000 mgr.y (mgr.34107) 5975 : cluster [DBG] pgmap v7176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:34 smithi115 bash[45495]: audit 2022-04-22T20:20:33.742626+0000 mon.a (mon.0) 9202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:34 smithi115 bash[46934]: cluster 2022-04-22T20:20:33.270814+0000 mgr.y (mgr.34107) 5975 : cluster [DBG] pgmap v7176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:34 smithi115 bash[46934]: audit 2022-04-22T20:20:33.742626+0000 mon.a (mon.0) 9202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:34 smithi121 bash[36902]: cluster 2022-04-22T20:20:33.270814+0000 mgr.y (mgr.34107) 5975 : cluster [DBG] pgmap v7176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:34 smithi121 bash[36902]: audit 2022-04-22T20:20:33.742626+0000 mon.a (mon.0) 9202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:36 smithi121 bash[36902]: cluster 2022-04-22T20:20:35.271807+0000 mgr.y (mgr.34107) 5976 : cluster [DBG] pgmap v7177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:36.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:36 smithi115 bash[45495]: cluster 2022-04-22T20:20:35.271807+0000 mgr.y (mgr.34107) 5976 : cluster [DBG] pgmap v7177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:36.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:36 smithi115 bash[46934]: cluster 2022-04-22T20:20:35.271807+0000 mgr.y (mgr.34107) 5976 : cluster [DBG] pgmap v7177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:37 smithi121 bash[36902]: audit 2022-04-22T20:20:37.120176+0000 mon.a (mon.0) 9203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:37 smithi121 bash[36902]: audit 2022-04-22T20:20:37.306701+0000 mon.a (mon.0) 9204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:37 smithi115 bash[45495]: audit 2022-04-22T20:20:37.120176+0000 mon.a (mon.0) 9203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:37 smithi115 bash[45495]: audit 2022-04-22T20:20:37.306701+0000 mon.a (mon.0) 9204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:37 smithi115 bash[46934]: audit 2022-04-22T20:20:37.120176+0000 mon.a (mon.0) 9203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:37 smithi115 bash[46934]: audit 2022-04-22T20:20:37.306701+0000 mon.a (mon.0) 9204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:38 smithi121 bash[36902]: cluster 2022-04-22T20:20:37.272818+0000 mgr.y (mgr.34107) 5977 : cluster [DBG] pgmap v7178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:38.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:38 smithi115 bash[45495]: cluster 2022-04-22T20:20:37.272818+0000 mgr.y (mgr.34107) 5977 : cluster [DBG] pgmap v7178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:38 smithi115 bash[46934]: cluster 2022-04-22T20:20:37.272818+0000 mgr.y (mgr.34107) 5977 : cluster [DBG] pgmap v7178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:39 smithi121 bash[36902]: audit 2022-04-22T20:20:38.745750+0000 mon.a (mon.0) 9205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:39 smithi115 bash[46934]: audit 2022-04-22T20:20:38.745750+0000 mon.a (mon.0) 9205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:39 smithi115 bash[45495]: audit 2022-04-22T20:20:38.745750+0000 mon.a (mon.0) 9205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:40 smithi121 bash[36902]: cluster 2022-04-22T20:20:39.273890+0000 mgr.y (mgr.34107) 5978 : cluster [DBG] pgmap v7179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:41.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:40 smithi115 bash[45495]: cluster 2022-04-22T20:20:39.273890+0000 mgr.y (mgr.34107) 5978 : cluster [DBG] pgmap v7179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:40 smithi115 bash[46934]: cluster 2022-04-22T20:20:39.273890+0000 mgr.y (mgr.34107) 5978 : cluster [DBG] pgmap v7179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:42.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:42.288Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:42 smithi121 bash[36902]: cluster 2022-04-22T20:20:41.274977+0000 mgr.y (mgr.34107) 5979 : cluster [DBG] pgmap v7180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:42 smithi121 bash[36902]: audit 2022-04-22T20:20:42.123756+0000 mon.a (mon.0) 9206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:42 smithi121 bash[36902]: audit 2022-04-22T20:20:42.309871+0000 mon.a (mon.0) 9207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[45495]: cluster 2022-04-22T20:20:41.274977+0000 mgr.y (mgr.34107) 5979 : cluster [DBG] pgmap v7180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[45495]: audit 2022-04-22T20:20:42.123756+0000 mon.a (mon.0) 9206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[45495]: audit 2022-04-22T20:20:42.309871+0000 mon.a (mon.0) 9207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[46934]: cluster 2022-04-22T20:20:41.274977+0000 mgr.y (mgr.34107) 5979 : cluster [DBG] pgmap v7180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[46934]: audit 2022-04-22T20:20:42.123756+0000 mon.a (mon.0) 9206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:42 smithi115 bash[46934]: audit 2022-04-22T20:20:42.309871+0000 mon.a (mon.0) 9207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:43 smithi121 bash[36902]: audit 2022-04-22T20:20:43.748919+0000 mon.a (mon.0) 9208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:44.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:43 smithi115 bash[45495]: audit 2022-04-22T20:20:43.748919+0000 mon.a (mon.0) 9208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:44.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:43 smithi115 bash[46934]: audit 2022-04-22T20:20:43.748919+0000 mon.a (mon.0) 9208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:44.298 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:44.171Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:44 smithi121 bash[36902]: cluster 2022-04-22T20:20:43.276147+0000 mgr.y (mgr.34107) 5980 : cluster [DBG] pgmap v7181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:45.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:44 smithi115 bash[45495]: cluster 2022-04-22T20:20:43.276147+0000 mgr.y (mgr.34107) 5980 : cluster [DBG] pgmap v7181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:44 smithi115 bash[46934]: cluster 2022-04-22T20:20:43.276147+0000 mgr.y (mgr.34107) 5980 : cluster [DBG] pgmap v7181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:46 smithi121 bash[36902]: cluster 2022-04-22T20:20:45.277177+0000 mgr.y (mgr.34107) 5981 : cluster [DBG] pgmap v7182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:46 smithi115 bash[45495]: cluster 2022-04-22T20:20:45.277177+0000 mgr.y (mgr.34107) 5981 : cluster [DBG] pgmap v7182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:46 smithi115 bash[46934]: cluster 2022-04-22T20:20:45.277177+0000 mgr.y (mgr.34107) 5981 : cluster [DBG] pgmap v7182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:47 smithi121 bash[36902]: audit 2022-04-22T20:20:47.127704+0000 mon.a (mon.0) 9209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:47 smithi121 bash[36902]: audit 2022-04-22T20:20:47.312791+0000 mon.a (mon.0) 9210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:47 smithi115 bash[45495]: audit 2022-04-22T20:20:47.127704+0000 mon.a (mon.0) 9209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:47 smithi115 bash[45495]: audit 2022-04-22T20:20:47.312791+0000 mon.a (mon.0) 9210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:47 smithi115 bash[46934]: audit 2022-04-22T20:20:47.127704+0000 mon.a (mon.0) 9209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:47 smithi115 bash[46934]: audit 2022-04-22T20:20:47.312791+0000 mon.a (mon.0) 9210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:48 smithi121 bash[36902]: cluster 2022-04-22T20:20:47.278154+0000 mgr.y (mgr.34107) 5982 : cluster [DBG] pgmap v7183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:48 smithi121 bash[36902]: audit 2022-04-22T20:20:48.751740+0000 mon.a (mon.0) 9211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:48 smithi115 bash[45495]: cluster 2022-04-22T20:20:47.278154+0000 mgr.y (mgr.34107) 5982 : cluster [DBG] pgmap v7183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:48 smithi115 bash[45495]: audit 2022-04-22T20:20:48.751740+0000 mon.a (mon.0) 9211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:48 smithi115 bash[46934]: cluster 2022-04-22T20:20:47.278154+0000 mgr.y (mgr.34107) 5982 : cluster [DBG] pgmap v7183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:48 smithi115 bash[46934]: audit 2022-04-22T20:20:48.751740+0000 mon.a (mon.0) 9211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:50 smithi121 bash[36902]: cluster 2022-04-22T20:20:49.279252+0000 mgr.y (mgr.34107) 5983 : cluster [DBG] pgmap v7184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:50 smithi115 bash[45495]: cluster 2022-04-22T20:20:49.279252+0000 mgr.y (mgr.34107) 5983 : cluster [DBG] pgmap v7184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:51.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:50 smithi115 bash[46934]: cluster 2022-04-22T20:20:49.279252+0000 mgr.y (mgr.34107) 5983 : cluster [DBG] pgmap v7184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:52.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:52.290Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:52 smithi121 bash[36902]: cluster 2022-04-22T20:20:51.280336+0000 mgr.y (mgr.34107) 5984 : cluster [DBG] pgmap v7185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:52 smithi121 bash[36902]: audit 2022-04-22T20:20:52.131319+0000 mon.a (mon.0) 9212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:52 smithi121 bash[36902]: audit 2022-04-22T20:20:52.316053+0000 mon.a (mon.0) 9213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[45495]: cluster 2022-04-22T20:20:51.280336+0000 mgr.y (mgr.34107) 5984 : cluster [DBG] pgmap v7185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[45495]: audit 2022-04-22T20:20:52.131319+0000 mon.a (mon.0) 9212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[45495]: audit 2022-04-22T20:20:52.316053+0000 mon.a (mon.0) 9213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[46934]: cluster 2022-04-22T20:20:51.280336+0000 mgr.y (mgr.34107) 5984 : cluster [DBG] pgmap v7185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[46934]: audit 2022-04-22T20:20:52.131319+0000 mon.a (mon.0) 9212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:52 smithi115 bash[46934]: audit 2022-04-22T20:20:52.316053+0000 mon.a (mon.0) 9213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:53 smithi121 bash[36902]: audit 2022-04-22T20:20:53.754768+0000 mon.a (mon.0) 9214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:53 smithi115 bash[45495]: audit 2022-04-22T20:20:53.754768+0000 mon.a (mon.0) 9214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:20:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:20:54.170Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:53 smithi115 bash[46934]: audit 2022-04-22T20:20:53.754768+0000 mon.a (mon.0) 9214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:54 smithi121 bash[36902]: cluster 2022-04-22T20:20:53.281432+0000 mgr.y (mgr.34107) 5985 : cluster [DBG] pgmap v7186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:54 smithi115 bash[45495]: cluster 2022-04-22T20:20:53.281432+0000 mgr.y (mgr.34107) 5985 : cluster [DBG] pgmap v7186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:54 smithi115 bash[46934]: cluster 2022-04-22T20:20:53.281432+0000 mgr.y (mgr.34107) 5985 : cluster [DBG] pgmap v7186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:57.190 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:20:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:56 smithi121 bash[36902]: cluster 2022-04-22T20:20:55.282537+0000 mgr.y (mgr.34107) 5986 : cluster [DBG] pgmap v7187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:56 smithi115 bash[46934]: cluster 2022-04-22T20:20:55.282537+0000 mgr.y (mgr.34107) 5986 : cluster [DBG] pgmap v7187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:56 smithi115 bash[45495]: cluster 2022-04-22T20:20:55.282537+0000 mgr.y (mgr.34107) 5986 : cluster [DBG] pgmap v7187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:57.530 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 9m ago 2h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 2h 21.8M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 8m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 8m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:20:57.531 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 9m ago 3h 202M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 8m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 2h 10.1M - 1dbe0e931976 f4b044642a96 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 8m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 3h 576M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:20:57.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 2h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:20:57.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:20:57.533 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 2h 71.8M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:20:57.930 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:57.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:20:57.932 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:20:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:57 smithi121 bash[36902]: audit 2022-04-22T20:20:57.135184+0000 mon.a (mon.0) 9215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:57 smithi121 bash[36902]: audit 2022-04-22T20:20:57.318995+0000 mon.a (mon.0) 9216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:57 smithi121 bash[36902]: audit 2022-04-22T20:20:57.921580+0000 mon.c (mon.1) 238 : audit [DBG] from='client.? 172.21.15.115:0/1571951262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[45495]: audit 2022-04-22T20:20:57.135184+0000 mon.a (mon.0) 9215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[45495]: audit 2022-04-22T20:20:57.318995+0000 mon.a (mon.0) 9216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:58.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[45495]: audit 2022-04-22T20:20:57.921580+0000 mon.c (mon.1) 238 : audit [DBG] from='client.? 172.21.15.115:0/1571951262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:58.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[46934]: audit 2022-04-22T20:20:57.135184+0000 mon.a (mon.0) 9215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:20:58.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[46934]: audit 2022-04-22T20:20:57.318995+0000 mon.a (mon.0) 9216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:20:58.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:57 smithi115 bash[46934]: audit 2022-04-22T20:20:57.921580+0000 mon.c (mon.1) 238 : audit [DBG] from='client.? 172.21.15.115:0/1571951262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:20:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:58 smithi121 bash[36902]: audit 2022-04-22T20:20:57.166398+0000 mgr.y (mgr.34107) 5987 : audit [DBG] from='client.51550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:58 smithi121 bash[36902]: cluster 2022-04-22T20:20:57.283609+0000 mgr.y (mgr.34107) 5988 : cluster [DBG] pgmap v7188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:59.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:58 smithi121 bash[36902]: audit 2022-04-22T20:20:57.503392+0000 mgr.y (mgr.34107) 5989 : audit [DBG] from='client.41622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:20:58 smithi121 bash[36902]: audit 2022-04-22T20:20:58.757686+0000 mon.a (mon.0) 9217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:59.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[45495]: audit 2022-04-22T20:20:57.166398+0000 mgr.y (mgr.34107) 5987 : audit [DBG] from='client.51550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[45495]: cluster 2022-04-22T20:20:57.283609+0000 mgr.y (mgr.34107) 5988 : cluster [DBG] pgmap v7188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:59.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[45495]: audit 2022-04-22T20:20:57.503392+0000 mgr.y (mgr.34107) 5989 : audit [DBG] from='client.41622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.321 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[45495]: audit 2022-04-22T20:20:58.757686+0000 mon.a (mon.0) 9217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:20:59.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[46934]: audit 2022-04-22T20:20:57.166398+0000 mgr.y (mgr.34107) 5987 : audit [DBG] from='client.51550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[46934]: cluster 2022-04-22T20:20:57.283609+0000 mgr.y (mgr.34107) 5988 : cluster [DBG] pgmap v7188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:20:59.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[46934]: audit 2022-04-22T20:20:57.503392+0000 mgr.y (mgr.34107) 5989 : audit [DBG] from='client.41622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:20:59.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:20:58 smithi115 bash[46934]: audit 2022-04-22T20:20:58.757686+0000 mon.a (mon.0) 9217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:00 smithi121 bash[36902]: cluster 2022-04-22T20:20:59.284780+0000 mgr.y (mgr.34107) 5990 : cluster [DBG] pgmap v7189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:00 smithi115 bash[45495]: cluster 2022-04-22T20:20:59.284780+0000 mgr.y (mgr.34107) 5990 : cluster [DBG] pgmap v7189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:00 smithi115 bash[46934]: cluster 2022-04-22T20:20:59.284780+0000 mgr.y (mgr.34107) 5990 : cluster [DBG] pgmap v7189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:02.290Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:03 smithi121 bash[36902]: cluster 2022-04-22T20:21:01.285916+0000 mgr.y (mgr.34107) 5991 : cluster [DBG] pgmap v7190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:03 smithi121 bash[36902]: audit 2022-04-22T20:21:02.138722+0000 mon.a (mon.0) 9218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:03 smithi121 bash[36902]: audit 2022-04-22T20:21:02.322062+0000 mon.a (mon.0) 9219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:03 smithi115 bash[46934]: cluster 2022-04-22T20:21:01.285916+0000 mgr.y (mgr.34107) 5991 : cluster [DBG] pgmap v7190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:03 smithi115 bash[46934]: audit 2022-04-22T20:21:02.138722+0000 mon.a (mon.0) 9218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:03 smithi115 bash[46934]: audit 2022-04-22T20:21:02.322062+0000 mon.a (mon.0) 9219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:02 smithi115 bash[45495]: cluster 2022-04-22T20:21:01.285916+0000 mgr.y (mgr.34107) 5991 : cluster [DBG] pgmap v7190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:02 smithi115 bash[45495]: audit 2022-04-22T20:21:02.138722+0000 mon.a (mon.0) 9218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:03.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:02 smithi115 bash[45495]: audit 2022-04-22T20:21:02.322062+0000 mon.a (mon.0) 9219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:04 smithi115 bash[45495]: audit 2022-04-22T20:21:03.760770+0000 mon.a (mon.0) 9220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:04.317 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:04.170Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:04.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:04 smithi115 bash[46934]: audit 2022-04-22T20:21:03.760770+0000 mon.a (mon.0) 9220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:04 smithi121 bash[36902]: audit 2022-04-22T20:21:03.760770+0000 mon.a (mon.0) 9220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:05.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:05 smithi115 bash[45495]: cluster 2022-04-22T20:21:03.287038+0000 mgr.y (mgr.34107) 5992 : cluster [DBG] pgmap v7191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:05 smithi115 bash[46934]: cluster 2022-04-22T20:21:03.287038+0000 mgr.y (mgr.34107) 5992 : cluster [DBG] pgmap v7191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:05 smithi121 bash[36902]: cluster 2022-04-22T20:21:03.287038+0000 mgr.y (mgr.34107) 5992 : cluster [DBG] pgmap v7191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:07 smithi115 bash[45495]: cluster 2022-04-22T20:21:05.288301+0000 mgr.y (mgr.34107) 5993 : cluster [DBG] pgmap v7192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:07.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:07 smithi115 bash[46934]: cluster 2022-04-22T20:21:05.288301+0000 mgr.y (mgr.34107) 5993 : cluster [DBG] pgmap v7192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:07 smithi121 bash[36902]: cluster 2022-04-22T20:21:05.288301+0000 mgr.y (mgr.34107) 5993 : cluster [DBG] pgmap v7192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:08 smithi115 bash[45495]: audit 2022-04-22T20:21:07.142288+0000 mon.a (mon.0) 9221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:08 smithi115 bash[45495]: audit 2022-04-22T20:21:07.325036+0000 mon.a (mon.0) 9222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:08 smithi115 bash[46934]: audit 2022-04-22T20:21:07.142288+0000 mon.a (mon.0) 9221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:08 smithi115 bash[46934]: audit 2022-04-22T20:21:07.325036+0000 mon.a (mon.0) 9222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:08 smithi121 bash[36902]: audit 2022-04-22T20:21:07.142288+0000 mon.a (mon.0) 9221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:08 smithi121 bash[36902]: audit 2022-04-22T20:21:07.325036+0000 mon.a (mon.0) 9222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:09 smithi115 bash[45495]: cluster 2022-04-22T20:21:07.289300+0000 mgr.y (mgr.34107) 5994 : cluster [DBG] pgmap v7193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:09 smithi115 bash[45495]: audit 2022-04-22T20:21:08.763834+0000 mon.a (mon.0) 9223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:09 smithi115 bash[46934]: cluster 2022-04-22T20:21:07.289300+0000 mgr.y (mgr.34107) 5994 : cluster [DBG] pgmap v7193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:09.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:09 smithi115 bash[46934]: audit 2022-04-22T20:21:08.763834+0000 mon.a (mon.0) 9223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:09 smithi121 bash[36902]: cluster 2022-04-22T20:21:07.289300+0000 mgr.y (mgr.34107) 5994 : cluster [DBG] pgmap v7193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:09 smithi121 bash[36902]: audit 2022-04-22T20:21:08.763834+0000 mon.a (mon.0) 9223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:11 smithi115 bash[45495]: cluster 2022-04-22T20:21:09.290509+0000 mgr.y (mgr.34107) 5995 : cluster [DBG] pgmap v7194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:11 smithi115 bash[46934]: cluster 2022-04-22T20:21:09.290509+0000 mgr.y (mgr.34107) 5995 : cluster [DBG] pgmap v7194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:11 smithi121 bash[36902]: cluster 2022-04-22T20:21:09.290509+0000 mgr.y (mgr.34107) 5995 : cluster [DBG] pgmap v7194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:12.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:12.290Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[45495]: cluster 2022-04-22T20:21:11.291647+0000 mgr.y (mgr.34107) 5996 : cluster [DBG] pgmap v7195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[45495]: audit 2022-04-22T20:21:12.145648+0000 mon.a (mon.0) 9224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:13.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[45495]: audit 2022-04-22T20:21:12.327932+0000 mon.a (mon.0) 9225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[46934]: cluster 2022-04-22T20:21:11.291647+0000 mgr.y (mgr.34107) 5996 : cluster [DBG] pgmap v7195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[46934]: audit 2022-04-22T20:21:12.145648+0000 mon.a (mon.0) 9224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:13.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:13 smithi115 bash[46934]: audit 2022-04-22T20:21:12.327932+0000 mon.a (mon.0) 9225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:13 smithi121 bash[36902]: cluster 2022-04-22T20:21:11.291647+0000 mgr.y (mgr.34107) 5996 : cluster [DBG] pgmap v7195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:13 smithi121 bash[36902]: audit 2022-04-22T20:21:12.145648+0000 mon.a (mon.0) 9224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:13 smithi121 bash[36902]: audit 2022-04-22T20:21:12.327932+0000 mon.a (mon.0) 9225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:14 smithi121 bash[36902]: audit 2022-04-22T20:21:13.766777+0000 mon.a (mon.0) 9226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:14 smithi115 bash[45495]: audit 2022-04-22T20:21:13.766777+0000 mon.a (mon.0) 9226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:14 smithi115 bash[46934]: audit 2022-04-22T20:21:13.766777+0000 mon.a (mon.0) 9226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:14.170Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:15 smithi121 bash[36902]: cluster 2022-04-22T20:21:13.292871+0000 mgr.y (mgr.34107) 5997 : cluster [DBG] pgmap v7196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:15 smithi115 bash[46934]: cluster 2022-04-22T20:21:13.292871+0000 mgr.y (mgr.34107) 5997 : cluster [DBG] pgmap v7196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:15 smithi115 bash[45495]: cluster 2022-04-22T20:21:13.292871+0000 mgr.y (mgr.34107) 5997 : cluster [DBG] pgmap v7196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:16 smithi121 bash[36902]: audit 2022-04-22T20:21:15.288957+0000 mon.a (mon.0) 9227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:21:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:16 smithi121 bash[36902]: audit 2022-04-22T20:21:15.289765+0000 mon.a (mon.0) 9228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:21:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:16 smithi115 bash[45495]: audit 2022-04-22T20:21:15.288957+0000 mon.a (mon.0) 9227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:21:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:16 smithi115 bash[45495]: audit 2022-04-22T20:21:15.289765+0000 mon.a (mon.0) 9228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:21:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:16 smithi115 bash[46934]: audit 2022-04-22T20:21:15.288957+0000 mon.a (mon.0) 9227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:21:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:16 smithi115 bash[46934]: audit 2022-04-22T20:21:15.289765+0000 mon.a (mon.0) 9228 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:21:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:17 smithi121 bash[36902]: cluster 2022-04-22T20:21:15.293826+0000 mgr.y (mgr.34107) 5998 : cluster [DBG] pgmap v7197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:17 smithi115 bash[45495]: cluster 2022-04-22T20:21:15.293826+0000 mgr.y (mgr.34107) 5998 : cluster [DBG] pgmap v7197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:17 smithi115 bash[46934]: cluster 2022-04-22T20:21:15.293826+0000 mgr.y (mgr.34107) 5998 : cluster [DBG] pgmap v7197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:18 smithi121 bash[36902]: audit 2022-04-22T20:21:17.149310+0000 mon.a (mon.0) 9229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:18 smithi121 bash[36902]: audit 2022-04-22T20:21:17.330937+0000 mon.a (mon.0) 9230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:18 smithi115 bash[45495]: audit 2022-04-22T20:21:17.149310+0000 mon.a (mon.0) 9229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:18 smithi115 bash[45495]: audit 2022-04-22T20:21:17.330937+0000 mon.a (mon.0) 9230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:18 smithi115 bash[46934]: audit 2022-04-22T20:21:17.149310+0000 mon.a (mon.0) 9229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:18 smithi115 bash[46934]: audit 2022-04-22T20:21:17.330937+0000 mon.a (mon.0) 9230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:19 smithi121 bash[36902]: cluster 2022-04-22T20:21:17.294988+0000 mgr.y (mgr.34107) 5999 : cluster [DBG] pgmap v7198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:19 smithi121 bash[36902]: audit 2022-04-22T20:21:18.770187+0000 mon.a (mon.0) 9231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:19 smithi115 bash[45495]: cluster 2022-04-22T20:21:17.294988+0000 mgr.y (mgr.34107) 5999 : cluster [DBG] pgmap v7198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:19.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:19 smithi115 bash[45495]: audit 2022-04-22T20:21:18.770187+0000 mon.a (mon.0) 9231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:19.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:19 smithi115 bash[46934]: cluster 2022-04-22T20:21:17.294988+0000 mgr.y (mgr.34107) 5999 : cluster [DBG] pgmap v7198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:19.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:19 smithi115 bash[46934]: audit 2022-04-22T20:21:18.770187+0000 mon.a (mon.0) 9231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:21 smithi121 bash[36902]: cluster 2022-04-22T20:21:19.296073+0000 mgr.y (mgr.34107) 6000 : cluster [DBG] pgmap v7199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:21 smithi115 bash[45495]: cluster 2022-04-22T20:21:19.296073+0000 mgr.y (mgr.34107) 6000 : cluster [DBG] pgmap v7199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:21 smithi115 bash[46934]: cluster 2022-04-22T20:21:19.296073+0000 mgr.y (mgr.34107) 6000 : cluster [DBG] pgmap v7199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:22.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:22.292Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:23 smithi121 bash[36902]: cluster 2022-04-22T20:21:21.297100+0000 mgr.y (mgr.34107) 6001 : cluster [DBG] pgmap v7200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:23 smithi121 bash[36902]: audit 2022-04-22T20:21:22.152836+0000 mon.a (mon.0) 9232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:23 smithi121 bash[36902]: audit 2022-04-22T20:21:22.334037+0000 mon.a (mon.0) 9233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[45495]: cluster 2022-04-22T20:21:21.297100+0000 mgr.y (mgr.34107) 6001 : cluster [DBG] pgmap v7200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[45495]: audit 2022-04-22T20:21:22.152836+0000 mon.a (mon.0) 9232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[45495]: audit 2022-04-22T20:21:22.334037+0000 mon.a (mon.0) 9233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[46934]: cluster 2022-04-22T20:21:21.297100+0000 mgr.y (mgr.34107) 6001 : cluster [DBG] pgmap v7200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[46934]: audit 2022-04-22T20:21:22.152836+0000 mon.a (mon.0) 9232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:23 smithi115 bash[46934]: audit 2022-04-22T20:21:22.334037+0000 mon.a (mon.0) 9233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:24.486 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:24.169Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:24.486 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:24 smithi115 bash[45495]: audit 2022-04-22T20:21:23.773086+0000 mon.a (mon.0) 9234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:24.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:24 smithi115 bash[46934]: audit 2022-04-22T20:21:23.773086+0000 mon.a (mon.0) 9234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:24 smithi121 bash[36902]: audit 2022-04-22T20:21:23.773086+0000 mon.a (mon.0) 9234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:25 smithi121 bash[36902]: cluster 2022-04-22T20:21:23.298229+0000 mgr.y (mgr.34107) 6002 : cluster [DBG] pgmap v7201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:25 smithi121 bash[36902]: audit 2022-04-22T20:21:24.110538+0000 mon.a (mon.0) 9235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:21:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:25 smithi121 bash[36902]: audit 2022-04-22T20:21:24.113523+0000 mon.a (mon.0) 9236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:21:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:25 smithi121 bash[36902]: audit 2022-04-22T20:21:24.115546+0000 mon.a (mon.0) 9237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:21:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[45495]: cluster 2022-04-22T20:21:23.298229+0000 mgr.y (mgr.34107) 6002 : cluster [DBG] pgmap v7201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[45495]: audit 2022-04-22T20:21:24.110538+0000 mon.a (mon.0) 9235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:21:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[45495]: audit 2022-04-22T20:21:24.113523+0000 mon.a (mon.0) 9236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:21:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[45495]: audit 2022-04-22T20:21:24.115546+0000 mon.a (mon.0) 9237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:21:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[46934]: cluster 2022-04-22T20:21:23.298229+0000 mgr.y (mgr.34107) 6002 : cluster [DBG] pgmap v7201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[46934]: audit 2022-04-22T20:21:24.110538+0000 mon.a (mon.0) 9235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:21:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[46934]: audit 2022-04-22T20:21:24.113523+0000 mon.a (mon.0) 9236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:21:25.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:25 smithi115 bash[46934]: audit 2022-04-22T20:21:24.115546+0000 mon.a (mon.0) 9237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:21:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:27 smithi121 bash[36902]: cluster 2022-04-22T20:21:25.299323+0000 mgr.y (mgr.34107) 6003 : cluster [DBG] pgmap v7202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:27 smithi115 bash[45495]: cluster 2022-04-22T20:21:25.299323+0000 mgr.y (mgr.34107) 6003 : cluster [DBG] pgmap v7202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:27 smithi115 bash[46934]: cluster 2022-04-22T20:21:25.299323+0000 mgr.y (mgr.34107) 6003 : cluster [DBG] pgmap v7202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:28.308 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:21:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:28 smithi121 bash[36902]: audit 2022-04-22T20:21:27.156374+0000 mon.a (mon.0) 9238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:28 smithi121 bash[36902]: audit 2022-04-22T20:21:27.337028+0000 mon.a (mon.0) 9239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:28 smithi115 bash[46934]: audit 2022-04-22T20:21:27.156374+0000 mon.a (mon.0) 9238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:28 smithi115 bash[46934]: audit 2022-04-22T20:21:27.337028+0000 mon.a (mon.0) 9239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:28 smithi115 bash[45495]: audit 2022-04-22T20:21:27.156374+0000 mon.a (mon.0) 9238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:28 smithi115 bash[45495]: audit 2022-04-22T20:21:27.337028+0000 mon.a (mon.0) 9239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 0s ago 2h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 0s ago 2h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 0s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:21:28.652 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 0s ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 0s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:21:28.653 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 0s ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:21:28.654 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 0s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:21:28.654 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 0s ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:21:28.654 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 0s ago 2h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:21:28.654 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 0s ago 2h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:21:29.057 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:21:29.058 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:21:29.058 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:21:29.058 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:21:29.058 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:21:29.058 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:21:29.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:21:29.059 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:21:29.059 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:21:29.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:21:29.060 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:21:29.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:21:29.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:21:29.060 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:21:29.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:21:29.061 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:21:29.061 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:21:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:29 smithi121 bash[36902]: cluster 2022-04-22T20:21:27.300516+0000 mgr.y (mgr.34107) 6004 : cluster [DBG] pgmap v7203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:29 smithi121 bash[36902]: audit 2022-04-22T20:21:28.573987+0000 mon.a (mon.0) 9240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:29 smithi121 bash[36902]: audit 2022-04-22T20:21:28.780989+0000 mon.a (mon.0) 9241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:29 smithi121 bash[36902]: audit 2022-04-22T20:21:28.911756+0000 mon.a (mon.0) 9242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:29 smithi121 bash[36902]: audit 2022-04-22T20:21:29.048463+0000 mon.a (mon.0) 9243 : audit [DBG] from='client.? 172.21.15.115:0/395020366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[45495]: cluster 2022-04-22T20:21:27.300516+0000 mgr.y (mgr.34107) 6004 : cluster [DBG] pgmap v7203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[45495]: audit 2022-04-22T20:21:28.573987+0000 mon.a (mon.0) 9240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[45495]: audit 2022-04-22T20:21:28.780989+0000 mon.a (mon.0) 9241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[45495]: audit 2022-04-22T20:21:28.911756+0000 mon.a (mon.0) 9242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[45495]: audit 2022-04-22T20:21:29.048463+0000 mon.a (mon.0) 9243 : audit [DBG] from='client.? 172.21.15.115:0/395020366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:21:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[46934]: cluster 2022-04-22T20:21:27.300516+0000 mgr.y (mgr.34107) 6004 : cluster [DBG] pgmap v7203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[46934]: audit 2022-04-22T20:21:28.573987+0000 mon.a (mon.0) 9240 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[46934]: audit 2022-04-22T20:21:28.780989+0000 mon.a (mon.0) 9241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[46934]: audit 2022-04-22T20:21:28.911756+0000 mon.a (mon.0) 9242 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:29 smithi115 bash[46934]: audit 2022-04-22T20:21:29.048463+0000 mon.a (mon.0) 9243 : audit [DBG] from='client.? 172.21.15.115:0/395020366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:21:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:30 smithi121 bash[36902]: audit 2022-04-22T20:21:28.284102+0000 mgr.y (mgr.34107) 6005 : audit [DBG] from='client.41634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:30 smithi121 bash[36902]: audit 2022-04-22T20:21:28.627794+0000 mgr.y (mgr.34107) 6006 : audit [DBG] from='client.51574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:30 smithi115 bash[45495]: audit 2022-04-22T20:21:28.284102+0000 mgr.y (mgr.34107) 6005 : audit [DBG] from='client.41634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:30 smithi115 bash[45495]: audit 2022-04-22T20:21:28.627794+0000 mgr.y (mgr.34107) 6006 : audit [DBG] from='client.51574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:30 smithi115 bash[46934]: audit 2022-04-22T20:21:28.284102+0000 mgr.y (mgr.34107) 6005 : audit [DBG] from='client.41634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:30 smithi115 bash[46934]: audit 2022-04-22T20:21:28.627794+0000 mgr.y (mgr.34107) 6006 : audit [DBG] from='client.51574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:21:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:31 smithi121 bash[36902]: cluster 2022-04-22T20:21:29.301633+0000 mgr.y (mgr.34107) 6007 : cluster [DBG] pgmap v7204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:31 smithi115 bash[46934]: cluster 2022-04-22T20:21:29.301633+0000 mgr.y (mgr.34107) 6007 : cluster [DBG] pgmap v7204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:31.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:31 smithi115 bash[45495]: cluster 2022-04-22T20:21:29.301633+0000 mgr.y (mgr.34107) 6007 : cluster [DBG] pgmap v7204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:32.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:32.293Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:33 smithi121 bash[36902]: cluster 2022-04-22T20:21:31.302709+0000 mgr.y (mgr.34107) 6008 : cluster [DBG] pgmap v7205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:33 smithi121 bash[36902]: audit 2022-04-22T20:21:32.159716+0000 mon.a (mon.0) 9244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:33 smithi121 bash[36902]: audit 2022-04-22T20:21:32.340292+0000 mon.a (mon.0) 9245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[45495]: cluster 2022-04-22T20:21:31.302709+0000 mgr.y (mgr.34107) 6008 : cluster [DBG] pgmap v7205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[45495]: audit 2022-04-22T20:21:32.159716+0000 mon.a (mon.0) 9244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[45495]: audit 2022-04-22T20:21:32.340292+0000 mon.a (mon.0) 9245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[46934]: cluster 2022-04-22T20:21:31.302709+0000 mgr.y (mgr.34107) 6008 : cluster [DBG] pgmap v7205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[46934]: audit 2022-04-22T20:21:32.159716+0000 mon.a (mon.0) 9244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:33.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:33 smithi115 bash[46934]: audit 2022-04-22T20:21:32.340292+0000 mon.a (mon.0) 9245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:34 smithi121 bash[36902]: audit 2022-04-22T20:21:33.777805+0000 mon.a (mon.0) 9246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:34 smithi115 bash[45495]: audit 2022-04-22T20:21:33.777805+0000 mon.a (mon.0) 9246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:34 smithi115 bash[46934]: audit 2022-04-22T20:21:33.777805+0000 mon.a (mon.0) 9246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:34.172Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:35 smithi121 bash[36902]: cluster 2022-04-22T20:21:33.303861+0000 mgr.y (mgr.34107) 6009 : cluster [DBG] pgmap v7206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:35 smithi115 bash[45495]: cluster 2022-04-22T20:21:33.303861+0000 mgr.y (mgr.34107) 6009 : cluster [DBG] pgmap v7206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:35.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:35 smithi115 bash[46934]: cluster 2022-04-22T20:21:33.303861+0000 mgr.y (mgr.34107) 6009 : cluster [DBG] pgmap v7206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:36 smithi121 bash[36902]: cluster 2022-04-22T20:21:35.304937+0000 mgr.y (mgr.34107) 6010 : cluster [DBG] pgmap v7207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:36.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:36 smithi115 bash[45495]: cluster 2022-04-22T20:21:35.304937+0000 mgr.y (mgr.34107) 6010 : cluster [DBG] pgmap v7207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:36 smithi115 bash[46934]: cluster 2022-04-22T20:21:35.304937+0000 mgr.y (mgr.34107) 6010 : cluster [DBG] pgmap v7207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:37 smithi121 bash[36902]: audit 2022-04-22T20:21:37.163380+0000 mon.a (mon.0) 9247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:37.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:37 smithi115 bash[45495]: audit 2022-04-22T20:21:37.163380+0000 mon.a (mon.0) 9247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:37.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:37 smithi115 bash[46934]: audit 2022-04-22T20:21:37.163380+0000 mon.a (mon.0) 9247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:38 smithi121 bash[36902]: cluster 2022-04-22T20:21:37.305994+0000 mgr.y (mgr.34107) 6011 : cluster [DBG] pgmap v7208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:38 smithi121 bash[36902]: audit 2022-04-22T20:21:37.343188+0000 mon.a (mon.0) 9248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:38 smithi115 bash[46934]: cluster 2022-04-22T20:21:37.305994+0000 mgr.y (mgr.34107) 6011 : cluster [DBG] pgmap v7208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:38 smithi115 bash[46934]: audit 2022-04-22T20:21:37.343188+0000 mon.a (mon.0) 9248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:38 smithi115 bash[45495]: cluster 2022-04-22T20:21:37.305994+0000 mgr.y (mgr.34107) 6011 : cluster [DBG] pgmap v7208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:38.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:38 smithi115 bash[45495]: audit 2022-04-22T20:21:37.343188+0000 mon.a (mon.0) 9248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:39 smithi121 bash[36902]: audit 2022-04-22T20:21:38.780942+0000 mon.a (mon.0) 9249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:39 smithi115 bash[45495]: audit 2022-04-22T20:21:38.780942+0000 mon.a (mon.0) 9249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:39 smithi115 bash[46934]: audit 2022-04-22T20:21:38.780942+0000 mon.a (mon.0) 9249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:40 smithi121 bash[36902]: cluster 2022-04-22T20:21:39.307093+0000 mgr.y (mgr.34107) 6012 : cluster [DBG] pgmap v7209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:40.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:40 smithi115 bash[46934]: cluster 2022-04-22T20:21:39.307093+0000 mgr.y (mgr.34107) 6012 : cluster [DBG] pgmap v7209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:40.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:40 smithi115 bash[45495]: cluster 2022-04-22T20:21:39.307093+0000 mgr.y (mgr.34107) 6012 : cluster [DBG] pgmap v7209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:42 smithi121 bash[36902]: cluster 2022-04-22T20:21:41.308159+0000 mgr.y (mgr.34107) 6013 : cluster [DBG] pgmap v7210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:42 smithi121 bash[36902]: audit 2022-04-22T20:21:42.167022+0000 mon.a (mon.0) 9250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:42 smithi121 bash[36902]: audit 2022-04-22T20:21:42.346342+0000 mon.a (mon.0) 9251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:42.294Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[45495]: cluster 2022-04-22T20:21:41.308159+0000 mgr.y (mgr.34107) 6013 : cluster [DBG] pgmap v7210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[45495]: audit 2022-04-22T20:21:42.167022+0000 mon.a (mon.0) 9250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[45495]: audit 2022-04-22T20:21:42.346342+0000 mon.a (mon.0) 9251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[46934]: cluster 2022-04-22T20:21:41.308159+0000 mgr.y (mgr.34107) 6013 : cluster [DBG] pgmap v7210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:42.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[46934]: audit 2022-04-22T20:21:42.167022+0000 mon.a (mon.0) 9250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:42.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:42 smithi115 bash[46934]: audit 2022-04-22T20:21:42.346342+0000 mon.a (mon.0) 9251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:44.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:44.173Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:44.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:44 smithi115 bash[46934]: cluster 2022-04-22T20:21:43.309297+0000 mgr.y (mgr.34107) 6014 : cluster [DBG] pgmap v7211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:44.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:44 smithi115 bash[46934]: audit 2022-04-22T20:21:43.784040+0000 mon.a (mon.0) 9252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:44.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:44 smithi115 bash[45495]: cluster 2022-04-22T20:21:43.309297+0000 mgr.y (mgr.34107) 6014 : cluster [DBG] pgmap v7211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:44.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:44 smithi115 bash[45495]: audit 2022-04-22T20:21:43.784040+0000 mon.a (mon.0) 9252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:44 smithi121 bash[36902]: cluster 2022-04-22T20:21:43.309297+0000 mgr.y (mgr.34107) 6014 : cluster [DBG] pgmap v7211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:44.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:44 smithi121 bash[36902]: audit 2022-04-22T20:21:43.784040+0000 mon.a (mon.0) 9252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:46 smithi121 bash[36902]: cluster 2022-04-22T20:21:45.310379+0000 mgr.y (mgr.34107) 6015 : cluster [DBG] pgmap v7212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:46.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:46 smithi115 bash[45495]: cluster 2022-04-22T20:21:45.310379+0000 mgr.y (mgr.34107) 6015 : cluster [DBG] pgmap v7212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:46.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:46 smithi115 bash[46934]: cluster 2022-04-22T20:21:45.310379+0000 mgr.y (mgr.34107) 6015 : cluster [DBG] pgmap v7212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:47 smithi121 bash[36902]: audit 2022-04-22T20:21:47.170793+0000 mon.a (mon.0) 9253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:47 smithi121 bash[36902]: audit 2022-04-22T20:21:47.349474+0000 mon.a (mon.0) 9254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:47 smithi115 bash[45495]: audit 2022-04-22T20:21:47.170793+0000 mon.a (mon.0) 9253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:47 smithi115 bash[45495]: audit 2022-04-22T20:21:47.349474+0000 mon.a (mon.0) 9254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:47 smithi115 bash[46934]: audit 2022-04-22T20:21:47.170793+0000 mon.a (mon.0) 9253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:47.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:47 smithi115 bash[46934]: audit 2022-04-22T20:21:47.349474+0000 mon.a (mon.0) 9254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:48 smithi121 bash[36902]: cluster 2022-04-22T20:21:47.311459+0000 mgr.y (mgr.34107) 6016 : cluster [DBG] pgmap v7213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:48 smithi115 bash[45495]: cluster 2022-04-22T20:21:47.311459+0000 mgr.y (mgr.34107) 6016 : cluster [DBG] pgmap v7213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:48 smithi115 bash[46934]: cluster 2022-04-22T20:21:47.311459+0000 mgr.y (mgr.34107) 6016 : cluster [DBG] pgmap v7213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:49 smithi121 bash[36902]: audit 2022-04-22T20:21:48.787052+0000 mon.a (mon.0) 9255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:49.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:49 smithi115 bash[45495]: audit 2022-04-22T20:21:48.787052+0000 mon.a (mon.0) 9255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:49.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:49 smithi115 bash[46934]: audit 2022-04-22T20:21:48.787052+0000 mon.a (mon.0) 9255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:50 smithi121 bash[36902]: cluster 2022-04-22T20:21:49.312524+0000 mgr.y (mgr.34107) 6017 : cluster [DBG] pgmap v7214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:50 smithi115 bash[45495]: cluster 2022-04-22T20:21:49.312524+0000 mgr.y (mgr.34107) 6017 : cluster [DBG] pgmap v7214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:50 smithi115 bash[46934]: cluster 2022-04-22T20:21:49.312524+0000 mgr.y (mgr.34107) 6017 : cluster [DBG] pgmap v7214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:52.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:52.294Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:52 smithi121 bash[36902]: cluster 2022-04-22T20:21:51.313595+0000 mgr.y (mgr.34107) 6018 : cluster [DBG] pgmap v7215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:52 smithi121 bash[36902]: audit 2022-04-22T20:21:52.174613+0000 mon.a (mon.0) 9256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:52 smithi121 bash[36902]: audit 2022-04-22T20:21:52.352760+0000 mon.a (mon.0) 9257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[45495]: cluster 2022-04-22T20:21:51.313595+0000 mgr.y (mgr.34107) 6018 : cluster [DBG] pgmap v7215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[45495]: audit 2022-04-22T20:21:52.174613+0000 mon.a (mon.0) 9256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[45495]: audit 2022-04-22T20:21:52.352760+0000 mon.a (mon.0) 9257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[46934]: cluster 2022-04-22T20:21:51.313595+0000 mgr.y (mgr.34107) 6018 : cluster [DBG] pgmap v7215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[46934]: audit 2022-04-22T20:21:52.174613+0000 mon.a (mon.0) 9256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:53.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:52 smithi115 bash[46934]: audit 2022-04-22T20:21:52.352760+0000 mon.a (mon.0) 9257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:53 smithi121 bash[36902]: audit 2022-04-22T20:21:53.790352+0000 mon.a (mon.0) 9258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:53 smithi115 bash[45495]: audit 2022-04-22T20:21:53.790352+0000 mon.a (mon.0) 9258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:53 smithi115 bash[46934]: audit 2022-04-22T20:21:53.790352+0000 mon.a (mon.0) 9258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:54.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:21:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:21:54.172Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:21:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:54 smithi121 bash[36902]: cluster 2022-04-22T20:21:53.314684+0000 mgr.y (mgr.34107) 6019 : cluster [DBG] pgmap v7216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:54 smithi115 bash[45495]: cluster 2022-04-22T20:21:53.314684+0000 mgr.y (mgr.34107) 6019 : cluster [DBG] pgmap v7216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:54 smithi115 bash[46934]: cluster 2022-04-22T20:21:53.314684+0000 mgr.y (mgr.34107) 6019 : cluster [DBG] pgmap v7216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:56 smithi121 bash[36902]: cluster 2022-04-22T20:21:55.315791+0000 mgr.y (mgr.34107) 6020 : cluster [DBG] pgmap v7217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:56 smithi115 bash[45495]: cluster 2022-04-22T20:21:55.315791+0000 mgr.y (mgr.34107) 6020 : cluster [DBG] pgmap v7217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:56 smithi115 bash[46934]: cluster 2022-04-22T20:21:55.315791+0000 mgr.y (mgr.34107) 6020 : cluster [DBG] pgmap v7217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:57 smithi121 bash[36902]: audit 2022-04-22T20:21:57.178108+0000 mon.a (mon.0) 9259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:57 smithi121 bash[36902]: audit 2022-04-22T20:21:57.355748+0000 mon.a (mon.0) 9260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:57 smithi115 bash[45495]: audit 2022-04-22T20:21:57.178108+0000 mon.a (mon.0) 9259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:57 smithi115 bash[45495]: audit 2022-04-22T20:21:57.355748+0000 mon.a (mon.0) 9260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:57 smithi115 bash[46934]: audit 2022-04-22T20:21:57.178108+0000 mon.a (mon.0) 9259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:21:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:57 smithi115 bash[46934]: audit 2022-04-22T20:21:57.355748+0000 mon.a (mon.0) 9260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:21:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:58 smithi121 bash[36902]: cluster 2022-04-22T20:21:57.316798+0000 mgr.y (mgr.34107) 6021 : cluster [DBG] pgmap v7218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:21:58 smithi121 bash[36902]: audit 2022-04-22T20:21:58.793435+0000 mon.a (mon.0) 9261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:58 smithi115 bash[45495]: cluster 2022-04-22T20:21:57.316798+0000 mgr.y (mgr.34107) 6021 : cluster [DBG] pgmap v7218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:21:58 smithi115 bash[45495]: audit 2022-04-22T20:21:58.793435+0000 mon.a (mon.0) 9261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:58 smithi115 bash[46934]: cluster 2022-04-22T20:21:57.316798+0000 mgr.y (mgr.34107) 6021 : cluster [DBG] pgmap v7218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:21:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:21:58 smithi115 bash[46934]: audit 2022-04-22T20:21:58.793435+0000 mon.a (mon.0) 9261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:21:59.428 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:21:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:21:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:21:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 31s ago 2h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 2h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 31s ago 2h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 31s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 9m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 9m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 31s ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 9m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 31s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:21:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 9m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 9m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 9m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 9m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 31s ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 31s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 31s ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 31s ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:21:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 31s ago 2h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:22:00.194 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:22:00.194 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:22:00.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:22:00.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:00.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:22:00.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:22:00.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:00.199 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:22:00.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:22:00.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:22:00.200 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:22:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:00 smithi121 bash[36902]: cluster 2022-04-22T20:21:59.317848+0000 mgr.y (mgr.34107) 6022 : cluster [DBG] pgmap v7219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:00 smithi121 bash[36902]: audit 2022-04-22T20:21:59.404966+0000 mgr.y (mgr.34107) 6023 : audit [DBG] from='client.41652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:00 smithi121 bash[36902]: audit 2022-04-22T20:21:59.745881+0000 mgr.y (mgr.34107) 6024 : audit [DBG] from='client.51592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:00 smithi121 bash[36902]: audit 2022-04-22T20:22:00.185455+0000 mon.c (mon.1) 239 : audit [DBG] from='client.? 172.21.15.115:0/1989088966' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:01.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[45495]: cluster 2022-04-22T20:21:59.317848+0000 mgr.y (mgr.34107) 6022 : cluster [DBG] pgmap v7219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:01.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[45495]: audit 2022-04-22T20:21:59.404966+0000 mgr.y (mgr.34107) 6023 : audit [DBG] from='client.41652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[45495]: audit 2022-04-22T20:21:59.745881+0000 mgr.y (mgr.34107) 6024 : audit [DBG] from='client.51592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[45495]: audit 2022-04-22T20:22:00.185455+0000 mon.c (mon.1) 239 : audit [DBG] from='client.? 172.21.15.115:0/1989088966' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[46934]: cluster 2022-04-22T20:21:59.317848+0000 mgr.y (mgr.34107) 6022 : cluster [DBG] pgmap v7219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[46934]: audit 2022-04-22T20:21:59.404966+0000 mgr.y (mgr.34107) 6023 : audit [DBG] from='client.41652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[46934]: audit 2022-04-22T20:21:59.745881+0000 mgr.y (mgr.34107) 6024 : audit [DBG] from='client.51592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:00 smithi115 bash[46934]: audit 2022-04-22T20:22:00.185455+0000 mon.c (mon.1) 239 : audit [DBG] from='client.? 172.21.15.115:0/1989088966' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:02.296Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:02 smithi121 bash[36902]: cluster 2022-04-22T20:22:01.318987+0000 mgr.y (mgr.34107) 6025 : cluster [DBG] pgmap v7220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:02 smithi121 bash[36902]: audit 2022-04-22T20:22:02.181784+0000 mon.a (mon.0) 9262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:02 smithi121 bash[36902]: audit 2022-04-22T20:22:02.358640+0000 mon.a (mon.0) 9263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[45495]: cluster 2022-04-22T20:22:01.318987+0000 mgr.y (mgr.34107) 6025 : cluster [DBG] pgmap v7220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[45495]: audit 2022-04-22T20:22:02.181784+0000 mon.a (mon.0) 9262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[45495]: audit 2022-04-22T20:22:02.358640+0000 mon.a (mon.0) 9263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[46934]: cluster 2022-04-22T20:22:01.318987+0000 mgr.y (mgr.34107) 6025 : cluster [DBG] pgmap v7220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[46934]: audit 2022-04-22T20:22:02.181784+0000 mon.a (mon.0) 9262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:02 smithi115 bash[46934]: audit 2022-04-22T20:22:02.358640+0000 mon.a (mon.0) 9263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:03 smithi121 bash[36902]: audit 2022-04-22T20:22:03.796610+0000 mon.a (mon.0) 9264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:04.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:04.174Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:04.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:03 smithi115 bash[45495]: audit 2022-04-22T20:22:03.796610+0000 mon.a (mon.0) 9264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:04.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:03 smithi115 bash[46934]: audit 2022-04-22T20:22:03.796610+0000 mon.a (mon.0) 9264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:04 smithi121 bash[36902]: cluster 2022-04-22T20:22:03.320065+0000 mgr.y (mgr.34107) 6026 : cluster [DBG] pgmap v7221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:05.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:04 smithi115 bash[45495]: cluster 2022-04-22T20:22:03.320065+0000 mgr.y (mgr.34107) 6026 : cluster [DBG] pgmap v7221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:05.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:04 smithi115 bash[46934]: cluster 2022-04-22T20:22:03.320065+0000 mgr.y (mgr.34107) 6026 : cluster [DBG] pgmap v7221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:06 smithi121 bash[36902]: cluster 2022-04-22T20:22:05.321067+0000 mgr.y (mgr.34107) 6027 : cluster [DBG] pgmap v7222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:07.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:06 smithi115 bash[46934]: cluster 2022-04-22T20:22:05.321067+0000 mgr.y (mgr.34107) 6027 : cluster [DBG] pgmap v7222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:07.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:06 smithi115 bash[45495]: cluster 2022-04-22T20:22:05.321067+0000 mgr.y (mgr.34107) 6027 : cluster [DBG] pgmap v7222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:08 smithi121 bash[36902]: audit 2022-04-22T20:22:07.185475+0000 mon.a (mon.0) 9265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:08 smithi121 bash[36902]: audit 2022-04-22T20:22:07.361131+0000 mon.a (mon.0) 9266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:08 smithi115 bash[46934]: audit 2022-04-22T20:22:07.185475+0000 mon.a (mon.0) 9265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:08.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:08 smithi115 bash[46934]: audit 2022-04-22T20:22:07.361131+0000 mon.a (mon.0) 9266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:08 smithi115 bash[45495]: audit 2022-04-22T20:22:07.185475+0000 mon.a (mon.0) 9265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:08.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:08 smithi115 bash[45495]: audit 2022-04-22T20:22:07.361131+0000 mon.a (mon.0) 9266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:09 smithi121 bash[36902]: cluster 2022-04-22T20:22:07.321668+0000 mgr.y (mgr.34107) 6028 : cluster [DBG] pgmap v7223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:09 smithi121 bash[36902]: audit 2022-04-22T20:22:08.799466+0000 mon.a (mon.0) 9267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:09 smithi115 bash[46934]: cluster 2022-04-22T20:22:07.321668+0000 mgr.y (mgr.34107) 6028 : cluster [DBG] pgmap v7223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:09.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:09 smithi115 bash[46934]: audit 2022-04-22T20:22:08.799466+0000 mon.a (mon.0) 9267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:09 smithi115 bash[45495]: cluster 2022-04-22T20:22:07.321668+0000 mgr.y (mgr.34107) 6028 : cluster [DBG] pgmap v7223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:09.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:09 smithi115 bash[45495]: audit 2022-04-22T20:22:08.799466+0000 mon.a (mon.0) 9267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:11.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:11 smithi115 bash[46934]: cluster 2022-04-22T20:22:09.322641+0000 mgr.y (mgr.34107) 6029 : cluster [DBG] pgmap v7224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:11.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:11 smithi115 bash[45495]: cluster 2022-04-22T20:22:09.322641+0000 mgr.y (mgr.34107) 6029 : cluster [DBG] pgmap v7224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:11 smithi121 bash[36902]: cluster 2022-04-22T20:22:09.322641+0000 mgr.y (mgr.34107) 6029 : cluster [DBG] pgmap v7224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:12.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:12.297Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[46934]: cluster 2022-04-22T20:22:11.323685+0000 mgr.y (mgr.34107) 6030 : cluster [DBG] pgmap v7225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[46934]: audit 2022-04-22T20:22:12.189142+0000 mon.a (mon.0) 9268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:13.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[46934]: audit 2022-04-22T20:22:12.363912+0000 mon.a (mon.0) 9269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[45495]: cluster 2022-04-22T20:22:11.323685+0000 mgr.y (mgr.34107) 6030 : cluster [DBG] pgmap v7225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[45495]: audit 2022-04-22T20:22:12.189142+0000 mon.a (mon.0) 9268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:13.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:13 smithi115 bash[45495]: audit 2022-04-22T20:22:12.363912+0000 mon.a (mon.0) 9269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:13 smithi121 bash[36902]: cluster 2022-04-22T20:22:11.323685+0000 mgr.y (mgr.34107) 6030 : cluster [DBG] pgmap v7225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:13 smithi121 bash[36902]: audit 2022-04-22T20:22:12.189142+0000 mon.a (mon.0) 9268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:13 smithi121 bash[36902]: audit 2022-04-22T20:22:12.363912+0000 mon.a (mon.0) 9269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:14 smithi115 bash[46934]: audit 2022-04-22T20:22:13.802551+0000 mon.a (mon.0) 9270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:14.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:14.174Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:14.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:14 smithi115 bash[45495]: audit 2022-04-22T20:22:13.802551+0000 mon.a (mon.0) 9270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:14 smithi121 bash[36902]: audit 2022-04-22T20:22:13.802551+0000 mon.a (mon.0) 9270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:15.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:15 smithi115 bash[46934]: cluster 2022-04-22T20:22:13.324811+0000 mgr.y (mgr.34107) 6031 : cluster [DBG] pgmap v7226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:15.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:15 smithi115 bash[45495]: cluster 2022-04-22T20:22:13.324811+0000 mgr.y (mgr.34107) 6031 : cluster [DBG] pgmap v7226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:15 smithi121 bash[36902]: cluster 2022-04-22T20:22:13.324811+0000 mgr.y (mgr.34107) 6031 : cluster [DBG] pgmap v7226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:16 smithi115 bash[45495]: audit 2022-04-22T20:22:15.300578+0000 mon.a (mon.0) 9271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:22:16.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:16 smithi115 bash[45495]: audit 2022-04-22T20:22:15.300953+0000 mon.a (mon.0) 9272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:22:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:16 smithi115 bash[46934]: audit 2022-04-22T20:22:15.300578+0000 mon.a (mon.0) 9271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:22:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:16 smithi115 bash[46934]: audit 2022-04-22T20:22:15.300953+0000 mon.a (mon.0) 9272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:22:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:16 smithi121 bash[36902]: audit 2022-04-22T20:22:15.300578+0000 mon.a (mon.0) 9271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:22:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:16 smithi121 bash[36902]: audit 2022-04-22T20:22:15.300953+0000 mon.a (mon.0) 9272 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:22:17.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:17 smithi115 bash[45495]: cluster 2022-04-22T20:22:15.325810+0000 mgr.y (mgr.34107) 6032 : cluster [DBG] pgmap v7227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:17.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:17 smithi115 bash[46934]: cluster 2022-04-22T20:22:15.325810+0000 mgr.y (mgr.34107) 6032 : cluster [DBG] pgmap v7227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:17 smithi121 bash[36902]: cluster 2022-04-22T20:22:15.325810+0000 mgr.y (mgr.34107) 6032 : cluster [DBG] pgmap v7227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:18 smithi121 bash[36902]: audit 2022-04-22T20:22:17.192836+0000 mon.a (mon.0) 9273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:18 smithi121 bash[36902]: audit 2022-04-22T20:22:17.366526+0000 mon.a (mon.0) 9274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:18 smithi115 bash[45495]: audit 2022-04-22T20:22:17.192836+0000 mon.a (mon.0) 9273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:18.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:18 smithi115 bash[45495]: audit 2022-04-22T20:22:17.366526+0000 mon.a (mon.0) 9274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:18 smithi115 bash[46934]: audit 2022-04-22T20:22:17.192836+0000 mon.a (mon.0) 9273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:18 smithi115 bash[46934]: audit 2022-04-22T20:22:17.366526+0000 mon.a (mon.0) 9274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:19 smithi121 bash[36902]: cluster 2022-04-22T20:22:17.326737+0000 mgr.y (mgr.34107) 6033 : cluster [DBG] pgmap v7228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:19 smithi121 bash[36902]: audit 2022-04-22T20:22:18.805664+0000 mon.a (mon.0) 9275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:19 smithi115 bash[45495]: cluster 2022-04-22T20:22:17.326737+0000 mgr.y (mgr.34107) 6033 : cluster [DBG] pgmap v7228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:19 smithi115 bash[45495]: audit 2022-04-22T20:22:18.805664+0000 mon.a (mon.0) 9275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:19 smithi115 bash[46934]: cluster 2022-04-22T20:22:17.326737+0000 mgr.y (mgr.34107) 6033 : cluster [DBG] pgmap v7228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:19 smithi115 bash[46934]: audit 2022-04-22T20:22:18.805664+0000 mon.a (mon.0) 9275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:21 smithi121 bash[36902]: cluster 2022-04-22T20:22:19.327850+0000 mgr.y (mgr.34107) 6034 : cluster [DBG] pgmap v7229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:21 smithi115 bash[45495]: cluster 2022-04-22T20:22:19.327850+0000 mgr.y (mgr.34107) 6034 : cluster [DBG] pgmap v7229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:21 smithi115 bash[46934]: cluster 2022-04-22T20:22:19.327850+0000 mgr.y (mgr.34107) 6034 : cluster [DBG] pgmap v7229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:22.298Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:23 smithi121 bash[36902]: cluster 2022-04-22T20:22:21.328941+0000 mgr.y (mgr.34107) 6035 : cluster [DBG] pgmap v7230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:23 smithi121 bash[36902]: audit 2022-04-22T20:22:22.196270+0000 mon.a (mon.0) 9276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:23 smithi121 bash[36902]: audit 2022-04-22T20:22:22.369367+0000 mon.a (mon.0) 9277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[45495]: cluster 2022-04-22T20:22:21.328941+0000 mgr.y (mgr.34107) 6035 : cluster [DBG] pgmap v7230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[45495]: audit 2022-04-22T20:22:22.196270+0000 mon.a (mon.0) 9276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[45495]: audit 2022-04-22T20:22:22.369367+0000 mon.a (mon.0) 9277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[46934]: cluster 2022-04-22T20:22:21.328941+0000 mgr.y (mgr.34107) 6035 : cluster [DBG] pgmap v7230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[46934]: audit 2022-04-22T20:22:22.196270+0000 mon.a (mon.0) 9276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:23 smithi115 bash[46934]: audit 2022-04-22T20:22:22.369367+0000 mon.a (mon.0) 9277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:24 smithi121 bash[36902]: audit 2022-04-22T20:22:23.808566+0000 mon.a (mon.0) 9278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:24 smithi115 bash[45495]: audit 2022-04-22T20:22:23.808566+0000 mon.a (mon.0) 9278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:24.175Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:24 smithi115 bash[46934]: audit 2022-04-22T20:22:23.808566+0000 mon.a (mon.0) 9278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:25 smithi121 bash[36902]: cluster 2022-04-22T20:22:23.330111+0000 mgr.y (mgr.34107) 6036 : cluster [DBG] pgmap v7231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:25 smithi115 bash[45495]: cluster 2022-04-22T20:22:23.330111+0000 mgr.y (mgr.34107) 6036 : cluster [DBG] pgmap v7231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:25.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:25 smithi115 bash[46934]: cluster 2022-04-22T20:22:23.330111+0000 mgr.y (mgr.34107) 6036 : cluster [DBG] pgmap v7231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:27 smithi121 bash[36902]: cluster 2022-04-22T20:22:25.331188+0000 mgr.y (mgr.34107) 6037 : cluster [DBG] pgmap v7232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:27.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:27 smithi115 bash[45495]: cluster 2022-04-22T20:22:25.331188+0000 mgr.y (mgr.34107) 6037 : cluster [DBG] pgmap v7232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:27 smithi115 bash[46934]: cluster 2022-04-22T20:22:25.331188+0000 mgr.y (mgr.34107) 6037 : cluster [DBG] pgmap v7232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:28 smithi121 bash[36902]: audit 2022-04-22T20:22:27.199937+0000 mon.a (mon.0) 9279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:28 smithi121 bash[36902]: audit 2022-04-22T20:22:27.372196+0000 mon.a (mon.0) 9280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:28 smithi115 bash[45495]: audit 2022-04-22T20:22:27.199937+0000 mon.a (mon.0) 9279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:28 smithi115 bash[45495]: audit 2022-04-22T20:22:27.372196+0000 mon.a (mon.0) 9280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:28 smithi115 bash[46934]: audit 2022-04-22T20:22:27.199937+0000 mon.a (mon.0) 9279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:28 smithi115 bash[46934]: audit 2022-04-22T20:22:27.372196+0000 mon.a (mon.0) 9280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:29.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[45495]: cluster 2022-04-22T20:22:27.332279+0000 mgr.y (mgr.34107) 6038 : cluster [DBG] pgmap v7233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:29.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[45495]: audit 2022-04-22T20:22:28.811620+0000 mon.a (mon.0) 9281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:29.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[45495]: audit 2022-04-22T20:22:28.920667+0000 mon.a (mon.0) 9282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:22:29.250 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[45495]: audit 2022-04-22T20:22:28.923856+0000 mon.a (mon.0) 9283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:22:29.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[45495]: audit 2022-04-22T20:22:28.925920+0000 mon.a (mon.0) 9284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:22:29.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[46934]: cluster 2022-04-22T20:22:27.332279+0000 mgr.y (mgr.34107) 6038 : cluster [DBG] pgmap v7233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:29.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[46934]: audit 2022-04-22T20:22:28.811620+0000 mon.a (mon.0) 9281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:29.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[46934]: audit 2022-04-22T20:22:28.920667+0000 mon.a (mon.0) 9282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:22:29.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[46934]: audit 2022-04-22T20:22:28.923856+0000 mon.a (mon.0) 9283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:22:29.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:29 smithi115 bash[46934]: audit 2022-04-22T20:22:28.925920+0000 mon.a (mon.0) 9284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:22:29.255 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:29 smithi121 bash[36902]: cluster 2022-04-22T20:22:27.332279+0000 mgr.y (mgr.34107) 6038 : cluster [DBG] pgmap v7233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:29.255 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:29 smithi121 bash[36902]: audit 2022-04-22T20:22:28.811620+0000 mon.a (mon.0) 9281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:29.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:29 smithi121 bash[36902]: audit 2022-04-22T20:22:28.920667+0000 mon.a (mon.0) 9282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:22:29.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:29 smithi121 bash[36902]: audit 2022-04-22T20:22:28.923856+0000 mon.a (mon.0) 9283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:22:29.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:29 smithi121 bash[36902]: audit 2022-04-22T20:22:28.925920+0000 mon.a (mon.0) 9284 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:22:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:30 smithi121 bash[36902]: audit 2022-04-22T20:22:29.255654+0000 mon.a (mon.0) 9285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:30 smithi121 bash[36902]: audit 2022-04-22T20:22:29.261241+0000 mon.a (mon.0) 9286 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:30 smithi121 bash[36902]: cluster 2022-04-22T20:22:29.333100+0000 mgr.y (mgr.34107) 6039 : cluster [DBG] pgmap v7234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[45495]: audit 2022-04-22T20:22:29.255654+0000 mon.a (mon.0) 9285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[45495]: audit 2022-04-22T20:22:29.261241+0000 mon.a (mon.0) 9286 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[45495]: cluster 2022-04-22T20:22:29.333100+0000 mgr.y (mgr.34107) 6039 : cluster [DBG] pgmap v7234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[46934]: audit 2022-04-22T20:22:29.255654+0000 mon.a (mon.0) 9285 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[46934]: audit 2022-04-22T20:22:29.261241+0000 mon.a (mon.0) 9286 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:30 smithi115 bash[46934]: cluster 2022-04-22T20:22:29.333100+0000 mgr.y (mgr.34107) 6039 : cluster [DBG] pgmap v7234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:30.576 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 10m ago 2h 15.4M - ba2b418f427c 9255ba08f72f 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 62s ago 2h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 10m ago 3h 23.5M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 62s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:22:30.928 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 62s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 10m ago 3h 484M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 10m ago 3h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 62s ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 10m ago 3h 207M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 10m ago 2h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 62s ago 2h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 10m ago 3h 704M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 10m ago 3h 784M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 10m ago 3h 530M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:22:30.929 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 10m ago 3h 488M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:22:30.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 62s ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:22:30.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 62s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:22:30.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 62s ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:22:30.930 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 62s ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:22:30.930 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 62s ago 2h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:22:31.335 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:22:31.335 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:22:31.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:22:31.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:31.335 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:22:31.336 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:22:31.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:31.336 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:22:31.336 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:22:31.337 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:22:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:22:31.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:31 smithi115 bash[45495]: audit 2022-04-22T20:22:30.553151+0000 mgr.y (mgr.34107) 6040 : audit [DBG] from='client.41670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:31.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:31 smithi115 bash[45495]: audit 2022-04-22T20:22:30.898244+0000 mgr.y (mgr.34107) 6041 : audit [DBG] from='client.51610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:31 smithi115 bash[46934]: audit 2022-04-22T20:22:30.553151+0000 mgr.y (mgr.34107) 6040 : audit [DBG] from='client.41670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:31 smithi115 bash[46934]: audit 2022-04-22T20:22:30.898244+0000 mgr.y (mgr.34107) 6041 : audit [DBG] from='client.51610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:31 smithi121 bash[36902]: audit 2022-04-22T20:22:30.553151+0000 mgr.y (mgr.34107) 6040 : audit [DBG] from='client.41670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:31 smithi121 bash[36902]: audit 2022-04-22T20:22:30.898244+0000 mgr.y (mgr.34107) 6041 : audit [DBG] from='client.51610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:22:32.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:32.296Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[45495]: audit 2022-04-22T20:22:31.326614+0000 mon.a (mon.0) 9287 : audit [DBG] from='client.? 172.21.15.115:0/1037542724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[45495]: cluster 2022-04-22T20:22:31.333654+0000 mgr.y (mgr.34107) 6042 : cluster [DBG] pgmap v7235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:32.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[45495]: audit 2022-04-22T20:22:32.203559+0000 mon.a (mon.0) 9288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:32.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[46934]: audit 2022-04-22T20:22:31.326614+0000 mon.a (mon.0) 9287 : audit [DBG] from='client.? 172.21.15.115:0/1037542724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:32.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[46934]: cluster 2022-04-22T20:22:31.333654+0000 mgr.y (mgr.34107) 6042 : cluster [DBG] pgmap v7235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:32.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:32 smithi115 bash[46934]: audit 2022-04-22T20:22:32.203559+0000 mon.a (mon.0) 9288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:32 smithi121 bash[36902]: audit 2022-04-22T20:22:31.326614+0000 mon.a (mon.0) 9287 : audit [DBG] from='client.? 172.21.15.115:0/1037542724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:22:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:32 smithi121 bash[36902]: cluster 2022-04-22T20:22:31.333654+0000 mgr.y (mgr.34107) 6042 : cluster [DBG] pgmap v7235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:32 smithi121 bash[36902]: audit 2022-04-22T20:22:32.203559+0000 mon.a (mon.0) 9288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:33 smithi115 bash[45495]: audit 2022-04-22T20:22:32.375004+0000 mon.a (mon.0) 9289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:33 smithi115 bash[46934]: audit 2022-04-22T20:22:32.375004+0000 mon.a (mon.0) 9289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:33 smithi121 bash[36902]: audit 2022-04-22T20:22:32.375004+0000 mon.a (mon.0) 9289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:34.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:34.175Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[45495]: cluster 2022-04-22T20:22:33.334452+0000 mgr.y (mgr.34107) 6043 : cluster [DBG] pgmap v7236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[45495]: audit 2022-04-22T20:22:33.381644+0000 mon.a (mon.0) 9290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[45495]: audit 2022-04-22T20:22:33.712763+0000 mon.a (mon.0) 9291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[45495]: audit 2022-04-22T20:22:33.814486+0000 mon.a (mon.0) 9292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[46934]: cluster 2022-04-22T20:22:33.334452+0000 mgr.y (mgr.34107) 6043 : cluster [DBG] pgmap v7236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[46934]: audit 2022-04-22T20:22:33.381644+0000 mon.a (mon.0) 9290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[46934]: audit 2022-04-22T20:22:33.712763+0000 mon.a (mon.0) 9291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:34 smithi115 bash[46934]: audit 2022-04-22T20:22:33.814486+0000 mon.a (mon.0) 9292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:34 smithi121 bash[36902]: cluster 2022-04-22T20:22:33.334452+0000 mgr.y (mgr.34107) 6043 : cluster [DBG] pgmap v7236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:34 smithi121 bash[36902]: audit 2022-04-22T20:22:33.381644+0000 mon.a (mon.0) 9290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:34 smithi121 bash[36902]: audit 2022-04-22T20:22:33.712763+0000 mon.a (mon.0) 9291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:22:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:34 smithi121 bash[36902]: audit 2022-04-22T20:22:33.814486+0000 mon.a (mon.0) 9292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:36 smithi121 bash[36902]: cluster 2022-04-22T20:22:35.335459+0000 mgr.y (mgr.34107) 6044 : cluster [DBG] pgmap v7237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:36 smithi115 bash[45495]: cluster 2022-04-22T20:22:35.335459+0000 mgr.y (mgr.34107) 6044 : cluster [DBG] pgmap v7237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:36 smithi115 bash[46934]: cluster 2022-04-22T20:22:35.335459+0000 mgr.y (mgr.34107) 6044 : cluster [DBG] pgmap v7237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:37 smithi121 bash[36902]: audit 2022-04-22T20:22:37.207043+0000 mon.a (mon.0) 9293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:37 smithi121 bash[36902]: audit 2022-04-22T20:22:37.377410+0000 mon.a (mon.0) 9294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:37 smithi115 bash[45495]: audit 2022-04-22T20:22:37.207043+0000 mon.a (mon.0) 9293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:37 smithi115 bash[45495]: audit 2022-04-22T20:22:37.377410+0000 mon.a (mon.0) 9294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:37 smithi115 bash[46934]: audit 2022-04-22T20:22:37.207043+0000 mon.a (mon.0) 9293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:37 smithi115 bash[46934]: audit 2022-04-22T20:22:37.377410+0000 mon.a (mon.0) 9294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:38 smithi121 bash[36902]: cluster 2022-04-22T20:22:37.336555+0000 mgr.y (mgr.34107) 6045 : cluster [DBG] pgmap v7238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:38.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:38 smithi115 bash[45495]: cluster 2022-04-22T20:22:37.336555+0000 mgr.y (mgr.34107) 6045 : cluster [DBG] pgmap v7238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:38 smithi115 bash[46934]: cluster 2022-04-22T20:22:37.336555+0000 mgr.y (mgr.34107) 6045 : cluster [DBG] pgmap v7238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:39 smithi121 bash[36902]: audit 2022-04-22T20:22:38.817695+0000 mon.a (mon.0) 9295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:39 smithi115 bash[45495]: audit 2022-04-22T20:22:38.817695+0000 mon.a (mon.0) 9295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:39.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:39 smithi115 bash[46934]: audit 2022-04-22T20:22:38.817695+0000 mon.a (mon.0) 9295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:40 smithi121 bash[36902]: cluster 2022-04-22T20:22:39.337726+0000 mgr.y (mgr.34107) 6046 : cluster [DBG] pgmap v7239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:40 smithi115 bash[45495]: cluster 2022-04-22T20:22:39.337726+0000 mgr.y (mgr.34107) 6046 : cluster [DBG] pgmap v7239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:40 smithi115 bash[46934]: cluster 2022-04-22T20:22:39.337726+0000 mgr.y (mgr.34107) 6046 : cluster [DBG] pgmap v7239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:42.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:42 smithi121 bash[36902]: cluster 2022-04-22T20:22:41.338799+0000 mgr.y (mgr.34107) 6047 : cluster [DBG] pgmap v7240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:43.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:42 smithi121 bash[36902]: audit 2022-04-22T20:22:42.210943+0000 mon.a (mon.0) 9296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:43.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:42 smithi121 bash[36902]: audit 2022-04-22T20:22:42.380029+0000 mon.a (mon.0) 9297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[45495]: cluster 2022-04-22T20:22:41.338799+0000 mgr.y (mgr.34107) 6047 : cluster [DBG] pgmap v7240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[45495]: audit 2022-04-22T20:22:42.210943+0000 mon.a (mon.0) 9296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[45495]: audit 2022-04-22T20:22:42.380029+0000 mon.a (mon.0) 9297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[46934]: cluster 2022-04-22T20:22:41.338799+0000 mgr.y (mgr.34107) 6047 : cluster [DBG] pgmap v7240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[46934]: audit 2022-04-22T20:22:42.210943+0000 mon.a (mon.0) 9296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:42 smithi115 bash[46934]: audit 2022-04-22T20:22:42.380029+0000 mon.a (mon.0) 9297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:43 smithi121 bash[36902]: audit 2022-04-22T20:22:43.820729+0000 mon.a (mon.0) 9298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:44.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:43 smithi115 bash[45495]: audit 2022-04-22T20:22:43.820729+0000 mon.a (mon.0) 9298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:43 smithi115 bash[46934]: audit 2022-04-22T20:22:43.820729+0000 mon.a (mon.0) 9298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:44.178Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:45.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:44 smithi121 bash[36902]: cluster 2022-04-22T20:22:43.339953+0000 mgr.y (mgr.34107) 6048 : cluster [DBG] pgmap v7241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:45.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:44 smithi115 bash[45495]: cluster 2022-04-22T20:22:43.339953+0000 mgr.y (mgr.34107) 6048 : cluster [DBG] pgmap v7241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:44 smithi115 bash[46934]: cluster 2022-04-22T20:22:43.339953+0000 mgr.y (mgr.34107) 6048 : cluster [DBG] pgmap v7241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:46 smithi121 bash[36902]: cluster 2022-04-22T20:22:45.341017+0000 mgr.y (mgr.34107) 6049 : cluster [DBG] pgmap v7242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:46 smithi115 bash[45495]: cluster 2022-04-22T20:22:45.341017+0000 mgr.y (mgr.34107) 6049 : cluster [DBG] pgmap v7242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:46 smithi115 bash[46934]: cluster 2022-04-22T20:22:45.341017+0000 mgr.y (mgr.34107) 6049 : cluster [DBG] pgmap v7242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:48 smithi121 bash[36902]: audit 2022-04-22T20:22:47.214593+0000 mon.a (mon.0) 9299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:48 smithi121 bash[36902]: audit 2022-04-22T20:22:47.382610+0000 mon.a (mon.0) 9300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:47 smithi115 bash[45495]: audit 2022-04-22T20:22:47.214593+0000 mon.a (mon.0) 9299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:47 smithi115 bash[45495]: audit 2022-04-22T20:22:47.382610+0000 mon.a (mon.0) 9300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:47 smithi115 bash[46934]: audit 2022-04-22T20:22:47.214593+0000 mon.a (mon.0) 9299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:47 smithi115 bash[46934]: audit 2022-04-22T20:22:47.382610+0000 mon.a (mon.0) 9300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:49 smithi121 bash[36902]: cluster 2022-04-22T20:22:47.342069+0000 mgr.y (mgr.34107) 6050 : cluster [DBG] pgmap v7243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:49.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:49 smithi121 bash[36902]: audit 2022-04-22T20:22:48.823728+0000 mon.a (mon.0) 9301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:48 smithi115 bash[45495]: cluster 2022-04-22T20:22:47.342069+0000 mgr.y (mgr.34107) 6050 : cluster [DBG] pgmap v7243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:48 smithi115 bash[45495]: audit 2022-04-22T20:22:48.823728+0000 mon.a (mon.0) 9301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:48 smithi115 bash[46934]: cluster 2022-04-22T20:22:47.342069+0000 mgr.y (mgr.34107) 6050 : cluster [DBG] pgmap v7243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:48 smithi115 bash[46934]: audit 2022-04-22T20:22:48.823728+0000 mon.a (mon.0) 9301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:51 smithi121 bash[36902]: cluster 2022-04-22T20:22:49.343142+0000 mgr.y (mgr.34107) 6051 : cluster [DBG] pgmap v7244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:51.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:51 smithi115 bash[45495]: cluster 2022-04-22T20:22:49.343142+0000 mgr.y (mgr.34107) 6051 : cluster [DBG] pgmap v7244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:51 smithi115 bash[46934]: cluster 2022-04-22T20:22:49.343142+0000 mgr.y (mgr.34107) 6051 : cluster [DBG] pgmap v7244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:52.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:53 smithi121 bash[36902]: cluster 2022-04-22T20:22:51.344086+0000 mgr.y (mgr.34107) 6052 : cluster [DBG] pgmap v7245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:53 smithi121 bash[36902]: audit 2022-04-22T20:22:52.218270+0000 mon.a (mon.0) 9302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:53 smithi121 bash[36902]: audit 2022-04-22T20:22:52.385534+0000 mon.a (mon.0) 9303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[45495]: cluster 2022-04-22T20:22:51.344086+0000 mgr.y (mgr.34107) 6052 : cluster [DBG] pgmap v7245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[45495]: audit 2022-04-22T20:22:52.218270+0000 mon.a (mon.0) 9302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[45495]: audit 2022-04-22T20:22:52.385534+0000 mon.a (mon.0) 9303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[46934]: cluster 2022-04-22T20:22:51.344086+0000 mgr.y (mgr.34107) 6052 : cluster [DBG] pgmap v7245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[46934]: audit 2022-04-22T20:22:52.218270+0000 mon.a (mon.0) 9302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:53 smithi115 bash[46934]: audit 2022-04-22T20:22:52.385534+0000 mon.a (mon.0) 9303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:54 smithi115 bash[45495]: audit 2022-04-22T20:22:53.826733+0000 mon.a (mon.0) 9304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:22:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:22:54.177Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:22:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:54 smithi115 bash[46934]: audit 2022-04-22T20:22:53.826733+0000 mon.a (mon.0) 9304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:54 smithi121 bash[36902]: audit 2022-04-22T20:22:53.826733+0000 mon.a (mon.0) 9304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:55.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:55 smithi115 bash[45495]: cluster 2022-04-22T20:22:53.345226+0000 mgr.y (mgr.34107) 6053 : cluster [DBG] pgmap v7246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:55 smithi115 bash[46934]: cluster 2022-04-22T20:22:53.345226+0000 mgr.y (mgr.34107) 6053 : cluster [DBG] pgmap v7246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:55 smithi121 bash[36902]: cluster 2022-04-22T20:22:53.345226+0000 mgr.y (mgr.34107) 6053 : cluster [DBG] pgmap v7246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:57.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:57 smithi115 bash[45495]: cluster 2022-04-22T20:22:55.346296+0000 mgr.y (mgr.34107) 6054 : cluster [DBG] pgmap v7247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:57 smithi115 bash[46934]: cluster 2022-04-22T20:22:55.346296+0000 mgr.y (mgr.34107) 6054 : cluster [DBG] pgmap v7247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:57 smithi121 bash[36902]: cluster 2022-04-22T20:22:55.346296+0000 mgr.y (mgr.34107) 6054 : cluster [DBG] pgmap v7247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:58 smithi115 bash[45495]: audit 2022-04-22T20:22:57.222157+0000 mon.a (mon.0) 9305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:58.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:58 smithi115 bash[45495]: audit 2022-04-22T20:22:57.388199+0000 mon.a (mon.0) 9306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:58 smithi115 bash[46934]: audit 2022-04-22T20:22:57.222157+0000 mon.a (mon.0) 9305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:58 smithi115 bash[46934]: audit 2022-04-22T20:22:57.388199+0000 mon.a (mon.0) 9306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:58 smithi121 bash[36902]: audit 2022-04-22T20:22:57.222157+0000 mon.a (mon.0) 9305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:58 smithi121 bash[36902]: audit 2022-04-22T20:22:57.388199+0000 mon.a (mon.0) 9306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:59 smithi115 bash[45495]: cluster 2022-04-22T20:22:57.347349+0000 mgr.y (mgr.34107) 6055 : cluster [DBG] pgmap v7248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:22:59 smithi115 bash[45495]: audit 2022-04-22T20:22:58.829811+0000 mon.a (mon.0) 9307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:59 smithi115 bash[46934]: cluster 2022-04-22T20:22:57.347349+0000 mgr.y (mgr.34107) 6055 : cluster [DBG] pgmap v7248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:59.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:22:59 smithi115 bash[46934]: audit 2022-04-22T20:22:58.829811+0000 mon.a (mon.0) 9307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:22:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:59 smithi121 bash[36902]: cluster 2022-04-22T20:22:57.347349+0000 mgr.y (mgr.34107) 6055 : cluster [DBG] pgmap v7248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:22:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:22:59 smithi121 bash[36902]: audit 2022-04-22T20:22:58.829811+0000 mon.a (mon.0) 9307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:01.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:01 smithi115 bash[45495]: cluster 2022-04-22T20:22:59.348446+0000 mgr.y (mgr.34107) 6056 : cluster [DBG] pgmap v7249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:01.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:01 smithi115 bash[46934]: cluster 2022-04-22T20:22:59.348446+0000 mgr.y (mgr.34107) 6056 : cluster [DBG] pgmap v7249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:01 smithi121 bash[36902]: cluster 2022-04-22T20:22:59.348446+0000 mgr.y (mgr.34107) 6056 : cluster [DBG] pgmap v7249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:01.709 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:23:02.053 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 28s ago 2h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (2h) 93s ago 2h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 28s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 93s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 93s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 28s ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 28s ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:23:02.054 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 93s ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 28s ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 28s ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 93s ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 28s ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 28s ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 28s ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 28s ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:23:02.055 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 93s ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:23:02.056 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 93s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:23:02.056 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 93s ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:23:02.056 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 93s ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:23:02.056 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 93s ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:23:02.458 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:23:02.458 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:23:02.458 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:23:02.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:02.460 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:23:02.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:23:02.460 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:23:02.460 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:23:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:02.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: cluster 2022-04-22T20:23:01.349519+0000 mgr.y (mgr.34107) 6057 : cluster [DBG] pgmap v7250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: audit 2022-04-22T20:23:01.686346+0000 mgr.y (mgr.34107) 6058 : audit [DBG] from='client.51622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: audit 2022-04-22T20:23:02.030269+0000 mgr.y (mgr.34107) 6059 : audit [DBG] from='client.51628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: audit 2022-04-22T20:23:02.225812+0000 mon.a (mon.0) 9308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: audit 2022-04-22T20:23:02.390885+0000 mon.a (mon.0) 9309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:03.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[45495]: audit 2022-04-22T20:23:02.450043+0000 mon.a (mon.0) 9310 : audit [DBG] from='client.? 172.21.15.115:0/3237744297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: cluster 2022-04-22T20:23:01.349519+0000 mgr.y (mgr.34107) 6057 : cluster [DBG] pgmap v7250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: audit 2022-04-22T20:23:01.686346+0000 mgr.y (mgr.34107) 6058 : audit [DBG] from='client.51622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: audit 2022-04-22T20:23:02.030269+0000 mgr.y (mgr.34107) 6059 : audit [DBG] from='client.51628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: audit 2022-04-22T20:23:02.225812+0000 mon.a (mon.0) 9308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: audit 2022-04-22T20:23:02.390885+0000 mon.a (mon.0) 9309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:03.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:03 smithi115 bash[46934]: audit 2022-04-22T20:23:02.450043+0000 mon.a (mon.0) 9310 : audit [DBG] from='client.? 172.21.15.115:0/3237744297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: cluster 2022-04-22T20:23:01.349519+0000 mgr.y (mgr.34107) 6057 : cluster [DBG] pgmap v7250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: audit 2022-04-22T20:23:01.686346+0000 mgr.y (mgr.34107) 6058 : audit [DBG] from='client.51622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: audit 2022-04-22T20:23:02.030269+0000 mgr.y (mgr.34107) 6059 : audit [DBG] from='client.51628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: audit 2022-04-22T20:23:02.225812+0000 mon.a (mon.0) 9308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: audit 2022-04-22T20:23:02.390885+0000 mon.a (mon.0) 9309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:03 smithi121 bash[36902]: audit 2022-04-22T20:23:02.450043+0000 mon.a (mon.0) 9310 : audit [DBG] from='client.? 172.21.15.115:0/3237744297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:04 smithi121 bash[36902]: audit 2022-04-22T20:23:03.832788+0000 mon.a (mon.0) 9311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:04 smithi115 bash[45495]: audit 2022-04-22T20:23:03.832788+0000 mon.a (mon.0) 9311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:04.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:04.178Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:04 smithi115 bash[46934]: audit 2022-04-22T20:23:03.832788+0000 mon.a (mon.0) 9311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:05 smithi121 bash[36902]: cluster 2022-04-22T20:23:03.350729+0000 mgr.y (mgr.34107) 6060 : cluster [DBG] pgmap v7251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:05 smithi115 bash[45495]: cluster 2022-04-22T20:23:03.350729+0000 mgr.y (mgr.34107) 6060 : cluster [DBG] pgmap v7251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:05 smithi115 bash[46934]: cluster 2022-04-22T20:23:03.350729+0000 mgr.y (mgr.34107) 6060 : cluster [DBG] pgmap v7251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:07 smithi121 bash[36902]: cluster 2022-04-22T20:23:05.351772+0000 mgr.y (mgr.34107) 6061 : cluster [DBG] pgmap v7252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:07 smithi115 bash[45495]: cluster 2022-04-22T20:23:05.351772+0000 mgr.y (mgr.34107) 6061 : cluster [DBG] pgmap v7252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:07 smithi115 bash[46934]: cluster 2022-04-22T20:23:05.351772+0000 mgr.y (mgr.34107) 6061 : cluster [DBG] pgmap v7252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:08 smithi121 bash[36902]: audit 2022-04-22T20:23:07.229222+0000 mon.a (mon.0) 9312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:08 smithi121 bash[36902]: audit 2022-04-22T20:23:07.393854+0000 mon.a (mon.0) 9313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:08 smithi115 bash[45495]: audit 2022-04-22T20:23:07.229222+0000 mon.a (mon.0) 9312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:08 smithi115 bash[45495]: audit 2022-04-22T20:23:07.393854+0000 mon.a (mon.0) 9313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:08 smithi115 bash[46934]: audit 2022-04-22T20:23:07.229222+0000 mon.a (mon.0) 9312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:08 smithi115 bash[46934]: audit 2022-04-22T20:23:07.393854+0000 mon.a (mon.0) 9313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:09 smithi121 bash[36902]: cluster 2022-04-22T20:23:07.352850+0000 mgr.y (mgr.34107) 6062 : cluster [DBG] pgmap v7253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:09 smithi121 bash[36902]: audit 2022-04-22T20:23:08.835871+0000 mon.a (mon.0) 9314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:09 smithi115 bash[45495]: cluster 2022-04-22T20:23:07.352850+0000 mgr.y (mgr.34107) 6062 : cluster [DBG] pgmap v7253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:09 smithi115 bash[45495]: audit 2022-04-22T20:23:08.835871+0000 mon.a (mon.0) 9314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:09 smithi115 bash[46934]: cluster 2022-04-22T20:23:07.352850+0000 mgr.y (mgr.34107) 6062 : cluster [DBG] pgmap v7253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:09 smithi115 bash[46934]: audit 2022-04-22T20:23:08.835871+0000 mon.a (mon.0) 9314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:11 smithi121 bash[36902]: cluster 2022-04-22T20:23:09.354003+0000 mgr.y (mgr.34107) 6063 : cluster [DBG] pgmap v7254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:11 smithi115 bash[45495]: cluster 2022-04-22T20:23:09.354003+0000 mgr.y (mgr.34107) 6063 : cluster [DBG] pgmap v7254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:11 smithi115 bash[46934]: cluster 2022-04-22T20:23:09.354003+0000 mgr.y (mgr.34107) 6063 : cluster [DBG] pgmap v7254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:12.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:13 smithi121 bash[36902]: cluster 2022-04-22T20:23:11.355112+0000 mgr.y (mgr.34107) 6064 : cluster [DBG] pgmap v7255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:13 smithi121 bash[36902]: audit 2022-04-22T20:23:12.233116+0000 mon.a (mon.0) 9315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:13 smithi121 bash[36902]: audit 2022-04-22T20:23:12.396355+0000 mon.a (mon.0) 9316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[45495]: cluster 2022-04-22T20:23:11.355112+0000 mgr.y (mgr.34107) 6064 : cluster [DBG] pgmap v7255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[45495]: audit 2022-04-22T20:23:12.233116+0000 mon.a (mon.0) 9315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[45495]: audit 2022-04-22T20:23:12.396355+0000 mon.a (mon.0) 9316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[46934]: cluster 2022-04-22T20:23:11.355112+0000 mgr.y (mgr.34107) 6064 : cluster [DBG] pgmap v7255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[46934]: audit 2022-04-22T20:23:12.233116+0000 mon.a (mon.0) 9315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:13 smithi115 bash[46934]: audit 2022-04-22T20:23:12.396355+0000 mon.a (mon.0) 9316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:14 smithi121 bash[36902]: audit 2022-04-22T20:23:13.838929+0000 mon.a (mon.0) 9317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:14 smithi115 bash[45495]: audit 2022-04-22T20:23:13.838929+0000 mon.a (mon.0) 9317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:14 smithi115 bash[46934]: audit 2022-04-22T20:23:13.838929+0000 mon.a (mon.0) 9317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:14.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:14.180Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:15 smithi121 bash[36902]: cluster 2022-04-22T20:23:13.356258+0000 mgr.y (mgr.34107) 6065 : cluster [DBG] pgmap v7256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:15 smithi115 bash[45495]: cluster 2022-04-22T20:23:13.356258+0000 mgr.y (mgr.34107) 6065 : cluster [DBG] pgmap v7256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:15 smithi115 bash[46934]: cluster 2022-04-22T20:23:13.356258+0000 mgr.y (mgr.34107) 6065 : cluster [DBG] pgmap v7256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:16 smithi121 bash[36902]: audit 2022-04-22T20:23:15.310274+0000 mon.a (mon.0) 9318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:23:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:16 smithi121 bash[36902]: audit 2022-04-22T20:23:15.310601+0000 mon.a (mon.0) 9319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:23:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:16 smithi115 bash[45495]: audit 2022-04-22T20:23:15.310274+0000 mon.a (mon.0) 9318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:23:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:16 smithi115 bash[45495]: audit 2022-04-22T20:23:15.310601+0000 mon.a (mon.0) 9319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:23:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:16 smithi115 bash[46934]: audit 2022-04-22T20:23:15.310274+0000 mon.a (mon.0) 9318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:23:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:16 smithi115 bash[46934]: audit 2022-04-22T20:23:15.310601+0000 mon.a (mon.0) 9319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:23:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:17 smithi121 bash[36902]: cluster 2022-04-22T20:23:15.357334+0000 mgr.y (mgr.34107) 6066 : cluster [DBG] pgmap v7257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:17 smithi115 bash[45495]: cluster 2022-04-22T20:23:15.357334+0000 mgr.y (mgr.34107) 6066 : cluster [DBG] pgmap v7257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:17 smithi115 bash[46934]: cluster 2022-04-22T20:23:15.357334+0000 mgr.y (mgr.34107) 6066 : cluster [DBG] pgmap v7257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:18 smithi121 bash[36902]: audit 2022-04-22T20:23:17.236718+0000 mon.a (mon.0) 9320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:18 smithi121 bash[36902]: audit 2022-04-22T20:23:17.398474+0000 mon.a (mon.0) 9321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:18 smithi115 bash[45495]: audit 2022-04-22T20:23:17.236718+0000 mon.a (mon.0) 9320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:18.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:18 smithi115 bash[45495]: audit 2022-04-22T20:23:17.398474+0000 mon.a (mon.0) 9321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:18 smithi115 bash[46934]: audit 2022-04-22T20:23:17.236718+0000 mon.a (mon.0) 9320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:18.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:18 smithi115 bash[46934]: audit 2022-04-22T20:23:17.398474+0000 mon.a (mon.0) 9321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:19 smithi121 bash[36902]: cluster 2022-04-22T20:23:17.358432+0000 mgr.y (mgr.34107) 6067 : cluster [DBG] pgmap v7258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:19 smithi121 bash[36902]: audit 2022-04-22T20:23:18.841930+0000 mon.a (mon.0) 9322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:19 smithi115 bash[45495]: cluster 2022-04-22T20:23:17.358432+0000 mgr.y (mgr.34107) 6067 : cluster [DBG] pgmap v7258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:19 smithi115 bash[45495]: audit 2022-04-22T20:23:18.841930+0000 mon.a (mon.0) 9322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:19 smithi115 bash[46934]: cluster 2022-04-22T20:23:17.358432+0000 mgr.y (mgr.34107) 6067 : cluster [DBG] pgmap v7258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:19 smithi115 bash[46934]: audit 2022-04-22T20:23:18.841930+0000 mon.a (mon.0) 9322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:21 smithi121 bash[36902]: cluster 2022-04-22T20:23:19.359425+0000 mgr.y (mgr.34107) 6068 : cluster [DBG] pgmap v7259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:21 smithi115 bash[45495]: cluster 2022-04-22T20:23:19.359425+0000 mgr.y (mgr.34107) 6068 : cluster [DBG] pgmap v7259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:21 smithi115 bash[46934]: cluster 2022-04-22T20:23:19.359425+0000 mgr.y (mgr.34107) 6068 : cluster [DBG] pgmap v7259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:22.301Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:23 smithi121 bash[36902]: cluster 2022-04-22T20:23:21.360526+0000 mgr.y (mgr.34107) 6069 : cluster [DBG] pgmap v7260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:23 smithi121 bash[36902]: audit 2022-04-22T20:23:22.240599+0000 mon.a (mon.0) 9323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:23 smithi121 bash[36902]: audit 2022-04-22T20:23:22.401292+0000 mon.a (mon.0) 9324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[45495]: cluster 2022-04-22T20:23:21.360526+0000 mgr.y (mgr.34107) 6069 : cluster [DBG] pgmap v7260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[45495]: audit 2022-04-22T20:23:22.240599+0000 mon.a (mon.0) 9323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[45495]: audit 2022-04-22T20:23:22.401292+0000 mon.a (mon.0) 9324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[46934]: cluster 2022-04-22T20:23:21.360526+0000 mgr.y (mgr.34107) 6069 : cluster [DBG] pgmap v7260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[46934]: audit 2022-04-22T20:23:22.240599+0000 mon.a (mon.0) 9323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:23 smithi115 bash[46934]: audit 2022-04-22T20:23:22.401292+0000 mon.a (mon.0) 9324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:24.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:24 smithi121 bash[36902]: audit 2022-04-22T20:23:23.844884+0000 mon.a (mon.0) 9325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:24 smithi115 bash[45495]: audit 2022-04-22T20:23:23.844884+0000 mon.a (mon.0) 9325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:24.557 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:24.180Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:24.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:24 smithi115 bash[46934]: audit 2022-04-22T20:23:23.844884+0000 mon.a (mon.0) 9325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:25 smithi121 bash[36902]: cluster 2022-04-22T20:23:23.361700+0000 mgr.y (mgr.34107) 6070 : cluster [DBG] pgmap v7261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:25 smithi115 bash[45495]: cluster 2022-04-22T20:23:23.361700+0000 mgr.y (mgr.34107) 6070 : cluster [DBG] pgmap v7261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:25 smithi115 bash[46934]: cluster 2022-04-22T20:23:23.361700+0000 mgr.y (mgr.34107) 6070 : cluster [DBG] pgmap v7261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:27 smithi121 bash[36902]: cluster 2022-04-22T20:23:25.362809+0000 mgr.y (mgr.34107) 6071 : cluster [DBG] pgmap v7262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:27 smithi115 bash[45495]: cluster 2022-04-22T20:23:25.362809+0000 mgr.y (mgr.34107) 6071 : cluster [DBG] pgmap v7262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:27 smithi115 bash[46934]: cluster 2022-04-22T20:23:25.362809+0000 mgr.y (mgr.34107) 6071 : cluster [DBG] pgmap v7262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:28 smithi121 bash[36902]: audit 2022-04-22T20:23:27.244250+0000 mon.a (mon.0) 9326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:28 smithi121 bash[36902]: audit 2022-04-22T20:23:27.404312+0000 mon.a (mon.0) 9327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:28 smithi115 bash[45495]: audit 2022-04-22T20:23:27.244250+0000 mon.a (mon.0) 9326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:28 smithi115 bash[45495]: audit 2022-04-22T20:23:27.404312+0000 mon.a (mon.0) 9327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:28 smithi115 bash[46934]: audit 2022-04-22T20:23:27.244250+0000 mon.a (mon.0) 9326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:28 smithi115 bash[46934]: audit 2022-04-22T20:23:27.404312+0000 mon.a (mon.0) 9327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:29 smithi121 bash[36902]: cluster 2022-04-22T20:23:27.363897+0000 mgr.y (mgr.34107) 6072 : cluster [DBG] pgmap v7263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:29 smithi121 bash[36902]: audit 2022-04-22T20:23:28.848034+0000 mon.a (mon.0) 9328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:29 smithi115 bash[46934]: cluster 2022-04-22T20:23:27.363897+0000 mgr.y (mgr.34107) 6072 : cluster [DBG] pgmap v7263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:29 smithi115 bash[46934]: audit 2022-04-22T20:23:28.848034+0000 mon.a (mon.0) 9328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:29 smithi115 bash[45495]: cluster 2022-04-22T20:23:27.363897+0000 mgr.y (mgr.34107) 6072 : cluster [DBG] pgmap v7263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:29.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:29 smithi115 bash[45495]: audit 2022-04-22T20:23:28.848034+0000 mon.a (mon.0) 9328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:30 smithi121 bash[36902]: cluster 2022-04-22T20:23:29.365137+0000 mgr.y (mgr.34107) 6073 : cluster [DBG] pgmap v7264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:30.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:30 smithi115 bash[45495]: cluster 2022-04-22T20:23:29.365137+0000 mgr.y (mgr.34107) 6073 : cluster [DBG] pgmap v7264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:30 smithi115 bash[46934]: cluster 2022-04-22T20:23:29.365137+0000 mgr.y (mgr.34107) 6073 : cluster [DBG] pgmap v7264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:32 smithi121 bash[36902]: cluster 2022-04-22T20:23:31.366255+0000 mgr.y (mgr.34107) 6074 : cluster [DBG] pgmap v7265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:32 smithi121 bash[36902]: audit 2022-04-22T20:23:32.247836+0000 mon.a (mon.0) 9329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:32 smithi121 bash[36902]: audit 2022-04-22T20:23:32.407055+0000 mon.a (mon.0) 9330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:32.301Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[45495]: cluster 2022-04-22T20:23:31.366255+0000 mgr.y (mgr.34107) 6074 : cluster [DBG] pgmap v7265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[45495]: audit 2022-04-22T20:23:32.247836+0000 mon.a (mon.0) 9329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[45495]: audit 2022-04-22T20:23:32.407055+0000 mon.a (mon.0) 9330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[46934]: cluster 2022-04-22T20:23:31.366255+0000 mgr.y (mgr.34107) 6074 : cluster [DBG] pgmap v7265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[46934]: audit 2022-04-22T20:23:32.247836+0000 mon.a (mon.0) 9329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:32 smithi115 bash[46934]: audit 2022-04-22T20:23:32.407055+0000 mon.a (mon.0) 9330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:32.836 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:23:33.182 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:23:33.183 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 59s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:23:33.183 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:23:33.183 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 59s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 59s ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 59s ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 59s ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 59s ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 59s ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 59s ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:23:33.184 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 59s ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 59s ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:23:33.185 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:23:33.594 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:23:33.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:33.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:23:33.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:23:33.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:23:33.596 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:23:33.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:23:33.602 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:23:33.602 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:23:33.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:33 smithi115 bash[46934]: audit 2022-04-22T20:23:32.811885+0000 mgr.y (mgr.34107) 6075 : audit [DBG] from='client.51640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:33.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:33 smithi115 bash[45495]: audit 2022-04-22T20:23:32.811885+0000 mgr.y (mgr.34107) 6075 : audit [DBG] from='client.51640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:33.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:33 smithi121 bash[36902]: audit 2022-04-22T20:23:32.811885+0000 mgr.y (mgr.34107) 6075 : audit [DBG] from='client.51640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:34.441 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:34.181Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.152159+0000 mgr.y (mgr.34107) 6076 : audit [DBG] from='client.41712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:34.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: cluster 2022-04-22T20:23:33.367349+0000 mgr.y (mgr.34107) 6077 : cluster [DBG] pgmap v7266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.586097+0000 mon.a (mon.0) 9331 : audit [DBG] from='client.? 172.21.15.115:0/2791930909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.720136+0000 mon.a (mon.0) 9332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.723131+0000 mon.a (mon.0) 9333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.724826+0000 mon.a (mon.0) 9334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:33.851015+0000 mon.a (mon.0) 9335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:34.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:34 smithi121 bash[36902]: audit 2022-04-22T20:23:34.082915+0000 mon.a (mon.0) 9336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.152159+0000 mgr.y (mgr.34107) 6076 : audit [DBG] from='client.41712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: cluster 2022-04-22T20:23:33.367349+0000 mgr.y (mgr.34107) 6077 : cluster [DBG] pgmap v7266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.586097+0000 mon.a (mon.0) 9331 : audit [DBG] from='client.? 172.21.15.115:0/2791930909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.720136+0000 mon.a (mon.0) 9332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.723131+0000 mon.a (mon.0) 9333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.724826+0000 mon.a (mon.0) 9334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:33.851015+0000 mon.a (mon.0) 9335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[45495]: audit 2022-04-22T20:23:34.082915+0000 mon.a (mon.0) 9336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.152159+0000 mgr.y (mgr.34107) 6076 : audit [DBG] from='client.41712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: cluster 2022-04-22T20:23:33.367349+0000 mgr.y (mgr.34107) 6077 : cluster [DBG] pgmap v7266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.586097+0000 mon.a (mon.0) 9331 : audit [DBG] from='client.? 172.21.15.115:0/2791930909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.720136+0000 mon.a (mon.0) 9332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.723131+0000 mon.a (mon.0) 9333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:23:34.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.724826+0000 mon.a (mon.0) 9334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:23:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:33.851015+0000 mon.a (mon.0) 9335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:34 smithi115 bash[46934]: audit 2022-04-22T20:23:34.082915+0000 mon.a (mon.0) 9336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:23:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:36 smithi121 bash[36902]: cluster 2022-04-22T20:23:35.368407+0000 mgr.y (mgr.34107) 6078 : cluster [DBG] pgmap v7267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:36 smithi115 bash[45495]: cluster 2022-04-22T20:23:35.368407+0000 mgr.y (mgr.34107) 6078 : cluster [DBG] pgmap v7267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:36 smithi115 bash[46934]: cluster 2022-04-22T20:23:35.368407+0000 mgr.y (mgr.34107) 6078 : cluster [DBG] pgmap v7267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:37 smithi121 bash[36902]: audit 2022-04-22T20:23:37.251343+0000 mon.a (mon.0) 9337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:37 smithi121 bash[36902]: audit 2022-04-22T20:23:37.409592+0000 mon.a (mon.0) 9338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:37 smithi115 bash[45495]: audit 2022-04-22T20:23:37.251343+0000 mon.a (mon.0) 9337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:37 smithi115 bash[45495]: audit 2022-04-22T20:23:37.409592+0000 mon.a (mon.0) 9338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:37 smithi115 bash[46934]: audit 2022-04-22T20:23:37.251343+0000 mon.a (mon.0) 9337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:37 smithi115 bash[46934]: audit 2022-04-22T20:23:37.409592+0000 mon.a (mon.0) 9338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:38 smithi121 bash[36902]: cluster 2022-04-22T20:23:37.369506+0000 mgr.y (mgr.34107) 6079 : cluster [DBG] pgmap v7268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:38.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:38 smithi115 bash[45495]: cluster 2022-04-22T20:23:37.369506+0000 mgr.y (mgr.34107) 6079 : cluster [DBG] pgmap v7268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:38 smithi115 bash[46934]: cluster 2022-04-22T20:23:37.369506+0000 mgr.y (mgr.34107) 6079 : cluster [DBG] pgmap v7268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:39 smithi121 bash[36902]: audit 2022-04-22T20:23:38.853936+0000 mon.a (mon.0) 9339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:39 smithi115 bash[45495]: audit 2022-04-22T20:23:38.853936+0000 mon.a (mon.0) 9339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:39 smithi115 bash[46934]: audit 2022-04-22T20:23:38.853936+0000 mon.a (mon.0) 9339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:40 smithi121 bash[36902]: cluster 2022-04-22T20:23:39.370575+0000 mgr.y (mgr.34107) 6080 : cluster [DBG] pgmap v7269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:40 smithi115 bash[45495]: cluster 2022-04-22T20:23:39.370575+0000 mgr.y (mgr.34107) 6080 : cluster [DBG] pgmap v7269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:40 smithi115 bash[46934]: cluster 2022-04-22T20:23:39.370575+0000 mgr.y (mgr.34107) 6080 : cluster [DBG] pgmap v7269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:42.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:42 smithi121 bash[36902]: cluster 2022-04-22T20:23:41.371730+0000 mgr.y (mgr.34107) 6081 : cluster [DBG] pgmap v7270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:42 smithi121 bash[36902]: audit 2022-04-22T20:23:42.254915+0000 mon.a (mon.0) 9340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:42 smithi121 bash[36902]: audit 2022-04-22T20:23:42.412249+0000 mon.a (mon.0) 9341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[45495]: cluster 2022-04-22T20:23:41.371730+0000 mgr.y (mgr.34107) 6081 : cluster [DBG] pgmap v7270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[45495]: audit 2022-04-22T20:23:42.254915+0000 mon.a (mon.0) 9340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[45495]: audit 2022-04-22T20:23:42.412249+0000 mon.a (mon.0) 9341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[46934]: cluster 2022-04-22T20:23:41.371730+0000 mgr.y (mgr.34107) 6081 : cluster [DBG] pgmap v7270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[46934]: audit 2022-04-22T20:23:42.254915+0000 mon.a (mon.0) 9340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:42 smithi115 bash[46934]: audit 2022-04-22T20:23:42.412249+0000 mon.a (mon.0) 9341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:43 smithi121 bash[36902]: audit 2022-04-22T20:23:43.857230+0000 mon.a (mon.0) 9342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:44.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:44.181Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:44.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:43 smithi115 bash[45495]: audit 2022-04-22T20:23:43.857230+0000 mon.a (mon.0) 9342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:43 smithi115 bash[46934]: audit 2022-04-22T20:23:43.857230+0000 mon.a (mon.0) 9342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:44 smithi121 bash[36902]: cluster 2022-04-22T20:23:43.372723+0000 mgr.y (mgr.34107) 6082 : cluster [DBG] pgmap v7271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:45.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:44 smithi115 bash[45495]: cluster 2022-04-22T20:23:43.372723+0000 mgr.y (mgr.34107) 6082 : cluster [DBG] pgmap v7271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:44 smithi115 bash[46934]: cluster 2022-04-22T20:23:43.372723+0000 mgr.y (mgr.34107) 6082 : cluster [DBG] pgmap v7271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:47 smithi121 bash[36902]: cluster 2022-04-22T20:23:45.373814+0000 mgr.y (mgr.34107) 6083 : cluster [DBG] pgmap v7272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:47.388 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:46 smithi115 bash[45495]: cluster 2022-04-22T20:23:45.373814+0000 mgr.y (mgr.34107) 6083 : cluster [DBG] pgmap v7272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:47.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:46 smithi115 bash[46934]: cluster 2022-04-22T20:23:45.373814+0000 mgr.y (mgr.34107) 6083 : cluster [DBG] pgmap v7272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:48 smithi121 bash[36902]: audit 2022-04-22T20:23:47.258820+0000 mon.a (mon.0) 9343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:48 smithi121 bash[36902]: audit 2022-04-22T20:23:47.415058+0000 mon.a (mon.0) 9344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:48 smithi115 bash[45495]: audit 2022-04-22T20:23:47.258820+0000 mon.a (mon.0) 9343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:48 smithi115 bash[45495]: audit 2022-04-22T20:23:47.415058+0000 mon.a (mon.0) 9344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:48 smithi115 bash[46934]: audit 2022-04-22T20:23:47.258820+0000 mon.a (mon.0) 9343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:48 smithi115 bash[46934]: audit 2022-04-22T20:23:47.415058+0000 mon.a (mon.0) 9344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:49.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:49 smithi115 bash[45495]: cluster 2022-04-22T20:23:47.374831+0000 mgr.y (mgr.34107) 6084 : cluster [DBG] pgmap v7273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:49 smithi115 bash[45495]: audit 2022-04-22T20:23:48.860368+0000 mon.a (mon.0) 9345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:49 smithi115 bash[46934]: cluster 2022-04-22T20:23:47.374831+0000 mgr.y (mgr.34107) 6084 : cluster [DBG] pgmap v7273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:49 smithi115 bash[46934]: audit 2022-04-22T20:23:48.860368+0000 mon.a (mon.0) 9345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:49 smithi121 bash[36902]: cluster 2022-04-22T20:23:47.374831+0000 mgr.y (mgr.34107) 6084 : cluster [DBG] pgmap v7273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:49 smithi121 bash[36902]: audit 2022-04-22T20:23:48.860368+0000 mon.a (mon.0) 9345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:51 smithi115 bash[45495]: cluster 2022-04-22T20:23:49.375958+0000 mgr.y (mgr.34107) 6085 : cluster [DBG] pgmap v7274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:51 smithi115 bash[46934]: cluster 2022-04-22T20:23:49.375958+0000 mgr.y (mgr.34107) 6085 : cluster [DBG] pgmap v7274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:51 smithi121 bash[36902]: cluster 2022-04-22T20:23:49.375958+0000 mgr.y (mgr.34107) 6085 : cluster [DBG] pgmap v7274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:52.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[45495]: cluster 2022-04-22T20:23:51.377036+0000 mgr.y (mgr.34107) 6086 : cluster [DBG] pgmap v7275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[45495]: audit 2022-04-22T20:23:52.262260+0000 mon.a (mon.0) 9346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:53.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[45495]: audit 2022-04-22T20:23:52.417960+0000 mon.a (mon.0) 9347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[46934]: cluster 2022-04-22T20:23:51.377036+0000 mgr.y (mgr.34107) 6086 : cluster [DBG] pgmap v7275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[46934]: audit 2022-04-22T20:23:52.262260+0000 mon.a (mon.0) 9346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:53 smithi115 bash[46934]: audit 2022-04-22T20:23:52.417960+0000 mon.a (mon.0) 9347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[36902]: cluster 2022-04-22T20:23:51.377036+0000 mgr.y (mgr.34107) 6086 : cluster [DBG] pgmap v7275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[36902]: audit 2022-04-22T20:23:52.262260+0000 mon.a (mon.0) 9346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[36902]: audit 2022-04-22T20:23:52.417960+0000 mon.a (mon.0) 9347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:54.015 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[49697]: ts=2022-04-22T20:23:53.541Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1650648232300 maxt=1650650400000 ulid=01G19F40DAKT77FZWCC92S7EKV duration=26.535427ms 2022-04-22T20:23:54.015 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[49697]: ts=2022-04-22T20:23:53.544Z caller=head.go:827 level=info component=tsdb msg="Head GC completed" duration=2.481074ms 2022-04-22T20:23:54.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[49697]: ts=2022-04-22T20:23:53.544Z caller=checkpoint.go:98 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 mint=1650650400000 2022-04-22T20:23:54.016 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 20:23:53 smithi121 bash[49697]: ts=2022-04-22T20:23:53.548Z caller=head.go:996 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=3.896322ms 2022-04-22T20:23:54.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:54 smithi115 bash[45495]: audit 2022-04-22T20:23:53.863469+0000 mon.a (mon.0) 9348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:54.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:54 smithi115 bash[46934]: audit 2022-04-22T20:23:53.863469+0000 mon.a (mon.0) 9348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:54.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:23:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:23:54.182Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:23:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:54 smithi121 bash[36902]: audit 2022-04-22T20:23:53.863469+0000 mon.a (mon.0) 9348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:55.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:55 smithi115 bash[45495]: cluster 2022-04-22T20:23:53.378177+0000 mgr.y (mgr.34107) 6087 : cluster [DBG] pgmap v7276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:55.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:55 smithi115 bash[46934]: cluster 2022-04-22T20:23:53.378177+0000 mgr.y (mgr.34107) 6087 : cluster [DBG] pgmap v7276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:55 smithi121 bash[36902]: cluster 2022-04-22T20:23:53.378177+0000 mgr.y (mgr.34107) 6087 : cluster [DBG] pgmap v7276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:57 smithi121 bash[36902]: cluster 2022-04-22T20:23:55.379375+0000 mgr.y (mgr.34107) 6088 : cluster [DBG] pgmap v7277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:57 smithi115 bash[45495]: cluster 2022-04-22T20:23:55.379375+0000 mgr.y (mgr.34107) 6088 : cluster [DBG] pgmap v7277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:57 smithi115 bash[46934]: cluster 2022-04-22T20:23:55.379375+0000 mgr.y (mgr.34107) 6088 : cluster [DBG] pgmap v7277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:58 smithi121 bash[36902]: audit 2022-04-22T20:23:57.265795+0000 mon.a (mon.0) 9349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:58 smithi121 bash[36902]: audit 2022-04-22T20:23:57.420732+0000 mon.a (mon.0) 9350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:58 smithi115 bash[46934]: audit 2022-04-22T20:23:57.265795+0000 mon.a (mon.0) 9349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:58 smithi115 bash[46934]: audit 2022-04-22T20:23:57.420732+0000 mon.a (mon.0) 9350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:58 smithi115 bash[45495]: audit 2022-04-22T20:23:57.265795+0000 mon.a (mon.0) 9349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:23:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:58 smithi115 bash[45495]: audit 2022-04-22T20:23:57.420732+0000 mon.a (mon.0) 9350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:23:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:59 smithi121 bash[36902]: cluster 2022-04-22T20:23:57.380547+0000 mgr.y (mgr.34107) 6089 : cluster [DBG] pgmap v7278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:23:59 smithi121 bash[36902]: audit 2022-04-22T20:23:58.866633+0000 mon.a (mon.0) 9351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:59 smithi115 bash[46934]: cluster 2022-04-22T20:23:57.380547+0000 mgr.y (mgr.34107) 6089 : cluster [DBG] pgmap v7278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:23:59 smithi115 bash[46934]: audit 2022-04-22T20:23:58.866633+0000 mon.a (mon.0) 9351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:23:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:59 smithi115 bash[45495]: cluster 2022-04-22T20:23:57.380547+0000 mgr.y (mgr.34107) 6089 : cluster [DBG] pgmap v7278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:23:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:23:59 smithi115 bash[45495]: audit 2022-04-22T20:23:58.866633+0000 mon.a (mon.0) 9351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:01 smithi121 bash[36902]: cluster 2022-04-22T20:23:59.381809+0000 mgr.y (mgr.34107) 6090 : cluster [DBG] pgmap v7279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:01 smithi115 bash[46934]: cluster 2022-04-22T20:23:59.381809+0000 mgr.y (mgr.34107) 6090 : cluster [DBG] pgmap v7279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:01.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:01 smithi115 bash[45495]: cluster 2022-04-22T20:23:59.381809+0000 mgr.y (mgr.34107) 6090 : cluster [DBG] pgmap v7279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:02.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:03 smithi121 bash[36902]: cluster 2022-04-22T20:24:01.382898+0000 mgr.y (mgr.34107) 6091 : cluster [DBG] pgmap v7280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:03 smithi121 bash[36902]: audit 2022-04-22T20:24:02.269443+0000 mon.a (mon.0) 9352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:03 smithi121 bash[36902]: audit 2022-04-22T20:24:02.423308+0000 mon.a (mon.0) 9353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[45495]: cluster 2022-04-22T20:24:01.382898+0000 mgr.y (mgr.34107) 6091 : cluster [DBG] pgmap v7280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[45495]: audit 2022-04-22T20:24:02.269443+0000 mon.a (mon.0) 9352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[45495]: audit 2022-04-22T20:24:02.423308+0000 mon.a (mon.0) 9353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[46934]: cluster 2022-04-22T20:24:01.382898+0000 mgr.y (mgr.34107) 6091 : cluster [DBG] pgmap v7280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[46934]: audit 2022-04-22T20:24:02.269443+0000 mon.a (mon.0) 9352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:03 smithi115 bash[46934]: audit 2022-04-22T20:24:02.423308+0000 mon.a (mon.0) 9353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:03.971 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 90s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 90s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 90s ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 90s ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:24:04.317 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 2m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 90s ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 90s ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 90s ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 90s ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 90s ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 90s ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 2m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 2m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 2m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:24:04.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:24:04.319 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:24:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:04 smithi121 bash[36902]: audit 2022-04-22T20:24:03.869615+0000 mon.a (mon.0) 9354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:04 smithi115 bash[46934]: audit 2022-04-22T20:24:03.869615+0000 mon.a (mon.0) 9354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:04.555 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:04.179Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:04.556 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:04 smithi115 bash[45495]: audit 2022-04-22T20:24:03.869615+0000 mon.a (mon.0) 9354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:04.720 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:24:04.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:24:04.722 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:24:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:05 smithi121 bash[36902]: cluster 2022-04-22T20:24:03.384061+0000 mgr.y (mgr.34107) 6092 : cluster [DBG] pgmap v7281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:05 smithi121 bash[36902]: audit 2022-04-22T20:24:03.942793+0000 mgr.y (mgr.34107) 6093 : audit [DBG] from='client.41724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:05 smithi121 bash[36902]: audit 2022-04-22T20:24:04.710960+0000 mon.c (mon.1) 240 : audit [DBG] from='client.? 172.21.15.115:0/1869234967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[46934]: cluster 2022-04-22T20:24:03.384061+0000 mgr.y (mgr.34107) 6092 : cluster [DBG] pgmap v7281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[46934]: audit 2022-04-22T20:24:03.942793+0000 mgr.y (mgr.34107) 6093 : audit [DBG] from='client.41724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[46934]: audit 2022-04-22T20:24:04.710960+0000 mon.c (mon.1) 240 : audit [DBG] from='client.? 172.21.15.115:0/1869234967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[45495]: cluster 2022-04-22T20:24:03.384061+0000 mgr.y (mgr.34107) 6092 : cluster [DBG] pgmap v7281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[45495]: audit 2022-04-22T20:24:03.942793+0000 mgr.y (mgr.34107) 6093 : audit [DBG] from='client.41724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:05.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:05 smithi115 bash[45495]: audit 2022-04-22T20:24:04.710960+0000 mon.c (mon.1) 240 : audit [DBG] from='client.? 172.21.15.115:0/1869234967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:06 smithi121 bash[36902]: audit 2022-04-22T20:24:04.288714+0000 mgr.y (mgr.34107) 6094 : audit [DBG] from='client.41730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:06.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:06 smithi115 bash[46934]: audit 2022-04-22T20:24:04.288714+0000 mgr.y (mgr.34107) 6094 : audit [DBG] from='client.41730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:06 smithi115 bash[45495]: audit 2022-04-22T20:24:04.288714+0000 mgr.y (mgr.34107) 6094 : audit [DBG] from='client.41730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:07 smithi121 bash[36902]: cluster 2022-04-22T20:24:05.385137+0000 mgr.y (mgr.34107) 6095 : cluster [DBG] pgmap v7282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:07 smithi115 bash[46934]: cluster 2022-04-22T20:24:05.385137+0000 mgr.y (mgr.34107) 6095 : cluster [DBG] pgmap v7282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:07 smithi115 bash[45495]: cluster 2022-04-22T20:24:05.385137+0000 mgr.y (mgr.34107) 6095 : cluster [DBG] pgmap v7282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:08 smithi121 bash[36902]: audit 2022-04-22T20:24:07.273018+0000 mon.a (mon.0) 9355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:08 smithi121 bash[36902]: audit 2022-04-22T20:24:07.426004+0000 mon.a (mon.0) 9356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:08 smithi115 bash[46934]: audit 2022-04-22T20:24:07.273018+0000 mon.a (mon.0) 9355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:08.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:08 smithi115 bash[46934]: audit 2022-04-22T20:24:07.426004+0000 mon.a (mon.0) 9356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:08 smithi115 bash[45495]: audit 2022-04-22T20:24:07.273018+0000 mon.a (mon.0) 9355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:08.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:08 smithi115 bash[45495]: audit 2022-04-22T20:24:07.426004+0000 mon.a (mon.0) 9356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:09.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:09 smithi121 bash[36902]: cluster 2022-04-22T20:24:07.386151+0000 mgr.y (mgr.34107) 6096 : cluster [DBG] pgmap v7283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:09.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:09 smithi121 bash[36902]: audit 2022-04-22T20:24:08.872345+0000 mon.a (mon.0) 9357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:09 smithi115 bash[46934]: cluster 2022-04-22T20:24:07.386151+0000 mgr.y (mgr.34107) 6096 : cluster [DBG] pgmap v7283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:09 smithi115 bash[46934]: audit 2022-04-22T20:24:08.872345+0000 mon.a (mon.0) 9357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:09 smithi115 bash[45495]: cluster 2022-04-22T20:24:07.386151+0000 mgr.y (mgr.34107) 6096 : cluster [DBG] pgmap v7283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:09 smithi115 bash[45495]: audit 2022-04-22T20:24:08.872345+0000 mon.a (mon.0) 9357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:11 smithi121 bash[36902]: cluster 2022-04-22T20:24:09.387271+0000 mgr.y (mgr.34107) 6097 : cluster [DBG] pgmap v7284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:11 smithi115 bash[45495]: cluster 2022-04-22T20:24:09.387271+0000 mgr.y (mgr.34107) 6097 : cluster [DBG] pgmap v7284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:11 smithi115 bash[46934]: cluster 2022-04-22T20:24:09.387271+0000 mgr.y (mgr.34107) 6097 : cluster [DBG] pgmap v7284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:12.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:12.303Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:13 smithi121 bash[36902]: cluster 2022-04-22T20:24:11.388372+0000 mgr.y (mgr.34107) 6098 : cluster [DBG] pgmap v7285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:13 smithi121 bash[36902]: audit 2022-04-22T20:24:12.276556+0000 mon.a (mon.0) 9358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:13 smithi121 bash[36902]: audit 2022-04-22T20:24:12.429110+0000 mon.a (mon.0) 9359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[45495]: cluster 2022-04-22T20:24:11.388372+0000 mgr.y (mgr.34107) 6098 : cluster [DBG] pgmap v7285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[45495]: audit 2022-04-22T20:24:12.276556+0000 mon.a (mon.0) 9358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[45495]: audit 2022-04-22T20:24:12.429110+0000 mon.a (mon.0) 9359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[46934]: cluster 2022-04-22T20:24:11.388372+0000 mgr.y (mgr.34107) 6098 : cluster [DBG] pgmap v7285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[46934]: audit 2022-04-22T20:24:12.276556+0000 mon.a (mon.0) 9358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:13.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:13 smithi115 bash[46934]: audit 2022-04-22T20:24:12.429110+0000 mon.a (mon.0) 9359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:14 smithi121 bash[36902]: audit 2022-04-22T20:24:13.875706+0000 mon.a (mon.0) 9360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:14 smithi115 bash[45495]: audit 2022-04-22T20:24:13.875706+0000 mon.a (mon.0) 9360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:14.183Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:14 smithi115 bash[46934]: audit 2022-04-22T20:24:13.875706+0000 mon.a (mon.0) 9360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:15 smithi121 bash[36902]: cluster 2022-04-22T20:24:13.389528+0000 mgr.y (mgr.34107) 6099 : cluster [DBG] pgmap v7286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:15 smithi115 bash[45495]: cluster 2022-04-22T20:24:13.389528+0000 mgr.y (mgr.34107) 6099 : cluster [DBG] pgmap v7286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:15 smithi115 bash[46934]: cluster 2022-04-22T20:24:13.389528+0000 mgr.y (mgr.34107) 6099 : cluster [DBG] pgmap v7286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:16 smithi121 bash[36902]: audit 2022-04-22T20:24:15.321587+0000 mon.a (mon.0) 9361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:24:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:16 smithi121 bash[36902]: audit 2022-04-22T20:24:15.325214+0000 mon.a (mon.0) 9362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:24:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:16 smithi115 bash[45495]: audit 2022-04-22T20:24:15.321587+0000 mon.a (mon.0) 9361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:24:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:16 smithi115 bash[45495]: audit 2022-04-22T20:24:15.325214+0000 mon.a (mon.0) 9362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:24:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:16 smithi115 bash[46934]: audit 2022-04-22T20:24:15.321587+0000 mon.a (mon.0) 9361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:24:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:16 smithi115 bash[46934]: audit 2022-04-22T20:24:15.325214+0000 mon.a (mon.0) 9362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:24:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:17 smithi121 bash[36902]: cluster 2022-04-22T20:24:15.390590+0000 mgr.y (mgr.34107) 6100 : cluster [DBG] pgmap v7287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:17 smithi115 bash[45495]: cluster 2022-04-22T20:24:15.390590+0000 mgr.y (mgr.34107) 6100 : cluster [DBG] pgmap v7287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:17 smithi115 bash[46934]: cluster 2022-04-22T20:24:15.390590+0000 mgr.y (mgr.34107) 6100 : cluster [DBG] pgmap v7287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:18 smithi121 bash[36902]: audit 2022-04-22T20:24:17.280409+0000 mon.a (mon.0) 9363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:18 smithi121 bash[36902]: audit 2022-04-22T20:24:17.432255+0000 mon.a (mon.0) 9364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:18 smithi115 bash[45495]: audit 2022-04-22T20:24:17.280409+0000 mon.a (mon.0) 9363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:18 smithi115 bash[45495]: audit 2022-04-22T20:24:17.432255+0000 mon.a (mon.0) 9364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:18 smithi115 bash[46934]: audit 2022-04-22T20:24:17.280409+0000 mon.a (mon.0) 9363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:18 smithi115 bash[46934]: audit 2022-04-22T20:24:17.432255+0000 mon.a (mon.0) 9364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:19 smithi121 bash[36902]: cluster 2022-04-22T20:24:17.391674+0000 mgr.y (mgr.34107) 6101 : cluster [DBG] pgmap v7288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:19 smithi121 bash[36902]: audit 2022-04-22T20:24:18.878597+0000 mon.a (mon.0) 9365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:19 smithi115 bash[45495]: cluster 2022-04-22T20:24:17.391674+0000 mgr.y (mgr.34107) 6101 : cluster [DBG] pgmap v7288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:19.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:19 smithi115 bash[45495]: audit 2022-04-22T20:24:18.878597+0000 mon.a (mon.0) 9365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:19 smithi115 bash[46934]: cluster 2022-04-22T20:24:17.391674+0000 mgr.y (mgr.34107) 6101 : cluster [DBG] pgmap v7288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:19 smithi115 bash[46934]: audit 2022-04-22T20:24:18.878597+0000 mon.a (mon.0) 9365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:21 smithi121 bash[36902]: cluster 2022-04-22T20:24:19.392701+0000 mgr.y (mgr.34107) 6102 : cluster [DBG] pgmap v7289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:21 smithi115 bash[45495]: cluster 2022-04-22T20:24:19.392701+0000 mgr.y (mgr.34107) 6102 : cluster [DBG] pgmap v7289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:21.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:21 smithi115 bash[46934]: cluster 2022-04-22T20:24:19.392701+0000 mgr.y (mgr.34107) 6102 : cluster [DBG] pgmap v7289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:22.303Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:23 smithi121 bash[36902]: cluster 2022-04-22T20:24:21.393768+0000 mgr.y (mgr.34107) 6103 : cluster [DBG] pgmap v7290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:23 smithi121 bash[36902]: audit 2022-04-22T20:24:22.284060+0000 mon.a (mon.0) 9366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:23 smithi121 bash[36902]: audit 2022-04-22T20:24:22.435302+0000 mon.a (mon.0) 9367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[45495]: cluster 2022-04-22T20:24:21.393768+0000 mgr.y (mgr.34107) 6103 : cluster [DBG] pgmap v7290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[45495]: audit 2022-04-22T20:24:22.284060+0000 mon.a (mon.0) 9366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[45495]: audit 2022-04-22T20:24:22.435302+0000 mon.a (mon.0) 9367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[46934]: cluster 2022-04-22T20:24:21.393768+0000 mgr.y (mgr.34107) 6103 : cluster [DBG] pgmap v7290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[46934]: audit 2022-04-22T20:24:22.284060+0000 mon.a (mon.0) 9366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:23 smithi115 bash[46934]: audit 2022-04-22T20:24:22.435302+0000 mon.a (mon.0) 9367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:24 smithi121 bash[36902]: audit 2022-04-22T20:24:23.881735+0000 mon.a (mon.0) 9368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:24 smithi115 bash[45495]: audit 2022-04-22T20:24:23.881735+0000 mon.a (mon.0) 9368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:24 smithi115 bash[46934]: audit 2022-04-22T20:24:23.881735+0000 mon.a (mon.0) 9368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:24.183Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:25 smithi121 bash[36902]: cluster 2022-04-22T20:24:23.394865+0000 mgr.y (mgr.34107) 6104 : cluster [DBG] pgmap v7291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:25 smithi115 bash[45495]: cluster 2022-04-22T20:24:23.394865+0000 mgr.y (mgr.34107) 6104 : cluster [DBG] pgmap v7291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:25 smithi115 bash[46934]: cluster 2022-04-22T20:24:23.394865+0000 mgr.y (mgr.34107) 6104 : cluster [DBG] pgmap v7291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:27 smithi121 bash[36902]: cluster 2022-04-22T20:24:25.395951+0000 mgr.y (mgr.34107) 6105 : cluster [DBG] pgmap v7292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:27 smithi115 bash[45495]: cluster 2022-04-22T20:24:25.395951+0000 mgr.y (mgr.34107) 6105 : cluster [DBG] pgmap v7292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:27.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:27 smithi115 bash[46934]: cluster 2022-04-22T20:24:25.395951+0000 mgr.y (mgr.34107) 6105 : cluster [DBG] pgmap v7292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:28 smithi121 bash[36902]: audit 2022-04-22T20:24:27.287766+0000 mon.a (mon.0) 9369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:28 smithi121 bash[36902]: audit 2022-04-22T20:24:27.438315+0000 mon.a (mon.0) 9370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:28 smithi115 bash[45495]: audit 2022-04-22T20:24:27.287766+0000 mon.a (mon.0) 9369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:28 smithi115 bash[45495]: audit 2022-04-22T20:24:27.438315+0000 mon.a (mon.0) 9370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:28 smithi115 bash[46934]: audit 2022-04-22T20:24:27.287766+0000 mon.a (mon.0) 9369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:28 smithi115 bash[46934]: audit 2022-04-22T20:24:27.438315+0000 mon.a (mon.0) 9370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:29 smithi121 bash[36902]: cluster 2022-04-22T20:24:27.397005+0000 mgr.y (mgr.34107) 6106 : cluster [DBG] pgmap v7293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:29 smithi121 bash[36902]: audit 2022-04-22T20:24:28.884744+0000 mon.a (mon.0) 9371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:29 smithi115 bash[45495]: cluster 2022-04-22T20:24:27.397005+0000 mgr.y (mgr.34107) 6106 : cluster [DBG] pgmap v7293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:29 smithi115 bash[45495]: audit 2022-04-22T20:24:28.884744+0000 mon.a (mon.0) 9371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:29 smithi115 bash[46934]: cluster 2022-04-22T20:24:27.397005+0000 mgr.y (mgr.34107) 6106 : cluster [DBG] pgmap v7293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:29 smithi115 bash[46934]: audit 2022-04-22T20:24:28.884744+0000 mon.a (mon.0) 9371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:30 smithi121 bash[36902]: cluster 2022-04-22T20:24:29.398107+0000 mgr.y (mgr.34107) 6107 : cluster [DBG] pgmap v7294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:30 smithi115 bash[45495]: cluster 2022-04-22T20:24:29.398107+0000 mgr.y (mgr.34107) 6107 : cluster [DBG] pgmap v7294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:30 smithi115 bash[46934]: cluster 2022-04-22T20:24:29.398107+0000 mgr.y (mgr.34107) 6107 : cluster [DBG] pgmap v7294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:32 smithi121 bash[36902]: cluster 2022-04-22T20:24:31.399174+0000 mgr.y (mgr.34107) 6108 : cluster [DBG] pgmap v7295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:32 smithi121 bash[36902]: audit 2022-04-22T20:24:32.291594+0000 mon.a (mon.0) 9372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:32 smithi121 bash[36902]: audit 2022-04-22T20:24:32.440980+0000 mon.a (mon.0) 9373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[45495]: cluster 2022-04-22T20:24:31.399174+0000 mgr.y (mgr.34107) 6108 : cluster [DBG] pgmap v7295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[45495]: audit 2022-04-22T20:24:32.291594+0000 mon.a (mon.0) 9372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[45495]: audit 2022-04-22T20:24:32.440980+0000 mon.a (mon.0) 9373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:32.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:32.304Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[46934]: cluster 2022-04-22T20:24:31.399174+0000 mgr.y (mgr.34107) 6108 : cluster [DBG] pgmap v7295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[46934]: audit 2022-04-22T20:24:32.291594+0000 mon.a (mon.0) 9372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:32 smithi115 bash[46934]: audit 2022-04-22T20:24:32.440980+0000 mon.a (mon.0) 9373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:34.466 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:34.183Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:34 smithi121 bash[36902]: cluster 2022-04-22T20:24:33.400461+0000 mgr.y (mgr.34107) 6109 : cluster [DBG] pgmap v7296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:34.841 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:34 smithi121 bash[36902]: audit 2022-04-22T20:24:33.887921+0000 mon.a (mon.0) 9374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:34.841 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:34 smithi121 bash[36902]: audit 2022-04-22T20:24:34.091670+0000 mon.a (mon.0) 9375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:24:34.841 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:34 smithi121 bash[36902]: audit 2022-04-22T20:24:34.094547+0000 mon.a (mon.0) 9376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:24:34.841 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:34 smithi121 bash[36902]: audit 2022-04-22T20:24:34.096216+0000 mon.a (mon.0) 9377 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[46934]: cluster 2022-04-22T20:24:33.400461+0000 mgr.y (mgr.34107) 6109 : cluster [DBG] pgmap v7296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[46934]: audit 2022-04-22T20:24:33.887921+0000 mon.a (mon.0) 9374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[46934]: audit 2022-04-22T20:24:34.091670+0000 mon.a (mon.0) 9375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[46934]: audit 2022-04-22T20:24:34.094547+0000 mon.a (mon.0) 9376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[46934]: audit 2022-04-22T20:24:34.096216+0000 mon.a (mon.0) 9377 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[45495]: cluster 2022-04-22T20:24:33.400461+0000 mgr.y (mgr.34107) 6109 : cluster [DBG] pgmap v7296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:34.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[45495]: audit 2022-04-22T20:24:33.887921+0000 mon.a (mon.0) 9374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[45495]: audit 2022-04-22T20:24:34.091670+0000 mon.a (mon.0) 9375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:24:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[45495]: audit 2022-04-22T20:24:34.094547+0000 mon.a (mon.0) 9376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:24:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:34 smithi115 bash[45495]: audit 2022-04-22T20:24:34.096216+0000 mon.a (mon.0) 9377 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:24:35.099 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:24:35.441 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:24:35.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:24:35.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:24:35.444 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:24:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:35 smithi121 bash[36902]: audit 2022-04-22T20:24:34.472923+0000 mon.a (mon.0) 9378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:24:35.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:35 smithi121 bash[36902]: audit 2022-04-22T20:24:35.074641+0000 mgr.y (mgr.34107) 6110 : audit [DBG] from='client.41742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:35 smithi115 bash[46934]: audit 2022-04-22T20:24:34.472923+0000 mon.a (mon.0) 9378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:24:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:35 smithi115 bash[46934]: audit 2022-04-22T20:24:35.074641+0000 mgr.y (mgr.34107) 6110 : audit [DBG] from='client.41742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:35 smithi115 bash[45495]: audit 2022-04-22T20:24:34.472923+0000 mon.a (mon.0) 9378 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:24:35.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:35 smithi115 bash[45495]: audit 2022-04-22T20:24:35.074641+0000 mgr.y (mgr.34107) 6110 : audit [DBG] from='client.41742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:35.844 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:24:35.844 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:24:35.844 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:24:35.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:24:35.846 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:24:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:36 smithi121 bash[36902]: cluster 2022-04-22T20:24:35.401550+0000 mgr.y (mgr.34107) 6111 : cluster [DBG] pgmap v7297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:36 smithi121 bash[36902]: audit 2022-04-22T20:24:35.414973+0000 mgr.y (mgr.34107) 6112 : audit [DBG] from='client.51682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:36 smithi121 bash[36902]: audit 2022-04-22T20:24:35.835344+0000 mon.c (mon.1) 241 : audit [DBG] from='client.? 172.21.15.115:0/4255991188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[45495]: cluster 2022-04-22T20:24:35.401550+0000 mgr.y (mgr.34107) 6111 : cluster [DBG] pgmap v7297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[45495]: audit 2022-04-22T20:24:35.414973+0000 mgr.y (mgr.34107) 6112 : audit [DBG] from='client.51682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[45495]: audit 2022-04-22T20:24:35.835344+0000 mon.c (mon.1) 241 : audit [DBG] from='client.? 172.21.15.115:0/4255991188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[46934]: cluster 2022-04-22T20:24:35.401550+0000 mgr.y (mgr.34107) 6111 : cluster [DBG] pgmap v7297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[46934]: audit 2022-04-22T20:24:35.414973+0000 mgr.y (mgr.34107) 6112 : audit [DBG] from='client.51682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:24:36.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:36 smithi115 bash[46934]: audit 2022-04-22T20:24:35.835344+0000 mon.c (mon.1) 241 : audit [DBG] from='client.? 172.21.15.115:0/4255991188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:24:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:37 smithi121 bash[36902]: audit 2022-04-22T20:24:37.295048+0000 mon.a (mon.0) 9379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:37 smithi121 bash[36902]: audit 2022-04-22T20:24:37.443675+0000 mon.a (mon.0) 9380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:37 smithi115 bash[45495]: audit 2022-04-22T20:24:37.295048+0000 mon.a (mon.0) 9379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:37.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:37 smithi115 bash[45495]: audit 2022-04-22T20:24:37.443675+0000 mon.a (mon.0) 9380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:37 smithi115 bash[46934]: audit 2022-04-22T20:24:37.295048+0000 mon.a (mon.0) 9379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:37 smithi115 bash[46934]: audit 2022-04-22T20:24:37.443675+0000 mon.a (mon.0) 9380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:38 smithi121 bash[36902]: cluster 2022-04-22T20:24:37.402620+0000 mgr.y (mgr.34107) 6113 : cluster [DBG] pgmap v7298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:38 smithi115 bash[46934]: cluster 2022-04-22T20:24:37.402620+0000 mgr.y (mgr.34107) 6113 : cluster [DBG] pgmap v7298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:38.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:38 smithi115 bash[45495]: cluster 2022-04-22T20:24:37.402620+0000 mgr.y (mgr.34107) 6113 : cluster [DBG] pgmap v7298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:39 smithi121 bash[36902]: audit 2022-04-22T20:24:38.891149+0000 mon.a (mon.0) 9381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:39 smithi115 bash[45495]: audit 2022-04-22T20:24:38.891149+0000 mon.a (mon.0) 9381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:39 smithi115 bash[46934]: audit 2022-04-22T20:24:38.891149+0000 mon.a (mon.0) 9381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:41 smithi121 bash[36902]: cluster 2022-04-22T20:24:39.403764+0000 mgr.y (mgr.34107) 6114 : cluster [DBG] pgmap v7299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:41 smithi115 bash[45495]: cluster 2022-04-22T20:24:39.403764+0000 mgr.y (mgr.34107) 6114 : cluster [DBG] pgmap v7299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:41.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:41 smithi115 bash[46934]: cluster 2022-04-22T20:24:39.403764+0000 mgr.y (mgr.34107) 6114 : cluster [DBG] pgmap v7299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:42.306Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:43 smithi121 bash[36902]: cluster 2022-04-22T20:24:41.404798+0000 mgr.y (mgr.34107) 6115 : cluster [DBG] pgmap v7300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:43 smithi121 bash[36902]: audit 2022-04-22T20:24:42.298734+0000 mon.a (mon.0) 9382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:43 smithi121 bash[36902]: audit 2022-04-22T20:24:42.446348+0000 mon.a (mon.0) 9383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[45495]: cluster 2022-04-22T20:24:41.404798+0000 mgr.y (mgr.34107) 6115 : cluster [DBG] pgmap v7300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[45495]: audit 2022-04-22T20:24:42.298734+0000 mon.a (mon.0) 9382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[45495]: audit 2022-04-22T20:24:42.446348+0000 mon.a (mon.0) 9383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[46934]: cluster 2022-04-22T20:24:41.404798+0000 mgr.y (mgr.34107) 6115 : cluster [DBG] pgmap v7300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[46934]: audit 2022-04-22T20:24:42.298734+0000 mon.a (mon.0) 9382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:43.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:43 smithi115 bash[46934]: audit 2022-04-22T20:24:42.446348+0000 mon.a (mon.0) 9383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:44.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:44 smithi115 bash[45495]: audit 2022-04-22T20:24:43.894173+0000 mon.a (mon.0) 9384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:44 smithi115 bash[46934]: audit 2022-04-22T20:24:43.894173+0000 mon.a (mon.0) 9384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:44.184Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:44 smithi121 bash[36902]: audit 2022-04-22T20:24:43.894173+0000 mon.a (mon.0) 9384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:45.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:45 smithi115 bash[45495]: cluster 2022-04-22T20:24:43.405908+0000 mgr.y (mgr.34107) 6116 : cluster [DBG] pgmap v7301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:45.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:45 smithi115 bash[46934]: cluster 2022-04-22T20:24:43.405908+0000 mgr.y (mgr.34107) 6116 : cluster [DBG] pgmap v7301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:45 smithi121 bash[36902]: cluster 2022-04-22T20:24:43.405908+0000 mgr.y (mgr.34107) 6116 : cluster [DBG] pgmap v7301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:47 smithi115 bash[45495]: cluster 2022-04-22T20:24:45.407028+0000 mgr.y (mgr.34107) 6117 : cluster [DBG] pgmap v7302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:47 smithi115 bash[46934]: cluster 2022-04-22T20:24:45.407028+0000 mgr.y (mgr.34107) 6117 : cluster [DBG] pgmap v7302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:47 smithi121 bash[36902]: cluster 2022-04-22T20:24:45.407028+0000 mgr.y (mgr.34107) 6117 : cluster [DBG] pgmap v7302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:48 smithi115 bash[45495]: audit 2022-04-22T20:24:47.302665+0000 mon.a (mon.0) 9385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:48 smithi115 bash[45495]: audit 2022-04-22T20:24:47.449480+0000 mon.a (mon.0) 9386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:48 smithi115 bash[46934]: audit 2022-04-22T20:24:47.302665+0000 mon.a (mon.0) 9385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:48 smithi115 bash[46934]: audit 2022-04-22T20:24:47.449480+0000 mon.a (mon.0) 9386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:48 smithi121 bash[36902]: audit 2022-04-22T20:24:47.302665+0000 mon.a (mon.0) 9385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:48 smithi121 bash[36902]: audit 2022-04-22T20:24:47.449480+0000 mon.a (mon.0) 9386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:49 smithi115 bash[45495]: cluster 2022-04-22T20:24:47.408187+0000 mgr.y (mgr.34107) 6118 : cluster [DBG] pgmap v7303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:49.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:49 smithi115 bash[45495]: audit 2022-04-22T20:24:48.897216+0000 mon.a (mon.0) 9387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:49.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:49 smithi115 bash[46934]: cluster 2022-04-22T20:24:47.408187+0000 mgr.y (mgr.34107) 6118 : cluster [DBG] pgmap v7303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:49.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:49 smithi115 bash[46934]: audit 2022-04-22T20:24:48.897216+0000 mon.a (mon.0) 9387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:49 smithi121 bash[36902]: cluster 2022-04-22T20:24:47.408187+0000 mgr.y (mgr.34107) 6118 : cluster [DBG] pgmap v7303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:49 smithi121 bash[36902]: audit 2022-04-22T20:24:48.897216+0000 mon.a (mon.0) 9387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:51.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:51 smithi115 bash[45495]: cluster 2022-04-22T20:24:49.409237+0000 mgr.y (mgr.34107) 6119 : cluster [DBG] pgmap v7304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:51.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:51 smithi115 bash[46934]: cluster 2022-04-22T20:24:49.409237+0000 mgr.y (mgr.34107) 6119 : cluster [DBG] pgmap v7304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:51 smithi121 bash[36902]: cluster 2022-04-22T20:24:49.409237+0000 mgr.y (mgr.34107) 6119 : cluster [DBG] pgmap v7304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:52.307Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:53 smithi121 bash[36902]: cluster 2022-04-22T20:24:51.410347+0000 mgr.y (mgr.34107) 6120 : cluster [DBG] pgmap v7305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:53.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:53 smithi121 bash[36902]: audit 2022-04-22T20:24:52.306547+0000 mon.a (mon.0) 9388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:53.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:53 smithi121 bash[36902]: audit 2022-04-22T20:24:52.452311+0000 mon.a (mon.0) 9389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[45495]: cluster 2022-04-22T20:24:51.410347+0000 mgr.y (mgr.34107) 6120 : cluster [DBG] pgmap v7305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[45495]: audit 2022-04-22T20:24:52.306547+0000 mon.a (mon.0) 9388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[45495]: audit 2022-04-22T20:24:52.452311+0000 mon.a (mon.0) 9389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[46934]: cluster 2022-04-22T20:24:51.410347+0000 mgr.y (mgr.34107) 6120 : cluster [DBG] pgmap v7305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[46934]: audit 2022-04-22T20:24:52.306547+0000 mon.a (mon.0) 9388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:53 smithi115 bash[46934]: audit 2022-04-22T20:24:52.452311+0000 mon.a (mon.0) 9389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:54 smithi121 bash[36902]: audit 2022-04-22T20:24:53.900161+0000 mon.a (mon.0) 9390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:54 smithi115 bash[45495]: audit 2022-04-22T20:24:53.900161+0000 mon.a (mon.0) 9390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:54 smithi115 bash[46934]: audit 2022-04-22T20:24:53.900161+0000 mon.a (mon.0) 9390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:24:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:24:54.185Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:24:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:55 smithi121 bash[36902]: cluster 2022-04-22T20:24:53.411870+0000 mgr.y (mgr.34107) 6121 : cluster [DBG] pgmap v7306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:55 smithi115 bash[45495]: cluster 2022-04-22T20:24:53.411870+0000 mgr.y (mgr.34107) 6121 : cluster [DBG] pgmap v7306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:55 smithi115 bash[46934]: cluster 2022-04-22T20:24:53.411870+0000 mgr.y (mgr.34107) 6121 : cluster [DBG] pgmap v7306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:57 smithi121 bash[36902]: cluster 2022-04-22T20:24:55.413077+0000 mgr.y (mgr.34107) 6122 : cluster [DBG] pgmap v7307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:57 smithi115 bash[45495]: cluster 2022-04-22T20:24:55.413077+0000 mgr.y (mgr.34107) 6122 : cluster [DBG] pgmap v7307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:57 smithi115 bash[46934]: cluster 2022-04-22T20:24:55.413077+0000 mgr.y (mgr.34107) 6122 : cluster [DBG] pgmap v7307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:58 smithi121 bash[36902]: audit 2022-04-22T20:24:57.310428+0000 mon.a (mon.0) 9391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:58 smithi121 bash[36902]: audit 2022-04-22T20:24:57.455172+0000 mon.a (mon.0) 9392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:58 smithi115 bash[45495]: audit 2022-04-22T20:24:57.310428+0000 mon.a (mon.0) 9391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:58 smithi115 bash[45495]: audit 2022-04-22T20:24:57.455172+0000 mon.a (mon.0) 9392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:58 smithi115 bash[46934]: audit 2022-04-22T20:24:57.310428+0000 mon.a (mon.0) 9391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:24:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:58 smithi115 bash[46934]: audit 2022-04-22T20:24:57.455172+0000 mon.a (mon.0) 9392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:24:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:59 smithi121 bash[36902]: cluster 2022-04-22T20:24:57.414110+0000 mgr.y (mgr.34107) 6123 : cluster [DBG] pgmap v7308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:24:59 smithi121 bash[36902]: audit 2022-04-22T20:24:58.903184+0000 mon.a (mon.0) 9393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:59 smithi115 bash[45495]: cluster 2022-04-22T20:24:57.414110+0000 mgr.y (mgr.34107) 6123 : cluster [DBG] pgmap v7308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:24:59 smithi115 bash[45495]: audit 2022-04-22T20:24:58.903184+0000 mon.a (mon.0) 9393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:24:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:59 smithi115 bash[46934]: cluster 2022-04-22T20:24:57.414110+0000 mgr.y (mgr.34107) 6123 : cluster [DBG] pgmap v7308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:24:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:24:59 smithi115 bash[46934]: audit 2022-04-22T20:24:58.903184+0000 mon.a (mon.0) 9393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:01 smithi121 bash[36902]: cluster 2022-04-22T20:24:59.415141+0000 mgr.y (mgr.34107) 6124 : cluster [DBG] pgmap v7309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:01 smithi115 bash[45495]: cluster 2022-04-22T20:24:59.415141+0000 mgr.y (mgr.34107) 6124 : cluster [DBG] pgmap v7309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:01.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:01 smithi115 bash[46934]: cluster 2022-04-22T20:24:59.415141+0000 mgr.y (mgr.34107) 6124 : cluster [DBG] pgmap v7309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:02.306Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:03 smithi121 bash[36902]: cluster 2022-04-22T20:25:01.416162+0000 mgr.y (mgr.34107) 6125 : cluster [DBG] pgmap v7310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:03 smithi121 bash[36902]: audit 2022-04-22T20:25:02.314170+0000 mon.a (mon.0) 9394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:03 smithi121 bash[36902]: audit 2022-04-22T20:25:02.457957+0000 mon.a (mon.0) 9395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[46934]: cluster 2022-04-22T20:25:01.416162+0000 mgr.y (mgr.34107) 6125 : cluster [DBG] pgmap v7310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:03.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[46934]: audit 2022-04-22T20:25:02.314170+0000 mon.a (mon.0) 9394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:03.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[46934]: audit 2022-04-22T20:25:02.457957+0000 mon.a (mon.0) 9395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:03.612 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[45495]: cluster 2022-04-22T20:25:01.416162+0000 mgr.y (mgr.34107) 6125 : cluster [DBG] pgmap v7310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:03.612 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[45495]: audit 2022-04-22T20:25:02.314170+0000 mon.a (mon.0) 9394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:03.612 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:03 smithi115 bash[45495]: audit 2022-04-22T20:25:02.457957+0000 mon.a (mon.0) 9395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:04 smithi121 bash[36902]: audit 2022-04-22T20:25:03.906127+0000 mon.a (mon.0) 9396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:04.185Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:04 smithi115 bash[45495]: audit 2022-04-22T20:25:03.906127+0000 mon.a (mon.0) 9396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:04 smithi115 bash[46934]: audit 2022-04-22T20:25:03.906127+0000 mon.a (mon.0) 9396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:05 smithi121 bash[36902]: cluster 2022-04-22T20:25:03.417313+0000 mgr.y (mgr.34107) 6126 : cluster [DBG] pgmap v7311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:05 smithi115 bash[45495]: cluster 2022-04-22T20:25:03.417313+0000 mgr.y (mgr.34107) 6126 : cluster [DBG] pgmap v7311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:05.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:05 smithi115 bash[46934]: cluster 2022-04-22T20:25:03.417313+0000 mgr.y (mgr.34107) 6126 : cluster [DBG] pgmap v7311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:06.235 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:25:06.577 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:25:06.577 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:25:06.577 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:25:06.577 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:25:06.578 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:25:06.578 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:25:06.578 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 2m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:25:06.581 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 2m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 3m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 2m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 2m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 2m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 2m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 2m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 3m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:25:06.582 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 3m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:25:06.583 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 3m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:25:06.583 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 3m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:25:06.583 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:25:07.001 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:25:07.002 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:25:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:07 smithi121 bash[36902]: cluster 2022-04-22T20:25:05.418366+0000 mgr.y (mgr.34107) 6127 : cluster [DBG] pgmap v7312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:07 smithi121 bash[36902]: audit 2022-04-22T20:25:06.992505+0000 mon.c (mon.1) 242 : audit [DBG] from='client.? 172.21.15.115:0/2878660141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:07 smithi115 bash[45495]: cluster 2022-04-22T20:25:05.418366+0000 mgr.y (mgr.34107) 6127 : cluster [DBG] pgmap v7312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:07.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:07 smithi115 bash[45495]: audit 2022-04-22T20:25:06.992505+0000 mon.c (mon.1) 242 : audit [DBG] from='client.? 172.21.15.115:0/2878660141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:07 smithi115 bash[46934]: cluster 2022-04-22T20:25:05.418366+0000 mgr.y (mgr.34107) 6127 : cluster [DBG] pgmap v7312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:07.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:07 smithi115 bash[46934]: audit 2022-04-22T20:25:06.992505+0000 mon.c (mon.1) 242 : audit [DBG] from='client.? 172.21.15.115:0/2878660141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:08 smithi121 bash[36902]: audit 2022-04-22T20:25:06.211123+0000 mgr.y (mgr.34107) 6128 : audit [DBG] from='client.41760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:08 smithi121 bash[36902]: audit 2022-04-22T20:25:06.549556+0000 mgr.y (mgr.34107) 6129 : audit [DBG] from='client.51697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:08 smithi121 bash[36902]: audit 2022-04-22T20:25:07.317454+0000 mon.a (mon.0) 9397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:08 smithi121 bash[36902]: audit 2022-04-22T20:25:07.460679+0000 mon.a (mon.0) 9398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[45495]: audit 2022-04-22T20:25:06.211123+0000 mgr.y (mgr.34107) 6128 : audit [DBG] from='client.41760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[45495]: audit 2022-04-22T20:25:06.549556+0000 mgr.y (mgr.34107) 6129 : audit [DBG] from='client.51697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[45495]: audit 2022-04-22T20:25:07.317454+0000 mon.a (mon.0) 9397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[45495]: audit 2022-04-22T20:25:07.460679+0000 mon.a (mon.0) 9398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[46934]: audit 2022-04-22T20:25:06.211123+0000 mgr.y (mgr.34107) 6128 : audit [DBG] from='client.41760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[46934]: audit 2022-04-22T20:25:06.549556+0000 mgr.y (mgr.34107) 6129 : audit [DBG] from='client.51697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[46934]: audit 2022-04-22T20:25:07.317454+0000 mon.a (mon.0) 9397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:08 smithi115 bash[46934]: audit 2022-04-22T20:25:07.460679+0000 mon.a (mon.0) 9398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:09 smithi121 bash[36902]: cluster 2022-04-22T20:25:07.419409+0000 mgr.y (mgr.34107) 6130 : cluster [DBG] pgmap v7313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:09 smithi121 bash[36902]: audit 2022-04-22T20:25:08.909200+0000 mon.a (mon.0) 9399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:09 smithi115 bash[46934]: cluster 2022-04-22T20:25:07.419409+0000 mgr.y (mgr.34107) 6130 : cluster [DBG] pgmap v7313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:09 smithi115 bash[46934]: audit 2022-04-22T20:25:08.909200+0000 mon.a (mon.0) 9399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:09 smithi115 bash[45495]: cluster 2022-04-22T20:25:07.419409+0000 mgr.y (mgr.34107) 6130 : cluster [DBG] pgmap v7313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:09 smithi115 bash[45495]: audit 2022-04-22T20:25:08.909200+0000 mon.a (mon.0) 9399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:11 smithi121 bash[36902]: cluster 2022-04-22T20:25:09.420246+0000 mgr.y (mgr.34107) 6131 : cluster [DBG] pgmap v7314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:11 smithi115 bash[45495]: cluster 2022-04-22T20:25:09.420246+0000 mgr.y (mgr.34107) 6131 : cluster [DBG] pgmap v7314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:11 smithi115 bash[46934]: cluster 2022-04-22T20:25:09.420246+0000 mgr.y (mgr.34107) 6131 : cluster [DBG] pgmap v7314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:12.308Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:13 smithi121 bash[36902]: cluster 2022-04-22T20:25:11.421269+0000 mgr.y (mgr.34107) 6132 : cluster [DBG] pgmap v7315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:13 smithi121 bash[36902]: audit 2022-04-22T20:25:12.321643+0000 mon.a (mon.0) 9400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:13 smithi121 bash[36902]: audit 2022-04-22T20:25:12.463356+0000 mon.a (mon.0) 9401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[46934]: cluster 2022-04-22T20:25:11.421269+0000 mgr.y (mgr.34107) 6132 : cluster [DBG] pgmap v7315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[46934]: audit 2022-04-22T20:25:12.321643+0000 mon.a (mon.0) 9400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[46934]: audit 2022-04-22T20:25:12.463356+0000 mon.a (mon.0) 9401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[45495]: cluster 2022-04-22T20:25:11.421269+0000 mgr.y (mgr.34107) 6132 : cluster [DBG] pgmap v7315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[45495]: audit 2022-04-22T20:25:12.321643+0000 mon.a (mon.0) 9400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:13.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:13 smithi115 bash[45495]: audit 2022-04-22T20:25:12.463356+0000 mon.a (mon.0) 9401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:14 smithi121 bash[36902]: audit 2022-04-22T20:25:13.912473+0000 mon.a (mon.0) 9402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:14 smithi115 bash[46934]: audit 2022-04-22T20:25:13.912473+0000 mon.a (mon.0) 9402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:14.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:14 smithi115 bash[45495]: audit 2022-04-22T20:25:13.912473+0000 mon.a (mon.0) 9402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:14.187Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:15 smithi121 bash[36902]: cluster 2022-04-22T20:25:13.422301+0000 mgr.y (mgr.34107) 6133 : cluster [DBG] pgmap v7316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:15.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:15 smithi115 bash[45495]: cluster 2022-04-22T20:25:13.422301+0000 mgr.y (mgr.34107) 6133 : cluster [DBG] pgmap v7316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:15 smithi115 bash[46934]: cluster 2022-04-22T20:25:13.422301+0000 mgr.y (mgr.34107) 6133 : cluster [DBG] pgmap v7316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:16 smithi121 bash[36902]: audit 2022-04-22T20:25:15.334271+0000 mon.a (mon.0) 9403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:25:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:16 smithi121 bash[36902]: audit 2022-04-22T20:25:15.334759+0000 mon.a (mon.0) 9404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:25:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:16 smithi115 bash[45495]: audit 2022-04-22T20:25:15.334271+0000 mon.a (mon.0) 9403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:25:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:16 smithi115 bash[45495]: audit 2022-04-22T20:25:15.334759+0000 mon.a (mon.0) 9404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:25:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:16 smithi115 bash[46934]: audit 2022-04-22T20:25:15.334271+0000 mon.a (mon.0) 9403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:25:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:16 smithi115 bash[46934]: audit 2022-04-22T20:25:15.334759+0000 mon.a (mon.0) 9404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:25:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:17 smithi121 bash[36902]: cluster 2022-04-22T20:25:15.423290+0000 mgr.y (mgr.34107) 6134 : cluster [DBG] pgmap v7317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:17 smithi115 bash[45495]: cluster 2022-04-22T20:25:15.423290+0000 mgr.y (mgr.34107) 6134 : cluster [DBG] pgmap v7317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:17 smithi115 bash[46934]: cluster 2022-04-22T20:25:15.423290+0000 mgr.y (mgr.34107) 6134 : cluster [DBG] pgmap v7317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:18 smithi121 bash[36902]: audit 2022-04-22T20:25:17.325218+0000 mon.a (mon.0) 9405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:18 smithi121 bash[36902]: audit 2022-04-22T20:25:17.466583+0000 mon.a (mon.0) 9406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:18 smithi115 bash[45495]: audit 2022-04-22T20:25:17.325218+0000 mon.a (mon.0) 9405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:18 smithi115 bash[45495]: audit 2022-04-22T20:25:17.466583+0000 mon.a (mon.0) 9406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:18 smithi115 bash[46934]: audit 2022-04-22T20:25:17.325218+0000 mon.a (mon.0) 9405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:18 smithi115 bash[46934]: audit 2022-04-22T20:25:17.466583+0000 mon.a (mon.0) 9406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:19 smithi121 bash[36902]: cluster 2022-04-22T20:25:17.424180+0000 mgr.y (mgr.34107) 6135 : cluster [DBG] pgmap v7318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:19 smithi121 bash[36902]: audit 2022-04-22T20:25:18.915623+0000 mon.a (mon.0) 9407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:19 smithi115 bash[45495]: cluster 2022-04-22T20:25:17.424180+0000 mgr.y (mgr.34107) 6135 : cluster [DBG] pgmap v7318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:19 smithi115 bash[45495]: audit 2022-04-22T20:25:18.915623+0000 mon.a (mon.0) 9407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:19 smithi115 bash[46934]: cluster 2022-04-22T20:25:17.424180+0000 mgr.y (mgr.34107) 6135 : cluster [DBG] pgmap v7318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:19 smithi115 bash[46934]: audit 2022-04-22T20:25:18.915623+0000 mon.a (mon.0) 9407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:21 smithi121 bash[36902]: cluster 2022-04-22T20:25:19.425289+0000 mgr.y (mgr.34107) 6136 : cluster [DBG] pgmap v7319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:21 smithi115 bash[46934]: cluster 2022-04-22T20:25:19.425289+0000 mgr.y (mgr.34107) 6136 : cluster [DBG] pgmap v7319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:21 smithi115 bash[45495]: cluster 2022-04-22T20:25:19.425289+0000 mgr.y (mgr.34107) 6136 : cluster [DBG] pgmap v7319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:22.309Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:23 smithi121 bash[36902]: cluster 2022-04-22T20:25:21.426155+0000 mgr.y (mgr.34107) 6137 : cluster [DBG] pgmap v7320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:23 smithi121 bash[36902]: audit 2022-04-22T20:25:22.328870+0000 mon.a (mon.0) 9408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:23 smithi121 bash[36902]: audit 2022-04-22T20:25:22.469387+0000 mon.a (mon.0) 9409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[45495]: cluster 2022-04-22T20:25:21.426155+0000 mgr.y (mgr.34107) 6137 : cluster [DBG] pgmap v7320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[45495]: audit 2022-04-22T20:25:22.328870+0000 mon.a (mon.0) 9408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[45495]: audit 2022-04-22T20:25:22.469387+0000 mon.a (mon.0) 9409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[46934]: cluster 2022-04-22T20:25:21.426155+0000 mgr.y (mgr.34107) 6137 : cluster [DBG] pgmap v7320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[46934]: audit 2022-04-22T20:25:22.328870+0000 mon.a (mon.0) 9408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:23 smithi115 bash[46934]: audit 2022-04-22T20:25:22.469387+0000 mon.a (mon.0) 9409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:24 smithi121 bash[36902]: audit 2022-04-22T20:25:23.918736+0000 mon.a (mon.0) 9410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:24.581 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:24 smithi115 bash[45495]: audit 2022-04-22T20:25:23.918736+0000 mon.a (mon.0) 9410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:24.582 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:24.188Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:24.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:24 smithi115 bash[46934]: audit 2022-04-22T20:25:23.918736+0000 mon.a (mon.0) 9410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:25 smithi121 bash[36902]: cluster 2022-04-22T20:25:23.427252+0000 mgr.y (mgr.34107) 6138 : cluster [DBG] pgmap v7321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:25 smithi115 bash[45495]: cluster 2022-04-22T20:25:23.427252+0000 mgr.y (mgr.34107) 6138 : cluster [DBG] pgmap v7321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:25 smithi115 bash[46934]: cluster 2022-04-22T20:25:23.427252+0000 mgr.y (mgr.34107) 6138 : cluster [DBG] pgmap v7321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:27.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:27 smithi121 bash[36902]: cluster 2022-04-22T20:25:25.428310+0000 mgr.y (mgr.34107) 6139 : cluster [DBG] pgmap v7322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:27 smithi115 bash[45495]: cluster 2022-04-22T20:25:25.428310+0000 mgr.y (mgr.34107) 6139 : cluster [DBG] pgmap v7322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:27.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:27 smithi115 bash[46934]: cluster 2022-04-22T20:25:25.428310+0000 mgr.y (mgr.34107) 6139 : cluster [DBG] pgmap v7322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:28 smithi121 bash[36902]: audit 2022-04-22T20:25:27.332241+0000 mon.a (mon.0) 9411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:28 smithi121 bash[36902]: cluster 2022-04-22T20:25:27.429381+0000 mgr.y (mgr.34107) 6140 : cluster [DBG] pgmap v7323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:28 smithi121 bash[36902]: audit 2022-04-22T20:25:27.471982+0000 mon.a (mon.0) 9412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[45495]: audit 2022-04-22T20:25:27.332241+0000 mon.a (mon.0) 9411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[45495]: cluster 2022-04-22T20:25:27.429381+0000 mgr.y (mgr.34107) 6140 : cluster [DBG] pgmap v7323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[45495]: audit 2022-04-22T20:25:27.471982+0000 mon.a (mon.0) 9412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[46934]: audit 2022-04-22T20:25:27.332241+0000 mon.a (mon.0) 9411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[46934]: cluster 2022-04-22T20:25:27.429381+0000 mgr.y (mgr.34107) 6140 : cluster [DBG] pgmap v7323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:28 smithi115 bash[46934]: audit 2022-04-22T20:25:27.471982+0000 mon.a (mon.0) 9412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:29 smithi121 bash[36902]: audit 2022-04-22T20:25:28.922071+0000 mon.a (mon.0) 9413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:29 smithi115 bash[46934]: audit 2022-04-22T20:25:28.922071+0000 mon.a (mon.0) 9413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:29.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:29 smithi115 bash[45495]: audit 2022-04-22T20:25:28.922071+0000 mon.a (mon.0) 9413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:30 smithi121 bash[36902]: cluster 2022-04-22T20:25:29.430365+0000 mgr.y (mgr.34107) 6141 : cluster [DBG] pgmap v7324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:30 smithi115 bash[45495]: cluster 2022-04-22T20:25:29.430365+0000 mgr.y (mgr.34107) 6141 : cluster [DBG] pgmap v7324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:30 smithi115 bash[46934]: cluster 2022-04-22T20:25:29.430365+0000 mgr.y (mgr.34107) 6141 : cluster [DBG] pgmap v7324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:32 smithi121 bash[36902]: cluster 2022-04-22T20:25:31.431391+0000 mgr.y (mgr.34107) 6142 : cluster [DBG] pgmap v7325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:32 smithi121 bash[36902]: audit 2022-04-22T20:25:32.335876+0000 mon.a (mon.0) 9414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:32 smithi121 bash[36902]: audit 2022-04-22T20:25:32.474407+0000 mon.a (mon.0) 9415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:32.308Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[45495]: cluster 2022-04-22T20:25:31.431391+0000 mgr.y (mgr.34107) 6142 : cluster [DBG] pgmap v7325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[45495]: audit 2022-04-22T20:25:32.335876+0000 mon.a (mon.0) 9414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:32.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[45495]: audit 2022-04-22T20:25:32.474407+0000 mon.a (mon.0) 9415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[46934]: cluster 2022-04-22T20:25:31.431391+0000 mgr.y (mgr.34107) 6142 : cluster [DBG] pgmap v7325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[46934]: audit 2022-04-22T20:25:32.335876+0000 mon.a (mon.0) 9414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:32.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:32 smithi115 bash[46934]: audit 2022-04-22T20:25:32.474407+0000 mon.a (mon.0) 9415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:34.500 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:34.188Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:34 smithi121 bash[36902]: cluster 2022-04-22T20:25:33.432472+0000 mgr.y (mgr.34107) 6143 : cluster [DBG] pgmap v7326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:34 smithi121 bash[36902]: audit 2022-04-22T20:25:33.925232+0000 mon.a (mon.0) 9416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:34 smithi121 bash[36902]: audit 2022-04-22T20:25:34.480288+0000 mon.a (mon.0) 9417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:25:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:34 smithi121 bash[36902]: audit 2022-04-22T20:25:34.483364+0000 mon.a (mon.0) 9418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:25:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:34 smithi121 bash[36902]: audit 2022-04-22T20:25:34.485075+0000 mon.a (mon.0) 9419 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[46934]: cluster 2022-04-22T20:25:33.432472+0000 mgr.y (mgr.34107) 6143 : cluster [DBG] pgmap v7326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[46934]: audit 2022-04-22T20:25:33.925232+0000 mon.a (mon.0) 9416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:34.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[46934]: audit 2022-04-22T20:25:34.480288+0000 mon.a (mon.0) 9417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[46934]: audit 2022-04-22T20:25:34.483364+0000 mon.a (mon.0) 9418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[46934]: audit 2022-04-22T20:25:34.485075+0000 mon.a (mon.0) 9419 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[45495]: cluster 2022-04-22T20:25:33.432472+0000 mgr.y (mgr.34107) 6143 : cluster [DBG] pgmap v7326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[45495]: audit 2022-04-22T20:25:33.925232+0000 mon.a (mon.0) 9416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[45495]: audit 2022-04-22T20:25:34.480288+0000 mon.a (mon.0) 9417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[45495]: audit 2022-04-22T20:25:34.483364+0000 mon.a (mon.0) 9418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:25:34.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:34 smithi115 bash[45495]: audit 2022-04-22T20:25:34.485075+0000 mon.a (mon.0) 9419 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:25:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:35 smithi121 bash[36902]: audit 2022-04-22T20:25:34.845600+0000 mon.a (mon.0) 9420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:25:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:35 smithi115 bash[45495]: audit 2022-04-22T20:25:34.845600+0000 mon.a (mon.0) 9420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:25:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:35 smithi115 bash[46934]: audit 2022-04-22T20:25:34.845600+0000 mon.a (mon.0) 9420 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:25:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:36 smithi121 bash[36902]: cluster 2022-04-22T20:25:35.433640+0000 mgr.y (mgr.34107) 6144 : cluster [DBG] pgmap v7327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:36 smithi115 bash[45495]: cluster 2022-04-22T20:25:35.433640+0000 mgr.y (mgr.34107) 6144 : cluster [DBG] pgmap v7327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:36 smithi115 bash[46934]: cluster 2022-04-22T20:25:35.433640+0000 mgr.y (mgr.34107) 6144 : cluster [DBG] pgmap v7327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:37.382 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:25:37.722 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:25:37.722 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:25:37.723 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:25:37.724 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:25:38.126 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:25:38.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:25:38.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:25:38.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:25:38.128 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:25:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:37 smithi121 bash[36902]: audit 2022-04-22T20:25:37.339686+0000 mon.a (mon.0) 9421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:38.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:37 smithi121 bash[36902]: audit 2022-04-22T20:25:37.476942+0000 mon.a (mon.0) 9422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:37 smithi115 bash[45495]: audit 2022-04-22T20:25:37.339686+0000 mon.a (mon.0) 9421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:38.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:37 smithi115 bash[45495]: audit 2022-04-22T20:25:37.476942+0000 mon.a (mon.0) 9422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:37 smithi115 bash[46934]: audit 2022-04-22T20:25:37.339686+0000 mon.a (mon.0) 9421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:38.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:37 smithi115 bash[46934]: audit 2022-04-22T20:25:37.476942+0000 mon.a (mon.0) 9422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:38 smithi121 bash[36902]: audit 2022-04-22T20:25:37.358268+0000 mgr.y (mgr.34107) 6145 : audit [DBG] from='client.51709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:38 smithi121 bash[36902]: cluster 2022-04-22T20:25:37.434650+0000 mgr.y (mgr.34107) 6146 : cluster [DBG] pgmap v7328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:38 smithi121 bash[36902]: audit 2022-04-22T20:25:37.695645+0000 mgr.y (mgr.34107) 6147 : audit [DBG] from='client.51715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:38 smithi121 bash[36902]: audit 2022-04-22T20:25:38.117176+0000 mon.a (mon.0) 9423 : audit [DBG] from='client.? 172.21.15.115:0/818734078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[45495]: audit 2022-04-22T20:25:37.358268+0000 mgr.y (mgr.34107) 6145 : audit [DBG] from='client.51709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[45495]: cluster 2022-04-22T20:25:37.434650+0000 mgr.y (mgr.34107) 6146 : cluster [DBG] pgmap v7328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[45495]: audit 2022-04-22T20:25:37.695645+0000 mgr.y (mgr.34107) 6147 : audit [DBG] from='client.51715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[45495]: audit 2022-04-22T20:25:38.117176+0000 mon.a (mon.0) 9423 : audit [DBG] from='client.? 172.21.15.115:0/818734078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[46934]: audit 2022-04-22T20:25:37.358268+0000 mgr.y (mgr.34107) 6145 : audit [DBG] from='client.51709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[46934]: cluster 2022-04-22T20:25:37.434650+0000 mgr.y (mgr.34107) 6146 : cluster [DBG] pgmap v7328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[46934]: audit 2022-04-22T20:25:37.695645+0000 mgr.y (mgr.34107) 6147 : audit [DBG] from='client.51715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:25:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:38 smithi115 bash[46934]: audit 2022-04-22T20:25:38.117176+0000 mon.a (mon.0) 9423 : audit [DBG] from='client.? 172.21.15.115:0/818734078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:25:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:39 smithi121 bash[36902]: audit 2022-04-22T20:25:38.928231+0000 mon.a (mon.0) 9424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:39 smithi115 bash[45495]: audit 2022-04-22T20:25:38.928231+0000 mon.a (mon.0) 9424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:39 smithi115 bash[46934]: audit 2022-04-22T20:25:38.928231+0000 mon.a (mon.0) 9424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:41.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:41 smithi115 bash[45495]: cluster 2022-04-22T20:25:39.435876+0000 mgr.y (mgr.34107) 6148 : cluster [DBG] pgmap v7329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:41.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:41 smithi115 bash[46934]: cluster 2022-04-22T20:25:39.435876+0000 mgr.y (mgr.34107) 6148 : cluster [DBG] pgmap v7329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:41 smithi121 bash[36902]: cluster 2022-04-22T20:25:39.435876+0000 mgr.y (mgr.34107) 6148 : cluster [DBG] pgmap v7329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:42.308Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[46934]: cluster 2022-04-22T20:25:41.437032+0000 mgr.y (mgr.34107) 6149 : cluster [DBG] pgmap v7330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[46934]: audit 2022-04-22T20:25:42.343181+0000 mon.a (mon.0) 9425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[46934]: audit 2022-04-22T20:25:42.479514+0000 mon.a (mon.0) 9426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[45495]: cluster 2022-04-22T20:25:41.437032+0000 mgr.y (mgr.34107) 6149 : cluster [DBG] pgmap v7330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[45495]: audit 2022-04-22T20:25:42.343181+0000 mon.a (mon.0) 9425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:43.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:43 smithi115 bash[45495]: audit 2022-04-22T20:25:42.479514+0000 mon.a (mon.0) 9426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:43 smithi121 bash[36902]: cluster 2022-04-22T20:25:41.437032+0000 mgr.y (mgr.34107) 6149 : cluster [DBG] pgmap v7330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:43 smithi121 bash[36902]: audit 2022-04-22T20:25:42.343181+0000 mon.a (mon.0) 9425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:43 smithi121 bash[36902]: audit 2022-04-22T20:25:42.479514+0000 mon.a (mon.0) 9426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:44.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:44 smithi115 bash[45495]: audit 2022-04-22T20:25:43.931274+0000 mon.a (mon.0) 9427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:44.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:44.188Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:44.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:44 smithi115 bash[46934]: audit 2022-04-22T20:25:43.931274+0000 mon.a (mon.0) 9427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:44 smithi121 bash[36902]: audit 2022-04-22T20:25:43.931274+0000 mon.a (mon.0) 9427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:45.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:45 smithi115 bash[45495]: cluster 2022-04-22T20:25:43.438183+0000 mgr.y (mgr.34107) 6150 : cluster [DBG] pgmap v7331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:45.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:45 smithi115 bash[46934]: cluster 2022-04-22T20:25:43.438183+0000 mgr.y (mgr.34107) 6150 : cluster [DBG] pgmap v7331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:45 smithi121 bash[36902]: cluster 2022-04-22T20:25:43.438183+0000 mgr.y (mgr.34107) 6150 : cluster [DBG] pgmap v7331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:47 smithi115 bash[45495]: cluster 2022-04-22T20:25:45.439298+0000 mgr.y (mgr.34107) 6151 : cluster [DBG] pgmap v7332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:47.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:47 smithi115 bash[46934]: cluster 2022-04-22T20:25:45.439298+0000 mgr.y (mgr.34107) 6151 : cluster [DBG] pgmap v7332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:47 smithi121 bash[36902]: cluster 2022-04-22T20:25:45.439298+0000 mgr.y (mgr.34107) 6151 : cluster [DBG] pgmap v7332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:48 smithi121 bash[36902]: audit 2022-04-22T20:25:47.346969+0000 mon.a (mon.0) 9428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:48 smithi121 bash[36902]: audit 2022-04-22T20:25:47.482066+0000 mon.a (mon.0) 9429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:48 smithi115 bash[45495]: audit 2022-04-22T20:25:47.346969+0000 mon.a (mon.0) 9428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:48 smithi115 bash[45495]: audit 2022-04-22T20:25:47.482066+0000 mon.a (mon.0) 9429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:48 smithi115 bash[46934]: audit 2022-04-22T20:25:47.346969+0000 mon.a (mon.0) 9428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:48 smithi115 bash[46934]: audit 2022-04-22T20:25:47.482066+0000 mon.a (mon.0) 9429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:49 smithi121 bash[36902]: cluster 2022-04-22T20:25:47.440367+0000 mgr.y (mgr.34107) 6152 : cluster [DBG] pgmap v7333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:49 smithi121 bash[36902]: audit 2022-04-22T20:25:48.934610+0000 mon.a (mon.0) 9430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:49 smithi115 bash[46934]: cluster 2022-04-22T20:25:47.440367+0000 mgr.y (mgr.34107) 6152 : cluster [DBG] pgmap v7333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:49 smithi115 bash[46934]: audit 2022-04-22T20:25:48.934610+0000 mon.a (mon.0) 9430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:49 smithi115 bash[45495]: cluster 2022-04-22T20:25:47.440367+0000 mgr.y (mgr.34107) 6152 : cluster [DBG] pgmap v7333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:49.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:49 smithi115 bash[45495]: audit 2022-04-22T20:25:48.934610+0000 mon.a (mon.0) 9430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:51 smithi121 bash[36902]: cluster 2022-04-22T20:25:49.441367+0000 mgr.y (mgr.34107) 6153 : cluster [DBG] pgmap v7334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:51 smithi115 bash[45495]: cluster 2022-04-22T20:25:49.441367+0000 mgr.y (mgr.34107) 6153 : cluster [DBG] pgmap v7334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:51 smithi115 bash[46934]: cluster 2022-04-22T20:25:49.441367+0000 mgr.y (mgr.34107) 6153 : cluster [DBG] pgmap v7334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:52.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:52.309Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:53 smithi121 bash[36902]: cluster 2022-04-22T20:25:51.442343+0000 mgr.y (mgr.34107) 6154 : cluster [DBG] pgmap v7335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:53 smithi121 bash[36902]: audit 2022-04-22T20:25:52.350501+0000 mon.a (mon.0) 9431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:53 smithi121 bash[36902]: audit 2022-04-22T20:25:52.484410+0000 mon.a (mon.0) 9432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[45495]: cluster 2022-04-22T20:25:51.442343+0000 mgr.y (mgr.34107) 6154 : cluster [DBG] pgmap v7335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[45495]: audit 2022-04-22T20:25:52.350501+0000 mon.a (mon.0) 9431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[45495]: audit 2022-04-22T20:25:52.484410+0000 mon.a (mon.0) 9432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[46934]: cluster 2022-04-22T20:25:51.442343+0000 mgr.y (mgr.34107) 6154 : cluster [DBG] pgmap v7335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[46934]: audit 2022-04-22T20:25:52.350501+0000 mon.a (mon.0) 9431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:53 smithi115 bash[46934]: audit 2022-04-22T20:25:52.484410+0000 mon.a (mon.0) 9432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:54 smithi121 bash[36902]: audit 2022-04-22T20:25:53.937747+0000 mon.a (mon.0) 9433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:54 smithi115 bash[45495]: audit 2022-04-22T20:25:53.937747+0000 mon.a (mon.0) 9433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:25:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:25:54.188Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:25:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:54 smithi115 bash[46934]: audit 2022-04-22T20:25:53.937747+0000 mon.a (mon.0) 9433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:55.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:55 smithi121 bash[36902]: cluster 2022-04-22T20:25:53.443574+0000 mgr.y (mgr.34107) 6155 : cluster [DBG] pgmap v7336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:55 smithi115 bash[46934]: cluster 2022-04-22T20:25:53.443574+0000 mgr.y (mgr.34107) 6155 : cluster [DBG] pgmap v7336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:55 smithi115 bash[45495]: cluster 2022-04-22T20:25:53.443574+0000 mgr.y (mgr.34107) 6155 : cluster [DBG] pgmap v7336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:57 smithi121 bash[36902]: cluster 2022-04-22T20:25:55.444559+0000 mgr.y (mgr.34107) 6156 : cluster [DBG] pgmap v7337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:57 smithi115 bash[46934]: cluster 2022-04-22T20:25:55.444559+0000 mgr.y (mgr.34107) 6156 : cluster [DBG] pgmap v7337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:57 smithi115 bash[45495]: cluster 2022-04-22T20:25:55.444559+0000 mgr.y (mgr.34107) 6156 : cluster [DBG] pgmap v7337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:58 smithi121 bash[36902]: audit 2022-04-22T20:25:57.354230+0000 mon.a (mon.0) 9434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:58 smithi121 bash[36902]: audit 2022-04-22T20:25:57.486830+0000 mon.a (mon.0) 9435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:58 smithi115 bash[45495]: audit 2022-04-22T20:25:57.354230+0000 mon.a (mon.0) 9434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:58.583 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:58 smithi115 bash[45495]: audit 2022-04-22T20:25:57.486830+0000 mon.a (mon.0) 9435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:58.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:58 smithi115 bash[46934]: audit 2022-04-22T20:25:57.354230+0000 mon.a (mon.0) 9434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:25:58.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:58 smithi115 bash[46934]: audit 2022-04-22T20:25:57.486830+0000 mon.a (mon.0) 9435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:25:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:59 smithi121 bash[36902]: cluster 2022-04-22T20:25:57.445575+0000 mgr.y (mgr.34107) 6157 : cluster [DBG] pgmap v7338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:25:59 smithi121 bash[36902]: audit 2022-04-22T20:25:58.940803+0000 mon.a (mon.0) 9436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:59 smithi115 bash[46934]: cluster 2022-04-22T20:25:57.445575+0000 mgr.y (mgr.34107) 6157 : cluster [DBG] pgmap v7338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:25:59 smithi115 bash[46934]: audit 2022-04-22T20:25:58.940803+0000 mon.a (mon.0) 9436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:25:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:59 smithi115 bash[45495]: cluster 2022-04-22T20:25:57.445575+0000 mgr.y (mgr.34107) 6157 : cluster [DBG] pgmap v7338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:25:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:25:59 smithi115 bash[45495]: audit 2022-04-22T20:25:58.940803+0000 mon.a (mon.0) 9436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:01 smithi121 bash[36902]: cluster 2022-04-22T20:25:59.446667+0000 mgr.y (mgr.34107) 6158 : cluster [DBG] pgmap v7339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:01 smithi115 bash[45495]: cluster 2022-04-22T20:25:59.446667+0000 mgr.y (mgr.34107) 6158 : cluster [DBG] pgmap v7339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:01 smithi115 bash[46934]: cluster 2022-04-22T20:25:59.446667+0000 mgr.y (mgr.34107) 6158 : cluster [DBG] pgmap v7339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:02.310Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:03 smithi121 bash[36902]: cluster 2022-04-22T20:26:01.447787+0000 mgr.y (mgr.34107) 6159 : cluster [DBG] pgmap v7340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:03 smithi121 bash[36902]: audit 2022-04-22T20:26:02.357935+0000 mon.a (mon.0) 9437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:03 smithi121 bash[36902]: audit 2022-04-22T20:26:02.489557+0000 mon.a (mon.0) 9438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[45495]: cluster 2022-04-22T20:26:01.447787+0000 mgr.y (mgr.34107) 6159 : cluster [DBG] pgmap v7340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[45495]: audit 2022-04-22T20:26:02.357935+0000 mon.a (mon.0) 9437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[45495]: audit 2022-04-22T20:26:02.489557+0000 mon.a (mon.0) 9438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[46934]: cluster 2022-04-22T20:26:01.447787+0000 mgr.y (mgr.34107) 6159 : cluster [DBG] pgmap v7340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[46934]: audit 2022-04-22T20:26:02.357935+0000 mon.a (mon.0) 9437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:03 smithi115 bash[46934]: audit 2022-04-22T20:26:02.489557+0000 mon.a (mon.0) 9438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:04 smithi121 bash[36902]: audit 2022-04-22T20:26:03.943730+0000 mon.a (mon.0) 9439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:04.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:04 smithi115 bash[45495]: audit 2022-04-22T20:26:03.943730+0000 mon.a (mon.0) 9439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:04.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:04.189Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:04.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:04 smithi115 bash[46934]: audit 2022-04-22T20:26:03.943730+0000 mon.a (mon.0) 9439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:05 smithi121 bash[36902]: cluster 2022-04-22T20:26:03.448898+0000 mgr.y (mgr.34107) 6160 : cluster [DBG] pgmap v7341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:05 smithi115 bash[45495]: cluster 2022-04-22T20:26:03.448898+0000 mgr.y (mgr.34107) 6160 : cluster [DBG] pgmap v7341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:05 smithi115 bash[46934]: cluster 2022-04-22T20:26:03.448898+0000 mgr.y (mgr.34107) 6160 : cluster [DBG] pgmap v7341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:07 smithi121 bash[36902]: cluster 2022-04-22T20:26:05.449984+0000 mgr.y (mgr.34107) 6161 : cluster [DBG] pgmap v7342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:07 smithi115 bash[45495]: cluster 2022-04-22T20:26:05.449984+0000 mgr.y (mgr.34107) 6161 : cluster [DBG] pgmap v7342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:07 smithi115 bash[46934]: cluster 2022-04-22T20:26:05.449984+0000 mgr.y (mgr.34107) 6161 : cluster [DBG] pgmap v7342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:08.501 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:26:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:08 smithi121 bash[36902]: audit 2022-04-22T20:26:07.361700+0000 mon.a (mon.0) 9440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:08 smithi121 bash[36902]: audit 2022-04-22T20:26:07.492005+0000 mon.a (mon.0) 9441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:08 smithi115 bash[45495]: audit 2022-04-22T20:26:07.361700+0000 mon.a (mon.0) 9440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:08 smithi115 bash[45495]: audit 2022-04-22T20:26:07.492005+0000 mon.a (mon.0) 9441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:08 smithi115 bash[46934]: audit 2022-04-22T20:26:07.361700+0000 mon.a (mon.0) 9440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:08 smithi115 bash[46934]: audit 2022-04-22T20:26:07.492005+0000 mon.a (mon.0) 9441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 3m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 3m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 4m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 3m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 3m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 4m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 3m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 3m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 3m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:26:08.852 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 3m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:26:08.853 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 4m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:26:08.853 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 4m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:26:08.853 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 4m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:26:08.853 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 4m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:26:08.853 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:26:09.252 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:26:09.252 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:26:09.252 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:26:09.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:26:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:26:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:26:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:26:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:26:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:26:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:26:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:26:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:26:09.255 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:26:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:09 smithi121 bash[36902]: cluster 2022-04-22T20:26:07.451018+0000 mgr.y (mgr.34107) 6162 : cluster [DBG] pgmap v7343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:09 smithi121 bash[36902]: audit 2022-04-22T20:26:08.946697+0000 mon.a (mon.0) 9442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:09 smithi115 bash[45495]: cluster 2022-04-22T20:26:07.451018+0000 mgr.y (mgr.34107) 6162 : cluster [DBG] pgmap v7343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:09 smithi115 bash[45495]: audit 2022-04-22T20:26:08.946697+0000 mon.a (mon.0) 9442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:09 smithi115 bash[46934]: cluster 2022-04-22T20:26:07.451018+0000 mgr.y (mgr.34107) 6162 : cluster [DBG] pgmap v7343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:09 smithi115 bash[46934]: audit 2022-04-22T20:26:08.946697+0000 mon.a (mon.0) 9442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:10 smithi121 bash[36902]: audit 2022-04-22T20:26:08.477761+0000 mgr.y (mgr.34107) 6163 : audit [DBG] from='client.41796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:10 smithi121 bash[36902]: audit 2022-04-22T20:26:08.819096+0000 mgr.y (mgr.34107) 6164 : audit [DBG] from='client.41802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:10 smithi121 bash[36902]: audit 2022-04-22T20:26:09.241772+0000 mon.a (mon.0) 9443 : audit [DBG] from='client.? 172.21.15.115:0/1863688104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:10.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[45495]: audit 2022-04-22T20:26:08.477761+0000 mgr.y (mgr.34107) 6163 : audit [DBG] from='client.41796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[45495]: audit 2022-04-22T20:26:08.819096+0000 mgr.y (mgr.34107) 6164 : audit [DBG] from='client.41802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[45495]: audit 2022-04-22T20:26:09.241772+0000 mon.a (mon.0) 9443 : audit [DBG] from='client.? 172.21.15.115:0/1863688104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:10.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[46934]: audit 2022-04-22T20:26:08.477761+0000 mgr.y (mgr.34107) 6163 : audit [DBG] from='client.41796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[46934]: audit 2022-04-22T20:26:08.819096+0000 mgr.y (mgr.34107) 6164 : audit [DBG] from='client.41802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:10.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:10 smithi115 bash[46934]: audit 2022-04-22T20:26:09.241772+0000 mon.a (mon.0) 9443 : audit [DBG] from='client.? 172.21.15.115:0/1863688104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:11 smithi121 bash[36902]: cluster 2022-04-22T20:26:09.452097+0000 mgr.y (mgr.34107) 6165 : cluster [DBG] pgmap v7344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:11 smithi115 bash[45495]: cluster 2022-04-22T20:26:09.452097+0000 mgr.y (mgr.34107) 6165 : cluster [DBG] pgmap v7344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:11 smithi115 bash[46934]: cluster 2022-04-22T20:26:09.452097+0000 mgr.y (mgr.34107) 6165 : cluster [DBG] pgmap v7344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:12.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:13 smithi121 bash[36902]: cluster 2022-04-22T20:26:11.453137+0000 mgr.y (mgr.34107) 6166 : cluster [DBG] pgmap v7345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:13 smithi121 bash[36902]: audit 2022-04-22T20:26:12.365471+0000 mon.a (mon.0) 9444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:13 smithi121 bash[36902]: audit 2022-04-22T20:26:12.495085+0000 mon.a (mon.0) 9445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[45495]: cluster 2022-04-22T20:26:11.453137+0000 mgr.y (mgr.34107) 6166 : cluster [DBG] pgmap v7345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[45495]: audit 2022-04-22T20:26:12.365471+0000 mon.a (mon.0) 9444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[45495]: audit 2022-04-22T20:26:12.495085+0000 mon.a (mon.0) 9445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[46934]: cluster 2022-04-22T20:26:11.453137+0000 mgr.y (mgr.34107) 6166 : cluster [DBG] pgmap v7345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[46934]: audit 2022-04-22T20:26:12.365471+0000 mon.a (mon.0) 9444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:13 smithi115 bash[46934]: audit 2022-04-22T20:26:12.495085+0000 mon.a (mon.0) 9445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:14 smithi121 bash[36902]: audit 2022-04-22T20:26:13.949411+0000 mon.a (mon.0) 9446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:14.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:14.190Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:14.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:14 smithi115 bash[45495]: audit 2022-04-22T20:26:13.949411+0000 mon.a (mon.0) 9446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:14 smithi115 bash[46934]: audit 2022-04-22T20:26:13.949411+0000 mon.a (mon.0) 9446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:15 smithi121 bash[36902]: cluster 2022-04-22T20:26:13.454268+0000 mgr.y (mgr.34107) 6167 : cluster [DBG] pgmap v7346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:15 smithi115 bash[45495]: cluster 2022-04-22T20:26:13.454268+0000 mgr.y (mgr.34107) 6167 : cluster [DBG] pgmap v7346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:15 smithi115 bash[46934]: cluster 2022-04-22T20:26:13.454268+0000 mgr.y (mgr.34107) 6167 : cluster [DBG] pgmap v7346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:16 smithi121 bash[36902]: audit 2022-04-22T20:26:15.343013+0000 mon.a (mon.0) 9447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:26:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:16 smithi121 bash[36902]: audit 2022-04-22T20:26:15.344523+0000 mon.a (mon.0) 9448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:26:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:16 smithi115 bash[45495]: audit 2022-04-22T20:26:15.343013+0000 mon.a (mon.0) 9447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:26:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:16 smithi115 bash[45495]: audit 2022-04-22T20:26:15.344523+0000 mon.a (mon.0) 9448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:26:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:16 smithi115 bash[46934]: audit 2022-04-22T20:26:15.343013+0000 mon.a (mon.0) 9447 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:26:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:16 smithi115 bash[46934]: audit 2022-04-22T20:26:15.344523+0000 mon.a (mon.0) 9448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:26:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:17 smithi121 bash[36902]: cluster 2022-04-22T20:26:15.455242+0000 mgr.y (mgr.34107) 6168 : cluster [DBG] pgmap v7347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:17 smithi115 bash[45495]: cluster 2022-04-22T20:26:15.455242+0000 mgr.y (mgr.34107) 6168 : cluster [DBG] pgmap v7347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:17 smithi115 bash[46934]: cluster 2022-04-22T20:26:15.455242+0000 mgr.y (mgr.34107) 6168 : cluster [DBG] pgmap v7347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:18 smithi121 bash[36902]: audit 2022-04-22T20:26:17.369222+0000 mon.a (mon.0) 9449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:18 smithi121 bash[36902]: audit 2022-04-22T20:26:17.498036+0000 mon.a (mon.0) 9450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:18 smithi115 bash[45495]: audit 2022-04-22T20:26:17.369222+0000 mon.a (mon.0) 9449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:18 smithi115 bash[45495]: audit 2022-04-22T20:26:17.498036+0000 mon.a (mon.0) 9450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:18 smithi115 bash[46934]: audit 2022-04-22T20:26:17.369222+0000 mon.a (mon.0) 9449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:18 smithi115 bash[46934]: audit 2022-04-22T20:26:17.498036+0000 mon.a (mon.0) 9450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:19 smithi121 bash[36902]: cluster 2022-04-22T20:26:17.456294+0000 mgr.y (mgr.34107) 6169 : cluster [DBG] pgmap v7348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:19 smithi121 bash[36902]: audit 2022-04-22T20:26:18.952307+0000 mon.a (mon.0) 9451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:19.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:19 smithi115 bash[45495]: cluster 2022-04-22T20:26:17.456294+0000 mgr.y (mgr.34107) 6169 : cluster [DBG] pgmap v7348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:19.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:19 smithi115 bash[45495]: audit 2022-04-22T20:26:18.952307+0000 mon.a (mon.0) 9451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:19.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:19 smithi115 bash[46934]: cluster 2022-04-22T20:26:17.456294+0000 mgr.y (mgr.34107) 6169 : cluster [DBG] pgmap v7348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:19.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:19 smithi115 bash[46934]: audit 2022-04-22T20:26:18.952307+0000 mon.a (mon.0) 9451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:21 smithi121 bash[36902]: cluster 2022-04-22T20:26:19.457362+0000 mgr.y (mgr.34107) 6170 : cluster [DBG] pgmap v7349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:21.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:21 smithi115 bash[45495]: cluster 2022-04-22T20:26:19.457362+0000 mgr.y (mgr.34107) 6170 : cluster [DBG] pgmap v7349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:21.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:21 smithi115 bash[46934]: cluster 2022-04-22T20:26:19.457362+0000 mgr.y (mgr.34107) 6170 : cluster [DBG] pgmap v7349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:22.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:23 smithi121 bash[36902]: cluster 2022-04-22T20:26:21.458422+0000 mgr.y (mgr.34107) 6171 : cluster [DBG] pgmap v7350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:23 smithi121 bash[36902]: audit 2022-04-22T20:26:22.372852+0000 mon.a (mon.0) 9452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:23 smithi121 bash[36902]: audit 2022-04-22T20:26:22.500685+0000 mon.a (mon.0) 9453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[45495]: cluster 2022-04-22T20:26:21.458422+0000 mgr.y (mgr.34107) 6171 : cluster [DBG] pgmap v7350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[45495]: audit 2022-04-22T20:26:22.372852+0000 mon.a (mon.0) 9452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:23.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[45495]: audit 2022-04-22T20:26:22.500685+0000 mon.a (mon.0) 9453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[46934]: cluster 2022-04-22T20:26:21.458422+0000 mgr.y (mgr.34107) 6171 : cluster [DBG] pgmap v7350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[46934]: audit 2022-04-22T20:26:22.372852+0000 mon.a (mon.0) 9452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:23 smithi115 bash[46934]: audit 2022-04-22T20:26:22.500685+0000 mon.a (mon.0) 9453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:24 smithi121 bash[36902]: audit 2022-04-22T20:26:23.955202+0000 mon.a (mon.0) 9454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:24 smithi115 bash[45495]: audit 2022-04-22T20:26:23.955202+0000 mon.a (mon.0) 9454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:24.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:24.189Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:24 smithi115 bash[46934]: audit 2022-04-22T20:26:23.955202+0000 mon.a (mon.0) 9454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:25 smithi121 bash[36902]: cluster 2022-04-22T20:26:23.459602+0000 mgr.y (mgr.34107) 6172 : cluster [DBG] pgmap v7351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:25 smithi115 bash[45495]: cluster 2022-04-22T20:26:23.459602+0000 mgr.y (mgr.34107) 6172 : cluster [DBG] pgmap v7351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:25 smithi115 bash[46934]: cluster 2022-04-22T20:26:23.459602+0000 mgr.y (mgr.34107) 6172 : cluster [DBG] pgmap v7351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:26 smithi121 bash[36902]: cluster 2022-04-22T20:26:25.460643+0000 mgr.y (mgr.34107) 6173 : cluster [DBG] pgmap v7352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:26 smithi115 bash[45495]: cluster 2022-04-22T20:26:25.460643+0000 mgr.y (mgr.34107) 6173 : cluster [DBG] pgmap v7352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:26 smithi115 bash[46934]: cluster 2022-04-22T20:26:25.460643+0000 mgr.y (mgr.34107) 6173 : cluster [DBG] pgmap v7352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:27 smithi121 bash[36902]: audit 2022-04-22T20:26:27.376560+0000 mon.a (mon.0) 9455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:27 smithi115 bash[46934]: audit 2022-04-22T20:26:27.376560+0000 mon.a (mon.0) 9455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:27.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:27 smithi115 bash[45495]: audit 2022-04-22T20:26:27.376560+0000 mon.a (mon.0) 9455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:28 smithi121 bash[36902]: cluster 2022-04-22T20:26:27.461739+0000 mgr.y (mgr.34107) 6174 : cluster [DBG] pgmap v7353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:28 smithi121 bash[36902]: audit 2022-04-22T20:26:27.503571+0000 mon.a (mon.0) 9456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:28 smithi115 bash[45495]: cluster 2022-04-22T20:26:27.461739+0000 mgr.y (mgr.34107) 6174 : cluster [DBG] pgmap v7353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:28 smithi115 bash[45495]: audit 2022-04-22T20:26:27.503571+0000 mon.a (mon.0) 9456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:28.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:28 smithi115 bash[46934]: cluster 2022-04-22T20:26:27.461739+0000 mgr.y (mgr.34107) 6174 : cluster [DBG] pgmap v7353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:28.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:28 smithi115 bash[46934]: audit 2022-04-22T20:26:27.503571+0000 mon.a (mon.0) 9456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:29 smithi121 bash[36902]: audit 2022-04-22T20:26:28.958043+0000 mon.a (mon.0) 9457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:29.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:29 smithi115 bash[45495]: audit 2022-04-22T20:26:28.958043+0000 mon.a (mon.0) 9457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:29.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:29 smithi115 bash[46934]: audit 2022-04-22T20:26:28.958043+0000 mon.a (mon.0) 9457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:30 smithi121 bash[36902]: cluster 2022-04-22T20:26:29.462855+0000 mgr.y (mgr.34107) 6175 : cluster [DBG] pgmap v7354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:30 smithi115 bash[45495]: cluster 2022-04-22T20:26:29.462855+0000 mgr.y (mgr.34107) 6175 : cluster [DBG] pgmap v7354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:30 smithi115 bash[46934]: cluster 2022-04-22T20:26:29.462855+0000 mgr.y (mgr.34107) 6175 : cluster [DBG] pgmap v7354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:32.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:32.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:32.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[45495]: cluster 2022-04-22T20:26:31.463832+0000 mgr.y (mgr.34107) 6176 : cluster [DBG] pgmap v7355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:32.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[45495]: audit 2022-04-22T20:26:32.380282+0000 mon.a (mon.0) 9458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:32.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[45495]: audit 2022-04-22T20:26:32.506692+0000 mon.a (mon.0) 9459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[46934]: cluster 2022-04-22T20:26:31.463832+0000 mgr.y (mgr.34107) 6176 : cluster [DBG] pgmap v7355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[46934]: audit 2022-04-22T20:26:32.380282+0000 mon.a (mon.0) 9458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:32.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:32 smithi115 bash[46934]: audit 2022-04-22T20:26:32.506692+0000 mon.a (mon.0) 9459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:32 smithi121 bash[36902]: cluster 2022-04-22T20:26:31.463832+0000 mgr.y (mgr.34107) 6176 : cluster [DBG] pgmap v7355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:32 smithi121 bash[36902]: audit 2022-04-22T20:26:32.380282+0000 mon.a (mon.0) 9458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:33.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:32 smithi121 bash[36902]: audit 2022-04-22T20:26:32.506692+0000 mon.a (mon.0) 9459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:34.532 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:34.191Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:34.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:34 smithi115 bash[45495]: cluster 2022-04-22T20:26:33.464813+0000 mgr.y (mgr.34107) 6177 : cluster [DBG] pgmap v7356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:34.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:34 smithi115 bash[45495]: audit 2022-04-22T20:26:33.961001+0000 mon.a (mon.0) 9460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:34 smithi115 bash[46934]: cluster 2022-04-22T20:26:33.464813+0000 mgr.y (mgr.34107) 6177 : cluster [DBG] pgmap v7356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:34.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:34 smithi115 bash[46934]: audit 2022-04-22T20:26:33.961001+0000 mon.a (mon.0) 9460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:34 smithi121 bash[36902]: cluster 2022-04-22T20:26:33.464813+0000 mgr.y (mgr.34107) 6177 : cluster [DBG] pgmap v7356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:34.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:34 smithi121 bash[36902]: audit 2022-04-22T20:26:33.961001+0000 mon.a (mon.0) 9460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:35.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[45495]: audit 2022-04-22T20:26:34.853854+0000 mon.a (mon.0) 9461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:26:35.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[45495]: audit 2022-04-22T20:26:34.857013+0000 mon.a (mon.0) 9462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:26:35.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[45495]: audit 2022-04-22T20:26:34.858862+0000 mon.a (mon.0) 9463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:26:35.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[45495]: audit 2022-04-22T20:26:35.218626+0000 mon.a (mon.0) 9464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:26:35.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[46934]: audit 2022-04-22T20:26:34.853854+0000 mon.a (mon.0) 9461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:26:35.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[46934]: audit 2022-04-22T20:26:34.857013+0000 mon.a (mon.0) 9462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:26:35.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[46934]: audit 2022-04-22T20:26:34.858862+0000 mon.a (mon.0) 9463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:26:35.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:35 smithi115 bash[46934]: audit 2022-04-22T20:26:35.218626+0000 mon.a (mon.0) 9464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:26:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:35 smithi121 bash[36902]: audit 2022-04-22T20:26:34.853854+0000 mon.a (mon.0) 9461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:26:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:35 smithi121 bash[36902]: audit 2022-04-22T20:26:34.857013+0000 mon.a (mon.0) 9462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:26:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:35 smithi121 bash[36902]: audit 2022-04-22T20:26:34.858862+0000 mon.a (mon.0) 9463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:26:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:35 smithi121 bash[36902]: audit 2022-04-22T20:26:35.218626+0000 mon.a (mon.0) 9464 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:26:36.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:36 smithi115 bash[45495]: cluster 2022-04-22T20:26:35.465937+0000 mgr.y (mgr.34107) 6178 : cluster [DBG] pgmap v7357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:36 smithi115 bash[46934]: cluster 2022-04-22T20:26:35.465937+0000 mgr.y (mgr.34107) 6178 : cluster [DBG] pgmap v7357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:36 smithi121 bash[36902]: cluster 2022-04-22T20:26:35.465937+0000 mgr.y (mgr.34107) 6178 : cluster [DBG] pgmap v7357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:37 smithi121 bash[36902]: audit 2022-04-22T20:26:37.383994+0000 mon.a (mon.0) 9465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:37 smithi121 bash[36902]: audit 2022-04-22T20:26:37.509265+0000 mon.a (mon.0) 9466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:37 smithi115 bash[46934]: audit 2022-04-22T20:26:37.383994+0000 mon.a (mon.0) 9465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:38.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:37 smithi115 bash[46934]: audit 2022-04-22T20:26:37.509265+0000 mon.a (mon.0) 9466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:38.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:37 smithi115 bash[45495]: audit 2022-04-22T20:26:37.383994+0000 mon.a (mon.0) 9465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:38.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:37 smithi115 bash[45495]: audit 2022-04-22T20:26:37.509265+0000 mon.a (mon.0) 9466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:38 smithi121 bash[36902]: cluster 2022-04-22T20:26:37.467020+0000 mgr.y (mgr.34107) 6179 : cluster [DBG] pgmap v7358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:39.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:38 smithi115 bash[46934]: cluster 2022-04-22T20:26:37.467020+0000 mgr.y (mgr.34107) 6179 : cluster [DBG] pgmap v7358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:39.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:38 smithi115 bash[45495]: cluster 2022-04-22T20:26:37.467020+0000 mgr.y (mgr.34107) 6179 : cluster [DBG] pgmap v7358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:39.625 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:26:39.974 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:26:39.974 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:26:39.974 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:26:39.974 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:26:39.974 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (2h) 4m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:26:39.975 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:26:39.976 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:26:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:39 smithi121 bash[36902]: audit 2022-04-22T20:26:38.964117+0000 mon.a (mon.0) 9467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:40.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:39 smithi115 bash[46934]: audit 2022-04-22T20:26:38.964117+0000 mon.a (mon.0) 9467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:40.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:39 smithi115 bash[45495]: audit 2022-04-22T20:26:38.964117+0000 mon.a (mon.0) 9467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:40.383 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:26:40.383 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:26:40.383 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:26:40.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:26:40.385 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:26:40.386 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:26:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:41 smithi121 bash[36902]: cluster 2022-04-22T20:26:39.468107+0000 mgr.y (mgr.34107) 6180 : cluster [DBG] pgmap v7359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:41 smithi121 bash[36902]: audit 2022-04-22T20:26:39.602302+0000 mgr.y (mgr.34107) 6181 : audit [DBG] from='client.41814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:41 smithi121 bash[36902]: audit 2022-04-22T20:26:39.945916+0000 mgr.y (mgr.34107) 6182 : audit [DBG] from='client.51751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:41 smithi121 bash[36902]: audit 2022-04-22T20:26:40.374874+0000 mon.a (mon.0) 9468 : audit [DBG] from='client.? 172.21.15.115:0/2486116907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[46934]: cluster 2022-04-22T20:26:39.468107+0000 mgr.y (mgr.34107) 6180 : cluster [DBG] pgmap v7359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[46934]: audit 2022-04-22T20:26:39.602302+0000 mgr.y (mgr.34107) 6181 : audit [DBG] from='client.41814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[46934]: audit 2022-04-22T20:26:39.945916+0000 mgr.y (mgr.34107) 6182 : audit [DBG] from='client.51751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[46934]: audit 2022-04-22T20:26:40.374874+0000 mon.a (mon.0) 9468 : audit [DBG] from='client.? 172.21.15.115:0/2486116907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[45495]: cluster 2022-04-22T20:26:39.468107+0000 mgr.y (mgr.34107) 6180 : cluster [DBG] pgmap v7359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[45495]: audit 2022-04-22T20:26:39.602302+0000 mgr.y (mgr.34107) 6181 : audit [DBG] from='client.41814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[45495]: audit 2022-04-22T20:26:39.945916+0000 mgr.y (mgr.34107) 6182 : audit [DBG] from='client.51751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:26:41.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:41 smithi115 bash[45495]: audit 2022-04-22T20:26:40.374874+0000 mon.a (mon.0) 9468 : audit [DBG] from='client.? 172.21.15.115:0/2486116907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:26:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:42.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:43 smithi121 bash[36902]: cluster 2022-04-22T20:26:41.469163+0000 mgr.y (mgr.34107) 6183 : cluster [DBG] pgmap v7360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:43 smithi121 bash[36902]: audit 2022-04-22T20:26:42.387853+0000 mon.a (mon.0) 9469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:43 smithi121 bash[36902]: audit 2022-04-22T20:26:42.511856+0000 mon.a (mon.0) 9470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[45495]: cluster 2022-04-22T20:26:41.469163+0000 mgr.y (mgr.34107) 6183 : cluster [DBG] pgmap v7360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[45495]: audit 2022-04-22T20:26:42.387853+0000 mon.a (mon.0) 9469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[45495]: audit 2022-04-22T20:26:42.511856+0000 mon.a (mon.0) 9470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[46934]: cluster 2022-04-22T20:26:41.469163+0000 mgr.y (mgr.34107) 6183 : cluster [DBG] pgmap v7360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[46934]: audit 2022-04-22T20:26:42.387853+0000 mon.a (mon.0) 9469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:43 smithi115 bash[46934]: audit 2022-04-22T20:26:42.511856+0000 mon.a (mon.0) 9470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:44 smithi121 bash[36902]: audit 2022-04-22T20:26:43.967131+0000 mon.a (mon.0) 9471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:44 smithi115 bash[45495]: audit 2022-04-22T20:26:43.967131+0000 mon.a (mon.0) 9471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:44.191Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:44 smithi115 bash[46934]: audit 2022-04-22T20:26:43.967131+0000 mon.a (mon.0) 9471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:45 smithi121 bash[36902]: cluster 2022-04-22T20:26:43.470423+0000 mgr.y (mgr.34107) 6184 : cluster [DBG] pgmap v7361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:45 smithi115 bash[46934]: cluster 2022-04-22T20:26:43.470423+0000 mgr.y (mgr.34107) 6184 : cluster [DBG] pgmap v7361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:45 smithi115 bash[45495]: cluster 2022-04-22T20:26:43.470423+0000 mgr.y (mgr.34107) 6184 : cluster [DBG] pgmap v7361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:47 smithi121 bash[36902]: cluster 2022-04-22T20:26:45.471498+0000 mgr.y (mgr.34107) 6185 : cluster [DBG] pgmap v7362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:47 smithi115 bash[45495]: cluster 2022-04-22T20:26:45.471498+0000 mgr.y (mgr.34107) 6185 : cluster [DBG] pgmap v7362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:47 smithi115 bash[46934]: cluster 2022-04-22T20:26:45.471498+0000 mgr.y (mgr.34107) 6185 : cluster [DBG] pgmap v7362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:48 smithi121 bash[36902]: audit 2022-04-22T20:26:47.391631+0000 mon.a (mon.0) 9472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:48.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:48 smithi121 bash[36902]: audit 2022-04-22T20:26:47.514745+0000 mon.a (mon.0) 9473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:48 smithi115 bash[45495]: audit 2022-04-22T20:26:47.391631+0000 mon.a (mon.0) 9472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:48 smithi115 bash[45495]: audit 2022-04-22T20:26:47.514745+0000 mon.a (mon.0) 9473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:48 smithi115 bash[46934]: audit 2022-04-22T20:26:47.391631+0000 mon.a (mon.0) 9472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:48.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:48 smithi115 bash[46934]: audit 2022-04-22T20:26:47.514745+0000 mon.a (mon.0) 9473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:49 smithi121 bash[36902]: cluster 2022-04-22T20:26:47.472550+0000 mgr.y (mgr.34107) 6186 : cluster [DBG] pgmap v7363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:49 smithi121 bash[36902]: audit 2022-04-22T20:26:48.970254+0000 mon.a (mon.0) 9474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:49 smithi115 bash[45495]: cluster 2022-04-22T20:26:47.472550+0000 mgr.y (mgr.34107) 6186 : cluster [DBG] pgmap v7363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:49 smithi115 bash[45495]: audit 2022-04-22T20:26:48.970254+0000 mon.a (mon.0) 9474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:49 smithi115 bash[46934]: cluster 2022-04-22T20:26:47.472550+0000 mgr.y (mgr.34107) 6186 : cluster [DBG] pgmap v7363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:49 smithi115 bash[46934]: audit 2022-04-22T20:26:48.970254+0000 mon.a (mon.0) 9474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:51 smithi121 bash[36902]: cluster 2022-04-22T20:26:49.473684+0000 mgr.y (mgr.34107) 6187 : cluster [DBG] pgmap v7364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:51 smithi115 bash[45495]: cluster 2022-04-22T20:26:49.473684+0000 mgr.y (mgr.34107) 6187 : cluster [DBG] pgmap v7364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:51 smithi115 bash[46934]: cluster 2022-04-22T20:26:49.473684+0000 mgr.y (mgr.34107) 6187 : cluster [DBG] pgmap v7364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:52.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:53 smithi121 bash[36902]: cluster 2022-04-22T20:26:51.474848+0000 mgr.y (mgr.34107) 6188 : cluster [DBG] pgmap v7365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:53 smithi121 bash[36902]: audit 2022-04-22T20:26:52.395369+0000 mon.a (mon.0) 9475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:53 smithi121 bash[36902]: audit 2022-04-22T20:26:52.520603+0000 mon.a (mon.0) 9476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[45495]: cluster 2022-04-22T20:26:51.474848+0000 mgr.y (mgr.34107) 6188 : cluster [DBG] pgmap v7365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[45495]: audit 2022-04-22T20:26:52.395369+0000 mon.a (mon.0) 9475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[45495]: audit 2022-04-22T20:26:52.520603+0000 mon.a (mon.0) 9476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[46934]: cluster 2022-04-22T20:26:51.474848+0000 mgr.y (mgr.34107) 6188 : cluster [DBG] pgmap v7365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[46934]: audit 2022-04-22T20:26:52.395369+0000 mon.a (mon.0) 9475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:53 smithi115 bash[46934]: audit 2022-04-22T20:26:52.520603+0000 mon.a (mon.0) 9476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:54 smithi121 bash[36902]: audit 2022-04-22T20:26:53.973232+0000 mon.a (mon.0) 9477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:54 smithi115 bash[45495]: audit 2022-04-22T20:26:53.973232+0000 mon.a (mon.0) 9477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:26:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:26:54.192Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:26:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:54 smithi115 bash[46934]: audit 2022-04-22T20:26:53.973232+0000 mon.a (mon.0) 9477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:55 smithi121 bash[36902]: cluster 2022-04-22T20:26:53.476002+0000 mgr.y (mgr.34107) 6189 : cluster [DBG] pgmap v7366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:55 smithi115 bash[45495]: cluster 2022-04-22T20:26:53.476002+0000 mgr.y (mgr.34107) 6189 : cluster [DBG] pgmap v7366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:55 smithi115 bash[46934]: cluster 2022-04-22T20:26:53.476002+0000 mgr.y (mgr.34107) 6189 : cluster [DBG] pgmap v7366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:57 smithi121 bash[36902]: cluster 2022-04-22T20:26:55.477058+0000 mgr.y (mgr.34107) 6190 : cluster [DBG] pgmap v7367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:57 smithi115 bash[45495]: cluster 2022-04-22T20:26:55.477058+0000 mgr.y (mgr.34107) 6190 : cluster [DBG] pgmap v7367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:57 smithi115 bash[46934]: cluster 2022-04-22T20:26:55.477058+0000 mgr.y (mgr.34107) 6190 : cluster [DBG] pgmap v7367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:58 smithi121 bash[36902]: audit 2022-04-22T20:26:57.399053+0000 mon.a (mon.0) 9478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:58 smithi121 bash[36902]: audit 2022-04-22T20:26:57.523323+0000 mon.a (mon.0) 9479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:58 smithi115 bash[45495]: audit 2022-04-22T20:26:57.399053+0000 mon.a (mon.0) 9478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:58 smithi115 bash[45495]: audit 2022-04-22T20:26:57.523323+0000 mon.a (mon.0) 9479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:58 smithi115 bash[46934]: audit 2022-04-22T20:26:57.399053+0000 mon.a (mon.0) 9478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:26:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:58 smithi115 bash[46934]: audit 2022-04-22T20:26:57.523323+0000 mon.a (mon.0) 9479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:26:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:59 smithi121 bash[36902]: cluster 2022-04-22T20:26:57.478095+0000 mgr.y (mgr.34107) 6191 : cluster [DBG] pgmap v7368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:26:59 smithi121 bash[36902]: audit 2022-04-22T20:26:58.976438+0000 mon.a (mon.0) 9480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:59 smithi115 bash[46934]: cluster 2022-04-22T20:26:57.478095+0000 mgr.y (mgr.34107) 6191 : cluster [DBG] pgmap v7368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:26:59 smithi115 bash[46934]: audit 2022-04-22T20:26:58.976438+0000 mon.a (mon.0) 9480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:26:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:59 smithi115 bash[45495]: cluster 2022-04-22T20:26:57.478095+0000 mgr.y (mgr.34107) 6191 : cluster [DBG] pgmap v7368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:26:59.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:26:59 smithi115 bash[45495]: audit 2022-04-22T20:26:58.976438+0000 mon.a (mon.0) 9480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:01 smithi121 bash[36902]: cluster 2022-04-22T20:26:59.479209+0000 mgr.y (mgr.34107) 6192 : cluster [DBG] pgmap v7369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:01 smithi115 bash[45495]: cluster 2022-04-22T20:26:59.479209+0000 mgr.y (mgr.34107) 6192 : cluster [DBG] pgmap v7369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:01 smithi115 bash[46934]: cluster 2022-04-22T20:26:59.479209+0000 mgr.y (mgr.34107) 6192 : cluster [DBG] pgmap v7369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:02.313Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:03 smithi121 bash[36902]: cluster 2022-04-22T20:27:01.480350+0000 mgr.y (mgr.34107) 6193 : cluster [DBG] pgmap v7370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:03.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:03 smithi121 bash[36902]: audit 2022-04-22T20:27:02.402917+0000 mon.a (mon.0) 9481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:03.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:03 smithi121 bash[36902]: audit 2022-04-22T20:27:02.526156+0000 mon.a (mon.0) 9482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:03.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[45495]: cluster 2022-04-22T20:27:01.480350+0000 mgr.y (mgr.34107) 6193 : cluster [DBG] pgmap v7370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:03.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[45495]: audit 2022-04-22T20:27:02.402917+0000 mon.a (mon.0) 9481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:03.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[45495]: audit 2022-04-22T20:27:02.526156+0000 mon.a (mon.0) 9482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[46934]: cluster 2022-04-22T20:27:01.480350+0000 mgr.y (mgr.34107) 6193 : cluster [DBG] pgmap v7370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[46934]: audit 2022-04-22T20:27:02.402917+0000 mon.a (mon.0) 9481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:03 smithi115 bash[46934]: audit 2022-04-22T20:27:02.526156+0000 mon.a (mon.0) 9482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:04 smithi121 bash[36902]: audit 2022-04-22T20:27:03.979369+0000 mon.a (mon.0) 9483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:04 smithi115 bash[45495]: audit 2022-04-22T20:27:03.979369+0000 mon.a (mon.0) 9483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:04.193Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:04 smithi115 bash[46934]: audit 2022-04-22T20:27:03.979369+0000 mon.a (mon.0) 9483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:05 smithi121 bash[36902]: cluster 2022-04-22T20:27:03.481500+0000 mgr.y (mgr.34107) 6194 : cluster [DBG] pgmap v7371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:05 smithi115 bash[45495]: cluster 2022-04-22T20:27:03.481500+0000 mgr.y (mgr.34107) 6194 : cluster [DBG] pgmap v7371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:05 smithi115 bash[46934]: cluster 2022-04-22T20:27:03.481500+0000 mgr.y (mgr.34107) 6194 : cluster [DBG] pgmap v7371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:07 smithi121 bash[36902]: cluster 2022-04-22T20:27:05.482630+0000 mgr.y (mgr.34107) 6195 : cluster [DBG] pgmap v7372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:07 smithi115 bash[45495]: cluster 2022-04-22T20:27:05.482630+0000 mgr.y (mgr.34107) 6195 : cluster [DBG] pgmap v7372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:07 smithi115 bash[46934]: cluster 2022-04-22T20:27:05.482630+0000 mgr.y (mgr.34107) 6195 : cluster [DBG] pgmap v7372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:08 smithi121 bash[36902]: audit 2022-04-22T20:27:07.406492+0000 mon.a (mon.0) 9484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:08 smithi121 bash[36902]: audit 2022-04-22T20:27:07.529126+0000 mon.a (mon.0) 9485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:08 smithi115 bash[45495]: audit 2022-04-22T20:27:07.406492+0000 mon.a (mon.0) 9484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:08 smithi115 bash[45495]: audit 2022-04-22T20:27:07.529126+0000 mon.a (mon.0) 9485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:08 smithi115 bash[46934]: audit 2022-04-22T20:27:07.406492+0000 mon.a (mon.0) 9484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:08 smithi115 bash[46934]: audit 2022-04-22T20:27:07.529126+0000 mon.a (mon.0) 9485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:09 smithi121 bash[36902]: cluster 2022-04-22T20:27:07.483696+0000 mgr.y (mgr.34107) 6196 : cluster [DBG] pgmap v7373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:09 smithi121 bash[36902]: audit 2022-04-22T20:27:08.982441+0000 mon.a (mon.0) 9486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:09 smithi115 bash[45495]: cluster 2022-04-22T20:27:07.483696+0000 mgr.y (mgr.34107) 6196 : cluster [DBG] pgmap v7373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:09 smithi115 bash[45495]: audit 2022-04-22T20:27:08.982441+0000 mon.a (mon.0) 9486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:09 smithi115 bash[46934]: cluster 2022-04-22T20:27:07.483696+0000 mgr.y (mgr.34107) 6196 : cluster [DBG] pgmap v7373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:09 smithi115 bash[46934]: audit 2022-04-22T20:27:08.982441+0000 mon.a (mon.0) 9486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:10.758 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:27:11.110 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:27:11.110 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 4m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:27:11.110 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:27:11.110 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (2h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 4m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 5m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 4m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 4m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 5m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 4m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:27:11.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 4m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 4m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 4m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 5m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 5m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 5m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 5m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:27:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 5m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:27:11.512 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:27:11.512 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:27:11.512 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:27:11.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:11.513 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:27:11.513 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:27:11.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:11.514 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:27:11.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:27:11.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:11.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:27:11.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:27:11.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:11.515 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:27:11.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:27:11.515 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:27:11.515 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:27:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:11 smithi121 bash[36902]: cluster 2022-04-22T20:27:09.484833+0000 mgr.y (mgr.34107) 6197 : cluster [DBG] pgmap v7374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:11 smithi115 bash[46934]: cluster 2022-04-22T20:27:09.484833+0000 mgr.y (mgr.34107) 6197 : cluster [DBG] pgmap v7374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:11 smithi115 bash[45495]: cluster 2022-04-22T20:27:09.484833+0000 mgr.y (mgr.34107) 6197 : cluster [DBG] pgmap v7374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:12 smithi121 bash[36902]: audit 2022-04-22T20:27:10.735204+0000 mgr.y (mgr.34107) 6198 : audit [DBG] from='client.41832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:12 smithi121 bash[36902]: audit 2022-04-22T20:27:11.080932+0000 mgr.y (mgr.34107) 6199 : audit [DBG] from='client.51769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:12 smithi121 bash[36902]: audit 2022-04-22T20:27:11.501710+0000 mon.c (mon.1) 243 : audit [DBG] from='client.? 172.21.15.115:0/1728595596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:12.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:12.313Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:12.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[45495]: audit 2022-04-22T20:27:10.735204+0000 mgr.y (mgr.34107) 6198 : audit [DBG] from='client.41832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[45495]: audit 2022-04-22T20:27:11.080932+0000 mgr.y (mgr.34107) 6199 : audit [DBG] from='client.51769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[45495]: audit 2022-04-22T20:27:11.501710+0000 mon.c (mon.1) 243 : audit [DBG] from='client.? 172.21.15.115:0/1728595596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:12.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[46934]: audit 2022-04-22T20:27:10.735204+0000 mgr.y (mgr.34107) 6198 : audit [DBG] from='client.41832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[46934]: audit 2022-04-22T20:27:11.080932+0000 mgr.y (mgr.34107) 6199 : audit [DBG] from='client.51769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:12.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:12 smithi115 bash[46934]: audit 2022-04-22T20:27:11.501710+0000 mon.c (mon.1) 243 : audit [DBG] from='client.? 172.21.15.115:0/1728595596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:13 smithi121 bash[36902]: cluster 2022-04-22T20:27:11.485822+0000 mgr.y (mgr.34107) 6200 : cluster [DBG] pgmap v7375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:13 smithi121 bash[36902]: audit 2022-04-22T20:27:12.410193+0000 mon.a (mon.0) 9487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:13 smithi121 bash[36902]: audit 2022-04-22T20:27:12.531990+0000 mon.a (mon.0) 9488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[45495]: cluster 2022-04-22T20:27:11.485822+0000 mgr.y (mgr.34107) 6200 : cluster [DBG] pgmap v7375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[45495]: audit 2022-04-22T20:27:12.410193+0000 mon.a (mon.0) 9487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[45495]: audit 2022-04-22T20:27:12.531990+0000 mon.a (mon.0) 9488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[46934]: cluster 2022-04-22T20:27:11.485822+0000 mgr.y (mgr.34107) 6200 : cluster [DBG] pgmap v7375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[46934]: audit 2022-04-22T20:27:12.410193+0000 mon.a (mon.0) 9487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:13 smithi115 bash[46934]: audit 2022-04-22T20:27:12.531990+0000 mon.a (mon.0) 9488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:14 smithi121 bash[36902]: audit 2022-04-22T20:27:13.985539+0000 mon.a (mon.0) 9489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:14 smithi115 bash[45495]: audit 2022-04-22T20:27:13.985539+0000 mon.a (mon.0) 9489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:14.194Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:14 smithi115 bash[46934]: audit 2022-04-22T20:27:13.985539+0000 mon.a (mon.0) 9489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:15 smithi121 bash[36902]: cluster 2022-04-22T20:27:13.487065+0000 mgr.y (mgr.34107) 6201 : cluster [DBG] pgmap v7376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:15 smithi115 bash[45495]: cluster 2022-04-22T20:27:13.487065+0000 mgr.y (mgr.34107) 6201 : cluster [DBG] pgmap v7376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:15 smithi115 bash[46934]: cluster 2022-04-22T20:27:13.487065+0000 mgr.y (mgr.34107) 6201 : cluster [DBG] pgmap v7376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:16 smithi121 bash[36902]: audit 2022-04-22T20:27:15.358120+0000 mon.a (mon.0) 9490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:27:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:16 smithi121 bash[36902]: audit 2022-04-22T20:27:15.358572+0000 mon.a (mon.0) 9491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:27:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:16 smithi115 bash[45495]: audit 2022-04-22T20:27:15.358120+0000 mon.a (mon.0) 9490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:27:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:16 smithi115 bash[45495]: audit 2022-04-22T20:27:15.358572+0000 mon.a (mon.0) 9491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:27:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:16 smithi115 bash[46934]: audit 2022-04-22T20:27:15.358120+0000 mon.a (mon.0) 9490 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:27:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:16 smithi115 bash[46934]: audit 2022-04-22T20:27:15.358572+0000 mon.a (mon.0) 9491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:27:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:17 smithi121 bash[36902]: cluster 2022-04-22T20:27:15.488210+0000 mgr.y (mgr.34107) 6202 : cluster [DBG] pgmap v7377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:17 smithi115 bash[45495]: cluster 2022-04-22T20:27:15.488210+0000 mgr.y (mgr.34107) 6202 : cluster [DBG] pgmap v7377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:17 smithi115 bash[46934]: cluster 2022-04-22T20:27:15.488210+0000 mgr.y (mgr.34107) 6202 : cluster [DBG] pgmap v7377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:18 smithi121 bash[36902]: audit 2022-04-22T20:27:17.413779+0000 mon.a (mon.0) 9492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:18 smithi121 bash[36902]: audit 2022-04-22T20:27:17.534934+0000 mon.a (mon.0) 9493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:18 smithi115 bash[45495]: audit 2022-04-22T20:27:17.413779+0000 mon.a (mon.0) 9492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:18 smithi115 bash[45495]: audit 2022-04-22T20:27:17.534934+0000 mon.a (mon.0) 9493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:18 smithi115 bash[46934]: audit 2022-04-22T20:27:17.413779+0000 mon.a (mon.0) 9492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:18 smithi115 bash[46934]: audit 2022-04-22T20:27:17.534934+0000 mon.a (mon.0) 9493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:19 smithi121 bash[36902]: cluster 2022-04-22T20:27:17.489176+0000 mgr.y (mgr.34107) 6203 : cluster [DBG] pgmap v7378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:19 smithi121 bash[36902]: audit 2022-04-22T20:27:18.988835+0000 mon.a (mon.0) 9494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:19 smithi115 bash[45495]: cluster 2022-04-22T20:27:17.489176+0000 mgr.y (mgr.34107) 6203 : cluster [DBG] pgmap v7378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:19 smithi115 bash[45495]: audit 2022-04-22T20:27:18.988835+0000 mon.a (mon.0) 9494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:19 smithi115 bash[46934]: cluster 2022-04-22T20:27:17.489176+0000 mgr.y (mgr.34107) 6203 : cluster [DBG] pgmap v7378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:19 smithi115 bash[46934]: audit 2022-04-22T20:27:18.988835+0000 mon.a (mon.0) 9494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:20 smithi121 bash[36902]: cluster 2022-04-22T20:27:19.490286+0000 mgr.y (mgr.34107) 6204 : cluster [DBG] pgmap v7379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:20 smithi115 bash[46934]: cluster 2022-04-22T20:27:19.490286+0000 mgr.y (mgr.34107) 6204 : cluster [DBG] pgmap v7379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:20.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:20 smithi115 bash[45495]: cluster 2022-04-22T20:27:19.490286+0000 mgr.y (mgr.34107) 6204 : cluster [DBG] pgmap v7379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:22.314Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[45495]: cluster 2022-04-22T20:27:21.491369+0000 mgr.y (mgr.34107) 6205 : cluster [DBG] pgmap v7380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[45495]: audit 2022-04-22T20:27:22.417682+0000 mon.a (mon.0) 9495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:22.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[45495]: audit 2022-04-22T20:27:22.537616+0000 mon.a (mon.0) 9496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[46934]: cluster 2022-04-22T20:27:21.491369+0000 mgr.y (mgr.34107) 6205 : cluster [DBG] pgmap v7380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[46934]: audit 2022-04-22T20:27:22.417682+0000 mon.a (mon.0) 9495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:22.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:22 smithi115 bash[46934]: audit 2022-04-22T20:27:22.537616+0000 mon.a (mon.0) 9496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:22 smithi121 bash[36902]: cluster 2022-04-22T20:27:21.491369+0000 mgr.y (mgr.34107) 6205 : cluster [DBG] pgmap v7380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:22 smithi121 bash[36902]: audit 2022-04-22T20:27:22.417682+0000 mon.a (mon.0) 9495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:22 smithi121 bash[36902]: audit 2022-04-22T20:27:22.537616+0000 mon.a (mon.0) 9496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:24.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:24.194Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:24 smithi121 bash[36902]: cluster 2022-04-22T20:27:23.492464+0000 mgr.y (mgr.34107) 6206 : cluster [DBG] pgmap v7381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:24 smithi121 bash[36902]: audit 2022-04-22T20:27:23.992015+0000 mon.a (mon.0) 9497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:25.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:24 smithi115 bash[45495]: cluster 2022-04-22T20:27:23.492464+0000 mgr.y (mgr.34107) 6206 : cluster [DBG] pgmap v7381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:25.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:24 smithi115 bash[45495]: audit 2022-04-22T20:27:23.992015+0000 mon.a (mon.0) 9497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:24 smithi115 bash[46934]: cluster 2022-04-22T20:27:23.492464+0000 mgr.y (mgr.34107) 6206 : cluster [DBG] pgmap v7381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:24 smithi115 bash[46934]: audit 2022-04-22T20:27:23.992015+0000 mon.a (mon.0) 9497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:26 smithi121 bash[36902]: cluster 2022-04-22T20:27:25.493600+0000 mgr.y (mgr.34107) 6207 : cluster [DBG] pgmap v7382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:27.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:26 smithi115 bash[45495]: cluster 2022-04-22T20:27:25.493600+0000 mgr.y (mgr.34107) 6207 : cluster [DBG] pgmap v7382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:27.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:26 smithi115 bash[46934]: cluster 2022-04-22T20:27:25.493600+0000 mgr.y (mgr.34107) 6207 : cluster [DBG] pgmap v7382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:27 smithi121 bash[36902]: audit 2022-04-22T20:27:27.421440+0000 mon.a (mon.0) 9498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:27 smithi121 bash[36902]: audit 2022-04-22T20:27:27.540423+0000 mon.a (mon.0) 9499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:27 smithi115 bash[45495]: audit 2022-04-22T20:27:27.421440+0000 mon.a (mon.0) 9498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:28.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:27 smithi115 bash[45495]: audit 2022-04-22T20:27:27.540423+0000 mon.a (mon.0) 9499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:27 smithi115 bash[46934]: audit 2022-04-22T20:27:27.421440+0000 mon.a (mon.0) 9498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:28.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:27 smithi115 bash[46934]: audit 2022-04-22T20:27:27.540423+0000 mon.a (mon.0) 9499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:28 smithi121 bash[36902]: cluster 2022-04-22T20:27:27.494657+0000 mgr.y (mgr.34107) 6208 : cluster [DBG] pgmap v7383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:29.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:28 smithi115 bash[45495]: cluster 2022-04-22T20:27:27.494657+0000 mgr.y (mgr.34107) 6208 : cluster [DBG] pgmap v7383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:29.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:28 smithi115 bash[46934]: cluster 2022-04-22T20:27:27.494657+0000 mgr.y (mgr.34107) 6208 : cluster [DBG] pgmap v7383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:29 smithi121 bash[36902]: audit 2022-04-22T20:27:28.995074+0000 mon.a (mon.0) 9500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:30.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:29 smithi115 bash[45495]: audit 2022-04-22T20:27:28.995074+0000 mon.a (mon.0) 9500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:30.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:29 smithi115 bash[46934]: audit 2022-04-22T20:27:28.995074+0000 mon.a (mon.0) 9500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:31 smithi121 bash[36902]: cluster 2022-04-22T20:27:29.495794+0000 mgr.y (mgr.34107) 6209 : cluster [DBG] pgmap v7384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:31 smithi115 bash[45495]: cluster 2022-04-22T20:27:29.495794+0000 mgr.y (mgr.34107) 6209 : cluster [DBG] pgmap v7384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:31 smithi115 bash[46934]: cluster 2022-04-22T20:27:29.495794+0000 mgr.y (mgr.34107) 6209 : cluster [DBG] pgmap v7384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:32.314Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:33 smithi121 bash[36902]: cluster 2022-04-22T20:27:31.496755+0000 mgr.y (mgr.34107) 6210 : cluster [DBG] pgmap v7385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:33 smithi121 bash[36902]: audit 2022-04-22T20:27:32.425009+0000 mon.a (mon.0) 9501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:33 smithi121 bash[36902]: audit 2022-04-22T20:27:32.543043+0000 mon.a (mon.0) 9502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[45495]: cluster 2022-04-22T20:27:31.496755+0000 mgr.y (mgr.34107) 6210 : cluster [DBG] pgmap v7385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[45495]: audit 2022-04-22T20:27:32.425009+0000 mon.a (mon.0) 9501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[45495]: audit 2022-04-22T20:27:32.543043+0000 mon.a (mon.0) 9502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[46934]: cluster 2022-04-22T20:27:31.496755+0000 mgr.y (mgr.34107) 6210 : cluster [DBG] pgmap v7385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[46934]: audit 2022-04-22T20:27:32.425009+0000 mon.a (mon.0) 9501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:33 smithi115 bash[46934]: audit 2022-04-22T20:27:32.543043+0000 mon.a (mon.0) 9502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:34 smithi121 bash[36902]: audit 2022-04-22T20:27:33.998326+0000 mon.a (mon.0) 9503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:34.564 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:34 smithi115 bash[45495]: audit 2022-04-22T20:27:33.998326+0000 mon.a (mon.0) 9503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:34.564 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:34.194Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:34.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:34 smithi115 bash[46934]: audit 2022-04-22T20:27:33.998326+0000 mon.a (mon.0) 9503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:35 smithi121 bash[36902]: cluster 2022-04-22T20:27:33.497890+0000 mgr.y (mgr.34107) 6211 : cluster [DBG] pgmap v7386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:35 smithi115 bash[45495]: cluster 2022-04-22T20:27:33.497890+0000 mgr.y (mgr.34107) 6211 : cluster [DBG] pgmap v7386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:35 smithi115 bash[46934]: cluster 2022-04-22T20:27:33.497890+0000 mgr.y (mgr.34107) 6211 : cluster [DBG] pgmap v7386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:36 smithi121 bash[36902]: audit 2022-04-22T20:27:35.228213+0000 mon.a (mon.0) 9504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:27:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:36 smithi121 bash[36902]: audit 2022-04-22T20:27:35.231392+0000 mon.a (mon.0) 9505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:27:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:36 smithi121 bash[36902]: audit 2022-04-22T20:27:35.233378+0000 mon.a (mon.0) 9506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:27:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:36 smithi121 bash[36902]: audit 2022-04-22T20:27:35.596481+0000 mon.a (mon.0) 9507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:27:36.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[45495]: audit 2022-04-22T20:27:35.228213+0000 mon.a (mon.0) 9504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:27:36.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[45495]: audit 2022-04-22T20:27:35.231392+0000 mon.a (mon.0) 9505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[45495]: audit 2022-04-22T20:27:35.233378+0000 mon.a (mon.0) 9506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[45495]: audit 2022-04-22T20:27:35.596481+0000 mon.a (mon.0) 9507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[46934]: audit 2022-04-22T20:27:35.228213+0000 mon.a (mon.0) 9504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[46934]: audit 2022-04-22T20:27:35.231392+0000 mon.a (mon.0) 9505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[46934]: audit 2022-04-22T20:27:35.233378+0000 mon.a (mon.0) 9506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:27:36.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:36 smithi115 bash[46934]: audit 2022-04-22T20:27:35.596481+0000 mon.a (mon.0) 9507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:27:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:37 smithi121 bash[36902]: cluster 2022-04-22T20:27:35.498976+0000 mgr.y (mgr.34107) 6212 : cluster [DBG] pgmap v7387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:37 smithi115 bash[45495]: cluster 2022-04-22T20:27:35.498976+0000 mgr.y (mgr.34107) 6212 : cluster [DBG] pgmap v7387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:37 smithi115 bash[46934]: cluster 2022-04-22T20:27:35.498976+0000 mgr.y (mgr.34107) 6212 : cluster [DBG] pgmap v7387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:38 smithi121 bash[36902]: audit 2022-04-22T20:27:37.428570+0000 mon.a (mon.0) 9508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:38 smithi121 bash[36902]: audit 2022-04-22T20:27:37.545558+0000 mon.a (mon.0) 9509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:38 smithi115 bash[45495]: audit 2022-04-22T20:27:37.428570+0000 mon.a (mon.0) 9508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:38 smithi115 bash[45495]: audit 2022-04-22T20:27:37.545558+0000 mon.a (mon.0) 9509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:38 smithi115 bash[46934]: audit 2022-04-22T20:27:37.428570+0000 mon.a (mon.0) 9508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:38 smithi115 bash[46934]: audit 2022-04-22T20:27:37.545558+0000 mon.a (mon.0) 9509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:39 smithi121 bash[36902]: cluster 2022-04-22T20:27:37.500111+0000 mgr.y (mgr.34107) 6213 : cluster [DBG] pgmap v7388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:39 smithi121 bash[36902]: audit 2022-04-22T20:27:39.001377+0000 mon.a (mon.0) 9510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:39 smithi115 bash[45495]: cluster 2022-04-22T20:27:37.500111+0000 mgr.y (mgr.34107) 6213 : cluster [DBG] pgmap v7388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:39 smithi115 bash[45495]: audit 2022-04-22T20:27:39.001377+0000 mon.a (mon.0) 9510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:39 smithi115 bash[46934]: cluster 2022-04-22T20:27:37.500111+0000 mgr.y (mgr.34107) 6213 : cluster [DBG] pgmap v7388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:39 smithi115 bash[46934]: audit 2022-04-22T20:27:39.001377+0000 mon.a (mon.0) 9510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:41 smithi121 bash[36902]: cluster 2022-04-22T20:27:39.501235+0000 mgr.y (mgr.34107) 6214 : cluster [DBG] pgmap v7389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:41 smithi115 bash[45495]: cluster 2022-04-22T20:27:39.501235+0000 mgr.y (mgr.34107) 6214 : cluster [DBG] pgmap v7389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:41 smithi115 bash[46934]: cluster 2022-04-22T20:27:39.501235+0000 mgr.y (mgr.34107) 6214 : cluster [DBG] pgmap v7389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:41.890 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:27:42.242 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:27:42.242 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:27:42.242 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:27:42.242 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:27:42.242 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2h) 5m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:27:42.243 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:27:42.244 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:27:42.652 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:27:42.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:27:42.654 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:27:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:42.314Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:43 smithi121 bash[36902]: cluster 2022-04-22T20:27:41.502280+0000 mgr.y (mgr.34107) 6215 : cluster [DBG] pgmap v7390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:43 smithi121 bash[36902]: audit 2022-04-22T20:27:41.866144+0000 mgr.y (mgr.34107) 6216 : audit [DBG] from='client.51781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:43 smithi121 bash[36902]: audit 2022-04-22T20:27:42.431897+0000 mon.a (mon.0) 9511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:43 smithi121 bash[36902]: audit 2022-04-22T20:27:42.548084+0000 mon.a (mon.0) 9512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:43 smithi121 bash[36902]: audit 2022-04-22T20:27:42.644816+0000 mon.a (mon.0) 9513 : audit [DBG] from='client.? 172.21.15.115:0/1895538024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[45495]: cluster 2022-04-22T20:27:41.502280+0000 mgr.y (mgr.34107) 6215 : cluster [DBG] pgmap v7390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[45495]: audit 2022-04-22T20:27:41.866144+0000 mgr.y (mgr.34107) 6216 : audit [DBG] from='client.51781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[45495]: audit 2022-04-22T20:27:42.431897+0000 mon.a (mon.0) 9511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[45495]: audit 2022-04-22T20:27:42.548084+0000 mon.a (mon.0) 9512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[45495]: audit 2022-04-22T20:27:42.644816+0000 mon.a (mon.0) 9513 : audit [DBG] from='client.? 172.21.15.115:0/1895538024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[46934]: cluster 2022-04-22T20:27:41.502280+0000 mgr.y (mgr.34107) 6215 : cluster [DBG] pgmap v7390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[46934]: audit 2022-04-22T20:27:41.866144+0000 mgr.y (mgr.34107) 6216 : audit [DBG] from='client.51781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[46934]: audit 2022-04-22T20:27:42.431897+0000 mon.a (mon.0) 9511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[46934]: audit 2022-04-22T20:27:42.548084+0000 mon.a (mon.0) 9512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:43 smithi115 bash[46934]: audit 2022-04-22T20:27:42.644816+0000 mon.a (mon.0) 9513 : audit [DBG] from='client.? 172.21.15.115:0/1895538024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:27:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:44 smithi121 bash[36902]: audit 2022-04-22T20:27:42.206923+0000 mgr.y (mgr.34107) 6217 : audit [DBG] from='client.51787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:44 smithi121 bash[36902]: audit 2022-04-22T20:27:44.004396+0000 mon.a (mon.0) 9514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:44 smithi115 bash[45495]: audit 2022-04-22T20:27:42.206923+0000 mgr.y (mgr.34107) 6217 : audit [DBG] from='client.51787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:44 smithi115 bash[45495]: audit 2022-04-22T20:27:44.004396+0000 mon.a (mon.0) 9514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:44 smithi115 bash[46934]: audit 2022-04-22T20:27:42.206923+0000 mgr.y (mgr.34107) 6217 : audit [DBG] from='client.51787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:27:44.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:44 smithi115 bash[46934]: audit 2022-04-22T20:27:44.004396+0000 mon.a (mon.0) 9514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:44.547 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:44.196Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:45 smithi121 bash[36902]: cluster 2022-04-22T20:27:43.503396+0000 mgr.y (mgr.34107) 6218 : cluster [DBG] pgmap v7391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:45 smithi115 bash[45495]: cluster 2022-04-22T20:27:43.503396+0000 mgr.y (mgr.34107) 6218 : cluster [DBG] pgmap v7391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:45 smithi115 bash[46934]: cluster 2022-04-22T20:27:43.503396+0000 mgr.y (mgr.34107) 6218 : cluster [DBG] pgmap v7391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:47 smithi121 bash[36902]: cluster 2022-04-22T20:27:45.504412+0000 mgr.y (mgr.34107) 6219 : cluster [DBG] pgmap v7392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:47 smithi115 bash[45495]: cluster 2022-04-22T20:27:45.504412+0000 mgr.y (mgr.34107) 6219 : cluster [DBG] pgmap v7392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:47 smithi115 bash[46934]: cluster 2022-04-22T20:27:45.504412+0000 mgr.y (mgr.34107) 6219 : cluster [DBG] pgmap v7392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:48 smithi121 bash[36902]: audit 2022-04-22T20:27:47.435161+0000 mon.a (mon.0) 9515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:48 smithi121 bash[36902]: audit 2022-04-22T20:27:47.550074+0000 mon.a (mon.0) 9516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:48 smithi115 bash[45495]: audit 2022-04-22T20:27:47.435161+0000 mon.a (mon.0) 9515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:48 smithi115 bash[45495]: audit 2022-04-22T20:27:47.550074+0000 mon.a (mon.0) 9516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:48 smithi115 bash[46934]: audit 2022-04-22T20:27:47.435161+0000 mon.a (mon.0) 9515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:48 smithi115 bash[46934]: audit 2022-04-22T20:27:47.550074+0000 mon.a (mon.0) 9516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:49 smithi121 bash[36902]: cluster 2022-04-22T20:27:47.505429+0000 mgr.y (mgr.34107) 6220 : cluster [DBG] pgmap v7393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:49 smithi121 bash[36902]: audit 2022-04-22T20:27:49.007412+0000 mon.a (mon.0) 9517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:49 smithi115 bash[45495]: cluster 2022-04-22T20:27:47.505429+0000 mgr.y (mgr.34107) 6220 : cluster [DBG] pgmap v7393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:49 smithi115 bash[45495]: audit 2022-04-22T20:27:49.007412+0000 mon.a (mon.0) 9517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:49 smithi115 bash[46934]: cluster 2022-04-22T20:27:47.505429+0000 mgr.y (mgr.34107) 6220 : cluster [DBG] pgmap v7393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:49 smithi115 bash[46934]: audit 2022-04-22T20:27:49.007412+0000 mon.a (mon.0) 9517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:51 smithi121 bash[36902]: cluster 2022-04-22T20:27:49.506527+0000 mgr.y (mgr.34107) 6221 : cluster [DBG] pgmap v7394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:51 smithi115 bash[45495]: cluster 2022-04-22T20:27:49.506527+0000 mgr.y (mgr.34107) 6221 : cluster [DBG] pgmap v7394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:51 smithi115 bash[46934]: cluster 2022-04-22T20:27:49.506527+0000 mgr.y (mgr.34107) 6221 : cluster [DBG] pgmap v7394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:52.316Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:53 smithi121 bash[36902]: cluster 2022-04-22T20:27:51.507587+0000 mgr.y (mgr.34107) 6222 : cluster [DBG] pgmap v7395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:53 smithi121 bash[36902]: audit 2022-04-22T20:27:52.439025+0000 mon.a (mon.0) 9518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:53 smithi121 bash[36902]: audit 2022-04-22T20:27:52.552387+0000 mon.a (mon.0) 9519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[45495]: cluster 2022-04-22T20:27:51.507587+0000 mgr.y (mgr.34107) 6222 : cluster [DBG] pgmap v7395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[45495]: audit 2022-04-22T20:27:52.439025+0000 mon.a (mon.0) 9518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[45495]: audit 2022-04-22T20:27:52.552387+0000 mon.a (mon.0) 9519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[46934]: cluster 2022-04-22T20:27:51.507587+0000 mgr.y (mgr.34107) 6222 : cluster [DBG] pgmap v7395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[46934]: audit 2022-04-22T20:27:52.439025+0000 mon.a (mon.0) 9518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:53 smithi115 bash[46934]: audit 2022-04-22T20:27:52.552387+0000 mon.a (mon.0) 9519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:54 smithi121 bash[36902]: audit 2022-04-22T20:27:54.010530+0000 mon.a (mon.0) 9520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:54 smithi115 bash[45495]: audit 2022-04-22T20:27:54.010530+0000 mon.a (mon.0) 9520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:54.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:27:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:27:54.196Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:27:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:54 smithi115 bash[46934]: audit 2022-04-22T20:27:54.010530+0000 mon.a (mon.0) 9520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:55 smithi121 bash[36902]: cluster 2022-04-22T20:27:53.508811+0000 mgr.y (mgr.34107) 6223 : cluster [DBG] pgmap v7396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:55 smithi115 bash[45495]: cluster 2022-04-22T20:27:53.508811+0000 mgr.y (mgr.34107) 6223 : cluster [DBG] pgmap v7396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:55 smithi115 bash[46934]: cluster 2022-04-22T20:27:53.508811+0000 mgr.y (mgr.34107) 6223 : cluster [DBG] pgmap v7396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:57 smithi121 bash[36902]: cluster 2022-04-22T20:27:55.509910+0000 mgr.y (mgr.34107) 6224 : cluster [DBG] pgmap v7397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:57 smithi115 bash[45495]: cluster 2022-04-22T20:27:55.509910+0000 mgr.y (mgr.34107) 6224 : cluster [DBG] pgmap v7397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:57 smithi115 bash[46934]: cluster 2022-04-22T20:27:55.509910+0000 mgr.y (mgr.34107) 6224 : cluster [DBG] pgmap v7397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:58 smithi121 bash[36902]: audit 2022-04-22T20:27:57.442439+0000 mon.a (mon.0) 9521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:58 smithi121 bash[36902]: audit 2022-04-22T20:27:57.554621+0000 mon.a (mon.0) 9522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:58 smithi115 bash[45495]: audit 2022-04-22T20:27:57.442439+0000 mon.a (mon.0) 9521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:58 smithi115 bash[45495]: audit 2022-04-22T20:27:57.554621+0000 mon.a (mon.0) 9522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:58 smithi115 bash[46934]: audit 2022-04-22T20:27:57.442439+0000 mon.a (mon.0) 9521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:27:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:58 smithi115 bash[46934]: audit 2022-04-22T20:27:57.554621+0000 mon.a (mon.0) 9522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:27:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:59 smithi121 bash[36902]: cluster 2022-04-22T20:27:57.510925+0000 mgr.y (mgr.34107) 6225 : cluster [DBG] pgmap v7398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:27:59 smithi121 bash[36902]: audit 2022-04-22T20:27:59.013612+0000 mon.a (mon.0) 9523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:59 smithi115 bash[45495]: cluster 2022-04-22T20:27:57.510925+0000 mgr.y (mgr.34107) 6225 : cluster [DBG] pgmap v7398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:27:59 smithi115 bash[45495]: audit 2022-04-22T20:27:59.013612+0000 mon.a (mon.0) 9523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:27:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:59 smithi115 bash[46934]: cluster 2022-04-22T20:27:57.510925+0000 mgr.y (mgr.34107) 6225 : cluster [DBG] pgmap v7398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:27:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:27:59 smithi115 bash[46934]: audit 2022-04-22T20:27:59.013612+0000 mon.a (mon.0) 9523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:01 smithi121 bash[36902]: cluster 2022-04-22T20:27:59.512049+0000 mgr.y (mgr.34107) 6226 : cluster [DBG] pgmap v7399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:01 smithi115 bash[45495]: cluster 2022-04-22T20:27:59.512049+0000 mgr.y (mgr.34107) 6226 : cluster [DBG] pgmap v7399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:01 smithi115 bash[46934]: cluster 2022-04-22T20:27:59.512049+0000 mgr.y (mgr.34107) 6226 : cluster [DBG] pgmap v7399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:02.317Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:03 smithi121 bash[36902]: cluster 2022-04-22T20:28:01.513144+0000 mgr.y (mgr.34107) 6227 : cluster [DBG] pgmap v7400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:03 smithi121 bash[36902]: audit 2022-04-22T20:28:02.446305+0000 mon.a (mon.0) 9524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:03 smithi121 bash[36902]: audit 2022-04-22T20:28:02.557240+0000 mon.a (mon.0) 9525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[45495]: cluster 2022-04-22T20:28:01.513144+0000 mgr.y (mgr.34107) 6227 : cluster [DBG] pgmap v7400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[45495]: audit 2022-04-22T20:28:02.446305+0000 mon.a (mon.0) 9524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[45495]: audit 2022-04-22T20:28:02.557240+0000 mon.a (mon.0) 9525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[46934]: cluster 2022-04-22T20:28:01.513144+0000 mgr.y (mgr.34107) 6227 : cluster [DBG] pgmap v7400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[46934]: audit 2022-04-22T20:28:02.446305+0000 mon.a (mon.0) 9524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:03 smithi115 bash[46934]: audit 2022-04-22T20:28:02.557240+0000 mon.a (mon.0) 9525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:04 smithi121 bash[36902]: audit 2022-04-22T20:28:04.016584+0000 mon.a (mon.0) 9526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:04 smithi115 bash[45495]: audit 2022-04-22T20:28:04.016584+0000 mon.a (mon.0) 9526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:04.193Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:04 smithi115 bash[46934]: audit 2022-04-22T20:28:04.016584+0000 mon.a (mon.0) 9526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:05 smithi121 bash[36902]: cluster 2022-04-22T20:28:03.514294+0000 mgr.y (mgr.34107) 6228 : cluster [DBG] pgmap v7401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:05 smithi115 bash[45495]: cluster 2022-04-22T20:28:03.514294+0000 mgr.y (mgr.34107) 6228 : cluster [DBG] pgmap v7401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:05 smithi115 bash[46934]: cluster 2022-04-22T20:28:03.514294+0000 mgr.y (mgr.34107) 6228 : cluster [DBG] pgmap v7401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:07 smithi121 bash[36902]: cluster 2022-04-22T20:28:05.515451+0000 mgr.y (mgr.34107) 6229 : cluster [DBG] pgmap v7402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:07 smithi115 bash[45495]: cluster 2022-04-22T20:28:05.515451+0000 mgr.y (mgr.34107) 6229 : cluster [DBG] pgmap v7402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:07 smithi115 bash[46934]: cluster 2022-04-22T20:28:05.515451+0000 mgr.y (mgr.34107) 6229 : cluster [DBG] pgmap v7402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:08 smithi121 bash[36902]: audit 2022-04-22T20:28:07.450050+0000 mon.a (mon.0) 9527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:08 smithi121 bash[36902]: audit 2022-04-22T20:28:07.560402+0000 mon.a (mon.0) 9528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:08 smithi115 bash[45495]: audit 2022-04-22T20:28:07.450050+0000 mon.a (mon.0) 9527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:08 smithi115 bash[45495]: audit 2022-04-22T20:28:07.560402+0000 mon.a (mon.0) 9528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:08 smithi115 bash[46934]: audit 2022-04-22T20:28:07.450050+0000 mon.a (mon.0) 9527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:08 smithi115 bash[46934]: audit 2022-04-22T20:28:07.560402+0000 mon.a (mon.0) 9528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:09 smithi121 bash[36902]: cluster 2022-04-22T20:28:07.516490+0000 mgr.y (mgr.34107) 6230 : cluster [DBG] pgmap v7403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:09 smithi121 bash[36902]: audit 2022-04-22T20:28:09.019856+0000 mon.a (mon.0) 9529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:09 smithi115 bash[45495]: cluster 2022-04-22T20:28:07.516490+0000 mgr.y (mgr.34107) 6230 : cluster [DBG] pgmap v7403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:09 smithi115 bash[45495]: audit 2022-04-22T20:28:09.019856+0000 mon.a (mon.0) 9529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:09 smithi115 bash[46934]: cluster 2022-04-22T20:28:07.516490+0000 mgr.y (mgr.34107) 6230 : cluster [DBG] pgmap v7403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:09 smithi115 bash[46934]: audit 2022-04-22T20:28:09.019856+0000 mon.a (mon.0) 9529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:10 smithi121 bash[36902]: cluster 2022-04-22T20:28:09.517631+0000 mgr.y (mgr.34107) 6231 : cluster [DBG] pgmap v7404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:10.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:10 smithi115 bash[45495]: cluster 2022-04-22T20:28:09.517631+0000 mgr.y (mgr.34107) 6231 : cluster [DBG] pgmap v7404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:10 smithi115 bash[46934]: cluster 2022-04-22T20:28:09.517631+0000 mgr.y (mgr.34107) 6231 : cluster [DBG] pgmap v7404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:12.582 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:12.317Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:12 smithi121 bash[36902]: cluster 2022-04-22T20:28:11.518864+0000 mgr.y (mgr.34107) 6232 : cluster [DBG] pgmap v7405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:12 smithi121 bash[36902]: audit 2022-04-22T20:28:12.453671+0000 mon.a (mon.0) 9530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:12 smithi121 bash[36902]: audit 2022-04-22T20:28:12.563152+0000 mon.a (mon.0) 9531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:13.029 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[46934]: cluster 2022-04-22T20:28:11.518864+0000 mgr.y (mgr.34107) 6232 : cluster [DBG] pgmap v7405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[46934]: audit 2022-04-22T20:28:12.453671+0000 mon.a (mon.0) 9530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[46934]: audit 2022-04-22T20:28:12.563152+0000 mon.a (mon.0) 9531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[45495]: cluster 2022-04-22T20:28:11.518864+0000 mgr.y (mgr.34107) 6232 : cluster [DBG] pgmap v7405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[45495]: audit 2022-04-22T20:28:12.453671+0000 mon.a (mon.0) 9530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:13.123 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:12 smithi115 bash[45495]: audit 2022-04-22T20:28:12.563152+0000 mon.a (mon.0) 9531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:13.372 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:28:13.466 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 5m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:28:13.467 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:28:13.467 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:28:13.467 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:28:13.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:28:13.467 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:28:13.468 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:28:13.468 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 6m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2h) 5m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 5m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 6m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 5m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 5m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 5m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:28:13.469 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 5m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:28:13.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 6m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:28:13.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 6m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:28:13.480 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 6m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:28:13.480 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 6m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:28:13.480 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 6m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:28:13.784 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:28:13.784 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:28:13.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:28:13.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:13.790 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:28:13.791 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:28:13.791 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:28:13.791 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:28:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:13 smithi121 bash[36902]: audit 2022-04-22T20:28:13.006198+0000 mgr.y (mgr.34107) 6233 : audit [DBG] from='client.41868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:14.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:13 smithi115 bash[45495]: audit 2022-04-22T20:28:13.006198+0000 mgr.y (mgr.34107) 6233 : audit [DBG] from='client.41868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:14.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:13 smithi115 bash[46934]: audit 2022-04-22T20:28:13.006198+0000 mgr.y (mgr.34107) 6233 : audit [DBG] from='client.41868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:14.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:14.197Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:14 smithi121 bash[36902]: audit 2022-04-22T20:28:13.346348+0000 mgr.y (mgr.34107) 6234 : audit [DBG] from='client.41874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:14 smithi121 bash[36902]: cluster 2022-04-22T20:28:13.519587+0000 mgr.y (mgr.34107) 6235 : cluster [DBG] pgmap v7406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:14 smithi121 bash[36902]: audit 2022-04-22T20:28:13.776501+0000 mon.c (mon.1) 244 : audit [DBG] from='client.? 172.21.15.115:0/4039117287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:14 smithi121 bash[36902]: audit 2022-04-22T20:28:14.022869+0000 mon.a (mon.0) 9532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:15.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[45495]: audit 2022-04-22T20:28:13.346348+0000 mgr.y (mgr.34107) 6234 : audit [DBG] from='client.41874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:15.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[45495]: cluster 2022-04-22T20:28:13.519587+0000 mgr.y (mgr.34107) 6235 : cluster [DBG] pgmap v7406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:15.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[45495]: audit 2022-04-22T20:28:13.776501+0000 mon.c (mon.1) 244 : audit [DBG] from='client.? 172.21.15.115:0/4039117287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:15.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[45495]: audit 2022-04-22T20:28:14.022869+0000 mon.a (mon.0) 9532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[46934]: audit 2022-04-22T20:28:13.346348+0000 mgr.y (mgr.34107) 6234 : audit [DBG] from='client.41874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[46934]: cluster 2022-04-22T20:28:13.519587+0000 mgr.y (mgr.34107) 6235 : cluster [DBG] pgmap v7406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[46934]: audit 2022-04-22T20:28:13.776501+0000 mon.c (mon.1) 244 : audit [DBG] from='client.? 172.21.15.115:0/4039117287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:15.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:14 smithi115 bash[46934]: audit 2022-04-22T20:28:14.022869+0000 mon.a (mon.0) 9532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:15 smithi121 bash[36902]: audit 2022-04-22T20:28:15.370594+0000 mon.a (mon.0) 9533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:28:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:15 smithi121 bash[36902]: audit 2022-04-22T20:28:15.371221+0000 mon.a (mon.0) 9534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:28:16.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:15 smithi115 bash[45495]: audit 2022-04-22T20:28:15.370594+0000 mon.a (mon.0) 9533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:28:16.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:15 smithi115 bash[45495]: audit 2022-04-22T20:28:15.371221+0000 mon.a (mon.0) 9534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:28:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:15 smithi115 bash[46934]: audit 2022-04-22T20:28:15.370594+0000 mon.a (mon.0) 9533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:28:16.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:15 smithi115 bash[46934]: audit 2022-04-22T20:28:15.371221+0000 mon.a (mon.0) 9534 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:28:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:16 smithi121 bash[36902]: cluster 2022-04-22T20:28:15.521264+0000 mgr.y (mgr.34107) 6236 : cluster [DBG] pgmap v7407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:17.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:16 smithi115 bash[45495]: cluster 2022-04-22T20:28:15.521264+0000 mgr.y (mgr.34107) 6236 : cluster [DBG] pgmap v7407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:17.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:16 smithi115 bash[46934]: cluster 2022-04-22T20:28:15.521264+0000 mgr.y (mgr.34107) 6236 : cluster [DBG] pgmap v7407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:17 smithi121 bash[36902]: audit 2022-04-22T20:28:17.457379+0000 mon.a (mon.0) 9535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:17 smithi121 bash[36902]: audit 2022-04-22T20:28:17.565657+0000 mon.a (mon.0) 9536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:17 smithi115 bash[45495]: audit 2022-04-22T20:28:17.457379+0000 mon.a (mon.0) 9535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:18.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:17 smithi115 bash[45495]: audit 2022-04-22T20:28:17.565657+0000 mon.a (mon.0) 9536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:17 smithi115 bash[46934]: audit 2022-04-22T20:28:17.457379+0000 mon.a (mon.0) 9535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:18.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:17 smithi115 bash[46934]: audit 2022-04-22T20:28:17.565657+0000 mon.a (mon.0) 9536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:18 smithi121 bash[36902]: cluster 2022-04-22T20:28:17.522570+0000 mgr.y (mgr.34107) 6237 : cluster [DBG] pgmap v7408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:19.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:18 smithi115 bash[45495]: cluster 2022-04-22T20:28:17.522570+0000 mgr.y (mgr.34107) 6237 : cluster [DBG] pgmap v7408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:19.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:18 smithi115 bash[46934]: cluster 2022-04-22T20:28:17.522570+0000 mgr.y (mgr.34107) 6237 : cluster [DBG] pgmap v7408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:19 smithi121 bash[36902]: audit 2022-04-22T20:28:19.025885+0000 mon.a (mon.0) 9537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:20.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:19 smithi115 bash[45495]: audit 2022-04-22T20:28:19.025885+0000 mon.a (mon.0) 9537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:20.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:19 smithi115 bash[46934]: audit 2022-04-22T20:28:19.025885+0000 mon.a (mon.0) 9537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:21 smithi121 bash[36902]: cluster 2022-04-22T20:28:19.523802+0000 mgr.y (mgr.34107) 6238 : cluster [DBG] pgmap v7409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:21 smithi115 bash[45495]: cluster 2022-04-22T20:28:19.523802+0000 mgr.y (mgr.34107) 6238 : cluster [DBG] pgmap v7409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:21 smithi115 bash[46934]: cluster 2022-04-22T20:28:19.523802+0000 mgr.y (mgr.34107) 6238 : cluster [DBG] pgmap v7409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:22.319Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:23 smithi121 bash[36902]: cluster 2022-04-22T20:28:21.524857+0000 mgr.y (mgr.34107) 6239 : cluster [DBG] pgmap v7410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:23 smithi121 bash[36902]: audit 2022-04-22T20:28:22.461268+0000 mon.a (mon.0) 9538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:23 smithi121 bash[36902]: audit 2022-04-22T20:28:22.568061+0000 mon.a (mon.0) 9539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[45495]: cluster 2022-04-22T20:28:21.524857+0000 mgr.y (mgr.34107) 6239 : cluster [DBG] pgmap v7410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[45495]: audit 2022-04-22T20:28:22.461268+0000 mon.a (mon.0) 9538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[45495]: audit 2022-04-22T20:28:22.568061+0000 mon.a (mon.0) 9539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[46934]: cluster 2022-04-22T20:28:21.524857+0000 mgr.y (mgr.34107) 6239 : cluster [DBG] pgmap v7410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[46934]: audit 2022-04-22T20:28:22.461268+0000 mon.a (mon.0) 9538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:23 smithi115 bash[46934]: audit 2022-04-22T20:28:22.568061+0000 mon.a (mon.0) 9539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:24 smithi121 bash[36902]: audit 2022-04-22T20:28:24.028810+0000 mon.a (mon.0) 9540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:24 smithi115 bash[45495]: audit 2022-04-22T20:28:24.028810+0000 mon.a (mon.0) 9540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:24.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:24.198Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:24 smithi115 bash[46934]: audit 2022-04-22T20:28:24.028810+0000 mon.a (mon.0) 9540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:25 smithi121 bash[36902]: cluster 2022-04-22T20:28:23.525790+0000 mgr.y (mgr.34107) 6240 : cluster [DBG] pgmap v7411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:25 smithi115 bash[45495]: cluster 2022-04-22T20:28:23.525790+0000 mgr.y (mgr.34107) 6240 : cluster [DBG] pgmap v7411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:25 smithi115 bash[46934]: cluster 2022-04-22T20:28:23.525790+0000 mgr.y (mgr.34107) 6240 : cluster [DBG] pgmap v7411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:27 smithi121 bash[36902]: cluster 2022-04-22T20:28:25.526920+0000 mgr.y (mgr.34107) 6241 : cluster [DBG] pgmap v7412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:27 smithi115 bash[45495]: cluster 2022-04-22T20:28:25.526920+0000 mgr.y (mgr.34107) 6241 : cluster [DBG] pgmap v7412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:27 smithi115 bash[46934]: cluster 2022-04-22T20:28:25.526920+0000 mgr.y (mgr.34107) 6241 : cluster [DBG] pgmap v7412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:28 smithi121 bash[36902]: audit 2022-04-22T20:28:27.465173+0000 mon.a (mon.0) 9541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:28 smithi121 bash[36902]: audit 2022-04-22T20:28:27.570780+0000 mon.a (mon.0) 9542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:28 smithi115 bash[45495]: audit 2022-04-22T20:28:27.465173+0000 mon.a (mon.0) 9541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:28 smithi115 bash[45495]: audit 2022-04-22T20:28:27.570780+0000 mon.a (mon.0) 9542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:28 smithi115 bash[46934]: audit 2022-04-22T20:28:27.465173+0000 mon.a (mon.0) 9541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:28.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:28 smithi115 bash[46934]: audit 2022-04-22T20:28:27.570780+0000 mon.a (mon.0) 9542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:29 smithi121 bash[36902]: cluster 2022-04-22T20:28:27.527670+0000 mgr.y (mgr.34107) 6242 : cluster [DBG] pgmap v7413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:29 smithi121 bash[36902]: audit 2022-04-22T20:28:29.032112+0000 mon.a (mon.0) 9543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:29 smithi115 bash[45495]: cluster 2022-04-22T20:28:27.527670+0000 mgr.y (mgr.34107) 6242 : cluster [DBG] pgmap v7413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:29 smithi115 bash[45495]: audit 2022-04-22T20:28:29.032112+0000 mon.a (mon.0) 9543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:29 smithi115 bash[46934]: cluster 2022-04-22T20:28:27.527670+0000 mgr.y (mgr.34107) 6242 : cluster [DBG] pgmap v7413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:29 smithi115 bash[46934]: audit 2022-04-22T20:28:29.032112+0000 mon.a (mon.0) 9543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:31 smithi121 bash[36902]: cluster 2022-04-22T20:28:29.528694+0000 mgr.y (mgr.34107) 6243 : cluster [DBG] pgmap v7414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:31 smithi115 bash[45495]: cluster 2022-04-22T20:28:29.528694+0000 mgr.y (mgr.34107) 6243 : cluster [DBG] pgmap v7414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:31.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:31 smithi115 bash[46934]: cluster 2022-04-22T20:28:29.528694+0000 mgr.y (mgr.34107) 6243 : cluster [DBG] pgmap v7414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:32.317Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:33 smithi121 bash[36902]: cluster 2022-04-22T20:28:31.529475+0000 mgr.y (mgr.34107) 6244 : cluster [DBG] pgmap v7415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:33 smithi121 bash[36902]: audit 2022-04-22T20:28:32.468927+0000 mon.a (mon.0) 9544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:33 smithi121 bash[36902]: audit 2022-04-22T20:28:32.573377+0000 mon.a (mon.0) 9545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[45495]: cluster 2022-04-22T20:28:31.529475+0000 mgr.y (mgr.34107) 6244 : cluster [DBG] pgmap v7415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[45495]: audit 2022-04-22T20:28:32.468927+0000 mon.a (mon.0) 9544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[45495]: audit 2022-04-22T20:28:32.573377+0000 mon.a (mon.0) 9545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[46934]: cluster 2022-04-22T20:28:31.529475+0000 mgr.y (mgr.34107) 6244 : cluster [DBG] pgmap v7415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[46934]: audit 2022-04-22T20:28:32.468927+0000 mon.a (mon.0) 9544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:33 smithi115 bash[46934]: audit 2022-04-22T20:28:32.573377+0000 mon.a (mon.0) 9545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:34 smithi121 bash[36902]: audit 2022-04-22T20:28:34.035236+0000 mon.a (mon.0) 9546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:34.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:34 smithi115 bash[46934]: audit 2022-04-22T20:28:34.035236+0000 mon.a (mon.0) 9546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:34.610 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:34.198Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:34.611 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:34 smithi115 bash[45495]: audit 2022-04-22T20:28:34.035236+0000 mon.a (mon.0) 9546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:35 smithi121 bash[36902]: cluster 2022-04-22T20:28:33.530557+0000 mgr.y (mgr.34107) 6245 : cluster [DBG] pgmap v7416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:35 smithi115 bash[45495]: cluster 2022-04-22T20:28:33.530557+0000 mgr.y (mgr.34107) 6245 : cluster [DBG] pgmap v7416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:35 smithi115 bash[46934]: cluster 2022-04-22T20:28:33.530557+0000 mgr.y (mgr.34107) 6245 : cluster [DBG] pgmap v7416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:36 smithi121 bash[36902]: audit 2022-04-22T20:28:35.605288+0000 mon.a (mon.0) 9547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:28:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:36 smithi121 bash[36902]: audit 2022-04-22T20:28:35.608307+0000 mon.a (mon.0) 9548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:28:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:36 smithi121 bash[36902]: audit 2022-04-22T20:28:35.609850+0000 mon.a (mon.0) 9549 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:28:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:36 smithi121 bash[36902]: audit 2022-04-22T20:28:35.957230+0000 mon.a (mon.0) 9550 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:28:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[45495]: audit 2022-04-22T20:28:35.605288+0000 mon.a (mon.0) 9547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:28:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[45495]: audit 2022-04-22T20:28:35.608307+0000 mon.a (mon.0) 9548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:28:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[45495]: audit 2022-04-22T20:28:35.609850+0000 mon.a (mon.0) 9549 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:28:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[45495]: audit 2022-04-22T20:28:35.957230+0000 mon.a (mon.0) 9550 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:28:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[46934]: audit 2022-04-22T20:28:35.605288+0000 mon.a (mon.0) 9547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:28:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[46934]: audit 2022-04-22T20:28:35.608307+0000 mon.a (mon.0) 9548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:28:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[46934]: audit 2022-04-22T20:28:35.609850+0000 mon.a (mon.0) 9549 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:28:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:36 smithi115 bash[46934]: audit 2022-04-22T20:28:35.957230+0000 mon.a (mon.0) 9550 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:28:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:37 smithi121 bash[36902]: cluster 2022-04-22T20:28:35.531673+0000 mgr.y (mgr.34107) 6246 : cluster [DBG] pgmap v7417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:37 smithi115 bash[45495]: cluster 2022-04-22T20:28:35.531673+0000 mgr.y (mgr.34107) 6246 : cluster [DBG] pgmap v7417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:37 smithi115 bash[46934]: cluster 2022-04-22T20:28:35.531673+0000 mgr.y (mgr.34107) 6246 : cluster [DBG] pgmap v7417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:38 smithi121 bash[36902]: audit 2022-04-22T20:28:37.472666+0000 mon.a (mon.0) 9551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:38 smithi121 bash[36902]: audit 2022-04-22T20:28:37.576182+0000 mon.a (mon.0) 9552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:38 smithi115 bash[45495]: audit 2022-04-22T20:28:37.472666+0000 mon.a (mon.0) 9551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:38 smithi115 bash[45495]: audit 2022-04-22T20:28:37.576182+0000 mon.a (mon.0) 9552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:38 smithi115 bash[46934]: audit 2022-04-22T20:28:37.472666+0000 mon.a (mon.0) 9551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:38 smithi115 bash[46934]: audit 2022-04-22T20:28:37.576182+0000 mon.a (mon.0) 9552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:39 smithi121 bash[36902]: cluster 2022-04-22T20:28:37.532754+0000 mgr.y (mgr.34107) 6247 : cluster [DBG] pgmap v7418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:39 smithi121 bash[36902]: audit 2022-04-22T20:28:39.038331+0000 mon.a (mon.0) 9553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:39 smithi115 bash[45495]: cluster 2022-04-22T20:28:37.532754+0000 mgr.y (mgr.34107) 6247 : cluster [DBG] pgmap v7418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:39 smithi115 bash[45495]: audit 2022-04-22T20:28:39.038331+0000 mon.a (mon.0) 9553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:39 smithi115 bash[46934]: cluster 2022-04-22T20:28:37.532754+0000 mgr.y (mgr.34107) 6247 : cluster [DBG] pgmap v7418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:39 smithi115 bash[46934]: audit 2022-04-22T20:28:39.038331+0000 mon.a (mon.0) 9553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:41 smithi121 bash[36902]: cluster 2022-04-22T20:28:39.533582+0000 mgr.y (mgr.34107) 6248 : cluster [DBG] pgmap v7419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:41 smithi115 bash[45495]: cluster 2022-04-22T20:28:39.533582+0000 mgr.y (mgr.34107) 6248 : cluster [DBG] pgmap v7419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:41 smithi115 bash[46934]: cluster 2022-04-22T20:28:39.533582+0000 mgr.y (mgr.34107) 6248 : cluster [DBG] pgmap v7419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:42.318Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:43 smithi121 bash[36902]: cluster 2022-04-22T20:28:41.534670+0000 mgr.y (mgr.34107) 6249 : cluster [DBG] pgmap v7420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:43 smithi121 bash[36902]: audit 2022-04-22T20:28:42.476159+0000 mon.a (mon.0) 9554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:43 smithi121 bash[36902]: audit 2022-04-22T20:28:42.579202+0000 mon.a (mon.0) 9555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[46934]: cluster 2022-04-22T20:28:41.534670+0000 mgr.y (mgr.34107) 6249 : cluster [DBG] pgmap v7420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[46934]: audit 2022-04-22T20:28:42.476159+0000 mon.a (mon.0) 9554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[46934]: audit 2022-04-22T20:28:42.579202+0000 mon.a (mon.0) 9555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[45495]: cluster 2022-04-22T20:28:41.534670+0000 mgr.y (mgr.34107) 6249 : cluster [DBG] pgmap v7420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[45495]: audit 2022-04-22T20:28:42.476159+0000 mon.a (mon.0) 9554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:43 smithi115 bash[45495]: audit 2022-04-22T20:28:42.579202+0000 mon.a (mon.0) 9555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:44.160 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:28:44.506 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (2h) 7m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:28:44.507 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:28:44.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:28:44.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:28:44.509 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:28:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:44 smithi121 bash[36902]: audit 2022-04-22T20:28:44.041355+0000 mon.a (mon.0) 9556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:44 smithi115 bash[45495]: audit 2022-04-22T20:28:44.041355+0000 mon.a (mon.0) 9556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:44 smithi115 bash[46934]: audit 2022-04-22T20:28:44.041355+0000 mon.a (mon.0) 9556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:44.197Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:44.910 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:28:44.910 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:28:44.910 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:28:44.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:44.910 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:28:44.911 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:28:44.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:44.911 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:28:44.911 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:28:44.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:28:44.912 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:28:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:45 smithi121 bash[36902]: cluster 2022-04-22T20:28:43.535867+0000 mgr.y (mgr.34107) 6250 : cluster [DBG] pgmap v7421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:45 smithi121 bash[36902]: audit 2022-04-22T20:28:44.137963+0000 mgr.y (mgr.34107) 6251 : audit [DBG] from='client.41886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:45 smithi121 bash[36902]: audit 2022-04-22T20:28:44.900187+0000 mon.a (mon.0) 9557 : audit [DBG] from='client.? 172.21.15.115:0/1205473826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[45495]: cluster 2022-04-22T20:28:43.535867+0000 mgr.y (mgr.34107) 6250 : cluster [DBG] pgmap v7421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[45495]: audit 2022-04-22T20:28:44.137963+0000 mgr.y (mgr.34107) 6251 : audit [DBG] from='client.41886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[45495]: audit 2022-04-22T20:28:44.900187+0000 mon.a (mon.0) 9557 : audit [DBG] from='client.? 172.21.15.115:0/1205473826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[46934]: cluster 2022-04-22T20:28:43.535867+0000 mgr.y (mgr.34107) 6250 : cluster [DBG] pgmap v7421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[46934]: audit 2022-04-22T20:28:44.137963+0000 mgr.y (mgr.34107) 6251 : audit [DBG] from='client.41886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:45 smithi115 bash[46934]: audit 2022-04-22T20:28:44.900187+0000 mon.a (mon.0) 9557 : audit [DBG] from='client.? 172.21.15.115:0/1205473826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:28:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:46 smithi121 bash[36902]: audit 2022-04-22T20:28:44.478107+0000 mgr.y (mgr.34107) 6252 : audit [DBG] from='client.41892 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:46 smithi115 bash[45495]: audit 2022-04-22T20:28:44.478107+0000 mgr.y (mgr.34107) 6252 : audit [DBG] from='client.41892 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:46 smithi115 bash[46934]: audit 2022-04-22T20:28:44.478107+0000 mgr.y (mgr.34107) 6252 : audit [DBG] from='client.41892 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:28:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:47 smithi121 bash[36902]: cluster 2022-04-22T20:28:45.536888+0000 mgr.y (mgr.34107) 6253 : cluster [DBG] pgmap v7422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:47 smithi115 bash[45495]: cluster 2022-04-22T20:28:45.536888+0000 mgr.y (mgr.34107) 6253 : cluster [DBG] pgmap v7422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:47 smithi115 bash[46934]: cluster 2022-04-22T20:28:45.536888+0000 mgr.y (mgr.34107) 6253 : cluster [DBG] pgmap v7422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:48 smithi121 bash[36902]: audit 2022-04-22T20:28:47.479931+0000 mon.a (mon.0) 9558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:48 smithi121 bash[36902]: audit 2022-04-22T20:28:47.581881+0000 mon.a (mon.0) 9559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:48 smithi115 bash[45495]: audit 2022-04-22T20:28:47.479931+0000 mon.a (mon.0) 9558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:48 smithi115 bash[45495]: audit 2022-04-22T20:28:47.581881+0000 mon.a (mon.0) 9559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:48 smithi115 bash[46934]: audit 2022-04-22T20:28:47.479931+0000 mon.a (mon.0) 9558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:48 smithi115 bash[46934]: audit 2022-04-22T20:28:47.581881+0000 mon.a (mon.0) 9559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:49 smithi121 bash[36902]: cluster 2022-04-22T20:28:47.537738+0000 mgr.y (mgr.34107) 6254 : cluster [DBG] pgmap v7423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:49 smithi121 bash[36902]: audit 2022-04-22T20:28:49.044649+0000 mon.a (mon.0) 9560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:49 smithi115 bash[45495]: cluster 2022-04-22T20:28:47.537738+0000 mgr.y (mgr.34107) 6254 : cluster [DBG] pgmap v7423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:49 smithi115 bash[45495]: audit 2022-04-22T20:28:49.044649+0000 mon.a (mon.0) 9560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:49 smithi115 bash[46934]: cluster 2022-04-22T20:28:47.537738+0000 mgr.y (mgr.34107) 6254 : cluster [DBG] pgmap v7423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:49 smithi115 bash[46934]: audit 2022-04-22T20:28:49.044649+0000 mon.a (mon.0) 9560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:51 smithi121 bash[36902]: cluster 2022-04-22T20:28:49.538937+0000 mgr.y (mgr.34107) 6255 : cluster [DBG] pgmap v7424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:51 smithi115 bash[45495]: cluster 2022-04-22T20:28:49.538937+0000 mgr.y (mgr.34107) 6255 : cluster [DBG] pgmap v7424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:51 smithi115 bash[46934]: cluster 2022-04-22T20:28:49.538937+0000 mgr.y (mgr.34107) 6255 : cluster [DBG] pgmap v7424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:52.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:53 smithi121 bash[36902]: cluster 2022-04-22T20:28:51.540029+0000 mgr.y (mgr.34107) 6256 : cluster [DBG] pgmap v7425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:53 smithi121 bash[36902]: audit 2022-04-22T20:28:52.483558+0000 mon.a (mon.0) 9561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:53.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:53 smithi121 bash[36902]: audit 2022-04-22T20:28:52.584655+0000 mon.a (mon.0) 9562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[45495]: cluster 2022-04-22T20:28:51.540029+0000 mgr.y (mgr.34107) 6256 : cluster [DBG] pgmap v7425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[45495]: audit 2022-04-22T20:28:52.483558+0000 mon.a (mon.0) 9561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[45495]: audit 2022-04-22T20:28:52.584655+0000 mon.a (mon.0) 9562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[46934]: cluster 2022-04-22T20:28:51.540029+0000 mgr.y (mgr.34107) 6256 : cluster [DBG] pgmap v7425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[46934]: audit 2022-04-22T20:28:52.483558+0000 mon.a (mon.0) 9561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:53.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:53 smithi115 bash[46934]: audit 2022-04-22T20:28:52.584655+0000 mon.a (mon.0) 9562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:54 smithi121 bash[36902]: audit 2022-04-22T20:28:54.047911+0000 mon.a (mon.0) 9563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:54 smithi115 bash[45495]: audit 2022-04-22T20:28:54.047911+0000 mon.a (mon.0) 9563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:28:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:28:54.200Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:28:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:54 smithi115 bash[46934]: audit 2022-04-22T20:28:54.047911+0000 mon.a (mon.0) 9563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:55 smithi121 bash[36902]: cluster 2022-04-22T20:28:53.541182+0000 mgr.y (mgr.34107) 6257 : cluster [DBG] pgmap v7426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:55 smithi115 bash[45495]: cluster 2022-04-22T20:28:53.541182+0000 mgr.y (mgr.34107) 6257 : cluster [DBG] pgmap v7426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:55 smithi115 bash[46934]: cluster 2022-04-22T20:28:53.541182+0000 mgr.y (mgr.34107) 6257 : cluster [DBG] pgmap v7426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:57 smithi121 bash[36902]: cluster 2022-04-22T20:28:55.542327+0000 mgr.y (mgr.34107) 6258 : cluster [DBG] pgmap v7427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:57 smithi115 bash[45495]: cluster 2022-04-22T20:28:55.542327+0000 mgr.y (mgr.34107) 6258 : cluster [DBG] pgmap v7427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:57 smithi115 bash[46934]: cluster 2022-04-22T20:28:55.542327+0000 mgr.y (mgr.34107) 6258 : cluster [DBG] pgmap v7427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:58 smithi121 bash[36902]: audit 2022-04-22T20:28:57.486850+0000 mon.a (mon.0) 9564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:58 smithi121 bash[36902]: audit 2022-04-22T20:28:57.587588+0000 mon.a (mon.0) 9565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:58 smithi115 bash[45495]: audit 2022-04-22T20:28:57.486850+0000 mon.a (mon.0) 9564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:58 smithi115 bash[45495]: audit 2022-04-22T20:28:57.587588+0000 mon.a (mon.0) 9565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:58 smithi115 bash[46934]: audit 2022-04-22T20:28:57.486850+0000 mon.a (mon.0) 9564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:28:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:58 smithi115 bash[46934]: audit 2022-04-22T20:28:57.587588+0000 mon.a (mon.0) 9565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:28:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:59 smithi121 bash[36902]: cluster 2022-04-22T20:28:57.543424+0000 mgr.y (mgr.34107) 6259 : cluster [DBG] pgmap v7428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:28:59 smithi121 bash[36902]: audit 2022-04-22T20:28:59.051169+0000 mon.a (mon.0) 9566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:59 smithi115 bash[45495]: cluster 2022-04-22T20:28:57.543424+0000 mgr.y (mgr.34107) 6259 : cluster [DBG] pgmap v7428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:28:59 smithi115 bash[45495]: audit 2022-04-22T20:28:59.051169+0000 mon.a (mon.0) 9566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:28:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:59 smithi115 bash[46934]: cluster 2022-04-22T20:28:57.543424+0000 mgr.y (mgr.34107) 6259 : cluster [DBG] pgmap v7428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:28:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:28:59 smithi115 bash[46934]: audit 2022-04-22T20:28:59.051169+0000 mon.a (mon.0) 9566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:00 smithi121 bash[36902]: cluster 2022-04-22T20:28:59.544644+0000 mgr.y (mgr.34107) 6260 : cluster [DBG] pgmap v7429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:00 smithi115 bash[46934]: cluster 2022-04-22T20:28:59.544644+0000 mgr.y (mgr.34107) 6260 : cluster [DBG] pgmap v7429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:00.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:00 smithi115 bash[45495]: cluster 2022-04-22T20:28:59.544644+0000 mgr.y (mgr.34107) 6260 : cluster [DBG] pgmap v7429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:02.609 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:02.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:02 smithi121 bash[36902]: cluster 2022-04-22T20:29:01.545841+0000 mgr.y (mgr.34107) 6261 : cluster [DBG] pgmap v7430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:02 smithi121 bash[36902]: audit 2022-04-22T20:29:02.490276+0000 mon.a (mon.0) 9567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:02 smithi121 bash[36902]: audit 2022-04-22T20:29:02.590854+0000 mon.a (mon.0) 9568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[45495]: cluster 2022-04-22T20:29:01.545841+0000 mgr.y (mgr.34107) 6261 : cluster [DBG] pgmap v7430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[45495]: audit 2022-04-22T20:29:02.490276+0000 mon.a (mon.0) 9567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[45495]: audit 2022-04-22T20:29:02.590854+0000 mon.a (mon.0) 9568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[46934]: cluster 2022-04-22T20:29:01.545841+0000 mgr.y (mgr.34107) 6261 : cluster [DBG] pgmap v7430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[46934]: audit 2022-04-22T20:29:02.490276+0000 mon.a (mon.0) 9567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:02 smithi115 bash[46934]: audit 2022-04-22T20:29:02.590854+0000 mon.a (mon.0) 9568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:04.201Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:04 smithi121 bash[36902]: cluster 2022-04-22T20:29:03.547038+0000 mgr.y (mgr.34107) 6262 : cluster [DBG] pgmap v7431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:04 smithi121 bash[36902]: audit 2022-04-22T20:29:04.054355+0000 mon.a (mon.0) 9569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:04 smithi115 bash[45495]: cluster 2022-04-22T20:29:03.547038+0000 mgr.y (mgr.34107) 6262 : cluster [DBG] pgmap v7431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:04 smithi115 bash[45495]: audit 2022-04-22T20:29:04.054355+0000 mon.a (mon.0) 9569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:04 smithi115 bash[46934]: cluster 2022-04-22T20:29:03.547038+0000 mgr.y (mgr.34107) 6262 : cluster [DBG] pgmap v7431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:04 smithi115 bash[46934]: audit 2022-04-22T20:29:04.054355+0000 mon.a (mon.0) 9569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:06 smithi121 bash[36902]: cluster 2022-04-22T20:29:05.548169+0000 mgr.y (mgr.34107) 6263 : cluster [DBG] pgmap v7432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:06 smithi115 bash[45495]: cluster 2022-04-22T20:29:05.548169+0000 mgr.y (mgr.34107) 6263 : cluster [DBG] pgmap v7432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:06 smithi115 bash[46934]: cluster 2022-04-22T20:29:05.548169+0000 mgr.y (mgr.34107) 6263 : cluster [DBG] pgmap v7432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:07 smithi121 bash[36902]: audit 2022-04-22T20:29:07.493814+0000 mon.a (mon.0) 9570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:07 smithi121 bash[36902]: audit 2022-04-22T20:29:07.594204+0000 mon.a (mon.0) 9571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:07 smithi115 bash[45495]: audit 2022-04-22T20:29:07.493814+0000 mon.a (mon.0) 9570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:07 smithi115 bash[45495]: audit 2022-04-22T20:29:07.594204+0000 mon.a (mon.0) 9571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:07 smithi115 bash[46934]: audit 2022-04-22T20:29:07.493814+0000 mon.a (mon.0) 9570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:07 smithi115 bash[46934]: audit 2022-04-22T20:29:07.594204+0000 mon.a (mon.0) 9571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:08 smithi121 bash[36902]: cluster 2022-04-22T20:29:07.549302+0000 mgr.y (mgr.34107) 6264 : cluster [DBG] pgmap v7433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:08 smithi115 bash[45495]: cluster 2022-04-22T20:29:07.549302+0000 mgr.y (mgr.34107) 6264 : cluster [DBG] pgmap v7433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:08 smithi115 bash[46934]: cluster 2022-04-22T20:29:07.549302+0000 mgr.y (mgr.34107) 6264 : cluster [DBG] pgmap v7433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:09 smithi121 bash[36902]: audit 2022-04-22T20:29:09.057519+0000 mon.a (mon.0) 9572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:10.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:09 smithi115 bash[45495]: audit 2022-04-22T20:29:09.057519+0000 mon.a (mon.0) 9572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:09 smithi115 bash[46934]: audit 2022-04-22T20:29:09.057519+0000 mon.a (mon.0) 9572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:11 smithi121 bash[36902]: cluster 2022-04-22T20:29:09.550397+0000 mgr.y (mgr.34107) 6265 : cluster [DBG] pgmap v7434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:11 smithi115 bash[45495]: cluster 2022-04-22T20:29:09.550397+0000 mgr.y (mgr.34107) 6265 : cluster [DBG] pgmap v7434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:11 smithi115 bash[46934]: cluster 2022-04-22T20:29:09.550397+0000 mgr.y (mgr.34107) 6265 : cluster [DBG] pgmap v7434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:12.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:12.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:13 smithi121 bash[36902]: cluster 2022-04-22T20:29:11.551580+0000 mgr.y (mgr.34107) 6266 : cluster [DBG] pgmap v7435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:13 smithi121 bash[36902]: audit 2022-04-22T20:29:12.497009+0000 mon.a (mon.0) 9573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:13 smithi121 bash[36902]: audit 2022-04-22T20:29:12.597194+0000 mon.a (mon.0) 9574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[45495]: cluster 2022-04-22T20:29:11.551580+0000 mgr.y (mgr.34107) 6266 : cluster [DBG] pgmap v7435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[45495]: audit 2022-04-22T20:29:12.497009+0000 mon.a (mon.0) 9573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[45495]: audit 2022-04-22T20:29:12.597194+0000 mon.a (mon.0) 9574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[46934]: cluster 2022-04-22T20:29:11.551580+0000 mgr.y (mgr.34107) 6266 : cluster [DBG] pgmap v7435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[46934]: audit 2022-04-22T20:29:12.497009+0000 mon.a (mon.0) 9573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:13 smithi115 bash[46934]: audit 2022-04-22T20:29:12.597194+0000 mon.a (mon.0) 9574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:14 smithi121 bash[36902]: audit 2022-04-22T20:29:14.060702+0000 mon.a (mon.0) 9575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:14 smithi115 bash[45495]: audit 2022-04-22T20:29:14.060702+0000 mon.a (mon.0) 9575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:14.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:14.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:14 smithi115 bash[46934]: audit 2022-04-22T20:29:14.060702+0000 mon.a (mon.0) 9575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:15.283 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:29:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:15 smithi121 bash[36902]: cluster 2022-04-22T20:29:13.552690+0000 mgr.y (mgr.34107) 6267 : cluster [DBG] pgmap v7436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:15 smithi115 bash[46934]: cluster 2022-04-22T20:29:13.552690+0000 mgr.y (mgr.34107) 6267 : cluster [DBG] pgmap v7436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:15 smithi115 bash[45495]: cluster 2022-04-22T20:29:13.552690+0000 mgr.y (mgr.34107) 6267 : cluster [DBG] pgmap v7436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:15.619 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:29:15.619 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 6m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:29:15.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 6m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 7m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 6m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 6m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 6m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 6m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 7m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 7m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:29:15.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 7m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:29:15.622 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 7m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:29:15.622 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 7m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:29:16.023 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:29:16.023 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:29:16.023 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:29:16.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:29:16.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:29:16.025 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:29:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:16 smithi121 bash[36902]: audit 2022-04-22T20:29:15.380738+0000 mon.a (mon.0) 9576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:29:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:16 smithi121 bash[36902]: audit 2022-04-22T20:29:15.380962+0000 mon.a (mon.0) 9577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:29:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:16 smithi121 bash[36902]: audit 2022-04-22T20:29:16.015002+0000 mon.a (mon.0) 9578 : audit [DBG] from='client.? 172.21.15.115:0/2329599912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[46934]: audit 2022-04-22T20:29:15.380738+0000 mon.a (mon.0) 9576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:29:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[46934]: audit 2022-04-22T20:29:15.380962+0000 mon.a (mon.0) 9577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:29:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[46934]: audit 2022-04-22T20:29:16.015002+0000 mon.a (mon.0) 9578 : audit [DBG] from='client.? 172.21.15.115:0/2329599912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[45495]: audit 2022-04-22T20:29:15.380738+0000 mon.a (mon.0) 9576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:29:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[45495]: audit 2022-04-22T20:29:15.380962+0000 mon.a (mon.0) 9577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:29:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:16 smithi115 bash[45495]: audit 2022-04-22T20:29:16.015002+0000 mon.a (mon.0) 9578 : audit [DBG] from='client.? 172.21.15.115:0/2329599912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:17 smithi121 bash[36902]: audit 2022-04-22T20:29:15.260482+0000 mgr.y (mgr.34107) 6268 : audit [DBG] from='client.51835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:17 smithi121 bash[36902]: cluster 2022-04-22T20:29:15.553747+0000 mgr.y (mgr.34107) 6269 : cluster [DBG] pgmap v7437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:17 smithi121 bash[36902]: audit 2022-04-22T20:29:15.590362+0000 mgr.y (mgr.34107) 6270 : audit [DBG] from='client.51841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[45495]: audit 2022-04-22T20:29:15.260482+0000 mgr.y (mgr.34107) 6268 : audit [DBG] from='client.51835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[45495]: cluster 2022-04-22T20:29:15.553747+0000 mgr.y (mgr.34107) 6269 : cluster [DBG] pgmap v7437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:17.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[45495]: audit 2022-04-22T20:29:15.590362+0000 mgr.y (mgr.34107) 6270 : audit [DBG] from='client.51841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:17.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[46934]: audit 2022-04-22T20:29:15.260482+0000 mgr.y (mgr.34107) 6268 : audit [DBG] from='client.51835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[46934]: cluster 2022-04-22T20:29:15.553747+0000 mgr.y (mgr.34107) 6269 : cluster [DBG] pgmap v7437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:17.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:17 smithi115 bash[46934]: audit 2022-04-22T20:29:15.590362+0000 mgr.y (mgr.34107) 6270 : audit [DBG] from='client.51841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:18 smithi121 bash[36902]: audit 2022-04-22T20:29:17.500551+0000 mon.a (mon.0) 9579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:18 smithi121 bash[36902]: audit 2022-04-22T20:29:17.600490+0000 mon.a (mon.0) 9580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:18 smithi115 bash[45495]: audit 2022-04-22T20:29:17.500551+0000 mon.a (mon.0) 9579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:18 smithi115 bash[45495]: audit 2022-04-22T20:29:17.600490+0000 mon.a (mon.0) 9580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:18 smithi115 bash[46934]: audit 2022-04-22T20:29:17.500551+0000 mon.a (mon.0) 9579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:18 smithi115 bash[46934]: audit 2022-04-22T20:29:17.600490+0000 mon.a (mon.0) 9580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:19 smithi121 bash[36902]: cluster 2022-04-22T20:29:17.554792+0000 mgr.y (mgr.34107) 6271 : cluster [DBG] pgmap v7438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:19 smithi121 bash[36902]: audit 2022-04-22T20:29:19.063877+0000 mon.a (mon.0) 9581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:19 smithi115 bash[45495]: cluster 2022-04-22T20:29:17.554792+0000 mgr.y (mgr.34107) 6271 : cluster [DBG] pgmap v7438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:19 smithi115 bash[45495]: audit 2022-04-22T20:29:19.063877+0000 mon.a (mon.0) 9581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:19 smithi115 bash[46934]: cluster 2022-04-22T20:29:17.554792+0000 mgr.y (mgr.34107) 6271 : cluster [DBG] pgmap v7438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:19 smithi115 bash[46934]: audit 2022-04-22T20:29:19.063877+0000 mon.a (mon.0) 9581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:21 smithi121 bash[36902]: cluster 2022-04-22T20:29:19.555919+0000 mgr.y (mgr.34107) 6272 : cluster [DBG] pgmap v7439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:21 smithi115 bash[45495]: cluster 2022-04-22T20:29:19.555919+0000 mgr.y (mgr.34107) 6272 : cluster [DBG] pgmap v7439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:21 smithi115 bash[46934]: cluster 2022-04-22T20:29:19.555919+0000 mgr.y (mgr.34107) 6272 : cluster [DBG] pgmap v7439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:22.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:23 smithi121 bash[36902]: cluster 2022-04-22T20:29:21.557022+0000 mgr.y (mgr.34107) 6273 : cluster [DBG] pgmap v7440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:23 smithi121 bash[36902]: audit 2022-04-22T20:29:22.504167+0000 mon.a (mon.0) 9582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:23 smithi121 bash[36902]: audit 2022-04-22T20:29:22.603395+0000 mon.a (mon.0) 9583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[45495]: cluster 2022-04-22T20:29:21.557022+0000 mgr.y (mgr.34107) 6273 : cluster [DBG] pgmap v7440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[45495]: audit 2022-04-22T20:29:22.504167+0000 mon.a (mon.0) 9582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[45495]: audit 2022-04-22T20:29:22.603395+0000 mon.a (mon.0) 9583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[46934]: cluster 2022-04-22T20:29:21.557022+0000 mgr.y (mgr.34107) 6273 : cluster [DBG] pgmap v7440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[46934]: audit 2022-04-22T20:29:22.504167+0000 mon.a (mon.0) 9582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:23 smithi115 bash[46934]: audit 2022-04-22T20:29:22.603395+0000 mon.a (mon.0) 9583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:24 smithi121 bash[36902]: audit 2022-04-22T20:29:24.066872+0000 mon.a (mon.0) 9584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:24.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:24.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:24.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:24 smithi115 bash[45495]: audit 2022-04-22T20:29:24.066872+0000 mon.a (mon.0) 9584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:24.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:24 smithi115 bash[46934]: audit 2022-04-22T20:29:24.066872+0000 mon.a (mon.0) 9584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:25 smithi121 bash[36902]: cluster 2022-04-22T20:29:23.558153+0000 mgr.y (mgr.34107) 6274 : cluster [DBG] pgmap v7441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:25 smithi115 bash[45495]: cluster 2022-04-22T20:29:23.558153+0000 mgr.y (mgr.34107) 6274 : cluster [DBG] pgmap v7441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:25 smithi115 bash[46934]: cluster 2022-04-22T20:29:23.558153+0000 mgr.y (mgr.34107) 6274 : cluster [DBG] pgmap v7441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:27 smithi121 bash[36902]: cluster 2022-04-22T20:29:25.559222+0000 mgr.y (mgr.34107) 6275 : cluster [DBG] pgmap v7442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:27 smithi115 bash[45495]: cluster 2022-04-22T20:29:25.559222+0000 mgr.y (mgr.34107) 6275 : cluster [DBG] pgmap v7442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:27 smithi115 bash[46934]: cluster 2022-04-22T20:29:25.559222+0000 mgr.y (mgr.34107) 6275 : cluster [DBG] pgmap v7442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:28 smithi121 bash[36902]: audit 2022-04-22T20:29:27.507655+0000 mon.a (mon.0) 9585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:28 smithi121 bash[36902]: audit 2022-04-22T20:29:27.606287+0000 mon.a (mon.0) 9586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:28 smithi115 bash[45495]: audit 2022-04-22T20:29:27.507655+0000 mon.a (mon.0) 9585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:28 smithi115 bash[45495]: audit 2022-04-22T20:29:27.606287+0000 mon.a (mon.0) 9586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:28 smithi115 bash[46934]: audit 2022-04-22T20:29:27.507655+0000 mon.a (mon.0) 9585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:28 smithi115 bash[46934]: audit 2022-04-22T20:29:27.606287+0000 mon.a (mon.0) 9586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:29 smithi121 bash[36902]: cluster 2022-04-22T20:29:27.560397+0000 mgr.y (mgr.34107) 6276 : cluster [DBG] pgmap v7443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:29 smithi121 bash[36902]: audit 2022-04-22T20:29:29.069883+0000 mon.a (mon.0) 9587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:29 smithi115 bash[45495]: cluster 2022-04-22T20:29:27.560397+0000 mgr.y (mgr.34107) 6276 : cluster [DBG] pgmap v7443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:29 smithi115 bash[45495]: audit 2022-04-22T20:29:29.069883+0000 mon.a (mon.0) 9587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:29 smithi115 bash[46934]: cluster 2022-04-22T20:29:27.560397+0000 mgr.y (mgr.34107) 6276 : cluster [DBG] pgmap v7443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:29 smithi115 bash[46934]: audit 2022-04-22T20:29:29.069883+0000 mon.a (mon.0) 9587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:31 smithi121 bash[36902]: cluster 2022-04-22T20:29:29.561522+0000 mgr.y (mgr.34107) 6277 : cluster [DBG] pgmap v7444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:31 smithi115 bash[45495]: cluster 2022-04-22T20:29:29.561522+0000 mgr.y (mgr.34107) 6277 : cluster [DBG] pgmap v7444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:31.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:31 smithi115 bash[46934]: cluster 2022-04-22T20:29:29.561522+0000 mgr.y (mgr.34107) 6277 : cluster [DBG] pgmap v7444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:32.322Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:33 smithi121 bash[36902]: cluster 2022-04-22T20:29:31.562618+0000 mgr.y (mgr.34107) 6278 : cluster [DBG] pgmap v7445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:33 smithi121 bash[36902]: audit 2022-04-22T20:29:32.511361+0000 mon.a (mon.0) 9588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:33 smithi121 bash[36902]: audit 2022-04-22T20:29:32.609281+0000 mon.a (mon.0) 9589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[45495]: cluster 2022-04-22T20:29:31.562618+0000 mgr.y (mgr.34107) 6278 : cluster [DBG] pgmap v7445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[45495]: audit 2022-04-22T20:29:32.511361+0000 mon.a (mon.0) 9588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[45495]: audit 2022-04-22T20:29:32.609281+0000 mon.a (mon.0) 9589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[46934]: cluster 2022-04-22T20:29:31.562618+0000 mgr.y (mgr.34107) 6278 : cluster [DBG] pgmap v7445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[46934]: audit 2022-04-22T20:29:32.511361+0000 mon.a (mon.0) 9588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:33 smithi115 bash[46934]: audit 2022-04-22T20:29:32.609281+0000 mon.a (mon.0) 9589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:34 smithi121 bash[36902]: audit 2022-04-22T20:29:34.072716+0000 mon.a (mon.0) 9590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:34.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:34.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:34 smithi115 bash[45495]: audit 2022-04-22T20:29:34.072716+0000 mon.a (mon.0) 9590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:34 smithi115 bash[46934]: audit 2022-04-22T20:29:34.072716+0000 mon.a (mon.0) 9590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:35 smithi121 bash[36902]: cluster 2022-04-22T20:29:33.563759+0000 mgr.y (mgr.34107) 6279 : cluster [DBG] pgmap v7446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:35 smithi115 bash[45495]: cluster 2022-04-22T20:29:33.563759+0000 mgr.y (mgr.34107) 6279 : cluster [DBG] pgmap v7446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:35 smithi115 bash[46934]: cluster 2022-04-22T20:29:33.563759+0000 mgr.y (mgr.34107) 6279 : cluster [DBG] pgmap v7446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:36 smithi121 bash[36902]: audit 2022-04-22T20:29:35.964128+0000 mon.a (mon.0) 9591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:29:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:36 smithi121 bash[36902]: audit 2022-04-22T20:29:35.967172+0000 mon.a (mon.0) 9592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:29:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:36 smithi121 bash[36902]: audit 2022-04-22T20:29:35.968859+0000 mon.a (mon.0) 9593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:29:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[45495]: audit 2022-04-22T20:29:35.964128+0000 mon.a (mon.0) 9591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:29:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[45495]: audit 2022-04-22T20:29:35.967172+0000 mon.a (mon.0) 9592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:29:36.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[45495]: audit 2022-04-22T20:29:35.968859+0000 mon.a (mon.0) 9593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:29:36.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[46934]: audit 2022-04-22T20:29:35.964128+0000 mon.a (mon.0) 9591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:29:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[46934]: audit 2022-04-22T20:29:35.967172+0000 mon.a (mon.0) 9592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:29:36.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:36 smithi115 bash[46934]: audit 2022-04-22T20:29:35.968859+0000 mon.a (mon.0) 9593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:29:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:37 smithi121 bash[36902]: cluster 2022-04-22T20:29:35.564835+0000 mgr.y (mgr.34107) 6280 : cluster [DBG] pgmap v7447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:37 smithi121 bash[36902]: audit 2022-04-22T20:29:36.331649+0000 mon.a (mon.0) 9594 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:29:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:37 smithi115 bash[45495]: cluster 2022-04-22T20:29:35.564835+0000 mgr.y (mgr.34107) 6280 : cluster [DBG] pgmap v7447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:37 smithi115 bash[45495]: audit 2022-04-22T20:29:36.331649+0000 mon.a (mon.0) 9594 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:29:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:37 smithi115 bash[46934]: cluster 2022-04-22T20:29:35.564835+0000 mgr.y (mgr.34107) 6280 : cluster [DBG] pgmap v7447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:37 smithi115 bash[46934]: audit 2022-04-22T20:29:36.331649+0000 mon.a (mon.0) 9594 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:29:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:38 smithi121 bash[36902]: audit 2022-04-22T20:29:37.514536+0000 mon.a (mon.0) 9595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:38 smithi121 bash[36902]: audit 2022-04-22T20:29:37.612233+0000 mon.a (mon.0) 9596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:38 smithi115 bash[45495]: audit 2022-04-22T20:29:37.514536+0000 mon.a (mon.0) 9595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:38 smithi115 bash[45495]: audit 2022-04-22T20:29:37.612233+0000 mon.a (mon.0) 9596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:38 smithi115 bash[46934]: audit 2022-04-22T20:29:37.514536+0000 mon.a (mon.0) 9595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:38 smithi115 bash[46934]: audit 2022-04-22T20:29:37.612233+0000 mon.a (mon.0) 9596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:39 smithi121 bash[36902]: cluster 2022-04-22T20:29:37.565893+0000 mgr.y (mgr.34107) 6281 : cluster [DBG] pgmap v7448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:39 smithi121 bash[36902]: audit 2022-04-22T20:29:39.075844+0000 mon.a (mon.0) 9597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:39 smithi115 bash[46934]: cluster 2022-04-22T20:29:37.565893+0000 mgr.y (mgr.34107) 6281 : cluster [DBG] pgmap v7448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:39 smithi115 bash[46934]: audit 2022-04-22T20:29:39.075844+0000 mon.a (mon.0) 9597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:39 smithi115 bash[45495]: cluster 2022-04-22T20:29:37.565893+0000 mgr.y (mgr.34107) 6281 : cluster [DBG] pgmap v7448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:39 smithi115 bash[45495]: audit 2022-04-22T20:29:39.075844+0000 mon.a (mon.0) 9597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:41 smithi121 bash[36902]: cluster 2022-04-22T20:29:39.566986+0000 mgr.y (mgr.34107) 6282 : cluster [DBG] pgmap v7449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:41 smithi115 bash[46934]: cluster 2022-04-22T20:29:39.566986+0000 mgr.y (mgr.34107) 6282 : cluster [DBG] pgmap v7449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:41 smithi115 bash[45495]: cluster 2022-04-22T20:29:39.566986+0000 mgr.y (mgr.34107) 6282 : cluster [DBG] pgmap v7449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:42.322Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:43 smithi121 bash[36902]: cluster 2022-04-22T20:29:41.568124+0000 mgr.y (mgr.34107) 6283 : cluster [DBG] pgmap v7450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:43 smithi121 bash[36902]: audit 2022-04-22T20:29:42.517639+0000 mon.a (mon.0) 9598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:43 smithi121 bash[36902]: audit 2022-04-22T20:29:42.615355+0000 mon.a (mon.0) 9599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[45495]: cluster 2022-04-22T20:29:41.568124+0000 mgr.y (mgr.34107) 6283 : cluster [DBG] pgmap v7450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[45495]: audit 2022-04-22T20:29:42.517639+0000 mon.a (mon.0) 9598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[45495]: audit 2022-04-22T20:29:42.615355+0000 mon.a (mon.0) 9599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[46934]: cluster 2022-04-22T20:29:41.568124+0000 mgr.y (mgr.34107) 6283 : cluster [DBG] pgmap v7450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[46934]: audit 2022-04-22T20:29:42.517639+0000 mon.a (mon.0) 9598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:43.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:43 smithi115 bash[46934]: audit 2022-04-22T20:29:42.615355+0000 mon.a (mon.0) 9599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:44 smithi121 bash[36902]: audit 2022-04-22T20:29:44.078976+0000 mon.a (mon.0) 9600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:44.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:44 smithi115 bash[45495]: audit 2022-04-22T20:29:44.078976+0000 mon.a (mon.0) 9600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:44.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:44.202Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:44.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:44 smithi115 bash[46934]: audit 2022-04-22T20:29:44.078976+0000 mon.a (mon.0) 9600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:45 smithi121 bash[36902]: cluster 2022-04-22T20:29:43.569283+0000 mgr.y (mgr.34107) 6284 : cluster [DBG] pgmap v7451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:45 smithi115 bash[45495]: cluster 2022-04-22T20:29:43.569283+0000 mgr.y (mgr.34107) 6284 : cluster [DBG] pgmap v7451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:45 smithi115 bash[46934]: cluster 2022-04-22T20:29:43.569283+0000 mgr.y (mgr.34107) 6284 : cluster [DBG] pgmap v7451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:46.401 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:29:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:46 smithi121 bash[36902]: cluster 2022-04-22T20:29:45.570441+0000 mgr.y (mgr.34107) 6285 : cluster [DBG] pgmap v7452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:46.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:46 smithi115 bash[45495]: cluster 2022-04-22T20:29:45.570441+0000 mgr.y (mgr.34107) 6285 : cluster [DBG] pgmap v7452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:46 smithi115 bash[46934]: cluster 2022-04-22T20:29:45.570441+0000 mgr.y (mgr.34107) 6285 : cluster [DBG] pgmap v7452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:46.751 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:29:46.751 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:29:46.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2h) 7m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:29:46.753 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:29:47.162 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:47.163 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:29:47.164 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:29:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:47 smithi121 bash[36902]: audit 2022-04-22T20:29:46.375578+0000 mgr.y (mgr.34107) 6286 : audit [DBG] from='client.41922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:47 smithi121 bash[36902]: audit 2022-04-22T20:29:46.723684+0000 mgr.y (mgr.34107) 6287 : audit [DBG] from='client.41928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:47 smithi121 bash[36902]: audit 2022-04-22T20:29:47.154878+0000 mon.a (mon.0) 9601 : audit [DBG] from='client.? 172.21.15.115:0/4288603055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[45495]: audit 2022-04-22T20:29:46.375578+0000 mgr.y (mgr.34107) 6286 : audit [DBG] from='client.41922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[45495]: audit 2022-04-22T20:29:46.723684+0000 mgr.y (mgr.34107) 6287 : audit [DBG] from='client.41928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[45495]: audit 2022-04-22T20:29:47.154878+0000 mon.a (mon.0) 9601 : audit [DBG] from='client.? 172.21.15.115:0/4288603055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[46934]: audit 2022-04-22T20:29:46.375578+0000 mgr.y (mgr.34107) 6286 : audit [DBG] from='client.41922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[46934]: audit 2022-04-22T20:29:46.723684+0000 mgr.y (mgr.34107) 6287 : audit [DBG] from='client.41928 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:29:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:47 smithi115 bash[46934]: audit 2022-04-22T20:29:47.154878+0000 mon.a (mon.0) 9601 : audit [DBG] from='client.? 172.21.15.115:0/4288603055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:29:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:48 smithi121 bash[36902]: audit 2022-04-22T20:29:47.520873+0000 mon.a (mon.0) 9602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:48 smithi121 bash[36902]: cluster 2022-04-22T20:29:47.571548+0000 mgr.y (mgr.34107) 6288 : cluster [DBG] pgmap v7453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:48 smithi121 bash[36902]: audit 2022-04-22T20:29:47.618536+0000 mon.a (mon.0) 9603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[45495]: audit 2022-04-22T20:29:47.520873+0000 mon.a (mon.0) 9602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[45495]: cluster 2022-04-22T20:29:47.571548+0000 mgr.y (mgr.34107) 6288 : cluster [DBG] pgmap v7453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[45495]: audit 2022-04-22T20:29:47.618536+0000 mon.a (mon.0) 9603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[46934]: audit 2022-04-22T20:29:47.520873+0000 mon.a (mon.0) 9602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[46934]: cluster 2022-04-22T20:29:47.571548+0000 mgr.y (mgr.34107) 6288 : cluster [DBG] pgmap v7453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:48 smithi115 bash[46934]: audit 2022-04-22T20:29:47.618536+0000 mon.a (mon.0) 9603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:49 smithi121 bash[36902]: audit 2022-04-22T20:29:49.082277+0000 mon.a (mon.0) 9604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:49 smithi115 bash[46934]: audit 2022-04-22T20:29:49.082277+0000 mon.a (mon.0) 9604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:49 smithi115 bash[45495]: audit 2022-04-22T20:29:49.082277+0000 mon.a (mon.0) 9604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:50 smithi121 bash[36902]: cluster 2022-04-22T20:29:49.572595+0000 mgr.y (mgr.34107) 6289 : cluster [DBG] pgmap v7454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:50 smithi115 bash[45495]: cluster 2022-04-22T20:29:49.572595+0000 mgr.y (mgr.34107) 6289 : cluster [DBG] pgmap v7454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:50 smithi115 bash[46934]: cluster 2022-04-22T20:29:49.572595+0000 mgr.y (mgr.34107) 6289 : cluster [DBG] pgmap v7454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:52.637 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:52.321Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:52 smithi121 bash[36902]: cluster 2022-04-22T20:29:51.573711+0000 mgr.y (mgr.34107) 6290 : cluster [DBG] pgmap v7455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:52 smithi121 bash[36902]: audit 2022-04-22T20:29:52.524293+0000 mon.a (mon.0) 9605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:52 smithi121 bash[36902]: audit 2022-04-22T20:29:52.621472+0000 mon.a (mon.0) 9606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[45495]: cluster 2022-04-22T20:29:51.573711+0000 mgr.y (mgr.34107) 6290 : cluster [DBG] pgmap v7455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[45495]: audit 2022-04-22T20:29:52.524293+0000 mon.a (mon.0) 9605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[45495]: audit 2022-04-22T20:29:52.621472+0000 mon.a (mon.0) 9606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[46934]: cluster 2022-04-22T20:29:51.573711+0000 mgr.y (mgr.34107) 6290 : cluster [DBG] pgmap v7455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[46934]: audit 2022-04-22T20:29:52.524293+0000 mon.a (mon.0) 9605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:52 smithi115 bash[46934]: audit 2022-04-22T20:29:52.621472+0000 mon.a (mon.0) 9606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:54.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:29:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:29:54.205Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:29:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:54 smithi121 bash[36902]: cluster 2022-04-22T20:29:53.574840+0000 mgr.y (mgr.34107) 6291 : cluster [DBG] pgmap v7456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:54 smithi121 bash[36902]: audit 2022-04-22T20:29:54.085257+0000 mon.a (mon.0) 9607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:54 smithi115 bash[45495]: cluster 2022-04-22T20:29:53.574840+0000 mgr.y (mgr.34107) 6291 : cluster [DBG] pgmap v7456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:55.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:54 smithi115 bash[45495]: audit 2022-04-22T20:29:54.085257+0000 mon.a (mon.0) 9607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:54 smithi115 bash[46934]: cluster 2022-04-22T20:29:53.574840+0000 mgr.y (mgr.34107) 6291 : cluster [DBG] pgmap v7456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:55.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:54 smithi115 bash[46934]: audit 2022-04-22T20:29:54.085257+0000 mon.a (mon.0) 9607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:29:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:56 smithi121 bash[36902]: cluster 2022-04-22T20:29:55.575948+0000 mgr.y (mgr.34107) 6292 : cluster [DBG] pgmap v7457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:57.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:56 smithi115 bash[45495]: cluster 2022-04-22T20:29:55.575948+0000 mgr.y (mgr.34107) 6292 : cluster [DBG] pgmap v7457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:56 smithi115 bash[46934]: cluster 2022-04-22T20:29:55.575948+0000 mgr.y (mgr.34107) 6292 : cluster [DBG] pgmap v7457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:57 smithi121 bash[36902]: audit 2022-04-22T20:29:57.527866+0000 mon.a (mon.0) 9608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:57 smithi121 bash[36902]: audit 2022-04-22T20:29:57.624199+0000 mon.a (mon.0) 9609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:57 smithi115 bash[45495]: audit 2022-04-22T20:29:57.527866+0000 mon.a (mon.0) 9608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:58.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:57 smithi115 bash[45495]: audit 2022-04-22T20:29:57.624199+0000 mon.a (mon.0) 9609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:57 smithi115 bash[46934]: audit 2022-04-22T20:29:57.527866+0000 mon.a (mon.0) 9608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:29:58.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:57 smithi115 bash[46934]: audit 2022-04-22T20:29:57.624199+0000 mon.a (mon.0) 9609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:29:59.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:58 smithi121 bash[36902]: cluster 2022-04-22T20:29:57.577033+0000 mgr.y (mgr.34107) 6293 : cluster [DBG] pgmap v7458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:59.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:58 smithi115 bash[45495]: cluster 2022-04-22T20:29:57.577033+0000 mgr.y (mgr.34107) 6293 : cluster [DBG] pgmap v7458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:29:59.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:58 smithi115 bash[46934]: cluster 2022-04-22T20:29:57.577033+0000 mgr.y (mgr.34107) 6293 : cluster [DBG] pgmap v7458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:29:59 smithi121 bash[36902]: audit 2022-04-22T20:29:59.088242+0000 mon.a (mon.0) 9610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:00.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:29:59 smithi115 bash[45495]: audit 2022-04-22T20:29:59.088242+0000 mon.a (mon.0) 9610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:00.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:29:59 smithi115 bash[46934]: audit 2022-04-22T20:29:59.088242+0000 mon.a (mon.0) 9610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:00 smithi121 bash[36902]: cluster 2022-04-22T20:29:59.578173+0000 mgr.y (mgr.34107) 6294 : cluster [DBG] pgmap v7459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:00 smithi121 bash[36902]: cluster 2022-04-22T20:30:00.000122+0000 mon.a (mon.0) 9611 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:30:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:00 smithi115 bash[45495]: cluster 2022-04-22T20:29:59.578173+0000 mgr.y (mgr.34107) 6294 : cluster [DBG] pgmap v7459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:01.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:00 smithi115 bash[45495]: cluster 2022-04-22T20:30:00.000122+0000 mon.a (mon.0) 9611 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:30:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:00 smithi115 bash[46934]: cluster 2022-04-22T20:29:59.578173+0000 mgr.y (mgr.34107) 6294 : cluster [DBG] pgmap v7459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:01.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:00 smithi115 bash[46934]: cluster 2022-04-22T20:30:00.000122+0000 mon.a (mon.0) 9611 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:30:02.662 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:02.322Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:02 smithi121 bash[36902]: cluster 2022-04-22T20:30:01.579228+0000 mgr.y (mgr.34107) 6295 : cluster [DBG] pgmap v7460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:02 smithi121 bash[36902]: audit 2022-04-22T20:30:02.531562+0000 mon.a (mon.0) 9612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:02 smithi121 bash[36902]: audit 2022-04-22T20:30:02.626779+0000 mon.a (mon.0) 9613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[45495]: cluster 2022-04-22T20:30:01.579228+0000 mgr.y (mgr.34107) 6295 : cluster [DBG] pgmap v7460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[45495]: audit 2022-04-22T20:30:02.531562+0000 mon.a (mon.0) 9612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:03.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[45495]: audit 2022-04-22T20:30:02.626779+0000 mon.a (mon.0) 9613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:03.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[46934]: cluster 2022-04-22T20:30:01.579228+0000 mgr.y (mgr.34107) 6295 : cluster [DBG] pgmap v7460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[46934]: audit 2022-04-22T20:30:02.531562+0000 mon.a (mon.0) 9612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:03.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:02 smithi115 bash[46934]: audit 2022-04-22T20:30:02.626779+0000 mon.a (mon.0) 9613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:04.204Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:04 smithi121 bash[36902]: cluster 2022-04-22T20:30:03.580399+0000 mgr.y (mgr.34107) 6296 : cluster [DBG] pgmap v7461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:04 smithi121 bash[36902]: audit 2022-04-22T20:30:04.091324+0000 mon.a (mon.0) 9614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:04 smithi115 bash[45495]: cluster 2022-04-22T20:30:03.580399+0000 mgr.y (mgr.34107) 6296 : cluster [DBG] pgmap v7461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:05.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:04 smithi115 bash[45495]: audit 2022-04-22T20:30:04.091324+0000 mon.a (mon.0) 9614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:05.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:04 smithi115 bash[46934]: cluster 2022-04-22T20:30:03.580399+0000 mgr.y (mgr.34107) 6296 : cluster [DBG] pgmap v7461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:05.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:04 smithi115 bash[46934]: audit 2022-04-22T20:30:04.091324+0000 mon.a (mon.0) 9614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:06 smithi121 bash[36902]: cluster 2022-04-22T20:30:05.581454+0000 mgr.y (mgr.34107) 6297 : cluster [DBG] pgmap v7462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:07.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:06 smithi115 bash[45495]: cluster 2022-04-22T20:30:05.581454+0000 mgr.y (mgr.34107) 6297 : cluster [DBG] pgmap v7462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:07.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:06 smithi115 bash[46934]: cluster 2022-04-22T20:30:05.581454+0000 mgr.y (mgr.34107) 6297 : cluster [DBG] pgmap v7462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:07 smithi121 bash[36902]: audit 2022-04-22T20:30:07.534687+0000 mon.a (mon.0) 9615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:07 smithi121 bash[36902]: audit 2022-04-22T20:30:07.629556+0000 mon.a (mon.0) 9616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:07 smithi115 bash[45495]: audit 2022-04-22T20:30:07.534687+0000 mon.a (mon.0) 9615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:08.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:07 smithi115 bash[45495]: audit 2022-04-22T20:30:07.629556+0000 mon.a (mon.0) 9616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:07 smithi115 bash[46934]: audit 2022-04-22T20:30:07.534687+0000 mon.a (mon.0) 9615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:08.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:07 smithi115 bash[46934]: audit 2022-04-22T20:30:07.629556+0000 mon.a (mon.0) 9616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:08 smithi121 bash[36902]: cluster 2022-04-22T20:30:07.582474+0000 mgr.y (mgr.34107) 6298 : cluster [DBG] pgmap v7463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:08 smithi115 bash[45495]: cluster 2022-04-22T20:30:07.582474+0000 mgr.y (mgr.34107) 6298 : cluster [DBG] pgmap v7463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:08 smithi115 bash[46934]: cluster 2022-04-22T20:30:07.582474+0000 mgr.y (mgr.34107) 6298 : cluster [DBG] pgmap v7463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:09 smithi121 bash[36902]: audit 2022-04-22T20:30:09.094335+0000 mon.a (mon.0) 9617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:10.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:09 smithi115 bash[45495]: audit 2022-04-22T20:30:09.094335+0000 mon.a (mon.0) 9617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:10.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:09 smithi115 bash[46934]: audit 2022-04-22T20:30:09.094335+0000 mon.a (mon.0) 9617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:11 smithi121 bash[36902]: cluster 2022-04-22T20:30:09.583577+0000 mgr.y (mgr.34107) 6299 : cluster [DBG] pgmap v7464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:11 smithi115 bash[45495]: cluster 2022-04-22T20:30:09.583577+0000 mgr.y (mgr.34107) 6299 : cluster [DBG] pgmap v7464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:11 smithi115 bash[46934]: cluster 2022-04-22T20:30:09.583577+0000 mgr.y (mgr.34107) 6299 : cluster [DBG] pgmap v7464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:12.323Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:13 smithi121 bash[36902]: cluster 2022-04-22T20:30:11.584686+0000 mgr.y (mgr.34107) 6300 : cluster [DBG] pgmap v7465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:13 smithi121 bash[36902]: audit 2022-04-22T20:30:12.537961+0000 mon.a (mon.0) 9618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:13 smithi121 bash[36902]: audit 2022-04-22T20:30:12.632585+0000 mon.a (mon.0) 9619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[45495]: cluster 2022-04-22T20:30:11.584686+0000 mgr.y (mgr.34107) 6300 : cluster [DBG] pgmap v7465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[45495]: audit 2022-04-22T20:30:12.537961+0000 mon.a (mon.0) 9618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[45495]: audit 2022-04-22T20:30:12.632585+0000 mon.a (mon.0) 9619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[46934]: cluster 2022-04-22T20:30:11.584686+0000 mgr.y (mgr.34107) 6300 : cluster [DBG] pgmap v7465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[46934]: audit 2022-04-22T20:30:12.537961+0000 mon.a (mon.0) 9618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:13.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:13 smithi115 bash[46934]: audit 2022-04-22T20:30:12.632585+0000 mon.a (mon.0) 9619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:14.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:14 smithi121 bash[36902]: audit 2022-04-22T20:30:14.097392+0000 mon.a (mon.0) 9620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:14.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:14.205Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:14 smithi115 bash[45495]: audit 2022-04-22T20:30:14.097392+0000 mon.a (mon.0) 9620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:14 smithi115 bash[46934]: audit 2022-04-22T20:30:14.097392+0000 mon.a (mon.0) 9620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:15 smithi121 bash[36902]: cluster 2022-04-22T20:30:13.585814+0000 mgr.y (mgr.34107) 6301 : cluster [DBG] pgmap v7466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:15 smithi115 bash[45495]: cluster 2022-04-22T20:30:13.585814+0000 mgr.y (mgr.34107) 6301 : cluster [DBG] pgmap v7466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:15 smithi115 bash[46934]: cluster 2022-04-22T20:30:13.585814+0000 mgr.y (mgr.34107) 6301 : cluster [DBG] pgmap v7466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:16 smithi121 bash[36902]: audit 2022-04-22T20:30:15.393482+0000 mon.a (mon.0) 9621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:30:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:16 smithi121 bash[36902]: audit 2022-04-22T20:30:15.394036+0000 mon.a (mon.0) 9622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:30:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:16 smithi115 bash[45495]: audit 2022-04-22T20:30:15.393482+0000 mon.a (mon.0) 9621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:30:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:16 smithi115 bash[45495]: audit 2022-04-22T20:30:15.394036+0000 mon.a (mon.0) 9622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:30:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:16 smithi115 bash[46934]: audit 2022-04-22T20:30:15.393482+0000 mon.a (mon.0) 9621 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:30:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:16 smithi115 bash[46934]: audit 2022-04-22T20:30:15.394036+0000 mon.a (mon.0) 9622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:30:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:17 smithi121 bash[36902]: cluster 2022-04-22T20:30:15.586900+0000 mgr.y (mgr.34107) 6302 : cluster [DBG] pgmap v7467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:17 smithi115 bash[45495]: cluster 2022-04-22T20:30:15.586900+0000 mgr.y (mgr.34107) 6302 : cluster [DBG] pgmap v7467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:17 smithi115 bash[46934]: cluster 2022-04-22T20:30:15.586900+0000 mgr.y (mgr.34107) 6302 : cluster [DBG] pgmap v7467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:17.546 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:30:17.891 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:30:17.891 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:30:17.891 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:30:17.892 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 8m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 7m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 7m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 7m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 8m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 8m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 8m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 8m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:30:17.893 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 8m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:30:18.294 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:30:18.294 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:30:18.295 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:30:18.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:18.296 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:30:18.296 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:30:18.296 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:30:18.296 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:30:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:18 smithi121 bash[36902]: audit 2022-04-22T20:30:17.541629+0000 mon.a (mon.0) 9623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:18 smithi121 bash[36902]: audit 2022-04-22T20:30:17.635756+0000 mon.a (mon.0) 9624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:18 smithi115 bash[45495]: audit 2022-04-22T20:30:17.541629+0000 mon.a (mon.0) 9623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:18 smithi115 bash[45495]: audit 2022-04-22T20:30:17.635756+0000 mon.a (mon.0) 9624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:18 smithi115 bash[46934]: audit 2022-04-22T20:30:17.541629+0000 mon.a (mon.0) 9623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:18 smithi115 bash[46934]: audit 2022-04-22T20:30:17.635756+0000 mon.a (mon.0) 9624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:19 smithi121 bash[36902]: audit 2022-04-22T20:30:17.521494+0000 mgr.y (mgr.34107) 6303 : audit [DBG] from='client.51871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:19 smithi121 bash[36902]: cluster 2022-04-22T20:30:17.587953+0000 mgr.y (mgr.34107) 6304 : cluster [DBG] pgmap v7468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:19 smithi121 bash[36902]: audit 2022-04-22T20:30:17.861083+0000 mgr.y (mgr.34107) 6305 : audit [DBG] from='client.51877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:19 smithi121 bash[36902]: audit 2022-04-22T20:30:18.286611+0000 mon.c (mon.1) 245 : audit [DBG] from='client.? 172.21.15.115:0/3715478162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:19 smithi121 bash[36902]: audit 2022-04-22T20:30:19.100493+0000 mon.a (mon.0) 9625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[45495]: audit 2022-04-22T20:30:17.521494+0000 mgr.y (mgr.34107) 6303 : audit [DBG] from='client.51871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[45495]: cluster 2022-04-22T20:30:17.587953+0000 mgr.y (mgr.34107) 6304 : cluster [DBG] pgmap v7468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[45495]: audit 2022-04-22T20:30:17.861083+0000 mgr.y (mgr.34107) 6305 : audit [DBG] from='client.51877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[45495]: audit 2022-04-22T20:30:18.286611+0000 mon.c (mon.1) 245 : audit [DBG] from='client.? 172.21.15.115:0/3715478162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[45495]: audit 2022-04-22T20:30:19.100493+0000 mon.a (mon.0) 9625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[46934]: audit 2022-04-22T20:30:17.521494+0000 mgr.y (mgr.34107) 6303 : audit [DBG] from='client.51871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[46934]: cluster 2022-04-22T20:30:17.587953+0000 mgr.y (mgr.34107) 6304 : cluster [DBG] pgmap v7468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[46934]: audit 2022-04-22T20:30:17.861083+0000 mgr.y (mgr.34107) 6305 : audit [DBG] from='client.51877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[46934]: audit 2022-04-22T20:30:18.286611+0000 mon.c (mon.1) 245 : audit [DBG] from='client.? 172.21.15.115:0/3715478162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:19 smithi115 bash[46934]: audit 2022-04-22T20:30:19.100493+0000 mon.a (mon.0) 9625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:21 smithi121 bash[36902]: cluster 2022-04-22T20:30:19.589047+0000 mgr.y (mgr.34107) 6306 : cluster [DBG] pgmap v7469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:21 smithi115 bash[45495]: cluster 2022-04-22T20:30:19.589047+0000 mgr.y (mgr.34107) 6306 : cluster [DBG] pgmap v7469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:21 smithi115 bash[46934]: cluster 2022-04-22T20:30:19.589047+0000 mgr.y (mgr.34107) 6306 : cluster [DBG] pgmap v7469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:22.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:22.324Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:23 smithi121 bash[36902]: cluster 2022-04-22T20:30:21.590110+0000 mgr.y (mgr.34107) 6307 : cluster [DBG] pgmap v7470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:23 smithi121 bash[36902]: audit 2022-04-22T20:30:22.545344+0000 mon.a (mon.0) 9626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:23 smithi121 bash[36902]: audit 2022-04-22T20:30:22.638649+0000 mon.a (mon.0) 9627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[45495]: cluster 2022-04-22T20:30:21.590110+0000 mgr.y (mgr.34107) 6307 : cluster [DBG] pgmap v7470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[45495]: audit 2022-04-22T20:30:22.545344+0000 mon.a (mon.0) 9626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[45495]: audit 2022-04-22T20:30:22.638649+0000 mon.a (mon.0) 9627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[46934]: cluster 2022-04-22T20:30:21.590110+0000 mgr.y (mgr.34107) 6307 : cluster [DBG] pgmap v7470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[46934]: audit 2022-04-22T20:30:22.545344+0000 mon.a (mon.0) 9626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:23 smithi115 bash[46934]: audit 2022-04-22T20:30:22.638649+0000 mon.a (mon.0) 9627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:24 smithi121 bash[36902]: audit 2022-04-22T20:30:24.103670+0000 mon.a (mon.0) 9628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:24 smithi115 bash[45495]: audit 2022-04-22T20:30:24.103670+0000 mon.a (mon.0) 9628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:24 smithi115 bash[46934]: audit 2022-04-22T20:30:24.103670+0000 mon.a (mon.0) 9628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:24.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:24.206Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:25 smithi121 bash[36902]: cluster 2022-04-22T20:30:23.591237+0000 mgr.y (mgr.34107) 6308 : cluster [DBG] pgmap v7471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:25 smithi115 bash[45495]: cluster 2022-04-22T20:30:23.591237+0000 mgr.y (mgr.34107) 6308 : cluster [DBG] pgmap v7471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:25 smithi115 bash[46934]: cluster 2022-04-22T20:30:23.591237+0000 mgr.y (mgr.34107) 6308 : cluster [DBG] pgmap v7471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:27 smithi121 bash[36902]: cluster 2022-04-22T20:30:25.592381+0000 mgr.y (mgr.34107) 6309 : cluster [DBG] pgmap v7472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:27 smithi115 bash[45495]: cluster 2022-04-22T20:30:25.592381+0000 mgr.y (mgr.34107) 6309 : cluster [DBG] pgmap v7472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:27 smithi115 bash[46934]: cluster 2022-04-22T20:30:25.592381+0000 mgr.y (mgr.34107) 6309 : cluster [DBG] pgmap v7472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:28 smithi121 bash[36902]: audit 2022-04-22T20:30:27.549295+0000 mon.a (mon.0) 9629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:28 smithi121 bash[36902]: audit 2022-04-22T20:30:27.641640+0000 mon.a (mon.0) 9630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:28 smithi115 bash[45495]: audit 2022-04-22T20:30:27.549295+0000 mon.a (mon.0) 9629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:28 smithi115 bash[45495]: audit 2022-04-22T20:30:27.641640+0000 mon.a (mon.0) 9630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:28 smithi115 bash[46934]: audit 2022-04-22T20:30:27.549295+0000 mon.a (mon.0) 9629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:28 smithi115 bash[46934]: audit 2022-04-22T20:30:27.641640+0000 mon.a (mon.0) 9630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:29 smithi121 bash[36902]: cluster 2022-04-22T20:30:27.593444+0000 mgr.y (mgr.34107) 6310 : cluster [DBG] pgmap v7473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:29 smithi121 bash[36902]: audit 2022-04-22T20:30:29.106813+0000 mon.a (mon.0) 9631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:29 smithi115 bash[45495]: cluster 2022-04-22T20:30:27.593444+0000 mgr.y (mgr.34107) 6310 : cluster [DBG] pgmap v7473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:29 smithi115 bash[45495]: audit 2022-04-22T20:30:29.106813+0000 mon.a (mon.0) 9631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:29 smithi115 bash[46934]: cluster 2022-04-22T20:30:27.593444+0000 mgr.y (mgr.34107) 6310 : cluster [DBG] pgmap v7473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:29 smithi115 bash[46934]: audit 2022-04-22T20:30:29.106813+0000 mon.a (mon.0) 9631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:31 smithi121 bash[36902]: cluster 2022-04-22T20:30:29.594537+0000 mgr.y (mgr.34107) 6311 : cluster [DBG] pgmap v7474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:31 smithi115 bash[45495]: cluster 2022-04-22T20:30:29.594537+0000 mgr.y (mgr.34107) 6311 : cluster [DBG] pgmap v7474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:31 smithi115 bash[46934]: cluster 2022-04-22T20:30:29.594537+0000 mgr.y (mgr.34107) 6311 : cluster [DBG] pgmap v7474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:32.324Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:33 smithi121 bash[36902]: cluster 2022-04-22T20:30:31.595649+0000 mgr.y (mgr.34107) 6312 : cluster [DBG] pgmap v7475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:33 smithi121 bash[36902]: audit 2022-04-22T20:30:32.552641+0000 mon.a (mon.0) 9632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:33 smithi121 bash[36902]: audit 2022-04-22T20:30:32.644568+0000 mon.a (mon.0) 9633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[45495]: cluster 2022-04-22T20:30:31.595649+0000 mgr.y (mgr.34107) 6312 : cluster [DBG] pgmap v7475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[45495]: audit 2022-04-22T20:30:32.552641+0000 mon.a (mon.0) 9632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[45495]: audit 2022-04-22T20:30:32.644568+0000 mon.a (mon.0) 9633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[46934]: cluster 2022-04-22T20:30:31.595649+0000 mgr.y (mgr.34107) 6312 : cluster [DBG] pgmap v7475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[46934]: audit 2022-04-22T20:30:32.552641+0000 mon.a (mon.0) 9632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:33.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:33 smithi115 bash[46934]: audit 2022-04-22T20:30:32.644568+0000 mon.a (mon.0) 9633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:34 smithi121 bash[36902]: audit 2022-04-22T20:30:34.110050+0000 mon.a (mon.0) 9634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:34.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:34 smithi115 bash[45495]: audit 2022-04-22T20:30:34.110050+0000 mon.a (mon.0) 9634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:34.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:34 smithi115 bash[46934]: audit 2022-04-22T20:30:34.110050+0000 mon.a (mon.0) 9634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:34.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:35 smithi121 bash[36902]: cluster 2022-04-22T20:30:33.596828+0000 mgr.y (mgr.34107) 6313 : cluster [DBG] pgmap v7476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:35 smithi115 bash[45495]: cluster 2022-04-22T20:30:33.596828+0000 mgr.y (mgr.34107) 6313 : cluster [DBG] pgmap v7476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:35 smithi115 bash[46934]: cluster 2022-04-22T20:30:33.596828+0000 mgr.y (mgr.34107) 6313 : cluster [DBG] pgmap v7476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:37 smithi121 bash[36902]: cluster 2022-04-22T20:30:35.597944+0000 mgr.y (mgr.34107) 6314 : cluster [DBG] pgmap v7477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:37 smithi121 bash[36902]: audit 2022-04-22T20:30:36.340054+0000 mon.a (mon.0) 9635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:30:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:37 smithi121 bash[36902]: audit 2022-04-22T20:30:36.342928+0000 mon.a (mon.0) 9636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:30:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:37 smithi121 bash[36902]: audit 2022-04-22T20:30:36.344733+0000 mon.a (mon.0) 9637 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:30:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:37 smithi121 bash[36902]: audit 2022-04-22T20:30:36.704888+0000 mon.a (mon.0) 9638 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:30:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[45495]: cluster 2022-04-22T20:30:35.597944+0000 mgr.y (mgr.34107) 6314 : cluster [DBG] pgmap v7477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[45495]: audit 2022-04-22T20:30:36.340054+0000 mon.a (mon.0) 9635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:30:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[45495]: audit 2022-04-22T20:30:36.342928+0000 mon.a (mon.0) 9636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:30:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[45495]: audit 2022-04-22T20:30:36.344733+0000 mon.a (mon.0) 9637 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:30:37.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[45495]: audit 2022-04-22T20:30:36.704888+0000 mon.a (mon.0) 9638 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:30:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[46934]: cluster 2022-04-22T20:30:35.597944+0000 mgr.y (mgr.34107) 6314 : cluster [DBG] pgmap v7477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[46934]: audit 2022-04-22T20:30:36.340054+0000 mon.a (mon.0) 9635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:30:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[46934]: audit 2022-04-22T20:30:36.342928+0000 mon.a (mon.0) 9636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:30:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[46934]: audit 2022-04-22T20:30:36.344733+0000 mon.a (mon.0) 9637 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:30:37.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:37 smithi115 bash[46934]: audit 2022-04-22T20:30:36.704888+0000 mon.a (mon.0) 9638 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:30:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:38 smithi121 bash[36902]: audit 2022-04-22T20:30:37.556145+0000 mon.a (mon.0) 9639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:38 smithi121 bash[36902]: audit 2022-04-22T20:30:37.647672+0000 mon.a (mon.0) 9640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:38 smithi115 bash[46934]: audit 2022-04-22T20:30:37.556145+0000 mon.a (mon.0) 9639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:38 smithi115 bash[46934]: audit 2022-04-22T20:30:37.647672+0000 mon.a (mon.0) 9640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:38 smithi115 bash[45495]: audit 2022-04-22T20:30:37.556145+0000 mon.a (mon.0) 9639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:38 smithi115 bash[45495]: audit 2022-04-22T20:30:37.647672+0000 mon.a (mon.0) 9640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:39 smithi121 bash[36902]: cluster 2022-04-22T20:30:37.598977+0000 mgr.y (mgr.34107) 6315 : cluster [DBG] pgmap v7478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:39 smithi121 bash[36902]: audit 2022-04-22T20:30:39.114812+0000 mon.a (mon.0) 9641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:39 smithi115 bash[45495]: cluster 2022-04-22T20:30:37.598977+0000 mgr.y (mgr.34107) 6315 : cluster [DBG] pgmap v7478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:39 smithi115 bash[45495]: audit 2022-04-22T20:30:39.114812+0000 mon.a (mon.0) 9641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:39 smithi115 bash[46934]: cluster 2022-04-22T20:30:37.598977+0000 mgr.y (mgr.34107) 6315 : cluster [DBG] pgmap v7478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:39 smithi115 bash[46934]: audit 2022-04-22T20:30:39.114812+0000 mon.a (mon.0) 9641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:41 smithi121 bash[36902]: cluster 2022-04-22T20:30:39.600129+0000 mgr.y (mgr.34107) 6316 : cluster [DBG] pgmap v7479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:41 smithi115 bash[45495]: cluster 2022-04-22T20:30:39.600129+0000 mgr.y (mgr.34107) 6316 : cluster [DBG] pgmap v7479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:41 smithi115 bash[46934]: cluster 2022-04-22T20:30:39.600129+0000 mgr.y (mgr.34107) 6316 : cluster [DBG] pgmap v7479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:42.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:42 smithi115 bash[45495]: cluster 2022-04-22T20:30:41.601226+0000 mgr.y (mgr.34107) 6317 : cluster [DBG] pgmap v7480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:42.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:42.324Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:42 smithi115 bash[46934]: cluster 2022-04-22T20:30:41.601226+0000 mgr.y (mgr.34107) 6317 : cluster [DBG] pgmap v7480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:42 smithi121 bash[36902]: cluster 2022-04-22T20:30:41.601226+0000 mgr.y (mgr.34107) 6317 : cluster [DBG] pgmap v7480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:43 smithi115 bash[45495]: audit 2022-04-22T20:30:42.559865+0000 mon.a (mon.0) 9642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:43 smithi115 bash[45495]: audit 2022-04-22T20:30:42.650500+0000 mon.a (mon.0) 9643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:43 smithi115 bash[46934]: audit 2022-04-22T20:30:42.559865+0000 mon.a (mon.0) 9642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:43 smithi115 bash[46934]: audit 2022-04-22T20:30:42.650500+0000 mon.a (mon.0) 9643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:43 smithi121 bash[36902]: audit 2022-04-22T20:30:42.559865+0000 mon.a (mon.0) 9642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:43 smithi121 bash[36902]: audit 2022-04-22T20:30:42.650500+0000 mon.a (mon.0) 9643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:44 smithi115 bash[46934]: cluster 2022-04-22T20:30:43.602380+0000 mgr.y (mgr.34107) 6318 : cluster [DBG] pgmap v7481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:44.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:44 smithi115 bash[46934]: audit 2022-04-22T20:30:44.116524+0000 mon.a (mon.0) 9644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:44.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:44.207Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:44 smithi115 bash[45495]: cluster 2022-04-22T20:30:43.602380+0000 mgr.y (mgr.34107) 6318 : cluster [DBG] pgmap v7481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:44 smithi115 bash[45495]: audit 2022-04-22T20:30:44.116524+0000 mon.a (mon.0) 9644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:44 smithi121 bash[36902]: cluster 2022-04-22T20:30:43.602380+0000 mgr.y (mgr.34107) 6318 : cluster [DBG] pgmap v7481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:44 smithi121 bash[36902]: audit 2022-04-22T20:30:44.116524+0000 mon.a (mon.0) 9644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:46 smithi121 bash[36902]: cluster 2022-04-22T20:30:45.603598+0000 mgr.y (mgr.34107) 6319 : cluster [DBG] pgmap v7482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:47.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:46 smithi115 bash[46934]: cluster 2022-04-22T20:30:45.603598+0000 mgr.y (mgr.34107) 6319 : cluster [DBG] pgmap v7482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:47.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:46 smithi115 bash[45495]: cluster 2022-04-22T20:30:45.603598+0000 mgr.y (mgr.34107) 6319 : cluster [DBG] pgmap v7482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:47 smithi121 bash[36902]: audit 2022-04-22T20:30:47.563608+0000 mon.a (mon.0) 9645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:47 smithi121 bash[36902]: audit 2022-04-22T20:30:47.652773+0000 mon.a (mon.0) 9646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:47 smithi115 bash[45495]: audit 2022-04-22T20:30:47.563608+0000 mon.a (mon.0) 9645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:48.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:47 smithi115 bash[45495]: audit 2022-04-22T20:30:47.652773+0000 mon.a (mon.0) 9646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:47 smithi115 bash[46934]: audit 2022-04-22T20:30:47.563608+0000 mon.a (mon.0) 9645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:47 smithi115 bash[46934]: audit 2022-04-22T20:30:47.652773+0000 mon.a (mon.0) 9646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:48.675 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:30:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:48 smithi121 bash[36902]: cluster 2022-04-22T20:30:47.604761+0000 mgr.y (mgr.34107) 6320 : cluster [DBG] pgmap v7483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:49.025 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:30:49.025 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:30:49.025 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:30:49.026 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2h) 8m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:30:49.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:30:49.028 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:30:49.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:48 smithi115 bash[45495]: cluster 2022-04-22T20:30:47.604761+0000 mgr.y (mgr.34107) 6320 : cluster [DBG] pgmap v7483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:49.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:48 smithi115 bash[46934]: cluster 2022-04-22T20:30:47.604761+0000 mgr.y (mgr.34107) 6320 : cluster [DBG] pgmap v7483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:49.432 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:30:49.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:30:49.434 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:30:49.435 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:30:49.435 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:30:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:50 smithi121 bash[36902]: audit 2022-04-22T20:30:48.652785+0000 mgr.y (mgr.34107) 6321 : audit [DBG] from='client.51889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:50 smithi121 bash[36902]: audit 2022-04-22T20:30:48.993137+0000 mgr.y (mgr.34107) 6322 : audit [DBG] from='client.51895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:50 smithi121 bash[36902]: audit 2022-04-22T20:30:49.123315+0000 mon.a (mon.0) 9647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:50 smithi121 bash[36902]: audit 2022-04-22T20:30:49.425068+0000 mon.c (mon.1) 246 : audit [DBG] from='client.? 172.21.15.115:0/514716118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[45495]: audit 2022-04-22T20:30:48.652785+0000 mgr.y (mgr.34107) 6321 : audit [DBG] from='client.51889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[45495]: audit 2022-04-22T20:30:48.993137+0000 mgr.y (mgr.34107) 6322 : audit [DBG] from='client.51895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[45495]: audit 2022-04-22T20:30:49.123315+0000 mon.a (mon.0) 9647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[45495]: audit 2022-04-22T20:30:49.425068+0000 mon.c (mon.1) 246 : audit [DBG] from='client.? 172.21.15.115:0/514716118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[46934]: audit 2022-04-22T20:30:48.652785+0000 mgr.y (mgr.34107) 6321 : audit [DBG] from='client.51889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[46934]: audit 2022-04-22T20:30:48.993137+0000 mgr.y (mgr.34107) 6322 : audit [DBG] from='client.51895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:30:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[46934]: audit 2022-04-22T20:30:49.123315+0000 mon.a (mon.0) 9647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:50.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:50 smithi115 bash[46934]: audit 2022-04-22T20:30:49.425068+0000 mon.c (mon.1) 246 : audit [DBG] from='client.? 172.21.15.115:0/514716118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:30:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:51 smithi121 bash[36902]: cluster 2022-04-22T20:30:49.605932+0000 mgr.y (mgr.34107) 6323 : cluster [DBG] pgmap v7484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:51 smithi115 bash[45495]: cluster 2022-04-22T20:30:49.605932+0000 mgr.y (mgr.34107) 6323 : cluster [DBG] pgmap v7484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:51 smithi115 bash[46934]: cluster 2022-04-22T20:30:49.605932+0000 mgr.y (mgr.34107) 6323 : cluster [DBG] pgmap v7484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:52.325Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:53 smithi121 bash[36902]: cluster 2022-04-22T20:30:51.607029+0000 mgr.y (mgr.34107) 6324 : cluster [DBG] pgmap v7485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:53.555 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:53 smithi121 bash[36902]: audit 2022-04-22T20:30:52.567467+0000 mon.a (mon.0) 9648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:53.555 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:53 smithi121 bash[36902]: audit 2022-04-22T20:30:52.655934+0000 mon.a (mon.0) 9649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:53.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[45495]: cluster 2022-04-22T20:30:51.607029+0000 mgr.y (mgr.34107) 6324 : cluster [DBG] pgmap v7485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:53.556 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[45495]: audit 2022-04-22T20:30:52.567467+0000 mon.a (mon.0) 9648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:53.556 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[45495]: audit 2022-04-22T20:30:52.655934+0000 mon.a (mon.0) 9649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:53.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[46934]: cluster 2022-04-22T20:30:51.607029+0000 mgr.y (mgr.34107) 6324 : cluster [DBG] pgmap v7485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:53.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[46934]: audit 2022-04-22T20:30:52.567467+0000 mon.a (mon.0) 9648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:53.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:53 smithi115 bash[46934]: audit 2022-04-22T20:30:52.655934+0000 mon.a (mon.0) 9649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:54.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:30:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:30:54.209Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:30:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:55 smithi121 bash[36902]: cluster 2022-04-22T20:30:53.608188+0000 mgr.y (mgr.34107) 6325 : cluster [DBG] pgmap v7486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:55 smithi121 bash[36902]: audit 2022-04-22T20:30:54.125676+0000 mon.a (mon.0) 9650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:55 smithi115 bash[45495]: cluster 2022-04-22T20:30:53.608188+0000 mgr.y (mgr.34107) 6325 : cluster [DBG] pgmap v7486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:55 smithi115 bash[45495]: audit 2022-04-22T20:30:54.125676+0000 mon.a (mon.0) 9650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:55 smithi115 bash[46934]: cluster 2022-04-22T20:30:53.608188+0000 mgr.y (mgr.34107) 6325 : cluster [DBG] pgmap v7486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:55 smithi115 bash[46934]: audit 2022-04-22T20:30:54.125676+0000 mon.a (mon.0) 9650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:57 smithi121 bash[36902]: cluster 2022-04-22T20:30:55.609209+0000 mgr.y (mgr.34107) 6326 : cluster [DBG] pgmap v7487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:57 smithi115 bash[45495]: cluster 2022-04-22T20:30:55.609209+0000 mgr.y (mgr.34107) 6326 : cluster [DBG] pgmap v7487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:57 smithi115 bash[46934]: cluster 2022-04-22T20:30:55.609209+0000 mgr.y (mgr.34107) 6326 : cluster [DBG] pgmap v7487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:58 smithi121 bash[36902]: audit 2022-04-22T20:30:57.570920+0000 mon.a (mon.0) 9651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:58 smithi121 bash[36902]: audit 2022-04-22T20:30:57.659227+0000 mon.a (mon.0) 9652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:58 smithi115 bash[45495]: audit 2022-04-22T20:30:57.570920+0000 mon.a (mon.0) 9651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:58 smithi115 bash[45495]: audit 2022-04-22T20:30:57.659227+0000 mon.a (mon.0) 9652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:58 smithi115 bash[46934]: audit 2022-04-22T20:30:57.570920+0000 mon.a (mon.0) 9651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:30:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:58 smithi115 bash[46934]: audit 2022-04-22T20:30:57.659227+0000 mon.a (mon.0) 9652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:30:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:59 smithi121 bash[36902]: cluster 2022-04-22T20:30:57.610344+0000 mgr.y (mgr.34107) 6327 : cluster [DBG] pgmap v7488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:30:59 smithi121 bash[36902]: audit 2022-04-22T20:30:59.127176+0000 mon.a (mon.0) 9653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:59 smithi115 bash[45495]: cluster 2022-04-22T20:30:57.610344+0000 mgr.y (mgr.34107) 6327 : cluster [DBG] pgmap v7488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:30:59 smithi115 bash[45495]: audit 2022-04-22T20:30:59.127176+0000 mon.a (mon.0) 9653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:30:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:59 smithi115 bash[46934]: cluster 2022-04-22T20:30:57.610344+0000 mgr.y (mgr.34107) 6327 : cluster [DBG] pgmap v7488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:30:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:30:59 smithi115 bash[46934]: audit 2022-04-22T20:30:59.127176+0000 mon.a (mon.0) 9653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:01 smithi121 bash[36902]: cluster 2022-04-22T20:30:59.611464+0000 mgr.y (mgr.34107) 6328 : cluster [DBG] pgmap v7489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:01 smithi115 bash[46934]: cluster 2022-04-22T20:30:59.611464+0000 mgr.y (mgr.34107) 6328 : cluster [DBG] pgmap v7489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:01.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:01 smithi115 bash[45495]: cluster 2022-04-22T20:30:59.611464+0000 mgr.y (mgr.34107) 6328 : cluster [DBG] pgmap v7489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:02.326Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:03 smithi121 bash[36902]: cluster 2022-04-22T20:31:01.612610+0000 mgr.y (mgr.34107) 6329 : cluster [DBG] pgmap v7490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:03 smithi121 bash[36902]: audit 2022-04-22T20:31:02.574640+0000 mon.a (mon.0) 9654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:03 smithi121 bash[36902]: audit 2022-04-22T20:31:02.662292+0000 mon.a (mon.0) 9655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[46934]: cluster 2022-04-22T20:31:01.612610+0000 mgr.y (mgr.34107) 6329 : cluster [DBG] pgmap v7490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[46934]: audit 2022-04-22T20:31:02.574640+0000 mon.a (mon.0) 9654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[46934]: audit 2022-04-22T20:31:02.662292+0000 mon.a (mon.0) 9655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[45495]: cluster 2022-04-22T20:31:01.612610+0000 mgr.y (mgr.34107) 6329 : cluster [DBG] pgmap v7490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[45495]: audit 2022-04-22T20:31:02.574640+0000 mon.a (mon.0) 9654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:03.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:03 smithi115 bash[45495]: audit 2022-04-22T20:31:02.662292+0000 mon.a (mon.0) 9655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:04.208Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:05 smithi121 bash[36902]: cluster 2022-04-22T20:31:03.613835+0000 mgr.y (mgr.34107) 6330 : cluster [DBG] pgmap v7491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:05 smithi121 bash[36902]: audit 2022-04-22T20:31:04.129878+0000 mon.a (mon.0) 9656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:05 smithi115 bash[45495]: cluster 2022-04-22T20:31:03.613835+0000 mgr.y (mgr.34107) 6330 : cluster [DBG] pgmap v7491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:05 smithi115 bash[45495]: audit 2022-04-22T20:31:04.129878+0000 mon.a (mon.0) 9656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:05 smithi115 bash[46934]: cluster 2022-04-22T20:31:03.613835+0000 mgr.y (mgr.34107) 6330 : cluster [DBG] pgmap v7491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:05 smithi115 bash[46934]: audit 2022-04-22T20:31:04.129878+0000 mon.a (mon.0) 9656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:07 smithi121 bash[36902]: cluster 2022-04-22T20:31:05.614915+0000 mgr.y (mgr.34107) 6331 : cluster [DBG] pgmap v7492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:07.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:07 smithi115 bash[45495]: cluster 2022-04-22T20:31:05.614915+0000 mgr.y (mgr.34107) 6331 : cluster [DBG] pgmap v7492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:07 smithi115 bash[46934]: cluster 2022-04-22T20:31:05.614915+0000 mgr.y (mgr.34107) 6331 : cluster [DBG] pgmap v7492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:08 smithi121 bash[36902]: audit 2022-04-22T20:31:07.578465+0000 mon.a (mon.0) 9657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:08 smithi121 bash[36902]: audit 2022-04-22T20:31:07.665387+0000 mon.a (mon.0) 9658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:08 smithi115 bash[45495]: audit 2022-04-22T20:31:07.578465+0000 mon.a (mon.0) 9657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:08 smithi115 bash[45495]: audit 2022-04-22T20:31:07.665387+0000 mon.a (mon.0) 9658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:08 smithi115 bash[46934]: audit 2022-04-22T20:31:07.578465+0000 mon.a (mon.0) 9657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:08 smithi115 bash[46934]: audit 2022-04-22T20:31:07.665387+0000 mon.a (mon.0) 9658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:09 smithi121 bash[36902]: cluster 2022-04-22T20:31:07.615969+0000 mgr.y (mgr.34107) 6332 : cluster [DBG] pgmap v7493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:09 smithi121 bash[36902]: audit 2022-04-22T20:31:09.132466+0000 mon.a (mon.0) 9659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:09 smithi115 bash[45495]: cluster 2022-04-22T20:31:07.615969+0000 mgr.y (mgr.34107) 6332 : cluster [DBG] pgmap v7493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:09 smithi115 bash[45495]: audit 2022-04-22T20:31:09.132466+0000 mon.a (mon.0) 9659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:09 smithi115 bash[46934]: cluster 2022-04-22T20:31:07.615969+0000 mgr.y (mgr.34107) 6332 : cluster [DBG] pgmap v7493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:09 smithi115 bash[46934]: audit 2022-04-22T20:31:09.132466+0000 mon.a (mon.0) 9659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:11 smithi121 bash[36902]: cluster 2022-04-22T20:31:09.616972+0000 mgr.y (mgr.34107) 6333 : cluster [DBG] pgmap v7494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:11.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:11 smithi115 bash[45495]: cluster 2022-04-22T20:31:09.616972+0000 mgr.y (mgr.34107) 6333 : cluster [DBG] pgmap v7494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:11 smithi115 bash[46934]: cluster 2022-04-22T20:31:09.616972+0000 mgr.y (mgr.34107) 6333 : cluster [DBG] pgmap v7494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:12.327Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:13 smithi121 bash[36902]: cluster 2022-04-22T20:31:11.618065+0000 mgr.y (mgr.34107) 6334 : cluster [DBG] pgmap v7495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:13 smithi121 bash[36902]: audit 2022-04-22T20:31:12.582603+0000 mon.a (mon.0) 9660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:13 smithi121 bash[36902]: audit 2022-04-22T20:31:12.668454+0000 mon.a (mon.0) 9661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[45495]: cluster 2022-04-22T20:31:11.618065+0000 mgr.y (mgr.34107) 6334 : cluster [DBG] pgmap v7495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[45495]: audit 2022-04-22T20:31:12.582603+0000 mon.a (mon.0) 9660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:13.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[45495]: audit 2022-04-22T20:31:12.668454+0000 mon.a (mon.0) 9661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[46934]: cluster 2022-04-22T20:31:11.618065+0000 mgr.y (mgr.34107) 6334 : cluster [DBG] pgmap v7495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[46934]: audit 2022-04-22T20:31:12.582603+0000 mon.a (mon.0) 9660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:13 smithi115 bash[46934]: audit 2022-04-22T20:31:12.668454+0000 mon.a (mon.0) 9661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:14.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:14.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:15 smithi121 bash[36902]: cluster 2022-04-22T20:31:13.619189+0000 mgr.y (mgr.34107) 6335 : cluster [DBG] pgmap v7496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:15 smithi121 bash[36902]: audit 2022-04-22T20:31:14.135136+0000 mon.a (mon.0) 9662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:15 smithi115 bash[45495]: cluster 2022-04-22T20:31:13.619189+0000 mgr.y (mgr.34107) 6335 : cluster [DBG] pgmap v7496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:15 smithi115 bash[45495]: audit 2022-04-22T20:31:14.135136+0000 mon.a (mon.0) 9662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:15 smithi115 bash[46934]: cluster 2022-04-22T20:31:13.619189+0000 mgr.y (mgr.34107) 6335 : cluster [DBG] pgmap v7496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:15 smithi115 bash[46934]: audit 2022-04-22T20:31:14.135136+0000 mon.a (mon.0) 9662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:16 smithi121 bash[36902]: audit 2022-04-22T20:31:15.403385+0000 mon.a (mon.0) 9663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:31:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:16 smithi121 bash[36902]: audit 2022-04-22T20:31:15.403801+0000 mon.a (mon.0) 9664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:31:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:16 smithi115 bash[45495]: audit 2022-04-22T20:31:15.403385+0000 mon.a (mon.0) 9663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:31:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:16 smithi115 bash[45495]: audit 2022-04-22T20:31:15.403801+0000 mon.a (mon.0) 9664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:31:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:16 smithi115 bash[46934]: audit 2022-04-22T20:31:15.403385+0000 mon.a (mon.0) 9663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:31:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:16 smithi115 bash[46934]: audit 2022-04-22T20:31:15.403801+0000 mon.a (mon.0) 9664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:31:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:17 smithi121 bash[36902]: cluster 2022-04-22T20:31:15.620288+0000 mgr.y (mgr.34107) 6336 : cluster [DBG] pgmap v7497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:17 smithi115 bash[45495]: cluster 2022-04-22T20:31:15.620288+0000 mgr.y (mgr.34107) 6336 : cluster [DBG] pgmap v7497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:17 smithi115 bash[46934]: cluster 2022-04-22T20:31:15.620288+0000 mgr.y (mgr.34107) 6336 : cluster [DBG] pgmap v7497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:18 smithi121 bash[36902]: audit 2022-04-22T20:31:17.586373+0000 mon.a (mon.0) 9665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:18 smithi121 bash[36902]: audit 2022-04-22T20:31:17.671626+0000 mon.a (mon.0) 9666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:18 smithi115 bash[45495]: audit 2022-04-22T20:31:17.586373+0000 mon.a (mon.0) 9665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:18 smithi115 bash[45495]: audit 2022-04-22T20:31:17.671626+0000 mon.a (mon.0) 9666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:18 smithi115 bash[46934]: audit 2022-04-22T20:31:17.586373+0000 mon.a (mon.0) 9665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:18 smithi115 bash[46934]: audit 2022-04-22T20:31:17.671626+0000 mon.a (mon.0) 9666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:19 smithi121 bash[36902]: cluster 2022-04-22T20:31:17.621307+0000 mgr.y (mgr.34107) 6337 : cluster [DBG] pgmap v7498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:19 smithi121 bash[36902]: audit 2022-04-22T20:31:19.137716+0000 mon.a (mon.0) 9667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:19 smithi115 bash[45495]: cluster 2022-04-22T20:31:17.621307+0000 mgr.y (mgr.34107) 6337 : cluster [DBG] pgmap v7498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:19.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:19 smithi115 bash[45495]: audit 2022-04-22T20:31:19.137716+0000 mon.a (mon.0) 9667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:19 smithi115 bash[46934]: cluster 2022-04-22T20:31:17.621307+0000 mgr.y (mgr.34107) 6337 : cluster [DBG] pgmap v7498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:19.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:19 smithi115 bash[46934]: audit 2022-04-22T20:31:19.137716+0000 mon.a (mon.0) 9667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:19.805 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:31:20.151 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:31:20.151 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 8m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:31:20.151 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:31:20.151 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:31:20.151 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 209M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 8m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 9m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:31:20.152 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2h) 8m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2h) 8m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 9m ago 3h 577M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 9m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 9m ago 3h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 9m ago 3h 638M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:31:20.153 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 9m ago 3h 72.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:31:20.553 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:31:20.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:20.555 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:31:20.555 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:31:20.555 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:31:20.556 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:31:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:21 smithi121 bash[36902]: cluster 2022-04-22T20:31:19.622380+0000 mgr.y (mgr.34107) 6338 : cluster [DBG] pgmap v7499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:21 smithi121 bash[36902]: audit 2022-04-22T20:31:19.782374+0000 mgr.y (mgr.34107) 6339 : audit [DBG] from='client.41976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:21 smithi121 bash[36902]: audit 2022-04-22T20:31:20.121987+0000 mgr.y (mgr.34107) 6340 : audit [DBG] from='client.41982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:21 smithi121 bash[36902]: audit 2022-04-22T20:31:20.545151+0000 mon.c (mon.1) 247 : audit [DBG] from='client.? 172.21.15.115:0/3072788163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[46934]: cluster 2022-04-22T20:31:19.622380+0000 mgr.y (mgr.34107) 6338 : cluster [DBG] pgmap v7499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[46934]: audit 2022-04-22T20:31:19.782374+0000 mgr.y (mgr.34107) 6339 : audit [DBG] from='client.41976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[46934]: audit 2022-04-22T20:31:20.121987+0000 mgr.y (mgr.34107) 6340 : audit [DBG] from='client.41982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[46934]: audit 2022-04-22T20:31:20.545151+0000 mon.c (mon.1) 247 : audit [DBG] from='client.? 172.21.15.115:0/3072788163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[45495]: cluster 2022-04-22T20:31:19.622380+0000 mgr.y (mgr.34107) 6338 : cluster [DBG] pgmap v7499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:21.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[45495]: audit 2022-04-22T20:31:19.782374+0000 mgr.y (mgr.34107) 6339 : audit [DBG] from='client.41976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[45495]: audit 2022-04-22T20:31:20.121987+0000 mgr.y (mgr.34107) 6340 : audit [DBG] from='client.41982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:21.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:21 smithi115 bash[45495]: audit 2022-04-22T20:31:20.545151+0000 mon.c (mon.1) 247 : audit [DBG] from='client.? 172.21.15.115:0/3072788163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:22.326Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:23 smithi121 bash[36902]: cluster 2022-04-22T20:31:21.623520+0000 mgr.y (mgr.34107) 6341 : cluster [DBG] pgmap v7500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:23 smithi121 bash[36902]: audit 2022-04-22T20:31:22.590134+0000 mon.a (mon.0) 9668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:23 smithi121 bash[36902]: audit 2022-04-22T20:31:22.674509+0000 mon.a (mon.0) 9669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[45495]: cluster 2022-04-22T20:31:21.623520+0000 mgr.y (mgr.34107) 6341 : cluster [DBG] pgmap v7500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[45495]: audit 2022-04-22T20:31:22.590134+0000 mon.a (mon.0) 9668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[45495]: audit 2022-04-22T20:31:22.674509+0000 mon.a (mon.0) 9669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[46934]: cluster 2022-04-22T20:31:21.623520+0000 mgr.y (mgr.34107) 6341 : cluster [DBG] pgmap v7500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[46934]: audit 2022-04-22T20:31:22.590134+0000 mon.a (mon.0) 9668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:23 smithi115 bash[46934]: audit 2022-04-22T20:31:22.674509+0000 mon.a (mon.0) 9669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:24.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:24.209Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:25 smithi121 bash[36902]: cluster 2022-04-22T20:31:23.624674+0000 mgr.y (mgr.34107) 6342 : cluster [DBG] pgmap v7501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:25 smithi121 bash[36902]: audit 2022-04-22T20:31:24.140221+0000 mon.a (mon.0) 9670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:25 smithi115 bash[46934]: cluster 2022-04-22T20:31:23.624674+0000 mgr.y (mgr.34107) 6342 : cluster [DBG] pgmap v7501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:25 smithi115 bash[46934]: audit 2022-04-22T20:31:24.140221+0000 mon.a (mon.0) 9670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:25 smithi115 bash[45495]: cluster 2022-04-22T20:31:23.624674+0000 mgr.y (mgr.34107) 6342 : cluster [DBG] pgmap v7501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:25 smithi115 bash[45495]: audit 2022-04-22T20:31:24.140221+0000 mon.a (mon.0) 9670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:27 smithi121 bash[36902]: cluster 2022-04-22T20:31:25.625883+0000 mgr.y (mgr.34107) 6343 : cluster [DBG] pgmap v7502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:27 smithi115 bash[45495]: cluster 2022-04-22T20:31:25.625883+0000 mgr.y (mgr.34107) 6343 : cluster [DBG] pgmap v7502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:27 smithi115 bash[46934]: cluster 2022-04-22T20:31:25.625883+0000 mgr.y (mgr.34107) 6343 : cluster [DBG] pgmap v7502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:28 smithi121 bash[36902]: audit 2022-04-22T20:31:27.593683+0000 mon.a (mon.0) 9671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:28.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:28 smithi121 bash[36902]: audit 2022-04-22T20:31:27.677385+0000 mon.a (mon.0) 9672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:28 smithi115 bash[45495]: audit 2022-04-22T20:31:27.593683+0000 mon.a (mon.0) 9671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:28 smithi115 bash[45495]: audit 2022-04-22T20:31:27.677385+0000 mon.a (mon.0) 9672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:28 smithi115 bash[46934]: audit 2022-04-22T20:31:27.593683+0000 mon.a (mon.0) 9671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:28.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:28 smithi115 bash[46934]: audit 2022-04-22T20:31:27.677385+0000 mon.a (mon.0) 9672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:29 smithi121 bash[36902]: cluster 2022-04-22T20:31:27.627022+0000 mgr.y (mgr.34107) 6344 : cluster [DBG] pgmap v7503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:29 smithi121 bash[36902]: audit 2022-04-22T20:31:29.142543+0000 mon.a (mon.0) 9673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:29 smithi115 bash[46934]: cluster 2022-04-22T20:31:27.627022+0000 mgr.y (mgr.34107) 6344 : cluster [DBG] pgmap v7503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:29 smithi115 bash[46934]: audit 2022-04-22T20:31:29.142543+0000 mon.a (mon.0) 9673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:29 smithi115 bash[45495]: cluster 2022-04-22T20:31:27.627022+0000 mgr.y (mgr.34107) 6344 : cluster [DBG] pgmap v7503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:29 smithi115 bash[45495]: audit 2022-04-22T20:31:29.142543+0000 mon.a (mon.0) 9673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:31 smithi121 bash[36902]: cluster 2022-04-22T20:31:29.628136+0000 mgr.y (mgr.34107) 6345 : cluster [DBG] pgmap v7504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:31 smithi115 bash[45495]: cluster 2022-04-22T20:31:29.628136+0000 mgr.y (mgr.34107) 6345 : cluster [DBG] pgmap v7504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:31 smithi115 bash[46934]: cluster 2022-04-22T20:31:29.628136+0000 mgr.y (mgr.34107) 6345 : cluster [DBG] pgmap v7504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:32.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:32.328Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:33 smithi121 bash[36902]: cluster 2022-04-22T20:31:31.629223+0000 mgr.y (mgr.34107) 6346 : cluster [DBG] pgmap v7505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:33 smithi121 bash[36902]: audit 2022-04-22T20:31:32.597376+0000 mon.a (mon.0) 9674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:33 smithi121 bash[36902]: audit 2022-04-22T20:31:32.680222+0000 mon.a (mon.0) 9675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[46934]: cluster 2022-04-22T20:31:31.629223+0000 mgr.y (mgr.34107) 6346 : cluster [DBG] pgmap v7505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[46934]: audit 2022-04-22T20:31:32.597376+0000 mon.a (mon.0) 9674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[46934]: audit 2022-04-22T20:31:32.680222+0000 mon.a (mon.0) 9675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[45495]: cluster 2022-04-22T20:31:31.629223+0000 mgr.y (mgr.34107) 6346 : cluster [DBG] pgmap v7505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[45495]: audit 2022-04-22T20:31:32.597376+0000 mon.a (mon.0) 9674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:33.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:33 smithi115 bash[45495]: audit 2022-04-22T20:31:32.680222+0000 mon.a (mon.0) 9675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:34.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:35 smithi121 bash[36902]: cluster 2022-04-22T20:31:33.630321+0000 mgr.y (mgr.34107) 6347 : cluster [DBG] pgmap v7506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:35 smithi121 bash[36902]: audit 2022-04-22T20:31:34.144709+0000 mon.a (mon.0) 9676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:35 smithi115 bash[45495]: cluster 2022-04-22T20:31:33.630321+0000 mgr.y (mgr.34107) 6347 : cluster [DBG] pgmap v7506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:35.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:35 smithi115 bash[45495]: audit 2022-04-22T20:31:34.144709+0000 mon.a (mon.0) 9676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:35 smithi115 bash[46934]: cluster 2022-04-22T20:31:33.630321+0000 mgr.y (mgr.34107) 6347 : cluster [DBG] pgmap v7506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:35.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:35 smithi115 bash[46934]: audit 2022-04-22T20:31:34.144709+0000 mon.a (mon.0) 9676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[45495]: cluster 2022-04-22T20:31:35.631555+0000 mgr.y (mgr.34107) 6348 : cluster [DBG] pgmap v7507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[45495]: audit 2022-04-22T20:31:36.715001+0000 mon.a (mon.0) 9677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:31:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[45495]: audit 2022-04-22T20:31:36.717998+0000 mon.a (mon.0) 9678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:31:37.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[45495]: audit 2022-04-22T20:31:36.719770+0000 mon.a (mon.0) 9679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:31:37.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[46934]: cluster 2022-04-22T20:31:35.631555+0000 mgr.y (mgr.34107) 6348 : cluster [DBG] pgmap v7507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[46934]: audit 2022-04-22T20:31:36.715001+0000 mon.a (mon.0) 9677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:31:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[46934]: audit 2022-04-22T20:31:36.717998+0000 mon.a (mon.0) 9678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:31:37.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:37 smithi115 bash[46934]: audit 2022-04-22T20:31:36.719770+0000 mon.a (mon.0) 9679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:31:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:37 smithi121 bash[36902]: cluster 2022-04-22T20:31:35.631555+0000 mgr.y (mgr.34107) 6348 : cluster [DBG] pgmap v7507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:37 smithi121 bash[36902]: audit 2022-04-22T20:31:36.715001+0000 mon.a (mon.0) 9677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:31:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:37 smithi121 bash[36902]: audit 2022-04-22T20:31:36.717998+0000 mon.a (mon.0) 9678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:31:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:37 smithi121 bash[36902]: audit 2022-04-22T20:31:36.719770+0000 mon.a (mon.0) 9679 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:31:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:38 smithi121 bash[36902]: audit 2022-04-22T20:31:37.601204+0000 mon.a (mon.0) 9680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:38 smithi121 bash[36902]: audit 2022-04-22T20:31:37.683190+0000 mon.a (mon.0) 9681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:38 smithi115 bash[45495]: audit 2022-04-22T20:31:37.601204+0000 mon.a (mon.0) 9680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:38 smithi115 bash[45495]: audit 2022-04-22T20:31:37.683190+0000 mon.a (mon.0) 9681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:38 smithi115 bash[46934]: audit 2022-04-22T20:31:37.601204+0000 mon.a (mon.0) 9680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:38 smithi115 bash[46934]: audit 2022-04-22T20:31:37.683190+0000 mon.a (mon.0) 9681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:39 smithi121 bash[36902]: cluster 2022-04-22T20:31:37.632599+0000 mgr.y (mgr.34107) 6349 : cluster [DBG] pgmap v7508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:39 smithi121 bash[36902]: audit 2022-04-22T20:31:39.147243+0000 mon.a (mon.0) 9682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:39 smithi115 bash[45495]: cluster 2022-04-22T20:31:37.632599+0000 mgr.y (mgr.34107) 6349 : cluster [DBG] pgmap v7508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:39 smithi115 bash[45495]: audit 2022-04-22T20:31:39.147243+0000 mon.a (mon.0) 9682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:39 smithi115 bash[46934]: cluster 2022-04-22T20:31:37.632599+0000 mgr.y (mgr.34107) 6349 : cluster [DBG] pgmap v7508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:39 smithi115 bash[46934]: audit 2022-04-22T20:31:39.147243+0000 mon.a (mon.0) 9682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:41.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:41 smithi121 bash[36902]: cluster 2022-04-22T20:31:39.633765+0000 mgr.y (mgr.34107) 6350 : cluster [DBG] pgmap v7509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:41.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:41 smithi121 bash[36902]: audit 2022-04-22T20:31:41.145852+0000 mon.a (mon.0) 9683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:41 smithi115 bash[45495]: cluster 2022-04-22T20:31:39.633765+0000 mgr.y (mgr.34107) 6350 : cluster [DBG] pgmap v7509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:41 smithi115 bash[45495]: audit 2022-04-22T20:31:41.145852+0000 mon.a (mon.0) 9683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:41 smithi115 bash[46934]: cluster 2022-04-22T20:31:39.633765+0000 mgr.y (mgr.34107) 6350 : cluster [DBG] pgmap v7509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:41 smithi115 bash[46934]: audit 2022-04-22T20:31:41.145852+0000 mon.a (mon.0) 9683 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:42 smithi121 bash[36902]: audit 2022-04-22T20:31:41.487159+0000 mon.a (mon.0) 9684 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:42 smithi121 bash[36902]: cluster 2022-04-22T20:31:41.634755+0000 mgr.y (mgr.34107) 6351 : cluster [DBG] pgmap v7510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:42.328Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:42 smithi115 bash[45495]: audit 2022-04-22T20:31:41.487159+0000 mon.a (mon.0) 9684 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:42 smithi115 bash[45495]: cluster 2022-04-22T20:31:41.634755+0000 mgr.y (mgr.34107) 6351 : cluster [DBG] pgmap v7510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:42 smithi115 bash[46934]: audit 2022-04-22T20:31:41.487159+0000 mon.a (mon.0) 9684 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:31:42.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:42 smithi115 bash[46934]: cluster 2022-04-22T20:31:41.634755+0000 mgr.y (mgr.34107) 6351 : cluster [DBG] pgmap v7510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:43 smithi121 bash[36902]: audit 2022-04-22T20:31:42.604784+0000 mon.a (mon.0) 9685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:43 smithi121 bash[36902]: audit 2022-04-22T20:31:42.686383+0000 mon.a (mon.0) 9686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:43 smithi115 bash[45495]: audit 2022-04-22T20:31:42.604784+0000 mon.a (mon.0) 9685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:43.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:43 smithi115 bash[45495]: audit 2022-04-22T20:31:42.686383+0000 mon.a (mon.0) 9686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:43 smithi115 bash[46934]: audit 2022-04-22T20:31:42.604784+0000 mon.a (mon.0) 9685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:43 smithi115 bash[46934]: audit 2022-04-22T20:31:42.686383+0000 mon.a (mon.0) 9686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:44.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:44.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:45 smithi121 bash[36902]: cluster 2022-04-22T20:31:43.635994+0000 mgr.y (mgr.34107) 6352 : cluster [DBG] pgmap v7511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:45 smithi121 bash[36902]: audit 2022-04-22T20:31:44.149785+0000 mon.a (mon.0) 9687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:45 smithi115 bash[45495]: cluster 2022-04-22T20:31:43.635994+0000 mgr.y (mgr.34107) 6352 : cluster [DBG] pgmap v7511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:45 smithi115 bash[45495]: audit 2022-04-22T20:31:44.149785+0000 mon.a (mon.0) 9687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:45 smithi115 bash[46934]: cluster 2022-04-22T20:31:43.635994+0000 mgr.y (mgr.34107) 6352 : cluster [DBG] pgmap v7511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:45 smithi115 bash[46934]: audit 2022-04-22T20:31:44.149785+0000 mon.a (mon.0) 9687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:47 smithi121 bash[36902]: cluster 2022-04-22T20:31:45.637091+0000 mgr.y (mgr.34107) 6353 : cluster [DBG] pgmap v7512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:47 smithi115 bash[45495]: cluster 2022-04-22T20:31:45.637091+0000 mgr.y (mgr.34107) 6353 : cluster [DBG] pgmap v7512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:47 smithi115 bash[46934]: cluster 2022-04-22T20:31:45.637091+0000 mgr.y (mgr.34107) 6353 : cluster [DBG] pgmap v7512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:48 smithi121 bash[36902]: audit 2022-04-22T20:31:47.608563+0000 mon.a (mon.0) 9688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:48 smithi121 bash[36902]: audit 2022-04-22T20:31:47.689396+0000 mon.a (mon.0) 9689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:48 smithi115 bash[45495]: audit 2022-04-22T20:31:47.608563+0000 mon.a (mon.0) 9688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:48 smithi115 bash[45495]: audit 2022-04-22T20:31:47.689396+0000 mon.a (mon.0) 9689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:48 smithi115 bash[46934]: audit 2022-04-22T20:31:47.608563+0000 mon.a (mon.0) 9688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:48 smithi115 bash[46934]: audit 2022-04-22T20:31:47.689396+0000 mon.a (mon.0) 9689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:49 smithi121 bash[36902]: cluster 2022-04-22T20:31:47.638156+0000 mgr.y (mgr.34107) 6354 : cluster [DBG] pgmap v7513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:49 smithi121 bash[36902]: audit 2022-04-22T20:31:49.152167+0000 mon.a (mon.0) 9690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:49 smithi115 bash[45495]: cluster 2022-04-22T20:31:47.638156+0000 mgr.y (mgr.34107) 6354 : cluster [DBG] pgmap v7513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:49 smithi115 bash[45495]: audit 2022-04-22T20:31:49.152167+0000 mon.a (mon.0) 9690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:49 smithi115 bash[46934]: cluster 2022-04-22T20:31:47.638156+0000 mgr.y (mgr.34107) 6354 : cluster [DBG] pgmap v7513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:49 smithi115 bash[46934]: audit 2022-04-22T20:31:49.152167+0000 mon.a (mon.0) 9690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:50.924 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:31:51.262 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:31:51.262 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:31:51.262 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:31:51.262 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:31:51.263 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 10s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:31:51.263 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:31:51.263 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:31:51.263 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:31:51.263 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10s ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:31:51.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:31:51.264 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:31:51.264 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 10s ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:31:51.264 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 10s ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 10s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 10s ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 10s ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:31:51.265 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 10s ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:31:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:51 smithi121 bash[36902]: cluster 2022-04-22T20:31:49.639215+0000 mgr.y (mgr.34107) 6355 : cluster [DBG] pgmap v7514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:51 smithi115 bash[45495]: cluster 2022-04-22T20:31:49.639215+0000 mgr.y (mgr.34107) 6355 : cluster [DBG] pgmap v7514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:51 smithi115 bash[46934]: cluster 2022-04-22T20:31:49.639215+0000 mgr.y (mgr.34107) 6355 : cluster [DBG] pgmap v7514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:51.663 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:31:51.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:31:51.665 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:31:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:52 smithi121 bash[36902]: audit 2022-04-22T20:31:50.901588+0000 mgr.y (mgr.34107) 6356 : audit [DBG] from='client.51925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:52 smithi121 bash[36902]: audit 2022-04-22T20:31:51.236212+0000 mgr.y (mgr.34107) 6357 : audit [DBG] from='client.42000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:52 smithi121 bash[36902]: audit 2022-04-22T20:31:51.655998+0000 mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.115:0/1152973804' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:52.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:52.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:52.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[45495]: audit 2022-04-22T20:31:50.901588+0000 mgr.y (mgr.34107) 6356 : audit [DBG] from='client.51925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[45495]: audit 2022-04-22T20:31:51.236212+0000 mgr.y (mgr.34107) 6357 : audit [DBG] from='client.42000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[45495]: audit 2022-04-22T20:31:51.655998+0000 mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.115:0/1152973804' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[46934]: audit 2022-04-22T20:31:50.901588+0000 mgr.y (mgr.34107) 6356 : audit [DBG] from='client.51925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[46934]: audit 2022-04-22T20:31:51.236212+0000 mgr.y (mgr.34107) 6357 : audit [DBG] from='client.42000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:31:52.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:52 smithi115 bash[46934]: audit 2022-04-22T20:31:51.655998+0000 mon.c (mon.1) 248 : audit [DBG] from='client.? 172.21.15.115:0/1152973804' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:31:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:53 smithi121 bash[36902]: cluster 2022-04-22T20:31:51.640261+0000 mgr.y (mgr.34107) 6358 : cluster [DBG] pgmap v7515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:53 smithi121 bash[36902]: audit 2022-04-22T20:31:52.612110+0000 mon.a (mon.0) 9691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:53 smithi121 bash[36902]: audit 2022-04-22T20:31:52.692427+0000 mon.a (mon.0) 9692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[45495]: cluster 2022-04-22T20:31:51.640261+0000 mgr.y (mgr.34107) 6358 : cluster [DBG] pgmap v7515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[45495]: audit 2022-04-22T20:31:52.612110+0000 mon.a (mon.0) 9691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[45495]: audit 2022-04-22T20:31:52.692427+0000 mon.a (mon.0) 9692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[46934]: cluster 2022-04-22T20:31:51.640261+0000 mgr.y (mgr.34107) 6358 : cluster [DBG] pgmap v7515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[46934]: audit 2022-04-22T20:31:52.612110+0000 mon.a (mon.0) 9691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:53 smithi115 bash[46934]: audit 2022-04-22T20:31:52.692427+0000 mon.a (mon.0) 9692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:54.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:31:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:31:54.209Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:31:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:55 smithi121 bash[36902]: cluster 2022-04-22T20:31:53.641433+0000 mgr.y (mgr.34107) 6359 : cluster [DBG] pgmap v7516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:55 smithi121 bash[36902]: audit 2022-04-22T20:31:54.154766+0000 mon.a (mon.0) 9693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:55 smithi115 bash[45495]: cluster 2022-04-22T20:31:53.641433+0000 mgr.y (mgr.34107) 6359 : cluster [DBG] pgmap v7516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:55 smithi115 bash[45495]: audit 2022-04-22T20:31:54.154766+0000 mon.a (mon.0) 9693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:55 smithi115 bash[46934]: cluster 2022-04-22T20:31:53.641433+0000 mgr.y (mgr.34107) 6359 : cluster [DBG] pgmap v7516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:55 smithi115 bash[46934]: audit 2022-04-22T20:31:54.154766+0000 mon.a (mon.0) 9693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:57 smithi121 bash[36902]: cluster 2022-04-22T20:31:55.642561+0000 mgr.y (mgr.34107) 6360 : cluster [DBG] pgmap v7517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:57 smithi115 bash[46934]: cluster 2022-04-22T20:31:55.642561+0000 mgr.y (mgr.34107) 6360 : cluster [DBG] pgmap v7517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:57 smithi115 bash[45495]: cluster 2022-04-22T20:31:55.642561+0000 mgr.y (mgr.34107) 6360 : cluster [DBG] pgmap v7517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:58 smithi121 bash[36902]: audit 2022-04-22T20:31:57.615736+0000 mon.a (mon.0) 9694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:58 smithi121 bash[36902]: audit 2022-04-22T20:31:57.695501+0000 mon.a (mon.0) 9695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:58 smithi115 bash[45495]: audit 2022-04-22T20:31:57.615736+0000 mon.a (mon.0) 9694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:58 smithi115 bash[45495]: audit 2022-04-22T20:31:57.695501+0000 mon.a (mon.0) 9695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:58 smithi115 bash[46934]: audit 2022-04-22T20:31:57.615736+0000 mon.a (mon.0) 9694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:31:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:58 smithi115 bash[46934]: audit 2022-04-22T20:31:57.695501+0000 mon.a (mon.0) 9695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:31:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:59 smithi121 bash[36902]: cluster 2022-04-22T20:31:57.643711+0000 mgr.y (mgr.34107) 6361 : cluster [DBG] pgmap v7518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:31:59 smithi121 bash[36902]: audit 2022-04-22T20:31:59.157186+0000 mon.a (mon.0) 9696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:59 smithi115 bash[45495]: cluster 2022-04-22T20:31:57.643711+0000 mgr.y (mgr.34107) 6361 : cluster [DBG] pgmap v7518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:59.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:31:59 smithi115 bash[45495]: audit 2022-04-22T20:31:59.157186+0000 mon.a (mon.0) 9696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:31:59.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:59 smithi115 bash[46934]: cluster 2022-04-22T20:31:57.643711+0000 mgr.y (mgr.34107) 6361 : cluster [DBG] pgmap v7518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:31:59.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:31:59 smithi115 bash[46934]: audit 2022-04-22T20:31:59.157186+0000 mon.a (mon.0) 9696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:01 smithi121 bash[36902]: cluster 2022-04-22T20:31:59.644793+0000 mgr.y (mgr.34107) 6362 : cluster [DBG] pgmap v7519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:01 smithi115 bash[45495]: cluster 2022-04-22T20:31:59.644793+0000 mgr.y (mgr.34107) 6362 : cluster [DBG] pgmap v7519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:01 smithi115 bash[46934]: cluster 2022-04-22T20:31:59.644793+0000 mgr.y (mgr.34107) 6362 : cluster [DBG] pgmap v7519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:02.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:02.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:03 smithi121 bash[36902]: cluster 2022-04-22T20:32:01.645886+0000 mgr.y (mgr.34107) 6363 : cluster [DBG] pgmap v7520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:03 smithi121 bash[36902]: audit 2022-04-22T20:32:02.619514+0000 mon.a (mon.0) 9697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:03 smithi121 bash[36902]: audit 2022-04-22T20:32:02.698738+0000 mon.a (mon.0) 9698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[45495]: cluster 2022-04-22T20:32:01.645886+0000 mgr.y (mgr.34107) 6363 : cluster [DBG] pgmap v7520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[45495]: audit 2022-04-22T20:32:02.619514+0000 mon.a (mon.0) 9697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[45495]: audit 2022-04-22T20:32:02.698738+0000 mon.a (mon.0) 9698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[46934]: cluster 2022-04-22T20:32:01.645886+0000 mgr.y (mgr.34107) 6363 : cluster [DBG] pgmap v7520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[46934]: audit 2022-04-22T20:32:02.619514+0000 mon.a (mon.0) 9697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:03 smithi115 bash[46934]: audit 2022-04-22T20:32:02.698738+0000 mon.a (mon.0) 9698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:04.211Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:05 smithi121 bash[36902]: cluster 2022-04-22T20:32:03.647086+0000 mgr.y (mgr.34107) 6364 : cluster [DBG] pgmap v7521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:05.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:05 smithi121 bash[36902]: audit 2022-04-22T20:32:04.159804+0000 mon.a (mon.0) 9699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:05 smithi115 bash[45495]: cluster 2022-04-22T20:32:03.647086+0000 mgr.y (mgr.34107) 6364 : cluster [DBG] pgmap v7521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:05 smithi115 bash[45495]: audit 2022-04-22T20:32:04.159804+0000 mon.a (mon.0) 9699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:05 smithi115 bash[46934]: cluster 2022-04-22T20:32:03.647086+0000 mgr.y (mgr.34107) 6364 : cluster [DBG] pgmap v7521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:05 smithi115 bash[46934]: audit 2022-04-22T20:32:04.159804+0000 mon.a (mon.0) 9699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:07 smithi121 bash[36902]: cluster 2022-04-22T20:32:05.648199+0000 mgr.y (mgr.34107) 6365 : cluster [DBG] pgmap v7522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:07 smithi115 bash[45495]: cluster 2022-04-22T20:32:05.648199+0000 mgr.y (mgr.34107) 6365 : cluster [DBG] pgmap v7522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:07 smithi115 bash[46934]: cluster 2022-04-22T20:32:05.648199+0000 mgr.y (mgr.34107) 6365 : cluster [DBG] pgmap v7522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:08 smithi121 bash[36902]: audit 2022-04-22T20:32:07.623083+0000 mon.a (mon.0) 9700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:08 smithi121 bash[36902]: audit 2022-04-22T20:32:07.701521+0000 mon.a (mon.0) 9701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:08 smithi115 bash[45495]: audit 2022-04-22T20:32:07.623083+0000 mon.a (mon.0) 9700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:08 smithi115 bash[45495]: audit 2022-04-22T20:32:07.701521+0000 mon.a (mon.0) 9701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:08 smithi115 bash[46934]: audit 2022-04-22T20:32:07.623083+0000 mon.a (mon.0) 9700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:08 smithi115 bash[46934]: audit 2022-04-22T20:32:07.701521+0000 mon.a (mon.0) 9701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:09 smithi121 bash[36902]: cluster 2022-04-22T20:32:07.649340+0000 mgr.y (mgr.34107) 6366 : cluster [DBG] pgmap v7523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:09 smithi115 bash[45495]: cluster 2022-04-22T20:32:07.649340+0000 mgr.y (mgr.34107) 6366 : cluster [DBG] pgmap v7523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:09 smithi115 bash[46934]: cluster 2022-04-22T20:32:07.649340+0000 mgr.y (mgr.34107) 6366 : cluster [DBG] pgmap v7523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:10 smithi121 bash[36902]: audit 2022-04-22T20:32:09.167115+0000 mon.a (mon.0) 9702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:10 smithi115 bash[45495]: audit 2022-04-22T20:32:09.167115+0000 mon.a (mon.0) 9702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:10 smithi115 bash[46934]: audit 2022-04-22T20:32:09.167115+0000 mon.a (mon.0) 9702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:11 smithi121 bash[36902]: cluster 2022-04-22T20:32:09.650479+0000 mgr.y (mgr.34107) 6367 : cluster [DBG] pgmap v7524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:11.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:11 smithi115 bash[45495]: cluster 2022-04-22T20:32:09.650479+0000 mgr.y (mgr.34107) 6367 : cluster [DBG] pgmap v7524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:11.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:11 smithi115 bash[46934]: cluster 2022-04-22T20:32:09.650479+0000 mgr.y (mgr.34107) 6367 : cluster [DBG] pgmap v7524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:12.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:13 smithi121 bash[36902]: cluster 2022-04-22T20:32:11.651598+0000 mgr.y (mgr.34107) 6368 : cluster [DBG] pgmap v7525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:13.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:13 smithi121 bash[36902]: audit 2022-04-22T20:32:12.626877+0000 mon.a (mon.0) 9703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:13.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:13 smithi121 bash[36902]: audit 2022-04-22T20:32:12.704565+0000 mon.a (mon.0) 9704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:13.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[46934]: cluster 2022-04-22T20:32:11.651598+0000 mgr.y (mgr.34107) 6368 : cluster [DBG] pgmap v7525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:13.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[46934]: audit 2022-04-22T20:32:12.626877+0000 mon.a (mon.0) 9703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:13.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[46934]: audit 2022-04-22T20:32:12.704565+0000 mon.a (mon.0) 9704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:13.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[45495]: cluster 2022-04-22T20:32:11.651598+0000 mgr.y (mgr.34107) 6368 : cluster [DBG] pgmap v7525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:13.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[45495]: audit 2022-04-22T20:32:12.626877+0000 mon.a (mon.0) 9703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:13.566 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:13 smithi115 bash[45495]: audit 2022-04-22T20:32:12.704565+0000 mon.a (mon.0) 9704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:14.213Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:15 smithi121 bash[36902]: cluster 2022-04-22T20:32:13.652687+0000 mgr.y (mgr.34107) 6369 : cluster [DBG] pgmap v7526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:15 smithi121 bash[36902]: audit 2022-04-22T20:32:14.165319+0000 mon.a (mon.0) 9705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:15 smithi115 bash[45495]: cluster 2022-04-22T20:32:13.652687+0000 mgr.y (mgr.34107) 6369 : cluster [DBG] pgmap v7526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:15 smithi115 bash[45495]: audit 2022-04-22T20:32:14.165319+0000 mon.a (mon.0) 9705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:15 smithi115 bash[46934]: cluster 2022-04-22T20:32:13.652687+0000 mgr.y (mgr.34107) 6369 : cluster [DBG] pgmap v7526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:15 smithi115 bash[46934]: audit 2022-04-22T20:32:14.165319+0000 mon.a (mon.0) 9705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:16 smithi121 bash[36902]: audit 2022-04-22T20:32:15.413552+0000 mon.a (mon.0) 9706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:32:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:16 smithi121 bash[36902]: audit 2022-04-22T20:32:15.414023+0000 mon.a (mon.0) 9707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:32:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:16 smithi115 bash[45495]: audit 2022-04-22T20:32:15.413552+0000 mon.a (mon.0) 9706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:32:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:16 smithi115 bash[45495]: audit 2022-04-22T20:32:15.414023+0000 mon.a (mon.0) 9707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:32:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:16 smithi115 bash[46934]: audit 2022-04-22T20:32:15.413552+0000 mon.a (mon.0) 9706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:32:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:16 smithi115 bash[46934]: audit 2022-04-22T20:32:15.414023+0000 mon.a (mon.0) 9707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:32:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:17 smithi121 bash[36902]: cluster 2022-04-22T20:32:15.653859+0000 mgr.y (mgr.34107) 6370 : cluster [DBG] pgmap v7527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:17 smithi115 bash[45495]: cluster 2022-04-22T20:32:15.653859+0000 mgr.y (mgr.34107) 6370 : cluster [DBG] pgmap v7527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:17 smithi115 bash[46934]: cluster 2022-04-22T20:32:15.653859+0000 mgr.y (mgr.34107) 6370 : cluster [DBG] pgmap v7527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:18 smithi121 bash[36902]: audit 2022-04-22T20:32:17.630577+0000 mon.a (mon.0) 9708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:18 smithi121 bash[36902]: audit 2022-04-22T20:32:17.707682+0000 mon.a (mon.0) 9709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:18 smithi115 bash[45495]: audit 2022-04-22T20:32:17.630577+0000 mon.a (mon.0) 9708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:18 smithi115 bash[45495]: audit 2022-04-22T20:32:17.707682+0000 mon.a (mon.0) 9709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:18 smithi115 bash[46934]: audit 2022-04-22T20:32:17.630577+0000 mon.a (mon.0) 9708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:18 smithi115 bash[46934]: audit 2022-04-22T20:32:17.707682+0000 mon.a (mon.0) 9709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:19 smithi121 bash[36902]: cluster 2022-04-22T20:32:17.654931+0000 mgr.y (mgr.34107) 6371 : cluster [DBG] pgmap v7528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:19 smithi115 bash[45495]: cluster 2022-04-22T20:32:17.654931+0000 mgr.y (mgr.34107) 6371 : cluster [DBG] pgmap v7528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:19.680 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:19 smithi115 bash[46934]: cluster 2022-04-22T20:32:17.654931+0000 mgr.y (mgr.34107) 6371 : cluster [DBG] pgmap v7528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:20 smithi121 bash[36902]: audit 2022-04-22T20:32:19.168242+0000 mon.a (mon.0) 9710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:20 smithi115 bash[45495]: audit 2022-04-22T20:32:19.168242+0000 mon.a (mon.0) 9710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:20 smithi115 bash[46934]: audit 2022-04-22T20:32:19.168242+0000 mon.a (mon.0) 9710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:21 smithi121 bash[36902]: cluster 2022-04-22T20:32:19.656077+0000 mgr.y (mgr.34107) 6372 : cluster [DBG] pgmap v7529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:21 smithi115 bash[45495]: cluster 2022-04-22T20:32:19.656077+0000 mgr.y (mgr.34107) 6372 : cluster [DBG] pgmap v7529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:21 smithi115 bash[46934]: cluster 2022-04-22T20:32:19.656077+0000 mgr.y (mgr.34107) 6372 : cluster [DBG] pgmap v7529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:22.030 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:32:22.377 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 9m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 41s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 41s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 41s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 485M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 331M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:32:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 41s ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:32:22.379 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 214M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:32:22.379 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 9m ago 3h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:32:22.379 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 41s ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:32:22.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 705M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:32:22.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 785M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 531M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 489M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (2h) 41s ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 41s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 41s ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 41s ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:32:22.380 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 41s ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:32:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:32:22.783 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:32:22.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:32:22.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:22.784 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:32:22.785 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:32:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:22.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:23 smithi121 bash[36902]: cluster 2022-04-22T20:32:21.657203+0000 mgr.y (mgr.34107) 6373 : cluster [DBG] pgmap v7530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:23 smithi121 bash[36902]: audit 2022-04-22T20:32:22.005829+0000 mgr.y (mgr.34107) 6374 : audit [DBG] from='client.51943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:23 smithi121 bash[36902]: audit 2022-04-22T20:32:22.634006+0000 mon.a (mon.0) 9711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:23 smithi121 bash[36902]: audit 2022-04-22T20:32:22.710557+0000 mon.a (mon.0) 9712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:23 smithi121 bash[36902]: audit 2022-04-22T20:32:22.775601+0000 mon.c (mon.1) 249 : audit [DBG] from='client.? 172.21.15.115:0/4207768025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[46934]: cluster 2022-04-22T20:32:21.657203+0000 mgr.y (mgr.34107) 6373 : cluster [DBG] pgmap v7530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[46934]: audit 2022-04-22T20:32:22.005829+0000 mgr.y (mgr.34107) 6374 : audit [DBG] from='client.51943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[46934]: audit 2022-04-22T20:32:22.634006+0000 mon.a (mon.0) 9711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[46934]: audit 2022-04-22T20:32:22.710557+0000 mon.a (mon.0) 9712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[46934]: audit 2022-04-22T20:32:22.775601+0000 mon.c (mon.1) 249 : audit [DBG] from='client.? 172.21.15.115:0/4207768025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[45495]: cluster 2022-04-22T20:32:21.657203+0000 mgr.y (mgr.34107) 6373 : cluster [DBG] pgmap v7530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[45495]: audit 2022-04-22T20:32:22.005829+0000 mgr.y (mgr.34107) 6374 : audit [DBG] from='client.51943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[45495]: audit 2022-04-22T20:32:22.634006+0000 mon.a (mon.0) 9711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:23.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[45495]: audit 2022-04-22T20:32:22.710557+0000 mon.a (mon.0) 9712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:23.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:23 smithi115 bash[45495]: audit 2022-04-22T20:32:22.775601+0000 mon.c (mon.1) 249 : audit [DBG] from='client.? 172.21.15.115:0/4207768025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:24 smithi121 bash[36902]: audit 2022-04-22T20:32:22.351563+0000 mgr.y (mgr.34107) 6375 : audit [DBG] from='client.51949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:24 smithi121 bash[36902]: audit 2022-04-22T20:32:24.171727+0000 mon.a (mon.0) 9713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:24.210Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:24 smithi115 bash[45495]: audit 2022-04-22T20:32:22.351563+0000 mgr.y (mgr.34107) 6375 : audit [DBG] from='client.51949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:24.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:24 smithi115 bash[45495]: audit 2022-04-22T20:32:24.171727+0000 mon.a (mon.0) 9713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:24 smithi115 bash[46934]: audit 2022-04-22T20:32:22.351563+0000 mgr.y (mgr.34107) 6375 : audit [DBG] from='client.51949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:24 smithi115 bash[46934]: audit 2022-04-22T20:32:24.171727+0000 mon.a (mon.0) 9713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:25 smithi121 bash[36902]: cluster 2022-04-22T20:32:23.658358+0000 mgr.y (mgr.34107) 6376 : cluster [DBG] pgmap v7531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:25 smithi115 bash[45495]: cluster 2022-04-22T20:32:23.658358+0000 mgr.y (mgr.34107) 6376 : cluster [DBG] pgmap v7531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:25 smithi115 bash[46934]: cluster 2022-04-22T20:32:23.658358+0000 mgr.y (mgr.34107) 6376 : cluster [DBG] pgmap v7531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:27 smithi121 bash[36902]: cluster 2022-04-22T20:32:25.659629+0000 mgr.y (mgr.34107) 6377 : cluster [DBG] pgmap v7532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:27 smithi115 bash[45495]: cluster 2022-04-22T20:32:25.659629+0000 mgr.y (mgr.34107) 6377 : cluster [DBG] pgmap v7532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:27 smithi115 bash[46934]: cluster 2022-04-22T20:32:25.659629+0000 mgr.y (mgr.34107) 6377 : cluster [DBG] pgmap v7532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:28 smithi121 bash[36902]: audit 2022-04-22T20:32:27.637529+0000 mon.a (mon.0) 9714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:28 smithi121 bash[36902]: audit 2022-04-22T20:32:27.713198+0000 mon.a (mon.0) 9715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:28 smithi115 bash[45495]: audit 2022-04-22T20:32:27.637529+0000 mon.a (mon.0) 9714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:28 smithi115 bash[45495]: audit 2022-04-22T20:32:27.713198+0000 mon.a (mon.0) 9715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:28 smithi115 bash[46934]: audit 2022-04-22T20:32:27.637529+0000 mon.a (mon.0) 9714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:28 smithi115 bash[46934]: audit 2022-04-22T20:32:27.713198+0000 mon.a (mon.0) 9715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:29 smithi121 bash[36902]: cluster 2022-04-22T20:32:27.660756+0000 mgr.y (mgr.34107) 6378 : cluster [DBG] pgmap v7533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:29 smithi121 bash[36902]: audit 2022-04-22T20:32:29.174121+0000 mon.a (mon.0) 9716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:29 smithi115 bash[45495]: cluster 2022-04-22T20:32:27.660756+0000 mgr.y (mgr.34107) 6378 : cluster [DBG] pgmap v7533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:29 smithi115 bash[45495]: audit 2022-04-22T20:32:29.174121+0000 mon.a (mon.0) 9716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:29 smithi115 bash[46934]: cluster 2022-04-22T20:32:27.660756+0000 mgr.y (mgr.34107) 6378 : cluster [DBG] pgmap v7533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:29 smithi115 bash[46934]: audit 2022-04-22T20:32:29.174121+0000 mon.a (mon.0) 9716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:31 smithi121 bash[36902]: cluster 2022-04-22T20:32:29.661875+0000 mgr.y (mgr.34107) 6379 : cluster [DBG] pgmap v7534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:31.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:31 smithi115 bash[46934]: cluster 2022-04-22T20:32:29.661875+0000 mgr.y (mgr.34107) 6379 : cluster [DBG] pgmap v7534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:31.583 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:31 smithi115 bash[45495]: cluster 2022-04-22T20:32:29.661875+0000 mgr.y (mgr.34107) 6379 : cluster [DBG] pgmap v7534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:32.331Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:33 smithi121 bash[36902]: cluster 2022-04-22T20:32:31.662970+0000 mgr.y (mgr.34107) 6380 : cluster [DBG] pgmap v7535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:33 smithi121 bash[36902]: audit 2022-04-22T20:32:32.641295+0000 mon.a (mon.0) 9717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:33 smithi121 bash[36902]: audit 2022-04-22T20:32:32.716300+0000 mon.a (mon.0) 9718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[45495]: cluster 2022-04-22T20:32:31.662970+0000 mgr.y (mgr.34107) 6380 : cluster [DBG] pgmap v7535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[45495]: audit 2022-04-22T20:32:32.641295+0000 mon.a (mon.0) 9717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[45495]: audit 2022-04-22T20:32:32.716300+0000 mon.a (mon.0) 9718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[46934]: cluster 2022-04-22T20:32:31.662970+0000 mgr.y (mgr.34107) 6380 : cluster [DBG] pgmap v7535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[46934]: audit 2022-04-22T20:32:32.641295+0000 mon.a (mon.0) 9717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:33 smithi115 bash[46934]: audit 2022-04-22T20:32:32.716300+0000 mon.a (mon.0) 9718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:34.214Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:35 smithi121 bash[36902]: cluster 2022-04-22T20:32:33.664206+0000 mgr.y (mgr.34107) 6381 : cluster [DBG] pgmap v7536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:35 smithi121 bash[36902]: audit 2022-04-22T20:32:34.177226+0000 mon.a (mon.0) 9719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:35 smithi115 bash[45495]: cluster 2022-04-22T20:32:33.664206+0000 mgr.y (mgr.34107) 6381 : cluster [DBG] pgmap v7536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:35.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:35 smithi115 bash[45495]: audit 2022-04-22T20:32:34.177226+0000 mon.a (mon.0) 9719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:35 smithi115 bash[46934]: cluster 2022-04-22T20:32:33.664206+0000 mgr.y (mgr.34107) 6381 : cluster [DBG] pgmap v7536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:35.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:35 smithi115 bash[46934]: audit 2022-04-22T20:32:34.177226+0000 mon.a (mon.0) 9719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:37 smithi121 bash[36902]: cluster 2022-04-22T20:32:35.665285+0000 mgr.y (mgr.34107) 6382 : cluster [DBG] pgmap v7537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:37.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:37 smithi115 bash[45495]: cluster 2022-04-22T20:32:35.665285+0000 mgr.y (mgr.34107) 6382 : cluster [DBG] pgmap v7537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:37.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:37 smithi115 bash[46934]: cluster 2022-04-22T20:32:35.665285+0000 mgr.y (mgr.34107) 6382 : cluster [DBG] pgmap v7537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:38 smithi121 bash[36902]: audit 2022-04-22T20:32:37.644997+0000 mon.a (mon.0) 9720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:38 smithi121 bash[36902]: audit 2022-04-22T20:32:37.719193+0000 mon.a (mon.0) 9721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:38 smithi115 bash[45495]: audit 2022-04-22T20:32:37.644997+0000 mon.a (mon.0) 9720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:38 smithi115 bash[45495]: audit 2022-04-22T20:32:37.719193+0000 mon.a (mon.0) 9721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:38 smithi115 bash[46934]: audit 2022-04-22T20:32:37.644997+0000 mon.a (mon.0) 9720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:38 smithi115 bash[46934]: audit 2022-04-22T20:32:37.719193+0000 mon.a (mon.0) 9721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:39 smithi121 bash[36902]: cluster 2022-04-22T20:32:37.666346+0000 mgr.y (mgr.34107) 6383 : cluster [DBG] pgmap v7538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:39 smithi115 bash[45495]: cluster 2022-04-22T20:32:37.666346+0000 mgr.y (mgr.34107) 6383 : cluster [DBG] pgmap v7538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:39 smithi115 bash[46934]: cluster 2022-04-22T20:32:37.666346+0000 mgr.y (mgr.34107) 6383 : cluster [DBG] pgmap v7538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:40 smithi121 bash[36902]: audit 2022-04-22T20:32:39.179973+0000 mon.a (mon.0) 9722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:40.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:40 smithi115 bash[45495]: audit 2022-04-22T20:32:39.179973+0000 mon.a (mon.0) 9722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:40.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:40 smithi115 bash[46934]: audit 2022-04-22T20:32:39.179973+0000 mon.a (mon.0) 9722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:41.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:41 smithi121 bash[36902]: cluster 2022-04-22T20:32:39.667938+0000 mgr.y (mgr.34107) 6384 : cluster [DBG] pgmap v7539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:41.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:41 smithi115 bash[46934]: cluster 2022-04-22T20:32:39.667938+0000 mgr.y (mgr.34107) 6384 : cluster [DBG] pgmap v7539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:41.528 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:41 smithi115 bash[45495]: cluster 2022-04-22T20:32:39.667938+0000 mgr.y (mgr.34107) 6384 : cluster [DBG] pgmap v7539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:42 smithi121 bash[36902]: audit 2022-04-22T20:32:41.496355+0000 mon.a (mon.0) 9723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:32:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:42 smithi121 bash[36902]: audit 2022-04-22T20:32:41.499361+0000 mon.a (mon.0) 9724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:32:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:42 smithi121 bash[36902]: audit 2022-04-22T20:32:41.501198+0000 mon.a (mon.0) 9725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:32:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:42 smithi121 bash[36902]: audit 2022-04-22T20:32:41.829324+0000 mon.a (mon.0) 9726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:42 smithi121 bash[36902]: audit 2022-04-22T20:32:41.842736+0000 mon.a (mon.0) 9727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:42.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[46934]: audit 2022-04-22T20:32:41.496355+0000 mon.a (mon.0) 9723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:32:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[46934]: audit 2022-04-22T20:32:41.499361+0000 mon.a (mon.0) 9724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:32:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[46934]: audit 2022-04-22T20:32:41.501198+0000 mon.a (mon.0) 9725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:32:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[46934]: audit 2022-04-22T20:32:41.829324+0000 mon.a (mon.0) 9726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:42.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[46934]: audit 2022-04-22T20:32:41.842736+0000 mon.a (mon.0) 9727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:42.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:42.330Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[45495]: audit 2022-04-22T20:32:41.496355+0000 mon.a (mon.0) 9723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:32:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[45495]: audit 2022-04-22T20:32:41.499361+0000 mon.a (mon.0) 9724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:32:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[45495]: audit 2022-04-22T20:32:41.501198+0000 mon.a (mon.0) 9725 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:32:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[45495]: audit 2022-04-22T20:32:41.829324+0000 mon.a (mon.0) 9726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:42.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:42 smithi115 bash[45495]: audit 2022-04-22T20:32:41.842736+0000 mon.a (mon.0) 9727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:43 smithi121 bash[36902]: cluster 2022-04-22T20:32:41.668917+0000 mgr.y (mgr.34107) 6385 : cluster [DBG] pgmap v7540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:43 smithi121 bash[36902]: audit 2022-04-22T20:32:42.648382+0000 mon.a (mon.0) 9728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:43 smithi121 bash[36902]: audit 2022-04-22T20:32:42.722344+0000 mon.a (mon.0) 9729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[45495]: cluster 2022-04-22T20:32:41.668917+0000 mgr.y (mgr.34107) 6385 : cluster [DBG] pgmap v7540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:43.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[45495]: audit 2022-04-22T20:32:42.648382+0000 mon.a (mon.0) 9728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:43.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[45495]: audit 2022-04-22T20:32:42.722344+0000 mon.a (mon.0) 9729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:43.670 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[46934]: cluster 2022-04-22T20:32:41.668917+0000 mgr.y (mgr.34107) 6385 : cluster [DBG] pgmap v7540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:43.670 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[46934]: audit 2022-04-22T20:32:42.648382+0000 mon.a (mon.0) 9728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:43.670 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:43 smithi115 bash[46934]: audit 2022-04-22T20:32:42.722344+0000 mon.a (mon.0) 9729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:44.212Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:45 smithi121 bash[36902]: cluster 2022-04-22T20:32:43.670113+0000 mgr.y (mgr.34107) 6386 : cluster [DBG] pgmap v7541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:45 smithi121 bash[36902]: audit 2022-04-22T20:32:44.182535+0000 mon.a (mon.0) 9730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:45 smithi115 bash[45495]: cluster 2022-04-22T20:32:43.670113+0000 mgr.y (mgr.34107) 6386 : cluster [DBG] pgmap v7541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:45.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:45 smithi115 bash[45495]: audit 2022-04-22T20:32:44.182535+0000 mon.a (mon.0) 9730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:45.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:45 smithi115 bash[46934]: cluster 2022-04-22T20:32:43.670113+0000 mgr.y (mgr.34107) 6386 : cluster [DBG] pgmap v7541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:45.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:45 smithi115 bash[46934]: audit 2022-04-22T20:32:44.182535+0000 mon.a (mon.0) 9730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:47 smithi121 bash[36902]: cluster 2022-04-22T20:32:45.671166+0000 mgr.y (mgr.34107) 6387 : cluster [DBG] pgmap v7542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:47 smithi121 bash[36902]: audit 2022-04-22T20:32:46.004516+0000 mon.a (mon.0) 9731 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:47 smithi121 bash[36902]: audit 2022-04-22T20:32:46.324825+0000 mon.a (mon.0) 9732 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:47.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[45495]: cluster 2022-04-22T20:32:45.671166+0000 mgr.y (mgr.34107) 6387 : cluster [DBG] pgmap v7542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[45495]: audit 2022-04-22T20:32:46.004516+0000 mon.a (mon.0) 9731 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[45495]: audit 2022-04-22T20:32:46.324825+0000 mon.a (mon.0) 9732 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[46934]: cluster 2022-04-22T20:32:45.671166+0000 mgr.y (mgr.34107) 6387 : cluster [DBG] pgmap v7542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[46934]: audit 2022-04-22T20:32:46.004516+0000 mon.a (mon.0) 9731 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:47.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:47 smithi115 bash[46934]: audit 2022-04-22T20:32:46.324825+0000 mon.a (mon.0) 9732 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:32:48.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:48 smithi115 bash[46934]: audit 2022-04-22T20:32:47.652009+0000 mon.a (mon.0) 9733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:48 smithi115 bash[46934]: audit 2022-04-22T20:32:47.725449+0000 mon.a (mon.0) 9734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:48 smithi115 bash[45495]: audit 2022-04-22T20:32:47.652009+0000 mon.a (mon.0) 9733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:48.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:48 smithi115 bash[45495]: audit 2022-04-22T20:32:47.725449+0000 mon.a (mon.0) 9734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:48 smithi121 bash[36902]: audit 2022-04-22T20:32:47.652009+0000 mon.a (mon.0) 9733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:48 smithi121 bash[36902]: audit 2022-04-22T20:32:47.725449+0000 mon.a (mon.0) 9734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:49.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:49 smithi115 bash[45495]: cluster 2022-04-22T20:32:47.672354+0000 mgr.y (mgr.34107) 6388 : cluster [DBG] pgmap v7543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:49.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:49 smithi115 bash[46934]: cluster 2022-04-22T20:32:47.672354+0000 mgr.y (mgr.34107) 6388 : cluster [DBG] pgmap v7543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:49 smithi121 bash[36902]: cluster 2022-04-22T20:32:47.672354+0000 mgr.y (mgr.34107) 6388 : cluster [DBG] pgmap v7543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:50 smithi121 bash[36902]: audit 2022-04-22T20:32:49.185411+0000 mon.a (mon.0) 9735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:50 smithi115 bash[45495]: audit 2022-04-22T20:32:49.185411+0000 mon.a (mon.0) 9735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:50 smithi115 bash[46934]: audit 2022-04-22T20:32:49.185411+0000 mon.a (mon.0) 9735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:51 smithi121 bash[36902]: cluster 2022-04-22T20:32:49.673478+0000 mgr.y (mgr.34107) 6389 : cluster [DBG] pgmap v7544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:51 smithi115 bash[45495]: cluster 2022-04-22T20:32:49.673478+0000 mgr.y (mgr.34107) 6389 : cluster [DBG] pgmap v7544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:51 smithi115 bash[46934]: cluster 2022-04-22T20:32:49.673478+0000 mgr.y (mgr.34107) 6389 : cluster [DBG] pgmap v7544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:52.332Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:53.155 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 7s ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 72s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 7s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 72s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 72s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7s ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7s ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:32:53.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 72s ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7s ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 7s ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 72s ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7s ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7s ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 72s ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (2h) 72s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:32:53.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 72s ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:32:53.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 72s ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:32:53.513 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 72s ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:32:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:53 smithi121 bash[36902]: cluster 2022-04-22T20:32:51.674538+0000 mgr.y (mgr.34107) 6390 : cluster [DBG] pgmap v7545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:53 smithi121 bash[36902]: audit 2022-04-22T20:32:52.655691+0000 mon.a (mon.0) 9736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:53 smithi121 bash[36902]: audit 2022-04-22T20:32:52.728612+0000 mon.a (mon.0) 9737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[46934]: cluster 2022-04-22T20:32:51.674538+0000 mgr.y (mgr.34107) 6390 : cluster [DBG] pgmap v7545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[46934]: audit 2022-04-22T20:32:52.655691+0000 mon.a (mon.0) 9736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[46934]: audit 2022-04-22T20:32:52.728612+0000 mon.a (mon.0) 9737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[45495]: cluster 2022-04-22T20:32:51.674538+0000 mgr.y (mgr.34107) 6390 : cluster [DBG] pgmap v7545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[45495]: audit 2022-04-22T20:32:52.655691+0000 mon.a (mon.0) 9736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:53.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:53 smithi115 bash[45495]: audit 2022-04-22T20:32:52.728612+0000 mon.a (mon.0) 9737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:53.911 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:32:53.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:32:53.913 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:32:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:54 smithi121 bash[36902]: audit 2022-04-22T20:32:53.132219+0000 mgr.y (mgr.34107) 6391 : audit [DBG] from='client.51961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:54 smithi121 bash[36902]: audit 2022-04-22T20:32:53.904182+0000 mon.c (mon.1) 250 : audit [DBG] from='client.? 172.21.15.115:0/4182969546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:32:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:32:54.227Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:32:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:54 smithi115 bash[45495]: audit 2022-04-22T20:32:53.132219+0000 mgr.y (mgr.34107) 6391 : audit [DBG] from='client.51961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:54 smithi115 bash[45495]: audit 2022-04-22T20:32:53.904182+0000 mon.c (mon.1) 250 : audit [DBG] from='client.? 172.21.15.115:0/4182969546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:54 smithi115 bash[46934]: audit 2022-04-22T20:32:53.132219+0000 mgr.y (mgr.34107) 6391 : audit [DBG] from='client.51961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:54 smithi115 bash[46934]: audit 2022-04-22T20:32:53.904182+0000 mon.c (mon.1) 250 : audit [DBG] from='client.? 172.21.15.115:0/4182969546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:32:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:55 smithi121 bash[36902]: audit 2022-04-22T20:32:53.481570+0000 mgr.y (mgr.34107) 6392 : audit [DBG] from='client.42036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:55 smithi121 bash[36902]: cluster 2022-04-22T20:32:53.675618+0000 mgr.y (mgr.34107) 6393 : cluster [DBG] pgmap v7546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:55 smithi121 bash[36902]: audit 2022-04-22T20:32:54.188144+0000 mon.a (mon.0) 9738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[45495]: audit 2022-04-22T20:32:53.481570+0000 mgr.y (mgr.34107) 6392 : audit [DBG] from='client.42036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[45495]: cluster 2022-04-22T20:32:53.675618+0000 mgr.y (mgr.34107) 6393 : cluster [DBG] pgmap v7546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[45495]: audit 2022-04-22T20:32:54.188144+0000 mon.a (mon.0) 9738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[46934]: audit 2022-04-22T20:32:53.481570+0000 mgr.y (mgr.34107) 6392 : audit [DBG] from='client.42036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[46934]: cluster 2022-04-22T20:32:53.675618+0000 mgr.y (mgr.34107) 6393 : cluster [DBG] pgmap v7546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:55 smithi115 bash[46934]: audit 2022-04-22T20:32:54.188144+0000 mon.a (mon.0) 9738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:57 smithi121 bash[36902]: cluster 2022-04-22T20:32:55.676690+0000 mgr.y (mgr.34107) 6394 : cluster [DBG] pgmap v7547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:57 smithi115 bash[45495]: cluster 2022-04-22T20:32:55.676690+0000 mgr.y (mgr.34107) 6394 : cluster [DBG] pgmap v7547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:57 smithi115 bash[46934]: cluster 2022-04-22T20:32:55.676690+0000 mgr.y (mgr.34107) 6394 : cluster [DBG] pgmap v7547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:58 smithi121 bash[36902]: audit 2022-04-22T20:32:57.659249+0000 mon.a (mon.0) 9739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:58 smithi121 bash[36902]: audit 2022-04-22T20:32:57.731747+0000 mon.a (mon.0) 9740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:58 smithi115 bash[45495]: audit 2022-04-22T20:32:57.659249+0000 mon.a (mon.0) 9739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:58 smithi115 bash[45495]: audit 2022-04-22T20:32:57.731747+0000 mon.a (mon.0) 9740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:58 smithi115 bash[46934]: audit 2022-04-22T20:32:57.659249+0000 mon.a (mon.0) 9739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:32:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:58 smithi115 bash[46934]: audit 2022-04-22T20:32:57.731747+0000 mon.a (mon.0) 9740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:32:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:59 smithi121 bash[36902]: cluster 2022-04-22T20:32:57.677701+0000 mgr.y (mgr.34107) 6395 : cluster [DBG] pgmap v7548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:32:59 smithi121 bash[36902]: audit 2022-04-22T20:32:59.191044+0000 mon.a (mon.0) 9741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:59 smithi115 bash[45495]: cluster 2022-04-22T20:32:57.677701+0000 mgr.y (mgr.34107) 6395 : cluster [DBG] pgmap v7548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:32:59 smithi115 bash[45495]: audit 2022-04-22T20:32:59.191044+0000 mon.a (mon.0) 9741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:32:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:59 smithi115 bash[46934]: cluster 2022-04-22T20:32:57.677701+0000 mgr.y (mgr.34107) 6395 : cluster [DBG] pgmap v7548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:32:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:32:59 smithi115 bash[46934]: audit 2022-04-22T20:32:59.191044+0000 mon.a (mon.0) 9741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:01 smithi121 bash[36902]: cluster 2022-04-22T20:32:59.678823+0000 mgr.y (mgr.34107) 6396 : cluster [DBG] pgmap v7549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:01 smithi115 bash[45495]: cluster 2022-04-22T20:32:59.678823+0000 mgr.y (mgr.34107) 6396 : cluster [DBG] pgmap v7549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:01 smithi115 bash[46934]: cluster 2022-04-22T20:32:59.678823+0000 mgr.y (mgr.34107) 6396 : cluster [DBG] pgmap v7549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:02.333Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:03 smithi121 bash[36902]: cluster 2022-04-22T20:33:01.679967+0000 mgr.y (mgr.34107) 6397 : cluster [DBG] pgmap v7550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:03 smithi121 bash[36902]: audit 2022-04-22T20:33:02.662918+0000 mon.a (mon.0) 9742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:03 smithi121 bash[36902]: audit 2022-04-22T20:33:02.734900+0000 mon.a (mon.0) 9743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[45495]: cluster 2022-04-22T20:33:01.679967+0000 mgr.y (mgr.34107) 6397 : cluster [DBG] pgmap v7550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[45495]: audit 2022-04-22T20:33:02.662918+0000 mon.a (mon.0) 9742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[45495]: audit 2022-04-22T20:33:02.734900+0000 mon.a (mon.0) 9743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[46934]: cluster 2022-04-22T20:33:01.679967+0000 mgr.y (mgr.34107) 6397 : cluster [DBG] pgmap v7550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[46934]: audit 2022-04-22T20:33:02.662918+0000 mon.a (mon.0) 9742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:03 smithi115 bash[46934]: audit 2022-04-22T20:33:02.734900+0000 mon.a (mon.0) 9743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:04.217Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:05 smithi121 bash[36902]: cluster 2022-04-22T20:33:03.681137+0000 mgr.y (mgr.34107) 6398 : cluster [DBG] pgmap v7551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:05 smithi121 bash[36902]: audit 2022-04-22T20:33:04.193851+0000 mon.a (mon.0) 9744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:05 smithi115 bash[45495]: cluster 2022-04-22T20:33:03.681137+0000 mgr.y (mgr.34107) 6398 : cluster [DBG] pgmap v7551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:05.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:05 smithi115 bash[45495]: audit 2022-04-22T20:33:04.193851+0000 mon.a (mon.0) 9744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:05.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:05 smithi115 bash[46934]: cluster 2022-04-22T20:33:03.681137+0000 mgr.y (mgr.34107) 6398 : cluster [DBG] pgmap v7551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:05.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:05 smithi115 bash[46934]: audit 2022-04-22T20:33:04.193851+0000 mon.a (mon.0) 9744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:07 smithi121 bash[36902]: cluster 2022-04-22T20:33:05.682182+0000 mgr.y (mgr.34107) 6399 : cluster [DBG] pgmap v7552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:07 smithi115 bash[45495]: cluster 2022-04-22T20:33:05.682182+0000 mgr.y (mgr.34107) 6399 : cluster [DBG] pgmap v7552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:07 smithi115 bash[46934]: cluster 2022-04-22T20:33:05.682182+0000 mgr.y (mgr.34107) 6399 : cluster [DBG] pgmap v7552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:08 smithi121 bash[36902]: audit 2022-04-22T20:33:07.666756+0000 mon.a (mon.0) 9745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:08 smithi121 bash[36902]: audit 2022-04-22T20:33:07.737688+0000 mon.a (mon.0) 9746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:08 smithi115 bash[45495]: audit 2022-04-22T20:33:07.666756+0000 mon.a (mon.0) 9745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:08 smithi115 bash[45495]: audit 2022-04-22T20:33:07.737688+0000 mon.a (mon.0) 9746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:08 smithi115 bash[46934]: audit 2022-04-22T20:33:07.666756+0000 mon.a (mon.0) 9745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:08 smithi115 bash[46934]: audit 2022-04-22T20:33:07.737688+0000 mon.a (mon.0) 9746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:09 smithi121 bash[36902]: cluster 2022-04-22T20:33:07.683282+0000 mgr.y (mgr.34107) 6400 : cluster [DBG] pgmap v7553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:09 smithi115 bash[45495]: cluster 2022-04-22T20:33:07.683282+0000 mgr.y (mgr.34107) 6400 : cluster [DBG] pgmap v7553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:09 smithi115 bash[46934]: cluster 2022-04-22T20:33:07.683282+0000 mgr.y (mgr.34107) 6400 : cluster [DBG] pgmap v7553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:10 smithi121 bash[36902]: audit 2022-04-22T20:33:09.196050+0000 mon.a (mon.0) 9747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:10 smithi115 bash[45495]: audit 2022-04-22T20:33:09.196050+0000 mon.a (mon.0) 9747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:10 smithi115 bash[46934]: audit 2022-04-22T20:33:09.196050+0000 mon.a (mon.0) 9747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:11 smithi121 bash[36902]: cluster 2022-04-22T20:33:09.684205+0000 mgr.y (mgr.34107) 6401 : cluster [DBG] pgmap v7554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:11 smithi115 bash[45495]: cluster 2022-04-22T20:33:09.684205+0000 mgr.y (mgr.34107) 6401 : cluster [DBG] pgmap v7554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:11 smithi115 bash[46934]: cluster 2022-04-22T20:33:09.684205+0000 mgr.y (mgr.34107) 6401 : cluster [DBG] pgmap v7554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:12.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:12.333Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:13 smithi121 bash[36902]: cluster 2022-04-22T20:33:11.685290+0000 mgr.y (mgr.34107) 6402 : cluster [DBG] pgmap v7555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:13 smithi121 bash[36902]: audit 2022-04-22T20:33:12.670467+0000 mon.a (mon.0) 9748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:13 smithi121 bash[36902]: audit 2022-04-22T20:33:12.740762+0000 mon.a (mon.0) 9749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[45495]: cluster 2022-04-22T20:33:11.685290+0000 mgr.y (mgr.34107) 6402 : cluster [DBG] pgmap v7555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[45495]: audit 2022-04-22T20:33:12.670467+0000 mon.a (mon.0) 9748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[45495]: audit 2022-04-22T20:33:12.740762+0000 mon.a (mon.0) 9749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[46934]: cluster 2022-04-22T20:33:11.685290+0000 mgr.y (mgr.34107) 6402 : cluster [DBG] pgmap v7555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[46934]: audit 2022-04-22T20:33:12.670467+0000 mon.a (mon.0) 9748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:13 smithi115 bash[46934]: audit 2022-04-22T20:33:12.740762+0000 mon.a (mon.0) 9749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:14.214Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:15 smithi121 bash[36902]: cluster 2022-04-22T20:33:13.686477+0000 mgr.y (mgr.34107) 6403 : cluster [DBG] pgmap v7556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:15.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:15 smithi121 bash[36902]: audit 2022-04-22T20:33:14.198753+0000 mon.a (mon.0) 9750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:15 smithi115 bash[45495]: cluster 2022-04-22T20:33:13.686477+0000 mgr.y (mgr.34107) 6403 : cluster [DBG] pgmap v7556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:15 smithi115 bash[45495]: audit 2022-04-22T20:33:14.198753+0000 mon.a (mon.0) 9750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:15 smithi115 bash[46934]: cluster 2022-04-22T20:33:13.686477+0000 mgr.y (mgr.34107) 6403 : cluster [DBG] pgmap v7556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:15 smithi115 bash[46934]: audit 2022-04-22T20:33:14.198753+0000 mon.a (mon.0) 9750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:16 smithi121 bash[36902]: audit 2022-04-22T20:33:15.426485+0000 mon.a (mon.0) 9751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:33:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:16 smithi121 bash[36902]: audit 2022-04-22T20:33:15.426943+0000 mon.a (mon.0) 9752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:33:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:16 smithi115 bash[45495]: audit 2022-04-22T20:33:15.426485+0000 mon.a (mon.0) 9751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:33:16.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:16 smithi115 bash[45495]: audit 2022-04-22T20:33:15.426943+0000 mon.a (mon.0) 9752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:33:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:16 smithi115 bash[46934]: audit 2022-04-22T20:33:15.426485+0000 mon.a (mon.0) 9751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:33:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:16 smithi115 bash[46934]: audit 2022-04-22T20:33:15.426943+0000 mon.a (mon.0) 9752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:33:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:17 smithi121 bash[36902]: cluster 2022-04-22T20:33:15.687606+0000 mgr.y (mgr.34107) 6404 : cluster [DBG] pgmap v7557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:17 smithi115 bash[46934]: cluster 2022-04-22T20:33:15.687606+0000 mgr.y (mgr.34107) 6404 : cluster [DBG] pgmap v7557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:17 smithi115 bash[45495]: cluster 2022-04-22T20:33:15.687606+0000 mgr.y (mgr.34107) 6404 : cluster [DBG] pgmap v7557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:18 smithi121 bash[36902]: audit 2022-04-22T20:33:17.674266+0000 mon.a (mon.0) 9753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:18 smithi121 bash[36902]: audit 2022-04-22T20:33:17.743772+0000 mon.a (mon.0) 9754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:18 smithi115 bash[45495]: audit 2022-04-22T20:33:17.674266+0000 mon.a (mon.0) 9753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:18 smithi115 bash[45495]: audit 2022-04-22T20:33:17.743772+0000 mon.a (mon.0) 9754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:18 smithi115 bash[46934]: audit 2022-04-22T20:33:17.674266+0000 mon.a (mon.0) 9753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:18 smithi115 bash[46934]: audit 2022-04-22T20:33:17.743772+0000 mon.a (mon.0) 9754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:19 smithi121 bash[36902]: cluster 2022-04-22T20:33:17.688645+0000 mgr.y (mgr.34107) 6405 : cluster [DBG] pgmap v7558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:19 smithi115 bash[45495]: cluster 2022-04-22T20:33:17.688645+0000 mgr.y (mgr.34107) 6405 : cluster [DBG] pgmap v7558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:19 smithi115 bash[46934]: cluster 2022-04-22T20:33:17.688645+0000 mgr.y (mgr.34107) 6405 : cluster [DBG] pgmap v7558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:20 smithi121 bash[36902]: audit 2022-04-22T20:33:19.201563+0000 mon.a (mon.0) 9755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:20 smithi115 bash[45495]: audit 2022-04-22T20:33:19.201563+0000 mon.a (mon.0) 9755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:20 smithi115 bash[46934]: audit 2022-04-22T20:33:19.201563+0000 mon.a (mon.0) 9755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:21 smithi121 bash[36902]: cluster 2022-04-22T20:33:19.689762+0000 mgr.y (mgr.34107) 6406 : cluster [DBG] pgmap v7559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:21 smithi115 bash[45495]: cluster 2022-04-22T20:33:19.689762+0000 mgr.y (mgr.34107) 6406 : cluster [DBG] pgmap v7559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:21 smithi115 bash[46934]: cluster 2022-04-22T20:33:19.689762+0000 mgr.y (mgr.34107) 6406 : cluster [DBG] pgmap v7559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:22.333Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:23 smithi121 bash[36902]: cluster 2022-04-22T20:33:21.690882+0000 mgr.y (mgr.34107) 6407 : cluster [DBG] pgmap v7560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:23 smithi121 bash[36902]: audit 2022-04-22T20:33:22.678220+0000 mon.a (mon.0) 9756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:23 smithi121 bash[36902]: audit 2022-04-22T20:33:22.746762+0000 mon.a (mon.0) 9757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[45495]: cluster 2022-04-22T20:33:21.690882+0000 mgr.y (mgr.34107) 6407 : cluster [DBG] pgmap v7560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[45495]: audit 2022-04-22T20:33:22.678220+0000 mon.a (mon.0) 9756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[45495]: audit 2022-04-22T20:33:22.746762+0000 mon.a (mon.0) 9757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[46934]: cluster 2022-04-22T20:33:21.690882+0000 mgr.y (mgr.34107) 6407 : cluster [DBG] pgmap v7560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[46934]: audit 2022-04-22T20:33:22.678220+0000 mon.a (mon.0) 9756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:23.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:23 smithi115 bash[46934]: audit 2022-04-22T20:33:22.746762+0000 mon.a (mon.0) 9757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:24.290 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:33:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:24.214Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:33:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 38s ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:33:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 103s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:33:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 38s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:33:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 103s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 103s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 38s ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 38s ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 103s ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 38s ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 38s ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 103s ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 38s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 38s ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:33:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 38s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:33:24.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 38s ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:33:24.637 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 103s ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:33:24.637 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 103s ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:33:24.637 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (2h) 103s ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:33:24.637 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 103s ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:33:24.637 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 103s ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:33:25.036 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:33:25.036 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:33:25.036 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:33:25.036 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:25.037 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:33:25.037 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:33:25.037 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:33:25.038 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:33:25.039 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:33:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:25 smithi121 bash[36902]: cluster 2022-04-22T20:33:23.692119+0000 mgr.y (mgr.34107) 6408 : cluster [DBG] pgmap v7561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:25 smithi121 bash[36902]: audit 2022-04-22T20:33:24.204219+0000 mon.a (mon.0) 9758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:25 smithi121 bash[36902]: audit 2022-04-22T20:33:25.028455+0000 mon.a (mon.0) 9759 : audit [DBG] from='client.? 172.21.15.115:0/46884844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[45495]: cluster 2022-04-22T20:33:23.692119+0000 mgr.y (mgr.34107) 6408 : cluster [DBG] pgmap v7561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[45495]: audit 2022-04-22T20:33:24.204219+0000 mon.a (mon.0) 9758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:25.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[45495]: audit 2022-04-22T20:33:25.028455+0000 mon.a (mon.0) 9759 : audit [DBG] from='client.? 172.21.15.115:0/46884844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[46934]: cluster 2022-04-22T20:33:23.692119+0000 mgr.y (mgr.34107) 6408 : cluster [DBG] pgmap v7561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[46934]: audit 2022-04-22T20:33:24.204219+0000 mon.a (mon.0) 9758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:25.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:25 smithi115 bash[46934]: audit 2022-04-22T20:33:25.028455+0000 mon.a (mon.0) 9759 : audit [DBG] from='client.? 172.21.15.115:0/46884844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:26 smithi121 bash[36902]: audit 2022-04-22T20:33:24.265977+0000 mgr.y (mgr.34107) 6409 : audit [DBG] from='client.42048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:26.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:26 smithi121 bash[36902]: audit 2022-04-22T20:33:24.609588+0000 mgr.y (mgr.34107) 6410 : audit [DBG] from='client.51985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:26 smithi115 bash[45495]: audit 2022-04-22T20:33:24.265977+0000 mgr.y (mgr.34107) 6409 : audit [DBG] from='client.42048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:26.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:26 smithi115 bash[45495]: audit 2022-04-22T20:33:24.609588+0000 mgr.y (mgr.34107) 6410 : audit [DBG] from='client.51985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:26 smithi115 bash[46934]: audit 2022-04-22T20:33:24.265977+0000 mgr.y (mgr.34107) 6409 : audit [DBG] from='client.42048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:26 smithi115 bash[46934]: audit 2022-04-22T20:33:24.609588+0000 mgr.y (mgr.34107) 6410 : audit [DBG] from='client.51985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:27 smithi121 bash[36902]: cluster 2022-04-22T20:33:25.693250+0000 mgr.y (mgr.34107) 6411 : cluster [DBG] pgmap v7562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:27 smithi115 bash[45495]: cluster 2022-04-22T20:33:25.693250+0000 mgr.y (mgr.34107) 6411 : cluster [DBG] pgmap v7562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:27 smithi115 bash[46934]: cluster 2022-04-22T20:33:25.693250+0000 mgr.y (mgr.34107) 6411 : cluster [DBG] pgmap v7562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:28 smithi121 bash[36902]: audit 2022-04-22T20:33:27.681851+0000 mon.a (mon.0) 9760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:28 smithi121 bash[36902]: audit 2022-04-22T20:33:27.749768+0000 mon.a (mon.0) 9761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:28 smithi115 bash[46934]: audit 2022-04-22T20:33:27.681851+0000 mon.a (mon.0) 9760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:28 smithi115 bash[46934]: audit 2022-04-22T20:33:27.749768+0000 mon.a (mon.0) 9761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:28 smithi115 bash[45495]: audit 2022-04-22T20:33:27.681851+0000 mon.a (mon.0) 9760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:28.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:28 smithi115 bash[45495]: audit 2022-04-22T20:33:27.749768+0000 mon.a (mon.0) 9761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:29 smithi121 bash[36902]: cluster 2022-04-22T20:33:27.694283+0000 mgr.y (mgr.34107) 6412 : cluster [DBG] pgmap v7563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:29 smithi115 bash[46934]: cluster 2022-04-22T20:33:27.694283+0000 mgr.y (mgr.34107) 6412 : cluster [DBG] pgmap v7563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:29 smithi115 bash[45495]: cluster 2022-04-22T20:33:27.694283+0000 mgr.y (mgr.34107) 6412 : cluster [DBG] pgmap v7563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:30 smithi121 bash[36902]: audit 2022-04-22T20:33:29.206780+0000 mon.a (mon.0) 9762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:30 smithi115 bash[45495]: audit 2022-04-22T20:33:29.206780+0000 mon.a (mon.0) 9762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:30 smithi115 bash[46934]: audit 2022-04-22T20:33:29.206780+0000 mon.a (mon.0) 9762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:31 smithi121 bash[36902]: cluster 2022-04-22T20:33:29.695351+0000 mgr.y (mgr.34107) 6413 : cluster [DBG] pgmap v7564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:31.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:31 smithi115 bash[45495]: cluster 2022-04-22T20:33:29.695351+0000 mgr.y (mgr.34107) 6413 : cluster [DBG] pgmap v7564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:31 smithi115 bash[46934]: cluster 2022-04-22T20:33:29.695351+0000 mgr.y (mgr.34107) 6413 : cluster [DBG] pgmap v7564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:32.334Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:33 smithi121 bash[36902]: cluster 2022-04-22T20:33:31.696394+0000 mgr.y (mgr.34107) 6414 : cluster [DBG] pgmap v7565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:33 smithi121 bash[36902]: audit 2022-04-22T20:33:32.685622+0000 mon.a (mon.0) 9763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:33 smithi121 bash[36902]: audit 2022-04-22T20:33:32.752610+0000 mon.a (mon.0) 9764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:33.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[45495]: cluster 2022-04-22T20:33:31.696394+0000 mgr.y (mgr.34107) 6414 : cluster [DBG] pgmap v7565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[45495]: audit 2022-04-22T20:33:32.685622+0000 mon.a (mon.0) 9763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[45495]: audit 2022-04-22T20:33:32.752610+0000 mon.a (mon.0) 9764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[46934]: cluster 2022-04-22T20:33:31.696394+0000 mgr.y (mgr.34107) 6414 : cluster [DBG] pgmap v7565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[46934]: audit 2022-04-22T20:33:32.685622+0000 mon.a (mon.0) 9763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:33 smithi115 bash[46934]: audit 2022-04-22T20:33:32.752610+0000 mon.a (mon.0) 9764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:34.218Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:35 smithi121 bash[36902]: cluster 2022-04-22T20:33:33.697531+0000 mgr.y (mgr.34107) 6415 : cluster [DBG] pgmap v7566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:35 smithi121 bash[36902]: audit 2022-04-22T20:33:34.209002+0000 mon.a (mon.0) 9765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:35 smithi115 bash[45495]: cluster 2022-04-22T20:33:33.697531+0000 mgr.y (mgr.34107) 6415 : cluster [DBG] pgmap v7566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:35.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:35 smithi115 bash[45495]: audit 2022-04-22T20:33:34.209002+0000 mon.a (mon.0) 9765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:35.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:35 smithi115 bash[46934]: cluster 2022-04-22T20:33:33.697531+0000 mgr.y (mgr.34107) 6415 : cluster [DBG] pgmap v7566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:35.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:35 smithi115 bash[46934]: audit 2022-04-22T20:33:34.209002+0000 mon.a (mon.0) 9765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:37 smithi121 bash[36902]: cluster 2022-04-22T20:33:35.698633+0000 mgr.y (mgr.34107) 6416 : cluster [DBG] pgmap v7567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:37.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:37 smithi115 bash[45495]: cluster 2022-04-22T20:33:35.698633+0000 mgr.y (mgr.34107) 6416 : cluster [DBG] pgmap v7567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:37 smithi115 bash[46934]: cluster 2022-04-22T20:33:35.698633+0000 mgr.y (mgr.34107) 6416 : cluster [DBG] pgmap v7567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:38 smithi121 bash[36902]: audit 2022-04-22T20:33:37.689386+0000 mon.a (mon.0) 9766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:38 smithi121 bash[36902]: audit 2022-04-22T20:33:37.755567+0000 mon.a (mon.0) 9767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:38.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:38 smithi115 bash[45495]: audit 2022-04-22T20:33:37.689386+0000 mon.a (mon.0) 9766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:38 smithi115 bash[45495]: audit 2022-04-22T20:33:37.755567+0000 mon.a (mon.0) 9767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:38 smithi115 bash[46934]: audit 2022-04-22T20:33:37.689386+0000 mon.a (mon.0) 9766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:38 smithi115 bash[46934]: audit 2022-04-22T20:33:37.755567+0000 mon.a (mon.0) 9767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:39 smithi121 bash[36902]: cluster 2022-04-22T20:33:37.699719+0000 mgr.y (mgr.34107) 6417 : cluster [DBG] pgmap v7568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:39 smithi115 bash[45495]: cluster 2022-04-22T20:33:37.699719+0000 mgr.y (mgr.34107) 6417 : cluster [DBG] pgmap v7568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:39.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:39 smithi115 bash[46934]: cluster 2022-04-22T20:33:37.699719+0000 mgr.y (mgr.34107) 6417 : cluster [DBG] pgmap v7568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:40 smithi121 bash[36902]: audit 2022-04-22T20:33:39.212130+0000 mon.a (mon.0) 9768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:40 smithi115 bash[46934]: audit 2022-04-22T20:33:39.212130+0000 mon.a (mon.0) 9768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:40.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:40 smithi115 bash[45495]: audit 2022-04-22T20:33:39.212130+0000 mon.a (mon.0) 9768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:41 smithi121 bash[36902]: cluster 2022-04-22T20:33:39.700736+0000 mgr.y (mgr.34107) 6418 : cluster [DBG] pgmap v7569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:41.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:41 smithi115 bash[45495]: cluster 2022-04-22T20:33:39.700736+0000 mgr.y (mgr.34107) 6418 : cluster [DBG] pgmap v7569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:41 smithi115 bash[46934]: cluster 2022-04-22T20:33:39.700736+0000 mgr.y (mgr.34107) 6418 : cluster [DBG] pgmap v7569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:42.334Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:43 smithi121 bash[36902]: cluster 2022-04-22T20:33:41.701852+0000 mgr.y (mgr.34107) 6419 : cluster [DBG] pgmap v7570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:43 smithi121 bash[36902]: audit 2022-04-22T20:33:42.693442+0000 mon.a (mon.0) 9769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:43 smithi121 bash[36902]: audit 2022-04-22T20:33:42.758604+0000 mon.a (mon.0) 9770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[45495]: cluster 2022-04-22T20:33:41.701852+0000 mgr.y (mgr.34107) 6419 : cluster [DBG] pgmap v7570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[45495]: audit 2022-04-22T20:33:42.693442+0000 mon.a (mon.0) 9769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[45495]: audit 2022-04-22T20:33:42.758604+0000 mon.a (mon.0) 9770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[46934]: cluster 2022-04-22T20:33:41.701852+0000 mgr.y (mgr.34107) 6419 : cluster [DBG] pgmap v7570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[46934]: audit 2022-04-22T20:33:42.693442+0000 mon.a (mon.0) 9769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:43 smithi115 bash[46934]: audit 2022-04-22T20:33:42.758604+0000 mon.a (mon.0) 9770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:44.219Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:45 smithi121 bash[36902]: cluster 2022-04-22T20:33:43.703015+0000 mgr.y (mgr.34107) 6420 : cluster [DBG] pgmap v7571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:45 smithi121 bash[36902]: audit 2022-04-22T20:33:44.215430+0000 mon.a (mon.0) 9771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:45 smithi115 bash[45495]: cluster 2022-04-22T20:33:43.703015+0000 mgr.y (mgr.34107) 6420 : cluster [DBG] pgmap v7571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:45 smithi115 bash[45495]: audit 2022-04-22T20:33:44.215430+0000 mon.a (mon.0) 9771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:45 smithi115 bash[46934]: cluster 2022-04-22T20:33:43.703015+0000 mgr.y (mgr.34107) 6420 : cluster [DBG] pgmap v7571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:45 smithi115 bash[46934]: audit 2022-04-22T20:33:44.215430+0000 mon.a (mon.0) 9771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:47 smithi121 bash[36902]: cluster 2022-04-22T20:33:45.704125+0000 mgr.y (mgr.34107) 6421 : cluster [DBG] pgmap v7572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:47.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:47 smithi121 bash[36902]: audit 2022-04-22T20:33:46.334752+0000 mon.a (mon.0) 9772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:33:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:47 smithi121 bash[36902]: audit 2022-04-22T20:33:46.337697+0000 mon.a (mon.0) 9773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:33:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:47 smithi121 bash[36902]: audit 2022-04-22T20:33:46.339509+0000 mon.a (mon.0) 9774 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:33:47.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:47 smithi121 bash[36902]: audit 2022-04-22T20:33:46.699389+0000 mon.a (mon.0) 9775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:33:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[45495]: cluster 2022-04-22T20:33:45.704125+0000 mgr.y (mgr.34107) 6421 : cluster [DBG] pgmap v7572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[45495]: audit 2022-04-22T20:33:46.334752+0000 mon.a (mon.0) 9772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:33:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[45495]: audit 2022-04-22T20:33:46.337697+0000 mon.a (mon.0) 9773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:33:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[45495]: audit 2022-04-22T20:33:46.339509+0000 mon.a (mon.0) 9774 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[45495]: audit 2022-04-22T20:33:46.699389+0000 mon.a (mon.0) 9775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[46934]: cluster 2022-04-22T20:33:45.704125+0000 mgr.y (mgr.34107) 6421 : cluster [DBG] pgmap v7572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[46934]: audit 2022-04-22T20:33:46.334752+0000 mon.a (mon.0) 9772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[46934]: audit 2022-04-22T20:33:46.337697+0000 mon.a (mon.0) 9773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[46934]: audit 2022-04-22T20:33:46.339509+0000 mon.a (mon.0) 9774 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:33:47.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:47 smithi115 bash[46934]: audit 2022-04-22T20:33:46.699389+0000 mon.a (mon.0) 9775 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:33:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:48 smithi121 bash[36902]: audit 2022-04-22T20:33:47.697214+0000 mon.a (mon.0) 9776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:48 smithi121 bash[36902]: audit 2022-04-22T20:33:47.761653+0000 mon.a (mon.0) 9777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:48 smithi115 bash[45495]: audit 2022-04-22T20:33:47.697214+0000 mon.a (mon.0) 9776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:48 smithi115 bash[45495]: audit 2022-04-22T20:33:47.761653+0000 mon.a (mon.0) 9777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:48 smithi115 bash[46934]: audit 2022-04-22T20:33:47.697214+0000 mon.a (mon.0) 9776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:48 smithi115 bash[46934]: audit 2022-04-22T20:33:47.761653+0000 mon.a (mon.0) 9777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:49 smithi121 bash[36902]: cluster 2022-04-22T20:33:47.705172+0000 mgr.y (mgr.34107) 6422 : cluster [DBG] pgmap v7573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:49 smithi121 bash[36902]: audit 2022-04-22T20:33:49.218395+0000 mon.a (mon.0) 9778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:49 smithi115 bash[46934]: cluster 2022-04-22T20:33:47.705172+0000 mgr.y (mgr.34107) 6422 : cluster [DBG] pgmap v7573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:49 smithi115 bash[46934]: audit 2022-04-22T20:33:49.218395+0000 mon.a (mon.0) 9778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:49 smithi115 bash[45495]: cluster 2022-04-22T20:33:47.705172+0000 mgr.y (mgr.34107) 6422 : cluster [DBG] pgmap v7573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:49 smithi115 bash[45495]: audit 2022-04-22T20:33:49.218395+0000 mon.a (mon.0) 9778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:51 smithi121 bash[36902]: cluster 2022-04-22T20:33:49.706292+0000 mgr.y (mgr.34107) 6423 : cluster [DBG] pgmap v7574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:51 smithi115 bash[46934]: cluster 2022-04-22T20:33:49.706292+0000 mgr.y (mgr.34107) 6423 : cluster [DBG] pgmap v7574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:51.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:51 smithi115 bash[45495]: cluster 2022-04-22T20:33:49.706292+0000 mgr.y (mgr.34107) 6423 : cluster [DBG] pgmap v7574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:52.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:52.336Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:53 smithi121 bash[36902]: cluster 2022-04-22T20:33:51.707375+0000 mgr.y (mgr.34107) 6424 : cluster [DBG] pgmap v7575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:53 smithi121 bash[36902]: audit 2022-04-22T20:33:52.700710+0000 mon.a (mon.0) 9779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:53 smithi121 bash[36902]: audit 2022-04-22T20:33:52.764333+0000 mon.a (mon.0) 9780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[45495]: cluster 2022-04-22T20:33:51.707375+0000 mgr.y (mgr.34107) 6424 : cluster [DBG] pgmap v7575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[45495]: audit 2022-04-22T20:33:52.700710+0000 mon.a (mon.0) 9779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[45495]: audit 2022-04-22T20:33:52.764333+0000 mon.a (mon.0) 9780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[46934]: cluster 2022-04-22T20:33:51.707375+0000 mgr.y (mgr.34107) 6424 : cluster [DBG] pgmap v7575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[46934]: audit 2022-04-22T20:33:52.700710+0000 mon.a (mon.0) 9779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:53 smithi115 bash[46934]: audit 2022-04-22T20:33:52.764333+0000 mon.a (mon.0) 9780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:33:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:33:54.217Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:33:55.406 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:33:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:55 smithi121 bash[36902]: cluster 2022-04-22T20:33:53.708436+0000 mgr.y (mgr.34107) 6425 : cluster [DBG] pgmap v7576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:55 smithi121 bash[36902]: audit 2022-04-22T20:33:54.220884+0000 mon.a (mon.0) 9781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:55 smithi115 bash[46934]: cluster 2022-04-22T20:33:53.708436+0000 mgr.y (mgr.34107) 6425 : cluster [DBG] pgmap v7576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:55 smithi115 bash[46934]: audit 2022-04-22T20:33:54.220884+0000 mon.a (mon.0) 9781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:55 smithi115 bash[45495]: cluster 2022-04-22T20:33:53.708436+0000 mgr.y (mgr.34107) 6425 : cluster [DBG] pgmap v7576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:55 smithi115 bash[45495]: audit 2022-04-22T20:33:54.220884+0000 mon.a (mon.0) 9781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:33:55.751 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:33:55.751 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 69s ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:33:55.751 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 69s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 69s ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 69s ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 69s ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 69s ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:33:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 69s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 69s ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 69s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 69s ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:33:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:33:56.166 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:33:56.167 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:33:56.168 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:33:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:56 smithi121 bash[36902]: audit 2022-04-22T20:33:56.158893+0000 mon.a (mon.0) 9782 : audit [DBG] from='client.? 172.21.15.115:0/568976136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:56 smithi115 bash[45495]: audit 2022-04-22T20:33:56.158893+0000 mon.a (mon.0) 9782 : audit [DBG] from='client.? 172.21.15.115:0/568976136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:56.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:56 smithi115 bash[46934]: audit 2022-04-22T20:33:56.158893+0000 mon.a (mon.0) 9782 : audit [DBG] from='client.? 172.21.15.115:0/568976136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:33:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:57 smithi121 bash[36902]: audit 2022-04-22T20:33:55.383707+0000 mgr.y (mgr.34107) 6426 : audit [DBG] from='client.51997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:57 smithi121 bash[36902]: cluster 2022-04-22T20:33:55.709428+0000 mgr.y (mgr.34107) 6427 : cluster [DBG] pgmap v7577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:57 smithi121 bash[36902]: audit 2022-04-22T20:33:55.726001+0000 mgr.y (mgr.34107) 6428 : audit [DBG] from='client.42072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[45495]: audit 2022-04-22T20:33:55.383707+0000 mgr.y (mgr.34107) 6426 : audit [DBG] from='client.51997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[45495]: cluster 2022-04-22T20:33:55.709428+0000 mgr.y (mgr.34107) 6427 : cluster [DBG] pgmap v7577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[45495]: audit 2022-04-22T20:33:55.726001+0000 mgr.y (mgr.34107) 6428 : audit [DBG] from='client.42072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[46934]: audit 2022-04-22T20:33:55.383707+0000 mgr.y (mgr.34107) 6426 : audit [DBG] from='client.51997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[46934]: cluster 2022-04-22T20:33:55.709428+0000 mgr.y (mgr.34107) 6427 : cluster [DBG] pgmap v7577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:57 smithi115 bash[46934]: audit 2022-04-22T20:33:55.726001+0000 mgr.y (mgr.34107) 6428 : audit [DBG] from='client.42072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:33:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:58 smithi121 bash[36902]: audit 2022-04-22T20:33:57.704574+0000 mon.a (mon.0) 9783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:58 smithi121 bash[36902]: audit 2022-04-22T20:33:57.767466+0000 mon.a (mon.0) 9784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:58 smithi115 bash[45495]: audit 2022-04-22T20:33:57.704574+0000 mon.a (mon.0) 9783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:58 smithi115 bash[45495]: audit 2022-04-22T20:33:57.767466+0000 mon.a (mon.0) 9784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:58 smithi115 bash[46934]: audit 2022-04-22T20:33:57.704574+0000 mon.a (mon.0) 9783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:33:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:58 smithi115 bash[46934]: audit 2022-04-22T20:33:57.767466+0000 mon.a (mon.0) 9784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:33:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:33:59 smithi121 bash[36902]: cluster 2022-04-22T20:33:57.710448+0000 mgr.y (mgr.34107) 6429 : cluster [DBG] pgmap v7578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:33:59 smithi115 bash[45495]: cluster 2022-04-22T20:33:57.710448+0000 mgr.y (mgr.34107) 6429 : cluster [DBG] pgmap v7578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:33:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:33:59 smithi115 bash[46934]: cluster 2022-04-22T20:33:57.710448+0000 mgr.y (mgr.34107) 6429 : cluster [DBG] pgmap v7578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:00 smithi121 bash[36902]: audit 2022-04-22T20:33:59.223701+0000 mon.a (mon.0) 9785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:00 smithi115 bash[45495]: audit 2022-04-22T20:33:59.223701+0000 mon.a (mon.0) 9785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:00 smithi115 bash[46934]: audit 2022-04-22T20:33:59.223701+0000 mon.a (mon.0) 9785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:01 smithi121 bash[36902]: cluster 2022-04-22T20:33:59.711510+0000 mgr.y (mgr.34107) 6430 : cluster [DBG] pgmap v7579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:01.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:01 smithi115 bash[45495]: cluster 2022-04-22T20:33:59.711510+0000 mgr.y (mgr.34107) 6430 : cluster [DBG] pgmap v7579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:01.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:01 smithi115 bash[46934]: cluster 2022-04-22T20:33:59.711510+0000 mgr.y (mgr.34107) 6430 : cluster [DBG] pgmap v7579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:02.336Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:03 smithi121 bash[36902]: cluster 2022-04-22T20:34:01.712485+0000 mgr.y (mgr.34107) 6431 : cluster [DBG] pgmap v7580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:03 smithi121 bash[36902]: audit 2022-04-22T20:34:02.708486+0000 mon.a (mon.0) 9786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:03 smithi121 bash[36902]: audit 2022-04-22T20:34:02.770575+0000 mon.a (mon.0) 9787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[45495]: cluster 2022-04-22T20:34:01.712485+0000 mgr.y (mgr.34107) 6431 : cluster [DBG] pgmap v7580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[45495]: audit 2022-04-22T20:34:02.708486+0000 mon.a (mon.0) 9786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[45495]: audit 2022-04-22T20:34:02.770575+0000 mon.a (mon.0) 9787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[46934]: cluster 2022-04-22T20:34:01.712485+0000 mgr.y (mgr.34107) 6431 : cluster [DBG] pgmap v7580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[46934]: audit 2022-04-22T20:34:02.708486+0000 mon.a (mon.0) 9786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:03 smithi115 bash[46934]: audit 2022-04-22T20:34:02.770575+0000 mon.a (mon.0) 9787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:04.217Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:05 smithi121 bash[36902]: cluster 2022-04-22T20:34:03.713571+0000 mgr.y (mgr.34107) 6432 : cluster [DBG] pgmap v7581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:05 smithi121 bash[36902]: audit 2022-04-22T20:34:04.226405+0000 mon.a (mon.0) 9788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:05 smithi115 bash[45495]: cluster 2022-04-22T20:34:03.713571+0000 mgr.y (mgr.34107) 6432 : cluster [DBG] pgmap v7581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:05 smithi115 bash[45495]: audit 2022-04-22T20:34:04.226405+0000 mon.a (mon.0) 9788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:05 smithi115 bash[46934]: cluster 2022-04-22T20:34:03.713571+0000 mgr.y (mgr.34107) 6432 : cluster [DBG] pgmap v7581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:05 smithi115 bash[46934]: audit 2022-04-22T20:34:04.226405+0000 mon.a (mon.0) 9788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:07 smithi121 bash[36902]: cluster 2022-04-22T20:34:05.714656+0000 mgr.y (mgr.34107) 6433 : cluster [DBG] pgmap v7582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:07 smithi115 bash[45495]: cluster 2022-04-22T20:34:05.714656+0000 mgr.y (mgr.34107) 6433 : cluster [DBG] pgmap v7582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:07 smithi115 bash[46934]: cluster 2022-04-22T20:34:05.714656+0000 mgr.y (mgr.34107) 6433 : cluster [DBG] pgmap v7582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:08 smithi121 bash[36902]: audit 2022-04-22T20:34:07.712042+0000 mon.a (mon.0) 9789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:08 smithi121 bash[36902]: audit 2022-04-22T20:34:07.773656+0000 mon.a (mon.0) 9790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:08 smithi115 bash[45495]: audit 2022-04-22T20:34:07.712042+0000 mon.a (mon.0) 9789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:08 smithi115 bash[45495]: audit 2022-04-22T20:34:07.773656+0000 mon.a (mon.0) 9790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:08 smithi115 bash[46934]: audit 2022-04-22T20:34:07.712042+0000 mon.a (mon.0) 9789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:08 smithi115 bash[46934]: audit 2022-04-22T20:34:07.773656+0000 mon.a (mon.0) 9790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:09 smithi121 bash[36902]: cluster 2022-04-22T20:34:07.715697+0000 mgr.y (mgr.34107) 6434 : cluster [DBG] pgmap v7583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:09 smithi121 bash[36902]: audit 2022-04-22T20:34:09.229324+0000 mon.a (mon.0) 9791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:09 smithi115 bash[45495]: cluster 2022-04-22T20:34:07.715697+0000 mgr.y (mgr.34107) 6434 : cluster [DBG] pgmap v7583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:09 smithi115 bash[45495]: audit 2022-04-22T20:34:09.229324+0000 mon.a (mon.0) 9791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:09 smithi115 bash[46934]: cluster 2022-04-22T20:34:07.715697+0000 mgr.y (mgr.34107) 6434 : cluster [DBG] pgmap v7583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:09 smithi115 bash[46934]: audit 2022-04-22T20:34:09.229324+0000 mon.a (mon.0) 9791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:11 smithi121 bash[36902]: cluster 2022-04-22T20:34:09.716695+0000 mgr.y (mgr.34107) 6435 : cluster [DBG] pgmap v7584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:11 smithi115 bash[45495]: cluster 2022-04-22T20:34:09.716695+0000 mgr.y (mgr.34107) 6435 : cluster [DBG] pgmap v7584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:11 smithi115 bash[46934]: cluster 2022-04-22T20:34:09.716695+0000 mgr.y (mgr.34107) 6435 : cluster [DBG] pgmap v7584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:12.337Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:13 smithi121 bash[36902]: cluster 2022-04-22T20:34:11.717743+0000 mgr.y (mgr.34107) 6436 : cluster [DBG] pgmap v7585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:13 smithi121 bash[36902]: audit 2022-04-22T20:34:12.715696+0000 mon.a (mon.0) 9792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:13 smithi121 bash[36902]: audit 2022-04-22T20:34:12.776762+0000 mon.a (mon.0) 9793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:13.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[46934]: cluster 2022-04-22T20:34:11.717743+0000 mgr.y (mgr.34107) 6436 : cluster [DBG] pgmap v7585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[46934]: audit 2022-04-22T20:34:12.715696+0000 mon.a (mon.0) 9792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[46934]: audit 2022-04-22T20:34:12.776762+0000 mon.a (mon.0) 9793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[45495]: cluster 2022-04-22T20:34:11.717743+0000 mgr.y (mgr.34107) 6436 : cluster [DBG] pgmap v7585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[45495]: audit 2022-04-22T20:34:12.715696+0000 mon.a (mon.0) 9792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:13 smithi115 bash[45495]: audit 2022-04-22T20:34:12.776762+0000 mon.a (mon.0) 9793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:14.218Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:15 smithi121 bash[36902]: cluster 2022-04-22T20:34:13.718883+0000 mgr.y (mgr.34107) 6437 : cluster [DBG] pgmap v7586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:15 smithi121 bash[36902]: audit 2022-04-22T20:34:14.232226+0000 mon.a (mon.0) 9794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:15 smithi115 bash[45495]: cluster 2022-04-22T20:34:13.718883+0000 mgr.y (mgr.34107) 6437 : cluster [DBG] pgmap v7586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:15 smithi115 bash[45495]: audit 2022-04-22T20:34:14.232226+0000 mon.a (mon.0) 9794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:15 smithi115 bash[46934]: cluster 2022-04-22T20:34:13.718883+0000 mgr.y (mgr.34107) 6437 : cluster [DBG] pgmap v7586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:15 smithi115 bash[46934]: audit 2022-04-22T20:34:14.232226+0000 mon.a (mon.0) 9794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:16 smithi121 bash[36902]: audit 2022-04-22T20:34:15.436442+0000 mon.a (mon.0) 9795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:34:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:16 smithi121 bash[36902]: audit 2022-04-22T20:34:15.436911+0000 mon.a (mon.0) 9796 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:34:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:16 smithi115 bash[45495]: audit 2022-04-22T20:34:15.436442+0000 mon.a (mon.0) 9795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:34:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:16 smithi115 bash[45495]: audit 2022-04-22T20:34:15.436911+0000 mon.a (mon.0) 9796 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:34:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:16 smithi115 bash[46934]: audit 2022-04-22T20:34:15.436442+0000 mon.a (mon.0) 9795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:34:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:16 smithi115 bash[46934]: audit 2022-04-22T20:34:15.436911+0000 mon.a (mon.0) 9796 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:34:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:17 smithi121 bash[36902]: cluster 2022-04-22T20:34:15.720034+0000 mgr.y (mgr.34107) 6438 : cluster [DBG] pgmap v7587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:17 smithi115 bash[45495]: cluster 2022-04-22T20:34:15.720034+0000 mgr.y (mgr.34107) 6438 : cluster [DBG] pgmap v7587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:17 smithi115 bash[46934]: cluster 2022-04-22T20:34:15.720034+0000 mgr.y (mgr.34107) 6438 : cluster [DBG] pgmap v7587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:18 smithi121 bash[36902]: audit 2022-04-22T20:34:17.722723+0000 mon.a (mon.0) 9797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:18 smithi121 bash[36902]: audit 2022-04-22T20:34:17.779807+0000 mon.a (mon.0) 9798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:18 smithi115 bash[45495]: audit 2022-04-22T20:34:17.722723+0000 mon.a (mon.0) 9797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:18 smithi115 bash[45495]: audit 2022-04-22T20:34:17.779807+0000 mon.a (mon.0) 9798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:18 smithi115 bash[46934]: audit 2022-04-22T20:34:17.722723+0000 mon.a (mon.0) 9797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:18 smithi115 bash[46934]: audit 2022-04-22T20:34:17.779807+0000 mon.a (mon.0) 9798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:19 smithi121 bash[36902]: cluster 2022-04-22T20:34:17.721133+0000 mgr.y (mgr.34107) 6439 : cluster [DBG] pgmap v7588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:19 smithi115 bash[45495]: cluster 2022-04-22T20:34:17.721133+0000 mgr.y (mgr.34107) 6439 : cluster [DBG] pgmap v7588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:19 smithi115 bash[46934]: cluster 2022-04-22T20:34:17.721133+0000 mgr.y (mgr.34107) 6439 : cluster [DBG] pgmap v7588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:20 smithi121 bash[36902]: audit 2022-04-22T20:34:19.235035+0000 mon.a (mon.0) 9799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:20 smithi115 bash[45495]: audit 2022-04-22T20:34:19.235035+0000 mon.a (mon.0) 9799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:20 smithi115 bash[46934]: audit 2022-04-22T20:34:19.235035+0000 mon.a (mon.0) 9799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:21 smithi121 bash[36902]: cluster 2022-04-22T20:34:19.722221+0000 mgr.y (mgr.34107) 6440 : cluster [DBG] pgmap v7589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:21 smithi115 bash[45495]: cluster 2022-04-22T20:34:19.722221+0000 mgr.y (mgr.34107) 6440 : cluster [DBG] pgmap v7589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:21 smithi115 bash[46934]: cluster 2022-04-22T20:34:19.722221+0000 mgr.y (mgr.34107) 6440 : cluster [DBG] pgmap v7589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:22.338Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:23 smithi121 bash[36902]: cluster 2022-04-22T20:34:21.723331+0000 mgr.y (mgr.34107) 6441 : cluster [DBG] pgmap v7590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:23 smithi121 bash[36902]: audit 2022-04-22T20:34:22.725718+0000 mon.a (mon.0) 9800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:23 smithi121 bash[36902]: audit 2022-04-22T20:34:22.782935+0000 mon.a (mon.0) 9801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[45495]: cluster 2022-04-22T20:34:21.723331+0000 mgr.y (mgr.34107) 6441 : cluster [DBG] pgmap v7590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[45495]: audit 2022-04-22T20:34:22.725718+0000 mon.a (mon.0) 9800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[45495]: audit 2022-04-22T20:34:22.782935+0000 mon.a (mon.0) 9801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[46934]: cluster 2022-04-22T20:34:21.723331+0000 mgr.y (mgr.34107) 6441 : cluster [DBG] pgmap v7590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[46934]: audit 2022-04-22T20:34:22.725718+0000 mon.a (mon.0) 9800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:23 smithi115 bash[46934]: audit 2022-04-22T20:34:22.782935+0000 mon.a (mon.0) 9801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:24.219Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:25 smithi121 bash[36902]: cluster 2022-04-22T20:34:23.724639+0000 mgr.y (mgr.34107) 6442 : cluster [DBG] pgmap v7591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:25 smithi121 bash[36902]: audit 2022-04-22T20:34:24.238060+0000 mon.a (mon.0) 9802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:25 smithi115 bash[46934]: cluster 2022-04-22T20:34:23.724639+0000 mgr.y (mgr.34107) 6442 : cluster [DBG] pgmap v7591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:25 smithi115 bash[46934]: audit 2022-04-22T20:34:24.238060+0000 mon.a (mon.0) 9802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:25 smithi115 bash[45495]: cluster 2022-04-22T20:34:23.724639+0000 mgr.y (mgr.34107) 6442 : cluster [DBG] pgmap v7591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:25 smithi115 bash[45495]: audit 2022-04-22T20:34:24.238060+0000 mon.a (mon.0) 9802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:26.552 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 100s ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (2h) 100s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:34:26.894 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 100s ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 100s ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 100s ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 100s ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 2m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 100s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 100s ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 100s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 100s ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:34:26.895 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:34:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:34:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:34:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (2h) 2m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:34:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 2m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:34:27.299 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:34:27.300 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:34:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:27 smithi121 bash[36902]: cluster 2022-04-22T20:34:25.725775+0000 mgr.y (mgr.34107) 6443 : cluster [DBG] pgmap v7592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:27 smithi115 bash[46934]: cluster 2022-04-22T20:34:25.725775+0000 mgr.y (mgr.34107) 6443 : cluster [DBG] pgmap v7592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:27.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:27 smithi115 bash[45495]: cluster 2022-04-22T20:34:25.725775+0000 mgr.y (mgr.34107) 6443 : cluster [DBG] pgmap v7592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:28 smithi121 bash[36902]: audit 2022-04-22T20:34:26.526617+0000 mgr.y (mgr.34107) 6444 : audit [DBG] from='client.52015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:28 smithi121 bash[36902]: audit 2022-04-22T20:34:26.862373+0000 mgr.y (mgr.34107) 6445 : audit [DBG] from='client.52021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:28 smithi121 bash[36902]: audit 2022-04-22T20:34:27.291385+0000 mon.a (mon.0) 9803 : audit [DBG] from='client.? 172.21.15.115:0/2701456698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:34:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:28 smithi121 bash[36902]: audit 2022-04-22T20:34:27.729120+0000 mon.a (mon.0) 9804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:28 smithi121 bash[36902]: audit 2022-04-22T20:34:27.785995+0000 mon.a (mon.0) 9805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[45495]: audit 2022-04-22T20:34:26.526617+0000 mgr.y (mgr.34107) 6444 : audit [DBG] from='client.52015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[45495]: audit 2022-04-22T20:34:26.862373+0000 mgr.y (mgr.34107) 6445 : audit [DBG] from='client.52021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[45495]: audit 2022-04-22T20:34:27.291385+0000 mon.a (mon.0) 9803 : audit [DBG] from='client.? 172.21.15.115:0/2701456698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[45495]: audit 2022-04-22T20:34:27.729120+0000 mon.a (mon.0) 9804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[45495]: audit 2022-04-22T20:34:27.785995+0000 mon.a (mon.0) 9805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[46934]: audit 2022-04-22T20:34:26.526617+0000 mgr.y (mgr.34107) 6444 : audit [DBG] from='client.52015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[46934]: audit 2022-04-22T20:34:26.862373+0000 mgr.y (mgr.34107) 6445 : audit [DBG] from='client.52021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[46934]: audit 2022-04-22T20:34:27.291385+0000 mon.a (mon.0) 9803 : audit [DBG] from='client.? 172.21.15.115:0/2701456698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[46934]: audit 2022-04-22T20:34:27.729120+0000 mon.a (mon.0) 9804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:28 smithi115 bash[46934]: audit 2022-04-22T20:34:27.785995+0000 mon.a (mon.0) 9805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:29 smithi121 bash[36902]: cluster 2022-04-22T20:34:27.726870+0000 mgr.y (mgr.34107) 6446 : cluster [DBG] pgmap v7593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:29 smithi121 bash[36902]: audit 2022-04-22T20:34:29.240460+0000 mon.a (mon.0) 9806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:29 smithi115 bash[45495]: cluster 2022-04-22T20:34:27.726870+0000 mgr.y (mgr.34107) 6446 : cluster [DBG] pgmap v7593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:29 smithi115 bash[45495]: audit 2022-04-22T20:34:29.240460+0000 mon.a (mon.0) 9806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:29 smithi115 bash[46934]: cluster 2022-04-22T20:34:27.726870+0000 mgr.y (mgr.34107) 6446 : cluster [DBG] pgmap v7593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:29 smithi115 bash[46934]: audit 2022-04-22T20:34:29.240460+0000 mon.a (mon.0) 9806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:31 smithi121 bash[36902]: cluster 2022-04-22T20:34:29.728148+0000 mgr.y (mgr.34107) 6447 : cluster [DBG] pgmap v7594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:31 smithi115 bash[45495]: cluster 2022-04-22T20:34:29.728148+0000 mgr.y (mgr.34107) 6447 : cluster [DBG] pgmap v7594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:31 smithi115 bash[46934]: cluster 2022-04-22T20:34:29.728148+0000 mgr.y (mgr.34107) 6447 : cluster [DBG] pgmap v7594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:32.338Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:33 smithi121 bash[36902]: cluster 2022-04-22T20:34:31.729396+0000 mgr.y (mgr.34107) 6448 : cluster [DBG] pgmap v7595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:33 smithi121 bash[36902]: audit 2022-04-22T20:34:32.732541+0000 mon.a (mon.0) 9807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:33 smithi121 bash[36902]: audit 2022-04-22T20:34:32.788947+0000 mon.a (mon.0) 9808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[46934]: cluster 2022-04-22T20:34:31.729396+0000 mgr.y (mgr.34107) 6448 : cluster [DBG] pgmap v7595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[46934]: audit 2022-04-22T20:34:32.732541+0000 mon.a (mon.0) 9807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:33.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[46934]: audit 2022-04-22T20:34:32.788947+0000 mon.a (mon.0) 9808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[45495]: cluster 2022-04-22T20:34:31.729396+0000 mgr.y (mgr.34107) 6448 : cluster [DBG] pgmap v7595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[45495]: audit 2022-04-22T20:34:32.732541+0000 mon.a (mon.0) 9807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:33.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:33 smithi115 bash[45495]: audit 2022-04-22T20:34:32.788947+0000 mon.a (mon.0) 9808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:34.218Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:35 smithi121 bash[36902]: cluster 2022-04-22T20:34:33.730545+0000 mgr.y (mgr.34107) 6449 : cluster [DBG] pgmap v7596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:35 smithi121 bash[36902]: audit 2022-04-22T20:34:34.243449+0000 mon.a (mon.0) 9809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:35 smithi115 bash[45495]: cluster 2022-04-22T20:34:33.730545+0000 mgr.y (mgr.34107) 6449 : cluster [DBG] pgmap v7596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:35 smithi115 bash[45495]: audit 2022-04-22T20:34:34.243449+0000 mon.a (mon.0) 9809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:35 smithi115 bash[46934]: cluster 2022-04-22T20:34:33.730545+0000 mgr.y (mgr.34107) 6449 : cluster [DBG] pgmap v7596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:35 smithi115 bash[46934]: audit 2022-04-22T20:34:34.243449+0000 mon.a (mon.0) 9809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:37 smithi121 bash[36902]: cluster 2022-04-22T20:34:35.731656+0000 mgr.y (mgr.34107) 6450 : cluster [DBG] pgmap v7597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:37.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:37 smithi115 bash[45495]: cluster 2022-04-22T20:34:35.731656+0000 mgr.y (mgr.34107) 6450 : cluster [DBG] pgmap v7597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:37 smithi115 bash[46934]: cluster 2022-04-22T20:34:35.731656+0000 mgr.y (mgr.34107) 6450 : cluster [DBG] pgmap v7597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:38 smithi121 bash[36902]: audit 2022-04-22T20:34:37.736168+0000 mon.a (mon.0) 9810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:38 smithi121 bash[36902]: audit 2022-04-22T20:34:37.791970+0000 mon.a (mon.0) 9811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:38 smithi115 bash[45495]: audit 2022-04-22T20:34:37.736168+0000 mon.a (mon.0) 9810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:38 smithi115 bash[45495]: audit 2022-04-22T20:34:37.791970+0000 mon.a (mon.0) 9811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:38 smithi115 bash[46934]: audit 2022-04-22T20:34:37.736168+0000 mon.a (mon.0) 9810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:38.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:38 smithi115 bash[46934]: audit 2022-04-22T20:34:37.791970+0000 mon.a (mon.0) 9811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:39 smithi121 bash[36902]: cluster 2022-04-22T20:34:37.732689+0000 mgr.y (mgr.34107) 6451 : cluster [DBG] pgmap v7598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:39 smithi115 bash[46934]: cluster 2022-04-22T20:34:37.732689+0000 mgr.y (mgr.34107) 6451 : cluster [DBG] pgmap v7598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:39.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:39 smithi115 bash[45495]: cluster 2022-04-22T20:34:37.732689+0000 mgr.y (mgr.34107) 6451 : cluster [DBG] pgmap v7598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:40 smithi121 bash[36902]: audit 2022-04-22T20:34:39.246225+0000 mon.a (mon.0) 9812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:40 smithi115 bash[45495]: audit 2022-04-22T20:34:39.246225+0000 mon.a (mon.0) 9812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:40 smithi115 bash[46934]: audit 2022-04-22T20:34:39.246225+0000 mon.a (mon.0) 9812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:41 smithi121 bash[36902]: cluster 2022-04-22T20:34:39.733787+0000 mgr.y (mgr.34107) 6452 : cluster [DBG] pgmap v7599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:41 smithi115 bash[45495]: cluster 2022-04-22T20:34:39.733787+0000 mgr.y (mgr.34107) 6452 : cluster [DBG] pgmap v7599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:41.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:41 smithi115 bash[46934]: cluster 2022-04-22T20:34:39.733787+0000 mgr.y (mgr.34107) 6452 : cluster [DBG] pgmap v7599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:42.795 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:42.340Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:43 smithi121 bash[36902]: cluster 2022-04-22T20:34:41.734913+0000 mgr.y (mgr.34107) 6453 : cluster [DBG] pgmap v7600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:43 smithi121 bash[36902]: audit 2022-04-22T20:34:42.739362+0000 mon.a (mon.0) 9813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:43 smithi121 bash[36902]: audit 2022-04-22T20:34:42.795207+0000 mon.a (mon.0) 9814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[46934]: cluster 2022-04-22T20:34:41.734913+0000 mgr.y (mgr.34107) 6453 : cluster [DBG] pgmap v7600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[46934]: audit 2022-04-22T20:34:42.739362+0000 mon.a (mon.0) 9813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[46934]: audit 2022-04-22T20:34:42.795207+0000 mon.a (mon.0) 9814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[45495]: cluster 2022-04-22T20:34:41.734913+0000 mgr.y (mgr.34107) 6453 : cluster [DBG] pgmap v7600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[45495]: audit 2022-04-22T20:34:42.739362+0000 mon.a (mon.0) 9813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:43 smithi115 bash[45495]: audit 2022-04-22T20:34:42.795207+0000 mon.a (mon.0) 9814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:44.219Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:45 smithi121 bash[36902]: cluster 2022-04-22T20:34:43.736004+0000 mgr.y (mgr.34107) 6454 : cluster [DBG] pgmap v7601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:45 smithi121 bash[36902]: audit 2022-04-22T20:34:44.248708+0000 mon.a (mon.0) 9815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:45 smithi115 bash[46934]: cluster 2022-04-22T20:34:43.736004+0000 mgr.y (mgr.34107) 6454 : cluster [DBG] pgmap v7601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:45 smithi115 bash[46934]: audit 2022-04-22T20:34:44.248708+0000 mon.a (mon.0) 9815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:45 smithi115 bash[45495]: cluster 2022-04-22T20:34:43.736004+0000 mgr.y (mgr.34107) 6454 : cluster [DBG] pgmap v7601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:45 smithi115 bash[45495]: audit 2022-04-22T20:34:44.248708+0000 mon.a (mon.0) 9815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[45495]: cluster 2022-04-22T20:34:45.737106+0000 mgr.y (mgr.34107) 6455 : cluster [DBG] pgmap v7602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[45495]: audit 2022-04-22T20:34:46.706395+0000 mon.a (mon.0) 9816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:34:47.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[45495]: audit 2022-04-22T20:34:46.709604+0000 mon.a (mon.0) 9817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:34:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[45495]: audit 2022-04-22T20:34:46.711583+0000 mon.a (mon.0) 9818 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:34:47.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[45495]: audit 2022-04-22T20:34:47.059119+0000 mon.a (mon.0) 9819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:34:47.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[46934]: cluster 2022-04-22T20:34:45.737106+0000 mgr.y (mgr.34107) 6455 : cluster [DBG] pgmap v7602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[46934]: audit 2022-04-22T20:34:46.706395+0000 mon.a (mon.0) 9816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:34:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[46934]: audit 2022-04-22T20:34:46.709604+0000 mon.a (mon.0) 9817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:34:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[46934]: audit 2022-04-22T20:34:46.711583+0000 mon.a (mon.0) 9818 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:34:47.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:47 smithi115 bash[46934]: audit 2022-04-22T20:34:47.059119+0000 mon.a (mon.0) 9819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:34:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:47 smithi121 bash[36902]: cluster 2022-04-22T20:34:45.737106+0000 mgr.y (mgr.34107) 6455 : cluster [DBG] pgmap v7602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:47 smithi121 bash[36902]: audit 2022-04-22T20:34:46.706395+0000 mon.a (mon.0) 9816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:34:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:47 smithi121 bash[36902]: audit 2022-04-22T20:34:46.709604+0000 mon.a (mon.0) 9817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:34:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:47 smithi121 bash[36902]: audit 2022-04-22T20:34:46.711583+0000 mon.a (mon.0) 9818 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:34:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:47 smithi121 bash[36902]: audit 2022-04-22T20:34:47.059119+0000 mon.a (mon.0) 9819 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:34:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:48 smithi121 bash[36902]: audit 2022-04-22T20:34:47.742941+0000 mon.a (mon.0) 9820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:48 smithi121 bash[36902]: audit 2022-04-22T20:34:47.798291+0000 mon.a (mon.0) 9821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:48 smithi115 bash[45495]: audit 2022-04-22T20:34:47.742941+0000 mon.a (mon.0) 9820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:48 smithi115 bash[45495]: audit 2022-04-22T20:34:47.798291+0000 mon.a (mon.0) 9821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:48 smithi115 bash[46934]: audit 2022-04-22T20:34:47.742941+0000 mon.a (mon.0) 9820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:48 smithi115 bash[46934]: audit 2022-04-22T20:34:47.798291+0000 mon.a (mon.0) 9821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:49 smithi121 bash[36902]: cluster 2022-04-22T20:34:47.738205+0000 mgr.y (mgr.34107) 6456 : cluster [DBG] pgmap v7603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:49 smithi115 bash[45495]: cluster 2022-04-22T20:34:47.738205+0000 mgr.y (mgr.34107) 6456 : cluster [DBG] pgmap v7603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:49 smithi115 bash[46934]: cluster 2022-04-22T20:34:47.738205+0000 mgr.y (mgr.34107) 6456 : cluster [DBG] pgmap v7603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:50.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:50 smithi115 bash[45495]: audit 2022-04-22T20:34:49.251637+0000 mon.a (mon.0) 9822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:50.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:50 smithi115 bash[46934]: audit 2022-04-22T20:34:49.251637+0000 mon.a (mon.0) 9822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:50.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:50 smithi121 bash[36902]: audit 2022-04-22T20:34:49.251637+0000 mon.a (mon.0) 9822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:51 smithi115 bash[45495]: cluster 2022-04-22T20:34:49.739321+0000 mgr.y (mgr.34107) 6457 : cluster [DBG] pgmap v7604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:51 smithi115 bash[46934]: cluster 2022-04-22T20:34:49.739321+0000 mgr.y (mgr.34107) 6457 : cluster [DBG] pgmap v7604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:51 smithi121 bash[36902]: cluster 2022-04-22T20:34:49.739321+0000 mgr.y (mgr.34107) 6457 : cluster [DBG] pgmap v7604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:52.341Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[45495]: cluster 2022-04-22T20:34:51.740418+0000 mgr.y (mgr.34107) 6458 : cluster [DBG] pgmap v7605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[45495]: audit 2022-04-22T20:34:52.746678+0000 mon.a (mon.0) 9823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[45495]: audit 2022-04-22T20:34:52.801433+0000 mon.a (mon.0) 9824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[46934]: cluster 2022-04-22T20:34:51.740418+0000 mgr.y (mgr.34107) 6458 : cluster [DBG] pgmap v7605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[46934]: audit 2022-04-22T20:34:52.746678+0000 mon.a (mon.0) 9823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:53 smithi115 bash[46934]: audit 2022-04-22T20:34:52.801433+0000 mon.a (mon.0) 9824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:53 smithi121 bash[36902]: cluster 2022-04-22T20:34:51.740418+0000 mgr.y (mgr.34107) 6458 : cluster [DBG] pgmap v7605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:53 smithi121 bash[36902]: audit 2022-04-22T20:34:52.746678+0000 mon.a (mon.0) 9823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:53 smithi121 bash[36902]: audit 2022-04-22T20:34:52.801433+0000 mon.a (mon.0) 9824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:34:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:34:54.219Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:34:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:55 smithi121 bash[36902]: cluster 2022-04-22T20:34:53.741525+0000 mgr.y (mgr.34107) 6459 : cluster [DBG] pgmap v7606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:55 smithi121 bash[36902]: audit 2022-04-22T20:34:54.254602+0000 mon.a (mon.0) 9825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:55 smithi115 bash[45495]: cluster 2022-04-22T20:34:53.741525+0000 mgr.y (mgr.34107) 6459 : cluster [DBG] pgmap v7606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:55 smithi115 bash[45495]: audit 2022-04-22T20:34:54.254602+0000 mon.a (mon.0) 9825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:55 smithi115 bash[46934]: cluster 2022-04-22T20:34:53.741525+0000 mgr.y (mgr.34107) 6459 : cluster [DBG] pgmap v7606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:55.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:55 smithi115 bash[46934]: audit 2022-04-22T20:34:54.254602+0000 mon.a (mon.0) 9825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:34:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:57 smithi121 bash[36902]: cluster 2022-04-22T20:34:55.742652+0000 mgr.y (mgr.34107) 6460 : cluster [DBG] pgmap v7607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:57 smithi115 bash[45495]: cluster 2022-04-22T20:34:55.742652+0000 mgr.y (mgr.34107) 6460 : cluster [DBG] pgmap v7607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:57 smithi115 bash[46934]: cluster 2022-04-22T20:34:55.742652+0000 mgr.y (mgr.34107) 6460 : cluster [DBG] pgmap v7607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:57.679 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:34:58.024 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:34:58.024 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (2h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:34:58.025 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:34:58.026 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:34:58.027 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:34:58.430 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:34:58.430 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:34:58.430 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:34:58.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:58.430 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:34:58.431 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:34:58.432 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:34:58.432 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:34:58.432 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:34:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:58 smithi121 bash[36902]: audit 2022-04-22T20:34:57.750167+0000 mon.a (mon.0) 9826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:58 smithi121 bash[36902]: audit 2022-04-22T20:34:57.804221+0000 mon.a (mon.0) 9827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:58 smithi115 bash[45495]: audit 2022-04-22T20:34:57.750167+0000 mon.a (mon.0) 9826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:58.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:58 smithi115 bash[45495]: audit 2022-04-22T20:34:57.804221+0000 mon.a (mon.0) 9827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:58 smithi115 bash[46934]: audit 2022-04-22T20:34:57.750167+0000 mon.a (mon.0) 9826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:34:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:58 smithi115 bash[46934]: audit 2022-04-22T20:34:57.804221+0000 mon.a (mon.0) 9827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:34:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:59 smithi121 bash[36902]: audit 2022-04-22T20:34:57.655267+0000 mgr.y (mgr.34107) 6461 : audit [DBG] from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:59 smithi121 bash[36902]: cluster 2022-04-22T20:34:57.743726+0000 mgr.y (mgr.34107) 6462 : cluster [DBG] pgmap v7608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:59 smithi121 bash[36902]: audit 2022-04-22T20:34:57.997146+0000 mgr.y (mgr.34107) 6463 : audit [DBG] from='client.42108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:34:59 smithi121 bash[36902]: audit 2022-04-22T20:34:58.422586+0000 mon.c (mon.1) 251 : audit [DBG] from='client.? 172.21.15.115:0/3035479070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:34:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[45495]: audit 2022-04-22T20:34:57.655267+0000 mgr.y (mgr.34107) 6461 : audit [DBG] from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[45495]: cluster 2022-04-22T20:34:57.743726+0000 mgr.y (mgr.34107) 6462 : cluster [DBG] pgmap v7608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[45495]: audit 2022-04-22T20:34:57.997146+0000 mgr.y (mgr.34107) 6463 : audit [DBG] from='client.42108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[45495]: audit 2022-04-22T20:34:58.422586+0000 mon.c (mon.1) 251 : audit [DBG] from='client.? 172.21.15.115:0/3035479070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:34:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[46934]: audit 2022-04-22T20:34:57.655267+0000 mgr.y (mgr.34107) 6461 : audit [DBG] from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[46934]: cluster 2022-04-22T20:34:57.743726+0000 mgr.y (mgr.34107) 6462 : cluster [DBG] pgmap v7608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:34:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[46934]: audit 2022-04-22T20:34:57.997146+0000 mgr.y (mgr.34107) 6463 : audit [DBG] from='client.42108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:34:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:34:59 smithi115 bash[46934]: audit 2022-04-22T20:34:58.422586+0000 mon.c (mon.1) 251 : audit [DBG] from='client.? 172.21.15.115:0/3035479070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:35:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:00 smithi121 bash[36902]: audit 2022-04-22T20:34:59.257677+0000 mon.a (mon.0) 9828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:00 smithi115 bash[45495]: audit 2022-04-22T20:34:59.257677+0000 mon.a (mon.0) 9828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:00 smithi115 bash[46934]: audit 2022-04-22T20:34:59.257677+0000 mon.a (mon.0) 9828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:01 smithi121 bash[36902]: cluster 2022-04-22T20:34:59.744809+0000 mgr.y (mgr.34107) 6464 : cluster [DBG] pgmap v7609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:01 smithi115 bash[45495]: cluster 2022-04-22T20:34:59.744809+0000 mgr.y (mgr.34107) 6464 : cluster [DBG] pgmap v7609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:01 smithi115 bash[46934]: cluster 2022-04-22T20:34:59.744809+0000 mgr.y (mgr.34107) 6464 : cluster [DBG] pgmap v7609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:02.339Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:03 smithi121 bash[36902]: cluster 2022-04-22T20:35:01.745957+0000 mgr.y (mgr.34107) 6465 : cluster [DBG] pgmap v7610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:03 smithi121 bash[36902]: audit 2022-04-22T20:35:02.753486+0000 mon.a (mon.0) 9829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:03 smithi121 bash[36902]: audit 2022-04-22T20:35:02.806718+0000 mon.a (mon.0) 9830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[45495]: cluster 2022-04-22T20:35:01.745957+0000 mgr.y (mgr.34107) 6465 : cluster [DBG] pgmap v7610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[45495]: audit 2022-04-22T20:35:02.753486+0000 mon.a (mon.0) 9829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[45495]: audit 2022-04-22T20:35:02.806718+0000 mon.a (mon.0) 9830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[46934]: cluster 2022-04-22T20:35:01.745957+0000 mgr.y (mgr.34107) 6465 : cluster [DBG] pgmap v7610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[46934]: audit 2022-04-22T20:35:02.753486+0000 mon.a (mon.0) 9829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:03 smithi115 bash[46934]: audit 2022-04-22T20:35:02.806718+0000 mon.a (mon.0) 9830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:04.221Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:05 smithi121 bash[36902]: cluster 2022-04-22T20:35:03.747071+0000 mgr.y (mgr.34107) 6466 : cluster [DBG] pgmap v7611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:05 smithi121 bash[36902]: audit 2022-04-22T20:35:04.260564+0000 mon.a (mon.0) 9831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:05 smithi115 bash[46934]: cluster 2022-04-22T20:35:03.747071+0000 mgr.y (mgr.34107) 6466 : cluster [DBG] pgmap v7611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:05 smithi115 bash[46934]: audit 2022-04-22T20:35:04.260564+0000 mon.a (mon.0) 9831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:05 smithi115 bash[45495]: cluster 2022-04-22T20:35:03.747071+0000 mgr.y (mgr.34107) 6466 : cluster [DBG] pgmap v7611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:05 smithi115 bash[45495]: audit 2022-04-22T20:35:04.260564+0000 mon.a (mon.0) 9831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:07 smithi121 bash[36902]: cluster 2022-04-22T20:35:05.748208+0000 mgr.y (mgr.34107) 6467 : cluster [DBG] pgmap v7612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:07 smithi115 bash[45495]: cluster 2022-04-22T20:35:05.748208+0000 mgr.y (mgr.34107) 6467 : cluster [DBG] pgmap v7612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:07 smithi115 bash[46934]: cluster 2022-04-22T20:35:05.748208+0000 mgr.y (mgr.34107) 6467 : cluster [DBG] pgmap v7612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:08 smithi121 bash[36902]: audit 2022-04-22T20:35:07.756990+0000 mon.a (mon.0) 9832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:08 smithi121 bash[36902]: audit 2022-04-22T20:35:07.809959+0000 mon.a (mon.0) 9833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:08 smithi115 bash[45495]: audit 2022-04-22T20:35:07.756990+0000 mon.a (mon.0) 9832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:08 smithi115 bash[45495]: audit 2022-04-22T20:35:07.809959+0000 mon.a (mon.0) 9833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:08 smithi115 bash[46934]: audit 2022-04-22T20:35:07.756990+0000 mon.a (mon.0) 9832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:08 smithi115 bash[46934]: audit 2022-04-22T20:35:07.809959+0000 mon.a (mon.0) 9833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:09 smithi121 bash[36902]: cluster 2022-04-22T20:35:07.749199+0000 mgr.y (mgr.34107) 6468 : cluster [DBG] pgmap v7613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:09 smithi115 bash[45495]: cluster 2022-04-22T20:35:07.749199+0000 mgr.y (mgr.34107) 6468 : cluster [DBG] pgmap v7613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:09 smithi115 bash[46934]: cluster 2022-04-22T20:35:07.749199+0000 mgr.y (mgr.34107) 6468 : cluster [DBG] pgmap v7613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:10 smithi115 bash[45495]: audit 2022-04-22T20:35:09.263288+0000 mon.a (mon.0) 9834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:10.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:10 smithi115 bash[46934]: audit 2022-04-22T20:35:09.263288+0000 mon.a (mon.0) 9834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:10 smithi121 bash[36902]: audit 2022-04-22T20:35:09.263288+0000 mon.a (mon.0) 9834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:11 smithi115 bash[45495]: cluster 2022-04-22T20:35:09.750295+0000 mgr.y (mgr.34107) 6469 : cluster [DBG] pgmap v7614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:11 smithi115 bash[46934]: cluster 2022-04-22T20:35:09.750295+0000 mgr.y (mgr.34107) 6469 : cluster [DBG] pgmap v7614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:11 smithi121 bash[36902]: cluster 2022-04-22T20:35:09.750295+0000 mgr.y (mgr.34107) 6469 : cluster [DBG] pgmap v7614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:12.343Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[45495]: cluster 2022-04-22T20:35:11.751382+0000 mgr.y (mgr.34107) 6470 : cluster [DBG] pgmap v7615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[45495]: audit 2022-04-22T20:35:12.760237+0000 mon.a (mon.0) 9835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:13.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[45495]: audit 2022-04-22T20:35:12.813068+0000 mon.a (mon.0) 9836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[46934]: cluster 2022-04-22T20:35:11.751382+0000 mgr.y (mgr.34107) 6470 : cluster [DBG] pgmap v7615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[46934]: audit 2022-04-22T20:35:12.760237+0000 mon.a (mon.0) 9835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:13 smithi115 bash[46934]: audit 2022-04-22T20:35:12.813068+0000 mon.a (mon.0) 9836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:13 smithi121 bash[36902]: cluster 2022-04-22T20:35:11.751382+0000 mgr.y (mgr.34107) 6470 : cluster [DBG] pgmap v7615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:13 smithi121 bash[36902]: audit 2022-04-22T20:35:12.760237+0000 mon.a (mon.0) 9835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:13 smithi121 bash[36902]: audit 2022-04-22T20:35:12.813068+0000 mon.a (mon.0) 9836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:14.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:14.222Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:15 smithi121 bash[36902]: cluster 2022-04-22T20:35:13.752494+0000 mgr.y (mgr.34107) 6471 : cluster [DBG] pgmap v7616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:15 smithi121 bash[36902]: audit 2022-04-22T20:35:14.269124+0000 mon.a (mon.0) 9837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:15 smithi115 bash[45495]: cluster 2022-04-22T20:35:13.752494+0000 mgr.y (mgr.34107) 6471 : cluster [DBG] pgmap v7616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:15 smithi115 bash[45495]: audit 2022-04-22T20:35:14.269124+0000 mon.a (mon.0) 9837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:15 smithi115 bash[46934]: cluster 2022-04-22T20:35:13.752494+0000 mgr.y (mgr.34107) 6471 : cluster [DBG] pgmap v7616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:15 smithi115 bash[46934]: audit 2022-04-22T20:35:14.269124+0000 mon.a (mon.0) 9837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:16 smithi121 bash[36902]: audit 2022-04-22T20:35:15.447359+0000 mon.a (mon.0) 9838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:35:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:16 smithi121 bash[36902]: audit 2022-04-22T20:35:15.448000+0000 mon.a (mon.0) 9839 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:35:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:16 smithi115 bash[45495]: audit 2022-04-22T20:35:15.447359+0000 mon.a (mon.0) 9838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:35:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:16 smithi115 bash[45495]: audit 2022-04-22T20:35:15.448000+0000 mon.a (mon.0) 9839 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:35:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:16 smithi115 bash[46934]: audit 2022-04-22T20:35:15.447359+0000 mon.a (mon.0) 9838 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:35:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:16 smithi115 bash[46934]: audit 2022-04-22T20:35:15.448000+0000 mon.a (mon.0) 9839 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:35:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:17 smithi121 bash[36902]: cluster 2022-04-22T20:35:15.753560+0000 mgr.y (mgr.34107) 6472 : cluster [DBG] pgmap v7617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:17 smithi115 bash[45495]: cluster 2022-04-22T20:35:15.753560+0000 mgr.y (mgr.34107) 6472 : cluster [DBG] pgmap v7617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:17 smithi115 bash[46934]: cluster 2022-04-22T20:35:15.753560+0000 mgr.y (mgr.34107) 6472 : cluster [DBG] pgmap v7617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:18 smithi121 bash[36902]: audit 2022-04-22T20:35:17.763951+0000 mon.a (mon.0) 9840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:18 smithi121 bash[36902]: audit 2022-04-22T20:35:17.816135+0000 mon.a (mon.0) 9841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:18 smithi115 bash[45495]: audit 2022-04-22T20:35:17.763951+0000 mon.a (mon.0) 9840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:18.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:18 smithi115 bash[45495]: audit 2022-04-22T20:35:17.816135+0000 mon.a (mon.0) 9841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:18 smithi115 bash[46934]: audit 2022-04-22T20:35:17.763951+0000 mon.a (mon.0) 9840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:18 smithi115 bash[46934]: audit 2022-04-22T20:35:17.816135+0000 mon.a (mon.0) 9841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:19 smithi121 bash[36902]: cluster 2022-04-22T20:35:17.754601+0000 mgr.y (mgr.34107) 6473 : cluster [DBG] pgmap v7618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:19 smithi115 bash[45495]: cluster 2022-04-22T20:35:17.754601+0000 mgr.y (mgr.34107) 6473 : cluster [DBG] pgmap v7618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:19 smithi115 bash[46934]: cluster 2022-04-22T20:35:17.754601+0000 mgr.y (mgr.34107) 6473 : cluster [DBG] pgmap v7618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:20 smithi121 bash[36902]: audit 2022-04-22T20:35:19.267825+0000 mon.a (mon.0) 9842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:20 smithi115 bash[45495]: audit 2022-04-22T20:35:19.267825+0000 mon.a (mon.0) 9842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:20.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:20 smithi115 bash[46934]: audit 2022-04-22T20:35:19.267825+0000 mon.a (mon.0) 9842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:21 smithi121 bash[36902]: cluster 2022-04-22T20:35:19.755747+0000 mgr.y (mgr.34107) 6474 : cluster [DBG] pgmap v7619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:21 smithi115 bash[45495]: cluster 2022-04-22T20:35:19.755747+0000 mgr.y (mgr.34107) 6474 : cluster [DBG] pgmap v7619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:21 smithi115 bash[46934]: cluster 2022-04-22T20:35:19.755747+0000 mgr.y (mgr.34107) 6474 : cluster [DBG] pgmap v7619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:22.342Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:23 smithi121 bash[36902]: cluster 2022-04-22T20:35:21.756915+0000 mgr.y (mgr.34107) 6475 : cluster [DBG] pgmap v7620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:23 smithi121 bash[36902]: audit 2022-04-22T20:35:22.767682+0000 mon.a (mon.0) 9843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:23 smithi121 bash[36902]: audit 2022-04-22T20:35:22.819165+0000 mon.a (mon.0) 9844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[45495]: cluster 2022-04-22T20:35:21.756915+0000 mgr.y (mgr.34107) 6475 : cluster [DBG] pgmap v7620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[45495]: audit 2022-04-22T20:35:22.767682+0000 mon.a (mon.0) 9843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[45495]: audit 2022-04-22T20:35:22.819165+0000 mon.a (mon.0) 9844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[46934]: cluster 2022-04-22T20:35:21.756915+0000 mgr.y (mgr.34107) 6475 : cluster [DBG] pgmap v7620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[46934]: audit 2022-04-22T20:35:22.767682+0000 mon.a (mon.0) 9843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:23 smithi115 bash[46934]: audit 2022-04-22T20:35:22.819165+0000 mon.a (mon.0) 9844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:24.222Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:25 smithi115 bash[45495]: cluster 2022-04-22T20:35:23.758087+0000 mgr.y (mgr.34107) 6476 : cluster [DBG] pgmap v7621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:25 smithi115 bash[45495]: audit 2022-04-22T20:35:24.270611+0000 mon.a (mon.0) 9845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:25 smithi115 bash[46934]: cluster 2022-04-22T20:35:23.758087+0000 mgr.y (mgr.34107) 6476 : cluster [DBG] pgmap v7621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:25 smithi115 bash[46934]: audit 2022-04-22T20:35:24.270611+0000 mon.a (mon.0) 9845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:25 smithi121 bash[36902]: cluster 2022-04-22T20:35:23.758087+0000 mgr.y (mgr.34107) 6476 : cluster [DBG] pgmap v7621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:25 smithi121 bash[36902]: audit 2022-04-22T20:35:24.270611+0000 mon.a (mon.0) 9845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:27 smithi115 bash[45495]: cluster 2022-04-22T20:35:25.759262+0000 mgr.y (mgr.34107) 6477 : cluster [DBG] pgmap v7622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:27 smithi115 bash[46934]: cluster 2022-04-22T20:35:25.759262+0000 mgr.y (mgr.34107) 6477 : cluster [DBG] pgmap v7622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:27 smithi121 bash[36902]: cluster 2022-04-22T20:35:25.759262+0000 mgr.y (mgr.34107) 6477 : cluster [DBG] pgmap v7622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:28 smithi115 bash[46934]: audit 2022-04-22T20:35:27.771255+0000 mon.a (mon.0) 9846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:28 smithi115 bash[46934]: audit 2022-04-22T20:35:27.822218+0000 mon.a (mon.0) 9847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:28 smithi115 bash[45495]: audit 2022-04-22T20:35:27.771255+0000 mon.a (mon.0) 9846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:28 smithi115 bash[45495]: audit 2022-04-22T20:35:27.822218+0000 mon.a (mon.0) 9847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:28 smithi121 bash[36902]: audit 2022-04-22T20:35:27.771255+0000 mon.a (mon.0) 9846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:28 smithi121 bash[36902]: audit 2022-04-22T20:35:27.822218+0000 mon.a (mon.0) 9847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:28.805 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:35:29.149 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:35:29.149 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 2m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:35:29.150 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2h) 2m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (2h) 3m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:35:29.151 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 3m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:35:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:29 smithi115 bash[46934]: cluster 2022-04-22T20:35:27.760416+0000 mgr.y (mgr.34107) 6478 : cluster [DBG] pgmap v7623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:29 smithi115 bash[45495]: cluster 2022-04-22T20:35:27.760416+0000 mgr.y (mgr.34107) 6478 : cluster [DBG] pgmap v7623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:29.558 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:35:29.558 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:35:29.559 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:35:29.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:35:29.559 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:35:29.559 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:35:29.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:35:29.560 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:35:29.560 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:35:29.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:35:29.560 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:35:29.561 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:35:29.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:35:29.561 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:35:29.561 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:35:29.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:35:29.562 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:35:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:29 smithi121 bash[36902]: cluster 2022-04-22T20:35:27.760416+0000 mgr.y (mgr.34107) 6478 : cluster [DBG] pgmap v7623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:30.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[45495]: audit 2022-04-22T20:35:28.781447+0000 mgr.y (mgr.34107) 6479 : audit [DBG] from='client.42120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[45495]: audit 2022-04-22T20:35:29.123979+0000 mgr.y (mgr.34107) 6480 : audit [DBG] from='client.42126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[45495]: audit 2022-04-22T20:35:29.278558+0000 mon.a (mon.0) 9848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[45495]: audit 2022-04-22T20:35:29.551040+0000 mon.c (mon.1) 252 : audit [DBG] from='client.? 172.21.15.115:0/388058642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[46934]: audit 2022-04-22T20:35:28.781447+0000 mgr.y (mgr.34107) 6479 : audit [DBG] from='client.42120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[46934]: audit 2022-04-22T20:35:29.123979+0000 mgr.y (mgr.34107) 6480 : audit [DBG] from='client.42126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[46934]: audit 2022-04-22T20:35:29.278558+0000 mon.a (mon.0) 9848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:30.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:30 smithi115 bash[46934]: audit 2022-04-22T20:35:29.551040+0000 mon.c (mon.1) 252 : audit [DBG] from='client.? 172.21.15.115:0/388058642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:35:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:30 smithi121 bash[36902]: audit 2022-04-22T20:35:28.781447+0000 mgr.y (mgr.34107) 6479 : audit [DBG] from='client.42120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:30 smithi121 bash[36902]: audit 2022-04-22T20:35:29.123979+0000 mgr.y (mgr.34107) 6480 : audit [DBG] from='client.42126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:35:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:30 smithi121 bash[36902]: audit 2022-04-22T20:35:29.278558+0000 mon.a (mon.0) 9848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:30 smithi121 bash[36902]: audit 2022-04-22T20:35:29.551040+0000 mon.c (mon.1) 252 : audit [DBG] from='client.? 172.21.15.115:0/388058642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:35:31.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:31 smithi115 bash[45495]: cluster 2022-04-22T20:35:29.761482+0000 mgr.y (mgr.34107) 6481 : cluster [DBG] pgmap v7624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:31.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:31 smithi115 bash[46934]: cluster 2022-04-22T20:35:29.761482+0000 mgr.y (mgr.34107) 6481 : cluster [DBG] pgmap v7624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:31 smithi121 bash[36902]: cluster 2022-04-22T20:35:29.761482+0000 mgr.y (mgr.34107) 6481 : cluster [DBG] pgmap v7624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:32.343Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:33 smithi121 bash[36902]: cluster 2022-04-22T20:35:31.762567+0000 mgr.y (mgr.34107) 6482 : cluster [DBG] pgmap v7625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:33 smithi121 bash[36902]: audit 2022-04-22T20:35:32.774912+0000 mon.a (mon.0) 9849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:33 smithi121 bash[36902]: audit 2022-04-22T20:35:32.825060+0000 mon.a (mon.0) 9850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[45495]: cluster 2022-04-22T20:35:31.762567+0000 mgr.y (mgr.34107) 6482 : cluster [DBG] pgmap v7625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[45495]: audit 2022-04-22T20:35:32.774912+0000 mon.a (mon.0) 9849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[45495]: audit 2022-04-22T20:35:32.825060+0000 mon.a (mon.0) 9850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[46934]: cluster 2022-04-22T20:35:31.762567+0000 mgr.y (mgr.34107) 6482 : cluster [DBG] pgmap v7625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[46934]: audit 2022-04-22T20:35:32.774912+0000 mon.a (mon.0) 9849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:33 smithi115 bash[46934]: audit 2022-04-22T20:35:32.825060+0000 mon.a (mon.0) 9850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:34.224Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:35 smithi121 bash[36902]: cluster 2022-04-22T20:35:33.763756+0000 mgr.y (mgr.34107) 6483 : cluster [DBG] pgmap v7626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:35 smithi121 bash[36902]: audit 2022-04-22T20:35:34.278166+0000 mon.a (mon.0) 9851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:35 smithi115 bash[45495]: cluster 2022-04-22T20:35:33.763756+0000 mgr.y (mgr.34107) 6483 : cluster [DBG] pgmap v7626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:35 smithi115 bash[45495]: audit 2022-04-22T20:35:34.278166+0000 mon.a (mon.0) 9851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:35 smithi115 bash[46934]: cluster 2022-04-22T20:35:33.763756+0000 mgr.y (mgr.34107) 6483 : cluster [DBG] pgmap v7626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:35.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:35 smithi115 bash[46934]: audit 2022-04-22T20:35:34.278166+0000 mon.a (mon.0) 9851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:37 smithi121 bash[36902]: cluster 2022-04-22T20:35:35.764780+0000 mgr.y (mgr.34107) 6484 : cluster [DBG] pgmap v7627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:37.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:37 smithi115 bash[45495]: cluster 2022-04-22T20:35:35.764780+0000 mgr.y (mgr.34107) 6484 : cluster [DBG] pgmap v7627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:37 smithi115 bash[46934]: cluster 2022-04-22T20:35:35.764780+0000 mgr.y (mgr.34107) 6484 : cluster [DBG] pgmap v7627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:38 smithi121 bash[36902]: audit 2022-04-22T20:35:37.778762+0000 mon.a (mon.0) 9852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:38 smithi121 bash[36902]: audit 2022-04-22T20:35:37.828014+0000 mon.a (mon.0) 9853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:38 smithi115 bash[46934]: audit 2022-04-22T20:35:37.778762+0000 mon.a (mon.0) 9852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:38 smithi115 bash[46934]: audit 2022-04-22T20:35:37.828014+0000 mon.a (mon.0) 9853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:38 smithi115 bash[45495]: audit 2022-04-22T20:35:37.778762+0000 mon.a (mon.0) 9852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:38 smithi115 bash[45495]: audit 2022-04-22T20:35:37.828014+0000 mon.a (mon.0) 9853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:39 smithi121 bash[36902]: cluster 2022-04-22T20:35:37.765876+0000 mgr.y (mgr.34107) 6485 : cluster [DBG] pgmap v7628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:39 smithi115 bash[45495]: cluster 2022-04-22T20:35:37.765876+0000 mgr.y (mgr.34107) 6485 : cluster [DBG] pgmap v7628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:39 smithi115 bash[46934]: cluster 2022-04-22T20:35:37.765876+0000 mgr.y (mgr.34107) 6485 : cluster [DBG] pgmap v7628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:40 smithi121 bash[36902]: audit 2022-04-22T20:35:39.281138+0000 mon.a (mon.0) 9854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:40 smithi115 bash[45495]: audit 2022-04-22T20:35:39.281138+0000 mon.a (mon.0) 9854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:40.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:40 smithi115 bash[46934]: audit 2022-04-22T20:35:39.281138+0000 mon.a (mon.0) 9854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:41 smithi121 bash[36902]: cluster 2022-04-22T20:35:39.766961+0000 mgr.y (mgr.34107) 6486 : cluster [DBG] pgmap v7629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:41 smithi115 bash[45495]: cluster 2022-04-22T20:35:39.766961+0000 mgr.y (mgr.34107) 6486 : cluster [DBG] pgmap v7629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:41.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:41 smithi115 bash[46934]: cluster 2022-04-22T20:35:39.766961+0000 mgr.y (mgr.34107) 6486 : cluster [DBG] pgmap v7629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:42.342Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:43 smithi121 bash[36902]: cluster 2022-04-22T20:35:41.768105+0000 mgr.y (mgr.34107) 6487 : cluster [DBG] pgmap v7630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:43 smithi121 bash[36902]: audit 2022-04-22T20:35:42.782397+0000 mon.a (mon.0) 9855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:43 smithi121 bash[36902]: audit 2022-04-22T20:35:42.830857+0000 mon.a (mon.0) 9856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[45495]: cluster 2022-04-22T20:35:41.768105+0000 mgr.y (mgr.34107) 6487 : cluster [DBG] pgmap v7630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[45495]: audit 2022-04-22T20:35:42.782397+0000 mon.a (mon.0) 9855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:43.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[45495]: audit 2022-04-22T20:35:42.830857+0000 mon.a (mon.0) 9856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[46934]: cluster 2022-04-22T20:35:41.768105+0000 mgr.y (mgr.34107) 6487 : cluster [DBG] pgmap v7630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[46934]: audit 2022-04-22T20:35:42.782397+0000 mon.a (mon.0) 9855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:43 smithi115 bash[46934]: audit 2022-04-22T20:35:42.830857+0000 mon.a (mon.0) 9856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:44.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:45 smithi121 bash[36902]: cluster 2022-04-22T20:35:43.769270+0000 mgr.y (mgr.34107) 6488 : cluster [DBG] pgmap v7631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:45 smithi121 bash[36902]: audit 2022-04-22T20:35:44.284320+0000 mon.a (mon.0) 9857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:45 smithi115 bash[46934]: cluster 2022-04-22T20:35:43.769270+0000 mgr.y (mgr.34107) 6488 : cluster [DBG] pgmap v7631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:45 smithi115 bash[46934]: audit 2022-04-22T20:35:44.284320+0000 mon.a (mon.0) 9857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:45 smithi115 bash[45495]: cluster 2022-04-22T20:35:43.769270+0000 mgr.y (mgr.34107) 6488 : cluster [DBG] pgmap v7631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:45.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:45 smithi115 bash[45495]: audit 2022-04-22T20:35:44.284320+0000 mon.a (mon.0) 9857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:47.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:47 smithi121 bash[36902]: cluster 2022-04-22T20:35:45.770400+0000 mgr.y (mgr.34107) 6489 : cluster [DBG] pgmap v7632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:47.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:47 smithi121 bash[36902]: audit 2022-04-22T20:35:47.067453+0000 mon.a (mon.0) 9858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:35:47.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:47 smithi121 bash[36902]: audit 2022-04-22T20:35:47.070419+0000 mon.a (mon.0) 9859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:35:47.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:47 smithi121 bash[36902]: audit 2022-04-22T20:35:47.072167+0000 mon.a (mon.0) 9860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:35:47.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[46934]: cluster 2022-04-22T20:35:45.770400+0000 mgr.y (mgr.34107) 6489 : cluster [DBG] pgmap v7632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[46934]: audit 2022-04-22T20:35:47.067453+0000 mon.a (mon.0) 9858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[46934]: audit 2022-04-22T20:35:47.070419+0000 mon.a (mon.0) 9859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[46934]: audit 2022-04-22T20:35:47.072167+0000 mon.a (mon.0) 9860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[45495]: cluster 2022-04-22T20:35:45.770400+0000 mgr.y (mgr.34107) 6489 : cluster [DBG] pgmap v7632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[45495]: audit 2022-04-22T20:35:47.067453+0000 mon.a (mon.0) 9858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[45495]: audit 2022-04-22T20:35:47.070419+0000 mon.a (mon.0) 9859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:35:47.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:47 smithi115 bash[45495]: audit 2022-04-22T20:35:47.072167+0000 mon.a (mon.0) 9860 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:35:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:48 smithi121 bash[36902]: audit 2022-04-22T20:35:47.432513+0000 mon.a (mon.0) 9861 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:35:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:48 smithi121 bash[36902]: cluster 2022-04-22T20:35:47.771596+0000 mgr.y (mgr.34107) 6490 : cluster [DBG] pgmap v7633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:48 smithi121 bash[36902]: audit 2022-04-22T20:35:47.786024+0000 mon.a (mon.0) 9862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:48 smithi121 bash[36902]: audit 2022-04-22T20:35:47.833848+0000 mon.a (mon.0) 9863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[45495]: audit 2022-04-22T20:35:47.432513+0000 mon.a (mon.0) 9861 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:35:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[45495]: cluster 2022-04-22T20:35:47.771596+0000 mgr.y (mgr.34107) 6490 : cluster [DBG] pgmap v7633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[45495]: audit 2022-04-22T20:35:47.786024+0000 mon.a (mon.0) 9862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:48.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[45495]: audit 2022-04-22T20:35:47.833848+0000 mon.a (mon.0) 9863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[46934]: audit 2022-04-22T20:35:47.432513+0000 mon.a (mon.0) 9861 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:35:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[46934]: cluster 2022-04-22T20:35:47.771596+0000 mgr.y (mgr.34107) 6490 : cluster [DBG] pgmap v7633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[46934]: audit 2022-04-22T20:35:47.786024+0000 mon.a (mon.0) 9862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:48.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:48 smithi115 bash[46934]: audit 2022-04-22T20:35:47.833848+0000 mon.a (mon.0) 9863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:50 smithi121 bash[36902]: audit 2022-04-22T20:35:49.287305+0000 mon.a (mon.0) 9864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:50 smithi115 bash[45495]: audit 2022-04-22T20:35:49.287305+0000 mon.a (mon.0) 9864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:50.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:50 smithi115 bash[46934]: audit 2022-04-22T20:35:49.287305+0000 mon.a (mon.0) 9864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:51 smithi121 bash[36902]: cluster 2022-04-22T20:35:49.772710+0000 mgr.y (mgr.34107) 6491 : cluster [DBG] pgmap v7634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:51 smithi115 bash[45495]: cluster 2022-04-22T20:35:49.772710+0000 mgr.y (mgr.34107) 6491 : cluster [DBG] pgmap v7634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:51.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:51 smithi115 bash[46934]: cluster 2022-04-22T20:35:49.772710+0000 mgr.y (mgr.34107) 6491 : cluster [DBG] pgmap v7634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:52.343Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:53 smithi121 bash[36902]: cluster 2022-04-22T20:35:51.773765+0000 mgr.y (mgr.34107) 6492 : cluster [DBG] pgmap v7635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:53 smithi121 bash[36902]: audit 2022-04-22T20:35:52.789656+0000 mon.a (mon.0) 9865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:53 smithi121 bash[36902]: audit 2022-04-22T20:35:52.836874+0000 mon.a (mon.0) 9866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:53.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[45495]: cluster 2022-04-22T20:35:51.773765+0000 mgr.y (mgr.34107) 6492 : cluster [DBG] pgmap v7635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[45495]: audit 2022-04-22T20:35:52.789656+0000 mon.a (mon.0) 9865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:53.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[45495]: audit 2022-04-22T20:35:52.836874+0000 mon.a (mon.0) 9866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[46934]: cluster 2022-04-22T20:35:51.773765+0000 mgr.y (mgr.34107) 6492 : cluster [DBG] pgmap v7635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[46934]: audit 2022-04-22T20:35:52.789656+0000 mon.a (mon.0) 9865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:53.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:53 smithi115 bash[46934]: audit 2022-04-22T20:35:52.836874+0000 mon.a (mon.0) 9866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:35:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:35:54.226Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:35:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:55 smithi121 bash[36902]: cluster 2022-04-22T20:35:53.775038+0000 mgr.y (mgr.34107) 6493 : cluster [DBG] pgmap v7636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:55 smithi121 bash[36902]: audit 2022-04-22T20:35:54.290571+0000 mon.a (mon.0) 9867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:55 smithi115 bash[45495]: cluster 2022-04-22T20:35:53.775038+0000 mgr.y (mgr.34107) 6493 : cluster [DBG] pgmap v7636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:55 smithi115 bash[45495]: audit 2022-04-22T20:35:54.290571+0000 mon.a (mon.0) 9867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:55 smithi115 bash[46934]: cluster 2022-04-22T20:35:53.775038+0000 mgr.y (mgr.34107) 6493 : cluster [DBG] pgmap v7636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:55 smithi115 bash[46934]: audit 2022-04-22T20:35:54.290571+0000 mon.a (mon.0) 9867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:35:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:57 smithi121 bash[36902]: cluster 2022-04-22T20:35:55.776232+0000 mgr.y (mgr.34107) 6494 : cluster [DBG] pgmap v7637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:57 smithi115 bash[45495]: cluster 2022-04-22T20:35:55.776232+0000 mgr.y (mgr.34107) 6494 : cluster [DBG] pgmap v7637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:57 smithi115 bash[46934]: cluster 2022-04-22T20:35:55.776232+0000 mgr.y (mgr.34107) 6494 : cluster [DBG] pgmap v7637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:58 smithi121 bash[36902]: audit 2022-04-22T20:35:57.793260+0000 mon.a (mon.0) 9868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:58 smithi121 bash[36902]: audit 2022-04-22T20:35:57.839910+0000 mon.a (mon.0) 9869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:58 smithi115 bash[45495]: audit 2022-04-22T20:35:57.793260+0000 mon.a (mon.0) 9868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:58.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:58 smithi115 bash[45495]: audit 2022-04-22T20:35:57.839910+0000 mon.a (mon.0) 9869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:58 smithi115 bash[46934]: audit 2022-04-22T20:35:57.793260+0000 mon.a (mon.0) 9868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:35:58.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:58 smithi115 bash[46934]: audit 2022-04-22T20:35:57.839910+0000 mon.a (mon.0) 9869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:35:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:35:59 smithi121 bash[36902]: cluster 2022-04-22T20:35:57.777402+0000 mgr.y (mgr.34107) 6495 : cluster [DBG] pgmap v7638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:35:59 smithi115 bash[45495]: cluster 2022-04-22T20:35:57.777402+0000 mgr.y (mgr.34107) 6495 : cluster [DBG] pgmap v7638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:35:59 smithi115 bash[46934]: cluster 2022-04-22T20:35:57.777402+0000 mgr.y (mgr.34107) 6495 : cluster [DBG] pgmap v7638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:35:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:36:00.275 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:36:00.276 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:36:00.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:36:00.278 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:36:00.278 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:36:00.278 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:36:00.278 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (2h) 4m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:36:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:00 smithi115 bash[45495]: audit 2022-04-22T20:35:59.293639+0000 mon.a (mon.0) 9870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:00.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:00 smithi115 bash[46934]: audit 2022-04-22T20:35:59.293639+0000 mon.a (mon.0) 9870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:36:00.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:36:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:36:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:00 smithi121 bash[36902]: audit 2022-04-22T20:35:59.293639+0000 mon.a (mon.0) 9870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[45495]: cluster 2022-04-22T20:35:59.778426+0000 mgr.y (mgr.34107) 6496 : cluster [DBG] pgmap v7639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[45495]: audit 2022-04-22T20:35:59.902455+0000 mgr.y (mgr.34107) 6497 : audit [DBG] from='client.42138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[45495]: audit 2022-04-22T20:36:00.246468+0000 mgr.y (mgr.34107) 6498 : audit [DBG] from='client.52075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[45495]: audit 2022-04-22T20:36:00.672176+0000 mon.a (mon.0) 9871 : audit [DBG] from='client.? 172.21.15.115:0/3984151743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[46934]: cluster 2022-04-22T20:35:59.778426+0000 mgr.y (mgr.34107) 6496 : cluster [DBG] pgmap v7639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[46934]: audit 2022-04-22T20:35:59.902455+0000 mgr.y (mgr.34107) 6497 : audit [DBG] from='client.42138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[46934]: audit 2022-04-22T20:36:00.246468+0000 mgr.y (mgr.34107) 6498 : audit [DBG] from='client.52075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:01 smithi115 bash[46934]: audit 2022-04-22T20:36:00.672176+0000 mon.a (mon.0) 9871 : audit [DBG] from='client.? 172.21.15.115:0/3984151743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:01 smithi121 bash[36902]: cluster 2022-04-22T20:35:59.778426+0000 mgr.y (mgr.34107) 6496 : cluster [DBG] pgmap v7639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:01 smithi121 bash[36902]: audit 2022-04-22T20:35:59.902455+0000 mgr.y (mgr.34107) 6497 : audit [DBG] from='client.42138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:01 smithi121 bash[36902]: audit 2022-04-22T20:36:00.246468+0000 mgr.y (mgr.34107) 6498 : audit [DBG] from='client.52075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:01 smithi121 bash[36902]: audit 2022-04-22T20:36:00.672176+0000 mon.a (mon.0) 9871 : audit [DBG] from='client.? 172.21.15.115:0/3984151743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:02.346Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[45495]: cluster 2022-04-22T20:36:01.779517+0000 mgr.y (mgr.34107) 6499 : cluster [DBG] pgmap v7640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[45495]: audit 2022-04-22T20:36:02.796998+0000 mon.a (mon.0) 9872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[45495]: audit 2022-04-22T20:36:02.842916+0000 mon.a (mon.0) 9873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[46934]: cluster 2022-04-22T20:36:01.779517+0000 mgr.y (mgr.34107) 6499 : cluster [DBG] pgmap v7640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[46934]: audit 2022-04-22T20:36:02.796998+0000 mon.a (mon.0) 9872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:03 smithi115 bash[46934]: audit 2022-04-22T20:36:02.842916+0000 mon.a (mon.0) 9873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:03 smithi121 bash[36902]: cluster 2022-04-22T20:36:01.779517+0000 mgr.y (mgr.34107) 6499 : cluster [DBG] pgmap v7640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:03 smithi121 bash[36902]: audit 2022-04-22T20:36:02.796998+0000 mon.a (mon.0) 9872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:03 smithi121 bash[36902]: audit 2022-04-22T20:36:02.842916+0000 mon.a (mon.0) 9873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:04.225Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:05 smithi115 bash[46934]: cluster 2022-04-22T20:36:03.780579+0000 mgr.y (mgr.34107) 6500 : cluster [DBG] pgmap v7641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:05 smithi115 bash[46934]: audit 2022-04-22T20:36:04.296181+0000 mon.a (mon.0) 9874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:05 smithi115 bash[45495]: cluster 2022-04-22T20:36:03.780579+0000 mgr.y (mgr.34107) 6500 : cluster [DBG] pgmap v7641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:05 smithi115 bash[45495]: audit 2022-04-22T20:36:04.296181+0000 mon.a (mon.0) 9874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:05 smithi121 bash[36902]: cluster 2022-04-22T20:36:03.780579+0000 mgr.y (mgr.34107) 6500 : cluster [DBG] pgmap v7641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:05 smithi121 bash[36902]: audit 2022-04-22T20:36:04.296181+0000 mon.a (mon.0) 9874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:07 smithi115 bash[45495]: cluster 2022-04-22T20:36:05.781698+0000 mgr.y (mgr.34107) 6501 : cluster [DBG] pgmap v7642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:07 smithi115 bash[46934]: cluster 2022-04-22T20:36:05.781698+0000 mgr.y (mgr.34107) 6501 : cluster [DBG] pgmap v7642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:07 smithi121 bash[36902]: cluster 2022-04-22T20:36:05.781698+0000 mgr.y (mgr.34107) 6501 : cluster [DBG] pgmap v7642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:08 smithi115 bash[45495]: audit 2022-04-22T20:36:07.800656+0000 mon.a (mon.0) 9875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:08 smithi115 bash[45495]: audit 2022-04-22T20:36:07.845804+0000 mon.a (mon.0) 9876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:08 smithi115 bash[46934]: audit 2022-04-22T20:36:07.800656+0000 mon.a (mon.0) 9875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:08 smithi115 bash[46934]: audit 2022-04-22T20:36:07.845804+0000 mon.a (mon.0) 9876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:08 smithi121 bash[36902]: audit 2022-04-22T20:36:07.800656+0000 mon.a (mon.0) 9875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:08 smithi121 bash[36902]: audit 2022-04-22T20:36:07.845804+0000 mon.a (mon.0) 9876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:09 smithi121 bash[36902]: cluster 2022-04-22T20:36:07.782809+0000 mgr.y (mgr.34107) 6502 : cluster [DBG] pgmap v7643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:09 smithi115 bash[45495]: cluster 2022-04-22T20:36:07.782809+0000 mgr.y (mgr.34107) 6502 : cluster [DBG] pgmap v7643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:09.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:09 smithi115 bash[46934]: cluster 2022-04-22T20:36:07.782809+0000 mgr.y (mgr.34107) 6502 : cluster [DBG] pgmap v7643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:10 smithi121 bash[36902]: audit 2022-04-22T20:36:09.298810+0000 mon.a (mon.0) 9877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:10 smithi115 bash[45495]: audit 2022-04-22T20:36:09.298810+0000 mon.a (mon.0) 9877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:10 smithi115 bash[46934]: audit 2022-04-22T20:36:09.298810+0000 mon.a (mon.0) 9877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:11 smithi121 bash[36902]: cluster 2022-04-22T20:36:09.783840+0000 mgr.y (mgr.34107) 6503 : cluster [DBG] pgmap v7644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:11 smithi115 bash[45495]: cluster 2022-04-22T20:36:09.783840+0000 mgr.y (mgr.34107) 6503 : cluster [DBG] pgmap v7644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:11 smithi115 bash[46934]: cluster 2022-04-22T20:36:09.783840+0000 mgr.y (mgr.34107) 6503 : cluster [DBG] pgmap v7644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:12 smithi121 bash[36902]: cluster 2022-04-22T20:36:11.784853+0000 mgr.y (mgr.34107) 6504 : cluster [DBG] pgmap v7645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:12.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:12 smithi115 bash[45495]: cluster 2022-04-22T20:36:11.784853+0000 mgr.y (mgr.34107) 6504 : cluster [DBG] pgmap v7645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:12.346Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:12 smithi115 bash[46934]: cluster 2022-04-22T20:36:11.784853+0000 mgr.y (mgr.34107) 6504 : cluster [DBG] pgmap v7645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:13 smithi121 bash[36902]: audit 2022-04-22T20:36:12.804234+0000 mon.a (mon.0) 9878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:13 smithi121 bash[36902]: audit 2022-04-22T20:36:12.849132+0000 mon.a (mon.0) 9879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:13 smithi115 bash[45495]: audit 2022-04-22T20:36:12.804234+0000 mon.a (mon.0) 9878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:13.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:13 smithi115 bash[45495]: audit 2022-04-22T20:36:12.849132+0000 mon.a (mon.0) 9879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:13 smithi115 bash[46934]: audit 2022-04-22T20:36:12.804234+0000 mon.a (mon.0) 9878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:13 smithi115 bash[46934]: audit 2022-04-22T20:36:12.849132+0000 mon.a (mon.0) 9879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:14.227Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:15 smithi121 bash[36902]: cluster 2022-04-22T20:36:13.786339+0000 mgr.y (mgr.34107) 6505 : cluster [DBG] pgmap v7646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:15 smithi121 bash[36902]: audit 2022-04-22T20:36:14.301123+0000 mon.a (mon.0) 9880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:15 smithi115 bash[45495]: cluster 2022-04-22T20:36:13.786339+0000 mgr.y (mgr.34107) 6505 : cluster [DBG] pgmap v7646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:15.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:15 smithi115 bash[45495]: audit 2022-04-22T20:36:14.301123+0000 mon.a (mon.0) 9880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:15 smithi115 bash[46934]: cluster 2022-04-22T20:36:13.786339+0000 mgr.y (mgr.34107) 6505 : cluster [DBG] pgmap v7646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:15.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:15 smithi115 bash[46934]: audit 2022-04-22T20:36:14.301123+0000 mon.a (mon.0) 9880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:16 smithi121 bash[36902]: audit 2022-04-22T20:36:15.458863+0000 mon.a (mon.0) 9881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:36:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:16 smithi121 bash[36902]: audit 2022-04-22T20:36:15.459636+0000 mon.a (mon.0) 9882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:36:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:16 smithi115 bash[45495]: audit 2022-04-22T20:36:15.458863+0000 mon.a (mon.0) 9881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:36:16.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:16 smithi115 bash[45495]: audit 2022-04-22T20:36:15.459636+0000 mon.a (mon.0) 9882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:36:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:16 smithi115 bash[46934]: audit 2022-04-22T20:36:15.458863+0000 mon.a (mon.0) 9881 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:36:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:16 smithi115 bash[46934]: audit 2022-04-22T20:36:15.459636+0000 mon.a (mon.0) 9882 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:36:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:17 smithi121 bash[36902]: cluster 2022-04-22T20:36:15.787509+0000 mgr.y (mgr.34107) 6506 : cluster [DBG] pgmap v7647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:17 smithi115 bash[45495]: cluster 2022-04-22T20:36:15.787509+0000 mgr.y (mgr.34107) 6506 : cluster [DBG] pgmap v7647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:17 smithi115 bash[46934]: cluster 2022-04-22T20:36:15.787509+0000 mgr.y (mgr.34107) 6506 : cluster [DBG] pgmap v7647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:18 smithi121 bash[36902]: audit 2022-04-22T20:36:17.808093+0000 mon.a (mon.0) 9883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:18 smithi121 bash[36902]: audit 2022-04-22T20:36:17.852252+0000 mon.a (mon.0) 9884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:18 smithi115 bash[45495]: audit 2022-04-22T20:36:17.808093+0000 mon.a (mon.0) 9883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:18 smithi115 bash[45495]: audit 2022-04-22T20:36:17.852252+0000 mon.a (mon.0) 9884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:18 smithi115 bash[46934]: audit 2022-04-22T20:36:17.808093+0000 mon.a (mon.0) 9883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:18 smithi115 bash[46934]: audit 2022-04-22T20:36:17.852252+0000 mon.a (mon.0) 9884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:19 smithi121 bash[36902]: cluster 2022-04-22T20:36:17.788559+0000 mgr.y (mgr.34107) 6507 : cluster [DBG] pgmap v7648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:19 smithi115 bash[45495]: cluster 2022-04-22T20:36:17.788559+0000 mgr.y (mgr.34107) 6507 : cluster [DBG] pgmap v7648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:19 smithi115 bash[46934]: cluster 2022-04-22T20:36:17.788559+0000 mgr.y (mgr.34107) 6507 : cluster [DBG] pgmap v7648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:20 smithi115 bash[45495]: audit 2022-04-22T20:36:19.303847+0000 mon.a (mon.0) 9885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:20.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:20 smithi115 bash[46934]: audit 2022-04-22T20:36:19.303847+0000 mon.a (mon.0) 9885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:20 smithi121 bash[36902]: audit 2022-04-22T20:36:19.303847+0000 mon.a (mon.0) 9885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:21 smithi115 bash[45495]: cluster 2022-04-22T20:36:19.789679+0000 mgr.y (mgr.34107) 6508 : cluster [DBG] pgmap v7649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:21.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:21 smithi115 bash[46934]: cluster 2022-04-22T20:36:19.789679+0000 mgr.y (mgr.34107) 6508 : cluster [DBG] pgmap v7649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:21 smithi121 bash[36902]: cluster 2022-04-22T20:36:19.789679+0000 mgr.y (mgr.34107) 6508 : cluster [DBG] pgmap v7649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:22.346Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[45495]: cluster 2022-04-22T20:36:21.790829+0000 mgr.y (mgr.34107) 6509 : cluster [DBG] pgmap v7650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[45495]: audit 2022-04-22T20:36:22.811962+0000 mon.a (mon.0) 9886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:23.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[45495]: audit 2022-04-22T20:36:22.855288+0000 mon.a (mon.0) 9887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[46934]: cluster 2022-04-22T20:36:21.790829+0000 mgr.y (mgr.34107) 6509 : cluster [DBG] pgmap v7650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[46934]: audit 2022-04-22T20:36:22.811962+0000 mon.a (mon.0) 9886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:23 smithi115 bash[46934]: audit 2022-04-22T20:36:22.855288+0000 mon.a (mon.0) 9887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:23 smithi121 bash[36902]: cluster 2022-04-22T20:36:21.790829+0000 mgr.y (mgr.34107) 6509 : cluster [DBG] pgmap v7650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:23 smithi121 bash[36902]: audit 2022-04-22T20:36:22.811962+0000 mon.a (mon.0) 9886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:23 smithi121 bash[36902]: audit 2022-04-22T20:36:22.855288+0000 mon.a (mon.0) 9887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:24.228Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:25 smithi115 bash[45495]: cluster 2022-04-22T20:36:23.792029+0000 mgr.y (mgr.34107) 6510 : cluster [DBG] pgmap v7651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:25 smithi115 bash[45495]: audit 2022-04-22T20:36:24.306740+0000 mon.a (mon.0) 9888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:25 smithi115 bash[46934]: cluster 2022-04-22T20:36:23.792029+0000 mgr.y (mgr.34107) 6510 : cluster [DBG] pgmap v7651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:25 smithi115 bash[46934]: audit 2022-04-22T20:36:24.306740+0000 mon.a (mon.0) 9888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:25 smithi121 bash[36902]: cluster 2022-04-22T20:36:23.792029+0000 mgr.y (mgr.34107) 6510 : cluster [DBG] pgmap v7651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:25 smithi121 bash[36902]: audit 2022-04-22T20:36:24.306740+0000 mon.a (mon.0) 9888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:27 smithi115 bash[45495]: cluster 2022-04-22T20:36:25.793174+0000 mgr.y (mgr.34107) 6511 : cluster [DBG] pgmap v7652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:27 smithi115 bash[46934]: cluster 2022-04-22T20:36:25.793174+0000 mgr.y (mgr.34107) 6511 : cluster [DBG] pgmap v7652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:27 smithi121 bash[36902]: cluster 2022-04-22T20:36:25.793174+0000 mgr.y (mgr.34107) 6511 : cluster [DBG] pgmap v7652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:28 smithi121 bash[36902]: audit 2022-04-22T20:36:27.815745+0000 mon.a (mon.0) 9889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:28 smithi121 bash[36902]: audit 2022-04-22T20:36:27.858244+0000 mon.a (mon.0) 9890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:28 smithi115 bash[45495]: audit 2022-04-22T20:36:27.815745+0000 mon.a (mon.0) 9889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:28 smithi115 bash[45495]: audit 2022-04-22T20:36:27.858244+0000 mon.a (mon.0) 9890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:28 smithi115 bash[46934]: audit 2022-04-22T20:36:27.815745+0000 mon.a (mon.0) 9889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:28 smithi115 bash[46934]: audit 2022-04-22T20:36:27.858244+0000 mon.a (mon.0) 9890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:29 smithi121 bash[36902]: cluster 2022-04-22T20:36:27.794330+0000 mgr.y (mgr.34107) 6512 : cluster [DBG] pgmap v7653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:29 smithi115 bash[45495]: cluster 2022-04-22T20:36:27.794330+0000 mgr.y (mgr.34107) 6512 : cluster [DBG] pgmap v7653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:29 smithi115 bash[46934]: cluster 2022-04-22T20:36:27.794330+0000 mgr.y (mgr.34107) 6512 : cluster [DBG] pgmap v7653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:30 smithi121 bash[36902]: audit 2022-04-22T20:36:29.309736+0000 mon.a (mon.0) 9891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:30 smithi115 bash[45495]: audit 2022-04-22T20:36:29.309736+0000 mon.a (mon.0) 9891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:30 smithi115 bash[46934]: audit 2022-04-22T20:36:29.309736+0000 mon.a (mon.0) 9891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:31.054 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:36:31.393 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:36:31.393 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2h) 3m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:36:31.394 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:36:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:36:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:31 smithi121 bash[36902]: cluster 2022-04-22T20:36:29.795490+0000 mgr.y (mgr.34107) 6513 : cluster [DBG] pgmap v7654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:31 smithi115 bash[46934]: cluster 2022-04-22T20:36:29.795490+0000 mgr.y (mgr.34107) 6513 : cluster [DBG] pgmap v7654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:31 smithi115 bash[45495]: cluster 2022-04-22T20:36:29.795490+0000 mgr.y (mgr.34107) 6513 : cluster [DBG] pgmap v7654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:36:31.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:36:31.804 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:36:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:32 smithi121 bash[36902]: audit 2022-04-22T20:36:31.031734+0000 mgr.y (mgr.34107) 6514 : audit [DBG] from='client.42156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:32 smithi121 bash[36902]: audit 2022-04-22T20:36:31.372809+0000 mgr.y (mgr.34107) 6515 : audit [DBG] from='client.52093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:32 smithi121 bash[36902]: audit 2022-04-22T20:36:31.795755+0000 mon.c (mon.1) 253 : audit [DBG] from='client.? 172.21.15.115:0/277618700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:32.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:32 smithi121 bash[36902]: cluster 2022-04-22T20:36:31.796480+0000 mgr.y (mgr.34107) 6516 : cluster [DBG] pgmap v7655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:32.347Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[45495]: audit 2022-04-22T20:36:31.031734+0000 mgr.y (mgr.34107) 6514 : audit [DBG] from='client.42156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[45495]: audit 2022-04-22T20:36:31.372809+0000 mgr.y (mgr.34107) 6515 : audit [DBG] from='client.52093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[45495]: audit 2022-04-22T20:36:31.795755+0000 mon.c (mon.1) 253 : audit [DBG] from='client.? 172.21.15.115:0/277618700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[45495]: cluster 2022-04-22T20:36:31.796480+0000 mgr.y (mgr.34107) 6516 : cluster [DBG] pgmap v7655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[46934]: audit 2022-04-22T20:36:31.031734+0000 mgr.y (mgr.34107) 6514 : audit [DBG] from='client.42156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[46934]: audit 2022-04-22T20:36:31.372809+0000 mgr.y (mgr.34107) 6515 : audit [DBG] from='client.52093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[46934]: audit 2022-04-22T20:36:31.795755+0000 mon.c (mon.1) 253 : audit [DBG] from='client.? 172.21.15.115:0/277618700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:36:32.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:32 smithi115 bash[46934]: cluster 2022-04-22T20:36:31.796480+0000 mgr.y (mgr.34107) 6516 : cluster [DBG] pgmap v7655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:33 smithi121 bash[36902]: audit 2022-04-22T20:36:32.819524+0000 mon.a (mon.0) 9892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:33 smithi121 bash[36902]: audit 2022-04-22T20:36:32.861109+0000 mon.a (mon.0) 9893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:33 smithi115 bash[45495]: audit 2022-04-22T20:36:32.819524+0000 mon.a (mon.0) 9892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:33 smithi115 bash[45495]: audit 2022-04-22T20:36:32.861109+0000 mon.a (mon.0) 9893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:33 smithi115 bash[46934]: audit 2022-04-22T20:36:32.819524+0000 mon.a (mon.0) 9892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:33 smithi115 bash[46934]: audit 2022-04-22T20:36:32.861109+0000 mon.a (mon.0) 9893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:34.228Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:35 smithi121 bash[36902]: cluster 2022-04-22T20:36:33.797619+0000 mgr.y (mgr.34107) 6517 : cluster [DBG] pgmap v7656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:35 smithi121 bash[36902]: audit 2022-04-22T20:36:34.312570+0000 mon.a (mon.0) 9894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:35 smithi115 bash[46934]: cluster 2022-04-22T20:36:33.797619+0000 mgr.y (mgr.34107) 6517 : cluster [DBG] pgmap v7656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:35 smithi115 bash[46934]: audit 2022-04-22T20:36:34.312570+0000 mon.a (mon.0) 9894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:35 smithi115 bash[45495]: cluster 2022-04-22T20:36:33.797619+0000 mgr.y (mgr.34107) 6517 : cluster [DBG] pgmap v7656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:35.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:35 smithi115 bash[45495]: audit 2022-04-22T20:36:34.312570+0000 mon.a (mon.0) 9894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:37 smithi121 bash[36902]: cluster 2022-04-22T20:36:35.798679+0000 mgr.y (mgr.34107) 6518 : cluster [DBG] pgmap v7657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:37.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:37 smithi115 bash[45495]: cluster 2022-04-22T20:36:35.798679+0000 mgr.y (mgr.34107) 6518 : cluster [DBG] pgmap v7657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:37.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:37 smithi115 bash[46934]: cluster 2022-04-22T20:36:35.798679+0000 mgr.y (mgr.34107) 6518 : cluster [DBG] pgmap v7657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:38 smithi121 bash[36902]: audit 2022-04-22T20:36:37.823179+0000 mon.a (mon.0) 9895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:38 smithi121 bash[36902]: audit 2022-04-22T20:36:37.864023+0000 mon.a (mon.0) 9896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:38 smithi115 bash[46934]: audit 2022-04-22T20:36:37.823179+0000 mon.a (mon.0) 9895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:38.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:38 smithi115 bash[46934]: audit 2022-04-22T20:36:37.864023+0000 mon.a (mon.0) 9896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:38 smithi115 bash[45495]: audit 2022-04-22T20:36:37.823179+0000 mon.a (mon.0) 9895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:38.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:38 smithi115 bash[45495]: audit 2022-04-22T20:36:37.864023+0000 mon.a (mon.0) 9896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:39 smithi121 bash[36902]: cluster 2022-04-22T20:36:37.799865+0000 mgr.y (mgr.34107) 6519 : cluster [DBG] pgmap v7658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:39 smithi115 bash[45495]: cluster 2022-04-22T20:36:37.799865+0000 mgr.y (mgr.34107) 6519 : cluster [DBG] pgmap v7658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:39 smithi115 bash[46934]: cluster 2022-04-22T20:36:37.799865+0000 mgr.y (mgr.34107) 6519 : cluster [DBG] pgmap v7658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:40 smithi115 bash[46934]: audit 2022-04-22T20:36:39.315562+0000 mon.a (mon.0) 9897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:40.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:40 smithi115 bash[45495]: audit 2022-04-22T20:36:39.315562+0000 mon.a (mon.0) 9897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:40 smithi121 bash[36902]: audit 2022-04-22T20:36:39.315562+0000 mon.a (mon.0) 9897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:41 smithi115 bash[45495]: cluster 2022-04-22T20:36:39.800896+0000 mgr.y (mgr.34107) 6520 : cluster [DBG] pgmap v7659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:41 smithi115 bash[46934]: cluster 2022-04-22T20:36:39.800896+0000 mgr.y (mgr.34107) 6520 : cluster [DBG] pgmap v7659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:41 smithi121 bash[36902]: cluster 2022-04-22T20:36:39.800896+0000 mgr.y (mgr.34107) 6520 : cluster [DBG] pgmap v7659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:42.348Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[45495]: cluster 2022-04-22T20:36:41.801964+0000 mgr.y (mgr.34107) 6521 : cluster [DBG] pgmap v7660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[45495]: audit 2022-04-22T20:36:42.826888+0000 mon.a (mon.0) 9898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[45495]: audit 2022-04-22T20:36:42.866990+0000 mon.a (mon.0) 9899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[46934]: cluster 2022-04-22T20:36:41.801964+0000 mgr.y (mgr.34107) 6521 : cluster [DBG] pgmap v7660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[46934]: audit 2022-04-22T20:36:42.826888+0000 mon.a (mon.0) 9898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:43 smithi115 bash[46934]: audit 2022-04-22T20:36:42.866990+0000 mon.a (mon.0) 9899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:43 smithi121 bash[36902]: cluster 2022-04-22T20:36:41.801964+0000 mgr.y (mgr.34107) 6521 : cluster [DBG] pgmap v7660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:43 smithi121 bash[36902]: audit 2022-04-22T20:36:42.826888+0000 mon.a (mon.0) 9898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:43 smithi121 bash[36902]: audit 2022-04-22T20:36:42.866990+0000 mon.a (mon.0) 9899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:44.228Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:45 smithi115 bash[45495]: cluster 2022-04-22T20:36:43.803140+0000 mgr.y (mgr.34107) 6522 : cluster [DBG] pgmap v7661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:45.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:45 smithi115 bash[45495]: audit 2022-04-22T20:36:44.318544+0000 mon.a (mon.0) 9900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:45 smithi115 bash[46934]: cluster 2022-04-22T20:36:43.803140+0000 mgr.y (mgr.34107) 6522 : cluster [DBG] pgmap v7661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:45.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:45 smithi115 bash[46934]: audit 2022-04-22T20:36:44.318544+0000 mon.a (mon.0) 9900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:45 smithi121 bash[36902]: cluster 2022-04-22T20:36:43.803140+0000 mgr.y (mgr.34107) 6522 : cluster [DBG] pgmap v7661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:45 smithi121 bash[36902]: audit 2022-04-22T20:36:44.318544+0000 mon.a (mon.0) 9900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:47 smithi115 bash[46934]: cluster 2022-04-22T20:36:45.804263+0000 mgr.y (mgr.34107) 6523 : cluster [DBG] pgmap v7662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:47 smithi115 bash[45495]: cluster 2022-04-22T20:36:45.804263+0000 mgr.y (mgr.34107) 6523 : cluster [DBG] pgmap v7662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:47 smithi121 bash[36902]: cluster 2022-04-22T20:36:45.804263+0000 mgr.y (mgr.34107) 6523 : cluster [DBG] pgmap v7662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.441218+0000 mon.a (mon.0) 9901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:36:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.444267+0000 mon.a (mon.0) 9902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.446181+0000 mon.a (mon.0) 9903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.809280+0000 mon.a (mon.0) 9904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.830464+0000 mon.a (mon.0) 9905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[45495]: audit 2022-04-22T20:36:47.869977+0000 mon.a (mon.0) 9906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.441218+0000 mon.a (mon.0) 9901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.444267+0000 mon.a (mon.0) 9902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:36:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.446181+0000 mon.a (mon.0) 9903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:36:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.809280+0000 mon.a (mon.0) 9904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:36:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.830464+0000 mon.a (mon.0) 9905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:48 smithi115 bash[46934]: audit 2022-04-22T20:36:47.869977+0000 mon.a (mon.0) 9906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.441218+0000 mon.a (mon.0) 9901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:36:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.444267+0000 mon.a (mon.0) 9902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:36:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.446181+0000 mon.a (mon.0) 9903 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:36:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.809280+0000 mon.a (mon.0) 9904 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:36:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.830464+0000 mon.a (mon.0) 9905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:48 smithi121 bash[36902]: audit 2022-04-22T20:36:47.869977+0000 mon.a (mon.0) 9906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:49 smithi115 bash[45495]: cluster 2022-04-22T20:36:47.805239+0000 mgr.y (mgr.34107) 6524 : cluster [DBG] pgmap v7663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:49 smithi115 bash[46934]: cluster 2022-04-22T20:36:47.805239+0000 mgr.y (mgr.34107) 6524 : cluster [DBG] pgmap v7663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:49 smithi121 bash[36902]: cluster 2022-04-22T20:36:47.805239+0000 mgr.y (mgr.34107) 6524 : cluster [DBG] pgmap v7663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:50 smithi121 bash[36902]: audit 2022-04-22T20:36:49.321466+0000 mon.a (mon.0) 9907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:50 smithi115 bash[45495]: audit 2022-04-22T20:36:49.321466+0000 mon.a (mon.0) 9907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:50.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:50 smithi115 bash[46934]: audit 2022-04-22T20:36:49.321466+0000 mon.a (mon.0) 9907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:51 smithi121 bash[36902]: cluster 2022-04-22T20:36:49.806295+0000 mgr.y (mgr.34107) 6525 : cluster [DBG] pgmap v7664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:51 smithi115 bash[45495]: cluster 2022-04-22T20:36:49.806295+0000 mgr.y (mgr.34107) 6525 : cluster [DBG] pgmap v7664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:51 smithi115 bash[46934]: cluster 2022-04-22T20:36:49.806295+0000 mgr.y (mgr.34107) 6525 : cluster [DBG] pgmap v7664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:52.349Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:53 smithi121 bash[36902]: cluster 2022-04-22T20:36:51.807425+0000 mgr.y (mgr.34107) 6526 : cluster [DBG] pgmap v7665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:53 smithi121 bash[36902]: audit 2022-04-22T20:36:52.833680+0000 mon.a (mon.0) 9908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:53 smithi121 bash[36902]: audit 2022-04-22T20:36:52.872730+0000 mon.a (mon.0) 9909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[45495]: cluster 2022-04-22T20:36:51.807425+0000 mgr.y (mgr.34107) 6526 : cluster [DBG] pgmap v7665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[45495]: audit 2022-04-22T20:36:52.833680+0000 mon.a (mon.0) 9908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[45495]: audit 2022-04-22T20:36:52.872730+0000 mon.a (mon.0) 9909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[46934]: cluster 2022-04-22T20:36:51.807425+0000 mgr.y (mgr.34107) 6526 : cluster [DBG] pgmap v7665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[46934]: audit 2022-04-22T20:36:52.833680+0000 mon.a (mon.0) 9908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:53 smithi115 bash[46934]: audit 2022-04-22T20:36:52.872730+0000 mon.a (mon.0) 9909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:36:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:36:54.230Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:36:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:55 smithi121 bash[36902]: cluster 2022-04-22T20:36:53.808505+0000 mgr.y (mgr.34107) 6527 : cluster [DBG] pgmap v7666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:55 smithi121 bash[36902]: audit 2022-04-22T20:36:54.324790+0000 mon.a (mon.0) 9910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:55 smithi115 bash[45495]: cluster 2022-04-22T20:36:53.808505+0000 mgr.y (mgr.34107) 6527 : cluster [DBG] pgmap v7666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:55 smithi115 bash[45495]: audit 2022-04-22T20:36:54.324790+0000 mon.a (mon.0) 9910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:55 smithi115 bash[46934]: cluster 2022-04-22T20:36:53.808505+0000 mgr.y (mgr.34107) 6527 : cluster [DBG] pgmap v7666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:55 smithi115 bash[46934]: audit 2022-04-22T20:36:54.324790+0000 mon.a (mon.0) 9910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:36:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:56 smithi121 bash[36902]: cluster 2022-04-22T20:36:55.809625+0000 mgr.y (mgr.34107) 6528 : cluster [DBG] pgmap v7667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:56 smithi115 bash[45495]: cluster 2022-04-22T20:36:55.809625+0000 mgr.y (mgr.34107) 6528 : cluster [DBG] pgmap v7667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:56.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:56 smithi115 bash[46934]: cluster 2022-04-22T20:36:55.809625+0000 mgr.y (mgr.34107) 6528 : cluster [DBG] pgmap v7667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:57 smithi121 bash[36902]: audit 2022-04-22T20:36:57.837643+0000 mon.a (mon.0) 9911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:57 smithi121 bash[36902]: audit 2022-04-22T20:36:57.875753+0000 mon.a (mon.0) 9912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:57 smithi115 bash[45495]: audit 2022-04-22T20:36:57.837643+0000 mon.a (mon.0) 9911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:58.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:57 smithi115 bash[45495]: audit 2022-04-22T20:36:57.875753+0000 mon.a (mon.0) 9912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:58.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:57 smithi115 bash[46934]: audit 2022-04-22T20:36:57.837643+0000 mon.a (mon.0) 9911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:36:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:57 smithi115 bash[46934]: audit 2022-04-22T20:36:57.875753+0000 mon.a (mon.0) 9912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:36:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:36:58 smithi121 bash[36902]: cluster 2022-04-22T20:36:57.810730+0000 mgr.y (mgr.34107) 6529 : cluster [DBG] pgmap v7668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:59.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:36:58 smithi115 bash[45495]: cluster 2022-04-22T20:36:57.810730+0000 mgr.y (mgr.34107) 6529 : cluster [DBG] pgmap v7668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:36:59.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:36:58 smithi115 bash[46934]: cluster 2022-04-22T20:36:57.810730+0000 mgr.y (mgr.34107) 6529 : cluster [DBG] pgmap v7668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:00 smithi121 bash[36902]: audit 2022-04-22T20:36:59.327636+0000 mon.a (mon.0) 9913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:00.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:00 smithi115 bash[45495]: audit 2022-04-22T20:36:59.327636+0000 mon.a (mon.0) 9913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:00.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:00 smithi115 bash[46934]: audit 2022-04-22T20:36:59.327636+0000 mon.a (mon.0) 9913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:01 smithi121 bash[36902]: cluster 2022-04-22T20:36:59.811872+0000 mgr.y (mgr.34107) 6530 : cluster [DBG] pgmap v7669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:01.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:01 smithi115 bash[45495]: cluster 2022-04-22T20:36:59.811872+0000 mgr.y (mgr.34107) 6530 : cluster [DBG] pgmap v7669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:01 smithi115 bash[46934]: cluster 2022-04-22T20:36:59.811872+0000 mgr.y (mgr.34107) 6530 : cluster [DBG] pgmap v7669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:02.180 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:37:02.517 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:37:02.517 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:37:02.518 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:37:02.519 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:37:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:02.346Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:37:02.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:37:02.918 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:37:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[45495]: cluster 2022-04-22T20:37:01.812972+0000 mgr.y (mgr.34107) 6531 : cluster [DBG] pgmap v7670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[45495]: audit 2022-04-22T20:37:02.152769+0000 mgr.y (mgr.34107) 6532 : audit [DBG] from='client.52105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[45495]: audit 2022-04-22T20:37:02.841289+0000 mon.a (mon.0) 9914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[45495]: audit 2022-04-22T20:37:02.878565+0000 mon.a (mon.0) 9915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:03.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[45495]: audit 2022-04-22T20:37:02.909873+0000 mon.a (mon.0) 9916 : audit [DBG] from='client.? 172.21.15.115:0/2510656185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[46934]: cluster 2022-04-22T20:37:01.812972+0000 mgr.y (mgr.34107) 6531 : cluster [DBG] pgmap v7670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[46934]: audit 2022-04-22T20:37:02.152769+0000 mgr.y (mgr.34107) 6532 : audit [DBG] from='client.52105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[46934]: audit 2022-04-22T20:37:02.841289+0000 mon.a (mon.0) 9914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:03.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[46934]: audit 2022-04-22T20:37:02.878565+0000 mon.a (mon.0) 9915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:03 smithi115 bash[46934]: audit 2022-04-22T20:37:02.909873+0000 mon.a (mon.0) 9916 : audit [DBG] from='client.? 172.21.15.115:0/2510656185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:03 smithi121 bash[36902]: cluster 2022-04-22T20:37:01.812972+0000 mgr.y (mgr.34107) 6531 : cluster [DBG] pgmap v7670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:03 smithi121 bash[36902]: audit 2022-04-22T20:37:02.152769+0000 mgr.y (mgr.34107) 6532 : audit [DBG] from='client.52105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:03 smithi121 bash[36902]: audit 2022-04-22T20:37:02.841289+0000 mon.a (mon.0) 9914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:03 smithi121 bash[36902]: audit 2022-04-22T20:37:02.878565+0000 mon.a (mon.0) 9915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:03 smithi121 bash[36902]: audit 2022-04-22T20:37:02.909873+0000 mon.a (mon.0) 9916 : audit [DBG] from='client.? 172.21.15.115:0/2510656185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:04 smithi115 bash[45495]: audit 2022-04-22T20:37:02.496376+0000 mgr.y (mgr.34107) 6533 : audit [DBG] from='client.52111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:04.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:04.229Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:04 smithi115 bash[46934]: audit 2022-04-22T20:37:02.496376+0000 mgr.y (mgr.34107) 6533 : audit [DBG] from='client.52111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:04 smithi121 bash[36902]: audit 2022-04-22T20:37:02.496376+0000 mgr.y (mgr.34107) 6533 : audit [DBG] from='client.52111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:05 smithi115 bash[45495]: cluster 2022-04-22T20:37:03.814102+0000 mgr.y (mgr.34107) 6534 : cluster [DBG] pgmap v7671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:05.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:05 smithi115 bash[45495]: audit 2022-04-22T20:37:04.330623+0000 mon.a (mon.0) 9917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:05 smithi115 bash[46934]: cluster 2022-04-22T20:37:03.814102+0000 mgr.y (mgr.34107) 6534 : cluster [DBG] pgmap v7671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:05.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:05 smithi115 bash[46934]: audit 2022-04-22T20:37:04.330623+0000 mon.a (mon.0) 9917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:05 smithi121 bash[36902]: cluster 2022-04-22T20:37:03.814102+0000 mgr.y (mgr.34107) 6534 : cluster [DBG] pgmap v7671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:05 smithi121 bash[36902]: audit 2022-04-22T20:37:04.330623+0000 mon.a (mon.0) 9917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:07 smithi115 bash[45495]: cluster 2022-04-22T20:37:05.815276+0000 mgr.y (mgr.34107) 6535 : cluster [DBG] pgmap v7672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:07.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:07 smithi115 bash[46934]: cluster 2022-04-22T20:37:05.815276+0000 mgr.y (mgr.34107) 6535 : cluster [DBG] pgmap v7672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:07 smithi121 bash[36902]: cluster 2022-04-22T20:37:05.815276+0000 mgr.y (mgr.34107) 6535 : cluster [DBG] pgmap v7672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:08 smithi115 bash[45495]: audit 2022-04-22T20:37:07.844984+0000 mon.a (mon.0) 9918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:08.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:08 smithi115 bash[45495]: audit 2022-04-22T20:37:07.881335+0000 mon.a (mon.0) 9919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:08 smithi115 bash[46934]: audit 2022-04-22T20:37:07.844984+0000 mon.a (mon.0) 9918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:08.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:08 smithi115 bash[46934]: audit 2022-04-22T20:37:07.881335+0000 mon.a (mon.0) 9919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:08 smithi121 bash[36902]: audit 2022-04-22T20:37:07.844984+0000 mon.a (mon.0) 9918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:08 smithi121 bash[36902]: audit 2022-04-22T20:37:07.881335+0000 mon.a (mon.0) 9919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:09 smithi115 bash[45495]: cluster 2022-04-22T20:37:07.816355+0000 mgr.y (mgr.34107) 6536 : cluster [DBG] pgmap v7673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:09 smithi115 bash[46934]: cluster 2022-04-22T20:37:07.816355+0000 mgr.y (mgr.34107) 6536 : cluster [DBG] pgmap v7673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:09 smithi121 bash[36902]: cluster 2022-04-22T20:37:07.816355+0000 mgr.y (mgr.34107) 6536 : cluster [DBG] pgmap v7673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:10 smithi121 bash[36902]: audit 2022-04-22T20:37:09.333712+0000 mon.a (mon.0) 9920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:10 smithi115 bash[45495]: audit 2022-04-22T20:37:09.333712+0000 mon.a (mon.0) 9920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:10 smithi115 bash[46934]: audit 2022-04-22T20:37:09.333712+0000 mon.a (mon.0) 9920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:11 smithi121 bash[36902]: cluster 2022-04-22T20:37:09.817432+0000 mgr.y (mgr.34107) 6537 : cluster [DBG] pgmap v7674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:11 smithi115 bash[45495]: cluster 2022-04-22T20:37:09.817432+0000 mgr.y (mgr.34107) 6537 : cluster [DBG] pgmap v7674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:11 smithi115 bash[46934]: cluster 2022-04-22T20:37:09.817432+0000 mgr.y (mgr.34107) 6537 : cluster [DBG] pgmap v7674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:12.351Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:13 smithi121 bash[36902]: cluster 2022-04-22T20:37:11.818470+0000 mgr.y (mgr.34107) 6538 : cluster [DBG] pgmap v7675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:13 smithi121 bash[36902]: audit 2022-04-22T20:37:12.848718+0000 mon.a (mon.0) 9921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:13 smithi121 bash[36902]: audit 2022-04-22T20:37:12.883906+0000 mon.a (mon.0) 9922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[45495]: cluster 2022-04-22T20:37:11.818470+0000 mgr.y (mgr.34107) 6538 : cluster [DBG] pgmap v7675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[45495]: audit 2022-04-22T20:37:12.848718+0000 mon.a (mon.0) 9921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[45495]: audit 2022-04-22T20:37:12.883906+0000 mon.a (mon.0) 9922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[46934]: cluster 2022-04-22T20:37:11.818470+0000 mgr.y (mgr.34107) 6538 : cluster [DBG] pgmap v7675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[46934]: audit 2022-04-22T20:37:12.848718+0000 mon.a (mon.0) 9921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:13 smithi115 bash[46934]: audit 2022-04-22T20:37:12.883906+0000 mon.a (mon.0) 9922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:14.232Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:15 smithi121 bash[36902]: cluster 2022-04-22T20:37:13.819625+0000 mgr.y (mgr.34107) 6539 : cluster [DBG] pgmap v7676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:15 smithi121 bash[36902]: audit 2022-04-22T20:37:14.336599+0000 mon.a (mon.0) 9923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:15 smithi115 bash[45495]: cluster 2022-04-22T20:37:13.819625+0000 mgr.y (mgr.34107) 6539 : cluster [DBG] pgmap v7676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:15 smithi115 bash[45495]: audit 2022-04-22T20:37:14.336599+0000 mon.a (mon.0) 9923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:15 smithi115 bash[46934]: cluster 2022-04-22T20:37:13.819625+0000 mgr.y (mgr.34107) 6539 : cluster [DBG] pgmap v7676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:15 smithi115 bash[46934]: audit 2022-04-22T20:37:14.336599+0000 mon.a (mon.0) 9923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:16 smithi121 bash[36902]: audit 2022-04-22T20:37:15.468644+0000 mon.a (mon.0) 9924 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:37:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:16 smithi121 bash[36902]: audit 2022-04-22T20:37:15.469139+0000 mon.a (mon.0) 9925 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:37:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:16 smithi115 bash[45495]: audit 2022-04-22T20:37:15.468644+0000 mon.a (mon.0) 9924 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:37:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:16 smithi115 bash[45495]: audit 2022-04-22T20:37:15.469139+0000 mon.a (mon.0) 9925 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:37:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:16 smithi115 bash[46934]: audit 2022-04-22T20:37:15.468644+0000 mon.a (mon.0) 9924 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:37:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:16 smithi115 bash[46934]: audit 2022-04-22T20:37:15.469139+0000 mon.a (mon.0) 9925 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:37:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:17 smithi121 bash[36902]: cluster 2022-04-22T20:37:15.820786+0000 mgr.y (mgr.34107) 6540 : cluster [DBG] pgmap v7677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:17 smithi115 bash[46934]: cluster 2022-04-22T20:37:15.820786+0000 mgr.y (mgr.34107) 6540 : cluster [DBG] pgmap v7677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:17 smithi115 bash[45495]: cluster 2022-04-22T20:37:15.820786+0000 mgr.y (mgr.34107) 6540 : cluster [DBG] pgmap v7677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:18 smithi121 bash[36902]: cluster 2022-04-22T20:37:17.821924+0000 mgr.y (mgr.34107) 6541 : cluster [DBG] pgmap v7678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:18.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:18 smithi121 bash[36902]: audit 2022-04-22T20:37:17.852209+0000 mon.a (mon.0) 9926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:18.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:18 smithi121 bash[36902]: audit 2022-04-22T20:37:17.887050+0000 mon.a (mon.0) 9927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[45495]: cluster 2022-04-22T20:37:17.821924+0000 mgr.y (mgr.34107) 6541 : cluster [DBG] pgmap v7678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[45495]: audit 2022-04-22T20:37:17.852209+0000 mon.a (mon.0) 9926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[45495]: audit 2022-04-22T20:37:17.887050+0000 mon.a (mon.0) 9927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[46934]: cluster 2022-04-22T20:37:17.821924+0000 mgr.y (mgr.34107) 6541 : cluster [DBG] pgmap v7678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[46934]: audit 2022-04-22T20:37:17.852209+0000 mon.a (mon.0) 9926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:18 smithi115 bash[46934]: audit 2022-04-22T20:37:17.887050+0000 mon.a (mon.0) 9927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:20.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:20 smithi115 bash[45495]: audit 2022-04-22T20:37:19.339353+0000 mon.a (mon.0) 9928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:20.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:20 smithi115 bash[46934]: audit 2022-04-22T20:37:19.339353+0000 mon.a (mon.0) 9928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:20 smithi121 bash[36902]: audit 2022-04-22T20:37:19.339353+0000 mon.a (mon.0) 9928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:21.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:21 smithi115 bash[45495]: cluster 2022-04-22T20:37:19.823007+0000 mgr.y (mgr.34107) 6542 : cluster [DBG] pgmap v7679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:21.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:21 smithi115 bash[46934]: cluster 2022-04-22T20:37:19.823007+0000 mgr.y (mgr.34107) 6542 : cluster [DBG] pgmap v7679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:21 smithi121 bash[36902]: cluster 2022-04-22T20:37:19.823007+0000 mgr.y (mgr.34107) 6542 : cluster [DBG] pgmap v7679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:22.352Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[45495]: cluster 2022-04-22T20:37:21.824085+0000 mgr.y (mgr.34107) 6543 : cluster [DBG] pgmap v7680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[45495]: audit 2022-04-22T20:37:22.855936+0000 mon.a (mon.0) 9929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[45495]: audit 2022-04-22T20:37:22.890253+0000 mon.a (mon.0) 9930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[46934]: cluster 2022-04-22T20:37:21.824085+0000 mgr.y (mgr.34107) 6543 : cluster [DBG] pgmap v7680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[46934]: audit 2022-04-22T20:37:22.855936+0000 mon.a (mon.0) 9929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:23.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:23 smithi115 bash[46934]: audit 2022-04-22T20:37:22.890253+0000 mon.a (mon.0) 9930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:23 smithi121 bash[36902]: cluster 2022-04-22T20:37:21.824085+0000 mgr.y (mgr.34107) 6543 : cluster [DBG] pgmap v7680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:23 smithi121 bash[36902]: audit 2022-04-22T20:37:22.855936+0000 mon.a (mon.0) 9929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:23 smithi121 bash[36902]: audit 2022-04-22T20:37:22.890253+0000 mon.a (mon.0) 9930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:24.231Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:25 smithi115 bash[45495]: cluster 2022-04-22T20:37:23.825218+0000 mgr.y (mgr.34107) 6544 : cluster [DBG] pgmap v7681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:25.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:25 smithi115 bash[45495]: audit 2022-04-22T20:37:24.342268+0000 mon.a (mon.0) 9931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:25 smithi115 bash[46934]: cluster 2022-04-22T20:37:23.825218+0000 mgr.y (mgr.34107) 6544 : cluster [DBG] pgmap v7681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:25 smithi115 bash[46934]: audit 2022-04-22T20:37:24.342268+0000 mon.a (mon.0) 9931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:25 smithi121 bash[36902]: cluster 2022-04-22T20:37:23.825218+0000 mgr.y (mgr.34107) 6544 : cluster [DBG] pgmap v7681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:25 smithi121 bash[36902]: audit 2022-04-22T20:37:24.342268+0000 mon.a (mon.0) 9931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:27 smithi115 bash[45495]: cluster 2022-04-22T20:37:25.826347+0000 mgr.y (mgr.34107) 6545 : cluster [DBG] pgmap v7682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:27.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:27 smithi115 bash[46934]: cluster 2022-04-22T20:37:25.826347+0000 mgr.y (mgr.34107) 6545 : cluster [DBG] pgmap v7682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:27 smithi121 bash[36902]: cluster 2022-04-22T20:37:25.826347+0000 mgr.y (mgr.34107) 6545 : cluster [DBG] pgmap v7682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:28.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:28 smithi115 bash[45495]: audit 2022-04-22T20:37:27.859808+0000 mon.a (mon.0) 9932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:28.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:28 smithi115 bash[45495]: audit 2022-04-22T20:37:27.893120+0000 mon.a (mon.0) 9933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:28 smithi115 bash[46934]: audit 2022-04-22T20:37:27.859808+0000 mon.a (mon.0) 9932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:28.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:28 smithi115 bash[46934]: audit 2022-04-22T20:37:27.893120+0000 mon.a (mon.0) 9933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:28 smithi121 bash[36902]: audit 2022-04-22T20:37:27.859808+0000 mon.a (mon.0) 9932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:28 smithi121 bash[36902]: audit 2022-04-22T20:37:27.893120+0000 mon.a (mon.0) 9933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:29 smithi115 bash[45495]: cluster 2022-04-22T20:37:27.827494+0000 mgr.y (mgr.34107) 6546 : cluster [DBG] pgmap v7683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:29 smithi115 bash[46934]: cluster 2022-04-22T20:37:27.827494+0000 mgr.y (mgr.34107) 6546 : cluster [DBG] pgmap v7683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:29 smithi121 bash[36902]: cluster 2022-04-22T20:37:27.827494+0000 mgr.y (mgr.34107) 6546 : cluster [DBG] pgmap v7683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:30 smithi121 bash[36902]: audit 2022-04-22T20:37:29.345338+0000 mon.a (mon.0) 9934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:30 smithi115 bash[45495]: audit 2022-04-22T20:37:29.345338+0000 mon.a (mon.0) 9934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:30 smithi115 bash[46934]: audit 2022-04-22T20:37:29.345338+0000 mon.a (mon.0) 9934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:31 smithi121 bash[36902]: cluster 2022-04-22T20:37:29.828592+0000 mgr.y (mgr.34107) 6547 : cluster [DBG] pgmap v7684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:31 smithi115 bash[45495]: cluster 2022-04-22T20:37:29.828592+0000 mgr.y (mgr.34107) 6547 : cluster [DBG] pgmap v7684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:31 smithi115 bash[46934]: cluster 2022-04-22T20:37:29.828592+0000 mgr.y (mgr.34107) 6547 : cluster [DBG] pgmap v7684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:32.351Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:33.291 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:37:33.643 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:37:33.643 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:37:33.643 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:37:33.644 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:37:33.645 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:37:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:33 smithi121 bash[36902]: cluster 2022-04-22T20:37:31.829831+0000 mgr.y (mgr.34107) 6548 : cluster [DBG] pgmap v7685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:33 smithi121 bash[36902]: audit 2022-04-22T20:37:32.863658+0000 mon.a (mon.0) 9935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:33 smithi121 bash[36902]: audit 2022-04-22T20:37:32.896152+0000 mon.a (mon.0) 9936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[45495]: cluster 2022-04-22T20:37:31.829831+0000 mgr.y (mgr.34107) 6548 : cluster [DBG] pgmap v7685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[45495]: audit 2022-04-22T20:37:32.863658+0000 mon.a (mon.0) 9935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[45495]: audit 2022-04-22T20:37:32.896152+0000 mon.a (mon.0) 9936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[46934]: cluster 2022-04-22T20:37:31.829831+0000 mgr.y (mgr.34107) 6548 : cluster [DBG] pgmap v7685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[46934]: audit 2022-04-22T20:37:32.863658+0000 mon.a (mon.0) 9935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:33 smithi115 bash[46934]: audit 2022-04-22T20:37:32.896152+0000 mon.a (mon.0) 9936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:34.051 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:34.052 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:37:34.053 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:37:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:34.231Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:34.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:34 smithi115 bash[45495]: audit 2022-04-22T20:37:33.268605+0000 mgr.y (mgr.34107) 6549 : audit [DBG] from='client.52123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:34.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:34 smithi115 bash[45495]: audit 2022-04-22T20:37:34.044509+0000 mon.c (mon.1) 254 : audit [DBG] from='client.? 172.21.15.115:0/3373748706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:34.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:34 smithi115 bash[46934]: audit 2022-04-22T20:37:33.268605+0000 mgr.y (mgr.34107) 6549 : audit [DBG] from='client.52123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:34.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:34 smithi115 bash[46934]: audit 2022-04-22T20:37:34.044509+0000 mon.c (mon.1) 254 : audit [DBG] from='client.? 172.21.15.115:0/3373748706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:34 smithi121 bash[36902]: audit 2022-04-22T20:37:33.268605+0000 mgr.y (mgr.34107) 6549 : audit [DBG] from='client.52123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:34 smithi121 bash[36902]: audit 2022-04-22T20:37:34.044509+0000 mon.c (mon.1) 254 : audit [DBG] from='client.? 172.21.15.115:0/3373748706' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:37:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:35 smithi121 bash[36902]: audit 2022-04-22T20:37:33.615499+0000 mgr.y (mgr.34107) 6550 : audit [DBG] from='client.52129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:35 smithi121 bash[36902]: cluster 2022-04-22T20:37:33.830963+0000 mgr.y (mgr.34107) 6551 : cluster [DBG] pgmap v7686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:35 smithi121 bash[36902]: audit 2022-04-22T20:37:34.348011+0000 mon.a (mon.0) 9937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[45495]: audit 2022-04-22T20:37:33.615499+0000 mgr.y (mgr.34107) 6550 : audit [DBG] from='client.52129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[45495]: cluster 2022-04-22T20:37:33.830963+0000 mgr.y (mgr.34107) 6551 : cluster [DBG] pgmap v7686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[45495]: audit 2022-04-22T20:37:34.348011+0000 mon.a (mon.0) 9937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[46934]: audit 2022-04-22T20:37:33.615499+0000 mgr.y (mgr.34107) 6550 : audit [DBG] from='client.52129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:37:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[46934]: cluster 2022-04-22T20:37:33.830963+0000 mgr.y (mgr.34107) 6551 : cluster [DBG] pgmap v7686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:35 smithi115 bash[46934]: audit 2022-04-22T20:37:34.348011+0000 mon.a (mon.0) 9937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:37 smithi121 bash[36902]: cluster 2022-04-22T20:37:35.832084+0000 mgr.y (mgr.34107) 6552 : cluster [DBG] pgmap v7687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:37 smithi115 bash[45495]: cluster 2022-04-22T20:37:35.832084+0000 mgr.y (mgr.34107) 6552 : cluster [DBG] pgmap v7687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:37 smithi115 bash[46934]: cluster 2022-04-22T20:37:35.832084+0000 mgr.y (mgr.34107) 6552 : cluster [DBG] pgmap v7687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:38 smithi121 bash[36902]: cluster 2022-04-22T20:37:37.833225+0000 mgr.y (mgr.34107) 6553 : cluster [DBG] pgmap v7688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:38 smithi121 bash[36902]: audit 2022-04-22T20:37:37.867249+0000 mon.a (mon.0) 9938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:38 smithi121 bash[36902]: audit 2022-04-22T20:37:37.899196+0000 mon.a (mon.0) 9939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[45495]: cluster 2022-04-22T20:37:37.833225+0000 mgr.y (mgr.34107) 6553 : cluster [DBG] pgmap v7688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[45495]: audit 2022-04-22T20:37:37.867249+0000 mon.a (mon.0) 9938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:38.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[45495]: audit 2022-04-22T20:37:37.899196+0000 mon.a (mon.0) 9939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[46934]: cluster 2022-04-22T20:37:37.833225+0000 mgr.y (mgr.34107) 6553 : cluster [DBG] pgmap v7688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[46934]: audit 2022-04-22T20:37:37.867249+0000 mon.a (mon.0) 9938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:38 smithi115 bash[46934]: audit 2022-04-22T20:37:37.899196+0000 mon.a (mon.0) 9939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:40.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:40 smithi115 bash[46934]: audit 2022-04-22T20:37:39.351135+0000 mon.a (mon.0) 9940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:40.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:40 smithi115 bash[45495]: audit 2022-04-22T20:37:39.351135+0000 mon.a (mon.0) 9940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:40 smithi121 bash[36902]: audit 2022-04-22T20:37:39.351135+0000 mon.a (mon.0) 9940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:41 smithi115 bash[45495]: cluster 2022-04-22T20:37:39.834373+0000 mgr.y (mgr.34107) 6554 : cluster [DBG] pgmap v7689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:41.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:41 smithi115 bash[46934]: cluster 2022-04-22T20:37:39.834373+0000 mgr.y (mgr.34107) 6554 : cluster [DBG] pgmap v7689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:41 smithi121 bash[36902]: cluster 2022-04-22T20:37:39.834373+0000 mgr.y (mgr.34107) 6554 : cluster [DBG] pgmap v7689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:42.352Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[45495]: cluster 2022-04-22T20:37:41.835415+0000 mgr.y (mgr.34107) 6555 : cluster [DBG] pgmap v7690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[45495]: audit 2022-04-22T20:37:42.871130+0000 mon.a (mon.0) 9941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[45495]: audit 2022-04-22T20:37:42.902038+0000 mon.a (mon.0) 9942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[46934]: cluster 2022-04-22T20:37:41.835415+0000 mgr.y (mgr.34107) 6555 : cluster [DBG] pgmap v7690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[46934]: audit 2022-04-22T20:37:42.871130+0000 mon.a (mon.0) 9941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:43.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:43 smithi115 bash[46934]: audit 2022-04-22T20:37:42.902038+0000 mon.a (mon.0) 9942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:43 smithi121 bash[36902]: cluster 2022-04-22T20:37:41.835415+0000 mgr.y (mgr.34107) 6555 : cluster [DBG] pgmap v7690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:43 smithi121 bash[36902]: audit 2022-04-22T20:37:42.871130+0000 mon.a (mon.0) 9941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:43 smithi121 bash[36902]: audit 2022-04-22T20:37:42.902038+0000 mon.a (mon.0) 9942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:44.232Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:45 smithi115 bash[45495]: cluster 2022-04-22T20:37:43.836500+0000 mgr.y (mgr.34107) 6556 : cluster [DBG] pgmap v7691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:45 smithi115 bash[45495]: audit 2022-04-22T20:37:44.354136+0000 mon.a (mon.0) 9943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:45 smithi115 bash[46934]: cluster 2022-04-22T20:37:43.836500+0000 mgr.y (mgr.34107) 6556 : cluster [DBG] pgmap v7691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:45 smithi115 bash[46934]: audit 2022-04-22T20:37:44.354136+0000 mon.a (mon.0) 9943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:45 smithi121 bash[36902]: cluster 2022-04-22T20:37:43.836500+0000 mgr.y (mgr.34107) 6556 : cluster [DBG] pgmap v7691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:45 smithi121 bash[36902]: audit 2022-04-22T20:37:44.354136+0000 mon.a (mon.0) 9943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:47 smithi115 bash[45495]: cluster 2022-04-22T20:37:45.837661+0000 mgr.y (mgr.34107) 6557 : cluster [DBG] pgmap v7692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:47 smithi115 bash[46934]: cluster 2022-04-22T20:37:45.837661+0000 mgr.y (mgr.34107) 6557 : cluster [DBG] pgmap v7692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:47 smithi121 bash[36902]: cluster 2022-04-22T20:37:45.837661+0000 mgr.y (mgr.34107) 6557 : cluster [DBG] pgmap v7692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:47.817754+0000 mon.a (mon.0) 9944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:37:48.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:47.820850+0000 mon.a (mon.0) 9945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:37:48.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:47.823014+0000 mon.a (mon.0) 9946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:37:48.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:47.874711+0000 mon.a (mon.0) 9947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:48.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:47.904756+0000 mon.a (mon.0) 9948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:48.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:48 smithi121 bash[36902]: audit 2022-04-22T20:37:48.173164+0000 mon.a (mon.0) 9949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:37:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:47.817754+0000 mon.a (mon.0) 9944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:37:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:47.820850+0000 mon.a (mon.0) 9945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:37:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:47.823014+0000 mon.a (mon.0) 9946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:37:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:47.874711+0000 mon.a (mon.0) 9947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:47.904756+0000 mon.a (mon.0) 9948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[45495]: audit 2022-04-22T20:37:48.173164+0000 mon.a (mon.0) 9949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:47.817754+0000 mon.a (mon.0) 9944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:47.820850+0000 mon.a (mon.0) 9945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:47.823014+0000 mon.a (mon.0) 9946 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:47.874711+0000 mon.a (mon.0) 9947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:47.904756+0000 mon.a (mon.0) 9948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:48 smithi115 bash[46934]: audit 2022-04-22T20:37:48.173164+0000 mon.a (mon.0) 9949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:37:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:49 smithi121 bash[36902]: cluster 2022-04-22T20:37:47.838735+0000 mgr.y (mgr.34107) 6558 : cluster [DBG] pgmap v7693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:49 smithi115 bash[45495]: cluster 2022-04-22T20:37:47.838735+0000 mgr.y (mgr.34107) 6558 : cluster [DBG] pgmap v7693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:49 smithi115 bash[46934]: cluster 2022-04-22T20:37:47.838735+0000 mgr.y (mgr.34107) 6558 : cluster [DBG] pgmap v7693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:50 smithi121 bash[36902]: audit 2022-04-22T20:37:49.357377+0000 mon.a (mon.0) 9950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:50 smithi121 bash[36902]: cluster 2022-04-22T20:37:49.840005+0000 mgr.y (mgr.34107) 6559 : cluster [DBG] pgmap v7694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:50 smithi115 bash[45495]: audit 2022-04-22T20:37:49.357377+0000 mon.a (mon.0) 9950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:50 smithi115 bash[45495]: cluster 2022-04-22T20:37:49.840005+0000 mgr.y (mgr.34107) 6559 : cluster [DBG] pgmap v7694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:50.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:50 smithi115 bash[46934]: audit 2022-04-22T20:37:49.357377+0000 mon.a (mon.0) 9950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:50.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:50 smithi115 bash[46934]: cluster 2022-04-22T20:37:49.840005+0000 mgr.y (mgr.34107) 6559 : cluster [DBG] pgmap v7694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:52.353Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:52 smithi121 bash[36902]: cluster 2022-04-22T20:37:51.841192+0000 mgr.y (mgr.34107) 6560 : cluster [DBG] pgmap v7695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:52 smithi121 bash[36902]: audit 2022-04-22T20:37:52.878331+0000 mon.a (mon.0) 9951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:52 smithi115 bash[45495]: cluster 2022-04-22T20:37:51.841192+0000 mgr.y (mgr.34107) 6560 : cluster [DBG] pgmap v7695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:52 smithi115 bash[45495]: audit 2022-04-22T20:37:52.878331+0000 mon.a (mon.0) 9951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:52 smithi115 bash[46934]: cluster 2022-04-22T20:37:51.841192+0000 mgr.y (mgr.34107) 6560 : cluster [DBG] pgmap v7695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:52 smithi115 bash[46934]: audit 2022-04-22T20:37:52.878331+0000 mon.a (mon.0) 9951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:54.239 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:53 smithi115 bash[45495]: audit 2022-04-22T20:37:52.906897+0000 mon.a (mon.0) 9952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:54.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:53 smithi115 bash[46934]: audit 2022-04-22T20:37:52.906897+0000 mon.a (mon.0) 9952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:53 smithi121 bash[36902]: audit 2022-04-22T20:37:52.906897+0000 mon.a (mon.0) 9952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:37:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:37:54.232Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:37:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:55 smithi115 bash[45495]: cluster 2022-04-22T20:37:53.842454+0000 mgr.y (mgr.34107) 6561 : cluster [DBG] pgmap v7696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:55 smithi115 bash[45495]: audit 2022-04-22T20:37:54.360343+0000 mon.a (mon.0) 9953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:55 smithi115 bash[46934]: cluster 2022-04-22T20:37:53.842454+0000 mgr.y (mgr.34107) 6561 : cluster [DBG] pgmap v7696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:55 smithi115 bash[46934]: audit 2022-04-22T20:37:54.360343+0000 mon.a (mon.0) 9953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:55 smithi121 bash[36902]: cluster 2022-04-22T20:37:53.842454+0000 mgr.y (mgr.34107) 6561 : cluster [DBG] pgmap v7696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:55 smithi121 bash[36902]: audit 2022-04-22T20:37:54.360343+0000 mon.a (mon.0) 9953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:37:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:57 smithi115 bash[45495]: cluster 2022-04-22T20:37:55.843608+0000 mgr.y (mgr.34107) 6562 : cluster [DBG] pgmap v7697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:57 smithi115 bash[46934]: cluster 2022-04-22T20:37:55.843608+0000 mgr.y (mgr.34107) 6562 : cluster [DBG] pgmap v7697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:57 smithi121 bash[36902]: cluster 2022-04-22T20:37:55.843608+0000 mgr.y (mgr.34107) 6562 : cluster [DBG] pgmap v7697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:58 smithi115 bash[45495]: audit 2022-04-22T20:37:57.881862+0000 mon.a (mon.0) 9954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:58 smithi115 bash[45495]: audit 2022-04-22T20:37:57.909570+0000 mon.a (mon.0) 9955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:58 smithi115 bash[46934]: audit 2022-04-22T20:37:57.881862+0000 mon.a (mon.0) 9954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:58.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:58 smithi115 bash[46934]: audit 2022-04-22T20:37:57.909570+0000 mon.a (mon.0) 9955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:58 smithi121 bash[36902]: audit 2022-04-22T20:37:57.881862+0000 mon.a (mon.0) 9954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:37:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:58 smithi121 bash[36902]: audit 2022-04-22T20:37:57.909570+0000 mon.a (mon.0) 9955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:37:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:37:59 smithi115 bash[45495]: cluster 2022-04-22T20:37:57.844736+0000 mgr.y (mgr.34107) 6563 : cluster [DBG] pgmap v7698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:37:59 smithi115 bash[46934]: cluster 2022-04-22T20:37:57.844736+0000 mgr.y (mgr.34107) 6563 : cluster [DBG] pgmap v7698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:37:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:37:59 smithi121 bash[36902]: cluster 2022-04-22T20:37:57.844736+0000 mgr.y (mgr.34107) 6563 : cluster [DBG] pgmap v7698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:00 smithi121 bash[36902]: audit 2022-04-22T20:37:59.363352+0000 mon.a (mon.0) 9956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:00 smithi115 bash[45495]: audit 2022-04-22T20:37:59.363352+0000 mon.a (mon.0) 9956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:00 smithi115 bash[46934]: audit 2022-04-22T20:37:59.363352+0000 mon.a (mon.0) 9956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:01 smithi121 bash[36902]: cluster 2022-04-22T20:37:59.845866+0000 mgr.y (mgr.34107) 6564 : cluster [DBG] pgmap v7699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:01 smithi115 bash[45495]: cluster 2022-04-22T20:37:59.845866+0000 mgr.y (mgr.34107) 6564 : cluster [DBG] pgmap v7699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:01.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:01 smithi115 bash[46934]: cluster 2022-04-22T20:37:59.845866+0000 mgr.y (mgr.34107) 6564 : cluster [DBG] pgmap v7699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:02.353Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:03 smithi121 bash[36902]: cluster 2022-04-22T20:38:01.846949+0000 mgr.y (mgr.34107) 6565 : cluster [DBG] pgmap v7700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:03 smithi121 bash[36902]: audit 2022-04-22T20:38:02.885773+0000 mon.a (mon.0) 9957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:03 smithi121 bash[36902]: audit 2022-04-22T20:38:02.912570+0000 mon.a (mon.0) 9958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[45495]: cluster 2022-04-22T20:38:01.846949+0000 mgr.y (mgr.34107) 6565 : cluster [DBG] pgmap v7700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[45495]: audit 2022-04-22T20:38:02.885773+0000 mon.a (mon.0) 9957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[45495]: audit 2022-04-22T20:38:02.912570+0000 mon.a (mon.0) 9958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[46934]: cluster 2022-04-22T20:38:01.846949+0000 mgr.y (mgr.34107) 6565 : cluster [DBG] pgmap v7700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[46934]: audit 2022-04-22T20:38:02.885773+0000 mon.a (mon.0) 9957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:03 smithi115 bash[46934]: audit 2022-04-22T20:38:02.912570+0000 mon.a (mon.0) 9958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:04.436 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:38:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:04.232Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:04.788 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:38:04.789 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:38:04.790 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:38:04.791 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:38:05.194 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:38:05.194 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:38:05.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:38:05.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:05.196 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:38:05.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:38:05.196 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:38:05.196 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:38:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:05 smithi121 bash[36902]: cluster 2022-04-22T20:38:03.848156+0000 mgr.y (mgr.34107) 6566 : cluster [DBG] pgmap v7701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:05.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:05 smithi121 bash[36902]: audit 2022-04-22T20:38:04.366172+0000 mon.a (mon.0) 9959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:05.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:05 smithi121 bash[36902]: audit 2022-04-22T20:38:05.187193+0000 mon.c (mon.1) 255 : audit [DBG] from='client.? 172.21.15.115:0/1213937207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[45495]: cluster 2022-04-22T20:38:03.848156+0000 mgr.y (mgr.34107) 6566 : cluster [DBG] pgmap v7701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[45495]: audit 2022-04-22T20:38:04.366172+0000 mon.a (mon.0) 9959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[45495]: audit 2022-04-22T20:38:05.187193+0000 mon.c (mon.1) 255 : audit [DBG] from='client.? 172.21.15.115:0/1213937207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[46934]: cluster 2022-04-22T20:38:03.848156+0000 mgr.y (mgr.34107) 6566 : cluster [DBG] pgmap v7701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[46934]: audit 2022-04-22T20:38:04.366172+0000 mon.a (mon.0) 9959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:05 smithi115 bash[46934]: audit 2022-04-22T20:38:05.187193+0000 mon.c (mon.1) 255 : audit [DBG] from='client.? 172.21.15.115:0/1213937207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:06 smithi121 bash[36902]: audit 2022-04-22T20:38:04.412351+0000 mgr.y (mgr.34107) 6567 : audit [DBG] from='client.52141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:06 smithi121 bash[36902]: audit 2022-04-22T20:38:04.761306+0000 mgr.y (mgr.34107) 6568 : audit [DBG] from='client.42216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:06 smithi115 bash[45495]: audit 2022-04-22T20:38:04.412351+0000 mgr.y (mgr.34107) 6567 : audit [DBG] from='client.52141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:06 smithi115 bash[45495]: audit 2022-04-22T20:38:04.761306+0000 mgr.y (mgr.34107) 6568 : audit [DBG] from='client.42216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:06.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:06 smithi115 bash[46934]: audit 2022-04-22T20:38:04.412351+0000 mgr.y (mgr.34107) 6567 : audit [DBG] from='client.52141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:06 smithi115 bash[46934]: audit 2022-04-22T20:38:04.761306+0000 mgr.y (mgr.34107) 6568 : audit [DBG] from='client.42216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:07 smithi121 bash[36902]: cluster 2022-04-22T20:38:05.849237+0000 mgr.y (mgr.34107) 6569 : cluster [DBG] pgmap v7702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:07 smithi115 bash[45495]: cluster 2022-04-22T20:38:05.849237+0000 mgr.y (mgr.34107) 6569 : cluster [DBG] pgmap v7702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:07 smithi115 bash[46934]: cluster 2022-04-22T20:38:05.849237+0000 mgr.y (mgr.34107) 6569 : cluster [DBG] pgmap v7702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:08 smithi121 bash[36902]: cluster 2022-04-22T20:38:07.850380+0000 mgr.y (mgr.34107) 6570 : cluster [DBG] pgmap v7703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:08 smithi121 bash[36902]: audit 2022-04-22T20:38:07.889229+0000 mon.a (mon.0) 9960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:08 smithi121 bash[36902]: audit 2022-04-22T20:38:07.915654+0000 mon.a (mon.0) 9961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[45495]: cluster 2022-04-22T20:38:07.850380+0000 mgr.y (mgr.34107) 6570 : cluster [DBG] pgmap v7703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[45495]: audit 2022-04-22T20:38:07.889229+0000 mon.a (mon.0) 9960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:08.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[45495]: audit 2022-04-22T20:38:07.915654+0000 mon.a (mon.0) 9961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[46934]: cluster 2022-04-22T20:38:07.850380+0000 mgr.y (mgr.34107) 6570 : cluster [DBG] pgmap v7703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[46934]: audit 2022-04-22T20:38:07.889229+0000 mon.a (mon.0) 9960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:08 smithi115 bash[46934]: audit 2022-04-22T20:38:07.915654+0000 mon.a (mon.0) 9961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:10.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:10 smithi115 bash[45495]: audit 2022-04-22T20:38:09.368760+0000 mon.a (mon.0) 9962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:10.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:10 smithi115 bash[46934]: audit 2022-04-22T20:38:09.368760+0000 mon.a (mon.0) 9962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:10 smithi121 bash[36902]: audit 2022-04-22T20:38:09.368760+0000 mon.a (mon.0) 9962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:11 smithi115 bash[45495]: cluster 2022-04-22T20:38:09.851465+0000 mgr.y (mgr.34107) 6571 : cluster [DBG] pgmap v7704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:11.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:11 smithi115 bash[46934]: cluster 2022-04-22T20:38:09.851465+0000 mgr.y (mgr.34107) 6571 : cluster [DBG] pgmap v7704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:11 smithi121 bash[36902]: cluster 2022-04-22T20:38:09.851465+0000 mgr.y (mgr.34107) 6571 : cluster [DBG] pgmap v7704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:12.355Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[45495]: cluster 2022-04-22T20:38:11.852477+0000 mgr.y (mgr.34107) 6572 : cluster [DBG] pgmap v7705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[45495]: audit 2022-04-22T20:38:12.893036+0000 mon.a (mon.0) 9963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[45495]: audit 2022-04-22T20:38:12.918512+0000 mon.a (mon.0) 9964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[46934]: cluster 2022-04-22T20:38:11.852477+0000 mgr.y (mgr.34107) 6572 : cluster [DBG] pgmap v7705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[46934]: audit 2022-04-22T20:38:12.893036+0000 mon.a (mon.0) 9963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:13 smithi115 bash[46934]: audit 2022-04-22T20:38:12.918512+0000 mon.a (mon.0) 9964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:13 smithi121 bash[36902]: cluster 2022-04-22T20:38:11.852477+0000 mgr.y (mgr.34107) 6572 : cluster [DBG] pgmap v7705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:13 smithi121 bash[36902]: audit 2022-04-22T20:38:12.893036+0000 mon.a (mon.0) 9963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:13 smithi121 bash[36902]: audit 2022-04-22T20:38:12.918512+0000 mon.a (mon.0) 9964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:14.234Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:15 smithi121 bash[36902]: cluster 2022-04-22T20:38:13.853586+0000 mgr.y (mgr.34107) 6573 : cluster [DBG] pgmap v7706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:15 smithi121 bash[36902]: audit 2022-04-22T20:38:14.371820+0000 mon.a (mon.0) 9965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:15 smithi115 bash[45495]: cluster 2022-04-22T20:38:13.853586+0000 mgr.y (mgr.34107) 6573 : cluster [DBG] pgmap v7706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:15 smithi115 bash[45495]: audit 2022-04-22T20:38:14.371820+0000 mon.a (mon.0) 9965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:15 smithi115 bash[46934]: cluster 2022-04-22T20:38:13.853586+0000 mgr.y (mgr.34107) 6573 : cluster [DBG] pgmap v7706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:15 smithi115 bash[46934]: audit 2022-04-22T20:38:14.371820+0000 mon.a (mon.0) 9965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:16 smithi121 bash[36902]: audit 2022-04-22T20:38:15.479127+0000 mon.a (mon.0) 9966 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:38:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:16 smithi121 bash[36902]: audit 2022-04-22T20:38:15.479649+0000 mon.a (mon.0) 9967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:38:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:16 smithi115 bash[45495]: audit 2022-04-22T20:38:15.479127+0000 mon.a (mon.0) 9966 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:38:16.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:16 smithi115 bash[45495]: audit 2022-04-22T20:38:15.479649+0000 mon.a (mon.0) 9967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:38:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:16 smithi115 bash[46934]: audit 2022-04-22T20:38:15.479127+0000 mon.a (mon.0) 9966 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:38:16.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:16 smithi115 bash[46934]: audit 2022-04-22T20:38:15.479649+0000 mon.a (mon.0) 9967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:38:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:17 smithi121 bash[36902]: cluster 2022-04-22T20:38:15.854629+0000 mgr.y (mgr.34107) 6574 : cluster [DBG] pgmap v7707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:17 smithi115 bash[45495]: cluster 2022-04-22T20:38:15.854629+0000 mgr.y (mgr.34107) 6574 : cluster [DBG] pgmap v7707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:17 smithi115 bash[46934]: cluster 2022-04-22T20:38:15.854629+0000 mgr.y (mgr.34107) 6574 : cluster [DBG] pgmap v7707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:18 smithi121 bash[36902]: audit 2022-04-22T20:38:17.896885+0000 mon.a (mon.0) 9968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:18 smithi121 bash[36902]: audit 2022-04-22T20:38:17.921306+0000 mon.a (mon.0) 9969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:18 smithi115 bash[46934]: audit 2022-04-22T20:38:17.896885+0000 mon.a (mon.0) 9968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:18 smithi115 bash[46934]: audit 2022-04-22T20:38:17.921306+0000 mon.a (mon.0) 9969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:18 smithi115 bash[45495]: audit 2022-04-22T20:38:17.896885+0000 mon.a (mon.0) 9968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:18.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:18 smithi115 bash[45495]: audit 2022-04-22T20:38:17.921306+0000 mon.a (mon.0) 9969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:19 smithi121 bash[36902]: cluster 2022-04-22T20:38:17.855680+0000 mgr.y (mgr.34107) 6575 : cluster [DBG] pgmap v7708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:19 smithi115 bash[45495]: cluster 2022-04-22T20:38:17.855680+0000 mgr.y (mgr.34107) 6575 : cluster [DBG] pgmap v7708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:19 smithi115 bash[46934]: cluster 2022-04-22T20:38:17.855680+0000 mgr.y (mgr.34107) 6575 : cluster [DBG] pgmap v7708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:20 smithi121 bash[36902]: audit 2022-04-22T20:38:19.374922+0000 mon.a (mon.0) 9970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:20 smithi115 bash[45495]: audit 2022-04-22T20:38:19.374922+0000 mon.a (mon.0) 9970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:20 smithi115 bash[46934]: audit 2022-04-22T20:38:19.374922+0000 mon.a (mon.0) 9970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:21 smithi121 bash[36902]: cluster 2022-04-22T20:38:19.856719+0000 mgr.y (mgr.34107) 6576 : cluster [DBG] pgmap v7709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:21 smithi115 bash[45495]: cluster 2022-04-22T20:38:19.856719+0000 mgr.y (mgr.34107) 6576 : cluster [DBG] pgmap v7709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:21 smithi115 bash[46934]: cluster 2022-04-22T20:38:19.856719+0000 mgr.y (mgr.34107) 6576 : cluster [DBG] pgmap v7709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:22 smithi121 bash[36902]: cluster 2022-04-22T20:38:21.857878+0000 mgr.y (mgr.34107) 6577 : cluster [DBG] pgmap v7710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:22 smithi115 bash[45495]: cluster 2022-04-22T20:38:21.857878+0000 mgr.y (mgr.34107) 6577 : cluster [DBG] pgmap v7710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:22.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:22 smithi115 bash[46934]: cluster 2022-04-22T20:38:21.857878+0000 mgr.y (mgr.34107) 6577 : cluster [DBG] pgmap v7710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:22.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:22.353Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:23 smithi121 bash[36902]: audit 2022-04-22T20:38:22.900725+0000 mon.a (mon.0) 9971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:23 smithi121 bash[36902]: audit 2022-04-22T20:38:22.924517+0000 mon.a (mon.0) 9972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:23 smithi115 bash[45495]: audit 2022-04-22T20:38:22.900725+0000 mon.a (mon.0) 9971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:23 smithi115 bash[45495]: audit 2022-04-22T20:38:22.924517+0000 mon.a (mon.0) 9972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:23 smithi115 bash[46934]: audit 2022-04-22T20:38:22.900725+0000 mon.a (mon.0) 9971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:23 smithi115 bash[46934]: audit 2022-04-22T20:38:22.924517+0000 mon.a (mon.0) 9972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:24.234Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:25 smithi115 bash[45495]: cluster 2022-04-22T20:38:23.859212+0000 mgr.y (mgr.34107) 6578 : cluster [DBG] pgmap v7711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:25 smithi115 bash[45495]: audit 2022-04-22T20:38:24.377889+0000 mon.a (mon.0) 9973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:25 smithi115 bash[46934]: cluster 2022-04-22T20:38:23.859212+0000 mgr.y (mgr.34107) 6578 : cluster [DBG] pgmap v7711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:25 smithi115 bash[46934]: audit 2022-04-22T20:38:24.377889+0000 mon.a (mon.0) 9973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:25 smithi121 bash[36902]: cluster 2022-04-22T20:38:23.859212+0000 mgr.y (mgr.34107) 6578 : cluster [DBG] pgmap v7711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:25 smithi121 bash[36902]: audit 2022-04-22T20:38:24.377889+0000 mon.a (mon.0) 9973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:27 smithi115 bash[45495]: cluster 2022-04-22T20:38:25.860362+0000 mgr.y (mgr.34107) 6579 : cluster [DBG] pgmap v7712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:27.633 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:27 smithi115 bash[46934]: cluster 2022-04-22T20:38:25.860362+0000 mgr.y (mgr.34107) 6579 : cluster [DBG] pgmap v7712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:27 smithi121 bash[36902]: cluster 2022-04-22T20:38:25.860362+0000 mgr.y (mgr.34107) 6579 : cluster [DBG] pgmap v7712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:28 smithi121 bash[36902]: audit 2022-04-22T20:38:27.904334+0000 mon.a (mon.0) 9974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:28 smithi121 bash[36902]: audit 2022-04-22T20:38:27.927690+0000 mon.a (mon.0) 9975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:28 smithi115 bash[46934]: audit 2022-04-22T20:38:27.904334+0000 mon.a (mon.0) 9974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:28 smithi115 bash[46934]: audit 2022-04-22T20:38:27.927690+0000 mon.a (mon.0) 9975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:28 smithi115 bash[45495]: audit 2022-04-22T20:38:27.904334+0000 mon.a (mon.0) 9974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:28 smithi115 bash[45495]: audit 2022-04-22T20:38:27.927690+0000 mon.a (mon.0) 9975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:29 smithi121 bash[36902]: cluster 2022-04-22T20:38:27.861433+0000 mgr.y (mgr.34107) 6580 : cluster [DBG] pgmap v7713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:29 smithi115 bash[45495]: cluster 2022-04-22T20:38:27.861433+0000 mgr.y (mgr.34107) 6580 : cluster [DBG] pgmap v7713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:29 smithi115 bash[46934]: cluster 2022-04-22T20:38:27.861433+0000 mgr.y (mgr.34107) 6580 : cluster [DBG] pgmap v7713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:30 smithi121 bash[36902]: audit 2022-04-22T20:38:29.380687+0000 mon.a (mon.0) 9976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:30 smithi115 bash[45495]: audit 2022-04-22T20:38:29.380687+0000 mon.a (mon.0) 9976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:30 smithi115 bash[46934]: audit 2022-04-22T20:38:29.380687+0000 mon.a (mon.0) 9976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:31 smithi121 bash[36902]: cluster 2022-04-22T20:38:29.862572+0000 mgr.y (mgr.34107) 6581 : cluster [DBG] pgmap v7714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:31 smithi115 bash[45495]: cluster 2022-04-22T20:38:29.862572+0000 mgr.y (mgr.34107) 6581 : cluster [DBG] pgmap v7714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:31 smithi115 bash[46934]: cluster 2022-04-22T20:38:29.862572+0000 mgr.y (mgr.34107) 6581 : cluster [DBG] pgmap v7714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:32.355Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:33 smithi121 bash[36902]: cluster 2022-04-22T20:38:31.863675+0000 mgr.y (mgr.34107) 6582 : cluster [DBG] pgmap v7715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:33 smithi121 bash[36902]: audit 2022-04-22T20:38:32.908233+0000 mon.a (mon.0) 9977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:33 smithi121 bash[36902]: audit 2022-04-22T20:38:32.930583+0000 mon.a (mon.0) 9978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[45495]: cluster 2022-04-22T20:38:31.863675+0000 mgr.y (mgr.34107) 6582 : cluster [DBG] pgmap v7715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[45495]: audit 2022-04-22T20:38:32.908233+0000 mon.a (mon.0) 9977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[45495]: audit 2022-04-22T20:38:32.930583+0000 mon.a (mon.0) 9978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[46934]: cluster 2022-04-22T20:38:31.863675+0000 mgr.y (mgr.34107) 6582 : cluster [DBG] pgmap v7715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[46934]: audit 2022-04-22T20:38:32.908233+0000 mon.a (mon.0) 9977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:33 smithi115 bash[46934]: audit 2022-04-22T20:38:32.930583+0000 mon.a (mon.0) 9978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:34.234Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:35.565 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:38:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:35 smithi121 bash[36902]: cluster 2022-04-22T20:38:33.864847+0000 mgr.y (mgr.34107) 6583 : cluster [DBG] pgmap v7716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:35 smithi121 bash[36902]: audit 2022-04-22T20:38:34.383659+0000 mon.a (mon.0) 9979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:35 smithi115 bash[46934]: cluster 2022-04-22T20:38:33.864847+0000 mgr.y (mgr.34107) 6583 : cluster [DBG] pgmap v7716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:35 smithi115 bash[46934]: audit 2022-04-22T20:38:34.383659+0000 mon.a (mon.0) 9979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:35 smithi115 bash[45495]: cluster 2022-04-22T20:38:33.864847+0000 mgr.y (mgr.34107) 6583 : cluster [DBG] pgmap v7716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:35.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:35 smithi115 bash[45495]: audit 2022-04-22T20:38:34.383659+0000 mon.a (mon.0) 9979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:35.909 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:38:35.910 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:38:35.911 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:38:35.912 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:38:36.309 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:38:36.310 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:38:36.311 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:38:36.311 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:38:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:36 smithi121 bash[36902]: audit 2022-04-22T20:38:36.301971+0000 mon.a (mon.0) 9980 : audit [DBG] from='client.? 172.21.15.115:0/4266785538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:36 smithi115 bash[45495]: audit 2022-04-22T20:38:36.301971+0000 mon.a (mon.0) 9980 : audit [DBG] from='client.? 172.21.15.115:0/4266785538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:36 smithi115 bash[46934]: audit 2022-04-22T20:38:36.301971+0000 mon.a (mon.0) 9980 : audit [DBG] from='client.? 172.21.15.115:0/4266785538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:38:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:37 smithi121 bash[36902]: audit 2022-04-22T20:38:35.541223+0000 mgr.y (mgr.34107) 6584 : audit [DBG] from='client.42228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:37 smithi121 bash[36902]: cluster 2022-04-22T20:38:35.865878+0000 mgr.y (mgr.34107) 6585 : cluster [DBG] pgmap v7717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:37 smithi121 bash[36902]: audit 2022-04-22T20:38:35.883751+0000 mgr.y (mgr.34107) 6586 : audit [DBG] from='client.52165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[45495]: audit 2022-04-22T20:38:35.541223+0000 mgr.y (mgr.34107) 6584 : audit [DBG] from='client.42228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[45495]: cluster 2022-04-22T20:38:35.865878+0000 mgr.y (mgr.34107) 6585 : cluster [DBG] pgmap v7717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[45495]: audit 2022-04-22T20:38:35.883751+0000 mgr.y (mgr.34107) 6586 : audit [DBG] from='client.52165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:37.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[46934]: audit 2022-04-22T20:38:35.541223+0000 mgr.y (mgr.34107) 6584 : audit [DBG] from='client.42228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:37.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[46934]: cluster 2022-04-22T20:38:35.865878+0000 mgr.y (mgr.34107) 6585 : cluster [DBG] pgmap v7717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:37.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:37 smithi115 bash[46934]: audit 2022-04-22T20:38:35.883751+0000 mgr.y (mgr.34107) 6586 : audit [DBG] from='client.52165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:38:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:38 smithi121 bash[36902]: audit 2022-04-22T20:38:37.911927+0000 mon.a (mon.0) 9981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:38 smithi121 bash[36902]: audit 2022-04-22T20:38:37.933569+0000 mon.a (mon.0) 9982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:38 smithi115 bash[45495]: audit 2022-04-22T20:38:37.911927+0000 mon.a (mon.0) 9981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:38 smithi115 bash[45495]: audit 2022-04-22T20:38:37.933569+0000 mon.a (mon.0) 9982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:38 smithi115 bash[46934]: audit 2022-04-22T20:38:37.911927+0000 mon.a (mon.0) 9981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:38 smithi115 bash[46934]: audit 2022-04-22T20:38:37.933569+0000 mon.a (mon.0) 9982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:39 smithi121 bash[36902]: cluster 2022-04-22T20:38:37.866982+0000 mgr.y (mgr.34107) 6587 : cluster [DBG] pgmap v7718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:39 smithi115 bash[45495]: cluster 2022-04-22T20:38:37.866982+0000 mgr.y (mgr.34107) 6587 : cluster [DBG] pgmap v7718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:39.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:39 smithi115 bash[46934]: cluster 2022-04-22T20:38:37.866982+0000 mgr.y (mgr.34107) 6587 : cluster [DBG] pgmap v7718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:40 smithi121 bash[36902]: audit 2022-04-22T20:38:39.386701+0000 mon.a (mon.0) 9983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:40 smithi121 bash[36902]: cluster 2022-04-22T20:38:39.868119+0000 mgr.y (mgr.34107) 6588 : cluster [DBG] pgmap v7719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:40 smithi115 bash[46934]: audit 2022-04-22T20:38:39.386701+0000 mon.a (mon.0) 9983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:40 smithi115 bash[46934]: cluster 2022-04-22T20:38:39.868119+0000 mgr.y (mgr.34107) 6588 : cluster [DBG] pgmap v7719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:40 smithi115 bash[45495]: audit 2022-04-22T20:38:39.386701+0000 mon.a (mon.0) 9983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:40.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:40 smithi115 bash[45495]: cluster 2022-04-22T20:38:39.868119+0000 mgr.y (mgr.34107) 6588 : cluster [DBG] pgmap v7719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:42.356Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:42 smithi121 bash[36902]: cluster 2022-04-22T20:38:41.869192+0000 mgr.y (mgr.34107) 6589 : cluster [DBG] pgmap v7720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:43.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:42 smithi121 bash[36902]: audit 2022-04-22T20:38:42.915789+0000 mon.a (mon.0) 9984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:42 smithi115 bash[45495]: cluster 2022-04-22T20:38:41.869192+0000 mgr.y (mgr.34107) 6589 : cluster [DBG] pgmap v7720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:43.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:42 smithi115 bash[45495]: audit 2022-04-22T20:38:42.915789+0000 mon.a (mon.0) 9984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:42 smithi115 bash[46934]: cluster 2022-04-22T20:38:41.869192+0000 mgr.y (mgr.34107) 6589 : cluster [DBG] pgmap v7720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:42 smithi115 bash[46934]: audit 2022-04-22T20:38:42.915789+0000 mon.a (mon.0) 9984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:44.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:43 smithi115 bash[46934]: audit 2022-04-22T20:38:42.936346+0000 mon.a (mon.0) 9985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:44.243 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:43 smithi115 bash[45495]: audit 2022-04-22T20:38:42.936346+0000 mon.a (mon.0) 9985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:43 smithi121 bash[36902]: audit 2022-04-22T20:38:42.936346+0000 mon.a (mon.0) 9985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:44.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:44.236Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:45 smithi121 bash[36902]: cluster 2022-04-22T20:38:43.870349+0000 mgr.y (mgr.34107) 6590 : cluster [DBG] pgmap v7721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:45 smithi121 bash[36902]: audit 2022-04-22T20:38:44.389732+0000 mon.a (mon.0) 9986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:45 smithi115 bash[45495]: cluster 2022-04-22T20:38:43.870349+0000 mgr.y (mgr.34107) 6590 : cluster [DBG] pgmap v7721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:45 smithi115 bash[45495]: audit 2022-04-22T20:38:44.389732+0000 mon.a (mon.0) 9986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:45 smithi115 bash[46934]: cluster 2022-04-22T20:38:43.870349+0000 mgr.y (mgr.34107) 6590 : cluster [DBG] pgmap v7721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:45 smithi115 bash[46934]: audit 2022-04-22T20:38:44.389732+0000 mon.a (mon.0) 9986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:47 smithi121 bash[36902]: cluster 2022-04-22T20:38:45.871512+0000 mgr.y (mgr.34107) 6591 : cluster [DBG] pgmap v7722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:47 smithi115 bash[45495]: cluster 2022-04-22T20:38:45.871512+0000 mgr.y (mgr.34107) 6591 : cluster [DBG] pgmap v7722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:47 smithi115 bash[46934]: cluster 2022-04-22T20:38:45.871512+0000 mgr.y (mgr.34107) 6591 : cluster [DBG] pgmap v7722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:48 smithi121 bash[36902]: audit 2022-04-22T20:38:47.919382+0000 mon.a (mon.0) 9987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:48 smithi121 bash[36902]: audit 2022-04-22T20:38:47.938683+0000 mon.a (mon.0) 9988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:48 smithi121 bash[36902]: audit 2022-04-22T20:38:48.181633+0000 mon.a (mon.0) 9989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:38:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:48 smithi121 bash[36902]: audit 2022-04-22T20:38:48.184567+0000 mon.a (mon.0) 9990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:38:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:48 smithi121 bash[36902]: audit 2022-04-22T20:38:48.186469+0000 mon.a (mon.0) 9991 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:38:48.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[45495]: audit 2022-04-22T20:38:47.919382+0000 mon.a (mon.0) 9987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[45495]: audit 2022-04-22T20:38:47.938683+0000 mon.a (mon.0) 9988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[45495]: audit 2022-04-22T20:38:48.181633+0000 mon.a (mon.0) 9989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[45495]: audit 2022-04-22T20:38:48.184567+0000 mon.a (mon.0) 9990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[45495]: audit 2022-04-22T20:38:48.186469+0000 mon.a (mon.0) 9991 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[46934]: audit 2022-04-22T20:38:47.919382+0000 mon.a (mon.0) 9987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:48.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[46934]: audit 2022-04-22T20:38:47.938683+0000 mon.a (mon.0) 9988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[46934]: audit 2022-04-22T20:38:48.181633+0000 mon.a (mon.0) 9989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:38:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[46934]: audit 2022-04-22T20:38:48.184567+0000 mon.a (mon.0) 9990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:38:48.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:48 smithi115 bash[46934]: audit 2022-04-22T20:38:48.186469+0000 mon.a (mon.0) 9991 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:38:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:49 smithi121 bash[36902]: cluster 2022-04-22T20:38:47.872627+0000 mgr.y (mgr.34107) 6592 : cluster [DBG] pgmap v7723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:49 smithi121 bash[36902]: audit 2022-04-22T20:38:48.553943+0000 mon.a (mon.0) 9992 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:38:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:49 smithi115 bash[45495]: cluster 2022-04-22T20:38:47.872627+0000 mgr.y (mgr.34107) 6592 : cluster [DBG] pgmap v7723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:49 smithi115 bash[45495]: audit 2022-04-22T20:38:48.553943+0000 mon.a (mon.0) 9992 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:38:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:49 smithi115 bash[46934]: cluster 2022-04-22T20:38:47.872627+0000 mgr.y (mgr.34107) 6592 : cluster [DBG] pgmap v7723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:49 smithi115 bash[46934]: audit 2022-04-22T20:38:48.553943+0000 mon.a (mon.0) 9992 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:38:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:50 smithi121 bash[36902]: audit 2022-04-22T20:38:49.392519+0000 mon.a (mon.0) 9993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:50 smithi115 bash[45495]: audit 2022-04-22T20:38:49.392519+0000 mon.a (mon.0) 9993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:50 smithi115 bash[46934]: audit 2022-04-22T20:38:49.392519+0000 mon.a (mon.0) 9993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:51 smithi121 bash[36902]: cluster 2022-04-22T20:38:49.873865+0000 mgr.y (mgr.34107) 6593 : cluster [DBG] pgmap v7724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:51 smithi115 bash[45495]: cluster 2022-04-22T20:38:49.873865+0000 mgr.y (mgr.34107) 6593 : cluster [DBG] pgmap v7724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:51 smithi115 bash[46934]: cluster 2022-04-22T20:38:49.873865+0000 mgr.y (mgr.34107) 6593 : cluster [DBG] pgmap v7724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:52.356Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:53 smithi121 bash[36902]: cluster 2022-04-22T20:38:51.874927+0000 mgr.y (mgr.34107) 6594 : cluster [DBG] pgmap v7725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:53 smithi121 bash[36902]: audit 2022-04-22T20:38:52.923137+0000 mon.a (mon.0) 9994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:53 smithi121 bash[36902]: audit 2022-04-22T20:38:52.941269+0000 mon.a (mon.0) 9995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[46934]: cluster 2022-04-22T20:38:51.874927+0000 mgr.y (mgr.34107) 6594 : cluster [DBG] pgmap v7725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[46934]: audit 2022-04-22T20:38:52.923137+0000 mon.a (mon.0) 9994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[46934]: audit 2022-04-22T20:38:52.941269+0000 mon.a (mon.0) 9995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:53.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[45495]: cluster 2022-04-22T20:38:51.874927+0000 mgr.y (mgr.34107) 6594 : cluster [DBG] pgmap v7725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:53.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[45495]: audit 2022-04-22T20:38:52.923137+0000 mon.a (mon.0) 9994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:53.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:53 smithi115 bash[45495]: audit 2022-04-22T20:38:52.941269+0000 mon.a (mon.0) 9995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:38:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:38:54.237Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:38:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:55 smithi121 bash[36902]: cluster 2022-04-22T20:38:53.876137+0000 mgr.y (mgr.34107) 6595 : cluster [DBG] pgmap v7726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:55 smithi121 bash[36902]: audit 2022-04-22T20:38:54.395492+0000 mon.a (mon.0) 9996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:55 smithi115 bash[46934]: cluster 2022-04-22T20:38:53.876137+0000 mgr.y (mgr.34107) 6595 : cluster [DBG] pgmap v7726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:55 smithi115 bash[46934]: audit 2022-04-22T20:38:54.395492+0000 mon.a (mon.0) 9996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:55 smithi115 bash[45495]: cluster 2022-04-22T20:38:53.876137+0000 mgr.y (mgr.34107) 6595 : cluster [DBG] pgmap v7726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:55 smithi115 bash[45495]: audit 2022-04-22T20:38:54.395492+0000 mon.a (mon.0) 9996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:38:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:57 smithi121 bash[36902]: cluster 2022-04-22T20:38:55.877271+0000 mgr.y (mgr.34107) 6596 : cluster [DBG] pgmap v7727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:57 smithi115 bash[45495]: cluster 2022-04-22T20:38:55.877271+0000 mgr.y (mgr.34107) 6596 : cluster [DBG] pgmap v7727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:57.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:57 smithi115 bash[46934]: cluster 2022-04-22T20:38:55.877271+0000 mgr.y (mgr.34107) 6596 : cluster [DBG] pgmap v7727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:58 smithi121 bash[36902]: audit 2022-04-22T20:38:57.927058+0000 mon.a (mon.0) 9997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:58 smithi121 bash[36902]: audit 2022-04-22T20:38:57.944051+0000 mon.a (mon.0) 9998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:58 smithi115 bash[45495]: audit 2022-04-22T20:38:57.927058+0000 mon.a (mon.0) 9997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:58 smithi115 bash[45495]: audit 2022-04-22T20:38:57.944051+0000 mon.a (mon.0) 9998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:58 smithi115 bash[46934]: audit 2022-04-22T20:38:57.927058+0000 mon.a (mon.0) 9997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:38:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:58 smithi115 bash[46934]: audit 2022-04-22T20:38:57.944051+0000 mon.a (mon.0) 9998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:38:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:38:59 smithi121 bash[36902]: cluster 2022-04-22T20:38:57.878322+0000 mgr.y (mgr.34107) 6597 : cluster [DBG] pgmap v7728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:38:59 smithi115 bash[45495]: cluster 2022-04-22T20:38:57.878322+0000 mgr.y (mgr.34107) 6597 : cluster [DBG] pgmap v7728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:38:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:38:59 smithi115 bash[46934]: cluster 2022-04-22T20:38:57.878322+0000 mgr.y (mgr.34107) 6597 : cluster [DBG] pgmap v7728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:00 smithi121 bash[36902]: audit 2022-04-22T20:38:59.398562+0000 mon.a (mon.0) 9999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:00 smithi121 bash[36902]: cluster 2022-04-22T20:38:59.879424+0000 mgr.y (mgr.34107) 6598 : cluster [DBG] pgmap v7729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:00 smithi115 bash[45495]: audit 2022-04-22T20:38:59.398562+0000 mon.a (mon.0) 9999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:00 smithi115 bash[45495]: cluster 2022-04-22T20:38:59.879424+0000 mgr.y (mgr.34107) 6598 : cluster [DBG] pgmap v7729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:00 smithi115 bash[46934]: audit 2022-04-22T20:38:59.398562+0000 mon.a (mon.0) 9999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:00 smithi115 bash[46934]: cluster 2022-04-22T20:38:59.879424+0000 mgr.y (mgr.34107) 6598 : cluster [DBG] pgmap v7729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:02.359Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:02 smithi121 bash[36902]: cluster 2022-04-22T20:39:01.880550+0000 mgr.y (mgr.34107) 6599 : cluster [DBG] pgmap v7730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:02 smithi121 bash[36902]: audit 2022-04-22T20:39:02.930789+0000 mon.a (mon.0) 10000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:02 smithi115 bash[46934]: cluster 2022-04-22T20:39:01.880550+0000 mgr.y (mgr.34107) 6599 : cluster [DBG] pgmap v7730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:02 smithi115 bash[46934]: audit 2022-04-22T20:39:02.930789+0000 mon.a (mon.0) 10000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:03.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:02 smithi115 bash[45495]: cluster 2022-04-22T20:39:01.880550+0000 mgr.y (mgr.34107) 6599 : cluster [DBG] pgmap v7730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:03.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:02 smithi115 bash[45495]: audit 2022-04-22T20:39:02.930789+0000 mon.a (mon.0) 10000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:04.244 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:03 smithi115 bash[45495]: audit 2022-04-22T20:39:02.946642+0000 mon.a (mon.0) 10001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:04.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:03 smithi115 bash[46934]: audit 2022-04-22T20:39:02.946642+0000 mon.a (mon.0) 10001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:03 smithi121 bash[36902]: audit 2022-04-22T20:39:02.946642+0000 mon.a (mon.0) 10001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:04.237Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:05 smithi121 bash[36902]: cluster 2022-04-22T20:39:03.881718+0000 mgr.y (mgr.34107) 6600 : cluster [DBG] pgmap v7731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:05 smithi121 bash[36902]: audit 2022-04-22T20:39:04.401740+0000 mon.a (mon.0) 10002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:05 smithi115 bash[45495]: cluster 2022-04-22T20:39:03.881718+0000 mgr.y (mgr.34107) 6600 : cluster [DBG] pgmap v7731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:05.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:05 smithi115 bash[45495]: audit 2022-04-22T20:39:04.401740+0000 mon.a (mon.0) 10002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:05 smithi115 bash[46934]: cluster 2022-04-22T20:39:03.881718+0000 mgr.y (mgr.34107) 6600 : cluster [DBG] pgmap v7731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:05.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:05 smithi115 bash[46934]: audit 2022-04-22T20:39:04.401740+0000 mon.a (mon.0) 10002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:06.690 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:39:07.026 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:39:07.026 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:39:07.026 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:39:07.027 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:39:07.028 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:39:07.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:39:07.428 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:39:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:07 smithi121 bash[36902]: cluster 2022-04-22T20:39:05.882874+0000 mgr.y (mgr.34107) 6601 : cluster [DBG] pgmap v7732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:07 smithi115 bash[45495]: cluster 2022-04-22T20:39:05.882874+0000 mgr.y (mgr.34107) 6601 : cluster [DBG] pgmap v7732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:07 smithi115 bash[46934]: cluster 2022-04-22T20:39:05.882874+0000 mgr.y (mgr.34107) 6601 : cluster [DBG] pgmap v7732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:08 smithi121 bash[36902]: audit 2022-04-22T20:39:06.668308+0000 mgr.y (mgr.34107) 6602 : audit [DBG] from='client.52177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:08 smithi121 bash[36902]: audit 2022-04-22T20:39:07.005327+0000 mgr.y (mgr.34107) 6603 : audit [DBG] from='client.52183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:08 smithi121 bash[36902]: audit 2022-04-22T20:39:07.419881+0000 mon.a (mon.0) 10003 : audit [DBG] from='client.? 172.21.15.115:0/569910587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:08 smithi121 bash[36902]: audit 2022-04-22T20:39:07.934717+0000 mon.a (mon.0) 10004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:08 smithi121 bash[36902]: audit 2022-04-22T20:39:07.949231+0000 mon.a (mon.0) 10005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[45495]: audit 2022-04-22T20:39:06.668308+0000 mgr.y (mgr.34107) 6602 : audit [DBG] from='client.52177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[45495]: audit 2022-04-22T20:39:07.005327+0000 mgr.y (mgr.34107) 6603 : audit [DBG] from='client.52183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[45495]: audit 2022-04-22T20:39:07.419881+0000 mon.a (mon.0) 10003 : audit [DBG] from='client.? 172.21.15.115:0/569910587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[45495]: audit 2022-04-22T20:39:07.934717+0000 mon.a (mon.0) 10004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[45495]: audit 2022-04-22T20:39:07.949231+0000 mon.a (mon.0) 10005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[46934]: audit 2022-04-22T20:39:06.668308+0000 mgr.y (mgr.34107) 6602 : audit [DBG] from='client.52177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[46934]: audit 2022-04-22T20:39:07.005327+0000 mgr.y (mgr.34107) 6603 : audit [DBG] from='client.52183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[46934]: audit 2022-04-22T20:39:07.419881+0000 mon.a (mon.0) 10003 : audit [DBG] from='client.? 172.21.15.115:0/569910587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[46934]: audit 2022-04-22T20:39:07.934717+0000 mon.a (mon.0) 10004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:08.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:08 smithi115 bash[46934]: audit 2022-04-22T20:39:07.949231+0000 mon.a (mon.0) 10005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:09 smithi121 bash[36902]: cluster 2022-04-22T20:39:07.884023+0000 mgr.y (mgr.34107) 6604 : cluster [DBG] pgmap v7733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:09 smithi115 bash[45495]: cluster 2022-04-22T20:39:07.884023+0000 mgr.y (mgr.34107) 6604 : cluster [DBG] pgmap v7733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:09 smithi115 bash[46934]: cluster 2022-04-22T20:39:07.884023+0000 mgr.y (mgr.34107) 6604 : cluster [DBG] pgmap v7733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:10 smithi121 bash[36902]: audit 2022-04-22T20:39:09.404863+0000 mon.a (mon.0) 10006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:10 smithi115 bash[45495]: audit 2022-04-22T20:39:09.404863+0000 mon.a (mon.0) 10006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:10.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:10 smithi115 bash[46934]: audit 2022-04-22T20:39:09.404863+0000 mon.a (mon.0) 10006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:11 smithi121 bash[36902]: cluster 2022-04-22T20:39:09.885128+0000 mgr.y (mgr.34107) 6605 : cluster [DBG] pgmap v7734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:11 smithi115 bash[45495]: cluster 2022-04-22T20:39:09.885128+0000 mgr.y (mgr.34107) 6605 : cluster [DBG] pgmap v7734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:11 smithi115 bash[46934]: cluster 2022-04-22T20:39:09.885128+0000 mgr.y (mgr.34107) 6605 : cluster [DBG] pgmap v7734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:12.358Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:13 smithi121 bash[36902]: cluster 2022-04-22T20:39:11.886181+0000 mgr.y (mgr.34107) 6606 : cluster [DBG] pgmap v7735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:13.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:13 smithi121 bash[36902]: audit 2022-04-22T20:39:12.938580+0000 mon.a (mon.0) 10007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:13.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:13 smithi121 bash[36902]: audit 2022-04-22T20:39:12.952094+0000 mon.a (mon.0) 10008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[45495]: cluster 2022-04-22T20:39:11.886181+0000 mgr.y (mgr.34107) 6606 : cluster [DBG] pgmap v7735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[45495]: audit 2022-04-22T20:39:12.938580+0000 mon.a (mon.0) 10007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:13.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[45495]: audit 2022-04-22T20:39:12.952094+0000 mon.a (mon.0) 10008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[46934]: cluster 2022-04-22T20:39:11.886181+0000 mgr.y (mgr.34107) 6606 : cluster [DBG] pgmap v7735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[46934]: audit 2022-04-22T20:39:12.938580+0000 mon.a (mon.0) 10007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:13 smithi115 bash[46934]: audit 2022-04-22T20:39:12.952094+0000 mon.a (mon.0) 10008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:14.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:15 smithi121 bash[36902]: cluster 2022-04-22T20:39:13.887353+0000 mgr.y (mgr.34107) 6607 : cluster [DBG] pgmap v7736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:15 smithi121 bash[36902]: audit 2022-04-22T20:39:14.407946+0000 mon.a (mon.0) 10009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:15 smithi115 bash[45495]: cluster 2022-04-22T20:39:13.887353+0000 mgr.y (mgr.34107) 6607 : cluster [DBG] pgmap v7736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:15.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:15 smithi115 bash[45495]: audit 2022-04-22T20:39:14.407946+0000 mon.a (mon.0) 10009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:15.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:15 smithi115 bash[46934]: cluster 2022-04-22T20:39:13.887353+0000 mgr.y (mgr.34107) 6607 : cluster [DBG] pgmap v7736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:15.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:15 smithi115 bash[46934]: audit 2022-04-22T20:39:14.407946+0000 mon.a (mon.0) 10009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:16 smithi121 bash[36902]: audit 2022-04-22T20:39:15.491678+0000 mon.a (mon.0) 10010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:39:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:16 smithi121 bash[36902]: audit 2022-04-22T20:39:15.491890+0000 mon.a (mon.0) 10011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:39:16.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:16 smithi115 bash[46934]: audit 2022-04-22T20:39:15.491678+0000 mon.a (mon.0) 10010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:39:16.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:16 smithi115 bash[46934]: audit 2022-04-22T20:39:15.491890+0000 mon.a (mon.0) 10011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:39:16.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:16 smithi115 bash[45495]: audit 2022-04-22T20:39:15.491678+0000 mon.a (mon.0) 10010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:39:16.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:16 smithi115 bash[45495]: audit 2022-04-22T20:39:15.491890+0000 mon.a (mon.0) 10011 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:39:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:17 smithi121 bash[36902]: cluster 2022-04-22T20:39:15.888547+0000 mgr.y (mgr.34107) 6608 : cluster [DBG] pgmap v7737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:17 smithi115 bash[45495]: cluster 2022-04-22T20:39:15.888547+0000 mgr.y (mgr.34107) 6608 : cluster [DBG] pgmap v7737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:17 smithi115 bash[46934]: cluster 2022-04-22T20:39:15.888547+0000 mgr.y (mgr.34107) 6608 : cluster [DBG] pgmap v7737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:18 smithi121 bash[36902]: audit 2022-04-22T20:39:17.942295+0000 mon.a (mon.0) 10012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:18 smithi121 bash[36902]: audit 2022-04-22T20:39:17.954986+0000 mon.a (mon.0) 10013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:18 smithi115 bash[45495]: audit 2022-04-22T20:39:17.942295+0000 mon.a (mon.0) 10012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:18 smithi115 bash[45495]: audit 2022-04-22T20:39:17.954986+0000 mon.a (mon.0) 10013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:18 smithi115 bash[46934]: audit 2022-04-22T20:39:17.942295+0000 mon.a (mon.0) 10012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:18 smithi115 bash[46934]: audit 2022-04-22T20:39:17.954986+0000 mon.a (mon.0) 10013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:19 smithi121 bash[36902]: cluster 2022-04-22T20:39:17.889700+0000 mgr.y (mgr.34107) 6609 : cluster [DBG] pgmap v7738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:19.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:19 smithi115 bash[45495]: cluster 2022-04-22T20:39:17.889700+0000 mgr.y (mgr.34107) 6609 : cluster [DBG] pgmap v7738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:19 smithi115 bash[46934]: cluster 2022-04-22T20:39:17.889700+0000 mgr.y (mgr.34107) 6609 : cluster [DBG] pgmap v7738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:20 smithi121 bash[36902]: audit 2022-04-22T20:39:19.411219+0000 mon.a (mon.0) 10014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:20 smithi121 bash[36902]: cluster 2022-04-22T20:39:19.890842+0000 mgr.y (mgr.34107) 6610 : cluster [DBG] pgmap v7739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:20 smithi115 bash[45495]: audit 2022-04-22T20:39:19.411219+0000 mon.a (mon.0) 10014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:20 smithi115 bash[45495]: cluster 2022-04-22T20:39:19.890842+0000 mgr.y (mgr.34107) 6610 : cluster [DBG] pgmap v7739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:20 smithi115 bash[46934]: audit 2022-04-22T20:39:19.411219+0000 mon.a (mon.0) 10014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:20 smithi115 bash[46934]: cluster 2022-04-22T20:39:19.890842+0000 mgr.y (mgr.34107) 6610 : cluster [DBG] pgmap v7739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:22.359Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:22 smithi121 bash[36902]: cluster 2022-04-22T20:39:21.891977+0000 mgr.y (mgr.34107) 6611 : cluster [DBG] pgmap v7740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:23.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:22 smithi115 bash[46934]: cluster 2022-04-22T20:39:21.891977+0000 mgr.y (mgr.34107) 6611 : cluster [DBG] pgmap v7740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:23.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:22 smithi115 bash[45495]: cluster 2022-04-22T20:39:21.891977+0000 mgr.y (mgr.34107) 6611 : cluster [DBG] pgmap v7740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:24.246 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:23 smithi115 bash[46934]: audit 2022-04-22T20:39:22.946393+0000 mon.a (mon.0) 10015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:24.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:23 smithi115 bash[46934]: audit 2022-04-22T20:39:22.957519+0000 mon.a (mon.0) 10016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:24.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:23 smithi115 bash[45495]: audit 2022-04-22T20:39:22.946393+0000 mon.a (mon.0) 10015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:24.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:23 smithi115 bash[45495]: audit 2022-04-22T20:39:22.957519+0000 mon.a (mon.0) 10016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:23 smithi121 bash[36902]: audit 2022-04-22T20:39:22.946393+0000 mon.a (mon.0) 10015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:23 smithi121 bash[36902]: audit 2022-04-22T20:39:22.957519+0000 mon.a (mon.0) 10016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:24.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:25 smithi121 bash[36902]: cluster 2022-04-22T20:39:23.893124+0000 mgr.y (mgr.34107) 6612 : cluster [DBG] pgmap v7741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:25 smithi121 bash[36902]: audit 2022-04-22T20:39:24.414123+0000 mon.a (mon.0) 10017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:25 smithi115 bash[46934]: cluster 2022-04-22T20:39:23.893124+0000 mgr.y (mgr.34107) 6612 : cluster [DBG] pgmap v7741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:25 smithi115 bash[46934]: audit 2022-04-22T20:39:24.414123+0000 mon.a (mon.0) 10017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:25 smithi115 bash[45495]: cluster 2022-04-22T20:39:23.893124+0000 mgr.y (mgr.34107) 6612 : cluster [DBG] pgmap v7741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:25.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:25 smithi115 bash[45495]: audit 2022-04-22T20:39:24.414123+0000 mon.a (mon.0) 10017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:27 smithi121 bash[36902]: cluster 2022-04-22T20:39:25.894299+0000 mgr.y (mgr.34107) 6613 : cluster [DBG] pgmap v7742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:27 smithi115 bash[45495]: cluster 2022-04-22T20:39:25.894299+0000 mgr.y (mgr.34107) 6613 : cluster [DBG] pgmap v7742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:27 smithi115 bash[46934]: cluster 2022-04-22T20:39:25.894299+0000 mgr.y (mgr.34107) 6613 : cluster [DBG] pgmap v7742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:28 smithi121 bash[36902]: audit 2022-04-22T20:39:27.950273+0000 mon.a (mon.0) 10018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:28 smithi121 bash[36902]: audit 2022-04-22T20:39:27.960055+0000 mon.a (mon.0) 10019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:28 smithi115 bash[46934]: audit 2022-04-22T20:39:27.950273+0000 mon.a (mon.0) 10018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:28 smithi115 bash[46934]: audit 2022-04-22T20:39:27.960055+0000 mon.a (mon.0) 10019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:28 smithi115 bash[45495]: audit 2022-04-22T20:39:27.950273+0000 mon.a (mon.0) 10018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:28.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:28 smithi115 bash[45495]: audit 2022-04-22T20:39:27.960055+0000 mon.a (mon.0) 10019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:29 smithi121 bash[36902]: cluster 2022-04-22T20:39:27.895378+0000 mgr.y (mgr.34107) 6614 : cluster [DBG] pgmap v7743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:29 smithi115 bash[46934]: cluster 2022-04-22T20:39:27.895378+0000 mgr.y (mgr.34107) 6614 : cluster [DBG] pgmap v7743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:29 smithi115 bash[45495]: cluster 2022-04-22T20:39:27.895378+0000 mgr.y (mgr.34107) 6614 : cluster [DBG] pgmap v7743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:30 smithi121 bash[36902]: audit 2022-04-22T20:39:29.417278+0000 mon.a (mon.0) 10020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:30 smithi115 bash[46934]: audit 2022-04-22T20:39:29.417278+0000 mon.a (mon.0) 10020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:30 smithi115 bash[45495]: audit 2022-04-22T20:39:29.417278+0000 mon.a (mon.0) 10020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:31 smithi121 bash[36902]: cluster 2022-04-22T20:39:29.896455+0000 mgr.y (mgr.34107) 6615 : cluster [DBG] pgmap v7744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:31 smithi115 bash[46934]: cluster 2022-04-22T20:39:29.896455+0000 mgr.y (mgr.34107) 6615 : cluster [DBG] pgmap v7744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:31.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:31 smithi115 bash[45495]: cluster 2022-04-22T20:39:29.896455+0000 mgr.y (mgr.34107) 6615 : cluster [DBG] pgmap v7744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:32.359Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:33 smithi121 bash[36902]: cluster 2022-04-22T20:39:31.897590+0000 mgr.y (mgr.34107) 6616 : cluster [DBG] pgmap v7745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:33 smithi121 bash[36902]: audit 2022-04-22T20:39:32.953943+0000 mon.a (mon.0) 10021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:33 smithi121 bash[36902]: audit 2022-04-22T20:39:32.962871+0000 mon.a (mon.0) 10022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[45495]: cluster 2022-04-22T20:39:31.897590+0000 mgr.y (mgr.34107) 6616 : cluster [DBG] pgmap v7745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[45495]: audit 2022-04-22T20:39:32.953943+0000 mon.a (mon.0) 10021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[45495]: audit 2022-04-22T20:39:32.962871+0000 mon.a (mon.0) 10022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[46934]: cluster 2022-04-22T20:39:31.897590+0000 mgr.y (mgr.34107) 6616 : cluster [DBG] pgmap v7745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[46934]: audit 2022-04-22T20:39:32.953943+0000 mon.a (mon.0) 10021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:33.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:33 smithi115 bash[46934]: audit 2022-04-22T20:39:32.962871+0000 mon.a (mon.0) 10022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:34.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:35 smithi121 bash[36902]: cluster 2022-04-22T20:39:33.898804+0000 mgr.y (mgr.34107) 6617 : cluster [DBG] pgmap v7746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:35 smithi121 bash[36902]: audit 2022-04-22T20:39:34.420190+0000 mon.a (mon.0) 10023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:35 smithi115 bash[45495]: cluster 2022-04-22T20:39:33.898804+0000 mgr.y (mgr.34107) 6617 : cluster [DBG] pgmap v7746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:35 smithi115 bash[45495]: audit 2022-04-22T20:39:34.420190+0000 mon.a (mon.0) 10023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:35 smithi115 bash[46934]: cluster 2022-04-22T20:39:33.898804+0000 mgr.y (mgr.34107) 6617 : cluster [DBG] pgmap v7746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:35 smithi115 bash[46934]: audit 2022-04-22T20:39:34.420190+0000 mon.a (mon.0) 10023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:36 smithi121 bash[36902]: cluster 2022-04-22T20:39:35.899927+0000 mgr.y (mgr.34107) 6618 : cluster [DBG] pgmap v7747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:36.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:36 smithi115 bash[45495]: cluster 2022-04-22T20:39:35.899927+0000 mgr.y (mgr.34107) 6618 : cluster [DBG] pgmap v7747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:36 smithi115 bash[46934]: cluster 2022-04-22T20:39:35.899927+0000 mgr.y (mgr.34107) 6618 : cluster [DBG] pgmap v7747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:37.798 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:39:38.147 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:39:38.148 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:39:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:39:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:39:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:39:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:39:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:39:38.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:38.557 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:39:38.557 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:39:38.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:38.557 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:39:38.557 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:39:38.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:39:38.558 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:39:38.558 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:39:38.558 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:39:38.559 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:39:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: audit 2022-04-22T20:39:37.775962+0000 mgr.y (mgr.34107) 6619 : audit [DBG] from='client.52195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: cluster 2022-04-22T20:39:37.900915+0000 mgr.y (mgr.34107) 6620 : cluster [DBG] pgmap v7748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: audit 2022-04-22T20:39:37.959205+0000 mon.a (mon.0) 10024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: audit 2022-04-22T20:39:37.965263+0000 mon.a (mon.0) 10025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: audit 2022-04-22T20:39:38.120795+0000 mgr.y (mgr.34107) 6621 : audit [DBG] from='client.42270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:38 smithi121 bash[36902]: audit 2022-04-22T20:39:38.548986+0000 mon.a (mon.0) 10026 : audit [DBG] from='client.? 172.21.15.115:0/1601881104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:39.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: audit 2022-04-22T20:39:37.775962+0000 mgr.y (mgr.34107) 6619 : audit [DBG] from='client.52195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: cluster 2022-04-22T20:39:37.900915+0000 mgr.y (mgr.34107) 6620 : cluster [DBG] pgmap v7748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: audit 2022-04-22T20:39:37.959205+0000 mon.a (mon.0) 10024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: audit 2022-04-22T20:39:37.965263+0000 mon.a (mon.0) 10025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: audit 2022-04-22T20:39:38.120795+0000 mgr.y (mgr.34107) 6621 : audit [DBG] from='client.42270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[45495]: audit 2022-04-22T20:39:38.548986+0000 mon.a (mon.0) 10026 : audit [DBG] from='client.? 172.21.15.115:0/1601881104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: audit 2022-04-22T20:39:37.775962+0000 mgr.y (mgr.34107) 6619 : audit [DBG] from='client.52195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: cluster 2022-04-22T20:39:37.900915+0000 mgr.y (mgr.34107) 6620 : cluster [DBG] pgmap v7748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: audit 2022-04-22T20:39:37.959205+0000 mon.a (mon.0) 10024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: audit 2022-04-22T20:39:37.965263+0000 mon.a (mon.0) 10025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: audit 2022-04-22T20:39:38.120795+0000 mgr.y (mgr.34107) 6621 : audit [DBG] from='client.42270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:39:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:38 smithi115 bash[46934]: audit 2022-04-22T20:39:38.548986+0000 mon.a (mon.0) 10026 : audit [DBG] from='client.? 172.21.15.115:0/1601881104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:39:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:40 smithi121 bash[36902]: audit 2022-04-22T20:39:39.423604+0000 mon.a (mon.0) 10027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:40 smithi115 bash[45495]: audit 2022-04-22T20:39:39.423604+0000 mon.a (mon.0) 10027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:40 smithi115 bash[46934]: audit 2022-04-22T20:39:39.423604+0000 mon.a (mon.0) 10027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:41 smithi121 bash[36902]: cluster 2022-04-22T20:39:39.901966+0000 mgr.y (mgr.34107) 6622 : cluster [DBG] pgmap v7749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:41.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:41 smithi115 bash[45495]: cluster 2022-04-22T20:39:39.901966+0000 mgr.y (mgr.34107) 6622 : cluster [DBG] pgmap v7749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:41.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:41 smithi115 bash[46934]: cluster 2022-04-22T20:39:39.901966+0000 mgr.y (mgr.34107) 6622 : cluster [DBG] pgmap v7749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:42.360Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:43 smithi121 bash[36902]: cluster 2022-04-22T20:39:41.903168+0000 mgr.y (mgr.34107) 6623 : cluster [DBG] pgmap v7750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:43 smithi121 bash[36902]: audit 2022-04-22T20:39:42.961501+0000 mon.a (mon.0) 10028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:43 smithi121 bash[36902]: audit 2022-04-22T20:39:42.967341+0000 mon.a (mon.0) 10029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[45495]: cluster 2022-04-22T20:39:41.903168+0000 mgr.y (mgr.34107) 6623 : cluster [DBG] pgmap v7750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:43.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[45495]: audit 2022-04-22T20:39:42.961501+0000 mon.a (mon.0) 10028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:43.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[45495]: audit 2022-04-22T20:39:42.967341+0000 mon.a (mon.0) 10029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[46934]: cluster 2022-04-22T20:39:41.903168+0000 mgr.y (mgr.34107) 6623 : cluster [DBG] pgmap v7750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[46934]: audit 2022-04-22T20:39:42.961501+0000 mon.a (mon.0) 10028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:43 smithi115 bash[46934]: audit 2022-04-22T20:39:42.967341+0000 mon.a (mon.0) 10029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:44.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:45 smithi121 bash[36902]: cluster 2022-04-22T20:39:43.904277+0000 mgr.y (mgr.34107) 6624 : cluster [DBG] pgmap v7751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:45 smithi121 bash[36902]: audit 2022-04-22T20:39:44.426599+0000 mon.a (mon.0) 10030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:45 smithi115 bash[45495]: cluster 2022-04-22T20:39:43.904277+0000 mgr.y (mgr.34107) 6624 : cluster [DBG] pgmap v7751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:45 smithi115 bash[45495]: audit 2022-04-22T20:39:44.426599+0000 mon.a (mon.0) 10030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:45 smithi115 bash[46934]: cluster 2022-04-22T20:39:43.904277+0000 mgr.y (mgr.34107) 6624 : cluster [DBG] pgmap v7751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:45 smithi115 bash[46934]: audit 2022-04-22T20:39:44.426599+0000 mon.a (mon.0) 10030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:47 smithi121 bash[36902]: cluster 2022-04-22T20:39:45.905324+0000 mgr.y (mgr.34107) 6625 : cluster [DBG] pgmap v7752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:47 smithi115 bash[45495]: cluster 2022-04-22T20:39:45.905324+0000 mgr.y (mgr.34107) 6625 : cluster [DBG] pgmap v7752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:47 smithi115 bash[46934]: cluster 2022-04-22T20:39:45.905324+0000 mgr.y (mgr.34107) 6625 : cluster [DBG] pgmap v7752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:48.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:48 smithi115 bash[45495]: audit 2022-04-22T20:39:47.965177+0000 mon.a (mon.0) 10031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:48.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:48 smithi115 bash[45495]: audit 2022-04-22T20:39:47.970163+0000 mon.a (mon.0) 10032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:48.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:48 smithi115 bash[46934]: audit 2022-04-22T20:39:47.965177+0000 mon.a (mon.0) 10031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:48.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:48 smithi115 bash[46934]: audit 2022-04-22T20:39:47.970163+0000 mon.a (mon.0) 10032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:48.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:48 smithi121 bash[36902]: audit 2022-04-22T20:39:47.965177+0000 mon.a (mon.0) 10031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:48.596 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:48 smithi121 bash[36902]: audit 2022-04-22T20:39:47.970163+0000 mon.a (mon.0) 10032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:49 smithi121 bash[36902]: cluster 2022-04-22T20:39:47.906450+0000 mgr.y (mgr.34107) 6626 : cluster [DBG] pgmap v7753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:49 smithi121 bash[36902]: audit 2022-04-22T20:39:48.563680+0000 mon.a (mon.0) 10033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:39:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:49 smithi121 bash[36902]: audit 2022-04-22T20:39:48.566817+0000 mon.a (mon.0) 10034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:39:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:49 smithi121 bash[36902]: audit 2022-04-22T20:39:48.568653+0000 mon.a (mon.0) 10035 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:39:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:49 smithi121 bash[36902]: audit 2022-04-22T20:39:48.924014+0000 mon.a (mon.0) 10036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:39:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[45495]: cluster 2022-04-22T20:39:47.906450+0000 mgr.y (mgr.34107) 6626 : cluster [DBG] pgmap v7753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[45495]: audit 2022-04-22T20:39:48.563680+0000 mon.a (mon.0) 10033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:39:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[45495]: audit 2022-04-22T20:39:48.566817+0000 mon.a (mon.0) 10034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:39:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[45495]: audit 2022-04-22T20:39:48.568653+0000 mon.a (mon.0) 10035 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[45495]: audit 2022-04-22T20:39:48.924014+0000 mon.a (mon.0) 10036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[46934]: cluster 2022-04-22T20:39:47.906450+0000 mgr.y (mgr.34107) 6626 : cluster [DBG] pgmap v7753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[46934]: audit 2022-04-22T20:39:48.563680+0000 mon.a (mon.0) 10033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[46934]: audit 2022-04-22T20:39:48.566817+0000 mon.a (mon.0) 10034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[46934]: audit 2022-04-22T20:39:48.568653+0000 mon.a (mon.0) 10035 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:39:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:49 smithi115 bash[46934]: audit 2022-04-22T20:39:48.924014+0000 mon.a (mon.0) 10036 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:39:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:50 smithi121 bash[36902]: audit 2022-04-22T20:39:49.429648+0000 mon.a (mon.0) 10037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:50 smithi115 bash[45495]: audit 2022-04-22T20:39:49.429648+0000 mon.a (mon.0) 10037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:50 smithi115 bash[46934]: audit 2022-04-22T20:39:49.429648+0000 mon.a (mon.0) 10037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:51 smithi121 bash[36902]: cluster 2022-04-22T20:39:49.907567+0000 mgr.y (mgr.34107) 6627 : cluster [DBG] pgmap v7754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:51 smithi115 bash[45495]: cluster 2022-04-22T20:39:49.907567+0000 mgr.y (mgr.34107) 6627 : cluster [DBG] pgmap v7754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:51 smithi115 bash[46934]: cluster 2022-04-22T20:39:49.907567+0000 mgr.y (mgr.34107) 6627 : cluster [DBG] pgmap v7754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:52 smithi121 bash[36902]: cluster 2022-04-22T20:39:51.908571+0000 mgr.y (mgr.34107) 6628 : cluster [DBG] pgmap v7755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:52 smithi115 bash[45495]: cluster 2022-04-22T20:39:51.908571+0000 mgr.y (mgr.34107) 6628 : cluster [DBG] pgmap v7755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:52.360Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:52.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:52 smithi115 bash[46934]: cluster 2022-04-22T20:39:51.908571+0000 mgr.y (mgr.34107) 6628 : cluster [DBG] pgmap v7755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:53 smithi121 bash[36902]: audit 2022-04-22T20:39:52.968852+0000 mon.a (mon.0) 10038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:53 smithi121 bash[36902]: audit 2022-04-22T20:39:52.973009+0000 mon.a (mon.0) 10039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:53 smithi115 bash[45495]: audit 2022-04-22T20:39:52.968852+0000 mon.a (mon.0) 10038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:53 smithi115 bash[45495]: audit 2022-04-22T20:39:52.973009+0000 mon.a (mon.0) 10039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:53 smithi115 bash[46934]: audit 2022-04-22T20:39:52.968852+0000 mon.a (mon.0) 10038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:53 smithi115 bash[46934]: audit 2022-04-22T20:39:52.973009+0000 mon.a (mon.0) 10039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:39:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:39:54.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:39:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:55 smithi121 bash[36902]: cluster 2022-04-22T20:39:53.909691+0000 mgr.y (mgr.34107) 6629 : cluster [DBG] pgmap v7756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:55 smithi121 bash[36902]: audit 2022-04-22T20:39:54.432546+0000 mon.a (mon.0) 10040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:55 smithi115 bash[45495]: cluster 2022-04-22T20:39:53.909691+0000 mgr.y (mgr.34107) 6629 : cluster [DBG] pgmap v7756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:55 smithi115 bash[45495]: audit 2022-04-22T20:39:54.432546+0000 mon.a (mon.0) 10040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:55 smithi115 bash[46934]: cluster 2022-04-22T20:39:53.909691+0000 mgr.y (mgr.34107) 6629 : cluster [DBG] pgmap v7756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:55 smithi115 bash[46934]: audit 2022-04-22T20:39:54.432546+0000 mon.a (mon.0) 10040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:39:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:57 smithi121 bash[36902]: cluster 2022-04-22T20:39:55.910796+0000 mgr.y (mgr.34107) 6630 : cluster [DBG] pgmap v7757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:57.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:57 smithi115 bash[46934]: cluster 2022-04-22T20:39:55.910796+0000 mgr.y (mgr.34107) 6630 : cluster [DBG] pgmap v7757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:57 smithi115 bash[45495]: cluster 2022-04-22T20:39:55.910796+0000 mgr.y (mgr.34107) 6630 : cluster [DBG] pgmap v7757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:58 smithi121 bash[36902]: audit 2022-04-22T20:39:57.972378+0000 mon.a (mon.0) 10041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:58 smithi121 bash[36902]: audit 2022-04-22T20:39:57.976205+0000 mon.a (mon.0) 10042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:58 smithi115 bash[45495]: audit 2022-04-22T20:39:57.972378+0000 mon.a (mon.0) 10041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:58 smithi115 bash[45495]: audit 2022-04-22T20:39:57.976205+0000 mon.a (mon.0) 10042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:58.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:58 smithi115 bash[46934]: audit 2022-04-22T20:39:57.972378+0000 mon.a (mon.0) 10041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:39:58.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:58 smithi115 bash[46934]: audit 2022-04-22T20:39:57.976205+0000 mon.a (mon.0) 10042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:39:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:39:59 smithi121 bash[36902]: cluster 2022-04-22T20:39:57.911938+0000 mgr.y (mgr.34107) 6631 : cluster [DBG] pgmap v7758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:39:59 smithi115 bash[45495]: cluster 2022-04-22T20:39:57.911938+0000 mgr.y (mgr.34107) 6631 : cluster [DBG] pgmap v7758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:39:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:39:59 smithi115 bash[46934]: cluster 2022-04-22T20:39:57.911938+0000 mgr.y (mgr.34107) 6631 : cluster [DBG] pgmap v7758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:00 smithi121 bash[36902]: audit 2022-04-22T20:39:59.435621+0000 mon.a (mon.0) 10043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:00 smithi121 bash[36902]: cluster 2022-04-22T20:40:00.000172+0000 mon.a (mon.0) 10044 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:40:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:00 smithi115 bash[45495]: audit 2022-04-22T20:39:59.435621+0000 mon.a (mon.0) 10043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:00 smithi115 bash[45495]: cluster 2022-04-22T20:40:00.000172+0000 mon.a (mon.0) 10044 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:40:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:00 smithi115 bash[46934]: audit 2022-04-22T20:39:59.435621+0000 mon.a (mon.0) 10043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:00 smithi115 bash[46934]: cluster 2022-04-22T20:40:00.000172+0000 mon.a (mon.0) 10044 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:40:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:01 smithi121 bash[36902]: cluster 2022-04-22T20:39:59.913067+0000 mgr.y (mgr.34107) 6632 : cluster [DBG] pgmap v7759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:01 smithi115 bash[45495]: cluster 2022-04-22T20:39:59.913067+0000 mgr.y (mgr.34107) 6632 : cluster [DBG] pgmap v7759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:01 smithi115 bash[46934]: cluster 2022-04-22T20:39:59.913067+0000 mgr.y (mgr.34107) 6632 : cluster [DBG] pgmap v7759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:02.359Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:03 smithi121 bash[36902]: cluster 2022-04-22T20:40:01.913991+0000 mgr.y (mgr.34107) 6633 : cluster [DBG] pgmap v7760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:03 smithi121 bash[36902]: audit 2022-04-22T20:40:02.976154+0000 mon.a (mon.0) 10045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:03 smithi121 bash[36902]: audit 2022-04-22T20:40:02.978960+0000 mon.a (mon.0) 10046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[45495]: cluster 2022-04-22T20:40:01.913991+0000 mgr.y (mgr.34107) 6633 : cluster [DBG] pgmap v7760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[45495]: audit 2022-04-22T20:40:02.976154+0000 mon.a (mon.0) 10045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[45495]: audit 2022-04-22T20:40:02.978960+0000 mon.a (mon.0) 10046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:03.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[46934]: cluster 2022-04-22T20:40:01.913991+0000 mgr.y (mgr.34107) 6633 : cluster [DBG] pgmap v7760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[46934]: audit 2022-04-22T20:40:02.976154+0000 mon.a (mon.0) 10045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:03 smithi115 bash[46934]: audit 2022-04-22T20:40:02.978960+0000 mon.a (mon.0) 10046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:04.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:05 smithi121 bash[36902]: cluster 2022-04-22T20:40:03.915102+0000 mgr.y (mgr.34107) 6634 : cluster [DBG] pgmap v7761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:05 smithi121 bash[36902]: audit 2022-04-22T20:40:04.438609+0000 mon.a (mon.0) 10047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:05 smithi115 bash[46934]: cluster 2022-04-22T20:40:03.915102+0000 mgr.y (mgr.34107) 6634 : cluster [DBG] pgmap v7761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:05 smithi115 bash[46934]: audit 2022-04-22T20:40:04.438609+0000 mon.a (mon.0) 10047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:05 smithi115 bash[45495]: cluster 2022-04-22T20:40:03.915102+0000 mgr.y (mgr.34107) 6634 : cluster [DBG] pgmap v7761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:05 smithi115 bash[45495]: audit 2022-04-22T20:40:04.438609+0000 mon.a (mon.0) 10047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:07 smithi121 bash[36902]: cluster 2022-04-22T20:40:05.916251+0000 mgr.y (mgr.34107) 6635 : cluster [DBG] pgmap v7762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:07 smithi115 bash[45495]: cluster 2022-04-22T20:40:05.916251+0000 mgr.y (mgr.34107) 6635 : cluster [DBG] pgmap v7762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:07 smithi115 bash[46934]: cluster 2022-04-22T20:40:05.916251+0000 mgr.y (mgr.34107) 6635 : cluster [DBG] pgmap v7762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:08 smithi121 bash[36902]: audit 2022-04-22T20:40:07.979876+0000 mon.a (mon.0) 10048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:08 smithi121 bash[36902]: audit 2022-04-22T20:40:07.982068+0000 mon.a (mon.0) 10049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:08 smithi115 bash[46934]: audit 2022-04-22T20:40:07.979876+0000 mon.a (mon.0) 10048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:08 smithi115 bash[46934]: audit 2022-04-22T20:40:07.982068+0000 mon.a (mon.0) 10049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:08 smithi115 bash[45495]: audit 2022-04-22T20:40:07.979876+0000 mon.a (mon.0) 10048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:08 smithi115 bash[45495]: audit 2022-04-22T20:40:07.982068+0000 mon.a (mon.0) 10049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:08.930 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:40:09.281 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:40:09.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:40:09.283 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:40:09.688 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:40:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:40:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:40:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:40:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:40:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:40:09.690 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:40:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:09 smithi121 bash[36902]: cluster 2022-04-22T20:40:07.917482+0000 mgr.y (mgr.34107) 6636 : cluster [DBG] pgmap v7763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:09 smithi115 bash[45495]: cluster 2022-04-22T20:40:07.917482+0000 mgr.y (mgr.34107) 6636 : cluster [DBG] pgmap v7763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:09 smithi115 bash[46934]: cluster 2022-04-22T20:40:07.917482+0000 mgr.y (mgr.34107) 6636 : cluster [DBG] pgmap v7763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:10 smithi121 bash[36902]: audit 2022-04-22T20:40:08.907617+0000 mgr.y (mgr.34107) 6637 : audit [DBG] from='client.42282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:10 smithi121 bash[36902]: audit 2022-04-22T20:40:09.255022+0000 mgr.y (mgr.34107) 6638 : audit [DBG] from='client.42288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:10 smithi121 bash[36902]: audit 2022-04-22T20:40:09.441581+0000 mon.a (mon.0) 10050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:10 smithi121 bash[36902]: audit 2022-04-22T20:40:09.681269+0000 mon.c (mon.1) 256 : audit [DBG] from='client.? 172.21.15.115:0/978662935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[45495]: audit 2022-04-22T20:40:08.907617+0000 mgr.y (mgr.34107) 6637 : audit [DBG] from='client.42282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[45495]: audit 2022-04-22T20:40:09.255022+0000 mgr.y (mgr.34107) 6638 : audit [DBG] from='client.42288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[45495]: audit 2022-04-22T20:40:09.441581+0000 mon.a (mon.0) 10050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:10.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[45495]: audit 2022-04-22T20:40:09.681269+0000 mon.c (mon.1) 256 : audit [DBG] from='client.? 172.21.15.115:0/978662935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:10.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[46934]: audit 2022-04-22T20:40:08.907617+0000 mgr.y (mgr.34107) 6637 : audit [DBG] from='client.42282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[46934]: audit 2022-04-22T20:40:09.255022+0000 mgr.y (mgr.34107) 6638 : audit [DBG] from='client.42288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:10.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[46934]: audit 2022-04-22T20:40:09.441581+0000 mon.a (mon.0) 10050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:10.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:10 smithi115 bash[46934]: audit 2022-04-22T20:40:09.681269+0000 mon.c (mon.1) 256 : audit [DBG] from='client.? 172.21.15.115:0/978662935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:11 smithi121 bash[36902]: cluster 2022-04-22T20:40:09.918599+0000 mgr.y (mgr.34107) 6639 : cluster [DBG] pgmap v7764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:11 smithi115 bash[45495]: cluster 2022-04-22T20:40:09.918599+0000 mgr.y (mgr.34107) 6639 : cluster [DBG] pgmap v7764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:11 smithi115 bash[46934]: cluster 2022-04-22T20:40:09.918599+0000 mgr.y (mgr.34107) 6639 : cluster [DBG] pgmap v7764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:12.358Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:13 smithi121 bash[36902]: cluster 2022-04-22T20:40:11.919832+0000 mgr.y (mgr.34107) 6640 : cluster [DBG] pgmap v7765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:13 smithi121 bash[36902]: audit 2022-04-22T20:40:12.984163+0000 mon.a (mon.0) 10051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:13 smithi121 bash[36902]: audit 2022-04-22T20:40:12.985140+0000 mon.a (mon.0) 10052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[45495]: cluster 2022-04-22T20:40:11.919832+0000 mgr.y (mgr.34107) 6640 : cluster [DBG] pgmap v7765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[45495]: audit 2022-04-22T20:40:12.984163+0000 mon.a (mon.0) 10051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[45495]: audit 2022-04-22T20:40:12.985140+0000 mon.a (mon.0) 10052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[46934]: cluster 2022-04-22T20:40:11.919832+0000 mgr.y (mgr.34107) 6640 : cluster [DBG] pgmap v7765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[46934]: audit 2022-04-22T20:40:12.984163+0000 mon.a (mon.0) 10051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:13 smithi115 bash[46934]: audit 2022-04-22T20:40:12.985140+0000 mon.a (mon.0) 10052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:14.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:15 smithi121 bash[36902]: cluster 2022-04-22T20:40:13.920944+0000 mgr.y (mgr.34107) 6641 : cluster [DBG] pgmap v7766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:15 smithi121 bash[36902]: audit 2022-04-22T20:40:14.444513+0000 mon.a (mon.0) 10053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:15 smithi115 bash[45495]: cluster 2022-04-22T20:40:13.920944+0000 mgr.y (mgr.34107) 6641 : cluster [DBG] pgmap v7766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:15 smithi115 bash[45495]: audit 2022-04-22T20:40:14.444513+0000 mon.a (mon.0) 10053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:15 smithi115 bash[46934]: cluster 2022-04-22T20:40:13.920944+0000 mgr.y (mgr.34107) 6641 : cluster [DBG] pgmap v7766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:15 smithi115 bash[46934]: audit 2022-04-22T20:40:14.444513+0000 mon.a (mon.0) 10053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:16 smithi121 bash[36902]: audit 2022-04-22T20:40:15.501863+0000 mon.a (mon.0) 10054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:40:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:16 smithi121 bash[36902]: audit 2022-04-22T20:40:15.502677+0000 mon.a (mon.0) 10055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:40:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:16 smithi115 bash[45495]: audit 2022-04-22T20:40:15.501863+0000 mon.a (mon.0) 10054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:40:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:16 smithi115 bash[45495]: audit 2022-04-22T20:40:15.502677+0000 mon.a (mon.0) 10055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:40:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:16 smithi115 bash[46934]: audit 2022-04-22T20:40:15.501863+0000 mon.a (mon.0) 10054 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:40:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:16 smithi115 bash[46934]: audit 2022-04-22T20:40:15.502677+0000 mon.a (mon.0) 10055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:40:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:17 smithi121 bash[36902]: cluster 2022-04-22T20:40:15.922060+0000 mgr.y (mgr.34107) 6642 : cluster [DBG] pgmap v7767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:17 smithi115 bash[46934]: cluster 2022-04-22T20:40:15.922060+0000 mgr.y (mgr.34107) 6642 : cluster [DBG] pgmap v7767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:17 smithi115 bash[45495]: cluster 2022-04-22T20:40:15.922060+0000 mgr.y (mgr.34107) 6642 : cluster [DBG] pgmap v7767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:18 smithi121 bash[36902]: cluster 2022-04-22T20:40:17.923191+0000 mgr.y (mgr.34107) 6643 : cluster [DBG] pgmap v7768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:18 smithi121 bash[36902]: audit 2022-04-22T20:40:17.988597+0000 mon.a (mon.0) 10056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:18 smithi121 bash[36902]: audit 2022-04-22T20:40:17.989006+0000 mon.a (mon.0) 10057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[45495]: cluster 2022-04-22T20:40:17.923191+0000 mgr.y (mgr.34107) 6643 : cluster [DBG] pgmap v7768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[45495]: audit 2022-04-22T20:40:17.988597+0000 mon.a (mon.0) 10056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[45495]: audit 2022-04-22T20:40:17.989006+0000 mon.a (mon.0) 10057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[46934]: cluster 2022-04-22T20:40:17.923191+0000 mgr.y (mgr.34107) 6643 : cluster [DBG] pgmap v7768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[46934]: audit 2022-04-22T20:40:17.988597+0000 mon.a (mon.0) 10056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:18.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:18 smithi115 bash[46934]: audit 2022-04-22T20:40:17.989006+0000 mon.a (mon.0) 10057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:20 smithi121 bash[36902]: audit 2022-04-22T20:40:19.447674+0000 mon.a (mon.0) 10058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:20 smithi115 bash[45495]: audit 2022-04-22T20:40:19.447674+0000 mon.a (mon.0) 10058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:20 smithi115 bash[46934]: audit 2022-04-22T20:40:19.447674+0000 mon.a (mon.0) 10058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:21 smithi121 bash[36902]: cluster 2022-04-22T20:40:19.924425+0000 mgr.y (mgr.34107) 6644 : cluster [DBG] pgmap v7769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:21 smithi115 bash[45495]: cluster 2022-04-22T20:40:19.924425+0000 mgr.y (mgr.34107) 6644 : cluster [DBG] pgmap v7769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:21 smithi115 bash[46934]: cluster 2022-04-22T20:40:19.924425+0000 mgr.y (mgr.34107) 6644 : cluster [DBG] pgmap v7769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:22.360Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:23 smithi121 bash[36902]: cluster 2022-04-22T20:40:21.925692+0000 mgr.y (mgr.34107) 6645 : cluster [DBG] pgmap v7770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:23 smithi121 bash[36902]: audit 2022-04-22T20:40:22.992126+0000 mon.a (mon.0) 10059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:23 smithi121 bash[36902]: audit 2022-04-22T20:40:22.992666+0000 mon.a (mon.0) 10060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[45495]: cluster 2022-04-22T20:40:21.925692+0000 mgr.y (mgr.34107) 6645 : cluster [DBG] pgmap v7770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[45495]: audit 2022-04-22T20:40:22.992126+0000 mon.a (mon.0) 10059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[45495]: audit 2022-04-22T20:40:22.992666+0000 mon.a (mon.0) 10060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[46934]: cluster 2022-04-22T20:40:21.925692+0000 mgr.y (mgr.34107) 6645 : cluster [DBG] pgmap v7770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[46934]: audit 2022-04-22T20:40:22.992126+0000 mon.a (mon.0) 10059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:23 smithi115 bash[46934]: audit 2022-04-22T20:40:22.992666+0000 mon.a (mon.0) 10060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:24.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:25 smithi121 bash[36902]: cluster 2022-04-22T20:40:23.926783+0000 mgr.y (mgr.34107) 6646 : cluster [DBG] pgmap v7771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:25 smithi121 bash[36902]: audit 2022-04-22T20:40:24.450733+0000 mon.a (mon.0) 10061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:25 smithi115 bash[45495]: cluster 2022-04-22T20:40:23.926783+0000 mgr.y (mgr.34107) 6646 : cluster [DBG] pgmap v7771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:25 smithi115 bash[45495]: audit 2022-04-22T20:40:24.450733+0000 mon.a (mon.0) 10061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:25 smithi115 bash[46934]: cluster 2022-04-22T20:40:23.926783+0000 mgr.y (mgr.34107) 6646 : cluster [DBG] pgmap v7771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:25 smithi115 bash[46934]: audit 2022-04-22T20:40:24.450733+0000 mon.a (mon.0) 10061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:27 smithi121 bash[36902]: cluster 2022-04-22T20:40:25.927864+0000 mgr.y (mgr.34107) 6647 : cluster [DBG] pgmap v7772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:27 smithi115 bash[46934]: cluster 2022-04-22T20:40:25.927864+0000 mgr.y (mgr.34107) 6647 : cluster [DBG] pgmap v7772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:27 smithi115 bash[45495]: cluster 2022-04-22T20:40:25.927864+0000 mgr.y (mgr.34107) 6647 : cluster [DBG] pgmap v7772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:28 smithi121 bash[36902]: audit 2022-04-22T20:40:27.995441+0000 mon.a (mon.0) 10062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:28 smithi121 bash[36902]: audit 2022-04-22T20:40:27.996811+0000 mon.a (mon.0) 10063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:28 smithi115 bash[45495]: audit 2022-04-22T20:40:27.995441+0000 mon.a (mon.0) 10062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:28 smithi115 bash[45495]: audit 2022-04-22T20:40:27.996811+0000 mon.a (mon.0) 10063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:28 smithi115 bash[46934]: audit 2022-04-22T20:40:27.995441+0000 mon.a (mon.0) 10062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:28 smithi115 bash[46934]: audit 2022-04-22T20:40:27.996811+0000 mon.a (mon.0) 10063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:29 smithi121 bash[36902]: cluster 2022-04-22T20:40:27.928916+0000 mgr.y (mgr.34107) 6648 : cluster [DBG] pgmap v7773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:29 smithi115 bash[46934]: cluster 2022-04-22T20:40:27.928916+0000 mgr.y (mgr.34107) 6648 : cluster [DBG] pgmap v7773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:29.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:29 smithi115 bash[45495]: cluster 2022-04-22T20:40:27.928916+0000 mgr.y (mgr.34107) 6648 : cluster [DBG] pgmap v7773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:30 smithi121 bash[36902]: audit 2022-04-22T20:40:29.453637+0000 mon.a (mon.0) 10064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:30 smithi115 bash[45495]: audit 2022-04-22T20:40:29.453637+0000 mon.a (mon.0) 10064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:30 smithi115 bash[46934]: audit 2022-04-22T20:40:29.453637+0000 mon.a (mon.0) 10064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:31 smithi121 bash[36902]: cluster 2022-04-22T20:40:29.929977+0000 mgr.y (mgr.34107) 6649 : cluster [DBG] pgmap v7774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:31 smithi115 bash[45495]: cluster 2022-04-22T20:40:29.929977+0000 mgr.y (mgr.34107) 6649 : cluster [DBG] pgmap v7774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:31 smithi115 bash[46934]: cluster 2022-04-22T20:40:29.929977+0000 mgr.y (mgr.34107) 6649 : cluster [DBG] pgmap v7774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:32 smithi121 bash[36902]: cluster 2022-04-22T20:40:31.931105+0000 mgr.y (mgr.34107) 6650 : cluster [DBG] pgmap v7775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:32 smithi115 bash[45495]: cluster 2022-04-22T20:40:31.931105+0000 mgr.y (mgr.34107) 6650 : cluster [DBG] pgmap v7775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:32.363Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:32 smithi115 bash[46934]: cluster 2022-04-22T20:40:31.931105+0000 mgr.y (mgr.34107) 6650 : cluster [DBG] pgmap v7775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:33 smithi121 bash[36902]: audit 2022-04-22T20:40:32.998983+0000 mon.a (mon.0) 10065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:33 smithi121 bash[36902]: audit 2022-04-22T20:40:33.000520+0000 mon.a (mon.0) 10066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:33 smithi115 bash[46934]: audit 2022-04-22T20:40:32.998983+0000 mon.a (mon.0) 10065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:33 smithi115 bash[46934]: audit 2022-04-22T20:40:33.000520+0000 mon.a (mon.0) 10066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:33 smithi115 bash[45495]: audit 2022-04-22T20:40:32.998983+0000 mon.a (mon.0) 10065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:33 smithi115 bash[45495]: audit 2022-04-22T20:40:33.000520+0000 mon.a (mon.0) 10066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:34.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:35 smithi121 bash[36902]: cluster 2022-04-22T20:40:33.932333+0000 mgr.y (mgr.34107) 6651 : cluster [DBG] pgmap v7776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:35 smithi121 bash[36902]: audit 2022-04-22T20:40:34.456613+0000 mon.a (mon.0) 10067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:35 smithi115 bash[46934]: cluster 2022-04-22T20:40:33.932333+0000 mgr.y (mgr.34107) 6651 : cluster [DBG] pgmap v7776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:35 smithi115 bash[46934]: audit 2022-04-22T20:40:34.456613+0000 mon.a (mon.0) 10067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:35 smithi115 bash[45495]: cluster 2022-04-22T20:40:33.932333+0000 mgr.y (mgr.34107) 6651 : cluster [DBG] pgmap v7776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:35 smithi115 bash[45495]: audit 2022-04-22T20:40:34.456613+0000 mon.a (mon.0) 10067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:37 smithi121 bash[36902]: cluster 2022-04-22T20:40:35.933618+0000 mgr.y (mgr.34107) 6652 : cluster [DBG] pgmap v7777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:37 smithi115 bash[45495]: cluster 2022-04-22T20:40:35.933618+0000 mgr.y (mgr.34107) 6652 : cluster [DBG] pgmap v7777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:37 smithi115 bash[46934]: cluster 2022-04-22T20:40:35.933618+0000 mgr.y (mgr.34107) 6652 : cluster [DBG] pgmap v7777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:38 smithi121 bash[36902]: audit 2022-04-22T20:40:38.002486+0000 mon.a (mon.0) 10068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:38 smithi121 bash[36902]: audit 2022-04-22T20:40:38.004418+0000 mon.a (mon.0) 10069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:38 smithi115 bash[45495]: audit 2022-04-22T20:40:38.002486+0000 mon.a (mon.0) 10068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:38 smithi115 bash[45495]: audit 2022-04-22T20:40:38.004418+0000 mon.a (mon.0) 10069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:38 smithi115 bash[46934]: audit 2022-04-22T20:40:38.002486+0000 mon.a (mon.0) 10068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:38 smithi115 bash[46934]: audit 2022-04-22T20:40:38.004418+0000 mon.a (mon.0) 10069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:39 smithi121 bash[36902]: cluster 2022-04-22T20:40:37.934832+0000 mgr.y (mgr.34107) 6653 : cluster [DBG] pgmap v7778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:39 smithi115 bash[46934]: cluster 2022-04-22T20:40:37.934832+0000 mgr.y (mgr.34107) 6653 : cluster [DBG] pgmap v7778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:39 smithi115 bash[45495]: cluster 2022-04-22T20:40:37.934832+0000 mgr.y (mgr.34107) 6653 : cluster [DBG] pgmap v7778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:40.067 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:40:40.420 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:40:40.421 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:40:40.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:40:40.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:40:40.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:40:40.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:40:40.422 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:40:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:40 smithi121 bash[36902]: audit 2022-04-22T20:40:39.459571+0000 mon.a (mon.0) 10070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:40 smithi115 bash[46934]: audit 2022-04-22T20:40:39.459571+0000 mon.a (mon.0) 10070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:40 smithi115 bash[45495]: audit 2022-04-22T20:40:39.459571+0000 mon.a (mon.0) 10070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:40.818 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:40:40.818 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:40:40.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:40:40.820 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:40:40.820 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:40:40.820 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:40:40.820 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:40:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:41 smithi121 bash[36902]: cluster 2022-04-22T20:40:39.935967+0000 mgr.y (mgr.34107) 6654 : cluster [DBG] pgmap v7779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:41 smithi121 bash[36902]: audit 2022-04-22T20:40:40.045339+0000 mgr.y (mgr.34107) 6655 : audit [DBG] from='client.42300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:41 smithi121 bash[36902]: audit 2022-04-22T20:40:40.811925+0000 mon.c (mon.1) 257 : audit [DBG] from='client.? 172.21.15.115:0/1815151307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[46934]: cluster 2022-04-22T20:40:39.935967+0000 mgr.y (mgr.34107) 6654 : cluster [DBG] pgmap v7779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:41.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[46934]: audit 2022-04-22T20:40:40.045339+0000 mgr.y (mgr.34107) 6655 : audit [DBG] from='client.42300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:41.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[46934]: audit 2022-04-22T20:40:40.811925+0000 mon.c (mon.1) 257 : audit [DBG] from='client.? 172.21.15.115:0/1815151307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:41.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[45495]: cluster 2022-04-22T20:40:39.935967+0000 mgr.y (mgr.34107) 6654 : cluster [DBG] pgmap v7779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:41.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[45495]: audit 2022-04-22T20:40:40.045339+0000 mgr.y (mgr.34107) 6655 : audit [DBG] from='client.42300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:41.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:41 smithi115 bash[45495]: audit 2022-04-22T20:40:40.811925+0000 mon.c (mon.1) 257 : audit [DBG] from='client.? 172.21.15.115:0/1815151307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:40:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:42 smithi121 bash[36902]: audit 2022-04-22T20:40:40.388970+0000 mgr.y (mgr.34107) 6656 : audit [DBG] from='client.42306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:42 smithi115 bash[45495]: audit 2022-04-22T20:40:40.388970+0000 mgr.y (mgr.34107) 6656 : audit [DBG] from='client.42306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:42.362Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:42.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:42 smithi115 bash[46934]: audit 2022-04-22T20:40:40.388970+0000 mgr.y (mgr.34107) 6656 : audit [DBG] from='client.42306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:40:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:43 smithi121 bash[36902]: cluster 2022-04-22T20:40:41.936891+0000 mgr.y (mgr.34107) 6657 : cluster [DBG] pgmap v7780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:43 smithi121 bash[36902]: audit 2022-04-22T20:40:43.005826+0000 mon.a (mon.0) 10071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:43 smithi121 bash[36902]: audit 2022-04-22T20:40:43.008055+0000 mon.a (mon.0) 10072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[45495]: cluster 2022-04-22T20:40:41.936891+0000 mgr.y (mgr.34107) 6657 : cluster [DBG] pgmap v7780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[45495]: audit 2022-04-22T20:40:43.005826+0000 mon.a (mon.0) 10071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[45495]: audit 2022-04-22T20:40:43.008055+0000 mon.a (mon.0) 10072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[46934]: cluster 2022-04-22T20:40:41.936891+0000 mgr.y (mgr.34107) 6657 : cluster [DBG] pgmap v7780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[46934]: audit 2022-04-22T20:40:43.005826+0000 mon.a (mon.0) 10071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:43 smithi115 bash[46934]: audit 2022-04-22T20:40:43.008055+0000 mon.a (mon.0) 10072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:44.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:45 smithi121 bash[36902]: cluster 2022-04-22T20:40:43.938149+0000 mgr.y (mgr.34107) 6658 : cluster [DBG] pgmap v7781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:45 smithi121 bash[36902]: audit 2022-04-22T20:40:44.462473+0000 mon.a (mon.0) 10073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:45 smithi115 bash[46934]: cluster 2022-04-22T20:40:43.938149+0000 mgr.y (mgr.34107) 6658 : cluster [DBG] pgmap v7781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:45.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:45 smithi115 bash[46934]: audit 2022-04-22T20:40:44.462473+0000 mon.a (mon.0) 10073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:45.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:45 smithi115 bash[45495]: cluster 2022-04-22T20:40:43.938149+0000 mgr.y (mgr.34107) 6658 : cluster [DBG] pgmap v7781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:45.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:45 smithi115 bash[45495]: audit 2022-04-22T20:40:44.462473+0000 mon.a (mon.0) 10073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:47 smithi121 bash[36902]: cluster 2022-04-22T20:40:45.939180+0000 mgr.y (mgr.34107) 6659 : cluster [DBG] pgmap v7782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:47 smithi115 bash[46934]: cluster 2022-04-22T20:40:45.939180+0000 mgr.y (mgr.34107) 6659 : cluster [DBG] pgmap v7782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:47 smithi115 bash[45495]: cluster 2022-04-22T20:40:45.939180+0000 mgr.y (mgr.34107) 6659 : cluster [DBG] pgmap v7782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:48 smithi121 bash[36902]: audit 2022-04-22T20:40:48.008953+0000 mon.a (mon.0) 10074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:48 smithi121 bash[36902]: audit 2022-04-22T20:40:48.011556+0000 mon.a (mon.0) 10075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:48 smithi115 bash[46934]: audit 2022-04-22T20:40:48.008953+0000 mon.a (mon.0) 10074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:48 smithi115 bash[46934]: audit 2022-04-22T20:40:48.011556+0000 mon.a (mon.0) 10075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:48 smithi115 bash[45495]: audit 2022-04-22T20:40:48.008953+0000 mon.a (mon.0) 10074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:48 smithi115 bash[45495]: audit 2022-04-22T20:40:48.011556+0000 mon.a (mon.0) 10075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:49.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:49 smithi121 bash[36902]: cluster 2022-04-22T20:40:47.940200+0000 mgr.y (mgr.34107) 6660 : cluster [DBG] pgmap v7783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:49.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:49 smithi121 bash[36902]: audit 2022-04-22T20:40:48.931605+0000 mon.a (mon.0) 10076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:40:49.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:49 smithi121 bash[36902]: audit 2022-04-22T20:40:48.934764+0000 mon.a (mon.0) 10077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:40:49.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:49 smithi121 bash[36902]: audit 2022-04-22T20:40:48.936773+0000 mon.a (mon.0) 10078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[46934]: cluster 2022-04-22T20:40:47.940200+0000 mgr.y (mgr.34107) 6660 : cluster [DBG] pgmap v7783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[46934]: audit 2022-04-22T20:40:48.931605+0000 mon.a (mon.0) 10076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[46934]: audit 2022-04-22T20:40:48.934764+0000 mon.a (mon.0) 10077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[46934]: audit 2022-04-22T20:40:48.936773+0000 mon.a (mon.0) 10078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[45495]: cluster 2022-04-22T20:40:47.940200+0000 mgr.y (mgr.34107) 6660 : cluster [DBG] pgmap v7783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[45495]: audit 2022-04-22T20:40:48.931605+0000 mon.a (mon.0) 10076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:40:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[45495]: audit 2022-04-22T20:40:48.934764+0000 mon.a (mon.0) 10077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:40:49.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:49 smithi115 bash[45495]: audit 2022-04-22T20:40:48.936773+0000 mon.a (mon.0) 10078 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:40:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:50 smithi121 bash[36902]: audit 2022-04-22T20:40:49.465420+0000 mon.a (mon.0) 10079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:50 smithi121 bash[36902]: audit 2022-04-22T20:40:49.551385+0000 mon.a (mon.0) 10080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:50 smithi115 bash[45495]: audit 2022-04-22T20:40:49.465420+0000 mon.a (mon.0) 10079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:50 smithi115 bash[45495]: audit 2022-04-22T20:40:49.551385+0000 mon.a (mon.0) 10080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:50 smithi115 bash[46934]: audit 2022-04-22T20:40:49.465420+0000 mon.a (mon.0) 10079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:50 smithi115 bash[46934]: audit 2022-04-22T20:40:49.551385+0000 mon.a (mon.0) 10080 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:51 smithi121 bash[36902]: cluster 2022-04-22T20:40:49.941333+0000 mgr.y (mgr.34107) 6661 : cluster [DBG] pgmap v7784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:51.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:51 smithi115 bash[45495]: cluster 2022-04-22T20:40:49.941333+0000 mgr.y (mgr.34107) 6661 : cluster [DBG] pgmap v7784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:51.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:51 smithi115 bash[46934]: cluster 2022-04-22T20:40:49.941333+0000 mgr.y (mgr.34107) 6661 : cluster [DBG] pgmap v7784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:52.362Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:53 smithi121 bash[36902]: cluster 2022-04-22T20:40:51.942546+0000 mgr.y (mgr.34107) 6662 : cluster [DBG] pgmap v7785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:53 smithi121 bash[36902]: audit 2022-04-22T20:40:53.012283+0000 mon.a (mon.0) 10081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:53 smithi121 bash[36902]: audit 2022-04-22T20:40:53.015436+0000 mon.a (mon.0) 10082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[45495]: cluster 2022-04-22T20:40:51.942546+0000 mgr.y (mgr.34107) 6662 : cluster [DBG] pgmap v7785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[45495]: audit 2022-04-22T20:40:53.012283+0000 mon.a (mon.0) 10081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[45495]: audit 2022-04-22T20:40:53.015436+0000 mon.a (mon.0) 10082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[46934]: cluster 2022-04-22T20:40:51.942546+0000 mgr.y (mgr.34107) 6662 : cluster [DBG] pgmap v7785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[46934]: audit 2022-04-22T20:40:53.012283+0000 mon.a (mon.0) 10081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:53 smithi115 bash[46934]: audit 2022-04-22T20:40:53.015436+0000 mon.a (mon.0) 10082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:40:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:40:54.245Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:40:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:55 smithi121 bash[36902]: cluster 2022-04-22T20:40:53.943888+0000 mgr.y (mgr.34107) 6663 : cluster [DBG] pgmap v7786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:55 smithi121 bash[36902]: audit 2022-04-22T20:40:54.469598+0000 mon.a (mon.0) 10083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:55 smithi115 bash[45495]: cluster 2022-04-22T20:40:53.943888+0000 mgr.y (mgr.34107) 6663 : cluster [DBG] pgmap v7786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:55 smithi115 bash[45495]: audit 2022-04-22T20:40:54.469598+0000 mon.a (mon.0) 10083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:55 smithi115 bash[46934]: cluster 2022-04-22T20:40:53.943888+0000 mgr.y (mgr.34107) 6663 : cluster [DBG] pgmap v7786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:55 smithi115 bash[46934]: audit 2022-04-22T20:40:54.469598+0000 mon.a (mon.0) 10083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:40:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:56 smithi121 bash[36902]: cluster 2022-04-22T20:40:55.944988+0000 mgr.y (mgr.34107) 6664 : cluster [DBG] pgmap v7787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:56.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:56 smithi115 bash[46934]: cluster 2022-04-22T20:40:55.944988+0000 mgr.y (mgr.34107) 6664 : cluster [DBG] pgmap v7787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:56.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:56 smithi115 bash[45495]: cluster 2022-04-22T20:40:55.944988+0000 mgr.y (mgr.34107) 6664 : cluster [DBG] pgmap v7787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:59.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:59 smithi121 bash[36902]: cluster 2022-04-22T20:40:57.945933+0000 mgr.y (mgr.34107) 6665 : cluster [DBG] pgmap v7788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:59.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:59 smithi121 bash[36902]: audit 2022-04-22T20:40:58.015206+0000 mon.a (mon.0) 10084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:59.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:59 smithi121 bash[36902]: audit 2022-04-22T20:40:58.018688+0000 mon.a (mon.0) 10085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:59.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:59 smithi121 bash[36902]: audit 2022-04-22T20:40:58.393218+0000 mon.a (mon.0) 10086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:59.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:40:59 smithi121 bash[36902]: audit 2022-04-22T20:40:58.399382+0000 mon.a (mon.0) 10087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:59.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[46934]: cluster 2022-04-22T20:40:57.945933+0000 mgr.y (mgr.34107) 6665 : cluster [DBG] pgmap v7788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:59.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[46934]: audit 2022-04-22T20:40:58.015206+0000 mon.a (mon.0) 10084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:59.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[46934]: audit 2022-04-22T20:40:58.018688+0000 mon.a (mon.0) 10085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:59.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[46934]: audit 2022-04-22T20:40:58.393218+0000 mon.a (mon.0) 10086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:59.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[46934]: audit 2022-04-22T20:40:58.399382+0000 mon.a (mon.0) 10087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:59.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[45495]: cluster 2022-04-22T20:40:57.945933+0000 mgr.y (mgr.34107) 6665 : cluster [DBG] pgmap v7788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:40:59.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[45495]: audit 2022-04-22T20:40:58.015206+0000 mon.a (mon.0) 10084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:40:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[45495]: audit 2022-04-22T20:40:58.018688+0000 mon.a (mon.0) 10085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:40:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[45495]: audit 2022-04-22T20:40:58.393218+0000 mon.a (mon.0) 10086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:40:59.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:40:59 smithi115 bash[45495]: audit 2022-04-22T20:40:58.399382+0000 mon.a (mon.0) 10087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:41:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:00 smithi121 bash[36902]: cephadm 2022-04-22T20:40:58.382614+0000 mgr.y (mgr.34107) 6666 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:41:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:00 smithi121 bash[36902]: audit 2022-04-22T20:40:59.472571+0000 mon.a (mon.0) 10088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:00 smithi115 bash[46934]: cephadm 2022-04-22T20:40:58.382614+0000 mgr.y (mgr.34107) 6666 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:41:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:00 smithi115 bash[46934]: audit 2022-04-22T20:40:59.472571+0000 mon.a (mon.0) 10088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:00 smithi115 bash[45495]: cephadm 2022-04-22T20:40:58.382614+0000 mgr.y (mgr.34107) 6666 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T20:41:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:00 smithi115 bash[45495]: audit 2022-04-22T20:40:59.472571+0000 mon.a (mon.0) 10088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:01 smithi121 bash[36902]: cluster 2022-04-22T20:40:59.947092+0000 mgr.y (mgr.34107) 6667 : cluster [DBG] pgmap v7789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:01 smithi115 bash[45495]: cluster 2022-04-22T20:40:59.947092+0000 mgr.y (mgr.34107) 6667 : cluster [DBG] pgmap v7789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:01 smithi115 bash[46934]: cluster 2022-04-22T20:40:59.947092+0000 mgr.y (mgr.34107) 6667 : cluster [DBG] pgmap v7789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:02.361Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:03 smithi121 bash[36902]: cluster 2022-04-22T20:41:01.948159+0000 mgr.y (mgr.34107) 6668 : cluster [DBG] pgmap v7790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:03 smithi121 bash[36902]: audit 2022-04-22T20:41:03.018282+0000 mon.a (mon.0) 10089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:03 smithi121 bash[36902]: audit 2022-04-22T20:41:03.021817+0000 mon.a (mon.0) 10090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[45495]: cluster 2022-04-22T20:41:01.948159+0000 mgr.y (mgr.34107) 6668 : cluster [DBG] pgmap v7790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[45495]: audit 2022-04-22T20:41:03.018282+0000 mon.a (mon.0) 10089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[45495]: audit 2022-04-22T20:41:03.021817+0000 mon.a (mon.0) 10090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:03.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[46934]: cluster 2022-04-22T20:41:01.948159+0000 mgr.y (mgr.34107) 6668 : cluster [DBG] pgmap v7790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[46934]: audit 2022-04-22T20:41:03.018282+0000 mon.a (mon.0) 10089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:03 smithi115 bash[46934]: audit 2022-04-22T20:41:03.021817+0000 mon.a (mon.0) 10090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:04.246Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:05 smithi121 bash[36902]: cluster 2022-04-22T20:41:03.949179+0000 mgr.y (mgr.34107) 6669 : cluster [DBG] pgmap v7791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:05 smithi121 bash[36902]: audit 2022-04-22T20:41:04.475429+0000 mon.a (mon.0) 10091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:05 smithi115 bash[45495]: cluster 2022-04-22T20:41:03.949179+0000 mgr.y (mgr.34107) 6669 : cluster [DBG] pgmap v7791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:05 smithi115 bash[45495]: audit 2022-04-22T20:41:04.475429+0000 mon.a (mon.0) 10091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:05 smithi115 bash[46934]: cluster 2022-04-22T20:41:03.949179+0000 mgr.y (mgr.34107) 6669 : cluster [DBG] pgmap v7791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:05 smithi115 bash[46934]: audit 2022-04-22T20:41:04.475429+0000 mon.a (mon.0) 10091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:07 smithi121 bash[36902]: cluster 2022-04-22T20:41:05.950264+0000 mgr.y (mgr.34107) 6670 : cluster [DBG] pgmap v7792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:07 smithi115 bash[45495]: cluster 2022-04-22T20:41:05.950264+0000 mgr.y (mgr.34107) 6670 : cluster [DBG] pgmap v7792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:07 smithi115 bash[46934]: cluster 2022-04-22T20:41:05.950264+0000 mgr.y (mgr.34107) 6670 : cluster [DBG] pgmap v7792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:08 smithi121 bash[36902]: audit 2022-04-22T20:41:08.021240+0000 mon.a (mon.0) 10092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:08 smithi121 bash[36902]: audit 2022-04-22T20:41:08.025128+0000 mon.a (mon.0) 10093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:08 smithi115 bash[45495]: audit 2022-04-22T20:41:08.021240+0000 mon.a (mon.0) 10092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:08 smithi115 bash[45495]: audit 2022-04-22T20:41:08.025128+0000 mon.a (mon.0) 10093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:08 smithi115 bash[46934]: audit 2022-04-22T20:41:08.021240+0000 mon.a (mon.0) 10092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:08 smithi115 bash[46934]: audit 2022-04-22T20:41:08.025128+0000 mon.a (mon.0) 10093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:09 smithi121 bash[36902]: cluster 2022-04-22T20:41:07.951268+0000 mgr.y (mgr.34107) 6671 : cluster [DBG] pgmap v7793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:09 smithi115 bash[45495]: cluster 2022-04-22T20:41:07.951268+0000 mgr.y (mgr.34107) 6671 : cluster [DBG] pgmap v7793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:09 smithi115 bash[46934]: cluster 2022-04-22T20:41:07.951268+0000 mgr.y (mgr.34107) 6671 : cluster [DBG] pgmap v7793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:10 smithi121 bash[36902]: audit 2022-04-22T20:41:09.478150+0000 mon.a (mon.0) 10094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:10.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:10 smithi121 bash[36902]: cluster 2022-04-22T20:41:09.952510+0000 mgr.y (mgr.34107) 6672 : cluster [DBG] pgmap v7794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:10 smithi115 bash[45495]: audit 2022-04-22T20:41:09.478150+0000 mon.a (mon.0) 10094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:10 smithi115 bash[45495]: cluster 2022-04-22T20:41:09.952510+0000 mgr.y (mgr.34107) 6672 : cluster [DBG] pgmap v7794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:10 smithi115 bash[46934]: audit 2022-04-22T20:41:09.478150+0000 mon.a (mon.0) 10094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:10 smithi115 bash[46934]: cluster 2022-04-22T20:41:09.952510+0000 mgr.y (mgr.34107) 6672 : cluster [DBG] pgmap v7794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:11.199 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:41:11.543 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:41:11.543 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:41:11.544 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:41:11.544 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:41:11.544 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:41:11.544 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:41:11.545 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:41:11.546 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:41:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:11 smithi121 bash[36902]: audit 2022-04-22T20:41:11.175659+0000 mgr.y (mgr.34107) 6673 : audit [DBG] from='client.52249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:11 smithi115 bash[46934]: audit 2022-04-22T20:41:11.175659+0000 mgr.y (mgr.34107) 6673 : audit [DBG] from='client.52249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:11.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:11 smithi115 bash[45495]: audit 2022-04-22T20:41:11.175659+0000 mgr.y (mgr.34107) 6673 : audit [DBG] from='client.52249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:11.954 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:41:11.955 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:41:11.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:41:11.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:11.956 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:41:11.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:41:11.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:11.957 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:41:11.957 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:41:11.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:11.958 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:41:11.959 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:41:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:12 smithi121 bash[36902]: audit 2022-04-22T20:41:11.518173+0000 mgr.y (mgr.34107) 6674 : audit [DBG] from='client.52255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:12 smithi121 bash[36902]: audit 2022-04-22T20:41:11.946721+0000 mon.a (mon.0) 10095 : audit [DBG] from='client.? 172.21.15.115:0/3009639006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:12 smithi121 bash[36902]: cluster 2022-04-22T20:41:11.953428+0000 mgr.y (mgr.34107) 6675 : cluster [DBG] pgmap v7795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:12.364Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[45495]: audit 2022-04-22T20:41:11.518173+0000 mgr.y (mgr.34107) 6674 : audit [DBG] from='client.52255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[45495]: audit 2022-04-22T20:41:11.946721+0000 mon.a (mon.0) 10095 : audit [DBG] from='client.? 172.21.15.115:0/3009639006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[45495]: cluster 2022-04-22T20:41:11.953428+0000 mgr.y (mgr.34107) 6675 : cluster [DBG] pgmap v7795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[46934]: audit 2022-04-22T20:41:11.518173+0000 mgr.y (mgr.34107) 6674 : audit [DBG] from='client.52255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[46934]: audit 2022-04-22T20:41:11.946721+0000 mon.a (mon.0) 10095 : audit [DBG] from='client.? 172.21.15.115:0/3009639006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:12 smithi115 bash[46934]: cluster 2022-04-22T20:41:11.953428+0000 mgr.y (mgr.34107) 6675 : cluster [DBG] pgmap v7795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:13 smithi121 bash[36902]: audit 2022-04-22T20:41:13.024327+0000 mon.a (mon.0) 10096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:13 smithi121 bash[36902]: audit 2022-04-22T20:41:13.028646+0000 mon.a (mon.0) 10097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:13 smithi115 bash[46934]: audit 2022-04-22T20:41:13.024327+0000 mon.a (mon.0) 10096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:13 smithi115 bash[46934]: audit 2022-04-22T20:41:13.028646+0000 mon.a (mon.0) 10097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:13 smithi115 bash[45495]: audit 2022-04-22T20:41:13.024327+0000 mon.a (mon.0) 10096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:13 smithi115 bash[45495]: audit 2022-04-22T20:41:13.028646+0000 mon.a (mon.0) 10097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:14.246Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:15 smithi121 bash[36902]: cluster 2022-04-22T20:41:13.954533+0000 mgr.y (mgr.34107) 6676 : cluster [DBG] pgmap v7796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:15 smithi121 bash[36902]: audit 2022-04-22T20:41:14.480675+0000 mon.a (mon.0) 10098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:15 smithi115 bash[45495]: cluster 2022-04-22T20:41:13.954533+0000 mgr.y (mgr.34107) 6676 : cluster [DBG] pgmap v7796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:15 smithi115 bash[45495]: audit 2022-04-22T20:41:14.480675+0000 mon.a (mon.0) 10098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:15 smithi115 bash[46934]: cluster 2022-04-22T20:41:13.954533+0000 mgr.y (mgr.34107) 6676 : cluster [DBG] pgmap v7796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:15 smithi115 bash[46934]: audit 2022-04-22T20:41:14.480675+0000 mon.a (mon.0) 10098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:16 smithi121 bash[36902]: audit 2022-04-22T20:41:15.513812+0000 mon.a (mon.0) 10099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:41:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:16 smithi121 bash[36902]: audit 2022-04-22T20:41:15.514371+0000 mon.a (mon.0) 10100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:41:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:16 smithi115 bash[45495]: audit 2022-04-22T20:41:15.513812+0000 mon.a (mon.0) 10099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:41:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:16 smithi115 bash[45495]: audit 2022-04-22T20:41:15.514371+0000 mon.a (mon.0) 10100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:41:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:16 smithi115 bash[46934]: audit 2022-04-22T20:41:15.513812+0000 mon.a (mon.0) 10099 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:41:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:16 smithi115 bash[46934]: audit 2022-04-22T20:41:15.514371+0000 mon.a (mon.0) 10100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:41:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:17 smithi121 bash[36902]: cluster 2022-04-22T20:41:15.955601+0000 mgr.y (mgr.34107) 6677 : cluster [DBG] pgmap v7797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:17.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:17 smithi115 bash[45495]: cluster 2022-04-22T20:41:15.955601+0000 mgr.y (mgr.34107) 6677 : cluster [DBG] pgmap v7797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:17.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:17 smithi115 bash[46934]: cluster 2022-04-22T20:41:15.955601+0000 mgr.y (mgr.34107) 6677 : cluster [DBG] pgmap v7797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:18 smithi121 bash[36902]: audit 2022-04-22T20:41:18.027552+0000 mon.a (mon.0) 10101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:18 smithi121 bash[36902]: audit 2022-04-22T20:41:18.031952+0000 mon.a (mon.0) 10102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:18 smithi115 bash[45495]: audit 2022-04-22T20:41:18.027552+0000 mon.a (mon.0) 10101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:18 smithi115 bash[45495]: audit 2022-04-22T20:41:18.031952+0000 mon.a (mon.0) 10102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:18 smithi115 bash[46934]: audit 2022-04-22T20:41:18.027552+0000 mon.a (mon.0) 10101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:18 smithi115 bash[46934]: audit 2022-04-22T20:41:18.031952+0000 mon.a (mon.0) 10102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:19 smithi121 bash[36902]: cluster 2022-04-22T20:41:17.956697+0000 mgr.y (mgr.34107) 6678 : cluster [DBG] pgmap v7798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:19 smithi115 bash[45495]: cluster 2022-04-22T20:41:17.956697+0000 mgr.y (mgr.34107) 6678 : cluster [DBG] pgmap v7798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:19 smithi115 bash[46934]: cluster 2022-04-22T20:41:17.956697+0000 mgr.y (mgr.34107) 6678 : cluster [DBG] pgmap v7798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:20 smithi121 bash[36902]: audit 2022-04-22T20:41:19.483374+0000 mon.a (mon.0) 10103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:20 smithi115 bash[45495]: audit 2022-04-22T20:41:19.483374+0000 mon.a (mon.0) 10103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:20 smithi115 bash[46934]: audit 2022-04-22T20:41:19.483374+0000 mon.a (mon.0) 10103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:21 smithi121 bash[36902]: cluster 2022-04-22T20:41:19.957990+0000 mgr.y (mgr.34107) 6679 : cluster [DBG] pgmap v7799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:21.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:21 smithi115 bash[45495]: cluster 2022-04-22T20:41:19.957990+0000 mgr.y (mgr.34107) 6679 : cluster [DBG] pgmap v7799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:21.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:21 smithi115 bash[46934]: cluster 2022-04-22T20:41:19.957990+0000 mgr.y (mgr.34107) 6679 : cluster [DBG] pgmap v7799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:22 smithi121 bash[36902]: cluster 2022-04-22T20:41:21.959030+0000 mgr.y (mgr.34107) 6680 : cluster [DBG] pgmap v7800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:22.364Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:22.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:22 smithi115 bash[45495]: cluster 2022-04-22T20:41:21.959030+0000 mgr.y (mgr.34107) 6680 : cluster [DBG] pgmap v7800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:22.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:22 smithi115 bash[46934]: cluster 2022-04-22T20:41:21.959030+0000 mgr.y (mgr.34107) 6680 : cluster [DBG] pgmap v7800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:23 smithi121 bash[36902]: audit 2022-04-22T20:41:23.030536+0000 mon.a (mon.0) 10104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:23 smithi121 bash[36902]: audit 2022-04-22T20:41:23.035416+0000 mon.a (mon.0) 10105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:23 smithi115 bash[46934]: audit 2022-04-22T20:41:23.030536+0000 mon.a (mon.0) 10104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:23 smithi115 bash[46934]: audit 2022-04-22T20:41:23.035416+0000 mon.a (mon.0) 10105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:23 smithi115 bash[45495]: audit 2022-04-22T20:41:23.030536+0000 mon.a (mon.0) 10104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:23 smithi115 bash[45495]: audit 2022-04-22T20:41:23.035416+0000 mon.a (mon.0) 10105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:24.248Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:25 smithi121 bash[36902]: cluster 2022-04-22T20:41:23.960297+0000 mgr.y (mgr.34107) 6681 : cluster [DBG] pgmap v7801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:25 smithi121 bash[36902]: audit 2022-04-22T20:41:24.486407+0000 mon.a (mon.0) 10106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:25 smithi115 bash[46934]: cluster 2022-04-22T20:41:23.960297+0000 mgr.y (mgr.34107) 6681 : cluster [DBG] pgmap v7801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:25 smithi115 bash[46934]: audit 2022-04-22T20:41:24.486407+0000 mon.a (mon.0) 10106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:25 smithi115 bash[45495]: cluster 2022-04-22T20:41:23.960297+0000 mgr.y (mgr.34107) 6681 : cluster [DBG] pgmap v7801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:25 smithi115 bash[45495]: audit 2022-04-22T20:41:24.486407+0000 mon.a (mon.0) 10106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:27 smithi121 bash[36902]: cluster 2022-04-22T20:41:25.961358+0000 mgr.y (mgr.34107) 6682 : cluster [DBG] pgmap v7802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:27 smithi115 bash[45495]: cluster 2022-04-22T20:41:25.961358+0000 mgr.y (mgr.34107) 6682 : cluster [DBG] pgmap v7802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:27 smithi115 bash[46934]: cluster 2022-04-22T20:41:25.961358+0000 mgr.y (mgr.34107) 6682 : cluster [DBG] pgmap v7802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:28 smithi121 bash[36902]: audit 2022-04-22T20:41:28.033605+0000 mon.a (mon.0) 10107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:28 smithi121 bash[36902]: audit 2022-04-22T20:41:28.038514+0000 mon.a (mon.0) 10108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:28.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:28 smithi115 bash[46934]: audit 2022-04-22T20:41:28.033605+0000 mon.a (mon.0) 10107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:28 smithi115 bash[46934]: audit 2022-04-22T20:41:28.038514+0000 mon.a (mon.0) 10108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:28 smithi115 bash[45495]: audit 2022-04-22T20:41:28.033605+0000 mon.a (mon.0) 10107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:28 smithi115 bash[45495]: audit 2022-04-22T20:41:28.038514+0000 mon.a (mon.0) 10108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:29 smithi121 bash[36902]: cluster 2022-04-22T20:41:27.962373+0000 mgr.y (mgr.34107) 6683 : cluster [DBG] pgmap v7803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:29 smithi115 bash[46934]: cluster 2022-04-22T20:41:27.962373+0000 mgr.y (mgr.34107) 6683 : cluster [DBG] pgmap v7803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:29 smithi115 bash[45495]: cluster 2022-04-22T20:41:27.962373+0000 mgr.y (mgr.34107) 6683 : cluster [DBG] pgmap v7803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:30 smithi121 bash[36902]: audit 2022-04-22T20:41:29.489375+0000 mon.a (mon.0) 10109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:30.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:30 smithi115 bash[45495]: audit 2022-04-22T20:41:29.489375+0000 mon.a (mon.0) 10109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:30 smithi115 bash[46934]: audit 2022-04-22T20:41:29.489375+0000 mon.a (mon.0) 10109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:31 smithi121 bash[36902]: cluster 2022-04-22T20:41:29.963554+0000 mgr.y (mgr.34107) 6684 : cluster [DBG] pgmap v7804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:31.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:31 smithi115 bash[45495]: cluster 2022-04-22T20:41:29.963554+0000 mgr.y (mgr.34107) 6684 : cluster [DBG] pgmap v7804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:31 smithi115 bash[46934]: cluster 2022-04-22T20:41:29.963554+0000 mgr.y (mgr.34107) 6684 : cluster [DBG] pgmap v7804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:32 smithi121 bash[36902]: cluster 2022-04-22T20:41:31.964626+0000 mgr.y (mgr.34107) 6685 : cluster [DBG] pgmap v7805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:32 smithi115 bash[45495]: cluster 2022-04-22T20:41:31.964626+0000 mgr.y (mgr.34107) 6685 : cluster [DBG] pgmap v7805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:32.366Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:32 smithi115 bash[46934]: cluster 2022-04-22T20:41:31.964626+0000 mgr.y (mgr.34107) 6685 : cluster [DBG] pgmap v7805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:33 smithi121 bash[36902]: audit 2022-04-22T20:41:33.036642+0000 mon.a (mon.0) 10110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:33 smithi121 bash[36902]: audit 2022-04-22T20:41:33.042109+0000 mon.a (mon.0) 10111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:33 smithi115 bash[45495]: audit 2022-04-22T20:41:33.036642+0000 mon.a (mon.0) 10110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:33 smithi115 bash[45495]: audit 2022-04-22T20:41:33.042109+0000 mon.a (mon.0) 10111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:33 smithi115 bash[46934]: audit 2022-04-22T20:41:33.036642+0000 mon.a (mon.0) 10110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:33 smithi115 bash[46934]: audit 2022-04-22T20:41:33.042109+0000 mon.a (mon.0) 10111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:34.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:34.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:35 smithi121 bash[36902]: cluster 2022-04-22T20:41:33.965803+0000 mgr.y (mgr.34107) 6686 : cluster [DBG] pgmap v7806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:35 smithi121 bash[36902]: audit 2022-04-22T20:41:34.492434+0000 mon.a (mon.0) 10112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:35 smithi115 bash[45495]: cluster 2022-04-22T20:41:33.965803+0000 mgr.y (mgr.34107) 6686 : cluster [DBG] pgmap v7806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:35 smithi115 bash[45495]: audit 2022-04-22T20:41:34.492434+0000 mon.a (mon.0) 10112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:35 smithi115 bash[46934]: cluster 2022-04-22T20:41:33.965803+0000 mgr.y (mgr.34107) 6686 : cluster [DBG] pgmap v7806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:35 smithi115 bash[46934]: audit 2022-04-22T20:41:34.492434+0000 mon.a (mon.0) 10112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:37 smithi121 bash[36902]: cluster 2022-04-22T20:41:35.966863+0000 mgr.y (mgr.34107) 6687 : cluster [DBG] pgmap v7807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:37 smithi115 bash[45495]: cluster 2022-04-22T20:41:35.966863+0000 mgr.y (mgr.34107) 6687 : cluster [DBG] pgmap v7807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:37.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:37 smithi115 bash[46934]: cluster 2022-04-22T20:41:35.966863+0000 mgr.y (mgr.34107) 6687 : cluster [DBG] pgmap v7807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:38 smithi121 bash[36902]: audit 2022-04-22T20:41:38.039827+0000 mon.a (mon.0) 10113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:38.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:38 smithi121 bash[36902]: audit 2022-04-22T20:41:38.045461+0000 mon.a (mon.0) 10114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:38.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:38 smithi115 bash[45495]: audit 2022-04-22T20:41:38.039827+0000 mon.a (mon.0) 10113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:38.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:38 smithi115 bash[45495]: audit 2022-04-22T20:41:38.045461+0000 mon.a (mon.0) 10114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:38.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:38 smithi115 bash[46934]: audit 2022-04-22T20:41:38.039827+0000 mon.a (mon.0) 10113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:38.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:38 smithi115 bash[46934]: audit 2022-04-22T20:41:38.045461+0000 mon.a (mon.0) 10114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:39 smithi121 bash[36902]: cluster 2022-04-22T20:41:37.967795+0000 mgr.y (mgr.34107) 6688 : cluster [DBG] pgmap v7808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:39.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:39 smithi115 bash[45495]: cluster 2022-04-22T20:41:37.967795+0000 mgr.y (mgr.34107) 6688 : cluster [DBG] pgmap v7808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:39.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:39 smithi115 bash[46934]: cluster 2022-04-22T20:41:37.967795+0000 mgr.y (mgr.34107) 6688 : cluster [DBG] pgmap v7808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:40 smithi121 bash[36902]: audit 2022-04-22T20:41:39.495714+0000 mon.a (mon.0) 10115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:40 smithi115 bash[45495]: audit 2022-04-22T20:41:39.495714+0000 mon.a (mon.0) 10115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:40 smithi115 bash[46934]: audit 2022-04-22T20:41:39.495714+0000 mon.a (mon.0) 10115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:41 smithi121 bash[36902]: cluster 2022-04-22T20:41:39.968929+0000 mgr.y (mgr.34107) 6689 : cluster [DBG] pgmap v7809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:41.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:41 smithi115 bash[46934]: cluster 2022-04-22T20:41:39.968929+0000 mgr.y (mgr.34107) 6689 : cluster [DBG] pgmap v7809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:41.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:41 smithi115 bash[45495]: cluster 2022-04-22T20:41:39.968929+0000 mgr.y (mgr.34107) 6689 : cluster [DBG] pgmap v7809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:42.327 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:41:42.685 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:41:42.686 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:41:42.686 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:41:42.686 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:41:42.687 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:41:42.687 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:41:42.687 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:41:42.687 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10m ago 3h 215M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10m ago 3h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:41:42.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:41:42.689 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10m ago 3h 578M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:41:42.689 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10m ago 3h 708M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:41:42.689 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10m ago 3h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:41:42.689 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10m ago 3h 639M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:41:42.689 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10m ago 3h 73.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:41:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:42 smithi121 bash[36902]: cluster 2022-04-22T20:41:41.970128+0000 mgr.y (mgr.34107) 6690 : cluster [DBG] pgmap v7810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:42 smithi121 bash[36902]: audit 2022-04-22T20:41:42.302212+0000 mgr.y (mgr.34107) 6691 : audit [DBG] from='client.52267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:42 smithi115 bash[45495]: cluster 2022-04-22T20:41:41.970128+0000 mgr.y (mgr.34107) 6690 : cluster [DBG] pgmap v7810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:42 smithi115 bash[45495]: audit 2022-04-22T20:41:42.302212+0000 mgr.y (mgr.34107) 6691 : audit [DBG] from='client.52267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:42.366Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:42.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:42 smithi115 bash[46934]: cluster 2022-04-22T20:41:41.970128+0000 mgr.y (mgr.34107) 6690 : cluster [DBG] pgmap v7810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:42.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:42 smithi115 bash[46934]: audit 2022-04-22T20:41:42.302212+0000 mgr.y (mgr.34107) 6691 : audit [DBG] from='client.52267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:43.086 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:43.087 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:41:43.088 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:41:43.088 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:43.088 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:41:43.088 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:41:43.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:41:43.090 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:41:43.090 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:41:43.090 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:41:43.091 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:41:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:43 smithi121 bash[36902]: audit 2022-04-22T20:41:42.652237+0000 mgr.y (mgr.34107) 6692 : audit [DBG] from='client.52273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:43 smithi121 bash[36902]: audit 2022-04-22T20:41:43.042692+0000 mon.a (mon.0) 10116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:43 smithi121 bash[36902]: audit 2022-04-22T20:41:43.048660+0000 mon.a (mon.0) 10117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:43 smithi121 bash[36902]: audit 2022-04-22T20:41:43.079627+0000 mon.a (mon.0) 10118 : audit [DBG] from='client.? 172.21.15.115:0/2038181762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[45495]: audit 2022-04-22T20:41:42.652237+0000 mgr.y (mgr.34107) 6692 : audit [DBG] from='client.52273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[45495]: audit 2022-04-22T20:41:43.042692+0000 mon.a (mon.0) 10116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[45495]: audit 2022-04-22T20:41:43.048660+0000 mon.a (mon.0) 10117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[45495]: audit 2022-04-22T20:41:43.079627+0000 mon.a (mon.0) 10118 : audit [DBG] from='client.? 172.21.15.115:0/2038181762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[46934]: audit 2022-04-22T20:41:42.652237+0000 mgr.y (mgr.34107) 6692 : audit [DBG] from='client.52273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:41:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[46934]: audit 2022-04-22T20:41:43.042692+0000 mon.a (mon.0) 10116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[46934]: audit 2022-04-22T20:41:43.048660+0000 mon.a (mon.0) 10117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:43.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:43 smithi115 bash[46934]: audit 2022-04-22T20:41:43.079627+0000 mon.a (mon.0) 10118 : audit [DBG] from='client.? 172.21.15.115:0/2038181762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:41:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:44.249Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:45 smithi121 bash[36902]: cluster 2022-04-22T20:41:43.971169+0000 mgr.y (mgr.34107) 6693 : cluster [DBG] pgmap v7811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:45 smithi121 bash[36902]: audit 2022-04-22T20:41:44.498546+0000 mon.a (mon.0) 10119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:45 smithi115 bash[45495]: cluster 2022-04-22T20:41:43.971169+0000 mgr.y (mgr.34107) 6693 : cluster [DBG] pgmap v7811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:45 smithi115 bash[45495]: audit 2022-04-22T20:41:44.498546+0000 mon.a (mon.0) 10119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:45 smithi115 bash[46934]: cluster 2022-04-22T20:41:43.971169+0000 mgr.y (mgr.34107) 6693 : cluster [DBG] pgmap v7811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:45 smithi115 bash[46934]: audit 2022-04-22T20:41:44.498546+0000 mon.a (mon.0) 10119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:47 smithi121 bash[36902]: cluster 2022-04-22T20:41:45.972233+0000 mgr.y (mgr.34107) 6694 : cluster [DBG] pgmap v7812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:47 smithi115 bash[45495]: cluster 2022-04-22T20:41:45.972233+0000 mgr.y (mgr.34107) 6694 : cluster [DBG] pgmap v7812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:47 smithi115 bash[46934]: cluster 2022-04-22T20:41:45.972233+0000 mgr.y (mgr.34107) 6694 : cluster [DBG] pgmap v7812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:48 smithi121 bash[36902]: audit 2022-04-22T20:41:48.045656+0000 mon.a (mon.0) 10120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:48 smithi121 bash[36902]: audit 2022-04-22T20:41:48.051804+0000 mon.a (mon.0) 10121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:48 smithi115 bash[45495]: audit 2022-04-22T20:41:48.045656+0000 mon.a (mon.0) 10120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:48 smithi115 bash[45495]: audit 2022-04-22T20:41:48.051804+0000 mon.a (mon.0) 10121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:48 smithi115 bash[46934]: audit 2022-04-22T20:41:48.045656+0000 mon.a (mon.0) 10120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:48 smithi115 bash[46934]: audit 2022-04-22T20:41:48.051804+0000 mon.a (mon.0) 10121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:49.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:49 smithi121 bash[36902]: cluster 2022-04-22T20:41:47.973311+0000 mgr.y (mgr.34107) 6695 : cluster [DBG] pgmap v7813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:49 smithi115 bash[45495]: cluster 2022-04-22T20:41:47.973311+0000 mgr.y (mgr.34107) 6695 : cluster [DBG] pgmap v7813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:49 smithi115 bash[46934]: cluster 2022-04-22T20:41:47.973311+0000 mgr.y (mgr.34107) 6695 : cluster [DBG] pgmap v7813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:50 smithi121 bash[36902]: audit 2022-04-22T20:41:49.501690+0000 mon.a (mon.0) 10122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:50 smithi121 bash[36902]: cluster 2022-04-22T20:41:49.974546+0000 mgr.y (mgr.34107) 6696 : cluster [DBG] pgmap v7814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:50 smithi115 bash[45495]: audit 2022-04-22T20:41:49.501690+0000 mon.a (mon.0) 10122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:50 smithi115 bash[45495]: cluster 2022-04-22T20:41:49.974546+0000 mgr.y (mgr.34107) 6696 : cluster [DBG] pgmap v7814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:50 smithi115 bash[46934]: audit 2022-04-22T20:41:49.501690+0000 mon.a (mon.0) 10122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:50 smithi115 bash[46934]: cluster 2022-04-22T20:41:49.974546+0000 mgr.y (mgr.34107) 6696 : cluster [DBG] pgmap v7814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:52.366Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:53.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:53 smithi115 bash[45495]: cluster 2022-04-22T20:41:51.975804+0000 mgr.y (mgr.34107) 6697 : cluster [DBG] pgmap v7815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:53.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:53 smithi115 bash[46934]: cluster 2022-04-22T20:41:51.975804+0000 mgr.y (mgr.34107) 6697 : cluster [DBG] pgmap v7815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:53 smithi121 bash[36902]: cluster 2022-04-22T20:41:51.975804+0000 mgr.y (mgr.34107) 6697 : cluster [DBG] pgmap v7815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:54.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:41:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:41:54.249Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:41:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:54 smithi115 bash[45495]: audit 2022-04-22T20:41:53.048423+0000 mon.a (mon.0) 10123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:54.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:54 smithi115 bash[45495]: audit 2022-04-22T20:41:53.055154+0000 mon.a (mon.0) 10124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:54.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:54 smithi115 bash[46934]: audit 2022-04-22T20:41:53.048423+0000 mon.a (mon.0) 10123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:54.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:54 smithi115 bash[46934]: audit 2022-04-22T20:41:53.055154+0000 mon.a (mon.0) 10124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:54 smithi121 bash[36902]: audit 2022-04-22T20:41:53.048423+0000 mon.a (mon.0) 10123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:54 smithi121 bash[36902]: audit 2022-04-22T20:41:53.055154+0000 mon.a (mon.0) 10124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:55 smithi121 bash[36902]: cluster 2022-04-22T20:41:53.977064+0000 mgr.y (mgr.34107) 6698 : cluster [DBG] pgmap v7816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:55 smithi121 bash[36902]: audit 2022-04-22T20:41:54.504283+0000 mon.a (mon.0) 10125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:55 smithi115 bash[45495]: cluster 2022-04-22T20:41:53.977064+0000 mgr.y (mgr.34107) 6698 : cluster [DBG] pgmap v7816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:55.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:55 smithi115 bash[45495]: audit 2022-04-22T20:41:54.504283+0000 mon.a (mon.0) 10125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:55 smithi115 bash[46934]: cluster 2022-04-22T20:41:53.977064+0000 mgr.y (mgr.34107) 6698 : cluster [DBG] pgmap v7816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:55 smithi115 bash[46934]: audit 2022-04-22T20:41:54.504283+0000 mon.a (mon.0) 10125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:41:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:57 smithi121 bash[36902]: cluster 2022-04-22T20:41:55.978145+0000 mgr.y (mgr.34107) 6699 : cluster [DBG] pgmap v7817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:57 smithi115 bash[45495]: cluster 2022-04-22T20:41:55.978145+0000 mgr.y (mgr.34107) 6699 : cluster [DBG] pgmap v7817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:57.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:57 smithi115 bash[46934]: cluster 2022-04-22T20:41:55.978145+0000 mgr.y (mgr.34107) 6699 : cluster [DBG] pgmap v7817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:58.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:58 smithi115 bash[45495]: audit 2022-04-22T20:41:58.051123+0000 mon.a (mon.0) 10126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:58.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:58 smithi115 bash[45495]: audit 2022-04-22T20:41:58.058889+0000 mon.a (mon.0) 10127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:58.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:58 smithi115 bash[46934]: audit 2022-04-22T20:41:58.051123+0000 mon.a (mon.0) 10126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:58.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:58 smithi115 bash[46934]: audit 2022-04-22T20:41:58.058889+0000 mon.a (mon.0) 10127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:58 smithi121 bash[36902]: audit 2022-04-22T20:41:58.051123+0000 mon.a (mon.0) 10126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:41:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:58 smithi121 bash[36902]: audit 2022-04-22T20:41:58.058889+0000 mon.a (mon.0) 10127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:41:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:59 smithi121 bash[36902]: cluster 2022-04-22T20:41:57.979048+0000 mgr.y (mgr.34107) 6700 : cluster [DBG] pgmap v7818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:59 smithi121 bash[36902]: audit 2022-04-22T20:41:58.408456+0000 mon.a (mon.0) 10128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:41:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:59 smithi121 bash[36902]: audit 2022-04-22T20:41:58.411634+0000 mon.a (mon.0) 10129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:41:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:59 smithi121 bash[36902]: audit 2022-04-22T20:41:58.413608+0000 mon.a (mon.0) 10130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:41:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:41:59 smithi121 bash[36902]: audit 2022-04-22T20:41:59.024490+0000 mon.a (mon.0) 10131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:41:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[45495]: cluster 2022-04-22T20:41:57.979048+0000 mgr.y (mgr.34107) 6700 : cluster [DBG] pgmap v7818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[45495]: audit 2022-04-22T20:41:58.408456+0000 mon.a (mon.0) 10128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:41:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[45495]: audit 2022-04-22T20:41:58.411634+0000 mon.a (mon.0) 10129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:41:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[45495]: audit 2022-04-22T20:41:58.413608+0000 mon.a (mon.0) 10130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:41:59.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[45495]: audit 2022-04-22T20:41:59.024490+0000 mon.a (mon.0) 10131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:41:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[46934]: cluster 2022-04-22T20:41:57.979048+0000 mgr.y (mgr.34107) 6700 : cluster [DBG] pgmap v7818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:41:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[46934]: audit 2022-04-22T20:41:58.408456+0000 mon.a (mon.0) 10128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:41:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[46934]: audit 2022-04-22T20:41:58.411634+0000 mon.a (mon.0) 10129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:41:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[46934]: audit 2022-04-22T20:41:58.413608+0000 mon.a (mon.0) 10130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:41:59.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:41:59 smithi115 bash[46934]: audit 2022-04-22T20:41:59.024490+0000 mon.a (mon.0) 10131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:00 smithi121 bash[36902]: audit 2022-04-22T20:41:59.507610+0000 mon.a (mon.0) 10132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:00 smithi115 bash[46934]: audit 2022-04-22T20:41:59.507610+0000 mon.a (mon.0) 10132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:00 smithi115 bash[45495]: audit 2022-04-22T20:41:59.507610+0000 mon.a (mon.0) 10132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:01.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:01 smithi121 bash[36902]: cluster 2022-04-22T20:41:59.980134+0000 mgr.y (mgr.34107) 6701 : cluster [DBG] pgmap v7819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:01.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:01 smithi115 bash[45495]: cluster 2022-04-22T20:41:59.980134+0000 mgr.y (mgr.34107) 6701 : cluster [DBG] pgmap v7819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:01.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:01 smithi115 bash[46934]: cluster 2022-04-22T20:41:59.980134+0000 mgr.y (mgr.34107) 6701 : cluster [DBG] pgmap v7819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:02 smithi121 bash[36902]: cluster 2022-04-22T20:42:01.981166+0000 mgr.y (mgr.34107) 6702 : cluster [DBG] pgmap v7820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:02.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:02 smithi115 bash[45495]: cluster 2022-04-22T20:42:01.981166+0000 mgr.y (mgr.34107) 6702 : cluster [DBG] pgmap v7820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:02.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:02 smithi115 bash[46934]: cluster 2022-04-22T20:42:01.981166+0000 mgr.y (mgr.34107) 6702 : cluster [DBG] pgmap v7820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:02.796 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:02.367Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:04.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[45495]: audit 2022-04-22T20:42:02.872773+0000 mon.a (mon.0) 10133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[45495]: audit 2022-04-22T20:42:03.054232+0000 mon.a (mon.0) 10134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[45495]: audit 2022-04-22T20:42:03.062384+0000 mon.a (mon.0) 10135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[45495]: audit 2022-04-22T20:42:03.198980+0000 mon.a (mon.0) 10136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[46934]: audit 2022-04-22T20:42:02.872773+0000 mon.a (mon.0) 10133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[46934]: audit 2022-04-22T20:42:03.054232+0000 mon.a (mon.0) 10134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[46934]: audit 2022-04-22T20:42:03.062384+0000 mon.a (mon.0) 10135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:04.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:03 smithi115 bash[46934]: audit 2022-04-22T20:42:03.198980+0000 mon.a (mon.0) 10136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:03 smithi121 bash[36902]: audit 2022-04-22T20:42:02.872773+0000 mon.a (mon.0) 10133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:03 smithi121 bash[36902]: audit 2022-04-22T20:42:03.054232+0000 mon.a (mon.0) 10134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:03 smithi121 bash[36902]: audit 2022-04-22T20:42:03.062384+0000 mon.a (mon.0) 10135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:03 smithi121 bash[36902]: audit 2022-04-22T20:42:03.198980+0000 mon.a (mon.0) 10136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:42:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:04.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:05 smithi121 bash[36902]: cluster 2022-04-22T20:42:03.982343+0000 mgr.y (mgr.34107) 6703 : cluster [DBG] pgmap v7821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:05 smithi121 bash[36902]: audit 2022-04-22T20:42:04.510291+0000 mon.a (mon.0) 10137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:05 smithi115 bash[45495]: cluster 2022-04-22T20:42:03.982343+0000 mgr.y (mgr.34107) 6703 : cluster [DBG] pgmap v7821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:05.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:05 smithi115 bash[45495]: audit 2022-04-22T20:42:04.510291+0000 mon.a (mon.0) 10137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:05 smithi115 bash[46934]: cluster 2022-04-22T20:42:03.982343+0000 mgr.y (mgr.34107) 6703 : cluster [DBG] pgmap v7821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:05 smithi115 bash[46934]: audit 2022-04-22T20:42:04.510291+0000 mon.a (mon.0) 10137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:07 smithi121 bash[36902]: cluster 2022-04-22T20:42:05.983397+0000 mgr.y (mgr.34107) 6704 : cluster [DBG] pgmap v7822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:07 smithi115 bash[45495]: cluster 2022-04-22T20:42:05.983397+0000 mgr.y (mgr.34107) 6704 : cluster [DBG] pgmap v7822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:07 smithi115 bash[46934]: cluster 2022-04-22T20:42:05.983397+0000 mgr.y (mgr.34107) 6704 : cluster [DBG] pgmap v7822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:08 smithi121 bash[36902]: audit 2022-04-22T20:42:08.057321+0000 mon.a (mon.0) 10138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:08 smithi121 bash[36902]: audit 2022-04-22T20:42:08.065505+0000 mon.a (mon.0) 10139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:08 smithi115 bash[45495]: audit 2022-04-22T20:42:08.057321+0000 mon.a (mon.0) 10138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:08.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:08 smithi115 bash[45495]: audit 2022-04-22T20:42:08.065505+0000 mon.a (mon.0) 10139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:08 smithi115 bash[46934]: audit 2022-04-22T20:42:08.057321+0000 mon.a (mon.0) 10138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:08 smithi115 bash[46934]: audit 2022-04-22T20:42:08.065505+0000 mon.a (mon.0) 10139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:09 smithi121 bash[36902]: cluster 2022-04-22T20:42:07.984357+0000 mgr.y (mgr.34107) 6705 : cluster [DBG] pgmap v7823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:09.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:09 smithi115 bash[45495]: cluster 2022-04-22T20:42:07.984357+0000 mgr.y (mgr.34107) 6705 : cluster [DBG] pgmap v7823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:09 smithi115 bash[46934]: cluster 2022-04-22T20:42:07.984357+0000 mgr.y (mgr.34107) 6705 : cluster [DBG] pgmap v7823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:10 smithi121 bash[36902]: audit 2022-04-22T20:42:09.513151+0000 mon.a (mon.0) 10140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:10 smithi121 bash[36902]: cluster 2022-04-22T20:42:09.985428+0000 mgr.y (mgr.34107) 6706 : cluster [DBG] pgmap v7824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:10 smithi115 bash[45495]: audit 2022-04-22T20:42:09.513151+0000 mon.a (mon.0) 10140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:10 smithi115 bash[45495]: cluster 2022-04-22T20:42:09.985428+0000 mgr.y (mgr.34107) 6706 : cluster [DBG] pgmap v7824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:10 smithi115 bash[46934]: audit 2022-04-22T20:42:09.513151+0000 mon.a (mon.0) 10140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:10 smithi115 bash[46934]: cluster 2022-04-22T20:42:09.985428+0000 mgr.y (mgr.34107) 6706 : cluster [DBG] pgmap v7824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:12.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:12.369Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:13.461 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:42:13.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:13 smithi121 bash[36902]: cluster 2022-04-22T20:42:11.986477+0000 mgr.y (mgr.34107) 6707 : cluster [DBG] pgmap v7825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:13 smithi115 bash[45495]: cluster 2022-04-22T20:42:11.986477+0000 mgr.y (mgr.34107) 6707 : cluster [DBG] pgmap v7825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:13 smithi115 bash[46934]: cluster 2022-04-22T20:42:11.986477+0000 mgr.y (mgr.34107) 6707 : cluster [DBG] pgmap v7825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:13.797 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:42:13.797 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:42:13.797 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:42:13.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:42:13.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10s ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10s ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:42:13.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10s ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10s ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:42:13.799 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10s ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:42:14.203 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:42:14.203 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:42:14.203 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:14.204 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:42:14.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:42:14.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:14.205 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:42:14.207 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:42:14.207 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:42:14.207 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:42:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:14 smithi121 bash[36902]: audit 2022-04-22T20:42:13.060332+0000 mon.a (mon.0) 10141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:14 smithi121 bash[36902]: audit 2022-04-22T20:42:13.068811+0000 mon.a (mon.0) 10142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:14 smithi115 bash[45495]: audit 2022-04-22T20:42:13.060332+0000 mon.a (mon.0) 10141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:14 smithi115 bash[45495]: audit 2022-04-22T20:42:13.068811+0000 mon.a (mon.0) 10142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:14.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:14 smithi115 bash[46934]: audit 2022-04-22T20:42:13.060332+0000 mon.a (mon.0) 10141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:14 smithi115 bash[46934]: audit 2022-04-22T20:42:13.068811+0000 mon.a (mon.0) 10142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:15 smithi121 bash[36902]: audit 2022-04-22T20:42:13.436165+0000 mgr.y (mgr.34107) 6708 : audit [DBG] from='client.52285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:15 smithi121 bash[36902]: audit 2022-04-22T20:42:13.771846+0000 mgr.y (mgr.34107) 6709 : audit [DBG] from='client.52291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:15 smithi121 bash[36902]: cluster 2022-04-22T20:42:13.987550+0000 mgr.y (mgr.34107) 6710 : cluster [DBG] pgmap v7826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:15 smithi121 bash[36902]: audit 2022-04-22T20:42:14.196765+0000 mon.a (mon.0) 10143 : audit [DBG] from='client.? 172.21.15.115:0/3717134152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:15 smithi121 bash[36902]: audit 2022-04-22T20:42:14.516215+0000 mon.a (mon.0) 10144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[45495]: audit 2022-04-22T20:42:13.436165+0000 mgr.y (mgr.34107) 6708 : audit [DBG] from='client.52285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[45495]: audit 2022-04-22T20:42:13.771846+0000 mgr.y (mgr.34107) 6709 : audit [DBG] from='client.52291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[45495]: cluster 2022-04-22T20:42:13.987550+0000 mgr.y (mgr.34107) 6710 : cluster [DBG] pgmap v7826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[45495]: audit 2022-04-22T20:42:14.196765+0000 mon.a (mon.0) 10143 : audit [DBG] from='client.? 172.21.15.115:0/3717134152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:15.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[45495]: audit 2022-04-22T20:42:14.516215+0000 mon.a (mon.0) 10144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[46934]: audit 2022-04-22T20:42:13.436165+0000 mgr.y (mgr.34107) 6708 : audit [DBG] from='client.52285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[46934]: audit 2022-04-22T20:42:13.771846+0000 mgr.y (mgr.34107) 6709 : audit [DBG] from='client.52291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[46934]: cluster 2022-04-22T20:42:13.987550+0000 mgr.y (mgr.34107) 6710 : cluster [DBG] pgmap v7826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[46934]: audit 2022-04-22T20:42:14.196765+0000 mon.a (mon.0) 10143 : audit [DBG] from='client.? 172.21.15.115:0/3717134152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:15.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:15 smithi115 bash[46934]: audit 2022-04-22T20:42:14.516215+0000 mon.a (mon.0) 10144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:16 smithi121 bash[36902]: audit 2022-04-22T20:42:15.525350+0000 mon.a (mon.0) 10145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:42:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:16 smithi121 bash[36902]: audit 2022-04-22T20:42:15.526013+0000 mon.a (mon.0) 10146 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:42:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:16 smithi115 bash[45495]: audit 2022-04-22T20:42:15.525350+0000 mon.a (mon.0) 10145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:42:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:16 smithi115 bash[45495]: audit 2022-04-22T20:42:15.526013+0000 mon.a (mon.0) 10146 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:42:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:16 smithi115 bash[46934]: audit 2022-04-22T20:42:15.525350+0000 mon.a (mon.0) 10145 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:42:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:16 smithi115 bash[46934]: audit 2022-04-22T20:42:15.526013+0000 mon.a (mon.0) 10146 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:42:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:17 smithi121 bash[36902]: cluster 2022-04-22T20:42:15.988839+0000 mgr.y (mgr.34107) 6711 : cluster [DBG] pgmap v7827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:17 smithi115 bash[45495]: cluster 2022-04-22T20:42:15.988839+0000 mgr.y (mgr.34107) 6711 : cluster [DBG] pgmap v7827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:17 smithi115 bash[46934]: cluster 2022-04-22T20:42:15.988839+0000 mgr.y (mgr.34107) 6711 : cluster [DBG] pgmap v7827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:18 smithi121 bash[36902]: audit 2022-04-22T20:42:18.063260+0000 mon.a (mon.0) 10147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:18.734 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:18 smithi121 bash[36902]: audit 2022-04-22T20:42:18.071821+0000 mon.a (mon.0) 10148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:18.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:18 smithi115 bash[45495]: audit 2022-04-22T20:42:18.063260+0000 mon.a (mon.0) 10147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:18.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:18 smithi115 bash[45495]: audit 2022-04-22T20:42:18.071821+0000 mon.a (mon.0) 10148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:18.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:18 smithi115 bash[46934]: audit 2022-04-22T20:42:18.063260+0000 mon.a (mon.0) 10147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:18.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:18 smithi115 bash[46934]: audit 2022-04-22T20:42:18.071821+0000 mon.a (mon.0) 10148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:19 smithi121 bash[36902]: cluster 2022-04-22T20:42:17.989847+0000 mgr.y (mgr.34107) 6712 : cluster [DBG] pgmap v7828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:19 smithi115 bash[45495]: cluster 2022-04-22T20:42:17.989847+0000 mgr.y (mgr.34107) 6712 : cluster [DBG] pgmap v7828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:19 smithi115 bash[46934]: cluster 2022-04-22T20:42:17.989847+0000 mgr.y (mgr.34107) 6712 : cluster [DBG] pgmap v7828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:20 smithi121 bash[36902]: audit 2022-04-22T20:42:19.519257+0000 mon.a (mon.0) 10149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:20 smithi115 bash[45495]: audit 2022-04-22T20:42:19.519257+0000 mon.a (mon.0) 10149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:20 smithi115 bash[46934]: audit 2022-04-22T20:42:19.519257+0000 mon.a (mon.0) 10149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:21 smithi121 bash[36902]: cluster 2022-04-22T20:42:19.991038+0000 mgr.y (mgr.34107) 6713 : cluster [DBG] pgmap v7829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:21 smithi115 bash[45495]: cluster 2022-04-22T20:42:19.991038+0000 mgr.y (mgr.34107) 6713 : cluster [DBG] pgmap v7829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:21 smithi115 bash[46934]: cluster 2022-04-22T20:42:19.991038+0000 mgr.y (mgr.34107) 6713 : cluster [DBG] pgmap v7829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:22.381Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:23 smithi121 bash[36902]: cluster 2022-04-22T20:42:21.992192+0000 mgr.y (mgr.34107) 6714 : cluster [DBG] pgmap v7830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:23 smithi121 bash[36902]: audit 2022-04-22T20:42:23.066570+0000 mon.a (mon.0) 10150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:23 smithi121 bash[36902]: audit 2022-04-22T20:42:23.074882+0000 mon.a (mon.0) 10151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[45495]: cluster 2022-04-22T20:42:21.992192+0000 mgr.y (mgr.34107) 6714 : cluster [DBG] pgmap v7830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:23.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[45495]: audit 2022-04-22T20:42:23.066570+0000 mon.a (mon.0) 10150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:23.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[45495]: audit 2022-04-22T20:42:23.074882+0000 mon.a (mon.0) 10151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:23.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[46934]: cluster 2022-04-22T20:42:21.992192+0000 mgr.y (mgr.34107) 6714 : cluster [DBG] pgmap v7830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:23.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[46934]: audit 2022-04-22T20:42:23.066570+0000 mon.a (mon.0) 10150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:23.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:23 smithi115 bash[46934]: audit 2022-04-22T20:42:23.074882+0000 mon.a (mon.0) 10151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:24.250Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:25 smithi121 bash[36902]: cluster 2022-04-22T20:42:23.993370+0000 mgr.y (mgr.34107) 6715 : cluster [DBG] pgmap v7831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:25 smithi121 bash[36902]: audit 2022-04-22T20:42:24.522279+0000 mon.a (mon.0) 10152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:25 smithi115 bash[45495]: cluster 2022-04-22T20:42:23.993370+0000 mgr.y (mgr.34107) 6715 : cluster [DBG] pgmap v7831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:25 smithi115 bash[45495]: audit 2022-04-22T20:42:24.522279+0000 mon.a (mon.0) 10152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:25 smithi115 bash[46934]: cluster 2022-04-22T20:42:23.993370+0000 mgr.y (mgr.34107) 6715 : cluster [DBG] pgmap v7831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:25 smithi115 bash[46934]: audit 2022-04-22T20:42:24.522279+0000 mon.a (mon.0) 10152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:27 smithi121 bash[36902]: cluster 2022-04-22T20:42:25.994469+0000 mgr.y (mgr.34107) 6716 : cluster [DBG] pgmap v7832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:27 smithi115 bash[45495]: cluster 2022-04-22T20:42:25.994469+0000 mgr.y (mgr.34107) 6716 : cluster [DBG] pgmap v7832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:27 smithi115 bash[46934]: cluster 2022-04-22T20:42:25.994469+0000 mgr.y (mgr.34107) 6716 : cluster [DBG] pgmap v7832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:28 smithi121 bash[36902]: audit 2022-04-22T20:42:28.069282+0000 mon.a (mon.0) 10153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:28 smithi121 bash[36902]: audit 2022-04-22T20:42:28.078046+0000 mon.a (mon.0) 10154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:28.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:28 smithi115 bash[45495]: audit 2022-04-22T20:42:28.069282+0000 mon.a (mon.0) 10153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:28 smithi115 bash[45495]: audit 2022-04-22T20:42:28.078046+0000 mon.a (mon.0) 10154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:28 smithi115 bash[46934]: audit 2022-04-22T20:42:28.069282+0000 mon.a (mon.0) 10153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:28 smithi115 bash[46934]: audit 2022-04-22T20:42:28.078046+0000 mon.a (mon.0) 10154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:29 smithi121 bash[36902]: cluster 2022-04-22T20:42:27.995544+0000 mgr.y (mgr.34107) 6717 : cluster [DBG] pgmap v7833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:29 smithi115 bash[45495]: cluster 2022-04-22T20:42:27.995544+0000 mgr.y (mgr.34107) 6717 : cluster [DBG] pgmap v7833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:29 smithi115 bash[46934]: cluster 2022-04-22T20:42:27.995544+0000 mgr.y (mgr.34107) 6717 : cluster [DBG] pgmap v7833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:30 smithi121 bash[36902]: audit 2022-04-22T20:42:29.525255+0000 mon.a (mon.0) 10155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:30 smithi121 bash[36902]: cluster 2022-04-22T20:42:29.996721+0000 mgr.y (mgr.34107) 6718 : cluster [DBG] pgmap v7834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:30 smithi115 bash[45495]: audit 2022-04-22T20:42:29.525255+0000 mon.a (mon.0) 10155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:30 smithi115 bash[45495]: cluster 2022-04-22T20:42:29.996721+0000 mgr.y (mgr.34107) 6718 : cluster [DBG] pgmap v7834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:30 smithi115 bash[46934]: audit 2022-04-22T20:42:29.525255+0000 mon.a (mon.0) 10155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:30 smithi115 bash[46934]: cluster 2022-04-22T20:42:29.996721+0000 mgr.y (mgr.34107) 6718 : cluster [DBG] pgmap v7834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:32.370Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:33 smithi121 bash[36902]: cluster 2022-04-22T20:42:31.997877+0000 mgr.y (mgr.34107) 6719 : cluster [DBG] pgmap v7835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:33.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:33 smithi115 bash[45495]: cluster 2022-04-22T20:42:31.997877+0000 mgr.y (mgr.34107) 6719 : cluster [DBG] pgmap v7835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:33 smithi115 bash[46934]: cluster 2022-04-22T20:42:31.997877+0000 mgr.y (mgr.34107) 6719 : cluster [DBG] pgmap v7835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:34 smithi121 bash[36902]: audit 2022-04-22T20:42:33.072351+0000 mon.a (mon.0) 10156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:34 smithi121 bash[36902]: audit 2022-04-22T20:42:33.081410+0000 mon.a (mon.0) 10157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:34 smithi115 bash[45495]: audit 2022-04-22T20:42:33.072351+0000 mon.a (mon.0) 10156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:34 smithi115 bash[45495]: audit 2022-04-22T20:42:33.081410+0000 mon.a (mon.0) 10157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:34.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:34.251Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:34 smithi115 bash[46934]: audit 2022-04-22T20:42:33.072351+0000 mon.a (mon.0) 10156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:34 smithi115 bash[46934]: audit 2022-04-22T20:42:33.081410+0000 mon.a (mon.0) 10157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:35 smithi121 bash[36902]: cluster 2022-04-22T20:42:33.999085+0000 mgr.y (mgr.34107) 6720 : cluster [DBG] pgmap v7836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:35 smithi121 bash[36902]: audit 2022-04-22T20:42:34.528388+0000 mon.a (mon.0) 10158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:35 smithi115 bash[46934]: cluster 2022-04-22T20:42:33.999085+0000 mgr.y (mgr.34107) 6720 : cluster [DBG] pgmap v7836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:35 smithi115 bash[46934]: audit 2022-04-22T20:42:34.528388+0000 mon.a (mon.0) 10158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:35.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:35 smithi115 bash[45495]: cluster 2022-04-22T20:42:33.999085+0000 mgr.y (mgr.34107) 6720 : cluster [DBG] pgmap v7836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:35.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:35 smithi115 bash[45495]: audit 2022-04-22T20:42:34.528388+0000 mon.a (mon.0) 10158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:37 smithi121 bash[36902]: cluster 2022-04-22T20:42:35.999865+0000 mgr.y (mgr.34107) 6721 : cluster [DBG] pgmap v7837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:37 smithi115 bash[45495]: cluster 2022-04-22T20:42:35.999865+0000 mgr.y (mgr.34107) 6721 : cluster [DBG] pgmap v7837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:37 smithi115 bash[46934]: cluster 2022-04-22T20:42:35.999865+0000 mgr.y (mgr.34107) 6721 : cluster [DBG] pgmap v7837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:38 smithi121 bash[36902]: audit 2022-04-22T20:42:38.075248+0000 mon.a (mon.0) 10159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:38 smithi121 bash[36902]: audit 2022-04-22T20:42:38.084576+0000 mon.a (mon.0) 10160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:38 smithi115 bash[45495]: audit 2022-04-22T20:42:38.075248+0000 mon.a (mon.0) 10159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:38.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:38 smithi115 bash[45495]: audit 2022-04-22T20:42:38.084576+0000 mon.a (mon.0) 10160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:38 smithi115 bash[46934]: audit 2022-04-22T20:42:38.075248+0000 mon.a (mon.0) 10159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:38 smithi115 bash[46934]: audit 2022-04-22T20:42:38.084576+0000 mon.a (mon.0) 10160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:39 smithi121 bash[36902]: cluster 2022-04-22T20:42:38.000860+0000 mgr.y (mgr.34107) 6722 : cluster [DBG] pgmap v7838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:39 smithi115 bash[45495]: cluster 2022-04-22T20:42:38.000860+0000 mgr.y (mgr.34107) 6722 : cluster [DBG] pgmap v7838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:39 smithi115 bash[46934]: cluster 2022-04-22T20:42:38.000860+0000 mgr.y (mgr.34107) 6722 : cluster [DBG] pgmap v7838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:40 smithi121 bash[36902]: audit 2022-04-22T20:42:39.531247+0000 mon.a (mon.0) 10161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:40 smithi121 bash[36902]: cluster 2022-04-22T20:42:40.002286+0000 mgr.y (mgr.34107) 6723 : cluster [DBG] pgmap v7839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:40 smithi115 bash[45495]: audit 2022-04-22T20:42:39.531247+0000 mon.a (mon.0) 10161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:40 smithi115 bash[45495]: cluster 2022-04-22T20:42:40.002286+0000 mgr.y (mgr.34107) 6723 : cluster [DBG] pgmap v7839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:40 smithi115 bash[46934]: audit 2022-04-22T20:42:39.531247+0000 mon.a (mon.0) 10161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:40 smithi115 bash[46934]: cluster 2022-04-22T20:42:40.002286+0000 mgr.y (mgr.34107) 6723 : cluster [DBG] pgmap v7839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:42.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:42.369Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:43 smithi121 bash[36902]: cluster 2022-04-22T20:42:42.003367+0000 mgr.y (mgr.34107) 6724 : cluster [DBG] pgmap v7840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:43.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:43 smithi115 bash[45495]: cluster 2022-04-22T20:42:42.003367+0000 mgr.y (mgr.34107) 6724 : cluster [DBG] pgmap v7840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:43 smithi115 bash[46934]: cluster 2022-04-22T20:42:42.003367+0000 mgr.y (mgr.34107) 6724 : cluster [DBG] pgmap v7840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:44 smithi121 bash[36902]: audit 2022-04-22T20:42:43.078332+0000 mon.a (mon.0) 10162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:44 smithi121 bash[36902]: audit 2022-04-22T20:42:43.087990+0000 mon.a (mon.0) 10163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:44.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:44.252Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:44 smithi115 bash[45495]: audit 2022-04-22T20:42:43.078332+0000 mon.a (mon.0) 10162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:44.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:44 smithi115 bash[45495]: audit 2022-04-22T20:42:43.087990+0000 mon.a (mon.0) 10163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:44 smithi115 bash[46934]: audit 2022-04-22T20:42:43.078332+0000 mon.a (mon.0) 10162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:44 smithi115 bash[46934]: audit 2022-04-22T20:42:43.087990+0000 mon.a (mon.0) 10163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:44.577 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:42:44.920 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:42:44.920 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.8M - ba2b418f427c 9255ba08f72f 2022-04-22T20:42:44.920 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 42s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:42:44.920 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 42s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 42s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 486M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 339M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 42s ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 220M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 42s ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:42:44.921 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 786M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 490M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 42s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 42s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 42s ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 42s ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:42:44.922 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 42s ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:42:45.327 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:42:45.327 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:42:45.328 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:42:45.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:45.328 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:42:45.328 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:42:45.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:45.329 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:42:45.329 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:42:45.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:45.329 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:42:45.330 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:42:45.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:42:45.330 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:42:45.330 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:42:45.331 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:42:45.331 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:42:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:45 smithi121 bash[36902]: cluster 2022-04-22T20:42:44.004508+0000 mgr.y (mgr.34107) 6725 : cluster [DBG] pgmap v7841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:45 smithi121 bash[36902]: audit 2022-04-22T20:42:44.534243+0000 mon.a (mon.0) 10164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:45 smithi121 bash[36902]: audit 2022-04-22T20:42:45.320033+0000 mon.a (mon.0) 10165 : audit [DBG] from='client.? 172.21.15.115:0/3763700098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[45495]: cluster 2022-04-22T20:42:44.004508+0000 mgr.y (mgr.34107) 6725 : cluster [DBG] pgmap v7841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[45495]: audit 2022-04-22T20:42:44.534243+0000 mon.a (mon.0) 10164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[45495]: audit 2022-04-22T20:42:45.320033+0000 mon.a (mon.0) 10165 : audit [DBG] from='client.? 172.21.15.115:0/3763700098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[46934]: cluster 2022-04-22T20:42:44.004508+0000 mgr.y (mgr.34107) 6725 : cluster [DBG] pgmap v7841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[46934]: audit 2022-04-22T20:42:44.534243+0000 mon.a (mon.0) 10164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:45 smithi115 bash[46934]: audit 2022-04-22T20:42:45.320033+0000 mon.a (mon.0) 10165 : audit [DBG] from='client.? 172.21.15.115:0/3763700098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:42:46.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:46 smithi121 bash[36902]: audit 2022-04-22T20:42:44.553534+0000 mgr.y (mgr.34107) 6726 : audit [DBG] from='client.42372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:46.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:46 smithi121 bash[36902]: audit 2022-04-22T20:42:44.893956+0000 mgr.y (mgr.34107) 6727 : audit [DBG] from='client.52309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:46 smithi115 bash[45495]: audit 2022-04-22T20:42:44.553534+0000 mgr.y (mgr.34107) 6726 : audit [DBG] from='client.42372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:46.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:46 smithi115 bash[45495]: audit 2022-04-22T20:42:44.893956+0000 mgr.y (mgr.34107) 6727 : audit [DBG] from='client.52309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:46 smithi115 bash[46934]: audit 2022-04-22T20:42:44.553534+0000 mgr.y (mgr.34107) 6726 : audit [DBG] from='client.42372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:46 smithi115 bash[46934]: audit 2022-04-22T20:42:44.893956+0000 mgr.y (mgr.34107) 6727 : audit [DBG] from='client.52309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:42:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:47 smithi121 bash[36902]: cluster 2022-04-22T20:42:46.005231+0000 mgr.y (mgr.34107) 6728 : cluster [DBG] pgmap v7842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:47 smithi115 bash[46934]: cluster 2022-04-22T20:42:46.005231+0000 mgr.y (mgr.34107) 6728 : cluster [DBG] pgmap v7842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:47.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:47 smithi115 bash[45495]: cluster 2022-04-22T20:42:46.005231+0000 mgr.y (mgr.34107) 6728 : cluster [DBG] pgmap v7842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:48 smithi121 bash[36902]: cluster 2022-04-22T20:42:48.006206+0000 mgr.y (mgr.34107) 6729 : cluster [DBG] pgmap v7843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:48 smithi121 bash[36902]: audit 2022-04-22T20:42:48.080965+0000 mon.a (mon.0) 10166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:48 smithi121 bash[36902]: audit 2022-04-22T20:42:48.091356+0000 mon.a (mon.0) 10167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[45495]: cluster 2022-04-22T20:42:48.006206+0000 mgr.y (mgr.34107) 6729 : cluster [DBG] pgmap v7843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[45495]: audit 2022-04-22T20:42:48.080965+0000 mon.a (mon.0) 10166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[45495]: audit 2022-04-22T20:42:48.091356+0000 mon.a (mon.0) 10167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[46934]: cluster 2022-04-22T20:42:48.006206+0000 mgr.y (mgr.34107) 6729 : cluster [DBG] pgmap v7843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[46934]: audit 2022-04-22T20:42:48.080965+0000 mon.a (mon.0) 10166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:48 smithi115 bash[46934]: audit 2022-04-22T20:42:48.091356+0000 mon.a (mon.0) 10167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:50 smithi121 bash[36902]: audit 2022-04-22T20:42:49.537029+0000 mon.a (mon.0) 10168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:50 smithi115 bash[45495]: audit 2022-04-22T20:42:49.537029+0000 mon.a (mon.0) 10168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:50.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:50 smithi115 bash[46934]: audit 2022-04-22T20:42:49.537029+0000 mon.a (mon.0) 10168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:51 smithi121 bash[36902]: cluster 2022-04-22T20:42:50.007345+0000 mgr.y (mgr.34107) 6730 : cluster [DBG] pgmap v7844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:51 smithi115 bash[45495]: cluster 2022-04-22T20:42:50.007345+0000 mgr.y (mgr.34107) 6730 : cluster [DBG] pgmap v7844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:51 smithi115 bash[46934]: cluster 2022-04-22T20:42:50.007345+0000 mgr.y (mgr.34107) 6730 : cluster [DBG] pgmap v7844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:52 smithi121 bash[36902]: cluster 2022-04-22T20:42:52.008358+0000 mgr.y (mgr.34107) 6731 : cluster [DBG] pgmap v7845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:52.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:52 smithi115 bash[45495]: cluster 2022-04-22T20:42:52.008358+0000 mgr.y (mgr.34107) 6731 : cluster [DBG] pgmap v7845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:52.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:52 smithi115 bash[46934]: cluster 2022-04-22T20:42:52.008358+0000 mgr.y (mgr.34107) 6731 : cluster [DBG] pgmap v7845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:52.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:52.370Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:53 smithi121 bash[36902]: audit 2022-04-22T20:42:53.084448+0000 mon.a (mon.0) 10169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:53 smithi121 bash[36902]: audit 2022-04-22T20:42:53.094645+0000 mon.a (mon.0) 10170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:53 smithi115 bash[45495]: audit 2022-04-22T20:42:53.084448+0000 mon.a (mon.0) 10169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:53.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:53 smithi115 bash[45495]: audit 2022-04-22T20:42:53.094645+0000 mon.a (mon.0) 10170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:53 smithi115 bash[46934]: audit 2022-04-22T20:42:53.084448+0000 mon.a (mon.0) 10169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:53 smithi115 bash[46934]: audit 2022-04-22T20:42:53.094645+0000 mon.a (mon.0) 10170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:54.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:42:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:42:54.253Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:55 smithi121 bash[36902]: cluster 2022-04-22T20:42:54.009549+0000 mgr.y (mgr.34107) 6732 : cluster [DBG] pgmap v7846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:55 smithi121 bash[36902]: audit 2022-04-22T20:42:54.539998+0000 mon.a (mon.0) 10171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:55 smithi115 bash[45495]: cluster 2022-04-22T20:42:54.009549+0000 mgr.y (mgr.34107) 6732 : cluster [DBG] pgmap v7846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:55.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:55 smithi115 bash[45495]: audit 2022-04-22T20:42:54.539998+0000 mon.a (mon.0) 10171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:55 smithi115 bash[46934]: cluster 2022-04-22T20:42:54.009549+0000 mgr.y (mgr.34107) 6732 : cluster [DBG] pgmap v7846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:55.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:55 smithi115 bash[46934]: audit 2022-04-22T20:42:54.539998+0000 mon.a (mon.0) 10171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:42:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:57 smithi121 bash[36902]: cluster 2022-04-22T20:42:56.010527+0000 mgr.y (mgr.34107) 6733 : cluster [DBG] pgmap v7847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:57.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:57 smithi115 bash[45495]: cluster 2022-04-22T20:42:56.010527+0000 mgr.y (mgr.34107) 6733 : cluster [DBG] pgmap v7847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:57.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:57 smithi115 bash[46934]: cluster 2022-04-22T20:42:56.010527+0000 mgr.y (mgr.34107) 6733 : cluster [DBG] pgmap v7847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:58 smithi121 bash[36902]: audit 2022-04-22T20:42:58.087605+0000 mon.a (mon.0) 10172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:58 smithi121 bash[36902]: audit 2022-04-22T20:42:58.097893+0000 mon.a (mon.0) 10173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:58.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:58 smithi115 bash[45495]: audit 2022-04-22T20:42:58.087605+0000 mon.a (mon.0) 10172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:58.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:58 smithi115 bash[45495]: audit 2022-04-22T20:42:58.097893+0000 mon.a (mon.0) 10173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:58 smithi115 bash[46934]: audit 2022-04-22T20:42:58.087605+0000 mon.a (mon.0) 10172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:42:58.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:58 smithi115 bash[46934]: audit 2022-04-22T20:42:58.097893+0000 mon.a (mon.0) 10173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:42:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:42:59 smithi121 bash[36902]: cluster 2022-04-22T20:42:58.011566+0000 mgr.y (mgr.34107) 6734 : cluster [DBG] pgmap v7848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:42:59 smithi115 bash[45495]: cluster 2022-04-22T20:42:58.011566+0000 mgr.y (mgr.34107) 6734 : cluster [DBG] pgmap v7848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:42:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:42:59 smithi115 bash[46934]: cluster 2022-04-22T20:42:58.011566+0000 mgr.y (mgr.34107) 6734 : cluster [DBG] pgmap v7848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:00 smithi121 bash[36902]: audit 2022-04-22T20:42:59.543183+0000 mon.a (mon.0) 10174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:00 smithi121 bash[36902]: cluster 2022-04-22T20:43:00.012697+0000 mgr.y (mgr.34107) 6735 : cluster [DBG] pgmap v7849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:00 smithi115 bash[45495]: audit 2022-04-22T20:42:59.543183+0000 mon.a (mon.0) 10174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:00 smithi115 bash[45495]: cluster 2022-04-22T20:43:00.012697+0000 mgr.y (mgr.34107) 6735 : cluster [DBG] pgmap v7849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:00 smithi115 bash[46934]: audit 2022-04-22T20:42:59.543183+0000 mon.a (mon.0) 10174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:00 smithi115 bash[46934]: cluster 2022-04-22T20:43:00.012697+0000 mgr.y (mgr.34107) 6735 : cluster [DBG] pgmap v7849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:02.370Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:03 smithi121 bash[36902]: cluster 2022-04-22T20:43:02.013806+0000 mgr.y (mgr.34107) 6736 : cluster [DBG] pgmap v7850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:03 smithi115 bash[45495]: cluster 2022-04-22T20:43:02.013806+0000 mgr.y (mgr.34107) 6736 : cluster [DBG] pgmap v7850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:03 smithi115 bash[46934]: cluster 2022-04-22T20:43:02.013806+0000 mgr.y (mgr.34107) 6736 : cluster [DBG] pgmap v7850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.090714+0000 mon.a (mon.0) 10175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.101704+0000 mon.a (mon.0) 10176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.206929+0000 mon.a (mon.0) 10177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:43:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.209846+0000 mon.a (mon.0) 10178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:43:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.211664+0000 mon.a (mon.0) 10179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:43:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.557536+0000 mon.a (mon.0) 10180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:04 smithi121 bash[36902]: audit 2022-04-22T20:43:03.562542+0000 mon.a (mon.0) 10181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:04.252Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.090714+0000 mon.a (mon.0) 10175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.101704+0000 mon.a (mon.0) 10176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.206929+0000 mon.a (mon.0) 10177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.209846+0000 mon.a (mon.0) 10178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.211664+0000 mon.a (mon.0) 10179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.557536+0000 mon.a (mon.0) 10180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[45495]: audit 2022-04-22T20:43:03.562542+0000 mon.a (mon.0) 10181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.090714+0000 mon.a (mon.0) 10175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.101704+0000 mon.a (mon.0) 10176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.206929+0000 mon.a (mon.0) 10177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:43:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.209846+0000 mon.a (mon.0) 10178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:43:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.211664+0000 mon.a (mon.0) 10179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:43:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.557536+0000 mon.a (mon.0) 10180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:04.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:04 smithi115 bash[46934]: audit 2022-04-22T20:43:03.562542+0000 mon.a (mon.0) 10181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:05 smithi121 bash[36902]: cluster 2022-04-22T20:43:04.015009+0000 mgr.y (mgr.34107) 6737 : cluster [DBG] pgmap v7851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:05 smithi121 bash[36902]: audit 2022-04-22T20:43:04.546133+0000 mon.a (mon.0) 10182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:05 smithi115 bash[45495]: cluster 2022-04-22T20:43:04.015009+0000 mgr.y (mgr.34107) 6737 : cluster [DBG] pgmap v7851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:05 smithi115 bash[45495]: audit 2022-04-22T20:43:04.546133+0000 mon.a (mon.0) 10182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:05 smithi115 bash[46934]: cluster 2022-04-22T20:43:04.015009+0000 mgr.y (mgr.34107) 6737 : cluster [DBG] pgmap v7851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:05 smithi115 bash[46934]: audit 2022-04-22T20:43:04.546133+0000 mon.a (mon.0) 10182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:07.694 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:07 smithi115 bash[45495]: cluster 2022-04-22T20:43:06.016027+0000 mgr.y (mgr.34107) 6738 : cluster [DBG] pgmap v7852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:07.695 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:07 smithi115 bash[46934]: cluster 2022-04-22T20:43:06.016027+0000 mgr.y (mgr.34107) 6738 : cluster [DBG] pgmap v7852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:07 smithi121 bash[36902]: cluster 2022-04-22T20:43:06.016027+0000 mgr.y (mgr.34107) 6738 : cluster [DBG] pgmap v7852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:08 smithi121 bash[36902]: audit 2022-04-22T20:43:07.704781+0000 mon.a (mon.0) 10183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:08 smithi121 bash[36902]: cluster 2022-04-22T20:43:08.016890+0000 mgr.y (mgr.34107) 6739 : cluster [DBG] pgmap v7853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:08 smithi121 bash[36902]: audit 2022-04-22T20:43:08.093412+0000 mon.a (mon.0) 10184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:08 smithi121 bash[36902]: audit 2022-04-22T20:43:08.105080+0000 mon.a (mon.0) 10185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:09.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[45495]: audit 2022-04-22T20:43:07.704781+0000 mon.a (mon.0) 10183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[45495]: cluster 2022-04-22T20:43:08.016890+0000 mgr.y (mgr.34107) 6739 : cluster [DBG] pgmap v7853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[45495]: audit 2022-04-22T20:43:08.093412+0000 mon.a (mon.0) 10184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[45495]: audit 2022-04-22T20:43:08.105080+0000 mon.a (mon.0) 10185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[46934]: audit 2022-04-22T20:43:07.704781+0000 mon.a (mon.0) 10183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[46934]: cluster 2022-04-22T20:43:08.016890+0000 mgr.y (mgr.34107) 6739 : cluster [DBG] pgmap v7853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[46934]: audit 2022-04-22T20:43:08.093412+0000 mon.a (mon.0) 10184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:09.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:08 smithi115 bash[46934]: audit 2022-04-22T20:43:08.105080+0000 mon.a (mon.0) 10185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:10 smithi121 bash[36902]: audit 2022-04-22T20:43:09.549061+0000 mon.a (mon.0) 10186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:10 smithi115 bash[45495]: audit 2022-04-22T20:43:09.549061+0000 mon.a (mon.0) 10186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:10 smithi115 bash[46934]: audit 2022-04-22T20:43:09.549061+0000 mon.a (mon.0) 10186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:11 smithi121 bash[36902]: cluster 2022-04-22T20:43:10.017911+0000 mgr.y (mgr.34107) 6740 : cluster [DBG] pgmap v7854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:11.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:11 smithi115 bash[45495]: cluster 2022-04-22T20:43:10.017911+0000 mgr.y (mgr.34107) 6740 : cluster [DBG] pgmap v7854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:11.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:11 smithi115 bash[46934]: cluster 2022-04-22T20:43:10.017911+0000 mgr.y (mgr.34107) 6740 : cluster [DBG] pgmap v7854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:12.719 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:12.368Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:12.720 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:12 smithi115 bash[45495]: cluster 2022-04-22T20:43:12.019029+0000 mgr.y (mgr.34107) 6741 : cluster [DBG] pgmap v7855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:12.720 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:12 smithi115 bash[46934]: cluster 2022-04-22T20:43:12.019029+0000 mgr.y (mgr.34107) 6741 : cluster [DBG] pgmap v7855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:12 smithi121 bash[36902]: cluster 2022-04-22T20:43:12.019029+0000 mgr.y (mgr.34107) 6741 : cluster [DBG] pgmap v7855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:13 smithi121 bash[36902]: audit 2022-04-22T20:43:13.096054+0000 mon.a (mon.0) 10187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:13 smithi121 bash[36902]: audit 2022-04-22T20:43:13.108156+0000 mon.a (mon.0) 10188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:13 smithi115 bash[45495]: audit 2022-04-22T20:43:13.096054+0000 mon.a (mon.0) 10187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:13.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:13 smithi115 bash[45495]: audit 2022-04-22T20:43:13.108156+0000 mon.a (mon.0) 10188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:13 smithi115 bash[46934]: audit 2022-04-22T20:43:13.096054+0000 mon.a (mon.0) 10187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:13.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:13 smithi115 bash[46934]: audit 2022-04-22T20:43:13.108156+0000 mon.a (mon.0) 10188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:14.548 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:14.251Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:15.708 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:43:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:15 smithi121 bash[36902]: cluster 2022-04-22T20:43:14.020204+0000 mgr.y (mgr.34107) 6742 : cluster [DBG] pgmap v7856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:15 smithi121 bash[36902]: audit 2022-04-22T20:43:14.552076+0000 mon.a (mon.0) 10189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:15 smithi115 bash[45495]: cluster 2022-04-22T20:43:14.020204+0000 mgr.y (mgr.34107) 6742 : cluster [DBG] pgmap v7856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:15 smithi115 bash[45495]: audit 2022-04-22T20:43:14.552076+0000 mon.a (mon.0) 10189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:15 smithi115 bash[46934]: cluster 2022-04-22T20:43:14.020204+0000 mgr.y (mgr.34107) 6742 : cluster [DBG] pgmap v7856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:15 smithi115 bash[46934]: audit 2022-04-22T20:43:14.552076+0000 mon.a (mon.0) 10189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:16.071 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:43:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8s ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:43:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 73s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:43:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:43:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 73s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:43:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 73s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:43:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:43:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8s ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:43:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 73s ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:43:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8s ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:43:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8s ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:43:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 73s ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:43:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8s ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 73s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 73s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:43:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 73s ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:43:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 73s ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:43:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 73s ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:43:16.472 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:43:16.472 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:43:16.472 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:43:16.472 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:43:16.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:43:16.474 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:43:16.474 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:43:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:16 smithi121 bash[36902]: audit 2022-04-22T20:43:15.538980+0000 mon.a (mon.0) 10190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:43:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:16 smithi121 bash[36902]: audit 2022-04-22T20:43:15.539270+0000 mon.a (mon.0) 10191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:43:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:16 smithi115 bash[45495]: audit 2022-04-22T20:43:15.538980+0000 mon.a (mon.0) 10190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:43:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:16 smithi115 bash[45495]: audit 2022-04-22T20:43:15.539270+0000 mon.a (mon.0) 10191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:43:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:16 smithi115 bash[46934]: audit 2022-04-22T20:43:15.538980+0000 mon.a (mon.0) 10190 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:43:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:16 smithi115 bash[46934]: audit 2022-04-22T20:43:15.539270+0000 mon.a (mon.0) 10191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:43:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: audit 2022-04-22T20:43:15.686202+0000 mgr.y (mgr.34107) 6743 : audit [DBG] from='client.42390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: cluster 2022-04-22T20:43:16.020970+0000 mgr.y (mgr.34107) 6744 : cluster [DBG] pgmap v7857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: audit 2022-04-22T20:43:16.044584+0000 mgr.y (mgr.34107) 6745 : audit [DBG] from='client.52327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: audit 2022-04-22T20:43:16.465716+0000 mon.c (mon.1) 258 : audit [DBG] from='client.? 172.21.15.115:0/3552728645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: cephadm 2022-04-22T20:43:16.792646+0000 mgr.y (mgr.34107) 6746 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:43:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: audit 2022-04-22T20:43:16.803921+0000 mon.a (mon.0) 10192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:17 smithi121 bash[36902]: audit 2022-04-22T20:43:16.811133+0000 mon.a (mon.0) 10193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: audit 2022-04-22T20:43:15.686202+0000 mgr.y (mgr.34107) 6743 : audit [DBG] from='client.42390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: cluster 2022-04-22T20:43:16.020970+0000 mgr.y (mgr.34107) 6744 : cluster [DBG] pgmap v7857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: audit 2022-04-22T20:43:16.044584+0000 mgr.y (mgr.34107) 6745 : audit [DBG] from='client.52327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: audit 2022-04-22T20:43:16.465716+0000 mon.c (mon.1) 258 : audit [DBG] from='client.? 172.21.15.115:0/3552728645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: cephadm 2022-04-22T20:43:16.792646+0000 mgr.y (mgr.34107) 6746 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: audit 2022-04-22T20:43:16.803921+0000 mon.a (mon.0) 10192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:17.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[46934]: audit 2022-04-22T20:43:16.811133+0000 mon.a (mon.0) 10193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: audit 2022-04-22T20:43:15.686202+0000 mgr.y (mgr.34107) 6743 : audit [DBG] from='client.42390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: cluster 2022-04-22T20:43:16.020970+0000 mgr.y (mgr.34107) 6744 : cluster [DBG] pgmap v7857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: audit 2022-04-22T20:43:16.044584+0000 mgr.y (mgr.34107) 6745 : audit [DBG] from='client.52327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: audit 2022-04-22T20:43:16.465716+0000 mon.c (mon.1) 258 : audit [DBG] from='client.? 172.21.15.115:0/3552728645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: cephadm 2022-04-22T20:43:16.792646+0000 mgr.y (mgr.34107) 6746 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: audit 2022-04-22T20:43:16.803921+0000 mon.a (mon.0) 10192 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:17.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:17 smithi115 bash[45495]: audit 2022-04-22T20:43:16.811133+0000 mon.a (mon.0) 10193 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:43:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:18 smithi121 bash[36902]: cluster 2022-04-22T20:43:18.021926+0000 mgr.y (mgr.34107) 6747 : cluster [DBG] pgmap v7858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:18 smithi121 bash[36902]: audit 2022-04-22T20:43:18.098775+0000 mon.a (mon.0) 10194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:18 smithi121 bash[36902]: audit 2022-04-22T20:43:18.111297+0000 mon.a (mon.0) 10195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[45495]: cluster 2022-04-22T20:43:18.021926+0000 mgr.y (mgr.34107) 6747 : cluster [DBG] pgmap v7858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[45495]: audit 2022-04-22T20:43:18.098775+0000 mon.a (mon.0) 10194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[45495]: audit 2022-04-22T20:43:18.111297+0000 mon.a (mon.0) 10195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[46934]: cluster 2022-04-22T20:43:18.021926+0000 mgr.y (mgr.34107) 6747 : cluster [DBG] pgmap v7858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[46934]: audit 2022-04-22T20:43:18.098775+0000 mon.a (mon.0) 10194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:18.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:18 smithi115 bash[46934]: audit 2022-04-22T20:43:18.111297+0000 mon.a (mon.0) 10195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:20 smithi121 bash[36902]: audit 2022-04-22T20:43:19.555004+0000 mon.a (mon.0) 10196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:20 smithi115 bash[45495]: audit 2022-04-22T20:43:19.555004+0000 mon.a (mon.0) 10196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:20 smithi115 bash[46934]: audit 2022-04-22T20:43:19.555004+0000 mon.a (mon.0) 10196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:21 smithi121 bash[36902]: cluster 2022-04-22T20:43:20.023028+0000 mgr.y (mgr.34107) 6748 : cluster [DBG] pgmap v7859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:21 smithi115 bash[45495]: cluster 2022-04-22T20:43:20.023028+0000 mgr.y (mgr.34107) 6748 : cluster [DBG] pgmap v7859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:21 smithi115 bash[46934]: cluster 2022-04-22T20:43:20.023028+0000 mgr.y (mgr.34107) 6748 : cluster [DBG] pgmap v7859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:22 smithi121 bash[36902]: cluster 2022-04-22T20:43:22.024094+0000 mgr.y (mgr.34107) 6749 : cluster [DBG] pgmap v7860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:22 smithi115 bash[45495]: cluster 2022-04-22T20:43:22.024094+0000 mgr.y (mgr.34107) 6749 : cluster [DBG] pgmap v7860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:22.370Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:22 smithi115 bash[46934]: cluster 2022-04-22T20:43:22.024094+0000 mgr.y (mgr.34107) 6749 : cluster [DBG] pgmap v7860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:23 smithi121 bash[36902]: audit 2022-04-22T20:43:23.101852+0000 mon.a (mon.0) 10197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:23 smithi121 bash[36902]: audit 2022-04-22T20:43:23.114747+0000 mon.a (mon.0) 10198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:23 smithi115 bash[45495]: audit 2022-04-22T20:43:23.101852+0000 mon.a (mon.0) 10197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:23 smithi115 bash[45495]: audit 2022-04-22T20:43:23.114747+0000 mon.a (mon.0) 10198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:23 smithi115 bash[46934]: audit 2022-04-22T20:43:23.101852+0000 mon.a (mon.0) 10197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:23 smithi115 bash[46934]: audit 2022-04-22T20:43:23.114747+0000 mon.a (mon.0) 10198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:24.255Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:25 smithi121 bash[36902]: cluster 2022-04-22T20:43:24.025287+0000 mgr.y (mgr.34107) 6750 : cluster [DBG] pgmap v7861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:25 smithi121 bash[36902]: audit 2022-04-22T20:43:24.557984+0000 mon.a (mon.0) 10199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:25 smithi115 bash[45495]: cluster 2022-04-22T20:43:24.025287+0000 mgr.y (mgr.34107) 6750 : cluster [DBG] pgmap v7861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:25 smithi115 bash[45495]: audit 2022-04-22T20:43:24.557984+0000 mon.a (mon.0) 10199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:25 smithi115 bash[46934]: cluster 2022-04-22T20:43:24.025287+0000 mgr.y (mgr.34107) 6750 : cluster [DBG] pgmap v7861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:25 smithi115 bash[46934]: audit 2022-04-22T20:43:24.557984+0000 mon.a (mon.0) 10199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:27 smithi121 bash[36902]: cluster 2022-04-22T20:43:26.026363+0000 mgr.y (mgr.34107) 6751 : cluster [DBG] pgmap v7862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:27.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:27 smithi115 bash[45495]: cluster 2022-04-22T20:43:26.026363+0000 mgr.y (mgr.34107) 6751 : cluster [DBG] pgmap v7862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:27 smithi115 bash[46934]: cluster 2022-04-22T20:43:26.026363+0000 mgr.y (mgr.34107) 6751 : cluster [DBG] pgmap v7862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:28 smithi121 bash[36902]: cluster 2022-04-22T20:43:28.027377+0000 mgr.y (mgr.34107) 6752 : cluster [DBG] pgmap v7863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:28 smithi121 bash[36902]: audit 2022-04-22T20:43:28.104937+0000 mon.a (mon.0) 10200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:28 smithi121 bash[36902]: audit 2022-04-22T20:43:28.118018+0000 mon.a (mon.0) 10201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[45495]: cluster 2022-04-22T20:43:28.027377+0000 mgr.y (mgr.34107) 6752 : cluster [DBG] pgmap v7863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[45495]: audit 2022-04-22T20:43:28.104937+0000 mon.a (mon.0) 10200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[45495]: audit 2022-04-22T20:43:28.118018+0000 mon.a (mon.0) 10201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[46934]: cluster 2022-04-22T20:43:28.027377+0000 mgr.y (mgr.34107) 6752 : cluster [DBG] pgmap v7863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[46934]: audit 2022-04-22T20:43:28.104937+0000 mon.a (mon.0) 10200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:28.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:28 smithi115 bash[46934]: audit 2022-04-22T20:43:28.118018+0000 mon.a (mon.0) 10201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:30 smithi121 bash[36902]: audit 2022-04-22T20:43:29.561234+0000 mon.a (mon.0) 10202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:30 smithi115 bash[45495]: audit 2022-04-22T20:43:29.561234+0000 mon.a (mon.0) 10202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:30 smithi115 bash[46934]: audit 2022-04-22T20:43:29.561234+0000 mon.a (mon.0) 10202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:31 smithi121 bash[36902]: cluster 2022-04-22T20:43:30.028485+0000 mgr.y (mgr.34107) 6753 : cluster [DBG] pgmap v7864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:31 smithi115 bash[45495]: cluster 2022-04-22T20:43:30.028485+0000 mgr.y (mgr.34107) 6753 : cluster [DBG] pgmap v7864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:31 smithi115 bash[46934]: cluster 2022-04-22T20:43:30.028485+0000 mgr.y (mgr.34107) 6753 : cluster [DBG] pgmap v7864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:32 smithi121 bash[36902]: cluster 2022-04-22T20:43:32.029524+0000 mgr.y (mgr.34107) 6754 : cluster [DBG] pgmap v7865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:32 smithi115 bash[45495]: cluster 2022-04-22T20:43:32.029524+0000 mgr.y (mgr.34107) 6754 : cluster [DBG] pgmap v7865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:32.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:32.371Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:32.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:32 smithi115 bash[46934]: cluster 2022-04-22T20:43:32.029524+0000 mgr.y (mgr.34107) 6754 : cluster [DBG] pgmap v7865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:33 smithi121 bash[36902]: audit 2022-04-22T20:43:33.108145+0000 mon.a (mon.0) 10203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:33 smithi121 bash[36902]: audit 2022-04-22T20:43:33.121471+0000 mon.a (mon.0) 10204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:33 smithi115 bash[45495]: audit 2022-04-22T20:43:33.108145+0000 mon.a (mon.0) 10203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:33.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:33 smithi115 bash[45495]: audit 2022-04-22T20:43:33.121471+0000 mon.a (mon.0) 10204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:33 smithi115 bash[46934]: audit 2022-04-22T20:43:33.108145+0000 mon.a (mon.0) 10203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:33 smithi115 bash[46934]: audit 2022-04-22T20:43:33.121471+0000 mon.a (mon.0) 10204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:34.256Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:35 smithi121 bash[36902]: cluster 2022-04-22T20:43:34.030724+0000 mgr.y (mgr.34107) 6755 : cluster [DBG] pgmap v7866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:35 smithi121 bash[36902]: audit 2022-04-22T20:43:34.564302+0000 mon.a (mon.0) 10205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:35 smithi115 bash[45495]: cluster 2022-04-22T20:43:34.030724+0000 mgr.y (mgr.34107) 6755 : cluster [DBG] pgmap v7866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:35 smithi115 bash[45495]: audit 2022-04-22T20:43:34.564302+0000 mon.a (mon.0) 10205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:35 smithi115 bash[46934]: cluster 2022-04-22T20:43:34.030724+0000 mgr.y (mgr.34107) 6755 : cluster [DBG] pgmap v7866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:35 smithi115 bash[46934]: audit 2022-04-22T20:43:34.564302+0000 mon.a (mon.0) 10205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:36 smithi121 bash[36902]: cluster 2022-04-22T20:43:36.031936+0000 mgr.y (mgr.34107) 6756 : cluster [DBG] pgmap v7867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:36.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:36 smithi115 bash[45495]: cluster 2022-04-22T20:43:36.031936+0000 mgr.y (mgr.34107) 6756 : cluster [DBG] pgmap v7867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:36 smithi115 bash[46934]: cluster 2022-04-22T20:43:36.031936+0000 mgr.y (mgr.34107) 6756 : cluster [DBG] pgmap v7867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:39 smithi121 bash[36902]: cluster 2022-04-22T20:43:38.032943+0000 mgr.y (mgr.34107) 6757 : cluster [DBG] pgmap v7868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:39 smithi121 bash[36902]: audit 2022-04-22T20:43:38.111270+0000 mon.a (mon.0) 10206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:39 smithi121 bash[36902]: audit 2022-04-22T20:43:38.124992+0000 mon.a (mon.0) 10207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[45495]: cluster 2022-04-22T20:43:38.032943+0000 mgr.y (mgr.34107) 6757 : cluster [DBG] pgmap v7868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[45495]: audit 2022-04-22T20:43:38.111270+0000 mon.a (mon.0) 10206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[45495]: audit 2022-04-22T20:43:38.124992+0000 mon.a (mon.0) 10207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[46934]: cluster 2022-04-22T20:43:38.032943+0000 mgr.y (mgr.34107) 6757 : cluster [DBG] pgmap v7868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[46934]: audit 2022-04-22T20:43:38.111270+0000 mon.a (mon.0) 10206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:39 smithi115 bash[46934]: audit 2022-04-22T20:43:38.124992+0000 mon.a (mon.0) 10207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:40 smithi121 bash[36902]: audit 2022-04-22T20:43:39.567204+0000 mon.a (mon.0) 10208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:40 smithi115 bash[45495]: audit 2022-04-22T20:43:39.567204+0000 mon.a (mon.0) 10208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:40 smithi115 bash[46934]: audit 2022-04-22T20:43:39.567204+0000 mon.a (mon.0) 10208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:41 smithi121 bash[36902]: cluster 2022-04-22T20:43:40.034097+0000 mgr.y (mgr.34107) 6758 : cluster [DBG] pgmap v7869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:41.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:41 smithi115 bash[45495]: cluster 2022-04-22T20:43:40.034097+0000 mgr.y (mgr.34107) 6758 : cluster [DBG] pgmap v7869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:41.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:41 smithi115 bash[46934]: cluster 2022-04-22T20:43:40.034097+0000 mgr.y (mgr.34107) 6758 : cluster [DBG] pgmap v7869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:42 smithi121 bash[36902]: cluster 2022-04-22T20:43:42.035182+0000 mgr.y (mgr.34107) 6759 : cluster [DBG] pgmap v7870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:42.372Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:42.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:42 smithi115 bash[45495]: cluster 2022-04-22T20:43:42.035182+0000 mgr.y (mgr.34107) 6759 : cluster [DBG] pgmap v7870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:42.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:42 smithi115 bash[46934]: cluster 2022-04-22T20:43:42.035182+0000 mgr.y (mgr.34107) 6759 : cluster [DBG] pgmap v7870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:43 smithi121 bash[36902]: audit 2022-04-22T20:43:43.114024+0000 mon.a (mon.0) 10209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:43 smithi121 bash[36902]: audit 2022-04-22T20:43:43.128198+0000 mon.a (mon.0) 10210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:43 smithi115 bash[45495]: audit 2022-04-22T20:43:43.114024+0000 mon.a (mon.0) 10209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:43.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:43 smithi115 bash[45495]: audit 2022-04-22T20:43:43.128198+0000 mon.a (mon.0) 10210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:43 smithi115 bash[46934]: audit 2022-04-22T20:43:43.114024+0000 mon.a (mon.0) 10209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:43 smithi115 bash[46934]: audit 2022-04-22T20:43:43.128198+0000 mon.a (mon.0) 10210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:44.255Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:45 smithi121 bash[36902]: cluster 2022-04-22T20:43:44.036418+0000 mgr.y (mgr.34107) 6760 : cluster [DBG] pgmap v7871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:45 smithi121 bash[36902]: audit 2022-04-22T20:43:44.570359+0000 mon.a (mon.0) 10211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:45 smithi115 bash[45495]: cluster 2022-04-22T20:43:44.036418+0000 mgr.y (mgr.34107) 6760 : cluster [DBG] pgmap v7871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:45.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:45 smithi115 bash[45495]: audit 2022-04-22T20:43:44.570359+0000 mon.a (mon.0) 10211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:45 smithi115 bash[46934]: cluster 2022-04-22T20:43:44.036418+0000 mgr.y (mgr.34107) 6760 : cluster [DBG] pgmap v7871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:45.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:45 smithi115 bash[46934]: audit 2022-04-22T20:43:44.570359+0000 mon.a (mon.0) 10211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:46 smithi121 bash[36902]: cluster 2022-04-22T20:43:46.037533+0000 mgr.y (mgr.34107) 6761 : cluster [DBG] pgmap v7872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:46 smithi115 bash[45495]: cluster 2022-04-22T20:43:46.037533+0000 mgr.y (mgr.34107) 6761 : cluster [DBG] pgmap v7872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:46 smithi115 bash[46934]: cluster 2022-04-22T20:43:46.037533+0000 mgr.y (mgr.34107) 6761 : cluster [DBG] pgmap v7872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:46.846 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:43:47.190 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 39s ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 104s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 39s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 104s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 104s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 39s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 39s ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:43:47.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 104s ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 39s ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 39s ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 104s ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 39s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 39s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 39s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 39s ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 104s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 104s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:43:47.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 104s ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:43:47.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 104s ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:43:47.193 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 104s ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:47.589 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:43:47.590 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:43:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:47 smithi121 bash[36902]: audit 2022-04-22T20:43:46.824285+0000 mgr.y (mgr.34107) 6762 : audit [DBG] from='client.42408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:47 smithi121 bash[36902]: audit 2022-04-22T20:43:47.164901+0000 mgr.y (mgr.34107) 6763 : audit [DBG] from='client.52345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:47.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:47 smithi115 bash[45495]: audit 2022-04-22T20:43:46.824285+0000 mgr.y (mgr.34107) 6762 : audit [DBG] from='client.42408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:47.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:47 smithi115 bash[45495]: audit 2022-04-22T20:43:47.164901+0000 mgr.y (mgr.34107) 6763 : audit [DBG] from='client.52345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:47 smithi115 bash[46934]: audit 2022-04-22T20:43:46.824285+0000 mgr.y (mgr.34107) 6762 : audit [DBG] from='client.42408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:47.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:47 smithi115 bash[46934]: audit 2022-04-22T20:43:47.164901+0000 mgr.y (mgr.34107) 6763 : audit [DBG] from='client.52345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:43:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:48 smithi121 bash[36902]: audit 2022-04-22T20:43:47.582250+0000 mon.c (mon.1) 259 : audit [DBG] from='client.? 172.21.15.115:0/2647922050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:48 smithi121 bash[36902]: cluster 2022-04-22T20:43:48.038559+0000 mgr.y (mgr.34107) 6764 : cluster [DBG] pgmap v7873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:48 smithi121 bash[36902]: audit 2022-04-22T20:43:48.116800+0000 mon.a (mon.0) 10212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:48 smithi121 bash[36902]: audit 2022-04-22T20:43:48.131594+0000 mon.a (mon.0) 10213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[45495]: audit 2022-04-22T20:43:47.582250+0000 mon.c (mon.1) 259 : audit [DBG] from='client.? 172.21.15.115:0/2647922050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[45495]: cluster 2022-04-22T20:43:48.038559+0000 mgr.y (mgr.34107) 6764 : cluster [DBG] pgmap v7873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[45495]: audit 2022-04-22T20:43:48.116800+0000 mon.a (mon.0) 10212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[45495]: audit 2022-04-22T20:43:48.131594+0000 mon.a (mon.0) 10213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[46934]: audit 2022-04-22T20:43:47.582250+0000 mon.c (mon.1) 259 : audit [DBG] from='client.? 172.21.15.115:0/2647922050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:43:48.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[46934]: cluster 2022-04-22T20:43:48.038559+0000 mgr.y (mgr.34107) 6764 : cluster [DBG] pgmap v7873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[46934]: audit 2022-04-22T20:43:48.116800+0000 mon.a (mon.0) 10212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:48.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:48 smithi115 bash[46934]: audit 2022-04-22T20:43:48.131594+0000 mon.a (mon.0) 10213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:50 smithi121 bash[36902]: audit 2022-04-22T20:43:49.573484+0000 mon.a (mon.0) 10214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:50.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:50 smithi121 bash[36902]: cluster 2022-04-22T20:43:50.039680+0000 mgr.y (mgr.34107) 6765 : cluster [DBG] pgmap v7874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:50 smithi115 bash[45495]: audit 2022-04-22T20:43:49.573484+0000 mon.a (mon.0) 10214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:50 smithi115 bash[45495]: cluster 2022-04-22T20:43:50.039680+0000 mgr.y (mgr.34107) 6765 : cluster [DBG] pgmap v7874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:50 smithi115 bash[46934]: audit 2022-04-22T20:43:49.573484+0000 mon.a (mon.0) 10214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:50 smithi115 bash[46934]: cluster 2022-04-22T20:43:50.039680+0000 mgr.y (mgr.34107) 6765 : cluster [DBG] pgmap v7874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:52.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:53 smithi121 bash[36902]: cluster 2022-04-22T20:43:52.040688+0000 mgr.y (mgr.34107) 6766 : cluster [DBG] pgmap v7875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:53 smithi115 bash[45495]: cluster 2022-04-22T20:43:52.040688+0000 mgr.y (mgr.34107) 6766 : cluster [DBG] pgmap v7875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:53 smithi115 bash[46934]: cluster 2022-04-22T20:43:52.040688+0000 mgr.y (mgr.34107) 6766 : cluster [DBG] pgmap v7875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:54 smithi121 bash[36902]: audit 2022-04-22T20:43:53.120162+0000 mon.a (mon.0) 10215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:54 smithi121 bash[36902]: audit 2022-04-22T20:43:53.135062+0000 mon.a (mon.0) 10216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:54.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:43:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:43:54.255Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:43:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:54 smithi115 bash[45495]: audit 2022-04-22T20:43:53.120162+0000 mon.a (mon.0) 10215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:54 smithi115 bash[45495]: audit 2022-04-22T20:43:53.135062+0000 mon.a (mon.0) 10216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:54 smithi115 bash[46934]: audit 2022-04-22T20:43:53.120162+0000 mon.a (mon.0) 10215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:54 smithi115 bash[46934]: audit 2022-04-22T20:43:53.135062+0000 mon.a (mon.0) 10216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:55 smithi121 bash[36902]: cluster 2022-04-22T20:43:54.041795+0000 mgr.y (mgr.34107) 6767 : cluster [DBG] pgmap v7876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:55 smithi121 bash[36902]: audit 2022-04-22T20:43:54.576637+0000 mon.a (mon.0) 10217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:55 smithi115 bash[45495]: cluster 2022-04-22T20:43:54.041795+0000 mgr.y (mgr.34107) 6767 : cluster [DBG] pgmap v7876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:55 smithi115 bash[45495]: audit 2022-04-22T20:43:54.576637+0000 mon.a (mon.0) 10217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:55 smithi115 bash[46934]: cluster 2022-04-22T20:43:54.041795+0000 mgr.y (mgr.34107) 6767 : cluster [DBG] pgmap v7876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:55 smithi115 bash[46934]: audit 2022-04-22T20:43:54.576637+0000 mon.a (mon.0) 10217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:43:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:57 smithi121 bash[36902]: cluster 2022-04-22T20:43:56.043081+0000 mgr.y (mgr.34107) 6768 : cluster [DBG] pgmap v7877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:57.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:57 smithi115 bash[45495]: cluster 2022-04-22T20:43:56.043081+0000 mgr.y (mgr.34107) 6768 : cluster [DBG] pgmap v7877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:57.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:57 smithi115 bash[46934]: cluster 2022-04-22T20:43:56.043081+0000 mgr.y (mgr.34107) 6768 : cluster [DBG] pgmap v7877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:59 smithi121 bash[36902]: cluster 2022-04-22T20:43:58.044035+0000 mgr.y (mgr.34107) 6769 : cluster [DBG] pgmap v7878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:59 smithi121 bash[36902]: audit 2022-04-22T20:43:58.122831+0000 mon.a (mon.0) 10218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:43:59 smithi121 bash[36902]: audit 2022-04-22T20:43:58.138744+0000 mon.a (mon.0) 10219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[45495]: cluster 2022-04-22T20:43:58.044035+0000 mgr.y (mgr.34107) 6769 : cluster [DBG] pgmap v7878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[45495]: audit 2022-04-22T20:43:58.122831+0000 mon.a (mon.0) 10218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[45495]: audit 2022-04-22T20:43:58.138744+0000 mon.a (mon.0) 10219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:43:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[46934]: cluster 2022-04-22T20:43:58.044035+0000 mgr.y (mgr.34107) 6769 : cluster [DBG] pgmap v7878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:43:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[46934]: audit 2022-04-22T20:43:58.122831+0000 mon.a (mon.0) 10218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:43:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:43:59 smithi115 bash[46934]: audit 2022-04-22T20:43:58.138744+0000 mon.a (mon.0) 10219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:00 smithi121 bash[36902]: audit 2022-04-22T20:43:59.580260+0000 mon.a (mon.0) 10220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:00 smithi121 bash[36902]: cluster 2022-04-22T20:44:00.045129+0000 mgr.y (mgr.34107) 6770 : cluster [DBG] pgmap v7879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:00 smithi115 bash[45495]: audit 2022-04-22T20:43:59.580260+0000 mon.a (mon.0) 10220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:00 smithi115 bash[45495]: cluster 2022-04-22T20:44:00.045129+0000 mgr.y (mgr.34107) 6770 : cluster [DBG] pgmap v7879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:00 smithi115 bash[46934]: audit 2022-04-22T20:43:59.580260+0000 mon.a (mon.0) 10220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:00 smithi115 bash[46934]: cluster 2022-04-22T20:44:00.045129+0000 mgr.y (mgr.34107) 6770 : cluster [DBG] pgmap v7879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:02.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:03.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:03 smithi121 bash[36902]: cluster 2022-04-22T20:44:02.046216+0000 mgr.y (mgr.34107) 6771 : cluster [DBG] pgmap v7880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:03 smithi115 bash[45495]: cluster 2022-04-22T20:44:02.046216+0000 mgr.y (mgr.34107) 6771 : cluster [DBG] pgmap v7880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:03.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:03 smithi115 bash[46934]: cluster 2022-04-22T20:44:02.046216+0000 mgr.y (mgr.34107) 6771 : cluster [DBG] pgmap v7880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:04 smithi121 bash[36902]: audit 2022-04-22T20:44:03.125693+0000 mon.a (mon.0) 10221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:04 smithi121 bash[36902]: audit 2022-04-22T20:44:03.142198+0000 mon.a (mon.0) 10222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:04.255Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:04 smithi115 bash[45495]: audit 2022-04-22T20:44:03.125693+0000 mon.a (mon.0) 10221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:04 smithi115 bash[45495]: audit 2022-04-22T20:44:03.142198+0000 mon.a (mon.0) 10222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:04 smithi115 bash[46934]: audit 2022-04-22T20:44:03.125693+0000 mon.a (mon.0) 10221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:04.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:04 smithi115 bash[46934]: audit 2022-04-22T20:44:03.142198+0000 mon.a (mon.0) 10222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:05 smithi121 bash[36902]: cluster 2022-04-22T20:44:04.047392+0000 mgr.y (mgr.34107) 6772 : cluster [DBG] pgmap v7881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:05 smithi121 bash[36902]: audit 2022-04-22T20:44:04.584189+0000 mon.a (mon.0) 10223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:05 smithi115 bash[45495]: cluster 2022-04-22T20:44:04.047392+0000 mgr.y (mgr.34107) 6772 : cluster [DBG] pgmap v7881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:05.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:05 smithi115 bash[45495]: audit 2022-04-22T20:44:04.584189+0000 mon.a (mon.0) 10223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:05.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:05 smithi115 bash[46934]: cluster 2022-04-22T20:44:04.047392+0000 mgr.y (mgr.34107) 6772 : cluster [DBG] pgmap v7881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:05.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:05 smithi115 bash[46934]: audit 2022-04-22T20:44:04.584189+0000 mon.a (mon.0) 10223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:07 smithi121 bash[36902]: cluster 2022-04-22T20:44:06.048619+0000 mgr.y (mgr.34107) 6773 : cluster [DBG] pgmap v7882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:07.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:07 smithi115 bash[45495]: cluster 2022-04-22T20:44:06.048619+0000 mgr.y (mgr.34107) 6773 : cluster [DBG] pgmap v7882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:07.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:07 smithi115 bash[46934]: cluster 2022-04-22T20:44:06.048619+0000 mgr.y (mgr.34107) 6773 : cluster [DBG] pgmap v7882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:09.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:09 smithi121 bash[36902]: cluster 2022-04-22T20:44:08.049650+0000 mgr.y (mgr.34107) 6774 : cluster [DBG] pgmap v7883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:09.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:09 smithi121 bash[36902]: audit 2022-04-22T20:44:08.128460+0000 mon.a (mon.0) 10224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:09.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:09 smithi121 bash[36902]: audit 2022-04-22T20:44:08.145628+0000 mon.a (mon.0) 10225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[45495]: cluster 2022-04-22T20:44:08.049650+0000 mgr.y (mgr.34107) 6774 : cluster [DBG] pgmap v7883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[45495]: audit 2022-04-22T20:44:08.128460+0000 mon.a (mon.0) 10224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:09.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[45495]: audit 2022-04-22T20:44:08.145628+0000 mon.a (mon.0) 10225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:09.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[46934]: cluster 2022-04-22T20:44:08.049650+0000 mgr.y (mgr.34107) 6774 : cluster [DBG] pgmap v7883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[46934]: audit 2022-04-22T20:44:08.128460+0000 mon.a (mon.0) 10224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:09 smithi115 bash[46934]: audit 2022-04-22T20:44:08.145628+0000 mon.a (mon.0) 10225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:10 smithi121 bash[36902]: audit 2022-04-22T20:44:09.587338+0000 mon.a (mon.0) 10226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:10 smithi121 bash[36902]: cluster 2022-04-22T20:44:10.050737+0000 mgr.y (mgr.34107) 6775 : cluster [DBG] pgmap v7884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:10 smithi115 bash[45495]: audit 2022-04-22T20:44:09.587338+0000 mon.a (mon.0) 10226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:10 smithi115 bash[45495]: cluster 2022-04-22T20:44:10.050737+0000 mgr.y (mgr.34107) 6775 : cluster [DBG] pgmap v7884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:10 smithi115 bash[46934]: audit 2022-04-22T20:44:09.587338+0000 mon.a (mon.0) 10226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:10 smithi115 bash[46934]: cluster 2022-04-22T20:44:10.050737+0000 mgr.y (mgr.34107) 6775 : cluster [DBG] pgmap v7884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:12.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:13 smithi121 bash[36902]: cluster 2022-04-22T20:44:12.052020+0000 mgr.y (mgr.34107) 6776 : cluster [DBG] pgmap v7885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:13 smithi115 bash[45495]: cluster 2022-04-22T20:44:12.052020+0000 mgr.y (mgr.34107) 6776 : cluster [DBG] pgmap v7885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:13.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:13 smithi115 bash[46934]: cluster 2022-04-22T20:44:12.052020+0000 mgr.y (mgr.34107) 6776 : cluster [DBG] pgmap v7885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:14 smithi121 bash[36902]: audit 2022-04-22T20:44:13.131059+0000 mon.a (mon.0) 10227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:14 smithi121 bash[36902]: audit 2022-04-22T20:44:13.149083+0000 mon.a (mon.0) 10228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:14.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:14.258Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:14 smithi115 bash[45495]: audit 2022-04-22T20:44:13.131059+0000 mon.a (mon.0) 10227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:14 smithi115 bash[45495]: audit 2022-04-22T20:44:13.149083+0000 mon.a (mon.0) 10228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:14 smithi115 bash[46934]: audit 2022-04-22T20:44:13.131059+0000 mon.a (mon.0) 10227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:14 smithi115 bash[46934]: audit 2022-04-22T20:44:13.149083+0000 mon.a (mon.0) 10228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:15 smithi121 bash[36902]: cluster 2022-04-22T20:44:14.053058+0000 mgr.y (mgr.34107) 6777 : cluster [DBG] pgmap v7886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:15 smithi121 bash[36902]: audit 2022-04-22T20:44:14.590542+0000 mon.a (mon.0) 10229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:15 smithi115 bash[45495]: cluster 2022-04-22T20:44:14.053058+0000 mgr.y (mgr.34107) 6777 : cluster [DBG] pgmap v7886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:15 smithi115 bash[45495]: audit 2022-04-22T20:44:14.590542+0000 mon.a (mon.0) 10229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:15 smithi115 bash[46934]: cluster 2022-04-22T20:44:14.053058+0000 mgr.y (mgr.34107) 6777 : cluster [DBG] pgmap v7886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:15.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:15 smithi115 bash[46934]: audit 2022-04-22T20:44:14.590542+0000 mon.a (mon.0) 10229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:16 smithi121 bash[36902]: audit 2022-04-22T20:44:15.550989+0000 mon.a (mon.0) 10230 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:44:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:16 smithi121 bash[36902]: audit 2022-04-22T20:44:15.551455+0000 mon.a (mon.0) 10231 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:44:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:16 smithi115 bash[45495]: audit 2022-04-22T20:44:15.550989+0000 mon.a (mon.0) 10230 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:44:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:16 smithi115 bash[45495]: audit 2022-04-22T20:44:15.551455+0000 mon.a (mon.0) 10231 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:44:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:16 smithi115 bash[46934]: audit 2022-04-22T20:44:15.550989+0000 mon.a (mon.0) 10230 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:44:16.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:16 smithi115 bash[46934]: audit 2022-04-22T20:44:15.551455+0000 mon.a (mon.0) 10231 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:44:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:17 smithi121 bash[36902]: cluster 2022-04-22T20:44:16.054232+0000 mgr.y (mgr.34107) 6778 : cluster [DBG] pgmap v7887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:17 smithi121 bash[36902]: audit 2022-04-22T20:44:16.818830+0000 mon.a (mon.0) 10232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:44:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:17 smithi121 bash[36902]: audit 2022-04-22T20:44:16.822046+0000 mon.a (mon.0) 10233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:44:17.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:17 smithi121 bash[36902]: audit 2022-04-22T20:44:16.823855+0000 mon.a (mon.0) 10234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:44:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[45495]: cluster 2022-04-22T20:44:16.054232+0000 mgr.y (mgr.34107) 6778 : cluster [DBG] pgmap v7887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[45495]: audit 2022-04-22T20:44:16.818830+0000 mon.a (mon.0) 10232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:44:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[45495]: audit 2022-04-22T20:44:16.822046+0000 mon.a (mon.0) 10233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:44:17.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[45495]: audit 2022-04-22T20:44:16.823855+0000 mon.a (mon.0) 10234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:44:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[46934]: cluster 2022-04-22T20:44:16.054232+0000 mgr.y (mgr.34107) 6778 : cluster [DBG] pgmap v7887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[46934]: audit 2022-04-22T20:44:16.818830+0000 mon.a (mon.0) 10232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:44:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[46934]: audit 2022-04-22T20:44:16.822046+0000 mon.a (mon.0) 10233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:44:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:17 smithi115 bash[46934]: audit 2022-04-22T20:44:16.823855+0000 mon.a (mon.0) 10234 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:44:17.960 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 70s ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 70s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:44:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 70s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 70s ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 70s ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 70s ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 70s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 70s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 70s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 70s ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:44:18.317 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:44:18.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:44:18.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:44:18.318 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:44:18.318 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:44:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:18 smithi121 bash[36902]: audit 2022-04-22T20:44:17.188660+0000 mon.a (mon.0) 10235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:44:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:18 smithi121 bash[36902]: audit 2022-04-22T20:44:18.134262+0000 mon.a (mon.0) 10236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:18 smithi121 bash[36902]: audit 2022-04-22T20:44:18.152455+0000 mon.a (mon.0) 10237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[46934]: audit 2022-04-22T20:44:17.188660+0000 mon.a (mon.0) 10235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:44:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[46934]: audit 2022-04-22T20:44:18.134262+0000 mon.a (mon.0) 10236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[46934]: audit 2022-04-22T20:44:18.152455+0000 mon.a (mon.0) 10237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[45495]: audit 2022-04-22T20:44:17.188660+0000 mon.a (mon.0) 10235 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:44:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[45495]: audit 2022-04-22T20:44:18.134262+0000 mon.a (mon.0) 10236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:18 smithi115 bash[45495]: audit 2022-04-22T20:44:18.152455+0000 mon.a (mon.0) 10237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:18.717 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:44:18.717 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:44:18.717 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:44:18.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:18.717 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:44:18.718 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:44:18.719 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:44:18.719 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:44:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:19 smithi121 bash[36902]: audit 2022-04-22T20:44:17.938603+0000 mgr.y (mgr.34107) 6779 : audit [DBG] from='client.52357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:19 smithi121 bash[36902]: cluster 2022-04-22T20:44:18.054929+0000 mgr.y (mgr.34107) 6780 : cluster [DBG] pgmap v7888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:19 smithi121 bash[36902]: audit 2022-04-22T20:44:18.289522+0000 mgr.y (mgr.34107) 6781 : audit [DBG] from='client.42432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:19 smithi121 bash[36902]: audit 2022-04-22T20:44:18.710907+0000 mon.a (mon.0) 10238 : audit [DBG] from='client.? 172.21.15.115:0/3757017401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[45495]: audit 2022-04-22T20:44:17.938603+0000 mgr.y (mgr.34107) 6779 : audit [DBG] from='client.52357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[45495]: cluster 2022-04-22T20:44:18.054929+0000 mgr.y (mgr.34107) 6780 : cluster [DBG] pgmap v7888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[45495]: audit 2022-04-22T20:44:18.289522+0000 mgr.y (mgr.34107) 6781 : audit [DBG] from='client.42432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[45495]: audit 2022-04-22T20:44:18.710907+0000 mon.a (mon.0) 10238 : audit [DBG] from='client.? 172.21.15.115:0/3757017401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[46934]: audit 2022-04-22T20:44:17.938603+0000 mgr.y (mgr.34107) 6779 : audit [DBG] from='client.52357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[46934]: cluster 2022-04-22T20:44:18.054929+0000 mgr.y (mgr.34107) 6780 : cluster [DBG] pgmap v7888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[46934]: audit 2022-04-22T20:44:18.289522+0000 mgr.y (mgr.34107) 6781 : audit [DBG] from='client.42432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:19 smithi115 bash[46934]: audit 2022-04-22T20:44:18.710907+0000 mon.a (mon.0) 10238 : audit [DBG] from='client.? 172.21.15.115:0/3757017401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:20 smithi121 bash[36902]: audit 2022-04-22T20:44:19.593406+0000 mon.a (mon.0) 10239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:20 smithi121 bash[36902]: cluster 2022-04-22T20:44:20.055980+0000 mgr.y (mgr.34107) 6782 : cluster [DBG] pgmap v7889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:20 smithi115 bash[45495]: audit 2022-04-22T20:44:19.593406+0000 mon.a (mon.0) 10239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:20 smithi115 bash[45495]: cluster 2022-04-22T20:44:20.055980+0000 mgr.y (mgr.34107) 6782 : cluster [DBG] pgmap v7889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:20 smithi115 bash[46934]: audit 2022-04-22T20:44:19.593406+0000 mon.a (mon.0) 10239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:20 smithi115 bash[46934]: cluster 2022-04-22T20:44:20.055980+0000 mgr.y (mgr.34107) 6782 : cluster [DBG] pgmap v7889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:22.376Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:23 smithi121 bash[36902]: cluster 2022-04-22T20:44:22.057158+0000 mgr.y (mgr.34107) 6783 : cluster [DBG] pgmap v7890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:23.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:23 smithi115 bash[45495]: cluster 2022-04-22T20:44:22.057158+0000 mgr.y (mgr.34107) 6783 : cluster [DBG] pgmap v7890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:23 smithi115 bash[46934]: cluster 2022-04-22T20:44:22.057158+0000 mgr.y (mgr.34107) 6783 : cluster [DBG] pgmap v7890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:24 smithi121 bash[36902]: audit 2022-04-22T20:44:23.137117+0000 mon.a (mon.0) 10240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:24 smithi121 bash[36902]: audit 2022-04-22T20:44:23.155994+0000 mon.a (mon.0) 10241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:24.257Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:24 smithi115 bash[45495]: audit 2022-04-22T20:44:23.137117+0000 mon.a (mon.0) 10240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:24 smithi115 bash[45495]: audit 2022-04-22T20:44:23.155994+0000 mon.a (mon.0) 10241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:24 smithi115 bash[46934]: audit 2022-04-22T20:44:23.137117+0000 mon.a (mon.0) 10240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:24 smithi115 bash[46934]: audit 2022-04-22T20:44:23.155994+0000 mon.a (mon.0) 10241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:25 smithi121 bash[36902]: cluster 2022-04-22T20:44:24.058229+0000 mgr.y (mgr.34107) 6784 : cluster [DBG] pgmap v7891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:25 smithi121 bash[36902]: audit 2022-04-22T20:44:24.596164+0000 mon.a (mon.0) 10242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:25.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:25 smithi115 bash[45495]: cluster 2022-04-22T20:44:24.058229+0000 mgr.y (mgr.34107) 6784 : cluster [DBG] pgmap v7891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:25 smithi115 bash[45495]: audit 2022-04-22T20:44:24.596164+0000 mon.a (mon.0) 10242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:25 smithi115 bash[46934]: cluster 2022-04-22T20:44:24.058229+0000 mgr.y (mgr.34107) 6784 : cluster [DBG] pgmap v7891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:25 smithi115 bash[46934]: audit 2022-04-22T20:44:24.596164+0000 mon.a (mon.0) 10242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:27 smithi121 bash[36902]: cluster 2022-04-22T20:44:26.059333+0000 mgr.y (mgr.34107) 6785 : cluster [DBG] pgmap v7892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:27.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:27 smithi115 bash[46934]: cluster 2022-04-22T20:44:26.059333+0000 mgr.y (mgr.34107) 6785 : cluster [DBG] pgmap v7892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:27 smithi115 bash[45495]: cluster 2022-04-22T20:44:26.059333+0000 mgr.y (mgr.34107) 6785 : cluster [DBG] pgmap v7892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:29 smithi121 bash[36902]: cluster 2022-04-22T20:44:28.060714+0000 mgr.y (mgr.34107) 6786 : cluster [DBG] pgmap v7893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:29 smithi121 bash[36902]: audit 2022-04-22T20:44:28.139843+0000 mon.a (mon.0) 10243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:29 smithi121 bash[36902]: audit 2022-04-22T20:44:28.159714+0000 mon.a (mon.0) 10244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[45495]: cluster 2022-04-22T20:44:28.060714+0000 mgr.y (mgr.34107) 6786 : cluster [DBG] pgmap v7893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[45495]: audit 2022-04-22T20:44:28.139843+0000 mon.a (mon.0) 10243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:29.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[45495]: audit 2022-04-22T20:44:28.159714+0000 mon.a (mon.0) 10244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[46934]: cluster 2022-04-22T20:44:28.060714+0000 mgr.y (mgr.34107) 6786 : cluster [DBG] pgmap v7893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:29.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[46934]: audit 2022-04-22T20:44:28.139843+0000 mon.a (mon.0) 10243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:29.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:29 smithi115 bash[46934]: audit 2022-04-22T20:44:28.159714+0000 mon.a (mon.0) 10244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:30 smithi121 bash[36902]: audit 2022-04-22T20:44:29.598788+0000 mon.a (mon.0) 10245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:30 smithi121 bash[36902]: cluster 2022-04-22T20:44:30.061842+0000 mgr.y (mgr.34107) 6787 : cluster [DBG] pgmap v7894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:30 smithi115 bash[45495]: audit 2022-04-22T20:44:29.598788+0000 mon.a (mon.0) 10245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:30 smithi115 bash[45495]: cluster 2022-04-22T20:44:30.061842+0000 mgr.y (mgr.34107) 6787 : cluster [DBG] pgmap v7894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:30 smithi115 bash[46934]: audit 2022-04-22T20:44:29.598788+0000 mon.a (mon.0) 10245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:30 smithi115 bash[46934]: cluster 2022-04-22T20:44:30.061842+0000 mgr.y (mgr.34107) 6787 : cluster [DBG] pgmap v7894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:32.377Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:33 smithi121 bash[36902]: cluster 2022-04-22T20:44:32.063031+0000 mgr.y (mgr.34107) 6788 : cluster [DBG] pgmap v7895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:33 smithi115 bash[45495]: cluster 2022-04-22T20:44:32.063031+0000 mgr.y (mgr.34107) 6788 : cluster [DBG] pgmap v7895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:33.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:33 smithi115 bash[46934]: cluster 2022-04-22T20:44:32.063031+0000 mgr.y (mgr.34107) 6788 : cluster [DBG] pgmap v7895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:34 smithi121 bash[36902]: audit 2022-04-22T20:44:33.142638+0000 mon.a (mon.0) 10246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:34 smithi121 bash[36902]: audit 2022-04-22T20:44:33.163375+0000 mon.a (mon.0) 10247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:34.259Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:34 smithi115 bash[45495]: audit 2022-04-22T20:44:33.142638+0000 mon.a (mon.0) 10246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:34 smithi115 bash[45495]: audit 2022-04-22T20:44:33.163375+0000 mon.a (mon.0) 10247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:34 smithi115 bash[46934]: audit 2022-04-22T20:44:33.142638+0000 mon.a (mon.0) 10246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:34 smithi115 bash[46934]: audit 2022-04-22T20:44:33.163375+0000 mon.a (mon.0) 10247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:35 smithi121 bash[36902]: cluster 2022-04-22T20:44:34.064185+0000 mgr.y (mgr.34107) 6789 : cluster [DBG] pgmap v7896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:35 smithi121 bash[36902]: audit 2022-04-22T20:44:34.601959+0000 mon.a (mon.0) 10248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:35 smithi115 bash[45495]: cluster 2022-04-22T20:44:34.064185+0000 mgr.y (mgr.34107) 6789 : cluster [DBG] pgmap v7896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:35.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:35 smithi115 bash[45495]: audit 2022-04-22T20:44:34.601959+0000 mon.a (mon.0) 10248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:35 smithi115 bash[46934]: cluster 2022-04-22T20:44:34.064185+0000 mgr.y (mgr.34107) 6789 : cluster [DBG] pgmap v7896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:35 smithi115 bash[46934]: audit 2022-04-22T20:44:34.601959+0000 mon.a (mon.0) 10248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:37 smithi121 bash[36902]: cluster 2022-04-22T20:44:36.065254+0000 mgr.y (mgr.34107) 6790 : cluster [DBG] pgmap v7897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:37.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:37 smithi115 bash[46934]: cluster 2022-04-22T20:44:36.065254+0000 mgr.y (mgr.34107) 6790 : cluster [DBG] pgmap v7897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:37.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:37 smithi115 bash[45495]: cluster 2022-04-22T20:44:36.065254+0000 mgr.y (mgr.34107) 6790 : cluster [DBG] pgmap v7897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:39 smithi121 bash[36902]: cluster 2022-04-22T20:44:38.066229+0000 mgr.y (mgr.34107) 6791 : cluster [DBG] pgmap v7898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:39 smithi121 bash[36902]: audit 2022-04-22T20:44:38.145100+0000 mon.a (mon.0) 10249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:39 smithi121 bash[36902]: audit 2022-04-22T20:44:38.166461+0000 mon.a (mon.0) 10250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:39.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[46934]: cluster 2022-04-22T20:44:38.066229+0000 mgr.y (mgr.34107) 6791 : cluster [DBG] pgmap v7898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[46934]: audit 2022-04-22T20:44:38.145100+0000 mon.a (mon.0) 10249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[46934]: audit 2022-04-22T20:44:38.166461+0000 mon.a (mon.0) 10250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[45495]: cluster 2022-04-22T20:44:38.066229+0000 mgr.y (mgr.34107) 6791 : cluster [DBG] pgmap v7898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[45495]: audit 2022-04-22T20:44:38.145100+0000 mon.a (mon.0) 10249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:39 smithi115 bash[45495]: audit 2022-04-22T20:44:38.166461+0000 mon.a (mon.0) 10250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:40 smithi121 bash[36902]: audit 2022-04-22T20:44:39.605154+0000 mon.a (mon.0) 10251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:40 smithi121 bash[36902]: cluster 2022-04-22T20:44:40.067309+0000 mgr.y (mgr.34107) 6792 : cluster [DBG] pgmap v7899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:40 smithi115 bash[45495]: audit 2022-04-22T20:44:39.605154+0000 mon.a (mon.0) 10251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:40.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:40 smithi115 bash[45495]: cluster 2022-04-22T20:44:40.067309+0000 mgr.y (mgr.34107) 6792 : cluster [DBG] pgmap v7899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:40 smithi115 bash[46934]: audit 2022-04-22T20:44:39.605154+0000 mon.a (mon.0) 10251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:40 smithi115 bash[46934]: cluster 2022-04-22T20:44:40.067309+0000 mgr.y (mgr.34107) 6792 : cluster [DBG] pgmap v7899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:42.378Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:43 smithi121 bash[36902]: cluster 2022-04-22T20:44:42.068412+0000 mgr.y (mgr.34107) 6793 : cluster [DBG] pgmap v7900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:43 smithi115 bash[45495]: cluster 2022-04-22T20:44:42.068412+0000 mgr.y (mgr.34107) 6793 : cluster [DBG] pgmap v7900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:43 smithi115 bash[46934]: cluster 2022-04-22T20:44:42.068412+0000 mgr.y (mgr.34107) 6793 : cluster [DBG] pgmap v7900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:44 smithi121 bash[36902]: audit 2022-04-22T20:44:43.147727+0000 mon.a (mon.0) 10252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:44 smithi121 bash[36902]: audit 2022-04-22T20:44:43.169817+0000 mon.a (mon.0) 10253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:44.259Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:44 smithi115 bash[45495]: audit 2022-04-22T20:44:43.147727+0000 mon.a (mon.0) 10252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:44 smithi115 bash[45495]: audit 2022-04-22T20:44:43.169817+0000 mon.a (mon.0) 10253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:44 smithi115 bash[46934]: audit 2022-04-22T20:44:43.147727+0000 mon.a (mon.0) 10252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:44 smithi115 bash[46934]: audit 2022-04-22T20:44:43.169817+0000 mon.a (mon.0) 10253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:45 smithi121 bash[36902]: cluster 2022-04-22T20:44:44.069632+0000 mgr.y (mgr.34107) 6794 : cluster [DBG] pgmap v7901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:45 smithi121 bash[36902]: audit 2022-04-22T20:44:44.607529+0000 mon.a (mon.0) 10254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:45 smithi115 bash[45495]: cluster 2022-04-22T20:44:44.069632+0000 mgr.y (mgr.34107) 6794 : cluster [DBG] pgmap v7901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:45.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:45 smithi115 bash[45495]: audit 2022-04-22T20:44:44.607529+0000 mon.a (mon.0) 10254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:45 smithi115 bash[46934]: cluster 2022-04-22T20:44:44.069632+0000 mgr.y (mgr.34107) 6794 : cluster [DBG] pgmap v7901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:45.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:45 smithi115 bash[46934]: audit 2022-04-22T20:44:44.607529+0000 mon.a (mon.0) 10254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:47 smithi121 bash[36902]: cluster 2022-04-22T20:44:46.070745+0000 mgr.y (mgr.34107) 6795 : cluster [DBG] pgmap v7902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:47 smithi115 bash[45495]: cluster 2022-04-22T20:44:46.070745+0000 mgr.y (mgr.34107) 6795 : cluster [DBG] pgmap v7902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:47 smithi115 bash[46934]: cluster 2022-04-22T20:44:46.070745+0000 mgr.y (mgr.34107) 6795 : cluster [DBG] pgmap v7902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:49.101 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 101s ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 101s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:44:49.454 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 101s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 101s ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 101s ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 101s ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 101s ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 101s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 101s ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:44:49.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 101s ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:44:49.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:44:49.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:44:49.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:44:49.456 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:44:49.456 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:49 smithi121 bash[36902]: cluster 2022-04-22T20:44:48.071797+0000 mgr.y (mgr.34107) 6796 : cluster [DBG] pgmap v7903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:49 smithi121 bash[36902]: audit 2022-04-22T20:44:48.151089+0000 mon.a (mon.0) 10255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:49 smithi121 bash[36902]: audit 2022-04-22T20:44:48.173208+0000 mon.a (mon.0) 10256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[45495]: cluster 2022-04-22T20:44:48.071797+0000 mgr.y (mgr.34107) 6796 : cluster [DBG] pgmap v7903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[45495]: audit 2022-04-22T20:44:48.151089+0000 mon.a (mon.0) 10255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[45495]: audit 2022-04-22T20:44:48.173208+0000 mon.a (mon.0) 10256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[46934]: cluster 2022-04-22T20:44:48.071797+0000 mgr.y (mgr.34107) 6796 : cluster [DBG] pgmap v7903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[46934]: audit 2022-04-22T20:44:48.151089+0000 mon.a (mon.0) 10255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:49 smithi115 bash[46934]: audit 2022-04-22T20:44:48.173208+0000 mon.a (mon.0) 10256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:49.852 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:44:49.852 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:44:49.852 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:44:49.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:49.853 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:44:49.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:44:49.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:49.854 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:44:49.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:44:49.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:49.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:44:49.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:44:49.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:44:49.855 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:44:49.855 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:44:49.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:44:49.855 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:44:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:50 smithi121 bash[36902]: audit 2022-04-22T20:44:49.080143+0000 mgr.y (mgr.34107) 6797 : audit [DBG] from='client.42444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:50 smithi121 bash[36902]: audit 2022-04-22T20:44:49.610473+0000 mon.a (mon.0) 10257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:50 smithi121 bash[36902]: audit 2022-04-22T20:44:49.845439+0000 mon.a (mon.0) 10258 : audit [DBG] from='client.? 172.21.15.115:0/1867494246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:50.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[45495]: audit 2022-04-22T20:44:49.080143+0000 mgr.y (mgr.34107) 6797 : audit [DBG] from='client.42444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[45495]: audit 2022-04-22T20:44:49.610473+0000 mon.a (mon.0) 10257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:50.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[45495]: audit 2022-04-22T20:44:49.845439+0000 mon.a (mon.0) 10258 : audit [DBG] from='client.? 172.21.15.115:0/1867494246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[46934]: audit 2022-04-22T20:44:49.080143+0000 mgr.y (mgr.34107) 6797 : audit [DBG] from='client.42444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[46934]: audit 2022-04-22T20:44:49.610473+0000 mon.a (mon.0) 10257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:50.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:50 smithi115 bash[46934]: audit 2022-04-22T20:44:49.845439+0000 mon.a (mon.0) 10258 : audit [DBG] from='client.? 172.21.15.115:0/1867494246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:44:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:51 smithi121 bash[36902]: audit 2022-04-22T20:44:49.421803+0000 mgr.y (mgr.34107) 6798 : audit [DBG] from='client.52381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:51 smithi121 bash[36902]: cluster 2022-04-22T20:44:50.072871+0000 mgr.y (mgr.34107) 6799 : cluster [DBG] pgmap v7904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:51 smithi115 bash[45495]: audit 2022-04-22T20:44:49.421803+0000 mgr.y (mgr.34107) 6798 : audit [DBG] from='client.52381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:51.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:51 smithi115 bash[45495]: cluster 2022-04-22T20:44:50.072871+0000 mgr.y (mgr.34107) 6799 : cluster [DBG] pgmap v7904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:51 smithi115 bash[46934]: audit 2022-04-22T20:44:49.421803+0000 mgr.y (mgr.34107) 6798 : audit [DBG] from='client.52381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:44:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:51 smithi115 bash[46934]: cluster 2022-04-22T20:44:50.072871+0000 mgr.y (mgr.34107) 6799 : cluster [DBG] pgmap v7904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:52.379Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:53 smithi121 bash[36902]: cluster 2022-04-22T20:44:52.074098+0000 mgr.y (mgr.34107) 6800 : cluster [DBG] pgmap v7905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:53 smithi121 bash[36902]: audit 2022-04-22T20:44:53.153735+0000 mon.a (mon.0) 10259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:53 smithi115 bash[45495]: cluster 2022-04-22T20:44:52.074098+0000 mgr.y (mgr.34107) 6800 : cluster [DBG] pgmap v7905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:53 smithi115 bash[45495]: audit 2022-04-22T20:44:53.153735+0000 mon.a (mon.0) 10259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:53 smithi115 bash[46934]: cluster 2022-04-22T20:44:52.074098+0000 mgr.y (mgr.34107) 6800 : cluster [DBG] pgmap v7905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:53.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:53 smithi115 bash[46934]: audit 2022-04-22T20:44:53.153735+0000 mon.a (mon.0) 10259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:54 smithi121 bash[36902]: audit 2022-04-22T20:44:53.176622+0000 mon.a (mon.0) 10260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:54 smithi115 bash[45495]: audit 2022-04-22T20:44:53.176622+0000 mon.a (mon.0) 10260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:44:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:44:54.261Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:44:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:54 smithi115 bash[46934]: audit 2022-04-22T20:44:53.176622+0000 mon.a (mon.0) 10260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:55 smithi121 bash[36902]: cluster 2022-04-22T20:44:54.075315+0000 mgr.y (mgr.34107) 6801 : cluster [DBG] pgmap v7906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:55 smithi121 bash[36902]: audit 2022-04-22T20:44:54.613458+0000 mon.a (mon.0) 10261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:55 smithi115 bash[45495]: cluster 2022-04-22T20:44:54.075315+0000 mgr.y (mgr.34107) 6801 : cluster [DBG] pgmap v7906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:55 smithi115 bash[45495]: audit 2022-04-22T20:44:54.613458+0000 mon.a (mon.0) 10261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:55 smithi115 bash[46934]: cluster 2022-04-22T20:44:54.075315+0000 mgr.y (mgr.34107) 6801 : cluster [DBG] pgmap v7906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:55 smithi115 bash[46934]: audit 2022-04-22T20:44:54.613458+0000 mon.a (mon.0) 10261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:44:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:57 smithi121 bash[36902]: cluster 2022-04-22T20:44:56.076394+0000 mgr.y (mgr.34107) 6802 : cluster [DBG] pgmap v7907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:57 smithi115 bash[45495]: cluster 2022-04-22T20:44:56.076394+0000 mgr.y (mgr.34107) 6802 : cluster [DBG] pgmap v7907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:57 smithi115 bash[46934]: cluster 2022-04-22T20:44:56.076394+0000 mgr.y (mgr.34107) 6802 : cluster [DBG] pgmap v7907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:58 smithi121 bash[36902]: audit 2022-04-22T20:44:58.156955+0000 mon.a (mon.0) 10262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:58 smithi115 bash[45495]: audit 2022-04-22T20:44:58.156955+0000 mon.a (mon.0) 10262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:58.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:58 smithi115 bash[46934]: audit 2022-04-22T20:44:58.156955+0000 mon.a (mon.0) 10262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:44:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:59 smithi121 bash[36902]: cluster 2022-04-22T20:44:58.077420+0000 mgr.y (mgr.34107) 6803 : cluster [DBG] pgmap v7908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:44:59 smithi121 bash[36902]: audit 2022-04-22T20:44:58.180110+0000 mon.a (mon.0) 10263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:59 smithi115 bash[45495]: cluster 2022-04-22T20:44:58.077420+0000 mgr.y (mgr.34107) 6803 : cluster [DBG] pgmap v7908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:44:59 smithi115 bash[45495]: audit 2022-04-22T20:44:58.180110+0000 mon.a (mon.0) 10263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:44:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:59 smithi115 bash[46934]: cluster 2022-04-22T20:44:58.077420+0000 mgr.y (mgr.34107) 6803 : cluster [DBG] pgmap v7908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:44:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:44:59 smithi115 bash[46934]: audit 2022-04-22T20:44:58.180110+0000 mon.a (mon.0) 10263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:00 smithi121 bash[36902]: audit 2022-04-22T20:44:59.616690+0000 mon.a (mon.0) 10264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:00 smithi121 bash[36902]: cluster 2022-04-22T20:45:00.078464+0000 mgr.y (mgr.34107) 6804 : cluster [DBG] pgmap v7909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:00 smithi115 bash[45495]: audit 2022-04-22T20:44:59.616690+0000 mon.a (mon.0) 10264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:00 smithi115 bash[45495]: cluster 2022-04-22T20:45:00.078464+0000 mgr.y (mgr.34107) 6804 : cluster [DBG] pgmap v7909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:00 smithi115 bash[46934]: audit 2022-04-22T20:44:59.616690+0000 mon.a (mon.0) 10264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:00 smithi115 bash[46934]: cluster 2022-04-22T20:45:00.078464+0000 mgr.y (mgr.34107) 6804 : cluster [DBG] pgmap v7909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:02.381Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:03 smithi121 bash[36902]: cluster 2022-04-22T20:45:02.079801+0000 mgr.y (mgr.34107) 6805 : cluster [DBG] pgmap v7910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:03.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:03 smithi115 bash[45495]: cluster 2022-04-22T20:45:02.079801+0000 mgr.y (mgr.34107) 6805 : cluster [DBG] pgmap v7910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:03 smithi115 bash[46934]: cluster 2022-04-22T20:45:02.079801+0000 mgr.y (mgr.34107) 6805 : cluster [DBG] pgmap v7910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:04 smithi121 bash[36902]: audit 2022-04-22T20:45:03.160003+0000 mon.a (mon.0) 10265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:04 smithi121 bash[36902]: audit 2022-04-22T20:45:03.183377+0000 mon.a (mon.0) 10266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:04 smithi115 bash[45495]: audit 2022-04-22T20:45:03.160003+0000 mon.a (mon.0) 10265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:04 smithi115 bash[45495]: audit 2022-04-22T20:45:03.183377+0000 mon.a (mon.0) 10266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:04.261Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:04 smithi115 bash[46934]: audit 2022-04-22T20:45:03.160003+0000 mon.a (mon.0) 10265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:04 smithi115 bash[46934]: audit 2022-04-22T20:45:03.183377+0000 mon.a (mon.0) 10266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:05 smithi121 bash[36902]: cluster 2022-04-22T20:45:04.080913+0000 mgr.y (mgr.34107) 6806 : cluster [DBG] pgmap v7911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:05 smithi121 bash[36902]: audit 2022-04-22T20:45:04.619618+0000 mon.a (mon.0) 10267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:05 smithi115 bash[45495]: cluster 2022-04-22T20:45:04.080913+0000 mgr.y (mgr.34107) 6806 : cluster [DBG] pgmap v7911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:05 smithi115 bash[45495]: audit 2022-04-22T20:45:04.619618+0000 mon.a (mon.0) 10267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:05 smithi115 bash[46934]: cluster 2022-04-22T20:45:04.080913+0000 mgr.y (mgr.34107) 6806 : cluster [DBG] pgmap v7911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:05 smithi115 bash[46934]: audit 2022-04-22T20:45:04.619618+0000 mon.a (mon.0) 10267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:07 smithi121 bash[36902]: cluster 2022-04-22T20:45:06.082034+0000 mgr.y (mgr.34107) 6807 : cluster [DBG] pgmap v7912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:07 smithi115 bash[45495]: cluster 2022-04-22T20:45:06.082034+0000 mgr.y (mgr.34107) 6807 : cluster [DBG] pgmap v7912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:07.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:07 smithi115 bash[46934]: cluster 2022-04-22T20:45:06.082034+0000 mgr.y (mgr.34107) 6807 : cluster [DBG] pgmap v7912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:09 smithi121 bash[36902]: cluster 2022-04-22T20:45:08.083318+0000 mgr.y (mgr.34107) 6808 : cluster [DBG] pgmap v7913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:09 smithi121 bash[36902]: audit 2022-04-22T20:45:08.162730+0000 mon.a (mon.0) 10268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:09 smithi121 bash[36902]: audit 2022-04-22T20:45:08.186856+0000 mon.a (mon.0) 10269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[45495]: cluster 2022-04-22T20:45:08.083318+0000 mgr.y (mgr.34107) 6808 : cluster [DBG] pgmap v7913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[45495]: audit 2022-04-22T20:45:08.162730+0000 mon.a (mon.0) 10268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[45495]: audit 2022-04-22T20:45:08.186856+0000 mon.a (mon.0) 10269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[46934]: cluster 2022-04-22T20:45:08.083318+0000 mgr.y (mgr.34107) 6808 : cluster [DBG] pgmap v7913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[46934]: audit 2022-04-22T20:45:08.162730+0000 mon.a (mon.0) 10268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:09 smithi115 bash[46934]: audit 2022-04-22T20:45:08.186856+0000 mon.a (mon.0) 10269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:10 smithi121 bash[36902]: audit 2022-04-22T20:45:09.622530+0000 mon.a (mon.0) 10270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:10 smithi121 bash[36902]: cluster 2022-04-22T20:45:10.084380+0000 mgr.y (mgr.34107) 6809 : cluster [DBG] pgmap v7914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:10 smithi115 bash[45495]: audit 2022-04-22T20:45:09.622530+0000 mon.a (mon.0) 10270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:10 smithi115 bash[45495]: cluster 2022-04-22T20:45:10.084380+0000 mgr.y (mgr.34107) 6809 : cluster [DBG] pgmap v7914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:10 smithi115 bash[46934]: audit 2022-04-22T20:45:09.622530+0000 mon.a (mon.0) 10270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:10 smithi115 bash[46934]: cluster 2022-04-22T20:45:10.084380+0000 mgr.y (mgr.34107) 6809 : cluster [DBG] pgmap v7914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:12.380Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:13 smithi121 bash[36902]: cluster 2022-04-22T20:45:12.085593+0000 mgr.y (mgr.34107) 6810 : cluster [DBG] pgmap v7915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:13 smithi115 bash[45495]: cluster 2022-04-22T20:45:12.085593+0000 mgr.y (mgr.34107) 6810 : cluster [DBG] pgmap v7915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:13.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:13 smithi115 bash[46934]: cluster 2022-04-22T20:45:12.085593+0000 mgr.y (mgr.34107) 6810 : cluster [DBG] pgmap v7915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:14 smithi121 bash[36902]: audit 2022-04-22T20:45:13.165272+0000 mon.a (mon.0) 10271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:14 smithi121 bash[36902]: audit 2022-04-22T20:45:13.190433+0000 mon.a (mon.0) 10272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:14.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:14.262Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:14 smithi115 bash[45495]: audit 2022-04-22T20:45:13.165272+0000 mon.a (mon.0) 10271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:14 smithi115 bash[45495]: audit 2022-04-22T20:45:13.190433+0000 mon.a (mon.0) 10272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:14 smithi115 bash[46934]: audit 2022-04-22T20:45:13.165272+0000 mon.a (mon.0) 10271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:14 smithi115 bash[46934]: audit 2022-04-22T20:45:13.190433+0000 mon.a (mon.0) 10272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:15 smithi121 bash[36902]: cluster 2022-04-22T20:45:14.086564+0000 mgr.y (mgr.34107) 6811 : cluster [DBG] pgmap v7916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:15.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:15 smithi121 bash[36902]: audit 2022-04-22T20:45:14.625684+0000 mon.a (mon.0) 10273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:15 smithi115 bash[46934]: cluster 2022-04-22T20:45:14.086564+0000 mgr.y (mgr.34107) 6811 : cluster [DBG] pgmap v7916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:15 smithi115 bash[46934]: audit 2022-04-22T20:45:14.625684+0000 mon.a (mon.0) 10273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:15 smithi115 bash[45495]: cluster 2022-04-22T20:45:14.086564+0000 mgr.y (mgr.34107) 6811 : cluster [DBG] pgmap v7916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:15 smithi115 bash[45495]: audit 2022-04-22T20:45:14.625684+0000 mon.a (mon.0) 10273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:16 smithi121 bash[36902]: audit 2022-04-22T20:45:15.560983+0000 mon.a (mon.0) 10274 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:45:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:16 smithi121 bash[36902]: audit 2022-04-22T20:45:15.561398+0000 mon.a (mon.0) 10275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:45:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:16 smithi115 bash[45495]: audit 2022-04-22T20:45:15.560983+0000 mon.a (mon.0) 10274 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:45:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:16 smithi115 bash[45495]: audit 2022-04-22T20:45:15.561398+0000 mon.a (mon.0) 10275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:45:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:16 smithi115 bash[46934]: audit 2022-04-22T20:45:15.560983+0000 mon.a (mon.0) 10274 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:45:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:16 smithi115 bash[46934]: audit 2022-04-22T20:45:15.561398+0000 mon.a (mon.0) 10275 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:45:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:17 smithi121 bash[36902]: cluster 2022-04-22T20:45:16.087821+0000 mgr.y (mgr.34107) 6812 : cluster [DBG] pgmap v7917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:17 smithi115 bash[45495]: cluster 2022-04-22T20:45:16.087821+0000 mgr.y (mgr.34107) 6812 : cluster [DBG] pgmap v7917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:17 smithi115 bash[46934]: cluster 2022-04-22T20:45:16.087821+0000 mgr.y (mgr.34107) 6812 : cluster [DBG] pgmap v7917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:18 smithi121 bash[36902]: audit 2022-04-22T20:45:17.198488+0000 mon.a (mon.0) 10276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:45:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:18 smithi121 bash[36902]: audit 2022-04-22T20:45:17.201728+0000 mon.a (mon.0) 10277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:45:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:18 smithi121 bash[36902]: audit 2022-04-22T20:45:17.203531+0000 mon.a (mon.0) 10278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:45:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:18 smithi121 bash[36902]: audit 2022-04-22T20:45:17.576199+0000 mon.a (mon.0) 10279 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:45:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[45495]: audit 2022-04-22T20:45:17.198488+0000 mon.a (mon.0) 10276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:45:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[45495]: audit 2022-04-22T20:45:17.201728+0000 mon.a (mon.0) 10277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:45:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[45495]: audit 2022-04-22T20:45:17.203531+0000 mon.a (mon.0) 10278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:45:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[45495]: audit 2022-04-22T20:45:17.576199+0000 mon.a (mon.0) 10279 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:45:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[46934]: audit 2022-04-22T20:45:17.198488+0000 mon.a (mon.0) 10276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:45:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[46934]: audit 2022-04-22T20:45:17.201728+0000 mon.a (mon.0) 10277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:45:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[46934]: audit 2022-04-22T20:45:17.203531+0000 mon.a (mon.0) 10278 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:45:18.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:18 smithi115 bash[46934]: audit 2022-04-22T20:45:17.576199+0000 mon.a (mon.0) 10279 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:45:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:19 smithi121 bash[36902]: cluster 2022-04-22T20:45:18.088900+0000 mgr.y (mgr.34107) 6813 : cluster [DBG] pgmap v7918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:19 smithi121 bash[36902]: audit 2022-04-22T20:45:18.175123+0000 mon.a (mon.0) 10280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:19 smithi121 bash[36902]: audit 2022-04-22T20:45:18.193830+0000 mon.a (mon.0) 10281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[45495]: cluster 2022-04-22T20:45:18.088900+0000 mgr.y (mgr.34107) 6813 : cluster [DBG] pgmap v7918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[45495]: audit 2022-04-22T20:45:18.175123+0000 mon.a (mon.0) 10280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[45495]: audit 2022-04-22T20:45:18.193830+0000 mon.a (mon.0) 10281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[46934]: cluster 2022-04-22T20:45:18.088900+0000 mgr.y (mgr.34107) 6813 : cluster [DBG] pgmap v7918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[46934]: audit 2022-04-22T20:45:18.175123+0000 mon.a (mon.0) 10280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:19 smithi115 bash[46934]: audit 2022-04-22T20:45:18.193830+0000 mon.a (mon.0) 10281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:20.227 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:45:20.574 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:45:20.574 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:45:20.574 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:45:20.574 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:45:20.574 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:45:20.575 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:45:20.575 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:45:20.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:45:20.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:45:20.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:45:20.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:45:20.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:45:20.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:45:20.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:45:20.577 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:45:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:20 smithi121 bash[36902]: audit 2022-04-22T20:45:19.628640+0000 mon.a (mon.0) 10282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:20 smithi121 bash[36902]: cluster 2022-04-22T20:45:20.089548+0000 mgr.y (mgr.34107) 6814 : cluster [DBG] pgmap v7919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:20 smithi121 bash[36902]: audit 2022-04-22T20:45:20.206206+0000 mgr.y (mgr.34107) 6815 : audit [DBG] from='client.52393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[45495]: audit 2022-04-22T20:45:19.628640+0000 mon.a (mon.0) 10282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[45495]: cluster 2022-04-22T20:45:20.089548+0000 mgr.y (mgr.34107) 6814 : cluster [DBG] pgmap v7919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[45495]: audit 2022-04-22T20:45:20.206206+0000 mgr.y (mgr.34107) 6815 : audit [DBG] from='client.52393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[46934]: audit 2022-04-22T20:45:19.628640+0000 mon.a (mon.0) 10282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[46934]: cluster 2022-04-22T20:45:20.089548+0000 mgr.y (mgr.34107) 6814 : cluster [DBG] pgmap v7919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:20 smithi115 bash[46934]: audit 2022-04-22T20:45:20.206206+0000 mgr.y (mgr.34107) 6815 : audit [DBG] from='client.52393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:20.982 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:45:20.983 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:45:20.984 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:45:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:21 smithi121 bash[36902]: audit 2022-04-22T20:45:20.547575+0000 mgr.y (mgr.34107) 6816 : audit [DBG] from='client.52399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:21 smithi121 bash[36902]: audit 2022-04-22T20:45:20.976007+0000 mon.a (mon.0) 10283 : audit [DBG] from='client.? 172.21.15.115:0/4023694445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:21 smithi115 bash[45495]: audit 2022-04-22T20:45:20.547575+0000 mgr.y (mgr.34107) 6816 : audit [DBG] from='client.52399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:21.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:21 smithi115 bash[45495]: audit 2022-04-22T20:45:20.976007+0000 mon.a (mon.0) 10283 : audit [DBG] from='client.? 172.21.15.115:0/4023694445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:21 smithi115 bash[46934]: audit 2022-04-22T20:45:20.547575+0000 mgr.y (mgr.34107) 6816 : audit [DBG] from='client.52399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:21.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:21 smithi115 bash[46934]: audit 2022-04-22T20:45:20.976007+0000 mon.a (mon.0) 10283 : audit [DBG] from='client.? 172.21.15.115:0/4023694445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:22 smithi121 bash[36902]: cluster 2022-04-22T20:45:22.090659+0000 mgr.y (mgr.34107) 6817 : cluster [DBG] pgmap v7920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:22 smithi115 bash[45495]: cluster 2022-04-22T20:45:22.090659+0000 mgr.y (mgr.34107) 6817 : cluster [DBG] pgmap v7920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:22.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:22.380Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:22 smithi115 bash[46934]: cluster 2022-04-22T20:45:22.090659+0000 mgr.y (mgr.34107) 6817 : cluster [DBG] pgmap v7920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:23 smithi121 bash[36902]: audit 2022-04-22T20:45:23.172256+0000 mon.a (mon.0) 10284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:23 smithi121 bash[36902]: audit 2022-04-22T20:45:23.197379+0000 mon.a (mon.0) 10285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:23 smithi115 bash[45495]: audit 2022-04-22T20:45:23.172256+0000 mon.a (mon.0) 10284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:23 smithi115 bash[45495]: audit 2022-04-22T20:45:23.197379+0000 mon.a (mon.0) 10285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:23 smithi115 bash[46934]: audit 2022-04-22T20:45:23.172256+0000 mon.a (mon.0) 10284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:23.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:23 smithi115 bash[46934]: audit 2022-04-22T20:45:23.197379+0000 mon.a (mon.0) 10285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:24 smithi115 bash[45495]: cluster 2022-04-22T20:45:24.091775+0000 mgr.y (mgr.34107) 6818 : cluster [DBG] pgmap v7921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:24.263Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:24 smithi115 bash[46934]: cluster 2022-04-22T20:45:24.091775+0000 mgr.y (mgr.34107) 6818 : cluster [DBG] pgmap v7921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:24 smithi121 bash[36902]: cluster 2022-04-22T20:45:24.091775+0000 mgr.y (mgr.34107) 6818 : cluster [DBG] pgmap v7921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:25 smithi121 bash[36902]: audit 2022-04-22T20:45:24.631698+0000 mon.a (mon.0) 10286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:25 smithi115 bash[45495]: audit 2022-04-22T20:45:24.631698+0000 mon.a (mon.0) 10286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:25 smithi115 bash[46934]: audit 2022-04-22T20:45:24.631698+0000 mon.a (mon.0) 10286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:26 smithi121 bash[36902]: cluster 2022-04-22T20:45:26.092894+0000 mgr.y (mgr.34107) 6819 : cluster [DBG] pgmap v7922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:26 smithi115 bash[45495]: cluster 2022-04-22T20:45:26.092894+0000 mgr.y (mgr.34107) 6819 : cluster [DBG] pgmap v7922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:26 smithi115 bash[46934]: cluster 2022-04-22T20:45:26.092894+0000 mgr.y (mgr.34107) 6819 : cluster [DBG] pgmap v7922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:29 smithi121 bash[36902]: cluster 2022-04-22T20:45:28.093997+0000 mgr.y (mgr.34107) 6820 : cluster [DBG] pgmap v7923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:29 smithi121 bash[36902]: audit 2022-04-22T20:45:28.175283+0000 mon.a (mon.0) 10287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:29 smithi121 bash[36902]: audit 2022-04-22T20:45:28.200954+0000 mon.a (mon.0) 10288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[45495]: cluster 2022-04-22T20:45:28.093997+0000 mgr.y (mgr.34107) 6820 : cluster [DBG] pgmap v7923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[45495]: audit 2022-04-22T20:45:28.175283+0000 mon.a (mon.0) 10287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[45495]: audit 2022-04-22T20:45:28.200954+0000 mon.a (mon.0) 10288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[46934]: cluster 2022-04-22T20:45:28.093997+0000 mgr.y (mgr.34107) 6820 : cluster [DBG] pgmap v7923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[46934]: audit 2022-04-22T20:45:28.175283+0000 mon.a (mon.0) 10287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:29 smithi115 bash[46934]: audit 2022-04-22T20:45:28.200954+0000 mon.a (mon.0) 10288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:30 smithi121 bash[36902]: audit 2022-04-22T20:45:29.634746+0000 mon.a (mon.0) 10289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:30 smithi121 bash[36902]: cluster 2022-04-22T20:45:30.094942+0000 mgr.y (mgr.34107) 6821 : cluster [DBG] pgmap v7924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:30 smithi115 bash[45495]: audit 2022-04-22T20:45:29.634746+0000 mon.a (mon.0) 10289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:30 smithi115 bash[45495]: cluster 2022-04-22T20:45:30.094942+0000 mgr.y (mgr.34107) 6821 : cluster [DBG] pgmap v7924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:30 smithi115 bash[46934]: audit 2022-04-22T20:45:29.634746+0000 mon.a (mon.0) 10289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:30 smithi115 bash[46934]: cluster 2022-04-22T20:45:30.094942+0000 mgr.y (mgr.34107) 6821 : cluster [DBG] pgmap v7924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:32.381Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:33 smithi121 bash[36902]: cluster 2022-04-22T20:45:32.096129+0000 mgr.y (mgr.34107) 6822 : cluster [DBG] pgmap v7925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:33 smithi115 bash[45495]: cluster 2022-04-22T20:45:32.096129+0000 mgr.y (mgr.34107) 6822 : cluster [DBG] pgmap v7925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:33 smithi115 bash[46934]: cluster 2022-04-22T20:45:32.096129+0000 mgr.y (mgr.34107) 6822 : cluster [DBG] pgmap v7925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:34 smithi121 bash[36902]: audit 2022-04-22T20:45:33.178734+0000 mon.a (mon.0) 10290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:34 smithi121 bash[36902]: audit 2022-04-22T20:45:33.204843+0000 mon.a (mon.0) 10291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:34.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:34 smithi115 bash[45495]: audit 2022-04-22T20:45:33.178734+0000 mon.a (mon.0) 10290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:34 smithi115 bash[45495]: audit 2022-04-22T20:45:33.204843+0000 mon.a (mon.0) 10291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:34 smithi115 bash[46934]: audit 2022-04-22T20:45:33.178734+0000 mon.a (mon.0) 10290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:34 smithi115 bash[46934]: audit 2022-04-22T20:45:33.204843+0000 mon.a (mon.0) 10291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:35 smithi121 bash[36902]: cluster 2022-04-22T20:45:34.097209+0000 mgr.y (mgr.34107) 6823 : cluster [DBG] pgmap v7926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:35 smithi121 bash[36902]: audit 2022-04-22T20:45:34.637941+0000 mon.a (mon.0) 10292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:35 smithi115 bash[45495]: cluster 2022-04-22T20:45:34.097209+0000 mgr.y (mgr.34107) 6823 : cluster [DBG] pgmap v7926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:35 smithi115 bash[45495]: audit 2022-04-22T20:45:34.637941+0000 mon.a (mon.0) 10292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:35 smithi115 bash[46934]: cluster 2022-04-22T20:45:34.097209+0000 mgr.y (mgr.34107) 6823 : cluster [DBG] pgmap v7926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:35 smithi115 bash[46934]: audit 2022-04-22T20:45:34.637941+0000 mon.a (mon.0) 10292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:37 smithi121 bash[36902]: cluster 2022-04-22T20:45:36.098223+0000 mgr.y (mgr.34107) 6824 : cluster [DBG] pgmap v7927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:37 smithi115 bash[45495]: cluster 2022-04-22T20:45:36.098223+0000 mgr.y (mgr.34107) 6824 : cluster [DBG] pgmap v7927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:37.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:37 smithi115 bash[46934]: cluster 2022-04-22T20:45:36.098223+0000 mgr.y (mgr.34107) 6824 : cluster [DBG] pgmap v7927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:39 smithi121 bash[36902]: cluster 2022-04-22T20:45:38.099366+0000 mgr.y (mgr.34107) 6825 : cluster [DBG] pgmap v7928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:39 smithi121 bash[36902]: audit 2022-04-22T20:45:38.181573+0000 mon.a (mon.0) 10293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:39 smithi121 bash[36902]: audit 2022-04-22T20:45:38.208409+0000 mon.a (mon.0) 10294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[45495]: cluster 2022-04-22T20:45:38.099366+0000 mgr.y (mgr.34107) 6825 : cluster [DBG] pgmap v7928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[45495]: audit 2022-04-22T20:45:38.181573+0000 mon.a (mon.0) 10293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[45495]: audit 2022-04-22T20:45:38.208409+0000 mon.a (mon.0) 10294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[46934]: cluster 2022-04-22T20:45:38.099366+0000 mgr.y (mgr.34107) 6825 : cluster [DBG] pgmap v7928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[46934]: audit 2022-04-22T20:45:38.181573+0000 mon.a (mon.0) 10293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:39 smithi115 bash[46934]: audit 2022-04-22T20:45:38.208409+0000 mon.a (mon.0) 10294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:40 smithi121 bash[36902]: audit 2022-04-22T20:45:39.640761+0000 mon.a (mon.0) 10295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:40 smithi121 bash[36902]: cluster 2022-04-22T20:45:40.100303+0000 mgr.y (mgr.34107) 6826 : cluster [DBG] pgmap v7929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:40 smithi115 bash[45495]: audit 2022-04-22T20:45:39.640761+0000 mon.a (mon.0) 10295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:40 smithi115 bash[45495]: cluster 2022-04-22T20:45:40.100303+0000 mgr.y (mgr.34107) 6826 : cluster [DBG] pgmap v7929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:40 smithi115 bash[46934]: audit 2022-04-22T20:45:39.640761+0000 mon.a (mon.0) 10295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:40 smithi115 bash[46934]: cluster 2022-04-22T20:45:40.100303+0000 mgr.y (mgr.34107) 6826 : cluster [DBG] pgmap v7929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:42.383Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:43 smithi121 bash[36902]: cluster 2022-04-22T20:45:42.101453+0000 mgr.y (mgr.34107) 6827 : cluster [DBG] pgmap v7930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:43 smithi115 bash[45495]: cluster 2022-04-22T20:45:42.101453+0000 mgr.y (mgr.34107) 6827 : cluster [DBG] pgmap v7930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:43 smithi115 bash[46934]: cluster 2022-04-22T20:45:42.101453+0000 mgr.y (mgr.34107) 6827 : cluster [DBG] pgmap v7930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:44 smithi121 bash[36902]: audit 2022-04-22T20:45:43.184304+0000 mon.a (mon.0) 10296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:44 smithi121 bash[36902]: audit 2022-04-22T20:45:43.211773+0000 mon.a (mon.0) 10297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:44.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:44.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:44 smithi115 bash[45495]: audit 2022-04-22T20:45:43.184304+0000 mon.a (mon.0) 10296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:44.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:44 smithi115 bash[45495]: audit 2022-04-22T20:45:43.211773+0000 mon.a (mon.0) 10297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:44.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:44 smithi115 bash[46934]: audit 2022-04-22T20:45:43.184304+0000 mon.a (mon.0) 10296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:44.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:44 smithi115 bash[46934]: audit 2022-04-22T20:45:43.211773+0000 mon.a (mon.0) 10297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:45 smithi121 bash[36902]: cluster 2022-04-22T20:45:44.102413+0000 mgr.y (mgr.34107) 6828 : cluster [DBG] pgmap v7931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:45 smithi121 bash[36902]: audit 2022-04-22T20:45:44.644074+0000 mon.a (mon.0) 10298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:45 smithi115 bash[45495]: cluster 2022-04-22T20:45:44.102413+0000 mgr.y (mgr.34107) 6828 : cluster [DBG] pgmap v7931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:45 smithi115 bash[45495]: audit 2022-04-22T20:45:44.644074+0000 mon.a (mon.0) 10298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:45 smithi115 bash[46934]: cluster 2022-04-22T20:45:44.102413+0000 mgr.y (mgr.34107) 6828 : cluster [DBG] pgmap v7931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:45 smithi115 bash[46934]: audit 2022-04-22T20:45:44.644074+0000 mon.a (mon.0) 10298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:47 smithi121 bash[36902]: cluster 2022-04-22T20:45:46.103535+0000 mgr.y (mgr.34107) 6829 : cluster [DBG] pgmap v7932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:47 smithi115 bash[45495]: cluster 2022-04-22T20:45:46.103535+0000 mgr.y (mgr.34107) 6829 : cluster [DBG] pgmap v7932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:47 smithi115 bash[46934]: cluster 2022-04-22T20:45:46.103535+0000 mgr.y (mgr.34107) 6829 : cluster [DBG] pgmap v7932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:49 smithi121 bash[36902]: cluster 2022-04-22T20:45:48.104562+0000 mgr.y (mgr.34107) 6830 : cluster [DBG] pgmap v7933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:49 smithi121 bash[36902]: audit 2022-04-22T20:45:48.187336+0000 mon.a (mon.0) 10299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:49 smithi121 bash[36902]: audit 2022-04-22T20:45:48.215326+0000 mon.a (mon.0) 10300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[45495]: cluster 2022-04-22T20:45:48.104562+0000 mgr.y (mgr.34107) 6830 : cluster [DBG] pgmap v7933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[45495]: audit 2022-04-22T20:45:48.187336+0000 mon.a (mon.0) 10299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[45495]: audit 2022-04-22T20:45:48.215326+0000 mon.a (mon.0) 10300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[46934]: cluster 2022-04-22T20:45:48.104562+0000 mgr.y (mgr.34107) 6830 : cluster [DBG] pgmap v7933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[46934]: audit 2022-04-22T20:45:48.187336+0000 mon.a (mon.0) 10299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:49 smithi115 bash[46934]: audit 2022-04-22T20:45:48.215326+0000 mon.a (mon.0) 10300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:50 smithi121 bash[36902]: audit 2022-04-22T20:45:49.647142+0000 mon.a (mon.0) 10301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:50 smithi121 bash[36902]: cluster 2022-04-22T20:45:50.105524+0000 mgr.y (mgr.34107) 6831 : cluster [DBG] pgmap v7934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:50 smithi115 bash[45495]: audit 2022-04-22T20:45:49.647142+0000 mon.a (mon.0) 10301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:50 smithi115 bash[45495]: cluster 2022-04-22T20:45:50.105524+0000 mgr.y (mgr.34107) 6831 : cluster [DBG] pgmap v7934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:50 smithi115 bash[46934]: audit 2022-04-22T20:45:49.647142+0000 mon.a (mon.0) 10301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:50 smithi115 bash[46934]: cluster 2022-04-22T20:45:50.105524+0000 mgr.y (mgr.34107) 6831 : cluster [DBG] pgmap v7934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:51.364 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:45:51.706 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:45:51.706 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:45:51.706 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:45:51.706 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:45:51.706 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:45:51.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:45:51.708 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:45:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:51 smithi121 bash[36902]: audit 2022-04-22T20:45:51.337859+0000 mgr.y (mgr.34107) 6832 : audit [DBG] from='client.42480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:51 smithi115 bash[45495]: audit 2022-04-22T20:45:51.337859+0000 mgr.y (mgr.34107) 6832 : audit [DBG] from='client.42480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:51 smithi115 bash[46934]: audit 2022-04-22T20:45:51.337859+0000 mgr.y (mgr.34107) 6832 : audit [DBG] from='client.42480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:52.120 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:52.121 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:45:52.122 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:45:52.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:52.122 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:45:52.122 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:45:52.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:45:52.125 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:45:52.125 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:45:52.125 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:45:52.125 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:45:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:52 smithi121 bash[36902]: audit 2022-04-22T20:45:51.680092+0000 mgr.y (mgr.34107) 6833 : audit [DBG] from='client.52417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:52 smithi121 bash[36902]: cluster 2022-04-22T20:45:52.106589+0000 mgr.y (mgr.34107) 6834 : cluster [DBG] pgmap v7935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:52 smithi121 bash[36902]: audit 2022-04-22T20:45:52.114425+0000 mon.c (mon.1) 260 : audit [DBG] from='client.? 172.21.15.115:0/3775418554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:52.384Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[45495]: audit 2022-04-22T20:45:51.680092+0000 mgr.y (mgr.34107) 6833 : audit [DBG] from='client.52417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[45495]: cluster 2022-04-22T20:45:52.106589+0000 mgr.y (mgr.34107) 6834 : cluster [DBG] pgmap v7935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:52.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[45495]: audit 2022-04-22T20:45:52.114425+0000 mon.c (mon.1) 260 : audit [DBG] from='client.? 172.21.15.115:0/3775418554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:52.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[46934]: audit 2022-04-22T20:45:51.680092+0000 mgr.y (mgr.34107) 6833 : audit [DBG] from='client.52417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:45:52.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[46934]: cluster 2022-04-22T20:45:52.106589+0000 mgr.y (mgr.34107) 6834 : cluster [DBG] pgmap v7935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:52.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:52 smithi115 bash[46934]: audit 2022-04-22T20:45:52.114425+0000 mon.c (mon.1) 260 : audit [DBG] from='client.? 172.21.15.115:0/3775418554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:45:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:53 smithi121 bash[36902]: audit 2022-04-22T20:45:53.189741+0000 mon.a (mon.0) 10302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:53 smithi121 bash[36902]: audit 2022-04-22T20:45:53.218945+0000 mon.a (mon.0) 10303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:53 smithi115 bash[45495]: audit 2022-04-22T20:45:53.189741+0000 mon.a (mon.0) 10302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:53 smithi115 bash[45495]: audit 2022-04-22T20:45:53.218945+0000 mon.a (mon.0) 10303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:53 smithi115 bash[46934]: audit 2022-04-22T20:45:53.189741+0000 mon.a (mon.0) 10302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:53 smithi115 bash[46934]: audit 2022-04-22T20:45:53.218945+0000 mon.a (mon.0) 10303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:54 smithi115 bash[45495]: cluster 2022-04-22T20:45:54.107644+0000 mgr.y (mgr.34107) 6835 : cluster [DBG] pgmap v7936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:45:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:45:54.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:45:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:54 smithi115 bash[46934]: cluster 2022-04-22T20:45:54.107644+0000 mgr.y (mgr.34107) 6835 : cluster [DBG] pgmap v7936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:54 smithi121 bash[36902]: cluster 2022-04-22T20:45:54.107644+0000 mgr.y (mgr.34107) 6835 : cluster [DBG] pgmap v7936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:55 smithi121 bash[36902]: audit 2022-04-22T20:45:54.650199+0000 mon.a (mon.0) 10304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:55 smithi115 bash[45495]: audit 2022-04-22T20:45:54.650199+0000 mon.a (mon.0) 10304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:55 smithi115 bash[46934]: audit 2022-04-22T20:45:54.650199+0000 mon.a (mon.0) 10304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:45:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:56 smithi121 bash[36902]: cluster 2022-04-22T20:45:56.108759+0000 mgr.y (mgr.34107) 6836 : cluster [DBG] pgmap v7937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:56.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:56 smithi115 bash[45495]: cluster 2022-04-22T20:45:56.108759+0000 mgr.y (mgr.34107) 6836 : cluster [DBG] pgmap v7937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:56.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:56 smithi115 bash[46934]: cluster 2022-04-22T20:45:56.108759+0000 mgr.y (mgr.34107) 6836 : cluster [DBG] pgmap v7937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:59 smithi121 bash[36902]: cluster 2022-04-22T20:45:58.109827+0000 mgr.y (mgr.34107) 6837 : cluster [DBG] pgmap v7938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:59 smithi121 bash[36902]: audit 2022-04-22T20:45:58.192648+0000 mon.a (mon.0) 10305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:45:59 smithi121 bash[36902]: audit 2022-04-22T20:45:58.222570+0000 mon.a (mon.0) 10306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[45495]: cluster 2022-04-22T20:45:58.109827+0000 mgr.y (mgr.34107) 6837 : cluster [DBG] pgmap v7938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[45495]: audit 2022-04-22T20:45:58.192648+0000 mon.a (mon.0) 10305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[45495]: audit 2022-04-22T20:45:58.222570+0000 mon.a (mon.0) 10306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:45:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[46934]: cluster 2022-04-22T20:45:58.109827+0000 mgr.y (mgr.34107) 6837 : cluster [DBG] pgmap v7938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:45:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[46934]: audit 2022-04-22T20:45:58.192648+0000 mon.a (mon.0) 10305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:45:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:45:59 smithi115 bash[46934]: audit 2022-04-22T20:45:58.222570+0000 mon.a (mon.0) 10306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:00 smithi121 bash[36902]: audit 2022-04-22T20:45:59.653381+0000 mon.a (mon.0) 10307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:00 smithi121 bash[36902]: cluster 2022-04-22T20:46:00.110847+0000 mgr.y (mgr.34107) 6838 : cluster [DBG] pgmap v7939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:00 smithi115 bash[45495]: audit 2022-04-22T20:45:59.653381+0000 mon.a (mon.0) 10307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:00 smithi115 bash[45495]: cluster 2022-04-22T20:46:00.110847+0000 mgr.y (mgr.34107) 6838 : cluster [DBG] pgmap v7939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:00 smithi115 bash[46934]: audit 2022-04-22T20:45:59.653381+0000 mon.a (mon.0) 10307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:00 smithi115 bash[46934]: cluster 2022-04-22T20:46:00.110847+0000 mgr.y (mgr.34107) 6838 : cluster [DBG] pgmap v7939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:02.384Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:03 smithi121 bash[36902]: cluster 2022-04-22T20:46:02.112077+0000 mgr.y (mgr.34107) 6839 : cluster [DBG] pgmap v7940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:03 smithi115 bash[45495]: cluster 2022-04-22T20:46:02.112077+0000 mgr.y (mgr.34107) 6839 : cluster [DBG] pgmap v7940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:03 smithi115 bash[46934]: cluster 2022-04-22T20:46:02.112077+0000 mgr.y (mgr.34107) 6839 : cluster [DBG] pgmap v7940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:04 smithi121 bash[36902]: audit 2022-04-22T20:46:03.195749+0000 mon.a (mon.0) 10308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:04 smithi121 bash[36902]: audit 2022-04-22T20:46:03.226061+0000 mon.a (mon.0) 10309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:04.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:04 smithi115 bash[46934]: audit 2022-04-22T20:46:03.195749+0000 mon.a (mon.0) 10308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:04 smithi115 bash[46934]: audit 2022-04-22T20:46:03.226061+0000 mon.a (mon.0) 10309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:04 smithi115 bash[45495]: audit 2022-04-22T20:46:03.195749+0000 mon.a (mon.0) 10308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:04 smithi115 bash[45495]: audit 2022-04-22T20:46:03.226061+0000 mon.a (mon.0) 10309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:05 smithi121 bash[36902]: cluster 2022-04-22T20:46:04.113095+0000 mgr.y (mgr.34107) 6840 : cluster [DBG] pgmap v7941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:05 smithi121 bash[36902]: audit 2022-04-22T20:46:04.656090+0000 mon.a (mon.0) 10310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:05.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:05 smithi115 bash[46934]: cluster 2022-04-22T20:46:04.113095+0000 mgr.y (mgr.34107) 6840 : cluster [DBG] pgmap v7941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:05.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:05 smithi115 bash[46934]: audit 2022-04-22T20:46:04.656090+0000 mon.a (mon.0) 10310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:05 smithi115 bash[45495]: cluster 2022-04-22T20:46:04.113095+0000 mgr.y (mgr.34107) 6840 : cluster [DBG] pgmap v7941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:05.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:05 smithi115 bash[45495]: audit 2022-04-22T20:46:04.656090+0000 mon.a (mon.0) 10310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:07 smithi121 bash[36902]: cluster 2022-04-22T20:46:06.114209+0000 mgr.y (mgr.34107) 6841 : cluster [DBG] pgmap v7942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:07 smithi115 bash[45495]: cluster 2022-04-22T20:46:06.114209+0000 mgr.y (mgr.34107) 6841 : cluster [DBG] pgmap v7942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:07.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:07 smithi115 bash[46934]: cluster 2022-04-22T20:46:06.114209+0000 mgr.y (mgr.34107) 6841 : cluster [DBG] pgmap v7942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:09 smithi121 bash[36902]: cluster 2022-04-22T20:46:08.115303+0000 mgr.y (mgr.34107) 6842 : cluster [DBG] pgmap v7943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:09 smithi121 bash[36902]: audit 2022-04-22T20:46:08.198479+0000 mon.a (mon.0) 10311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:09 smithi121 bash[36902]: audit 2022-04-22T20:46:08.229711+0000 mon.a (mon.0) 10312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[45495]: cluster 2022-04-22T20:46:08.115303+0000 mgr.y (mgr.34107) 6842 : cluster [DBG] pgmap v7943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:09.616 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[45495]: audit 2022-04-22T20:46:08.198479+0000 mon.a (mon.0) 10311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:09.616 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[45495]: audit 2022-04-22T20:46:08.229711+0000 mon.a (mon.0) 10312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:09.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[46934]: cluster 2022-04-22T20:46:08.115303+0000 mgr.y (mgr.34107) 6842 : cluster [DBG] pgmap v7943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:09.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[46934]: audit 2022-04-22T20:46:08.198479+0000 mon.a (mon.0) 10311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:09.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:09 smithi115 bash[46934]: audit 2022-04-22T20:46:08.229711+0000 mon.a (mon.0) 10312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:10 smithi121 bash[36902]: audit 2022-04-22T20:46:09.659199+0000 mon.a (mon.0) 10313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:10 smithi121 bash[36902]: cluster 2022-04-22T20:46:10.116269+0000 mgr.y (mgr.34107) 6843 : cluster [DBG] pgmap v7944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:10 smithi115 bash[45495]: audit 2022-04-22T20:46:09.659199+0000 mon.a (mon.0) 10313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:10.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:10 smithi115 bash[45495]: cluster 2022-04-22T20:46:10.116269+0000 mgr.y (mgr.34107) 6843 : cluster [DBG] pgmap v7944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:10 smithi115 bash[46934]: audit 2022-04-22T20:46:09.659199+0000 mon.a (mon.0) 10313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:10 smithi115 bash[46934]: cluster 2022-04-22T20:46:10.116269+0000 mgr.y (mgr.34107) 6843 : cluster [DBG] pgmap v7944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:12.384Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:13 smithi121 bash[36902]: cluster 2022-04-22T20:46:12.117448+0000 mgr.y (mgr.34107) 6844 : cluster [DBG] pgmap v7945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:13 smithi115 bash[45495]: cluster 2022-04-22T20:46:12.117448+0000 mgr.y (mgr.34107) 6844 : cluster [DBG] pgmap v7945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:13 smithi115 bash[46934]: cluster 2022-04-22T20:46:12.117448+0000 mgr.y (mgr.34107) 6844 : cluster [DBG] pgmap v7945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:14 smithi121 bash[36902]: audit 2022-04-22T20:46:13.201014+0000 mon.a (mon.0) 10314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:14 smithi121 bash[36902]: audit 2022-04-22T20:46:13.233301+0000 mon.a (mon.0) 10315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:14 smithi115 bash[46934]: audit 2022-04-22T20:46:13.201014+0000 mon.a (mon.0) 10314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:14 smithi115 bash[46934]: audit 2022-04-22T20:46:13.233301+0000 mon.a (mon.0) 10315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:14.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:14 smithi115 bash[45495]: audit 2022-04-22T20:46:13.201014+0000 mon.a (mon.0) 10314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:14 smithi115 bash[45495]: audit 2022-04-22T20:46:13.233301+0000 mon.a (mon.0) 10315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:15 smithi121 bash[36902]: cluster 2022-04-22T20:46:14.118515+0000 mgr.y (mgr.34107) 6845 : cluster [DBG] pgmap v7946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:15 smithi121 bash[36902]: audit 2022-04-22T20:46:14.662335+0000 mon.a (mon.0) 10316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:15 smithi115 bash[45495]: cluster 2022-04-22T20:46:14.118515+0000 mgr.y (mgr.34107) 6845 : cluster [DBG] pgmap v7946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:15 smithi115 bash[45495]: audit 2022-04-22T20:46:14.662335+0000 mon.a (mon.0) 10316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:15 smithi115 bash[46934]: cluster 2022-04-22T20:46:14.118515+0000 mgr.y (mgr.34107) 6845 : cluster [DBG] pgmap v7946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:15 smithi115 bash[46934]: audit 2022-04-22T20:46:14.662335+0000 mon.a (mon.0) 10316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:16 smithi121 bash[36902]: audit 2022-04-22T20:46:15.571399+0000 mon.a (mon.0) 10317 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:46:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:16 smithi121 bash[36902]: audit 2022-04-22T20:46:15.572093+0000 mon.a (mon.0) 10318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:46:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:16 smithi115 bash[45495]: audit 2022-04-22T20:46:15.571399+0000 mon.a (mon.0) 10317 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:46:16.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:16 smithi115 bash[45495]: audit 2022-04-22T20:46:15.572093+0000 mon.a (mon.0) 10318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:46:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:16 smithi115 bash[46934]: audit 2022-04-22T20:46:15.571399+0000 mon.a (mon.0) 10317 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:46:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:16 smithi115 bash[46934]: audit 2022-04-22T20:46:15.572093+0000 mon.a (mon.0) 10318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:46:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:17 smithi121 bash[36902]: cluster 2022-04-22T20:46:16.119517+0000 mgr.y (mgr.34107) 6846 : cluster [DBG] pgmap v7947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:17 smithi115 bash[45495]: cluster 2022-04-22T20:46:16.119517+0000 mgr.y (mgr.34107) 6846 : cluster [DBG] pgmap v7947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:17 smithi115 bash[46934]: cluster 2022-04-22T20:46:16.119517+0000 mgr.y (mgr.34107) 6846 : cluster [DBG] pgmap v7947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:18 smithi121 bash[36902]: audit 2022-04-22T20:46:17.584191+0000 mon.a (mon.0) 10319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:46:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:18 smithi121 bash[36902]: audit 2022-04-22T20:46:17.587270+0000 mon.a (mon.0) 10320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:46:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:18 smithi121 bash[36902]: audit 2022-04-22T20:46:17.589046+0000 mon.a (mon.0) 10321 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:46:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:18 smithi121 bash[36902]: audit 2022-04-22T20:46:17.938381+0000 mon.a (mon.0) 10322 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:46:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[46934]: audit 2022-04-22T20:46:17.584191+0000 mon.a (mon.0) 10319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:46:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[46934]: audit 2022-04-22T20:46:17.587270+0000 mon.a (mon.0) 10320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:46:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[46934]: audit 2022-04-22T20:46:17.589046+0000 mon.a (mon.0) 10321 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:46:18.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[46934]: audit 2022-04-22T20:46:17.938381+0000 mon.a (mon.0) 10322 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:46:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[45495]: audit 2022-04-22T20:46:17.584191+0000 mon.a (mon.0) 10319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:46:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[45495]: audit 2022-04-22T20:46:17.587270+0000 mon.a (mon.0) 10320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:46:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[45495]: audit 2022-04-22T20:46:17.589046+0000 mon.a (mon.0) 10321 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:46:18.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:18 smithi115 bash[45495]: audit 2022-04-22T20:46:17.938381+0000 mon.a (mon.0) 10322 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:46:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:19 smithi121 bash[36902]: cluster 2022-04-22T20:46:18.120505+0000 mgr.y (mgr.34107) 6847 : cluster [DBG] pgmap v7948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:19 smithi121 bash[36902]: audit 2022-04-22T20:46:18.211971+0000 mon.a (mon.0) 10323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:19 smithi121 bash[36902]: audit 2022-04-22T20:46:18.236771+0000 mon.a (mon.0) 10324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[46934]: cluster 2022-04-22T20:46:18.120505+0000 mgr.y (mgr.34107) 6847 : cluster [DBG] pgmap v7948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[46934]: audit 2022-04-22T20:46:18.211971+0000 mon.a (mon.0) 10323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[46934]: audit 2022-04-22T20:46:18.236771+0000 mon.a (mon.0) 10324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[45495]: cluster 2022-04-22T20:46:18.120505+0000 mgr.y (mgr.34107) 6847 : cluster [DBG] pgmap v7948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[45495]: audit 2022-04-22T20:46:18.211971+0000 mon.a (mon.0) 10323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:19 smithi115 bash[45495]: audit 2022-04-22T20:46:18.236771+0000 mon.a (mon.0) 10324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:20 smithi121 bash[36902]: audit 2022-04-22T20:46:19.665418+0000 mon.a (mon.0) 10325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:20 smithi121 bash[36902]: cluster 2022-04-22T20:46:20.121442+0000 mgr.y (mgr.34107) 6848 : cluster [DBG] pgmap v7949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:20 smithi115 bash[46934]: audit 2022-04-22T20:46:19.665418+0000 mon.a (mon.0) 10325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:20 smithi115 bash[46934]: cluster 2022-04-22T20:46:20.121442+0000 mgr.y (mgr.34107) 6848 : cluster [DBG] pgmap v7949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:20 smithi115 bash[45495]: audit 2022-04-22T20:46:19.665418+0000 mon.a (mon.0) 10325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:20 smithi115 bash[45495]: cluster 2022-04-22T20:46:20.121442+0000 mgr.y (mgr.34107) 6848 : cluster [DBG] pgmap v7949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:22.506 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:46:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:22.385Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:22.855 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:46:22.855 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:46:22.855 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:46:22.855 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:46:22.856 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:46:22.857 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:46:23.264 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:46:23.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:46:23.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:46:23.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:23.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:46:23.265 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:46:23.266 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:46:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:23 smithi121 bash[36902]: cluster 2022-04-22T20:46:22.122728+0000 mgr.y (mgr.34107) 6849 : cluster [DBG] pgmap v7950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:23 smithi115 bash[45495]: cluster 2022-04-22T20:46:22.122728+0000 mgr.y (mgr.34107) 6849 : cluster [DBG] pgmap v7950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:23.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:23 smithi115 bash[46934]: cluster 2022-04-22T20:46:22.122728+0000 mgr.y (mgr.34107) 6849 : cluster [DBG] pgmap v7950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:24 smithi121 bash[36902]: audit 2022-04-22T20:46:22.481606+0000 mgr.y (mgr.34107) 6850 : audit [DBG] from='client.52429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:24 smithi121 bash[36902]: audit 2022-04-22T20:46:22.828700+0000 mgr.y (mgr.34107) 6851 : audit [DBG] from='client.52435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:24 smithi121 bash[36902]: audit 2022-04-22T20:46:23.208048+0000 mon.a (mon.0) 10326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:24 smithi121 bash[36902]: audit 2022-04-22T20:46:23.240316+0000 mon.a (mon.0) 10327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:24 smithi121 bash[36902]: audit 2022-04-22T20:46:23.258028+0000 mon.a (mon.0) 10328 : audit [DBG] from='client.? 172.21.15.115:0/1151638407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:24.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[45495]: audit 2022-04-22T20:46:22.481606+0000 mgr.y (mgr.34107) 6850 : audit [DBG] from='client.52429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[45495]: audit 2022-04-22T20:46:22.828700+0000 mgr.y (mgr.34107) 6851 : audit [DBG] from='client.52435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[45495]: audit 2022-04-22T20:46:23.208048+0000 mon.a (mon.0) 10326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[45495]: audit 2022-04-22T20:46:23.240316+0000 mon.a (mon.0) 10327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[45495]: audit 2022-04-22T20:46:23.258028+0000 mon.a (mon.0) 10328 : audit [DBG] from='client.? 172.21.15.115:0/1151638407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[46934]: audit 2022-04-22T20:46:22.481606+0000 mgr.y (mgr.34107) 6850 : audit [DBG] from='client.52429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[46934]: audit 2022-04-22T20:46:22.828700+0000 mgr.y (mgr.34107) 6851 : audit [DBG] from='client.52435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[46934]: audit 2022-04-22T20:46:23.208048+0000 mon.a (mon.0) 10326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[46934]: audit 2022-04-22T20:46:23.240316+0000 mon.a (mon.0) 10327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:24 smithi115 bash[46934]: audit 2022-04-22T20:46:23.258028+0000 mon.a (mon.0) 10328 : audit [DBG] from='client.? 172.21.15.115:0/1151638407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:25 smithi121 bash[36902]: cluster 2022-04-22T20:46:24.123848+0000 mgr.y (mgr.34107) 6852 : cluster [DBG] pgmap v7951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:25 smithi121 bash[36902]: audit 2022-04-22T20:46:24.668471+0000 mon.a (mon.0) 10329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:25 smithi115 bash[45495]: cluster 2022-04-22T20:46:24.123848+0000 mgr.y (mgr.34107) 6852 : cluster [DBG] pgmap v7951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:25 smithi115 bash[45495]: audit 2022-04-22T20:46:24.668471+0000 mon.a (mon.0) 10329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:25 smithi115 bash[46934]: cluster 2022-04-22T20:46:24.123848+0000 mgr.y (mgr.34107) 6852 : cluster [DBG] pgmap v7951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:25 smithi115 bash[46934]: audit 2022-04-22T20:46:24.668471+0000 mon.a (mon.0) 10329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:27 smithi121 bash[36902]: cluster 2022-04-22T20:46:26.124890+0000 mgr.y (mgr.34107) 6853 : cluster [DBG] pgmap v7952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:27.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:27 smithi115 bash[45495]: cluster 2022-04-22T20:46:26.124890+0000 mgr.y (mgr.34107) 6853 : cluster [DBG] pgmap v7952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:27 smithi115 bash[46934]: cluster 2022-04-22T20:46:26.124890+0000 mgr.y (mgr.34107) 6853 : cluster [DBG] pgmap v7952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:29 smithi121 bash[36902]: cluster 2022-04-22T20:46:28.125950+0000 mgr.y (mgr.34107) 6854 : cluster [DBG] pgmap v7953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:29 smithi121 bash[36902]: audit 2022-04-22T20:46:28.210919+0000 mon.a (mon.0) 10330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:29 smithi121 bash[36902]: audit 2022-04-22T20:46:28.243715+0000 mon.a (mon.0) 10331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[45495]: cluster 2022-04-22T20:46:28.125950+0000 mgr.y (mgr.34107) 6854 : cluster [DBG] pgmap v7953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[45495]: audit 2022-04-22T20:46:28.210919+0000 mon.a (mon.0) 10330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[45495]: audit 2022-04-22T20:46:28.243715+0000 mon.a (mon.0) 10331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[46934]: cluster 2022-04-22T20:46:28.125950+0000 mgr.y (mgr.34107) 6854 : cluster [DBG] pgmap v7953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[46934]: audit 2022-04-22T20:46:28.210919+0000 mon.a (mon.0) 10330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:29 smithi115 bash[46934]: audit 2022-04-22T20:46:28.243715+0000 mon.a (mon.0) 10331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:30 smithi121 bash[36902]: audit 2022-04-22T20:46:29.671423+0000 mon.a (mon.0) 10332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:30 smithi121 bash[36902]: cluster 2022-04-22T20:46:30.126995+0000 mgr.y (mgr.34107) 6855 : cluster [DBG] pgmap v7954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:30 smithi115 bash[46934]: audit 2022-04-22T20:46:29.671423+0000 mon.a (mon.0) 10332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:30 smithi115 bash[46934]: cluster 2022-04-22T20:46:30.126995+0000 mgr.y (mgr.34107) 6855 : cluster [DBG] pgmap v7954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:30 smithi115 bash[45495]: audit 2022-04-22T20:46:29.671423+0000 mon.a (mon.0) 10332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:30 smithi115 bash[45495]: cluster 2022-04-22T20:46:30.126995+0000 mgr.y (mgr.34107) 6855 : cluster [DBG] pgmap v7954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:32.387Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:33 smithi121 bash[36902]: cluster 2022-04-22T20:46:32.128206+0000 mgr.y (mgr.34107) 6856 : cluster [DBG] pgmap v7955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:33 smithi115 bash[45495]: cluster 2022-04-22T20:46:32.128206+0000 mgr.y (mgr.34107) 6856 : cluster [DBG] pgmap v7955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:33 smithi115 bash[46934]: cluster 2022-04-22T20:46:32.128206+0000 mgr.y (mgr.34107) 6856 : cluster [DBG] pgmap v7955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:34 smithi121 bash[36902]: audit 2022-04-22T20:46:33.213480+0000 mon.a (mon.0) 10333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:34 smithi121 bash[36902]: audit 2022-04-22T20:46:33.247224+0000 mon.a (mon.0) 10334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:34.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:34 smithi115 bash[45495]: audit 2022-04-22T20:46:33.213480+0000 mon.a (mon.0) 10333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:34 smithi115 bash[45495]: audit 2022-04-22T20:46:33.247224+0000 mon.a (mon.0) 10334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:34 smithi115 bash[46934]: audit 2022-04-22T20:46:33.213480+0000 mon.a (mon.0) 10333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:34 smithi115 bash[46934]: audit 2022-04-22T20:46:33.247224+0000 mon.a (mon.0) 10334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:35 smithi121 bash[36902]: cluster 2022-04-22T20:46:34.129260+0000 mgr.y (mgr.34107) 6857 : cluster [DBG] pgmap v7956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:35 smithi121 bash[36902]: audit 2022-04-22T20:46:34.674599+0000 mon.a (mon.0) 10335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:35 smithi115 bash[45495]: cluster 2022-04-22T20:46:34.129260+0000 mgr.y (mgr.34107) 6857 : cluster [DBG] pgmap v7956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:35 smithi115 bash[45495]: audit 2022-04-22T20:46:34.674599+0000 mon.a (mon.0) 10335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:35.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:35 smithi115 bash[46934]: cluster 2022-04-22T20:46:34.129260+0000 mgr.y (mgr.34107) 6857 : cluster [DBG] pgmap v7956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:35 smithi115 bash[46934]: audit 2022-04-22T20:46:34.674599+0000 mon.a (mon.0) 10335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:37 smithi121 bash[36902]: cluster 2022-04-22T20:46:36.130375+0000 mgr.y (mgr.34107) 6858 : cluster [DBG] pgmap v7957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:37 smithi115 bash[45495]: cluster 2022-04-22T20:46:36.130375+0000 mgr.y (mgr.34107) 6858 : cluster [DBG] pgmap v7957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:37.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:37 smithi115 bash[46934]: cluster 2022-04-22T20:46:36.130375+0000 mgr.y (mgr.34107) 6858 : cluster [DBG] pgmap v7957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:39 smithi121 bash[36902]: cluster 2022-04-22T20:46:38.131559+0000 mgr.y (mgr.34107) 6859 : cluster [DBG] pgmap v7958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:39 smithi121 bash[36902]: audit 2022-04-22T20:46:38.216293+0000 mon.a (mon.0) 10336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:39 smithi121 bash[36902]: audit 2022-04-22T20:46:38.250746+0000 mon.a (mon.0) 10337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[45495]: cluster 2022-04-22T20:46:38.131559+0000 mgr.y (mgr.34107) 6859 : cluster [DBG] pgmap v7958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[45495]: audit 2022-04-22T20:46:38.216293+0000 mon.a (mon.0) 10336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[45495]: audit 2022-04-22T20:46:38.250746+0000 mon.a (mon.0) 10337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[46934]: cluster 2022-04-22T20:46:38.131559+0000 mgr.y (mgr.34107) 6859 : cluster [DBG] pgmap v7958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[46934]: audit 2022-04-22T20:46:38.216293+0000 mon.a (mon.0) 10336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:39.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:39 smithi115 bash[46934]: audit 2022-04-22T20:46:38.250746+0000 mon.a (mon.0) 10337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:40 smithi121 bash[36902]: audit 2022-04-22T20:46:39.677690+0000 mon.a (mon.0) 10338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:40 smithi121 bash[36902]: cluster 2022-04-22T20:46:40.132500+0000 mgr.y (mgr.34107) 6860 : cluster [DBG] pgmap v7959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:40 smithi115 bash[45495]: audit 2022-04-22T20:46:39.677690+0000 mon.a (mon.0) 10338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:40 smithi115 bash[45495]: cluster 2022-04-22T20:46:40.132500+0000 mgr.y (mgr.34107) 6860 : cluster [DBG] pgmap v7959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:40 smithi115 bash[46934]: audit 2022-04-22T20:46:39.677690+0000 mon.a (mon.0) 10338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:40 smithi115 bash[46934]: cluster 2022-04-22T20:46:40.132500+0000 mgr.y (mgr.34107) 6860 : cluster [DBG] pgmap v7959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:42.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:43 smithi121 bash[36902]: cluster 2022-04-22T20:46:42.133754+0000 mgr.y (mgr.34107) 6861 : cluster [DBG] pgmap v7960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:43 smithi115 bash[45495]: cluster 2022-04-22T20:46:42.133754+0000 mgr.y (mgr.34107) 6861 : cluster [DBG] pgmap v7960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:43.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:43 smithi115 bash[46934]: cluster 2022-04-22T20:46:42.133754+0000 mgr.y (mgr.34107) 6861 : cluster [DBG] pgmap v7960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:44 smithi121 bash[36902]: audit 2022-04-22T20:46:43.218937+0000 mon.a (mon.0) 10339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:44 smithi121 bash[36902]: audit 2022-04-22T20:46:43.254246+0000 mon.a (mon.0) 10340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:44.268Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:44 smithi115 bash[45495]: audit 2022-04-22T20:46:43.218937+0000 mon.a (mon.0) 10339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:44 smithi115 bash[45495]: audit 2022-04-22T20:46:43.254246+0000 mon.a (mon.0) 10340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:44 smithi115 bash[46934]: audit 2022-04-22T20:46:43.218937+0000 mon.a (mon.0) 10339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:44 smithi115 bash[46934]: audit 2022-04-22T20:46:43.254246+0000 mon.a (mon.0) 10340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:45 smithi121 bash[36902]: cluster 2022-04-22T20:46:44.134866+0000 mgr.y (mgr.34107) 6862 : cluster [DBG] pgmap v7961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:45 smithi121 bash[36902]: audit 2022-04-22T20:46:44.681031+0000 mon.a (mon.0) 10341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:45 smithi115 bash[45495]: cluster 2022-04-22T20:46:44.134866+0000 mgr.y (mgr.34107) 6862 : cluster [DBG] pgmap v7961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:45 smithi115 bash[45495]: audit 2022-04-22T20:46:44.681031+0000 mon.a (mon.0) 10341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:45 smithi115 bash[46934]: cluster 2022-04-22T20:46:44.134866+0000 mgr.y (mgr.34107) 6862 : cluster [DBG] pgmap v7961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:45 smithi115 bash[46934]: audit 2022-04-22T20:46:44.681031+0000 mon.a (mon.0) 10341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:47 smithi121 bash[36902]: cluster 2022-04-22T20:46:46.136008+0000 mgr.y (mgr.34107) 6863 : cluster [DBG] pgmap v7962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:47 smithi115 bash[45495]: cluster 2022-04-22T20:46:46.136008+0000 mgr.y (mgr.34107) 6863 : cluster [DBG] pgmap v7962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:47 smithi115 bash[46934]: cluster 2022-04-22T20:46:46.136008+0000 mgr.y (mgr.34107) 6863 : cluster [DBG] pgmap v7962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:49 smithi121 bash[36902]: cluster 2022-04-22T20:46:48.137293+0000 mgr.y (mgr.34107) 6864 : cluster [DBG] pgmap v7963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:49 smithi121 bash[36902]: audit 2022-04-22T20:46:48.221751+0000 mon.a (mon.0) 10342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:49 smithi121 bash[36902]: audit 2022-04-22T20:46:48.257842+0000 mon.a (mon.0) 10343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[45495]: cluster 2022-04-22T20:46:48.137293+0000 mgr.y (mgr.34107) 6864 : cluster [DBG] pgmap v7963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[45495]: audit 2022-04-22T20:46:48.221751+0000 mon.a (mon.0) 10342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[45495]: audit 2022-04-22T20:46:48.257842+0000 mon.a (mon.0) 10343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[46934]: cluster 2022-04-22T20:46:48.137293+0000 mgr.y (mgr.34107) 6864 : cluster [DBG] pgmap v7963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[46934]: audit 2022-04-22T20:46:48.221751+0000 mon.a (mon.0) 10342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:49 smithi115 bash[46934]: audit 2022-04-22T20:46:48.257842+0000 mon.a (mon.0) 10343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:50 smithi121 bash[36902]: audit 2022-04-22T20:46:49.684127+0000 mon.a (mon.0) 10344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:50 smithi121 bash[36902]: cluster 2022-04-22T20:46:50.138296+0000 mgr.y (mgr.34107) 6865 : cluster [DBG] pgmap v7964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:50 smithi115 bash[45495]: audit 2022-04-22T20:46:49.684127+0000 mon.a (mon.0) 10344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:50.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:50 smithi115 bash[45495]: cluster 2022-04-22T20:46:50.138296+0000 mgr.y (mgr.34107) 6865 : cluster [DBG] pgmap v7964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:50 smithi115 bash[46934]: audit 2022-04-22T20:46:49.684127+0000 mon.a (mon.0) 10344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:50.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:50 smithi115 bash[46934]: cluster 2022-04-22T20:46:50.138296+0000 mgr.y (mgr.34107) 6865 : cluster [DBG] pgmap v7964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:52.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:53 smithi121 bash[36902]: cluster 2022-04-22T20:46:52.139599+0000 mgr.y (mgr.34107) 6866 : cluster [DBG] pgmap v7965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:53 smithi115 bash[45495]: cluster 2022-04-22T20:46:52.139599+0000 mgr.y (mgr.34107) 6866 : cluster [DBG] pgmap v7965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:53 smithi115 bash[46934]: cluster 2022-04-22T20:46:52.139599+0000 mgr.y (mgr.34107) 6866 : cluster [DBG] pgmap v7965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:53.636 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:46:53.987 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:46:53.987 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:46:53.987 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:46:53.987 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:46:53.988 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:46:53.988 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:46:53.988 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:46:53.988 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:46:53.989 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:46:53.989 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:46:53.989 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:46:53.989 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:46:53.989 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:46:53.990 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:46:54.400 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:46:54.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:46:54.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:54.401 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:46:54.402 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:46:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:54 smithi121 bash[36902]: audit 2022-04-22T20:46:53.224457+0000 mon.a (mon.0) 10345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:54 smithi121 bash[36902]: audit 2022-04-22T20:46:53.261439+0000 mon.a (mon.0) 10346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:54.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:46:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:46:54.268Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:46:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:54 smithi115 bash[45495]: audit 2022-04-22T20:46:53.224457+0000 mon.a (mon.0) 10345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:54 smithi115 bash[45495]: audit 2022-04-22T20:46:53.261439+0000 mon.a (mon.0) 10346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:54 smithi115 bash[46934]: audit 2022-04-22T20:46:53.224457+0000 mon.a (mon.0) 10345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:54 smithi115 bash[46934]: audit 2022-04-22T20:46:53.261439+0000 mon.a (mon.0) 10346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:55 smithi121 bash[36902]: audit 2022-04-22T20:46:53.614578+0000 mgr.y (mgr.34107) 6867 : audit [DBG] from='client.42516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:55 smithi121 bash[36902]: audit 2022-04-22T20:46:53.961270+0000 mgr.y (mgr.34107) 6868 : audit [DBG] from='client.42522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:55 smithi121 bash[36902]: cluster 2022-04-22T20:46:54.140630+0000 mgr.y (mgr.34107) 6869 : cluster [DBG] pgmap v7966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:55 smithi121 bash[36902]: audit 2022-04-22T20:46:54.394127+0000 mon.a (mon.0) 10347 : audit [DBG] from='client.? 172.21.15.115:0/1806996587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:55 smithi121 bash[36902]: audit 2022-04-22T20:46:54.687388+0000 mon.a (mon.0) 10348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[45495]: audit 2022-04-22T20:46:53.614578+0000 mgr.y (mgr.34107) 6867 : audit [DBG] from='client.42516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[45495]: audit 2022-04-22T20:46:53.961270+0000 mgr.y (mgr.34107) 6868 : audit [DBG] from='client.42522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[45495]: cluster 2022-04-22T20:46:54.140630+0000 mgr.y (mgr.34107) 6869 : cluster [DBG] pgmap v7966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[45495]: audit 2022-04-22T20:46:54.394127+0000 mon.a (mon.0) 10347 : audit [DBG] from='client.? 172.21.15.115:0/1806996587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[45495]: audit 2022-04-22T20:46:54.687388+0000 mon.a (mon.0) 10348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[46934]: audit 2022-04-22T20:46:53.614578+0000 mgr.y (mgr.34107) 6867 : audit [DBG] from='client.42516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[46934]: audit 2022-04-22T20:46:53.961270+0000 mgr.y (mgr.34107) 6868 : audit [DBG] from='client.42522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[46934]: cluster 2022-04-22T20:46:54.140630+0000 mgr.y (mgr.34107) 6869 : cluster [DBG] pgmap v7966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[46934]: audit 2022-04-22T20:46:54.394127+0000 mon.a (mon.0) 10347 : audit [DBG] from='client.? 172.21.15.115:0/1806996587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:46:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:55 smithi115 bash[46934]: audit 2022-04-22T20:46:54.687388+0000 mon.a (mon.0) 10348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:46:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:57 smithi121 bash[36902]: cluster 2022-04-22T20:46:56.141777+0000 mgr.y (mgr.34107) 6870 : cluster [DBG] pgmap v7967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:57 smithi115 bash[45495]: cluster 2022-04-22T20:46:56.141777+0000 mgr.y (mgr.34107) 6870 : cluster [DBG] pgmap v7967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:57 smithi115 bash[46934]: cluster 2022-04-22T20:46:56.141777+0000 mgr.y (mgr.34107) 6870 : cluster [DBG] pgmap v7967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:59 smithi121 bash[36902]: cluster 2022-04-22T20:46:58.142910+0000 mgr.y (mgr.34107) 6871 : cluster [DBG] pgmap v7968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:59 smithi121 bash[36902]: audit 2022-04-22T20:46:58.227590+0000 mon.a (mon.0) 10349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:46:59 smithi121 bash[36902]: audit 2022-04-22T20:46:58.265039+0000 mon.a (mon.0) 10350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[46934]: cluster 2022-04-22T20:46:58.142910+0000 mgr.y (mgr.34107) 6871 : cluster [DBG] pgmap v7968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[46934]: audit 2022-04-22T20:46:58.227590+0000 mon.a (mon.0) 10349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[46934]: audit 2022-04-22T20:46:58.265039+0000 mon.a (mon.0) 10350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:46:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[45495]: cluster 2022-04-22T20:46:58.142910+0000 mgr.y (mgr.34107) 6871 : cluster [DBG] pgmap v7968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:46:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[45495]: audit 2022-04-22T20:46:58.227590+0000 mon.a (mon.0) 10349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:46:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:46:59 smithi115 bash[45495]: audit 2022-04-22T20:46:58.265039+0000 mon.a (mon.0) 10350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:00 smithi121 bash[36902]: audit 2022-04-22T20:46:59.690442+0000 mon.a (mon.0) 10351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:00 smithi121 bash[36902]: cluster 2022-04-22T20:47:00.143913+0000 mgr.y (mgr.34107) 6872 : cluster [DBG] pgmap v7969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:00 smithi115 bash[45495]: audit 2022-04-22T20:46:59.690442+0000 mon.a (mon.0) 10351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:00 smithi115 bash[45495]: cluster 2022-04-22T20:47:00.143913+0000 mgr.y (mgr.34107) 6872 : cluster [DBG] pgmap v7969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:00 smithi115 bash[46934]: audit 2022-04-22T20:46:59.690442+0000 mon.a (mon.0) 10351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:00 smithi115 bash[46934]: cluster 2022-04-22T20:47:00.143913+0000 mgr.y (mgr.34107) 6872 : cluster [DBG] pgmap v7969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:02.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:03 smithi121 bash[36902]: cluster 2022-04-22T20:47:02.145038+0000 mgr.y (mgr.34107) 6873 : cluster [DBG] pgmap v7970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:03.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:03 smithi115 bash[46934]: cluster 2022-04-22T20:47:02.145038+0000 mgr.y (mgr.34107) 6873 : cluster [DBG] pgmap v7970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:03 smithi115 bash[45495]: cluster 2022-04-22T20:47:02.145038+0000 mgr.y (mgr.34107) 6873 : cluster [DBG] pgmap v7970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:04 smithi121 bash[36902]: audit 2022-04-22T20:47:03.230306+0000 mon.a (mon.0) 10352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:04 smithi121 bash[36902]: audit 2022-04-22T20:47:03.268186+0000 mon.a (mon.0) 10353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:04.268Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:04.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:04 smithi115 bash[45495]: audit 2022-04-22T20:47:03.230306+0000 mon.a (mon.0) 10352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:04.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:04 smithi115 bash[45495]: audit 2022-04-22T20:47:03.268186+0000 mon.a (mon.0) 10353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:04.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:04 smithi115 bash[46934]: audit 2022-04-22T20:47:03.230306+0000 mon.a (mon.0) 10352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:04.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:04 smithi115 bash[46934]: audit 2022-04-22T20:47:03.268186+0000 mon.a (mon.0) 10353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:05 smithi121 bash[36902]: cluster 2022-04-22T20:47:04.146142+0000 mgr.y (mgr.34107) 6874 : cluster [DBG] pgmap v7971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:05 smithi121 bash[36902]: audit 2022-04-22T20:47:04.693628+0000 mon.a (mon.0) 10354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:05 smithi115 bash[45495]: cluster 2022-04-22T20:47:04.146142+0000 mgr.y (mgr.34107) 6874 : cluster [DBG] pgmap v7971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:05 smithi115 bash[45495]: audit 2022-04-22T20:47:04.693628+0000 mon.a (mon.0) 10354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:05.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:05 smithi115 bash[46934]: cluster 2022-04-22T20:47:04.146142+0000 mgr.y (mgr.34107) 6874 : cluster [DBG] pgmap v7971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:05 smithi115 bash[46934]: audit 2022-04-22T20:47:04.693628+0000 mon.a (mon.0) 10354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:07 smithi121 bash[36902]: cluster 2022-04-22T20:47:06.147177+0000 mgr.y (mgr.34107) 6875 : cluster [DBG] pgmap v7972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:07 smithi115 bash[45495]: cluster 2022-04-22T20:47:06.147177+0000 mgr.y (mgr.34107) 6875 : cluster [DBG] pgmap v7972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:07.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:07 smithi115 bash[46934]: cluster 2022-04-22T20:47:06.147177+0000 mgr.y (mgr.34107) 6875 : cluster [DBG] pgmap v7972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:09 smithi121 bash[36902]: cluster 2022-04-22T20:47:08.148363+0000 mgr.y (mgr.34107) 6876 : cluster [DBG] pgmap v7973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:09 smithi121 bash[36902]: audit 2022-04-22T20:47:08.233464+0000 mon.a (mon.0) 10355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:09 smithi121 bash[36902]: audit 2022-04-22T20:47:08.271848+0000 mon.a (mon.0) 10356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[45495]: cluster 2022-04-22T20:47:08.148363+0000 mgr.y (mgr.34107) 6876 : cluster [DBG] pgmap v7973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[45495]: audit 2022-04-22T20:47:08.233464+0000 mon.a (mon.0) 10355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:09.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[45495]: audit 2022-04-22T20:47:08.271848+0000 mon.a (mon.0) 10356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[46934]: cluster 2022-04-22T20:47:08.148363+0000 mgr.y (mgr.34107) 6876 : cluster [DBG] pgmap v7973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[46934]: audit 2022-04-22T20:47:08.233464+0000 mon.a (mon.0) 10355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:09.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:09 smithi115 bash[46934]: audit 2022-04-22T20:47:08.271848+0000 mon.a (mon.0) 10356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:10 smithi121 bash[36902]: audit 2022-04-22T20:47:09.696892+0000 mon.a (mon.0) 10357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:10 smithi121 bash[36902]: cluster 2022-04-22T20:47:10.149258+0000 mgr.y (mgr.34107) 6877 : cluster [DBG] pgmap v7974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:10 smithi115 bash[45495]: audit 2022-04-22T20:47:09.696892+0000 mon.a (mon.0) 10357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:10 smithi115 bash[45495]: cluster 2022-04-22T20:47:10.149258+0000 mgr.y (mgr.34107) 6877 : cluster [DBG] pgmap v7974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:10 smithi115 bash[46934]: audit 2022-04-22T20:47:09.696892+0000 mon.a (mon.0) 10357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:10 smithi115 bash[46934]: cluster 2022-04-22T20:47:10.149258+0000 mgr.y (mgr.34107) 6877 : cluster [DBG] pgmap v7974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:12.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:13 smithi121 bash[36902]: cluster 2022-04-22T20:47:12.150428+0000 mgr.y (mgr.34107) 6878 : cluster [DBG] pgmap v7975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:13 smithi115 bash[45495]: cluster 2022-04-22T20:47:12.150428+0000 mgr.y (mgr.34107) 6878 : cluster [DBG] pgmap v7975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:13 smithi115 bash[46934]: cluster 2022-04-22T20:47:12.150428+0000 mgr.y (mgr.34107) 6878 : cluster [DBG] pgmap v7975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:14 smithi121 bash[36902]: audit 2022-04-22T20:47:13.236178+0000 mon.a (mon.0) 10358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:14 smithi121 bash[36902]: audit 2022-04-22T20:47:13.275109+0000 mon.a (mon.0) 10359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:14.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:14.269Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:14 smithi115 bash[45495]: audit 2022-04-22T20:47:13.236178+0000 mon.a (mon.0) 10358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:14.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:14 smithi115 bash[45495]: audit 2022-04-22T20:47:13.275109+0000 mon.a (mon.0) 10359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:14 smithi115 bash[46934]: audit 2022-04-22T20:47:13.236178+0000 mon.a (mon.0) 10358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:14 smithi115 bash[46934]: audit 2022-04-22T20:47:13.275109+0000 mon.a (mon.0) 10359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:15 smithi121 bash[36902]: cluster 2022-04-22T20:47:14.151553+0000 mgr.y (mgr.34107) 6879 : cluster [DBG] pgmap v7976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:15 smithi121 bash[36902]: audit 2022-04-22T20:47:14.699876+0000 mon.a (mon.0) 10360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:15 smithi115 bash[45495]: cluster 2022-04-22T20:47:14.151553+0000 mgr.y (mgr.34107) 6879 : cluster [DBG] pgmap v7976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:15 smithi115 bash[45495]: audit 2022-04-22T20:47:14.699876+0000 mon.a (mon.0) 10360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:15 smithi115 bash[46934]: cluster 2022-04-22T20:47:14.151553+0000 mgr.y (mgr.34107) 6879 : cluster [DBG] pgmap v7976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:15 smithi115 bash[46934]: audit 2022-04-22T20:47:14.699876+0000 mon.a (mon.0) 10360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:16 smithi121 bash[36902]: audit 2022-04-22T20:47:15.581758+0000 mon.a (mon.0) 10361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:47:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:16 smithi121 bash[36902]: audit 2022-04-22T20:47:15.583882+0000 mon.a (mon.0) 10362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:47:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:16 smithi115 bash[45495]: audit 2022-04-22T20:47:15.581758+0000 mon.a (mon.0) 10361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:47:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:16 smithi115 bash[45495]: audit 2022-04-22T20:47:15.583882+0000 mon.a (mon.0) 10362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:47:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:16 smithi115 bash[46934]: audit 2022-04-22T20:47:15.581758+0000 mon.a (mon.0) 10361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:47:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:16 smithi115 bash[46934]: audit 2022-04-22T20:47:15.583882+0000 mon.a (mon.0) 10362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:47:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:17 smithi121 bash[36902]: cluster 2022-04-22T20:47:16.152535+0000 mgr.y (mgr.34107) 6880 : cluster [DBG] pgmap v7977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:17 smithi115 bash[45495]: cluster 2022-04-22T20:47:16.152535+0000 mgr.y (mgr.34107) 6880 : cluster [DBG] pgmap v7977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:17 smithi115 bash[46934]: cluster 2022-04-22T20:47:16.152535+0000 mgr.y (mgr.34107) 6880 : cluster [DBG] pgmap v7977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:18.247 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[45495]: audit 2022-04-22T20:47:17.947593+0000 mon.a (mon.0) 10363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:47:18.248 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[45495]: audit 2022-04-22T20:47:17.950522+0000 mon.a (mon.0) 10364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:47:18.248 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[45495]: audit 2022-04-22T20:47:17.952255+0000 mon.a (mon.0) 10365 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:47:18.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[46934]: audit 2022-04-22T20:47:17.947593+0000 mon.a (mon.0) 10363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:47:18.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[46934]: audit 2022-04-22T20:47:17.950522+0000 mon.a (mon.0) 10364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:47:18.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:18 smithi115 bash[46934]: audit 2022-04-22T20:47:17.952255+0000 mon.a (mon.0) 10365 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:47:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:18 smithi121 bash[36902]: audit 2022-04-22T20:47:17.947593+0000 mon.a (mon.0) 10363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:47:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:18 smithi121 bash[36902]: audit 2022-04-22T20:47:17.950522+0000 mon.a (mon.0) 10364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:47:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:18 smithi121 bash[36902]: audit 2022-04-22T20:47:17.952255+0000 mon.a (mon.0) 10365 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:47:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:19 smithi121 bash[36902]: cluster 2022-04-22T20:47:18.153548+0000 mgr.y (mgr.34107) 6881 : cluster [DBG] pgmap v7978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:19 smithi121 bash[36902]: audit 2022-04-22T20:47:18.238651+0000 mon.a (mon.0) 10366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:19 smithi121 bash[36902]: audit 2022-04-22T20:47:18.278808+0000 mon.a (mon.0) 10367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:19 smithi121 bash[36902]: audit 2022-04-22T20:47:18.321277+0000 mon.a (mon.0) 10368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:47:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[45495]: cluster 2022-04-22T20:47:18.153548+0000 mgr.y (mgr.34107) 6881 : cluster [DBG] pgmap v7978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[45495]: audit 2022-04-22T20:47:18.238651+0000 mon.a (mon.0) 10366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[45495]: audit 2022-04-22T20:47:18.278808+0000 mon.a (mon.0) 10367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[45495]: audit 2022-04-22T20:47:18.321277+0000 mon.a (mon.0) 10368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[46934]: cluster 2022-04-22T20:47:18.153548+0000 mgr.y (mgr.34107) 6881 : cluster [DBG] pgmap v7978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[46934]: audit 2022-04-22T20:47:18.238651+0000 mon.a (mon.0) 10366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[46934]: audit 2022-04-22T20:47:18.278808+0000 mon.a (mon.0) 10367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:19 smithi115 bash[46934]: audit 2022-04-22T20:47:18.321277+0000 mon.a (mon.0) 10368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:47:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:20 smithi121 bash[36902]: audit 2022-04-22T20:47:19.702763+0000 mon.a (mon.0) 10369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:20 smithi121 bash[36902]: cluster 2022-04-22T20:47:20.154494+0000 mgr.y (mgr.34107) 6882 : cluster [DBG] pgmap v7979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:20 smithi115 bash[45495]: audit 2022-04-22T20:47:19.702763+0000 mon.a (mon.0) 10369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:20 smithi115 bash[45495]: cluster 2022-04-22T20:47:20.154494+0000 mgr.y (mgr.34107) 6882 : cluster [DBG] pgmap v7979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:20 smithi115 bash[46934]: audit 2022-04-22T20:47:19.702763+0000 mon.a (mon.0) 10369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:20 smithi115 bash[46934]: cluster 2022-04-22T20:47:20.154494+0000 mgr.y (mgr.34107) 6882 : cluster [DBG] pgmap v7979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:22.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:23 smithi121 bash[36902]: cluster 2022-04-22T20:47:22.155727+0000 mgr.y (mgr.34107) 6883 : cluster [DBG] pgmap v7980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:23 smithi115 bash[45495]: cluster 2022-04-22T20:47:22.155727+0000 mgr.y (mgr.34107) 6883 : cluster [DBG] pgmap v7980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:23.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:23 smithi115 bash[46934]: cluster 2022-04-22T20:47:22.155727+0000 mgr.y (mgr.34107) 6883 : cluster [DBG] pgmap v7980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:24 smithi121 bash[36902]: audit 2022-04-22T20:47:23.241260+0000 mon.a (mon.0) 10370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:24 smithi121 bash[36902]: audit 2022-04-22T20:47:23.282100+0000 mon.a (mon.0) 10371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:24.269Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:24 smithi115 bash[45495]: audit 2022-04-22T20:47:23.241260+0000 mon.a (mon.0) 10370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:24 smithi115 bash[45495]: audit 2022-04-22T20:47:23.282100+0000 mon.a (mon.0) 10371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:24 smithi115 bash[46934]: audit 2022-04-22T20:47:23.241260+0000 mon.a (mon.0) 10370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:24 smithi115 bash[46934]: audit 2022-04-22T20:47:23.282100+0000 mon.a (mon.0) 10371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:24.778 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:47:25.119 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:47:25.120 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:47:25.121 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:47:25.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:47:25.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:47:25.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:47:25.122 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:47:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:25 smithi121 bash[36902]: cluster 2022-04-22T20:47:24.156741+0000 mgr.y (mgr.34107) 6884 : cluster [DBG] pgmap v7981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:25.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:25 smithi121 bash[36902]: audit 2022-04-22T20:47:24.705795+0000 mon.a (mon.0) 10372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:25.519 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:47:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:47:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:47:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:47:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:47:25.521 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:47:25.521 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:47:25.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:25 smithi115 bash[45495]: cluster 2022-04-22T20:47:24.156741+0000 mgr.y (mgr.34107) 6884 : cluster [DBG] pgmap v7981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:25.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:25 smithi115 bash[45495]: audit 2022-04-22T20:47:24.705795+0000 mon.a (mon.0) 10372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:25 smithi115 bash[46934]: cluster 2022-04-22T20:47:24.156741+0000 mgr.y (mgr.34107) 6884 : cluster [DBG] pgmap v7981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:25.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:25 smithi115 bash[46934]: audit 2022-04-22T20:47:24.705795+0000 mon.a (mon.0) 10372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:26 smithi121 bash[36902]: audit 2022-04-22T20:47:24.754224+0000 mgr.y (mgr.34107) 6885 : audit [DBG] from='client.52465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:26 smithi121 bash[36902]: audit 2022-04-22T20:47:25.099334+0000 mgr.y (mgr.34107) 6886 : audit [DBG] from='client.52471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:26 smithi121 bash[36902]: audit 2022-04-22T20:47:25.513233+0000 mon.c (mon.1) 261 : audit [DBG] from='client.? 172.21.15.115:0/1789609836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[45495]: audit 2022-04-22T20:47:24.754224+0000 mgr.y (mgr.34107) 6885 : audit [DBG] from='client.52465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[45495]: audit 2022-04-22T20:47:25.099334+0000 mgr.y (mgr.34107) 6886 : audit [DBG] from='client.52471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[45495]: audit 2022-04-22T20:47:25.513233+0000 mon.c (mon.1) 261 : audit [DBG] from='client.? 172.21.15.115:0/1789609836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:26.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[46934]: audit 2022-04-22T20:47:24.754224+0000 mgr.y (mgr.34107) 6885 : audit [DBG] from='client.52465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[46934]: audit 2022-04-22T20:47:25.099334+0000 mgr.y (mgr.34107) 6886 : audit [DBG] from='client.52471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:26.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:26 smithi115 bash[46934]: audit 2022-04-22T20:47:25.513233+0000 mon.c (mon.1) 261 : audit [DBG] from='client.? 172.21.15.115:0/1789609836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:27 smithi121 bash[36902]: cluster 2022-04-22T20:47:26.157845+0000 mgr.y (mgr.34107) 6887 : cluster [DBG] pgmap v7982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:27.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:27 smithi115 bash[45495]: cluster 2022-04-22T20:47:26.157845+0000 mgr.y (mgr.34107) 6887 : cluster [DBG] pgmap v7982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:27 smithi115 bash[46934]: cluster 2022-04-22T20:47:26.157845+0000 mgr.y (mgr.34107) 6887 : cluster [DBG] pgmap v7982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:29 smithi121 bash[36902]: cluster 2022-04-22T20:47:28.159041+0000 mgr.y (mgr.34107) 6888 : cluster [DBG] pgmap v7983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:29 smithi121 bash[36902]: audit 2022-04-22T20:47:28.244037+0000 mon.a (mon.0) 10373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:29 smithi121 bash[36902]: audit 2022-04-22T20:47:28.285882+0000 mon.a (mon.0) 10374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[45495]: cluster 2022-04-22T20:47:28.159041+0000 mgr.y (mgr.34107) 6888 : cluster [DBG] pgmap v7983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[45495]: audit 2022-04-22T20:47:28.244037+0000 mon.a (mon.0) 10373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:29.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[45495]: audit 2022-04-22T20:47:28.285882+0000 mon.a (mon.0) 10374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[46934]: cluster 2022-04-22T20:47:28.159041+0000 mgr.y (mgr.34107) 6888 : cluster [DBG] pgmap v7983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[46934]: audit 2022-04-22T20:47:28.244037+0000 mon.a (mon.0) 10373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:29.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:29 smithi115 bash[46934]: audit 2022-04-22T20:47:28.285882+0000 mon.a (mon.0) 10374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:30 smithi121 bash[36902]: audit 2022-04-22T20:47:29.708669+0000 mon.a (mon.0) 10375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:30 smithi121 bash[36902]: cluster 2022-04-22T20:47:30.160170+0000 mgr.y (mgr.34107) 6889 : cluster [DBG] pgmap v7984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:30 smithi115 bash[45495]: audit 2022-04-22T20:47:29.708669+0000 mon.a (mon.0) 10375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:30 smithi115 bash[45495]: cluster 2022-04-22T20:47:30.160170+0000 mgr.y (mgr.34107) 6889 : cluster [DBG] pgmap v7984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:30 smithi115 bash[46934]: audit 2022-04-22T20:47:29.708669+0000 mon.a (mon.0) 10375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:30 smithi115 bash[46934]: cluster 2022-04-22T20:47:30.160170+0000 mgr.y (mgr.34107) 6889 : cluster [DBG] pgmap v7984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:32.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:33 smithi121 bash[36902]: cluster 2022-04-22T20:47:32.161273+0000 mgr.y (mgr.34107) 6890 : cluster [DBG] pgmap v7985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:33 smithi115 bash[45495]: cluster 2022-04-22T20:47:32.161273+0000 mgr.y (mgr.34107) 6890 : cluster [DBG] pgmap v7985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:33 smithi115 bash[46934]: cluster 2022-04-22T20:47:32.161273+0000 mgr.y (mgr.34107) 6890 : cluster [DBG] pgmap v7985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:34 smithi121 bash[36902]: audit 2022-04-22T20:47:33.246874+0000 mon.a (mon.0) 10376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:34 smithi121 bash[36902]: audit 2022-04-22T20:47:33.289535+0000 mon.a (mon.0) 10377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:34.270Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:34 smithi115 bash[45495]: audit 2022-04-22T20:47:33.246874+0000 mon.a (mon.0) 10376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:34 smithi115 bash[45495]: audit 2022-04-22T20:47:33.289535+0000 mon.a (mon.0) 10377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:34 smithi115 bash[46934]: audit 2022-04-22T20:47:33.246874+0000 mon.a (mon.0) 10376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:34 smithi115 bash[46934]: audit 2022-04-22T20:47:33.289535+0000 mon.a (mon.0) 10377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:35 smithi121 bash[36902]: cluster 2022-04-22T20:47:34.162373+0000 mgr.y (mgr.34107) 6891 : cluster [DBG] pgmap v7986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:35 smithi121 bash[36902]: audit 2022-04-22T20:47:34.711811+0000 mon.a (mon.0) 10378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:35 smithi115 bash[45495]: cluster 2022-04-22T20:47:34.162373+0000 mgr.y (mgr.34107) 6891 : cluster [DBG] pgmap v7986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:35 smithi115 bash[45495]: audit 2022-04-22T20:47:34.711811+0000 mon.a (mon.0) 10378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:35.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:35 smithi115 bash[46934]: cluster 2022-04-22T20:47:34.162373+0000 mgr.y (mgr.34107) 6891 : cluster [DBG] pgmap v7986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:35 smithi115 bash[46934]: audit 2022-04-22T20:47:34.711811+0000 mon.a (mon.0) 10378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:37 smithi121 bash[36902]: cluster 2022-04-22T20:47:36.163444+0000 mgr.y (mgr.34107) 6892 : cluster [DBG] pgmap v7987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:37 smithi115 bash[45495]: cluster 2022-04-22T20:47:36.163444+0000 mgr.y (mgr.34107) 6892 : cluster [DBG] pgmap v7987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:37.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:37 smithi115 bash[46934]: cluster 2022-04-22T20:47:36.163444+0000 mgr.y (mgr.34107) 6892 : cluster [DBG] pgmap v7987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:39 smithi121 bash[36902]: cluster 2022-04-22T20:47:38.164465+0000 mgr.y (mgr.34107) 6893 : cluster [DBG] pgmap v7988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:39 smithi121 bash[36902]: audit 2022-04-22T20:47:38.249957+0000 mon.a (mon.0) 10379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:39 smithi121 bash[36902]: audit 2022-04-22T20:47:38.293200+0000 mon.a (mon.0) 10380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:39.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[45495]: cluster 2022-04-22T20:47:38.164465+0000 mgr.y (mgr.34107) 6893 : cluster [DBG] pgmap v7988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:39.615 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[45495]: audit 2022-04-22T20:47:38.249957+0000 mon.a (mon.0) 10379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:39.615 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[45495]: audit 2022-04-22T20:47:38.293200+0000 mon.a (mon.0) 10380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[46934]: cluster 2022-04-22T20:47:38.164465+0000 mgr.y (mgr.34107) 6893 : cluster [DBG] pgmap v7988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[46934]: audit 2022-04-22T20:47:38.249957+0000 mon.a (mon.0) 10379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:39 smithi115 bash[46934]: audit 2022-04-22T20:47:38.293200+0000 mon.a (mon.0) 10380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:40 smithi121 bash[36902]: audit 2022-04-22T20:47:39.714816+0000 mon.a (mon.0) 10381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:40 smithi121 bash[36902]: cluster 2022-04-22T20:47:40.165490+0000 mgr.y (mgr.34107) 6894 : cluster [DBG] pgmap v7989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:40 smithi115 bash[45495]: audit 2022-04-22T20:47:39.714816+0000 mon.a (mon.0) 10381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:40 smithi115 bash[45495]: cluster 2022-04-22T20:47:40.165490+0000 mgr.y (mgr.34107) 6894 : cluster [DBG] pgmap v7989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:40 smithi115 bash[46934]: audit 2022-04-22T20:47:39.714816+0000 mon.a (mon.0) 10381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:40 smithi115 bash[46934]: cluster 2022-04-22T20:47:40.165490+0000 mgr.y (mgr.34107) 6894 : cluster [DBG] pgmap v7989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:42.390Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:43 smithi121 bash[36902]: cluster 2022-04-22T20:47:42.166490+0000 mgr.y (mgr.34107) 6895 : cluster [DBG] pgmap v7990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:43 smithi115 bash[45495]: cluster 2022-04-22T20:47:42.166490+0000 mgr.y (mgr.34107) 6895 : cluster [DBG] pgmap v7990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:43 smithi115 bash[46934]: cluster 2022-04-22T20:47:42.166490+0000 mgr.y (mgr.34107) 6895 : cluster [DBG] pgmap v7990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:44 smithi121 bash[36902]: audit 2022-04-22T20:47:43.252776+0000 mon.a (mon.0) 10382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:44 smithi121 bash[36902]: audit 2022-04-22T20:47:43.296690+0000 mon.a (mon.0) 10383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:44 smithi115 bash[45495]: audit 2022-04-22T20:47:43.252776+0000 mon.a (mon.0) 10382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:44 smithi115 bash[45495]: audit 2022-04-22T20:47:43.296690+0000 mon.a (mon.0) 10383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:44.272Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:44 smithi115 bash[46934]: audit 2022-04-22T20:47:43.252776+0000 mon.a (mon.0) 10382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:44 smithi115 bash[46934]: audit 2022-04-22T20:47:43.296690+0000 mon.a (mon.0) 10383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:45 smithi121 bash[36902]: cluster 2022-04-22T20:47:44.167594+0000 mgr.y (mgr.34107) 6896 : cluster [DBG] pgmap v7991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:45 smithi121 bash[36902]: audit 2022-04-22T20:47:44.718042+0000 mon.a (mon.0) 10384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:45 smithi115 bash[45495]: cluster 2022-04-22T20:47:44.167594+0000 mgr.y (mgr.34107) 6896 : cluster [DBG] pgmap v7991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:45.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:45 smithi115 bash[45495]: audit 2022-04-22T20:47:44.718042+0000 mon.a (mon.0) 10384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:45 smithi115 bash[46934]: cluster 2022-04-22T20:47:44.167594+0000 mgr.y (mgr.34107) 6896 : cluster [DBG] pgmap v7991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:45.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:45 smithi115 bash[46934]: audit 2022-04-22T20:47:44.718042+0000 mon.a (mon.0) 10384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:47 smithi121 bash[36902]: cluster 2022-04-22T20:47:46.168740+0000 mgr.y (mgr.34107) 6897 : cluster [DBG] pgmap v7992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:47 smithi115 bash[45495]: cluster 2022-04-22T20:47:46.168740+0000 mgr.y (mgr.34107) 6897 : cluster [DBG] pgmap v7992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:47 smithi115 bash[46934]: cluster 2022-04-22T20:47:46.168740+0000 mgr.y (mgr.34107) 6897 : cluster [DBG] pgmap v7992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:49 smithi121 bash[36902]: cluster 2022-04-22T20:47:48.169840+0000 mgr.y (mgr.34107) 6898 : cluster [DBG] pgmap v7993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:49 smithi121 bash[36902]: audit 2022-04-22T20:47:48.255464+0000 mon.a (mon.0) 10385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:49 smithi121 bash[36902]: audit 2022-04-22T20:47:48.300157+0000 mon.a (mon.0) 10386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[45495]: cluster 2022-04-22T20:47:48.169840+0000 mgr.y (mgr.34107) 6898 : cluster [DBG] pgmap v7993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[45495]: audit 2022-04-22T20:47:48.255464+0000 mon.a (mon.0) 10385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:49.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[45495]: audit 2022-04-22T20:47:48.300157+0000 mon.a (mon.0) 10386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[46934]: cluster 2022-04-22T20:47:48.169840+0000 mgr.y (mgr.34107) 6898 : cluster [DBG] pgmap v7993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[46934]: audit 2022-04-22T20:47:48.255464+0000 mon.a (mon.0) 10385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:49.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:49 smithi115 bash[46934]: audit 2022-04-22T20:47:48.300157+0000 mon.a (mon.0) 10386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:50 smithi121 bash[36902]: audit 2022-04-22T20:47:49.720817+0000 mon.a (mon.0) 10387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:50 smithi121 bash[36902]: cluster 2022-04-22T20:47:50.170759+0000 mgr.y (mgr.34107) 6899 : cluster [DBG] pgmap v7994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:50 smithi115 bash[45495]: audit 2022-04-22T20:47:49.720817+0000 mon.a (mon.0) 10387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:50 smithi115 bash[45495]: cluster 2022-04-22T20:47:50.170759+0000 mgr.y (mgr.34107) 6899 : cluster [DBG] pgmap v7994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:50 smithi115 bash[46934]: audit 2022-04-22T20:47:49.720817+0000 mon.a (mon.0) 10387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:50 smithi115 bash[46934]: cluster 2022-04-22T20:47:50.170759+0000 mgr.y (mgr.34107) 6899 : cluster [DBG] pgmap v7994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:52.390Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:53 smithi121 bash[36902]: cluster 2022-04-22T20:47:52.171853+0000 mgr.y (mgr.34107) 6900 : cluster [DBG] pgmap v7995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:53 smithi115 bash[45495]: cluster 2022-04-22T20:47:52.171853+0000 mgr.y (mgr.34107) 6900 : cluster [DBG] pgmap v7995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:53.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:53 smithi115 bash[46934]: cluster 2022-04-22T20:47:52.171853+0000 mgr.y (mgr.34107) 6900 : cluster [DBG] pgmap v7995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:54 smithi121 bash[36902]: audit 2022-04-22T20:47:53.258233+0000 mon.a (mon.0) 10388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:54 smithi121 bash[36902]: audit 2022-04-22T20:47:53.303903+0000 mon.a (mon.0) 10389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:54.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:47:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:47:54.270Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:47:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:54 smithi115 bash[45495]: audit 2022-04-22T20:47:53.258233+0000 mon.a (mon.0) 10388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:54 smithi115 bash[45495]: audit 2022-04-22T20:47:53.303903+0000 mon.a (mon.0) 10389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:54 smithi115 bash[46934]: audit 2022-04-22T20:47:53.258233+0000 mon.a (mon.0) 10388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:54 smithi115 bash[46934]: audit 2022-04-22T20:47:53.303903+0000 mon.a (mon.0) 10389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:55 smithi121 bash[36902]: cluster 2022-04-22T20:47:54.173086+0000 mgr.y (mgr.34107) 6901 : cluster [DBG] pgmap v7996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:55 smithi121 bash[36902]: audit 2022-04-22T20:47:54.723830+0000 mon.a (mon.0) 10390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:55 smithi115 bash[46934]: cluster 2022-04-22T20:47:54.173086+0000 mgr.y (mgr.34107) 6901 : cluster [DBG] pgmap v7996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:55.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:55 smithi115 bash[46934]: audit 2022-04-22T20:47:54.723830+0000 mon.a (mon.0) 10390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:55.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:55 smithi115 bash[45495]: cluster 2022-04-22T20:47:54.173086+0000 mgr.y (mgr.34107) 6901 : cluster [DBG] pgmap v7996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:55.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:55 smithi115 bash[45495]: audit 2022-04-22T20:47:54.723830+0000 mon.a (mon.0) 10390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:47:55.893 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:47:56.233 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:47:56.233 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:47:56.234 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:47:56.235 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:47:56.638 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:47:56.638 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:47:56.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:47:56.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:56.638 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:47:56.639 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:47:56.640 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:47:56.640 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:47:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:57 smithi121 bash[36902]: audit 2022-04-22T20:47:55.871903+0000 mgr.y (mgr.34107) 6902 : audit [DBG] from='client.42552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:57 smithi121 bash[36902]: cluster 2022-04-22T20:47:56.174035+0000 mgr.y (mgr.34107) 6903 : cluster [DBG] pgmap v7997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:57 smithi121 bash[36902]: audit 2022-04-22T20:47:56.206634+0000 mgr.y (mgr.34107) 6904 : audit [DBG] from='client.42558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:57 smithi121 bash[36902]: audit 2022-04-22T20:47:56.632142+0000 mon.a (mon.0) 10391 : audit [DBG] from='client.? 172.21.15.115:0/1105698971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[45495]: audit 2022-04-22T20:47:55.871903+0000 mgr.y (mgr.34107) 6902 : audit [DBG] from='client.42552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[45495]: cluster 2022-04-22T20:47:56.174035+0000 mgr.y (mgr.34107) 6903 : cluster [DBG] pgmap v7997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[45495]: audit 2022-04-22T20:47:56.206634+0000 mgr.y (mgr.34107) 6904 : audit [DBG] from='client.42558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[45495]: audit 2022-04-22T20:47:56.632142+0000 mon.a (mon.0) 10391 : audit [DBG] from='client.? 172.21.15.115:0/1105698971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[46934]: audit 2022-04-22T20:47:55.871903+0000 mgr.y (mgr.34107) 6902 : audit [DBG] from='client.42552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[46934]: cluster 2022-04-22T20:47:56.174035+0000 mgr.y (mgr.34107) 6903 : cluster [DBG] pgmap v7997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[46934]: audit 2022-04-22T20:47:56.206634+0000 mgr.y (mgr.34107) 6904 : audit [DBG] from='client.42558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:47:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:57 smithi115 bash[46934]: audit 2022-04-22T20:47:56.632142+0000 mon.a (mon.0) 10391 : audit [DBG] from='client.? 172.21.15.115:0/1105698971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:47:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:59 smithi121 bash[36902]: cluster 2022-04-22T20:47:58.175069+0000 mgr.y (mgr.34107) 6905 : cluster [DBG] pgmap v7998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:59 smithi121 bash[36902]: audit 2022-04-22T20:47:58.261050+0000 mon.a (mon.0) 10392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:47:59 smithi121 bash[36902]: audit 2022-04-22T20:47:58.307628+0000 mon.a (mon.0) 10393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[45495]: cluster 2022-04-22T20:47:58.175069+0000 mgr.y (mgr.34107) 6905 : cluster [DBG] pgmap v7998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[45495]: audit 2022-04-22T20:47:58.261050+0000 mon.a (mon.0) 10392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[45495]: audit 2022-04-22T20:47:58.307628+0000 mon.a (mon.0) 10393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:47:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[46934]: cluster 2022-04-22T20:47:58.175069+0000 mgr.y (mgr.34107) 6905 : cluster [DBG] pgmap v7998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:47:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[46934]: audit 2022-04-22T20:47:58.261050+0000 mon.a (mon.0) 10392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:47:59.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:47:59 smithi115 bash[46934]: audit 2022-04-22T20:47:58.307628+0000 mon.a (mon.0) 10393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:00 smithi121 bash[36902]: audit 2022-04-22T20:47:59.726818+0000 mon.a (mon.0) 10394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:00 smithi121 bash[36902]: cluster 2022-04-22T20:48:00.176178+0000 mgr.y (mgr.34107) 6906 : cluster [DBG] pgmap v7999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:00 smithi115 bash[45495]: audit 2022-04-22T20:47:59.726818+0000 mon.a (mon.0) 10394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:00 smithi115 bash[45495]: cluster 2022-04-22T20:48:00.176178+0000 mgr.y (mgr.34107) 6906 : cluster [DBG] pgmap v7999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:00 smithi115 bash[46934]: audit 2022-04-22T20:47:59.726818+0000 mon.a (mon.0) 10394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:00 smithi115 bash[46934]: cluster 2022-04-22T20:48:00.176178+0000 mgr.y (mgr.34107) 6906 : cluster [DBG] pgmap v7999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:02.391Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:03 smithi121 bash[36902]: cluster 2022-04-22T20:48:02.177194+0000 mgr.y (mgr.34107) 6907 : cluster [DBG] pgmap v8000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:03.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:03 smithi115 bash[45495]: cluster 2022-04-22T20:48:02.177194+0000 mgr.y (mgr.34107) 6907 : cluster [DBG] pgmap v8000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:03.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:03 smithi115 bash[46934]: cluster 2022-04-22T20:48:02.177194+0000 mgr.y (mgr.34107) 6907 : cluster [DBG] pgmap v8000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:04 smithi121 bash[36902]: audit 2022-04-22T20:48:03.263937+0000 mon.a (mon.0) 10395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:04 smithi121 bash[36902]: audit 2022-04-22T20:48:03.311118+0000 mon.a (mon.0) 10396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:04 smithi115 bash[45495]: audit 2022-04-22T20:48:03.263937+0000 mon.a (mon.0) 10395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:04 smithi115 bash[45495]: audit 2022-04-22T20:48:03.311118+0000 mon.a (mon.0) 10396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:04.271Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:04 smithi115 bash[46934]: audit 2022-04-22T20:48:03.263937+0000 mon.a (mon.0) 10395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:04 smithi115 bash[46934]: audit 2022-04-22T20:48:03.311118+0000 mon.a (mon.0) 10396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:05 smithi121 bash[36902]: cluster 2022-04-22T20:48:04.178315+0000 mgr.y (mgr.34107) 6908 : cluster [DBG] pgmap v8001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:05 smithi121 bash[36902]: audit 2022-04-22T20:48:04.729853+0000 mon.a (mon.0) 10397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:05 smithi115 bash[45495]: cluster 2022-04-22T20:48:04.178315+0000 mgr.y (mgr.34107) 6908 : cluster [DBG] pgmap v8001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:05.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:05 smithi115 bash[45495]: audit 2022-04-22T20:48:04.729853+0000 mon.a (mon.0) 10397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:05 smithi115 bash[46934]: cluster 2022-04-22T20:48:04.178315+0000 mgr.y (mgr.34107) 6908 : cluster [DBG] pgmap v8001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:05.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:05 smithi115 bash[46934]: audit 2022-04-22T20:48:04.729853+0000 mon.a (mon.0) 10397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:07 smithi121 bash[36902]: cluster 2022-04-22T20:48:06.179436+0000 mgr.y (mgr.34107) 6909 : cluster [DBG] pgmap v8002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:07.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:07 smithi115 bash[45495]: cluster 2022-04-22T20:48:06.179436+0000 mgr.y (mgr.34107) 6909 : cluster [DBG] pgmap v8002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:07.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:07 smithi115 bash[46934]: cluster 2022-04-22T20:48:06.179436+0000 mgr.y (mgr.34107) 6909 : cluster [DBG] pgmap v8002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:09 smithi121 bash[36902]: cluster 2022-04-22T20:48:08.180614+0000 mgr.y (mgr.34107) 6910 : cluster [DBG] pgmap v8003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:09 smithi121 bash[36902]: audit 2022-04-22T20:48:08.266843+0000 mon.a (mon.0) 10398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:09 smithi121 bash[36902]: audit 2022-04-22T20:48:08.314570+0000 mon.a (mon.0) 10399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[45495]: cluster 2022-04-22T20:48:08.180614+0000 mgr.y (mgr.34107) 6910 : cluster [DBG] pgmap v8003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[45495]: audit 2022-04-22T20:48:08.266843+0000 mon.a (mon.0) 10398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:09.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[45495]: audit 2022-04-22T20:48:08.314570+0000 mon.a (mon.0) 10399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:09.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[46934]: cluster 2022-04-22T20:48:08.180614+0000 mgr.y (mgr.34107) 6910 : cluster [DBG] pgmap v8003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:09.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[46934]: audit 2022-04-22T20:48:08.266843+0000 mon.a (mon.0) 10398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:09.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:09 smithi115 bash[46934]: audit 2022-04-22T20:48:08.314570+0000 mon.a (mon.0) 10399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:10 smithi121 bash[36902]: audit 2022-04-22T20:48:09.733003+0000 mon.a (mon.0) 10400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:10.773 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:10 smithi121 bash[36902]: cluster 2022-04-22T20:48:10.181683+0000 mgr.y (mgr.34107) 6911 : cluster [DBG] pgmap v8004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:10 smithi115 bash[45495]: audit 2022-04-22T20:48:09.733003+0000 mon.a (mon.0) 10400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:10 smithi115 bash[45495]: cluster 2022-04-22T20:48:10.181683+0000 mgr.y (mgr.34107) 6911 : cluster [DBG] pgmap v8004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:10 smithi115 bash[46934]: audit 2022-04-22T20:48:09.733003+0000 mon.a (mon.0) 10400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:10 smithi115 bash[46934]: cluster 2022-04-22T20:48:10.181683+0000 mgr.y (mgr.34107) 6911 : cluster [DBG] pgmap v8004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:12.394Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:13 smithi121 bash[36902]: cluster 2022-04-22T20:48:12.182812+0000 mgr.y (mgr.34107) 6912 : cluster [DBG] pgmap v8005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:13 smithi115 bash[45495]: cluster 2022-04-22T20:48:12.182812+0000 mgr.y (mgr.34107) 6912 : cluster [DBG] pgmap v8005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:13 smithi115 bash[46934]: cluster 2022-04-22T20:48:12.182812+0000 mgr.y (mgr.34107) 6912 : cluster [DBG] pgmap v8005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:14 smithi121 bash[36902]: audit 2022-04-22T20:48:13.269610+0000 mon.a (mon.0) 10401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:14 smithi121 bash[36902]: audit 2022-04-22T20:48:13.318201+0000 mon.a (mon.0) 10402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:14 smithi115 bash[45495]: audit 2022-04-22T20:48:13.269610+0000 mon.a (mon.0) 10401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:14 smithi115 bash[45495]: audit 2022-04-22T20:48:13.318201+0000 mon.a (mon.0) 10402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:14.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:14.272Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:14 smithi115 bash[46934]: audit 2022-04-22T20:48:13.269610+0000 mon.a (mon.0) 10401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:14 smithi115 bash[46934]: audit 2022-04-22T20:48:13.318201+0000 mon.a (mon.0) 10402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:15 smithi121 bash[36902]: cluster 2022-04-22T20:48:14.183872+0000 mgr.y (mgr.34107) 6913 : cluster [DBG] pgmap v8006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:15 smithi121 bash[36902]: audit 2022-04-22T20:48:14.736194+0000 mon.a (mon.0) 10403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:15 smithi115 bash[45495]: cluster 2022-04-22T20:48:14.183872+0000 mgr.y (mgr.34107) 6913 : cluster [DBG] pgmap v8006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:15.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:15 smithi115 bash[45495]: audit 2022-04-22T20:48:14.736194+0000 mon.a (mon.0) 10403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:15 smithi115 bash[46934]: cluster 2022-04-22T20:48:14.183872+0000 mgr.y (mgr.34107) 6913 : cluster [DBG] pgmap v8006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:15.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:15 smithi115 bash[46934]: audit 2022-04-22T20:48:14.736194+0000 mon.a (mon.0) 10403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:16 smithi115 bash[45495]: audit 2022-04-22T20:48:15.592436+0000 mon.a (mon.0) 10404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:48:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:16 smithi115 bash[45495]: audit 2022-04-22T20:48:15.598084+0000 mon.a (mon.0) 10405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:48:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:16 smithi115 bash[46934]: audit 2022-04-22T20:48:15.592436+0000 mon.a (mon.0) 10404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:48:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:16 smithi115 bash[46934]: audit 2022-04-22T20:48:15.598084+0000 mon.a (mon.0) 10405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:48:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:16 smithi121 bash[36902]: audit 2022-04-22T20:48:15.592436+0000 mon.a (mon.0) 10404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:48:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:16 smithi121 bash[36902]: audit 2022-04-22T20:48:15.598084+0000 mon.a (mon.0) 10405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:48:17.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:17 smithi115 bash[45495]: cluster 2022-04-22T20:48:16.184928+0000 mgr.y (mgr.34107) 6914 : cluster [DBG] pgmap v8007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:17.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:17 smithi115 bash[46934]: cluster 2022-04-22T20:48:16.184928+0000 mgr.y (mgr.34107) 6914 : cluster [DBG] pgmap v8007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:17 smithi121 bash[36902]: cluster 2022-04-22T20:48:16.184928+0000 mgr.y (mgr.34107) 6914 : cluster [DBG] pgmap v8007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: cluster 2022-04-22T20:48:18.186047+0000 mgr.y (mgr.34107) 6915 : cluster [DBG] pgmap v8008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.272408+0000 mon.a (mon.0) 10406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.321740+0000 mon.a (mon.0) 10407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.329832+0000 mon.a (mon.0) 10408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.332516+0000 mon.a (mon.0) 10409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.334059+0000 mon.a (mon.0) 10410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[45495]: audit 2022-04-22T20:48:18.702119+0000 mon.a (mon.0) 10411 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:48:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: cluster 2022-04-22T20:48:18.186047+0000 mgr.y (mgr.34107) 6915 : cluster [DBG] pgmap v8008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.272408+0000 mon.a (mon.0) 10406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.321740+0000 mon.a (mon.0) 10407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.329832+0000 mon.a (mon.0) 10408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.332516+0000 mon.a (mon.0) 10409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.334059+0000 mon.a (mon.0) 10410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:48:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:19 smithi115 bash[46934]: audit 2022-04-22T20:48:18.702119+0000 mon.a (mon.0) 10411 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:48:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: cluster 2022-04-22T20:48:18.186047+0000 mgr.y (mgr.34107) 6915 : cluster [DBG] pgmap v8008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.272408+0000 mon.a (mon.0) 10406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.321740+0000 mon.a (mon.0) 10407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.329832+0000 mon.a (mon.0) 10408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.332516+0000 mon.a (mon.0) 10409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.334059+0000 mon.a (mon.0) 10410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:48:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:19 smithi121 bash[36902]: audit 2022-04-22T20:48:18.702119+0000 mon.a (mon.0) 10411 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:48:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:20 smithi121 bash[36902]: audit 2022-04-22T20:48:19.739370+0000 mon.a (mon.0) 10412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:20 smithi121 bash[36902]: cluster 2022-04-22T20:48:20.186996+0000 mgr.y (mgr.34107) 6916 : cluster [DBG] pgmap v8009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:20 smithi115 bash[45495]: audit 2022-04-22T20:48:19.739370+0000 mon.a (mon.0) 10412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:20 smithi115 bash[45495]: cluster 2022-04-22T20:48:20.186996+0000 mgr.y (mgr.34107) 6916 : cluster [DBG] pgmap v8009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:20 smithi115 bash[46934]: audit 2022-04-22T20:48:19.739370+0000 mon.a (mon.0) 10412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:20 smithi115 bash[46934]: cluster 2022-04-22T20:48:20.186996+0000 mgr.y (mgr.34107) 6916 : cluster [DBG] pgmap v8009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:22.797 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:22.393Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:23 smithi121 bash[36902]: cluster 2022-04-22T20:48:22.188111+0000 mgr.y (mgr.34107) 6917 : cluster [DBG] pgmap v8010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:23 smithi115 bash[45495]: cluster 2022-04-22T20:48:22.188111+0000 mgr.y (mgr.34107) 6917 : cluster [DBG] pgmap v8010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:23 smithi115 bash[46934]: cluster 2022-04-22T20:48:22.188111+0000 mgr.y (mgr.34107) 6917 : cluster [DBG] pgmap v8010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:24 smithi121 bash[36902]: audit 2022-04-22T20:48:23.275346+0000 mon.a (mon.0) 10413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:24 smithi121 bash[36902]: audit 2022-04-22T20:48:23.325288+0000 mon.a (mon.0) 10414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:24 smithi115 bash[45495]: audit 2022-04-22T20:48:23.275346+0000 mon.a (mon.0) 10413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:24 smithi115 bash[45495]: audit 2022-04-22T20:48:23.325288+0000 mon.a (mon.0) 10414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:24.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:24.271Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:24 smithi115 bash[46934]: audit 2022-04-22T20:48:23.275346+0000 mon.a (mon.0) 10413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:24 smithi115 bash[46934]: audit 2022-04-22T20:48:23.325288+0000 mon.a (mon.0) 10414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:25 smithi121 bash[36902]: cluster 2022-04-22T20:48:24.189159+0000 mgr.y (mgr.34107) 6918 : cluster [DBG] pgmap v8011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:25 smithi121 bash[36902]: audit 2022-04-22T20:48:24.742585+0000 mon.a (mon.0) 10415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:25.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:25 smithi115 bash[45495]: cluster 2022-04-22T20:48:24.189159+0000 mgr.y (mgr.34107) 6918 : cluster [DBG] pgmap v8011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:25 smithi115 bash[45495]: audit 2022-04-22T20:48:24.742585+0000 mon.a (mon.0) 10415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:25 smithi115 bash[46934]: cluster 2022-04-22T20:48:24.189159+0000 mgr.y (mgr.34107) 6918 : cluster [DBG] pgmap v8011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:25 smithi115 bash[46934]: audit 2022-04-22T20:48:24.742585+0000 mon.a (mon.0) 10415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:27.026 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:48:27.371 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:48:27.371 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:48:27.372 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:48:27.372 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:48:27.372 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:48:27.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:48:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:48:27.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:27 smithi115 bash[45495]: cluster 2022-04-22T20:48:26.190272+0000 mgr.y (mgr.34107) 6919 : cluster [DBG] pgmap v8012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:27.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:27 smithi115 bash[46934]: cluster 2022-04-22T20:48:26.190272+0000 mgr.y (mgr.34107) 6919 : cluster [DBG] pgmap v8012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:27 smithi121 bash[36902]: cluster 2022-04-22T20:48:26.190272+0000 mgr.y (mgr.34107) 6919 : cluster [DBG] pgmap v8012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:27.772 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:48:27.773 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:48:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:48:27.775 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:48:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[45495]: audit 2022-04-22T20:48:27.003041+0000 mgr.y (mgr.34107) 6920 : audit [DBG] from='client.42570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[45495]: audit 2022-04-22T20:48:27.346263+0000 mgr.y (mgr.34107) 6921 : audit [DBG] from='client.42576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[45495]: audit 2022-04-22T20:48:27.766580+0000 mon.a (mon.0) 10416 : audit [DBG] from='client.? 172.21.15.115:0/3140517133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:48:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[46934]: audit 2022-04-22T20:48:27.003041+0000 mgr.y (mgr.34107) 6920 : audit [DBG] from='client.42570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[46934]: audit 2022-04-22T20:48:27.346263+0000 mgr.y (mgr.34107) 6921 : audit [DBG] from='client.42576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:28 smithi115 bash[46934]: audit 2022-04-22T20:48:27.766580+0000 mon.a (mon.0) 10416 : audit [DBG] from='client.? 172.21.15.115:0/3140517133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:48:28.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:28 smithi121 bash[36902]: audit 2022-04-22T20:48:27.003041+0000 mgr.y (mgr.34107) 6920 : audit [DBG] from='client.42570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:28 smithi121 bash[36902]: audit 2022-04-22T20:48:27.346263+0000 mgr.y (mgr.34107) 6921 : audit [DBG] from='client.42576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:28.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:28 smithi121 bash[36902]: audit 2022-04-22T20:48:27.766580+0000 mon.a (mon.0) 10416 : audit [DBG] from='client.? 172.21.15.115:0/3140517133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:48:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[45495]: cluster 2022-04-22T20:48:28.191322+0000 mgr.y (mgr.34107) 6922 : cluster [DBG] pgmap v8013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[45495]: audit 2022-04-22T20:48:28.278041+0000 mon.a (mon.0) 10417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[45495]: audit 2022-04-22T20:48:28.328720+0000 mon.a (mon.0) 10418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[46934]: cluster 2022-04-22T20:48:28.191322+0000 mgr.y (mgr.34107) 6922 : cluster [DBG] pgmap v8013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[46934]: audit 2022-04-22T20:48:28.278041+0000 mon.a (mon.0) 10417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:29.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:29 smithi115 bash[46934]: audit 2022-04-22T20:48:28.328720+0000 mon.a (mon.0) 10418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:29.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:29 smithi121 bash[36902]: cluster 2022-04-22T20:48:28.191322+0000 mgr.y (mgr.34107) 6922 : cluster [DBG] pgmap v8013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:29.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:29 smithi121 bash[36902]: audit 2022-04-22T20:48:28.278041+0000 mon.a (mon.0) 10417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:29.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:29 smithi121 bash[36902]: audit 2022-04-22T20:48:28.328720+0000 mon.a (mon.0) 10418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:30 smithi121 bash[36902]: audit 2022-04-22T20:48:29.745512+0000 mon.a (mon.0) 10419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:30 smithi121 bash[36902]: cluster 2022-04-22T20:48:30.192364+0000 mgr.y (mgr.34107) 6923 : cluster [DBG] pgmap v8014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:30 smithi115 bash[45495]: audit 2022-04-22T20:48:29.745512+0000 mon.a (mon.0) 10419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:30 smithi115 bash[45495]: cluster 2022-04-22T20:48:30.192364+0000 mgr.y (mgr.34107) 6923 : cluster [DBG] pgmap v8014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:30 smithi115 bash[46934]: audit 2022-04-22T20:48:29.745512+0000 mon.a (mon.0) 10419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:30.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:30 smithi115 bash[46934]: cluster 2022-04-22T20:48:30.192364+0000 mgr.y (mgr.34107) 6923 : cluster [DBG] pgmap v8014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:32.395Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:33 smithi121 bash[36902]: cluster 2022-04-22T20:48:32.193413+0000 mgr.y (mgr.34107) 6924 : cluster [DBG] pgmap v8015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:33.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:33 smithi115 bash[45495]: cluster 2022-04-22T20:48:32.193413+0000 mgr.y (mgr.34107) 6924 : cluster [DBG] pgmap v8015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:33 smithi115 bash[46934]: cluster 2022-04-22T20:48:32.193413+0000 mgr.y (mgr.34107) 6924 : cluster [DBG] pgmap v8015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:34 smithi121 bash[36902]: audit 2022-04-22T20:48:33.280369+0000 mon.a (mon.0) 10420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:34 smithi121 bash[36902]: audit 2022-04-22T20:48:33.332414+0000 mon.a (mon.0) 10421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:34.270Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:34 smithi115 bash[45495]: audit 2022-04-22T20:48:33.280369+0000 mon.a (mon.0) 10420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:34 smithi115 bash[45495]: audit 2022-04-22T20:48:33.332414+0000 mon.a (mon.0) 10421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:34 smithi115 bash[46934]: audit 2022-04-22T20:48:33.280369+0000 mon.a (mon.0) 10420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:34 smithi115 bash[46934]: audit 2022-04-22T20:48:33.332414+0000 mon.a (mon.0) 10421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:35 smithi115 bash[45495]: cluster 2022-04-22T20:48:34.194598+0000 mgr.y (mgr.34107) 6925 : cluster [DBG] pgmap v8016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:35 smithi115 bash[45495]: audit 2022-04-22T20:48:34.748756+0000 mon.a (mon.0) 10422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:35 smithi115 bash[46934]: cluster 2022-04-22T20:48:34.194598+0000 mgr.y (mgr.34107) 6925 : cluster [DBG] pgmap v8016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:35 smithi115 bash[46934]: audit 2022-04-22T20:48:34.748756+0000 mon.a (mon.0) 10422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:35.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:35 smithi121 bash[36902]: cluster 2022-04-22T20:48:34.194598+0000 mgr.y (mgr.34107) 6925 : cluster [DBG] pgmap v8016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:35.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:35 smithi121 bash[36902]: audit 2022-04-22T20:48:34.748756+0000 mon.a (mon.0) 10422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:37 smithi115 bash[45495]: cluster 2022-04-22T20:48:36.195894+0000 mgr.y (mgr.34107) 6926 : cluster [DBG] pgmap v8017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:37.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:37 smithi115 bash[46934]: cluster 2022-04-22T20:48:36.195894+0000 mgr.y (mgr.34107) 6926 : cluster [DBG] pgmap v8017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:37.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:37 smithi121 bash[36902]: cluster 2022-04-22T20:48:36.195894+0000 mgr.y (mgr.34107) 6926 : cluster [DBG] pgmap v8017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[45495]: cluster 2022-04-22T20:48:38.197066+0000 mgr.y (mgr.34107) 6927 : cluster [DBG] pgmap v8018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[45495]: audit 2022-04-22T20:48:38.283203+0000 mon.a (mon.0) 10423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[45495]: audit 2022-04-22T20:48:38.336103+0000 mon.a (mon.0) 10424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[46934]: cluster 2022-04-22T20:48:38.197066+0000 mgr.y (mgr.34107) 6927 : cluster [DBG] pgmap v8018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[46934]: audit 2022-04-22T20:48:38.283203+0000 mon.a (mon.0) 10423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:39.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:39 smithi115 bash[46934]: audit 2022-04-22T20:48:38.336103+0000 mon.a (mon.0) 10424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:39 smithi121 bash[36902]: cluster 2022-04-22T20:48:38.197066+0000 mgr.y (mgr.34107) 6927 : cluster [DBG] pgmap v8018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:39 smithi121 bash[36902]: audit 2022-04-22T20:48:38.283203+0000 mon.a (mon.0) 10423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:39 smithi121 bash[36902]: audit 2022-04-22T20:48:38.336103+0000 mon.a (mon.0) 10424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:40 smithi121 bash[36902]: audit 2022-04-22T20:48:39.751793+0000 mon.a (mon.0) 10425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:40.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:40 smithi121 bash[36902]: cluster 2022-04-22T20:48:40.198110+0000 mgr.y (mgr.34107) 6928 : cluster [DBG] pgmap v8019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:40 smithi115 bash[45495]: audit 2022-04-22T20:48:39.751793+0000 mon.a (mon.0) 10425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:40 smithi115 bash[45495]: cluster 2022-04-22T20:48:40.198110+0000 mgr.y (mgr.34107) 6928 : cluster [DBG] pgmap v8019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:40 smithi115 bash[46934]: audit 2022-04-22T20:48:39.751793+0000 mon.a (mon.0) 10425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:40 smithi115 bash[46934]: cluster 2022-04-22T20:48:40.198110+0000 mgr.y (mgr.34107) 6928 : cluster [DBG] pgmap v8019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:42.395Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:43 smithi121 bash[36902]: cluster 2022-04-22T20:48:42.199197+0000 mgr.y (mgr.34107) 6929 : cluster [DBG] pgmap v8020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:43.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:43 smithi115 bash[45495]: cluster 2022-04-22T20:48:42.199197+0000 mgr.y (mgr.34107) 6929 : cluster [DBG] pgmap v8020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:43.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:43 smithi115 bash[46934]: cluster 2022-04-22T20:48:42.199197+0000 mgr.y (mgr.34107) 6929 : cluster [DBG] pgmap v8020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:44 smithi115 bash[46934]: audit 2022-04-22T20:48:43.286202+0000 mon.a (mon.0) 10426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:44.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:44 smithi115 bash[46934]: audit 2022-04-22T20:48:43.339586+0000 mon.a (mon.0) 10427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:44.550 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:44.273Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:44.553 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:44 smithi115 bash[45495]: audit 2022-04-22T20:48:43.286202+0000 mon.a (mon.0) 10426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:44.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:44 smithi115 bash[45495]: audit 2022-04-22T20:48:43.339586+0000 mon.a (mon.0) 10427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:44 smithi121 bash[36902]: audit 2022-04-22T20:48:43.286202+0000 mon.a (mon.0) 10426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:44 smithi121 bash[36902]: audit 2022-04-22T20:48:43.339586+0000 mon.a (mon.0) 10427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:45 smithi115 bash[46934]: cluster 2022-04-22T20:48:44.200151+0000 mgr.y (mgr.34107) 6930 : cluster [DBG] pgmap v8021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:45.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:45 smithi115 bash[46934]: audit 2022-04-22T20:48:44.755184+0000 mon.a (mon.0) 10428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:45 smithi115 bash[45495]: cluster 2022-04-22T20:48:44.200151+0000 mgr.y (mgr.34107) 6930 : cluster [DBG] pgmap v8021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:45.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:45 smithi115 bash[45495]: audit 2022-04-22T20:48:44.755184+0000 mon.a (mon.0) 10428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:45 smithi121 bash[36902]: cluster 2022-04-22T20:48:44.200151+0000 mgr.y (mgr.34107) 6930 : cluster [DBG] pgmap v8021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:45 smithi121 bash[36902]: audit 2022-04-22T20:48:44.755184+0000 mon.a (mon.0) 10428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:47 smithi115 bash[45495]: cluster 2022-04-22T20:48:46.201247+0000 mgr.y (mgr.34107) 6931 : cluster [DBG] pgmap v8022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:47 smithi115 bash[46934]: cluster 2022-04-22T20:48:46.201247+0000 mgr.y (mgr.34107) 6931 : cluster [DBG] pgmap v8022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:47 smithi121 bash[36902]: cluster 2022-04-22T20:48:46.201247+0000 mgr.y (mgr.34107) 6931 : cluster [DBG] pgmap v8022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[45495]: cluster 2022-04-22T20:48:48.202297+0000 mgr.y (mgr.34107) 6932 : cluster [DBG] pgmap v8023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[45495]: audit 2022-04-22T20:48:48.288841+0000 mon.a (mon.0) 10429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:49.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[45495]: audit 2022-04-22T20:48:48.343281+0000 mon.a (mon.0) 10430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[46934]: cluster 2022-04-22T20:48:48.202297+0000 mgr.y (mgr.34107) 6932 : cluster [DBG] pgmap v8023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[46934]: audit 2022-04-22T20:48:48.288841+0000 mon.a (mon.0) 10429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:49.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:49 smithi115 bash[46934]: audit 2022-04-22T20:48:48.343281+0000 mon.a (mon.0) 10430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:49.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:49 smithi121 bash[36902]: cluster 2022-04-22T20:48:48.202297+0000 mgr.y (mgr.34107) 6932 : cluster [DBG] pgmap v8023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:49.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:49 smithi121 bash[36902]: audit 2022-04-22T20:48:48.288841+0000 mon.a (mon.0) 10429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:49.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:49 smithi121 bash[36902]: audit 2022-04-22T20:48:48.343281+0000 mon.a (mon.0) 10430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:50 smithi115 bash[45495]: audit 2022-04-22T20:48:49.758308+0000 mon.a (mon.0) 10431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:50 smithi115 bash[45495]: cluster 2022-04-22T20:48:50.203339+0000 mgr.y (mgr.34107) 6933 : cluster [DBG] pgmap v8024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:50.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:50 smithi115 bash[46934]: audit 2022-04-22T20:48:49.758308+0000 mon.a (mon.0) 10431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:50 smithi115 bash[46934]: cluster 2022-04-22T20:48:50.203339+0000 mgr.y (mgr.34107) 6933 : cluster [DBG] pgmap v8024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:50 smithi121 bash[36902]: audit 2022-04-22T20:48:49.758308+0000 mon.a (mon.0) 10431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:50 smithi121 bash[36902]: cluster 2022-04-22T20:48:50.203339+0000 mgr.y (mgr.34107) 6933 : cluster [DBG] pgmap v8024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:52.395Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:53.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:53 smithi115 bash[45495]: cluster 2022-04-22T20:48:52.204358+0000 mgr.y (mgr.34107) 6934 : cluster [DBG] pgmap v8025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:53.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:53 smithi115 bash[46934]: cluster 2022-04-22T20:48:52.204358+0000 mgr.y (mgr.34107) 6934 : cluster [DBG] pgmap v8025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:53 smithi121 bash[36902]: cluster 2022-04-22T20:48:52.204358+0000 mgr.y (mgr.34107) 6934 : cluster [DBG] pgmap v8025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:54.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:48:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:48:54.274Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:48:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:54 smithi115 bash[45495]: audit 2022-04-22T20:48:53.291785+0000 mon.a (mon.0) 10432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:54.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:54 smithi115 bash[45495]: audit 2022-04-22T20:48:53.346856+0000 mon.a (mon.0) 10433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:54 smithi115 bash[46934]: audit 2022-04-22T20:48:53.291785+0000 mon.a (mon.0) 10432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:54 smithi115 bash[46934]: audit 2022-04-22T20:48:53.346856+0000 mon.a (mon.0) 10433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:54 smithi121 bash[36902]: audit 2022-04-22T20:48:53.291785+0000 mon.a (mon.0) 10432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:54 smithi121 bash[36902]: audit 2022-04-22T20:48:53.346856+0000 mon.a (mon.0) 10433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:55.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:55 smithi115 bash[45495]: cluster 2022-04-22T20:48:54.205419+0000 mgr.y (mgr.34107) 6935 : cluster [DBG] pgmap v8026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:55.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:55 smithi115 bash[45495]: audit 2022-04-22T20:48:54.761443+0000 mon.a (mon.0) 10434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:55 smithi115 bash[46934]: cluster 2022-04-22T20:48:54.205419+0000 mgr.y (mgr.34107) 6935 : cluster [DBG] pgmap v8026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:55.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:55 smithi115 bash[46934]: audit 2022-04-22T20:48:54.761443+0000 mon.a (mon.0) 10434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:55 smithi121 bash[36902]: cluster 2022-04-22T20:48:54.205419+0000 mgr.y (mgr.34107) 6935 : cluster [DBG] pgmap v8026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:55 smithi121 bash[36902]: audit 2022-04-22T20:48:54.761443+0000 mon.a (mon.0) 10434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:48:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:57 smithi115 bash[45495]: cluster 2022-04-22T20:48:56.206491+0000 mgr.y (mgr.34107) 6936 : cluster [DBG] pgmap v8027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:57 smithi115 bash[46934]: cluster 2022-04-22T20:48:56.206491+0000 mgr.y (mgr.34107) 6936 : cluster [DBG] pgmap v8027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:57 smithi121 bash[36902]: cluster 2022-04-22T20:48:56.206491+0000 mgr.y (mgr.34107) 6936 : cluster [DBG] pgmap v8027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:58.142 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:48:58.493 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:48:58.504 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:48:58.505 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:48:58.505 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:48:58.505 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:48:58.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:48:58.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:48:58.506 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:48:58.897 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:48:58.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:48:58.899 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:48:58.899 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:48:58.899 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:48:58.900 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:48:59.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[45495]: audit 2022-04-22T20:48:58.121369+0000 mgr.y (mgr.34107) 6937 : audit [DBG] from='client.52519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[45495]: cluster 2022-04-22T20:48:58.207559+0000 mgr.y (mgr.34107) 6938 : cluster [DBG] pgmap v8028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[45495]: audit 2022-04-22T20:48:58.294590+0000 mon.a (mon.0) 10435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[45495]: audit 2022-04-22T20:48:58.350714+0000 mon.a (mon.0) 10436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:59.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[45495]: audit 2022-04-22T20:48:58.890895+0000 mon.a (mon.0) 10437 : audit [DBG] from='client.? 172.21.15.115:0/785367433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:48:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[46934]: audit 2022-04-22T20:48:58.121369+0000 mgr.y (mgr.34107) 6937 : audit [DBG] from='client.52519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[46934]: cluster 2022-04-22T20:48:58.207559+0000 mgr.y (mgr.34107) 6938 : cluster [DBG] pgmap v8028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:59.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[46934]: audit 2022-04-22T20:48:58.294590+0000 mon.a (mon.0) 10435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[46934]: audit 2022-04-22T20:48:58.350714+0000 mon.a (mon.0) 10436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:59.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:48:59 smithi115 bash[46934]: audit 2022-04-22T20:48:58.890895+0000 mon.a (mon.0) 10437 : audit [DBG] from='client.? 172.21.15.115:0/785367433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:48:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:59 smithi121 bash[36902]: audit 2022-04-22T20:48:58.121369+0000 mgr.y (mgr.34107) 6937 : audit [DBG] from='client.52519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:48:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:59 smithi121 bash[36902]: cluster 2022-04-22T20:48:58.207559+0000 mgr.y (mgr.34107) 6938 : cluster [DBG] pgmap v8028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:48:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:59 smithi121 bash[36902]: audit 2022-04-22T20:48:58.294590+0000 mon.a (mon.0) 10435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:48:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:59 smithi121 bash[36902]: audit 2022-04-22T20:48:58.350714+0000 mon.a (mon.0) 10436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:48:59.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:48:59 smithi121 bash[36902]: audit 2022-04-22T20:48:58.890895+0000 mon.a (mon.0) 10437 : audit [DBG] from='client.? 172.21.15.115:0/785367433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:49:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:00 smithi115 bash[45495]: audit 2022-04-22T20:48:58.468694+0000 mgr.y (mgr.34107) 6939 : audit [DBG] from='client.52525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:00 smithi115 bash[45495]: audit 2022-04-22T20:48:59.764406+0000 mon.a (mon.0) 10438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:00 smithi115 bash[46934]: audit 2022-04-22T20:48:58.468694+0000 mgr.y (mgr.34107) 6939 : audit [DBG] from='client.52525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:00.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:00 smithi115 bash[46934]: audit 2022-04-22T20:48:59.764406+0000 mon.a (mon.0) 10438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:00 smithi121 bash[36902]: audit 2022-04-22T20:48:58.468694+0000 mgr.y (mgr.34107) 6939 : audit [DBG] from='client.52525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:00 smithi121 bash[36902]: audit 2022-04-22T20:48:59.764406+0000 mon.a (mon.0) 10438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:01.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:01 smithi115 bash[45495]: cluster 2022-04-22T20:49:00.208528+0000 mgr.y (mgr.34107) 6940 : cluster [DBG] pgmap v8029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:01.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:01 smithi115 bash[46934]: cluster 2022-04-22T20:49:00.208528+0000 mgr.y (mgr.34107) 6940 : cluster [DBG] pgmap v8029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:01 smithi121 bash[36902]: cluster 2022-04-22T20:49:00.208528+0000 mgr.y (mgr.34107) 6940 : cluster [DBG] pgmap v8029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:02.396Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:03 smithi121 bash[36902]: cluster 2022-04-22T20:49:02.209579+0000 mgr.y (mgr.34107) 6941 : cluster [DBG] pgmap v8030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:03 smithi115 bash[46934]: cluster 2022-04-22T20:49:02.209579+0000 mgr.y (mgr.34107) 6941 : cluster [DBG] pgmap v8030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:03.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:03 smithi115 bash[45495]: cluster 2022-04-22T20:49:02.209579+0000 mgr.y (mgr.34107) 6941 : cluster [DBG] pgmap v8030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:04.276Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:04 smithi115 bash[45495]: audit 2022-04-22T20:49:03.297012+0000 mon.a (mon.0) 10439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:04 smithi115 bash[45495]: audit 2022-04-22T20:49:03.354235+0000 mon.a (mon.0) 10440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:04 smithi115 bash[46934]: audit 2022-04-22T20:49:03.297012+0000 mon.a (mon.0) 10439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:04 smithi115 bash[46934]: audit 2022-04-22T20:49:03.354235+0000 mon.a (mon.0) 10440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:04 smithi121 bash[36902]: audit 2022-04-22T20:49:03.297012+0000 mon.a (mon.0) 10439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:04 smithi121 bash[36902]: audit 2022-04-22T20:49:03.354235+0000 mon.a (mon.0) 10440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:05 smithi121 bash[36902]: cluster 2022-04-22T20:49:04.210639+0000 mgr.y (mgr.34107) 6942 : cluster [DBG] pgmap v8031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:05 smithi121 bash[36902]: audit 2022-04-22T20:49:04.767719+0000 mon.a (mon.0) 10441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:05 smithi115 bash[45495]: cluster 2022-04-22T20:49:04.210639+0000 mgr.y (mgr.34107) 6942 : cluster [DBG] pgmap v8031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:05 smithi115 bash[45495]: audit 2022-04-22T20:49:04.767719+0000 mon.a (mon.0) 10441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:05 smithi115 bash[46934]: cluster 2022-04-22T20:49:04.210639+0000 mgr.y (mgr.34107) 6942 : cluster [DBG] pgmap v8031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:05 smithi115 bash[46934]: audit 2022-04-22T20:49:04.767719+0000 mon.a (mon.0) 10441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:07 smithi121 bash[36902]: cluster 2022-04-22T20:49:06.211722+0000 mgr.y (mgr.34107) 6943 : cluster [DBG] pgmap v8032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:07 smithi115 bash[45495]: cluster 2022-04-22T20:49:06.211722+0000 mgr.y (mgr.34107) 6943 : cluster [DBG] pgmap v8032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:07 smithi115 bash[46934]: cluster 2022-04-22T20:49:06.211722+0000 mgr.y (mgr.34107) 6943 : cluster [DBG] pgmap v8032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:08 smithi121 bash[36902]: audit 2022-04-22T20:49:08.299579+0000 mon.a (mon.0) 10442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:08 smithi115 bash[45495]: audit 2022-04-22T20:49:08.299579+0000 mon.a (mon.0) 10442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:08.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:08 smithi115 bash[46934]: audit 2022-04-22T20:49:08.299579+0000 mon.a (mon.0) 10442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:09 smithi121 bash[36902]: cluster 2022-04-22T20:49:08.212960+0000 mgr.y (mgr.34107) 6944 : cluster [DBG] pgmap v8033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:09 smithi121 bash[36902]: audit 2022-04-22T20:49:08.357967+0000 mon.a (mon.0) 10443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:09 smithi115 bash[45495]: cluster 2022-04-22T20:49:08.212960+0000 mgr.y (mgr.34107) 6944 : cluster [DBG] pgmap v8033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:09 smithi115 bash[45495]: audit 2022-04-22T20:49:08.357967+0000 mon.a (mon.0) 10443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:09 smithi115 bash[46934]: cluster 2022-04-22T20:49:08.212960+0000 mgr.y (mgr.34107) 6944 : cluster [DBG] pgmap v8033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:09 smithi115 bash[46934]: audit 2022-04-22T20:49:08.357967+0000 mon.a (mon.0) 10443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:10.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:10 smithi115 bash[45495]: audit 2022-04-22T20:49:09.770734+0000 mon.a (mon.0) 10444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:10.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:10 smithi115 bash[45495]: cluster 2022-04-22T20:49:10.214063+0000 mgr.y (mgr.34107) 6945 : cluster [DBG] pgmap v8034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:10.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:10 smithi115 bash[46934]: audit 2022-04-22T20:49:09.770734+0000 mon.a (mon.0) 10444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:10.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:10 smithi115 bash[46934]: cluster 2022-04-22T20:49:10.214063+0000 mgr.y (mgr.34107) 6945 : cluster [DBG] pgmap v8034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:10 smithi121 bash[36902]: audit 2022-04-22T20:49:09.770734+0000 mon.a (mon.0) 10444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:10 smithi121 bash[36902]: cluster 2022-04-22T20:49:10.214063+0000 mgr.y (mgr.34107) 6945 : cluster [DBG] pgmap v8034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:12.398Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:13 smithi115 bash[45495]: cluster 2022-04-22T20:49:12.215047+0000 mgr.y (mgr.34107) 6946 : cluster [DBG] pgmap v8035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:13.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:13 smithi115 bash[46934]: cluster 2022-04-22T20:49:12.215047+0000 mgr.y (mgr.34107) 6946 : cluster [DBG] pgmap v8035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:13 smithi121 bash[36902]: cluster 2022-04-22T20:49:12.215047+0000 mgr.y (mgr.34107) 6946 : cluster [DBG] pgmap v8035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:14 smithi115 bash[45495]: audit 2022-04-22T20:49:13.302650+0000 mon.a (mon.0) 10445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:14.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:14 smithi115 bash[45495]: audit 2022-04-22T20:49:13.361699+0000 mon.a (mon.0) 10446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:14 smithi115 bash[46934]: audit 2022-04-22T20:49:13.302650+0000 mon.a (mon.0) 10445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:14 smithi115 bash[46934]: audit 2022-04-22T20:49:13.361699+0000 mon.a (mon.0) 10446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:14.546 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:14.276Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:14 smithi121 bash[36902]: audit 2022-04-22T20:49:13.302650+0000 mon.a (mon.0) 10445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:14 smithi121 bash[36902]: audit 2022-04-22T20:49:13.361699+0000 mon.a (mon.0) 10446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:15 smithi115 bash[46934]: cluster 2022-04-22T20:49:14.216140+0000 mgr.y (mgr.34107) 6947 : cluster [DBG] pgmap v8036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:15 smithi115 bash[46934]: audit 2022-04-22T20:49:14.773930+0000 mon.a (mon.0) 10447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:15 smithi115 bash[45495]: cluster 2022-04-22T20:49:14.216140+0000 mgr.y (mgr.34107) 6947 : cluster [DBG] pgmap v8036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:15.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:15 smithi115 bash[45495]: audit 2022-04-22T20:49:14.773930+0000 mon.a (mon.0) 10447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:15 smithi121 bash[36902]: cluster 2022-04-22T20:49:14.216140+0000 mgr.y (mgr.34107) 6947 : cluster [DBG] pgmap v8036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:15 smithi121 bash[36902]: audit 2022-04-22T20:49:14.773930+0000 mon.a (mon.0) 10447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:16 smithi121 bash[36902]: audit 2022-04-22T20:49:15.609532+0000 mon.a (mon.0) 10448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:49:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:16 smithi121 bash[36902]: audit 2022-04-22T20:49:15.609993+0000 mon.a (mon.0) 10449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:49:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:16 smithi115 bash[45495]: audit 2022-04-22T20:49:15.609532+0000 mon.a (mon.0) 10448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:49:16.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:16 smithi115 bash[45495]: audit 2022-04-22T20:49:15.609993+0000 mon.a (mon.0) 10449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:49:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:16 smithi115 bash[46934]: audit 2022-04-22T20:49:15.609532+0000 mon.a (mon.0) 10448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:49:16.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:16 smithi115 bash[46934]: audit 2022-04-22T20:49:15.609993+0000 mon.a (mon.0) 10449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:49:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:17 smithi121 bash[36902]: cluster 2022-04-22T20:49:16.217220+0000 mgr.y (mgr.34107) 6948 : cluster [DBG] pgmap v8037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:17 smithi115 bash[45495]: cluster 2022-04-22T20:49:16.217220+0000 mgr.y (mgr.34107) 6948 : cluster [DBG] pgmap v8037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:17.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:17 smithi115 bash[46934]: cluster 2022-04-22T20:49:16.217220+0000 mgr.y (mgr.34107) 6948 : cluster [DBG] pgmap v8037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: cluster 2022-04-22T20:49:18.218243+0000 mgr.y (mgr.34107) 6949 : cluster [DBG] pgmap v8038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:18.305693+0000 mon.a (mon.0) 10450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:18.365246+0000 mon.a (mon.0) 10451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:18.709728+0000 mon.a (mon.0) 10452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:49:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:18.712818+0000 mon.a (mon.0) 10453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:49:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:18.714575+0000 mon.a (mon.0) 10454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:49:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:19 smithi121 bash[36902]: audit 2022-04-22T20:49:19.089795+0000 mon.a (mon.0) 10455 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:49:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: cluster 2022-04-22T20:49:18.218243+0000 mgr.y (mgr.34107) 6949 : cluster [DBG] pgmap v8038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:18.305693+0000 mon.a (mon.0) 10450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:18.365246+0000 mon.a (mon.0) 10451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:18.709728+0000 mon.a (mon.0) 10452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:18.712818+0000 mon.a (mon.0) 10453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:18.714575+0000 mon.a (mon.0) 10454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[45495]: audit 2022-04-22T20:49:19.089795+0000 mon.a (mon.0) 10455 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: cluster 2022-04-22T20:49:18.218243+0000 mgr.y (mgr.34107) 6949 : cluster [DBG] pgmap v8038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:19.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:18.305693+0000 mon.a (mon.0) 10450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:18.365246+0000 mon.a (mon.0) 10451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:18.709728+0000 mon.a (mon.0) 10452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:49:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:18.712818+0000 mon.a (mon.0) 10453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:49:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:18.714575+0000 mon.a (mon.0) 10454 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:49:19.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:19 smithi115 bash[46934]: audit 2022-04-22T20:49:19.089795+0000 mon.a (mon.0) 10455 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:49:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:20 smithi115 bash[46934]: audit 2022-04-22T20:49:19.776925+0000 mon.a (mon.0) 10456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:20 smithi115 bash[46934]: cluster 2022-04-22T20:49:20.219299+0000 mgr.y (mgr.34107) 6950 : cluster [DBG] pgmap v8039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:20 smithi115 bash[45495]: audit 2022-04-22T20:49:19.776925+0000 mon.a (mon.0) 10456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:20 smithi115 bash[45495]: cluster 2022-04-22T20:49:20.219299+0000 mgr.y (mgr.34107) 6950 : cluster [DBG] pgmap v8039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:20 smithi121 bash[36902]: audit 2022-04-22T20:49:19.776925+0000 mon.a (mon.0) 10456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:20 smithi121 bash[36902]: cluster 2022-04-22T20:49:20.219299+0000 mgr.y (mgr.34107) 6950 : cluster [DBG] pgmap v8039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:22.399Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:23.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:23 smithi115 bash[45495]: cluster 2022-04-22T20:49:22.220317+0000 mgr.y (mgr.34107) 6951 : cluster [DBG] pgmap v8040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:23.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:23 smithi115 bash[46934]: cluster 2022-04-22T20:49:22.220317+0000 mgr.y (mgr.34107) 6951 : cluster [DBG] pgmap v8040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:23.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:23 smithi121 bash[36902]: cluster 2022-04-22T20:49:22.220317+0000 mgr.y (mgr.34107) 6951 : cluster [DBG] pgmap v8040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:24.277Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:24 smithi115 bash[45495]: audit 2022-04-22T20:49:23.308693+0000 mon.a (mon.0) 10457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:24 smithi115 bash[45495]: audit 2022-04-22T20:49:23.368773+0000 mon.a (mon.0) 10458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:24 smithi115 bash[46934]: audit 2022-04-22T20:49:23.308693+0000 mon.a (mon.0) 10457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:24 smithi115 bash[46934]: audit 2022-04-22T20:49:23.368773+0000 mon.a (mon.0) 10458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:24 smithi121 bash[36902]: audit 2022-04-22T20:49:23.308693+0000 mon.a (mon.0) 10457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:24 smithi121 bash[36902]: audit 2022-04-22T20:49:23.368773+0000 mon.a (mon.0) 10458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:25 smithi121 bash[36902]: cluster 2022-04-22T20:49:24.221513+0000 mgr.y (mgr.34107) 6952 : cluster [DBG] pgmap v8041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:25 smithi121 bash[36902]: audit 2022-04-22T20:49:24.780036+0000 mon.a (mon.0) 10459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:25 smithi115 bash[45495]: cluster 2022-04-22T20:49:24.221513+0000 mgr.y (mgr.34107) 6952 : cluster [DBG] pgmap v8041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:25 smithi115 bash[45495]: audit 2022-04-22T20:49:24.780036+0000 mon.a (mon.0) 10459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:25 smithi115 bash[46934]: cluster 2022-04-22T20:49:24.221513+0000 mgr.y (mgr.34107) 6952 : cluster [DBG] pgmap v8041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:25 smithi115 bash[46934]: audit 2022-04-22T20:49:24.780036+0000 mon.a (mon.0) 10459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:27 smithi121 bash[36902]: cluster 2022-04-22T20:49:26.222673+0000 mgr.y (mgr.34107) 6953 : cluster [DBG] pgmap v8042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:27 smithi115 bash[45495]: cluster 2022-04-22T20:49:26.222673+0000 mgr.y (mgr.34107) 6953 : cluster [DBG] pgmap v8042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:27 smithi115 bash[46934]: cluster 2022-04-22T20:49:26.222673+0000 mgr.y (mgr.34107) 6953 : cluster [DBG] pgmap v8042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:29.266 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:49:29.613 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:49:29.613 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:49:29.614 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:49:29.615 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:49:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:29 smithi121 bash[36902]: cluster 2022-04-22T20:49:28.223814+0000 mgr.y (mgr.34107) 6954 : cluster [DBG] pgmap v8043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:29 smithi121 bash[36902]: audit 2022-04-22T20:49:28.311693+0000 mon.a (mon.0) 10460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:29 smithi121 bash[36902]: audit 2022-04-22T20:49:28.372100+0000 mon.a (mon.0) 10461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[45495]: cluster 2022-04-22T20:49:28.223814+0000 mgr.y (mgr.34107) 6954 : cluster [DBG] pgmap v8043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[45495]: audit 2022-04-22T20:49:28.311693+0000 mon.a (mon.0) 10460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[45495]: audit 2022-04-22T20:49:28.372100+0000 mon.a (mon.0) 10461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[46934]: cluster 2022-04-22T20:49:28.223814+0000 mgr.y (mgr.34107) 6954 : cluster [DBG] pgmap v8043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[46934]: audit 2022-04-22T20:49:28.311693+0000 mon.a (mon.0) 10460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:29.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:29 smithi115 bash[46934]: audit 2022-04-22T20:49:28.372100+0000 mon.a (mon.0) 10461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:30.019 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:49:30.020 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:49:30.020 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:49:30.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:49:30.020 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:49:30.021 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:49:30.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:49:30.021 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:49:30.021 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:49:30.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:49:30.022 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:49:30.022 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:49:30.022 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:49:30.022 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:49:30.023 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:49:30.023 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:49:30.024 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:49:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:30 smithi121 bash[36902]: audit 2022-04-22T20:49:29.244429+0000 mgr.y (mgr.34107) 6955 : audit [DBG] from='client.42606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:30 smithi121 bash[36902]: audit 2022-04-22T20:49:29.783023+0000 mon.a (mon.0) 10462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:30 smithi121 bash[36902]: audit 2022-04-22T20:49:30.013926+0000 mon.a (mon.0) 10463 : audit [DBG] from='client.? 172.21.15.115:0/154463125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:49:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[45495]: audit 2022-04-22T20:49:29.244429+0000 mgr.y (mgr.34107) 6955 : audit [DBG] from='client.42606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[45495]: audit 2022-04-22T20:49:29.783023+0000 mon.a (mon.0) 10462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[45495]: audit 2022-04-22T20:49:30.013926+0000 mon.a (mon.0) 10463 : audit [DBG] from='client.? 172.21.15.115:0/154463125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:49:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[46934]: audit 2022-04-22T20:49:29.244429+0000 mgr.y (mgr.34107) 6955 : audit [DBG] from='client.42606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[46934]: audit 2022-04-22T20:49:29.783023+0000 mon.a (mon.0) 10462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:30.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:30 smithi115 bash[46934]: audit 2022-04-22T20:49:30.013926+0000 mon.a (mon.0) 10463 : audit [DBG] from='client.? 172.21.15.115:0/154463125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:49:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:31 smithi121 bash[36902]: audit 2022-04-22T20:49:29.586648+0000 mgr.y (mgr.34107) 6956 : audit [DBG] from='client.52543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:31 smithi121 bash[36902]: cluster 2022-04-22T20:49:30.225044+0000 mgr.y (mgr.34107) 6957 : cluster [DBG] pgmap v8044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:31.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:31 smithi115 bash[45495]: audit 2022-04-22T20:49:29.586648+0000 mgr.y (mgr.34107) 6956 : audit [DBG] from='client.52543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:31 smithi115 bash[45495]: cluster 2022-04-22T20:49:30.225044+0000 mgr.y (mgr.34107) 6957 : cluster [DBG] pgmap v8044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:31 smithi115 bash[46934]: audit 2022-04-22T20:49:29.586648+0000 mgr.y (mgr.34107) 6956 : audit [DBG] from='client.52543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:49:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:31 smithi115 bash[46934]: cluster 2022-04-22T20:49:30.225044+0000 mgr.y (mgr.34107) 6957 : cluster [DBG] pgmap v8044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:32.399Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:33 smithi121 bash[36902]: cluster 2022-04-22T20:49:32.226235+0000 mgr.y (mgr.34107) 6958 : cluster [DBG] pgmap v8045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:33 smithi115 bash[45495]: cluster 2022-04-22T20:49:32.226235+0000 mgr.y (mgr.34107) 6958 : cluster [DBG] pgmap v8045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:33 smithi115 bash[46934]: cluster 2022-04-22T20:49:32.226235+0000 mgr.y (mgr.34107) 6958 : cluster [DBG] pgmap v8045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:34 smithi115 bash[45495]: audit 2022-04-22T20:49:33.314197+0000 mon.a (mon.0) 10464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:34 smithi115 bash[45495]: audit 2022-04-22T20:49:33.375736+0000 mon.a (mon.0) 10465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:34.278Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:34 smithi115 bash[46934]: audit 2022-04-22T20:49:33.314197+0000 mon.a (mon.0) 10464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:34.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:34 smithi115 bash[46934]: audit 2022-04-22T20:49:33.375736+0000 mon.a (mon.0) 10465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:34 smithi121 bash[36902]: audit 2022-04-22T20:49:33.314197+0000 mon.a (mon.0) 10464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:34 smithi121 bash[36902]: audit 2022-04-22T20:49:33.375736+0000 mon.a (mon.0) 10465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:35 smithi121 bash[36902]: cluster 2022-04-22T20:49:34.227307+0000 mgr.y (mgr.34107) 6959 : cluster [DBG] pgmap v8046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:35 smithi121 bash[36902]: audit 2022-04-22T20:49:34.786548+0000 mon.a (mon.0) 10466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:35 smithi115 bash[45495]: cluster 2022-04-22T20:49:34.227307+0000 mgr.y (mgr.34107) 6959 : cluster [DBG] pgmap v8046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:35 smithi115 bash[45495]: audit 2022-04-22T20:49:34.786548+0000 mon.a (mon.0) 10466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:35 smithi115 bash[46934]: cluster 2022-04-22T20:49:34.227307+0000 mgr.y (mgr.34107) 6959 : cluster [DBG] pgmap v8046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:35 smithi115 bash[46934]: audit 2022-04-22T20:49:34.786548+0000 mon.a (mon.0) 10466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:37 smithi121 bash[36902]: cluster 2022-04-22T20:49:36.228427+0000 mgr.y (mgr.34107) 6960 : cluster [DBG] pgmap v8047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:37 smithi115 bash[45495]: cluster 2022-04-22T20:49:36.228427+0000 mgr.y (mgr.34107) 6960 : cluster [DBG] pgmap v8047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:37 smithi115 bash[46934]: cluster 2022-04-22T20:49:36.228427+0000 mgr.y (mgr.34107) 6960 : cluster [DBG] pgmap v8047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:38 smithi121 bash[36902]: audit 2022-04-22T20:49:38.316903+0000 mon.a (mon.0) 10467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:38.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:38 smithi115 bash[45495]: audit 2022-04-22T20:49:38.316903+0000 mon.a (mon.0) 10467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:38 smithi115 bash[46934]: audit 2022-04-22T20:49:38.316903+0000 mon.a (mon.0) 10467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:39 smithi121 bash[36902]: cluster 2022-04-22T20:49:38.229543+0000 mgr.y (mgr.34107) 6961 : cluster [DBG] pgmap v8048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:39 smithi121 bash[36902]: audit 2022-04-22T20:49:38.379356+0000 mon.a (mon.0) 10468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:39 smithi115 bash[45495]: cluster 2022-04-22T20:49:38.229543+0000 mgr.y (mgr.34107) 6961 : cluster [DBG] pgmap v8048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:39 smithi115 bash[45495]: audit 2022-04-22T20:49:38.379356+0000 mon.a (mon.0) 10468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:39 smithi115 bash[46934]: cluster 2022-04-22T20:49:38.229543+0000 mgr.y (mgr.34107) 6961 : cluster [DBG] pgmap v8048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:39.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:39 smithi115 bash[46934]: audit 2022-04-22T20:49:38.379356+0000 mon.a (mon.0) 10468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:40.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:40 smithi115 bash[45495]: audit 2022-04-22T20:49:39.789656+0000 mon.a (mon.0) 10469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:40.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:40 smithi115 bash[45495]: cluster 2022-04-22T20:49:40.230636+0000 mgr.y (mgr.34107) 6962 : cluster [DBG] pgmap v8049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:40 smithi115 bash[46934]: audit 2022-04-22T20:49:39.789656+0000 mon.a (mon.0) 10469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:40.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:40 smithi115 bash[46934]: cluster 2022-04-22T20:49:40.230636+0000 mgr.y (mgr.34107) 6962 : cluster [DBG] pgmap v8049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:40 smithi121 bash[36902]: audit 2022-04-22T20:49:39.789656+0000 mon.a (mon.0) 10469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:40 smithi121 bash[36902]: cluster 2022-04-22T20:49:40.230636+0000 mgr.y (mgr.34107) 6962 : cluster [DBG] pgmap v8049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:42.400Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:43 smithi121 bash[36902]: cluster 2022-04-22T20:49:42.231697+0000 mgr.y (mgr.34107) 6963 : cluster [DBG] pgmap v8050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:43 smithi115 bash[45495]: cluster 2022-04-22T20:49:42.231697+0000 mgr.y (mgr.34107) 6963 : cluster [DBG] pgmap v8050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:43 smithi115 bash[46934]: cluster 2022-04-22T20:49:42.231697+0000 mgr.y (mgr.34107) 6963 : cluster [DBG] pgmap v8050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:44.278Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:44 smithi115 bash[45495]: audit 2022-04-22T20:49:43.319949+0000 mon.a (mon.0) 10470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:44 smithi115 bash[45495]: audit 2022-04-22T20:49:43.382890+0000 mon.a (mon.0) 10471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:44 smithi115 bash[46934]: audit 2022-04-22T20:49:43.319949+0000 mon.a (mon.0) 10470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:44 smithi115 bash[46934]: audit 2022-04-22T20:49:43.382890+0000 mon.a (mon.0) 10471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:44 smithi121 bash[36902]: audit 2022-04-22T20:49:43.319949+0000 mon.a (mon.0) 10470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:44 smithi121 bash[36902]: audit 2022-04-22T20:49:43.382890+0000 mon.a (mon.0) 10471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:45.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:45 smithi121 bash[36902]: cluster 2022-04-22T20:49:44.232747+0000 mgr.y (mgr.34107) 6964 : cluster [DBG] pgmap v8051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:45.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:45 smithi121 bash[36902]: audit 2022-04-22T20:49:44.792686+0000 mon.a (mon.0) 10472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:45 smithi115 bash[45495]: cluster 2022-04-22T20:49:44.232747+0000 mgr.y (mgr.34107) 6964 : cluster [DBG] pgmap v8051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:45 smithi115 bash[45495]: audit 2022-04-22T20:49:44.792686+0000 mon.a (mon.0) 10472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:45 smithi115 bash[46934]: cluster 2022-04-22T20:49:44.232747+0000 mgr.y (mgr.34107) 6964 : cluster [DBG] pgmap v8051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:45.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:45 smithi115 bash[46934]: audit 2022-04-22T20:49:44.792686+0000 mon.a (mon.0) 10472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:47 smithi121 bash[36902]: cluster 2022-04-22T20:49:46.233841+0000 mgr.y (mgr.34107) 6965 : cluster [DBG] pgmap v8052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:47.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:47 smithi115 bash[45495]: cluster 2022-04-22T20:49:46.233841+0000 mgr.y (mgr.34107) 6965 : cluster [DBG] pgmap v8052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:47.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:47 smithi115 bash[46934]: cluster 2022-04-22T20:49:46.233841+0000 mgr.y (mgr.34107) 6965 : cluster [DBG] pgmap v8052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:49 smithi121 bash[36902]: cluster 2022-04-22T20:49:48.234995+0000 mgr.y (mgr.34107) 6966 : cluster [DBG] pgmap v8053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:49 smithi121 bash[36902]: audit 2022-04-22T20:49:48.322831+0000 mon.a (mon.0) 10473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:49 smithi121 bash[36902]: audit 2022-04-22T20:49:48.386420+0000 mon.a (mon.0) 10474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[45495]: cluster 2022-04-22T20:49:48.234995+0000 mgr.y (mgr.34107) 6966 : cluster [DBG] pgmap v8053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[45495]: audit 2022-04-22T20:49:48.322831+0000 mon.a (mon.0) 10473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[45495]: audit 2022-04-22T20:49:48.386420+0000 mon.a (mon.0) 10474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[46934]: cluster 2022-04-22T20:49:48.234995+0000 mgr.y (mgr.34107) 6966 : cluster [DBG] pgmap v8053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[46934]: audit 2022-04-22T20:49:48.322831+0000 mon.a (mon.0) 10473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:49 smithi115 bash[46934]: audit 2022-04-22T20:49:48.386420+0000 mon.a (mon.0) 10474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:50 smithi115 bash[45495]: audit 2022-04-22T20:49:49.795840+0000 mon.a (mon.0) 10475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:50.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:50 smithi115 bash[45495]: cluster 2022-04-22T20:49:50.236041+0000 mgr.y (mgr.34107) 6967 : cluster [DBG] pgmap v8054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:50 smithi115 bash[46934]: audit 2022-04-22T20:49:49.795840+0000 mon.a (mon.0) 10475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:50.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:50 smithi115 bash[46934]: cluster 2022-04-22T20:49:50.236041+0000 mgr.y (mgr.34107) 6967 : cluster [DBG] pgmap v8054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:50 smithi121 bash[36902]: audit 2022-04-22T20:49:49.795840+0000 mon.a (mon.0) 10475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:51.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:50 smithi121 bash[36902]: cluster 2022-04-22T20:49:50.236041+0000 mgr.y (mgr.34107) 6967 : cluster [DBG] pgmap v8054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:52.399Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:53 smithi121 bash[36902]: cluster 2022-04-22T20:49:52.237138+0000 mgr.y (mgr.34107) 6968 : cluster [DBG] pgmap v8055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:53 smithi115 bash[45495]: cluster 2022-04-22T20:49:52.237138+0000 mgr.y (mgr.34107) 6968 : cluster [DBG] pgmap v8055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:53 smithi115 bash[46934]: cluster 2022-04-22T20:49:52.237138+0000 mgr.y (mgr.34107) 6968 : cluster [DBG] pgmap v8055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:54.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:49:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:49:54.278Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:49:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:54 smithi115 bash[45495]: audit 2022-04-22T20:49:53.325437+0000 mon.a (mon.0) 10476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:54.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:54 smithi115 bash[45495]: audit 2022-04-22T20:49:53.389828+0000 mon.a (mon.0) 10477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:54 smithi115 bash[46934]: audit 2022-04-22T20:49:53.325437+0000 mon.a (mon.0) 10476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:54 smithi115 bash[46934]: audit 2022-04-22T20:49:53.389828+0000 mon.a (mon.0) 10477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:54 smithi121 bash[36902]: audit 2022-04-22T20:49:53.325437+0000 mon.a (mon.0) 10476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:54 smithi121 bash[36902]: audit 2022-04-22T20:49:53.389828+0000 mon.a (mon.0) 10477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:55 smithi121 bash[36902]: cluster 2022-04-22T20:49:54.238168+0000 mgr.y (mgr.34107) 6969 : cluster [DBG] pgmap v8056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:55 smithi121 bash[36902]: audit 2022-04-22T20:49:54.799024+0000 mon.a (mon.0) 10478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:55 smithi115 bash[45495]: cluster 2022-04-22T20:49:54.238168+0000 mgr.y (mgr.34107) 6969 : cluster [DBG] pgmap v8056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:55 smithi115 bash[45495]: audit 2022-04-22T20:49:54.799024+0000 mon.a (mon.0) 10478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:55 smithi115 bash[46934]: cluster 2022-04-22T20:49:54.238168+0000 mgr.y (mgr.34107) 6969 : cluster [DBG] pgmap v8056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:55 smithi115 bash[46934]: audit 2022-04-22T20:49:54.799024+0000 mon.a (mon.0) 10478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:49:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:57 smithi121 bash[36902]: cluster 2022-04-22T20:49:56.239221+0000 mgr.y (mgr.34107) 6970 : cluster [DBG] pgmap v8057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:57.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:57 smithi115 bash[45495]: cluster 2022-04-22T20:49:56.239221+0000 mgr.y (mgr.34107) 6970 : cluster [DBG] pgmap v8057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:57.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:57 smithi115 bash[46934]: cluster 2022-04-22T20:49:56.239221+0000 mgr.y (mgr.34107) 6970 : cluster [DBG] pgmap v8057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:59 smithi121 bash[36902]: cluster 2022-04-22T20:49:58.240355+0000 mgr.y (mgr.34107) 6971 : cluster [DBG] pgmap v8058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:59 smithi121 bash[36902]: audit 2022-04-22T20:49:58.328594+0000 mon.a (mon.0) 10479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:59.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:49:59 smithi121 bash[36902]: audit 2022-04-22T20:49:58.393343+0000 mon.a (mon.0) 10480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[45495]: cluster 2022-04-22T20:49:58.240355+0000 mgr.y (mgr.34107) 6971 : cluster [DBG] pgmap v8058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[45495]: audit 2022-04-22T20:49:58.328594+0000 mon.a (mon.0) 10479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[45495]: audit 2022-04-22T20:49:58.393343+0000 mon.a (mon.0) 10480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:49:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[46934]: cluster 2022-04-22T20:49:58.240355+0000 mgr.y (mgr.34107) 6971 : cluster [DBG] pgmap v8058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:49:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[46934]: audit 2022-04-22T20:49:58.328594+0000 mon.a (mon.0) 10479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:49:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:49:59 smithi115 bash[46934]: audit 2022-04-22T20:49:58.393343+0000 mon.a (mon.0) 10480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:50:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:50:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:50:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:50:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:50:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:50:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[46934]: audit 2022-04-22T20:49:59.802219+0000 mon.a (mon.0) 10481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[46934]: cluster 2022-04-22T20:50:00.000149+0000 mon.a (mon.0) 10482 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:50:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[46934]: cluster 2022-04-22T20:50:00.241421+0000 mgr.y (mgr.34107) 6972 : cluster [DBG] pgmap v8059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[46934]: audit 2022-04-22T20:50:00.380462+0000 mgr.y (mgr.34107) 6973 : audit [DBG] from='client.52555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[45495]: audit 2022-04-22T20:49:59.802219+0000 mon.a (mon.0) 10481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[45495]: cluster 2022-04-22T20:50:00.000149+0000 mon.a (mon.0) 10482 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:50:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[45495]: cluster 2022-04-22T20:50:00.241421+0000 mgr.y (mgr.34107) 6972 : cluster [DBG] pgmap v8059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:00.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:00 smithi115 bash[45495]: audit 2022-04-22T20:50:00.380462+0000 mgr.y (mgr.34107) 6973 : audit [DBG] from='client.52555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:00 smithi121 bash[36902]: audit 2022-04-22T20:49:59.802219+0000 mon.a (mon.0) 10481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:00 smithi121 bash[36902]: cluster 2022-04-22T20:50:00.000149+0000 mon.a (mon.0) 10482 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T20:50:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:00 smithi121 bash[36902]: cluster 2022-04-22T20:50:00.241421+0000 mgr.y (mgr.34107) 6972 : cluster [DBG] pgmap v8059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:00 smithi121 bash[36902]: audit 2022-04-22T20:50:00.380462+0000 mgr.y (mgr.34107) 6973 : audit [DBG] from='client.52555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:01.156 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:50:01.157 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:50:01.157 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:50:01.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:01.157 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:50:01.158 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:50:01.158 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:01.158 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:50:01.158 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:50:01.159 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:01.159 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:50:01.159 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:50:01.159 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:01.159 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:50:01.160 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:50:01.160 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:50:01.160 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:50:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:01 smithi115 bash[46934]: audit 2022-04-22T20:50:00.726100+0000 mgr.y (mgr.34107) 6974 : audit [DBG] from='client.52561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:01 smithi115 bash[46934]: audit 2022-04-22T20:50:01.150394+0000 mon.a (mon.0) 10483 : audit [DBG] from='client.? 172.21.15.115:0/2468908772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:01.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:01 smithi115 bash[45495]: audit 2022-04-22T20:50:00.726100+0000 mgr.y (mgr.34107) 6974 : audit [DBG] from='client.52561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:01.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:01 smithi115 bash[45495]: audit 2022-04-22T20:50:01.150394+0000 mon.a (mon.0) 10483 : audit [DBG] from='client.? 172.21.15.115:0/2468908772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:01 smithi121 bash[36902]: audit 2022-04-22T20:50:00.726100+0000 mgr.y (mgr.34107) 6974 : audit [DBG] from='client.52561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:01 smithi121 bash[36902]: audit 2022-04-22T20:50:01.150394+0000 mon.a (mon.0) 10483 : audit [DBG] from='client.? 172.21.15.115:0/2468908772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:02 smithi115 bash[46934]: cluster 2022-04-22T20:50:02.242587+0000 mgr.y (mgr.34107) 6975 : cluster [DBG] pgmap v8060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:02.400Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:02.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:02 smithi115 bash[45495]: cluster 2022-04-22T20:50:02.242587+0000 mgr.y (mgr.34107) 6975 : cluster [DBG] pgmap v8060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:02 smithi121 bash[36902]: cluster 2022-04-22T20:50:02.242587+0000 mgr.y (mgr.34107) 6975 : cluster [DBG] pgmap v8060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:03 smithi121 bash[36902]: audit 2022-04-22T20:50:03.331562+0000 mon.a (mon.0) 10484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:03 smithi121 bash[36902]: audit 2022-04-22T20:50:03.396900+0000 mon.a (mon.0) 10485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:04.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:03 smithi115 bash[46934]: audit 2022-04-22T20:50:03.331562+0000 mon.a (mon.0) 10484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:04.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:03 smithi115 bash[46934]: audit 2022-04-22T20:50:03.396900+0000 mon.a (mon.0) 10485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:03 smithi115 bash[45495]: audit 2022-04-22T20:50:03.331562+0000 mon.a (mon.0) 10484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:04.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:03 smithi115 bash[45495]: audit 2022-04-22T20:50:03.396900+0000 mon.a (mon.0) 10485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:04.280Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:04 smithi121 bash[36902]: cluster 2022-04-22T20:50:04.243765+0000 mgr.y (mgr.34107) 6976 : cluster [DBG] pgmap v8061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:04 smithi115 bash[46934]: cluster 2022-04-22T20:50:04.243765+0000 mgr.y (mgr.34107) 6976 : cluster [DBG] pgmap v8061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:05.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:04 smithi115 bash[45495]: cluster 2022-04-22T20:50:04.243765+0000 mgr.y (mgr.34107) 6976 : cluster [DBG] pgmap v8061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:05 smithi121 bash[36902]: audit 2022-04-22T20:50:04.805320+0000 mon.a (mon.0) 10486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:06.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:05 smithi115 bash[45495]: audit 2022-04-22T20:50:04.805320+0000 mon.a (mon.0) 10486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:06.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:05 smithi115 bash[46934]: audit 2022-04-22T20:50:04.805320+0000 mon.a (mon.0) 10486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:06 smithi121 bash[36902]: cluster 2022-04-22T20:50:06.244814+0000 mgr.y (mgr.34107) 6977 : cluster [DBG] pgmap v8062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:07.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:06 smithi115 bash[45495]: cluster 2022-04-22T20:50:06.244814+0000 mgr.y (mgr.34107) 6977 : cluster [DBG] pgmap v8062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:07.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:06 smithi115 bash[46934]: cluster 2022-04-22T20:50:06.244814+0000 mgr.y (mgr.34107) 6977 : cluster [DBG] pgmap v8062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:09 smithi121 bash[36902]: cluster 2022-04-22T20:50:08.245900+0000 mgr.y (mgr.34107) 6978 : cluster [DBG] pgmap v8063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:09 smithi121 bash[36902]: audit 2022-04-22T20:50:08.334794+0000 mon.a (mon.0) 10487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:09 smithi121 bash[36902]: audit 2022-04-22T20:50:08.400391+0000 mon.a (mon.0) 10488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[45495]: cluster 2022-04-22T20:50:08.245900+0000 mgr.y (mgr.34107) 6978 : cluster [DBG] pgmap v8063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[45495]: audit 2022-04-22T20:50:08.334794+0000 mon.a (mon.0) 10487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[45495]: audit 2022-04-22T20:50:08.400391+0000 mon.a (mon.0) 10488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[46934]: cluster 2022-04-22T20:50:08.245900+0000 mgr.y (mgr.34107) 6978 : cluster [DBG] pgmap v8063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[46934]: audit 2022-04-22T20:50:08.334794+0000 mon.a (mon.0) 10487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:09 smithi115 bash[46934]: audit 2022-04-22T20:50:08.400391+0000 mon.a (mon.0) 10488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:10 smithi115 bash[45495]: audit 2022-04-22T20:50:09.808353+0000 mon.a (mon.0) 10489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:10.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:10 smithi115 bash[45495]: cluster 2022-04-22T20:50:10.246979+0000 mgr.y (mgr.34107) 6979 : cluster [DBG] pgmap v8064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:10 smithi115 bash[46934]: audit 2022-04-22T20:50:09.808353+0000 mon.a (mon.0) 10489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:10 smithi115 bash[46934]: cluster 2022-04-22T20:50:10.246979+0000 mgr.y (mgr.34107) 6979 : cluster [DBG] pgmap v8064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:10 smithi121 bash[36902]: audit 2022-04-22T20:50:09.808353+0000 mon.a (mon.0) 10489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:10 smithi121 bash[36902]: cluster 2022-04-22T20:50:10.246979+0000 mgr.y (mgr.34107) 6979 : cluster [DBG] pgmap v8064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:12.402Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:13 smithi121 bash[36902]: cluster 2022-04-22T20:50:12.248076+0000 mgr.y (mgr.34107) 6980 : cluster [DBG] pgmap v8065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:13 smithi115 bash[46934]: cluster 2022-04-22T20:50:12.248076+0000 mgr.y (mgr.34107) 6980 : cluster [DBG] pgmap v8065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:13 smithi115 bash[45495]: cluster 2022-04-22T20:50:12.248076+0000 mgr.y (mgr.34107) 6980 : cluster [DBG] pgmap v8065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:14 smithi115 bash[45495]: audit 2022-04-22T20:50:13.337699+0000 mon.a (mon.0) 10490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:14 smithi115 bash[45495]: audit 2022-04-22T20:50:13.403763+0000 mon.a (mon.0) 10491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:14.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:14.280Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:14 smithi115 bash[46934]: audit 2022-04-22T20:50:13.337699+0000 mon.a (mon.0) 10490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:14 smithi115 bash[46934]: audit 2022-04-22T20:50:13.403763+0000 mon.a (mon.0) 10491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:14 smithi121 bash[36902]: audit 2022-04-22T20:50:13.337699+0000 mon.a (mon.0) 10490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:14 smithi121 bash[36902]: audit 2022-04-22T20:50:13.403763+0000 mon.a (mon.0) 10491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:15 smithi121 bash[36902]: cluster 2022-04-22T20:50:14.249174+0000 mgr.y (mgr.34107) 6981 : cluster [DBG] pgmap v8066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:15 smithi121 bash[36902]: audit 2022-04-22T20:50:14.811428+0000 mon.a (mon.0) 10492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:15 smithi115 bash[45495]: cluster 2022-04-22T20:50:14.249174+0000 mgr.y (mgr.34107) 6981 : cluster [DBG] pgmap v8066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:15 smithi115 bash[45495]: audit 2022-04-22T20:50:14.811428+0000 mon.a (mon.0) 10492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:15 smithi115 bash[46934]: cluster 2022-04-22T20:50:14.249174+0000 mgr.y (mgr.34107) 6981 : cluster [DBG] pgmap v8066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:15 smithi115 bash[46934]: audit 2022-04-22T20:50:14.811428+0000 mon.a (mon.0) 10492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:16 smithi121 bash[36902]: audit 2022-04-22T20:50:15.618923+0000 mon.a (mon.0) 10493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:50:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:16 smithi121 bash[36902]: audit 2022-04-22T20:50:15.619152+0000 mon.a (mon.0) 10494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:50:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:16 smithi115 bash[45495]: audit 2022-04-22T20:50:15.618923+0000 mon.a (mon.0) 10493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:50:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:16 smithi115 bash[45495]: audit 2022-04-22T20:50:15.619152+0000 mon.a (mon.0) 10494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:50:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:16 smithi115 bash[46934]: audit 2022-04-22T20:50:15.618923+0000 mon.a (mon.0) 10493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:50:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:16 smithi115 bash[46934]: audit 2022-04-22T20:50:15.619152+0000 mon.a (mon.0) 10494 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:50:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:17 smithi121 bash[36902]: cluster 2022-04-22T20:50:16.250283+0000 mgr.y (mgr.34107) 6982 : cluster [DBG] pgmap v8067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:17 smithi115 bash[45495]: cluster 2022-04-22T20:50:16.250283+0000 mgr.y (mgr.34107) 6982 : cluster [DBG] pgmap v8067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:17 smithi115 bash[46934]: cluster 2022-04-22T20:50:16.250283+0000 mgr.y (mgr.34107) 6982 : cluster [DBG] pgmap v8067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:19.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: cluster 2022-04-22T20:50:18.251413+0000 mgr.y (mgr.34107) 6983 : cluster [DBG] pgmap v8068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:19.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: audit 2022-04-22T20:50:18.340455+0000 mon.a (mon.0) 10495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:19.466 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: audit 2022-04-22T20:50:18.407661+0000 mon.a (mon.0) 10496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:19.466 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: audit 2022-04-22T20:50:19.098113+0000 mon.a (mon.0) 10497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:50:19.466 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: audit 2022-04-22T20:50:19.101170+0000 mon.a (mon.0) 10498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:50:19.466 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:19 smithi121 bash[36902]: audit 2022-04-22T20:50:19.103038+0000 mon.a (mon.0) 10499 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:50:19.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: cluster 2022-04-22T20:50:18.251413+0000 mgr.y (mgr.34107) 6983 : cluster [DBG] pgmap v8068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:19.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: audit 2022-04-22T20:50:18.340455+0000 mon.a (mon.0) 10495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:19.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: audit 2022-04-22T20:50:18.407661+0000 mon.a (mon.0) 10496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:19.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: audit 2022-04-22T20:50:19.098113+0000 mon.a (mon.0) 10497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: audit 2022-04-22T20:50:19.101170+0000 mon.a (mon.0) 10498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[45495]: audit 2022-04-22T20:50:19.103038+0000 mon.a (mon.0) 10499 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: cluster 2022-04-22T20:50:18.251413+0000 mgr.y (mgr.34107) 6983 : cluster [DBG] pgmap v8068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: audit 2022-04-22T20:50:18.340455+0000 mon.a (mon.0) 10495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: audit 2022-04-22T20:50:18.407661+0000 mon.a (mon.0) 10496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: audit 2022-04-22T20:50:19.098113+0000 mon.a (mon.0) 10497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: audit 2022-04-22T20:50:19.101170+0000 mon.a (mon.0) 10498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:50:19.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:19 smithi115 bash[46934]: audit 2022-04-22T20:50:19.103038+0000 mon.a (mon.0) 10499 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:50:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:20 smithi121 bash[36902]: audit 2022-04-22T20:50:19.479564+0000 mon.a (mon.0) 10500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:50:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:20 smithi121 bash[36902]: audit 2022-04-22T20:50:19.814600+0000 mon.a (mon.0) 10501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:20 smithi121 bash[36902]: cluster 2022-04-22T20:50:20.252383+0000 mgr.y (mgr.34107) 6984 : cluster [DBG] pgmap v8069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[45495]: audit 2022-04-22T20:50:19.479564+0000 mon.a (mon.0) 10500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:50:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[45495]: audit 2022-04-22T20:50:19.814600+0000 mon.a (mon.0) 10501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[45495]: cluster 2022-04-22T20:50:20.252383+0000 mgr.y (mgr.34107) 6984 : cluster [DBG] pgmap v8069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[46934]: audit 2022-04-22T20:50:19.479564+0000 mon.a (mon.0) 10500 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:50:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[46934]: audit 2022-04-22T20:50:19.814600+0000 mon.a (mon.0) 10501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:20 smithi115 bash[46934]: cluster 2022-04-22T20:50:20.252383+0000 mgr.y (mgr.34107) 6984 : cluster [DBG] pgmap v8069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:22.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:23 smithi121 bash[36902]: cluster 2022-04-22T20:50:22.253417+0000 mgr.y (mgr.34107) 6985 : cluster [DBG] pgmap v8070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:23 smithi115 bash[46934]: cluster 2022-04-22T20:50:22.253417+0000 mgr.y (mgr.34107) 6985 : cluster [DBG] pgmap v8070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:23.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:23 smithi115 bash[45495]: cluster 2022-04-22T20:50:22.253417+0000 mgr.y (mgr.34107) 6985 : cluster [DBG] pgmap v8070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:24.281Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:24 smithi115 bash[46934]: audit 2022-04-22T20:50:23.343117+0000 mon.a (mon.0) 10502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:24 smithi115 bash[46934]: audit 2022-04-22T20:50:23.411037+0000 mon.a (mon.0) 10503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:24 smithi115 bash[45495]: audit 2022-04-22T20:50:23.343117+0000 mon.a (mon.0) 10502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:24.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:24 smithi115 bash[45495]: audit 2022-04-22T20:50:23.411037+0000 mon.a (mon.0) 10503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:24 smithi121 bash[36902]: audit 2022-04-22T20:50:23.343117+0000 mon.a (mon.0) 10502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:24 smithi121 bash[36902]: audit 2022-04-22T20:50:23.411037+0000 mon.a (mon.0) 10503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:25 smithi121 bash[36902]: cluster 2022-04-22T20:50:24.254408+0000 mgr.y (mgr.34107) 6986 : cluster [DBG] pgmap v8071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:25 smithi121 bash[36902]: audit 2022-04-22T20:50:24.817819+0000 mon.a (mon.0) 10504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:25 smithi115 bash[46934]: cluster 2022-04-22T20:50:24.254408+0000 mgr.y (mgr.34107) 6986 : cluster [DBG] pgmap v8071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:25 smithi115 bash[46934]: audit 2022-04-22T20:50:24.817819+0000 mon.a (mon.0) 10504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:25 smithi115 bash[45495]: cluster 2022-04-22T20:50:24.254408+0000 mgr.y (mgr.34107) 6986 : cluster [DBG] pgmap v8071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:25 smithi115 bash[45495]: audit 2022-04-22T20:50:24.817819+0000 mon.a (mon.0) 10504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:27 smithi121 bash[36902]: cluster 2022-04-22T20:50:26.255589+0000 mgr.y (mgr.34107) 6987 : cluster [DBG] pgmap v8072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:27 smithi115 bash[46934]: cluster 2022-04-22T20:50:26.255589+0000 mgr.y (mgr.34107) 6987 : cluster [DBG] pgmap v8072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:27 smithi115 bash[45495]: cluster 2022-04-22T20:50:26.255589+0000 mgr.y (mgr.34107) 6987 : cluster [DBG] pgmap v8072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:29.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:29 smithi121 bash[36902]: cluster 2022-04-22T20:50:28.256685+0000 mgr.y (mgr.34107) 6988 : cluster [DBG] pgmap v8073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:29.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:29 smithi121 bash[36902]: audit 2022-04-22T20:50:28.346171+0000 mon.a (mon.0) 10505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:29.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:29 smithi121 bash[36902]: audit 2022-04-22T20:50:28.414729+0000 mon.a (mon.0) 10506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[46934]: cluster 2022-04-22T20:50:28.256685+0000 mgr.y (mgr.34107) 6988 : cluster [DBG] pgmap v8073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[46934]: audit 2022-04-22T20:50:28.346171+0000 mon.a (mon.0) 10505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:29.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[46934]: audit 2022-04-22T20:50:28.414729+0000 mon.a (mon.0) 10506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[45495]: cluster 2022-04-22T20:50:28.256685+0000 mgr.y (mgr.34107) 6988 : cluster [DBG] pgmap v8073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[45495]: audit 2022-04-22T20:50:28.346171+0000 mon.a (mon.0) 10505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:29.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:29 smithi115 bash[45495]: audit 2022-04-22T20:50:28.414729+0000 mon.a (mon.0) 10506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:30 smithi115 bash[46934]: audit 2022-04-22T20:50:29.820860+0000 mon.a (mon.0) 10507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:30 smithi115 bash[46934]: cluster 2022-04-22T20:50:30.257800+0000 mgr.y (mgr.34107) 6989 : cluster [DBG] pgmap v8074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:30 smithi115 bash[45495]: audit 2022-04-22T20:50:29.820860+0000 mon.a (mon.0) 10507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:30.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:30 smithi115 bash[45495]: cluster 2022-04-22T20:50:30.257800+0000 mgr.y (mgr.34107) 6989 : cluster [DBG] pgmap v8074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:30 smithi121 bash[36902]: audit 2022-04-22T20:50:29.820860+0000 mon.a (mon.0) 10507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:30 smithi121 bash[36902]: cluster 2022-04-22T20:50:30.257800+0000 mgr.y (mgr.34107) 6989 : cluster [DBG] pgmap v8074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:31.533 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:50:31.884 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:50:31.884 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:50:31.885 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:50:31.886 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:50:32.284 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:50:32.284 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:50:32.284 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:50:32.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:50:32.286 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:50:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:32 smithi121 bash[36902]: audit 2022-04-22T20:50:32.278648+0000 mon.c (mon.1) 262 : audit [DBG] from='client.? 172.21.15.115:0/902984927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:32 smithi115 bash[46934]: audit 2022-04-22T20:50:32.278648+0000 mon.c (mon.1) 262 : audit [DBG] from='client.? 172.21.15.115:0/902984927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:32.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:32.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:32 smithi115 bash[45495]: audit 2022-04-22T20:50:32.278648+0000 mon.c (mon.1) 262 : audit [DBG] from='client.? 172.21.15.115:0/902984927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:50:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:33 smithi121 bash[36902]: audit 2022-04-22T20:50:31.512097+0000 mgr.y (mgr.34107) 6990 : audit [DBG] from='client.52573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:33 smithi121 bash[36902]: audit 2022-04-22T20:50:31.858359+0000 mgr.y (mgr.34107) 6991 : audit [DBG] from='client.52579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:33 smithi121 bash[36902]: cluster 2022-04-22T20:50:32.258794+0000 mgr.y (mgr.34107) 6992 : cluster [DBG] pgmap v8075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[45495]: audit 2022-04-22T20:50:31.512097+0000 mgr.y (mgr.34107) 6990 : audit [DBG] from='client.52573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[45495]: audit 2022-04-22T20:50:31.858359+0000 mgr.y (mgr.34107) 6991 : audit [DBG] from='client.52579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[45495]: cluster 2022-04-22T20:50:32.258794+0000 mgr.y (mgr.34107) 6992 : cluster [DBG] pgmap v8075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[46934]: audit 2022-04-22T20:50:31.512097+0000 mgr.y (mgr.34107) 6990 : audit [DBG] from='client.52573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[46934]: audit 2022-04-22T20:50:31.858359+0000 mgr.y (mgr.34107) 6991 : audit [DBG] from='client.52579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:50:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:33 smithi115 bash[46934]: cluster 2022-04-22T20:50:32.258794+0000 mgr.y (mgr.34107) 6992 : cluster [DBG] pgmap v8075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:34.282Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:34 smithi115 bash[46934]: audit 2022-04-22T20:50:33.348857+0000 mon.a (mon.0) 10508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:34 smithi115 bash[46934]: audit 2022-04-22T20:50:33.418362+0000 mon.a (mon.0) 10509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:34 smithi115 bash[45495]: audit 2022-04-22T20:50:33.348857+0000 mon.a (mon.0) 10508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:34.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:34 smithi115 bash[45495]: audit 2022-04-22T20:50:33.418362+0000 mon.a (mon.0) 10509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:34 smithi121 bash[36902]: audit 2022-04-22T20:50:33.348857+0000 mon.a (mon.0) 10508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:34 smithi121 bash[36902]: audit 2022-04-22T20:50:33.418362+0000 mon.a (mon.0) 10509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:35 smithi121 bash[36902]: cluster 2022-04-22T20:50:34.259867+0000 mgr.y (mgr.34107) 6993 : cluster [DBG] pgmap v8076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:35 smithi121 bash[36902]: audit 2022-04-22T20:50:34.824051+0000 mon.a (mon.0) 10510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:35 smithi115 bash[46934]: cluster 2022-04-22T20:50:34.259867+0000 mgr.y (mgr.34107) 6993 : cluster [DBG] pgmap v8076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:35 smithi115 bash[46934]: audit 2022-04-22T20:50:34.824051+0000 mon.a (mon.0) 10510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:35 smithi115 bash[45495]: cluster 2022-04-22T20:50:34.259867+0000 mgr.y (mgr.34107) 6993 : cluster [DBG] pgmap v8076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:35 smithi115 bash[45495]: audit 2022-04-22T20:50:34.824051+0000 mon.a (mon.0) 10510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:37 smithi121 bash[36902]: cluster 2022-04-22T20:50:36.261040+0000 mgr.y (mgr.34107) 6994 : cluster [DBG] pgmap v8077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:37 smithi115 bash[46934]: cluster 2022-04-22T20:50:36.261040+0000 mgr.y (mgr.34107) 6994 : cluster [DBG] pgmap v8077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:37 smithi115 bash[45495]: cluster 2022-04-22T20:50:36.261040+0000 mgr.y (mgr.34107) 6994 : cluster [DBG] pgmap v8077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:39 smithi121 bash[36902]: cluster 2022-04-22T20:50:38.262215+0000 mgr.y (mgr.34107) 6995 : cluster [DBG] pgmap v8078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:39 smithi121 bash[36902]: audit 2022-04-22T20:50:38.354387+0000 mon.a (mon.0) 10511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:39 smithi121 bash[36902]: audit 2022-04-22T20:50:38.422118+0000 mon.a (mon.0) 10512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[46934]: cluster 2022-04-22T20:50:38.262215+0000 mgr.y (mgr.34107) 6995 : cluster [DBG] pgmap v8078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[46934]: audit 2022-04-22T20:50:38.354387+0000 mon.a (mon.0) 10511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[46934]: audit 2022-04-22T20:50:38.422118+0000 mon.a (mon.0) 10512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[45495]: cluster 2022-04-22T20:50:38.262215+0000 mgr.y (mgr.34107) 6995 : cluster [DBG] pgmap v8078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[45495]: audit 2022-04-22T20:50:38.354387+0000 mon.a (mon.0) 10511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T20:50:39.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:39 smithi115 bash[45495]: audit 2022-04-22T20:50:38.422118+0000 mon.a (mon.0) 10512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:40 smithi121 bash[36902]: audit 2022-04-22T20:50:39.827164+0000 mon.a (mon.0) 10513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:40 smithi121 bash[36902]: cluster 2022-04-22T20:50:40.263312+0000 mgr.y (mgr.34107) 6996 : cluster [DBG] pgmap v8079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:40 smithi115 bash[45495]: audit 2022-04-22T20:50:39.827164+0000 mon.a (mon.0) 10513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:40 smithi115 bash[45495]: cluster 2022-04-22T20:50:40.263312+0000 mgr.y (mgr.34107) 6996 : cluster [DBG] pgmap v8079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:40 smithi115 bash[46934]: audit 2022-04-22T20:50:39.827164+0000 mon.a (mon.0) 10513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:40 smithi115 bash[46934]: cluster 2022-04-22T20:50:40.263312+0000 mgr.y (mgr.34107) 6996 : cluster [DBG] pgmap v8079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:42.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:43 smithi121 bash[36902]: cluster 2022-04-22T20:50:42.264384+0000 mgr.y (mgr.34107) 6997 : cluster [DBG] pgmap v8080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:43 smithi115 bash[45495]: cluster 2022-04-22T20:50:42.264384+0000 mgr.y (mgr.34107) 6997 : cluster [DBG] pgmap v8080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:43 smithi115 bash[46934]: cluster 2022-04-22T20:50:42.264384+0000 mgr.y (mgr.34107) 6997 : cluster [DBG] pgmap v8080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:44 smithi115 bash[46934]: audit 2022-04-22T20:50:43.425723+0000 mon.a (mon.0) 10514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:44.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:44.283Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:44.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:44 smithi115 bash[45495]: audit 2022-04-22T20:50:43.425723+0000 mon.a (mon.0) 10514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:44 smithi121 bash[36902]: audit 2022-04-22T20:50:43.425723+0000 mon.a (mon.0) 10514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:45 smithi121 bash[36902]: cluster 2022-04-22T20:50:44.265401+0000 mgr.y (mgr.34107) 6998 : cluster [DBG] pgmap v8081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:45 smithi121 bash[36902]: audit 2022-04-22T20:50:44.830277+0000 mon.a (mon.0) 10515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:45 smithi115 bash[45495]: cluster 2022-04-22T20:50:44.265401+0000 mgr.y (mgr.34107) 6998 : cluster [DBG] pgmap v8081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:45 smithi115 bash[45495]: audit 2022-04-22T20:50:44.830277+0000 mon.a (mon.0) 10515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:45 smithi115 bash[46934]: cluster 2022-04-22T20:50:44.265401+0000 mgr.y (mgr.34107) 6998 : cluster [DBG] pgmap v8081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:45 smithi115 bash[46934]: audit 2022-04-22T20:50:44.830277+0000 mon.a (mon.0) 10515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:47 smithi121 bash[36902]: cluster 2022-04-22T20:50:46.266493+0000 mgr.y (mgr.34107) 6999 : cluster [DBG] pgmap v8082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:47.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:47 smithi115 bash[45495]: cluster 2022-04-22T20:50:46.266493+0000 mgr.y (mgr.34107) 6999 : cluster [DBG] pgmap v8082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:47 smithi115 bash[46934]: cluster 2022-04-22T20:50:46.266493+0000 mgr.y (mgr.34107) 6999 : cluster [DBG] pgmap v8082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:49 smithi121 bash[36902]: cluster 2022-04-22T20:50:48.267653+0000 mgr.y (mgr.34107) 7000 : cluster [DBG] pgmap v8083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:49 smithi121 bash[36902]: audit 2022-04-22T20:50:48.429222+0000 mon.a (mon.0) 10516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:49 smithi115 bash[45495]: cluster 2022-04-22T20:50:48.267653+0000 mgr.y (mgr.34107) 7000 : cluster [DBG] pgmap v8083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:49 smithi115 bash[45495]: audit 2022-04-22T20:50:48.429222+0000 mon.a (mon.0) 10516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:49 smithi115 bash[46934]: cluster 2022-04-22T20:50:48.267653+0000 mgr.y (mgr.34107) 7000 : cluster [DBG] pgmap v8083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:49 smithi115 bash[46934]: audit 2022-04-22T20:50:48.429222+0000 mon.a (mon.0) 10516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:50 smithi121 bash[36902]: audit 2022-04-22T20:50:49.833334+0000 mon.a (mon.0) 10517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:50 smithi121 bash[36902]: cluster 2022-04-22T20:50:50.268752+0000 mgr.y (mgr.34107) 7001 : cluster [DBG] pgmap v8084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:50 smithi115 bash[45495]: audit 2022-04-22T20:50:49.833334+0000 mon.a (mon.0) 10517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:50 smithi115 bash[45495]: cluster 2022-04-22T20:50:50.268752+0000 mgr.y (mgr.34107) 7001 : cluster [DBG] pgmap v8084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:50 smithi115 bash[46934]: audit 2022-04-22T20:50:49.833334+0000 mon.a (mon.0) 10517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:50 smithi115 bash[46934]: cluster 2022-04-22T20:50:50.268752+0000 mgr.y (mgr.34107) 7001 : cluster [DBG] pgmap v8084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:52.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:53 smithi121 bash[36902]: cluster 2022-04-22T20:50:52.269878+0000 mgr.y (mgr.34107) 7002 : cluster [DBG] pgmap v8085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:53 smithi115 bash[45495]: cluster 2022-04-22T20:50:52.269878+0000 mgr.y (mgr.34107) 7002 : cluster [DBG] pgmap v8085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:53 smithi115 bash[46934]: cluster 2022-04-22T20:50:52.269878+0000 mgr.y (mgr.34107) 7002 : cluster [DBG] pgmap v8085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:54 smithi115 bash[45495]: audit 2022-04-22T20:50:53.432947+0000 mon.a (mon.0) 10518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:54.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:54 smithi115 bash[46934]: audit 2022-04-22T20:50:53.432947+0000 mon.a (mon.0) 10518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:54.599 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:50:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:50:54.282Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:50:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:54 smithi121 bash[36902]: audit 2022-04-22T20:50:53.432947+0000 mon.a (mon.0) 10518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:55 smithi121 bash[36902]: cluster 2022-04-22T20:50:54.270938+0000 mgr.y (mgr.34107) 7003 : cluster [DBG] pgmap v8086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:55 smithi121 bash[36902]: audit 2022-04-22T20:50:54.836376+0000 mon.a (mon.0) 10519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:55 smithi115 bash[45495]: cluster 2022-04-22T20:50:54.270938+0000 mgr.y (mgr.34107) 7003 : cluster [DBG] pgmap v8086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:55 smithi115 bash[45495]: audit 2022-04-22T20:50:54.836376+0000 mon.a (mon.0) 10519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:55 smithi115 bash[46934]: cluster 2022-04-22T20:50:54.270938+0000 mgr.y (mgr.34107) 7003 : cluster [DBG] pgmap v8086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:55 smithi115 bash[46934]: audit 2022-04-22T20:50:54.836376+0000 mon.a (mon.0) 10519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:50:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:57 smithi121 bash[36902]: cluster 2022-04-22T20:50:56.272115+0000 mgr.y (mgr.34107) 7004 : cluster [DBG] pgmap v8087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:57 smithi115 bash[46934]: cluster 2022-04-22T20:50:56.272115+0000 mgr.y (mgr.34107) 7004 : cluster [DBG] pgmap v8087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:57.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:57 smithi115 bash[45495]: cluster 2022-04-22T20:50:56.272115+0000 mgr.y (mgr.34107) 7004 : cluster [DBG] pgmap v8087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:59 smithi121 bash[36902]: cluster 2022-04-22T20:50:58.273127+0000 mgr.y (mgr.34107) 7005 : cluster [DBG] pgmap v8088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:50:59 smithi121 bash[36902]: audit 2022-04-22T20:50:58.436946+0000 mon.a (mon.0) 10520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:59 smithi115 bash[45495]: cluster 2022-04-22T20:50:58.273127+0000 mgr.y (mgr.34107) 7005 : cluster [DBG] pgmap v8088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:50:59 smithi115 bash[45495]: audit 2022-04-22T20:50:58.436946+0000 mon.a (mon.0) 10520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:50:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:59 smithi115 bash[46934]: cluster 2022-04-22T20:50:58.273127+0000 mgr.y (mgr.34107) 7005 : cluster [DBG] pgmap v8088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:50:59.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:50:59 smithi115 bash[46934]: audit 2022-04-22T20:50:58.436946+0000 mon.a (mon.0) 10520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:00 smithi121 bash[36902]: audit 2022-04-22T20:50:59.839509+0000 mon.a (mon.0) 10521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:00 smithi121 bash[36902]: cluster 2022-04-22T20:51:00.274211+0000 mgr.y (mgr.34107) 7006 : cluster [DBG] pgmap v8089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:00 smithi115 bash[45495]: audit 2022-04-22T20:50:59.839509+0000 mon.a (mon.0) 10521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:00 smithi115 bash[45495]: cluster 2022-04-22T20:51:00.274211+0000 mgr.y (mgr.34107) 7006 : cluster [DBG] pgmap v8089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:00 smithi115 bash[46934]: audit 2022-04-22T20:50:59.839509+0000 mon.a (mon.0) 10521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:00 smithi115 bash[46934]: cluster 2022-04-22T20:51:00.274211+0000 mgr.y (mgr.34107) 7006 : cluster [DBG] pgmap v8089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:02.647 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:51:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:02.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:03.000 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:51:03.000 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:51:03.001 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:51:03.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:51:03.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:51:03.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:51:03.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:51:03.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:51:03.003 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:51:03.411 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:51:03.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:51:03.411 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:51:03.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:51:03.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:51:03.413 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:51:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:03 smithi121 bash[36902]: cluster 2022-04-22T20:51:02.275337+0000 mgr.y (mgr.34107) 7007 : cluster [DBG] pgmap v8090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:03 smithi115 bash[46934]: cluster 2022-04-22T20:51:02.275337+0000 mgr.y (mgr.34107) 7007 : cluster [DBG] pgmap v8090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:03.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:03 smithi115 bash[45495]: cluster 2022-04-22T20:51:02.275337+0000 mgr.y (mgr.34107) 7007 : cluster [DBG] pgmap v8090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:04.283Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[46934]: audit 2022-04-22T20:51:02.626666+0000 mgr.y (mgr.34107) 7008 : audit [DBG] from='client.52591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[46934]: audit 2022-04-22T20:51:02.974561+0000 mgr.y (mgr.34107) 7009 : audit [DBG] from='client.52597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[46934]: audit 2022-04-22T20:51:03.410188+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.115:0/3641640329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:04.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[46934]: audit 2022-04-22T20:51:03.440581+0000 mon.a (mon.0) 10522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[45495]: audit 2022-04-22T20:51:02.626666+0000 mgr.y (mgr.34107) 7008 : audit [DBG] from='client.52591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[45495]: audit 2022-04-22T20:51:02.974561+0000 mgr.y (mgr.34107) 7009 : audit [DBG] from='client.52597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[45495]: audit 2022-04-22T20:51:03.410188+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.115:0/3641640329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:04.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:04 smithi115 bash[45495]: audit 2022-04-22T20:51:03.440581+0000 mon.a (mon.0) 10522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:04 smithi121 bash[36902]: audit 2022-04-22T20:51:02.626666+0000 mgr.y (mgr.34107) 7008 : audit [DBG] from='client.52591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:04 smithi121 bash[36902]: audit 2022-04-22T20:51:02.974561+0000 mgr.y (mgr.34107) 7009 : audit [DBG] from='client.52597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:04 smithi121 bash[36902]: audit 2022-04-22T20:51:03.410188+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.115:0/3641640329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:04 smithi121 bash[36902]: audit 2022-04-22T20:51:03.440581+0000 mon.a (mon.0) 10522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:05 smithi121 bash[36902]: cluster 2022-04-22T20:51:04.276509+0000 mgr.y (mgr.34107) 7010 : cluster [DBG] pgmap v8091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:05 smithi121 bash[36902]: audit 2022-04-22T20:51:04.842384+0000 mon.a (mon.0) 10523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:05 smithi115 bash[45495]: cluster 2022-04-22T20:51:04.276509+0000 mgr.y (mgr.34107) 7010 : cluster [DBG] pgmap v8091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:05 smithi115 bash[45495]: audit 2022-04-22T20:51:04.842384+0000 mon.a (mon.0) 10523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:05 smithi115 bash[46934]: cluster 2022-04-22T20:51:04.276509+0000 mgr.y (mgr.34107) 7010 : cluster [DBG] pgmap v8091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:05 smithi115 bash[46934]: audit 2022-04-22T20:51:04.842384+0000 mon.a (mon.0) 10523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:07 smithi121 bash[36902]: cluster 2022-04-22T20:51:06.277686+0000 mgr.y (mgr.34107) 7011 : cluster [DBG] pgmap v8092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:07 smithi115 bash[45495]: cluster 2022-04-22T20:51:06.277686+0000 mgr.y (mgr.34107) 7011 : cluster [DBG] pgmap v8092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:07 smithi115 bash[46934]: cluster 2022-04-22T20:51:06.277686+0000 mgr.y (mgr.34107) 7011 : cluster [DBG] pgmap v8092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:09 smithi121 bash[36902]: cluster 2022-04-22T20:51:08.278948+0000 mgr.y (mgr.34107) 7012 : cluster [DBG] pgmap v8093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:09 smithi121 bash[36902]: audit 2022-04-22T20:51:08.444027+0000 mon.a (mon.0) 10524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:09 smithi115 bash[45495]: cluster 2022-04-22T20:51:08.278948+0000 mgr.y (mgr.34107) 7012 : cluster [DBG] pgmap v8093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:09 smithi115 bash[45495]: audit 2022-04-22T20:51:08.444027+0000 mon.a (mon.0) 10524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:09 smithi115 bash[46934]: cluster 2022-04-22T20:51:08.278948+0000 mgr.y (mgr.34107) 7012 : cluster [DBG] pgmap v8093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:09 smithi115 bash[46934]: audit 2022-04-22T20:51:08.444027+0000 mon.a (mon.0) 10524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:10 smithi121 bash[36902]: audit 2022-04-22T20:51:09.845155+0000 mon.a (mon.0) 10525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:10 smithi121 bash[36902]: cluster 2022-04-22T20:51:10.280100+0000 mgr.y (mgr.34107) 7013 : cluster [DBG] pgmap v8094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:10 smithi115 bash[45495]: audit 2022-04-22T20:51:09.845155+0000 mon.a (mon.0) 10525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:10 smithi115 bash[45495]: cluster 2022-04-22T20:51:10.280100+0000 mgr.y (mgr.34107) 7013 : cluster [DBG] pgmap v8094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:10 smithi115 bash[46934]: audit 2022-04-22T20:51:09.845155+0000 mon.a (mon.0) 10525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:10 smithi115 bash[46934]: cluster 2022-04-22T20:51:10.280100+0000 mgr.y (mgr.34107) 7013 : cluster [DBG] pgmap v8094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:12.402Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:13 smithi121 bash[36902]: cluster 2022-04-22T20:51:12.281306+0000 mgr.y (mgr.34107) 7014 : cluster [DBG] pgmap v8095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:13 smithi115 bash[45495]: cluster 2022-04-22T20:51:12.281306+0000 mgr.y (mgr.34107) 7014 : cluster [DBG] pgmap v8095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:13 smithi115 bash[46934]: cluster 2022-04-22T20:51:12.281306+0000 mgr.y (mgr.34107) 7014 : cluster [DBG] pgmap v8095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:14 smithi115 bash[45495]: audit 2022-04-22T20:51:13.447622+0000 mon.a (mon.0) 10526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:14.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:14.285Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:14 smithi115 bash[46934]: audit 2022-04-22T20:51:13.447622+0000 mon.a (mon.0) 10526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:14 smithi121 bash[36902]: audit 2022-04-22T20:51:13.447622+0000 mon.a (mon.0) 10526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:15 smithi121 bash[36902]: cluster 2022-04-22T20:51:14.282701+0000 mgr.y (mgr.34107) 7015 : cluster [DBG] pgmap v8096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:15 smithi121 bash[36902]: audit 2022-04-22T20:51:14.848197+0000 mon.a (mon.0) 10527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:15 smithi115 bash[45495]: cluster 2022-04-22T20:51:14.282701+0000 mgr.y (mgr.34107) 7015 : cluster [DBG] pgmap v8096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:15 smithi115 bash[45495]: audit 2022-04-22T20:51:14.848197+0000 mon.a (mon.0) 10527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:15 smithi115 bash[46934]: cluster 2022-04-22T20:51:14.282701+0000 mgr.y (mgr.34107) 7015 : cluster [DBG] pgmap v8096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:15 smithi115 bash[46934]: audit 2022-04-22T20:51:14.848197+0000 mon.a (mon.0) 10527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:16 smithi121 bash[36902]: audit 2022-04-22T20:51:15.631023+0000 mon.a (mon.0) 10528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:51:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:16 smithi121 bash[36902]: audit 2022-04-22T20:51:15.631357+0000 mon.a (mon.0) 10529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:51:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:16 smithi115 bash[45495]: audit 2022-04-22T20:51:15.631023+0000 mon.a (mon.0) 10528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:51:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:16 smithi115 bash[45495]: audit 2022-04-22T20:51:15.631357+0000 mon.a (mon.0) 10529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:51:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:16 smithi115 bash[46934]: audit 2022-04-22T20:51:15.631023+0000 mon.a (mon.0) 10528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:51:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:16 smithi115 bash[46934]: audit 2022-04-22T20:51:15.631357+0000 mon.a (mon.0) 10529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:51:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:17 smithi121 bash[36902]: cluster 2022-04-22T20:51:16.283884+0000 mgr.y (mgr.34107) 7016 : cluster [DBG] pgmap v8097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:17 smithi115 bash[45495]: cluster 2022-04-22T20:51:16.283884+0000 mgr.y (mgr.34107) 7016 : cluster [DBG] pgmap v8097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:17 smithi115 bash[46934]: cluster 2022-04-22T20:51:16.283884+0000 mgr.y (mgr.34107) 7016 : cluster [DBG] pgmap v8097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:19 smithi121 bash[36902]: cluster 2022-04-22T20:51:18.284959+0000 mgr.y (mgr.34107) 7017 : cluster [DBG] pgmap v8098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:19 smithi121 bash[36902]: audit 2022-04-22T20:51:18.451155+0000 mon.a (mon.0) 10530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:19.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:19 smithi115 bash[46934]: cluster 2022-04-22T20:51:18.284959+0000 mgr.y (mgr.34107) 7017 : cluster [DBG] pgmap v8098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:19.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:19 smithi115 bash[46934]: audit 2022-04-22T20:51:18.451155+0000 mon.a (mon.0) 10530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:19 smithi115 bash[45495]: cluster 2022-04-22T20:51:18.284959+0000 mgr.y (mgr.34107) 7017 : cluster [DBG] pgmap v8098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:19 smithi115 bash[45495]: audit 2022-04-22T20:51:18.451155+0000 mon.a (mon.0) 10530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:20 smithi121 bash[36902]: audit 2022-04-22T20:51:19.488258+0000 mon.a (mon.0) 10531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:51:20.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:20 smithi121 bash[36902]: audit 2022-04-22T20:51:19.491244+0000 mon.a (mon.0) 10532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:51:20.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:20 smithi121 bash[36902]: audit 2022-04-22T20:51:19.492892+0000 mon.a (mon.0) 10533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:51:20.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:20 smithi121 bash[36902]: audit 2022-04-22T20:51:19.851205+0000 mon.a (mon.0) 10534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:20.771 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:20 smithi121 bash[36902]: audit 2022-04-22T20:51:19.859122+0000 mon.a (mon.0) 10535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:51:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[45495]: audit 2022-04-22T20:51:19.488258+0000 mon.a (mon.0) 10531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:51:20.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[45495]: audit 2022-04-22T20:51:19.491244+0000 mon.a (mon.0) 10532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:51:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[45495]: audit 2022-04-22T20:51:19.492892+0000 mon.a (mon.0) 10533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:51:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[45495]: audit 2022-04-22T20:51:19.851205+0000 mon.a (mon.0) 10534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[45495]: audit 2022-04-22T20:51:19.859122+0000 mon.a (mon.0) 10535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:51:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[46934]: audit 2022-04-22T20:51:19.488258+0000 mon.a (mon.0) 10531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:51:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[46934]: audit 2022-04-22T20:51:19.491244+0000 mon.a (mon.0) 10532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:51:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[46934]: audit 2022-04-22T20:51:19.492892+0000 mon.a (mon.0) 10533 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:51:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[46934]: audit 2022-04-22T20:51:19.851205+0000 mon.a (mon.0) 10534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:20.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:20 smithi115 bash[46934]: audit 2022-04-22T20:51:19.859122+0000 mon.a (mon.0) 10535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:51:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:21 smithi121 bash[36902]: cluster 2022-04-22T20:51:20.286054+0000 mgr.y (mgr.34107) 7018 : cluster [DBG] pgmap v8099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:21.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:21 smithi115 bash[45495]: cluster 2022-04-22T20:51:20.286054+0000 mgr.y (mgr.34107) 7018 : cluster [DBG] pgmap v8099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:21.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:21 smithi115 bash[46934]: cluster 2022-04-22T20:51:20.286054+0000 mgr.y (mgr.34107) 7018 : cluster [DBG] pgmap v8099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:22.405Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:23 smithi121 bash[36902]: cluster 2022-04-22T20:51:22.287110+0000 mgr.y (mgr.34107) 7019 : cluster [DBG] pgmap v8100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:23 smithi115 bash[45495]: cluster 2022-04-22T20:51:22.287110+0000 mgr.y (mgr.34107) 7019 : cluster [DBG] pgmap v8100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:23 smithi115 bash[46934]: cluster 2022-04-22T20:51:22.287110+0000 mgr.y (mgr.34107) 7019 : cluster [DBG] pgmap v8100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:24 smithi115 bash[45495]: audit 2022-04-22T20:51:23.454669+0000 mon.a (mon.0) 10536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:24.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:24.283Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:24 smithi115 bash[46934]: audit 2022-04-22T20:51:23.454669+0000 mon.a (mon.0) 10536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:24 smithi121 bash[36902]: audit 2022-04-22T20:51:23.454669+0000 mon.a (mon.0) 10536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:25 smithi121 bash[36902]: cluster 2022-04-22T20:51:24.287821+0000 mgr.y (mgr.34107) 7020 : cluster [DBG] pgmap v8101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:25 smithi121 bash[36902]: audit 2022-04-22T20:51:24.853432+0000 mon.a (mon.0) 10537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:25 smithi115 bash[45495]: cluster 2022-04-22T20:51:24.287821+0000 mgr.y (mgr.34107) 7020 : cluster [DBG] pgmap v8101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:25 smithi115 bash[45495]: audit 2022-04-22T20:51:24.853432+0000 mon.a (mon.0) 10537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:25 smithi115 bash[46934]: cluster 2022-04-22T20:51:24.287821+0000 mgr.y (mgr.34107) 7020 : cluster [DBG] pgmap v8101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:25 smithi115 bash[46934]: audit 2022-04-22T20:51:24.853432+0000 mon.a (mon.0) 10537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:27 smithi121 bash[36902]: cluster 2022-04-22T20:51:26.288783+0000 mgr.y (mgr.34107) 7021 : cluster [DBG] pgmap v8102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:27 smithi115 bash[45495]: cluster 2022-04-22T20:51:26.288783+0000 mgr.y (mgr.34107) 7021 : cluster [DBG] pgmap v8102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:27 smithi115 bash[46934]: cluster 2022-04-22T20:51:26.288783+0000 mgr.y (mgr.34107) 7021 : cluster [DBG] pgmap v8102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:29 smithi121 bash[36902]: cluster 2022-04-22T20:51:28.289924+0000 mgr.y (mgr.34107) 7022 : cluster [DBG] pgmap v8103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:29.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:29 smithi121 bash[36902]: audit 2022-04-22T20:51:28.458525+0000 mon.a (mon.0) 10538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:29 smithi115 bash[45495]: cluster 2022-04-22T20:51:28.289924+0000 mgr.y (mgr.34107) 7022 : cluster [DBG] pgmap v8103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:29 smithi115 bash[45495]: audit 2022-04-22T20:51:28.458525+0000 mon.a (mon.0) 10538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:29 smithi115 bash[46934]: cluster 2022-04-22T20:51:28.289924+0000 mgr.y (mgr.34107) 7022 : cluster [DBG] pgmap v8103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:29 smithi115 bash[46934]: audit 2022-04-22T20:51:28.458525+0000 mon.a (mon.0) 10538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:30 smithi121 bash[36902]: audit 2022-04-22T20:51:29.856750+0000 mon.a (mon.0) 10539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:30 smithi121 bash[36902]: cluster 2022-04-22T20:51:30.290967+0000 mgr.y (mgr.34107) 7023 : cluster [DBG] pgmap v8104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:30 smithi115 bash[45495]: audit 2022-04-22T20:51:29.856750+0000 mon.a (mon.0) 10539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:30 smithi115 bash[45495]: cluster 2022-04-22T20:51:30.290967+0000 mgr.y (mgr.34107) 7023 : cluster [DBG] pgmap v8104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:30 smithi115 bash[46934]: audit 2022-04-22T20:51:29.856750+0000 mon.a (mon.0) 10539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:30 smithi115 bash[46934]: cluster 2022-04-22T20:51:30.290967+0000 mgr.y (mgr.34107) 7023 : cluster [DBG] pgmap v8104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:32.406Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:33 smithi121 bash[36902]: cluster 2022-04-22T20:51:32.292120+0000 mgr.y (mgr.34107) 7024 : cluster [DBG] pgmap v8105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:33.781 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:51:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:33 smithi115 bash[45495]: cluster 2022-04-22T20:51:32.292120+0000 mgr.y (mgr.34107) 7024 : cluster [DBG] pgmap v8105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:33 smithi115 bash[46934]: cluster 2022-04-22T20:51:32.292120+0000 mgr.y (mgr.34107) 7024 : cluster [DBG] pgmap v8105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:51:34.121 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:51:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:51:34.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:51:34.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:51:34.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:51:34.123 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:51:34.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:51:34.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:51:34.535 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:51:34.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:51:34.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:51:34.535 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:51:34.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:34.284Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:34 smithi115 bash[45495]: audit 2022-04-22T20:51:33.462163+0000 mon.a (mon.0) 10540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:34 smithi115 bash[46934]: audit 2022-04-22T20:51:33.462163+0000 mon.a (mon.0) 10540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:34 smithi121 bash[36902]: audit 2022-04-22T20:51:33.462163+0000 mon.a (mon.0) 10540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:35 smithi121 bash[36902]: audit 2022-04-22T20:51:33.759272+0000 mgr.y (mgr.34107) 7025 : audit [DBG] from='client.52609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:35 smithi121 bash[36902]: audit 2022-04-22T20:51:34.094941+0000 mgr.y (mgr.34107) 7026 : audit [DBG] from='client.42684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:35 smithi121 bash[36902]: cluster 2022-04-22T20:51:34.292871+0000 mgr.y (mgr.34107) 7027 : cluster [DBG] pgmap v8106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:35 smithi121 bash[36902]: audit 2022-04-22T20:51:34.527263+0000 mon.a (mon.0) 10541 : audit [DBG] from='client.? 172.21.15.115:0/4263527563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:35 smithi121 bash[36902]: audit 2022-04-22T20:51:34.859881+0000 mon.a (mon.0) 10542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[45495]: audit 2022-04-22T20:51:33.759272+0000 mgr.y (mgr.34107) 7025 : audit [DBG] from='client.52609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[45495]: audit 2022-04-22T20:51:34.094941+0000 mgr.y (mgr.34107) 7026 : audit [DBG] from='client.42684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[45495]: cluster 2022-04-22T20:51:34.292871+0000 mgr.y (mgr.34107) 7027 : cluster [DBG] pgmap v8106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[45495]: audit 2022-04-22T20:51:34.527263+0000 mon.a (mon.0) 10541 : audit [DBG] from='client.? 172.21.15.115:0/4263527563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:35.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[45495]: audit 2022-04-22T20:51:34.859881+0000 mon.a (mon.0) 10542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[46934]: audit 2022-04-22T20:51:33.759272+0000 mgr.y (mgr.34107) 7025 : audit [DBG] from='client.52609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[46934]: audit 2022-04-22T20:51:34.094941+0000 mgr.y (mgr.34107) 7026 : audit [DBG] from='client.42684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:51:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[46934]: cluster 2022-04-22T20:51:34.292871+0000 mgr.y (mgr.34107) 7027 : cluster [DBG] pgmap v8106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[46934]: audit 2022-04-22T20:51:34.527263+0000 mon.a (mon.0) 10541 : audit [DBG] from='client.? 172.21.15.115:0/4263527563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:51:35.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:35 smithi115 bash[46934]: audit 2022-04-22T20:51:34.859881+0000 mon.a (mon.0) 10542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:37 smithi121 bash[36902]: cluster 2022-04-22T20:51:36.293949+0000 mgr.y (mgr.34107) 7028 : cluster [DBG] pgmap v8107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:37 smithi115 bash[45495]: cluster 2022-04-22T20:51:36.293949+0000 mgr.y (mgr.34107) 7028 : cluster [DBG] pgmap v8107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:37 smithi115 bash[46934]: cluster 2022-04-22T20:51:36.293949+0000 mgr.y (mgr.34107) 7028 : cluster [DBG] pgmap v8107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:39 smithi121 bash[36902]: cluster 2022-04-22T20:51:38.294975+0000 mgr.y (mgr.34107) 7029 : cluster [DBG] pgmap v8108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:39 smithi121 bash[36902]: audit 2022-04-22T20:51:38.465853+0000 mon.a (mon.0) 10543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:39 smithi115 bash[45495]: cluster 2022-04-22T20:51:38.294975+0000 mgr.y (mgr.34107) 7029 : cluster [DBG] pgmap v8108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:39 smithi115 bash[45495]: audit 2022-04-22T20:51:38.465853+0000 mon.a (mon.0) 10543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:39 smithi115 bash[46934]: cluster 2022-04-22T20:51:38.294975+0000 mgr.y (mgr.34107) 7029 : cluster [DBG] pgmap v8108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:39 smithi115 bash[46934]: audit 2022-04-22T20:51:38.465853+0000 mon.a (mon.0) 10543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:40 smithi121 bash[36902]: audit 2022-04-22T20:51:39.862816+0000 mon.a (mon.0) 10544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:40 smithi121 bash[36902]: cluster 2022-04-22T20:51:40.296065+0000 mgr.y (mgr.34107) 7030 : cluster [DBG] pgmap v8109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:40 smithi115 bash[45495]: audit 2022-04-22T20:51:39.862816+0000 mon.a (mon.0) 10544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:40 smithi115 bash[45495]: cluster 2022-04-22T20:51:40.296065+0000 mgr.y (mgr.34107) 7030 : cluster [DBG] pgmap v8109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:40 smithi115 bash[46934]: audit 2022-04-22T20:51:39.862816+0000 mon.a (mon.0) 10544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:40 smithi115 bash[46934]: cluster 2022-04-22T20:51:40.296065+0000 mgr.y (mgr.34107) 7030 : cluster [DBG] pgmap v8109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:42.408Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:43 smithi121 bash[36902]: cluster 2022-04-22T20:51:42.297151+0000 mgr.y (mgr.34107) 7031 : cluster [DBG] pgmap v8110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:43 smithi115 bash[45495]: cluster 2022-04-22T20:51:42.297151+0000 mgr.y (mgr.34107) 7031 : cluster [DBG] pgmap v8110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:43 smithi115 bash[46934]: cluster 2022-04-22T20:51:42.297151+0000 mgr.y (mgr.34107) 7031 : cluster [DBG] pgmap v8110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:44.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:44 smithi115 bash[45495]: audit 2022-04-22T20:51:43.469357+0000 mon.a (mon.0) 10545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:44.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:44.285Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:44.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:44 smithi115 bash[46934]: audit 2022-04-22T20:51:43.469357+0000 mon.a (mon.0) 10545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:44 smithi121 bash[36902]: audit 2022-04-22T20:51:43.469357+0000 mon.a (mon.0) 10545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:45 smithi121 bash[36902]: cluster 2022-04-22T20:51:44.298172+0000 mgr.y (mgr.34107) 7032 : cluster [DBG] pgmap v8111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:45 smithi121 bash[36902]: audit 2022-04-22T20:51:44.866117+0000 mon.a (mon.0) 10546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:45.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:45 smithi115 bash[46934]: cluster 2022-04-22T20:51:44.298172+0000 mgr.y (mgr.34107) 7032 : cluster [DBG] pgmap v8111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:45 smithi115 bash[46934]: audit 2022-04-22T20:51:44.866117+0000 mon.a (mon.0) 10546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:45 smithi115 bash[45495]: cluster 2022-04-22T20:51:44.298172+0000 mgr.y (mgr.34107) 7032 : cluster [DBG] pgmap v8111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:45 smithi115 bash[45495]: audit 2022-04-22T20:51:44.866117+0000 mon.a (mon.0) 10546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:47 smithi121 bash[36902]: cluster 2022-04-22T20:51:46.299318+0000 mgr.y (mgr.34107) 7033 : cluster [DBG] pgmap v8112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:47 smithi115 bash[45495]: cluster 2022-04-22T20:51:46.299318+0000 mgr.y (mgr.34107) 7033 : cluster [DBG] pgmap v8112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:47 smithi115 bash[46934]: cluster 2022-04-22T20:51:46.299318+0000 mgr.y (mgr.34107) 7033 : cluster [DBG] pgmap v8112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:49 smithi121 bash[36902]: cluster 2022-04-22T20:51:48.300460+0000 mgr.y (mgr.34107) 7034 : cluster [DBG] pgmap v8113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:49 smithi121 bash[36902]: audit 2022-04-22T20:51:48.472751+0000 mon.a (mon.0) 10547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:49 smithi115 bash[45495]: cluster 2022-04-22T20:51:48.300460+0000 mgr.y (mgr.34107) 7034 : cluster [DBG] pgmap v8113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:49 smithi115 bash[45495]: audit 2022-04-22T20:51:48.472751+0000 mon.a (mon.0) 10547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:49 smithi115 bash[46934]: cluster 2022-04-22T20:51:48.300460+0000 mgr.y (mgr.34107) 7034 : cluster [DBG] pgmap v8113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:49 smithi115 bash[46934]: audit 2022-04-22T20:51:48.472751+0000 mon.a (mon.0) 10547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:50 smithi121 bash[36902]: audit 2022-04-22T20:51:49.869205+0000 mon.a (mon.0) 10548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:50 smithi121 bash[36902]: cluster 2022-04-22T20:51:50.301611+0000 mgr.y (mgr.34107) 7035 : cluster [DBG] pgmap v8114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:50 smithi115 bash[45495]: audit 2022-04-22T20:51:49.869205+0000 mon.a (mon.0) 10548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:50 smithi115 bash[45495]: cluster 2022-04-22T20:51:50.301611+0000 mgr.y (mgr.34107) 7035 : cluster [DBG] pgmap v8114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:50 smithi115 bash[46934]: audit 2022-04-22T20:51:49.869205+0000 mon.a (mon.0) 10548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:50 smithi115 bash[46934]: cluster 2022-04-22T20:51:50.301611+0000 mgr.y (mgr.34107) 7035 : cluster [DBG] pgmap v8114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:52.407Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:53 smithi121 bash[36902]: cluster 2022-04-22T20:51:52.302700+0000 mgr.y (mgr.34107) 7036 : cluster [DBG] pgmap v8115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:53 smithi115 bash[45495]: cluster 2022-04-22T20:51:52.302700+0000 mgr.y (mgr.34107) 7036 : cluster [DBG] pgmap v8115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:53 smithi115 bash[46934]: cluster 2022-04-22T20:51:52.302700+0000 mgr.y (mgr.34107) 7036 : cluster [DBG] pgmap v8115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:54.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:54 smithi115 bash[45495]: audit 2022-04-22T20:51:53.476524+0000 mon.a (mon.0) 10549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:51:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:51:54.285Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:51:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:54 smithi115 bash[46934]: audit 2022-04-22T20:51:53.476524+0000 mon.a (mon.0) 10549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:54 smithi121 bash[36902]: audit 2022-04-22T20:51:53.476524+0000 mon.a (mon.0) 10549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:55 smithi121 bash[36902]: cluster 2022-04-22T20:51:54.303857+0000 mgr.y (mgr.34107) 7037 : cluster [DBG] pgmap v8116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:55 smithi121 bash[36902]: audit 2022-04-22T20:51:54.872561+0000 mon.a (mon.0) 10550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:55 smithi115 bash[45495]: cluster 2022-04-22T20:51:54.303857+0000 mgr.y (mgr.34107) 7037 : cluster [DBG] pgmap v8116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:55 smithi115 bash[45495]: audit 2022-04-22T20:51:54.872561+0000 mon.a (mon.0) 10550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:55 smithi115 bash[46934]: cluster 2022-04-22T20:51:54.303857+0000 mgr.y (mgr.34107) 7037 : cluster [DBG] pgmap v8116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:55.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:55 smithi115 bash[46934]: audit 2022-04-22T20:51:54.872561+0000 mon.a (mon.0) 10550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:51:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:57 smithi121 bash[36902]: cluster 2022-04-22T20:51:56.304880+0000 mgr.y (mgr.34107) 7038 : cluster [DBG] pgmap v8117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:57.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:57 smithi115 bash[45495]: cluster 2022-04-22T20:51:56.304880+0000 mgr.y (mgr.34107) 7038 : cluster [DBG] pgmap v8117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:57 smithi115 bash[46934]: cluster 2022-04-22T20:51:56.304880+0000 mgr.y (mgr.34107) 7038 : cluster [DBG] pgmap v8117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:59 smithi121 bash[36902]: cluster 2022-04-22T20:51:58.306020+0000 mgr.y (mgr.34107) 7039 : cluster [DBG] pgmap v8118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:51:59 smithi121 bash[36902]: audit 2022-04-22T20:51:58.480362+0000 mon.a (mon.0) 10551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:59 smithi115 bash[45495]: cluster 2022-04-22T20:51:58.306020+0000 mgr.y (mgr.34107) 7039 : cluster [DBG] pgmap v8118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:51:59 smithi115 bash[45495]: audit 2022-04-22T20:51:58.480362+0000 mon.a (mon.0) 10551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:51:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:59 smithi115 bash[46934]: cluster 2022-04-22T20:51:58.306020+0000 mgr.y (mgr.34107) 7039 : cluster [DBG] pgmap v8118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:51:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:51:59 smithi115 bash[46934]: audit 2022-04-22T20:51:58.480362+0000 mon.a (mon.0) 10551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:00 smithi121 bash[36902]: audit 2022-04-22T20:51:59.875813+0000 mon.a (mon.0) 10552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:00 smithi121 bash[36902]: cluster 2022-04-22T20:52:00.307067+0000 mgr.y (mgr.34107) 7040 : cluster [DBG] pgmap v8119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:00 smithi115 bash[45495]: audit 2022-04-22T20:51:59.875813+0000 mon.a (mon.0) 10552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:00 smithi115 bash[45495]: cluster 2022-04-22T20:52:00.307067+0000 mgr.y (mgr.34107) 7040 : cluster [DBG] pgmap v8119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:00 smithi115 bash[46934]: audit 2022-04-22T20:51:59.875813+0000 mon.a (mon.0) 10552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:00 smithi115 bash[46934]: cluster 2022-04-22T20:52:00.307067+0000 mgr.y (mgr.34107) 7040 : cluster [DBG] pgmap v8119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:02.409Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:03 smithi121 bash[36902]: cluster 2022-04-22T20:52:02.308199+0000 mgr.y (mgr.34107) 7041 : cluster [DBG] pgmap v8120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:03 smithi115 bash[45495]: cluster 2022-04-22T20:52:02.308199+0000 mgr.y (mgr.34107) 7041 : cluster [DBG] pgmap v8120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:03 smithi115 bash[46934]: cluster 2022-04-22T20:52:02.308199+0000 mgr.y (mgr.34107) 7041 : cluster [DBG] pgmap v8120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:04.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:04 smithi115 bash[45495]: audit 2022-04-22T20:52:03.484054+0000 mon.a (mon.0) 10553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:04.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:04.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:04 smithi115 bash[46934]: audit 2022-04-22T20:52:03.484054+0000 mon.a (mon.0) 10553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:04 smithi121 bash[36902]: audit 2022-04-22T20:52:03.484054+0000 mon.a (mon.0) 10553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:04.907 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:52:05.246 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:52:05.246 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:52:05.246 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:52:05.246 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10m ago 3h 222M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10m ago 3h 10.0M - 1dbe0e931976 f4b044642a96 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:52:05.247 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10m ago 3h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10m ago 3h 640M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:52:05.248 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10m ago 3h 65.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:52:05.657 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:52:05.657 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:52:05.657 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:52:05.658 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:52:05.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:05.659 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:52:05.659 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:52:05.659 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:52:05.659 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:52:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:05 smithi121 bash[36902]: cluster 2022-04-22T20:52:04.309228+0000 mgr.y (mgr.34107) 7042 : cluster [DBG] pgmap v8121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:05 smithi121 bash[36902]: audit 2022-04-22T20:52:04.878926+0000 mon.a (mon.0) 10554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:05 smithi115 bash[45495]: cluster 2022-04-22T20:52:04.309228+0000 mgr.y (mgr.34107) 7042 : cluster [DBG] pgmap v8121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:05 smithi115 bash[45495]: audit 2022-04-22T20:52:04.878926+0000 mon.a (mon.0) 10554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:05 smithi115 bash[46934]: cluster 2022-04-22T20:52:04.309228+0000 mgr.y (mgr.34107) 7042 : cluster [DBG] pgmap v8121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:05 smithi115 bash[46934]: audit 2022-04-22T20:52:04.878926+0000 mon.a (mon.0) 10554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:06 smithi121 bash[36902]: audit 2022-04-22T20:52:04.888151+0000 mgr.y (mgr.34107) 7043 : audit [DBG] from='client.42696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:06 smithi121 bash[36902]: audit 2022-04-22T20:52:05.220536+0000 mgr.y (mgr.34107) 7044 : audit [DBG] from='client.52633 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:06 smithi121 bash[36902]: audit 2022-04-22T20:52:05.651745+0000 mon.c (mon.1) 263 : audit [DBG] from='client.? 172.21.15.115:0/4098730037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[45495]: audit 2022-04-22T20:52:04.888151+0000 mgr.y (mgr.34107) 7043 : audit [DBG] from='client.42696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[45495]: audit 2022-04-22T20:52:05.220536+0000 mgr.y (mgr.34107) 7044 : audit [DBG] from='client.52633 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[45495]: audit 2022-04-22T20:52:05.651745+0000 mon.c (mon.1) 263 : audit [DBG] from='client.? 172.21.15.115:0/4098730037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:06.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[46934]: audit 2022-04-22T20:52:04.888151+0000 mgr.y (mgr.34107) 7043 : audit [DBG] from='client.42696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[46934]: audit 2022-04-22T20:52:05.220536+0000 mgr.y (mgr.34107) 7044 : audit [DBG] from='client.52633 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:06.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:06 smithi115 bash[46934]: audit 2022-04-22T20:52:05.651745+0000 mon.c (mon.1) 263 : audit [DBG] from='client.? 172.21.15.115:0/4098730037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:07 smithi121 bash[36902]: cluster 2022-04-22T20:52:06.310282+0000 mgr.y (mgr.34107) 7045 : cluster [DBG] pgmap v8122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:07 smithi115 bash[45495]: cluster 2022-04-22T20:52:06.310282+0000 mgr.y (mgr.34107) 7045 : cluster [DBG] pgmap v8122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:07.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:07 smithi115 bash[46934]: cluster 2022-04-22T20:52:06.310282+0000 mgr.y (mgr.34107) 7045 : cluster [DBG] pgmap v8122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:09 smithi121 bash[36902]: cluster 2022-04-22T20:52:08.311435+0000 mgr.y (mgr.34107) 7046 : cluster [DBG] pgmap v8123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:09 smithi121 bash[36902]: audit 2022-04-22T20:52:08.488007+0000 mon.a (mon.0) 10555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:09 smithi115 bash[45495]: cluster 2022-04-22T20:52:08.311435+0000 mgr.y (mgr.34107) 7046 : cluster [DBG] pgmap v8123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:09 smithi115 bash[45495]: audit 2022-04-22T20:52:08.488007+0000 mon.a (mon.0) 10555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:09 smithi115 bash[46934]: cluster 2022-04-22T20:52:08.311435+0000 mgr.y (mgr.34107) 7046 : cluster [DBG] pgmap v8123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:09 smithi115 bash[46934]: audit 2022-04-22T20:52:08.488007+0000 mon.a (mon.0) 10555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:10 smithi121 bash[36902]: audit 2022-04-22T20:52:09.881888+0000 mon.a (mon.0) 10556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:10 smithi121 bash[36902]: cluster 2022-04-22T20:52:10.312480+0000 mgr.y (mgr.34107) 7047 : cluster [DBG] pgmap v8124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:10 smithi115 bash[45495]: audit 2022-04-22T20:52:09.881888+0000 mon.a (mon.0) 10556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:10 smithi115 bash[45495]: cluster 2022-04-22T20:52:10.312480+0000 mgr.y (mgr.34107) 7047 : cluster [DBG] pgmap v8124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:10 smithi115 bash[46934]: audit 2022-04-22T20:52:09.881888+0000 mon.a (mon.0) 10556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:10 smithi115 bash[46934]: cluster 2022-04-22T20:52:10.312480+0000 mgr.y (mgr.34107) 7047 : cluster [DBG] pgmap v8124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:12.410Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:13 smithi121 bash[36902]: cluster 2022-04-22T20:52:12.313532+0000 mgr.y (mgr.34107) 7048 : cluster [DBG] pgmap v8125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:13 smithi115 bash[45495]: cluster 2022-04-22T20:52:12.313532+0000 mgr.y (mgr.34107) 7048 : cluster [DBG] pgmap v8125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:13.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:13 smithi115 bash[46934]: cluster 2022-04-22T20:52:12.313532+0000 mgr.y (mgr.34107) 7048 : cluster [DBG] pgmap v8125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:14 smithi121 bash[36902]: audit 2022-04-22T20:52:13.492072+0000 mon.a (mon.0) 10557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:14.287Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:14 smithi115 bash[45495]: audit 2022-04-22T20:52:13.492072+0000 mon.a (mon.0) 10557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:14 smithi115 bash[46934]: audit 2022-04-22T20:52:13.492072+0000 mon.a (mon.0) 10557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:15 smithi121 bash[36902]: cluster 2022-04-22T20:52:14.314649+0000 mgr.y (mgr.34107) 7049 : cluster [DBG] pgmap v8126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:15 smithi121 bash[36902]: audit 2022-04-22T20:52:14.885227+0000 mon.a (mon.0) 10558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:15 smithi115 bash[45495]: cluster 2022-04-22T20:52:14.314649+0000 mgr.y (mgr.34107) 7049 : cluster [DBG] pgmap v8126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:15 smithi115 bash[45495]: audit 2022-04-22T20:52:14.885227+0000 mon.a (mon.0) 10558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:15 smithi115 bash[46934]: cluster 2022-04-22T20:52:14.314649+0000 mgr.y (mgr.34107) 7049 : cluster [DBG] pgmap v8126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:15 smithi115 bash[46934]: audit 2022-04-22T20:52:14.885227+0000 mon.a (mon.0) 10558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:16 smithi121 bash[36902]: audit 2022-04-22T20:52:15.643089+0000 mon.a (mon.0) 10559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:52:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:16 smithi121 bash[36902]: audit 2022-04-22T20:52:15.643371+0000 mon.a (mon.0) 10560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:52:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:16 smithi115 bash[46934]: audit 2022-04-22T20:52:15.643089+0000 mon.a (mon.0) 10559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:52:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:16 smithi115 bash[46934]: audit 2022-04-22T20:52:15.643371+0000 mon.a (mon.0) 10560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:52:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:16 smithi115 bash[45495]: audit 2022-04-22T20:52:15.643089+0000 mon.a (mon.0) 10559 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:52:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:16 smithi115 bash[45495]: audit 2022-04-22T20:52:15.643371+0000 mon.a (mon.0) 10560 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:52:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:17 smithi121 bash[36902]: cluster 2022-04-22T20:52:16.315751+0000 mgr.y (mgr.34107) 7050 : cluster [DBG] pgmap v8127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:17 smithi115 bash[45495]: cluster 2022-04-22T20:52:16.315751+0000 mgr.y (mgr.34107) 7050 : cluster [DBG] pgmap v8127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:17 smithi115 bash[46934]: cluster 2022-04-22T20:52:16.315751+0000 mgr.y (mgr.34107) 7050 : cluster [DBG] pgmap v8127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:19 smithi121 bash[36902]: cluster 2022-04-22T20:52:18.316868+0000 mgr.y (mgr.34107) 7051 : cluster [DBG] pgmap v8128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:19 smithi121 bash[36902]: audit 2022-04-22T20:52:18.495645+0000 mon.a (mon.0) 10561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:19 smithi115 bash[45495]: cluster 2022-04-22T20:52:18.316868+0000 mgr.y (mgr.34107) 7051 : cluster [DBG] pgmap v8128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:19 smithi115 bash[45495]: audit 2022-04-22T20:52:18.495645+0000 mon.a (mon.0) 10561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:19 smithi115 bash[46934]: cluster 2022-04-22T20:52:18.316868+0000 mgr.y (mgr.34107) 7051 : cluster [DBG] pgmap v8128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:19 smithi115 bash[46934]: audit 2022-04-22T20:52:18.495645+0000 mon.a (mon.0) 10561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:20 smithi121 bash[36902]: audit 2022-04-22T20:52:19.867446+0000 mon.a (mon.0) 10562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:52:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:20 smithi121 bash[36902]: audit 2022-04-22T20:52:19.870469+0000 mon.a (mon.0) 10563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:52:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:20 smithi121 bash[36902]: audit 2022-04-22T20:52:19.872314+0000 mon.a (mon.0) 10564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:52:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:20 smithi121 bash[36902]: audit 2022-04-22T20:52:19.887976+0000 mon.a (mon.0) 10565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:20 smithi121 bash[36902]: cluster 2022-04-22T20:52:20.317815+0000 mgr.y (mgr.34107) 7052 : cluster [DBG] pgmap v8129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[45495]: audit 2022-04-22T20:52:19.867446+0000 mon.a (mon.0) 10562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:52:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[45495]: audit 2022-04-22T20:52:19.870469+0000 mon.a (mon.0) 10563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:52:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[45495]: audit 2022-04-22T20:52:19.872314+0000 mon.a (mon.0) 10564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:52:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[45495]: audit 2022-04-22T20:52:19.887976+0000 mon.a (mon.0) 10565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[45495]: cluster 2022-04-22T20:52:20.317815+0000 mgr.y (mgr.34107) 7052 : cluster [DBG] pgmap v8129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[46934]: audit 2022-04-22T20:52:19.867446+0000 mon.a (mon.0) 10562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:52:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[46934]: audit 2022-04-22T20:52:19.870469+0000 mon.a (mon.0) 10563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:52:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[46934]: audit 2022-04-22T20:52:19.872314+0000 mon.a (mon.0) 10564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:52:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[46934]: audit 2022-04-22T20:52:19.887976+0000 mon.a (mon.0) 10565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:21.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:20 smithi115 bash[46934]: cluster 2022-04-22T20:52:20.317815+0000 mgr.y (mgr.34107) 7052 : cluster [DBG] pgmap v8129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:22.410Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:23 smithi121 bash[36902]: cluster 2022-04-22T20:52:22.318882+0000 mgr.y (mgr.34107) 7053 : cluster [DBG] pgmap v8130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:23 smithi115 bash[45495]: cluster 2022-04-22T20:52:22.318882+0000 mgr.y (mgr.34107) 7053 : cluster [DBG] pgmap v8130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:23 smithi115 bash[46934]: cluster 2022-04-22T20:52:22.318882+0000 mgr.y (mgr.34107) 7053 : cluster [DBG] pgmap v8130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:24.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:24 smithi121 bash[36902]: audit 2022-04-22T20:52:23.499265+0000 mon.a (mon.0) 10566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:24.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:24 smithi121 bash[36902]: audit 2022-04-22T20:52:24.307511+0000 mon.a (mon.0) 10567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:24.291Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:24 smithi115 bash[46934]: audit 2022-04-22T20:52:23.499265+0000 mon.a (mon.0) 10566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:24 smithi115 bash[46934]: audit 2022-04-22T20:52:24.307511+0000 mon.a (mon.0) 10567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:24 smithi115 bash[45495]: audit 2022-04-22T20:52:23.499265+0000 mon.a (mon.0) 10566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:24 smithi115 bash[45495]: audit 2022-04-22T20:52:24.307511+0000 mon.a (mon.0) 10567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:25.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:25 smithi121 bash[36902]: cluster 2022-04-22T20:52:24.319894+0000 mgr.y (mgr.34107) 7054 : cluster [DBG] pgmap v8131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:25.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:25 smithi121 bash[36902]: audit 2022-04-22T20:52:24.625043+0000 mon.a (mon.0) 10568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:25.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:25 smithi121 bash[36902]: audit 2022-04-22T20:52:24.890420+0000 mon.a (mon.0) 10569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[45495]: cluster 2022-04-22T20:52:24.319894+0000 mgr.y (mgr.34107) 7054 : cluster [DBG] pgmap v8131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[45495]: audit 2022-04-22T20:52:24.625043+0000 mon.a (mon.0) 10568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[45495]: audit 2022-04-22T20:52:24.890420+0000 mon.a (mon.0) 10569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[46934]: cluster 2022-04-22T20:52:24.319894+0000 mgr.y (mgr.34107) 7054 : cluster [DBG] pgmap v8131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[46934]: audit 2022-04-22T20:52:24.625043+0000 mon.a (mon.0) 10568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:52:25.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:25 smithi115 bash[46934]: audit 2022-04-22T20:52:24.890420+0000 mon.a (mon.0) 10569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:27 smithi121 bash[36902]: cluster 2022-04-22T20:52:26.321005+0000 mgr.y (mgr.34107) 7055 : cluster [DBG] pgmap v8132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:27 smithi115 bash[45495]: cluster 2022-04-22T20:52:26.321005+0000 mgr.y (mgr.34107) 7055 : cluster [DBG] pgmap v8132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:27.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:27 smithi115 bash[46934]: cluster 2022-04-22T20:52:26.321005+0000 mgr.y (mgr.34107) 7055 : cluster [DBG] pgmap v8132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:29 smithi121 bash[36902]: cluster 2022-04-22T20:52:28.322076+0000 mgr.y (mgr.34107) 7056 : cluster [DBG] pgmap v8133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:29.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:29 smithi121 bash[36902]: audit 2022-04-22T20:52:28.503080+0000 mon.a (mon.0) 10570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:29 smithi115 bash[45495]: cluster 2022-04-22T20:52:28.322076+0000 mgr.y (mgr.34107) 7056 : cluster [DBG] pgmap v8133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:29 smithi115 bash[45495]: audit 2022-04-22T20:52:28.503080+0000 mon.a (mon.0) 10570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:29 smithi115 bash[46934]: cluster 2022-04-22T20:52:28.322076+0000 mgr.y (mgr.34107) 7056 : cluster [DBG] pgmap v8133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:29 smithi115 bash[46934]: audit 2022-04-22T20:52:28.503080+0000 mon.a (mon.0) 10570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:30 smithi121 bash[36902]: audit 2022-04-22T20:52:29.893478+0000 mon.a (mon.0) 10571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:30 smithi121 bash[36902]: cluster 2022-04-22T20:52:30.323091+0000 mgr.y (mgr.34107) 7057 : cluster [DBG] pgmap v8134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:30 smithi115 bash[45495]: audit 2022-04-22T20:52:29.893478+0000 mon.a (mon.0) 10571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:30 smithi115 bash[45495]: cluster 2022-04-22T20:52:30.323091+0000 mgr.y (mgr.34107) 7057 : cluster [DBG] pgmap v8134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:30 smithi115 bash[46934]: audit 2022-04-22T20:52:29.893478+0000 mon.a (mon.0) 10571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:30 smithi115 bash[46934]: cluster 2022-04-22T20:52:30.323091+0000 mgr.y (mgr.34107) 7057 : cluster [DBG] pgmap v8134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:32.412Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:33 smithi121 bash[36902]: cluster 2022-04-22T20:52:32.324197+0000 mgr.y (mgr.34107) 7058 : cluster [DBG] pgmap v8135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:33.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:33 smithi115 bash[45495]: cluster 2022-04-22T20:52:32.324197+0000 mgr.y (mgr.34107) 7058 : cluster [DBG] pgmap v8135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:33 smithi115 bash[46934]: cluster 2022-04-22T20:52:32.324197+0000 mgr.y (mgr.34107) 7058 : cluster [DBG] pgmap v8135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:34 smithi121 bash[36902]: audit 2022-04-22T20:52:33.506732+0000 mon.a (mon.0) 10572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:34 smithi115 bash[45495]: audit 2022-04-22T20:52:33.506732+0000 mon.a (mon.0) 10572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:34 smithi115 bash[46934]: audit 2022-04-22T20:52:33.506732+0000 mon.a (mon.0) 10572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:34.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:34.289Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:35 smithi121 bash[36902]: cluster 2022-04-22T20:52:34.325201+0000 mgr.y (mgr.34107) 7059 : cluster [DBG] pgmap v8136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:35 smithi121 bash[36902]: audit 2022-04-22T20:52:34.896669+0000 mon.a (mon.0) 10573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:35 smithi115 bash[46934]: cluster 2022-04-22T20:52:34.325201+0000 mgr.y (mgr.34107) 7059 : cluster [DBG] pgmap v8136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:35 smithi115 bash[46934]: audit 2022-04-22T20:52:34.896669+0000 mon.a (mon.0) 10573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:35 smithi115 bash[45495]: cluster 2022-04-22T20:52:34.325201+0000 mgr.y (mgr.34107) 7059 : cluster [DBG] pgmap v8136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:35 smithi115 bash[45495]: audit 2022-04-22T20:52:34.896669+0000 mon.a (mon.0) 10573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:36.039 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:52:36.377 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:52:36.377 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 12s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 12s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 12s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 12s ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:52:36.378 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 12s ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 12s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 12s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 12s ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 12s ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:52:36.379 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 12s ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:52:36.782 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:52:36.782 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:52:36.783 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:52:36.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:52:36.784 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:52:36.784 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:52:36.784 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:52:36.784 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:52:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:37 smithi121 bash[36902]: audit 2022-04-22T20:52:36.017582+0000 mgr.y (mgr.34107) 7060 : audit [DBG] from='client.42714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:37 smithi121 bash[36902]: cluster 2022-04-22T20:52:36.326283+0000 mgr.y (mgr.34107) 7061 : cluster [DBG] pgmap v8137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:37 smithi121 bash[36902]: audit 2022-04-22T20:52:36.351589+0000 mgr.y (mgr.34107) 7062 : audit [DBG] from='client.42720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:37 smithi121 bash[36902]: audit 2022-04-22T20:52:36.777045+0000 mon.c (mon.1) 264 : audit [DBG] from='client.? 172.21.15.115:0/1621174186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[45495]: audit 2022-04-22T20:52:36.017582+0000 mgr.y (mgr.34107) 7060 : audit [DBG] from='client.42714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[45495]: cluster 2022-04-22T20:52:36.326283+0000 mgr.y (mgr.34107) 7061 : cluster [DBG] pgmap v8137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[45495]: audit 2022-04-22T20:52:36.351589+0000 mgr.y (mgr.34107) 7062 : audit [DBG] from='client.42720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[45495]: audit 2022-04-22T20:52:36.777045+0000 mon.c (mon.1) 264 : audit [DBG] from='client.? 172.21.15.115:0/1621174186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[46934]: audit 2022-04-22T20:52:36.017582+0000 mgr.y (mgr.34107) 7060 : audit [DBG] from='client.42714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[46934]: cluster 2022-04-22T20:52:36.326283+0000 mgr.y (mgr.34107) 7061 : cluster [DBG] pgmap v8137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[46934]: audit 2022-04-22T20:52:36.351589+0000 mgr.y (mgr.34107) 7062 : audit [DBG] from='client.42720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:52:37.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:37 smithi115 bash[46934]: audit 2022-04-22T20:52:36.777045+0000 mon.c (mon.1) 264 : audit [DBG] from='client.? 172.21.15.115:0/1621174186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:52:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:39 smithi121 bash[36902]: cluster 2022-04-22T20:52:38.327393+0000 mgr.y (mgr.34107) 7063 : cluster [DBG] pgmap v8138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:39 smithi121 bash[36902]: audit 2022-04-22T20:52:38.511020+0000 mon.a (mon.0) 10574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:39 smithi115 bash[45495]: cluster 2022-04-22T20:52:38.327393+0000 mgr.y (mgr.34107) 7063 : cluster [DBG] pgmap v8138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:39.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:39 smithi115 bash[45495]: audit 2022-04-22T20:52:38.511020+0000 mon.a (mon.0) 10574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:39 smithi115 bash[46934]: cluster 2022-04-22T20:52:38.327393+0000 mgr.y (mgr.34107) 7063 : cluster [DBG] pgmap v8138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:39 smithi115 bash[46934]: audit 2022-04-22T20:52:38.511020+0000 mon.a (mon.0) 10574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:40 smithi121 bash[36902]: audit 2022-04-22T20:52:39.899856+0000 mon.a (mon.0) 10575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:40 smithi121 bash[36902]: cluster 2022-04-22T20:52:40.328466+0000 mgr.y (mgr.34107) 7064 : cluster [DBG] pgmap v8139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:40 smithi115 bash[45495]: audit 2022-04-22T20:52:39.899856+0000 mon.a (mon.0) 10575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:40 smithi115 bash[45495]: cluster 2022-04-22T20:52:40.328466+0000 mgr.y (mgr.34107) 7064 : cluster [DBG] pgmap v8139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:40 smithi115 bash[46934]: audit 2022-04-22T20:52:39.899856+0000 mon.a (mon.0) 10575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:40 smithi115 bash[46934]: cluster 2022-04-22T20:52:40.328466+0000 mgr.y (mgr.34107) 7064 : cluster [DBG] pgmap v8139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:42.412Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:43 smithi121 bash[36902]: cluster 2022-04-22T20:52:42.329543+0000 mgr.y (mgr.34107) 7065 : cluster [DBG] pgmap v8140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:43 smithi115 bash[45495]: cluster 2022-04-22T20:52:42.329543+0000 mgr.y (mgr.34107) 7065 : cluster [DBG] pgmap v8140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:43.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:43 smithi115 bash[46934]: cluster 2022-04-22T20:52:42.329543+0000 mgr.y (mgr.34107) 7065 : cluster [DBG] pgmap v8140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:44 smithi121 bash[36902]: audit 2022-04-22T20:52:43.514487+0000 mon.a (mon.0) 10576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:44 smithi115 bash[46934]: audit 2022-04-22T20:52:43.514487+0000 mon.a (mon.0) 10576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:44.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:44.290Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:44 smithi115 bash[45495]: audit 2022-04-22T20:52:43.514487+0000 mon.a (mon.0) 10576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:45 smithi121 bash[36902]: cluster 2022-04-22T20:52:44.330590+0000 mgr.y (mgr.34107) 7066 : cluster [DBG] pgmap v8141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:45 smithi121 bash[36902]: audit 2022-04-22T20:52:44.902937+0000 mon.a (mon.0) 10577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:45 smithi115 bash[45495]: cluster 2022-04-22T20:52:44.330590+0000 mgr.y (mgr.34107) 7066 : cluster [DBG] pgmap v8141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:45 smithi115 bash[45495]: audit 2022-04-22T20:52:44.902937+0000 mon.a (mon.0) 10577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:45 smithi115 bash[46934]: cluster 2022-04-22T20:52:44.330590+0000 mgr.y (mgr.34107) 7066 : cluster [DBG] pgmap v8141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:45 smithi115 bash[46934]: audit 2022-04-22T20:52:44.902937+0000 mon.a (mon.0) 10577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:47 smithi121 bash[36902]: cluster 2022-04-22T20:52:46.331744+0000 mgr.y (mgr.34107) 7067 : cluster [DBG] pgmap v8142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:47.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:47 smithi115 bash[45495]: cluster 2022-04-22T20:52:46.331744+0000 mgr.y (mgr.34107) 7067 : cluster [DBG] pgmap v8142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:47 smithi115 bash[46934]: cluster 2022-04-22T20:52:46.331744+0000 mgr.y (mgr.34107) 7067 : cluster [DBG] pgmap v8142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:49 smithi121 bash[36902]: cluster 2022-04-22T20:52:48.332892+0000 mgr.y (mgr.34107) 7068 : cluster [DBG] pgmap v8143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:49 smithi121 bash[36902]: audit 2022-04-22T20:52:48.518129+0000 mon.a (mon.0) 10578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:49 smithi115 bash[46934]: cluster 2022-04-22T20:52:48.332892+0000 mgr.y (mgr.34107) 7068 : cluster [DBG] pgmap v8143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:49 smithi115 bash[46934]: audit 2022-04-22T20:52:48.518129+0000 mon.a (mon.0) 10578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:49 smithi115 bash[45495]: cluster 2022-04-22T20:52:48.332892+0000 mgr.y (mgr.34107) 7068 : cluster [DBG] pgmap v8143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:49.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:49 smithi115 bash[45495]: audit 2022-04-22T20:52:48.518129+0000 mon.a (mon.0) 10578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:50 smithi121 bash[36902]: audit 2022-04-22T20:52:49.906095+0000 mon.a (mon.0) 10579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:50 smithi121 bash[36902]: cluster 2022-04-22T20:52:50.333944+0000 mgr.y (mgr.34107) 7069 : cluster [DBG] pgmap v8144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:50 smithi115 bash[46934]: audit 2022-04-22T20:52:49.906095+0000 mon.a (mon.0) 10579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:50 smithi115 bash[46934]: cluster 2022-04-22T20:52:50.333944+0000 mgr.y (mgr.34107) 7069 : cluster [DBG] pgmap v8144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:50 smithi115 bash[45495]: audit 2022-04-22T20:52:49.906095+0000 mon.a (mon.0) 10579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:50 smithi115 bash[45495]: cluster 2022-04-22T20:52:50.333944+0000 mgr.y (mgr.34107) 7069 : cluster [DBG] pgmap v8144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:52.411Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:53 smithi121 bash[36902]: cluster 2022-04-22T20:52:52.335011+0000 mgr.y (mgr.34107) 7070 : cluster [DBG] pgmap v8145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:53 smithi115 bash[45495]: cluster 2022-04-22T20:52:52.335011+0000 mgr.y (mgr.34107) 7070 : cluster [DBG] pgmap v8145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:53 smithi115 bash[46934]: cluster 2022-04-22T20:52:52.335011+0000 mgr.y (mgr.34107) 7070 : cluster [DBG] pgmap v8145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:54 smithi121 bash[36902]: audit 2022-04-22T20:52:53.521647+0000 mon.a (mon.0) 10580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:54 smithi115 bash[45495]: audit 2022-04-22T20:52:53.521647+0000 mon.a (mon.0) 10580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:54 smithi115 bash[46934]: audit 2022-04-22T20:52:53.521647+0000 mon.a (mon.0) 10580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:54.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:52:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:52:54.291Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:52:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:55 smithi121 bash[36902]: cluster 2022-04-22T20:52:54.336091+0000 mgr.y (mgr.34107) 7071 : cluster [DBG] pgmap v8146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:55 smithi121 bash[36902]: audit 2022-04-22T20:52:54.909285+0000 mon.a (mon.0) 10581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:55 smithi115 bash[45495]: cluster 2022-04-22T20:52:54.336091+0000 mgr.y (mgr.34107) 7071 : cluster [DBG] pgmap v8146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:55 smithi115 bash[45495]: audit 2022-04-22T20:52:54.909285+0000 mon.a (mon.0) 10581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:55 smithi115 bash[46934]: cluster 2022-04-22T20:52:54.336091+0000 mgr.y (mgr.34107) 7071 : cluster [DBG] pgmap v8146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:55 smithi115 bash[46934]: audit 2022-04-22T20:52:54.909285+0000 mon.a (mon.0) 10581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:52:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:57 smithi121 bash[36902]: cluster 2022-04-22T20:52:56.337207+0000 mgr.y (mgr.34107) 7072 : cluster [DBG] pgmap v8147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:57.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:57 smithi115 bash[45495]: cluster 2022-04-22T20:52:56.337207+0000 mgr.y (mgr.34107) 7072 : cluster [DBG] pgmap v8147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:57 smithi115 bash[46934]: cluster 2022-04-22T20:52:56.337207+0000 mgr.y (mgr.34107) 7072 : cluster [DBG] pgmap v8147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:59 smithi121 bash[36902]: cluster 2022-04-22T20:52:58.338322+0000 mgr.y (mgr.34107) 7073 : cluster [DBG] pgmap v8148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:52:59 smithi121 bash[36902]: audit 2022-04-22T20:52:58.525320+0000 mon.a (mon.0) 10582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:59 smithi115 bash[45495]: cluster 2022-04-22T20:52:58.338322+0000 mgr.y (mgr.34107) 7073 : cluster [DBG] pgmap v8148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:52:59 smithi115 bash[45495]: audit 2022-04-22T20:52:58.525320+0000 mon.a (mon.0) 10582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:52:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:59 smithi115 bash[46934]: cluster 2022-04-22T20:52:58.338322+0000 mgr.y (mgr.34107) 7073 : cluster [DBG] pgmap v8148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:52:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:52:59 smithi115 bash[46934]: audit 2022-04-22T20:52:58.525320+0000 mon.a (mon.0) 10582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:00 smithi121 bash[36902]: audit 2022-04-22T20:52:59.912533+0000 mon.a (mon.0) 10583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:00 smithi121 bash[36902]: cluster 2022-04-22T20:53:00.339386+0000 mgr.y (mgr.34107) 7074 : cluster [DBG] pgmap v8149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:00 smithi115 bash[45495]: audit 2022-04-22T20:52:59.912533+0000 mon.a (mon.0) 10583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:00 smithi115 bash[45495]: cluster 2022-04-22T20:53:00.339386+0000 mgr.y (mgr.34107) 7074 : cluster [DBG] pgmap v8149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:00 smithi115 bash[46934]: audit 2022-04-22T20:52:59.912533+0000 mon.a (mon.0) 10583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:00 smithi115 bash[46934]: cluster 2022-04-22T20:53:00.339386+0000 mgr.y (mgr.34107) 7074 : cluster [DBG] pgmap v8149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:02.412Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:03 smithi121 bash[36902]: cluster 2022-04-22T20:53:02.340444+0000 mgr.y (mgr.34107) 7075 : cluster [DBG] pgmap v8150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:03.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:03 smithi115 bash[45495]: cluster 2022-04-22T20:53:02.340444+0000 mgr.y (mgr.34107) 7075 : cluster [DBG] pgmap v8150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:03.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:03 smithi115 bash[46934]: cluster 2022-04-22T20:53:02.340444+0000 mgr.y (mgr.34107) 7075 : cluster [DBG] pgmap v8150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:04.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:04 smithi121 bash[36902]: audit 2022-04-22T20:53:03.528932+0000 mon.a (mon.0) 10584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:04 smithi115 bash[45495]: audit 2022-04-22T20:53:03.528932+0000 mon.a (mon.0) 10584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:04.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:04.291Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:04 smithi115 bash[46934]: audit 2022-04-22T20:53:03.528932+0000 mon.a (mon.0) 10584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:05 smithi121 bash[36902]: cluster 2022-04-22T20:53:04.341505+0000 mgr.y (mgr.34107) 7076 : cluster [DBG] pgmap v8151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:05 smithi121 bash[36902]: audit 2022-04-22T20:53:04.915935+0000 mon.a (mon.0) 10585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:05 smithi115 bash[45495]: cluster 2022-04-22T20:53:04.341505+0000 mgr.y (mgr.34107) 7076 : cluster [DBG] pgmap v8151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:05 smithi115 bash[45495]: audit 2022-04-22T20:53:04.915935+0000 mon.a (mon.0) 10585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:05 smithi115 bash[46934]: cluster 2022-04-22T20:53:04.341505+0000 mgr.y (mgr.34107) 7076 : cluster [DBG] pgmap v8151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:05 smithi115 bash[46934]: audit 2022-04-22T20:53:04.915935+0000 mon.a (mon.0) 10585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:07.161 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:53:07.496 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 43s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 43s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 43s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 347M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:53:07.497 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 43s ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 226M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 43s ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 707M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 532M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 491M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 43s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:53:07.498 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 43s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:53:07.499 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 43s ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:53:07.499 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 43s ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:53:07.499 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 43s ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:53:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:07 smithi121 bash[36902]: cluster 2022-04-22T20:53:06.342597+0000 mgr.y (mgr.34107) 7077 : cluster [DBG] pgmap v8152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:07 smithi115 bash[45495]: cluster 2022-04-22T20:53:06.342597+0000 mgr.y (mgr.34107) 7077 : cluster [DBG] pgmap v8152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:07 smithi115 bash[46934]: cluster 2022-04-22T20:53:06.342597+0000 mgr.y (mgr.34107) 7077 : cluster [DBG] pgmap v8152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:53:07.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:53:07.908 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:53:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:08 smithi121 bash[36902]: audit 2022-04-22T20:53:07.136553+0000 mgr.y (mgr.34107) 7078 : audit [DBG] from='client.42732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:08 smithi121 bash[36902]: audit 2022-04-22T20:53:07.472891+0000 mgr.y (mgr.34107) 7079 : audit [DBG] from='client.42738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:08 smithi121 bash[36902]: audit 2022-04-22T20:53:07.901305+0000 mon.c (mon.1) 265 : audit [DBG] from='client.? 172.21.15.115:0/542261768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[45495]: audit 2022-04-22T20:53:07.136553+0000 mgr.y (mgr.34107) 7078 : audit [DBG] from='client.42732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[45495]: audit 2022-04-22T20:53:07.472891+0000 mgr.y (mgr.34107) 7079 : audit [DBG] from='client.42738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[45495]: audit 2022-04-22T20:53:07.901305+0000 mon.c (mon.1) 265 : audit [DBG] from='client.? 172.21.15.115:0/542261768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[46934]: audit 2022-04-22T20:53:07.136553+0000 mgr.y (mgr.34107) 7078 : audit [DBG] from='client.42732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[46934]: audit 2022-04-22T20:53:07.472891+0000 mgr.y (mgr.34107) 7079 : audit [DBG] from='client.42738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:08 smithi115 bash[46934]: audit 2022-04-22T20:53:07.901305+0000 mon.c (mon.1) 265 : audit [DBG] from='client.? 172.21.15.115:0/542261768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:09 smithi121 bash[36902]: cluster 2022-04-22T20:53:08.343802+0000 mgr.y (mgr.34107) 7080 : cluster [DBG] pgmap v8153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:09 smithi121 bash[36902]: audit 2022-04-22T20:53:08.532457+0000 mon.a (mon.0) 10586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:09 smithi115 bash[45495]: cluster 2022-04-22T20:53:08.343802+0000 mgr.y (mgr.34107) 7080 : cluster [DBG] pgmap v8153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:09 smithi115 bash[45495]: audit 2022-04-22T20:53:08.532457+0000 mon.a (mon.0) 10586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:09 smithi115 bash[46934]: cluster 2022-04-22T20:53:08.343802+0000 mgr.y (mgr.34107) 7080 : cluster [DBG] pgmap v8153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:09 smithi115 bash[46934]: audit 2022-04-22T20:53:08.532457+0000 mon.a (mon.0) 10586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:10 smithi121 bash[36902]: audit 2022-04-22T20:53:09.918671+0000 mon.a (mon.0) 10587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:10 smithi121 bash[36902]: cluster 2022-04-22T20:53:10.344838+0000 mgr.y (mgr.34107) 7081 : cluster [DBG] pgmap v8154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:10 smithi115 bash[46934]: audit 2022-04-22T20:53:09.918671+0000 mon.a (mon.0) 10587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:10 smithi115 bash[46934]: cluster 2022-04-22T20:53:10.344838+0000 mgr.y (mgr.34107) 7081 : cluster [DBG] pgmap v8154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:10 smithi115 bash[45495]: audit 2022-04-22T20:53:09.918671+0000 mon.a (mon.0) 10587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:10 smithi115 bash[45495]: cluster 2022-04-22T20:53:10.344838+0000 mgr.y (mgr.34107) 7081 : cluster [DBG] pgmap v8154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:12.411Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:13 smithi121 bash[36902]: cluster 2022-04-22T20:53:12.345914+0000 mgr.y (mgr.34107) 7082 : cluster [DBG] pgmap v8155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:13 smithi115 bash[45495]: cluster 2022-04-22T20:53:12.345914+0000 mgr.y (mgr.34107) 7082 : cluster [DBG] pgmap v8155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:13 smithi115 bash[46934]: cluster 2022-04-22T20:53:12.345914+0000 mgr.y (mgr.34107) 7082 : cluster [DBG] pgmap v8155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:14 smithi121 bash[36902]: audit 2022-04-22T20:53:13.536122+0000 mon.a (mon.0) 10588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:14 smithi115 bash[45495]: audit 2022-04-22T20:53:13.536122+0000 mon.a (mon.0) 10588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:14 smithi115 bash[46934]: audit 2022-04-22T20:53:13.536122+0000 mon.a (mon.0) 10588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:14.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:14.293Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:15 smithi121 bash[36902]: cluster 2022-04-22T20:53:14.346963+0000 mgr.y (mgr.34107) 7083 : cluster [DBG] pgmap v8156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:15 smithi121 bash[36902]: audit 2022-04-22T20:53:14.921819+0000 mon.a (mon.0) 10589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:15 smithi115 bash[45495]: cluster 2022-04-22T20:53:14.346963+0000 mgr.y (mgr.34107) 7083 : cluster [DBG] pgmap v8156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:15 smithi115 bash[45495]: audit 2022-04-22T20:53:14.921819+0000 mon.a (mon.0) 10589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:15 smithi115 bash[46934]: cluster 2022-04-22T20:53:14.346963+0000 mgr.y (mgr.34107) 7083 : cluster [DBG] pgmap v8156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:15 smithi115 bash[46934]: audit 2022-04-22T20:53:14.921819+0000 mon.a (mon.0) 10589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:16 smithi121 bash[36902]: audit 2022-04-22T20:53:15.654781+0000 mon.a (mon.0) 10590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:53:16.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:16 smithi121 bash[36902]: audit 2022-04-22T20:53:15.655201+0000 mon.a (mon.0) 10591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:53:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:16 smithi115 bash[45495]: audit 2022-04-22T20:53:15.654781+0000 mon.a (mon.0) 10590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:53:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:16 smithi115 bash[45495]: audit 2022-04-22T20:53:15.655201+0000 mon.a (mon.0) 10591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:53:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:16 smithi115 bash[46934]: audit 2022-04-22T20:53:15.654781+0000 mon.a (mon.0) 10590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:53:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:16 smithi115 bash[46934]: audit 2022-04-22T20:53:15.655201+0000 mon.a (mon.0) 10591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:53:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:17 smithi121 bash[36902]: cluster 2022-04-22T20:53:16.348041+0000 mgr.y (mgr.34107) 7084 : cluster [DBG] pgmap v8157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:17 smithi115 bash[45495]: cluster 2022-04-22T20:53:16.348041+0000 mgr.y (mgr.34107) 7084 : cluster [DBG] pgmap v8157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:17 smithi115 bash[46934]: cluster 2022-04-22T20:53:16.348041+0000 mgr.y (mgr.34107) 7084 : cluster [DBG] pgmap v8157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:19 smithi121 bash[36902]: cluster 2022-04-22T20:53:18.349149+0000 mgr.y (mgr.34107) 7085 : cluster [DBG] pgmap v8158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:19 smithi121 bash[36902]: audit 2022-04-22T20:53:18.539828+0000 mon.a (mon.0) 10592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:19 smithi115 bash[45495]: cluster 2022-04-22T20:53:18.349149+0000 mgr.y (mgr.34107) 7085 : cluster [DBG] pgmap v8158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:19 smithi115 bash[45495]: audit 2022-04-22T20:53:18.539828+0000 mon.a (mon.0) 10592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:19 smithi115 bash[46934]: cluster 2022-04-22T20:53:18.349149+0000 mgr.y (mgr.34107) 7085 : cluster [DBG] pgmap v8158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:19 smithi115 bash[46934]: audit 2022-04-22T20:53:18.539828+0000 mon.a (mon.0) 10592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:20 smithi121 bash[36902]: audit 2022-04-22T20:53:19.925234+0000 mon.a (mon.0) 10593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:20 smithi121 bash[36902]: cluster 2022-04-22T20:53:20.350142+0000 mgr.y (mgr.34107) 7086 : cluster [DBG] pgmap v8159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:20 smithi115 bash[45495]: audit 2022-04-22T20:53:19.925234+0000 mon.a (mon.0) 10593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:21.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:20 smithi115 bash[45495]: cluster 2022-04-22T20:53:20.350142+0000 mgr.y (mgr.34107) 7086 : cluster [DBG] pgmap v8159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:20 smithi115 bash[46934]: audit 2022-04-22T20:53:19.925234+0000 mon.a (mon.0) 10593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:20 smithi115 bash[46934]: cluster 2022-04-22T20:53:20.350142+0000 mgr.y (mgr.34107) 7086 : cluster [DBG] pgmap v8159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:22.414Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:23 smithi121 bash[36902]: cluster 2022-04-22T20:53:22.351174+0000 mgr.y (mgr.34107) 7087 : cluster [DBG] pgmap v8160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:23 smithi115 bash[45495]: cluster 2022-04-22T20:53:22.351174+0000 mgr.y (mgr.34107) 7087 : cluster [DBG] pgmap v8160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:23 smithi115 bash[46934]: cluster 2022-04-22T20:53:22.351174+0000 mgr.y (mgr.34107) 7087 : cluster [DBG] pgmap v8160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:24.661 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:24 smithi115 bash[46934]: audit 2022-04-22T20:53:23.543403+0000 mon.a (mon.0) 10594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:24.661 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:24.291Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:24.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:24 smithi115 bash[45495]: audit 2022-04-22T20:53:23.543403+0000 mon.a (mon.0) 10594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:24.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:24 smithi121 bash[36902]: audit 2022-04-22T20:53:23.543403+0000 mon.a (mon.0) 10594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: cluster 2022-04-22T20:53:24.352275+0000 mgr.y (mgr.34107) 7088 : cluster [DBG] pgmap v8161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.632820+0000 mon.a (mon.0) 10595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:53:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.635840+0000 mon.a (mon.0) 10596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:53:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.637631+0000 mon.a (mon.0) 10597 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:53:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.928179+0000 mon.a (mon.0) 10598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.975132+0000 mon.a (mon.0) 10599 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:25 smithi121 bash[36902]: audit 2022-04-22T20:53:24.982136+0000 mon.a (mon.0) 10600 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: cluster 2022-04-22T20:53:24.352275+0000 mgr.y (mgr.34107) 7088 : cluster [DBG] pgmap v8161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.632820+0000 mon.a (mon.0) 10595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:53:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.635840+0000 mon.a (mon.0) 10596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:53:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.637631+0000 mon.a (mon.0) 10597 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:53:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.928179+0000 mon.a (mon.0) 10598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.975132+0000 mon.a (mon.0) 10599 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:25.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[45495]: audit 2022-04-22T20:53:24.982136+0000 mon.a (mon.0) 10600 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: cluster 2022-04-22T20:53:24.352275+0000 mgr.y (mgr.34107) 7088 : cluster [DBG] pgmap v8161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.632820+0000 mon.a (mon.0) 10595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.635840+0000 mon.a (mon.0) 10596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.637631+0000 mon.a (mon.0) 10597 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:53:25.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.928179+0000 mon.a (mon.0) 10598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.975132+0000 mon.a (mon.0) 10599 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:25.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:25 smithi115 bash[46934]: audit 2022-04-22T20:53:24.982136+0000 mon.a (mon.0) 10600 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:27 smithi121 bash[36902]: cluster 2022-04-22T20:53:26.353394+0000 mgr.y (mgr.34107) 7089 : cluster [DBG] pgmap v8162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:27 smithi115 bash[45495]: cluster 2022-04-22T20:53:26.353394+0000 mgr.y (mgr.34107) 7089 : cluster [DBG] pgmap v8162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:27 smithi115 bash[46934]: cluster 2022-04-22T20:53:26.353394+0000 mgr.y (mgr.34107) 7089 : cluster [DBG] pgmap v8162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:29 smithi121 bash[36902]: cluster 2022-04-22T20:53:28.354497+0000 mgr.y (mgr.34107) 7090 : cluster [DBG] pgmap v8163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:29 smithi121 bash[36902]: audit 2022-04-22T20:53:28.547294+0000 mon.a (mon.0) 10601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:29 smithi121 bash[36902]: audit 2022-04-22T20:53:29.174395+0000 mon.a (mon.0) 10602 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[45495]: cluster 2022-04-22T20:53:28.354497+0000 mgr.y (mgr.34107) 7090 : cluster [DBG] pgmap v8163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[45495]: audit 2022-04-22T20:53:28.547294+0000 mon.a (mon.0) 10601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:29.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[45495]: audit 2022-04-22T20:53:29.174395+0000 mon.a (mon.0) 10602 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[46934]: cluster 2022-04-22T20:53:28.354497+0000 mgr.y (mgr.34107) 7090 : cluster [DBG] pgmap v8163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[46934]: audit 2022-04-22T20:53:28.547294+0000 mon.a (mon.0) 10601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:29 smithi115 bash[46934]: audit 2022-04-22T20:53:29.174395+0000 mon.a (mon.0) 10602 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[45495]: audit 2022-04-22T20:53:29.524734+0000 mon.a (mon.0) 10603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[45495]: audit 2022-04-22T20:53:29.930930+0000 mon.a (mon.0) 10604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[45495]: cluster 2022-04-22T20:53:30.355546+0000 mgr.y (mgr.34107) 7091 : cluster [DBG] pgmap v8164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[46934]: audit 2022-04-22T20:53:29.524734+0000 mon.a (mon.0) 10603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[46934]: audit 2022-04-22T20:53:29.930930+0000 mon.a (mon.0) 10604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:30 smithi115 bash[46934]: cluster 2022-04-22T20:53:30.355546+0000 mgr.y (mgr.34107) 7091 : cluster [DBG] pgmap v8164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:30 smithi121 bash[36902]: audit 2022-04-22T20:53:29.524734+0000 mon.a (mon.0) 10603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:53:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:30 smithi121 bash[36902]: audit 2022-04-22T20:53:29.930930+0000 mon.a (mon.0) 10604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:30 smithi121 bash[36902]: cluster 2022-04-22T20:53:30.355546+0000 mgr.y (mgr.34107) 7091 : cluster [DBG] pgmap v8164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:32.416Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:33 smithi121 bash[36902]: cluster 2022-04-22T20:53:32.356599+0000 mgr.y (mgr.34107) 7092 : cluster [DBG] pgmap v8165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:33 smithi115 bash[45495]: cluster 2022-04-22T20:53:32.356599+0000 mgr.y (mgr.34107) 7092 : cluster [DBG] pgmap v8165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:33 smithi115 bash[46934]: cluster 2022-04-22T20:53:32.356599+0000 mgr.y (mgr.34107) 7092 : cluster [DBG] pgmap v8165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:34 smithi121 bash[36902]: audit 2022-04-22T20:53:33.551035+0000 mon.a (mon.0) 10605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:34 smithi115 bash[45495]: audit 2022-04-22T20:53:33.551035+0000 mon.a (mon.0) 10605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:34.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:34.294Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:34 smithi115 bash[46934]: audit 2022-04-22T20:53:33.551035+0000 mon.a (mon.0) 10605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:35 smithi121 bash[36902]: cluster 2022-04-22T20:53:34.357656+0000 mgr.y (mgr.34107) 7093 : cluster [DBG] pgmap v8166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:35 smithi121 bash[36902]: audit 2022-04-22T20:53:34.933994+0000 mon.a (mon.0) 10606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:35 smithi115 bash[45495]: cluster 2022-04-22T20:53:34.357656+0000 mgr.y (mgr.34107) 7093 : cluster [DBG] pgmap v8166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:35 smithi115 bash[45495]: audit 2022-04-22T20:53:34.933994+0000 mon.a (mon.0) 10606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:35 smithi115 bash[46934]: cluster 2022-04-22T20:53:34.357656+0000 mgr.y (mgr.34107) 7093 : cluster [DBG] pgmap v8166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:35 smithi115 bash[46934]: audit 2022-04-22T20:53:34.933994+0000 mon.a (mon.0) 10606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:37 smithi121 bash[36902]: cluster 2022-04-22T20:53:36.358732+0000 mgr.y (mgr.34107) 7094 : cluster [DBG] pgmap v8167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:37.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:37 smithi115 bash[45495]: cluster 2022-04-22T20:53:36.358732+0000 mgr.y (mgr.34107) 7094 : cluster [DBG] pgmap v8167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:37 smithi115 bash[46934]: cluster 2022-04-22T20:53:36.358732+0000 mgr.y (mgr.34107) 7094 : cluster [DBG] pgmap v8167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:38.282 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:53:38.633 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:53:38.634 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9s ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:53:38.634 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 74s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:53:38.634 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:53:38.634 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 74s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:53:38.634 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 74s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9s ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 74s ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9s ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 74s ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:53:38.635 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9s ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9s ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9s ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 74s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 74s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 74s ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 74s ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:53:38.636 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 74s ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:53:39.033 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:53:39.034 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:53:39.035 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:53:39.035 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:53:39.035 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:53:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:39 smithi121 bash[36902]: audit 2022-04-22T20:53:38.261539+0000 mgr.y (mgr.34107) 7095 : audit [DBG] from='client.42750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:39 smithi121 bash[36902]: cluster 2022-04-22T20:53:38.359867+0000 mgr.y (mgr.34107) 7096 : cluster [DBG] pgmap v8168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:39 smithi121 bash[36902]: audit 2022-04-22T20:53:38.554128+0000 mon.a (mon.0) 10607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:39 smithi121 bash[36902]: audit 2022-04-22T20:53:39.031620+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.115:0/2199541301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:39.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[45495]: audit 2022-04-22T20:53:38.261539+0000 mgr.y (mgr.34107) 7095 : audit [DBG] from='client.42750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[45495]: cluster 2022-04-22T20:53:38.359867+0000 mgr.y (mgr.34107) 7096 : cluster [DBG] pgmap v8168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:39.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[45495]: audit 2022-04-22T20:53:38.554128+0000 mon.a (mon.0) 10607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:39.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[45495]: audit 2022-04-22T20:53:39.031620+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.115:0/2199541301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[46934]: audit 2022-04-22T20:53:38.261539+0000 mgr.y (mgr.34107) 7095 : audit [DBG] from='client.42750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:39.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[46934]: cluster 2022-04-22T20:53:38.359867+0000 mgr.y (mgr.34107) 7096 : cluster [DBG] pgmap v8168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[46934]: audit 2022-04-22T20:53:38.554128+0000 mon.a (mon.0) 10607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:39.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:39 smithi115 bash[46934]: audit 2022-04-22T20:53:39.031620+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.115:0/2199541301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:53:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:40 smithi121 bash[36902]: audit 2022-04-22T20:53:38.607679+0000 mgr.y (mgr.34107) 7097 : audit [DBG] from='client.52687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:40 smithi121 bash[36902]: audit 2022-04-22T20:53:39.936902+0000 mon.a (mon.0) 10608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:40 smithi115 bash[45495]: audit 2022-04-22T20:53:38.607679+0000 mgr.y (mgr.34107) 7097 : audit [DBG] from='client.52687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:40 smithi115 bash[45495]: audit 2022-04-22T20:53:39.936902+0000 mon.a (mon.0) 10608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:40 smithi115 bash[46934]: audit 2022-04-22T20:53:38.607679+0000 mgr.y (mgr.34107) 7097 : audit [DBG] from='client.52687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:53:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:40 smithi115 bash[46934]: audit 2022-04-22T20:53:39.936902+0000 mon.a (mon.0) 10608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:41 smithi121 bash[36902]: cluster 2022-04-22T20:53:40.360907+0000 mgr.y (mgr.34107) 7098 : cluster [DBG] pgmap v8169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:41 smithi115 bash[45495]: cluster 2022-04-22T20:53:40.360907+0000 mgr.y (mgr.34107) 7098 : cluster [DBG] pgmap v8169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:41.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:41 smithi115 bash[46934]: cluster 2022-04-22T20:53:40.360907+0000 mgr.y (mgr.34107) 7098 : cluster [DBG] pgmap v8169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:42.413Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:43 smithi121 bash[36902]: cluster 2022-04-22T20:53:42.361994+0000 mgr.y (mgr.34107) 7099 : cluster [DBG] pgmap v8170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:43 smithi115 bash[46934]: cluster 2022-04-22T20:53:42.361994+0000 mgr.y (mgr.34107) 7099 : cluster [DBG] pgmap v8170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:43 smithi115 bash[45495]: cluster 2022-04-22T20:53:42.361994+0000 mgr.y (mgr.34107) 7099 : cluster [DBG] pgmap v8170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:44 smithi121 bash[36902]: audit 2022-04-22T20:53:43.557308+0000 mon.a (mon.0) 10609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:44 smithi115 bash[46934]: audit 2022-04-22T20:53:43.557308+0000 mon.a (mon.0) 10609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:44.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:44.295Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:44 smithi115 bash[45495]: audit 2022-04-22T20:53:43.557308+0000 mon.a (mon.0) 10609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:45 smithi121 bash[36902]: cluster 2022-04-22T20:53:44.363070+0000 mgr.y (mgr.34107) 7100 : cluster [DBG] pgmap v8171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:45 smithi121 bash[36902]: audit 2022-04-22T20:53:44.940037+0000 mon.a (mon.0) 10610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:45 smithi115 bash[45495]: cluster 2022-04-22T20:53:44.363070+0000 mgr.y (mgr.34107) 7100 : cluster [DBG] pgmap v8171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:45 smithi115 bash[45495]: audit 2022-04-22T20:53:44.940037+0000 mon.a (mon.0) 10610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:45 smithi115 bash[46934]: cluster 2022-04-22T20:53:44.363070+0000 mgr.y (mgr.34107) 7100 : cluster [DBG] pgmap v8171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:45 smithi115 bash[46934]: audit 2022-04-22T20:53:44.940037+0000 mon.a (mon.0) 10610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:47 smithi121 bash[36902]: cluster 2022-04-22T20:53:46.364211+0000 mgr.y (mgr.34107) 7101 : cluster [DBG] pgmap v8172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:47 smithi115 bash[45495]: cluster 2022-04-22T20:53:46.364211+0000 mgr.y (mgr.34107) 7101 : cluster [DBG] pgmap v8172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:47 smithi115 bash[46934]: cluster 2022-04-22T20:53:46.364211+0000 mgr.y (mgr.34107) 7101 : cluster [DBG] pgmap v8172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:49 smithi121 bash[36902]: cluster 2022-04-22T20:53:48.365302+0000 mgr.y (mgr.34107) 7102 : cluster [DBG] pgmap v8173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:49 smithi121 bash[36902]: audit 2022-04-22T20:53:48.560894+0000 mon.a (mon.0) 10611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:49 smithi115 bash[45495]: cluster 2022-04-22T20:53:48.365302+0000 mgr.y (mgr.34107) 7102 : cluster [DBG] pgmap v8173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:49 smithi115 bash[45495]: audit 2022-04-22T20:53:48.560894+0000 mon.a (mon.0) 10611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:49 smithi115 bash[46934]: cluster 2022-04-22T20:53:48.365302+0000 mgr.y (mgr.34107) 7102 : cluster [DBG] pgmap v8173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:49 smithi115 bash[46934]: audit 2022-04-22T20:53:48.560894+0000 mon.a (mon.0) 10611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:50 smithi121 bash[36902]: audit 2022-04-22T20:53:49.943056+0000 mon.a (mon.0) 10612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:50 smithi121 bash[36902]: cluster 2022-04-22T20:53:50.366301+0000 mgr.y (mgr.34107) 7103 : cluster [DBG] pgmap v8174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:50 smithi115 bash[45495]: audit 2022-04-22T20:53:49.943056+0000 mon.a (mon.0) 10612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:50 smithi115 bash[45495]: cluster 2022-04-22T20:53:50.366301+0000 mgr.y (mgr.34107) 7103 : cluster [DBG] pgmap v8174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:50 smithi115 bash[46934]: audit 2022-04-22T20:53:49.943056+0000 mon.a (mon.0) 10612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:50 smithi115 bash[46934]: cluster 2022-04-22T20:53:50.366301+0000 mgr.y (mgr.34107) 7103 : cluster [DBG] pgmap v8174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:52.414Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:53 smithi121 bash[36902]: cluster 2022-04-22T20:53:52.367289+0000 mgr.y (mgr.34107) 7104 : cluster [DBG] pgmap v8175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:53.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:53 smithi115 bash[45495]: cluster 2022-04-22T20:53:52.367289+0000 mgr.y (mgr.34107) 7104 : cluster [DBG] pgmap v8175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:53.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:53 smithi115 bash[46934]: cluster 2022-04-22T20:53:52.367289+0000 mgr.y (mgr.34107) 7104 : cluster [DBG] pgmap v8175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:54 smithi121 bash[36902]: audit 2022-04-22T20:53:53.564402+0000 mon.a (mon.0) 10613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:54 smithi115 bash[45495]: audit 2022-04-22T20:53:53.564402+0000 mon.a (mon.0) 10613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:54.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:53:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:53:54.294Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:53:54.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:54 smithi115 bash[46934]: audit 2022-04-22T20:53:53.564402+0000 mon.a (mon.0) 10613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:55 smithi121 bash[36902]: cluster 2022-04-22T20:53:54.368466+0000 mgr.y (mgr.34107) 7105 : cluster [DBG] pgmap v8176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:55 smithi121 bash[36902]: audit 2022-04-22T20:53:54.946060+0000 mon.a (mon.0) 10614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:55 smithi115 bash[45495]: cluster 2022-04-22T20:53:54.368466+0000 mgr.y (mgr.34107) 7105 : cluster [DBG] pgmap v8176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:55.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:55 smithi115 bash[45495]: audit 2022-04-22T20:53:54.946060+0000 mon.a (mon.0) 10614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:55 smithi115 bash[46934]: cluster 2022-04-22T20:53:54.368466+0000 mgr.y (mgr.34107) 7105 : cluster [DBG] pgmap v8176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:55 smithi115 bash[46934]: audit 2022-04-22T20:53:54.946060+0000 mon.a (mon.0) 10614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:53:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:57 smithi121 bash[36902]: cluster 2022-04-22T20:53:56.369497+0000 mgr.y (mgr.34107) 7106 : cluster [DBG] pgmap v8177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:57.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:57 smithi115 bash[45495]: cluster 2022-04-22T20:53:56.369497+0000 mgr.y (mgr.34107) 7106 : cluster [DBG] pgmap v8177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:57 smithi115 bash[46934]: cluster 2022-04-22T20:53:56.369497+0000 mgr.y (mgr.34107) 7106 : cluster [DBG] pgmap v8177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:59 smithi121 bash[36902]: cluster 2022-04-22T20:53:58.370592+0000 mgr.y (mgr.34107) 7107 : cluster [DBG] pgmap v8178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:59.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:53:59 smithi121 bash[36902]: audit 2022-04-22T20:53:58.568175+0000 mon.a (mon.0) 10615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:59 smithi115 bash[46934]: cluster 2022-04-22T20:53:58.370592+0000 mgr.y (mgr.34107) 7107 : cluster [DBG] pgmap v8178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:53:59 smithi115 bash[46934]: audit 2022-04-22T20:53:58.568175+0000 mon.a (mon.0) 10615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:53:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:59 smithi115 bash[45495]: cluster 2022-04-22T20:53:58.370592+0000 mgr.y (mgr.34107) 7107 : cluster [DBG] pgmap v8178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:53:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:53:59 smithi115 bash[45495]: audit 2022-04-22T20:53:58.568175+0000 mon.a (mon.0) 10615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:00 smithi121 bash[36902]: audit 2022-04-22T20:53:59.949183+0000 mon.a (mon.0) 10616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:00 smithi121 bash[36902]: cluster 2022-04-22T20:54:00.371659+0000 mgr.y (mgr.34107) 7108 : cluster [DBG] pgmap v8179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:00 smithi115 bash[45495]: audit 2022-04-22T20:53:59.949183+0000 mon.a (mon.0) 10616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:00 smithi115 bash[45495]: cluster 2022-04-22T20:54:00.371659+0000 mgr.y (mgr.34107) 7108 : cluster [DBG] pgmap v8179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:00 smithi115 bash[46934]: audit 2022-04-22T20:53:59.949183+0000 mon.a (mon.0) 10616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:00 smithi115 bash[46934]: cluster 2022-04-22T20:54:00.371659+0000 mgr.y (mgr.34107) 7108 : cluster [DBG] pgmap v8179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:02.415Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:03 smithi121 bash[36902]: cluster 2022-04-22T20:54:02.372647+0000 mgr.y (mgr.34107) 7109 : cluster [DBG] pgmap v8180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:03 smithi115 bash[45495]: cluster 2022-04-22T20:54:02.372647+0000 mgr.y (mgr.34107) 7109 : cluster [DBG] pgmap v8180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:03 smithi115 bash[46934]: cluster 2022-04-22T20:54:02.372647+0000 mgr.y (mgr.34107) 7109 : cluster [DBG] pgmap v8180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:04 smithi121 bash[36902]: audit 2022-04-22T20:54:03.571598+0000 mon.a (mon.0) 10617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:04 smithi115 bash[45495]: audit 2022-04-22T20:54:03.571598+0000 mon.a (mon.0) 10617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:04.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:04.294Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:04 smithi115 bash[46934]: audit 2022-04-22T20:54:03.571598+0000 mon.a (mon.0) 10617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:05 smithi121 bash[36902]: cluster 2022-04-22T20:54:04.373649+0000 mgr.y (mgr.34107) 7110 : cluster [DBG] pgmap v8181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:05 smithi121 bash[36902]: audit 2022-04-22T20:54:04.952266+0000 mon.a (mon.0) 10618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:05 smithi115 bash[45495]: cluster 2022-04-22T20:54:04.373649+0000 mgr.y (mgr.34107) 7110 : cluster [DBG] pgmap v8181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:05.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:05 smithi115 bash[45495]: audit 2022-04-22T20:54:04.952266+0000 mon.a (mon.0) 10618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:05.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:05 smithi115 bash[46934]: cluster 2022-04-22T20:54:04.373649+0000 mgr.y (mgr.34107) 7110 : cluster [DBG] pgmap v8181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:05.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:05 smithi115 bash[46934]: audit 2022-04-22T20:54:04.952266+0000 mon.a (mon.0) 10618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:07 smithi121 bash[36902]: cluster 2022-04-22T20:54:06.374614+0000 mgr.y (mgr.34107) 7111 : cluster [DBG] pgmap v8182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:07 smithi115 bash[45495]: cluster 2022-04-22T20:54:06.374614+0000 mgr.y (mgr.34107) 7111 : cluster [DBG] pgmap v8182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:07 smithi115 bash[46934]: cluster 2022-04-22T20:54:06.374614+0000 mgr.y (mgr.34107) 7111 : cluster [DBG] pgmap v8182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:09.400 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:54:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:54:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 40s ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:54:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 105s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:54:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 40s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:54:09.751 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 105s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 105s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 40s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 40s ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 105s ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 40s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 40s ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 105s ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 40s ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 40s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:54:09.752 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 40s ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 40s ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 105s ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 105s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 105s ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 105s ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:54:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 105s ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:54:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:09 smithi121 bash[36902]: cluster 2022-04-22T20:54:08.375697+0000 mgr.y (mgr.34107) 7112 : cluster [DBG] pgmap v8183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:09 smithi121 bash[36902]: audit 2022-04-22T20:54:08.575557+0000 mon.a (mon.0) 10619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:09 smithi115 bash[45495]: cluster 2022-04-22T20:54:08.375697+0000 mgr.y (mgr.34107) 7112 : cluster [DBG] pgmap v8183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:09 smithi115 bash[45495]: audit 2022-04-22T20:54:08.575557+0000 mon.a (mon.0) 10619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:09 smithi115 bash[46934]: cluster 2022-04-22T20:54:08.375697+0000 mgr.y (mgr.34107) 7112 : cluster [DBG] pgmap v8183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:09 smithi115 bash[46934]: audit 2022-04-22T20:54:08.575557+0000 mon.a (mon.0) 10619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:10.155 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:54:10.155 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:54:10.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:54:10.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:10.157 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:54:10.157 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:54:10.157 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:54:10.157 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:54:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:10 smithi121 bash[36902]: audit 2022-04-22T20:54:09.380173+0000 mgr.y (mgr.34107) 7113 : audit [DBG] from='client.42768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:10 smithi121 bash[36902]: audit 2022-04-22T20:54:09.954831+0000 mon.a (mon.0) 10620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:10 smithi121 bash[36902]: audit 2022-04-22T20:54:10.149975+0000 mon.c (mon.1) 266 : audit [DBG] from='client.? 172.21.15.115:0/404486848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[46934]: audit 2022-04-22T20:54:09.380173+0000 mgr.y (mgr.34107) 7113 : audit [DBG] from='client.42768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[46934]: audit 2022-04-22T20:54:09.954831+0000 mon.a (mon.0) 10620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:10.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[46934]: audit 2022-04-22T20:54:10.149975+0000 mon.c (mon.1) 266 : audit [DBG] from='client.? 172.21.15.115:0/404486848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[45495]: audit 2022-04-22T20:54:09.380173+0000 mgr.y (mgr.34107) 7113 : audit [DBG] from='client.42768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[45495]: audit 2022-04-22T20:54:09.954831+0000 mon.a (mon.0) 10620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:10.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:10 smithi115 bash[45495]: audit 2022-04-22T20:54:10.149975+0000 mon.c (mon.1) 266 : audit [DBG] from='client.? 172.21.15.115:0/404486848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:11 smithi121 bash[36902]: audit 2022-04-22T20:54:09.725440+0000 mgr.y (mgr.34107) 7114 : audit [DBG] from='client.52705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:11 smithi121 bash[36902]: cluster 2022-04-22T20:54:10.376719+0000 mgr.y (mgr.34107) 7115 : cluster [DBG] pgmap v8184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:11.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:11 smithi115 bash[45495]: audit 2022-04-22T20:54:09.725440+0000 mgr.y (mgr.34107) 7114 : audit [DBG] from='client.52705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:11.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:11 smithi115 bash[45495]: cluster 2022-04-22T20:54:10.376719+0000 mgr.y (mgr.34107) 7115 : cluster [DBG] pgmap v8184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:11 smithi115 bash[46934]: audit 2022-04-22T20:54:09.725440+0000 mgr.y (mgr.34107) 7114 : audit [DBG] from='client.52705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:11.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:11 smithi115 bash[46934]: cluster 2022-04-22T20:54:10.376719+0000 mgr.y (mgr.34107) 7115 : cluster [DBG] pgmap v8184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:12.416Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:13 smithi121 bash[36902]: cluster 2022-04-22T20:54:12.377794+0000 mgr.y (mgr.34107) 7116 : cluster [DBG] pgmap v8185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:13 smithi115 bash[45495]: cluster 2022-04-22T20:54:12.377794+0000 mgr.y (mgr.34107) 7116 : cluster [DBG] pgmap v8185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:13 smithi115 bash[46934]: cluster 2022-04-22T20:54:12.377794+0000 mgr.y (mgr.34107) 7116 : cluster [DBG] pgmap v8185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:14 smithi121 bash[36902]: audit 2022-04-22T20:54:13.579319+0000 mon.a (mon.0) 10621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:14 smithi115 bash[45495]: audit 2022-04-22T20:54:13.579319+0000 mon.a (mon.0) 10621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:14 smithi115 bash[46934]: audit 2022-04-22T20:54:13.579319+0000 mon.a (mon.0) 10621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:14.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:14.296Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:15 smithi121 bash[36902]: cluster 2022-04-22T20:54:14.378962+0000 mgr.y (mgr.34107) 7117 : cluster [DBG] pgmap v8186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:15 smithi121 bash[36902]: audit 2022-04-22T20:54:14.958100+0000 mon.a (mon.0) 10622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:15 smithi115 bash[46934]: cluster 2022-04-22T20:54:14.378962+0000 mgr.y (mgr.34107) 7117 : cluster [DBG] pgmap v8186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:15.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:15 smithi115 bash[46934]: audit 2022-04-22T20:54:14.958100+0000 mon.a (mon.0) 10622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:15.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:15 smithi115 bash[45495]: cluster 2022-04-22T20:54:14.378962+0000 mgr.y (mgr.34107) 7117 : cluster [DBG] pgmap v8186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:15.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:15 smithi115 bash[45495]: audit 2022-04-22T20:54:14.958100+0000 mon.a (mon.0) 10622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:16 smithi121 bash[36902]: audit 2022-04-22T20:54:15.666813+0000 mon.a (mon.0) 10623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:54:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:16 smithi121 bash[36902]: audit 2022-04-22T20:54:15.667281+0000 mon.a (mon.0) 10624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:54:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:16 smithi115 bash[45495]: audit 2022-04-22T20:54:15.666813+0000 mon.a (mon.0) 10623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:54:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:16 smithi115 bash[45495]: audit 2022-04-22T20:54:15.667281+0000 mon.a (mon.0) 10624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:54:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:16 smithi115 bash[46934]: audit 2022-04-22T20:54:15.666813+0000 mon.a (mon.0) 10623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:54:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:16 smithi115 bash[46934]: audit 2022-04-22T20:54:15.667281+0000 mon.a (mon.0) 10624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:54:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:17 smithi121 bash[36902]: cluster 2022-04-22T20:54:16.380134+0000 mgr.y (mgr.34107) 7118 : cluster [DBG] pgmap v8187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:17 smithi115 bash[45495]: cluster 2022-04-22T20:54:16.380134+0000 mgr.y (mgr.34107) 7118 : cluster [DBG] pgmap v8187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:17 smithi115 bash[46934]: cluster 2022-04-22T20:54:16.380134+0000 mgr.y (mgr.34107) 7118 : cluster [DBG] pgmap v8187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:19 smithi121 bash[36902]: cluster 2022-04-22T20:54:18.381255+0000 mgr.y (mgr.34107) 7119 : cluster [DBG] pgmap v8188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:19 smithi121 bash[36902]: audit 2022-04-22T20:54:18.582827+0000 mon.a (mon.0) 10625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:19 smithi115 bash[45495]: cluster 2022-04-22T20:54:18.381255+0000 mgr.y (mgr.34107) 7119 : cluster [DBG] pgmap v8188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:19 smithi115 bash[45495]: audit 2022-04-22T20:54:18.582827+0000 mon.a (mon.0) 10625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:19 smithi115 bash[46934]: cluster 2022-04-22T20:54:18.381255+0000 mgr.y (mgr.34107) 7119 : cluster [DBG] pgmap v8188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:19 smithi115 bash[46934]: audit 2022-04-22T20:54:18.582827+0000 mon.a (mon.0) 10625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:20 smithi121 bash[36902]: audit 2022-04-22T20:54:19.961149+0000 mon.a (mon.0) 10626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:20 smithi121 bash[36902]: cluster 2022-04-22T20:54:20.382387+0000 mgr.y (mgr.34107) 7120 : cluster [DBG] pgmap v8189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:20 smithi115 bash[45495]: audit 2022-04-22T20:54:19.961149+0000 mon.a (mon.0) 10626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:20 smithi115 bash[45495]: cluster 2022-04-22T20:54:20.382387+0000 mgr.y (mgr.34107) 7120 : cluster [DBG] pgmap v8189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:21.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:20 smithi115 bash[46934]: audit 2022-04-22T20:54:19.961149+0000 mon.a (mon.0) 10626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:21.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:20 smithi115 bash[46934]: cluster 2022-04-22T20:54:20.382387+0000 mgr.y (mgr.34107) 7120 : cluster [DBG] pgmap v8189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:22.417Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:23 smithi121 bash[36902]: cluster 2022-04-22T20:54:22.383606+0000 mgr.y (mgr.34107) 7121 : cluster [DBG] pgmap v8190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:23 smithi115 bash[45495]: cluster 2022-04-22T20:54:22.383606+0000 mgr.y (mgr.34107) 7121 : cluster [DBG] pgmap v8190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:23 smithi115 bash[46934]: cluster 2022-04-22T20:54:22.383606+0000 mgr.y (mgr.34107) 7121 : cluster [DBG] pgmap v8190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:24 smithi121 bash[36902]: audit 2022-04-22T20:54:23.586316+0000 mon.a (mon.0) 10627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:24 smithi115 bash[45495]: audit 2022-04-22T20:54:23.586316+0000 mon.a (mon.0) 10627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:24.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:24.297Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:24 smithi115 bash[46934]: audit 2022-04-22T20:54:23.586316+0000 mon.a (mon.0) 10627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:25 smithi121 bash[36902]: cluster 2022-04-22T20:54:24.384727+0000 mgr.y (mgr.34107) 7122 : cluster [DBG] pgmap v8191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:25 smithi121 bash[36902]: audit 2022-04-22T20:54:24.964203+0000 mon.a (mon.0) 10628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:25 smithi115 bash[45495]: cluster 2022-04-22T20:54:24.384727+0000 mgr.y (mgr.34107) 7122 : cluster [DBG] pgmap v8191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:25 smithi115 bash[45495]: audit 2022-04-22T20:54:24.964203+0000 mon.a (mon.0) 10628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:25 smithi115 bash[46934]: cluster 2022-04-22T20:54:24.384727+0000 mgr.y (mgr.34107) 7122 : cluster [DBG] pgmap v8191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:25 smithi115 bash[46934]: audit 2022-04-22T20:54:24.964203+0000 mon.a (mon.0) 10628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:27 smithi121 bash[36902]: cluster 2022-04-22T20:54:26.385673+0000 mgr.y (mgr.34107) 7123 : cluster [DBG] pgmap v8192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:27 smithi115 bash[45495]: cluster 2022-04-22T20:54:26.385673+0000 mgr.y (mgr.34107) 7123 : cluster [DBG] pgmap v8192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:27 smithi115 bash[46934]: cluster 2022-04-22T20:54:26.385673+0000 mgr.y (mgr.34107) 7123 : cluster [DBG] pgmap v8192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:29 smithi121 bash[36902]: cluster 2022-04-22T20:54:28.386731+0000 mgr.y (mgr.34107) 7124 : cluster [DBG] pgmap v8193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:29.855 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:29 smithi121 bash[36902]: audit 2022-04-22T20:54:28.590162+0000 mon.a (mon.0) 10629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:29.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:29 smithi115 bash[45495]: cluster 2022-04-22T20:54:28.386731+0000 mgr.y (mgr.34107) 7124 : cluster [DBG] pgmap v8193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:29.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:29 smithi115 bash[45495]: audit 2022-04-22T20:54:28.590162+0000 mon.a (mon.0) 10629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:29.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:29 smithi115 bash[46934]: cluster 2022-04-22T20:54:28.386731+0000 mgr.y (mgr.34107) 7124 : cluster [DBG] pgmap v8193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:29.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:29 smithi115 bash[46934]: audit 2022-04-22T20:54:28.590162+0000 mon.a (mon.0) 10629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:30 smithi121 bash[36902]: audit 2022-04-22T20:54:29.532472+0000 mon.a (mon.0) 10630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:54:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:30 smithi121 bash[36902]: audit 2022-04-22T20:54:29.534800+0000 mon.a (mon.0) 10631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:54:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:30 smithi121 bash[36902]: audit 2022-04-22T20:54:29.536417+0000 mon.a (mon.0) 10632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:54:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:30 smithi121 bash[36902]: audit 2022-04-22T20:54:29.905795+0000 mon.a (mon.0) 10633 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:54:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:30 smithi121 bash[36902]: audit 2022-04-22T20:54:29.967163+0000 mon.a (mon.0) 10634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[45495]: audit 2022-04-22T20:54:29.532472+0000 mon.a (mon.0) 10630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:54:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[45495]: audit 2022-04-22T20:54:29.534800+0000 mon.a (mon.0) 10631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:54:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[45495]: audit 2022-04-22T20:54:29.536417+0000 mon.a (mon.0) 10632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[45495]: audit 2022-04-22T20:54:29.905795+0000 mon.a (mon.0) 10633 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[45495]: audit 2022-04-22T20:54:29.967163+0000 mon.a (mon.0) 10634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[46934]: audit 2022-04-22T20:54:29.532472+0000 mon.a (mon.0) 10630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[46934]: audit 2022-04-22T20:54:29.534800+0000 mon.a (mon.0) 10631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[46934]: audit 2022-04-22T20:54:29.536417+0000 mon.a (mon.0) 10632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[46934]: audit 2022-04-22T20:54:29.905795+0000 mon.a (mon.0) 10633 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:54:30.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:30 smithi115 bash[46934]: audit 2022-04-22T20:54:29.967163+0000 mon.a (mon.0) 10634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:31 smithi121 bash[36902]: cluster 2022-04-22T20:54:30.387808+0000 mgr.y (mgr.34107) 7125 : cluster [DBG] pgmap v8194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:31.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:31 smithi115 bash[45495]: cluster 2022-04-22T20:54:30.387808+0000 mgr.y (mgr.34107) 7125 : cluster [DBG] pgmap v8194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:31.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:31 smithi115 bash[46934]: cluster 2022-04-22T20:54:30.387808+0000 mgr.y (mgr.34107) 7125 : cluster [DBG] pgmap v8194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:32.418Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:33 smithi121 bash[36902]: cluster 2022-04-22T20:54:32.388961+0000 mgr.y (mgr.34107) 7126 : cluster [DBG] pgmap v8195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:33 smithi115 bash[45495]: cluster 2022-04-22T20:54:32.388961+0000 mgr.y (mgr.34107) 7126 : cluster [DBG] pgmap v8195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:33 smithi115 bash[46934]: cluster 2022-04-22T20:54:32.388961+0000 mgr.y (mgr.34107) 7126 : cluster [DBG] pgmap v8195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:34 smithi121 bash[36902]: audit 2022-04-22T20:54:33.593751+0000 mon.a (mon.0) 10635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:34 smithi115 bash[45495]: audit 2022-04-22T20:54:33.593751+0000 mon.a (mon.0) 10635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:34.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:34.297Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:34 smithi115 bash[46934]: audit 2022-04-22T20:54:33.593751+0000 mon.a (mon.0) 10635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:35 smithi121 bash[36902]: cluster 2022-04-22T20:54:34.390188+0000 mgr.y (mgr.34107) 7127 : cluster [DBG] pgmap v8196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:35.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:35 smithi121 bash[36902]: audit 2022-04-22T20:54:34.970130+0000 mon.a (mon.0) 10636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:35 smithi115 bash[45495]: cluster 2022-04-22T20:54:34.390188+0000 mgr.y (mgr.34107) 7127 : cluster [DBG] pgmap v8196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:35 smithi115 bash[45495]: audit 2022-04-22T20:54:34.970130+0000 mon.a (mon.0) 10636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:35 smithi115 bash[46934]: cluster 2022-04-22T20:54:34.390188+0000 mgr.y (mgr.34107) 7127 : cluster [DBG] pgmap v8196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:35 smithi115 bash[46934]: audit 2022-04-22T20:54:34.970130+0000 mon.a (mon.0) 10636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:37 smithi121 bash[36902]: cluster 2022-04-22T20:54:36.391465+0000 mgr.y (mgr.34107) 7128 : cluster [DBG] pgmap v8197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:37 smithi115 bash[45495]: cluster 2022-04-22T20:54:36.391465+0000 mgr.y (mgr.34107) 7128 : cluster [DBG] pgmap v8197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:37 smithi115 bash[46934]: cluster 2022-04-22T20:54:36.391465+0000 mgr.y (mgr.34107) 7128 : cluster [DBG] pgmap v8197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:39 smithi121 bash[36902]: cluster 2022-04-22T20:54:38.392575+0000 mgr.y (mgr.34107) 7129 : cluster [DBG] pgmap v8198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:39 smithi121 bash[36902]: audit 2022-04-22T20:54:38.597369+0000 mon.a (mon.0) 10637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:39 smithi115 bash[45495]: cluster 2022-04-22T20:54:38.392575+0000 mgr.y (mgr.34107) 7129 : cluster [DBG] pgmap v8198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:39 smithi115 bash[45495]: audit 2022-04-22T20:54:38.597369+0000 mon.a (mon.0) 10637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:39 smithi115 bash[46934]: cluster 2022-04-22T20:54:38.392575+0000 mgr.y (mgr.34107) 7129 : cluster [DBG] pgmap v8198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:39 smithi115 bash[46934]: audit 2022-04-22T20:54:38.597369+0000 mon.a (mon.0) 10637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:40.537 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:54:40.877 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:54:40.877 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 71s ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:54:40.877 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:54:40.877 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 71s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 71s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 71s ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 71s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 71s ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 71s ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:54:40.878 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 71s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 71s ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 71s ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:54:40.879 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:54:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:40 smithi121 bash[36902]: audit 2022-04-22T20:54:39.973076+0000 mon.a (mon.0) 10638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:40 smithi121 bash[36902]: cluster 2022-04-22T20:54:40.393608+0000 mgr.y (mgr.34107) 7130 : cluster [DBG] pgmap v8199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:40 smithi121 bash[36902]: audit 2022-04-22T20:54:40.516667+0000 mgr.y (mgr.34107) 7131 : audit [DBG] from='client.42786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[46934]: audit 2022-04-22T20:54:39.973076+0000 mon.a (mon.0) 10638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[46934]: cluster 2022-04-22T20:54:40.393608+0000 mgr.y (mgr.34107) 7130 : cluster [DBG] pgmap v8199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[46934]: audit 2022-04-22T20:54:40.516667+0000 mgr.y (mgr.34107) 7131 : audit [DBG] from='client.42786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[45495]: audit 2022-04-22T20:54:39.973076+0000 mon.a (mon.0) 10638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[45495]: cluster 2022-04-22T20:54:40.393608+0000 mgr.y (mgr.34107) 7130 : cluster [DBG] pgmap v8199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:40 smithi115 bash[45495]: audit 2022-04-22T20:54:40.516667+0000 mgr.y (mgr.34107) 7131 : audit [DBG] from='client.42786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:41.280 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:54:41.280 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:54:41.280 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:54:41.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:41.280 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:54:41.281 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:54:41.282 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:54:41.282 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:54:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:41 smithi121 bash[36902]: audit 2022-04-22T20:54:40.852532+0000 mgr.y (mgr.34107) 7132 : audit [DBG] from='client.52723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:41 smithi121 bash[36902]: audit 2022-04-22T20:54:41.274455+0000 mon.c (mon.1) 267 : audit [DBG] from='client.? 172.21.15.115:0/1002272836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:42.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:41 smithi115 bash[45495]: audit 2022-04-22T20:54:40.852532+0000 mgr.y (mgr.34107) 7132 : audit [DBG] from='client.52723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:42.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:41 smithi115 bash[45495]: audit 2022-04-22T20:54:41.274455+0000 mon.c (mon.1) 267 : audit [DBG] from='client.? 172.21.15.115:0/1002272836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:42.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:41 smithi115 bash[46934]: audit 2022-04-22T20:54:40.852532+0000 mgr.y (mgr.34107) 7132 : audit [DBG] from='client.52723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:54:42.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:41 smithi115 bash[46934]: audit 2022-04-22T20:54:41.274455+0000 mon.c (mon.1) 267 : audit [DBG] from='client.? 172.21.15.115:0/1002272836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:54:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:42 smithi115 bash[45495]: cluster 2022-04-22T20:54:42.394694+0000 mgr.y (mgr.34107) 7133 : cluster [DBG] pgmap v8200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:42.419Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:42 smithi115 bash[46934]: cluster 2022-04-22T20:54:42.394694+0000 mgr.y (mgr.34107) 7133 : cluster [DBG] pgmap v8200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:42 smithi121 bash[36902]: cluster 2022-04-22T20:54:42.394694+0000 mgr.y (mgr.34107) 7133 : cluster [DBG] pgmap v8200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:43 smithi121 bash[36902]: audit 2022-04-22T20:54:43.601329+0000 mon.a (mon.0) 10639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:44.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:43 smithi115 bash[45495]: audit 2022-04-22T20:54:43.601329+0000 mon.a (mon.0) 10639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:44.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:43 smithi115 bash[46934]: audit 2022-04-22T20:54:43.601329+0000 mon.a (mon.0) 10639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:44.653 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:44.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:44.653 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:44 smithi115 bash[45495]: cluster 2022-04-22T20:54:44.395772+0000 mgr.y (mgr.34107) 7134 : cluster [DBG] pgmap v8201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:44 smithi121 bash[36902]: cluster 2022-04-22T20:54:44.395772+0000 mgr.y (mgr.34107) 7134 : cluster [DBG] pgmap v8201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:44 smithi115 bash[46934]: cluster 2022-04-22T20:54:44.395772+0000 mgr.y (mgr.34107) 7134 : cluster [DBG] pgmap v8201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:45 smithi121 bash[36902]: audit 2022-04-22T20:54:44.976045+0000 mon.a (mon.0) 10640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:46.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:45 smithi115 bash[45495]: audit 2022-04-22T20:54:44.976045+0000 mon.a (mon.0) 10640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:45 smithi115 bash[46934]: audit 2022-04-22T20:54:44.976045+0000 mon.a (mon.0) 10640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:46 smithi121 bash[36902]: cluster 2022-04-22T20:54:46.396813+0000 mgr.y (mgr.34107) 7135 : cluster [DBG] pgmap v8202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:47.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:46 smithi115 bash[45495]: cluster 2022-04-22T20:54:46.396813+0000 mgr.y (mgr.34107) 7135 : cluster [DBG] pgmap v8202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:47.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:46 smithi115 bash[46934]: cluster 2022-04-22T20:54:46.396813+0000 mgr.y (mgr.34107) 7135 : cluster [DBG] pgmap v8202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:49 smithi121 bash[36902]: cluster 2022-04-22T20:54:48.398006+0000 mgr.y (mgr.34107) 7136 : cluster [DBG] pgmap v8203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:49 smithi121 bash[36902]: audit 2022-04-22T20:54:48.604978+0000 mon.a (mon.0) 10641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:49 smithi115 bash[45495]: cluster 2022-04-22T20:54:48.398006+0000 mgr.y (mgr.34107) 7136 : cluster [DBG] pgmap v8203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:49 smithi115 bash[45495]: audit 2022-04-22T20:54:48.604978+0000 mon.a (mon.0) 10641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:49 smithi115 bash[46934]: cluster 2022-04-22T20:54:48.398006+0000 mgr.y (mgr.34107) 7136 : cluster [DBG] pgmap v8203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:49 smithi115 bash[46934]: audit 2022-04-22T20:54:48.604978+0000 mon.a (mon.0) 10641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:50 smithi121 bash[36902]: audit 2022-04-22T20:54:49.979181+0000 mon.a (mon.0) 10642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:50 smithi121 bash[36902]: cluster 2022-04-22T20:54:50.399251+0000 mgr.y (mgr.34107) 7137 : cluster [DBG] pgmap v8204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:50 smithi115 bash[45495]: audit 2022-04-22T20:54:49.979181+0000 mon.a (mon.0) 10642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:50 smithi115 bash[45495]: cluster 2022-04-22T20:54:50.399251+0000 mgr.y (mgr.34107) 7137 : cluster [DBG] pgmap v8204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:50 smithi115 bash[46934]: audit 2022-04-22T20:54:49.979181+0000 mon.a (mon.0) 10642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:50 smithi115 bash[46934]: cluster 2022-04-22T20:54:50.399251+0000 mgr.y (mgr.34107) 7137 : cluster [DBG] pgmap v8204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:52.419Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:53 smithi121 bash[36902]: cluster 2022-04-22T20:54:52.400306+0000 mgr.y (mgr.34107) 7138 : cluster [DBG] pgmap v8205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:53 smithi115 bash[45495]: cluster 2022-04-22T20:54:52.400306+0000 mgr.y (mgr.34107) 7138 : cluster [DBG] pgmap v8205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:53 smithi115 bash[46934]: cluster 2022-04-22T20:54:52.400306+0000 mgr.y (mgr.34107) 7138 : cluster [DBG] pgmap v8205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:54 smithi121 bash[36902]: audit 2022-04-22T20:54:53.608657+0000 mon.a (mon.0) 10643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:54 smithi115 bash[45495]: audit 2022-04-22T20:54:53.608657+0000 mon.a (mon.0) 10643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:54.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:54:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:54:54.298Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:54:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:54 smithi115 bash[46934]: audit 2022-04-22T20:54:53.608657+0000 mon.a (mon.0) 10643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:55 smithi121 bash[36902]: cluster 2022-04-22T20:54:54.401403+0000 mgr.y (mgr.34107) 7139 : cluster [DBG] pgmap v8206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:55 smithi121 bash[36902]: audit 2022-04-22T20:54:54.982251+0000 mon.a (mon.0) 10644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:55 smithi115 bash[45495]: cluster 2022-04-22T20:54:54.401403+0000 mgr.y (mgr.34107) 7139 : cluster [DBG] pgmap v8206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:55 smithi115 bash[45495]: audit 2022-04-22T20:54:54.982251+0000 mon.a (mon.0) 10644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:55 smithi115 bash[46934]: cluster 2022-04-22T20:54:54.401403+0000 mgr.y (mgr.34107) 7139 : cluster [DBG] pgmap v8206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:55 smithi115 bash[46934]: audit 2022-04-22T20:54:54.982251+0000 mon.a (mon.0) 10644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:54:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:57 smithi121 bash[36902]: cluster 2022-04-22T20:54:56.402510+0000 mgr.y (mgr.34107) 7140 : cluster [DBG] pgmap v8207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:57.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:57 smithi115 bash[45495]: cluster 2022-04-22T20:54:56.402510+0000 mgr.y (mgr.34107) 7140 : cluster [DBG] pgmap v8207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:57 smithi115 bash[46934]: cluster 2022-04-22T20:54:56.402510+0000 mgr.y (mgr.34107) 7140 : cluster [DBG] pgmap v8207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:59 smithi121 bash[36902]: cluster 2022-04-22T20:54:58.403635+0000 mgr.y (mgr.34107) 7141 : cluster [DBG] pgmap v8208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:54:59 smithi121 bash[36902]: audit 2022-04-22T20:54:58.612510+0000 mon.a (mon.0) 10645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:59 smithi115 bash[45495]: cluster 2022-04-22T20:54:58.403635+0000 mgr.y (mgr.34107) 7141 : cluster [DBG] pgmap v8208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:54:59 smithi115 bash[45495]: audit 2022-04-22T20:54:58.612510+0000 mon.a (mon.0) 10645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:54:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:59 smithi115 bash[46934]: cluster 2022-04-22T20:54:58.403635+0000 mgr.y (mgr.34107) 7141 : cluster [DBG] pgmap v8208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:54:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:54:59 smithi115 bash[46934]: audit 2022-04-22T20:54:58.612510+0000 mon.a (mon.0) 10645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:00 smithi121 bash[36902]: audit 2022-04-22T20:54:59.985199+0000 mon.a (mon.0) 10646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:00 smithi121 bash[36902]: cluster 2022-04-22T20:55:00.404675+0000 mgr.y (mgr.34107) 7142 : cluster [DBG] pgmap v8209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:00 smithi115 bash[45495]: audit 2022-04-22T20:54:59.985199+0000 mon.a (mon.0) 10646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:00 smithi115 bash[45495]: cluster 2022-04-22T20:55:00.404675+0000 mgr.y (mgr.34107) 7142 : cluster [DBG] pgmap v8209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:00 smithi115 bash[46934]: audit 2022-04-22T20:54:59.985199+0000 mon.a (mon.0) 10646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:00 smithi115 bash[46934]: cluster 2022-04-22T20:55:00.404675+0000 mgr.y (mgr.34107) 7142 : cluster [DBG] pgmap v8209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:02.420Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:03 smithi121 bash[36902]: cluster 2022-04-22T20:55:02.405759+0000 mgr.y (mgr.34107) 7143 : cluster [DBG] pgmap v8210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:03 smithi115 bash[45495]: cluster 2022-04-22T20:55:02.405759+0000 mgr.y (mgr.34107) 7143 : cluster [DBG] pgmap v8210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:03 smithi115 bash[46934]: cluster 2022-04-22T20:55:02.405759+0000 mgr.y (mgr.34107) 7143 : cluster [DBG] pgmap v8210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:04 smithi121 bash[36902]: audit 2022-04-22T20:55:03.616344+0000 mon.a (mon.0) 10647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:04 smithi115 bash[45495]: audit 2022-04-22T20:55:03.616344+0000 mon.a (mon.0) 10647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:04.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:04.299Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:04 smithi115 bash[46934]: audit 2022-04-22T20:55:03.616344+0000 mon.a (mon.0) 10647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:05 smithi121 bash[36902]: cluster 2022-04-22T20:55:04.406775+0000 mgr.y (mgr.34107) 7144 : cluster [DBG] pgmap v8211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:05 smithi121 bash[36902]: audit 2022-04-22T20:55:04.988227+0000 mon.a (mon.0) 10648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:05 smithi115 bash[45495]: cluster 2022-04-22T20:55:04.406775+0000 mgr.y (mgr.34107) 7144 : cluster [DBG] pgmap v8211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:05.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:05 smithi115 bash[45495]: audit 2022-04-22T20:55:04.988227+0000 mon.a (mon.0) 10648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:05 smithi115 bash[46934]: cluster 2022-04-22T20:55:04.406775+0000 mgr.y (mgr.34107) 7144 : cluster [DBG] pgmap v8211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:05.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:05 smithi115 bash[46934]: audit 2022-04-22T20:55:04.988227+0000 mon.a (mon.0) 10648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:07 smithi121 bash[36902]: cluster 2022-04-22T20:55:06.407880+0000 mgr.y (mgr.34107) 7145 : cluster [DBG] pgmap v8212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:07 smithi115 bash[45495]: cluster 2022-04-22T20:55:06.407880+0000 mgr.y (mgr.34107) 7145 : cluster [DBG] pgmap v8212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:07.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:07 smithi115 bash[46934]: cluster 2022-04-22T20:55:06.407880+0000 mgr.y (mgr.34107) 7145 : cluster [DBG] pgmap v8212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:09 smithi121 bash[36902]: cluster 2022-04-22T20:55:08.409138+0000 mgr.y (mgr.34107) 7146 : cluster [DBG] pgmap v8213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:09 smithi121 bash[36902]: audit 2022-04-22T20:55:08.621919+0000 mon.a (mon.0) 10649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:09 smithi115 bash[45495]: cluster 2022-04-22T20:55:08.409138+0000 mgr.y (mgr.34107) 7146 : cluster [DBG] pgmap v8213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:09 smithi115 bash[45495]: audit 2022-04-22T20:55:08.621919+0000 mon.a (mon.0) 10649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:09 smithi115 bash[46934]: cluster 2022-04-22T20:55:08.409138+0000 mgr.y (mgr.34107) 7146 : cluster [DBG] pgmap v8213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:09 smithi115 bash[46934]: audit 2022-04-22T20:55:08.621919+0000 mon.a (mon.0) 10649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:10 smithi121 bash[36902]: audit 2022-04-22T20:55:09.991386+0000 mon.a (mon.0) 10650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:10 smithi121 bash[36902]: cluster 2022-04-22T20:55:10.410227+0000 mgr.y (mgr.34107) 7147 : cluster [DBG] pgmap v8214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:10 smithi115 bash[46934]: audit 2022-04-22T20:55:09.991386+0000 mon.a (mon.0) 10650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:10 smithi115 bash[46934]: cluster 2022-04-22T20:55:10.410227+0000 mgr.y (mgr.34107) 7147 : cluster [DBG] pgmap v8214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:10 smithi115 bash[45495]: audit 2022-04-22T20:55:09.991386+0000 mon.a (mon.0) 10650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:10 smithi115 bash[45495]: cluster 2022-04-22T20:55:10.410227+0000 mgr.y (mgr.34107) 7147 : cluster [DBG] pgmap v8214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:11.649 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:55:11.996 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:55:11.996 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 102s ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:55:11.996 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:55:11.996 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 102s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 102s ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 102s ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 102s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 102s ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:55:11.997 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:55:11.998 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 102s ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:55:11.998 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 102s ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:55:11.998 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 102s ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:55:11.998 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 102s ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:55:11.998 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:55:11.999 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:55:11.999 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:55:11.999 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:55:11.999 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:55:12.398 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:55:12.398 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:55:12.398 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:55:12.398 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:12.398 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:12.399 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:55:12.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:55:12.400 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:55:12.400 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:55:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:12 smithi121 bash[36902]: audit 2022-04-22T20:55:12.392710+0000 mon.a (mon.0) 10651 : audit [DBG] from='client.? 172.21.15.115:0/2519775395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:12 smithi115 bash[46934]: audit 2022-04-22T20:55:12.392710+0000 mon.a (mon.0) 10651 : audit [DBG] from='client.? 172.21.15.115:0/2519775395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:12.419Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:12.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:12 smithi115 bash[45495]: audit 2022-04-22T20:55:12.392710+0000 mon.a (mon.0) 10651 : audit [DBG] from='client.? 172.21.15.115:0/2519775395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:13 smithi121 bash[36902]: audit 2022-04-22T20:55:11.629892+0000 mgr.y (mgr.34107) 7148 : audit [DBG] from='client.42804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:13 smithi121 bash[36902]: audit 2022-04-22T20:55:11.969777+0000 mgr.y (mgr.34107) 7149 : audit [DBG] from='client.52741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:13 smithi121 bash[36902]: cluster 2022-04-22T20:55:12.411126+0000 mgr.y (mgr.34107) 7150 : cluster [DBG] pgmap v8215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[45495]: audit 2022-04-22T20:55:11.629892+0000 mgr.y (mgr.34107) 7148 : audit [DBG] from='client.42804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[45495]: audit 2022-04-22T20:55:11.969777+0000 mgr.y (mgr.34107) 7149 : audit [DBG] from='client.52741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[45495]: cluster 2022-04-22T20:55:12.411126+0000 mgr.y (mgr.34107) 7150 : cluster [DBG] pgmap v8215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[46934]: audit 2022-04-22T20:55:11.629892+0000 mgr.y (mgr.34107) 7148 : audit [DBG] from='client.42804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[46934]: audit 2022-04-22T20:55:11.969777+0000 mgr.y (mgr.34107) 7149 : audit [DBG] from='client.52741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:13 smithi115 bash[46934]: cluster 2022-04-22T20:55:12.411126+0000 mgr.y (mgr.34107) 7150 : cluster [DBG] pgmap v8215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:14 smithi121 bash[36902]: audit 2022-04-22T20:55:13.625667+0000 mon.a (mon.0) 10652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:14.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:14 smithi115 bash[45495]: audit 2022-04-22T20:55:13.625667+0000 mon.a (mon.0) 10652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:14 smithi115 bash[46934]: audit 2022-04-22T20:55:13.625667+0000 mon.a (mon.0) 10652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:15 smithi121 bash[36902]: cluster 2022-04-22T20:55:14.412191+0000 mgr.y (mgr.34107) 7151 : cluster [DBG] pgmap v8216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:15 smithi121 bash[36902]: audit 2022-04-22T20:55:14.994421+0000 mon.a (mon.0) 10653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:15 smithi115 bash[46934]: cluster 2022-04-22T20:55:14.412191+0000 mgr.y (mgr.34107) 7151 : cluster [DBG] pgmap v8216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:15 smithi115 bash[46934]: audit 2022-04-22T20:55:14.994421+0000 mon.a (mon.0) 10653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:15 smithi115 bash[45495]: cluster 2022-04-22T20:55:14.412191+0000 mgr.y (mgr.34107) 7151 : cluster [DBG] pgmap v8216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:15 smithi115 bash[45495]: audit 2022-04-22T20:55:14.994421+0000 mon.a (mon.0) 10653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:16 smithi121 bash[36902]: audit 2022-04-22T20:55:15.677832+0000 mon.a (mon.0) 10654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:55:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:16 smithi121 bash[36902]: audit 2022-04-22T20:55:15.678137+0000 mon.a (mon.0) 10655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:55:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:16 smithi115 bash[45495]: audit 2022-04-22T20:55:15.677832+0000 mon.a (mon.0) 10654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:55:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:16 smithi115 bash[45495]: audit 2022-04-22T20:55:15.678137+0000 mon.a (mon.0) 10655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:55:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:16 smithi115 bash[46934]: audit 2022-04-22T20:55:15.677832+0000 mon.a (mon.0) 10654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:55:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:16 smithi115 bash[46934]: audit 2022-04-22T20:55:15.678137+0000 mon.a (mon.0) 10655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:55:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:17 smithi121 bash[36902]: cluster 2022-04-22T20:55:16.413268+0000 mgr.y (mgr.34107) 7152 : cluster [DBG] pgmap v8217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:17 smithi115 bash[45495]: cluster 2022-04-22T20:55:16.413268+0000 mgr.y (mgr.34107) 7152 : cluster [DBG] pgmap v8217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:17 smithi115 bash[46934]: cluster 2022-04-22T20:55:16.413268+0000 mgr.y (mgr.34107) 7152 : cluster [DBG] pgmap v8217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:19 smithi121 bash[36902]: cluster 2022-04-22T20:55:18.414380+0000 mgr.y (mgr.34107) 7153 : cluster [DBG] pgmap v8218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:19 smithi121 bash[36902]: audit 2022-04-22T20:55:18.629350+0000 mon.a (mon.0) 10656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:19 smithi115 bash[45495]: cluster 2022-04-22T20:55:18.414380+0000 mgr.y (mgr.34107) 7153 : cluster [DBG] pgmap v8218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:19.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:19 smithi115 bash[45495]: audit 2022-04-22T20:55:18.629350+0000 mon.a (mon.0) 10656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:19 smithi115 bash[46934]: cluster 2022-04-22T20:55:18.414380+0000 mgr.y (mgr.34107) 7153 : cluster [DBG] pgmap v8218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:19 smithi115 bash[46934]: audit 2022-04-22T20:55:18.629350+0000 mon.a (mon.0) 10656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:20 smithi121 bash[36902]: audit 2022-04-22T20:55:19.997654+0000 mon.a (mon.0) 10657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:20 smithi121 bash[36902]: cluster 2022-04-22T20:55:20.415598+0000 mgr.y (mgr.34107) 7154 : cluster [DBG] pgmap v8219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:20 smithi115 bash[45495]: audit 2022-04-22T20:55:19.997654+0000 mon.a (mon.0) 10657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:20 smithi115 bash[45495]: cluster 2022-04-22T20:55:20.415598+0000 mgr.y (mgr.34107) 7154 : cluster [DBG] pgmap v8219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:21.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:20 smithi115 bash[46934]: audit 2022-04-22T20:55:19.997654+0000 mon.a (mon.0) 10657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:21.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:20 smithi115 bash[46934]: cluster 2022-04-22T20:55:20.415598+0000 mgr.y (mgr.34107) 7154 : cluster [DBG] pgmap v8219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:22.422Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:23 smithi121 bash[36902]: cluster 2022-04-22T20:55:22.416793+0000 mgr.y (mgr.34107) 7155 : cluster [DBG] pgmap v8220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:23 smithi115 bash[45495]: cluster 2022-04-22T20:55:22.416793+0000 mgr.y (mgr.34107) 7155 : cluster [DBG] pgmap v8220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:23 smithi115 bash[46934]: cluster 2022-04-22T20:55:22.416793+0000 mgr.y (mgr.34107) 7155 : cluster [DBG] pgmap v8220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:24 smithi121 bash[36902]: audit 2022-04-22T20:55:23.633205+0000 mon.a (mon.0) 10658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:24 smithi115 bash[45495]: audit 2022-04-22T20:55:23.633205+0000 mon.a (mon.0) 10658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:24.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:24.301Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:24 smithi115 bash[46934]: audit 2022-04-22T20:55:23.633205+0000 mon.a (mon.0) 10658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:25 smithi121 bash[36902]: cluster 2022-04-22T20:55:24.417981+0000 mgr.y (mgr.34107) 7156 : cluster [DBG] pgmap v8221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:25 smithi121 bash[36902]: audit 2022-04-22T20:55:25.000666+0000 mon.a (mon.0) 10659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:25 smithi115 bash[45495]: cluster 2022-04-22T20:55:24.417981+0000 mgr.y (mgr.34107) 7156 : cluster [DBG] pgmap v8221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:25 smithi115 bash[45495]: audit 2022-04-22T20:55:25.000666+0000 mon.a (mon.0) 10659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:25 smithi115 bash[46934]: cluster 2022-04-22T20:55:24.417981+0000 mgr.y (mgr.34107) 7156 : cluster [DBG] pgmap v8221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:25 smithi115 bash[46934]: audit 2022-04-22T20:55:25.000666+0000 mon.a (mon.0) 10659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:27 smithi121 bash[36902]: cluster 2022-04-22T20:55:26.419239+0000 mgr.y (mgr.34107) 7157 : cluster [DBG] pgmap v8222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:27 smithi115 bash[45495]: cluster 2022-04-22T20:55:26.419239+0000 mgr.y (mgr.34107) 7157 : cluster [DBG] pgmap v8222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:27 smithi115 bash[46934]: cluster 2022-04-22T20:55:26.419239+0000 mgr.y (mgr.34107) 7157 : cluster [DBG] pgmap v8222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:29 smithi121 bash[36902]: cluster 2022-04-22T20:55:28.420424+0000 mgr.y (mgr.34107) 7158 : cluster [DBG] pgmap v8223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:29 smithi121 bash[36902]: audit 2022-04-22T20:55:28.636804+0000 mon.a (mon.0) 10660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:29 smithi115 bash[45495]: cluster 2022-04-22T20:55:28.420424+0000 mgr.y (mgr.34107) 7158 : cluster [DBG] pgmap v8223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:29 smithi115 bash[45495]: audit 2022-04-22T20:55:28.636804+0000 mon.a (mon.0) 10660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:29 smithi115 bash[46934]: cluster 2022-04-22T20:55:28.420424+0000 mgr.y (mgr.34107) 7158 : cluster [DBG] pgmap v8223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:29 smithi115 bash[46934]: audit 2022-04-22T20:55:28.636804+0000 mon.a (mon.0) 10660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: audit 2022-04-22T20:55:29.915631+0000 mon.a (mon.0) 10661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:55:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: audit 2022-04-22T20:55:29.918651+0000 mon.a (mon.0) 10662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:55:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: audit 2022-04-22T20:55:29.920556+0000 mon.a (mon.0) 10663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:55:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: audit 2022-04-22T20:55:30.003602+0000 mon.a (mon.0) 10664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: audit 2022-04-22T20:55:30.275537+0000 mon.a (mon.0) 10665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:55:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:30 smithi121 bash[36902]: cluster 2022-04-22T20:55:30.421501+0000 mgr.y (mgr.34107) 7159 : cluster [DBG] pgmap v8224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: audit 2022-04-22T20:55:29.915631+0000 mon.a (mon.0) 10661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: audit 2022-04-22T20:55:29.918651+0000 mon.a (mon.0) 10662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: audit 2022-04-22T20:55:29.920556+0000 mon.a (mon.0) 10663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: audit 2022-04-22T20:55:30.003602+0000 mon.a (mon.0) 10664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: audit 2022-04-22T20:55:30.275537+0000 mon.a (mon.0) 10665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[45495]: cluster 2022-04-22T20:55:30.421501+0000 mgr.y (mgr.34107) 7159 : cluster [DBG] pgmap v8224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: audit 2022-04-22T20:55:29.915631+0000 mon.a (mon.0) 10661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:55:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: audit 2022-04-22T20:55:29.918651+0000 mon.a (mon.0) 10662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:55:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: audit 2022-04-22T20:55:29.920556+0000 mon.a (mon.0) 10663 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:55:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: audit 2022-04-22T20:55:30.003602+0000 mon.a (mon.0) 10664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: audit 2022-04-22T20:55:30.275537+0000 mon.a (mon.0) 10665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:55:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:30 smithi115 bash[46934]: cluster 2022-04-22T20:55:30.421501+0000 mgr.y (mgr.34107) 7159 : cluster [DBG] pgmap v8224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:32.424Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:33 smithi121 bash[36902]: cluster 2022-04-22T20:55:32.422784+0000 mgr.y (mgr.34107) 7160 : cluster [DBG] pgmap v8225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:33 smithi115 bash[45495]: cluster 2022-04-22T20:55:32.422784+0000 mgr.y (mgr.34107) 7160 : cluster [DBG] pgmap v8225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:33 smithi115 bash[46934]: cluster 2022-04-22T20:55:32.422784+0000 mgr.y (mgr.34107) 7160 : cluster [DBG] pgmap v8225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:34 smithi121 bash[36902]: audit 2022-04-22T20:55:33.640232+0000 mon.a (mon.0) 10666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:34 smithi115 bash[45495]: audit 2022-04-22T20:55:33.640232+0000 mon.a (mon.0) 10666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:34.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:34.301Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:34 smithi115 bash[46934]: audit 2022-04-22T20:55:33.640232+0000 mon.a (mon.0) 10666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:35 smithi121 bash[36902]: cluster 2022-04-22T20:55:34.423963+0000 mgr.y (mgr.34107) 7161 : cluster [DBG] pgmap v8226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:35 smithi121 bash[36902]: audit 2022-04-22T20:55:35.006576+0000 mon.a (mon.0) 10667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:35 smithi115 bash[45495]: cluster 2022-04-22T20:55:34.423963+0000 mgr.y (mgr.34107) 7161 : cluster [DBG] pgmap v8226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:35 smithi115 bash[45495]: audit 2022-04-22T20:55:35.006576+0000 mon.a (mon.0) 10667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:35 smithi115 bash[46934]: cluster 2022-04-22T20:55:34.423963+0000 mgr.y (mgr.34107) 7161 : cluster [DBG] pgmap v8226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:35 smithi115 bash[46934]: audit 2022-04-22T20:55:35.006576+0000 mon.a (mon.0) 10667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:37 smithi121 bash[36902]: cluster 2022-04-22T20:55:36.425000+0000 mgr.y (mgr.34107) 7162 : cluster [DBG] pgmap v8227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:37 smithi115 bash[46934]: cluster 2022-04-22T20:55:36.425000+0000 mgr.y (mgr.34107) 7162 : cluster [DBG] pgmap v8227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:37 smithi115 bash[45495]: cluster 2022-04-22T20:55:36.425000+0000 mgr.y (mgr.34107) 7162 : cluster [DBG] pgmap v8227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:39 smithi121 bash[36902]: cluster 2022-04-22T20:55:38.426113+0000 mgr.y (mgr.34107) 7163 : cluster [DBG] pgmap v8228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:39 smithi121 bash[36902]: audit 2022-04-22T20:55:38.643977+0000 mon.a (mon.0) 10668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:39 smithi115 bash[46934]: cluster 2022-04-22T20:55:38.426113+0000 mgr.y (mgr.34107) 7163 : cluster [DBG] pgmap v8228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:39.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:39 smithi115 bash[46934]: audit 2022-04-22T20:55:38.643977+0000 mon.a (mon.0) 10668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:39.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:39 smithi115 bash[45495]: cluster 2022-04-22T20:55:38.426113+0000 mgr.y (mgr.34107) 7163 : cluster [DBG] pgmap v8228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:39.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:39 smithi115 bash[45495]: audit 2022-04-22T20:55:38.643977+0000 mon.a (mon.0) 10668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:40 smithi121 bash[36902]: audit 2022-04-22T20:55:40.009599+0000 mon.a (mon.0) 10669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:40 smithi121 bash[36902]: cluster 2022-04-22T20:55:40.427286+0000 mgr.y (mgr.34107) 7164 : cluster [DBG] pgmap v8229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:40 smithi115 bash[45495]: audit 2022-04-22T20:55:40.009599+0000 mon.a (mon.0) 10669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:40 smithi115 bash[45495]: cluster 2022-04-22T20:55:40.427286+0000 mgr.y (mgr.34107) 7164 : cluster [DBG] pgmap v8229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:40 smithi115 bash[46934]: audit 2022-04-22T20:55:40.009599+0000 mon.a (mon.0) 10669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:40 smithi115 bash[46934]: cluster 2022-04-22T20:55:40.427286+0000 mgr.y (mgr.34107) 7164 : cluster [DBG] pgmap v8229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:42.767 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:55:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:42.424Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:43.101 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:55:43.101 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:55:43.102 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:55:43.103 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:55:43.513 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:55:43.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:55:43.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:55:43.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:43.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:55:43.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:55:43.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:43.515 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:55:43.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:55:43.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:43.516 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:55:43.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:55:43.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:55:43.516 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:55:43.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:55:43.517 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:55:43.517 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:55:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:43 smithi121 bash[36902]: cluster 2022-04-22T20:55:42.427989+0000 mgr.y (mgr.34107) 7165 : cluster [DBG] pgmap v8230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:43.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:43 smithi115 bash[45495]: cluster 2022-04-22T20:55:42.427989+0000 mgr.y (mgr.34107) 7165 : cluster [DBG] pgmap v8230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:43 smithi115 bash[46934]: cluster 2022-04-22T20:55:42.427989+0000 mgr.y (mgr.34107) 7165 : cluster [DBG] pgmap v8230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:44 smithi121 bash[36902]: audit 2022-04-22T20:55:42.744989+0000 mgr.y (mgr.34107) 7166 : audit [DBG] from='client.42822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:44 smithi121 bash[36902]: audit 2022-04-22T20:55:43.077402+0000 mgr.y (mgr.34107) 7167 : audit [DBG] from='client.52759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:44 smithi121 bash[36902]: audit 2022-04-22T20:55:43.508584+0000 mon.c (mon.1) 268 : audit [DBG] from='client.? 172.21.15.115:0/1455320369' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:44 smithi121 bash[36902]: audit 2022-04-22T20:55:43.647534+0000 mon.a (mon.0) 10670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:44.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[45495]: audit 2022-04-22T20:55:42.744989+0000 mgr.y (mgr.34107) 7166 : audit [DBG] from='client.42822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[45495]: audit 2022-04-22T20:55:43.077402+0000 mgr.y (mgr.34107) 7167 : audit [DBG] from='client.52759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[45495]: audit 2022-04-22T20:55:43.508584+0000 mon.c (mon.1) 268 : audit [DBG] from='client.? 172.21.15.115:0/1455320369' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[45495]: audit 2022-04-22T20:55:43.647534+0000 mon.a (mon.0) 10670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[46934]: audit 2022-04-22T20:55:42.744989+0000 mgr.y (mgr.34107) 7166 : audit [DBG] from='client.42822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[46934]: audit 2022-04-22T20:55:43.077402+0000 mgr.y (mgr.34107) 7167 : audit [DBG] from='client.52759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:55:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[46934]: audit 2022-04-22T20:55:43.508584+0000 mon.c (mon.1) 268 : audit [DBG] from='client.? 172.21.15.115:0/1455320369' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:55:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:44 smithi115 bash[46934]: audit 2022-04-22T20:55:43.647534+0000 mon.a (mon.0) 10670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:45 smithi121 bash[36902]: cluster 2022-04-22T20:55:44.429011+0000 mgr.y (mgr.34107) 7168 : cluster [DBG] pgmap v8231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:45 smithi121 bash[36902]: audit 2022-04-22T20:55:45.012579+0000 mon.a (mon.0) 10671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:45 smithi115 bash[45495]: cluster 2022-04-22T20:55:44.429011+0000 mgr.y (mgr.34107) 7168 : cluster [DBG] pgmap v8231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:45.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:45 smithi115 bash[45495]: audit 2022-04-22T20:55:45.012579+0000 mon.a (mon.0) 10671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:45 smithi115 bash[46934]: cluster 2022-04-22T20:55:44.429011+0000 mgr.y (mgr.34107) 7168 : cluster [DBG] pgmap v8231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:45 smithi115 bash[46934]: audit 2022-04-22T20:55:45.012579+0000 mon.a (mon.0) 10671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:47 smithi115 bash[45495]: cluster 2022-04-22T20:55:46.430144+0000 mgr.y (mgr.34107) 7169 : cluster [DBG] pgmap v8232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:47 smithi115 bash[46934]: cluster 2022-04-22T20:55:46.430144+0000 mgr.y (mgr.34107) 7169 : cluster [DBG] pgmap v8232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:47 smithi121 bash[36902]: cluster 2022-04-22T20:55:46.430144+0000 mgr.y (mgr.34107) 7169 : cluster [DBG] pgmap v8232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:49 smithi115 bash[45495]: cluster 2022-04-22T20:55:48.431290+0000 mgr.y (mgr.34107) 7170 : cluster [DBG] pgmap v8233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:49.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:49 smithi115 bash[45495]: audit 2022-04-22T20:55:48.650729+0000 mon.a (mon.0) 10672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:49 smithi115 bash[46934]: cluster 2022-04-22T20:55:48.431290+0000 mgr.y (mgr.34107) 7170 : cluster [DBG] pgmap v8233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:49.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:49 smithi115 bash[46934]: audit 2022-04-22T20:55:48.650729+0000 mon.a (mon.0) 10672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:49 smithi121 bash[36902]: cluster 2022-04-22T20:55:48.431290+0000 mgr.y (mgr.34107) 7170 : cluster [DBG] pgmap v8233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:49 smithi121 bash[36902]: audit 2022-04-22T20:55:48.650729+0000 mon.a (mon.0) 10672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:50 smithi121 bash[36902]: audit 2022-04-22T20:55:50.015448+0000 mon.a (mon.0) 10673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:50 smithi121 bash[36902]: cluster 2022-04-22T20:55:50.432372+0000 mgr.y (mgr.34107) 7171 : cluster [DBG] pgmap v8234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:50 smithi115 bash[45495]: audit 2022-04-22T20:55:50.015448+0000 mon.a (mon.0) 10673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:50 smithi115 bash[45495]: cluster 2022-04-22T20:55:50.432372+0000 mgr.y (mgr.34107) 7171 : cluster [DBG] pgmap v8234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:50 smithi115 bash[46934]: audit 2022-04-22T20:55:50.015448+0000 mon.a (mon.0) 10673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:50 smithi115 bash[46934]: cluster 2022-04-22T20:55:50.432372+0000 mgr.y (mgr.34107) 7171 : cluster [DBG] pgmap v8234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:52.425Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:53 smithi121 bash[36902]: cluster 2022-04-22T20:55:52.433434+0000 mgr.y (mgr.34107) 7172 : cluster [DBG] pgmap v8235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:53 smithi115 bash[45495]: cluster 2022-04-22T20:55:52.433434+0000 mgr.y (mgr.34107) 7172 : cluster [DBG] pgmap v8235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:53 smithi115 bash[46934]: cluster 2022-04-22T20:55:52.433434+0000 mgr.y (mgr.34107) 7172 : cluster [DBG] pgmap v8235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:54 smithi121 bash[36902]: audit 2022-04-22T20:55:53.654411+0000 mon.a (mon.0) 10674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:54 smithi115 bash[45495]: audit 2022-04-22T20:55:53.654411+0000 mon.a (mon.0) 10674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:54.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:55:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:55:54.304Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:55:54.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:54 smithi115 bash[46934]: audit 2022-04-22T20:55:53.654411+0000 mon.a (mon.0) 10674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:55 smithi121 bash[36902]: cluster 2022-04-22T20:55:54.434495+0000 mgr.y (mgr.34107) 7173 : cluster [DBG] pgmap v8236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:55 smithi121 bash[36902]: audit 2022-04-22T20:55:55.018566+0000 mon.a (mon.0) 10675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:55 smithi115 bash[45495]: cluster 2022-04-22T20:55:54.434495+0000 mgr.y (mgr.34107) 7173 : cluster [DBG] pgmap v8236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:55 smithi115 bash[45495]: audit 2022-04-22T20:55:55.018566+0000 mon.a (mon.0) 10675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:55 smithi115 bash[46934]: cluster 2022-04-22T20:55:54.434495+0000 mgr.y (mgr.34107) 7173 : cluster [DBG] pgmap v8236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:55.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:55 smithi115 bash[46934]: audit 2022-04-22T20:55:55.018566+0000 mon.a (mon.0) 10675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:55:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:57 smithi121 bash[36902]: cluster 2022-04-22T20:55:56.435726+0000 mgr.y (mgr.34107) 7174 : cluster [DBG] pgmap v8237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:57.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:57 smithi115 bash[45495]: cluster 2022-04-22T20:55:56.435726+0000 mgr.y (mgr.34107) 7174 : cluster [DBG] pgmap v8237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:57.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:57 smithi115 bash[46934]: cluster 2022-04-22T20:55:56.435726+0000 mgr.y (mgr.34107) 7174 : cluster [DBG] pgmap v8237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:59 smithi115 bash[45495]: cluster 2022-04-22T20:55:58.436817+0000 mgr.y (mgr.34107) 7175 : cluster [DBG] pgmap v8238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:55:59 smithi115 bash[45495]: audit 2022-04-22T20:55:58.658129+0000 mon.a (mon.0) 10676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:55:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:59 smithi115 bash[46934]: cluster 2022-04-22T20:55:58.436817+0000 mgr.y (mgr.34107) 7175 : cluster [DBG] pgmap v8238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:55:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:55:59 smithi115 bash[46934]: audit 2022-04-22T20:55:58.658129+0000 mon.a (mon.0) 10676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:59 smithi121 bash[36902]: cluster 2022-04-22T20:55:58.436817+0000 mgr.y (mgr.34107) 7175 : cluster [DBG] pgmap v8238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:55:59 smithi121 bash[36902]: audit 2022-04-22T20:55:58.658129+0000 mon.a (mon.0) 10676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:00 smithi121 bash[36902]: audit 2022-04-22T20:56:00.021498+0000 mon.a (mon.0) 10677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:00 smithi121 bash[36902]: cluster 2022-04-22T20:56:00.437874+0000 mgr.y (mgr.34107) 7176 : cluster [DBG] pgmap v8239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:00 smithi115 bash[45495]: audit 2022-04-22T20:56:00.021498+0000 mon.a (mon.0) 10677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:00 smithi115 bash[45495]: cluster 2022-04-22T20:56:00.437874+0000 mgr.y (mgr.34107) 7176 : cluster [DBG] pgmap v8239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:00 smithi115 bash[46934]: audit 2022-04-22T20:56:00.021498+0000 mon.a (mon.0) 10677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:00 smithi115 bash[46934]: cluster 2022-04-22T20:56:00.437874+0000 mgr.y (mgr.34107) 7176 : cluster [DBG] pgmap v8239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:02.426Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:03 smithi121 bash[36902]: cluster 2022-04-22T20:56:02.438892+0000 mgr.y (mgr.34107) 7177 : cluster [DBG] pgmap v8240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:03.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:03 smithi115 bash[45495]: cluster 2022-04-22T20:56:02.438892+0000 mgr.y (mgr.34107) 7177 : cluster [DBG] pgmap v8240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:03.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:03 smithi115 bash[46934]: cluster 2022-04-22T20:56:02.438892+0000 mgr.y (mgr.34107) 7177 : cluster [DBG] pgmap v8240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:04 smithi121 bash[36902]: audit 2022-04-22T20:56:03.662396+0000 mon.a (mon.0) 10678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:04 smithi115 bash[45495]: audit 2022-04-22T20:56:03.662396+0000 mon.a (mon.0) 10678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:04.303Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:04 smithi115 bash[46934]: audit 2022-04-22T20:56:03.662396+0000 mon.a (mon.0) 10678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:05 smithi121 bash[36902]: cluster 2022-04-22T20:56:04.439975+0000 mgr.y (mgr.34107) 7178 : cluster [DBG] pgmap v8241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:05 smithi121 bash[36902]: audit 2022-04-22T20:56:05.024682+0000 mon.a (mon.0) 10679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:05 smithi115 bash[45495]: cluster 2022-04-22T20:56:04.439975+0000 mgr.y (mgr.34107) 7178 : cluster [DBG] pgmap v8241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:05 smithi115 bash[45495]: audit 2022-04-22T20:56:05.024682+0000 mon.a (mon.0) 10679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:05 smithi115 bash[46934]: cluster 2022-04-22T20:56:04.439975+0000 mgr.y (mgr.34107) 7178 : cluster [DBG] pgmap v8241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:05 smithi115 bash[46934]: audit 2022-04-22T20:56:05.024682+0000 mon.a (mon.0) 10679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:07 smithi115 bash[45495]: cluster 2022-04-22T20:56:06.441080+0000 mgr.y (mgr.34107) 7179 : cluster [DBG] pgmap v8242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:07 smithi115 bash[46934]: cluster 2022-04-22T20:56:06.441080+0000 mgr.y (mgr.34107) 7179 : cluster [DBG] pgmap v8242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:07 smithi121 bash[36902]: cluster 2022-04-22T20:56:06.441080+0000 mgr.y (mgr.34107) 7179 : cluster [DBG] pgmap v8242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:09 smithi115 bash[46934]: cluster 2022-04-22T20:56:08.442203+0000 mgr.y (mgr.34107) 7180 : cluster [DBG] pgmap v8243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:09.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:09 smithi115 bash[46934]: audit 2022-04-22T20:56:08.665998+0000 mon.a (mon.0) 10680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:09.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:09 smithi115 bash[45495]: cluster 2022-04-22T20:56:08.442203+0000 mgr.y (mgr.34107) 7180 : cluster [DBG] pgmap v8243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:09.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:09 smithi115 bash[45495]: audit 2022-04-22T20:56:08.665998+0000 mon.a (mon.0) 10680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:09 smithi121 bash[36902]: cluster 2022-04-22T20:56:08.442203+0000 mgr.y (mgr.34107) 7180 : cluster [DBG] pgmap v8243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:09 smithi121 bash[36902]: audit 2022-04-22T20:56:08.665998+0000 mon.a (mon.0) 10680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:10 smithi121 bash[36902]: audit 2022-04-22T20:56:10.027987+0000 mon.a (mon.0) 10681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:10 smithi121 bash[36902]: cluster 2022-04-22T20:56:10.443284+0000 mgr.y (mgr.34107) 7181 : cluster [DBG] pgmap v8244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:10 smithi115 bash[45495]: audit 2022-04-22T20:56:10.027987+0000 mon.a (mon.0) 10681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:11.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:10 smithi115 bash[45495]: cluster 2022-04-22T20:56:10.443284+0000 mgr.y (mgr.34107) 7181 : cluster [DBG] pgmap v8244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:10 smithi115 bash[46934]: audit 2022-04-22T20:56:10.027987+0000 mon.a (mon.0) 10681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:10 smithi115 bash[46934]: cluster 2022-04-22T20:56:10.443284+0000 mgr.y (mgr.34107) 7181 : cluster [DBG] pgmap v8244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:12.426Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:13 smithi121 bash[36902]: cluster 2022-04-22T20:56:12.444384+0000 mgr.y (mgr.34107) 7182 : cluster [DBG] pgmap v8245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:13 smithi115 bash[46934]: cluster 2022-04-22T20:56:12.444384+0000 mgr.y (mgr.34107) 7182 : cluster [DBG] pgmap v8245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:13.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:13 smithi115 bash[45495]: cluster 2022-04-22T20:56:12.444384+0000 mgr.y (mgr.34107) 7182 : cluster [DBG] pgmap v8245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:13.873 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:56:14.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:56:14.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:56:14.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:56:14.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:56:14.221 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:56:14.221 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:56:14.620 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:56:14.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:56:14.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:56:14.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:14.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:56:14.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:56:14.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:56:14.622 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:56:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:14 smithi115 bash[45495]: audit 2022-04-22T20:56:13.669470+0000 mon.a (mon.0) 10682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:14.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:14.302Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:14.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:14 smithi115 bash[46934]: audit 2022-04-22T20:56:13.669470+0000 mon.a (mon.0) 10682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:14 smithi121 bash[36902]: audit 2022-04-22T20:56:13.669470+0000 mon.a (mon.0) 10682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[45495]: audit 2022-04-22T20:56:13.854383+0000 mgr.y (mgr.34107) 7183 : audit [DBG] from='client.52771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[45495]: audit 2022-04-22T20:56:14.189354+0000 mgr.y (mgr.34107) 7184 : audit [DBG] from='client.52777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[45495]: cluster 2022-04-22T20:56:14.445411+0000 mgr.y (mgr.34107) 7185 : cluster [DBG] pgmap v8246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[45495]: audit 2022-04-22T20:56:14.614957+0000 mon.a (mon.0) 10683 : audit [DBG] from='client.? 172.21.15.115:0/1694854810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[45495]: audit 2022-04-22T20:56:15.031246+0000 mon.a (mon.0) 10684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[46934]: audit 2022-04-22T20:56:13.854383+0000 mgr.y (mgr.34107) 7183 : audit [DBG] from='client.52771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[46934]: audit 2022-04-22T20:56:14.189354+0000 mgr.y (mgr.34107) 7184 : audit [DBG] from='client.52777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[46934]: cluster 2022-04-22T20:56:14.445411+0000 mgr.y (mgr.34107) 7185 : cluster [DBG] pgmap v8246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:15.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[46934]: audit 2022-04-22T20:56:14.614957+0000 mon.a (mon.0) 10683 : audit [DBG] from='client.? 172.21.15.115:0/1694854810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:15.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:15 smithi115 bash[46934]: audit 2022-04-22T20:56:15.031246+0000 mon.a (mon.0) 10684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:15 smithi121 bash[36902]: audit 2022-04-22T20:56:13.854383+0000 mgr.y (mgr.34107) 7183 : audit [DBG] from='client.52771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:15 smithi121 bash[36902]: audit 2022-04-22T20:56:14.189354+0000 mgr.y (mgr.34107) 7184 : audit [DBG] from='client.52777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:15 smithi121 bash[36902]: cluster 2022-04-22T20:56:14.445411+0000 mgr.y (mgr.34107) 7185 : cluster [DBG] pgmap v8246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:15 smithi121 bash[36902]: audit 2022-04-22T20:56:14.614957+0000 mon.a (mon.0) 10683 : audit [DBG] from='client.? 172.21.15.115:0/1694854810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:15 smithi121 bash[36902]: audit 2022-04-22T20:56:15.031246+0000 mon.a (mon.0) 10684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:16 smithi115 bash[46934]: audit 2022-04-22T20:56:15.686484+0000 mon.a (mon.0) 10685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:56:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:16 smithi115 bash[46934]: audit 2022-04-22T20:56:15.689542+0000 mon.a (mon.0) 10686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:56:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:16 smithi115 bash[45495]: audit 2022-04-22T20:56:15.686484+0000 mon.a (mon.0) 10685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:56:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:16 smithi115 bash[45495]: audit 2022-04-22T20:56:15.689542+0000 mon.a (mon.0) 10686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:56:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:16 smithi121 bash[36902]: audit 2022-04-22T20:56:15.686484+0000 mon.a (mon.0) 10685 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:56:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:16 smithi121 bash[36902]: audit 2022-04-22T20:56:15.689542+0000 mon.a (mon.0) 10686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:56:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:17 smithi115 bash[45495]: cluster 2022-04-22T20:56:16.446489+0000 mgr.y (mgr.34107) 7186 : cluster [DBG] pgmap v8247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:17 smithi115 bash[46934]: cluster 2022-04-22T20:56:16.446489+0000 mgr.y (mgr.34107) 7186 : cluster [DBG] pgmap v8247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:17 smithi121 bash[36902]: cluster 2022-04-22T20:56:16.446489+0000 mgr.y (mgr.34107) 7186 : cluster [DBG] pgmap v8247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:19 smithi115 bash[45495]: cluster 2022-04-22T20:56:18.447675+0000 mgr.y (mgr.34107) 7187 : cluster [DBG] pgmap v8248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:19.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:19 smithi115 bash[45495]: audit 2022-04-22T20:56:18.673124+0000 mon.a (mon.0) 10687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:19 smithi115 bash[46934]: cluster 2022-04-22T20:56:18.447675+0000 mgr.y (mgr.34107) 7187 : cluster [DBG] pgmap v8248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:19 smithi115 bash[46934]: audit 2022-04-22T20:56:18.673124+0000 mon.a (mon.0) 10687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:19 smithi121 bash[36902]: cluster 2022-04-22T20:56:18.447675+0000 mgr.y (mgr.34107) 7187 : cluster [DBG] pgmap v8248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:19 smithi121 bash[36902]: audit 2022-04-22T20:56:18.673124+0000 mon.a (mon.0) 10687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:20 smithi121 bash[36902]: audit 2022-04-22T20:56:20.034290+0000 mon.a (mon.0) 10688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:20 smithi121 bash[36902]: cluster 2022-04-22T20:56:20.448743+0000 mgr.y (mgr.34107) 7188 : cluster [DBG] pgmap v8249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:20 smithi115 bash[46934]: audit 2022-04-22T20:56:20.034290+0000 mon.a (mon.0) 10688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:20 smithi115 bash[46934]: cluster 2022-04-22T20:56:20.448743+0000 mgr.y (mgr.34107) 7188 : cluster [DBG] pgmap v8249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:20 smithi115 bash[45495]: audit 2022-04-22T20:56:20.034290+0000 mon.a (mon.0) 10688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:20 smithi115 bash[45495]: cluster 2022-04-22T20:56:20.448743+0000 mgr.y (mgr.34107) 7188 : cluster [DBG] pgmap v8249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:22.427Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:23 smithi121 bash[36902]: cluster 2022-04-22T20:56:22.449868+0000 mgr.y (mgr.34107) 7189 : cluster [DBG] pgmap v8250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:23 smithi115 bash[45495]: cluster 2022-04-22T20:56:22.449868+0000 mgr.y (mgr.34107) 7189 : cluster [DBG] pgmap v8250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:23 smithi115 bash[46934]: cluster 2022-04-22T20:56:22.449868+0000 mgr.y (mgr.34107) 7189 : cluster [DBG] pgmap v8250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:24.304Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:24.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:24 smithi115 bash[45495]: audit 2022-04-22T20:56:23.676735+0000 mon.a (mon.0) 10689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:24 smithi115 bash[46934]: audit 2022-04-22T20:56:23.676735+0000 mon.a (mon.0) 10689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:24 smithi121 bash[36902]: audit 2022-04-22T20:56:23.676735+0000 mon.a (mon.0) 10689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:25 smithi115 bash[45495]: cluster 2022-04-22T20:56:24.450937+0000 mgr.y (mgr.34107) 7190 : cluster [DBG] pgmap v8251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:25.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:25 smithi115 bash[45495]: audit 2022-04-22T20:56:25.037364+0000 mon.a (mon.0) 10690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:25 smithi115 bash[46934]: cluster 2022-04-22T20:56:24.450937+0000 mgr.y (mgr.34107) 7190 : cluster [DBG] pgmap v8251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:25.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:25 smithi115 bash[46934]: audit 2022-04-22T20:56:25.037364+0000 mon.a (mon.0) 10690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:25 smithi121 bash[36902]: cluster 2022-04-22T20:56:24.450937+0000 mgr.y (mgr.34107) 7190 : cluster [DBG] pgmap v8251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:25 smithi121 bash[36902]: audit 2022-04-22T20:56:25.037364+0000 mon.a (mon.0) 10690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:27.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:27 smithi115 bash[45495]: cluster 2022-04-22T20:56:26.452129+0000 mgr.y (mgr.34107) 7191 : cluster [DBG] pgmap v8252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:27.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:27 smithi115 bash[46934]: cluster 2022-04-22T20:56:26.452129+0000 mgr.y (mgr.34107) 7191 : cluster [DBG] pgmap v8252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:27 smithi121 bash[36902]: cluster 2022-04-22T20:56:26.452129+0000 mgr.y (mgr.34107) 7191 : cluster [DBG] pgmap v8252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:29 smithi115 bash[45495]: cluster 2022-04-22T20:56:28.453225+0000 mgr.y (mgr.34107) 7192 : cluster [DBG] pgmap v8253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:29 smithi115 bash[45495]: audit 2022-04-22T20:56:28.680423+0000 mon.a (mon.0) 10691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:29 smithi115 bash[46934]: cluster 2022-04-22T20:56:28.453225+0000 mgr.y (mgr.34107) 7192 : cluster [DBG] pgmap v8253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:29.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:29 smithi115 bash[46934]: audit 2022-04-22T20:56:28.680423+0000 mon.a (mon.0) 10691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:29 smithi121 bash[36902]: cluster 2022-04-22T20:56:28.453225+0000 mgr.y (mgr.34107) 7192 : cluster [DBG] pgmap v8253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:29 smithi121 bash[36902]: audit 2022-04-22T20:56:28.680423+0000 mon.a (mon.0) 10691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: audit 2022-04-22T20:56:30.040497+0000 mon.a (mon.0) 10692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: audit 2022-04-22T20:56:30.282446+0000 mon.a (mon.0) 10693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:56:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: audit 2022-04-22T20:56:30.285508+0000 mon.a (mon.0) 10694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:56:31.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: audit 2022-04-22T20:56:30.287305+0000 mon.a (mon.0) 10695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:56:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: cluster 2022-04-22T20:56:30.454267+0000 mgr.y (mgr.34107) 7193 : cluster [DBG] pgmap v8254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:31.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:30 smithi121 bash[36902]: audit 2022-04-22T20:56:30.645498+0000 mon.a (mon.0) 10696 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:56:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: audit 2022-04-22T20:56:30.040497+0000 mon.a (mon.0) 10692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: audit 2022-04-22T20:56:30.282446+0000 mon.a (mon.0) 10693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: audit 2022-04-22T20:56:30.285508+0000 mon.a (mon.0) 10694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: audit 2022-04-22T20:56:30.287305+0000 mon.a (mon.0) 10695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: cluster 2022-04-22T20:56:30.454267+0000 mgr.y (mgr.34107) 7193 : cluster [DBG] pgmap v8254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[45495]: audit 2022-04-22T20:56:30.645498+0000 mon.a (mon.0) 10696 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:56:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: audit 2022-04-22T20:56:30.040497+0000 mon.a (mon.0) 10692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: audit 2022-04-22T20:56:30.282446+0000 mon.a (mon.0) 10693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: audit 2022-04-22T20:56:30.285508+0000 mon.a (mon.0) 10694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: audit 2022-04-22T20:56:30.287305+0000 mon.a (mon.0) 10695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: cluster 2022-04-22T20:56:30.454267+0000 mgr.y (mgr.34107) 7193 : cluster [DBG] pgmap v8254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:30 smithi115 bash[46934]: audit 2022-04-22T20:56:30.645498+0000 mon.a (mon.0) 10696 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:56:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:32.428Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:33 smithi115 bash[45495]: cluster 2022-04-22T20:56:32.455403+0000 mgr.y (mgr.34107) 7194 : cluster [DBG] pgmap v8255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:33 smithi115 bash[46934]: cluster 2022-04-22T20:56:32.455403+0000 mgr.y (mgr.34107) 7194 : cluster [DBG] pgmap v8255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:33 smithi121 bash[36902]: cluster 2022-04-22T20:56:32.455403+0000 mgr.y (mgr.34107) 7194 : cluster [DBG] pgmap v8255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:34 smithi115 bash[45495]: audit 2022-04-22T20:56:33.684131+0000 mon.a (mon.0) 10697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:34 smithi115 bash[46934]: audit 2022-04-22T20:56:33.684131+0000 mon.a (mon.0) 10697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:34.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:34.304Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:34 smithi121 bash[36902]: audit 2022-04-22T20:56:33.684131+0000 mon.a (mon.0) 10697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:35 smithi115 bash[45495]: cluster 2022-04-22T20:56:34.456473+0000 mgr.y (mgr.34107) 7195 : cluster [DBG] pgmap v8256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:35.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:35 smithi115 bash[45495]: audit 2022-04-22T20:56:35.043680+0000 mon.a (mon.0) 10698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:35 smithi115 bash[46934]: cluster 2022-04-22T20:56:34.456473+0000 mgr.y (mgr.34107) 7195 : cluster [DBG] pgmap v8256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:35.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:35 smithi115 bash[46934]: audit 2022-04-22T20:56:35.043680+0000 mon.a (mon.0) 10698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:35 smithi121 bash[36902]: cluster 2022-04-22T20:56:34.456473+0000 mgr.y (mgr.34107) 7195 : cluster [DBG] pgmap v8256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:35 smithi121 bash[36902]: audit 2022-04-22T20:56:35.043680+0000 mon.a (mon.0) 10698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:37.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:37 smithi115 bash[45495]: cluster 2022-04-22T20:56:36.457503+0000 mgr.y (mgr.34107) 7196 : cluster [DBG] pgmap v8257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:37 smithi115 bash[46934]: cluster 2022-04-22T20:56:36.457503+0000 mgr.y (mgr.34107) 7196 : cluster [DBG] pgmap v8257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:37 smithi121 bash[36902]: cluster 2022-04-22T20:56:36.457503+0000 mgr.y (mgr.34107) 7196 : cluster [DBG] pgmap v8257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:39 smithi115 bash[45495]: cluster 2022-04-22T20:56:38.458673+0000 mgr.y (mgr.34107) 7197 : cluster [DBG] pgmap v8258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:39 smithi115 bash[45495]: audit 2022-04-22T20:56:38.687836+0000 mon.a (mon.0) 10699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:39.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:39 smithi115 bash[46934]: cluster 2022-04-22T20:56:38.458673+0000 mgr.y (mgr.34107) 7197 : cluster [DBG] pgmap v8258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:39.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:39 smithi115 bash[46934]: audit 2022-04-22T20:56:38.687836+0000 mon.a (mon.0) 10699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:39 smithi121 bash[36902]: cluster 2022-04-22T20:56:38.458673+0000 mgr.y (mgr.34107) 7197 : cluster [DBG] pgmap v8258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:39 smithi121 bash[36902]: audit 2022-04-22T20:56:38.687836+0000 mon.a (mon.0) 10699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:40 smithi121 bash[36902]: audit 2022-04-22T20:56:40.046550+0000 mon.a (mon.0) 10700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:40 smithi121 bash[36902]: cluster 2022-04-22T20:56:40.459760+0000 mgr.y (mgr.34107) 7198 : cluster [DBG] pgmap v8259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:40 smithi115 bash[46934]: audit 2022-04-22T20:56:40.046550+0000 mon.a (mon.0) 10700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:40 smithi115 bash[46934]: cluster 2022-04-22T20:56:40.459760+0000 mgr.y (mgr.34107) 7198 : cluster [DBG] pgmap v8259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:40 smithi115 bash[45495]: audit 2022-04-22T20:56:40.046550+0000 mon.a (mon.0) 10700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:40 smithi115 bash[45495]: cluster 2022-04-22T20:56:40.459760+0000 mgr.y (mgr.34107) 7198 : cluster [DBG] pgmap v8259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:42.430Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:43.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:43 smithi115 bash[46934]: cluster 2022-04-22T20:56:42.460849+0000 mgr.y (mgr.34107) 7199 : cluster [DBG] pgmap v8260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:43.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:43 smithi115 bash[45495]: cluster 2022-04-22T20:56:42.460849+0000 mgr.y (mgr.34107) 7199 : cluster [DBG] pgmap v8260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:43 smithi121 bash[36902]: cluster 2022-04-22T20:56:42.460849+0000 mgr.y (mgr.34107) 7199 : cluster [DBG] pgmap v8260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:44 smithi115 bash[46934]: audit 2022-04-22T20:56:43.691544+0000 mon.a (mon.0) 10701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:44.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:44.306Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:44 smithi115 bash[45495]: audit 2022-04-22T20:56:43.691544+0000 mon.a (mon.0) 10701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:44.996 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:56:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:44 smithi121 bash[36902]: audit 2022-04-22T20:56:43.691544+0000 mon.a (mon.0) 10701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:45.344 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:56:45.353 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:56:45.353 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:56:45.354 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:56:45.355 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:56:45.747 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:56:45.747 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:56:45.747 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:56:45.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:45.747 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:56:45.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:56:45.749 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:56:45.749 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:56:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:45 smithi115 bash[45495]: cluster 2022-04-22T20:56:44.461954+0000 mgr.y (mgr.34107) 7200 : cluster [DBG] pgmap v8261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:45.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:45 smithi115 bash[45495]: audit 2022-04-22T20:56:45.049586+0000 mon.a (mon.0) 10702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:45 smithi115 bash[46934]: cluster 2022-04-22T20:56:44.461954+0000 mgr.y (mgr.34107) 7200 : cluster [DBG] pgmap v8261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:45.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:45 smithi115 bash[46934]: audit 2022-04-22T20:56:45.049586+0000 mon.a (mon.0) 10702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:45 smithi121 bash[36902]: cluster 2022-04-22T20:56:44.461954+0000 mgr.y (mgr.34107) 7200 : cluster [DBG] pgmap v8261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:45 smithi121 bash[36902]: audit 2022-04-22T20:56:45.049586+0000 mon.a (mon.0) 10702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[45495]: audit 2022-04-22T20:56:44.974754+0000 mgr.y (mgr.34107) 7201 : audit [DBG] from='client.42858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[45495]: audit 2022-04-22T20:56:45.321949+0000 mgr.y (mgr.34107) 7202 : audit [DBG] from='client.42864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[45495]: audit 2022-04-22T20:56:45.741947+0000 mon.c (mon.1) 269 : audit [DBG] from='client.? 172.21.15.115:0/3697652573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[46934]: audit 2022-04-22T20:56:44.974754+0000 mgr.y (mgr.34107) 7201 : audit [DBG] from='client.42858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[46934]: audit 2022-04-22T20:56:45.321949+0000 mgr.y (mgr.34107) 7202 : audit [DBG] from='client.42864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:46 smithi115 bash[46934]: audit 2022-04-22T20:56:45.741947+0000 mon.c (mon.1) 269 : audit [DBG] from='client.? 172.21.15.115:0/3697652573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:46 smithi121 bash[36902]: audit 2022-04-22T20:56:44.974754+0000 mgr.y (mgr.34107) 7201 : audit [DBG] from='client.42858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:46 smithi121 bash[36902]: audit 2022-04-22T20:56:45.321949+0000 mgr.y (mgr.34107) 7202 : audit [DBG] from='client.42864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:56:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:46 smithi121 bash[36902]: audit 2022-04-22T20:56:45.741947+0000 mon.c (mon.1) 269 : audit [DBG] from='client.? 172.21.15.115:0/3697652573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:56:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:47 smithi115 bash[46934]: cluster 2022-04-22T20:56:46.463075+0000 mgr.y (mgr.34107) 7203 : cluster [DBG] pgmap v8262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:47.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:47 smithi115 bash[45495]: cluster 2022-04-22T20:56:46.463075+0000 mgr.y (mgr.34107) 7203 : cluster [DBG] pgmap v8262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:47 smithi121 bash[36902]: cluster 2022-04-22T20:56:46.463075+0000 mgr.y (mgr.34107) 7203 : cluster [DBG] pgmap v8262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:49 smithi121 bash[36902]: cluster 2022-04-22T20:56:48.464194+0000 mgr.y (mgr.34107) 7204 : cluster [DBG] pgmap v8263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:49 smithi121 bash[36902]: audit 2022-04-22T20:56:48.695208+0000 mon.a (mon.0) 10703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:49 smithi115 bash[45495]: cluster 2022-04-22T20:56:48.464194+0000 mgr.y (mgr.34107) 7204 : cluster [DBG] pgmap v8263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:49 smithi115 bash[45495]: audit 2022-04-22T20:56:48.695208+0000 mon.a (mon.0) 10703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:49 smithi115 bash[46934]: cluster 2022-04-22T20:56:48.464194+0000 mgr.y (mgr.34107) 7204 : cluster [DBG] pgmap v8263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:49 smithi115 bash[46934]: audit 2022-04-22T20:56:48.695208+0000 mon.a (mon.0) 10703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:50 smithi121 bash[36902]: audit 2022-04-22T20:56:50.052113+0000 mon.a (mon.0) 10704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:50 smithi121 bash[36902]: cluster 2022-04-22T20:56:50.465246+0000 mgr.y (mgr.34107) 7205 : cluster [DBG] pgmap v8264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:50 smithi115 bash[46934]: audit 2022-04-22T20:56:50.052113+0000 mon.a (mon.0) 10704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:50 smithi115 bash[46934]: cluster 2022-04-22T20:56:50.465246+0000 mgr.y (mgr.34107) 7205 : cluster [DBG] pgmap v8264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:50 smithi115 bash[45495]: audit 2022-04-22T20:56:50.052113+0000 mon.a (mon.0) 10704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:50 smithi115 bash[45495]: cluster 2022-04-22T20:56:50.465246+0000 mgr.y (mgr.34107) 7205 : cluster [DBG] pgmap v8264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:52.430Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:53 smithi115 bash[45495]: cluster 2022-04-22T20:56:52.466255+0000 mgr.y (mgr.34107) 7206 : cluster [DBG] pgmap v8265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:53 smithi115 bash[46934]: cluster 2022-04-22T20:56:52.466255+0000 mgr.y (mgr.34107) 7206 : cluster [DBG] pgmap v8265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:53 smithi121 bash[36902]: cluster 2022-04-22T20:56:52.466255+0000 mgr.y (mgr.34107) 7206 : cluster [DBG] pgmap v8265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:56:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:56:54.307Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:56:54.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:54 smithi115 bash[45495]: audit 2022-04-22T20:56:53.698755+0000 mon.a (mon.0) 10705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:54 smithi115 bash[46934]: audit 2022-04-22T20:56:53.698755+0000 mon.a (mon.0) 10705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:54 smithi121 bash[36902]: audit 2022-04-22T20:56:53.698755+0000 mon.a (mon.0) 10705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:56:55.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:55 smithi115 bash[45495]: cluster 2022-04-22T20:56:54.467325+0000 mgr.y (mgr.34107) 7207 : cluster [DBG] pgmap v8266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:55.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:55 smithi115 bash[45495]: audit 2022-04-22T20:56:55.054887+0000 mon.a (mon.0) 10706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:55.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:55 smithi115 bash[46934]: cluster 2022-04-22T20:56:54.467325+0000 mgr.y (mgr.34107) 7207 : cluster [DBG] pgmap v8266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:55.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:55 smithi115 bash[46934]: audit 2022-04-22T20:56:55.054887+0000 mon.a (mon.0) 10706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:55 smithi121 bash[36902]: cluster 2022-04-22T20:56:54.467325+0000 mgr.y (mgr.34107) 7207 : cluster [DBG] pgmap v8266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:55 smithi121 bash[36902]: audit 2022-04-22T20:56:55.054887+0000 mon.a (mon.0) 10706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:56:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:57 smithi121 bash[36902]: cluster 2022-04-22T20:56:56.468376+0000 mgr.y (mgr.34107) 7208 : cluster [DBG] pgmap v8267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:58.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:57 smithi115 bash[46934]: cluster 2022-04-22T20:56:56.468376+0000 mgr.y (mgr.34107) 7208 : cluster [DBG] pgmap v8267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:56:58.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:57 smithi115 bash[45495]: cluster 2022-04-22T20:56:56.468376+0000 mgr.y (mgr.34107) 7208 : cluster [DBG] pgmap v8267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:59 smithi121 bash[36902]: cluster 2022-04-22T20:56:58.469503+0000 mgr.y (mgr.34107) 7209 : cluster [DBG] pgmap v8268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:56:59 smithi121 bash[36902]: audit 2022-04-22T20:56:58.702526+0000 mon.a (mon.0) 10707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:59 smithi115 bash[45495]: cluster 2022-04-22T20:56:58.469503+0000 mgr.y (mgr.34107) 7209 : cluster [DBG] pgmap v8268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:56:59 smithi115 bash[45495]: audit 2022-04-22T20:56:58.702526+0000 mon.a (mon.0) 10707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:00.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:59 smithi115 bash[46934]: cluster 2022-04-22T20:56:58.469503+0000 mgr.y (mgr.34107) 7209 : cluster [DBG] pgmap v8268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:00.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:56:59 smithi115 bash[46934]: audit 2022-04-22T20:56:58.702526+0000 mon.a (mon.0) 10707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:00 smithi121 bash[36902]: audit 2022-04-22T20:57:00.057751+0000 mon.a (mon.0) 10708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:00 smithi121 bash[36902]: cluster 2022-04-22T20:57:00.470567+0000 mgr.y (mgr.34107) 7210 : cluster [DBG] pgmap v8269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:00 smithi115 bash[45495]: audit 2022-04-22T20:57:00.057751+0000 mon.a (mon.0) 10708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:00 smithi115 bash[45495]: cluster 2022-04-22T20:57:00.470567+0000 mgr.y (mgr.34107) 7210 : cluster [DBG] pgmap v8269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:00 smithi115 bash[46934]: audit 2022-04-22T20:57:00.057751+0000 mon.a (mon.0) 10708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:00 smithi115 bash[46934]: cluster 2022-04-22T20:57:00.470567+0000 mgr.y (mgr.34107) 7210 : cluster [DBG] pgmap v8269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:02.431Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:03 smithi115 bash[45495]: cluster 2022-04-22T20:57:02.471653+0000 mgr.y (mgr.34107) 7211 : cluster [DBG] pgmap v8270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:03 smithi115 bash[46934]: cluster 2022-04-22T20:57:02.471653+0000 mgr.y (mgr.34107) 7211 : cluster [DBG] pgmap v8270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:03 smithi121 bash[36902]: cluster 2022-04-22T20:57:02.471653+0000 mgr.y (mgr.34107) 7211 : cluster [DBG] pgmap v8270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:04.306Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:04 smithi115 bash[45495]: audit 2022-04-22T20:57:03.706338+0000 mon.a (mon.0) 10709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:04 smithi115 bash[46934]: audit 2022-04-22T20:57:03.706338+0000 mon.a (mon.0) 10709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:04 smithi121 bash[36902]: audit 2022-04-22T20:57:03.706338+0000 mon.a (mon.0) 10709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:05 smithi121 bash[36902]: cluster 2022-04-22T20:57:04.472750+0000 mgr.y (mgr.34107) 7212 : cluster [DBG] pgmap v8271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:05 smithi121 bash[36902]: audit 2022-04-22T20:57:05.060639+0000 mon.a (mon.0) 10710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:05 smithi115 bash[45495]: cluster 2022-04-22T20:57:04.472750+0000 mgr.y (mgr.34107) 7212 : cluster [DBG] pgmap v8271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:05 smithi115 bash[45495]: audit 2022-04-22T20:57:05.060639+0000 mon.a (mon.0) 10710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:06.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:05 smithi115 bash[46934]: cluster 2022-04-22T20:57:04.472750+0000 mgr.y (mgr.34107) 7212 : cluster [DBG] pgmap v8271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:05 smithi115 bash[46934]: audit 2022-04-22T20:57:05.060639+0000 mon.a (mon.0) 10710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:07 smithi121 bash[36902]: cluster 2022-04-22T20:57:06.473837+0000 mgr.y (mgr.34107) 7213 : cluster [DBG] pgmap v8272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:08.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:07 smithi115 bash[45495]: cluster 2022-04-22T20:57:06.473837+0000 mgr.y (mgr.34107) 7213 : cluster [DBG] pgmap v8272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:07 smithi115 bash[46934]: cluster 2022-04-22T20:57:06.473837+0000 mgr.y (mgr.34107) 7213 : cluster [DBG] pgmap v8272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:09 smithi121 bash[36902]: cluster 2022-04-22T20:57:08.474975+0000 mgr.y (mgr.34107) 7214 : cluster [DBG] pgmap v8273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:09 smithi121 bash[36902]: audit 2022-04-22T20:57:08.710011+0000 mon.a (mon.0) 10711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:09 smithi115 bash[45495]: cluster 2022-04-22T20:57:08.474975+0000 mgr.y (mgr.34107) 7214 : cluster [DBG] pgmap v8273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:09 smithi115 bash[45495]: audit 2022-04-22T20:57:08.710011+0000 mon.a (mon.0) 10711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:09 smithi115 bash[46934]: cluster 2022-04-22T20:57:08.474975+0000 mgr.y (mgr.34107) 7214 : cluster [DBG] pgmap v8273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:10.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:09 smithi115 bash[46934]: audit 2022-04-22T20:57:08.710011+0000 mon.a (mon.0) 10711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:10 smithi121 bash[36902]: audit 2022-04-22T20:57:10.063849+0000 mon.a (mon.0) 10712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:10 smithi121 bash[36902]: cluster 2022-04-22T20:57:10.476046+0000 mgr.y (mgr.34107) 7215 : cluster [DBG] pgmap v8274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:10 smithi115 bash[45495]: audit 2022-04-22T20:57:10.063849+0000 mon.a (mon.0) 10712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:10 smithi115 bash[45495]: cluster 2022-04-22T20:57:10.476046+0000 mgr.y (mgr.34107) 7215 : cluster [DBG] pgmap v8274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:10 smithi115 bash[46934]: audit 2022-04-22T20:57:10.063849+0000 mon.a (mon.0) 10712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:11.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:10 smithi115 bash[46934]: cluster 2022-04-22T20:57:10.476046+0000 mgr.y (mgr.34107) 7215 : cluster [DBG] pgmap v8274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:12.432Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:13 smithi115 bash[45495]: cluster 2022-04-22T20:57:12.477131+0000 mgr.y (mgr.34107) 7216 : cluster [DBG] pgmap v8275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:13 smithi115 bash[46934]: cluster 2022-04-22T20:57:12.477131+0000 mgr.y (mgr.34107) 7216 : cluster [DBG] pgmap v8275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:13 smithi121 bash[36902]: cluster 2022-04-22T20:57:12.477131+0000 mgr.y (mgr.34107) 7216 : cluster [DBG] pgmap v8275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:14 smithi115 bash[45495]: audit 2022-04-22T20:57:13.713587+0000 mon.a (mon.0) 10713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:14 smithi115 bash[46934]: audit 2022-04-22T20:57:13.713587+0000 mon.a (mon.0) 10713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:14.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:14.308Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:14 smithi121 bash[36902]: audit 2022-04-22T20:57:13.713587+0000 mon.a (mon.0) 10713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:15 smithi121 bash[36902]: cluster 2022-04-22T20:57:14.478257+0000 mgr.y (mgr.34107) 7217 : cluster [DBG] pgmap v8276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:15 smithi121 bash[36902]: audit 2022-04-22T20:57:15.066612+0000 mon.a (mon.0) 10714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:16.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:15 smithi115 bash[46934]: cluster 2022-04-22T20:57:14.478257+0000 mgr.y (mgr.34107) 7217 : cluster [DBG] pgmap v8276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:16.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:15 smithi115 bash[46934]: audit 2022-04-22T20:57:15.066612+0000 mon.a (mon.0) 10714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:16.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:15 smithi115 bash[45495]: cluster 2022-04-22T20:57:14.478257+0000 mgr.y (mgr.34107) 7217 : cluster [DBG] pgmap v8276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:16.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:15 smithi115 bash[45495]: audit 2022-04-22T20:57:15.066612+0000 mon.a (mon.0) 10714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:16.141 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:57:16.483 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:57:16.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:57:16.485 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:57:16.883 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:57:16.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:57:16.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:57:16.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:16.885 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:57:16.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:57:16.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:57:16.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:57:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:16 smithi121 bash[36902]: audit 2022-04-22T20:57:15.697816+0000 mon.a (mon.0) 10715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:57:17.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:16 smithi121 bash[36902]: audit 2022-04-22T20:57:15.698320+0000 mon.a (mon.0) 10716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:57:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:16 smithi115 bash[45495]: audit 2022-04-22T20:57:15.697816+0000 mon.a (mon.0) 10715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:57:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:16 smithi115 bash[45495]: audit 2022-04-22T20:57:15.698320+0000 mon.a (mon.0) 10716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:57:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:16 smithi115 bash[46934]: audit 2022-04-22T20:57:15.697816+0000 mon.a (mon.0) 10715 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:57:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:16 smithi115 bash[46934]: audit 2022-04-22T20:57:15.698320+0000 mon.a (mon.0) 10716 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:57:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:17 smithi121 bash[36902]: audit 2022-04-22T20:57:16.118912+0000 mgr.y (mgr.34107) 7218 : audit [DBG] from='client.42876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:17 smithi121 bash[36902]: audit 2022-04-22T20:57:16.458914+0000 mgr.y (mgr.34107) 7219 : audit [DBG] from='client.42882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:17 smithi121 bash[36902]: cluster 2022-04-22T20:57:16.478844+0000 mgr.y (mgr.34107) 7220 : cluster [DBG] pgmap v8277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:17 smithi121 bash[36902]: audit 2022-04-22T20:57:16.878296+0000 mon.a (mon.0) 10717 : audit [DBG] from='client.? 172.21.15.115:0/2347178529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[45495]: audit 2022-04-22T20:57:16.118912+0000 mgr.y (mgr.34107) 7218 : audit [DBG] from='client.42876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[45495]: audit 2022-04-22T20:57:16.458914+0000 mgr.y (mgr.34107) 7219 : audit [DBG] from='client.42882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[45495]: cluster 2022-04-22T20:57:16.478844+0000 mgr.y (mgr.34107) 7220 : cluster [DBG] pgmap v8277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[45495]: audit 2022-04-22T20:57:16.878296+0000 mon.a (mon.0) 10717 : audit [DBG] from='client.? 172.21.15.115:0/2347178529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:18.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[46934]: audit 2022-04-22T20:57:16.118912+0000 mgr.y (mgr.34107) 7218 : audit [DBG] from='client.42876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[46934]: audit 2022-04-22T20:57:16.458914+0000 mgr.y (mgr.34107) 7219 : audit [DBG] from='client.42882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:18.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[46934]: cluster 2022-04-22T20:57:16.478844+0000 mgr.y (mgr.34107) 7220 : cluster [DBG] pgmap v8277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:18.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:17 smithi115 bash[46934]: audit 2022-04-22T20:57:16.878296+0000 mon.a (mon.0) 10717 : audit [DBG] from='client.? 172.21.15.115:0/2347178529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:18 smithi121 bash[36902]: cluster 2022-04-22T20:57:18.479873+0000 mgr.y (mgr.34107) 7221 : cluster [DBG] pgmap v8278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:19.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:18 smithi115 bash[45495]: cluster 2022-04-22T20:57:18.479873+0000 mgr.y (mgr.34107) 7221 : cluster [DBG] pgmap v8278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:18 smithi115 bash[46934]: cluster 2022-04-22T20:57:18.479873+0000 mgr.y (mgr.34107) 7221 : cluster [DBG] pgmap v8278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:19 smithi121 bash[36902]: audit 2022-04-22T20:57:18.717269+0000 mon.a (mon.0) 10718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:19 smithi115 bash[45495]: audit 2022-04-22T20:57:18.717269+0000 mon.a (mon.0) 10718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:19 smithi115 bash[46934]: audit 2022-04-22T20:57:18.717269+0000 mon.a (mon.0) 10718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:20 smithi121 bash[36902]: audit 2022-04-22T20:57:20.069739+0000 mon.a (mon.0) 10719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:20 smithi121 bash[36902]: cluster 2022-04-22T20:57:20.480851+0000 mgr.y (mgr.34107) 7222 : cluster [DBG] pgmap v8279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:20 smithi115 bash[46934]: audit 2022-04-22T20:57:20.069739+0000 mon.a (mon.0) 10719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:20 smithi115 bash[46934]: cluster 2022-04-22T20:57:20.480851+0000 mgr.y (mgr.34107) 7222 : cluster [DBG] pgmap v8279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:20 smithi115 bash[45495]: audit 2022-04-22T20:57:20.069739+0000 mon.a (mon.0) 10719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:20 smithi115 bash[45495]: cluster 2022-04-22T20:57:20.480851+0000 mgr.y (mgr.34107) 7222 : cluster [DBG] pgmap v8279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:22.434Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:23 smithi121 bash[36902]: cluster 2022-04-22T20:57:22.481831+0000 mgr.y (mgr.34107) 7223 : cluster [DBG] pgmap v8280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:23 smithi115 bash[46934]: cluster 2022-04-22T20:57:22.481831+0000 mgr.y (mgr.34107) 7223 : cluster [DBG] pgmap v8280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:24.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:23 smithi115 bash[45495]: cluster 2022-04-22T20:57:22.481831+0000 mgr.y (mgr.34107) 7223 : cluster [DBG] pgmap v8280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:24 smithi115 bash[45495]: audit 2022-04-22T20:57:23.720986+0000 mon.a (mon.0) 10720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:24.309Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:24 smithi115 bash[46934]: audit 2022-04-22T20:57:23.720986+0000 mon.a (mon.0) 10720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:24 smithi121 bash[36902]: audit 2022-04-22T20:57:23.720986+0000 mon.a (mon.0) 10720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:25 smithi121 bash[36902]: cluster 2022-04-22T20:57:24.482893+0000 mgr.y (mgr.34107) 7224 : cluster [DBG] pgmap v8281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:25 smithi121 bash[36902]: audit 2022-04-22T20:57:25.072679+0000 mon.a (mon.0) 10721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:26.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:25 smithi115 bash[45495]: cluster 2022-04-22T20:57:24.482893+0000 mgr.y (mgr.34107) 7224 : cluster [DBG] pgmap v8281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:26.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:25 smithi115 bash[45495]: audit 2022-04-22T20:57:25.072679+0000 mon.a (mon.0) 10721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:26.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:25 smithi115 bash[46934]: cluster 2022-04-22T20:57:24.482893+0000 mgr.y (mgr.34107) 7224 : cluster [DBG] pgmap v8281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:26.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:25 smithi115 bash[46934]: audit 2022-04-22T20:57:25.072679+0000 mon.a (mon.0) 10721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:27 smithi121 bash[36902]: cluster 2022-04-22T20:57:26.483953+0000 mgr.y (mgr.34107) 7225 : cluster [DBG] pgmap v8282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:27 smithi115 bash[45495]: cluster 2022-04-22T20:57:26.483953+0000 mgr.y (mgr.34107) 7225 : cluster [DBG] pgmap v8282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:27 smithi115 bash[46934]: cluster 2022-04-22T20:57:26.483953+0000 mgr.y (mgr.34107) 7225 : cluster [DBG] pgmap v8282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:29 smithi121 bash[36902]: cluster 2022-04-22T20:57:28.485031+0000 mgr.y (mgr.34107) 7226 : cluster [DBG] pgmap v8283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:29 smithi121 bash[36902]: audit 2022-04-22T20:57:28.724874+0000 mon.a (mon.0) 10722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:29 smithi115 bash[45495]: cluster 2022-04-22T20:57:28.485031+0000 mgr.y (mgr.34107) 7226 : cluster [DBG] pgmap v8283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:29 smithi115 bash[45495]: audit 2022-04-22T20:57:28.724874+0000 mon.a (mon.0) 10722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:30.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:29 smithi115 bash[46934]: cluster 2022-04-22T20:57:28.485031+0000 mgr.y (mgr.34107) 7226 : cluster [DBG] pgmap v8283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:29 smithi115 bash[46934]: audit 2022-04-22T20:57:28.724874+0000 mon.a (mon.0) 10722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:30.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[46934]: audit 2022-04-22T20:57:30.075469+0000 mon.a (mon.0) 10723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:30.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[46934]: cluster 2022-04-22T20:57:30.486075+0000 mgr.y (mgr.34107) 7227 : cluster [DBG] pgmap v8284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[46934]: audit 2022-04-22T20:57:30.653653+0000 mon.a (mon.0) 10724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:57:30.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[46934]: audit 2022-04-22T20:57:30.656684+0000 mon.a (mon.0) 10725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:57:30.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[46934]: audit 2022-04-22T20:57:30.658283+0000 mon.a (mon.0) 10726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:57:30.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[45495]: audit 2022-04-22T20:57:30.075469+0000 mon.a (mon.0) 10723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:30.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[45495]: cluster 2022-04-22T20:57:30.486075+0000 mgr.y (mgr.34107) 7227 : cluster [DBG] pgmap v8284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[45495]: audit 2022-04-22T20:57:30.653653+0000 mon.a (mon.0) 10724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:57:30.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[45495]: audit 2022-04-22T20:57:30.656684+0000 mon.a (mon.0) 10725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:57:30.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:30 smithi115 bash[45495]: audit 2022-04-22T20:57:30.658283+0000 mon.a (mon.0) 10726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:57:30.995 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:30 smithi121 bash[36902]: audit 2022-04-22T20:57:30.075469+0000 mon.a (mon.0) 10723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:30.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:30 smithi121 bash[36902]: cluster 2022-04-22T20:57:30.486075+0000 mgr.y (mgr.34107) 7227 : cluster [DBG] pgmap v8284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:30.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:30 smithi121 bash[36902]: audit 2022-04-22T20:57:30.653653+0000 mon.a (mon.0) 10724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:57:30.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:30 smithi121 bash[36902]: audit 2022-04-22T20:57:30.656684+0000 mon.a (mon.0) 10725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:57:30.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:30 smithi121 bash[36902]: audit 2022-04-22T20:57:30.658283+0000 mon.a (mon.0) 10726 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:32 smithi121 bash[36902]: audit 2022-04-22T20:57:31.002477+0000 mon.a (mon.0) 10727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:57:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:31 smithi115 bash[45495]: audit 2022-04-22T20:57:31.002477+0000 mon.a (mon.0) 10727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:57:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:32 smithi115 bash[46934]: audit 2022-04-22T20:57:31.002477+0000 mon.a (mon.0) 10727 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:57:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:32.435Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:33 smithi115 bash[45495]: cluster 2022-04-22T20:57:32.487174+0000 mgr.y (mgr.34107) 7228 : cluster [DBG] pgmap v8285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:33 smithi115 bash[46934]: cluster 2022-04-22T20:57:32.487174+0000 mgr.y (mgr.34107) 7228 : cluster [DBG] pgmap v8285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:33 smithi121 bash[36902]: cluster 2022-04-22T20:57:32.487174+0000 mgr.y (mgr.34107) 7228 : cluster [DBG] pgmap v8285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:34 smithi115 bash[45495]: audit 2022-04-22T20:57:33.728649+0000 mon.a (mon.0) 10728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:34 smithi115 bash[46934]: audit 2022-04-22T20:57:33.728649+0000 mon.a (mon.0) 10728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:34 smithi121 bash[36902]: audit 2022-04-22T20:57:33.728649+0000 mon.a (mon.0) 10728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:34.310Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:35 smithi115 bash[45495]: cluster 2022-04-22T20:57:34.488318+0000 mgr.y (mgr.34107) 7229 : cluster [DBG] pgmap v8286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:35 smithi115 bash[46934]: cluster 2022-04-22T20:57:34.488318+0000 mgr.y (mgr.34107) 7229 : cluster [DBG] pgmap v8286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:35 smithi121 bash[36902]: cluster 2022-04-22T20:57:34.488318+0000 mgr.y (mgr.34107) 7229 : cluster [DBG] pgmap v8286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:36 smithi115 bash[45495]: audit 2022-04-22T20:57:35.078455+0000 mon.a (mon.0) 10729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:36 smithi115 bash[46934]: audit 2022-04-22T20:57:35.078455+0000 mon.a (mon.0) 10729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:36 smithi121 bash[36902]: audit 2022-04-22T20:57:35.078455+0000 mon.a (mon.0) 10729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:37 smithi115 bash[45495]: cluster 2022-04-22T20:57:36.489266+0000 mgr.y (mgr.34107) 7230 : cluster [DBG] pgmap v8287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:37 smithi115 bash[46934]: cluster 2022-04-22T20:57:36.489266+0000 mgr.y (mgr.34107) 7230 : cluster [DBG] pgmap v8287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:37 smithi121 bash[36902]: cluster 2022-04-22T20:57:36.489266+0000 mgr.y (mgr.34107) 7230 : cluster [DBG] pgmap v8287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:39 smithi121 bash[36902]: cluster 2022-04-22T20:57:38.490354+0000 mgr.y (mgr.34107) 7231 : cluster [DBG] pgmap v8288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:39 smithi121 bash[36902]: audit 2022-04-22T20:57:38.731970+0000 mon.a (mon.0) 10730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:39 smithi115 bash[45495]: cluster 2022-04-22T20:57:38.490354+0000 mgr.y (mgr.34107) 7231 : cluster [DBG] pgmap v8288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:39 smithi115 bash[45495]: audit 2022-04-22T20:57:38.731970+0000 mon.a (mon.0) 10730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:39 smithi115 bash[46934]: cluster 2022-04-22T20:57:38.490354+0000 mgr.y (mgr.34107) 7231 : cluster [DBG] pgmap v8288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:40.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:39 smithi115 bash[46934]: audit 2022-04-22T20:57:38.731970+0000 mon.a (mon.0) 10730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:40 smithi121 bash[36902]: audit 2022-04-22T20:57:40.081567+0000 mon.a (mon.0) 10731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:40 smithi121 bash[36902]: cluster 2022-04-22T20:57:40.491408+0000 mgr.y (mgr.34107) 7232 : cluster [DBG] pgmap v8289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:40 smithi115 bash[46934]: audit 2022-04-22T20:57:40.081567+0000 mon.a (mon.0) 10731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:40 smithi115 bash[46934]: cluster 2022-04-22T20:57:40.491408+0000 mgr.y (mgr.34107) 7232 : cluster [DBG] pgmap v8289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:40 smithi115 bash[45495]: audit 2022-04-22T20:57:40.081567+0000 mon.a (mon.0) 10731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:40 smithi115 bash[45495]: cluster 2022-04-22T20:57:40.491408+0000 mgr.y (mgr.34107) 7232 : cluster [DBG] pgmap v8289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:42.436Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:43 smithi121 bash[36902]: cluster 2022-04-22T20:57:42.492485+0000 mgr.y (mgr.34107) 7233 : cluster [DBG] pgmap v8290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:44.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:43 smithi115 bash[45495]: cluster 2022-04-22T20:57:42.492485+0000 mgr.y (mgr.34107) 7233 : cluster [DBG] pgmap v8290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:44.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:43 smithi115 bash[46934]: cluster 2022-04-22T20:57:42.492485+0000 mgr.y (mgr.34107) 7233 : cluster [DBG] pgmap v8290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:44 smithi115 bash[45495]: audit 2022-04-22T20:57:43.734652+0000 mon.a (mon.0) 10732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:44.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:44.310Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:44 smithi115 bash[46934]: audit 2022-04-22T20:57:43.734652+0000 mon.a (mon.0) 10732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:44 smithi121 bash[36902]: audit 2022-04-22T20:57:43.734652+0000 mon.a (mon.0) 10732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:45 smithi121 bash[36902]: cluster 2022-04-22T20:57:44.493614+0000 mgr.y (mgr.34107) 7234 : cluster [DBG] pgmap v8291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:45 smithi121 bash[36902]: audit 2022-04-22T20:57:45.084455+0000 mon.a (mon.0) 10733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:46.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:45 smithi115 bash[45495]: cluster 2022-04-22T20:57:44.493614+0000 mgr.y (mgr.34107) 7234 : cluster [DBG] pgmap v8291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:46.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:45 smithi115 bash[45495]: audit 2022-04-22T20:57:45.084455+0000 mon.a (mon.0) 10733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:45 smithi115 bash[46934]: cluster 2022-04-22T20:57:44.493614+0000 mgr.y (mgr.34107) 7234 : cluster [DBG] pgmap v8291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:46.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:45 smithi115 bash[46934]: audit 2022-04-22T20:57:45.084455+0000 mon.a (mon.0) 10733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:46 smithi121 bash[36902]: cluster 2022-04-22T20:57:46.494799+0000 mgr.y (mgr.34107) 7235 : cluster [DBG] pgmap v8292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:46 smithi115 bash[46934]: cluster 2022-04-22T20:57:46.494799+0000 mgr.y (mgr.34107) 7235 : cluster [DBG] pgmap v8292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:47.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:46 smithi115 bash[45495]: cluster 2022-04-22T20:57:46.494799+0000 mgr.y (mgr.34107) 7235 : cluster [DBG] pgmap v8292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:47.264 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:57:47.611 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:57:47.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:57:47.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:57:47.614 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:48.010 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:57:48.011 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:57:48.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:47 smithi121 bash[36902]: audit 2022-04-22T20:57:47.242077+0000 mgr.y (mgr.34107) 7236 : audit [DBG] from='client.42894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:48.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:47 smithi115 bash[45495]: audit 2022-04-22T20:57:47.242077+0000 mgr.y (mgr.34107) 7236 : audit [DBG] from='client.42894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:48.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:47 smithi115 bash[46934]: audit 2022-04-22T20:57:47.242077+0000 mgr.y (mgr.34107) 7236 : audit [DBG] from='client.42894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:48 smithi121 bash[36902]: audit 2022-04-22T20:57:47.587587+0000 mgr.y (mgr.34107) 7237 : audit [DBG] from='client.52831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:48 smithi121 bash[36902]: audit 2022-04-22T20:57:48.004705+0000 mon.a (mon.0) 10734 : audit [DBG] from='client.? 172.21.15.115:0/1901321198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:48 smithi121 bash[36902]: cluster 2022-04-22T20:57:48.495934+0000 mgr.y (mgr.34107) 7238 : cluster [DBG] pgmap v8293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[45495]: audit 2022-04-22T20:57:47.587587+0000 mgr.y (mgr.34107) 7237 : audit [DBG] from='client.52831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[45495]: audit 2022-04-22T20:57:48.004705+0000 mon.a (mon.0) 10734 : audit [DBG] from='client.? 172.21.15.115:0/1901321198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:49.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[45495]: cluster 2022-04-22T20:57:48.495934+0000 mgr.y (mgr.34107) 7238 : cluster [DBG] pgmap v8293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[46934]: audit 2022-04-22T20:57:47.587587+0000 mgr.y (mgr.34107) 7237 : audit [DBG] from='client.52831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:57:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[46934]: audit 2022-04-22T20:57:48.004705+0000 mon.a (mon.0) 10734 : audit [DBG] from='client.? 172.21.15.115:0/1901321198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:57:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:48 smithi115 bash[46934]: cluster 2022-04-22T20:57:48.495934+0000 mgr.y (mgr.34107) 7238 : cluster [DBG] pgmap v8293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:49 smithi121 bash[36902]: audit 2022-04-22T20:57:48.738447+0000 mon.a (mon.0) 10735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:50.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:49 smithi115 bash[46934]: audit 2022-04-22T20:57:48.738447+0000 mon.a (mon.0) 10735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:49 smithi115 bash[45495]: audit 2022-04-22T20:57:48.738447+0000 mon.a (mon.0) 10735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:50 smithi121 bash[36902]: audit 2022-04-22T20:57:50.087369+0000 mon.a (mon.0) 10736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:50 smithi121 bash[36902]: cluster 2022-04-22T20:57:50.496977+0000 mgr.y (mgr.34107) 7239 : cluster [DBG] pgmap v8294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:50 smithi115 bash[45495]: audit 2022-04-22T20:57:50.087369+0000 mon.a (mon.0) 10736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:50 smithi115 bash[45495]: cluster 2022-04-22T20:57:50.496977+0000 mgr.y (mgr.34107) 7239 : cluster [DBG] pgmap v8294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:50 smithi115 bash[46934]: audit 2022-04-22T20:57:50.087369+0000 mon.a (mon.0) 10736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:50 smithi115 bash[46934]: cluster 2022-04-22T20:57:50.496977+0000 mgr.y (mgr.34107) 7239 : cluster [DBG] pgmap v8294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:52.435Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:53 smithi121 bash[36902]: cluster 2022-04-22T20:57:52.498058+0000 mgr.y (mgr.34107) 7240 : cluster [DBG] pgmap v8295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:54.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:53 smithi115 bash[45495]: cluster 2022-04-22T20:57:52.498058+0000 mgr.y (mgr.34107) 7240 : cluster [DBG] pgmap v8295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:54.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:53 smithi115 bash[46934]: cluster 2022-04-22T20:57:52.498058+0000 mgr.y (mgr.34107) 7240 : cluster [DBG] pgmap v8295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:54.567 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:57:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:57:54.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:57:54.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:54 smithi115 bash[45495]: audit 2022-04-22T20:57:53.742013+0000 mon.a (mon.0) 10737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:54 smithi121 bash[36902]: audit 2022-04-22T20:57:53.742013+0000 mon.a (mon.0) 10737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:55.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:54 smithi115 bash[46934]: audit 2022-04-22T20:57:53.742013+0000 mon.a (mon.0) 10737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:57:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:55 smithi121 bash[36902]: cluster 2022-04-22T20:57:54.499146+0000 mgr.y (mgr.34107) 7241 : cluster [DBG] pgmap v8296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:55 smithi121 bash[36902]: audit 2022-04-22T20:57:55.090291+0000 mon.a (mon.0) 10738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:55 smithi115 bash[45495]: cluster 2022-04-22T20:57:54.499146+0000 mgr.y (mgr.34107) 7241 : cluster [DBG] pgmap v8296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:55 smithi115 bash[45495]: audit 2022-04-22T20:57:55.090291+0000 mon.a (mon.0) 10738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:55 smithi115 bash[46934]: cluster 2022-04-22T20:57:54.499146+0000 mgr.y (mgr.34107) 7241 : cluster [DBG] pgmap v8296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:55 smithi115 bash[46934]: audit 2022-04-22T20:57:55.090291+0000 mon.a (mon.0) 10738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:57:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:56 smithi121 bash[36902]: cluster 2022-04-22T20:57:56.500303+0000 mgr.y (mgr.34107) 7242 : cluster [DBG] pgmap v8297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:57.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:56 smithi115 bash[45495]: cluster 2022-04-22T20:57:56.500303+0000 mgr.y (mgr.34107) 7242 : cluster [DBG] pgmap v8297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:57:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:56 smithi115 bash[46934]: cluster 2022-04-22T20:57:56.500303+0000 mgr.y (mgr.34107) 7242 : cluster [DBG] pgmap v8297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:59 smithi121 bash[36902]: cluster 2022-04-22T20:57:58.501432+0000 mgr.y (mgr.34107) 7243 : cluster [DBG] pgmap v8298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:57:59 smithi121 bash[36902]: audit 2022-04-22T20:57:58.745676+0000 mon.a (mon.0) 10739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:59 smithi115 bash[46934]: cluster 2022-04-22T20:57:58.501432+0000 mgr.y (mgr.34107) 7243 : cluster [DBG] pgmap v8298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:57:59 smithi115 bash[46934]: audit 2022-04-22T20:57:58.745676+0000 mon.a (mon.0) 10739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:59 smithi115 bash[45495]: cluster 2022-04-22T20:57:58.501432+0000 mgr.y (mgr.34107) 7243 : cluster [DBG] pgmap v8298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:57:59 smithi115 bash[45495]: audit 2022-04-22T20:57:58.745676+0000 mon.a (mon.0) 10739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:00 smithi121 bash[36902]: audit 2022-04-22T20:58:00.092986+0000 mon.a (mon.0) 10740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:00 smithi121 bash[36902]: cluster 2022-04-22T20:58:00.502490+0000 mgr.y (mgr.34107) 7244 : cluster [DBG] pgmap v8299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:00 smithi115 bash[46934]: audit 2022-04-22T20:58:00.092986+0000 mon.a (mon.0) 10740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:00 smithi115 bash[46934]: cluster 2022-04-22T20:58:00.502490+0000 mgr.y (mgr.34107) 7244 : cluster [DBG] pgmap v8299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:00 smithi115 bash[45495]: audit 2022-04-22T20:58:00.092986+0000 mon.a (mon.0) 10740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:00 smithi115 bash[45495]: cluster 2022-04-22T20:58:00.502490+0000 mgr.y (mgr.34107) 7244 : cluster [DBG] pgmap v8299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:02.435Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:03 smithi121 bash[36902]: cluster 2022-04-22T20:58:02.503595+0000 mgr.y (mgr.34107) 7245 : cluster [DBG] pgmap v8300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:04.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:03 smithi115 bash[45495]: cluster 2022-04-22T20:58:02.503595+0000 mgr.y (mgr.34107) 7245 : cluster [DBG] pgmap v8300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:03 smithi115 bash[46934]: cluster 2022-04-22T20:58:02.503595+0000 mgr.y (mgr.34107) 7245 : cluster [DBG] pgmap v8300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:04.575 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:04.311Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:04 smithi121 bash[36902]: audit 2022-04-22T20:58:03.749315+0000 mon.a (mon.0) 10741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:04 smithi115 bash[45495]: audit 2022-04-22T20:58:03.749315+0000 mon.a (mon.0) 10741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:04 smithi115 bash[46934]: audit 2022-04-22T20:58:03.749315+0000 mon.a (mon.0) 10741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:05 smithi121 bash[36902]: cluster 2022-04-22T20:58:04.504611+0000 mgr.y (mgr.34107) 7246 : cluster [DBG] pgmap v8301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:05 smithi121 bash[36902]: audit 2022-04-22T20:58:05.096417+0000 mon.a (mon.0) 10742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:05 smithi115 bash[45495]: cluster 2022-04-22T20:58:04.504611+0000 mgr.y (mgr.34107) 7246 : cluster [DBG] pgmap v8301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:05 smithi115 bash[45495]: audit 2022-04-22T20:58:05.096417+0000 mon.a (mon.0) 10742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:06.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:05 smithi115 bash[46934]: cluster 2022-04-22T20:58:04.504611+0000 mgr.y (mgr.34107) 7246 : cluster [DBG] pgmap v8301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:05 smithi115 bash[46934]: audit 2022-04-22T20:58:05.096417+0000 mon.a (mon.0) 10742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:06 smithi121 bash[36902]: cluster 2022-04-22T20:58:06.505741+0000 mgr.y (mgr.34107) 7247 : cluster [DBG] pgmap v8302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:06 smithi115 bash[45495]: cluster 2022-04-22T20:58:06.505741+0000 mgr.y (mgr.34107) 7247 : cluster [DBG] pgmap v8302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:06 smithi115 bash[46934]: cluster 2022-04-22T20:58:06.505741+0000 mgr.y (mgr.34107) 7247 : cluster [DBG] pgmap v8302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:09 smithi121 bash[36902]: cluster 2022-04-22T20:58:08.506844+0000 mgr.y (mgr.34107) 7248 : cluster [DBG] pgmap v8303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:09 smithi121 bash[36902]: audit 2022-04-22T20:58:08.752921+0000 mon.a (mon.0) 10743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:09 smithi115 bash[46934]: cluster 2022-04-22T20:58:08.506844+0000 mgr.y (mgr.34107) 7248 : cluster [DBG] pgmap v8303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:09 smithi115 bash[46934]: audit 2022-04-22T20:58:08.752921+0000 mon.a (mon.0) 10743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:09 smithi115 bash[45495]: cluster 2022-04-22T20:58:08.506844+0000 mgr.y (mgr.34107) 7248 : cluster [DBG] pgmap v8303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:10.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:09 smithi115 bash[45495]: audit 2022-04-22T20:58:08.752921+0000 mon.a (mon.0) 10743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:10 smithi121 bash[36902]: audit 2022-04-22T20:58:10.099560+0000 mon.a (mon.0) 10744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:10 smithi121 bash[36902]: cluster 2022-04-22T20:58:10.508054+0000 mgr.y (mgr.34107) 7249 : cluster [DBG] pgmap v8304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:10 smithi115 bash[46934]: audit 2022-04-22T20:58:10.099560+0000 mon.a (mon.0) 10744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:10 smithi115 bash[46934]: cluster 2022-04-22T20:58:10.508054+0000 mgr.y (mgr.34107) 7249 : cluster [DBG] pgmap v8304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:10 smithi115 bash[45495]: audit 2022-04-22T20:58:10.099560+0000 mon.a (mon.0) 10744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:10 smithi115 bash[45495]: cluster 2022-04-22T20:58:10.508054+0000 mgr.y (mgr.34107) 7249 : cluster [DBG] pgmap v8304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:12.436Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:13 smithi121 bash[36902]: cluster 2022-04-22T20:58:12.509131+0000 mgr.y (mgr.34107) 7250 : cluster [DBG] pgmap v8305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:14.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:13 smithi115 bash[46934]: cluster 2022-04-22T20:58:12.509131+0000 mgr.y (mgr.34107) 7250 : cluster [DBG] pgmap v8305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:14.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:13 smithi115 bash[45495]: cluster 2022-04-22T20:58:12.509131+0000 mgr.y (mgr.34107) 7250 : cluster [DBG] pgmap v8305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:14.579 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:14.312Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:14 smithi121 bash[36902]: audit 2022-04-22T20:58:13.756554+0000 mon.a (mon.0) 10745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:15.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:14 smithi115 bash[45495]: audit 2022-04-22T20:58:13.756554+0000 mon.a (mon.0) 10745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:14 smithi115 bash[46934]: audit 2022-04-22T20:58:13.756554+0000 mon.a (mon.0) 10745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:15 smithi121 bash[36902]: cluster 2022-04-22T20:58:14.510171+0000 mgr.y (mgr.34107) 7251 : cluster [DBG] pgmap v8306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:15 smithi121 bash[36902]: audit 2022-04-22T20:58:15.102644+0000 mon.a (mon.0) 10746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:15 smithi115 bash[45495]: cluster 2022-04-22T20:58:14.510171+0000 mgr.y (mgr.34107) 7251 : cluster [DBG] pgmap v8306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:16.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:15 smithi115 bash[45495]: audit 2022-04-22T20:58:15.102644+0000 mon.a (mon.0) 10746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:16.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:15 smithi115 bash[46934]: cluster 2022-04-22T20:58:14.510171+0000 mgr.y (mgr.34107) 7251 : cluster [DBG] pgmap v8306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:16.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:15 smithi115 bash[46934]: audit 2022-04-22T20:58:15.102644+0000 mon.a (mon.0) 10746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:16 smithi121 bash[36902]: audit 2022-04-22T20:58:15.709356+0000 mon.a (mon.0) 10747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:58:17.079 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:16 smithi121 bash[36902]: audit 2022-04-22T20:58:15.710417+0000 mon.a (mon.0) 10748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:58:17.079 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:16 smithi121 bash[36902]: cluster 2022-04-22T20:58:16.511182+0000 mgr.y (mgr.34107) 7252 : cluster [DBG] pgmap v8307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:17.079 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[45495]: audit 2022-04-22T20:58:15.709356+0000 mon.a (mon.0) 10747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:58:17.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[45495]: audit 2022-04-22T20:58:15.710417+0000 mon.a (mon.0) 10748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:58:17.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[45495]: cluster 2022-04-22T20:58:16.511182+0000 mgr.y (mgr.34107) 7252 : cluster [DBG] pgmap v8307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:17.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[46934]: audit 2022-04-22T20:58:15.709356+0000 mon.a (mon.0) 10747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:58:17.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[46934]: audit 2022-04-22T20:58:15.710417+0000 mon.a (mon.0) 10748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:58:17.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:16 smithi115 bash[46934]: cluster 2022-04-22T20:58:16.511182+0000 mgr.y (mgr.34107) 7252 : cluster [DBG] pgmap v8307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:18.381 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:58:18.729 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:58:18.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:58:18.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:58:18.732 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:58:19.130 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:58:19.131 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:58:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:19 smithi121 bash[36902]: audit 2022-04-22T20:58:18.359233+0000 mgr.y (mgr.34107) 7253 : audit [DBG] from='client.42912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:19 smithi121 bash[36902]: cluster 2022-04-22T20:58:18.512197+0000 mgr.y (mgr.34107) 7254 : cluster [DBG] pgmap v8308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:19 smithi121 bash[36902]: audit 2022-04-22T20:58:18.760036+0000 mon.a (mon.0) 10749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:19 smithi121 bash[36902]: audit 2022-04-22T20:58:19.125001+0000 mon.a (mon.0) 10750 : audit [DBG] from='client.? 172.21.15.115:0/3540517850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[45495]: audit 2022-04-22T20:58:18.359233+0000 mgr.y (mgr.34107) 7253 : audit [DBG] from='client.42912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[45495]: cluster 2022-04-22T20:58:18.512197+0000 mgr.y (mgr.34107) 7254 : cluster [DBG] pgmap v8308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[45495]: audit 2022-04-22T20:58:18.760036+0000 mon.a (mon.0) 10749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[45495]: audit 2022-04-22T20:58:19.125001+0000 mon.a (mon.0) 10750 : audit [DBG] from='client.? 172.21.15.115:0/3540517850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:20.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[46934]: audit 2022-04-22T20:58:18.359233+0000 mgr.y (mgr.34107) 7253 : audit [DBG] from='client.42912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:20.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[46934]: cluster 2022-04-22T20:58:18.512197+0000 mgr.y (mgr.34107) 7254 : cluster [DBG] pgmap v8308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:20.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[46934]: audit 2022-04-22T20:58:18.760036+0000 mon.a (mon.0) 10749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:20.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:19 smithi115 bash[46934]: audit 2022-04-22T20:58:19.125001+0000 mon.a (mon.0) 10750 : audit [DBG] from='client.? 172.21.15.115:0/3540517850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:20 smithi121 bash[36902]: audit 2022-04-22T20:58:18.704421+0000 mgr.y (mgr.34107) 7255 : audit [DBG] from='client.42918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:20 smithi121 bash[36902]: audit 2022-04-22T20:58:20.105762+0000 mon.a (mon.0) 10751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:20 smithi115 bash[45495]: audit 2022-04-22T20:58:18.704421+0000 mgr.y (mgr.34107) 7255 : audit [DBG] from='client.42918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:20 smithi115 bash[45495]: audit 2022-04-22T20:58:20.105762+0000 mon.a (mon.0) 10751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:20 smithi115 bash[46934]: audit 2022-04-22T20:58:18.704421+0000 mgr.y (mgr.34107) 7255 : audit [DBG] from='client.42918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:20 smithi115 bash[46934]: audit 2022-04-22T20:58:20.105762+0000 mon.a (mon.0) 10751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:21 smithi121 bash[36902]: cluster 2022-04-22T20:58:20.513253+0000 mgr.y (mgr.34107) 7256 : cluster [DBG] pgmap v8309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:22.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:21 smithi115 bash[45495]: cluster 2022-04-22T20:58:20.513253+0000 mgr.y (mgr.34107) 7256 : cluster [DBG] pgmap v8309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:22.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:21 smithi115 bash[46934]: cluster 2022-04-22T20:58:20.513253+0000 mgr.y (mgr.34107) 7256 : cluster [DBG] pgmap v8309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:22 smithi115 bash[45495]: cluster 2022-04-22T20:58:22.514299+0000 mgr.y (mgr.34107) 7257 : cluster [DBG] pgmap v8310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:22.438Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:22 smithi115 bash[46934]: cluster 2022-04-22T20:58:22.514299+0000 mgr.y (mgr.34107) 7257 : cluster [DBG] pgmap v8310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:22 smithi121 bash[36902]: cluster 2022-04-22T20:58:22.514299+0000 mgr.y (mgr.34107) 7257 : cluster [DBG] pgmap v8310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:23 smithi121 bash[36902]: audit 2022-04-22T20:58:23.763251+0000 mon.a (mon.0) 10752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:24.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:23 smithi115 bash[45495]: audit 2022-04-22T20:58:23.763251+0000 mon.a (mon.0) 10752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:23 smithi115 bash[46934]: audit 2022-04-22T20:58:23.763251+0000 mon.a (mon.0) 10752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:24.311Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:24 smithi121 bash[36902]: cluster 2022-04-22T20:58:24.515363+0000 mgr.y (mgr.34107) 7258 : cluster [DBG] pgmap v8311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:25.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:24 smithi115 bash[45495]: cluster 2022-04-22T20:58:24.515363+0000 mgr.y (mgr.34107) 7258 : cluster [DBG] pgmap v8311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:24 smithi115 bash[46934]: cluster 2022-04-22T20:58:24.515363+0000 mgr.y (mgr.34107) 7258 : cluster [DBG] pgmap v8311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:25 smithi121 bash[36902]: audit 2022-04-22T20:58:25.108650+0000 mon.a (mon.0) 10753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:25 smithi115 bash[46934]: audit 2022-04-22T20:58:25.108650+0000 mon.a (mon.0) 10753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:25 smithi115 bash[45495]: audit 2022-04-22T20:58:25.108650+0000 mon.a (mon.0) 10753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:26 smithi121 bash[36902]: cluster 2022-04-22T20:58:26.516487+0000 mgr.y (mgr.34107) 7259 : cluster [DBG] pgmap v8312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:27.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:26 smithi115 bash[45495]: cluster 2022-04-22T20:58:26.516487+0000 mgr.y (mgr.34107) 7259 : cluster [DBG] pgmap v8312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:27.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:26 smithi115 bash[46934]: cluster 2022-04-22T20:58:26.516487+0000 mgr.y (mgr.34107) 7259 : cluster [DBG] pgmap v8312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:28 smithi121 bash[36902]: cluster 2022-04-22T20:58:28.517577+0000 mgr.y (mgr.34107) 7260 : cluster [DBG] pgmap v8313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:28 smithi115 bash[46934]: cluster 2022-04-22T20:58:28.517577+0000 mgr.y (mgr.34107) 7260 : cluster [DBG] pgmap v8313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:29.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:28 smithi115 bash[45495]: cluster 2022-04-22T20:58:28.517577+0000 mgr.y (mgr.34107) 7260 : cluster [DBG] pgmap v8313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:29 smithi121 bash[36902]: audit 2022-04-22T20:58:28.767039+0000 mon.a (mon.0) 10754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:29 smithi115 bash[46934]: audit 2022-04-22T20:58:28.767039+0000 mon.a (mon.0) 10754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:29 smithi115 bash[45495]: audit 2022-04-22T20:58:28.767039+0000 mon.a (mon.0) 10754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:30 smithi121 bash[36902]: audit 2022-04-22T20:58:30.111629+0000 mon.a (mon.0) 10755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:30 smithi121 bash[36902]: cluster 2022-04-22T20:58:30.518614+0000 mgr.y (mgr.34107) 7261 : cluster [DBG] pgmap v8314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:30 smithi115 bash[45495]: audit 2022-04-22T20:58:30.111629+0000 mon.a (mon.0) 10755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:30 smithi115 bash[45495]: cluster 2022-04-22T20:58:30.518614+0000 mgr.y (mgr.34107) 7261 : cluster [DBG] pgmap v8314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:30 smithi115 bash[46934]: audit 2022-04-22T20:58:30.111629+0000 mon.a (mon.0) 10755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:30 smithi115 bash[46934]: cluster 2022-04-22T20:58:30.518614+0000 mgr.y (mgr.34107) 7261 : cluster [DBG] pgmap v8314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:31 smithi121 bash[36902]: audit 2022-04-22T20:58:31.011914+0000 mon.a (mon.0) 10756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:58:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:31 smithi121 bash[36902]: audit 2022-04-22T20:58:31.014925+0000 mon.a (mon.0) 10757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:58:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:31 smithi121 bash[36902]: audit 2022-04-22T20:58:31.016817+0000 mon.a (mon.0) 10758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:58:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:31 smithi121 bash[36902]: audit 2022-04-22T20:58:31.383861+0000 mon.a (mon.0) 10759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:58:32.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[45495]: audit 2022-04-22T20:58:31.011914+0000 mon.a (mon.0) 10756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:58:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[45495]: audit 2022-04-22T20:58:31.014925+0000 mon.a (mon.0) 10757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:58:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[45495]: audit 2022-04-22T20:58:31.016817+0000 mon.a (mon.0) 10758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:58:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[45495]: audit 2022-04-22T20:58:31.383861+0000 mon.a (mon.0) 10759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:58:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[46934]: audit 2022-04-22T20:58:31.011914+0000 mon.a (mon.0) 10756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:58:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[46934]: audit 2022-04-22T20:58:31.014925+0000 mon.a (mon.0) 10757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:58:32.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[46934]: audit 2022-04-22T20:58:31.016817+0000 mon.a (mon.0) 10758 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:58:32.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:31 smithi115 bash[46934]: audit 2022-04-22T20:58:31.383861+0000 mon.a (mon.0) 10759 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:58:32.723 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:32.438Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:32 smithi121 bash[36902]: cluster 2022-04-22T20:58:32.519717+0000 mgr.y (mgr.34107) 7262 : cluster [DBG] pgmap v8315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:33.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:32 smithi115 bash[46934]: cluster 2022-04-22T20:58:32.519717+0000 mgr.y (mgr.34107) 7262 : cluster [DBG] pgmap v8315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:32 smithi115 bash[45495]: cluster 2022-04-22T20:58:32.519717+0000 mgr.y (mgr.34107) 7262 : cluster [DBG] pgmap v8315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:33 smithi121 bash[36902]: audit 2022-04-22T20:58:33.770534+0000 mon.a (mon.0) 10760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:33 smithi115 bash[45495]: audit 2022-04-22T20:58:33.770534+0000 mon.a (mon.0) 10760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:33 smithi115 bash[46934]: audit 2022-04-22T20:58:33.770534+0000 mon.a (mon.0) 10760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:34.311Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:34 smithi121 bash[36902]: cluster 2022-04-22T20:58:34.520841+0000 mgr.y (mgr.34107) 7263 : cluster [DBG] pgmap v8316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:34 smithi115 bash[45495]: cluster 2022-04-22T20:58:34.520841+0000 mgr.y (mgr.34107) 7263 : cluster [DBG] pgmap v8316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:34 smithi115 bash[46934]: cluster 2022-04-22T20:58:34.520841+0000 mgr.y (mgr.34107) 7263 : cluster [DBG] pgmap v8316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:35 smithi121 bash[36902]: audit 2022-04-22T20:58:35.114720+0000 mon.a (mon.0) 10761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:35 smithi115 bash[45495]: audit 2022-04-22T20:58:35.114720+0000 mon.a (mon.0) 10761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:35 smithi115 bash[46934]: audit 2022-04-22T20:58:35.114720+0000 mon.a (mon.0) 10761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:36 smithi121 bash[36902]: cluster 2022-04-22T20:58:36.521913+0000 mgr.y (mgr.34107) 7264 : cluster [DBG] pgmap v8317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:36 smithi115 bash[45495]: cluster 2022-04-22T20:58:36.521913+0000 mgr.y (mgr.34107) 7264 : cluster [DBG] pgmap v8317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:36 smithi115 bash[46934]: cluster 2022-04-22T20:58:36.521913+0000 mgr.y (mgr.34107) 7264 : cluster [DBG] pgmap v8317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:38 smithi121 bash[36902]: cluster 2022-04-22T20:58:38.523036+0000 mgr.y (mgr.34107) 7265 : cluster [DBG] pgmap v8318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:38 smithi115 bash[45495]: cluster 2022-04-22T20:58:38.523036+0000 mgr.y (mgr.34107) 7265 : cluster [DBG] pgmap v8318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:38 smithi115 bash[46934]: cluster 2022-04-22T20:58:38.523036+0000 mgr.y (mgr.34107) 7265 : cluster [DBG] pgmap v8318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:39 smithi121 bash[36902]: audit 2022-04-22T20:58:38.774208+0000 mon.a (mon.0) 10762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:39 smithi115 bash[45495]: audit 2022-04-22T20:58:38.774208+0000 mon.a (mon.0) 10762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:39 smithi115 bash[46934]: audit 2022-04-22T20:58:38.774208+0000 mon.a (mon.0) 10762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:40 smithi121 bash[36902]: audit 2022-04-22T20:58:40.118099+0000 mon.a (mon.0) 10763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:40 smithi121 bash[36902]: cluster 2022-04-22T20:58:40.524137+0000 mgr.y (mgr.34107) 7266 : cluster [DBG] pgmap v8319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:40 smithi115 bash[45495]: audit 2022-04-22T20:58:40.118099+0000 mon.a (mon.0) 10763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:40 smithi115 bash[45495]: cluster 2022-04-22T20:58:40.524137+0000 mgr.y (mgr.34107) 7266 : cluster [DBG] pgmap v8319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:40 smithi115 bash[46934]: audit 2022-04-22T20:58:40.118099+0000 mon.a (mon.0) 10763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:40 smithi115 bash[46934]: cluster 2022-04-22T20:58:40.524137+0000 mgr.y (mgr.34107) 7266 : cluster [DBG] pgmap v8319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:42 smithi115 bash[45495]: cluster 2022-04-22T20:58:42.525239+0000 mgr.y (mgr.34107) 7267 : cluster [DBG] pgmap v8320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:42.439Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:42 smithi115 bash[46934]: cluster 2022-04-22T20:58:42.525239+0000 mgr.y (mgr.34107) 7267 : cluster [DBG] pgmap v8320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:42 smithi121 bash[36902]: cluster 2022-04-22T20:58:42.525239+0000 mgr.y (mgr.34107) 7267 : cluster [DBG] pgmap v8320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:43 smithi121 bash[36902]: audit 2022-04-22T20:58:43.777875+0000 mon.a (mon.0) 10764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:43 smithi115 bash[45495]: audit 2022-04-22T20:58:43.777875+0000 mon.a (mon.0) 10764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:43 smithi115 bash[46934]: audit 2022-04-22T20:58:43.777875+0000 mon.a (mon.0) 10764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:44.314Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:44 smithi121 bash[36902]: cluster 2022-04-22T20:58:44.526457+0000 mgr.y (mgr.34107) 7268 : cluster [DBG] pgmap v8321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:44 smithi115 bash[45495]: cluster 2022-04-22T20:58:44.526457+0000 mgr.y (mgr.34107) 7268 : cluster [DBG] pgmap v8321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:44 smithi115 bash[46934]: cluster 2022-04-22T20:58:44.526457+0000 mgr.y (mgr.34107) 7268 : cluster [DBG] pgmap v8321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:45 smithi121 bash[36902]: audit 2022-04-22T20:58:45.121102+0000 mon.a (mon.0) 10765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:45 smithi115 bash[45495]: audit 2022-04-22T20:58:45.121102+0000 mon.a (mon.0) 10765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:45 smithi115 bash[46934]: audit 2022-04-22T20:58:45.121102+0000 mon.a (mon.0) 10765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:46 smithi121 bash[36902]: cluster 2022-04-22T20:58:46.527559+0000 mgr.y (mgr.34107) 7269 : cluster [DBG] pgmap v8322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:46 smithi115 bash[45495]: cluster 2022-04-22T20:58:46.527559+0000 mgr.y (mgr.34107) 7269 : cluster [DBG] pgmap v8322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:46 smithi115 bash[46934]: cluster 2022-04-22T20:58:46.527559+0000 mgr.y (mgr.34107) 7269 : cluster [DBG] pgmap v8322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:48 smithi121 bash[36902]: cluster 2022-04-22T20:58:48.528711+0000 mgr.y (mgr.34107) 7270 : cluster [DBG] pgmap v8323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:48 smithi115 bash[45495]: cluster 2022-04-22T20:58:48.528711+0000 mgr.y (mgr.34107) 7270 : cluster [DBG] pgmap v8323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:48 smithi115 bash[46934]: cluster 2022-04-22T20:58:48.528711+0000 mgr.y (mgr.34107) 7270 : cluster [DBG] pgmap v8323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:49.506 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:58:49.856 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:58:49.856 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:58:49.856 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:58:49.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:58:49.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:58:49.858 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:58:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:49 smithi121 bash[36902]: audit 2022-04-22T20:58:48.781637+0000 mon.a (mon.0) 10766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:49 smithi121 bash[36902]: audit 2022-04-22T20:58:49.482745+0000 mgr.y (mgr.34107) 7271 : audit [DBG] from='client.42930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:49 smithi115 bash[45495]: audit 2022-04-22T20:58:48.781637+0000 mon.a (mon.0) 10766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:49 smithi115 bash[45495]: audit 2022-04-22T20:58:49.482745+0000 mgr.y (mgr.34107) 7271 : audit [DBG] from='client.42930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:50.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:49 smithi115 bash[46934]: audit 2022-04-22T20:58:48.781637+0000 mon.a (mon.0) 10766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:50.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:49 smithi115 bash[46934]: audit 2022-04-22T20:58:49.482745+0000 mgr.y (mgr.34107) 7271 : audit [DBG] from='client.42930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:58:50.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:58:50.256 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:58:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:50 smithi121 bash[36902]: audit 2022-04-22T20:58:49.830151+0000 mgr.y (mgr.34107) 7272 : audit [DBG] from='client.52867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:50 smithi121 bash[36902]: audit 2022-04-22T20:58:50.124109+0000 mon.a (mon.0) 10767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:50 smithi121 bash[36902]: audit 2022-04-22T20:58:50.249821+0000 mon.c (mon.1) 270 : audit [DBG] from='client.? 172.21.15.115:0/1105718436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:50 smithi121 bash[36902]: cluster 2022-04-22T20:58:50.529816+0000 mgr.y (mgr.34107) 7273 : cluster [DBG] pgmap v8324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[45495]: audit 2022-04-22T20:58:49.830151+0000 mgr.y (mgr.34107) 7272 : audit [DBG] from='client.52867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[45495]: audit 2022-04-22T20:58:50.124109+0000 mon.a (mon.0) 10767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[45495]: audit 2022-04-22T20:58:50.249821+0000 mon.c (mon.1) 270 : audit [DBG] from='client.? 172.21.15.115:0/1105718436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[45495]: cluster 2022-04-22T20:58:50.529816+0000 mgr.y (mgr.34107) 7273 : cluster [DBG] pgmap v8324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[46934]: audit 2022-04-22T20:58:49.830151+0000 mgr.y (mgr.34107) 7272 : audit [DBG] from='client.52867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:58:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[46934]: audit 2022-04-22T20:58:50.124109+0000 mon.a (mon.0) 10767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[46934]: audit 2022-04-22T20:58:50.249821+0000 mon.c (mon.1) 270 : audit [DBG] from='client.? 172.21.15.115:0/1105718436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:58:51.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:50 smithi115 bash[46934]: cluster 2022-04-22T20:58:50.529816+0000 mgr.y (mgr.34107) 7273 : cluster [DBG] pgmap v8324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:52 smithi115 bash[45495]: cluster 2022-04-22T20:58:52.530899+0000 mgr.y (mgr.34107) 7274 : cluster [DBG] pgmap v8325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:52.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:52 smithi115 bash[46934]: cluster 2022-04-22T20:58:52.530899+0000 mgr.y (mgr.34107) 7274 : cluster [DBG] pgmap v8325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:52.800 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:52.439Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:52 smithi121 bash[36902]: cluster 2022-04-22T20:58:52.530899+0000 mgr.y (mgr.34107) 7274 : cluster [DBG] pgmap v8325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:53 smithi121 bash[36902]: audit 2022-04-22T20:58:53.785290+0000 mon.a (mon.0) 10768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:53 smithi115 bash[45495]: audit 2022-04-22T20:58:53.785290+0000 mon.a (mon.0) 10768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:54.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:53 smithi115 bash[46934]: audit 2022-04-22T20:58:53.785290+0000 mon.a (mon.0) 10768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:58:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:58:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:58:54.314Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:58:55.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:54 smithi121 bash[36902]: cluster 2022-04-22T20:58:54.531976+0000 mgr.y (mgr.34107) 7275 : cluster [DBG] pgmap v8326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:54 smithi115 bash[45495]: cluster 2022-04-22T20:58:54.531976+0000 mgr.y (mgr.34107) 7275 : cluster [DBG] pgmap v8326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:54 smithi115 bash[46934]: cluster 2022-04-22T20:58:54.531976+0000 mgr.y (mgr.34107) 7275 : cluster [DBG] pgmap v8326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:55 smithi121 bash[36902]: audit 2022-04-22T20:58:55.127189+0000 mon.a (mon.0) 10769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:55 smithi115 bash[45495]: audit 2022-04-22T20:58:55.127189+0000 mon.a (mon.0) 10769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:55 smithi115 bash[46934]: audit 2022-04-22T20:58:55.127189+0000 mon.a (mon.0) 10769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:58:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:56 smithi121 bash[36902]: cluster 2022-04-22T20:58:56.533057+0000 mgr.y (mgr.34107) 7276 : cluster [DBG] pgmap v8327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:56 smithi115 bash[45495]: cluster 2022-04-22T20:58:56.533057+0000 mgr.y (mgr.34107) 7276 : cluster [DBG] pgmap v8327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:56 smithi115 bash[46934]: cluster 2022-04-22T20:58:56.533057+0000 mgr.y (mgr.34107) 7276 : cluster [DBG] pgmap v8327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:58 smithi121 bash[36902]: cluster 2022-04-22T20:58:58.534328+0000 mgr.y (mgr.34107) 7277 : cluster [DBG] pgmap v8328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:59.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:58 smithi115 bash[45495]: cluster 2022-04-22T20:58:58.534328+0000 mgr.y (mgr.34107) 7277 : cluster [DBG] pgmap v8328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:58:59.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:58 smithi115 bash[46934]: cluster 2022-04-22T20:58:58.534328+0000 mgr.y (mgr.34107) 7277 : cluster [DBG] pgmap v8328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:58:59 smithi121 bash[36902]: audit 2022-04-22T20:58:58.788924+0000 mon.a (mon.0) 10770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:58:59 smithi115 bash[45495]: audit 2022-04-22T20:58:58.788924+0000 mon.a (mon.0) 10770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:58:59 smithi115 bash[46934]: audit 2022-04-22T20:58:58.788924+0000 mon.a (mon.0) 10770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:00 smithi121 bash[36902]: audit 2022-04-22T20:59:00.130044+0000 mon.a (mon.0) 10771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:00 smithi121 bash[36902]: cluster 2022-04-22T20:59:00.535465+0000 mgr.y (mgr.34107) 7278 : cluster [DBG] pgmap v8329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:00 smithi115 bash[45495]: audit 2022-04-22T20:59:00.130044+0000 mon.a (mon.0) 10771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:00 smithi115 bash[45495]: cluster 2022-04-22T20:59:00.535465+0000 mgr.y (mgr.34107) 7278 : cluster [DBG] pgmap v8329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:00 smithi115 bash[46934]: audit 2022-04-22T20:59:00.130044+0000 mon.a (mon.0) 10771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:00 smithi115 bash[46934]: cluster 2022-04-22T20:59:00.535465+0000 mgr.y (mgr.34107) 7278 : cluster [DBG] pgmap v8329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:02 smithi115 bash[45495]: cluster 2022-04-22T20:59:02.536542+0000 mgr.y (mgr.34107) 7279 : cluster [DBG] pgmap v8330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:02 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:02.440Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:02 smithi115 bash[46934]: cluster 2022-04-22T20:59:02.536542+0000 mgr.y (mgr.34107) 7279 : cluster [DBG] pgmap v8330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:02 smithi121 bash[36902]: cluster 2022-04-22T20:59:02.536542+0000 mgr.y (mgr.34107) 7279 : cluster [DBG] pgmap v8330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:03 smithi121 bash[36902]: audit 2022-04-22T20:59:03.792413+0000 mon.a (mon.0) 10772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:03 smithi115 bash[45495]: audit 2022-04-22T20:59:03.792413+0000 mon.a (mon.0) 10772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:03 smithi115 bash[46934]: audit 2022-04-22T20:59:03.792413+0000 mon.a (mon.0) 10772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:04 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:04.315Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:04 smithi121 bash[36902]: cluster 2022-04-22T20:59:04.537667+0000 mgr.y (mgr.34107) 7280 : cluster [DBG] pgmap v8331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:05.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:04 smithi115 bash[45495]: cluster 2022-04-22T20:59:04.537667+0000 mgr.y (mgr.34107) 7280 : cluster [DBG] pgmap v8331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:04 smithi115 bash[46934]: cluster 2022-04-22T20:59:04.537667+0000 mgr.y (mgr.34107) 7280 : cluster [DBG] pgmap v8331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:05 smithi121 bash[36902]: audit 2022-04-22T20:59:05.132948+0000 mon.a (mon.0) 10773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:05 smithi115 bash[45495]: audit 2022-04-22T20:59:05.132948+0000 mon.a (mon.0) 10773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:05 smithi115 bash[46934]: audit 2022-04-22T20:59:05.132948+0000 mon.a (mon.0) 10773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:06 smithi121 bash[36902]: cluster 2022-04-22T20:59:06.538725+0000 mgr.y (mgr.34107) 7281 : cluster [DBG] pgmap v8332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:07.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:06 smithi115 bash[45495]: cluster 2022-04-22T20:59:06.538725+0000 mgr.y (mgr.34107) 7281 : cluster [DBG] pgmap v8332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:07.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:06 smithi115 bash[46934]: cluster 2022-04-22T20:59:06.538725+0000 mgr.y (mgr.34107) 7281 : cluster [DBG] pgmap v8332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:08 smithi121 bash[36902]: cluster 2022-04-22T20:59:08.539883+0000 mgr.y (mgr.34107) 7282 : cluster [DBG] pgmap v8333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:09.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:08 smithi115 bash[45495]: cluster 2022-04-22T20:59:08.539883+0000 mgr.y (mgr.34107) 7282 : cluster [DBG] pgmap v8333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:08 smithi115 bash[46934]: cluster 2022-04-22T20:59:08.539883+0000 mgr.y (mgr.34107) 7282 : cluster [DBG] pgmap v8333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:09 smithi121 bash[36902]: audit 2022-04-22T20:59:08.796332+0000 mon.a (mon.0) 10774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:09 smithi115 bash[45495]: audit 2022-04-22T20:59:08.796332+0000 mon.a (mon.0) 10774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:09 smithi115 bash[46934]: audit 2022-04-22T20:59:08.796332+0000 mon.a (mon.0) 10774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:10 smithi121 bash[36902]: audit 2022-04-22T20:59:10.135907+0000 mon.a (mon.0) 10775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:10 smithi121 bash[36902]: cluster 2022-04-22T20:59:10.540978+0000 mgr.y (mgr.34107) 7283 : cluster [DBG] pgmap v8334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:11.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:10 smithi115 bash[45495]: audit 2022-04-22T20:59:10.135907+0000 mon.a (mon.0) 10775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:10 smithi115 bash[45495]: cluster 2022-04-22T20:59:10.540978+0000 mgr.y (mgr.34107) 7283 : cluster [DBG] pgmap v8334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:10 smithi115 bash[46934]: audit 2022-04-22T20:59:10.135907+0000 mon.a (mon.0) 10775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:11.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:10 smithi115 bash[46934]: cluster 2022-04-22T20:59:10.540978+0000 mgr.y (mgr.34107) 7283 : cluster [DBG] pgmap v8334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:12 smithi115 bash[45495]: cluster 2022-04-22T20:59:12.542067+0000 mgr.y (mgr.34107) 7284 : cluster [DBG] pgmap v8335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:12 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:12.442Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:12 smithi115 bash[46934]: cluster 2022-04-22T20:59:12.542067+0000 mgr.y (mgr.34107) 7284 : cluster [DBG] pgmap v8335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:12 smithi121 bash[36902]: cluster 2022-04-22T20:59:12.542067+0000 mgr.y (mgr.34107) 7284 : cluster [DBG] pgmap v8335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:13 smithi121 bash[36902]: audit 2022-04-22T20:59:13.800006+0000 mon.a (mon.0) 10776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:13 smithi115 bash[45495]: audit 2022-04-22T20:59:13.800006+0000 mon.a (mon.0) 10776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:13 smithi115 bash[46934]: audit 2022-04-22T20:59:13.800006+0000 mon.a (mon.0) 10776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:14 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:14.315Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:14 smithi121 bash[36902]: cluster 2022-04-22T20:59:14.543195+0000 mgr.y (mgr.34107) 7285 : cluster [DBG] pgmap v8336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:15.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:14 smithi115 bash[45495]: cluster 2022-04-22T20:59:14.543195+0000 mgr.y (mgr.34107) 7285 : cluster [DBG] pgmap v8336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:14 smithi115 bash[46934]: cluster 2022-04-22T20:59:14.543195+0000 mgr.y (mgr.34107) 7285 : cluster [DBG] pgmap v8336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:15 smithi121 bash[36902]: audit 2022-04-22T20:59:15.138849+0000 mon.a (mon.0) 10777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:15 smithi121 bash[36902]: audit 2022-04-22T20:59:15.722735+0000 mon.a (mon.0) 10778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:59:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:15 smithi121 bash[36902]: audit 2022-04-22T20:59:15.723420+0000 mon.a (mon.0) 10779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:59:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[45495]: audit 2022-04-22T20:59:15.138849+0000 mon.a (mon.0) 10777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[45495]: audit 2022-04-22T20:59:15.722735+0000 mon.a (mon.0) 10778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:59:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[45495]: audit 2022-04-22T20:59:15.723420+0000 mon.a (mon.0) 10779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:59:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[46934]: audit 2022-04-22T20:59:15.138849+0000 mon.a (mon.0) 10777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[46934]: audit 2022-04-22T20:59:15.722735+0000 mon.a (mon.0) 10778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T20:59:16.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:15 smithi115 bash[46934]: audit 2022-04-22T20:59:15.723420+0000 mon.a (mon.0) 10779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T20:59:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:16 smithi121 bash[36902]: cluster 2022-04-22T20:59:16.544293+0000 mgr.y (mgr.34107) 7286 : cluster [DBG] pgmap v8337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:16 smithi115 bash[45495]: cluster 2022-04-22T20:59:16.544293+0000 mgr.y (mgr.34107) 7286 : cluster [DBG] pgmap v8337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:16 smithi115 bash[46934]: cluster 2022-04-22T20:59:16.544293+0000 mgr.y (mgr.34107) 7286 : cluster [DBG] pgmap v8337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:18 smithi121 bash[36902]: cluster 2022-04-22T20:59:18.545411+0000 mgr.y (mgr.34107) 7287 : cluster [DBG] pgmap v8338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:19.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:18 smithi115 bash[45495]: cluster 2022-04-22T20:59:18.545411+0000 mgr.y (mgr.34107) 7287 : cluster [DBG] pgmap v8338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:18 smithi115 bash[46934]: cluster 2022-04-22T20:59:18.545411+0000 mgr.y (mgr.34107) 7287 : cluster [DBG] pgmap v8338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:19 smithi121 bash[36902]: audit 2022-04-22T20:59:18.803638+0000 mon.a (mon.0) 10780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:19 smithi115 bash[46934]: audit 2022-04-22T20:59:18.803638+0000 mon.a (mon.0) 10780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:19 smithi115 bash[45495]: audit 2022-04-22T20:59:18.803638+0000 mon.a (mon.0) 10780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:20.631 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:59:20.977 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:59:20.977 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:59:20.978 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:59:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:59:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:20 smithi121 bash[36902]: audit 2022-04-22T20:59:20.142256+0000 mon.a (mon.0) 10781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:20 smithi121 bash[36902]: cluster 2022-04-22T20:59:20.546499+0000 mgr.y (mgr.34107) 7288 : cluster [DBG] pgmap v8339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:20 smithi115 bash[46934]: audit 2022-04-22T20:59:20.142256+0000 mon.a (mon.0) 10781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:20 smithi115 bash[46934]: cluster 2022-04-22T20:59:20.546499+0000 mgr.y (mgr.34107) 7288 : cluster [DBG] pgmap v8339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:20 smithi115 bash[45495]: audit 2022-04-22T20:59:20.142256+0000 mon.a (mon.0) 10781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:20 smithi115 bash[45495]: cluster 2022-04-22T20:59:20.546499+0000 mgr.y (mgr.34107) 7288 : cluster [DBG] pgmap v8339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:59:21.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:59:21.386 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:59:21.387 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:59:21.387 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:59:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:21 smithi121 bash[36902]: audit 2022-04-22T20:59:20.610700+0000 mgr.y (mgr.34107) 7289 : audit [DBG] from='client.52879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:21 smithi121 bash[36902]: audit 2022-04-22T20:59:20.953567+0000 mgr.y (mgr.34107) 7290 : audit [DBG] from='client.42954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:21 smithi121 bash[36902]: audit 2022-04-22T20:59:21.379904+0000 mon.a (mon.0) 10782 : audit [DBG] from='client.? 172.21.15.115:0/2016048535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[45495]: audit 2022-04-22T20:59:20.610700+0000 mgr.y (mgr.34107) 7289 : audit [DBG] from='client.52879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[45495]: audit 2022-04-22T20:59:20.953567+0000 mgr.y (mgr.34107) 7290 : audit [DBG] from='client.42954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[45495]: audit 2022-04-22T20:59:21.379904+0000 mon.a (mon.0) 10782 : audit [DBG] from='client.? 172.21.15.115:0/2016048535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:22.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[46934]: audit 2022-04-22T20:59:20.610700+0000 mgr.y (mgr.34107) 7289 : audit [DBG] from='client.52879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[46934]: audit 2022-04-22T20:59:20.953567+0000 mgr.y (mgr.34107) 7290 : audit [DBG] from='client.42954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:22.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:21 smithi115 bash[46934]: audit 2022-04-22T20:59:21.379904+0000 mon.a (mon.0) 10782 : audit [DBG] from='client.? 172.21.15.115:0/2016048535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:22.734 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:22 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:22.441Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:22 smithi121 bash[36902]: cluster 2022-04-22T20:59:22.547606+0000 mgr.y (mgr.34107) 7291 : cluster [DBG] pgmap v8340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:23.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:22 smithi115 bash[45495]: cluster 2022-04-22T20:59:22.547606+0000 mgr.y (mgr.34107) 7291 : cluster [DBG] pgmap v8340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:23.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:22 smithi115 bash[46934]: cluster 2022-04-22T20:59:22.547606+0000 mgr.y (mgr.34107) 7291 : cluster [DBG] pgmap v8340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:23 smithi121 bash[36902]: audit 2022-04-22T20:59:23.807236+0000 mon.a (mon.0) 10783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:24.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:23 smithi115 bash[45495]: audit 2022-04-22T20:59:23.807236+0000 mon.a (mon.0) 10783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:23 smithi115 bash[46934]: audit 2022-04-22T20:59:23.807236+0000 mon.a (mon.0) 10783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:24 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:24.315Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:24 smithi121 bash[36902]: cluster 2022-04-22T20:59:24.548642+0000 mgr.y (mgr.34107) 7292 : cluster [DBG] pgmap v8341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:25.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:24 smithi115 bash[45495]: cluster 2022-04-22T20:59:24.548642+0000 mgr.y (mgr.34107) 7292 : cluster [DBG] pgmap v8341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:25.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:24 smithi115 bash[46934]: cluster 2022-04-22T20:59:24.548642+0000 mgr.y (mgr.34107) 7292 : cluster [DBG] pgmap v8341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:25 smithi121 bash[36902]: audit 2022-04-22T20:59:25.145275+0000 mon.a (mon.0) 10784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:25 smithi115 bash[45495]: audit 2022-04-22T20:59:25.145275+0000 mon.a (mon.0) 10784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:25 smithi115 bash[46934]: audit 2022-04-22T20:59:25.145275+0000 mon.a (mon.0) 10784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:26 smithi121 bash[36902]: cluster 2022-04-22T20:59:26.549761+0000 mgr.y (mgr.34107) 7293 : cluster [DBG] pgmap v8342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:27.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:26 smithi115 bash[45495]: cluster 2022-04-22T20:59:26.549761+0000 mgr.y (mgr.34107) 7293 : cluster [DBG] pgmap v8342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:27.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:26 smithi115 bash[46934]: cluster 2022-04-22T20:59:26.549761+0000 mgr.y (mgr.34107) 7293 : cluster [DBG] pgmap v8342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:28 smithi121 bash[36902]: cluster 2022-04-22T20:59:28.550926+0000 mgr.y (mgr.34107) 7294 : cluster [DBG] pgmap v8343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:28 smithi115 bash[46934]: cluster 2022-04-22T20:59:28.550926+0000 mgr.y (mgr.34107) 7294 : cluster [DBG] pgmap v8343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:29.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:28 smithi115 bash[45495]: cluster 2022-04-22T20:59:28.550926+0000 mgr.y (mgr.34107) 7294 : cluster [DBG] pgmap v8343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:29 smithi121 bash[36902]: audit 2022-04-22T20:59:28.811010+0000 mon.a (mon.0) 10785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:29 smithi115 bash[45495]: audit 2022-04-22T20:59:28.811010+0000 mon.a (mon.0) 10785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:29 smithi115 bash[46934]: audit 2022-04-22T20:59:28.811010+0000 mon.a (mon.0) 10785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:30 smithi121 bash[36902]: audit 2022-04-22T20:59:30.148445+0000 mon.a (mon.0) 10786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:30 smithi121 bash[36902]: cluster 2022-04-22T20:59:30.552008+0000 mgr.y (mgr.34107) 7295 : cluster [DBG] pgmap v8344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:30 smithi115 bash[46934]: audit 2022-04-22T20:59:30.148445+0000 mon.a (mon.0) 10786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:30 smithi115 bash[46934]: cluster 2022-04-22T20:59:30.552008+0000 mgr.y (mgr.34107) 7295 : cluster [DBG] pgmap v8344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:30 smithi115 bash[45495]: audit 2022-04-22T20:59:30.148445+0000 mon.a (mon.0) 10786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:30 smithi115 bash[45495]: cluster 2022-04-22T20:59:30.552008+0000 mgr.y (mgr.34107) 7295 : cluster [DBG] pgmap v8344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:31 smithi121 bash[36902]: audit 2022-04-22T20:59:31.390557+0000 mon.a (mon.0) 10787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:59:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:31 smithi121 bash[36902]: audit 2022-04-22T20:59:31.393710+0000 mon.a (mon.0) 10788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:59:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:31 smithi121 bash[36902]: audit 2022-04-22T20:59:31.395225+0000 mon.a (mon.0) 10789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:59:32.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[45495]: audit 2022-04-22T20:59:31.390557+0000 mon.a (mon.0) 10787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:59:32.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[45495]: audit 2022-04-22T20:59:31.393710+0000 mon.a (mon.0) 10788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:59:32.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[45495]: audit 2022-04-22T20:59:31.395225+0000 mon.a (mon.0) 10789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[46934]: audit 2022-04-22T20:59:31.390557+0000 mon.a (mon.0) 10787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T20:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[46934]: audit 2022-04-22T20:59:31.393710+0000 mon.a (mon.0) 10788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T20:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:31 smithi115 bash[46934]: audit 2022-04-22T20:59:31.395225+0000 mon.a (mon.0) 10789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T20:59:32.768 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:32 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:32.443Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:32 smithi121 bash[36902]: audit 2022-04-22T20:59:31.745875+0000 mon.a (mon.0) 10790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:59:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:32 smithi121 bash[36902]: cluster 2022-04-22T20:59:32.553170+0000 mgr.y (mgr.34107) 7296 : cluster [DBG] pgmap v8345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:33.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:32 smithi115 bash[45495]: audit 2022-04-22T20:59:31.745875+0000 mon.a (mon.0) 10790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:59:33.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:32 smithi115 bash[45495]: cluster 2022-04-22T20:59:32.553170+0000 mgr.y (mgr.34107) 7296 : cluster [DBG] pgmap v8345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:33.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:32 smithi115 bash[46934]: audit 2022-04-22T20:59:31.745875+0000 mon.a (mon.0) 10790 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T20:59:33.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:32 smithi115 bash[46934]: cluster 2022-04-22T20:59:32.553170+0000 mgr.y (mgr.34107) 7296 : cluster [DBG] pgmap v8345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:33 smithi121 bash[36902]: audit 2022-04-22T20:59:33.814891+0000 mon.a (mon.0) 10791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:33 smithi115 bash[45495]: audit 2022-04-22T20:59:33.814891+0000 mon.a (mon.0) 10791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:34.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:33 smithi115 bash[46934]: audit 2022-04-22T20:59:33.814891+0000 mon.a (mon.0) 10791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:34 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:34.316Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:34 smithi121 bash[36902]: cluster 2022-04-22T20:59:34.554296+0000 mgr.y (mgr.34107) 7297 : cluster [DBG] pgmap v8346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:34 smithi115 bash[45495]: cluster 2022-04-22T20:59:34.554296+0000 mgr.y (mgr.34107) 7297 : cluster [DBG] pgmap v8346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:35.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:34 smithi115 bash[46934]: cluster 2022-04-22T20:59:34.554296+0000 mgr.y (mgr.34107) 7297 : cluster [DBG] pgmap v8346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:35 smithi121 bash[36902]: audit 2022-04-22T20:59:35.151381+0000 mon.a (mon.0) 10792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:35 smithi115 bash[45495]: audit 2022-04-22T20:59:35.151381+0000 mon.a (mon.0) 10792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:35 smithi115 bash[46934]: audit 2022-04-22T20:59:35.151381+0000 mon.a (mon.0) 10792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:36 smithi121 bash[36902]: cluster 2022-04-22T20:59:36.555573+0000 mgr.y (mgr.34107) 7298 : cluster [DBG] pgmap v8347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:36 smithi115 bash[45495]: cluster 2022-04-22T20:59:36.555573+0000 mgr.y (mgr.34107) 7298 : cluster [DBG] pgmap v8347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:36 smithi115 bash[46934]: cluster 2022-04-22T20:59:36.555573+0000 mgr.y (mgr.34107) 7298 : cluster [DBG] pgmap v8347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:38 smithi121 bash[36902]: cluster 2022-04-22T20:59:38.556745+0000 mgr.y (mgr.34107) 7299 : cluster [DBG] pgmap v8348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:38 smithi115 bash[45495]: cluster 2022-04-22T20:59:38.556745+0000 mgr.y (mgr.34107) 7299 : cluster [DBG] pgmap v8348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:39.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:38 smithi115 bash[46934]: cluster 2022-04-22T20:59:38.556745+0000 mgr.y (mgr.34107) 7299 : cluster [DBG] pgmap v8348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:39 smithi121 bash[36902]: audit 2022-04-22T20:59:38.818729+0000 mon.a (mon.0) 10793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:39 smithi115 bash[45495]: audit 2022-04-22T20:59:38.818729+0000 mon.a (mon.0) 10793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:39 smithi115 bash[46934]: audit 2022-04-22T20:59:38.818729+0000 mon.a (mon.0) 10793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:40 smithi121 bash[36902]: audit 2022-04-22T20:59:40.154417+0000 mon.a (mon.0) 10794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:40 smithi121 bash[36902]: cluster 2022-04-22T20:59:40.557811+0000 mgr.y (mgr.34107) 7300 : cluster [DBG] pgmap v8349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:40 smithi115 bash[45495]: audit 2022-04-22T20:59:40.154417+0000 mon.a (mon.0) 10794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:40 smithi115 bash[45495]: cluster 2022-04-22T20:59:40.557811+0000 mgr.y (mgr.34107) 7300 : cluster [DBG] pgmap v8349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:40 smithi115 bash[46934]: audit 2022-04-22T20:59:40.154417+0000 mon.a (mon.0) 10794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:40 smithi115 bash[46934]: cluster 2022-04-22T20:59:40.557811+0000 mgr.y (mgr.34107) 7300 : cluster [DBG] pgmap v8349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:42 smithi115 bash[45495]: cluster 2022-04-22T20:59:42.558823+0000 mgr.y (mgr.34107) 7301 : cluster [DBG] pgmap v8350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:42 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:42.442Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:42 smithi115 bash[46934]: cluster 2022-04-22T20:59:42.558823+0000 mgr.y (mgr.34107) 7301 : cluster [DBG] pgmap v8350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:42 smithi121 bash[36902]: cluster 2022-04-22T20:59:42.558823+0000 mgr.y (mgr.34107) 7301 : cluster [DBG] pgmap v8350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:43 smithi121 bash[36902]: audit 2022-04-22T20:59:43.822412+0000 mon.a (mon.0) 10795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:43 smithi115 bash[45495]: audit 2022-04-22T20:59:43.822412+0000 mon.a (mon.0) 10795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:43 smithi115 bash[46934]: audit 2022-04-22T20:59:43.822412+0000 mon.a (mon.0) 10795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:44 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:44.317Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:44 smithi121 bash[36902]: cluster 2022-04-22T20:59:44.559890+0000 mgr.y (mgr.34107) 7302 : cluster [DBG] pgmap v8351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:44 smithi115 bash[45495]: cluster 2022-04-22T20:59:44.559890+0000 mgr.y (mgr.34107) 7302 : cluster [DBG] pgmap v8351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:44 smithi115 bash[46934]: cluster 2022-04-22T20:59:44.559890+0000 mgr.y (mgr.34107) 7302 : cluster [DBG] pgmap v8351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:45 smithi121 bash[36902]: audit 2022-04-22T20:59:45.157341+0000 mon.a (mon.0) 10796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:45 smithi115 bash[45495]: audit 2022-04-22T20:59:45.157341+0000 mon.a (mon.0) 10796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:46.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:45 smithi115 bash[46934]: audit 2022-04-22T20:59:45.157341+0000 mon.a (mon.0) 10796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:46 smithi121 bash[36902]: cluster 2022-04-22T20:59:46.561028+0000 mgr.y (mgr.34107) 7303 : cluster [DBG] pgmap v8352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:46 smithi115 bash[45495]: cluster 2022-04-22T20:59:46.561028+0000 mgr.y (mgr.34107) 7303 : cluster [DBG] pgmap v8352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:47.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:46 smithi115 bash[46934]: cluster 2022-04-22T20:59:46.561028+0000 mgr.y (mgr.34107) 7303 : cluster [DBG] pgmap v8352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:48 smithi121 bash[36902]: cluster 2022-04-22T20:59:48.562293+0000 mgr.y (mgr.34107) 7304 : cluster [DBG] pgmap v8353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:48 smithi115 bash[45495]: cluster 2022-04-22T20:59:48.562293+0000 mgr.y (mgr.34107) 7304 : cluster [DBG] pgmap v8353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:49.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:48 smithi115 bash[46934]: cluster 2022-04-22T20:59:48.562293+0000 mgr.y (mgr.34107) 7304 : cluster [DBG] pgmap v8353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:49 smithi121 bash[36902]: audit 2022-04-22T20:59:48.826129+0000 mon.a (mon.0) 10797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:49 smithi115 bash[45495]: audit 2022-04-22T20:59:48.826129+0000 mon.a (mon.0) 10797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:49 smithi115 bash[46934]: audit 2022-04-22T20:59:48.826129+0000 mon.a (mon.0) 10797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:50 smithi121 bash[36902]: audit 2022-04-22T20:59:50.160435+0000 mon.a (mon.0) 10798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:50 smithi121 bash[36902]: cluster 2022-04-22T20:59:50.563340+0000 mgr.y (mgr.34107) 7305 : cluster [DBG] pgmap v8354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:50 smithi115 bash[45495]: audit 2022-04-22T20:59:50.160435+0000 mon.a (mon.0) 10798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:51.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:50 smithi115 bash[45495]: cluster 2022-04-22T20:59:50.563340+0000 mgr.y (mgr.34107) 7305 : cluster [DBG] pgmap v8354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:50 smithi115 bash[46934]: audit 2022-04-22T20:59:50.160435+0000 mon.a (mon.0) 10798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:50 smithi115 bash[46934]: cluster 2022-04-22T20:59:50.563340+0000 mgr.y (mgr.34107) 7305 : cluster [DBG] pgmap v8354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:51.763 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T20:59:52.109 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T20:59:52.110 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T20:59:52.111 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T20:59:52.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T20:59:52.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T20:59:52.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T20:59:52.112 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T20:59:52.112 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T20:59:52.515 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T20:59:52.515 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T20:59:52.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T20:59:52.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:52.515 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T20:59:52.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T20:59:52.517 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T20:59:52.517 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T20:59:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:52 smithi115 bash[45495]: audit 2022-04-22T20:59:52.510077+0000 mon.a (mon.0) 10799 : audit [DBG] from='client.? 172.21.15.115:0/417804620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:52 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:52.444Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:52 smithi115 bash[46934]: audit 2022-04-22T20:59:52.510077+0000 mon.a (mon.0) 10799 : audit [DBG] from='client.? 172.21.15.115:0/417804620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:52 smithi121 bash[36902]: audit 2022-04-22T20:59:52.510077+0000 mon.a (mon.0) 10799 : audit [DBG] from='client.? 172.21.15.115:0/417804620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T20:59:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:53 smithi121 bash[36902]: audit 2022-04-22T20:59:51.742999+0000 mgr.y (mgr.34107) 7306 : audit [DBG] from='client.42966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:53 smithi121 bash[36902]: audit 2022-04-22T20:59:52.084185+0000 mgr.y (mgr.34107) 7307 : audit [DBG] from='client.42972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:53 smithi121 bash[36902]: cluster 2022-04-22T20:59:52.564407+0000 mgr.y (mgr.34107) 7308 : cluster [DBG] pgmap v8355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:54.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[46934]: audit 2022-04-22T20:59:51.742999+0000 mgr.y (mgr.34107) 7306 : audit [DBG] from='client.42966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[46934]: audit 2022-04-22T20:59:52.084185+0000 mgr.y (mgr.34107) 7307 : audit [DBG] from='client.42972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[46934]: cluster 2022-04-22T20:59:52.564407+0000 mgr.y (mgr.34107) 7308 : cluster [DBG] pgmap v8355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[45495]: audit 2022-04-22T20:59:51.742999+0000 mgr.y (mgr.34107) 7306 : audit [DBG] from='client.42966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[45495]: audit 2022-04-22T20:59:52.084185+0000 mgr.y (mgr.34107) 7307 : audit [DBG] from='client.42972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T20:59:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:53 smithi115 bash[45495]: cluster 2022-04-22T20:59:52.564407+0000 mgr.y (mgr.34107) 7308 : cluster [DBG] pgmap v8355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:54.585 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 20:59:54 smithi115 bash[60466]: level=error ts=2022-04-22T20:59:54.317Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T20:59:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:54 smithi121 bash[36902]: audit 2022-04-22T20:59:53.829743+0000 mon.a (mon.0) 10800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:54 smithi115 bash[46934]: audit 2022-04-22T20:59:53.829743+0000 mon.a (mon.0) 10800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:55.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:54 smithi115 bash[45495]: audit 2022-04-22T20:59:53.829743+0000 mon.a (mon.0) 10800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T20:59:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:55 smithi121 bash[36902]: cluster 2022-04-22T20:59:54.565492+0000 mgr.y (mgr.34107) 7309 : cluster [DBG] pgmap v8356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:55 smithi121 bash[36902]: audit 2022-04-22T20:59:55.163392+0000 mon.a (mon.0) 10801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:55 smithi115 bash[45495]: cluster 2022-04-22T20:59:54.565492+0000 mgr.y (mgr.34107) 7309 : cluster [DBG] pgmap v8356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:55 smithi115 bash[45495]: audit 2022-04-22T20:59:55.163392+0000 mon.a (mon.0) 10801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:55 smithi115 bash[46934]: cluster 2022-04-22T20:59:54.565492+0000 mgr.y (mgr.34107) 7309 : cluster [DBG] pgmap v8356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:55 smithi115 bash[46934]: audit 2022-04-22T20:59:55.163392+0000 mon.a (mon.0) 10801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T20:59:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:57 smithi121 bash[36902]: cluster 2022-04-22T20:59:56.566566+0000 mgr.y (mgr.34107) 7310 : cluster [DBG] pgmap v8357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:58.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:57 smithi115 bash[45495]: cluster 2022-04-22T20:59:56.566566+0000 mgr.y (mgr.34107) 7310 : cluster [DBG] pgmap v8357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:58.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:57 smithi115 bash[46934]: cluster 2022-04-22T20:59:56.566566+0000 mgr.y (mgr.34107) 7310 : cluster [DBG] pgmap v8357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:58 smithi121 bash[36902]: cluster 2022-04-22T20:59:58.567935+0000 mgr.y (mgr.34107) 7311 : cluster [DBG] pgmap v8358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:59.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:58 smithi115 bash[45495]: cluster 2022-04-22T20:59:58.567935+0000 mgr.y (mgr.34107) 7311 : cluster [DBG] pgmap v8358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T20:59:59.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:58 smithi115 bash[46934]: cluster 2022-04-22T20:59:58.567935+0000 mgr.y (mgr.34107) 7311 : cluster [DBG] pgmap v8358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 20:59:59 smithi121 bash[36902]: audit 2022-04-22T20:59:58.833442+0000 mon.a (mon.0) 10802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:00.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 20:59:59 smithi115 bash[45495]: audit 2022-04-22T20:59:58.833442+0000 mon.a (mon.0) 10802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 20:59:59 smithi115 bash[46934]: audit 2022-04-22T20:59:58.833442+0000 mon.a (mon.0) 10802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:00 smithi121 bash[36902]: cluster 2022-04-22T21:00:00.000162+0000 mon.a (mon.0) 10803 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:00:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:00 smithi121 bash[36902]: audit 2022-04-22T21:00:00.166279+0000 mon.a (mon.0) 10804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:00 smithi121 bash[36902]: cluster 2022-04-22T21:00:00.568973+0000 mgr.y (mgr.34107) 7312 : cluster [DBG] pgmap v8359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[45495]: cluster 2022-04-22T21:00:00.000162+0000 mon.a (mon.0) 10803 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:00:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[45495]: audit 2022-04-22T21:00:00.166279+0000 mon.a (mon.0) 10804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:01.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[45495]: cluster 2022-04-22T21:00:00.568973+0000 mgr.y (mgr.34107) 7312 : cluster [DBG] pgmap v8359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[46934]: cluster 2022-04-22T21:00:00.000162+0000 mon.a (mon.0) 10803 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:00:01.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[46934]: audit 2022-04-22T21:00:00.166279+0000 mon.a (mon.0) 10804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:00 smithi115 bash[46934]: cluster 2022-04-22T21:00:00.568973+0000 mgr.y (mgr.34107) 7312 : cluster [DBG] pgmap v8359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:02.014 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 21:00:01 smithi121 bash[49697]: ts=2022-04-22T21:00:01.672Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1650650401220 maxt=1650657600000 ulid=01G19H65PRB5HMZ6G98YVGQ06H duration=47.918593ms 2022-04-22T21:00:02.014 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 21:00:01 smithi121 bash[49697]: ts=2022-04-22T21:00:01.675Z caller=head.go:827 level=info component=tsdb msg="Head GC completed" duration=2.124644ms 2022-04-22T21:00:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:02 smithi115 bash[45495]: cluster 2022-04-22T21:00:02.570097+0000 mgr.y (mgr.34107) 7313 : cluster [DBG] pgmap v8360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:02 smithi115 bash[46934]: cluster 2022-04-22T21:00:02.570097+0000 mgr.y (mgr.34107) 7313 : cluster [DBG] pgmap v8360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:02.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:02.445Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:02 smithi121 bash[36902]: cluster 2022-04-22T21:00:02.570097+0000 mgr.y (mgr.34107) 7313 : cluster [DBG] pgmap v8360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:03 smithi121 bash[36902]: audit 2022-04-22T21:00:03.837304+0000 mon.a (mon.0) 10805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:03 smithi115 bash[45495]: audit 2022-04-22T21:00:03.837304+0000 mon.a (mon.0) 10805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:03 smithi115 bash[46934]: audit 2022-04-22T21:00:03.837304+0000 mon.a (mon.0) 10805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:04.318Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:04 smithi121 bash[36902]: cluster 2022-04-22T21:00:04.571542+0000 mgr.y (mgr.34107) 7314 : cluster [DBG] pgmap v8361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:05.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:04 smithi115 bash[45495]: cluster 2022-04-22T21:00:04.571542+0000 mgr.y (mgr.34107) 7314 : cluster [DBG] pgmap v8361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:04 smithi115 bash[46934]: cluster 2022-04-22T21:00:04.571542+0000 mgr.y (mgr.34107) 7314 : cluster [DBG] pgmap v8361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:05 smithi121 bash[36902]: audit 2022-04-22T21:00:05.169529+0000 mon.a (mon.0) 10806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:05 smithi115 bash[45495]: audit 2022-04-22T21:00:05.169529+0000 mon.a (mon.0) 10806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:05 smithi115 bash[46934]: audit 2022-04-22T21:00:05.169529+0000 mon.a (mon.0) 10806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:06 smithi121 bash[36902]: cluster 2022-04-22T21:00:06.572679+0000 mgr.y (mgr.34107) 7315 : cluster [DBG] pgmap v8362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:07.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:06 smithi115 bash[45495]: cluster 2022-04-22T21:00:06.572679+0000 mgr.y (mgr.34107) 7315 : cluster [DBG] pgmap v8362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:07.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:06 smithi115 bash[46934]: cluster 2022-04-22T21:00:06.572679+0000 mgr.y (mgr.34107) 7315 : cluster [DBG] pgmap v8362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:08 smithi121 bash[36902]: cluster 2022-04-22T21:00:08.573796+0000 mgr.y (mgr.34107) 7316 : cluster [DBG] pgmap v8363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:09.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:08 smithi115 bash[45495]: cluster 2022-04-22T21:00:08.573796+0000 mgr.y (mgr.34107) 7316 : cluster [DBG] pgmap v8363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:08 smithi115 bash[46934]: cluster 2022-04-22T21:00:08.573796+0000 mgr.y (mgr.34107) 7316 : cluster [DBG] pgmap v8363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:09 smithi121 bash[36902]: audit 2022-04-22T21:00:08.841104+0000 mon.a (mon.0) 10807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:09 smithi115 bash[45495]: audit 2022-04-22T21:00:08.841104+0000 mon.a (mon.0) 10807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:09 smithi115 bash[46934]: audit 2022-04-22T21:00:08.841104+0000 mon.a (mon.0) 10807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:10 smithi121 bash[36902]: audit 2022-04-22T21:00:10.172617+0000 mon.a (mon.0) 10808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:10 smithi121 bash[36902]: cluster 2022-04-22T21:00:10.574770+0000 mgr.y (mgr.34107) 7317 : cluster [DBG] pgmap v8364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:10 smithi115 bash[45495]: audit 2022-04-22T21:00:10.172617+0000 mon.a (mon.0) 10808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:11.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:10 smithi115 bash[45495]: cluster 2022-04-22T21:00:10.574770+0000 mgr.y (mgr.34107) 7317 : cluster [DBG] pgmap v8364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:10 smithi115 bash[46934]: audit 2022-04-22T21:00:10.172617+0000 mon.a (mon.0) 10808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:11.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:10 smithi115 bash[46934]: cluster 2022-04-22T21:00:10.574770+0000 mgr.y (mgr.34107) 7317 : cluster [DBG] pgmap v8364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:12 smithi115 bash[45495]: cluster 2022-04-22T21:00:12.575794+0000 mgr.y (mgr.34107) 7318 : cluster [DBG] pgmap v8365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:12.443Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:12.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:12 smithi115 bash[46934]: cluster 2022-04-22T21:00:12.575794+0000 mgr.y (mgr.34107) 7318 : cluster [DBG] pgmap v8365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:12 smithi121 bash[36902]: cluster 2022-04-22T21:00:12.575794+0000 mgr.y (mgr.34107) 7318 : cluster [DBG] pgmap v8365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:13 smithi121 bash[36902]: audit 2022-04-22T21:00:13.844658+0000 mon.a (mon.0) 10809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:13 smithi115 bash[45495]: audit 2022-04-22T21:00:13.844658+0000 mon.a (mon.0) 10809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:13 smithi115 bash[46934]: audit 2022-04-22T21:00:13.844658+0000 mon.a (mon.0) 10809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:14.318Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:14 smithi121 bash[36902]: cluster 2022-04-22T21:00:14.576878+0000 mgr.y (mgr.34107) 7319 : cluster [DBG] pgmap v8366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:15.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:14 smithi115 bash[45495]: cluster 2022-04-22T21:00:14.576878+0000 mgr.y (mgr.34107) 7319 : cluster [DBG] pgmap v8366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:15.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:14 smithi115 bash[46934]: cluster 2022-04-22T21:00:14.576878+0000 mgr.y (mgr.34107) 7319 : cluster [DBG] pgmap v8366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:15 smithi121 bash[36902]: audit 2022-04-22T21:00:15.175642+0000 mon.a (mon.0) 10810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:15 smithi121 bash[36902]: audit 2022-04-22T21:00:15.733508+0000 mon.a (mon.0) 10811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:00:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:15 smithi121 bash[36902]: audit 2022-04-22T21:00:15.734264+0000 mon.a (mon.0) 10812 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:00:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[45495]: audit 2022-04-22T21:00:15.175642+0000 mon.a (mon.0) 10810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[45495]: audit 2022-04-22T21:00:15.733508+0000 mon.a (mon.0) 10811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:00:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[45495]: audit 2022-04-22T21:00:15.734264+0000 mon.a (mon.0) 10812 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:00:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[46934]: audit 2022-04-22T21:00:15.175642+0000 mon.a (mon.0) 10810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[46934]: audit 2022-04-22T21:00:15.733508+0000 mon.a (mon.0) 10811 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:00:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:15 smithi115 bash[46934]: audit 2022-04-22T21:00:15.734264+0000 mon.a (mon.0) 10812 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:00:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:16 smithi121 bash[36902]: cluster 2022-04-22T21:00:16.577975+0000 mgr.y (mgr.34107) 7320 : cluster [DBG] pgmap v8367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:16 smithi115 bash[45495]: cluster 2022-04-22T21:00:16.577975+0000 mgr.y (mgr.34107) 7320 : cluster [DBG] pgmap v8367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:17.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:16 smithi115 bash[46934]: cluster 2022-04-22T21:00:16.577975+0000 mgr.y (mgr.34107) 7320 : cluster [DBG] pgmap v8367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:18 smithi121 bash[36902]: cluster 2022-04-22T21:00:18.579137+0000 mgr.y (mgr.34107) 7321 : cluster [DBG] pgmap v8368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:19.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:18 smithi115 bash[45495]: cluster 2022-04-22T21:00:18.579137+0000 mgr.y (mgr.34107) 7321 : cluster [DBG] pgmap v8368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:18 smithi115 bash[46934]: cluster 2022-04-22T21:00:18.579137+0000 mgr.y (mgr.34107) 7321 : cluster [DBG] pgmap v8368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:19 smithi121 bash[36902]: audit 2022-04-22T21:00:18.848323+0000 mon.a (mon.0) 10813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:20.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:19 smithi115 bash[45495]: audit 2022-04-22T21:00:18.848323+0000 mon.a (mon.0) 10813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:19 smithi115 bash[46934]: audit 2022-04-22T21:00:18.848323+0000 mon.a (mon.0) 10813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:20 smithi121 bash[36902]: audit 2022-04-22T21:00:20.178761+0000 mon.a (mon.0) 10814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:20 smithi121 bash[36902]: cluster 2022-04-22T21:00:20.579998+0000 mgr.y (mgr.34107) 7322 : cluster [DBG] pgmap v8369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:20 smithi115 bash[45495]: audit 2022-04-22T21:00:20.178761+0000 mon.a (mon.0) 10814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:20 smithi115 bash[45495]: cluster 2022-04-22T21:00:20.579998+0000 mgr.y (mgr.34107) 7322 : cluster [DBG] pgmap v8369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:20 smithi115 bash[46934]: audit 2022-04-22T21:00:20.178761+0000 mon.a (mon.0) 10814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:21.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:20 smithi115 bash[46934]: cluster 2022-04-22T21:00:20.579998+0000 mgr.y (mgr.34107) 7322 : cluster [DBG] pgmap v8369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:22 smithi115 bash[46934]: cluster 2022-04-22T21:00:22.581046+0000 mgr.y (mgr.34107) 7323 : cluster [DBG] pgmap v8370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:22.445Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:22.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:22 smithi115 bash[45495]: cluster 2022-04-22T21:00:22.581046+0000 mgr.y (mgr.34107) 7323 : cluster [DBG] pgmap v8370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:22.890 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:00:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:22 smithi121 bash[36902]: cluster 2022-04-22T21:00:22.581046+0000 mgr.y (mgr.34107) 7323 : cluster [DBG] pgmap v8370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:00:23.228 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:00:23.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:00:23.230 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:00:23.230 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:00:23.230 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:00:23.230 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:00:23.626 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:00:23.626 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:00:23.626 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:00:23.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:00:23.627 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:00:23.628 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:00:23.628 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:00:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:23 smithi121 bash[36902]: audit 2022-04-22T21:00:22.870002+0000 mgr.y (mgr.34107) 7324 : audit [DBG] from='client.52915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:23 smithi121 bash[36902]: audit 2022-04-22T21:00:23.208655+0000 mgr.y (mgr.34107) 7325 : audit [DBG] from='client.42990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:23 smithi121 bash[36902]: audit 2022-04-22T21:00:23.621161+0000 mon.c (mon.1) 271 : audit [DBG] from='client.? 172.21.15.115:0/2713220362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:24.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[45495]: audit 2022-04-22T21:00:22.870002+0000 mgr.y (mgr.34107) 7324 : audit [DBG] from='client.52915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[45495]: audit 2022-04-22T21:00:23.208655+0000 mgr.y (mgr.34107) 7325 : audit [DBG] from='client.42990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[45495]: audit 2022-04-22T21:00:23.621161+0000 mon.c (mon.1) 271 : audit [DBG] from='client.? 172.21.15.115:0/2713220362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[46934]: audit 2022-04-22T21:00:22.870002+0000 mgr.y (mgr.34107) 7324 : audit [DBG] from='client.52915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[46934]: audit 2022-04-22T21:00:23.208655+0000 mgr.y (mgr.34107) 7325 : audit [DBG] from='client.42990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:24.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:23 smithi115 bash[46934]: audit 2022-04-22T21:00:23.621161+0000 mon.c (mon.1) 271 : audit [DBG] from='client.? 172.21.15.115:0/2713220362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:24.659 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:24.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:24.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:24 smithi115 bash[46934]: audit 2022-04-22T21:00:23.851902+0000 mon.a (mon.0) 10815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:24.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:24 smithi115 bash[46934]: cluster 2022-04-22T21:00:24.582121+0000 mgr.y (mgr.34107) 7326 : cluster [DBG] pgmap v8371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:24 smithi121 bash[36902]: audit 2022-04-22T21:00:23.851902+0000 mon.a (mon.0) 10815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:24 smithi121 bash[36902]: cluster 2022-04-22T21:00:24.582121+0000 mgr.y (mgr.34107) 7326 : cluster [DBG] pgmap v8371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:25.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:24 smithi115 bash[45495]: audit 2022-04-22T21:00:23.851902+0000 mon.a (mon.0) 10815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:25.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:24 smithi115 bash[45495]: cluster 2022-04-22T21:00:24.582121+0000 mgr.y (mgr.34107) 7326 : cluster [DBG] pgmap v8371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:25 smithi121 bash[36902]: audit 2022-04-22T21:00:25.181810+0000 mon.a (mon.0) 10816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:26.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:25 smithi115 bash[45495]: audit 2022-04-22T21:00:25.181810+0000 mon.a (mon.0) 10816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:26.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:25 smithi115 bash[46934]: audit 2022-04-22T21:00:25.181810+0000 mon.a (mon.0) 10816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:26 smithi121 bash[36902]: cluster 2022-04-22T21:00:26.583250+0000 mgr.y (mgr.34107) 7327 : cluster [DBG] pgmap v8372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:26 smithi115 bash[45495]: cluster 2022-04-22T21:00:26.583250+0000 mgr.y (mgr.34107) 7327 : cluster [DBG] pgmap v8372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:26 smithi115 bash[46934]: cluster 2022-04-22T21:00:26.583250+0000 mgr.y (mgr.34107) 7327 : cluster [DBG] pgmap v8372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:28 smithi121 bash[36902]: cluster 2022-04-22T21:00:28.584255+0000 mgr.y (mgr.34107) 7328 : cluster [DBG] pgmap v8373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:29.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:28 smithi115 bash[45495]: cluster 2022-04-22T21:00:28.584255+0000 mgr.y (mgr.34107) 7328 : cluster [DBG] pgmap v8373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:28 smithi115 bash[46934]: cluster 2022-04-22T21:00:28.584255+0000 mgr.y (mgr.34107) 7328 : cluster [DBG] pgmap v8373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:29 smithi121 bash[36902]: audit 2022-04-22T21:00:28.856201+0000 mon.a (mon.0) 10817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:29 smithi115 bash[45495]: audit 2022-04-22T21:00:28.856201+0000 mon.a (mon.0) 10817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:30.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:29 smithi115 bash[46934]: audit 2022-04-22T21:00:28.856201+0000 mon.a (mon.0) 10817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:30 smithi121 bash[36902]: audit 2022-04-22T21:00:30.184800+0000 mon.a (mon.0) 10818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:30 smithi121 bash[36902]: cluster 2022-04-22T21:00:30.585276+0000 mgr.y (mgr.34107) 7329 : cluster [DBG] pgmap v8374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:30 smithi115 bash[45495]: audit 2022-04-22T21:00:30.184800+0000 mon.a (mon.0) 10818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:31.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:30 smithi115 bash[45495]: cluster 2022-04-22T21:00:30.585276+0000 mgr.y (mgr.34107) 7329 : cluster [DBG] pgmap v8374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:30 smithi115 bash[46934]: audit 2022-04-22T21:00:30.184800+0000 mon.a (mon.0) 10818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:31.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:30 smithi115 bash[46934]: cluster 2022-04-22T21:00:30.585276+0000 mgr.y (mgr.34107) 7329 : cluster [DBG] pgmap v8374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[46934]: audit 2022-04-22T21:00:31.754962+0000 mon.a (mon.0) 10819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:00:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[46934]: audit 2022-04-22T21:00:31.757907+0000 mon.a (mon.0) 10820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:00:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[46934]: audit 2022-04-22T21:00:31.759646+0000 mon.a (mon.0) 10821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:00:32.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[45495]: audit 2022-04-22T21:00:31.754962+0000 mon.a (mon.0) 10819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:00:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[45495]: audit 2022-04-22T21:00:31.757907+0000 mon.a (mon.0) 10820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:00:32.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:31 smithi115 bash[45495]: audit 2022-04-22T21:00:31.759646+0000 mon.a (mon.0) 10821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:00:32.113 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:31 smithi121 bash[36902]: audit 2022-04-22T21:00:31.754962+0000 mon.a (mon.0) 10819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:00:32.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:31 smithi121 bash[36902]: audit 2022-04-22T21:00:31.757907+0000 mon.a (mon.0) 10820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:00:32.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:31 smithi121 bash[36902]: audit 2022-04-22T21:00:31.759646+0000 mon.a (mon.0) 10821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:00:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:32.447Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:33 smithi121 bash[36902]: audit 2022-04-22T21:00:32.126836+0000 mon.a (mon.0) 10822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:00:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:33 smithi121 bash[36902]: cluster 2022-04-22T21:00:32.586473+0000 mgr.y (mgr.34107) 7330 : cluster [DBG] pgmap v8375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:33 smithi115 bash[45495]: audit 2022-04-22T21:00:32.126836+0000 mon.a (mon.0) 10822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:00:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:33 smithi115 bash[45495]: cluster 2022-04-22T21:00:32.586473+0000 mgr.y (mgr.34107) 7330 : cluster [DBG] pgmap v8375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:33 smithi115 bash[46934]: audit 2022-04-22T21:00:32.126836+0000 mon.a (mon.0) 10822 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:00:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:33 smithi115 bash[46934]: cluster 2022-04-22T21:00:32.586473+0000 mgr.y (mgr.34107) 7330 : cluster [DBG] pgmap v8375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:34 smithi121 bash[36902]: audit 2022-04-22T21:00:33.859758+0000 mon.a (mon.0) 10823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:34 smithi115 bash[45495]: audit 2022-04-22T21:00:33.859758+0000 mon.a (mon.0) 10823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:34 smithi115 bash[46934]: audit 2022-04-22T21:00:33.859758+0000 mon.a (mon.0) 10823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:34.543 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:34.322Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:35 smithi121 bash[36902]: cluster 2022-04-22T21:00:34.587599+0000 mgr.y (mgr.34107) 7331 : cluster [DBG] pgmap v8376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:35.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:35 smithi115 bash[45495]: cluster 2022-04-22T21:00:34.587599+0000 mgr.y (mgr.34107) 7331 : cluster [DBG] pgmap v8376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:35.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:35 smithi115 bash[46934]: cluster 2022-04-22T21:00:34.587599+0000 mgr.y (mgr.34107) 7331 : cluster [DBG] pgmap v8376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:36 smithi121 bash[36902]: audit 2022-04-22T21:00:35.187837+0000 mon.a (mon.0) 10824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:36 smithi115 bash[45495]: audit 2022-04-22T21:00:35.187837+0000 mon.a (mon.0) 10824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:36 smithi115 bash[46934]: audit 2022-04-22T21:00:35.187837+0000 mon.a (mon.0) 10824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:37 smithi121 bash[36902]: cluster 2022-04-22T21:00:36.588511+0000 mgr.y (mgr.34107) 7332 : cluster [DBG] pgmap v8377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:37.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:37 smithi115 bash[45495]: cluster 2022-04-22T21:00:36.588511+0000 mgr.y (mgr.34107) 7332 : cluster [DBG] pgmap v8377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:37 smithi115 bash[46934]: cluster 2022-04-22T21:00:36.588511+0000 mgr.y (mgr.34107) 7332 : cluster [DBG] pgmap v8377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:38 smithi121 bash[36902]: cluster 2022-04-22T21:00:38.589587+0000 mgr.y (mgr.34107) 7333 : cluster [DBG] pgmap v8378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:38 smithi115 bash[45495]: cluster 2022-04-22T21:00:38.589587+0000 mgr.y (mgr.34107) 7333 : cluster [DBG] pgmap v8378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:38 smithi115 bash[46934]: cluster 2022-04-22T21:00:38.589587+0000 mgr.y (mgr.34107) 7333 : cluster [DBG] pgmap v8378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:39 smithi121 bash[36902]: audit 2022-04-22T21:00:38.863431+0000 mon.a (mon.0) 10825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:39 smithi115 bash[45495]: audit 2022-04-22T21:00:38.863431+0000 mon.a (mon.0) 10825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:39 smithi115 bash[46934]: audit 2022-04-22T21:00:38.863431+0000 mon.a (mon.0) 10825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:40 smithi121 bash[36902]: audit 2022-04-22T21:00:40.191074+0000 mon.a (mon.0) 10826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:40 smithi121 bash[36902]: cluster 2022-04-22T21:00:40.590628+0000 mgr.y (mgr.34107) 7334 : cluster [DBG] pgmap v8379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:40 smithi115 bash[46934]: audit 2022-04-22T21:00:40.191074+0000 mon.a (mon.0) 10826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:40 smithi115 bash[46934]: cluster 2022-04-22T21:00:40.590628+0000 mgr.y (mgr.34107) 7334 : cluster [DBG] pgmap v8379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:40 smithi115 bash[45495]: audit 2022-04-22T21:00:40.191074+0000 mon.a (mon.0) 10826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:40 smithi115 bash[45495]: cluster 2022-04-22T21:00:40.590628+0000 mgr.y (mgr.34107) 7334 : cluster [DBG] pgmap v8379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:42 smithi115 bash[45495]: cluster 2022-04-22T21:00:42.591749+0000 mgr.y (mgr.34107) 7335 : cluster [DBG] pgmap v8380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:42.448Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:42.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:42 smithi115 bash[46934]: cluster 2022-04-22T21:00:42.591749+0000 mgr.y (mgr.34107) 7335 : cluster [DBG] pgmap v8380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:42 smithi121 bash[36902]: cluster 2022-04-22T21:00:42.591749+0000 mgr.y (mgr.34107) 7335 : cluster [DBG] pgmap v8380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:43 smithi121 bash[36902]: audit 2022-04-22T21:00:43.867113+0000 mon.a (mon.0) 10827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:44.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:43 smithi115 bash[45495]: audit 2022-04-22T21:00:43.867113+0000 mon.a (mon.0) 10827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:43 smithi115 bash[46934]: audit 2022-04-22T21:00:43.867113+0000 mon.a (mon.0) 10827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:44.321Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:44 smithi121 bash[36902]: cluster 2022-04-22T21:00:44.592870+0000 mgr.y (mgr.34107) 7336 : cluster [DBG] pgmap v8381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:44 smithi115 bash[45495]: cluster 2022-04-22T21:00:44.592870+0000 mgr.y (mgr.34107) 7336 : cluster [DBG] pgmap v8381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:44 smithi115 bash[46934]: cluster 2022-04-22T21:00:44.592870+0000 mgr.y (mgr.34107) 7336 : cluster [DBG] pgmap v8381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:45 smithi121 bash[36902]: audit 2022-04-22T21:00:45.194060+0000 mon.a (mon.0) 10828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:45 smithi115 bash[45495]: audit 2022-04-22T21:00:45.194060+0000 mon.a (mon.0) 10828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:45 smithi115 bash[46934]: audit 2022-04-22T21:00:45.194060+0000 mon.a (mon.0) 10828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:46 smithi121 bash[36902]: cluster 2022-04-22T21:00:46.593981+0000 mgr.y (mgr.34107) 7337 : cluster [DBG] pgmap v8382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:46 smithi115 bash[45495]: cluster 2022-04-22T21:00:46.593981+0000 mgr.y (mgr.34107) 7337 : cluster [DBG] pgmap v8382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:46 smithi115 bash[46934]: cluster 2022-04-22T21:00:46.593981+0000 mgr.y (mgr.34107) 7337 : cluster [DBG] pgmap v8382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:48 smithi121 bash[36902]: cluster 2022-04-22T21:00:48.595136+0000 mgr.y (mgr.34107) 7338 : cluster [DBG] pgmap v8383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:48 smithi115 bash[45495]: cluster 2022-04-22T21:00:48.595136+0000 mgr.y (mgr.34107) 7338 : cluster [DBG] pgmap v8383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:48 smithi115 bash[46934]: cluster 2022-04-22T21:00:48.595136+0000 mgr.y (mgr.34107) 7338 : cluster [DBG] pgmap v8383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:49 smithi121 bash[36902]: audit 2022-04-22T21:00:48.870943+0000 mon.a (mon.0) 10829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:49 smithi115 bash[45495]: audit 2022-04-22T21:00:48.870943+0000 mon.a (mon.0) 10829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:50.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:49 smithi115 bash[46934]: audit 2022-04-22T21:00:48.870943+0000 mon.a (mon.0) 10829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:50 smithi121 bash[36902]: audit 2022-04-22T21:00:50.197029+0000 mon.a (mon.0) 10830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:50 smithi121 bash[36902]: cluster 2022-04-22T21:00:50.596341+0000 mgr.y (mgr.34107) 7339 : cluster [DBG] pgmap v8384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:50 smithi115 bash[45495]: audit 2022-04-22T21:00:50.197029+0000 mon.a (mon.0) 10830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:50 smithi115 bash[45495]: cluster 2022-04-22T21:00:50.596341+0000 mgr.y (mgr.34107) 7339 : cluster [DBG] pgmap v8384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:50 smithi115 bash[46934]: audit 2022-04-22T21:00:50.197029+0000 mon.a (mon.0) 10830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:51.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:50 smithi115 bash[46934]: cluster 2022-04-22T21:00:50.596341+0000 mgr.y (mgr.34107) 7339 : cluster [DBG] pgmap v8384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:52.447Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:54.005 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:00:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:53 smithi121 bash[36902]: cluster 2022-04-22T21:00:52.597565+0000 mgr.y (mgr.34107) 7340 : cluster [DBG] pgmap v8385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:54.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:53 smithi115 bash[45495]: cluster 2022-04-22T21:00:52.597565+0000 mgr.y (mgr.34107) 7340 : cluster [DBG] pgmap v8385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:54.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:53 smithi115 bash[46934]: cluster 2022-04-22T21:00:52.597565+0000 mgr.y (mgr.34107) 7340 : cluster [DBG] pgmap v8385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:00:54.357 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:00:54.358 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:00:54.358 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:00:54.358 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:00:54.358 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:00:54.358 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T21:00:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:00:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:00:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:00:54.359 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:00:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:00:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:00:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:00:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:00:54.360 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:00:54.666 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:00:54.320Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:00:54.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[45495]: audit 2022-04-22T21:00:53.874569+0000 mon.a (mon.0) 10831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:54.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[45495]: audit 2022-04-22T21:00:53.982005+0000 mgr.y (mgr.34107) 7341 : audit [DBG] from='client.52933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:54.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[45495]: audit 2022-04-22T21:00:54.331816+0000 mgr.y (mgr.34107) 7342 : audit [DBG] from='client.52939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:54.755 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:54.756 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:00:54.757 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:00:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:54 smithi121 bash[36902]: audit 2022-04-22T21:00:53.874569+0000 mon.a (mon.0) 10831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:54 smithi121 bash[36902]: audit 2022-04-22T21:00:53.982005+0000 mgr.y (mgr.34107) 7341 : audit [DBG] from='client.52933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:54 smithi121 bash[36902]: audit 2022-04-22T21:00:54.331816+0000 mgr.y (mgr.34107) 7342 : audit [DBG] from='client.52939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[46934]: audit 2022-04-22T21:00:53.874569+0000 mon.a (mon.0) 10831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:00:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[46934]: audit 2022-04-22T21:00:53.982005+0000 mgr.y (mgr.34107) 7341 : audit [DBG] from='client.52933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:54 smithi115 bash[46934]: audit 2022-04-22T21:00:54.331816+0000 mgr.y (mgr.34107) 7342 : audit [DBG] from='client.52939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:00:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:55 smithi121 bash[36902]: cluster 2022-04-22T21:00:54.598630+0000 mgr.y (mgr.34107) 7343 : cluster [DBG] pgmap v8386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:55 smithi121 bash[36902]: audit 2022-04-22T21:00:54.750552+0000 mon.a (mon.0) 10832 : audit [DBG] from='client.? 172.21.15.115:0/925494579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:55 smithi121 bash[36902]: audit 2022-04-22T21:00:55.200214+0000 mon.a (mon.0) 10833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[45495]: cluster 2022-04-22T21:00:54.598630+0000 mgr.y (mgr.34107) 7343 : cluster [DBG] pgmap v8386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[45495]: audit 2022-04-22T21:00:54.750552+0000 mon.a (mon.0) 10832 : audit [DBG] from='client.? 172.21.15.115:0/925494579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:56.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[45495]: audit 2022-04-22T21:00:55.200214+0000 mon.a (mon.0) 10833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[46934]: cluster 2022-04-22T21:00:54.598630+0000 mgr.y (mgr.34107) 7343 : cluster [DBG] pgmap v8386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[46934]: audit 2022-04-22T21:00:54.750552+0000 mon.a (mon.0) 10832 : audit [DBG] from='client.? 172.21.15.115:0/925494579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:00:56.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:55 smithi115 bash[46934]: audit 2022-04-22T21:00:55.200214+0000 mon.a (mon.0) 10833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:00:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:57 smithi121 bash[36902]: cluster 2022-04-22T21:00:56.599746+0000 mgr.y (mgr.34107) 7344 : cluster [DBG] pgmap v8387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:58.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:57 smithi115 bash[46934]: cluster 2022-04-22T21:00:56.599746+0000 mgr.y (mgr.34107) 7344 : cluster [DBG] pgmap v8387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:00:58.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:57 smithi115 bash[45495]: cluster 2022-04-22T21:00:56.599746+0000 mgr.y (mgr.34107) 7344 : cluster [DBG] pgmap v8387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:59 smithi121 bash[36902]: cluster 2022-04-22T21:00:58.600880+0000 mgr.y (mgr.34107) 7345 : cluster [DBG] pgmap v8388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:00:59 smithi121 bash[36902]: audit 2022-04-22T21:00:58.878103+0000 mon.a (mon.0) 10834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:00.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:59 smithi115 bash[45495]: cluster 2022-04-22T21:00:58.600880+0000 mgr.y (mgr.34107) 7345 : cluster [DBG] pgmap v8388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:00.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:00:59 smithi115 bash[45495]: audit 2022-04-22T21:00:58.878103+0000 mon.a (mon.0) 10834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:00.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:59 smithi115 bash[46934]: cluster 2022-04-22T21:00:58.600880+0000 mgr.y (mgr.34107) 7345 : cluster [DBG] pgmap v8388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:00.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:00:59 smithi115 bash[46934]: audit 2022-04-22T21:00:58.878103+0000 mon.a (mon.0) 10834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:00 smithi115 bash[45495]: audit 2022-04-22T21:01:00.203095+0000 mon.a (mon.0) 10835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:01.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:00 smithi115 bash[46934]: audit 2022-04-22T21:01:00.203095+0000 mon.a (mon.0) 10835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:00 smithi121 bash[36902]: audit 2022-04-22T21:01:00.203095+0000 mon.a (mon.0) 10835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:02.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:01 smithi115 bash[45495]: cluster 2022-04-22T21:01:00.601933+0000 mgr.y (mgr.34107) 7346 : cluster [DBG] pgmap v8389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:01 smithi115 bash[46934]: cluster 2022-04-22T21:01:00.601933+0000 mgr.y (mgr.34107) 7346 : cluster [DBG] pgmap v8389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:01 smithi121 bash[36902]: cluster 2022-04-22T21:01:00.601933+0000 mgr.y (mgr.34107) 7346 : cluster [DBG] pgmap v8389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:02.449Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:04.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:03 smithi115 bash[45495]: cluster 2022-04-22T21:01:02.603145+0000 mgr.y (mgr.34107) 7347 : cluster [DBG] pgmap v8390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:03 smithi115 bash[46934]: cluster 2022-04-22T21:01:02.603145+0000 mgr.y (mgr.34107) 7347 : cluster [DBG] pgmap v8390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:03 smithi121 bash[36902]: cluster 2022-04-22T21:01:02.603145+0000 mgr.y (mgr.34107) 7347 : cluster [DBG] pgmap v8390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:04.779 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:04.322Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:04 smithi115 bash[45495]: audit 2022-04-22T21:01:03.881882+0000 mon.a (mon.0) 10836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:04 smithi115 bash[46934]: audit 2022-04-22T21:01:03.881882+0000 mon.a (mon.0) 10836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:04 smithi121 bash[36902]: audit 2022-04-22T21:01:03.881882+0000 mon.a (mon.0) 10836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:05 smithi115 bash[45495]: cluster 2022-04-22T21:01:04.604284+0000 mgr.y (mgr.34107) 7348 : cluster [DBG] pgmap v8391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:05 smithi115 bash[45495]: audit 2022-04-22T21:01:05.205973+0000 mon.a (mon.0) 10837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:05 smithi115 bash[46934]: cluster 2022-04-22T21:01:04.604284+0000 mgr.y (mgr.34107) 7348 : cluster [DBG] pgmap v8391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:05 smithi115 bash[46934]: audit 2022-04-22T21:01:05.205973+0000 mon.a (mon.0) 10837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:05 smithi121 bash[36902]: cluster 2022-04-22T21:01:04.604284+0000 mgr.y (mgr.34107) 7348 : cluster [DBG] pgmap v8391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:05 smithi121 bash[36902]: audit 2022-04-22T21:01:05.205973+0000 mon.a (mon.0) 10837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:08.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:07 smithi115 bash[45495]: cluster 2022-04-22T21:01:06.605395+0000 mgr.y (mgr.34107) 7349 : cluster [DBG] pgmap v8392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:08.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:07 smithi115 bash[46934]: cluster 2022-04-22T21:01:06.605395+0000 mgr.y (mgr.34107) 7349 : cluster [DBG] pgmap v8392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:07 smithi121 bash[36902]: cluster 2022-04-22T21:01:06.605395+0000 mgr.y (mgr.34107) 7349 : cluster [DBG] pgmap v8392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:09 smithi115 bash[45495]: cluster 2022-04-22T21:01:08.606570+0000 mgr.y (mgr.34107) 7350 : cluster [DBG] pgmap v8393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:10.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:09 smithi115 bash[45495]: audit 2022-04-22T21:01:08.885696+0000 mon.a (mon.0) 10838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:10.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:09 smithi115 bash[46934]: cluster 2022-04-22T21:01:08.606570+0000 mgr.y (mgr.34107) 7350 : cluster [DBG] pgmap v8393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:10.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:09 smithi115 bash[46934]: audit 2022-04-22T21:01:08.885696+0000 mon.a (mon.0) 10838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:09 smithi121 bash[36902]: cluster 2022-04-22T21:01:08.606570+0000 mgr.y (mgr.34107) 7350 : cluster [DBG] pgmap v8393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:09 smithi121 bash[36902]: audit 2022-04-22T21:01:08.885696+0000 mon.a (mon.0) 10838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:10 smithi121 bash[36902]: audit 2022-04-22T21:01:10.209029+0000 mon.a (mon.0) 10839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:11.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:10 smithi115 bash[46934]: audit 2022-04-22T21:01:10.209029+0000 mon.a (mon.0) 10839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:11.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:10 smithi115 bash[45495]: audit 2022-04-22T21:01:10.209029+0000 mon.a (mon.0) 10839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:11 smithi121 bash[36902]: cluster 2022-04-22T21:01:10.607665+0000 mgr.y (mgr.34107) 7351 : cluster [DBG] pgmap v8394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:12.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:11 smithi115 bash[45495]: cluster 2022-04-22T21:01:10.607665+0000 mgr.y (mgr.34107) 7351 : cluster [DBG] pgmap v8394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:11 smithi115 bash[46934]: cluster 2022-04-22T21:01:10.607665+0000 mgr.y (mgr.34107) 7351 : cluster [DBG] pgmap v8394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:12.448Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:13 smithi121 bash[36902]: cluster 2022-04-22T21:01:12.608859+0000 mgr.y (mgr.34107) 7352 : cluster [DBG] pgmap v8395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:13 smithi115 bash[45495]: cluster 2022-04-22T21:01:12.608859+0000 mgr.y (mgr.34107) 7352 : cluster [DBG] pgmap v8395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:14.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:13 smithi115 bash[46934]: cluster 2022-04-22T21:01:12.608859+0000 mgr.y (mgr.34107) 7352 : cluster [DBG] pgmap v8395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:14.325Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:14 smithi121 bash[36902]: audit 2022-04-22T21:01:13.889489+0000 mon.a (mon.0) 10840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:15.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:14 smithi115 bash[45495]: audit 2022-04-22T21:01:13.889489+0000 mon.a (mon.0) 10840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:14 smithi115 bash[46934]: audit 2022-04-22T21:01:13.889489+0000 mon.a (mon.0) 10840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:15 smithi121 bash[36902]: cluster 2022-04-22T21:01:14.609972+0000 mgr.y (mgr.34107) 7353 : cluster [DBG] pgmap v8396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:15 smithi121 bash[36902]: audit 2022-04-22T21:01:15.211985+0000 mon.a (mon.0) 10841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:15 smithi121 bash[36902]: audit 2022-04-22T21:01:15.745877+0000 mon.a (mon.0) 10842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:01:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:15 smithi121 bash[36902]: audit 2022-04-22T21:01:15.746722+0000 mon.a (mon.0) 10843 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:01:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[45495]: cluster 2022-04-22T21:01:14.609972+0000 mgr.y (mgr.34107) 7353 : cluster [DBG] pgmap v8396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[45495]: audit 2022-04-22T21:01:15.211985+0000 mon.a (mon.0) 10841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[45495]: audit 2022-04-22T21:01:15.745877+0000 mon.a (mon.0) 10842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:01:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[45495]: audit 2022-04-22T21:01:15.746722+0000 mon.a (mon.0) 10843 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:01:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[46934]: cluster 2022-04-22T21:01:14.609972+0000 mgr.y (mgr.34107) 7353 : cluster [DBG] pgmap v8396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[46934]: audit 2022-04-22T21:01:15.211985+0000 mon.a (mon.0) 10841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[46934]: audit 2022-04-22T21:01:15.745877+0000 mon.a (mon.0) 10842 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:01:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:15 smithi115 bash[46934]: audit 2022-04-22T21:01:15.746722+0000 mon.a (mon.0) 10843 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:01:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:17 smithi121 bash[36902]: cluster 2022-04-22T21:01:16.610951+0000 mgr.y (mgr.34107) 7354 : cluster [DBG] pgmap v8397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:17 smithi115 bash[45495]: cluster 2022-04-22T21:01:16.610951+0000 mgr.y (mgr.34107) 7354 : cluster [DBG] pgmap v8397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:17 smithi115 bash[46934]: cluster 2022-04-22T21:01:16.610951+0000 mgr.y (mgr.34107) 7354 : cluster [DBG] pgmap v8397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:19 smithi121 bash[36902]: cluster 2022-04-22T21:01:18.612112+0000 mgr.y (mgr.34107) 7355 : cluster [DBG] pgmap v8398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:19 smithi121 bash[36902]: audit 2022-04-22T21:01:18.893259+0000 mon.a (mon.0) 10844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:19 smithi115 bash[45495]: cluster 2022-04-22T21:01:18.612112+0000 mgr.y (mgr.34107) 7355 : cluster [DBG] pgmap v8398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:19 smithi115 bash[45495]: audit 2022-04-22T21:01:18.893259+0000 mon.a (mon.0) 10844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:20.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:19 smithi115 bash[46934]: cluster 2022-04-22T21:01:18.612112+0000 mgr.y (mgr.34107) 7355 : cluster [DBG] pgmap v8398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:20.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:19 smithi115 bash[46934]: audit 2022-04-22T21:01:18.893259+0000 mon.a (mon.0) 10844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:20 smithi121 bash[36902]: audit 2022-04-22T21:01:20.215113+0000 mon.a (mon.0) 10845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:21.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:20 smithi115 bash[45495]: audit 2022-04-22T21:01:20.215113+0000 mon.a (mon.0) 10845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:21.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:20 smithi115 bash[46934]: audit 2022-04-22T21:01:20.215113+0000 mon.a (mon.0) 10845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:21 smithi121 bash[36902]: cluster 2022-04-22T21:01:20.613101+0000 mgr.y (mgr.34107) 7356 : cluster [DBG] pgmap v8399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:22.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:21 smithi115 bash[45495]: cluster 2022-04-22T21:01:20.613101+0000 mgr.y (mgr.34107) 7356 : cluster [DBG] pgmap v8399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:21 smithi115 bash[46934]: cluster 2022-04-22T21:01:20.613101+0000 mgr.y (mgr.34107) 7356 : cluster [DBG] pgmap v8399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:22.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:23 smithi121 bash[36902]: cluster 2022-04-22T21:01:22.614177+0000 mgr.y (mgr.34107) 7357 : cluster [DBG] pgmap v8400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:24.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:23 smithi115 bash[45495]: cluster 2022-04-22T21:01:22.614177+0000 mgr.y (mgr.34107) 7357 : cluster [DBG] pgmap v8400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:23 smithi115 bash[46934]: cluster 2022-04-22T21:01:22.614177+0000 mgr.y (mgr.34107) 7357 : cluster [DBG] pgmap v8400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:24.325Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:25.122 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:01:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:24 smithi121 bash[36902]: audit 2022-04-22T21:01:23.896999+0000 mon.a (mon.0) 10846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:25.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:24 smithi115 bash[45495]: audit 2022-04-22T21:01:23.896999+0000 mon.a (mon.0) 10846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:25.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:24 smithi115 bash[46934]: audit 2022-04-22T21:01:23.896999+0000 mon.a (mon.0) 10846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:01:25.469 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:01:25.470 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:01:25.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:01:25.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:01:25.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:01:25.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:01:25.471 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:01:25.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:01:25.875 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:01:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:25 smithi121 bash[36902]: cluster 2022-04-22T21:01:24.615126+0000 mgr.y (mgr.34107) 7358 : cluster [DBG] pgmap v8401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:26.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:25 smithi121 bash[36902]: audit 2022-04-22T21:01:25.101638+0000 mgr.y (mgr.34107) 7359 : audit [DBG] from='client.43020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:26.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:25 smithi121 bash[36902]: audit 2022-04-22T21:01:25.217964+0000 mon.a (mon.0) 10847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:26.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:25 smithi121 bash[36902]: audit 2022-04-22T21:01:25.444678+0000 mgr.y (mgr.34107) 7360 : audit [DBG] from='client.52957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[45495]: cluster 2022-04-22T21:01:24.615126+0000 mgr.y (mgr.34107) 7358 : cluster [DBG] pgmap v8401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[45495]: audit 2022-04-22T21:01:25.101638+0000 mgr.y (mgr.34107) 7359 : audit [DBG] from='client.43020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[45495]: audit 2022-04-22T21:01:25.217964+0000 mon.a (mon.0) 10847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[45495]: audit 2022-04-22T21:01:25.444678+0000 mgr.y (mgr.34107) 7360 : audit [DBG] from='client.52957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[46934]: cluster 2022-04-22T21:01:24.615126+0000 mgr.y (mgr.34107) 7358 : cluster [DBG] pgmap v8401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[46934]: audit 2022-04-22T21:01:25.101638+0000 mgr.y (mgr.34107) 7359 : audit [DBG] from='client.43020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[46934]: audit 2022-04-22T21:01:25.217964+0000 mon.a (mon.0) 10847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:25 smithi115 bash[46934]: audit 2022-04-22T21:01:25.444678+0000 mgr.y (mgr.34107) 7360 : audit [DBG] from='client.52957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:26 smithi121 bash[36902]: audit 2022-04-22T21:01:25.869566+0000 mon.c (mon.1) 272 : audit [DBG] from='client.? 172.21.15.115:0/1319350424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:01:27.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:26 smithi115 bash[45495]: audit 2022-04-22T21:01:25.869566+0000 mon.c (mon.1) 272 : audit [DBG] from='client.? 172.21.15.115:0/1319350424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:01:27.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:26 smithi115 bash[46934]: audit 2022-04-22T21:01:25.869566+0000 mon.c (mon.1) 272 : audit [DBG] from='client.? 172.21.15.115:0/1319350424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:01:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:27 smithi121 bash[36902]: cluster 2022-04-22T21:01:26.616226+0000 mgr.y (mgr.34107) 7361 : cluster [DBG] pgmap v8402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:28.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:27 smithi115 bash[45495]: cluster 2022-04-22T21:01:26.616226+0000 mgr.y (mgr.34107) 7361 : cluster [DBG] pgmap v8402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:28.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:27 smithi115 bash[46934]: cluster 2022-04-22T21:01:26.616226+0000 mgr.y (mgr.34107) 7361 : cluster [DBG] pgmap v8402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:29 smithi121 bash[36902]: cluster 2022-04-22T21:01:28.617358+0000 mgr.y (mgr.34107) 7362 : cluster [DBG] pgmap v8403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:29 smithi121 bash[36902]: audit 2022-04-22T21:01:28.900665+0000 mon.a (mon.0) 10848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:29 smithi115 bash[45495]: cluster 2022-04-22T21:01:28.617358+0000 mgr.y (mgr.34107) 7362 : cluster [DBG] pgmap v8403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:29 smithi115 bash[45495]: audit 2022-04-22T21:01:28.900665+0000 mon.a (mon.0) 10848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:29 smithi115 bash[46934]: cluster 2022-04-22T21:01:28.617358+0000 mgr.y (mgr.34107) 7362 : cluster [DBG] pgmap v8403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:29 smithi115 bash[46934]: audit 2022-04-22T21:01:28.900665+0000 mon.a (mon.0) 10848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:30 smithi121 bash[36902]: audit 2022-04-22T21:01:30.220970+0000 mon.a (mon.0) 10849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:30 smithi115 bash[45495]: audit 2022-04-22T21:01:30.220970+0000 mon.a (mon.0) 10849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:31.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:30 smithi115 bash[46934]: audit 2022-04-22T21:01:30.220970+0000 mon.a (mon.0) 10849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:32.163 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:31 smithi115 bash[45495]: cluster 2022-04-22T21:01:30.618376+0000 mgr.y (mgr.34107) 7363 : cluster [DBG] pgmap v8404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:32.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:31 smithi115 bash[46934]: cluster 2022-04-22T21:01:30.618376+0000 mgr.y (mgr.34107) 7363 : cluster [DBG] pgmap v8404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:32.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:31 smithi121 bash[36902]: cluster 2022-04-22T21:01:30.618376+0000 mgr.y (mgr.34107) 7363 : cluster [DBG] pgmap v8404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:32.449Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:32 smithi121 bash[36902]: audit 2022-04-22T21:01:32.135643+0000 mon.a (mon.0) 10850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:01:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:32 smithi121 bash[36902]: audit 2022-04-22T21:01:32.138676+0000 mon.a (mon.0) 10851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:01:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:32 smithi121 bash[36902]: audit 2022-04-22T21:01:32.140364+0000 mon.a (mon.0) 10852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:01:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:32 smithi121 bash[36902]: audit 2022-04-22T21:01:32.511531+0000 mon.a (mon.0) 10853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:01:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[45495]: audit 2022-04-22T21:01:32.135643+0000 mon.a (mon.0) 10850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:01:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[45495]: audit 2022-04-22T21:01:32.138676+0000 mon.a (mon.0) 10851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:01:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[45495]: audit 2022-04-22T21:01:32.140364+0000 mon.a (mon.0) 10852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:01:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[45495]: audit 2022-04-22T21:01:32.511531+0000 mon.a (mon.0) 10853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:01:33.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[46934]: audit 2022-04-22T21:01:32.135643+0000 mon.a (mon.0) 10850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:01:33.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[46934]: audit 2022-04-22T21:01:32.138676+0000 mon.a (mon.0) 10851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:01:33.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[46934]: audit 2022-04-22T21:01:32.140364+0000 mon.a (mon.0) 10852 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:01:33.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:32 smithi115 bash[46934]: audit 2022-04-22T21:01:32.511531+0000 mon.a (mon.0) 10853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:01:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:34.326Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:34 smithi115 bash[46934]: cluster 2022-04-22T21:01:32.619464+0000 mgr.y (mgr.34107) 7364 : cluster [DBG] pgmap v8405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:34 smithi115 bash[46934]: audit 2022-04-22T21:01:33.904074+0000 mon.a (mon.0) 10854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:34 smithi115 bash[45495]: cluster 2022-04-22T21:01:32.619464+0000 mgr.y (mgr.34107) 7364 : cluster [DBG] pgmap v8405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:34 smithi115 bash[45495]: audit 2022-04-22T21:01:33.904074+0000 mon.a (mon.0) 10854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:34 smithi121 bash[36902]: cluster 2022-04-22T21:01:32.619464+0000 mgr.y (mgr.34107) 7364 : cluster [DBG] pgmap v8405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:34 smithi121 bash[36902]: audit 2022-04-22T21:01:33.904074+0000 mon.a (mon.0) 10854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:35 smithi121 bash[36902]: cluster 2022-04-22T21:01:34.620550+0000 mgr.y (mgr.34107) 7365 : cluster [DBG] pgmap v8406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:35 smithi121 bash[36902]: audit 2022-04-22T21:01:35.224072+0000 mon.a (mon.0) 10855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:36.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:35 smithi115 bash[46934]: cluster 2022-04-22T21:01:34.620550+0000 mgr.y (mgr.34107) 7365 : cluster [DBG] pgmap v8406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:36.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:35 smithi115 bash[46934]: audit 2022-04-22T21:01:35.224072+0000 mon.a (mon.0) 10855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:36.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:35 smithi115 bash[45495]: cluster 2022-04-22T21:01:34.620550+0000 mgr.y (mgr.34107) 7365 : cluster [DBG] pgmap v8406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:36.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:35 smithi115 bash[45495]: audit 2022-04-22T21:01:35.224072+0000 mon.a (mon.0) 10855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:38.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:37 smithi115 bash[46934]: cluster 2022-04-22T21:01:36.621545+0000 mgr.y (mgr.34107) 7366 : cluster [DBG] pgmap v8407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:38.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:37 smithi115 bash[45495]: cluster 2022-04-22T21:01:36.621545+0000 mgr.y (mgr.34107) 7366 : cluster [DBG] pgmap v8407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:37 smithi121 bash[36902]: cluster 2022-04-22T21:01:36.621545+0000 mgr.y (mgr.34107) 7366 : cluster [DBG] pgmap v8407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:39 smithi115 bash[46934]: cluster 2022-04-22T21:01:38.622674+0000 mgr.y (mgr.34107) 7367 : cluster [DBG] pgmap v8408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:39 smithi115 bash[46934]: audit 2022-04-22T21:01:38.907770+0000 mon.a (mon.0) 10856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:40.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:39 smithi115 bash[45495]: cluster 2022-04-22T21:01:38.622674+0000 mgr.y (mgr.34107) 7367 : cluster [DBG] pgmap v8408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:40.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:39 smithi115 bash[45495]: audit 2022-04-22T21:01:38.907770+0000 mon.a (mon.0) 10856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:39 smithi121 bash[36902]: cluster 2022-04-22T21:01:38.622674+0000 mgr.y (mgr.34107) 7367 : cluster [DBG] pgmap v8408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:39 smithi121 bash[36902]: audit 2022-04-22T21:01:38.907770+0000 mon.a (mon.0) 10856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:40 smithi115 bash[45495]: audit 2022-04-22T21:01:40.227092+0000 mon.a (mon.0) 10857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:40 smithi115 bash[46934]: audit 2022-04-22T21:01:40.227092+0000 mon.a (mon.0) 10857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:40 smithi121 bash[36902]: audit 2022-04-22T21:01:40.227092+0000 mon.a (mon.0) 10857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:42.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:41 smithi115 bash[45495]: cluster 2022-04-22T21:01:40.623624+0000 mgr.y (mgr.34107) 7368 : cluster [DBG] pgmap v8409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:42.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:41 smithi115 bash[46934]: cluster 2022-04-22T21:01:40.623624+0000 mgr.y (mgr.34107) 7368 : cluster [DBG] pgmap v8409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:41 smithi121 bash[36902]: cluster 2022-04-22T21:01:40.623624+0000 mgr.y (mgr.34107) 7368 : cluster [DBG] pgmap v8409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:42.450Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:44.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:43 smithi115 bash[45495]: cluster 2022-04-22T21:01:42.624689+0000 mgr.y (mgr.34107) 7369 : cluster [DBG] pgmap v8410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:44.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:43 smithi115 bash[46934]: cluster 2022-04-22T21:01:42.624689+0000 mgr.y (mgr.34107) 7369 : cluster [DBG] pgmap v8410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:43 smithi121 bash[36902]: cluster 2022-04-22T21:01:42.624689+0000 mgr.y (mgr.34107) 7369 : cluster [DBG] pgmap v8410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:44.326Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:44 smithi121 bash[36902]: audit 2022-04-22T21:01:43.911801+0000 mon.a (mon.0) 10858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:44 smithi115 bash[45495]: audit 2022-04-22T21:01:43.911801+0000 mon.a (mon.0) 10858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:44 smithi115 bash[46934]: audit 2022-04-22T21:01:43.911801+0000 mon.a (mon.0) 10858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:45 smithi121 bash[36902]: cluster 2022-04-22T21:01:44.625794+0000 mgr.y (mgr.34107) 7370 : cluster [DBG] pgmap v8411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:45 smithi121 bash[36902]: audit 2022-04-22T21:01:45.230379+0000 mon.a (mon.0) 10859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:45 smithi115 bash[45495]: cluster 2022-04-22T21:01:44.625794+0000 mgr.y (mgr.34107) 7370 : cluster [DBG] pgmap v8411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:45 smithi115 bash[45495]: audit 2022-04-22T21:01:45.230379+0000 mon.a (mon.0) 10859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:45 smithi115 bash[46934]: cluster 2022-04-22T21:01:44.625794+0000 mgr.y (mgr.34107) 7370 : cluster [DBG] pgmap v8411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:45 smithi115 bash[46934]: audit 2022-04-22T21:01:45.230379+0000 mon.a (mon.0) 10859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:47 smithi121 bash[36902]: cluster 2022-04-22T21:01:46.626808+0000 mgr.y (mgr.34107) 7371 : cluster [DBG] pgmap v8412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:47 smithi115 bash[45495]: cluster 2022-04-22T21:01:46.626808+0000 mgr.y (mgr.34107) 7371 : cluster [DBG] pgmap v8412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:48.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:47 smithi115 bash[46934]: cluster 2022-04-22T21:01:46.626808+0000 mgr.y (mgr.34107) 7371 : cluster [DBG] pgmap v8412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:49 smithi121 bash[36902]: cluster 2022-04-22T21:01:48.627926+0000 mgr.y (mgr.34107) 7372 : cluster [DBG] pgmap v8413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:50.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:49 smithi121 bash[36902]: audit 2022-04-22T21:01:48.915632+0000 mon.a (mon.0) 10860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:49 smithi115 bash[45495]: cluster 2022-04-22T21:01:48.627926+0000 mgr.y (mgr.34107) 7372 : cluster [DBG] pgmap v8413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:49 smithi115 bash[45495]: audit 2022-04-22T21:01:48.915632+0000 mon.a (mon.0) 10860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:49 smithi115 bash[46934]: cluster 2022-04-22T21:01:48.627926+0000 mgr.y (mgr.34107) 7372 : cluster [DBG] pgmap v8413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:50.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:49 smithi115 bash[46934]: audit 2022-04-22T21:01:48.915632+0000 mon.a (mon.0) 10860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:50 smithi121 bash[36902]: audit 2022-04-22T21:01:50.233656+0000 mon.a (mon.0) 10861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:50 smithi115 bash[45495]: audit 2022-04-22T21:01:50.233656+0000 mon.a (mon.0) 10861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:50 smithi115 bash[46934]: audit 2022-04-22T21:01:50.233656+0000 mon.a (mon.0) 10861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:51 smithi121 bash[36902]: cluster 2022-04-22T21:01:50.628992+0000 mgr.y (mgr.34107) 7373 : cluster [DBG] pgmap v8414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:51 smithi115 bash[45495]: cluster 2022-04-22T21:01:50.628992+0000 mgr.y (mgr.34107) 7373 : cluster [DBG] pgmap v8414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:51 smithi115 bash[46934]: cluster 2022-04-22T21:01:50.628992+0000 mgr.y (mgr.34107) 7373 : cluster [DBG] pgmap v8414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:52.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:53 smithi121 bash[36902]: cluster 2022-04-22T21:01:52.630135+0000 mgr.y (mgr.34107) 7374 : cluster [DBG] pgmap v8415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:53 smithi115 bash[45495]: cluster 2022-04-22T21:01:52.630135+0000 mgr.y (mgr.34107) 7374 : cluster [DBG] pgmap v8415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:54.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:53 smithi115 bash[46934]: cluster 2022-04-22T21:01:52.630135+0000 mgr.y (mgr.34107) 7374 : cluster [DBG] pgmap v8415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:01:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:01:54.326Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:01:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:54 smithi121 bash[36902]: audit 2022-04-22T21:01:53.919314+0000 mon.a (mon.0) 10862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:54 smithi115 bash[45495]: audit 2022-04-22T21:01:53.919314+0000 mon.a (mon.0) 10862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:54 smithi115 bash[46934]: audit 2022-04-22T21:01:53.919314+0000 mon.a (mon.0) 10862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:01:56.241 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:01:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:55 smithi121 bash[36902]: cluster 2022-04-22T21:01:54.631216+0000 mgr.y (mgr.34107) 7375 : cluster [DBG] pgmap v8416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:55 smithi121 bash[36902]: audit 2022-04-22T21:01:55.236743+0000 mon.a (mon.0) 10863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:55 smithi115 bash[46934]: cluster 2022-04-22T21:01:54.631216+0000 mgr.y (mgr.34107) 7375 : cluster [DBG] pgmap v8416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:55 smithi115 bash[46934]: audit 2022-04-22T21:01:55.236743+0000 mon.a (mon.0) 10863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:55 smithi115 bash[45495]: cluster 2022-04-22T21:01:54.631216+0000 mgr.y (mgr.34107) 7375 : cluster [DBG] pgmap v8416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:55 smithi115 bash[45495]: audit 2022-04-22T21:01:55.236743+0000 mon.a (mon.0) 10863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:01:56.594 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:01:56.595 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T21:01:56.596 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:01:56.597 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:01:56.598 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:01:57.006 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:01:57.006 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:57.007 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:01:57.008 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:01:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:56 smithi121 bash[36902]: audit 2022-04-22T21:01:56.221555+0000 mgr.y (mgr.34107) 7376 : audit [DBG] from='client.43038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:57.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:56 smithi121 bash[36902]: audit 2022-04-22T21:01:56.569423+0000 mgr.y (mgr.34107) 7377 : audit [DBG] from='client.52975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:57.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:56 smithi115 bash[45495]: audit 2022-04-22T21:01:56.221555+0000 mgr.y (mgr.34107) 7376 : audit [DBG] from='client.43038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:57.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:56 smithi115 bash[45495]: audit 2022-04-22T21:01:56.569423+0000 mgr.y (mgr.34107) 7377 : audit [DBG] from='client.52975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:57.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:56 smithi115 bash[46934]: audit 2022-04-22T21:01:56.221555+0000 mgr.y (mgr.34107) 7376 : audit [DBG] from='client.43038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:57.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:56 smithi115 bash[46934]: audit 2022-04-22T21:01:56.569423+0000 mgr.y (mgr.34107) 7377 : audit [DBG] from='client.52975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:01:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:57 smithi121 bash[36902]: cluster 2022-04-22T21:01:56.632282+0000 mgr.y (mgr.34107) 7378 : cluster [DBG] pgmap v8417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:57 smithi121 bash[36902]: audit 2022-04-22T21:01:57.001372+0000 mon.a (mon.0) 10864 : audit [DBG] from='client.? 172.21.15.115:0/3143687876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:01:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:57 smithi115 bash[46934]: cluster 2022-04-22T21:01:56.632282+0000 mgr.y (mgr.34107) 7378 : cluster [DBG] pgmap v8417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:57 smithi115 bash[46934]: audit 2022-04-22T21:01:57.001372+0000 mon.a (mon.0) 10864 : audit [DBG] from='client.? 172.21.15.115:0/3143687876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:01:58.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:57 smithi115 bash[45495]: cluster 2022-04-22T21:01:56.632282+0000 mgr.y (mgr.34107) 7378 : cluster [DBG] pgmap v8417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:01:58.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:57 smithi115 bash[45495]: audit 2022-04-22T21:01:57.001372+0000 mon.a (mon.0) 10864 : audit [DBG] from='client.? 172.21.15.115:0/3143687876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:02:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:59 smithi121 bash[36902]: cluster 2022-04-22T21:01:58.633387+0000 mgr.y (mgr.34107) 7379 : cluster [DBG] pgmap v8418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:01:59 smithi121 bash[36902]: audit 2022-04-22T21:01:58.922917+0000 mon.a (mon.0) 10865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:59 smithi115 bash[45495]: cluster 2022-04-22T21:01:58.633387+0000 mgr.y (mgr.34107) 7379 : cluster [DBG] pgmap v8418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:01:59 smithi115 bash[45495]: audit 2022-04-22T21:01:58.922917+0000 mon.a (mon.0) 10865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:59 smithi115 bash[46934]: cluster 2022-04-22T21:01:58.633387+0000 mgr.y (mgr.34107) 7379 : cluster [DBG] pgmap v8418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:01:59 smithi115 bash[46934]: audit 2022-04-22T21:01:58.922917+0000 mon.a (mon.0) 10865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:00 smithi121 bash[36902]: audit 2022-04-22T21:02:00.239877+0000 mon.a (mon.0) 10866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:01.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:00 smithi115 bash[45495]: audit 2022-04-22T21:02:00.239877+0000 mon.a (mon.0) 10866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:00 smithi115 bash[46934]: audit 2022-04-22T21:02:00.239877+0000 mon.a (mon.0) 10866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:01 smithi121 bash[36902]: cluster 2022-04-22T21:02:00.634441+0000 mgr.y (mgr.34107) 7380 : cluster [DBG] pgmap v8419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:02.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:01 smithi115 bash[45495]: cluster 2022-04-22T21:02:00.634441+0000 mgr.y (mgr.34107) 7380 : cluster [DBG] pgmap v8419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:01 smithi115 bash[46934]: cluster 2022-04-22T21:02:00.634441+0000 mgr.y (mgr.34107) 7380 : cluster [DBG] pgmap v8419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:02.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:03 smithi121 bash[36902]: cluster 2022-04-22T21:02:02.635560+0000 mgr.y (mgr.34107) 7381 : cluster [DBG] pgmap v8420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:03 smithi115 bash[45495]: cluster 2022-04-22T21:02:02.635560+0000 mgr.y (mgr.34107) 7381 : cluster [DBG] pgmap v8420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:04.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:03 smithi115 bash[46934]: cluster 2022-04-22T21:02:02.635560+0000 mgr.y (mgr.34107) 7381 : cluster [DBG] pgmap v8420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:04.327Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:04 smithi121 bash[36902]: audit 2022-04-22T21:02:03.926786+0000 mon.a (mon.0) 10867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:04 smithi115 bash[46934]: audit 2022-04-22T21:02:03.926786+0000 mon.a (mon.0) 10867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:05.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:04 smithi115 bash[45495]: audit 2022-04-22T21:02:03.926786+0000 mon.a (mon.0) 10867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:05 smithi121 bash[36902]: cluster 2022-04-22T21:02:04.636605+0000 mgr.y (mgr.34107) 7382 : cluster [DBG] pgmap v8421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:05 smithi121 bash[36902]: audit 2022-04-22T21:02:05.242739+0000 mon.a (mon.0) 10868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:05 smithi115 bash[45495]: cluster 2022-04-22T21:02:04.636605+0000 mgr.y (mgr.34107) 7382 : cluster [DBG] pgmap v8421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:05 smithi115 bash[45495]: audit 2022-04-22T21:02:05.242739+0000 mon.a (mon.0) 10868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:05 smithi115 bash[46934]: cluster 2022-04-22T21:02:04.636605+0000 mgr.y (mgr.34107) 7382 : cluster [DBG] pgmap v8421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:05 smithi115 bash[46934]: audit 2022-04-22T21:02:05.242739+0000 mon.a (mon.0) 10868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:07 smithi121 bash[36902]: cluster 2022-04-22T21:02:06.637674+0000 mgr.y (mgr.34107) 7383 : cluster [DBG] pgmap v8422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:08.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:07 smithi115 bash[45495]: cluster 2022-04-22T21:02:06.637674+0000 mgr.y (mgr.34107) 7383 : cluster [DBG] pgmap v8422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:07 smithi115 bash[46934]: cluster 2022-04-22T21:02:06.637674+0000 mgr.y (mgr.34107) 7383 : cluster [DBG] pgmap v8422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:09 smithi121 bash[36902]: cluster 2022-04-22T21:02:08.638786+0000 mgr.y (mgr.34107) 7384 : cluster [DBG] pgmap v8423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:09 smithi121 bash[36902]: audit 2022-04-22T21:02:08.930179+0000 mon.a (mon.0) 10869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:09 smithi115 bash[45495]: cluster 2022-04-22T21:02:08.638786+0000 mgr.y (mgr.34107) 7384 : cluster [DBG] pgmap v8423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:09 smithi115 bash[45495]: audit 2022-04-22T21:02:08.930179+0000 mon.a (mon.0) 10869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:10.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:09 smithi115 bash[46934]: cluster 2022-04-22T21:02:08.638786+0000 mgr.y (mgr.34107) 7384 : cluster [DBG] pgmap v8423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:10.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:09 smithi115 bash[46934]: audit 2022-04-22T21:02:08.930179+0000 mon.a (mon.0) 10869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:10 smithi121 bash[36902]: audit 2022-04-22T21:02:10.246009+0000 mon.a (mon.0) 10870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:10 smithi115 bash[45495]: audit 2022-04-22T21:02:10.246009+0000 mon.a (mon.0) 10870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:10 smithi115 bash[46934]: audit 2022-04-22T21:02:10.246009+0000 mon.a (mon.0) 10870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:11 smithi121 bash[36902]: cluster 2022-04-22T21:02:10.639857+0000 mgr.y (mgr.34107) 7385 : cluster [DBG] pgmap v8424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:12.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:11 smithi115 bash[45495]: cluster 2022-04-22T21:02:10.639857+0000 mgr.y (mgr.34107) 7385 : cluster [DBG] pgmap v8424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:11 smithi115 bash[46934]: cluster 2022-04-22T21:02:10.639857+0000 mgr.y (mgr.34107) 7385 : cluster [DBG] pgmap v8424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:12.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:12.452Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:13 smithi121 bash[36902]: cluster 2022-04-22T21:02:12.641020+0000 mgr.y (mgr.34107) 7386 : cluster [DBG] pgmap v8425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:13 smithi115 bash[45495]: cluster 2022-04-22T21:02:12.641020+0000 mgr.y (mgr.34107) 7386 : cluster [DBG] pgmap v8425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:13 smithi115 bash[46934]: cluster 2022-04-22T21:02:12.641020+0000 mgr.y (mgr.34107) 7386 : cluster [DBG] pgmap v8425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:14.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:14 smithi121 bash[36902]: audit 2022-04-22T21:02:13.933738+0000 mon.a (mon.0) 10871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:15.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:14 smithi115 bash[45495]: audit 2022-04-22T21:02:13.933738+0000 mon.a (mon.0) 10871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:14 smithi115 bash[46934]: audit 2022-04-22T21:02:13.933738+0000 mon.a (mon.0) 10871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:15 smithi121 bash[36902]: cluster 2022-04-22T21:02:14.642068+0000 mgr.y (mgr.34107) 7387 : cluster [DBG] pgmap v8426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:15 smithi121 bash[36902]: audit 2022-04-22T21:02:15.249336+0000 mon.a (mon.0) 10872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:15 smithi121 bash[36902]: audit 2022-04-22T21:02:15.759086+0000 mon.a (mon.0) 10873 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:02:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:15 smithi121 bash[36902]: audit 2022-04-22T21:02:15.759764+0000 mon.a (mon.0) 10874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:02:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[45495]: cluster 2022-04-22T21:02:14.642068+0000 mgr.y (mgr.34107) 7387 : cluster [DBG] pgmap v8426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[45495]: audit 2022-04-22T21:02:15.249336+0000 mon.a (mon.0) 10872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[45495]: audit 2022-04-22T21:02:15.759086+0000 mon.a (mon.0) 10873 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:02:16.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[45495]: audit 2022-04-22T21:02:15.759764+0000 mon.a (mon.0) 10874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:02:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[46934]: cluster 2022-04-22T21:02:14.642068+0000 mgr.y (mgr.34107) 7387 : cluster [DBG] pgmap v8426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[46934]: audit 2022-04-22T21:02:15.249336+0000 mon.a (mon.0) 10872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[46934]: audit 2022-04-22T21:02:15.759086+0000 mon.a (mon.0) 10873 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:02:16.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:15 smithi115 bash[46934]: audit 2022-04-22T21:02:15.759764+0000 mon.a (mon.0) 10874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:02:18.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:17 smithi121 bash[36902]: cluster 2022-04-22T21:02:16.643132+0000 mgr.y (mgr.34107) 7388 : cluster [DBG] pgmap v8427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:17 smithi115 bash[45495]: cluster 2022-04-22T21:02:16.643132+0000 mgr.y (mgr.34107) 7388 : cluster [DBG] pgmap v8427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:17 smithi115 bash[46934]: cluster 2022-04-22T21:02:16.643132+0000 mgr.y (mgr.34107) 7388 : cluster [DBG] pgmap v8427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:19 smithi121 bash[36902]: cluster 2022-04-22T21:02:18.644242+0000 mgr.y (mgr.34107) 7389 : cluster [DBG] pgmap v8428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:19 smithi121 bash[36902]: audit 2022-04-22T21:02:18.937398+0000 mon.a (mon.0) 10875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:19 smithi115 bash[45495]: cluster 2022-04-22T21:02:18.644242+0000 mgr.y (mgr.34107) 7389 : cluster [DBG] pgmap v8428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:19 smithi115 bash[45495]: audit 2022-04-22T21:02:18.937398+0000 mon.a (mon.0) 10875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:20.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:19 smithi115 bash[46934]: cluster 2022-04-22T21:02:18.644242+0000 mgr.y (mgr.34107) 7389 : cluster [DBG] pgmap v8428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:20.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:19 smithi115 bash[46934]: audit 2022-04-22T21:02:18.937398+0000 mon.a (mon.0) 10875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:20 smithi121 bash[36902]: audit 2022-04-22T21:02:20.252240+0000 mon.a (mon.0) 10876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:21.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:20 smithi115 bash[45495]: audit 2022-04-22T21:02:20.252240+0000 mon.a (mon.0) 10876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:21.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:20 smithi115 bash[46934]: audit 2022-04-22T21:02:20.252240+0000 mon.a (mon.0) 10876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:21 smithi121 bash[36902]: cluster 2022-04-22T21:02:20.645265+0000 mgr.y (mgr.34107) 7390 : cluster [DBG] pgmap v8429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:22.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:21 smithi115 bash[45495]: cluster 2022-04-22T21:02:20.645265+0000 mgr.y (mgr.34107) 7390 : cluster [DBG] pgmap v8429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:22.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:21 smithi115 bash[46934]: cluster 2022-04-22T21:02:20.645265+0000 mgr.y (mgr.34107) 7390 : cluster [DBG] pgmap v8429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:22.453Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:23 smithi121 bash[36902]: cluster 2022-04-22T21:02:22.646324+0000 mgr.y (mgr.34107) 7391 : cluster [DBG] pgmap v8430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:24.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:23 smithi115 bash[45495]: cluster 2022-04-22T21:02:22.646324+0000 mgr.y (mgr.34107) 7391 : cluster [DBG] pgmap v8430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:23 smithi115 bash[46934]: cluster 2022-04-22T21:02:22.646324+0000 mgr.y (mgr.34107) 7391 : cluster [DBG] pgmap v8430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:24.329Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:24 smithi121 bash[36902]: audit 2022-04-22T21:02:23.940990+0000 mon.a (mon.0) 10877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:25.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:24 smithi115 bash[45495]: audit 2022-04-22T21:02:23.940990+0000 mon.a (mon.0) 10877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:25.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:24 smithi115 bash[46934]: audit 2022-04-22T21:02:23.940990+0000 mon.a (mon.0) 10877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:25 smithi121 bash[36902]: cluster 2022-04-22T21:02:24.647366+0000 mgr.y (mgr.34107) 7392 : cluster [DBG] pgmap v8431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:25 smithi121 bash[36902]: audit 2022-04-22T21:02:25.255439+0000 mon.a (mon.0) 10878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:25 smithi115 bash[45495]: cluster 2022-04-22T21:02:24.647366+0000 mgr.y (mgr.34107) 7392 : cluster [DBG] pgmap v8431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:25 smithi115 bash[45495]: audit 2022-04-22T21:02:25.255439+0000 mon.a (mon.0) 10878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:25 smithi115 bash[46934]: cluster 2022-04-22T21:02:24.647366+0000 mgr.y (mgr.34107) 7392 : cluster [DBG] pgmap v8431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:25 smithi115 bash[46934]: audit 2022-04-22T21:02:25.255439+0000 mon.a (mon.0) 10878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:27.376 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:02:27.720 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10m ago 3h 227M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10m ago 3h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:02:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:02:27.722 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10m ago 3h 579M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:02:27.722 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:02:27.722 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10m ago 3h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:02:27.722 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10m ago 3h 641M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:02:27.722 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10m ago 3h 66.4M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:02:28.122 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:02:28.600 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:02:28.600 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:02:28.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:28.600 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:28.601 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:02:28.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:02:28.602 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:02:28.602 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:02:28.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:27 smithi121 bash[36902]: cluster 2022-04-22T21:02:26.648459+0000 mgr.y (mgr.34107) 7393 : cluster [DBG] pgmap v8432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:28.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:27 smithi121 bash[36902]: audit 2022-04-22T21:02:27.356753+0000 mgr.y (mgr.34107) 7394 : audit [DBG] from='client.43056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:27 smithi115 bash[45495]: cluster 2022-04-22T21:02:26.648459+0000 mgr.y (mgr.34107) 7393 : cluster [DBG] pgmap v8432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:28.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:27 smithi115 bash[45495]: audit 2022-04-22T21:02:27.356753+0000 mgr.y (mgr.34107) 7394 : audit [DBG] from='client.43056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:28.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:27 smithi115 bash[46934]: cluster 2022-04-22T21:02:26.648459+0000 mgr.y (mgr.34107) 7393 : cluster [DBG] pgmap v8432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:28.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:27 smithi115 bash[46934]: audit 2022-04-22T21:02:27.356753+0000 mgr.y (mgr.34107) 7394 : audit [DBG] from='client.43056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:28 smithi121 bash[36902]: audit 2022-04-22T21:02:27.697006+0000 mgr.y (mgr.34107) 7395 : audit [DBG] from='client.52993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:28 smithi121 bash[36902]: audit 2022-04-22T21:02:28.115653+0000 mon.a (mon.0) 10879 : audit [DBG] from='client.? 172.21.15.115:0/3719445087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:02:29.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:28 smithi115 bash[45495]: audit 2022-04-22T21:02:27.697006+0000 mgr.y (mgr.34107) 7395 : audit [DBG] from='client.52993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:29.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:28 smithi115 bash[45495]: audit 2022-04-22T21:02:28.115653+0000 mon.a (mon.0) 10879 : audit [DBG] from='client.? 172.21.15.115:0/3719445087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:02:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:28 smithi115 bash[46934]: audit 2022-04-22T21:02:27.697006+0000 mgr.y (mgr.34107) 7395 : audit [DBG] from='client.52993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:28 smithi115 bash[46934]: audit 2022-04-22T21:02:28.115653+0000 mon.a (mon.0) 10879 : audit [DBG] from='client.? 172.21.15.115:0/3719445087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:02:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:29 smithi121 bash[36902]: cluster 2022-04-22T21:02:28.649608+0000 mgr.y (mgr.34107) 7396 : cluster [DBG] pgmap v8433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:29 smithi121 bash[36902]: audit 2022-04-22T21:02:28.950953+0000 mon.a (mon.0) 10880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:29 smithi115 bash[45495]: cluster 2022-04-22T21:02:28.649608+0000 mgr.y (mgr.34107) 7396 : cluster [DBG] pgmap v8433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:29 smithi115 bash[45495]: audit 2022-04-22T21:02:28.950953+0000 mon.a (mon.0) 10880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:29 smithi115 bash[46934]: cluster 2022-04-22T21:02:28.649608+0000 mgr.y (mgr.34107) 7396 : cluster [DBG] pgmap v8433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:29 smithi115 bash[46934]: audit 2022-04-22T21:02:28.950953+0000 mon.a (mon.0) 10880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:30 smithi121 bash[36902]: audit 2022-04-22T21:02:30.258553+0000 mon.a (mon.0) 10881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:30 smithi115 bash[45495]: audit 2022-04-22T21:02:30.258553+0000 mon.a (mon.0) 10881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:30 smithi115 bash[46934]: audit 2022-04-22T21:02:30.258553+0000 mon.a (mon.0) 10881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:31 smithi121 bash[36902]: cluster 2022-04-22T21:02:30.650710+0000 mgr.y (mgr.34107) 7397 : cluster [DBG] pgmap v8434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:32.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:31 smithi115 bash[45495]: cluster 2022-04-22T21:02:30.650710+0000 mgr.y (mgr.34107) 7397 : cluster [DBG] pgmap v8434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:31 smithi115 bash[46934]: cluster 2022-04-22T21:02:30.650710+0000 mgr.y (mgr.34107) 7397 : cluster [DBG] pgmap v8434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:32.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:32.453Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:32 smithi121 bash[36902]: audit 2022-04-22T21:02:32.518527+0000 mon.a (mon.0) 10882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:02:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:32 smithi121 bash[36902]: audit 2022-04-22T21:02:32.522316+0000 mon.a (mon.0) 10883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:02:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:32 smithi121 bash[36902]: audit 2022-04-22T21:02:32.524127+0000 mon.a (mon.0) 10884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:02:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[45495]: audit 2022-04-22T21:02:32.518527+0000 mon.a (mon.0) 10882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:02:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[45495]: audit 2022-04-22T21:02:32.522316+0000 mon.a (mon.0) 10883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:02:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[45495]: audit 2022-04-22T21:02:32.524127+0000 mon.a (mon.0) 10884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:02:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[46934]: audit 2022-04-22T21:02:32.518527+0000 mon.a (mon.0) 10882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:02:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[46934]: audit 2022-04-22T21:02:32.522316+0000 mon.a (mon.0) 10883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:02:33.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:32 smithi115 bash[46934]: audit 2022-04-22T21:02:32.524127+0000 mon.a (mon.0) 10884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:02:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:33 smithi121 bash[36902]: cluster 2022-04-22T21:02:32.651784+0000 mgr.y (mgr.34107) 7398 : cluster [DBG] pgmap v8435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:33 smithi121 bash[36902]: audit 2022-04-22T21:02:33.949716+0000 mon.a (mon.0) 10885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:33 smithi115 bash[45495]: cluster 2022-04-22T21:02:32.651784+0000 mgr.y (mgr.34107) 7398 : cluster [DBG] pgmap v8435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:33 smithi115 bash[45495]: audit 2022-04-22T21:02:33.949716+0000 mon.a (mon.0) 10885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:33 smithi115 bash[46934]: cluster 2022-04-22T21:02:32.651784+0000 mgr.y (mgr.34107) 7398 : cluster [DBG] pgmap v8435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:33 smithi115 bash[46934]: audit 2022-04-22T21:02:33.949716+0000 mon.a (mon.0) 10885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:34.330Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:35 smithi121 bash[36902]: cluster 2022-04-22T21:02:34.652879+0000 mgr.y (mgr.34107) 7399 : cluster [DBG] pgmap v8436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:35 smithi121 bash[36902]: audit 2022-04-22T21:02:35.261669+0000 mon.a (mon.0) 10886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:35 smithi115 bash[45495]: cluster 2022-04-22T21:02:34.652879+0000 mgr.y (mgr.34107) 7399 : cluster [DBG] pgmap v8436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:35 smithi115 bash[45495]: audit 2022-04-22T21:02:35.261669+0000 mon.a (mon.0) 10886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:35 smithi115 bash[46934]: cluster 2022-04-22T21:02:34.652879+0000 mgr.y (mgr.34107) 7399 : cluster [DBG] pgmap v8436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:35 smithi115 bash[46934]: audit 2022-04-22T21:02:35.261669+0000 mon.a (mon.0) 10886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:37 smithi121 bash[36902]: cluster 2022-04-22T21:02:36.653897+0000 mgr.y (mgr.34107) 7400 : cluster [DBG] pgmap v8437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:37 smithi121 bash[36902]: audit 2022-04-22T21:02:36.957195+0000 mon.a (mon.0) 10887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:37 smithi121 bash[36902]: audit 2022-04-22T21:02:37.309058+0000 mon.a (mon.0) 10888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[45495]: cluster 2022-04-22T21:02:36.653897+0000 mgr.y (mgr.34107) 7400 : cluster [DBG] pgmap v8437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[45495]: audit 2022-04-22T21:02:36.957195+0000 mon.a (mon.0) 10887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[45495]: audit 2022-04-22T21:02:37.309058+0000 mon.a (mon.0) 10888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[46934]: cluster 2022-04-22T21:02:36.653897+0000 mgr.y (mgr.34107) 7400 : cluster [DBG] pgmap v8437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[46934]: audit 2022-04-22T21:02:36.957195+0000 mon.a (mon.0) 10887 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:37 smithi115 bash[46934]: audit 2022-04-22T21:02:37.309058+0000 mon.a (mon.0) 10888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:02:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:38 smithi121 bash[36902]: audit 2022-04-22T21:02:38.953728+0000 mon.a (mon.0) 10889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:38 smithi115 bash[45495]: audit 2022-04-22T21:02:38.953728+0000 mon.a (mon.0) 10889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:38 smithi115 bash[46934]: audit 2022-04-22T21:02:38.953728+0000 mon.a (mon.0) 10889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:39 smithi121 bash[36902]: cluster 2022-04-22T21:02:38.654970+0000 mgr.y (mgr.34107) 7401 : cluster [DBG] pgmap v8438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:39 smithi115 bash[45495]: cluster 2022-04-22T21:02:38.654970+0000 mgr.y (mgr.34107) 7401 : cluster [DBG] pgmap v8438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:39 smithi115 bash[46934]: cluster 2022-04-22T21:02:38.654970+0000 mgr.y (mgr.34107) 7401 : cluster [DBG] pgmap v8438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:40 smithi121 bash[36902]: audit 2022-04-22T21:02:40.264843+0000 mon.a (mon.0) 10890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:40 smithi115 bash[45495]: audit 2022-04-22T21:02:40.264843+0000 mon.a (mon.0) 10890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:40 smithi115 bash[46934]: audit 2022-04-22T21:02:40.264843+0000 mon.a (mon.0) 10890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:41 smithi121 bash[36902]: cluster 2022-04-22T21:02:40.656376+0000 mgr.y (mgr.34107) 7402 : cluster [DBG] pgmap v8439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:42.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:41 smithi115 bash[45495]: cluster 2022-04-22T21:02:40.656376+0000 mgr.y (mgr.34107) 7402 : cluster [DBG] pgmap v8439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:41 smithi115 bash[46934]: cluster 2022-04-22T21:02:40.656376+0000 mgr.y (mgr.34107) 7402 : cluster [DBG] pgmap v8439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:42.454Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:43 smithi121 bash[36902]: cluster 2022-04-22T21:02:42.657453+0000 mgr.y (mgr.34107) 7403 : cluster [DBG] pgmap v8440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:43 smithi121 bash[36902]: audit 2022-04-22T21:02:43.957499+0000 mon.a (mon.0) 10891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:43 smithi115 bash[45495]: cluster 2022-04-22T21:02:42.657453+0000 mgr.y (mgr.34107) 7403 : cluster [DBG] pgmap v8440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:43 smithi115 bash[45495]: audit 2022-04-22T21:02:43.957499+0000 mon.a (mon.0) 10891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:43 smithi115 bash[46934]: cluster 2022-04-22T21:02:42.657453+0000 mgr.y (mgr.34107) 7403 : cluster [DBG] pgmap v8440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:43 smithi115 bash[46934]: audit 2022-04-22T21:02:43.957499+0000 mon.a (mon.0) 10891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:44.331Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:45 smithi121 bash[36902]: cluster 2022-04-22T21:02:44.658495+0000 mgr.y (mgr.34107) 7404 : cluster [DBG] pgmap v8441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:45 smithi121 bash[36902]: audit 2022-04-22T21:02:45.267884+0000 mon.a (mon.0) 10892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:45 smithi115 bash[45495]: cluster 2022-04-22T21:02:44.658495+0000 mgr.y (mgr.34107) 7404 : cluster [DBG] pgmap v8441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:45 smithi115 bash[45495]: audit 2022-04-22T21:02:45.267884+0000 mon.a (mon.0) 10892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:46.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:45 smithi115 bash[46934]: cluster 2022-04-22T21:02:44.658495+0000 mgr.y (mgr.34107) 7404 : cluster [DBG] pgmap v8441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:46.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:45 smithi115 bash[46934]: audit 2022-04-22T21:02:45.267884+0000 mon.a (mon.0) 10892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:47 smithi121 bash[36902]: cluster 2022-04-22T21:02:46.659602+0000 mgr.y (mgr.34107) 7405 : cluster [DBG] pgmap v8442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:48.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:47 smithi115 bash[45495]: cluster 2022-04-22T21:02:46.659602+0000 mgr.y (mgr.34107) 7405 : cluster [DBG] pgmap v8442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:48.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:47 smithi115 bash[46934]: cluster 2022-04-22T21:02:46.659602+0000 mgr.y (mgr.34107) 7405 : cluster [DBG] pgmap v8442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:49 smithi121 bash[36902]: audit 2022-04-22T21:02:48.961498+0000 mon.a (mon.0) 10893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:49.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:49 smithi115 bash[45495]: audit 2022-04-22T21:02:48.961498+0000 mon.a (mon.0) 10893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:49.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:49 smithi115 bash[46934]: audit 2022-04-22T21:02:48.961498+0000 mon.a (mon.0) 10893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:50 smithi121 bash[36902]: cluster 2022-04-22T21:02:48.660929+0000 mgr.y (mgr.34107) 7406 : cluster [DBG] pgmap v8443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:50 smithi115 bash[45495]: cluster 2022-04-22T21:02:48.660929+0000 mgr.y (mgr.34107) 7406 : cluster [DBG] pgmap v8443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:50.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:50 smithi115 bash[46934]: cluster 2022-04-22T21:02:48.660929+0000 mgr.y (mgr.34107) 7406 : cluster [DBG] pgmap v8443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:51 smithi115 bash[45495]: audit 2022-04-22T21:02:50.271084+0000 mon.a (mon.0) 10894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:51 smithi115 bash[46934]: audit 2022-04-22T21:02:50.271084+0000 mon.a (mon.0) 10894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:51 smithi121 bash[36902]: audit 2022-04-22T21:02:50.271084+0000 mon.a (mon.0) 10894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:52.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:52 smithi115 bash[45495]: cluster 2022-04-22T21:02:50.662147+0000 mgr.y (mgr.34107) 7407 : cluster [DBG] pgmap v8444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:52.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:52 smithi115 bash[46934]: cluster 2022-04-22T21:02:50.662147+0000 mgr.y (mgr.34107) 7407 : cluster [DBG] pgmap v8444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:52 smithi121 bash[36902]: cluster 2022-04-22T21:02:50.662147+0000 mgr.y (mgr.34107) 7407 : cluster [DBG] pgmap v8444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:52.455Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:54 smithi115 bash[45495]: cluster 2022-04-22T21:02:52.663272+0000 mgr.y (mgr.34107) 7408 : cluster [DBG] pgmap v8445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:54 smithi115 bash[45495]: audit 2022-04-22T21:02:53.965413+0000 mon.a (mon.0) 10895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:54.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:54 smithi115 bash[46934]: cluster 2022-04-22T21:02:52.663272+0000 mgr.y (mgr.34107) 7408 : cluster [DBG] pgmap v8445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:54.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:54 smithi115 bash[46934]: audit 2022-04-22T21:02:53.965413+0000 mon.a (mon.0) 10895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:54 smithi121 bash[36902]: cluster 2022-04-22T21:02:52.663272+0000 mgr.y (mgr.34107) 7408 : cluster [DBG] pgmap v8445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:54 smithi121 bash[36902]: audit 2022-04-22T21:02:53.965413+0000 mon.a (mon.0) 10895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:02:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:02:54.330Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:02:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:56 smithi115 bash[45495]: cluster 2022-04-22T21:02:54.664390+0000 mgr.y (mgr.34107) 7409 : cluster [DBG] pgmap v8446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:56 smithi115 bash[45495]: audit 2022-04-22T21:02:55.274401+0000 mon.a (mon.0) 10896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:56 smithi115 bash[46934]: cluster 2022-04-22T21:02:54.664390+0000 mgr.y (mgr.34107) 7409 : cluster [DBG] pgmap v8446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:56 smithi115 bash[46934]: audit 2022-04-22T21:02:55.274401+0000 mon.a (mon.0) 10896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:56 smithi121 bash[36902]: cluster 2022-04-22T21:02:54.664390+0000 mgr.y (mgr.34107) 7409 : cluster [DBG] pgmap v8446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:56 smithi121 bash[36902]: audit 2022-04-22T21:02:55.274401+0000 mon.a (mon.0) 10896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:02:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:58 smithi115 bash[46934]: cluster 2022-04-22T21:02:56.665331+0000 mgr.y (mgr.34107) 7410 : cluster [DBG] pgmap v8447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:58.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:58 smithi115 bash[45495]: cluster 2022-04-22T21:02:56.665331+0000 mgr.y (mgr.34107) 7410 : cluster [DBG] pgmap v8447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:58.490 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:02:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:58 smithi121 bash[36902]: cluster 2022-04-22T21:02:56.665331+0000 mgr.y (mgr.34107) 7410 : cluster [DBG] pgmap v8447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:02:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:02:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 21s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 21s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 21s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 21s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:02:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 21s ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 21s ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 21s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 21s ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:02:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 21s ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:02:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 21s ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:02:59.241 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:02:59.306 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:02:59.307 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:02:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:02:59.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:59 smithi115 bash[45495]: audit 2022-04-22T21:02:58.469023+0000 mgr.y (mgr.34107) 7411 : audit [DBG] from='client.43074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:59.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:02:59 smithi115 bash[45495]: audit 2022-04-22T21:02:58.968702+0000 mon.a (mon.0) 10897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:59.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:59 smithi115 bash[46934]: audit 2022-04-22T21:02:58.469023+0000 mgr.y (mgr.34107) 7411 : audit [DBG] from='client.43074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:59.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:02:59 smithi115 bash[46934]: audit 2022-04-22T21:02:58.968702+0000 mon.a (mon.0) 10897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:02:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:59 smithi121 bash[36902]: audit 2022-04-22T21:02:58.469023+0000 mgr.y (mgr.34107) 7411 : audit [DBG] from='client.43074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:02:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:02:59 smithi121 bash[36902]: audit 2022-04-22T21:02:58.968702+0000 mon.a (mon.0) 10897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:00 smithi121 bash[36902]: cluster 2022-04-22T21:02:58.666303+0000 mgr.y (mgr.34107) 7412 : cluster [DBG] pgmap v8448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:00 smithi121 bash[36902]: audit 2022-04-22T21:02:58.816086+0000 mgr.y (mgr.34107) 7413 : audit [DBG] from='client.53011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:00 smithi121 bash[36902]: audit 2022-04-22T21:02:59.236331+0000 mon.c (mon.1) 273 : audit [DBG] from='client.? 172.21.15.115:0/3770599093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[45495]: cluster 2022-04-22T21:02:58.666303+0000 mgr.y (mgr.34107) 7412 : cluster [DBG] pgmap v8448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[45495]: audit 2022-04-22T21:02:58.816086+0000 mgr.y (mgr.34107) 7413 : audit [DBG] from='client.53011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[45495]: audit 2022-04-22T21:02:59.236331+0000 mon.c (mon.1) 273 : audit [DBG] from='client.? 172.21.15.115:0/3770599093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[46934]: cluster 2022-04-22T21:02:58.666303+0000 mgr.y (mgr.34107) 7412 : cluster [DBG] pgmap v8448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[46934]: audit 2022-04-22T21:02:58.816086+0000 mgr.y (mgr.34107) 7413 : audit [DBG] from='client.53011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:00 smithi115 bash[46934]: audit 2022-04-22T21:02:59.236331+0000 mon.c (mon.1) 273 : audit [DBG] from='client.? 172.21.15.115:0/3770599093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:01 smithi121 bash[36902]: audit 2022-04-22T21:03:00.277479+0000 mon.a (mon.0) 10898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:01 smithi115 bash[45495]: audit 2022-04-22T21:03:00.277479+0000 mon.a (mon.0) 10898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:01 smithi115 bash[46934]: audit 2022-04-22T21:03:00.277479+0000 mon.a (mon.0) 10898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:02.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:02 smithi115 bash[46934]: cluster 2022-04-22T21:03:00.667360+0000 mgr.y (mgr.34107) 7414 : cluster [DBG] pgmap v8449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:02.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:02 smithi115 bash[45495]: cluster 2022-04-22T21:03:00.667360+0000 mgr.y (mgr.34107) 7414 : cluster [DBG] pgmap v8449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:02 smithi121 bash[36902]: cluster 2022-04-22T21:03:00.667360+0000 mgr.y (mgr.34107) 7414 : cluster [DBG] pgmap v8449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:02.456Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:04.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:04 smithi115 bash[46934]: cluster 2022-04-22T21:03:02.668514+0000 mgr.y (mgr.34107) 7415 : cluster [DBG] pgmap v8450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:04.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:04 smithi115 bash[46934]: audit 2022-04-22T21:03:03.971732+0000 mon.a (mon.0) 10899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:04.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:04 smithi115 bash[45495]: cluster 2022-04-22T21:03:02.668514+0000 mgr.y (mgr.34107) 7415 : cluster [DBG] pgmap v8450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:04.336 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:04 smithi115 bash[45495]: audit 2022-04-22T21:03:03.971732+0000 mon.a (mon.0) 10899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:04 smithi121 bash[36902]: cluster 2022-04-22T21:03:02.668514+0000 mgr.y (mgr.34107) 7415 : cluster [DBG] pgmap v8450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:04 smithi121 bash[36902]: audit 2022-04-22T21:03:03.971732+0000 mon.a (mon.0) 10899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:04.331Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:06 smithi121 bash[36902]: cluster 2022-04-22T21:03:04.669281+0000 mgr.y (mgr.34107) 7416 : cluster [DBG] pgmap v8451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:06 smithi121 bash[36902]: audit 2022-04-22T21:03:05.280384+0000 mon.a (mon.0) 10900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:06 smithi115 bash[45495]: cluster 2022-04-22T21:03:04.669281+0000 mgr.y (mgr.34107) 7416 : cluster [DBG] pgmap v8451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:06 smithi115 bash[45495]: audit 2022-04-22T21:03:05.280384+0000 mon.a (mon.0) 10900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:06 smithi115 bash[46934]: cluster 2022-04-22T21:03:04.669281+0000 mgr.y (mgr.34107) 7416 : cluster [DBG] pgmap v8451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:06 smithi115 bash[46934]: audit 2022-04-22T21:03:05.280384+0000 mon.a (mon.0) 10900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:08 smithi121 bash[36902]: cluster 2022-04-22T21:03:06.670373+0000 mgr.y (mgr.34107) 7417 : cluster [DBG] pgmap v8452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:08.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:08 smithi115 bash[45495]: cluster 2022-04-22T21:03:06.670373+0000 mgr.y (mgr.34107) 7417 : cluster [DBG] pgmap v8452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:08.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:08 smithi115 bash[46934]: cluster 2022-04-22T21:03:06.670373+0000 mgr.y (mgr.34107) 7417 : cluster [DBG] pgmap v8452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:09 smithi121 bash[36902]: audit 2022-04-22T21:03:08.975376+0000 mon.a (mon.0) 10901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:09 smithi115 bash[45495]: audit 2022-04-22T21:03:08.975376+0000 mon.a (mon.0) 10901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:09 smithi115 bash[46934]: audit 2022-04-22T21:03:08.975376+0000 mon.a (mon.0) 10901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:10 smithi121 bash[36902]: cluster 2022-04-22T21:03:08.671551+0000 mgr.y (mgr.34107) 7418 : cluster [DBG] pgmap v8453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:10 smithi115 bash[45495]: cluster 2022-04-22T21:03:08.671551+0000 mgr.y (mgr.34107) 7418 : cluster [DBG] pgmap v8453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:10 smithi115 bash[46934]: cluster 2022-04-22T21:03:08.671551+0000 mgr.y (mgr.34107) 7418 : cluster [DBG] pgmap v8453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:11 smithi121 bash[36902]: audit 2022-04-22T21:03:10.283214+0000 mon.a (mon.0) 10902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:11 smithi115 bash[45495]: audit 2022-04-22T21:03:10.283214+0000 mon.a (mon.0) 10902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:11 smithi115 bash[46934]: audit 2022-04-22T21:03:10.283214+0000 mon.a (mon.0) 10902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:12.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:12 smithi115 bash[45495]: cluster 2022-04-22T21:03:10.672628+0000 mgr.y (mgr.34107) 7419 : cluster [DBG] pgmap v8454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:12.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:12 smithi115 bash[46934]: cluster 2022-04-22T21:03:10.672628+0000 mgr.y (mgr.34107) 7419 : cluster [DBG] pgmap v8454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:12 smithi121 bash[36902]: cluster 2022-04-22T21:03:10.672628+0000 mgr.y (mgr.34107) 7419 : cluster [DBG] pgmap v8454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:12.456Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:14 smithi121 bash[36902]: cluster 2022-04-22T21:03:12.673515+0000 mgr.y (mgr.34107) 7420 : cluster [DBG] pgmap v8455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:14 smithi121 bash[36902]: audit 2022-04-22T21:03:13.979135+0000 mon.a (mon.0) 10903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:14.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:14.332Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:14 smithi115 bash[45495]: cluster 2022-04-22T21:03:12.673515+0000 mgr.y (mgr.34107) 7420 : cluster [DBG] pgmap v8455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:14 smithi115 bash[45495]: audit 2022-04-22T21:03:13.979135+0000 mon.a (mon.0) 10903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:14 smithi115 bash[46934]: cluster 2022-04-22T21:03:12.673515+0000 mgr.y (mgr.34107) 7420 : cluster [DBG] pgmap v8455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:14 smithi115 bash[46934]: audit 2022-04-22T21:03:13.979135+0000 mon.a (mon.0) 10903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:16 smithi121 bash[36902]: cluster 2022-04-22T21:03:14.674571+0000 mgr.y (mgr.34107) 7421 : cluster [DBG] pgmap v8456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:16 smithi121 bash[36902]: audit 2022-04-22T21:03:15.286337+0000 mon.a (mon.0) 10904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:16 smithi121 bash[36902]: audit 2022-04-22T21:03:15.768910+0000 mon.a (mon.0) 10905 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:03:16.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:16 smithi121 bash[36902]: audit 2022-04-22T21:03:15.769656+0000 mon.a (mon.0) 10906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:03:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[45495]: cluster 2022-04-22T21:03:14.674571+0000 mgr.y (mgr.34107) 7421 : cluster [DBG] pgmap v8456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[45495]: audit 2022-04-22T21:03:15.286337+0000 mon.a (mon.0) 10904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[45495]: audit 2022-04-22T21:03:15.768910+0000 mon.a (mon.0) 10905 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:03:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[45495]: audit 2022-04-22T21:03:15.769656+0000 mon.a (mon.0) 10906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:03:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[46934]: cluster 2022-04-22T21:03:14.674571+0000 mgr.y (mgr.34107) 7421 : cluster [DBG] pgmap v8456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[46934]: audit 2022-04-22T21:03:15.286337+0000 mon.a (mon.0) 10904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[46934]: audit 2022-04-22T21:03:15.768910+0000 mon.a (mon.0) 10905 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:03:16.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:16 smithi115 bash[46934]: audit 2022-04-22T21:03:15.769656+0000 mon.a (mon.0) 10906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:03:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:18 smithi121 bash[36902]: cluster 2022-04-22T21:03:16.675655+0000 mgr.y (mgr.34107) 7422 : cluster [DBG] pgmap v8457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:18 smithi115 bash[45495]: cluster 2022-04-22T21:03:16.675655+0000 mgr.y (mgr.34107) 7422 : cluster [DBG] pgmap v8457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:18 smithi115 bash[46934]: cluster 2022-04-22T21:03:16.675655+0000 mgr.y (mgr.34107) 7422 : cluster [DBG] pgmap v8457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:19 smithi121 bash[36902]: audit 2022-04-22T21:03:18.982701+0000 mon.a (mon.0) 10907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:19 smithi115 bash[45495]: audit 2022-04-22T21:03:18.982701+0000 mon.a (mon.0) 10907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:19 smithi115 bash[46934]: audit 2022-04-22T21:03:18.982701+0000 mon.a (mon.0) 10907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:20 smithi121 bash[36902]: cluster 2022-04-22T21:03:18.676829+0000 mgr.y (mgr.34107) 7423 : cluster [DBG] pgmap v8458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:20 smithi115 bash[45495]: cluster 2022-04-22T21:03:18.676829+0000 mgr.y (mgr.34107) 7423 : cluster [DBG] pgmap v8458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:20 smithi115 bash[46934]: cluster 2022-04-22T21:03:18.676829+0000 mgr.y (mgr.34107) 7423 : cluster [DBG] pgmap v8458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:21 smithi121 bash[36902]: audit 2022-04-22T21:03:20.289631+0000 mon.a (mon.0) 10908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:21 smithi115 bash[45495]: audit 2022-04-22T21:03:20.289631+0000 mon.a (mon.0) 10908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:21 smithi115 bash[46934]: audit 2022-04-22T21:03:20.289631+0000 mon.a (mon.0) 10908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:22.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:22 smithi115 bash[45495]: cluster 2022-04-22T21:03:20.677922+0000 mgr.y (mgr.34107) 7424 : cluster [DBG] pgmap v8459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:22.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:22 smithi115 bash[46934]: cluster 2022-04-22T21:03:20.677922+0000 mgr.y (mgr.34107) 7424 : cluster [DBG] pgmap v8459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:22 smithi121 bash[36902]: cluster 2022-04-22T21:03:20.677922+0000 mgr.y (mgr.34107) 7424 : cluster [DBG] pgmap v8459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:22.458Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:24 smithi121 bash[36902]: cluster 2022-04-22T21:03:22.679016+0000 mgr.y (mgr.34107) 7425 : cluster [DBG] pgmap v8460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:24 smithi121 bash[36902]: audit 2022-04-22T21:03:23.986379+0000 mon.a (mon.0) 10909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:24.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:24.332Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:24 smithi115 bash[45495]: cluster 2022-04-22T21:03:22.679016+0000 mgr.y (mgr.34107) 7425 : cluster [DBG] pgmap v8460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:24.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:24 smithi115 bash[45495]: audit 2022-04-22T21:03:23.986379+0000 mon.a (mon.0) 10909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:24 smithi115 bash[46934]: cluster 2022-04-22T21:03:22.679016+0000 mgr.y (mgr.34107) 7425 : cluster [DBG] pgmap v8460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:24 smithi115 bash[46934]: audit 2022-04-22T21:03:23.986379+0000 mon.a (mon.0) 10909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:26 smithi121 bash[36902]: cluster 2022-04-22T21:03:24.680195+0000 mgr.y (mgr.34107) 7426 : cluster [DBG] pgmap v8461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:26 smithi121 bash[36902]: audit 2022-04-22T21:03:25.292688+0000 mon.a (mon.0) 10910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:26 smithi115 bash[45495]: cluster 2022-04-22T21:03:24.680195+0000 mgr.y (mgr.34107) 7426 : cluster [DBG] pgmap v8461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:26 smithi115 bash[45495]: audit 2022-04-22T21:03:25.292688+0000 mon.a (mon.0) 10910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:26 smithi115 bash[46934]: cluster 2022-04-22T21:03:24.680195+0000 mgr.y (mgr.34107) 7426 : cluster [DBG] pgmap v8461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:26 smithi115 bash[46934]: audit 2022-04-22T21:03:25.292688+0000 mon.a (mon.0) 10910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:28 smithi121 bash[36902]: cluster 2022-04-22T21:03:26.681307+0000 mgr.y (mgr.34107) 7427 : cluster [DBG] pgmap v8462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:28 smithi115 bash[45495]: cluster 2022-04-22T21:03:26.681307+0000 mgr.y (mgr.34107) 7427 : cluster [DBG] pgmap v8462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:28 smithi115 bash[46934]: cluster 2022-04-22T21:03:26.681307+0000 mgr.y (mgr.34107) 7427 : cluster [DBG] pgmap v8462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:29 smithi121 bash[36902]: audit 2022-04-22T21:03:28.990386+0000 mon.a (mon.0) 10911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:29 smithi115 bash[46934]: audit 2022-04-22T21:03:28.990386+0000 mon.a (mon.0) 10911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:29.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:29 smithi115 bash[45495]: audit 2022-04-22T21:03:28.990386+0000 mon.a (mon.0) 10911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:29.623 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:03:29.969 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 10m ago 3h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 53s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 10m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 53s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 53s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 10m ago 3h 487M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:03:29.970 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 10m ago 3h 355M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 53s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 10m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 10m ago 3h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 53s ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 10m ago 3h 709M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:03:29.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 10m ago 3h 787M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:03:29.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 10m ago 3h 533M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:03:29.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 10m ago 3h 492M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:03:29.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 53s ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:03:29.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 53s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:03:29.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 53s ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:03:29.973 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 53s ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:03:29.973 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 53s ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:03:30.387 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:03:30.387 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:03:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:03:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:03:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:03:30.388 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:03:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:03:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:03:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:03:30.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:03:30.391 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:03:30.391 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:03:30.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:03:30.391 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:03:30.392 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:03:30.392 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:03:30.393 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:03:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:30 smithi121 bash[36902]: cluster 2022-04-22T21:03:28.682395+0000 mgr.y (mgr.34107) 7428 : cluster [DBG] pgmap v8463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:30.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:30 smithi121 bash[36902]: audit 2022-04-22T21:03:29.603764+0000 mgr.y (mgr.34107) 7429 : audit [DBG] from='client.53023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:30 smithi115 bash[45495]: cluster 2022-04-22T21:03:28.682395+0000 mgr.y (mgr.34107) 7428 : cluster [DBG] pgmap v8463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:30 smithi115 bash[45495]: audit 2022-04-22T21:03:29.603764+0000 mgr.y (mgr.34107) 7429 : audit [DBG] from='client.53023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:30 smithi115 bash[46934]: cluster 2022-04-22T21:03:28.682395+0000 mgr.y (mgr.34107) 7428 : cluster [DBG] pgmap v8463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:30 smithi115 bash[46934]: audit 2022-04-22T21:03:29.603764+0000 mgr.y (mgr.34107) 7429 : audit [DBG] from='client.53023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:31 smithi121 bash[36902]: audit 2022-04-22T21:03:29.946327+0000 mgr.y (mgr.34107) 7430 : audit [DBG] from='client.53029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:31 smithi121 bash[36902]: audit 2022-04-22T21:03:30.295696+0000 mon.a (mon.0) 10912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:31 smithi121 bash[36902]: audit 2022-04-22T21:03:30.382363+0000 mon.c (mon.1) 274 : audit [DBG] from='client.? 172.21.15.115:0/1672705210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[45495]: audit 2022-04-22T21:03:29.946327+0000 mgr.y (mgr.34107) 7430 : audit [DBG] from='client.53029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[45495]: audit 2022-04-22T21:03:30.295696+0000 mon.a (mon.0) 10912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[45495]: audit 2022-04-22T21:03:30.382363+0000 mon.c (mon.1) 274 : audit [DBG] from='client.? 172.21.15.115:0/1672705210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[46934]: audit 2022-04-22T21:03:29.946327+0000 mgr.y (mgr.34107) 7430 : audit [DBG] from='client.53029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:03:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[46934]: audit 2022-04-22T21:03:30.295696+0000 mon.a (mon.0) 10912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:31 smithi115 bash[46934]: audit 2022-04-22T21:03:30.382363+0000 mon.c (mon.1) 274 : audit [DBG] from='client.? 172.21.15.115:0/1672705210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:03:32.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:32 smithi115 bash[46934]: cluster 2022-04-22T21:03:30.683521+0000 mgr.y (mgr.34107) 7431 : cluster [DBG] pgmap v8464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:32.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:32 smithi115 bash[45495]: cluster 2022-04-22T21:03:30.683521+0000 mgr.y (mgr.34107) 7431 : cluster [DBG] pgmap v8464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:32 smithi121 bash[36902]: cluster 2022-04-22T21:03:30.683521+0000 mgr.y (mgr.34107) 7431 : cluster [DBG] pgmap v8464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:32.459Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:34 smithi121 bash[36902]: cluster 2022-04-22T21:03:32.684632+0000 mgr.y (mgr.34107) 7432 : cluster [DBG] pgmap v8465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:34 smithi121 bash[36902]: audit 2022-04-22T21:03:33.993998+0000 mon.a (mon.0) 10913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:34.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:34.333Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:34 smithi115 bash[46934]: cluster 2022-04-22T21:03:32.684632+0000 mgr.y (mgr.34107) 7432 : cluster [DBG] pgmap v8465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:34 smithi115 bash[46934]: audit 2022-04-22T21:03:33.993998+0000 mon.a (mon.0) 10913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:34 smithi115 bash[45495]: cluster 2022-04-22T21:03:32.684632+0000 mgr.y (mgr.34107) 7432 : cluster [DBG] pgmap v8465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:34 smithi115 bash[45495]: audit 2022-04-22T21:03:33.993998+0000 mon.a (mon.0) 10913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:36 smithi121 bash[36902]: cluster 2022-04-22T21:03:34.685767+0000 mgr.y (mgr.34107) 7433 : cluster [DBG] pgmap v8466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:36.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:36 smithi121 bash[36902]: audit 2022-04-22T21:03:35.298557+0000 mon.a (mon.0) 10914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:36 smithi115 bash[45495]: cluster 2022-04-22T21:03:34.685767+0000 mgr.y (mgr.34107) 7433 : cluster [DBG] pgmap v8466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:36 smithi115 bash[45495]: audit 2022-04-22T21:03:35.298557+0000 mon.a (mon.0) 10914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:36 smithi115 bash[46934]: cluster 2022-04-22T21:03:34.685767+0000 mgr.y (mgr.34107) 7433 : cluster [DBG] pgmap v8466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:36 smithi115 bash[46934]: audit 2022-04-22T21:03:35.298557+0000 mon.a (mon.0) 10914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: cluster 2022-04-22T21:03:36.686906+0000 mgr.y (mgr.34107) 7434 : cluster [DBG] pgmap v8467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: audit 2022-04-22T21:03:37.316408+0000 mon.a (mon.0) 10915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:03:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: audit 2022-04-22T21:03:37.319549+0000 mon.a (mon.0) 10916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:03:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: audit 2022-04-22T21:03:37.321225+0000 mon.a (mon.0) 10917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:03:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: audit 2022-04-22T21:03:37.654635+0000 mon.a (mon.0) 10918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:38 smithi121 bash[36902]: audit 2022-04-22T21:03:37.661258+0000 mon.a (mon.0) 10919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: cluster 2022-04-22T21:03:36.686906+0000 mgr.y (mgr.34107) 7434 : cluster [DBG] pgmap v8467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: audit 2022-04-22T21:03:37.316408+0000 mon.a (mon.0) 10915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: audit 2022-04-22T21:03:37.319549+0000 mon.a (mon.0) 10916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: audit 2022-04-22T21:03:37.321225+0000 mon.a (mon.0) 10917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: audit 2022-04-22T21:03:37.654635+0000 mon.a (mon.0) 10918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[45495]: audit 2022-04-22T21:03:37.661258+0000 mon.a (mon.0) 10919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: cluster 2022-04-22T21:03:36.686906+0000 mgr.y (mgr.34107) 7434 : cluster [DBG] pgmap v8467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: audit 2022-04-22T21:03:37.316408+0000 mon.a (mon.0) 10915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: audit 2022-04-22T21:03:37.319549+0000 mon.a (mon.0) 10916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: audit 2022-04-22T21:03:37.321225+0000 mon.a (mon.0) 10917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: audit 2022-04-22T21:03:37.654635+0000 mon.a (mon.0) 10918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:38 smithi115 bash[46934]: audit 2022-04-22T21:03:37.661258+0000 mon.a (mon.0) 10919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:39 smithi121 bash[36902]: audit 2022-04-22T21:03:38.997858+0000 mon.a (mon.0) 10920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:39.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:39 smithi115 bash[46934]: audit 2022-04-22T21:03:38.997858+0000 mon.a (mon.0) 10920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:39.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:39 smithi115 bash[45495]: audit 2022-04-22T21:03:38.997858+0000 mon.a (mon.0) 10920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:40 smithi121 bash[36902]: cluster 2022-04-22T21:03:38.688046+0000 mgr.y (mgr.34107) 7435 : cluster [DBG] pgmap v8468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:40 smithi115 bash[46934]: cluster 2022-04-22T21:03:38.688046+0000 mgr.y (mgr.34107) 7435 : cluster [DBG] pgmap v8468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:40.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:40 smithi115 bash[45495]: cluster 2022-04-22T21:03:38.688046+0000 mgr.y (mgr.34107) 7435 : cluster [DBG] pgmap v8468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:41 smithi121 bash[36902]: audit 2022-04-22T21:03:40.301528+0000 mon.a (mon.0) 10921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:41 smithi115 bash[45495]: audit 2022-04-22T21:03:40.301528+0000 mon.a (mon.0) 10921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:41 smithi115 bash[46934]: audit 2022-04-22T21:03:40.301528+0000 mon.a (mon.0) 10921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[45495]: cluster 2022-04-22T21:03:40.689459+0000 mgr.y (mgr.34107) 7436 : cluster [DBG] pgmap v8469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[45495]: audit 2022-04-22T21:03:41.774247+0000 mon.a (mon.0) 10922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[45495]: audit 2022-04-22T21:03:42.092848+0000 mon.a (mon.0) 10923 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[46934]: cluster 2022-04-22T21:03:40.689459+0000 mgr.y (mgr.34107) 7436 : cluster [DBG] pgmap v8469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:42.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[46934]: audit 2022-04-22T21:03:41.774247+0000 mon.a (mon.0) 10922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[46934]: audit 2022-04-22T21:03:42.092848+0000 mon.a (mon.0) 10923 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:42 smithi121 bash[36902]: cluster 2022-04-22T21:03:40.689459+0000 mgr.y (mgr.34107) 7436 : cluster [DBG] pgmap v8469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:42 smithi121 bash[36902]: audit 2022-04-22T21:03:41.774247+0000 mon.a (mon.0) 10922 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:42 smithi121 bash[36902]: audit 2022-04-22T21:03:42.092848+0000 mon.a (mon.0) 10923 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:03:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:42.459Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:44 smithi121 bash[36902]: cluster 2022-04-22T21:03:42.690599+0000 mgr.y (mgr.34107) 7437 : cluster [DBG] pgmap v8470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:44 smithi121 bash[36902]: audit 2022-04-22T21:03:44.001652+0000 mon.a (mon.0) 10924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:44.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:44.334Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:44 smithi115 bash[45495]: cluster 2022-04-22T21:03:42.690599+0000 mgr.y (mgr.34107) 7437 : cluster [DBG] pgmap v8470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:44 smithi115 bash[45495]: audit 2022-04-22T21:03:44.001652+0000 mon.a (mon.0) 10924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:44 smithi115 bash[46934]: cluster 2022-04-22T21:03:42.690599+0000 mgr.y (mgr.34107) 7437 : cluster [DBG] pgmap v8470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:44 smithi115 bash[46934]: audit 2022-04-22T21:03:44.001652+0000 mon.a (mon.0) 10924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:46 smithi121 bash[36902]: cluster 2022-04-22T21:03:44.691624+0000 mgr.y (mgr.34107) 7438 : cluster [DBG] pgmap v8471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:46 smithi121 bash[36902]: audit 2022-04-22T21:03:45.304614+0000 mon.a (mon.0) 10925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:46 smithi115 bash[45495]: cluster 2022-04-22T21:03:44.691624+0000 mgr.y (mgr.34107) 7438 : cluster [DBG] pgmap v8471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:46 smithi115 bash[45495]: audit 2022-04-22T21:03:45.304614+0000 mon.a (mon.0) 10925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:46 smithi115 bash[46934]: cluster 2022-04-22T21:03:44.691624+0000 mgr.y (mgr.34107) 7438 : cluster [DBG] pgmap v8471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:46 smithi115 bash[46934]: audit 2022-04-22T21:03:45.304614+0000 mon.a (mon.0) 10925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:48 smithi121 bash[36902]: cluster 2022-04-22T21:03:46.692751+0000 mgr.y (mgr.34107) 7439 : cluster [DBG] pgmap v8472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:48 smithi115 bash[45495]: cluster 2022-04-22T21:03:46.692751+0000 mgr.y (mgr.34107) 7439 : cluster [DBG] pgmap v8472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:48 smithi115 bash[46934]: cluster 2022-04-22T21:03:46.692751+0000 mgr.y (mgr.34107) 7439 : cluster [DBG] pgmap v8472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:49 smithi121 bash[36902]: audit 2022-04-22T21:03:49.005418+0000 mon.a (mon.0) 10926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:49 smithi115 bash[45495]: audit 2022-04-22T21:03:49.005418+0000 mon.a (mon.0) 10926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:49 smithi115 bash[46934]: audit 2022-04-22T21:03:49.005418+0000 mon.a (mon.0) 10926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:50 smithi121 bash[36902]: cluster 2022-04-22T21:03:48.693954+0000 mgr.y (mgr.34107) 7440 : cluster [DBG] pgmap v8473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:50 smithi115 bash[45495]: cluster 2022-04-22T21:03:48.693954+0000 mgr.y (mgr.34107) 7440 : cluster [DBG] pgmap v8473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:50 smithi115 bash[46934]: cluster 2022-04-22T21:03:48.693954+0000 mgr.y (mgr.34107) 7440 : cluster [DBG] pgmap v8473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:51 smithi121 bash[36902]: audit 2022-04-22T21:03:50.307912+0000 mon.a (mon.0) 10927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:51 smithi115 bash[45495]: audit 2022-04-22T21:03:50.307912+0000 mon.a (mon.0) 10927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:51 smithi115 bash[46934]: audit 2022-04-22T21:03:50.307912+0000 mon.a (mon.0) 10927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:52 smithi121 bash[36902]: cluster 2022-04-22T21:03:50.695150+0000 mgr.y (mgr.34107) 7441 : cluster [DBG] pgmap v8474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:52 smithi115 bash[45495]: cluster 2022-04-22T21:03:50.695150+0000 mgr.y (mgr.34107) 7441 : cluster [DBG] pgmap v8474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:52 smithi115 bash[46934]: cluster 2022-04-22T21:03:50.695150+0000 mgr.y (mgr.34107) 7441 : cluster [DBG] pgmap v8474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:52.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:52.458Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:54 smithi121 bash[36902]: cluster 2022-04-22T21:03:52.696396+0000 mgr.y (mgr.34107) 7442 : cluster [DBG] pgmap v8475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:54 smithi121 bash[36902]: audit 2022-04-22T21:03:54.008982+0000 mon.a (mon.0) 10928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:54.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:03:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:03:54.335Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:03:54.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:54 smithi115 bash[46934]: cluster 2022-04-22T21:03:52.696396+0000 mgr.y (mgr.34107) 7442 : cluster [DBG] pgmap v8475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:54.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:54 smithi115 bash[46934]: audit 2022-04-22T21:03:54.008982+0000 mon.a (mon.0) 10928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:54 smithi115 bash[45495]: cluster 2022-04-22T21:03:52.696396+0000 mgr.y (mgr.34107) 7442 : cluster [DBG] pgmap v8475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:54.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:54 smithi115 bash[45495]: audit 2022-04-22T21:03:54.008982+0000 mon.a (mon.0) 10928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:56 smithi121 bash[36902]: cluster 2022-04-22T21:03:54.697500+0000 mgr.y (mgr.34107) 7443 : cluster [DBG] pgmap v8476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:56 smithi121 bash[36902]: audit 2022-04-22T21:03:55.310980+0000 mon.a (mon.0) 10929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:56 smithi115 bash[45495]: cluster 2022-04-22T21:03:54.697500+0000 mgr.y (mgr.34107) 7443 : cluster [DBG] pgmap v8476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:56 smithi115 bash[45495]: audit 2022-04-22T21:03:55.310980+0000 mon.a (mon.0) 10929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:56 smithi115 bash[46934]: cluster 2022-04-22T21:03:54.697500+0000 mgr.y (mgr.34107) 7443 : cluster [DBG] pgmap v8476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:56 smithi115 bash[46934]: audit 2022-04-22T21:03:55.310980+0000 mon.a (mon.0) 10929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:03:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:58 smithi121 bash[36902]: cluster 2022-04-22T21:03:56.698646+0000 mgr.y (mgr.34107) 7444 : cluster [DBG] pgmap v8477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:58 smithi115 bash[45495]: cluster 2022-04-22T21:03:56.698646+0000 mgr.y (mgr.34107) 7444 : cluster [DBG] pgmap v8477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:58 smithi115 bash[46934]: cluster 2022-04-22T21:03:56.698646+0000 mgr.y (mgr.34107) 7444 : cluster [DBG] pgmap v8477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:03:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:03:59 smithi121 bash[36902]: audit 2022-04-22T21:03:59.012203+0000 mon.a (mon.0) 10930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:59.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:03:59 smithi115 bash[45495]: audit 2022-04-22T21:03:59.012203+0000 mon.a (mon.0) 10930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:03:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:03:59 smithi115 bash[46934]: audit 2022-04-22T21:03:59.012203+0000 mon.a (mon.0) 10930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:00 smithi121 bash[36902]: cluster 2022-04-22T21:03:58.699819+0000 mgr.y (mgr.34107) 7445 : cluster [DBG] pgmap v8478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:00 smithi115 bash[46934]: cluster 2022-04-22T21:03:58.699819+0000 mgr.y (mgr.34107) 7445 : cluster [DBG] pgmap v8478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:00.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:00 smithi115 bash[45495]: cluster 2022-04-22T21:03:58.699819+0000 mgr.y (mgr.34107) 7445 : cluster [DBG] pgmap v8478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 19s ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 84s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 19s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 84s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 84s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:04:01.123 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 19s ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 19s ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 84s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 19s ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 19s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 84s ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 19s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 19s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 19s ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 19s ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 84s ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:04:01.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 84s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:04:01.125 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 84s ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:04:01.125 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 84s ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:04:01.125 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 84s ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:04:01.533 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:04:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:04:01.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:04:01.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:01.535 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:04:01.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:04:01.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:04:01.535 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:04:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:01 smithi115 bash[46934]: audit 2022-04-22T21:04:00.313976+0000 mon.a (mon.0) 10931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:01 smithi115 bash[45495]: audit 2022-04-22T21:04:00.313976+0000 mon.a (mon.0) 10931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:01 smithi121 bash[36902]: audit 2022-04-22T21:04:00.313976+0000 mon.a (mon.0) 10931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:02.545 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:02.460Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[45495]: cluster 2022-04-22T21:04:00.700557+0000 mgr.y (mgr.34107) 7446 : cluster [DBG] pgmap v8479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[45495]: audit 2022-04-22T21:04:00.749176+0000 mgr.y (mgr.34107) 7447 : audit [DBG] from='client.53041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[45495]: audit 2022-04-22T21:04:01.090079+0000 mgr.y (mgr.34107) 7448 : audit [DBG] from='client.43116 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[45495]: audit 2022-04-22T21:04:01.528914+0000 mon.c (mon.1) 275 : audit [DBG] from='client.? 172.21.15.115:0/883062882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[46934]: cluster 2022-04-22T21:04:00.700557+0000 mgr.y (mgr.34107) 7446 : cluster [DBG] pgmap v8479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[46934]: audit 2022-04-22T21:04:00.749176+0000 mgr.y (mgr.34107) 7447 : audit [DBG] from='client.53041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[46934]: audit 2022-04-22T21:04:01.090079+0000 mgr.y (mgr.34107) 7448 : audit [DBG] from='client.43116 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:02 smithi115 bash[46934]: audit 2022-04-22T21:04:01.528914+0000 mon.c (mon.1) 275 : audit [DBG] from='client.? 172.21.15.115:0/883062882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:02 smithi121 bash[36902]: cluster 2022-04-22T21:04:00.700557+0000 mgr.y (mgr.34107) 7446 : cluster [DBG] pgmap v8479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:02 smithi121 bash[36902]: audit 2022-04-22T21:04:00.749176+0000 mgr.y (mgr.34107) 7447 : audit [DBG] from='client.53041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:02 smithi121 bash[36902]: audit 2022-04-22T21:04:01.090079+0000 mgr.y (mgr.34107) 7448 : audit [DBG] from='client.43116 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:02 smithi121 bash[36902]: audit 2022-04-22T21:04:01.528914+0000 mon.c (mon.1) 275 : audit [DBG] from='client.? 172.21.15.115:0/883062882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:04.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:04.336Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:04 smithi115 bash[45495]: cluster 2022-04-22T21:04:02.701620+0000 mgr.y (mgr.34107) 7449 : cluster [DBG] pgmap v8480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:04 smithi115 bash[45495]: audit 2022-04-22T21:04:04.015815+0000 mon.a (mon.0) 10932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:04 smithi115 bash[46934]: cluster 2022-04-22T21:04:02.701620+0000 mgr.y (mgr.34107) 7449 : cluster [DBG] pgmap v8480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:04 smithi115 bash[46934]: audit 2022-04-22T21:04:04.015815+0000 mon.a (mon.0) 10932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:04 smithi121 bash[36902]: cluster 2022-04-22T21:04:02.701620+0000 mgr.y (mgr.34107) 7449 : cluster [DBG] pgmap v8480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:04 smithi121 bash[36902]: audit 2022-04-22T21:04:04.015815+0000 mon.a (mon.0) 10932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:06 smithi115 bash[45495]: cluster 2022-04-22T21:04:04.702729+0000 mgr.y (mgr.34107) 7450 : cluster [DBG] pgmap v8481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:06 smithi115 bash[45495]: audit 2022-04-22T21:04:05.316840+0000 mon.a (mon.0) 10933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:06 smithi115 bash[46934]: cluster 2022-04-22T21:04:04.702729+0000 mgr.y (mgr.34107) 7450 : cluster [DBG] pgmap v8481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:06 smithi115 bash[46934]: audit 2022-04-22T21:04:05.316840+0000 mon.a (mon.0) 10933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:06 smithi121 bash[36902]: cluster 2022-04-22T21:04:04.702729+0000 mgr.y (mgr.34107) 7450 : cluster [DBG] pgmap v8481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:06 smithi121 bash[36902]: audit 2022-04-22T21:04:05.316840+0000 mon.a (mon.0) 10933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:08 smithi115 bash[45495]: cluster 2022-04-22T21:04:06.704165+0000 mgr.y (mgr.34107) 7451 : cluster [DBG] pgmap v8482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:08.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:08 smithi115 bash[46934]: cluster 2022-04-22T21:04:06.704165+0000 mgr.y (mgr.34107) 7451 : cluster [DBG] pgmap v8482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:08 smithi121 bash[36902]: cluster 2022-04-22T21:04:06.704165+0000 mgr.y (mgr.34107) 7451 : cluster [DBG] pgmap v8482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:09 smithi121 bash[36902]: audit 2022-04-22T21:04:09.019624+0000 mon.a (mon.0) 10934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:09 smithi115 bash[45495]: audit 2022-04-22T21:04:09.019624+0000 mon.a (mon.0) 10934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:09 smithi115 bash[46934]: audit 2022-04-22T21:04:09.019624+0000 mon.a (mon.0) 10934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:10 smithi121 bash[36902]: cluster 2022-04-22T21:04:08.705295+0000 mgr.y (mgr.34107) 7452 : cluster [DBG] pgmap v8483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:10 smithi115 bash[45495]: cluster 2022-04-22T21:04:08.705295+0000 mgr.y (mgr.34107) 7452 : cluster [DBG] pgmap v8483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:10 smithi115 bash[46934]: cluster 2022-04-22T21:04:08.705295+0000 mgr.y (mgr.34107) 7452 : cluster [DBG] pgmap v8483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:11 smithi121 bash[36902]: audit 2022-04-22T21:04:10.319860+0000 mon.a (mon.0) 10935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:11.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:11 smithi115 bash[45495]: audit 2022-04-22T21:04:10.319860+0000 mon.a (mon.0) 10935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:11 smithi115 bash[46934]: audit 2022-04-22T21:04:10.319860+0000 mon.a (mon.0) 10935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:12 smithi121 bash[36902]: cluster 2022-04-22T21:04:10.706378+0000 mgr.y (mgr.34107) 7453 : cluster [DBG] pgmap v8484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:12 smithi115 bash[45495]: cluster 2022-04-22T21:04:10.706378+0000 mgr.y (mgr.34107) 7453 : cluster [DBG] pgmap v8484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:12.460Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:12 smithi115 bash[46934]: cluster 2022-04-22T21:04:10.706378+0000 mgr.y (mgr.34107) 7453 : cluster [DBG] pgmap v8484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:14 smithi121 bash[36902]: cluster 2022-04-22T21:04:12.707628+0000 mgr.y (mgr.34107) 7454 : cluster [DBG] pgmap v8485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:14 smithi121 bash[36902]: audit 2022-04-22T21:04:14.023242+0000 mon.a (mon.0) 10936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:14.338Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:14 smithi115 bash[45495]: cluster 2022-04-22T21:04:12.707628+0000 mgr.y (mgr.34107) 7454 : cluster [DBG] pgmap v8485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:14 smithi115 bash[45495]: audit 2022-04-22T21:04:14.023242+0000 mon.a (mon.0) 10936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:14 smithi115 bash[46934]: cluster 2022-04-22T21:04:12.707628+0000 mgr.y (mgr.34107) 7454 : cluster [DBG] pgmap v8485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:14 smithi115 bash[46934]: audit 2022-04-22T21:04:14.023242+0000 mon.a (mon.0) 10936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:16 smithi121 bash[36902]: cluster 2022-04-22T21:04:14.708891+0000 mgr.y (mgr.34107) 7455 : cluster [DBG] pgmap v8486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:16 smithi121 bash[36902]: audit 2022-04-22T21:04:15.322611+0000 mon.a (mon.0) 10937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:16 smithi121 bash[36902]: audit 2022-04-22T21:04:15.777540+0000 mon.a (mon.0) 10938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:16 smithi121 bash[36902]: audit 2022-04-22T21:04:15.783865+0000 mon.a (mon.0) 10939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:04:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[46934]: cluster 2022-04-22T21:04:14.708891+0000 mgr.y (mgr.34107) 7455 : cluster [DBG] pgmap v8486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[46934]: audit 2022-04-22T21:04:15.322611+0000 mon.a (mon.0) 10937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[46934]: audit 2022-04-22T21:04:15.777540+0000 mon.a (mon.0) 10938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:04:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[46934]: audit 2022-04-22T21:04:15.783865+0000 mon.a (mon.0) 10939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:04:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[45495]: cluster 2022-04-22T21:04:14.708891+0000 mgr.y (mgr.34107) 7455 : cluster [DBG] pgmap v8486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[45495]: audit 2022-04-22T21:04:15.322611+0000 mon.a (mon.0) 10937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[45495]: audit 2022-04-22T21:04:15.777540+0000 mon.a (mon.0) 10938 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:04:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:16 smithi115 bash[45495]: audit 2022-04-22T21:04:15.783865+0000 mon.a (mon.0) 10939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:04:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:18 smithi121 bash[36902]: cluster 2022-04-22T21:04:16.710025+0000 mgr.y (mgr.34107) 7456 : cluster [DBG] pgmap v8487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:18.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:18 smithi115 bash[45495]: cluster 2022-04-22T21:04:16.710025+0000 mgr.y (mgr.34107) 7456 : cluster [DBG] pgmap v8487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:18 smithi115 bash[46934]: cluster 2022-04-22T21:04:16.710025+0000 mgr.y (mgr.34107) 7456 : cluster [DBG] pgmap v8487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:19 smithi121 bash[36902]: audit 2022-04-22T21:04:19.026961+0000 mon.a (mon.0) 10940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:19 smithi115 bash[45495]: audit 2022-04-22T21:04:19.026961+0000 mon.a (mon.0) 10940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:19 smithi115 bash[46934]: audit 2022-04-22T21:04:19.026961+0000 mon.a (mon.0) 10940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:20 smithi121 bash[36902]: cluster 2022-04-22T21:04:18.711180+0000 mgr.y (mgr.34107) 7457 : cluster [DBG] pgmap v8488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:20 smithi121 bash[36902]: audit 2022-04-22T21:04:20.325170+0000 mon.a (mon.0) 10941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:20 smithi115 bash[45495]: cluster 2022-04-22T21:04:18.711180+0000 mgr.y (mgr.34107) 7457 : cluster [DBG] pgmap v8488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:20 smithi115 bash[45495]: audit 2022-04-22T21:04:20.325170+0000 mon.a (mon.0) 10941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:20 smithi115 bash[46934]: cluster 2022-04-22T21:04:18.711180+0000 mgr.y (mgr.34107) 7457 : cluster [DBG] pgmap v8488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:20 smithi115 bash[46934]: audit 2022-04-22T21:04:20.325170+0000 mon.a (mon.0) 10941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:22 smithi121 bash[36902]: cluster 2022-04-22T21:04:20.712292+0000 mgr.y (mgr.34107) 7458 : cluster [DBG] pgmap v8489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:22 smithi115 bash[45495]: cluster 2022-04-22T21:04:20.712292+0000 mgr.y (mgr.34107) 7458 : cluster [DBG] pgmap v8489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:22 smithi115 bash[46934]: cluster 2022-04-22T21:04:20.712292+0000 mgr.y (mgr.34107) 7458 : cluster [DBG] pgmap v8489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:22.462Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:24 smithi121 bash[36902]: cluster 2022-04-22T21:04:22.713409+0000 mgr.y (mgr.34107) 7459 : cluster [DBG] pgmap v8490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:24 smithi121 bash[36902]: audit 2022-04-22T21:04:24.030757+0000 mon.a (mon.0) 10942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:24.337Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:24 smithi115 bash[45495]: cluster 2022-04-22T21:04:22.713409+0000 mgr.y (mgr.34107) 7459 : cluster [DBG] pgmap v8490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:24 smithi115 bash[45495]: audit 2022-04-22T21:04:24.030757+0000 mon.a (mon.0) 10942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:24 smithi115 bash[46934]: cluster 2022-04-22T21:04:22.713409+0000 mgr.y (mgr.34107) 7459 : cluster [DBG] pgmap v8490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:24 smithi115 bash[46934]: audit 2022-04-22T21:04:24.030757+0000 mon.a (mon.0) 10942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:25 smithi121 bash[36902]: cluster 2022-04-22T21:04:24.714481+0000 mgr.y (mgr.34107) 7460 : cluster [DBG] pgmap v8491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:25 smithi121 bash[36902]: audit 2022-04-22T21:04:25.328029+0000 mon.a (mon.0) 10943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:25 smithi115 bash[45495]: cluster 2022-04-22T21:04:24.714481+0000 mgr.y (mgr.34107) 7460 : cluster [DBG] pgmap v8491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:25 smithi115 bash[45495]: audit 2022-04-22T21:04:25.328029+0000 mon.a (mon.0) 10943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:25 smithi115 bash[46934]: cluster 2022-04-22T21:04:24.714481+0000 mgr.y (mgr.34107) 7460 : cluster [DBG] pgmap v8491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:26.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:25 smithi115 bash[46934]: audit 2022-04-22T21:04:25.328029+0000 mon.a (mon.0) 10943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:27 smithi121 bash[36902]: cluster 2022-04-22T21:04:26.715630+0000 mgr.y (mgr.34107) 7461 : cluster [DBG] pgmap v8492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:28.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:27 smithi115 bash[45495]: cluster 2022-04-22T21:04:26.715630+0000 mgr.y (mgr.34107) 7461 : cluster [DBG] pgmap v8492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:27 smithi115 bash[46934]: cluster 2022-04-22T21:04:26.715630+0000 mgr.y (mgr.34107) 7461 : cluster [DBG] pgmap v8492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:29 smithi121 bash[36902]: cluster 2022-04-22T21:04:28.716739+0000 mgr.y (mgr.34107) 7462 : cluster [DBG] pgmap v8493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:29 smithi121 bash[36902]: audit 2022-04-22T21:04:29.034580+0000 mon.a (mon.0) 10944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:29 smithi115 bash[45495]: cluster 2022-04-22T21:04:28.716739+0000 mgr.y (mgr.34107) 7462 : cluster [DBG] pgmap v8493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:29 smithi115 bash[45495]: audit 2022-04-22T21:04:29.034580+0000 mon.a (mon.0) 10944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:29 smithi115 bash[46934]: cluster 2022-04-22T21:04:28.716739+0000 mgr.y (mgr.34107) 7462 : cluster [DBG] pgmap v8493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:29 smithi115 bash[46934]: audit 2022-04-22T21:04:29.034580+0000 mon.a (mon.0) 10944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:30 smithi121 bash[36902]: audit 2022-04-22T21:04:30.331294+0000 mon.a (mon.0) 10945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:30 smithi115 bash[45495]: audit 2022-04-22T21:04:30.331294+0000 mon.a (mon.0) 10945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:31.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:30 smithi115 bash[46934]: audit 2022-04-22T21:04:30.331294+0000 mon.a (mon.0) 10945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:31.917 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:04:32.263 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 50s ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 115s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 50s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 115s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 115s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 50s ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 50s ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 115s ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:04:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 50s ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 50s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 115s ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 50s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 50s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 50s ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 50s ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 115s ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 115s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:04:32.265 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 115s ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:04:32.266 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 115s ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:04:32.266 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 115s ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:04:32.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:31 smithi121 bash[36902]: cluster 2022-04-22T21:04:30.717754+0000 mgr.y (mgr.34107) 7463 : cluster [DBG] pgmap v8494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:31 smithi115 bash[45495]: cluster 2022-04-22T21:04:30.717754+0000 mgr.y (mgr.34107) 7463 : cluster [DBG] pgmap v8494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:31 smithi115 bash[46934]: cluster 2022-04-22T21:04:30.717754+0000 mgr.y (mgr.34107) 7463 : cluster [DBG] pgmap v8494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:04:32.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:04:32.672 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:04:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:32.462Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:32 smithi121 bash[36902]: audit 2022-04-22T21:04:31.894478+0000 mgr.y (mgr.34107) 7464 : audit [DBG] from='client.43128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:32 smithi121 bash[36902]: audit 2022-04-22T21:04:32.238588+0000 mgr.y (mgr.34107) 7465 : audit [DBG] from='client.53065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:32 smithi121 bash[36902]: audit 2022-04-22T21:04:32.665940+0000 mon.a (mon.0) 10946 : audit [DBG] from='client.? 172.21.15.115:0/4020041047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[45495]: audit 2022-04-22T21:04:31.894478+0000 mgr.y (mgr.34107) 7464 : audit [DBG] from='client.43128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[45495]: audit 2022-04-22T21:04:32.238588+0000 mgr.y (mgr.34107) 7465 : audit [DBG] from='client.53065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[45495]: audit 2022-04-22T21:04:32.665940+0000 mon.a (mon.0) 10946 : audit [DBG] from='client.? 172.21.15.115:0/4020041047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[46934]: audit 2022-04-22T21:04:31.894478+0000 mgr.y (mgr.34107) 7464 : audit [DBG] from='client.43128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[46934]: audit 2022-04-22T21:04:32.238588+0000 mgr.y (mgr.34107) 7465 : audit [DBG] from='client.53065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:04:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:32 smithi115 bash[46934]: audit 2022-04-22T21:04:32.665940+0000 mon.a (mon.0) 10946 : audit [DBG] from='client.? 172.21.15.115:0/4020041047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:04:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:33 smithi121 bash[36902]: cluster 2022-04-22T21:04:32.718818+0000 mgr.y (mgr.34107) 7466 : cluster [DBG] pgmap v8495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:33 smithi115 bash[45495]: cluster 2022-04-22T21:04:32.718818+0000 mgr.y (mgr.34107) 7466 : cluster [DBG] pgmap v8495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:33 smithi115 bash[46934]: cluster 2022-04-22T21:04:32.718818+0000 mgr.y (mgr.34107) 7466 : cluster [DBG] pgmap v8495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:34.338Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:34 smithi121 bash[36902]: audit 2022-04-22T21:04:34.038019+0000 mon.a (mon.0) 10947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:34 smithi115 bash[46934]: audit 2022-04-22T21:04:34.038019+0000 mon.a (mon.0) 10947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:34 smithi115 bash[45495]: audit 2022-04-22T21:04:34.038019+0000 mon.a (mon.0) 10947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:35 smithi121 bash[36902]: cluster 2022-04-22T21:04:34.719929+0000 mgr.y (mgr.34107) 7467 : cluster [DBG] pgmap v8496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:35 smithi121 bash[36902]: audit 2022-04-22T21:04:35.334207+0000 mon.a (mon.0) 10948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:36.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:35 smithi115 bash[45495]: cluster 2022-04-22T21:04:34.719929+0000 mgr.y (mgr.34107) 7467 : cluster [DBG] pgmap v8496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:36.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:35 smithi115 bash[45495]: audit 2022-04-22T21:04:35.334207+0000 mon.a (mon.0) 10948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:35 smithi115 bash[46934]: cluster 2022-04-22T21:04:34.719929+0000 mgr.y (mgr.34107) 7467 : cluster [DBG] pgmap v8496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:36.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:35 smithi115 bash[46934]: audit 2022-04-22T21:04:35.334207+0000 mon.a (mon.0) 10948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:37 smithi121 bash[36902]: cluster 2022-04-22T21:04:36.721028+0000 mgr.y (mgr.34107) 7468 : cluster [DBG] pgmap v8497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:38.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:37 smithi115 bash[45495]: cluster 2022-04-22T21:04:36.721028+0000 mgr.y (mgr.34107) 7468 : cluster [DBG] pgmap v8497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:37 smithi115 bash[46934]: cluster 2022-04-22T21:04:36.721028+0000 mgr.y (mgr.34107) 7468 : cluster [DBG] pgmap v8497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:39 smithi121 bash[36902]: cluster 2022-04-22T21:04:38.722182+0000 mgr.y (mgr.34107) 7469 : cluster [DBG] pgmap v8498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:39 smithi121 bash[36902]: audit 2022-04-22T21:04:39.041536+0000 mon.a (mon.0) 10949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:39 smithi115 bash[45495]: cluster 2022-04-22T21:04:38.722182+0000 mgr.y (mgr.34107) 7469 : cluster [DBG] pgmap v8498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:39 smithi115 bash[45495]: audit 2022-04-22T21:04:39.041536+0000 mon.a (mon.0) 10949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:39 smithi115 bash[46934]: cluster 2022-04-22T21:04:38.722182+0000 mgr.y (mgr.34107) 7469 : cluster [DBG] pgmap v8498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:39 smithi115 bash[46934]: audit 2022-04-22T21:04:39.041536+0000 mon.a (mon.0) 10949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:40 smithi121 bash[36902]: audit 2022-04-22T21:04:40.337280+0000 mon.a (mon.0) 10950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:40 smithi115 bash[45495]: audit 2022-04-22T21:04:40.337280+0000 mon.a (mon.0) 10950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:40 smithi115 bash[46934]: audit 2022-04-22T21:04:40.337280+0000 mon.a (mon.0) 10950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:42.132 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:41 smithi115 bash[45495]: cluster 2022-04-22T21:04:40.723281+0000 mgr.y (mgr.34107) 7470 : cluster [DBG] pgmap v8499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:42.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:41 smithi115 bash[46934]: cluster 2022-04-22T21:04:40.723281+0000 mgr.y (mgr.34107) 7470 : cluster [DBG] pgmap v8499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:42.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:41 smithi121 bash[36902]: cluster 2022-04-22T21:04:40.723281+0000 mgr.y (mgr.34107) 7470 : cluster [DBG] pgmap v8499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:42.475Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:42 smithi121 bash[36902]: audit 2022-04-22T21:04:42.102985+0000 mon.a (mon.0) 10951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:04:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:42 smithi121 bash[36902]: audit 2022-04-22T21:04:42.106096+0000 mon.a (mon.0) 10952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:04:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:42 smithi121 bash[36902]: audit 2022-04-22T21:04:42.107851+0000 mon.a (mon.0) 10953 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:04:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:42 smithi121 bash[36902]: audit 2022-04-22T21:04:42.486922+0000 mon.a (mon.0) 10954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:04:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[45495]: audit 2022-04-22T21:04:42.102985+0000 mon.a (mon.0) 10951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:04:43.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[45495]: audit 2022-04-22T21:04:42.106096+0000 mon.a (mon.0) 10952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:04:43.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[45495]: audit 2022-04-22T21:04:42.107851+0000 mon.a (mon.0) 10953 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:04:43.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[45495]: audit 2022-04-22T21:04:42.486922+0000 mon.a (mon.0) 10954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:04:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[46934]: audit 2022-04-22T21:04:42.102985+0000 mon.a (mon.0) 10951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:04:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[46934]: audit 2022-04-22T21:04:42.106096+0000 mon.a (mon.0) 10952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:04:43.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[46934]: audit 2022-04-22T21:04:42.107851+0000 mon.a (mon.0) 10953 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:04:43.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:42 smithi115 bash[46934]: audit 2022-04-22T21:04:42.486922+0000 mon.a (mon.0) 10954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:04:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:44 smithi121 bash[36902]: cluster 2022-04-22T21:04:42.724439+0000 mgr.y (mgr.34107) 7471 : cluster [DBG] pgmap v8500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:44 smithi121 bash[36902]: audit 2022-04-22T21:04:44.045047+0000 mon.a (mon.0) 10955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:44.338Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:44 smithi115 bash[46934]: cluster 2022-04-22T21:04:42.724439+0000 mgr.y (mgr.34107) 7471 : cluster [DBG] pgmap v8500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:44 smithi115 bash[46934]: audit 2022-04-22T21:04:44.045047+0000 mon.a (mon.0) 10955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:44 smithi115 bash[45495]: cluster 2022-04-22T21:04:42.724439+0000 mgr.y (mgr.34107) 7471 : cluster [DBG] pgmap v8500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:44.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:44 smithi115 bash[45495]: audit 2022-04-22T21:04:44.045047+0000 mon.a (mon.0) 10955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:45 smithi121 bash[36902]: cluster 2022-04-22T21:04:44.725518+0000 mgr.y (mgr.34107) 7472 : cluster [DBG] pgmap v8501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:45 smithi121 bash[36902]: audit 2022-04-22T21:04:45.340476+0000 mon.a (mon.0) 10956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:45 smithi115 bash[46934]: cluster 2022-04-22T21:04:44.725518+0000 mgr.y (mgr.34107) 7472 : cluster [DBG] pgmap v8501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:45 smithi115 bash[46934]: audit 2022-04-22T21:04:45.340476+0000 mon.a (mon.0) 10956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:45 smithi115 bash[45495]: cluster 2022-04-22T21:04:44.725518+0000 mgr.y (mgr.34107) 7472 : cluster [DBG] pgmap v8501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:45 smithi115 bash[45495]: audit 2022-04-22T21:04:45.340476+0000 mon.a (mon.0) 10956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:47 smithi121 bash[36902]: cluster 2022-04-22T21:04:46.726630+0000 mgr.y (mgr.34107) 7473 : cluster [DBG] pgmap v8502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:47 smithi115 bash[45495]: cluster 2022-04-22T21:04:46.726630+0000 mgr.y (mgr.34107) 7473 : cluster [DBG] pgmap v8502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:48.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:47 smithi115 bash[46934]: cluster 2022-04-22T21:04:46.726630+0000 mgr.y (mgr.34107) 7473 : cluster [DBG] pgmap v8502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:49 smithi121 bash[36902]: cluster 2022-04-22T21:04:48.727780+0000 mgr.y (mgr.34107) 7474 : cluster [DBG] pgmap v8503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:49 smithi121 bash[36902]: audit 2022-04-22T21:04:49.048662+0000 mon.a (mon.0) 10957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:49 smithi115 bash[45495]: cluster 2022-04-22T21:04:48.727780+0000 mgr.y (mgr.34107) 7474 : cluster [DBG] pgmap v8503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:49 smithi115 bash[45495]: audit 2022-04-22T21:04:49.048662+0000 mon.a (mon.0) 10957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:49 smithi115 bash[46934]: cluster 2022-04-22T21:04:48.727780+0000 mgr.y (mgr.34107) 7474 : cluster [DBG] pgmap v8503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:49 smithi115 bash[46934]: audit 2022-04-22T21:04:49.048662+0000 mon.a (mon.0) 10957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:50 smithi121 bash[36902]: audit 2022-04-22T21:04:50.343733+0000 mon.a (mon.0) 10958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:50 smithi115 bash[45495]: audit 2022-04-22T21:04:50.343733+0000 mon.a (mon.0) 10958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:50 smithi115 bash[46934]: audit 2022-04-22T21:04:50.343733+0000 mon.a (mon.0) 10958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:51 smithi121 bash[36902]: cluster 2022-04-22T21:04:50.728845+0000 mgr.y (mgr.34107) 7475 : cluster [DBG] pgmap v8504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:51 smithi115 bash[46934]: cluster 2022-04-22T21:04:50.728845+0000 mgr.y (mgr.34107) 7475 : cluster [DBG] pgmap v8504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:51 smithi115 bash[45495]: cluster 2022-04-22T21:04:50.728845+0000 mgr.y (mgr.34107) 7475 : cluster [DBG] pgmap v8504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:52.464Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:54.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:53 smithi121 bash[36902]: cluster 2022-04-22T21:04:52.729975+0000 mgr.y (mgr.34107) 7476 : cluster [DBG] pgmap v8505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:53 smithi115 bash[45495]: cluster 2022-04-22T21:04:52.729975+0000 mgr.y (mgr.34107) 7476 : cluster [DBG] pgmap v8505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:54.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:53 smithi115 bash[46934]: cluster 2022-04-22T21:04:52.729975+0000 mgr.y (mgr.34107) 7476 : cluster [DBG] pgmap v8505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:04:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:04:54.340Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:04:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:54 smithi121 bash[36902]: audit 2022-04-22T21:04:54.052195+0000 mon.a (mon.0) 10959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:54 smithi115 bash[45495]: audit 2022-04-22T21:04:54.052195+0000 mon.a (mon.0) 10959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:54 smithi115 bash[46934]: audit 2022-04-22T21:04:54.052195+0000 mon.a (mon.0) 10959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:04:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:55 smithi121 bash[36902]: cluster 2022-04-22T21:04:54.731044+0000 mgr.y (mgr.34107) 7477 : cluster [DBG] pgmap v8506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:55 smithi121 bash[36902]: audit 2022-04-22T21:04:55.346630+0000 mon.a (mon.0) 10960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:55 smithi115 bash[45495]: cluster 2022-04-22T21:04:54.731044+0000 mgr.y (mgr.34107) 7477 : cluster [DBG] pgmap v8506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:56.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:55 smithi115 bash[45495]: audit 2022-04-22T21:04:55.346630+0000 mon.a (mon.0) 10960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:55 smithi115 bash[46934]: cluster 2022-04-22T21:04:54.731044+0000 mgr.y (mgr.34107) 7477 : cluster [DBG] pgmap v8506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:56.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:55 smithi115 bash[46934]: audit 2022-04-22T21:04:55.346630+0000 mon.a (mon.0) 10960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:04:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:57 smithi121 bash[36902]: cluster 2022-04-22T21:04:56.732164+0000 mgr.y (mgr.34107) 7478 : cluster [DBG] pgmap v8507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:58.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:57 smithi115 bash[45495]: cluster 2022-04-22T21:04:56.732164+0000 mgr.y (mgr.34107) 7478 : cluster [DBG] pgmap v8507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:04:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:57 smithi115 bash[46934]: cluster 2022-04-22T21:04:56.732164+0000 mgr.y (mgr.34107) 7478 : cluster [DBG] pgmap v8507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:59 smithi121 bash[36902]: cluster 2022-04-22T21:04:58.733338+0000 mgr.y (mgr.34107) 7479 : cluster [DBG] pgmap v8508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:04:59 smithi121 bash[36902]: audit 2022-04-22T21:04:59.055827+0000 mon.a (mon.0) 10961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:59 smithi115 bash[45495]: cluster 2022-04-22T21:04:58.733338+0000 mgr.y (mgr.34107) 7479 : cluster [DBG] pgmap v8508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:04:59 smithi115 bash[45495]: audit 2022-04-22T21:04:59.055827+0000 mon.a (mon.0) 10961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:59 smithi115 bash[46934]: cluster 2022-04-22T21:04:58.733338+0000 mgr.y (mgr.34107) 7479 : cluster [DBG] pgmap v8508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:04:59 smithi115 bash[46934]: audit 2022-04-22T21:04:59.055827+0000 mon.a (mon.0) 10961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:00 smithi121 bash[36902]: audit 2022-04-22T21:05:00.349758+0000 mon.a (mon.0) 10962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:01.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:00 smithi115 bash[45495]: audit 2022-04-22T21:05:00.349758+0000 mon.a (mon.0) 10962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:00 smithi115 bash[46934]: audit 2022-04-22T21:05:00.349758+0000 mon.a (mon.0) 10962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:01 smithi121 bash[36902]: cluster 2022-04-22T21:05:00.734414+0000 mgr.y (mgr.34107) 7480 : cluster [DBG] pgmap v8509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:02.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:01 smithi115 bash[45495]: cluster 2022-04-22T21:05:00.734414+0000 mgr.y (mgr.34107) 7480 : cluster [DBG] pgmap v8509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:02.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:01 smithi115 bash[46934]: cluster 2022-04-22T21:05:00.734414+0000 mgr.y (mgr.34107) 7480 : cluster [DBG] pgmap v8509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:02.464Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:03.046 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:05:03.385 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:05:03.385 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 81s ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:05:03.385 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:05:03.386 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 81s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:05:03.386 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:05:03.386 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:05:03.386 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 81s ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 81s ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 81s ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 81s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 81s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 81s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 81s ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 81s ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:05:03.387 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:05:03.388 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:05:03.388 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:05:03.388 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:05:03.388 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:05:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:05:03.792 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:05:03.792 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:05:03.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:03.793 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:05:03.793 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:05:03.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:03.793 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:05:03.793 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:05:03.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:03.794 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:05:03.794 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:05:03.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:03.794 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:05:03.795 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:05:03.795 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:05:03.795 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:05:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:03 smithi121 bash[36902]: cluster 2022-04-22T21:05:02.735544+0000 mgr.y (mgr.34107) 7481 : cluster [DBG] pgmap v8510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:03 smithi121 bash[36902]: audit 2022-04-22T21:05:03.025839+0000 mgr.y (mgr.34107) 7482 : audit [DBG] from='client.53077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:03 smithi121 bash[36902]: audit 2022-04-22T21:05:03.366625+0000 mgr.y (mgr.34107) 7483 : audit [DBG] from='client.43152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:03 smithi121 bash[36902]: audit 2022-04-22T21:05:03.786484+0000 mon.c (mon.1) 276 : audit [DBG] from='client.? 172.21.15.115:0/2104528726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[45495]: cluster 2022-04-22T21:05:02.735544+0000 mgr.y (mgr.34107) 7481 : cluster [DBG] pgmap v8510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[45495]: audit 2022-04-22T21:05:03.025839+0000 mgr.y (mgr.34107) 7482 : audit [DBG] from='client.53077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[45495]: audit 2022-04-22T21:05:03.366625+0000 mgr.y (mgr.34107) 7483 : audit [DBG] from='client.43152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[45495]: audit 2022-04-22T21:05:03.786484+0000 mon.c (mon.1) 276 : audit [DBG] from='client.? 172.21.15.115:0/2104528726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[46934]: cluster 2022-04-22T21:05:02.735544+0000 mgr.y (mgr.34107) 7481 : cluster [DBG] pgmap v8510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:04.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[46934]: audit 2022-04-22T21:05:03.025839+0000 mgr.y (mgr.34107) 7482 : audit [DBG] from='client.53077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[46934]: audit 2022-04-22T21:05:03.366625+0000 mgr.y (mgr.34107) 7483 : audit [DBG] from='client.43152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:04.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:03 smithi115 bash[46934]: audit 2022-04-22T21:05:03.786484+0000 mon.c (mon.1) 276 : audit [DBG] from='client.? 172.21.15.115:0/2104528726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:04.340Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:04 smithi121 bash[36902]: audit 2022-04-22T21:05:04.059349+0000 mon.a (mon.0) 10963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:05.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:04 smithi115 bash[45495]: audit 2022-04-22T21:05:04.059349+0000 mon.a (mon.0) 10963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:05.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:04 smithi115 bash[46934]: audit 2022-04-22T21:05:04.059349+0000 mon.a (mon.0) 10963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:05 smithi121 bash[36902]: cluster 2022-04-22T21:05:04.736791+0000 mgr.y (mgr.34107) 7484 : cluster [DBG] pgmap v8511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:05 smithi121 bash[36902]: audit 2022-04-22T21:05:05.352576+0000 mon.a (mon.0) 10964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:05 smithi115 bash[45495]: cluster 2022-04-22T21:05:04.736791+0000 mgr.y (mgr.34107) 7484 : cluster [DBG] pgmap v8511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:05 smithi115 bash[45495]: audit 2022-04-22T21:05:05.352576+0000 mon.a (mon.0) 10964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:05 smithi115 bash[46934]: cluster 2022-04-22T21:05:04.736791+0000 mgr.y (mgr.34107) 7484 : cluster [DBG] pgmap v8511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:05 smithi115 bash[46934]: audit 2022-04-22T21:05:05.352576+0000 mon.a (mon.0) 10964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:07 smithi121 bash[36902]: cluster 2022-04-22T21:05:06.737962+0000 mgr.y (mgr.34107) 7485 : cluster [DBG] pgmap v8512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:08.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:07 smithi115 bash[45495]: cluster 2022-04-22T21:05:06.737962+0000 mgr.y (mgr.34107) 7485 : cluster [DBG] pgmap v8512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:07 smithi115 bash[46934]: cluster 2022-04-22T21:05:06.737962+0000 mgr.y (mgr.34107) 7485 : cluster [DBG] pgmap v8512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:09 smithi121 bash[36902]: cluster 2022-04-22T21:05:08.739124+0000 mgr.y (mgr.34107) 7486 : cluster [DBG] pgmap v8513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:09 smithi121 bash[36902]: audit 2022-04-22T21:05:09.062909+0000 mon.a (mon.0) 10965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:09 smithi115 bash[45495]: cluster 2022-04-22T21:05:08.739124+0000 mgr.y (mgr.34107) 7486 : cluster [DBG] pgmap v8513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:09 smithi115 bash[45495]: audit 2022-04-22T21:05:09.062909+0000 mon.a (mon.0) 10965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:10.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:09 smithi115 bash[46934]: cluster 2022-04-22T21:05:08.739124+0000 mgr.y (mgr.34107) 7486 : cluster [DBG] pgmap v8513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:10.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:09 smithi115 bash[46934]: audit 2022-04-22T21:05:09.062909+0000 mon.a (mon.0) 10965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:10 smithi121 bash[36902]: audit 2022-04-22T21:05:10.355602+0000 mon.a (mon.0) 10966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:10 smithi115 bash[46934]: audit 2022-04-22T21:05:10.355602+0000 mon.a (mon.0) 10966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:10 smithi115 bash[45495]: audit 2022-04-22T21:05:10.355602+0000 mon.a (mon.0) 10966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:11 smithi121 bash[36902]: cluster 2022-04-22T21:05:10.740250+0000 mgr.y (mgr.34107) 7487 : cluster [DBG] pgmap v8514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:12.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:11 smithi115 bash[45495]: cluster 2022-04-22T21:05:10.740250+0000 mgr.y (mgr.34107) 7487 : cluster [DBG] pgmap v8514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:11 smithi115 bash[46934]: cluster 2022-04-22T21:05:10.740250+0000 mgr.y (mgr.34107) 7487 : cluster [DBG] pgmap v8514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:12.464Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:13 smithi121 bash[36902]: cluster 2022-04-22T21:05:12.741412+0000 mgr.y (mgr.34107) 7488 : cluster [DBG] pgmap v8515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:13 smithi115 bash[45495]: cluster 2022-04-22T21:05:12.741412+0000 mgr.y (mgr.34107) 7488 : cluster [DBG] pgmap v8515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:13 smithi115 bash[46934]: cluster 2022-04-22T21:05:12.741412+0000 mgr.y (mgr.34107) 7488 : cluster [DBG] pgmap v8515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:14.341Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:14 smithi121 bash[36902]: audit 2022-04-22T21:05:14.066534+0000 mon.a (mon.0) 10967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:15.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:14 smithi115 bash[45495]: audit 2022-04-22T21:05:14.066534+0000 mon.a (mon.0) 10967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:14 smithi115 bash[46934]: audit 2022-04-22T21:05:14.066534+0000 mon.a (mon.0) 10967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:15 smithi121 bash[36902]: cluster 2022-04-22T21:05:14.742424+0000 mgr.y (mgr.34107) 7489 : cluster [DBG] pgmap v8516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:15 smithi121 bash[36902]: audit 2022-04-22T21:05:15.358582+0000 mon.a (mon.0) 10968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:15 smithi121 bash[36902]: audit 2022-04-22T21:05:15.794055+0000 mon.a (mon.0) 10969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:05:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:15 smithi121 bash[36902]: audit 2022-04-22T21:05:15.795356+0000 mon.a (mon.0) 10970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:05:16.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[45495]: cluster 2022-04-22T21:05:14.742424+0000 mgr.y (mgr.34107) 7489 : cluster [DBG] pgmap v8516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[45495]: audit 2022-04-22T21:05:15.358582+0000 mon.a (mon.0) 10968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[45495]: audit 2022-04-22T21:05:15.794055+0000 mon.a (mon.0) 10969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[45495]: audit 2022-04-22T21:05:15.795356+0000 mon.a (mon.0) 10970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[46934]: cluster 2022-04-22T21:05:14.742424+0000 mgr.y (mgr.34107) 7489 : cluster [DBG] pgmap v8516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[46934]: audit 2022-04-22T21:05:15.358582+0000 mon.a (mon.0) 10968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[46934]: audit 2022-04-22T21:05:15.794055+0000 mon.a (mon.0) 10969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:05:16.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:15 smithi115 bash[46934]: audit 2022-04-22T21:05:15.795356+0000 mon.a (mon.0) 10970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:05:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:17 smithi121 bash[36902]: cluster 2022-04-22T21:05:16.743564+0000 mgr.y (mgr.34107) 7490 : cluster [DBG] pgmap v8517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:17 smithi115 bash[45495]: cluster 2022-04-22T21:05:16.743564+0000 mgr.y (mgr.34107) 7490 : cluster [DBG] pgmap v8517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:17 smithi115 bash[46934]: cluster 2022-04-22T21:05:16.743564+0000 mgr.y (mgr.34107) 7490 : cluster [DBG] pgmap v8517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:19 smithi121 bash[36902]: cluster 2022-04-22T21:05:18.744637+0000 mgr.y (mgr.34107) 7491 : cluster [DBG] pgmap v8518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:19 smithi121 bash[36902]: audit 2022-04-22T21:05:19.070228+0000 mon.a (mon.0) 10971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:19 smithi115 bash[45495]: cluster 2022-04-22T21:05:18.744637+0000 mgr.y (mgr.34107) 7491 : cluster [DBG] pgmap v8518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:20.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:19 smithi115 bash[45495]: audit 2022-04-22T21:05:19.070228+0000 mon.a (mon.0) 10971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:20.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:19 smithi115 bash[46934]: cluster 2022-04-22T21:05:18.744637+0000 mgr.y (mgr.34107) 7491 : cluster [DBG] pgmap v8518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:20.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:19 smithi115 bash[46934]: audit 2022-04-22T21:05:19.070228+0000 mon.a (mon.0) 10971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:20 smithi121 bash[36902]: audit 2022-04-22T21:05:20.361615+0000 mon.a (mon.0) 10972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:21.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:20 smithi115 bash[45495]: audit 2022-04-22T21:05:20.361615+0000 mon.a (mon.0) 10972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:21.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:20 smithi115 bash[46934]: audit 2022-04-22T21:05:20.361615+0000 mon.a (mon.0) 10972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:21 smithi121 bash[36902]: cluster 2022-04-22T21:05:20.745758+0000 mgr.y (mgr.34107) 7492 : cluster [DBG] pgmap v8519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:22.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:21 smithi115 bash[45495]: cluster 2022-04-22T21:05:20.745758+0000 mgr.y (mgr.34107) 7492 : cluster [DBG] pgmap v8519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:21 smithi115 bash[46934]: cluster 2022-04-22T21:05:20.745758+0000 mgr.y (mgr.34107) 7492 : cluster [DBG] pgmap v8519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:22.465Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:23 smithi121 bash[36902]: cluster 2022-04-22T21:05:22.747154+0000 mgr.y (mgr.34107) 7493 : cluster [DBG] pgmap v8520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:24.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:23 smithi115 bash[45495]: cluster 2022-04-22T21:05:22.747154+0000 mgr.y (mgr.34107) 7493 : cluster [DBG] pgmap v8520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:23 smithi115 bash[46934]: cluster 2022-04-22T21:05:22.747154+0000 mgr.y (mgr.34107) 7493 : cluster [DBG] pgmap v8520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:24.341Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:24 smithi121 bash[36902]: audit 2022-04-22T21:05:24.073948+0000 mon.a (mon.0) 10973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:24 smithi115 bash[46934]: audit 2022-04-22T21:05:24.073948+0000 mon.a (mon.0) 10973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:25.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:24 smithi115 bash[45495]: audit 2022-04-22T21:05:24.073948+0000 mon.a (mon.0) 10973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:25 smithi121 bash[36902]: cluster 2022-04-22T21:05:24.748287+0000 mgr.y (mgr.34107) 7494 : cluster [DBG] pgmap v8521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:25 smithi121 bash[36902]: audit 2022-04-22T21:05:25.364376+0000 mon.a (mon.0) 10974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:25 smithi115 bash[45495]: cluster 2022-04-22T21:05:24.748287+0000 mgr.y (mgr.34107) 7494 : cluster [DBG] pgmap v8521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:25 smithi115 bash[45495]: audit 2022-04-22T21:05:25.364376+0000 mon.a (mon.0) 10974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:25 smithi115 bash[46934]: cluster 2022-04-22T21:05:24.748287+0000 mgr.y (mgr.34107) 7494 : cluster [DBG] pgmap v8521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:25 smithi115 bash[46934]: audit 2022-04-22T21:05:25.364376+0000 mon.a (mon.0) 10974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:27 smithi121 bash[36902]: cluster 2022-04-22T21:05:26.749352+0000 mgr.y (mgr.34107) 7495 : cluster [DBG] pgmap v8522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:27 smithi115 bash[45495]: cluster 2022-04-22T21:05:26.749352+0000 mgr.y (mgr.34107) 7495 : cluster [DBG] pgmap v8522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:27 smithi115 bash[46934]: cluster 2022-04-22T21:05:26.749352+0000 mgr.y (mgr.34107) 7495 : cluster [DBG] pgmap v8522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:29 smithi121 bash[36902]: cluster 2022-04-22T21:05:28.750440+0000 mgr.y (mgr.34107) 7496 : cluster [DBG] pgmap v8523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:30.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:29 smithi121 bash[36902]: audit 2022-04-22T21:05:29.077679+0000 mon.a (mon.0) 10975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:29 smithi115 bash[45495]: cluster 2022-04-22T21:05:28.750440+0000 mgr.y (mgr.34107) 7496 : cluster [DBG] pgmap v8523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:29 smithi115 bash[45495]: audit 2022-04-22T21:05:29.077679+0000 mon.a (mon.0) 10975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:29 smithi115 bash[46934]: cluster 2022-04-22T21:05:28.750440+0000 mgr.y (mgr.34107) 7496 : cluster [DBG] pgmap v8523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:29 smithi115 bash[46934]: audit 2022-04-22T21:05:29.077679+0000 mon.a (mon.0) 10975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:30 smithi121 bash[36902]: audit 2022-04-22T21:05:30.367244+0000 mon.a (mon.0) 10976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:30 smithi115 bash[45495]: audit 2022-04-22T21:05:30.367244+0000 mon.a (mon.0) 10976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:30 smithi115 bash[46934]: audit 2022-04-22T21:05:30.367244+0000 mon.a (mon.0) 10976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:31 smithi121 bash[36902]: cluster 2022-04-22T21:05:30.751549+0000 mgr.y (mgr.34107) 7497 : cluster [DBG] pgmap v8524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:32.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:31 smithi115 bash[45495]: cluster 2022-04-22T21:05:30.751549+0000 mgr.y (mgr.34107) 7497 : cluster [DBG] pgmap v8524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:31 smithi115 bash[46934]: cluster 2022-04-22T21:05:30.751549+0000 mgr.y (mgr.34107) 7497 : cluster [DBG] pgmap v8524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:32.468Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:34.168 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:05:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:33 smithi121 bash[36902]: cluster 2022-04-22T21:05:32.752599+0000 mgr.y (mgr.34107) 7498 : cluster [DBG] pgmap v8525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:33 smithi115 bash[46934]: cluster 2022-04-22T21:05:32.752599+0000 mgr.y (mgr.34107) 7498 : cluster [DBG] pgmap v8525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:33 smithi115 bash[45495]: cluster 2022-04-22T21:05:32.752599+0000 mgr.y (mgr.34107) 7498 : cluster [DBG] pgmap v8525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:05:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 112s ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:05:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:05:34.522 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 112s ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 112s ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 112s ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 112s ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 112s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 112s ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:05:34.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 112s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 112s ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 112s ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:05:34.524 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:05:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:34.342Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:05:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:05:34.937 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:05:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:34 smithi121 bash[36902]: audit 2022-04-22T21:05:34.081360+0000 mon.a (mon.0) 10977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:34 smithi121 bash[36902]: audit 2022-04-22T21:05:34.148426+0000 mgr.y (mgr.34107) 7499 : audit [DBG] from='client.43164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:34 smithi121 bash[36902]: audit 2022-04-22T21:05:34.497328+0000 mgr.y (mgr.34107) 7500 : audit [DBG] from='client.53101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:34 smithi121 bash[36902]: audit 2022-04-22T21:05:34.931092+0000 mon.c (mon.1) 277 : audit [DBG] from='client.? 172.21.15.115:0/3936375686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[45495]: audit 2022-04-22T21:05:34.081360+0000 mon.a (mon.0) 10977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[45495]: audit 2022-04-22T21:05:34.148426+0000 mgr.y (mgr.34107) 7499 : audit [DBG] from='client.43164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[45495]: audit 2022-04-22T21:05:34.497328+0000 mgr.y (mgr.34107) 7500 : audit [DBG] from='client.53101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[45495]: audit 2022-04-22T21:05:34.931092+0000 mon.c (mon.1) 277 : audit [DBG] from='client.? 172.21.15.115:0/3936375686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[46934]: audit 2022-04-22T21:05:34.081360+0000 mon.a (mon.0) 10977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:35.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[46934]: audit 2022-04-22T21:05:34.148426+0000 mgr.y (mgr.34107) 7499 : audit [DBG] from='client.43164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[46934]: audit 2022-04-22T21:05:34.497328+0000 mgr.y (mgr.34107) 7500 : audit [DBG] from='client.53101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:05:35.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:34 smithi115 bash[46934]: audit 2022-04-22T21:05:34.931092+0000 mon.c (mon.1) 277 : audit [DBG] from='client.? 172.21.15.115:0/3936375686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:05:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:36 smithi121 bash[36902]: cluster 2022-04-22T21:05:34.753626+0000 mgr.y (mgr.34107) 7501 : cluster [DBG] pgmap v8526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:36 smithi121 bash[36902]: audit 2022-04-22T21:05:35.370243+0000 mon.a (mon.0) 10978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:35 smithi115 bash[45495]: cluster 2022-04-22T21:05:34.753626+0000 mgr.y (mgr.34107) 7501 : cluster [DBG] pgmap v8526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:35 smithi115 bash[45495]: audit 2022-04-22T21:05:35.370243+0000 mon.a (mon.0) 10978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:35 smithi115 bash[46934]: cluster 2022-04-22T21:05:34.753626+0000 mgr.y (mgr.34107) 7501 : cluster [DBG] pgmap v8526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:35 smithi115 bash[46934]: audit 2022-04-22T21:05:35.370243+0000 mon.a (mon.0) 10978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:38 smithi121 bash[36902]: cluster 2022-04-22T21:05:36.754715+0000 mgr.y (mgr.34107) 7502 : cluster [DBG] pgmap v8527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:38 smithi115 bash[45495]: cluster 2022-04-22T21:05:36.754715+0000 mgr.y (mgr.34107) 7502 : cluster [DBG] pgmap v8527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:38 smithi115 bash[46934]: cluster 2022-04-22T21:05:36.754715+0000 mgr.y (mgr.34107) 7502 : cluster [DBG] pgmap v8527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:40 smithi121 bash[36902]: cluster 2022-04-22T21:05:38.755714+0000 mgr.y (mgr.34107) 7503 : cluster [DBG] pgmap v8528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:40 smithi121 bash[36902]: audit 2022-04-22T21:05:39.085071+0000 mon.a (mon.0) 10979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:40 smithi115 bash[45495]: cluster 2022-04-22T21:05:38.755714+0000 mgr.y (mgr.34107) 7503 : cluster [DBG] pgmap v8528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:40 smithi115 bash[45495]: audit 2022-04-22T21:05:39.085071+0000 mon.a (mon.0) 10979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:40 smithi115 bash[46934]: cluster 2022-04-22T21:05:38.755714+0000 mgr.y (mgr.34107) 7503 : cluster [DBG] pgmap v8528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:40 smithi115 bash[46934]: audit 2022-04-22T21:05:39.085071+0000 mon.a (mon.0) 10979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:41 smithi121 bash[36902]: audit 2022-04-22T21:05:40.373284+0000 mon.a (mon.0) 10980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:41 smithi115 bash[45495]: audit 2022-04-22T21:05:40.373284+0000 mon.a (mon.0) 10980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:41 smithi115 bash[46934]: audit 2022-04-22T21:05:40.373284+0000 mon.a (mon.0) 10980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:42.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:42 smithi115 bash[45495]: cluster 2022-04-22T21:05:40.756892+0000 mgr.y (mgr.34107) 7504 : cluster [DBG] pgmap v8529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:42 smithi115 bash[46934]: cluster 2022-04-22T21:05:40.756892+0000 mgr.y (mgr.34107) 7504 : cluster [DBG] pgmap v8529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:42 smithi121 bash[36902]: cluster 2022-04-22T21:05:40.756892+0000 mgr.y (mgr.34107) 7504 : cluster [DBG] pgmap v8529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:42.468Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:43 smithi121 bash[36902]: audit 2022-04-22T21:05:42.494337+0000 mon.a (mon.0) 10981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:05:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:43 smithi121 bash[36902]: audit 2022-04-22T21:05:42.497225+0000 mon.a (mon.0) 10982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:05:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:43 smithi121 bash[36902]: audit 2022-04-22T21:05:42.498869+0000 mon.a (mon.0) 10983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:05:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:43 smithi121 bash[36902]: audit 2022-04-22T21:05:42.861591+0000 mon.a (mon.0) 10984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:05:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[45495]: audit 2022-04-22T21:05:42.494337+0000 mon.a (mon.0) 10981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:05:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[45495]: audit 2022-04-22T21:05:42.497225+0000 mon.a (mon.0) 10982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:05:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[45495]: audit 2022-04-22T21:05:42.498869+0000 mon.a (mon.0) 10983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:05:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[45495]: audit 2022-04-22T21:05:42.861591+0000 mon.a (mon.0) 10984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:05:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[46934]: audit 2022-04-22T21:05:42.494337+0000 mon.a (mon.0) 10981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:05:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[46934]: audit 2022-04-22T21:05:42.497225+0000 mon.a (mon.0) 10982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:05:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[46934]: audit 2022-04-22T21:05:42.498869+0000 mon.a (mon.0) 10983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:05:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:43 smithi115 bash[46934]: audit 2022-04-22T21:05:42.861591+0000 mon.a (mon.0) 10984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:05:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:44 smithi115 bash[45495]: cluster 2022-04-22T21:05:42.757847+0000 mgr.y (mgr.34107) 7505 : cluster [DBG] pgmap v8530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:44 smithi115 bash[45495]: audit 2022-04-22T21:05:43.358765+0000 mon.a (mon.0) 10985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:44 smithi115 bash[46934]: cluster 2022-04-22T21:05:42.757847+0000 mgr.y (mgr.34107) 7505 : cluster [DBG] pgmap v8530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:44 smithi115 bash[46934]: audit 2022-04-22T21:05:43.358765+0000 mon.a (mon.0) 10985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:44 smithi121 bash[36902]: cluster 2022-04-22T21:05:42.757847+0000 mgr.y (mgr.34107) 7505 : cluster [DBG] pgmap v8530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:44 smithi121 bash[36902]: audit 2022-04-22T21:05:43.358765+0000 mon.a (mon.0) 10985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:44.342Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:45 smithi121 bash[36902]: audit 2022-04-22T21:05:44.088676+0000 mon.a (mon.0) 10986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:45 smithi115 bash[45495]: audit 2022-04-22T21:05:44.088676+0000 mon.a (mon.0) 10986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:45 smithi115 bash[46934]: audit 2022-04-22T21:05:44.088676+0000 mon.a (mon.0) 10986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:46 smithi121 bash[36902]: cluster 2022-04-22T21:05:44.758944+0000 mgr.y (mgr.34107) 7506 : cluster [DBG] pgmap v8531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:46 smithi121 bash[36902]: audit 2022-04-22T21:05:45.376321+0000 mon.a (mon.0) 10987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:46 smithi115 bash[45495]: cluster 2022-04-22T21:05:44.758944+0000 mgr.y (mgr.34107) 7506 : cluster [DBG] pgmap v8531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:46 smithi115 bash[45495]: audit 2022-04-22T21:05:45.376321+0000 mon.a (mon.0) 10987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:46 smithi115 bash[46934]: cluster 2022-04-22T21:05:44.758944+0000 mgr.y (mgr.34107) 7506 : cluster [DBG] pgmap v8531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:46 smithi115 bash[46934]: audit 2022-04-22T21:05:45.376321+0000 mon.a (mon.0) 10987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:48 smithi121 bash[36902]: cluster 2022-04-22T21:05:46.759691+0000 mgr.y (mgr.34107) 7507 : cluster [DBG] pgmap v8532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:48 smithi115 bash[45495]: cluster 2022-04-22T21:05:46.759691+0000 mgr.y (mgr.34107) 7507 : cluster [DBG] pgmap v8532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:48 smithi115 bash[46934]: cluster 2022-04-22T21:05:46.759691+0000 mgr.y (mgr.34107) 7507 : cluster [DBG] pgmap v8532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:49 smithi121 bash[36902]: audit 2022-04-22T21:05:48.361751+0000 mon.a (mon.0) 10988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:49 smithi115 bash[45495]: audit 2022-04-22T21:05:48.361751+0000 mon.a (mon.0) 10988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:49.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:49 smithi115 bash[46934]: audit 2022-04-22T21:05:48.361751+0000 mon.a (mon.0) 10988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:50 smithi121 bash[36902]: cluster 2022-04-22T21:05:48.760758+0000 mgr.y (mgr.34107) 7508 : cluster [DBG] pgmap v8533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:50 smithi121 bash[36902]: audit 2022-04-22T21:05:49.092420+0000 mon.a (mon.0) 10989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:50 smithi115 bash[45495]: cluster 2022-04-22T21:05:48.760758+0000 mgr.y (mgr.34107) 7508 : cluster [DBG] pgmap v8533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:50 smithi115 bash[45495]: audit 2022-04-22T21:05:49.092420+0000 mon.a (mon.0) 10989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:50 smithi115 bash[46934]: cluster 2022-04-22T21:05:48.760758+0000 mgr.y (mgr.34107) 7508 : cluster [DBG] pgmap v8533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:50 smithi115 bash[46934]: audit 2022-04-22T21:05:49.092420+0000 mon.a (mon.0) 10989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:51 smithi121 bash[36902]: audit 2022-04-22T21:05:50.379210+0000 mon.a (mon.0) 10990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:51 smithi115 bash[45495]: audit 2022-04-22T21:05:50.379210+0000 mon.a (mon.0) 10990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:51 smithi115 bash[46934]: audit 2022-04-22T21:05:50.379210+0000 mon.a (mon.0) 10990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:52.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:52 smithi115 bash[45495]: cluster 2022-04-22T21:05:50.761871+0000 mgr.y (mgr.34107) 7509 : cluster [DBG] pgmap v8534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:52.473 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:52 smithi115 bash[46934]: cluster 2022-04-22T21:05:50.761871+0000 mgr.y (mgr.34107) 7509 : cluster [DBG] pgmap v8534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:52 smithi121 bash[36902]: cluster 2022-04-22T21:05:50.761871+0000 mgr.y (mgr.34107) 7509 : cluster [DBG] pgmap v8534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:52.468Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:54.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:54 smithi115 bash[45495]: cluster 2022-04-22T21:05:52.762958+0000 mgr.y (mgr.34107) 7510 : cluster [DBG] pgmap v8535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:54.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:54 smithi115 bash[45495]: audit 2022-04-22T21:05:53.364747+0000 mon.a (mon.0) 10991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:54 smithi115 bash[46934]: cluster 2022-04-22T21:05:52.762958+0000 mgr.y (mgr.34107) 7510 : cluster [DBG] pgmap v8535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:54 smithi115 bash[46934]: audit 2022-04-22T21:05:53.364747+0000 mon.a (mon.0) 10991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:54 smithi121 bash[36902]: cluster 2022-04-22T21:05:52.762958+0000 mgr.y (mgr.34107) 7510 : cluster [DBG] pgmap v8535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:54 smithi121 bash[36902]: audit 2022-04-22T21:05:53.364747+0000 mon.a (mon.0) 10991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:05:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:05:54.343Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:05:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:55 smithi121 bash[36902]: audit 2022-04-22T21:05:54.096029+0000 mon.a (mon.0) 10992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:55.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:55 smithi115 bash[45495]: audit 2022-04-22T21:05:54.096029+0000 mon.a (mon.0) 10992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:55 smithi115 bash[46934]: audit 2022-04-22T21:05:54.096029+0000 mon.a (mon.0) 10992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:05:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:56 smithi121 bash[36902]: cluster 2022-04-22T21:05:54.763697+0000 mgr.y (mgr.34107) 7511 : cluster [DBG] pgmap v8536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:56 smithi121 bash[36902]: audit 2022-04-22T21:05:55.382155+0000 mon.a (mon.0) 10993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:56 smithi115 bash[45495]: cluster 2022-04-22T21:05:54.763697+0000 mgr.y (mgr.34107) 7511 : cluster [DBG] pgmap v8536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:56 smithi115 bash[45495]: audit 2022-04-22T21:05:55.382155+0000 mon.a (mon.0) 10993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:56 smithi115 bash[46934]: cluster 2022-04-22T21:05:54.763697+0000 mgr.y (mgr.34107) 7511 : cluster [DBG] pgmap v8536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:56 smithi115 bash[46934]: audit 2022-04-22T21:05:55.382155+0000 mon.a (mon.0) 10993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:05:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:58 smithi121 bash[36902]: cluster 2022-04-22T21:05:56.764725+0000 mgr.y (mgr.34107) 7512 : cluster [DBG] pgmap v8537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:58 smithi115 bash[45495]: cluster 2022-04-22T21:05:56.764725+0000 mgr.y (mgr.34107) 7512 : cluster [DBG] pgmap v8537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:58 smithi115 bash[46934]: cluster 2022-04-22T21:05:56.764725+0000 mgr.y (mgr.34107) 7512 : cluster [DBG] pgmap v8537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:05:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:05:59 smithi121 bash[36902]: audit 2022-04-22T21:05:58.367852+0000 mon.a (mon.0) 10994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:59.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:05:59 smithi115 bash[45495]: audit 2022-04-22T21:05:58.367852+0000 mon.a (mon.0) 10994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:05:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:05:59 smithi115 bash[46934]: audit 2022-04-22T21:05:58.367852+0000 mon.a (mon.0) 10994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:00 smithi121 bash[36902]: cluster 2022-04-22T21:05:58.765793+0000 mgr.y (mgr.34107) 7513 : cluster [DBG] pgmap v8538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:00 smithi121 bash[36902]: audit 2022-04-22T21:05:59.104212+0000 mon.a (mon.0) 10995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:00 smithi115 bash[46934]: cluster 2022-04-22T21:05:58.765793+0000 mgr.y (mgr.34107) 7513 : cluster [DBG] pgmap v8538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:00 smithi115 bash[46934]: audit 2022-04-22T21:05:59.104212+0000 mon.a (mon.0) 10995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:00 smithi115 bash[45495]: cluster 2022-04-22T21:05:58.765793+0000 mgr.y (mgr.34107) 7513 : cluster [DBG] pgmap v8538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:00 smithi115 bash[45495]: audit 2022-04-22T21:05:59.104212+0000 mon.a (mon.0) 10995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:01 smithi121 bash[36902]: audit 2022-04-22T21:06:00.385150+0000 mon.a (mon.0) 10996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:01 smithi115 bash[45495]: audit 2022-04-22T21:06:00.385150+0000 mon.a (mon.0) 10996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:01 smithi115 bash[46934]: audit 2022-04-22T21:06:00.385150+0000 mon.a (mon.0) 10996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:02.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:02 smithi115 bash[45495]: cluster 2022-04-22T21:06:00.766901+0000 mgr.y (mgr.34107) 7514 : cluster [DBG] pgmap v8539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:02.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:02 smithi115 bash[46934]: cluster 2022-04-22T21:06:00.766901+0000 mgr.y (mgr.34107) 7514 : cluster [DBG] pgmap v8539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:02 smithi121 bash[36902]: cluster 2022-04-22T21:06:00.766901+0000 mgr.y (mgr.34107) 7514 : cluster [DBG] pgmap v8539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:02.469Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:04 smithi121 bash[36902]: cluster 2022-04-22T21:06:02.768025+0000 mgr.y (mgr.34107) 7515 : cluster [DBG] pgmap v8540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:04 smithi121 bash[36902]: audit 2022-04-22T21:06:03.371236+0000 mon.a (mon.0) 10997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:04 smithi121 bash[36902]: audit 2022-04-22T21:06:04.103212+0000 mon.a (mon.0) 10998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:04.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:04.343Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[45495]: cluster 2022-04-22T21:06:02.768025+0000 mgr.y (mgr.34107) 7515 : cluster [DBG] pgmap v8540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[45495]: audit 2022-04-22T21:06:03.371236+0000 mon.a (mon.0) 10997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[45495]: audit 2022-04-22T21:06:04.103212+0000 mon.a (mon.0) 10998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[46934]: cluster 2022-04-22T21:06:02.768025+0000 mgr.y (mgr.34107) 7515 : cluster [DBG] pgmap v8540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[46934]: audit 2022-04-22T21:06:03.371236+0000 mon.a (mon.0) 10997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:04 smithi115 bash[46934]: audit 2022-04-22T21:06:04.103212+0000 mon.a (mon.0) 10998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:05.315 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:06:05.663 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:06:05.663 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:06:05.663 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:06:05.664 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:06:05.665 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:06:06.069 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:06:06.070 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:06:06.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:06:06.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:06:06.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:06.072 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:06:06.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:06:06.072 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:06:06.072 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:06:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:06 smithi121 bash[36902]: cluster 2022-04-22T21:06:04.769108+0000 mgr.y (mgr.34107) 7516 : cluster [DBG] pgmap v8541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:06 smithi121 bash[36902]: audit 2022-04-22T21:06:05.295532+0000 mgr.y (mgr.34107) 7517 : audit [DBG] from='client.53113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:06 smithi121 bash[36902]: audit 2022-04-22T21:06:05.388177+0000 mon.a (mon.0) 10999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:06 smithi121 bash[36902]: audit 2022-04-22T21:06:05.637640+0000 mgr.y (mgr.34107) 7518 : audit [DBG] from='client.53119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:06 smithi121 bash[36902]: audit 2022-04-22T21:06:06.064856+0000 mon.a (mon.0) 11000 : audit [DBG] from='client.? 172.21.15.115:0/1773603289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[45495]: cluster 2022-04-22T21:06:04.769108+0000 mgr.y (mgr.34107) 7516 : cluster [DBG] pgmap v8541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[45495]: audit 2022-04-22T21:06:05.295532+0000 mgr.y (mgr.34107) 7517 : audit [DBG] from='client.53113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[45495]: audit 2022-04-22T21:06:05.388177+0000 mon.a (mon.0) 10999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[45495]: audit 2022-04-22T21:06:05.637640+0000 mgr.y (mgr.34107) 7518 : audit [DBG] from='client.53119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[45495]: audit 2022-04-22T21:06:06.064856+0000 mon.a (mon.0) 11000 : audit [DBG] from='client.? 172.21.15.115:0/1773603289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[46934]: cluster 2022-04-22T21:06:04.769108+0000 mgr.y (mgr.34107) 7516 : cluster [DBG] pgmap v8541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[46934]: audit 2022-04-22T21:06:05.295532+0000 mgr.y (mgr.34107) 7517 : audit [DBG] from='client.53113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[46934]: audit 2022-04-22T21:06:05.388177+0000 mon.a (mon.0) 10999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[46934]: audit 2022-04-22T21:06:05.637640+0000 mgr.y (mgr.34107) 7518 : audit [DBG] from='client.53119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:06.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:06 smithi115 bash[46934]: audit 2022-04-22T21:06:06.064856+0000 mon.a (mon.0) 11000 : audit [DBG] from='client.? 172.21.15.115:0/1773603289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:08 smithi121 bash[36902]: cluster 2022-04-22T21:06:06.770308+0000 mgr.y (mgr.34107) 7519 : cluster [DBG] pgmap v8542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:08 smithi115 bash[45495]: cluster 2022-04-22T21:06:06.770308+0000 mgr.y (mgr.34107) 7519 : cluster [DBG] pgmap v8542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:08.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:08 smithi115 bash[46934]: cluster 2022-04-22T21:06:06.770308+0000 mgr.y (mgr.34107) 7519 : cluster [DBG] pgmap v8542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:09 smithi121 bash[36902]: audit 2022-04-22T21:06:08.374393+0000 mon.a (mon.0) 11001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:09 smithi121 bash[36902]: audit 2022-04-22T21:06:09.106938+0000 mon.a (mon.0) 11002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:09 smithi115 bash[46934]: audit 2022-04-22T21:06:08.374393+0000 mon.a (mon.0) 11001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:09 smithi115 bash[46934]: audit 2022-04-22T21:06:09.106938+0000 mon.a (mon.0) 11002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:09 smithi115 bash[45495]: audit 2022-04-22T21:06:08.374393+0000 mon.a (mon.0) 11001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:09.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:09 smithi115 bash[45495]: audit 2022-04-22T21:06:09.106938+0000 mon.a (mon.0) 11002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:10.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:10 smithi121 bash[36902]: cluster 2022-04-22T21:06:08.771621+0000 mgr.y (mgr.34107) 7520 : cluster [DBG] pgmap v8543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:10 smithi115 bash[45495]: cluster 2022-04-22T21:06:08.771621+0000 mgr.y (mgr.34107) 7520 : cluster [DBG] pgmap v8543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:10 smithi115 bash[46934]: cluster 2022-04-22T21:06:08.771621+0000 mgr.y (mgr.34107) 7520 : cluster [DBG] pgmap v8543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:11 smithi121 bash[36902]: audit 2022-04-22T21:06:10.390988+0000 mon.a (mon.0) 11003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:11 smithi115 bash[45495]: audit 2022-04-22T21:06:10.390988+0000 mon.a (mon.0) 11003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:11 smithi115 bash[46934]: audit 2022-04-22T21:06:10.390988+0000 mon.a (mon.0) 11003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:12.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:12 smithi115 bash[45495]: cluster 2022-04-22T21:06:10.772798+0000 mgr.y (mgr.34107) 7521 : cluster [DBG] pgmap v8544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:12.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:12 smithi115 bash[46934]: cluster 2022-04-22T21:06:10.772798+0000 mgr.y (mgr.34107) 7521 : cluster [DBG] pgmap v8544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:12 smithi121 bash[36902]: cluster 2022-04-22T21:06:10.772798+0000 mgr.y (mgr.34107) 7521 : cluster [DBG] pgmap v8544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:12.470Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:14 smithi121 bash[36902]: cluster 2022-04-22T21:06:12.773876+0000 mgr.y (mgr.34107) 7522 : cluster [DBG] pgmap v8545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:14 smithi121 bash[36902]: audit 2022-04-22T21:06:13.377596+0000 mon.a (mon.0) 11004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:14 smithi121 bash[36902]: audit 2022-04-22T21:06:14.110557+0000 mon.a (mon.0) 11005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:14.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:14.347Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[45495]: cluster 2022-04-22T21:06:12.773876+0000 mgr.y (mgr.34107) 7522 : cluster [DBG] pgmap v8545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[45495]: audit 2022-04-22T21:06:13.377596+0000 mon.a (mon.0) 11004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[45495]: audit 2022-04-22T21:06:14.110557+0000 mon.a (mon.0) 11005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[46934]: cluster 2022-04-22T21:06:12.773876+0000 mgr.y (mgr.34107) 7522 : cluster [DBG] pgmap v8545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[46934]: audit 2022-04-22T21:06:13.377596+0000 mon.a (mon.0) 11004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:14 smithi115 bash[46934]: audit 2022-04-22T21:06:14.110557+0000 mon.a (mon.0) 11005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:16 smithi121 bash[36902]: cluster 2022-04-22T21:06:14.774957+0000 mgr.y (mgr.34107) 7523 : cluster [DBG] pgmap v8546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:16 smithi121 bash[36902]: audit 2022-04-22T21:06:15.393780+0000 mon.a (mon.0) 11006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:16 smithi121 bash[36902]: audit 2022-04-22T21:06:15.805674+0000 mon.a (mon.0) 11007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:16 smithi121 bash[36902]: audit 2022-04-22T21:06:15.805883+0000 mon.a (mon.0) 11008 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:06:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[45495]: cluster 2022-04-22T21:06:14.774957+0000 mgr.y (mgr.34107) 7523 : cluster [DBG] pgmap v8546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[45495]: audit 2022-04-22T21:06:15.393780+0000 mon.a (mon.0) 11006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[45495]: audit 2022-04-22T21:06:15.805674+0000 mon.a (mon.0) 11007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:06:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[45495]: audit 2022-04-22T21:06:15.805883+0000 mon.a (mon.0) 11008 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:06:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[46934]: cluster 2022-04-22T21:06:14.774957+0000 mgr.y (mgr.34107) 7523 : cluster [DBG] pgmap v8546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[46934]: audit 2022-04-22T21:06:15.393780+0000 mon.a (mon.0) 11006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[46934]: audit 2022-04-22T21:06:15.805674+0000 mon.a (mon.0) 11007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:06:16.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:16 smithi115 bash[46934]: audit 2022-04-22T21:06:15.805883+0000 mon.a (mon.0) 11008 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:06:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:18 smithi121 bash[36902]: cluster 2022-04-22T21:06:16.776063+0000 mgr.y (mgr.34107) 7524 : cluster [DBG] pgmap v8547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:18 smithi115 bash[45495]: cluster 2022-04-22T21:06:16.776063+0000 mgr.y (mgr.34107) 7524 : cluster [DBG] pgmap v8547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:18 smithi115 bash[46934]: cluster 2022-04-22T21:06:16.776063+0000 mgr.y (mgr.34107) 7524 : cluster [DBG] pgmap v8547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:19 smithi121 bash[36902]: audit 2022-04-22T21:06:18.380860+0000 mon.a (mon.0) 11009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:19 smithi121 bash[36902]: audit 2022-04-22T21:06:19.114322+0000 mon.a (mon.0) 11010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:19.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:19 smithi115 bash[45495]: audit 2022-04-22T21:06:18.380860+0000 mon.a (mon.0) 11009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:19.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:19 smithi115 bash[45495]: audit 2022-04-22T21:06:19.114322+0000 mon.a (mon.0) 11010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:19 smithi115 bash[46934]: audit 2022-04-22T21:06:18.380860+0000 mon.a (mon.0) 11009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:19 smithi115 bash[46934]: audit 2022-04-22T21:06:19.114322+0000 mon.a (mon.0) 11010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:20 smithi121 bash[36902]: cluster 2022-04-22T21:06:18.777131+0000 mgr.y (mgr.34107) 7525 : cluster [DBG] pgmap v8548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:20 smithi115 bash[45495]: cluster 2022-04-22T21:06:18.777131+0000 mgr.y (mgr.34107) 7525 : cluster [DBG] pgmap v8548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:20 smithi115 bash[46934]: cluster 2022-04-22T21:06:18.777131+0000 mgr.y (mgr.34107) 7525 : cluster [DBG] pgmap v8548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:21 smithi121 bash[36902]: audit 2022-04-22T21:06:20.396707+0000 mon.a (mon.0) 11011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:21 smithi115 bash[45495]: audit 2022-04-22T21:06:20.396707+0000 mon.a (mon.0) 11011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:21.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:21 smithi115 bash[46934]: audit 2022-04-22T21:06:20.396707+0000 mon.a (mon.0) 11011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:22.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:22 smithi115 bash[45495]: cluster 2022-04-22T21:06:20.778243+0000 mgr.y (mgr.34107) 7526 : cluster [DBG] pgmap v8549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:22.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:22 smithi115 bash[46934]: cluster 2022-04-22T21:06:20.778243+0000 mgr.y (mgr.34107) 7526 : cluster [DBG] pgmap v8549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:22 smithi121 bash[36902]: cluster 2022-04-22T21:06:20.778243+0000 mgr.y (mgr.34107) 7526 : cluster [DBG] pgmap v8549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:22.469Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:24 smithi121 bash[36902]: cluster 2022-04-22T21:06:22.779279+0000 mgr.y (mgr.34107) 7527 : cluster [DBG] pgmap v8550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:24 smithi121 bash[36902]: audit 2022-04-22T21:06:23.384005+0000 mon.a (mon.0) 11012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:24 smithi121 bash[36902]: audit 2022-04-22T21:06:24.118012+0000 mon.a (mon.0) 11013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:24.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:24.348Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[45495]: cluster 2022-04-22T21:06:22.779279+0000 mgr.y (mgr.34107) 7527 : cluster [DBG] pgmap v8550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[45495]: audit 2022-04-22T21:06:23.384005+0000 mon.a (mon.0) 11012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[45495]: audit 2022-04-22T21:06:24.118012+0000 mon.a (mon.0) 11013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[46934]: cluster 2022-04-22T21:06:22.779279+0000 mgr.y (mgr.34107) 7527 : cluster [DBG] pgmap v8550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[46934]: audit 2022-04-22T21:06:23.384005+0000 mon.a (mon.0) 11012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:24 smithi115 bash[46934]: audit 2022-04-22T21:06:24.118012+0000 mon.a (mon.0) 11013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:26 smithi121 bash[36902]: cluster 2022-04-22T21:06:24.780368+0000 mgr.y (mgr.34107) 7528 : cluster [DBG] pgmap v8551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:26 smithi121 bash[36902]: audit 2022-04-22T21:06:25.399467+0000 mon.a (mon.0) 11014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:26 smithi115 bash[45495]: cluster 2022-04-22T21:06:24.780368+0000 mgr.y (mgr.34107) 7528 : cluster [DBG] pgmap v8551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:26 smithi115 bash[45495]: audit 2022-04-22T21:06:25.399467+0000 mon.a (mon.0) 11014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:26 smithi115 bash[46934]: cluster 2022-04-22T21:06:24.780368+0000 mgr.y (mgr.34107) 7528 : cluster [DBG] pgmap v8551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:26 smithi115 bash[46934]: audit 2022-04-22T21:06:25.399467+0000 mon.a (mon.0) 11014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:28 smithi121 bash[36902]: cluster 2022-04-22T21:06:26.781477+0000 mgr.y (mgr.34107) 7529 : cluster [DBG] pgmap v8552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:28 smithi115 bash[45495]: cluster 2022-04-22T21:06:26.781477+0000 mgr.y (mgr.34107) 7529 : cluster [DBG] pgmap v8552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:28 smithi115 bash[46934]: cluster 2022-04-22T21:06:26.781477+0000 mgr.y (mgr.34107) 7529 : cluster [DBG] pgmap v8552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:29 smithi121 bash[36902]: audit 2022-04-22T21:06:28.387117+0000 mon.a (mon.0) 11015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:29 smithi121 bash[36902]: audit 2022-04-22T21:06:29.121763+0000 mon.a (mon.0) 11016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:29 smithi115 bash[45495]: audit 2022-04-22T21:06:28.387117+0000 mon.a (mon.0) 11015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:29 smithi115 bash[45495]: audit 2022-04-22T21:06:29.121763+0000 mon.a (mon.0) 11016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:29.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:29 smithi115 bash[46934]: audit 2022-04-22T21:06:28.387117+0000 mon.a (mon.0) 11015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:29.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:29 smithi115 bash[46934]: audit 2022-04-22T21:06:29.121763+0000 mon.a (mon.0) 11016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:30 smithi121 bash[36902]: cluster 2022-04-22T21:06:28.782736+0000 mgr.y (mgr.34107) 7530 : cluster [DBG] pgmap v8553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:30 smithi115 bash[45495]: cluster 2022-04-22T21:06:28.782736+0000 mgr.y (mgr.34107) 7530 : cluster [DBG] pgmap v8553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:30 smithi115 bash[46934]: cluster 2022-04-22T21:06:28.782736+0000 mgr.y (mgr.34107) 7530 : cluster [DBG] pgmap v8553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:31 smithi121 bash[36902]: audit 2022-04-22T21:06:30.402503+0000 mon.a (mon.0) 11017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:31 smithi115 bash[45495]: audit 2022-04-22T21:06:30.402503+0000 mon.a (mon.0) 11017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:31 smithi115 bash[46934]: audit 2022-04-22T21:06:30.402503+0000 mon.a (mon.0) 11017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:32.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:32 smithi115 bash[46934]: cluster 2022-04-22T21:06:30.783763+0000 mgr.y (mgr.34107) 7531 : cluster [DBG] pgmap v8554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:32.476 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:32 smithi115 bash[45495]: cluster 2022-04-22T21:06:30.783763+0000 mgr.y (mgr.34107) 7531 : cluster [DBG] pgmap v8554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:32 smithi121 bash[36902]: cluster 2022-04-22T21:06:30.783763+0000 mgr.y (mgr.34107) 7531 : cluster [DBG] pgmap v8554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:32.471Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:34 smithi121 bash[36902]: cluster 2022-04-22T21:06:32.784777+0000 mgr.y (mgr.34107) 7532 : cluster [DBG] pgmap v8555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:34 smithi121 bash[36902]: audit 2022-04-22T21:06:33.390436+0000 mon.a (mon.0) 11018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:34 smithi121 bash[36902]: audit 2022-04-22T21:06:34.125456+0000 mon.a (mon.0) 11019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:34.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:34.349Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[46934]: cluster 2022-04-22T21:06:32.784777+0000 mgr.y (mgr.34107) 7532 : cluster [DBG] pgmap v8555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[46934]: audit 2022-04-22T21:06:33.390436+0000 mon.a (mon.0) 11018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[46934]: audit 2022-04-22T21:06:34.125456+0000 mon.a (mon.0) 11019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[45495]: cluster 2022-04-22T21:06:32.784777+0000 mgr.y (mgr.34107) 7532 : cluster [DBG] pgmap v8555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[45495]: audit 2022-04-22T21:06:33.390436+0000 mon.a (mon.0) 11018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:34.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:34 smithi115 bash[45495]: audit 2022-04-22T21:06:34.125456+0000 mon.a (mon.0) 11019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:36.453 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:06:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:36 smithi121 bash[36902]: cluster 2022-04-22T21:06:34.785712+0000 mgr.y (mgr.34107) 7533 : cluster [DBG] pgmap v8556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:36 smithi121 bash[36902]: audit 2022-04-22T21:06:35.405452+0000 mon.a (mon.0) 11020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:36 smithi115 bash[45495]: cluster 2022-04-22T21:06:34.785712+0000 mgr.y (mgr.34107) 7533 : cluster [DBG] pgmap v8556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:36 smithi115 bash[45495]: audit 2022-04-22T21:06:35.405452+0000 mon.a (mon.0) 11020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:36 smithi115 bash[46934]: cluster 2022-04-22T21:06:34.785712+0000 mgr.y (mgr.34107) 7533 : cluster [DBG] pgmap v8556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:36 smithi115 bash[46934]: audit 2022-04-22T21:06:35.405452+0000 mon.a (mon.0) 11020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:06:36.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:06:36.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:06:36.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:06:36.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:06:36.801 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:06:36.801 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:06:37.198 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:37.199 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:06:37.200 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:06:37.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:37 smithi121 bash[36902]: audit 2022-04-22T21:06:36.433481+0000 mgr.y (mgr.34107) 7534 : audit [DBG] from='client.43200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:37 smithi121 bash[36902]: audit 2022-04-22T21:06:37.194094+0000 mon.a (mon.0) 11021 : audit [DBG] from='client.? 172.21.15.115:0/1106827150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:37.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:37 smithi115 bash[45495]: audit 2022-04-22T21:06:36.433481+0000 mgr.y (mgr.34107) 7534 : audit [DBG] from='client.43200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:37.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:37 smithi115 bash[45495]: audit 2022-04-22T21:06:37.194094+0000 mon.a (mon.0) 11021 : audit [DBG] from='client.? 172.21.15.115:0/1106827150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:37 smithi115 bash[46934]: audit 2022-04-22T21:06:36.433481+0000 mgr.y (mgr.34107) 7534 : audit [DBG] from='client.43200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:37 smithi115 bash[46934]: audit 2022-04-22T21:06:37.194094+0000 mon.a (mon.0) 11021 : audit [DBG] from='client.? 172.21.15.115:0/1106827150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:06:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:38 smithi121 bash[36902]: audit 2022-04-22T21:06:36.774159+0000 mgr.y (mgr.34107) 7535 : audit [DBG] from='client.43206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:38 smithi121 bash[36902]: cluster 2022-04-22T21:06:36.786689+0000 mgr.y (mgr.34107) 7536 : cluster [DBG] pgmap v8557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:38 smithi115 bash[46934]: audit 2022-04-22T21:06:36.774159+0000 mgr.y (mgr.34107) 7535 : audit [DBG] from='client.43206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:38 smithi115 bash[46934]: cluster 2022-04-22T21:06:36.786689+0000 mgr.y (mgr.34107) 7536 : cluster [DBG] pgmap v8557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:38 smithi115 bash[45495]: audit 2022-04-22T21:06:36.774159+0000 mgr.y (mgr.34107) 7535 : audit [DBG] from='client.43206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:06:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:38 smithi115 bash[45495]: cluster 2022-04-22T21:06:36.786689+0000 mgr.y (mgr.34107) 7536 : cluster [DBG] pgmap v8557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:39 smithi121 bash[36902]: audit 2022-04-22T21:06:38.393606+0000 mon.a (mon.0) 11022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:39 smithi121 bash[36902]: audit 2022-04-22T21:06:39.129156+0000 mon.a (mon.0) 11023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:39 smithi115 bash[45495]: audit 2022-04-22T21:06:38.393606+0000 mon.a (mon.0) 11022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:39 smithi115 bash[45495]: audit 2022-04-22T21:06:39.129156+0000 mon.a (mon.0) 11023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:39 smithi115 bash[46934]: audit 2022-04-22T21:06:38.393606+0000 mon.a (mon.0) 11022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:39 smithi115 bash[46934]: audit 2022-04-22T21:06:39.129156+0000 mon.a (mon.0) 11023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:40 smithi121 bash[36902]: cluster 2022-04-22T21:06:38.787870+0000 mgr.y (mgr.34107) 7537 : cluster [DBG] pgmap v8558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:40 smithi115 bash[45495]: cluster 2022-04-22T21:06:38.787870+0000 mgr.y (mgr.34107) 7537 : cluster [DBG] pgmap v8558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:40 smithi115 bash[46934]: cluster 2022-04-22T21:06:38.787870+0000 mgr.y (mgr.34107) 7537 : cluster [DBG] pgmap v8558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:41 smithi115 bash[45495]: audit 2022-04-22T21:06:40.408454+0000 mon.a (mon.0) 11024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:41 smithi115 bash[46934]: audit 2022-04-22T21:06:40.408454+0000 mon.a (mon.0) 11024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:41 smithi121 bash[36902]: audit 2022-04-22T21:06:40.408454+0000 mon.a (mon.0) 11024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:42 smithi115 bash[46934]: cluster 2022-04-22T21:06:40.788901+0000 mgr.y (mgr.34107) 7538 : cluster [DBG] pgmap v8559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:42.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:42.470Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:42 smithi115 bash[45495]: cluster 2022-04-22T21:06:40.788901+0000 mgr.y (mgr.34107) 7538 : cluster [DBG] pgmap v8559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:42 smithi121 bash[36902]: cluster 2022-04-22T21:06:40.788901+0000 mgr.y (mgr.34107) 7538 : cluster [DBG] pgmap v8559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:43.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:43 smithi121 bash[36902]: audit 2022-04-22T21:06:42.869610+0000 mon.a (mon.0) 11025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:06:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:43 smithi121 bash[36902]: audit 2022-04-22T21:06:42.872132+0000 mon.a (mon.0) 11026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:06:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:43 smithi121 bash[36902]: audit 2022-04-22T21:06:42.873704+0000 mon.a (mon.0) 11027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:06:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:43 smithi121 bash[36902]: audit 2022-04-22T21:06:43.237063+0000 mon.a (mon.0) 11028 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:06:43.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[45495]: audit 2022-04-22T21:06:42.869610+0000 mon.a (mon.0) 11025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[45495]: audit 2022-04-22T21:06:42.872132+0000 mon.a (mon.0) 11026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[45495]: audit 2022-04-22T21:06:42.873704+0000 mon.a (mon.0) 11027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[45495]: audit 2022-04-22T21:06:43.237063+0000 mon.a (mon.0) 11028 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[46934]: audit 2022-04-22T21:06:42.869610+0000 mon.a (mon.0) 11025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[46934]: audit 2022-04-22T21:06:42.872132+0000 mon.a (mon.0) 11026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:06:43.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[46934]: audit 2022-04-22T21:06:42.873704+0000 mon.a (mon.0) 11027 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:06:43.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:43 smithi115 bash[46934]: audit 2022-04-22T21:06:43.237063+0000 mon.a (mon.0) 11028 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:06:44.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:44.348Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:44.585 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[45495]: cluster 2022-04-22T21:06:42.790040+0000 mgr.y (mgr.34107) 7539 : cluster [DBG] pgmap v8560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:44.585 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[45495]: audit 2022-04-22T21:06:43.396823+0000 mon.a (mon.0) 11029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:44.585 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[45495]: audit 2022-04-22T21:06:44.132930+0000 mon.a (mon.0) 11030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:44.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[46934]: cluster 2022-04-22T21:06:42.790040+0000 mgr.y (mgr.34107) 7539 : cluster [DBG] pgmap v8560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:44.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[46934]: audit 2022-04-22T21:06:43.396823+0000 mon.a (mon.0) 11029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:44.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:44 smithi115 bash[46934]: audit 2022-04-22T21:06:44.132930+0000 mon.a (mon.0) 11030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:44 smithi121 bash[36902]: cluster 2022-04-22T21:06:42.790040+0000 mgr.y (mgr.34107) 7539 : cluster [DBG] pgmap v8560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:44 smithi121 bash[36902]: audit 2022-04-22T21:06:43.396823+0000 mon.a (mon.0) 11029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:44 smithi121 bash[36902]: audit 2022-04-22T21:06:44.132930+0000 mon.a (mon.0) 11030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:46 smithi115 bash[45495]: cluster 2022-04-22T21:06:44.791174+0000 mgr.y (mgr.34107) 7540 : cluster [DBG] pgmap v8561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:46 smithi115 bash[45495]: audit 2022-04-22T21:06:45.411619+0000 mon.a (mon.0) 11031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:46 smithi115 bash[46934]: cluster 2022-04-22T21:06:44.791174+0000 mgr.y (mgr.34107) 7540 : cluster [DBG] pgmap v8561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:46 smithi115 bash[46934]: audit 2022-04-22T21:06:45.411619+0000 mon.a (mon.0) 11031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:46 smithi121 bash[36902]: cluster 2022-04-22T21:06:44.791174+0000 mgr.y (mgr.34107) 7540 : cluster [DBG] pgmap v8561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:46 smithi121 bash[36902]: audit 2022-04-22T21:06:45.411619+0000 mon.a (mon.0) 11031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:48 smithi115 bash[46934]: cluster 2022-04-22T21:06:46.792325+0000 mgr.y (mgr.34107) 7541 : cluster [DBG] pgmap v8562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:48 smithi115 bash[45495]: cluster 2022-04-22T21:06:46.792325+0000 mgr.y (mgr.34107) 7541 : cluster [DBG] pgmap v8562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:48 smithi121 bash[36902]: cluster 2022-04-22T21:06:46.792325+0000 mgr.y (mgr.34107) 7541 : cluster [DBG] pgmap v8562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:49 smithi121 bash[36902]: audit 2022-04-22T21:06:48.399738+0000 mon.a (mon.0) 11032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:49 smithi121 bash[36902]: audit 2022-04-22T21:06:49.136678+0000 mon.a (mon.0) 11033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:49 smithi115 bash[46934]: audit 2022-04-22T21:06:48.399738+0000 mon.a (mon.0) 11032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:49 smithi115 bash[46934]: audit 2022-04-22T21:06:49.136678+0000 mon.a (mon.0) 11033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:49 smithi115 bash[45495]: audit 2022-04-22T21:06:48.399738+0000 mon.a (mon.0) 11032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:49 smithi115 bash[45495]: audit 2022-04-22T21:06:49.136678+0000 mon.a (mon.0) 11033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:50 smithi121 bash[36902]: cluster 2022-04-22T21:06:48.793468+0000 mgr.y (mgr.34107) 7542 : cluster [DBG] pgmap v8563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:50 smithi115 bash[45495]: cluster 2022-04-22T21:06:48.793468+0000 mgr.y (mgr.34107) 7542 : cluster [DBG] pgmap v8563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:50 smithi115 bash[46934]: cluster 2022-04-22T21:06:48.793468+0000 mgr.y (mgr.34107) 7542 : cluster [DBG] pgmap v8563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:51 smithi121 bash[36902]: audit 2022-04-22T21:06:50.414608+0000 mon.a (mon.0) 11034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:51 smithi115 bash[45495]: audit 2022-04-22T21:06:50.414608+0000 mon.a (mon.0) 11034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:51 smithi115 bash[46934]: audit 2022-04-22T21:06:50.414608+0000 mon.a (mon.0) 11034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:52 smithi121 bash[36902]: cluster 2022-04-22T21:06:50.794534+0000 mgr.y (mgr.34107) 7543 : cluster [DBG] pgmap v8564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:52 smithi115 bash[45495]: cluster 2022-04-22T21:06:50.794534+0000 mgr.y (mgr.34107) 7543 : cluster [DBG] pgmap v8564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:52 smithi115 bash[46934]: cluster 2022-04-22T21:06:50.794534+0000 mgr.y (mgr.34107) 7543 : cluster [DBG] pgmap v8564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:52.471Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:54 smithi121 bash[36902]: cluster 2022-04-22T21:06:52.795660+0000 mgr.y (mgr.34107) 7544 : cluster [DBG] pgmap v8565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:54 smithi121 bash[36902]: audit 2022-04-22T21:06:53.402943+0000 mon.a (mon.0) 11035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:54 smithi121 bash[36902]: audit 2022-04-22T21:06:54.140379+0000 mon.a (mon.0) 11036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:06:54.349Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:06:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[45495]: cluster 2022-04-22T21:06:52.795660+0000 mgr.y (mgr.34107) 7544 : cluster [DBG] pgmap v8565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[45495]: audit 2022-04-22T21:06:53.402943+0000 mon.a (mon.0) 11035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[45495]: audit 2022-04-22T21:06:54.140379+0000 mon.a (mon.0) 11036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[46934]: cluster 2022-04-22T21:06:52.795660+0000 mgr.y (mgr.34107) 7544 : cluster [DBG] pgmap v8565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[46934]: audit 2022-04-22T21:06:53.402943+0000 mon.a (mon.0) 11035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:54 smithi115 bash[46934]: audit 2022-04-22T21:06:54.140379+0000 mon.a (mon.0) 11036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:56 smithi121 bash[36902]: cluster 2022-04-22T21:06:54.796696+0000 mgr.y (mgr.34107) 7545 : cluster [DBG] pgmap v8566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:56 smithi121 bash[36902]: audit 2022-04-22T21:06:55.417460+0000 mon.a (mon.0) 11037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:56 smithi115 bash[45495]: cluster 2022-04-22T21:06:54.796696+0000 mgr.y (mgr.34107) 7545 : cluster [DBG] pgmap v8566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:56 smithi115 bash[45495]: audit 2022-04-22T21:06:55.417460+0000 mon.a (mon.0) 11037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:56 smithi115 bash[46934]: cluster 2022-04-22T21:06:54.796696+0000 mgr.y (mgr.34107) 7545 : cluster [DBG] pgmap v8566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:56 smithi115 bash[46934]: audit 2022-04-22T21:06:55.417460+0000 mon.a (mon.0) 11037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:06:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:58 smithi121 bash[36902]: cluster 2022-04-22T21:06:56.797881+0000 mgr.y (mgr.34107) 7546 : cluster [DBG] pgmap v8567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:58 smithi115 bash[45495]: cluster 2022-04-22T21:06:56.797881+0000 mgr.y (mgr.34107) 7546 : cluster [DBG] pgmap v8567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:58.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:58 smithi115 bash[46934]: cluster 2022-04-22T21:06:56.797881+0000 mgr.y (mgr.34107) 7546 : cluster [DBG] pgmap v8567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:06:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:59 smithi121 bash[36902]: audit 2022-04-22T21:06:58.406113+0000 mon.a (mon.0) 11038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:06:59 smithi121 bash[36902]: audit 2022-04-22T21:06:59.144150+0000 mon.a (mon.0) 11039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:59 smithi115 bash[45495]: audit 2022-04-22T21:06:58.406113+0000 mon.a (mon.0) 11038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:06:59 smithi115 bash[45495]: audit 2022-04-22T21:06:59.144150+0000 mon.a (mon.0) 11039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:06:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:59 smithi115 bash[46934]: audit 2022-04-22T21:06:58.406113+0000 mon.a (mon.0) 11038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:06:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:06:59 smithi115 bash[46934]: audit 2022-04-22T21:06:59.144150+0000 mon.a (mon.0) 11039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:00 smithi121 bash[36902]: cluster 2022-04-22T21:06:58.798964+0000 mgr.y (mgr.34107) 7547 : cluster [DBG] pgmap v8568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:00 smithi115 bash[45495]: cluster 2022-04-22T21:06:58.798964+0000 mgr.y (mgr.34107) 7547 : cluster [DBG] pgmap v8568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:00 smithi115 bash[46934]: cluster 2022-04-22T21:06:58.798964+0000 mgr.y (mgr.34107) 7547 : cluster [DBG] pgmap v8568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:01 smithi121 bash[36902]: audit 2022-04-22T21:07:00.420425+0000 mon.a (mon.0) 11040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:01 smithi115 bash[45495]: audit 2022-04-22T21:07:00.420425+0000 mon.a (mon.0) 11040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:01 smithi115 bash[46934]: audit 2022-04-22T21:07:00.420425+0000 mon.a (mon.0) 11040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:02 smithi121 bash[36902]: cluster 2022-04-22T21:07:00.800016+0000 mgr.y (mgr.34107) 7548 : cluster [DBG] pgmap v8569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:02 smithi115 bash[45495]: cluster 2022-04-22T21:07:00.800016+0000 mgr.y (mgr.34107) 7548 : cluster [DBG] pgmap v8569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:02.473Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:02 smithi115 bash[46934]: cluster 2022-04-22T21:07:00.800016+0000 mgr.y (mgr.34107) 7548 : cluster [DBG] pgmap v8569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:04 smithi121 bash[36902]: cluster 2022-04-22T21:07:02.801024+0000 mgr.y (mgr.34107) 7549 : cluster [DBG] pgmap v8570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:04 smithi121 bash[36902]: audit 2022-04-22T21:07:03.409142+0000 mon.a (mon.0) 11041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:04 smithi121 bash[36902]: audit 2022-04-22T21:07:04.147947+0000 mon.a (mon.0) 11042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:04.349Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[45495]: cluster 2022-04-22T21:07:02.801024+0000 mgr.y (mgr.34107) 7549 : cluster [DBG] pgmap v8570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[45495]: audit 2022-04-22T21:07:03.409142+0000 mon.a (mon.0) 11041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[45495]: audit 2022-04-22T21:07:04.147947+0000 mon.a (mon.0) 11042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[46934]: cluster 2022-04-22T21:07:02.801024+0000 mgr.y (mgr.34107) 7549 : cluster [DBG] pgmap v8570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[46934]: audit 2022-04-22T21:07:03.409142+0000 mon.a (mon.0) 11041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:04 smithi115 bash[46934]: audit 2022-04-22T21:07:04.147947+0000 mon.a (mon.0) 11042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:06 smithi121 bash[36902]: cluster 2022-04-22T21:07:04.802085+0000 mgr.y (mgr.34107) 7550 : cluster [DBG] pgmap v8571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:06 smithi121 bash[36902]: audit 2022-04-22T21:07:05.423143+0000 mon.a (mon.0) 11043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:06 smithi115 bash[45495]: cluster 2022-04-22T21:07:04.802085+0000 mgr.y (mgr.34107) 7550 : cluster [DBG] pgmap v8571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:06.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:06 smithi115 bash[45495]: audit 2022-04-22T21:07:05.423143+0000 mon.a (mon.0) 11043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:06 smithi115 bash[46934]: cluster 2022-04-22T21:07:04.802085+0000 mgr.y (mgr.34107) 7550 : cluster [DBG] pgmap v8571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:06.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:06 smithi115 bash[46934]: audit 2022-04-22T21:07:05.423143+0000 mon.a (mon.0) 11043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:07.572 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:07:07.917 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:07:07.918 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:07:07.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:07:07.920 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:07:08.325 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:07:08.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:08.326 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:07:08.327 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:07:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:08 smithi121 bash[36902]: cluster 2022-04-22T21:07:06.803219+0000 mgr.y (mgr.34107) 7551 : cluster [DBG] pgmap v8572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:08 smithi121 bash[36902]: audit 2022-04-22T21:07:07.549785+0000 mgr.y (mgr.34107) 7552 : audit [DBG] from='client.43218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:08 smithi121 bash[36902]: audit 2022-04-22T21:07:08.321011+0000 mon.a (mon.0) 11044 : audit [DBG] from='client.? 172.21.15.115:0/3409486472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[45495]: cluster 2022-04-22T21:07:06.803219+0000 mgr.y (mgr.34107) 7551 : cluster [DBG] pgmap v8572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[45495]: audit 2022-04-22T21:07:07.549785+0000 mgr.y (mgr.34107) 7552 : audit [DBG] from='client.43218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[45495]: audit 2022-04-22T21:07:08.321011+0000 mon.a (mon.0) 11044 : audit [DBG] from='client.? 172.21.15.115:0/3409486472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[46934]: cluster 2022-04-22T21:07:06.803219+0000 mgr.y (mgr.34107) 7551 : cluster [DBG] pgmap v8572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[46934]: audit 2022-04-22T21:07:07.549785+0000 mgr.y (mgr.34107) 7552 : audit [DBG] from='client.43218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:08 smithi115 bash[46934]: audit 2022-04-22T21:07:08.321011+0000 mon.a (mon.0) 11044 : audit [DBG] from='client.? 172.21.15.115:0/3409486472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:09 smithi121 bash[36902]: audit 2022-04-22T21:07:07.887543+0000 mgr.y (mgr.34107) 7553 : audit [DBG] from='client.43224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:09 smithi121 bash[36902]: audit 2022-04-22T21:07:08.412174+0000 mon.a (mon.0) 11045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:09 smithi121 bash[36902]: audit 2022-04-22T21:07:09.151967+0000 mon.a (mon.0) 11046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[45495]: audit 2022-04-22T21:07:07.887543+0000 mgr.y (mgr.34107) 7553 : audit [DBG] from='client.43224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[45495]: audit 2022-04-22T21:07:08.412174+0000 mon.a (mon.0) 11045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[45495]: audit 2022-04-22T21:07:09.151967+0000 mon.a (mon.0) 11046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[46934]: audit 2022-04-22T21:07:07.887543+0000 mgr.y (mgr.34107) 7553 : audit [DBG] from='client.43224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[46934]: audit 2022-04-22T21:07:08.412174+0000 mon.a (mon.0) 11045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:09 smithi115 bash[46934]: audit 2022-04-22T21:07:09.151967+0000 mon.a (mon.0) 11046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:10 smithi121 bash[36902]: cluster 2022-04-22T21:07:08.804429+0000 mgr.y (mgr.34107) 7554 : cluster [DBG] pgmap v8573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:10 smithi115 bash[45495]: cluster 2022-04-22T21:07:08.804429+0000 mgr.y (mgr.34107) 7554 : cluster [DBG] pgmap v8573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:10 smithi115 bash[46934]: cluster 2022-04-22T21:07:08.804429+0000 mgr.y (mgr.34107) 7554 : cluster [DBG] pgmap v8573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:11 smithi121 bash[36902]: audit 2022-04-22T21:07:10.426248+0000 mon.a (mon.0) 11047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:11 smithi115 bash[45495]: audit 2022-04-22T21:07:10.426248+0000 mon.a (mon.0) 11047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:11 smithi115 bash[46934]: audit 2022-04-22T21:07:10.426248+0000 mon.a (mon.0) 11047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:12 smithi121 bash[36902]: cluster 2022-04-22T21:07:10.805452+0000 mgr.y (mgr.34107) 7555 : cluster [DBG] pgmap v8574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:12 smithi115 bash[45495]: cluster 2022-04-22T21:07:10.805452+0000 mgr.y (mgr.34107) 7555 : cluster [DBG] pgmap v8574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:12.472Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:12 smithi115 bash[46934]: cluster 2022-04-22T21:07:10.805452+0000 mgr.y (mgr.34107) 7555 : cluster [DBG] pgmap v8574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:14 smithi121 bash[36902]: cluster 2022-04-22T21:07:12.806544+0000 mgr.y (mgr.34107) 7556 : cluster [DBG] pgmap v8575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:14 smithi121 bash[36902]: audit 2022-04-22T21:07:13.415224+0000 mon.a (mon.0) 11048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:14 smithi121 bash[36902]: audit 2022-04-22T21:07:14.155679+0000 mon.a (mon.0) 11049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[46934]: cluster 2022-04-22T21:07:12.806544+0000 mgr.y (mgr.34107) 7556 : cluster [DBG] pgmap v8575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[46934]: audit 2022-04-22T21:07:13.415224+0000 mon.a (mon.0) 11048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[46934]: audit 2022-04-22T21:07:14.155679+0000 mon.a (mon.0) 11049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[45495]: cluster 2022-04-22T21:07:12.806544+0000 mgr.y (mgr.34107) 7556 : cluster [DBG] pgmap v8575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[45495]: audit 2022-04-22T21:07:13.415224+0000 mon.a (mon.0) 11048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[45495]: audit 2022-04-22T21:07:14.155679+0000 mon.a (mon.0) 11049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:14.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:14.350Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:16 smithi121 bash[36902]: cluster 2022-04-22T21:07:14.807615+0000 mgr.y (mgr.34107) 7557 : cluster [DBG] pgmap v8576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:16 smithi121 bash[36902]: audit 2022-04-22T21:07:15.429014+0000 mon.a (mon.0) 11050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:16 smithi121 bash[36902]: audit 2022-04-22T21:07:15.818809+0000 mon.a (mon.0) 11051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:16 smithi121 bash[36902]: audit 2022-04-22T21:07:15.819109+0000 mon.a (mon.0) 11052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:07:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[45495]: cluster 2022-04-22T21:07:14.807615+0000 mgr.y (mgr.34107) 7557 : cluster [DBG] pgmap v8576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[45495]: audit 2022-04-22T21:07:15.429014+0000 mon.a (mon.0) 11050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[45495]: audit 2022-04-22T21:07:15.818809+0000 mon.a (mon.0) 11051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[45495]: audit 2022-04-22T21:07:15.819109+0000 mon.a (mon.0) 11052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[46934]: cluster 2022-04-22T21:07:14.807615+0000 mgr.y (mgr.34107) 7557 : cluster [DBG] pgmap v8576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[46934]: audit 2022-04-22T21:07:15.429014+0000 mon.a (mon.0) 11050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[46934]: audit 2022-04-22T21:07:15.818809+0000 mon.a (mon.0) 11051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:07:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:16 smithi115 bash[46934]: audit 2022-04-22T21:07:15.819109+0000 mon.a (mon.0) 11052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:07:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:18 smithi121 bash[36902]: cluster 2022-04-22T21:07:16.808637+0000 mgr.y (mgr.34107) 7558 : cluster [DBG] pgmap v8577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:18 smithi115 bash[45495]: cluster 2022-04-22T21:07:16.808637+0000 mgr.y (mgr.34107) 7558 : cluster [DBG] pgmap v8577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:18 smithi115 bash[46934]: cluster 2022-04-22T21:07:16.808637+0000 mgr.y (mgr.34107) 7558 : cluster [DBG] pgmap v8577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:19 smithi121 bash[36902]: audit 2022-04-22T21:07:18.418115+0000 mon.a (mon.0) 11053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:19 smithi121 bash[36902]: audit 2022-04-22T21:07:19.159324+0000 mon.a (mon.0) 11054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:19 smithi115 bash[45495]: audit 2022-04-22T21:07:18.418115+0000 mon.a (mon.0) 11053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:19 smithi115 bash[45495]: audit 2022-04-22T21:07:19.159324+0000 mon.a (mon.0) 11054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:19 smithi115 bash[46934]: audit 2022-04-22T21:07:18.418115+0000 mon.a (mon.0) 11053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:19 smithi115 bash[46934]: audit 2022-04-22T21:07:19.159324+0000 mon.a (mon.0) 11054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:20 smithi121 bash[36902]: cluster 2022-04-22T21:07:18.809712+0000 mgr.y (mgr.34107) 7559 : cluster [DBG] pgmap v8578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:20 smithi115 bash[45495]: cluster 2022-04-22T21:07:18.809712+0000 mgr.y (mgr.34107) 7559 : cluster [DBG] pgmap v8578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:20 smithi115 bash[46934]: cluster 2022-04-22T21:07:18.809712+0000 mgr.y (mgr.34107) 7559 : cluster [DBG] pgmap v8578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:21 smithi121 bash[36902]: audit 2022-04-22T21:07:20.432134+0000 mon.a (mon.0) 11055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:21 smithi115 bash[46934]: audit 2022-04-22T21:07:20.432134+0000 mon.a (mon.0) 11055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:21.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:21 smithi115 bash[45495]: audit 2022-04-22T21:07:20.432134+0000 mon.a (mon.0) 11055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:22 smithi121 bash[36902]: cluster 2022-04-22T21:07:20.810754+0000 mgr.y (mgr.34107) 7560 : cluster [DBG] pgmap v8579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:22 smithi115 bash[45495]: cluster 2022-04-22T21:07:20.810754+0000 mgr.y (mgr.34107) 7560 : cluster [DBG] pgmap v8579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:22 smithi115 bash[46934]: cluster 2022-04-22T21:07:20.810754+0000 mgr.y (mgr.34107) 7560 : cluster [DBG] pgmap v8579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:22.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:24 smithi121 bash[36902]: cluster 2022-04-22T21:07:22.811798+0000 mgr.y (mgr.34107) 7561 : cluster [DBG] pgmap v8580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:24 smithi121 bash[36902]: audit 2022-04-22T21:07:23.421086+0000 mon.a (mon.0) 11056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:24 smithi121 bash[36902]: audit 2022-04-22T21:07:24.162731+0000 mon.a (mon.0) 11057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:24.351Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[45495]: cluster 2022-04-22T21:07:22.811798+0000 mgr.y (mgr.34107) 7561 : cluster [DBG] pgmap v8580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[45495]: audit 2022-04-22T21:07:23.421086+0000 mon.a (mon.0) 11056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[45495]: audit 2022-04-22T21:07:24.162731+0000 mon.a (mon.0) 11057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[46934]: cluster 2022-04-22T21:07:22.811798+0000 mgr.y (mgr.34107) 7561 : cluster [DBG] pgmap v8580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[46934]: audit 2022-04-22T21:07:23.421086+0000 mon.a (mon.0) 11056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:24 smithi115 bash[46934]: audit 2022-04-22T21:07:24.162731+0000 mon.a (mon.0) 11057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:26 smithi121 bash[36902]: cluster 2022-04-22T21:07:24.812828+0000 mgr.y (mgr.34107) 7562 : cluster [DBG] pgmap v8581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:26.773 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:26 smithi121 bash[36902]: audit 2022-04-22T21:07:25.434791+0000 mon.a (mon.0) 11058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:26 smithi115 bash[45495]: cluster 2022-04-22T21:07:24.812828+0000 mgr.y (mgr.34107) 7562 : cluster [DBG] pgmap v8581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:26 smithi115 bash[45495]: audit 2022-04-22T21:07:25.434791+0000 mon.a (mon.0) 11058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:26 smithi115 bash[46934]: cluster 2022-04-22T21:07:24.812828+0000 mgr.y (mgr.34107) 7562 : cluster [DBG] pgmap v8581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:26 smithi115 bash[46934]: audit 2022-04-22T21:07:25.434791+0000 mon.a (mon.0) 11058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:28 smithi121 bash[36902]: cluster 2022-04-22T21:07:26.813946+0000 mgr.y (mgr.34107) 7563 : cluster [DBG] pgmap v8582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:28 smithi121 bash[36902]: audit 2022-04-22T21:07:28.424654+0000 mon.a (mon.0) 11059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:28 smithi115 bash[46934]: cluster 2022-04-22T21:07:26.813946+0000 mgr.y (mgr.34107) 7563 : cluster [DBG] pgmap v8582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:28.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:28 smithi115 bash[46934]: audit 2022-04-22T21:07:28.424654+0000 mon.a (mon.0) 11059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:28 smithi115 bash[45495]: cluster 2022-04-22T21:07:26.813946+0000 mgr.y (mgr.34107) 7563 : cluster [DBG] pgmap v8582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:28.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:28 smithi115 bash[45495]: audit 2022-04-22T21:07:28.424654+0000 mon.a (mon.0) 11059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:29 smithi121 bash[36902]: audit 2022-04-22T21:07:29.166401+0000 mon.a (mon.0) 11060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:29 smithi115 bash[45495]: audit 2022-04-22T21:07:29.166401+0000 mon.a (mon.0) 11060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:29 smithi115 bash[46934]: audit 2022-04-22T21:07:29.166401+0000 mon.a (mon.0) 11060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:30 smithi121 bash[36902]: cluster 2022-04-22T21:07:28.815120+0000 mgr.y (mgr.34107) 7564 : cluster [DBG] pgmap v8583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:30 smithi115 bash[45495]: cluster 2022-04-22T21:07:28.815120+0000 mgr.y (mgr.34107) 7564 : cluster [DBG] pgmap v8583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:30 smithi115 bash[46934]: cluster 2022-04-22T21:07:28.815120+0000 mgr.y (mgr.34107) 7564 : cluster [DBG] pgmap v8583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:31.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:31 smithi121 bash[36902]: audit 2022-04-22T21:07:30.437562+0000 mon.a (mon.0) 11061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:31.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:31 smithi115 bash[45495]: audit 2022-04-22T21:07:30.437562+0000 mon.a (mon.0) 11061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:31 smithi115 bash[46934]: audit 2022-04-22T21:07:30.437562+0000 mon.a (mon.0) 11061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:32 smithi121 bash[36902]: cluster 2022-04-22T21:07:30.816261+0000 mgr.y (mgr.34107) 7565 : cluster [DBG] pgmap v8584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:32 smithi115 bash[46934]: cluster 2022-04-22T21:07:30.816261+0000 mgr.y (mgr.34107) 7565 : cluster [DBG] pgmap v8584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:32.473Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:32.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:32 smithi115 bash[45495]: cluster 2022-04-22T21:07:30.816261+0000 mgr.y (mgr.34107) 7565 : cluster [DBG] pgmap v8584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:33 smithi121 bash[36902]: audit 2022-04-22T21:07:33.427593+0000 mon.a (mon.0) 11062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:33 smithi115 bash[46934]: audit 2022-04-22T21:07:33.427593+0000 mon.a (mon.0) 11062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:33.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:33 smithi115 bash[45495]: audit 2022-04-22T21:07:33.427593+0000 mon.a (mon.0) 11062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:34 smithi121 bash[36902]: cluster 2022-04-22T21:07:32.817398+0000 mgr.y (mgr.34107) 7566 : cluster [DBG] pgmap v8585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:34 smithi121 bash[36902]: audit 2022-04-22T21:07:34.170224+0000 mon.a (mon.0) 11063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:34.351Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:34 smithi115 bash[45495]: cluster 2022-04-22T21:07:32.817398+0000 mgr.y (mgr.34107) 7566 : cluster [DBG] pgmap v8585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:34 smithi115 bash[45495]: audit 2022-04-22T21:07:34.170224+0000 mon.a (mon.0) 11063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:34 smithi115 bash[46934]: cluster 2022-04-22T21:07:32.817398+0000 mgr.y (mgr.34107) 7566 : cluster [DBG] pgmap v8585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:34 smithi115 bash[46934]: audit 2022-04-22T21:07:34.170224+0000 mon.a (mon.0) 11063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:35 smithi121 bash[36902]: cluster 2022-04-22T21:07:34.818529+0000 mgr.y (mgr.34107) 7567 : cluster [DBG] pgmap v8586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:35 smithi121 bash[36902]: audit 2022-04-22T21:07:35.441024+0000 mon.a (mon.0) 11064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:35 smithi115 bash[45495]: cluster 2022-04-22T21:07:34.818529+0000 mgr.y (mgr.34107) 7567 : cluster [DBG] pgmap v8586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:35 smithi115 bash[45495]: audit 2022-04-22T21:07:35.441024+0000 mon.a (mon.0) 11064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:35 smithi115 bash[46934]: cluster 2022-04-22T21:07:34.818529+0000 mgr.y (mgr.34107) 7567 : cluster [DBG] pgmap v8586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:35 smithi115 bash[46934]: audit 2022-04-22T21:07:35.441024+0000 mon.a (mon.0) 11064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:37 smithi121 bash[36902]: cluster 2022-04-22T21:07:36.819645+0000 mgr.y (mgr.34107) 7568 : cluster [DBG] pgmap v8587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:37 smithi115 bash[46934]: cluster 2022-04-22T21:07:36.819645+0000 mgr.y (mgr.34107) 7568 : cluster [DBG] pgmap v8587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:37 smithi115 bash[45495]: cluster 2022-04-22T21:07:36.819645+0000 mgr.y (mgr.34107) 7568 : cluster [DBG] pgmap v8587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:07:39.039 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:07:39.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:07:39.041 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:07:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:38 smithi121 bash[36902]: audit 2022-04-22T21:07:38.430620+0000 mon.a (mon.0) 11065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:38 smithi121 bash[36902]: audit 2022-04-22T21:07:38.674941+0000 mgr.y (mgr.34107) 7569 : audit [DBG] from='client.43236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:38 smithi115 bash[46934]: audit 2022-04-22T21:07:38.430620+0000 mon.a (mon.0) 11065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:38 smithi115 bash[46934]: audit 2022-04-22T21:07:38.674941+0000 mgr.y (mgr.34107) 7569 : audit [DBG] from='client.43236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:38 smithi115 bash[45495]: audit 2022-04-22T21:07:38.430620+0000 mon.a (mon.0) 11065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:38 smithi115 bash[45495]: audit 2022-04-22T21:07:38.674941+0000 mgr.y (mgr.34107) 7569 : audit [DBG] from='client.43236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:39.442 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:07:39.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:07:39.444 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:07:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:39 smithi121 bash[36902]: cluster 2022-04-22T21:07:38.820586+0000 mgr.y (mgr.34107) 7570 : cluster [DBG] pgmap v8588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:39 smithi121 bash[36902]: audit 2022-04-22T21:07:39.020258+0000 mgr.y (mgr.34107) 7571 : audit [DBG] from='client.53173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:39 smithi121 bash[36902]: audit 2022-04-22T21:07:39.173643+0000 mon.a (mon.0) 11066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:39 smithi121 bash[36902]: audit 2022-04-22T21:07:39.437926+0000 mon.c (mon.1) 278 : audit [DBG] from='client.? 172.21.15.115:0/208123691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[45495]: cluster 2022-04-22T21:07:38.820586+0000 mgr.y (mgr.34107) 7570 : cluster [DBG] pgmap v8588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[45495]: audit 2022-04-22T21:07:39.020258+0000 mgr.y (mgr.34107) 7571 : audit [DBG] from='client.53173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[45495]: audit 2022-04-22T21:07:39.173643+0000 mon.a (mon.0) 11066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[45495]: audit 2022-04-22T21:07:39.437926+0000 mon.c (mon.1) 278 : audit [DBG] from='client.? 172.21.15.115:0/208123691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[46934]: cluster 2022-04-22T21:07:38.820586+0000 mgr.y (mgr.34107) 7570 : cluster [DBG] pgmap v8588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[46934]: audit 2022-04-22T21:07:39.020258+0000 mgr.y (mgr.34107) 7571 : audit [DBG] from='client.53173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[46934]: audit 2022-04-22T21:07:39.173643+0000 mon.a (mon.0) 11066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:39 smithi115 bash[46934]: audit 2022-04-22T21:07:39.437926+0000 mon.c (mon.1) 278 : audit [DBG] from='client.? 172.21.15.115:0/208123691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:07:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:40 smithi121 bash[36902]: audit 2022-04-22T21:07:40.444169+0000 mon.a (mon.0) 11067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:40 smithi115 bash[45495]: audit 2022-04-22T21:07:40.444169+0000 mon.a (mon.0) 11067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:40 smithi115 bash[46934]: audit 2022-04-22T21:07:40.444169+0000 mon.a (mon.0) 11067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:41 smithi121 bash[36902]: cluster 2022-04-22T21:07:40.821627+0000 mgr.y (mgr.34107) 7572 : cluster [DBG] pgmap v8589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:42.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:41 smithi115 bash[45495]: cluster 2022-04-22T21:07:40.821627+0000 mgr.y (mgr.34107) 7572 : cluster [DBG] pgmap v8589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:41 smithi115 bash[46934]: cluster 2022-04-22T21:07:40.821627+0000 mgr.y (mgr.34107) 7572 : cluster [DBG] pgmap v8589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:42.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: cluster 2022-04-22T21:07:42.822736+0000 mgr.y (mgr.34107) 7573 : cluster [DBG] pgmap v8590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: audit 2022-04-22T21:07:43.244797+0000 mon.a (mon.0) 11068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: audit 2022-04-22T21:07:43.247928+0000 mon.a (mon.0) 11069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: audit 2022-04-22T21:07:43.249663+0000 mon.a (mon.0) 11070 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: audit 2022-04-22T21:07:43.432867+0000 mon.a (mon.0) 11071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:43 smithi121 bash[36902]: audit 2022-04-22T21:07:43.615612+0000 mon.a (mon.0) 11072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: cluster 2022-04-22T21:07:42.822736+0000 mgr.y (mgr.34107) 7573 : cluster [DBG] pgmap v8590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: audit 2022-04-22T21:07:43.244797+0000 mon.a (mon.0) 11068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: audit 2022-04-22T21:07:43.247928+0000 mon.a (mon.0) 11069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: audit 2022-04-22T21:07:43.249663+0000 mon.a (mon.0) 11070 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: audit 2022-04-22T21:07:43.432867+0000 mon.a (mon.0) 11071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[45495]: audit 2022-04-22T21:07:43.615612+0000 mon.a (mon.0) 11072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: cluster 2022-04-22T21:07:42.822736+0000 mgr.y (mgr.34107) 7573 : cluster [DBG] pgmap v8590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: audit 2022-04-22T21:07:43.244797+0000 mon.a (mon.0) 11068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: audit 2022-04-22T21:07:43.247928+0000 mon.a (mon.0) 11069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: audit 2022-04-22T21:07:43.249663+0000 mon.a (mon.0) 11070 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: audit 2022-04-22T21:07:43.432867+0000 mon.a (mon.0) 11071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:43 smithi115 bash[46934]: audit 2022-04-22T21:07:43.615612+0000 mon.a (mon.0) 11072 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:07:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:44.354Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:44 smithi121 bash[36902]: audit 2022-04-22T21:07:44.176937+0000 mon.a (mon.0) 11073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:44 smithi115 bash[45495]: audit 2022-04-22T21:07:44.176937+0000 mon.a (mon.0) 11073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:44 smithi115 bash[46934]: audit 2022-04-22T21:07:44.176937+0000 mon.a (mon.0) 11073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:45 smithi121 bash[36902]: cluster 2022-04-22T21:07:44.823799+0000 mgr.y (mgr.34107) 7574 : cluster [DBG] pgmap v8591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:45 smithi121 bash[36902]: audit 2022-04-22T21:07:45.447342+0000 mon.a (mon.0) 11074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:45 smithi115 bash[45495]: cluster 2022-04-22T21:07:44.823799+0000 mgr.y (mgr.34107) 7574 : cluster [DBG] pgmap v8591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:45 smithi115 bash[45495]: audit 2022-04-22T21:07:45.447342+0000 mon.a (mon.0) 11074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:45 smithi115 bash[46934]: cluster 2022-04-22T21:07:44.823799+0000 mgr.y (mgr.34107) 7574 : cluster [DBG] pgmap v8591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:45 smithi115 bash[46934]: audit 2022-04-22T21:07:45.447342+0000 mon.a (mon.0) 11074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:47 smithi121 bash[36902]: cluster 2022-04-22T21:07:46.824869+0000 mgr.y (mgr.34107) 7575 : cluster [DBG] pgmap v8592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:47 smithi115 bash[45495]: cluster 2022-04-22T21:07:46.824869+0000 mgr.y (mgr.34107) 7575 : cluster [DBG] pgmap v8592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:48.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:47 smithi115 bash[46934]: cluster 2022-04-22T21:07:46.824869+0000 mgr.y (mgr.34107) 7575 : cluster [DBG] pgmap v8592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:48 smithi121 bash[36902]: audit 2022-04-22T21:07:48.435619+0000 mon.a (mon.0) 11075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:48 smithi115 bash[45495]: audit 2022-04-22T21:07:48.435619+0000 mon.a (mon.0) 11075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:48 smithi115 bash[46934]: audit 2022-04-22T21:07:48.435619+0000 mon.a (mon.0) 11075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:50.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:49 smithi121 bash[36902]: cluster 2022-04-22T21:07:48.825942+0000 mgr.y (mgr.34107) 7576 : cluster [DBG] pgmap v8593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:49 smithi121 bash[36902]: audit 2022-04-22T21:07:49.180985+0000 mon.a (mon.0) 11076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:49 smithi115 bash[45495]: cluster 2022-04-22T21:07:48.825942+0000 mgr.y (mgr.34107) 7576 : cluster [DBG] pgmap v8593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:50.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:49 smithi115 bash[45495]: audit 2022-04-22T21:07:49.180985+0000 mon.a (mon.0) 11076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:49 smithi115 bash[46934]: cluster 2022-04-22T21:07:48.825942+0000 mgr.y (mgr.34107) 7576 : cluster [DBG] pgmap v8593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:49 smithi115 bash[46934]: audit 2022-04-22T21:07:49.180985+0000 mon.a (mon.0) 11076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:50 smithi121 bash[36902]: audit 2022-04-22T21:07:50.450619+0000 mon.a (mon.0) 11077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:50 smithi115 bash[45495]: audit 2022-04-22T21:07:50.450619+0000 mon.a (mon.0) 11077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:50 smithi115 bash[46934]: audit 2022-04-22T21:07:50.450619+0000 mon.a (mon.0) 11077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:51 smithi121 bash[36902]: cluster 2022-04-22T21:07:50.827046+0000 mgr.y (mgr.34107) 7577 : cluster [DBG] pgmap v8594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:51 smithi115 bash[45495]: cluster 2022-04-22T21:07:50.827046+0000 mgr.y (mgr.34107) 7577 : cluster [DBG] pgmap v8594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:51 smithi115 bash[46934]: cluster 2022-04-22T21:07:50.827046+0000 mgr.y (mgr.34107) 7577 : cluster [DBG] pgmap v8594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:52.476Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:53 smithi121 bash[36902]: cluster 2022-04-22T21:07:52.828077+0000 mgr.y (mgr.34107) 7578 : cluster [DBG] pgmap v8595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:53 smithi121 bash[36902]: audit 2022-04-22T21:07:53.438715+0000 mon.a (mon.0) 11078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:53 smithi115 bash[45495]: cluster 2022-04-22T21:07:52.828077+0000 mgr.y (mgr.34107) 7578 : cluster [DBG] pgmap v8595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:54.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:53 smithi115 bash[45495]: audit 2022-04-22T21:07:53.438715+0000 mon.a (mon.0) 11078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:54.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:53 smithi115 bash[46934]: cluster 2022-04-22T21:07:52.828077+0000 mgr.y (mgr.34107) 7578 : cluster [DBG] pgmap v8595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:54.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:53 smithi115 bash[46934]: audit 2022-04-22T21:07:53.438715+0000 mon.a (mon.0) 11078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:07:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:07:54.352Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:07:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:54 smithi121 bash[36902]: audit 2022-04-22T21:07:54.184701+0000 mon.a (mon.0) 11079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:54 smithi115 bash[45495]: audit 2022-04-22T21:07:54.184701+0000 mon.a (mon.0) 11079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:54 smithi115 bash[46934]: audit 2022-04-22T21:07:54.184701+0000 mon.a (mon.0) 11079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:07:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:55 smithi121 bash[36902]: cluster 2022-04-22T21:07:54.829205+0000 mgr.y (mgr.34107) 7579 : cluster [DBG] pgmap v8596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:55 smithi121 bash[36902]: audit 2022-04-22T21:07:55.453653+0000 mon.a (mon.0) 11080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:55 smithi115 bash[45495]: cluster 2022-04-22T21:07:54.829205+0000 mgr.y (mgr.34107) 7579 : cluster [DBG] pgmap v8596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:55 smithi115 bash[45495]: audit 2022-04-22T21:07:55.453653+0000 mon.a (mon.0) 11080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:55 smithi115 bash[46934]: cluster 2022-04-22T21:07:54.829205+0000 mgr.y (mgr.34107) 7579 : cluster [DBG] pgmap v8596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:55 smithi115 bash[46934]: audit 2022-04-22T21:07:55.453653+0000 mon.a (mon.0) 11080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:07:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:57 smithi121 bash[36902]: cluster 2022-04-22T21:07:56.830322+0000 mgr.y (mgr.34107) 7580 : cluster [DBG] pgmap v8597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:57 smithi115 bash[45495]: cluster 2022-04-22T21:07:56.830322+0000 mgr.y (mgr.34107) 7580 : cluster [DBG] pgmap v8597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:57 smithi115 bash[46934]: cluster 2022-04-22T21:07:56.830322+0000 mgr.y (mgr.34107) 7580 : cluster [DBG] pgmap v8597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:07:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:58 smithi121 bash[36902]: audit 2022-04-22T21:07:58.442099+0000 mon.a (mon.0) 11081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:59.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:58 smithi115 bash[45495]: audit 2022-04-22T21:07:58.442099+0000 mon.a (mon.0) 11081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:07:59.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:58 smithi115 bash[46934]: audit 2022-04-22T21:07:58.442099+0000 mon.a (mon.0) 11081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:59 smithi121 bash[36902]: cluster 2022-04-22T21:07:58.831545+0000 mgr.y (mgr.34107) 7581 : cluster [DBG] pgmap v8598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:07:59 smithi121 bash[36902]: audit 2022-04-22T21:07:59.188526+0000 mon.a (mon.0) 11082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:59 smithi115 bash[46934]: cluster 2022-04-22T21:07:58.831545+0000 mgr.y (mgr.34107) 7581 : cluster [DBG] pgmap v8598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:07:59 smithi115 bash[46934]: audit 2022-04-22T21:07:59.188526+0000 mon.a (mon.0) 11082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:59 smithi115 bash[45495]: cluster 2022-04-22T21:07:58.831545+0000 mgr.y (mgr.34107) 7581 : cluster [DBG] pgmap v8598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:00.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:07:59 smithi115 bash[45495]: audit 2022-04-22T21:07:59.188526+0000 mon.a (mon.0) 11082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:00 smithi121 bash[36902]: audit 2022-04-22T21:08:00.456731+0000 mon.a (mon.0) 11083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:00 smithi115 bash[45495]: audit 2022-04-22T21:08:00.456731+0000 mon.a (mon.0) 11083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:00 smithi115 bash[46934]: audit 2022-04-22T21:08:00.456731+0000 mon.a (mon.0) 11083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:02.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:01 smithi121 bash[36902]: cluster 2022-04-22T21:08:00.832648+0000 mgr.y (mgr.34107) 7582 : cluster [DBG] pgmap v8599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:01 smithi115 bash[45495]: cluster 2022-04-22T21:08:00.832648+0000 mgr.y (mgr.34107) 7582 : cluster [DBG] pgmap v8599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:01 smithi115 bash[46934]: cluster 2022-04-22T21:08:00.832648+0000 mgr.y (mgr.34107) 7582 : cluster [DBG] pgmap v8599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:02.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:04.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:03 smithi121 bash[36902]: cluster 2022-04-22T21:08:02.833860+0000 mgr.y (mgr.34107) 7583 : cluster [DBG] pgmap v8600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:03 smithi121 bash[36902]: audit 2022-04-22T21:08:03.445250+0000 mon.a (mon.0) 11084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:03 smithi115 bash[45495]: cluster 2022-04-22T21:08:02.833860+0000 mgr.y (mgr.34107) 7583 : cluster [DBG] pgmap v8600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:04.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:03 smithi115 bash[45495]: audit 2022-04-22T21:08:03.445250+0000 mon.a (mon.0) 11084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:03 smithi115 bash[46934]: cluster 2022-04-22T21:08:02.833860+0000 mgr.y (mgr.34107) 7583 : cluster [DBG] pgmap v8600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:03 smithi115 bash[46934]: audit 2022-04-22T21:08:03.445250+0000 mon.a (mon.0) 11084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:04.353Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:05 smithi121 bash[36902]: audit 2022-04-22T21:08:04.192393+0000 mon.a (mon.0) 11085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:05 smithi115 bash[45495]: audit 2022-04-22T21:08:04.192393+0000 mon.a (mon.0) 11085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:05 smithi115 bash[46934]: audit 2022-04-22T21:08:04.192393+0000 mon.a (mon.0) 11085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:06 smithi115 bash[45495]: cluster 2022-04-22T21:08:04.835080+0000 mgr.y (mgr.34107) 7584 : cluster [DBG] pgmap v8601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:06 smithi115 bash[45495]: audit 2022-04-22T21:08:05.459835+0000 mon.a (mon.0) 11086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:06 smithi115 bash[46934]: cluster 2022-04-22T21:08:04.835080+0000 mgr.y (mgr.34107) 7584 : cluster [DBG] pgmap v8601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:06 smithi115 bash[46934]: audit 2022-04-22T21:08:05.459835+0000 mon.a (mon.0) 11086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:06 smithi121 bash[36902]: cluster 2022-04-22T21:08:04.835080+0000 mgr.y (mgr.34107) 7584 : cluster [DBG] pgmap v8601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:06 smithi121 bash[36902]: audit 2022-04-22T21:08:05.459835+0000 mon.a (mon.0) 11086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:08.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:08 smithi115 bash[45495]: cluster 2022-04-22T21:08:06.836264+0000 mgr.y (mgr.34107) 7585 : cluster [DBG] pgmap v8602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:08 smithi115 bash[46934]: cluster 2022-04-22T21:08:06.836264+0000 mgr.y (mgr.34107) 7585 : cluster [DBG] pgmap v8602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:08 smithi121 bash[36902]: cluster 2022-04-22T21:08:06.836264+0000 mgr.y (mgr.34107) 7585 : cluster [DBG] pgmap v8602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:09.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:09 smithi115 bash[45495]: audit 2022-04-22T21:08:08.448428+0000 mon.a (mon.0) 11087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:09.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:09 smithi115 bash[46934]: audit 2022-04-22T21:08:08.448428+0000 mon.a (mon.0) 11087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:09 smithi121 bash[36902]: audit 2022-04-22T21:08:08.448428+0000 mon.a (mon.0) 11087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:09.821 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:08:10.172 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:08:10.173 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:08:10.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:08:10.175 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:08:10.175 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:08:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:10 smithi115 bash[45495]: cluster 2022-04-22T21:08:08.837411+0000 mgr.y (mgr.34107) 7586 : cluster [DBG] pgmap v8603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:10.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:10 smithi115 bash[45495]: audit 2022-04-22T21:08:09.195947+0000 mon.a (mon.0) 11088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:10.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:10 smithi115 bash[46934]: cluster 2022-04-22T21:08:08.837411+0000 mgr.y (mgr.34107) 7586 : cluster [DBG] pgmap v8603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:10.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:10 smithi115 bash[46934]: audit 2022-04-22T21:08:09.195947+0000 mon.a (mon.0) 11088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:10 smithi121 bash[36902]: cluster 2022-04-22T21:08:08.837411+0000 mgr.y (mgr.34107) 7586 : cluster [DBG] pgmap v8603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:10.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:10 smithi121 bash[36902]: audit 2022-04-22T21:08:09.195947+0000 mon.a (mon.0) 11088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:10.580 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:08:10.580 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:08:10.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:08:10.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:10.582 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:08:10.582 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:08:10.582 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:08:10.582 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:08:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[45495]: audit 2022-04-22T21:08:09.800737+0000 mgr.y (mgr.34107) 7587 : audit [DBG] from='client.53185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[45495]: audit 2022-04-22T21:08:10.147715+0000 mgr.y (mgr.34107) 7588 : audit [DBG] from='client.43260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[45495]: audit 2022-04-22T21:08:10.462843+0000 mon.a (mon.0) 11089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[45495]: audit 2022-04-22T21:08:10.575843+0000 mon.a (mon.0) 11090 : audit [DBG] from='client.? 172.21.15.115:0/2686391838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[46934]: audit 2022-04-22T21:08:09.800737+0000 mgr.y (mgr.34107) 7587 : audit [DBG] from='client.53185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[46934]: audit 2022-04-22T21:08:10.147715+0000 mgr.y (mgr.34107) 7588 : audit [DBG] from='client.43260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[46934]: audit 2022-04-22T21:08:10.462843+0000 mon.a (mon.0) 11089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:11 smithi115 bash[46934]: audit 2022-04-22T21:08:10.575843+0000 mon.a (mon.0) 11090 : audit [DBG] from='client.? 172.21.15.115:0/2686391838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:11 smithi121 bash[36902]: audit 2022-04-22T21:08:09.800737+0000 mgr.y (mgr.34107) 7587 : audit [DBG] from='client.53185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:11 smithi121 bash[36902]: audit 2022-04-22T21:08:10.147715+0000 mgr.y (mgr.34107) 7588 : audit [DBG] from='client.43260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:11 smithi121 bash[36902]: audit 2022-04-22T21:08:10.462843+0000 mon.a (mon.0) 11089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:11 smithi121 bash[36902]: audit 2022-04-22T21:08:10.575843+0000 mon.a (mon.0) 11090 : audit [DBG] from='client.? 172.21.15.115:0/2686391838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:12 smithi115 bash[45495]: cluster 2022-04-22T21:08:10.838495+0000 mgr.y (mgr.34107) 7589 : cluster [DBG] pgmap v8604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:12 smithi115 bash[46934]: cluster 2022-04-22T21:08:10.838495+0000 mgr.y (mgr.34107) 7589 : cluster [DBG] pgmap v8604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:12 smithi121 bash[36902]: cluster 2022-04-22T21:08:10.838495+0000 mgr.y (mgr.34107) 7589 : cluster [DBG] pgmap v8604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:12.476Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:14 smithi115 bash[46934]: cluster 2022-04-22T21:08:12.839647+0000 mgr.y (mgr.34107) 7590 : cluster [DBG] pgmap v8605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:14 smithi115 bash[46934]: audit 2022-04-22T21:08:13.451596+0000 mon.a (mon.0) 11091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:14.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:14 smithi115 bash[45495]: cluster 2022-04-22T21:08:12.839647+0000 mgr.y (mgr.34107) 7590 : cluster [DBG] pgmap v8605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:14.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:14 smithi115 bash[45495]: audit 2022-04-22T21:08:13.451596+0000 mon.a (mon.0) 11091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:14 smithi121 bash[36902]: cluster 2022-04-22T21:08:12.839647+0000 mgr.y (mgr.34107) 7590 : cluster [DBG] pgmap v8605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:14 smithi121 bash[36902]: audit 2022-04-22T21:08:13.451596+0000 mon.a (mon.0) 11091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:14.354Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:15 smithi121 bash[36902]: audit 2022-04-22T21:08:14.199674+0000 mon.a (mon.0) 11092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:15.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:15 smithi115 bash[45495]: audit 2022-04-22T21:08:14.199674+0000 mon.a (mon.0) 11092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:15 smithi115 bash[46934]: audit 2022-04-22T21:08:14.199674+0000 mon.a (mon.0) 11092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:16 smithi121 bash[36902]: cluster 2022-04-22T21:08:14.840741+0000 mgr.y (mgr.34107) 7591 : cluster [DBG] pgmap v8606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:16 smithi121 bash[36902]: audit 2022-04-22T21:08:15.465754+0000 mon.a (mon.0) 11093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:16 smithi121 bash[36902]: audit 2022-04-22T21:08:15.830016+0000 mon.a (mon.0) 11094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:08:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:16 smithi121 bash[36902]: audit 2022-04-22T21:08:15.831829+0000 mon.a (mon.0) 11095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:08:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[45495]: cluster 2022-04-22T21:08:14.840741+0000 mgr.y (mgr.34107) 7591 : cluster [DBG] pgmap v8606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[45495]: audit 2022-04-22T21:08:15.465754+0000 mon.a (mon.0) 11093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[45495]: audit 2022-04-22T21:08:15.830016+0000 mon.a (mon.0) 11094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[45495]: audit 2022-04-22T21:08:15.831829+0000 mon.a (mon.0) 11095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[46934]: cluster 2022-04-22T21:08:14.840741+0000 mgr.y (mgr.34107) 7591 : cluster [DBG] pgmap v8606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[46934]: audit 2022-04-22T21:08:15.465754+0000 mon.a (mon.0) 11093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[46934]: audit 2022-04-22T21:08:15.830016+0000 mon.a (mon.0) 11094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:08:16.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:16 smithi115 bash[46934]: audit 2022-04-22T21:08:15.831829+0000 mon.a (mon.0) 11095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:08:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:18 smithi121 bash[36902]: cluster 2022-04-22T21:08:16.841837+0000 mgr.y (mgr.34107) 7592 : cluster [DBG] pgmap v8607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:18 smithi115 bash[45495]: cluster 2022-04-22T21:08:16.841837+0000 mgr.y (mgr.34107) 7592 : cluster [DBG] pgmap v8607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:18 smithi115 bash[46934]: cluster 2022-04-22T21:08:16.841837+0000 mgr.y (mgr.34107) 7592 : cluster [DBG] pgmap v8607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:19 smithi121 bash[36902]: audit 2022-04-22T21:08:18.455042+0000 mon.a (mon.0) 11096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:19.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:19 smithi115 bash[45495]: audit 2022-04-22T21:08:18.455042+0000 mon.a (mon.0) 11096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:19 smithi115 bash[46934]: audit 2022-04-22T21:08:18.455042+0000 mon.a (mon.0) 11096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:20 smithi121 bash[36902]: cluster 2022-04-22T21:08:18.843022+0000 mgr.y (mgr.34107) 7593 : cluster [DBG] pgmap v8608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:20 smithi121 bash[36902]: audit 2022-04-22T21:08:19.203151+0000 mon.a (mon.0) 11097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:20 smithi115 bash[45495]: cluster 2022-04-22T21:08:18.843022+0000 mgr.y (mgr.34107) 7593 : cluster [DBG] pgmap v8608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:20 smithi115 bash[45495]: audit 2022-04-22T21:08:19.203151+0000 mon.a (mon.0) 11097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:20 smithi115 bash[46934]: cluster 2022-04-22T21:08:18.843022+0000 mgr.y (mgr.34107) 7593 : cluster [DBG] pgmap v8608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:20 smithi115 bash[46934]: audit 2022-04-22T21:08:19.203151+0000 mon.a (mon.0) 11097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:21 smithi121 bash[36902]: audit 2022-04-22T21:08:20.468895+0000 mon.a (mon.0) 11098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:21 smithi115 bash[45495]: audit 2022-04-22T21:08:20.468895+0000 mon.a (mon.0) 11098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:21 smithi115 bash[46934]: audit 2022-04-22T21:08:20.468895+0000 mon.a (mon.0) 11098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:22.482 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:22 smithi115 bash[45495]: cluster 2022-04-22T21:08:20.844119+0000 mgr.y (mgr.34107) 7594 : cluster [DBG] pgmap v8609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:22.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:22 smithi115 bash[46934]: cluster 2022-04-22T21:08:20.844119+0000 mgr.y (mgr.34107) 7594 : cluster [DBG] pgmap v8609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:22 smithi121 bash[36902]: cluster 2022-04-22T21:08:20.844119+0000 mgr.y (mgr.34107) 7594 : cluster [DBG] pgmap v8609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:22.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:24 smithi115 bash[46934]: cluster 2022-04-22T21:08:22.845209+0000 mgr.y (mgr.34107) 7595 : cluster [DBG] pgmap v8610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:24 smithi115 bash[46934]: audit 2022-04-22T21:08:23.458105+0000 mon.a (mon.0) 11099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:24.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:24 smithi115 bash[45495]: cluster 2022-04-22T21:08:22.845209+0000 mgr.y (mgr.34107) 7595 : cluster [DBG] pgmap v8610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:24.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:24 smithi115 bash[45495]: audit 2022-04-22T21:08:23.458105+0000 mon.a (mon.0) 11099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:24 smithi121 bash[36902]: cluster 2022-04-22T21:08:22.845209+0000 mgr.y (mgr.34107) 7595 : cluster [DBG] pgmap v8610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:24 smithi121 bash[36902]: audit 2022-04-22T21:08:23.458105+0000 mon.a (mon.0) 11099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:24.353Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:25 smithi121 bash[36902]: audit 2022-04-22T21:08:24.206771+0000 mon.a (mon.0) 11100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:25 smithi115 bash[45495]: audit 2022-04-22T21:08:24.206771+0000 mon.a (mon.0) 11100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:25.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:25 smithi115 bash[46934]: audit 2022-04-22T21:08:24.206771+0000 mon.a (mon.0) 11100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:26 smithi121 bash[36902]: cluster 2022-04-22T21:08:24.846296+0000 mgr.y (mgr.34107) 7596 : cluster [DBG] pgmap v8611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:26 smithi121 bash[36902]: audit 2022-04-22T21:08:25.471953+0000 mon.a (mon.0) 11101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:26 smithi115 bash[45495]: cluster 2022-04-22T21:08:24.846296+0000 mgr.y (mgr.34107) 7596 : cluster [DBG] pgmap v8611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:26 smithi115 bash[45495]: audit 2022-04-22T21:08:25.471953+0000 mon.a (mon.0) 11101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:26 smithi115 bash[46934]: cluster 2022-04-22T21:08:24.846296+0000 mgr.y (mgr.34107) 7596 : cluster [DBG] pgmap v8611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:26 smithi115 bash[46934]: audit 2022-04-22T21:08:25.471953+0000 mon.a (mon.0) 11101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:28 smithi121 bash[36902]: cluster 2022-04-22T21:08:26.847336+0000 mgr.y (mgr.34107) 7597 : cluster [DBG] pgmap v8612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:28.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:28 smithi115 bash[45495]: cluster 2022-04-22T21:08:26.847336+0000 mgr.y (mgr.34107) 7597 : cluster [DBG] pgmap v8612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:28.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:28 smithi115 bash[46934]: cluster 2022-04-22T21:08:26.847336+0000 mgr.y (mgr.34107) 7597 : cluster [DBG] pgmap v8612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:29 smithi121 bash[36902]: audit 2022-04-22T21:08:28.461410+0000 mon.a (mon.0) 11102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:29 smithi115 bash[45495]: audit 2022-04-22T21:08:28.461410+0000 mon.a (mon.0) 11102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:29 smithi115 bash[46934]: audit 2022-04-22T21:08:28.461410+0000 mon.a (mon.0) 11102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:30 smithi121 bash[36902]: cluster 2022-04-22T21:08:28.848506+0000 mgr.y (mgr.34107) 7598 : cluster [DBG] pgmap v8613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:30 smithi121 bash[36902]: audit 2022-04-22T21:08:29.210319+0000 mon.a (mon.0) 11103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:30 smithi115 bash[45495]: cluster 2022-04-22T21:08:28.848506+0000 mgr.y (mgr.34107) 7598 : cluster [DBG] pgmap v8613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:30 smithi115 bash[45495]: audit 2022-04-22T21:08:29.210319+0000 mon.a (mon.0) 11103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:30 smithi115 bash[46934]: cluster 2022-04-22T21:08:28.848506+0000 mgr.y (mgr.34107) 7598 : cluster [DBG] pgmap v8613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:30 smithi115 bash[46934]: audit 2022-04-22T21:08:29.210319+0000 mon.a (mon.0) 11103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:31 smithi121 bash[36902]: audit 2022-04-22T21:08:30.474928+0000 mon.a (mon.0) 11104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:31 smithi115 bash[46934]: audit 2022-04-22T21:08:30.474928+0000 mon.a (mon.0) 11104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:31 smithi115 bash[45495]: audit 2022-04-22T21:08:30.474928+0000 mon.a (mon.0) 11104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:32.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:32 smithi115 bash[46934]: cluster 2022-04-22T21:08:30.849581+0000 mgr.y (mgr.34107) 7599 : cluster [DBG] pgmap v8614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:32.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:32 smithi115 bash[45495]: cluster 2022-04-22T21:08:30.849581+0000 mgr.y (mgr.34107) 7599 : cluster [DBG] pgmap v8614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:32 smithi121 bash[36902]: cluster 2022-04-22T21:08:30.849581+0000 mgr.y (mgr.34107) 7599 : cluster [DBG] pgmap v8614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:32.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:34 smithi121 bash[36902]: cluster 2022-04-22T21:08:32.850842+0000 mgr.y (mgr.34107) 7600 : cluster [DBG] pgmap v8615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:34 smithi121 bash[36902]: audit 2022-04-22T21:08:33.464486+0000 mon.a (mon.0) 11105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:34.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:34.355Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:34 smithi115 bash[46934]: cluster 2022-04-22T21:08:32.850842+0000 mgr.y (mgr.34107) 7600 : cluster [DBG] pgmap v8615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:34 smithi115 bash[46934]: audit 2022-04-22T21:08:33.464486+0000 mon.a (mon.0) 11105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:34 smithi115 bash[45495]: cluster 2022-04-22T21:08:32.850842+0000 mgr.y (mgr.34107) 7600 : cluster [DBG] pgmap v8615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:34 smithi115 bash[45495]: audit 2022-04-22T21:08:33.464486+0000 mon.a (mon.0) 11105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:35 smithi121 bash[36902]: audit 2022-04-22T21:08:34.214246+0000 mon.a (mon.0) 11106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:35.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:35 smithi115 bash[46934]: audit 2022-04-22T21:08:34.214246+0000 mon.a (mon.0) 11106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:35 smithi115 bash[45495]: audit 2022-04-22T21:08:34.214246+0000 mon.a (mon.0) 11106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:36 smithi121 bash[36902]: cluster 2022-04-22T21:08:34.852113+0000 mgr.y (mgr.34107) 7601 : cluster [DBG] pgmap v8616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:36 smithi121 bash[36902]: audit 2022-04-22T21:08:35.478116+0000 mon.a (mon.0) 11107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:36 smithi115 bash[46934]: cluster 2022-04-22T21:08:34.852113+0000 mgr.y (mgr.34107) 7601 : cluster [DBG] pgmap v8616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:36 smithi115 bash[46934]: audit 2022-04-22T21:08:35.478116+0000 mon.a (mon.0) 11107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:36 smithi115 bash[45495]: cluster 2022-04-22T21:08:34.852113+0000 mgr.y (mgr.34107) 7601 : cluster [DBG] pgmap v8616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:36 smithi115 bash[45495]: audit 2022-04-22T21:08:35.478116+0000 mon.a (mon.0) 11107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:38 smithi121 bash[36902]: cluster 2022-04-22T21:08:36.853368+0000 mgr.y (mgr.34107) 7602 : cluster [DBG] pgmap v8617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:38 smithi115 bash[45495]: cluster 2022-04-22T21:08:36.853368+0000 mgr.y (mgr.34107) 7602 : cluster [DBG] pgmap v8617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:38 smithi115 bash[46934]: cluster 2022-04-22T21:08:36.853368+0000 mgr.y (mgr.34107) 7602 : cluster [DBG] pgmap v8617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:39.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:39 smithi121 bash[36902]: audit 2022-04-22T21:08:38.467777+0000 mon.a (mon.0) 11108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:39 smithi115 bash[45495]: audit 2022-04-22T21:08:38.467777+0000 mon.a (mon.0) 11108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:39 smithi115 bash[46934]: audit 2022-04-22T21:08:38.467777+0000 mon.a (mon.0) 11108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:40 smithi121 bash[36902]: cluster 2022-04-22T21:08:38.854613+0000 mgr.y (mgr.34107) 7603 : cluster [DBG] pgmap v8618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:40 smithi121 bash[36902]: audit 2022-04-22T21:08:39.217749+0000 mon.a (mon.0) 11109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:40 smithi115 bash[45495]: cluster 2022-04-22T21:08:38.854613+0000 mgr.y (mgr.34107) 7603 : cluster [DBG] pgmap v8618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:40 smithi115 bash[45495]: audit 2022-04-22T21:08:39.217749+0000 mon.a (mon.0) 11109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:40 smithi115 bash[46934]: cluster 2022-04-22T21:08:38.854613+0000 mgr.y (mgr.34107) 7603 : cluster [DBG] pgmap v8618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:40 smithi115 bash[46934]: audit 2022-04-22T21:08:39.217749+0000 mon.a (mon.0) 11109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:40.960 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:08:41.305 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:08:41.305 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:08:41.306 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:08:41.306 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:08:41.306 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:08:41.307 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:08:41.308 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:08:41.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:41 smithi121 bash[36902]: audit 2022-04-22T21:08:40.480907+0000 mon.a (mon.0) 11110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:41 smithi115 bash[46934]: audit 2022-04-22T21:08:40.480907+0000 mon.a (mon.0) 11110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:41 smithi115 bash[45495]: audit 2022-04-22T21:08:40.480907+0000 mon.a (mon.0) 11110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:41.710 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:08:41.710 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:08:41.710 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:08:41.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:08:41.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:08:41.712 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:08:41.712 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:08:41.712 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:08:41.712 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:08:42.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[45495]: cluster 2022-04-22T21:08:40.855748+0000 mgr.y (mgr.34107) 7604 : cluster [DBG] pgmap v8619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:42.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[45495]: audit 2022-04-22T21:08:40.940147+0000 mgr.y (mgr.34107) 7605 : audit [DBG] from='client.43272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[45495]: audit 2022-04-22T21:08:41.280901+0000 mgr.y (mgr.34107) 7606 : audit [DBG] from='client.43278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[45495]: audit 2022-04-22T21:08:41.705603+0000 mon.a (mon.0) 11111 : audit [DBG] from='client.? 172.21.15.115:0/3086222376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:42.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[46934]: cluster 2022-04-22T21:08:40.855748+0000 mgr.y (mgr.34107) 7604 : cluster [DBG] pgmap v8619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:42.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[46934]: audit 2022-04-22T21:08:40.940147+0000 mgr.y (mgr.34107) 7605 : audit [DBG] from='client.43272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[46934]: audit 2022-04-22T21:08:41.280901+0000 mgr.y (mgr.34107) 7606 : audit [DBG] from='client.43278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[46934]: audit 2022-04-22T21:08:41.705603+0000 mon.a (mon.0) 11111 : audit [DBG] from='client.? 172.21.15.115:0/3086222376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:42 smithi121 bash[36902]: cluster 2022-04-22T21:08:40.855748+0000 mgr.y (mgr.34107) 7604 : cluster [DBG] pgmap v8619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:42 smithi121 bash[36902]: audit 2022-04-22T21:08:40.940147+0000 mgr.y (mgr.34107) 7605 : audit [DBG] from='client.43272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:42 smithi121 bash[36902]: audit 2022-04-22T21:08:41.280901+0000 mgr.y (mgr.34107) 7606 : audit [DBG] from='client.43278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:08:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:42 smithi121 bash[36902]: audit 2022-04-22T21:08:41.705603+0000 mon.a (mon.0) 11111 : audit [DBG] from='client.? 172.21.15.115:0/3086222376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:08:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:42.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: cluster 2022-04-22T21:08:42.856985+0000 mgr.y (mgr.34107) 7607 : cluster [DBG] pgmap v8620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: audit 2022-04-22T21:08:43.470753+0000 mon.a (mon.0) 11112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: audit 2022-04-22T21:08:43.623893+0000 mon.a (mon.0) 11113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: audit 2022-04-22T21:08:43.626658+0000 mon.a (mon.0) 11114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: audit 2022-04-22T21:08:43.628420+0000 mon.a (mon.0) 11115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:08:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:44 smithi121 bash[36902]: audit 2022-04-22T21:08:43.990639+0000 mon.a (mon.0) 11116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: cluster 2022-04-22T21:08:42.856985+0000 mgr.y (mgr.34107) 7607 : cluster [DBG] pgmap v8620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: audit 2022-04-22T21:08:43.470753+0000 mon.a (mon.0) 11112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: audit 2022-04-22T21:08:43.623893+0000 mon.a (mon.0) 11113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: audit 2022-04-22T21:08:43.626658+0000 mon.a (mon.0) 11114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: audit 2022-04-22T21:08:43.628420+0000 mon.a (mon.0) 11115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:08:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[45495]: audit 2022-04-22T21:08:43.990639+0000 mon.a (mon.0) 11116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: cluster 2022-04-22T21:08:42.856985+0000 mgr.y (mgr.34107) 7607 : cluster [DBG] pgmap v8620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: audit 2022-04-22T21:08:43.470753+0000 mon.a (mon.0) 11112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: audit 2022-04-22T21:08:43.623893+0000 mon.a (mon.0) 11113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: audit 2022-04-22T21:08:43.626658+0000 mon.a (mon.0) 11114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: audit 2022-04-22T21:08:43.628420+0000 mon.a (mon.0) 11115 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:08:44.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[46934]: audit 2022-04-22T21:08:43.990639+0000 mon.a (mon.0) 11116 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:08:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:44.356Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:45 smithi121 bash[36902]: audit 2022-04-22T21:08:44.221181+0000 mon.a (mon.0) 11117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:45 smithi115 bash[46934]: audit 2022-04-22T21:08:44.221181+0000 mon.a (mon.0) 11117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:45 smithi115 bash[45495]: audit 2022-04-22T21:08:44.221181+0000 mon.a (mon.0) 11117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:46 smithi121 bash[36902]: cluster 2022-04-22T21:08:44.858062+0000 mgr.y (mgr.34107) 7608 : cluster [DBG] pgmap v8621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:46 smithi121 bash[36902]: audit 2022-04-22T21:08:45.484016+0000 mon.a (mon.0) 11118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:46 smithi115 bash[45495]: cluster 2022-04-22T21:08:44.858062+0000 mgr.y (mgr.34107) 7608 : cluster [DBG] pgmap v8621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:46 smithi115 bash[45495]: audit 2022-04-22T21:08:45.484016+0000 mon.a (mon.0) 11118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:46 smithi115 bash[46934]: cluster 2022-04-22T21:08:44.858062+0000 mgr.y (mgr.34107) 7608 : cluster [DBG] pgmap v8621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:46 smithi115 bash[46934]: audit 2022-04-22T21:08:45.484016+0000 mon.a (mon.0) 11118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:48 smithi121 bash[36902]: cluster 2022-04-22T21:08:46.859077+0000 mgr.y (mgr.34107) 7609 : cluster [DBG] pgmap v8622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:48 smithi115 bash[45495]: cluster 2022-04-22T21:08:46.859077+0000 mgr.y (mgr.34107) 7609 : cluster [DBG] pgmap v8622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:48 smithi115 bash[46934]: cluster 2022-04-22T21:08:46.859077+0000 mgr.y (mgr.34107) 7609 : cluster [DBG] pgmap v8622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:49 smithi121 bash[36902]: audit 2022-04-22T21:08:48.474173+0000 mon.a (mon.0) 11119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:49 smithi115 bash[45495]: audit 2022-04-22T21:08:48.474173+0000 mon.a (mon.0) 11119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:49 smithi115 bash[46934]: audit 2022-04-22T21:08:48.474173+0000 mon.a (mon.0) 11119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:50 smithi121 bash[36902]: cluster 2022-04-22T21:08:48.860234+0000 mgr.y (mgr.34107) 7610 : cluster [DBG] pgmap v8623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:50 smithi121 bash[36902]: audit 2022-04-22T21:08:49.224876+0000 mon.a (mon.0) 11120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:50 smithi115 bash[45495]: cluster 2022-04-22T21:08:48.860234+0000 mgr.y (mgr.34107) 7610 : cluster [DBG] pgmap v8623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:50 smithi115 bash[45495]: audit 2022-04-22T21:08:49.224876+0000 mon.a (mon.0) 11120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:50 smithi115 bash[46934]: cluster 2022-04-22T21:08:48.860234+0000 mgr.y (mgr.34107) 7610 : cluster [DBG] pgmap v8623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:50 smithi115 bash[46934]: audit 2022-04-22T21:08:49.224876+0000 mon.a (mon.0) 11120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:51 smithi121 bash[36902]: audit 2022-04-22T21:08:50.487098+0000 mon.a (mon.0) 11121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:51 smithi115 bash[45495]: audit 2022-04-22T21:08:50.487098+0000 mon.a (mon.0) 11121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:51 smithi115 bash[46934]: audit 2022-04-22T21:08:50.487098+0000 mon.a (mon.0) 11121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:52.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:52 smithi115 bash[45495]: cluster 2022-04-22T21:08:50.861324+0000 mgr.y (mgr.34107) 7611 : cluster [DBG] pgmap v8624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:52.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:52 smithi115 bash[46934]: cluster 2022-04-22T21:08:50.861324+0000 mgr.y (mgr.34107) 7611 : cluster [DBG] pgmap v8624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:52 smithi121 bash[36902]: cluster 2022-04-22T21:08:50.861324+0000 mgr.y (mgr.34107) 7611 : cluster [DBG] pgmap v8624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:52.480Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:54 smithi121 bash[36902]: cluster 2022-04-22T21:08:52.862361+0000 mgr.y (mgr.34107) 7612 : cluster [DBG] pgmap v8625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:54 smithi121 bash[36902]: audit 2022-04-22T21:08:53.477405+0000 mon.a (mon.0) 11122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:54.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:08:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:08:54.357Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:08:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:54 smithi115 bash[45495]: cluster 2022-04-22T21:08:52.862361+0000 mgr.y (mgr.34107) 7612 : cluster [DBG] pgmap v8625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:54 smithi115 bash[45495]: audit 2022-04-22T21:08:53.477405+0000 mon.a (mon.0) 11122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:54 smithi115 bash[46934]: cluster 2022-04-22T21:08:52.862361+0000 mgr.y (mgr.34107) 7612 : cluster [DBG] pgmap v8625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:54 smithi115 bash[46934]: audit 2022-04-22T21:08:53.477405+0000 mon.a (mon.0) 11122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:55 smithi121 bash[36902]: audit 2022-04-22T21:08:54.228482+0000 mon.a (mon.0) 11123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:55.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:55 smithi115 bash[45495]: audit 2022-04-22T21:08:54.228482+0000 mon.a (mon.0) 11123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:55 smithi115 bash[46934]: audit 2022-04-22T21:08:54.228482+0000 mon.a (mon.0) 11123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:08:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:56 smithi121 bash[36902]: cluster 2022-04-22T21:08:54.863430+0000 mgr.y (mgr.34107) 7613 : cluster [DBG] pgmap v8626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:56 smithi121 bash[36902]: audit 2022-04-22T21:08:55.490146+0000 mon.a (mon.0) 11124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:56 smithi115 bash[45495]: cluster 2022-04-22T21:08:54.863430+0000 mgr.y (mgr.34107) 7613 : cluster [DBG] pgmap v8626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:56 smithi115 bash[45495]: audit 2022-04-22T21:08:55.490146+0000 mon.a (mon.0) 11124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:56 smithi115 bash[46934]: cluster 2022-04-22T21:08:54.863430+0000 mgr.y (mgr.34107) 7613 : cluster [DBG] pgmap v8626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:56 smithi115 bash[46934]: audit 2022-04-22T21:08:55.490146+0000 mon.a (mon.0) 11124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:08:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:58 smithi121 bash[36902]: cluster 2022-04-22T21:08:56.864506+0000 mgr.y (mgr.34107) 7614 : cluster [DBG] pgmap v8627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:58 smithi115 bash[45495]: cluster 2022-04-22T21:08:56.864506+0000 mgr.y (mgr.34107) 7614 : cluster [DBG] pgmap v8627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:58 smithi115 bash[46934]: cluster 2022-04-22T21:08:56.864506+0000 mgr.y (mgr.34107) 7614 : cluster [DBG] pgmap v8627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:08:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:08:59 smithi121 bash[36902]: audit 2022-04-22T21:08:58.480540+0000 mon.a (mon.0) 11125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:08:59 smithi115 bash[45495]: audit 2022-04-22T21:08:58.480540+0000 mon.a (mon.0) 11125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:08:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:08:59 smithi115 bash[46934]: audit 2022-04-22T21:08:58.480540+0000 mon.a (mon.0) 11125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:00 smithi121 bash[36902]: cluster 2022-04-22T21:08:58.865616+0000 mgr.y (mgr.34107) 7615 : cluster [DBG] pgmap v8628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:00 smithi121 bash[36902]: audit 2022-04-22T21:08:59.234174+0000 mon.a (mon.0) 11126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:00 smithi115 bash[45495]: cluster 2022-04-22T21:08:58.865616+0000 mgr.y (mgr.34107) 7615 : cluster [DBG] pgmap v8628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:00 smithi115 bash[45495]: audit 2022-04-22T21:08:59.234174+0000 mon.a (mon.0) 11126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:00 smithi115 bash[46934]: cluster 2022-04-22T21:08:58.865616+0000 mgr.y (mgr.34107) 7615 : cluster [DBG] pgmap v8628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:00 smithi115 bash[46934]: audit 2022-04-22T21:08:59.234174+0000 mon.a (mon.0) 11126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:01 smithi121 bash[36902]: audit 2022-04-22T21:09:00.493173+0000 mon.a (mon.0) 11127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:01 smithi115 bash[45495]: audit 2022-04-22T21:09:00.493173+0000 mon.a (mon.0) 11127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:01 smithi115 bash[46934]: audit 2022-04-22T21:09:00.493173+0000 mon.a (mon.0) 11127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:02 smithi121 bash[36902]: cluster 2022-04-22T21:09:00.866689+0000 mgr.y (mgr.34107) 7616 : cluster [DBG] pgmap v8629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:02 smithi115 bash[46934]: cluster 2022-04-22T21:09:00.866689+0000 mgr.y (mgr.34107) 7616 : cluster [DBG] pgmap v8629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:02.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:02.479Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:02.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:02 smithi115 bash[45495]: cluster 2022-04-22T21:09:00.866689+0000 mgr.y (mgr.34107) 7616 : cluster [DBG] pgmap v8629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:04.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:04 smithi121 bash[36902]: cluster 2022-04-22T21:09:02.867851+0000 mgr.y (mgr.34107) 7617 : cluster [DBG] pgmap v8630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:04 smithi121 bash[36902]: audit 2022-04-22T21:09:03.483735+0000 mon.a (mon.0) 11128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:04 smithi121 bash[36902]: audit 2022-04-22T21:09:04.235904+0000 mon.a (mon.0) 11129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:04.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:04.357Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[45495]: cluster 2022-04-22T21:09:02.867851+0000 mgr.y (mgr.34107) 7617 : cluster [DBG] pgmap v8630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[45495]: audit 2022-04-22T21:09:03.483735+0000 mon.a (mon.0) 11128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[45495]: audit 2022-04-22T21:09:04.235904+0000 mon.a (mon.0) 11129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[46934]: cluster 2022-04-22T21:09:02.867851+0000 mgr.y (mgr.34107) 7617 : cluster [DBG] pgmap v8630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[46934]: audit 2022-04-22T21:09:03.483735+0000 mon.a (mon.0) 11128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:04.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:04 smithi115 bash[46934]: audit 2022-04-22T21:09:04.235904+0000 mon.a (mon.0) 11129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:06 smithi121 bash[36902]: cluster 2022-04-22T21:09:04.868871+0000 mgr.y (mgr.34107) 7618 : cluster [DBG] pgmap v8631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:06 smithi121 bash[36902]: audit 2022-04-22T21:09:05.496205+0000 mon.a (mon.0) 11130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:06 smithi115 bash[45495]: cluster 2022-04-22T21:09:04.868871+0000 mgr.y (mgr.34107) 7618 : cluster [DBG] pgmap v8631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:06 smithi115 bash[45495]: audit 2022-04-22T21:09:05.496205+0000 mon.a (mon.0) 11130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:06 smithi115 bash[46934]: cluster 2022-04-22T21:09:04.868871+0000 mgr.y (mgr.34107) 7618 : cluster [DBG] pgmap v8631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:06 smithi115 bash[46934]: audit 2022-04-22T21:09:05.496205+0000 mon.a (mon.0) 11130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:08 smithi121 bash[36902]: cluster 2022-04-22T21:09:06.870190+0000 mgr.y (mgr.34107) 7619 : cluster [DBG] pgmap v8632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:08 smithi115 bash[45495]: cluster 2022-04-22T21:09:06.870190+0000 mgr.y (mgr.34107) 7619 : cluster [DBG] pgmap v8632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:08.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:08 smithi115 bash[46934]: cluster 2022-04-22T21:09:06.870190+0000 mgr.y (mgr.34107) 7619 : cluster [DBG] pgmap v8632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:09 smithi115 bash[45495]: audit 2022-04-22T21:09:08.486868+0000 mon.a (mon.0) 11131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:09 smithi115 bash[45495]: audit 2022-04-22T21:09:09.239572+0000 mon.a (mon.0) 11132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:09 smithi115 bash[46934]: audit 2022-04-22T21:09:08.486868+0000 mon.a (mon.0) 11131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:09 smithi115 bash[46934]: audit 2022-04-22T21:09:09.239572+0000 mon.a (mon.0) 11132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:09 smithi121 bash[36902]: audit 2022-04-22T21:09:08.486868+0000 mon.a (mon.0) 11131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:09 smithi121 bash[36902]: audit 2022-04-22T21:09:09.239572+0000 mon.a (mon.0) 11132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:10 smithi115 bash[45495]: cluster 2022-04-22T21:09:08.871322+0000 mgr.y (mgr.34107) 7620 : cluster [DBG] pgmap v8633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:10 smithi115 bash[46934]: cluster 2022-04-22T21:09:08.871322+0000 mgr.y (mgr.34107) 7620 : cluster [DBG] pgmap v8633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:10 smithi121 bash[36902]: cluster 2022-04-22T21:09:08.871322+0000 mgr.y (mgr.34107) 7620 : cluster [DBG] pgmap v8633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:11 smithi115 bash[46934]: audit 2022-04-22T21:09:10.499143+0000 mon.a (mon.0) 11133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:11 smithi115 bash[45495]: audit 2022-04-22T21:09:10.499143+0000 mon.a (mon.0) 11133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:11 smithi121 bash[36902]: audit 2022-04-22T21:09:10.499143+0000 mon.a (mon.0) 11133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:12.093 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:09:12.438 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:09:12.439 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:09:12.440 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:09:12.441 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:09:12.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:12.480Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:12.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:12 smithi115 bash[45495]: cluster 2022-04-22T21:09:10.872411+0000 mgr.y (mgr.34107) 7621 : cluster [DBG] pgmap v8634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:12 smithi115 bash[46934]: cluster 2022-04-22T21:09:10.872411+0000 mgr.y (mgr.34107) 7621 : cluster [DBG] pgmap v8634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:12 smithi121 bash[36902]: cluster 2022-04-22T21:09:10.872411+0000 mgr.y (mgr.34107) 7621 : cluster [DBG] pgmap v8634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:12.838 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:09:12.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:09:12.840 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:09:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[46934]: audit 2022-04-22T21:09:12.070556+0000 mgr.y (mgr.34107) 7622 : audit [DBG] from='client.43290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[46934]: audit 2022-04-22T21:09:12.414807+0000 mgr.y (mgr.34107) 7623 : audit [DBG] from='client.53227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[46934]: audit 2022-04-22T21:09:12.833938+0000 mon.a (mon.0) 11134 : audit [DBG] from='client.? 172.21.15.115:0/2346527729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[45495]: audit 2022-04-22T21:09:12.070556+0000 mgr.y (mgr.34107) 7622 : audit [DBG] from='client.43290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[45495]: audit 2022-04-22T21:09:12.414807+0000 mgr.y (mgr.34107) 7623 : audit [DBG] from='client.53227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:13 smithi115 bash[45495]: audit 2022-04-22T21:09:12.833938+0000 mon.a (mon.0) 11134 : audit [DBG] from='client.? 172.21.15.115:0/2346527729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:13.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:13 smithi121 bash[36902]: audit 2022-04-22T21:09:12.070556+0000 mgr.y (mgr.34107) 7622 : audit [DBG] from='client.43290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:13 smithi121 bash[36902]: audit 2022-04-22T21:09:12.414807+0000 mgr.y (mgr.34107) 7623 : audit [DBG] from='client.53227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:13 smithi121 bash[36902]: audit 2022-04-22T21:09:12.833938+0000 mon.a (mon.0) 11134 : audit [DBG] from='client.? 172.21.15.115:0/2346527729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:14.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:14.358Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[45495]: cluster 2022-04-22T21:09:12.873522+0000 mgr.y (mgr.34107) 7624 : cluster [DBG] pgmap v8635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[45495]: audit 2022-04-22T21:09:13.489937+0000 mon.a (mon.0) 11135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:14.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[45495]: audit 2022-04-22T21:09:14.243352+0000 mon.a (mon.0) 11136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[46934]: cluster 2022-04-22T21:09:12.873522+0000 mgr.y (mgr.34107) 7624 : cluster [DBG] pgmap v8635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[46934]: audit 2022-04-22T21:09:13.489937+0000 mon.a (mon.0) 11135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:14 smithi115 bash[46934]: audit 2022-04-22T21:09:14.243352+0000 mon.a (mon.0) 11136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:14 smithi121 bash[36902]: cluster 2022-04-22T21:09:12.873522+0000 mgr.y (mgr.34107) 7624 : cluster [DBG] pgmap v8635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:14 smithi121 bash[36902]: audit 2022-04-22T21:09:13.489937+0000 mon.a (mon.0) 11135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:14 smithi121 bash[36902]: audit 2022-04-22T21:09:14.243352+0000 mon.a (mon.0) 11136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:16 smithi121 bash[36902]: cluster 2022-04-22T21:09:14.874633+0000 mgr.y (mgr.34107) 7625 : cluster [DBG] pgmap v8636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:16 smithi121 bash[36902]: audit 2022-04-22T21:09:15.501962+0000 mon.a (mon.0) 11137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:16 smithi121 bash[36902]: audit 2022-04-22T21:09:15.842894+0000 mon.a (mon.0) 11138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:09:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:16 smithi121 bash[36902]: audit 2022-04-22T21:09:15.843607+0000 mon.a (mon.0) 11139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:09:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[45495]: cluster 2022-04-22T21:09:14.874633+0000 mgr.y (mgr.34107) 7625 : cluster [DBG] pgmap v8636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[45495]: audit 2022-04-22T21:09:15.501962+0000 mon.a (mon.0) 11137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[45495]: audit 2022-04-22T21:09:15.842894+0000 mon.a (mon.0) 11138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[45495]: audit 2022-04-22T21:09:15.843607+0000 mon.a (mon.0) 11139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[46934]: cluster 2022-04-22T21:09:14.874633+0000 mgr.y (mgr.34107) 7625 : cluster [DBG] pgmap v8636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[46934]: audit 2022-04-22T21:09:15.501962+0000 mon.a (mon.0) 11137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[46934]: audit 2022-04-22T21:09:15.842894+0000 mon.a (mon.0) 11138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:09:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:16 smithi115 bash[46934]: audit 2022-04-22T21:09:15.843607+0000 mon.a (mon.0) 11139 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:09:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:18 smithi121 bash[36902]: cluster 2022-04-22T21:09:16.875780+0000 mgr.y (mgr.34107) 7626 : cluster [DBG] pgmap v8637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:18 smithi115 bash[45495]: cluster 2022-04-22T21:09:16.875780+0000 mgr.y (mgr.34107) 7626 : cluster [DBG] pgmap v8637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:18 smithi115 bash[46934]: cluster 2022-04-22T21:09:16.875780+0000 mgr.y (mgr.34107) 7626 : cluster [DBG] pgmap v8637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:19 smithi121 bash[36902]: audit 2022-04-22T21:09:18.492961+0000 mon.a (mon.0) 11140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:19 smithi121 bash[36902]: audit 2022-04-22T21:09:19.246950+0000 mon.a (mon.0) 11141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:19 smithi115 bash[46934]: audit 2022-04-22T21:09:18.492961+0000 mon.a (mon.0) 11140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:19 smithi115 bash[46934]: audit 2022-04-22T21:09:19.246950+0000 mon.a (mon.0) 11141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:19 smithi115 bash[45495]: audit 2022-04-22T21:09:18.492961+0000 mon.a (mon.0) 11140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:19 smithi115 bash[45495]: audit 2022-04-22T21:09:19.246950+0000 mon.a (mon.0) 11141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:20 smithi121 bash[36902]: cluster 2022-04-22T21:09:18.876935+0000 mgr.y (mgr.34107) 7627 : cluster [DBG] pgmap v8638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:20 smithi115 bash[45495]: cluster 2022-04-22T21:09:18.876935+0000 mgr.y (mgr.34107) 7627 : cluster [DBG] pgmap v8638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:20 smithi115 bash[46934]: cluster 2022-04-22T21:09:18.876935+0000 mgr.y (mgr.34107) 7627 : cluster [DBG] pgmap v8638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:21 smithi121 bash[36902]: audit 2022-04-22T21:09:20.504869+0000 mon.a (mon.0) 11142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:21 smithi115 bash[45495]: audit 2022-04-22T21:09:20.504869+0000 mon.a (mon.0) 11142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:21 smithi115 bash[46934]: audit 2022-04-22T21:09:20.504869+0000 mon.a (mon.0) 11142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:22 smithi121 bash[36902]: cluster 2022-04-22T21:09:20.878034+0000 mgr.y (mgr.34107) 7628 : cluster [DBG] pgmap v8639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:22 smithi115 bash[45495]: cluster 2022-04-22T21:09:20.878034+0000 mgr.y (mgr.34107) 7628 : cluster [DBG] pgmap v8639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:22.482Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:22 smithi115 bash[46934]: cluster 2022-04-22T21:09:20.878034+0000 mgr.y (mgr.34107) 7628 : cluster [DBG] pgmap v8639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:24 smithi121 bash[36902]: cluster 2022-04-22T21:09:22.879175+0000 mgr.y (mgr.34107) 7629 : cluster [DBG] pgmap v8640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:24 smithi121 bash[36902]: audit 2022-04-22T21:09:23.496104+0000 mon.a (mon.0) 11143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:24 smithi121 bash[36902]: audit 2022-04-22T21:09:24.250602+0000 mon.a (mon.0) 11144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:24.359Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[45495]: cluster 2022-04-22T21:09:22.879175+0000 mgr.y (mgr.34107) 7629 : cluster [DBG] pgmap v8640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[45495]: audit 2022-04-22T21:09:23.496104+0000 mon.a (mon.0) 11143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[45495]: audit 2022-04-22T21:09:24.250602+0000 mon.a (mon.0) 11144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[46934]: cluster 2022-04-22T21:09:22.879175+0000 mgr.y (mgr.34107) 7629 : cluster [DBG] pgmap v8640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[46934]: audit 2022-04-22T21:09:23.496104+0000 mon.a (mon.0) 11143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:24.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:24 smithi115 bash[46934]: audit 2022-04-22T21:09:24.250602+0000 mon.a (mon.0) 11144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:26 smithi121 bash[36902]: cluster 2022-04-22T21:09:24.880303+0000 mgr.y (mgr.34107) 7630 : cluster [DBG] pgmap v8641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:26 smithi121 bash[36902]: audit 2022-04-22T21:09:25.507839+0000 mon.a (mon.0) 11145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:26 smithi115 bash[45495]: cluster 2022-04-22T21:09:24.880303+0000 mgr.y (mgr.34107) 7630 : cluster [DBG] pgmap v8641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:26.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:26 smithi115 bash[45495]: audit 2022-04-22T21:09:25.507839+0000 mon.a (mon.0) 11145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:26.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:26 smithi115 bash[46934]: cluster 2022-04-22T21:09:24.880303+0000 mgr.y (mgr.34107) 7630 : cluster [DBG] pgmap v8641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:26.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:26 smithi115 bash[46934]: audit 2022-04-22T21:09:25.507839+0000 mon.a (mon.0) 11145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:28 smithi121 bash[36902]: cluster 2022-04-22T21:09:26.881436+0000 mgr.y (mgr.34107) 7631 : cluster [DBG] pgmap v8642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:28 smithi115 bash[45495]: cluster 2022-04-22T21:09:26.881436+0000 mgr.y (mgr.34107) 7631 : cluster [DBG] pgmap v8642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:28.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:28 smithi115 bash[46934]: cluster 2022-04-22T21:09:26.881436+0000 mgr.y (mgr.34107) 7631 : cluster [DBG] pgmap v8642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:29.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:29 smithi121 bash[36902]: audit 2022-04-22T21:09:28.499222+0000 mon.a (mon.0) 11146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:29 smithi121 bash[36902]: audit 2022-04-22T21:09:29.254262+0000 mon.a (mon.0) 11147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:29 smithi115 bash[45495]: audit 2022-04-22T21:09:28.499222+0000 mon.a (mon.0) 11146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:29.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:29 smithi115 bash[45495]: audit 2022-04-22T21:09:29.254262+0000 mon.a (mon.0) 11147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:29 smithi115 bash[46934]: audit 2022-04-22T21:09:28.499222+0000 mon.a (mon.0) 11146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:29 smithi115 bash[46934]: audit 2022-04-22T21:09:29.254262+0000 mon.a (mon.0) 11147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:30.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:30 smithi121 bash[36902]: cluster 2022-04-22T21:09:28.882577+0000 mgr.y (mgr.34107) 7632 : cluster [DBG] pgmap v8643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:30 smithi115 bash[45495]: cluster 2022-04-22T21:09:28.882577+0000 mgr.y (mgr.34107) 7632 : cluster [DBG] pgmap v8643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:30 smithi115 bash[46934]: cluster 2022-04-22T21:09:28.882577+0000 mgr.y (mgr.34107) 7632 : cluster [DBG] pgmap v8643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:31.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:31 smithi121 bash[36902]: audit 2022-04-22T21:09:30.510888+0000 mon.a (mon.0) 11148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:31 smithi115 bash[45495]: audit 2022-04-22T21:09:30.510888+0000 mon.a (mon.0) 11148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:31 smithi115 bash[46934]: audit 2022-04-22T21:09:30.510888+0000 mon.a (mon.0) 11148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:32 smithi121 bash[36902]: cluster 2022-04-22T21:09:30.883731+0000 mgr.y (mgr.34107) 7633 : cluster [DBG] pgmap v8644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:32 smithi115 bash[46934]: cluster 2022-04-22T21:09:30.883731+0000 mgr.y (mgr.34107) 7633 : cluster [DBG] pgmap v8644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:32.481Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:32.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:32 smithi115 bash[45495]: cluster 2022-04-22T21:09:30.883731+0000 mgr.y (mgr.34107) 7633 : cluster [DBG] pgmap v8644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:34 smithi121 bash[36902]: cluster 2022-04-22T21:09:32.884788+0000 mgr.y (mgr.34107) 7634 : cluster [DBG] pgmap v8645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:34 smithi121 bash[36902]: audit 2022-04-22T21:09:33.502600+0000 mon.a (mon.0) 11149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:34 smithi121 bash[36902]: audit 2022-04-22T21:09:34.258120+0000 mon.a (mon.0) 11150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:34.360Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[46934]: cluster 2022-04-22T21:09:32.884788+0000 mgr.y (mgr.34107) 7634 : cluster [DBG] pgmap v8645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[46934]: audit 2022-04-22T21:09:33.502600+0000 mon.a (mon.0) 11149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[46934]: audit 2022-04-22T21:09:34.258120+0000 mon.a (mon.0) 11150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[45495]: cluster 2022-04-22T21:09:32.884788+0000 mgr.y (mgr.34107) 7634 : cluster [DBG] pgmap v8645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[45495]: audit 2022-04-22T21:09:33.502600+0000 mon.a (mon.0) 11149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:34.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:34 smithi115 bash[45495]: audit 2022-04-22T21:09:34.258120+0000 mon.a (mon.0) 11150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:36 smithi121 bash[36902]: cluster 2022-04-22T21:09:34.885957+0000 mgr.y (mgr.34107) 7635 : cluster [DBG] pgmap v8646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:36 smithi121 bash[36902]: audit 2022-04-22T21:09:35.513910+0000 mon.a (mon.0) 11151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:36 smithi115 bash[45495]: cluster 2022-04-22T21:09:34.885957+0000 mgr.y (mgr.34107) 7635 : cluster [DBG] pgmap v8646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:36 smithi115 bash[45495]: audit 2022-04-22T21:09:35.513910+0000 mon.a (mon.0) 11151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:36 smithi115 bash[46934]: cluster 2022-04-22T21:09:34.885957+0000 mgr.y (mgr.34107) 7635 : cluster [DBG] pgmap v8646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:36 smithi115 bash[46934]: audit 2022-04-22T21:09:35.513910+0000 mon.a (mon.0) 11151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:38 smithi121 bash[36902]: cluster 2022-04-22T21:09:36.887085+0000 mgr.y (mgr.34107) 7636 : cluster [DBG] pgmap v8647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:38 smithi115 bash[45495]: cluster 2022-04-22T21:09:36.887085+0000 mgr.y (mgr.34107) 7636 : cluster [DBG] pgmap v8647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:38 smithi115 bash[46934]: cluster 2022-04-22T21:09:36.887085+0000 mgr.y (mgr.34107) 7636 : cluster [DBG] pgmap v8647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:39 smithi121 bash[36902]: audit 2022-04-22T21:09:38.505734+0000 mon.a (mon.0) 11152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:39 smithi121 bash[36902]: audit 2022-04-22T21:09:39.262038+0000 mon.a (mon.0) 11153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:39 smithi115 bash[45495]: audit 2022-04-22T21:09:38.505734+0000 mon.a (mon.0) 11152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:39 smithi115 bash[45495]: audit 2022-04-22T21:09:39.262038+0000 mon.a (mon.0) 11153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:39 smithi115 bash[46934]: audit 2022-04-22T21:09:38.505734+0000 mon.a (mon.0) 11152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:39 smithi115 bash[46934]: audit 2022-04-22T21:09:39.262038+0000 mon.a (mon.0) 11153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:40 smithi121 bash[36902]: cluster 2022-04-22T21:09:38.888254+0000 mgr.y (mgr.34107) 7637 : cluster [DBG] pgmap v8648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:40 smithi115 bash[45495]: cluster 2022-04-22T21:09:38.888254+0000 mgr.y (mgr.34107) 7637 : cluster [DBG] pgmap v8648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:40 smithi115 bash[46934]: cluster 2022-04-22T21:09:38.888254+0000 mgr.y (mgr.34107) 7637 : cluster [DBG] pgmap v8648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:41 smithi121 bash[36902]: audit 2022-04-22T21:09:40.517111+0000 mon.a (mon.0) 11154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:41 smithi115 bash[45495]: audit 2022-04-22T21:09:40.517111+0000 mon.a (mon.0) 11154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:41.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:41 smithi115 bash[46934]: audit 2022-04-22T21:09:40.517111+0000 mon.a (mon.0) 11154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:42 smithi121 bash[36902]: cluster 2022-04-22T21:09:40.889352+0000 mgr.y (mgr.34107) 7638 : cluster [DBG] pgmap v8649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:42 smithi115 bash[46934]: cluster 2022-04-22T21:09:40.889352+0000 mgr.y (mgr.34107) 7638 : cluster [DBG] pgmap v8649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:42.482Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:42 smithi115 bash[45495]: cluster 2022-04-22T21:09:40.889352+0000 mgr.y (mgr.34107) 7638 : cluster [DBG] pgmap v8649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:43.206 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:09:43.550 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:09:43.551 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:09:43.552 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:09:43.552 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:09:43.552 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:09:43.954 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:09:43.954 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:09:43.954 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:09:43.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:09:43.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:09:43.957 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:09:43.957 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:09:43.957 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:09:43.957 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: cluster 2022-04-22T21:09:42.890378+0000 mgr.y (mgr.34107) 7639 : cluster [DBG] pgmap v8650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:43.186783+0000 mgr.y (mgr.34107) 7640 : audit [DBG] from='client.53239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:43.508568+0000 mon.a (mon.0) 11155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:43.531416+0000 mgr.y (mgr.34107) 7641 : audit [DBG] from='client.53245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:43.949973+0000 mon.a (mon.0) 11156 : audit [DBG] from='client.? 172.21.15.115:0/4078295724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:44.000783+0000 mon.a (mon.0) 11157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:09:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:44.003674+0000 mon.a (mon.0) 11158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:09:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:44.005350+0000 mon.a (mon.0) 11159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:09:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:44.265500+0000 mon.a (mon.0) 11160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:44 smithi121 bash[36902]: audit 2022-04-22T21:09:44.371737+0000 mon.a (mon.0) 11161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:09:44.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:44.364Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: cluster 2022-04-22T21:09:42.890378+0000 mgr.y (mgr.34107) 7639 : cluster [DBG] pgmap v8650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:43.186783+0000 mgr.y (mgr.34107) 7640 : audit [DBG] from='client.53239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:43.508568+0000 mon.a (mon.0) 11155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:43.531416+0000 mgr.y (mgr.34107) 7641 : audit [DBG] from='client.53245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:43.949973+0000 mon.a (mon.0) 11156 : audit [DBG] from='client.? 172.21.15.115:0/4078295724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:44.000783+0000 mon.a (mon.0) 11157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:44.003674+0000 mon.a (mon.0) 11158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:09:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:44.005350+0000 mon.a (mon.0) 11159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:44.265500+0000 mon.a (mon.0) 11160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[45495]: audit 2022-04-22T21:09:44.371737+0000 mon.a (mon.0) 11161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: cluster 2022-04-22T21:09:42.890378+0000 mgr.y (mgr.34107) 7639 : cluster [DBG] pgmap v8650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:43.186783+0000 mgr.y (mgr.34107) 7640 : audit [DBG] from='client.53239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:43.508568+0000 mon.a (mon.0) 11155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:43.531416+0000 mgr.y (mgr.34107) 7641 : audit [DBG] from='client.53245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:43.949973+0000 mon.a (mon.0) 11156 : audit [DBG] from='client.? 172.21.15.115:0/4078295724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:09:44.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:44.000783+0000 mon.a (mon.0) 11157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:09:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:44.003674+0000 mon.a (mon.0) 11158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:09:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:44.005350+0000 mon.a (mon.0) 11159 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:09:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:44.265500+0000 mon.a (mon.0) 11160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:44.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:44 smithi115 bash[46934]: audit 2022-04-22T21:09:44.371737+0000 mon.a (mon.0) 11161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:09:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:46 smithi121 bash[36902]: cluster 2022-04-22T21:09:44.891426+0000 mgr.y (mgr.34107) 7642 : cluster [DBG] pgmap v8651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:46 smithi121 bash[36902]: audit 2022-04-22T21:09:45.520103+0000 mon.a (mon.0) 11162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:46 smithi115 bash[45495]: cluster 2022-04-22T21:09:44.891426+0000 mgr.y (mgr.34107) 7642 : cluster [DBG] pgmap v8651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:46 smithi115 bash[45495]: audit 2022-04-22T21:09:45.520103+0000 mon.a (mon.0) 11162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:46 smithi115 bash[46934]: cluster 2022-04-22T21:09:44.891426+0000 mgr.y (mgr.34107) 7642 : cluster [DBG] pgmap v8651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:46 smithi115 bash[46934]: audit 2022-04-22T21:09:45.520103+0000 mon.a (mon.0) 11162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:48 smithi121 bash[36902]: cluster 2022-04-22T21:09:46.892570+0000 mgr.y (mgr.34107) 7643 : cluster [DBG] pgmap v8652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:48 smithi115 bash[45495]: cluster 2022-04-22T21:09:46.892570+0000 mgr.y (mgr.34107) 7643 : cluster [DBG] pgmap v8652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:48 smithi115 bash[46934]: cluster 2022-04-22T21:09:46.892570+0000 mgr.y (mgr.34107) 7643 : cluster [DBG] pgmap v8652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:49 smithi121 bash[36902]: audit 2022-04-22T21:09:48.511399+0000 mon.a (mon.0) 11163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:49 smithi121 bash[36902]: audit 2022-04-22T21:09:49.268762+0000 mon.a (mon.0) 11164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:49 smithi115 bash[45495]: audit 2022-04-22T21:09:48.511399+0000 mon.a (mon.0) 11163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:49 smithi115 bash[45495]: audit 2022-04-22T21:09:49.268762+0000 mon.a (mon.0) 11164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:49 smithi115 bash[46934]: audit 2022-04-22T21:09:48.511399+0000 mon.a (mon.0) 11163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:49 smithi115 bash[46934]: audit 2022-04-22T21:09:49.268762+0000 mon.a (mon.0) 11164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:50 smithi121 bash[36902]: cluster 2022-04-22T21:09:48.893770+0000 mgr.y (mgr.34107) 7644 : cluster [DBG] pgmap v8653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:50 smithi115 bash[45495]: cluster 2022-04-22T21:09:48.893770+0000 mgr.y (mgr.34107) 7644 : cluster [DBG] pgmap v8653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:50 smithi115 bash[46934]: cluster 2022-04-22T21:09:48.893770+0000 mgr.y (mgr.34107) 7644 : cluster [DBG] pgmap v8653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:51 smithi121 bash[36902]: audit 2022-04-22T21:09:50.523065+0000 mon.a (mon.0) 11165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:51 smithi115 bash[45495]: audit 2022-04-22T21:09:50.523065+0000 mon.a (mon.0) 11165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:51 smithi115 bash[46934]: audit 2022-04-22T21:09:50.523065+0000 mon.a (mon.0) 11165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:52 smithi121 bash[36902]: cluster 2022-04-22T21:09:50.894858+0000 mgr.y (mgr.34107) 7645 : cluster [DBG] pgmap v8654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:52 smithi115 bash[45495]: cluster 2022-04-22T21:09:50.894858+0000 mgr.y (mgr.34107) 7645 : cluster [DBG] pgmap v8654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:52.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:52.483Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:52.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:52 smithi115 bash[46934]: cluster 2022-04-22T21:09:50.894858+0000 mgr.y (mgr.34107) 7645 : cluster [DBG] pgmap v8654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:54 smithi121 bash[36902]: cluster 2022-04-22T21:09:52.896003+0000 mgr.y (mgr.34107) 7646 : cluster [DBG] pgmap v8655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:54 smithi121 bash[36902]: audit 2022-04-22T21:09:53.514509+0000 mon.a (mon.0) 11166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:54 smithi121 bash[36902]: audit 2022-04-22T21:09:54.272523+0000 mon.a (mon.0) 11167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:09:54.361Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:09:54.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[45495]: cluster 2022-04-22T21:09:52.896003+0000 mgr.y (mgr.34107) 7646 : cluster [DBG] pgmap v8655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:54.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[45495]: audit 2022-04-22T21:09:53.514509+0000 mon.a (mon.0) 11166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:54.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[45495]: audit 2022-04-22T21:09:54.272523+0000 mon.a (mon.0) 11167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:54.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[46934]: cluster 2022-04-22T21:09:52.896003+0000 mgr.y (mgr.34107) 7646 : cluster [DBG] pgmap v8655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:54.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[46934]: audit 2022-04-22T21:09:53.514509+0000 mon.a (mon.0) 11166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:54.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:54 smithi115 bash[46934]: audit 2022-04-22T21:09:54.272523+0000 mon.a (mon.0) 11167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:56 smithi121 bash[36902]: cluster 2022-04-22T21:09:54.897065+0000 mgr.y (mgr.34107) 7647 : cluster [DBG] pgmap v8656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:56 smithi121 bash[36902]: audit 2022-04-22T21:09:55.526040+0000 mon.a (mon.0) 11168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:56 smithi115 bash[45495]: cluster 2022-04-22T21:09:54.897065+0000 mgr.y (mgr.34107) 7647 : cluster [DBG] pgmap v8656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:56 smithi115 bash[45495]: audit 2022-04-22T21:09:55.526040+0000 mon.a (mon.0) 11168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:56 smithi115 bash[46934]: cluster 2022-04-22T21:09:54.897065+0000 mgr.y (mgr.34107) 7647 : cluster [DBG] pgmap v8656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:56 smithi115 bash[46934]: audit 2022-04-22T21:09:55.526040+0000 mon.a (mon.0) 11168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:09:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:58 smithi121 bash[36902]: cluster 2022-04-22T21:09:56.898258+0000 mgr.y (mgr.34107) 7648 : cluster [DBG] pgmap v8657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:58 smithi115 bash[45495]: cluster 2022-04-22T21:09:56.898258+0000 mgr.y (mgr.34107) 7648 : cluster [DBG] pgmap v8657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:58.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:58 smithi115 bash[46934]: cluster 2022-04-22T21:09:56.898258+0000 mgr.y (mgr.34107) 7648 : cluster [DBG] pgmap v8657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:09:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:59 smithi121 bash[36902]: audit 2022-04-22T21:09:58.517439+0000 mon.a (mon.0) 11169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:09:59 smithi121 bash[36902]: audit 2022-04-22T21:09:59.276466+0000 mon.a (mon.0) 11170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:59 smithi115 bash[45495]: audit 2022-04-22T21:09:58.517439+0000 mon.a (mon.0) 11169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:59.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:09:59 smithi115 bash[45495]: audit 2022-04-22T21:09:59.276466+0000 mon.a (mon.0) 11170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:09:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:59 smithi115 bash[46934]: audit 2022-04-22T21:09:58.517439+0000 mon.a (mon.0) 11169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:09:59.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:09:59 smithi115 bash[46934]: audit 2022-04-22T21:09:59.276466+0000 mon.a (mon.0) 11170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:00 smithi121 bash[36902]: cluster 2022-04-22T21:09:58.899327+0000 mgr.y (mgr.34107) 7649 : cluster [DBG] pgmap v8658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:00 smithi121 bash[36902]: cluster 2022-04-22T21:10:00.000158+0000 mon.a (mon.0) 11171 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:10:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:00 smithi115 bash[45495]: cluster 2022-04-22T21:09:58.899327+0000 mgr.y (mgr.34107) 7649 : cluster [DBG] pgmap v8658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:00 smithi115 bash[45495]: cluster 2022-04-22T21:10:00.000158+0000 mon.a (mon.0) 11171 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:10:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:00 smithi115 bash[46934]: cluster 2022-04-22T21:09:58.899327+0000 mgr.y (mgr.34107) 7649 : cluster [DBG] pgmap v8658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:00 smithi115 bash[46934]: cluster 2022-04-22T21:10:00.000158+0000 mon.a (mon.0) 11171 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:10:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:01 smithi121 bash[36902]: audit 2022-04-22T21:10:00.528951+0000 mon.a (mon.0) 11172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:01 smithi115 bash[45495]: audit 2022-04-22T21:10:00.528951+0000 mon.a (mon.0) 11172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:01.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:01 smithi115 bash[46934]: audit 2022-04-22T21:10:00.528951+0000 mon.a (mon.0) 11172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:02 smithi121 bash[36902]: cluster 2022-04-22T21:10:00.900447+0000 mgr.y (mgr.34107) 7650 : cluster [DBG] pgmap v8659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:02 smithi115 bash[45495]: cluster 2022-04-22T21:10:00.900447+0000 mgr.y (mgr.34107) 7650 : cluster [DBG] pgmap v8659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:02.483Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:02 smithi115 bash[46934]: cluster 2022-04-22T21:10:00.900447+0000 mgr.y (mgr.34107) 7650 : cluster [DBG] pgmap v8659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:04 smithi121 bash[36902]: cluster 2022-04-22T21:10:02.901480+0000 mgr.y (mgr.34107) 7651 : cluster [DBG] pgmap v8660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:04 smithi121 bash[36902]: audit 2022-04-22T21:10:03.519902+0000 mon.a (mon.0) 11173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:04 smithi121 bash[36902]: audit 2022-04-22T21:10:04.280151+0000 mon.a (mon.0) 11174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:04.361Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[45495]: cluster 2022-04-22T21:10:02.901480+0000 mgr.y (mgr.34107) 7651 : cluster [DBG] pgmap v8660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[45495]: audit 2022-04-22T21:10:03.519902+0000 mon.a (mon.0) 11173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:04.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[45495]: audit 2022-04-22T21:10:04.280151+0000 mon.a (mon.0) 11174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[46934]: cluster 2022-04-22T21:10:02.901480+0000 mgr.y (mgr.34107) 7651 : cluster [DBG] pgmap v8660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[46934]: audit 2022-04-22T21:10:03.519902+0000 mon.a (mon.0) 11173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:04 smithi115 bash[46934]: audit 2022-04-22T21:10:04.280151+0000 mon.a (mon.0) 11174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:06 smithi121 bash[36902]: cluster 2022-04-22T21:10:04.902495+0000 mgr.y (mgr.34107) 7652 : cluster [DBG] pgmap v8661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:06 smithi121 bash[36902]: audit 2022-04-22T21:10:05.532025+0000 mon.a (mon.0) 11175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:06 smithi115 bash[45495]: cluster 2022-04-22T21:10:04.902495+0000 mgr.y (mgr.34107) 7652 : cluster [DBG] pgmap v8661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:06 smithi115 bash[45495]: audit 2022-04-22T21:10:05.532025+0000 mon.a (mon.0) 11175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:06 smithi115 bash[46934]: cluster 2022-04-22T21:10:04.902495+0000 mgr.y (mgr.34107) 7652 : cluster [DBG] pgmap v8661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:06 smithi115 bash[46934]: audit 2022-04-22T21:10:05.532025+0000 mon.a (mon.0) 11175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:08 smithi121 bash[36902]: cluster 2022-04-22T21:10:06.903602+0000 mgr.y (mgr.34107) 7653 : cluster [DBG] pgmap v8662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:08 smithi115 bash[45495]: cluster 2022-04-22T21:10:06.903602+0000 mgr.y (mgr.34107) 7653 : cluster [DBG] pgmap v8662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:08 smithi115 bash[46934]: cluster 2022-04-22T21:10:06.903602+0000 mgr.y (mgr.34107) 7653 : cluster [DBG] pgmap v8662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:09 smithi121 bash[36902]: audit 2022-04-22T21:10:08.523030+0000 mon.a (mon.0) 11176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:09 smithi121 bash[36902]: audit 2022-04-22T21:10:09.283724+0000 mon.a (mon.0) 11177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:09 smithi115 bash[45495]: audit 2022-04-22T21:10:08.523030+0000 mon.a (mon.0) 11176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:09 smithi115 bash[45495]: audit 2022-04-22T21:10:09.283724+0000 mon.a (mon.0) 11177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:09 smithi115 bash[46934]: audit 2022-04-22T21:10:08.523030+0000 mon.a (mon.0) 11176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:09 smithi115 bash[46934]: audit 2022-04-22T21:10:09.283724+0000 mon.a (mon.0) 11177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:10 smithi121 bash[36902]: cluster 2022-04-22T21:10:08.904694+0000 mgr.y (mgr.34107) 7654 : cluster [DBG] pgmap v8663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:10 smithi115 bash[45495]: cluster 2022-04-22T21:10:08.904694+0000 mgr.y (mgr.34107) 7654 : cluster [DBG] pgmap v8663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:10 smithi115 bash[46934]: cluster 2022-04-22T21:10:08.904694+0000 mgr.y (mgr.34107) 7654 : cluster [DBG] pgmap v8663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:11 smithi121 bash[36902]: audit 2022-04-22T21:10:10.534766+0000 mon.a (mon.0) 11178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:11.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:11 smithi115 bash[45495]: audit 2022-04-22T21:10:10.534766+0000 mon.a (mon.0) 11178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:11.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:11 smithi115 bash[46934]: audit 2022-04-22T21:10:10.534766+0000 mon.a (mon.0) 11178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:12 smithi115 bash[45495]: cluster 2022-04-22T21:10:10.905749+0000 mgr.y (mgr.34107) 7655 : cluster [DBG] pgmap v8664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:12.484Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:12 smithi115 bash[46934]: cluster 2022-04-22T21:10:10.905749+0000 mgr.y (mgr.34107) 7655 : cluster [DBG] pgmap v8664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:12 smithi121 bash[36902]: cluster 2022-04-22T21:10:10.905749+0000 mgr.y (mgr.34107) 7655 : cluster [DBG] pgmap v8664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:10:14.680 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:10:14.680 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:10:14.680 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:10:14.681 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:10:14.681 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:10:14.681 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:10:14.681 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:10:14.681 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:10:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:10:14.683 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:10:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:14.361Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[45495]: cluster 2022-04-22T21:10:12.906811+0000 mgr.y (mgr.34107) 7656 : cluster [DBG] pgmap v8665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[45495]: audit 2022-04-22T21:10:13.525872+0000 mon.a (mon.0) 11179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[45495]: audit 2022-04-22T21:10:14.287256+0000 mon.a (mon.0) 11180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[46934]: cluster 2022-04-22T21:10:12.906811+0000 mgr.y (mgr.34107) 7656 : cluster [DBG] pgmap v8665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[46934]: audit 2022-04-22T21:10:13.525872+0000 mon.a (mon.0) 11179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:14 smithi115 bash[46934]: audit 2022-04-22T21:10:14.287256+0000 mon.a (mon.0) 11180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:14 smithi121 bash[36902]: cluster 2022-04-22T21:10:12.906811+0000 mgr.y (mgr.34107) 7656 : cluster [DBG] pgmap v8665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:14 smithi121 bash[36902]: audit 2022-04-22T21:10:13.525872+0000 mon.a (mon.0) 11179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:14 smithi121 bash[36902]: audit 2022-04-22T21:10:14.287256+0000 mon.a (mon.0) 11180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:15.081 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:10:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:10:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:10:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:10:15.083 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:10:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[46934]: audit 2022-04-22T21:10:14.303979+0000 mgr.y (mgr.34107) 7657 : audit [DBG] from='client.53257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[46934]: audit 2022-04-22T21:10:14.650431+0000 mgr.y (mgr.34107) 7658 : audit [DBG] from='client.43332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[46934]: audit 2022-04-22T21:10:15.076039+0000 mon.a (mon.0) 11181 : audit [DBG] from='client.? 172.21.15.115:0/2167847934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[45495]: audit 2022-04-22T21:10:14.303979+0000 mgr.y (mgr.34107) 7657 : audit [DBG] from='client.53257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[45495]: audit 2022-04-22T21:10:14.650431+0000 mgr.y (mgr.34107) 7658 : audit [DBG] from='client.43332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:15 smithi115 bash[45495]: audit 2022-04-22T21:10:15.076039+0000 mon.a (mon.0) 11181 : audit [DBG] from='client.? 172.21.15.115:0/2167847934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:15 smithi121 bash[36902]: audit 2022-04-22T21:10:14.303979+0000 mgr.y (mgr.34107) 7657 : audit [DBG] from='client.53257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:15 smithi121 bash[36902]: audit 2022-04-22T21:10:14.650431+0000 mgr.y (mgr.34107) 7658 : audit [DBG] from='client.43332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:15 smithi121 bash[36902]: audit 2022-04-22T21:10:15.076039+0000 mon.a (mon.0) 11181 : audit [DBG] from='client.? 172.21.15.115:0/2167847934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[46934]: cluster 2022-04-22T21:10:14.907908+0000 mgr.y (mgr.34107) 7659 : cluster [DBG] pgmap v8666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[46934]: audit 2022-04-22T21:10:15.537757+0000 mon.a (mon.0) 11182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[46934]: audit 2022-04-22T21:10:15.850305+0000 mon.a (mon.0) 11183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:10:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[46934]: audit 2022-04-22T21:10:15.856694+0000 mon.a (mon.0) 11184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:10:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[45495]: cluster 2022-04-22T21:10:14.907908+0000 mgr.y (mgr.34107) 7659 : cluster [DBG] pgmap v8666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[45495]: audit 2022-04-22T21:10:15.537757+0000 mon.a (mon.0) 11182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[45495]: audit 2022-04-22T21:10:15.850305+0000 mon.a (mon.0) 11183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:10:16.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:16 smithi115 bash[45495]: audit 2022-04-22T21:10:15.856694+0000 mon.a (mon.0) 11184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:10:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:16 smithi121 bash[36902]: cluster 2022-04-22T21:10:14.907908+0000 mgr.y (mgr.34107) 7659 : cluster [DBG] pgmap v8666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:16 smithi121 bash[36902]: audit 2022-04-22T21:10:15.537757+0000 mon.a (mon.0) 11182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:16 smithi121 bash[36902]: audit 2022-04-22T21:10:15.850305+0000 mon.a (mon.0) 11183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:10:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:16 smithi121 bash[36902]: audit 2022-04-22T21:10:15.856694+0000 mon.a (mon.0) 11184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:10:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:18 smithi115 bash[45495]: cluster 2022-04-22T21:10:16.909030+0000 mgr.y (mgr.34107) 7660 : cluster [DBG] pgmap v8667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:18 smithi115 bash[46934]: cluster 2022-04-22T21:10:16.909030+0000 mgr.y (mgr.34107) 7660 : cluster [DBG] pgmap v8667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:19.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:18 smithi121 bash[36902]: cluster 2022-04-22T21:10:16.909030+0000 mgr.y (mgr.34107) 7660 : cluster [DBG] pgmap v8667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:19 smithi115 bash[45495]: audit 2022-04-22T21:10:18.533845+0000 mon.a (mon.0) 11185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:19 smithi115 bash[45495]: audit 2022-04-22T21:10:19.291020+0000 mon.a (mon.0) 11186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:19 smithi115 bash[46934]: audit 2022-04-22T21:10:18.533845+0000 mon.a (mon.0) 11185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:19 smithi115 bash[46934]: audit 2022-04-22T21:10:19.291020+0000 mon.a (mon.0) 11186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:19 smithi121 bash[36902]: audit 2022-04-22T21:10:18.533845+0000 mon.a (mon.0) 11185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:19 smithi121 bash[36902]: audit 2022-04-22T21:10:19.291020+0000 mon.a (mon.0) 11186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:20 smithi121 bash[36902]: cluster 2022-04-22T21:10:18.910193+0000 mgr.y (mgr.34107) 7661 : cluster [DBG] pgmap v8668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:20 smithi115 bash[45495]: cluster 2022-04-22T21:10:18.910193+0000 mgr.y (mgr.34107) 7661 : cluster [DBG] pgmap v8668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:20 smithi115 bash[46934]: cluster 2022-04-22T21:10:18.910193+0000 mgr.y (mgr.34107) 7661 : cluster [DBG] pgmap v8668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:21 smithi121 bash[36902]: audit 2022-04-22T21:10:20.545281+0000 mon.a (mon.0) 11187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:21 smithi115 bash[45495]: audit 2022-04-22T21:10:20.545281+0000 mon.a (mon.0) 11187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:22.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:21 smithi115 bash[46934]: audit 2022-04-22T21:10:20.545281+0000 mon.a (mon.0) 11187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:22 smithi115 bash[45495]: cluster 2022-04-22T21:10:20.911282+0000 mgr.y (mgr.34107) 7662 : cluster [DBG] pgmap v8669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:22.486Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:22 smithi115 bash[46934]: cluster 2022-04-22T21:10:20.911282+0000 mgr.y (mgr.34107) 7662 : cluster [DBG] pgmap v8669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:22 smithi121 bash[36902]: cluster 2022-04-22T21:10:20.911282+0000 mgr.y (mgr.34107) 7662 : cluster [DBG] pgmap v8669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:24.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:23 smithi121 bash[36902]: audit 2022-04-22T21:10:23.532058+0000 mon.a (mon.0) 11188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:23 smithi115 bash[45495]: audit 2022-04-22T21:10:23.532058+0000 mon.a (mon.0) 11188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:23 smithi115 bash[46934]: audit 2022-04-22T21:10:23.532058+0000 mon.a (mon.0) 11188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:24.363Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:24 smithi115 bash[45495]: cluster 2022-04-22T21:10:22.912286+0000 mgr.y (mgr.34107) 7663 : cluster [DBG] pgmap v8670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:24 smithi115 bash[45495]: audit 2022-04-22T21:10:24.294545+0000 mon.a (mon.0) 11189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:24 smithi115 bash[46934]: cluster 2022-04-22T21:10:22.912286+0000 mgr.y (mgr.34107) 7663 : cluster [DBG] pgmap v8670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:24 smithi115 bash[46934]: audit 2022-04-22T21:10:24.294545+0000 mon.a (mon.0) 11189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:24 smithi121 bash[36902]: cluster 2022-04-22T21:10:22.912286+0000 mgr.y (mgr.34107) 7663 : cluster [DBG] pgmap v8670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:24 smithi121 bash[36902]: audit 2022-04-22T21:10:24.294545+0000 mon.a (mon.0) 11189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:25 smithi121 bash[36902]: cluster 2022-04-22T21:10:24.913339+0000 mgr.y (mgr.34107) 7664 : cluster [DBG] pgmap v8671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:26.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:25 smithi121 bash[36902]: audit 2022-04-22T21:10:25.545271+0000 mon.a (mon.0) 11190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:25 smithi115 bash[45495]: cluster 2022-04-22T21:10:24.913339+0000 mgr.y (mgr.34107) 7664 : cluster [DBG] pgmap v8671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:26.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:25 smithi115 bash[45495]: audit 2022-04-22T21:10:25.545271+0000 mon.a (mon.0) 11190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:25 smithi115 bash[46934]: cluster 2022-04-22T21:10:24.913339+0000 mgr.y (mgr.34107) 7664 : cluster [DBG] pgmap v8671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:25 smithi115 bash[46934]: audit 2022-04-22T21:10:25.545271+0000 mon.a (mon.0) 11190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:28.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:27 smithi121 bash[36902]: cluster 2022-04-22T21:10:26.914405+0000 mgr.y (mgr.34107) 7665 : cluster [DBG] pgmap v8672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:27 smithi115 bash[45495]: cluster 2022-04-22T21:10:26.914405+0000 mgr.y (mgr.34107) 7665 : cluster [DBG] pgmap v8672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:27 smithi115 bash[46934]: cluster 2022-04-22T21:10:26.914405+0000 mgr.y (mgr.34107) 7665 : cluster [DBG] pgmap v8672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:28 smithi121 bash[36902]: audit 2022-04-22T21:10:28.535527+0000 mon.a (mon.0) 11191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:28 smithi115 bash[45495]: audit 2022-04-22T21:10:28.535527+0000 mon.a (mon.0) 11191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:28 smithi115 bash[46934]: audit 2022-04-22T21:10:28.535527+0000 mon.a (mon.0) 11191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:29 smithi121 bash[36902]: cluster 2022-04-22T21:10:28.915556+0000 mgr.y (mgr.34107) 7666 : cluster [DBG] pgmap v8673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:29 smithi121 bash[36902]: audit 2022-04-22T21:10:29.298226+0000 mon.a (mon.0) 11192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:29 smithi115 bash[45495]: cluster 2022-04-22T21:10:28.915556+0000 mgr.y (mgr.34107) 7666 : cluster [DBG] pgmap v8673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:30.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:29 smithi115 bash[45495]: audit 2022-04-22T21:10:29.298226+0000 mon.a (mon.0) 11192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:29 smithi115 bash[46934]: cluster 2022-04-22T21:10:28.915556+0000 mgr.y (mgr.34107) 7666 : cluster [DBG] pgmap v8673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:30.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:29 smithi115 bash[46934]: audit 2022-04-22T21:10:29.298226+0000 mon.a (mon.0) 11192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:30 smithi121 bash[36902]: audit 2022-04-22T21:10:30.548317+0000 mon.a (mon.0) 11193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:30 smithi115 bash[45495]: audit 2022-04-22T21:10:30.548317+0000 mon.a (mon.0) 11193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:30 smithi115 bash[46934]: audit 2022-04-22T21:10:30.548317+0000 mon.a (mon.0) 11193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:31 smithi121 bash[36902]: cluster 2022-04-22T21:10:30.916571+0000 mgr.y (mgr.34107) 7667 : cluster [DBG] pgmap v8674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:31 smithi115 bash[45495]: cluster 2022-04-22T21:10:30.916571+0000 mgr.y (mgr.34107) 7667 : cluster [DBG] pgmap v8674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:31 smithi115 bash[46934]: cluster 2022-04-22T21:10:30.916571+0000 mgr.y (mgr.34107) 7667 : cluster [DBG] pgmap v8674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:32.486Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:34.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:33 smithi121 bash[36902]: cluster 2022-04-22T21:10:32.917697+0000 mgr.y (mgr.34107) 7668 : cluster [DBG] pgmap v8675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:33 smithi121 bash[36902]: audit 2022-04-22T21:10:33.538624+0000 mon.a (mon.0) 11194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:33 smithi115 bash[46934]: cluster 2022-04-22T21:10:32.917697+0000 mgr.y (mgr.34107) 7668 : cluster [DBG] pgmap v8675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:33 smithi115 bash[46934]: audit 2022-04-22T21:10:33.538624+0000 mon.a (mon.0) 11194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:33 smithi115 bash[45495]: cluster 2022-04-22T21:10:32.917697+0000 mgr.y (mgr.34107) 7668 : cluster [DBG] pgmap v8675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:33 smithi115 bash[45495]: audit 2022-04-22T21:10:33.538624+0000 mon.a (mon.0) 11194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:34.364Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:34 smithi121 bash[36902]: audit 2022-04-22T21:10:34.301943+0000 mon.a (mon.0) 11195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:34 smithi115 bash[45495]: audit 2022-04-22T21:10:34.301943+0000 mon.a (mon.0) 11195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:34 smithi115 bash[46934]: audit 2022-04-22T21:10:34.301943+0000 mon.a (mon.0) 11195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:35 smithi121 bash[36902]: cluster 2022-04-22T21:10:34.918723+0000 mgr.y (mgr.34107) 7669 : cluster [DBG] pgmap v8676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:35 smithi121 bash[36902]: audit 2022-04-22T21:10:35.551376+0000 mon.a (mon.0) 11196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:35 smithi115 bash[45495]: cluster 2022-04-22T21:10:34.918723+0000 mgr.y (mgr.34107) 7669 : cluster [DBG] pgmap v8676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:36.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:35 smithi115 bash[45495]: audit 2022-04-22T21:10:35.551376+0000 mon.a (mon.0) 11196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:35 smithi115 bash[46934]: cluster 2022-04-22T21:10:34.918723+0000 mgr.y (mgr.34107) 7669 : cluster [DBG] pgmap v8676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:35 smithi115 bash[46934]: audit 2022-04-22T21:10:35.551376+0000 mon.a (mon.0) 11196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:37 smithi121 bash[36902]: cluster 2022-04-22T21:10:36.919979+0000 mgr.y (mgr.34107) 7670 : cluster [DBG] pgmap v8677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:37 smithi115 bash[45495]: cluster 2022-04-22T21:10:36.919979+0000 mgr.y (mgr.34107) 7670 : cluster [DBG] pgmap v8677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:37 smithi115 bash[46934]: cluster 2022-04-22T21:10:36.919979+0000 mgr.y (mgr.34107) 7670 : cluster [DBG] pgmap v8677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:39 smithi121 bash[36902]: audit 2022-04-22T21:10:38.542285+0000 mon.a (mon.0) 11197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:38 smithi115 bash[45495]: audit 2022-04-22T21:10:38.542285+0000 mon.a (mon.0) 11197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:39 smithi115 bash[46934]: audit 2022-04-22T21:10:38.542285+0000 mon.a (mon.0) 11197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:40 smithi121 bash[36902]: cluster 2022-04-22T21:10:38.921095+0000 mgr.y (mgr.34107) 7671 : cluster [DBG] pgmap v8678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:40 smithi121 bash[36902]: audit 2022-04-22T21:10:39.305611+0000 mon.a (mon.0) 11198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:40 smithi115 bash[45495]: cluster 2022-04-22T21:10:38.921095+0000 mgr.y (mgr.34107) 7671 : cluster [DBG] pgmap v8678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:40 smithi115 bash[45495]: audit 2022-04-22T21:10:39.305611+0000 mon.a (mon.0) 11198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:40 smithi115 bash[46934]: cluster 2022-04-22T21:10:38.921095+0000 mgr.y (mgr.34107) 7671 : cluster [DBG] pgmap v8678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:40 smithi115 bash[46934]: audit 2022-04-22T21:10:39.305611+0000 mon.a (mon.0) 11198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:41 smithi115 bash[45495]: audit 2022-04-22T21:10:40.554532+0000 mon.a (mon.0) 11199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:41 smithi115 bash[46934]: audit 2022-04-22T21:10:40.554532+0000 mon.a (mon.0) 11199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:41 smithi121 bash[36902]: audit 2022-04-22T21:10:40.554532+0000 mon.a (mon.0) 11199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:42 smithi115 bash[45495]: cluster 2022-04-22T21:10:40.922205+0000 mgr.y (mgr.34107) 7672 : cluster [DBG] pgmap v8679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:42 smithi115 bash[46934]: cluster 2022-04-22T21:10:40.922205+0000 mgr.y (mgr.34107) 7672 : cluster [DBG] pgmap v8679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:42 smithi121 bash[36902]: cluster 2022-04-22T21:10:40.922205+0000 mgr.y (mgr.34107) 7672 : cluster [DBG] pgmap v8679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:42.488Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:44 smithi115 bash[45495]: cluster 2022-04-22T21:10:42.923300+0000 mgr.y (mgr.34107) 7673 : cluster [DBG] pgmap v8680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:44 smithi115 bash[45495]: audit 2022-04-22T21:10:43.545433+0000 mon.a (mon.0) 11200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:44 smithi115 bash[46934]: cluster 2022-04-22T21:10:42.923300+0000 mgr.y (mgr.34107) 7673 : cluster [DBG] pgmap v8680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:44 smithi115 bash[46934]: audit 2022-04-22T21:10:43.545433+0000 mon.a (mon.0) 11200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:44.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:44 smithi121 bash[36902]: cluster 2022-04-22T21:10:42.923300+0000 mgr.y (mgr.34107) 7673 : cluster [DBG] pgmap v8680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:44.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:44 smithi121 bash[36902]: audit 2022-04-22T21:10:43.545433+0000 mon.a (mon.0) 11200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:44.729 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:44.364Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[45495]: audit 2022-04-22T21:10:44.309527+0000 mon.a (mon.0) 11201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[45495]: audit 2022-04-22T21:10:44.381523+0000 mon.a (mon.0) 11202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[45495]: audit 2022-04-22T21:10:44.383975+0000 mon.a (mon.0) 11203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[45495]: audit 2022-04-22T21:10:44.385735+0000 mon.a (mon.0) 11204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[45495]: audit 2022-04-22T21:10:44.739599+0000 mon.a (mon.0) 11205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[46934]: audit 2022-04-22T21:10:44.309527+0000 mon.a (mon.0) 11201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[46934]: audit 2022-04-22T21:10:44.381523+0000 mon.a (mon.0) 11202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:10:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[46934]: audit 2022-04-22T21:10:44.383975+0000 mon.a (mon.0) 11203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:10:45.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[46934]: audit 2022-04-22T21:10:44.385735+0000 mon.a (mon.0) 11204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:10:45.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:45 smithi115 bash[46934]: audit 2022-04-22T21:10:44.739599+0000 mon.a (mon.0) 11205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:10:45.457 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:45 smithi121 bash[36902]: audit 2022-04-22T21:10:44.309527+0000 mon.a (mon.0) 11201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:45 smithi121 bash[36902]: audit 2022-04-22T21:10:44.381523+0000 mon.a (mon.0) 11202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:45 smithi121 bash[36902]: audit 2022-04-22T21:10:44.383975+0000 mon.a (mon.0) 11203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:45 smithi121 bash[36902]: audit 2022-04-22T21:10:44.385735+0000 mon.a (mon.0) 11204 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:10:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:45 smithi121 bash[36902]: audit 2022-04-22T21:10:44.739599+0000 mon.a (mon.0) 11205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:10:45.793 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:10:45.796 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:10:45.797 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:10:45.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:10:45.799 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:10:46.204 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:10:46.205 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:10:46.206 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:10:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[45495]: cluster 2022-04-22T21:10:44.924370+0000 mgr.y (mgr.34107) 7674 : cluster [DBG] pgmap v8681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[45495]: audit 2022-04-22T21:10:45.434976+0000 mgr.y (mgr.34107) 7675 : audit [DBG] from='client.53275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[45495]: audit 2022-04-22T21:10:45.557681+0000 mon.a (mon.0) 11206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[46934]: cluster 2022-04-22T21:10:44.924370+0000 mgr.y (mgr.34107) 7674 : cluster [DBG] pgmap v8681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[46934]: audit 2022-04-22T21:10:45.434976+0000 mgr.y (mgr.34107) 7675 : audit [DBG] from='client.53275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:46 smithi115 bash[46934]: audit 2022-04-22T21:10:45.557681+0000 mon.a (mon.0) 11206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:46.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:46 smithi121 bash[36902]: cluster 2022-04-22T21:10:44.924370+0000 mgr.y (mgr.34107) 7674 : cluster [DBG] pgmap v8681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:46 smithi121 bash[36902]: audit 2022-04-22T21:10:45.434976+0000 mgr.y (mgr.34107) 7675 : audit [DBG] from='client.53275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:46.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:46 smithi121 bash[36902]: audit 2022-04-22T21:10:45.557681+0000 mon.a (mon.0) 11206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:47 smithi115 bash[45495]: audit 2022-04-22T21:10:45.769378+0000 mgr.y (mgr.34107) 7676 : audit [DBG] from='client.53281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:47 smithi115 bash[45495]: audit 2022-04-22T21:10:46.199518+0000 mon.c (mon.1) 279 : audit [DBG] from='client.? 172.21.15.115:0/1794985018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:47 smithi115 bash[46934]: audit 2022-04-22T21:10:45.769378+0000 mgr.y (mgr.34107) 7676 : audit [DBG] from='client.53281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:47 smithi115 bash[46934]: audit 2022-04-22T21:10:46.199518+0000 mon.c (mon.1) 279 : audit [DBG] from='client.? 172.21.15.115:0/1794985018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:47.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:47 smithi121 bash[36902]: audit 2022-04-22T21:10:45.769378+0000 mgr.y (mgr.34107) 7676 : audit [DBG] from='client.53281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:10:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:47 smithi121 bash[36902]: audit 2022-04-22T21:10:46.199518+0000 mon.c (mon.1) 279 : audit [DBG] from='client.? 172.21.15.115:0/1794985018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:10:48.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:48 smithi115 bash[45495]: cluster 2022-04-22T21:10:46.925488+0000 mgr.y (mgr.34107) 7677 : cluster [DBG] pgmap v8682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:48.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:48 smithi115 bash[46934]: cluster 2022-04-22T21:10:46.925488+0000 mgr.y (mgr.34107) 7677 : cluster [DBG] pgmap v8682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:48 smithi121 bash[36902]: cluster 2022-04-22T21:10:46.925488+0000 mgr.y (mgr.34107) 7677 : cluster [DBG] pgmap v8682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:49 smithi121 bash[36902]: audit 2022-04-22T21:10:48.548590+0000 mon.a (mon.0) 11207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:49 smithi115 bash[45495]: audit 2022-04-22T21:10:48.548590+0000 mon.a (mon.0) 11207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:49 smithi115 bash[46934]: audit 2022-04-22T21:10:48.548590+0000 mon.a (mon.0) 11207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:50 smithi121 bash[36902]: cluster 2022-04-22T21:10:48.926604+0000 mgr.y (mgr.34107) 7678 : cluster [DBG] pgmap v8683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:50 smithi121 bash[36902]: audit 2022-04-22T21:10:49.313197+0000 mon.a (mon.0) 11208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:50 smithi115 bash[45495]: cluster 2022-04-22T21:10:48.926604+0000 mgr.y (mgr.34107) 7678 : cluster [DBG] pgmap v8683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:50.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:50 smithi115 bash[45495]: audit 2022-04-22T21:10:49.313197+0000 mon.a (mon.0) 11208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:50 smithi115 bash[46934]: cluster 2022-04-22T21:10:48.926604+0000 mgr.y (mgr.34107) 7678 : cluster [DBG] pgmap v8683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:50 smithi115 bash[46934]: audit 2022-04-22T21:10:49.313197+0000 mon.a (mon.0) 11208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:51 smithi121 bash[36902]: audit 2022-04-22T21:10:50.560539+0000 mon.a (mon.0) 11209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:51 smithi115 bash[45495]: audit 2022-04-22T21:10:50.560539+0000 mon.a (mon.0) 11209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:51.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:51 smithi115 bash[46934]: audit 2022-04-22T21:10:50.560539+0000 mon.a (mon.0) 11209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:52.492 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:52 smithi115 bash[45495]: cluster 2022-04-22T21:10:50.927765+0000 mgr.y (mgr.34107) 7679 : cluster [DBG] pgmap v8684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:52.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:52 smithi115 bash[46934]: cluster 2022-04-22T21:10:50.927765+0000 mgr.y (mgr.34107) 7679 : cluster [DBG] pgmap v8684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:52 smithi121 bash[36902]: cluster 2022-04-22T21:10:50.927765+0000 mgr.y (mgr.34107) 7679 : cluster [DBG] pgmap v8684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:52.488Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:54.369 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:54 smithi115 bash[45495]: cluster 2022-04-22T21:10:52.928709+0000 mgr.y (mgr.34107) 7680 : cluster [DBG] pgmap v8685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:54.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:54 smithi115 bash[45495]: audit 2022-04-22T21:10:53.551740+0000 mon.a (mon.0) 11210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:54.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:54 smithi115 bash[46934]: cluster 2022-04-22T21:10:52.928709+0000 mgr.y (mgr.34107) 7680 : cluster [DBG] pgmap v8685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:54.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:54 smithi115 bash[46934]: audit 2022-04-22T21:10:53.551740+0000 mon.a (mon.0) 11210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:54 smithi121 bash[36902]: cluster 2022-04-22T21:10:52.928709+0000 mgr.y (mgr.34107) 7680 : cluster [DBG] pgmap v8685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:54 smithi121 bash[36902]: audit 2022-04-22T21:10:53.551740+0000 mon.a (mon.0) 11210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:10:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:10:54.365Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:10:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:55 smithi121 bash[36902]: audit 2022-04-22T21:10:54.316979+0000 mon.a (mon.0) 11211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:55 smithi115 bash[45495]: audit 2022-04-22T21:10:54.316979+0000 mon.a (mon.0) 11211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:55.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:55 smithi115 bash[46934]: audit 2022-04-22T21:10:54.316979+0000 mon.a (mon.0) 11211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:10:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:56 smithi121 bash[36902]: cluster 2022-04-22T21:10:54.929770+0000 mgr.y (mgr.34107) 7681 : cluster [DBG] pgmap v8686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:56.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:56 smithi121 bash[36902]: audit 2022-04-22T21:10:55.563656+0000 mon.a (mon.0) 11212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:56.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:56 smithi115 bash[45495]: cluster 2022-04-22T21:10:54.929770+0000 mgr.y (mgr.34107) 7681 : cluster [DBG] pgmap v8686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:56.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:56 smithi115 bash[45495]: audit 2022-04-22T21:10:55.563656+0000 mon.a (mon.0) 11212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:56.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:56 smithi115 bash[46934]: cluster 2022-04-22T21:10:54.929770+0000 mgr.y (mgr.34107) 7681 : cluster [DBG] pgmap v8686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:56.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:56 smithi115 bash[46934]: audit 2022-04-22T21:10:55.563656+0000 mon.a (mon.0) 11212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:10:58.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:58 smithi121 bash[36902]: cluster 2022-04-22T21:10:56.931026+0000 mgr.y (mgr.34107) 7682 : cluster [DBG] pgmap v8687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:58 smithi115 bash[45495]: cluster 2022-04-22T21:10:56.931026+0000 mgr.y (mgr.34107) 7682 : cluster [DBG] pgmap v8687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:58 smithi115 bash[46934]: cluster 2022-04-22T21:10:56.931026+0000 mgr.y (mgr.34107) 7682 : cluster [DBG] pgmap v8687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:10:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:10:59 smithi121 bash[36902]: audit 2022-04-22T21:10:58.554790+0000 mon.a (mon.0) 11213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:10:59 smithi115 bash[45495]: audit 2022-04-22T21:10:58.554790+0000 mon.a (mon.0) 11213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:10:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:10:59 smithi115 bash[46934]: audit 2022-04-22T21:10:58.554790+0000 mon.a (mon.0) 11213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:00.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:00 smithi121 bash[36902]: cluster 2022-04-22T21:10:58.932208+0000 mgr.y (mgr.34107) 7683 : cluster [DBG] pgmap v8688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:00 smithi121 bash[36902]: audit 2022-04-22T21:10:59.320606+0000 mon.a (mon.0) 11214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:00 smithi115 bash[45495]: cluster 2022-04-22T21:10:58.932208+0000 mgr.y (mgr.34107) 7683 : cluster [DBG] pgmap v8688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:00 smithi115 bash[45495]: audit 2022-04-22T21:10:59.320606+0000 mon.a (mon.0) 11214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:00 smithi115 bash[46934]: cluster 2022-04-22T21:10:58.932208+0000 mgr.y (mgr.34107) 7683 : cluster [DBG] pgmap v8688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:00 smithi115 bash[46934]: audit 2022-04-22T21:10:59.320606+0000 mon.a (mon.0) 11214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:01.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:01 smithi121 bash[36902]: audit 2022-04-22T21:11:00.566753+0000 mon.a (mon.0) 11215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:01 smithi115 bash[45495]: audit 2022-04-22T21:11:00.566753+0000 mon.a (mon.0) 11215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:01 smithi115 bash[46934]: audit 2022-04-22T21:11:00.566753+0000 mon.a (mon.0) 11215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:02.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:02 smithi115 bash[46934]: cluster 2022-04-22T21:11:00.933305+0000 mgr.y (mgr.34107) 7684 : cluster [DBG] pgmap v8689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:02.493 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:02 smithi115 bash[45495]: cluster 2022-04-22T21:11:00.933305+0000 mgr.y (mgr.34107) 7684 : cluster [DBG] pgmap v8689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:02 smithi121 bash[36902]: cluster 2022-04-22T21:11:00.933305+0000 mgr.y (mgr.34107) 7684 : cluster [DBG] pgmap v8689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:02.488Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:04.369 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:04 smithi115 bash[45495]: cluster 2022-04-22T21:11:02.934400+0000 mgr.y (mgr.34107) 7685 : cluster [DBG] pgmap v8690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:04.369 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:04 smithi115 bash[45495]: audit 2022-04-22T21:11:03.557823+0000 mon.a (mon.0) 11216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:04 smithi115 bash[46934]: cluster 2022-04-22T21:11:02.934400+0000 mgr.y (mgr.34107) 7685 : cluster [DBG] pgmap v8690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:04 smithi115 bash[46934]: audit 2022-04-22T21:11:03.557823+0000 mon.a (mon.0) 11216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:04 smithi121 bash[36902]: cluster 2022-04-22T21:11:02.934400+0000 mgr.y (mgr.34107) 7685 : cluster [DBG] pgmap v8690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:04 smithi121 bash[36902]: audit 2022-04-22T21:11:03.557823+0000 mon.a (mon.0) 11216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:04.365Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:05.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:05 smithi121 bash[36902]: audit 2022-04-22T21:11:04.324395+0000 mon.a (mon.0) 11217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:05 smithi115 bash[45495]: audit 2022-04-22T21:11:04.324395+0000 mon.a (mon.0) 11217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:05 smithi115 bash[46934]: audit 2022-04-22T21:11:04.324395+0000 mon.a (mon.0) 11217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:06 smithi121 bash[36902]: cluster 2022-04-22T21:11:04.935538+0000 mgr.y (mgr.34107) 7686 : cluster [DBG] pgmap v8691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:06 smithi121 bash[36902]: audit 2022-04-22T21:11:05.569780+0000 mon.a (mon.0) 11218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:06 smithi115 bash[45495]: cluster 2022-04-22T21:11:04.935538+0000 mgr.y (mgr.34107) 7686 : cluster [DBG] pgmap v8691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:06 smithi115 bash[45495]: audit 2022-04-22T21:11:05.569780+0000 mon.a (mon.0) 11218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:06 smithi115 bash[46934]: cluster 2022-04-22T21:11:04.935538+0000 mgr.y (mgr.34107) 7686 : cluster [DBG] pgmap v8691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:06 smithi115 bash[46934]: audit 2022-04-22T21:11:05.569780+0000 mon.a (mon.0) 11218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:08 smithi121 bash[36902]: cluster 2022-04-22T21:11:06.936623+0000 mgr.y (mgr.34107) 7687 : cluster [DBG] pgmap v8692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:08 smithi115 bash[46934]: cluster 2022-04-22T21:11:06.936623+0000 mgr.y (mgr.34107) 7687 : cluster [DBG] pgmap v8692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:08 smithi115 bash[45495]: cluster 2022-04-22T21:11:06.936623+0000 mgr.y (mgr.34107) 7687 : cluster [DBG] pgmap v8692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:09.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:09 smithi121 bash[36902]: audit 2022-04-22T21:11:08.560851+0000 mon.a (mon.0) 11219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:09 smithi115 bash[46934]: audit 2022-04-22T21:11:08.560851+0000 mon.a (mon.0) 11219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:09 smithi115 bash[45495]: audit 2022-04-22T21:11:08.560851+0000 mon.a (mon.0) 11219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:10.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:10 smithi121 bash[36902]: cluster 2022-04-22T21:11:08.937716+0000 mgr.y (mgr.34107) 7688 : cluster [DBG] pgmap v8693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:10 smithi121 bash[36902]: audit 2022-04-22T21:11:09.328034+0000 mon.a (mon.0) 11220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:10 smithi115 bash[46934]: cluster 2022-04-22T21:11:08.937716+0000 mgr.y (mgr.34107) 7688 : cluster [DBG] pgmap v8693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:10 smithi115 bash[46934]: audit 2022-04-22T21:11:09.328034+0000 mon.a (mon.0) 11220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:10 smithi115 bash[45495]: cluster 2022-04-22T21:11:08.937716+0000 mgr.y (mgr.34107) 7688 : cluster [DBG] pgmap v8693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:10 smithi115 bash[45495]: audit 2022-04-22T21:11:09.328034+0000 mon.a (mon.0) 11220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:11 smithi121 bash[36902]: audit 2022-04-22T21:11:10.573230+0000 mon.a (mon.0) 11221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:11 smithi115 bash[45495]: audit 2022-04-22T21:11:10.573230+0000 mon.a (mon.0) 11221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:11 smithi115 bash[46934]: audit 2022-04-22T21:11:10.573230+0000 mon.a (mon.0) 11221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:12.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:12 smithi115 bash[45495]: cluster 2022-04-22T21:11:10.938798+0000 mgr.y (mgr.34107) 7689 : cluster [DBG] pgmap v8694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:12.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:12 smithi115 bash[46934]: cluster 2022-04-22T21:11:10.938798+0000 mgr.y (mgr.34107) 7689 : cluster [DBG] pgmap v8694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:12.515 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:12.489Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:12 smithi121 bash[36902]: cluster 2022-04-22T21:11:10.938798+0000 mgr.y (mgr.34107) 7689 : cluster [DBG] pgmap v8694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:14.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:14 smithi121 bash[36902]: cluster 2022-04-22T21:11:12.939902+0000 mgr.y (mgr.34107) 7690 : cluster [DBG] pgmap v8695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:14 smithi121 bash[36902]: audit 2022-04-22T21:11:13.564062+0000 mon.a (mon.0) 11222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:14.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:14.365Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:14 smithi115 bash[46934]: cluster 2022-04-22T21:11:12.939902+0000 mgr.y (mgr.34107) 7690 : cluster [DBG] pgmap v8695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:14 smithi115 bash[46934]: audit 2022-04-22T21:11:13.564062+0000 mon.a (mon.0) 11222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:14 smithi115 bash[45495]: cluster 2022-04-22T21:11:12.939902+0000 mgr.y (mgr.34107) 7690 : cluster [DBG] pgmap v8695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:14 smithi115 bash[45495]: audit 2022-04-22T21:11:13.564062+0000 mon.a (mon.0) 11222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:15 smithi121 bash[36902]: audit 2022-04-22T21:11:14.331724+0000 mon.a (mon.0) 11223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:15 smithi115 bash[45495]: audit 2022-04-22T21:11:14.331724+0000 mon.a (mon.0) 11223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:15 smithi115 bash[46934]: audit 2022-04-22T21:11:14.331724+0000 mon.a (mon.0) 11223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:16 smithi121 bash[36902]: cluster 2022-04-22T21:11:14.941018+0000 mgr.y (mgr.34107) 7691 : cluster [DBG] pgmap v8696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:16 smithi121 bash[36902]: audit 2022-04-22T21:11:15.576481+0000 mon.a (mon.0) 11224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:16 smithi121 bash[36902]: audit 2022-04-22T21:11:15.862481+0000 mon.a (mon.0) 11225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:11:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:16 smithi121 bash[36902]: audit 2022-04-22T21:11:15.868109+0000 mon.a (mon.0) 11226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:11:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[46934]: cluster 2022-04-22T21:11:14.941018+0000 mgr.y (mgr.34107) 7691 : cluster [DBG] pgmap v8696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[46934]: audit 2022-04-22T21:11:15.576481+0000 mon.a (mon.0) 11224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[46934]: audit 2022-04-22T21:11:15.862481+0000 mon.a (mon.0) 11225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[46934]: audit 2022-04-22T21:11:15.868109+0000 mon.a (mon.0) 11226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[45495]: cluster 2022-04-22T21:11:14.941018+0000 mgr.y (mgr.34107) 7691 : cluster [DBG] pgmap v8696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[45495]: audit 2022-04-22T21:11:15.576481+0000 mon.a (mon.0) 11224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[45495]: audit 2022-04-22T21:11:15.862481+0000 mon.a (mon.0) 11225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:11:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:16 smithi115 bash[45495]: audit 2022-04-22T21:11:15.868109+0000 mon.a (mon.0) 11226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:11:16.576 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:11:16.914 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:11:16.915 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:11:16.916 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:11:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:11:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:11:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:11:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:11:16.917 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:11:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:11:17.320 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:11:17.320 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:11:17.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:11:17.321 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:11:17.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:17.322 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:11:17.322 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:11:17.322 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:11:17.322 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:11:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:17 smithi121 bash[36902]: audit 2022-04-22T21:11:16.555877+0000 mgr.y (mgr.34107) 7692 : audit [DBG] from='client.43362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:17 smithi115 bash[45495]: audit 2022-04-22T21:11:16.555877+0000 mgr.y (mgr.34107) 7692 : audit [DBG] from='client.43362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:17 smithi115 bash[46934]: audit 2022-04-22T21:11:16.555877+0000 mgr.y (mgr.34107) 7692 : audit [DBG] from='client.43362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:18 smithi121 bash[36902]: audit 2022-04-22T21:11:16.891520+0000 mgr.y (mgr.34107) 7693 : audit [DBG] from='client.53299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:18 smithi121 bash[36902]: cluster 2022-04-22T21:11:16.942164+0000 mgr.y (mgr.34107) 7694 : cluster [DBG] pgmap v8697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:18 smithi121 bash[36902]: audit 2022-04-22T21:11:17.315863+0000 mon.a (mon.0) 11227 : audit [DBG] from='client.? 172.21.15.115:0/1475386189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[45495]: audit 2022-04-22T21:11:16.891520+0000 mgr.y (mgr.34107) 7693 : audit [DBG] from='client.53299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[45495]: cluster 2022-04-22T21:11:16.942164+0000 mgr.y (mgr.34107) 7694 : cluster [DBG] pgmap v8697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:18.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[45495]: audit 2022-04-22T21:11:17.315863+0000 mon.a (mon.0) 11227 : audit [DBG] from='client.? 172.21.15.115:0/1475386189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[46934]: audit 2022-04-22T21:11:16.891520+0000 mgr.y (mgr.34107) 7693 : audit [DBG] from='client.53299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[46934]: cluster 2022-04-22T21:11:16.942164+0000 mgr.y (mgr.34107) 7694 : cluster [DBG] pgmap v8697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:18 smithi115 bash[46934]: audit 2022-04-22T21:11:17.315863+0000 mon.a (mon.0) 11227 : audit [DBG] from='client.? 172.21.15.115:0/1475386189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:19 smithi121 bash[36902]: audit 2022-04-22T21:11:18.567238+0000 mon.a (mon.0) 11228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:19 smithi115 bash[45495]: audit 2022-04-22T21:11:18.567238+0000 mon.a (mon.0) 11228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:19 smithi115 bash[46934]: audit 2022-04-22T21:11:18.567238+0000 mon.a (mon.0) 11228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:20 smithi121 bash[36902]: cluster 2022-04-22T21:11:18.943319+0000 mgr.y (mgr.34107) 7695 : cluster [DBG] pgmap v8698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:20 smithi121 bash[36902]: audit 2022-04-22T21:11:19.335561+0000 mon.a (mon.0) 11229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:20 smithi115 bash[45495]: cluster 2022-04-22T21:11:18.943319+0000 mgr.y (mgr.34107) 7695 : cluster [DBG] pgmap v8698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:20 smithi115 bash[45495]: audit 2022-04-22T21:11:19.335561+0000 mon.a (mon.0) 11229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:20 smithi115 bash[46934]: cluster 2022-04-22T21:11:18.943319+0000 mgr.y (mgr.34107) 7695 : cluster [DBG] pgmap v8698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:20 smithi115 bash[46934]: audit 2022-04-22T21:11:19.335561+0000 mon.a (mon.0) 11229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:21 smithi121 bash[36902]: audit 2022-04-22T21:11:20.579602+0000 mon.a (mon.0) 11230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:21 smithi115 bash[45495]: audit 2022-04-22T21:11:20.579602+0000 mon.a (mon.0) 11230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:21 smithi115 bash[46934]: audit 2022-04-22T21:11:20.579602+0000 mon.a (mon.0) 11230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:22.495 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:22 smithi115 bash[45495]: cluster 2022-04-22T21:11:20.944480+0000 mgr.y (mgr.34107) 7696 : cluster [DBG] pgmap v8699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:22.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:22 smithi115 bash[46934]: cluster 2022-04-22T21:11:20.944480+0000 mgr.y (mgr.34107) 7696 : cluster [DBG] pgmap v8699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:22 smithi121 bash[36902]: cluster 2022-04-22T21:11:20.944480+0000 mgr.y (mgr.34107) 7696 : cluster [DBG] pgmap v8699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:22.490Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:24 smithi121 bash[36902]: cluster 2022-04-22T21:11:22.945655+0000 mgr.y (mgr.34107) 7697 : cluster [DBG] pgmap v8700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:24 smithi121 bash[36902]: audit 2022-04-22T21:11:23.570375+0000 mon.a (mon.0) 11231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:24.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:24.367Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:24 smithi115 bash[46934]: cluster 2022-04-22T21:11:22.945655+0000 mgr.y (mgr.34107) 7697 : cluster [DBG] pgmap v8700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:24 smithi115 bash[46934]: audit 2022-04-22T21:11:23.570375+0000 mon.a (mon.0) 11231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:24 smithi115 bash[45495]: cluster 2022-04-22T21:11:22.945655+0000 mgr.y (mgr.34107) 7697 : cluster [DBG] pgmap v8700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:24 smithi115 bash[45495]: audit 2022-04-22T21:11:23.570375+0000 mon.a (mon.0) 11231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:25 smithi121 bash[36902]: audit 2022-04-22T21:11:24.339004+0000 mon.a (mon.0) 11232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:25 smithi115 bash[46934]: audit 2022-04-22T21:11:24.339004+0000 mon.a (mon.0) 11232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:25.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:25 smithi115 bash[45495]: audit 2022-04-22T21:11:24.339004+0000 mon.a (mon.0) 11232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:26.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:26 smithi121 bash[36902]: cluster 2022-04-22T21:11:24.946708+0000 mgr.y (mgr.34107) 7698 : cluster [DBG] pgmap v8701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:26 smithi121 bash[36902]: audit 2022-04-22T21:11:25.582496+0000 mon.a (mon.0) 11233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:26 smithi115 bash[45495]: cluster 2022-04-22T21:11:24.946708+0000 mgr.y (mgr.34107) 7698 : cluster [DBG] pgmap v8701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:26 smithi115 bash[45495]: audit 2022-04-22T21:11:25.582496+0000 mon.a (mon.0) 11233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:26 smithi115 bash[46934]: cluster 2022-04-22T21:11:24.946708+0000 mgr.y (mgr.34107) 7698 : cluster [DBG] pgmap v8701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:26 smithi115 bash[46934]: audit 2022-04-22T21:11:25.582496+0000 mon.a (mon.0) 11233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:28 smithi121 bash[36902]: cluster 2022-04-22T21:11:26.947798+0000 mgr.y (mgr.34107) 7699 : cluster [DBG] pgmap v8702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:28 smithi115 bash[45495]: cluster 2022-04-22T21:11:26.947798+0000 mgr.y (mgr.34107) 7699 : cluster [DBG] pgmap v8702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:28 smithi115 bash[46934]: cluster 2022-04-22T21:11:26.947798+0000 mgr.y (mgr.34107) 7699 : cluster [DBG] pgmap v8702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:29.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:29 smithi121 bash[36902]: audit 2022-04-22T21:11:28.573299+0000 mon.a (mon.0) 11234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:29 smithi115 bash[45495]: audit 2022-04-22T21:11:28.573299+0000 mon.a (mon.0) 11234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:29 smithi115 bash[46934]: audit 2022-04-22T21:11:28.573299+0000 mon.a (mon.0) 11234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:30 smithi121 bash[36902]: cluster 2022-04-22T21:11:28.948868+0000 mgr.y (mgr.34107) 7700 : cluster [DBG] pgmap v8703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:30 smithi121 bash[36902]: audit 2022-04-22T21:11:29.342597+0000 mon.a (mon.0) 11235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:30 smithi115 bash[45495]: cluster 2022-04-22T21:11:28.948868+0000 mgr.y (mgr.34107) 7700 : cluster [DBG] pgmap v8703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:30 smithi115 bash[45495]: audit 2022-04-22T21:11:29.342597+0000 mon.a (mon.0) 11235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:30 smithi115 bash[46934]: cluster 2022-04-22T21:11:28.948868+0000 mgr.y (mgr.34107) 7700 : cluster [DBG] pgmap v8703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:30.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:30 smithi115 bash[46934]: audit 2022-04-22T21:11:29.342597+0000 mon.a (mon.0) 11235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:31 smithi121 bash[36902]: audit 2022-04-22T21:11:30.585156+0000 mon.a (mon.0) 11236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:31 smithi115 bash[45495]: audit 2022-04-22T21:11:30.585156+0000 mon.a (mon.0) 11236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:31 smithi115 bash[46934]: audit 2022-04-22T21:11:30.585156+0000 mon.a (mon.0) 11236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:32.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:32 smithi115 bash[45495]: cluster 2022-04-22T21:11:30.949895+0000 mgr.y (mgr.34107) 7701 : cluster [DBG] pgmap v8704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:32.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:32 smithi115 bash[46934]: cluster 2022-04-22T21:11:30.949895+0000 mgr.y (mgr.34107) 7701 : cluster [DBG] pgmap v8704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:32 smithi121 bash[36902]: cluster 2022-04-22T21:11:30.949895+0000 mgr.y (mgr.34107) 7701 : cluster [DBG] pgmap v8704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:32.490Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:34 smithi121 bash[36902]: cluster 2022-04-22T21:11:32.950939+0000 mgr.y (mgr.34107) 7702 : cluster [DBG] pgmap v8705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:34 smithi121 bash[36902]: audit 2022-04-22T21:11:33.576382+0000 mon.a (mon.0) 11237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:34.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:34.367Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:34 smithi115 bash[46934]: cluster 2022-04-22T21:11:32.950939+0000 mgr.y (mgr.34107) 7702 : cluster [DBG] pgmap v8705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:34 smithi115 bash[46934]: audit 2022-04-22T21:11:33.576382+0000 mon.a (mon.0) 11237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:34 smithi115 bash[45495]: cluster 2022-04-22T21:11:32.950939+0000 mgr.y (mgr.34107) 7702 : cluster [DBG] pgmap v8705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:34 smithi115 bash[45495]: audit 2022-04-22T21:11:33.576382+0000 mon.a (mon.0) 11237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:35 smithi121 bash[36902]: audit 2022-04-22T21:11:34.346356+0000 mon.a (mon.0) 11238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:35 smithi115 bash[45495]: audit 2022-04-22T21:11:34.346356+0000 mon.a (mon.0) 11238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:35.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:35 smithi115 bash[46934]: audit 2022-04-22T21:11:34.346356+0000 mon.a (mon.0) 11238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:36 smithi121 bash[36902]: cluster 2022-04-22T21:11:34.952061+0000 mgr.y (mgr.34107) 7703 : cluster [DBG] pgmap v8706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:36 smithi121 bash[36902]: audit 2022-04-22T21:11:35.588179+0000 mon.a (mon.0) 11239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:36 smithi115 bash[45495]: cluster 2022-04-22T21:11:34.952061+0000 mgr.y (mgr.34107) 7703 : cluster [DBG] pgmap v8706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:36 smithi115 bash[45495]: audit 2022-04-22T21:11:35.588179+0000 mon.a (mon.0) 11239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:36 smithi115 bash[46934]: cluster 2022-04-22T21:11:34.952061+0000 mgr.y (mgr.34107) 7703 : cluster [DBG] pgmap v8706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:36 smithi115 bash[46934]: audit 2022-04-22T21:11:35.588179+0000 mon.a (mon.0) 11239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:38.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:38 smithi121 bash[36902]: cluster 2022-04-22T21:11:36.953184+0000 mgr.y (mgr.34107) 7704 : cluster [DBG] pgmap v8707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:38 smithi115 bash[45495]: cluster 2022-04-22T21:11:36.953184+0000 mgr.y (mgr.34107) 7704 : cluster [DBG] pgmap v8707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:38 smithi115 bash[46934]: cluster 2022-04-22T21:11:36.953184+0000 mgr.y (mgr.34107) 7704 : cluster [DBG] pgmap v8707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:39 smithi115 bash[45495]: audit 2022-04-22T21:11:38.578913+0000 mon.a (mon.0) 11240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:39 smithi115 bash[46934]: audit 2022-04-22T21:11:38.578913+0000 mon.a (mon.0) 11240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:39 smithi121 bash[36902]: audit 2022-04-22T21:11:38.578913+0000 mon.a (mon.0) 11240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:40 smithi115 bash[45495]: cluster 2022-04-22T21:11:38.954299+0000 mgr.y (mgr.34107) 7705 : cluster [DBG] pgmap v8708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:40 smithi115 bash[45495]: audit 2022-04-22T21:11:39.350036+0000 mon.a (mon.0) 11241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:40 smithi115 bash[46934]: cluster 2022-04-22T21:11:38.954299+0000 mgr.y (mgr.34107) 7705 : cluster [DBG] pgmap v8708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:40 smithi115 bash[46934]: audit 2022-04-22T21:11:39.350036+0000 mon.a (mon.0) 11241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:40 smithi121 bash[36902]: cluster 2022-04-22T21:11:38.954299+0000 mgr.y (mgr.34107) 7705 : cluster [DBG] pgmap v8708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:40 smithi121 bash[36902]: audit 2022-04-22T21:11:39.350036+0000 mon.a (mon.0) 11241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:41 smithi115 bash[45495]: audit 2022-04-22T21:11:40.591424+0000 mon.a (mon.0) 11242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:41 smithi115 bash[46934]: audit 2022-04-22T21:11:40.591424+0000 mon.a (mon.0) 11242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:41.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:41 smithi121 bash[36902]: audit 2022-04-22T21:11:40.591424+0000 mon.a (mon.0) 11242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:42 smithi115 bash[45495]: cluster 2022-04-22T21:11:40.955382+0000 mgr.y (mgr.34107) 7706 : cluster [DBG] pgmap v8709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:42.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:42.492Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:42 smithi115 bash[46934]: cluster 2022-04-22T21:11:40.955382+0000 mgr.y (mgr.34107) 7706 : cluster [DBG] pgmap v8709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:42 smithi121 bash[36902]: cluster 2022-04-22T21:11:40.955382+0000 mgr.y (mgr.34107) 7706 : cluster [DBG] pgmap v8709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:44.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:44.367Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:44 smithi115 bash[45495]: cluster 2022-04-22T21:11:42.956347+0000 mgr.y (mgr.34107) 7707 : cluster [DBG] pgmap v8710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:44 smithi115 bash[45495]: audit 2022-04-22T21:11:43.582115+0000 mon.a (mon.0) 11243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:44 smithi115 bash[46934]: cluster 2022-04-22T21:11:42.956347+0000 mgr.y (mgr.34107) 7707 : cluster [DBG] pgmap v8710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:44 smithi115 bash[46934]: audit 2022-04-22T21:11:43.582115+0000 mon.a (mon.0) 11243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:44 smithi121 bash[36902]: cluster 2022-04-22T21:11:42.956347+0000 mgr.y (mgr.34107) 7707 : cluster [DBG] pgmap v8710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:44 smithi121 bash[36902]: audit 2022-04-22T21:11:43.582115+0000 mon.a (mon.0) 11243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:45.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:45 smithi121 bash[36902]: audit 2022-04-22T21:11:44.353676+0000 mon.a (mon.0) 11244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:45.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:45 smithi121 bash[36902]: audit 2022-04-22T21:11:44.746596+0000 mon.a (mon.0) 11245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:11:45.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:45 smithi121 bash[36902]: audit 2022-04-22T21:11:44.749621+0000 mon.a (mon.0) 11246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:11:45.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:45 smithi121 bash[36902]: audit 2022-04-22T21:11:44.751349+0000 mon.a (mon.0) 11247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:11:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[45495]: audit 2022-04-22T21:11:44.353676+0000 mon.a (mon.0) 11244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[45495]: audit 2022-04-22T21:11:44.746596+0000 mon.a (mon.0) 11245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:11:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[45495]: audit 2022-04-22T21:11:44.749621+0000 mon.a (mon.0) 11246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:11:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[45495]: audit 2022-04-22T21:11:44.751349+0000 mon.a (mon.0) 11247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:11:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[46934]: audit 2022-04-22T21:11:44.353676+0000 mon.a (mon.0) 11244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[46934]: audit 2022-04-22T21:11:44.746596+0000 mon.a (mon.0) 11245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:11:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[46934]: audit 2022-04-22T21:11:44.749621+0000 mon.a (mon.0) 11246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:11:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:45 smithi115 bash[46934]: audit 2022-04-22T21:11:44.751349+0000 mon.a (mon.0) 11247 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:11:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:46 smithi121 bash[36902]: cluster 2022-04-22T21:11:44.957412+0000 mgr.y (mgr.34107) 7708 : cluster [DBG] pgmap v8711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:46 smithi121 bash[36902]: audit 2022-04-22T21:11:45.338140+0000 mon.a (mon.0) 11248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:46 smithi121 bash[36902]: audit 2022-04-22T21:11:45.594379+0000 mon.a (mon.0) 11249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[45495]: cluster 2022-04-22T21:11:44.957412+0000 mgr.y (mgr.34107) 7708 : cluster [DBG] pgmap v8711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[45495]: audit 2022-04-22T21:11:45.338140+0000 mon.a (mon.0) 11248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[45495]: audit 2022-04-22T21:11:45.594379+0000 mon.a (mon.0) 11249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[46934]: cluster 2022-04-22T21:11:44.957412+0000 mgr.y (mgr.34107) 7708 : cluster [DBG] pgmap v8711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[46934]: audit 2022-04-22T21:11:45.338140+0000 mon.a (mon.0) 11248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:46 smithi115 bash[46934]: audit 2022-04-22T21:11:45.594379+0000 mon.a (mon.0) 11249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:47.690 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:11:48.032 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:11:48.032 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:11:48.032 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:11:48.033 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:11:48.034 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:11:48.439 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:11:48.439 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:11:48.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:11:48.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:11:48.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:11:48.441 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:11:48.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:11:48.441 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:11:48.441 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:11:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:48 smithi121 bash[36902]: cluster 2022-04-22T21:11:46.958540+0000 mgr.y (mgr.34107) 7709 : cluster [DBG] pgmap v8712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:48 smithi121 bash[36902]: audit 2022-04-22T21:11:47.670369+0000 mgr.y (mgr.34107) 7710 : audit [DBG] from='client.43380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:48 smithi115 bash[45495]: cluster 2022-04-22T21:11:46.958540+0000 mgr.y (mgr.34107) 7709 : cluster [DBG] pgmap v8712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:48.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:48 smithi115 bash[45495]: audit 2022-04-22T21:11:47.670369+0000 mgr.y (mgr.34107) 7710 : audit [DBG] from='client.43380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:48 smithi115 bash[46934]: cluster 2022-04-22T21:11:46.958540+0000 mgr.y (mgr.34107) 7709 : cluster [DBG] pgmap v8712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:48 smithi115 bash[46934]: audit 2022-04-22T21:11:47.670369+0000 mgr.y (mgr.34107) 7710 : audit [DBG] from='client.43380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:49.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:49 smithi121 bash[36902]: audit 2022-04-22T21:11:48.012785+0000 mgr.y (mgr.34107) 7711 : audit [DBG] from='client.53317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:49.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:49 smithi121 bash[36902]: audit 2022-04-22T21:11:48.434793+0000 mon.c (mon.1) 280 : audit [DBG] from='client.? 172.21.15.115:0/1219117735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:49.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:49 smithi121 bash[36902]: audit 2022-04-22T21:11:48.585424+0000 mon.a (mon.0) 11250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[45495]: audit 2022-04-22T21:11:48.012785+0000 mgr.y (mgr.34107) 7711 : audit [DBG] from='client.53317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[45495]: audit 2022-04-22T21:11:48.434793+0000 mon.c (mon.1) 280 : audit [DBG] from='client.? 172.21.15.115:0/1219117735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:49.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[45495]: audit 2022-04-22T21:11:48.585424+0000 mon.a (mon.0) 11250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[46934]: audit 2022-04-22T21:11:48.012785+0000 mgr.y (mgr.34107) 7711 : audit [DBG] from='client.53317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:11:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[46934]: audit 2022-04-22T21:11:48.434793+0000 mon.c (mon.1) 280 : audit [DBG] from='client.? 172.21.15.115:0/1219117735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:11:49.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:49 smithi115 bash[46934]: audit 2022-04-22T21:11:48.585424+0000 mon.a (mon.0) 11250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:50 smithi121 bash[36902]: cluster 2022-04-22T21:11:48.959718+0000 mgr.y (mgr.34107) 7712 : cluster [DBG] pgmap v8713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:50 smithi121 bash[36902]: audit 2022-04-22T21:11:49.357183+0000 mon.a (mon.0) 11251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:50 smithi115 bash[45495]: cluster 2022-04-22T21:11:48.959718+0000 mgr.y (mgr.34107) 7712 : cluster [DBG] pgmap v8713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:50 smithi115 bash[45495]: audit 2022-04-22T21:11:49.357183+0000 mon.a (mon.0) 11251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:50 smithi115 bash[46934]: cluster 2022-04-22T21:11:48.959718+0000 mgr.y (mgr.34107) 7712 : cluster [DBG] pgmap v8713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:50 smithi115 bash[46934]: audit 2022-04-22T21:11:49.357183+0000 mon.a (mon.0) 11251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:51 smithi121 bash[36902]: audit 2022-04-22T21:11:50.597245+0000 mon.a (mon.0) 11252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:51 smithi115 bash[45495]: audit 2022-04-22T21:11:50.597245+0000 mon.a (mon.0) 11252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:51 smithi115 bash[46934]: audit 2022-04-22T21:11:50.597245+0000 mon.a (mon.0) 11252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:52 smithi121 bash[36902]: cluster 2022-04-22T21:11:50.960718+0000 mgr.y (mgr.34107) 7713 : cluster [DBG] pgmap v8714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:52 smithi115 bash[45495]: cluster 2022-04-22T21:11:50.960718+0000 mgr.y (mgr.34107) 7713 : cluster [DBG] pgmap v8714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:52.493Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:52 smithi115 bash[46934]: cluster 2022-04-22T21:11:50.960718+0000 mgr.y (mgr.34107) 7713 : cluster [DBG] pgmap v8714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:54 smithi121 bash[36902]: cluster 2022-04-22T21:11:52.961732+0000 mgr.y (mgr.34107) 7714 : cluster [DBG] pgmap v8715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:54 smithi121 bash[36902]: audit 2022-04-22T21:11:53.588502+0000 mon.a (mon.0) 11253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:54 smithi121 bash[36902]: audit 2022-04-22T21:11:54.351776+0000 mon.a (mon.0) 11254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:54 smithi121 bash[36902]: audit 2022-04-22T21:11:54.358590+0000 mon.a (mon.0) 11255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:54 smithi121 bash[36902]: audit 2022-04-22T21:11:54.359971+0000 mon.a (mon.0) 11256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:11:54.372Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[45495]: cluster 2022-04-22T21:11:52.961732+0000 mgr.y (mgr.34107) 7714 : cluster [DBG] pgmap v8715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[45495]: audit 2022-04-22T21:11:53.588502+0000 mon.a (mon.0) 11253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[45495]: audit 2022-04-22T21:11:54.351776+0000 mon.a (mon.0) 11254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[45495]: audit 2022-04-22T21:11:54.358590+0000 mon.a (mon.0) 11255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[45495]: audit 2022-04-22T21:11:54.359971+0000 mon.a (mon.0) 11256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[46934]: cluster 2022-04-22T21:11:52.961732+0000 mgr.y (mgr.34107) 7714 : cluster [DBG] pgmap v8715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[46934]: audit 2022-04-22T21:11:53.588502+0000 mon.a (mon.0) 11253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[46934]: audit 2022-04-22T21:11:54.351776+0000 mon.a (mon.0) 11254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[46934]: audit 2022-04-22T21:11:54.358590+0000 mon.a (mon.0) 11255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:11:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:54 smithi115 bash[46934]: audit 2022-04-22T21:11:54.359971+0000 mon.a (mon.0) 11256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:55 smithi121 bash[36902]: cephadm 2022-04-22T21:11:54.340204+0000 mgr.y (mgr.34107) 7715 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:11:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:55 smithi115 bash[45495]: cephadm 2022-04-22T21:11:54.340204+0000 mgr.y (mgr.34107) 7715 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:11:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:55 smithi115 bash[46934]: cephadm 2022-04-22T21:11:54.340204+0000 mgr.y (mgr.34107) 7715 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:11:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:56 smithi121 bash[36902]: cluster 2022-04-22T21:11:54.962819+0000 mgr.y (mgr.34107) 7716 : cluster [DBG] pgmap v8716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:56 smithi121 bash[36902]: audit 2022-04-22T21:11:55.600223+0000 mon.a (mon.0) 11257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:56 smithi115 bash[45495]: cluster 2022-04-22T21:11:54.962819+0000 mgr.y (mgr.34107) 7716 : cluster [DBG] pgmap v8716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:56 smithi115 bash[45495]: audit 2022-04-22T21:11:55.600223+0000 mon.a (mon.0) 11257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:56 smithi115 bash[46934]: cluster 2022-04-22T21:11:54.962819+0000 mgr.y (mgr.34107) 7716 : cluster [DBG] pgmap v8716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:56 smithi115 bash[46934]: audit 2022-04-22T21:11:55.600223+0000 mon.a (mon.0) 11257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:11:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:58 smithi121 bash[36902]: cluster 2022-04-22T21:11:56.963937+0000 mgr.y (mgr.34107) 7717 : cluster [DBG] pgmap v8717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:58 smithi115 bash[46934]: cluster 2022-04-22T21:11:56.963937+0000 mgr.y (mgr.34107) 7717 : cluster [DBG] pgmap v8717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:58.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:58 smithi115 bash[45495]: cluster 2022-04-22T21:11:56.963937+0000 mgr.y (mgr.34107) 7717 : cluster [DBG] pgmap v8717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:11:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:59 smithi121 bash[36902]: audit 2022-04-22T21:11:58.591425+0000 mon.a (mon.0) 11258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:11:59 smithi121 bash[36902]: audit 2022-04-22T21:11:59.363584+0000 mon.a (mon.0) 11259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:59 smithi115 bash[45495]: audit 2022-04-22T21:11:58.591425+0000 mon.a (mon.0) 11258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:11:59 smithi115 bash[45495]: audit 2022-04-22T21:11:59.363584+0000 mon.a (mon.0) 11259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:11:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:59 smithi115 bash[46934]: audit 2022-04-22T21:11:58.591425+0000 mon.a (mon.0) 11258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:11:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:11:59 smithi115 bash[46934]: audit 2022-04-22T21:11:59.363584+0000 mon.a (mon.0) 11259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:00 smithi121 bash[36902]: cluster 2022-04-22T21:11:58.965060+0000 mgr.y (mgr.34107) 7718 : cluster [DBG] pgmap v8718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:00 smithi115 bash[45495]: cluster 2022-04-22T21:11:58.965060+0000 mgr.y (mgr.34107) 7718 : cluster [DBG] pgmap v8718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:00 smithi115 bash[46934]: cluster 2022-04-22T21:11:58.965060+0000 mgr.y (mgr.34107) 7718 : cluster [DBG] pgmap v8718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:01 smithi121 bash[36902]: audit 2022-04-22T21:12:00.603368+0000 mon.a (mon.0) 11260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:01 smithi115 bash[45495]: audit 2022-04-22T21:12:00.603368+0000 mon.a (mon.0) 11260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:01 smithi115 bash[46934]: audit 2022-04-22T21:12:00.603368+0000 mon.a (mon.0) 11260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:02 smithi121 bash[36902]: cluster 2022-04-22T21:12:00.966151+0000 mgr.y (mgr.34107) 7719 : cluster [DBG] pgmap v8719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:02.492Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:02 smithi115 bash[45495]: cluster 2022-04-22T21:12:00.966151+0000 mgr.y (mgr.34107) 7719 : cluster [DBG] pgmap v8719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:02 smithi115 bash[46934]: cluster 2022-04-22T21:12:00.966151+0000 mgr.y (mgr.34107) 7719 : cluster [DBG] pgmap v8719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:04 smithi121 bash[36902]: cluster 2022-04-22T21:12:02.967175+0000 mgr.y (mgr.34107) 7720 : cluster [DBG] pgmap v8720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:04 smithi121 bash[36902]: audit 2022-04-22T21:12:03.594575+0000 mon.a (mon.0) 11261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:04 smithi121 bash[36902]: audit 2022-04-22T21:12:04.367192+0000 mon.a (mon.0) 11262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:04.369Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[45495]: cluster 2022-04-22T21:12:02.967175+0000 mgr.y (mgr.34107) 7720 : cluster [DBG] pgmap v8720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[45495]: audit 2022-04-22T21:12:03.594575+0000 mon.a (mon.0) 11261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[45495]: audit 2022-04-22T21:12:04.367192+0000 mon.a (mon.0) 11262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[46934]: cluster 2022-04-22T21:12:02.967175+0000 mgr.y (mgr.34107) 7720 : cluster [DBG] pgmap v8720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[46934]: audit 2022-04-22T21:12:03.594575+0000 mon.a (mon.0) 11261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:04 smithi115 bash[46934]: audit 2022-04-22T21:12:04.367192+0000 mon.a (mon.0) 11262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:06 smithi121 bash[36902]: cluster 2022-04-22T21:12:04.968297+0000 mgr.y (mgr.34107) 7721 : cluster [DBG] pgmap v8721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:06 smithi121 bash[36902]: audit 2022-04-22T21:12:05.606548+0000 mon.a (mon.0) 11263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:06 smithi115 bash[45495]: cluster 2022-04-22T21:12:04.968297+0000 mgr.y (mgr.34107) 7721 : cluster [DBG] pgmap v8721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:06.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:06 smithi115 bash[45495]: audit 2022-04-22T21:12:05.606548+0000 mon.a (mon.0) 11263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:06.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:06 smithi115 bash[46934]: cluster 2022-04-22T21:12:04.968297+0000 mgr.y (mgr.34107) 7721 : cluster [DBG] pgmap v8721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:06.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:06 smithi115 bash[46934]: audit 2022-04-22T21:12:05.606548+0000 mon.a (mon.0) 11263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:08 smithi121 bash[36902]: cluster 2022-04-22T21:12:06.969380+0000 mgr.y (mgr.34107) 7722 : cluster [DBG] pgmap v8722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:08 smithi115 bash[45495]: cluster 2022-04-22T21:12:06.969380+0000 mgr.y (mgr.34107) 7722 : cluster [DBG] pgmap v8722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:08 smithi115 bash[46934]: cluster 2022-04-22T21:12:06.969380+0000 mgr.y (mgr.34107) 7722 : cluster [DBG] pgmap v8722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:09 smithi121 bash[36902]: audit 2022-04-22T21:12:08.597793+0000 mon.a (mon.0) 11264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:09 smithi121 bash[36902]: audit 2022-04-22T21:12:09.371041+0000 mon.a (mon.0) 11265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:09 smithi115 bash[45495]: audit 2022-04-22T21:12:08.597793+0000 mon.a (mon.0) 11264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:09 smithi115 bash[45495]: audit 2022-04-22T21:12:09.371041+0000 mon.a (mon.0) 11265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:09 smithi115 bash[46934]: audit 2022-04-22T21:12:08.597793+0000 mon.a (mon.0) 11264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:09 smithi115 bash[46934]: audit 2022-04-22T21:12:09.371041+0000 mon.a (mon.0) 11265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:10 smithi121 bash[36902]: cluster 2022-04-22T21:12:08.970525+0000 mgr.y (mgr.34107) 7723 : cluster [DBG] pgmap v8723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:10 smithi115 bash[45495]: cluster 2022-04-22T21:12:08.970525+0000 mgr.y (mgr.34107) 7723 : cluster [DBG] pgmap v8723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:10 smithi115 bash[46934]: cluster 2022-04-22T21:12:08.970525+0000 mgr.y (mgr.34107) 7723 : cluster [DBG] pgmap v8723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:11 smithi121 bash[36902]: audit 2022-04-22T21:12:10.609325+0000 mon.a (mon.0) 11266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:11.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:11 smithi115 bash[45495]: audit 2022-04-22T21:12:10.609325+0000 mon.a (mon.0) 11266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:11.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:11 smithi115 bash[46934]: audit 2022-04-22T21:12:10.609325+0000 mon.a (mon.0) 11266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:12 smithi121 bash[36902]: cluster 2022-04-22T21:12:10.971651+0000 mgr.y (mgr.34107) 7724 : cluster [DBG] pgmap v8724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:12.493Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:12 smithi115 bash[45495]: cluster 2022-04-22T21:12:10.971651+0000 mgr.y (mgr.34107) 7724 : cluster [DBG] pgmap v8724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:12 smithi115 bash[46934]: cluster 2022-04-22T21:12:10.971651+0000 mgr.y (mgr.34107) 7724 : cluster [DBG] pgmap v8724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:14 smithi121 bash[36902]: cluster 2022-04-22T21:12:12.972587+0000 mgr.y (mgr.34107) 7725 : cluster [DBG] pgmap v8725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:14 smithi121 bash[36902]: audit 2022-04-22T21:12:13.600970+0000 mon.a (mon.0) 11267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:14 smithi121 bash[36902]: audit 2022-04-22T21:12:14.374758+0000 mon.a (mon.0) 11268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:14.371Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[45495]: cluster 2022-04-22T21:12:12.972587+0000 mgr.y (mgr.34107) 7725 : cluster [DBG] pgmap v8725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[45495]: audit 2022-04-22T21:12:13.600970+0000 mon.a (mon.0) 11267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[45495]: audit 2022-04-22T21:12:14.374758+0000 mon.a (mon.0) 11268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[46934]: cluster 2022-04-22T21:12:12.972587+0000 mgr.y (mgr.34107) 7725 : cluster [DBG] pgmap v8725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[46934]: audit 2022-04-22T21:12:13.600970+0000 mon.a (mon.0) 11267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:14 smithi115 bash[46934]: audit 2022-04-22T21:12:14.374758+0000 mon.a (mon.0) 11268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:16 smithi121 bash[36902]: cluster 2022-04-22T21:12:14.973707+0000 mgr.y (mgr.34107) 7726 : cluster [DBG] pgmap v8726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:16 smithi121 bash[36902]: audit 2022-04-22T21:12:15.612429+0000 mon.a (mon.0) 11269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:16 smithi121 bash[36902]: audit 2022-04-22T21:12:15.881795+0000 mon.a (mon.0) 11270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:12:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:16 smithi121 bash[36902]: audit 2022-04-22T21:12:15.882249+0000 mon.a (mon.0) 11271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[45495]: cluster 2022-04-22T21:12:14.973707+0000 mgr.y (mgr.34107) 7726 : cluster [DBG] pgmap v8726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[45495]: audit 2022-04-22T21:12:15.612429+0000 mon.a (mon.0) 11269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[45495]: audit 2022-04-22T21:12:15.881795+0000 mon.a (mon.0) 11270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[45495]: audit 2022-04-22T21:12:15.882249+0000 mon.a (mon.0) 11271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[46934]: cluster 2022-04-22T21:12:14.973707+0000 mgr.y (mgr.34107) 7726 : cluster [DBG] pgmap v8726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[46934]: audit 2022-04-22T21:12:15.612429+0000 mon.a (mon.0) 11269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[46934]: audit 2022-04-22T21:12:15.881795+0000 mon.a (mon.0) 11270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:12:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:16 smithi115 bash[46934]: audit 2022-04-22T21:12:15.882249+0000 mon.a (mon.0) 11271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:12:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:18 smithi121 bash[36902]: cluster 2022-04-22T21:12:16.974824+0000 mgr.y (mgr.34107) 7727 : cluster [DBG] pgmap v8727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:18 smithi115 bash[45495]: cluster 2022-04-22T21:12:16.974824+0000 mgr.y (mgr.34107) 7727 : cluster [DBG] pgmap v8727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:18 smithi115 bash[46934]: cluster 2022-04-22T21:12:16.974824+0000 mgr.y (mgr.34107) 7727 : cluster [DBG] pgmap v8727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:18.811 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:12:19.153 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:12:19.153 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:12:19.154 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:12:19.155 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:12:19.562 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:12:19.562 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:12:19.562 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:12:19.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:19.564 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:12:19.564 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:12:19.564 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:12:19.564 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:12:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:19 smithi121 bash[36902]: audit 2022-04-22T21:12:18.604112+0000 mon.a (mon.0) 11272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:19 smithi121 bash[36902]: audit 2022-04-22T21:12:19.378209+0000 mon.a (mon.0) 11273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:19 smithi115 bash[45495]: audit 2022-04-22T21:12:18.604112+0000 mon.a (mon.0) 11272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:19.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:19 smithi115 bash[45495]: audit 2022-04-22T21:12:19.378209+0000 mon.a (mon.0) 11273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:19 smithi115 bash[46934]: audit 2022-04-22T21:12:18.604112+0000 mon.a (mon.0) 11272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:19 smithi115 bash[46934]: audit 2022-04-22T21:12:19.378209+0000 mon.a (mon.0) 11273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:20 smithi121 bash[36902]: audit 2022-04-22T21:12:18.790901+0000 mgr.y (mgr.34107) 7728 : audit [DBG] from='client.53329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:20 smithi121 bash[36902]: cluster 2022-04-22T21:12:18.975913+0000 mgr.y (mgr.34107) 7729 : cluster [DBG] pgmap v8728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:20 smithi121 bash[36902]: audit 2022-04-22T21:12:19.130574+0000 mgr.y (mgr.34107) 7730 : audit [DBG] from='client.43404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:20 smithi121 bash[36902]: audit 2022-04-22T21:12:19.557858+0000 mon.a (mon.0) 11274 : audit [DBG] from='client.? 172.21.15.115:0/1596285922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[45495]: audit 2022-04-22T21:12:18.790901+0000 mgr.y (mgr.34107) 7728 : audit [DBG] from='client.53329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[45495]: cluster 2022-04-22T21:12:18.975913+0000 mgr.y (mgr.34107) 7729 : cluster [DBG] pgmap v8728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[45495]: audit 2022-04-22T21:12:19.130574+0000 mgr.y (mgr.34107) 7730 : audit [DBG] from='client.43404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[45495]: audit 2022-04-22T21:12:19.557858+0000 mon.a (mon.0) 11274 : audit [DBG] from='client.? 172.21.15.115:0/1596285922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[46934]: audit 2022-04-22T21:12:18.790901+0000 mgr.y (mgr.34107) 7728 : audit [DBG] from='client.53329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[46934]: cluster 2022-04-22T21:12:18.975913+0000 mgr.y (mgr.34107) 7729 : cluster [DBG] pgmap v8728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[46934]: audit 2022-04-22T21:12:19.130574+0000 mgr.y (mgr.34107) 7730 : audit [DBG] from='client.43404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:20 smithi115 bash[46934]: audit 2022-04-22T21:12:19.557858+0000 mon.a (mon.0) 11274 : audit [DBG] from='client.? 172.21.15.115:0/1596285922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:21 smithi121 bash[36902]: audit 2022-04-22T21:12:20.615369+0000 mon.a (mon.0) 11275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:21 smithi115 bash[45495]: audit 2022-04-22T21:12:20.615369+0000 mon.a (mon.0) 11275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:21 smithi115 bash[46934]: audit 2022-04-22T21:12:20.615369+0000 mon.a (mon.0) 11275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:22 smithi121 bash[36902]: cluster 2022-04-22T21:12:20.976970+0000 mgr.y (mgr.34107) 7731 : cluster [DBG] pgmap v8729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:22 smithi115 bash[45495]: cluster 2022-04-22T21:12:20.976970+0000 mgr.y (mgr.34107) 7731 : cluster [DBG] pgmap v8729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:22 smithi115 bash[46934]: cluster 2022-04-22T21:12:20.976970+0000 mgr.y (mgr.34107) 7731 : cluster [DBG] pgmap v8729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:22.492Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:24.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:24 smithi121 bash[36902]: cluster 2022-04-22T21:12:22.978066+0000 mgr.y (mgr.34107) 7732 : cluster [DBG] pgmap v8730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:24 smithi121 bash[36902]: audit 2022-04-22T21:12:23.607171+0000 mon.a (mon.0) 11276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:24 smithi121 bash[36902]: audit 2022-04-22T21:12:24.381773+0000 mon.a (mon.0) 11277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:24.370Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:24.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[45495]: cluster 2022-04-22T21:12:22.978066+0000 mgr.y (mgr.34107) 7732 : cluster [DBG] pgmap v8730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:24.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[45495]: audit 2022-04-22T21:12:23.607171+0000 mon.a (mon.0) 11276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:24.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[45495]: audit 2022-04-22T21:12:24.381773+0000 mon.a (mon.0) 11277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:24.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[46934]: cluster 2022-04-22T21:12:22.978066+0000 mgr.y (mgr.34107) 7732 : cluster [DBG] pgmap v8730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:24.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[46934]: audit 2022-04-22T21:12:23.607171+0000 mon.a (mon.0) 11276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:24.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:24 smithi115 bash[46934]: audit 2022-04-22T21:12:24.381773+0000 mon.a (mon.0) 11277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:26 smithi121 bash[36902]: cluster 2022-04-22T21:12:24.979269+0000 mgr.y (mgr.34107) 7733 : cluster [DBG] pgmap v8731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:26 smithi121 bash[36902]: audit 2022-04-22T21:12:25.619791+0000 mon.a (mon.0) 11278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:26 smithi115 bash[45495]: cluster 2022-04-22T21:12:24.979269+0000 mgr.y (mgr.34107) 7733 : cluster [DBG] pgmap v8731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:26 smithi115 bash[45495]: audit 2022-04-22T21:12:25.619791+0000 mon.a (mon.0) 11278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:26 smithi115 bash[46934]: cluster 2022-04-22T21:12:24.979269+0000 mgr.y (mgr.34107) 7733 : cluster [DBG] pgmap v8731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:26 smithi115 bash[46934]: audit 2022-04-22T21:12:25.619791+0000 mon.a (mon.0) 11278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:28 smithi121 bash[36902]: cluster 2022-04-22T21:12:26.980472+0000 mgr.y (mgr.34107) 7734 : cluster [DBG] pgmap v8732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:28 smithi115 bash[45495]: cluster 2022-04-22T21:12:26.980472+0000 mgr.y (mgr.34107) 7734 : cluster [DBG] pgmap v8732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:28.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:28 smithi115 bash[46934]: cluster 2022-04-22T21:12:26.980472+0000 mgr.y (mgr.34107) 7734 : cluster [DBG] pgmap v8732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:29 smithi121 bash[36902]: audit 2022-04-22T21:12:28.610250+0000 mon.a (mon.0) 11279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:29 smithi121 bash[36902]: audit 2022-04-22T21:12:29.385304+0000 mon.a (mon.0) 11280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:29 smithi115 bash[45495]: audit 2022-04-22T21:12:28.610250+0000 mon.a (mon.0) 11279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:29 smithi115 bash[45495]: audit 2022-04-22T21:12:29.385304+0000 mon.a (mon.0) 11280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:29 smithi115 bash[46934]: audit 2022-04-22T21:12:28.610250+0000 mon.a (mon.0) 11279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:29 smithi115 bash[46934]: audit 2022-04-22T21:12:29.385304+0000 mon.a (mon.0) 11280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:30.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:30 smithi121 bash[36902]: cluster 2022-04-22T21:12:28.981658+0000 mgr.y (mgr.34107) 7735 : cluster [DBG] pgmap v8733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:30 smithi115 bash[45495]: cluster 2022-04-22T21:12:28.981658+0000 mgr.y (mgr.34107) 7735 : cluster [DBG] pgmap v8733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:30 smithi115 bash[46934]: cluster 2022-04-22T21:12:28.981658+0000 mgr.y (mgr.34107) 7735 : cluster [DBG] pgmap v8733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:31 smithi115 bash[46934]: audit 2022-04-22T21:12:30.622558+0000 mon.a (mon.0) 11281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:31.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:31 smithi115 bash[45495]: audit 2022-04-22T21:12:30.622558+0000 mon.a (mon.0) 11281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:32.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:31 smithi121 bash[36902]: audit 2022-04-22T21:12:30.622558+0000 mon.a (mon.0) 11281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:32 smithi115 bash[46934]: cluster 2022-04-22T21:12:30.982764+0000 mgr.y (mgr.34107) 7736 : cluster [DBG] pgmap v8734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:32.495Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:32.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:32 smithi115 bash[45495]: cluster 2022-04-22T21:12:30.982764+0000 mgr.y (mgr.34107) 7736 : cluster [DBG] pgmap v8734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:33.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:32 smithi121 bash[36902]: cluster 2022-04-22T21:12:30.982764+0000 mgr.y (mgr.34107) 7736 : cluster [DBG] pgmap v8734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:34.373Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[45495]: cluster 2022-04-22T21:12:32.983795+0000 mgr.y (mgr.34107) 7737 : cluster [DBG] pgmap v8735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[45495]: audit 2022-04-22T21:12:33.613396+0000 mon.a (mon.0) 11282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[45495]: audit 2022-04-22T21:12:34.389088+0000 mon.a (mon.0) 11283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[46934]: cluster 2022-04-22T21:12:32.983795+0000 mgr.y (mgr.34107) 7737 : cluster [DBG] pgmap v8735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[46934]: audit 2022-04-22T21:12:33.613396+0000 mon.a (mon.0) 11282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:34 smithi115 bash[46934]: audit 2022-04-22T21:12:34.389088+0000 mon.a (mon.0) 11283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:34 smithi121 bash[36902]: cluster 2022-04-22T21:12:32.983795+0000 mgr.y (mgr.34107) 7737 : cluster [DBG] pgmap v8735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:34 smithi121 bash[36902]: audit 2022-04-22T21:12:33.613396+0000 mon.a (mon.0) 11282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:34 smithi121 bash[36902]: audit 2022-04-22T21:12:34.389088+0000 mon.a (mon.0) 11283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:36 smithi115 bash[45495]: cluster 2022-04-22T21:12:34.984936+0000 mgr.y (mgr.34107) 7738 : cluster [DBG] pgmap v8736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:36 smithi115 bash[45495]: audit 2022-04-22T21:12:35.625628+0000 mon.a (mon.0) 11284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:36 smithi115 bash[46934]: cluster 2022-04-22T21:12:34.984936+0000 mgr.y (mgr.34107) 7738 : cluster [DBG] pgmap v8736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:36 smithi115 bash[46934]: audit 2022-04-22T21:12:35.625628+0000 mon.a (mon.0) 11284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:36 smithi121 bash[36902]: cluster 2022-04-22T21:12:34.984936+0000 mgr.y (mgr.34107) 7738 : cluster [DBG] pgmap v8736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:36 smithi121 bash[36902]: audit 2022-04-22T21:12:35.625628+0000 mon.a (mon.0) 11284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:38 smithi115 bash[45495]: cluster 2022-04-22T21:12:36.986070+0000 mgr.y (mgr.34107) 7739 : cluster [DBG] pgmap v8737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:38 smithi115 bash[46934]: cluster 2022-04-22T21:12:36.986070+0000 mgr.y (mgr.34107) 7739 : cluster [DBG] pgmap v8737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:38 smithi121 bash[36902]: cluster 2022-04-22T21:12:36.986070+0000 mgr.y (mgr.34107) 7739 : cluster [DBG] pgmap v8737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:39 smithi121 bash[36902]: audit 2022-04-22T21:12:38.616619+0000 mon.a (mon.0) 11285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:39 smithi121 bash[36902]: audit 2022-04-22T21:12:39.392715+0000 mon.a (mon.0) 11286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:39 smithi115 bash[45495]: audit 2022-04-22T21:12:38.616619+0000 mon.a (mon.0) 11285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:40.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:39 smithi115 bash[45495]: audit 2022-04-22T21:12:39.392715+0000 mon.a (mon.0) 11286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:39 smithi115 bash[46934]: audit 2022-04-22T21:12:38.616619+0000 mon.a (mon.0) 11285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:40.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:39 smithi115 bash[46934]: audit 2022-04-22T21:12:39.392715+0000 mon.a (mon.0) 11286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:40 smithi121 bash[36902]: cluster 2022-04-22T21:12:38.987227+0000 mgr.y (mgr.34107) 7740 : cluster [DBG] pgmap v8738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:40 smithi115 bash[45495]: cluster 2022-04-22T21:12:38.987227+0000 mgr.y (mgr.34107) 7740 : cluster [DBG] pgmap v8738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:40 smithi115 bash[46934]: cluster 2022-04-22T21:12:38.987227+0000 mgr.y (mgr.34107) 7740 : cluster [DBG] pgmap v8738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:42.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:41 smithi121 bash[36902]: audit 2022-04-22T21:12:40.628701+0000 mon.a (mon.0) 11287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:41 smithi115 bash[45495]: audit 2022-04-22T21:12:40.628701+0000 mon.a (mon.0) 11287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:42.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:41 smithi115 bash[46934]: audit 2022-04-22T21:12:40.628701+0000 mon.a (mon.0) 11287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:42 smithi115 bash[46934]: cluster 2022-04-22T21:12:40.988347+0000 mgr.y (mgr.34107) 7741 : cluster [DBG] pgmap v8739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:42.495Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:42 smithi115 bash[45495]: cluster 2022-04-22T21:12:40.988347+0000 mgr.y (mgr.34107) 7741 : cluster [DBG] pgmap v8739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:42 smithi121 bash[36902]: cluster 2022-04-22T21:12:40.988347+0000 mgr.y (mgr.34107) 7741 : cluster [DBG] pgmap v8739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:44.372Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[45495]: cluster 2022-04-22T21:12:42.989344+0000 mgr.y (mgr.34107) 7742 : cluster [DBG] pgmap v8740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[45495]: audit 2022-04-22T21:12:43.619616+0000 mon.a (mon.0) 11288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[45495]: audit 2022-04-22T21:12:44.396642+0000 mon.a (mon.0) 11289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[46934]: cluster 2022-04-22T21:12:42.989344+0000 mgr.y (mgr.34107) 7742 : cluster [DBG] pgmap v8740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[46934]: audit 2022-04-22T21:12:43.619616+0000 mon.a (mon.0) 11288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:44 smithi115 bash[46934]: audit 2022-04-22T21:12:44.396642+0000 mon.a (mon.0) 11289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:44 smithi121 bash[36902]: cluster 2022-04-22T21:12:42.989344+0000 mgr.y (mgr.34107) 7742 : cluster [DBG] pgmap v8740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:44 smithi121 bash[36902]: audit 2022-04-22T21:12:43.619616+0000 mon.a (mon.0) 11288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:44 smithi121 bash[36902]: audit 2022-04-22T21:12:44.396642+0000 mon.a (mon.0) 11289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:46 smithi121 bash[36902]: cluster 2022-04-22T21:12:44.990713+0000 mgr.y (mgr.34107) 7743 : cluster [DBG] pgmap v8741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:46 smithi121 bash[36902]: audit 2022-04-22T21:12:45.631558+0000 mon.a (mon.0) 11290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:46 smithi115 bash[45495]: cluster 2022-04-22T21:12:44.990713+0000 mgr.y (mgr.34107) 7743 : cluster [DBG] pgmap v8741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:46 smithi115 bash[45495]: audit 2022-04-22T21:12:45.631558+0000 mon.a (mon.0) 11290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:46 smithi115 bash[46934]: cluster 2022-04-22T21:12:44.990713+0000 mgr.y (mgr.34107) 7743 : cluster [DBG] pgmap v8741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:46 smithi115 bash[46934]: audit 2022-04-22T21:12:45.631558+0000 mon.a (mon.0) 11290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:48 smithi121 bash[36902]: cluster 2022-04-22T21:12:46.991983+0000 mgr.y (mgr.34107) 7744 : cluster [DBG] pgmap v8742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:48.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:48 smithi115 bash[45495]: cluster 2022-04-22T21:12:46.991983+0000 mgr.y (mgr.34107) 7744 : cluster [DBG] pgmap v8742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:48.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:48 smithi115 bash[46934]: cluster 2022-04-22T21:12:46.991983+0000 mgr.y (mgr.34107) 7744 : cluster [DBG] pgmap v8742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:49 smithi115 bash[45495]: audit 2022-04-22T21:12:48.622596+0000 mon.a (mon.0) 11291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:49 smithi115 bash[46934]: audit 2022-04-22T21:12:48.622596+0000 mon.a (mon.0) 11291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:49 smithi121 bash[36902]: audit 2022-04-22T21:12:48.622596+0000 mon.a (mon.0) 11291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:49.932 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:12:50.283 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10m ago 3h 232M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10m ago 3h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:12:50.284 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:12:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10m ago 3h 580M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:12:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:12:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10m ago 3h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:12:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10m ago 3h 642M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:12:50.285 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10m ago 3h 65.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:12:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:50 smithi115 bash[45495]: cluster 2022-04-22T21:12:48.993172+0000 mgr.y (mgr.34107) 7745 : cluster [DBG] pgmap v8743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:50 smithi115 bash[45495]: audit 2022-04-22T21:12:49.400205+0000 mon.a (mon.0) 11292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:50 smithi115 bash[46934]: cluster 2022-04-22T21:12:48.993172+0000 mgr.y (mgr.34107) 7745 : cluster [DBG] pgmap v8743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:50 smithi115 bash[46934]: audit 2022-04-22T21:12:49.400205+0000 mon.a (mon.0) 11292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:50 smithi121 bash[36902]: cluster 2022-04-22T21:12:48.993172+0000 mgr.y (mgr.34107) 7745 : cluster [DBG] pgmap v8743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:50 smithi121 bash[36902]: audit 2022-04-22T21:12:49.400205+0000 mon.a (mon.0) 11292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:50.691 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:12:50.692 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:12:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[46934]: audit 2022-04-22T21:12:49.911016+0000 mgr.y (mgr.34107) 7746 : audit [DBG] from='client.53347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[46934]: audit 2022-04-22T21:12:50.257561+0000 mgr.y (mgr.34107) 7747 : audit [DBG] from='client.53353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[46934]: audit 2022-04-22T21:12:50.634716+0000 mon.a (mon.0) 11293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[46934]: audit 2022-04-22T21:12:50.686251+0000 mon.c (mon.1) 281 : audit [DBG] from='client.? 172.21.15.115:0/793690042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[45495]: audit 2022-04-22T21:12:49.911016+0000 mgr.y (mgr.34107) 7746 : audit [DBG] from='client.53347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[45495]: audit 2022-04-22T21:12:50.257561+0000 mgr.y (mgr.34107) 7747 : audit [DBG] from='client.53353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[45495]: audit 2022-04-22T21:12:50.634716+0000 mon.a (mon.0) 11293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:51.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:51 smithi115 bash[45495]: audit 2022-04-22T21:12:50.686251+0000 mon.c (mon.1) 281 : audit [DBG] from='client.? 172.21.15.115:0/793690042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:51 smithi121 bash[36902]: audit 2022-04-22T21:12:49.911016+0000 mgr.y (mgr.34107) 7746 : audit [DBG] from='client.53347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:51 smithi121 bash[36902]: audit 2022-04-22T21:12:50.257561+0000 mgr.y (mgr.34107) 7747 : audit [DBG] from='client.53353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:12:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:51 smithi121 bash[36902]: audit 2022-04-22T21:12:50.634716+0000 mon.a (mon.0) 11293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:51 smithi121 bash[36902]: audit 2022-04-22T21:12:50.686251+0000 mon.c (mon.1) 281 : audit [DBG] from='client.? 172.21.15.115:0/793690042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:12:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:52 smithi115 bash[46934]: cluster 2022-04-22T21:12:50.994193+0000 mgr.y (mgr.34107) 7748 : cluster [DBG] pgmap v8744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:52 smithi115 bash[45495]: cluster 2022-04-22T21:12:50.994193+0000 mgr.y (mgr.34107) 7748 : cluster [DBG] pgmap v8744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:52 smithi121 bash[36902]: cluster 2022-04-22T21:12:50.994193+0000 mgr.y (mgr.34107) 7748 : cluster [DBG] pgmap v8744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:52.496Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:54 smithi115 bash[45495]: cluster 2022-04-22T21:12:52.995207+0000 mgr.y (mgr.34107) 7749 : cluster [DBG] pgmap v8745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:54 smithi115 bash[45495]: audit 2022-04-22T21:12:53.625658+0000 mon.a (mon.0) 11294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:54.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:54 smithi115 bash[46934]: cluster 2022-04-22T21:12:52.995207+0000 mgr.y (mgr.34107) 7749 : cluster [DBG] pgmap v8745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:54.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:54 smithi115 bash[46934]: audit 2022-04-22T21:12:53.625658+0000 mon.a (mon.0) 11294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:54.403 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:54 smithi121 bash[36902]: cluster 2022-04-22T21:12:52.995207+0000 mgr.y (mgr.34107) 7749 : cluster [DBG] pgmap v8745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:54.403 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:54 smithi121 bash[36902]: audit 2022-04-22T21:12:53.625658+0000 mon.a (mon.0) 11294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:54.707 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:12:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:12:54.373Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:12:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[45495]: audit 2022-04-22T21:12:54.374812+0000 mon.a (mon.0) 11295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[45495]: audit 2022-04-22T21:12:54.377453+0000 mon.a (mon.0) 11296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[45495]: audit 2022-04-22T21:12:54.378988+0000 mon.a (mon.0) 11297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[45495]: audit 2022-04-22T21:12:54.404032+0000 mon.a (mon.0) 11298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[45495]: audit 2022-04-22T21:12:54.974121+0000 mon.a (mon.0) 11299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[46934]: audit 2022-04-22T21:12:54.374812+0000 mon.a (mon.0) 11295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:12:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[46934]: audit 2022-04-22T21:12:54.377453+0000 mon.a (mon.0) 11296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:12:55.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[46934]: audit 2022-04-22T21:12:54.378988+0000 mon.a (mon.0) 11297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:12:55.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[46934]: audit 2022-04-22T21:12:54.404032+0000 mon.a (mon.0) 11298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:55.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:55 smithi115 bash[46934]: audit 2022-04-22T21:12:54.974121+0000 mon.a (mon.0) 11299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:12:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:55 smithi121 bash[36902]: audit 2022-04-22T21:12:54.374812+0000 mon.a (mon.0) 11295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:12:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:55 smithi121 bash[36902]: audit 2022-04-22T21:12:54.377453+0000 mon.a (mon.0) 11296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:12:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:55 smithi121 bash[36902]: audit 2022-04-22T21:12:54.378988+0000 mon.a (mon.0) 11297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:12:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:55 smithi121 bash[36902]: audit 2022-04-22T21:12:54.404032+0000 mon.a (mon.0) 11298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:12:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:55 smithi121 bash[36902]: audit 2022-04-22T21:12:54.974121+0000 mon.a (mon.0) 11299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:12:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:56 smithi115 bash[45495]: cluster 2022-04-22T21:12:54.996258+0000 mgr.y (mgr.34107) 7750 : cluster [DBG] pgmap v8746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:56.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:56 smithi115 bash[45495]: audit 2022-04-22T21:12:55.637640+0000 mon.a (mon.0) 11300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:56 smithi115 bash[46934]: cluster 2022-04-22T21:12:54.996258+0000 mgr.y (mgr.34107) 7750 : cluster [DBG] pgmap v8746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:56.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:56 smithi115 bash[46934]: audit 2022-04-22T21:12:55.637640+0000 mon.a (mon.0) 11300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:56 smithi121 bash[36902]: cluster 2022-04-22T21:12:54.996258+0000 mgr.y (mgr.34107) 7750 : cluster [DBG] pgmap v8746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:56 smithi121 bash[36902]: audit 2022-04-22T21:12:55.637640+0000 mon.a (mon.0) 11300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:12:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:58 smithi121 bash[36902]: cluster 2022-04-22T21:12:56.997329+0000 mgr.y (mgr.34107) 7751 : cluster [DBG] pgmap v8747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:58 smithi115 bash[45495]: cluster 2022-04-22T21:12:56.997329+0000 mgr.y (mgr.34107) 7751 : cluster [DBG] pgmap v8747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:58 smithi115 bash[46934]: cluster 2022-04-22T21:12:56.997329+0000 mgr.y (mgr.34107) 7751 : cluster [DBG] pgmap v8747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:12:59.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:59 smithi121 bash[36902]: audit 2022-04-22T21:12:58.628814+0000 mon.a (mon.0) 11301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:59.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:12:59 smithi121 bash[36902]: audit 2022-04-22T21:12:58.833201+0000 mon.a (mon.0) 11302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:12:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:59 smithi115 bash[45495]: audit 2022-04-22T21:12:58.628814+0000 mon.a (mon.0) 11301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:59.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:12:59 smithi115 bash[45495]: audit 2022-04-22T21:12:58.833201+0000 mon.a (mon.0) 11302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:12:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:59 smithi115 bash[46934]: audit 2022-04-22T21:12:58.628814+0000 mon.a (mon.0) 11301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:12:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:12:59 smithi115 bash[46934]: audit 2022-04-22T21:12:58.833201+0000 mon.a (mon.0) 11302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:13:00.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:00 smithi121 bash[36902]: cluster 2022-04-22T21:12:58.998449+0000 mgr.y (mgr.34107) 7752 : cluster [DBG] pgmap v8748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:00 smithi121 bash[36902]: audit 2022-04-22T21:12:59.183634+0000 mon.a (mon.0) 11303 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:13:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:00 smithi121 bash[36902]: audit 2022-04-22T21:12:59.407688+0000 mon.a (mon.0) 11304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[45495]: cluster 2022-04-22T21:12:58.998449+0000 mgr.y (mgr.34107) 7752 : cluster [DBG] pgmap v8748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[45495]: audit 2022-04-22T21:12:59.183634+0000 mon.a (mon.0) 11303 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:13:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[45495]: audit 2022-04-22T21:12:59.407688+0000 mon.a (mon.0) 11304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[46934]: cluster 2022-04-22T21:12:58.998449+0000 mgr.y (mgr.34107) 7752 : cluster [DBG] pgmap v8748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[46934]: audit 2022-04-22T21:12:59.183634+0000 mon.a (mon.0) 11303 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:13:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:00 smithi115 bash[46934]: audit 2022-04-22T21:12:59.407688+0000 mon.a (mon.0) 11304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:01.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:01 smithi121 bash[36902]: audit 2022-04-22T21:13:00.640677+0000 mon.a (mon.0) 11305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:01 smithi115 bash[45495]: audit 2022-04-22T21:13:00.640677+0000 mon.a (mon.0) 11305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:01 smithi115 bash[46934]: audit 2022-04-22T21:13:00.640677+0000 mon.a (mon.0) 11305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:02.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:02 smithi115 bash[45495]: cluster 2022-04-22T21:13:00.999493+0000 mgr.y (mgr.34107) 7753 : cluster [DBG] pgmap v8749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:02.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:02 smithi115 bash[46934]: cluster 2022-04-22T21:13:00.999493+0000 mgr.y (mgr.34107) 7753 : cluster [DBG] pgmap v8749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:02 smithi121 bash[36902]: cluster 2022-04-22T21:13:00.999493+0000 mgr.y (mgr.34107) 7753 : cluster [DBG] pgmap v8749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:02.498Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:04.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:04 smithi121 bash[36902]: cluster 2022-04-22T21:13:03.000413+0000 mgr.y (mgr.34107) 7754 : cluster [DBG] pgmap v8750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:04 smithi121 bash[36902]: audit 2022-04-22T21:13:03.631704+0000 mon.a (mon.0) 11306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:04 smithi115 bash[46934]: cluster 2022-04-22T21:13:03.000413+0000 mgr.y (mgr.34107) 7754 : cluster [DBG] pgmap v8750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:04.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:04 smithi115 bash[46934]: audit 2022-04-22T21:13:03.631704+0000 mon.a (mon.0) 11306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:04.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:04.373Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:04 smithi115 bash[45495]: cluster 2022-04-22T21:13:03.000413+0000 mgr.y (mgr.34107) 7754 : cluster [DBG] pgmap v8750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:04 smithi115 bash[45495]: audit 2022-04-22T21:13:03.631704+0000 mon.a (mon.0) 11306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:05.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:05 smithi121 bash[36902]: audit 2022-04-22T21:13:04.411357+0000 mon.a (mon.0) 11307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:05.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:05 smithi115 bash[45495]: audit 2022-04-22T21:13:04.411357+0000 mon.a (mon.0) 11307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:05 smithi115 bash[46934]: audit 2022-04-22T21:13:04.411357+0000 mon.a (mon.0) 11307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:06.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:06 smithi121 bash[36902]: cluster 2022-04-22T21:13:05.001492+0000 mgr.y (mgr.34107) 7755 : cluster [DBG] pgmap v8751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:06 smithi121 bash[36902]: audit 2022-04-22T21:13:05.643687+0000 mon.a (mon.0) 11308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:06 smithi115 bash[45495]: cluster 2022-04-22T21:13:05.001492+0000 mgr.y (mgr.34107) 7755 : cluster [DBG] pgmap v8751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:06 smithi115 bash[45495]: audit 2022-04-22T21:13:05.643687+0000 mon.a (mon.0) 11308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:06 smithi115 bash[46934]: cluster 2022-04-22T21:13:05.001492+0000 mgr.y (mgr.34107) 7755 : cluster [DBG] pgmap v8751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:06 smithi115 bash[46934]: audit 2022-04-22T21:13:05.643687+0000 mon.a (mon.0) 11308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:08 smithi121 bash[36902]: cluster 2022-04-22T21:13:07.002613+0000 mgr.y (mgr.34107) 7756 : cluster [DBG] pgmap v8752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:08 smithi115 bash[46934]: cluster 2022-04-22T21:13:07.002613+0000 mgr.y (mgr.34107) 7756 : cluster [DBG] pgmap v8752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:08 smithi115 bash[45495]: cluster 2022-04-22T21:13:07.002613+0000 mgr.y (mgr.34107) 7756 : cluster [DBG] pgmap v8752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:09 smithi121 bash[36902]: audit 2022-04-22T21:13:08.634670+0000 mon.a (mon.0) 11309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:09 smithi115 bash[45495]: audit 2022-04-22T21:13:08.634670+0000 mon.a (mon.0) 11309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:09 smithi115 bash[46934]: audit 2022-04-22T21:13:08.634670+0000 mon.a (mon.0) 11309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:10.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:10 smithi121 bash[36902]: cluster 2022-04-22T21:13:09.003817+0000 mgr.y (mgr.34107) 7757 : cluster [DBG] pgmap v8753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:10 smithi121 bash[36902]: audit 2022-04-22T21:13:09.415033+0000 mon.a (mon.0) 11310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:10 smithi115 bash[45495]: cluster 2022-04-22T21:13:09.003817+0000 mgr.y (mgr.34107) 7757 : cluster [DBG] pgmap v8753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:10.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:10 smithi115 bash[45495]: audit 2022-04-22T21:13:09.415033+0000 mon.a (mon.0) 11310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:10 smithi115 bash[46934]: cluster 2022-04-22T21:13:09.003817+0000 mgr.y (mgr.34107) 7757 : cluster [DBG] pgmap v8753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:10 smithi115 bash[46934]: audit 2022-04-22T21:13:09.415033+0000 mon.a (mon.0) 11310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:11.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:11 smithi121 bash[36902]: audit 2022-04-22T21:13:10.646657+0000 mon.a (mon.0) 11311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:11 smithi115 bash[45495]: audit 2022-04-22T21:13:10.646657+0000 mon.a (mon.0) 11311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:11 smithi115 bash[46934]: audit 2022-04-22T21:13:10.646657+0000 mon.a (mon.0) 11311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:12.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:12 smithi115 bash[45495]: cluster 2022-04-22T21:13:11.004822+0000 mgr.y (mgr.34107) 7758 : cluster [DBG] pgmap v8754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:12.507 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:12.497Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:12.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:12 smithi115 bash[46934]: cluster 2022-04-22T21:13:11.004822+0000 mgr.y (mgr.34107) 7758 : cluster [DBG] pgmap v8754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:12.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:12 smithi121 bash[36902]: cluster 2022-04-22T21:13:11.004822+0000 mgr.y (mgr.34107) 7758 : cluster [DBG] pgmap v8754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:14 smithi121 bash[36902]: cluster 2022-04-22T21:13:13.005761+0000 mgr.y (mgr.34107) 7759 : cluster [DBG] pgmap v8755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:14 smithi121 bash[36902]: audit 2022-04-22T21:13:13.637402+0000 mon.a (mon.0) 11312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:14.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:14.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:14 smithi115 bash[45495]: cluster 2022-04-22T21:13:13.005761+0000 mgr.y (mgr.34107) 7759 : cluster [DBG] pgmap v8755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:14 smithi115 bash[45495]: audit 2022-04-22T21:13:13.637402+0000 mon.a (mon.0) 11312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:14 smithi115 bash[46934]: cluster 2022-04-22T21:13:13.005761+0000 mgr.y (mgr.34107) 7759 : cluster [DBG] pgmap v8755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:14 smithi115 bash[46934]: audit 2022-04-22T21:13:13.637402+0000 mon.a (mon.0) 11312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:15 smithi121 bash[36902]: audit 2022-04-22T21:13:14.418890+0000 mon.a (mon.0) 11313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:15 smithi115 bash[45495]: audit 2022-04-22T21:13:14.418890+0000 mon.a (mon.0) 11313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:15 smithi115 bash[46934]: audit 2022-04-22T21:13:14.418890+0000 mon.a (mon.0) 11313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:16 smithi121 bash[36902]: cluster 2022-04-22T21:13:15.006729+0000 mgr.y (mgr.34107) 7760 : cluster [DBG] pgmap v8756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:16 smithi121 bash[36902]: audit 2022-04-22T21:13:15.649699+0000 mon.a (mon.0) 11314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:16 smithi121 bash[36902]: audit 2022-04-22T21:13:15.893267+0000 mon.a (mon.0) 11315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:13:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:16 smithi121 bash[36902]: audit 2022-04-22T21:13:15.894008+0000 mon.a (mon.0) 11316 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:13:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[45495]: cluster 2022-04-22T21:13:15.006729+0000 mgr.y (mgr.34107) 7760 : cluster [DBG] pgmap v8756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[45495]: audit 2022-04-22T21:13:15.649699+0000 mon.a (mon.0) 11314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[45495]: audit 2022-04-22T21:13:15.893267+0000 mon.a (mon.0) 11315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[45495]: audit 2022-04-22T21:13:15.894008+0000 mon.a (mon.0) 11316 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[46934]: cluster 2022-04-22T21:13:15.006729+0000 mgr.y (mgr.34107) 7760 : cluster [DBG] pgmap v8756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[46934]: audit 2022-04-22T21:13:15.649699+0000 mon.a (mon.0) 11314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[46934]: audit 2022-04-22T21:13:15.893267+0000 mon.a (mon.0) 11315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:13:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:16 smithi115 bash[46934]: audit 2022-04-22T21:13:15.894008+0000 mon.a (mon.0) 11316 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:13:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:18 smithi121 bash[36902]: cluster 2022-04-22T21:13:17.007869+0000 mgr.y (mgr.34107) 7761 : cluster [DBG] pgmap v8757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:18 smithi115 bash[45495]: cluster 2022-04-22T21:13:17.007869+0000 mgr.y (mgr.34107) 7761 : cluster [DBG] pgmap v8757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:18 smithi115 bash[46934]: cluster 2022-04-22T21:13:17.007869+0000 mgr.y (mgr.34107) 7761 : cluster [DBG] pgmap v8757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:19 smithi121 bash[36902]: audit 2022-04-22T21:13:18.640668+0000 mon.a (mon.0) 11317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:19 smithi115 bash[45495]: audit 2022-04-22T21:13:18.640668+0000 mon.a (mon.0) 11317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:19 smithi115 bash[46934]: audit 2022-04-22T21:13:18.640668+0000 mon.a (mon.0) 11317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:20 smithi115 bash[45495]: cluster 2022-04-22T21:13:19.008989+0000 mgr.y (mgr.34107) 7762 : cluster [DBG] pgmap v8758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:20 smithi115 bash[45495]: audit 2022-04-22T21:13:19.422602+0000 mon.a (mon.0) 11318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:20 smithi115 bash[46934]: cluster 2022-04-22T21:13:19.008989+0000 mgr.y (mgr.34107) 7762 : cluster [DBG] pgmap v8758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:20 smithi115 bash[46934]: audit 2022-04-22T21:13:19.422602+0000 mon.a (mon.0) 11318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:20 smithi121 bash[36902]: cluster 2022-04-22T21:13:19.008989+0000 mgr.y (mgr.34107) 7762 : cluster [DBG] pgmap v8758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:20 smithi121 bash[36902]: audit 2022-04-22T21:13:19.422602+0000 mon.a (mon.0) 11318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:21.070 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:13:21.420 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:13:21.420 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 22s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 22s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 22s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 22s ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:13:21.421 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 22s ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 22s ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 22s ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 22s ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 22s ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:13:21.422 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 22s ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:13:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:21 smithi115 bash[45495]: audit 2022-04-22T21:13:20.652780+0000 mon.a (mon.0) 11319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:21 smithi115 bash[46934]: audit 2022-04-22T21:13:20.652780+0000 mon.a (mon.0) 11319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:21 smithi121 bash[36902]: audit 2022-04-22T21:13:20.652780+0000 mon.a (mon.0) 11319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:21.827 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:13:21.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:13:21.829 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:13:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[46934]: cluster 2022-04-22T21:13:21.009987+0000 mgr.y (mgr.34107) 7763 : cluster [DBG] pgmap v8759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[46934]: audit 2022-04-22T21:13:21.049300+0000 mgr.y (mgr.34107) 7764 : audit [DBG] from='client.43434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[46934]: audit 2022-04-22T21:13:21.394495+0000 mgr.y (mgr.34107) 7765 : audit [DBG] from='client.53371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[46934]: audit 2022-04-22T21:13:21.823157+0000 mon.a (mon.0) 11320 : audit [DBG] from='client.? 172.21.15.115:0/3575793890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:22.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:22.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:22.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[45495]: cluster 2022-04-22T21:13:21.009987+0000 mgr.y (mgr.34107) 7763 : cluster [DBG] pgmap v8759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:22.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[45495]: audit 2022-04-22T21:13:21.049300+0000 mgr.y (mgr.34107) 7764 : audit [DBG] from='client.43434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[45495]: audit 2022-04-22T21:13:21.394495+0000 mgr.y (mgr.34107) 7765 : audit [DBG] from='client.53371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:22 smithi115 bash[45495]: audit 2022-04-22T21:13:21.823157+0000 mon.a (mon.0) 11320 : audit [DBG] from='client.? 172.21.15.115:0/3575793890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:22 smithi121 bash[36902]: cluster 2022-04-22T21:13:21.009987+0000 mgr.y (mgr.34107) 7763 : cluster [DBG] pgmap v8759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:22 smithi121 bash[36902]: audit 2022-04-22T21:13:21.049300+0000 mgr.y (mgr.34107) 7764 : audit [DBG] from='client.43434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:22 smithi121 bash[36902]: audit 2022-04-22T21:13:21.394495+0000 mgr.y (mgr.34107) 7765 : audit [DBG] from='client.53371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:22 smithi121 bash[36902]: audit 2022-04-22T21:13:21.823157+0000 mon.a (mon.0) 11320 : audit [DBG] from='client.? 172.21.15.115:0/3575793890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:24.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:24.373Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:24 smithi115 bash[45495]: cluster 2022-04-22T21:13:23.011006+0000 mgr.y (mgr.34107) 7766 : cluster [DBG] pgmap v8760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:24 smithi115 bash[45495]: audit 2022-04-22T21:13:23.643842+0000 mon.a (mon.0) 11321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:24 smithi115 bash[46934]: cluster 2022-04-22T21:13:23.011006+0000 mgr.y (mgr.34107) 7766 : cluster [DBG] pgmap v8760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:24 smithi115 bash[46934]: audit 2022-04-22T21:13:23.643842+0000 mon.a (mon.0) 11321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:24.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:24 smithi121 bash[36902]: cluster 2022-04-22T21:13:23.011006+0000 mgr.y (mgr.34107) 7766 : cluster [DBG] pgmap v8760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:24 smithi121 bash[36902]: audit 2022-04-22T21:13:23.643842+0000 mon.a (mon.0) 11321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:25 smithi115 bash[45495]: audit 2022-04-22T21:13:24.426225+0000 mon.a (mon.0) 11322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:25.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:25 smithi115 bash[46934]: audit 2022-04-22T21:13:24.426225+0000 mon.a (mon.0) 11322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:25.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:25 smithi121 bash[36902]: audit 2022-04-22T21:13:24.426225+0000 mon.a (mon.0) 11322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:26 smithi115 bash[45495]: cluster 2022-04-22T21:13:25.012127+0000 mgr.y (mgr.34107) 7767 : cluster [DBG] pgmap v8761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:26 smithi115 bash[45495]: audit 2022-04-22T21:13:25.655843+0000 mon.a (mon.0) 11323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:26 smithi115 bash[46934]: cluster 2022-04-22T21:13:25.012127+0000 mgr.y (mgr.34107) 7767 : cluster [DBG] pgmap v8761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:26 smithi115 bash[46934]: audit 2022-04-22T21:13:25.655843+0000 mon.a (mon.0) 11323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:26 smithi121 bash[36902]: cluster 2022-04-22T21:13:25.012127+0000 mgr.y (mgr.34107) 7767 : cluster [DBG] pgmap v8761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:26 smithi121 bash[36902]: audit 2022-04-22T21:13:25.655843+0000 mon.a (mon.0) 11323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:28 smithi115 bash[45495]: cluster 2022-04-22T21:13:27.013216+0000 mgr.y (mgr.34107) 7768 : cluster [DBG] pgmap v8762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:28 smithi115 bash[46934]: cluster 2022-04-22T21:13:27.013216+0000 mgr.y (mgr.34107) 7768 : cluster [DBG] pgmap v8762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:28 smithi121 bash[36902]: cluster 2022-04-22T21:13:27.013216+0000 mgr.y (mgr.34107) 7768 : cluster [DBG] pgmap v8762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:29.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:29 smithi121 bash[36902]: audit 2022-04-22T21:13:28.646933+0000 mon.a (mon.0) 11324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:29 smithi115 bash[45495]: audit 2022-04-22T21:13:28.646933+0000 mon.a (mon.0) 11324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:29 smithi115 bash[46934]: audit 2022-04-22T21:13:28.646933+0000 mon.a (mon.0) 11324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:30.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:30 smithi121 bash[36902]: cluster 2022-04-22T21:13:29.014411+0000 mgr.y (mgr.34107) 7769 : cluster [DBG] pgmap v8763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:30 smithi121 bash[36902]: audit 2022-04-22T21:13:29.429774+0000 mon.a (mon.0) 11325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:30 smithi115 bash[45495]: cluster 2022-04-22T21:13:29.014411+0000 mgr.y (mgr.34107) 7769 : cluster [DBG] pgmap v8763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:30 smithi115 bash[45495]: audit 2022-04-22T21:13:29.429774+0000 mon.a (mon.0) 11325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:30 smithi115 bash[46934]: cluster 2022-04-22T21:13:29.014411+0000 mgr.y (mgr.34107) 7769 : cluster [DBG] pgmap v8763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:30 smithi115 bash[46934]: audit 2022-04-22T21:13:29.429774+0000 mon.a (mon.0) 11325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:31 smithi121 bash[36902]: audit 2022-04-22T21:13:30.658743+0000 mon.a (mon.0) 11326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:31 smithi115 bash[45495]: audit 2022-04-22T21:13:30.658743+0000 mon.a (mon.0) 11326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:31 smithi115 bash[46934]: audit 2022-04-22T21:13:30.658743+0000 mon.a (mon.0) 11326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:32 smithi121 bash[36902]: cluster 2022-04-22T21:13:31.015649+0000 mgr.y (mgr.34107) 7770 : cluster [DBG] pgmap v8764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:32 smithi115 bash[45495]: cluster 2022-04-22T21:13:31.015649+0000 mgr.y (mgr.34107) 7770 : cluster [DBG] pgmap v8764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:32.825 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:32.500Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:32.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:32 smithi115 bash[46934]: cluster 2022-04-22T21:13:31.015649+0000 mgr.y (mgr.34107) 7770 : cluster [DBG] pgmap v8764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:34.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:34 smithi121 bash[36902]: cluster 2022-04-22T21:13:33.016696+0000 mgr.y (mgr.34107) 7771 : cluster [DBG] pgmap v8765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:34 smithi121 bash[36902]: audit 2022-04-22T21:13:33.649934+0000 mon.a (mon.0) 11327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:34.376Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:34 smithi115 bash[45495]: cluster 2022-04-22T21:13:33.016696+0000 mgr.y (mgr.34107) 7771 : cluster [DBG] pgmap v8765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:34 smithi115 bash[45495]: audit 2022-04-22T21:13:33.649934+0000 mon.a (mon.0) 11327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:34 smithi115 bash[46934]: cluster 2022-04-22T21:13:33.016696+0000 mgr.y (mgr.34107) 7771 : cluster [DBG] pgmap v8765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:34 smithi115 bash[46934]: audit 2022-04-22T21:13:33.649934+0000 mon.a (mon.0) 11327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:35.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:35 smithi121 bash[36902]: audit 2022-04-22T21:13:34.433620+0000 mon.a (mon.0) 11328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:35 smithi115 bash[45495]: audit 2022-04-22T21:13:34.433620+0000 mon.a (mon.0) 11328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:35 smithi115 bash[46934]: audit 2022-04-22T21:13:34.433620+0000 mon.a (mon.0) 11328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:36 smithi121 bash[36902]: cluster 2022-04-22T21:13:35.017762+0000 mgr.y (mgr.34107) 7772 : cluster [DBG] pgmap v8766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:36 smithi121 bash[36902]: audit 2022-04-22T21:13:35.661866+0000 mon.a (mon.0) 11329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:36 smithi115 bash[45495]: cluster 2022-04-22T21:13:35.017762+0000 mgr.y (mgr.34107) 7772 : cluster [DBG] pgmap v8766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:36.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:36 smithi115 bash[45495]: audit 2022-04-22T21:13:35.661866+0000 mon.a (mon.0) 11329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:36 smithi115 bash[46934]: cluster 2022-04-22T21:13:35.017762+0000 mgr.y (mgr.34107) 7772 : cluster [DBG] pgmap v8766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:36 smithi115 bash[46934]: audit 2022-04-22T21:13:35.661866+0000 mon.a (mon.0) 11329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:38.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:38 smithi121 bash[36902]: cluster 2022-04-22T21:13:37.018790+0000 mgr.y (mgr.34107) 7773 : cluster [DBG] pgmap v8767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:38 smithi115 bash[45495]: cluster 2022-04-22T21:13:37.018790+0000 mgr.y (mgr.34107) 7773 : cluster [DBG] pgmap v8767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:38 smithi115 bash[46934]: cluster 2022-04-22T21:13:37.018790+0000 mgr.y (mgr.34107) 7773 : cluster [DBG] pgmap v8767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:39 smithi121 bash[36902]: audit 2022-04-22T21:13:38.653150+0000 mon.a (mon.0) 11330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:39 smithi115 bash[46934]: audit 2022-04-22T21:13:38.653150+0000 mon.a (mon.0) 11330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:39 smithi115 bash[45495]: audit 2022-04-22T21:13:38.653150+0000 mon.a (mon.0) 11330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:40 smithi121 bash[36902]: cluster 2022-04-22T21:13:39.020022+0000 mgr.y (mgr.34107) 7774 : cluster [DBG] pgmap v8768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:40 smithi121 bash[36902]: audit 2022-04-22T21:13:39.437196+0000 mon.a (mon.0) 11331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:40 smithi115 bash[45495]: cluster 2022-04-22T21:13:39.020022+0000 mgr.y (mgr.34107) 7774 : cluster [DBG] pgmap v8768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:40 smithi115 bash[45495]: audit 2022-04-22T21:13:39.437196+0000 mon.a (mon.0) 11331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:40 smithi115 bash[46934]: cluster 2022-04-22T21:13:39.020022+0000 mgr.y (mgr.34107) 7774 : cluster [DBG] pgmap v8768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:40 smithi115 bash[46934]: audit 2022-04-22T21:13:39.437196+0000 mon.a (mon.0) 11331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:41 smithi121 bash[36902]: audit 2022-04-22T21:13:40.664846+0000 mon.a (mon.0) 11332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:41 smithi115 bash[45495]: audit 2022-04-22T21:13:40.664846+0000 mon.a (mon.0) 11332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:41.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:41 smithi115 bash[46934]: audit 2022-04-22T21:13:40.664846+0000 mon.a (mon.0) 11332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:42 smithi121 bash[36902]: cluster 2022-04-22T21:13:41.021169+0000 mgr.y (mgr.34107) 7775 : cluster [DBG] pgmap v8769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:42 smithi115 bash[46934]: cluster 2022-04-22T21:13:41.021169+0000 mgr.y (mgr.34107) 7775 : cluster [DBG] pgmap v8769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:42.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:42.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:42 smithi115 bash[45495]: cluster 2022-04-22T21:13:41.021169+0000 mgr.y (mgr.34107) 7775 : cluster [DBG] pgmap v8769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:44 smithi121 bash[36902]: cluster 2022-04-22T21:13:43.022281+0000 mgr.y (mgr.34107) 7776 : cluster [DBG] pgmap v8770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:44 smithi121 bash[36902]: audit 2022-04-22T21:13:43.656346+0000 mon.a (mon.0) 11333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:44.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:44 smithi115 bash[45495]: cluster 2022-04-22T21:13:43.022281+0000 mgr.y (mgr.34107) 7776 : cluster [DBG] pgmap v8770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:44 smithi115 bash[45495]: audit 2022-04-22T21:13:43.656346+0000 mon.a (mon.0) 11333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:44 smithi115 bash[46934]: cluster 2022-04-22T21:13:43.022281+0000 mgr.y (mgr.34107) 7776 : cluster [DBG] pgmap v8770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:44 smithi115 bash[46934]: audit 2022-04-22T21:13:43.656346+0000 mon.a (mon.0) 11333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:45.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:45 smithi121 bash[36902]: audit 2022-04-22T21:13:44.441121+0000 mon.a (mon.0) 11334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:45 smithi115 bash[45495]: audit 2022-04-22T21:13:44.441121+0000 mon.a (mon.0) 11334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:45.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:45 smithi115 bash[46934]: audit 2022-04-22T21:13:44.441121+0000 mon.a (mon.0) 11334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:46 smithi121 bash[36902]: cluster 2022-04-22T21:13:45.023426+0000 mgr.y (mgr.34107) 7777 : cluster [DBG] pgmap v8771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:46 smithi121 bash[36902]: audit 2022-04-22T21:13:45.667871+0000 mon.a (mon.0) 11335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:46 smithi115 bash[45495]: cluster 2022-04-22T21:13:45.023426+0000 mgr.y (mgr.34107) 7777 : cluster [DBG] pgmap v8771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:46 smithi115 bash[45495]: audit 2022-04-22T21:13:45.667871+0000 mon.a (mon.0) 11335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:46 smithi115 bash[46934]: cluster 2022-04-22T21:13:45.023426+0000 mgr.y (mgr.34107) 7777 : cluster [DBG] pgmap v8771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:46 smithi115 bash[46934]: audit 2022-04-22T21:13:45.667871+0000 mon.a (mon.0) 11335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:48 smithi121 bash[36902]: cluster 2022-04-22T21:13:47.024474+0000 mgr.y (mgr.34107) 7778 : cluster [DBG] pgmap v8772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:48 smithi115 bash[45495]: cluster 2022-04-22T21:13:47.024474+0000 mgr.y (mgr.34107) 7778 : cluster [DBG] pgmap v8772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:48 smithi115 bash[46934]: cluster 2022-04-22T21:13:47.024474+0000 mgr.y (mgr.34107) 7778 : cluster [DBG] pgmap v8772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:49 smithi121 bash[36902]: audit 2022-04-22T21:13:48.659078+0000 mon.a (mon.0) 11336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:49 smithi115 bash[45495]: audit 2022-04-22T21:13:48.659078+0000 mon.a (mon.0) 11336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:49 smithi115 bash[46934]: audit 2022-04-22T21:13:48.659078+0000 mon.a (mon.0) 11336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:50 smithi121 bash[36902]: cluster 2022-04-22T21:13:49.025595+0000 mgr.y (mgr.34107) 7779 : cluster [DBG] pgmap v8773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:50 smithi121 bash[36902]: audit 2022-04-22T21:13:49.444821+0000 mon.a (mon.0) 11337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:50 smithi115 bash[46934]: cluster 2022-04-22T21:13:49.025595+0000 mgr.y (mgr.34107) 7779 : cluster [DBG] pgmap v8773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:50 smithi115 bash[46934]: audit 2022-04-22T21:13:49.444821+0000 mon.a (mon.0) 11337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:50 smithi115 bash[45495]: cluster 2022-04-22T21:13:49.025595+0000 mgr.y (mgr.34107) 7779 : cluster [DBG] pgmap v8773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:50 smithi115 bash[45495]: audit 2022-04-22T21:13:49.444821+0000 mon.a (mon.0) 11337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:51 smithi121 bash[36902]: audit 2022-04-22T21:13:50.670869+0000 mon.a (mon.0) 11338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:51 smithi115 bash[45495]: audit 2022-04-22T21:13:50.670869+0000 mon.a (mon.0) 11338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:51 smithi115 bash[46934]: audit 2022-04-22T21:13:50.670869+0000 mon.a (mon.0) 11338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:52.199 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:13:52.547 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:13:52.547 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 10m ago 3h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 53s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 10m ago 3h 23.6M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 53s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 53s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 10m ago 3h 488M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 10m ago 3h 364M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 53s ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 10m ago 3h 238M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 10m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:13:52.548 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 53s ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 10m ago 3h 710M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 10m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 10m ago 3h 534M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 10m ago 3h 493M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 53s ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 53s ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 53s ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 53s ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:13:52.549 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 53s ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:13:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:52 smithi121 bash[36902]: cluster 2022-04-22T21:13:51.026660+0000 mgr.y (mgr.34107) 7780 : cluster [DBG] pgmap v8774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:52.501Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:52 smithi115 bash[45495]: cluster 2022-04-22T21:13:51.026660+0000 mgr.y (mgr.34107) 7780 : cluster [DBG] pgmap v8774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:52 smithi115 bash[46934]: cluster 2022-04-22T21:13:51.026660+0000 mgr.y (mgr.34107) 7780 : cluster [DBG] pgmap v8774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:13:52.950 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:13:52.950 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:13:52.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:13:52.951 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:13:52.952 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:13:52.952 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:13:52.952 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:13:53.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:53 smithi121 bash[36902]: audit 2022-04-22T21:13:52.180429+0000 mgr.y (mgr.34107) 7781 : audit [DBG] from='client.53383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:53 smithi121 bash[36902]: audit 2022-04-22T21:13:52.522599+0000 mgr.y (mgr.34107) 7782 : audit [DBG] from='client.53389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:53 smithi121 bash[36902]: audit 2022-04-22T21:13:52.945946+0000 mon.c (mon.1) 282 : audit [DBG] from='client.? 172.21.15.115:0/3697101105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[45495]: audit 2022-04-22T21:13:52.180429+0000 mgr.y (mgr.34107) 7781 : audit [DBG] from='client.53383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[45495]: audit 2022-04-22T21:13:52.522599+0000 mgr.y (mgr.34107) 7782 : audit [DBG] from='client.53389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[45495]: audit 2022-04-22T21:13:52.945946+0000 mon.c (mon.1) 282 : audit [DBG] from='client.? 172.21.15.115:0/3697101105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[46934]: audit 2022-04-22T21:13:52.180429+0000 mgr.y (mgr.34107) 7781 : audit [DBG] from='client.53383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[46934]: audit 2022-04-22T21:13:52.522599+0000 mgr.y (mgr.34107) 7782 : audit [DBG] from='client.53389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:13:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:53 smithi115 bash[46934]: audit 2022-04-22T21:13:52.945946+0000 mon.c (mon.1) 282 : audit [DBG] from='client.? 172.21.15.115:0/3697101105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:13:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:54 smithi121 bash[36902]: cluster 2022-04-22T21:13:53.027402+0000 mgr.y (mgr.34107) 7783 : cluster [DBG] pgmap v8775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:54 smithi121 bash[36902]: audit 2022-04-22T21:13:53.662156+0000 mon.a (mon.0) 11339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:13:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:13:54.374Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:13:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:54 smithi115 bash[45495]: cluster 2022-04-22T21:13:53.027402+0000 mgr.y (mgr.34107) 7783 : cluster [DBG] pgmap v8775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:54 smithi115 bash[45495]: audit 2022-04-22T21:13:53.662156+0000 mon.a (mon.0) 11339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:54 smithi115 bash[46934]: cluster 2022-04-22T21:13:53.027402+0000 mgr.y (mgr.34107) 7783 : cluster [DBG] pgmap v8775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:54 smithi115 bash[46934]: audit 2022-04-22T21:13:53.662156+0000 mon.a (mon.0) 11339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:55 smithi121 bash[36902]: audit 2022-04-22T21:13:54.448493+0000 mon.a (mon.0) 11340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:55 smithi115 bash[45495]: audit 2022-04-22T21:13:54.448493+0000 mon.a (mon.0) 11340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:55 smithi115 bash[46934]: audit 2022-04-22T21:13:54.448493+0000 mon.a (mon.0) 11340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:13:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:56 smithi121 bash[36902]: cluster 2022-04-22T21:13:55.028543+0000 mgr.y (mgr.34107) 7784 : cluster [DBG] pgmap v8776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:56 smithi121 bash[36902]: audit 2022-04-22T21:13:55.673932+0000 mon.a (mon.0) 11341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:56 smithi115 bash[45495]: cluster 2022-04-22T21:13:55.028543+0000 mgr.y (mgr.34107) 7784 : cluster [DBG] pgmap v8776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:56 smithi115 bash[45495]: audit 2022-04-22T21:13:55.673932+0000 mon.a (mon.0) 11341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:56 smithi115 bash[46934]: cluster 2022-04-22T21:13:55.028543+0000 mgr.y (mgr.34107) 7784 : cluster [DBG] pgmap v8776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:56 smithi115 bash[46934]: audit 2022-04-22T21:13:55.673932+0000 mon.a (mon.0) 11341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:13:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:58 smithi121 bash[36902]: cluster 2022-04-22T21:13:57.029663+0000 mgr.y (mgr.34107) 7785 : cluster [DBG] pgmap v8777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:58 smithi115 bash[45495]: cluster 2022-04-22T21:13:57.029663+0000 mgr.y (mgr.34107) 7785 : cluster [DBG] pgmap v8777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:58.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:58 smithi115 bash[46934]: cluster 2022-04-22T21:13:57.029663+0000 mgr.y (mgr.34107) 7785 : cluster [DBG] pgmap v8777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:13:59.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[45495]: audit 2022-04-22T21:13:58.665352+0000 mon.a (mon.0) 11342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:59.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[45495]: audit 2022-04-22T21:13:59.192731+0000 mon.a (mon.0) 11343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:13:59.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[45495]: audit 2022-04-22T21:13:59.195877+0000 mon.a (mon.0) 11344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:13:59.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[45495]: audit 2022-04-22T21:13:59.197798+0000 mon.a (mon.0) 11345 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:13:59.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[46934]: audit 2022-04-22T21:13:58.665352+0000 mon.a (mon.0) 11342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:59.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[46934]: audit 2022-04-22T21:13:59.192731+0000 mon.a (mon.0) 11343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:13:59.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[46934]: audit 2022-04-22T21:13:59.195877+0000 mon.a (mon.0) 11344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:13:59.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:13:59 smithi115 bash[46934]: audit 2022-04-22T21:13:59.197798+0000 mon.a (mon.0) 11345 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:13:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:59 smithi121 bash[36902]: audit 2022-04-22T21:13:58.665352+0000 mon.a (mon.0) 11342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:13:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:59 smithi121 bash[36902]: audit 2022-04-22T21:13:59.192731+0000 mon.a (mon.0) 11343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:13:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:59 smithi121 bash[36902]: audit 2022-04-22T21:13:59.195877+0000 mon.a (mon.0) 11344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:13:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:13:59 smithi121 bash[36902]: audit 2022-04-22T21:13:59.197798+0000 mon.a (mon.0) 11345 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:14:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:00 smithi121 bash[36902]: cluster 2022-04-22T21:13:59.030919+0000 mgr.y (mgr.34107) 7786 : cluster [DBG] pgmap v8778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:00 smithi121 bash[36902]: audit 2022-04-22T21:13:59.451941+0000 mon.a (mon.0) 11346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:00 smithi121 bash[36902]: audit 2022-04-22T21:13:59.514644+0000 mon.a (mon.0) 11347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:00 smithi121 bash[36902]: audit 2022-04-22T21:13:59.534593+0000 mon.a (mon.0) 11348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[45495]: cluster 2022-04-22T21:13:59.030919+0000 mgr.y (mgr.34107) 7786 : cluster [DBG] pgmap v8778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[45495]: audit 2022-04-22T21:13:59.451941+0000 mon.a (mon.0) 11346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[45495]: audit 2022-04-22T21:13:59.514644+0000 mon.a (mon.0) 11347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[45495]: audit 2022-04-22T21:13:59.534593+0000 mon.a (mon.0) 11348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[46934]: cluster 2022-04-22T21:13:59.030919+0000 mgr.y (mgr.34107) 7786 : cluster [DBG] pgmap v8778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[46934]: audit 2022-04-22T21:13:59.451941+0000 mon.a (mon.0) 11346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[46934]: audit 2022-04-22T21:13:59.514644+0000 mon.a (mon.0) 11347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:00.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:00 smithi115 bash[46934]: audit 2022-04-22T21:13:59.534593+0000 mon.a (mon.0) 11348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:01.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:01 smithi121 bash[36902]: audit 2022-04-22T21:14:00.676911+0000 mon.a (mon.0) 11349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:01 smithi115 bash[45495]: audit 2022-04-22T21:14:00.676911+0000 mon.a (mon.0) 11349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:01 smithi115 bash[46934]: audit 2022-04-22T21:14:00.676911+0000 mon.a (mon.0) 11349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:02 smithi121 bash[36902]: cluster 2022-04-22T21:14:01.031990+0000 mgr.y (mgr.34107) 7787 : cluster [DBG] pgmap v8779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:02 smithi115 bash[46934]: cluster 2022-04-22T21:14:01.031990+0000 mgr.y (mgr.34107) 7787 : cluster [DBG] pgmap v8779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:02.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:02 smithi115 bash[45495]: cluster 2022-04-22T21:14:01.031990+0000 mgr.y (mgr.34107) 7787 : cluster [DBG] pgmap v8779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:04.673 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:04.376Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:04 smithi121 bash[36902]: cluster 2022-04-22T21:14:03.032736+0000 mgr.y (mgr.34107) 7788 : cluster [DBG] pgmap v8780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:04 smithi121 bash[36902]: audit 2022-04-22T21:14:03.671857+0000 mon.a (mon.0) 11350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:04 smithi121 bash[36902]: audit 2022-04-22T21:14:03.672418+0000 mon.a (mon.0) 11351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:04 smithi121 bash[36902]: audit 2022-04-22T21:14:04.455104+0000 mon.a (mon.0) 11352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[45495]: cluster 2022-04-22T21:14:03.032736+0000 mgr.y (mgr.34107) 7788 : cluster [DBG] pgmap v8780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[45495]: audit 2022-04-22T21:14:03.671857+0000 mon.a (mon.0) 11350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[45495]: audit 2022-04-22T21:14:03.672418+0000 mon.a (mon.0) 11351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[45495]: audit 2022-04-22T21:14:04.455104+0000 mon.a (mon.0) 11352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[46934]: cluster 2022-04-22T21:14:03.032736+0000 mgr.y (mgr.34107) 7788 : cluster [DBG] pgmap v8780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[46934]: audit 2022-04-22T21:14:03.671857+0000 mon.a (mon.0) 11350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[46934]: audit 2022-04-22T21:14:03.672418+0000 mon.a (mon.0) 11351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:04 smithi115 bash[46934]: audit 2022-04-22T21:14:04.455104+0000 mon.a (mon.0) 11352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:06 smithi115 bash[45495]: cluster 2022-04-22T21:14:05.033834+0000 mgr.y (mgr.34107) 7789 : cluster [DBG] pgmap v8781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:06 smithi115 bash[45495]: audit 2022-04-22T21:14:05.680134+0000 mon.a (mon.0) 11353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:06 smithi115 bash[46934]: cluster 2022-04-22T21:14:05.033834+0000 mgr.y (mgr.34107) 7789 : cluster [DBG] pgmap v8781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:06 smithi115 bash[46934]: audit 2022-04-22T21:14:05.680134+0000 mon.a (mon.0) 11353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:06.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:06 smithi121 bash[36902]: cluster 2022-04-22T21:14:05.033834+0000 mgr.y (mgr.34107) 7789 : cluster [DBG] pgmap v8781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:06 smithi121 bash[36902]: audit 2022-04-22T21:14:05.680134+0000 mon.a (mon.0) 11353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:08.400 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:08 smithi115 bash[45495]: cluster 2022-04-22T21:14:07.034931+0000 mgr.y (mgr.34107) 7790 : cluster [DBG] pgmap v8782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:08.401 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:08 smithi115 bash[46934]: cluster 2022-04-22T21:14:07.034931+0000 mgr.y (mgr.34107) 7790 : cluster [DBG] pgmap v8782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:08 smithi121 bash[36902]: cluster 2022-04-22T21:14:07.034931+0000 mgr.y (mgr.34107) 7790 : cluster [DBG] pgmap v8782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:09 smithi115 bash[46934]: audit 2022-04-22T21:14:08.671785+0000 mon.a (mon.0) 11354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:09.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:09 smithi115 bash[45495]: audit 2022-04-22T21:14:08.671785+0000 mon.a (mon.0) 11354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:09.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:09 smithi121 bash[36902]: audit 2022-04-22T21:14:08.671785+0000 mon.a (mon.0) 11354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:10.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:10 smithi121 bash[36902]: cluster 2022-04-22T21:14:09.036102+0000 mgr.y (mgr.34107) 7791 : cluster [DBG] pgmap v8783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:10 smithi121 bash[36902]: audit 2022-04-22T21:14:09.458945+0000 mon.a (mon.0) 11355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:10.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:10 smithi115 bash[46934]: cluster 2022-04-22T21:14:09.036102+0000 mgr.y (mgr.34107) 7791 : cluster [DBG] pgmap v8783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:10.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:10 smithi115 bash[46934]: audit 2022-04-22T21:14:09.458945+0000 mon.a (mon.0) 11355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:10.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:10 smithi115 bash[45495]: cluster 2022-04-22T21:14:09.036102+0000 mgr.y (mgr.34107) 7791 : cluster [DBG] pgmap v8783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:10.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:10 smithi115 bash[45495]: audit 2022-04-22T21:14:09.458945+0000 mon.a (mon.0) 11355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:11 smithi121 bash[36902]: audit 2022-04-22T21:14:10.683012+0000 mon.a (mon.0) 11356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:11 smithi115 bash[45495]: audit 2022-04-22T21:14:10.683012+0000 mon.a (mon.0) 11356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:11 smithi115 bash[46934]: audit 2022-04-22T21:14:10.683012+0000 mon.a (mon.0) 11356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:12.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:12 smithi115 bash[45495]: cluster 2022-04-22T21:14:11.037230+0000 mgr.y (mgr.34107) 7792 : cluster [DBG] pgmap v8784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:12.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:12 smithi115 bash[46934]: cluster 2022-04-22T21:14:11.037230+0000 mgr.y (mgr.34107) 7792 : cluster [DBG] pgmap v8784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:12.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:12 smithi121 bash[36902]: cluster 2022-04-22T21:14:11.037230+0000 mgr.y (mgr.34107) 7792 : cluster [DBG] pgmap v8784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:12.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:14.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:13 smithi121 bash[36902]: cephadm 2022-04-22T21:14:12.900583+0000 mgr.y (mgr.34107) 7793 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:14:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:13 smithi121 bash[36902]: audit 2022-04-22T21:14:12.912113+0000 mon.a (mon.0) 11357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:13 smithi121 bash[36902]: audit 2022-04-22T21:14:12.918986+0000 mon.a (mon.0) 11358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:13 smithi121 bash[36902]: cluster 2022-04-22T21:14:13.038215+0000 mgr.y (mgr.34107) 7794 : cluster [DBG] pgmap v8785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:13 smithi121 bash[36902]: audit 2022-04-22T21:14:13.674311+0000 mon.a (mon.0) 11359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[45495]: cephadm 2022-04-22T21:14:12.900583+0000 mgr.y (mgr.34107) 7793 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:14:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[45495]: audit 2022-04-22T21:14:12.912113+0000 mon.a (mon.0) 11357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[45495]: audit 2022-04-22T21:14:12.918986+0000 mon.a (mon.0) 11358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[45495]: cluster 2022-04-22T21:14:13.038215+0000 mgr.y (mgr.34107) 7794 : cluster [DBG] pgmap v8785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[45495]: audit 2022-04-22T21:14:13.674311+0000 mon.a (mon.0) 11359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[46934]: cephadm 2022-04-22T21:14:12.900583+0000 mgr.y (mgr.34107) 7793 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[46934]: audit 2022-04-22T21:14:12.912113+0000 mon.a (mon.0) 11357 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[46934]: audit 2022-04-22T21:14:12.918986+0000 mon.a (mon.0) 11358 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:14:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[46934]: cluster 2022-04-22T21:14:13.038215+0000 mgr.y (mgr.34107) 7794 : cluster [DBG] pgmap v8785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:14.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:13 smithi115 bash[46934]: audit 2022-04-22T21:14:13.674311+0000 mon.a (mon.0) 11359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:14.377Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:14 smithi121 bash[36902]: audit 2022-04-22T21:14:14.461919+0000 mon.a (mon.0) 11360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:14 smithi115 bash[45495]: audit 2022-04-22T21:14:14.461919+0000 mon.a (mon.0) 11360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:14 smithi115 bash[46934]: audit 2022-04-22T21:14:14.461919+0000 mon.a (mon.0) 11360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[45495]: cluster 2022-04-22T21:14:15.039314+0000 mgr.y (mgr.34107) 7795 : cluster [DBG] pgmap v8786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:16.392 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[45495]: audit 2022-04-22T21:14:15.685644+0000 mon.a (mon.0) 11361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:16.392 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[45495]: audit 2022-04-22T21:14:15.904770+0000 mon.a (mon.0) 11362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:14:16.392 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[45495]: audit 2022-04-22T21:14:15.905410+0000 mon.a (mon.0) 11363 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:14:16.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[46934]: cluster 2022-04-22T21:14:15.039314+0000 mgr.y (mgr.34107) 7795 : cluster [DBG] pgmap v8786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:16.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[46934]: audit 2022-04-22T21:14:15.685644+0000 mon.a (mon.0) 11361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:16.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[46934]: audit 2022-04-22T21:14:15.904770+0000 mon.a (mon.0) 11362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:14:16.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:16 smithi115 bash[46934]: audit 2022-04-22T21:14:15.905410+0000 mon.a (mon.0) 11363 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:14:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:16 smithi121 bash[36902]: cluster 2022-04-22T21:14:15.039314+0000 mgr.y (mgr.34107) 7795 : cluster [DBG] pgmap v8786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:16 smithi121 bash[36902]: audit 2022-04-22T21:14:15.685644+0000 mon.a (mon.0) 11361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:16 smithi121 bash[36902]: audit 2022-04-22T21:14:15.904770+0000 mon.a (mon.0) 11362 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:14:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:16 smithi121 bash[36902]: audit 2022-04-22T21:14:15.905410+0000 mon.a (mon.0) 11363 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:14:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:18 smithi121 bash[36902]: cluster 2022-04-22T21:14:17.040372+0000 mgr.y (mgr.34107) 7796 : cluster [DBG] pgmap v8787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:18 smithi115 bash[45495]: cluster 2022-04-22T21:14:17.040372+0000 mgr.y (mgr.34107) 7796 : cluster [DBG] pgmap v8787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:18 smithi115 bash[46934]: cluster 2022-04-22T21:14:17.040372+0000 mgr.y (mgr.34107) 7796 : cluster [DBG] pgmap v8787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:19 smithi121 bash[36902]: audit 2022-04-22T21:14:18.677730+0000 mon.a (mon.0) 11364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:19.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:19 smithi115 bash[45495]: audit 2022-04-22T21:14:18.677730+0000 mon.a (mon.0) 11364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:19.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:19 smithi115 bash[46934]: audit 2022-04-22T21:14:18.677730+0000 mon.a (mon.0) 11364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:20 smithi121 bash[36902]: cluster 2022-04-22T21:14:19.041635+0000 mgr.y (mgr.34107) 7797 : cluster [DBG] pgmap v8788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:20 smithi121 bash[36902]: audit 2022-04-22T21:14:19.465499+0000 mon.a (mon.0) 11365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:20 smithi115 bash[45495]: cluster 2022-04-22T21:14:19.041635+0000 mgr.y (mgr.34107) 7797 : cluster [DBG] pgmap v8788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:20 smithi115 bash[45495]: audit 2022-04-22T21:14:19.465499+0000 mon.a (mon.0) 11365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:20 smithi115 bash[46934]: cluster 2022-04-22T21:14:19.041635+0000 mgr.y (mgr.34107) 7797 : cluster [DBG] pgmap v8788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:20 smithi115 bash[46934]: audit 2022-04-22T21:14:19.465499+0000 mon.a (mon.0) 11365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:21 smithi121 bash[36902]: audit 2022-04-22T21:14:20.688423+0000 mon.a (mon.0) 11366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:21 smithi115 bash[45495]: audit 2022-04-22T21:14:20.688423+0000 mon.a (mon.0) 11366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:21 smithi115 bash[46934]: audit 2022-04-22T21:14:20.688423+0000 mon.a (mon.0) 11366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:22.507 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:22 smithi115 bash[45495]: cluster 2022-04-22T21:14:21.042838+0000 mgr.y (mgr.34107) 7798 : cluster [DBG] pgmap v8789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:22.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:22 smithi115 bash[46934]: cluster 2022-04-22T21:14:21.042838+0000 mgr.y (mgr.34107) 7798 : cluster [DBG] pgmap v8789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:22 smithi121 bash[36902]: cluster 2022-04-22T21:14:21.042838+0000 mgr.y (mgr.34107) 7798 : cluster [DBG] pgmap v8789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:22.502Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:23.332 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:14:23.667 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:14:23.667 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 19s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:14:23.667 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 84s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:14:23.667 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 19s ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:14:23.667 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 84s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 84s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 19s ago 3h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 19s ago 3h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 84s ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 19s ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 19s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 84s ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 19s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 19s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 19s ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:14:23.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 19s ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:14:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 84s ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:14:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 84s ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:14:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 84s ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:14:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 84s ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:14:23.669 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 84s ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:14:24.071 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:14:24.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:14:24.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:14:24.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:14:24.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:24.073 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:14:24.073 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:14:24.073 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:14:24.073 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[45495]: cluster 2022-04-22T21:14:23.043506+0000 mgr.y (mgr.34107) 7799 : cluster [DBG] pgmap v8790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[45495]: audit 2022-04-22T21:14:23.312321+0000 mgr.y (mgr.34107) 7800 : audit [DBG] from='client.53401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[45495]: audit 2022-04-22T21:14:23.646958+0000 mgr.y (mgr.34107) 7801 : audit [DBG] from='client.53407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[45495]: audit 2022-04-22T21:14:23.680863+0000 mon.a (mon.0) 11367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[45495]: audit 2022-04-22T21:14:24.067036+0000 mon.a (mon.0) 11368 : audit [DBG] from='client.? 172.21.15.115:0/1735192465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:24.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[46934]: cluster 2022-04-22T21:14:23.043506+0000 mgr.y (mgr.34107) 7799 : cluster [DBG] pgmap v8790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:24.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[46934]: audit 2022-04-22T21:14:23.312321+0000 mgr.y (mgr.34107) 7800 : audit [DBG] from='client.53401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[46934]: audit 2022-04-22T21:14:23.646958+0000 mgr.y (mgr.34107) 7801 : audit [DBG] from='client.53407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[46934]: audit 2022-04-22T21:14:23.680863+0000 mon.a (mon.0) 11367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:24.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[46934]: audit 2022-04-22T21:14:24.067036+0000 mon.a (mon.0) 11368 : audit [DBG] from='client.? 172.21.15.115:0/1735192465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:24 smithi121 bash[36902]: cluster 2022-04-22T21:14:23.043506+0000 mgr.y (mgr.34107) 7799 : cluster [DBG] pgmap v8790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:24 smithi121 bash[36902]: audit 2022-04-22T21:14:23.312321+0000 mgr.y (mgr.34107) 7800 : audit [DBG] from='client.53401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:24 smithi121 bash[36902]: audit 2022-04-22T21:14:23.646958+0000 mgr.y (mgr.34107) 7801 : audit [DBG] from='client.53407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:24 smithi121 bash[36902]: audit 2022-04-22T21:14:23.680863+0000 mon.a (mon.0) 11367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:24 smithi121 bash[36902]: audit 2022-04-22T21:14:24.067036+0000 mon.a (mon.0) 11368 : audit [DBG] from='client.? 172.21.15.115:0/1735192465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:24.377Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:25 smithi121 bash[36902]: audit 2022-04-22T21:14:24.469000+0000 mon.a (mon.0) 11369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:25 smithi115 bash[45495]: audit 2022-04-22T21:14:24.469000+0000 mon.a (mon.0) 11369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:25.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:25 smithi115 bash[46934]: audit 2022-04-22T21:14:24.469000+0000 mon.a (mon.0) 11369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:26.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:26 smithi121 bash[36902]: cluster 2022-04-22T21:14:25.044656+0000 mgr.y (mgr.34107) 7802 : cluster [DBG] pgmap v8791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:26 smithi121 bash[36902]: audit 2022-04-22T21:14:25.691507+0000 mon.a (mon.0) 11370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:26 smithi115 bash[45495]: cluster 2022-04-22T21:14:25.044656+0000 mgr.y (mgr.34107) 7802 : cluster [DBG] pgmap v8791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:26 smithi115 bash[45495]: audit 2022-04-22T21:14:25.691507+0000 mon.a (mon.0) 11370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:26 smithi115 bash[46934]: cluster 2022-04-22T21:14:25.044656+0000 mgr.y (mgr.34107) 7802 : cluster [DBG] pgmap v8791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:26.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:26 smithi115 bash[46934]: audit 2022-04-22T21:14:25.691507+0000 mon.a (mon.0) 11370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:28 smithi121 bash[36902]: cluster 2022-04-22T21:14:27.045917+0000 mgr.y (mgr.34107) 7803 : cluster [DBG] pgmap v8792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:28 smithi115 bash[45495]: cluster 2022-04-22T21:14:27.045917+0000 mgr.y (mgr.34107) 7803 : cluster [DBG] pgmap v8792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:28 smithi115 bash[46934]: cluster 2022-04-22T21:14:27.045917+0000 mgr.y (mgr.34107) 7803 : cluster [DBG] pgmap v8792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:29.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:29 smithi121 bash[36902]: audit 2022-04-22T21:14:28.685931+0000 mon.a (mon.0) 11371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:29 smithi115 bash[45495]: audit 2022-04-22T21:14:28.685931+0000 mon.a (mon.0) 11371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:29 smithi115 bash[46934]: audit 2022-04-22T21:14:28.685931+0000 mon.a (mon.0) 11371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:30 smithi121 bash[36902]: cluster 2022-04-22T21:14:29.047098+0000 mgr.y (mgr.34107) 7804 : cluster [DBG] pgmap v8793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:30 smithi121 bash[36902]: audit 2022-04-22T21:14:29.472816+0000 mon.a (mon.0) 11372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:30 smithi115 bash[45495]: cluster 2022-04-22T21:14:29.047098+0000 mgr.y (mgr.34107) 7804 : cluster [DBG] pgmap v8793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:30 smithi115 bash[45495]: audit 2022-04-22T21:14:29.472816+0000 mon.a (mon.0) 11372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:30 smithi115 bash[46934]: cluster 2022-04-22T21:14:29.047098+0000 mgr.y (mgr.34107) 7804 : cluster [DBG] pgmap v8793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:30 smithi115 bash[46934]: audit 2022-04-22T21:14:29.472816+0000 mon.a (mon.0) 11372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:31 smithi121 bash[36902]: audit 2022-04-22T21:14:30.694703+0000 mon.a (mon.0) 11373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:31 smithi115 bash[45495]: audit 2022-04-22T21:14:30.694703+0000 mon.a (mon.0) 11373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:31 smithi115 bash[46934]: audit 2022-04-22T21:14:30.694703+0000 mon.a (mon.0) 11373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:32.507 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:32 smithi115 bash[45495]: cluster 2022-04-22T21:14:31.048186+0000 mgr.y (mgr.34107) 7805 : cluster [DBG] pgmap v8794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:32.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:32 smithi115 bash[46934]: cluster 2022-04-22T21:14:31.048186+0000 mgr.y (mgr.34107) 7805 : cluster [DBG] pgmap v8794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:32 smithi121 bash[36902]: cluster 2022-04-22T21:14:31.048186+0000 mgr.y (mgr.34107) 7805 : cluster [DBG] pgmap v8794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:32.503Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:34 smithi121 bash[36902]: cluster 2022-04-22T21:14:33.049183+0000 mgr.y (mgr.34107) 7806 : cluster [DBG] pgmap v8795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:34 smithi121 bash[36902]: audit 2022-04-22T21:14:33.688925+0000 mon.a (mon.0) 11374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:34.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:34.376Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:34 smithi115 bash[45495]: cluster 2022-04-22T21:14:33.049183+0000 mgr.y (mgr.34107) 7806 : cluster [DBG] pgmap v8795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:34.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:34 smithi115 bash[45495]: audit 2022-04-22T21:14:33.688925+0000 mon.a (mon.0) 11374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:34 smithi115 bash[46934]: cluster 2022-04-22T21:14:33.049183+0000 mgr.y (mgr.34107) 7806 : cluster [DBG] pgmap v8795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:34 smithi115 bash[46934]: audit 2022-04-22T21:14:33.688925+0000 mon.a (mon.0) 11374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:35 smithi121 bash[36902]: audit 2022-04-22T21:14:34.476386+0000 mon.a (mon.0) 11375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:35 smithi115 bash[45495]: audit 2022-04-22T21:14:34.476386+0000 mon.a (mon.0) 11375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:35 smithi115 bash[46934]: audit 2022-04-22T21:14:34.476386+0000 mon.a (mon.0) 11375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:36 smithi121 bash[36902]: cluster 2022-04-22T21:14:35.050321+0000 mgr.y (mgr.34107) 7807 : cluster [DBG] pgmap v8796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:36 smithi121 bash[36902]: audit 2022-04-22T21:14:35.697561+0000 mon.a (mon.0) 11376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:36.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:36 smithi115 bash[45495]: cluster 2022-04-22T21:14:35.050321+0000 mgr.y (mgr.34107) 7807 : cluster [DBG] pgmap v8796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:36.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:36 smithi115 bash[45495]: audit 2022-04-22T21:14:35.697561+0000 mon.a (mon.0) 11376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:36.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:36 smithi115 bash[46934]: cluster 2022-04-22T21:14:35.050321+0000 mgr.y (mgr.34107) 7807 : cluster [DBG] pgmap v8796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:36.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:36 smithi115 bash[46934]: audit 2022-04-22T21:14:35.697561+0000 mon.a (mon.0) 11376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:38.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:38 smithi121 bash[36902]: cluster 2022-04-22T21:14:37.051299+0000 mgr.y (mgr.34107) 7808 : cluster [DBG] pgmap v8797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:38 smithi115 bash[45495]: cluster 2022-04-22T21:14:37.051299+0000 mgr.y (mgr.34107) 7808 : cluster [DBG] pgmap v8797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:38 smithi115 bash[46934]: cluster 2022-04-22T21:14:37.051299+0000 mgr.y (mgr.34107) 7808 : cluster [DBG] pgmap v8797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:39.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:39 smithi121 bash[36902]: audit 2022-04-22T21:14:38.692118+0000 mon.a (mon.0) 11377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:39 smithi115 bash[45495]: audit 2022-04-22T21:14:38.692118+0000 mon.a (mon.0) 11377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:39 smithi115 bash[46934]: audit 2022-04-22T21:14:38.692118+0000 mon.a (mon.0) 11377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:40 smithi121 bash[36902]: cluster 2022-04-22T21:14:39.052469+0000 mgr.y (mgr.34107) 7809 : cluster [DBG] pgmap v8798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:40 smithi121 bash[36902]: audit 2022-04-22T21:14:39.480039+0000 mon.a (mon.0) 11378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:40 smithi115 bash[45495]: cluster 2022-04-22T21:14:39.052469+0000 mgr.y (mgr.34107) 7809 : cluster [DBG] pgmap v8798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:40 smithi115 bash[45495]: audit 2022-04-22T21:14:39.480039+0000 mon.a (mon.0) 11378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:40 smithi115 bash[46934]: cluster 2022-04-22T21:14:39.052469+0000 mgr.y (mgr.34107) 7809 : cluster [DBG] pgmap v8798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:40 smithi115 bash[46934]: audit 2022-04-22T21:14:39.480039+0000 mon.a (mon.0) 11378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:41 smithi121 bash[36902]: audit 2022-04-22T21:14:40.700856+0000 mon.a (mon.0) 11379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:41 smithi115 bash[46934]: audit 2022-04-22T21:14:40.700856+0000 mon.a (mon.0) 11379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:41 smithi115 bash[45495]: audit 2022-04-22T21:14:40.700856+0000 mon.a (mon.0) 11379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:42.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:42 smithi115 bash[45495]: cluster 2022-04-22T21:14:41.053552+0000 mgr.y (mgr.34107) 7810 : cluster [DBG] pgmap v8799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:42.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:42 smithi115 bash[46934]: cluster 2022-04-22T21:14:41.053552+0000 mgr.y (mgr.34107) 7810 : cluster [DBG] pgmap v8799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:42 smithi121 bash[36902]: cluster 2022-04-22T21:14:41.053552+0000 mgr.y (mgr.34107) 7810 : cluster [DBG] pgmap v8799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:42.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:44 smithi121 bash[36902]: cluster 2022-04-22T21:14:43.054569+0000 mgr.y (mgr.34107) 7811 : cluster [DBG] pgmap v8800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:44 smithi121 bash[36902]: audit 2022-04-22T21:14:43.695161+0000 mon.a (mon.0) 11380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:44.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:44.377Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:44.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:44 smithi115 bash[45495]: cluster 2022-04-22T21:14:43.054569+0000 mgr.y (mgr.34107) 7811 : cluster [DBG] pgmap v8800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:44.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:44 smithi115 bash[45495]: audit 2022-04-22T21:14:43.695161+0000 mon.a (mon.0) 11380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:44.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:44 smithi115 bash[46934]: cluster 2022-04-22T21:14:43.054569+0000 mgr.y (mgr.34107) 7811 : cluster [DBG] pgmap v8800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:44.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:44 smithi115 bash[46934]: audit 2022-04-22T21:14:43.695161+0000 mon.a (mon.0) 11380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:45 smithi121 bash[36902]: audit 2022-04-22T21:14:44.483550+0000 mon.a (mon.0) 11381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:45 smithi115 bash[46934]: audit 2022-04-22T21:14:44.483550+0000 mon.a (mon.0) 11381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:45 smithi115 bash[45495]: audit 2022-04-22T21:14:44.483550+0000 mon.a (mon.0) 11381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:46 smithi121 bash[36902]: cluster 2022-04-22T21:14:45.055585+0000 mgr.y (mgr.34107) 7812 : cluster [DBG] pgmap v8801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:46 smithi121 bash[36902]: audit 2022-04-22T21:14:45.704799+0000 mon.a (mon.0) 11382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:46 smithi115 bash[45495]: cluster 2022-04-22T21:14:45.055585+0000 mgr.y (mgr.34107) 7812 : cluster [DBG] pgmap v8801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:46 smithi115 bash[45495]: audit 2022-04-22T21:14:45.704799+0000 mon.a (mon.0) 11382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:46 smithi115 bash[46934]: cluster 2022-04-22T21:14:45.055585+0000 mgr.y (mgr.34107) 7812 : cluster [DBG] pgmap v8801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:46 smithi115 bash[46934]: audit 2022-04-22T21:14:45.704799+0000 mon.a (mon.0) 11382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:48 smithi121 bash[36902]: cluster 2022-04-22T21:14:47.056649+0000 mgr.y (mgr.34107) 7813 : cluster [DBG] pgmap v8802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:48 smithi115 bash[45495]: cluster 2022-04-22T21:14:47.056649+0000 mgr.y (mgr.34107) 7813 : cluster [DBG] pgmap v8802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:48 smithi115 bash[46934]: cluster 2022-04-22T21:14:47.056649+0000 mgr.y (mgr.34107) 7813 : cluster [DBG] pgmap v8802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:49 smithi121 bash[36902]: audit 2022-04-22T21:14:48.698251+0000 mon.a (mon.0) 11383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:49 smithi115 bash[45495]: audit 2022-04-22T21:14:48.698251+0000 mon.a (mon.0) 11383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:49 smithi115 bash[46934]: audit 2022-04-22T21:14:48.698251+0000 mon.a (mon.0) 11383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:50 smithi121 bash[36902]: cluster 2022-04-22T21:14:49.057686+0000 mgr.y (mgr.34107) 7814 : cluster [DBG] pgmap v8803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:50 smithi121 bash[36902]: audit 2022-04-22T21:14:49.486986+0000 mon.a (mon.0) 11384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:50 smithi115 bash[45495]: cluster 2022-04-22T21:14:49.057686+0000 mgr.y (mgr.34107) 7814 : cluster [DBG] pgmap v8803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:50 smithi115 bash[45495]: audit 2022-04-22T21:14:49.486986+0000 mon.a (mon.0) 11384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:50 smithi115 bash[46934]: cluster 2022-04-22T21:14:49.057686+0000 mgr.y (mgr.34107) 7814 : cluster [DBG] pgmap v8803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:50 smithi115 bash[46934]: audit 2022-04-22T21:14:49.486986+0000 mon.a (mon.0) 11384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:51 smithi121 bash[36902]: audit 2022-04-22T21:14:50.707894+0000 mon.a (mon.0) 11385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:51 smithi115 bash[45495]: audit 2022-04-22T21:14:50.707894+0000 mon.a (mon.0) 11385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:51 smithi115 bash[46934]: audit 2022-04-22T21:14:50.707894+0000 mon.a (mon.0) 11385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:52.510 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:52 smithi115 bash[45495]: cluster 2022-04-22T21:14:51.058691+0000 mgr.y (mgr.34107) 7815 : cluster [DBG] pgmap v8804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:52.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:52 smithi115 bash[46934]: cluster 2022-04-22T21:14:51.058691+0000 mgr.y (mgr.34107) 7815 : cluster [DBG] pgmap v8804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:52 smithi121 bash[36902]: cluster 2022-04-22T21:14:51.058691+0000 mgr.y (mgr.34107) 7815 : cluster [DBG] pgmap v8804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:52.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:54.445 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:14:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:54 smithi121 bash[36902]: cluster 2022-04-22T21:14:53.059694+0000 mgr.y (mgr.34107) 7816 : cluster [DBG] pgmap v8805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:54 smithi121 bash[36902]: audit 2022-04-22T21:14:53.702761+0000 mon.a (mon.0) 11386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:54 smithi115 bash[45495]: cluster 2022-04-22T21:14:53.059694+0000 mgr.y (mgr.34107) 7816 : cluster [DBG] pgmap v8805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:54 smithi115 bash[45495]: audit 2022-04-22T21:14:53.702761+0000 mon.a (mon.0) 11386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:54 smithi115 bash[46934]: cluster 2022-04-22T21:14:53.059694+0000 mgr.y (mgr.34107) 7816 : cluster [DBG] pgmap v8805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:54 smithi115 bash[46934]: audit 2022-04-22T21:14:53.702761+0000 mon.a (mon.0) 11386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:54.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:14:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:14:54.377Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:14:54.798 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:14:54.798 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 51s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:14:54.798 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 115s ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 51s ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 115s ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 115s ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 51s ago 3h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 51s ago 3h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 115s ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 51s ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:14:54.799 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 51s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 115s ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 51s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 51s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 51s ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 51s ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 115s ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 115s ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 115s ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 115s ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:14:54.800 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 115s ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:14:55.199 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:14:55.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:14:55.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:14:55.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:55.200 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:14:55.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:14:55.201 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:14:55.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:14:55.202 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:14:55.202 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:14:55.202 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:14:55.203 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:14:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:55 smithi121 bash[36902]: audit 2022-04-22T21:14:54.424045+0000 mgr.y (mgr.34107) 7817 : audit [DBG] from='client.43488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:55 smithi121 bash[36902]: audit 2022-04-22T21:14:54.490630+0000 mon.a (mon.0) 11387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:55 smithi121 bash[36902]: audit 2022-04-22T21:14:54.767231+0000 mgr.y (mgr.34107) 7818 : audit [DBG] from='client.53425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:55 smithi121 bash[36902]: audit 2022-04-22T21:14:55.194869+0000 mon.a (mon.0) 11388 : audit [DBG] from='client.? 172.21.15.115:0/4176438051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[46934]: audit 2022-04-22T21:14:54.424045+0000 mgr.y (mgr.34107) 7817 : audit [DBG] from='client.43488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[46934]: audit 2022-04-22T21:14:54.490630+0000 mon.a (mon.0) 11387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[46934]: audit 2022-04-22T21:14:54.767231+0000 mgr.y (mgr.34107) 7818 : audit [DBG] from='client.53425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[46934]: audit 2022-04-22T21:14:55.194869+0000 mon.a (mon.0) 11388 : audit [DBG] from='client.? 172.21.15.115:0/4176438051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:55.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[45495]: audit 2022-04-22T21:14:54.424045+0000 mgr.y (mgr.34107) 7817 : audit [DBG] from='client.43488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[45495]: audit 2022-04-22T21:14:54.490630+0000 mon.a (mon.0) 11387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:14:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[45495]: audit 2022-04-22T21:14:54.767231+0000 mgr.y (mgr.34107) 7818 : audit [DBG] from='client.53425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:14:55.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:55 smithi115 bash[45495]: audit 2022-04-22T21:14:55.194869+0000 mon.a (mon.0) 11388 : audit [DBG] from='client.? 172.21.15.115:0/4176438051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:14:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:56 smithi121 bash[36902]: cluster 2022-04-22T21:14:55.060744+0000 mgr.y (mgr.34107) 7819 : cluster [DBG] pgmap v8806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:56 smithi121 bash[36902]: audit 2022-04-22T21:14:55.711046+0000 mon.a (mon.0) 11389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:56 smithi115 bash[46934]: cluster 2022-04-22T21:14:55.060744+0000 mgr.y (mgr.34107) 7819 : cluster [DBG] pgmap v8806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:56 smithi115 bash[46934]: audit 2022-04-22T21:14:55.711046+0000 mon.a (mon.0) 11389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:56 smithi115 bash[45495]: cluster 2022-04-22T21:14:55.060744+0000 mgr.y (mgr.34107) 7819 : cluster [DBG] pgmap v8806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:56.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:56 smithi115 bash[45495]: audit 2022-04-22T21:14:55.711046+0000 mon.a (mon.0) 11389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:14:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:58 smithi121 bash[36902]: cluster 2022-04-22T21:14:57.061979+0000 mgr.y (mgr.34107) 7820 : cluster [DBG] pgmap v8807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:58.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:58 smithi115 bash[45495]: cluster 2022-04-22T21:14:57.061979+0000 mgr.y (mgr.34107) 7820 : cluster [DBG] pgmap v8807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:58 smithi115 bash[46934]: cluster 2022-04-22T21:14:57.061979+0000 mgr.y (mgr.34107) 7820 : cluster [DBG] pgmap v8807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:14:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:14:59 smithi121 bash[36902]: audit 2022-04-22T21:14:58.705769+0000 mon.a (mon.0) 11390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:14:59 smithi115 bash[45495]: audit 2022-04-22T21:14:58.705769+0000 mon.a (mon.0) 11390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:14:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:14:59 smithi115 bash[46934]: audit 2022-04-22T21:14:58.705769+0000 mon.a (mon.0) 11390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:00.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:00 smithi121 bash[36902]: cluster 2022-04-22T21:14:59.063073+0000 mgr.y (mgr.34107) 7821 : cluster [DBG] pgmap v8808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:00 smithi121 bash[36902]: audit 2022-04-22T21:14:59.494170+0000 mon.a (mon.0) 11391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:00 smithi115 bash[45495]: cluster 2022-04-22T21:14:59.063073+0000 mgr.y (mgr.34107) 7821 : cluster [DBG] pgmap v8808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:00 smithi115 bash[45495]: audit 2022-04-22T21:14:59.494170+0000 mon.a (mon.0) 11391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:00 smithi115 bash[46934]: cluster 2022-04-22T21:14:59.063073+0000 mgr.y (mgr.34107) 7821 : cluster [DBG] pgmap v8808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:00 smithi115 bash[46934]: audit 2022-04-22T21:14:59.494170+0000 mon.a (mon.0) 11391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:01.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:01 smithi121 bash[36902]: audit 2022-04-22T21:15:00.714007+0000 mon.a (mon.0) 11392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:01 smithi115 bash[45495]: audit 2022-04-22T21:15:00.714007+0000 mon.a (mon.0) 11392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:01 smithi115 bash[46934]: audit 2022-04-22T21:15:00.714007+0000 mon.a (mon.0) 11392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:02.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:02 smithi115 bash[46934]: cluster 2022-04-22T21:15:01.064231+0000 mgr.y (mgr.34107) 7822 : cluster [DBG] pgmap v8809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:02.511 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:02 smithi115 bash[45495]: cluster 2022-04-22T21:15:01.064231+0000 mgr.y (mgr.34107) 7822 : cluster [DBG] pgmap v8809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:02 smithi121 bash[36902]: cluster 2022-04-22T21:15:01.064231+0000 mgr.y (mgr.34107) 7822 : cluster [DBG] pgmap v8809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:02.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:04.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:04 smithi121 bash[36902]: cluster 2022-04-22T21:15:03.065224+0000 mgr.y (mgr.34107) 7823 : cluster [DBG] pgmap v8810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:04 smithi121 bash[36902]: audit 2022-04-22T21:15:03.708485+0000 mon.a (mon.0) 11393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:04 smithi115 bash[45495]: cluster 2022-04-22T21:15:03.065224+0000 mgr.y (mgr.34107) 7823 : cluster [DBG] pgmap v8810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:04 smithi115 bash[45495]: audit 2022-04-22T21:15:03.708485+0000 mon.a (mon.0) 11393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:04.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:04.380Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:04 smithi115 bash[46934]: cluster 2022-04-22T21:15:03.065224+0000 mgr.y (mgr.34107) 7823 : cluster [DBG] pgmap v8810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:04.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:04 smithi115 bash[46934]: audit 2022-04-22T21:15:03.708485+0000 mon.a (mon.0) 11393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:05.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:05 smithi121 bash[36902]: audit 2022-04-22T21:15:04.497948+0000 mon.a (mon.0) 11394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:05.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:05 smithi115 bash[45495]: audit 2022-04-22T21:15:04.497948+0000 mon.a (mon.0) 11394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:05 smithi115 bash[46934]: audit 2022-04-22T21:15:04.497948+0000 mon.a (mon.0) 11394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:06 smithi115 bash[45495]: cluster 2022-04-22T21:15:05.066368+0000 mgr.y (mgr.34107) 7824 : cluster [DBG] pgmap v8811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:06 smithi115 bash[45495]: audit 2022-04-22T21:15:05.717068+0000 mon.a (mon.0) 11395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:06 smithi115 bash[46934]: cluster 2022-04-22T21:15:05.066368+0000 mgr.y (mgr.34107) 7824 : cluster [DBG] pgmap v8811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:06 smithi115 bash[46934]: audit 2022-04-22T21:15:05.717068+0000 mon.a (mon.0) 11395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:06 smithi121 bash[36902]: cluster 2022-04-22T21:15:05.066368+0000 mgr.y (mgr.34107) 7824 : cluster [DBG] pgmap v8811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:06 smithi121 bash[36902]: audit 2022-04-22T21:15:05.717068+0000 mon.a (mon.0) 11395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:08 smithi115 bash[45495]: cluster 2022-04-22T21:15:07.067544+0000 mgr.y (mgr.34107) 7825 : cluster [DBG] pgmap v8812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:08.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:08 smithi115 bash[46934]: cluster 2022-04-22T21:15:07.067544+0000 mgr.y (mgr.34107) 7825 : cluster [DBG] pgmap v8812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:08 smithi121 bash[36902]: cluster 2022-04-22T21:15:07.067544+0000 mgr.y (mgr.34107) 7825 : cluster [DBG] pgmap v8812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:09 smithi115 bash[45495]: audit 2022-04-22T21:15:08.711338+0000 mon.a (mon.0) 11396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:09 smithi115 bash[46934]: audit 2022-04-22T21:15:08.711338+0000 mon.a (mon.0) 11396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:09 smithi121 bash[36902]: audit 2022-04-22T21:15:08.711338+0000 mon.a (mon.0) 11396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:10 smithi115 bash[45495]: cluster 2022-04-22T21:15:09.068598+0000 mgr.y (mgr.34107) 7826 : cluster [DBG] pgmap v8813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:10 smithi115 bash[45495]: audit 2022-04-22T21:15:09.501616+0000 mon.a (mon.0) 11397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:10 smithi115 bash[46934]: cluster 2022-04-22T21:15:09.068598+0000 mgr.y (mgr.34107) 7826 : cluster [DBG] pgmap v8813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:10 smithi115 bash[46934]: audit 2022-04-22T21:15:09.501616+0000 mon.a (mon.0) 11397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:10 smithi121 bash[36902]: cluster 2022-04-22T21:15:09.068598+0000 mgr.y (mgr.34107) 7826 : cluster [DBG] pgmap v8813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:10 smithi121 bash[36902]: audit 2022-04-22T21:15:09.501616+0000 mon.a (mon.0) 11397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:11 smithi115 bash[45495]: audit 2022-04-22T21:15:10.720216+0000 mon.a (mon.0) 11398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:11 smithi115 bash[46934]: audit 2022-04-22T21:15:10.720216+0000 mon.a (mon.0) 11398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:11 smithi121 bash[36902]: audit 2022-04-22T21:15:10.720216+0000 mon.a (mon.0) 11398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:12 smithi121 bash[36902]: cluster 2022-04-22T21:15:11.069689+0000 mgr.y (mgr.34107) 7827 : cluster [DBG] pgmap v8814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:12 smithi115 bash[45495]: cluster 2022-04-22T21:15:11.069689+0000 mgr.y (mgr.34107) 7827 : cluster [DBG] pgmap v8814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:12.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:12 smithi115 bash[46934]: cluster 2022-04-22T21:15:11.069689+0000 mgr.y (mgr.34107) 7827 : cluster [DBG] pgmap v8814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[45495]: audit 2022-04-22T21:15:12.926794+0000 mon.a (mon.0) 11399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:15:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[45495]: audit 2022-04-22T21:15:12.930048+0000 mon.a (mon.0) 11400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:15:13.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[45495]: audit 2022-04-22T21:15:12.931925+0000 mon.a (mon.0) 11401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:15:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[46934]: audit 2022-04-22T21:15:12.926794+0000 mon.a (mon.0) 11399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:15:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[46934]: audit 2022-04-22T21:15:12.930048+0000 mon.a (mon.0) 11400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:15:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:13 smithi115 bash[46934]: audit 2022-04-22T21:15:12.931925+0000 mon.a (mon.0) 11401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:15:13.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:13 smithi121 bash[36902]: audit 2022-04-22T21:15:12.926794+0000 mon.a (mon.0) 11399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:15:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:13 smithi121 bash[36902]: audit 2022-04-22T21:15:12.930048+0000 mon.a (mon.0) 11400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:15:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:13 smithi121 bash[36902]: audit 2022-04-22T21:15:12.931925+0000 mon.a (mon.0) 11401 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:15:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:14 smithi121 bash[36902]: cluster 2022-04-22T21:15:13.070679+0000 mgr.y (mgr.34107) 7828 : cluster [DBG] pgmap v8815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:14 smithi121 bash[36902]: audit 2022-04-22T21:15:13.308159+0000 mon.a (mon.0) 11402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:15:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:14 smithi121 bash[36902]: audit 2022-04-22T21:15:13.714435+0000 mon.a (mon.0) 11403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:14.380Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[45495]: cluster 2022-04-22T21:15:13.070679+0000 mgr.y (mgr.34107) 7828 : cluster [DBG] pgmap v8815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[45495]: audit 2022-04-22T21:15:13.308159+0000 mon.a (mon.0) 11402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:15:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[45495]: audit 2022-04-22T21:15:13.714435+0000 mon.a (mon.0) 11403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[46934]: cluster 2022-04-22T21:15:13.070679+0000 mgr.y (mgr.34107) 7828 : cluster [DBG] pgmap v8815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[46934]: audit 2022-04-22T21:15:13.308159+0000 mon.a (mon.0) 11402 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:15:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:14 smithi115 bash[46934]: audit 2022-04-22T21:15:13.714435+0000 mon.a (mon.0) 11403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:15.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:15 smithi121 bash[36902]: audit 2022-04-22T21:15:14.505368+0000 mon.a (mon.0) 11404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:15 smithi115 bash[45495]: audit 2022-04-22T21:15:14.505368+0000 mon.a (mon.0) 11404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:15 smithi115 bash[46934]: audit 2022-04-22T21:15:14.505368+0000 mon.a (mon.0) 11404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:16 smithi121 bash[36902]: cluster 2022-04-22T21:15:15.071862+0000 mgr.y (mgr.34107) 7829 : cluster [DBG] pgmap v8816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:16 smithi121 bash[36902]: audit 2022-04-22T21:15:15.723366+0000 mon.a (mon.0) 11405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:16 smithi121 bash[36902]: audit 2022-04-22T21:15:15.917685+0000 mon.a (mon.0) 11406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:15:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:16 smithi121 bash[36902]: audit 2022-04-22T21:15:15.918229+0000 mon.a (mon.0) 11407 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:15:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[45495]: cluster 2022-04-22T21:15:15.071862+0000 mgr.y (mgr.34107) 7829 : cluster [DBG] pgmap v8816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[45495]: audit 2022-04-22T21:15:15.723366+0000 mon.a (mon.0) 11405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[45495]: audit 2022-04-22T21:15:15.917685+0000 mon.a (mon.0) 11406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[45495]: audit 2022-04-22T21:15:15.918229+0000 mon.a (mon.0) 11407 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[46934]: cluster 2022-04-22T21:15:15.071862+0000 mgr.y (mgr.34107) 7829 : cluster [DBG] pgmap v8816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[46934]: audit 2022-04-22T21:15:15.723366+0000 mon.a (mon.0) 11405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[46934]: audit 2022-04-22T21:15:15.917685+0000 mon.a (mon.0) 11406 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:15:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:16 smithi115 bash[46934]: audit 2022-04-22T21:15:15.918229+0000 mon.a (mon.0) 11407 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:15:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:18 smithi121 bash[36902]: cluster 2022-04-22T21:15:17.073018+0000 mgr.y (mgr.34107) 7830 : cluster [DBG] pgmap v8817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:18 smithi115 bash[45495]: cluster 2022-04-22T21:15:17.073018+0000 mgr.y (mgr.34107) 7830 : cluster [DBG] pgmap v8817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:18 smithi115 bash[46934]: cluster 2022-04-22T21:15:17.073018+0000 mgr.y (mgr.34107) 7830 : cluster [DBG] pgmap v8817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:19 smithi121 bash[36902]: audit 2022-04-22T21:15:18.717406+0000 mon.a (mon.0) 11408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:19 smithi115 bash[45495]: audit 2022-04-22T21:15:18.717406+0000 mon.a (mon.0) 11408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:19 smithi115 bash[46934]: audit 2022-04-22T21:15:18.717406+0000 mon.a (mon.0) 11408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:20 smithi121 bash[36902]: cluster 2022-04-22T21:15:19.073972+0000 mgr.y (mgr.34107) 7831 : cluster [DBG] pgmap v8818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:20 smithi121 bash[36902]: audit 2022-04-22T21:15:19.509065+0000 mon.a (mon.0) 11409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:20 smithi115 bash[45495]: cluster 2022-04-22T21:15:19.073972+0000 mgr.y (mgr.34107) 7831 : cluster [DBG] pgmap v8818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:20 smithi115 bash[45495]: audit 2022-04-22T21:15:19.509065+0000 mon.a (mon.0) 11409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:20 smithi115 bash[46934]: cluster 2022-04-22T21:15:19.073972+0000 mgr.y (mgr.34107) 7831 : cluster [DBG] pgmap v8818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:20 smithi115 bash[46934]: audit 2022-04-22T21:15:19.509065+0000 mon.a (mon.0) 11409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:21 smithi121 bash[36902]: audit 2022-04-22T21:15:20.726216+0000 mon.a (mon.0) 11410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:21 smithi115 bash[45495]: audit 2022-04-22T21:15:20.726216+0000 mon.a (mon.0) 11410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:21 smithi115 bash[46934]: audit 2022-04-22T21:15:20.726216+0000 mon.a (mon.0) 11410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:22 smithi121 bash[36902]: cluster 2022-04-22T21:15:21.075040+0000 mgr.y (mgr.34107) 7832 : cluster [DBG] pgmap v8819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:22 smithi115 bash[45495]: cluster 2022-04-22T21:15:21.075040+0000 mgr.y (mgr.34107) 7832 : cluster [DBG] pgmap v8819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:22 smithi115 bash[46934]: cluster 2022-04-22T21:15:21.075040+0000 mgr.y (mgr.34107) 7832 : cluster [DBG] pgmap v8819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:22.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:24.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:24 smithi121 bash[36902]: cluster 2022-04-22T21:15:23.076016+0000 mgr.y (mgr.34107) 7833 : cluster [DBG] pgmap v8820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:24.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:24 smithi121 bash[36902]: audit 2022-04-22T21:15:23.720407+0000 mon.a (mon.0) 11411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:24.380Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:24 smithi115 bash[45495]: cluster 2022-04-22T21:15:23.076016+0000 mgr.y (mgr.34107) 7833 : cluster [DBG] pgmap v8820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:24 smithi115 bash[45495]: audit 2022-04-22T21:15:23.720407+0000 mon.a (mon.0) 11411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:24 smithi115 bash[46934]: cluster 2022-04-22T21:15:23.076016+0000 mgr.y (mgr.34107) 7833 : cluster [DBG] pgmap v8820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:24 smithi115 bash[46934]: audit 2022-04-22T21:15:23.720407+0000 mon.a (mon.0) 11411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:25.581 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:15:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:25 smithi121 bash[36902]: audit 2022-04-22T21:15:24.512753+0000 mon.a (mon.0) 11412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:25 smithi115 bash[46934]: audit 2022-04-22T21:15:24.512753+0000 mon.a (mon.0) 11412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:25 smithi115 bash[45495]: audit 2022-04-22T21:15:24.512753+0000 mon.a (mon.0) 11412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:25.926 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:15:25.926 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 82s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:15:25.926 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 82s ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 82s ago 3h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 82s ago 3h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 82s ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 82s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:15:25.927 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 82s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 82s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 82s ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 82s ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:15:25.928 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:15:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:15:26.329 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:15:26.329 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:15:26.330 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:15:26.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:26.331 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:15:26.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:15:26.331 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:15:26.331 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:15:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:26 smithi121 bash[36902]: cluster 2022-04-22T21:15:25.077015+0000 mgr.y (mgr.34107) 7834 : cluster [DBG] pgmap v8821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:26 smithi121 bash[36902]: audit 2022-04-22T21:15:25.557532+0000 mgr.y (mgr.34107) 7835 : audit [DBG] from='client.43506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:26 smithi121 bash[36902]: audit 2022-04-22T21:15:25.728984+0000 mon.a (mon.0) 11413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:26 smithi121 bash[36902]: audit 2022-04-22T21:15:26.325065+0000 mon.c (mon.1) 283 : audit [DBG] from='client.? 172.21.15.115:0/2030633200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[45495]: cluster 2022-04-22T21:15:25.077015+0000 mgr.y (mgr.34107) 7834 : cluster [DBG] pgmap v8821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[45495]: audit 2022-04-22T21:15:25.557532+0000 mgr.y (mgr.34107) 7835 : audit [DBG] from='client.43506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[45495]: audit 2022-04-22T21:15:25.728984+0000 mon.a (mon.0) 11413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[45495]: audit 2022-04-22T21:15:26.325065+0000 mon.c (mon.1) 283 : audit [DBG] from='client.? 172.21.15.115:0/2030633200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[46934]: cluster 2022-04-22T21:15:25.077015+0000 mgr.y (mgr.34107) 7834 : cluster [DBG] pgmap v8821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[46934]: audit 2022-04-22T21:15:25.557532+0000 mgr.y (mgr.34107) 7835 : audit [DBG] from='client.43506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[46934]: audit 2022-04-22T21:15:25.728984+0000 mon.a (mon.0) 11413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:26 smithi115 bash[46934]: audit 2022-04-22T21:15:26.325065+0000 mon.c (mon.1) 283 : audit [DBG] from='client.? 172.21.15.115:0/2030633200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:27.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:27 smithi121 bash[36902]: audit 2022-04-22T21:15:25.900349+0000 mgr.y (mgr.34107) 7836 : audit [DBG] from='client.53443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:27 smithi115 bash[45495]: audit 2022-04-22T21:15:25.900349+0000 mgr.y (mgr.34107) 7836 : audit [DBG] from='client.53443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:27 smithi115 bash[46934]: audit 2022-04-22T21:15:25.900349+0000 mgr.y (mgr.34107) 7836 : audit [DBG] from='client.53443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:28 smithi121 bash[36902]: cluster 2022-04-22T21:15:27.078300+0000 mgr.y (mgr.34107) 7837 : cluster [DBG] pgmap v8822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:28 smithi115 bash[45495]: cluster 2022-04-22T21:15:27.078300+0000 mgr.y (mgr.34107) 7837 : cluster [DBG] pgmap v8822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:28 smithi115 bash[46934]: cluster 2022-04-22T21:15:27.078300+0000 mgr.y (mgr.34107) 7837 : cluster [DBG] pgmap v8822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:29.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:29 smithi121 bash[36902]: audit 2022-04-22T21:15:28.723450+0000 mon.a (mon.0) 11414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:29 smithi115 bash[45495]: audit 2022-04-22T21:15:28.723450+0000 mon.a (mon.0) 11414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:29 smithi115 bash[46934]: audit 2022-04-22T21:15:28.723450+0000 mon.a (mon.0) 11414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:30 smithi121 bash[36902]: cluster 2022-04-22T21:15:29.079320+0000 mgr.y (mgr.34107) 7838 : cluster [DBG] pgmap v8823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:30 smithi121 bash[36902]: audit 2022-04-22T21:15:29.516441+0000 mon.a (mon.0) 11415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:30 smithi115 bash[46934]: cluster 2022-04-22T21:15:29.079320+0000 mgr.y (mgr.34107) 7838 : cluster [DBG] pgmap v8823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:30 smithi115 bash[46934]: audit 2022-04-22T21:15:29.516441+0000 mon.a (mon.0) 11415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:30 smithi115 bash[45495]: cluster 2022-04-22T21:15:29.079320+0000 mgr.y (mgr.34107) 7838 : cluster [DBG] pgmap v8823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:30 smithi115 bash[45495]: audit 2022-04-22T21:15:29.516441+0000 mon.a (mon.0) 11415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:31.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:31 smithi121 bash[36902]: audit 2022-04-22T21:15:30.731931+0000 mon.a (mon.0) 11416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:31 smithi115 bash[45495]: audit 2022-04-22T21:15:30.731931+0000 mon.a (mon.0) 11416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:31 smithi115 bash[46934]: audit 2022-04-22T21:15:30.731931+0000 mon.a (mon.0) 11416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:32 smithi121 bash[36902]: cluster 2022-04-22T21:15:31.080409+0000 mgr.y (mgr.34107) 7839 : cluster [DBG] pgmap v8824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:32 smithi115 bash[45495]: cluster 2022-04-22T21:15:31.080409+0000 mgr.y (mgr.34107) 7839 : cluster [DBG] pgmap v8824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:32.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:32 smithi115 bash[46934]: cluster 2022-04-22T21:15:31.080409+0000 mgr.y (mgr.34107) 7839 : cluster [DBG] pgmap v8824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:34.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:34 smithi121 bash[36902]: cluster 2022-04-22T21:15:33.081483+0000 mgr.y (mgr.34107) 7840 : cluster [DBG] pgmap v8825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:34 smithi121 bash[36902]: audit 2022-04-22T21:15:33.726617+0000 mon.a (mon.0) 11417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:34.381Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:34 smithi115 bash[45495]: cluster 2022-04-22T21:15:33.081483+0000 mgr.y (mgr.34107) 7840 : cluster [DBG] pgmap v8825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:34 smithi115 bash[45495]: audit 2022-04-22T21:15:33.726617+0000 mon.a (mon.0) 11417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:34 smithi115 bash[46934]: cluster 2022-04-22T21:15:33.081483+0000 mgr.y (mgr.34107) 7840 : cluster [DBG] pgmap v8825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:34 smithi115 bash[46934]: audit 2022-04-22T21:15:33.726617+0000 mon.a (mon.0) 11417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:35.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:35 smithi121 bash[36902]: audit 2022-04-22T21:15:34.520083+0000 mon.a (mon.0) 11418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:35 smithi115 bash[45495]: audit 2022-04-22T21:15:34.520083+0000 mon.a (mon.0) 11418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:35 smithi115 bash[46934]: audit 2022-04-22T21:15:34.520083+0000 mon.a (mon.0) 11418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:36.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:36 smithi121 bash[36902]: cluster 2022-04-22T21:15:35.082515+0000 mgr.y (mgr.34107) 7841 : cluster [DBG] pgmap v8826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:36 smithi121 bash[36902]: audit 2022-04-22T21:15:35.734968+0000 mon.a (mon.0) 11419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:36 smithi115 bash[45495]: cluster 2022-04-22T21:15:35.082515+0000 mgr.y (mgr.34107) 7841 : cluster [DBG] pgmap v8826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:36 smithi115 bash[45495]: audit 2022-04-22T21:15:35.734968+0000 mon.a (mon.0) 11419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:36 smithi115 bash[46934]: cluster 2022-04-22T21:15:35.082515+0000 mgr.y (mgr.34107) 7841 : cluster [DBG] pgmap v8826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:36 smithi115 bash[46934]: audit 2022-04-22T21:15:35.734968+0000 mon.a (mon.0) 11419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:38.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:38 smithi121 bash[36902]: cluster 2022-04-22T21:15:37.083823+0000 mgr.y (mgr.34107) 7842 : cluster [DBG] pgmap v8827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:38 smithi115 bash[46934]: cluster 2022-04-22T21:15:37.083823+0000 mgr.y (mgr.34107) 7842 : cluster [DBG] pgmap v8827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:38 smithi115 bash[45495]: cluster 2022-04-22T21:15:37.083823+0000 mgr.y (mgr.34107) 7842 : cluster [DBG] pgmap v8827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:39 smithi121 bash[36902]: audit 2022-04-22T21:15:38.729776+0000 mon.a (mon.0) 11420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:39 smithi115 bash[46934]: audit 2022-04-22T21:15:38.729776+0000 mon.a (mon.0) 11420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:39.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:39 smithi115 bash[45495]: audit 2022-04-22T21:15:38.729776+0000 mon.a (mon.0) 11420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:40 smithi121 bash[36902]: cluster 2022-04-22T21:15:39.084921+0000 mgr.y (mgr.34107) 7843 : cluster [DBG] pgmap v8828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:40 smithi121 bash[36902]: audit 2022-04-22T21:15:39.523557+0000 mon.a (mon.0) 11421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:40 smithi115 bash[46934]: cluster 2022-04-22T21:15:39.084921+0000 mgr.y (mgr.34107) 7843 : cluster [DBG] pgmap v8828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:40 smithi115 bash[46934]: audit 2022-04-22T21:15:39.523557+0000 mon.a (mon.0) 11421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:40 smithi115 bash[45495]: cluster 2022-04-22T21:15:39.084921+0000 mgr.y (mgr.34107) 7843 : cluster [DBG] pgmap v8828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:40 smithi115 bash[45495]: audit 2022-04-22T21:15:39.523557+0000 mon.a (mon.0) 11421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:41.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:41 smithi121 bash[36902]: audit 2022-04-22T21:15:40.738085+0000 mon.a (mon.0) 11422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:41 smithi115 bash[46934]: audit 2022-04-22T21:15:40.738085+0000 mon.a (mon.0) 11422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:41 smithi115 bash[45495]: audit 2022-04-22T21:15:40.738085+0000 mon.a (mon.0) 11422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:42 smithi121 bash[36902]: cluster 2022-04-22T21:15:41.085995+0000 mgr.y (mgr.34107) 7844 : cluster [DBG] pgmap v8829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:42 smithi115 bash[46934]: cluster 2022-04-22T21:15:41.085995+0000 mgr.y (mgr.34107) 7844 : cluster [DBG] pgmap v8829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:42.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:42 smithi115 bash[45495]: cluster 2022-04-22T21:15:41.085995+0000 mgr.y (mgr.34107) 7844 : cluster [DBG] pgmap v8829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:44 smithi121 bash[36902]: cluster 2022-04-22T21:15:43.087011+0000 mgr.y (mgr.34107) 7845 : cluster [DBG] pgmap v8830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:44 smithi121 bash[36902]: audit 2022-04-22T21:15:43.732841+0000 mon.a (mon.0) 11423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:44.381Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:44 smithi115 bash[45495]: cluster 2022-04-22T21:15:43.087011+0000 mgr.y (mgr.34107) 7845 : cluster [DBG] pgmap v8830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:44 smithi115 bash[45495]: audit 2022-04-22T21:15:43.732841+0000 mon.a (mon.0) 11423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:44 smithi115 bash[46934]: cluster 2022-04-22T21:15:43.087011+0000 mgr.y (mgr.34107) 7845 : cluster [DBG] pgmap v8830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:44 smithi115 bash[46934]: audit 2022-04-22T21:15:43.732841+0000 mon.a (mon.0) 11423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:45.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:45 smithi121 bash[36902]: audit 2022-04-22T21:15:44.527194+0000 mon.a (mon.0) 11424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:45 smithi115 bash[46934]: audit 2022-04-22T21:15:44.527194+0000 mon.a (mon.0) 11424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:45 smithi115 bash[45495]: audit 2022-04-22T21:15:44.527194+0000 mon.a (mon.0) 11424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:46 smithi121 bash[36902]: cluster 2022-04-22T21:15:45.088017+0000 mgr.y (mgr.34107) 7846 : cluster [DBG] pgmap v8831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:46 smithi121 bash[36902]: audit 2022-04-22T21:15:45.741012+0000 mon.a (mon.0) 11425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:46 smithi115 bash[45495]: cluster 2022-04-22T21:15:45.088017+0000 mgr.y (mgr.34107) 7846 : cluster [DBG] pgmap v8831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:46 smithi115 bash[45495]: audit 2022-04-22T21:15:45.741012+0000 mon.a (mon.0) 11425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:46 smithi115 bash[46934]: cluster 2022-04-22T21:15:45.088017+0000 mgr.y (mgr.34107) 7846 : cluster [DBG] pgmap v8831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:46 smithi115 bash[46934]: audit 2022-04-22T21:15:45.741012+0000 mon.a (mon.0) 11425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:48 smithi121 bash[36902]: cluster 2022-04-22T21:15:47.089294+0000 mgr.y (mgr.34107) 7847 : cluster [DBG] pgmap v8832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:48 smithi115 bash[45495]: cluster 2022-04-22T21:15:47.089294+0000 mgr.y (mgr.34107) 7847 : cluster [DBG] pgmap v8832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:48 smithi115 bash[46934]: cluster 2022-04-22T21:15:47.089294+0000 mgr.y (mgr.34107) 7847 : cluster [DBG] pgmap v8832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:49 smithi121 bash[36902]: audit 2022-04-22T21:15:48.736007+0000 mon.a (mon.0) 11426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:49 smithi115 bash[45495]: audit 2022-04-22T21:15:48.736007+0000 mon.a (mon.0) 11426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:49 smithi115 bash[46934]: audit 2022-04-22T21:15:48.736007+0000 mon.a (mon.0) 11426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:50 smithi121 bash[36902]: cluster 2022-04-22T21:15:49.090340+0000 mgr.y (mgr.34107) 7848 : cluster [DBG] pgmap v8833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:50 smithi121 bash[36902]: audit 2022-04-22T21:15:49.530810+0000 mon.a (mon.0) 11427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:50 smithi115 bash[46934]: cluster 2022-04-22T21:15:49.090340+0000 mgr.y (mgr.34107) 7848 : cluster [DBG] pgmap v8833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:50 smithi115 bash[46934]: audit 2022-04-22T21:15:49.530810+0000 mon.a (mon.0) 11427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:50 smithi115 bash[45495]: cluster 2022-04-22T21:15:49.090340+0000 mgr.y (mgr.34107) 7848 : cluster [DBG] pgmap v8833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:50 smithi115 bash[45495]: audit 2022-04-22T21:15:49.530810+0000 mon.a (mon.0) 11427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:51 smithi121 bash[36902]: audit 2022-04-22T21:15:50.744160+0000 mon.a (mon.0) 11428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:51 smithi115 bash[45495]: audit 2022-04-22T21:15:50.744160+0000 mon.a (mon.0) 11428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:51 smithi115 bash[46934]: audit 2022-04-22T21:15:50.744160+0000 mon.a (mon.0) 11428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:52 smithi121 bash[36902]: cluster 2022-04-22T21:15:51.091424+0000 mgr.y (mgr.34107) 7849 : cluster [DBG] pgmap v8834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:52 smithi115 bash[45495]: cluster 2022-04-22T21:15:51.091424+0000 mgr.y (mgr.34107) 7849 : cluster [DBG] pgmap v8834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:52.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:52 smithi115 bash[46934]: cluster 2022-04-22T21:15:51.091424+0000 mgr.y (mgr.34107) 7849 : cluster [DBG] pgmap v8834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:54 smithi121 bash[36902]: cluster 2022-04-22T21:15:53.092481+0000 mgr.y (mgr.34107) 7850 : cluster [DBG] pgmap v8835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:54 smithi121 bash[36902]: audit 2022-04-22T21:15:53.739059+0000 mon.a (mon.0) 11429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:15:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:15:54.384Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:15:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:54 smithi115 bash[45495]: cluster 2022-04-22T21:15:53.092481+0000 mgr.y (mgr.34107) 7850 : cluster [DBG] pgmap v8835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:54 smithi115 bash[45495]: audit 2022-04-22T21:15:53.739059+0000 mon.a (mon.0) 11429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:54 smithi115 bash[46934]: cluster 2022-04-22T21:15:53.092481+0000 mgr.y (mgr.34107) 7850 : cluster [DBG] pgmap v8835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:54 smithi115 bash[46934]: audit 2022-04-22T21:15:53.739059+0000 mon.a (mon.0) 11429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:55 smithi121 bash[36902]: audit 2022-04-22T21:15:54.534287+0000 mon.a (mon.0) 11430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:55 smithi115 bash[45495]: audit 2022-04-22T21:15:54.534287+0000 mon.a (mon.0) 11430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:55 smithi115 bash[46934]: audit 2022-04-22T21:15:54.534287+0000 mon.a (mon.0) 11430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:15:56.704 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:15:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:56 smithi121 bash[36902]: cluster 2022-04-22T21:15:55.093457+0000 mgr.y (mgr.34107) 7851 : cluster [DBG] pgmap v8836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:56 smithi121 bash[36902]: audit 2022-04-22T21:15:55.747030+0000 mon.a (mon.0) 11431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:56 smithi115 bash[45495]: cluster 2022-04-22T21:15:55.093457+0000 mgr.y (mgr.34107) 7851 : cluster [DBG] pgmap v8836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:56 smithi115 bash[45495]: audit 2022-04-22T21:15:55.747030+0000 mon.a (mon.0) 11431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:56 smithi115 bash[46934]: cluster 2022-04-22T21:15:55.093457+0000 mgr.y (mgr.34107) 7851 : cluster [DBG] pgmap v8836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:56 smithi115 bash[46934]: audit 2022-04-22T21:15:55.747030+0000 mon.a (mon.0) 11431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:15:57.059 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:15:57.059 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 113s ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:15:57.059 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 2m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:15:57.059 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 113s ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:15:57.059 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 113s ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 113s ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 113s ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 113s ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 113s ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 113s ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:15:57.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 113s ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 113s ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:15:57.061 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:15:57.459 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:15:57.460 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:15:57.461 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:15:57.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:57 smithi121 bash[36902]: audit 2022-04-22T21:15:56.681378+0000 mgr.y (mgr.34107) 7852 : audit [DBG] from='client.43524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:57.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:57 smithi121 bash[36902]: audit 2022-04-22T21:15:57.454505+0000 mon.c (mon.1) 284 : audit [DBG] from='client.? 172.21.15.115:0/3411073434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:57 smithi115 bash[45495]: audit 2022-04-22T21:15:56.681378+0000 mgr.y (mgr.34107) 7852 : audit [DBG] from='client.43524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:57 smithi115 bash[45495]: audit 2022-04-22T21:15:57.454505+0000 mon.c (mon.1) 284 : audit [DBG] from='client.? 172.21.15.115:0/3411073434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:57 smithi115 bash[46934]: audit 2022-04-22T21:15:56.681378+0000 mgr.y (mgr.34107) 7852 : audit [DBG] from='client.43524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:57 smithi115 bash[46934]: audit 2022-04-22T21:15:57.454505+0000 mon.c (mon.1) 284 : audit [DBG] from='client.? 172.21.15.115:0/3411073434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:15:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:58 smithi121 bash[36902]: audit 2022-04-22T21:15:57.034992+0000 mgr.y (mgr.34107) 7853 : audit [DBG] from='client.43530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:58 smithi121 bash[36902]: cluster 2022-04-22T21:15:57.094622+0000 mgr.y (mgr.34107) 7854 : cluster [DBG] pgmap v8837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:58 smithi115 bash[46934]: audit 2022-04-22T21:15:57.034992+0000 mgr.y (mgr.34107) 7853 : audit [DBG] from='client.43530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:58 smithi115 bash[46934]: cluster 2022-04-22T21:15:57.094622+0000 mgr.y (mgr.34107) 7854 : cluster [DBG] pgmap v8837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:58.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:58 smithi115 bash[45495]: audit 2022-04-22T21:15:57.034992+0000 mgr.y (mgr.34107) 7853 : audit [DBG] from='client.43530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:15:58.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:58 smithi115 bash[45495]: cluster 2022-04-22T21:15:57.094622+0000 mgr.y (mgr.34107) 7854 : cluster [DBG] pgmap v8837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:15:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:15:59 smithi121 bash[36902]: audit 2022-04-22T21:15:58.742343+0000 mon.a (mon.0) 11432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:15:59 smithi115 bash[46934]: audit 2022-04-22T21:15:58.742343+0000 mon.a (mon.0) 11432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:15:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:15:59 smithi115 bash[45495]: audit 2022-04-22T21:15:58.742343+0000 mon.a (mon.0) 11432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:00 smithi121 bash[36902]: cluster 2022-04-22T21:15:59.095742+0000 mgr.y (mgr.34107) 7855 : cluster [DBG] pgmap v8838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:00 smithi121 bash[36902]: audit 2022-04-22T21:15:59.538002+0000 mon.a (mon.0) 11433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:00 smithi115 bash[45495]: cluster 2022-04-22T21:15:59.095742+0000 mgr.y (mgr.34107) 7855 : cluster [DBG] pgmap v8838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:00 smithi115 bash[45495]: audit 2022-04-22T21:15:59.538002+0000 mon.a (mon.0) 11433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:00 smithi115 bash[46934]: cluster 2022-04-22T21:15:59.095742+0000 mgr.y (mgr.34107) 7855 : cluster [DBG] pgmap v8838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:00 smithi115 bash[46934]: audit 2022-04-22T21:15:59.538002+0000 mon.a (mon.0) 11433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:01 smithi121 bash[36902]: audit 2022-04-22T21:16:00.750069+0000 mon.a (mon.0) 11434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:01 smithi115 bash[45495]: audit 2022-04-22T21:16:00.750069+0000 mon.a (mon.0) 11434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:01 smithi115 bash[46934]: audit 2022-04-22T21:16:00.750069+0000 mon.a (mon.0) 11434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:02 smithi121 bash[36902]: cluster 2022-04-22T21:16:01.096837+0000 mgr.y (mgr.34107) 7856 : cluster [DBG] pgmap v8839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:02 smithi115 bash[46934]: cluster 2022-04-22T21:16:01.096837+0000 mgr.y (mgr.34107) 7856 : cluster [DBG] pgmap v8839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:02.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:02 smithi115 bash[45495]: cluster 2022-04-22T21:16:01.096837+0000 mgr.y (mgr.34107) 7856 : cluster [DBG] pgmap v8839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:04 smithi115 bash[45495]: cluster 2022-04-22T21:16:03.097997+0000 mgr.y (mgr.34107) 7857 : cluster [DBG] pgmap v8840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:04 smithi115 bash[45495]: audit 2022-04-22T21:16:03.745395+0000 mon.a (mon.0) 11435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:04.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:04.382Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:04 smithi115 bash[46934]: cluster 2022-04-22T21:16:03.097997+0000 mgr.y (mgr.34107) 7857 : cluster [DBG] pgmap v8840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:04 smithi115 bash[46934]: audit 2022-04-22T21:16:03.745395+0000 mon.a (mon.0) 11435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:04 smithi121 bash[36902]: cluster 2022-04-22T21:16:03.097997+0000 mgr.y (mgr.34107) 7857 : cluster [DBG] pgmap v8840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:04 smithi121 bash[36902]: audit 2022-04-22T21:16:03.745395+0000 mon.a (mon.0) 11435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:05 smithi115 bash[45495]: audit 2022-04-22T21:16:04.541605+0000 mon.a (mon.0) 11436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:05.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:05 smithi115 bash[46934]: audit 2022-04-22T21:16:04.541605+0000 mon.a (mon.0) 11436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:05 smithi121 bash[36902]: audit 2022-04-22T21:16:04.541605+0000 mon.a (mon.0) 11436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:06 smithi115 bash[46934]: cluster 2022-04-22T21:16:05.099085+0000 mgr.y (mgr.34107) 7858 : cluster [DBG] pgmap v8841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:06 smithi115 bash[46934]: audit 2022-04-22T21:16:05.753057+0000 mon.a (mon.0) 11437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:06 smithi115 bash[45495]: cluster 2022-04-22T21:16:05.099085+0000 mgr.y (mgr.34107) 7858 : cluster [DBG] pgmap v8841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:06 smithi115 bash[45495]: audit 2022-04-22T21:16:05.753057+0000 mon.a (mon.0) 11437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:06 smithi121 bash[36902]: cluster 2022-04-22T21:16:05.099085+0000 mgr.y (mgr.34107) 7858 : cluster [DBG] pgmap v8841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:06 smithi121 bash[36902]: audit 2022-04-22T21:16:05.753057+0000 mon.a (mon.0) 11437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:08 smithi115 bash[46934]: cluster 2022-04-22T21:16:07.100269+0000 mgr.y (mgr.34107) 7859 : cluster [DBG] pgmap v8842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:08 smithi115 bash[45495]: cluster 2022-04-22T21:16:07.100269+0000 mgr.y (mgr.34107) 7859 : cluster [DBG] pgmap v8842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:08 smithi121 bash[36902]: cluster 2022-04-22T21:16:07.100269+0000 mgr.y (mgr.34107) 7859 : cluster [DBG] pgmap v8842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:09 smithi115 bash[46934]: audit 2022-04-22T21:16:08.748641+0000 mon.a (mon.0) 11438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:09 smithi115 bash[45495]: audit 2022-04-22T21:16:08.748641+0000 mon.a (mon.0) 11438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:09 smithi121 bash[36902]: audit 2022-04-22T21:16:08.748641+0000 mon.a (mon.0) 11438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:10 smithi115 bash[46934]: cluster 2022-04-22T21:16:09.101267+0000 mgr.y (mgr.34107) 7860 : cluster [DBG] pgmap v8843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:10 smithi115 bash[46934]: audit 2022-04-22T21:16:09.545161+0000 mon.a (mon.0) 11439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:10 smithi115 bash[45495]: cluster 2022-04-22T21:16:09.101267+0000 mgr.y (mgr.34107) 7860 : cluster [DBG] pgmap v8843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:10 smithi115 bash[45495]: audit 2022-04-22T21:16:09.545161+0000 mon.a (mon.0) 11439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:10 smithi121 bash[36902]: cluster 2022-04-22T21:16:09.101267+0000 mgr.y (mgr.34107) 7860 : cluster [DBG] pgmap v8843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:10 smithi121 bash[36902]: audit 2022-04-22T21:16:09.545161+0000 mon.a (mon.0) 11439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:11 smithi121 bash[36902]: audit 2022-04-22T21:16:10.756212+0000 mon.a (mon.0) 11440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:11 smithi115 bash[45495]: audit 2022-04-22T21:16:10.756212+0000 mon.a (mon.0) 11440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:11 smithi115 bash[46934]: audit 2022-04-22T21:16:10.756212+0000 mon.a (mon.0) 11440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:12 smithi115 bash[45495]: cluster 2022-04-22T21:16:11.102354+0000 mgr.y (mgr.34107) 7861 : cluster [DBG] pgmap v8844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:12 smithi115 bash[46934]: cluster 2022-04-22T21:16:11.102354+0000 mgr.y (mgr.34107) 7861 : cluster [DBG] pgmap v8844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:12.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:12 smithi121 bash[36902]: cluster 2022-04-22T21:16:11.102354+0000 mgr.y (mgr.34107) 7861 : cluster [DBG] pgmap v8844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:13.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[46934]: audit 2022-04-22T21:16:13.315775+0000 mon.a (mon.0) 11441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:16:13.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[46934]: audit 2022-04-22T21:16:13.318662+0000 mon.a (mon.0) 11442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:16:13.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[46934]: audit 2022-04-22T21:16:13.320495+0000 mon.a (mon.0) 11443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:16:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[45495]: audit 2022-04-22T21:16:13.315775+0000 mon.a (mon.0) 11441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:16:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[45495]: audit 2022-04-22T21:16:13.318662+0000 mon.a (mon.0) 11442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:16:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:13 smithi115 bash[45495]: audit 2022-04-22T21:16:13.320495+0000 mon.a (mon.0) 11443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:16:13.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:13 smithi121 bash[36902]: audit 2022-04-22T21:16:13.315775+0000 mon.a (mon.0) 11441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:16:13.667 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:13 smithi121 bash[36902]: audit 2022-04-22T21:16:13.318662+0000 mon.a (mon.0) 11442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:16:13.667 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:13 smithi121 bash[36902]: audit 2022-04-22T21:16:13.320495+0000 mon.a (mon.0) 11443 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:16:14.677 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:14.384Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:15.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:14 smithi121 bash[36902]: cluster 2022-04-22T21:16:13.103547+0000 mgr.y (mgr.34107) 7862 : cluster [DBG] pgmap v8845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:14 smithi121 bash[36902]: audit 2022-04-22T21:16:13.674321+0000 mon.a (mon.0) 11444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:16:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:14 smithi121 bash[36902]: audit 2022-04-22T21:16:13.751861+0000 mon.a (mon.0) 11445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:14 smithi121 bash[36902]: audit 2022-04-22T21:16:14.548764+0000 mon.a (mon.0) 11446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[45495]: cluster 2022-04-22T21:16:13.103547+0000 mgr.y (mgr.34107) 7862 : cluster [DBG] pgmap v8845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[45495]: audit 2022-04-22T21:16:13.674321+0000 mon.a (mon.0) 11444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[45495]: audit 2022-04-22T21:16:13.751861+0000 mon.a (mon.0) 11445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[45495]: audit 2022-04-22T21:16:14.548764+0000 mon.a (mon.0) 11446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[46934]: cluster 2022-04-22T21:16:13.103547+0000 mgr.y (mgr.34107) 7862 : cluster [DBG] pgmap v8845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[46934]: audit 2022-04-22T21:16:13.674321+0000 mon.a (mon.0) 11444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[46934]: audit 2022-04-22T21:16:13.751861+0000 mon.a (mon.0) 11445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:14 smithi115 bash[46934]: audit 2022-04-22T21:16:14.548764+0000 mon.a (mon.0) 11446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:16.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:16 smithi121 bash[36902]: cluster 2022-04-22T21:16:15.104395+0000 mgr.y (mgr.34107) 7863 : cluster [DBG] pgmap v8846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:16 smithi121 bash[36902]: audit 2022-04-22T21:16:15.759166+0000 mon.a (mon.0) 11447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:16 smithi121 bash[36902]: audit 2022-04-22T21:16:15.929314+0000 mon.a (mon.0) 11448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:16:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:16 smithi121 bash[36902]: audit 2022-04-22T21:16:15.929985+0000 mon.a (mon.0) 11449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:16:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[45495]: cluster 2022-04-22T21:16:15.104395+0000 mgr.y (mgr.34107) 7863 : cluster [DBG] pgmap v8846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[45495]: audit 2022-04-22T21:16:15.759166+0000 mon.a (mon.0) 11447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[45495]: audit 2022-04-22T21:16:15.929314+0000 mon.a (mon.0) 11448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:16:16.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[45495]: audit 2022-04-22T21:16:15.929985+0000 mon.a (mon.0) 11449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:16:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[46934]: cluster 2022-04-22T21:16:15.104395+0000 mgr.y (mgr.34107) 7863 : cluster [DBG] pgmap v8846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[46934]: audit 2022-04-22T21:16:15.759166+0000 mon.a (mon.0) 11447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[46934]: audit 2022-04-22T21:16:15.929314+0000 mon.a (mon.0) 11448 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:16:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:16 smithi115 bash[46934]: audit 2022-04-22T21:16:15.929985+0000 mon.a (mon.0) 11449 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:16:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:18 smithi121 bash[36902]: cluster 2022-04-22T21:16:17.105536+0000 mgr.y (mgr.34107) 7864 : cluster [DBG] pgmap v8847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:18 smithi115 bash[46934]: cluster 2022-04-22T21:16:17.105536+0000 mgr.y (mgr.34107) 7864 : cluster [DBG] pgmap v8847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:18 smithi115 bash[45495]: cluster 2022-04-22T21:16:17.105536+0000 mgr.y (mgr.34107) 7864 : cluster [DBG] pgmap v8847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:19 smithi121 bash[36902]: audit 2022-04-22T21:16:18.754869+0000 mon.a (mon.0) 11450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:19 smithi115 bash[45495]: audit 2022-04-22T21:16:18.754869+0000 mon.a (mon.0) 11450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:19 smithi115 bash[46934]: audit 2022-04-22T21:16:18.754869+0000 mon.a (mon.0) 11450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:20 smithi121 bash[36902]: cluster 2022-04-22T21:16:19.106577+0000 mgr.y (mgr.34107) 7865 : cluster [DBG] pgmap v8848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:20 smithi121 bash[36902]: audit 2022-04-22T21:16:19.552447+0000 mon.a (mon.0) 11451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:20 smithi115 bash[45495]: cluster 2022-04-22T21:16:19.106577+0000 mgr.y (mgr.34107) 7865 : cluster [DBG] pgmap v8848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:20 smithi115 bash[45495]: audit 2022-04-22T21:16:19.552447+0000 mon.a (mon.0) 11451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:20 smithi115 bash[46934]: cluster 2022-04-22T21:16:19.106577+0000 mgr.y (mgr.34107) 7865 : cluster [DBG] pgmap v8848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:20 smithi115 bash[46934]: audit 2022-04-22T21:16:19.552447+0000 mon.a (mon.0) 11451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:21 smithi121 bash[36902]: audit 2022-04-22T21:16:20.762131+0000 mon.a (mon.0) 11452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:21 smithi115 bash[46934]: audit 2022-04-22T21:16:20.762131+0000 mon.a (mon.0) 11452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:21 smithi115 bash[45495]: audit 2022-04-22T21:16:20.762131+0000 mon.a (mon.0) 11452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:22 smithi121 bash[36902]: cluster 2022-04-22T21:16:21.107671+0000 mgr.y (mgr.34107) 7866 : cluster [DBG] pgmap v8849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:22.515 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:22 smithi115 bash[45495]: cluster 2022-04-22T21:16:21.107671+0000 mgr.y (mgr.34107) 7866 : cluster [DBG] pgmap v8849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:22.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:22 smithi115 bash[46934]: cluster 2022-04-22T21:16:21.107671+0000 mgr.y (mgr.34107) 7866 : cluster [DBG] pgmap v8849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:22.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:24 smithi121 bash[36902]: cluster 2022-04-22T21:16:23.108738+0000 mgr.y (mgr.34107) 7867 : cluster [DBG] pgmap v8850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:24 smithi121 bash[36902]: audit 2022-04-22T21:16:23.757999+0000 mon.a (mon.0) 11453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:24.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:24.383Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:24 smithi115 bash[45495]: cluster 2022-04-22T21:16:23.108738+0000 mgr.y (mgr.34107) 7867 : cluster [DBG] pgmap v8850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:24 smithi115 bash[45495]: audit 2022-04-22T21:16:23.757999+0000 mon.a (mon.0) 11453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:24 smithi115 bash[46934]: cluster 2022-04-22T21:16:23.108738+0000 mgr.y (mgr.34107) 7867 : cluster [DBG] pgmap v8850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:24 smithi115 bash[46934]: audit 2022-04-22T21:16:23.757999+0000 mon.a (mon.0) 11453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:25 smithi121 bash[36902]: audit 2022-04-22T21:16:24.556229+0000 mon.a (mon.0) 11454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:25.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:25 smithi115 bash[45495]: audit 2022-04-22T21:16:24.556229+0000 mon.a (mon.0) 11454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:25.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:25 smithi115 bash[46934]: audit 2022-04-22T21:16:24.556229+0000 mon.a (mon.0) 11454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:26 smithi121 bash[36902]: cluster 2022-04-22T21:16:25.109758+0000 mgr.y (mgr.34107) 7868 : cluster [DBG] pgmap v8851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:26 smithi121 bash[36902]: audit 2022-04-22T21:16:25.765351+0000 mon.a (mon.0) 11455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:26 smithi115 bash[45495]: cluster 2022-04-22T21:16:25.109758+0000 mgr.y (mgr.34107) 7868 : cluster [DBG] pgmap v8851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:26.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:26 smithi115 bash[45495]: audit 2022-04-22T21:16:25.765351+0000 mon.a (mon.0) 11455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:26 smithi115 bash[46934]: cluster 2022-04-22T21:16:25.109758+0000 mgr.y (mgr.34107) 7868 : cluster [DBG] pgmap v8851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:26 smithi115 bash[46934]: audit 2022-04-22T21:16:25.765351+0000 mon.a (mon.0) 11455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:27.834 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:16:28.183 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:16:28.183 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:16:28.183 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 3m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:16:28.183 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:16:28.183 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:16:28.184 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:16:28.185 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:16:28.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:16:28.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:16:28.186 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:16:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:28 smithi121 bash[36902]: cluster 2022-04-22T21:16:27.110925+0000 mgr.y (mgr.34107) 7869 : cluster [DBG] pgmap v8852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:28 smithi115 bash[46934]: cluster 2022-04-22T21:16:27.110925+0000 mgr.y (mgr.34107) 7869 : cluster [DBG] pgmap v8852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:28 smithi115 bash[45495]: cluster 2022-04-22T21:16:27.110925+0000 mgr.y (mgr.34107) 7869 : cluster [DBG] pgmap v8852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:28.580 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:16:28.580 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:16:28.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:16:28.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:28.581 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:16:28.582 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:16:28.582 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:16:28.582 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:16:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:29 smithi121 bash[36902]: audit 2022-04-22T21:16:27.815130+0000 mgr.y (mgr.34107) 7870 : audit [DBG] from='client.53473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:29 smithi121 bash[36902]: audit 2022-04-22T21:16:28.159948+0000 mgr.y (mgr.34107) 7871 : audit [DBG] from='client.53479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:29 smithi121 bash[36902]: audit 2022-04-22T21:16:28.576153+0000 mon.a (mon.0) 11456 : audit [DBG] from='client.? 172.21.15.115:0/2119974581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:16:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:29 smithi121 bash[36902]: audit 2022-04-22T21:16:28.761659+0000 mon.a (mon.0) 11457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[45495]: audit 2022-04-22T21:16:27.815130+0000 mgr.y (mgr.34107) 7870 : audit [DBG] from='client.53473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[45495]: audit 2022-04-22T21:16:28.159948+0000 mgr.y (mgr.34107) 7871 : audit [DBG] from='client.53479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[45495]: audit 2022-04-22T21:16:28.576153+0000 mon.a (mon.0) 11456 : audit [DBG] from='client.? 172.21.15.115:0/2119974581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[45495]: audit 2022-04-22T21:16:28.761659+0000 mon.a (mon.0) 11457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[46934]: audit 2022-04-22T21:16:27.815130+0000 mgr.y (mgr.34107) 7870 : audit [DBG] from='client.53473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[46934]: audit 2022-04-22T21:16:28.159948+0000 mgr.y (mgr.34107) 7871 : audit [DBG] from='client.53479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[46934]: audit 2022-04-22T21:16:28.576153+0000 mon.a (mon.0) 11456 : audit [DBG] from='client.? 172.21.15.115:0/2119974581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:16:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:29 smithi115 bash[46934]: audit 2022-04-22T21:16:28.761659+0000 mon.a (mon.0) 11457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:30 smithi121 bash[36902]: cluster 2022-04-22T21:16:29.111989+0000 mgr.y (mgr.34107) 7872 : cluster [DBG] pgmap v8853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:30.935 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:30 smithi121 bash[36902]: audit 2022-04-22T21:16:29.559695+0000 mon.a (mon.0) 11458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:30.936 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:30 smithi115 bash[45495]: cluster 2022-04-22T21:16:29.111989+0000 mgr.y (mgr.34107) 7872 : cluster [DBG] pgmap v8853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:30.936 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:30 smithi115 bash[45495]: audit 2022-04-22T21:16:29.559695+0000 mon.a (mon.0) 11458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:30.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:30 smithi115 bash[46934]: cluster 2022-04-22T21:16:29.111989+0000 mgr.y (mgr.34107) 7872 : cluster [DBG] pgmap v8853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:30.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:30 smithi115 bash[46934]: audit 2022-04-22T21:16:29.559695+0000 mon.a (mon.0) 11458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:31 smithi121 bash[36902]: audit 2022-04-22T21:16:30.768308+0000 mon.a (mon.0) 11459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:31 smithi115 bash[45495]: audit 2022-04-22T21:16:30.768308+0000 mon.a (mon.0) 11459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:31 smithi115 bash[46934]: audit 2022-04-22T21:16:30.768308+0000 mon.a (mon.0) 11459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:32.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:32 smithi121 bash[36902]: cluster 2022-04-22T21:16:31.112984+0000 mgr.y (mgr.34107) 7873 : cluster [DBG] pgmap v8854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:32.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:32 smithi115 bash[46934]: cluster 2022-04-22T21:16:31.112984+0000 mgr.y (mgr.34107) 7873 : cluster [DBG] pgmap v8854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:32.516 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:32 smithi115 bash[45495]: cluster 2022-04-22T21:16:31.112984+0000 mgr.y (mgr.34107) 7873 : cluster [DBG] pgmap v8854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:32.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:34 smithi121 bash[36902]: cluster 2022-04-22T21:16:33.114021+0000 mgr.y (mgr.34107) 7874 : cluster [DBG] pgmap v8855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:34 smithi121 bash[36902]: audit 2022-04-22T21:16:33.764881+0000 mon.a (mon.0) 11460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:34 smithi115 bash[45495]: cluster 2022-04-22T21:16:33.114021+0000 mgr.y (mgr.34107) 7874 : cluster [DBG] pgmap v8855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:34 smithi115 bash[45495]: audit 2022-04-22T21:16:33.764881+0000 mon.a (mon.0) 11460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:34.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:34.386Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:34 smithi115 bash[46934]: cluster 2022-04-22T21:16:33.114021+0000 mgr.y (mgr.34107) 7874 : cluster [DBG] pgmap v8855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:34 smithi115 bash[46934]: audit 2022-04-22T21:16:33.764881+0000 mon.a (mon.0) 11460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:35 smithi121 bash[36902]: audit 2022-04-22T21:16:34.563266+0000 mon.a (mon.0) 11461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:35 smithi115 bash[45495]: audit 2022-04-22T21:16:34.563266+0000 mon.a (mon.0) 11461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:35.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:35 smithi115 bash[46934]: audit 2022-04-22T21:16:34.563266+0000 mon.a (mon.0) 11461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:36 smithi121 bash[36902]: cluster 2022-04-22T21:16:35.114979+0000 mgr.y (mgr.34107) 7875 : cluster [DBG] pgmap v8856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:36 smithi121 bash[36902]: audit 2022-04-22T21:16:35.771489+0000 mon.a (mon.0) 11462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:36 smithi115 bash[45495]: cluster 2022-04-22T21:16:35.114979+0000 mgr.y (mgr.34107) 7875 : cluster [DBG] pgmap v8856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:36 smithi115 bash[45495]: audit 2022-04-22T21:16:35.771489+0000 mon.a (mon.0) 11462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:36 smithi115 bash[46934]: cluster 2022-04-22T21:16:35.114979+0000 mgr.y (mgr.34107) 7875 : cluster [DBG] pgmap v8856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:36 smithi115 bash[46934]: audit 2022-04-22T21:16:35.771489+0000 mon.a (mon.0) 11462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:38.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:38 smithi121 bash[36902]: cluster 2022-04-22T21:16:37.116254+0000 mgr.y (mgr.34107) 7876 : cluster [DBG] pgmap v8857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:38.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:38 smithi115 bash[45495]: cluster 2022-04-22T21:16:37.116254+0000 mgr.y (mgr.34107) 7876 : cluster [DBG] pgmap v8857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:38.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:38 smithi115 bash[46934]: cluster 2022-04-22T21:16:37.116254+0000 mgr.y (mgr.34107) 7876 : cluster [DBG] pgmap v8857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:39.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:39 smithi121 bash[36902]: audit 2022-04-22T21:16:38.768180+0000 mon.a (mon.0) 11463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:39 smithi115 bash[45495]: audit 2022-04-22T21:16:38.768180+0000 mon.a (mon.0) 11463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:39 smithi115 bash[46934]: audit 2022-04-22T21:16:38.768180+0000 mon.a (mon.0) 11463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:40.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:40 smithi121 bash[36902]: cluster 2022-04-22T21:16:39.117386+0000 mgr.y (mgr.34107) 7877 : cluster [DBG] pgmap v8858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:40 smithi121 bash[36902]: audit 2022-04-22T21:16:39.567030+0000 mon.a (mon.0) 11464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:40 smithi115 bash[45495]: cluster 2022-04-22T21:16:39.117386+0000 mgr.y (mgr.34107) 7877 : cluster [DBG] pgmap v8858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:40.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:40 smithi115 bash[45495]: audit 2022-04-22T21:16:39.567030+0000 mon.a (mon.0) 11464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:40 smithi115 bash[46934]: cluster 2022-04-22T21:16:39.117386+0000 mgr.y (mgr.34107) 7877 : cluster [DBG] pgmap v8858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:40.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:40 smithi115 bash[46934]: audit 2022-04-22T21:16:39.567030+0000 mon.a (mon.0) 11464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:41.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:41 smithi121 bash[36902]: audit 2022-04-22T21:16:40.774491+0000 mon.a (mon.0) 11465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:41 smithi115 bash[45495]: audit 2022-04-22T21:16:40.774491+0000 mon.a (mon.0) 11465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:41 smithi115 bash[46934]: audit 2022-04-22T21:16:40.774491+0000 mon.a (mon.0) 11465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:42 smithi121 bash[36902]: cluster 2022-04-22T21:16:41.118523+0000 mgr.y (mgr.34107) 7878 : cluster [DBG] pgmap v8859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:42.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:42 smithi115 bash[46934]: cluster 2022-04-22T21:16:41.118523+0000 mgr.y (mgr.34107) 7878 : cluster [DBG] pgmap v8859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:42.514 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:42 smithi115 bash[45495]: cluster 2022-04-22T21:16:41.118523+0000 mgr.y (mgr.34107) 7878 : cluster [DBG] pgmap v8859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:42.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:44 smithi121 bash[36902]: cluster 2022-04-22T21:16:43.119731+0000 mgr.y (mgr.34107) 7879 : cluster [DBG] pgmap v8860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:44 smithi121 bash[36902]: audit 2022-04-22T21:16:43.771171+0000 mon.a (mon.0) 11466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:44.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:44.386Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:44 smithi115 bash[45495]: cluster 2022-04-22T21:16:43.119731+0000 mgr.y (mgr.34107) 7879 : cluster [DBG] pgmap v8860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:44 smithi115 bash[45495]: audit 2022-04-22T21:16:43.771171+0000 mon.a (mon.0) 11466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:44 smithi115 bash[46934]: cluster 2022-04-22T21:16:43.119731+0000 mgr.y (mgr.34107) 7879 : cluster [DBG] pgmap v8860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:44 smithi115 bash[46934]: audit 2022-04-22T21:16:43.771171+0000 mon.a (mon.0) 11466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:45 smithi115 bash[45495]: audit 2022-04-22T21:16:44.570867+0000 mon.a (mon.0) 11467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:45 smithi115 bash[46934]: audit 2022-04-22T21:16:44.570867+0000 mon.a (mon.0) 11467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:45.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:45 smithi121 bash[36902]: audit 2022-04-22T21:16:44.570867+0000 mon.a (mon.0) 11467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:46 smithi115 bash[45495]: cluster 2022-04-22T21:16:45.120689+0000 mgr.y (mgr.34107) 7880 : cluster [DBG] pgmap v8861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:46 smithi115 bash[45495]: audit 2022-04-22T21:16:45.777987+0000 mon.a (mon.0) 11468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:46 smithi115 bash[46934]: cluster 2022-04-22T21:16:45.120689+0000 mgr.y (mgr.34107) 7880 : cluster [DBG] pgmap v8861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:46 smithi115 bash[46934]: audit 2022-04-22T21:16:45.777987+0000 mon.a (mon.0) 11468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:46 smithi121 bash[36902]: cluster 2022-04-22T21:16:45.120689+0000 mgr.y (mgr.34107) 7880 : cluster [DBG] pgmap v8861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:46 smithi121 bash[36902]: audit 2022-04-22T21:16:45.777987+0000 mon.a (mon.0) 11468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:48 smithi115 bash[45495]: cluster 2022-04-22T21:16:47.121883+0000 mgr.y (mgr.34107) 7881 : cluster [DBG] pgmap v8862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:48 smithi115 bash[46934]: cluster 2022-04-22T21:16:47.121883+0000 mgr.y (mgr.34107) 7881 : cluster [DBG] pgmap v8862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:48 smithi121 bash[36902]: cluster 2022-04-22T21:16:47.121883+0000 mgr.y (mgr.34107) 7881 : cluster [DBG] pgmap v8862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:49 smithi115 bash[45495]: audit 2022-04-22T21:16:48.774470+0000 mon.a (mon.0) 11469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:49 smithi115 bash[46934]: audit 2022-04-22T21:16:48.774470+0000 mon.a (mon.0) 11469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:49 smithi121 bash[36902]: audit 2022-04-22T21:16:48.774470+0000 mon.a (mon.0) 11469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:50 smithi115 bash[45495]: cluster 2022-04-22T21:16:49.122924+0000 mgr.y (mgr.34107) 7882 : cluster [DBG] pgmap v8863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:50 smithi115 bash[45495]: audit 2022-04-22T21:16:49.574844+0000 mon.a (mon.0) 11470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:50 smithi115 bash[46934]: cluster 2022-04-22T21:16:49.122924+0000 mgr.y (mgr.34107) 7882 : cluster [DBG] pgmap v8863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:50 smithi115 bash[46934]: audit 2022-04-22T21:16:49.574844+0000 mon.a (mon.0) 11470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:50 smithi121 bash[36902]: cluster 2022-04-22T21:16:49.122924+0000 mgr.y (mgr.34107) 7882 : cluster [DBG] pgmap v8863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:50 smithi121 bash[36902]: audit 2022-04-22T21:16:49.574844+0000 mon.a (mon.0) 11470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:51 smithi115 bash[45495]: audit 2022-04-22T21:16:50.781135+0000 mon.a (mon.0) 11471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:51 smithi115 bash[46934]: audit 2022-04-22T21:16:50.781135+0000 mon.a (mon.0) 11471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:51 smithi121 bash[36902]: audit 2022-04-22T21:16:50.781135+0000 mon.a (mon.0) 11471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:52 smithi115 bash[45495]: cluster 2022-04-22T21:16:51.124146+0000 mgr.y (mgr.34107) 7883 : cluster [DBG] pgmap v8864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:52.542 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:52.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:52 smithi115 bash[46934]: cluster 2022-04-22T21:16:51.124146+0000 mgr.y (mgr.34107) 7883 : cluster [DBG] pgmap v8864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:52 smithi121 bash[36902]: cluster 2022-04-22T21:16:51.124146+0000 mgr.y (mgr.34107) 7883 : cluster [DBG] pgmap v8864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:54 smithi121 bash[36902]: cluster 2022-04-22T21:16:53.125410+0000 mgr.y (mgr.34107) 7884 : cluster [DBG] pgmap v8865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:54 smithi121 bash[36902]: audit 2022-04-22T21:16:53.777776+0000 mon.a (mon.0) 11472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:16:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:16:54.387Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:16:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:54 smithi115 bash[45495]: cluster 2022-04-22T21:16:53.125410+0000 mgr.y (mgr.34107) 7884 : cluster [DBG] pgmap v8865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:54 smithi115 bash[45495]: audit 2022-04-22T21:16:53.777776+0000 mon.a (mon.0) 11472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:54 smithi115 bash[46934]: cluster 2022-04-22T21:16:53.125410+0000 mgr.y (mgr.34107) 7884 : cluster [DBG] pgmap v8865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:54 smithi115 bash[46934]: audit 2022-04-22T21:16:53.777776+0000 mon.a (mon.0) 11472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:55 smithi121 bash[36902]: audit 2022-04-22T21:16:54.578424+0000 mon.a (mon.0) 11473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:55 smithi115 bash[45495]: audit 2022-04-22T21:16:54.578424+0000 mon.a (mon.0) 11473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:55 smithi115 bash[46934]: audit 2022-04-22T21:16:54.578424+0000 mon.a (mon.0) 11473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:16:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:56 smithi121 bash[36902]: cluster 2022-04-22T21:16:55.126363+0000 mgr.y (mgr.34107) 7885 : cluster [DBG] pgmap v8866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:56 smithi121 bash[36902]: audit 2022-04-22T21:16:55.784134+0000 mon.a (mon.0) 11474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:56 smithi115 bash[45495]: cluster 2022-04-22T21:16:55.126363+0000 mgr.y (mgr.34107) 7885 : cluster [DBG] pgmap v8866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:56 smithi115 bash[45495]: audit 2022-04-22T21:16:55.784134+0000 mon.a (mon.0) 11474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:56 smithi115 bash[46934]: cluster 2022-04-22T21:16:55.126363+0000 mgr.y (mgr.34107) 7885 : cluster [DBG] pgmap v8866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:56 smithi115 bash[46934]: audit 2022-04-22T21:16:55.784134+0000 mon.a (mon.0) 11474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:16:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:58 smithi121 bash[36902]: cluster 2022-04-22T21:16:57.127569+0000 mgr.y (mgr.34107) 7886 : cluster [DBG] pgmap v8867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:58 smithi115 bash[45495]: cluster 2022-04-22T21:16:57.127569+0000 mgr.y (mgr.34107) 7886 : cluster [DBG] pgmap v8867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:58 smithi115 bash[46934]: cluster 2022-04-22T21:16:57.127569+0000 mgr.y (mgr.34107) 7886 : cluster [DBG] pgmap v8867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:16:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:16:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:16:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:16:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:16:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:16:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 3h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:16:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:16:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:16:59.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:16:59.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:16:59.701 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:16:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:16:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:16:59 smithi121 bash[36902]: audit 2022-04-22T21:16:58.780670+0000 mon.a (mon.0) 11475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:16:59 smithi115 bash[46934]: audit 2022-04-22T21:16:58.780670+0000 mon.a (mon.0) 11475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:16:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:16:59 smithi115 bash[45495]: audit 2022-04-22T21:16:58.780670+0000 mon.a (mon.0) 11475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:00 smithi121 bash[36902]: audit 2022-04-22T21:16:58.938216+0000 mgr.y (mgr.34107) 7887 : audit [DBG] from='client.43560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:00 smithi121 bash[36902]: cluster 2022-04-22T21:16:59.128611+0000 mgr.y (mgr.34107) 7888 : cluster [DBG] pgmap v8868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:00 smithi121 bash[36902]: audit 2022-04-22T21:16:59.280201+0000 mgr.y (mgr.34107) 7889 : audit [DBG] from='client.53497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:00 smithi121 bash[36902]: audit 2022-04-22T21:16:59.582179+0000 mon.a (mon.0) 11476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:00 smithi121 bash[36902]: audit 2022-04-22T21:16:59.694455+0000 mon.c (mon.1) 285 : audit [DBG] from='client.? 172.21.15.115:0/3286387358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[45495]: audit 2022-04-22T21:16:58.938216+0000 mgr.y (mgr.34107) 7887 : audit [DBG] from='client.43560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[45495]: cluster 2022-04-22T21:16:59.128611+0000 mgr.y (mgr.34107) 7888 : cluster [DBG] pgmap v8868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[45495]: audit 2022-04-22T21:16:59.280201+0000 mgr.y (mgr.34107) 7889 : audit [DBG] from='client.53497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[45495]: audit 2022-04-22T21:16:59.582179+0000 mon.a (mon.0) 11476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[45495]: audit 2022-04-22T21:16:59.694455+0000 mon.c (mon.1) 285 : audit [DBG] from='client.? 172.21.15.115:0/3286387358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[46934]: audit 2022-04-22T21:16:58.938216+0000 mgr.y (mgr.34107) 7887 : audit [DBG] from='client.43560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[46934]: cluster 2022-04-22T21:16:59.128611+0000 mgr.y (mgr.34107) 7888 : cluster [DBG] pgmap v8868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[46934]: audit 2022-04-22T21:16:59.280201+0000 mgr.y (mgr.34107) 7889 : audit [DBG] from='client.53497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[46934]: audit 2022-04-22T21:16:59.582179+0000 mon.a (mon.0) 11476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:00 smithi115 bash[46934]: audit 2022-04-22T21:16:59.694455+0000 mon.c (mon.1) 285 : audit [DBG] from='client.? 172.21.15.115:0/3286387358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:01.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:01 smithi121 bash[36902]: audit 2022-04-22T21:17:00.787311+0000 mon.a (mon.0) 11477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:01 smithi115 bash[45495]: audit 2022-04-22T21:17:00.787311+0000 mon.a (mon.0) 11477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:01 smithi115 bash[46934]: audit 2022-04-22T21:17:00.787311+0000 mon.a (mon.0) 11477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:02 smithi121 bash[36902]: cluster 2022-04-22T21:17:01.129727+0000 mgr.y (mgr.34107) 7890 : cluster [DBG] pgmap v8869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:02.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:02 smithi115 bash[45495]: cluster 2022-04-22T21:17:01.129727+0000 mgr.y (mgr.34107) 7890 : cluster [DBG] pgmap v8869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:02 smithi115 bash[46934]: cluster 2022-04-22T21:17:01.129727+0000 mgr.y (mgr.34107) 7890 : cluster [DBG] pgmap v8869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:04 smithi121 bash[36902]: cluster 2022-04-22T21:17:03.130959+0000 mgr.y (mgr.34107) 7891 : cluster [DBG] pgmap v8870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:04 smithi121 bash[36902]: audit 2022-04-22T21:17:03.783242+0000 mon.a (mon.0) 11478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:04.386Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:04 smithi115 bash[46934]: cluster 2022-04-22T21:17:03.130959+0000 mgr.y (mgr.34107) 7891 : cluster [DBG] pgmap v8870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:04 smithi115 bash[46934]: audit 2022-04-22T21:17:03.783242+0000 mon.a (mon.0) 11478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:04 smithi115 bash[45495]: cluster 2022-04-22T21:17:03.130959+0000 mgr.y (mgr.34107) 7891 : cluster [DBG] pgmap v8870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:04 smithi115 bash[45495]: audit 2022-04-22T21:17:03.783242+0000 mon.a (mon.0) 11478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:05.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:05 smithi121 bash[36902]: audit 2022-04-22T21:17:04.585732+0000 mon.a (mon.0) 11479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:05 smithi115 bash[46934]: audit 2022-04-22T21:17:04.585732+0000 mon.a (mon.0) 11479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:05 smithi115 bash[45495]: audit 2022-04-22T21:17:04.585732+0000 mon.a (mon.0) 11479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:06 smithi121 bash[36902]: cluster 2022-04-22T21:17:05.131925+0000 mgr.y (mgr.34107) 7892 : cluster [DBG] pgmap v8871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:06 smithi121 bash[36902]: audit 2022-04-22T21:17:05.790658+0000 mon.a (mon.0) 11480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:06 smithi115 bash[45495]: cluster 2022-04-22T21:17:05.131925+0000 mgr.y (mgr.34107) 7892 : cluster [DBG] pgmap v8871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:06 smithi115 bash[45495]: audit 2022-04-22T21:17:05.790658+0000 mon.a (mon.0) 11480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:06 smithi115 bash[46934]: cluster 2022-04-22T21:17:05.131925+0000 mgr.y (mgr.34107) 7892 : cluster [DBG] pgmap v8871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:06 smithi115 bash[46934]: audit 2022-04-22T21:17:05.790658+0000 mon.a (mon.0) 11480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:08 smithi121 bash[36902]: cluster 2022-04-22T21:17:07.133122+0000 mgr.y (mgr.34107) 7893 : cluster [DBG] pgmap v8872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:08 smithi115 bash[46934]: cluster 2022-04-22T21:17:07.133122+0000 mgr.y (mgr.34107) 7893 : cluster [DBG] pgmap v8872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:08 smithi115 bash[45495]: cluster 2022-04-22T21:17:07.133122+0000 mgr.y (mgr.34107) 7893 : cluster [DBG] pgmap v8872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:09 smithi121 bash[36902]: audit 2022-04-22T21:17:08.786340+0000 mon.a (mon.0) 11481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:09 smithi115 bash[45495]: audit 2022-04-22T21:17:08.786340+0000 mon.a (mon.0) 11481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:09 smithi115 bash[46934]: audit 2022-04-22T21:17:08.786340+0000 mon.a (mon.0) 11481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:10 smithi121 bash[36902]: cluster 2022-04-22T21:17:09.134186+0000 mgr.y (mgr.34107) 7894 : cluster [DBG] pgmap v8873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:10 smithi121 bash[36902]: audit 2022-04-22T21:17:09.589473+0000 mon.a (mon.0) 11482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:10 smithi115 bash[45495]: cluster 2022-04-22T21:17:09.134186+0000 mgr.y (mgr.34107) 7894 : cluster [DBG] pgmap v8873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:10 smithi115 bash[45495]: audit 2022-04-22T21:17:09.589473+0000 mon.a (mon.0) 11482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:10 smithi115 bash[46934]: cluster 2022-04-22T21:17:09.134186+0000 mgr.y (mgr.34107) 7894 : cluster [DBG] pgmap v8873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:10 smithi115 bash[46934]: audit 2022-04-22T21:17:09.589473+0000 mon.a (mon.0) 11482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:11 smithi121 bash[36902]: audit 2022-04-22T21:17:10.793669+0000 mon.a (mon.0) 11483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:11 smithi115 bash[45495]: audit 2022-04-22T21:17:10.793669+0000 mon.a (mon.0) 11483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:11 smithi115 bash[46934]: audit 2022-04-22T21:17:10.793669+0000 mon.a (mon.0) 11483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:12 smithi121 bash[36902]: cluster 2022-04-22T21:17:11.135244+0000 mgr.y (mgr.34107) 7895 : cluster [DBG] pgmap v8874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:12 smithi115 bash[45495]: cluster 2022-04-22T21:17:11.135244+0000 mgr.y (mgr.34107) 7895 : cluster [DBG] pgmap v8874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:12 smithi115 bash[46934]: cluster 2022-04-22T21:17:11.135244+0000 mgr.y (mgr.34107) 7895 : cluster [DBG] pgmap v8874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:12.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: cluster 2022-04-22T21:17:13.136440+0000 mgr.y (mgr.34107) 7896 : cluster [DBG] pgmap v8875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: audit 2022-04-22T21:17:13.684313+0000 mon.a (mon.0) 11484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:17:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: audit 2022-04-22T21:17:13.687616+0000 mon.a (mon.0) 11485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:17:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: audit 2022-04-22T21:17:13.689427+0000 mon.a (mon.0) 11486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:17:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: audit 2022-04-22T21:17:13.789620+0000 mon.a (mon.0) 11487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:14 smithi121 bash[36902]: audit 2022-04-22T21:17:14.030214+0000 mon.a (mon.0) 11488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:17:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:14.386Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: cluster 2022-04-22T21:17:13.136440+0000 mgr.y (mgr.34107) 7896 : cluster [DBG] pgmap v8875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: audit 2022-04-22T21:17:13.684313+0000 mon.a (mon.0) 11484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: audit 2022-04-22T21:17:13.687616+0000 mon.a (mon.0) 11485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: audit 2022-04-22T21:17:13.689427+0000 mon.a (mon.0) 11486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: audit 2022-04-22T21:17:13.789620+0000 mon.a (mon.0) 11487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[46934]: audit 2022-04-22T21:17:14.030214+0000 mon.a (mon.0) 11488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: cluster 2022-04-22T21:17:13.136440+0000 mgr.y (mgr.34107) 7896 : cluster [DBG] pgmap v8875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: audit 2022-04-22T21:17:13.684313+0000 mon.a (mon.0) 11484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: audit 2022-04-22T21:17:13.687616+0000 mon.a (mon.0) 11485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: audit 2022-04-22T21:17:13.689427+0000 mon.a (mon.0) 11486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: audit 2022-04-22T21:17:13.789620+0000 mon.a (mon.0) 11487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:14.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:14 smithi115 bash[45495]: audit 2022-04-22T21:17:14.030214+0000 mon.a (mon.0) 11488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:17:15.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:15 smithi121 bash[36902]: audit 2022-04-22T21:17:14.594840+0000 mon.a (mon.0) 11489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:15 smithi115 bash[45495]: audit 2022-04-22T21:17:14.594840+0000 mon.a (mon.0) 11489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:15 smithi115 bash[46934]: audit 2022-04-22T21:17:14.594840+0000 mon.a (mon.0) 11489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:16 smithi121 bash[36902]: cluster 2022-04-22T21:17:15.137456+0000 mgr.y (mgr.34107) 7897 : cluster [DBG] pgmap v8876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:16 smithi121 bash[36902]: audit 2022-04-22T21:17:15.796540+0000 mon.a (mon.0) 11490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:16 smithi121 bash[36902]: audit 2022-04-22T21:17:15.941850+0000 mon.a (mon.0) 11491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:17:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:16 smithi121 bash[36902]: audit 2022-04-22T21:17:15.942568+0000 mon.a (mon.0) 11492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:17:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[45495]: cluster 2022-04-22T21:17:15.137456+0000 mgr.y (mgr.34107) 7897 : cluster [DBG] pgmap v8876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[45495]: audit 2022-04-22T21:17:15.796540+0000 mon.a (mon.0) 11490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[45495]: audit 2022-04-22T21:17:15.941850+0000 mon.a (mon.0) 11491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:17:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[45495]: audit 2022-04-22T21:17:15.942568+0000 mon.a (mon.0) 11492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:17:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[46934]: cluster 2022-04-22T21:17:15.137456+0000 mgr.y (mgr.34107) 7897 : cluster [DBG] pgmap v8876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[46934]: audit 2022-04-22T21:17:15.796540+0000 mon.a (mon.0) 11490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[46934]: audit 2022-04-22T21:17:15.941850+0000 mon.a (mon.0) 11491 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:17:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:16 smithi115 bash[46934]: audit 2022-04-22T21:17:15.942568+0000 mon.a (mon.0) 11492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:17:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:18 smithi121 bash[36902]: cluster 2022-04-22T21:17:17.138644+0000 mgr.y (mgr.34107) 7898 : cluster [DBG] pgmap v8877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:18 smithi115 bash[45495]: cluster 2022-04-22T21:17:17.138644+0000 mgr.y (mgr.34107) 7898 : cluster [DBG] pgmap v8877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:18 smithi115 bash[46934]: cluster 2022-04-22T21:17:17.138644+0000 mgr.y (mgr.34107) 7898 : cluster [DBG] pgmap v8877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:19 smithi121 bash[36902]: audit 2022-04-22T21:17:18.792799+0000 mon.a (mon.0) 11493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:19 smithi115 bash[45495]: audit 2022-04-22T21:17:18.792799+0000 mon.a (mon.0) 11493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:19 smithi115 bash[46934]: audit 2022-04-22T21:17:18.792799+0000 mon.a (mon.0) 11493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:20 smithi121 bash[36902]: cluster 2022-04-22T21:17:19.139720+0000 mgr.y (mgr.34107) 7899 : cluster [DBG] pgmap v8878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:20 smithi121 bash[36902]: audit 2022-04-22T21:17:19.598712+0000 mon.a (mon.0) 11494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:20 smithi115 bash[45495]: cluster 2022-04-22T21:17:19.139720+0000 mgr.y (mgr.34107) 7899 : cluster [DBG] pgmap v8878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:20 smithi115 bash[45495]: audit 2022-04-22T21:17:19.598712+0000 mon.a (mon.0) 11494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:20 smithi115 bash[46934]: cluster 2022-04-22T21:17:19.139720+0000 mgr.y (mgr.34107) 7899 : cluster [DBG] pgmap v8878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:20 smithi115 bash[46934]: audit 2022-04-22T21:17:19.598712+0000 mon.a (mon.0) 11494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:21 smithi121 bash[36902]: audit 2022-04-22T21:17:20.799597+0000 mon.a (mon.0) 11495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:21 smithi115 bash[45495]: audit 2022-04-22T21:17:20.799597+0000 mon.a (mon.0) 11495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:21 smithi115 bash[46934]: audit 2022-04-22T21:17:20.799597+0000 mon.a (mon.0) 11495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:22 smithi121 bash[36902]: cluster 2022-04-22T21:17:21.140799+0000 mgr.y (mgr.34107) 7900 : cluster [DBG] pgmap v8879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:22 smithi115 bash[45495]: cluster 2022-04-22T21:17:21.140799+0000 mgr.y (mgr.34107) 7900 : cluster [DBG] pgmap v8879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:22 smithi115 bash[46934]: cluster 2022-04-22T21:17:21.140799+0000 mgr.y (mgr.34107) 7900 : cluster [DBG] pgmap v8879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:22.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:24 smithi121 bash[36902]: cluster 2022-04-22T21:17:23.141918+0000 mgr.y (mgr.34107) 7901 : cluster [DBG] pgmap v8880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:24.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:24 smithi121 bash[36902]: audit 2022-04-22T21:17:23.795836+0000 mon.a (mon.0) 11496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:24.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:24 smithi115 bash[45495]: cluster 2022-04-22T21:17:23.141918+0000 mgr.y (mgr.34107) 7901 : cluster [DBG] pgmap v8880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:24 smithi115 bash[45495]: audit 2022-04-22T21:17:23.795836+0000 mon.a (mon.0) 11496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:24 smithi115 bash[46934]: cluster 2022-04-22T21:17:23.141918+0000 mgr.y (mgr.34107) 7901 : cluster [DBG] pgmap v8880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:24 smithi115 bash[46934]: audit 2022-04-22T21:17:23.795836+0000 mon.a (mon.0) 11496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:25.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:25 smithi121 bash[36902]: audit 2022-04-22T21:17:24.602442+0000 mon.a (mon.0) 11497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:25 smithi115 bash[45495]: audit 2022-04-22T21:17:24.602442+0000 mon.a (mon.0) 11497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:25 smithi115 bash[46934]: audit 2022-04-22T21:17:24.602442+0000 mon.a (mon.0) 11497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:26 smithi121 bash[36902]: cluster 2022-04-22T21:17:25.142940+0000 mgr.y (mgr.34107) 7902 : cluster [DBG] pgmap v8881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:26 smithi121 bash[36902]: audit 2022-04-22T21:17:25.802445+0000 mon.a (mon.0) 11498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:26 smithi115 bash[45495]: cluster 2022-04-22T21:17:25.142940+0000 mgr.y (mgr.34107) 7902 : cluster [DBG] pgmap v8881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:26 smithi115 bash[45495]: audit 2022-04-22T21:17:25.802445+0000 mon.a (mon.0) 11498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:26 smithi115 bash[46934]: cluster 2022-04-22T21:17:25.142940+0000 mgr.y (mgr.34107) 7902 : cluster [DBG] pgmap v8881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:26 smithi115 bash[46934]: audit 2022-04-22T21:17:25.802445+0000 mon.a (mon.0) 11498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:28 smithi121 bash[36902]: cluster 2022-04-22T21:17:27.144284+0000 mgr.y (mgr.34107) 7903 : cluster [DBG] pgmap v8882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:28 smithi115 bash[45495]: cluster 2022-04-22T21:17:27.144284+0000 mgr.y (mgr.34107) 7903 : cluster [DBG] pgmap v8882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:28 smithi115 bash[46934]: cluster 2022-04-22T21:17:27.144284+0000 mgr.y (mgr.34107) 7903 : cluster [DBG] pgmap v8882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:29.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:29 smithi121 bash[36902]: audit 2022-04-22T21:17:28.798968+0000 mon.a (mon.0) 11499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:29 smithi115 bash[45495]: audit 2022-04-22T21:17:28.798968+0000 mon.a (mon.0) 11499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:29 smithi115 bash[46934]: audit 2022-04-22T21:17:28.798968+0000 mon.a (mon.0) 11499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:30.075 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:17:30.421 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:17:30.421 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:17:30.421 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 4m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 3h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 3h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:17:30.422 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:17:30.423 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:17:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:30 smithi121 bash[36902]: cluster 2022-04-22T21:17:29.145334+0000 mgr.y (mgr.34107) 7904 : cluster [DBG] pgmap v8883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:30 smithi121 bash[36902]: audit 2022-04-22T21:17:29.606158+0000 mon.a (mon.0) 11500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:30 smithi115 bash[45495]: cluster 2022-04-22T21:17:29.145334+0000 mgr.y (mgr.34107) 7904 : cluster [DBG] pgmap v8883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:30.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:30 smithi115 bash[45495]: audit 2022-04-22T21:17:29.606158+0000 mon.a (mon.0) 11500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:30 smithi115 bash[46934]: cluster 2022-04-22T21:17:29.145334+0000 mgr.y (mgr.34107) 7904 : cluster [DBG] pgmap v8883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:30 smithi115 bash[46934]: audit 2022-04-22T21:17:29.606158+0000 mon.a (mon.0) 11500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:30.832 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:17:30.833 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:17:30.833 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:17:30.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:17:30.833 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:17:30.834 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:17:30.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:17:30.834 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:17:30.834 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:17:30.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:17:30.835 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:17:30.835 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:17:30.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:17:30.836 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:17:30.836 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:17:30.836 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:17:30.836 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:17:31.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:31 smithi121 bash[36902]: audit 2022-04-22T21:17:30.055730+0000 mgr.y (mgr.34107) 7905 : audit [DBG] from='client.53509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:31 smithi121 bash[36902]: audit 2022-04-22T21:17:30.397373+0000 mgr.y (mgr.34107) 7906 : audit [DBG] from='client.53515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:31 smithi121 bash[36902]: audit 2022-04-22T21:17:30.805096+0000 mon.a (mon.0) 11501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:31 smithi121 bash[36902]: audit 2022-04-22T21:17:30.828124+0000 mon.a (mon.0) 11502 : audit [DBG] from='client.? 172.21.15.115:0/2351042464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[45495]: audit 2022-04-22T21:17:30.055730+0000 mgr.y (mgr.34107) 7905 : audit [DBG] from='client.53509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[45495]: audit 2022-04-22T21:17:30.397373+0000 mgr.y (mgr.34107) 7906 : audit [DBG] from='client.53515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[45495]: audit 2022-04-22T21:17:30.805096+0000 mon.a (mon.0) 11501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[45495]: audit 2022-04-22T21:17:30.828124+0000 mon.a (mon.0) 11502 : audit [DBG] from='client.? 172.21.15.115:0/2351042464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[46934]: audit 2022-04-22T21:17:30.055730+0000 mgr.y (mgr.34107) 7905 : audit [DBG] from='client.53509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[46934]: audit 2022-04-22T21:17:30.397373+0000 mgr.y (mgr.34107) 7906 : audit [DBG] from='client.53515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:17:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[46934]: audit 2022-04-22T21:17:30.805096+0000 mon.a (mon.0) 11501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:31 smithi115 bash[46934]: audit 2022-04-22T21:17:30.828124+0000 mon.a (mon.0) 11502 : audit [DBG] from='client.? 172.21.15.115:0/2351042464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:17:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:32 smithi121 bash[36902]: cluster 2022-04-22T21:17:31.146460+0000 mgr.y (mgr.34107) 7907 : cluster [DBG] pgmap v8884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:32 smithi115 bash[45495]: cluster 2022-04-22T21:17:31.146460+0000 mgr.y (mgr.34107) 7907 : cluster [DBG] pgmap v8884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:32 smithi115 bash[46934]: cluster 2022-04-22T21:17:31.146460+0000 mgr.y (mgr.34107) 7907 : cluster [DBG] pgmap v8884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:32.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:34.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:34 smithi121 bash[36902]: cluster 2022-04-22T21:17:33.147624+0000 mgr.y (mgr.34107) 7908 : cluster [DBG] pgmap v8885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:34 smithi121 bash[36902]: audit 2022-04-22T21:17:33.801994+0000 mon.a (mon.0) 11503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:34.388Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:34 smithi115 bash[45495]: cluster 2022-04-22T21:17:33.147624+0000 mgr.y (mgr.34107) 7908 : cluster [DBG] pgmap v8885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:34 smithi115 bash[45495]: audit 2022-04-22T21:17:33.801994+0000 mon.a (mon.0) 11503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:34 smithi115 bash[46934]: cluster 2022-04-22T21:17:33.147624+0000 mgr.y (mgr.34107) 7908 : cluster [DBG] pgmap v8885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:34 smithi115 bash[46934]: audit 2022-04-22T21:17:33.801994+0000 mon.a (mon.0) 11503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:35.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:35 smithi121 bash[36902]: audit 2022-04-22T21:17:34.609929+0000 mon.a (mon.0) 11504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:35 smithi115 bash[45495]: audit 2022-04-22T21:17:34.609929+0000 mon.a (mon.0) 11504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:35 smithi115 bash[46934]: audit 2022-04-22T21:17:34.609929+0000 mon.a (mon.0) 11504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:36.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:36 smithi121 bash[36902]: cluster 2022-04-22T21:17:35.148672+0000 mgr.y (mgr.34107) 7909 : cluster [DBG] pgmap v8886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:36 smithi121 bash[36902]: audit 2022-04-22T21:17:35.807455+0000 mon.a (mon.0) 11505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:36 smithi115 bash[45495]: cluster 2022-04-22T21:17:35.148672+0000 mgr.y (mgr.34107) 7909 : cluster [DBG] pgmap v8886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:36 smithi115 bash[45495]: audit 2022-04-22T21:17:35.807455+0000 mon.a (mon.0) 11505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:36 smithi115 bash[46934]: cluster 2022-04-22T21:17:35.148672+0000 mgr.y (mgr.34107) 7909 : cluster [DBG] pgmap v8886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:36 smithi115 bash[46934]: audit 2022-04-22T21:17:35.807455+0000 mon.a (mon.0) 11505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:38.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:38 smithi121 bash[36902]: cluster 2022-04-22T21:17:37.149890+0000 mgr.y (mgr.34107) 7910 : cluster [DBG] pgmap v8887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:38 smithi115 bash[45495]: cluster 2022-04-22T21:17:37.149890+0000 mgr.y (mgr.34107) 7910 : cluster [DBG] pgmap v8887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:38 smithi115 bash[46934]: cluster 2022-04-22T21:17:37.149890+0000 mgr.y (mgr.34107) 7910 : cluster [DBG] pgmap v8887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:39 smithi121 bash[36902]: audit 2022-04-22T21:17:38.804733+0000 mon.a (mon.0) 11506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:39 smithi115 bash[45495]: audit 2022-04-22T21:17:38.804733+0000 mon.a (mon.0) 11506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:39 smithi115 bash[46934]: audit 2022-04-22T21:17:38.804733+0000 mon.a (mon.0) 11506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:40 smithi121 bash[36902]: cluster 2022-04-22T21:17:39.150953+0000 mgr.y (mgr.34107) 7911 : cluster [DBG] pgmap v8888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:40 smithi121 bash[36902]: audit 2022-04-22T21:17:39.613410+0000 mon.a (mon.0) 11507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:40 smithi115 bash[45495]: cluster 2022-04-22T21:17:39.150953+0000 mgr.y (mgr.34107) 7911 : cluster [DBG] pgmap v8888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:40 smithi115 bash[45495]: audit 2022-04-22T21:17:39.613410+0000 mon.a (mon.0) 11507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:40 smithi115 bash[46934]: cluster 2022-04-22T21:17:39.150953+0000 mgr.y (mgr.34107) 7911 : cluster [DBG] pgmap v8888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:40 smithi115 bash[46934]: audit 2022-04-22T21:17:39.613410+0000 mon.a (mon.0) 11507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:41.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:41 smithi121 bash[36902]: audit 2022-04-22T21:17:40.810603+0000 mon.a (mon.0) 11508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:41 smithi115 bash[45495]: audit 2022-04-22T21:17:40.810603+0000 mon.a (mon.0) 11508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:41 smithi115 bash[46934]: audit 2022-04-22T21:17:40.810603+0000 mon.a (mon.0) 11508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:42 smithi121 bash[36902]: cluster 2022-04-22T21:17:41.152054+0000 mgr.y (mgr.34107) 7912 : cluster [DBG] pgmap v8889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:42 smithi115 bash[45495]: cluster 2022-04-22T21:17:41.152054+0000 mgr.y (mgr.34107) 7912 : cluster [DBG] pgmap v8889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:42 smithi115 bash[46934]: cluster 2022-04-22T21:17:41.152054+0000 mgr.y (mgr.34107) 7912 : cluster [DBG] pgmap v8889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:42.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:44 smithi121 bash[36902]: cluster 2022-04-22T21:17:43.153212+0000 mgr.y (mgr.34107) 7913 : cluster [DBG] pgmap v8890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:44 smithi121 bash[36902]: audit 2022-04-22T21:17:43.807112+0000 mon.a (mon.0) 11509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:44.390Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:44 smithi115 bash[45495]: cluster 2022-04-22T21:17:43.153212+0000 mgr.y (mgr.34107) 7913 : cluster [DBG] pgmap v8890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:44 smithi115 bash[45495]: audit 2022-04-22T21:17:43.807112+0000 mon.a (mon.0) 11509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:44 smithi115 bash[46934]: cluster 2022-04-22T21:17:43.153212+0000 mgr.y (mgr.34107) 7913 : cluster [DBG] pgmap v8890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:44 smithi115 bash[46934]: audit 2022-04-22T21:17:43.807112+0000 mon.a (mon.0) 11509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:45.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:45 smithi121 bash[36902]: audit 2022-04-22T21:17:44.617529+0000 mon.a (mon.0) 11510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:45 smithi115 bash[45495]: audit 2022-04-22T21:17:44.617529+0000 mon.a (mon.0) 11510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:45 smithi115 bash[46934]: audit 2022-04-22T21:17:44.617529+0000 mon.a (mon.0) 11510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:46 smithi121 bash[36902]: cluster 2022-04-22T21:17:45.154128+0000 mgr.y (mgr.34107) 7914 : cluster [DBG] pgmap v8891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:46 smithi121 bash[36902]: audit 2022-04-22T21:17:45.813665+0000 mon.a (mon.0) 11511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:46 smithi115 bash[45495]: cluster 2022-04-22T21:17:45.154128+0000 mgr.y (mgr.34107) 7914 : cluster [DBG] pgmap v8891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:46 smithi115 bash[45495]: audit 2022-04-22T21:17:45.813665+0000 mon.a (mon.0) 11511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:46 smithi115 bash[46934]: cluster 2022-04-22T21:17:45.154128+0000 mgr.y (mgr.34107) 7914 : cluster [DBG] pgmap v8891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:46 smithi115 bash[46934]: audit 2022-04-22T21:17:45.813665+0000 mon.a (mon.0) 11511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:48 smithi121 bash[36902]: cluster 2022-04-22T21:17:47.155251+0000 mgr.y (mgr.34107) 7915 : cluster [DBG] pgmap v8892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:48 smithi115 bash[45495]: cluster 2022-04-22T21:17:47.155251+0000 mgr.y (mgr.34107) 7915 : cluster [DBG] pgmap v8892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:48 smithi115 bash[46934]: cluster 2022-04-22T21:17:47.155251+0000 mgr.y (mgr.34107) 7915 : cluster [DBG] pgmap v8892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:49 smithi121 bash[36902]: audit 2022-04-22T21:17:48.810356+0000 mon.a (mon.0) 11512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:49 smithi115 bash[45495]: audit 2022-04-22T21:17:48.810356+0000 mon.a (mon.0) 11512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:49 smithi115 bash[46934]: audit 2022-04-22T21:17:48.810356+0000 mon.a (mon.0) 11512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:50 smithi121 bash[36902]: cluster 2022-04-22T21:17:49.156379+0000 mgr.y (mgr.34107) 7916 : cluster [DBG] pgmap v8893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:50 smithi121 bash[36902]: audit 2022-04-22T21:17:49.620838+0000 mon.a (mon.0) 11513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:50 smithi115 bash[45495]: cluster 2022-04-22T21:17:49.156379+0000 mgr.y (mgr.34107) 7916 : cluster [DBG] pgmap v8893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:50 smithi115 bash[45495]: audit 2022-04-22T21:17:49.620838+0000 mon.a (mon.0) 11513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:50 smithi115 bash[46934]: cluster 2022-04-22T21:17:49.156379+0000 mgr.y (mgr.34107) 7916 : cluster [DBG] pgmap v8893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:50 smithi115 bash[46934]: audit 2022-04-22T21:17:49.620838+0000 mon.a (mon.0) 11513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:51 smithi121 bash[36902]: audit 2022-04-22T21:17:50.816682+0000 mon.a (mon.0) 11514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:51 smithi115 bash[45495]: audit 2022-04-22T21:17:50.816682+0000 mon.a (mon.0) 11514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:51 smithi115 bash[46934]: audit 2022-04-22T21:17:50.816682+0000 mon.a (mon.0) 11514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:52 smithi121 bash[36902]: cluster 2022-04-22T21:17:51.157530+0000 mgr.y (mgr.34107) 7917 : cluster [DBG] pgmap v8894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:52.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:52 smithi115 bash[45495]: cluster 2022-04-22T21:17:51.157530+0000 mgr.y (mgr.34107) 7917 : cluster [DBG] pgmap v8894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:52.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:52 smithi115 bash[46934]: cluster 2022-04-22T21:17:51.157530+0000 mgr.y (mgr.34107) 7917 : cluster [DBG] pgmap v8894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:54 smithi121 bash[36902]: cluster 2022-04-22T21:17:53.158757+0000 mgr.y (mgr.34107) 7918 : cluster [DBG] pgmap v8895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:54 smithi121 bash[36902]: audit 2022-04-22T21:17:53.813367+0000 mon.a (mon.0) 11515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:17:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:17:54.390Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:17:54.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:54 smithi115 bash[45495]: cluster 2022-04-22T21:17:53.158757+0000 mgr.y (mgr.34107) 7918 : cluster [DBG] pgmap v8895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:54.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:54 smithi115 bash[45495]: audit 2022-04-22T21:17:53.813367+0000 mon.a (mon.0) 11515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:54.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:54 smithi115 bash[46934]: cluster 2022-04-22T21:17:53.158757+0000 mgr.y (mgr.34107) 7918 : cluster [DBG] pgmap v8895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:54.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:54 smithi115 bash[46934]: audit 2022-04-22T21:17:53.813367+0000 mon.a (mon.0) 11515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:55 smithi115 bash[45495]: audit 2022-04-22T21:17:54.624259+0000 mon.a (mon.0) 11516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:55 smithi115 bash[46934]: audit 2022-04-22T21:17:54.624259+0000 mon.a (mon.0) 11516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:55 smithi121 bash[36902]: audit 2022-04-22T21:17:54.624259+0000 mon.a (mon.0) 11516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:17:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:56 smithi115 bash[45495]: cluster 2022-04-22T21:17:55.159790+0000 mgr.y (mgr.34107) 7919 : cluster [DBG] pgmap v8896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:56 smithi115 bash[45495]: audit 2022-04-22T21:17:55.819802+0000 mon.a (mon.0) 11517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:56 smithi115 bash[46934]: cluster 2022-04-22T21:17:55.159790+0000 mgr.y (mgr.34107) 7919 : cluster [DBG] pgmap v8896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:56 smithi115 bash[46934]: audit 2022-04-22T21:17:55.819802+0000 mon.a (mon.0) 11517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:56 smithi121 bash[36902]: cluster 2022-04-22T21:17:55.159790+0000 mgr.y (mgr.34107) 7919 : cluster [DBG] pgmap v8896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:56 smithi121 bash[36902]: audit 2022-04-22T21:17:55.819802+0000 mon.a (mon.0) 11517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:17:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:58 smithi115 bash[45495]: cluster 2022-04-22T21:17:57.160937+0000 mgr.y (mgr.34107) 7920 : cluster [DBG] pgmap v8897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:58.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:58 smithi115 bash[46934]: cluster 2022-04-22T21:17:57.160937+0000 mgr.y (mgr.34107) 7920 : cluster [DBG] pgmap v8897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:58 smithi121 bash[36902]: cluster 2022-04-22T21:17:57.160937+0000 mgr.y (mgr.34107) 7920 : cluster [DBG] pgmap v8897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:17:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:17:59 smithi115 bash[45495]: audit 2022-04-22T21:17:58.816334+0000 mon.a (mon.0) 11518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:17:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:17:59 smithi115 bash[46934]: audit 2022-04-22T21:17:58.816334+0000 mon.a (mon.0) 11518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:17:59 smithi121 bash[36902]: audit 2022-04-22T21:17:58.816334+0000 mon.a (mon.0) 11518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:00 smithi115 bash[45495]: cluster 2022-04-22T21:17:59.161955+0000 mgr.y (mgr.34107) 7921 : cluster [DBG] pgmap v8898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:00 smithi115 bash[45495]: audit 2022-04-22T21:17:59.628192+0000 mon.a (mon.0) 11519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:00 smithi115 bash[46934]: cluster 2022-04-22T21:17:59.161955+0000 mgr.y (mgr.34107) 7921 : cluster [DBG] pgmap v8898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:00 smithi115 bash[46934]: audit 2022-04-22T21:17:59.628192+0000 mon.a (mon.0) 11519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:00 smithi121 bash[36902]: cluster 2022-04-22T21:17:59.161955+0000 mgr.y (mgr.34107) 7921 : cluster [DBG] pgmap v8898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:00 smithi121 bash[36902]: audit 2022-04-22T21:17:59.628192+0000 mon.a (mon.0) 11519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:01.205 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 3m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:18:01.545 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 3h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:18:01.546 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:18:01.547 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:18:01.547 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:18:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:01 smithi115 bash[45495]: audit 2022-04-22T21:18:00.822896+0000 mon.a (mon.0) 11520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:01 smithi115 bash[46934]: audit 2022-04-22T21:18:00.822896+0000 mon.a (mon.0) 11520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:01.944 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:18:01.944 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:18:01.944 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:18:01.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:01.944 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:18:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:18:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:18:01.946 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:18:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:01 smithi121 bash[36902]: audit 2022-04-22T21:18:00.822896+0000 mon.a (mon.0) 11520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:02.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[45495]: cluster 2022-04-22T21:18:01.163017+0000 mgr.y (mgr.34107) 7922 : cluster [DBG] pgmap v8899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[45495]: audit 2022-04-22T21:18:01.183648+0000 mgr.y (mgr.34107) 7923 : audit [DBG] from='client.53527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[45495]: audit 2022-04-22T21:18:01.519277+0000 mgr.y (mgr.34107) 7924 : audit [DBG] from='client.43602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[45495]: audit 2022-04-22T21:18:01.939998+0000 mon.a (mon.0) 11521 : audit [DBG] from='client.? 172.21.15.115:0/3854224446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[46934]: cluster 2022-04-22T21:18:01.163017+0000 mgr.y (mgr.34107) 7922 : cluster [DBG] pgmap v8899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[46934]: audit 2022-04-22T21:18:01.183648+0000 mgr.y (mgr.34107) 7923 : audit [DBG] from='client.53527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[46934]: audit 2022-04-22T21:18:01.519277+0000 mgr.y (mgr.34107) 7924 : audit [DBG] from='client.43602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:02 smithi115 bash[46934]: audit 2022-04-22T21:18:01.939998+0000 mon.a (mon.0) 11521 : audit [DBG] from='client.? 172.21.15.115:0/3854224446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:02 smithi121 bash[36902]: cluster 2022-04-22T21:18:01.163017+0000 mgr.y (mgr.34107) 7922 : cluster [DBG] pgmap v8899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:02 smithi121 bash[36902]: audit 2022-04-22T21:18:01.183648+0000 mgr.y (mgr.34107) 7923 : audit [DBG] from='client.53527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:02 smithi121 bash[36902]: audit 2022-04-22T21:18:01.519277+0000 mgr.y (mgr.34107) 7924 : audit [DBG] from='client.43602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:02 smithi121 bash[36902]: audit 2022-04-22T21:18:01.939998+0000 mon.a (mon.0) 11521 : audit [DBG] from='client.? 172.21.15.115:0/3854224446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:04.391Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:04 smithi115 bash[45495]: cluster 2022-04-22T21:18:03.164154+0000 mgr.y (mgr.34107) 7925 : cluster [DBG] pgmap v8900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:04 smithi115 bash[45495]: audit 2022-04-22T21:18:03.819458+0000 mon.a (mon.0) 11522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:04 smithi115 bash[46934]: cluster 2022-04-22T21:18:03.164154+0000 mgr.y (mgr.34107) 7925 : cluster [DBG] pgmap v8900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:04 smithi115 bash[46934]: audit 2022-04-22T21:18:03.819458+0000 mon.a (mon.0) 11522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:04 smithi121 bash[36902]: cluster 2022-04-22T21:18:03.164154+0000 mgr.y (mgr.34107) 7925 : cluster [DBG] pgmap v8900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:04 smithi121 bash[36902]: audit 2022-04-22T21:18:03.819458+0000 mon.a (mon.0) 11522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:05 smithi121 bash[36902]: audit 2022-04-22T21:18:04.631986+0000 mon.a (mon.0) 11523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:05 smithi115 bash[45495]: audit 2022-04-22T21:18:04.631986+0000 mon.a (mon.0) 11523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:05 smithi115 bash[46934]: audit 2022-04-22T21:18:04.631986+0000 mon.a (mon.0) 11523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:06 smithi121 bash[36902]: cluster 2022-04-22T21:18:05.164797+0000 mgr.y (mgr.34107) 7926 : cluster [DBG] pgmap v8901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:06 smithi121 bash[36902]: audit 2022-04-22T21:18:05.825837+0000 mon.a (mon.0) 11524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:06 smithi115 bash[45495]: cluster 2022-04-22T21:18:05.164797+0000 mgr.y (mgr.34107) 7926 : cluster [DBG] pgmap v8901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:06 smithi115 bash[45495]: audit 2022-04-22T21:18:05.825837+0000 mon.a (mon.0) 11524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:06 smithi115 bash[46934]: cluster 2022-04-22T21:18:05.164797+0000 mgr.y (mgr.34107) 7926 : cluster [DBG] pgmap v8901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:06 smithi115 bash[46934]: audit 2022-04-22T21:18:05.825837+0000 mon.a (mon.0) 11524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:08 smithi121 bash[36902]: cluster 2022-04-22T21:18:07.166022+0000 mgr.y (mgr.34107) 7927 : cluster [DBG] pgmap v8902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:08 smithi115 bash[46934]: cluster 2022-04-22T21:18:07.166022+0000 mgr.y (mgr.34107) 7927 : cluster [DBG] pgmap v8902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:08 smithi115 bash[45495]: cluster 2022-04-22T21:18:07.166022+0000 mgr.y (mgr.34107) 7927 : cluster [DBG] pgmap v8902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:09 smithi121 bash[36902]: audit 2022-04-22T21:18:08.822635+0000 mon.a (mon.0) 11525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:09 smithi115 bash[45495]: audit 2022-04-22T21:18:08.822635+0000 mon.a (mon.0) 11525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:09 smithi115 bash[46934]: audit 2022-04-22T21:18:08.822635+0000 mon.a (mon.0) 11525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:10 smithi121 bash[36902]: cluster 2022-04-22T21:18:09.167023+0000 mgr.y (mgr.34107) 7928 : cluster [DBG] pgmap v8903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:10 smithi121 bash[36902]: audit 2022-04-22T21:18:09.635748+0000 mon.a (mon.0) 11526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:10 smithi115 bash[46934]: cluster 2022-04-22T21:18:09.167023+0000 mgr.y (mgr.34107) 7928 : cluster [DBG] pgmap v8903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:10 smithi115 bash[46934]: audit 2022-04-22T21:18:09.635748+0000 mon.a (mon.0) 11526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:10 smithi115 bash[45495]: cluster 2022-04-22T21:18:09.167023+0000 mgr.y (mgr.34107) 7928 : cluster [DBG] pgmap v8903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:11.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:10 smithi115 bash[45495]: audit 2022-04-22T21:18:09.635748+0000 mon.a (mon.0) 11526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:11 smithi121 bash[36902]: audit 2022-04-22T21:18:10.829137+0000 mon.a (mon.0) 11527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:11 smithi115 bash[46934]: audit 2022-04-22T21:18:10.829137+0000 mon.a (mon.0) 11527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:12.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:11 smithi115 bash[45495]: audit 2022-04-22T21:18:10.829137+0000 mon.a (mon.0) 11527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:12.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:12 smithi115 bash[45495]: cluster 2022-04-22T21:18:11.168045+0000 mgr.y (mgr.34107) 7929 : cluster [DBG] pgmap v8904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:12 smithi115 bash[46934]: cluster 2022-04-22T21:18:11.168045+0000 mgr.y (mgr.34107) 7929 : cluster [DBG] pgmap v8904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:12 smithi121 bash[36902]: cluster 2022-04-22T21:18:11.168045+0000 mgr.y (mgr.34107) 7929 : cluster [DBG] pgmap v8904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: cluster 2022-04-22T21:18:13.169214+0000 mgr.y (mgr.34107) 7930 : cluster [DBG] pgmap v8905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: audit 2022-04-22T21:18:13.825763+0000 mon.a (mon.0) 11528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: audit 2022-04-22T21:18:14.038431+0000 mon.a (mon.0) 11529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:18:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: audit 2022-04-22T21:18:14.041442+0000 mon.a (mon.0) 11530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:18:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: audit 2022-04-22T21:18:14.043276+0000 mon.a (mon.0) 11531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:18:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:14 smithi121 bash[36902]: audit 2022-04-22T21:18:14.420487+0000 mon.a (mon.0) 11532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:18:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: cluster 2022-04-22T21:18:13.169214+0000 mgr.y (mgr.34107) 7930 : cluster [DBG] pgmap v8905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: audit 2022-04-22T21:18:13.825763+0000 mon.a (mon.0) 11528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: audit 2022-04-22T21:18:14.038431+0000 mon.a (mon.0) 11529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:18:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: audit 2022-04-22T21:18:14.041442+0000 mon.a (mon.0) 11530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:18:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: audit 2022-04-22T21:18:14.043276+0000 mon.a (mon.0) 11531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:18:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[45495]: audit 2022-04-22T21:18:14.420487+0000 mon.a (mon.0) 11532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:18:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:14.409Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: cluster 2022-04-22T21:18:13.169214+0000 mgr.y (mgr.34107) 7930 : cluster [DBG] pgmap v8905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: audit 2022-04-22T21:18:13.825763+0000 mon.a (mon.0) 11528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: audit 2022-04-22T21:18:14.038431+0000 mon.a (mon.0) 11529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: audit 2022-04-22T21:18:14.041442+0000 mon.a (mon.0) 11530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: audit 2022-04-22T21:18:14.043276+0000 mon.a (mon.0) 11531 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:18:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:14 smithi115 bash[46934]: audit 2022-04-22T21:18:14.420487+0000 mon.a (mon.0) 11532 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:18:16.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:15 smithi121 bash[36902]: audit 2022-04-22T21:18:14.639194+0000 mon.a (mon.0) 11533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:15 smithi115 bash[45495]: audit 2022-04-22T21:18:14.639194+0000 mon.a (mon.0) 11533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:15 smithi115 bash[46934]: audit 2022-04-22T21:18:14.639194+0000 mon.a (mon.0) 11533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:17.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:16 smithi121 bash[36902]: cluster 2022-04-22T21:18:15.170397+0000 mgr.y (mgr.34107) 7931 : cluster [DBG] pgmap v8906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:16 smithi121 bash[36902]: audit 2022-04-22T21:18:15.832244+0000 mon.a (mon.0) 11534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:16 smithi121 bash[36902]: audit 2022-04-22T21:18:15.954146+0000 mon.a (mon.0) 11535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:18:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:16 smithi121 bash[36902]: audit 2022-04-22T21:18:15.954840+0000 mon.a (mon.0) 11536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[45495]: cluster 2022-04-22T21:18:15.170397+0000 mgr.y (mgr.34107) 7931 : cluster [DBG] pgmap v8906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[45495]: audit 2022-04-22T21:18:15.832244+0000 mon.a (mon.0) 11534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[45495]: audit 2022-04-22T21:18:15.954146+0000 mon.a (mon.0) 11535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[45495]: audit 2022-04-22T21:18:15.954840+0000 mon.a (mon.0) 11536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[46934]: cluster 2022-04-22T21:18:15.170397+0000 mgr.y (mgr.34107) 7931 : cluster [DBG] pgmap v8906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[46934]: audit 2022-04-22T21:18:15.832244+0000 mon.a (mon.0) 11534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[46934]: audit 2022-04-22T21:18:15.954146+0000 mon.a (mon.0) 11535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:18:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:16 smithi115 bash[46934]: audit 2022-04-22T21:18:15.954840+0000 mon.a (mon.0) 11536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:18:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:18 smithi121 bash[36902]: cluster 2022-04-22T21:18:17.171596+0000 mgr.y (mgr.34107) 7932 : cluster [DBG] pgmap v8907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:18 smithi115 bash[45495]: cluster 2022-04-22T21:18:17.171596+0000 mgr.y (mgr.34107) 7932 : cluster [DBG] pgmap v8907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:18 smithi115 bash[46934]: cluster 2022-04-22T21:18:17.171596+0000 mgr.y (mgr.34107) 7932 : cluster [DBG] pgmap v8907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:19 smithi121 bash[36902]: audit 2022-04-22T21:18:18.828520+0000 mon.a (mon.0) 11537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:19 smithi115 bash[45495]: audit 2022-04-22T21:18:18.828520+0000 mon.a (mon.0) 11537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:19 smithi115 bash[46934]: audit 2022-04-22T21:18:18.828520+0000 mon.a (mon.0) 11537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:20 smithi121 bash[36902]: cluster 2022-04-22T21:18:19.172583+0000 mgr.y (mgr.34107) 7933 : cluster [DBG] pgmap v8908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:20 smithi121 bash[36902]: audit 2022-04-22T21:18:19.643092+0000 mon.a (mon.0) 11538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:20 smithi115 bash[45495]: cluster 2022-04-22T21:18:19.172583+0000 mgr.y (mgr.34107) 7933 : cluster [DBG] pgmap v8908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:21.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:20 smithi115 bash[45495]: audit 2022-04-22T21:18:19.643092+0000 mon.a (mon.0) 11538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:20 smithi115 bash[46934]: cluster 2022-04-22T21:18:19.172583+0000 mgr.y (mgr.34107) 7933 : cluster [DBG] pgmap v8908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:20 smithi115 bash[46934]: audit 2022-04-22T21:18:19.643092+0000 mon.a (mon.0) 11538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:21 smithi121 bash[36902]: audit 2022-04-22T21:18:20.835452+0000 mon.a (mon.0) 11539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:21 smithi115 bash[46934]: audit 2022-04-22T21:18:20.835452+0000 mon.a (mon.0) 11539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:21 smithi115 bash[45495]: audit 2022-04-22T21:18:20.835452+0000 mon.a (mon.0) 11539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:22 smithi115 bash[45495]: cluster 2022-04-22T21:18:21.173601+0000 mgr.y (mgr.34107) 7934 : cluster [DBG] pgmap v8909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:22 smithi115 bash[46934]: cluster 2022-04-22T21:18:21.173601+0000 mgr.y (mgr.34107) 7934 : cluster [DBG] pgmap v8909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:22.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:22 smithi121 bash[36902]: cluster 2022-04-22T21:18:21.173601+0000 mgr.y (mgr.34107) 7934 : cluster [DBG] pgmap v8909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:24.392Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:24 smithi115 bash[45495]: cluster 2022-04-22T21:18:23.174696+0000 mgr.y (mgr.34107) 7935 : cluster [DBG] pgmap v8910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:24 smithi115 bash[45495]: audit 2022-04-22T21:18:23.831593+0000 mon.a (mon.0) 11540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:24 smithi115 bash[46934]: cluster 2022-04-22T21:18:23.174696+0000 mgr.y (mgr.34107) 7935 : cluster [DBG] pgmap v8910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:24 smithi115 bash[46934]: audit 2022-04-22T21:18:23.831593+0000 mon.a (mon.0) 11540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:24 smithi121 bash[36902]: cluster 2022-04-22T21:18:23.174696+0000 mgr.y (mgr.34107) 7935 : cluster [DBG] pgmap v8910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:24 smithi121 bash[36902]: audit 2022-04-22T21:18:23.831593+0000 mon.a (mon.0) 11540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:25 smithi121 bash[36902]: audit 2022-04-22T21:18:24.646574+0000 mon.a (mon.0) 11541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:25 smithi115 bash[45495]: audit 2022-04-22T21:18:24.646574+0000 mon.a (mon.0) 11541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:25 smithi115 bash[46934]: audit 2022-04-22T21:18:24.646574+0000 mon.a (mon.0) 11541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:26 smithi121 bash[36902]: cluster 2022-04-22T21:18:25.175803+0000 mgr.y (mgr.34107) 7936 : cluster [DBG] pgmap v8911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:26 smithi121 bash[36902]: audit 2022-04-22T21:18:25.838528+0000 mon.a (mon.0) 11542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:26 smithi115 bash[45495]: cluster 2022-04-22T21:18:25.175803+0000 mgr.y (mgr.34107) 7936 : cluster [DBG] pgmap v8911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:26 smithi115 bash[45495]: audit 2022-04-22T21:18:25.838528+0000 mon.a (mon.0) 11542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:26 smithi115 bash[46934]: cluster 2022-04-22T21:18:25.175803+0000 mgr.y (mgr.34107) 7936 : cluster [DBG] pgmap v8911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:26 smithi115 bash[46934]: audit 2022-04-22T21:18:25.838528+0000 mon.a (mon.0) 11542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:28 smithi121 bash[36902]: cluster 2022-04-22T21:18:27.176911+0000 mgr.y (mgr.34107) 7937 : cluster [DBG] pgmap v8912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:28 smithi115 bash[45495]: cluster 2022-04-22T21:18:27.176911+0000 mgr.y (mgr.34107) 7937 : cluster [DBG] pgmap v8912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:28 smithi115 bash[46934]: cluster 2022-04-22T21:18:27.176911+0000 mgr.y (mgr.34107) 7937 : cluster [DBG] pgmap v8912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:29 smithi121 bash[36902]: audit 2022-04-22T21:18:28.834741+0000 mon.a (mon.0) 11543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:29 smithi115 bash[45495]: audit 2022-04-22T21:18:28.834741+0000 mon.a (mon.0) 11543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:29 smithi115 bash[46934]: audit 2022-04-22T21:18:28.834741+0000 mon.a (mon.0) 11543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:30 smithi121 bash[36902]: cluster 2022-04-22T21:18:29.177979+0000 mgr.y (mgr.34107) 7938 : cluster [DBG] pgmap v8913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:30 smithi121 bash[36902]: audit 2022-04-22T21:18:29.650248+0000 mon.a (mon.0) 11544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:30 smithi115 bash[45495]: cluster 2022-04-22T21:18:29.177979+0000 mgr.y (mgr.34107) 7938 : cluster [DBG] pgmap v8913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:30 smithi115 bash[45495]: audit 2022-04-22T21:18:29.650248+0000 mon.a (mon.0) 11544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:30 smithi115 bash[46934]: cluster 2022-04-22T21:18:29.177979+0000 mgr.y (mgr.34107) 7938 : cluster [DBG] pgmap v8913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:30 smithi115 bash[46934]: audit 2022-04-22T21:18:29.650248+0000 mon.a (mon.0) 11544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:32.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:31 smithi121 bash[36902]: audit 2022-04-22T21:18:30.841510+0000 mon.a (mon.0) 11545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:31 smithi115 bash[45495]: audit 2022-04-22T21:18:30.841510+0000 mon.a (mon.0) 11545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:31 smithi115 bash[46934]: audit 2022-04-22T21:18:30.841510+0000 mon.a (mon.0) 11545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:32.320 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:18:32.667 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:18:32.667 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:18:32.667 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 5m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:18:32.667 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:18:32.668 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:18:32.668 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:18:32.668 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 4m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:18:32.668 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 4m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:18:32.669 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:18:32.669 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 4m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:18:32.669 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:18:32.669 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:18:32.669 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 3h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:18:32.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:18:32.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:18:32.671 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:18:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:32 smithi115 bash[45495]: cluster 2022-04-22T21:18:31.179023+0000 mgr.y (mgr.34107) 7939 : cluster [DBG] pgmap v8914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:32.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:32.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:32 smithi115 bash[46934]: cluster 2022-04-22T21:18:31.179023+0000 mgr.y (mgr.34107) 7939 : cluster [DBG] pgmap v8914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:33.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:32 smithi121 bash[36902]: cluster 2022-04-22T21:18:31.179023+0000 mgr.y (mgr.34107) 7939 : cluster [DBG] pgmap v8914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:33.076 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:18:33.077 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:18:33.078 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:18:33.079 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:18:33.079 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:18:34.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:33 smithi121 bash[36902]: audit 2022-04-22T21:18:32.298747+0000 mgr.y (mgr.34107) 7940 : audit [DBG] from='client.53545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:33 smithi121 bash[36902]: audit 2022-04-22T21:18:32.641153+0000 mgr.y (mgr.34107) 7941 : audit [DBG] from='client.43620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:33 smithi121 bash[36902]: audit 2022-04-22T21:18:33.072365+0000 mon.c (mon.1) 286 : audit [DBG] from='client.? 172.21.15.115:0/1589912879' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[45495]: audit 2022-04-22T21:18:32.298747+0000 mgr.y (mgr.34107) 7940 : audit [DBG] from='client.53545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[45495]: audit 2022-04-22T21:18:32.641153+0000 mgr.y (mgr.34107) 7941 : audit [DBG] from='client.43620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[45495]: audit 2022-04-22T21:18:33.072365+0000 mon.c (mon.1) 286 : audit [DBG] from='client.? 172.21.15.115:0/1589912879' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[46934]: audit 2022-04-22T21:18:32.298747+0000 mgr.y (mgr.34107) 7940 : audit [DBG] from='client.53545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[46934]: audit 2022-04-22T21:18:32.641153+0000 mgr.y (mgr.34107) 7941 : audit [DBG] from='client.43620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:18:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:33 smithi115 bash[46934]: audit 2022-04-22T21:18:33.072365+0000 mon.c (mon.1) 286 : audit [DBG] from='client.? 172.21.15.115:0/1589912879' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:18:34.657 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:34.393Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:34 smithi121 bash[36902]: cluster 2022-04-22T21:18:33.180201+0000 mgr.y (mgr.34107) 7942 : cluster [DBG] pgmap v8915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:34 smithi121 bash[36902]: audit 2022-04-22T21:18:33.837907+0000 mon.a (mon.0) 11546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:34 smithi115 bash[46934]: cluster 2022-04-22T21:18:33.180201+0000 mgr.y (mgr.34107) 7942 : cluster [DBG] pgmap v8915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:34 smithi115 bash[46934]: audit 2022-04-22T21:18:33.837907+0000 mon.a (mon.0) 11546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:34 smithi115 bash[45495]: cluster 2022-04-22T21:18:33.180201+0000 mgr.y (mgr.34107) 7942 : cluster [DBG] pgmap v8915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:34 smithi115 bash[45495]: audit 2022-04-22T21:18:33.837907+0000 mon.a (mon.0) 11546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:35 smithi121 bash[36902]: audit 2022-04-22T21:18:34.659196+0000 mon.a (mon.0) 11547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:35 smithi115 bash[45495]: audit 2022-04-22T21:18:34.659196+0000 mon.a (mon.0) 11547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:35 smithi115 bash[46934]: audit 2022-04-22T21:18:34.659196+0000 mon.a (mon.0) 11547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:36 smithi121 bash[36902]: cluster 2022-04-22T21:18:35.181286+0000 mgr.y (mgr.34107) 7943 : cluster [DBG] pgmap v8916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:36 smithi121 bash[36902]: audit 2022-04-22T21:18:35.844409+0000 mon.a (mon.0) 11548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:36 smithi115 bash[45495]: cluster 2022-04-22T21:18:35.181286+0000 mgr.y (mgr.34107) 7943 : cluster [DBG] pgmap v8916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:36 smithi115 bash[45495]: audit 2022-04-22T21:18:35.844409+0000 mon.a (mon.0) 11548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:36 smithi115 bash[46934]: cluster 2022-04-22T21:18:35.181286+0000 mgr.y (mgr.34107) 7943 : cluster [DBG] pgmap v8916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:36 smithi115 bash[46934]: audit 2022-04-22T21:18:35.844409+0000 mon.a (mon.0) 11548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:38 smithi121 bash[36902]: cluster 2022-04-22T21:18:37.182373+0000 mgr.y (mgr.34107) 7944 : cluster [DBG] pgmap v8917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:38 smithi115 bash[45495]: cluster 2022-04-22T21:18:37.182373+0000 mgr.y (mgr.34107) 7944 : cluster [DBG] pgmap v8917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:38 smithi115 bash[46934]: cluster 2022-04-22T21:18:37.182373+0000 mgr.y (mgr.34107) 7944 : cluster [DBG] pgmap v8917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:39 smithi121 bash[36902]: audit 2022-04-22T21:18:38.841015+0000 mon.a (mon.0) 11549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:39 smithi121 bash[36902]: audit 2022-04-22T21:18:39.658294+0000 mon.a (mon.0) 11550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:39 smithi115 bash[45495]: audit 2022-04-22T21:18:38.841015+0000 mon.a (mon.0) 11549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:39 smithi115 bash[45495]: audit 2022-04-22T21:18:39.658294+0000 mon.a (mon.0) 11550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:39 smithi115 bash[46934]: audit 2022-04-22T21:18:38.841015+0000 mon.a (mon.0) 11549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:39 smithi115 bash[46934]: audit 2022-04-22T21:18:39.658294+0000 mon.a (mon.0) 11550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:41.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:40 smithi121 bash[36902]: cluster 2022-04-22T21:18:39.183561+0000 mgr.y (mgr.34107) 7945 : cluster [DBG] pgmap v8918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:40 smithi115 bash[45495]: cluster 2022-04-22T21:18:39.183561+0000 mgr.y (mgr.34107) 7945 : cluster [DBG] pgmap v8918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:40 smithi115 bash[46934]: cluster 2022-04-22T21:18:39.183561+0000 mgr.y (mgr.34107) 7945 : cluster [DBG] pgmap v8918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:42.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:41 smithi121 bash[36902]: audit 2022-04-22T21:18:40.847318+0000 mon.a (mon.0) 11551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:41 smithi115 bash[45495]: audit 2022-04-22T21:18:40.847318+0000 mon.a (mon.0) 11551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:41 smithi115 bash[46934]: audit 2022-04-22T21:18:40.847318+0000 mon.a (mon.0) 11551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:42 smithi115 bash[45495]: cluster 2022-04-22T21:18:41.184654+0000 mgr.y (mgr.34107) 7946 : cluster [DBG] pgmap v8919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:42 smithi115 bash[46934]: cluster 2022-04-22T21:18:41.184654+0000 mgr.y (mgr.34107) 7946 : cluster [DBG] pgmap v8919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:42.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:42.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:42 smithi121 bash[36902]: cluster 2022-04-22T21:18:41.184654+0000 mgr.y (mgr.34107) 7946 : cluster [DBG] pgmap v8919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:44.697 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:44.394Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:44 smithi121 bash[36902]: cluster 2022-04-22T21:18:43.185893+0000 mgr.y (mgr.34107) 7947 : cluster [DBG] pgmap v8920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:44 smithi121 bash[36902]: audit 2022-04-22T21:18:43.844034+0000 mon.a (mon.0) 11552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:44 smithi121 bash[36902]: audit 2022-04-22T21:18:44.661818+0000 mon.a (mon.0) 11553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[45495]: cluster 2022-04-22T21:18:43.185893+0000 mgr.y (mgr.34107) 7947 : cluster [DBG] pgmap v8920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:45.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[45495]: audit 2022-04-22T21:18:43.844034+0000 mon.a (mon.0) 11552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:45.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[45495]: audit 2022-04-22T21:18:44.661818+0000 mon.a (mon.0) 11553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[46934]: cluster 2022-04-22T21:18:43.185893+0000 mgr.y (mgr.34107) 7947 : cluster [DBG] pgmap v8920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[46934]: audit 2022-04-22T21:18:43.844034+0000 mon.a (mon.0) 11552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:44 smithi115 bash[46934]: audit 2022-04-22T21:18:44.661818+0000 mon.a (mon.0) 11553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:46 smithi121 bash[36902]: cluster 2022-04-22T21:18:45.186943+0000 mgr.y (mgr.34107) 7948 : cluster [DBG] pgmap v8921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:46 smithi121 bash[36902]: audit 2022-04-22T21:18:45.850309+0000 mon.a (mon.0) 11554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:46 smithi115 bash[45495]: cluster 2022-04-22T21:18:45.186943+0000 mgr.y (mgr.34107) 7948 : cluster [DBG] pgmap v8921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:46 smithi115 bash[45495]: audit 2022-04-22T21:18:45.850309+0000 mon.a (mon.0) 11554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:46 smithi115 bash[46934]: cluster 2022-04-22T21:18:45.186943+0000 mgr.y (mgr.34107) 7948 : cluster [DBG] pgmap v8921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:46 smithi115 bash[46934]: audit 2022-04-22T21:18:45.850309+0000 mon.a (mon.0) 11554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:48.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:48 smithi121 bash[36902]: cluster 2022-04-22T21:18:47.188071+0000 mgr.y (mgr.34107) 7949 : cluster [DBG] pgmap v8922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:48 smithi115 bash[45495]: cluster 2022-04-22T21:18:47.188071+0000 mgr.y (mgr.34107) 7949 : cluster [DBG] pgmap v8922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:48 smithi115 bash[46934]: cluster 2022-04-22T21:18:47.188071+0000 mgr.y (mgr.34107) 7949 : cluster [DBG] pgmap v8922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:49 smithi121 bash[36902]: audit 2022-04-22T21:18:48.847059+0000 mon.a (mon.0) 11555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:49 smithi115 bash[46934]: audit 2022-04-22T21:18:48.847059+0000 mon.a (mon.0) 11555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:49 smithi115 bash[45495]: audit 2022-04-22T21:18:48.847059+0000 mon.a (mon.0) 11555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:50 smithi121 bash[36902]: cluster 2022-04-22T21:18:49.189090+0000 mgr.y (mgr.34107) 7950 : cluster [DBG] pgmap v8923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:50 smithi121 bash[36902]: audit 2022-04-22T21:18:49.665365+0000 mon.a (mon.0) 11556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:50 smithi115 bash[45495]: cluster 2022-04-22T21:18:49.189090+0000 mgr.y (mgr.34107) 7950 : cluster [DBG] pgmap v8923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:50 smithi115 bash[45495]: audit 2022-04-22T21:18:49.665365+0000 mon.a (mon.0) 11556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:50 smithi115 bash[46934]: cluster 2022-04-22T21:18:49.189090+0000 mgr.y (mgr.34107) 7950 : cluster [DBG] pgmap v8923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:50 smithi115 bash[46934]: audit 2022-04-22T21:18:49.665365+0000 mon.a (mon.0) 11556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:51 smithi115 bash[46934]: audit 2022-04-22T21:18:50.853485+0000 mon.a (mon.0) 11557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:51 smithi115 bash[45495]: audit 2022-04-22T21:18:50.853485+0000 mon.a (mon.0) 11557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:51 smithi121 bash[36902]: audit 2022-04-22T21:18:50.853485+0000 mon.a (mon.0) 11557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:52.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:52 smithi115 bash[45495]: cluster 2022-04-22T21:18:51.190168+0000 mgr.y (mgr.34107) 7951 : cluster [DBG] pgmap v8924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:52.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:52 smithi115 bash[46934]: cluster 2022-04-22T21:18:51.190168+0000 mgr.y (mgr.34107) 7951 : cluster [DBG] pgmap v8924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:52 smithi121 bash[36902]: cluster 2022-04-22T21:18:51.190168+0000 mgr.y (mgr.34107) 7951 : cluster [DBG] pgmap v8924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:52.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:54.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:18:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:18:54.396Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:18:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:54 smithi115 bash[45495]: cluster 2022-04-22T21:18:53.191296+0000 mgr.y (mgr.34107) 7952 : cluster [DBG] pgmap v8925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:54 smithi115 bash[45495]: audit 2022-04-22T21:18:53.850112+0000 mon.a (mon.0) 11558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:54 smithi115 bash[46934]: cluster 2022-04-22T21:18:53.191296+0000 mgr.y (mgr.34107) 7952 : cluster [DBG] pgmap v8925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:54.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:54 smithi115 bash[46934]: audit 2022-04-22T21:18:53.850112+0000 mon.a (mon.0) 11558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:54 smithi121 bash[36902]: cluster 2022-04-22T21:18:53.191296+0000 mgr.y (mgr.34107) 7952 : cluster [DBG] pgmap v8925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:54 smithi121 bash[36902]: audit 2022-04-22T21:18:53.850112+0000 mon.a (mon.0) 11558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:55 smithi115 bash[45495]: audit 2022-04-22T21:18:54.669256+0000 mon.a (mon.0) 11559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:55 smithi115 bash[46934]: audit 2022-04-22T21:18:54.669256+0000 mon.a (mon.0) 11559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:55 smithi121 bash[36902]: audit 2022-04-22T21:18:54.669256+0000 mon.a (mon.0) 11559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:18:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:56 smithi115 bash[45495]: cluster 2022-04-22T21:18:55.192361+0000 mgr.y (mgr.34107) 7953 : cluster [DBG] pgmap v8926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:56 smithi115 bash[45495]: audit 2022-04-22T21:18:55.856395+0000 mon.a (mon.0) 11560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:56 smithi115 bash[46934]: cluster 2022-04-22T21:18:55.192361+0000 mgr.y (mgr.34107) 7953 : cluster [DBG] pgmap v8926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:56 smithi115 bash[46934]: audit 2022-04-22T21:18:55.856395+0000 mon.a (mon.0) 11560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:56 smithi121 bash[36902]: cluster 2022-04-22T21:18:55.192361+0000 mgr.y (mgr.34107) 7953 : cluster [DBG] pgmap v8926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:56 smithi121 bash[36902]: audit 2022-04-22T21:18:55.856395+0000 mon.a (mon.0) 11560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:18:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:58 smithi115 bash[45495]: cluster 2022-04-22T21:18:57.193436+0000 mgr.y (mgr.34107) 7954 : cluster [DBG] pgmap v8927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:58 smithi115 bash[46934]: cluster 2022-04-22T21:18:57.193436+0000 mgr.y (mgr.34107) 7954 : cluster [DBG] pgmap v8927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:58 smithi121 bash[36902]: cluster 2022-04-22T21:18:57.193436+0000 mgr.y (mgr.34107) 7954 : cluster [DBG] pgmap v8927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:18:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:18:59 smithi115 bash[45495]: audit 2022-04-22T21:18:58.853507+0000 mon.a (mon.0) 11561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:18:59 smithi115 bash[46934]: audit 2022-04-22T21:18:58.853507+0000 mon.a (mon.0) 11561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:18:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:18:59 smithi121 bash[36902]: audit 2022-04-22T21:18:58.853507+0000 mon.a (mon.0) 11561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:00 smithi121 bash[36902]: cluster 2022-04-22T21:18:59.194426+0000 mgr.y (mgr.34107) 7955 : cluster [DBG] pgmap v8928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:00 smithi121 bash[36902]: audit 2022-04-22T21:18:59.673368+0000 mon.a (mon.0) 11562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:00.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:00 smithi115 bash[45495]: cluster 2022-04-22T21:18:59.194426+0000 mgr.y (mgr.34107) 7955 : cluster [DBG] pgmap v8928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:00.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:00 smithi115 bash[45495]: audit 2022-04-22T21:18:59.673368+0000 mon.a (mon.0) 11562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:00.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:00 smithi115 bash[46934]: cluster 2022-04-22T21:18:59.194426+0000 mgr.y (mgr.34107) 7955 : cluster [DBG] pgmap v8928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:00.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:00 smithi115 bash[46934]: audit 2022-04-22T21:18:59.673368+0000 mon.a (mon.0) 11562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:01.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:01 smithi121 bash[36902]: audit 2022-04-22T21:19:00.859451+0000 mon.a (mon.0) 11563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:01 smithi115 bash[45495]: audit 2022-04-22T21:19:00.859451+0000 mon.a (mon.0) 11563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:01 smithi115 bash[46934]: audit 2022-04-22T21:19:00.859451+0000 mon.a (mon.0) 11563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:02 smithi121 bash[36902]: cluster 2022-04-22T21:19:01.195421+0000 mgr.y (mgr.34107) 7956 : cluster [DBG] pgmap v8929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:02 smithi115 bash[45495]: cluster 2022-04-22T21:19:01.195421+0000 mgr.y (mgr.34107) 7956 : cluster [DBG] pgmap v8929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:02 smithi115 bash[46934]: cluster 2022-04-22T21:19:01.195421+0000 mgr.y (mgr.34107) 7956 : cluster [DBG] pgmap v8929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:02.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:03.449 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:19:03.791 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:19:03.792 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 3h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:19:03.793 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:19:04.195 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:19:04.195 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:04.196 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:19:04.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:19:04.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:04.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:19:04.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:19:04.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:04.198 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:19:04.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:19:04.198 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:19:04.198 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:19:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:04 smithi121 bash[36902]: cluster 2022-04-22T21:19:03.196532+0000 mgr.y (mgr.34107) 7957 : cluster [DBG] pgmap v8930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:04 smithi121 bash[36902]: audit 2022-04-22T21:19:03.429279+0000 mgr.y (mgr.34107) 7958 : audit [DBG] from='client.43632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:04 smithi121 bash[36902]: audit 2022-04-22T21:19:03.773038+0000 mgr.y (mgr.34107) 7959 : audit [DBG] from='client.43638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:04 smithi121 bash[36902]: audit 2022-04-22T21:19:03.856555+0000 mon.a (mon.0) 11564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:04 smithi121 bash[36902]: audit 2022-04-22T21:19:04.191537+0000 mon.c (mon.1) 287 : audit [DBG] from='client.? 172.21.15.115:0/3682505565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:04.392Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[45495]: cluster 2022-04-22T21:19:03.196532+0000 mgr.y (mgr.34107) 7957 : cluster [DBG] pgmap v8930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[45495]: audit 2022-04-22T21:19:03.429279+0000 mgr.y (mgr.34107) 7958 : audit [DBG] from='client.43632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[45495]: audit 2022-04-22T21:19:03.773038+0000 mgr.y (mgr.34107) 7959 : audit [DBG] from='client.43638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[45495]: audit 2022-04-22T21:19:03.856555+0000 mon.a (mon.0) 11564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:04.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[45495]: audit 2022-04-22T21:19:04.191537+0000 mon.c (mon.1) 287 : audit [DBG] from='client.? 172.21.15.115:0/3682505565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[46934]: cluster 2022-04-22T21:19:03.196532+0000 mgr.y (mgr.34107) 7957 : cluster [DBG] pgmap v8930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[46934]: audit 2022-04-22T21:19:03.429279+0000 mgr.y (mgr.34107) 7958 : audit [DBG] from='client.43632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[46934]: audit 2022-04-22T21:19:03.773038+0000 mgr.y (mgr.34107) 7959 : audit [DBG] from='client.43638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[46934]: audit 2022-04-22T21:19:03.856555+0000 mon.a (mon.0) 11564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:04.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:04 smithi115 bash[46934]: audit 2022-04-22T21:19:04.191537+0000 mon.c (mon.1) 287 : audit [DBG] from='client.? 172.21.15.115:0/3682505565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:05.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:05 smithi121 bash[36902]: audit 2022-04-22T21:19:04.677089+0000 mon.a (mon.0) 11565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:05 smithi115 bash[46934]: audit 2022-04-22T21:19:04.677089+0000 mon.a (mon.0) 11565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:05 smithi115 bash[45495]: audit 2022-04-22T21:19:04.677089+0000 mon.a (mon.0) 11565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:06 smithi121 bash[36902]: cluster 2022-04-22T21:19:05.197613+0000 mgr.y (mgr.34107) 7960 : cluster [DBG] pgmap v8931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:06 smithi121 bash[36902]: audit 2022-04-22T21:19:05.862484+0000 mon.a (mon.0) 11566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:06 smithi115 bash[46934]: cluster 2022-04-22T21:19:05.197613+0000 mgr.y (mgr.34107) 7960 : cluster [DBG] pgmap v8931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:06 smithi115 bash[46934]: audit 2022-04-22T21:19:05.862484+0000 mon.a (mon.0) 11566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:06 smithi115 bash[45495]: cluster 2022-04-22T21:19:05.197613+0000 mgr.y (mgr.34107) 7960 : cluster [DBG] pgmap v8931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:06 smithi115 bash[45495]: audit 2022-04-22T21:19:05.862484+0000 mon.a (mon.0) 11566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:08 smithi121 bash[36902]: cluster 2022-04-22T21:19:07.198854+0000 mgr.y (mgr.34107) 7961 : cluster [DBG] pgmap v8932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:08 smithi115 bash[45495]: cluster 2022-04-22T21:19:07.198854+0000 mgr.y (mgr.34107) 7961 : cluster [DBG] pgmap v8932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:08 smithi115 bash[46934]: cluster 2022-04-22T21:19:07.198854+0000 mgr.y (mgr.34107) 7961 : cluster [DBG] pgmap v8932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:09 smithi121 bash[36902]: audit 2022-04-22T21:19:08.859526+0000 mon.a (mon.0) 11567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:09 smithi115 bash[45495]: audit 2022-04-22T21:19:08.859526+0000 mon.a (mon.0) 11567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:09 smithi115 bash[46934]: audit 2022-04-22T21:19:08.859526+0000 mon.a (mon.0) 11567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:10 smithi121 bash[36902]: cluster 2022-04-22T21:19:09.199993+0000 mgr.y (mgr.34107) 7962 : cluster [DBG] pgmap v8933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:10.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:10 smithi121 bash[36902]: audit 2022-04-22T21:19:09.680787+0000 mon.a (mon.0) 11568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:10 smithi115 bash[45495]: cluster 2022-04-22T21:19:09.199993+0000 mgr.y (mgr.34107) 7962 : cluster [DBG] pgmap v8933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:10 smithi115 bash[45495]: audit 2022-04-22T21:19:09.680787+0000 mon.a (mon.0) 11568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:10 smithi115 bash[46934]: cluster 2022-04-22T21:19:09.199993+0000 mgr.y (mgr.34107) 7962 : cluster [DBG] pgmap v8933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:10 smithi115 bash[46934]: audit 2022-04-22T21:19:09.680787+0000 mon.a (mon.0) 11568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:11.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:11 smithi121 bash[36902]: audit 2022-04-22T21:19:10.865460+0000 mon.a (mon.0) 11569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:11 smithi115 bash[45495]: audit 2022-04-22T21:19:10.865460+0000 mon.a (mon.0) 11569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:11 smithi115 bash[46934]: audit 2022-04-22T21:19:10.865460+0000 mon.a (mon.0) 11569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:12.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:12 smithi121 bash[36902]: cluster 2022-04-22T21:19:11.201101+0000 mgr.y (mgr.34107) 7963 : cluster [DBG] pgmap v8934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:12.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:12.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:12 smithi115 bash[45495]: cluster 2022-04-22T21:19:11.201101+0000 mgr.y (mgr.34107) 7963 : cluster [DBG] pgmap v8934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:12.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:12 smithi115 bash[46934]: cluster 2022-04-22T21:19:11.201101+0000 mgr.y (mgr.34107) 7963 : cluster [DBG] pgmap v8934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:14 smithi121 bash[36902]: cluster 2022-04-22T21:19:13.202334+0000 mgr.y (mgr.34107) 7964 : cluster [DBG] pgmap v8935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:14 smithi121 bash[36902]: audit 2022-04-22T21:19:13.862681+0000 mon.a (mon.0) 11570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:14.396Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:14 smithi115 bash[45495]: cluster 2022-04-22T21:19:13.202334+0000 mgr.y (mgr.34107) 7964 : cluster [DBG] pgmap v8935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:14 smithi115 bash[45495]: audit 2022-04-22T21:19:13.862681+0000 mon.a (mon.0) 11570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:14 smithi115 bash[46934]: cluster 2022-04-22T21:19:13.202334+0000 mgr.y (mgr.34107) 7964 : cluster [DBG] pgmap v8935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:14 smithi115 bash[46934]: audit 2022-04-22T21:19:13.862681+0000 mon.a (mon.0) 11570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:15 smithi121 bash[36902]: audit 2022-04-22T21:19:14.427171+0000 mon.a (mon.0) 11571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:19:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:15 smithi121 bash[36902]: audit 2022-04-22T21:19:14.431112+0000 mon.a (mon.0) 11572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:19:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:15 smithi121 bash[36902]: audit 2022-04-22T21:19:14.432918+0000 mon.a (mon.0) 11573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:19:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:15 smithi121 bash[36902]: audit 2022-04-22T21:19:14.684777+0000 mon.a (mon.0) 11574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:15 smithi121 bash[36902]: audit 2022-04-22T21:19:14.805087+0000 mon.a (mon.0) 11575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:19:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[45495]: audit 2022-04-22T21:19:14.427171+0000 mon.a (mon.0) 11571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:19:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[45495]: audit 2022-04-22T21:19:14.431112+0000 mon.a (mon.0) 11572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:19:15.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[45495]: audit 2022-04-22T21:19:14.432918+0000 mon.a (mon.0) 11573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:19:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[45495]: audit 2022-04-22T21:19:14.684777+0000 mon.a (mon.0) 11574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:15.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[45495]: audit 2022-04-22T21:19:14.805087+0000 mon.a (mon.0) 11575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:19:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[46934]: audit 2022-04-22T21:19:14.427171+0000 mon.a (mon.0) 11571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:19:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[46934]: audit 2022-04-22T21:19:14.431112+0000 mon.a (mon.0) 11572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:19:15.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[46934]: audit 2022-04-22T21:19:14.432918+0000 mon.a (mon.0) 11573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:19:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[46934]: audit 2022-04-22T21:19:14.684777+0000 mon.a (mon.0) 11574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:15 smithi115 bash[46934]: audit 2022-04-22T21:19:14.805087+0000 mon.a (mon.0) 11575 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:19:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:16 smithi121 bash[36902]: cluster 2022-04-22T21:19:15.203578+0000 mgr.y (mgr.34107) 7965 : cluster [DBG] pgmap v8936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:16 smithi121 bash[36902]: audit 2022-04-22T21:19:15.868525+0000 mon.a (mon.0) 11576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:16 smithi121 bash[36902]: audit 2022-04-22T21:19:15.965879+0000 mon.a (mon.0) 11577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:19:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:16 smithi121 bash[36902]: audit 2022-04-22T21:19:15.966328+0000 mon.a (mon.0) 11578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:19:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[45495]: cluster 2022-04-22T21:19:15.203578+0000 mgr.y (mgr.34107) 7965 : cluster [DBG] pgmap v8936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[45495]: audit 2022-04-22T21:19:15.868525+0000 mon.a (mon.0) 11576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[45495]: audit 2022-04-22T21:19:15.965879+0000 mon.a (mon.0) 11577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[45495]: audit 2022-04-22T21:19:15.966328+0000 mon.a (mon.0) 11578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[46934]: cluster 2022-04-22T21:19:15.203578+0000 mgr.y (mgr.34107) 7965 : cluster [DBG] pgmap v8936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[46934]: audit 2022-04-22T21:19:15.868525+0000 mon.a (mon.0) 11576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[46934]: audit 2022-04-22T21:19:15.965879+0000 mon.a (mon.0) 11577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:19:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:16 smithi115 bash[46934]: audit 2022-04-22T21:19:15.966328+0000 mon.a (mon.0) 11578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:19:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:18 smithi121 bash[36902]: cluster 2022-04-22T21:19:17.204617+0000 mgr.y (mgr.34107) 7966 : cluster [DBG] pgmap v8937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:18 smithi115 bash[45495]: cluster 2022-04-22T21:19:17.204617+0000 mgr.y (mgr.34107) 7966 : cluster [DBG] pgmap v8937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:18 smithi115 bash[46934]: cluster 2022-04-22T21:19:17.204617+0000 mgr.y (mgr.34107) 7966 : cluster [DBG] pgmap v8937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:19 smithi121 bash[36902]: audit 2022-04-22T21:19:18.865800+0000 mon.a (mon.0) 11579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:19 smithi115 bash[45495]: audit 2022-04-22T21:19:18.865800+0000 mon.a (mon.0) 11579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:19 smithi115 bash[46934]: audit 2022-04-22T21:19:18.865800+0000 mon.a (mon.0) 11579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:20 smithi121 bash[36902]: cluster 2022-04-22T21:19:19.205642+0000 mgr.y (mgr.34107) 7967 : cluster [DBG] pgmap v8938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:20 smithi121 bash[36902]: audit 2022-04-22T21:19:19.688946+0000 mon.a (mon.0) 11580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:20 smithi115 bash[46934]: cluster 2022-04-22T21:19:19.205642+0000 mgr.y (mgr.34107) 7967 : cluster [DBG] pgmap v8938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:20 smithi115 bash[46934]: audit 2022-04-22T21:19:19.688946+0000 mon.a (mon.0) 11580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:20 smithi115 bash[45495]: cluster 2022-04-22T21:19:19.205642+0000 mgr.y (mgr.34107) 7967 : cluster [DBG] pgmap v8938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:20.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:20 smithi115 bash[45495]: audit 2022-04-22T21:19:19.688946+0000 mon.a (mon.0) 11580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:21 smithi121 bash[36902]: audit 2022-04-22T21:19:20.871576+0000 mon.a (mon.0) 11581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:21 smithi115 bash[46934]: audit 2022-04-22T21:19:20.871576+0000 mon.a (mon.0) 11581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:21 smithi115 bash[45495]: audit 2022-04-22T21:19:20.871576+0000 mon.a (mon.0) 11581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:22 smithi121 bash[36902]: cluster 2022-04-22T21:19:21.206642+0000 mgr.y (mgr.34107) 7968 : cluster [DBG] pgmap v8939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:22 smithi115 bash[45495]: cluster 2022-04-22T21:19:21.206642+0000 mgr.y (mgr.34107) 7968 : cluster [DBG] pgmap v8939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:22 smithi115 bash[46934]: cluster 2022-04-22T21:19:21.206642+0000 mgr.y (mgr.34107) 7968 : cluster [DBG] pgmap v8939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:22.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:24 smithi121 bash[36902]: cluster 2022-04-22T21:19:23.207761+0000 mgr.y (mgr.34107) 7969 : cluster [DBG] pgmap v8940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:24 smithi121 bash[36902]: audit 2022-04-22T21:19:23.868898+0000 mon.a (mon.0) 11582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:24 smithi115 bash[45495]: cluster 2022-04-22T21:19:23.207761+0000 mgr.y (mgr.34107) 7969 : cluster [DBG] pgmap v8940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:24 smithi115 bash[45495]: audit 2022-04-22T21:19:23.868898+0000 mon.a (mon.0) 11582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:24.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:24.394Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:24 smithi115 bash[46934]: cluster 2022-04-22T21:19:23.207761+0000 mgr.y (mgr.34107) 7969 : cluster [DBG] pgmap v8940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:24 smithi115 bash[46934]: audit 2022-04-22T21:19:23.868898+0000 mon.a (mon.0) 11582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:25.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:25 smithi121 bash[36902]: audit 2022-04-22T21:19:24.692886+0000 mon.a (mon.0) 11583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:25 smithi115 bash[45495]: audit 2022-04-22T21:19:24.692886+0000 mon.a (mon.0) 11583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:25 smithi115 bash[46934]: audit 2022-04-22T21:19:24.692886+0000 mon.a (mon.0) 11583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:26 smithi121 bash[36902]: cluster 2022-04-22T21:19:25.208986+0000 mgr.y (mgr.34107) 7970 : cluster [DBG] pgmap v8941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:26 smithi121 bash[36902]: audit 2022-04-22T21:19:25.874310+0000 mon.a (mon.0) 11584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:26 smithi115 bash[45495]: cluster 2022-04-22T21:19:25.208986+0000 mgr.y (mgr.34107) 7970 : cluster [DBG] pgmap v8941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:26 smithi115 bash[45495]: audit 2022-04-22T21:19:25.874310+0000 mon.a (mon.0) 11584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:26 smithi115 bash[46934]: cluster 2022-04-22T21:19:25.208986+0000 mgr.y (mgr.34107) 7970 : cluster [DBG] pgmap v8941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:26 smithi115 bash[46934]: audit 2022-04-22T21:19:25.874310+0000 mon.a (mon.0) 11584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:28 smithi121 bash[36902]: cluster 2022-04-22T21:19:27.210134+0000 mgr.y (mgr.34107) 7971 : cluster [DBG] pgmap v8942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:28 smithi115 bash[45495]: cluster 2022-04-22T21:19:27.210134+0000 mgr.y (mgr.34107) 7971 : cluster [DBG] pgmap v8942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:28 smithi115 bash[46934]: cluster 2022-04-22T21:19:27.210134+0000 mgr.y (mgr.34107) 7971 : cluster [DBG] pgmap v8942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:29.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:29 smithi121 bash[36902]: audit 2022-04-22T21:19:28.872045+0000 mon.a (mon.0) 11585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:29 smithi115 bash[45495]: audit 2022-04-22T21:19:28.872045+0000 mon.a (mon.0) 11585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:29 smithi115 bash[46934]: audit 2022-04-22T21:19:28.872045+0000 mon.a (mon.0) 11585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:30 smithi121 bash[36902]: cluster 2022-04-22T21:19:29.211193+0000 mgr.y (mgr.34107) 7972 : cluster [DBG] pgmap v8943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:30 smithi121 bash[36902]: audit 2022-04-22T21:19:29.696567+0000 mon.a (mon.0) 11586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:30 smithi115 bash[45495]: cluster 2022-04-22T21:19:29.211193+0000 mgr.y (mgr.34107) 7972 : cluster [DBG] pgmap v8943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:30 smithi115 bash[45495]: audit 2022-04-22T21:19:29.696567+0000 mon.a (mon.0) 11586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:30 smithi115 bash[46934]: cluster 2022-04-22T21:19:29.211193+0000 mgr.y (mgr.34107) 7972 : cluster [DBG] pgmap v8943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:30.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:30 smithi115 bash[46934]: audit 2022-04-22T21:19:29.696567+0000 mon.a (mon.0) 11586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:31.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:31 smithi121 bash[36902]: audit 2022-04-22T21:19:30.877335+0000 mon.a (mon.0) 11587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:31 smithi115 bash[45495]: audit 2022-04-22T21:19:30.877335+0000 mon.a (mon.0) 11587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:31.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:31 smithi115 bash[46934]: audit 2022-04-22T21:19:30.877335+0000 mon.a (mon.0) 11587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:32 smithi121 bash[36902]: cluster 2022-04-22T21:19:31.212262+0000 mgr.y (mgr.34107) 7973 : cluster [DBG] pgmap v8944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:32 smithi115 bash[45495]: cluster 2022-04-22T21:19:31.212262+0000 mgr.y (mgr.34107) 7973 : cluster [DBG] pgmap v8944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:32.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:32 smithi115 bash[46934]: cluster 2022-04-22T21:19:31.212262+0000 mgr.y (mgr.34107) 7973 : cluster [DBG] pgmap v8944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:34.573 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:19:34.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:34 smithi121 bash[36902]: cluster 2022-04-22T21:19:33.213329+0000 mgr.y (mgr.34107) 7974 : cluster [DBG] pgmap v8945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:34 smithi121 bash[36902]: audit 2022-04-22T21:19:33.875560+0000 mon.a (mon.0) 11588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:34 smithi115 bash[46934]: cluster 2022-04-22T21:19:33.213329+0000 mgr.y (mgr.34107) 7974 : cluster [DBG] pgmap v8945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:34 smithi115 bash[46934]: audit 2022-04-22T21:19:33.875560+0000 mon.a (mon.0) 11588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:34.396Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:34 smithi115 bash[45495]: cluster 2022-04-22T21:19:33.213329+0000 mgr.y (mgr.34107) 7974 : cluster [DBG] pgmap v8945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:34 smithi115 bash[45495]: audit 2022-04-22T21:19:33.875560+0000 mon.a (mon.0) 11588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:34.922 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:19:34.923 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:19:34.923 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 6m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:19:34.923 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:19:34.923 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:19:34.923 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:19:34.924 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 5m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:19:34.924 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 5m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:19:34.924 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 6m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:19:34.925 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 5m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:19:34.925 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:19:34.925 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:19:34.925 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:19:34.925 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:19:34.926 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:19:34.926 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 3h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:19:34.926 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:19:34.926 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:19:34.927 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:19:34.927 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:19:34.927 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:19:35.328 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:19:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:19:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:19:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:19:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:19:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:19:35.332 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:19:35.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:35 smithi121 bash[36902]: audit 2022-04-22T21:19:34.554282+0000 mgr.y (mgr.34107) 7975 : audit [DBG] from='client.53581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:35 smithi121 bash[36902]: audit 2022-04-22T21:19:34.700058+0000 mon.a (mon.0) 11589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:35 smithi121 bash[36902]: audit 2022-04-22T21:19:35.324698+0000 mon.a (mon.0) 11590 : audit [DBG] from='client.? 172.21.15.115:0/2024785291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[45495]: audit 2022-04-22T21:19:34.554282+0000 mgr.y (mgr.34107) 7975 : audit [DBG] from='client.53581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[45495]: audit 2022-04-22T21:19:34.700058+0000 mon.a (mon.0) 11589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[45495]: audit 2022-04-22T21:19:35.324698+0000 mon.a (mon.0) 11590 : audit [DBG] from='client.? 172.21.15.115:0/2024785291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[46934]: audit 2022-04-22T21:19:34.554282+0000 mgr.y (mgr.34107) 7975 : audit [DBG] from='client.53581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[46934]: audit 2022-04-22T21:19:34.700058+0000 mon.a (mon.0) 11589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:35.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:35 smithi115 bash[46934]: audit 2022-04-22T21:19:35.324698+0000 mon.a (mon.0) 11590 : audit [DBG] from='client.? 172.21.15.115:0/2024785291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:19:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:36 smithi121 bash[36902]: audit 2022-04-22T21:19:34.899989+0000 mgr.y (mgr.34107) 7976 : audit [DBG] from='client.43656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:36 smithi121 bash[36902]: cluster 2022-04-22T21:19:35.214312+0000 mgr.y (mgr.34107) 7977 : cluster [DBG] pgmap v8946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:36 smithi121 bash[36902]: audit 2022-04-22T21:19:35.879896+0000 mon.a (mon.0) 11591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[45495]: audit 2022-04-22T21:19:34.899989+0000 mgr.y (mgr.34107) 7976 : audit [DBG] from='client.43656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[45495]: cluster 2022-04-22T21:19:35.214312+0000 mgr.y (mgr.34107) 7977 : cluster [DBG] pgmap v8946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[45495]: audit 2022-04-22T21:19:35.879896+0000 mon.a (mon.0) 11591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[46934]: audit 2022-04-22T21:19:34.899989+0000 mgr.y (mgr.34107) 7976 : audit [DBG] from='client.43656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:19:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[46934]: cluster 2022-04-22T21:19:35.214312+0000 mgr.y (mgr.34107) 7977 : cluster [DBG] pgmap v8946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:36 smithi115 bash[46934]: audit 2022-04-22T21:19:35.879896+0000 mon.a (mon.0) 11591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:38.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:38 smithi121 bash[36902]: cluster 2022-04-22T21:19:37.215451+0000 mgr.y (mgr.34107) 7978 : cluster [DBG] pgmap v8947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:38 smithi115 bash[45495]: cluster 2022-04-22T21:19:37.215451+0000 mgr.y (mgr.34107) 7978 : cluster [DBG] pgmap v8947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:38 smithi115 bash[46934]: cluster 2022-04-22T21:19:37.215451+0000 mgr.y (mgr.34107) 7978 : cluster [DBG] pgmap v8947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:39 smithi121 bash[36902]: audit 2022-04-22T21:19:38.878641+0000 mon.a (mon.0) 11592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:39 smithi115 bash[45495]: audit 2022-04-22T21:19:38.878641+0000 mon.a (mon.0) 11592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:39 smithi115 bash[46934]: audit 2022-04-22T21:19:38.878641+0000 mon.a (mon.0) 11592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:40 smithi121 bash[36902]: cluster 2022-04-22T21:19:39.216519+0000 mgr.y (mgr.34107) 7979 : cluster [DBG] pgmap v8948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:40 smithi121 bash[36902]: audit 2022-04-22T21:19:39.703733+0000 mon.a (mon.0) 11593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:40 smithi115 bash[45495]: cluster 2022-04-22T21:19:39.216519+0000 mgr.y (mgr.34107) 7979 : cluster [DBG] pgmap v8948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:40.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:40 smithi115 bash[45495]: audit 2022-04-22T21:19:39.703733+0000 mon.a (mon.0) 11593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:40.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:40 smithi115 bash[46934]: cluster 2022-04-22T21:19:39.216519+0000 mgr.y (mgr.34107) 7979 : cluster [DBG] pgmap v8948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:40.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:40 smithi115 bash[46934]: audit 2022-04-22T21:19:39.703733+0000 mon.a (mon.0) 11593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:41.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:41 smithi121 bash[36902]: audit 2022-04-22T21:19:40.883052+0000 mon.a (mon.0) 11594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:41 smithi115 bash[46934]: audit 2022-04-22T21:19:40.883052+0000 mon.a (mon.0) 11594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:41 smithi115 bash[45495]: audit 2022-04-22T21:19:40.883052+0000 mon.a (mon.0) 11594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:42 smithi121 bash[36902]: cluster 2022-04-22T21:19:41.217583+0000 mgr.y (mgr.34107) 7980 : cluster [DBG] pgmap v8949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:42 smithi115 bash[46934]: cluster 2022-04-22T21:19:41.217583+0000 mgr.y (mgr.34107) 7980 : cluster [DBG] pgmap v8949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:42.524Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:42 smithi115 bash[45495]: cluster 2022-04-22T21:19:41.217583+0000 mgr.y (mgr.34107) 7980 : cluster [DBG] pgmap v8949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:44.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:44 smithi121 bash[36902]: cluster 2022-04-22T21:19:43.218660+0000 mgr.y (mgr.34107) 7981 : cluster [DBG] pgmap v8950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:44 smithi121 bash[36902]: audit 2022-04-22T21:19:43.881830+0000 mon.a (mon.0) 11595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:44.399Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:44 smithi115 bash[46934]: cluster 2022-04-22T21:19:43.218660+0000 mgr.y (mgr.34107) 7981 : cluster [DBG] pgmap v8950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:44 smithi115 bash[46934]: audit 2022-04-22T21:19:43.881830+0000 mon.a (mon.0) 11595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:44 smithi115 bash[45495]: cluster 2022-04-22T21:19:43.218660+0000 mgr.y (mgr.34107) 7981 : cluster [DBG] pgmap v8950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:44 smithi115 bash[45495]: audit 2022-04-22T21:19:43.881830+0000 mon.a (mon.0) 11595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:45.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:45 smithi121 bash[36902]: audit 2022-04-22T21:19:44.707299+0000 mon.a (mon.0) 11596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:45 smithi115 bash[46934]: audit 2022-04-22T21:19:44.707299+0000 mon.a (mon.0) 11596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:45 smithi115 bash[45495]: audit 2022-04-22T21:19:44.707299+0000 mon.a (mon.0) 11596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:46.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:46 smithi121 bash[36902]: cluster 2022-04-22T21:19:45.219790+0000 mgr.y (mgr.34107) 7982 : cluster [DBG] pgmap v8951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:46 smithi121 bash[36902]: audit 2022-04-22T21:19:45.886168+0000 mon.a (mon.0) 11597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:46 smithi115 bash[45495]: cluster 2022-04-22T21:19:45.219790+0000 mgr.y (mgr.34107) 7982 : cluster [DBG] pgmap v8951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:46 smithi115 bash[45495]: audit 2022-04-22T21:19:45.886168+0000 mon.a (mon.0) 11597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:46 smithi115 bash[46934]: cluster 2022-04-22T21:19:45.219790+0000 mgr.y (mgr.34107) 7982 : cluster [DBG] pgmap v8951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:46 smithi115 bash[46934]: audit 2022-04-22T21:19:45.886168+0000 mon.a (mon.0) 11597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:48 smithi121 bash[36902]: cluster 2022-04-22T21:19:47.220867+0000 mgr.y (mgr.34107) 7983 : cluster [DBG] pgmap v8952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:48 smithi115 bash[45495]: cluster 2022-04-22T21:19:47.220867+0000 mgr.y (mgr.34107) 7983 : cluster [DBG] pgmap v8952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:48 smithi115 bash[46934]: cluster 2022-04-22T21:19:47.220867+0000 mgr.y (mgr.34107) 7983 : cluster [DBG] pgmap v8952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:49 smithi121 bash[36902]: audit 2022-04-22T21:19:48.884940+0000 mon.a (mon.0) 11598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:49 smithi115 bash[46934]: audit 2022-04-22T21:19:48.884940+0000 mon.a (mon.0) 11598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:49 smithi115 bash[45495]: audit 2022-04-22T21:19:48.884940+0000 mon.a (mon.0) 11598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:50 smithi121 bash[36902]: cluster 2022-04-22T21:19:49.221915+0000 mgr.y (mgr.34107) 7984 : cluster [DBG] pgmap v8953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:50 smithi121 bash[36902]: audit 2022-04-22T21:19:49.711113+0000 mon.a (mon.0) 11599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:50 smithi115 bash[46934]: cluster 2022-04-22T21:19:49.221915+0000 mgr.y (mgr.34107) 7984 : cluster [DBG] pgmap v8953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:50 smithi115 bash[46934]: audit 2022-04-22T21:19:49.711113+0000 mon.a (mon.0) 11599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:50 smithi115 bash[45495]: cluster 2022-04-22T21:19:49.221915+0000 mgr.y (mgr.34107) 7984 : cluster [DBG] pgmap v8953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:50.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:50 smithi115 bash[45495]: audit 2022-04-22T21:19:49.711113+0000 mon.a (mon.0) 11599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:51 smithi121 bash[36902]: audit 2022-04-22T21:19:50.889331+0000 mon.a (mon.0) 11600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:51 smithi115 bash[45495]: audit 2022-04-22T21:19:50.889331+0000 mon.a (mon.0) 11600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:51 smithi115 bash[46934]: audit 2022-04-22T21:19:50.889331+0000 mon.a (mon.0) 11600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:52 smithi121 bash[36902]: cluster 2022-04-22T21:19:51.222928+0000 mgr.y (mgr.34107) 7985 : cluster [DBG] pgmap v8954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:52.525Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:52 smithi115 bash[45495]: cluster 2022-04-22T21:19:51.222928+0000 mgr.y (mgr.34107) 7985 : cluster [DBG] pgmap v8954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:52 smithi115 bash[46934]: cluster 2022-04-22T21:19:51.222928+0000 mgr.y (mgr.34107) 7985 : cluster [DBG] pgmap v8954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:54 smithi121 bash[36902]: cluster 2022-04-22T21:19:53.224108+0000 mgr.y (mgr.34107) 7986 : cluster [DBG] pgmap v8955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:54 smithi121 bash[36902]: audit 2022-04-22T21:19:53.888139+0000 mon.a (mon.0) 11601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:19:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:19:54.400Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:19:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:54 smithi115 bash[45495]: cluster 2022-04-22T21:19:53.224108+0000 mgr.y (mgr.34107) 7986 : cluster [DBG] pgmap v8955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:54 smithi115 bash[45495]: audit 2022-04-22T21:19:53.888139+0000 mon.a (mon.0) 11601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:54 smithi115 bash[46934]: cluster 2022-04-22T21:19:53.224108+0000 mgr.y (mgr.34107) 7986 : cluster [DBG] pgmap v8955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:54 smithi115 bash[46934]: audit 2022-04-22T21:19:53.888139+0000 mon.a (mon.0) 11601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:55 smithi121 bash[36902]: audit 2022-04-22T21:19:54.714668+0000 mon.a (mon.0) 11602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:55 smithi115 bash[45495]: audit 2022-04-22T21:19:54.714668+0000 mon.a (mon.0) 11602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:55 smithi115 bash[46934]: audit 2022-04-22T21:19:54.714668+0000 mon.a (mon.0) 11602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:19:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:56 smithi121 bash[36902]: cluster 2022-04-22T21:19:55.225199+0000 mgr.y (mgr.34107) 7987 : cluster [DBG] pgmap v8956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:56 smithi121 bash[36902]: audit 2022-04-22T21:19:55.892634+0000 mon.a (mon.0) 11603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:56 smithi115 bash[46934]: cluster 2022-04-22T21:19:55.225199+0000 mgr.y (mgr.34107) 7987 : cluster [DBG] pgmap v8956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:56 smithi115 bash[46934]: audit 2022-04-22T21:19:55.892634+0000 mon.a (mon.0) 11603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:56 smithi115 bash[45495]: cluster 2022-04-22T21:19:55.225199+0000 mgr.y (mgr.34107) 7987 : cluster [DBG] pgmap v8956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:56 smithi115 bash[45495]: audit 2022-04-22T21:19:55.892634+0000 mon.a (mon.0) 11603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:19:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:58 smithi121 bash[36902]: cluster 2022-04-22T21:19:57.226282+0000 mgr.y (mgr.34107) 7988 : cluster [DBG] pgmap v8957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:58 smithi115 bash[46934]: cluster 2022-04-22T21:19:57.226282+0000 mgr.y (mgr.34107) 7988 : cluster [DBG] pgmap v8957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:58 smithi115 bash[45495]: cluster 2022-04-22T21:19:57.226282+0000 mgr.y (mgr.34107) 7988 : cluster [DBG] pgmap v8957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:19:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:19:59 smithi115 bash[45495]: audit 2022-04-22T21:19:58.891065+0000 mon.a (mon.0) 11604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:19:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:19:59 smithi115 bash[46934]: audit 2022-04-22T21:19:58.891065+0000 mon.a (mon.0) 11604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:19:59 smithi121 bash[36902]: audit 2022-04-22T21:19:58.891065+0000 mon.a (mon.0) 11604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[45495]: cluster 2022-04-22T21:19:59.227363+0000 mgr.y (mgr.34107) 7989 : cluster [DBG] pgmap v8958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[45495]: audit 2022-04-22T21:19:59.718085+0000 mon.a (mon.0) 11605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:00.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[45495]: cluster 2022-04-22T21:20:00.000124+0000 mon.a (mon.0) 11606 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:20:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[46934]: cluster 2022-04-22T21:19:59.227363+0000 mgr.y (mgr.34107) 7989 : cluster [DBG] pgmap v8958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[46934]: audit 2022-04-22T21:19:59.718085+0000 mon.a (mon.0) 11605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:00 smithi115 bash[46934]: cluster 2022-04-22T21:20:00.000124+0000 mon.a (mon.0) 11606 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:20:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:00 smithi121 bash[36902]: cluster 2022-04-22T21:19:59.227363+0000 mgr.y (mgr.34107) 7989 : cluster [DBG] pgmap v8958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:00 smithi121 bash[36902]: audit 2022-04-22T21:19:59.718085+0000 mon.a (mon.0) 11605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:00 smithi121 bash[36902]: cluster 2022-04-22T21:20:00.000124+0000 mon.a (mon.0) 11606 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:20:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:01 smithi115 bash[45495]: audit 2022-04-22T21:20:00.895805+0000 mon.a (mon.0) 11607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:01 smithi115 bash[46934]: audit 2022-04-22T21:20:00.895805+0000 mon.a (mon.0) 11607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:01 smithi121 bash[36902]: audit 2022-04-22T21:20:00.895805+0000 mon.a (mon.0) 11607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:02 smithi115 bash[45495]: cluster 2022-04-22T21:20:01.228620+0000 mgr.y (mgr.34107) 7990 : cluster [DBG] pgmap v8959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:02.526Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:02 smithi115 bash[46934]: cluster 2022-04-22T21:20:01.228620+0000 mgr.y (mgr.34107) 7990 : cluster [DBG] pgmap v8959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:02 smithi121 bash[36902]: cluster 2022-04-22T21:20:01.228620+0000 mgr.y (mgr.34107) 7990 : cluster [DBG] pgmap v8959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:04 smithi115 bash[45495]: cluster 2022-04-22T21:20:03.229905+0000 mgr.y (mgr.34107) 7991 : cluster [DBG] pgmap v8960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:04 smithi115 bash[45495]: audit 2022-04-22T21:20:03.894281+0000 mon.a (mon.0) 11608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:04.400Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:04 smithi115 bash[46934]: cluster 2022-04-22T21:20:03.229905+0000 mgr.y (mgr.34107) 7991 : cluster [DBG] pgmap v8960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:04 smithi115 bash[46934]: audit 2022-04-22T21:20:03.894281+0000 mon.a (mon.0) 11608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:04 smithi121 bash[36902]: cluster 2022-04-22T21:20:03.229905+0000 mgr.y (mgr.34107) 7991 : cluster [DBG] pgmap v8960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:04 smithi121 bash[36902]: audit 2022-04-22T21:20:03.894281+0000 mon.a (mon.0) 11608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:05.705 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:20:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:05 smithi115 bash[45495]: audit 2022-04-22T21:20:04.721525+0000 mon.a (mon.0) 11609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:05 smithi115 bash[46934]: audit 2022-04-22T21:20:04.721525+0000 mon.a (mon.0) 11609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:05 smithi121 bash[36902]: audit 2022-04-22T21:20:04.721525+0000 mon.a (mon.0) 11609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:20:06.058 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:20:06.059 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 3h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:20:06.060 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:20:06.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:20:06.466 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:20:06.467 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:20:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[45495]: cluster 2022-04-22T21:20:05.231100+0000 mgr.y (mgr.34107) 7992 : cluster [DBG] pgmap v8961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[45495]: audit 2022-04-22T21:20:05.685266+0000 mgr.y (mgr.34107) 7993 : audit [DBG] from='client.53599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[45495]: audit 2022-04-22T21:20:05.898878+0000 mon.a (mon.0) 11610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[45495]: audit 2022-04-22T21:20:06.461055+0000 mon.a (mon.0) 11611 : audit [DBG] from='client.? 172.21.15.115:0/1547161924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[46934]: cluster 2022-04-22T21:20:05.231100+0000 mgr.y (mgr.34107) 7992 : cluster [DBG] pgmap v8961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[46934]: audit 2022-04-22T21:20:05.685266+0000 mgr.y (mgr.34107) 7993 : audit [DBG] from='client.53599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[46934]: audit 2022-04-22T21:20:05.898878+0000 mon.a (mon.0) 11610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:06 smithi115 bash[46934]: audit 2022-04-22T21:20:06.461055+0000 mon.a (mon.0) 11611 : audit [DBG] from='client.? 172.21.15.115:0/1547161924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:06 smithi121 bash[36902]: cluster 2022-04-22T21:20:05.231100+0000 mgr.y (mgr.34107) 7992 : cluster [DBG] pgmap v8961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:06 smithi121 bash[36902]: audit 2022-04-22T21:20:05.685266+0000 mgr.y (mgr.34107) 7993 : audit [DBG] from='client.53599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:06 smithi121 bash[36902]: audit 2022-04-22T21:20:05.898878+0000 mon.a (mon.0) 11610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:06 smithi121 bash[36902]: audit 2022-04-22T21:20:06.461055+0000 mon.a (mon.0) 11611 : audit [DBG] from='client.? 172.21.15.115:0/1547161924' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:08.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:07 smithi121 bash[36902]: audit 2022-04-22T21:20:06.033623+0000 mgr.y (mgr.34107) 7994 : audit [DBG] from='client.53605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:07 smithi115 bash[45495]: audit 2022-04-22T21:20:06.033623+0000 mgr.y (mgr.34107) 7994 : audit [DBG] from='client.53605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:07 smithi115 bash[46934]: audit 2022-04-22T21:20:06.033623+0000 mgr.y (mgr.34107) 7994 : audit [DBG] from='client.53605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:08 smithi121 bash[36902]: cluster 2022-04-22T21:20:07.232401+0000 mgr.y (mgr.34107) 7995 : cluster [DBG] pgmap v8962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:08 smithi115 bash[45495]: cluster 2022-04-22T21:20:07.232401+0000 mgr.y (mgr.34107) 7995 : cluster [DBG] pgmap v8962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:08 smithi115 bash[46934]: cluster 2022-04-22T21:20:07.232401+0000 mgr.y (mgr.34107) 7995 : cluster [DBG] pgmap v8962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:09 smithi121 bash[36902]: audit 2022-04-22T21:20:08.897517+0000 mon.a (mon.0) 11612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:09 smithi115 bash[45495]: audit 2022-04-22T21:20:08.897517+0000 mon.a (mon.0) 11612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:09 smithi115 bash[46934]: audit 2022-04-22T21:20:08.897517+0000 mon.a (mon.0) 11612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:10 smithi121 bash[36902]: cluster 2022-04-22T21:20:09.233569+0000 mgr.y (mgr.34107) 7996 : cluster [DBG] pgmap v8963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:10 smithi121 bash[36902]: audit 2022-04-22T21:20:09.725263+0000 mon.a (mon.0) 11613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:10 smithi115 bash[46934]: cluster 2022-04-22T21:20:09.233569+0000 mgr.y (mgr.34107) 7996 : cluster [DBG] pgmap v8963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:10 smithi115 bash[46934]: audit 2022-04-22T21:20:09.725263+0000 mon.a (mon.0) 11613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:11.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:10 smithi115 bash[45495]: cluster 2022-04-22T21:20:09.233569+0000 mgr.y (mgr.34107) 7996 : cluster [DBG] pgmap v8963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:11.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:10 smithi115 bash[45495]: audit 2022-04-22T21:20:09.725263+0000 mon.a (mon.0) 11613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:11 smithi121 bash[36902]: audit 2022-04-22T21:20:10.901889+0000 mon.a (mon.0) 11614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:11 smithi115 bash[45495]: audit 2022-04-22T21:20:10.901889+0000 mon.a (mon.0) 11614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:11 smithi115 bash[46934]: audit 2022-04-22T21:20:10.901889+0000 mon.a (mon.0) 11614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:12 smithi115 bash[45495]: cluster 2022-04-22T21:20:11.234764+0000 mgr.y (mgr.34107) 7997 : cluster [DBG] pgmap v8964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:12 smithi115 bash[46934]: cluster 2022-04-22T21:20:11.234764+0000 mgr.y (mgr.34107) 7997 : cluster [DBG] pgmap v8964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:12.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:12.525Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:12 smithi121 bash[36902]: cluster 2022-04-22T21:20:11.234764+0000 mgr.y (mgr.34107) 7997 : cluster [DBG] pgmap v8964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:14.400Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:14 smithi115 bash[46934]: cluster 2022-04-22T21:20:13.235967+0000 mgr.y (mgr.34107) 7998 : cluster [DBG] pgmap v8965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:14 smithi115 bash[46934]: audit 2022-04-22T21:20:13.900558+0000 mon.a (mon.0) 11615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:14 smithi115 bash[45495]: cluster 2022-04-22T21:20:13.235967+0000 mgr.y (mgr.34107) 7998 : cluster [DBG] pgmap v8965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:14 smithi115 bash[45495]: audit 2022-04-22T21:20:13.900558+0000 mon.a (mon.0) 11615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:14.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:14 smithi121 bash[36902]: cluster 2022-04-22T21:20:13.235967+0000 mgr.y (mgr.34107) 7998 : cluster [DBG] pgmap v8965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:14.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:14 smithi121 bash[36902]: audit 2022-04-22T21:20:13.900558+0000 mon.a (mon.0) 11615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:16.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:15 smithi121 bash[36902]: audit 2022-04-22T21:20:14.728953+0000 mon.a (mon.0) 11616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:15 smithi121 bash[36902]: audit 2022-04-22T21:20:14.811563+0000 mon.a (mon.0) 11617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:20:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:15 smithi121 bash[36902]: audit 2022-04-22T21:20:14.814579+0000 mon.a (mon.0) 11618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:20:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:15 smithi121 bash[36902]: audit 2022-04-22T21:20:14.816354+0000 mon.a (mon.0) 11619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:20:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:15 smithi121 bash[36902]: audit 2022-04-22T21:20:15.176900+0000 mon.a (mon.0) 11620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:20:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[46934]: audit 2022-04-22T21:20:14.728953+0000 mon.a (mon.0) 11616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[46934]: audit 2022-04-22T21:20:14.811563+0000 mon.a (mon.0) 11617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:20:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[46934]: audit 2022-04-22T21:20:14.814579+0000 mon.a (mon.0) 11618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:20:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[46934]: audit 2022-04-22T21:20:14.816354+0000 mon.a (mon.0) 11619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:20:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[46934]: audit 2022-04-22T21:20:15.176900+0000 mon.a (mon.0) 11620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:20:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[45495]: audit 2022-04-22T21:20:14.728953+0000 mon.a (mon.0) 11616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[45495]: audit 2022-04-22T21:20:14.811563+0000 mon.a (mon.0) 11617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:20:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[45495]: audit 2022-04-22T21:20:14.814579+0000 mon.a (mon.0) 11618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:20:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[45495]: audit 2022-04-22T21:20:14.816354+0000 mon.a (mon.0) 11619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:20:16.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:15 smithi115 bash[45495]: audit 2022-04-22T21:20:15.176900+0000 mon.a (mon.0) 11620 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:20:17.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:16 smithi121 bash[36902]: cluster 2022-04-22T21:20:15.237008+0000 mgr.y (mgr.34107) 7999 : cluster [DBG] pgmap v8966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:16 smithi121 bash[36902]: audit 2022-04-22T21:20:15.904942+0000 mon.a (mon.0) 11621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:16 smithi121 bash[36902]: audit 2022-04-22T21:20:15.975653+0000 mon.a (mon.0) 11622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:20:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:16 smithi121 bash[36902]: audit 2022-04-22T21:20:15.978437+0000 mon.a (mon.0) 11623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[45495]: cluster 2022-04-22T21:20:15.237008+0000 mgr.y (mgr.34107) 7999 : cluster [DBG] pgmap v8966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[45495]: audit 2022-04-22T21:20:15.904942+0000 mon.a (mon.0) 11621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[45495]: audit 2022-04-22T21:20:15.975653+0000 mon.a (mon.0) 11622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[45495]: audit 2022-04-22T21:20:15.978437+0000 mon.a (mon.0) 11623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:20:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[46934]: cluster 2022-04-22T21:20:15.237008+0000 mgr.y (mgr.34107) 7999 : cluster [DBG] pgmap v8966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[46934]: audit 2022-04-22T21:20:15.904942+0000 mon.a (mon.0) 11621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[46934]: audit 2022-04-22T21:20:15.975653+0000 mon.a (mon.0) 11622 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:20:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:16 smithi115 bash[46934]: audit 2022-04-22T21:20:15.978437+0000 mon.a (mon.0) 11623 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:20:19.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:18 smithi121 bash[36902]: cluster 2022-04-22T21:20:17.238124+0000 mgr.y (mgr.34107) 8000 : cluster [DBG] pgmap v8967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:18 smithi115 bash[45495]: cluster 2022-04-22T21:20:17.238124+0000 mgr.y (mgr.34107) 8000 : cluster [DBG] pgmap v8967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:18 smithi115 bash[46934]: cluster 2022-04-22T21:20:17.238124+0000 mgr.y (mgr.34107) 8000 : cluster [DBG] pgmap v8967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:19 smithi121 bash[36902]: audit 2022-04-22T21:20:18.903638+0000 mon.a (mon.0) 11624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:19 smithi115 bash[45495]: audit 2022-04-22T21:20:18.903638+0000 mon.a (mon.0) 11624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:19 smithi115 bash[46934]: audit 2022-04-22T21:20:18.903638+0000 mon.a (mon.0) 11624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:20 smithi121 bash[36902]: cluster 2022-04-22T21:20:19.239208+0000 mgr.y (mgr.34107) 8001 : cluster [DBG] pgmap v8968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:20 smithi121 bash[36902]: audit 2022-04-22T21:20:19.732764+0000 mon.a (mon.0) 11625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:20 smithi115 bash[45495]: cluster 2022-04-22T21:20:19.239208+0000 mgr.y (mgr.34107) 8001 : cluster [DBG] pgmap v8968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:20 smithi115 bash[45495]: audit 2022-04-22T21:20:19.732764+0000 mon.a (mon.0) 11625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:20 smithi115 bash[46934]: cluster 2022-04-22T21:20:19.239208+0000 mgr.y (mgr.34107) 8001 : cluster [DBG] pgmap v8968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:20 smithi115 bash[46934]: audit 2022-04-22T21:20:19.732764+0000 mon.a (mon.0) 11625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:21 smithi121 bash[36902]: audit 2022-04-22T21:20:20.907882+0000 mon.a (mon.0) 11626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:21 smithi115 bash[45495]: audit 2022-04-22T21:20:20.907882+0000 mon.a (mon.0) 11626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:21 smithi115 bash[46934]: audit 2022-04-22T21:20:20.907882+0000 mon.a (mon.0) 11626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:22 smithi115 bash[45495]: cluster 2022-04-22T21:20:21.240249+0000 mgr.y (mgr.34107) 8002 : cluster [DBG] pgmap v8969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:22 smithi115 bash[46934]: cluster 2022-04-22T21:20:21.240249+0000 mgr.y (mgr.34107) 8002 : cluster [DBG] pgmap v8969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:22.527Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:22 smithi121 bash[36902]: cluster 2022-04-22T21:20:21.240249+0000 mgr.y (mgr.34107) 8002 : cluster [DBG] pgmap v8969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:24.402Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:24 smithi115 bash[45495]: cluster 2022-04-22T21:20:23.241383+0000 mgr.y (mgr.34107) 8003 : cluster [DBG] pgmap v8970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:24 smithi115 bash[45495]: audit 2022-04-22T21:20:23.906756+0000 mon.a (mon.0) 11627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:24 smithi115 bash[46934]: cluster 2022-04-22T21:20:23.241383+0000 mgr.y (mgr.34107) 8003 : cluster [DBG] pgmap v8970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:24 smithi115 bash[46934]: audit 2022-04-22T21:20:23.906756+0000 mon.a (mon.0) 11627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:24 smithi121 bash[36902]: cluster 2022-04-22T21:20:23.241383+0000 mgr.y (mgr.34107) 8003 : cluster [DBG] pgmap v8970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:24 smithi121 bash[36902]: audit 2022-04-22T21:20:23.906756+0000 mon.a (mon.0) 11627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:25 smithi121 bash[36902]: audit 2022-04-22T21:20:24.736232+0000 mon.a (mon.0) 11628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:25 smithi115 bash[45495]: audit 2022-04-22T21:20:24.736232+0000 mon.a (mon.0) 11628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:25 smithi115 bash[46934]: audit 2022-04-22T21:20:24.736232+0000 mon.a (mon.0) 11628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:26 smithi121 bash[36902]: cluster 2022-04-22T21:20:25.242557+0000 mgr.y (mgr.34107) 8004 : cluster [DBG] pgmap v8971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:26 smithi121 bash[36902]: audit 2022-04-22T21:20:25.910926+0000 mon.a (mon.0) 11629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:26 smithi115 bash[45495]: cluster 2022-04-22T21:20:25.242557+0000 mgr.y (mgr.34107) 8004 : cluster [DBG] pgmap v8971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:26 smithi115 bash[45495]: audit 2022-04-22T21:20:25.910926+0000 mon.a (mon.0) 11629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:26 smithi115 bash[46934]: cluster 2022-04-22T21:20:25.242557+0000 mgr.y (mgr.34107) 8004 : cluster [DBG] pgmap v8971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:26 smithi115 bash[46934]: audit 2022-04-22T21:20:25.910926+0000 mon.a (mon.0) 11629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:28 smithi121 bash[36902]: cluster 2022-04-22T21:20:27.243704+0000 mgr.y (mgr.34107) 8005 : cluster [DBG] pgmap v8972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:28 smithi115 bash[45495]: cluster 2022-04-22T21:20:27.243704+0000 mgr.y (mgr.34107) 8005 : cluster [DBG] pgmap v8972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:28 smithi115 bash[46934]: cluster 2022-04-22T21:20:27.243704+0000 mgr.y (mgr.34107) 8005 : cluster [DBG] pgmap v8972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:29 smithi121 bash[36902]: audit 2022-04-22T21:20:28.909782+0000 mon.a (mon.0) 11630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:29 smithi115 bash[45495]: audit 2022-04-22T21:20:28.909782+0000 mon.a (mon.0) 11630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:29 smithi115 bash[46934]: audit 2022-04-22T21:20:28.909782+0000 mon.a (mon.0) 11630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:30 smithi121 bash[36902]: cluster 2022-04-22T21:20:29.244903+0000 mgr.y (mgr.34107) 8006 : cluster [DBG] pgmap v8973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:31.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:30 smithi121 bash[36902]: audit 2022-04-22T21:20:29.739293+0000 mon.a (mon.0) 11631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:30 smithi115 bash[45495]: cluster 2022-04-22T21:20:29.244903+0000 mgr.y (mgr.34107) 8006 : cluster [DBG] pgmap v8973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:30 smithi115 bash[45495]: audit 2022-04-22T21:20:29.739293+0000 mon.a (mon.0) 11631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:30 smithi115 bash[46934]: cluster 2022-04-22T21:20:29.244903+0000 mgr.y (mgr.34107) 8006 : cluster [DBG] pgmap v8973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:30 smithi115 bash[46934]: audit 2022-04-22T21:20:29.739293+0000 mon.a (mon.0) 11631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:31 smithi121 bash[36902]: audit 2022-04-22T21:20:30.913727+0000 mon.a (mon.0) 11632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:31 smithi115 bash[45495]: audit 2022-04-22T21:20:30.913727+0000 mon.a (mon.0) 11632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:31 smithi115 bash[46934]: audit 2022-04-22T21:20:30.913727+0000 mon.a (mon.0) 11632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:32 smithi115 bash[45495]: cluster 2022-04-22T21:20:31.246120+0000 mgr.y (mgr.34107) 8007 : cluster [DBG] pgmap v8974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:32.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:32 smithi115 bash[46934]: cluster 2022-04-22T21:20:31.246120+0000 mgr.y (mgr.34107) 8007 : cluster [DBG] pgmap v8974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:32 smithi121 bash[36902]: cluster 2022-04-22T21:20:31.246120+0000 mgr.y (mgr.34107) 8007 : cluster [DBG] pgmap v8974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:34.657 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:34.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:34.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:34 smithi115 bash[45495]: cluster 2022-04-22T21:20:33.247349+0000 mgr.y (mgr.34107) 8008 : cluster [DBG] pgmap v8975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:34.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:34 smithi115 bash[45495]: audit 2022-04-22T21:20:33.912753+0000 mon.a (mon.0) 11633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:34.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:34 smithi115 bash[46934]: cluster 2022-04-22T21:20:33.247349+0000 mgr.y (mgr.34107) 8008 : cluster [DBG] pgmap v8975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:34.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:34 smithi115 bash[46934]: audit 2022-04-22T21:20:33.912753+0000 mon.a (mon.0) 11633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:34 smithi121 bash[36902]: cluster 2022-04-22T21:20:33.247349+0000 mgr.y (mgr.34107) 8008 : cluster [DBG] pgmap v8975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:34 smithi121 bash[36902]: audit 2022-04-22T21:20:33.912753+0000 mon.a (mon.0) 11633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:35 smithi121 bash[36902]: audit 2022-04-22T21:20:34.742876+0000 mon.a (mon.0) 11634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:35 smithi115 bash[45495]: audit 2022-04-22T21:20:34.742876+0000 mon.a (mon.0) 11634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:35 smithi115 bash[46934]: audit 2022-04-22T21:20:34.742876+0000 mon.a (mon.0) 11634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:36.844 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:20:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:36 smithi121 bash[36902]: cluster 2022-04-22T21:20:35.248500+0000 mgr.y (mgr.34107) 8009 : cluster [DBG] pgmap v8976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:36 smithi121 bash[36902]: audit 2022-04-22T21:20:35.916455+0000 mon.a (mon.0) 11635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:36 smithi115 bash[45495]: cluster 2022-04-22T21:20:35.248500+0000 mgr.y (mgr.34107) 8009 : cluster [DBG] pgmap v8976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:36 smithi115 bash[45495]: audit 2022-04-22T21:20:35.916455+0000 mon.a (mon.0) 11635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:36 smithi115 bash[46934]: cluster 2022-04-22T21:20:35.248500+0000 mgr.y (mgr.34107) 8009 : cluster [DBG] pgmap v8976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:36 smithi115 bash[46934]: audit 2022-04-22T21:20:35.916455+0000 mon.a (mon.0) 11635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:37.190 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:20:37.190 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:20:37.190 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 7m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:20:37.190 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 6m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 6m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 7m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 6m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 6m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:20:37.191 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 3h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 3h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:20:37.192 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:20:37.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:20:37.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:20:37.598 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:20:38.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:37 smithi121 bash[36902]: audit 2022-04-22T21:20:36.823512+0000 mgr.y (mgr.34107) 8010 : audit [DBG] from='client.53617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:38.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:37 smithi121 bash[36902]: audit 2022-04-22T21:20:37.592086+0000 mon.c (mon.1) 288 : audit [DBG] from='client.? 172.21.15.115:0/1487628125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:37 smithi115 bash[45495]: audit 2022-04-22T21:20:36.823512+0000 mgr.y (mgr.34107) 8010 : audit [DBG] from='client.53617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:37 smithi115 bash[45495]: audit 2022-04-22T21:20:37.592086+0000 mon.c (mon.1) 288 : audit [DBG] from='client.? 172.21.15.115:0/1487628125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:38.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:37 smithi115 bash[46934]: audit 2022-04-22T21:20:36.823512+0000 mgr.y (mgr.34107) 8010 : audit [DBG] from='client.53617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:38.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:37 smithi115 bash[46934]: audit 2022-04-22T21:20:37.592086+0000 mon.c (mon.1) 288 : audit [DBG] from='client.? 172.21.15.115:0/1487628125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:20:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:38 smithi121 bash[36902]: audit 2022-04-22T21:20:37.166833+0000 mgr.y (mgr.34107) 8011 : audit [DBG] from='client.53623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:38 smithi121 bash[36902]: cluster 2022-04-22T21:20:37.249679+0000 mgr.y (mgr.34107) 8012 : cluster [DBG] pgmap v8977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:38 smithi115 bash[45495]: audit 2022-04-22T21:20:37.166833+0000 mgr.y (mgr.34107) 8011 : audit [DBG] from='client.53623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:38 smithi115 bash[45495]: cluster 2022-04-22T21:20:37.249679+0000 mgr.y (mgr.34107) 8012 : cluster [DBG] pgmap v8977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:38 smithi115 bash[46934]: audit 2022-04-22T21:20:37.166833+0000 mgr.y (mgr.34107) 8011 : audit [DBG] from='client.53623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:20:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:38 smithi115 bash[46934]: cluster 2022-04-22T21:20:37.249679+0000 mgr.y (mgr.34107) 8012 : cluster [DBG] pgmap v8977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:39 smithi121 bash[36902]: audit 2022-04-22T21:20:38.915702+0000 mon.a (mon.0) 11636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:39 smithi115 bash[45495]: audit 2022-04-22T21:20:38.915702+0000 mon.a (mon.0) 11636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:39 smithi115 bash[46934]: audit 2022-04-22T21:20:38.915702+0000 mon.a (mon.0) 11636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:40 smithi121 bash[36902]: cluster 2022-04-22T21:20:39.250884+0000 mgr.y (mgr.34107) 8013 : cluster [DBG] pgmap v8978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:40 smithi121 bash[36902]: audit 2022-04-22T21:20:39.746578+0000 mon.a (mon.0) 11637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:40 smithi115 bash[45495]: cluster 2022-04-22T21:20:39.250884+0000 mgr.y (mgr.34107) 8013 : cluster [DBG] pgmap v8978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:40 smithi115 bash[45495]: audit 2022-04-22T21:20:39.746578+0000 mon.a (mon.0) 11637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:40 smithi115 bash[46934]: cluster 2022-04-22T21:20:39.250884+0000 mgr.y (mgr.34107) 8013 : cluster [DBG] pgmap v8978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:40 smithi115 bash[46934]: audit 2022-04-22T21:20:39.746578+0000 mon.a (mon.0) 11637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:42.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:41 smithi121 bash[36902]: audit 2022-04-22T21:20:40.919360+0000 mon.a (mon.0) 11638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:41 smithi115 bash[45495]: audit 2022-04-22T21:20:40.919360+0000 mon.a (mon.0) 11638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:41 smithi115 bash[46934]: audit 2022-04-22T21:20:40.919360+0000 mon.a (mon.0) 11638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:42.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:42 smithi115 bash[45495]: cluster 2022-04-22T21:20:41.252050+0000 mgr.y (mgr.34107) 8014 : cluster [DBG] pgmap v8979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:42 smithi115 bash[46934]: cluster 2022-04-22T21:20:41.252050+0000 mgr.y (mgr.34107) 8014 : cluster [DBG] pgmap v8979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:42.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:42.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:42 smithi121 bash[36902]: cluster 2022-04-22T21:20:41.252050+0000 mgr.y (mgr.34107) 8014 : cluster [DBG] pgmap v8979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:44.693 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:44.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:44 smithi121 bash[36902]: cluster 2022-04-22T21:20:43.253181+0000 mgr.y (mgr.34107) 8015 : cluster [DBG] pgmap v8980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:44 smithi121 bash[36902]: audit 2022-04-22T21:20:43.918829+0000 mon.a (mon.0) 11639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:44 smithi115 bash[45495]: cluster 2022-04-22T21:20:43.253181+0000 mgr.y (mgr.34107) 8015 : cluster [DBG] pgmap v8980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:44 smithi115 bash[45495]: audit 2022-04-22T21:20:43.918829+0000 mon.a (mon.0) 11639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:44 smithi115 bash[46934]: cluster 2022-04-22T21:20:43.253181+0000 mgr.y (mgr.34107) 8015 : cluster [DBG] pgmap v8980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:44 smithi115 bash[46934]: audit 2022-04-22T21:20:43.918829+0000 mon.a (mon.0) 11639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:45 smithi121 bash[36902]: audit 2022-04-22T21:20:44.749892+0000 mon.a (mon.0) 11640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:45 smithi115 bash[45495]: audit 2022-04-22T21:20:44.749892+0000 mon.a (mon.0) 11640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:45 smithi115 bash[46934]: audit 2022-04-22T21:20:44.749892+0000 mon.a (mon.0) 11640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:47.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:46 smithi121 bash[36902]: cluster 2022-04-22T21:20:45.254290+0000 mgr.y (mgr.34107) 8016 : cluster [DBG] pgmap v8981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:46 smithi121 bash[36902]: audit 2022-04-22T21:20:45.922400+0000 mon.a (mon.0) 11641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:46 smithi115 bash[45495]: cluster 2022-04-22T21:20:45.254290+0000 mgr.y (mgr.34107) 8016 : cluster [DBG] pgmap v8981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:46 smithi115 bash[45495]: audit 2022-04-22T21:20:45.922400+0000 mon.a (mon.0) 11641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:46 smithi115 bash[46934]: cluster 2022-04-22T21:20:45.254290+0000 mgr.y (mgr.34107) 8016 : cluster [DBG] pgmap v8981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:46 smithi115 bash[46934]: audit 2022-04-22T21:20:45.922400+0000 mon.a (mon.0) 11641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:48 smithi121 bash[36902]: cluster 2022-04-22T21:20:47.255407+0000 mgr.y (mgr.34107) 8017 : cluster [DBG] pgmap v8982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:49.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:48 smithi115 bash[45495]: cluster 2022-04-22T21:20:47.255407+0000 mgr.y (mgr.34107) 8017 : cluster [DBG] pgmap v8982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:49.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:48 smithi115 bash[46934]: cluster 2022-04-22T21:20:47.255407+0000 mgr.y (mgr.34107) 8017 : cluster [DBG] pgmap v8982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:49 smithi121 bash[36902]: audit 2022-04-22T21:20:48.922005+0000 mon.a (mon.0) 11642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:49 smithi115 bash[45495]: audit 2022-04-22T21:20:48.922005+0000 mon.a (mon.0) 11642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:49 smithi115 bash[46934]: audit 2022-04-22T21:20:48.922005+0000 mon.a (mon.0) 11642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:50 smithi121 bash[36902]: cluster 2022-04-22T21:20:49.256407+0000 mgr.y (mgr.34107) 8018 : cluster [DBG] pgmap v8983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:50 smithi121 bash[36902]: audit 2022-04-22T21:20:49.753737+0000 mon.a (mon.0) 11643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:50 smithi115 bash[45495]: cluster 2022-04-22T21:20:49.256407+0000 mgr.y (mgr.34107) 8018 : cluster [DBG] pgmap v8983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:51.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:50 smithi115 bash[45495]: audit 2022-04-22T21:20:49.753737+0000 mon.a (mon.0) 11643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:50 smithi115 bash[46934]: cluster 2022-04-22T21:20:49.256407+0000 mgr.y (mgr.34107) 8018 : cluster [DBG] pgmap v8983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:50 smithi115 bash[46934]: audit 2022-04-22T21:20:49.753737+0000 mon.a (mon.0) 11643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:52.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:51 smithi121 bash[36902]: audit 2022-04-22T21:20:50.925469+0000 mon.a (mon.0) 11644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:51 smithi115 bash[45495]: audit 2022-04-22T21:20:50.925469+0000 mon.a (mon.0) 11644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:51 smithi115 bash[46934]: audit 2022-04-22T21:20:50.925469+0000 mon.a (mon.0) 11644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:52 smithi115 bash[45495]: cluster 2022-04-22T21:20:51.257479+0000 mgr.y (mgr.34107) 8019 : cluster [DBG] pgmap v8984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:52.527Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:52 smithi115 bash[46934]: cluster 2022-04-22T21:20:51.257479+0000 mgr.y (mgr.34107) 8019 : cluster [DBG] pgmap v8984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:52 smithi121 bash[36902]: cluster 2022-04-22T21:20:51.257479+0000 mgr.y (mgr.34107) 8019 : cluster [DBG] pgmap v8984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:54.738 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:20:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:20:54.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:20:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:54 smithi121 bash[36902]: cluster 2022-04-22T21:20:53.258632+0000 mgr.y (mgr.34107) 8020 : cluster [DBG] pgmap v8985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:54 smithi121 bash[36902]: audit 2022-04-22T21:20:53.924946+0000 mon.a (mon.0) 11645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:54 smithi115 bash[45495]: cluster 2022-04-22T21:20:53.258632+0000 mgr.y (mgr.34107) 8020 : cluster [DBG] pgmap v8985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:54 smithi115 bash[45495]: audit 2022-04-22T21:20:53.924946+0000 mon.a (mon.0) 11645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:54 smithi115 bash[46934]: cluster 2022-04-22T21:20:53.258632+0000 mgr.y (mgr.34107) 8020 : cluster [DBG] pgmap v8985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:54 smithi115 bash[46934]: audit 2022-04-22T21:20:53.924946+0000 mon.a (mon.0) 11645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:20:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:55 smithi121 bash[36902]: audit 2022-04-22T21:20:54.757470+0000 mon.a (mon.0) 11646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:55 smithi115 bash[45495]: audit 2022-04-22T21:20:54.757470+0000 mon.a (mon.0) 11646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:55 smithi115 bash[46934]: audit 2022-04-22T21:20:54.757470+0000 mon.a (mon.0) 11646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:20:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:56 smithi121 bash[36902]: cluster 2022-04-22T21:20:55.259862+0000 mgr.y (mgr.34107) 8021 : cluster [DBG] pgmap v8986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:56 smithi121 bash[36902]: audit 2022-04-22T21:20:55.928548+0000 mon.a (mon.0) 11647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:56 smithi115 bash[45495]: cluster 2022-04-22T21:20:55.259862+0000 mgr.y (mgr.34107) 8021 : cluster [DBG] pgmap v8986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:56 smithi115 bash[45495]: audit 2022-04-22T21:20:55.928548+0000 mon.a (mon.0) 11647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:56 smithi115 bash[46934]: cluster 2022-04-22T21:20:55.259862+0000 mgr.y (mgr.34107) 8021 : cluster [DBG] pgmap v8986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:56 smithi115 bash[46934]: audit 2022-04-22T21:20:55.928548+0000 mon.a (mon.0) 11647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:20:59.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:58 smithi121 bash[36902]: cluster 2022-04-22T21:20:57.261059+0000 mgr.y (mgr.34107) 8022 : cluster [DBG] pgmap v8987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:58 smithi115 bash[45495]: cluster 2022-04-22T21:20:57.261059+0000 mgr.y (mgr.34107) 8022 : cluster [DBG] pgmap v8987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:20:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:58 smithi115 bash[46934]: cluster 2022-04-22T21:20:57.261059+0000 mgr.y (mgr.34107) 8022 : cluster [DBG] pgmap v8987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:20:59 smithi121 bash[36902]: audit 2022-04-22T21:20:58.928284+0000 mon.a (mon.0) 11648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:20:59 smithi115 bash[45495]: audit 2022-04-22T21:20:58.928284+0000 mon.a (mon.0) 11648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:20:59 smithi115 bash[46934]: audit 2022-04-22T21:20:58.928284+0000 mon.a (mon.0) 11648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:00 smithi121 bash[36902]: cluster 2022-04-22T21:20:59.262122+0000 mgr.y (mgr.34107) 8023 : cluster [DBG] pgmap v8988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:00 smithi121 bash[36902]: audit 2022-04-22T21:20:59.761012+0000 mon.a (mon.0) 11649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:00 smithi115 bash[45495]: cluster 2022-04-22T21:20:59.262122+0000 mgr.y (mgr.34107) 8023 : cluster [DBG] pgmap v8988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:00 smithi115 bash[45495]: audit 2022-04-22T21:20:59.761012+0000 mon.a (mon.0) 11649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:00 smithi115 bash[46934]: cluster 2022-04-22T21:20:59.262122+0000 mgr.y (mgr.34107) 8023 : cluster [DBG] pgmap v8988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:00 smithi115 bash[46934]: audit 2022-04-22T21:20:59.761012+0000 mon.a (mon.0) 11649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:01 smithi115 bash[45495]: audit 2022-04-22T21:21:00.931524+0000 mon.a (mon.0) 11650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:02.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:01 smithi115 bash[46934]: audit 2022-04-22T21:21:00.931524+0000 mon.a (mon.0) 11650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:02.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:01 smithi121 bash[36902]: audit 2022-04-22T21:21:00.931524+0000 mon.a (mon.0) 11650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:02 smithi115 bash[45495]: cluster 2022-04-22T21:21:01.263210+0000 mgr.y (mgr.34107) 8024 : cluster [DBG] pgmap v8989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:02.529Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:02 smithi115 bash[46934]: cluster 2022-04-22T21:21:01.263210+0000 mgr.y (mgr.34107) 8024 : cluster [DBG] pgmap v8989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:02 smithi121 bash[36902]: cluster 2022-04-22T21:21:01.263210+0000 mgr.y (mgr.34107) 8024 : cluster [DBG] pgmap v8989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:04.771 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:04.403Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:04.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[45495]: cluster 2022-04-22T21:21:03.264402+0000 mgr.y (mgr.34107) 8025 : cluster [DBG] pgmap v8990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:04.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[45495]: audit 2022-04-22T21:21:03.931771+0000 mon.a (mon.0) 11651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:04.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[45495]: audit 2022-04-22T21:21:04.764172+0000 mon.a (mon.0) 11652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[46934]: cluster 2022-04-22T21:21:03.264402+0000 mgr.y (mgr.34107) 8025 : cluster [DBG] pgmap v8990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[46934]: audit 2022-04-22T21:21:03.931771+0000 mon.a (mon.0) 11651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:04 smithi115 bash[46934]: audit 2022-04-22T21:21:04.764172+0000 mon.a (mon.0) 11652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:04 smithi121 bash[36902]: cluster 2022-04-22T21:21:03.264402+0000 mgr.y (mgr.34107) 8025 : cluster [DBG] pgmap v8990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:04 smithi121 bash[36902]: audit 2022-04-22T21:21:03.931771+0000 mon.a (mon.0) 11651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:04 smithi121 bash[36902]: audit 2022-04-22T21:21:04.764172+0000 mon.a (mon.0) 11652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:06 smithi121 bash[36902]: cluster 2022-04-22T21:21:05.265508+0000 mgr.y (mgr.34107) 8026 : cluster [DBG] pgmap v8991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:06 smithi121 bash[36902]: audit 2022-04-22T21:21:05.934784+0000 mon.a (mon.0) 11653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:06 smithi115 bash[45495]: cluster 2022-04-22T21:21:05.265508+0000 mgr.y (mgr.34107) 8026 : cluster [DBG] pgmap v8991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:06 smithi115 bash[45495]: audit 2022-04-22T21:21:05.934784+0000 mon.a (mon.0) 11653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:06 smithi115 bash[46934]: cluster 2022-04-22T21:21:05.265508+0000 mgr.y (mgr.34107) 8026 : cluster [DBG] pgmap v8991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:06 smithi115 bash[46934]: audit 2022-04-22T21:21:05.934784+0000 mon.a (mon.0) 11653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:07.967 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:21:08.308 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:21:08.308 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:21:08.309 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:21:08.309 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:21:08.309 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:21:08.309 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:21:08.309 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:21:08.310 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:21:08.311 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:21:08.311 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:21:08.311 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:21:08.311 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 3h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:21:08.311 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:21:08.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:21:08.726 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:21:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:08 smithi121 bash[36902]: cluster 2022-04-22T21:21:07.266594+0000 mgr.y (mgr.34107) 8027 : cluster [DBG] pgmap v8992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:08 smithi115 bash[45495]: cluster 2022-04-22T21:21:07.266594+0000 mgr.y (mgr.34107) 8027 : cluster [DBG] pgmap v8992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:08 smithi115 bash[46934]: cluster 2022-04-22T21:21:07.266594+0000 mgr.y (mgr.34107) 8027 : cluster [DBG] pgmap v8992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:09 smithi121 bash[36902]: audit 2022-04-22T21:21:07.949213+0000 mgr.y (mgr.34107) 8028 : audit [DBG] from='client.53635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:09 smithi121 bash[36902]: audit 2022-04-22T21:21:08.284988+0000 mgr.y (mgr.34107) 8029 : audit [DBG] from='client.43710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:09 smithi121 bash[36902]: audit 2022-04-22T21:21:08.720808+0000 mon.c (mon.1) 289 : audit [DBG] from='client.? 172.21.15.115:0/1891532320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:09 smithi121 bash[36902]: audit 2022-04-22T21:21:08.934882+0000 mon.a (mon.0) 11654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[45495]: audit 2022-04-22T21:21:07.949213+0000 mgr.y (mgr.34107) 8028 : audit [DBG] from='client.53635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[45495]: audit 2022-04-22T21:21:08.284988+0000 mgr.y (mgr.34107) 8029 : audit [DBG] from='client.43710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[45495]: audit 2022-04-22T21:21:08.720808+0000 mon.c (mon.1) 289 : audit [DBG] from='client.? 172.21.15.115:0/1891532320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:09.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[45495]: audit 2022-04-22T21:21:08.934882+0000 mon.a (mon.0) 11654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[46934]: audit 2022-04-22T21:21:07.949213+0000 mgr.y (mgr.34107) 8028 : audit [DBG] from='client.53635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[46934]: audit 2022-04-22T21:21:08.284988+0000 mgr.y (mgr.34107) 8029 : audit [DBG] from='client.43710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[46934]: audit 2022-04-22T21:21:08.720808+0000 mon.c (mon.1) 289 : audit [DBG] from='client.? 172.21.15.115:0/1891532320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:09.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:09 smithi115 bash[46934]: audit 2022-04-22T21:21:08.934882+0000 mon.a (mon.0) 11654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:10 smithi121 bash[36902]: cluster 2022-04-22T21:21:09.267724+0000 mgr.y (mgr.34107) 8030 : cluster [DBG] pgmap v8993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:10 smithi121 bash[36902]: audit 2022-04-22T21:21:09.767939+0000 mon.a (mon.0) 11655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:10 smithi115 bash[45495]: cluster 2022-04-22T21:21:09.267724+0000 mgr.y (mgr.34107) 8030 : cluster [DBG] pgmap v8993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:10.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:10 smithi115 bash[45495]: audit 2022-04-22T21:21:09.767939+0000 mon.a (mon.0) 11655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:10 smithi115 bash[46934]: cluster 2022-04-22T21:21:09.267724+0000 mgr.y (mgr.34107) 8030 : cluster [DBG] pgmap v8993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:10.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:10 smithi115 bash[46934]: audit 2022-04-22T21:21:09.767939+0000 mon.a (mon.0) 11655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:11 smithi121 bash[36902]: audit 2022-04-22T21:21:10.937766+0000 mon.a (mon.0) 11656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:11 smithi115 bash[45495]: audit 2022-04-22T21:21:10.937766+0000 mon.a (mon.0) 11656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:11 smithi115 bash[46934]: audit 2022-04-22T21:21:10.937766+0000 mon.a (mon.0) 11656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:12 smithi121 bash[36902]: cluster 2022-04-22T21:21:11.268787+0000 mgr.y (mgr.34107) 8031 : cluster [DBG] pgmap v8994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:12 smithi115 bash[45495]: cluster 2022-04-22T21:21:11.268787+0000 mgr.y (mgr.34107) 8031 : cluster [DBG] pgmap v8994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:12.529Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:12 smithi115 bash[46934]: cluster 2022-04-22T21:21:11.268787+0000 mgr.y (mgr.34107) 8031 : cluster [DBG] pgmap v8994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:14 smithi121 bash[36902]: cluster 2022-04-22T21:21:13.269922+0000 mgr.y (mgr.34107) 8032 : cluster [DBG] pgmap v8995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:14 smithi121 bash[36902]: audit 2022-04-22T21:21:13.937877+0000 mon.a (mon.0) 11657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:14.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:14.406Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:14.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:14 smithi115 bash[45495]: cluster 2022-04-22T21:21:13.269922+0000 mgr.y (mgr.34107) 8032 : cluster [DBG] pgmap v8995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:14.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:14 smithi115 bash[45495]: audit 2022-04-22T21:21:13.937877+0000 mon.a (mon.0) 11657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:14 smithi115 bash[46934]: cluster 2022-04-22T21:21:13.269922+0000 mgr.y (mgr.34107) 8032 : cluster [DBG] pgmap v8995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:14.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:14 smithi115 bash[46934]: audit 2022-04-22T21:21:13.937877+0000 mon.a (mon.0) 11657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:15 smithi121 bash[36902]: audit 2022-04-22T21:21:14.771695+0000 mon.a (mon.0) 11658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:15 smithi121 bash[36902]: audit 2022-04-22T21:21:15.184622+0000 mon.a (mon.0) 11659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:21:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:15 smithi121 bash[36902]: audit 2022-04-22T21:21:15.188576+0000 mon.a (mon.0) 11660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:21:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:15 smithi121 bash[36902]: audit 2022-04-22T21:21:15.190237+0000 mon.a (mon.0) 11661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:21:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[45495]: audit 2022-04-22T21:21:14.771695+0000 mon.a (mon.0) 11658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[45495]: audit 2022-04-22T21:21:15.184622+0000 mon.a (mon.0) 11659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:21:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[45495]: audit 2022-04-22T21:21:15.188576+0000 mon.a (mon.0) 11660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:21:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[45495]: audit 2022-04-22T21:21:15.190237+0000 mon.a (mon.0) 11661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:21:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[46934]: audit 2022-04-22T21:21:14.771695+0000 mon.a (mon.0) 11658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[46934]: audit 2022-04-22T21:21:15.184622+0000 mon.a (mon.0) 11659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:21:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[46934]: audit 2022-04-22T21:21:15.188576+0000 mon.a (mon.0) 11660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:21:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:15 smithi115 bash[46934]: audit 2022-04-22T21:21:15.190237+0000 mon.a (mon.0) 11661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:21:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:16 smithi121 bash[36902]: cluster 2022-04-22T21:21:15.270955+0000 mgr.y (mgr.34107) 8033 : cluster [DBG] pgmap v8996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:16 smithi121 bash[36902]: audit 2022-04-22T21:21:15.554971+0000 mon.a (mon.0) 11662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:21:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:16 smithi121 bash[36902]: audit 2022-04-22T21:21:15.940932+0000 mon.a (mon.0) 11663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:16 smithi121 bash[36902]: audit 2022-04-22T21:21:15.987356+0000 mon.a (mon.0) 11664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:21:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:16 smithi121 bash[36902]: audit 2022-04-22T21:21:15.987712+0000 mon.a (mon.0) 11665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:21:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[45495]: cluster 2022-04-22T21:21:15.270955+0000 mgr.y (mgr.34107) 8033 : cluster [DBG] pgmap v8996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[45495]: audit 2022-04-22T21:21:15.554971+0000 mon.a (mon.0) 11662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:21:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[45495]: audit 2022-04-22T21:21:15.940932+0000 mon.a (mon.0) 11663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[45495]: audit 2022-04-22T21:21:15.987356+0000 mon.a (mon.0) 11664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[45495]: audit 2022-04-22T21:21:15.987712+0000 mon.a (mon.0) 11665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[46934]: cluster 2022-04-22T21:21:15.270955+0000 mgr.y (mgr.34107) 8033 : cluster [DBG] pgmap v8996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[46934]: audit 2022-04-22T21:21:15.554971+0000 mon.a (mon.0) 11662 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[46934]: audit 2022-04-22T21:21:15.940932+0000 mon.a (mon.0) 11663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[46934]: audit 2022-04-22T21:21:15.987356+0000 mon.a (mon.0) 11664 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:21:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:16 smithi115 bash[46934]: audit 2022-04-22T21:21:15.987712+0000 mon.a (mon.0) 11665 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:21:19.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:18 smithi121 bash[36902]: cluster 2022-04-22T21:21:17.272197+0000 mgr.y (mgr.34107) 8034 : cluster [DBG] pgmap v8997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:18 smithi115 bash[45495]: cluster 2022-04-22T21:21:17.272197+0000 mgr.y (mgr.34107) 8034 : cluster [DBG] pgmap v8997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:18 smithi115 bash[46934]: cluster 2022-04-22T21:21:17.272197+0000 mgr.y (mgr.34107) 8034 : cluster [DBG] pgmap v8997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:19 smithi121 bash[36902]: audit 2022-04-22T21:21:18.941363+0000 mon.a (mon.0) 11666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:19 smithi115 bash[45495]: audit 2022-04-22T21:21:18.941363+0000 mon.a (mon.0) 11666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:19 smithi115 bash[46934]: audit 2022-04-22T21:21:18.941363+0000 mon.a (mon.0) 11666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:20 smithi121 bash[36902]: cluster 2022-04-22T21:21:19.273395+0000 mgr.y (mgr.34107) 8035 : cluster [DBG] pgmap v8998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:20 smithi121 bash[36902]: audit 2022-04-22T21:21:19.775379+0000 mon.a (mon.0) 11667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:20 smithi115 bash[45495]: cluster 2022-04-22T21:21:19.273395+0000 mgr.y (mgr.34107) 8035 : cluster [DBG] pgmap v8998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:20 smithi115 bash[45495]: audit 2022-04-22T21:21:19.775379+0000 mon.a (mon.0) 11667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:20 smithi115 bash[46934]: cluster 2022-04-22T21:21:19.273395+0000 mgr.y (mgr.34107) 8035 : cluster [DBG] pgmap v8998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:20 smithi115 bash[46934]: audit 2022-04-22T21:21:19.775379+0000 mon.a (mon.0) 11667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:21 smithi121 bash[36902]: audit 2022-04-22T21:21:20.943887+0000 mon.a (mon.0) 11668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:21 smithi115 bash[45495]: audit 2022-04-22T21:21:20.943887+0000 mon.a (mon.0) 11668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:22.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:21 smithi115 bash[46934]: audit 2022-04-22T21:21:20.943887+0000 mon.a (mon.0) 11668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:22.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:22.530Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:22.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:22 smithi115 bash[45495]: cluster 2022-04-22T21:21:21.274421+0000 mgr.y (mgr.34107) 8036 : cluster [DBG] pgmap v8999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:22.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:22 smithi115 bash[46934]: cluster 2022-04-22T21:21:21.274421+0000 mgr.y (mgr.34107) 8036 : cluster [DBG] pgmap v8999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:22 smithi121 bash[36902]: cluster 2022-04-22T21:21:21.274421+0000 mgr.y (mgr.34107) 8036 : cluster [DBG] pgmap v8999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:24 smithi115 bash[45495]: cluster 2022-04-22T21:21:23.275546+0000 mgr.y (mgr.34107) 8037 : cluster [DBG] pgmap v9000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:24 smithi115 bash[45495]: audit 2022-04-22T21:21:23.944556+0000 mon.a (mon.0) 11669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:24.405Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:24 smithi115 bash[46934]: cluster 2022-04-22T21:21:23.275546+0000 mgr.y (mgr.34107) 8037 : cluster [DBG] pgmap v9000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:24 smithi115 bash[46934]: audit 2022-04-22T21:21:23.944556+0000 mon.a (mon.0) 11669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:24 smithi121 bash[36902]: cluster 2022-04-22T21:21:23.275546+0000 mgr.y (mgr.34107) 8037 : cluster [DBG] pgmap v9000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:24 smithi121 bash[36902]: audit 2022-04-22T21:21:23.944556+0000 mon.a (mon.0) 11669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:25 smithi121 bash[36902]: audit 2022-04-22T21:21:24.779171+0000 mon.a (mon.0) 11670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:25 smithi115 bash[45495]: audit 2022-04-22T21:21:24.779171+0000 mon.a (mon.0) 11670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:26.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:25 smithi115 bash[46934]: audit 2022-04-22T21:21:24.779171+0000 mon.a (mon.0) 11670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:26 smithi121 bash[36902]: cluster 2022-04-22T21:21:25.276473+0000 mgr.y (mgr.34107) 8038 : cluster [DBG] pgmap v9001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:26 smithi121 bash[36902]: audit 2022-04-22T21:21:25.947285+0000 mon.a (mon.0) 11671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:26 smithi115 bash[45495]: cluster 2022-04-22T21:21:25.276473+0000 mgr.y (mgr.34107) 8038 : cluster [DBG] pgmap v9001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:26 smithi115 bash[45495]: audit 2022-04-22T21:21:25.947285+0000 mon.a (mon.0) 11671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:26 smithi115 bash[46934]: cluster 2022-04-22T21:21:25.276473+0000 mgr.y (mgr.34107) 8038 : cluster [DBG] pgmap v9001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:26 smithi115 bash[46934]: audit 2022-04-22T21:21:25.947285+0000 mon.a (mon.0) 11671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:28 smithi121 bash[36902]: cluster 2022-04-22T21:21:27.277564+0000 mgr.y (mgr.34107) 8039 : cluster [DBG] pgmap v9002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:28 smithi115 bash[45495]: cluster 2022-04-22T21:21:27.277564+0000 mgr.y (mgr.34107) 8039 : cluster [DBG] pgmap v9002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:28 smithi115 bash[46934]: cluster 2022-04-22T21:21:27.277564+0000 mgr.y (mgr.34107) 8039 : cluster [DBG] pgmap v9002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:29 smithi121 bash[36902]: audit 2022-04-22T21:21:28.947910+0000 mon.a (mon.0) 11672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:29 smithi115 bash[45495]: audit 2022-04-22T21:21:28.947910+0000 mon.a (mon.0) 11672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:29 smithi115 bash[46934]: audit 2022-04-22T21:21:28.947910+0000 mon.a (mon.0) 11672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:30 smithi121 bash[36902]: cluster 2022-04-22T21:21:29.278605+0000 mgr.y (mgr.34107) 8040 : cluster [DBG] pgmap v9003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:30 smithi121 bash[36902]: audit 2022-04-22T21:21:29.782887+0000 mon.a (mon.0) 11673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:30 smithi115 bash[45495]: cluster 2022-04-22T21:21:29.278605+0000 mgr.y (mgr.34107) 8040 : cluster [DBG] pgmap v9003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:30 smithi115 bash[45495]: audit 2022-04-22T21:21:29.782887+0000 mon.a (mon.0) 11673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:30 smithi115 bash[46934]: cluster 2022-04-22T21:21:29.278605+0000 mgr.y (mgr.34107) 8040 : cluster [DBG] pgmap v9003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:30 smithi115 bash[46934]: audit 2022-04-22T21:21:29.782887+0000 mon.a (mon.0) 11673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:32.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:31 smithi121 bash[36902]: audit 2022-04-22T21:21:30.950278+0000 mon.a (mon.0) 11674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:31 smithi115 bash[45495]: audit 2022-04-22T21:21:30.950278+0000 mon.a (mon.0) 11674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:31 smithi115 bash[46934]: audit 2022-04-22T21:21:30.950278+0000 mon.a (mon.0) 11674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:32 smithi115 bash[45495]: cluster 2022-04-22T21:21:31.279720+0000 mgr.y (mgr.34107) 8041 : cluster [DBG] pgmap v9004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:32.531Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:32.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:32 smithi115 bash[46934]: cluster 2022-04-22T21:21:31.279720+0000 mgr.y (mgr.34107) 8041 : cluster [DBG] pgmap v9004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:32 smithi121 bash[36902]: cluster 2022-04-22T21:21:31.279720+0000 mgr.y (mgr.34107) 8041 : cluster [DBG] pgmap v9004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:34 smithi115 bash[45495]: cluster 2022-04-22T21:21:33.280782+0000 mgr.y (mgr.34107) 8042 : cluster [DBG] pgmap v9005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:34 smithi115 bash[45495]: audit 2022-04-22T21:21:33.951320+0000 mon.a (mon.0) 11675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:34 smithi115 bash[46934]: cluster 2022-04-22T21:21:33.280782+0000 mgr.y (mgr.34107) 8042 : cluster [DBG] pgmap v9005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:34 smithi115 bash[46934]: audit 2022-04-22T21:21:33.951320+0000 mon.a (mon.0) 11675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:34.405Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:34 smithi121 bash[36902]: cluster 2022-04-22T21:21:33.280782+0000 mgr.y (mgr.34107) 8042 : cluster [DBG] pgmap v9005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:34 smithi121 bash[36902]: audit 2022-04-22T21:21:33.951320+0000 mon.a (mon.0) 11675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:35 smithi121 bash[36902]: audit 2022-04-22T21:21:34.786803+0000 mon.a (mon.0) 11676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:35 smithi115 bash[45495]: audit 2022-04-22T21:21:34.786803+0000 mon.a (mon.0) 11676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:35 smithi115 bash[46934]: audit 2022-04-22T21:21:34.786803+0000 mon.a (mon.0) 11676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:36 smithi121 bash[36902]: cluster 2022-04-22T21:21:35.281869+0000 mgr.y (mgr.34107) 8043 : cluster [DBG] pgmap v9006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:36 smithi121 bash[36902]: audit 2022-04-22T21:21:35.953449+0000 mon.a (mon.0) 11677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:36 smithi115 bash[46934]: cluster 2022-04-22T21:21:35.281869+0000 mgr.y (mgr.34107) 8043 : cluster [DBG] pgmap v9006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:36 smithi115 bash[46934]: audit 2022-04-22T21:21:35.953449+0000 mon.a (mon.0) 11677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:36 smithi115 bash[45495]: cluster 2022-04-22T21:21:35.281869+0000 mgr.y (mgr.34107) 8043 : cluster [DBG] pgmap v9006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:36 smithi115 bash[45495]: audit 2022-04-22T21:21:35.953449+0000 mon.a (mon.0) 11677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:38 smithi121 bash[36902]: cluster 2022-04-22T21:21:37.283067+0000 mgr.y (mgr.34107) 8044 : cluster [DBG] pgmap v9007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:38 smithi115 bash[45495]: cluster 2022-04-22T21:21:37.283067+0000 mgr.y (mgr.34107) 8044 : cluster [DBG] pgmap v9007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:38 smithi115 bash[46934]: cluster 2022-04-22T21:21:37.283067+0000 mgr.y (mgr.34107) 8044 : cluster [DBG] pgmap v9007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:39.099 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 8m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 3h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 3h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 7m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:21:39.440 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 7m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 8m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 7m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 7m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 7m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 7m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:21:39.441 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:21:39.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:21:39.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 4h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:21:39.442 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:21:39.841 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:21:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:21:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:21:39.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:39.843 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:21:39.844 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:21:39.844 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:39.845 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:21:39.845 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:21:39.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:39.846 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:21:39.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:21:39.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:21:39.847 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:21:39.847 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:21:39.847 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:21:39.847 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:21:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:39 smithi121 bash[36902]: audit 2022-04-22T21:21:38.954219+0000 mon.a (mon.0) 11678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:39 smithi115 bash[45495]: audit 2022-04-22T21:21:38.954219+0000 mon.a (mon.0) 11678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:40.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:39 smithi115 bash[46934]: audit 2022-04-22T21:21:38.954219+0000 mon.a (mon.0) 11678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:40 smithi121 bash[36902]: audit 2022-04-22T21:21:39.075988+0000 mgr.y (mgr.34107) 8045 : audit [DBG] from='client.53653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:40 smithi121 bash[36902]: cluster 2022-04-22T21:21:39.284129+0000 mgr.y (mgr.34107) 8046 : cluster [DBG] pgmap v9008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:40 smithi121 bash[36902]: audit 2022-04-22T21:21:39.416631+0000 mgr.y (mgr.34107) 8047 : audit [DBG] from='client.43728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:40 smithi121 bash[36902]: audit 2022-04-22T21:21:39.790719+0000 mon.a (mon.0) 11679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:41.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:40 smithi121 bash[36902]: audit 2022-04-22T21:21:39.837395+0000 mon.a (mon.0) 11680 : audit [DBG] from='client.? 172.21.15.115:0/3420067697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[45495]: audit 2022-04-22T21:21:39.075988+0000 mgr.y (mgr.34107) 8045 : audit [DBG] from='client.53653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[45495]: cluster 2022-04-22T21:21:39.284129+0000 mgr.y (mgr.34107) 8046 : cluster [DBG] pgmap v9008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[45495]: audit 2022-04-22T21:21:39.416631+0000 mgr.y (mgr.34107) 8047 : audit [DBG] from='client.43728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[45495]: audit 2022-04-22T21:21:39.790719+0000 mon.a (mon.0) 11679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:41.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[45495]: audit 2022-04-22T21:21:39.837395+0000 mon.a (mon.0) 11680 : audit [DBG] from='client.? 172.21.15.115:0/3420067697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[46934]: audit 2022-04-22T21:21:39.075988+0000 mgr.y (mgr.34107) 8045 : audit [DBG] from='client.53653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[46934]: cluster 2022-04-22T21:21:39.284129+0000 mgr.y (mgr.34107) 8046 : cluster [DBG] pgmap v9008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[46934]: audit 2022-04-22T21:21:39.416631+0000 mgr.y (mgr.34107) 8047 : audit [DBG] from='client.43728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:21:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[46934]: audit 2022-04-22T21:21:39.790719+0000 mon.a (mon.0) 11679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:41.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:40 smithi115 bash[46934]: audit 2022-04-22T21:21:39.837395+0000 mon.a (mon.0) 11680 : audit [DBG] from='client.? 172.21.15.115:0/3420067697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:21:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:41 smithi121 bash[36902]: audit 2022-04-22T21:21:40.956370+0000 mon.a (mon.0) 11681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:41 smithi115 bash[45495]: audit 2022-04-22T21:21:40.956370+0000 mon.a (mon.0) 11681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:41 smithi115 bash[46934]: audit 2022-04-22T21:21:40.956370+0000 mon.a (mon.0) 11681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:42 smithi115 bash[45495]: cluster 2022-04-22T21:21:41.285132+0000 mgr.y (mgr.34107) 8048 : cluster [DBG] pgmap v9009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:42.532Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:42 smithi115 bash[46934]: cluster 2022-04-22T21:21:41.285132+0000 mgr.y (mgr.34107) 8048 : cluster [DBG] pgmap v9009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:42 smithi121 bash[36902]: cluster 2022-04-22T21:21:41.285132+0000 mgr.y (mgr.34107) 8048 : cluster [DBG] pgmap v9009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:44.406Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:44 smithi115 bash[45495]: cluster 2022-04-22T21:21:43.286231+0000 mgr.y (mgr.34107) 8049 : cluster [DBG] pgmap v9010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:44 smithi115 bash[45495]: audit 2022-04-22T21:21:43.957076+0000 mon.a (mon.0) 11682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:44 smithi115 bash[46934]: cluster 2022-04-22T21:21:43.286231+0000 mgr.y (mgr.34107) 8049 : cluster [DBG] pgmap v9010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:44 smithi115 bash[46934]: audit 2022-04-22T21:21:43.957076+0000 mon.a (mon.0) 11682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:44 smithi121 bash[36902]: cluster 2022-04-22T21:21:43.286231+0000 mgr.y (mgr.34107) 8049 : cluster [DBG] pgmap v9010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:44 smithi121 bash[36902]: audit 2022-04-22T21:21:43.957076+0000 mon.a (mon.0) 11682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:45 smithi121 bash[36902]: audit 2022-04-22T21:21:44.794590+0000 mon.a (mon.0) 11683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:45 smithi115 bash[45495]: audit 2022-04-22T21:21:44.794590+0000 mon.a (mon.0) 11683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:45 smithi115 bash[46934]: audit 2022-04-22T21:21:44.794590+0000 mon.a (mon.0) 11683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:47.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:46 smithi121 bash[36902]: cluster 2022-04-22T21:21:45.287332+0000 mgr.y (mgr.34107) 8050 : cluster [DBG] pgmap v9011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:46 smithi121 bash[36902]: audit 2022-04-22T21:21:45.959416+0000 mon.a (mon.0) 11684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:46 smithi115 bash[45495]: cluster 2022-04-22T21:21:45.287332+0000 mgr.y (mgr.34107) 8050 : cluster [DBG] pgmap v9011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:46 smithi115 bash[45495]: audit 2022-04-22T21:21:45.959416+0000 mon.a (mon.0) 11684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:46 smithi115 bash[46934]: cluster 2022-04-22T21:21:45.287332+0000 mgr.y (mgr.34107) 8050 : cluster [DBG] pgmap v9011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:46 smithi115 bash[46934]: audit 2022-04-22T21:21:45.959416+0000 mon.a (mon.0) 11684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:48 smithi121 bash[36902]: cluster 2022-04-22T21:21:47.288428+0000 mgr.y (mgr.34107) 8051 : cluster [DBG] pgmap v9012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:48 smithi115 bash[45495]: cluster 2022-04-22T21:21:47.288428+0000 mgr.y (mgr.34107) 8051 : cluster [DBG] pgmap v9012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:49.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:48 smithi115 bash[46934]: cluster 2022-04-22T21:21:47.288428+0000 mgr.y (mgr.34107) 8051 : cluster [DBG] pgmap v9012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:50.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:49 smithi121 bash[36902]: audit 2022-04-22T21:21:48.960177+0000 mon.a (mon.0) 11685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:49 smithi115 bash[45495]: audit 2022-04-22T21:21:48.960177+0000 mon.a (mon.0) 11685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:49 smithi115 bash[46934]: audit 2022-04-22T21:21:48.960177+0000 mon.a (mon.0) 11685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:50 smithi121 bash[36902]: cluster 2022-04-22T21:21:49.289548+0000 mgr.y (mgr.34107) 8052 : cluster [DBG] pgmap v9013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:50 smithi121 bash[36902]: audit 2022-04-22T21:21:49.798441+0000 mon.a (mon.0) 11686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:50 smithi115 bash[45495]: cluster 2022-04-22T21:21:49.289548+0000 mgr.y (mgr.34107) 8052 : cluster [DBG] pgmap v9013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:50 smithi115 bash[45495]: audit 2022-04-22T21:21:49.798441+0000 mon.a (mon.0) 11686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:50 smithi115 bash[46934]: cluster 2022-04-22T21:21:49.289548+0000 mgr.y (mgr.34107) 8052 : cluster [DBG] pgmap v9013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:51.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:50 smithi115 bash[46934]: audit 2022-04-22T21:21:49.798441+0000 mon.a (mon.0) 11686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:52.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:51 smithi121 bash[36902]: audit 2022-04-22T21:21:50.962366+0000 mon.a (mon.0) 11687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:51 smithi115 bash[45495]: audit 2022-04-22T21:21:50.962366+0000 mon.a (mon.0) 11687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:51 smithi115 bash[46934]: audit 2022-04-22T21:21:50.962366+0000 mon.a (mon.0) 11687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:52 smithi115 bash[45495]: cluster 2022-04-22T21:21:51.290464+0000 mgr.y (mgr.34107) 8053 : cluster [DBG] pgmap v9014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:52.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:52.532Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:52.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:52 smithi115 bash[46934]: cluster 2022-04-22T21:21:51.290464+0000 mgr.y (mgr.34107) 8053 : cluster [DBG] pgmap v9014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:52 smithi121 bash[36902]: cluster 2022-04-22T21:21:51.290464+0000 mgr.y (mgr.34107) 8053 : cluster [DBG] pgmap v9014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:54.691 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:21:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:21:54.407Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:21:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:54 smithi121 bash[36902]: cluster 2022-04-22T21:21:53.291682+0000 mgr.y (mgr.34107) 8054 : cluster [DBG] pgmap v9015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:54 smithi121 bash[36902]: audit 2022-04-22T21:21:53.963190+0000 mon.a (mon.0) 11688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:54 smithi115 bash[45495]: cluster 2022-04-22T21:21:53.291682+0000 mgr.y (mgr.34107) 8054 : cluster [DBG] pgmap v9015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:54 smithi115 bash[45495]: audit 2022-04-22T21:21:53.963190+0000 mon.a (mon.0) 11688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:54 smithi115 bash[46934]: cluster 2022-04-22T21:21:53.291682+0000 mgr.y (mgr.34107) 8054 : cluster [DBG] pgmap v9015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:54 smithi115 bash[46934]: audit 2022-04-22T21:21:53.963190+0000 mon.a (mon.0) 11688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:21:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:55 smithi121 bash[36902]: audit 2022-04-22T21:21:54.802157+0000 mon.a (mon.0) 11689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:55 smithi115 bash[45495]: audit 2022-04-22T21:21:54.802157+0000 mon.a (mon.0) 11689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:55 smithi115 bash[46934]: audit 2022-04-22T21:21:54.802157+0000 mon.a (mon.0) 11689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:21:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:56 smithi121 bash[36902]: cluster 2022-04-22T21:21:55.292709+0000 mgr.y (mgr.34107) 8055 : cluster [DBG] pgmap v9016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:56 smithi121 bash[36902]: audit 2022-04-22T21:21:55.965451+0000 mon.a (mon.0) 11690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:56 smithi115 bash[45495]: cluster 2022-04-22T21:21:55.292709+0000 mgr.y (mgr.34107) 8055 : cluster [DBG] pgmap v9016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:56 smithi115 bash[45495]: audit 2022-04-22T21:21:55.965451+0000 mon.a (mon.0) 11690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:56 smithi115 bash[46934]: cluster 2022-04-22T21:21:55.292709+0000 mgr.y (mgr.34107) 8055 : cluster [DBG] pgmap v9016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:56 smithi115 bash[46934]: audit 2022-04-22T21:21:55.965451+0000 mon.a (mon.0) 11690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:21:59.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:58 smithi121 bash[36902]: cluster 2022-04-22T21:21:57.293777+0000 mgr.y (mgr.34107) 8056 : cluster [DBG] pgmap v9017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:58 smithi115 bash[45495]: cluster 2022-04-22T21:21:57.293777+0000 mgr.y (mgr.34107) 8056 : cluster [DBG] pgmap v9017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:21:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:58 smithi115 bash[46934]: cluster 2022-04-22T21:21:57.293777+0000 mgr.y (mgr.34107) 8056 : cluster [DBG] pgmap v9017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:21:59 smithi121 bash[36902]: audit 2022-04-22T21:21:58.966480+0000 mon.a (mon.0) 11691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:21:59 smithi115 bash[45495]: audit 2022-04-22T21:21:58.966480+0000 mon.a (mon.0) 11691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:21:59 smithi115 bash[46934]: audit 2022-04-22T21:21:58.966480+0000 mon.a (mon.0) 11691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:01.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:00 smithi121 bash[36902]: cluster 2022-04-22T21:21:59.294828+0000 mgr.y (mgr.34107) 8057 : cluster [DBG] pgmap v9018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:01.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:00 smithi121 bash[36902]: audit 2022-04-22T21:21:59.807296+0000 mon.a (mon.0) 11692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:00 smithi115 bash[45495]: cluster 2022-04-22T21:21:59.294828+0000 mgr.y (mgr.34107) 8057 : cluster [DBG] pgmap v9018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:00 smithi115 bash[45495]: audit 2022-04-22T21:21:59.807296+0000 mon.a (mon.0) 11692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:00 smithi115 bash[46934]: cluster 2022-04-22T21:21:59.294828+0000 mgr.y (mgr.34107) 8057 : cluster [DBG] pgmap v9018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:00 smithi115 bash[46934]: audit 2022-04-22T21:21:59.807296+0000 mon.a (mon.0) 11692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:01 smithi121 bash[36902]: audit 2022-04-22T21:22:00.968649+0000 mon.a (mon.0) 11693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:01 smithi115 bash[45495]: audit 2022-04-22T21:22:00.968649+0000 mon.a (mon.0) 11693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:01 smithi115 bash[46934]: audit 2022-04-22T21:22:00.968649+0000 mon.a (mon.0) 11693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:02.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:02 smithi115 bash[45495]: cluster 2022-04-22T21:22:01.295920+0000 mgr.y (mgr.34107) 8058 : cluster [DBG] pgmap v9019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:02 smithi115 bash[46934]: cluster 2022-04-22T21:22:01.295920+0000 mgr.y (mgr.34107) 8058 : cluster [DBG] pgmap v9019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:02 smithi121 bash[36902]: cluster 2022-04-22T21:22:01.295920+0000 mgr.y (mgr.34107) 8058 : cluster [DBG] pgmap v9019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:04.725 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:04.408Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:04 smithi121 bash[36902]: cluster 2022-04-22T21:22:03.297057+0000 mgr.y (mgr.34107) 8059 : cluster [DBG] pgmap v9020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:04 smithi121 bash[36902]: audit 2022-04-22T21:22:03.969512+0000 mon.a (mon.0) 11694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:04 smithi115 bash[45495]: cluster 2022-04-22T21:22:03.297057+0000 mgr.y (mgr.34107) 8059 : cluster [DBG] pgmap v9020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:04 smithi115 bash[45495]: audit 2022-04-22T21:22:03.969512+0000 mon.a (mon.0) 11694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:04 smithi115 bash[46934]: cluster 2022-04-22T21:22:03.297057+0000 mgr.y (mgr.34107) 8059 : cluster [DBG] pgmap v9020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:04 smithi115 bash[46934]: audit 2022-04-22T21:22:03.969512+0000 mon.a (mon.0) 11694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:05 smithi121 bash[36902]: audit 2022-04-22T21:22:04.811011+0000 mon.a (mon.0) 11695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:05 smithi115 bash[45495]: audit 2022-04-22T21:22:04.811011+0000 mon.a (mon.0) 11695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:05 smithi115 bash[46934]: audit 2022-04-22T21:22:04.811011+0000 mon.a (mon.0) 11695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:06 smithi121 bash[36902]: cluster 2022-04-22T21:22:05.298218+0000 mgr.y (mgr.34107) 8060 : cluster [DBG] pgmap v9021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:06 smithi121 bash[36902]: audit 2022-04-22T21:22:05.971733+0000 mon.a (mon.0) 11696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:06 smithi115 bash[45495]: cluster 2022-04-22T21:22:05.298218+0000 mgr.y (mgr.34107) 8060 : cluster [DBG] pgmap v9021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:06 smithi115 bash[45495]: audit 2022-04-22T21:22:05.971733+0000 mon.a (mon.0) 11696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:06 smithi115 bash[46934]: cluster 2022-04-22T21:22:05.298218+0000 mgr.y (mgr.34107) 8060 : cluster [DBG] pgmap v9021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:06 smithi115 bash[46934]: audit 2022-04-22T21:22:05.971733+0000 mon.a (mon.0) 11696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:09.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:08 smithi121 bash[36902]: cluster 2022-04-22T21:22:07.299266+0000 mgr.y (mgr.34107) 8061 : cluster [DBG] pgmap v9022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:08 smithi115 bash[45495]: cluster 2022-04-22T21:22:07.299266+0000 mgr.y (mgr.34107) 8061 : cluster [DBG] pgmap v9022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:08 smithi115 bash[46934]: cluster 2022-04-22T21:22:07.299266+0000 mgr.y (mgr.34107) 8061 : cluster [DBG] pgmap v9022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:09 smithi121 bash[36902]: audit 2022-04-22T21:22:08.973028+0000 mon.a (mon.0) 11697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:09 smithi115 bash[45495]: audit 2022-04-22T21:22:08.973028+0000 mon.a (mon.0) 11697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:09 smithi115 bash[46934]: audit 2022-04-22T21:22:08.973028+0000 mon.a (mon.0) 11697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:10.220 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:22:10.566 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:22:10.566 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:22:10.567 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:22:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:22:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:22:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:22:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:22:10.568 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:22:10.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:22:10.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:22:10.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:22:10.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:22:10.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:22:10.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:22:10.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 4h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:22:10.570 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:22:10.968 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:22:10.969 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:22:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:10 smithi121 bash[36902]: cluster 2022-04-22T21:22:09.300320+0000 mgr.y (mgr.34107) 8062 : cluster [DBG] pgmap v9023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:11.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:10 smithi121 bash[36902]: audit 2022-04-22T21:22:09.814832+0000 mon.a (mon.0) 11698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:10 smithi115 bash[46934]: cluster 2022-04-22T21:22:09.300320+0000 mgr.y (mgr.34107) 8062 : cluster [DBG] pgmap v9023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:10 smithi115 bash[46934]: audit 2022-04-22T21:22:09.814832+0000 mon.a (mon.0) 11698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:10 smithi115 bash[45495]: cluster 2022-04-22T21:22:09.300320+0000 mgr.y (mgr.34107) 8062 : cluster [DBG] pgmap v9023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:10 smithi115 bash[45495]: audit 2022-04-22T21:22:09.814832+0000 mon.a (mon.0) 11698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:12.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:11 smithi121 bash[36902]: audit 2022-04-22T21:22:10.201966+0000 mgr.y (mgr.34107) 8063 : audit [DBG] from='client.43740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:11 smithi121 bash[36902]: audit 2022-04-22T21:22:10.542408+0000 mgr.y (mgr.34107) 8064 : audit [DBG] from='client.53677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:11 smithi121 bash[36902]: audit 2022-04-22T21:22:10.963947+0000 mon.c (mon.1) 290 : audit [DBG] from='client.? 172.21.15.115:0/4006511763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:12.020 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:11 smithi121 bash[36902]: audit 2022-04-22T21:22:10.974876+0000 mon.a (mon.0) 11699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[45495]: audit 2022-04-22T21:22:10.201966+0000 mgr.y (mgr.34107) 8063 : audit [DBG] from='client.43740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[45495]: audit 2022-04-22T21:22:10.542408+0000 mgr.y (mgr.34107) 8064 : audit [DBG] from='client.53677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[45495]: audit 2022-04-22T21:22:10.963947+0000 mon.c (mon.1) 290 : audit [DBG] from='client.? 172.21.15.115:0/4006511763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[45495]: audit 2022-04-22T21:22:10.974876+0000 mon.a (mon.0) 11699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[46934]: audit 2022-04-22T21:22:10.201966+0000 mgr.y (mgr.34107) 8063 : audit [DBG] from='client.43740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[46934]: audit 2022-04-22T21:22:10.542408+0000 mgr.y (mgr.34107) 8064 : audit [DBG] from='client.53677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[46934]: audit 2022-04-22T21:22:10.963947+0000 mon.c (mon.1) 290 : audit [DBG] from='client.? 172.21.15.115:0/4006511763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:11 smithi115 bash[46934]: audit 2022-04-22T21:22:10.974876+0000 mon.a (mon.0) 11699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:12 smithi115 bash[45495]: cluster 2022-04-22T21:22:11.301356+0000 mgr.y (mgr.34107) 8065 : cluster [DBG] pgmap v9024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:12.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:12 smithi115 bash[46934]: cluster 2022-04-22T21:22:11.301356+0000 mgr.y (mgr.34107) 8065 : cluster [DBG] pgmap v9024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:12 smithi121 bash[36902]: cluster 2022-04-22T21:22:11.301356+0000 mgr.y (mgr.34107) 8065 : cluster [DBG] pgmap v9024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:14.763 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:14.408Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:15.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:14 smithi121 bash[36902]: cluster 2022-04-22T21:22:13.302481+0000 mgr.y (mgr.34107) 8066 : cluster [DBG] pgmap v9025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:14 smithi121 bash[36902]: audit 2022-04-22T21:22:13.976182+0000 mon.a (mon.0) 11700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:14 smithi115 bash[45495]: cluster 2022-04-22T21:22:13.302481+0000 mgr.y (mgr.34107) 8066 : cluster [DBG] pgmap v9025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:14 smithi115 bash[45495]: audit 2022-04-22T21:22:13.976182+0000 mon.a (mon.0) 11700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:14 smithi115 bash[46934]: cluster 2022-04-22T21:22:13.302481+0000 mgr.y (mgr.34107) 8066 : cluster [DBG] pgmap v9025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:14 smithi115 bash[46934]: audit 2022-04-22T21:22:13.976182+0000 mon.a (mon.0) 11700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:15.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[46934]: audit 2022-04-22T21:22:14.818690+0000 mon.a (mon.0) 11701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:15.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[46934]: audit 2022-04-22T21:22:15.560896+0000 mon.a (mon.0) 11702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:22:15.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[46934]: audit 2022-04-22T21:22:15.563714+0000 mon.a (mon.0) 11703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:22:15.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[46934]: audit 2022-04-22T21:22:15.565257+0000 mon.a (mon.0) 11704 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:22:15.877 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[45495]: audit 2022-04-22T21:22:14.818690+0000 mon.a (mon.0) 11701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:15.878 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[45495]: audit 2022-04-22T21:22:15.560896+0000 mon.a (mon.0) 11702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:22:15.878 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[45495]: audit 2022-04-22T21:22:15.563714+0000 mon.a (mon.0) 11703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:22:15.878 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:15 smithi115 bash[45495]: audit 2022-04-22T21:22:15.565257+0000 mon.a (mon.0) 11704 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:22:15.918 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:15 smithi121 bash[36902]: audit 2022-04-22T21:22:14.818690+0000 mon.a (mon.0) 11701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:15.918 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:15 smithi121 bash[36902]: audit 2022-04-22T21:22:15.560896+0000 mon.a (mon.0) 11702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:22:15.918 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:15 smithi121 bash[36902]: audit 2022-04-22T21:22:15.563714+0000 mon.a (mon.0) 11703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:22:15.919 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:15 smithi121 bash[36902]: audit 2022-04-22T21:22:15.565257+0000 mon.a (mon.0) 11704 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:22:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[45495]: cluster 2022-04-22T21:22:15.303534+0000 mgr.y (mgr.34107) 8067 : cluster [DBG] pgmap v9026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[45495]: audit 2022-04-22T21:22:15.927715+0000 mon.a (mon.0) 11705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:22:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[45495]: audit 2022-04-22T21:22:15.977477+0000 mon.a (mon.0) 11706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[45495]: audit 2022-04-22T21:22:15.998631+0000 mon.a (mon.0) 11707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[45495]: audit 2022-04-22T21:22:15.998966+0000 mon.a (mon.0) 11708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[46934]: cluster 2022-04-22T21:22:15.303534+0000 mgr.y (mgr.34107) 8067 : cluster [DBG] pgmap v9026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[46934]: audit 2022-04-22T21:22:15.927715+0000 mon.a (mon.0) 11705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[46934]: audit 2022-04-22T21:22:15.977477+0000 mon.a (mon.0) 11706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[46934]: audit 2022-04-22T21:22:15.998631+0000 mon.a (mon.0) 11707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:16 smithi115 bash[46934]: audit 2022-04-22T21:22:15.998966+0000 mon.a (mon.0) 11708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:22:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:16 smithi121 bash[36902]: cluster 2022-04-22T21:22:15.303534+0000 mgr.y (mgr.34107) 8067 : cluster [DBG] pgmap v9026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:16 smithi121 bash[36902]: audit 2022-04-22T21:22:15.927715+0000 mon.a (mon.0) 11705 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:22:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:16 smithi121 bash[36902]: audit 2022-04-22T21:22:15.977477+0000 mon.a (mon.0) 11706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:16 smithi121 bash[36902]: audit 2022-04-22T21:22:15.998631+0000 mon.a (mon.0) 11707 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:22:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:16 smithi121 bash[36902]: audit 2022-04-22T21:22:15.998966+0000 mon.a (mon.0) 11708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:22:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:18 smithi115 bash[45495]: cluster 2022-04-22T21:22:17.304577+0000 mgr.y (mgr.34107) 8068 : cluster [DBG] pgmap v9027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:18 smithi115 bash[46934]: cluster 2022-04-22T21:22:17.304577+0000 mgr.y (mgr.34107) 8068 : cluster [DBG] pgmap v9027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:19.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:18 smithi121 bash[36902]: cluster 2022-04-22T21:22:17.304577+0000 mgr.y (mgr.34107) 8068 : cluster [DBG] pgmap v9027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:19 smithi115 bash[45495]: audit 2022-04-22T21:22:18.979626+0000 mon.a (mon.0) 11709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:19 smithi115 bash[46934]: audit 2022-04-22T21:22:18.979626+0000 mon.a (mon.0) 11709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:19 smithi121 bash[36902]: audit 2022-04-22T21:22:18.979626+0000 mon.a (mon.0) 11709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:20 smithi115 bash[45495]: cluster 2022-04-22T21:22:19.305696+0000 mgr.y (mgr.34107) 8069 : cluster [DBG] pgmap v9028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:20 smithi115 bash[45495]: audit 2022-04-22T21:22:19.822373+0000 mon.a (mon.0) 11710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:20 smithi115 bash[46934]: cluster 2022-04-22T21:22:19.305696+0000 mgr.y (mgr.34107) 8069 : cluster [DBG] pgmap v9028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:20 smithi115 bash[46934]: audit 2022-04-22T21:22:19.822373+0000 mon.a (mon.0) 11710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:20 smithi121 bash[36902]: cluster 2022-04-22T21:22:19.305696+0000 mgr.y (mgr.34107) 8069 : cluster [DBG] pgmap v9028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:20 smithi121 bash[36902]: audit 2022-04-22T21:22:19.822373+0000 mon.a (mon.0) 11710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:21 smithi115 bash[45495]: audit 2022-04-22T21:22:20.980604+0000 mon.a (mon.0) 11711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:21 smithi115 bash[46934]: audit 2022-04-22T21:22:20.980604+0000 mon.a (mon.0) 11711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:21 smithi121 bash[36902]: audit 2022-04-22T21:22:20.980604+0000 mon.a (mon.0) 11711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:22.534Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:22 smithi121 bash[36902]: cluster 2022-04-22T21:22:21.306827+0000 mgr.y (mgr.34107) 8070 : cluster [DBG] pgmap v9029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:22 smithi115 bash[45495]: cluster 2022-04-22T21:22:21.306827+0000 mgr.y (mgr.34107) 8070 : cluster [DBG] pgmap v9029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:22 smithi115 bash[46934]: cluster 2022-04-22T21:22:21.306827+0000 mgr.y (mgr.34107) 8070 : cluster [DBG] pgmap v9029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:24.408Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:24 smithi121 bash[36902]: cluster 2022-04-22T21:22:23.307958+0000 mgr.y (mgr.34107) 8071 : cluster [DBG] pgmap v9030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:24 smithi121 bash[36902]: audit 2022-04-22T21:22:23.982491+0000 mon.a (mon.0) 11712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:24 smithi115 bash[45495]: cluster 2022-04-22T21:22:23.307958+0000 mgr.y (mgr.34107) 8071 : cluster [DBG] pgmap v9030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:24 smithi115 bash[45495]: audit 2022-04-22T21:22:23.982491+0000 mon.a (mon.0) 11712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:24 smithi115 bash[46934]: cluster 2022-04-22T21:22:23.307958+0000 mgr.y (mgr.34107) 8071 : cluster [DBG] pgmap v9030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:24 smithi115 bash[46934]: audit 2022-04-22T21:22:23.982491+0000 mon.a (mon.0) 11712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:25 smithi121 bash[36902]: audit 2022-04-22T21:22:24.825994+0000 mon.a (mon.0) 11713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:25 smithi115 bash[45495]: audit 2022-04-22T21:22:24.825994+0000 mon.a (mon.0) 11713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:25 smithi115 bash[46934]: audit 2022-04-22T21:22:24.825994+0000 mon.a (mon.0) 11713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:27.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:26 smithi121 bash[36902]: cluster 2022-04-22T21:22:25.308986+0000 mgr.y (mgr.34107) 8072 : cluster [DBG] pgmap v9031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:27.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:26 smithi121 bash[36902]: audit 2022-04-22T21:22:25.983630+0000 mon.a (mon.0) 11714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:26 smithi115 bash[45495]: cluster 2022-04-22T21:22:25.308986+0000 mgr.y (mgr.34107) 8072 : cluster [DBG] pgmap v9031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:26 smithi115 bash[45495]: audit 2022-04-22T21:22:25.983630+0000 mon.a (mon.0) 11714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:26 smithi115 bash[46934]: cluster 2022-04-22T21:22:25.308986+0000 mgr.y (mgr.34107) 8072 : cluster [DBG] pgmap v9031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:26 smithi115 bash[46934]: audit 2022-04-22T21:22:25.983630+0000 mon.a (mon.0) 11714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:28 smithi121 bash[36902]: cluster 2022-04-22T21:22:27.310084+0000 mgr.y (mgr.34107) 8073 : cluster [DBG] pgmap v9032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:28 smithi115 bash[45495]: cluster 2022-04-22T21:22:27.310084+0000 mgr.y (mgr.34107) 8073 : cluster [DBG] pgmap v9032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:28 smithi115 bash[46934]: cluster 2022-04-22T21:22:27.310084+0000 mgr.y (mgr.34107) 8073 : cluster [DBG] pgmap v9032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:29 smithi121 bash[36902]: audit 2022-04-22T21:22:28.985874+0000 mon.a (mon.0) 11715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:29 smithi115 bash[46934]: audit 2022-04-22T21:22:28.985874+0000 mon.a (mon.0) 11715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:29 smithi115 bash[45495]: audit 2022-04-22T21:22:28.985874+0000 mon.a (mon.0) 11715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:30 smithi121 bash[36902]: cluster 2022-04-22T21:22:29.311077+0000 mgr.y (mgr.34107) 8074 : cluster [DBG] pgmap v9033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:30 smithi121 bash[36902]: audit 2022-04-22T21:22:29.829421+0000 mon.a (mon.0) 11716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:30 smithi115 bash[46934]: cluster 2022-04-22T21:22:29.311077+0000 mgr.y (mgr.34107) 8074 : cluster [DBG] pgmap v9033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:30 smithi115 bash[46934]: audit 2022-04-22T21:22:29.829421+0000 mon.a (mon.0) 11716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:30 smithi115 bash[45495]: cluster 2022-04-22T21:22:29.311077+0000 mgr.y (mgr.34107) 8074 : cluster [DBG] pgmap v9033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:30 smithi115 bash[45495]: audit 2022-04-22T21:22:29.829421+0000 mon.a (mon.0) 11716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:31 smithi121 bash[36902]: audit 2022-04-22T21:22:30.986572+0000 mon.a (mon.0) 11717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:31 smithi115 bash[45495]: audit 2022-04-22T21:22:30.986572+0000 mon.a (mon.0) 11717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:31 smithi115 bash[46934]: audit 2022-04-22T21:22:30.986572+0000 mon.a (mon.0) 11717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:32.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:32.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:32 smithi121 bash[36902]: cluster 2022-04-22T21:22:31.312158+0000 mgr.y (mgr.34107) 8075 : cluster [DBG] pgmap v9034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:32 smithi115 bash[45495]: cluster 2022-04-22T21:22:31.312158+0000 mgr.y (mgr.34107) 8075 : cluster [DBG] pgmap v9034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:32 smithi115 bash[46934]: cluster 2022-04-22T21:22:31.312158+0000 mgr.y (mgr.34107) 8075 : cluster [DBG] pgmap v9034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:34.410Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:34 smithi121 bash[36902]: cluster 2022-04-22T21:22:33.313199+0000 mgr.y (mgr.34107) 8076 : cluster [DBG] pgmap v9035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:34 smithi121 bash[36902]: audit 2022-04-22T21:22:33.989015+0000 mon.a (mon.0) 11718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:34 smithi115 bash[45495]: cluster 2022-04-22T21:22:33.313199+0000 mgr.y (mgr.34107) 8076 : cluster [DBG] pgmap v9035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:34 smithi115 bash[45495]: audit 2022-04-22T21:22:33.989015+0000 mon.a (mon.0) 11718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:34 smithi115 bash[46934]: cluster 2022-04-22T21:22:33.313199+0000 mgr.y (mgr.34107) 8076 : cluster [DBG] pgmap v9035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:34 smithi115 bash[46934]: audit 2022-04-22T21:22:33.989015+0000 mon.a (mon.0) 11718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:35 smithi121 bash[36902]: audit 2022-04-22T21:22:34.833929+0000 mon.a (mon.0) 11719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:35 smithi115 bash[46934]: audit 2022-04-22T21:22:34.833929+0000 mon.a (mon.0) 11719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:35 smithi115 bash[45495]: audit 2022-04-22T21:22:34.833929+0000 mon.a (mon.0) 11719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:36 smithi121 bash[36902]: cluster 2022-04-22T21:22:35.314256+0000 mgr.y (mgr.34107) 8077 : cluster [DBG] pgmap v9036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:36 smithi121 bash[36902]: audit 2022-04-22T21:22:35.989855+0000 mon.a (mon.0) 11720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:36 smithi115 bash[45495]: cluster 2022-04-22T21:22:35.314256+0000 mgr.y (mgr.34107) 8077 : cluster [DBG] pgmap v9036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:36 smithi115 bash[45495]: audit 2022-04-22T21:22:35.989855+0000 mon.a (mon.0) 11720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:36 smithi115 bash[46934]: cluster 2022-04-22T21:22:35.314256+0000 mgr.y (mgr.34107) 8077 : cluster [DBG] pgmap v9036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:36 smithi115 bash[46934]: audit 2022-04-22T21:22:35.989855+0000 mon.a (mon.0) 11720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:38 smithi121 bash[36902]: cluster 2022-04-22T21:22:37.315296+0000 mgr.y (mgr.34107) 8078 : cluster [DBG] pgmap v9037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:38 smithi115 bash[45495]: cluster 2022-04-22T21:22:37.315296+0000 mgr.y (mgr.34107) 8078 : cluster [DBG] pgmap v9037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:38 smithi115 bash[46934]: cluster 2022-04-22T21:22:37.315296+0000 mgr.y (mgr.34107) 8078 : cluster [DBG] pgmap v9037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:39 smithi121 bash[36902]: audit 2022-04-22T21:22:38.992268+0000 mon.a (mon.0) 11721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:39 smithi115 bash[45495]: audit 2022-04-22T21:22:38.992268+0000 mon.a (mon.0) 11721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:39 smithi115 bash[46934]: audit 2022-04-22T21:22:38.992268+0000 mon.a (mon.0) 11721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:40 smithi121 bash[36902]: cluster 2022-04-22T21:22:39.316370+0000 mgr.y (mgr.34107) 8079 : cluster [DBG] pgmap v9038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:40 smithi121 bash[36902]: audit 2022-04-22T21:22:39.835764+0000 mon.a (mon.0) 11722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:40 smithi115 bash[45495]: cluster 2022-04-22T21:22:39.316370+0000 mgr.y (mgr.34107) 8079 : cluster [DBG] pgmap v9038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:40 smithi115 bash[45495]: audit 2022-04-22T21:22:39.835764+0000 mon.a (mon.0) 11722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:40 smithi115 bash[46934]: cluster 2022-04-22T21:22:39.316370+0000 mgr.y (mgr.34107) 8079 : cluster [DBG] pgmap v9038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:40 smithi115 bash[46934]: audit 2022-04-22T21:22:39.835764+0000 mon.a (mon.0) 11722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:41.345 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 3h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 9m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 8m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:22:41.691 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 8m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 9m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 8m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 3h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 3h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 8m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 8m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 8m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 8m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 9m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:22:41.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:22:41.693 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 4h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:22:41.693 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 3h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:22:42.099 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:22:42.099 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:22:42.099 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:42.100 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:22:42.101 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:22:42.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:41 smithi121 bash[36902]: audit 2022-04-22T21:22:40.994149+0000 mon.a (mon.0) 11723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:41 smithi115 bash[45495]: audit 2022-04-22T21:22:40.994149+0000 mon.a (mon.0) 11723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:41 smithi115 bash[46934]: audit 2022-04-22T21:22:40.994149+0000 mon.a (mon.0) 11723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:42.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:42 smithi121 bash[36902]: cluster 2022-04-22T21:22:41.317051+0000 mgr.y (mgr.34107) 8080 : cluster [DBG] pgmap v9039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:42 smithi121 bash[36902]: audit 2022-04-22T21:22:41.326044+0000 mgr.y (mgr.34107) 8081 : audit [DBG] from='client.43758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:42 smithi121 bash[36902]: audit 2022-04-22T21:22:41.666647+0000 mgr.y (mgr.34107) 8082 : audit [DBG] from='client.53695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:42 smithi121 bash[36902]: audit 2022-04-22T21:22:42.094593+0000 mon.c (mon.1) 291 : audit [DBG] from='client.? 172.21.15.115:0/123283990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[45495]: cluster 2022-04-22T21:22:41.317051+0000 mgr.y (mgr.34107) 8080 : cluster [DBG] pgmap v9039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[45495]: audit 2022-04-22T21:22:41.326044+0000 mgr.y (mgr.34107) 8081 : audit [DBG] from='client.43758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[45495]: audit 2022-04-22T21:22:41.666647+0000 mgr.y (mgr.34107) 8082 : audit [DBG] from='client.53695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[45495]: audit 2022-04-22T21:22:42.094593+0000 mon.c (mon.1) 291 : audit [DBG] from='client.? 172.21.15.115:0/123283990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[46934]: cluster 2022-04-22T21:22:41.317051+0000 mgr.y (mgr.34107) 8080 : cluster [DBG] pgmap v9039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[46934]: audit 2022-04-22T21:22:41.326044+0000 mgr.y (mgr.34107) 8081 : audit [DBG] from='client.43758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[46934]: audit 2022-04-22T21:22:41.666647+0000 mgr.y (mgr.34107) 8082 : audit [DBG] from='client.53695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:22:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:42 smithi115 bash[46934]: audit 2022-04-22T21:22:42.094593+0000 mon.c (mon.1) 291 : audit [DBG] from='client.? 172.21.15.115:0/123283990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:22:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:44.410Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:44 smithi121 bash[36902]: cluster 2022-04-22T21:22:43.318237+0000 mgr.y (mgr.34107) 8083 : cluster [DBG] pgmap v9040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:44 smithi121 bash[36902]: audit 2022-04-22T21:22:43.995328+0000 mon.a (mon.0) 11724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:44 smithi121 bash[36902]: audit 2022-04-22T21:22:44.839670+0000 mon.a (mon.0) 11725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[46934]: cluster 2022-04-22T21:22:43.318237+0000 mgr.y (mgr.34107) 8083 : cluster [DBG] pgmap v9040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[46934]: audit 2022-04-22T21:22:43.995328+0000 mon.a (mon.0) 11724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[46934]: audit 2022-04-22T21:22:44.839670+0000 mon.a (mon.0) 11725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[45495]: cluster 2022-04-22T21:22:43.318237+0000 mgr.y (mgr.34107) 8083 : cluster [DBG] pgmap v9040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[45495]: audit 2022-04-22T21:22:43.995328+0000 mon.a (mon.0) 11724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:44 smithi115 bash[45495]: audit 2022-04-22T21:22:44.839670+0000 mon.a (mon.0) 11725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:46 smithi121 bash[36902]: cluster 2022-04-22T21:22:45.319306+0000 mgr.y (mgr.34107) 8084 : cluster [DBG] pgmap v9041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:46 smithi121 bash[36902]: audit 2022-04-22T21:22:45.996857+0000 mon.a (mon.0) 11726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:46 smithi115 bash[46934]: cluster 2022-04-22T21:22:45.319306+0000 mgr.y (mgr.34107) 8084 : cluster [DBG] pgmap v9041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:46 smithi115 bash[46934]: audit 2022-04-22T21:22:45.996857+0000 mon.a (mon.0) 11726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:46 smithi115 bash[45495]: cluster 2022-04-22T21:22:45.319306+0000 mgr.y (mgr.34107) 8084 : cluster [DBG] pgmap v9041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:46 smithi115 bash[45495]: audit 2022-04-22T21:22:45.996857+0000 mon.a (mon.0) 11726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:48 smithi121 bash[36902]: cluster 2022-04-22T21:22:47.320429+0000 mgr.y (mgr.34107) 8085 : cluster [DBG] pgmap v9042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:48 smithi115 bash[46934]: cluster 2022-04-22T21:22:47.320429+0000 mgr.y (mgr.34107) 8085 : cluster [DBG] pgmap v9042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:48 smithi115 bash[45495]: cluster 2022-04-22T21:22:47.320429+0000 mgr.y (mgr.34107) 8085 : cluster [DBG] pgmap v9042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:49 smithi121 bash[36902]: audit 2022-04-22T21:22:48.998875+0000 mon.a (mon.0) 11727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:49 smithi115 bash[45495]: audit 2022-04-22T21:22:48.998875+0000 mon.a (mon.0) 11727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:49 smithi115 bash[46934]: audit 2022-04-22T21:22:48.998875+0000 mon.a (mon.0) 11727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:50 smithi121 bash[36902]: cluster 2022-04-22T21:22:49.321465+0000 mgr.y (mgr.34107) 8086 : cluster [DBG] pgmap v9043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:50 smithi121 bash[36902]: audit 2022-04-22T21:22:49.843256+0000 mon.a (mon.0) 11728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:50 smithi115 bash[45495]: cluster 2022-04-22T21:22:49.321465+0000 mgr.y (mgr.34107) 8086 : cluster [DBG] pgmap v9043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:50 smithi115 bash[45495]: audit 2022-04-22T21:22:49.843256+0000 mon.a (mon.0) 11728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:50 smithi115 bash[46934]: cluster 2022-04-22T21:22:49.321465+0000 mgr.y (mgr.34107) 8086 : cluster [DBG] pgmap v9043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:50.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:50 smithi115 bash[46934]: audit 2022-04-22T21:22:49.843256+0000 mon.a (mon.0) 11728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:51.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:51 smithi121 bash[36902]: audit 2022-04-22T21:22:50.999874+0000 mon.a (mon.0) 11729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:51 smithi115 bash[45495]: audit 2022-04-22T21:22:50.999874+0000 mon.a (mon.0) 11729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:51 smithi115 bash[46934]: audit 2022-04-22T21:22:50.999874+0000 mon.a (mon.0) 11729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:52.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:52 smithi121 bash[36902]: cluster 2022-04-22T21:22:51.322582+0000 mgr.y (mgr.34107) 8087 : cluster [DBG] pgmap v9044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:52 smithi115 bash[45495]: cluster 2022-04-22T21:22:51.322582+0000 mgr.y (mgr.34107) 8087 : cluster [DBG] pgmap v9044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:52 smithi115 bash[46934]: cluster 2022-04-22T21:22:51.322582+0000 mgr.y (mgr.34107) 8087 : cluster [DBG] pgmap v9044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:52.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:52.536Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:54 smithi121 bash[36902]: cluster 2022-04-22T21:22:53.323762+0000 mgr.y (mgr.34107) 8088 : cluster [DBG] pgmap v9045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:54 smithi121 bash[36902]: audit 2022-04-22T21:22:54.001927+0000 mon.a (mon.0) 11730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:22:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:22:54.411Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:22:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:54 smithi115 bash[45495]: cluster 2022-04-22T21:22:53.323762+0000 mgr.y (mgr.34107) 8088 : cluster [DBG] pgmap v9045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:54 smithi115 bash[45495]: audit 2022-04-22T21:22:54.001927+0000 mon.a (mon.0) 11730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:54 smithi115 bash[46934]: cluster 2022-04-22T21:22:53.323762+0000 mgr.y (mgr.34107) 8088 : cluster [DBG] pgmap v9045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:54 smithi115 bash[46934]: audit 2022-04-22T21:22:54.001927+0000 mon.a (mon.0) 11730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:55 smithi121 bash[36902]: audit 2022-04-22T21:22:54.846676+0000 mon.a (mon.0) 11731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:55 smithi115 bash[45495]: audit 2022-04-22T21:22:54.846676+0000 mon.a (mon.0) 11731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:55 smithi115 bash[46934]: audit 2022-04-22T21:22:54.846676+0000 mon.a (mon.0) 11731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:22:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:56 smithi121 bash[36902]: cluster 2022-04-22T21:22:55.324897+0000 mgr.y (mgr.34107) 8089 : cluster [DBG] pgmap v9046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:56 smithi121 bash[36902]: audit 2022-04-22T21:22:56.002640+0000 mon.a (mon.0) 11732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:56 smithi115 bash[45495]: cluster 2022-04-22T21:22:55.324897+0000 mgr.y (mgr.34107) 8089 : cluster [DBG] pgmap v9046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:56 smithi115 bash[45495]: audit 2022-04-22T21:22:56.002640+0000 mon.a (mon.0) 11732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:56 smithi115 bash[46934]: cluster 2022-04-22T21:22:55.324897+0000 mgr.y (mgr.34107) 8089 : cluster [DBG] pgmap v9046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:56 smithi115 bash[46934]: audit 2022-04-22T21:22:56.002640+0000 mon.a (mon.0) 11732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:22:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:58 smithi121 bash[36902]: cluster 2022-04-22T21:22:57.326132+0000 mgr.y (mgr.34107) 8090 : cluster [DBG] pgmap v9047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:58 smithi115 bash[45495]: cluster 2022-04-22T21:22:57.326132+0000 mgr.y (mgr.34107) 8090 : cluster [DBG] pgmap v9047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:58 smithi115 bash[46934]: cluster 2022-04-22T21:22:57.326132+0000 mgr.y (mgr.34107) 8090 : cluster [DBG] pgmap v9047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:22:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:22:59 smithi121 bash[36902]: audit 2022-04-22T21:22:59.005060+0000 mon.a (mon.0) 11733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:22:59 smithi115 bash[45495]: audit 2022-04-22T21:22:59.005060+0000 mon.a (mon.0) 11733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:22:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:22:59 smithi115 bash[46934]: audit 2022-04-22T21:22:59.005060+0000 mon.a (mon.0) 11733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:00 smithi121 bash[36902]: cluster 2022-04-22T21:22:59.327199+0000 mgr.y (mgr.34107) 8091 : cluster [DBG] pgmap v9048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:00 smithi121 bash[36902]: audit 2022-04-22T21:22:59.850383+0000 mon.a (mon.0) 11734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:00 smithi115 bash[45495]: cluster 2022-04-22T21:22:59.327199+0000 mgr.y (mgr.34107) 8091 : cluster [DBG] pgmap v9048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:00 smithi115 bash[45495]: audit 2022-04-22T21:22:59.850383+0000 mon.a (mon.0) 11734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:00 smithi115 bash[46934]: cluster 2022-04-22T21:22:59.327199+0000 mgr.y (mgr.34107) 8091 : cluster [DBG] pgmap v9048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:00 smithi115 bash[46934]: audit 2022-04-22T21:22:59.850383+0000 mon.a (mon.0) 11734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:01 smithi121 bash[36902]: audit 2022-04-22T21:23:01.004972+0000 mon.a (mon.0) 11735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:01 smithi115 bash[45495]: audit 2022-04-22T21:23:01.004972+0000 mon.a (mon.0) 11735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:01 smithi115 bash[46934]: audit 2022-04-22T21:23:01.004972+0000 mon.a (mon.0) 11735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:02 smithi121 bash[36902]: cluster 2022-04-22T21:23:01.328312+0000 mgr.y (mgr.34107) 8092 : cluster [DBG] pgmap v9049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:02 smithi115 bash[45495]: cluster 2022-04-22T21:23:01.328312+0000 mgr.y (mgr.34107) 8092 : cluster [DBG] pgmap v9049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:02.537Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:02 smithi115 bash[46934]: cluster 2022-04-22T21:23:01.328312+0000 mgr.y (mgr.34107) 8092 : cluster [DBG] pgmap v9049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:04 smithi121 bash[36902]: cluster 2022-04-22T21:23:03.329450+0000 mgr.y (mgr.34107) 8093 : cluster [DBG] pgmap v9050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:04 smithi121 bash[36902]: audit 2022-04-22T21:23:04.008197+0000 mon.a (mon.0) 11736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:04.412Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:04 smithi115 bash[45495]: cluster 2022-04-22T21:23:03.329450+0000 mgr.y (mgr.34107) 8093 : cluster [DBG] pgmap v9050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:04 smithi115 bash[45495]: audit 2022-04-22T21:23:04.008197+0000 mon.a (mon.0) 11736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:04 smithi115 bash[46934]: cluster 2022-04-22T21:23:03.329450+0000 mgr.y (mgr.34107) 8093 : cluster [DBG] pgmap v9050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:04 smithi115 bash[46934]: audit 2022-04-22T21:23:04.008197+0000 mon.a (mon.0) 11736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:05.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:05 smithi121 bash[36902]: audit 2022-04-22T21:23:04.854182+0000 mon.a (mon.0) 11737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:05 smithi115 bash[45495]: audit 2022-04-22T21:23:04.854182+0000 mon.a (mon.0) 11737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:05 smithi115 bash[46934]: audit 2022-04-22T21:23:04.854182+0000 mon.a (mon.0) 11737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:06 smithi121 bash[36902]: cluster 2022-04-22T21:23:05.330538+0000 mgr.y (mgr.34107) 8094 : cluster [DBG] pgmap v9051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:06.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:06 smithi121 bash[36902]: audit 2022-04-22T21:23:06.007720+0000 mon.a (mon.0) 11738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:06 smithi115 bash[45495]: cluster 2022-04-22T21:23:05.330538+0000 mgr.y (mgr.34107) 8094 : cluster [DBG] pgmap v9051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:06 smithi115 bash[45495]: audit 2022-04-22T21:23:06.007720+0000 mon.a (mon.0) 11738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:06 smithi115 bash[46934]: cluster 2022-04-22T21:23:05.330538+0000 mgr.y (mgr.34107) 8094 : cluster [DBG] pgmap v9051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:06 smithi115 bash[46934]: audit 2022-04-22T21:23:06.007720+0000 mon.a (mon.0) 11738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:08 smithi121 bash[36902]: cluster 2022-04-22T21:23:07.331697+0000 mgr.y (mgr.34107) 8095 : cluster [DBG] pgmap v9052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:08 smithi115 bash[45495]: cluster 2022-04-22T21:23:07.331697+0000 mgr.y (mgr.34107) 8095 : cluster [DBG] pgmap v9052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:08 smithi115 bash[46934]: cluster 2022-04-22T21:23:07.331697+0000 mgr.y (mgr.34107) 8095 : cluster [DBG] pgmap v9052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:09 smithi121 bash[36902]: audit 2022-04-22T21:23:09.011454+0000 mon.a (mon.0) 11739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:09 smithi115 bash[46934]: audit 2022-04-22T21:23:09.011454+0000 mon.a (mon.0) 11739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:09.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:09 smithi115 bash[45495]: audit 2022-04-22T21:23:09.011454+0000 mon.a (mon.0) 11739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:10 smithi121 bash[36902]: cluster 2022-04-22T21:23:09.332690+0000 mgr.y (mgr.34107) 8096 : cluster [DBG] pgmap v9053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:10 smithi121 bash[36902]: audit 2022-04-22T21:23:09.857849+0000 mon.a (mon.0) 11740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:10 smithi115 bash[46934]: cluster 2022-04-22T21:23:09.332690+0000 mgr.y (mgr.34107) 8096 : cluster [DBG] pgmap v9053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:10 smithi115 bash[46934]: audit 2022-04-22T21:23:09.857849+0000 mon.a (mon.0) 11740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:10 smithi115 bash[45495]: cluster 2022-04-22T21:23:09.332690+0000 mgr.y (mgr.34107) 8096 : cluster [DBG] pgmap v9053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:10 smithi115 bash[45495]: audit 2022-04-22T21:23:09.857849+0000 mon.a (mon.0) 11740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:11 smithi121 bash[36902]: audit 2022-04-22T21:23:11.010072+0000 mon.a (mon.0) 11741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:11 smithi115 bash[46934]: audit 2022-04-22T21:23:11.010072+0000 mon.a (mon.0) 11741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:11 smithi115 bash[45495]: audit 2022-04-22T21:23:11.010072+0000 mon.a (mon.0) 11741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:12.472 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:23:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:12 smithi121 bash[36902]: cluster 2022-04-22T21:23:11.333791+0000 mgr.y (mgr.34107) 8097 : cluster [DBG] pgmap v9054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:12 smithi115 bash[45495]: cluster 2022-04-22T21:23:11.333791+0000 mgr.y (mgr.34107) 8097 : cluster [DBG] pgmap v9054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:12.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:12 smithi115 bash[46934]: cluster 2022-04-22T21:23:11.333791+0000 mgr.y (mgr.34107) 8097 : cluster [DBG] pgmap v9054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:12.814 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:23:12.814 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:23:12.814 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (3h) 10m ago 3h 25.8M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 10m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 10m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 10m ago 4h 240M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 4h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 10m ago 4h 11.0M - 1dbe0e931976 f4b044642a96 2022-04-22T21:23:12.815 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 10m ago 4h 581M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 10m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 10m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 10m ago 4h 643M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:23:12.816 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 10m ago 4h 64.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:23:13.215 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:23:13.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:23:13.217 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:23:13.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:13 smithi121 bash[36902]: audit 2022-04-22T21:23:12.450897+0000 mgr.y (mgr.34107) 8098 : audit [DBG] from='client.53707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:13 smithi121 bash[36902]: audit 2022-04-22T21:23:12.796720+0000 mgr.y (mgr.34107) 8099 : audit [DBG] from='client.43782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:13 smithi121 bash[36902]: audit 2022-04-22T21:23:13.211605+0000 mon.c (mon.1) 292 : audit [DBG] from='client.? 172.21.15.115:0/46162566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[45495]: audit 2022-04-22T21:23:12.450897+0000 mgr.y (mgr.34107) 8098 : audit [DBG] from='client.53707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[45495]: audit 2022-04-22T21:23:12.796720+0000 mgr.y (mgr.34107) 8099 : audit [DBG] from='client.43782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[45495]: audit 2022-04-22T21:23:13.211605+0000 mon.c (mon.1) 292 : audit [DBG] from='client.? 172.21.15.115:0/46162566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[46934]: audit 2022-04-22T21:23:12.450897+0000 mgr.y (mgr.34107) 8098 : audit [DBG] from='client.53707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[46934]: audit 2022-04-22T21:23:12.796720+0000 mgr.y (mgr.34107) 8099 : audit [DBG] from='client.43782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:13 smithi115 bash[46934]: audit 2022-04-22T21:23:13.211605+0000 mon.c (mon.1) 292 : audit [DBG] from='client.? 172.21.15.115:0/46162566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:14 smithi121 bash[36902]: cluster 2022-04-22T21:23:13.334912+0000 mgr.y (mgr.34107) 8100 : cluster [DBG] pgmap v9055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:14 smithi121 bash[36902]: audit 2022-04-22T21:23:14.014819+0000 mon.a (mon.0) 11742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:14.412Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:14 smithi115 bash[46934]: cluster 2022-04-22T21:23:13.334912+0000 mgr.y (mgr.34107) 8100 : cluster [DBG] pgmap v9055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:14 smithi115 bash[46934]: audit 2022-04-22T21:23:14.014819+0000 mon.a (mon.0) 11742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:14 smithi115 bash[45495]: cluster 2022-04-22T21:23:13.334912+0000 mgr.y (mgr.34107) 8100 : cluster [DBG] pgmap v9055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:14 smithi115 bash[45495]: audit 2022-04-22T21:23:14.014819+0000 mon.a (mon.0) 11742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:15.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:15 smithi121 bash[36902]: audit 2022-04-22T21:23:14.861505+0000 mon.a (mon.0) 11743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:15 smithi115 bash[45495]: audit 2022-04-22T21:23:14.861505+0000 mon.a (mon.0) 11743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:15 smithi115 bash[46934]: audit 2022-04-22T21:23:14.861505+0000 mon.a (mon.0) 11743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: cluster 2022-04-22T21:23:15.336030+0000 mgr.y (mgr.34107) 8101 : cluster [DBG] pgmap v9056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:15.935835+0000 mon.a (mon.0) 11744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:15.939909+0000 mon.a (mon.0) 11745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:15.941696+0000 mon.a (mon.0) 11746 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:16.005410+0000 mon.a (mon.0) 11747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:16.005975+0000 mon.a (mon.0) 11748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:23:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:16 smithi121 bash[36902]: audit 2022-04-22T21:23:16.012867+0000 mon.a (mon.0) 11749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: cluster 2022-04-22T21:23:15.336030+0000 mgr.y (mgr.34107) 8101 : cluster [DBG] pgmap v9056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:15.935835+0000 mon.a (mon.0) 11744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:23:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:15.939909+0000 mon.a (mon.0) 11745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:15.941696+0000 mon.a (mon.0) 11746 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:16.005410+0000 mon.a (mon.0) 11747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:16.005975+0000 mon.a (mon.0) 11748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[45495]: audit 2022-04-22T21:23:16.012867+0000 mon.a (mon.0) 11749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: cluster 2022-04-22T21:23:15.336030+0000 mgr.y (mgr.34107) 8101 : cluster [DBG] pgmap v9056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:15.935835+0000 mon.a (mon.0) 11744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:15.939909+0000 mon.a (mon.0) 11745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:23:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:15.941696+0000 mon.a (mon.0) 11746 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:23:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:16.005410+0000 mon.a (mon.0) 11747 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:23:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:16.005975+0000 mon.a (mon.0) 11748 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:23:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:16 smithi115 bash[46934]: audit 2022-04-22T21:23:16.012867+0000 mon.a (mon.0) 11749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:18 smithi121 bash[36902]: cluster 2022-04-22T21:23:17.337127+0000 mgr.y (mgr.34107) 8102 : cluster [DBG] pgmap v9057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:18 smithi115 bash[45495]: cluster 2022-04-22T21:23:17.337127+0000 mgr.y (mgr.34107) 8102 : cluster [DBG] pgmap v9057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:18 smithi115 bash[46934]: cluster 2022-04-22T21:23:17.337127+0000 mgr.y (mgr.34107) 8102 : cluster [DBG] pgmap v9057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:19 smithi121 bash[36902]: audit 2022-04-22T21:23:19.017951+0000 mon.a (mon.0) 11750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:19 smithi115 bash[45495]: audit 2022-04-22T21:23:19.017951+0000 mon.a (mon.0) 11750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:19 smithi115 bash[46934]: audit 2022-04-22T21:23:19.017951+0000 mon.a (mon.0) 11750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:20 smithi121 bash[36902]: cluster 2022-04-22T21:23:19.338214+0000 mgr.y (mgr.34107) 8103 : cluster [DBG] pgmap v9058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:20 smithi121 bash[36902]: audit 2022-04-22T21:23:19.864953+0000 mon.a (mon.0) 11751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:20 smithi121 bash[36902]: audit 2022-04-22T21:23:20.430659+0000 mon.a (mon.0) 11752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[45495]: cluster 2022-04-22T21:23:19.338214+0000 mgr.y (mgr.34107) 8103 : cluster [DBG] pgmap v9058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[45495]: audit 2022-04-22T21:23:19.864953+0000 mon.a (mon.0) 11751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[45495]: audit 2022-04-22T21:23:20.430659+0000 mon.a (mon.0) 11752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[46934]: cluster 2022-04-22T21:23:19.338214+0000 mgr.y (mgr.34107) 8103 : cluster [DBG] pgmap v9058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[46934]: audit 2022-04-22T21:23:19.864953+0000 mon.a (mon.0) 11751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:20 smithi115 bash[46934]: audit 2022-04-22T21:23:20.430659+0000 mon.a (mon.0) 11752 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:21 smithi121 bash[36902]: audit 2022-04-22T21:23:20.793150+0000 mon.a (mon.0) 11753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:21 smithi121 bash[36902]: audit 2022-04-22T21:23:21.015312+0000 mon.a (mon.0) 11754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:21 smithi115 bash[45495]: audit 2022-04-22T21:23:20.793150+0000 mon.a (mon.0) 11753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:21 smithi115 bash[45495]: audit 2022-04-22T21:23:21.015312+0000 mon.a (mon.0) 11754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:21 smithi115 bash[46934]: audit 2022-04-22T21:23:20.793150+0000 mon.a (mon.0) 11753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:23:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:21 smithi115 bash[46934]: audit 2022-04-22T21:23:21.015312+0000 mon.a (mon.0) 11754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:22.803 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:22.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:22 smithi121 bash[36902]: cluster 2022-04-22T21:23:21.339303+0000 mgr.y (mgr.34107) 8104 : cluster [DBG] pgmap v9059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:22 smithi115 bash[45495]: cluster 2022-04-22T21:23:21.339303+0000 mgr.y (mgr.34107) 8104 : cluster [DBG] pgmap v9059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:22 smithi115 bash[46934]: cluster 2022-04-22T21:23:21.339303+0000 mgr.y (mgr.34107) 8104 : cluster [DBG] pgmap v9059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:24.413Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:24 smithi121 bash[36902]: cluster 2022-04-22T21:23:23.340478+0000 mgr.y (mgr.34107) 8105 : cluster [DBG] pgmap v9060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:24 smithi121 bash[36902]: audit 2022-04-22T21:23:24.021082+0000 mon.a (mon.0) 11755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:24 smithi115 bash[45495]: cluster 2022-04-22T21:23:23.340478+0000 mgr.y (mgr.34107) 8105 : cluster [DBG] pgmap v9060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:24 smithi115 bash[45495]: audit 2022-04-22T21:23:24.021082+0000 mon.a (mon.0) 11755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:24 smithi115 bash[46934]: cluster 2022-04-22T21:23:23.340478+0000 mgr.y (mgr.34107) 8105 : cluster [DBG] pgmap v9060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:24 smithi115 bash[46934]: audit 2022-04-22T21:23:24.021082+0000 mon.a (mon.0) 11755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:25 smithi121 bash[36902]: audit 2022-04-22T21:23:24.868780+0000 mon.a (mon.0) 11756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:25 smithi115 bash[45495]: audit 2022-04-22T21:23:24.868780+0000 mon.a (mon.0) 11756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:25 smithi115 bash[46934]: audit 2022-04-22T21:23:24.868780+0000 mon.a (mon.0) 11756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:26 smithi121 bash[36902]: cluster 2022-04-22T21:23:25.341553+0000 mgr.y (mgr.34107) 8106 : cluster [DBG] pgmap v9061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:26 smithi121 bash[36902]: audit 2022-04-22T21:23:26.018483+0000 mon.a (mon.0) 11757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:26 smithi115 bash[45495]: cluster 2022-04-22T21:23:25.341553+0000 mgr.y (mgr.34107) 8106 : cluster [DBG] pgmap v9061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:26 smithi115 bash[45495]: audit 2022-04-22T21:23:26.018483+0000 mon.a (mon.0) 11757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:26 smithi115 bash[46934]: cluster 2022-04-22T21:23:25.341553+0000 mgr.y (mgr.34107) 8106 : cluster [DBG] pgmap v9061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:26 smithi115 bash[46934]: audit 2022-04-22T21:23:26.018483+0000 mon.a (mon.0) 11757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:28 smithi121 bash[36902]: cluster 2022-04-22T21:23:27.342623+0000 mgr.y (mgr.34107) 8107 : cluster [DBG] pgmap v9062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:29.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:28 smithi115 bash[45495]: cluster 2022-04-22T21:23:27.342623+0000 mgr.y (mgr.34107) 8107 : cluster [DBG] pgmap v9062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:29.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:28 smithi115 bash[46934]: cluster 2022-04-22T21:23:27.342623+0000 mgr.y (mgr.34107) 8107 : cluster [DBG] pgmap v9062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:29 smithi121 bash[36902]: audit 2022-04-22T21:23:29.024179+0000 mon.a (mon.0) 11758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:29 smithi115 bash[45495]: audit 2022-04-22T21:23:29.024179+0000 mon.a (mon.0) 11758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:29 smithi115 bash[46934]: audit 2022-04-22T21:23:29.024179+0000 mon.a (mon.0) 11758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:30 smithi121 bash[36902]: cluster 2022-04-22T21:23:29.343707+0000 mgr.y (mgr.34107) 8108 : cluster [DBG] pgmap v9063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:30 smithi121 bash[36902]: audit 2022-04-22T21:23:29.872620+0000 mon.a (mon.0) 11759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:30 smithi115 bash[45495]: cluster 2022-04-22T21:23:29.343707+0000 mgr.y (mgr.34107) 8108 : cluster [DBG] pgmap v9063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:30 smithi115 bash[45495]: audit 2022-04-22T21:23:29.872620+0000 mon.a (mon.0) 11759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:30 smithi115 bash[46934]: cluster 2022-04-22T21:23:29.343707+0000 mgr.y (mgr.34107) 8108 : cluster [DBG] pgmap v9063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:30 smithi115 bash[46934]: audit 2022-04-22T21:23:29.872620+0000 mon.a (mon.0) 11759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:31 smithi121 bash[36902]: audit 2022-04-22T21:23:31.022243+0000 mon.a (mon.0) 11760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:31 smithi115 bash[45495]: audit 2022-04-22T21:23:31.022243+0000 mon.a (mon.0) 11760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:31 smithi115 bash[46934]: audit 2022-04-22T21:23:31.022243+0000 mon.a (mon.0) 11760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:32.842 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:32.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:33.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:32 smithi121 bash[36902]: cluster 2022-04-22T21:23:31.344808+0000 mgr.y (mgr.34107) 8109 : cluster [DBG] pgmap v9064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:32 smithi115 bash[45495]: cluster 2022-04-22T21:23:31.344808+0000 mgr.y (mgr.34107) 8109 : cluster [DBG] pgmap v9064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:32 smithi115 bash[46934]: cluster 2022-04-22T21:23:31.344808+0000 mgr.y (mgr.34107) 8109 : cluster [DBG] pgmap v9064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:34.413Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:34 smithi121 bash[36902]: cluster 2022-04-22T21:23:33.345981+0000 mgr.y (mgr.34107) 8110 : cluster [DBG] pgmap v9065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:34 smithi121 bash[36902]: audit 2022-04-22T21:23:34.027286+0000 mon.a (mon.0) 11761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:34 smithi115 bash[45495]: cluster 2022-04-22T21:23:33.345981+0000 mgr.y (mgr.34107) 8110 : cluster [DBG] pgmap v9065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:34 smithi115 bash[45495]: audit 2022-04-22T21:23:34.027286+0000 mon.a (mon.0) 11761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:34 smithi115 bash[46934]: cluster 2022-04-22T21:23:33.345981+0000 mgr.y (mgr.34107) 8110 : cluster [DBG] pgmap v9065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:34 smithi115 bash[46934]: audit 2022-04-22T21:23:34.027286+0000 mon.a (mon.0) 11761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:35 smithi121 bash[36902]: audit 2022-04-22T21:23:34.876229+0000 mon.a (mon.0) 11762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:35 smithi115 bash[45495]: audit 2022-04-22T21:23:34.876229+0000 mon.a (mon.0) 11762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:35 smithi115 bash[46934]: audit 2022-04-22T21:23:34.876229+0000 mon.a (mon.0) 11762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:36 smithi121 bash[36902]: cluster 2022-04-22T21:23:35.347091+0000 mgr.y (mgr.34107) 8111 : cluster [DBG] pgmap v9066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:36 smithi121 bash[36902]: audit 2022-04-22T21:23:36.025255+0000 mon.a (mon.0) 11763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:36 smithi115 bash[45495]: cluster 2022-04-22T21:23:35.347091+0000 mgr.y (mgr.34107) 8111 : cluster [DBG] pgmap v9066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:36 smithi115 bash[45495]: audit 2022-04-22T21:23:36.025255+0000 mon.a (mon.0) 11763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:36 smithi115 bash[46934]: cluster 2022-04-22T21:23:35.347091+0000 mgr.y (mgr.34107) 8111 : cluster [DBG] pgmap v9066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:36 smithi115 bash[46934]: audit 2022-04-22T21:23:36.025255+0000 mon.a (mon.0) 11763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:38 smithi121 bash[36902]: cluster 2022-04-22T21:23:37.348211+0000 mgr.y (mgr.34107) 8112 : cluster [DBG] pgmap v9067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:38 smithi115 bash[45495]: cluster 2022-04-22T21:23:37.348211+0000 mgr.y (mgr.34107) 8112 : cluster [DBG] pgmap v9067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:38 smithi115 bash[46934]: cluster 2022-04-22T21:23:37.348211+0000 mgr.y (mgr.34107) 8112 : cluster [DBG] pgmap v9067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:39 smithi121 bash[36902]: audit 2022-04-22T21:23:39.030570+0000 mon.a (mon.0) 11764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:39 smithi115 bash[45495]: audit 2022-04-22T21:23:39.030570+0000 mon.a (mon.0) 11764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:39 smithi115 bash[46934]: audit 2022-04-22T21:23:39.030570+0000 mon.a (mon.0) 11764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:40 smithi121 bash[36902]: cluster 2022-04-22T21:23:39.349266+0000 mgr.y (mgr.34107) 8113 : cluster [DBG] pgmap v9068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:40 smithi121 bash[36902]: audit 2022-04-22T21:23:39.879866+0000 mon.a (mon.0) 11765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:40 smithi115 bash[45495]: cluster 2022-04-22T21:23:39.349266+0000 mgr.y (mgr.34107) 8113 : cluster [DBG] pgmap v9068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:40 smithi115 bash[45495]: audit 2022-04-22T21:23:39.879866+0000 mon.a (mon.0) 11765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:40 smithi115 bash[46934]: cluster 2022-04-22T21:23:39.349266+0000 mgr.y (mgr.34107) 8113 : cluster [DBG] pgmap v9068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:40 smithi115 bash[46934]: audit 2022-04-22T21:23:39.879866+0000 mon.a (mon.0) 11765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:42.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:41 smithi121 bash[36902]: audit 2022-04-22T21:23:41.028027+0000 mon.a (mon.0) 11766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:41 smithi115 bash[45495]: audit 2022-04-22T21:23:41.028027+0000 mon.a (mon.0) 11766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:41 smithi115 bash[46934]: audit 2022-04-22T21:23:41.028027+0000 mon.a (mon.0) 11766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:42.874 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:42.540Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:42 smithi121 bash[36902]: cluster 2022-04-22T21:23:41.350364+0000 mgr.y (mgr.34107) 8114 : cluster [DBG] pgmap v9069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:42 smithi115 bash[45495]: cluster 2022-04-22T21:23:41.350364+0000 mgr.y (mgr.34107) 8114 : cluster [DBG] pgmap v9069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:42 smithi115 bash[46934]: cluster 2022-04-22T21:23:41.350364+0000 mgr.y (mgr.34107) 8114 : cluster [DBG] pgmap v9069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:43.595 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:23:43.936 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:23:43.936 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 23s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 23s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 23s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 9m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 9m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 23s ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:23:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 9m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:23:43.938 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 4h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:23:43.938 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 23s ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:23:43.938 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 9m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 9m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 9m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 9m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 23s ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 23s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 23s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 23s ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:23:43.939 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 23s ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:23:44.340 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:23:44.340 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:23:44.340 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:23:44.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:23:44.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:23:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:23:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:23:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:23:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:23:44.343 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:23:44.343 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:23:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:44.413Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:44 smithi121 bash[36902]: cluster 2022-04-22T21:23:43.351199+0000 mgr.y (mgr.34107) 8115 : cluster [DBG] pgmap v9070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:44 smithi121 bash[36902]: audit 2022-04-22T21:23:43.577325+0000 mgr.y (mgr.34107) 8116 : audit [DBG] from='client.53725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:44 smithi121 bash[36902]: audit 2022-04-22T21:23:44.033743+0000 mon.a (mon.0) 11767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:44 smithi121 bash[36902]: audit 2022-04-22T21:23:44.336157+0000 mon.c (mon.1) 293 : audit [DBG] from='client.? 172.21.15.115:0/1961186679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[45495]: cluster 2022-04-22T21:23:43.351199+0000 mgr.y (mgr.34107) 8115 : cluster [DBG] pgmap v9070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[45495]: audit 2022-04-22T21:23:43.577325+0000 mgr.y (mgr.34107) 8116 : audit [DBG] from='client.53725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[45495]: audit 2022-04-22T21:23:44.033743+0000 mon.a (mon.0) 11767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[45495]: audit 2022-04-22T21:23:44.336157+0000 mon.c (mon.1) 293 : audit [DBG] from='client.? 172.21.15.115:0/1961186679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[46934]: cluster 2022-04-22T21:23:43.351199+0000 mgr.y (mgr.34107) 8115 : cluster [DBG] pgmap v9070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[46934]: audit 2022-04-22T21:23:43.577325+0000 mgr.y (mgr.34107) 8116 : audit [DBG] from='client.53725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[46934]: audit 2022-04-22T21:23:44.033743+0000 mon.a (mon.0) 11767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:45.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:44 smithi115 bash[46934]: audit 2022-04-22T21:23:44.336157+0000 mon.c (mon.1) 293 : audit [DBG] from='client.? 172.21.15.115:0/1961186679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:23:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:45 smithi121 bash[36902]: audit 2022-04-22T21:23:43.918482+0000 mgr.y (mgr.34107) 8117 : audit [DBG] from='client.53731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:45 smithi121 bash[36902]: audit 2022-04-22T21:23:44.882993+0000 mon.a (mon.0) 11768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:45 smithi115 bash[45495]: audit 2022-04-22T21:23:43.918482+0000 mgr.y (mgr.34107) 8117 : audit [DBG] from='client.53731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:45 smithi115 bash[45495]: audit 2022-04-22T21:23:44.882993+0000 mon.a (mon.0) 11768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:45 smithi115 bash[46934]: audit 2022-04-22T21:23:43.918482+0000 mgr.y (mgr.34107) 8117 : audit [DBG] from='client.53731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:23:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:45 smithi115 bash[46934]: audit 2022-04-22T21:23:44.882993+0000 mon.a (mon.0) 11768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:46 smithi121 bash[36902]: cluster 2022-04-22T21:23:45.352271+0000 mgr.y (mgr.34107) 8118 : cluster [DBG] pgmap v9071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:46 smithi121 bash[36902]: audit 2022-04-22T21:23:46.030692+0000 mon.a (mon.0) 11769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:46 smithi115 bash[45495]: cluster 2022-04-22T21:23:45.352271+0000 mgr.y (mgr.34107) 8118 : cluster [DBG] pgmap v9071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:46 smithi115 bash[45495]: audit 2022-04-22T21:23:46.030692+0000 mon.a (mon.0) 11769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:46 smithi115 bash[46934]: cluster 2022-04-22T21:23:45.352271+0000 mgr.y (mgr.34107) 8118 : cluster [DBG] pgmap v9071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:46 smithi115 bash[46934]: audit 2022-04-22T21:23:46.030692+0000 mon.a (mon.0) 11769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:47 smithi121 bash[36902]: cluster 2022-04-22T21:23:47.353441+0000 mgr.y (mgr.34107) 8119 : cluster [DBG] pgmap v9072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:47 smithi115 bash[45495]: cluster 2022-04-22T21:23:47.353441+0000 mgr.y (mgr.34107) 8119 : cluster [DBG] pgmap v9072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:47 smithi115 bash[46934]: cluster 2022-04-22T21:23:47.353441+0000 mgr.y (mgr.34107) 8119 : cluster [DBG] pgmap v9072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:49 smithi121 bash[36902]: audit 2022-04-22T21:23:49.035820+0000 mon.a (mon.0) 11770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:49 smithi115 bash[45495]: audit 2022-04-22T21:23:49.035820+0000 mon.a (mon.0) 11770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:49 smithi115 bash[46934]: audit 2022-04-22T21:23:49.035820+0000 mon.a (mon.0) 11770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:50 smithi121 bash[36902]: cluster 2022-04-22T21:23:49.354639+0000 mgr.y (mgr.34107) 8120 : cluster [DBG] pgmap v9073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:50.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:50 smithi121 bash[36902]: audit 2022-04-22T21:23:49.886252+0000 mon.a (mon.0) 11771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:50 smithi115 bash[45495]: cluster 2022-04-22T21:23:49.354639+0000 mgr.y (mgr.34107) 8120 : cluster [DBG] pgmap v9073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:50 smithi115 bash[45495]: audit 2022-04-22T21:23:49.886252+0000 mon.a (mon.0) 11771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:50 smithi115 bash[46934]: cluster 2022-04-22T21:23:49.354639+0000 mgr.y (mgr.34107) 8120 : cluster [DBG] pgmap v9073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:50 smithi115 bash[46934]: audit 2022-04-22T21:23:49.886252+0000 mon.a (mon.0) 11771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:51.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:51 smithi121 bash[36902]: audit 2022-04-22T21:23:51.033101+0000 mon.a (mon.0) 11772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:51 smithi115 bash[45495]: audit 2022-04-22T21:23:51.033101+0000 mon.a (mon.0) 11772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:51 smithi115 bash[46934]: audit 2022-04-22T21:23:51.033101+0000 mon.a (mon.0) 11772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:52.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:52 smithi121 bash[36902]: cluster 2022-04-22T21:23:51.355756+0000 mgr.y (mgr.34107) 8121 : cluster [DBG] pgmap v9074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:52 smithi115 bash[45495]: cluster 2022-04-22T21:23:51.355756+0000 mgr.y (mgr.34107) 8121 : cluster [DBG] pgmap v9074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:52 smithi115 bash[46934]: cluster 2022-04-22T21:23:51.355756+0000 mgr.y (mgr.34107) 8121 : cluster [DBG] pgmap v9074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:52.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:54 smithi121 bash[36902]: cluster 2022-04-22T21:23:53.356812+0000 mgr.y (mgr.34107) 8122 : cluster [DBG] pgmap v9075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:54 smithi121 bash[36902]: audit 2022-04-22T21:23:54.039182+0000 mon.a (mon.0) 11773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:23:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:23:54.415Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:23:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:54 smithi115 bash[45495]: cluster 2022-04-22T21:23:53.356812+0000 mgr.y (mgr.34107) 8122 : cluster [DBG] pgmap v9075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:54 smithi115 bash[45495]: audit 2022-04-22T21:23:54.039182+0000 mon.a (mon.0) 11773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:54 smithi115 bash[46934]: cluster 2022-04-22T21:23:53.356812+0000 mgr.y (mgr.34107) 8122 : cluster [DBG] pgmap v9075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:54 smithi115 bash[46934]: audit 2022-04-22T21:23:54.039182+0000 mon.a (mon.0) 11773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:55 smithi121 bash[36902]: audit 2022-04-22T21:23:54.890217+0000 mon.a (mon.0) 11774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:55 smithi115 bash[45495]: audit 2022-04-22T21:23:54.890217+0000 mon.a (mon.0) 11774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:55 smithi115 bash[46934]: audit 2022-04-22T21:23:54.890217+0000 mon.a (mon.0) 11774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:23:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:56 smithi121 bash[36902]: cluster 2022-04-22T21:23:55.357853+0000 mgr.y (mgr.34107) 8123 : cluster [DBG] pgmap v9076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:56 smithi121 bash[36902]: audit 2022-04-22T21:23:56.036238+0000 mon.a (mon.0) 11775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:56 smithi115 bash[45495]: cluster 2022-04-22T21:23:55.357853+0000 mgr.y (mgr.34107) 8123 : cluster [DBG] pgmap v9076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:56 smithi115 bash[45495]: audit 2022-04-22T21:23:56.036238+0000 mon.a (mon.0) 11775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:56 smithi115 bash[46934]: cluster 2022-04-22T21:23:55.357853+0000 mgr.y (mgr.34107) 8123 : cluster [DBG] pgmap v9076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:56 smithi115 bash[46934]: audit 2022-04-22T21:23:56.036238+0000 mon.a (mon.0) 11775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:23:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:58 smithi121 bash[36902]: cluster 2022-04-22T21:23:57.359010+0000 mgr.y (mgr.34107) 8124 : cluster [DBG] pgmap v9077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:58 smithi115 bash[45495]: cluster 2022-04-22T21:23:57.359010+0000 mgr.y (mgr.34107) 8124 : cluster [DBG] pgmap v9077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:58 smithi115 bash[46934]: cluster 2022-04-22T21:23:57.359010+0000 mgr.y (mgr.34107) 8124 : cluster [DBG] pgmap v9077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:23:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:23:59 smithi121 bash[36902]: audit 2022-04-22T21:23:59.042320+0000 mon.a (mon.0) 11776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:23:59 smithi115 bash[45495]: audit 2022-04-22T21:23:59.042320+0000 mon.a (mon.0) 11776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:23:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:23:59 smithi115 bash[46934]: audit 2022-04-22T21:23:59.042320+0000 mon.a (mon.0) 11776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:00 smithi121 bash[36902]: cluster 2022-04-22T21:23:59.360165+0000 mgr.y (mgr.34107) 8125 : cluster [DBG] pgmap v9078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:00.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:00 smithi121 bash[36902]: audit 2022-04-22T21:23:59.893911+0000 mon.a (mon.0) 11777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:00 smithi115 bash[45495]: cluster 2022-04-22T21:23:59.360165+0000 mgr.y (mgr.34107) 8125 : cluster [DBG] pgmap v9078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:00 smithi115 bash[45495]: audit 2022-04-22T21:23:59.893911+0000 mon.a (mon.0) 11777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:00 smithi115 bash[46934]: cluster 2022-04-22T21:23:59.360165+0000 mgr.y (mgr.34107) 8125 : cluster [DBG] pgmap v9078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:00 smithi115 bash[46934]: audit 2022-04-22T21:23:59.893911+0000 mon.a (mon.0) 11777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:01.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:01 smithi121 bash[36902]: audit 2022-04-22T21:24:01.039523+0000 mon.a (mon.0) 11778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:01 smithi115 bash[45495]: audit 2022-04-22T21:24:01.039523+0000 mon.a (mon.0) 11778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:01 smithi115 bash[46934]: audit 2022-04-22T21:24:01.039523+0000 mon.a (mon.0) 11778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:02.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:02 smithi121 bash[36902]: cluster 2022-04-22T21:24:01.361286+0000 mgr.y (mgr.34107) 8126 : cluster [DBG] pgmap v9079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:02 smithi115 bash[46934]: cluster 2022-04-22T21:24:01.361286+0000 mgr.y (mgr.34107) 8126 : cluster [DBG] pgmap v9079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:02.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:02.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:02 smithi115 bash[45495]: cluster 2022-04-22T21:24:01.361286+0000 mgr.y (mgr.34107) 8126 : cluster [DBG] pgmap v9079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:04.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:04 smithi121 bash[36902]: cluster 2022-04-22T21:24:03.362398+0000 mgr.y (mgr.34107) 8127 : cluster [DBG] pgmap v9080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:04 smithi121 bash[36902]: audit 2022-04-22T21:24:04.045531+0000 mon.a (mon.0) 11779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:04.415Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:04 smithi115 bash[45495]: cluster 2022-04-22T21:24:03.362398+0000 mgr.y (mgr.34107) 8127 : cluster [DBG] pgmap v9080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:04 smithi115 bash[45495]: audit 2022-04-22T21:24:04.045531+0000 mon.a (mon.0) 11779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:04 smithi115 bash[46934]: cluster 2022-04-22T21:24:03.362398+0000 mgr.y (mgr.34107) 8127 : cluster [DBG] pgmap v9080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:04 smithi115 bash[46934]: audit 2022-04-22T21:24:04.045531+0000 mon.a (mon.0) 11779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:05.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:05 smithi121 bash[36902]: audit 2022-04-22T21:24:04.897685+0000 mon.a (mon.0) 11780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:05 smithi115 bash[45495]: audit 2022-04-22T21:24:04.897685+0000 mon.a (mon.0) 11780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:05 smithi115 bash[46934]: audit 2022-04-22T21:24:04.897685+0000 mon.a (mon.0) 11780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:06 smithi121 bash[36902]: cluster 2022-04-22T21:24:05.363530+0000 mgr.y (mgr.34107) 8128 : cluster [DBG] pgmap v9081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:06 smithi121 bash[36902]: audit 2022-04-22T21:24:06.042611+0000 mon.a (mon.0) 11781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:06 smithi115 bash[45495]: cluster 2022-04-22T21:24:05.363530+0000 mgr.y (mgr.34107) 8128 : cluster [DBG] pgmap v9081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:06 smithi115 bash[45495]: audit 2022-04-22T21:24:06.042611+0000 mon.a (mon.0) 11781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:06 smithi115 bash[46934]: cluster 2022-04-22T21:24:05.363530+0000 mgr.y (mgr.34107) 8128 : cluster [DBG] pgmap v9081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:06 smithi115 bash[46934]: audit 2022-04-22T21:24:06.042611+0000 mon.a (mon.0) 11781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:08 smithi121 bash[36902]: cluster 2022-04-22T21:24:07.364716+0000 mgr.y (mgr.34107) 8129 : cluster [DBG] pgmap v9082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:08 smithi115 bash[45495]: cluster 2022-04-22T21:24:07.364716+0000 mgr.y (mgr.34107) 8129 : cluster [DBG] pgmap v9082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:08.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:08 smithi115 bash[46934]: cluster 2022-04-22T21:24:07.364716+0000 mgr.y (mgr.34107) 8129 : cluster [DBG] pgmap v9082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:09 smithi121 bash[36902]: audit 2022-04-22T21:24:09.048594+0000 mon.a (mon.0) 11782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:09 smithi115 bash[45495]: audit 2022-04-22T21:24:09.048594+0000 mon.a (mon.0) 11782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:09 smithi115 bash[46934]: audit 2022-04-22T21:24:09.048594+0000 mon.a (mon.0) 11782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:10 smithi121 bash[36902]: cluster 2022-04-22T21:24:09.365795+0000 mgr.y (mgr.34107) 8130 : cluster [DBG] pgmap v9083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:10 smithi121 bash[36902]: audit 2022-04-22T21:24:09.901664+0000 mon.a (mon.0) 11783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:10 smithi115 bash[45495]: cluster 2022-04-22T21:24:09.365795+0000 mgr.y (mgr.34107) 8130 : cluster [DBG] pgmap v9083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:10 smithi115 bash[45495]: audit 2022-04-22T21:24:09.901664+0000 mon.a (mon.0) 11783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:10 smithi115 bash[46934]: cluster 2022-04-22T21:24:09.365795+0000 mgr.y (mgr.34107) 8130 : cluster [DBG] pgmap v9083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:10 smithi115 bash[46934]: audit 2022-04-22T21:24:09.901664+0000 mon.a (mon.0) 11783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:11 smithi121 bash[36902]: audit 2022-04-22T21:24:11.045496+0000 mon.a (mon.0) 11784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:11 smithi115 bash[45495]: audit 2022-04-22T21:24:11.045496+0000 mon.a (mon.0) 11784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:11 smithi115 bash[46934]: audit 2022-04-22T21:24:11.045496+0000 mon.a (mon.0) 11784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:12 smithi121 bash[36902]: cluster 2022-04-22T21:24:11.366989+0000 mgr.y (mgr.34107) 8131 : cluster [DBG] pgmap v9084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:12.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:12 smithi115 bash[45495]: cluster 2022-04-22T21:24:11.366989+0000 mgr.y (mgr.34107) 8131 : cluster [DBG] pgmap v9084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:12.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:12 smithi115 bash[46934]: cluster 2022-04-22T21:24:11.366989+0000 mgr.y (mgr.34107) 8131 : cluster [DBG] pgmap v9084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:12.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:12.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:14.714 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:24:14.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:14 smithi121 bash[36902]: cluster 2022-04-22T21:24:13.368221+0000 mgr.y (mgr.34107) 8132 : cluster [DBG] pgmap v9085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:14 smithi121 bash[36902]: audit 2022-04-22T21:24:14.051821+0000 mon.a (mon.0) 11785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:14.414Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:14 smithi115 bash[45495]: cluster 2022-04-22T21:24:13.368221+0000 mgr.y (mgr.34107) 8132 : cluster [DBG] pgmap v9085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:14 smithi115 bash[45495]: audit 2022-04-22T21:24:14.051821+0000 mon.a (mon.0) 11785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:14 smithi115 bash[46934]: cluster 2022-04-22T21:24:13.368221+0000 mgr.y (mgr.34107) 8132 : cluster [DBG] pgmap v9085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:14.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:14 smithi115 bash[46934]: audit 2022-04-22T21:24:14.051821+0000 mon.a (mon.0) 11785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:15.059 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 10m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 54s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 10m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 54s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 54s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 10m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 10m ago 4h 373M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 54s ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:24:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 10m ago 4h 243M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 10m ago 4h 10.9M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 54s ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 10m ago 4h 711M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 10m ago 4h 789M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 10m ago 4h 535M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 10m ago 4h 494M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 54s ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 54s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 54s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:24:15.061 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 54s ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:24:15.062 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 54s ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:24:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:24:15.464 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:24:15.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:15 smithi121 bash[36902]: audit 2022-04-22T21:24:14.696036+0000 mgr.y (mgr.34107) 8133 : audit [DBG] from='client.53743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:15 smithi121 bash[36902]: audit 2022-04-22T21:24:14.905180+0000 mon.a (mon.0) 11786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:15 smithi121 bash[36902]: audit 2022-04-22T21:24:15.459075+0000 mon.a (mon.0) 11787 : audit [DBG] from='client.? 172.21.15.115:0/1873230347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[45495]: audit 2022-04-22T21:24:14.696036+0000 mgr.y (mgr.34107) 8133 : audit [DBG] from='client.53743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[45495]: audit 2022-04-22T21:24:14.905180+0000 mon.a (mon.0) 11786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[45495]: audit 2022-04-22T21:24:15.459075+0000 mon.a (mon.0) 11787 : audit [DBG] from='client.? 172.21.15.115:0/1873230347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[46934]: audit 2022-04-22T21:24:14.696036+0000 mgr.y (mgr.34107) 8133 : audit [DBG] from='client.53743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[46934]: audit 2022-04-22T21:24:14.905180+0000 mon.a (mon.0) 11786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:15 smithi115 bash[46934]: audit 2022-04-22T21:24:15.459075+0000 mon.a (mon.0) 11787 : audit [DBG] from='client.? 172.21.15.115:0/1873230347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:16 smithi121 bash[36902]: audit 2022-04-22T21:24:15.036233+0000 mgr.y (mgr.34107) 8134 : audit [DBG] from='client.53749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:16 smithi121 bash[36902]: cluster 2022-04-22T21:24:15.369259+0000 mgr.y (mgr.34107) 8135 : cluster [DBG] pgmap v9086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:16.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:16 smithi121 bash[36902]: audit 2022-04-22T21:24:16.011031+0000 mon.a (mon.0) 11788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:24:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:16 smithi121 bash[36902]: audit 2022-04-22T21:24:16.011790+0000 mon.a (mon.0) 11789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:24:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:16 smithi121 bash[36902]: audit 2022-04-22T21:24:16.048590+0000 mon.a (mon.0) 11790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[45495]: audit 2022-04-22T21:24:15.036233+0000 mgr.y (mgr.34107) 8134 : audit [DBG] from='client.53749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[45495]: cluster 2022-04-22T21:24:15.369259+0000 mgr.y (mgr.34107) 8135 : cluster [DBG] pgmap v9086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[45495]: audit 2022-04-22T21:24:16.011031+0000 mon.a (mon.0) 11788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[45495]: audit 2022-04-22T21:24:16.011790+0000 mon.a (mon.0) 11789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[45495]: audit 2022-04-22T21:24:16.048590+0000 mon.a (mon.0) 11790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[46934]: audit 2022-04-22T21:24:15.036233+0000 mgr.y (mgr.34107) 8134 : audit [DBG] from='client.53749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[46934]: cluster 2022-04-22T21:24:15.369259+0000 mgr.y (mgr.34107) 8135 : cluster [DBG] pgmap v9086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[46934]: audit 2022-04-22T21:24:16.011031+0000 mon.a (mon.0) 11788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:24:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[46934]: audit 2022-04-22T21:24:16.011790+0000 mon.a (mon.0) 11789 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:24:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:16 smithi115 bash[46934]: audit 2022-04-22T21:24:16.048590+0000 mon.a (mon.0) 11790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:18.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:18 smithi121 bash[36902]: cluster 2022-04-22T21:24:17.370385+0000 mgr.y (mgr.34107) 8136 : cluster [DBG] pgmap v9087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:18 smithi115 bash[46934]: cluster 2022-04-22T21:24:17.370385+0000 mgr.y (mgr.34107) 8136 : cluster [DBG] pgmap v9087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:18 smithi115 bash[45495]: cluster 2022-04-22T21:24:17.370385+0000 mgr.y (mgr.34107) 8136 : cluster [DBG] pgmap v9087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:19 smithi121 bash[36902]: audit 2022-04-22T21:24:19.055214+0000 mon.a (mon.0) 11791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:19 smithi115 bash[45495]: audit 2022-04-22T21:24:19.055214+0000 mon.a (mon.0) 11791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:19 smithi115 bash[46934]: audit 2022-04-22T21:24:19.055214+0000 mon.a (mon.0) 11791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:20 smithi121 bash[36902]: cluster 2022-04-22T21:24:19.371451+0000 mgr.y (mgr.34107) 8137 : cluster [DBG] pgmap v9088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:20 smithi121 bash[36902]: audit 2022-04-22T21:24:19.908702+0000 mon.a (mon.0) 11792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:20 smithi115 bash[45495]: cluster 2022-04-22T21:24:19.371451+0000 mgr.y (mgr.34107) 8137 : cluster [DBG] pgmap v9088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:20 smithi115 bash[45495]: audit 2022-04-22T21:24:19.908702+0000 mon.a (mon.0) 11792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:20 smithi115 bash[46934]: cluster 2022-04-22T21:24:19.371451+0000 mgr.y (mgr.34107) 8137 : cluster [DBG] pgmap v9088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:20 smithi115 bash[46934]: audit 2022-04-22T21:24:19.908702+0000 mon.a (mon.0) 11792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:20.803170+0000 mon.a (mon.0) 11793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:24:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:20.806242+0000 mon.a (mon.0) 11794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:24:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:20.807973+0000 mon.a (mon.0) 11795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:24:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:21.051541+0000 mon.a (mon.0) 11796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:21.132617+0000 mon.a (mon.0) 11797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:21 smithi121 bash[36902]: audit 2022-04-22T21:24:21.143251+0000 mon.a (mon.0) 11798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:20.803170+0000 mon.a (mon.0) 11793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:20.806242+0000 mon.a (mon.0) 11794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:20.807973+0000 mon.a (mon.0) 11795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:21.051541+0000 mon.a (mon.0) 11796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:21.132617+0000 mon.a (mon.0) 11797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[45495]: audit 2022-04-22T21:24:21.143251+0000 mon.a (mon.0) 11798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:20.803170+0000 mon.a (mon.0) 11793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:20.806242+0000 mon.a (mon.0) 11794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:20.807973+0000 mon.a (mon.0) 11795 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:21.051541+0000 mon.a (mon.0) 11796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:21.132617+0000 mon.a (mon.0) 11797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:21 smithi115 bash[46934]: audit 2022-04-22T21:24:21.143251+0000 mon.a (mon.0) 11798 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:22 smithi115 bash[45495]: cluster 2022-04-22T21:24:21.372347+0000 mgr.y (mgr.34107) 8138 : cluster [DBG] pgmap v9089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:22.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:22.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:22.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:22 smithi115 bash[46934]: cluster 2022-04-22T21:24:21.372347+0000 mgr.y (mgr.34107) 8138 : cluster [DBG] pgmap v9089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:22 smithi121 bash[36902]: cluster 2022-04-22T21:24:21.372347+0000 mgr.y (mgr.34107) 8138 : cluster [DBG] pgmap v9089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:24.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:24 smithi115 bash[46934]: cluster 2022-04-22T21:24:23.373316+0000 mgr.y (mgr.34107) 8139 : cluster [DBG] pgmap v9090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:24 smithi115 bash[46934]: audit 2022-04-22T21:24:24.058517+0000 mon.a (mon.0) 11799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:24.413Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:24 smithi115 bash[45495]: cluster 2022-04-22T21:24:23.373316+0000 mgr.y (mgr.34107) 8139 : cluster [DBG] pgmap v9090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:24 smithi115 bash[45495]: audit 2022-04-22T21:24:24.058517+0000 mon.a (mon.0) 11799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:24 smithi121 bash[36902]: cluster 2022-04-22T21:24:23.373316+0000 mgr.y (mgr.34107) 8139 : cluster [DBG] pgmap v9090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:25.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:24 smithi121 bash[36902]: audit 2022-04-22T21:24:24.058517+0000 mon.a (mon.0) 11799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:25 smithi115 bash[45495]: audit 2022-04-22T21:24:24.912378+0000 mon.a (mon.0) 11800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:25.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:25 smithi115 bash[45495]: audit 2022-04-22T21:24:25.288022+0000 mon.a (mon.0) 11801 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:25 smithi115 bash[46934]: audit 2022-04-22T21:24:24.912378+0000 mon.a (mon.0) 11800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:25.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:25 smithi115 bash[46934]: audit 2022-04-22T21:24:25.288022+0000 mon.a (mon.0) 11801 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:25 smithi121 bash[36902]: audit 2022-04-22T21:24:24.912378+0000 mon.a (mon.0) 11800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:25 smithi121 bash[36902]: audit 2022-04-22T21:24:25.288022+0000 mon.a (mon.0) 11801 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:26 smithi121 bash[36902]: cluster 2022-04-22T21:24:25.374399+0000 mgr.y (mgr.34107) 8140 : cluster [DBG] pgmap v9091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:26 smithi121 bash[36902]: audit 2022-04-22T21:24:25.634253+0000 mon.a (mon.0) 11802 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:26 smithi121 bash[36902]: audit 2022-04-22T21:24:26.053800+0000 mon.a (mon.0) 11803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[46934]: cluster 2022-04-22T21:24:25.374399+0000 mgr.y (mgr.34107) 8140 : cluster [DBG] pgmap v9091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[46934]: audit 2022-04-22T21:24:25.634253+0000 mon.a (mon.0) 11802 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[46934]: audit 2022-04-22T21:24:26.053800+0000 mon.a (mon.0) 11803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[45495]: cluster 2022-04-22T21:24:25.374399+0000 mgr.y (mgr.34107) 8140 : cluster [DBG] pgmap v9091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[45495]: audit 2022-04-22T21:24:25.634253+0000 mon.a (mon.0) 11802 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:24:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:26 smithi115 bash[45495]: audit 2022-04-22T21:24:26.053800+0000 mon.a (mon.0) 11803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:28 smithi121 bash[36902]: cluster 2022-04-22T21:24:27.375561+0000 mgr.y (mgr.34107) 8141 : cluster [DBG] pgmap v9092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:28 smithi115 bash[45495]: cluster 2022-04-22T21:24:27.375561+0000 mgr.y (mgr.34107) 8141 : cluster [DBG] pgmap v9092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:28 smithi115 bash[46934]: cluster 2022-04-22T21:24:27.375561+0000 mgr.y (mgr.34107) 8141 : cluster [DBG] pgmap v9092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:29 smithi121 bash[36902]: audit 2022-04-22T21:24:29.061582+0000 mon.a (mon.0) 11804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:29 smithi115 bash[45495]: audit 2022-04-22T21:24:29.061582+0000 mon.a (mon.0) 11804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:29 smithi115 bash[46934]: audit 2022-04-22T21:24:29.061582+0000 mon.a (mon.0) 11804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:30 smithi121 bash[36902]: cluster 2022-04-22T21:24:29.376637+0000 mgr.y (mgr.34107) 8142 : cluster [DBG] pgmap v9093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:30 smithi121 bash[36902]: audit 2022-04-22T21:24:29.915735+0000 mon.a (mon.0) 11805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:30 smithi115 bash[45495]: cluster 2022-04-22T21:24:29.376637+0000 mgr.y (mgr.34107) 8142 : cluster [DBG] pgmap v9093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:30 smithi115 bash[45495]: audit 2022-04-22T21:24:29.915735+0000 mon.a (mon.0) 11805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:30 smithi115 bash[46934]: cluster 2022-04-22T21:24:29.376637+0000 mgr.y (mgr.34107) 8142 : cluster [DBG] pgmap v9093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:30 smithi115 bash[46934]: audit 2022-04-22T21:24:29.915735+0000 mon.a (mon.0) 11805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:32.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:31 smithi121 bash[36902]: audit 2022-04-22T21:24:31.056844+0000 mon.a (mon.0) 11806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:31 smithi115 bash[45495]: audit 2022-04-22T21:24:31.056844+0000 mon.a (mon.0) 11806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:31 smithi115 bash[46934]: audit 2022-04-22T21:24:31.056844+0000 mon.a (mon.0) 11806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:33.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:32 smithi121 bash[36902]: cluster 2022-04-22T21:24:31.377740+0000 mgr.y (mgr.34107) 8143 : cluster [DBG] pgmap v9094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:33.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:32 smithi115 bash[45495]: cluster 2022-04-22T21:24:31.377740+0000 mgr.y (mgr.34107) 8143 : cluster [DBG] pgmap v9094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:32.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:32 smithi115 bash[46934]: cluster 2022-04-22T21:24:31.377740+0000 mgr.y (mgr.34107) 8143 : cluster [DBG] pgmap v9094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:34 smithi115 bash[45495]: cluster 2022-04-22T21:24:33.378889+0000 mgr.y (mgr.34107) 8144 : cluster [DBG] pgmap v9095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:34 smithi115 bash[45495]: audit 2022-04-22T21:24:34.064607+0000 mon.a (mon.0) 11807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:34.415Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:34 smithi115 bash[46934]: cluster 2022-04-22T21:24:33.378889+0000 mgr.y (mgr.34107) 8144 : cluster [DBG] pgmap v9095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:34 smithi115 bash[46934]: audit 2022-04-22T21:24:34.064607+0000 mon.a (mon.0) 11807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:34 smithi121 bash[36902]: cluster 2022-04-22T21:24:33.378889+0000 mgr.y (mgr.34107) 8144 : cluster [DBG] pgmap v9095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:34 smithi121 bash[36902]: audit 2022-04-22T21:24:34.064607+0000 mon.a (mon.0) 11807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:35 smithi121 bash[36902]: audit 2022-04-22T21:24:34.919532+0000 mon.a (mon.0) 11808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:35 smithi115 bash[45495]: audit 2022-04-22T21:24:34.919532+0000 mon.a (mon.0) 11808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:35 smithi115 bash[46934]: audit 2022-04-22T21:24:34.919532+0000 mon.a (mon.0) 11808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:36 smithi121 bash[36902]: cluster 2022-04-22T21:24:35.380112+0000 mgr.y (mgr.34107) 8145 : cluster [DBG] pgmap v9096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:36 smithi121 bash[36902]: audit 2022-04-22T21:24:36.060017+0000 mon.a (mon.0) 11809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:36 smithi115 bash[45495]: cluster 2022-04-22T21:24:35.380112+0000 mgr.y (mgr.34107) 8145 : cluster [DBG] pgmap v9096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:36 smithi115 bash[45495]: audit 2022-04-22T21:24:36.060017+0000 mon.a (mon.0) 11809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:36 smithi115 bash[46934]: cluster 2022-04-22T21:24:35.380112+0000 mgr.y (mgr.34107) 8145 : cluster [DBG] pgmap v9096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:36 smithi115 bash[46934]: audit 2022-04-22T21:24:36.060017+0000 mon.a (mon.0) 11809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:38 smithi121 bash[36902]: cluster 2022-04-22T21:24:37.381200+0000 mgr.y (mgr.34107) 8146 : cluster [DBG] pgmap v9097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:38 smithi115 bash[45495]: cluster 2022-04-22T21:24:37.381200+0000 mgr.y (mgr.34107) 8146 : cluster [DBG] pgmap v9097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:38 smithi115 bash[46934]: cluster 2022-04-22T21:24:37.381200+0000 mgr.y (mgr.34107) 8146 : cluster [DBG] pgmap v9097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:39 smithi121 bash[36902]: audit 2022-04-22T21:24:39.067800+0000 mon.a (mon.0) 11810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:39 smithi115 bash[45495]: audit 2022-04-22T21:24:39.067800+0000 mon.a (mon.0) 11810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:39 smithi115 bash[46934]: audit 2022-04-22T21:24:39.067800+0000 mon.a (mon.0) 11810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:41.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:40 smithi121 bash[36902]: cluster 2022-04-22T21:24:39.382270+0000 mgr.y (mgr.34107) 8147 : cluster [DBG] pgmap v9098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:40 smithi121 bash[36902]: audit 2022-04-22T21:24:39.923032+0000 mon.a (mon.0) 11811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:40 smithi115 bash[45495]: cluster 2022-04-22T21:24:39.382270+0000 mgr.y (mgr.34107) 8147 : cluster [DBG] pgmap v9098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:40 smithi115 bash[45495]: audit 2022-04-22T21:24:39.923032+0000 mon.a (mon.0) 11811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:40 smithi115 bash[46934]: cluster 2022-04-22T21:24:39.382270+0000 mgr.y (mgr.34107) 8147 : cluster [DBG] pgmap v9098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:40 smithi115 bash[46934]: audit 2022-04-22T21:24:39.923032+0000 mon.a (mon.0) 11811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:42.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:41 smithi121 bash[36902]: audit 2022-04-22T21:24:41.063059+0000 mon.a (mon.0) 11812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:41 smithi115 bash[45495]: audit 2022-04-22T21:24:41.063059+0000 mon.a (mon.0) 11812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:41 smithi115 bash[46934]: audit 2022-04-22T21:24:41.063059+0000 mon.a (mon.0) 11812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:42 smithi121 bash[36902]: cluster 2022-04-22T21:24:41.383362+0000 mgr.y (mgr.34107) 8148 : cluster [DBG] pgmap v9099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:42 smithi115 bash[45495]: cluster 2022-04-22T21:24:41.383362+0000 mgr.y (mgr.34107) 8148 : cluster [DBG] pgmap v9099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:42 smithi115 bash[46934]: cluster 2022-04-22T21:24:41.383362+0000 mgr.y (mgr.34107) 8148 : cluster [DBG] pgmap v9099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:43.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:42.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:44.703 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:44.417Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:44 smithi121 bash[36902]: cluster 2022-04-22T21:24:43.384430+0000 mgr.y (mgr.34107) 8149 : cluster [DBG] pgmap v9100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:44 smithi121 bash[36902]: audit 2022-04-22T21:24:44.070729+0000 mon.a (mon.0) 11813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:44 smithi115 bash[45495]: cluster 2022-04-22T21:24:43.384430+0000 mgr.y (mgr.34107) 8149 : cluster [DBG] pgmap v9100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:44 smithi115 bash[45495]: audit 2022-04-22T21:24:44.070729+0000 mon.a (mon.0) 11813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:44 smithi115 bash[46934]: cluster 2022-04-22T21:24:43.384430+0000 mgr.y (mgr.34107) 8149 : cluster [DBG] pgmap v9100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:44 smithi115 bash[46934]: audit 2022-04-22T21:24:44.070729+0000 mon.a (mon.0) 11813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:45.836 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:24:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:45 smithi121 bash[36902]: audit 2022-04-22T21:24:44.926697+0000 mon.a (mon.0) 11814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:45 smithi115 bash[46934]: audit 2022-04-22T21:24:44.926697+0000 mon.a (mon.0) 11814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:45 smithi115 bash[45495]: audit 2022-04-22T21:24:44.926697+0000 mon.a (mon.0) 11814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 20s ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 85s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 20s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 85s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:24:46.188 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 85s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 20s ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 20s ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 85s ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 20s ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 20s ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 85s ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 20s ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 20s ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 20s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:24:46.189 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 20s ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:24:46.190 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 85s ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:24:46.190 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 85s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:24:46.190 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 85s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:24:46.190 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 85s ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:24:46.190 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 85s ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:24:46.603 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:24:46.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:24:46.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:24:46.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:24:46.605 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:24:46.605 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:24:46.605 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:24:46.605 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:24:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:46 smithi121 bash[36902]: cluster 2022-04-22T21:24:45.385517+0000 mgr.y (mgr.34107) 8150 : cluster [DBG] pgmap v9101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:46 smithi121 bash[36902]: audit 2022-04-22T21:24:45.817091+0000 mgr.y (mgr.34107) 8151 : audit [DBG] from='client.43830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:46 smithi121 bash[36902]: audit 2022-04-22T21:24:46.066183+0000 mon.a (mon.0) 11815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:46 smithi121 bash[36902]: audit 2022-04-22T21:24:46.599571+0000 mon.a (mon.0) 11816 : audit [DBG] from='client.? 172.21.15.115:0/924869024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[45495]: cluster 2022-04-22T21:24:45.385517+0000 mgr.y (mgr.34107) 8150 : cluster [DBG] pgmap v9101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[45495]: audit 2022-04-22T21:24:45.817091+0000 mgr.y (mgr.34107) 8151 : audit [DBG] from='client.43830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[45495]: audit 2022-04-22T21:24:46.066183+0000 mon.a (mon.0) 11815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[45495]: audit 2022-04-22T21:24:46.599571+0000 mon.a (mon.0) 11816 : audit [DBG] from='client.? 172.21.15.115:0/924869024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[46934]: cluster 2022-04-22T21:24:45.385517+0000 mgr.y (mgr.34107) 8150 : cluster [DBG] pgmap v9101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[46934]: audit 2022-04-22T21:24:45.817091+0000 mgr.y (mgr.34107) 8151 : audit [DBG] from='client.43830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[46934]: audit 2022-04-22T21:24:46.066183+0000 mon.a (mon.0) 11815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:46 smithi115 bash[46934]: audit 2022-04-22T21:24:46.599571+0000 mon.a (mon.0) 11816 : audit [DBG] from='client.? 172.21.15.115:0/924869024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:24:48.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:47 smithi121 bash[36902]: audit 2022-04-22T21:24:46.163393+0000 mgr.y (mgr.34107) 8152 : audit [DBG] from='client.53767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:47 smithi115 bash[45495]: audit 2022-04-22T21:24:46.163393+0000 mgr.y (mgr.34107) 8152 : audit [DBG] from='client.53767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:47 smithi115 bash[46934]: audit 2022-04-22T21:24:46.163393+0000 mgr.y (mgr.34107) 8152 : audit [DBG] from='client.53767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:48 smithi121 bash[36902]: cluster 2022-04-22T21:24:47.386634+0000 mgr.y (mgr.34107) 8153 : cluster [DBG] pgmap v9102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:48 smithi115 bash[45495]: cluster 2022-04-22T21:24:47.386634+0000 mgr.y (mgr.34107) 8153 : cluster [DBG] pgmap v9102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:48 smithi115 bash[46934]: cluster 2022-04-22T21:24:47.386634+0000 mgr.y (mgr.34107) 8153 : cluster [DBG] pgmap v9102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:49 smithi121 bash[36902]: audit 2022-04-22T21:24:49.073888+0000 mon.a (mon.0) 11817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:49 smithi115 bash[45495]: audit 2022-04-22T21:24:49.073888+0000 mon.a (mon.0) 11817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:49 smithi115 bash[46934]: audit 2022-04-22T21:24:49.073888+0000 mon.a (mon.0) 11817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:50 smithi121 bash[36902]: cluster 2022-04-22T21:24:49.387726+0000 mgr.y (mgr.34107) 8154 : cluster [DBG] pgmap v9103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:50 smithi121 bash[36902]: audit 2022-04-22T21:24:49.930325+0000 mon.a (mon.0) 11818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:50 smithi115 bash[45495]: cluster 2022-04-22T21:24:49.387726+0000 mgr.y (mgr.34107) 8154 : cluster [DBG] pgmap v9103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:50 smithi115 bash[45495]: audit 2022-04-22T21:24:49.930325+0000 mon.a (mon.0) 11818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:50 smithi115 bash[46934]: cluster 2022-04-22T21:24:49.387726+0000 mgr.y (mgr.34107) 8154 : cluster [DBG] pgmap v9103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:50 smithi115 bash[46934]: audit 2022-04-22T21:24:49.930325+0000 mon.a (mon.0) 11818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:51 smithi121 bash[36902]: audit 2022-04-22T21:24:51.069005+0000 mon.a (mon.0) 11819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:51 smithi115 bash[45495]: audit 2022-04-22T21:24:51.069005+0000 mon.a (mon.0) 11819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:51 smithi115 bash[46934]: audit 2022-04-22T21:24:51.069005+0000 mon.a (mon.0) 11819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:52 smithi121 bash[36902]: cluster 2022-04-22T21:24:51.388962+0000 mgr.y (mgr.34107) 8155 : cluster [DBG] pgmap v9104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:52 smithi115 bash[45495]: cluster 2022-04-22T21:24:51.388962+0000 mgr.y (mgr.34107) 8155 : cluster [DBG] pgmap v9104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:52.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:52 smithi115 bash[46934]: cluster 2022-04-22T21:24:51.388962+0000 mgr.y (mgr.34107) 8155 : cluster [DBG] pgmap v9104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:54.742 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:24:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:24:54.417Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:24:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:54 smithi121 bash[36902]: cluster 2022-04-22T21:24:53.390124+0000 mgr.y (mgr.34107) 8156 : cluster [DBG] pgmap v9105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:54 smithi121 bash[36902]: audit 2022-04-22T21:24:54.076964+0000 mon.a (mon.0) 11820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:54 smithi115 bash[46934]: cluster 2022-04-22T21:24:53.390124+0000 mgr.y (mgr.34107) 8156 : cluster [DBG] pgmap v9105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:54 smithi115 bash[46934]: audit 2022-04-22T21:24:54.076964+0000 mon.a (mon.0) 11820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:54 smithi115 bash[45495]: cluster 2022-04-22T21:24:53.390124+0000 mgr.y (mgr.34107) 8156 : cluster [DBG] pgmap v9105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:54 smithi115 bash[45495]: audit 2022-04-22T21:24:54.076964+0000 mon.a (mon.0) 11820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:24:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:55 smithi121 bash[36902]: audit 2022-04-22T21:24:54.934087+0000 mon.a (mon.0) 11821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:56.143 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:55 smithi115 bash[45495]: audit 2022-04-22T21:24:54.934087+0000 mon.a (mon.0) 11821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:56.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:55 smithi115 bash[46934]: audit 2022-04-22T21:24:54.934087+0000 mon.a (mon.0) 11821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:24:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:56 smithi121 bash[36902]: cluster 2022-04-22T21:24:55.391208+0000 mgr.y (mgr.34107) 8157 : cluster [DBG] pgmap v9106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:56 smithi121 bash[36902]: audit 2022-04-22T21:24:56.071939+0000 mon.a (mon.0) 11822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:56 smithi115 bash[45495]: cluster 2022-04-22T21:24:55.391208+0000 mgr.y (mgr.34107) 8157 : cluster [DBG] pgmap v9106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:56 smithi115 bash[45495]: audit 2022-04-22T21:24:56.071939+0000 mon.a (mon.0) 11822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:56 smithi115 bash[46934]: cluster 2022-04-22T21:24:55.391208+0000 mgr.y (mgr.34107) 8157 : cluster [DBG] pgmap v9106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:56 smithi115 bash[46934]: audit 2022-04-22T21:24:56.071939+0000 mon.a (mon.0) 11822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:24:59.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:58 smithi121 bash[36902]: cluster 2022-04-22T21:24:57.392367+0000 mgr.y (mgr.34107) 8158 : cluster [DBG] pgmap v9107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:58 smithi115 bash[45495]: cluster 2022-04-22T21:24:57.392367+0000 mgr.y (mgr.34107) 8158 : cluster [DBG] pgmap v9107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:24:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:58 smithi115 bash[46934]: cluster 2022-04-22T21:24:57.392367+0000 mgr.y (mgr.34107) 8158 : cluster [DBG] pgmap v9107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:24:59 smithi121 bash[36902]: audit 2022-04-22T21:24:59.080159+0000 mon.a (mon.0) 11823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:24:59 smithi115 bash[45495]: audit 2022-04-22T21:24:59.080159+0000 mon.a (mon.0) 11823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:24:59 smithi115 bash[46934]: audit 2022-04-22T21:24:59.080159+0000 mon.a (mon.0) 11823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:00 smithi121 bash[36902]: cluster 2022-04-22T21:24:59.393472+0000 mgr.y (mgr.34107) 8159 : cluster [DBG] pgmap v9108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:00 smithi121 bash[36902]: audit 2022-04-22T21:24:59.937707+0000 mon.a (mon.0) 11824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:00 smithi115 bash[45495]: cluster 2022-04-22T21:24:59.393472+0000 mgr.y (mgr.34107) 8159 : cluster [DBG] pgmap v9108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:00 smithi115 bash[45495]: audit 2022-04-22T21:24:59.937707+0000 mon.a (mon.0) 11824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:00 smithi115 bash[46934]: cluster 2022-04-22T21:24:59.393472+0000 mgr.y (mgr.34107) 8159 : cluster [DBG] pgmap v9108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:00 smithi115 bash[46934]: audit 2022-04-22T21:24:59.937707+0000 mon.a (mon.0) 11824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:01 smithi121 bash[36902]: audit 2022-04-22T21:25:01.074934+0000 mon.a (mon.0) 11825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:01 smithi115 bash[46934]: audit 2022-04-22T21:25:01.074934+0000 mon.a (mon.0) 11825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:01 smithi115 bash[45495]: audit 2022-04-22T21:25:01.074934+0000 mon.a (mon.0) 11825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:02 smithi115 bash[46934]: cluster 2022-04-22T21:25:01.394661+0000 mgr.y (mgr.34107) 8160 : cluster [DBG] pgmap v9109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:02.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:02 smithi115 bash[45495]: cluster 2022-04-22T21:25:01.394661+0000 mgr.y (mgr.34107) 8160 : cluster [DBG] pgmap v9109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:02 smithi121 bash[36902]: cluster 2022-04-22T21:25:01.394661+0000 mgr.y (mgr.34107) 8160 : cluster [DBG] pgmap v9109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:04.777 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:04.418Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:04 smithi115 bash[45495]: cluster 2022-04-22T21:25:03.395968+0000 mgr.y (mgr.34107) 8161 : cluster [DBG] pgmap v9110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:04 smithi115 bash[45495]: audit 2022-04-22T21:25:04.083171+0000 mon.a (mon.0) 11826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:04 smithi115 bash[46934]: cluster 2022-04-22T21:25:03.395968+0000 mgr.y (mgr.34107) 8161 : cluster [DBG] pgmap v9110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:04 smithi115 bash[46934]: audit 2022-04-22T21:25:04.083171+0000 mon.a (mon.0) 11826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:04 smithi121 bash[36902]: cluster 2022-04-22T21:25:03.395968+0000 mgr.y (mgr.34107) 8161 : cluster [DBG] pgmap v9110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:04 smithi121 bash[36902]: audit 2022-04-22T21:25:04.083171+0000 mon.a (mon.0) 11826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:05 smithi115 bash[45495]: audit 2022-04-22T21:25:04.941645+0000 mon.a (mon.0) 11827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:05 smithi115 bash[46934]: audit 2022-04-22T21:25:04.941645+0000 mon.a (mon.0) 11827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:05 smithi121 bash[36902]: audit 2022-04-22T21:25:04.941645+0000 mon.a (mon.0) 11827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:06 smithi115 bash[45495]: cluster 2022-04-22T21:25:05.397220+0000 mgr.y (mgr.34107) 8162 : cluster [DBG] pgmap v9111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:06 smithi115 bash[45495]: audit 2022-04-22T21:25:06.078173+0000 mon.a (mon.0) 11828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:06 smithi115 bash[46934]: cluster 2022-04-22T21:25:05.397220+0000 mgr.y (mgr.34107) 8162 : cluster [DBG] pgmap v9111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:06 smithi115 bash[46934]: audit 2022-04-22T21:25:06.078173+0000 mon.a (mon.0) 11828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:07.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:06 smithi121 bash[36902]: cluster 2022-04-22T21:25:05.397220+0000 mgr.y (mgr.34107) 8162 : cluster [DBG] pgmap v9111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:06 smithi121 bash[36902]: audit 2022-04-22T21:25:06.078173+0000 mon.a (mon.0) 11828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:08 smithi115 bash[46934]: cluster 2022-04-22T21:25:07.398376+0000 mgr.y (mgr.34107) 8163 : cluster [DBG] pgmap v9112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:08 smithi115 bash[45495]: cluster 2022-04-22T21:25:07.398376+0000 mgr.y (mgr.34107) 8163 : cluster [DBG] pgmap v9112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:08 smithi121 bash[36902]: cluster 2022-04-22T21:25:07.398376+0000 mgr.y (mgr.34107) 8163 : cluster [DBG] pgmap v9112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:10.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:09 smithi121 bash[36902]: audit 2022-04-22T21:25:09.086396+0000 mon.a (mon.0) 11829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:09 smithi115 bash[45495]: audit 2022-04-22T21:25:09.086396+0000 mon.a (mon.0) 11829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:09 smithi115 bash[46934]: audit 2022-04-22T21:25:09.086396+0000 mon.a (mon.0) 11829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:10 smithi121 bash[36902]: cluster 2022-04-22T21:25:09.399427+0000 mgr.y (mgr.34107) 8164 : cluster [DBG] pgmap v9113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:10 smithi121 bash[36902]: audit 2022-04-22T21:25:09.944943+0000 mon.a (mon.0) 11830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:10 smithi115 bash[45495]: cluster 2022-04-22T21:25:09.399427+0000 mgr.y (mgr.34107) 8164 : cluster [DBG] pgmap v9113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:11.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:10 smithi115 bash[45495]: audit 2022-04-22T21:25:09.944943+0000 mon.a (mon.0) 11830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:10 smithi115 bash[46934]: cluster 2022-04-22T21:25:09.399427+0000 mgr.y (mgr.34107) 8164 : cluster [DBG] pgmap v9113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:11.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:10 smithi115 bash[46934]: audit 2022-04-22T21:25:09.944943+0000 mon.a (mon.0) 11830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:11 smithi121 bash[36902]: audit 2022-04-22T21:25:11.081105+0000 mon.a (mon.0) 11831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:11 smithi115 bash[46934]: audit 2022-04-22T21:25:11.081105+0000 mon.a (mon.0) 11831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:11 smithi115 bash[45495]: audit 2022-04-22T21:25:11.081105+0000 mon.a (mon.0) 11831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:12.814 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:12.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:12 smithi121 bash[36902]: cluster 2022-04-22T21:25:11.400581+0000 mgr.y (mgr.34107) 8165 : cluster [DBG] pgmap v9114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:12 smithi115 bash[45495]: cluster 2022-04-22T21:25:11.400581+0000 mgr.y (mgr.34107) 8165 : cluster [DBG] pgmap v9114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:12 smithi115 bash[46934]: cluster 2022-04-22T21:25:11.400581+0000 mgr.y (mgr.34107) 8165 : cluster [DBG] pgmap v9114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:14.419Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:14 smithi121 bash[36902]: cluster 2022-04-22T21:25:13.401726+0000 mgr.y (mgr.34107) 8166 : cluster [DBG] pgmap v9115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:14 smithi121 bash[36902]: audit 2022-04-22T21:25:14.089543+0000 mon.a (mon.0) 11832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:14 smithi115 bash[45495]: cluster 2022-04-22T21:25:13.401726+0000 mgr.y (mgr.34107) 8166 : cluster [DBG] pgmap v9115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:14 smithi115 bash[45495]: audit 2022-04-22T21:25:14.089543+0000 mon.a (mon.0) 11832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:14 smithi115 bash[46934]: cluster 2022-04-22T21:25:13.401726+0000 mgr.y (mgr.34107) 8166 : cluster [DBG] pgmap v9115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:14 smithi115 bash[46934]: audit 2022-04-22T21:25:14.089543+0000 mon.a (mon.0) 11832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:15 smithi121 bash[36902]: audit 2022-04-22T21:25:14.948784+0000 mon.a (mon.0) 11833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:15 smithi115 bash[45495]: audit 2022-04-22T21:25:14.948784+0000 mon.a (mon.0) 11833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:15 smithi115 bash[46934]: audit 2022-04-22T21:25:14.948784+0000 mon.a (mon.0) 11833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:16.984 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:25:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:16 smithi121 bash[36902]: cluster 2022-04-22T21:25:15.402770+0000 mgr.y (mgr.34107) 8167 : cluster [DBG] pgmap v9116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:16 smithi121 bash[36902]: audit 2022-04-22T21:25:16.018458+0000 mon.a (mon.0) 11834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:25:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:16 smithi121 bash[36902]: audit 2022-04-22T21:25:16.019124+0000 mon.a (mon.0) 11835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:25:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:16 smithi121 bash[36902]: audit 2022-04-22T21:25:16.084191+0000 mon.a (mon.0) 11836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[46934]: cluster 2022-04-22T21:25:15.402770+0000 mgr.y (mgr.34107) 8167 : cluster [DBG] pgmap v9116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[46934]: audit 2022-04-22T21:25:16.018458+0000 mon.a (mon.0) 11834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[46934]: audit 2022-04-22T21:25:16.019124+0000 mon.a (mon.0) 11835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[46934]: audit 2022-04-22T21:25:16.084191+0000 mon.a (mon.0) 11836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[45495]: cluster 2022-04-22T21:25:15.402770+0000 mgr.y (mgr.34107) 8167 : cluster [DBG] pgmap v9116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[45495]: audit 2022-04-22T21:25:16.018458+0000 mon.a (mon.0) 11834 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:25:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[45495]: audit 2022-04-22T21:25:16.019124+0000 mon.a (mon.0) 11835 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:25:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:16 smithi115 bash[45495]: audit 2022-04-22T21:25:16.084191+0000 mon.a (mon.0) 11836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:17.319 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:25:17.319 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 52s ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:25:17.319 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 116s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:25:17.319 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 52s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 116s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 116s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 52s ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 52s ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 116s ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 52s ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 52s ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 116s ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:25:17.320 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 52s ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 52s ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 52s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 52s ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 116s ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 116s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 116s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 116s ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:25:17.321 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 116s ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:25:17.728 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:25:17.729 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:25:17.730 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:25:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:17 smithi121 bash[36902]: audit 2022-04-22T21:25:17.724421+0000 mon.a (mon.0) 11837 : audit [DBG] from='client.? 172.21.15.115:0/1550096975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:17 smithi115 bash[45495]: audit 2022-04-22T21:25:17.724421+0000 mon.a (mon.0) 11837 : audit [DBG] from='client.? 172.21.15.115:0/1550096975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:17 smithi115 bash[46934]: audit 2022-04-22T21:25:17.724421+0000 mon.a (mon.0) 11837 : audit [DBG] from='client.? 172.21.15.115:0/1550096975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:19.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:18 smithi121 bash[36902]: audit 2022-04-22T21:25:16.965615+0000 mgr.y (mgr.34107) 8168 : audit [DBG] from='client.53779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:18 smithi121 bash[36902]: audit 2022-04-22T21:25:17.296217+0000 mgr.y (mgr.34107) 8169 : audit [DBG] from='client.53785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:18 smithi121 bash[36902]: cluster 2022-04-22T21:25:17.403961+0000 mgr.y (mgr.34107) 8170 : cluster [DBG] pgmap v9117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[45495]: audit 2022-04-22T21:25:16.965615+0000 mgr.y (mgr.34107) 8168 : audit [DBG] from='client.53779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[45495]: audit 2022-04-22T21:25:17.296217+0000 mgr.y (mgr.34107) 8169 : audit [DBG] from='client.53785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[45495]: cluster 2022-04-22T21:25:17.403961+0000 mgr.y (mgr.34107) 8170 : cluster [DBG] pgmap v9117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[46934]: audit 2022-04-22T21:25:16.965615+0000 mgr.y (mgr.34107) 8168 : audit [DBG] from='client.53779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[46934]: audit 2022-04-22T21:25:17.296217+0000 mgr.y (mgr.34107) 8169 : audit [DBG] from='client.53785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:18 smithi115 bash[46934]: cluster 2022-04-22T21:25:17.403961+0000 mgr.y (mgr.34107) 8170 : cluster [DBG] pgmap v9117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:19 smithi121 bash[36902]: audit 2022-04-22T21:25:19.092826+0000 mon.a (mon.0) 11838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:19 smithi115 bash[45495]: audit 2022-04-22T21:25:19.092826+0000 mon.a (mon.0) 11838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:20.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:19 smithi115 bash[46934]: audit 2022-04-22T21:25:19.092826+0000 mon.a (mon.0) 11838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:20 smithi121 bash[36902]: cluster 2022-04-22T21:25:19.405045+0000 mgr.y (mgr.34107) 8171 : cluster [DBG] pgmap v9118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:20 smithi121 bash[36902]: audit 2022-04-22T21:25:19.952353+0000 mon.a (mon.0) 11839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:20 smithi115 bash[45495]: cluster 2022-04-22T21:25:19.405045+0000 mgr.y (mgr.34107) 8171 : cluster [DBG] pgmap v9118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:20 smithi115 bash[45495]: audit 2022-04-22T21:25:19.952353+0000 mon.a (mon.0) 11839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:20 smithi115 bash[46934]: cluster 2022-04-22T21:25:19.405045+0000 mgr.y (mgr.34107) 8171 : cluster [DBG] pgmap v9118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:20 smithi115 bash[46934]: audit 2022-04-22T21:25:19.952353+0000 mon.a (mon.0) 11839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:21 smithi121 bash[36902]: audit 2022-04-22T21:25:21.087015+0000 mon.a (mon.0) 11840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:21 smithi115 bash[45495]: audit 2022-04-22T21:25:21.087015+0000 mon.a (mon.0) 11840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:21 smithi115 bash[46934]: audit 2022-04-22T21:25:21.087015+0000 mon.a (mon.0) 11840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:22.848 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:22.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:22 smithi121 bash[36902]: cluster 2022-04-22T21:25:21.406119+0000 mgr.y (mgr.34107) 8172 : cluster [DBG] pgmap v9119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:23.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:22 smithi115 bash[45495]: cluster 2022-04-22T21:25:21.406119+0000 mgr.y (mgr.34107) 8172 : cluster [DBG] pgmap v9119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:22 smithi115 bash[46934]: cluster 2022-04-22T21:25:21.406119+0000 mgr.y (mgr.34107) 8172 : cluster [DBG] pgmap v9119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:24.418Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:24 smithi121 bash[36902]: cluster 2022-04-22T21:25:23.407177+0000 mgr.y (mgr.34107) 8173 : cluster [DBG] pgmap v9120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:24 smithi121 bash[36902]: audit 2022-04-22T21:25:24.096044+0000 mon.a (mon.0) 11841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:24 smithi115 bash[45495]: cluster 2022-04-22T21:25:23.407177+0000 mgr.y (mgr.34107) 8173 : cluster [DBG] pgmap v9120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:24 smithi115 bash[45495]: audit 2022-04-22T21:25:24.096044+0000 mon.a (mon.0) 11841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:24 smithi115 bash[46934]: cluster 2022-04-22T21:25:23.407177+0000 mgr.y (mgr.34107) 8173 : cluster [DBG] pgmap v9120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:24 smithi115 bash[46934]: audit 2022-04-22T21:25:24.096044+0000 mon.a (mon.0) 11841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:25.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[45495]: audit 2022-04-22T21:25:24.956127+0000 mon.a (mon.0) 11842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:25.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[45495]: audit 2022-04-22T21:25:25.641160+0000 mon.a (mon.0) 11843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:25:25.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[45495]: audit 2022-04-22T21:25:25.643190+0000 mon.a (mon.0) 11844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:25:25.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[45495]: audit 2022-04-22T21:25:25.644670+0000 mon.a (mon.0) 11845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:25:25.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[46934]: audit 2022-04-22T21:25:24.956127+0000 mon.a (mon.0) 11842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:25.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[46934]: audit 2022-04-22T21:25:25.641160+0000 mon.a (mon.0) 11843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:25:25.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[46934]: audit 2022-04-22T21:25:25.643190+0000 mon.a (mon.0) 11844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:25:25.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:25 smithi115 bash[46934]: audit 2022-04-22T21:25:25.644670+0000 mon.a (mon.0) 11845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:25:26.005 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:25 smithi121 bash[36902]: audit 2022-04-22T21:25:24.956127+0000 mon.a (mon.0) 11842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:26.005 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:25 smithi121 bash[36902]: audit 2022-04-22T21:25:25.641160+0000 mon.a (mon.0) 11843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:25:26.005 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:25 smithi121 bash[36902]: audit 2022-04-22T21:25:25.643190+0000 mon.a (mon.0) 11844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:25:26.006 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:25 smithi121 bash[36902]: audit 2022-04-22T21:25:25.644670+0000 mon.a (mon.0) 11845 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:25:27.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:26 smithi121 bash[36902]: cluster 2022-04-22T21:25:25.408212+0000 mgr.y (mgr.34107) 8174 : cluster [DBG] pgmap v9121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:26 smithi121 bash[36902]: audit 2022-04-22T21:25:26.013493+0000 mon.a (mon.0) 11846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:25:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:26 smithi121 bash[36902]: audit 2022-04-22T21:25:26.090183+0000 mon.a (mon.0) 11847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[45495]: cluster 2022-04-22T21:25:25.408212+0000 mgr.y (mgr.34107) 8174 : cluster [DBG] pgmap v9121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[45495]: audit 2022-04-22T21:25:26.013493+0000 mon.a (mon.0) 11846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[45495]: audit 2022-04-22T21:25:26.090183+0000 mon.a (mon.0) 11847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[46934]: cluster 2022-04-22T21:25:25.408212+0000 mgr.y (mgr.34107) 8174 : cluster [DBG] pgmap v9121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[46934]: audit 2022-04-22T21:25:26.013493+0000 mon.a (mon.0) 11846 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:25:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:26 smithi115 bash[46934]: audit 2022-04-22T21:25:26.090183+0000 mon.a (mon.0) 11847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:28 smithi121 bash[36902]: cluster 2022-04-22T21:25:27.409315+0000 mgr.y (mgr.34107) 8175 : cluster [DBG] pgmap v9122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:28 smithi115 bash[45495]: cluster 2022-04-22T21:25:27.409315+0000 mgr.y (mgr.34107) 8175 : cluster [DBG] pgmap v9122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:29.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:28 smithi115 bash[46934]: cluster 2022-04-22T21:25:27.409315+0000 mgr.y (mgr.34107) 8175 : cluster [DBG] pgmap v9122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:29 smithi121 bash[36902]: audit 2022-04-22T21:25:29.099249+0000 mon.a (mon.0) 11848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:29 smithi115 bash[45495]: audit 2022-04-22T21:25:29.099249+0000 mon.a (mon.0) 11848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:29 smithi115 bash[46934]: audit 2022-04-22T21:25:29.099249+0000 mon.a (mon.0) 11848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:30 smithi121 bash[36902]: cluster 2022-04-22T21:25:29.410477+0000 mgr.y (mgr.34107) 8176 : cluster [DBG] pgmap v9123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:30 smithi121 bash[36902]: audit 2022-04-22T21:25:29.959704+0000 mon.a (mon.0) 11849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:30 smithi115 bash[45495]: cluster 2022-04-22T21:25:29.410477+0000 mgr.y (mgr.34107) 8176 : cluster [DBG] pgmap v9123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:30 smithi115 bash[45495]: audit 2022-04-22T21:25:29.959704+0000 mon.a (mon.0) 11849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:30 smithi115 bash[46934]: cluster 2022-04-22T21:25:29.410477+0000 mgr.y (mgr.34107) 8176 : cluster [DBG] pgmap v9123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:30 smithi115 bash[46934]: audit 2022-04-22T21:25:29.959704+0000 mon.a (mon.0) 11849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:31 smithi121 bash[36902]: audit 2022-04-22T21:25:31.093189+0000 mon.a (mon.0) 11850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:31 smithi115 bash[46934]: audit 2022-04-22T21:25:31.093189+0000 mon.a (mon.0) 11850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:31 smithi115 bash[45495]: audit 2022-04-22T21:25:31.093189+0000 mon.a (mon.0) 11850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:32.882 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:32.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:32 smithi121 bash[36902]: cluster 2022-04-22T21:25:31.411729+0000 mgr.y (mgr.34107) 8177 : cluster [DBG] pgmap v9124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:32 smithi115 bash[45495]: cluster 2022-04-22T21:25:31.411729+0000 mgr.y (mgr.34107) 8177 : cluster [DBG] pgmap v9124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:32 smithi115 bash[46934]: cluster 2022-04-22T21:25:31.411729+0000 mgr.y (mgr.34107) 8177 : cluster [DBG] pgmap v9124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:34.418Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:34 smithi121 bash[36902]: cluster 2022-04-22T21:25:33.412832+0000 mgr.y (mgr.34107) 8178 : cluster [DBG] pgmap v9125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:34 smithi121 bash[36902]: audit 2022-04-22T21:25:34.102388+0000 mon.a (mon.0) 11851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:34 smithi115 bash[45495]: cluster 2022-04-22T21:25:33.412832+0000 mgr.y (mgr.34107) 8178 : cluster [DBG] pgmap v9125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:34 smithi115 bash[45495]: audit 2022-04-22T21:25:34.102388+0000 mon.a (mon.0) 11851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:34 smithi115 bash[46934]: cluster 2022-04-22T21:25:33.412832+0000 mgr.y (mgr.34107) 8178 : cluster [DBG] pgmap v9125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:34 smithi115 bash[46934]: audit 2022-04-22T21:25:34.102388+0000 mon.a (mon.0) 11851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:35 smithi121 bash[36902]: audit 2022-04-22T21:25:34.963306+0000 mon.a (mon.0) 11852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:35 smithi115 bash[45495]: audit 2022-04-22T21:25:34.963306+0000 mon.a (mon.0) 11852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:35 smithi115 bash[46934]: audit 2022-04-22T21:25:34.963306+0000 mon.a (mon.0) 11852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:36 smithi121 bash[36902]: cluster 2022-04-22T21:25:35.413909+0000 mgr.y (mgr.34107) 8179 : cluster [DBG] pgmap v9126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:36 smithi121 bash[36902]: audit 2022-04-22T21:25:36.096306+0000 mon.a (mon.0) 11853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:36 smithi115 bash[45495]: cluster 2022-04-22T21:25:35.413909+0000 mgr.y (mgr.34107) 8179 : cluster [DBG] pgmap v9126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:36 smithi115 bash[45495]: audit 2022-04-22T21:25:36.096306+0000 mon.a (mon.0) 11853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:36 smithi115 bash[46934]: cluster 2022-04-22T21:25:35.413909+0000 mgr.y (mgr.34107) 8179 : cluster [DBG] pgmap v9126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:36 smithi115 bash[46934]: audit 2022-04-22T21:25:36.096306+0000 mon.a (mon.0) 11853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:38 smithi121 bash[36902]: cluster 2022-04-22T21:25:37.415024+0000 mgr.y (mgr.34107) 8180 : cluster [DBG] pgmap v9127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:38 smithi115 bash[45495]: cluster 2022-04-22T21:25:37.415024+0000 mgr.y (mgr.34107) 8180 : cluster [DBG] pgmap v9127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:38 smithi115 bash[46934]: cluster 2022-04-22T21:25:37.415024+0000 mgr.y (mgr.34107) 8180 : cluster [DBG] pgmap v9127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:39 smithi121 bash[36902]: audit 2022-04-22T21:25:39.105525+0000 mon.a (mon.0) 11854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:39 smithi115 bash[45495]: audit 2022-04-22T21:25:39.105525+0000 mon.a (mon.0) 11854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:39 smithi115 bash[46934]: audit 2022-04-22T21:25:39.105525+0000 mon.a (mon.0) 11854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:40 smithi121 bash[36902]: cluster 2022-04-22T21:25:39.416116+0000 mgr.y (mgr.34107) 8181 : cluster [DBG] pgmap v9128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:40 smithi121 bash[36902]: audit 2022-04-22T21:25:39.967220+0000 mon.a (mon.0) 11855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:40 smithi115 bash[45495]: cluster 2022-04-22T21:25:39.416116+0000 mgr.y (mgr.34107) 8181 : cluster [DBG] pgmap v9128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:40 smithi115 bash[45495]: audit 2022-04-22T21:25:39.967220+0000 mon.a (mon.0) 11855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:40 smithi115 bash[46934]: cluster 2022-04-22T21:25:39.416116+0000 mgr.y (mgr.34107) 8181 : cluster [DBG] pgmap v9128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:40 smithi115 bash[46934]: audit 2022-04-22T21:25:39.967220+0000 mon.a (mon.0) 11855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:41 smithi121 bash[36902]: audit 2022-04-22T21:25:41.099096+0000 mon.a (mon.0) 11856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:41 smithi115 bash[45495]: audit 2022-04-22T21:25:41.099096+0000 mon.a (mon.0) 11856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:41 smithi115 bash[46934]: audit 2022-04-22T21:25:41.099096+0000 mon.a (mon.0) 11856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:42.914 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:42.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:42 smithi121 bash[36902]: cluster 2022-04-22T21:25:41.417180+0000 mgr.y (mgr.34107) 8182 : cluster [DBG] pgmap v9129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:42 smithi115 bash[45495]: cluster 2022-04-22T21:25:41.417180+0000 mgr.y (mgr.34107) 8182 : cluster [DBG] pgmap v9129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:42 smithi115 bash[46934]: cluster 2022-04-22T21:25:41.417180+0000 mgr.y (mgr.34107) 8182 : cluster [DBG] pgmap v9129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:44.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:43 smithi121 bash[36902]: cluster 2022-04-22T21:25:43.418266+0000 mgr.y (mgr.34107) 8183 : cluster [DBG] pgmap v9130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:43 smithi115 bash[45495]: cluster 2022-04-22T21:25:43.418266+0000 mgr.y (mgr.34107) 8183 : cluster [DBG] pgmap v9130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:43 smithi115 bash[46934]: cluster 2022-04-22T21:25:43.418266+0000 mgr.y (mgr.34107) 8183 : cluster [DBG] pgmap v9130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:44.420Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:44 smithi121 bash[36902]: audit 2022-04-22T21:25:44.108745+0000 mon.a (mon.0) 11857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:44 smithi115 bash[45495]: audit 2022-04-22T21:25:44.108745+0000 mon.a (mon.0) 11857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:44 smithi115 bash[46934]: audit 2022-04-22T21:25:44.108745+0000 mon.a (mon.0) 11857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:45 smithi121 bash[36902]: audit 2022-04-22T21:25:44.970727+0000 mon.a (mon.0) 11858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:45 smithi121 bash[36902]: cluster 2022-04-22T21:25:45.419333+0000 mgr.y (mgr.34107) 8184 : cluster [DBG] pgmap v9131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:45 smithi115 bash[45495]: audit 2022-04-22T21:25:44.970727+0000 mon.a (mon.0) 11858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:45 smithi115 bash[45495]: cluster 2022-04-22T21:25:45.419333+0000 mgr.y (mgr.34107) 8184 : cluster [DBG] pgmap v9131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:45 smithi115 bash[46934]: audit 2022-04-22T21:25:44.970727+0000 mon.a (mon.0) 11858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:45 smithi115 bash[46934]: cluster 2022-04-22T21:25:45.419333+0000 mgr.y (mgr.34107) 8184 : cluster [DBG] pgmap v9131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:46 smithi121 bash[36902]: audit 2022-04-22T21:25:46.102367+0000 mon.a (mon.0) 11859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:46 smithi115 bash[45495]: audit 2022-04-22T21:25:46.102367+0000 mon.a (mon.0) 11859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:46 smithi115 bash[46934]: audit 2022-04-22T21:25:46.102367+0000 mon.a (mon.0) 11859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:48.106 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:25:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:47 smithi121 bash[36902]: cluster 2022-04-22T21:25:47.420352+0000 mgr.y (mgr.34107) 8185 : cluster [DBG] pgmap v9132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:47 smithi115 bash[45495]: cluster 2022-04-22T21:25:47.420352+0000 mgr.y (mgr.34107) 8185 : cluster [DBG] pgmap v9132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:47 smithi115 bash[46934]: cluster 2022-04-22T21:25:47.420352+0000 mgr.y (mgr.34107) 8185 : cluster [DBG] pgmap v9132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 83s ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 83s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:25:48.449 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 83s ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 83s ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 83s ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 83s ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 83s ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 83s ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:25:48.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 83s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 83s ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:25:48.451 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:25:48.852 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:25:48.853 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:25:48.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:25:48.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:25:48.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:25:48.855 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:25:48.855 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:25:48.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:25:48.855 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:25:49.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:48 smithi121 bash[36902]: audit 2022-04-22T21:25:48.086951+0000 mgr.y (mgr.34107) 8186 : audit [DBG] from='client.53797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:48 smithi121 bash[36902]: audit 2022-04-22T21:25:48.425213+0000 mgr.y (mgr.34107) 8187 : audit [DBG] from='client.43872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:48 smithi121 bash[36902]: audit 2022-04-22T21:25:48.848952+0000 mon.c (mon.1) 294 : audit [DBG] from='client.? 172.21.15.115:0/3687100053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[45495]: audit 2022-04-22T21:25:48.086951+0000 mgr.y (mgr.34107) 8186 : audit [DBG] from='client.53797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[45495]: audit 2022-04-22T21:25:48.425213+0000 mgr.y (mgr.34107) 8187 : audit [DBG] from='client.43872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[45495]: audit 2022-04-22T21:25:48.848952+0000 mon.c (mon.1) 294 : audit [DBG] from='client.? 172.21.15.115:0/3687100053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[46934]: audit 2022-04-22T21:25:48.086951+0000 mgr.y (mgr.34107) 8186 : audit [DBG] from='client.53797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[46934]: audit 2022-04-22T21:25:48.425213+0000 mgr.y (mgr.34107) 8187 : audit [DBG] from='client.43872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:25:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:48 smithi115 bash[46934]: audit 2022-04-22T21:25:48.848952+0000 mon.c (mon.1) 294 : audit [DBG] from='client.? 172.21.15.115:0/3687100053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:25:50.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:49 smithi121 bash[36902]: audit 2022-04-22T21:25:49.111885+0000 mon.a (mon.0) 11860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:49 smithi121 bash[36902]: cluster 2022-04-22T21:25:49.421359+0000 mgr.y (mgr.34107) 8188 : cluster [DBG] pgmap v9133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:49 smithi115 bash[45495]: audit 2022-04-22T21:25:49.111885+0000 mon.a (mon.0) 11860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:50.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:49 smithi115 bash[45495]: cluster 2022-04-22T21:25:49.421359+0000 mgr.y (mgr.34107) 8188 : cluster [DBG] pgmap v9133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:50.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:49 smithi115 bash[46934]: audit 2022-04-22T21:25:49.111885+0000 mon.a (mon.0) 11860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:50.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:49 smithi115 bash[46934]: cluster 2022-04-22T21:25:49.421359+0000 mgr.y (mgr.34107) 8188 : cluster [DBG] pgmap v9133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:50 smithi121 bash[36902]: audit 2022-04-22T21:25:49.974407+0000 mon.a (mon.0) 11861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:50 smithi115 bash[45495]: audit 2022-04-22T21:25:49.974407+0000 mon.a (mon.0) 11861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:50 smithi115 bash[46934]: audit 2022-04-22T21:25:49.974407+0000 mon.a (mon.0) 11861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:51 smithi121 bash[36902]: audit 2022-04-22T21:25:51.105496+0000 mon.a (mon.0) 11862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:51 smithi121 bash[36902]: cluster 2022-04-22T21:25:51.422395+0000 mgr.y (mgr.34107) 8189 : cluster [DBG] pgmap v9134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:51 smithi115 bash[45495]: audit 2022-04-22T21:25:51.105496+0000 mon.a (mon.0) 11862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:51 smithi115 bash[45495]: cluster 2022-04-22T21:25:51.422395+0000 mgr.y (mgr.34107) 8189 : cluster [DBG] pgmap v9134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:51 smithi115 bash[46934]: audit 2022-04-22T21:25:51.105496+0000 mon.a (mon.0) 11862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:51 smithi115 bash[46934]: cluster 2022-04-22T21:25:51.422395+0000 mgr.y (mgr.34107) 8189 : cluster [DBG] pgmap v9134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:52.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:54.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:54 smithi121 bash[36902]: cluster 2022-04-22T21:25:53.423521+0000 mgr.y (mgr.34107) 8190 : cluster [DBG] pgmap v9135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:54 smithi121 bash[36902]: audit 2022-04-22T21:25:54.114908+0000 mon.a (mon.0) 11863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:25:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:25:54.421Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:25:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:54 smithi115 bash[46934]: cluster 2022-04-22T21:25:53.423521+0000 mgr.y (mgr.34107) 8190 : cluster [DBG] pgmap v9135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:54 smithi115 bash[46934]: audit 2022-04-22T21:25:54.114908+0000 mon.a (mon.0) 11863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:54 smithi115 bash[45495]: cluster 2022-04-22T21:25:53.423521+0000 mgr.y (mgr.34107) 8190 : cluster [DBG] pgmap v9135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:54 smithi115 bash[45495]: audit 2022-04-22T21:25:54.114908+0000 mon.a (mon.0) 11863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:55.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:55 smithi121 bash[36902]: audit 2022-04-22T21:25:54.978417+0000 mon.a (mon.0) 11864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:55 smithi115 bash[45495]: audit 2022-04-22T21:25:54.978417+0000 mon.a (mon.0) 11864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:55 smithi115 bash[46934]: audit 2022-04-22T21:25:54.978417+0000 mon.a (mon.0) 11864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:25:56.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:56 smithi121 bash[36902]: cluster 2022-04-22T21:25:55.424560+0000 mgr.y (mgr.34107) 8191 : cluster [DBG] pgmap v9136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:56 smithi121 bash[36902]: audit 2022-04-22T21:25:56.108856+0000 mon.a (mon.0) 11865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:56 smithi115 bash[45495]: cluster 2022-04-22T21:25:55.424560+0000 mgr.y (mgr.34107) 8191 : cluster [DBG] pgmap v9136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:56.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:56 smithi115 bash[45495]: audit 2022-04-22T21:25:56.108856+0000 mon.a (mon.0) 11865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:56.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:56 smithi115 bash[46934]: cluster 2022-04-22T21:25:55.424560+0000 mgr.y (mgr.34107) 8191 : cluster [DBG] pgmap v9136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:56.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:56 smithi115 bash[46934]: audit 2022-04-22T21:25:56.108856+0000 mon.a (mon.0) 11865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:25:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:58 smithi121 bash[36902]: cluster 2022-04-22T21:25:57.425669+0000 mgr.y (mgr.34107) 8192 : cluster [DBG] pgmap v9137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:58 smithi115 bash[45495]: cluster 2022-04-22T21:25:57.425669+0000 mgr.y (mgr.34107) 8192 : cluster [DBG] pgmap v9137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:58 smithi115 bash[46934]: cluster 2022-04-22T21:25:57.425669+0000 mgr.y (mgr.34107) 8192 : cluster [DBG] pgmap v9137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:25:59.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:25:59 smithi121 bash[36902]: audit 2022-04-22T21:25:59.118032+0000 mon.a (mon.0) 11866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:25:59 smithi115 bash[45495]: audit 2022-04-22T21:25:59.118032+0000 mon.a (mon.0) 11866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:25:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:25:59 smithi115 bash[46934]: audit 2022-04-22T21:25:59.118032+0000 mon.a (mon.0) 11866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:00 smithi121 bash[36902]: cluster 2022-04-22T21:25:59.426655+0000 mgr.y (mgr.34107) 8193 : cluster [DBG] pgmap v9138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:00 smithi121 bash[36902]: audit 2022-04-22T21:25:59.982159+0000 mon.a (mon.0) 11867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:00 smithi115 bash[45495]: cluster 2022-04-22T21:25:59.426655+0000 mgr.y (mgr.34107) 8193 : cluster [DBG] pgmap v9138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:00 smithi115 bash[45495]: audit 2022-04-22T21:25:59.982159+0000 mon.a (mon.0) 11867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:00 smithi115 bash[46934]: cluster 2022-04-22T21:25:59.426655+0000 mgr.y (mgr.34107) 8193 : cluster [DBG] pgmap v9138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:00 smithi115 bash[46934]: audit 2022-04-22T21:25:59.982159+0000 mon.a (mon.0) 11867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:01.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:01 smithi121 bash[36902]: audit 2022-04-22T21:26:01.111966+0000 mon.a (mon.0) 11868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:01 smithi115 bash[45495]: audit 2022-04-22T21:26:01.111966+0000 mon.a (mon.0) 11868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:01 smithi115 bash[46934]: audit 2022-04-22T21:26:01.111966+0000 mon.a (mon.0) 11868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:02.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:02 smithi115 bash[45495]: cluster 2022-04-22T21:26:01.427776+0000 mgr.y (mgr.34107) 8194 : cluster [DBG] pgmap v9139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:02.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:02.552Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:02.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:02 smithi115 bash[46934]: cluster 2022-04-22T21:26:01.427776+0000 mgr.y (mgr.34107) 8194 : cluster [DBG] pgmap v9139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:02 smithi121 bash[36902]: cluster 2022-04-22T21:26:01.427776+0000 mgr.y (mgr.34107) 8194 : cluster [DBG] pgmap v9139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:04.421Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:04 smithi115 bash[45495]: cluster 2022-04-22T21:26:03.428904+0000 mgr.y (mgr.34107) 8195 : cluster [DBG] pgmap v9140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:04 smithi115 bash[45495]: audit 2022-04-22T21:26:04.121431+0000 mon.a (mon.0) 11869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:04 smithi115 bash[46934]: cluster 2022-04-22T21:26:03.428904+0000 mgr.y (mgr.34107) 8195 : cluster [DBG] pgmap v9140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:04 smithi115 bash[46934]: audit 2022-04-22T21:26:04.121431+0000 mon.a (mon.0) 11869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:04 smithi121 bash[36902]: cluster 2022-04-22T21:26:03.428904+0000 mgr.y (mgr.34107) 8195 : cluster [DBG] pgmap v9140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:04 smithi121 bash[36902]: audit 2022-04-22T21:26:04.121431+0000 mon.a (mon.0) 11869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:05 smithi115 bash[45495]: audit 2022-04-22T21:26:04.985927+0000 mon.a (mon.0) 11870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:05.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:05 smithi115 bash[46934]: audit 2022-04-22T21:26:04.985927+0000 mon.a (mon.0) 11870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:06.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:05 smithi121 bash[36902]: audit 2022-04-22T21:26:04.985927+0000 mon.a (mon.0) 11870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:06 smithi115 bash[45495]: cluster 2022-04-22T21:26:05.429979+0000 mgr.y (mgr.34107) 8196 : cluster [DBG] pgmap v9141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:06 smithi115 bash[45495]: audit 2022-04-22T21:26:06.115188+0000 mon.a (mon.0) 11871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:06 smithi115 bash[46934]: cluster 2022-04-22T21:26:05.429979+0000 mgr.y (mgr.34107) 8196 : cluster [DBG] pgmap v9141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:06 smithi115 bash[46934]: audit 2022-04-22T21:26:06.115188+0000 mon.a (mon.0) 11871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:06 smithi121 bash[36902]: cluster 2022-04-22T21:26:05.429979+0000 mgr.y (mgr.34107) 8196 : cluster [DBG] pgmap v9141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:06 smithi121 bash[36902]: audit 2022-04-22T21:26:06.115188+0000 mon.a (mon.0) 11871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:08 smithi115 bash[45495]: cluster 2022-04-22T21:26:07.431050+0000 mgr.y (mgr.34107) 8197 : cluster [DBG] pgmap v9142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:08 smithi115 bash[46934]: cluster 2022-04-22T21:26:07.431050+0000 mgr.y (mgr.34107) 8197 : cluster [DBG] pgmap v9142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:08 smithi121 bash[36902]: cluster 2022-04-22T21:26:07.431050+0000 mgr.y (mgr.34107) 8197 : cluster [DBG] pgmap v9142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:09 smithi115 bash[45495]: audit 2022-04-22T21:26:09.125696+0000 mon.a (mon.0) 11872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:09 smithi115 bash[46934]: audit 2022-04-22T21:26:09.125696+0000 mon.a (mon.0) 11872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:09 smithi121 bash[36902]: audit 2022-04-22T21:26:09.125696+0000 mon.a (mon.0) 11872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:10 smithi115 bash[45495]: cluster 2022-04-22T21:26:09.432130+0000 mgr.y (mgr.34107) 8198 : cluster [DBG] pgmap v9143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:10 smithi115 bash[45495]: audit 2022-04-22T21:26:09.989581+0000 mon.a (mon.0) 11873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:10 smithi115 bash[46934]: cluster 2022-04-22T21:26:09.432130+0000 mgr.y (mgr.34107) 8198 : cluster [DBG] pgmap v9143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:10 smithi115 bash[46934]: audit 2022-04-22T21:26:09.989581+0000 mon.a (mon.0) 11873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:10 smithi121 bash[36902]: cluster 2022-04-22T21:26:09.432130+0000 mgr.y (mgr.34107) 8198 : cluster [DBG] pgmap v9143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:10 smithi121 bash[36902]: audit 2022-04-22T21:26:09.989581+0000 mon.a (mon.0) 11873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:11 smithi121 bash[36902]: audit 2022-04-22T21:26:11.118132+0000 mon.a (mon.0) 11874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:11 smithi115 bash[45495]: audit 2022-04-22T21:26:11.118132+0000 mon.a (mon.0) 11874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:11 smithi115 bash[46934]: audit 2022-04-22T21:26:11.118132+0000 mon.a (mon.0) 11874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:12 smithi121 bash[36902]: cluster 2022-04-22T21:26:11.433167+0000 mgr.y (mgr.34107) 8199 : cluster [DBG] pgmap v9144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:12.552Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:12 smithi115 bash[45495]: cluster 2022-04-22T21:26:11.433167+0000 mgr.y (mgr.34107) 8199 : cluster [DBG] pgmap v9144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:12 smithi115 bash[46934]: cluster 2022-04-22T21:26:11.433167+0000 mgr.y (mgr.34107) 8199 : cluster [DBG] pgmap v9144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:14.425Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:14 smithi115 bash[46934]: cluster 2022-04-22T21:26:13.434285+0000 mgr.y (mgr.34107) 8200 : cluster [DBG] pgmap v9145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:14 smithi115 bash[46934]: audit 2022-04-22T21:26:14.127773+0000 mon.a (mon.0) 11875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:14 smithi115 bash[45495]: cluster 2022-04-22T21:26:13.434285+0000 mgr.y (mgr.34107) 8200 : cluster [DBG] pgmap v9145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:14 smithi115 bash[45495]: audit 2022-04-22T21:26:14.127773+0000 mon.a (mon.0) 11875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:15.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:14 smithi121 bash[36902]: cluster 2022-04-22T21:26:13.434285+0000 mgr.y (mgr.34107) 8200 : cluster [DBG] pgmap v9145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:14 smithi121 bash[36902]: audit 2022-04-22T21:26:14.127773+0000 mon.a (mon.0) 11875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:16.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:15 smithi121 bash[36902]: audit 2022-04-22T21:26:14.993283+0000 mon.a (mon.0) 11876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:15 smithi115 bash[45495]: audit 2022-04-22T21:26:14.993283+0000 mon.a (mon.0) 11876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:15 smithi115 bash[46934]: audit 2022-04-22T21:26:14.993283+0000 mon.a (mon.0) 11876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:17.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:16 smithi121 bash[36902]: cluster 2022-04-22T21:26:15.435365+0000 mgr.y (mgr.34107) 8201 : cluster [DBG] pgmap v9146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:16 smithi121 bash[36902]: audit 2022-04-22T21:26:16.030156+0000 mon.a (mon.0) 11877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:26:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:16 smithi121 bash[36902]: audit 2022-04-22T21:26:16.030625+0000 mon.a (mon.0) 11878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:26:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:16 smithi121 bash[36902]: audit 2022-04-22T21:26:16.121125+0000 mon.a (mon.0) 11879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[45495]: cluster 2022-04-22T21:26:15.435365+0000 mgr.y (mgr.34107) 8201 : cluster [DBG] pgmap v9146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[45495]: audit 2022-04-22T21:26:16.030156+0000 mon.a (mon.0) 11877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:26:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[45495]: audit 2022-04-22T21:26:16.030625+0000 mon.a (mon.0) 11878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:26:17.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[45495]: audit 2022-04-22T21:26:16.121125+0000 mon.a (mon.0) 11879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[46934]: cluster 2022-04-22T21:26:15.435365+0000 mgr.y (mgr.34107) 8201 : cluster [DBG] pgmap v9146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[46934]: audit 2022-04-22T21:26:16.030156+0000 mon.a (mon.0) 11877 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[46934]: audit 2022-04-22T21:26:16.030625+0000 mon.a (mon.0) 11878 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:16 smithi115 bash[46934]: audit 2022-04-22T21:26:16.121125+0000 mon.a (mon.0) 11879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:19.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:18 smithi121 bash[36902]: cluster 2022-04-22T21:26:17.436488+0000 mgr.y (mgr.34107) 8202 : cluster [DBG] pgmap v9147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:18 smithi115 bash[45495]: cluster 2022-04-22T21:26:17.436488+0000 mgr.y (mgr.34107) 8202 : cluster [DBG] pgmap v9147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:18 smithi115 bash[46934]: cluster 2022-04-22T21:26:17.436488+0000 mgr.y (mgr.34107) 8202 : cluster [DBG] pgmap v9147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:19.227 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:26:19.578 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:26:19.617 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 114s ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 114s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 114s ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 114s ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 2m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 114s ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:26:19.618 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 114s ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 114s ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 114s ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 114s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 114s ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 2m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:26:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:26:19.985 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:26:19.985 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:26:19.985 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:26:19.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:19.985 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:26:19.986 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:26:19.987 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:26:19.987 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:26:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:19 smithi121 bash[36902]: audit 2022-04-22T21:26:19.130795+0000 mon.a (mon.0) 11880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:19 smithi115 bash[45495]: audit 2022-04-22T21:26:19.130795+0000 mon.a (mon.0) 11880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:19 smithi115 bash[46934]: audit 2022-04-22T21:26:19.130795+0000 mon.a (mon.0) 11880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:20 smithi121 bash[36902]: audit 2022-04-22T21:26:19.208113+0000 mgr.y (mgr.34107) 8203 : audit [DBG] from='client.53815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:20 smithi121 bash[36902]: cluster 2022-04-22T21:26:19.437554+0000 mgr.y (mgr.34107) 8204 : cluster [DBG] pgmap v9148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:20 smithi121 bash[36902]: audit 2022-04-22T21:26:19.555668+0000 mgr.y (mgr.34107) 8205 : audit [DBG] from='client.43890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:20 smithi121 bash[36902]: audit 2022-04-22T21:26:19.981517+0000 mon.a (mon.0) 11881 : audit [DBG] from='client.? 172.21.15.115:0/1669479325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:20 smithi121 bash[36902]: audit 2022-04-22T21:26:19.996968+0000 mon.a (mon.0) 11882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[45495]: audit 2022-04-22T21:26:19.208113+0000 mgr.y (mgr.34107) 8203 : audit [DBG] from='client.53815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[45495]: cluster 2022-04-22T21:26:19.437554+0000 mgr.y (mgr.34107) 8204 : cluster [DBG] pgmap v9148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[45495]: audit 2022-04-22T21:26:19.555668+0000 mgr.y (mgr.34107) 8205 : audit [DBG] from='client.43890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[45495]: audit 2022-04-22T21:26:19.981517+0000 mon.a (mon.0) 11881 : audit [DBG] from='client.? 172.21.15.115:0/1669479325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[45495]: audit 2022-04-22T21:26:19.996968+0000 mon.a (mon.0) 11882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[46934]: audit 2022-04-22T21:26:19.208113+0000 mgr.y (mgr.34107) 8203 : audit [DBG] from='client.53815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[46934]: cluster 2022-04-22T21:26:19.437554+0000 mgr.y (mgr.34107) 8204 : cluster [DBG] pgmap v9148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[46934]: audit 2022-04-22T21:26:19.555668+0000 mgr.y (mgr.34107) 8205 : audit [DBG] from='client.43890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[46934]: audit 2022-04-22T21:26:19.981517+0000 mon.a (mon.0) 11881 : audit [DBG] from='client.? 172.21.15.115:0/1669479325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:20 smithi115 bash[46934]: audit 2022-04-22T21:26:19.996968+0000 mon.a (mon.0) 11882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:21 smithi121 bash[36902]: audit 2022-04-22T21:26:21.124295+0000 mon.a (mon.0) 11883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:21 smithi115 bash[46934]: audit 2022-04-22T21:26:21.124295+0000 mon.a (mon.0) 11883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:21 smithi115 bash[45495]: audit 2022-04-22T21:26:21.124295+0000 mon.a (mon.0) 11883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:22 smithi121 bash[36902]: cluster 2022-04-22T21:26:21.438649+0000 mgr.y (mgr.34107) 8206 : cluster [DBG] pgmap v9149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:22 smithi115 bash[45495]: cluster 2022-04-22T21:26:21.438649+0000 mgr.y (mgr.34107) 8206 : cluster [DBG] pgmap v9149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:22.552Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:22 smithi115 bash[46934]: cluster 2022-04-22T21:26:21.438649+0000 mgr.y (mgr.34107) 8206 : cluster [DBG] pgmap v9149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:24.424Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:24 smithi115 bash[45495]: cluster 2022-04-22T21:26:23.439758+0000 mgr.y (mgr.34107) 8207 : cluster [DBG] pgmap v9150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:24 smithi115 bash[45495]: audit 2022-04-22T21:26:24.133401+0000 mon.a (mon.0) 11884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:24 smithi115 bash[46934]: cluster 2022-04-22T21:26:23.439758+0000 mgr.y (mgr.34107) 8207 : cluster [DBG] pgmap v9150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:24 smithi115 bash[46934]: audit 2022-04-22T21:26:24.133401+0000 mon.a (mon.0) 11884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:24 smithi121 bash[36902]: cluster 2022-04-22T21:26:23.439758+0000 mgr.y (mgr.34107) 8207 : cluster [DBG] pgmap v9150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:24 smithi121 bash[36902]: audit 2022-04-22T21:26:24.133401+0000 mon.a (mon.0) 11884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:25 smithi121 bash[36902]: audit 2022-04-22T21:26:25.000328+0000 mon.a (mon.0) 11885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:25 smithi115 bash[46934]: audit 2022-04-22T21:26:25.000328+0000 mon.a (mon.0) 11885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:25 smithi115 bash[45495]: audit 2022-04-22T21:26:25.000328+0000 mon.a (mon.0) 11885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:26.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: cluster 2022-04-22T21:26:25.440825+0000 mgr.y (mgr.34107) 8208 : cluster [DBG] pgmap v9151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: audit 2022-04-22T21:26:26.022145+0000 mon.a (mon.0) 11886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:26:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: audit 2022-04-22T21:26:26.024851+0000 mon.a (mon.0) 11887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:26:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: audit 2022-04-22T21:26:26.026244+0000 mon.a (mon.0) 11888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:26:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: audit 2022-04-22T21:26:26.127241+0000 mon.a (mon.0) 11889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:26 smithi121 bash[36902]: audit 2022-04-22T21:26:26.391894+0000 mon.a (mon.0) 11890 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: cluster 2022-04-22T21:26:25.440825+0000 mgr.y (mgr.34107) 8208 : cluster [DBG] pgmap v9151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: audit 2022-04-22T21:26:26.022145+0000 mon.a (mon.0) 11886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: audit 2022-04-22T21:26:26.024851+0000 mon.a (mon.0) 11887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: audit 2022-04-22T21:26:26.026244+0000 mon.a (mon.0) 11888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: audit 2022-04-22T21:26:26.127241+0000 mon.a (mon.0) 11889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[45495]: audit 2022-04-22T21:26:26.391894+0000 mon.a (mon.0) 11890 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: cluster 2022-04-22T21:26:25.440825+0000 mgr.y (mgr.34107) 8208 : cluster [DBG] pgmap v9151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: audit 2022-04-22T21:26:26.022145+0000 mon.a (mon.0) 11886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: audit 2022-04-22T21:26:26.024851+0000 mon.a (mon.0) 11887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: audit 2022-04-22T21:26:26.026244+0000 mon.a (mon.0) 11888 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: audit 2022-04-22T21:26:26.127241+0000 mon.a (mon.0) 11889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:26 smithi115 bash[46934]: audit 2022-04-22T21:26:26.391894+0000 mon.a (mon.0) 11890 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:26:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:28 smithi121 bash[36902]: cluster 2022-04-22T21:26:27.441915+0000 mgr.y (mgr.34107) 8209 : cluster [DBG] pgmap v9152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:28 smithi115 bash[45495]: cluster 2022-04-22T21:26:27.441915+0000 mgr.y (mgr.34107) 8209 : cluster [DBG] pgmap v9152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:28 smithi115 bash[46934]: cluster 2022-04-22T21:26:27.441915+0000 mgr.y (mgr.34107) 8209 : cluster [DBG] pgmap v9152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:29 smithi121 bash[36902]: audit 2022-04-22T21:26:29.136605+0000 mon.a (mon.0) 11891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:29 smithi115 bash[45495]: audit 2022-04-22T21:26:29.136605+0000 mon.a (mon.0) 11891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:29 smithi115 bash[46934]: audit 2022-04-22T21:26:29.136605+0000 mon.a (mon.0) 11891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:30 smithi121 bash[36902]: cluster 2022-04-22T21:26:29.442944+0000 mgr.y (mgr.34107) 8210 : cluster [DBG] pgmap v9153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:30 smithi121 bash[36902]: audit 2022-04-22T21:26:30.004049+0000 mon.a (mon.0) 11892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:30 smithi115 bash[45495]: cluster 2022-04-22T21:26:29.442944+0000 mgr.y (mgr.34107) 8210 : cluster [DBG] pgmap v9153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:30 smithi115 bash[45495]: audit 2022-04-22T21:26:30.004049+0000 mon.a (mon.0) 11892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:30 smithi115 bash[46934]: cluster 2022-04-22T21:26:29.442944+0000 mgr.y (mgr.34107) 8210 : cluster [DBG] pgmap v9153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:30 smithi115 bash[46934]: audit 2022-04-22T21:26:30.004049+0000 mon.a (mon.0) 11892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:32.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:31 smithi121 bash[36902]: audit 2022-04-22T21:26:31.130473+0000 mon.a (mon.0) 11893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:31 smithi115 bash[46934]: audit 2022-04-22T21:26:31.130473+0000 mon.a (mon.0) 11893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:31 smithi115 bash[45495]: audit 2022-04-22T21:26:31.130473+0000 mon.a (mon.0) 11893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:33.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:32 smithi121 bash[36902]: cluster 2022-04-22T21:26:31.444065+0000 mgr.y (mgr.34107) 8211 : cluster [DBG] pgmap v9154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:32 smithi115 bash[46934]: cluster 2022-04-22T21:26:31.444065+0000 mgr.y (mgr.34107) 8211 : cluster [DBG] pgmap v9154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:32.552Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:32 smithi115 bash[45495]: cluster 2022-04-22T21:26:31.444065+0000 mgr.y (mgr.34107) 8211 : cluster [DBG] pgmap v9154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:34.425Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:34 smithi115 bash[45495]: cluster 2022-04-22T21:26:33.445327+0000 mgr.y (mgr.34107) 8212 : cluster [DBG] pgmap v9155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:34 smithi115 bash[45495]: audit 2022-04-22T21:26:34.139767+0000 mon.a (mon.0) 11894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:34 smithi115 bash[46934]: cluster 2022-04-22T21:26:33.445327+0000 mgr.y (mgr.34107) 8212 : cluster [DBG] pgmap v9155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:34 smithi115 bash[46934]: audit 2022-04-22T21:26:34.139767+0000 mon.a (mon.0) 11894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:34 smithi121 bash[36902]: cluster 2022-04-22T21:26:33.445327+0000 mgr.y (mgr.34107) 8212 : cluster [DBG] pgmap v9155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:34 smithi121 bash[36902]: audit 2022-04-22T21:26:34.139767+0000 mon.a (mon.0) 11894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:35 smithi121 bash[36902]: audit 2022-04-22T21:26:35.007862+0000 mon.a (mon.0) 11895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:35 smithi115 bash[45495]: audit 2022-04-22T21:26:35.007862+0000 mon.a (mon.0) 11895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:35 smithi115 bash[46934]: audit 2022-04-22T21:26:35.007862+0000 mon.a (mon.0) 11895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:36 smithi121 bash[36902]: cluster 2022-04-22T21:26:35.446403+0000 mgr.y (mgr.34107) 8213 : cluster [DBG] pgmap v9156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:36 smithi121 bash[36902]: audit 2022-04-22T21:26:36.133668+0000 mon.a (mon.0) 11896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:36 smithi115 bash[45495]: cluster 2022-04-22T21:26:35.446403+0000 mgr.y (mgr.34107) 8213 : cluster [DBG] pgmap v9156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:36 smithi115 bash[45495]: audit 2022-04-22T21:26:36.133668+0000 mon.a (mon.0) 11896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:36 smithi115 bash[46934]: cluster 2022-04-22T21:26:35.446403+0000 mgr.y (mgr.34107) 8213 : cluster [DBG] pgmap v9156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:36 smithi115 bash[46934]: audit 2022-04-22T21:26:36.133668+0000 mon.a (mon.0) 11896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:38 smithi121 bash[36902]: cluster 2022-04-22T21:26:37.447501+0000 mgr.y (mgr.34107) 8214 : cluster [DBG] pgmap v9157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:38 smithi115 bash[45495]: cluster 2022-04-22T21:26:37.447501+0000 mgr.y (mgr.34107) 8214 : cluster [DBG] pgmap v9157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:38 smithi115 bash[46934]: cluster 2022-04-22T21:26:37.447501+0000 mgr.y (mgr.34107) 8214 : cluster [DBG] pgmap v9157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:39 smithi121 bash[36902]: audit 2022-04-22T21:26:39.143070+0000 mon.a (mon.0) 11897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:39 smithi115 bash[45495]: audit 2022-04-22T21:26:39.143070+0000 mon.a (mon.0) 11897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:39 smithi115 bash[46934]: audit 2022-04-22T21:26:39.143070+0000 mon.a (mon.0) 11897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:40 smithi121 bash[36902]: cluster 2022-04-22T21:26:39.448535+0000 mgr.y (mgr.34107) 8215 : cluster [DBG] pgmap v9158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:41.024 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:40 smithi121 bash[36902]: audit 2022-04-22T21:26:40.011615+0000 mon.a (mon.0) 11898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:40 smithi115 bash[45495]: cluster 2022-04-22T21:26:39.448535+0000 mgr.y (mgr.34107) 8215 : cluster [DBG] pgmap v9158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:40 smithi115 bash[45495]: audit 2022-04-22T21:26:40.011615+0000 mon.a (mon.0) 11898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:40 smithi115 bash[46934]: cluster 2022-04-22T21:26:39.448535+0000 mgr.y (mgr.34107) 8215 : cluster [DBG] pgmap v9158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:40 smithi115 bash[46934]: audit 2022-04-22T21:26:40.011615+0000 mon.a (mon.0) 11898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:42.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:41 smithi121 bash[36902]: audit 2022-04-22T21:26:41.136641+0000 mon.a (mon.0) 11899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:41 smithi115 bash[45495]: audit 2022-04-22T21:26:41.136641+0000 mon.a (mon.0) 11899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:41 smithi115 bash[46934]: audit 2022-04-22T21:26:41.136641+0000 mon.a (mon.0) 11899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:42 smithi121 bash[36902]: cluster 2022-04-22T21:26:41.449558+0000 mgr.y (mgr.34107) 8216 : cluster [DBG] pgmap v9159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:42.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:43.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:42 smithi115 bash[45495]: cluster 2022-04-22T21:26:41.449558+0000 mgr.y (mgr.34107) 8216 : cluster [DBG] pgmap v9159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:43.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:42 smithi115 bash[46934]: cluster 2022-04-22T21:26:41.449558+0000 mgr.y (mgr.34107) 8216 : cluster [DBG] pgmap v9159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:44.427Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:44 smithi115 bash[45495]: cluster 2022-04-22T21:26:43.450712+0000 mgr.y (mgr.34107) 8217 : cluster [DBG] pgmap v9160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:44 smithi115 bash[45495]: audit 2022-04-22T21:26:44.146135+0000 mon.a (mon.0) 11900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:44 smithi115 bash[46934]: cluster 2022-04-22T21:26:43.450712+0000 mgr.y (mgr.34107) 8217 : cluster [DBG] pgmap v9160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:44 smithi115 bash[46934]: audit 2022-04-22T21:26:44.146135+0000 mon.a (mon.0) 11900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:44 smithi121 bash[36902]: cluster 2022-04-22T21:26:43.450712+0000 mgr.y (mgr.34107) 8217 : cluster [DBG] pgmap v9160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:44 smithi121 bash[36902]: audit 2022-04-22T21:26:44.146135+0000 mon.a (mon.0) 11900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:45 smithi121 bash[36902]: audit 2022-04-22T21:26:45.014884+0000 mon.a (mon.0) 11901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:45 smithi115 bash[46934]: audit 2022-04-22T21:26:45.014884+0000 mon.a (mon.0) 11901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:45 smithi115 bash[45495]: audit 2022-04-22T21:26:45.014884+0000 mon.a (mon.0) 11901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:46 smithi121 bash[36902]: cluster 2022-04-22T21:26:45.451839+0000 mgr.y (mgr.34107) 8218 : cluster [DBG] pgmap v9161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:47.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:46 smithi121 bash[36902]: audit 2022-04-22T21:26:46.139895+0000 mon.a (mon.0) 11902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:47.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:46 smithi115 bash[45495]: cluster 2022-04-22T21:26:45.451839+0000 mgr.y (mgr.34107) 8218 : cluster [DBG] pgmap v9161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:47.073 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:46 smithi115 bash[45495]: audit 2022-04-22T21:26:46.139895+0000 mon.a (mon.0) 11902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:47.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:46 smithi115 bash[46934]: cluster 2022-04-22T21:26:45.451839+0000 mgr.y (mgr.34107) 8218 : cluster [DBG] pgmap v9161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:47.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:46 smithi115 bash[46934]: audit 2022-04-22T21:26:46.139895+0000 mon.a (mon.0) 11902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:48 smithi121 bash[36902]: cluster 2022-04-22T21:26:47.452913+0000 mgr.y (mgr.34107) 8219 : cluster [DBG] pgmap v9162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:48 smithi115 bash[45495]: cluster 2022-04-22T21:26:47.452913+0000 mgr.y (mgr.34107) 8219 : cluster [DBG] pgmap v9162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:48 smithi115 bash[46934]: cluster 2022-04-22T21:26:47.452913+0000 mgr.y (mgr.34107) 8219 : cluster [DBG] pgmap v9162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:50.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:49 smithi121 bash[36902]: audit 2022-04-22T21:26:49.149292+0000 mon.a (mon.0) 11903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:49 smithi115 bash[45495]: audit 2022-04-22T21:26:49.149292+0000 mon.a (mon.0) 11903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:49 smithi115 bash[46934]: audit 2022-04-22T21:26:49.149292+0000 mon.a (mon.0) 11903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:50.356 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:26:50.704 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:26:50.704 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (3h) 2m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 3m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:26:50.705 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 3m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 3m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:26:50.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 3m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:26:50.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 3m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:26:50.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 3m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:26:50.707 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 3m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:26:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:50 smithi121 bash[36902]: cluster 2022-04-22T21:26:49.453944+0000 mgr.y (mgr.34107) 8220 : cluster [DBG] pgmap v9163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:50 smithi121 bash[36902]: audit 2022-04-22T21:26:50.018576+0000 mon.a (mon.0) 11904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:50 smithi115 bash[45495]: cluster 2022-04-22T21:26:49.453944+0000 mgr.y (mgr.34107) 8220 : cluster [DBG] pgmap v9163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:50 smithi115 bash[45495]: audit 2022-04-22T21:26:50.018576+0000 mon.a (mon.0) 11904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:50 smithi115 bash[46934]: cluster 2022-04-22T21:26:49.453944+0000 mgr.y (mgr.34107) 8220 : cluster [DBG] pgmap v9163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:50 smithi115 bash[46934]: audit 2022-04-22T21:26:50.018576+0000 mon.a (mon.0) 11904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:51.103 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:26:51.103 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:26:51.103 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:26:51.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:26:51.104 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:26:51.105 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:26:51.105 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:26:51.105 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:26:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:51 smithi121 bash[36902]: audit 2022-04-22T21:26:50.338200+0000 mgr.y (mgr.34107) 8221 : audit [DBG] from='client.43902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:51 smithi121 bash[36902]: audit 2022-04-22T21:26:50.681662+0000 mgr.y (mgr.34107) 8222 : audit [DBG] from='client.43908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:51 smithi121 bash[36902]: audit 2022-04-22T21:26:51.099230+0000 mon.c (mon.1) 295 : audit [DBG] from='client.? 172.21.15.115:0/3539616293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:51 smithi121 bash[36902]: audit 2022-04-22T21:26:51.142980+0000 mon.a (mon.0) 11905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[46934]: audit 2022-04-22T21:26:50.338200+0000 mgr.y (mgr.34107) 8221 : audit [DBG] from='client.43902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[46934]: audit 2022-04-22T21:26:50.681662+0000 mgr.y (mgr.34107) 8222 : audit [DBG] from='client.43908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[46934]: audit 2022-04-22T21:26:51.099230+0000 mon.c (mon.1) 295 : audit [DBG] from='client.? 172.21.15.115:0/3539616293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[46934]: audit 2022-04-22T21:26:51.142980+0000 mon.a (mon.0) 11905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[45495]: audit 2022-04-22T21:26:50.338200+0000 mgr.y (mgr.34107) 8221 : audit [DBG] from='client.43902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[45495]: audit 2022-04-22T21:26:50.681662+0000 mgr.y (mgr.34107) 8222 : audit [DBG] from='client.43908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:26:52.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[45495]: audit 2022-04-22T21:26:51.099230+0000 mon.c (mon.1) 295 : audit [DBG] from='client.? 172.21.15.115:0/3539616293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:26:52.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:51 smithi115 bash[45495]: audit 2022-04-22T21:26:51.142980+0000 mon.a (mon.0) 11905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:52 smithi121 bash[36902]: cluster 2022-04-22T21:26:51.454958+0000 mgr.y (mgr.34107) 8223 : cluster [DBG] pgmap v9164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:52 smithi115 bash[45495]: cluster 2022-04-22T21:26:51.454958+0000 mgr.y (mgr.34107) 8223 : cluster [DBG] pgmap v9164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:52.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:52 smithi115 bash[46934]: cluster 2022-04-22T21:26:51.454958+0000 mgr.y (mgr.34107) 8223 : cluster [DBG] pgmap v9164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:54.694 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:26:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:26:54.426Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:26:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:54 smithi121 bash[36902]: cluster 2022-04-22T21:26:53.456146+0000 mgr.y (mgr.34107) 8224 : cluster [DBG] pgmap v9165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:54 smithi121 bash[36902]: audit 2022-04-22T21:26:54.152442+0000 mon.a (mon.0) 11906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:54 smithi115 bash[45495]: cluster 2022-04-22T21:26:53.456146+0000 mgr.y (mgr.34107) 8224 : cluster [DBG] pgmap v9165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:54 smithi115 bash[45495]: audit 2022-04-22T21:26:54.152442+0000 mon.a (mon.0) 11906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:54 smithi115 bash[46934]: cluster 2022-04-22T21:26:53.456146+0000 mgr.y (mgr.34107) 8224 : cluster [DBG] pgmap v9165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:54 smithi115 bash[46934]: audit 2022-04-22T21:26:54.152442+0000 mon.a (mon.0) 11906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:26:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:55 smithi121 bash[36902]: audit 2022-04-22T21:26:55.022226+0000 mon.a (mon.0) 11907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:55 smithi115 bash[45495]: audit 2022-04-22T21:26:55.022226+0000 mon.a (mon.0) 11907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:55 smithi115 bash[46934]: audit 2022-04-22T21:26:55.022226+0000 mon.a (mon.0) 11907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:26:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:56 smithi121 bash[36902]: cluster 2022-04-22T21:26:55.457256+0000 mgr.y (mgr.34107) 8225 : cluster [DBG] pgmap v9166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:56 smithi121 bash[36902]: audit 2022-04-22T21:26:56.146003+0000 mon.a (mon.0) 11908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:56 smithi115 bash[45495]: cluster 2022-04-22T21:26:55.457256+0000 mgr.y (mgr.34107) 8225 : cluster [DBG] pgmap v9166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:56 smithi115 bash[45495]: audit 2022-04-22T21:26:56.146003+0000 mon.a (mon.0) 11908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:56 smithi115 bash[46934]: cluster 2022-04-22T21:26:55.457256+0000 mgr.y (mgr.34107) 8225 : cluster [DBG] pgmap v9166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:56 smithi115 bash[46934]: audit 2022-04-22T21:26:56.146003+0000 mon.a (mon.0) 11908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:26:59.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:58 smithi121 bash[36902]: cluster 2022-04-22T21:26:57.458326+0000 mgr.y (mgr.34107) 8226 : cluster [DBG] pgmap v9167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:59.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:58 smithi115 bash[45495]: cluster 2022-04-22T21:26:57.458326+0000 mgr.y (mgr.34107) 8226 : cluster [DBG] pgmap v9167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:26:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:58 smithi115 bash[46934]: cluster 2022-04-22T21:26:57.458326+0000 mgr.y (mgr.34107) 8226 : cluster [DBG] pgmap v9167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:26:59 smithi121 bash[36902]: audit 2022-04-22T21:26:59.155407+0000 mon.a (mon.0) 11909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:26:59 smithi115 bash[45495]: audit 2022-04-22T21:26:59.155407+0000 mon.a (mon.0) 11909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:26:59 smithi115 bash[46934]: audit 2022-04-22T21:26:59.155407+0000 mon.a (mon.0) 11909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:01.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:00 smithi121 bash[36902]: cluster 2022-04-22T21:26:59.459367+0000 mgr.y (mgr.34107) 8227 : cluster [DBG] pgmap v9168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:00 smithi121 bash[36902]: audit 2022-04-22T21:27:00.026123+0000 mon.a (mon.0) 11910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:00 smithi115 bash[45495]: cluster 2022-04-22T21:26:59.459367+0000 mgr.y (mgr.34107) 8227 : cluster [DBG] pgmap v9168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:00 smithi115 bash[45495]: audit 2022-04-22T21:27:00.026123+0000 mon.a (mon.0) 11910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:00 smithi115 bash[46934]: cluster 2022-04-22T21:26:59.459367+0000 mgr.y (mgr.34107) 8227 : cluster [DBG] pgmap v9168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:00 smithi115 bash[46934]: audit 2022-04-22T21:27:00.026123+0000 mon.a (mon.0) 11910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:01 smithi121 bash[36902]: audit 2022-04-22T21:27:01.149021+0000 mon.a (mon.0) 11911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:01 smithi115 bash[45495]: audit 2022-04-22T21:27:01.149021+0000 mon.a (mon.0) 11911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:01 smithi115 bash[46934]: audit 2022-04-22T21:27:01.149021+0000 mon.a (mon.0) 11911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:02 smithi121 bash[36902]: cluster 2022-04-22T21:27:01.460612+0000 mgr.y (mgr.34107) 8228 : cluster [DBG] pgmap v9169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:02 smithi115 bash[45495]: cluster 2022-04-22T21:27:01.460612+0000 mgr.y (mgr.34107) 8228 : cluster [DBG] pgmap v9169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:02.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:02 smithi115 bash[46934]: cluster 2022-04-22T21:27:01.460612+0000 mgr.y (mgr.34107) 8228 : cluster [DBG] pgmap v9169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:04.726 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:04.428Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:04 smithi121 bash[36902]: cluster 2022-04-22T21:27:03.461661+0000 mgr.y (mgr.34107) 8229 : cluster [DBG] pgmap v9170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:05.043 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:04 smithi121 bash[36902]: audit 2022-04-22T21:27:04.158673+0000 mon.a (mon.0) 11912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:05.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:04 smithi115 bash[45495]: cluster 2022-04-22T21:27:03.461661+0000 mgr.y (mgr.34107) 8229 : cluster [DBG] pgmap v9170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:05.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:04 smithi115 bash[45495]: audit 2022-04-22T21:27:04.158673+0000 mon.a (mon.0) 11912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:05.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:04 smithi115 bash[46934]: cluster 2022-04-22T21:27:03.461661+0000 mgr.y (mgr.34107) 8229 : cluster [DBG] pgmap v9170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:05.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:04 smithi115 bash[46934]: audit 2022-04-22T21:27:04.158673+0000 mon.a (mon.0) 11912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:05 smithi121 bash[36902]: audit 2022-04-22T21:27:05.029952+0000 mon.a (mon.0) 11913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:05 smithi115 bash[45495]: audit 2022-04-22T21:27:05.029952+0000 mon.a (mon.0) 11913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:06.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:05 smithi115 bash[46934]: audit 2022-04-22T21:27:05.029952+0000 mon.a (mon.0) 11913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:06 smithi121 bash[36902]: cluster 2022-04-22T21:27:05.462731+0000 mgr.y (mgr.34107) 8230 : cluster [DBG] pgmap v9171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:06 smithi121 bash[36902]: audit 2022-04-22T21:27:06.152179+0000 mon.a (mon.0) 11914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:06 smithi115 bash[46934]: cluster 2022-04-22T21:27:05.462731+0000 mgr.y (mgr.34107) 8230 : cluster [DBG] pgmap v9171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:06 smithi115 bash[46934]: audit 2022-04-22T21:27:06.152179+0000 mon.a (mon.0) 11914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:06 smithi115 bash[45495]: cluster 2022-04-22T21:27:05.462731+0000 mgr.y (mgr.34107) 8230 : cluster [DBG] pgmap v9171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:06 smithi115 bash[45495]: audit 2022-04-22T21:27:06.152179+0000 mon.a (mon.0) 11914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:08 smithi121 bash[36902]: cluster 2022-04-22T21:27:07.463835+0000 mgr.y (mgr.34107) 8231 : cluster [DBG] pgmap v9172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:08 smithi115 bash[45495]: cluster 2022-04-22T21:27:07.463835+0000 mgr.y (mgr.34107) 8231 : cluster [DBG] pgmap v9172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:08 smithi115 bash[46934]: cluster 2022-04-22T21:27:07.463835+0000 mgr.y (mgr.34107) 8231 : cluster [DBG] pgmap v9172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:09 smithi121 bash[36902]: audit 2022-04-22T21:27:09.161883+0000 mon.a (mon.0) 11915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:09 smithi115 bash[45495]: audit 2022-04-22T21:27:09.161883+0000 mon.a (mon.0) 11915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:10.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:09 smithi115 bash[46934]: audit 2022-04-22T21:27:09.161883+0000 mon.a (mon.0) 11915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:10 smithi121 bash[36902]: cluster 2022-04-22T21:27:09.464917+0000 mgr.y (mgr.34107) 8232 : cluster [DBG] pgmap v9173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:11.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:10 smithi121 bash[36902]: audit 2022-04-22T21:27:10.033562+0000 mon.a (mon.0) 11916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:10 smithi115 bash[45495]: cluster 2022-04-22T21:27:09.464917+0000 mgr.y (mgr.34107) 8232 : cluster [DBG] pgmap v9173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:10 smithi115 bash[45495]: audit 2022-04-22T21:27:10.033562+0000 mon.a (mon.0) 11916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:11.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:10 smithi115 bash[46934]: cluster 2022-04-22T21:27:09.464917+0000 mgr.y (mgr.34107) 8232 : cluster [DBG] pgmap v9173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:11.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:10 smithi115 bash[46934]: audit 2022-04-22T21:27:10.033562+0000 mon.a (mon.0) 11916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:11 smithi121 bash[36902]: audit 2022-04-22T21:27:11.155059+0000 mon.a (mon.0) 11917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:11 smithi115 bash[45495]: audit 2022-04-22T21:27:11.155059+0000 mon.a (mon.0) 11917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:11 smithi115 bash[46934]: audit 2022-04-22T21:27:11.155059+0000 mon.a (mon.0) 11917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:12 smithi121 bash[36902]: cluster 2022-04-22T21:27:11.466038+0000 mgr.y (mgr.34107) 8233 : cluster [DBG] pgmap v9174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:12 smithi115 bash[45495]: cluster 2022-04-22T21:27:11.466038+0000 mgr.y (mgr.34107) 8233 : cluster [DBG] pgmap v9174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:12 smithi115 bash[46934]: cluster 2022-04-22T21:27:11.466038+0000 mgr.y (mgr.34107) 8233 : cluster [DBG] pgmap v9174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:12.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:14.766 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:14.428Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:14 smithi115 bash[45495]: cluster 2022-04-22T21:27:13.467155+0000 mgr.y (mgr.34107) 8234 : cluster [DBG] pgmap v9175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:14 smithi115 bash[45495]: audit 2022-04-22T21:27:14.164972+0000 mon.a (mon.0) 11918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:14 smithi115 bash[46934]: cluster 2022-04-22T21:27:13.467155+0000 mgr.y (mgr.34107) 8234 : cluster [DBG] pgmap v9175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:14 smithi115 bash[46934]: audit 2022-04-22T21:27:14.164972+0000 mon.a (mon.0) 11918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:14 smithi121 bash[36902]: cluster 2022-04-22T21:27:13.467155+0000 mgr.y (mgr.34107) 8234 : cluster [DBG] pgmap v9175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:14 smithi121 bash[36902]: audit 2022-04-22T21:27:14.164972+0000 mon.a (mon.0) 11918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:15 smithi115 bash[46934]: audit 2022-04-22T21:27:15.037407+0000 mon.a (mon.0) 11919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:15 smithi115 bash[45495]: audit 2022-04-22T21:27:15.037407+0000 mon.a (mon.0) 11919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:16.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:15 smithi121 bash[36902]: audit 2022-04-22T21:27:15.037407+0000 mon.a (mon.0) 11919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[45495]: cluster 2022-04-22T21:27:15.468188+0000 mgr.y (mgr.34107) 8235 : cluster [DBG] pgmap v9176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:17.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[45495]: audit 2022-04-22T21:27:16.040136+0000 mon.a (mon.0) 11920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:27:17.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[45495]: audit 2022-04-22T21:27:16.043078+0000 mon.a (mon.0) 11921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:27:17.068 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[45495]: audit 2022-04-22T21:27:16.158356+0000 mon.a (mon.0) 11922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:17.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[46934]: cluster 2022-04-22T21:27:15.468188+0000 mgr.y (mgr.34107) 8235 : cluster [DBG] pgmap v9176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:17.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[46934]: audit 2022-04-22T21:27:16.040136+0000 mon.a (mon.0) 11920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:27:17.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[46934]: audit 2022-04-22T21:27:16.043078+0000 mon.a (mon.0) 11921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:27:17.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:16 smithi115 bash[46934]: audit 2022-04-22T21:27:16.158356+0000 mon.a (mon.0) 11922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:16 smithi121 bash[36902]: cluster 2022-04-22T21:27:15.468188+0000 mgr.y (mgr.34107) 8235 : cluster [DBG] pgmap v9176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:16 smithi121 bash[36902]: audit 2022-04-22T21:27:16.040136+0000 mon.a (mon.0) 11920 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:27:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:16 smithi121 bash[36902]: audit 2022-04-22T21:27:16.043078+0000 mon.a (mon.0) 11921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:27:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:16 smithi121 bash[36902]: audit 2022-04-22T21:27:16.158356+0000 mon.a (mon.0) 11922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:18 smithi115 bash[45495]: cluster 2022-04-22T21:27:17.469253+0000 mgr.y (mgr.34107) 8236 : cluster [DBG] pgmap v9177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:18 smithi115 bash[46934]: cluster 2022-04-22T21:27:17.469253+0000 mgr.y (mgr.34107) 8236 : cluster [DBG] pgmap v9177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:19.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:18 smithi121 bash[36902]: cluster 2022-04-22T21:27:17.469253+0000 mgr.y (mgr.34107) 8236 : cluster [DBG] pgmap v9177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:19 smithi115 bash[45495]: audit 2022-04-22T21:27:19.167747+0000 mon.a (mon.0) 11923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:19 smithi115 bash[46934]: audit 2022-04-22T21:27:19.167747+0000 mon.a (mon.0) 11923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:19 smithi121 bash[36902]: audit 2022-04-22T21:27:19.167747+0000 mon.a (mon.0) 11923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:20 smithi115 bash[45495]: cluster 2022-04-22T21:27:19.470305+0000 mgr.y (mgr.34107) 8237 : cluster [DBG] pgmap v9178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:20 smithi115 bash[45495]: audit 2022-04-22T21:27:20.041056+0000 mon.a (mon.0) 11924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:20 smithi115 bash[46934]: cluster 2022-04-22T21:27:19.470305+0000 mgr.y (mgr.34107) 8237 : cluster [DBG] pgmap v9178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:20 smithi115 bash[46934]: audit 2022-04-22T21:27:20.041056+0000 mon.a (mon.0) 11924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:20 smithi121 bash[36902]: cluster 2022-04-22T21:27:19.470305+0000 mgr.y (mgr.34107) 8237 : cluster [DBG] pgmap v9178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:20 smithi121 bash[36902]: audit 2022-04-22T21:27:20.041056+0000 mon.a (mon.0) 11924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:21.474 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:27:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:27:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:27:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:27:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:27:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (3h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 2m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 2m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 2m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 2m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:27:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 2m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 2m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 2m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:27:21.819 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:27:22.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:22.226 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:27:22.227 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:27:22.227 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:27:22.227 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:27:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:21 smithi121 bash[36902]: audit 2022-04-22T21:27:21.161267+0000 mon.a (mon.0) 11925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:21 smithi115 bash[45495]: audit 2022-04-22T21:27:21.161267+0000 mon.a (mon.0) 11925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:21 smithi115 bash[46934]: audit 2022-04-22T21:27:21.161267+0000 mon.a (mon.0) 11925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:22.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[45495]: audit 2022-04-22T21:27:21.453358+0000 mgr.y (mgr.34107) 8238 : audit [DBG] from='client.53851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[45495]: cluster 2022-04-22T21:27:21.471512+0000 mgr.y (mgr.34107) 8239 : cluster [DBG] pgmap v9179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[45495]: audit 2022-04-22T21:27:21.795711+0000 mgr.y (mgr.34107) 8240 : audit [DBG] from='client.53857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[45495]: audit 2022-04-22T21:27:22.221258+0000 mon.a (mon.0) 11926 : audit [DBG] from='client.? 172.21.15.115:0/4121949255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[46934]: audit 2022-04-22T21:27:21.453358+0000 mgr.y (mgr.34107) 8238 : audit [DBG] from='client.53851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[46934]: cluster 2022-04-22T21:27:21.471512+0000 mgr.y (mgr.34107) 8239 : cluster [DBG] pgmap v9179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[46934]: audit 2022-04-22T21:27:21.795711+0000 mgr.y (mgr.34107) 8240 : audit [DBG] from='client.53857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:22 smithi115 bash[46934]: audit 2022-04-22T21:27:22.221258+0000 mon.a (mon.0) 11926 : audit [DBG] from='client.? 172.21.15.115:0/4121949255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:22 smithi121 bash[36902]: audit 2022-04-22T21:27:21.453358+0000 mgr.y (mgr.34107) 8238 : audit [DBG] from='client.53851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:22 smithi121 bash[36902]: cluster 2022-04-22T21:27:21.471512+0000 mgr.y (mgr.34107) 8239 : cluster [DBG] pgmap v9179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:22 smithi121 bash[36902]: audit 2022-04-22T21:27:21.795711+0000 mgr.y (mgr.34107) 8240 : audit [DBG] from='client.53857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:22 smithi121 bash[36902]: audit 2022-04-22T21:27:22.221258+0000 mon.a (mon.0) 11926 : audit [DBG] from='client.? 172.21.15.115:0/4121949255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:24.429Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:24 smithi121 bash[36902]: cluster 2022-04-22T21:27:23.472691+0000 mgr.y (mgr.34107) 8241 : cluster [DBG] pgmap v9180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:24 smithi121 bash[36902]: audit 2022-04-22T21:27:24.170818+0000 mon.a (mon.0) 11927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:24 smithi115 bash[45495]: cluster 2022-04-22T21:27:23.472691+0000 mgr.y (mgr.34107) 8241 : cluster [DBG] pgmap v9180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:24 smithi115 bash[45495]: audit 2022-04-22T21:27:24.170818+0000 mon.a (mon.0) 11927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:24 smithi115 bash[46934]: cluster 2022-04-22T21:27:23.472691+0000 mgr.y (mgr.34107) 8241 : cluster [DBG] pgmap v9180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:24 smithi115 bash[46934]: audit 2022-04-22T21:27:24.170818+0000 mon.a (mon.0) 11927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:25 smithi121 bash[36902]: audit 2022-04-22T21:27:25.044893+0000 mon.a (mon.0) 11928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:25 smithi115 bash[45495]: audit 2022-04-22T21:27:25.044893+0000 mon.a (mon.0) 11928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:25 smithi115 bash[46934]: audit 2022-04-22T21:27:25.044893+0000 mon.a (mon.0) 11928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: cluster 2022-04-22T21:27:25.473919+0000 mgr.y (mgr.34107) 8242 : cluster [DBG] pgmap v9181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: audit 2022-04-22T21:27:26.164145+0000 mon.a (mon.0) 11929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: audit 2022-04-22T21:27:26.401964+0000 mon.a (mon.0) 11930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: audit 2022-04-22T21:27:26.404869+0000 mon.a (mon.0) 11931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: audit 2022-04-22T21:27:26.406493+0000 mon.a (mon.0) 11932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:26 smithi121 bash[36902]: audit 2022-04-22T21:27:26.770794+0000 mon.a (mon.0) 11933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: cluster 2022-04-22T21:27:25.473919+0000 mgr.y (mgr.34107) 8242 : cluster [DBG] pgmap v9181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: audit 2022-04-22T21:27:26.164145+0000 mon.a (mon.0) 11929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: audit 2022-04-22T21:27:26.401964+0000 mon.a (mon.0) 11930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: audit 2022-04-22T21:27:26.404869+0000 mon.a (mon.0) 11931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: audit 2022-04-22T21:27:26.406493+0000 mon.a (mon.0) 11932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:27:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[45495]: audit 2022-04-22T21:27:26.770794+0000 mon.a (mon.0) 11933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:27:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: cluster 2022-04-22T21:27:25.473919+0000 mgr.y (mgr.34107) 8242 : cluster [DBG] pgmap v9181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: audit 2022-04-22T21:27:26.164145+0000 mon.a (mon.0) 11929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: audit 2022-04-22T21:27:26.401964+0000 mon.a (mon.0) 11930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:27:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: audit 2022-04-22T21:27:26.404869+0000 mon.a (mon.0) 11931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:27:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: audit 2022-04-22T21:27:26.406493+0000 mon.a (mon.0) 11932 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:27:27.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:26 smithi115 bash[46934]: audit 2022-04-22T21:27:26.770794+0000 mon.a (mon.0) 11933 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:27:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:28 smithi121 bash[36902]: cluster 2022-04-22T21:27:27.475096+0000 mgr.y (mgr.34107) 8243 : cluster [DBG] pgmap v9182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:28 smithi115 bash[45495]: cluster 2022-04-22T21:27:27.475096+0000 mgr.y (mgr.34107) 8243 : cluster [DBG] pgmap v9182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:28 smithi115 bash[46934]: cluster 2022-04-22T21:27:27.475096+0000 mgr.y (mgr.34107) 8243 : cluster [DBG] pgmap v9182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:29 smithi121 bash[36902]: audit 2022-04-22T21:27:29.174019+0000 mon.a (mon.0) 11934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:29 smithi115 bash[45495]: audit 2022-04-22T21:27:29.174019+0000 mon.a (mon.0) 11934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:29 smithi115 bash[46934]: audit 2022-04-22T21:27:29.174019+0000 mon.a (mon.0) 11934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:30 smithi121 bash[36902]: cluster 2022-04-22T21:27:29.476319+0000 mgr.y (mgr.34107) 8244 : cluster [DBG] pgmap v9183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:31.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:30 smithi121 bash[36902]: audit 2022-04-22T21:27:30.048742+0000 mon.a (mon.0) 11935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:31.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:30 smithi115 bash[45495]: cluster 2022-04-22T21:27:29.476319+0000 mgr.y (mgr.34107) 8244 : cluster [DBG] pgmap v9183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:31.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:30 smithi115 bash[45495]: audit 2022-04-22T21:27:30.048742+0000 mon.a (mon.0) 11935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:31.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:30 smithi115 bash[46934]: cluster 2022-04-22T21:27:29.476319+0000 mgr.y (mgr.34107) 8244 : cluster [DBG] pgmap v9183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:31.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:30 smithi115 bash[46934]: audit 2022-04-22T21:27:30.048742+0000 mon.a (mon.0) 11935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:31 smithi121 bash[36902]: audit 2022-04-22T21:27:31.166983+0000 mon.a (mon.0) 11936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:31 smithi115 bash[45495]: audit 2022-04-22T21:27:31.166983+0000 mon.a (mon.0) 11936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:31 smithi115 bash[46934]: audit 2022-04-22T21:27:31.166983+0000 mon.a (mon.0) 11936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:32.826 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:32.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:32 smithi121 bash[36902]: cluster 2022-04-22T21:27:31.477539+0000 mgr.y (mgr.34107) 8245 : cluster [DBG] pgmap v9184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:32 smithi115 bash[45495]: cluster 2022-04-22T21:27:31.477539+0000 mgr.y (mgr.34107) 8245 : cluster [DBG] pgmap v9184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:32 smithi115 bash[46934]: cluster 2022-04-22T21:27:31.477539+0000 mgr.y (mgr.34107) 8245 : cluster [DBG] pgmap v9184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:34.429Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:34 smithi121 bash[36902]: cluster 2022-04-22T21:27:33.478670+0000 mgr.y (mgr.34107) 8246 : cluster [DBG] pgmap v9185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:34 smithi121 bash[36902]: audit 2022-04-22T21:27:34.176998+0000 mon.a (mon.0) 11937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:34 smithi115 bash[45495]: cluster 2022-04-22T21:27:33.478670+0000 mgr.y (mgr.34107) 8246 : cluster [DBG] pgmap v9185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:34 smithi115 bash[45495]: audit 2022-04-22T21:27:34.176998+0000 mon.a (mon.0) 11937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:34 smithi115 bash[46934]: cluster 2022-04-22T21:27:33.478670+0000 mgr.y (mgr.34107) 8246 : cluster [DBG] pgmap v9185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:34 smithi115 bash[46934]: audit 2022-04-22T21:27:34.176998+0000 mon.a (mon.0) 11937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:35 smithi121 bash[36902]: audit 2022-04-22T21:27:35.052405+0000 mon.a (mon.0) 11938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:35 smithi115 bash[45495]: audit 2022-04-22T21:27:35.052405+0000 mon.a (mon.0) 11938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:35 smithi115 bash[46934]: audit 2022-04-22T21:27:35.052405+0000 mon.a (mon.0) 11938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:36 smithi121 bash[36902]: cluster 2022-04-22T21:27:35.479795+0000 mgr.y (mgr.34107) 8247 : cluster [DBG] pgmap v9186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:36 smithi121 bash[36902]: audit 2022-04-22T21:27:36.170165+0000 mon.a (mon.0) 11939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:36 smithi115 bash[45495]: cluster 2022-04-22T21:27:35.479795+0000 mgr.y (mgr.34107) 8247 : cluster [DBG] pgmap v9186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:36 smithi115 bash[45495]: audit 2022-04-22T21:27:36.170165+0000 mon.a (mon.0) 11939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:36 smithi115 bash[46934]: cluster 2022-04-22T21:27:35.479795+0000 mgr.y (mgr.34107) 8247 : cluster [DBG] pgmap v9186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:36 smithi115 bash[46934]: audit 2022-04-22T21:27:36.170165+0000 mon.a (mon.0) 11939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:38 smithi121 bash[36902]: cluster 2022-04-22T21:27:37.481153+0000 mgr.y (mgr.34107) 8248 : cluster [DBG] pgmap v9187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:38 smithi115 bash[45495]: cluster 2022-04-22T21:27:37.481153+0000 mgr.y (mgr.34107) 8248 : cluster [DBG] pgmap v9187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:38 smithi115 bash[46934]: cluster 2022-04-22T21:27:37.481153+0000 mgr.y (mgr.34107) 8248 : cluster [DBG] pgmap v9187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:39 smithi121 bash[36902]: audit 2022-04-22T21:27:39.180244+0000 mon.a (mon.0) 11940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:39 smithi115 bash[45495]: audit 2022-04-22T21:27:39.180244+0000 mon.a (mon.0) 11940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:39 smithi115 bash[46934]: audit 2022-04-22T21:27:39.180244+0000 mon.a (mon.0) 11940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:40 smithi121 bash[36902]: cluster 2022-04-22T21:27:39.482207+0000 mgr.y (mgr.34107) 8249 : cluster [DBG] pgmap v9188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:40 smithi121 bash[36902]: audit 2022-04-22T21:27:40.056088+0000 mon.a (mon.0) 11941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:40 smithi115 bash[45495]: cluster 2022-04-22T21:27:39.482207+0000 mgr.y (mgr.34107) 8249 : cluster [DBG] pgmap v9188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:40 smithi115 bash[45495]: audit 2022-04-22T21:27:40.056088+0000 mon.a (mon.0) 11941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:40 smithi115 bash[46934]: cluster 2022-04-22T21:27:39.482207+0000 mgr.y (mgr.34107) 8249 : cluster [DBG] pgmap v9188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:40 smithi115 bash[46934]: audit 2022-04-22T21:27:40.056088+0000 mon.a (mon.0) 11941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:42.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:41 smithi121 bash[36902]: audit 2022-04-22T21:27:41.173062+0000 mon.a (mon.0) 11942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:41 smithi115 bash[45495]: audit 2022-04-22T21:27:41.173062+0000 mon.a (mon.0) 11942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:41 smithi115 bash[46934]: audit 2022-04-22T21:27:41.173062+0000 mon.a (mon.0) 11942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:42.859 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:42.557Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:42 smithi121 bash[36902]: cluster 2022-04-22T21:27:41.483205+0000 mgr.y (mgr.34107) 8250 : cluster [DBG] pgmap v9189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:42 smithi115 bash[45495]: cluster 2022-04-22T21:27:41.483205+0000 mgr.y (mgr.34107) 8250 : cluster [DBG] pgmap v9189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:42 smithi115 bash[46934]: cluster 2022-04-22T21:27:41.483205+0000 mgr.y (mgr.34107) 8250 : cluster [DBG] pgmap v9189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:44.430Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:44 smithi121 bash[36902]: cluster 2022-04-22T21:27:43.484356+0000 mgr.y (mgr.34107) 8251 : cluster [DBG] pgmap v9190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:44 smithi121 bash[36902]: audit 2022-04-22T21:27:44.183400+0000 mon.a (mon.0) 11943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:44 smithi115 bash[45495]: cluster 2022-04-22T21:27:43.484356+0000 mgr.y (mgr.34107) 8251 : cluster [DBG] pgmap v9190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:44 smithi115 bash[45495]: audit 2022-04-22T21:27:44.183400+0000 mon.a (mon.0) 11943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:44 smithi115 bash[46934]: cluster 2022-04-22T21:27:43.484356+0000 mgr.y (mgr.34107) 8251 : cluster [DBG] pgmap v9190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:44 smithi115 bash[46934]: audit 2022-04-22T21:27:44.183400+0000 mon.a (mon.0) 11943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:45 smithi121 bash[36902]: audit 2022-04-22T21:27:45.059674+0000 mon.a (mon.0) 11944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:45 smithi115 bash[45495]: audit 2022-04-22T21:27:45.059674+0000 mon.a (mon.0) 11944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:45 smithi115 bash[46934]: audit 2022-04-22T21:27:45.059674+0000 mon.a (mon.0) 11944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:46 smithi121 bash[36902]: cluster 2022-04-22T21:27:45.485514+0000 mgr.y (mgr.34107) 8252 : cluster [DBG] pgmap v9191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:46 smithi121 bash[36902]: audit 2022-04-22T21:27:46.175902+0000 mon.a (mon.0) 11945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:46 smithi115 bash[45495]: cluster 2022-04-22T21:27:45.485514+0000 mgr.y (mgr.34107) 8252 : cluster [DBG] pgmap v9191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:46 smithi115 bash[45495]: audit 2022-04-22T21:27:46.175902+0000 mon.a (mon.0) 11945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:46 smithi115 bash[46934]: cluster 2022-04-22T21:27:45.485514+0000 mgr.y (mgr.34107) 8252 : cluster [DBG] pgmap v9191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:46 smithi115 bash[46934]: audit 2022-04-22T21:27:46.175902+0000 mon.a (mon.0) 11945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:49.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:48 smithi121 bash[36902]: cluster 2022-04-22T21:27:47.486673+0000 mgr.y (mgr.34107) 8253 : cluster [DBG] pgmap v9192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:48 smithi115 bash[45495]: cluster 2022-04-22T21:27:47.486673+0000 mgr.y (mgr.34107) 8253 : cluster [DBG] pgmap v9192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:48 smithi115 bash[46934]: cluster 2022-04-22T21:27:47.486673+0000 mgr.y (mgr.34107) 8253 : cluster [DBG] pgmap v9192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:50.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:49 smithi121 bash[36902]: audit 2022-04-22T21:27:49.186564+0000 mon.a (mon.0) 11946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:49 smithi115 bash[45495]: audit 2022-04-22T21:27:49.186564+0000 mon.a (mon.0) 11946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:49 smithi115 bash[46934]: audit 2022-04-22T21:27:49.186564+0000 mon.a (mon.0) 11946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:50 smithi121 bash[36902]: cluster 2022-04-22T21:27:49.487870+0000 mgr.y (mgr.34107) 8254 : cluster [DBG] pgmap v9193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:50 smithi121 bash[36902]: audit 2022-04-22T21:27:50.063292+0000 mon.a (mon.0) 11947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:50 smithi115 bash[45495]: cluster 2022-04-22T21:27:49.487870+0000 mgr.y (mgr.34107) 8254 : cluster [DBG] pgmap v9193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:50 smithi115 bash[45495]: audit 2022-04-22T21:27:50.063292+0000 mon.a (mon.0) 11947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:50 smithi115 bash[46934]: cluster 2022-04-22T21:27:49.487870+0000 mgr.y (mgr.34107) 8254 : cluster [DBG] pgmap v9193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:50 smithi115 bash[46934]: audit 2022-04-22T21:27:50.063292+0000 mon.a (mon.0) 11947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:51 smithi121 bash[36902]: audit 2022-04-22T21:27:51.178944+0000 mon.a (mon.0) 11948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:51 smithi115 bash[46934]: audit 2022-04-22T21:27:51.178944+0000 mon.a (mon.0) 11948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:51 smithi115 bash[45495]: audit 2022-04-22T21:27:51.178944+0000 mon.a (mon.0) 11948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:52.598 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:27:52.892 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:52.557Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:52.947 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:27:52.947 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:27:52.947 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:27:52.947 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:27:52.948 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:27:52.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:27:52.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:27:52.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3h) 3m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:27:52.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 4m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:27:52.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:27:52.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:27:52.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 4m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:27:52.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:27:52.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:27:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:27:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:27:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 4m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:27:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:27:52.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 4m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:27:52.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 4m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:27:52.951 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 4m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:27:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:52 smithi121 bash[36902]: cluster 2022-04-22T21:27:51.488964+0000 mgr.y (mgr.34107) 8255 : cluster [DBG] pgmap v9194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:53.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:52 smithi115 bash[45495]: cluster 2022-04-22T21:27:51.488964+0000 mgr.y (mgr.34107) 8255 : cluster [DBG] pgmap v9194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:53.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:52 smithi115 bash[46934]: cluster 2022-04-22T21:27:51.488964+0000 mgr.y (mgr.34107) 8255 : cluster [DBG] pgmap v9194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:53.343 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:27:53.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:27:53.345 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:27:54.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:53 smithi121 bash[36902]: audit 2022-04-22T21:27:52.579175+0000 mgr.y (mgr.34107) 8256 : audit [DBG] from='client.53869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:53 smithi121 bash[36902]: audit 2022-04-22T21:27:52.923421+0000 mgr.y (mgr.34107) 8257 : audit [DBG] from='client.53875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:53 smithi121 bash[36902]: audit 2022-04-22T21:27:53.339681+0000 mon.c (mon.1) 296 : audit [DBG] from='client.? 172.21.15.115:0/1123849008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[45495]: audit 2022-04-22T21:27:52.579175+0000 mgr.y (mgr.34107) 8256 : audit [DBG] from='client.53869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[45495]: audit 2022-04-22T21:27:52.923421+0000 mgr.y (mgr.34107) 8257 : audit [DBG] from='client.53875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[45495]: audit 2022-04-22T21:27:53.339681+0000 mon.c (mon.1) 296 : audit [DBG] from='client.? 172.21.15.115:0/1123849008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[46934]: audit 2022-04-22T21:27:52.579175+0000 mgr.y (mgr.34107) 8256 : audit [DBG] from='client.53869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[46934]: audit 2022-04-22T21:27:52.923421+0000 mgr.y (mgr.34107) 8257 : audit [DBG] from='client.53875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:27:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:53 smithi115 bash[46934]: audit 2022-04-22T21:27:53.339681+0000 mon.c (mon.1) 296 : audit [DBG] from='client.? 172.21.15.115:0/1123849008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:27:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:27:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:27:54.429Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:27:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:54 smithi121 bash[36902]: cluster 2022-04-22T21:27:53.490072+0000 mgr.y (mgr.34107) 8258 : cluster [DBG] pgmap v9195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:54 smithi121 bash[36902]: audit 2022-04-22T21:27:54.189884+0000 mon.a (mon.0) 11949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:54 smithi115 bash[45495]: cluster 2022-04-22T21:27:53.490072+0000 mgr.y (mgr.34107) 8258 : cluster [DBG] pgmap v9195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:54 smithi115 bash[45495]: audit 2022-04-22T21:27:54.189884+0000 mon.a (mon.0) 11949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:54 smithi115 bash[46934]: cluster 2022-04-22T21:27:53.490072+0000 mgr.y (mgr.34107) 8258 : cluster [DBG] pgmap v9195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:54 smithi115 bash[46934]: audit 2022-04-22T21:27:54.189884+0000 mon.a (mon.0) 11949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:27:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:55 smithi121 bash[36902]: audit 2022-04-22T21:27:55.066792+0000 mon.a (mon.0) 11950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:55 smithi115 bash[45495]: audit 2022-04-22T21:27:55.066792+0000 mon.a (mon.0) 11950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:55 smithi115 bash[46934]: audit 2022-04-22T21:27:55.066792+0000 mon.a (mon.0) 11950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:27:57.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:56 smithi121 bash[36902]: cluster 2022-04-22T21:27:55.491138+0000 mgr.y (mgr.34107) 8259 : cluster [DBG] pgmap v9196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:57.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:56 smithi121 bash[36902]: audit 2022-04-22T21:27:56.181812+0000 mon.a (mon.0) 11951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:56 smithi115 bash[45495]: cluster 2022-04-22T21:27:55.491138+0000 mgr.y (mgr.34107) 8259 : cluster [DBG] pgmap v9196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:57.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:56 smithi115 bash[45495]: audit 2022-04-22T21:27:56.181812+0000 mon.a (mon.0) 11951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:56 smithi115 bash[46934]: cluster 2022-04-22T21:27:55.491138+0000 mgr.y (mgr.34107) 8259 : cluster [DBG] pgmap v9196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:57.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:56 smithi115 bash[46934]: audit 2022-04-22T21:27:56.181812+0000 mon.a (mon.0) 11951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:27:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:58 smithi121 bash[36902]: cluster 2022-04-22T21:27:57.492231+0000 mgr.y (mgr.34107) 8260 : cluster [DBG] pgmap v9197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:58 smithi115 bash[45495]: cluster 2022-04-22T21:27:57.492231+0000 mgr.y (mgr.34107) 8260 : cluster [DBG] pgmap v9197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:27:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:58 smithi115 bash[46934]: cluster 2022-04-22T21:27:57.492231+0000 mgr.y (mgr.34107) 8260 : cluster [DBG] pgmap v9197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:27:59 smithi121 bash[36902]: audit 2022-04-22T21:27:59.192895+0000 mon.a (mon.0) 11952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:27:59 smithi115 bash[45495]: audit 2022-04-22T21:27:59.192895+0000 mon.a (mon.0) 11952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:27:59 smithi115 bash[46934]: audit 2022-04-22T21:27:59.192895+0000 mon.a (mon.0) 11952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:00 smithi121 bash[36902]: cluster 2022-04-22T21:27:59.493174+0000 mgr.y (mgr.34107) 8261 : cluster [DBG] pgmap v9198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:00 smithi121 bash[36902]: audit 2022-04-22T21:28:00.070374+0000 mon.a (mon.0) 11953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:00 smithi115 bash[45495]: cluster 2022-04-22T21:27:59.493174+0000 mgr.y (mgr.34107) 8261 : cluster [DBG] pgmap v9198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:00 smithi115 bash[45495]: audit 2022-04-22T21:28:00.070374+0000 mon.a (mon.0) 11953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:00 smithi115 bash[46934]: cluster 2022-04-22T21:27:59.493174+0000 mgr.y (mgr.34107) 8261 : cluster [DBG] pgmap v9198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:00 smithi115 bash[46934]: audit 2022-04-22T21:28:00.070374+0000 mon.a (mon.0) 11953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:02.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:01 smithi121 bash[36902]: audit 2022-04-22T21:28:01.184807+0000 mon.a (mon.0) 11954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:01 smithi115 bash[45495]: audit 2022-04-22T21:28:01.184807+0000 mon.a (mon.0) 11954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:01 smithi115 bash[46934]: audit 2022-04-22T21:28:01.184807+0000 mon.a (mon.0) 11954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:02.931 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:02.558Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:02 smithi121 bash[36902]: cluster 2022-04-22T21:28:01.494195+0000 mgr.y (mgr.34107) 8262 : cluster [DBG] pgmap v9199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:02 smithi115 bash[45495]: cluster 2022-04-22T21:28:01.494195+0000 mgr.y (mgr.34107) 8262 : cluster [DBG] pgmap v9199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:02 smithi115 bash[46934]: cluster 2022-04-22T21:28:01.494195+0000 mgr.y (mgr.34107) 8262 : cluster [DBG] pgmap v9199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:04.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:03 smithi121 bash[36902]: cluster 2022-04-22T21:28:03.495284+0000 mgr.y (mgr.34107) 8263 : cluster [DBG] pgmap v9200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:03 smithi115 bash[45495]: cluster 2022-04-22T21:28:03.495284+0000 mgr.y (mgr.34107) 8263 : cluster [DBG] pgmap v9200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:03 smithi115 bash[46934]: cluster 2022-04-22T21:28:03.495284+0000 mgr.y (mgr.34107) 8263 : cluster [DBG] pgmap v9200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:04.430Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:04 smithi121 bash[36902]: audit 2022-04-22T21:28:04.195918+0000 mon.a (mon.0) 11955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:04 smithi115 bash[45495]: audit 2022-04-22T21:28:04.195918+0000 mon.a (mon.0) 11955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:05.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:04 smithi115 bash[46934]: audit 2022-04-22T21:28:04.195918+0000 mon.a (mon.0) 11955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:05 smithi121 bash[36902]: audit 2022-04-22T21:28:05.074177+0000 mon.a (mon.0) 11956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:05 smithi121 bash[36902]: cluster 2022-04-22T21:28:05.496354+0000 mgr.y (mgr.34107) 8264 : cluster [DBG] pgmap v9201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:05 smithi115 bash[45495]: audit 2022-04-22T21:28:05.074177+0000 mon.a (mon.0) 11956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:06.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:05 smithi115 bash[45495]: cluster 2022-04-22T21:28:05.496354+0000 mgr.y (mgr.34107) 8264 : cluster [DBG] pgmap v9201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:05 smithi115 bash[46934]: audit 2022-04-22T21:28:05.074177+0000 mon.a (mon.0) 11956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:06.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:05 smithi115 bash[46934]: cluster 2022-04-22T21:28:05.496354+0000 mgr.y (mgr.34107) 8264 : cluster [DBG] pgmap v9201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:07.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:06 smithi121 bash[36902]: audit 2022-04-22T21:28:06.188069+0000 mon.a (mon.0) 11957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:06 smithi115 bash[46934]: audit 2022-04-22T21:28:06.188069+0000 mon.a (mon.0) 11957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:06 smithi115 bash[45495]: audit 2022-04-22T21:28:06.188069+0000 mon.a (mon.0) 11957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:08 smithi121 bash[36902]: cluster 2022-04-22T21:28:07.497404+0000 mgr.y (mgr.34107) 8265 : cluster [DBG] pgmap v9202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:07 smithi115 bash[45495]: cluster 2022-04-22T21:28:07.497404+0000 mgr.y (mgr.34107) 8265 : cluster [DBG] pgmap v9202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:07 smithi115 bash[46934]: cluster 2022-04-22T21:28:07.497404+0000 mgr.y (mgr.34107) 8265 : cluster [DBG] pgmap v9202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:09.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:09 smithi121 bash[36902]: audit 2022-04-22T21:28:09.198792+0000 mon.a (mon.0) 11958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:09 smithi115 bash[45495]: audit 2022-04-22T21:28:09.198792+0000 mon.a (mon.0) 11958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:09 smithi115 bash[46934]: audit 2022-04-22T21:28:09.198792+0000 mon.a (mon.0) 11958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:10 smithi115 bash[45495]: cluster 2022-04-22T21:28:09.498483+0000 mgr.y (mgr.34107) 8266 : cluster [DBG] pgmap v9203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:10 smithi115 bash[45495]: audit 2022-04-22T21:28:10.077869+0000 mon.a (mon.0) 11959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:10 smithi115 bash[46934]: cluster 2022-04-22T21:28:09.498483+0000 mgr.y (mgr.34107) 8266 : cluster [DBG] pgmap v9203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:10 smithi115 bash[46934]: audit 2022-04-22T21:28:10.077869+0000 mon.a (mon.0) 11959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:10 smithi121 bash[36902]: cluster 2022-04-22T21:28:09.498483+0000 mgr.y (mgr.34107) 8266 : cluster [DBG] pgmap v9203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:10 smithi121 bash[36902]: audit 2022-04-22T21:28:10.077869+0000 mon.a (mon.0) 11959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:11.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:11 smithi121 bash[36902]: audit 2022-04-22T21:28:11.190995+0000 mon.a (mon.0) 11960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:11 smithi115 bash[45495]: audit 2022-04-22T21:28:11.190995+0000 mon.a (mon.0) 11960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:11 smithi115 bash[46934]: audit 2022-04-22T21:28:11.190995+0000 mon.a (mon.0) 11960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:12.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:12 smithi121 bash[36902]: cluster 2022-04-22T21:28:11.499609+0000 mgr.y (mgr.34107) 8267 : cluster [DBG] pgmap v9204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:12.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:12 smithi115 bash[46934]: cluster 2022-04-22T21:28:11.499609+0000 mgr.y (mgr.34107) 8267 : cluster [DBG] pgmap v9204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:12.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:12.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:12 smithi115 bash[45495]: cluster 2022-04-22T21:28:11.499609+0000 mgr.y (mgr.34107) 8267 : cluster [DBG] pgmap v9204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:14.430Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:14 smithi115 bash[45495]: cluster 2022-04-22T21:28:13.500662+0000 mgr.y (mgr.34107) 8268 : cluster [DBG] pgmap v9205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:14 smithi115 bash[45495]: audit 2022-04-22T21:28:14.202704+0000 mon.a (mon.0) 11961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:14 smithi115 bash[46934]: cluster 2022-04-22T21:28:13.500662+0000 mgr.y (mgr.34107) 8268 : cluster [DBG] pgmap v9205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:14 smithi115 bash[46934]: audit 2022-04-22T21:28:14.202704+0000 mon.a (mon.0) 11961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:15.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:14 smithi121 bash[36902]: cluster 2022-04-22T21:28:13.500662+0000 mgr.y (mgr.34107) 8268 : cluster [DBG] pgmap v9205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:14 smithi121 bash[36902]: audit 2022-04-22T21:28:14.202704+0000 mon.a (mon.0) 11961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:16.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:15 smithi121 bash[36902]: audit 2022-04-22T21:28:15.081588+0000 mon.a (mon.0) 11962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:15 smithi115 bash[45495]: audit 2022-04-22T21:28:15.081588+0000 mon.a (mon.0) 11962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:15 smithi115 bash[46934]: audit 2022-04-22T21:28:15.081588+0000 mon.a (mon.0) 11962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:17.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:16 smithi121 bash[36902]: cluster 2022-04-22T21:28:15.501734+0000 mgr.y (mgr.34107) 8269 : cluster [DBG] pgmap v9206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:16 smithi121 bash[36902]: audit 2022-04-22T21:28:16.051663+0000 mon.a (mon.0) 11963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:28:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:16 smithi121 bash[36902]: audit 2022-04-22T21:28:16.052175+0000 mon.a (mon.0) 11964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:28:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:16 smithi121 bash[36902]: audit 2022-04-22T21:28:16.193976+0000 mon.a (mon.0) 11965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[45495]: cluster 2022-04-22T21:28:15.501734+0000 mgr.y (mgr.34107) 8269 : cluster [DBG] pgmap v9206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[45495]: audit 2022-04-22T21:28:16.051663+0000 mon.a (mon.0) 11963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:28:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[45495]: audit 2022-04-22T21:28:16.052175+0000 mon.a (mon.0) 11964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:28:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[45495]: audit 2022-04-22T21:28:16.193976+0000 mon.a (mon.0) 11965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[46934]: cluster 2022-04-22T21:28:15.501734+0000 mgr.y (mgr.34107) 8269 : cluster [DBG] pgmap v9206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[46934]: audit 2022-04-22T21:28:16.051663+0000 mon.a (mon.0) 11963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:28:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[46934]: audit 2022-04-22T21:28:16.052175+0000 mon.a (mon.0) 11964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:28:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:16 smithi115 bash[46934]: audit 2022-04-22T21:28:16.193976+0000 mon.a (mon.0) 11965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:19.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:18 smithi121 bash[36902]: cluster 2022-04-22T21:28:17.502956+0000 mgr.y (mgr.34107) 8270 : cluster [DBG] pgmap v9207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:18 smithi115 bash[45495]: cluster 2022-04-22T21:28:17.502956+0000 mgr.y (mgr.34107) 8270 : cluster [DBG] pgmap v9207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:18 smithi115 bash[46934]: cluster 2022-04-22T21:28:17.502956+0000 mgr.y (mgr.34107) 8270 : cluster [DBG] pgmap v9207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:19 smithi121 bash[36902]: audit 2022-04-22T21:28:19.205925+0000 mon.a (mon.0) 11966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:19 smithi115 bash[45495]: audit 2022-04-22T21:28:19.205925+0000 mon.a (mon.0) 11966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:19 smithi115 bash[46934]: audit 2022-04-22T21:28:19.205925+0000 mon.a (mon.0) 11966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:20 smithi121 bash[36902]: cluster 2022-04-22T21:28:19.504164+0000 mgr.y (mgr.34107) 8271 : cluster [DBG] pgmap v9208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:20 smithi121 bash[36902]: audit 2022-04-22T21:28:20.085405+0000 mon.a (mon.0) 11967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:20 smithi115 bash[45495]: cluster 2022-04-22T21:28:19.504164+0000 mgr.y (mgr.34107) 8271 : cluster [DBG] pgmap v9208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:20 smithi115 bash[45495]: audit 2022-04-22T21:28:20.085405+0000 mon.a (mon.0) 11967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:20 smithi115 bash[46934]: cluster 2022-04-22T21:28:19.504164+0000 mgr.y (mgr.34107) 8271 : cluster [DBG] pgmap v9208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:20 smithi115 bash[46934]: audit 2022-04-22T21:28:20.085405+0000 mon.a (mon.0) 11967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:21 smithi121 bash[36902]: audit 2022-04-22T21:28:21.197036+0000 mon.a (mon.0) 11968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:21 smithi115 bash[45495]: audit 2022-04-22T21:28:21.197036+0000 mon.a (mon.0) 11968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:21 smithi115 bash[46934]: audit 2022-04-22T21:28:21.197036+0000 mon.a (mon.0) 11968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:22 smithi121 bash[36902]: cluster 2022-04-22T21:28:21.505301+0000 mgr.y (mgr.34107) 8272 : cluster [DBG] pgmap v9209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:22.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:22 smithi115 bash[45495]: cluster 2022-04-22T21:28:21.505301+0000 mgr.y (mgr.34107) 8272 : cluster [DBG] pgmap v9209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:22 smithi115 bash[46934]: cluster 2022-04-22T21:28:21.505301+0000 mgr.y (mgr.34107) 8272 : cluster [DBG] pgmap v9209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:23.720 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 3m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:28:24.067 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3h) 3m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 3m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 3m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 3m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 3m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 3m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:28:24.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:28:24.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:28:24.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:28:24.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:28:24.069 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:28:24.469 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:28:24.469 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:28:24.469 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:28:24.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:24.469 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:24.470 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:28:24.471 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:28:24.471 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:28:24.471 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:28:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:24.429Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[45495]: cluster 2022-04-22T21:28:23.506030+0000 mgr.y (mgr.34107) 8273 : cluster [DBG] pgmap v9210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[45495]: audit 2022-04-22T21:28:23.700642+0000 mgr.y (mgr.34107) 8274 : audit [DBG] from='client.43956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[45495]: audit 2022-04-22T21:28:24.208633+0000 mon.a (mon.0) 11969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[45495]: audit 2022-04-22T21:28:24.465199+0000 mon.a (mon.0) 11970 : audit [DBG] from='client.? 172.21.15.115:0/3800448054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[46934]: cluster 2022-04-22T21:28:23.506030+0000 mgr.y (mgr.34107) 8273 : cluster [DBG] pgmap v9210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[46934]: audit 2022-04-22T21:28:23.700642+0000 mgr.y (mgr.34107) 8274 : audit [DBG] from='client.43956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[46934]: audit 2022-04-22T21:28:24.208633+0000 mon.a (mon.0) 11969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:24 smithi115 bash[46934]: audit 2022-04-22T21:28:24.465199+0000 mon.a (mon.0) 11970 : audit [DBG] from='client.? 172.21.15.115:0/3800448054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:24 smithi121 bash[36902]: cluster 2022-04-22T21:28:23.506030+0000 mgr.y (mgr.34107) 8273 : cluster [DBG] pgmap v9210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:24 smithi121 bash[36902]: audit 2022-04-22T21:28:23.700642+0000 mgr.y (mgr.34107) 8274 : audit [DBG] from='client.43956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:24 smithi121 bash[36902]: audit 2022-04-22T21:28:24.208633+0000 mon.a (mon.0) 11969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:24 smithi121 bash[36902]: audit 2022-04-22T21:28:24.465199+0000 mon.a (mon.0) 11970 : audit [DBG] from='client.? 172.21.15.115:0/3800448054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:25 smithi121 bash[36902]: audit 2022-04-22T21:28:24.042725+0000 mgr.y (mgr.34107) 8275 : audit [DBG] from='client.43962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:25 smithi121 bash[36902]: audit 2022-04-22T21:28:25.089035+0000 mon.a (mon.0) 11971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:25 smithi115 bash[45495]: audit 2022-04-22T21:28:24.042725+0000 mgr.y (mgr.34107) 8275 : audit [DBG] from='client.43962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:25 smithi115 bash[45495]: audit 2022-04-22T21:28:25.089035+0000 mon.a (mon.0) 11971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:25 smithi115 bash[46934]: audit 2022-04-22T21:28:24.042725+0000 mgr.y (mgr.34107) 8275 : audit [DBG] from='client.43962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:25 smithi115 bash[46934]: audit 2022-04-22T21:28:25.089035+0000 mon.a (mon.0) 11971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:26.858 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:26 smithi121 bash[36902]: cluster 2022-04-22T21:28:25.507029+0000 mgr.y (mgr.34107) 8276 : cluster [DBG] pgmap v9211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:26.858 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:26 smithi121 bash[36902]: audit 2022-04-22T21:28:26.200272+0000 mon.a (mon.0) 11972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:26 smithi115 bash[46934]: cluster 2022-04-22T21:28:25.507029+0000 mgr.y (mgr.34107) 8276 : cluster [DBG] pgmap v9211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:26 smithi115 bash[46934]: audit 2022-04-22T21:28:26.200272+0000 mon.a (mon.0) 11972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:26.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:26 smithi115 bash[45495]: cluster 2022-04-22T21:28:25.507029+0000 mgr.y (mgr.34107) 8276 : cluster [DBG] pgmap v9211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:26.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:26 smithi115 bash[45495]: audit 2022-04-22T21:28:26.200272+0000 mon.a (mon.0) 11972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:27 smithi121 bash[36902]: audit 2022-04-22T21:28:26.779538+0000 mon.a (mon.0) 11973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:28:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:27 smithi121 bash[36902]: audit 2022-04-22T21:28:26.782858+0000 mon.a (mon.0) 11974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:28:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:27 smithi121 bash[36902]: audit 2022-04-22T21:28:26.784811+0000 mon.a (mon.0) 11975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:28:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:27 smithi121 bash[36902]: audit 2022-04-22T21:28:27.175706+0000 mon.a (mon.0) 11976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:28:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[45495]: audit 2022-04-22T21:28:26.779538+0000 mon.a (mon.0) 11973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:28:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[45495]: audit 2022-04-22T21:28:26.782858+0000 mon.a (mon.0) 11974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:28:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[45495]: audit 2022-04-22T21:28:26.784811+0000 mon.a (mon.0) 11975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:28:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[45495]: audit 2022-04-22T21:28:27.175706+0000 mon.a (mon.0) 11976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:28:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[46934]: audit 2022-04-22T21:28:26.779538+0000 mon.a (mon.0) 11973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:28:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[46934]: audit 2022-04-22T21:28:26.782858+0000 mon.a (mon.0) 11974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:28:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[46934]: audit 2022-04-22T21:28:26.784811+0000 mon.a (mon.0) 11975 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:28:28.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:27 smithi115 bash[46934]: audit 2022-04-22T21:28:27.175706+0000 mon.a (mon.0) 11976 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:28:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:28 smithi121 bash[36902]: cluster 2022-04-22T21:28:27.508154+0000 mgr.y (mgr.34107) 8277 : cluster [DBG] pgmap v9212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:28 smithi115 bash[45495]: cluster 2022-04-22T21:28:27.508154+0000 mgr.y (mgr.34107) 8277 : cluster [DBG] pgmap v9212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:28 smithi115 bash[46934]: cluster 2022-04-22T21:28:27.508154+0000 mgr.y (mgr.34107) 8277 : cluster [DBG] pgmap v9212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:29 smithi121 bash[36902]: audit 2022-04-22T21:28:29.211830+0000 mon.a (mon.0) 11977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:29 smithi115 bash[45495]: audit 2022-04-22T21:28:29.211830+0000 mon.a (mon.0) 11977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:29 smithi115 bash[46934]: audit 2022-04-22T21:28:29.211830+0000 mon.a (mon.0) 11977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:30 smithi121 bash[36902]: cluster 2022-04-22T21:28:29.509208+0000 mgr.y (mgr.34107) 8278 : cluster [DBG] pgmap v9213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:30 smithi121 bash[36902]: audit 2022-04-22T21:28:30.092525+0000 mon.a (mon.0) 11978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:30 smithi115 bash[45495]: cluster 2022-04-22T21:28:29.509208+0000 mgr.y (mgr.34107) 8278 : cluster [DBG] pgmap v9213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:30 smithi115 bash[45495]: audit 2022-04-22T21:28:30.092525+0000 mon.a (mon.0) 11978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:30 smithi115 bash[46934]: cluster 2022-04-22T21:28:29.509208+0000 mgr.y (mgr.34107) 8278 : cluster [DBG] pgmap v9213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:30 smithi115 bash[46934]: audit 2022-04-22T21:28:30.092525+0000 mon.a (mon.0) 11978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:31 smithi121 bash[36902]: audit 2022-04-22T21:28:31.203277+0000 mon.a (mon.0) 11979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:31 smithi115 bash[45495]: audit 2022-04-22T21:28:31.203277+0000 mon.a (mon.0) 11979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:31 smithi115 bash[46934]: audit 2022-04-22T21:28:31.203277+0000 mon.a (mon.0) 11979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:33.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:32 smithi121 bash[36902]: cluster 2022-04-22T21:28:31.510275+0000 mgr.y (mgr.34107) 8279 : cluster [DBG] pgmap v9214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:33.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:32 smithi115 bash[45495]: cluster 2022-04-22T21:28:31.510275+0000 mgr.y (mgr.34107) 8279 : cluster [DBG] pgmap v9214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:32.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:32 smithi115 bash[46934]: cluster 2022-04-22T21:28:31.510275+0000 mgr.y (mgr.34107) 8279 : cluster [DBG] pgmap v9214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:34.432Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:34 smithi115 bash[45495]: cluster 2022-04-22T21:28:33.511414+0000 mgr.y (mgr.34107) 8280 : cluster [DBG] pgmap v9215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:34 smithi115 bash[45495]: audit 2022-04-22T21:28:34.214918+0000 mon.a (mon.0) 11980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:34 smithi115 bash[46934]: cluster 2022-04-22T21:28:33.511414+0000 mgr.y (mgr.34107) 8280 : cluster [DBG] pgmap v9215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:34.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:34 smithi115 bash[46934]: audit 2022-04-22T21:28:34.214918+0000 mon.a (mon.0) 11980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:34 smithi121 bash[36902]: cluster 2022-04-22T21:28:33.511414+0000 mgr.y (mgr.34107) 8280 : cluster [DBG] pgmap v9215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:34 smithi121 bash[36902]: audit 2022-04-22T21:28:34.214918+0000 mon.a (mon.0) 11980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:35 smithi121 bash[36902]: audit 2022-04-22T21:28:35.096258+0000 mon.a (mon.0) 11981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:36.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:35 smithi115 bash[45495]: audit 2022-04-22T21:28:35.096258+0000 mon.a (mon.0) 11981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:36.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:35 smithi115 bash[46934]: audit 2022-04-22T21:28:35.096258+0000 mon.a (mon.0) 11981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:36 smithi121 bash[36902]: cluster 2022-04-22T21:28:35.512440+0000 mgr.y (mgr.34107) 8281 : cluster [DBG] pgmap v9216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:36 smithi121 bash[36902]: audit 2022-04-22T21:28:36.206447+0000 mon.a (mon.0) 11982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:36 smithi115 bash[45495]: cluster 2022-04-22T21:28:35.512440+0000 mgr.y (mgr.34107) 8281 : cluster [DBG] pgmap v9216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:36 smithi115 bash[45495]: audit 2022-04-22T21:28:36.206447+0000 mon.a (mon.0) 11982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:36 smithi115 bash[46934]: cluster 2022-04-22T21:28:35.512440+0000 mgr.y (mgr.34107) 8281 : cluster [DBG] pgmap v9216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:36 smithi115 bash[46934]: audit 2022-04-22T21:28:36.206447+0000 mon.a (mon.0) 11982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:39.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:38 smithi121 bash[36902]: cluster 2022-04-22T21:28:37.513600+0000 mgr.y (mgr.34107) 8282 : cluster [DBG] pgmap v9217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:38 smithi115 bash[46934]: cluster 2022-04-22T21:28:37.513600+0000 mgr.y (mgr.34107) 8282 : cluster [DBG] pgmap v9217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:38 smithi115 bash[45495]: cluster 2022-04-22T21:28:37.513600+0000 mgr.y (mgr.34107) 8282 : cluster [DBG] pgmap v9217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:39 smithi121 bash[36902]: audit 2022-04-22T21:28:39.217992+0000 mon.a (mon.0) 11983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:39 smithi115 bash[46934]: audit 2022-04-22T21:28:39.217992+0000 mon.a (mon.0) 11983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:39 smithi115 bash[45495]: audit 2022-04-22T21:28:39.217992+0000 mon.a (mon.0) 11983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:41.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:40 smithi121 bash[36902]: cluster 2022-04-22T21:28:39.514614+0000 mgr.y (mgr.34107) 8283 : cluster [DBG] pgmap v9218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:40 smithi121 bash[36902]: audit 2022-04-22T21:28:40.100067+0000 mon.a (mon.0) 11984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:40 smithi115 bash[46934]: cluster 2022-04-22T21:28:39.514614+0000 mgr.y (mgr.34107) 8283 : cluster [DBG] pgmap v9218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:41.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:40 smithi115 bash[46934]: audit 2022-04-22T21:28:40.100067+0000 mon.a (mon.0) 11984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:41.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:40 smithi115 bash[45495]: cluster 2022-04-22T21:28:39.514614+0000 mgr.y (mgr.34107) 8283 : cluster [DBG] pgmap v9218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:41.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:40 smithi115 bash[45495]: audit 2022-04-22T21:28:40.100067+0000 mon.a (mon.0) 11984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:41 smithi121 bash[36902]: audit 2022-04-22T21:28:41.209534+0000 mon.a (mon.0) 11985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:41 smithi115 bash[45495]: audit 2022-04-22T21:28:41.209534+0000 mon.a (mon.0) 11985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:41 smithi115 bash[46934]: audit 2022-04-22T21:28:41.209534+0000 mon.a (mon.0) 11985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:43.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:42 smithi121 bash[36902]: cluster 2022-04-22T21:28:41.515695+0000 mgr.y (mgr.34107) 8284 : cluster [DBG] pgmap v9219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:42 smithi115 bash[45495]: cluster 2022-04-22T21:28:41.515695+0000 mgr.y (mgr.34107) 8284 : cluster [DBG] pgmap v9219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:42.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:42 smithi115 bash[46934]: cluster 2022-04-22T21:28:41.515695+0000 mgr.y (mgr.34107) 8284 : cluster [DBG] pgmap v9219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:44.432Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:44 smithi115 bash[45495]: cluster 2022-04-22T21:28:43.516846+0000 mgr.y (mgr.34107) 8285 : cluster [DBG] pgmap v9220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:44 smithi115 bash[45495]: audit 2022-04-22T21:28:44.221021+0000 mon.a (mon.0) 11986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:44 smithi115 bash[46934]: cluster 2022-04-22T21:28:43.516846+0000 mgr.y (mgr.34107) 8285 : cluster [DBG] pgmap v9220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:44 smithi115 bash[46934]: audit 2022-04-22T21:28:44.221021+0000 mon.a (mon.0) 11986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:44 smithi121 bash[36902]: cluster 2022-04-22T21:28:43.516846+0000 mgr.y (mgr.34107) 8285 : cluster [DBG] pgmap v9220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:44 smithi121 bash[36902]: audit 2022-04-22T21:28:44.221021+0000 mon.a (mon.0) 11986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:45 smithi121 bash[36902]: audit 2022-04-22T21:28:45.103687+0000 mon.a (mon.0) 11987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:45 smithi115 bash[45495]: audit 2022-04-22T21:28:45.103687+0000 mon.a (mon.0) 11987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:45 smithi115 bash[46934]: audit 2022-04-22T21:28:45.103687+0000 mon.a (mon.0) 11987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:46 smithi121 bash[36902]: cluster 2022-04-22T21:28:45.517900+0000 mgr.y (mgr.34107) 8286 : cluster [DBG] pgmap v9221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:46 smithi121 bash[36902]: audit 2022-04-22T21:28:46.212697+0000 mon.a (mon.0) 11988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:46 smithi115 bash[45495]: cluster 2022-04-22T21:28:45.517900+0000 mgr.y (mgr.34107) 8286 : cluster [DBG] pgmap v9221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:46 smithi115 bash[45495]: audit 2022-04-22T21:28:46.212697+0000 mon.a (mon.0) 11988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:46 smithi115 bash[46934]: cluster 2022-04-22T21:28:45.517900+0000 mgr.y (mgr.34107) 8286 : cluster [DBG] pgmap v9221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:46 smithi115 bash[46934]: audit 2022-04-22T21:28:46.212697+0000 mon.a (mon.0) 11988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:48 smithi121 bash[36902]: cluster 2022-04-22T21:28:47.518953+0000 mgr.y (mgr.34107) 8287 : cluster [DBG] pgmap v9222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:48 smithi115 bash[45495]: cluster 2022-04-22T21:28:47.518953+0000 mgr.y (mgr.34107) 8287 : cluster [DBG] pgmap v9222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:48 smithi115 bash[46934]: cluster 2022-04-22T21:28:47.518953+0000 mgr.y (mgr.34107) 8287 : cluster [DBG] pgmap v9222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:49 smithi121 bash[36902]: audit 2022-04-22T21:28:49.224338+0000 mon.a (mon.0) 11989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:49 smithi115 bash[45495]: audit 2022-04-22T21:28:49.224338+0000 mon.a (mon.0) 11989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:49 smithi115 bash[46934]: audit 2022-04-22T21:28:49.224338+0000 mon.a (mon.0) 11989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:50 smithi121 bash[36902]: cluster 2022-04-22T21:28:49.520024+0000 mgr.y (mgr.34107) 8288 : cluster [DBG] pgmap v9223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:50 smithi121 bash[36902]: audit 2022-04-22T21:28:50.107395+0000 mon.a (mon.0) 11990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:50 smithi115 bash[45495]: cluster 2022-04-22T21:28:49.520024+0000 mgr.y (mgr.34107) 8288 : cluster [DBG] pgmap v9223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:50 smithi115 bash[45495]: audit 2022-04-22T21:28:50.107395+0000 mon.a (mon.0) 11990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:50 smithi115 bash[46934]: cluster 2022-04-22T21:28:49.520024+0000 mgr.y (mgr.34107) 8288 : cluster [DBG] pgmap v9223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:50 smithi115 bash[46934]: audit 2022-04-22T21:28:50.107395+0000 mon.a (mon.0) 11990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:52.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:51 smithi121 bash[36902]: audit 2022-04-22T21:28:51.215591+0000 mon.a (mon.0) 11991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:51 smithi115 bash[45495]: audit 2022-04-22T21:28:51.215591+0000 mon.a (mon.0) 11991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:51 smithi115 bash[46934]: audit 2022-04-22T21:28:51.215591+0000 mon.a (mon.0) 11991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:52 smithi121 bash[36902]: cluster 2022-04-22T21:28:51.521133+0000 mgr.y (mgr.34107) 8289 : cluster [DBG] pgmap v9224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:52 smithi115 bash[46934]: cluster 2022-04-22T21:28:51.521133+0000 mgr.y (mgr.34107) 8289 : cluster [DBG] pgmap v9224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:52.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:52 smithi115 bash[45495]: cluster 2022-04-22T21:28:51.521133+0000 mgr.y (mgr.34107) 8289 : cluster [DBG] pgmap v9224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:54.710 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:54 smithi115 bash[45495]: cluster 2022-04-22T21:28:53.522204+0000 mgr.y (mgr.34107) 8290 : cluster [DBG] pgmap v9225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:54.710 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:54 smithi115 bash[45495]: audit 2022-04-22T21:28:54.227526+0000 mon.a (mon.0) 11992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:54.710 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:28:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:28:54.434Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:28:54.849 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:28:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:54 smithi121 bash[36902]: cluster 2022-04-22T21:28:53.522204+0000 mgr.y (mgr.34107) 8290 : cluster [DBG] pgmap v9225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:54 smithi121 bash[36902]: audit 2022-04-22T21:28:54.227526+0000 mon.a (mon.0) 11992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:54 smithi115 bash[46934]: cluster 2022-04-22T21:28:53.522204+0000 mgr.y (mgr.34107) 8290 : cluster [DBG] pgmap v9225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:54 smithi115 bash[46934]: audit 2022-04-22T21:28:54.227526+0000 mon.a (mon.0) 11992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:28:55.192 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 4m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (3h) 5m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:28:55.193 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 4m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 5m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 4m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 4m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 4m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 4m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 5m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 5m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:28:55.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 5m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:28:55.195 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 5m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:28:55.602 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:28:55.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:28:55.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:28:55.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:55.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:28:55.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:28:55.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:55.604 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:28:55.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:28:55.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:55.604 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:28:55.605 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:28:55.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:28:55.605 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:28:55.605 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:28:55.605 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:28:55.606 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:28:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:55 smithi121 bash[36902]: audit 2022-04-22T21:28:54.829444+0000 mgr.y (mgr.34107) 8291 : audit [DBG] from='client.53905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:55 smithi121 bash[36902]: audit 2022-04-22T21:28:55.110693+0000 mon.a (mon.0) 11993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:55 smithi121 bash[36902]: audit 2022-04-22T21:28:55.597710+0000 mon.c (mon.1) 297 : audit [DBG] from='client.? 172.21.15.115:0/1429917235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[45495]: audit 2022-04-22T21:28:54.829444+0000 mgr.y (mgr.34107) 8291 : audit [DBG] from='client.53905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[45495]: audit 2022-04-22T21:28:55.110693+0000 mon.a (mon.0) 11993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[45495]: audit 2022-04-22T21:28:55.597710+0000 mon.c (mon.1) 297 : audit [DBG] from='client.? 172.21.15.115:0/1429917235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[46934]: audit 2022-04-22T21:28:54.829444+0000 mgr.y (mgr.34107) 8291 : audit [DBG] from='client.53905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[46934]: audit 2022-04-22T21:28:55.110693+0000 mon.a (mon.0) 11993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:28:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:55 smithi115 bash[46934]: audit 2022-04-22T21:28:55.597710+0000 mon.c (mon.1) 297 : audit [DBG] from='client.? 172.21.15.115:0/1429917235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:28:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:56 smithi121 bash[36902]: audit 2022-04-22T21:28:55.174639+0000 mgr.y (mgr.34107) 8292 : audit [DBG] from='client.53911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:56 smithi121 bash[36902]: cluster 2022-04-22T21:28:55.523054+0000 mgr.y (mgr.34107) 8293 : cluster [DBG] pgmap v9226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:56 smithi121 bash[36902]: audit 2022-04-22T21:28:56.218511+0000 mon.a (mon.0) 11994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[45495]: audit 2022-04-22T21:28:55.174639+0000 mgr.y (mgr.34107) 8292 : audit [DBG] from='client.53911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[45495]: cluster 2022-04-22T21:28:55.523054+0000 mgr.y (mgr.34107) 8293 : cluster [DBG] pgmap v9226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[45495]: audit 2022-04-22T21:28:56.218511+0000 mon.a (mon.0) 11994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[46934]: audit 2022-04-22T21:28:55.174639+0000 mgr.y (mgr.34107) 8292 : audit [DBG] from='client.53911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:28:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[46934]: cluster 2022-04-22T21:28:55.523054+0000 mgr.y (mgr.34107) 8293 : cluster [DBG] pgmap v9226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:56 smithi115 bash[46934]: audit 2022-04-22T21:28:56.218511+0000 mon.a (mon.0) 11994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:28:59.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:58 smithi121 bash[36902]: cluster 2022-04-22T21:28:57.524152+0000 mgr.y (mgr.34107) 8294 : cluster [DBG] pgmap v9227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:58 smithi115 bash[45495]: cluster 2022-04-22T21:28:57.524152+0000 mgr.y (mgr.34107) 8294 : cluster [DBG] pgmap v9227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:28:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:58 smithi115 bash[46934]: cluster 2022-04-22T21:28:57.524152+0000 mgr.y (mgr.34107) 8294 : cluster [DBG] pgmap v9227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:28:59 smithi121 bash[36902]: audit 2022-04-22T21:28:59.230595+0000 mon.a (mon.0) 11995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:28:59 smithi115 bash[45495]: audit 2022-04-22T21:28:59.230595+0000 mon.a (mon.0) 11995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:28:59 smithi115 bash[46934]: audit 2022-04-22T21:28:59.230595+0000 mon.a (mon.0) 11995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:01.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:00 smithi121 bash[36902]: cluster 2022-04-22T21:28:59.525220+0000 mgr.y (mgr.34107) 8295 : cluster [DBG] pgmap v9228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:00 smithi121 bash[36902]: audit 2022-04-22T21:29:00.114139+0000 mon.a (mon.0) 11996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:00 smithi115 bash[45495]: cluster 2022-04-22T21:28:59.525220+0000 mgr.y (mgr.34107) 8295 : cluster [DBG] pgmap v9228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:00 smithi115 bash[45495]: audit 2022-04-22T21:29:00.114139+0000 mon.a (mon.0) 11996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:00 smithi115 bash[46934]: cluster 2022-04-22T21:28:59.525220+0000 mgr.y (mgr.34107) 8295 : cluster [DBG] pgmap v9228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:00 smithi115 bash[46934]: audit 2022-04-22T21:29:00.114139+0000 mon.a (mon.0) 11996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:01 smithi121 bash[36902]: audit 2022-04-22T21:29:01.221696+0000 mon.a (mon.0) 11997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:01 smithi115 bash[45495]: audit 2022-04-22T21:29:01.221696+0000 mon.a (mon.0) 11997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:02.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:01 smithi115 bash[46934]: audit 2022-04-22T21:29:01.221696+0000 mon.a (mon.0) 11997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:02 smithi121 bash[36902]: cluster 2022-04-22T21:29:01.526328+0000 mgr.y (mgr.34107) 8296 : cluster [DBG] pgmap v9229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:02 smithi115 bash[45495]: cluster 2022-04-22T21:29:01.526328+0000 mgr.y (mgr.34107) 8296 : cluster [DBG] pgmap v9229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:02.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:02 smithi115 bash[46934]: cluster 2022-04-22T21:29:01.526328+0000 mgr.y (mgr.34107) 8296 : cluster [DBG] pgmap v9229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:04.743 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:04.432Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:04 smithi121 bash[36902]: cluster 2022-04-22T21:29:03.527461+0000 mgr.y (mgr.34107) 8297 : cluster [DBG] pgmap v9230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:04 smithi121 bash[36902]: audit 2022-04-22T21:29:04.233826+0000 mon.a (mon.0) 11998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:04 smithi115 bash[45495]: cluster 2022-04-22T21:29:03.527461+0000 mgr.y (mgr.34107) 8297 : cluster [DBG] pgmap v9230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:04 smithi115 bash[45495]: audit 2022-04-22T21:29:04.233826+0000 mon.a (mon.0) 11998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:04 smithi115 bash[46934]: cluster 2022-04-22T21:29:03.527461+0000 mgr.y (mgr.34107) 8297 : cluster [DBG] pgmap v9230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:04 smithi115 bash[46934]: audit 2022-04-22T21:29:04.233826+0000 mon.a (mon.0) 11998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:05 smithi121 bash[36902]: audit 2022-04-22T21:29:05.117958+0000 mon.a (mon.0) 11999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:05 smithi115 bash[45495]: audit 2022-04-22T21:29:05.117958+0000 mon.a (mon.0) 11999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:06.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:05 smithi115 bash[46934]: audit 2022-04-22T21:29:05.117958+0000 mon.a (mon.0) 11999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:06 smithi121 bash[36902]: cluster 2022-04-22T21:29:05.528501+0000 mgr.y (mgr.34107) 8298 : cluster [DBG] pgmap v9231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:06 smithi121 bash[36902]: audit 2022-04-22T21:29:06.224863+0000 mon.a (mon.0) 12000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:06 smithi115 bash[45495]: cluster 2022-04-22T21:29:05.528501+0000 mgr.y (mgr.34107) 8298 : cluster [DBG] pgmap v9231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:06 smithi115 bash[45495]: audit 2022-04-22T21:29:06.224863+0000 mon.a (mon.0) 12000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:06 smithi115 bash[46934]: cluster 2022-04-22T21:29:05.528501+0000 mgr.y (mgr.34107) 8298 : cluster [DBG] pgmap v9231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:06 smithi115 bash[46934]: audit 2022-04-22T21:29:06.224863+0000 mon.a (mon.0) 12000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:08 smithi121 bash[36902]: cluster 2022-04-22T21:29:07.529619+0000 mgr.y (mgr.34107) 8299 : cluster [DBG] pgmap v9232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:08 smithi115 bash[45495]: cluster 2022-04-22T21:29:07.529619+0000 mgr.y (mgr.34107) 8299 : cluster [DBG] pgmap v9232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:08 smithi115 bash[46934]: cluster 2022-04-22T21:29:07.529619+0000 mgr.y (mgr.34107) 8299 : cluster [DBG] pgmap v9232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:09 smithi121 bash[36902]: audit 2022-04-22T21:29:09.236769+0000 mon.a (mon.0) 12001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:09 smithi115 bash[45495]: audit 2022-04-22T21:29:09.236769+0000 mon.a (mon.0) 12001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:09 smithi115 bash[46934]: audit 2022-04-22T21:29:09.236769+0000 mon.a (mon.0) 12001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:10 smithi115 bash[46934]: cluster 2022-04-22T21:29:09.530713+0000 mgr.y (mgr.34107) 8300 : cluster [DBG] pgmap v9233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:10 smithi115 bash[46934]: audit 2022-04-22T21:29:10.121815+0000 mon.a (mon.0) 12002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:10 smithi115 bash[45495]: cluster 2022-04-22T21:29:09.530713+0000 mgr.y (mgr.34107) 8300 : cluster [DBG] pgmap v9233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:10 smithi115 bash[45495]: audit 2022-04-22T21:29:10.121815+0000 mon.a (mon.0) 12002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:10 smithi121 bash[36902]: cluster 2022-04-22T21:29:09.530713+0000 mgr.y (mgr.34107) 8300 : cluster [DBG] pgmap v9233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:10 smithi121 bash[36902]: audit 2022-04-22T21:29:10.121815+0000 mon.a (mon.0) 12002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:11 smithi115 bash[46934]: audit 2022-04-22T21:29:11.227787+0000 mon.a (mon.0) 12003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:11 smithi115 bash[45495]: audit 2022-04-22T21:29:11.227787+0000 mon.a (mon.0) 12003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:11 smithi121 bash[36902]: audit 2022-04-22T21:29:11.227787+0000 mon.a (mon.0) 12003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:12 smithi115 bash[46934]: cluster 2022-04-22T21:29:11.531887+0000 mgr.y (mgr.34107) 8301 : cluster [DBG] pgmap v9234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:13.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:12.561Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:12 smithi115 bash[45495]: cluster 2022-04-22T21:29:11.531887+0000 mgr.y (mgr.34107) 8301 : cluster [DBG] pgmap v9234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:12 smithi121 bash[36902]: cluster 2022-04-22T21:29:11.531887+0000 mgr.y (mgr.34107) 8301 : cluster [DBG] pgmap v9234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:14.782 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:14.434Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:14 smithi115 bash[45495]: cluster 2022-04-22T21:29:13.533049+0000 mgr.y (mgr.34107) 8302 : cluster [DBG] pgmap v9235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:14 smithi115 bash[45495]: audit 2022-04-22T21:29:14.239803+0000 mon.a (mon.0) 12004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:14 smithi115 bash[46934]: cluster 2022-04-22T21:29:13.533049+0000 mgr.y (mgr.34107) 8302 : cluster [DBG] pgmap v9235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:14 smithi115 bash[46934]: audit 2022-04-22T21:29:14.239803+0000 mon.a (mon.0) 12004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:14 smithi121 bash[36902]: cluster 2022-04-22T21:29:13.533049+0000 mgr.y (mgr.34107) 8302 : cluster [DBG] pgmap v9235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:14 smithi121 bash[36902]: audit 2022-04-22T21:29:14.239803+0000 mon.a (mon.0) 12004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:15 smithi115 bash[45495]: audit 2022-04-22T21:29:15.125459+0000 mon.a (mon.0) 12005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:15 smithi115 bash[46934]: audit 2022-04-22T21:29:15.125459+0000 mon.a (mon.0) 12005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:16.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:15 smithi121 bash[36902]: audit 2022-04-22T21:29:15.125459+0000 mon.a (mon.0) 12005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[46934]: cluster 2022-04-22T21:29:15.534255+0000 mgr.y (mgr.34107) 8303 : cluster [DBG] pgmap v9236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[46934]: audit 2022-04-22T21:29:16.058707+0000 mon.a (mon.0) 12006 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[46934]: audit 2022-04-22T21:29:16.064173+0000 mon.a (mon.0) 12007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[46934]: audit 2022-04-22T21:29:16.230958+0000 mon.a (mon.0) 12008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[45495]: cluster 2022-04-22T21:29:15.534255+0000 mgr.y (mgr.34107) 8303 : cluster [DBG] pgmap v9236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[45495]: audit 2022-04-22T21:29:16.058707+0000 mon.a (mon.0) 12006 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:29:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[45495]: audit 2022-04-22T21:29:16.064173+0000 mon.a (mon.0) 12007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:29:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:16 smithi115 bash[45495]: audit 2022-04-22T21:29:16.230958+0000 mon.a (mon.0) 12008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:16 smithi121 bash[36902]: cluster 2022-04-22T21:29:15.534255+0000 mgr.y (mgr.34107) 8303 : cluster [DBG] pgmap v9236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:16 smithi121 bash[36902]: audit 2022-04-22T21:29:16.058707+0000 mon.a (mon.0) 12006 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:29:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:16 smithi121 bash[36902]: audit 2022-04-22T21:29:16.064173+0000 mon.a (mon.0) 12007 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:29:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:16 smithi121 bash[36902]: audit 2022-04-22T21:29:16.230958+0000 mon.a (mon.0) 12008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:18 smithi115 bash[45495]: cluster 2022-04-22T21:29:17.535555+0000 mgr.y (mgr.34107) 8304 : cluster [DBG] pgmap v9237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:18 smithi115 bash[46934]: cluster 2022-04-22T21:29:17.535555+0000 mgr.y (mgr.34107) 8304 : cluster [DBG] pgmap v9237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:19.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:18 smithi121 bash[36902]: cluster 2022-04-22T21:29:17.535555+0000 mgr.y (mgr.34107) 8304 : cluster [DBG] pgmap v9237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:19 smithi121 bash[36902]: audit 2022-04-22T21:29:19.242843+0000 mon.a (mon.0) 12009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:19 smithi115 bash[45495]: audit 2022-04-22T21:29:19.242843+0000 mon.a (mon.0) 12009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:19 smithi115 bash[46934]: audit 2022-04-22T21:29:19.242843+0000 mon.a (mon.0) 12009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:20 smithi121 bash[36902]: cluster 2022-04-22T21:29:19.536615+0000 mgr.y (mgr.34107) 8305 : cluster [DBG] pgmap v9238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:20 smithi121 bash[36902]: audit 2022-04-22T21:29:20.129038+0000 mon.a (mon.0) 12010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:20 smithi115 bash[45495]: cluster 2022-04-22T21:29:19.536615+0000 mgr.y (mgr.34107) 8305 : cluster [DBG] pgmap v9238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:20 smithi115 bash[45495]: audit 2022-04-22T21:29:20.129038+0000 mon.a (mon.0) 12010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:20 smithi115 bash[46934]: cluster 2022-04-22T21:29:19.536615+0000 mgr.y (mgr.34107) 8305 : cluster [DBG] pgmap v9238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:20 smithi115 bash[46934]: audit 2022-04-22T21:29:20.129038+0000 mon.a (mon.0) 12010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:21 smithi121 bash[36902]: audit 2022-04-22T21:29:21.233989+0000 mon.a (mon.0) 12011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:21 smithi115 bash[45495]: audit 2022-04-22T21:29:21.233989+0000 mon.a (mon.0) 12011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:21 smithi115 bash[46934]: audit 2022-04-22T21:29:21.233989+0000 mon.a (mon.0) 12011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:22 smithi115 bash[45495]: cluster 2022-04-22T21:29:21.537634+0000 mgr.y (mgr.34107) 8306 : cluster [DBG] pgmap v9239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:22.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:22 smithi115 bash[46934]: cluster 2022-04-22T21:29:21.537634+0000 mgr.y (mgr.34107) 8306 : cluster [DBG] pgmap v9239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:22 smithi121 bash[36902]: cluster 2022-04-22T21:29:21.537634+0000 mgr.y (mgr.34107) 8306 : cluster [DBG] pgmap v9239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:24.435Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:24 smithi121 bash[36902]: cluster 2022-04-22T21:29:23.538629+0000 mgr.y (mgr.34107) 8307 : cluster [DBG] pgmap v9240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:24 smithi121 bash[36902]: audit 2022-04-22T21:29:24.246056+0000 mon.a (mon.0) 12012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:24 smithi115 bash[45495]: cluster 2022-04-22T21:29:23.538629+0000 mgr.y (mgr.34107) 8307 : cluster [DBG] pgmap v9240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:24 smithi115 bash[45495]: audit 2022-04-22T21:29:24.246056+0000 mon.a (mon.0) 12012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:24 smithi115 bash[46934]: cluster 2022-04-22T21:29:23.538629+0000 mgr.y (mgr.34107) 8307 : cluster [DBG] pgmap v9240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:24 smithi115 bash[46934]: audit 2022-04-22T21:29:24.246056+0000 mon.a (mon.0) 12012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:25.985 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:29:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:25 smithi121 bash[36902]: audit 2022-04-22T21:29:25.132989+0000 mon.a (mon.0) 12013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:25 smithi115 bash[45495]: audit 2022-04-22T21:29:25.132989+0000 mon.a (mon.0) 12013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:25 smithi115 bash[46934]: audit 2022-04-22T21:29:25.132989+0000 mon.a (mon.0) 12013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:29:26.328 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:29:26.329 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:29:26.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:29:26.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:29:26.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:29:26.330 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:29:26.726 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:29:26.726 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:29:26.726 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:29:26.727 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:29:26.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:26.728 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:29:26.728 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:29:26.728 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:29:26.728 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:29:27.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[46934]: cluster 2022-04-22T21:29:25.539821+0000 mgr.y (mgr.34107) 8308 : cluster [DBG] pgmap v9241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:27.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[46934]: audit 2022-04-22T21:29:26.236790+0000 mon.a (mon.0) 12014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:27.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[46934]: audit 2022-04-22T21:29:26.722559+0000 mon.a (mon.0) 12015 : audit [DBG] from='client.? 172.21.15.115:0/2838449018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:27.212 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[45495]: cluster 2022-04-22T21:29:25.539821+0000 mgr.y (mgr.34107) 8308 : cluster [DBG] pgmap v9241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:27.212 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[45495]: audit 2022-04-22T21:29:26.236790+0000 mon.a (mon.0) 12014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:27.212 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:26 smithi115 bash[45495]: audit 2022-04-22T21:29:26.722559+0000 mon.a (mon.0) 12015 : audit [DBG] from='client.? 172.21.15.115:0/2838449018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:27.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:26 smithi121 bash[36902]: cluster 2022-04-22T21:29:25.539821+0000 mgr.y (mgr.34107) 8308 : cluster [DBG] pgmap v9241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:27.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:26 smithi121 bash[36902]: audit 2022-04-22T21:29:26.236790+0000 mon.a (mon.0) 12014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:27.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:26 smithi121 bash[36902]: audit 2022-04-22T21:29:26.722559+0000 mon.a (mon.0) 12015 : audit [DBG] from='client.? 172.21.15.115:0/2838449018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:28.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:25.965143+0000 mgr.y (mgr.34107) 8309 : audit [DBG] from='client.43992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:26.310054+0000 mgr.y (mgr.34107) 8310 : audit [DBG] from='client.53929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:27.183991+0000 mon.a (mon.0) 12016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:27.186817+0000 mon.a (mon.0) 12017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:27.188551+0000 mon.a (mon.0) 12018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:27 smithi121 bash[36902]: audit 2022-04-22T21:29:27.558469+0000 mon.a (mon.0) 12019 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:25.965143+0000 mgr.y (mgr.34107) 8309 : audit [DBG] from='client.43992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:26.310054+0000 mgr.y (mgr.34107) 8310 : audit [DBG] from='client.53929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:27.183991+0000 mon.a (mon.0) 12016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:27.186817+0000 mon.a (mon.0) 12017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:27.188551+0000 mon.a (mon.0) 12018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:29:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[45495]: audit 2022-04-22T21:29:27.558469+0000 mon.a (mon.0) 12019 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:25.965143+0000 mgr.y (mgr.34107) 8309 : audit [DBG] from='client.43992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:26.310054+0000 mgr.y (mgr.34107) 8310 : audit [DBG] from='client.53929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:27.183991+0000 mon.a (mon.0) 12016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:27.186817+0000 mon.a (mon.0) 12017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:27.188551+0000 mon.a (mon.0) 12018 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:29:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:27 smithi115 bash[46934]: audit 2022-04-22T21:29:27.558469+0000 mon.a (mon.0) 12019 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:29:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:28 smithi121 bash[36902]: cluster 2022-04-22T21:29:27.540844+0000 mgr.y (mgr.34107) 8311 : cluster [DBG] pgmap v9242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:28 smithi115 bash[45495]: cluster 2022-04-22T21:29:27.540844+0000 mgr.y (mgr.34107) 8311 : cluster [DBG] pgmap v9242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:28 smithi115 bash[46934]: cluster 2022-04-22T21:29:27.540844+0000 mgr.y (mgr.34107) 8311 : cluster [DBG] pgmap v9242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:29 smithi121 bash[36902]: audit 2022-04-22T21:29:29.249156+0000 mon.a (mon.0) 12020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:29 smithi115 bash[45495]: audit 2022-04-22T21:29:29.249156+0000 mon.a (mon.0) 12020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:29 smithi115 bash[46934]: audit 2022-04-22T21:29:29.249156+0000 mon.a (mon.0) 12020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:30 smithi121 bash[36902]: cluster 2022-04-22T21:29:29.541953+0000 mgr.y (mgr.34107) 8312 : cluster [DBG] pgmap v9243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:30 smithi121 bash[36902]: audit 2022-04-22T21:29:30.136549+0000 mon.a (mon.0) 12021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:30 smithi115 bash[45495]: cluster 2022-04-22T21:29:29.541953+0000 mgr.y (mgr.34107) 8312 : cluster [DBG] pgmap v9243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:30 smithi115 bash[45495]: audit 2022-04-22T21:29:30.136549+0000 mon.a (mon.0) 12021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:30 smithi115 bash[46934]: cluster 2022-04-22T21:29:29.541953+0000 mgr.y (mgr.34107) 8312 : cluster [DBG] pgmap v9243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:30 smithi115 bash[46934]: audit 2022-04-22T21:29:30.136549+0000 mon.a (mon.0) 12021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:31 smithi121 bash[36902]: audit 2022-04-22T21:29:31.239979+0000 mon.a (mon.0) 12022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:31 smithi115 bash[45495]: audit 2022-04-22T21:29:31.239979+0000 mon.a (mon.0) 12022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:31 smithi115 bash[46934]: audit 2022-04-22T21:29:31.239979+0000 mon.a (mon.0) 12022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:32.860 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:32.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:32 smithi121 bash[36902]: cluster 2022-04-22T21:29:31.543197+0000 mgr.y (mgr.34107) 8313 : cluster [DBG] pgmap v9244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:32 smithi115 bash[45495]: cluster 2022-04-22T21:29:31.543197+0000 mgr.y (mgr.34107) 8313 : cluster [DBG] pgmap v9244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:32 smithi115 bash[46934]: cluster 2022-04-22T21:29:31.543197+0000 mgr.y (mgr.34107) 8313 : cluster [DBG] pgmap v9244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:34.436Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:34 smithi121 bash[36902]: cluster 2022-04-22T21:29:33.544433+0000 mgr.y (mgr.34107) 8314 : cluster [DBG] pgmap v9245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:34 smithi121 bash[36902]: audit 2022-04-22T21:29:34.252205+0000 mon.a (mon.0) 12023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:34 smithi115 bash[45495]: cluster 2022-04-22T21:29:33.544433+0000 mgr.y (mgr.34107) 8314 : cluster [DBG] pgmap v9245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:34 smithi115 bash[45495]: audit 2022-04-22T21:29:34.252205+0000 mon.a (mon.0) 12023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:34 smithi115 bash[46934]: cluster 2022-04-22T21:29:33.544433+0000 mgr.y (mgr.34107) 8314 : cluster [DBG] pgmap v9245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:34 smithi115 bash[46934]: audit 2022-04-22T21:29:34.252205+0000 mon.a (mon.0) 12023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:35 smithi121 bash[36902]: audit 2022-04-22T21:29:35.140324+0000 mon.a (mon.0) 12024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:35 smithi115 bash[45495]: audit 2022-04-22T21:29:35.140324+0000 mon.a (mon.0) 12024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:35 smithi115 bash[46934]: audit 2022-04-22T21:29:35.140324+0000 mon.a (mon.0) 12024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:36 smithi121 bash[36902]: cluster 2022-04-22T21:29:35.545575+0000 mgr.y (mgr.34107) 8315 : cluster [DBG] pgmap v9246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:36 smithi121 bash[36902]: audit 2022-04-22T21:29:36.243105+0000 mon.a (mon.0) 12025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:36 smithi115 bash[45495]: cluster 2022-04-22T21:29:35.545575+0000 mgr.y (mgr.34107) 8315 : cluster [DBG] pgmap v9246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:36 smithi115 bash[45495]: audit 2022-04-22T21:29:36.243105+0000 mon.a (mon.0) 12025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:36 smithi115 bash[46934]: cluster 2022-04-22T21:29:35.545575+0000 mgr.y (mgr.34107) 8315 : cluster [DBG] pgmap v9246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:36 smithi115 bash[46934]: audit 2022-04-22T21:29:36.243105+0000 mon.a (mon.0) 12025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:39.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:38 smithi121 bash[36902]: cluster 2022-04-22T21:29:37.546696+0000 mgr.y (mgr.34107) 8316 : cluster [DBG] pgmap v9247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:38 smithi115 bash[45495]: cluster 2022-04-22T21:29:37.546696+0000 mgr.y (mgr.34107) 8316 : cluster [DBG] pgmap v9247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:38 smithi115 bash[46934]: cluster 2022-04-22T21:29:37.546696+0000 mgr.y (mgr.34107) 8316 : cluster [DBG] pgmap v9247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:39 smithi121 bash[36902]: audit 2022-04-22T21:29:39.255442+0000 mon.a (mon.0) 12026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:39 smithi115 bash[45495]: audit 2022-04-22T21:29:39.255442+0000 mon.a (mon.0) 12026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:39 smithi115 bash[46934]: audit 2022-04-22T21:29:39.255442+0000 mon.a (mon.0) 12026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:40 smithi121 bash[36902]: cluster 2022-04-22T21:29:39.547862+0000 mgr.y (mgr.34107) 8317 : cluster [DBG] pgmap v9248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:40 smithi121 bash[36902]: audit 2022-04-22T21:29:40.143801+0000 mon.a (mon.0) 12027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:40 smithi115 bash[45495]: cluster 2022-04-22T21:29:39.547862+0000 mgr.y (mgr.34107) 8317 : cluster [DBG] pgmap v9248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:40 smithi115 bash[45495]: audit 2022-04-22T21:29:40.143801+0000 mon.a (mon.0) 12027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:40 smithi115 bash[46934]: cluster 2022-04-22T21:29:39.547862+0000 mgr.y (mgr.34107) 8317 : cluster [DBG] pgmap v9248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:40 smithi115 bash[46934]: audit 2022-04-22T21:29:40.143801+0000 mon.a (mon.0) 12027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:41 smithi121 bash[36902]: audit 2022-04-22T21:29:41.246046+0000 mon.a (mon.0) 12028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:42.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:41 smithi115 bash[45495]: audit 2022-04-22T21:29:41.246046+0000 mon.a (mon.0) 12028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:41 smithi115 bash[46934]: audit 2022-04-22T21:29:41.246046+0000 mon.a (mon.0) 12028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:42.895 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:42.563Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:42 smithi121 bash[36902]: cluster 2022-04-22T21:29:41.548924+0000 mgr.y (mgr.34107) 8318 : cluster [DBG] pgmap v9249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:43.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:42 smithi115 bash[45495]: cluster 2022-04-22T21:29:41.548924+0000 mgr.y (mgr.34107) 8318 : cluster [DBG] pgmap v9249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:43.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:42 smithi115 bash[46934]: cluster 2022-04-22T21:29:41.548924+0000 mgr.y (mgr.34107) 8318 : cluster [DBG] pgmap v9249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:44.437Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:44 smithi121 bash[36902]: cluster 2022-04-22T21:29:43.550048+0000 mgr.y (mgr.34107) 8319 : cluster [DBG] pgmap v9250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:44 smithi121 bash[36902]: audit 2022-04-22T21:29:44.258538+0000 mon.a (mon.0) 12029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:44 smithi115 bash[45495]: cluster 2022-04-22T21:29:43.550048+0000 mgr.y (mgr.34107) 8319 : cluster [DBG] pgmap v9250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:44 smithi115 bash[45495]: audit 2022-04-22T21:29:44.258538+0000 mon.a (mon.0) 12029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:44 smithi115 bash[46934]: cluster 2022-04-22T21:29:43.550048+0000 mgr.y (mgr.34107) 8319 : cluster [DBG] pgmap v9250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:44 smithi115 bash[46934]: audit 2022-04-22T21:29:44.258538+0000 mon.a (mon.0) 12029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:45 smithi121 bash[36902]: audit 2022-04-22T21:29:45.147430+0000 mon.a (mon.0) 12030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:45 smithi115 bash[45495]: audit 2022-04-22T21:29:45.147430+0000 mon.a (mon.0) 12030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:45 smithi115 bash[46934]: audit 2022-04-22T21:29:45.147430+0000 mon.a (mon.0) 12030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:46 smithi121 bash[36902]: cluster 2022-04-22T21:29:45.551125+0000 mgr.y (mgr.34107) 8320 : cluster [DBG] pgmap v9251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:47.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:46 smithi121 bash[36902]: audit 2022-04-22T21:29:46.249003+0000 mon.a (mon.0) 12031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:47.318 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:46 smithi115 bash[45495]: cluster 2022-04-22T21:29:45.551125+0000 mgr.y (mgr.34107) 8320 : cluster [DBG] pgmap v9251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:47.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:46 smithi115 bash[45495]: audit 2022-04-22T21:29:46.249003+0000 mon.a (mon.0) 12031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:47.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:46 smithi115 bash[46934]: cluster 2022-04-22T21:29:45.551125+0000 mgr.y (mgr.34107) 8320 : cluster [DBG] pgmap v9251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:47.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:46 smithi115 bash[46934]: audit 2022-04-22T21:29:46.249003+0000 mon.a (mon.0) 12031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:49.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:48 smithi121 bash[36902]: cluster 2022-04-22T21:29:47.552177+0000 mgr.y (mgr.34107) 8321 : cluster [DBG] pgmap v9252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:48 smithi115 bash[45495]: cluster 2022-04-22T21:29:47.552177+0000 mgr.y (mgr.34107) 8321 : cluster [DBG] pgmap v9252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:48 smithi115 bash[46934]: cluster 2022-04-22T21:29:47.552177+0000 mgr.y (mgr.34107) 8321 : cluster [DBG] pgmap v9252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:50.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:49 smithi121 bash[36902]: audit 2022-04-22T21:29:49.261883+0000 mon.a (mon.0) 12032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:49 smithi115 bash[45495]: audit 2022-04-22T21:29:49.261883+0000 mon.a (mon.0) 12032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:49 smithi115 bash[46934]: audit 2022-04-22T21:29:49.261883+0000 mon.a (mon.0) 12032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:50 smithi121 bash[36902]: cluster 2022-04-22T21:29:49.553236+0000 mgr.y (mgr.34107) 8322 : cluster [DBG] pgmap v9253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:50 smithi121 bash[36902]: audit 2022-04-22T21:29:50.151019+0000 mon.a (mon.0) 12033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:50 smithi115 bash[45495]: cluster 2022-04-22T21:29:49.553236+0000 mgr.y (mgr.34107) 8322 : cluster [DBG] pgmap v9253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:50 smithi115 bash[45495]: audit 2022-04-22T21:29:50.151019+0000 mon.a (mon.0) 12033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:50 smithi115 bash[46934]: cluster 2022-04-22T21:29:49.553236+0000 mgr.y (mgr.34107) 8322 : cluster [DBG] pgmap v9253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:50 smithi115 bash[46934]: audit 2022-04-22T21:29:50.151019+0000 mon.a (mon.0) 12033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:51 smithi121 bash[36902]: audit 2022-04-22T21:29:51.252146+0000 mon.a (mon.0) 12034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:51 smithi115 bash[45495]: audit 2022-04-22T21:29:51.252146+0000 mon.a (mon.0) 12034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:51 smithi115 bash[46934]: audit 2022-04-22T21:29:51.252146+0000 mon.a (mon.0) 12034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:52.930 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:52.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:52 smithi121 bash[36902]: cluster 2022-04-22T21:29:51.554362+0000 mgr.y (mgr.34107) 8323 : cluster [DBG] pgmap v9254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:52 smithi115 bash[45495]: cluster 2022-04-22T21:29:51.554362+0000 mgr.y (mgr.34107) 8323 : cluster [DBG] pgmap v9254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:52 smithi115 bash[46934]: cluster 2022-04-22T21:29:51.554362+0000 mgr.y (mgr.34107) 8323 : cluster [DBG] pgmap v9254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:29:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:29:54.437Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:29:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:54 smithi121 bash[36902]: cluster 2022-04-22T21:29:53.555436+0000 mgr.y (mgr.34107) 8324 : cluster [DBG] pgmap v9255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:54 smithi121 bash[36902]: audit 2022-04-22T21:29:54.264994+0000 mon.a (mon.0) 12035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:54 smithi115 bash[45495]: cluster 2022-04-22T21:29:53.555436+0000 mgr.y (mgr.34107) 8324 : cluster [DBG] pgmap v9255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:54 smithi115 bash[45495]: audit 2022-04-22T21:29:54.264994+0000 mon.a (mon.0) 12035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:54 smithi115 bash[46934]: cluster 2022-04-22T21:29:53.555436+0000 mgr.y (mgr.34107) 8324 : cluster [DBG] pgmap v9255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:54 smithi115 bash[46934]: audit 2022-04-22T21:29:54.264994+0000 mon.a (mon.0) 12035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:29:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:55 smithi121 bash[36902]: audit 2022-04-22T21:29:55.154545+0000 mon.a (mon.0) 12036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:56.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:55 smithi115 bash[45495]: audit 2022-04-22T21:29:55.154545+0000 mon.a (mon.0) 12036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:56.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:55 smithi115 bash[46934]: audit 2022-04-22T21:29:55.154545+0000 mon.a (mon.0) 12036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:29:57.105 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:29:57.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:56 smithi121 bash[36902]: cluster 2022-04-22T21:29:55.556601+0000 mgr.y (mgr.34107) 8325 : cluster [DBG] pgmap v9256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:56 smithi121 bash[36902]: audit 2022-04-22T21:29:56.255386+0000 mon.a (mon.0) 12037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:56 smithi115 bash[45495]: cluster 2022-04-22T21:29:55.556601+0000 mgr.y (mgr.34107) 8325 : cluster [DBG] pgmap v9256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:56 smithi115 bash[45495]: audit 2022-04-22T21:29:56.255386+0000 mon.a (mon.0) 12037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:56 smithi115 bash[46934]: cluster 2022-04-22T21:29:55.556601+0000 mgr.y (mgr.34107) 8325 : cluster [DBG] pgmap v9256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:56 smithi115 bash[46934]: audit 2022-04-22T21:29:56.255386+0000 mon.a (mon.0) 12037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:29:57.446 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:29:57.446 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 5m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:29:57.446 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:29:57.447 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:29:57.447 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:29:57.447 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:29:57.447 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:29:57.447 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:29:57.448 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:29:57.448 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:29:57.448 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 5m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:29:57.448 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 6m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:29:57.449 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3h) 5m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:29:57.449 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 5m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:29:57.449 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 5m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:29:57.449 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 5m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:29:57.449 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 6m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:29:57.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:29:57.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 6m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:29:57.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 6m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:29:57.450 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 6m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:57.853 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:29:57.854 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:29:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:57 smithi121 bash[36902]: audit 2022-04-22T21:29:57.849145+0000 mon.a (mon.0) 12038 : audit [DBG] from='client.? 172.21.15.115:0/2025737260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:57 smithi115 bash[45495]: audit 2022-04-22T21:29:57.849145+0000 mon.a (mon.0) 12038 : audit [DBG] from='client.? 172.21.15.115:0/2025737260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:57 smithi115 bash[46934]: audit 2022-04-22T21:29:57.849145+0000 mon.a (mon.0) 12038 : audit [DBG] from='client.? 172.21.15.115:0/2025737260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:29:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:58 smithi121 bash[36902]: audit 2022-04-22T21:29:57.085722+0000 mgr.y (mgr.34107) 8326 : audit [DBG] from='client.53941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:58 smithi121 bash[36902]: audit 2022-04-22T21:29:57.422754+0000 mgr.y (mgr.34107) 8327 : audit [DBG] from='client.53947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:58 smithi121 bash[36902]: cluster 2022-04-22T21:29:57.557630+0000 mgr.y (mgr.34107) 8328 : cluster [DBG] pgmap v9257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[45495]: audit 2022-04-22T21:29:57.085722+0000 mgr.y (mgr.34107) 8326 : audit [DBG] from='client.53941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[45495]: audit 2022-04-22T21:29:57.422754+0000 mgr.y (mgr.34107) 8327 : audit [DBG] from='client.53947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[45495]: cluster 2022-04-22T21:29:57.557630+0000 mgr.y (mgr.34107) 8328 : cluster [DBG] pgmap v9257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:29:59.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[46934]: audit 2022-04-22T21:29:57.085722+0000 mgr.y (mgr.34107) 8326 : audit [DBG] from='client.53941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[46934]: audit 2022-04-22T21:29:57.422754+0000 mgr.y (mgr.34107) 8327 : audit [DBG] from='client.53947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:29:59.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:58 smithi115 bash[46934]: cluster 2022-04-22T21:29:57.557630+0000 mgr.y (mgr.34107) 8328 : cluster [DBG] pgmap v9257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:29:59 smithi121 bash[36902]: audit 2022-04-22T21:29:59.268421+0000 mon.a (mon.0) 12039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:29:59 smithi115 bash[45495]: audit 2022-04-22T21:29:59.268421+0000 mon.a (mon.0) 12039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:29:59 smithi115 bash[46934]: audit 2022-04-22T21:29:59.268421+0000 mon.a (mon.0) 12039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:00 smithi121 bash[36902]: cluster 2022-04-22T21:29:59.558707+0000 mgr.y (mgr.34107) 8329 : cluster [DBG] pgmap v9258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:00 smithi121 bash[36902]: cluster 2022-04-22T21:30:00.000130+0000 mon.a (mon.0) 12040 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:30:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:00 smithi121 bash[36902]: audit 2022-04-22T21:30:00.158117+0000 mon.a (mon.0) 12041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[45495]: cluster 2022-04-22T21:29:59.558707+0000 mgr.y (mgr.34107) 8329 : cluster [DBG] pgmap v9258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[45495]: cluster 2022-04-22T21:30:00.000130+0000 mon.a (mon.0) 12040 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[45495]: audit 2022-04-22T21:30:00.158117+0000 mon.a (mon.0) 12041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[46934]: cluster 2022-04-22T21:29:59.558707+0000 mgr.y (mgr.34107) 8329 : cluster [DBG] pgmap v9258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[46934]: cluster 2022-04-22T21:30:00.000130+0000 mon.a (mon.0) 12040 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:30:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:00 smithi115 bash[46934]: audit 2022-04-22T21:30:00.158117+0000 mon.a (mon.0) 12041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:01 smithi121 bash[36902]: audit 2022-04-22T21:30:01.258455+0000 mon.a (mon.0) 12042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:01 smithi121 bash[36902]: cluster 2022-04-22T21:30:01.559808+0000 mgr.y (mgr.34107) 8330 : cluster [DBG] pgmap v9259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:01 smithi115 bash[45495]: audit 2022-04-22T21:30:01.258455+0000 mon.a (mon.0) 12042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:01 smithi115 bash[45495]: cluster 2022-04-22T21:30:01.559808+0000 mgr.y (mgr.34107) 8330 : cluster [DBG] pgmap v9259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:01 smithi115 bash[46934]: audit 2022-04-22T21:30:01.258455+0000 mon.a (mon.0) 12042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:01 smithi115 bash[46934]: cluster 2022-04-22T21:30:01.559808+0000 mgr.y (mgr.34107) 8330 : cluster [DBG] pgmap v9259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:02.565Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:04.437Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:04 smithi115 bash[45495]: cluster 2022-04-22T21:30:03.560932+0000 mgr.y (mgr.34107) 8331 : cluster [DBG] pgmap v9260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:04 smithi115 bash[45495]: audit 2022-04-22T21:30:04.271687+0000 mon.a (mon.0) 12043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:04 smithi115 bash[46934]: cluster 2022-04-22T21:30:03.560932+0000 mgr.y (mgr.34107) 8331 : cluster [DBG] pgmap v9260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:04 smithi115 bash[46934]: audit 2022-04-22T21:30:04.271687+0000 mon.a (mon.0) 12043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:05.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:04 smithi121 bash[36902]: cluster 2022-04-22T21:30:03.560932+0000 mgr.y (mgr.34107) 8331 : cluster [DBG] pgmap v9260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:04 smithi121 bash[36902]: audit 2022-04-22T21:30:04.271687+0000 mon.a (mon.0) 12043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:05 smithi121 bash[36902]: audit 2022-04-22T21:30:05.162086+0000 mon.a (mon.0) 12044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:05 smithi115 bash[46934]: audit 2022-04-22T21:30:05.162086+0000 mon.a (mon.0) 12044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:05 smithi115 bash[45495]: audit 2022-04-22T21:30:05.162086+0000 mon.a (mon.0) 12044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:06 smithi121 bash[36902]: cluster 2022-04-22T21:30:05.562038+0000 mgr.y (mgr.34107) 8332 : cluster [DBG] pgmap v9261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:06 smithi121 bash[36902]: audit 2022-04-22T21:30:06.261464+0000 mon.a (mon.0) 12045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:06 smithi115 bash[45495]: cluster 2022-04-22T21:30:05.562038+0000 mgr.y (mgr.34107) 8332 : cluster [DBG] pgmap v9261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:06 smithi115 bash[45495]: audit 2022-04-22T21:30:06.261464+0000 mon.a (mon.0) 12045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:06 smithi115 bash[46934]: cluster 2022-04-22T21:30:05.562038+0000 mgr.y (mgr.34107) 8332 : cluster [DBG] pgmap v9261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:06 smithi115 bash[46934]: audit 2022-04-22T21:30:06.261464+0000 mon.a (mon.0) 12045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:08 smithi121 bash[36902]: cluster 2022-04-22T21:30:07.563108+0000 mgr.y (mgr.34107) 8333 : cluster [DBG] pgmap v9262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:08 smithi115 bash[45495]: cluster 2022-04-22T21:30:07.563108+0000 mgr.y (mgr.34107) 8333 : cluster [DBG] pgmap v9262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:08 smithi115 bash[46934]: cluster 2022-04-22T21:30:07.563108+0000 mgr.y (mgr.34107) 8333 : cluster [DBG] pgmap v9262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:09 smithi121 bash[36902]: audit 2022-04-22T21:30:09.274788+0000 mon.a (mon.0) 12046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:09 smithi115 bash[45495]: audit 2022-04-22T21:30:09.274788+0000 mon.a (mon.0) 12046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:09 smithi115 bash[46934]: audit 2022-04-22T21:30:09.274788+0000 mon.a (mon.0) 12046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:10 smithi121 bash[36902]: cluster 2022-04-22T21:30:09.564299+0000 mgr.y (mgr.34107) 8334 : cluster [DBG] pgmap v9263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:10 smithi121 bash[36902]: audit 2022-04-22T21:30:10.165887+0000 mon.a (mon.0) 12047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:10 smithi115 bash[45495]: cluster 2022-04-22T21:30:09.564299+0000 mgr.y (mgr.34107) 8334 : cluster [DBG] pgmap v9263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:10 smithi115 bash[45495]: audit 2022-04-22T21:30:10.165887+0000 mon.a (mon.0) 12047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:10 smithi115 bash[46934]: cluster 2022-04-22T21:30:09.564299+0000 mgr.y (mgr.34107) 8334 : cluster [DBG] pgmap v9263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:10 smithi115 bash[46934]: audit 2022-04-22T21:30:10.165887+0000 mon.a (mon.0) 12047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:11 smithi121 bash[36902]: audit 2022-04-22T21:30:11.264612+0000 mon.a (mon.0) 12048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:11 smithi115 bash[45495]: audit 2022-04-22T21:30:11.264612+0000 mon.a (mon.0) 12048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:11 smithi115 bash[46934]: audit 2022-04-22T21:30:11.264612+0000 mon.a (mon.0) 12048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:12 smithi121 bash[36902]: cluster 2022-04-22T21:30:11.565518+0000 mgr.y (mgr.34107) 8335 : cluster [DBG] pgmap v9264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:12 smithi115 bash[45495]: cluster 2022-04-22T21:30:11.565518+0000 mgr.y (mgr.34107) 8335 : cluster [DBG] pgmap v9264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:12.567Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:12 smithi115 bash[46934]: cluster 2022-04-22T21:30:11.565518+0000 mgr.y (mgr.34107) 8335 : cluster [DBG] pgmap v9264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:14.438Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:14 smithi115 bash[45495]: cluster 2022-04-22T21:30:13.566669+0000 mgr.y (mgr.34107) 8336 : cluster [DBG] pgmap v9265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:14 smithi115 bash[45495]: audit 2022-04-22T21:30:14.278017+0000 mon.a (mon.0) 12049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:14 smithi115 bash[46934]: cluster 2022-04-22T21:30:13.566669+0000 mgr.y (mgr.34107) 8336 : cluster [DBG] pgmap v9265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:14 smithi115 bash[46934]: audit 2022-04-22T21:30:14.278017+0000 mon.a (mon.0) 12049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:15.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:14 smithi121 bash[36902]: cluster 2022-04-22T21:30:13.566669+0000 mgr.y (mgr.34107) 8336 : cluster [DBG] pgmap v9265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:14 smithi121 bash[36902]: audit 2022-04-22T21:30:14.278017+0000 mon.a (mon.0) 12049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:15 smithi121 bash[36902]: audit 2022-04-22T21:30:15.169695+0000 mon.a (mon.0) 12050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:15 smithi115 bash[45495]: audit 2022-04-22T21:30:15.169695+0000 mon.a (mon.0) 12050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:15 smithi115 bash[46934]: audit 2022-04-22T21:30:15.169695+0000 mon.a (mon.0) 12050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:16 smithi121 bash[36902]: cluster 2022-04-22T21:30:15.567799+0000 mgr.y (mgr.34107) 8337 : cluster [DBG] pgmap v9266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:16 smithi121 bash[36902]: audit 2022-04-22T21:30:16.075152+0000 mon.a (mon.0) 12051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:16 smithi121 bash[36902]: audit 2022-04-22T21:30:16.075434+0000 mon.a (mon.0) 12052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:16 smithi121 bash[36902]: audit 2022-04-22T21:30:16.267511+0000 mon.a (mon.0) 12053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[45495]: cluster 2022-04-22T21:30:15.567799+0000 mgr.y (mgr.34107) 8337 : cluster [DBG] pgmap v9266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[45495]: audit 2022-04-22T21:30:16.075152+0000 mon.a (mon.0) 12051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:30:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[45495]: audit 2022-04-22T21:30:16.075434+0000 mon.a (mon.0) 12052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:30:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[45495]: audit 2022-04-22T21:30:16.267511+0000 mon.a (mon.0) 12053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[46934]: cluster 2022-04-22T21:30:15.567799+0000 mgr.y (mgr.34107) 8337 : cluster [DBG] pgmap v9266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[46934]: audit 2022-04-22T21:30:16.075152+0000 mon.a (mon.0) 12051 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:30:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[46934]: audit 2022-04-22T21:30:16.075434+0000 mon.a (mon.0) 12052 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:30:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:16 smithi115 bash[46934]: audit 2022-04-22T21:30:16.267511+0000 mon.a (mon.0) 12053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:18 smithi121 bash[36902]: cluster 2022-04-22T21:30:17.568854+0000 mgr.y (mgr.34107) 8338 : cluster [DBG] pgmap v9267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:18 smithi115 bash[45495]: cluster 2022-04-22T21:30:17.568854+0000 mgr.y (mgr.34107) 8338 : cluster [DBG] pgmap v9267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:18 smithi115 bash[46934]: cluster 2022-04-22T21:30:17.568854+0000 mgr.y (mgr.34107) 8338 : cluster [DBG] pgmap v9267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:20.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:19 smithi121 bash[36902]: audit 2022-04-22T21:30:19.281163+0000 mon.a (mon.0) 12054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:19 smithi115 bash[45495]: audit 2022-04-22T21:30:19.281163+0000 mon.a (mon.0) 12054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:19 smithi115 bash[46934]: audit 2022-04-22T21:30:19.281163+0000 mon.a (mon.0) 12054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:21.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:20 smithi121 bash[36902]: cluster 2022-04-22T21:30:19.569909+0000 mgr.y (mgr.34107) 8339 : cluster [DBG] pgmap v9268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:20 smithi121 bash[36902]: audit 2022-04-22T21:30:20.173437+0000 mon.a (mon.0) 12055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:20 smithi115 bash[45495]: cluster 2022-04-22T21:30:19.569909+0000 mgr.y (mgr.34107) 8339 : cluster [DBG] pgmap v9268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:20 smithi115 bash[45495]: audit 2022-04-22T21:30:20.173437+0000 mon.a (mon.0) 12055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:20 smithi115 bash[46934]: cluster 2022-04-22T21:30:19.569909+0000 mgr.y (mgr.34107) 8339 : cluster [DBG] pgmap v9268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:20 smithi115 bash[46934]: audit 2022-04-22T21:30:20.173437+0000 mon.a (mon.0) 12055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:22.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:21 smithi121 bash[36902]: audit 2022-04-22T21:30:21.270849+0000 mon.a (mon.0) 12056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:21 smithi115 bash[45495]: audit 2022-04-22T21:30:21.270849+0000 mon.a (mon.0) 12056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:22.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:21 smithi115 bash[46934]: audit 2022-04-22T21:30:21.270849+0000 mon.a (mon.0) 12056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:23.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:22 smithi121 bash[36902]: cluster 2022-04-22T21:30:21.570927+0000 mgr.y (mgr.34107) 8340 : cluster [DBG] pgmap v9269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:22 smithi115 bash[45495]: cluster 2022-04-22T21:30:21.570927+0000 mgr.y (mgr.34107) 8340 : cluster [DBG] pgmap v9269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:22.566Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:22 smithi115 bash[46934]: cluster 2022-04-22T21:30:21.570927+0000 mgr.y (mgr.34107) 8340 : cluster [DBG] pgmap v9269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:24.439Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:24 smithi115 bash[45495]: cluster 2022-04-22T21:30:23.572196+0000 mgr.y (mgr.34107) 8341 : cluster [DBG] pgmap v9270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:24 smithi115 bash[45495]: audit 2022-04-22T21:30:24.284165+0000 mon.a (mon.0) 12057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:24 smithi115 bash[46934]: cluster 2022-04-22T21:30:23.572196+0000 mgr.y (mgr.34107) 8341 : cluster [DBG] pgmap v9270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:24 smithi115 bash[46934]: audit 2022-04-22T21:30:24.284165+0000 mon.a (mon.0) 12057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:24 smithi121 bash[36902]: cluster 2022-04-22T21:30:23.572196+0000 mgr.y (mgr.34107) 8341 : cluster [DBG] pgmap v9270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:24 smithi121 bash[36902]: audit 2022-04-22T21:30:24.284165+0000 mon.a (mon.0) 12057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:25 smithi121 bash[36902]: audit 2022-04-22T21:30:25.177062+0000 mon.a (mon.0) 12058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:25 smithi115 bash[45495]: audit 2022-04-22T21:30:25.177062+0000 mon.a (mon.0) 12058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:25 smithi115 bash[46934]: audit 2022-04-22T21:30:25.177062+0000 mon.a (mon.0) 12058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:26 smithi121 bash[36902]: cluster 2022-04-22T21:30:25.573279+0000 mgr.y (mgr.34107) 8342 : cluster [DBG] pgmap v9271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:26 smithi121 bash[36902]: audit 2022-04-22T21:30:26.273868+0000 mon.a (mon.0) 12059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:26 smithi115 bash[45495]: cluster 2022-04-22T21:30:25.573279+0000 mgr.y (mgr.34107) 8342 : cluster [DBG] pgmap v9271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:26 smithi115 bash[45495]: audit 2022-04-22T21:30:26.273868+0000 mon.a (mon.0) 12059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:26 smithi115 bash[46934]: cluster 2022-04-22T21:30:25.573279+0000 mgr.y (mgr.34107) 8342 : cluster [DBG] pgmap v9271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:26 smithi115 bash[46934]: audit 2022-04-22T21:30:26.273868+0000 mon.a (mon.0) 12059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:27.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:27 smithi121 bash[36902]: audit 2022-04-22T21:30:27.567916+0000 mon.a (mon.0) 12060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:30:27.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:27 smithi121 bash[36902]: audit 2022-04-22T21:30:27.570987+0000 mon.a (mon.0) 12061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:30:27.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:27 smithi121 bash[36902]: audit 2022-04-22T21:30:27.572828+0000 mon.a (mon.0) 12062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:30:27.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[45495]: audit 2022-04-22T21:30:27.567916+0000 mon.a (mon.0) 12060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:30:27.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[45495]: audit 2022-04-22T21:30:27.570987+0000 mon.a (mon.0) 12061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:30:27.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[45495]: audit 2022-04-22T21:30:27.572828+0000 mon.a (mon.0) 12062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:30:27.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[46934]: audit 2022-04-22T21:30:27.567916+0000 mon.a (mon.0) 12060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:30:27.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[46934]: audit 2022-04-22T21:30:27.570987+0000 mon.a (mon.0) 12061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:30:27.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:27 smithi115 bash[46934]: audit 2022-04-22T21:30:27.572828+0000 mon.a (mon.0) 12062 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:30:28.227 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:30:28.570 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:30:28.570 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:30:28.571 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3h) 6m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:30:28.572 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:30:28.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:30:28.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:30:28.982 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:30:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:28 smithi121 bash[36902]: cluster 2022-04-22T21:30:27.574596+0000 mgr.y (mgr.34107) 8343 : cluster [DBG] pgmap v9272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:28 smithi121 bash[36902]: audit 2022-04-22T21:30:27.932682+0000 mon.a (mon.0) 12063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:30:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:28 smithi115 bash[45495]: cluster 2022-04-22T21:30:27.574596+0000 mgr.y (mgr.34107) 8343 : cluster [DBG] pgmap v9272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:28 smithi115 bash[45495]: audit 2022-04-22T21:30:27.932682+0000 mon.a (mon.0) 12063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:30:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:28 smithi115 bash[46934]: cluster 2022-04-22T21:30:27.574596+0000 mgr.y (mgr.34107) 8343 : cluster [DBG] pgmap v9272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:28 smithi115 bash[46934]: audit 2022-04-22T21:30:27.932682+0000 mon.a (mon.0) 12063 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:30:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:29 smithi121 bash[36902]: audit 2022-04-22T21:30:28.203317+0000 mgr.y (mgr.34107) 8344 : audit [DBG] from='client.44028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:29 smithi121 bash[36902]: audit 2022-04-22T21:30:28.546462+0000 mgr.y (mgr.34107) 8345 : audit [DBG] from='client.53965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:29 smithi121 bash[36902]: audit 2022-04-22T21:30:28.976359+0000 mon.c (mon.1) 298 : audit [DBG] from='client.? 172.21.15.115:0/3722381216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:30:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:29 smithi121 bash[36902]: audit 2022-04-22T21:30:29.287462+0000 mon.a (mon.0) 12064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[45495]: audit 2022-04-22T21:30:28.203317+0000 mgr.y (mgr.34107) 8344 : audit [DBG] from='client.44028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[45495]: audit 2022-04-22T21:30:28.546462+0000 mgr.y (mgr.34107) 8345 : audit [DBG] from='client.53965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[45495]: audit 2022-04-22T21:30:28.976359+0000 mon.c (mon.1) 298 : audit [DBG] from='client.? 172.21.15.115:0/3722381216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:30:30.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[45495]: audit 2022-04-22T21:30:29.287462+0000 mon.a (mon.0) 12064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:30.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[46934]: audit 2022-04-22T21:30:28.203317+0000 mgr.y (mgr.34107) 8344 : audit [DBG] from='client.44028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[46934]: audit 2022-04-22T21:30:28.546462+0000 mgr.y (mgr.34107) 8345 : audit [DBG] from='client.53965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:30:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[46934]: audit 2022-04-22T21:30:28.976359+0000 mon.c (mon.1) 298 : audit [DBG] from='client.? 172.21.15.115:0/3722381216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:30:30.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:29 smithi115 bash[46934]: audit 2022-04-22T21:30:29.287462+0000 mon.a (mon.0) 12064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:30 smithi121 bash[36902]: cluster 2022-04-22T21:30:29.575720+0000 mgr.y (mgr.34107) 8346 : cluster [DBG] pgmap v9273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:30 smithi121 bash[36902]: audit 2022-04-22T21:30:30.180727+0000 mon.a (mon.0) 12065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:30 smithi115 bash[45495]: cluster 2022-04-22T21:30:29.575720+0000 mgr.y (mgr.34107) 8346 : cluster [DBG] pgmap v9273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:31.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:30 smithi115 bash[45495]: audit 2022-04-22T21:30:30.180727+0000 mon.a (mon.0) 12065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:31.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:30 smithi115 bash[46934]: cluster 2022-04-22T21:30:29.575720+0000 mgr.y (mgr.34107) 8346 : cluster [DBG] pgmap v9273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:31.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:30 smithi115 bash[46934]: audit 2022-04-22T21:30:30.180727+0000 mon.a (mon.0) 12065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:31 smithi121 bash[36902]: audit 2022-04-22T21:30:31.276837+0000 mon.a (mon.0) 12066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:31 smithi115 bash[45495]: audit 2022-04-22T21:30:31.276837+0000 mon.a (mon.0) 12066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:31 smithi115 bash[46934]: audit 2022-04-22T21:30:31.276837+0000 mon.a (mon.0) 12066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:32.956 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:32.566Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:32 smithi121 bash[36902]: cluster 2022-04-22T21:30:31.576695+0000 mgr.y (mgr.34107) 8347 : cluster [DBG] pgmap v9274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:32 smithi115 bash[45495]: cluster 2022-04-22T21:30:31.576695+0000 mgr.y (mgr.34107) 8347 : cluster [DBG] pgmap v9274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:32 smithi115 bash[46934]: cluster 2022-04-22T21:30:31.576695+0000 mgr.y (mgr.34107) 8347 : cluster [DBG] pgmap v9274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:34.441Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:34 smithi121 bash[36902]: cluster 2022-04-22T21:30:33.577824+0000 mgr.y (mgr.34107) 8348 : cluster [DBG] pgmap v9275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:34 smithi121 bash[36902]: audit 2022-04-22T21:30:34.291004+0000 mon.a (mon.0) 12067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:34 smithi115 bash[45495]: cluster 2022-04-22T21:30:33.577824+0000 mgr.y (mgr.34107) 8348 : cluster [DBG] pgmap v9275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:34 smithi115 bash[45495]: audit 2022-04-22T21:30:34.291004+0000 mon.a (mon.0) 12067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:34 smithi115 bash[46934]: cluster 2022-04-22T21:30:33.577824+0000 mgr.y (mgr.34107) 8348 : cluster [DBG] pgmap v9275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:34 smithi115 bash[46934]: audit 2022-04-22T21:30:34.291004+0000 mon.a (mon.0) 12067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:35 smithi121 bash[36902]: audit 2022-04-22T21:30:35.184596+0000 mon.a (mon.0) 12068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:35 smithi115 bash[45495]: audit 2022-04-22T21:30:35.184596+0000 mon.a (mon.0) 12068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:35 smithi115 bash[46934]: audit 2022-04-22T21:30:35.184596+0000 mon.a (mon.0) 12068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:36 smithi121 bash[36902]: cluster 2022-04-22T21:30:35.578753+0000 mgr.y (mgr.34107) 8349 : cluster [DBG] pgmap v9276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:36 smithi121 bash[36902]: audit 2022-04-22T21:30:36.280007+0000 mon.a (mon.0) 12069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:36 smithi115 bash[45495]: cluster 2022-04-22T21:30:35.578753+0000 mgr.y (mgr.34107) 8349 : cluster [DBG] pgmap v9276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:36 smithi115 bash[45495]: audit 2022-04-22T21:30:36.280007+0000 mon.a (mon.0) 12069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:36 smithi115 bash[46934]: cluster 2022-04-22T21:30:35.578753+0000 mgr.y (mgr.34107) 8349 : cluster [DBG] pgmap v9276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:36 smithi115 bash[46934]: audit 2022-04-22T21:30:36.280007+0000 mon.a (mon.0) 12069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:38.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:37 smithi121 bash[36902]: cluster 2022-04-22T21:30:37.579848+0000 mgr.y (mgr.34107) 8350 : cluster [DBG] pgmap v9277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:37 smithi115 bash[45495]: cluster 2022-04-22T21:30:37.579848+0000 mgr.y (mgr.34107) 8350 : cluster [DBG] pgmap v9277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:37 smithi115 bash[46934]: cluster 2022-04-22T21:30:37.579848+0000 mgr.y (mgr.34107) 8350 : cluster [DBG] pgmap v9277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:39 smithi121 bash[36902]: audit 2022-04-22T21:30:39.294104+0000 mon.a (mon.0) 12070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:39 smithi115 bash[45495]: audit 2022-04-22T21:30:39.294104+0000 mon.a (mon.0) 12070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:39 smithi115 bash[46934]: audit 2022-04-22T21:30:39.294104+0000 mon.a (mon.0) 12070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:40 smithi121 bash[36902]: cluster 2022-04-22T21:30:39.580867+0000 mgr.y (mgr.34107) 8351 : cluster [DBG] pgmap v9278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:40.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:40 smithi121 bash[36902]: audit 2022-04-22T21:30:40.188385+0000 mon.a (mon.0) 12071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:40 smithi115 bash[46934]: cluster 2022-04-22T21:30:39.580867+0000 mgr.y (mgr.34107) 8351 : cluster [DBG] pgmap v9278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:40 smithi115 bash[46934]: audit 2022-04-22T21:30:40.188385+0000 mon.a (mon.0) 12071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:40 smithi115 bash[45495]: cluster 2022-04-22T21:30:39.580867+0000 mgr.y (mgr.34107) 8351 : cluster [DBG] pgmap v9278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:40 smithi115 bash[45495]: audit 2022-04-22T21:30:40.188385+0000 mon.a (mon.0) 12071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:41.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:41 smithi121 bash[36902]: audit 2022-04-22T21:30:41.283252+0000 mon.a (mon.0) 12072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:41 smithi115 bash[45495]: audit 2022-04-22T21:30:41.283252+0000 mon.a (mon.0) 12072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:41 smithi115 bash[46934]: audit 2022-04-22T21:30:41.283252+0000 mon.a (mon.0) 12072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:42 smithi121 bash[36902]: cluster 2022-04-22T21:30:41.581910+0000 mgr.y (mgr.34107) 8352 : cluster [DBG] pgmap v9279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:42 smithi115 bash[45495]: cluster 2022-04-22T21:30:41.581910+0000 mgr.y (mgr.34107) 8352 : cluster [DBG] pgmap v9279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:42.567Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:42 smithi115 bash[46934]: cluster 2022-04-22T21:30:41.581910+0000 mgr.y (mgr.34107) 8352 : cluster [DBG] pgmap v9279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:44.439Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:44 smithi115 bash[45495]: cluster 2022-04-22T21:30:43.583003+0000 mgr.y (mgr.34107) 8353 : cluster [DBG] pgmap v9280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:44 smithi115 bash[45495]: audit 2022-04-22T21:30:44.297280+0000 mon.a (mon.0) 12073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:44 smithi115 bash[46934]: cluster 2022-04-22T21:30:43.583003+0000 mgr.y (mgr.34107) 8353 : cluster [DBG] pgmap v9280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:44 smithi115 bash[46934]: audit 2022-04-22T21:30:44.297280+0000 mon.a (mon.0) 12073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:44 smithi121 bash[36902]: cluster 2022-04-22T21:30:43.583003+0000 mgr.y (mgr.34107) 8353 : cluster [DBG] pgmap v9280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:44 smithi121 bash[36902]: audit 2022-04-22T21:30:44.297280+0000 mon.a (mon.0) 12073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:45 smithi121 bash[36902]: audit 2022-04-22T21:30:45.192144+0000 mon.a (mon.0) 12074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:45 smithi115 bash[45495]: audit 2022-04-22T21:30:45.192144+0000 mon.a (mon.0) 12074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:45 smithi115 bash[46934]: audit 2022-04-22T21:30:45.192144+0000 mon.a (mon.0) 12074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:46 smithi121 bash[36902]: cluster 2022-04-22T21:30:45.584123+0000 mgr.y (mgr.34107) 8354 : cluster [DBG] pgmap v9281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:46 smithi121 bash[36902]: audit 2022-04-22T21:30:46.286469+0000 mon.a (mon.0) 12075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:46 smithi115 bash[45495]: cluster 2022-04-22T21:30:45.584123+0000 mgr.y (mgr.34107) 8354 : cluster [DBG] pgmap v9281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:46 smithi115 bash[45495]: audit 2022-04-22T21:30:46.286469+0000 mon.a (mon.0) 12075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:46 smithi115 bash[46934]: cluster 2022-04-22T21:30:45.584123+0000 mgr.y (mgr.34107) 8354 : cluster [DBG] pgmap v9281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:46 smithi115 bash[46934]: audit 2022-04-22T21:30:46.286469+0000 mon.a (mon.0) 12075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:48 smithi121 bash[36902]: cluster 2022-04-22T21:30:47.585228+0000 mgr.y (mgr.34107) 8355 : cluster [DBG] pgmap v9282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:48 smithi115 bash[45495]: cluster 2022-04-22T21:30:47.585228+0000 mgr.y (mgr.34107) 8355 : cluster [DBG] pgmap v9282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:48 smithi115 bash[46934]: cluster 2022-04-22T21:30:47.585228+0000 mgr.y (mgr.34107) 8355 : cluster [DBG] pgmap v9282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:49 smithi121 bash[36902]: audit 2022-04-22T21:30:49.300465+0000 mon.a (mon.0) 12076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:50.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:49 smithi115 bash[45495]: audit 2022-04-22T21:30:49.300465+0000 mon.a (mon.0) 12076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:49 smithi115 bash[46934]: audit 2022-04-22T21:30:49.300465+0000 mon.a (mon.0) 12076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:51.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:50 smithi121 bash[36902]: cluster 2022-04-22T21:30:49.586309+0000 mgr.y (mgr.34107) 8356 : cluster [DBG] pgmap v9283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:50 smithi121 bash[36902]: audit 2022-04-22T21:30:50.195985+0000 mon.a (mon.0) 12077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:50 smithi115 bash[45495]: cluster 2022-04-22T21:30:49.586309+0000 mgr.y (mgr.34107) 8356 : cluster [DBG] pgmap v9283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:50 smithi115 bash[45495]: audit 2022-04-22T21:30:50.195985+0000 mon.a (mon.0) 12077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:50 smithi115 bash[46934]: cluster 2022-04-22T21:30:49.586309+0000 mgr.y (mgr.34107) 8356 : cluster [DBG] pgmap v9283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:50 smithi115 bash[46934]: audit 2022-04-22T21:30:50.195985+0000 mon.a (mon.0) 12077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:51 smithi121 bash[36902]: audit 2022-04-22T21:30:51.289533+0000 mon.a (mon.0) 12078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:51 smithi115 bash[46934]: audit 2022-04-22T21:30:51.289533+0000 mon.a (mon.0) 12078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:51 smithi115 bash[45495]: audit 2022-04-22T21:30:51.289533+0000 mon.a (mon.0) 12078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:53.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:52 smithi121 bash[36902]: cluster 2022-04-22T21:30:51.587316+0000 mgr.y (mgr.34107) 8357 : cluster [DBG] pgmap v9284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:52 smithi115 bash[45495]: cluster 2022-04-22T21:30:51.587316+0000 mgr.y (mgr.34107) 8357 : cluster [DBG] pgmap v9284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:52 smithi115 bash[46934]: cluster 2022-04-22T21:30:51.587316+0000 mgr.y (mgr.34107) 8357 : cluster [DBG] pgmap v9284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:52.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:54 smithi115 bash[45495]: cluster 2022-04-22T21:30:53.588481+0000 mgr.y (mgr.34107) 8358 : cluster [DBG] pgmap v9285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:54 smithi115 bash[45495]: audit 2022-04-22T21:30:54.303460+0000 mon.a (mon.0) 12079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:30:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:30:54.441Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:30:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:54 smithi115 bash[46934]: cluster 2022-04-22T21:30:53.588481+0000 mgr.y (mgr.34107) 8358 : cluster [DBG] pgmap v9285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:54 smithi115 bash[46934]: audit 2022-04-22T21:30:54.303460+0000 mon.a (mon.0) 12079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:55.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:54 smithi121 bash[36902]: cluster 2022-04-22T21:30:53.588481+0000 mgr.y (mgr.34107) 8358 : cluster [DBG] pgmap v9285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:54 smithi121 bash[36902]: audit 2022-04-22T21:30:54.303460+0000 mon.a (mon.0) 12079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:30:56.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:55 smithi121 bash[36902]: audit 2022-04-22T21:30:55.199450+0000 mon.a (mon.0) 12080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:55 smithi115 bash[45495]: audit 2022-04-22T21:30:55.199450+0000 mon.a (mon.0) 12080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:55 smithi115 bash[46934]: audit 2022-04-22T21:30:55.199450+0000 mon.a (mon.0) 12080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:30:57.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:56 smithi121 bash[36902]: cluster 2022-04-22T21:30:55.589540+0000 mgr.y (mgr.34107) 8359 : cluster [DBG] pgmap v9286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:56 smithi121 bash[36902]: audit 2022-04-22T21:30:56.292741+0000 mon.a (mon.0) 12081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:56 smithi115 bash[46934]: cluster 2022-04-22T21:30:55.589540+0000 mgr.y (mgr.34107) 8359 : cluster [DBG] pgmap v9286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:56 smithi115 bash[46934]: audit 2022-04-22T21:30:56.292741+0000 mon.a (mon.0) 12081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:56 smithi115 bash[45495]: cluster 2022-04-22T21:30:55.589540+0000 mgr.y (mgr.34107) 8359 : cluster [DBG] pgmap v9286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:57.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:56 smithi115 bash[45495]: audit 2022-04-22T21:30:56.292741+0000 mon.a (mon.0) 12081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:30:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:58 smithi121 bash[36902]: cluster 2022-04-22T21:30:57.590594+0000 mgr.y (mgr.34107) 8360 : cluster [DBG] pgmap v9287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:58 smithi115 bash[45495]: cluster 2022-04-22T21:30:57.590594+0000 mgr.y (mgr.34107) 8360 : cluster [DBG] pgmap v9287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:58 smithi115 bash[46934]: cluster 2022-04-22T21:30:57.590594+0000 mgr.y (mgr.34107) 8360 : cluster [DBG] pgmap v9287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:30:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:30:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:30:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 6m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:30:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:30:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:30:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:30:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:30:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:30:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:30:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:30:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:30:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 6m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:30:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 7m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:30:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:30:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:30:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3h) 6m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:30:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 6m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:30:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 7m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:30:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:30:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 7m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:30:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 7m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:30:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 7m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:31:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:30:59 smithi121 bash[36902]: audit 2022-04-22T21:30:59.306423+0000 mon.a (mon.0) 12082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:30:59 smithi115 bash[46934]: audit 2022-04-22T21:30:59.306423+0000 mon.a (mon.0) 12082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:30:59 smithi115 bash[45495]: audit 2022-04-22T21:30:59.306423+0000 mon.a (mon.0) 12082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:31:00.094 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:31:00.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:31:00.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:00.096 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:31:00.096 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:31:00.096 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:31:00.096 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:00 smithi121 bash[36902]: audit 2022-04-22T21:30:59.334813+0000 mgr.y (mgr.34107) 8361 : audit [DBG] from='client.53977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:00 smithi121 bash[36902]: cluster 2022-04-22T21:30:59.591674+0000 mgr.y (mgr.34107) 8362 : cluster [DBG] pgmap v9288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:00 smithi121 bash[36902]: audit 2022-04-22T21:30:59.671347+0000 mgr.y (mgr.34107) 8363 : audit [DBG] from='client.44052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:00 smithi121 bash[36902]: audit 2022-04-22T21:31:00.090727+0000 mon.a (mon.0) 12083 : audit [DBG] from='client.? 172.21.15.115:0/2591731973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:00 smithi121 bash[36902]: audit 2022-04-22T21:31:00.202901+0000 mon.a (mon.0) 12084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[46934]: audit 2022-04-22T21:30:59.334813+0000 mgr.y (mgr.34107) 8361 : audit [DBG] from='client.53977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[46934]: cluster 2022-04-22T21:30:59.591674+0000 mgr.y (mgr.34107) 8362 : cluster [DBG] pgmap v9288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[46934]: audit 2022-04-22T21:30:59.671347+0000 mgr.y (mgr.34107) 8363 : audit [DBG] from='client.44052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[46934]: audit 2022-04-22T21:31:00.090727+0000 mon.a (mon.0) 12083 : audit [DBG] from='client.? 172.21.15.115:0/2591731973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[46934]: audit 2022-04-22T21:31:00.202901+0000 mon.a (mon.0) 12084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[45495]: audit 2022-04-22T21:30:59.334813+0000 mgr.y (mgr.34107) 8361 : audit [DBG] from='client.53977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[45495]: cluster 2022-04-22T21:30:59.591674+0000 mgr.y (mgr.34107) 8362 : cluster [DBG] pgmap v9288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[45495]: audit 2022-04-22T21:30:59.671347+0000 mgr.y (mgr.34107) 8363 : audit [DBG] from='client.44052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[45495]: audit 2022-04-22T21:31:00.090727+0000 mon.a (mon.0) 12083 : audit [DBG] from='client.? 172.21.15.115:0/2591731973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:01.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:00 smithi115 bash[45495]: audit 2022-04-22T21:31:00.202901+0000 mon.a (mon.0) 12084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:02.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:01 smithi121 bash[36902]: audit 2022-04-22T21:31:01.295791+0000 mon.a (mon.0) 12085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:01 smithi115 bash[45495]: audit 2022-04-22T21:31:01.295791+0000 mon.a (mon.0) 12085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:01 smithi115 bash[46934]: audit 2022-04-22T21:31:01.295791+0000 mon.a (mon.0) 12085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:02 smithi121 bash[36902]: cluster 2022-04-22T21:31:01.592779+0000 mgr.y (mgr.34107) 8364 : cluster [DBG] pgmap v9289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:02 smithi115 bash[46934]: cluster 2022-04-22T21:31:01.592779+0000 mgr.y (mgr.34107) 8364 : cluster [DBG] pgmap v9289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:02.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:02 smithi115 bash[45495]: cluster 2022-04-22T21:31:01.592779+0000 mgr.y (mgr.34107) 8364 : cluster [DBG] pgmap v9289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:04.715 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:04.440Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:04 smithi121 bash[36902]: cluster 2022-04-22T21:31:03.593980+0000 mgr.y (mgr.34107) 8365 : cluster [DBG] pgmap v9290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:04 smithi121 bash[36902]: audit 2022-04-22T21:31:04.309241+0000 mon.a (mon.0) 12086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:04 smithi115 bash[45495]: cluster 2022-04-22T21:31:03.593980+0000 mgr.y (mgr.34107) 8365 : cluster [DBG] pgmap v9290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:04 smithi115 bash[45495]: audit 2022-04-22T21:31:04.309241+0000 mon.a (mon.0) 12086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:04 smithi115 bash[46934]: cluster 2022-04-22T21:31:03.593980+0000 mgr.y (mgr.34107) 8365 : cluster [DBG] pgmap v9290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:04 smithi115 bash[46934]: audit 2022-04-22T21:31:04.309241+0000 mon.a (mon.0) 12086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:06.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:05 smithi121 bash[36902]: audit 2022-04-22T21:31:05.206229+0000 mon.a (mon.0) 12087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:05 smithi115 bash[45495]: audit 2022-04-22T21:31:05.206229+0000 mon.a (mon.0) 12087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:05 smithi115 bash[46934]: audit 2022-04-22T21:31:05.206229+0000 mon.a (mon.0) 12087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:06 smithi121 bash[36902]: cluster 2022-04-22T21:31:05.595062+0000 mgr.y (mgr.34107) 8366 : cluster [DBG] pgmap v9291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:06 smithi121 bash[36902]: audit 2022-04-22T21:31:06.298961+0000 mon.a (mon.0) 12088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:06 smithi115 bash[45495]: cluster 2022-04-22T21:31:05.595062+0000 mgr.y (mgr.34107) 8366 : cluster [DBG] pgmap v9291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:06 smithi115 bash[45495]: audit 2022-04-22T21:31:06.298961+0000 mon.a (mon.0) 12088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:06 smithi115 bash[46934]: cluster 2022-04-22T21:31:05.595062+0000 mgr.y (mgr.34107) 8366 : cluster [DBG] pgmap v9291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:06 smithi115 bash[46934]: audit 2022-04-22T21:31:06.298961+0000 mon.a (mon.0) 12088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:09.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:08 smithi121 bash[36902]: cluster 2022-04-22T21:31:07.596240+0000 mgr.y (mgr.34107) 8367 : cluster [DBG] pgmap v9292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:08 smithi115 bash[45495]: cluster 2022-04-22T21:31:07.596240+0000 mgr.y (mgr.34107) 8367 : cluster [DBG] pgmap v9292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:08 smithi115 bash[46934]: cluster 2022-04-22T21:31:07.596240+0000 mgr.y (mgr.34107) 8367 : cluster [DBG] pgmap v9292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:10.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:09 smithi121 bash[36902]: audit 2022-04-22T21:31:09.312373+0000 mon.a (mon.0) 12089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:09 smithi115 bash[45495]: audit 2022-04-22T21:31:09.312373+0000 mon.a (mon.0) 12089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:09 smithi115 bash[46934]: audit 2022-04-22T21:31:09.312373+0000 mon.a (mon.0) 12089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:11.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:10 smithi121 bash[36902]: cluster 2022-04-22T21:31:09.597323+0000 mgr.y (mgr.34107) 8368 : cluster [DBG] pgmap v9293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:10 smithi121 bash[36902]: audit 2022-04-22T21:31:10.210059+0000 mon.a (mon.0) 12090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:10 smithi115 bash[45495]: cluster 2022-04-22T21:31:09.597323+0000 mgr.y (mgr.34107) 8368 : cluster [DBG] pgmap v9293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:10 smithi115 bash[45495]: audit 2022-04-22T21:31:10.210059+0000 mon.a (mon.0) 12090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:10 smithi115 bash[46934]: cluster 2022-04-22T21:31:09.597323+0000 mgr.y (mgr.34107) 8368 : cluster [DBG] pgmap v9293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:10 smithi115 bash[46934]: audit 2022-04-22T21:31:10.210059+0000 mon.a (mon.0) 12090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:11 smithi121 bash[36902]: audit 2022-04-22T21:31:11.302069+0000 mon.a (mon.0) 12091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:11 smithi115 bash[45495]: audit 2022-04-22T21:31:11.302069+0000 mon.a (mon.0) 12091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:11 smithi115 bash[46934]: audit 2022-04-22T21:31:11.302069+0000 mon.a (mon.0) 12091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:13.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:12 smithi121 bash[36902]: cluster 2022-04-22T21:31:11.598390+0000 mgr.y (mgr.34107) 8369 : cluster [DBG] pgmap v9294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:12 smithi115 bash[45495]: cluster 2022-04-22T21:31:11.598390+0000 mgr.y (mgr.34107) 8369 : cluster [DBG] pgmap v9294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:12 smithi115 bash[46934]: cluster 2022-04-22T21:31:11.598390+0000 mgr.y (mgr.34107) 8369 : cluster [DBG] pgmap v9294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:12.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:14.752 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:14.444Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:14 smithi121 bash[36902]: cluster 2022-04-22T21:31:13.599551+0000 mgr.y (mgr.34107) 8370 : cluster [DBG] pgmap v9295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:15.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:14 smithi121 bash[36902]: audit 2022-04-22T21:31:14.315647+0000 mon.a (mon.0) 12092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:14 smithi115 bash[45495]: cluster 2022-04-22T21:31:13.599551+0000 mgr.y (mgr.34107) 8370 : cluster [DBG] pgmap v9295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:14 smithi115 bash[45495]: audit 2022-04-22T21:31:14.315647+0000 mon.a (mon.0) 12092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:14 smithi115 bash[46934]: cluster 2022-04-22T21:31:13.599551+0000 mgr.y (mgr.34107) 8370 : cluster [DBG] pgmap v9295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:14 smithi115 bash[46934]: audit 2022-04-22T21:31:14.315647+0000 mon.a (mon.0) 12092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:16.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:15 smithi121 bash[36902]: audit 2022-04-22T21:31:15.214121+0000 mon.a (mon.0) 12093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:15 smithi115 bash[46934]: audit 2022-04-22T21:31:15.214121+0000 mon.a (mon.0) 12093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:15 smithi115 bash[45495]: audit 2022-04-22T21:31:15.214121+0000 mon.a (mon.0) 12093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:17.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:16 smithi121 bash[36902]: cluster 2022-04-22T21:31:15.600547+0000 mgr.y (mgr.34107) 8371 : cluster [DBG] pgmap v9296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:16 smithi121 bash[36902]: audit 2022-04-22T21:31:16.086680+0000 mon.a (mon.0) 12094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:31:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:16 smithi121 bash[36902]: audit 2022-04-22T21:31:16.087262+0000 mon.a (mon.0) 12095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:31:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:16 smithi121 bash[36902]: audit 2022-04-22T21:31:16.304978+0000 mon.a (mon.0) 12096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[45495]: cluster 2022-04-22T21:31:15.600547+0000 mgr.y (mgr.34107) 8371 : cluster [DBG] pgmap v9296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[45495]: audit 2022-04-22T21:31:16.086680+0000 mon.a (mon.0) 12094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:31:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[45495]: audit 2022-04-22T21:31:16.087262+0000 mon.a (mon.0) 12095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:31:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[45495]: audit 2022-04-22T21:31:16.304978+0000 mon.a (mon.0) 12096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[46934]: cluster 2022-04-22T21:31:15.600547+0000 mgr.y (mgr.34107) 8371 : cluster [DBG] pgmap v9296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[46934]: audit 2022-04-22T21:31:16.086680+0000 mon.a (mon.0) 12094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:31:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[46934]: audit 2022-04-22T21:31:16.087262+0000 mon.a (mon.0) 12095 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:31:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:16 smithi115 bash[46934]: audit 2022-04-22T21:31:16.304978+0000 mon.a (mon.0) 12096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:18 smithi121 bash[36902]: cluster 2022-04-22T21:31:17.601605+0000 mgr.y (mgr.34107) 8372 : cluster [DBG] pgmap v9297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:18 smithi115 bash[46934]: cluster 2022-04-22T21:31:17.601605+0000 mgr.y (mgr.34107) 8372 : cluster [DBG] pgmap v9297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:18 smithi115 bash[45495]: cluster 2022-04-22T21:31:17.601605+0000 mgr.y (mgr.34107) 8372 : cluster [DBG] pgmap v9297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:19 smithi115 bash[46934]: audit 2022-04-22T21:31:19.318875+0000 mon.a (mon.0) 12097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:19 smithi115 bash[45495]: audit 2022-04-22T21:31:19.318875+0000 mon.a (mon.0) 12097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:19 smithi121 bash[36902]: audit 2022-04-22T21:31:19.318875+0000 mon.a (mon.0) 12097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:20 smithi115 bash[45495]: cluster 2022-04-22T21:31:19.602659+0000 mgr.y (mgr.34107) 8373 : cluster [DBG] pgmap v9298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:21.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:20 smithi115 bash[45495]: audit 2022-04-22T21:31:20.217891+0000 mon.a (mon.0) 12098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:20 smithi115 bash[46934]: cluster 2022-04-22T21:31:19.602659+0000 mgr.y (mgr.34107) 8373 : cluster [DBG] pgmap v9298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:21.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:20 smithi115 bash[46934]: audit 2022-04-22T21:31:20.217891+0000 mon.a (mon.0) 12098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:20 smithi121 bash[36902]: cluster 2022-04-22T21:31:19.602659+0000 mgr.y (mgr.34107) 8373 : cluster [DBG] pgmap v9298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:21.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:20 smithi121 bash[36902]: audit 2022-04-22T21:31:20.217891+0000 mon.a (mon.0) 12098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:21 smithi115 bash[45495]: audit 2022-04-22T21:31:21.308072+0000 mon.a (mon.0) 12099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:21 smithi115 bash[46934]: audit 2022-04-22T21:31:21.308072+0000 mon.a (mon.0) 12099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:21 smithi121 bash[36902]: audit 2022-04-22T21:31:21.308072+0000 mon.a (mon.0) 12099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:22 smithi115 bash[45495]: cluster 2022-04-22T21:31:21.603775+0000 mgr.y (mgr.34107) 8374 : cluster [DBG] pgmap v9299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:22.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:22 smithi115 bash[46934]: cluster 2022-04-22T21:31:21.603775+0000 mgr.y (mgr.34107) 8374 : cluster [DBG] pgmap v9299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:22 smithi121 bash[36902]: cluster 2022-04-22T21:31:21.603775+0000 mgr.y (mgr.34107) 8374 : cluster [DBG] pgmap v9299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:24.788 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:24.444Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:24 smithi115 bash[45495]: cluster 2022-04-22T21:31:23.604822+0000 mgr.y (mgr.34107) 8375 : cluster [DBG] pgmap v9300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:25.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:24 smithi115 bash[45495]: audit 2022-04-22T21:31:24.322128+0000 mon.a (mon.0) 12100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:25.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:24 smithi115 bash[46934]: cluster 2022-04-22T21:31:23.604822+0000 mgr.y (mgr.34107) 8375 : cluster [DBG] pgmap v9300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:25.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:24 smithi115 bash[46934]: audit 2022-04-22T21:31:24.322128+0000 mon.a (mon.0) 12100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:25.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:24 smithi121 bash[36902]: cluster 2022-04-22T21:31:23.604822+0000 mgr.y (mgr.34107) 8375 : cluster [DBG] pgmap v9300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:24 smithi121 bash[36902]: audit 2022-04-22T21:31:24.322128+0000 mon.a (mon.0) 12100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:25 smithi115 bash[45495]: audit 2022-04-22T21:31:25.221766+0000 mon.a (mon.0) 12101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:25 smithi115 bash[46934]: audit 2022-04-22T21:31:25.221766+0000 mon.a (mon.0) 12101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:25 smithi121 bash[36902]: audit 2022-04-22T21:31:25.221766+0000 mon.a (mon.0) 12101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:26 smithi121 bash[36902]: cluster 2022-04-22T21:31:25.605887+0000 mgr.y (mgr.34107) 8376 : cluster [DBG] pgmap v9301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:26 smithi121 bash[36902]: audit 2022-04-22T21:31:26.310984+0000 mon.a (mon.0) 12102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:26 smithi115 bash[46934]: cluster 2022-04-22T21:31:25.605887+0000 mgr.y (mgr.34107) 8376 : cluster [DBG] pgmap v9301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:26 smithi115 bash[46934]: audit 2022-04-22T21:31:26.310984+0000 mon.a (mon.0) 12102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:26 smithi115 bash[45495]: cluster 2022-04-22T21:31:25.605887+0000 mgr.y (mgr.34107) 8376 : cluster [DBG] pgmap v9301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:27.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:26 smithi115 bash[45495]: audit 2022-04-22T21:31:26.310984+0000 mon.a (mon.0) 12102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:28 smithi121 bash[36902]: cluster 2022-04-22T21:31:27.606968+0000 mgr.y (mgr.34107) 8377 : cluster [DBG] pgmap v9302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:28 smithi121 bash[36902]: audit 2022-04-22T21:31:27.939960+0000 mon.a (mon.0) 12103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:31:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:28 smithi121 bash[36902]: audit 2022-04-22T21:31:27.943234+0000 mon.a (mon.0) 12104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:31:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:28 smithi121 bash[36902]: audit 2022-04-22T21:31:27.945187+0000 mon.a (mon.0) 12105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:31:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:28 smithi121 bash[36902]: audit 2022-04-22T21:31:28.274476+0000 mon.a (mon.0) 12106 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[45495]: cluster 2022-04-22T21:31:27.606968+0000 mgr.y (mgr.34107) 8377 : cluster [DBG] pgmap v9302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[45495]: audit 2022-04-22T21:31:27.939960+0000 mon.a (mon.0) 12103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[45495]: audit 2022-04-22T21:31:27.943234+0000 mon.a (mon.0) 12104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[45495]: audit 2022-04-22T21:31:27.945187+0000 mon.a (mon.0) 12105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[45495]: audit 2022-04-22T21:31:28.274476+0000 mon.a (mon.0) 12106 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:31:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[46934]: cluster 2022-04-22T21:31:27.606968+0000 mgr.y (mgr.34107) 8377 : cluster [DBG] pgmap v9302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[46934]: audit 2022-04-22T21:31:27.939960+0000 mon.a (mon.0) 12103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:31:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[46934]: audit 2022-04-22T21:31:27.943234+0000 mon.a (mon.0) 12104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:31:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[46934]: audit 2022-04-22T21:31:27.945187+0000 mon.a (mon.0) 12105 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:31:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:28 smithi115 bash[46934]: audit 2022-04-22T21:31:28.274476+0000 mon.a (mon.0) 12106 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:31:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:29 smithi121 bash[36902]: audit 2022-04-22T21:31:29.325076+0000 mon.a (mon.0) 12107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:29 smithi115 bash[46934]: audit 2022-04-22T21:31:29.325076+0000 mon.a (mon.0) 12107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:29 smithi115 bash[45495]: audit 2022-04-22T21:31:29.325076+0000 mon.a (mon.0) 12107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:30.479 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:31:30.818 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:31:30.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3h) 7m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:31:30.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:31:30.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:31:30.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:31:30.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:31:30.820 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:31:31.226 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:31:31.226 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:31:31.226 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:31:31.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:31:31.227 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:31:31.228 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:31:31.228 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:31:31.228 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:31:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:30 smithi121 bash[36902]: cluster 2022-04-22T21:31:29.608061+0000 mgr.y (mgr.34107) 8378 : cluster [DBG] pgmap v9303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:30 smithi121 bash[36902]: audit 2022-04-22T21:31:30.225299+0000 mon.a (mon.0) 12108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:30 smithi115 bash[45495]: cluster 2022-04-22T21:31:29.608061+0000 mgr.y (mgr.34107) 8378 : cluster [DBG] pgmap v9303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:30 smithi115 bash[45495]: audit 2022-04-22T21:31:30.225299+0000 mon.a (mon.0) 12108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:30 smithi115 bash[46934]: cluster 2022-04-22T21:31:29.608061+0000 mgr.y (mgr.34107) 8378 : cluster [DBG] pgmap v9303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:30 smithi115 bash[46934]: audit 2022-04-22T21:31:30.225299+0000 mon.a (mon.0) 12108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:31 smithi121 bash[36902]: audit 2022-04-22T21:31:30.461055+0000 mgr.y (mgr.34107) 8379 : audit [DBG] from='client.53995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:31 smithi121 bash[36902]: audit 2022-04-22T21:31:30.795442+0000 mgr.y (mgr.34107) 8380 : audit [DBG] from='client.54001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:31 smithi121 bash[36902]: audit 2022-04-22T21:31:31.222561+0000 mon.c (mon.1) 299 : audit [DBG] from='client.? 172.21.15.115:0/1428966389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:31 smithi121 bash[36902]: audit 2022-04-22T21:31:31.313968+0000 mon.a (mon.0) 12109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[45495]: audit 2022-04-22T21:31:30.461055+0000 mgr.y (mgr.34107) 8379 : audit [DBG] from='client.53995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[45495]: audit 2022-04-22T21:31:30.795442+0000 mgr.y (mgr.34107) 8380 : audit [DBG] from='client.54001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[45495]: audit 2022-04-22T21:31:31.222561+0000 mon.c (mon.1) 299 : audit [DBG] from='client.? 172.21.15.115:0/1428966389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[45495]: audit 2022-04-22T21:31:31.313968+0000 mon.a (mon.0) 12109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[46934]: audit 2022-04-22T21:31:30.461055+0000 mgr.y (mgr.34107) 8379 : audit [DBG] from='client.53995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[46934]: audit 2022-04-22T21:31:30.795442+0000 mgr.y (mgr.34107) 8380 : audit [DBG] from='client.54001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:31:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[46934]: audit 2022-04-22T21:31:31.222561+0000 mon.c (mon.1) 299 : audit [DBG] from='client.? 172.21.15.115:0/1428966389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:31:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:31 smithi115 bash[46934]: audit 2022-04-22T21:31:31.313968+0000 mon.a (mon.0) 12109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:32.826 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:32.571Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:32.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:32 smithi115 bash[45495]: cluster 2022-04-22T21:31:31.609134+0000 mgr.y (mgr.34107) 8381 : cluster [DBG] pgmap v9304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:32 smithi121 bash[36902]: cluster 2022-04-22T21:31:31.609134+0000 mgr.y (mgr.34107) 8381 : cluster [DBG] pgmap v9304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:33.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:32 smithi115 bash[46934]: cluster 2022-04-22T21:31:31.609134+0000 mgr.y (mgr.34107) 8381 : cluster [DBG] pgmap v9304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:34.442Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:34 smithi121 bash[36902]: cluster 2022-04-22T21:31:33.610279+0000 mgr.y (mgr.34107) 8382 : cluster [DBG] pgmap v9305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:34 smithi121 bash[36902]: audit 2022-04-22T21:31:34.328054+0000 mon.a (mon.0) 12110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:34 smithi115 bash[45495]: cluster 2022-04-22T21:31:33.610279+0000 mgr.y (mgr.34107) 8382 : cluster [DBG] pgmap v9305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:34 smithi115 bash[45495]: audit 2022-04-22T21:31:34.328054+0000 mon.a (mon.0) 12110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:34 smithi115 bash[46934]: cluster 2022-04-22T21:31:33.610279+0000 mgr.y (mgr.34107) 8382 : cluster [DBG] pgmap v9305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:34 smithi115 bash[46934]: audit 2022-04-22T21:31:34.328054+0000 mon.a (mon.0) 12110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:35 smithi121 bash[36902]: audit 2022-04-22T21:31:35.228898+0000 mon.a (mon.0) 12111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:36.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:35 smithi115 bash[46934]: audit 2022-04-22T21:31:35.228898+0000 mon.a (mon.0) 12111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:35 smithi115 bash[45495]: audit 2022-04-22T21:31:35.228898+0000 mon.a (mon.0) 12111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:36 smithi121 bash[36902]: cluster 2022-04-22T21:31:35.611333+0000 mgr.y (mgr.34107) 8383 : cluster [DBG] pgmap v9306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:36 smithi121 bash[36902]: audit 2022-04-22T21:31:36.316776+0000 mon.a (mon.0) 12112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:36 smithi115 bash[45495]: cluster 2022-04-22T21:31:35.611333+0000 mgr.y (mgr.34107) 8383 : cluster [DBG] pgmap v9306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:36 smithi115 bash[45495]: audit 2022-04-22T21:31:36.316776+0000 mon.a (mon.0) 12112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:36 smithi115 bash[46934]: cluster 2022-04-22T21:31:35.611333+0000 mgr.y (mgr.34107) 8383 : cluster [DBG] pgmap v9306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:36 smithi115 bash[46934]: audit 2022-04-22T21:31:36.316776+0000 mon.a (mon.0) 12112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:38 smithi121 bash[36902]: cluster 2022-04-22T21:31:37.612411+0000 mgr.y (mgr.34107) 8384 : cluster [DBG] pgmap v9307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:38 smithi115 bash[45495]: cluster 2022-04-22T21:31:37.612411+0000 mgr.y (mgr.34107) 8384 : cluster [DBG] pgmap v9307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:38 smithi115 bash[46934]: cluster 2022-04-22T21:31:37.612411+0000 mgr.y (mgr.34107) 8384 : cluster [DBG] pgmap v9307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:39 smithi121 bash[36902]: audit 2022-04-22T21:31:39.331199+0000 mon.a (mon.0) 12113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:39 smithi115 bash[45495]: audit 2022-04-22T21:31:39.331199+0000 mon.a (mon.0) 12113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:39 smithi115 bash[46934]: audit 2022-04-22T21:31:39.331199+0000 mon.a (mon.0) 12113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:40 smithi121 bash[36902]: cluster 2022-04-22T21:31:39.613449+0000 mgr.y (mgr.34107) 8385 : cluster [DBG] pgmap v9308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:41.274 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:40 smithi121 bash[36902]: audit 2022-04-22T21:31:40.232439+0000 mon.a (mon.0) 12114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:40 smithi115 bash[45495]: cluster 2022-04-22T21:31:39.613449+0000 mgr.y (mgr.34107) 8385 : cluster [DBG] pgmap v9308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:40 smithi115 bash[45495]: audit 2022-04-22T21:31:40.232439+0000 mon.a (mon.0) 12114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:40 smithi115 bash[46934]: cluster 2022-04-22T21:31:39.613449+0000 mgr.y (mgr.34107) 8385 : cluster [DBG] pgmap v9308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:40 smithi115 bash[46934]: audit 2022-04-22T21:31:40.232439+0000 mon.a (mon.0) 12114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:42.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:41 smithi121 bash[36902]: audit 2022-04-22T21:31:41.319602+0000 mon.a (mon.0) 12115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:41 smithi115 bash[45495]: audit 2022-04-22T21:31:41.319602+0000 mon.a (mon.0) 12115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:41 smithi115 bash[46934]: audit 2022-04-22T21:31:41.319602+0000 mon.a (mon.0) 12115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:42.875 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:42.571Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:42 smithi121 bash[36902]: cluster 2022-04-22T21:31:41.614518+0000 mgr.y (mgr.34107) 8386 : cluster [DBG] pgmap v9309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:42 smithi115 bash[45495]: cluster 2022-04-22T21:31:41.614518+0000 mgr.y (mgr.34107) 8386 : cluster [DBG] pgmap v9309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:42 smithi115 bash[46934]: cluster 2022-04-22T21:31:41.614518+0000 mgr.y (mgr.34107) 8386 : cluster [DBG] pgmap v9309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:44.444Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:44 smithi121 bash[36902]: cluster 2022-04-22T21:31:43.615661+0000 mgr.y (mgr.34107) 8387 : cluster [DBG] pgmap v9310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:44 smithi121 bash[36902]: audit 2022-04-22T21:31:44.334276+0000 mon.a (mon.0) 12116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:45.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:44 smithi115 bash[45495]: cluster 2022-04-22T21:31:43.615661+0000 mgr.y (mgr.34107) 8387 : cluster [DBG] pgmap v9310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:44 smithi115 bash[45495]: audit 2022-04-22T21:31:44.334276+0000 mon.a (mon.0) 12116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:44 smithi115 bash[46934]: cluster 2022-04-22T21:31:43.615661+0000 mgr.y (mgr.34107) 8387 : cluster [DBG] pgmap v9310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:44 smithi115 bash[46934]: audit 2022-04-22T21:31:44.334276+0000 mon.a (mon.0) 12116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:45 smithi121 bash[36902]: audit 2022-04-22T21:31:45.236025+0000 mon.a (mon.0) 12117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:45 smithi115 bash[45495]: audit 2022-04-22T21:31:45.236025+0000 mon.a (mon.0) 12117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:45 smithi115 bash[46934]: audit 2022-04-22T21:31:45.236025+0000 mon.a (mon.0) 12117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:46 smithi121 bash[36902]: cluster 2022-04-22T21:31:45.616811+0000 mgr.y (mgr.34107) 8388 : cluster [DBG] pgmap v9311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:46 smithi121 bash[36902]: audit 2022-04-22T21:31:46.322792+0000 mon.a (mon.0) 12118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:46 smithi115 bash[45495]: cluster 2022-04-22T21:31:45.616811+0000 mgr.y (mgr.34107) 8388 : cluster [DBG] pgmap v9311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:46 smithi115 bash[45495]: audit 2022-04-22T21:31:46.322792+0000 mon.a (mon.0) 12118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:46 smithi115 bash[46934]: cluster 2022-04-22T21:31:45.616811+0000 mgr.y (mgr.34107) 8388 : cluster [DBG] pgmap v9311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:46 smithi115 bash[46934]: audit 2022-04-22T21:31:46.322792+0000 mon.a (mon.0) 12118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:48 smithi121 bash[36902]: cluster 2022-04-22T21:31:47.617900+0000 mgr.y (mgr.34107) 8389 : cluster [DBG] pgmap v9312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:48 smithi115 bash[45495]: cluster 2022-04-22T21:31:47.617900+0000 mgr.y (mgr.34107) 8389 : cluster [DBG] pgmap v9312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:48 smithi115 bash[46934]: cluster 2022-04-22T21:31:47.617900+0000 mgr.y (mgr.34107) 8389 : cluster [DBG] pgmap v9312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:49 smithi121 bash[36902]: audit 2022-04-22T21:31:49.337506+0000 mon.a (mon.0) 12119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:49 smithi115 bash[45495]: audit 2022-04-22T21:31:49.337506+0000 mon.a (mon.0) 12119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:50.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:49 smithi115 bash[46934]: audit 2022-04-22T21:31:49.337506+0000 mon.a (mon.0) 12119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:50 smithi121 bash[36902]: cluster 2022-04-22T21:31:49.618958+0000 mgr.y (mgr.34107) 8390 : cluster [DBG] pgmap v9313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:50 smithi121 bash[36902]: audit 2022-04-22T21:31:50.239890+0000 mon.a (mon.0) 12120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:50 smithi115 bash[45495]: cluster 2022-04-22T21:31:49.618958+0000 mgr.y (mgr.34107) 8390 : cluster [DBG] pgmap v9313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:50 smithi115 bash[45495]: audit 2022-04-22T21:31:50.239890+0000 mon.a (mon.0) 12120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:50 smithi115 bash[46934]: cluster 2022-04-22T21:31:49.618958+0000 mgr.y (mgr.34107) 8390 : cluster [DBG] pgmap v9313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:50 smithi115 bash[46934]: audit 2022-04-22T21:31:50.239890+0000 mon.a (mon.0) 12120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:51 smithi121 bash[36902]: audit 2022-04-22T21:31:51.325809+0000 mon.a (mon.0) 12121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:51 smithi115 bash[45495]: audit 2022-04-22T21:31:51.325809+0000 mon.a (mon.0) 12121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:51 smithi115 bash[46934]: audit 2022-04-22T21:31:51.325809+0000 mon.a (mon.0) 12121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:52.911 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:52.573Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:52.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:52 smithi115 bash[46934]: cluster 2022-04-22T21:31:51.620006+0000 mgr.y (mgr.34107) 8391 : cluster [DBG] pgmap v9314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:52 smithi121 bash[36902]: cluster 2022-04-22T21:31:51.620006+0000 mgr.y (mgr.34107) 8391 : cluster [DBG] pgmap v9314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:52 smithi115 bash[45495]: cluster 2022-04-22T21:31:51.620006+0000 mgr.y (mgr.34107) 8391 : cluster [DBG] pgmap v9314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:31:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:31:54.446Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:31:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:54 smithi121 bash[36902]: cluster 2022-04-22T21:31:53.621156+0000 mgr.y (mgr.34107) 8392 : cluster [DBG] pgmap v9315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:54 smithi121 bash[36902]: audit 2022-04-22T21:31:54.340557+0000 mon.a (mon.0) 12122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:54 smithi115 bash[45495]: cluster 2022-04-22T21:31:53.621156+0000 mgr.y (mgr.34107) 8392 : cluster [DBG] pgmap v9315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:55.388 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:54 smithi115 bash[45495]: audit 2022-04-22T21:31:54.340557+0000 mon.a (mon.0) 12122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:55.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:54 smithi115 bash[46934]: cluster 2022-04-22T21:31:53.621156+0000 mgr.y (mgr.34107) 8392 : cluster [DBG] pgmap v9315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:55.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:54 smithi115 bash[46934]: audit 2022-04-22T21:31:54.340557+0000 mon.a (mon.0) 12122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:31:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:55 smithi121 bash[36902]: audit 2022-04-22T21:31:55.243409+0000 mon.a (mon.0) 12123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:55 smithi115 bash[45495]: audit 2022-04-22T21:31:55.243409+0000 mon.a (mon.0) 12123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:55 smithi115 bash[46934]: audit 2022-04-22T21:31:55.243409+0000 mon.a (mon.0) 12123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:31:57.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:56 smithi121 bash[36902]: cluster 2022-04-22T21:31:55.622256+0000 mgr.y (mgr.34107) 8393 : cluster [DBG] pgmap v9316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:56 smithi121 bash[36902]: audit 2022-04-22T21:31:56.328996+0000 mon.a (mon.0) 12124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:56 smithi115 bash[45495]: cluster 2022-04-22T21:31:55.622256+0000 mgr.y (mgr.34107) 8393 : cluster [DBG] pgmap v9316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:56 smithi115 bash[45495]: audit 2022-04-22T21:31:56.328996+0000 mon.a (mon.0) 12124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:56 smithi115 bash[46934]: cluster 2022-04-22T21:31:55.622256+0000 mgr.y (mgr.34107) 8393 : cluster [DBG] pgmap v9316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:56 smithi115 bash[46934]: audit 2022-04-22T21:31:56.328996+0000 mon.a (mon.0) 12124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:31:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:58 smithi121 bash[36902]: cluster 2022-04-22T21:31:57.623364+0000 mgr.y (mgr.34107) 8394 : cluster [DBG] pgmap v9317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:58 smithi115 bash[45495]: cluster 2022-04-22T21:31:57.623364+0000 mgr.y (mgr.34107) 8394 : cluster [DBG] pgmap v9317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:31:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:58 smithi115 bash[46934]: cluster 2022-04-22T21:31:57.623364+0000 mgr.y (mgr.34107) 8394 : cluster [DBG] pgmap v9317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:31:59 smithi121 bash[36902]: audit 2022-04-22T21:31:59.343629+0000 mon.a (mon.0) 12125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:31:59 smithi115 bash[45495]: audit 2022-04-22T21:31:59.343629+0000 mon.a (mon.0) 12125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:31:59 smithi115 bash[46934]: audit 2022-04-22T21:31:59.343629+0000 mon.a (mon.0) 12125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:00 smithi121 bash[36902]: cluster 2022-04-22T21:31:59.624347+0000 mgr.y (mgr.34107) 8395 : cluster [DBG] pgmap v9318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:00 smithi121 bash[36902]: audit 2022-04-22T21:32:00.247012+0000 mon.a (mon.0) 12126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:00 smithi115 bash[45495]: cluster 2022-04-22T21:31:59.624347+0000 mgr.y (mgr.34107) 8395 : cluster [DBG] pgmap v9318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:01.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:00 smithi115 bash[45495]: audit 2022-04-22T21:32:00.247012+0000 mon.a (mon.0) 12126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:01.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:00 smithi115 bash[46934]: cluster 2022-04-22T21:31:59.624347+0000 mgr.y (mgr.34107) 8395 : cluster [DBG] pgmap v9318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:01.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:00 smithi115 bash[46934]: audit 2022-04-22T21:32:00.247012+0000 mon.a (mon.0) 12126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:01.607 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:32:01.949 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:32:01.949 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 7m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:32:01.950 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 7m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 8m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (3h) 8m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 8m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 8m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:32:01.951 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 8m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:32:02.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:01 smithi121 bash[36902]: audit 2022-04-22T21:32:01.332018+0000 mon.a (mon.0) 12127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:01 smithi115 bash[45495]: audit 2022-04-22T21:32:01.332018+0000 mon.a (mon.0) 12127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:01 smithi115 bash[46934]: audit 2022-04-22T21:32:01.332018+0000 mon.a (mon.0) 12127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:02.353 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:32:02.354 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:32:02.354 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:32:02.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:02.355 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:32:02.355 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:32:02.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:02.356 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:32:02.356 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:32:02.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:02.357 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:32:02.357 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:32:02.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:02.358 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:32:02.358 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:32:02.359 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:32:02.359 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:32:02.943 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:02.574Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:02 smithi121 bash[36902]: audit 2022-04-22T21:32:01.589034+0000 mgr.y (mgr.34107) 8396 : audit [DBG] from='client.44082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:02 smithi121 bash[36902]: cluster 2022-04-22T21:32:01.625320+0000 mgr.y (mgr.34107) 8397 : cluster [DBG] pgmap v9319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:02 smithi121 bash[36902]: audit 2022-04-22T21:32:01.930172+0000 mgr.y (mgr.34107) 8398 : audit [DBG] from='client.44088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:02 smithi121 bash[36902]: audit 2022-04-22T21:32:02.348745+0000 mon.c (mon.1) 300 : audit [DBG] from='client.? 172.21.15.115:0/1928153255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[45495]: audit 2022-04-22T21:32:01.589034+0000 mgr.y (mgr.34107) 8396 : audit [DBG] from='client.44082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[45495]: cluster 2022-04-22T21:32:01.625320+0000 mgr.y (mgr.34107) 8397 : cluster [DBG] pgmap v9319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[45495]: audit 2022-04-22T21:32:01.930172+0000 mgr.y (mgr.34107) 8398 : audit [DBG] from='client.44088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[45495]: audit 2022-04-22T21:32:02.348745+0000 mon.c (mon.1) 300 : audit [DBG] from='client.? 172.21.15.115:0/1928153255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[46934]: audit 2022-04-22T21:32:01.589034+0000 mgr.y (mgr.34107) 8396 : audit [DBG] from='client.44082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[46934]: cluster 2022-04-22T21:32:01.625320+0000 mgr.y (mgr.34107) 8397 : cluster [DBG] pgmap v9319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[46934]: audit 2022-04-22T21:32:01.930172+0000 mgr.y (mgr.34107) 8398 : audit [DBG] from='client.44088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:02 smithi115 bash[46934]: audit 2022-04-22T21:32:02.348745+0000 mon.c (mon.1) 300 : audit [DBG] from='client.? 172.21.15.115:0/1928153255' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:04.446Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:04 smithi121 bash[36902]: cluster 2022-04-22T21:32:03.626460+0000 mgr.y (mgr.34107) 8399 : cluster [DBG] pgmap v9320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:04 smithi121 bash[36902]: audit 2022-04-22T21:32:04.346637+0000 mon.a (mon.0) 12128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:04 smithi115 bash[45495]: cluster 2022-04-22T21:32:03.626460+0000 mgr.y (mgr.34107) 8399 : cluster [DBG] pgmap v9320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:04 smithi115 bash[45495]: audit 2022-04-22T21:32:04.346637+0000 mon.a (mon.0) 12128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:04 smithi115 bash[46934]: cluster 2022-04-22T21:32:03.626460+0000 mgr.y (mgr.34107) 8399 : cluster [DBG] pgmap v9320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:04 smithi115 bash[46934]: audit 2022-04-22T21:32:04.346637+0000 mon.a (mon.0) 12128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:05 smithi121 bash[36902]: audit 2022-04-22T21:32:05.252374+0000 mon.a (mon.0) 12129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:05 smithi115 bash[45495]: audit 2022-04-22T21:32:05.252374+0000 mon.a (mon.0) 12129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:05 smithi115 bash[46934]: audit 2022-04-22T21:32:05.252374+0000 mon.a (mon.0) 12129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:06 smithi121 bash[36902]: cluster 2022-04-22T21:32:05.627583+0000 mgr.y (mgr.34107) 8400 : cluster [DBG] pgmap v9321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:06 smithi121 bash[36902]: audit 2022-04-22T21:32:06.335354+0000 mon.a (mon.0) 12130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:06 smithi115 bash[45495]: cluster 2022-04-22T21:32:05.627583+0000 mgr.y (mgr.34107) 8400 : cluster [DBG] pgmap v9321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:06 smithi115 bash[45495]: audit 2022-04-22T21:32:06.335354+0000 mon.a (mon.0) 12130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:06 smithi115 bash[46934]: cluster 2022-04-22T21:32:05.627583+0000 mgr.y (mgr.34107) 8400 : cluster [DBG] pgmap v9321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:06 smithi115 bash[46934]: audit 2022-04-22T21:32:06.335354+0000 mon.a (mon.0) 12130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:08 smithi121 bash[36902]: cluster 2022-04-22T21:32:07.628772+0000 mgr.y (mgr.34107) 8401 : cluster [DBG] pgmap v9322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:08 smithi115 bash[45495]: cluster 2022-04-22T21:32:07.628772+0000 mgr.y (mgr.34107) 8401 : cluster [DBG] pgmap v9322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:08 smithi115 bash[46934]: cluster 2022-04-22T21:32:07.628772+0000 mgr.y (mgr.34107) 8401 : cluster [DBG] pgmap v9322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:10.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:09 smithi121 bash[36902]: audit 2022-04-22T21:32:09.349561+0000 mon.a (mon.0) 12131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:09 smithi115 bash[45495]: audit 2022-04-22T21:32:09.349561+0000 mon.a (mon.0) 12131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:09 smithi115 bash[46934]: audit 2022-04-22T21:32:09.349561+0000 mon.a (mon.0) 12131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:10 smithi121 bash[36902]: cluster 2022-04-22T21:32:09.629973+0000 mgr.y (mgr.34107) 8402 : cluster [DBG] pgmap v9323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:11.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:10 smithi121 bash[36902]: audit 2022-04-22T21:32:10.256063+0000 mon.a (mon.0) 12132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:10 smithi115 bash[45495]: cluster 2022-04-22T21:32:09.629973+0000 mgr.y (mgr.34107) 8402 : cluster [DBG] pgmap v9323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:10 smithi115 bash[45495]: audit 2022-04-22T21:32:10.256063+0000 mon.a (mon.0) 12132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:10 smithi115 bash[46934]: cluster 2022-04-22T21:32:09.629973+0000 mgr.y (mgr.34107) 8402 : cluster [DBG] pgmap v9323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:10 smithi115 bash[46934]: audit 2022-04-22T21:32:10.256063+0000 mon.a (mon.0) 12132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:11 smithi121 bash[36902]: audit 2022-04-22T21:32:11.338532+0000 mon.a (mon.0) 12133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:11 smithi115 bash[45495]: audit 2022-04-22T21:32:11.338532+0000 mon.a (mon.0) 12133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:11 smithi115 bash[46934]: audit 2022-04-22T21:32:11.338532+0000 mon.a (mon.0) 12133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:12.980 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:12.575Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:12 smithi121 bash[36902]: cluster 2022-04-22T21:32:11.631164+0000 mgr.y (mgr.34107) 8403 : cluster [DBG] pgmap v9324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:12 smithi115 bash[45495]: cluster 2022-04-22T21:32:11.631164+0000 mgr.y (mgr.34107) 8403 : cluster [DBG] pgmap v9324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:12 smithi115 bash[46934]: cluster 2022-04-22T21:32:11.631164+0000 mgr.y (mgr.34107) 8403 : cluster [DBG] pgmap v9324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:14.445Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:14 smithi121 bash[36902]: cluster 2022-04-22T21:32:13.632463+0000 mgr.y (mgr.34107) 8404 : cluster [DBG] pgmap v9325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:14 smithi121 bash[36902]: audit 2022-04-22T21:32:14.352519+0000 mon.a (mon.0) 12134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:14 smithi115 bash[45495]: cluster 2022-04-22T21:32:13.632463+0000 mgr.y (mgr.34107) 8404 : cluster [DBG] pgmap v9325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:14 smithi115 bash[45495]: audit 2022-04-22T21:32:14.352519+0000 mon.a (mon.0) 12134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:14 smithi115 bash[46934]: cluster 2022-04-22T21:32:13.632463+0000 mgr.y (mgr.34107) 8404 : cluster [DBG] pgmap v9325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:14 smithi115 bash[46934]: audit 2022-04-22T21:32:14.352519+0000 mon.a (mon.0) 12134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:15 smithi121 bash[36902]: audit 2022-04-22T21:32:15.259871+0000 mon.a (mon.0) 12135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:15 smithi121 bash[36902]: cluster 2022-04-22T21:32:15.633552+0000 mgr.y (mgr.34107) 8405 : cluster [DBG] pgmap v9326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:15 smithi115 bash[45495]: audit 2022-04-22T21:32:15.259871+0000 mon.a (mon.0) 12135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:15 smithi115 bash[45495]: cluster 2022-04-22T21:32:15.633552+0000 mgr.y (mgr.34107) 8405 : cluster [DBG] pgmap v9326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:15 smithi115 bash[46934]: audit 2022-04-22T21:32:15.259871+0000 mon.a (mon.0) 12135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:15 smithi115 bash[46934]: cluster 2022-04-22T21:32:15.633552+0000 mgr.y (mgr.34107) 8405 : cluster [DBG] pgmap v9326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:17.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:16 smithi121 bash[36902]: audit 2022-04-22T21:32:16.094465+0000 mon.a (mon.0) 12136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:32:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:16 smithi121 bash[36902]: audit 2022-04-22T21:32:16.099605+0000 mon.a (mon.0) 12137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:32:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:16 smithi121 bash[36902]: audit 2022-04-22T21:32:16.341868+0000 mon.a (mon.0) 12138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[45495]: audit 2022-04-22T21:32:16.094465+0000 mon.a (mon.0) 12136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:32:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[45495]: audit 2022-04-22T21:32:16.099605+0000 mon.a (mon.0) 12137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:32:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[45495]: audit 2022-04-22T21:32:16.341868+0000 mon.a (mon.0) 12138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[46934]: audit 2022-04-22T21:32:16.094465+0000 mon.a (mon.0) 12136 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[46934]: audit 2022-04-22T21:32:16.099605+0000 mon.a (mon.0) 12137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:32:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:16 smithi115 bash[46934]: audit 2022-04-22T21:32:16.341868+0000 mon.a (mon.0) 12138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:18 smithi121 bash[36902]: cluster 2022-04-22T21:32:17.634636+0000 mgr.y (mgr.34107) 8406 : cluster [DBG] pgmap v9327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:17 smithi115 bash[46934]: cluster 2022-04-22T21:32:17.634636+0000 mgr.y (mgr.34107) 8406 : cluster [DBG] pgmap v9327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:17 smithi115 bash[45495]: cluster 2022-04-22T21:32:17.634636+0000 mgr.y (mgr.34107) 8406 : cluster [DBG] pgmap v9327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:19.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:19 smithi121 bash[36902]: audit 2022-04-22T21:32:19.355697+0000 mon.a (mon.0) 12139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:19 smithi115 bash[45495]: audit 2022-04-22T21:32:19.355697+0000 mon.a (mon.0) 12139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:19 smithi115 bash[46934]: audit 2022-04-22T21:32:19.355697+0000 mon.a (mon.0) 12139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:20.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:20 smithi121 bash[36902]: cluster 2022-04-22T21:32:19.635761+0000 mgr.y (mgr.34107) 8407 : cluster [DBG] pgmap v9328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:20 smithi121 bash[36902]: audit 2022-04-22T21:32:20.264049+0000 mon.a (mon.0) 12140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:20 smithi115 bash[45495]: cluster 2022-04-22T21:32:19.635761+0000 mgr.y (mgr.34107) 8407 : cluster [DBG] pgmap v9328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:20 smithi115 bash[45495]: audit 2022-04-22T21:32:20.264049+0000 mon.a (mon.0) 12140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:20 smithi115 bash[46934]: cluster 2022-04-22T21:32:19.635761+0000 mgr.y (mgr.34107) 8407 : cluster [DBG] pgmap v9328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:20 smithi115 bash[46934]: audit 2022-04-22T21:32:20.264049+0000 mon.a (mon.0) 12140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:21.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:21 smithi121 bash[36902]: audit 2022-04-22T21:32:21.344918+0000 mon.a (mon.0) 12141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:21 smithi115 bash[45495]: audit 2022-04-22T21:32:21.344918+0000 mon.a (mon.0) 12141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:21 smithi115 bash[46934]: audit 2022-04-22T21:32:21.344918+0000 mon.a (mon.0) 12141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:22.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:22 smithi121 bash[36902]: cluster 2022-04-22T21:32:21.636906+0000 mgr.y (mgr.34107) 8408 : cluster [DBG] pgmap v9329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:22 smithi115 bash[45495]: cluster 2022-04-22T21:32:21.636906+0000 mgr.y (mgr.34107) 8408 : cluster [DBG] pgmap v9329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:22.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:22.576Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:22 smithi115 bash[46934]: cluster 2022-04-22T21:32:21.636906+0000 mgr.y (mgr.34107) 8408 : cluster [DBG] pgmap v9329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:24.447Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:24 smithi115 bash[45495]: cluster 2022-04-22T21:32:23.638061+0000 mgr.y (mgr.34107) 8409 : cluster [DBG] pgmap v9330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:24 smithi115 bash[45495]: audit 2022-04-22T21:32:24.358876+0000 mon.a (mon.0) 12142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:24 smithi115 bash[46934]: cluster 2022-04-22T21:32:23.638061+0000 mgr.y (mgr.34107) 8409 : cluster [DBG] pgmap v9330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:24 smithi115 bash[46934]: audit 2022-04-22T21:32:24.358876+0000 mon.a (mon.0) 12142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:25.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:24 smithi121 bash[36902]: cluster 2022-04-22T21:32:23.638061+0000 mgr.y (mgr.34107) 8409 : cluster [DBG] pgmap v9330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:24 smithi121 bash[36902]: audit 2022-04-22T21:32:24.358876+0000 mon.a (mon.0) 12142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:26.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:25 smithi121 bash[36902]: audit 2022-04-22T21:32:25.268017+0000 mon.a (mon.0) 12143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:25 smithi115 bash[45495]: audit 2022-04-22T21:32:25.268017+0000 mon.a (mon.0) 12143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:25 smithi115 bash[46934]: audit 2022-04-22T21:32:25.268017+0000 mon.a (mon.0) 12143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:26 smithi121 bash[36902]: cluster 2022-04-22T21:32:25.639149+0000 mgr.y (mgr.34107) 8410 : cluster [DBG] pgmap v9331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:26 smithi121 bash[36902]: audit 2022-04-22T21:32:26.347894+0000 mon.a (mon.0) 12144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:27.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:26 smithi115 bash[46934]: cluster 2022-04-22T21:32:25.639149+0000 mgr.y (mgr.34107) 8410 : cluster [DBG] pgmap v9331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:26 smithi115 bash[46934]: audit 2022-04-22T21:32:26.347894+0000 mon.a (mon.0) 12144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:26 smithi115 bash[45495]: cluster 2022-04-22T21:32:25.639149+0000 mgr.y (mgr.34107) 8410 : cluster [DBG] pgmap v9331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:26 smithi115 bash[45495]: audit 2022-04-22T21:32:26.347894+0000 mon.a (mon.0) 12144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:28 smithi121 bash[36902]: cluster 2022-04-22T21:32:27.640281+0000 mgr.y (mgr.34107) 8411 : cluster [DBG] pgmap v9332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:28 smithi121 bash[36902]: audit 2022-04-22T21:32:28.283456+0000 mon.a (mon.0) 12145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:28 smithi121 bash[36902]: audit 2022-04-22T21:32:28.286587+0000 mon.a (mon.0) 12146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:28 smithi121 bash[36902]: audit 2022-04-22T21:32:28.288418+0000 mon.a (mon.0) 12147 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:28 smithi121 bash[36902]: audit 2022-04-22T21:32:28.654616+0000 mon.a (mon.0) 12148 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[45495]: cluster 2022-04-22T21:32:27.640281+0000 mgr.y (mgr.34107) 8411 : cluster [DBG] pgmap v9332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[45495]: audit 2022-04-22T21:32:28.283456+0000 mon.a (mon.0) 12145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[45495]: audit 2022-04-22T21:32:28.286587+0000 mon.a (mon.0) 12146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[45495]: audit 2022-04-22T21:32:28.288418+0000 mon.a (mon.0) 12147 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[45495]: audit 2022-04-22T21:32:28.654616+0000 mon.a (mon.0) 12148 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:32:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[46934]: cluster 2022-04-22T21:32:27.640281+0000 mgr.y (mgr.34107) 8411 : cluster [DBG] pgmap v9332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[46934]: audit 2022-04-22T21:32:28.283456+0000 mon.a (mon.0) 12145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:32:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[46934]: audit 2022-04-22T21:32:28.286587+0000 mon.a (mon.0) 12146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:32:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[46934]: audit 2022-04-22T21:32:28.288418+0000 mon.a (mon.0) 12147 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:32:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:28 smithi115 bash[46934]: audit 2022-04-22T21:32:28.654616+0000 mon.a (mon.0) 12148 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:32:30.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:29 smithi121 bash[36902]: audit 2022-04-22T21:32:29.361927+0000 mon.a (mon.0) 12149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:29 smithi115 bash[45495]: audit 2022-04-22T21:32:29.361927+0000 mon.a (mon.0) 12149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:29 smithi115 bash[46934]: audit 2022-04-22T21:32:29.361927+0000 mon.a (mon.0) 12149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:31.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:30 smithi121 bash[36902]: cluster 2022-04-22T21:32:29.641562+0000 mgr.y (mgr.34107) 8412 : cluster [DBG] pgmap v9333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:30 smithi121 bash[36902]: audit 2022-04-22T21:32:30.271831+0000 mon.a (mon.0) 12150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:30 smithi115 bash[45495]: cluster 2022-04-22T21:32:29.641562+0000 mgr.y (mgr.34107) 8412 : cluster [DBG] pgmap v9333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:30 smithi115 bash[45495]: audit 2022-04-22T21:32:30.271831+0000 mon.a (mon.0) 12150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:30 smithi115 bash[46934]: cluster 2022-04-22T21:32:29.641562+0000 mgr.y (mgr.34107) 8412 : cluster [DBG] pgmap v9333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:30 smithi115 bash[46934]: audit 2022-04-22T21:32:30.271831+0000 mon.a (mon.0) 12150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:31 smithi121 bash[36902]: audit 2022-04-22T21:32:31.350827+0000 mon.a (mon.0) 12151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:31 smithi115 bash[45495]: audit 2022-04-22T21:32:31.350827+0000 mon.a (mon.0) 12151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:31 smithi115 bash[46934]: audit 2022-04-22T21:32:31.350827+0000 mon.a (mon.0) 12151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:32.737 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:32:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:32 smithi121 bash[36902]: cluster 2022-04-22T21:32:31.642751+0000 mgr.y (mgr.34107) 8413 : cluster [DBG] pgmap v9334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:33.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:32 smithi115 bash[46934]: cluster 2022-04-22T21:32:31.642751+0000 mgr.y (mgr.34107) 8413 : cluster [DBG] pgmap v9334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:32.575Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:32 smithi115 bash[45495]: cluster 2022-04-22T21:32:31.642751+0000 mgr.y (mgr.34107) 8413 : cluster [DBG] pgmap v9334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:33.076 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:32:33.077 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:32:33.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:32:33.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:32:33.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:32:33.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:32:33.079 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:32:33.479 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:32:33.479 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:32:33.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:32:33.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:33.479 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:32:33.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:32:33.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:32:33.481 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:32:33.481 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:32:33.481 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:32:33.482 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:32:34.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:33 smithi121 bash[36902]: audit 2022-04-22T21:32:32.714073+0000 mgr.y (mgr.34107) 8414 : audit [DBG] from='client.54031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:33 smithi121 bash[36902]: audit 2022-04-22T21:32:33.475071+0000 mon.a (mon.0) 12152 : audit [DBG] from='client.? 172.21.15.115:0/3670389444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:34.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:33 smithi115 bash[46934]: audit 2022-04-22T21:32:32.714073+0000 mgr.y (mgr.34107) 8414 : audit [DBG] from='client.54031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:33 smithi115 bash[46934]: audit 2022-04-22T21:32:33.475071+0000 mon.a (mon.0) 12152 : audit [DBG] from='client.? 172.21.15.115:0/3670389444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:33 smithi115 bash[45495]: audit 2022-04-22T21:32:32.714073+0000 mgr.y (mgr.34107) 8414 : audit [DBG] from='client.54031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:33 smithi115 bash[45495]: audit 2022-04-22T21:32:33.475071+0000 mon.a (mon.0) 12152 : audit [DBG] from='client.? 172.21.15.115:0/3670389444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:32:34.738 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:34.447Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:34 smithi121 bash[36902]: audit 2022-04-22T21:32:33.059011+0000 mgr.y (mgr.34107) 8415 : audit [DBG] from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:34 smithi121 bash[36902]: cluster 2022-04-22T21:32:33.643891+0000 mgr.y (mgr.34107) 8416 : cluster [DBG] pgmap v9335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:34 smithi121 bash[36902]: audit 2022-04-22T21:32:34.365243+0000 mon.a (mon.0) 12153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[45495]: audit 2022-04-22T21:32:33.059011+0000 mgr.y (mgr.34107) 8415 : audit [DBG] from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[45495]: cluster 2022-04-22T21:32:33.643891+0000 mgr.y (mgr.34107) 8416 : cluster [DBG] pgmap v9335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[45495]: audit 2022-04-22T21:32:34.365243+0000 mon.a (mon.0) 12153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[46934]: audit 2022-04-22T21:32:33.059011+0000 mgr.y (mgr.34107) 8415 : audit [DBG] from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:32:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[46934]: cluster 2022-04-22T21:32:33.643891+0000 mgr.y (mgr.34107) 8416 : cluster [DBG] pgmap v9335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:34 smithi115 bash[46934]: audit 2022-04-22T21:32:34.365243+0000 mon.a (mon.0) 12153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:36.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:35 smithi121 bash[36902]: audit 2022-04-22T21:32:35.275665+0000 mon.a (mon.0) 12154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:35 smithi115 bash[45495]: audit 2022-04-22T21:32:35.275665+0000 mon.a (mon.0) 12154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:35 smithi115 bash[46934]: audit 2022-04-22T21:32:35.275665+0000 mon.a (mon.0) 12154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:36 smithi121 bash[36902]: cluster 2022-04-22T21:32:35.644933+0000 mgr.y (mgr.34107) 8417 : cluster [DBG] pgmap v9336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:36 smithi121 bash[36902]: audit 2022-04-22T21:32:36.354138+0000 mon.a (mon.0) 12155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:36 smithi115 bash[45495]: cluster 2022-04-22T21:32:35.644933+0000 mgr.y (mgr.34107) 8417 : cluster [DBG] pgmap v9336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:36 smithi115 bash[45495]: audit 2022-04-22T21:32:36.354138+0000 mon.a (mon.0) 12155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:36 smithi115 bash[46934]: cluster 2022-04-22T21:32:35.644933+0000 mgr.y (mgr.34107) 8417 : cluster [DBG] pgmap v9336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:36 smithi115 bash[46934]: audit 2022-04-22T21:32:36.354138+0000 mon.a (mon.0) 12155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:38 smithi121 bash[36902]: cluster 2022-04-22T21:32:37.646211+0000 mgr.y (mgr.34107) 8418 : cluster [DBG] pgmap v9337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:38 smithi115 bash[45495]: cluster 2022-04-22T21:32:37.646211+0000 mgr.y (mgr.34107) 8418 : cluster [DBG] pgmap v9337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:38 smithi115 bash[46934]: cluster 2022-04-22T21:32:37.646211+0000 mgr.y (mgr.34107) 8418 : cluster [DBG] pgmap v9337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:40.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:39 smithi121 bash[36902]: audit 2022-04-22T21:32:39.368299+0000 mon.a (mon.0) 12156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:39 smithi115 bash[45495]: audit 2022-04-22T21:32:39.368299+0000 mon.a (mon.0) 12156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:39 smithi115 bash[46934]: audit 2022-04-22T21:32:39.368299+0000 mon.a (mon.0) 12156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:40 smithi121 bash[36902]: cluster 2022-04-22T21:32:39.647276+0000 mgr.y (mgr.34107) 8419 : cluster [DBG] pgmap v9338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:40 smithi121 bash[36902]: audit 2022-04-22T21:32:40.279161+0000 mon.a (mon.0) 12157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:40 smithi115 bash[45495]: cluster 2022-04-22T21:32:39.647276+0000 mgr.y (mgr.34107) 8419 : cluster [DBG] pgmap v9338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:40 smithi115 bash[45495]: audit 2022-04-22T21:32:40.279161+0000 mon.a (mon.0) 12157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:40 smithi115 bash[46934]: cluster 2022-04-22T21:32:39.647276+0000 mgr.y (mgr.34107) 8419 : cluster [DBG] pgmap v9338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:40 smithi115 bash[46934]: audit 2022-04-22T21:32:40.279161+0000 mon.a (mon.0) 12157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:41 smithi115 bash[45495]: audit 2022-04-22T21:32:41.356954+0000 mon.a (mon.0) 12158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:41 smithi115 bash[46934]: audit 2022-04-22T21:32:41.356954+0000 mon.a (mon.0) 12158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:42.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:41 smithi121 bash[36902]: audit 2022-04-22T21:32:41.356954+0000 mon.a (mon.0) 12158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:42 smithi115 bash[45495]: cluster 2022-04-22T21:32:41.648348+0000 mgr.y (mgr.34107) 8420 : cluster [DBG] pgmap v9339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:42.577Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:42 smithi115 bash[46934]: cluster 2022-04-22T21:32:41.648348+0000 mgr.y (mgr.34107) 8420 : cluster [DBG] pgmap v9339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:42 smithi121 bash[36902]: cluster 2022-04-22T21:32:41.648348+0000 mgr.y (mgr.34107) 8420 : cluster [DBG] pgmap v9339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:44.774 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:44.449Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:44 smithi115 bash[45495]: cluster 2022-04-22T21:32:43.649398+0000 mgr.y (mgr.34107) 8421 : cluster [DBG] pgmap v9340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:44 smithi115 bash[45495]: audit 2022-04-22T21:32:44.371245+0000 mon.a (mon.0) 12159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:44 smithi115 bash[46934]: cluster 2022-04-22T21:32:43.649398+0000 mgr.y (mgr.34107) 8421 : cluster [DBG] pgmap v9340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:44 smithi115 bash[46934]: audit 2022-04-22T21:32:44.371245+0000 mon.a (mon.0) 12159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:44 smithi121 bash[36902]: cluster 2022-04-22T21:32:43.649398+0000 mgr.y (mgr.34107) 8421 : cluster [DBG] pgmap v9340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:44 smithi121 bash[36902]: audit 2022-04-22T21:32:44.371245+0000 mon.a (mon.0) 12159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:45 smithi115 bash[45495]: audit 2022-04-22T21:32:45.282676+0000 mon.a (mon.0) 12160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:45 smithi115 bash[46934]: audit 2022-04-22T21:32:45.282676+0000 mon.a (mon.0) 12160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:45 smithi121 bash[36902]: audit 2022-04-22T21:32:45.282676+0000 mon.a (mon.0) 12160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:46 smithi115 bash[45495]: cluster 2022-04-22T21:32:45.650230+0000 mgr.y (mgr.34107) 8422 : cluster [DBG] pgmap v9341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:46 smithi115 bash[45495]: audit 2022-04-22T21:32:46.360178+0000 mon.a (mon.0) 12161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:46 smithi115 bash[46934]: cluster 2022-04-22T21:32:45.650230+0000 mgr.y (mgr.34107) 8422 : cluster [DBG] pgmap v9341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:46 smithi115 bash[46934]: audit 2022-04-22T21:32:46.360178+0000 mon.a (mon.0) 12161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:46 smithi121 bash[36902]: cluster 2022-04-22T21:32:45.650230+0000 mgr.y (mgr.34107) 8422 : cluster [DBG] pgmap v9341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:46 smithi121 bash[36902]: audit 2022-04-22T21:32:46.360178+0000 mon.a (mon.0) 12161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:48 smithi115 bash[45495]: cluster 2022-04-22T21:32:47.651388+0000 mgr.y (mgr.34107) 8423 : cluster [DBG] pgmap v9342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:49.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:48 smithi115 bash[46934]: cluster 2022-04-22T21:32:47.651388+0000 mgr.y (mgr.34107) 8423 : cluster [DBG] pgmap v9342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:49.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:48 smithi121 bash[36902]: cluster 2022-04-22T21:32:47.651388+0000 mgr.y (mgr.34107) 8423 : cluster [DBG] pgmap v9342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:49 smithi121 bash[36902]: audit 2022-04-22T21:32:49.374374+0000 mon.a (mon.0) 12162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:49 smithi115 bash[45495]: audit 2022-04-22T21:32:49.374374+0000 mon.a (mon.0) 12162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:49 smithi115 bash[46934]: audit 2022-04-22T21:32:49.374374+0000 mon.a (mon.0) 12162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:50 smithi121 bash[36902]: cluster 2022-04-22T21:32:49.652431+0000 mgr.y (mgr.34107) 8424 : cluster [DBG] pgmap v9343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:50 smithi121 bash[36902]: audit 2022-04-22T21:32:50.286434+0000 mon.a (mon.0) 12163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:50 smithi115 bash[45495]: cluster 2022-04-22T21:32:49.652431+0000 mgr.y (mgr.34107) 8424 : cluster [DBG] pgmap v9343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:50 smithi115 bash[45495]: audit 2022-04-22T21:32:50.286434+0000 mon.a (mon.0) 12163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:50 smithi115 bash[46934]: cluster 2022-04-22T21:32:49.652431+0000 mgr.y (mgr.34107) 8424 : cluster [DBG] pgmap v9343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:50 smithi115 bash[46934]: audit 2022-04-22T21:32:50.286434+0000 mon.a (mon.0) 12163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:51 smithi121 bash[36902]: audit 2022-04-22T21:32:51.363321+0000 mon.a (mon.0) 12164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:51 smithi115 bash[45495]: audit 2022-04-22T21:32:51.363321+0000 mon.a (mon.0) 12164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:52.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:51 smithi115 bash[46934]: audit 2022-04-22T21:32:51.363321+0000 mon.a (mon.0) 12164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:52 smithi115 bash[45495]: cluster 2022-04-22T21:32:51.653542+0000 mgr.y (mgr.34107) 8425 : cluster [DBG] pgmap v9344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:52.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:52 smithi115 bash[46934]: cluster 2022-04-22T21:32:51.653542+0000 mgr.y (mgr.34107) 8425 : cluster [DBG] pgmap v9344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:52 smithi121 bash[36902]: cluster 2022-04-22T21:32:51.653542+0000 mgr.y (mgr.34107) 8425 : cluster [DBG] pgmap v9344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:32:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:32:54.462Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:32:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:54 smithi121 bash[36902]: cluster 2022-04-22T21:32:53.654340+0000 mgr.y (mgr.34107) 8426 : cluster [DBG] pgmap v9345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:54 smithi121 bash[36902]: audit 2022-04-22T21:32:54.377522+0000 mon.a (mon.0) 12165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:54 smithi115 bash[45495]: cluster 2022-04-22T21:32:53.654340+0000 mgr.y (mgr.34107) 8426 : cluster [DBG] pgmap v9345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:54 smithi115 bash[45495]: audit 2022-04-22T21:32:54.377522+0000 mon.a (mon.0) 12165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:54 smithi115 bash[46934]: cluster 2022-04-22T21:32:53.654340+0000 mgr.y (mgr.34107) 8426 : cluster [DBG] pgmap v9345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:54 smithi115 bash[46934]: audit 2022-04-22T21:32:54.377522+0000 mon.a (mon.0) 12165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:32:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:55 smithi121 bash[36902]: audit 2022-04-22T21:32:55.290305+0000 mon.a (mon.0) 12166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:55 smithi115 bash[45495]: audit 2022-04-22T21:32:55.290305+0000 mon.a (mon.0) 12166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:55 smithi115 bash[46934]: audit 2022-04-22T21:32:55.290305+0000 mon.a (mon.0) 12166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:32:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:56 smithi121 bash[36902]: cluster 2022-04-22T21:32:55.655393+0000 mgr.y (mgr.34107) 8427 : cluster [DBG] pgmap v9346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:57.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:56 smithi121 bash[36902]: audit 2022-04-22T21:32:56.366493+0000 mon.a (mon.0) 12167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:57.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:56 smithi115 bash[45495]: cluster 2022-04-22T21:32:55.655393+0000 mgr.y (mgr.34107) 8427 : cluster [DBG] pgmap v9346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:57.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:56 smithi115 bash[45495]: audit 2022-04-22T21:32:56.366493+0000 mon.a (mon.0) 12167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:56 smithi115 bash[46934]: cluster 2022-04-22T21:32:55.655393+0000 mgr.y (mgr.34107) 8427 : cluster [DBG] pgmap v9346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:56 smithi115 bash[46934]: audit 2022-04-22T21:32:56.366493+0000 mon.a (mon.0) 12167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:32:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:58 smithi121 bash[36902]: cluster 2022-04-22T21:32:57.656373+0000 mgr.y (mgr.34107) 8428 : cluster [DBG] pgmap v9347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:58 smithi115 bash[45495]: cluster 2022-04-22T21:32:57.656373+0000 mgr.y (mgr.34107) 8428 : cluster [DBG] pgmap v9347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:32:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:58 smithi115 bash[46934]: cluster 2022-04-22T21:32:57.656373+0000 mgr.y (mgr.34107) 8428 : cluster [DBG] pgmap v9347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:32:59 smithi121 bash[36902]: audit 2022-04-22T21:32:59.380686+0000 mon.a (mon.0) 12168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:00.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:32:59 smithi115 bash[45495]: audit 2022-04-22T21:32:59.380686+0000 mon.a (mon.0) 12168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:00.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:32:59 smithi115 bash[46934]: audit 2022-04-22T21:32:59.380686+0000 mon.a (mon.0) 12168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:00 smithi121 bash[36902]: cluster 2022-04-22T21:32:59.657400+0000 mgr.y (mgr.34107) 8429 : cluster [DBG] pgmap v9348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:00 smithi121 bash[36902]: audit 2022-04-22T21:33:00.294236+0000 mon.a (mon.0) 12169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:00 smithi115 bash[45495]: cluster 2022-04-22T21:32:59.657400+0000 mgr.y (mgr.34107) 8429 : cluster [DBG] pgmap v9348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:00 smithi115 bash[45495]: audit 2022-04-22T21:33:00.294236+0000 mon.a (mon.0) 12169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:00 smithi115 bash[46934]: cluster 2022-04-22T21:32:59.657400+0000 mgr.y (mgr.34107) 8429 : cluster [DBG] pgmap v9348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:00 smithi115 bash[46934]: audit 2022-04-22T21:33:00.294236+0000 mon.a (mon.0) 12169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:01 smithi121 bash[36902]: audit 2022-04-22T21:33:01.369620+0000 mon.a (mon.0) 12170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:01 smithi115 bash[45495]: audit 2022-04-22T21:33:01.369620+0000 mon.a (mon.0) 12170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:01 smithi115 bash[46934]: audit 2022-04-22T21:33:01.369620+0000 mon.a (mon.0) 12170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:02.847 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:02.579Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:02 smithi121 bash[36902]: cluster 2022-04-22T21:33:01.658281+0000 mgr.y (mgr.34107) 8430 : cluster [DBG] pgmap v9349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:02 smithi115 bash[45495]: cluster 2022-04-22T21:33:01.658281+0000 mgr.y (mgr.34107) 8430 : cluster [DBG] pgmap v9349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:02 smithi115 bash[46934]: cluster 2022-04-22T21:33:01.658281+0000 mgr.y (mgr.34107) 8430 : cluster [DBG] pgmap v9349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:03.858 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 8m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:33:04.203 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 246M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 8m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 9m ago 4h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:33:04.204 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:33:04.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 582M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:33:04.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (3h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:33:04.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 9m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:33:04.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 9m ago 4h 644M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:33:04.205 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 9m ago 4h 64.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:33:04.611 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:33:04.611 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:04.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:33:04.613 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:33:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:04.447Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:04 smithi121 bash[36902]: cluster 2022-04-22T21:33:03.659320+0000 mgr.y (mgr.34107) 8431 : cluster [DBG] pgmap v9350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:04 smithi121 bash[36902]: audit 2022-04-22T21:33:03.835343+0000 mgr.y (mgr.34107) 8432 : audit [DBG] from='client.54049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:04 smithi121 bash[36902]: audit 2022-04-22T21:33:04.383623+0000 mon.a (mon.0) 12171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:04 smithi121 bash[36902]: audit 2022-04-22T21:33:04.607806+0000 mon.a (mon.0) 12172 : audit [DBG] from='client.? 172.21.15.115:0/807556894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[45495]: cluster 2022-04-22T21:33:03.659320+0000 mgr.y (mgr.34107) 8431 : cluster [DBG] pgmap v9350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[45495]: audit 2022-04-22T21:33:03.835343+0000 mgr.y (mgr.34107) 8432 : audit [DBG] from='client.54049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[45495]: audit 2022-04-22T21:33:04.383623+0000 mon.a (mon.0) 12171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[45495]: audit 2022-04-22T21:33:04.607806+0000 mon.a (mon.0) 12172 : audit [DBG] from='client.? 172.21.15.115:0/807556894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[46934]: cluster 2022-04-22T21:33:03.659320+0000 mgr.y (mgr.34107) 8431 : cluster [DBG] pgmap v9350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[46934]: audit 2022-04-22T21:33:03.835343+0000 mgr.y (mgr.34107) 8432 : audit [DBG] from='client.54049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[46934]: audit 2022-04-22T21:33:04.383623+0000 mon.a (mon.0) 12171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:04 smithi115 bash[46934]: audit 2022-04-22T21:33:04.607806+0000 mon.a (mon.0) 12172 : audit [DBG] from='client.? 172.21.15.115:0/807556894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:05 smithi121 bash[36902]: audit 2022-04-22T21:33:04.181361+0000 mgr.y (mgr.34107) 8433 : audit [DBG] from='client.54055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:05 smithi121 bash[36902]: audit 2022-04-22T21:33:05.297659+0000 mon.a (mon.0) 12173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:05 smithi115 bash[45495]: audit 2022-04-22T21:33:04.181361+0000 mgr.y (mgr.34107) 8433 : audit [DBG] from='client.54055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:05 smithi115 bash[45495]: audit 2022-04-22T21:33:05.297659+0000 mon.a (mon.0) 12173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:05 smithi115 bash[46934]: audit 2022-04-22T21:33:04.181361+0000 mgr.y (mgr.34107) 8433 : audit [DBG] from='client.54055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:05 smithi115 bash[46934]: audit 2022-04-22T21:33:05.297659+0000 mon.a (mon.0) 12173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:06 smithi121 bash[36902]: cluster 2022-04-22T21:33:05.660297+0000 mgr.y (mgr.34107) 8434 : cluster [DBG] pgmap v9351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:06 smithi121 bash[36902]: audit 2022-04-22T21:33:06.372489+0000 mon.a (mon.0) 12174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:07.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:06 smithi115 bash[46934]: cluster 2022-04-22T21:33:05.660297+0000 mgr.y (mgr.34107) 8434 : cluster [DBG] pgmap v9351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:06 smithi115 bash[46934]: audit 2022-04-22T21:33:06.372489+0000 mon.a (mon.0) 12174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:06 smithi115 bash[45495]: cluster 2022-04-22T21:33:05.660297+0000 mgr.y (mgr.34107) 8434 : cluster [DBG] pgmap v9351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:06 smithi115 bash[45495]: audit 2022-04-22T21:33:06.372489+0000 mon.a (mon.0) 12174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:08 smithi121 bash[36902]: cluster 2022-04-22T21:33:07.661535+0000 mgr.y (mgr.34107) 8435 : cluster [DBG] pgmap v9352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:08 smithi115 bash[45495]: cluster 2022-04-22T21:33:07.661535+0000 mgr.y (mgr.34107) 8435 : cluster [DBG] pgmap v9352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:08 smithi115 bash[46934]: cluster 2022-04-22T21:33:07.661535+0000 mgr.y (mgr.34107) 8435 : cluster [DBG] pgmap v9352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:10.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:09 smithi121 bash[36902]: audit 2022-04-22T21:33:09.386380+0000 mon.a (mon.0) 12175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:09 smithi115 bash[46934]: audit 2022-04-22T21:33:09.386380+0000 mon.a (mon.0) 12175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:09 smithi115 bash[45495]: audit 2022-04-22T21:33:09.386380+0000 mon.a (mon.0) 12175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:10 smithi121 bash[36902]: cluster 2022-04-22T21:33:09.662498+0000 mgr.y (mgr.34107) 8436 : cluster [DBG] pgmap v9353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:10 smithi121 bash[36902]: audit 2022-04-22T21:33:10.301544+0000 mon.a (mon.0) 12176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:10 smithi115 bash[45495]: cluster 2022-04-22T21:33:09.662498+0000 mgr.y (mgr.34107) 8436 : cluster [DBG] pgmap v9353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:10 smithi115 bash[45495]: audit 2022-04-22T21:33:10.301544+0000 mon.a (mon.0) 12176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:10 smithi115 bash[46934]: cluster 2022-04-22T21:33:09.662498+0000 mgr.y (mgr.34107) 8436 : cluster [DBG] pgmap v9353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:10 smithi115 bash[46934]: audit 2022-04-22T21:33:10.301544+0000 mon.a (mon.0) 12176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:11 smithi121 bash[36902]: audit 2022-04-22T21:33:11.375565+0000 mon.a (mon.0) 12177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:12.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:11 smithi115 bash[45495]: audit 2022-04-22T21:33:11.375565+0000 mon.a (mon.0) 12177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:12.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:11 smithi115 bash[46934]: audit 2022-04-22T21:33:11.375565+0000 mon.a (mon.0) 12177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:12.884 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:12.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:12 smithi121 bash[36902]: cluster 2022-04-22T21:33:11.663518+0000 mgr.y (mgr.34107) 8437 : cluster [DBG] pgmap v9354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:12 smithi115 bash[45495]: cluster 2022-04-22T21:33:11.663518+0000 mgr.y (mgr.34107) 8437 : cluster [DBG] pgmap v9354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:12 smithi115 bash[46934]: cluster 2022-04-22T21:33:11.663518+0000 mgr.y (mgr.34107) 8437 : cluster [DBG] pgmap v9354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:14.450Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:15.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:14 smithi121 bash[36902]: cluster 2022-04-22T21:33:13.664566+0000 mgr.y (mgr.34107) 8438 : cluster [DBG] pgmap v9355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:14 smithi121 bash[36902]: audit 2022-04-22T21:33:14.389297+0000 mon.a (mon.0) 12178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:14 smithi115 bash[45495]: cluster 2022-04-22T21:33:13.664566+0000 mgr.y (mgr.34107) 8438 : cluster [DBG] pgmap v9355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:14 smithi115 bash[45495]: audit 2022-04-22T21:33:14.389297+0000 mon.a (mon.0) 12178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:14 smithi115 bash[46934]: cluster 2022-04-22T21:33:13.664566+0000 mgr.y (mgr.34107) 8438 : cluster [DBG] pgmap v9355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:14 smithi115 bash[46934]: audit 2022-04-22T21:33:14.389297+0000 mon.a (mon.0) 12178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:15 smithi121 bash[36902]: audit 2022-04-22T21:33:15.305168+0000 mon.a (mon.0) 12179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:15 smithi115 bash[45495]: audit 2022-04-22T21:33:15.305168+0000 mon.a (mon.0) 12179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:16.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:15 smithi115 bash[46934]: audit 2022-04-22T21:33:15.305168+0000 mon.a (mon.0) 12179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:16 smithi121 bash[36902]: cluster 2022-04-22T21:33:15.665653+0000 mgr.y (mgr.34107) 8439 : cluster [DBG] pgmap v9356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:16 smithi121 bash[36902]: audit 2022-04-22T21:33:16.105685+0000 mon.a (mon.0) 12180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:33:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:16 smithi121 bash[36902]: audit 2022-04-22T21:33:16.113851+0000 mon.a (mon.0) 12181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:33:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:16 smithi121 bash[36902]: audit 2022-04-22T21:33:16.378835+0000 mon.a (mon.0) 12182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[45495]: cluster 2022-04-22T21:33:15.665653+0000 mgr.y (mgr.34107) 8439 : cluster [DBG] pgmap v9356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[45495]: audit 2022-04-22T21:33:16.105685+0000 mon.a (mon.0) 12180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:33:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[45495]: audit 2022-04-22T21:33:16.113851+0000 mon.a (mon.0) 12181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:33:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[45495]: audit 2022-04-22T21:33:16.378835+0000 mon.a (mon.0) 12182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[46934]: cluster 2022-04-22T21:33:15.665653+0000 mgr.y (mgr.34107) 8439 : cluster [DBG] pgmap v9356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[46934]: audit 2022-04-22T21:33:16.105685+0000 mon.a (mon.0) 12180 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:33:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[46934]: audit 2022-04-22T21:33:16.113851+0000 mon.a (mon.0) 12181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:33:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:16 smithi115 bash[46934]: audit 2022-04-22T21:33:16.378835+0000 mon.a (mon.0) 12182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:19.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:18 smithi121 bash[36902]: cluster 2022-04-22T21:33:17.666838+0000 mgr.y (mgr.34107) 8440 : cluster [DBG] pgmap v9357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:18 smithi115 bash[45495]: cluster 2022-04-22T21:33:17.666838+0000 mgr.y (mgr.34107) 8440 : cluster [DBG] pgmap v9357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:18 smithi115 bash[46934]: cluster 2022-04-22T21:33:17.666838+0000 mgr.y (mgr.34107) 8440 : cluster [DBG] pgmap v9357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:19 smithi121 bash[36902]: audit 2022-04-22T21:33:19.392586+0000 mon.a (mon.0) 12183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:19 smithi115 bash[46934]: audit 2022-04-22T21:33:19.392586+0000 mon.a (mon.0) 12183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:19 smithi115 bash[45495]: audit 2022-04-22T21:33:19.392586+0000 mon.a (mon.0) 12183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:20 smithi121 bash[36902]: cluster 2022-04-22T21:33:19.667978+0000 mgr.y (mgr.34107) 8441 : cluster [DBG] pgmap v9358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:20 smithi121 bash[36902]: audit 2022-04-22T21:33:20.308913+0000 mon.a (mon.0) 12184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:20 smithi115 bash[45495]: cluster 2022-04-22T21:33:19.667978+0000 mgr.y (mgr.34107) 8441 : cluster [DBG] pgmap v9358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:20 smithi115 bash[45495]: audit 2022-04-22T21:33:20.308913+0000 mon.a (mon.0) 12184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:20 smithi115 bash[46934]: cluster 2022-04-22T21:33:19.667978+0000 mgr.y (mgr.34107) 8441 : cluster [DBG] pgmap v9358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:20 smithi115 bash[46934]: audit 2022-04-22T21:33:20.308913+0000 mon.a (mon.0) 12184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:21 smithi121 bash[36902]: audit 2022-04-22T21:33:21.381808+0000 mon.a (mon.0) 12185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:21 smithi115 bash[45495]: audit 2022-04-22T21:33:21.381808+0000 mon.a (mon.0) 12185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:21 smithi115 bash[46934]: audit 2022-04-22T21:33:21.381808+0000 mon.a (mon.0) 12185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:22.916 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:22.581Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:22 smithi121 bash[36902]: cluster 2022-04-22T21:33:21.669048+0000 mgr.y (mgr.34107) 8442 : cluster [DBG] pgmap v9359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:22 smithi115 bash[45495]: cluster 2022-04-22T21:33:21.669048+0000 mgr.y (mgr.34107) 8442 : cluster [DBG] pgmap v9359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:22 smithi115 bash[46934]: cluster 2022-04-22T21:33:21.669048+0000 mgr.y (mgr.34107) 8442 : cluster [DBG] pgmap v9359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:24.450Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:24 smithi121 bash[36902]: cluster 2022-04-22T21:33:23.670256+0000 mgr.y (mgr.34107) 8443 : cluster [DBG] pgmap v9360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:24 smithi121 bash[36902]: audit 2022-04-22T21:33:24.395700+0000 mon.a (mon.0) 12186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:24 smithi115 bash[46934]: cluster 2022-04-22T21:33:23.670256+0000 mgr.y (mgr.34107) 8443 : cluster [DBG] pgmap v9360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:24 smithi115 bash[46934]: audit 2022-04-22T21:33:24.395700+0000 mon.a (mon.0) 12186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:24 smithi115 bash[45495]: cluster 2022-04-22T21:33:23.670256+0000 mgr.y (mgr.34107) 8443 : cluster [DBG] pgmap v9360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:24 smithi115 bash[45495]: audit 2022-04-22T21:33:24.395700+0000 mon.a (mon.0) 12186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:25 smithi121 bash[36902]: audit 2022-04-22T21:33:25.312573+0000 mon.a (mon.0) 12187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:25 smithi115 bash[45495]: audit 2022-04-22T21:33:25.312573+0000 mon.a (mon.0) 12187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:25 smithi115 bash[46934]: audit 2022-04-22T21:33:25.312573+0000 mon.a (mon.0) 12187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:27.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:26 smithi121 bash[36902]: cluster 2022-04-22T21:33:25.671453+0000 mgr.y (mgr.34107) 8444 : cluster [DBG] pgmap v9361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:26 smithi121 bash[36902]: audit 2022-04-22T21:33:26.384881+0000 mon.a (mon.0) 12188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:26 smithi115 bash[45495]: cluster 2022-04-22T21:33:25.671453+0000 mgr.y (mgr.34107) 8444 : cluster [DBG] pgmap v9361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:26 smithi115 bash[45495]: audit 2022-04-22T21:33:26.384881+0000 mon.a (mon.0) 12188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:26 smithi115 bash[46934]: cluster 2022-04-22T21:33:25.671453+0000 mgr.y (mgr.34107) 8444 : cluster [DBG] pgmap v9361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:26 smithi115 bash[46934]: audit 2022-04-22T21:33:26.384881+0000 mon.a (mon.0) 12188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:29.006 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[45495]: cluster 2022-04-22T21:33:27.672580+0000 mgr.y (mgr.34107) 8445 : cluster [DBG] pgmap v9362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:29.006 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[45495]: audit 2022-04-22T21:33:28.662921+0000 mon.a (mon.0) 12189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[45495]: audit 2022-04-22T21:33:28.665956+0000 mon.a (mon.0) 12190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[45495]: audit 2022-04-22T21:33:28.667735+0000 mon.a (mon.0) 12191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[46934]: cluster 2022-04-22T21:33:27.672580+0000 mgr.y (mgr.34107) 8445 : cluster [DBG] pgmap v9362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[46934]: audit 2022-04-22T21:33:28.662921+0000 mon.a (mon.0) 12189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[46934]: audit 2022-04-22T21:33:28.665956+0000 mon.a (mon.0) 12190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:33:29.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:28 smithi115 bash[46934]: audit 2022-04-22T21:33:28.667735+0000 mon.a (mon.0) 12191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:33:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:28 smithi121 bash[36902]: cluster 2022-04-22T21:33:27.672580+0000 mgr.y (mgr.34107) 8445 : cluster [DBG] pgmap v9362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:29.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:28 smithi121 bash[36902]: audit 2022-04-22T21:33:28.662921+0000 mon.a (mon.0) 12189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:33:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:28 smithi121 bash[36902]: audit 2022-04-22T21:33:28.665956+0000 mon.a (mon.0) 12190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:33:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:28 smithi121 bash[36902]: audit 2022-04-22T21:33:28.667735+0000 mon.a (mon.0) 12191 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:33:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:29 smithi121 bash[36902]: audit 2022-04-22T21:33:29.398636+0000 mon.a (mon.0) 12192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:29 smithi115 bash[45495]: audit 2022-04-22T21:33:29.398636+0000 mon.a (mon.0) 12192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:29 smithi115 bash[46934]: audit 2022-04-22T21:33:29.398636+0000 mon.a (mon.0) 12192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:30 smithi121 bash[36902]: cluster 2022-04-22T21:33:29.673624+0000 mgr.y (mgr.34107) 8446 : cluster [DBG] pgmap v9363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:30 smithi121 bash[36902]: audit 2022-04-22T21:33:30.316270+0000 mon.a (mon.0) 12193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:30 smithi115 bash[45495]: cluster 2022-04-22T21:33:29.673624+0000 mgr.y (mgr.34107) 8446 : cluster [DBG] pgmap v9363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:30 smithi115 bash[45495]: audit 2022-04-22T21:33:30.316270+0000 mon.a (mon.0) 12193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:30 smithi115 bash[46934]: cluster 2022-04-22T21:33:29.673624+0000 mgr.y (mgr.34107) 8446 : cluster [DBG] pgmap v9363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:30 smithi115 bash[46934]: audit 2022-04-22T21:33:30.316270+0000 mon.a (mon.0) 12193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:32.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:31 smithi121 bash[36902]: audit 2022-04-22T21:33:31.387728+0000 mon.a (mon.0) 12194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:31 smithi115 bash[45495]: audit 2022-04-22T21:33:31.387728+0000 mon.a (mon.0) 12194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:31 smithi115 bash[46934]: audit 2022-04-22T21:33:31.387728+0000 mon.a (mon.0) 12194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:32.955 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:32.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:33.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:32 smithi121 bash[36902]: cluster 2022-04-22T21:33:31.674692+0000 mgr.y (mgr.34107) 8447 : cluster [DBG] pgmap v9364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:32 smithi115 bash[45495]: cluster 2022-04-22T21:33:31.674692+0000 mgr.y (mgr.34107) 8447 : cluster [DBG] pgmap v9364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:32 smithi115 bash[46934]: cluster 2022-04-22T21:33:31.674692+0000 mgr.y (mgr.34107) 8447 : cluster [DBG] pgmap v9364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:34.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[45495]: audit 2022-04-22T21:33:33.135490+0000 mon.a (mon.0) 12195 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[45495]: audit 2022-04-22T21:33:33.466921+0000 mon.a (mon.0) 12196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[45495]: cluster 2022-04-22T21:33:33.675881+0000 mgr.y (mgr.34107) 8448 : cluster [DBG] pgmap v9365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:34.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[46934]: audit 2022-04-22T21:33:33.135490+0000 mon.a (mon.0) 12195 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[46934]: audit 2022-04-22T21:33:33.466921+0000 mon.a (mon.0) 12196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[46934]: cluster 2022-04-22T21:33:33.675881+0000 mgr.y (mgr.34107) 8448 : cluster [DBG] pgmap v9365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:34 smithi121 bash[36902]: audit 2022-04-22T21:33:33.135490+0000 mon.a (mon.0) 12195 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:34 smithi121 bash[36902]: audit 2022-04-22T21:33:33.466921+0000 mon.a (mon.0) 12196 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:33:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:34 smithi121 bash[36902]: cluster 2022-04-22T21:33:33.675881+0000 mgr.y (mgr.34107) 8448 : cluster [DBG] pgmap v9365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:34.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:34.976 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:33:35.312 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:33:35.312 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2s ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 2s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2s ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:33:35.313 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2s ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (3h) 2s ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 2s ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:33:35.314 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2s ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:33:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:35 smithi121 bash[36902]: audit 2022-04-22T21:33:34.401795+0000 mon.a (mon.0) 12197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:35 smithi121 bash[36902]: audit 2022-04-22T21:33:34.958843+0000 mgr.y (mgr.34107) 8449 : audit [DBG] from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:35.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:35 smithi115 bash[46934]: audit 2022-04-22T21:33:34.401795+0000 mon.a (mon.0) 12197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:35 smithi115 bash[46934]: audit 2022-04-22T21:33:34.958843+0000 mgr.y (mgr.34107) 8449 : audit [DBG] from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:35 smithi115 bash[45495]: audit 2022-04-22T21:33:34.401795+0000 mon.a (mon.0) 12197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:35 smithi115 bash[45495]: audit 2022-04-22T21:33:34.958843+0000 mgr.y (mgr.34107) 8449 : audit [DBG] from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:35.724 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:33:35.724 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:33:35.724 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:33:35.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:35.724 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:33:35.725 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:33:35.726 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:33:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:36 smithi121 bash[36902]: audit 2022-04-22T21:33:35.292053+0000 mgr.y (mgr.34107) 8450 : audit [DBG] from='client.54073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:36.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:36 smithi121 bash[36902]: audit 2022-04-22T21:33:35.320195+0000 mon.a (mon.0) 12198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:36.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:36 smithi121 bash[36902]: cluster 2022-04-22T21:33:35.676876+0000 mgr.y (mgr.34107) 8451 : cluster [DBG] pgmap v9366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:36.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:36 smithi121 bash[36902]: audit 2022-04-22T21:33:35.720395+0000 mon.c (mon.1) 301 : audit [DBG] from='client.? 172.21.15.115:0/930425512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[45495]: audit 2022-04-22T21:33:35.292053+0000 mgr.y (mgr.34107) 8450 : audit [DBG] from='client.54073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[45495]: audit 2022-04-22T21:33:35.320195+0000 mon.a (mon.0) 12198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[45495]: cluster 2022-04-22T21:33:35.676876+0000 mgr.y (mgr.34107) 8451 : cluster [DBG] pgmap v9366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[45495]: audit 2022-04-22T21:33:35.720395+0000 mon.c (mon.1) 301 : audit [DBG] from='client.? 172.21.15.115:0/930425512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[46934]: audit 2022-04-22T21:33:35.292053+0000 mgr.y (mgr.34107) 8450 : audit [DBG] from='client.54073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[46934]: audit 2022-04-22T21:33:35.320195+0000 mon.a (mon.0) 12198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:36.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[46934]: cluster 2022-04-22T21:33:35.676876+0000 mgr.y (mgr.34107) 8451 : cluster [DBG] pgmap v9366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:36.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:36 smithi115 bash[46934]: audit 2022-04-22T21:33:35.720395+0000 mon.c (mon.1) 301 : audit [DBG] from='client.? 172.21.15.115:0/930425512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:33:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:37 smithi121 bash[36902]: audit 2022-04-22T21:33:36.390625+0000 mon.a (mon.0) 12199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:37 smithi115 bash[45495]: audit 2022-04-22T21:33:36.390625+0000 mon.a (mon.0) 12199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:37 smithi115 bash[46934]: audit 2022-04-22T21:33:36.390625+0000 mon.a (mon.0) 12199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:38 smithi121 bash[36902]: cluster 2022-04-22T21:33:37.677967+0000 mgr.y (mgr.34107) 8452 : cluster [DBG] pgmap v9367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:38 smithi115 bash[45495]: cluster 2022-04-22T21:33:37.677967+0000 mgr.y (mgr.34107) 8452 : cluster [DBG] pgmap v9367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:38 smithi115 bash[46934]: cluster 2022-04-22T21:33:37.677967+0000 mgr.y (mgr.34107) 8452 : cluster [DBG] pgmap v9367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:39.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:39 smithi121 bash[36902]: audit 2022-04-22T21:33:39.404862+0000 mon.a (mon.0) 12200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:39 smithi115 bash[45495]: audit 2022-04-22T21:33:39.404862+0000 mon.a (mon.0) 12200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:39 smithi115 bash[46934]: audit 2022-04-22T21:33:39.404862+0000 mon.a (mon.0) 12200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:40 smithi121 bash[36902]: cluster 2022-04-22T21:33:39.679052+0000 mgr.y (mgr.34107) 8453 : cluster [DBG] pgmap v9368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:40 smithi121 bash[36902]: audit 2022-04-22T21:33:40.323606+0000 mon.a (mon.0) 12201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:40 smithi115 bash[45495]: cluster 2022-04-22T21:33:39.679052+0000 mgr.y (mgr.34107) 8453 : cluster [DBG] pgmap v9368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:40 smithi115 bash[45495]: audit 2022-04-22T21:33:40.323606+0000 mon.a (mon.0) 12201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:40 smithi115 bash[46934]: cluster 2022-04-22T21:33:39.679052+0000 mgr.y (mgr.34107) 8453 : cluster [DBG] pgmap v9368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:40 smithi115 bash[46934]: audit 2022-04-22T21:33:40.323606+0000 mon.a (mon.0) 12201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:41 smithi121 bash[36902]: audit 2022-04-22T21:33:41.393566+0000 mon.a (mon.0) 12202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:41 smithi115 bash[45495]: audit 2022-04-22T21:33:41.393566+0000 mon.a (mon.0) 12202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:41 smithi115 bash[46934]: audit 2022-04-22T21:33:41.393566+0000 mon.a (mon.0) 12202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:42.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:42 smithi121 bash[36902]: cluster 2022-04-22T21:33:41.679923+0000 mgr.y (mgr.34107) 8454 : cluster [DBG] pgmap v9369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:42 smithi115 bash[46934]: cluster 2022-04-22T21:33:41.679923+0000 mgr.y (mgr.34107) 8454 : cluster [DBG] pgmap v9369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:42.584Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:42 smithi115 bash[45495]: cluster 2022-04-22T21:33:41.679923+0000 mgr.y (mgr.34107) 8454 : cluster [DBG] pgmap v9369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:44.740 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:44.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:45.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:44 smithi121 bash[36902]: cluster 2022-04-22T21:33:43.681029+0000 mgr.y (mgr.34107) 8455 : cluster [DBG] pgmap v9370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:44 smithi121 bash[36902]: audit 2022-04-22T21:33:44.407841+0000 mon.a (mon.0) 12203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:44 smithi115 bash[45495]: cluster 2022-04-22T21:33:43.681029+0000 mgr.y (mgr.34107) 8455 : cluster [DBG] pgmap v9370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:44 smithi115 bash[45495]: audit 2022-04-22T21:33:44.407841+0000 mon.a (mon.0) 12203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:44 smithi115 bash[46934]: cluster 2022-04-22T21:33:43.681029+0000 mgr.y (mgr.34107) 8455 : cluster [DBG] pgmap v9370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:44 smithi115 bash[46934]: audit 2022-04-22T21:33:44.407841+0000 mon.a (mon.0) 12203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:46.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:45 smithi121 bash[36902]: audit 2022-04-22T21:33:45.327332+0000 mon.a (mon.0) 12204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:45 smithi115 bash[45495]: audit 2022-04-22T21:33:45.327332+0000 mon.a (mon.0) 12204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:45 smithi115 bash[46934]: audit 2022-04-22T21:33:45.327332+0000 mon.a (mon.0) 12204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:46 smithi121 bash[36902]: cluster 2022-04-22T21:33:45.682086+0000 mgr.y (mgr.34107) 8456 : cluster [DBG] pgmap v9371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:46 smithi121 bash[36902]: audit 2022-04-22T21:33:46.396783+0000 mon.a (mon.0) 12205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:46 smithi115 bash[45495]: cluster 2022-04-22T21:33:45.682086+0000 mgr.y (mgr.34107) 8456 : cluster [DBG] pgmap v9371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:46 smithi115 bash[45495]: audit 2022-04-22T21:33:46.396783+0000 mon.a (mon.0) 12205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:46 smithi115 bash[46934]: cluster 2022-04-22T21:33:45.682086+0000 mgr.y (mgr.34107) 8456 : cluster [DBG] pgmap v9371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:46 smithi115 bash[46934]: audit 2022-04-22T21:33:46.396783+0000 mon.a (mon.0) 12205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:48 smithi121 bash[36902]: cluster 2022-04-22T21:33:47.683340+0000 mgr.y (mgr.34107) 8457 : cluster [DBG] pgmap v9372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:48 smithi115 bash[45495]: cluster 2022-04-22T21:33:47.683340+0000 mgr.y (mgr.34107) 8457 : cluster [DBG] pgmap v9372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:48 smithi115 bash[46934]: cluster 2022-04-22T21:33:47.683340+0000 mgr.y (mgr.34107) 8457 : cluster [DBG] pgmap v9372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:49 smithi121 bash[36902]: audit 2022-04-22T21:33:49.411072+0000 mon.a (mon.0) 12206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:49 smithi115 bash[46934]: audit 2022-04-22T21:33:49.411072+0000 mon.a (mon.0) 12206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:49 smithi115 bash[45495]: audit 2022-04-22T21:33:49.411072+0000 mon.a (mon.0) 12206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:50 smithi115 bash[45495]: cluster 2022-04-22T21:33:49.684154+0000 mgr.y (mgr.34107) 8458 : cluster [DBG] pgmap v9373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:50 smithi115 bash[45495]: audit 2022-04-22T21:33:50.331170+0000 mon.a (mon.0) 12207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:50 smithi115 bash[46934]: cluster 2022-04-22T21:33:49.684154+0000 mgr.y (mgr.34107) 8458 : cluster [DBG] pgmap v9373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:50 smithi115 bash[46934]: audit 2022-04-22T21:33:50.331170+0000 mon.a (mon.0) 12207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:50 smithi121 bash[36902]: cluster 2022-04-22T21:33:49.684154+0000 mgr.y (mgr.34107) 8458 : cluster [DBG] pgmap v9373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:50 smithi121 bash[36902]: audit 2022-04-22T21:33:50.331170+0000 mon.a (mon.0) 12207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:51 smithi115 bash[45495]: audit 2022-04-22T21:33:51.399442+0000 mon.a (mon.0) 12208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:51 smithi115 bash[46934]: audit 2022-04-22T21:33:51.399442+0000 mon.a (mon.0) 12208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:51 smithi121 bash[36902]: audit 2022-04-22T21:33:51.399442+0000 mon.a (mon.0) 12208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:52 smithi115 bash[45495]: cluster 2022-04-22T21:33:51.685167+0000 mgr.y (mgr.34107) 8459 : cluster [DBG] pgmap v9374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:52 smithi115 bash[46934]: cluster 2022-04-22T21:33:51.685167+0000 mgr.y (mgr.34107) 8459 : cluster [DBG] pgmap v9374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:52.584Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:52 smithi121 bash[36902]: cluster 2022-04-22T21:33:51.685167+0000 mgr.y (mgr.34107) 8459 : cluster [DBG] pgmap v9374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:54.777 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:33:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:33:54.450Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:33:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:54 smithi115 bash[46934]: cluster 2022-04-22T21:33:53.686190+0000 mgr.y (mgr.34107) 8460 : cluster [DBG] pgmap v9375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:55.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:54 smithi115 bash[46934]: audit 2022-04-22T21:33:54.414092+0000 mon.a (mon.0) 12209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:54 smithi115 bash[45495]: cluster 2022-04-22T21:33:53.686190+0000 mgr.y (mgr.34107) 8460 : cluster [DBG] pgmap v9375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:54 smithi115 bash[45495]: audit 2022-04-22T21:33:54.414092+0000 mon.a (mon.0) 12209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:54 smithi121 bash[36902]: cluster 2022-04-22T21:33:53.686190+0000 mgr.y (mgr.34107) 8460 : cluster [DBG] pgmap v9375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:54 smithi121 bash[36902]: audit 2022-04-22T21:33:54.414092+0000 mon.a (mon.0) 12209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:33:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:55 smithi115 bash[45495]: audit 2022-04-22T21:33:55.334960+0000 mon.a (mon.0) 12210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:55 smithi115 bash[46934]: audit 2022-04-22T21:33:55.334960+0000 mon.a (mon.0) 12210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:55 smithi121 bash[36902]: audit 2022-04-22T21:33:55.334960+0000 mon.a (mon.0) 12210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:33:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:56 smithi115 bash[45495]: cluster 2022-04-22T21:33:55.687258+0000 mgr.y (mgr.34107) 8461 : cluster [DBG] pgmap v9376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:56 smithi115 bash[45495]: audit 2022-04-22T21:33:56.402457+0000 mon.a (mon.0) 12211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:56 smithi115 bash[46934]: cluster 2022-04-22T21:33:55.687258+0000 mgr.y (mgr.34107) 8461 : cluster [DBG] pgmap v9376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:56 smithi115 bash[46934]: audit 2022-04-22T21:33:56.402457+0000 mon.a (mon.0) 12211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:56 smithi121 bash[36902]: cluster 2022-04-22T21:33:55.687258+0000 mgr.y (mgr.34107) 8461 : cluster [DBG] pgmap v9376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:56 smithi121 bash[36902]: audit 2022-04-22T21:33:56.402457+0000 mon.a (mon.0) 12211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:33:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:58 smithi115 bash[45495]: cluster 2022-04-22T21:33:57.688545+0000 mgr.y (mgr.34107) 8462 : cluster [DBG] pgmap v9377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:58 smithi115 bash[46934]: cluster 2022-04-22T21:33:57.688545+0000 mgr.y (mgr.34107) 8462 : cluster [DBG] pgmap v9377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:33:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:58 smithi121 bash[36902]: cluster 2022-04-22T21:33:57.688545+0000 mgr.y (mgr.34107) 8462 : cluster [DBG] pgmap v9377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:33:59 smithi121 bash[36902]: audit 2022-04-22T21:33:59.417047+0000 mon.a (mon.0) 12212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:33:59 smithi115 bash[45495]: audit 2022-04-22T21:33:59.417047+0000 mon.a (mon.0) 12212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:33:59 smithi115 bash[46934]: audit 2022-04-22T21:33:59.417047+0000 mon.a (mon.0) 12212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:00 smithi121 bash[36902]: cluster 2022-04-22T21:33:59.689635+0000 mgr.y (mgr.34107) 8463 : cluster [DBG] pgmap v9378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:00 smithi121 bash[36902]: audit 2022-04-22T21:34:00.338798+0000 mon.a (mon.0) 12213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:00 smithi115 bash[45495]: cluster 2022-04-22T21:33:59.689635+0000 mgr.y (mgr.34107) 8463 : cluster [DBG] pgmap v9378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:00 smithi115 bash[45495]: audit 2022-04-22T21:34:00.338798+0000 mon.a (mon.0) 12213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:00 smithi115 bash[46934]: cluster 2022-04-22T21:33:59.689635+0000 mgr.y (mgr.34107) 8463 : cluster [DBG] pgmap v9378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:00 smithi115 bash[46934]: audit 2022-04-22T21:34:00.338798+0000 mon.a (mon.0) 12213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:02.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:01 smithi121 bash[36902]: audit 2022-04-22T21:34:01.405447+0000 mon.a (mon.0) 12214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:02.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:01 smithi115 bash[46934]: audit 2022-04-22T21:34:01.405447+0000 mon.a (mon.0) 12214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:01 smithi115 bash[45495]: audit 2022-04-22T21:34:01.405447+0000 mon.a (mon.0) 12214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:02 smithi115 bash[45495]: cluster 2022-04-22T21:34:01.690702+0000 mgr.y (mgr.34107) 8464 : cluster [DBG] pgmap v9379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:02.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:02 smithi115 bash[46934]: cluster 2022-04-22T21:34:01.690702+0000 mgr.y (mgr.34107) 8464 : cluster [DBG] pgmap v9379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:02 smithi121 bash[36902]: cluster 2022-04-22T21:34:01.690702+0000 mgr.y (mgr.34107) 8464 : cluster [DBG] pgmap v9379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:04.450Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:04 smithi121 bash[36902]: cluster 2022-04-22T21:34:03.691838+0000 mgr.y (mgr.34107) 8465 : cluster [DBG] pgmap v9380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:05.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:04 smithi121 bash[36902]: audit 2022-04-22T21:34:04.420228+0000 mon.a (mon.0) 12215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:05.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:04 smithi115 bash[45495]: cluster 2022-04-22T21:34:03.691838+0000 mgr.y (mgr.34107) 8465 : cluster [DBG] pgmap v9380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:05.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:04 smithi115 bash[45495]: audit 2022-04-22T21:34:04.420228+0000 mon.a (mon.0) 12215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:05.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:04 smithi115 bash[46934]: cluster 2022-04-22T21:34:03.691838+0000 mgr.y (mgr.34107) 8465 : cluster [DBG] pgmap v9380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:05.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:04 smithi115 bash[46934]: audit 2022-04-22T21:34:04.420228+0000 mon.a (mon.0) 12215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:06.112 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:34:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:05 smithi121 bash[36902]: audit 2022-04-22T21:34:05.342593+0000 mon.a (mon.0) 12216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:05 smithi115 bash[46934]: audit 2022-04-22T21:34:05.342593+0000 mon.a (mon.0) 12216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:05 smithi115 bash[45495]: audit 2022-04-22T21:34:05.342593+0000 mon.a (mon.0) 12216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:06.456 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:34:06.457 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 9m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:34:06.457 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 33s ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:34:06.457 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:34:06.457 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 33s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:34:06.457 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 33s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 33s ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 9m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 33s ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:34:06.458 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:34:06.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:34:06.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:34:06.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:34:06.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 33s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:34:06.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 33s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:34:06.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 33s ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:34:06.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (3h) 33s ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:34:06.460 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 33s ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:34:06.872 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:34:06.872 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:34:06.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:34:06.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:06.873 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:34:06.873 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:34:06.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:06.873 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:34:06.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:34:06.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:06.874 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:34:06.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:34:06.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:06.875 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:34:06.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:34:06.875 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:34:06.875 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:34:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:06 smithi121 bash[36902]: cluster 2022-04-22T21:34:05.692846+0000 mgr.y (mgr.34107) 8466 : cluster [DBG] pgmap v9381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:06 smithi121 bash[36902]: audit 2022-04-22T21:34:06.408201+0000 mon.a (mon.0) 12217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:06 smithi115 bash[45495]: cluster 2022-04-22T21:34:05.692846+0000 mgr.y (mgr.34107) 8466 : cluster [DBG] pgmap v9381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:06 smithi115 bash[45495]: audit 2022-04-22T21:34:06.408201+0000 mon.a (mon.0) 12217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:06 smithi115 bash[46934]: cluster 2022-04-22T21:34:05.692846+0000 mgr.y (mgr.34107) 8466 : cluster [DBG] pgmap v9381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:06 smithi115 bash[46934]: audit 2022-04-22T21:34:06.408201+0000 mon.a (mon.0) 12217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:07 smithi121 bash[36902]: audit 2022-04-22T21:34:06.093815+0000 mgr.y (mgr.34107) 8467 : audit [DBG] from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:07 smithi121 bash[36902]: audit 2022-04-22T21:34:06.431793+0000 mgr.y (mgr.34107) 8468 : audit [DBG] from='client.54091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:07 smithi121 bash[36902]: audit 2022-04-22T21:34:06.868161+0000 mon.c (mon.1) 302 : audit [DBG] from='client.? 172.21.15.115:0/881839335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[45495]: audit 2022-04-22T21:34:06.093815+0000 mgr.y (mgr.34107) 8467 : audit [DBG] from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[45495]: audit 2022-04-22T21:34:06.431793+0000 mgr.y (mgr.34107) 8468 : audit [DBG] from='client.54091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[45495]: audit 2022-04-22T21:34:06.868161+0000 mon.c (mon.1) 302 : audit [DBG] from='client.? 172.21.15.115:0/881839335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[46934]: audit 2022-04-22T21:34:06.093815+0000 mgr.y (mgr.34107) 8467 : audit [DBG] from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[46934]: audit 2022-04-22T21:34:06.431793+0000 mgr.y (mgr.34107) 8468 : audit [DBG] from='client.54091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:07 smithi115 bash[46934]: audit 2022-04-22T21:34:06.868161+0000 mon.c (mon.1) 302 : audit [DBG] from='client.? 172.21.15.115:0/881839335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:08 smithi121 bash[36902]: cluster 2022-04-22T21:34:07.693911+0000 mgr.y (mgr.34107) 8469 : cluster [DBG] pgmap v9382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:08 smithi115 bash[45495]: cluster 2022-04-22T21:34:07.693911+0000 mgr.y (mgr.34107) 8469 : cluster [DBG] pgmap v9382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:08 smithi115 bash[46934]: cluster 2022-04-22T21:34:07.693911+0000 mgr.y (mgr.34107) 8469 : cluster [DBG] pgmap v9382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:10.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:09 smithi121 bash[36902]: audit 2022-04-22T21:34:09.423320+0000 mon.a (mon.0) 12218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:09 smithi115 bash[45495]: audit 2022-04-22T21:34:09.423320+0000 mon.a (mon.0) 12218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:09 smithi115 bash[46934]: audit 2022-04-22T21:34:09.423320+0000 mon.a (mon.0) 12218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:10 smithi121 bash[36902]: cluster 2022-04-22T21:34:09.695064+0000 mgr.y (mgr.34107) 8470 : cluster [DBG] pgmap v9383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:10 smithi121 bash[36902]: audit 2022-04-22T21:34:10.346557+0000 mon.a (mon.0) 12219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:10 smithi115 bash[45495]: cluster 2022-04-22T21:34:09.695064+0000 mgr.y (mgr.34107) 8470 : cluster [DBG] pgmap v9383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:10 smithi115 bash[45495]: audit 2022-04-22T21:34:10.346557+0000 mon.a (mon.0) 12219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:10 smithi115 bash[46934]: cluster 2022-04-22T21:34:09.695064+0000 mgr.y (mgr.34107) 8470 : cluster [DBG] pgmap v9383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:10 smithi115 bash[46934]: audit 2022-04-22T21:34:10.346557+0000 mon.a (mon.0) 12219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:12.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:11 smithi121 bash[36902]: audit 2022-04-22T21:34:11.411161+0000 mon.a (mon.0) 12220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:11 smithi115 bash[45495]: audit 2022-04-22T21:34:11.411161+0000 mon.a (mon.0) 12220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:11 smithi115 bash[46934]: audit 2022-04-22T21:34:11.411161+0000 mon.a (mon.0) 12220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:12.848 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:12.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:12 smithi121 bash[36902]: cluster 2022-04-22T21:34:11.696237+0000 mgr.y (mgr.34107) 8471 : cluster [DBG] pgmap v9384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:12 smithi115 bash[45495]: cluster 2022-04-22T21:34:11.696237+0000 mgr.y (mgr.34107) 8471 : cluster [DBG] pgmap v9384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:12 smithi115 bash[46934]: cluster 2022-04-22T21:34:11.696237+0000 mgr.y (mgr.34107) 8471 : cluster [DBG] pgmap v9384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:14.451Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:14 smithi121 bash[36902]: cluster 2022-04-22T21:34:13.697362+0000 mgr.y (mgr.34107) 8472 : cluster [DBG] pgmap v9385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:14 smithi121 bash[36902]: audit 2022-04-22T21:34:14.426964+0000 mon.a (mon.0) 12221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:15.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:14 smithi115 bash[46934]: cluster 2022-04-22T21:34:13.697362+0000 mgr.y (mgr.34107) 8472 : cluster [DBG] pgmap v9385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:14 smithi115 bash[46934]: audit 2022-04-22T21:34:14.426964+0000 mon.a (mon.0) 12221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:14 smithi115 bash[45495]: cluster 2022-04-22T21:34:13.697362+0000 mgr.y (mgr.34107) 8472 : cluster [DBG] pgmap v9385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:14 smithi115 bash[45495]: audit 2022-04-22T21:34:14.426964+0000 mon.a (mon.0) 12221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:15 smithi121 bash[36902]: audit 2022-04-22T21:34:15.350359+0000 mon.a (mon.0) 12222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:16.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:15 smithi115 bash[46934]: audit 2022-04-22T21:34:15.350359+0000 mon.a (mon.0) 12222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:15 smithi115 bash[45495]: audit 2022-04-22T21:34:15.350359+0000 mon.a (mon.0) 12222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:16 smithi121 bash[36902]: cluster 2022-04-22T21:34:15.698450+0000 mgr.y (mgr.34107) 8473 : cluster [DBG] pgmap v9386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:16 smithi121 bash[36902]: audit 2022-04-22T21:34:16.126372+0000 mon.a (mon.0) 12223 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:34:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:16 smithi121 bash[36902]: audit 2022-04-22T21:34:16.128073+0000 mon.a (mon.0) 12224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:34:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:16 smithi121 bash[36902]: audit 2022-04-22T21:34:16.414222+0000 mon.a (mon.0) 12225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[45495]: cluster 2022-04-22T21:34:15.698450+0000 mgr.y (mgr.34107) 8473 : cluster [DBG] pgmap v9386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[45495]: audit 2022-04-22T21:34:16.126372+0000 mon.a (mon.0) 12223 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:34:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[45495]: audit 2022-04-22T21:34:16.128073+0000 mon.a (mon.0) 12224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:34:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[45495]: audit 2022-04-22T21:34:16.414222+0000 mon.a (mon.0) 12225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[46934]: cluster 2022-04-22T21:34:15.698450+0000 mgr.y (mgr.34107) 8473 : cluster [DBG] pgmap v9386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[46934]: audit 2022-04-22T21:34:16.126372+0000 mon.a (mon.0) 12223 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:34:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[46934]: audit 2022-04-22T21:34:16.128073+0000 mon.a (mon.0) 12224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:34:17.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:16 smithi115 bash[46934]: audit 2022-04-22T21:34:16.414222+0000 mon.a (mon.0) 12225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:18 smithi121 bash[36902]: cluster 2022-04-22T21:34:17.699609+0000 mgr.y (mgr.34107) 8474 : cluster [DBG] pgmap v9387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:18 smithi115 bash[46934]: cluster 2022-04-22T21:34:17.699609+0000 mgr.y (mgr.34107) 8474 : cluster [DBG] pgmap v9387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:18 smithi115 bash[45495]: cluster 2022-04-22T21:34:17.699609+0000 mgr.y (mgr.34107) 8474 : cluster [DBG] pgmap v9387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:19 smithi121 bash[36902]: audit 2022-04-22T21:34:19.430163+0000 mon.a (mon.0) 12226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:19 smithi115 bash[45495]: audit 2022-04-22T21:34:19.430163+0000 mon.a (mon.0) 12226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:19 smithi115 bash[46934]: audit 2022-04-22T21:34:19.430163+0000 mon.a (mon.0) 12226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:20 smithi121 bash[36902]: cluster 2022-04-22T21:34:19.700660+0000 mgr.y (mgr.34107) 8475 : cluster [DBG] pgmap v9388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:20 smithi121 bash[36902]: audit 2022-04-22T21:34:20.354115+0000 mon.a (mon.0) 12227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:20 smithi115 bash[45495]: cluster 2022-04-22T21:34:19.700660+0000 mgr.y (mgr.34107) 8475 : cluster [DBG] pgmap v9388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:21.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:20 smithi115 bash[45495]: audit 2022-04-22T21:34:20.354115+0000 mon.a (mon.0) 12227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:21.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:20 smithi115 bash[46934]: cluster 2022-04-22T21:34:19.700660+0000 mgr.y (mgr.34107) 8475 : cluster [DBG] pgmap v9388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:21.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:20 smithi115 bash[46934]: audit 2022-04-22T21:34:20.354115+0000 mon.a (mon.0) 12227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:22.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:21 smithi121 bash[36902]: audit 2022-04-22T21:34:21.417142+0000 mon.a (mon.0) 12228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:22.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:21 smithi115 bash[45495]: audit 2022-04-22T21:34:21.417142+0000 mon.a (mon.0) 12228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:22.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:21 smithi115 bash[46934]: audit 2022-04-22T21:34:21.417142+0000 mon.a (mon.0) 12228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:22.897 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:22.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:22 smithi121 bash[36902]: cluster 2022-04-22T21:34:21.701772+0000 mgr.y (mgr.34107) 8476 : cluster [DBG] pgmap v9389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:22 smithi115 bash[45495]: cluster 2022-04-22T21:34:21.701772+0000 mgr.y (mgr.34107) 8476 : cluster [DBG] pgmap v9389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:22 smithi115 bash[46934]: cluster 2022-04-22T21:34:21.701772+0000 mgr.y (mgr.34107) 8476 : cluster [DBG] pgmap v9389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:24.452Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:24 smithi121 bash[36902]: cluster 2022-04-22T21:34:23.702924+0000 mgr.y (mgr.34107) 8477 : cluster [DBG] pgmap v9390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:24 smithi121 bash[36902]: audit 2022-04-22T21:34:24.433584+0000 mon.a (mon.0) 12229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:24 smithi115 bash[45495]: cluster 2022-04-22T21:34:23.702924+0000 mgr.y (mgr.34107) 8477 : cluster [DBG] pgmap v9390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:24 smithi115 bash[45495]: audit 2022-04-22T21:34:24.433584+0000 mon.a (mon.0) 12229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:24 smithi115 bash[46934]: cluster 2022-04-22T21:34:23.702924+0000 mgr.y (mgr.34107) 8477 : cluster [DBG] pgmap v9390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:24 smithi115 bash[46934]: audit 2022-04-22T21:34:24.433584+0000 mon.a (mon.0) 12229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:26.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:25 smithi121 bash[36902]: audit 2022-04-22T21:34:25.357835+0000 mon.a (mon.0) 12230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:25 smithi115 bash[45495]: audit 2022-04-22T21:34:25.357835+0000 mon.a (mon.0) 12230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:25 smithi115 bash[46934]: audit 2022-04-22T21:34:25.357835+0000 mon.a (mon.0) 12230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:26 smithi121 bash[36902]: cluster 2022-04-22T21:34:25.704009+0000 mgr.y (mgr.34107) 8478 : cluster [DBG] pgmap v9391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:26 smithi121 bash[36902]: audit 2022-04-22T21:34:26.420225+0000 mon.a (mon.0) 12231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:26 smithi115 bash[45495]: cluster 2022-04-22T21:34:25.704009+0000 mgr.y (mgr.34107) 8478 : cluster [DBG] pgmap v9391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:26 smithi115 bash[45495]: audit 2022-04-22T21:34:26.420225+0000 mon.a (mon.0) 12231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:27.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:26 smithi115 bash[46934]: cluster 2022-04-22T21:34:25.704009+0000 mgr.y (mgr.34107) 8478 : cluster [DBG] pgmap v9391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:27.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:26 smithi115 bash[46934]: audit 2022-04-22T21:34:26.420225+0000 mon.a (mon.0) 12231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:28 smithi121 bash[36902]: cluster 2022-04-22T21:34:27.705116+0000 mgr.y (mgr.34107) 8479 : cluster [DBG] pgmap v9392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:28 smithi115 bash[45495]: cluster 2022-04-22T21:34:27.705116+0000 mgr.y (mgr.34107) 8479 : cluster [DBG] pgmap v9392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:28 smithi115 bash[46934]: cluster 2022-04-22T21:34:27.705116+0000 mgr.y (mgr.34107) 8479 : cluster [DBG] pgmap v9392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:29 smithi121 bash[36902]: audit 2022-04-22T21:34:29.436724+0000 mon.a (mon.0) 12232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:29 smithi115 bash[45495]: audit 2022-04-22T21:34:29.436724+0000 mon.a (mon.0) 12232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:29 smithi115 bash[46934]: audit 2022-04-22T21:34:29.436724+0000 mon.a (mon.0) 12232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:30 smithi121 bash[36902]: cluster 2022-04-22T21:34:29.706166+0000 mgr.y (mgr.34107) 8480 : cluster [DBG] pgmap v9393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:30 smithi121 bash[36902]: audit 2022-04-22T21:34:30.361670+0000 mon.a (mon.0) 12233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:30 smithi115 bash[45495]: cluster 2022-04-22T21:34:29.706166+0000 mgr.y (mgr.34107) 8480 : cluster [DBG] pgmap v9393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:30 smithi115 bash[45495]: audit 2022-04-22T21:34:30.361670+0000 mon.a (mon.0) 12233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:30 smithi115 bash[46934]: cluster 2022-04-22T21:34:29.706166+0000 mgr.y (mgr.34107) 8480 : cluster [DBG] pgmap v9393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:30 smithi115 bash[46934]: audit 2022-04-22T21:34:30.361670+0000 mon.a (mon.0) 12233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:31 smithi121 bash[36902]: audit 2022-04-22T21:34:31.423331+0000 mon.a (mon.0) 12234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:31 smithi115 bash[45495]: audit 2022-04-22T21:34:31.423331+0000 mon.a (mon.0) 12234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:31 smithi115 bash[46934]: audit 2022-04-22T21:34:31.423331+0000 mon.a (mon.0) 12234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:32.927 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:32.586Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:32 smithi121 bash[36902]: cluster 2022-04-22T21:34:31.707235+0000 mgr.y (mgr.34107) 8481 : cluster [DBG] pgmap v9394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:32 smithi115 bash[45495]: cluster 2022-04-22T21:34:31.707235+0000 mgr.y (mgr.34107) 8481 : cluster [DBG] pgmap v9394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:32 smithi115 bash[46934]: cluster 2022-04-22T21:34:31.707235+0000 mgr.y (mgr.34107) 8481 : cluster [DBG] pgmap v9394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:34.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:33 smithi121 bash[36902]: audit 2022-04-22T21:34:33.477310+0000 mon.a (mon.0) 12235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:34:34.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:33 smithi121 bash[36902]: audit 2022-04-22T21:34:33.480420+0000 mon.a (mon.0) 12236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:34:34.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:33 smithi121 bash[36902]: audit 2022-04-22T21:34:33.482411+0000 mon.a (mon.0) 12237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:34:34.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:33 smithi121 bash[36902]: audit 2022-04-22T21:34:33.818007+0000 mon.a (mon.0) 12238 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:33 smithi121 bash[36902]: audit 2022-04-22T21:34:33.823300+0000 mon.a (mon.0) 12239 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[46934]: audit 2022-04-22T21:34:33.477310+0000 mon.a (mon.0) 12235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[46934]: audit 2022-04-22T21:34:33.480420+0000 mon.a (mon.0) 12236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[46934]: audit 2022-04-22T21:34:33.482411+0000 mon.a (mon.0) 12237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[46934]: audit 2022-04-22T21:34:33.818007+0000 mon.a (mon.0) 12238 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[46934]: audit 2022-04-22T21:34:33.823300+0000 mon.a (mon.0) 12239 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[45495]: audit 2022-04-22T21:34:33.477310+0000 mon.a (mon.0) 12235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:34:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[45495]: audit 2022-04-22T21:34:33.480420+0000 mon.a (mon.0) 12236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:34:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[45495]: audit 2022-04-22T21:34:33.482411+0000 mon.a (mon.0) 12237 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:34:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[45495]: audit 2022-04-22T21:34:33.818007+0000 mon.a (mon.0) 12238 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:33 smithi115 bash[45495]: audit 2022-04-22T21:34:33.823300+0000 mon.a (mon.0) 12239 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:34.452Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:34 smithi121 bash[36902]: cluster 2022-04-22T21:34:33.708072+0000 mgr.y (mgr.34107) 8482 : cluster [DBG] pgmap v9395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:34 smithi121 bash[36902]: audit 2022-04-22T21:34:34.439905+0000 mon.a (mon.0) 12240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:34 smithi115 bash[45495]: cluster 2022-04-22T21:34:33.708072+0000 mgr.y (mgr.34107) 8482 : cluster [DBG] pgmap v9395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:34 smithi115 bash[45495]: audit 2022-04-22T21:34:34.439905+0000 mon.a (mon.0) 12240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:34 smithi115 bash[46934]: cluster 2022-04-22T21:34:33.708072+0000 mgr.y (mgr.34107) 8482 : cluster [DBG] pgmap v9395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:34 smithi115 bash[46934]: audit 2022-04-22T21:34:34.439905+0000 mon.a (mon.0) 12240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:35 smithi121 bash[36902]: audit 2022-04-22T21:34:35.365591+0000 mon.a (mon.0) 12241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:35 smithi115 bash[45495]: audit 2022-04-22T21:34:35.365591+0000 mon.a (mon.0) 12241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:35 smithi115 bash[46934]: audit 2022-04-22T21:34:35.365591+0000 mon.a (mon.0) 12241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:37.244 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:34:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:36 smithi121 bash[36902]: cluster 2022-04-22T21:34:35.709109+0000 mgr.y (mgr.34107) 8483 : cluster [DBG] pgmap v9396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:36 smithi121 bash[36902]: audit 2022-04-22T21:34:36.426179+0000 mon.a (mon.0) 12242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:36 smithi115 bash[45495]: cluster 2022-04-22T21:34:35.709109+0000 mgr.y (mgr.34107) 8483 : cluster [DBG] pgmap v9396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:36 smithi115 bash[45495]: audit 2022-04-22T21:34:36.426179+0000 mon.a (mon.0) 12242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:36 smithi115 bash[46934]: cluster 2022-04-22T21:34:35.709109+0000 mgr.y (mgr.34107) 8483 : cluster [DBG] pgmap v9396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:36 smithi115 bash[46934]: audit 2022-04-22T21:34:36.426179+0000 mon.a (mon.0) 12242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:34:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 10m ago 4h 15.6M - ba2b418f427c 9255ba08f72f 2022-04-22T21:34:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 64s ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:34:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (3h) 10m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 64s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 64s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 10m ago 4h 489M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 10m ago 4h 381M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 64s ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 10m ago 4h 251M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 10m ago 4h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 64s ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 10m ago 4h 712M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:34:37.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 10m ago 4h 791M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 10m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 10m ago 4h 495M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 64s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 64s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 64s ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 64s ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:34:37.613 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 64s ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:34:38.051 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:34:38.051 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:34:38.051 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:34:38.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:34:38.053 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:34:38.053 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:34:38.053 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:34:38.053 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:34:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: audit 2022-04-22T21:34:37.222181+0000 mgr.y (mgr.34107) 8484 : audit [DBG] from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: audit 2022-04-22T21:34:37.587502+0000 mgr.y (mgr.34107) 8485 : audit [DBG] from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: cluster 2022-04-22T21:34:37.710138+0000 mgr.y (mgr.34107) 8486 : cluster [DBG] pgmap v9397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: audit 2022-04-22T21:34:37.941141+0000 mon.a (mon.0) 12243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: audit 2022-04-22T21:34:38.047437+0000 mon.a (mon.0) 12244 : audit [DBG] from='client.? 172.21.15.115:0/3986536955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:38 smithi121 bash[36902]: audit 2022-04-22T21:34:38.247842+0000 mon.a (mon.0) 12245 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: audit 2022-04-22T21:34:37.222181+0000 mgr.y (mgr.34107) 8484 : audit [DBG] from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: audit 2022-04-22T21:34:37.587502+0000 mgr.y (mgr.34107) 8485 : audit [DBG] from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: cluster 2022-04-22T21:34:37.710138+0000 mgr.y (mgr.34107) 8486 : cluster [DBG] pgmap v9397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: audit 2022-04-22T21:34:37.941141+0000 mon.a (mon.0) 12243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: audit 2022-04-22T21:34:38.047437+0000 mon.a (mon.0) 12244 : audit [DBG] from='client.? 172.21.15.115:0/3986536955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:39.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[45495]: audit 2022-04-22T21:34:38.247842+0000 mon.a (mon.0) 12245 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:39.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: audit 2022-04-22T21:34:37.222181+0000 mgr.y (mgr.34107) 8484 : audit [DBG] from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: audit 2022-04-22T21:34:37.587502+0000 mgr.y (mgr.34107) 8485 : audit [DBG] from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:34:39.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: cluster 2022-04-22T21:34:37.710138+0000 mgr.y (mgr.34107) 8486 : cluster [DBG] pgmap v9397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:39.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: audit 2022-04-22T21:34:37.941141+0000 mon.a (mon.0) 12243 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:39.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: audit 2022-04-22T21:34:38.047437+0000 mon.a (mon.0) 12244 : audit [DBG] from='client.? 172.21.15.115:0/3986536955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:34:39.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:38 smithi115 bash[46934]: audit 2022-04-22T21:34:38.247842+0000 mon.a (mon.0) 12245 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:34:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:39 smithi121 bash[36902]: audit 2022-04-22T21:34:39.443220+0000 mon.a (mon.0) 12246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:40.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:39 smithi115 bash[46934]: audit 2022-04-22T21:34:39.443220+0000 mon.a (mon.0) 12246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:39 smithi115 bash[45495]: audit 2022-04-22T21:34:39.443220+0000 mon.a (mon.0) 12246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:40 smithi121 bash[36902]: cluster 2022-04-22T21:34:39.711206+0000 mgr.y (mgr.34107) 8487 : cluster [DBG] pgmap v9398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:40 smithi121 bash[36902]: audit 2022-04-22T21:34:40.369528+0000 mon.a (mon.0) 12247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:40 smithi115 bash[45495]: cluster 2022-04-22T21:34:39.711206+0000 mgr.y (mgr.34107) 8487 : cluster [DBG] pgmap v9398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:40 smithi115 bash[45495]: audit 2022-04-22T21:34:40.369528+0000 mon.a (mon.0) 12247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:40 smithi115 bash[46934]: cluster 2022-04-22T21:34:39.711206+0000 mgr.y (mgr.34107) 8487 : cluster [DBG] pgmap v9398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:40 smithi115 bash[46934]: audit 2022-04-22T21:34:40.369528+0000 mon.a (mon.0) 12247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:41 smithi121 bash[36902]: audit 2022-04-22T21:34:41.429230+0000 mon.a (mon.0) 12248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:41 smithi115 bash[45495]: audit 2022-04-22T21:34:41.429230+0000 mon.a (mon.0) 12248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:41 smithi115 bash[46934]: audit 2022-04-22T21:34:41.429230+0000 mon.a (mon.0) 12248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:42.965 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:42.587Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:42 smithi121 bash[36902]: cluster 2022-04-22T21:34:41.712440+0000 mgr.y (mgr.34107) 8488 : cluster [DBG] pgmap v9399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:42 smithi115 bash[45495]: cluster 2022-04-22T21:34:41.712440+0000 mgr.y (mgr.34107) 8488 : cluster [DBG] pgmap v9399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:42 smithi115 bash[46934]: cluster 2022-04-22T21:34:41.712440+0000 mgr.y (mgr.34107) 8488 : cluster [DBG] pgmap v9399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:44.455Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:44 smithi121 bash[36902]: cluster 2022-04-22T21:34:43.713723+0000 mgr.y (mgr.34107) 8489 : cluster [DBG] pgmap v9400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:44 smithi121 bash[36902]: audit 2022-04-22T21:34:44.446418+0000 mon.a (mon.0) 12249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:44 smithi115 bash[46934]: cluster 2022-04-22T21:34:43.713723+0000 mgr.y (mgr.34107) 8489 : cluster [DBG] pgmap v9400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:44 smithi115 bash[46934]: audit 2022-04-22T21:34:44.446418+0000 mon.a (mon.0) 12249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:44 smithi115 bash[45495]: cluster 2022-04-22T21:34:43.713723+0000 mgr.y (mgr.34107) 8489 : cluster [DBG] pgmap v9400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:44 smithi115 bash[45495]: audit 2022-04-22T21:34:44.446418+0000 mon.a (mon.0) 12249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:45 smithi121 bash[36902]: audit 2022-04-22T21:34:45.373186+0000 mon.a (mon.0) 12250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:45 smithi115 bash[46934]: audit 2022-04-22T21:34:45.373186+0000 mon.a (mon.0) 12250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:45 smithi115 bash[45495]: audit 2022-04-22T21:34:45.373186+0000 mon.a (mon.0) 12250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:46 smithi121 bash[36902]: cluster 2022-04-22T21:34:45.714834+0000 mgr.y (mgr.34107) 8490 : cluster [DBG] pgmap v9401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:46 smithi121 bash[36902]: audit 2022-04-22T21:34:46.432489+0000 mon.a (mon.0) 12251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:46 smithi115 bash[46934]: cluster 2022-04-22T21:34:45.714834+0000 mgr.y (mgr.34107) 8490 : cluster [DBG] pgmap v9401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:46 smithi115 bash[46934]: audit 2022-04-22T21:34:46.432489+0000 mon.a (mon.0) 12251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:46 smithi115 bash[45495]: cluster 2022-04-22T21:34:45.714834+0000 mgr.y (mgr.34107) 8490 : cluster [DBG] pgmap v9401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:46 smithi115 bash[45495]: audit 2022-04-22T21:34:46.432489+0000 mon.a (mon.0) 12251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:48 smithi121 bash[36902]: cluster 2022-04-22T21:34:47.716001+0000 mgr.y (mgr.34107) 8491 : cluster [DBG] pgmap v9402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:48 smithi115 bash[45495]: cluster 2022-04-22T21:34:47.716001+0000 mgr.y (mgr.34107) 8491 : cluster [DBG] pgmap v9402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:48 smithi115 bash[46934]: cluster 2022-04-22T21:34:47.716001+0000 mgr.y (mgr.34107) 8491 : cluster [DBG] pgmap v9402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:49 smithi121 bash[36902]: audit 2022-04-22T21:34:49.449447+0000 mon.a (mon.0) 12252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:49 smithi115 bash[45495]: audit 2022-04-22T21:34:49.449447+0000 mon.a (mon.0) 12252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:49 smithi115 bash[46934]: audit 2022-04-22T21:34:49.449447+0000 mon.a (mon.0) 12252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:50 smithi121 bash[36902]: cluster 2022-04-22T21:34:49.717160+0000 mgr.y (mgr.34107) 8492 : cluster [DBG] pgmap v9403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:50 smithi121 bash[36902]: audit 2022-04-22T21:34:50.377069+0000 mon.a (mon.0) 12253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:50 smithi115 bash[45495]: cluster 2022-04-22T21:34:49.717160+0000 mgr.y (mgr.34107) 8492 : cluster [DBG] pgmap v9403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:50 smithi115 bash[45495]: audit 2022-04-22T21:34:50.377069+0000 mon.a (mon.0) 12253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:50 smithi115 bash[46934]: cluster 2022-04-22T21:34:49.717160+0000 mgr.y (mgr.34107) 8492 : cluster [DBG] pgmap v9403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:50 smithi115 bash[46934]: audit 2022-04-22T21:34:50.377069+0000 mon.a (mon.0) 12253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:51 smithi121 bash[36902]: audit 2022-04-22T21:34:51.435793+0000 mon.a (mon.0) 12254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:51 smithi115 bash[45495]: audit 2022-04-22T21:34:51.435793+0000 mon.a (mon.0) 12254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:51 smithi115 bash[46934]: audit 2022-04-22T21:34:51.435793+0000 mon.a (mon.0) 12254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:53.001 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:52.587Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:53 smithi121 bash[36902]: cluster 2022-04-22T21:34:51.718332+0000 mgr.y (mgr.34107) 8493 : cluster [DBG] pgmap v9404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:52 smithi115 bash[45495]: cluster 2022-04-22T21:34:51.718332+0000 mgr.y (mgr.34107) 8493 : cluster [DBG] pgmap v9404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:52 smithi115 bash[46934]: cluster 2022-04-22T21:34:51.718332+0000 mgr.y (mgr.34107) 8493 : cluster [DBG] pgmap v9404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:34:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:34:54.456Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:34:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:55 smithi121 bash[36902]: cluster 2022-04-22T21:34:53.719535+0000 mgr.y (mgr.34107) 8494 : cluster [DBG] pgmap v9405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:55 smithi121 bash[36902]: audit 2022-04-22T21:34:54.452460+0000 mon.a (mon.0) 12255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:55 smithi115 bash[45495]: cluster 2022-04-22T21:34:53.719535+0000 mgr.y (mgr.34107) 8494 : cluster [DBG] pgmap v9405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:55 smithi115 bash[45495]: audit 2022-04-22T21:34:54.452460+0000 mon.a (mon.0) 12255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:55 smithi115 bash[46934]: cluster 2022-04-22T21:34:53.719535+0000 mgr.y (mgr.34107) 8494 : cluster [DBG] pgmap v9405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:55 smithi115 bash[46934]: audit 2022-04-22T21:34:54.452460+0000 mon.a (mon.0) 12255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:34:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:56 smithi121 bash[36902]: audit 2022-04-22T21:34:55.380526+0000 mon.a (mon.0) 12256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:56 smithi115 bash[45495]: audit 2022-04-22T21:34:55.380526+0000 mon.a (mon.0) 12256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:56 smithi115 bash[46934]: audit 2022-04-22T21:34:55.380526+0000 mon.a (mon.0) 12256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:34:57.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:57 smithi121 bash[36902]: cluster 2022-04-22T21:34:55.720554+0000 mgr.y (mgr.34107) 8495 : cluster [DBG] pgmap v9406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:57.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:57 smithi121 bash[36902]: audit 2022-04-22T21:34:56.439029+0000 mon.a (mon.0) 12257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:57 smithi115 bash[45495]: cluster 2022-04-22T21:34:55.720554+0000 mgr.y (mgr.34107) 8495 : cluster [DBG] pgmap v9406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:57 smithi115 bash[45495]: audit 2022-04-22T21:34:56.439029+0000 mon.a (mon.0) 12257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:57 smithi115 bash[46934]: cluster 2022-04-22T21:34:55.720554+0000 mgr.y (mgr.34107) 8495 : cluster [DBG] pgmap v9406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:57.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:57 smithi115 bash[46934]: audit 2022-04-22T21:34:56.439029+0000 mon.a (mon.0) 12257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:34:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:34:59 smithi121 bash[36902]: cluster 2022-04-22T21:34:57.721668+0000 mgr.y (mgr.34107) 8496 : cluster [DBG] pgmap v9407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:34:59 smithi115 bash[45495]: cluster 2022-04-22T21:34:57.721668+0000 mgr.y (mgr.34107) 8496 : cluster [DBG] pgmap v9407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:34:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:34:59 smithi115 bash[46934]: cluster 2022-04-22T21:34:57.721668+0000 mgr.y (mgr.34107) 8496 : cluster [DBG] pgmap v9407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:00 smithi115 bash[46934]: audit 2022-04-22T21:34:59.456031+0000 mon.a (mon.0) 12258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:00.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:00 smithi115 bash[45495]: audit 2022-04-22T21:34:59.456031+0000 mon.a (mon.0) 12258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:00 smithi121 bash[36902]: audit 2022-04-22T21:34:59.456031+0000 mon.a (mon.0) 12258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:01 smithi115 bash[45495]: cluster 2022-04-22T21:34:59.722735+0000 mgr.y (mgr.34107) 8497 : cluster [DBG] pgmap v9408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:01 smithi115 bash[45495]: audit 2022-04-22T21:35:00.384388+0000 mon.a (mon.0) 12259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:01 smithi115 bash[46934]: cluster 2022-04-22T21:34:59.722735+0000 mgr.y (mgr.34107) 8497 : cluster [DBG] pgmap v9408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:01 smithi115 bash[46934]: audit 2022-04-22T21:35:00.384388+0000 mon.a (mon.0) 12259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:01 smithi121 bash[36902]: cluster 2022-04-22T21:34:59.722735+0000 mgr.y (mgr.34107) 8497 : cluster [DBG] pgmap v9408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:01 smithi121 bash[36902]: audit 2022-04-22T21:35:00.384388+0000 mon.a (mon.0) 12259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:02 smithi115 bash[45495]: audit 2022-04-22T21:35:01.442130+0000 mon.a (mon.0) 12260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:02 smithi115 bash[45495]: cluster 2022-04-22T21:35:01.723894+0000 mgr.y (mgr.34107) 8498 : cluster [DBG] pgmap v9409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:02 smithi115 bash[46934]: audit 2022-04-22T21:35:01.442130+0000 mon.a (mon.0) 12260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:02 smithi115 bash[46934]: cluster 2022-04-22T21:35:01.723894+0000 mgr.y (mgr.34107) 8498 : cluster [DBG] pgmap v9409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:02 smithi121 bash[36902]: audit 2022-04-22T21:35:01.442130+0000 mon.a (mon.0) 12260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:02 smithi121 bash[36902]: cluster 2022-04-22T21:35:01.723894+0000 mgr.y (mgr.34107) 8498 : cluster [DBG] pgmap v9409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:02.588Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:04.785 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:04.455Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:04 smithi115 bash[45495]: cluster 2022-04-22T21:35:03.725064+0000 mgr.y (mgr.34107) 8499 : cluster [DBG] pgmap v9410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:04 smithi115 bash[45495]: audit 2022-04-22T21:35:04.459294+0000 mon.a (mon.0) 12261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:04 smithi115 bash[46934]: cluster 2022-04-22T21:35:03.725064+0000 mgr.y (mgr.34107) 8499 : cluster [DBG] pgmap v9410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:04 smithi115 bash[46934]: audit 2022-04-22T21:35:04.459294+0000 mon.a (mon.0) 12261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:04 smithi121 bash[36902]: cluster 2022-04-22T21:35:03.725064+0000 mgr.y (mgr.34107) 8499 : cluster [DBG] pgmap v9410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:04 smithi121 bash[36902]: audit 2022-04-22T21:35:04.459294+0000 mon.a (mon.0) 12261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:05 smithi115 bash[45495]: audit 2022-04-22T21:35:05.388017+0000 mon.a (mon.0) 12262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:05 smithi115 bash[46934]: audit 2022-04-22T21:35:05.388017+0000 mon.a (mon.0) 12262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:05 smithi121 bash[36902]: audit 2022-04-22T21:35:05.388017+0000 mon.a (mon.0) 12262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:06 smithi121 bash[36902]: cluster 2022-04-22T21:35:05.726212+0000 mgr.y (mgr.34107) 8500 : cluster [DBG] pgmap v9411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:06 smithi121 bash[36902]: audit 2022-04-22T21:35:06.445159+0000 mon.a (mon.0) 12263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:06 smithi115 bash[45495]: cluster 2022-04-22T21:35:05.726212+0000 mgr.y (mgr.34107) 8500 : cluster [DBG] pgmap v9411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:06 smithi115 bash[45495]: audit 2022-04-22T21:35:06.445159+0000 mon.a (mon.0) 12263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:06 smithi115 bash[46934]: cluster 2022-04-22T21:35:05.726212+0000 mgr.y (mgr.34107) 8500 : cluster [DBG] pgmap v9411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:06 smithi115 bash[46934]: audit 2022-04-22T21:35:06.445159+0000 mon.a (mon.0) 12263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:08.427 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:35:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:35:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 30s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:35:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 95s ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 30s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (3h) 95s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 95s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 30s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 30s ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 95s ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 30s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 30s ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 95s ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:35:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 30s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 30s ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 30s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 30s ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 95s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 95s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 95s ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 95s ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:35:08.774 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 95s ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:35:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:35:09.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:35:09.200 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:35:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:08 smithi121 bash[36902]: cluster 2022-04-22T21:35:07.727373+0000 mgr.y (mgr.34107) 8501 : cluster [DBG] pgmap v9412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:09.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:08 smithi115 bash[45495]: cluster 2022-04-22T21:35:07.727373+0000 mgr.y (mgr.34107) 8501 : cluster [DBG] pgmap v9412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:09.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:08 smithi115 bash[46934]: cluster 2022-04-22T21:35:07.727373+0000 mgr.y (mgr.34107) 8501 : cluster [DBG] pgmap v9412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:09 smithi121 bash[36902]: audit 2022-04-22T21:35:08.408121+0000 mgr.y (mgr.34107) 8502 : audit [DBG] from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:09 smithi121 bash[36902]: audit 2022-04-22T21:35:08.749303+0000 mgr.y (mgr.34107) 8503 : audit [DBG] from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:09 smithi121 bash[36902]: audit 2022-04-22T21:35:09.174257+0000 mon.a (mon.0) 12264 : audit [DBG] from='client.? 172.21.15.115:0/1337541541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:09 smithi121 bash[36902]: audit 2022-04-22T21:35:09.462132+0000 mon.a (mon.0) 12265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[45495]: audit 2022-04-22T21:35:08.408121+0000 mgr.y (mgr.34107) 8502 : audit [DBG] from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[45495]: audit 2022-04-22T21:35:08.749303+0000 mgr.y (mgr.34107) 8503 : audit [DBG] from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.326 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[45495]: audit 2022-04-22T21:35:09.174257+0000 mon.a (mon.0) 12264 : audit [DBG] from='client.? 172.21.15.115:0/1337541541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:10.326 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[45495]: audit 2022-04-22T21:35:09.462132+0000 mon.a (mon.0) 12265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:10.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[46934]: audit 2022-04-22T21:35:08.408121+0000 mgr.y (mgr.34107) 8502 : audit [DBG] from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[46934]: audit 2022-04-22T21:35:08.749303+0000 mgr.y (mgr.34107) 8503 : audit [DBG] from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:10.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[46934]: audit 2022-04-22T21:35:09.174257+0000 mon.a (mon.0) 12264 : audit [DBG] from='client.? 172.21.15.115:0/1337541541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:10.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:09 smithi115 bash[46934]: audit 2022-04-22T21:35:09.462132+0000 mon.a (mon.0) 12265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:10 smithi121 bash[36902]: cluster 2022-04-22T21:35:09.728407+0000 mgr.y (mgr.34107) 8504 : cluster [DBG] pgmap v9413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:10 smithi121 bash[36902]: audit 2022-04-22T21:35:10.391968+0000 mon.a (mon.0) 12266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:10 smithi115 bash[45495]: cluster 2022-04-22T21:35:09.728407+0000 mgr.y (mgr.34107) 8504 : cluster [DBG] pgmap v9413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:10 smithi115 bash[45495]: audit 2022-04-22T21:35:10.391968+0000 mon.a (mon.0) 12266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:10 smithi115 bash[46934]: cluster 2022-04-22T21:35:09.728407+0000 mgr.y (mgr.34107) 8504 : cluster [DBG] pgmap v9413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:10 smithi115 bash[46934]: audit 2022-04-22T21:35:10.391968+0000 mon.a (mon.0) 12266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:11 smithi121 bash[36902]: audit 2022-04-22T21:35:11.448198+0000 mon.a (mon.0) 12267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:11 smithi115 bash[45495]: audit 2022-04-22T21:35:11.448198+0000 mon.a (mon.0) 12267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:11 smithi115 bash[46934]: audit 2022-04-22T21:35:11.448198+0000 mon.a (mon.0) 12267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:13.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:12 smithi115 bash[45495]: cluster 2022-04-22T21:35:11.729506+0000 mgr.y (mgr.34107) 8505 : cluster [DBG] pgmap v9414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:13.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:12.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:13.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:12 smithi115 bash[46934]: cluster 2022-04-22T21:35:11.729506+0000 mgr.y (mgr.34107) 8505 : cluster [DBG] pgmap v9414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:12 smithi121 bash[36902]: cluster 2022-04-22T21:35:11.729506+0000 mgr.y (mgr.34107) 8505 : cluster [DBG] pgmap v9414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:14.457Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:14 smithi121 bash[36902]: cluster 2022-04-22T21:35:13.730765+0000 mgr.y (mgr.34107) 8506 : cluster [DBG] pgmap v9415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:14 smithi121 bash[36902]: audit 2022-04-22T21:35:14.464955+0000 mon.a (mon.0) 12268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:14 smithi115 bash[45495]: cluster 2022-04-22T21:35:13.730765+0000 mgr.y (mgr.34107) 8506 : cluster [DBG] pgmap v9415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:14 smithi115 bash[45495]: audit 2022-04-22T21:35:14.464955+0000 mon.a (mon.0) 12268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:14 smithi115 bash[46934]: cluster 2022-04-22T21:35:13.730765+0000 mgr.y (mgr.34107) 8506 : cluster [DBG] pgmap v9415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:14 smithi115 bash[46934]: audit 2022-04-22T21:35:14.464955+0000 mon.a (mon.0) 12268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:15 smithi121 bash[36902]: audit 2022-04-22T21:35:15.395715+0000 mon.a (mon.0) 12269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:16.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:15 smithi115 bash[45495]: audit 2022-04-22T21:35:15.395715+0000 mon.a (mon.0) 12269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:16.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:15 smithi115 bash[46934]: audit 2022-04-22T21:35:15.395715+0000 mon.a (mon.0) 12269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:16 smithi121 bash[36902]: cluster 2022-04-22T21:35:15.731904+0000 mgr.y (mgr.34107) 8507 : cluster [DBG] pgmap v9416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:16 smithi121 bash[36902]: audit 2022-04-22T21:35:16.138120+0000 mon.a (mon.0) 12270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:35:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:16 smithi121 bash[36902]: audit 2022-04-22T21:35:16.138713+0000 mon.a (mon.0) 12271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:35:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:16 smithi121 bash[36902]: audit 2022-04-22T21:35:16.451644+0000 mon.a (mon.0) 12272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[46934]: cluster 2022-04-22T21:35:15.731904+0000 mgr.y (mgr.34107) 8507 : cluster [DBG] pgmap v9416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[46934]: audit 2022-04-22T21:35:16.138120+0000 mon.a (mon.0) 12270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[46934]: audit 2022-04-22T21:35:16.138713+0000 mon.a (mon.0) 12271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[46934]: audit 2022-04-22T21:35:16.451644+0000 mon.a (mon.0) 12272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[45495]: cluster 2022-04-22T21:35:15.731904+0000 mgr.y (mgr.34107) 8507 : cluster [DBG] pgmap v9416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[45495]: audit 2022-04-22T21:35:16.138120+0000 mon.a (mon.0) 12270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:35:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[45495]: audit 2022-04-22T21:35:16.138713+0000 mon.a (mon.0) 12271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:35:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:16 smithi115 bash[45495]: audit 2022-04-22T21:35:16.451644+0000 mon.a (mon.0) 12272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:18 smithi121 bash[36902]: cluster 2022-04-22T21:35:17.733181+0000 mgr.y (mgr.34107) 8508 : cluster [DBG] pgmap v9417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:19.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:18 smithi115 bash[46934]: cluster 2022-04-22T21:35:17.733181+0000 mgr.y (mgr.34107) 8508 : cluster [DBG] pgmap v9417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:18 smithi115 bash[45495]: cluster 2022-04-22T21:35:17.733181+0000 mgr.y (mgr.34107) 8508 : cluster [DBG] pgmap v9417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:19 smithi121 bash[36902]: audit 2022-04-22T21:35:19.467561+0000 mon.a (mon.0) 12273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:19 smithi115 bash[45495]: audit 2022-04-22T21:35:19.467561+0000 mon.a (mon.0) 12273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:20.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:19 smithi115 bash[46934]: audit 2022-04-22T21:35:19.467561+0000 mon.a (mon.0) 12273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:20 smithi121 bash[36902]: cluster 2022-04-22T21:35:19.734410+0000 mgr.y (mgr.34107) 8509 : cluster [DBG] pgmap v9418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:20 smithi121 bash[36902]: audit 2022-04-22T21:35:20.399469+0000 mon.a (mon.0) 12274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:20 smithi115 bash[45495]: cluster 2022-04-22T21:35:19.734410+0000 mgr.y (mgr.34107) 8509 : cluster [DBG] pgmap v9418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:20 smithi115 bash[45495]: audit 2022-04-22T21:35:20.399469+0000 mon.a (mon.0) 12274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:20 smithi115 bash[46934]: cluster 2022-04-22T21:35:19.734410+0000 mgr.y (mgr.34107) 8509 : cluster [DBG] pgmap v9418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:20 smithi115 bash[46934]: audit 2022-04-22T21:35:20.399469+0000 mon.a (mon.0) 12274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:21 smithi121 bash[36902]: audit 2022-04-22T21:35:21.454554+0000 mon.a (mon.0) 12275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:21 smithi115 bash[45495]: audit 2022-04-22T21:35:21.454554+0000 mon.a (mon.0) 12275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:21 smithi115 bash[46934]: audit 2022-04-22T21:35:21.454554+0000 mon.a (mon.0) 12275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:22.851 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:22.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:22 smithi121 bash[36902]: cluster 2022-04-22T21:35:21.735517+0000 mgr.y (mgr.34107) 8510 : cluster [DBG] pgmap v9419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:22 smithi115 bash[46934]: cluster 2022-04-22T21:35:21.735517+0000 mgr.y (mgr.34107) 8510 : cluster [DBG] pgmap v9419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:23.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:22 smithi115 bash[45495]: cluster 2022-04-22T21:35:21.735517+0000 mgr.y (mgr.34107) 8510 : cluster [DBG] pgmap v9419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:24.457Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:24 smithi121 bash[36902]: cluster 2022-04-22T21:35:23.736591+0000 mgr.y (mgr.34107) 8511 : cluster [DBG] pgmap v9420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:24 smithi121 bash[36902]: audit 2022-04-22T21:35:24.470411+0000 mon.a (mon.0) 12276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:24 smithi115 bash[46934]: cluster 2022-04-22T21:35:23.736591+0000 mgr.y (mgr.34107) 8511 : cluster [DBG] pgmap v9420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:24 smithi115 bash[46934]: audit 2022-04-22T21:35:24.470411+0000 mon.a (mon.0) 12276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:24 smithi115 bash[45495]: cluster 2022-04-22T21:35:23.736591+0000 mgr.y (mgr.34107) 8511 : cluster [DBG] pgmap v9420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:24 smithi115 bash[45495]: audit 2022-04-22T21:35:24.470411+0000 mon.a (mon.0) 12276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:25 smithi121 bash[36902]: audit 2022-04-22T21:35:25.403376+0000 mon.a (mon.0) 12277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:25 smithi115 bash[45495]: audit 2022-04-22T21:35:25.403376+0000 mon.a (mon.0) 12277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:25 smithi115 bash[46934]: audit 2022-04-22T21:35:25.403376+0000 mon.a (mon.0) 12277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:26 smithi121 bash[36902]: cluster 2022-04-22T21:35:25.737673+0000 mgr.y (mgr.34107) 8512 : cluster [DBG] pgmap v9421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:26 smithi121 bash[36902]: audit 2022-04-22T21:35:26.457723+0000 mon.a (mon.0) 12278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:26 smithi115 bash[45495]: cluster 2022-04-22T21:35:25.737673+0000 mgr.y (mgr.34107) 8512 : cluster [DBG] pgmap v9421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:26 smithi115 bash[45495]: audit 2022-04-22T21:35:26.457723+0000 mon.a (mon.0) 12278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:26 smithi115 bash[46934]: cluster 2022-04-22T21:35:25.737673+0000 mgr.y (mgr.34107) 8512 : cluster [DBG] pgmap v9421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:26 smithi115 bash[46934]: audit 2022-04-22T21:35:26.457723+0000 mon.a (mon.0) 12278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:28 smithi121 bash[36902]: cluster 2022-04-22T21:35:27.738935+0000 mgr.y (mgr.34107) 8513 : cluster [DBG] pgmap v9422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:28 smithi115 bash[45495]: cluster 2022-04-22T21:35:27.738935+0000 mgr.y (mgr.34107) 8513 : cluster [DBG] pgmap v9422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:28 smithi115 bash[46934]: cluster 2022-04-22T21:35:27.738935+0000 mgr.y (mgr.34107) 8513 : cluster [DBG] pgmap v9422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:29 smithi121 bash[36902]: audit 2022-04-22T21:35:29.473301+0000 mon.a (mon.0) 12279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:29 smithi115 bash[45495]: audit 2022-04-22T21:35:29.473301+0000 mon.a (mon.0) 12279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:29 smithi115 bash[46934]: audit 2022-04-22T21:35:29.473301+0000 mon.a (mon.0) 12279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:30 smithi121 bash[36902]: cluster 2022-04-22T21:35:29.740200+0000 mgr.y (mgr.34107) 8514 : cluster [DBG] pgmap v9423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:30 smithi121 bash[36902]: audit 2022-04-22T21:35:30.406879+0000 mon.a (mon.0) 12280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:30 smithi115 bash[45495]: cluster 2022-04-22T21:35:29.740200+0000 mgr.y (mgr.34107) 8514 : cluster [DBG] pgmap v9423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:30 smithi115 bash[45495]: audit 2022-04-22T21:35:30.406879+0000 mon.a (mon.0) 12280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:30 smithi115 bash[46934]: cluster 2022-04-22T21:35:29.740200+0000 mgr.y (mgr.34107) 8514 : cluster [DBG] pgmap v9423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:30 smithi115 bash[46934]: audit 2022-04-22T21:35:30.406879+0000 mon.a (mon.0) 12280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:31 smithi121 bash[36902]: audit 2022-04-22T21:35:31.460956+0000 mon.a (mon.0) 12281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:32.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:31 smithi115 bash[45495]: audit 2022-04-22T21:35:31.460956+0000 mon.a (mon.0) 12281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:31 smithi115 bash[46934]: audit 2022-04-22T21:35:31.460956+0000 mon.a (mon.0) 12281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:32.889 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:32.590Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:32 smithi121 bash[36902]: cluster 2022-04-22T21:35:31.741278+0000 mgr.y (mgr.34107) 8515 : cluster [DBG] pgmap v9424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:32 smithi115 bash[45495]: cluster 2022-04-22T21:35:31.741278+0000 mgr.y (mgr.34107) 8515 : cluster [DBG] pgmap v9424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:33.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:32 smithi115 bash[46934]: cluster 2022-04-22T21:35:31.741278+0000 mgr.y (mgr.34107) 8515 : cluster [DBG] pgmap v9424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:34.457Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:35.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:34 smithi121 bash[36902]: cluster 2022-04-22T21:35:33.742521+0000 mgr.y (mgr.34107) 8516 : cluster [DBG] pgmap v9425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:34 smithi121 bash[36902]: audit 2022-04-22T21:35:34.476071+0000 mon.a (mon.0) 12282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:34 smithi115 bash[45495]: cluster 2022-04-22T21:35:33.742521+0000 mgr.y (mgr.34107) 8516 : cluster [DBG] pgmap v9425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:34 smithi115 bash[45495]: audit 2022-04-22T21:35:34.476071+0000 mon.a (mon.0) 12282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:34 smithi115 bash[46934]: cluster 2022-04-22T21:35:33.742521+0000 mgr.y (mgr.34107) 8516 : cluster [DBG] pgmap v9425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:34 smithi115 bash[46934]: audit 2022-04-22T21:35:34.476071+0000 mon.a (mon.0) 12282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:35 smithi121 bash[36902]: audit 2022-04-22T21:35:35.410557+0000 mon.a (mon.0) 12283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:35 smithi115 bash[45495]: audit 2022-04-22T21:35:35.410557+0000 mon.a (mon.0) 12283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:35 smithi115 bash[46934]: audit 2022-04-22T21:35:35.410557+0000 mon.a (mon.0) 12283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:36 smithi121 bash[36902]: cluster 2022-04-22T21:35:35.743712+0000 mgr.y (mgr.34107) 8517 : cluster [DBG] pgmap v9426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:36 smithi121 bash[36902]: audit 2022-04-22T21:35:36.468570+0000 mon.a (mon.0) 12284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:36 smithi115 bash[45495]: cluster 2022-04-22T21:35:35.743712+0000 mgr.y (mgr.34107) 8517 : cluster [DBG] pgmap v9426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:36 smithi115 bash[45495]: audit 2022-04-22T21:35:36.468570+0000 mon.a (mon.0) 12284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:36 smithi115 bash[46934]: cluster 2022-04-22T21:35:35.743712+0000 mgr.y (mgr.34107) 8517 : cluster [DBG] pgmap v9426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:36 smithi115 bash[46934]: audit 2022-04-22T21:35:36.468570+0000 mon.a (mon.0) 12284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:35:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:38 smithi121 bash[36902]: cluster 2022-04-22T21:35:37.744872+0000 mgr.y (mgr.34107) 8518 : cluster [DBG] pgmap v9427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:38 smithi121 bash[36902]: audit 2022-04-22T21:35:38.255779+0000 mon.a (mon.0) 12285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:35:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:38 smithi121 bash[36902]: audit 2022-04-22T21:35:38.258517+0000 mon.a (mon.0) 12286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:35:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:38 smithi121 bash[36902]: audit 2022-04-22T21:35:38.260207+0000 mon.a (mon.0) 12287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:35:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:38 smithi121 bash[36902]: audit 2022-04-22T21:35:38.604985+0000 mon.a (mon.0) 12288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:35:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[45495]: cluster 2022-04-22T21:35:37.744872+0000 mgr.y (mgr.34107) 8518 : cluster [DBG] pgmap v9427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[45495]: audit 2022-04-22T21:35:38.255779+0000 mon.a (mon.0) 12285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:35:39.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[45495]: audit 2022-04-22T21:35:38.258517+0000 mon.a (mon.0) 12286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:35:39.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[45495]: audit 2022-04-22T21:35:38.260207+0000 mon.a (mon.0) 12287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:35:39.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[45495]: audit 2022-04-22T21:35:38.604985+0000 mon.a (mon.0) 12288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:35:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[46934]: cluster 2022-04-22T21:35:37.744872+0000 mgr.y (mgr.34107) 8518 : cluster [DBG] pgmap v9427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[46934]: audit 2022-04-22T21:35:38.255779+0000 mon.a (mon.0) 12285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:35:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[46934]: audit 2022-04-22T21:35:38.258517+0000 mon.a (mon.0) 12286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:35:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[46934]: audit 2022-04-22T21:35:38.260207+0000 mon.a (mon.0) 12287 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:35:39.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:38 smithi115 bash[46934]: audit 2022-04-22T21:35:38.604985+0000 mon.a (mon.0) 12288 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:35:39.550 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:35:39.903 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:35:39.903 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 61s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:35:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:35:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 61s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:35:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:35:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:35:39.904 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 61s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 61s ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 61s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3h) 61s ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (3h) 2m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:35:39.905 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 61s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 61s ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 61s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 61s ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:35:39.906 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:35:39.907 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:35:39.907 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:35:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:39 smithi121 bash[36902]: audit 2022-04-22T21:35:39.478884+0000 mon.a (mon.0) 12289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:39 smithi115 bash[45495]: audit 2022-04-22T21:35:39.478884+0000 mon.a (mon.0) 12289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:39 smithi115 bash[46934]: audit 2022-04-22T21:35:39.478884+0000 mon.a (mon.0) 12289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:40.321 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:35:40.321 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:35:40.322 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:35:40.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:40.322 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:35:40.322 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:35:40.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:40.323 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:35:40.323 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:35:40.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:40.324 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:35:40.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:35:40.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:35:40.324 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:35:40.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:35:40.327 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:35:40.327 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:35:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:40 smithi121 bash[36902]: audit 2022-04-22T21:35:39.530621+0000 mgr.y (mgr.34107) 8519 : audit [DBG] from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:40 smithi121 bash[36902]: cluster 2022-04-22T21:35:39.745914+0000 mgr.y (mgr.34107) 8520 : cluster [DBG] pgmap v9428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:40 smithi121 bash[36902]: audit 2022-04-22T21:35:39.879885+0000 mgr.y (mgr.34107) 8521 : audit [DBG] from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:40 smithi121 bash[36902]: audit 2022-04-22T21:35:40.317530+0000 mon.a (mon.0) 12290 : audit [DBG] from='client.? 172.21.15.115:0/1839622925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:40 smithi121 bash[36902]: audit 2022-04-22T21:35:40.414519+0000 mon.a (mon.0) 12291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[45495]: audit 2022-04-22T21:35:39.530621+0000 mgr.y (mgr.34107) 8519 : audit [DBG] from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[45495]: cluster 2022-04-22T21:35:39.745914+0000 mgr.y (mgr.34107) 8520 : cluster [DBG] pgmap v9428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[45495]: audit 2022-04-22T21:35:39.879885+0000 mgr.y (mgr.34107) 8521 : audit [DBG] from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[45495]: audit 2022-04-22T21:35:40.317530+0000 mon.a (mon.0) 12290 : audit [DBG] from='client.? 172.21.15.115:0/1839622925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:41.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[45495]: audit 2022-04-22T21:35:40.414519+0000 mon.a (mon.0) 12291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[46934]: audit 2022-04-22T21:35:39.530621+0000 mgr.y (mgr.34107) 8519 : audit [DBG] from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[46934]: cluster 2022-04-22T21:35:39.745914+0000 mgr.y (mgr.34107) 8520 : cluster [DBG] pgmap v9428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[46934]: audit 2022-04-22T21:35:39.879885+0000 mgr.y (mgr.34107) 8521 : audit [DBG] from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:35:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[46934]: audit 2022-04-22T21:35:40.317530+0000 mon.a (mon.0) 12290 : audit [DBG] from='client.? 172.21.15.115:0/1839622925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:35:41.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:40 smithi115 bash[46934]: audit 2022-04-22T21:35:40.414519+0000 mon.a (mon.0) 12291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:42.918 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:42.590Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:42 smithi121 bash[36902]: cluster 2022-04-22T21:35:41.747006+0000 mgr.y (mgr.34107) 8522 : cluster [DBG] pgmap v9429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:42 smithi115 bash[45495]: cluster 2022-04-22T21:35:41.747006+0000 mgr.y (mgr.34107) 8522 : cluster [DBG] pgmap v9429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:42 smithi115 bash[46934]: cluster 2022-04-22T21:35:41.747006+0000 mgr.y (mgr.34107) 8522 : cluster [DBG] pgmap v9429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:44.459Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:44 smithi121 bash[36902]: cluster 2022-04-22T21:35:43.748217+0000 mgr.y (mgr.34107) 8523 : cluster [DBG] pgmap v9430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:44 smithi121 bash[36902]: audit 2022-04-22T21:35:44.481543+0000 mon.a (mon.0) 12292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:44 smithi115 bash[45495]: cluster 2022-04-22T21:35:43.748217+0000 mgr.y (mgr.34107) 8523 : cluster [DBG] pgmap v9430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:44 smithi115 bash[45495]: audit 2022-04-22T21:35:44.481543+0000 mon.a (mon.0) 12292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:44 smithi115 bash[46934]: cluster 2022-04-22T21:35:43.748217+0000 mgr.y (mgr.34107) 8523 : cluster [DBG] pgmap v9430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:44 smithi115 bash[46934]: audit 2022-04-22T21:35:44.481543+0000 mon.a (mon.0) 12292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:46.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:45 smithi121 bash[36902]: audit 2022-04-22T21:35:45.418263+0000 mon.a (mon.0) 12293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:45 smithi115 bash[45495]: audit 2022-04-22T21:35:45.418263+0000 mon.a (mon.0) 12293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:45 smithi115 bash[46934]: audit 2022-04-22T21:35:45.418263+0000 mon.a (mon.0) 12293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:47.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:46 smithi121 bash[36902]: cluster 2022-04-22T21:35:45.749292+0000 mgr.y (mgr.34107) 8524 : cluster [DBG] pgmap v9431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:47.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:46 smithi115 bash[45495]: cluster 2022-04-22T21:35:45.749292+0000 mgr.y (mgr.34107) 8524 : cluster [DBG] pgmap v9431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:47.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:46 smithi115 bash[46934]: cluster 2022-04-22T21:35:45.749292+0000 mgr.y (mgr.34107) 8524 : cluster [DBG] pgmap v9431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:48 smithi121 bash[36902]: cluster 2022-04-22T21:35:47.750429+0000 mgr.y (mgr.34107) 8525 : cluster [DBG] pgmap v9432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:48 smithi115 bash[45495]: cluster 2022-04-22T21:35:47.750429+0000 mgr.y (mgr.34107) 8525 : cluster [DBG] pgmap v9432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:48 smithi115 bash[46934]: cluster 2022-04-22T21:35:47.750429+0000 mgr.y (mgr.34107) 8525 : cluster [DBG] pgmap v9432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:49 smithi121 bash[36902]: audit 2022-04-22T21:35:49.484602+0000 mon.a (mon.0) 12294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:49 smithi115 bash[45495]: audit 2022-04-22T21:35:49.484602+0000 mon.a (mon.0) 12294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:49 smithi115 bash[46934]: audit 2022-04-22T21:35:49.484602+0000 mon.a (mon.0) 12294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:50 smithi121 bash[36902]: cluster 2022-04-22T21:35:49.751660+0000 mgr.y (mgr.34107) 8526 : cluster [DBG] pgmap v9433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:50 smithi121 bash[36902]: audit 2022-04-22T21:35:50.421895+0000 mon.a (mon.0) 12295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:50 smithi115 bash[45495]: cluster 2022-04-22T21:35:49.751660+0000 mgr.y (mgr.34107) 8526 : cluster [DBG] pgmap v9433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:50 smithi115 bash[45495]: audit 2022-04-22T21:35:50.421895+0000 mon.a (mon.0) 12295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:50 smithi115 bash[46934]: cluster 2022-04-22T21:35:49.751660+0000 mgr.y (mgr.34107) 8526 : cluster [DBG] pgmap v9433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:51.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:50 smithi115 bash[46934]: audit 2022-04-22T21:35:50.421895+0000 mon.a (mon.0) 12295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:52.946 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:52.591Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:52 smithi121 bash[36902]: cluster 2022-04-22T21:35:51.752859+0000 mgr.y (mgr.34107) 8527 : cluster [DBG] pgmap v9434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:52 smithi115 bash[45495]: cluster 2022-04-22T21:35:51.752859+0000 mgr.y (mgr.34107) 8527 : cluster [DBG] pgmap v9434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:52 smithi115 bash[46934]: cluster 2022-04-22T21:35:51.752859+0000 mgr.y (mgr.34107) 8527 : cluster [DBG] pgmap v9434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:35:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:35:54.459Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:35:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:54 smithi121 bash[36902]: cluster 2022-04-22T21:35:53.754072+0000 mgr.y (mgr.34107) 8528 : cluster [DBG] pgmap v9435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:54 smithi121 bash[36902]: audit 2022-04-22T21:35:54.487664+0000 mon.a (mon.0) 12296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:54 smithi115 bash[45495]: cluster 2022-04-22T21:35:53.754072+0000 mgr.y (mgr.34107) 8528 : cluster [DBG] pgmap v9435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:54 smithi115 bash[45495]: audit 2022-04-22T21:35:54.487664+0000 mon.a (mon.0) 12296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:54 smithi115 bash[46934]: cluster 2022-04-22T21:35:53.754072+0000 mgr.y (mgr.34107) 8528 : cluster [DBG] pgmap v9435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:54 smithi115 bash[46934]: audit 2022-04-22T21:35:54.487664+0000 mon.a (mon.0) 12296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:35:56.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:55 smithi121 bash[36902]: audit 2022-04-22T21:35:55.425814+0000 mon.a (mon.0) 12297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:55 smithi115 bash[45495]: audit 2022-04-22T21:35:55.425814+0000 mon.a (mon.0) 12297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:55 smithi115 bash[46934]: audit 2022-04-22T21:35:55.425814+0000 mon.a (mon.0) 12297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:35:57.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:56 smithi121 bash[36902]: cluster 2022-04-22T21:35:55.755164+0000 mgr.y (mgr.34107) 8529 : cluster [DBG] pgmap v9436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:56 smithi115 bash[45495]: cluster 2022-04-22T21:35:55.755164+0000 mgr.y (mgr.34107) 8529 : cluster [DBG] pgmap v9436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:56 smithi115 bash[46934]: cluster 2022-04-22T21:35:55.755164+0000 mgr.y (mgr.34107) 8529 : cluster [DBG] pgmap v9436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:58 smithi121 bash[36902]: cluster 2022-04-22T21:35:57.756412+0000 mgr.y (mgr.34107) 8530 : cluster [DBG] pgmap v9437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:58 smithi115 bash[46934]: cluster 2022-04-22T21:35:57.756412+0000 mgr.y (mgr.34107) 8530 : cluster [DBG] pgmap v9437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:35:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:58 smithi115 bash[45495]: cluster 2022-04-22T21:35:57.756412+0000 mgr.y (mgr.34107) 8530 : cluster [DBG] pgmap v9437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:35:59 smithi121 bash[36902]: audit 2022-04-22T21:35:59.490723+0000 mon.a (mon.0) 12298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:35:59 smithi115 bash[46934]: audit 2022-04-22T21:35:59.490723+0000 mon.a (mon.0) 12298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:35:59 smithi115 bash[45495]: audit 2022-04-22T21:35:59.490723+0000 mon.a (mon.0) 12298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:00 smithi121 bash[36902]: cluster 2022-04-22T21:35:59.757693+0000 mgr.y (mgr.34107) 8531 : cluster [DBG] pgmap v9438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:00 smithi121 bash[36902]: audit 2022-04-22T21:36:00.429503+0000 mon.a (mon.0) 12299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:00 smithi115 bash[45495]: cluster 2022-04-22T21:35:59.757693+0000 mgr.y (mgr.34107) 8531 : cluster [DBG] pgmap v9438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:00 smithi115 bash[45495]: audit 2022-04-22T21:36:00.429503+0000 mon.a (mon.0) 12299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:00 smithi115 bash[46934]: cluster 2022-04-22T21:35:59.757693+0000 mgr.y (mgr.34107) 8531 : cluster [DBG] pgmap v9438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:00 smithi115 bash[46934]: audit 2022-04-22T21:36:00.429503+0000 mon.a (mon.0) 12299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:02.972 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:02.593Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:02 smithi121 bash[36902]: cluster 2022-04-22T21:36:01.758847+0000 mgr.y (mgr.34107) 8532 : cluster [DBG] pgmap v9439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:02 smithi115 bash[45495]: cluster 2022-04-22T21:36:01.758847+0000 mgr.y (mgr.34107) 8532 : cluster [DBG] pgmap v9439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:02 smithi115 bash[46934]: cluster 2022-04-22T21:36:01.758847+0000 mgr.y (mgr.34107) 8532 : cluster [DBG] pgmap v9439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:04.459Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:04 smithi121 bash[36902]: cluster 2022-04-22T21:36:03.759899+0000 mgr.y (mgr.34107) 8533 : cluster [DBG] pgmap v9440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:04 smithi121 bash[36902]: audit 2022-04-22T21:36:04.493801+0000 mon.a (mon.0) 12300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:04 smithi115 bash[45495]: cluster 2022-04-22T21:36:03.759899+0000 mgr.y (mgr.34107) 8533 : cluster [DBG] pgmap v9440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:04 smithi115 bash[45495]: audit 2022-04-22T21:36:04.493801+0000 mon.a (mon.0) 12300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:04 smithi115 bash[46934]: cluster 2022-04-22T21:36:03.759899+0000 mgr.y (mgr.34107) 8533 : cluster [DBG] pgmap v9440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:04 smithi115 bash[46934]: audit 2022-04-22T21:36:04.493801+0000 mon.a (mon.0) 12300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:06.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:05 smithi121 bash[36902]: audit 2022-04-22T21:36:05.433161+0000 mon.a (mon.0) 12301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:05 smithi115 bash[45495]: audit 2022-04-22T21:36:05.433161+0000 mon.a (mon.0) 12301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:05 smithi115 bash[46934]: audit 2022-04-22T21:36:05.433161+0000 mon.a (mon.0) 12301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:06 smithi121 bash[36902]: cluster 2022-04-22T21:36:05.760902+0000 mgr.y (mgr.34107) 8534 : cluster [DBG] pgmap v9441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:07.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:06 smithi115 bash[46934]: cluster 2022-04-22T21:36:05.760902+0000 mgr.y (mgr.34107) 8534 : cluster [DBG] pgmap v9441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:06 smithi115 bash[45495]: cluster 2022-04-22T21:36:05.760902+0000 mgr.y (mgr.34107) 8534 : cluster [DBG] pgmap v9441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:08 smithi121 bash[36902]: cluster 2022-04-22T21:36:07.761917+0000 mgr.y (mgr.34107) 8535 : cluster [DBG] pgmap v9442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:08 smithi115 bash[45495]: cluster 2022-04-22T21:36:07.761917+0000 mgr.y (mgr.34107) 8535 : cluster [DBG] pgmap v9442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:08 smithi115 bash[46934]: cluster 2022-04-22T21:36:07.761917+0000 mgr.y (mgr.34107) 8535 : cluster [DBG] pgmap v9442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:09 smithi121 bash[36902]: audit 2022-04-22T21:36:09.496914+0000 mon.a (mon.0) 12302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:09 smithi115 bash[45495]: audit 2022-04-22T21:36:09.496914+0000 mon.a (mon.0) 12302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:10.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:09 smithi115 bash[46934]: audit 2022-04-22T21:36:09.496914+0000 mon.a (mon.0) 12302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:10.719 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:36:11.073 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 93s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 93s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:36:11.074 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 93s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 93s ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 93s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 93s ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 93s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 93s ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:36:11.075 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 93s ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 93s ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:36:11.076 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (3h) 2m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:36:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:10 smithi121 bash[36902]: cluster 2022-04-22T21:36:09.762975+0000 mgr.y (mgr.34107) 8536 : cluster [DBG] pgmap v9443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:10 smithi121 bash[36902]: audit 2022-04-22T21:36:10.436998+0000 mon.a (mon.0) 12303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:10 smithi115 bash[46934]: cluster 2022-04-22T21:36:09.762975+0000 mgr.y (mgr.34107) 8536 : cluster [DBG] pgmap v9443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:11.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:10 smithi115 bash[46934]: audit 2022-04-22T21:36:10.436998+0000 mon.a (mon.0) 12303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:11.323 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:10 smithi115 bash[45495]: cluster 2022-04-22T21:36:09.762975+0000 mgr.y (mgr.34107) 8536 : cluster [DBG] pgmap v9443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:11.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:10 smithi115 bash[45495]: audit 2022-04-22T21:36:10.436998+0000 mon.a (mon.0) 12303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:11.474 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:36:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:36:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:36:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:36:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:36:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:36:11.479 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:12 smithi121 bash[36902]: audit 2022-04-22T21:36:10.700157+0000 mgr.y (mgr.34107) 8537 : audit [DBG] from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:12 smithi121 bash[36902]: audit 2022-04-22T21:36:11.470604+0000 mon.c (mon.1) 303 : audit [DBG] from='client.? 172.21.15.115:0/3197315695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:12 smithi115 bash[45495]: audit 2022-04-22T21:36:10.700157+0000 mgr.y (mgr.34107) 8537 : audit [DBG] from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:12 smithi115 bash[45495]: audit 2022-04-22T21:36:11.470604+0000 mon.c (mon.1) 303 : audit [DBG] from='client.? 172.21.15.115:0/3197315695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:12 smithi115 bash[46934]: audit 2022-04-22T21:36:10.700157+0000 mgr.y (mgr.34107) 8537 : audit [DBG] from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:12 smithi115 bash[46934]: audit 2022-04-22T21:36:11.470604+0000 mon.c (mon.1) 303 : audit [DBG] from='client.? 172.21.15.115:0/3197315695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:13.008 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:12.593Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:13 smithi121 bash[36902]: audit 2022-04-22T21:36:11.047029+0000 mgr.y (mgr.34107) 8538 : audit [DBG] from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:13 smithi121 bash[36902]: cluster 2022-04-22T21:36:11.764115+0000 mgr.y (mgr.34107) 8539 : cluster [DBG] pgmap v9444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:13 smithi115 bash[45495]: audit 2022-04-22T21:36:11.047029+0000 mgr.y (mgr.34107) 8538 : audit [DBG] from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:13 smithi115 bash[45495]: cluster 2022-04-22T21:36:11.764115+0000 mgr.y (mgr.34107) 8539 : cluster [DBG] pgmap v9444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:13 smithi115 bash[46934]: audit 2022-04-22T21:36:11.047029+0000 mgr.y (mgr.34107) 8538 : audit [DBG] from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:13 smithi115 bash[46934]: cluster 2022-04-22T21:36:11.764115+0000 mgr.y (mgr.34107) 8539 : cluster [DBG] pgmap v9444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:14.460Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:15 smithi121 bash[36902]: cluster 2022-04-22T21:36:13.765329+0000 mgr.y (mgr.34107) 8540 : cluster [DBG] pgmap v9445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:15 smithi121 bash[36902]: audit 2022-04-22T21:36:14.500097+0000 mon.a (mon.0) 12304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:15 smithi115 bash[45495]: cluster 2022-04-22T21:36:13.765329+0000 mgr.y (mgr.34107) 8540 : cluster [DBG] pgmap v9445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:15 smithi115 bash[45495]: audit 2022-04-22T21:36:14.500097+0000 mon.a (mon.0) 12304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:15 smithi115 bash[46934]: cluster 2022-04-22T21:36:13.765329+0000 mgr.y (mgr.34107) 8540 : cluster [DBG] pgmap v9445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:15 smithi115 bash[46934]: audit 2022-04-22T21:36:14.500097+0000 mon.a (mon.0) 12304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:16.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:16 smithi115 bash[45495]: audit 2022-04-22T21:36:15.440512+0000 mon.a (mon.0) 12305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:16.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:16 smithi115 bash[46934]: audit 2022-04-22T21:36:15.440512+0000 mon.a (mon.0) 12305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:16.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:16 smithi121 bash[36902]: audit 2022-04-22T21:36:15.440512+0000 mon.a (mon.0) 12305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:17.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[45495]: cluster 2022-04-22T21:36:15.766208+0000 mgr.y (mgr.34107) 8541 : cluster [DBG] pgmap v9446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:17.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[45495]: audit 2022-04-22T21:36:16.151739+0000 mon.a (mon.0) 12306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:36:17.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[45495]: audit 2022-04-22T21:36:16.152219+0000 mon.a (mon.0) 12307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:36:17.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[46934]: cluster 2022-04-22T21:36:15.766208+0000 mgr.y (mgr.34107) 8541 : cluster [DBG] pgmap v9446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:17.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[46934]: audit 2022-04-22T21:36:16.151739+0000 mon.a (mon.0) 12306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:36:17.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:17 smithi115 bash[46934]: audit 2022-04-22T21:36:16.152219+0000 mon.a (mon.0) 12307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:36:17.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:17 smithi121 bash[36902]: cluster 2022-04-22T21:36:15.766208+0000 mgr.y (mgr.34107) 8541 : cluster [DBG] pgmap v9446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:17.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:17 smithi121 bash[36902]: audit 2022-04-22T21:36:16.151739+0000 mon.a (mon.0) 12306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:36:17.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:17 smithi121 bash[36902]: audit 2022-04-22T21:36:16.152219+0000 mon.a (mon.0) 12307 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:36:19.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:19 smithi115 bash[46934]: cluster 2022-04-22T21:36:17.767303+0000 mgr.y (mgr.34107) 8542 : cluster [DBG] pgmap v9447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:19.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:19 smithi115 bash[45495]: cluster 2022-04-22T21:36:17.767303+0000 mgr.y (mgr.34107) 8542 : cluster [DBG] pgmap v9447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:19 smithi121 bash[36902]: cluster 2022-04-22T21:36:17.767303+0000 mgr.y (mgr.34107) 8542 : cluster [DBG] pgmap v9447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:20 smithi115 bash[46934]: audit 2022-04-22T21:36:19.503238+0000 mon.a (mon.0) 12308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:20.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:20 smithi115 bash[45495]: audit 2022-04-22T21:36:19.503238+0000 mon.a (mon.0) 12308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:20 smithi121 bash[36902]: audit 2022-04-22T21:36:19.503238+0000 mon.a (mon.0) 12308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:21 smithi115 bash[45495]: cluster 2022-04-22T21:36:19.768397+0000 mgr.y (mgr.34107) 8543 : cluster [DBG] pgmap v9448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:21 smithi115 bash[45495]: audit 2022-04-22T21:36:20.444146+0000 mon.a (mon.0) 12309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:21 smithi115 bash[46934]: cluster 2022-04-22T21:36:19.768397+0000 mgr.y (mgr.34107) 8543 : cluster [DBG] pgmap v9448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:21 smithi115 bash[46934]: audit 2022-04-22T21:36:20.444146+0000 mon.a (mon.0) 12309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:21 smithi121 bash[36902]: cluster 2022-04-22T21:36:19.768397+0000 mgr.y (mgr.34107) 8543 : cluster [DBG] pgmap v9448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:21 smithi121 bash[36902]: audit 2022-04-22T21:36:20.444146+0000 mon.a (mon.0) 12309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:23.035 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:22.596Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:23 smithi115 bash[45495]: cluster 2022-04-22T21:36:21.769483+0000 mgr.y (mgr.34107) 8544 : cluster [DBG] pgmap v9449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:23.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:23 smithi115 bash[46934]: cluster 2022-04-22T21:36:21.769483+0000 mgr.y (mgr.34107) 8544 : cluster [DBG] pgmap v9449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:23.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:23 smithi121 bash[36902]: cluster 2022-04-22T21:36:21.769483+0000 mgr.y (mgr.34107) 8544 : cluster [DBG] pgmap v9449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:24.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:24 smithi115 bash[45495]: cluster 2022-04-22T21:36:23.770499+0000 mgr.y (mgr.34107) 8545 : cluster [DBG] pgmap v9450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:24.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:24 smithi115 bash[46934]: cluster 2022-04-22T21:36:23.770499+0000 mgr.y (mgr.34107) 8545 : cluster [DBG] pgmap v9450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:24 smithi121 bash[36902]: cluster 2022-04-22T21:36:23.770499+0000 mgr.y (mgr.34107) 8545 : cluster [DBG] pgmap v9450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:24.460Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:25 smithi121 bash[36902]: audit 2022-04-22T21:36:24.506302+0000 mon.a (mon.0) 12310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:25 smithi115 bash[45495]: audit 2022-04-22T21:36:24.506302+0000 mon.a (mon.0) 12310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:25 smithi115 bash[46934]: audit 2022-04-22T21:36:24.506302+0000 mon.a (mon.0) 12310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:26.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:26 smithi121 bash[36902]: audit 2022-04-22T21:36:25.448080+0000 mon.a (mon.0) 12311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:26 smithi121 bash[36902]: cluster 2022-04-22T21:36:25.771627+0000 mgr.y (mgr.34107) 8546 : cluster [DBG] pgmap v9451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:26 smithi115 bash[45495]: audit 2022-04-22T21:36:25.448080+0000 mon.a (mon.0) 12311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:26 smithi115 bash[45495]: cluster 2022-04-22T21:36:25.771627+0000 mgr.y (mgr.34107) 8546 : cluster [DBG] pgmap v9451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:26 smithi115 bash[46934]: audit 2022-04-22T21:36:25.448080+0000 mon.a (mon.0) 12311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:26 smithi115 bash[46934]: cluster 2022-04-22T21:36:25.771627+0000 mgr.y (mgr.34107) 8546 : cluster [DBG] pgmap v9451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:28 smithi121 bash[36902]: cluster 2022-04-22T21:36:27.772670+0000 mgr.y (mgr.34107) 8547 : cluster [DBG] pgmap v9452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:28 smithi115 bash[45495]: cluster 2022-04-22T21:36:27.772670+0000 mgr.y (mgr.34107) 8547 : cluster [DBG] pgmap v9452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:29.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:28 smithi115 bash[46934]: cluster 2022-04-22T21:36:27.772670+0000 mgr.y (mgr.34107) 8547 : cluster [DBG] pgmap v9452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:29 smithi121 bash[36902]: audit 2022-04-22T21:36:29.509509+0000 mon.a (mon.0) 12312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:29 smithi115 bash[45495]: audit 2022-04-22T21:36:29.509509+0000 mon.a (mon.0) 12312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:30.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:29 smithi115 bash[46934]: audit 2022-04-22T21:36:29.509509+0000 mon.a (mon.0) 12312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:30 smithi121 bash[36902]: cluster 2022-04-22T21:36:29.773835+0000 mgr.y (mgr.34107) 8548 : cluster [DBG] pgmap v9453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:30 smithi121 bash[36902]: audit 2022-04-22T21:36:30.451868+0000 mon.a (mon.0) 12313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:30 smithi115 bash[45495]: cluster 2022-04-22T21:36:29.773835+0000 mgr.y (mgr.34107) 8548 : cluster [DBG] pgmap v9453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:31.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:30 smithi115 bash[45495]: audit 2022-04-22T21:36:30.451868+0000 mon.a (mon.0) 12313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:31.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:30 smithi115 bash[46934]: cluster 2022-04-22T21:36:29.773835+0000 mgr.y (mgr.34107) 8548 : cluster [DBG] pgmap v9453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:31.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:30 smithi115 bash[46934]: audit 2022-04-22T21:36:30.451868+0000 mon.a (mon.0) 12313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:33.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:32.595Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:33.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:32 smithi115 bash[46934]: cluster 2022-04-22T21:36:31.775009+0000 mgr.y (mgr.34107) 8549 : cluster [DBG] pgmap v9454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:33.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:32 smithi115 bash[45495]: cluster 2022-04-22T21:36:31.775009+0000 mgr.y (mgr.34107) 8549 : cluster [DBG] pgmap v9454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:32 smithi121 bash[36902]: cluster 2022-04-22T21:36:31.775009+0000 mgr.y (mgr.34107) 8549 : cluster [DBG] pgmap v9454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:34.461Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:34 smithi121 bash[36902]: cluster 2022-04-22T21:36:33.776174+0000 mgr.y (mgr.34107) 8550 : cluster [DBG] pgmap v9455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:34 smithi121 bash[36902]: audit 2022-04-22T21:36:34.512397+0000 mon.a (mon.0) 12314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:34 smithi115 bash[46934]: cluster 2022-04-22T21:36:33.776174+0000 mgr.y (mgr.34107) 8550 : cluster [DBG] pgmap v9455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:34 smithi115 bash[46934]: audit 2022-04-22T21:36:34.512397+0000 mon.a (mon.0) 12314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:34 smithi115 bash[45495]: cluster 2022-04-22T21:36:33.776174+0000 mgr.y (mgr.34107) 8550 : cluster [DBG] pgmap v9455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:34 smithi115 bash[45495]: audit 2022-04-22T21:36:34.512397+0000 mon.a (mon.0) 12314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:36.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:35 smithi121 bash[36902]: audit 2022-04-22T21:36:35.455654+0000 mon.a (mon.0) 12315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:35 smithi115 bash[45495]: audit 2022-04-22T21:36:35.455654+0000 mon.a (mon.0) 12315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:35 smithi115 bash[46934]: audit 2022-04-22T21:36:35.455654+0000 mon.a (mon.0) 12315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:36 smithi121 bash[36902]: cluster 2022-04-22T21:36:35.777282+0000 mgr.y (mgr.34107) 8551 : cluster [DBG] pgmap v9456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:36 smithi115 bash[45495]: cluster 2022-04-22T21:36:35.777282+0000 mgr.y (mgr.34107) 8551 : cluster [DBG] pgmap v9456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:36 smithi115 bash[46934]: cluster 2022-04-22T21:36:35.777282+0000 mgr.y (mgr.34107) 8551 : cluster [DBG] pgmap v9456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:38.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[45495]: cluster 2022-04-22T21:36:37.778403+0000 mgr.y (mgr.34107) 8552 : cluster [DBG] pgmap v9457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[45495]: audit 2022-04-22T21:36:38.611996+0000 mon.a (mon.0) 12316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[45495]: audit 2022-04-22T21:36:38.615004+0000 mon.a (mon.0) 12317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[45495]: audit 2022-04-22T21:36:38.616860+0000 mon.a (mon.0) 12318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[46934]: cluster 2022-04-22T21:36:37.778403+0000 mgr.y (mgr.34107) 8552 : cluster [DBG] pgmap v9457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[46934]: audit 2022-04-22T21:36:38.611996+0000 mon.a (mon.0) 12316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:36:38.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[46934]: audit 2022-04-22T21:36:38.615004+0000 mon.a (mon.0) 12317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:36:38.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:38 smithi115 bash[46934]: audit 2022-04-22T21:36:38.616860+0000 mon.a (mon.0) 12318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:36:38.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:38 smithi121 bash[36902]: cluster 2022-04-22T21:36:37.778403+0000 mgr.y (mgr.34107) 8552 : cluster [DBG] pgmap v9457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:38.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:38 smithi121 bash[36902]: audit 2022-04-22T21:36:38.611996+0000 mon.a (mon.0) 12316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:36:38.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:38 smithi121 bash[36902]: audit 2022-04-22T21:36:38.615004+0000 mon.a (mon.0) 12317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:36:38.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:38 smithi121 bash[36902]: audit 2022-04-22T21:36:38.616860+0000 mon.a (mon.0) 12318 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:36:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:39 smithi121 bash[36902]: audit 2022-04-22T21:36:38.982683+0000 mon.a (mon.0) 12319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:36:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:39 smithi121 bash[36902]: audit 2022-04-22T21:36:39.515579+0000 mon.a (mon.0) 12320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:39 smithi115 bash[46934]: audit 2022-04-22T21:36:38.982683+0000 mon.a (mon.0) 12319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:36:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:39 smithi115 bash[46934]: audit 2022-04-22T21:36:39.515579+0000 mon.a (mon.0) 12320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:39 smithi115 bash[45495]: audit 2022-04-22T21:36:38.982683+0000 mon.a (mon.0) 12319 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:36:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:39 smithi115 bash[45495]: audit 2022-04-22T21:36:39.515579+0000 mon.a (mon.0) 12320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:40 smithi121 bash[36902]: cluster 2022-04-22T21:36:39.779501+0000 mgr.y (mgr.34107) 8553 : cluster [DBG] pgmap v9458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:40 smithi121 bash[36902]: audit 2022-04-22T21:36:40.459329+0000 mon.a (mon.0) 12321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:40 smithi115 bash[45495]: cluster 2022-04-22T21:36:39.779501+0000 mgr.y (mgr.34107) 8553 : cluster [DBG] pgmap v9458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:40 smithi115 bash[45495]: audit 2022-04-22T21:36:40.459329+0000 mon.a (mon.0) 12321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:40 smithi115 bash[46934]: cluster 2022-04-22T21:36:39.779501+0000 mgr.y (mgr.34107) 8553 : cluster [DBG] pgmap v9458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:40 smithi115 bash[46934]: audit 2022-04-22T21:36:40.459329+0000 mon.a (mon.0) 12321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:41.849 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:36:42.196 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:36:42.197 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:36:42.197 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:36:42.197 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:36:42.198 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:36:42.198 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:36:42.198 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:36:42.199 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:36:42.199 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:36:42.199 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:36:42.199 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:36:42.199 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:36:42.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:36:42.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:36:42.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:36:42.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:36:42.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:36:42.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:36:42.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:36:42.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:36:42.201 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:36:42.599 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:36:42.600 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:36:42.601 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:36:42.996 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:42.595Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:42 smithi121 bash[36902]: cluster 2022-04-22T21:36:41.780542+0000 mgr.y (mgr.34107) 8554 : cluster [DBG] pgmap v9459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:42 smithi121 bash[36902]: audit 2022-04-22T21:36:41.830991+0000 mgr.y (mgr.34107) 8555 : audit [DBG] from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:42 smithi121 bash[36902]: audit 2022-04-22T21:36:42.591684+0000 mon.c (mon.1) 304 : audit [DBG] from='client.? 172.21.15.115:0/904315121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[45495]: cluster 2022-04-22T21:36:41.780542+0000 mgr.y (mgr.34107) 8554 : cluster [DBG] pgmap v9459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[45495]: audit 2022-04-22T21:36:41.830991+0000 mgr.y (mgr.34107) 8555 : audit [DBG] from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[45495]: audit 2022-04-22T21:36:42.591684+0000 mon.c (mon.1) 304 : audit [DBG] from='client.? 172.21.15.115:0/904315121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[46934]: cluster 2022-04-22T21:36:41.780542+0000 mgr.y (mgr.34107) 8554 : cluster [DBG] pgmap v9459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[46934]: audit 2022-04-22T21:36:41.830991+0000 mgr.y (mgr.34107) 8555 : audit [DBG] from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:42 smithi115 bash[46934]: audit 2022-04-22T21:36:42.591684+0000 mon.c (mon.1) 304 : audit [DBG] from='client.? 172.21.15.115:0/904315121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:36:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:44 smithi121 bash[36902]: audit 2022-04-22T21:36:42.173120+0000 mgr.y (mgr.34107) 8556 : audit [DBG] from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:44 smithi115 bash[45495]: audit 2022-04-22T21:36:42.173120+0000 mgr.y (mgr.34107) 8556 : audit [DBG] from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:44.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:44 smithi115 bash[46934]: audit 2022-04-22T21:36:42.173120+0000 mgr.y (mgr.34107) 8556 : audit [DBG] from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:36:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:44.462Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:45 smithi121 bash[36902]: cluster 2022-04-22T21:36:43.781692+0000 mgr.y (mgr.34107) 8557 : cluster [DBG] pgmap v9460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:45 smithi121 bash[36902]: audit 2022-04-22T21:36:44.518779+0000 mon.a (mon.0) 12322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:45.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:45 smithi115 bash[45495]: cluster 2022-04-22T21:36:43.781692+0000 mgr.y (mgr.34107) 8557 : cluster [DBG] pgmap v9460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:45.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:45 smithi115 bash[45495]: audit 2022-04-22T21:36:44.518779+0000 mon.a (mon.0) 12322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:45.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:45 smithi115 bash[46934]: cluster 2022-04-22T21:36:43.781692+0000 mgr.y (mgr.34107) 8557 : cluster [DBG] pgmap v9460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:45.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:45 smithi115 bash[46934]: audit 2022-04-22T21:36:44.518779+0000 mon.a (mon.0) 12322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:46 smithi121 bash[36902]: audit 2022-04-22T21:36:45.462713+0000 mon.a (mon.0) 12323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:46 smithi115 bash[45495]: audit 2022-04-22T21:36:45.462713+0000 mon.a (mon.0) 12323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:46 smithi115 bash[46934]: audit 2022-04-22T21:36:45.462713+0000 mon.a (mon.0) 12323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:47 smithi121 bash[36902]: cluster 2022-04-22T21:36:45.782776+0000 mgr.y (mgr.34107) 8558 : cluster [DBG] pgmap v9461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:47 smithi115 bash[45495]: cluster 2022-04-22T21:36:45.782776+0000 mgr.y (mgr.34107) 8558 : cluster [DBG] pgmap v9461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:47 smithi115 bash[46934]: cluster 2022-04-22T21:36:45.782776+0000 mgr.y (mgr.34107) 8558 : cluster [DBG] pgmap v9461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:49 smithi115 bash[45495]: cluster 2022-04-22T21:36:47.783837+0000 mgr.y (mgr.34107) 8559 : cluster [DBG] pgmap v9462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:49 smithi115 bash[46934]: cluster 2022-04-22T21:36:47.783837+0000 mgr.y (mgr.34107) 8559 : cluster [DBG] pgmap v9462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:49 smithi121 bash[36902]: cluster 2022-04-22T21:36:47.783837+0000 mgr.y (mgr.34107) 8559 : cluster [DBG] pgmap v9462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:50 smithi115 bash[45495]: audit 2022-04-22T21:36:49.521898+0000 mon.a (mon.0) 12324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:50 smithi115 bash[46934]: audit 2022-04-22T21:36:49.521898+0000 mon.a (mon.0) 12324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:50 smithi121 bash[36902]: audit 2022-04-22T21:36:49.521898+0000 mon.a (mon.0) 12324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:51 smithi115 bash[45495]: cluster 2022-04-22T21:36:49.784907+0000 mgr.y (mgr.34107) 8560 : cluster [DBG] pgmap v9463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:51 smithi115 bash[45495]: audit 2022-04-22T21:36:50.466486+0000 mon.a (mon.0) 12325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:51 smithi115 bash[46934]: cluster 2022-04-22T21:36:49.784907+0000 mgr.y (mgr.34107) 8560 : cluster [DBG] pgmap v9463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:51 smithi115 bash[46934]: audit 2022-04-22T21:36:50.466486+0000 mon.a (mon.0) 12325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:51 smithi121 bash[36902]: cluster 2022-04-22T21:36:49.784907+0000 mgr.y (mgr.34107) 8560 : cluster [DBG] pgmap v9463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:51 smithi121 bash[36902]: audit 2022-04-22T21:36:50.466486+0000 mon.a (mon.0) 12325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:53.031 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:52.596Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:53 smithi115 bash[45495]: cluster 2022-04-22T21:36:51.785984+0000 mgr.y (mgr.34107) 8561 : cluster [DBG] pgmap v9464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:53.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:53 smithi115 bash[46934]: cluster 2022-04-22T21:36:51.785984+0000 mgr.y (mgr.34107) 8561 : cluster [DBG] pgmap v9464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:53.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:53 smithi121 bash[36902]: cluster 2022-04-22T21:36:51.785984+0000 mgr.y (mgr.34107) 8561 : cluster [DBG] pgmap v9464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:36:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:36:54.462Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:36:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:55 smithi115 bash[46934]: cluster 2022-04-22T21:36:53.787120+0000 mgr.y (mgr.34107) 8562 : cluster [DBG] pgmap v9465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:55 smithi115 bash[46934]: audit 2022-04-22T21:36:54.524922+0000 mon.a (mon.0) 12326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:55 smithi115 bash[45495]: cluster 2022-04-22T21:36:53.787120+0000 mgr.y (mgr.34107) 8562 : cluster [DBG] pgmap v9465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:55 smithi115 bash[45495]: audit 2022-04-22T21:36:54.524922+0000 mon.a (mon.0) 12326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:55 smithi121 bash[36902]: cluster 2022-04-22T21:36:53.787120+0000 mgr.y (mgr.34107) 8562 : cluster [DBG] pgmap v9465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:55 smithi121 bash[36902]: audit 2022-04-22T21:36:54.524922+0000 mon.a (mon.0) 12326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:36:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:56 smithi115 bash[45495]: audit 2022-04-22T21:36:55.470406+0000 mon.a (mon.0) 12327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:56 smithi115 bash[46934]: audit 2022-04-22T21:36:55.470406+0000 mon.a (mon.0) 12327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:56 smithi121 bash[36902]: audit 2022-04-22T21:36:55.470406+0000 mon.a (mon.0) 12327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:36:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:57 smithi115 bash[45495]: cluster 2022-04-22T21:36:55.787991+0000 mgr.y (mgr.34107) 8563 : cluster [DBG] pgmap v9466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:57 smithi115 bash[46934]: cluster 2022-04-22T21:36:55.787991+0000 mgr.y (mgr.34107) 8563 : cluster [DBG] pgmap v9466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:57.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:57 smithi121 bash[36902]: cluster 2022-04-22T21:36:55.787991+0000 mgr.y (mgr.34107) 8563 : cluster [DBG] pgmap v9466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:58 smithi121 bash[36902]: cluster 2022-04-22T21:36:57.789066+0000 mgr.y (mgr.34107) 8564 : cluster [DBG] pgmap v9467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:58 smithi115 bash[45495]: cluster 2022-04-22T21:36:57.789066+0000 mgr.y (mgr.34107) 8564 : cluster [DBG] pgmap v9467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:36:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:58 smithi115 bash[46934]: cluster 2022-04-22T21:36:57.789066+0000 mgr.y (mgr.34107) 8564 : cluster [DBG] pgmap v9467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:36:59 smithi121 bash[36902]: audit 2022-04-22T21:36:59.528116+0000 mon.a (mon.0) 12328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:00.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:36:59 smithi115 bash[45495]: audit 2022-04-22T21:36:59.528116+0000 mon.a (mon.0) 12328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:36:59 smithi115 bash[46934]: audit 2022-04-22T21:36:59.528116+0000 mon.a (mon.0) 12328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:00 smithi121 bash[36902]: cluster 2022-04-22T21:36:59.790140+0000 mgr.y (mgr.34107) 8565 : cluster [DBG] pgmap v9468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:00 smithi121 bash[36902]: audit 2022-04-22T21:37:00.474035+0000 mon.a (mon.0) 12329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:00 smithi115 bash[45495]: cluster 2022-04-22T21:36:59.790140+0000 mgr.y (mgr.34107) 8565 : cluster [DBG] pgmap v9468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:00 smithi115 bash[45495]: audit 2022-04-22T21:37:00.474035+0000 mon.a (mon.0) 12329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:00 smithi115 bash[46934]: cluster 2022-04-22T21:36:59.790140+0000 mgr.y (mgr.34107) 8565 : cluster [DBG] pgmap v9468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:00 smithi115 bash[46934]: audit 2022-04-22T21:37:00.474035+0000 mon.a (mon.0) 12329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:02.853 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:02.597Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:02.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:02 smithi115 bash[45495]: cluster 2022-04-22T21:37:01.791213+0000 mgr.y (mgr.34107) 8566 : cluster [DBG] pgmap v9469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:02 smithi121 bash[36902]: cluster 2022-04-22T21:37:01.791213+0000 mgr.y (mgr.34107) 8566 : cluster [DBG] pgmap v9469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:02 smithi115 bash[46934]: cluster 2022-04-22T21:37:01.791213+0000 mgr.y (mgr.34107) 8566 : cluster [DBG] pgmap v9469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:04.463Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:04 smithi121 bash[36902]: cluster 2022-04-22T21:37:03.792369+0000 mgr.y (mgr.34107) 8567 : cluster [DBG] pgmap v9470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:04 smithi121 bash[36902]: audit 2022-04-22T21:37:04.531250+0000 mon.a (mon.0) 12330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:04 smithi115 bash[45495]: cluster 2022-04-22T21:37:03.792369+0000 mgr.y (mgr.34107) 8567 : cluster [DBG] pgmap v9470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:04 smithi115 bash[45495]: audit 2022-04-22T21:37:04.531250+0000 mon.a (mon.0) 12330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:04 smithi115 bash[46934]: cluster 2022-04-22T21:37:03.792369+0000 mgr.y (mgr.34107) 8567 : cluster [DBG] pgmap v9470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:04 smithi115 bash[46934]: audit 2022-04-22T21:37:04.531250+0000 mon.a (mon.0) 12330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:05 smithi121 bash[36902]: audit 2022-04-22T21:37:05.477939+0000 mon.a (mon.0) 12331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:06.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:05 smithi115 bash[45495]: audit 2022-04-22T21:37:05.477939+0000 mon.a (mon.0) 12331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:05 smithi115 bash[46934]: audit 2022-04-22T21:37:05.477939+0000 mon.a (mon.0) 12331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:07.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:06 smithi121 bash[36902]: cluster 2022-04-22T21:37:05.793490+0000 mgr.y (mgr.34107) 8568 : cluster [DBG] pgmap v9471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:06 smithi115 bash[45495]: cluster 2022-04-22T21:37:05.793490+0000 mgr.y (mgr.34107) 8568 : cluster [DBG] pgmap v9471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:06 smithi115 bash[46934]: cluster 2022-04-22T21:37:05.793490+0000 mgr.y (mgr.34107) 8568 : cluster [DBG] pgmap v9471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:08 smithi121 bash[36902]: cluster 2022-04-22T21:37:07.794604+0000 mgr.y (mgr.34107) 8569 : cluster [DBG] pgmap v9472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:08 smithi115 bash[45495]: cluster 2022-04-22T21:37:07.794604+0000 mgr.y (mgr.34107) 8569 : cluster [DBG] pgmap v9472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:08 smithi115 bash[46934]: cluster 2022-04-22T21:37:07.794604+0000 mgr.y (mgr.34107) 8569 : cluster [DBG] pgmap v9472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:09 smithi121 bash[36902]: audit 2022-04-22T21:37:09.534573+0000 mon.a (mon.0) 12332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:09 smithi115 bash[45495]: audit 2022-04-22T21:37:09.534573+0000 mon.a (mon.0) 12332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:09 smithi115 bash[46934]: audit 2022-04-22T21:37:09.534573+0000 mon.a (mon.0) 12332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:10 smithi121 bash[36902]: cluster 2022-04-22T21:37:09.795712+0000 mgr.y (mgr.34107) 8570 : cluster [DBG] pgmap v9473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:10 smithi121 bash[36902]: audit 2022-04-22T21:37:10.481602+0000 mon.a (mon.0) 12333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:10 smithi115 bash[45495]: cluster 2022-04-22T21:37:09.795712+0000 mgr.y (mgr.34107) 8570 : cluster [DBG] pgmap v9473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:10 smithi115 bash[45495]: audit 2022-04-22T21:37:10.481602+0000 mon.a (mon.0) 12333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:10 smithi115 bash[46934]: cluster 2022-04-22T21:37:09.795712+0000 mgr.y (mgr.34107) 8570 : cluster [DBG] pgmap v9473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:10 smithi115 bash[46934]: audit 2022-04-22T21:37:10.481602+0000 mon.a (mon.0) 12333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:12.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:12 smithi115 bash[45495]: cluster 2022-04-22T21:37:11.796831+0000 mgr.y (mgr.34107) 8571 : cluster [DBG] pgmap v9474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:12.890 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:12.597Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:12.976 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:37:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:12 smithi121 bash[36902]: cluster 2022-04-22T21:37:11.796831+0000 mgr.y (mgr.34107) 8571 : cluster [DBG] pgmap v9474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:12 smithi115 bash[46934]: cluster 2022-04-22T21:37:11.796831+0000 mgr.y (mgr.34107) 8571 : cluster [DBG] pgmap v9474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:13.317 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:37:13.318 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:37:13.319 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:37:13.320 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:37:13.320 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:37:13.320 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:37:13.709 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:37:13.709 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:37:13.709 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:37:13.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:13.709 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:37:13.710 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:37:13.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:13.714 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:37:13.714 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:37:13.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:13.714 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:37:13.715 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:37:13.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:13.715 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:37:13.716 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:37:13.717 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:37:13.717 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:37:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:13 smithi121 bash[36902]: audit 2022-04-22T21:37:12.956823+0000 mgr.y (mgr.34107) 8572 : audit [DBG] from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:14.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:13 smithi121 bash[36902]: audit 2022-04-22T21:37:13.705189+0000 mon.c (mon.1) 305 : audit [DBG] from='client.? 172.21.15.115:0/329614137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:13 smithi115 bash[45495]: audit 2022-04-22T21:37:12.956823+0000 mgr.y (mgr.34107) 8572 : audit [DBG] from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:13 smithi115 bash[45495]: audit 2022-04-22T21:37:13.705189+0000 mon.c (mon.1) 305 : audit [DBG] from='client.? 172.21.15.115:0/329614137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:13 smithi115 bash[46934]: audit 2022-04-22T21:37:12.956823+0000 mgr.y (mgr.34107) 8572 : audit [DBG] from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:13 smithi115 bash[46934]: audit 2022-04-22T21:37:13.705189+0000 mon.c (mon.1) 305 : audit [DBG] from='client.? 172.21.15.115:0/329614137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:14.464Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:14 smithi121 bash[36902]: audit 2022-04-22T21:37:13.294253+0000 mgr.y (mgr.34107) 8573 : audit [DBG] from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:14 smithi121 bash[36902]: cluster 2022-04-22T21:37:13.797973+0000 mgr.y (mgr.34107) 8574 : cluster [DBG] pgmap v9475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:14 smithi121 bash[36902]: audit 2022-04-22T21:37:14.537964+0000 mon.a (mon.0) 12334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[45495]: audit 2022-04-22T21:37:13.294253+0000 mgr.y (mgr.34107) 8573 : audit [DBG] from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[45495]: cluster 2022-04-22T21:37:13.797973+0000 mgr.y (mgr.34107) 8574 : cluster [DBG] pgmap v9475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[45495]: audit 2022-04-22T21:37:14.537964+0000 mon.a (mon.0) 12334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[46934]: audit 2022-04-22T21:37:13.294253+0000 mgr.y (mgr.34107) 8573 : audit [DBG] from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[46934]: cluster 2022-04-22T21:37:13.797973+0000 mgr.y (mgr.34107) 8574 : cluster [DBG] pgmap v9475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:14 smithi115 bash[46934]: audit 2022-04-22T21:37:14.537964+0000 mon.a (mon.0) 12334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:16.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:15 smithi121 bash[36902]: audit 2022-04-22T21:37:15.485169+0000 mon.a (mon.0) 12335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:15 smithi115 bash[45495]: audit 2022-04-22T21:37:15.485169+0000 mon.a (mon.0) 12335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:15 smithi115 bash[46934]: audit 2022-04-22T21:37:15.485169+0000 mon.a (mon.0) 12335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:16 smithi121 bash[36902]: cluster 2022-04-22T21:37:15.799059+0000 mgr.y (mgr.34107) 8575 : cluster [DBG] pgmap v9476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:16 smithi121 bash[36902]: audit 2022-04-22T21:37:16.159663+0000 mon.a (mon.0) 12336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:37:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:16 smithi121 bash[36902]: audit 2022-04-22T21:37:16.162407+0000 mon.a (mon.0) 12337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[45495]: cluster 2022-04-22T21:37:15.799059+0000 mgr.y (mgr.34107) 8575 : cluster [DBG] pgmap v9476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[45495]: audit 2022-04-22T21:37:16.159663+0000 mon.a (mon.0) 12336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[45495]: audit 2022-04-22T21:37:16.162407+0000 mon.a (mon.0) 12337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[46934]: cluster 2022-04-22T21:37:15.799059+0000 mgr.y (mgr.34107) 8575 : cluster [DBG] pgmap v9476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[46934]: audit 2022-04-22T21:37:16.159663+0000 mon.a (mon.0) 12336 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:37:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:16 smithi115 bash[46934]: audit 2022-04-22T21:37:16.162407+0000 mon.a (mon.0) 12337 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:37:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:18 smithi121 bash[36902]: cluster 2022-04-22T21:37:17.800174+0000 mgr.y (mgr.34107) 8576 : cluster [DBG] pgmap v9477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:18 smithi115 bash[45495]: cluster 2022-04-22T21:37:17.800174+0000 mgr.y (mgr.34107) 8576 : cluster [DBG] pgmap v9477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:18 smithi115 bash[46934]: cluster 2022-04-22T21:37:17.800174+0000 mgr.y (mgr.34107) 8576 : cluster [DBG] pgmap v9477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:19 smithi121 bash[36902]: audit 2022-04-22T21:37:19.541105+0000 mon.a (mon.0) 12338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:19 smithi115 bash[45495]: audit 2022-04-22T21:37:19.541105+0000 mon.a (mon.0) 12338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:19 smithi115 bash[46934]: audit 2022-04-22T21:37:19.541105+0000 mon.a (mon.0) 12338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:20 smithi121 bash[36902]: cluster 2022-04-22T21:37:19.800995+0000 mgr.y (mgr.34107) 8577 : cluster [DBG] pgmap v9478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:20 smithi121 bash[36902]: audit 2022-04-22T21:37:20.488985+0000 mon.a (mon.0) 12339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:20 smithi115 bash[45495]: cluster 2022-04-22T21:37:19.800995+0000 mgr.y (mgr.34107) 8577 : cluster [DBG] pgmap v9478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:20 smithi115 bash[45495]: audit 2022-04-22T21:37:20.488985+0000 mon.a (mon.0) 12339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:20 smithi115 bash[46934]: cluster 2022-04-22T21:37:19.800995+0000 mgr.y (mgr.34107) 8577 : cluster [DBG] pgmap v9478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:20 smithi115 bash[46934]: audit 2022-04-22T21:37:20.488985+0000 mon.a (mon.0) 12339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:22.922 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:22.598Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:22 smithi121 bash[36902]: cluster 2022-04-22T21:37:21.802204+0000 mgr.y (mgr.34107) 8578 : cluster [DBG] pgmap v9479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:22 smithi115 bash[45495]: cluster 2022-04-22T21:37:21.802204+0000 mgr.y (mgr.34107) 8578 : cluster [DBG] pgmap v9479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:22 smithi115 bash[46934]: cluster 2022-04-22T21:37:21.802204+0000 mgr.y (mgr.34107) 8578 : cluster [DBG] pgmap v9479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:24.466Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:24 smithi121 bash[36902]: cluster 2022-04-22T21:37:23.803392+0000 mgr.y (mgr.34107) 8579 : cluster [DBG] pgmap v9480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:24 smithi121 bash[36902]: audit 2022-04-22T21:37:24.544154+0000 mon.a (mon.0) 12340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:25.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:24 smithi115 bash[46934]: cluster 2022-04-22T21:37:23.803392+0000 mgr.y (mgr.34107) 8579 : cluster [DBG] pgmap v9480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:24 smithi115 bash[46934]: audit 2022-04-22T21:37:24.544154+0000 mon.a (mon.0) 12340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:24 smithi115 bash[45495]: cluster 2022-04-22T21:37:23.803392+0000 mgr.y (mgr.34107) 8579 : cluster [DBG] pgmap v9480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:24 smithi115 bash[45495]: audit 2022-04-22T21:37:24.544154+0000 mon.a (mon.0) 12340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:25 smithi121 bash[36902]: audit 2022-04-22T21:37:25.492827+0000 mon.a (mon.0) 12341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:25 smithi115 bash[45495]: audit 2022-04-22T21:37:25.492827+0000 mon.a (mon.0) 12341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:25 smithi115 bash[46934]: audit 2022-04-22T21:37:25.492827+0000 mon.a (mon.0) 12341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:26 smithi121 bash[36902]: cluster 2022-04-22T21:37:25.804468+0000 mgr.y (mgr.34107) 8580 : cluster [DBG] pgmap v9481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:26 smithi115 bash[46934]: cluster 2022-04-22T21:37:25.804468+0000 mgr.y (mgr.34107) 8580 : cluster [DBG] pgmap v9481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:26 smithi115 bash[45495]: cluster 2022-04-22T21:37:25.804468+0000 mgr.y (mgr.34107) 8580 : cluster [DBG] pgmap v9481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:28 smithi121 bash[36902]: cluster 2022-04-22T21:37:27.805399+0000 mgr.y (mgr.34107) 8581 : cluster [DBG] pgmap v9482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:28 smithi115 bash[45495]: cluster 2022-04-22T21:37:27.805399+0000 mgr.y (mgr.34107) 8581 : cluster [DBG] pgmap v9482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:28 smithi115 bash[46934]: cluster 2022-04-22T21:37:27.805399+0000 mgr.y (mgr.34107) 8581 : cluster [DBG] pgmap v9482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:29 smithi121 bash[36902]: audit 2022-04-22T21:37:29.547114+0000 mon.a (mon.0) 12342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:29 smithi115 bash[45495]: audit 2022-04-22T21:37:29.547114+0000 mon.a (mon.0) 12342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:29 smithi115 bash[46934]: audit 2022-04-22T21:37:29.547114+0000 mon.a (mon.0) 12342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:30 smithi121 bash[36902]: cluster 2022-04-22T21:37:29.806459+0000 mgr.y (mgr.34107) 8582 : cluster [DBG] pgmap v9483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:30 smithi121 bash[36902]: audit 2022-04-22T21:37:30.496460+0000 mon.a (mon.0) 12343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:30 smithi115 bash[46934]: cluster 2022-04-22T21:37:29.806459+0000 mgr.y (mgr.34107) 8582 : cluster [DBG] pgmap v9483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:30 smithi115 bash[46934]: audit 2022-04-22T21:37:30.496460+0000 mon.a (mon.0) 12343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:30 smithi115 bash[45495]: cluster 2022-04-22T21:37:29.806459+0000 mgr.y (mgr.34107) 8582 : cluster [DBG] pgmap v9483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:30 smithi115 bash[45495]: audit 2022-04-22T21:37:30.496460+0000 mon.a (mon.0) 12343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:32.952 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:32.598Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:32 smithi121 bash[36902]: cluster 2022-04-22T21:37:31.807578+0000 mgr.y (mgr.34107) 8583 : cluster [DBG] pgmap v9484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:32 smithi115 bash[45495]: cluster 2022-04-22T21:37:31.807578+0000 mgr.y (mgr.34107) 8583 : cluster [DBG] pgmap v9484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:32 smithi115 bash[46934]: cluster 2022-04-22T21:37:31.807578+0000 mgr.y (mgr.34107) 8583 : cluster [DBG] pgmap v9484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:34.467Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:34 smithi121 bash[36902]: cluster 2022-04-22T21:37:33.808691+0000 mgr.y (mgr.34107) 8584 : cluster [DBG] pgmap v9485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:34 smithi121 bash[36902]: audit 2022-04-22T21:37:34.550297+0000 mon.a (mon.0) 12344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:34 smithi115 bash[45495]: cluster 2022-04-22T21:37:33.808691+0000 mgr.y (mgr.34107) 8584 : cluster [DBG] pgmap v9485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:34 smithi115 bash[45495]: audit 2022-04-22T21:37:34.550297+0000 mon.a (mon.0) 12344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:34 smithi115 bash[46934]: cluster 2022-04-22T21:37:33.808691+0000 mgr.y (mgr.34107) 8584 : cluster [DBG] pgmap v9485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:34 smithi115 bash[46934]: audit 2022-04-22T21:37:34.550297+0000 mon.a (mon.0) 12344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:36.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:35 smithi121 bash[36902]: audit 2022-04-22T21:37:35.500401+0000 mon.a (mon.0) 12345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:35 smithi115 bash[45495]: audit 2022-04-22T21:37:35.500401+0000 mon.a (mon.0) 12345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:35 smithi115 bash[46934]: audit 2022-04-22T21:37:35.500401+0000 mon.a (mon.0) 12345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:36 smithi121 bash[36902]: cluster 2022-04-22T21:37:35.809748+0000 mgr.y (mgr.34107) 8585 : cluster [DBG] pgmap v9486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:36 smithi115 bash[45495]: cluster 2022-04-22T21:37:35.809748+0000 mgr.y (mgr.34107) 8585 : cluster [DBG] pgmap v9486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:36 smithi115 bash[46934]: cluster 2022-04-22T21:37:35.809748+0000 mgr.y (mgr.34107) 8585 : cluster [DBG] pgmap v9486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:38 smithi121 bash[36902]: cluster 2022-04-22T21:37:37.810873+0000 mgr.y (mgr.34107) 8586 : cluster [DBG] pgmap v9487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:38 smithi115 bash[45495]: cluster 2022-04-22T21:37:37.810873+0000 mgr.y (mgr.34107) 8586 : cluster [DBG] pgmap v9487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:38 smithi115 bash[46934]: cluster 2022-04-22T21:37:37.810873+0000 mgr.y (mgr.34107) 8586 : cluster [DBG] pgmap v9487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:39 smithi121 bash[36902]: audit 2022-04-22T21:37:38.989839+0000 mon.a (mon.0) 12346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:39 smithi121 bash[36902]: audit 2022-04-22T21:37:38.992803+0000 mon.a (mon.0) 12347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:39 smithi121 bash[36902]: audit 2022-04-22T21:37:38.994587+0000 mon.a (mon.0) 12348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:39 smithi121 bash[36902]: audit 2022-04-22T21:37:39.347521+0000 mon.a (mon.0) 12349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:39 smithi121 bash[36902]: audit 2022-04-22T21:37:39.553536+0000 mon.a (mon.0) 12350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[45495]: audit 2022-04-22T21:37:38.989839+0000 mon.a (mon.0) 12346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:37:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[45495]: audit 2022-04-22T21:37:38.992803+0000 mon.a (mon.0) 12347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:37:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[45495]: audit 2022-04-22T21:37:38.994587+0000 mon.a (mon.0) 12348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:37:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[45495]: audit 2022-04-22T21:37:39.347521+0000 mon.a (mon.0) 12349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:37:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[45495]: audit 2022-04-22T21:37:39.553536+0000 mon.a (mon.0) 12350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[46934]: audit 2022-04-22T21:37:38.989839+0000 mon.a (mon.0) 12346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:37:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[46934]: audit 2022-04-22T21:37:38.992803+0000 mon.a (mon.0) 12347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:37:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[46934]: audit 2022-04-22T21:37:38.994587+0000 mon.a (mon.0) 12348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:37:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[46934]: audit 2022-04-22T21:37:39.347521+0000 mon.a (mon.0) 12349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:37:40.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:39 smithi115 bash[46934]: audit 2022-04-22T21:37:39.553536+0000 mon.a (mon.0) 12350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:40 smithi121 bash[36902]: cluster 2022-04-22T21:37:39.811952+0000 mgr.y (mgr.34107) 8587 : cluster [DBG] pgmap v9488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:40 smithi121 bash[36902]: audit 2022-04-22T21:37:40.504240+0000 mon.a (mon.0) 12351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:40 smithi115 bash[45495]: cluster 2022-04-22T21:37:39.811952+0000 mgr.y (mgr.34107) 8587 : cluster [DBG] pgmap v9488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:40 smithi115 bash[45495]: audit 2022-04-22T21:37:40.504240+0000 mon.a (mon.0) 12351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:40 smithi115 bash[46934]: cluster 2022-04-22T21:37:39.811952+0000 mgr.y (mgr.34107) 8587 : cluster [DBG] pgmap v9488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:40 smithi115 bash[46934]: audit 2022-04-22T21:37:40.504240+0000 mon.a (mon.0) 12351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:42.976 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:42.600Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:42 smithi121 bash[36902]: cluster 2022-04-22T21:37:41.813164+0000 mgr.y (mgr.34107) 8588 : cluster [DBG] pgmap v9489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:42 smithi115 bash[45495]: cluster 2022-04-22T21:37:41.813164+0000 mgr.y (mgr.34107) 8588 : cluster [DBG] pgmap v9489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:42 smithi115 bash[46934]: cluster 2022-04-22T21:37:41.813164+0000 mgr.y (mgr.34107) 8588 : cluster [DBG] pgmap v9489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:44.086 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:37:44.429 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:37:44.430 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:37:44.430 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:37:44.430 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:37:44.430 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:37:44.431 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:37:44.431 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:37:44.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:37:44.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:37:44.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:37:44.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:37:44.433 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:37:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:44.466Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:44.839 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:37:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:37:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:37:44.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:37:44.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:37:44.842 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:37:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:44 smithi121 bash[36902]: cluster 2022-04-22T21:37:43.814319+0000 mgr.y (mgr.34107) 8589 : cluster [DBG] pgmap v9490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:45.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:44 smithi121 bash[36902]: audit 2022-04-22T21:37:44.556230+0000 mon.a (mon.0) 12352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:45.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:44 smithi121 bash[36902]: audit 2022-04-22T21:37:44.835334+0000 mon.a (mon.0) 12353 : audit [DBG] from='client.? 172.21.15.115:0/1622189833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:45.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[45495]: cluster 2022-04-22T21:37:43.814319+0000 mgr.y (mgr.34107) 8589 : cluster [DBG] pgmap v9490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:45.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[45495]: audit 2022-04-22T21:37:44.556230+0000 mon.a (mon.0) 12352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:45.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[45495]: audit 2022-04-22T21:37:44.835334+0000 mon.a (mon.0) 12353 : audit [DBG] from='client.? 172.21.15.115:0/1622189833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:45.410 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[46934]: cluster 2022-04-22T21:37:43.814319+0000 mgr.y (mgr.34107) 8589 : cluster [DBG] pgmap v9490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:45.410 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[46934]: audit 2022-04-22T21:37:44.556230+0000 mon.a (mon.0) 12352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:45.410 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:44 smithi115 bash[46934]: audit 2022-04-22T21:37:44.835334+0000 mon.a (mon.0) 12353 : audit [DBG] from='client.? 172.21.15.115:0/1622189833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:37:46.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:45 smithi121 bash[36902]: audit 2022-04-22T21:37:44.067699+0000 mgr.y (mgr.34107) 8590 : audit [DBG] from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:45 smithi121 bash[36902]: audit 2022-04-22T21:37:44.407821+0000 mgr.y (mgr.34107) 8591 : audit [DBG] from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:45 smithi121 bash[36902]: audit 2022-04-22T21:37:45.509704+0000 mon.a (mon.0) 12354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[45495]: audit 2022-04-22T21:37:44.067699+0000 mgr.y (mgr.34107) 8590 : audit [DBG] from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[45495]: audit 2022-04-22T21:37:44.407821+0000 mgr.y (mgr.34107) 8591 : audit [DBG] from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[45495]: audit 2022-04-22T21:37:45.509704+0000 mon.a (mon.0) 12354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[46934]: audit 2022-04-22T21:37:44.067699+0000 mgr.y (mgr.34107) 8590 : audit [DBG] from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[46934]: audit 2022-04-22T21:37:44.407821+0000 mgr.y (mgr.34107) 8591 : audit [DBG] from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:37:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:45 smithi115 bash[46934]: audit 2022-04-22T21:37:45.509704+0000 mon.a (mon.0) 12354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:46 smithi121 bash[36902]: cluster 2022-04-22T21:37:45.815411+0000 mgr.y (mgr.34107) 8592 : cluster [DBG] pgmap v9491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:46 smithi115 bash[45495]: cluster 2022-04-22T21:37:45.815411+0000 mgr.y (mgr.34107) 8592 : cluster [DBG] pgmap v9491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:46 smithi115 bash[46934]: cluster 2022-04-22T21:37:45.815411+0000 mgr.y (mgr.34107) 8592 : cluster [DBG] pgmap v9491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:48 smithi121 bash[36902]: cluster 2022-04-22T21:37:47.816466+0000 mgr.y (mgr.34107) 8593 : cluster [DBG] pgmap v9492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:48 smithi115 bash[45495]: cluster 2022-04-22T21:37:47.816466+0000 mgr.y (mgr.34107) 8593 : cluster [DBG] pgmap v9492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:48 smithi115 bash[46934]: cluster 2022-04-22T21:37:47.816466+0000 mgr.y (mgr.34107) 8593 : cluster [DBG] pgmap v9492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:50 smithi121 bash[36902]: audit 2022-04-22T21:37:49.558984+0000 mon.a (mon.0) 12355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:49 smithi115 bash[45495]: audit 2022-04-22T21:37:49.558984+0000 mon.a (mon.0) 12355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:49 smithi115 bash[46934]: audit 2022-04-22T21:37:49.558984+0000 mon.a (mon.0) 12355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:51 smithi121 bash[36902]: cluster 2022-04-22T21:37:49.817558+0000 mgr.y (mgr.34107) 8594 : cluster [DBG] pgmap v9493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:51 smithi121 bash[36902]: audit 2022-04-22T21:37:50.511849+0000 mon.a (mon.0) 12356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:50 smithi115 bash[45495]: cluster 2022-04-22T21:37:49.817558+0000 mgr.y (mgr.34107) 8594 : cluster [DBG] pgmap v9493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:50 smithi115 bash[45495]: audit 2022-04-22T21:37:50.511849+0000 mon.a (mon.0) 12356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:51 smithi115 bash[46934]: cluster 2022-04-22T21:37:49.817558+0000 mgr.y (mgr.34107) 8594 : cluster [DBG] pgmap v9493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:51 smithi115 bash[46934]: audit 2022-04-22T21:37:50.511849+0000 mon.a (mon.0) 12356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:53.008 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:52.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:53 smithi121 bash[36902]: cluster 2022-04-22T21:37:51.818456+0000 mgr.y (mgr.34107) 8595 : cluster [DBG] pgmap v9494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:53 smithi115 bash[46934]: cluster 2022-04-22T21:37:51.818456+0000 mgr.y (mgr.34107) 8595 : cluster [DBG] pgmap v9494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:53 smithi115 bash[45495]: cluster 2022-04-22T21:37:51.818456+0000 mgr.y (mgr.34107) 8595 : cluster [DBG] pgmap v9494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:37:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:37:54.468Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:37:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:55 smithi121 bash[36902]: cluster 2022-04-22T21:37:53.819623+0000 mgr.y (mgr.34107) 8596 : cluster [DBG] pgmap v9495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:55 smithi121 bash[36902]: audit 2022-04-22T21:37:54.562037+0000 mon.a (mon.0) 12357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:55 smithi115 bash[45495]: cluster 2022-04-22T21:37:53.819623+0000 mgr.y (mgr.34107) 8596 : cluster [DBG] pgmap v9495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:55 smithi115 bash[45495]: audit 2022-04-22T21:37:54.562037+0000 mon.a (mon.0) 12357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:55 smithi115 bash[46934]: cluster 2022-04-22T21:37:53.819623+0000 mgr.y (mgr.34107) 8596 : cluster [DBG] pgmap v9495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:55 smithi115 bash[46934]: audit 2022-04-22T21:37:54.562037+0000 mon.a (mon.0) 12357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:37:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:56 smithi115 bash[45495]: audit 2022-04-22T21:37:55.519322+0000 mon.a (mon.0) 12358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:56 smithi115 bash[45495]: cluster 2022-04-22T21:37:55.820995+0000 mgr.y (mgr.34107) 8597 : cluster [DBG] pgmap v9496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:56 smithi115 bash[46934]: audit 2022-04-22T21:37:55.519322+0000 mon.a (mon.0) 12358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:56 smithi115 bash[46934]: cluster 2022-04-22T21:37:55.820995+0000 mgr.y (mgr.34107) 8597 : cluster [DBG] pgmap v9496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:56 smithi121 bash[36902]: audit 2022-04-22T21:37:55.519322+0000 mon.a (mon.0) 12358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:37:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:56 smithi121 bash[36902]: cluster 2022-04-22T21:37:55.820995+0000 mgr.y (mgr.34107) 8597 : cluster [DBG] pgmap v9496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:58 smithi121 bash[36902]: cluster 2022-04-22T21:37:57.822094+0000 mgr.y (mgr.34107) 8598 : cluster [DBG] pgmap v9497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:58 smithi115 bash[45495]: cluster 2022-04-22T21:37:57.822094+0000 mgr.y (mgr.34107) 8598 : cluster [DBG] pgmap v9497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:37:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:58 smithi115 bash[46934]: cluster 2022-04-22T21:37:57.822094+0000 mgr.y (mgr.34107) 8598 : cluster [DBG] pgmap v9497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:37:59 smithi121 bash[36902]: audit 2022-04-22T21:37:59.565039+0000 mon.a (mon.0) 12359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:37:59 smithi115 bash[45495]: audit 2022-04-22T21:37:59.565039+0000 mon.a (mon.0) 12359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:37:59 smithi115 bash[46934]: audit 2022-04-22T21:37:59.565039+0000 mon.a (mon.0) 12359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:00 smithi121 bash[36902]: cluster 2022-04-22T21:37:59.823140+0000 mgr.y (mgr.34107) 8599 : cluster [DBG] pgmap v9498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:01.278 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:00 smithi121 bash[36902]: audit 2022-04-22T21:38:00.520331+0000 mon.a (mon.0) 12360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:00 smithi115 bash[45495]: cluster 2022-04-22T21:37:59.823140+0000 mgr.y (mgr.34107) 8599 : cluster [DBG] pgmap v9498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:00 smithi115 bash[45495]: audit 2022-04-22T21:38:00.520331+0000 mon.a (mon.0) 12360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:00 smithi115 bash[46934]: cluster 2022-04-22T21:37:59.823140+0000 mgr.y (mgr.34107) 8599 : cluster [DBG] pgmap v9498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:00 smithi115 bash[46934]: audit 2022-04-22T21:38:00.520331+0000 mon.a (mon.0) 12360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:02.896 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:02.600Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:02 smithi121 bash[36902]: cluster 2022-04-22T21:38:01.824263+0000 mgr.y (mgr.34107) 8600 : cluster [DBG] pgmap v9499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:02 smithi115 bash[45495]: cluster 2022-04-22T21:38:01.824263+0000 mgr.y (mgr.34107) 8600 : cluster [DBG] pgmap v9499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:02 smithi115 bash[46934]: cluster 2022-04-22T21:38:01.824263+0000 mgr.y (mgr.34107) 8600 : cluster [DBG] pgmap v9499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:04.468Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:05.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:04 smithi121 bash[36902]: cluster 2022-04-22T21:38:03.825441+0000 mgr.y (mgr.34107) 8601 : cluster [DBG] pgmap v9500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:04 smithi121 bash[36902]: audit 2022-04-22T21:38:04.568261+0000 mon.a (mon.0) 12361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:04 smithi115 bash[45495]: cluster 2022-04-22T21:38:03.825441+0000 mgr.y (mgr.34107) 8601 : cluster [DBG] pgmap v9500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:04 smithi115 bash[45495]: audit 2022-04-22T21:38:04.568261+0000 mon.a (mon.0) 12361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:04 smithi115 bash[46934]: cluster 2022-04-22T21:38:03.825441+0000 mgr.y (mgr.34107) 8601 : cluster [DBG] pgmap v9500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:04 smithi115 bash[46934]: audit 2022-04-22T21:38:04.568261+0000 mon.a (mon.0) 12361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:06 smithi115 bash[45495]: audit 2022-04-22T21:38:05.523362+0000 mon.a (mon.0) 12362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:06 smithi115 bash[45495]: cluster 2022-04-22T21:38:05.826486+0000 mgr.y (mgr.34107) 8602 : cluster [DBG] pgmap v9501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:06 smithi115 bash[46934]: audit 2022-04-22T21:38:05.523362+0000 mon.a (mon.0) 12362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:06 smithi115 bash[46934]: cluster 2022-04-22T21:38:05.826486+0000 mgr.y (mgr.34107) 8602 : cluster [DBG] pgmap v9501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:06 smithi121 bash[36902]: audit 2022-04-22T21:38:05.523362+0000 mon.a (mon.0) 12362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:06 smithi121 bash[36902]: cluster 2022-04-22T21:38:05.826486+0000 mgr.y (mgr.34107) 8602 : cluster [DBG] pgmap v9501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:08 smithi121 bash[36902]: cluster 2022-04-22T21:38:07.827626+0000 mgr.y (mgr.34107) 8603 : cluster [DBG] pgmap v9502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:08 smithi115 bash[45495]: cluster 2022-04-22T21:38:07.827626+0000 mgr.y (mgr.34107) 8603 : cluster [DBG] pgmap v9502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:08 smithi115 bash[46934]: cluster 2022-04-22T21:38:07.827626+0000 mgr.y (mgr.34107) 8603 : cluster [DBG] pgmap v9502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:09 smithi121 bash[36902]: audit 2022-04-22T21:38:09.571405+0000 mon.a (mon.0) 12363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:10.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:09 smithi115 bash[45495]: audit 2022-04-22T21:38:09.571405+0000 mon.a (mon.0) 12363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:10.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:09 smithi115 bash[46934]: audit 2022-04-22T21:38:09.571405+0000 mon.a (mon.0) 12363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:10 smithi121 bash[36902]: cluster 2022-04-22T21:38:09.828690+0000 mgr.y (mgr.34107) 8604 : cluster [DBG] pgmap v9503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:10 smithi121 bash[36902]: audit 2022-04-22T21:38:10.526847+0000 mon.a (mon.0) 12364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:10 smithi115 bash[45495]: cluster 2022-04-22T21:38:09.828690+0000 mgr.y (mgr.34107) 8604 : cluster [DBG] pgmap v9503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:10 smithi115 bash[45495]: audit 2022-04-22T21:38:10.526847+0000 mon.a (mon.0) 12364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:10 smithi115 bash[46934]: cluster 2022-04-22T21:38:09.828690+0000 mgr.y (mgr.34107) 8604 : cluster [DBG] pgmap v9503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:10 smithi115 bash[46934]: audit 2022-04-22T21:38:10.526847+0000 mon.a (mon.0) 12364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:12.911 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:12.602Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:12 smithi121 bash[36902]: cluster 2022-04-22T21:38:11.829774+0000 mgr.y (mgr.34107) 8605 : cluster [DBG] pgmap v9504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:12 smithi115 bash[45495]: cluster 2022-04-22T21:38:11.829774+0000 mgr.y (mgr.34107) 8605 : cluster [DBG] pgmap v9504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:12 smithi115 bash[46934]: cluster 2022-04-22T21:38:11.829774+0000 mgr.y (mgr.34107) 8605 : cluster [DBG] pgmap v9504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:14.470Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:15.216 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:38:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:14 smithi121 bash[36902]: cluster 2022-04-22T21:38:13.830851+0000 mgr.y (mgr.34107) 8606 : cluster [DBG] pgmap v9505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:14 smithi121 bash[36902]: audit 2022-04-22T21:38:14.574627+0000 mon.a (mon.0) 12365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:14 smithi115 bash[45495]: cluster 2022-04-22T21:38:13.830851+0000 mgr.y (mgr.34107) 8606 : cluster [DBG] pgmap v9505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:14 smithi115 bash[45495]: audit 2022-04-22T21:38:14.574627+0000 mon.a (mon.0) 12365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:14 smithi115 bash[46934]: cluster 2022-04-22T21:38:13.830851+0000 mgr.y (mgr.34107) 8606 : cluster [DBG] pgmap v9505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:14 smithi115 bash[46934]: audit 2022-04-22T21:38:14.574627+0000 mon.a (mon.0) 12365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:15.560 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:38:15.560 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:38:15.561 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:38:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:38:15.961 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:38:15.995 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:38:15.995 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:38:15.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:15.997 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:38:15.997 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:38:15.997 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:38:15.997 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:15.194405+0000 mgr.y (mgr.34107) 8607 : audit [DBG] from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:15.529685+0000 mon.a (mon.0) 12366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:15.538777+0000 mgr.y (mgr.34107) 8608 : audit [DBG] from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: cluster 2022-04-22T21:38:15.831909+0000 mgr.y (mgr.34107) 8609 : cluster [DBG] pgmap v9506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:15.957376+0000 mon.c (mon.1) 306 : audit [DBG] from='client.? 172.21.15.115:0/1761482298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:16.169839+0000 mon.a (mon.0) 12367 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[46934]: audit 2022-04-22T21:38:16.175944+0000 mon.a (mon.0) 12368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:15.194405+0000 mgr.y (mgr.34107) 8607 : audit [DBG] from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:15.529685+0000 mon.a (mon.0) 12366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:15.538777+0000 mgr.y (mgr.34107) 8608 : audit [DBG] from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: cluster 2022-04-22T21:38:15.831909+0000 mgr.y (mgr.34107) 8609 : cluster [DBG] pgmap v9506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:15.957376+0000 mon.c (mon.1) 306 : audit [DBG] from='client.? 172.21.15.115:0/1761482298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:16.169839+0000 mon.a (mon.0) 12367 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:38:16.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:16 smithi115 bash[45495]: audit 2022-04-22T21:38:16.175944+0000 mon.a (mon.0) 12368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:38:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:15.194405+0000 mgr.y (mgr.34107) 8607 : audit [DBG] from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:15.529685+0000 mon.a (mon.0) 12366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:15.538777+0000 mgr.y (mgr.34107) 8608 : audit [DBG] from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: cluster 2022-04-22T21:38:15.831909+0000 mgr.y (mgr.34107) 8609 : cluster [DBG] pgmap v9506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:15.957376+0000 mon.c (mon.1) 306 : audit [DBG] from='client.? 172.21.15.115:0/1761482298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:16.169839+0000 mon.a (mon.0) 12367 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:38:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:16 smithi121 bash[36902]: audit 2022-04-22T21:38:16.175944+0000 mon.a (mon.0) 12368 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:38:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:18 smithi121 bash[36902]: cluster 2022-04-22T21:38:17.832997+0000 mgr.y (mgr.34107) 8610 : cluster [DBG] pgmap v9507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:18 smithi115 bash[45495]: cluster 2022-04-22T21:38:17.832997+0000 mgr.y (mgr.34107) 8610 : cluster [DBG] pgmap v9507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:18 smithi115 bash[46934]: cluster 2022-04-22T21:38:17.832997+0000 mgr.y (mgr.34107) 8610 : cluster [DBG] pgmap v9507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:19 smithi121 bash[36902]: audit 2022-04-22T21:38:19.577663+0000 mon.a (mon.0) 12369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:19 smithi115 bash[45495]: audit 2022-04-22T21:38:19.577663+0000 mon.a (mon.0) 12369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:19 smithi115 bash[46934]: audit 2022-04-22T21:38:19.577663+0000 mon.a (mon.0) 12369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:20 smithi121 bash[36902]: cluster 2022-04-22T21:38:19.834032+0000 mgr.y (mgr.34107) 8611 : cluster [DBG] pgmap v9508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:21.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:20 smithi121 bash[36902]: audit 2022-04-22T21:38:20.532674+0000 mon.a (mon.0) 12370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:20 smithi115 bash[45495]: cluster 2022-04-22T21:38:19.834032+0000 mgr.y (mgr.34107) 8611 : cluster [DBG] pgmap v9508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:20 smithi115 bash[45495]: audit 2022-04-22T21:38:20.532674+0000 mon.a (mon.0) 12370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:20 smithi115 bash[46934]: cluster 2022-04-22T21:38:19.834032+0000 mgr.y (mgr.34107) 8611 : cluster [DBG] pgmap v9508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:20 smithi115 bash[46934]: audit 2022-04-22T21:38:20.532674+0000 mon.a (mon.0) 12370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:22.907 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:22.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:22 smithi121 bash[36902]: cluster 2022-04-22T21:38:21.835041+0000 mgr.y (mgr.34107) 8612 : cluster [DBG] pgmap v9509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:22 smithi115 bash[45495]: cluster 2022-04-22T21:38:21.835041+0000 mgr.y (mgr.34107) 8612 : cluster [DBG] pgmap v9509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:22 smithi115 bash[46934]: cluster 2022-04-22T21:38:21.835041+0000 mgr.y (mgr.34107) 8612 : cluster [DBG] pgmap v9509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:24.470Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:24 smithi121 bash[36902]: cluster 2022-04-22T21:38:23.836232+0000 mgr.y (mgr.34107) 8613 : cluster [DBG] pgmap v9510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:24 smithi121 bash[36902]: audit 2022-04-22T21:38:24.580786+0000 mon.a (mon.0) 12371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:24 smithi115 bash[45495]: cluster 2022-04-22T21:38:23.836232+0000 mgr.y (mgr.34107) 8613 : cluster [DBG] pgmap v9510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:24 smithi115 bash[45495]: audit 2022-04-22T21:38:24.580786+0000 mon.a (mon.0) 12371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:24 smithi115 bash[46934]: cluster 2022-04-22T21:38:23.836232+0000 mgr.y (mgr.34107) 8613 : cluster [DBG] pgmap v9510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:24 smithi115 bash[46934]: audit 2022-04-22T21:38:24.580786+0000 mon.a (mon.0) 12371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:26 smithi115 bash[45495]: audit 2022-04-22T21:38:25.535675+0000 mon.a (mon.0) 12372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:26 smithi115 bash[45495]: cluster 2022-04-22T21:38:25.837265+0000 mgr.y (mgr.34107) 8614 : cluster [DBG] pgmap v9511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:26 smithi115 bash[46934]: audit 2022-04-22T21:38:25.535675+0000 mon.a (mon.0) 12372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:26 smithi115 bash[46934]: cluster 2022-04-22T21:38:25.837265+0000 mgr.y (mgr.34107) 8614 : cluster [DBG] pgmap v9511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:26 smithi121 bash[36902]: audit 2022-04-22T21:38:25.535675+0000 mon.a (mon.0) 12372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:26 smithi121 bash[36902]: cluster 2022-04-22T21:38:25.837265+0000 mgr.y (mgr.34107) 8614 : cluster [DBG] pgmap v9511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:28 smithi121 bash[36902]: cluster 2022-04-22T21:38:27.838343+0000 mgr.y (mgr.34107) 8615 : cluster [DBG] pgmap v9512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:28 smithi115 bash[45495]: cluster 2022-04-22T21:38:27.838343+0000 mgr.y (mgr.34107) 8615 : cluster [DBG] pgmap v9512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:28 smithi115 bash[46934]: cluster 2022-04-22T21:38:27.838343+0000 mgr.y (mgr.34107) 8615 : cluster [DBG] pgmap v9512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:29 smithi121 bash[36902]: audit 2022-04-22T21:38:29.583921+0000 mon.a (mon.0) 12373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:29 smithi115 bash[45495]: audit 2022-04-22T21:38:29.583921+0000 mon.a (mon.0) 12373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:29 smithi115 bash[46934]: audit 2022-04-22T21:38:29.583921+0000 mon.a (mon.0) 12373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:30 smithi121 bash[36902]: cluster 2022-04-22T21:38:29.839394+0000 mgr.y (mgr.34107) 8616 : cluster [DBG] pgmap v9513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:30 smithi121 bash[36902]: audit 2022-04-22T21:38:30.538582+0000 mon.a (mon.0) 12374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:30 smithi115 bash[45495]: cluster 2022-04-22T21:38:29.839394+0000 mgr.y (mgr.34107) 8616 : cluster [DBG] pgmap v9513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:30 smithi115 bash[45495]: audit 2022-04-22T21:38:30.538582+0000 mon.a (mon.0) 12374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:31.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:30 smithi115 bash[46934]: cluster 2022-04-22T21:38:29.839394+0000 mgr.y (mgr.34107) 8616 : cluster [DBG] pgmap v9513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:31.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:30 smithi115 bash[46934]: audit 2022-04-22T21:38:30.538582+0000 mon.a (mon.0) 12374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:32.915 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:32.603Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:32 smithi121 bash[36902]: cluster 2022-04-22T21:38:31.840501+0000 mgr.y (mgr.34107) 8617 : cluster [DBG] pgmap v9514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:32 smithi115 bash[45495]: cluster 2022-04-22T21:38:31.840501+0000 mgr.y (mgr.34107) 8617 : cluster [DBG] pgmap v9514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:32 smithi115 bash[46934]: cluster 2022-04-22T21:38:31.840501+0000 mgr.y (mgr.34107) 8617 : cluster [DBG] pgmap v9514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:34.470Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:34 smithi121 bash[36902]: cluster 2022-04-22T21:38:33.841653+0000 mgr.y (mgr.34107) 8618 : cluster [DBG] pgmap v9515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:34 smithi121 bash[36902]: audit 2022-04-22T21:38:34.586874+0000 mon.a (mon.0) 12375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:34 smithi115 bash[46934]: cluster 2022-04-22T21:38:33.841653+0000 mgr.y (mgr.34107) 8618 : cluster [DBG] pgmap v9515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:34 smithi115 bash[46934]: audit 2022-04-22T21:38:34.586874+0000 mon.a (mon.0) 12375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:34 smithi115 bash[45495]: cluster 2022-04-22T21:38:33.841653+0000 mgr.y (mgr.34107) 8618 : cluster [DBG] pgmap v9515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:35.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:34 smithi115 bash[45495]: audit 2022-04-22T21:38:34.586874+0000 mon.a (mon.0) 12375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:36 smithi115 bash[46934]: audit 2022-04-22T21:38:35.541833+0000 mon.a (mon.0) 12376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:36 smithi115 bash[46934]: cluster 2022-04-22T21:38:35.842697+0000 mgr.y (mgr.34107) 8619 : cluster [DBG] pgmap v9516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:36 smithi115 bash[45495]: audit 2022-04-22T21:38:35.541833+0000 mon.a (mon.0) 12376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:36 smithi115 bash[45495]: cluster 2022-04-22T21:38:35.842697+0000 mgr.y (mgr.34107) 8619 : cluster [DBG] pgmap v9516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:36 smithi121 bash[36902]: audit 2022-04-22T21:38:35.541833+0000 mon.a (mon.0) 12376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:36 smithi121 bash[36902]: cluster 2022-04-22T21:38:35.842697+0000 mgr.y (mgr.34107) 8619 : cluster [DBG] pgmap v9516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:38 smithi121 bash[36902]: cluster 2022-04-22T21:38:37.843818+0000 mgr.y (mgr.34107) 8620 : cluster [DBG] pgmap v9517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:38 smithi115 bash[45495]: cluster 2022-04-22T21:38:37.843818+0000 mgr.y (mgr.34107) 8620 : cluster [DBG] pgmap v9517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:38 smithi115 bash[46934]: cluster 2022-04-22T21:38:37.843818+0000 mgr.y (mgr.34107) 8620 : cluster [DBG] pgmap v9517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:39 smithi121 bash[36902]: audit 2022-04-22T21:38:39.355577+0000 mon.a (mon.0) 12377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:38:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:39 smithi121 bash[36902]: audit 2022-04-22T21:38:39.358726+0000 mon.a (mon.0) 12378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:38:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:39 smithi121 bash[36902]: audit 2022-04-22T21:38:39.360701+0000 mon.a (mon.0) 12379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:38:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:39 smithi121 bash[36902]: audit 2022-04-22T21:38:39.590062+0000 mon.a (mon.0) 12380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:39 smithi121 bash[36902]: audit 2022-04-22T21:38:39.724067+0000 mon.a (mon.0) 12381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:38:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[45495]: audit 2022-04-22T21:38:39.355577+0000 mon.a (mon.0) 12377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:38:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[45495]: audit 2022-04-22T21:38:39.358726+0000 mon.a (mon.0) 12378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:38:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[45495]: audit 2022-04-22T21:38:39.360701+0000 mon.a (mon.0) 12379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:38:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[45495]: audit 2022-04-22T21:38:39.590062+0000 mon.a (mon.0) 12380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[45495]: audit 2022-04-22T21:38:39.724067+0000 mon.a (mon.0) 12381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:38:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[46934]: audit 2022-04-22T21:38:39.355577+0000 mon.a (mon.0) 12377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:38:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[46934]: audit 2022-04-22T21:38:39.358726+0000 mon.a (mon.0) 12378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:38:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[46934]: audit 2022-04-22T21:38:39.360701+0000 mon.a (mon.0) 12379 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:38:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[46934]: audit 2022-04-22T21:38:39.590062+0000 mon.a (mon.0) 12380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:39 smithi115 bash[46934]: audit 2022-04-22T21:38:39.724067+0000 mon.a (mon.0) 12381 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:38:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:40 smithi121 bash[36902]: cluster 2022-04-22T21:38:39.844824+0000 mgr.y (mgr.34107) 8621 : cluster [DBG] pgmap v9518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:40 smithi121 bash[36902]: audit 2022-04-22T21:38:40.545282+0000 mon.a (mon.0) 12382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:40 smithi115 bash[45495]: cluster 2022-04-22T21:38:39.844824+0000 mgr.y (mgr.34107) 8621 : cluster [DBG] pgmap v9518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:40 smithi115 bash[45495]: audit 2022-04-22T21:38:40.545282+0000 mon.a (mon.0) 12382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:40 smithi115 bash[46934]: cluster 2022-04-22T21:38:39.844824+0000 mgr.y (mgr.34107) 8621 : cluster [DBG] pgmap v9518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:40 smithi115 bash[46934]: audit 2022-04-22T21:38:40.545282+0000 mon.a (mon.0) 12382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:42.916 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:42.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:42 smithi121 bash[36902]: cluster 2022-04-22T21:38:41.845888+0000 mgr.y (mgr.34107) 8622 : cluster [DBG] pgmap v9519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:43.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:42 smithi115 bash[46934]: cluster 2022-04-22T21:38:41.845888+0000 mgr.y (mgr.34107) 8622 : cluster [DBG] pgmap v9519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:43.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:42 smithi115 bash[45495]: cluster 2022-04-22T21:38:41.845888+0000 mgr.y (mgr.34107) 8622 : cluster [DBG] pgmap v9519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:44.471Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:44 smithi121 bash[36902]: cluster 2022-04-22T21:38:43.847005+0000 mgr.y (mgr.34107) 8623 : cluster [DBG] pgmap v9520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:44 smithi121 bash[36902]: audit 2022-04-22T21:38:44.593204+0000 mon.a (mon.0) 12383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:44 smithi115 bash[45495]: cluster 2022-04-22T21:38:43.847005+0000 mgr.y (mgr.34107) 8623 : cluster [DBG] pgmap v9520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:44 smithi115 bash[45495]: audit 2022-04-22T21:38:44.593204+0000 mon.a (mon.0) 12383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:44 smithi115 bash[46934]: cluster 2022-04-22T21:38:43.847005+0000 mgr.y (mgr.34107) 8623 : cluster [DBG] pgmap v9520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:44 smithi115 bash[46934]: audit 2022-04-22T21:38:44.593204+0000 mon.a (mon.0) 12383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:46.345 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:38:46.690 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:38:46.745 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:38:46.782 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:38:46.782 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:38:46.783 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:38:46.784 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:38:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:46 smithi121 bash[36902]: audit 2022-04-22T21:38:45.548695+0000 mon.a (mon.0) 12384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:46 smithi121 bash[36902]: cluster 2022-04-22T21:38:45.848117+0000 mgr.y (mgr.34107) 8624 : cluster [DBG] pgmap v9521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:46 smithi115 bash[45495]: audit 2022-04-22T21:38:45.548695+0000 mon.a (mon.0) 12384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:46 smithi115 bash[45495]: cluster 2022-04-22T21:38:45.848117+0000 mgr.y (mgr.34107) 8624 : cluster [DBG] pgmap v9521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:46 smithi115 bash[46934]: audit 2022-04-22T21:38:45.548695+0000 mon.a (mon.0) 12384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:46 smithi115 bash[46934]: cluster 2022-04-22T21:38:45.848117+0000 mgr.y (mgr.34107) 8624 : cluster [DBG] pgmap v9521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:47.091 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:38:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:38:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:38:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:38:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:38:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:38:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:38:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:38:47.094 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:38:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:47 smithi121 bash[36902]: audit 2022-04-22T21:38:46.326466+0000 mgr.y (mgr.34107) 8625 : audit [DBG] from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:47 smithi121 bash[36902]: audit 2022-04-22T21:38:46.668653+0000 mgr.y (mgr.34107) 8626 : audit [DBG] from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:47 smithi121 bash[36902]: audit 2022-04-22T21:38:47.087145+0000 mon.a (mon.0) 12385 : audit [DBG] from='client.? 172.21.15.115:0/3922823868' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[45495]: audit 2022-04-22T21:38:46.326466+0000 mgr.y (mgr.34107) 8625 : audit [DBG] from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[45495]: audit 2022-04-22T21:38:46.668653+0000 mgr.y (mgr.34107) 8626 : audit [DBG] from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[45495]: audit 2022-04-22T21:38:47.087145+0000 mon.a (mon.0) 12385 : audit [DBG] from='client.? 172.21.15.115:0/3922823868' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[46934]: audit 2022-04-22T21:38:46.326466+0000 mgr.y (mgr.34107) 8625 : audit [DBG] from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[46934]: audit 2022-04-22T21:38:46.668653+0000 mgr.y (mgr.34107) 8626 : audit [DBG] from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:38:48.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:47 smithi115 bash[46934]: audit 2022-04-22T21:38:47.087145+0000 mon.a (mon.0) 12385 : audit [DBG] from='client.? 172.21.15.115:0/3922823868' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:38:49.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:48 smithi121 bash[36902]: cluster 2022-04-22T21:38:47.849174+0000 mgr.y (mgr.34107) 8627 : cluster [DBG] pgmap v9522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:48 smithi115 bash[45495]: cluster 2022-04-22T21:38:47.849174+0000 mgr.y (mgr.34107) 8627 : cluster [DBG] pgmap v9522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:48 smithi115 bash[46934]: cluster 2022-04-22T21:38:47.849174+0000 mgr.y (mgr.34107) 8627 : cluster [DBG] pgmap v9522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:49 smithi121 bash[36902]: audit 2022-04-22T21:38:49.596218+0000 mon.a (mon.0) 12386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:49 smithi115 bash[46934]: audit 2022-04-22T21:38:49.596218+0000 mon.a (mon.0) 12386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:49 smithi115 bash[45495]: audit 2022-04-22T21:38:49.596218+0000 mon.a (mon.0) 12386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:50 smithi121 bash[36902]: cluster 2022-04-22T21:38:49.850366+0000 mgr.y (mgr.34107) 8628 : cluster [DBG] pgmap v9523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:50 smithi121 bash[36902]: audit 2022-04-22T21:38:50.552041+0000 mon.a (mon.0) 12387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:50 smithi115 bash[45495]: cluster 2022-04-22T21:38:49.850366+0000 mgr.y (mgr.34107) 8628 : cluster [DBG] pgmap v9523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:50 smithi115 bash[45495]: audit 2022-04-22T21:38:50.552041+0000 mon.a (mon.0) 12387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:50 smithi115 bash[46934]: cluster 2022-04-22T21:38:49.850366+0000 mgr.y (mgr.34107) 8628 : cluster [DBG] pgmap v9523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:50 smithi115 bash[46934]: audit 2022-04-22T21:38:50.552041+0000 mon.a (mon.0) 12387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:52.910 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:52.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:52 smithi121 bash[36902]: cluster 2022-04-22T21:38:51.851463+0000 mgr.y (mgr.34107) 8629 : cluster [DBG] pgmap v9524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:52 smithi115 bash[45495]: cluster 2022-04-22T21:38:51.851463+0000 mgr.y (mgr.34107) 8629 : cluster [DBG] pgmap v9524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:52 smithi115 bash[46934]: cluster 2022-04-22T21:38:51.851463+0000 mgr.y (mgr.34107) 8629 : cluster [DBG] pgmap v9524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:38:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:38:54.471Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:38:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:54 smithi121 bash[36902]: cluster 2022-04-22T21:38:53.852596+0000 mgr.y (mgr.34107) 8630 : cluster [DBG] pgmap v9525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:54 smithi121 bash[36902]: audit 2022-04-22T21:38:54.599362+0000 mon.a (mon.0) 12388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:54 smithi115 bash[45495]: cluster 2022-04-22T21:38:53.852596+0000 mgr.y (mgr.34107) 8630 : cluster [DBG] pgmap v9525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:54 smithi115 bash[45495]: audit 2022-04-22T21:38:54.599362+0000 mon.a (mon.0) 12388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:54 smithi115 bash[46934]: cluster 2022-04-22T21:38:53.852596+0000 mgr.y (mgr.34107) 8630 : cluster [DBG] pgmap v9525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:54 smithi115 bash[46934]: audit 2022-04-22T21:38:54.599362+0000 mon.a (mon.0) 12388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:38:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:56 smithi121 bash[36902]: audit 2022-04-22T21:38:55.555741+0000 mon.a (mon.0) 12389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:56 smithi121 bash[36902]: cluster 2022-04-22T21:38:55.853782+0000 mgr.y (mgr.34107) 8631 : cluster [DBG] pgmap v9526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:56 smithi115 bash[45495]: audit 2022-04-22T21:38:55.555741+0000 mon.a (mon.0) 12389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:57.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:56 smithi115 bash[45495]: cluster 2022-04-22T21:38:55.853782+0000 mgr.y (mgr.34107) 8631 : cluster [DBG] pgmap v9526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:56 smithi115 bash[46934]: audit 2022-04-22T21:38:55.555741+0000 mon.a (mon.0) 12389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:38:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:56 smithi115 bash[46934]: cluster 2022-04-22T21:38:55.853782+0000 mgr.y (mgr.34107) 8631 : cluster [DBG] pgmap v9526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:59.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:58 smithi121 bash[36902]: cluster 2022-04-22T21:38:57.854926+0000 mgr.y (mgr.34107) 8632 : cluster [DBG] pgmap v9527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:58 smithi115 bash[45495]: cluster 2022-04-22T21:38:57.854926+0000 mgr.y (mgr.34107) 8632 : cluster [DBG] pgmap v9527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:38:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:58 smithi115 bash[46934]: cluster 2022-04-22T21:38:57.854926+0000 mgr.y (mgr.34107) 8632 : cluster [DBG] pgmap v9527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:38:59 smithi121 bash[36902]: audit 2022-04-22T21:38:59.602522+0000 mon.a (mon.0) 12390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:00.303 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:38:59 smithi115 bash[45495]: audit 2022-04-22T21:38:59.602522+0000 mon.a (mon.0) 12390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:00.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:38:59 smithi115 bash[46934]: audit 2022-04-22T21:38:59.602522+0000 mon.a (mon.0) 12390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:00 smithi121 bash[36902]: cluster 2022-04-22T21:38:59.856047+0000 mgr.y (mgr.34107) 8633 : cluster [DBG] pgmap v9528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:00 smithi121 bash[36902]: audit 2022-04-22T21:39:00.559604+0000 mon.a (mon.0) 12391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:00 smithi115 bash[45495]: cluster 2022-04-22T21:38:59.856047+0000 mgr.y (mgr.34107) 8633 : cluster [DBG] pgmap v9528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:00 smithi115 bash[45495]: audit 2022-04-22T21:39:00.559604+0000 mon.a (mon.0) 12391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:00 smithi115 bash[46934]: cluster 2022-04-22T21:38:59.856047+0000 mgr.y (mgr.34107) 8633 : cluster [DBG] pgmap v9528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:00 smithi115 bash[46934]: audit 2022-04-22T21:39:00.559604+0000 mon.a (mon.0) 12391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:02.930 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:02.606Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:02 smithi121 bash[36902]: cluster 2022-04-22T21:39:01.857153+0000 mgr.y (mgr.34107) 8634 : cluster [DBG] pgmap v9529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:02 smithi115 bash[45495]: cluster 2022-04-22T21:39:01.857153+0000 mgr.y (mgr.34107) 8634 : cluster [DBG] pgmap v9529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:02 smithi115 bash[46934]: cluster 2022-04-22T21:39:01.857153+0000 mgr.y (mgr.34107) 8634 : cluster [DBG] pgmap v9529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:04.472Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:04 smithi121 bash[36902]: cluster 2022-04-22T21:39:03.858308+0000 mgr.y (mgr.34107) 8635 : cluster [DBG] pgmap v9530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:04 smithi121 bash[36902]: audit 2022-04-22T21:39:04.605655+0000 mon.a (mon.0) 12392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:04 smithi115 bash[45495]: cluster 2022-04-22T21:39:03.858308+0000 mgr.y (mgr.34107) 8635 : cluster [DBG] pgmap v9530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:04 smithi115 bash[45495]: audit 2022-04-22T21:39:04.605655+0000 mon.a (mon.0) 12392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:04 smithi115 bash[46934]: cluster 2022-04-22T21:39:03.858308+0000 mgr.y (mgr.34107) 8635 : cluster [DBG] pgmap v9530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:04 smithi115 bash[46934]: audit 2022-04-22T21:39:04.605655+0000 mon.a (mon.0) 12392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:07.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:06 smithi121 bash[36902]: audit 2022-04-22T21:39:05.563108+0000 mon.a (mon.0) 12393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:06 smithi121 bash[36902]: cluster 2022-04-22T21:39:05.859349+0000 mgr.y (mgr.34107) 8636 : cluster [DBG] pgmap v9531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:06 smithi115 bash[45495]: audit 2022-04-22T21:39:05.563108+0000 mon.a (mon.0) 12393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:06 smithi115 bash[45495]: cluster 2022-04-22T21:39:05.859349+0000 mgr.y (mgr.34107) 8636 : cluster [DBG] pgmap v9531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:06 smithi115 bash[46934]: audit 2022-04-22T21:39:05.563108+0000 mon.a (mon.0) 12393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:06 smithi115 bash[46934]: cluster 2022-04-22T21:39:05.859349+0000 mgr.y (mgr.34107) 8636 : cluster [DBG] pgmap v9531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:09.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:08 smithi121 bash[36902]: cluster 2022-04-22T21:39:07.860389+0000 mgr.y (mgr.34107) 8637 : cluster [DBG] pgmap v9532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:08 smithi115 bash[45495]: cluster 2022-04-22T21:39:07.860389+0000 mgr.y (mgr.34107) 8637 : cluster [DBG] pgmap v9532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:09.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:08 smithi115 bash[46934]: cluster 2022-04-22T21:39:07.860389+0000 mgr.y (mgr.34107) 8637 : cluster [DBG] pgmap v9532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:09 smithi121 bash[36902]: audit 2022-04-22T21:39:09.608780+0000 mon.a (mon.0) 12394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:09 smithi115 bash[45495]: audit 2022-04-22T21:39:09.608780+0000 mon.a (mon.0) 12394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:09 smithi115 bash[46934]: audit 2022-04-22T21:39:09.608780+0000 mon.a (mon.0) 12394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:10 smithi121 bash[36902]: cluster 2022-04-22T21:39:09.861552+0000 mgr.y (mgr.34107) 8638 : cluster [DBG] pgmap v9533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:10 smithi121 bash[36902]: audit 2022-04-22T21:39:10.566786+0000 mon.a (mon.0) 12395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:10 smithi115 bash[45495]: cluster 2022-04-22T21:39:09.861552+0000 mgr.y (mgr.34107) 8638 : cluster [DBG] pgmap v9533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:10 smithi115 bash[45495]: audit 2022-04-22T21:39:10.566786+0000 mon.a (mon.0) 12395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:10 smithi115 bash[46934]: cluster 2022-04-22T21:39:09.861552+0000 mgr.y (mgr.34107) 8638 : cluster [DBG] pgmap v9533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:10 smithi115 bash[46934]: audit 2022-04-22T21:39:10.566786+0000 mon.a (mon.0) 12395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:12.933 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:12.606Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:12 smithi121 bash[36902]: cluster 2022-04-22T21:39:11.862754+0000 mgr.y (mgr.34107) 8639 : cluster [DBG] pgmap v9534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:12 smithi115 bash[45495]: cluster 2022-04-22T21:39:11.862754+0000 mgr.y (mgr.34107) 8639 : cluster [DBG] pgmap v9534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:12 smithi115 bash[46934]: cluster 2022-04-22T21:39:11.862754+0000 mgr.y (mgr.34107) 8639 : cluster [DBG] pgmap v9534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:14.472Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:14 smithi121 bash[36902]: cluster 2022-04-22T21:39:13.863979+0000 mgr.y (mgr.34107) 8640 : cluster [DBG] pgmap v9535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:14 smithi121 bash[36902]: audit 2022-04-22T21:39:14.611995+0000 mon.a (mon.0) 12396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:14 smithi115 bash[45495]: cluster 2022-04-22T21:39:13.863979+0000 mgr.y (mgr.34107) 8640 : cluster [DBG] pgmap v9535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:14 smithi115 bash[45495]: audit 2022-04-22T21:39:14.611995+0000 mon.a (mon.0) 12396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:14 smithi115 bash[46934]: cluster 2022-04-22T21:39:13.863979+0000 mgr.y (mgr.34107) 8640 : cluster [DBG] pgmap v9535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:14 smithi115 bash[46934]: audit 2022-04-22T21:39:14.611995+0000 mon.a (mon.0) 12396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:16 smithi121 bash[36902]: audit 2022-04-22T21:39:15.570291+0000 mon.a (mon.0) 12397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:16 smithi121 bash[36902]: cluster 2022-04-22T21:39:15.864997+0000 mgr.y (mgr.34107) 8641 : cluster [DBG] pgmap v9536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:16 smithi121 bash[36902]: audit 2022-04-22T21:39:16.187363+0000 mon.a (mon.0) 12398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:39:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:16 smithi121 bash[36902]: audit 2022-04-22T21:39:16.187987+0000 mon.a (mon.0) 12399 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:39:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[45495]: audit 2022-04-22T21:39:15.570291+0000 mon.a (mon.0) 12397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[45495]: cluster 2022-04-22T21:39:15.864997+0000 mgr.y (mgr.34107) 8641 : cluster [DBG] pgmap v9536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[45495]: audit 2022-04-22T21:39:16.187363+0000 mon.a (mon.0) 12398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:39:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[45495]: audit 2022-04-22T21:39:16.187987+0000 mon.a (mon.0) 12399 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:39:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[46934]: audit 2022-04-22T21:39:15.570291+0000 mon.a (mon.0) 12397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[46934]: cluster 2022-04-22T21:39:15.864997+0000 mgr.y (mgr.34107) 8641 : cluster [DBG] pgmap v9536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[46934]: audit 2022-04-22T21:39:16.187363+0000 mon.a (mon.0) 12398 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:39:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:16 smithi115 bash[46934]: audit 2022-04-22T21:39:16.187987+0000 mon.a (mon.0) 12399 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:39:17.468 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:39:17.818 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:39:17.819 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:39:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:39:18.216 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:18.217 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:39:18.218 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:39:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:18 smithi121 bash[36902]: audit 2022-04-22T21:39:17.445124+0000 mgr.y (mgr.34107) 8642 : audit [DBG] from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:18 smithi121 bash[36902]: audit 2022-04-22T21:39:17.793895+0000 mgr.y (mgr.34107) 8643 : audit [DBG] from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:18 smithi121 bash[36902]: cluster 2022-04-22T21:39:17.865989+0000 mgr.y (mgr.34107) 8644 : cluster [DBG] pgmap v9537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:18 smithi121 bash[36902]: audit 2022-04-22T21:39:18.212925+0000 mon.a (mon.0) 12400 : audit [DBG] from='client.? 172.21.15.115:0/3525871431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[45495]: audit 2022-04-22T21:39:17.445124+0000 mgr.y (mgr.34107) 8642 : audit [DBG] from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[45495]: audit 2022-04-22T21:39:17.793895+0000 mgr.y (mgr.34107) 8643 : audit [DBG] from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[45495]: cluster 2022-04-22T21:39:17.865989+0000 mgr.y (mgr.34107) 8644 : cluster [DBG] pgmap v9537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[45495]: audit 2022-04-22T21:39:18.212925+0000 mon.a (mon.0) 12400 : audit [DBG] from='client.? 172.21.15.115:0/3525871431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[46934]: audit 2022-04-22T21:39:17.445124+0000 mgr.y (mgr.34107) 8642 : audit [DBG] from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[46934]: audit 2022-04-22T21:39:17.793895+0000 mgr.y (mgr.34107) 8643 : audit [DBG] from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[46934]: cluster 2022-04-22T21:39:17.865989+0000 mgr.y (mgr.34107) 8644 : cluster [DBG] pgmap v9537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:18 smithi115 bash[46934]: audit 2022-04-22T21:39:18.212925+0000 mon.a (mon.0) 12400 : audit [DBG] from='client.? 172.21.15.115:0/3525871431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:19 smithi121 bash[36902]: audit 2022-04-22T21:39:19.615138+0000 mon.a (mon.0) 12401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:19 smithi115 bash[45495]: audit 2022-04-22T21:39:19.615138+0000 mon.a (mon.0) 12401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:19 smithi115 bash[46934]: audit 2022-04-22T21:39:19.615138+0000 mon.a (mon.0) 12401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:20 smithi121 bash[36902]: cluster 2022-04-22T21:39:19.867004+0000 mgr.y (mgr.34107) 8645 : cluster [DBG] pgmap v9538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:20 smithi121 bash[36902]: audit 2022-04-22T21:39:20.574040+0000 mon.a (mon.0) 12402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:20 smithi115 bash[45495]: cluster 2022-04-22T21:39:19.867004+0000 mgr.y (mgr.34107) 8645 : cluster [DBG] pgmap v9538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:20 smithi115 bash[45495]: audit 2022-04-22T21:39:20.574040+0000 mon.a (mon.0) 12402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:20 smithi115 bash[46934]: cluster 2022-04-22T21:39:19.867004+0000 mgr.y (mgr.34107) 8645 : cluster [DBG] pgmap v9538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:20 smithi115 bash[46934]: audit 2022-04-22T21:39:20.574040+0000 mon.a (mon.0) 12402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:22.939 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:22.607Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:22 smithi121 bash[36902]: cluster 2022-04-22T21:39:21.868096+0000 mgr.y (mgr.34107) 8646 : cluster [DBG] pgmap v9539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:22 smithi115 bash[45495]: cluster 2022-04-22T21:39:21.868096+0000 mgr.y (mgr.34107) 8646 : cluster [DBG] pgmap v9539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:22 smithi115 bash[46934]: cluster 2022-04-22T21:39:21.868096+0000 mgr.y (mgr.34107) 8646 : cluster [DBG] pgmap v9539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:24.472Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:24 smithi121 bash[36902]: cluster 2022-04-22T21:39:23.869299+0000 mgr.y (mgr.34107) 8647 : cluster [DBG] pgmap v9540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:24 smithi121 bash[36902]: audit 2022-04-22T21:39:24.618314+0000 mon.a (mon.0) 12403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:24 smithi115 bash[45495]: cluster 2022-04-22T21:39:23.869299+0000 mgr.y (mgr.34107) 8647 : cluster [DBG] pgmap v9540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:25.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:24 smithi115 bash[45495]: audit 2022-04-22T21:39:24.618314+0000 mon.a (mon.0) 12403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:25.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:24 smithi115 bash[46934]: cluster 2022-04-22T21:39:23.869299+0000 mgr.y (mgr.34107) 8647 : cluster [DBG] pgmap v9540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:25.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:24 smithi115 bash[46934]: audit 2022-04-22T21:39:24.618314+0000 mon.a (mon.0) 12403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:26 smithi121 bash[36902]: audit 2022-04-22T21:39:25.577628+0000 mon.a (mon.0) 12404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:26 smithi121 bash[36902]: cluster 2022-04-22T21:39:25.870429+0000 mgr.y (mgr.34107) 8648 : cluster [DBG] pgmap v9541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:26 smithi115 bash[45495]: audit 2022-04-22T21:39:25.577628+0000 mon.a (mon.0) 12404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:26 smithi115 bash[45495]: cluster 2022-04-22T21:39:25.870429+0000 mgr.y (mgr.34107) 8648 : cluster [DBG] pgmap v9541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:26 smithi115 bash[46934]: audit 2022-04-22T21:39:25.577628+0000 mon.a (mon.0) 12404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:26 smithi115 bash[46934]: cluster 2022-04-22T21:39:25.870429+0000 mgr.y (mgr.34107) 8648 : cluster [DBG] pgmap v9541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:28 smithi121 bash[36902]: cluster 2022-04-22T21:39:27.871427+0000 mgr.y (mgr.34107) 8649 : cluster [DBG] pgmap v9542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:28 smithi115 bash[45495]: cluster 2022-04-22T21:39:27.871427+0000 mgr.y (mgr.34107) 8649 : cluster [DBG] pgmap v9542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:28 smithi115 bash[46934]: cluster 2022-04-22T21:39:27.871427+0000 mgr.y (mgr.34107) 8649 : cluster [DBG] pgmap v9542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:29 smithi121 bash[36902]: audit 2022-04-22T21:39:29.621524+0000 mon.a (mon.0) 12405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:29 smithi115 bash[45495]: audit 2022-04-22T21:39:29.621524+0000 mon.a (mon.0) 12405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:29 smithi115 bash[46934]: audit 2022-04-22T21:39:29.621524+0000 mon.a (mon.0) 12405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:30 smithi121 bash[36902]: cluster 2022-04-22T21:39:29.872485+0000 mgr.y (mgr.34107) 8650 : cluster [DBG] pgmap v9543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:30 smithi121 bash[36902]: audit 2022-04-22T21:39:30.581240+0000 mon.a (mon.0) 12406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:30 smithi115 bash[45495]: cluster 2022-04-22T21:39:29.872485+0000 mgr.y (mgr.34107) 8650 : cluster [DBG] pgmap v9543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:30 smithi115 bash[45495]: audit 2022-04-22T21:39:30.581240+0000 mon.a (mon.0) 12406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:30 smithi115 bash[46934]: cluster 2022-04-22T21:39:29.872485+0000 mgr.y (mgr.34107) 8650 : cluster [DBG] pgmap v9543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:30 smithi115 bash[46934]: audit 2022-04-22T21:39:30.581240+0000 mon.a (mon.0) 12406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:32.946 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:32.605Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:32 smithi121 bash[36902]: cluster 2022-04-22T21:39:31.873548+0000 mgr.y (mgr.34107) 8651 : cluster [DBG] pgmap v9544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:32 smithi115 bash[45495]: cluster 2022-04-22T21:39:31.873548+0000 mgr.y (mgr.34107) 8651 : cluster [DBG] pgmap v9544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:32 smithi115 bash[46934]: cluster 2022-04-22T21:39:31.873548+0000 mgr.y (mgr.34107) 8651 : cluster [DBG] pgmap v9544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:34.473Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:34 smithi121 bash[36902]: cluster 2022-04-22T21:39:33.874681+0000 mgr.y (mgr.34107) 8652 : cluster [DBG] pgmap v9545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:34 smithi121 bash[36902]: audit 2022-04-22T21:39:34.624541+0000 mon.a (mon.0) 12407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:34 smithi115 bash[46934]: cluster 2022-04-22T21:39:33.874681+0000 mgr.y (mgr.34107) 8652 : cluster [DBG] pgmap v9545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:34 smithi115 bash[46934]: audit 2022-04-22T21:39:34.624541+0000 mon.a (mon.0) 12407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:34 smithi115 bash[45495]: cluster 2022-04-22T21:39:33.874681+0000 mgr.y (mgr.34107) 8652 : cluster [DBG] pgmap v9545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:34 smithi115 bash[45495]: audit 2022-04-22T21:39:34.624541+0000 mon.a (mon.0) 12407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:36 smithi121 bash[36902]: audit 2022-04-22T21:39:35.584577+0000 mon.a (mon.0) 12408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:36 smithi121 bash[36902]: cluster 2022-04-22T21:39:35.875807+0000 mgr.y (mgr.34107) 8653 : cluster [DBG] pgmap v9546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:36 smithi115 bash[46934]: audit 2022-04-22T21:39:35.584577+0000 mon.a (mon.0) 12408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:36 smithi115 bash[46934]: cluster 2022-04-22T21:39:35.875807+0000 mgr.y (mgr.34107) 8653 : cluster [DBG] pgmap v9546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:36 smithi115 bash[45495]: audit 2022-04-22T21:39:35.584577+0000 mon.a (mon.0) 12408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:36 smithi115 bash[45495]: cluster 2022-04-22T21:39:35.875807+0000 mgr.y (mgr.34107) 8653 : cluster [DBG] pgmap v9546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:38 smithi121 bash[36902]: cluster 2022-04-22T21:39:37.876826+0000 mgr.y (mgr.34107) 8654 : cluster [DBG] pgmap v9547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:38 smithi115 bash[46934]: cluster 2022-04-22T21:39:37.876826+0000 mgr.y (mgr.34107) 8654 : cluster [DBG] pgmap v9547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:38 smithi115 bash[45495]: cluster 2022-04-22T21:39:37.876826+0000 mgr.y (mgr.34107) 8654 : cluster [DBG] pgmap v9547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:39 smithi121 bash[36902]: audit 2022-04-22T21:39:39.627894+0000 mon.a (mon.0) 12409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:39 smithi121 bash[36902]: audit 2022-04-22T21:39:39.731261+0000 mon.a (mon.0) 12410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:39:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:39 smithi121 bash[36902]: audit 2022-04-22T21:39:39.734368+0000 mon.a (mon.0) 12411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:39:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:39 smithi121 bash[36902]: audit 2022-04-22T21:39:39.736336+0000 mon.a (mon.0) 12412 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:39:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[45495]: audit 2022-04-22T21:39:39.627894+0000 mon.a (mon.0) 12409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[45495]: audit 2022-04-22T21:39:39.731261+0000 mon.a (mon.0) 12410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:39:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[45495]: audit 2022-04-22T21:39:39.734368+0000 mon.a (mon.0) 12411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:39:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[45495]: audit 2022-04-22T21:39:39.736336+0000 mon.a (mon.0) 12412 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:39:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[46934]: audit 2022-04-22T21:39:39.627894+0000 mon.a (mon.0) 12409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[46934]: audit 2022-04-22T21:39:39.731261+0000 mon.a (mon.0) 12410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:39:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[46934]: audit 2022-04-22T21:39:39.734368+0000 mon.a (mon.0) 12411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:39:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:39 smithi115 bash[46934]: audit 2022-04-22T21:39:39.736336+0000 mon.a (mon.0) 12412 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:39:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:40 smithi121 bash[36902]: cluster 2022-04-22T21:39:39.877570+0000 mgr.y (mgr.34107) 8655 : cluster [DBG] pgmap v9548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:40 smithi121 bash[36902]: audit 2022-04-22T21:39:40.101159+0000 mon.a (mon.0) 12413 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:39:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:40 smithi121 bash[36902]: audit 2022-04-22T21:39:40.588063+0000 mon.a (mon.0) 12414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[45495]: cluster 2022-04-22T21:39:39.877570+0000 mgr.y (mgr.34107) 8655 : cluster [DBG] pgmap v9548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[45495]: audit 2022-04-22T21:39:40.101159+0000 mon.a (mon.0) 12413 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:39:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[45495]: audit 2022-04-22T21:39:40.588063+0000 mon.a (mon.0) 12414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[46934]: cluster 2022-04-22T21:39:39.877570+0000 mgr.y (mgr.34107) 8655 : cluster [DBG] pgmap v9548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[46934]: audit 2022-04-22T21:39:40.101159+0000 mon.a (mon.0) 12413 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:39:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:40 smithi115 bash[46934]: audit 2022-04-22T21:39:40.588063+0000 mon.a (mon.0) 12414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:42.951 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:42.608Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:42 smithi121 bash[36902]: cluster 2022-04-22T21:39:41.878617+0000 mgr.y (mgr.34107) 8656 : cluster [DBG] pgmap v9549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:42 smithi115 bash[45495]: cluster 2022-04-22T21:39:41.878617+0000 mgr.y (mgr.34107) 8656 : cluster [DBG] pgmap v9549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:42 smithi115 bash[46934]: cluster 2022-04-22T21:39:41.878617+0000 mgr.y (mgr.34107) 8656 : cluster [DBG] pgmap v9549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:44.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:45.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:44 smithi121 bash[36902]: cluster 2022-04-22T21:39:43.879826+0000 mgr.y (mgr.34107) 8657 : cluster [DBG] pgmap v9550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:44 smithi121 bash[36902]: audit 2022-04-22T21:39:44.631084+0000 mon.a (mon.0) 12415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:45.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:44 smithi115 bash[46934]: cluster 2022-04-22T21:39:43.879826+0000 mgr.y (mgr.34107) 8657 : cluster [DBG] pgmap v9550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:44 smithi115 bash[46934]: audit 2022-04-22T21:39:44.631084+0000 mon.a (mon.0) 12415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:44 smithi115 bash[45495]: cluster 2022-04-22T21:39:43.879826+0000 mgr.y (mgr.34107) 8657 : cluster [DBG] pgmap v9550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:44 smithi115 bash[45495]: audit 2022-04-22T21:39:44.631084+0000 mon.a (mon.0) 12415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:46 smithi121 bash[36902]: audit 2022-04-22T21:39:45.591698+0000 mon.a (mon.0) 12416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:46 smithi121 bash[36902]: cluster 2022-04-22T21:39:45.880842+0000 mgr.y (mgr.34107) 8658 : cluster [DBG] pgmap v9551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:46 smithi115 bash[46934]: audit 2022-04-22T21:39:45.591698+0000 mon.a (mon.0) 12416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:46 smithi115 bash[46934]: cluster 2022-04-22T21:39:45.880842+0000 mgr.y (mgr.34107) 8658 : cluster [DBG] pgmap v9551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:46 smithi115 bash[45495]: audit 2022-04-22T21:39:45.591698+0000 mon.a (mon.0) 12416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:46 smithi115 bash[45495]: cluster 2022-04-22T21:39:45.880842+0000 mgr.y (mgr.34107) 8658 : cluster [DBG] pgmap v9551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:48.591 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:39:48.929 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:39:48.930 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:39:48.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:39:48.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:39:48.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:39:48.932 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:39:49.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:48 smithi121 bash[36902]: cluster 2022-04-22T21:39:47.881912+0000 mgr.y (mgr.34107) 8659 : cluster [DBG] pgmap v9552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:49.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:48 smithi115 bash[46934]: cluster 2022-04-22T21:39:47.881912+0000 mgr.y (mgr.34107) 8659 : cluster [DBG] pgmap v9552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:48 smithi115 bash[45495]: cluster 2022-04-22T21:39:47.881912+0000 mgr.y (mgr.34107) 8659 : cluster [DBG] pgmap v9552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:49.333 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:39:49.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:39:49.335 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:39:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:49 smithi121 bash[36902]: audit 2022-04-22T21:39:48.571869+0000 mgr.y (mgr.34107) 8660 : audit [DBG] from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:49 smithi121 bash[36902]: audit 2022-04-22T21:39:48.905882+0000 mgr.y (mgr.34107) 8661 : audit [DBG] from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:49 smithi121 bash[36902]: audit 2022-04-22T21:39:49.330109+0000 mon.a (mon.0) 12417 : audit [DBG] from='client.? 172.21.15.115:0/2935847189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:49 smithi121 bash[36902]: audit 2022-04-22T21:39:49.634331+0000 mon.a (mon.0) 12418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[45495]: audit 2022-04-22T21:39:48.571869+0000 mgr.y (mgr.34107) 8660 : audit [DBG] from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[45495]: audit 2022-04-22T21:39:48.905882+0000 mgr.y (mgr.34107) 8661 : audit [DBG] from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[45495]: audit 2022-04-22T21:39:49.330109+0000 mon.a (mon.0) 12417 : audit [DBG] from='client.? 172.21.15.115:0/2935847189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[45495]: audit 2022-04-22T21:39:49.634331+0000 mon.a (mon.0) 12418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[46934]: audit 2022-04-22T21:39:48.571869+0000 mgr.y (mgr.34107) 8660 : audit [DBG] from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[46934]: audit 2022-04-22T21:39:48.905882+0000 mgr.y (mgr.34107) 8661 : audit [DBG] from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:39:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[46934]: audit 2022-04-22T21:39:49.330109+0000 mon.a (mon.0) 12417 : audit [DBG] from='client.? 172.21.15.115:0/2935847189' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:39:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:49 smithi115 bash[46934]: audit 2022-04-22T21:39:49.634331+0000 mon.a (mon.0) 12418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:50 smithi121 bash[36902]: cluster 2022-04-22T21:39:49.883000+0000 mgr.y (mgr.34107) 8662 : cluster [DBG] pgmap v9553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:50 smithi121 bash[36902]: audit 2022-04-22T21:39:50.595404+0000 mon.a (mon.0) 12419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:50 smithi115 bash[45495]: cluster 2022-04-22T21:39:49.883000+0000 mgr.y (mgr.34107) 8662 : cluster [DBG] pgmap v9553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:50 smithi115 bash[45495]: audit 2022-04-22T21:39:50.595404+0000 mon.a (mon.0) 12419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:50 smithi115 bash[46934]: cluster 2022-04-22T21:39:49.883000+0000 mgr.y (mgr.34107) 8662 : cluster [DBG] pgmap v9553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:50 smithi115 bash[46934]: audit 2022-04-22T21:39:50.595404+0000 mon.a (mon.0) 12419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:52.960 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:52.609Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:53.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:52 smithi121 bash[36902]: cluster 2022-04-22T21:39:51.884116+0000 mgr.y (mgr.34107) 8663 : cluster [DBG] pgmap v9554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:52 smithi115 bash[45495]: cluster 2022-04-22T21:39:51.884116+0000 mgr.y (mgr.34107) 8663 : cluster [DBG] pgmap v9554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:52 smithi115 bash[46934]: cluster 2022-04-22T21:39:51.884116+0000 mgr.y (mgr.34107) 8663 : cluster [DBG] pgmap v9554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:39:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:39:54.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:39:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:54 smithi121 bash[36902]: cluster 2022-04-22T21:39:53.885263+0000 mgr.y (mgr.34107) 8664 : cluster [DBG] pgmap v9555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:54 smithi121 bash[36902]: audit 2022-04-22T21:39:54.637590+0000 mon.a (mon.0) 12420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:55.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:54 smithi115 bash[45495]: cluster 2022-04-22T21:39:53.885263+0000 mgr.y (mgr.34107) 8664 : cluster [DBG] pgmap v9555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:55.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:54 smithi115 bash[45495]: audit 2022-04-22T21:39:54.637590+0000 mon.a (mon.0) 12420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:54 smithi115 bash[46934]: cluster 2022-04-22T21:39:53.885263+0000 mgr.y (mgr.34107) 8664 : cluster [DBG] pgmap v9555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:54 smithi115 bash[46934]: audit 2022-04-22T21:39:54.637590+0000 mon.a (mon.0) 12420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:39:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:56 smithi121 bash[36902]: audit 2022-04-22T21:39:55.599129+0000 mon.a (mon.0) 12421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:56 smithi121 bash[36902]: cluster 2022-04-22T21:39:55.886502+0000 mgr.y (mgr.34107) 8665 : cluster [DBG] pgmap v9556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:56 smithi115 bash[45495]: audit 2022-04-22T21:39:55.599129+0000 mon.a (mon.0) 12421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:56 smithi115 bash[45495]: cluster 2022-04-22T21:39:55.886502+0000 mgr.y (mgr.34107) 8665 : cluster [DBG] pgmap v9556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:56 smithi115 bash[46934]: audit 2022-04-22T21:39:55.599129+0000 mon.a (mon.0) 12421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:39:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:56 smithi115 bash[46934]: cluster 2022-04-22T21:39:55.886502+0000 mgr.y (mgr.34107) 8665 : cluster [DBG] pgmap v9556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:58 smithi121 bash[36902]: cluster 2022-04-22T21:39:57.887712+0000 mgr.y (mgr.34107) 8666 : cluster [DBG] pgmap v9557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:58 smithi115 bash[45495]: cluster 2022-04-22T21:39:57.887712+0000 mgr.y (mgr.34107) 8666 : cluster [DBG] pgmap v9557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:39:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:58 smithi115 bash[46934]: cluster 2022-04-22T21:39:57.887712+0000 mgr.y (mgr.34107) 8666 : cluster [DBG] pgmap v9557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:39:59 smithi121 bash[36902]: audit 2022-04-22T21:39:59.640525+0000 mon.a (mon.0) 12422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:39:59 smithi115 bash[45495]: audit 2022-04-22T21:39:59.640525+0000 mon.a (mon.0) 12422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:39:59 smithi115 bash[46934]: audit 2022-04-22T21:39:59.640525+0000 mon.a (mon.0) 12422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:00 smithi121 bash[36902]: cluster 2022-04-22T21:39:59.888717+0000 mgr.y (mgr.34107) 8667 : cluster [DBG] pgmap v9558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:00 smithi121 bash[36902]: cluster 2022-04-22T21:40:00.000217+0000 mon.a (mon.0) 12423 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:40:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:00 smithi121 bash[36902]: audit 2022-04-22T21:40:00.602761+0000 mon.a (mon.0) 12424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[45495]: cluster 2022-04-22T21:39:59.888717+0000 mgr.y (mgr.34107) 8667 : cluster [DBG] pgmap v9558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[45495]: cluster 2022-04-22T21:40:00.000217+0000 mon.a (mon.0) 12423 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[45495]: audit 2022-04-22T21:40:00.602761+0000 mon.a (mon.0) 12424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[46934]: cluster 2022-04-22T21:39:59.888717+0000 mgr.y (mgr.34107) 8667 : cluster [DBG] pgmap v9558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[46934]: cluster 2022-04-22T21:40:00.000217+0000 mon.a (mon.0) 12423 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:40:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:00 smithi115 bash[46934]: audit 2022-04-22T21:40:00.602761+0000 mon.a (mon.0) 12424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:02.968 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:02.609Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:03.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:02 smithi121 bash[36902]: cluster 2022-04-22T21:40:01.889721+0000 mgr.y (mgr.34107) 8668 : cluster [DBG] pgmap v9559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:02 smithi115 bash[45495]: cluster 2022-04-22T21:40:01.889721+0000 mgr.y (mgr.34107) 8668 : cluster [DBG] pgmap v9559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:02 smithi115 bash[46934]: cluster 2022-04-22T21:40:01.889721+0000 mgr.y (mgr.34107) 8668 : cluster [DBG] pgmap v9559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:04.474Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:04 smithi121 bash[36902]: cluster 2022-04-22T21:40:03.890908+0000 mgr.y (mgr.34107) 8669 : cluster [DBG] pgmap v9560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:04 smithi121 bash[36902]: audit 2022-04-22T21:40:04.643373+0000 mon.a (mon.0) 12425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:04 smithi115 bash[45495]: cluster 2022-04-22T21:40:03.890908+0000 mgr.y (mgr.34107) 8669 : cluster [DBG] pgmap v9560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:04 smithi115 bash[45495]: audit 2022-04-22T21:40:04.643373+0000 mon.a (mon.0) 12425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:04 smithi115 bash[46934]: cluster 2022-04-22T21:40:03.890908+0000 mgr.y (mgr.34107) 8669 : cluster [DBG] pgmap v9560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:04 smithi115 bash[46934]: audit 2022-04-22T21:40:04.643373+0000 mon.a (mon.0) 12425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:06 smithi121 bash[36902]: audit 2022-04-22T21:40:05.606494+0000 mon.a (mon.0) 12426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:06 smithi121 bash[36902]: cluster 2022-04-22T21:40:05.892025+0000 mgr.y (mgr.34107) 8670 : cluster [DBG] pgmap v9561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:06 smithi115 bash[45495]: audit 2022-04-22T21:40:05.606494+0000 mon.a (mon.0) 12426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:06 smithi115 bash[45495]: cluster 2022-04-22T21:40:05.892025+0000 mgr.y (mgr.34107) 8670 : cluster [DBG] pgmap v9561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:06 smithi115 bash[46934]: audit 2022-04-22T21:40:05.606494+0000 mon.a (mon.0) 12426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:06 smithi115 bash[46934]: cluster 2022-04-22T21:40:05.892025+0000 mgr.y (mgr.34107) 8670 : cluster [DBG] pgmap v9561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:08 smithi121 bash[36902]: cluster 2022-04-22T21:40:07.893131+0000 mgr.y (mgr.34107) 8671 : cluster [DBG] pgmap v9562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:08 smithi115 bash[46934]: cluster 2022-04-22T21:40:07.893131+0000 mgr.y (mgr.34107) 8671 : cluster [DBG] pgmap v9562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:08 smithi115 bash[45495]: cluster 2022-04-22T21:40:07.893131+0000 mgr.y (mgr.34107) 8671 : cluster [DBG] pgmap v9562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:09 smithi121 bash[36902]: audit 2022-04-22T21:40:09.646409+0000 mon.a (mon.0) 12427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:09 smithi115 bash[45495]: audit 2022-04-22T21:40:09.646409+0000 mon.a (mon.0) 12427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:09 smithi115 bash[46934]: audit 2022-04-22T21:40:09.646409+0000 mon.a (mon.0) 12427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:10 smithi121 bash[36902]: cluster 2022-04-22T21:40:09.894180+0000 mgr.y (mgr.34107) 8672 : cluster [DBG] pgmap v9563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:10 smithi121 bash[36902]: audit 2022-04-22T21:40:10.609336+0000 mon.a (mon.0) 12428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:10 smithi115 bash[45495]: cluster 2022-04-22T21:40:09.894180+0000 mgr.y (mgr.34107) 8672 : cluster [DBG] pgmap v9563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:10 smithi115 bash[45495]: audit 2022-04-22T21:40:10.609336+0000 mon.a (mon.0) 12428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:10 smithi115 bash[46934]: cluster 2022-04-22T21:40:09.894180+0000 mgr.y (mgr.34107) 8672 : cluster [DBG] pgmap v9563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:10 smithi115 bash[46934]: audit 2022-04-22T21:40:10.609336+0000 mon.a (mon.0) 12428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:12.966 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:12.612Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:12 smithi121 bash[36902]: cluster 2022-04-22T21:40:11.895215+0000 mgr.y (mgr.34107) 8673 : cluster [DBG] pgmap v9564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:12 smithi115 bash[45495]: cluster 2022-04-22T21:40:11.895215+0000 mgr.y (mgr.34107) 8673 : cluster [DBG] pgmap v9564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:13.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:12 smithi115 bash[46934]: cluster 2022-04-22T21:40:11.895215+0000 mgr.y (mgr.34107) 8673 : cluster [DBG] pgmap v9564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:14.476Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:14 smithi121 bash[36902]: cluster 2022-04-22T21:40:13.896448+0000 mgr.y (mgr.34107) 8674 : cluster [DBG] pgmap v9565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:14 smithi121 bash[36902]: audit 2022-04-22T21:40:14.649605+0000 mon.a (mon.0) 12429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:15.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:14 smithi115 bash[46934]: cluster 2022-04-22T21:40:13.896448+0000 mgr.y (mgr.34107) 8674 : cluster [DBG] pgmap v9565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:14 smithi115 bash[46934]: audit 2022-04-22T21:40:14.649605+0000 mon.a (mon.0) 12429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:14 smithi115 bash[45495]: cluster 2022-04-22T21:40:13.896448+0000 mgr.y (mgr.34107) 8674 : cluster [DBG] pgmap v9565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:14 smithi115 bash[45495]: audit 2022-04-22T21:40:14.649605+0000 mon.a (mon.0) 12429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:16 smithi121 bash[36902]: audit 2022-04-22T21:40:15.612866+0000 mon.a (mon.0) 12430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:16 smithi121 bash[36902]: cluster 2022-04-22T21:40:15.897538+0000 mgr.y (mgr.34107) 8675 : cluster [DBG] pgmap v9566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:16 smithi121 bash[36902]: audit 2022-04-22T21:40:16.199172+0000 mon.a (mon.0) 12431 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:16 smithi121 bash[36902]: audit 2022-04-22T21:40:16.199676+0000 mon.a (mon.0) 12432 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[45495]: audit 2022-04-22T21:40:15.612866+0000 mon.a (mon.0) 12430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[45495]: cluster 2022-04-22T21:40:15.897538+0000 mgr.y (mgr.34107) 8675 : cluster [DBG] pgmap v9566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[45495]: audit 2022-04-22T21:40:16.199172+0000 mon.a (mon.0) 12431 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[45495]: audit 2022-04-22T21:40:16.199676+0000 mon.a (mon.0) 12432 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[46934]: audit 2022-04-22T21:40:15.612866+0000 mon.a (mon.0) 12430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[46934]: cluster 2022-04-22T21:40:15.897538+0000 mgr.y (mgr.34107) 8675 : cluster [DBG] pgmap v9566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[46934]: audit 2022-04-22T21:40:16.199172+0000 mon.a (mon.0) 12431 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:40:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:16 smithi115 bash[46934]: audit 2022-04-22T21:40:16.199676+0000 mon.a (mon.0) 12432 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:40:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:18 smithi121 bash[36902]: cluster 2022-04-22T21:40:17.898620+0000 mgr.y (mgr.34107) 8676 : cluster [DBG] pgmap v9567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:18 smithi115 bash[45495]: cluster 2022-04-22T21:40:17.898620+0000 mgr.y (mgr.34107) 8676 : cluster [DBG] pgmap v9567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:18 smithi115 bash[46934]: cluster 2022-04-22T21:40:17.898620+0000 mgr.y (mgr.34107) 8676 : cluster [DBG] pgmap v9567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:19.711 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:40:20.067 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:40:20.067 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:40:20.068 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:40:20.069 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:40:20.070 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:40:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:19 smithi121 bash[36902]: audit 2022-04-22T21:40:19.652536+0000 mon.a (mon.0) 12433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:19 smithi115 bash[45495]: audit 2022-04-22T21:40:19.652536+0000 mon.a (mon.0) 12433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:19 smithi115 bash[46934]: audit 2022-04-22T21:40:19.652536+0000 mon.a (mon.0) 12433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:40:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:40:20.480 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:20 smithi121 bash[36902]: audit 2022-04-22T21:40:19.691955+0000 mgr.y (mgr.34107) 8677 : audit [DBG] from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:20 smithi121 bash[36902]: cluster 2022-04-22T21:40:19.899689+0000 mgr.y (mgr.34107) 8678 : cluster [DBG] pgmap v9568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:20 smithi121 bash[36902]: audit 2022-04-22T21:40:20.044126+0000 mgr.y (mgr.34107) 8679 : audit [DBG] from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:20 smithi121 bash[36902]: audit 2022-04-22T21:40:20.474986+0000 mon.c (mon.1) 307 : audit [DBG] from='client.? 172.21.15.115:0/1710894910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:20 smithi121 bash[36902]: audit 2022-04-22T21:40:20.616516+0000 mon.a (mon.0) 12434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[45495]: audit 2022-04-22T21:40:19.691955+0000 mgr.y (mgr.34107) 8677 : audit [DBG] from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[45495]: cluster 2022-04-22T21:40:19.899689+0000 mgr.y (mgr.34107) 8678 : cluster [DBG] pgmap v9568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[45495]: audit 2022-04-22T21:40:20.044126+0000 mgr.y (mgr.34107) 8679 : audit [DBG] from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[45495]: audit 2022-04-22T21:40:20.474986+0000 mon.c (mon.1) 307 : audit [DBG] from='client.? 172.21.15.115:0/1710894910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[45495]: audit 2022-04-22T21:40:20.616516+0000 mon.a (mon.0) 12434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[46934]: audit 2022-04-22T21:40:19.691955+0000 mgr.y (mgr.34107) 8677 : audit [DBG] from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[46934]: cluster 2022-04-22T21:40:19.899689+0000 mgr.y (mgr.34107) 8678 : cluster [DBG] pgmap v9568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[46934]: audit 2022-04-22T21:40:20.044126+0000 mgr.y (mgr.34107) 8679 : audit [DBG] from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[46934]: audit 2022-04-22T21:40:20.474986+0000 mon.c (mon.1) 307 : audit [DBG] from='client.? 172.21.15.115:0/1710894910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:21.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:20 smithi115 bash[46934]: audit 2022-04-22T21:40:20.616516+0000 mon.a (mon.0) 12434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:22.971 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:22.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:22 smithi121 bash[36902]: cluster 2022-04-22T21:40:21.900883+0000 mgr.y (mgr.34107) 8680 : cluster [DBG] pgmap v9569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:22 smithi115 bash[45495]: cluster 2022-04-22T21:40:21.900883+0000 mgr.y (mgr.34107) 8680 : cluster [DBG] pgmap v9569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:22 smithi115 bash[46934]: cluster 2022-04-22T21:40:21.900883+0000 mgr.y (mgr.34107) 8680 : cluster [DBG] pgmap v9569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:24.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:24 smithi121 bash[36902]: cluster 2022-04-22T21:40:23.902021+0000 mgr.y (mgr.34107) 8681 : cluster [DBG] pgmap v9570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:24 smithi121 bash[36902]: audit 2022-04-22T21:40:24.655543+0000 mon.a (mon.0) 12435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:24 smithi115 bash[45495]: cluster 2022-04-22T21:40:23.902021+0000 mgr.y (mgr.34107) 8681 : cluster [DBG] pgmap v9570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:24 smithi115 bash[45495]: audit 2022-04-22T21:40:24.655543+0000 mon.a (mon.0) 12435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:24 smithi115 bash[46934]: cluster 2022-04-22T21:40:23.902021+0000 mgr.y (mgr.34107) 8681 : cluster [DBG] pgmap v9570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:24 smithi115 bash[46934]: audit 2022-04-22T21:40:24.655543+0000 mon.a (mon.0) 12435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:26 smithi121 bash[36902]: audit 2022-04-22T21:40:25.620055+0000 mon.a (mon.0) 12436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:26 smithi121 bash[36902]: cluster 2022-04-22T21:40:25.903105+0000 mgr.y (mgr.34107) 8682 : cluster [DBG] pgmap v9571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:27.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:26 smithi115 bash[46934]: audit 2022-04-22T21:40:25.620055+0000 mon.a (mon.0) 12436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:27.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:26 smithi115 bash[46934]: cluster 2022-04-22T21:40:25.903105+0000 mgr.y (mgr.34107) 8682 : cluster [DBG] pgmap v9571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:26 smithi115 bash[45495]: audit 2022-04-22T21:40:25.620055+0000 mon.a (mon.0) 12436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:26 smithi115 bash[45495]: cluster 2022-04-22T21:40:25.903105+0000 mgr.y (mgr.34107) 8682 : cluster [DBG] pgmap v9571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:29.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:28 smithi121 bash[36902]: cluster 2022-04-22T21:40:27.904244+0000 mgr.y (mgr.34107) 8683 : cluster [DBG] pgmap v9572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:29.467 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:28 smithi115 bash[45495]: cluster 2022-04-22T21:40:27.904244+0000 mgr.y (mgr.34107) 8683 : cluster [DBG] pgmap v9572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:29.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:28 smithi115 bash[46934]: cluster 2022-04-22T21:40:27.904244+0000 mgr.y (mgr.34107) 8683 : cluster [DBG] pgmap v9572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:29 smithi121 bash[36902]: audit 2022-04-22T21:40:29.658720+0000 mon.a (mon.0) 12437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:29 smithi115 bash[45495]: audit 2022-04-22T21:40:29.658720+0000 mon.a (mon.0) 12437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:29 smithi115 bash[46934]: audit 2022-04-22T21:40:29.658720+0000 mon.a (mon.0) 12437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:30 smithi121 bash[36902]: cluster 2022-04-22T21:40:29.905306+0000 mgr.y (mgr.34107) 8684 : cluster [DBG] pgmap v9573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:30 smithi121 bash[36902]: audit 2022-04-22T21:40:30.623842+0000 mon.a (mon.0) 12438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:31.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:30 smithi115 bash[46934]: cluster 2022-04-22T21:40:29.905306+0000 mgr.y (mgr.34107) 8684 : cluster [DBG] pgmap v9573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:31.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:30 smithi115 bash[46934]: audit 2022-04-22T21:40:30.623842+0000 mon.a (mon.0) 12438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:30 smithi115 bash[45495]: cluster 2022-04-22T21:40:29.905306+0000 mgr.y (mgr.34107) 8684 : cluster [DBG] pgmap v9573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:30 smithi115 bash[45495]: audit 2022-04-22T21:40:30.623842+0000 mon.a (mon.0) 12438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:32.979 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:32.612Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:32 smithi121 bash[36902]: cluster 2022-04-22T21:40:31.906444+0000 mgr.y (mgr.34107) 8685 : cluster [DBG] pgmap v9574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:32 smithi115 bash[45495]: cluster 2022-04-22T21:40:31.906444+0000 mgr.y (mgr.34107) 8685 : cluster [DBG] pgmap v9574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:32 smithi115 bash[46934]: cluster 2022-04-22T21:40:31.906444+0000 mgr.y (mgr.34107) 8685 : cluster [DBG] pgmap v9574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:34.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:34 smithi121 bash[36902]: cluster 2022-04-22T21:40:33.907717+0000 mgr.y (mgr.34107) 8686 : cluster [DBG] pgmap v9575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:34 smithi121 bash[36902]: audit 2022-04-22T21:40:34.661862+0000 mon.a (mon.0) 12439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:34 smithi115 bash[46934]: cluster 2022-04-22T21:40:33.907717+0000 mgr.y (mgr.34107) 8686 : cluster [DBG] pgmap v9575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:34 smithi115 bash[46934]: audit 2022-04-22T21:40:34.661862+0000 mon.a (mon.0) 12439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:34 smithi115 bash[45495]: cluster 2022-04-22T21:40:33.907717+0000 mgr.y (mgr.34107) 8686 : cluster [DBG] pgmap v9575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:34 smithi115 bash[45495]: audit 2022-04-22T21:40:34.661862+0000 mon.a (mon.0) 12439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:36 smithi121 bash[36902]: audit 2022-04-22T21:40:35.627274+0000 mon.a (mon.0) 12440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:36 smithi121 bash[36902]: cluster 2022-04-22T21:40:35.908792+0000 mgr.y (mgr.34107) 8687 : cluster [DBG] pgmap v9576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:36 smithi115 bash[45495]: audit 2022-04-22T21:40:35.627274+0000 mon.a (mon.0) 12440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:36 smithi115 bash[45495]: cluster 2022-04-22T21:40:35.908792+0000 mgr.y (mgr.34107) 8687 : cluster [DBG] pgmap v9576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:36 smithi115 bash[46934]: audit 2022-04-22T21:40:35.627274+0000 mon.a (mon.0) 12440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:36 smithi115 bash[46934]: cluster 2022-04-22T21:40:35.908792+0000 mgr.y (mgr.34107) 8687 : cluster [DBG] pgmap v9576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:38 smithi121 bash[36902]: cluster 2022-04-22T21:40:37.909888+0000 mgr.y (mgr.34107) 8688 : cluster [DBG] pgmap v9577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:38 smithi115 bash[45495]: cluster 2022-04-22T21:40:37.909888+0000 mgr.y (mgr.34107) 8688 : cluster [DBG] pgmap v9577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:38 smithi115 bash[46934]: cluster 2022-04-22T21:40:37.909888+0000 mgr.y (mgr.34107) 8688 : cluster [DBG] pgmap v9577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:39 smithi121 bash[36902]: audit 2022-04-22T21:40:39.664968+0000 mon.a (mon.0) 12441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:39 smithi115 bash[45495]: audit 2022-04-22T21:40:39.664968+0000 mon.a (mon.0) 12441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:39 smithi115 bash[46934]: audit 2022-04-22T21:40:39.664968+0000 mon.a (mon.0) 12441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: cluster 2022-04-22T21:40:39.910955+0000 mgr.y (mgr.34107) 8689 : cluster [DBG] pgmap v9578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:41.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: audit 2022-04-22T21:40:40.110039+0000 mon.a (mon.0) 12442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:40:41.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: audit 2022-04-22T21:40:40.112938+0000 mon.a (mon.0) 12443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:40:41.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: audit 2022-04-22T21:40:40.114713+0000 mon.a (mon.0) 12444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:40:41.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: audit 2022-04-22T21:40:40.483178+0000 mon.a (mon.0) 12445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:40:41.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:40 smithi121 bash[36902]: audit 2022-04-22T21:40:40.631015+0000 mon.a (mon.0) 12446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: cluster 2022-04-22T21:40:39.910955+0000 mgr.y (mgr.34107) 8689 : cluster [DBG] pgmap v9578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: audit 2022-04-22T21:40:40.110039+0000 mon.a (mon.0) 12442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: audit 2022-04-22T21:40:40.112938+0000 mon.a (mon.0) 12443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: audit 2022-04-22T21:40:40.114713+0000 mon.a (mon.0) 12444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: audit 2022-04-22T21:40:40.483178+0000 mon.a (mon.0) 12445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[45495]: audit 2022-04-22T21:40:40.631015+0000 mon.a (mon.0) 12446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: cluster 2022-04-22T21:40:39.910955+0000 mgr.y (mgr.34107) 8689 : cluster [DBG] pgmap v9578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: audit 2022-04-22T21:40:40.110039+0000 mon.a (mon.0) 12442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: audit 2022-04-22T21:40:40.112938+0000 mon.a (mon.0) 12443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: audit 2022-04-22T21:40:40.114713+0000 mon.a (mon.0) 12444 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: audit 2022-04-22T21:40:40.483178+0000 mon.a (mon.0) 12445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:40:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:40 smithi115 bash[46934]: audit 2022-04-22T21:40:40.631015+0000 mon.a (mon.0) 12446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:42.985 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:42.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:42 smithi121 bash[36902]: cluster 2022-04-22T21:40:41.912043+0000 mgr.y (mgr.34107) 8690 : cluster [DBG] pgmap v9579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:42 smithi115 bash[45495]: cluster 2022-04-22T21:40:41.912043+0000 mgr.y (mgr.34107) 8690 : cluster [DBG] pgmap v9579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:42 smithi115 bash[46934]: cluster 2022-04-22T21:40:41.912043+0000 mgr.y (mgr.34107) 8690 : cluster [DBG] pgmap v9579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:44.478Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:44 smithi121 bash[36902]: cluster 2022-04-22T21:40:43.913176+0000 mgr.y (mgr.34107) 8691 : cluster [DBG] pgmap v9580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:44 smithi121 bash[36902]: audit 2022-04-22T21:40:44.668026+0000 mon.a (mon.0) 12447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:45.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:44 smithi115 bash[45495]: cluster 2022-04-22T21:40:43.913176+0000 mgr.y (mgr.34107) 8691 : cluster [DBG] pgmap v9580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:44 smithi115 bash[45495]: audit 2022-04-22T21:40:44.668026+0000 mon.a (mon.0) 12447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:44 smithi115 bash[46934]: cluster 2022-04-22T21:40:43.913176+0000 mgr.y (mgr.34107) 8691 : cluster [DBG] pgmap v9580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:44 smithi115 bash[46934]: audit 2022-04-22T21:40:44.668026+0000 mon.a (mon.0) 12447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:46 smithi121 bash[36902]: audit 2022-04-22T21:40:45.634636+0000 mon.a (mon.0) 12448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:46 smithi121 bash[36902]: cluster 2022-04-22T21:40:45.914217+0000 mgr.y (mgr.34107) 8692 : cluster [DBG] pgmap v9581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:46 smithi115 bash[45495]: audit 2022-04-22T21:40:45.634636+0000 mon.a (mon.0) 12448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:46 smithi115 bash[45495]: cluster 2022-04-22T21:40:45.914217+0000 mgr.y (mgr.34107) 8692 : cluster [DBG] pgmap v9581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:46 smithi115 bash[46934]: audit 2022-04-22T21:40:45.634636+0000 mon.a (mon.0) 12448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:46 smithi115 bash[46934]: cluster 2022-04-22T21:40:45.914217+0000 mgr.y (mgr.34107) 8692 : cluster [DBG] pgmap v9581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:48 smithi121 bash[36902]: cluster 2022-04-22T21:40:47.915352+0000 mgr.y (mgr.34107) 8693 : cluster [DBG] pgmap v9582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:48 smithi115 bash[45495]: cluster 2022-04-22T21:40:47.915352+0000 mgr.y (mgr.34107) 8693 : cluster [DBG] pgmap v9582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:48 smithi115 bash[46934]: cluster 2022-04-22T21:40:47.915352+0000 mgr.y (mgr.34107) 8693 : cluster [DBG] pgmap v9582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:49 smithi121 bash[36902]: audit 2022-04-22T21:40:49.671116+0000 mon.a (mon.0) 12449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:49 smithi115 bash[45495]: audit 2022-04-22T21:40:49.671116+0000 mon.a (mon.0) 12449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:49 smithi115 bash[46934]: audit 2022-04-22T21:40:49.671116+0000 mon.a (mon.0) 12449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:50.859 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:40:51.199 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:40:51.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:40:51.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:40:51.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:40:51.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:40:51.201 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:40:51.201 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:40:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:50 smithi121 bash[36902]: cluster 2022-04-22T21:40:49.916498+0000 mgr.y (mgr.34107) 8694 : cluster [DBG] pgmap v9583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:50 smithi121 bash[36902]: audit 2022-04-22T21:40:50.638202+0000 mon.a (mon.0) 12450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:50 smithi115 bash[45495]: cluster 2022-04-22T21:40:49.916498+0000 mgr.y (mgr.34107) 8694 : cluster [DBG] pgmap v9583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:50 smithi115 bash[45495]: audit 2022-04-22T21:40:50.638202+0000 mon.a (mon.0) 12450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:50 smithi115 bash[46934]: cluster 2022-04-22T21:40:49.916498+0000 mgr.y (mgr.34107) 8694 : cluster [DBG] pgmap v9583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:50 smithi115 bash[46934]: audit 2022-04-22T21:40:50.638202+0000 mon.a (mon.0) 12450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:51.602 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:40:51.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:40:51.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:40:51.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:40:51.604 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:40:51.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:40:51.604 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:40:51.604 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:40:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:51 smithi121 bash[36902]: audit 2022-04-22T21:40:50.837389+0000 mgr.y (mgr.34107) 8695 : audit [DBG] from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:51 smithi121 bash[36902]: audit 2022-04-22T21:40:51.599161+0000 mon.a (mon.0) 12451 : audit [DBG] from='client.? 172.21.15.115:0/961651466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:51 smithi115 bash[45495]: audit 2022-04-22T21:40:50.837389+0000 mgr.y (mgr.34107) 8695 : audit [DBG] from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:51 smithi115 bash[45495]: audit 2022-04-22T21:40:51.599161+0000 mon.a (mon.0) 12451 : audit [DBG] from='client.? 172.21.15.115:0/961651466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:51 smithi115 bash[46934]: audit 2022-04-22T21:40:50.837389+0000 mgr.y (mgr.34107) 8695 : audit [DBG] from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:51 smithi115 bash[46934]: audit 2022-04-22T21:40:51.599161+0000 mon.a (mon.0) 12451 : audit [DBG] from='client.? 172.21.15.115:0/961651466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:40:52.998 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:52.612Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:52 smithi121 bash[36902]: audit 2022-04-22T21:40:51.181388+0000 mgr.y (mgr.34107) 8696 : audit [DBG] from='client.54325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:52 smithi121 bash[36902]: cluster 2022-04-22T21:40:51.917593+0000 mgr.y (mgr.34107) 8697 : cluster [DBG] pgmap v9584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:52 smithi115 bash[45495]: audit 2022-04-22T21:40:51.181388+0000 mgr.y (mgr.34107) 8696 : audit [DBG] from='client.54325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:52 smithi115 bash[45495]: cluster 2022-04-22T21:40:51.917593+0000 mgr.y (mgr.34107) 8697 : cluster [DBG] pgmap v9584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:52 smithi115 bash[46934]: audit 2022-04-22T21:40:51.181388+0000 mgr.y (mgr.34107) 8696 : audit [DBG] from='client.54325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:40:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:52 smithi115 bash[46934]: cluster 2022-04-22T21:40:51.917593+0000 mgr.y (mgr.34107) 8697 : cluster [DBG] pgmap v9584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:40:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:40:54.480Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:40:55.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:55 smithi121 bash[36902]: cluster 2022-04-22T21:40:53.918717+0000 mgr.y (mgr.34107) 8698 : cluster [DBG] pgmap v9585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:55 smithi121 bash[36902]: audit 2022-04-22T21:40:54.674427+0000 mon.a (mon.0) 12452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:55 smithi115 bash[46934]: cluster 2022-04-22T21:40:53.918717+0000 mgr.y (mgr.34107) 8698 : cluster [DBG] pgmap v9585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:55 smithi115 bash[46934]: audit 2022-04-22T21:40:54.674427+0000 mon.a (mon.0) 12452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:54 smithi115 bash[45495]: cluster 2022-04-22T21:40:53.918717+0000 mgr.y (mgr.34107) 8698 : cluster [DBG] pgmap v9585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:54 smithi115 bash[45495]: audit 2022-04-22T21:40:54.674427+0000 mon.a (mon.0) 12452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:40:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:56 smithi121 bash[36902]: audit 2022-04-22T21:40:55.641345+0000 mon.a (mon.0) 12453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:56 smithi121 bash[36902]: cluster 2022-04-22T21:40:55.919815+0000 mgr.y (mgr.34107) 8699 : cluster [DBG] pgmap v9586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:56 smithi115 bash[45495]: audit 2022-04-22T21:40:55.641345+0000 mon.a (mon.0) 12453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:56 smithi115 bash[45495]: cluster 2022-04-22T21:40:55.919815+0000 mgr.y (mgr.34107) 8699 : cluster [DBG] pgmap v9586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:56 smithi115 bash[46934]: audit 2022-04-22T21:40:55.641345+0000 mon.a (mon.0) 12453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:40:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:56 smithi115 bash[46934]: cluster 2022-04-22T21:40:55.919815+0000 mgr.y (mgr.34107) 8699 : cluster [DBG] pgmap v9586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:58 smithi121 bash[36902]: cluster 2022-04-22T21:40:57.920893+0000 mgr.y (mgr.34107) 8700 : cluster [DBG] pgmap v9587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:58 smithi115 bash[45495]: cluster 2022-04-22T21:40:57.920893+0000 mgr.y (mgr.34107) 8700 : cluster [DBG] pgmap v9587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:40:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:58 smithi115 bash[46934]: cluster 2022-04-22T21:40:57.920893+0000 mgr.y (mgr.34107) 8700 : cluster [DBG] pgmap v9587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:40:59 smithi121 bash[36902]: audit 2022-04-22T21:40:59.677565+0000 mon.a (mon.0) 12454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:40:59 smithi115 bash[45495]: audit 2022-04-22T21:40:59.677565+0000 mon.a (mon.0) 12454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:40:59 smithi115 bash[46934]: audit 2022-04-22T21:40:59.677565+0000 mon.a (mon.0) 12454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:00 smithi121 bash[36902]: cluster 2022-04-22T21:40:59.922098+0000 mgr.y (mgr.34107) 8701 : cluster [DBG] pgmap v9588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:00 smithi121 bash[36902]: audit 2022-04-22T21:41:00.645347+0000 mon.a (mon.0) 12455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:00 smithi115 bash[45495]: cluster 2022-04-22T21:40:59.922098+0000 mgr.y (mgr.34107) 8701 : cluster [DBG] pgmap v9588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:00 smithi115 bash[45495]: audit 2022-04-22T21:41:00.645347+0000 mon.a (mon.0) 12455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:00 smithi115 bash[46934]: cluster 2022-04-22T21:40:59.922098+0000 mgr.y (mgr.34107) 8701 : cluster [DBG] pgmap v9588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:00 smithi115 bash[46934]: audit 2022-04-22T21:41:00.645347+0000 mon.a (mon.0) 12455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:02.994 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:02.613Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:02 smithi121 bash[36902]: cluster 2022-04-22T21:41:01.923225+0000 mgr.y (mgr.34107) 8702 : cluster [DBG] pgmap v9589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:02 smithi115 bash[45495]: cluster 2022-04-22T21:41:01.923225+0000 mgr.y (mgr.34107) 8702 : cluster [DBG] pgmap v9589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:02 smithi115 bash[46934]: cluster 2022-04-22T21:41:01.923225+0000 mgr.y (mgr.34107) 8702 : cluster [DBG] pgmap v9589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:04.479Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:04 smithi121 bash[36902]: cluster 2022-04-22T21:41:03.924430+0000 mgr.y (mgr.34107) 8703 : cluster [DBG] pgmap v9590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:04 smithi121 bash[36902]: audit 2022-04-22T21:41:04.680795+0000 mon.a (mon.0) 12456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:04 smithi115 bash[45495]: cluster 2022-04-22T21:41:03.924430+0000 mgr.y (mgr.34107) 8703 : cluster [DBG] pgmap v9590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:04 smithi115 bash[45495]: audit 2022-04-22T21:41:04.680795+0000 mon.a (mon.0) 12456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:04 smithi115 bash[46934]: cluster 2022-04-22T21:41:03.924430+0000 mgr.y (mgr.34107) 8703 : cluster [DBG] pgmap v9590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:04 smithi115 bash[46934]: audit 2022-04-22T21:41:04.680795+0000 mon.a (mon.0) 12456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:06 smithi121 bash[36902]: audit 2022-04-22T21:41:05.648871+0000 mon.a (mon.0) 12457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:06 smithi121 bash[36902]: cluster 2022-04-22T21:41:05.925497+0000 mgr.y (mgr.34107) 8704 : cluster [DBG] pgmap v9591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:06 smithi115 bash[45495]: audit 2022-04-22T21:41:05.648871+0000 mon.a (mon.0) 12457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:06 smithi115 bash[45495]: cluster 2022-04-22T21:41:05.925497+0000 mgr.y (mgr.34107) 8704 : cluster [DBG] pgmap v9591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:06 smithi115 bash[46934]: audit 2022-04-22T21:41:05.648871+0000 mon.a (mon.0) 12457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:06 smithi115 bash[46934]: cluster 2022-04-22T21:41:05.925497+0000 mgr.y (mgr.34107) 8704 : cluster [DBG] pgmap v9591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:09.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:08 smithi121 bash[36902]: cluster 2022-04-22T21:41:07.926592+0000 mgr.y (mgr.34107) 8705 : cluster [DBG] pgmap v9592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:08 smithi115 bash[45495]: cluster 2022-04-22T21:41:07.926592+0000 mgr.y (mgr.34107) 8705 : cluster [DBG] pgmap v9592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:08 smithi115 bash[46934]: cluster 2022-04-22T21:41:07.926592+0000 mgr.y (mgr.34107) 8705 : cluster [DBG] pgmap v9592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:09 smithi121 bash[36902]: audit 2022-04-22T21:41:09.683788+0000 mon.a (mon.0) 12458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:09 smithi115 bash[45495]: audit 2022-04-22T21:41:09.683788+0000 mon.a (mon.0) 12458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:09 smithi115 bash[46934]: audit 2022-04-22T21:41:09.683788+0000 mon.a (mon.0) 12458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:10 smithi121 bash[36902]: cluster 2022-04-22T21:41:09.927602+0000 mgr.y (mgr.34107) 8706 : cluster [DBG] pgmap v9593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:10 smithi121 bash[36902]: audit 2022-04-22T21:41:10.652311+0000 mon.a (mon.0) 12459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:10 smithi115 bash[45495]: cluster 2022-04-22T21:41:09.927602+0000 mgr.y (mgr.34107) 8706 : cluster [DBG] pgmap v9593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:10 smithi115 bash[45495]: audit 2022-04-22T21:41:10.652311+0000 mon.a (mon.0) 12459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:10 smithi115 bash[46934]: cluster 2022-04-22T21:41:09.927602+0000 mgr.y (mgr.34107) 8706 : cluster [DBG] pgmap v9593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:10 smithi115 bash[46934]: audit 2022-04-22T21:41:10.652311+0000 mon.a (mon.0) 12459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:12.999 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:12.614Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:13.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:13 smithi121 bash[36902]: cluster 2022-04-22T21:41:11.928648+0000 mgr.y (mgr.34107) 8707 : cluster [DBG] pgmap v9594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:12 smithi115 bash[45495]: cluster 2022-04-22T21:41:11.928648+0000 mgr.y (mgr.34107) 8707 : cluster [DBG] pgmap v9594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:12 smithi115 bash[46934]: cluster 2022-04-22T21:41:11.928648+0000 mgr.y (mgr.34107) 8707 : cluster [DBG] pgmap v9594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:14.481Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:15 smithi121 bash[36902]: cluster 2022-04-22T21:41:13.929736+0000 mgr.y (mgr.34107) 8708 : cluster [DBG] pgmap v9595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:15 smithi121 bash[36902]: audit 2022-04-22T21:41:14.686227+0000 mon.a (mon.0) 12460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:15 smithi115 bash[45495]: cluster 2022-04-22T21:41:13.929736+0000 mgr.y (mgr.34107) 8708 : cluster [DBG] pgmap v9595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:15 smithi115 bash[45495]: audit 2022-04-22T21:41:14.686227+0000 mon.a (mon.0) 12460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:15 smithi115 bash[46934]: cluster 2022-04-22T21:41:13.929736+0000 mgr.y (mgr.34107) 8708 : cluster [DBG] pgmap v9595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:15 smithi115 bash[46934]: audit 2022-04-22T21:41:14.686227+0000 mon.a (mon.0) 12460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:16 smithi121 bash[36902]: audit 2022-04-22T21:41:15.656162+0000 mon.a (mon.0) 12461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:16 smithi121 bash[36902]: cluster 2022-04-22T21:41:15.930762+0000 mgr.y (mgr.34107) 8709 : cluster [DBG] pgmap v9596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:16 smithi121 bash[36902]: audit 2022-04-22T21:41:16.209521+0000 mon.a (mon.0) 12462 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:41:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:16 smithi121 bash[36902]: audit 2022-04-22T21:41:16.210293+0000 mon.a (mon.0) 12463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[45495]: audit 2022-04-22T21:41:15.656162+0000 mon.a (mon.0) 12461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[45495]: cluster 2022-04-22T21:41:15.930762+0000 mgr.y (mgr.34107) 8709 : cluster [DBG] pgmap v9596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[45495]: audit 2022-04-22T21:41:16.209521+0000 mon.a (mon.0) 12462 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[45495]: audit 2022-04-22T21:41:16.210293+0000 mon.a (mon.0) 12463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[46934]: audit 2022-04-22T21:41:15.656162+0000 mon.a (mon.0) 12461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[46934]: cluster 2022-04-22T21:41:15.930762+0000 mgr.y (mgr.34107) 8709 : cluster [DBG] pgmap v9596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[46934]: audit 2022-04-22T21:41:16.209521+0000 mon.a (mon.0) 12462 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:41:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:16 smithi115 bash[46934]: audit 2022-04-22T21:41:16.210293+0000 mon.a (mon.0) 12463 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:41:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:18 smithi121 bash[36902]: cluster 2022-04-22T21:41:17.931896+0000 mgr.y (mgr.34107) 8710 : cluster [DBG] pgmap v9597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:18 smithi115 bash[45495]: cluster 2022-04-22T21:41:17.931896+0000 mgr.y (mgr.34107) 8710 : cluster [DBG] pgmap v9597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:18 smithi115 bash[46934]: cluster 2022-04-22T21:41:17.931896+0000 mgr.y (mgr.34107) 8710 : cluster [DBG] pgmap v9597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:19 smithi121 bash[36902]: audit 2022-04-22T21:41:19.689253+0000 mon.a (mon.0) 12464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:20.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:19 smithi115 bash[46934]: audit 2022-04-22T21:41:19.689253+0000 mon.a (mon.0) 12464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:20.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:19 smithi115 bash[45495]: audit 2022-04-22T21:41:19.689253+0000 mon.a (mon.0) 12464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:21 smithi121 bash[36902]: cluster 2022-04-22T21:41:19.932887+0000 mgr.y (mgr.34107) 8711 : cluster [DBG] pgmap v9598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:21 smithi121 bash[36902]: audit 2022-04-22T21:41:20.659867+0000 mon.a (mon.0) 12465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:20 smithi115 bash[45495]: cluster 2022-04-22T21:41:19.932887+0000 mgr.y (mgr.34107) 8711 : cluster [DBG] pgmap v9598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:20 smithi115 bash[45495]: audit 2022-04-22T21:41:20.659867+0000 mon.a (mon.0) 12465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:20 smithi115 bash[46934]: cluster 2022-04-22T21:41:19.932887+0000 mgr.y (mgr.34107) 8711 : cluster [DBG] pgmap v9598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:20 smithi115 bash[46934]: audit 2022-04-22T21:41:20.659867+0000 mon.a (mon.0) 12465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:21.977 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:41:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:41:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:41:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:41:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:41:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:41:22.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:41:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:41:22.733 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:41:22.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:41:22.735 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:41:23.007 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:22.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:23.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:23 smithi121 bash[36902]: cluster 2022-04-22T21:41:21.933886+0000 mgr.y (mgr.34107) 8712 : cluster [DBG] pgmap v9599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:23 smithi121 bash[36902]: audit 2022-04-22T21:41:21.956479+0000 mgr.y (mgr.34107) 8713 : audit [DBG] from='client.44406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:23 smithi121 bash[36902]: audit 2022-04-22T21:41:22.729994+0000 mon.c (mon.1) 308 : audit [DBG] from='client.? 172.21.15.115:0/1431568047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[45495]: cluster 2022-04-22T21:41:21.933886+0000 mgr.y (mgr.34107) 8712 : cluster [DBG] pgmap v9599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[45495]: audit 2022-04-22T21:41:21.956479+0000 mgr.y (mgr.34107) 8713 : audit [DBG] from='client.44406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[45495]: audit 2022-04-22T21:41:22.729994+0000 mon.c (mon.1) 308 : audit [DBG] from='client.? 172.21.15.115:0/1431568047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[46934]: cluster 2022-04-22T21:41:21.933886+0000 mgr.y (mgr.34107) 8712 : cluster [DBG] pgmap v9599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[46934]: audit 2022-04-22T21:41:21.956479+0000 mgr.y (mgr.34107) 8713 : audit [DBG] from='client.44406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:23 smithi115 bash[46934]: audit 2022-04-22T21:41:22.729994+0000 mon.c (mon.1) 308 : audit [DBG] from='client.? 172.21.15.115:0/1431568047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:24.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:24 smithi115 bash[46934]: audit 2022-04-22T21:41:22.303049+0000 mgr.y (mgr.34107) 8714 : audit [DBG] from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:24.298 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:24 smithi115 bash[45495]: audit 2022-04-22T21:41:22.303049+0000 mgr.y (mgr.34107) 8714 : audit [DBG] from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:24 smithi121 bash[36902]: audit 2022-04-22T21:41:22.303049+0000 mgr.y (mgr.34107) 8714 : audit [DBG] from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:24.481Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:25 smithi115 bash[45495]: cluster 2022-04-22T21:41:23.935047+0000 mgr.y (mgr.34107) 8715 : cluster [DBG] pgmap v9600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:25 smithi115 bash[45495]: audit 2022-04-22T21:41:24.692407+0000 mon.a (mon.0) 12466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:25 smithi115 bash[46934]: cluster 2022-04-22T21:41:23.935047+0000 mgr.y (mgr.34107) 8715 : cluster [DBG] pgmap v9600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:25 smithi115 bash[46934]: audit 2022-04-22T21:41:24.692407+0000 mon.a (mon.0) 12466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:25 smithi121 bash[36902]: cluster 2022-04-22T21:41:23.935047+0000 mgr.y (mgr.34107) 8715 : cluster [DBG] pgmap v9600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:25 smithi121 bash[36902]: audit 2022-04-22T21:41:24.692407+0000 mon.a (mon.0) 12466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:27.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:26 smithi121 bash[36902]: audit 2022-04-22T21:41:25.663108+0000 mon.a (mon.0) 12467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:26 smithi121 bash[36902]: cluster 2022-04-22T21:41:25.936099+0000 mgr.y (mgr.34107) 8716 : cluster [DBG] pgmap v9601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:26 smithi115 bash[45495]: audit 2022-04-22T21:41:25.663108+0000 mon.a (mon.0) 12467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:27.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:26 smithi115 bash[45495]: cluster 2022-04-22T21:41:25.936099+0000 mgr.y (mgr.34107) 8716 : cluster [DBG] pgmap v9601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:27.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:26 smithi115 bash[46934]: audit 2022-04-22T21:41:25.663108+0000 mon.a (mon.0) 12467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:27.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:26 smithi115 bash[46934]: cluster 2022-04-22T21:41:25.936099+0000 mgr.y (mgr.34107) 8716 : cluster [DBG] pgmap v9601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:28 smithi121 bash[36902]: cluster 2022-04-22T21:41:27.937337+0000 mgr.y (mgr.34107) 8717 : cluster [DBG] pgmap v9602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:28 smithi115 bash[45495]: cluster 2022-04-22T21:41:27.937337+0000 mgr.y (mgr.34107) 8717 : cluster [DBG] pgmap v9602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:28 smithi115 bash[46934]: cluster 2022-04-22T21:41:27.937337+0000 mgr.y (mgr.34107) 8717 : cluster [DBG] pgmap v9602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:30 smithi121 bash[36902]: audit 2022-04-22T21:41:29.695575+0000 mon.a (mon.0) 12468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:30 smithi115 bash[45495]: audit 2022-04-22T21:41:29.695575+0000 mon.a (mon.0) 12468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:30 smithi115 bash[46934]: audit 2022-04-22T21:41:29.695575+0000 mon.a (mon.0) 12468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:31 smithi121 bash[36902]: cluster 2022-04-22T21:41:29.938543+0000 mgr.y (mgr.34107) 8718 : cluster [DBG] pgmap v9603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:31 smithi121 bash[36902]: audit 2022-04-22T21:41:30.666898+0000 mon.a (mon.0) 12469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:31 smithi115 bash[45495]: cluster 2022-04-22T21:41:29.938543+0000 mgr.y (mgr.34107) 8718 : cluster [DBG] pgmap v9603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:31 smithi115 bash[45495]: audit 2022-04-22T21:41:30.666898+0000 mon.a (mon.0) 12469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:31 smithi115 bash[46934]: cluster 2022-04-22T21:41:29.938543+0000 mgr.y (mgr.34107) 8718 : cluster [DBG] pgmap v9603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:31 smithi115 bash[46934]: audit 2022-04-22T21:41:30.666898+0000 mon.a (mon.0) 12469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:33.013 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:32.616Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:33.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:33 smithi121 bash[36902]: cluster 2022-04-22T21:41:31.939816+0000 mgr.y (mgr.34107) 8719 : cluster [DBG] pgmap v9604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:33 smithi115 bash[45495]: cluster 2022-04-22T21:41:31.939816+0000 mgr.y (mgr.34107) 8719 : cluster [DBG] pgmap v9604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:33 smithi115 bash[46934]: cluster 2022-04-22T21:41:31.939816+0000 mgr.y (mgr.34107) 8719 : cluster [DBG] pgmap v9604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:34.481Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:35 smithi115 bash[45495]: cluster 2022-04-22T21:41:33.941050+0000 mgr.y (mgr.34107) 8720 : cluster [DBG] pgmap v9605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:35 smithi115 bash[45495]: audit 2022-04-22T21:41:34.698567+0000 mon.a (mon.0) 12470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:35 smithi115 bash[46934]: cluster 2022-04-22T21:41:33.941050+0000 mgr.y (mgr.34107) 8720 : cluster [DBG] pgmap v9605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:35 smithi115 bash[46934]: audit 2022-04-22T21:41:34.698567+0000 mon.a (mon.0) 12470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:35 smithi121 bash[36902]: cluster 2022-04-22T21:41:33.941050+0000 mgr.y (mgr.34107) 8720 : cluster [DBG] pgmap v9605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:35 smithi121 bash[36902]: audit 2022-04-22T21:41:34.698567+0000 mon.a (mon.0) 12470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:36 smithi121 bash[36902]: audit 2022-04-22T21:41:35.670562+0000 mon.a (mon.0) 12471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:36 smithi121 bash[36902]: cluster 2022-04-22T21:41:35.942220+0000 mgr.y (mgr.34107) 8721 : cluster [DBG] pgmap v9606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:36 smithi115 bash[46934]: audit 2022-04-22T21:41:35.670562+0000 mon.a (mon.0) 12471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:36 smithi115 bash[46934]: cluster 2022-04-22T21:41:35.942220+0000 mgr.y (mgr.34107) 8721 : cluster [DBG] pgmap v9606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:36 smithi115 bash[45495]: audit 2022-04-22T21:41:35.670562+0000 mon.a (mon.0) 12471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:36 smithi115 bash[45495]: cluster 2022-04-22T21:41:35.942220+0000 mgr.y (mgr.34107) 8721 : cluster [DBG] pgmap v9606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:39.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:39 smithi121 bash[36902]: cluster 2022-04-22T21:41:37.943334+0000 mgr.y (mgr.34107) 8722 : cluster [DBG] pgmap v9607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:39 smithi115 bash[46934]: cluster 2022-04-22T21:41:37.943334+0000 mgr.y (mgr.34107) 8722 : cluster [DBG] pgmap v9607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:38 smithi115 bash[45495]: cluster 2022-04-22T21:41:37.943334+0000 mgr.y (mgr.34107) 8722 : cluster [DBG] pgmap v9607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:40.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:40 smithi121 bash[36902]: audit 2022-04-22T21:41:39.701834+0000 mon.a (mon.0) 12472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:40.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:40 smithi115 bash[46934]: audit 2022-04-22T21:41:39.701834+0000 mon.a (mon.0) 12472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:40 smithi115 bash[45495]: audit 2022-04-22T21:41:39.701834+0000 mon.a (mon.0) 12472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: cluster 2022-04-22T21:41:39.944470+0000 mgr.y (mgr.34107) 8723 : cluster [DBG] pgmap v9608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: audit 2022-04-22T21:41:40.492363+0000 mon.a (mon.0) 12473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: audit 2022-04-22T21:41:40.495597+0000 mon.a (mon.0) 12474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: audit 2022-04-22T21:41:40.497393+0000 mon.a (mon.0) 12475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: audit 2022-04-22T21:41:40.674100+0000 mon.a (mon.0) 12476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:41 smithi121 bash[36902]: audit 2022-04-22T21:41:40.849804+0000 mon.a (mon.0) 12477 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: cluster 2022-04-22T21:41:39.944470+0000 mgr.y (mgr.34107) 8723 : cluster [DBG] pgmap v9608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: audit 2022-04-22T21:41:40.492363+0000 mon.a (mon.0) 12473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: audit 2022-04-22T21:41:40.495597+0000 mon.a (mon.0) 12474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: audit 2022-04-22T21:41:40.497393+0000 mon.a (mon.0) 12475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: audit 2022-04-22T21:41:40.674100+0000 mon.a (mon.0) 12476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[45495]: audit 2022-04-22T21:41:40.849804+0000 mon.a (mon.0) 12477 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: cluster 2022-04-22T21:41:39.944470+0000 mgr.y (mgr.34107) 8723 : cluster [DBG] pgmap v9608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: audit 2022-04-22T21:41:40.492363+0000 mon.a (mon.0) 12473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: audit 2022-04-22T21:41:40.495597+0000 mon.a (mon.0) 12474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: audit 2022-04-22T21:41:40.497393+0000 mon.a (mon.0) 12475 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: audit 2022-04-22T21:41:40.674100+0000 mon.a (mon.0) 12476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:41 smithi115 bash[46934]: audit 2022-04-22T21:41:40.849804+0000 mon.a (mon.0) 12477 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:41:43.018 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:42.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:43 smithi115 bash[45495]: cluster 2022-04-22T21:41:41.945560+0000 mgr.y (mgr.34107) 8724 : cluster [DBG] pgmap v9609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:43 smithi115 bash[46934]: cluster 2022-04-22T21:41:41.945560+0000 mgr.y (mgr.34107) 8724 : cluster [DBG] pgmap v9609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:43.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:43 smithi121 bash[36902]: cluster 2022-04-22T21:41:41.945560+0000 mgr.y (mgr.34107) 8724 : cluster [DBG] pgmap v9609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:44.482Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:45.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:45 smithi115 bash[46934]: cluster 2022-04-22T21:41:43.946724+0000 mgr.y (mgr.34107) 8725 : cluster [DBG] pgmap v9610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:45 smithi115 bash[46934]: audit 2022-04-22T21:41:44.704986+0000 mon.a (mon.0) 12478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:45 smithi115 bash[45495]: cluster 2022-04-22T21:41:43.946724+0000 mgr.y (mgr.34107) 8725 : cluster [DBG] pgmap v9610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:45 smithi115 bash[45495]: audit 2022-04-22T21:41:44.704986+0000 mon.a (mon.0) 12478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:45 smithi121 bash[36902]: cluster 2022-04-22T21:41:43.946724+0000 mgr.y (mgr.34107) 8725 : cluster [DBG] pgmap v9610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:45 smithi121 bash[36902]: audit 2022-04-22T21:41:44.704986+0000 mon.a (mon.0) 12478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:46 smithi121 bash[36902]: audit 2022-04-22T21:41:45.677772+0000 mon.a (mon.0) 12479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:46 smithi121 bash[36902]: cluster 2022-04-22T21:41:45.947755+0000 mgr.y (mgr.34107) 8726 : cluster [DBG] pgmap v9611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:46 smithi115 bash[45495]: audit 2022-04-22T21:41:45.677772+0000 mon.a (mon.0) 12479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:46 smithi115 bash[45495]: cluster 2022-04-22T21:41:45.947755+0000 mgr.y (mgr.34107) 8726 : cluster [DBG] pgmap v9611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:46 smithi115 bash[46934]: audit 2022-04-22T21:41:45.677772+0000 mon.a (mon.0) 12479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:46 smithi115 bash[46934]: cluster 2022-04-22T21:41:45.947755+0000 mgr.y (mgr.34107) 8726 : cluster [DBG] pgmap v9611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:49 smithi121 bash[36902]: cluster 2022-04-22T21:41:47.948834+0000 mgr.y (mgr.34107) 8727 : cluster [DBG] pgmap v9612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:49 smithi115 bash[45495]: cluster 2022-04-22T21:41:47.948834+0000 mgr.y (mgr.34107) 8727 : cluster [DBG] pgmap v9612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:49 smithi115 bash[46934]: cluster 2022-04-22T21:41:47.948834+0000 mgr.y (mgr.34107) 8727 : cluster [DBG] pgmap v9612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:50 smithi115 bash[45495]: audit 2022-04-22T21:41:49.707850+0000 mon.a (mon.0) 12480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:50 smithi115 bash[46934]: audit 2022-04-22T21:41:49.707850+0000 mon.a (mon.0) 12480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:50 smithi121 bash[36902]: audit 2022-04-22T21:41:49.707850+0000 mon.a (mon.0) 12480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:51 smithi115 bash[45495]: cluster 2022-04-22T21:41:49.949903+0000 mgr.y (mgr.34107) 8728 : cluster [DBG] pgmap v9613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:51 smithi115 bash[45495]: audit 2022-04-22T21:41:50.681707+0000 mon.a (mon.0) 12481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:51 smithi115 bash[46934]: cluster 2022-04-22T21:41:49.949903+0000 mgr.y (mgr.34107) 8728 : cluster [DBG] pgmap v9613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:51 smithi115 bash[46934]: audit 2022-04-22T21:41:50.681707+0000 mon.a (mon.0) 12481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:51 smithi121 bash[36902]: cluster 2022-04-22T21:41:49.949903+0000 mgr.y (mgr.34107) 8728 : cluster [DBG] pgmap v9613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:51 smithi121 bash[36902]: audit 2022-04-22T21:41:50.681707+0000 mon.a (mon.0) 12481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:53.028 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:52.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:53.106 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:41:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:53 smithi115 bash[46934]: cluster 2022-04-22T21:41:51.950999+0000 mgr.y (mgr.34107) 8729 : cluster [DBG] pgmap v9614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:53 smithi115 bash[45495]: cluster 2022-04-22T21:41:51.950999+0000 mgr.y (mgr.34107) 8729 : cluster [DBG] pgmap v9614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:53.451 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:41:53.451 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:41:53.451 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:41:53.452 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:41:53.453 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:41:53.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:41:53.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:41:53.454 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:41:53.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:53 smithi121 bash[36902]: cluster 2022-04-22T21:41:51.950999+0000 mgr.y (mgr.34107) 8729 : cluster [DBG] pgmap v9614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:53.853 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:41:53.853 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:41:53.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:41:53.854 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:41:53.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:41:53.855 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:41:53.855 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:41:53.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:41:53.855 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:41:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:54 smithi115 bash[45495]: audit 2022-04-22T21:41:53.086898+0000 mgr.y (mgr.34107) 8730 : audit [DBG] from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:54 smithi115 bash[45495]: audit 2022-04-22T21:41:53.849808+0000 mon.c (mon.1) 309 : audit [DBG] from='client.? 172.21.15.115:0/2814916545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:54 smithi115 bash[46934]: audit 2022-04-22T21:41:53.086898+0000 mgr.y (mgr.34107) 8730 : audit [DBG] from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:54 smithi115 bash[46934]: audit 2022-04-22T21:41:53.849808+0000 mon.c (mon.1) 309 : audit [DBG] from='client.? 172.21.15.115:0/2814916545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:54 smithi121 bash[36902]: audit 2022-04-22T21:41:53.086898+0000 mgr.y (mgr.34107) 8730 : audit [DBG] from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:54 smithi121 bash[36902]: audit 2022-04-22T21:41:53.849808+0000 mon.c (mon.1) 309 : audit [DBG] from='client.? 172.21.15.115:0/2814916545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:41:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:41:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:41:54.481Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:41:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[45495]: audit 2022-04-22T21:41:53.426881+0000 mgr.y (mgr.34107) 8731 : audit [DBG] from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[45495]: cluster 2022-04-22T21:41:53.952193+0000 mgr.y (mgr.34107) 8732 : cluster [DBG] pgmap v9615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[45495]: audit 2022-04-22T21:41:54.710780+0000 mon.a (mon.0) 12482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[46934]: audit 2022-04-22T21:41:53.426881+0000 mgr.y (mgr.34107) 8731 : audit [DBG] from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[46934]: cluster 2022-04-22T21:41:53.952193+0000 mgr.y (mgr.34107) 8732 : cluster [DBG] pgmap v9615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:55 smithi115 bash[46934]: audit 2022-04-22T21:41:54.710780+0000 mon.a (mon.0) 12482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:55 smithi121 bash[36902]: audit 2022-04-22T21:41:53.426881+0000 mgr.y (mgr.34107) 8731 : audit [DBG] from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:41:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:55 smithi121 bash[36902]: cluster 2022-04-22T21:41:53.952193+0000 mgr.y (mgr.34107) 8732 : cluster [DBG] pgmap v9615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:55 smithi121 bash[36902]: audit 2022-04-22T21:41:54.710780+0000 mon.a (mon.0) 12482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:41:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:56 smithi121 bash[36902]: audit 2022-04-22T21:41:55.685348+0000 mon.a (mon.0) 12483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:56 smithi121 bash[36902]: cluster 2022-04-22T21:41:55.953314+0000 mgr.y (mgr.34107) 8733 : cluster [DBG] pgmap v9616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:56 smithi115 bash[45495]: audit 2022-04-22T21:41:55.685348+0000 mon.a (mon.0) 12483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:56 smithi115 bash[45495]: cluster 2022-04-22T21:41:55.953314+0000 mgr.y (mgr.34107) 8733 : cluster [DBG] pgmap v9616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:56 smithi115 bash[46934]: audit 2022-04-22T21:41:55.685348+0000 mon.a (mon.0) 12483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:41:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:56 smithi115 bash[46934]: cluster 2022-04-22T21:41:55.953314+0000 mgr.y (mgr.34107) 8733 : cluster [DBG] pgmap v9616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:41:59 smithi115 bash[45495]: cluster 2022-04-22T21:41:57.954594+0000 mgr.y (mgr.34107) 8734 : cluster [DBG] pgmap v9617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:59.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:41:59 smithi115 bash[46934]: cluster 2022-04-22T21:41:57.954594+0000 mgr.y (mgr.34107) 8734 : cluster [DBG] pgmap v9617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:41:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:41:59 smithi121 bash[36902]: cluster 2022-04-22T21:41:57.954594+0000 mgr.y (mgr.34107) 8734 : cluster [DBG] pgmap v9617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:00 smithi115 bash[45495]: audit 2022-04-22T21:41:59.714034+0000 mon.a (mon.0) 12484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:00 smithi115 bash[46934]: audit 2022-04-22T21:41:59.714034+0000 mon.a (mon.0) 12484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:00.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:00 smithi121 bash[36902]: audit 2022-04-22T21:41:59.714034+0000 mon.a (mon.0) 12484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:01 smithi115 bash[45495]: cluster 2022-04-22T21:41:59.955671+0000 mgr.y (mgr.34107) 8735 : cluster [DBG] pgmap v9618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:01 smithi115 bash[45495]: audit 2022-04-22T21:42:00.688824+0000 mon.a (mon.0) 12485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:01 smithi115 bash[46934]: cluster 2022-04-22T21:41:59.955671+0000 mgr.y (mgr.34107) 8735 : cluster [DBG] pgmap v9618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:01 smithi115 bash[46934]: audit 2022-04-22T21:42:00.688824+0000 mon.a (mon.0) 12485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:01 smithi121 bash[36902]: cluster 2022-04-22T21:41:59.955671+0000 mgr.y (mgr.34107) 8735 : cluster [DBG] pgmap v9618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:01 smithi121 bash[36902]: audit 2022-04-22T21:42:00.688824+0000 mon.a (mon.0) 12485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:03.027 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:02.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:03 smithi115 bash[45495]: cluster 2022-04-22T21:42:01.956800+0000 mgr.y (mgr.34107) 8736 : cluster [DBG] pgmap v9619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:03 smithi115 bash[46934]: cluster 2022-04-22T21:42:01.956800+0000 mgr.y (mgr.34107) 8736 : cluster [DBG] pgmap v9619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:03 smithi121 bash[36902]: cluster 2022-04-22T21:42:01.956800+0000 mgr.y (mgr.34107) 8736 : cluster [DBG] pgmap v9619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:04.483Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:05 smithi115 bash[45495]: cluster 2022-04-22T21:42:03.957967+0000 mgr.y (mgr.34107) 8737 : cluster [DBG] pgmap v9620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:05 smithi115 bash[45495]: audit 2022-04-22T21:42:04.717297+0000 mon.a (mon.0) 12486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:05 smithi115 bash[46934]: cluster 2022-04-22T21:42:03.957967+0000 mgr.y (mgr.34107) 8737 : cluster [DBG] pgmap v9620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:05 smithi115 bash[46934]: audit 2022-04-22T21:42:04.717297+0000 mon.a (mon.0) 12486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:05 smithi121 bash[36902]: cluster 2022-04-22T21:42:03.957967+0000 mgr.y (mgr.34107) 8737 : cluster [DBG] pgmap v9620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:05 smithi121 bash[36902]: audit 2022-04-22T21:42:04.717297+0000 mon.a (mon.0) 12486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:06 smithi121 bash[36902]: audit 2022-04-22T21:42:05.692255+0000 mon.a (mon.0) 12487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:06 smithi121 bash[36902]: cluster 2022-04-22T21:42:05.958923+0000 mgr.y (mgr.34107) 8738 : cluster [DBG] pgmap v9621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:06 smithi115 bash[46934]: audit 2022-04-22T21:42:05.692255+0000 mon.a (mon.0) 12487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:06 smithi115 bash[46934]: cluster 2022-04-22T21:42:05.958923+0000 mgr.y (mgr.34107) 8738 : cluster [DBG] pgmap v9621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:06 smithi115 bash[45495]: audit 2022-04-22T21:42:05.692255+0000 mon.a (mon.0) 12487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:06 smithi115 bash[45495]: cluster 2022-04-22T21:42:05.958923+0000 mgr.y (mgr.34107) 8738 : cluster [DBG] pgmap v9621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:09 smithi115 bash[46934]: cluster 2022-04-22T21:42:07.960030+0000 mgr.y (mgr.34107) 8739 : cluster [DBG] pgmap v9622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:09 smithi115 bash[45495]: cluster 2022-04-22T21:42:07.960030+0000 mgr.y (mgr.34107) 8739 : cluster [DBG] pgmap v9622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:09 smithi121 bash[36902]: cluster 2022-04-22T21:42:07.960030+0000 mgr.y (mgr.34107) 8739 : cluster [DBG] pgmap v9622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:10 smithi115 bash[45495]: audit 2022-04-22T21:42:09.720444+0000 mon.a (mon.0) 12488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:10 smithi115 bash[46934]: audit 2022-04-22T21:42:09.720444+0000 mon.a (mon.0) 12488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:10.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:10 smithi121 bash[36902]: audit 2022-04-22T21:42:09.720444+0000 mon.a (mon.0) 12488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:11 smithi115 bash[45495]: cluster 2022-04-22T21:42:09.961107+0000 mgr.y (mgr.34107) 8740 : cluster [DBG] pgmap v9623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:11 smithi115 bash[45495]: audit 2022-04-22T21:42:10.695667+0000 mon.a (mon.0) 12489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:11 smithi115 bash[46934]: cluster 2022-04-22T21:42:09.961107+0000 mgr.y (mgr.34107) 8740 : cluster [DBG] pgmap v9623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:11 smithi115 bash[46934]: audit 2022-04-22T21:42:10.695667+0000 mon.a (mon.0) 12489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:11 smithi121 bash[36902]: cluster 2022-04-22T21:42:09.961107+0000 mgr.y (mgr.34107) 8740 : cluster [DBG] pgmap v9623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:11 smithi121 bash[36902]: audit 2022-04-22T21:42:10.695667+0000 mon.a (mon.0) 12489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:13.033 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:12.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:13 smithi115 bash[45495]: cluster 2022-04-22T21:42:11.962210+0000 mgr.y (mgr.34107) 8741 : cluster [DBG] pgmap v9624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:13 smithi115 bash[46934]: cluster 2022-04-22T21:42:11.962210+0000 mgr.y (mgr.34107) 8741 : cluster [DBG] pgmap v9624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:13.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:13 smithi121 bash[36902]: cluster 2022-04-22T21:42:11.962210+0000 mgr.y (mgr.34107) 8741 : cluster [DBG] pgmap v9624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:14.483Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:15 smithi115 bash[45495]: cluster 2022-04-22T21:42:13.963272+0000 mgr.y (mgr.34107) 8742 : cluster [DBG] pgmap v9625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:15 smithi115 bash[45495]: audit 2022-04-22T21:42:14.723441+0000 mon.a (mon.0) 12490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:15 smithi115 bash[46934]: cluster 2022-04-22T21:42:13.963272+0000 mgr.y (mgr.34107) 8742 : cluster [DBG] pgmap v9625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:15 smithi115 bash[46934]: audit 2022-04-22T21:42:14.723441+0000 mon.a (mon.0) 12490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:15 smithi121 bash[36902]: cluster 2022-04-22T21:42:13.963272+0000 mgr.y (mgr.34107) 8742 : cluster [DBG] pgmap v9625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:15 smithi121 bash[36902]: audit 2022-04-22T21:42:14.723441+0000 mon.a (mon.0) 12490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:16 smithi121 bash[36902]: audit 2022-04-22T21:42:15.699225+0000 mon.a (mon.0) 12491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:16 smithi121 bash[36902]: cluster 2022-04-22T21:42:15.964306+0000 mgr.y (mgr.34107) 8743 : cluster [DBG] pgmap v9626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:16 smithi121 bash[36902]: audit 2022-04-22T21:42:16.221589+0000 mon.a (mon.0) 12492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:42:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:16 smithi121 bash[36902]: audit 2022-04-22T21:42:16.224474+0000 mon.a (mon.0) 12493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:42:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[45495]: audit 2022-04-22T21:42:15.699225+0000 mon.a (mon.0) 12491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[45495]: cluster 2022-04-22T21:42:15.964306+0000 mgr.y (mgr.34107) 8743 : cluster [DBG] pgmap v9626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[45495]: audit 2022-04-22T21:42:16.221589+0000 mon.a (mon.0) 12492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[45495]: audit 2022-04-22T21:42:16.224474+0000 mon.a (mon.0) 12493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[46934]: audit 2022-04-22T21:42:15.699225+0000 mon.a (mon.0) 12491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[46934]: cluster 2022-04-22T21:42:15.964306+0000 mgr.y (mgr.34107) 8743 : cluster [DBG] pgmap v9626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[46934]: audit 2022-04-22T21:42:16.221589+0000 mon.a (mon.0) 12492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:42:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:16 smithi115 bash[46934]: audit 2022-04-22T21:42:16.224474+0000 mon.a (mon.0) 12493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:42:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:19 smithi115 bash[45495]: cluster 2022-04-22T21:42:17.965275+0000 mgr.y (mgr.34107) 8744 : cluster [DBG] pgmap v9627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:19 smithi115 bash[46934]: cluster 2022-04-22T21:42:17.965275+0000 mgr.y (mgr.34107) 8744 : cluster [DBG] pgmap v9627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:19.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:19 smithi121 bash[36902]: cluster 2022-04-22T21:42:17.965275+0000 mgr.y (mgr.34107) 8744 : cluster [DBG] pgmap v9627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:20 smithi115 bash[45495]: audit 2022-04-22T21:42:19.726617+0000 mon.a (mon.0) 12494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:20 smithi115 bash[46934]: audit 2022-04-22T21:42:19.726617+0000 mon.a (mon.0) 12494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:20 smithi121 bash[36902]: audit 2022-04-22T21:42:19.726617+0000 mon.a (mon.0) 12494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:21 smithi115 bash[45495]: cluster 2022-04-22T21:42:19.966325+0000 mgr.y (mgr.34107) 8745 : cluster [DBG] pgmap v9628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:21 smithi115 bash[45495]: audit 2022-04-22T21:42:20.702962+0000 mon.a (mon.0) 12495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:21 smithi115 bash[46934]: cluster 2022-04-22T21:42:19.966325+0000 mgr.y (mgr.34107) 8745 : cluster [DBG] pgmap v9628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:21 smithi115 bash[46934]: audit 2022-04-22T21:42:20.702962+0000 mon.a (mon.0) 12495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:21 smithi121 bash[36902]: cluster 2022-04-22T21:42:19.966325+0000 mgr.y (mgr.34107) 8745 : cluster [DBG] pgmap v9628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:21 smithi121 bash[36902]: audit 2022-04-22T21:42:20.702962+0000 mon.a (mon.0) 12495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:22.630Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:23 smithi121 bash[36902]: cluster 2022-04-22T21:42:21.967282+0000 mgr.y (mgr.34107) 8746 : cluster [DBG] pgmap v9629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:23.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:23 smithi115 bash[45495]: cluster 2022-04-22T21:42:21.967282+0000 mgr.y (mgr.34107) 8746 : cluster [DBG] pgmap v9629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:23.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:23 smithi115 bash[46934]: cluster 2022-04-22T21:42:21.967282+0000 mgr.y (mgr.34107) 8746 : cluster [DBG] pgmap v9629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:24.230 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:42:24.572 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:42:24.573 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:42:24.573 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:42:24.573 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:42:24.573 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:42:24.574 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:42:24.574 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:42:24.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:42:24.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:42:24.575 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:42:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:42:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:42:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:42:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:42:24.576 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:42:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:42:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:42:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:42:24.577 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:42:24.578 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:42:24.578 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:42:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:24.482Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:24.980 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:42:24.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:42:24.982 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:42:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:25 smithi121 bash[36902]: cluster 2022-04-22T21:42:23.968397+0000 mgr.y (mgr.34107) 8747 : cluster [DBG] pgmap v9630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:25 smithi121 bash[36902]: audit 2022-04-22T21:42:24.729785+0000 mon.a (mon.0) 12496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:25 smithi121 bash[36902]: audit 2022-04-22T21:42:24.975776+0000 mon.c (mon.1) 310 : audit [DBG] from='client.? 172.21.15.115:0/1754401809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[45495]: cluster 2022-04-22T21:42:23.968397+0000 mgr.y (mgr.34107) 8747 : cluster [DBG] pgmap v9630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[45495]: audit 2022-04-22T21:42:24.729785+0000 mon.a (mon.0) 12496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[45495]: audit 2022-04-22T21:42:24.975776+0000 mon.c (mon.1) 310 : audit [DBG] from='client.? 172.21.15.115:0/1754401809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[46934]: cluster 2022-04-22T21:42:23.968397+0000 mgr.y (mgr.34107) 8747 : cluster [DBG] pgmap v9630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[46934]: audit 2022-04-22T21:42:24.729785+0000 mon.a (mon.0) 12496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:25 smithi115 bash[46934]: audit 2022-04-22T21:42:24.975776+0000 mon.c (mon.1) 310 : audit [DBG] from='client.? 172.21.15.115:0/1754401809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:26 smithi121 bash[36902]: audit 2022-04-22T21:42:24.208077+0000 mgr.y (mgr.34107) 8748 : audit [DBG] from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:26 smithi121 bash[36902]: audit 2022-04-22T21:42:24.550411+0000 mgr.y (mgr.34107) 8749 : audit [DBG] from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:26 smithi121 bash[36902]: audit 2022-04-22T21:42:25.706599+0000 mon.a (mon.0) 12497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[45495]: audit 2022-04-22T21:42:24.208077+0000 mgr.y (mgr.34107) 8748 : audit [DBG] from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[45495]: audit 2022-04-22T21:42:24.550411+0000 mgr.y (mgr.34107) 8749 : audit [DBG] from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[45495]: audit 2022-04-22T21:42:25.706599+0000 mon.a (mon.0) 12497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[46934]: audit 2022-04-22T21:42:24.208077+0000 mgr.y (mgr.34107) 8748 : audit [DBG] from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[46934]: audit 2022-04-22T21:42:24.550411+0000 mgr.y (mgr.34107) 8749 : audit [DBG] from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:26 smithi115 bash[46934]: audit 2022-04-22T21:42:25.706599+0000 mon.a (mon.0) 12497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:27 smithi121 bash[36902]: cluster 2022-04-22T21:42:25.969476+0000 mgr.y (mgr.34107) 8750 : cluster [DBG] pgmap v9631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:27.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:27 smithi115 bash[46934]: cluster 2022-04-22T21:42:25.969476+0000 mgr.y (mgr.34107) 8750 : cluster [DBG] pgmap v9631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:27 smithi115 bash[45495]: cluster 2022-04-22T21:42:25.969476+0000 mgr.y (mgr.34107) 8750 : cluster [DBG] pgmap v9631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:29 smithi121 bash[36902]: cluster 2022-04-22T21:42:27.970586+0000 mgr.y (mgr.34107) 8751 : cluster [DBG] pgmap v9632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:29.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:29 smithi115 bash[46934]: cluster 2022-04-22T21:42:27.970586+0000 mgr.y (mgr.34107) 8751 : cluster [DBG] pgmap v9632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:29 smithi115 bash[45495]: cluster 2022-04-22T21:42:27.970586+0000 mgr.y (mgr.34107) 8751 : cluster [DBG] pgmap v9632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:30 smithi121 bash[36902]: audit 2022-04-22T21:42:29.732751+0000 mon.a (mon.0) 12498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:30 smithi115 bash[46934]: audit 2022-04-22T21:42:29.732751+0000 mon.a (mon.0) 12498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:30 smithi115 bash[45495]: audit 2022-04-22T21:42:29.732751+0000 mon.a (mon.0) 12498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:31 smithi121 bash[36902]: cluster 2022-04-22T21:42:29.971819+0000 mgr.y (mgr.34107) 8752 : cluster [DBG] pgmap v9633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:31 smithi121 bash[36902]: audit 2022-04-22T21:42:30.710293+0000 mon.a (mon.0) 12499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:31 smithi115 bash[45495]: cluster 2022-04-22T21:42:29.971819+0000 mgr.y (mgr.34107) 8752 : cluster [DBG] pgmap v9633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:31 smithi115 bash[45495]: audit 2022-04-22T21:42:30.710293+0000 mon.a (mon.0) 12499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:31 smithi115 bash[46934]: cluster 2022-04-22T21:42:29.971819+0000 mgr.y (mgr.34107) 8752 : cluster [DBG] pgmap v9633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:31 smithi115 bash[46934]: audit 2022-04-22T21:42:30.710293+0000 mon.a (mon.0) 12499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:32.620Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:33.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:33 smithi121 bash[36902]: cluster 2022-04-22T21:42:31.972924+0000 mgr.y (mgr.34107) 8753 : cluster [DBG] pgmap v9634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:33 smithi115 bash[45495]: cluster 2022-04-22T21:42:31.972924+0000 mgr.y (mgr.34107) 8753 : cluster [DBG] pgmap v9634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:33 smithi115 bash[46934]: cluster 2022-04-22T21:42:31.972924+0000 mgr.y (mgr.34107) 8753 : cluster [DBG] pgmap v9634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:34.485Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:35 smithi121 bash[36902]: cluster 2022-04-22T21:42:33.974077+0000 mgr.y (mgr.34107) 8754 : cluster [DBG] pgmap v9635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:35 smithi121 bash[36902]: audit 2022-04-22T21:42:34.735764+0000 mon.a (mon.0) 12500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:35 smithi115 bash[45495]: cluster 2022-04-22T21:42:33.974077+0000 mgr.y (mgr.34107) 8754 : cluster [DBG] pgmap v9635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:35 smithi115 bash[45495]: audit 2022-04-22T21:42:34.735764+0000 mon.a (mon.0) 12500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:35 smithi115 bash[46934]: cluster 2022-04-22T21:42:33.974077+0000 mgr.y (mgr.34107) 8754 : cluster [DBG] pgmap v9635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:35 smithi115 bash[46934]: audit 2022-04-22T21:42:34.735764+0000 mon.a (mon.0) 12500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:36 smithi121 bash[36902]: audit 2022-04-22T21:42:35.713894+0000 mon.a (mon.0) 12501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:36 smithi121 bash[36902]: cluster 2022-04-22T21:42:35.975121+0000 mgr.y (mgr.34107) 8755 : cluster [DBG] pgmap v9636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:37.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:36 smithi115 bash[46934]: audit 2022-04-22T21:42:35.713894+0000 mon.a (mon.0) 12501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:36 smithi115 bash[46934]: cluster 2022-04-22T21:42:35.975121+0000 mgr.y (mgr.34107) 8755 : cluster [DBG] pgmap v9636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:36 smithi115 bash[45495]: audit 2022-04-22T21:42:35.713894+0000 mon.a (mon.0) 12501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:36 smithi115 bash[45495]: cluster 2022-04-22T21:42:35.975121+0000 mgr.y (mgr.34107) 8755 : cluster [DBG] pgmap v9636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:39 smithi115 bash[45495]: cluster 2022-04-22T21:42:37.976180+0000 mgr.y (mgr.34107) 8756 : cluster [DBG] pgmap v9637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:39 smithi115 bash[46934]: cluster 2022-04-22T21:42:37.976180+0000 mgr.y (mgr.34107) 8756 : cluster [DBG] pgmap v9637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:39 smithi121 bash[36902]: cluster 2022-04-22T21:42:37.976180+0000 mgr.y (mgr.34107) 8756 : cluster [DBG] pgmap v9637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:40.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:40 smithi121 bash[36902]: audit 2022-04-22T21:42:39.738924+0000 mon.a (mon.0) 12502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:40 smithi115 bash[45495]: audit 2022-04-22T21:42:39.738924+0000 mon.a (mon.0) 12502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:40 smithi115 bash[46934]: audit 2022-04-22T21:42:39.738924+0000 mon.a (mon.0) 12502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:41.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:41 smithi121 bash[36902]: cluster 2022-04-22T21:42:39.977306+0000 mgr.y (mgr.34107) 8757 : cluster [DBG] pgmap v9638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:41.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:41 smithi121 bash[36902]: audit 2022-04-22T21:42:40.717634+0000 mon.a (mon.0) 12503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:41.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:41 smithi121 bash[36902]: audit 2022-04-22T21:42:40.858570+0000 mon.a (mon.0) 12504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:42:41.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:41 smithi121 bash[36902]: audit 2022-04-22T21:42:40.861521+0000 mon.a (mon.0) 12505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:42:41.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:41 smithi121 bash[36902]: audit 2022-04-22T21:42:40.863259+0000 mon.a (mon.0) 12506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:42:41.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[46934]: cluster 2022-04-22T21:42:39.977306+0000 mgr.y (mgr.34107) 8757 : cluster [DBG] pgmap v9638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:41.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[46934]: audit 2022-04-22T21:42:40.717634+0000 mon.a (mon.0) 12503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[46934]: audit 2022-04-22T21:42:40.858570+0000 mon.a (mon.0) 12504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[46934]: audit 2022-04-22T21:42:40.861521+0000 mon.a (mon.0) 12505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[46934]: audit 2022-04-22T21:42:40.863259+0000 mon.a (mon.0) 12506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[45495]: cluster 2022-04-22T21:42:39.977306+0000 mgr.y (mgr.34107) 8757 : cluster [DBG] pgmap v9638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[45495]: audit 2022-04-22T21:42:40.717634+0000 mon.a (mon.0) 12503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[45495]: audit 2022-04-22T21:42:40.858570+0000 mon.a (mon.0) 12504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[45495]: audit 2022-04-22T21:42:40.861521+0000 mon.a (mon.0) 12505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:42:41.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:41 smithi115 bash[45495]: audit 2022-04-22T21:42:40.863259+0000 mon.a (mon.0) 12506 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:42:42.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:42 smithi121 bash[36902]: audit 2022-04-22T21:42:41.489243+0000 mon.a (mon.0) 12507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:42 smithi121 bash[36902]: cluster 2022-04-22T21:42:41.978369+0000 mgr.y (mgr.34107) 8758 : cluster [DBG] pgmap v9639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:42.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:42 smithi115 bash[45495]: audit 2022-04-22T21:42:41.489243+0000 mon.a (mon.0) 12507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:42.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:42 smithi115 bash[45495]: cluster 2022-04-22T21:42:41.978369+0000 mgr.y (mgr.34107) 8758 : cluster [DBG] pgmap v9639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:42.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:42.619Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:42 smithi115 bash[46934]: audit 2022-04-22T21:42:41.489243+0000 mon.a (mon.0) 12507 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:42 smithi115 bash[46934]: cluster 2022-04-22T21:42:41.978369+0000 mgr.y (mgr.34107) 8758 : cluster [DBG] pgmap v9639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:44.487Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:45.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:45 smithi115 bash[45495]: cluster 2022-04-22T21:42:43.979556+0000 mgr.y (mgr.34107) 8759 : cluster [DBG] pgmap v9640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:45 smithi115 bash[45495]: audit 2022-04-22T21:42:44.742023+0000 mon.a (mon.0) 12508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:45 smithi115 bash[46934]: cluster 2022-04-22T21:42:43.979556+0000 mgr.y (mgr.34107) 8759 : cluster [DBG] pgmap v9640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:45 smithi115 bash[46934]: audit 2022-04-22T21:42:44.742023+0000 mon.a (mon.0) 12508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:45 smithi121 bash[36902]: cluster 2022-04-22T21:42:43.979556+0000 mgr.y (mgr.34107) 8759 : cluster [DBG] pgmap v9640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:45 smithi121 bash[36902]: audit 2022-04-22T21:42:44.742023+0000 mon.a (mon.0) 12508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:46 smithi121 bash[36902]: audit 2022-04-22T21:42:45.721172+0000 mon.a (mon.0) 12509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:46 smithi121 bash[36902]: cluster 2022-04-22T21:42:45.980561+0000 mgr.y (mgr.34107) 8760 : cluster [DBG] pgmap v9641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:46 smithi115 bash[45495]: audit 2022-04-22T21:42:45.721172+0000 mon.a (mon.0) 12509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:46 smithi115 bash[45495]: cluster 2022-04-22T21:42:45.980561+0000 mgr.y (mgr.34107) 8760 : cluster [DBG] pgmap v9641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:46 smithi115 bash[46934]: audit 2022-04-22T21:42:45.721172+0000 mon.a (mon.0) 12509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:46 smithi115 bash[46934]: cluster 2022-04-22T21:42:45.980561+0000 mgr.y (mgr.34107) 8760 : cluster [DBG] pgmap v9641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:49 smithi115 bash[45495]: cluster 2022-04-22T21:42:47.981601+0000 mgr.y (mgr.34107) 8761 : cluster [DBG] pgmap v9642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:49.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:49 smithi115 bash[46934]: cluster 2022-04-22T21:42:47.981601+0000 mgr.y (mgr.34107) 8761 : cluster [DBG] pgmap v9642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:49.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:49 smithi121 bash[36902]: cluster 2022-04-22T21:42:47.981601+0000 mgr.y (mgr.34107) 8761 : cluster [DBG] pgmap v9642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:50 smithi121 bash[36902]: audit 2022-04-22T21:42:49.744649+0000 mon.a (mon.0) 12510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:50 smithi115 bash[45495]: audit 2022-04-22T21:42:49.744649+0000 mon.a (mon.0) 12510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:50 smithi115 bash[46934]: audit 2022-04-22T21:42:49.744649+0000 mon.a (mon.0) 12510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:51 smithi121 bash[36902]: cluster 2022-04-22T21:42:49.982751+0000 mgr.y (mgr.34107) 8762 : cluster [DBG] pgmap v9643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:51 smithi121 bash[36902]: audit 2022-04-22T21:42:50.301191+0000 mon.a (mon.0) 12511 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:51 smithi121 bash[36902]: audit 2022-04-22T21:42:50.310039+0000 mon.a (mon.0) 12512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:51 smithi121 bash[36902]: audit 2022-04-22T21:42:50.725004+0000 mon.a (mon.0) 12513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[45495]: cluster 2022-04-22T21:42:49.982751+0000 mgr.y (mgr.34107) 8762 : cluster [DBG] pgmap v9643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[45495]: audit 2022-04-22T21:42:50.301191+0000 mon.a (mon.0) 12511 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[45495]: audit 2022-04-22T21:42:50.310039+0000 mon.a (mon.0) 12512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[45495]: audit 2022-04-22T21:42:50.725004+0000 mon.a (mon.0) 12513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[46934]: cluster 2022-04-22T21:42:49.982751+0000 mgr.y (mgr.34107) 8762 : cluster [DBG] pgmap v9643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[46934]: audit 2022-04-22T21:42:50.301191+0000 mon.a (mon.0) 12511 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[46934]: audit 2022-04-22T21:42:50.310039+0000 mon.a (mon.0) 12512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:42:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:51 smithi115 bash[46934]: audit 2022-04-22T21:42:50.725004+0000 mon.a (mon.0) 12513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:52 smithi121 bash[36902]: cephadm 2022-04-22T21:42:50.290470+0000 mgr.y (mgr.34107) 8763 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:42:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:52 smithi115 bash[45495]: cephadm 2022-04-22T21:42:50.290470+0000 mgr.y (mgr.34107) 8763 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:42:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:52 smithi115 bash[46934]: cephadm 2022-04-22T21:42:50.290470+0000 mgr.y (mgr.34107) 8763 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T21:42:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:52.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:53.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:53 smithi121 bash[36902]: cluster 2022-04-22T21:42:51.983839+0000 mgr.y (mgr.34107) 8764 : cluster [DBG] pgmap v9644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:53.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:53 smithi115 bash[45495]: cluster 2022-04-22T21:42:51.983839+0000 mgr.y (mgr.34107) 8764 : cluster [DBG] pgmap v9644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:53.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:53 smithi115 bash[46934]: cluster 2022-04-22T21:42:51.983839+0000 mgr.y (mgr.34107) 8764 : cluster [DBG] pgmap v9644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:42:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:42:54.487Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:42:55.359 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:42:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:55 smithi121 bash[36902]: cluster 2022-04-22T21:42:53.985001+0000 mgr.y (mgr.34107) 8765 : cluster [DBG] pgmap v9645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:55 smithi121 bash[36902]: audit 2022-04-22T21:42:54.747620+0000 mon.a (mon.0) 12514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:55 smithi115 bash[45495]: cluster 2022-04-22T21:42:53.985001+0000 mgr.y (mgr.34107) 8765 : cluster [DBG] pgmap v9645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:55 smithi115 bash[45495]: audit 2022-04-22T21:42:54.747620+0000 mon.a (mon.0) 12514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:55 smithi115 bash[46934]: cluster 2022-04-22T21:42:53.985001+0000 mgr.y (mgr.34107) 8765 : cluster [DBG] pgmap v9645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:55 smithi115 bash[46934]: audit 2022-04-22T21:42:54.747620+0000 mon.a (mon.0) 12514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:42:55.704 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:42:55.705 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:42:55.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:42:55.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:42:55.706 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:42:55.706 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:42:56.106 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:42:56.106 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:42:56.107 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:42:56.108 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:42:56.108 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:42:56.108 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:42:56.108 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:42:56.108 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:42:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:56 smithi121 bash[36902]: audit 2022-04-22T21:42:55.339061+0000 mgr.y (mgr.34107) 8766 : audit [DBG] from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:56 smithi121 bash[36902]: audit 2022-04-22T21:42:55.682036+0000 mgr.y (mgr.34107) 8767 : audit [DBG] from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:56 smithi121 bash[36902]: audit 2022-04-22T21:42:55.728518+0000 mon.a (mon.0) 12515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:56 smithi121 bash[36902]: cluster 2022-04-22T21:42:55.986012+0000 mgr.y (mgr.34107) 8768 : cluster [DBG] pgmap v9646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:56 smithi121 bash[36902]: audit 2022-04-22T21:42:56.102853+0000 mon.c (mon.1) 311 : audit [DBG] from='client.? 172.21.15.115:0/1427488676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[45495]: audit 2022-04-22T21:42:55.339061+0000 mgr.y (mgr.34107) 8766 : audit [DBG] from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[45495]: audit 2022-04-22T21:42:55.682036+0000 mgr.y (mgr.34107) 8767 : audit [DBG] from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[45495]: audit 2022-04-22T21:42:55.728518+0000 mon.a (mon.0) 12515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[45495]: cluster 2022-04-22T21:42:55.986012+0000 mgr.y (mgr.34107) 8768 : cluster [DBG] pgmap v9646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[45495]: audit 2022-04-22T21:42:56.102853+0000 mon.c (mon.1) 311 : audit [DBG] from='client.? 172.21.15.115:0/1427488676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[46934]: audit 2022-04-22T21:42:55.339061+0000 mgr.y (mgr.34107) 8766 : audit [DBG] from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[46934]: audit 2022-04-22T21:42:55.682036+0000 mgr.y (mgr.34107) 8767 : audit [DBG] from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:42:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[46934]: audit 2022-04-22T21:42:55.728518+0000 mon.a (mon.0) 12515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:42:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[46934]: cluster 2022-04-22T21:42:55.986012+0000 mgr.y (mgr.34107) 8768 : cluster [DBG] pgmap v9646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:56 smithi115 bash[46934]: audit 2022-04-22T21:42:56.102853+0000 mon.c (mon.1) 311 : audit [DBG] from='client.? 172.21.15.115:0/1427488676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:42:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:42:59 smithi121 bash[36902]: cluster 2022-04-22T21:42:57.986907+0000 mgr.y (mgr.34107) 8769 : cluster [DBG] pgmap v9647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:42:59 smithi115 bash[45495]: cluster 2022-04-22T21:42:57.986907+0000 mgr.y (mgr.34107) 8769 : cluster [DBG] pgmap v9647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:42:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:42:59 smithi115 bash[46934]: cluster 2022-04-22T21:42:57.986907+0000 mgr.y (mgr.34107) 8769 : cluster [DBG] pgmap v9647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:00 smithi121 bash[36902]: audit 2022-04-22T21:42:59.750862+0000 mon.a (mon.0) 12516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:00 smithi115 bash[45495]: audit 2022-04-22T21:42:59.750862+0000 mon.a (mon.0) 12516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:00 smithi115 bash[46934]: audit 2022-04-22T21:42:59.750862+0000 mon.a (mon.0) 12516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:01 smithi121 bash[36902]: cluster 2022-04-22T21:42:59.988035+0000 mgr.y (mgr.34107) 8770 : cluster [DBG] pgmap v9648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:01 smithi121 bash[36902]: audit 2022-04-22T21:43:00.731848+0000 mon.a (mon.0) 12517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:01 smithi115 bash[45495]: cluster 2022-04-22T21:42:59.988035+0000 mgr.y (mgr.34107) 8770 : cluster [DBG] pgmap v9648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:01 smithi115 bash[45495]: audit 2022-04-22T21:43:00.731848+0000 mon.a (mon.0) 12517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:01 smithi115 bash[46934]: cluster 2022-04-22T21:42:59.988035+0000 mgr.y (mgr.34107) 8770 : cluster [DBG] pgmap v9648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:01 smithi115 bash[46934]: audit 2022-04-22T21:43:00.731848+0000 mon.a (mon.0) 12517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:02.622Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:03.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:03 smithi121 bash[36902]: cluster 2022-04-22T21:43:01.989058+0000 mgr.y (mgr.34107) 8771 : cluster [DBG] pgmap v9649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:03 smithi115 bash[45495]: cluster 2022-04-22T21:43:01.989058+0000 mgr.y (mgr.34107) 8771 : cluster [DBG] pgmap v9649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:03 smithi115 bash[46934]: cluster 2022-04-22T21:43:01.989058+0000 mgr.y (mgr.34107) 8771 : cluster [DBG] pgmap v9649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:04.487Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:05 smithi121 bash[36902]: cluster 2022-04-22T21:43:03.990299+0000 mgr.y (mgr.34107) 8772 : cluster [DBG] pgmap v9650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:05 smithi121 bash[36902]: audit 2022-04-22T21:43:04.753736+0000 mon.a (mon.0) 12518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:05 smithi115 bash[45495]: cluster 2022-04-22T21:43:03.990299+0000 mgr.y (mgr.34107) 8772 : cluster [DBG] pgmap v9650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:05 smithi115 bash[45495]: audit 2022-04-22T21:43:04.753736+0000 mon.a (mon.0) 12518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:05 smithi115 bash[46934]: cluster 2022-04-22T21:43:03.990299+0000 mgr.y (mgr.34107) 8772 : cluster [DBG] pgmap v9650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:05 smithi115 bash[46934]: audit 2022-04-22T21:43:04.753736+0000 mon.a (mon.0) 12518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:06 smithi121 bash[36902]: audit 2022-04-22T21:43:05.736113+0000 mon.a (mon.0) 12519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:06 smithi121 bash[36902]: cluster 2022-04-22T21:43:05.991358+0000 mgr.y (mgr.34107) 8773 : cluster [DBG] pgmap v9651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:06 smithi115 bash[45495]: audit 2022-04-22T21:43:05.736113+0000 mon.a (mon.0) 12519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:06 smithi115 bash[45495]: cluster 2022-04-22T21:43:05.991358+0000 mgr.y (mgr.34107) 8773 : cluster [DBG] pgmap v9651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:06 smithi115 bash[46934]: audit 2022-04-22T21:43:05.736113+0000 mon.a (mon.0) 12519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:06 smithi115 bash[46934]: cluster 2022-04-22T21:43:05.991358+0000 mgr.y (mgr.34107) 8773 : cluster [DBG] pgmap v9651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:09 smithi121 bash[36902]: cluster 2022-04-22T21:43:07.992400+0000 mgr.y (mgr.34107) 8774 : cluster [DBG] pgmap v9652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:09 smithi115 bash[45495]: cluster 2022-04-22T21:43:07.992400+0000 mgr.y (mgr.34107) 8774 : cluster [DBG] pgmap v9652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:09 smithi115 bash[46934]: cluster 2022-04-22T21:43:07.992400+0000 mgr.y (mgr.34107) 8774 : cluster [DBG] pgmap v9652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:10 smithi121 bash[36902]: audit 2022-04-22T21:43:09.756918+0000 mon.a (mon.0) 12520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:10 smithi115 bash[45495]: audit 2022-04-22T21:43:09.756918+0000 mon.a (mon.0) 12520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:10 smithi115 bash[46934]: audit 2022-04-22T21:43:09.756918+0000 mon.a (mon.0) 12520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:11 smithi121 bash[36902]: cluster 2022-04-22T21:43:09.993449+0000 mgr.y (mgr.34107) 8775 : cluster [DBG] pgmap v9653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:11 smithi121 bash[36902]: audit 2022-04-22T21:43:10.740063+0000 mon.a (mon.0) 12521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:11 smithi115 bash[45495]: cluster 2022-04-22T21:43:09.993449+0000 mgr.y (mgr.34107) 8775 : cluster [DBG] pgmap v9653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:11 smithi115 bash[45495]: audit 2022-04-22T21:43:10.740063+0000 mon.a (mon.0) 12521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:11 smithi115 bash[46934]: cluster 2022-04-22T21:43:09.993449+0000 mgr.y (mgr.34107) 8775 : cluster [DBG] pgmap v9653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:11 smithi115 bash[46934]: audit 2022-04-22T21:43:10.740063+0000 mon.a (mon.0) 12521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:12.622Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:13 smithi121 bash[36902]: cluster 2022-04-22T21:43:11.994523+0000 mgr.y (mgr.34107) 8776 : cluster [DBG] pgmap v9654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:13.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:13 smithi115 bash[45495]: cluster 2022-04-22T21:43:11.994523+0000 mgr.y (mgr.34107) 8776 : cluster [DBG] pgmap v9654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:13.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:13 smithi115 bash[46934]: cluster 2022-04-22T21:43:11.994523+0000 mgr.y (mgr.34107) 8776 : cluster [DBG] pgmap v9654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:14.489Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:15 smithi121 bash[36902]: cluster 2022-04-22T21:43:13.995777+0000 mgr.y (mgr.34107) 8777 : cluster [DBG] pgmap v9655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:15 smithi121 bash[36902]: audit 2022-04-22T21:43:14.760022+0000 mon.a (mon.0) 12522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:15 smithi115 bash[45495]: cluster 2022-04-22T21:43:13.995777+0000 mgr.y (mgr.34107) 8777 : cluster [DBG] pgmap v9655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:15 smithi115 bash[45495]: audit 2022-04-22T21:43:14.760022+0000 mon.a (mon.0) 12522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:15 smithi115 bash[46934]: cluster 2022-04-22T21:43:13.995777+0000 mgr.y (mgr.34107) 8777 : cluster [DBG] pgmap v9655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:15 smithi115 bash[46934]: audit 2022-04-22T21:43:14.760022+0000 mon.a (mon.0) 12522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:16 smithi121 bash[36902]: audit 2022-04-22T21:43:15.743686+0000 mon.a (mon.0) 12523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:16 smithi121 bash[36902]: cluster 2022-04-22T21:43:15.996909+0000 mgr.y (mgr.34107) 8778 : cluster [DBG] pgmap v9656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:16 smithi121 bash[36902]: audit 2022-04-22T21:43:16.232525+0000 mon.a (mon.0) 12524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:43:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:16 smithi121 bash[36902]: audit 2022-04-22T21:43:16.232930+0000 mon.a (mon.0) 12525 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:43:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[45495]: audit 2022-04-22T21:43:15.743686+0000 mon.a (mon.0) 12523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[45495]: cluster 2022-04-22T21:43:15.996909+0000 mgr.y (mgr.34107) 8778 : cluster [DBG] pgmap v9656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[45495]: audit 2022-04-22T21:43:16.232525+0000 mon.a (mon.0) 12524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:43:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[45495]: audit 2022-04-22T21:43:16.232930+0000 mon.a (mon.0) 12525 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:43:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[46934]: audit 2022-04-22T21:43:15.743686+0000 mon.a (mon.0) 12523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[46934]: cluster 2022-04-22T21:43:15.996909+0000 mgr.y (mgr.34107) 8778 : cluster [DBG] pgmap v9656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[46934]: audit 2022-04-22T21:43:16.232525+0000 mon.a (mon.0) 12524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:43:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:16 smithi115 bash[46934]: audit 2022-04-22T21:43:16.232930+0000 mon.a (mon.0) 12525 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:43:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:19 smithi121 bash[36902]: cluster 2022-04-22T21:43:17.997902+0000 mgr.y (mgr.34107) 8779 : cluster [DBG] pgmap v9657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:19.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:19 smithi115 bash[46934]: cluster 2022-04-22T21:43:17.997902+0000 mgr.y (mgr.34107) 8779 : cluster [DBG] pgmap v9657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:19 smithi115 bash[45495]: cluster 2022-04-22T21:43:17.997902+0000 mgr.y (mgr.34107) 8779 : cluster [DBG] pgmap v9657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:20 smithi121 bash[36902]: audit 2022-04-22T21:43:19.762955+0000 mon.a (mon.0) 12526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:20.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:20 smithi115 bash[46934]: audit 2022-04-22T21:43:19.762955+0000 mon.a (mon.0) 12526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:20 smithi115 bash[45495]: audit 2022-04-22T21:43:19.762955+0000 mon.a (mon.0) 12526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:21 smithi121 bash[36902]: cluster 2022-04-22T21:43:19.998964+0000 mgr.y (mgr.34107) 8780 : cluster [DBG] pgmap v9658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:21 smithi121 bash[36902]: audit 2022-04-22T21:43:20.747541+0000 mon.a (mon.0) 12527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:21 smithi115 bash[45495]: cluster 2022-04-22T21:43:19.998964+0000 mgr.y (mgr.34107) 8780 : cluster [DBG] pgmap v9658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:21 smithi115 bash[45495]: audit 2022-04-22T21:43:20.747541+0000 mon.a (mon.0) 12527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:21 smithi115 bash[46934]: cluster 2022-04-22T21:43:19.998964+0000 mgr.y (mgr.34107) 8780 : cluster [DBG] pgmap v9658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:21 smithi115 bash[46934]: audit 2022-04-22T21:43:20.747541+0000 mon.a (mon.0) 12527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:22.624Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:23.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:23 smithi121 bash[36902]: cluster 2022-04-22T21:43:22.000208+0000 mgr.y (mgr.34107) 8781 : cluster [DBG] pgmap v9659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:23 smithi115 bash[46934]: cluster 2022-04-22T21:43:22.000208+0000 mgr.y (mgr.34107) 8781 : cluster [DBG] pgmap v9659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:23 smithi115 bash[45495]: cluster 2022-04-22T21:43:22.000208+0000 mgr.y (mgr.34107) 8781 : cluster [DBG] pgmap v9659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:24.487Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:25 smithi121 bash[36902]: cluster 2022-04-22T21:43:24.001421+0000 mgr.y (mgr.34107) 8782 : cluster [DBG] pgmap v9660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:25 smithi121 bash[36902]: audit 2022-04-22T21:43:24.766101+0000 mon.a (mon.0) 12528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:25 smithi115 bash[45495]: cluster 2022-04-22T21:43:24.001421+0000 mgr.y (mgr.34107) 8782 : cluster [DBG] pgmap v9660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:25 smithi115 bash[45495]: audit 2022-04-22T21:43:24.766101+0000 mon.a (mon.0) 12528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:25 smithi115 bash[46934]: cluster 2022-04-22T21:43:24.001421+0000 mgr.y (mgr.34107) 8782 : cluster [DBG] pgmap v9660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:25 smithi115 bash[46934]: audit 2022-04-22T21:43:24.766101+0000 mon.a (mon.0) 12528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:26.480 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:43:26.823 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:43:26.823 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.7M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 252M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:43:26.824 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:43:26.825 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:43:26.826 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 645M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:43:26.826 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 64.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:43:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:26 smithi121 bash[36902]: audit 2022-04-22T21:43:25.751228+0000 mon.a (mon.0) 12529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:26 smithi121 bash[36902]: cluster 2022-04-22T21:43:26.002127+0000 mgr.y (mgr.34107) 8783 : cluster [DBG] pgmap v9661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:26 smithi115 bash[45495]: audit 2022-04-22T21:43:25.751228+0000 mon.a (mon.0) 12529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:26 smithi115 bash[45495]: cluster 2022-04-22T21:43:26.002127+0000 mgr.y (mgr.34107) 8783 : cluster [DBG] pgmap v9661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:26 smithi115 bash[46934]: audit 2022-04-22T21:43:25.751228+0000 mon.a (mon.0) 12529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:26 smithi115 bash[46934]: cluster 2022-04-22T21:43:26.002127+0000 mgr.y (mgr.34107) 8783 : cluster [DBG] pgmap v9661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:27.227 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:43:27.352 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:43:27.352 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:43:27.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:27.353 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:43:27.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:43:27.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:27.356 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:43:27.357 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:43:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:27 smithi121 bash[36902]: audit 2022-04-22T21:43:26.460332+0000 mgr.y (mgr.34107) 8784 : audit [DBG] from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:27 smithi121 bash[36902]: audit 2022-04-22T21:43:26.800720+0000 mgr.y (mgr.34107) 8785 : audit [DBG] from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:27 smithi121 bash[36902]: audit 2022-04-22T21:43:27.224519+0000 mon.c (mon.1) 312 : audit [DBG] from='client.? 172.21.15.115:0/2906746613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:43:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[45495]: audit 2022-04-22T21:43:26.460332+0000 mgr.y (mgr.34107) 8784 : audit [DBG] from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[45495]: audit 2022-04-22T21:43:26.800720+0000 mgr.y (mgr.34107) 8785 : audit [DBG] from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[45495]: audit 2022-04-22T21:43:27.224519+0000 mon.c (mon.1) 312 : audit [DBG] from='client.? 172.21.15.115:0/2906746613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:43:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[46934]: audit 2022-04-22T21:43:26.460332+0000 mgr.y (mgr.34107) 8784 : audit [DBG] from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[46934]: audit 2022-04-22T21:43:26.800720+0000 mgr.y (mgr.34107) 8785 : audit [DBG] from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:27 smithi115 bash[46934]: audit 2022-04-22T21:43:27.224519+0000 mon.c (mon.1) 312 : audit [DBG] from='client.? 172.21.15.115:0/2906746613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:43:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:28 smithi121 bash[36902]: cluster 2022-04-22T21:43:28.003135+0000 mgr.y (mgr.34107) 8786 : cluster [DBG] pgmap v9662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:29.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:28 smithi115 bash[46934]: cluster 2022-04-22T21:43:28.003135+0000 mgr.y (mgr.34107) 8786 : cluster [DBG] pgmap v9662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:28 smithi115 bash[45495]: cluster 2022-04-22T21:43:28.003135+0000 mgr.y (mgr.34107) 8786 : cluster [DBG] pgmap v9662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:29 smithi121 bash[36902]: audit 2022-04-22T21:43:29.769039+0000 mon.a (mon.0) 12530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:29 smithi115 bash[45495]: audit 2022-04-22T21:43:29.769039+0000 mon.a (mon.0) 12530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:29 smithi115 bash[46934]: audit 2022-04-22T21:43:29.769039+0000 mon.a (mon.0) 12530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:30 smithi121 bash[36902]: cluster 2022-04-22T21:43:30.004274+0000 mgr.y (mgr.34107) 8787 : cluster [DBG] pgmap v9663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:30 smithi121 bash[36902]: audit 2022-04-22T21:43:30.755014+0000 mon.a (mon.0) 12531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:31.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:30 smithi115 bash[46934]: cluster 2022-04-22T21:43:30.004274+0000 mgr.y (mgr.34107) 8787 : cluster [DBG] pgmap v9663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:30 smithi115 bash[46934]: audit 2022-04-22T21:43:30.755014+0000 mon.a (mon.0) 12531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:30 smithi115 bash[45495]: cluster 2022-04-22T21:43:30.004274+0000 mgr.y (mgr.34107) 8787 : cluster [DBG] pgmap v9663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:30 smithi115 bash[45495]: audit 2022-04-22T21:43:30.755014+0000 mon.a (mon.0) 12531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:32.625Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:33.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:33 smithi121 bash[36902]: cluster 2022-04-22T21:43:32.005367+0000 mgr.y (mgr.34107) 8788 : cluster [DBG] pgmap v9664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:33 smithi115 bash[45495]: cluster 2022-04-22T21:43:32.005367+0000 mgr.y (mgr.34107) 8788 : cluster [DBG] pgmap v9664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:33 smithi115 bash[46934]: cluster 2022-04-22T21:43:32.005367+0000 mgr.y (mgr.34107) 8788 : cluster [DBG] pgmap v9664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:34.487Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:35 smithi121 bash[36902]: cluster 2022-04-22T21:43:34.006509+0000 mgr.y (mgr.34107) 8789 : cluster [DBG] pgmap v9665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:35 smithi121 bash[36902]: audit 2022-04-22T21:43:34.772155+0000 mon.a (mon.0) 12532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:35 smithi115 bash[45495]: cluster 2022-04-22T21:43:34.006509+0000 mgr.y (mgr.34107) 8789 : cluster [DBG] pgmap v9665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:35.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:35 smithi115 bash[45495]: audit 2022-04-22T21:43:34.772155+0000 mon.a (mon.0) 12532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:35 smithi115 bash[46934]: cluster 2022-04-22T21:43:34.006509+0000 mgr.y (mgr.34107) 8789 : cluster [DBG] pgmap v9665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:35.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:35 smithi115 bash[46934]: audit 2022-04-22T21:43:34.772155+0000 mon.a (mon.0) 12532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:36 smithi121 bash[36902]: audit 2022-04-22T21:43:35.758738+0000 mon.a (mon.0) 12533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:36 smithi121 bash[36902]: cluster 2022-04-22T21:43:36.007169+0000 mgr.y (mgr.34107) 8790 : cluster [DBG] pgmap v9666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:36 smithi115 bash[45495]: audit 2022-04-22T21:43:35.758738+0000 mon.a (mon.0) 12533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:36 smithi115 bash[45495]: cluster 2022-04-22T21:43:36.007169+0000 mgr.y (mgr.34107) 8790 : cluster [DBG] pgmap v9666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:36 smithi115 bash[46934]: audit 2022-04-22T21:43:35.758738+0000 mon.a (mon.0) 12533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:36 smithi115 bash[46934]: cluster 2022-04-22T21:43:36.007169+0000 mgr.y (mgr.34107) 8790 : cluster [DBG] pgmap v9666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:39 smithi121 bash[36902]: cluster 2022-04-22T21:43:38.008276+0000 mgr.y (mgr.34107) 8791 : cluster [DBG] pgmap v9667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:39 smithi115 bash[45495]: cluster 2022-04-22T21:43:38.008276+0000 mgr.y (mgr.34107) 8791 : cluster [DBG] pgmap v9667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:39 smithi115 bash[46934]: cluster 2022-04-22T21:43:38.008276+0000 mgr.y (mgr.34107) 8791 : cluster [DBG] pgmap v9667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:40.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:40 smithi121 bash[36902]: audit 2022-04-22T21:43:39.775116+0000 mon.a (mon.0) 12534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:40 smithi115 bash[45495]: audit 2022-04-22T21:43:39.775116+0000 mon.a (mon.0) 12534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:40 smithi115 bash[46934]: audit 2022-04-22T21:43:39.775116+0000 mon.a (mon.0) 12534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:41 smithi121 bash[36902]: cluster 2022-04-22T21:43:40.009416+0000 mgr.y (mgr.34107) 8792 : cluster [DBG] pgmap v9668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:41 smithi121 bash[36902]: audit 2022-04-22T21:43:40.762439+0000 mon.a (mon.0) 12535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:41 smithi115 bash[46934]: cluster 2022-04-22T21:43:40.009416+0000 mgr.y (mgr.34107) 8792 : cluster [DBG] pgmap v9668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:41 smithi115 bash[46934]: audit 2022-04-22T21:43:40.762439+0000 mon.a (mon.0) 12535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:41 smithi115 bash[45495]: cluster 2022-04-22T21:43:40.009416+0000 mgr.y (mgr.34107) 8792 : cluster [DBG] pgmap v9668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:41 smithi115 bash[45495]: audit 2022-04-22T21:43:40.762439+0000 mon.a (mon.0) 12535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:42.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:43 smithi121 bash[36902]: cluster 2022-04-22T21:43:42.010452+0000 mgr.y (mgr.34107) 8793 : cluster [DBG] pgmap v9669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:43.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:43 smithi115 bash[45495]: cluster 2022-04-22T21:43:42.010452+0000 mgr.y (mgr.34107) 8793 : cluster [DBG] pgmap v9669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:43 smithi115 bash[46934]: cluster 2022-04-22T21:43:42.010452+0000 mgr.y (mgr.34107) 8793 : cluster [DBG] pgmap v9669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:44.488Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:45 smithi121 bash[36902]: cluster 2022-04-22T21:43:44.011644+0000 mgr.y (mgr.34107) 8794 : cluster [DBG] pgmap v9670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:45 smithi121 bash[36902]: audit 2022-04-22T21:43:44.778221+0000 mon.a (mon.0) 12536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:45 smithi115 bash[45495]: cluster 2022-04-22T21:43:44.011644+0000 mgr.y (mgr.34107) 8794 : cluster [DBG] pgmap v9670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:45 smithi115 bash[45495]: audit 2022-04-22T21:43:44.778221+0000 mon.a (mon.0) 12536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:45 smithi115 bash[46934]: cluster 2022-04-22T21:43:44.011644+0000 mgr.y (mgr.34107) 8794 : cluster [DBG] pgmap v9670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:45 smithi115 bash[46934]: audit 2022-04-22T21:43:44.778221+0000 mon.a (mon.0) 12536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:46 smithi121 bash[36902]: audit 2022-04-22T21:43:45.766227+0000 mon.a (mon.0) 12537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:46 smithi121 bash[36902]: cluster 2022-04-22T21:43:46.012578+0000 mgr.y (mgr.34107) 8795 : cluster [DBG] pgmap v9671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:46 smithi115 bash[45495]: audit 2022-04-22T21:43:45.766227+0000 mon.a (mon.0) 12537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:46 smithi115 bash[45495]: cluster 2022-04-22T21:43:46.012578+0000 mgr.y (mgr.34107) 8795 : cluster [DBG] pgmap v9671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:46 smithi115 bash[46934]: audit 2022-04-22T21:43:45.766227+0000 mon.a (mon.0) 12537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:46 smithi115 bash[46934]: cluster 2022-04-22T21:43:46.012578+0000 mgr.y (mgr.34107) 8795 : cluster [DBG] pgmap v9671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:49 smithi121 bash[36902]: cluster 2022-04-22T21:43:48.013592+0000 mgr.y (mgr.34107) 8796 : cluster [DBG] pgmap v9672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:49 smithi115 bash[45495]: cluster 2022-04-22T21:43:48.013592+0000 mgr.y (mgr.34107) 8796 : cluster [DBG] pgmap v9672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:49 smithi115 bash[46934]: cluster 2022-04-22T21:43:48.013592+0000 mgr.y (mgr.34107) 8796 : cluster [DBG] pgmap v9672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:50.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:50 smithi115 bash[45495]: audit 2022-04-22T21:43:49.781605+0000 mon.a (mon.0) 12538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:50.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:50 smithi115 bash[46934]: audit 2022-04-22T21:43:49.781605+0000 mon.a (mon.0) 12538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:50.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:50 smithi121 bash[36902]: audit 2022-04-22T21:43:49.781605+0000 mon.a (mon.0) 12538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: cluster 2022-04-22T21:43:50.014705+0000 mgr.y (mgr.34107) 8797 : cluster [DBG] pgmap v9673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: audit 2022-04-22T21:43:50.318853+0000 mon.a (mon.0) 12539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: audit 2022-04-22T21:43:50.321989+0000 mon.a (mon.0) 12540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: audit 2022-04-22T21:43:50.323961+0000 mon.a (mon.0) 12541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: audit 2022-04-22T21:43:50.770067+0000 mon.a (mon.0) 12542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:51 smithi121 bash[36902]: audit 2022-04-22T21:43:50.904482+0000 mon.a (mon.0) 12543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: cluster 2022-04-22T21:43:50.014705+0000 mgr.y (mgr.34107) 8797 : cluster [DBG] pgmap v9673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: audit 2022-04-22T21:43:50.318853+0000 mon.a (mon.0) 12539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: audit 2022-04-22T21:43:50.321989+0000 mon.a (mon.0) 12540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: audit 2022-04-22T21:43:50.323961+0000 mon.a (mon.0) 12541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: audit 2022-04-22T21:43:50.770067+0000 mon.a (mon.0) 12542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[45495]: audit 2022-04-22T21:43:50.904482+0000 mon.a (mon.0) 12543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: cluster 2022-04-22T21:43:50.014705+0000 mgr.y (mgr.34107) 8797 : cluster [DBG] pgmap v9673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: audit 2022-04-22T21:43:50.318853+0000 mon.a (mon.0) 12539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:43:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: audit 2022-04-22T21:43:50.321989+0000 mon.a (mon.0) 12540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:43:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: audit 2022-04-22T21:43:50.323961+0000 mon.a (mon.0) 12541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:43:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: audit 2022-04-22T21:43:50.770067+0000 mon.a (mon.0) 12542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:51 smithi115 bash[46934]: audit 2022-04-22T21:43:50.904482+0000 mon.a (mon.0) 12543 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:52.625Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:53 smithi121 bash[36902]: cluster 2022-04-22T21:43:52.015835+0000 mgr.y (mgr.34107) 8798 : cluster [DBG] pgmap v9674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:53 smithi115 bash[46934]: cluster 2022-04-22T21:43:52.015835+0000 mgr.y (mgr.34107) 8798 : cluster [DBG] pgmap v9674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:53 smithi115 bash[45495]: cluster 2022-04-22T21:43:52.015835+0000 mgr.y (mgr.34107) 8798 : cluster [DBG] pgmap v9674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:43:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:43:54.489Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:43:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:55 smithi121 bash[36902]: cluster 2022-04-22T21:43:54.016951+0000 mgr.y (mgr.34107) 8799 : cluster [DBG] pgmap v9675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:55 smithi121 bash[36902]: audit 2022-04-22T21:43:54.734235+0000 mon.a (mon.0) 12544 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:55 smithi121 bash[36902]: audit 2022-04-22T21:43:54.784585+0000 mon.a (mon.0) 12545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[45495]: cluster 2022-04-22T21:43:54.016951+0000 mgr.y (mgr.34107) 8799 : cluster [DBG] pgmap v9675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[45495]: audit 2022-04-22T21:43:54.734235+0000 mon.a (mon.0) 12544 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[45495]: audit 2022-04-22T21:43:54.784585+0000 mon.a (mon.0) 12545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[46934]: cluster 2022-04-22T21:43:54.016951+0000 mgr.y (mgr.34107) 8799 : cluster [DBG] pgmap v9675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[46934]: audit 2022-04-22T21:43:54.734235+0000 mon.a (mon.0) 12544 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:55.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:55 smithi115 bash[46934]: audit 2022-04-22T21:43:54.784585+0000 mon.a (mon.0) 12545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:43:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:56 smithi121 bash[36902]: audit 2022-04-22T21:43:55.100281+0000 mon.a (mon.0) 12546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:56 smithi121 bash[36902]: audit 2022-04-22T21:43:55.773507+0000 mon.a (mon.0) 12547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:56 smithi115 bash[45495]: audit 2022-04-22T21:43:55.100281+0000 mon.a (mon.0) 12546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:56 smithi115 bash[45495]: audit 2022-04-22T21:43:55.773507+0000 mon.a (mon.0) 12547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:56 smithi115 bash[46934]: audit 2022-04-22T21:43:55.100281+0000 mon.a (mon.0) 12546 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:43:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:56 smithi115 bash[46934]: audit 2022-04-22T21:43:55.773507+0000 mon.a (mon.0) 12547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:43:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:57 smithi121 bash[36902]: cluster 2022-04-22T21:43:56.017948+0000 mgr.y (mgr.34107) 8800 : cluster [DBG] pgmap v9676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:57.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:57 smithi115 bash[46934]: cluster 2022-04-22T21:43:56.017948+0000 mgr.y (mgr.34107) 8800 : cluster [DBG] pgmap v9676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:57 smithi115 bash[45495]: cluster 2022-04-22T21:43:56.017948+0000 mgr.y (mgr.34107) 8800 : cluster [DBG] pgmap v9676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:57.603 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:43:57.946 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:43:57.946 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:43:57.946 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:43:57.947 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3s ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:43:57.948 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3s ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:43:57.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:43:57.949 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:43:58.353 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:43:58.354 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:43:58.354 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:43:58.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:58.354 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:43:58.354 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:43:58.355 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:43:58.356 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:59 smithi121 bash[36902]: audit 2022-04-22T21:43:57.584853+0000 mgr.y (mgr.34107) 8801 : audit [DBG] from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:59 smithi121 bash[36902]: audit 2022-04-22T21:43:57.923660+0000 mgr.y (mgr.34107) 8802 : audit [DBG] from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:59 smithi121 bash[36902]: cluster 2022-04-22T21:43:58.018883+0000 mgr.y (mgr.34107) 8803 : cluster [DBG] pgmap v9677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:43:59 smithi121 bash[36902]: audit 2022-04-22T21:43:58.350172+0000 mon.a (mon.0) 12548 : audit [DBG] from='client.? 172.21.15.115:0/3676612530' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:43:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[45495]: audit 2022-04-22T21:43:57.584853+0000 mgr.y (mgr.34107) 8801 : audit [DBG] from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[45495]: audit 2022-04-22T21:43:57.923660+0000 mgr.y (mgr.34107) 8802 : audit [DBG] from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[45495]: cluster 2022-04-22T21:43:58.018883+0000 mgr.y (mgr.34107) 8803 : cluster [DBG] pgmap v9677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[45495]: audit 2022-04-22T21:43:58.350172+0000 mon.a (mon.0) 12548 : audit [DBG] from='client.? 172.21.15.115:0/3676612530' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[46934]: audit 2022-04-22T21:43:57.584853+0000 mgr.y (mgr.34107) 8801 : audit [DBG] from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[46934]: audit 2022-04-22T21:43:57.923660+0000 mgr.y (mgr.34107) 8802 : audit [DBG] from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[46934]: cluster 2022-04-22T21:43:58.018883+0000 mgr.y (mgr.34107) 8803 : cluster [DBG] pgmap v9677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:43:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:43:59 smithi115 bash[46934]: audit 2022-04-22T21:43:58.350172+0000 mon.a (mon.0) 12548 : audit [DBG] from='client.? 172.21.15.115:0/3676612530' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:44:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:00 smithi121 bash[36902]: audit 2022-04-22T21:43:59.787337+0000 mon.a (mon.0) 12549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:00 smithi115 bash[45495]: audit 2022-04-22T21:43:59.787337+0000 mon.a (mon.0) 12549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:00 smithi115 bash[46934]: audit 2022-04-22T21:43:59.787337+0000 mon.a (mon.0) 12549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:01 smithi121 bash[36902]: cluster 2022-04-22T21:44:00.019987+0000 mgr.y (mgr.34107) 8804 : cluster [DBG] pgmap v9678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:01 smithi121 bash[36902]: audit 2022-04-22T21:44:00.777237+0000 mon.a (mon.0) 12550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:01 smithi115 bash[45495]: cluster 2022-04-22T21:44:00.019987+0000 mgr.y (mgr.34107) 8804 : cluster [DBG] pgmap v9678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:01 smithi115 bash[45495]: audit 2022-04-22T21:44:00.777237+0000 mon.a (mon.0) 12550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:01 smithi115 bash[46934]: cluster 2022-04-22T21:44:00.019987+0000 mgr.y (mgr.34107) 8804 : cluster [DBG] pgmap v9678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:01 smithi115 bash[46934]: audit 2022-04-22T21:44:00.777237+0000 mon.a (mon.0) 12550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:02.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:03.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:03 smithi121 bash[36902]: cluster 2022-04-22T21:44:02.021081+0000 mgr.y (mgr.34107) 8805 : cluster [DBG] pgmap v9679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:03 smithi115 bash[45495]: cluster 2022-04-22T21:44:02.021081+0000 mgr.y (mgr.34107) 8805 : cluster [DBG] pgmap v9679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:03 smithi115 bash[46934]: cluster 2022-04-22T21:44:02.021081+0000 mgr.y (mgr.34107) 8805 : cluster [DBG] pgmap v9679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:04.494 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:04 smithi115 bash[45495]: cluster 2022-04-22T21:44:04.022286+0000 mgr.y (mgr.34107) 8806 : cluster [DBG] pgmap v9680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:04.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:04 smithi115 bash[46934]: cluster 2022-04-22T21:44:04.022286+0000 mgr.y (mgr.34107) 8806 : cluster [DBG] pgmap v9680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:04.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:04 smithi121 bash[36902]: cluster 2022-04-22T21:44:04.022286+0000 mgr.y (mgr.34107) 8806 : cluster [DBG] pgmap v9680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:04.490Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:05.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:05 smithi121 bash[36902]: audit 2022-04-22T21:44:04.790205+0000 mon.a (mon.0) 12551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:05 smithi115 bash[45495]: audit 2022-04-22T21:44:04.790205+0000 mon.a (mon.0) 12551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:05 smithi115 bash[46934]: audit 2022-04-22T21:44:04.790205+0000 mon.a (mon.0) 12551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:06 smithi121 bash[36902]: audit 2022-04-22T21:44:05.780988+0000 mon.a (mon.0) 12552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:07.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:06 smithi121 bash[36902]: cluster 2022-04-22T21:44:06.023428+0000 mgr.y (mgr.34107) 8807 : cluster [DBG] pgmap v9681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:06 smithi115 bash[45495]: audit 2022-04-22T21:44:05.780988+0000 mon.a (mon.0) 12552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:06 smithi115 bash[45495]: cluster 2022-04-22T21:44:06.023428+0000 mgr.y (mgr.34107) 8807 : cluster [DBG] pgmap v9681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:07.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:06 smithi115 bash[46934]: audit 2022-04-22T21:44:05.780988+0000 mon.a (mon.0) 12552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:07.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:06 smithi115 bash[46934]: cluster 2022-04-22T21:44:06.023428+0000 mgr.y (mgr.34107) 8807 : cluster [DBG] pgmap v9681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:09 smithi121 bash[36902]: cluster 2022-04-22T21:44:08.024467+0000 mgr.y (mgr.34107) 8808 : cluster [DBG] pgmap v9682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:09.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:09 smithi115 bash[45495]: cluster 2022-04-22T21:44:08.024467+0000 mgr.y (mgr.34107) 8808 : cluster [DBG] pgmap v9682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:09.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:09 smithi115 bash[46934]: cluster 2022-04-22T21:44:08.024467+0000 mgr.y (mgr.34107) 8808 : cluster [DBG] pgmap v9682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:10 smithi121 bash[36902]: audit 2022-04-22T21:44:09.793120+0000 mon.a (mon.0) 12553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:10 smithi115 bash[45495]: audit 2022-04-22T21:44:09.793120+0000 mon.a (mon.0) 12553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:10 smithi115 bash[46934]: audit 2022-04-22T21:44:09.793120+0000 mon.a (mon.0) 12553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:11 smithi121 bash[36902]: cluster 2022-04-22T21:44:10.025597+0000 mgr.y (mgr.34107) 8809 : cluster [DBG] pgmap v9683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:11 smithi121 bash[36902]: audit 2022-04-22T21:44:10.784514+0000 mon.a (mon.0) 12554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:11 smithi115 bash[45495]: cluster 2022-04-22T21:44:10.025597+0000 mgr.y (mgr.34107) 8809 : cluster [DBG] pgmap v9683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:11 smithi115 bash[45495]: audit 2022-04-22T21:44:10.784514+0000 mon.a (mon.0) 12554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:11 smithi115 bash[46934]: cluster 2022-04-22T21:44:10.025597+0000 mgr.y (mgr.34107) 8809 : cluster [DBG] pgmap v9683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:11 smithi115 bash[46934]: audit 2022-04-22T21:44:10.784514+0000 mon.a (mon.0) 12554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:12.627Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:13.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:13 smithi121 bash[36902]: cluster 2022-04-22T21:44:12.026731+0000 mgr.y (mgr.34107) 8810 : cluster [DBG] pgmap v9684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:13 smithi115 bash[45495]: cluster 2022-04-22T21:44:12.026731+0000 mgr.y (mgr.34107) 8810 : cluster [DBG] pgmap v9684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:13 smithi115 bash[46934]: cluster 2022-04-22T21:44:12.026731+0000 mgr.y (mgr.34107) 8810 : cluster [DBG] pgmap v9684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:14.490Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:15 smithi121 bash[36902]: cluster 2022-04-22T21:44:14.027937+0000 mgr.y (mgr.34107) 8811 : cluster [DBG] pgmap v9685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:15 smithi121 bash[36902]: audit 2022-04-22T21:44:14.795972+0000 mon.a (mon.0) 12555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:15 smithi115 bash[45495]: cluster 2022-04-22T21:44:14.027937+0000 mgr.y (mgr.34107) 8811 : cluster [DBG] pgmap v9685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:15 smithi115 bash[45495]: audit 2022-04-22T21:44:14.795972+0000 mon.a (mon.0) 12555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:15 smithi115 bash[46934]: cluster 2022-04-22T21:44:14.027937+0000 mgr.y (mgr.34107) 8811 : cluster [DBG] pgmap v9685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:15 smithi115 bash[46934]: audit 2022-04-22T21:44:14.795972+0000 mon.a (mon.0) 12555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:16 smithi121 bash[36902]: audit 2022-04-22T21:44:15.787881+0000 mon.a (mon.0) 12556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:16 smithi121 bash[36902]: cluster 2022-04-22T21:44:16.028930+0000 mgr.y (mgr.34107) 8812 : cluster [DBG] pgmap v9686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:16 smithi121 bash[36902]: audit 2022-04-22T21:44:16.244501+0000 mon.a (mon.0) 12557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:44:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:16 smithi121 bash[36902]: audit 2022-04-22T21:44:16.244819+0000 mon.a (mon.0) 12558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:44:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[45495]: audit 2022-04-22T21:44:15.787881+0000 mon.a (mon.0) 12556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[45495]: cluster 2022-04-22T21:44:16.028930+0000 mgr.y (mgr.34107) 8812 : cluster [DBG] pgmap v9686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[45495]: audit 2022-04-22T21:44:16.244501+0000 mon.a (mon.0) 12557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[45495]: audit 2022-04-22T21:44:16.244819+0000 mon.a (mon.0) 12558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[46934]: audit 2022-04-22T21:44:15.787881+0000 mon.a (mon.0) 12556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[46934]: cluster 2022-04-22T21:44:16.028930+0000 mgr.y (mgr.34107) 8812 : cluster [DBG] pgmap v9686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[46934]: audit 2022-04-22T21:44:16.244501+0000 mon.a (mon.0) 12557 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:44:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:16 smithi115 bash[46934]: audit 2022-04-22T21:44:16.244819+0000 mon.a (mon.0) 12558 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:44:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:19 smithi121 bash[36902]: cluster 2022-04-22T21:44:18.029946+0000 mgr.y (mgr.34107) 8813 : cluster [DBG] pgmap v9687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:19 smithi115 bash[45495]: cluster 2022-04-22T21:44:18.029946+0000 mgr.y (mgr.34107) 8813 : cluster [DBG] pgmap v9687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:19 smithi115 bash[46934]: cluster 2022-04-22T21:44:18.029946+0000 mgr.y (mgr.34107) 8813 : cluster [DBG] pgmap v9687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:20 smithi121 bash[36902]: audit 2022-04-22T21:44:19.799538+0000 mon.a (mon.0) 12559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:20 smithi115 bash[45495]: audit 2022-04-22T21:44:19.799538+0000 mon.a (mon.0) 12559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:20 smithi115 bash[46934]: audit 2022-04-22T21:44:19.799538+0000 mon.a (mon.0) 12559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:21 smithi121 bash[36902]: cluster 2022-04-22T21:44:20.031070+0000 mgr.y (mgr.34107) 8814 : cluster [DBG] pgmap v9688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:21.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:21 smithi121 bash[36902]: audit 2022-04-22T21:44:20.791503+0000 mon.a (mon.0) 12560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:21 smithi115 bash[45495]: cluster 2022-04-22T21:44:20.031070+0000 mgr.y (mgr.34107) 8814 : cluster [DBG] pgmap v9688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:21.587 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:21 smithi115 bash[45495]: audit 2022-04-22T21:44:20.791503+0000 mon.a (mon.0) 12560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:21.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:21 smithi115 bash[46934]: cluster 2022-04-22T21:44:20.031070+0000 mgr.y (mgr.34107) 8814 : cluster [DBG] pgmap v9688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:21 smithi115 bash[46934]: audit 2022-04-22T21:44:20.791503+0000 mon.a (mon.0) 12560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:22.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:23 smithi121 bash[36902]: cluster 2022-04-22T21:44:22.032204+0000 mgr.y (mgr.34107) 8815 : cluster [DBG] pgmap v9689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:23 smithi115 bash[45495]: cluster 2022-04-22T21:44:22.032204+0000 mgr.y (mgr.34107) 8815 : cluster [DBG] pgmap v9689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:23 smithi115 bash[46934]: cluster 2022-04-22T21:44:22.032204+0000 mgr.y (mgr.34107) 8815 : cluster [DBG] pgmap v9689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:24.491Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:25 smithi121 bash[36902]: cluster 2022-04-22T21:44:24.033366+0000 mgr.y (mgr.34107) 8816 : cluster [DBG] pgmap v9690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:25 smithi121 bash[36902]: audit 2022-04-22T21:44:24.802844+0000 mon.a (mon.0) 12561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:25 smithi115 bash[45495]: cluster 2022-04-22T21:44:24.033366+0000 mgr.y (mgr.34107) 8816 : cluster [DBG] pgmap v9690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:25 smithi115 bash[45495]: audit 2022-04-22T21:44:24.802844+0000 mon.a (mon.0) 12561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:25 smithi115 bash[46934]: cluster 2022-04-22T21:44:24.033366+0000 mgr.y (mgr.34107) 8816 : cluster [DBG] pgmap v9690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:25 smithi115 bash[46934]: audit 2022-04-22T21:44:24.802844+0000 mon.a (mon.0) 12561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:26 smithi121 bash[36902]: audit 2022-04-22T21:44:25.794943+0000 mon.a (mon.0) 12562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:26 smithi121 bash[36902]: cluster 2022-04-22T21:44:26.034435+0000 mgr.y (mgr.34107) 8817 : cluster [DBG] pgmap v9691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:26 smithi115 bash[45495]: audit 2022-04-22T21:44:25.794943+0000 mon.a (mon.0) 12562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:26 smithi115 bash[45495]: cluster 2022-04-22T21:44:26.034435+0000 mgr.y (mgr.34107) 8817 : cluster [DBG] pgmap v9691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:26 smithi115 bash[46934]: audit 2022-04-22T21:44:25.794943+0000 mon.a (mon.0) 12562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:26 smithi115 bash[46934]: cluster 2022-04-22T21:44:26.034435+0000 mgr.y (mgr.34107) 8817 : cluster [DBG] pgmap v9691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:28.730 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:44:29.077 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 34s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 34s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 34s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 390M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:44:29.078 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 34s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 12.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 34s ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 792M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 536M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 496M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 34s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:44:29.079 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 34s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:44:29.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 34s ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:44:29.080 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 34s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:44:29.080 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 34s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:44:29.487 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:44:29.487 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:44:29.487 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:44:29.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:44:29.487 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:44:29.488 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:44:29.489 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:44:29.489 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:44:29.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:29 smithi121 bash[36902]: cluster 2022-04-22T21:44:28.035436+0000 mgr.y (mgr.34107) 8818 : cluster [DBG] pgmap v9692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:29 smithi115 bash[45495]: cluster 2022-04-22T21:44:28.035436+0000 mgr.y (mgr.34107) 8818 : cluster [DBG] pgmap v9692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:29 smithi115 bash[46934]: cluster 2022-04-22T21:44:28.035436+0000 mgr.y (mgr.34107) 8818 : cluster [DBG] pgmap v9692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:30 smithi121 bash[36902]: audit 2022-04-22T21:44:28.709857+0000 mgr.y (mgr.34107) 8819 : audit [DBG] from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:30 smithi121 bash[36902]: audit 2022-04-22T21:44:29.053189+0000 mgr.y (mgr.34107) 8820 : audit [DBG] from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:30 smithi121 bash[36902]: audit 2022-04-22T21:44:29.483655+0000 mon.c (mon.1) 313 : audit [DBG] from='client.? 172.21.15.115:0/2432822568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:44:30.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:30 smithi121 bash[36902]: audit 2022-04-22T21:44:29.806001+0000 mon.a (mon.0) 12563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[45495]: audit 2022-04-22T21:44:28.709857+0000 mgr.y (mgr.34107) 8819 : audit [DBG] from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[45495]: audit 2022-04-22T21:44:29.053189+0000 mgr.y (mgr.34107) 8820 : audit [DBG] from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[45495]: audit 2022-04-22T21:44:29.483655+0000 mon.c (mon.1) 313 : audit [DBG] from='client.? 172.21.15.115:0/2432822568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[45495]: audit 2022-04-22T21:44:29.806001+0000 mon.a (mon.0) 12563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[46934]: audit 2022-04-22T21:44:28.709857+0000 mgr.y (mgr.34107) 8819 : audit [DBG] from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[46934]: audit 2022-04-22T21:44:29.053189+0000 mgr.y (mgr.34107) 8820 : audit [DBG] from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[46934]: audit 2022-04-22T21:44:29.483655+0000 mon.c (mon.1) 313 : audit [DBG] from='client.? 172.21.15.115:0/2432822568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:44:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:30 smithi115 bash[46934]: audit 2022-04-22T21:44:29.806001+0000 mon.a (mon.0) 12563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:31 smithi121 bash[36902]: cluster 2022-04-22T21:44:30.036707+0000 mgr.y (mgr.34107) 8821 : cluster [DBG] pgmap v9693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:31 smithi121 bash[36902]: audit 2022-04-22T21:44:30.798952+0000 mon.a (mon.0) 12564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:31 smithi115 bash[45495]: cluster 2022-04-22T21:44:30.036707+0000 mgr.y (mgr.34107) 8821 : cluster [DBG] pgmap v9693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:31 smithi115 bash[45495]: audit 2022-04-22T21:44:30.798952+0000 mon.a (mon.0) 12564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:31 smithi115 bash[46934]: cluster 2022-04-22T21:44:30.036707+0000 mgr.y (mgr.34107) 8821 : cluster [DBG] pgmap v9693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:31 smithi115 bash[46934]: audit 2022-04-22T21:44:30.798952+0000 mon.a (mon.0) 12564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:32.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:33 smithi121 bash[36902]: cluster 2022-04-22T21:44:32.037818+0000 mgr.y (mgr.34107) 8822 : cluster [DBG] pgmap v9694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:33 smithi115 bash[45495]: cluster 2022-04-22T21:44:32.037818+0000 mgr.y (mgr.34107) 8822 : cluster [DBG] pgmap v9694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:33 smithi115 bash[46934]: cluster 2022-04-22T21:44:32.037818+0000 mgr.y (mgr.34107) 8822 : cluster [DBG] pgmap v9694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:34.490Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:35 smithi121 bash[36902]: cluster 2022-04-22T21:44:34.039094+0000 mgr.y (mgr.34107) 8823 : cluster [DBG] pgmap v9695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:35 smithi121 bash[36902]: audit 2022-04-22T21:44:34.809370+0000 mon.a (mon.0) 12565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:35 smithi115 bash[45495]: cluster 2022-04-22T21:44:34.039094+0000 mgr.y (mgr.34107) 8823 : cluster [DBG] pgmap v9695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:35 smithi115 bash[45495]: audit 2022-04-22T21:44:34.809370+0000 mon.a (mon.0) 12565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:35 smithi115 bash[46934]: cluster 2022-04-22T21:44:34.039094+0000 mgr.y (mgr.34107) 8823 : cluster [DBG] pgmap v9695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:35 smithi115 bash[46934]: audit 2022-04-22T21:44:34.809370+0000 mon.a (mon.0) 12565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:36 smithi121 bash[36902]: audit 2022-04-22T21:44:35.802966+0000 mon.a (mon.0) 12566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:36 smithi121 bash[36902]: cluster 2022-04-22T21:44:36.040216+0000 mgr.y (mgr.34107) 8824 : cluster [DBG] pgmap v9696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:36 smithi115 bash[45495]: audit 2022-04-22T21:44:35.802966+0000 mon.a (mon.0) 12566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:36 smithi115 bash[45495]: cluster 2022-04-22T21:44:36.040216+0000 mgr.y (mgr.34107) 8824 : cluster [DBG] pgmap v9696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:36 smithi115 bash[46934]: audit 2022-04-22T21:44:35.802966+0000 mon.a (mon.0) 12566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:36 smithi115 bash[46934]: cluster 2022-04-22T21:44:36.040216+0000 mgr.y (mgr.34107) 8824 : cluster [DBG] pgmap v9696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:39 smithi121 bash[36902]: cluster 2022-04-22T21:44:38.041297+0000 mgr.y (mgr.34107) 8825 : cluster [DBG] pgmap v9697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:39.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:39 smithi115 bash[45495]: cluster 2022-04-22T21:44:38.041297+0000 mgr.y (mgr.34107) 8825 : cluster [DBG] pgmap v9697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:39.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:39 smithi115 bash[46934]: cluster 2022-04-22T21:44:38.041297+0000 mgr.y (mgr.34107) 8825 : cluster [DBG] pgmap v9697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:40 smithi121 bash[36902]: audit 2022-04-22T21:44:39.812198+0000 mon.a (mon.0) 12567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:40 smithi115 bash[46934]: audit 2022-04-22T21:44:39.812198+0000 mon.a (mon.0) 12567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:40 smithi115 bash[45495]: audit 2022-04-22T21:44:39.812198+0000 mon.a (mon.0) 12567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:41 smithi121 bash[36902]: cluster 2022-04-22T21:44:40.042500+0000 mgr.y (mgr.34107) 8826 : cluster [DBG] pgmap v9698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:41 smithi121 bash[36902]: audit 2022-04-22T21:44:40.807027+0000 mon.a (mon.0) 12568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:41 smithi115 bash[45495]: cluster 2022-04-22T21:44:40.042500+0000 mgr.y (mgr.34107) 8826 : cluster [DBG] pgmap v9698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:41 smithi115 bash[45495]: audit 2022-04-22T21:44:40.807027+0000 mon.a (mon.0) 12568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:41 smithi115 bash[46934]: cluster 2022-04-22T21:44:40.042500+0000 mgr.y (mgr.34107) 8826 : cluster [DBG] pgmap v9698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:41 smithi115 bash[46934]: audit 2022-04-22T21:44:40.807027+0000 mon.a (mon.0) 12568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:42.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:43 smithi121 bash[36902]: cluster 2022-04-22T21:44:42.043667+0000 mgr.y (mgr.34107) 8827 : cluster [DBG] pgmap v9699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:43 smithi115 bash[45495]: cluster 2022-04-22T21:44:42.043667+0000 mgr.y (mgr.34107) 8827 : cluster [DBG] pgmap v9699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:43 smithi115 bash[46934]: cluster 2022-04-22T21:44:42.043667+0000 mgr.y (mgr.34107) 8827 : cluster [DBG] pgmap v9699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:44.491Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:45 smithi121 bash[36902]: cluster 2022-04-22T21:44:44.044736+0000 mgr.y (mgr.34107) 8828 : cluster [DBG] pgmap v9700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:45.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:45 smithi121 bash[36902]: audit 2022-04-22T21:44:44.815360+0000 mon.a (mon.0) 12569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:45 smithi115 bash[45495]: cluster 2022-04-22T21:44:44.044736+0000 mgr.y (mgr.34107) 8828 : cluster [DBG] pgmap v9700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:45.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:45 smithi115 bash[45495]: audit 2022-04-22T21:44:44.815360+0000 mon.a (mon.0) 12569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:45 smithi115 bash[46934]: cluster 2022-04-22T21:44:44.044736+0000 mgr.y (mgr.34107) 8828 : cluster [DBG] pgmap v9700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:45.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:45 smithi115 bash[46934]: audit 2022-04-22T21:44:44.815360+0000 mon.a (mon.0) 12569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:46 smithi121 bash[36902]: audit 2022-04-22T21:44:45.812579+0000 mon.a (mon.0) 12570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:47.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:46 smithi121 bash[36902]: cluster 2022-04-22T21:44:46.045865+0000 mgr.y (mgr.34107) 8829 : cluster [DBG] pgmap v9701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:46 smithi115 bash[45495]: audit 2022-04-22T21:44:45.812579+0000 mon.a (mon.0) 12570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:46 smithi115 bash[45495]: cluster 2022-04-22T21:44:46.045865+0000 mgr.y (mgr.34107) 8829 : cluster [DBG] pgmap v9701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:46 smithi115 bash[46934]: audit 2022-04-22T21:44:45.812579+0000 mon.a (mon.0) 12570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:46 smithi115 bash[46934]: cluster 2022-04-22T21:44:46.045865+0000 mgr.y (mgr.34107) 8829 : cluster [DBG] pgmap v9701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:49 smithi121 bash[36902]: cluster 2022-04-22T21:44:48.046859+0000 mgr.y (mgr.34107) 8830 : cluster [DBG] pgmap v9702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:49 smithi115 bash[45495]: cluster 2022-04-22T21:44:48.046859+0000 mgr.y (mgr.34107) 8830 : cluster [DBG] pgmap v9702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:49 smithi115 bash[46934]: cluster 2022-04-22T21:44:48.046859+0000 mgr.y (mgr.34107) 8830 : cluster [DBG] pgmap v9702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:50 smithi121 bash[36902]: audit 2022-04-22T21:44:49.818735+0000 mon.a (mon.0) 12571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:50 smithi115 bash[45495]: audit 2022-04-22T21:44:49.818735+0000 mon.a (mon.0) 12571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:50 smithi115 bash[46934]: audit 2022-04-22T21:44:49.818735+0000 mon.a (mon.0) 12571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:51 smithi121 bash[36902]: cluster 2022-04-22T21:44:50.047967+0000 mgr.y (mgr.34107) 8831 : cluster [DBG] pgmap v9703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:51 smithi121 bash[36902]: audit 2022-04-22T21:44:50.816417+0000 mon.a (mon.0) 12572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:51 smithi115 bash[45495]: cluster 2022-04-22T21:44:50.047967+0000 mgr.y (mgr.34107) 8831 : cluster [DBG] pgmap v9703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:51 smithi115 bash[45495]: audit 2022-04-22T21:44:50.816417+0000 mon.a (mon.0) 12572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:51 smithi115 bash[46934]: cluster 2022-04-22T21:44:50.047967+0000 mgr.y (mgr.34107) 8831 : cluster [DBG] pgmap v9703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:51 smithi115 bash[46934]: audit 2022-04-22T21:44:50.816417+0000 mon.a (mon.0) 12572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:52.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:53 smithi121 bash[36902]: cluster 2022-04-22T21:44:52.049072+0000 mgr.y (mgr.34107) 8832 : cluster [DBG] pgmap v9704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:53 smithi115 bash[46934]: cluster 2022-04-22T21:44:52.049072+0000 mgr.y (mgr.34107) 8832 : cluster [DBG] pgmap v9704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:53 smithi115 bash[45495]: cluster 2022-04-22T21:44:52.049072+0000 mgr.y (mgr.34107) 8832 : cluster [DBG] pgmap v9704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:44:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:44:54.492Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:44:55.414 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[45495]: cluster 2022-04-22T21:44:54.050234+0000 mgr.y (mgr.34107) 8833 : cluster [DBG] pgmap v9705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:55.415 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[45495]: audit 2022-04-22T21:44:54.822034+0000 mon.a (mon.0) 12573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:55.415 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[45495]: audit 2022-04-22T21:44:55.108784+0000 mon.a (mon.0) 12574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:44:55.415 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[45495]: audit 2022-04-22T21:44:55.111858+0000 mon.a (mon.0) 12575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:44:55.415 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[45495]: audit 2022-04-22T21:44:55.113649+0000 mon.a (mon.0) 12576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:44:55.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[46934]: cluster 2022-04-22T21:44:54.050234+0000 mgr.y (mgr.34107) 8833 : cluster [DBG] pgmap v9705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:55.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[46934]: audit 2022-04-22T21:44:54.822034+0000 mon.a (mon.0) 12573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:55.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[46934]: audit 2022-04-22T21:44:55.108784+0000 mon.a (mon.0) 12574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:44:55.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[46934]: audit 2022-04-22T21:44:55.111858+0000 mon.a (mon.0) 12575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:44:55.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:55 smithi115 bash[46934]: audit 2022-04-22T21:44:55.113649+0000 mon.a (mon.0) 12576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:44:55.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:55 smithi121 bash[36902]: cluster 2022-04-22T21:44:54.050234+0000 mgr.y (mgr.34107) 8833 : cluster [DBG] pgmap v9705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:55.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:55 smithi121 bash[36902]: audit 2022-04-22T21:44:54.822034+0000 mon.a (mon.0) 12573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:44:55.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:55 smithi121 bash[36902]: audit 2022-04-22T21:44:55.108784+0000 mon.a (mon.0) 12574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:44:55.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:55 smithi121 bash[36902]: audit 2022-04-22T21:44:55.111858+0000 mon.a (mon.0) 12575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:44:55.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:55 smithi121 bash[36902]: audit 2022-04-22T21:44:55.113649+0000 mon.a (mon.0) 12576 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:44:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:56 smithi121 bash[36902]: audit 2022-04-22T21:44:55.429027+0000 mon.a (mon.0) 12577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:56 smithi121 bash[36902]: audit 2022-04-22T21:44:55.439034+0000 mon.a (mon.0) 12578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:56 smithi121 bash[36902]: audit 2022-04-22T21:44:55.820278+0000 mon.a (mon.0) 12579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:56 smithi121 bash[36902]: cluster 2022-04-22T21:44:56.051360+0000 mgr.y (mgr.34107) 8834 : cluster [DBG] pgmap v9706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[45495]: audit 2022-04-22T21:44:55.429027+0000 mon.a (mon.0) 12577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[45495]: audit 2022-04-22T21:44:55.439034+0000 mon.a (mon.0) 12578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[45495]: audit 2022-04-22T21:44:55.820278+0000 mon.a (mon.0) 12579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[45495]: cluster 2022-04-22T21:44:56.051360+0000 mgr.y (mgr.34107) 8834 : cluster [DBG] pgmap v9706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[46934]: audit 2022-04-22T21:44:55.429027+0000 mon.a (mon.0) 12577 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[46934]: audit 2022-04-22T21:44:55.439034+0000 mon.a (mon.0) 12578 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:44:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[46934]: audit 2022-04-22T21:44:55.820278+0000 mon.a (mon.0) 12579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:44:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:56 smithi115 bash[46934]: cluster 2022-04-22T21:44:56.051360+0000 mgr.y (mgr.34107) 8834 : cluster [DBG] pgmap v9706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:59.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:44:59 smithi121 bash[36902]: cluster 2022-04-22T21:44:58.052403+0000 mgr.y (mgr.34107) 8835 : cluster [DBG] pgmap v9707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:44:59 smithi115 bash[45495]: cluster 2022-04-22T21:44:58.052403+0000 mgr.y (mgr.34107) 8835 : cluster [DBG] pgmap v9707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:44:59 smithi115 bash[46934]: cluster 2022-04-22T21:44:58.052403+0000 mgr.y (mgr.34107) 8835 : cluster [DBG] pgmap v9707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:44:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:45:00.197 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 0s ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 65s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 0s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 65s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 65s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 0s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 0s ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:45:00.198 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 65s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 0s ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 0s ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 65s ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 0s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 0s ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 0s ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 0s ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 65s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:45:00.199 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 65s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:45:00.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 65s ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:45:00.200 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 65s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:45:00.200 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 65s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:00.602 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:45:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:45:01.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[45495]: audit 2022-04-22T21:44:59.551740+0000 mon.a (mon.0) 12580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:01.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[45495]: audit 2022-04-22T21:44:59.824865+0000 mon.a (mon.0) 12581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:01.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[45495]: audit 2022-04-22T21:44:59.833436+0000 mgr.y (mgr.34107) 8836 : audit [DBG] from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:01.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[45495]: cluster 2022-04-22T21:45:00.053588+0000 mgr.y (mgr.34107) 8837 : cluster [DBG] pgmap v9708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:01.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[46934]: audit 2022-04-22T21:44:59.551740+0000 mon.a (mon.0) 12580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:01.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[46934]: audit 2022-04-22T21:44:59.824865+0000 mon.a (mon.0) 12581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:01.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[46934]: audit 2022-04-22T21:44:59.833436+0000 mgr.y (mgr.34107) 8836 : audit [DBG] from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:01.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:00 smithi115 bash[46934]: cluster 2022-04-22T21:45:00.053588+0000 mgr.y (mgr.34107) 8837 : cluster [DBG] pgmap v9708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:00 smithi121 bash[36902]: audit 2022-04-22T21:44:59.551740+0000 mon.a (mon.0) 12580 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:00 smithi121 bash[36902]: audit 2022-04-22T21:44:59.824865+0000 mon.a (mon.0) 12581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:00 smithi121 bash[36902]: audit 2022-04-22T21:44:59.833436+0000 mgr.y (mgr.34107) 8836 : audit [DBG] from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:00 smithi121 bash[36902]: cluster 2022-04-22T21:45:00.053588+0000 mgr.y (mgr.34107) 8837 : cluster [DBG] pgmap v9708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:01 smithi121 bash[36902]: audit 2022-04-22T21:45:00.179926+0000 mgr.y (mgr.34107) 8838 : audit [DBG] from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:01 smithi121 bash[36902]: audit 2022-04-22T21:45:00.598304+0000 mon.a (mon.0) 12582 : audit [DBG] from='client.? 172.21.15.115:0/3349917934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:01 smithi121 bash[36902]: audit 2022-04-22T21:45:00.823838+0000 mon.a (mon.0) 12583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[45495]: audit 2022-04-22T21:45:00.179926+0000 mgr.y (mgr.34107) 8838 : audit [DBG] from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:02.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[45495]: audit 2022-04-22T21:45:00.598304+0000 mon.a (mon.0) 12582 : audit [DBG] from='client.? 172.21.15.115:0/3349917934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:02.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[45495]: audit 2022-04-22T21:45:00.823838+0000 mon.a (mon.0) 12583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[46934]: audit 2022-04-22T21:45:00.179926+0000 mgr.y (mgr.34107) 8838 : audit [DBG] from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[46934]: audit 2022-04-22T21:45:00.598304+0000 mon.a (mon.0) 12582 : audit [DBG] from='client.? 172.21.15.115:0/3349917934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:01 smithi115 bash[46934]: audit 2022-04-22T21:45:00.823838+0000 mon.a (mon.0) 12583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:02.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:02 smithi115 bash[45495]: cluster 2022-04-22T21:45:02.054752+0000 mgr.y (mgr.34107) 8839 : cluster [DBG] pgmap v9709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:02.900 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:02.630Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:02.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:02 smithi115 bash[46934]: cluster 2022-04-22T21:45:02.054752+0000 mgr.y (mgr.34107) 8839 : cluster [DBG] pgmap v9709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:03.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:02 smithi121 bash[36902]: cluster 2022-04-22T21:45:02.054752+0000 mgr.y (mgr.34107) 8839 : cluster [DBG] pgmap v9709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:04.491Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:05 smithi121 bash[36902]: cluster 2022-04-22T21:45:04.055905+0000 mgr.y (mgr.34107) 8840 : cluster [DBG] pgmap v9710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:05 smithi121 bash[36902]: audit 2022-04-22T21:45:04.827365+0000 mon.a (mon.0) 12584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:05 smithi115 bash[45495]: cluster 2022-04-22T21:45:04.055905+0000 mgr.y (mgr.34107) 8840 : cluster [DBG] pgmap v9710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:05 smithi115 bash[45495]: audit 2022-04-22T21:45:04.827365+0000 mon.a (mon.0) 12584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:05 smithi115 bash[46934]: cluster 2022-04-22T21:45:04.055905+0000 mgr.y (mgr.34107) 8840 : cluster [DBG] pgmap v9710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:05 smithi115 bash[46934]: audit 2022-04-22T21:45:04.827365+0000 mon.a (mon.0) 12584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:06 smithi121 bash[36902]: audit 2022-04-22T21:45:05.827676+0000 mon.a (mon.0) 12585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:06 smithi121 bash[36902]: cluster 2022-04-22T21:45:06.056683+0000 mgr.y (mgr.34107) 8841 : cluster [DBG] pgmap v9711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:06 smithi115 bash[45495]: audit 2022-04-22T21:45:05.827676+0000 mon.a (mon.0) 12585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:06 smithi115 bash[45495]: cluster 2022-04-22T21:45:06.056683+0000 mgr.y (mgr.34107) 8841 : cluster [DBG] pgmap v9711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:06 smithi115 bash[46934]: audit 2022-04-22T21:45:05.827676+0000 mon.a (mon.0) 12585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:06 smithi115 bash[46934]: cluster 2022-04-22T21:45:06.056683+0000 mgr.y (mgr.34107) 8841 : cluster [DBG] pgmap v9711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:09 smithi121 bash[36902]: cluster 2022-04-22T21:45:08.057672+0000 mgr.y (mgr.34107) 8842 : cluster [DBG] pgmap v9712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:09 smithi121 bash[36902]: audit 2022-04-22T21:45:08.829256+0000 mon.a (mon.0) 12586 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:09 smithi121 bash[36902]: audit 2022-04-22T21:45:08.837771+0000 mon.a (mon.0) 12587 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[45495]: cluster 2022-04-22T21:45:08.057672+0000 mgr.y (mgr.34107) 8842 : cluster [DBG] pgmap v9712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[45495]: audit 2022-04-22T21:45:08.829256+0000 mon.a (mon.0) 12586 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[45495]: audit 2022-04-22T21:45:08.837771+0000 mon.a (mon.0) 12587 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[46934]: cluster 2022-04-22T21:45:08.057672+0000 mgr.y (mgr.34107) 8842 : cluster [DBG] pgmap v9712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[46934]: audit 2022-04-22T21:45:08.829256+0000 mon.a (mon.0) 12586 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:09 smithi115 bash[46934]: audit 2022-04-22T21:45:08.837771+0000 mon.a (mon.0) 12587 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:45:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:10 smithi121 bash[36902]: cephadm 2022-04-22T21:45:08.818080+0000 mgr.y (mgr.34107) 8843 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:45:10.557 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:10 smithi121 bash[36902]: audit 2022-04-22T21:45:09.830198+0000 mon.a (mon.0) 12588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:10.558 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:10 smithi115 bash[45495]: cephadm 2022-04-22T21:45:08.818080+0000 mgr.y (mgr.34107) 8843 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:45:10.558 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:10 smithi115 bash[45495]: audit 2022-04-22T21:45:09.830198+0000 mon.a (mon.0) 12588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:10.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:10 smithi115 bash[46934]: cephadm 2022-04-22T21:45:08.818080+0000 mgr.y (mgr.34107) 8843 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T21:45:10.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:10 smithi115 bash[46934]: audit 2022-04-22T21:45:09.830198+0000 mon.a (mon.0) 12588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:11 smithi121 bash[36902]: cluster 2022-04-22T21:45:10.058794+0000 mgr.y (mgr.34107) 8844 : cluster [DBG] pgmap v9713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:11 smithi121 bash[36902]: audit 2022-04-22T21:45:10.831333+0000 mon.a (mon.0) 12589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:11 smithi115 bash[45495]: cluster 2022-04-22T21:45:10.058794+0000 mgr.y (mgr.34107) 8844 : cluster [DBG] pgmap v9713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:11 smithi115 bash[45495]: audit 2022-04-22T21:45:10.831333+0000 mon.a (mon.0) 12589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:11 smithi115 bash[46934]: cluster 2022-04-22T21:45:10.058794+0000 mgr.y (mgr.34107) 8844 : cluster [DBG] pgmap v9713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:11 smithi115 bash[46934]: audit 2022-04-22T21:45:10.831333+0000 mon.a (mon.0) 12589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:12.630Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:13 smithi121 bash[36902]: cluster 2022-04-22T21:45:12.059886+0000 mgr.y (mgr.34107) 8845 : cluster [DBG] pgmap v9714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:13 smithi115 bash[45495]: cluster 2022-04-22T21:45:12.059886+0000 mgr.y (mgr.34107) 8845 : cluster [DBG] pgmap v9714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:13 smithi115 bash[46934]: cluster 2022-04-22T21:45:12.059886+0000 mgr.y (mgr.34107) 8845 : cluster [DBG] pgmap v9714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:14.493Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:15 smithi121 bash[36902]: cluster 2022-04-22T21:45:14.060991+0000 mgr.y (mgr.34107) 8846 : cluster [DBG] pgmap v9715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:15 smithi121 bash[36902]: audit 2022-04-22T21:45:14.833367+0000 mon.a (mon.0) 12590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:15 smithi115 bash[45495]: cluster 2022-04-22T21:45:14.060991+0000 mgr.y (mgr.34107) 8846 : cluster [DBG] pgmap v9715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:15.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:15 smithi115 bash[45495]: audit 2022-04-22T21:45:14.833367+0000 mon.a (mon.0) 12590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:15 smithi115 bash[46934]: cluster 2022-04-22T21:45:14.060991+0000 mgr.y (mgr.34107) 8846 : cluster [DBG] pgmap v9715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:15.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:15 smithi115 bash[46934]: audit 2022-04-22T21:45:14.833367+0000 mon.a (mon.0) 12590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:16 smithi121 bash[36902]: audit 2022-04-22T21:45:15.835047+0000 mon.a (mon.0) 12591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:16 smithi121 bash[36902]: cluster 2022-04-22T21:45:16.062077+0000 mgr.y (mgr.34107) 8847 : cluster [DBG] pgmap v9716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:16 smithi121 bash[36902]: audit 2022-04-22T21:45:16.253978+0000 mon.a (mon.0) 12592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:45:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:16 smithi121 bash[36902]: audit 2022-04-22T21:45:16.255771+0000 mon.a (mon.0) 12593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:45:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[45495]: audit 2022-04-22T21:45:15.835047+0000 mon.a (mon.0) 12591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[45495]: cluster 2022-04-22T21:45:16.062077+0000 mgr.y (mgr.34107) 8847 : cluster [DBG] pgmap v9716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[45495]: audit 2022-04-22T21:45:16.253978+0000 mon.a (mon.0) 12592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:45:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[45495]: audit 2022-04-22T21:45:16.255771+0000 mon.a (mon.0) 12593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:45:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[46934]: audit 2022-04-22T21:45:15.835047+0000 mon.a (mon.0) 12591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[46934]: cluster 2022-04-22T21:45:16.062077+0000 mgr.y (mgr.34107) 8847 : cluster [DBG] pgmap v9716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[46934]: audit 2022-04-22T21:45:16.253978+0000 mon.a (mon.0) 12592 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:45:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:16 smithi115 bash[46934]: audit 2022-04-22T21:45:16.255771+0000 mon.a (mon.0) 12593 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:45:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:19 smithi121 bash[36902]: cluster 2022-04-22T21:45:18.063284+0000 mgr.y (mgr.34107) 8848 : cluster [DBG] pgmap v9717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:19 smithi115 bash[45495]: cluster 2022-04-22T21:45:18.063284+0000 mgr.y (mgr.34107) 8848 : cluster [DBG] pgmap v9717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:19 smithi115 bash[46934]: cluster 2022-04-22T21:45:18.063284+0000 mgr.y (mgr.34107) 8848 : cluster [DBG] pgmap v9717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:20 smithi121 bash[36902]: audit 2022-04-22T21:45:19.836562+0000 mon.a (mon.0) 12594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:20 smithi115 bash[45495]: audit 2022-04-22T21:45:19.836562+0000 mon.a (mon.0) 12594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:20 smithi115 bash[46934]: audit 2022-04-22T21:45:19.836562+0000 mon.a (mon.0) 12594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:21 smithi121 bash[36902]: cluster 2022-04-22T21:45:20.064270+0000 mgr.y (mgr.34107) 8849 : cluster [DBG] pgmap v9718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:21 smithi121 bash[36902]: audit 2022-04-22T21:45:20.838741+0000 mon.a (mon.0) 12595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:21 smithi115 bash[45495]: cluster 2022-04-22T21:45:20.064270+0000 mgr.y (mgr.34107) 8849 : cluster [DBG] pgmap v9718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:21 smithi115 bash[45495]: audit 2022-04-22T21:45:20.838741+0000 mon.a (mon.0) 12595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:21 smithi115 bash[46934]: cluster 2022-04-22T21:45:20.064270+0000 mgr.y (mgr.34107) 8849 : cluster [DBG] pgmap v9718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:21 smithi115 bash[46934]: audit 2022-04-22T21:45:20.838741+0000 mon.a (mon.0) 12595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:22.633Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:23 smithi121 bash[36902]: cluster 2022-04-22T21:45:22.065340+0000 mgr.y (mgr.34107) 8850 : cluster [DBG] pgmap v9719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:23 smithi115 bash[45495]: cluster 2022-04-22T21:45:22.065340+0000 mgr.y (mgr.34107) 8850 : cluster [DBG] pgmap v9719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:23.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:23 smithi115 bash[46934]: cluster 2022-04-22T21:45:22.065340+0000 mgr.y (mgr.34107) 8850 : cluster [DBG] pgmap v9719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:24.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:24 smithi115 bash[45495]: cluster 2022-04-22T21:45:24.066508+0000 mgr.y (mgr.34107) 8851 : cluster [DBG] pgmap v9720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:24.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:24 smithi115 bash[46934]: cluster 2022-04-22T21:45:24.066508+0000 mgr.y (mgr.34107) 8851 : cluster [DBG] pgmap v9720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:24 smithi121 bash[36902]: cluster 2022-04-22T21:45:24.066508+0000 mgr.y (mgr.34107) 8851 : cluster [DBG] pgmap v9720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:24.493Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:25 smithi121 bash[36902]: audit 2022-04-22T21:45:24.839622+0000 mon.a (mon.0) 12596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:25 smithi115 bash[45495]: audit 2022-04-22T21:45:24.839622+0000 mon.a (mon.0) 12596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:25 smithi115 bash[46934]: audit 2022-04-22T21:45:24.839622+0000 mon.a (mon.0) 12596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:26 smithi121 bash[36902]: audit 2022-04-22T21:45:25.842134+0000 mon.a (mon.0) 12597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:27.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:26 smithi121 bash[36902]: cluster 2022-04-22T21:45:26.067571+0000 mgr.y (mgr.34107) 8852 : cluster [DBG] pgmap v9721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:26 smithi115 bash[45495]: audit 2022-04-22T21:45:25.842134+0000 mon.a (mon.0) 12597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:26 smithi115 bash[45495]: cluster 2022-04-22T21:45:26.067571+0000 mgr.y (mgr.34107) 8852 : cluster [DBG] pgmap v9721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:26 smithi115 bash[46934]: audit 2022-04-22T21:45:25.842134+0000 mon.a (mon.0) 12597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:27.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:26 smithi115 bash[46934]: cluster 2022-04-22T21:45:26.067571+0000 mgr.y (mgr.34107) 8852 : cluster [DBG] pgmap v9721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:29 smithi121 bash[36902]: cluster 2022-04-22T21:45:28.068644+0000 mgr.y (mgr.34107) 8853 : cluster [DBG] pgmap v9722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:29 smithi115 bash[45495]: cluster 2022-04-22T21:45:28.068644+0000 mgr.y (mgr.34107) 8853 : cluster [DBG] pgmap v9722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:29 smithi115 bash[46934]: cluster 2022-04-22T21:45:28.068644+0000 mgr.y (mgr.34107) 8853 : cluster [DBG] pgmap v9722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:30 smithi121 bash[36902]: audit 2022-04-22T21:45:29.842956+0000 mon.a (mon.0) 12598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:30 smithi115 bash[45495]: audit 2022-04-22T21:45:29.842956+0000 mon.a (mon.0) 12598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:30 smithi115 bash[46934]: audit 2022-04-22T21:45:29.842956+0000 mon.a (mon.0) 12598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:30.984 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:45:31.333 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:45:31.333 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 31s ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:45:31.333 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 96s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:45:31.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 31s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:45:31.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 96s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:45:31.334 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 96s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:45:31.335 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 31s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:45:31.335 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 31s ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:45:31.335 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 96s ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:45:31.336 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 31s ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:45:31.336 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 31s ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:45:31.336 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 96s ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:45:31.336 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 31s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:45:31.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 31s ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:45:31.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 31s ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:45:31.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 31s ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:45:31.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 96s ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:45:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 96s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:45:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 96s ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:45:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 96s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:45:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 96s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:45:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:31 smithi121 bash[36902]: cluster 2022-04-22T21:45:30.069715+0000 mgr.y (mgr.34107) 8854 : cluster [DBG] pgmap v9723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:31 smithi121 bash[36902]: audit 2022-04-22T21:45:30.845487+0000 mon.a (mon.0) 12599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:31 smithi115 bash[46934]: cluster 2022-04-22T21:45:30.069715+0000 mgr.y (mgr.34107) 8854 : cluster [DBG] pgmap v9723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:31 smithi115 bash[46934]: audit 2022-04-22T21:45:30.845487+0000 mon.a (mon.0) 12599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:31 smithi115 bash[45495]: cluster 2022-04-22T21:45:30.069715+0000 mgr.y (mgr.34107) 8854 : cluster [DBG] pgmap v9723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:31.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:31 smithi115 bash[45495]: audit 2022-04-22T21:45:30.845487+0000 mon.a (mon.0) 12599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:45:31.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:45:31.731 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:45:31.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:45:31.732 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:45:31.732 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:45:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:32 smithi121 bash[36902]: audit 2022-04-22T21:45:30.965358+0000 mgr.y (mgr.34107) 8855 : audit [DBG] from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:32 smithi121 bash[36902]: audit 2022-04-22T21:45:31.726419+0000 mon.a (mon.0) 12600 : audit [DBG] from='client.? 172.21.15.115:0/1101078374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:32 smithi115 bash[45495]: audit 2022-04-22T21:45:30.965358+0000 mgr.y (mgr.34107) 8855 : audit [DBG] from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:32 smithi115 bash[45495]: audit 2022-04-22T21:45:31.726419+0000 mon.a (mon.0) 12600 : audit [DBG] from='client.? 172.21.15.115:0/1101078374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:32 smithi115 bash[46934]: audit 2022-04-22T21:45:30.965358+0000 mgr.y (mgr.34107) 8855 : audit [DBG] from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:32 smithi115 bash[46934]: audit 2022-04-22T21:45:31.726419+0000 mon.a (mon.0) 12600 : audit [DBG] from='client.? 172.21.15.115:0/1101078374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:45:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:32.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:33 smithi121 bash[36902]: audit 2022-04-22T21:45:31.310625+0000 mgr.y (mgr.34107) 8856 : audit [DBG] from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:33 smithi121 bash[36902]: cluster 2022-04-22T21:45:32.071018+0000 mgr.y (mgr.34107) 8857 : cluster [DBG] pgmap v9724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:33 smithi115 bash[45495]: audit 2022-04-22T21:45:31.310625+0000 mgr.y (mgr.34107) 8856 : audit [DBG] from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:33 smithi115 bash[45495]: cluster 2022-04-22T21:45:32.071018+0000 mgr.y (mgr.34107) 8857 : cluster [DBG] pgmap v9724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:33 smithi115 bash[46934]: audit 2022-04-22T21:45:31.310625+0000 mgr.y (mgr.34107) 8856 : audit [DBG] from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:45:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:33 smithi115 bash[46934]: cluster 2022-04-22T21:45:32.071018+0000 mgr.y (mgr.34107) 8857 : cluster [DBG] pgmap v9724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:34.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:34 smithi115 bash[45495]: cluster 2022-04-22T21:45:34.072044+0000 mgr.y (mgr.34107) 8858 : cluster [DBG] pgmap v9725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:34.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:34 smithi115 bash[46934]: cluster 2022-04-22T21:45:34.072044+0000 mgr.y (mgr.34107) 8858 : cluster [DBG] pgmap v9725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:34 smithi121 bash[36902]: cluster 2022-04-22T21:45:34.072044+0000 mgr.y (mgr.34107) 8858 : cluster [DBG] pgmap v9725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:34.493Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:35 smithi121 bash[36902]: audit 2022-04-22T21:45:34.845841+0000 mon.a (mon.0) 12601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:35 smithi115 bash[45495]: audit 2022-04-22T21:45:34.845841+0000 mon.a (mon.0) 12601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:35 smithi115 bash[46934]: audit 2022-04-22T21:45:34.845841+0000 mon.a (mon.0) 12601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:36 smithi121 bash[36902]: audit 2022-04-22T21:45:35.848521+0000 mon.a (mon.0) 12602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:36 smithi121 bash[36902]: cluster 2022-04-22T21:45:36.073039+0000 mgr.y (mgr.34107) 8859 : cluster [DBG] pgmap v9726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:36 smithi115 bash[45495]: audit 2022-04-22T21:45:35.848521+0000 mon.a (mon.0) 12602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:36 smithi115 bash[45495]: cluster 2022-04-22T21:45:36.073039+0000 mgr.y (mgr.34107) 8859 : cluster [DBG] pgmap v9726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:36 smithi115 bash[46934]: audit 2022-04-22T21:45:35.848521+0000 mon.a (mon.0) 12602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:36 smithi115 bash[46934]: cluster 2022-04-22T21:45:36.073039+0000 mgr.y (mgr.34107) 8859 : cluster [DBG] pgmap v9726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:39 smithi121 bash[36902]: cluster 2022-04-22T21:45:38.074061+0000 mgr.y (mgr.34107) 8860 : cluster [DBG] pgmap v9727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:39 smithi115 bash[45495]: cluster 2022-04-22T21:45:38.074061+0000 mgr.y (mgr.34107) 8860 : cluster [DBG] pgmap v9727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:39 smithi115 bash[46934]: cluster 2022-04-22T21:45:38.074061+0000 mgr.y (mgr.34107) 8860 : cluster [DBG] pgmap v9727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:40 smithi121 bash[36902]: audit 2022-04-22T21:45:39.849254+0000 mon.a (mon.0) 12603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:40 smithi115 bash[45495]: audit 2022-04-22T21:45:39.849254+0000 mon.a (mon.0) 12603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:40 smithi115 bash[46934]: audit 2022-04-22T21:45:39.849254+0000 mon.a (mon.0) 12603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:41 smithi121 bash[36902]: cluster 2022-04-22T21:45:40.075172+0000 mgr.y (mgr.34107) 8861 : cluster [DBG] pgmap v9728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:41 smithi121 bash[36902]: audit 2022-04-22T21:45:40.852199+0000 mon.a (mon.0) 12604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:41 smithi115 bash[45495]: cluster 2022-04-22T21:45:40.075172+0000 mgr.y (mgr.34107) 8861 : cluster [DBG] pgmap v9728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:41 smithi115 bash[45495]: audit 2022-04-22T21:45:40.852199+0000 mon.a (mon.0) 12604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:41 smithi115 bash[46934]: cluster 2022-04-22T21:45:40.075172+0000 mgr.y (mgr.34107) 8861 : cluster [DBG] pgmap v9728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:41 smithi115 bash[46934]: audit 2022-04-22T21:45:40.852199+0000 mon.a (mon.0) 12604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:42 smithi121 bash[36902]: cluster 2022-04-22T21:45:42.076482+0000 mgr.y (mgr.34107) 8862 : cluster [DBG] pgmap v9729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:42 smithi115 bash[45495]: cluster 2022-04-22T21:45:42.076482+0000 mgr.y (mgr.34107) 8862 : cluster [DBG] pgmap v9729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:42 smithi115 bash[46934]: cluster 2022-04-22T21:45:42.076482+0000 mgr.y (mgr.34107) 8862 : cluster [DBG] pgmap v9729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:42.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:44.494Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:45 smithi121 bash[36902]: cluster 2022-04-22T21:45:44.077572+0000 mgr.y (mgr.34107) 8863 : cluster [DBG] pgmap v9730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:45 smithi121 bash[36902]: audit 2022-04-22T21:45:44.852411+0000 mon.a (mon.0) 12605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:45.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:45 smithi115 bash[46934]: cluster 2022-04-22T21:45:44.077572+0000 mgr.y (mgr.34107) 8863 : cluster [DBG] pgmap v9730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:45.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:45 smithi115 bash[46934]: audit 2022-04-22T21:45:44.852411+0000 mon.a (mon.0) 12605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:45 smithi115 bash[45495]: cluster 2022-04-22T21:45:44.077572+0000 mgr.y (mgr.34107) 8863 : cluster [DBG] pgmap v9730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:45 smithi115 bash[45495]: audit 2022-04-22T21:45:44.852411+0000 mon.a (mon.0) 12605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:46 smithi121 bash[36902]: audit 2022-04-22T21:45:45.855947+0000 mon.a (mon.0) 12606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:46 smithi121 bash[36902]: cluster 2022-04-22T21:45:46.078819+0000 mgr.y (mgr.34107) 8864 : cluster [DBG] pgmap v9731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:46 smithi115 bash[45495]: audit 2022-04-22T21:45:45.855947+0000 mon.a (mon.0) 12606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:46 smithi115 bash[45495]: cluster 2022-04-22T21:45:46.078819+0000 mgr.y (mgr.34107) 8864 : cluster [DBG] pgmap v9731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:46 smithi115 bash[46934]: audit 2022-04-22T21:45:45.855947+0000 mon.a (mon.0) 12606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:46 smithi115 bash[46934]: cluster 2022-04-22T21:45:46.078819+0000 mgr.y (mgr.34107) 8864 : cluster [DBG] pgmap v9731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:49.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:49 smithi121 bash[36902]: cluster 2022-04-22T21:45:48.079956+0000 mgr.y (mgr.34107) 8865 : cluster [DBG] pgmap v9732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:49.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:49 smithi115 bash[45495]: cluster 2022-04-22T21:45:48.079956+0000 mgr.y (mgr.34107) 8865 : cluster [DBG] pgmap v9732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:49.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:49 smithi115 bash[46934]: cluster 2022-04-22T21:45:48.079956+0000 mgr.y (mgr.34107) 8865 : cluster [DBG] pgmap v9732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:50 smithi121 bash[36902]: audit 2022-04-22T21:45:49.855592+0000 mon.a (mon.0) 12607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:50 smithi115 bash[45495]: audit 2022-04-22T21:45:49.855592+0000 mon.a (mon.0) 12607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:50 smithi115 bash[46934]: audit 2022-04-22T21:45:49.855592+0000 mon.a (mon.0) 12607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:51 smithi121 bash[36902]: cluster 2022-04-22T21:45:50.081028+0000 mgr.y (mgr.34107) 8866 : cluster [DBG] pgmap v9733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:51 smithi121 bash[36902]: audit 2022-04-22T21:45:50.859630+0000 mon.a (mon.0) 12608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:51 smithi115 bash[45495]: cluster 2022-04-22T21:45:50.081028+0000 mgr.y (mgr.34107) 8866 : cluster [DBG] pgmap v9733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:51.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:51 smithi115 bash[45495]: audit 2022-04-22T21:45:50.859630+0000 mon.a (mon.0) 12608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:51 smithi115 bash[46934]: cluster 2022-04-22T21:45:50.081028+0000 mgr.y (mgr.34107) 8866 : cluster [DBG] pgmap v9733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:51.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:51 smithi115 bash[46934]: audit 2022-04-22T21:45:50.859630+0000 mon.a (mon.0) 12608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:52 smithi121 bash[36902]: cluster 2022-04-22T21:45:52.082311+0000 mgr.y (mgr.34107) 8867 : cluster [DBG] pgmap v9734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:52 smithi115 bash[45495]: cluster 2022-04-22T21:45:52.082311+0000 mgr.y (mgr.34107) 8867 : cluster [DBG] pgmap v9734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:52 smithi115 bash[46934]: cluster 2022-04-22T21:45:52.082311+0000 mgr.y (mgr.34107) 8867 : cluster [DBG] pgmap v9734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:52.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:45:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:45:54.494Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:45:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:55 smithi121 bash[36902]: cluster 2022-04-22T21:45:54.083445+0000 mgr.y (mgr.34107) 8868 : cluster [DBG] pgmap v9735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:55 smithi121 bash[36902]: audit 2022-04-22T21:45:54.858721+0000 mon.a (mon.0) 12609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:55 smithi115 bash[45495]: cluster 2022-04-22T21:45:54.083445+0000 mgr.y (mgr.34107) 8868 : cluster [DBG] pgmap v9735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:55 smithi115 bash[45495]: audit 2022-04-22T21:45:54.858721+0000 mon.a (mon.0) 12609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:55 smithi115 bash[46934]: cluster 2022-04-22T21:45:54.083445+0000 mgr.y (mgr.34107) 8868 : cluster [DBG] pgmap v9735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:55 smithi115 bash[46934]: audit 2022-04-22T21:45:54.858721+0000 mon.a (mon.0) 12609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:45:57.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:56 smithi121 bash[36902]: audit 2022-04-22T21:45:55.863416+0000 mon.a (mon.0) 12610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:57.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:56 smithi121 bash[36902]: cluster 2022-04-22T21:45:56.084595+0000 mgr.y (mgr.34107) 8869 : cluster [DBG] pgmap v9736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:56 smithi115 bash[45495]: audit 2022-04-22T21:45:55.863416+0000 mon.a (mon.0) 12610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:56 smithi115 bash[45495]: cluster 2022-04-22T21:45:56.084595+0000 mgr.y (mgr.34107) 8869 : cluster [DBG] pgmap v9736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:56 smithi115 bash[46934]: audit 2022-04-22T21:45:55.863416+0000 mon.a (mon.0) 12610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:45:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:56 smithi115 bash[46934]: cluster 2022-04-22T21:45:56.084595+0000 mgr.y (mgr.34107) 8869 : cluster [DBG] pgmap v9736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:45:59 smithi121 bash[36902]: cluster 2022-04-22T21:45:58.085598+0000 mgr.y (mgr.34107) 8870 : cluster [DBG] pgmap v9737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:45:59 smithi115 bash[45495]: cluster 2022-04-22T21:45:58.085598+0000 mgr.y (mgr.34107) 8870 : cluster [DBG] pgmap v9737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:45:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:45:59 smithi115 bash[46934]: cluster 2022-04-22T21:45:58.085598+0000 mgr.y (mgr.34107) 8870 : cluster [DBG] pgmap v9737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:00 smithi121 bash[36902]: audit 2022-04-22T21:45:59.861971+0000 mon.a (mon.0) 12611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:00 smithi115 bash[45495]: audit 2022-04-22T21:45:59.861971+0000 mon.a (mon.0) 12611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:00 smithi115 bash[46934]: audit 2022-04-22T21:45:59.861971+0000 mon.a (mon.0) 12611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:01 smithi121 bash[36902]: cluster 2022-04-22T21:46:00.086630+0000 mgr.y (mgr.34107) 8871 : cluster [DBG] pgmap v9738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:01 smithi121 bash[36902]: audit 2022-04-22T21:46:00.866993+0000 mon.a (mon.0) 12612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:01 smithi115 bash[45495]: cluster 2022-04-22T21:46:00.086630+0000 mgr.y (mgr.34107) 8871 : cluster [DBG] pgmap v9738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:01 smithi115 bash[45495]: audit 2022-04-22T21:46:00.866993+0000 mon.a (mon.0) 12612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:01 smithi115 bash[46934]: cluster 2022-04-22T21:46:00.086630+0000 mgr.y (mgr.34107) 8871 : cluster [DBG] pgmap v9738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:01 smithi115 bash[46934]: audit 2022-04-22T21:46:00.866993+0000 mon.a (mon.0) 12612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:02.104 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:46:02.441 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:46:02.441 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 62s ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:46:02.441 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 62s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 62s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 62s ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 62s ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 62s ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:46:02.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 62s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 62s ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 62s ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 62s ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:46:02.443 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:46:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:02 smithi121 bash[36902]: audit 2022-04-22T21:46:02.085242+0000 mgr.y (mgr.34107) 8872 : audit [DBG] from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:02 smithi121 bash[36902]: cluster 2022-04-22T21:46:02.087362+0000 mgr.y (mgr.34107) 8873 : cluster [DBG] pgmap v9739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:02.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:02 smithi115 bash[45495]: audit 2022-04-22T21:46:02.085242+0000 mgr.y (mgr.34107) 8872 : audit [DBG] from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:02.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:02 smithi115 bash[45495]: cluster 2022-04-22T21:46:02.087362+0000 mgr.y (mgr.34107) 8873 : cluster [DBG] pgmap v9739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:02 smithi115 bash[46934]: audit 2022-04-22T21:46:02.085242+0000 mgr.y (mgr.34107) 8872 : audit [DBG] from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:02.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:02 smithi115 bash[46934]: cluster 2022-04-22T21:46:02.087362+0000 mgr.y (mgr.34107) 8873 : cluster [DBG] pgmap v9739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:46:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:02.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:03.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:03 smithi121 bash[36902]: audit 2022-04-22T21:46:02.424162+0000 mgr.y (mgr.34107) 8874 : audit [DBG] from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:03.547 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:03 smithi121 bash[36902]: audit 2022-04-22T21:46:02.851801+0000 mon.a (mon.0) 12613 : audit [DBG] from='client.? 172.21.15.115:0/1516756223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:03 smithi115 bash[45495]: audit 2022-04-22T21:46:02.424162+0000 mgr.y (mgr.34107) 8874 : audit [DBG] from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:03.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:03 smithi115 bash[45495]: audit 2022-04-22T21:46:02.851801+0000 mon.a (mon.0) 12613 : audit [DBG] from='client.? 172.21.15.115:0/1516756223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:03 smithi115 bash[46934]: audit 2022-04-22T21:46:02.424162+0000 mgr.y (mgr.34107) 8874 : audit [DBG] from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:03 smithi115 bash[46934]: audit 2022-04-22T21:46:02.851801+0000 mon.a (mon.0) 12613 : audit [DBG] from='client.? 172.21.15.115:0/1516756223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:04.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:04 smithi115 bash[46934]: cluster 2022-04-22T21:46:04.088393+0000 mgr.y (mgr.34107) 8875 : cluster [DBG] pgmap v9740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:04.498 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:04 smithi115 bash[45495]: cluster 2022-04-22T21:46:04.088393+0000 mgr.y (mgr.34107) 8875 : cluster [DBG] pgmap v9740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:04.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:04 smithi121 bash[36902]: cluster 2022-04-22T21:46:04.088393+0000 mgr.y (mgr.34107) 8875 : cluster [DBG] pgmap v9740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:04.494Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:05 smithi121 bash[36902]: audit 2022-04-22T21:46:04.865069+0000 mon.a (mon.0) 12614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:05 smithi115 bash[45495]: audit 2022-04-22T21:46:04.865069+0000 mon.a (mon.0) 12614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:05 smithi115 bash[46934]: audit 2022-04-22T21:46:04.865069+0000 mon.a (mon.0) 12614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:06 smithi121 bash[36902]: audit 2022-04-22T21:46:05.870978+0000 mon.a (mon.0) 12615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:06 smithi121 bash[36902]: cluster 2022-04-22T21:46:06.089594+0000 mgr.y (mgr.34107) 8876 : cluster [DBG] pgmap v9741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:06 smithi115 bash[45495]: audit 2022-04-22T21:46:05.870978+0000 mon.a (mon.0) 12615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:06 smithi115 bash[45495]: cluster 2022-04-22T21:46:06.089594+0000 mgr.y (mgr.34107) 8876 : cluster [DBG] pgmap v9741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:06 smithi115 bash[46934]: audit 2022-04-22T21:46:05.870978+0000 mon.a (mon.0) 12615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:06 smithi115 bash[46934]: cluster 2022-04-22T21:46:06.089594+0000 mgr.y (mgr.34107) 8876 : cluster [DBG] pgmap v9741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:09 smithi121 bash[36902]: cluster 2022-04-22T21:46:08.090743+0000 mgr.y (mgr.34107) 8877 : cluster [DBG] pgmap v9742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:09 smithi121 bash[36902]: audit 2022-04-22T21:46:08.845600+0000 mon.a (mon.0) 12616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:09 smithi121 bash[36902]: audit 2022-04-22T21:46:08.847761+0000 mon.a (mon.0) 12617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:46:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:09 smithi121 bash[36902]: audit 2022-04-22T21:46:08.849023+0000 mon.a (mon.0) 12618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[45495]: cluster 2022-04-22T21:46:08.090743+0000 mgr.y (mgr.34107) 8877 : cluster [DBG] pgmap v9742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[45495]: audit 2022-04-22T21:46:08.845600+0000 mon.a (mon.0) 12616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[45495]: audit 2022-04-22T21:46:08.847761+0000 mon.a (mon.0) 12617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[45495]: audit 2022-04-22T21:46:08.849023+0000 mon.a (mon.0) 12618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[46934]: cluster 2022-04-22T21:46:08.090743+0000 mgr.y (mgr.34107) 8877 : cluster [DBG] pgmap v9742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[46934]: audit 2022-04-22T21:46:08.845600+0000 mon.a (mon.0) 12616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:46:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[46934]: audit 2022-04-22T21:46:08.847761+0000 mon.a (mon.0) 12617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:46:09.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:09 smithi115 bash[46934]: audit 2022-04-22T21:46:08.849023+0000 mon.a (mon.0) 12618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:46:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:10 smithi121 bash[36902]: audit 2022-04-22T21:46:09.218292+0000 mon.a (mon.0) 12619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:46:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:10 smithi121 bash[36902]: audit 2022-04-22T21:46:09.868116+0000 mon.a (mon.0) 12620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:10 smithi121 bash[36902]: cluster 2022-04-22T21:46:10.091773+0000 mgr.y (mgr.34107) 8878 : cluster [DBG] pgmap v9743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[45495]: audit 2022-04-22T21:46:09.218292+0000 mon.a (mon.0) 12619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:46:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[45495]: audit 2022-04-22T21:46:09.868116+0000 mon.a (mon.0) 12620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[45495]: cluster 2022-04-22T21:46:10.091773+0000 mgr.y (mgr.34107) 8878 : cluster [DBG] pgmap v9743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[46934]: audit 2022-04-22T21:46:09.218292+0000 mon.a (mon.0) 12619 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:46:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[46934]: audit 2022-04-22T21:46:09.868116+0000 mon.a (mon.0) 12620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:10 smithi115 bash[46934]: cluster 2022-04-22T21:46:10.091773+0000 mgr.y (mgr.34107) 8878 : cluster [DBG] pgmap v9743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:12.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:11 smithi121 bash[36902]: audit 2022-04-22T21:46:10.874906+0000 mon.a (mon.0) 12621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:11 smithi115 bash[45495]: audit 2022-04-22T21:46:10.874906+0000 mon.a (mon.0) 12621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:11 smithi115 bash[46934]: audit 2022-04-22T21:46:10.874906+0000 mon.a (mon.0) 12621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:12 smithi121 bash[36902]: cluster 2022-04-22T21:46:12.092837+0000 mgr.y (mgr.34107) 8879 : cluster [DBG] pgmap v9744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:12 smithi115 bash[45495]: cluster 2022-04-22T21:46:12.092837+0000 mgr.y (mgr.34107) 8879 : cluster [DBG] pgmap v9744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:12.635Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:12 smithi115 bash[46934]: cluster 2022-04-22T21:46:12.092837+0000 mgr.y (mgr.34107) 8879 : cluster [DBG] pgmap v9744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:14.496Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:15 smithi121 bash[36902]: cluster 2022-04-22T21:46:14.093894+0000 mgr.y (mgr.34107) 8880 : cluster [DBG] pgmap v9745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:15 smithi121 bash[36902]: audit 2022-04-22T21:46:14.871175+0000 mon.a (mon.0) 12622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:15 smithi115 bash[45495]: cluster 2022-04-22T21:46:14.093894+0000 mgr.y (mgr.34107) 8880 : cluster [DBG] pgmap v9745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:15 smithi115 bash[45495]: audit 2022-04-22T21:46:14.871175+0000 mon.a (mon.0) 12622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:15 smithi115 bash[46934]: cluster 2022-04-22T21:46:14.093894+0000 mgr.y (mgr.34107) 8880 : cluster [DBG] pgmap v9745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:15 smithi115 bash[46934]: audit 2022-04-22T21:46:14.871175+0000 mon.a (mon.0) 12622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:16 smithi121 bash[36902]: audit 2022-04-22T21:46:15.878542+0000 mon.a (mon.0) 12623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:16 smithi121 bash[36902]: cluster 2022-04-22T21:46:16.094955+0000 mgr.y (mgr.34107) 8881 : cluster [DBG] pgmap v9746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:16 smithi121 bash[36902]: audit 2022-04-22T21:46:16.264400+0000 mon.a (mon.0) 12624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:46:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:16 smithi121 bash[36902]: audit 2022-04-22T21:46:16.265659+0000 mon.a (mon.0) 12625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[45495]: audit 2022-04-22T21:46:15.878542+0000 mon.a (mon.0) 12623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[45495]: cluster 2022-04-22T21:46:16.094955+0000 mgr.y (mgr.34107) 8881 : cluster [DBG] pgmap v9746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[45495]: audit 2022-04-22T21:46:16.264400+0000 mon.a (mon.0) 12624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[45495]: audit 2022-04-22T21:46:16.265659+0000 mon.a (mon.0) 12625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[46934]: audit 2022-04-22T21:46:15.878542+0000 mon.a (mon.0) 12623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[46934]: cluster 2022-04-22T21:46:16.094955+0000 mgr.y (mgr.34107) 8881 : cluster [DBG] pgmap v9746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[46934]: audit 2022-04-22T21:46:16.264400+0000 mon.a (mon.0) 12624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:46:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:16 smithi115 bash[46934]: audit 2022-04-22T21:46:16.265659+0000 mon.a (mon.0) 12625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:46:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:18 smithi121 bash[36902]: cluster 2022-04-22T21:46:18.096058+0000 mgr.y (mgr.34107) 8882 : cluster [DBG] pgmap v9747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:18 smithi115 bash[45495]: cluster 2022-04-22T21:46:18.096058+0000 mgr.y (mgr.34107) 8882 : cluster [DBG] pgmap v9747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:18 smithi115 bash[46934]: cluster 2022-04-22T21:46:18.096058+0000 mgr.y (mgr.34107) 8882 : cluster [DBG] pgmap v9747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:19 smithi121 bash[36902]: audit 2022-04-22T21:46:19.874360+0000 mon.a (mon.0) 12626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:19 smithi115 bash[45495]: audit 2022-04-22T21:46:19.874360+0000 mon.a (mon.0) 12626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:20.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:19 smithi115 bash[46934]: audit 2022-04-22T21:46:19.874360+0000 mon.a (mon.0) 12626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:20 smithi121 bash[36902]: cluster 2022-04-22T21:46:20.097037+0000 mgr.y (mgr.34107) 8883 : cluster [DBG] pgmap v9748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:20 smithi121 bash[36902]: audit 2022-04-22T21:46:20.882124+0000 mon.a (mon.0) 12627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:20 smithi115 bash[46934]: cluster 2022-04-22T21:46:20.097037+0000 mgr.y (mgr.34107) 8883 : cluster [DBG] pgmap v9748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:20 smithi115 bash[46934]: audit 2022-04-22T21:46:20.882124+0000 mon.a (mon.0) 12627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:20 smithi115 bash[45495]: cluster 2022-04-22T21:46:20.097037+0000 mgr.y (mgr.34107) 8883 : cluster [DBG] pgmap v9748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:21.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:20 smithi115 bash[45495]: audit 2022-04-22T21:46:20.882124+0000 mon.a (mon.0) 12627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:22 smithi121 bash[36902]: cluster 2022-04-22T21:46:22.098144+0000 mgr.y (mgr.34107) 8884 : cluster [DBG] pgmap v9749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:22 smithi115 bash[46934]: cluster 2022-04-22T21:46:22.098144+0000 mgr.y (mgr.34107) 8884 : cluster [DBG] pgmap v9749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:22.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:22 smithi115 bash[45495]: cluster 2022-04-22T21:46:22.098144+0000 mgr.y (mgr.34107) 8884 : cluster [DBG] pgmap v9749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:22.635Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:24.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:24 smithi115 bash[45495]: cluster 2022-04-22T21:46:24.099185+0000 mgr.y (mgr.34107) 8885 : cluster [DBG] pgmap v9750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:24.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:24 smithi115 bash[46934]: cluster 2022-04-22T21:46:24.099185+0000 mgr.y (mgr.34107) 8885 : cluster [DBG] pgmap v9750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:24 smithi121 bash[36902]: cluster 2022-04-22T21:46:24.099185+0000 mgr.y (mgr.34107) 8885 : cluster [DBG] pgmap v9750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:24.496Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:25 smithi121 bash[36902]: audit 2022-04-22T21:46:24.877465+0000 mon.a (mon.0) 12628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:25 smithi115 bash[45495]: audit 2022-04-22T21:46:24.877465+0000 mon.a (mon.0) 12628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:25 smithi115 bash[46934]: audit 2022-04-22T21:46:24.877465+0000 mon.a (mon.0) 12628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:26 smithi121 bash[36902]: audit 2022-04-22T21:46:25.885711+0000 mon.a (mon.0) 12629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:26 smithi121 bash[36902]: cluster 2022-04-22T21:46:26.100291+0000 mgr.y (mgr.34107) 8886 : cluster [DBG] pgmap v9751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:26 smithi115 bash[45495]: audit 2022-04-22T21:46:25.885711+0000 mon.a (mon.0) 12629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:26 smithi115 bash[45495]: cluster 2022-04-22T21:46:26.100291+0000 mgr.y (mgr.34107) 8886 : cluster [DBG] pgmap v9751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:26 smithi115 bash[46934]: audit 2022-04-22T21:46:25.885711+0000 mon.a (mon.0) 12629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:26 smithi115 bash[46934]: cluster 2022-04-22T21:46:26.100291+0000 mgr.y (mgr.34107) 8886 : cluster [DBG] pgmap v9751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:28 smithi121 bash[36902]: cluster 2022-04-22T21:46:28.101348+0000 mgr.y (mgr.34107) 8887 : cluster [DBG] pgmap v9752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:28 smithi115 bash[45495]: cluster 2022-04-22T21:46:28.101348+0000 mgr.y (mgr.34107) 8887 : cluster [DBG] pgmap v9752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:28 smithi115 bash[46934]: cluster 2022-04-22T21:46:28.101348+0000 mgr.y (mgr.34107) 8887 : cluster [DBG] pgmap v9752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:29 smithi121 bash[36902]: audit 2022-04-22T21:46:29.880555+0000 mon.a (mon.0) 12630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:29 smithi115 bash[45495]: audit 2022-04-22T21:46:29.880555+0000 mon.a (mon.0) 12630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:30.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:29 smithi115 bash[46934]: audit 2022-04-22T21:46:29.880555+0000 mon.a (mon.0) 12630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:30 smithi121 bash[36902]: cluster 2022-04-22T21:46:30.102365+0000 mgr.y (mgr.34107) 8888 : cluster [DBG] pgmap v9753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:30 smithi121 bash[36902]: audit 2022-04-22T21:46:30.889436+0000 mon.a (mon.0) 12631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:30 smithi115 bash[45495]: cluster 2022-04-22T21:46:30.102365+0000 mgr.y (mgr.34107) 8888 : cluster [DBG] pgmap v9753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:30 smithi115 bash[45495]: audit 2022-04-22T21:46:30.889436+0000 mon.a (mon.0) 12631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:30 smithi115 bash[46934]: cluster 2022-04-22T21:46:30.102365+0000 mgr.y (mgr.34107) 8888 : cluster [DBG] pgmap v9753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:30 smithi115 bash[46934]: audit 2022-04-22T21:46:30.889436+0000 mon.a (mon.0) 12631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:32 smithi121 bash[36902]: cluster 2022-04-22T21:46:32.103593+0000 mgr.y (mgr.34107) 8889 : cluster [DBG] pgmap v9754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:32 smithi115 bash[45495]: cluster 2022-04-22T21:46:32.103593+0000 mgr.y (mgr.34107) 8889 : cluster [DBG] pgmap v9754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:32 smithi115 bash[46934]: cluster 2022-04-22T21:46:32.103593+0000 mgr.y (mgr.34107) 8889 : cluster [DBG] pgmap v9754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:32.635Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:33.227 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:46:33.567 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:46:33.568 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 94s ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:46:33.568 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:46:33.568 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 94s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:46:33.568 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:46:33.568 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 94s ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 94s ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 94s ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 94s ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 94s ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:46:33.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 94s ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 94s ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 94s ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:46:33.570 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:46:33.974 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:46:33.974 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:46:33.974 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:46:33.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:33.974 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:46:33.975 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:46:33.976 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:46:33.976 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:46:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:34 smithi115 bash[45495]: audit 2022-04-22T21:46:33.973161+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.115:0/213017684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:34.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:34 smithi115 bash[46934]: audit 2022-04-22T21:46:33.973161+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.115:0/213017684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:34 smithi121 bash[36902]: audit 2022-04-22T21:46:33.973161+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.115:0/213017684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:46:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:34.497Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[45495]: audit 2022-04-22T21:46:33.206763+0000 mgr.y (mgr.34107) 8890 : audit [DBG] from='client.54517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[45495]: audit 2022-04-22T21:46:33.545714+0000 mgr.y (mgr.34107) 8891 : audit [DBG] from='client.44592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[45495]: cluster 2022-04-22T21:46:34.104598+0000 mgr.y (mgr.34107) 8892 : cluster [DBG] pgmap v9755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[45495]: audit 2022-04-22T21:46:34.883545+0000 mon.a (mon.0) 12632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[46934]: audit 2022-04-22T21:46:33.206763+0000 mgr.y (mgr.34107) 8890 : audit [DBG] from='client.54517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[46934]: audit 2022-04-22T21:46:33.545714+0000 mgr.y (mgr.34107) 8891 : audit [DBG] from='client.44592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[46934]: cluster 2022-04-22T21:46:34.104598+0000 mgr.y (mgr.34107) 8892 : cluster [DBG] pgmap v9755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:35.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:35 smithi115 bash[46934]: audit 2022-04-22T21:46:34.883545+0000 mon.a (mon.0) 12632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:35 smithi121 bash[36902]: audit 2022-04-22T21:46:33.206763+0000 mgr.y (mgr.34107) 8890 : audit [DBG] from='client.54517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:35 smithi121 bash[36902]: audit 2022-04-22T21:46:33.545714+0000 mgr.y (mgr.34107) 8891 : audit [DBG] from='client.44592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:46:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:35 smithi121 bash[36902]: cluster 2022-04-22T21:46:34.104598+0000 mgr.y (mgr.34107) 8892 : cluster [DBG] pgmap v9755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:35 smithi121 bash[36902]: audit 2022-04-22T21:46:34.883545+0000 mon.a (mon.0) 12632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:36 smithi121 bash[36902]: audit 2022-04-22T21:46:35.893229+0000 mon.a (mon.0) 12633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:36 smithi121 bash[36902]: cluster 2022-04-22T21:46:36.105655+0000 mgr.y (mgr.34107) 8893 : cluster [DBG] pgmap v9756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:36 smithi115 bash[45495]: audit 2022-04-22T21:46:35.893229+0000 mon.a (mon.0) 12633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:36 smithi115 bash[45495]: cluster 2022-04-22T21:46:36.105655+0000 mgr.y (mgr.34107) 8893 : cluster [DBG] pgmap v9756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:36 smithi115 bash[46934]: audit 2022-04-22T21:46:35.893229+0000 mon.a (mon.0) 12633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:36 smithi115 bash[46934]: cluster 2022-04-22T21:46:36.105655+0000 mgr.y (mgr.34107) 8893 : cluster [DBG] pgmap v9756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:38 smithi121 bash[36902]: cluster 2022-04-22T21:46:38.106730+0000 mgr.y (mgr.34107) 8894 : cluster [DBG] pgmap v9757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:38 smithi115 bash[45495]: cluster 2022-04-22T21:46:38.106730+0000 mgr.y (mgr.34107) 8894 : cluster [DBG] pgmap v9757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:38 smithi115 bash[46934]: cluster 2022-04-22T21:46:38.106730+0000 mgr.y (mgr.34107) 8894 : cluster [DBG] pgmap v9757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:39 smithi121 bash[36902]: audit 2022-04-22T21:46:39.886468+0000 mon.a (mon.0) 12634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:39 smithi115 bash[45495]: audit 2022-04-22T21:46:39.886468+0000 mon.a (mon.0) 12634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:39 smithi115 bash[46934]: audit 2022-04-22T21:46:39.886468+0000 mon.a (mon.0) 12634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:40 smithi121 bash[36902]: cluster 2022-04-22T21:46:40.107628+0000 mgr.y (mgr.34107) 8895 : cluster [DBG] pgmap v9758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:40 smithi121 bash[36902]: audit 2022-04-22T21:46:40.896988+0000 mon.a (mon.0) 12635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:40 smithi115 bash[46934]: cluster 2022-04-22T21:46:40.107628+0000 mgr.y (mgr.34107) 8895 : cluster [DBG] pgmap v9758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:40 smithi115 bash[46934]: audit 2022-04-22T21:46:40.896988+0000 mon.a (mon.0) 12635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:40 smithi115 bash[45495]: cluster 2022-04-22T21:46:40.107628+0000 mgr.y (mgr.34107) 8895 : cluster [DBG] pgmap v9758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:40 smithi115 bash[45495]: audit 2022-04-22T21:46:40.896988+0000 mon.a (mon.0) 12635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:42 smithi121 bash[36902]: cluster 2022-04-22T21:46:42.108776+0000 mgr.y (mgr.34107) 8896 : cluster [DBG] pgmap v9759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:42 smithi115 bash[45495]: cluster 2022-04-22T21:46:42.108776+0000 mgr.y (mgr.34107) 8896 : cluster [DBG] pgmap v9759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:42 smithi115 bash[46934]: cluster 2022-04-22T21:46:42.108776+0000 mgr.y (mgr.34107) 8896 : cluster [DBG] pgmap v9759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:42.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:44.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:44 smithi115 bash[45495]: cluster 2022-04-22T21:46:44.109830+0000 mgr.y (mgr.34107) 8897 : cluster [DBG] pgmap v9760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:44.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:44 smithi115 bash[46934]: cluster 2022-04-22T21:46:44.109830+0000 mgr.y (mgr.34107) 8897 : cluster [DBG] pgmap v9760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:44 smithi121 bash[36902]: cluster 2022-04-22T21:46:44.109830+0000 mgr.y (mgr.34107) 8897 : cluster [DBG] pgmap v9760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:44.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:45.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:45 smithi121 bash[36902]: audit 2022-04-22T21:46:44.889763+0000 mon.a (mon.0) 12636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:45 smithi115 bash[45495]: audit 2022-04-22T21:46:44.889763+0000 mon.a (mon.0) 12636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:45 smithi115 bash[46934]: audit 2022-04-22T21:46:44.889763+0000 mon.a (mon.0) 12636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:46 smithi121 bash[36902]: audit 2022-04-22T21:46:45.900453+0000 mon.a (mon.0) 12637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:46 smithi121 bash[36902]: cluster 2022-04-22T21:46:46.110977+0000 mgr.y (mgr.34107) 8898 : cluster [DBG] pgmap v9761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:46 smithi115 bash[45495]: audit 2022-04-22T21:46:45.900453+0000 mon.a (mon.0) 12637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:46 smithi115 bash[45495]: cluster 2022-04-22T21:46:46.110977+0000 mgr.y (mgr.34107) 8898 : cluster [DBG] pgmap v9761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:46 smithi115 bash[46934]: audit 2022-04-22T21:46:45.900453+0000 mon.a (mon.0) 12637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:46 smithi115 bash[46934]: cluster 2022-04-22T21:46:46.110977+0000 mgr.y (mgr.34107) 8898 : cluster [DBG] pgmap v9761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:48 smithi121 bash[36902]: cluster 2022-04-22T21:46:48.112160+0000 mgr.y (mgr.34107) 8899 : cluster [DBG] pgmap v9762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:48 smithi115 bash[45495]: cluster 2022-04-22T21:46:48.112160+0000 mgr.y (mgr.34107) 8899 : cluster [DBG] pgmap v9762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:48 smithi115 bash[46934]: cluster 2022-04-22T21:46:48.112160+0000 mgr.y (mgr.34107) 8899 : cluster [DBG] pgmap v9762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:49 smithi121 bash[36902]: audit 2022-04-22T21:46:49.892903+0000 mon.a (mon.0) 12638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:49 smithi115 bash[45495]: audit 2022-04-22T21:46:49.892903+0000 mon.a (mon.0) 12638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:49 smithi115 bash[46934]: audit 2022-04-22T21:46:49.892903+0000 mon.a (mon.0) 12638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:50 smithi121 bash[36902]: cluster 2022-04-22T21:46:50.113142+0000 mgr.y (mgr.34107) 8900 : cluster [DBG] pgmap v9763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:50 smithi121 bash[36902]: audit 2022-04-22T21:46:50.903509+0000 mon.a (mon.0) 12639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:50 smithi115 bash[46934]: cluster 2022-04-22T21:46:50.113142+0000 mgr.y (mgr.34107) 8900 : cluster [DBG] pgmap v9763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:50 smithi115 bash[46934]: audit 2022-04-22T21:46:50.903509+0000 mon.a (mon.0) 12639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:50 smithi115 bash[45495]: cluster 2022-04-22T21:46:50.113142+0000 mgr.y (mgr.34107) 8900 : cluster [DBG] pgmap v9763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:50 smithi115 bash[45495]: audit 2022-04-22T21:46:50.903509+0000 mon.a (mon.0) 12639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:52 smithi121 bash[36902]: cluster 2022-04-22T21:46:52.114277+0000 mgr.y (mgr.34107) 8901 : cluster [DBG] pgmap v9764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:52 smithi115 bash[46934]: cluster 2022-04-22T21:46:52.114277+0000 mgr.y (mgr.34107) 8901 : cluster [DBG] pgmap v9764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:52 smithi115 bash[45495]: cluster 2022-04-22T21:46:52.114277+0000 mgr.y (mgr.34107) 8901 : cluster [DBG] pgmap v9764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:52.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:54.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:54 smithi115 bash[45495]: cluster 2022-04-22T21:46:54.115296+0000 mgr.y (mgr.34107) 8902 : cluster [DBG] pgmap v9765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:54.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:54 smithi115 bash[46934]: cluster 2022-04-22T21:46:54.115296+0000 mgr.y (mgr.34107) 8902 : cluster [DBG] pgmap v9765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:54 smithi121 bash[36902]: cluster 2022-04-22T21:46:54.115296+0000 mgr.y (mgr.34107) 8902 : cluster [DBG] pgmap v9765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:46:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:46:54.498Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:46:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:55 smithi121 bash[36902]: audit 2022-04-22T21:46:54.896231+0000 mon.a (mon.0) 12640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:55 smithi115 bash[45495]: audit 2022-04-22T21:46:54.896231+0000 mon.a (mon.0) 12640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:55 smithi115 bash[46934]: audit 2022-04-22T21:46:54.896231+0000 mon.a (mon.0) 12640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:46:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:56 smithi121 bash[36902]: audit 2022-04-22T21:46:55.906168+0000 mon.a (mon.0) 12641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:57.055 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:56 smithi121 bash[36902]: cluster 2022-04-22T21:46:56.116381+0000 mgr.y (mgr.34107) 8903 : cluster [DBG] pgmap v9766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:57.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:56 smithi115 bash[45495]: audit 2022-04-22T21:46:55.906168+0000 mon.a (mon.0) 12641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:57.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:56 smithi115 bash[45495]: cluster 2022-04-22T21:46:56.116381+0000 mgr.y (mgr.34107) 8903 : cluster [DBG] pgmap v9766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:57.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:56 smithi115 bash[46934]: audit 2022-04-22T21:46:55.906168+0000 mon.a (mon.0) 12641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:46:57.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:56 smithi115 bash[46934]: cluster 2022-04-22T21:46:56.116381+0000 mgr.y (mgr.34107) 8903 : cluster [DBG] pgmap v9766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:58 smithi121 bash[36902]: cluster 2022-04-22T21:46:58.117512+0000 mgr.y (mgr.34107) 8904 : cluster [DBG] pgmap v9767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:58 smithi115 bash[45495]: cluster 2022-04-22T21:46:58.117512+0000 mgr.y (mgr.34107) 8904 : cluster [DBG] pgmap v9767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:46:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:58 smithi115 bash[46934]: cluster 2022-04-22T21:46:58.117512+0000 mgr.y (mgr.34107) 8904 : cluster [DBG] pgmap v9767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:46:59 smithi121 bash[36902]: audit 2022-04-22T21:46:59.899761+0000 mon.a (mon.0) 12642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:46:59 smithi115 bash[45495]: audit 2022-04-22T21:46:59.899761+0000 mon.a (mon.0) 12642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:46:59 smithi115 bash[46934]: audit 2022-04-22T21:46:59.899761+0000 mon.a (mon.0) 12642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:00 smithi121 bash[36902]: cluster 2022-04-22T21:47:00.118518+0000 mgr.y (mgr.34107) 8905 : cluster [DBG] pgmap v9768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:00 smithi121 bash[36902]: audit 2022-04-22T21:47:00.910055+0000 mon.a (mon.0) 12643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:00 smithi115 bash[45495]: cluster 2022-04-22T21:47:00.118518+0000 mgr.y (mgr.34107) 8905 : cluster [DBG] pgmap v9768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:00 smithi115 bash[45495]: audit 2022-04-22T21:47:00.910055+0000 mon.a (mon.0) 12643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:00 smithi115 bash[46934]: cluster 2022-04-22T21:47:00.118518+0000 mgr.y (mgr.34107) 8905 : cluster [DBG] pgmap v9768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:00 smithi115 bash[46934]: audit 2022-04-22T21:47:00.910055+0000 mon.a (mon.0) 12643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:02 smithi121 bash[36902]: cluster 2022-04-22T21:47:02.119841+0000 mgr.y (mgr.34107) 8906 : cluster [DBG] pgmap v9769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:02 smithi115 bash[45495]: cluster 2022-04-22T21:47:02.119841+0000 mgr.y (mgr.34107) 8906 : cluster [DBG] pgmap v9769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:02 smithi115 bash[46934]: cluster 2022-04-22T21:47:02.119841+0000 mgr.y (mgr.34107) 8906 : cluster [DBG] pgmap v9769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:02.639Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:04.345 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:47:04.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:04 smithi115 bash[46934]: cluster 2022-04-22T21:47:04.120795+0000 mgr.y (mgr.34107) 8907 : cluster [DBG] pgmap v9770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:04.524 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:04 smithi115 bash[45495]: cluster 2022-04-22T21:47:04.120795+0000 mgr.y (mgr.34107) 8907 : cluster [DBG] pgmap v9770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:04.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:04 smithi121 bash[36902]: cluster 2022-04-22T21:47:04.120795+0000 mgr.y (mgr.34107) 8907 : cluster [DBG] pgmap v9770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:04.693 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:47:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:47:04.695 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:47:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:04.499Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:05.102 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:47:05.102 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:47:05.102 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:47:05.102 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:05.102 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:47:05.103 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:47:05.104 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:47:05.104 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:05 smithi121 bash[36902]: audit 2022-04-22T21:47:04.326399+0000 mgr.y (mgr.34107) 8908 : audit [DBG] from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:05 smithi121 bash[36902]: audit 2022-04-22T21:47:04.671891+0000 mgr.y (mgr.34107) 8909 : audit [DBG] from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:05 smithi121 bash[36902]: audit 2022-04-22T21:47:04.902928+0000 mon.a (mon.0) 12644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:05 smithi121 bash[36902]: audit 2022-04-22T21:47:05.098288+0000 mon.a (mon.0) 12645 : audit [DBG] from='client.? 172.21.15.115:0/773367074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[45495]: audit 2022-04-22T21:47:04.326399+0000 mgr.y (mgr.34107) 8908 : audit [DBG] from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[45495]: audit 2022-04-22T21:47:04.671891+0000 mgr.y (mgr.34107) 8909 : audit [DBG] from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[45495]: audit 2022-04-22T21:47:04.902928+0000 mon.a (mon.0) 12644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[45495]: audit 2022-04-22T21:47:05.098288+0000 mon.a (mon.0) 12645 : audit [DBG] from='client.? 172.21.15.115:0/773367074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[46934]: audit 2022-04-22T21:47:04.326399+0000 mgr.y (mgr.34107) 8908 : audit [DBG] from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[46934]: audit 2022-04-22T21:47:04.671891+0000 mgr.y (mgr.34107) 8909 : audit [DBG] from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[46934]: audit 2022-04-22T21:47:04.902928+0000 mon.a (mon.0) 12644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:05 smithi115 bash[46934]: audit 2022-04-22T21:47:05.098288+0000 mon.a (mon.0) 12645 : audit [DBG] from='client.? 172.21.15.115:0/773367074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:06 smithi121 bash[36902]: audit 2022-04-22T21:47:05.913772+0000 mon.a (mon.0) 12646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:06 smithi121 bash[36902]: cluster 2022-04-22T21:47:06.121845+0000 mgr.y (mgr.34107) 8910 : cluster [DBG] pgmap v9771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:06 smithi115 bash[45495]: audit 2022-04-22T21:47:05.913772+0000 mon.a (mon.0) 12646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:06 smithi115 bash[45495]: cluster 2022-04-22T21:47:06.121845+0000 mgr.y (mgr.34107) 8910 : cluster [DBG] pgmap v9771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:06 smithi115 bash[46934]: audit 2022-04-22T21:47:05.913772+0000 mon.a (mon.0) 12646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:06 smithi115 bash[46934]: cluster 2022-04-22T21:47:06.121845+0000 mgr.y (mgr.34107) 8910 : cluster [DBG] pgmap v9771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:08 smithi121 bash[36902]: cluster 2022-04-22T21:47:08.122962+0000 mgr.y (mgr.34107) 8911 : cluster [DBG] pgmap v9772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:08.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:08 smithi115 bash[45495]: cluster 2022-04-22T21:47:08.122962+0000 mgr.y (mgr.34107) 8911 : cluster [DBG] pgmap v9772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:08.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:08 smithi115 bash[46934]: cluster 2022-04-22T21:47:08.122962+0000 mgr.y (mgr.34107) 8911 : cluster [DBG] pgmap v9772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:09 smithi121 bash[36902]: audit 2022-04-22T21:47:09.227044+0000 mon.a (mon.0) 12647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:47:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:09 smithi121 bash[36902]: audit 2022-04-22T21:47:09.230135+0000 mon.a (mon.0) 12648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:47:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:09 smithi121 bash[36902]: audit 2022-04-22T21:47:09.232089+0000 mon.a (mon.0) 12649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:47:09.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[46934]: audit 2022-04-22T21:47:09.227044+0000 mon.a (mon.0) 12647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:47:09.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[46934]: audit 2022-04-22T21:47:09.230135+0000 mon.a (mon.0) 12648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:47:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[46934]: audit 2022-04-22T21:47:09.232089+0000 mon.a (mon.0) 12649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:47:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[45495]: audit 2022-04-22T21:47:09.227044+0000 mon.a (mon.0) 12647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:47:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[45495]: audit 2022-04-22T21:47:09.230135+0000 mon.a (mon.0) 12648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:47:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:09 smithi115 bash[45495]: audit 2022-04-22T21:47:09.232089+0000 mon.a (mon.0) 12649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:47:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:10 smithi121 bash[36902]: audit 2022-04-22T21:47:09.593325+0000 mon.a (mon.0) 12650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:47:11.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:10 smithi121 bash[36902]: audit 2022-04-22T21:47:09.905866+0000 mon.a (mon.0) 12651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:11.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:10 smithi121 bash[36902]: cluster 2022-04-22T21:47:10.123929+0000 mgr.y (mgr.34107) 8912 : cluster [DBG] pgmap v9773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[45495]: audit 2022-04-22T21:47:09.593325+0000 mon.a (mon.0) 12650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:47:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[45495]: audit 2022-04-22T21:47:09.905866+0000 mon.a (mon.0) 12651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[45495]: cluster 2022-04-22T21:47:10.123929+0000 mgr.y (mgr.34107) 8912 : cluster [DBG] pgmap v9773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[46934]: audit 2022-04-22T21:47:09.593325+0000 mon.a (mon.0) 12650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:47:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[46934]: audit 2022-04-22T21:47:09.905866+0000 mon.a (mon.0) 12651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:10 smithi115 bash[46934]: cluster 2022-04-22T21:47:10.123929+0000 mgr.y (mgr.34107) 8912 : cluster [DBG] pgmap v9773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:11 smithi121 bash[36902]: audit 2022-04-22T21:47:10.917333+0000 mon.a (mon.0) 12652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:11 smithi115 bash[45495]: audit 2022-04-22T21:47:10.917333+0000 mon.a (mon.0) 12652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:11 smithi115 bash[46934]: audit 2022-04-22T21:47:10.917333+0000 mon.a (mon.0) 12652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:12 smithi121 bash[36902]: cluster 2022-04-22T21:47:12.124963+0000 mgr.y (mgr.34107) 8913 : cluster [DBG] pgmap v9774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:12 smithi115 bash[45495]: cluster 2022-04-22T21:47:12.124963+0000 mgr.y (mgr.34107) 8913 : cluster [DBG] pgmap v9774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:12.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:12 smithi115 bash[46934]: cluster 2022-04-22T21:47:12.124963+0000 mgr.y (mgr.34107) 8913 : cluster [DBG] pgmap v9774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:14.504 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:14 smithi115 bash[45495]: cluster 2022-04-22T21:47:14.126042+0000 mgr.y (mgr.34107) 8914 : cluster [DBG] pgmap v9775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:14.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:14 smithi115 bash[46934]: cluster 2022-04-22T21:47:14.126042+0000 mgr.y (mgr.34107) 8914 : cluster [DBG] pgmap v9775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:14.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:14 smithi121 bash[36902]: cluster 2022-04-22T21:47:14.126042+0000 mgr.y (mgr.34107) 8914 : cluster [DBG] pgmap v9775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:14.500Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:15 smithi121 bash[36902]: audit 2022-04-22T21:47:14.908729+0000 mon.a (mon.0) 12653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:15 smithi115 bash[45495]: audit 2022-04-22T21:47:14.908729+0000 mon.a (mon.0) 12653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:15 smithi115 bash[46934]: audit 2022-04-22T21:47:14.908729+0000 mon.a (mon.0) 12653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:16 smithi121 bash[36902]: audit 2022-04-22T21:47:15.920906+0000 mon.a (mon.0) 12654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:16 smithi121 bash[36902]: cluster 2022-04-22T21:47:16.127157+0000 mgr.y (mgr.34107) 8915 : cluster [DBG] pgmap v9776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:16 smithi121 bash[36902]: audit 2022-04-22T21:47:16.276690+0000 mon.a (mon.0) 12655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:47:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:16 smithi121 bash[36902]: audit 2022-04-22T21:47:16.277162+0000 mon.a (mon.0) 12656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:47:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[45495]: audit 2022-04-22T21:47:15.920906+0000 mon.a (mon.0) 12654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[45495]: cluster 2022-04-22T21:47:16.127157+0000 mgr.y (mgr.34107) 8915 : cluster [DBG] pgmap v9776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[45495]: audit 2022-04-22T21:47:16.276690+0000 mon.a (mon.0) 12655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[45495]: audit 2022-04-22T21:47:16.277162+0000 mon.a (mon.0) 12656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[46934]: audit 2022-04-22T21:47:15.920906+0000 mon.a (mon.0) 12654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[46934]: cluster 2022-04-22T21:47:16.127157+0000 mgr.y (mgr.34107) 8915 : cluster [DBG] pgmap v9776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[46934]: audit 2022-04-22T21:47:16.276690+0000 mon.a (mon.0) 12655 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:47:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:16 smithi115 bash[46934]: audit 2022-04-22T21:47:16.277162+0000 mon.a (mon.0) 12656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:47:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:18 smithi121 bash[36902]: cluster 2022-04-22T21:47:18.128325+0000 mgr.y (mgr.34107) 8916 : cluster [DBG] pgmap v9777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:18 smithi115 bash[45495]: cluster 2022-04-22T21:47:18.128325+0000 mgr.y (mgr.34107) 8916 : cluster [DBG] pgmap v9777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:18 smithi115 bash[46934]: cluster 2022-04-22T21:47:18.128325+0000 mgr.y (mgr.34107) 8916 : cluster [DBG] pgmap v9777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:19 smithi121 bash[36902]: audit 2022-04-22T21:47:19.911027+0000 mon.a (mon.0) 12657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:19 smithi115 bash[45495]: audit 2022-04-22T21:47:19.911027+0000 mon.a (mon.0) 12657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:19 smithi115 bash[46934]: audit 2022-04-22T21:47:19.911027+0000 mon.a (mon.0) 12657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:20 smithi121 bash[36902]: cluster 2022-04-22T21:47:20.129258+0000 mgr.y (mgr.34107) 8917 : cluster [DBG] pgmap v9778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:20 smithi121 bash[36902]: audit 2022-04-22T21:47:20.924738+0000 mon.a (mon.0) 12658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:20 smithi115 bash[45495]: cluster 2022-04-22T21:47:20.129258+0000 mgr.y (mgr.34107) 8917 : cluster [DBG] pgmap v9778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:20 smithi115 bash[45495]: audit 2022-04-22T21:47:20.924738+0000 mon.a (mon.0) 12658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:20 smithi115 bash[46934]: cluster 2022-04-22T21:47:20.129258+0000 mgr.y (mgr.34107) 8917 : cluster [DBG] pgmap v9778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:20 smithi115 bash[46934]: audit 2022-04-22T21:47:20.924738+0000 mon.a (mon.0) 12658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:22.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:22 smithi121 bash[36902]: cluster 2022-04-22T21:47:22.130372+0000 mgr.y (mgr.34107) 8918 : cluster [DBG] pgmap v9779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:22 smithi115 bash[45495]: cluster 2022-04-22T21:47:22.130372+0000 mgr.y (mgr.34107) 8918 : cluster [DBG] pgmap v9779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:22 smithi115 bash[46934]: cluster 2022-04-22T21:47:22.130372+0000 mgr.y (mgr.34107) 8918 : cluster [DBG] pgmap v9779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:22.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:24.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:24 smithi115 bash[45495]: cluster 2022-04-22T21:47:24.131401+0000 mgr.y (mgr.34107) 8919 : cluster [DBG] pgmap v9780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:24.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:24 smithi115 bash[46934]: cluster 2022-04-22T21:47:24.131401+0000 mgr.y (mgr.34107) 8919 : cluster [DBG] pgmap v9780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:24 smithi121 bash[36902]: cluster 2022-04-22T21:47:24.131401+0000 mgr.y (mgr.34107) 8919 : cluster [DBG] pgmap v9780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:24.500Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:25 smithi121 bash[36902]: audit 2022-04-22T21:47:24.914223+0000 mon.a (mon.0) 12659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:25 smithi115 bash[45495]: audit 2022-04-22T21:47:24.914223+0000 mon.a (mon.0) 12659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:25 smithi115 bash[46934]: audit 2022-04-22T21:47:24.914223+0000 mon.a (mon.0) 12659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:26 smithi121 bash[36902]: audit 2022-04-22T21:47:25.928340+0000 mon.a (mon.0) 12660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:26 smithi121 bash[36902]: cluster 2022-04-22T21:47:26.132465+0000 mgr.y (mgr.34107) 8920 : cluster [DBG] pgmap v9781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:26 smithi115 bash[45495]: audit 2022-04-22T21:47:25.928340+0000 mon.a (mon.0) 12660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:26 smithi115 bash[45495]: cluster 2022-04-22T21:47:26.132465+0000 mgr.y (mgr.34107) 8920 : cluster [DBG] pgmap v9781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:26 smithi115 bash[46934]: audit 2022-04-22T21:47:25.928340+0000 mon.a (mon.0) 12660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:26 smithi115 bash[46934]: cluster 2022-04-22T21:47:26.132465+0000 mgr.y (mgr.34107) 8920 : cluster [DBG] pgmap v9781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:28 smithi121 bash[36902]: cluster 2022-04-22T21:47:28.133553+0000 mgr.y (mgr.34107) 8921 : cluster [DBG] pgmap v9782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:28 smithi115 bash[45495]: cluster 2022-04-22T21:47:28.133553+0000 mgr.y (mgr.34107) 8921 : cluster [DBG] pgmap v9782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:28 smithi115 bash[46934]: cluster 2022-04-22T21:47:28.133553+0000 mgr.y (mgr.34107) 8921 : cluster [DBG] pgmap v9782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:29 smithi121 bash[36902]: audit 2022-04-22T21:47:29.917349+0000 mon.a (mon.0) 12661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:29 smithi115 bash[45495]: audit 2022-04-22T21:47:29.917349+0000 mon.a (mon.0) 12661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:30.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:29 smithi115 bash[46934]: audit 2022-04-22T21:47:29.917349+0000 mon.a (mon.0) 12661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:30 smithi121 bash[36902]: cluster 2022-04-22T21:47:30.134520+0000 mgr.y (mgr.34107) 8922 : cluster [DBG] pgmap v9783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:30 smithi121 bash[36902]: audit 2022-04-22T21:47:30.932154+0000 mon.a (mon.0) 12662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:30 smithi115 bash[45495]: cluster 2022-04-22T21:47:30.134520+0000 mgr.y (mgr.34107) 8922 : cluster [DBG] pgmap v9783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:30 smithi115 bash[45495]: audit 2022-04-22T21:47:30.932154+0000 mon.a (mon.0) 12662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:30 smithi115 bash[46934]: cluster 2022-04-22T21:47:30.134520+0000 mgr.y (mgr.34107) 8922 : cluster [DBG] pgmap v9783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:30 smithi115 bash[46934]: audit 2022-04-22T21:47:30.932154+0000 mon.a (mon.0) 12662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:32 smithi121 bash[36902]: cluster 2022-04-22T21:47:32.135709+0000 mgr.y (mgr.34107) 8923 : cluster [DBG] pgmap v9784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:32 smithi115 bash[45495]: cluster 2022-04-22T21:47:32.135709+0000 mgr.y (mgr.34107) 8923 : cluster [DBG] pgmap v9784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:32 smithi115 bash[46934]: cluster 2022-04-22T21:47:32.135709+0000 mgr.y (mgr.34107) 8923 : cluster [DBG] pgmap v9784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:32.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:34.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:34 smithi115 bash[45495]: cluster 2022-04-22T21:47:34.136780+0000 mgr.y (mgr.34107) 8924 : cluster [DBG] pgmap v9785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:34.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:34 smithi115 bash[46934]: cluster 2022-04-22T21:47:34.136780+0000 mgr.y (mgr.34107) 8924 : cluster [DBG] pgmap v9785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:34 smithi121 bash[36902]: cluster 2022-04-22T21:47:34.136780+0000 mgr.y (mgr.34107) 8924 : cluster [DBG] pgmap v9785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:34.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:34.502Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:35.477 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:47:35.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:35 smithi121 bash[36902]: audit 2022-04-22T21:47:34.920763+0000 mon.a (mon.0) 12663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:35 smithi115 bash[45495]: audit 2022-04-22T21:47:34.920763+0000 mon.a (mon.0) 12663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:35 smithi115 bash[46934]: audit 2022-04-22T21:47:34.920763+0000 mon.a (mon.0) 12663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:47:35.827 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:47:35.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:47:35.829 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:47:36.242 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:47:36.242 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:47:36.242 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:47:36.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:47:36.244 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:47:36.244 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:47:36.244 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:47:36.244 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:47:37.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:36 smithi121 bash[36902]: audit 2022-04-22T21:47:35.457088+0000 mgr.y (mgr.34107) 8925 : audit [DBG] from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:36 smithi121 bash[36902]: audit 2022-04-22T21:47:35.806616+0000 mgr.y (mgr.34107) 8926 : audit [DBG] from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:36 smithi121 bash[36902]: audit 2022-04-22T21:47:35.935525+0000 mon.a (mon.0) 12664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:36 smithi121 bash[36902]: cluster 2022-04-22T21:47:36.137797+0000 mgr.y (mgr.34107) 8927 : cluster [DBG] pgmap v9786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:36 smithi121 bash[36902]: audit 2022-04-22T21:47:36.239023+0000 mon.a (mon.0) 12665 : audit [DBG] from='client.? 172.21.15.115:0/2692990629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[45495]: audit 2022-04-22T21:47:35.457088+0000 mgr.y (mgr.34107) 8925 : audit [DBG] from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[45495]: audit 2022-04-22T21:47:35.806616+0000 mgr.y (mgr.34107) 8926 : audit [DBG] from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[45495]: audit 2022-04-22T21:47:35.935525+0000 mon.a (mon.0) 12664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[45495]: cluster 2022-04-22T21:47:36.137797+0000 mgr.y (mgr.34107) 8927 : cluster [DBG] pgmap v9786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[45495]: audit 2022-04-22T21:47:36.239023+0000 mon.a (mon.0) 12665 : audit [DBG] from='client.? 172.21.15.115:0/2692990629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[46934]: audit 2022-04-22T21:47:35.457088+0000 mgr.y (mgr.34107) 8925 : audit [DBG] from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[46934]: audit 2022-04-22T21:47:35.806616+0000 mgr.y (mgr.34107) 8926 : audit [DBG] from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:47:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[46934]: audit 2022-04-22T21:47:35.935525+0000 mon.a (mon.0) 12664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[46934]: cluster 2022-04-22T21:47:36.137797+0000 mgr.y (mgr.34107) 8927 : cluster [DBG] pgmap v9786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:37.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:36 smithi115 bash[46934]: audit 2022-04-22T21:47:36.239023+0000 mon.a (mon.0) 12665 : audit [DBG] from='client.? 172.21.15.115:0/2692990629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:47:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:38 smithi121 bash[36902]: cluster 2022-04-22T21:47:38.138848+0000 mgr.y (mgr.34107) 8928 : cluster [DBG] pgmap v9787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:38 smithi115 bash[45495]: cluster 2022-04-22T21:47:38.138848+0000 mgr.y (mgr.34107) 8928 : cluster [DBG] pgmap v9787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:38 smithi115 bash[46934]: cluster 2022-04-22T21:47:38.138848+0000 mgr.y (mgr.34107) 8928 : cluster [DBG] pgmap v9787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:39 smithi121 bash[36902]: audit 2022-04-22T21:47:39.923641+0000 mon.a (mon.0) 12666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:39 smithi115 bash[45495]: audit 2022-04-22T21:47:39.923641+0000 mon.a (mon.0) 12666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:40.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:39 smithi115 bash[46934]: audit 2022-04-22T21:47:39.923641+0000 mon.a (mon.0) 12666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:40 smithi121 bash[36902]: cluster 2022-04-22T21:47:40.139806+0000 mgr.y (mgr.34107) 8929 : cluster [DBG] pgmap v9788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:40 smithi121 bash[36902]: audit 2022-04-22T21:47:40.939235+0000 mon.a (mon.0) 12667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:40 smithi115 bash[45495]: cluster 2022-04-22T21:47:40.139806+0000 mgr.y (mgr.34107) 8929 : cluster [DBG] pgmap v9788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:40 smithi115 bash[45495]: audit 2022-04-22T21:47:40.939235+0000 mon.a (mon.0) 12667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:40 smithi115 bash[46934]: cluster 2022-04-22T21:47:40.139806+0000 mgr.y (mgr.34107) 8929 : cluster [DBG] pgmap v9788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:40 smithi115 bash[46934]: audit 2022-04-22T21:47:40.939235+0000 mon.a (mon.0) 12667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:42.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:42 smithi121 bash[36902]: cluster 2022-04-22T21:47:42.141015+0000 mgr.y (mgr.34107) 8930 : cluster [DBG] pgmap v9789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:42 smithi115 bash[45495]: cluster 2022-04-22T21:47:42.141015+0000 mgr.y (mgr.34107) 8930 : cluster [DBG] pgmap v9789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:42 smithi115 bash[46934]: cluster 2022-04-22T21:47:42.141015+0000 mgr.y (mgr.34107) 8930 : cluster [DBG] pgmap v9789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:42.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:44.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:44 smithi115 bash[45495]: cluster 2022-04-22T21:47:44.142087+0000 mgr.y (mgr.34107) 8931 : cluster [DBG] pgmap v9790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:44.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:44 smithi115 bash[46934]: cluster 2022-04-22T21:47:44.142087+0000 mgr.y (mgr.34107) 8931 : cluster [DBG] pgmap v9790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:44.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:44 smithi121 bash[36902]: cluster 2022-04-22T21:47:44.142087+0000 mgr.y (mgr.34107) 8931 : cluster [DBG] pgmap v9790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:44.503Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:45 smithi121 bash[36902]: audit 2022-04-22T21:47:44.926591+0000 mon.a (mon.0) 12668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:45 smithi115 bash[45495]: audit 2022-04-22T21:47:44.926591+0000 mon.a (mon.0) 12668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:45 smithi115 bash[46934]: audit 2022-04-22T21:47:44.926591+0000 mon.a (mon.0) 12668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:46 smithi121 bash[36902]: audit 2022-04-22T21:47:45.942897+0000 mon.a (mon.0) 12669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:46 smithi121 bash[36902]: cluster 2022-04-22T21:47:46.143161+0000 mgr.y (mgr.34107) 8932 : cluster [DBG] pgmap v9791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:46 smithi115 bash[45495]: audit 2022-04-22T21:47:45.942897+0000 mon.a (mon.0) 12669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:46 smithi115 bash[45495]: cluster 2022-04-22T21:47:46.143161+0000 mgr.y (mgr.34107) 8932 : cluster [DBG] pgmap v9791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:46 smithi115 bash[46934]: audit 2022-04-22T21:47:45.942897+0000 mon.a (mon.0) 12669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:46 smithi115 bash[46934]: cluster 2022-04-22T21:47:46.143161+0000 mgr.y (mgr.34107) 8932 : cluster [DBG] pgmap v9791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:48 smithi121 bash[36902]: cluster 2022-04-22T21:47:48.144449+0000 mgr.y (mgr.34107) 8933 : cluster [DBG] pgmap v9792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:48 smithi115 bash[45495]: cluster 2022-04-22T21:47:48.144449+0000 mgr.y (mgr.34107) 8933 : cluster [DBG] pgmap v9792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:48 smithi115 bash[46934]: cluster 2022-04-22T21:47:48.144449+0000 mgr.y (mgr.34107) 8933 : cluster [DBG] pgmap v9792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:49 smithi121 bash[36902]: audit 2022-04-22T21:47:49.930059+0000 mon.a (mon.0) 12670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:49 smithi115 bash[45495]: audit 2022-04-22T21:47:49.930059+0000 mon.a (mon.0) 12670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:49 smithi115 bash[46934]: audit 2022-04-22T21:47:49.930059+0000 mon.a (mon.0) 12670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:50 smithi121 bash[36902]: cluster 2022-04-22T21:47:50.145347+0000 mgr.y (mgr.34107) 8934 : cluster [DBG] pgmap v9793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:50 smithi121 bash[36902]: audit 2022-04-22T21:47:50.946634+0000 mon.a (mon.0) 12671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:50 smithi115 bash[45495]: cluster 2022-04-22T21:47:50.145347+0000 mgr.y (mgr.34107) 8934 : cluster [DBG] pgmap v9793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:50 smithi115 bash[45495]: audit 2022-04-22T21:47:50.946634+0000 mon.a (mon.0) 12671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:50 smithi115 bash[46934]: cluster 2022-04-22T21:47:50.145347+0000 mgr.y (mgr.34107) 8934 : cluster [DBG] pgmap v9793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:50 smithi115 bash[46934]: audit 2022-04-22T21:47:50.946634+0000 mon.a (mon.0) 12671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:52 smithi121 bash[36902]: cluster 2022-04-22T21:47:52.146552+0000 mgr.y (mgr.34107) 8935 : cluster [DBG] pgmap v9794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:52 smithi115 bash[45495]: cluster 2022-04-22T21:47:52.146552+0000 mgr.y (mgr.34107) 8935 : cluster [DBG] pgmap v9794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:52 smithi115 bash[46934]: cluster 2022-04-22T21:47:52.146552+0000 mgr.y (mgr.34107) 8935 : cluster [DBG] pgmap v9794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:52.642Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:54.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:54 smithi115 bash[45495]: cluster 2022-04-22T21:47:54.147625+0000 mgr.y (mgr.34107) 8936 : cluster [DBG] pgmap v9795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:54.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:54 smithi115 bash[46934]: cluster 2022-04-22T21:47:54.147625+0000 mgr.y (mgr.34107) 8936 : cluster [DBG] pgmap v9795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:54 smithi121 bash[36902]: cluster 2022-04-22T21:47:54.147625+0000 mgr.y (mgr.34107) 8936 : cluster [DBG] pgmap v9795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:47:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:47:54.501Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:47:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:55 smithi121 bash[36902]: audit 2022-04-22T21:47:54.933098+0000 mon.a (mon.0) 12672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:55 smithi115 bash[45495]: audit 2022-04-22T21:47:54.933098+0000 mon.a (mon.0) 12672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:55 smithi115 bash[46934]: audit 2022-04-22T21:47:54.933098+0000 mon.a (mon.0) 12672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:47:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:56 smithi121 bash[36902]: audit 2022-04-22T21:47:55.950340+0000 mon.a (mon.0) 12673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:56 smithi121 bash[36902]: cluster 2022-04-22T21:47:56.148688+0000 mgr.y (mgr.34107) 8937 : cluster [DBG] pgmap v9796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:56 smithi115 bash[45495]: audit 2022-04-22T21:47:55.950340+0000 mon.a (mon.0) 12673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:56 smithi115 bash[45495]: cluster 2022-04-22T21:47:56.148688+0000 mgr.y (mgr.34107) 8937 : cluster [DBG] pgmap v9796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:56 smithi115 bash[46934]: audit 2022-04-22T21:47:55.950340+0000 mon.a (mon.0) 12673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:47:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:56 smithi115 bash[46934]: cluster 2022-04-22T21:47:56.148688+0000 mgr.y (mgr.34107) 8937 : cluster [DBG] pgmap v9796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:58.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:58 smithi121 bash[36902]: cluster 2022-04-22T21:47:58.149916+0000 mgr.y (mgr.34107) 8938 : cluster [DBG] pgmap v9797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:58.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:58 smithi115 bash[46934]: cluster 2022-04-22T21:47:58.149916+0000 mgr.y (mgr.34107) 8938 : cluster [DBG] pgmap v9797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:47:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:58 smithi115 bash[45495]: cluster 2022-04-22T21:47:58.149916+0000 mgr.y (mgr.34107) 8938 : cluster [DBG] pgmap v9797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:47:59 smithi121 bash[36902]: audit 2022-04-22T21:47:59.936134+0000 mon.a (mon.0) 12674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:00.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:47:59 smithi115 bash[46934]: audit 2022-04-22T21:47:59.936134+0000 mon.a (mon.0) 12674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:47:59 smithi115 bash[45495]: audit 2022-04-22T21:47:59.936134+0000 mon.a (mon.0) 12674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:01 smithi121 bash[36902]: cluster 2022-04-22T21:48:00.150920+0000 mgr.y (mgr.34107) 8939 : cluster [DBG] pgmap v9798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:01 smithi121 bash[36902]: audit 2022-04-22T21:48:00.954210+0000 mon.a (mon.0) 12675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:00 smithi115 bash[45495]: cluster 2022-04-22T21:48:00.150920+0000 mgr.y (mgr.34107) 8939 : cluster [DBG] pgmap v9798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:00 smithi115 bash[45495]: audit 2022-04-22T21:48:00.954210+0000 mon.a (mon.0) 12675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:01.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:01 smithi115 bash[46934]: cluster 2022-04-22T21:48:00.150920+0000 mgr.y (mgr.34107) 8939 : cluster [DBG] pgmap v9798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:01.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:01 smithi115 bash[46934]: audit 2022-04-22T21:48:00.954210+0000 mon.a (mon.0) 12675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:02 smithi121 bash[36902]: cluster 2022-04-22T21:48:02.152145+0000 mgr.y (mgr.34107) 8940 : cluster [DBG] pgmap v9799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:02 smithi115 bash[45495]: cluster 2022-04-22T21:48:02.152145+0000 mgr.y (mgr.34107) 8940 : cluster [DBG] pgmap v9799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:02 smithi115 bash[46934]: cluster 2022-04-22T21:48:02.152145+0000 mgr.y (mgr.34107) 8940 : cluster [DBG] pgmap v9799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:02.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:04.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:04 smithi115 bash[46934]: cluster 2022-04-22T21:48:04.153200+0000 mgr.y (mgr.34107) 8941 : cluster [DBG] pgmap v9800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:04.507 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:04 smithi115 bash[45495]: cluster 2022-04-22T21:48:04.153200+0000 mgr.y (mgr.34107) 8941 : cluster [DBG] pgmap v9800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:04 smithi121 bash[36902]: cluster 2022-04-22T21:48:04.153200+0000 mgr.y (mgr.34107) 8941 : cluster [DBG] pgmap v9800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:04.503Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:05 smithi121 bash[36902]: audit 2022-04-22T21:48:04.939326+0000 mon.a (mon.0) 12676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:05 smithi115 bash[45495]: audit 2022-04-22T21:48:04.939326+0000 mon.a (mon.0) 12676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:05 smithi115 bash[46934]: audit 2022-04-22T21:48:04.939326+0000 mon.a (mon.0) 12676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:06.617 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:48:06.958 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:48:06.959 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:48:06.960 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:48:06.960 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:48:06.960 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:48:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:06 smithi121 bash[36902]: audit 2022-04-22T21:48:05.957867+0000 mon.a (mon.0) 12677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:07.021 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:06 smithi121 bash[36902]: cluster 2022-04-22T21:48:06.154308+0000 mgr.y (mgr.34107) 8942 : cluster [DBG] pgmap v9801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:06 smithi115 bash[45495]: audit 2022-04-22T21:48:05.957867+0000 mon.a (mon.0) 12677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:06 smithi115 bash[45495]: cluster 2022-04-22T21:48:06.154308+0000 mgr.y (mgr.34107) 8942 : cluster [DBG] pgmap v9801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:06 smithi115 bash[46934]: audit 2022-04-22T21:48:05.957867+0000 mon.a (mon.0) 12677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:06 smithi115 bash[46934]: cluster 2022-04-22T21:48:06.154308+0000 mgr.y (mgr.34107) 8942 : cluster [DBG] pgmap v9801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:07.363 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:48:07.363 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:48:07.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:48:07.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:07.365 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:48:07.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:48:07.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:48:07.365 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:48:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:07 smithi121 bash[36902]: audit 2022-04-22T21:48:06.597789+0000 mgr.y (mgr.34107) 8943 : audit [DBG] from='client.44640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:07 smithi121 bash[36902]: audit 2022-04-22T21:48:06.934315+0000 mgr.y (mgr.34107) 8944 : audit [DBG] from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:07 smithi121 bash[36902]: audit 2022-04-22T21:48:07.359834+0000 mon.c (mon.1) 314 : audit [DBG] from='client.? 172.21.15.115:0/1549256241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[45495]: audit 2022-04-22T21:48:06.597789+0000 mgr.y (mgr.34107) 8943 : audit [DBG] from='client.44640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[45495]: audit 2022-04-22T21:48:06.934315+0000 mgr.y (mgr.34107) 8944 : audit [DBG] from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[45495]: audit 2022-04-22T21:48:07.359834+0000 mon.c (mon.1) 314 : audit [DBG] from='client.? 172.21.15.115:0/1549256241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[46934]: audit 2022-04-22T21:48:06.597789+0000 mgr.y (mgr.34107) 8943 : audit [DBG] from='client.44640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[46934]: audit 2022-04-22T21:48:06.934315+0000 mgr.y (mgr.34107) 8944 : audit [DBG] from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:07 smithi115 bash[46934]: audit 2022-04-22T21:48:07.359834+0000 mon.c (mon.1) 314 : audit [DBG] from='client.? 172.21.15.115:0/1549256241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:08 smithi121 bash[36902]: cluster 2022-04-22T21:48:08.155443+0000 mgr.y (mgr.34107) 8945 : cluster [DBG] pgmap v9802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:08 smithi115 bash[45495]: cluster 2022-04-22T21:48:08.155443+0000 mgr.y (mgr.34107) 8945 : cluster [DBG] pgmap v9802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:08 smithi115 bash[46934]: cluster 2022-04-22T21:48:08.155443+0000 mgr.y (mgr.34107) 8945 : cluster [DBG] pgmap v9802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:09.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:09 smithi121 bash[36902]: audit 2022-04-22T21:48:09.602841+0000 mon.a (mon.0) 12678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:48:09.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:09 smithi121 bash[36902]: audit 2022-04-22T21:48:09.605993+0000 mon.a (mon.0) 12679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:48:09.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:09 smithi121 bash[36902]: audit 2022-04-22T21:48:09.607752+0000 mon.a (mon.0) 12680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:48:09.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[46934]: audit 2022-04-22T21:48:09.602841+0000 mon.a (mon.0) 12678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:48:09.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[46934]: audit 2022-04-22T21:48:09.605993+0000 mon.a (mon.0) 12679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:48:09.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[46934]: audit 2022-04-22T21:48:09.607752+0000 mon.a (mon.0) 12680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:48:09.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[45495]: audit 2022-04-22T21:48:09.602841+0000 mon.a (mon.0) 12678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:48:09.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[45495]: audit 2022-04-22T21:48:09.605993+0000 mon.a (mon.0) 12679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:48:09.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:09 smithi115 bash[45495]: audit 2022-04-22T21:48:09.607752+0000 mon.a (mon.0) 12680 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:48:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:10 smithi121 bash[36902]: audit 2022-04-22T21:48:09.942485+0000 mon.a (mon.0) 12681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:10 smithi121 bash[36902]: audit 2022-04-22T21:48:09.977906+0000 mon.a (mon.0) 12682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:48:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:10 smithi121 bash[36902]: cluster 2022-04-22T21:48:10.156434+0000 mgr.y (mgr.34107) 8946 : cluster [DBG] pgmap v9803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[45495]: audit 2022-04-22T21:48:09.942485+0000 mon.a (mon.0) 12681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[45495]: audit 2022-04-22T21:48:09.977906+0000 mon.a (mon.0) 12682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:48:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[45495]: cluster 2022-04-22T21:48:10.156434+0000 mgr.y (mgr.34107) 8946 : cluster [DBG] pgmap v9803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[46934]: audit 2022-04-22T21:48:09.942485+0000 mon.a (mon.0) 12681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[46934]: audit 2022-04-22T21:48:09.977906+0000 mon.a (mon.0) 12682 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:48:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:10 smithi115 bash[46934]: cluster 2022-04-22T21:48:10.156434+0000 mgr.y (mgr.34107) 8946 : cluster [DBG] pgmap v9803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:11 smithi121 bash[36902]: audit 2022-04-22T21:48:10.961492+0000 mon.a (mon.0) 12683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:11 smithi115 bash[46934]: audit 2022-04-22T21:48:10.961492+0000 mon.a (mon.0) 12683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:11 smithi115 bash[45495]: audit 2022-04-22T21:48:10.961492+0000 mon.a (mon.0) 12683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:12 smithi121 bash[36902]: cluster 2022-04-22T21:48:12.157583+0000 mgr.y (mgr.34107) 8947 : cluster [DBG] pgmap v9804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:12 smithi115 bash[46934]: cluster 2022-04-22T21:48:12.157583+0000 mgr.y (mgr.34107) 8947 : cluster [DBG] pgmap v9804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:12.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:12 smithi115 bash[45495]: cluster 2022-04-22T21:48:12.157583+0000 mgr.y (mgr.34107) 8947 : cluster [DBG] pgmap v9804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:14.508 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:14 smithi115 bash[45495]: cluster 2022-04-22T21:48:14.158599+0000 mgr.y (mgr.34107) 8948 : cluster [DBG] pgmap v9805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:14.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:14 smithi115 bash[46934]: cluster 2022-04-22T21:48:14.158599+0000 mgr.y (mgr.34107) 8948 : cluster [DBG] pgmap v9805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:14.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:14 smithi121 bash[36902]: cluster 2022-04-22T21:48:14.158599+0000 mgr.y (mgr.34107) 8948 : cluster [DBG] pgmap v9805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:14.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:15 smithi121 bash[36902]: audit 2022-04-22T21:48:14.944920+0000 mon.a (mon.0) 12684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:15 smithi115 bash[45495]: audit 2022-04-22T21:48:14.944920+0000 mon.a (mon.0) 12684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:15 smithi115 bash[46934]: audit 2022-04-22T21:48:14.944920+0000 mon.a (mon.0) 12684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:16 smithi121 bash[36902]: audit 2022-04-22T21:48:15.965021+0000 mon.a (mon.0) 12685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:16 smithi121 bash[36902]: cluster 2022-04-22T21:48:16.159796+0000 mgr.y (mgr.34107) 8949 : cluster [DBG] pgmap v9806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:16 smithi121 bash[36902]: audit 2022-04-22T21:48:16.285519+0000 mon.a (mon.0) 12686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:48:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:16 smithi121 bash[36902]: audit 2022-04-22T21:48:16.286071+0000 mon.a (mon.0) 12687 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:48:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[45495]: audit 2022-04-22T21:48:15.965021+0000 mon.a (mon.0) 12685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:17.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[45495]: cluster 2022-04-22T21:48:16.159796+0000 mgr.y (mgr.34107) 8949 : cluster [DBG] pgmap v9806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:17.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[45495]: audit 2022-04-22T21:48:16.285519+0000 mon.a (mon.0) 12686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:48:17.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[45495]: audit 2022-04-22T21:48:16.286071+0000 mon.a (mon.0) 12687 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:48:17.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[46934]: audit 2022-04-22T21:48:15.965021+0000 mon.a (mon.0) 12685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:17.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[46934]: cluster 2022-04-22T21:48:16.159796+0000 mgr.y (mgr.34107) 8949 : cluster [DBG] pgmap v9806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:17.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[46934]: audit 2022-04-22T21:48:16.285519+0000 mon.a (mon.0) 12686 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:48:17.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:16 smithi115 bash[46934]: audit 2022-04-22T21:48:16.286071+0000 mon.a (mon.0) 12687 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:48:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:18 smithi121 bash[36902]: cluster 2022-04-22T21:48:18.160897+0000 mgr.y (mgr.34107) 8950 : cluster [DBG] pgmap v9807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:18 smithi115 bash[45495]: cluster 2022-04-22T21:48:18.160897+0000 mgr.y (mgr.34107) 8950 : cluster [DBG] pgmap v9807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:18 smithi115 bash[46934]: cluster 2022-04-22T21:48:18.160897+0000 mgr.y (mgr.34107) 8950 : cluster [DBG] pgmap v9807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:20 smithi121 bash[36902]: audit 2022-04-22T21:48:19.948155+0000 mon.a (mon.0) 12688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:20 smithi115 bash[45495]: audit 2022-04-22T21:48:19.948155+0000 mon.a (mon.0) 12688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:20.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:20 smithi115 bash[46934]: audit 2022-04-22T21:48:19.948155+0000 mon.a (mon.0) 12688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:21 smithi115 bash[45495]: cluster 2022-04-22T21:48:20.161849+0000 mgr.y (mgr.34107) 8951 : cluster [DBG] pgmap v9808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:21 smithi115 bash[45495]: audit 2022-04-22T21:48:20.968716+0000 mon.a (mon.0) 12689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:21 smithi115 bash[46934]: cluster 2022-04-22T21:48:20.161849+0000 mgr.y (mgr.34107) 8951 : cluster [DBG] pgmap v9808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:21 smithi115 bash[46934]: audit 2022-04-22T21:48:20.968716+0000 mon.a (mon.0) 12689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:21.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:21 smithi121 bash[36902]: cluster 2022-04-22T21:48:20.161849+0000 mgr.y (mgr.34107) 8951 : cluster [DBG] pgmap v9808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:21.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:21 smithi121 bash[36902]: audit 2022-04-22T21:48:20.968716+0000 mon.a (mon.0) 12689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:22 smithi121 bash[36902]: cluster 2022-04-22T21:48:22.163028+0000 mgr.y (mgr.34107) 8952 : cluster [DBG] pgmap v9809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:22 smithi115 bash[46934]: cluster 2022-04-22T21:48:22.163028+0000 mgr.y (mgr.34107) 8952 : cluster [DBG] pgmap v9809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:22.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:22 smithi115 bash[45495]: cluster 2022-04-22T21:48:22.163028+0000 mgr.y (mgr.34107) 8952 : cluster [DBG] pgmap v9809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:22.644Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:24.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:24 smithi115 bash[45495]: cluster 2022-04-22T21:48:24.164098+0000 mgr.y (mgr.34107) 8953 : cluster [DBG] pgmap v9810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:24.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:24 smithi115 bash[46934]: cluster 2022-04-22T21:48:24.164098+0000 mgr.y (mgr.34107) 8953 : cluster [DBG] pgmap v9810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:24 smithi121 bash[36902]: cluster 2022-04-22T21:48:24.164098+0000 mgr.y (mgr.34107) 8953 : cluster [DBG] pgmap v9810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:24.503Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:25 smithi121 bash[36902]: audit 2022-04-22T21:48:24.951106+0000 mon.a (mon.0) 12690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:25 smithi115 bash[45495]: audit 2022-04-22T21:48:24.951106+0000 mon.a (mon.0) 12690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:25 smithi115 bash[46934]: audit 2022-04-22T21:48:24.951106+0000 mon.a (mon.0) 12690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:26 smithi121 bash[36902]: audit 2022-04-22T21:48:25.972334+0000 mon.a (mon.0) 12691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:26 smithi121 bash[36902]: cluster 2022-04-22T21:48:26.165198+0000 mgr.y (mgr.34107) 8954 : cluster [DBG] pgmap v9811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:26 smithi115 bash[45495]: audit 2022-04-22T21:48:25.972334+0000 mon.a (mon.0) 12691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:26 smithi115 bash[45495]: cluster 2022-04-22T21:48:26.165198+0000 mgr.y (mgr.34107) 8954 : cluster [DBG] pgmap v9811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:26 smithi115 bash[46934]: audit 2022-04-22T21:48:25.972334+0000 mon.a (mon.0) 12691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:26 smithi115 bash[46934]: cluster 2022-04-22T21:48:26.165198+0000 mgr.y (mgr.34107) 8954 : cluster [DBG] pgmap v9811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:28 smithi121 bash[36902]: cluster 2022-04-22T21:48:28.166366+0000 mgr.y (mgr.34107) 8955 : cluster [DBG] pgmap v9812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:28 smithi115 bash[45495]: cluster 2022-04-22T21:48:28.166366+0000 mgr.y (mgr.34107) 8955 : cluster [DBG] pgmap v9812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:28 smithi115 bash[46934]: cluster 2022-04-22T21:48:28.166366+0000 mgr.y (mgr.34107) 8955 : cluster [DBG] pgmap v9812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:30.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:30 smithi121 bash[36902]: audit 2022-04-22T21:48:29.954070+0000 mon.a (mon.0) 12692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:30 smithi115 bash[45495]: audit 2022-04-22T21:48:29.954070+0000 mon.a (mon.0) 12692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:30.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:30 smithi115 bash[46934]: audit 2022-04-22T21:48:29.954070+0000 mon.a (mon.0) 12692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:31 smithi115 bash[45495]: cluster 2022-04-22T21:48:30.167380+0000 mgr.y (mgr.34107) 8956 : cluster [DBG] pgmap v9813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:31 smithi115 bash[45495]: audit 2022-04-22T21:48:30.975911+0000 mon.a (mon.0) 12693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:31 smithi115 bash[46934]: cluster 2022-04-22T21:48:30.167380+0000 mgr.y (mgr.34107) 8956 : cluster [DBG] pgmap v9813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:31 smithi115 bash[46934]: audit 2022-04-22T21:48:30.975911+0000 mon.a (mon.0) 12693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:31 smithi121 bash[36902]: cluster 2022-04-22T21:48:30.167380+0000 mgr.y (mgr.34107) 8956 : cluster [DBG] pgmap v9813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:31 smithi121 bash[36902]: audit 2022-04-22T21:48:30.975911+0000 mon.a (mon.0) 12693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:32 smithi121 bash[36902]: cluster 2022-04-22T21:48:32.168700+0000 mgr.y (mgr.34107) 8957 : cluster [DBG] pgmap v9814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:32 smithi115 bash[45495]: cluster 2022-04-22T21:48:32.168700+0000 mgr.y (mgr.34107) 8957 : cluster [DBG] pgmap v9814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:32 smithi115 bash[46934]: cluster 2022-04-22T21:48:32.168700+0000 mgr.y (mgr.34107) 8957 : cluster [DBG] pgmap v9814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:32.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:34.508 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:34 smithi115 bash[45495]: cluster 2022-04-22T21:48:34.169769+0000 mgr.y (mgr.34107) 8958 : cluster [DBG] pgmap v9815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:34.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:34 smithi115 bash[46934]: cluster 2022-04-22T21:48:34.169769+0000 mgr.y (mgr.34107) 8958 : cluster [DBG] pgmap v9815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:34.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:34 smithi121 bash[36902]: cluster 2022-04-22T21:48:34.169769+0000 mgr.y (mgr.34107) 8958 : cluster [DBG] pgmap v9815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:34.504Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:35 smithi121 bash[36902]: audit 2022-04-22T21:48:34.957167+0000 mon.a (mon.0) 12694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:35 smithi115 bash[45495]: audit 2022-04-22T21:48:34.957167+0000 mon.a (mon.0) 12694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:35 smithi115 bash[46934]: audit 2022-04-22T21:48:34.957167+0000 mon.a (mon.0) 12694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:36 smithi121 bash[36902]: audit 2022-04-22T21:48:35.979645+0000 mon.a (mon.0) 12695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:36 smithi121 bash[36902]: cluster 2022-04-22T21:48:36.170894+0000 mgr.y (mgr.34107) 8959 : cluster [DBG] pgmap v9816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:36 smithi115 bash[45495]: audit 2022-04-22T21:48:35.979645+0000 mon.a (mon.0) 12695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:36 smithi115 bash[45495]: cluster 2022-04-22T21:48:36.170894+0000 mgr.y (mgr.34107) 8959 : cluster [DBG] pgmap v9816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:36 smithi115 bash[46934]: audit 2022-04-22T21:48:35.979645+0000 mon.a (mon.0) 12695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:36 smithi115 bash[46934]: cluster 2022-04-22T21:48:36.170894+0000 mgr.y (mgr.34107) 8959 : cluster [DBG] pgmap v9816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:37.745 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:48:38.093 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:48:38.094 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:48:38.095 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:48:38.096 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:48:38.096 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:48:38.499 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:48:38.499 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:48:38.499 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:48:38.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:48:38.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:48:38.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:48:38.501 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:48:38.501 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:48:38.501 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:48:38.501 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:48:38.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:38 smithi121 bash[36902]: audit 2022-04-22T21:48:37.725598+0000 mgr.y (mgr.34107) 8960 : audit [DBG] from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:38 smithi121 bash[36902]: audit 2022-04-22T21:48:38.070380+0000 mgr.y (mgr.34107) 8961 : audit [DBG] from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:38 smithi121 bash[36902]: cluster 2022-04-22T21:48:38.172047+0000 mgr.y (mgr.34107) 8962 : cluster [DBG] pgmap v9817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[46934]: audit 2022-04-22T21:48:37.725598+0000 mgr.y (mgr.34107) 8960 : audit [DBG] from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[46934]: audit 2022-04-22T21:48:38.070380+0000 mgr.y (mgr.34107) 8961 : audit [DBG] from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[46934]: cluster 2022-04-22T21:48:38.172047+0000 mgr.y (mgr.34107) 8962 : cluster [DBG] pgmap v9817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[45495]: audit 2022-04-22T21:48:37.725598+0000 mgr.y (mgr.34107) 8960 : audit [DBG] from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[45495]: audit 2022-04-22T21:48:38.070380+0000 mgr.y (mgr.34107) 8961 : audit [DBG] from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:48:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:38 smithi115 bash[45495]: cluster 2022-04-22T21:48:38.172047+0000 mgr.y (mgr.34107) 8962 : cluster [DBG] pgmap v9817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:39 smithi121 bash[36902]: audit 2022-04-22T21:48:38.495572+0000 mon.c (mon.1) 315 : audit [DBG] from='client.? 172.21.15.115:0/2833743946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:39 smithi115 bash[45495]: audit 2022-04-22T21:48:38.495572+0000 mon.c (mon.1) 315 : audit [DBG] from='client.? 172.21.15.115:0/2833743946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:39 smithi115 bash[46934]: audit 2022-04-22T21:48:38.495572+0000 mon.c (mon.1) 315 : audit [DBG] from='client.? 172.21.15.115:0/2833743946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:48:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:40 smithi121 bash[36902]: audit 2022-04-22T21:48:39.960151+0000 mon.a (mon.0) 12696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:40.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:40 smithi121 bash[36902]: cluster 2022-04-22T21:48:40.173076+0000 mgr.y (mgr.34107) 8963 : cluster [DBG] pgmap v9818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:40 smithi115 bash[46934]: audit 2022-04-22T21:48:39.960151+0000 mon.a (mon.0) 12696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:40 smithi115 bash[46934]: cluster 2022-04-22T21:48:40.173076+0000 mgr.y (mgr.34107) 8963 : cluster [DBG] pgmap v9818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:40 smithi115 bash[45495]: audit 2022-04-22T21:48:39.960151+0000 mon.a (mon.0) 12696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:40 smithi115 bash[45495]: cluster 2022-04-22T21:48:40.173076+0000 mgr.y (mgr.34107) 8963 : cluster [DBG] pgmap v9818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:41 smithi121 bash[36902]: audit 2022-04-22T21:48:40.983291+0000 mon.a (mon.0) 12697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:41 smithi115 bash[45495]: audit 2022-04-22T21:48:40.983291+0000 mon.a (mon.0) 12697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:41 smithi115 bash[46934]: audit 2022-04-22T21:48:40.983291+0000 mon.a (mon.0) 12697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:42 smithi121 bash[36902]: cluster 2022-04-22T21:48:42.174352+0000 mgr.y (mgr.34107) 8964 : cluster [DBG] pgmap v9819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:43.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:42 smithi115 bash[45495]: cluster 2022-04-22T21:48:42.174352+0000 mgr.y (mgr.34107) 8964 : cluster [DBG] pgmap v9819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:42.645Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:42 smithi115 bash[46934]: cluster 2022-04-22T21:48:42.174352+0000 mgr.y (mgr.34107) 8964 : cluster [DBG] pgmap v9819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:44.508 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:44 smithi115 bash[45495]: cluster 2022-04-22T21:48:44.175583+0000 mgr.y (mgr.34107) 8965 : cluster [DBG] pgmap v9820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:44.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:44 smithi115 bash[46934]: cluster 2022-04-22T21:48:44.175583+0000 mgr.y (mgr.34107) 8965 : cluster [DBG] pgmap v9820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:44 smithi121 bash[36902]: cluster 2022-04-22T21:48:44.175583+0000 mgr.y (mgr.34107) 8965 : cluster [DBG] pgmap v9820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:44.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:45 smithi121 bash[36902]: audit 2022-04-22T21:48:44.963599+0000 mon.a (mon.0) 12698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:45 smithi115 bash[45495]: audit 2022-04-22T21:48:44.963599+0000 mon.a (mon.0) 12698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:45 smithi115 bash[46934]: audit 2022-04-22T21:48:44.963599+0000 mon.a (mon.0) 12698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:46 smithi121 bash[36902]: audit 2022-04-22T21:48:45.986734+0000 mon.a (mon.0) 12699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:46 smithi121 bash[36902]: cluster 2022-04-22T21:48:46.176701+0000 mgr.y (mgr.34107) 8966 : cluster [DBG] pgmap v9821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:46 smithi115 bash[45495]: audit 2022-04-22T21:48:45.986734+0000 mon.a (mon.0) 12699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:46 smithi115 bash[45495]: cluster 2022-04-22T21:48:46.176701+0000 mgr.y (mgr.34107) 8966 : cluster [DBG] pgmap v9821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:46 smithi115 bash[46934]: audit 2022-04-22T21:48:45.986734+0000 mon.a (mon.0) 12699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:46 smithi115 bash[46934]: cluster 2022-04-22T21:48:46.176701+0000 mgr.y (mgr.34107) 8966 : cluster [DBG] pgmap v9821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:48 smithi121 bash[36902]: cluster 2022-04-22T21:48:48.177835+0000 mgr.y (mgr.34107) 8967 : cluster [DBG] pgmap v9822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:48 smithi115 bash[45495]: cluster 2022-04-22T21:48:48.177835+0000 mgr.y (mgr.34107) 8967 : cluster [DBG] pgmap v9822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:48 smithi115 bash[46934]: cluster 2022-04-22T21:48:48.177835+0000 mgr.y (mgr.34107) 8967 : cluster [DBG] pgmap v9822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:50 smithi115 bash[45495]: audit 2022-04-22T21:48:49.966711+0000 mon.a (mon.0) 12700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:50 smithi115 bash[46934]: audit 2022-04-22T21:48:49.966711+0000 mon.a (mon.0) 12700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:50.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:50 smithi121 bash[36902]: audit 2022-04-22T21:48:49.966711+0000 mon.a (mon.0) 12700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:51 smithi121 bash[36902]: cluster 2022-04-22T21:48:50.178816+0000 mgr.y (mgr.34107) 8968 : cluster [DBG] pgmap v9823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:51 smithi121 bash[36902]: audit 2022-04-22T21:48:50.990310+0000 mon.a (mon.0) 12701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:51 smithi115 bash[45495]: cluster 2022-04-22T21:48:50.178816+0000 mgr.y (mgr.34107) 8968 : cluster [DBG] pgmap v9823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:51 smithi115 bash[45495]: audit 2022-04-22T21:48:50.990310+0000 mon.a (mon.0) 12701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:51 smithi115 bash[46934]: cluster 2022-04-22T21:48:50.178816+0000 mgr.y (mgr.34107) 8968 : cluster [DBG] pgmap v9823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:51 smithi115 bash[46934]: audit 2022-04-22T21:48:50.990310+0000 mon.a (mon.0) 12701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:52.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:53 smithi121 bash[36902]: cluster 2022-04-22T21:48:52.180005+0000 mgr.y (mgr.34107) 8969 : cluster [DBG] pgmap v9824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:53 smithi115 bash[45495]: cluster 2022-04-22T21:48:52.180005+0000 mgr.y (mgr.34107) 8969 : cluster [DBG] pgmap v9824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:53 smithi115 bash[46934]: cluster 2022-04-22T21:48:52.180005+0000 mgr.y (mgr.34107) 8969 : cluster [DBG] pgmap v9824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:48:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:48:54.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:48:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:55 smithi121 bash[36902]: cluster 2022-04-22T21:48:54.181064+0000 mgr.y (mgr.34107) 8970 : cluster [DBG] pgmap v9825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:55 smithi121 bash[36902]: audit 2022-04-22T21:48:54.969923+0000 mon.a (mon.0) 12702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:55 smithi115 bash[45495]: cluster 2022-04-22T21:48:54.181064+0000 mgr.y (mgr.34107) 8970 : cluster [DBG] pgmap v9825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:55 smithi115 bash[45495]: audit 2022-04-22T21:48:54.969923+0000 mon.a (mon.0) 12702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:55 smithi115 bash[46934]: cluster 2022-04-22T21:48:54.181064+0000 mgr.y (mgr.34107) 8970 : cluster [DBG] pgmap v9825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:55 smithi115 bash[46934]: audit 2022-04-22T21:48:54.969923+0000 mon.a (mon.0) 12702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:48:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:56 smithi121 bash[36902]: audit 2022-04-22T21:48:55.994985+0000 mon.a (mon.0) 12703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:56 smithi115 bash[45495]: audit 2022-04-22T21:48:55.994985+0000 mon.a (mon.0) 12703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:56 smithi115 bash[46934]: audit 2022-04-22T21:48:55.994985+0000 mon.a (mon.0) 12703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:48:58.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:57 smithi121 bash[36902]: cluster 2022-04-22T21:48:56.182177+0000 mgr.y (mgr.34107) 8971 : cluster [DBG] pgmap v9826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:57 smithi115 bash[45495]: cluster 2022-04-22T21:48:56.182177+0000 mgr.y (mgr.34107) 8971 : cluster [DBG] pgmap v9826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:48:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:57 smithi115 bash[46934]: cluster 2022-04-22T21:48:56.182177+0000 mgr.y (mgr.34107) 8971 : cluster [DBG] pgmap v9826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:00.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:48:59 smithi121 bash[36902]: cluster 2022-04-22T21:48:58.183283+0000 mgr.y (mgr.34107) 8972 : cluster [DBG] pgmap v9827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:00.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:48:59 smithi115 bash[46934]: cluster 2022-04-22T21:48:58.183283+0000 mgr.y (mgr.34107) 8972 : cluster [DBG] pgmap v9827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:48:59 smithi115 bash[45495]: cluster 2022-04-22T21:48:58.183283+0000 mgr.y (mgr.34107) 8972 : cluster [DBG] pgmap v9827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:00 smithi115 bash[45495]: audit 2022-04-22T21:48:59.972961+0000 mon.a (mon.0) 12704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:00 smithi115 bash[46934]: audit 2022-04-22T21:48:59.972961+0000 mon.a (mon.0) 12704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:01.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:00 smithi121 bash[36902]: audit 2022-04-22T21:48:59.972961+0000 mon.a (mon.0) 12704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:01 smithi115 bash[45495]: cluster 2022-04-22T21:49:00.184296+0000 mgr.y (mgr.34107) 8973 : cluster [DBG] pgmap v9828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:01 smithi115 bash[45495]: audit 2022-04-22T21:49:01.001300+0000 mon.a (mon.0) 12705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:01 smithi115 bash[46934]: cluster 2022-04-22T21:49:00.184296+0000 mgr.y (mgr.34107) 8973 : cluster [DBG] pgmap v9828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:01 smithi115 bash[46934]: audit 2022-04-22T21:49:01.001300+0000 mon.a (mon.0) 12705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:01 smithi121 bash[36902]: cluster 2022-04-22T21:49:00.184296+0000 mgr.y (mgr.34107) 8973 : cluster [DBG] pgmap v9828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:01 smithi121 bash[36902]: audit 2022-04-22T21:49:01.001300+0000 mon.a (mon.0) 12705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:02.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:03 smithi121 bash[36902]: cluster 2022-04-22T21:49:02.185445+0000 mgr.y (mgr.34107) 8974 : cluster [DBG] pgmap v9829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:03 smithi115 bash[45495]: cluster 2022-04-22T21:49:02.185445+0000 mgr.y (mgr.34107) 8974 : cluster [DBG] pgmap v9829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:03 smithi115 bash[46934]: cluster 2022-04-22T21:49:02.185445+0000 mgr.y (mgr.34107) 8974 : cluster [DBG] pgmap v9829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:04.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:05 smithi121 bash[36902]: cluster 2022-04-22T21:49:04.186572+0000 mgr.y (mgr.34107) 8975 : cluster [DBG] pgmap v9830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:05 smithi121 bash[36902]: audit 2022-04-22T21:49:04.976035+0000 mon.a (mon.0) 12706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:05 smithi115 bash[45495]: cluster 2022-04-22T21:49:04.186572+0000 mgr.y (mgr.34107) 8975 : cluster [DBG] pgmap v9830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:05 smithi115 bash[45495]: audit 2022-04-22T21:49:04.976035+0000 mon.a (mon.0) 12706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:05 smithi115 bash[46934]: cluster 2022-04-22T21:49:04.186572+0000 mgr.y (mgr.34107) 8975 : cluster [DBG] pgmap v9830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:05 smithi115 bash[46934]: audit 2022-04-22T21:49:04.976035+0000 mon.a (mon.0) 12706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:07.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:06 smithi121 bash[36902]: audit 2022-04-22T21:49:06.004679+0000 mon.a (mon.0) 12707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:06 smithi115 bash[45495]: audit 2022-04-22T21:49:06.004679+0000 mon.a (mon.0) 12707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:06 smithi115 bash[46934]: audit 2022-04-22T21:49:06.004679+0000 mon.a (mon.0) 12707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:07 smithi121 bash[36902]: cluster 2022-04-22T21:49:06.187697+0000 mgr.y (mgr.34107) 8976 : cluster [DBG] pgmap v9831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:07 smithi115 bash[45495]: cluster 2022-04-22T21:49:06.187697+0000 mgr.y (mgr.34107) 8976 : cluster [DBG] pgmap v9831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:07 smithi115 bash[46934]: cluster 2022-04-22T21:49:06.187697+0000 mgr.y (mgr.34107) 8976 : cluster [DBG] pgmap v9831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:08.878 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:49:09.226 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:49:09.226 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:49:09.226 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:49:09.226 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:49:09.226 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:49:09.227 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:49:09.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:49:09.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:49:09.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:49:09.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:49:09.228 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:49:09.229 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:49:09.630 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:49:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:49:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:49:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:49:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:49:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:49:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:49:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:49:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:49:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:49:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:49:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:49:09.633 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:49:10.074 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:09 smithi121 bash[36902]: cluster 2022-04-22T21:49:08.188739+0000 mgr.y (mgr.34107) 8977 : cluster [DBG] pgmap v9832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:10.074 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:09 smithi121 bash[36902]: audit 2022-04-22T21:49:08.858025+0000 mgr.y (mgr.34107) 8978 : audit [DBG] from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:10.075 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:09 smithi121 bash[36902]: audit 2022-04-22T21:49:09.626036+0000 mon.a (mon.0) 12708 : audit [DBG] from='client.? 172.21.15.115:0/2344890547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:10.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[45495]: cluster 2022-04-22T21:49:08.188739+0000 mgr.y (mgr.34107) 8977 : cluster [DBG] pgmap v9832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:10.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[45495]: audit 2022-04-22T21:49:08.858025+0000 mgr.y (mgr.34107) 8978 : audit [DBG] from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:10.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[45495]: audit 2022-04-22T21:49:09.626036+0000 mon.a (mon.0) 12708 : audit [DBG] from='client.? 172.21.15.115:0/2344890547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:10.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[46934]: cluster 2022-04-22T21:49:08.188739+0000 mgr.y (mgr.34107) 8977 : cluster [DBG] pgmap v9832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:10.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[46934]: audit 2022-04-22T21:49:08.858025+0000 mgr.y (mgr.34107) 8978 : audit [DBG] from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:10.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:09 smithi115 bash[46934]: audit 2022-04-22T21:49:09.626036+0000 mon.a (mon.0) 12708 : audit [DBG] from='client.? 172.21.15.115:0/2344890547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:09.203853+0000 mgr.y (mgr.34107) 8979 : audit [DBG] from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:09.979396+0000 mon.a (mon.0) 12709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:09.986640+0000 mon.a (mon.0) 12710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:09.989342+0000 mon.a (mon.0) 12711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:09.990821+0000 mon.a (mon.0) 12712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:49:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:10 smithi121 bash[36902]: audit 2022-04-22T21:49:10.352004+0000 mon.a (mon.0) 12713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:09.203853+0000 mgr.y (mgr.34107) 8979 : audit [DBG] from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:09.979396+0000 mon.a (mon.0) 12709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:09.986640+0000 mon.a (mon.0) 12710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:09.989342+0000 mon.a (mon.0) 12711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:09.990821+0000 mon.a (mon.0) 12712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:49:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[46934]: audit 2022-04-22T21:49:10.352004+0000 mon.a (mon.0) 12713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:09.203853+0000 mgr.y (mgr.34107) 8979 : audit [DBG] from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:09.979396+0000 mon.a (mon.0) 12709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:09.986640+0000 mon.a (mon.0) 12710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:09.989342+0000 mon.a (mon.0) 12711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:09.990821+0000 mon.a (mon.0) 12712 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:49:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:10 smithi115 bash[45495]: audit 2022-04-22T21:49:10.352004+0000 mon.a (mon.0) 12713 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:49:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:11 smithi121 bash[36902]: cluster 2022-04-22T21:49:10.189560+0000 mgr.y (mgr.34107) 8980 : cluster [DBG] pgmap v9833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:11 smithi121 bash[36902]: audit 2022-04-22T21:49:11.008074+0000 mon.a (mon.0) 12714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:11 smithi115 bash[45495]: cluster 2022-04-22T21:49:10.189560+0000 mgr.y (mgr.34107) 8980 : cluster [DBG] pgmap v9833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:11 smithi115 bash[45495]: audit 2022-04-22T21:49:11.008074+0000 mon.a (mon.0) 12714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:11 smithi115 bash[46934]: cluster 2022-04-22T21:49:10.189560+0000 mgr.y (mgr.34107) 8980 : cluster [DBG] pgmap v9833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:11 smithi115 bash[46934]: audit 2022-04-22T21:49:11.008074+0000 mon.a (mon.0) 12714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:12.647Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:13 smithi121 bash[36902]: cluster 2022-04-22T21:49:12.190725+0000 mgr.y (mgr.34107) 8981 : cluster [DBG] pgmap v9834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:13 smithi115 bash[45495]: cluster 2022-04-22T21:49:12.190725+0000 mgr.y (mgr.34107) 8981 : cluster [DBG] pgmap v9834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:13 smithi115 bash[46934]: cluster 2022-04-22T21:49:12.190725+0000 mgr.y (mgr.34107) 8981 : cluster [DBG] pgmap v9834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:14.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:15 smithi121 bash[36902]: cluster 2022-04-22T21:49:14.191800+0000 mgr.y (mgr.34107) 8982 : cluster [DBG] pgmap v9835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:16.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:15 smithi121 bash[36902]: audit 2022-04-22T21:49:14.982472+0000 mon.a (mon.0) 12715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:15 smithi115 bash[45495]: cluster 2022-04-22T21:49:14.191800+0000 mgr.y (mgr.34107) 8982 : cluster [DBG] pgmap v9835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:15 smithi115 bash[45495]: audit 2022-04-22T21:49:14.982472+0000 mon.a (mon.0) 12715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:15 smithi115 bash[46934]: cluster 2022-04-22T21:49:14.191800+0000 mgr.y (mgr.34107) 8982 : cluster [DBG] pgmap v9835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:15 smithi115 bash[46934]: audit 2022-04-22T21:49:14.982472+0000 mon.a (mon.0) 12715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:16 smithi121 bash[36902]: audit 2022-04-22T21:49:16.011791+0000 mon.a (mon.0) 12716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:16 smithi121 bash[36902]: audit 2022-04-22T21:49:16.304370+0000 mon.a (mon.0) 12717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:49:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:16 smithi121 bash[36902]: audit 2022-04-22T21:49:16.304809+0000 mon.a (mon.0) 12718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:49:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[45495]: audit 2022-04-22T21:49:16.011791+0000 mon.a (mon.0) 12716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[45495]: audit 2022-04-22T21:49:16.304370+0000 mon.a (mon.0) 12717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:49:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[45495]: audit 2022-04-22T21:49:16.304809+0000 mon.a (mon.0) 12718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:49:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[46934]: audit 2022-04-22T21:49:16.011791+0000 mon.a (mon.0) 12716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[46934]: audit 2022-04-22T21:49:16.304370+0000 mon.a (mon.0) 12717 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:49:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:16 smithi115 bash[46934]: audit 2022-04-22T21:49:16.304809+0000 mon.a (mon.0) 12718 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:49:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:17 smithi121 bash[36902]: cluster 2022-04-22T21:49:16.192828+0000 mgr.y (mgr.34107) 8983 : cluster [DBG] pgmap v9836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:17 smithi115 bash[45495]: cluster 2022-04-22T21:49:16.192828+0000 mgr.y (mgr.34107) 8983 : cluster [DBG] pgmap v9836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:17 smithi115 bash[46934]: cluster 2022-04-22T21:49:16.192828+0000 mgr.y (mgr.34107) 8983 : cluster [DBG] pgmap v9836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:19 smithi121 bash[36902]: cluster 2022-04-22T21:49:18.194000+0000 mgr.y (mgr.34107) 8984 : cluster [DBG] pgmap v9837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:19 smithi115 bash[45495]: cluster 2022-04-22T21:49:18.194000+0000 mgr.y (mgr.34107) 8984 : cluster [DBG] pgmap v9837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:19 smithi115 bash[46934]: cluster 2022-04-22T21:49:18.194000+0000 mgr.y (mgr.34107) 8984 : cluster [DBG] pgmap v9837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:20 smithi121 bash[36902]: audit 2022-04-22T21:49:19.985779+0000 mon.a (mon.0) 12719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:20 smithi115 bash[45495]: audit 2022-04-22T21:49:19.985779+0000 mon.a (mon.0) 12719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:20 smithi115 bash[46934]: audit 2022-04-22T21:49:19.985779+0000 mon.a (mon.0) 12719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:21 smithi121 bash[36902]: cluster 2022-04-22T21:49:20.194955+0000 mgr.y (mgr.34107) 8985 : cluster [DBG] pgmap v9838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:22.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:21 smithi121 bash[36902]: audit 2022-04-22T21:49:21.015076+0000 mon.a (mon.0) 12720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:21 smithi115 bash[45495]: cluster 2022-04-22T21:49:20.194955+0000 mgr.y (mgr.34107) 8985 : cluster [DBG] pgmap v9838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:21 smithi115 bash[45495]: audit 2022-04-22T21:49:21.015076+0000 mon.a (mon.0) 12720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:21 smithi115 bash[46934]: cluster 2022-04-22T21:49:20.194955+0000 mgr.y (mgr.34107) 8985 : cluster [DBG] pgmap v9838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:21 smithi115 bash[46934]: audit 2022-04-22T21:49:21.015076+0000 mon.a (mon.0) 12720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:22.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:24.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:23 smithi121 bash[36902]: cluster 2022-04-22T21:49:22.196266+0000 mgr.y (mgr.34107) 8986 : cluster [DBG] pgmap v9839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:23 smithi115 bash[45495]: cluster 2022-04-22T21:49:22.196266+0000 mgr.y (mgr.34107) 8986 : cluster [DBG] pgmap v9839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:23 smithi115 bash[46934]: cluster 2022-04-22T21:49:22.196266+0000 mgr.y (mgr.34107) 8986 : cluster [DBG] pgmap v9839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:24.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:25 smithi121 bash[36902]: cluster 2022-04-22T21:49:24.197354+0000 mgr.y (mgr.34107) 8987 : cluster [DBG] pgmap v9840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:26.277 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:25 smithi121 bash[36902]: audit 2022-04-22T21:49:24.988873+0000 mon.a (mon.0) 12721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:25 smithi115 bash[45495]: cluster 2022-04-22T21:49:24.197354+0000 mgr.y (mgr.34107) 8987 : cluster [DBG] pgmap v9840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:25 smithi115 bash[45495]: audit 2022-04-22T21:49:24.988873+0000 mon.a (mon.0) 12721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:25 smithi115 bash[46934]: cluster 2022-04-22T21:49:24.197354+0000 mgr.y (mgr.34107) 8987 : cluster [DBG] pgmap v9840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:25 smithi115 bash[46934]: audit 2022-04-22T21:49:24.988873+0000 mon.a (mon.0) 12721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:27.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:26 smithi121 bash[36902]: audit 2022-04-22T21:49:26.018830+0000 mon.a (mon.0) 12722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:27.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:26 smithi115 bash[45495]: audit 2022-04-22T21:49:26.018830+0000 mon.a (mon.0) 12722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:27.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:26 smithi115 bash[46934]: audit 2022-04-22T21:49:26.018830+0000 mon.a (mon.0) 12722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:27 smithi121 bash[36902]: cluster 2022-04-22T21:49:26.198414+0000 mgr.y (mgr.34107) 8988 : cluster [DBG] pgmap v9841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:27 smithi115 bash[45495]: cluster 2022-04-22T21:49:26.198414+0000 mgr.y (mgr.34107) 8988 : cluster [DBG] pgmap v9841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:27 smithi115 bash[46934]: cluster 2022-04-22T21:49:26.198414+0000 mgr.y (mgr.34107) 8988 : cluster [DBG] pgmap v9841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:29 smithi121 bash[36902]: cluster 2022-04-22T21:49:28.199552+0000 mgr.y (mgr.34107) 8989 : cluster [DBG] pgmap v9842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:29 smithi115 bash[45495]: cluster 2022-04-22T21:49:28.199552+0000 mgr.y (mgr.34107) 8989 : cluster [DBG] pgmap v9842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:29 smithi115 bash[46934]: cluster 2022-04-22T21:49:28.199552+0000 mgr.y (mgr.34107) 8989 : cluster [DBG] pgmap v9842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:30 smithi121 bash[36902]: audit 2022-04-22T21:49:29.992330+0000 mon.a (mon.0) 12723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:31.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:30 smithi115 bash[46934]: audit 2022-04-22T21:49:29.992330+0000 mon.a (mon.0) 12723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:31.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:30 smithi115 bash[45495]: audit 2022-04-22T21:49:29.992330+0000 mon.a (mon.0) 12723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:31 smithi121 bash[36902]: cluster 2022-04-22T21:49:30.200561+0000 mgr.y (mgr.34107) 8990 : cluster [DBG] pgmap v9843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:32.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:31 smithi121 bash[36902]: audit 2022-04-22T21:49:31.022946+0000 mon.a (mon.0) 12724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:31 smithi115 bash[45495]: cluster 2022-04-22T21:49:30.200561+0000 mgr.y (mgr.34107) 8990 : cluster [DBG] pgmap v9843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:31 smithi115 bash[45495]: audit 2022-04-22T21:49:31.022946+0000 mon.a (mon.0) 12724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:31 smithi115 bash[46934]: cluster 2022-04-22T21:49:30.200561+0000 mgr.y (mgr.34107) 8990 : cluster [DBG] pgmap v9843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:31 smithi115 bash[46934]: audit 2022-04-22T21:49:31.022946+0000 mon.a (mon.0) 12724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:32.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:34.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:33 smithi121 bash[36902]: cluster 2022-04-22T21:49:32.201746+0000 mgr.y (mgr.34107) 8991 : cluster [DBG] pgmap v9844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:34.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:33 smithi115 bash[46934]: cluster 2022-04-22T21:49:32.201746+0000 mgr.y (mgr.34107) 8991 : cluster [DBG] pgmap v9844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:33 smithi115 bash[45495]: cluster 2022-04-22T21:49:32.201746+0000 mgr.y (mgr.34107) 8991 : cluster [DBG] pgmap v9844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:34.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:35 smithi121 bash[36902]: cluster 2022-04-22T21:49:34.202836+0000 mgr.y (mgr.34107) 8992 : cluster [DBG] pgmap v9845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:35 smithi121 bash[36902]: audit 2022-04-22T21:49:34.995213+0000 mon.a (mon.0) 12725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:35 smithi115 bash[45495]: cluster 2022-04-22T21:49:34.202836+0000 mgr.y (mgr.34107) 8992 : cluster [DBG] pgmap v9845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:35 smithi115 bash[45495]: audit 2022-04-22T21:49:34.995213+0000 mon.a (mon.0) 12725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:35 smithi115 bash[46934]: cluster 2022-04-22T21:49:34.202836+0000 mgr.y (mgr.34107) 8992 : cluster [DBG] pgmap v9845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:35 smithi115 bash[46934]: audit 2022-04-22T21:49:34.995213+0000 mon.a (mon.0) 12725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:37.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:36 smithi121 bash[36902]: audit 2022-04-22T21:49:36.026726+0000 mon.a (mon.0) 12726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:36 smithi115 bash[45495]: audit 2022-04-22T21:49:36.026726+0000 mon.a (mon.0) 12726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:36 smithi115 bash[46934]: audit 2022-04-22T21:49:36.026726+0000 mon.a (mon.0) 12726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:37 smithi121 bash[36902]: cluster 2022-04-22T21:49:36.203937+0000 mgr.y (mgr.34107) 8993 : cluster [DBG] pgmap v9846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:37 smithi115 bash[45495]: cluster 2022-04-22T21:49:36.203937+0000 mgr.y (mgr.34107) 8993 : cluster [DBG] pgmap v9846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:37 smithi115 bash[46934]: cluster 2022-04-22T21:49:36.203937+0000 mgr.y (mgr.34107) 8993 : cluster [DBG] pgmap v9846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:39.995 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:49:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:39 smithi121 bash[36902]: cluster 2022-04-22T21:49:38.205060+0000 mgr.y (mgr.34107) 8994 : cluster [DBG] pgmap v9847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:39 smithi115 bash[45495]: cluster 2022-04-22T21:49:38.205060+0000 mgr.y (mgr.34107) 8994 : cluster [DBG] pgmap v9847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:39 smithi115 bash[46934]: cluster 2022-04-22T21:49:38.205060+0000 mgr.y (mgr.34107) 8994 : cluster [DBG] pgmap v9847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:40.333 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:49:40.334 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:49:40.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:49:40.336 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:49:40.730 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:49:40.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:49:40.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:49:40.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:40.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:49:40.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:49:40.732 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:49:40.732 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:49:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:40 smithi121 bash[36902]: audit 2022-04-22T21:49:39.974216+0000 mgr.y (mgr.34107) 8995 : audit [DBG] from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:40 smithi121 bash[36902]: audit 2022-04-22T21:49:39.998583+0000 mon.a (mon.0) 12727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:40 smithi121 bash[36902]: audit 2022-04-22T21:49:40.726635+0000 mon.a (mon.0) 12728 : audit [DBG] from='client.? 172.21.15.115:0/1260157613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:41.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[46934]: audit 2022-04-22T21:49:39.974216+0000 mgr.y (mgr.34107) 8995 : audit [DBG] from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[46934]: audit 2022-04-22T21:49:39.998583+0000 mon.a (mon.0) 12727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[46934]: audit 2022-04-22T21:49:40.726635+0000 mon.a (mon.0) 12728 : audit [DBG] from='client.? 172.21.15.115:0/1260157613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[45495]: audit 2022-04-22T21:49:39.974216+0000 mgr.y (mgr.34107) 8995 : audit [DBG] from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[45495]: audit 2022-04-22T21:49:39.998583+0000 mon.a (mon.0) 12727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:40 smithi115 bash[45495]: audit 2022-04-22T21:49:40.726635+0000 mon.a (mon.0) 12728 : audit [DBG] from='client.? 172.21.15.115:0/1260157613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:49:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:41 smithi121 bash[36902]: cluster 2022-04-22T21:49:40.206089+0000 mgr.y (mgr.34107) 8996 : cluster [DBG] pgmap v9848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:41 smithi121 bash[36902]: audit 2022-04-22T21:49:40.316282+0000 mgr.y (mgr.34107) 8997 : audit [DBG] from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:41 smithi121 bash[36902]: audit 2022-04-22T21:49:41.030417+0000 mon.a (mon.0) 12729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[45495]: cluster 2022-04-22T21:49:40.206089+0000 mgr.y (mgr.34107) 8996 : cluster [DBG] pgmap v9848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[45495]: audit 2022-04-22T21:49:40.316282+0000 mgr.y (mgr.34107) 8997 : audit [DBG] from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[45495]: audit 2022-04-22T21:49:41.030417+0000 mon.a (mon.0) 12729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[46934]: cluster 2022-04-22T21:49:40.206089+0000 mgr.y (mgr.34107) 8996 : cluster [DBG] pgmap v9848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[46934]: audit 2022-04-22T21:49:40.316282+0000 mgr.y (mgr.34107) 8997 : audit [DBG] from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:49:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:41 smithi115 bash[46934]: audit 2022-04-22T21:49:41.030417+0000 mon.a (mon.0) 12729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:42.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:43 smithi121 bash[36902]: cluster 2022-04-22T21:49:42.207309+0000 mgr.y (mgr.34107) 8998 : cluster [DBG] pgmap v9849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:43 smithi115 bash[45495]: cluster 2022-04-22T21:49:42.207309+0000 mgr.y (mgr.34107) 8998 : cluster [DBG] pgmap v9849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:43 smithi115 bash[46934]: cluster 2022-04-22T21:49:42.207309+0000 mgr.y (mgr.34107) 8998 : cluster [DBG] pgmap v9849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:44.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:45 smithi121 bash[36902]: cluster 2022-04-22T21:49:44.208502+0000 mgr.y (mgr.34107) 8999 : cluster [DBG] pgmap v9850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:45 smithi121 bash[36902]: audit 2022-04-22T21:49:45.001799+0000 mon.a (mon.0) 12730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:45 smithi115 bash[45495]: cluster 2022-04-22T21:49:44.208502+0000 mgr.y (mgr.34107) 8999 : cluster [DBG] pgmap v9850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:45 smithi115 bash[45495]: audit 2022-04-22T21:49:45.001799+0000 mon.a (mon.0) 12730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:45 smithi115 bash[46934]: cluster 2022-04-22T21:49:44.208502+0000 mgr.y (mgr.34107) 8999 : cluster [DBG] pgmap v9850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:45 smithi115 bash[46934]: audit 2022-04-22T21:49:45.001799+0000 mon.a (mon.0) 12730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:46 smithi121 bash[36902]: audit 2022-04-22T21:49:46.033915+0000 mon.a (mon.0) 12731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:46 smithi115 bash[45495]: audit 2022-04-22T21:49:46.033915+0000 mon.a (mon.0) 12731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:46 smithi115 bash[46934]: audit 2022-04-22T21:49:46.033915+0000 mon.a (mon.0) 12731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:47 smithi121 bash[36902]: cluster 2022-04-22T21:49:46.209619+0000 mgr.y (mgr.34107) 9000 : cluster [DBG] pgmap v9851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:47 smithi115 bash[45495]: cluster 2022-04-22T21:49:46.209619+0000 mgr.y (mgr.34107) 9000 : cluster [DBG] pgmap v9851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:47 smithi115 bash[46934]: cluster 2022-04-22T21:49:46.209619+0000 mgr.y (mgr.34107) 9000 : cluster [DBG] pgmap v9851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:49 smithi121 bash[36902]: cluster 2022-04-22T21:49:48.210857+0000 mgr.y (mgr.34107) 9001 : cluster [DBG] pgmap v9852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:49 smithi115 bash[45495]: cluster 2022-04-22T21:49:48.210857+0000 mgr.y (mgr.34107) 9001 : cluster [DBG] pgmap v9852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:49 smithi115 bash[46934]: cluster 2022-04-22T21:49:48.210857+0000 mgr.y (mgr.34107) 9001 : cluster [DBG] pgmap v9852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:50 smithi121 bash[36902]: audit 2022-04-22T21:49:50.004793+0000 mon.a (mon.0) 12732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:50 smithi115 bash[45495]: audit 2022-04-22T21:49:50.004793+0000 mon.a (mon.0) 12732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:50 smithi115 bash[46934]: audit 2022-04-22T21:49:50.004793+0000 mon.a (mon.0) 12732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:51 smithi121 bash[36902]: cluster 2022-04-22T21:49:50.211982+0000 mgr.y (mgr.34107) 9002 : cluster [DBG] pgmap v9853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:51 smithi121 bash[36902]: audit 2022-04-22T21:49:51.037100+0000 mon.a (mon.0) 12733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:51 smithi115 bash[45495]: cluster 2022-04-22T21:49:50.211982+0000 mgr.y (mgr.34107) 9002 : cluster [DBG] pgmap v9853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:51 smithi115 bash[45495]: audit 2022-04-22T21:49:51.037100+0000 mon.a (mon.0) 12733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:51 smithi115 bash[46934]: cluster 2022-04-22T21:49:50.211982+0000 mgr.y (mgr.34107) 9002 : cluster [DBG] pgmap v9853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:51 smithi115 bash[46934]: audit 2022-04-22T21:49:51.037100+0000 mon.a (mon.0) 12733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:52.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:53 smithi121 bash[36902]: cluster 2022-04-22T21:49:52.213076+0000 mgr.y (mgr.34107) 9003 : cluster [DBG] pgmap v9854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:53 smithi115 bash[45495]: cluster 2022-04-22T21:49:52.213076+0000 mgr.y (mgr.34107) 9003 : cluster [DBG] pgmap v9854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:53 smithi115 bash[46934]: cluster 2022-04-22T21:49:52.213076+0000 mgr.y (mgr.34107) 9003 : cluster [DBG] pgmap v9854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:49:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:49:54.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:49:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:55 smithi121 bash[36902]: cluster 2022-04-22T21:49:54.214201+0000 mgr.y (mgr.34107) 9004 : cluster [DBG] pgmap v9855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:55 smithi121 bash[36902]: audit 2022-04-22T21:49:55.008150+0000 mon.a (mon.0) 12734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:55 smithi115 bash[45495]: cluster 2022-04-22T21:49:54.214201+0000 mgr.y (mgr.34107) 9004 : cluster [DBG] pgmap v9855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:55 smithi115 bash[45495]: audit 2022-04-22T21:49:55.008150+0000 mon.a (mon.0) 12734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:55 smithi115 bash[46934]: cluster 2022-04-22T21:49:54.214201+0000 mgr.y (mgr.34107) 9004 : cluster [DBG] pgmap v9855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:55 smithi115 bash[46934]: audit 2022-04-22T21:49:55.008150+0000 mon.a (mon.0) 12734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:49:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:56 smithi121 bash[36902]: audit 2022-04-22T21:49:56.040803+0000 mon.a (mon.0) 12735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:56 smithi115 bash[45495]: audit 2022-04-22T21:49:56.040803+0000 mon.a (mon.0) 12735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:56 smithi115 bash[46934]: audit 2022-04-22T21:49:56.040803+0000 mon.a (mon.0) 12735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:49:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:57 smithi121 bash[36902]: cluster 2022-04-22T21:49:56.215176+0000 mgr.y (mgr.34107) 9005 : cluster [DBG] pgmap v9856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:57 smithi115 bash[45495]: cluster 2022-04-22T21:49:56.215176+0000 mgr.y (mgr.34107) 9005 : cluster [DBG] pgmap v9856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:49:58.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:57 smithi115 bash[46934]: cluster 2022-04-22T21:49:56.215176+0000 mgr.y (mgr.34107) 9005 : cluster [DBG] pgmap v9856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:49:59 smithi121 bash[36902]: cluster 2022-04-22T21:49:58.216324+0000 mgr.y (mgr.34107) 9006 : cluster [DBG] pgmap v9857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:49:59 smithi115 bash[45495]: cluster 2022-04-22T21:49:58.216324+0000 mgr.y (mgr.34107) 9006 : cluster [DBG] pgmap v9857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:49:59 smithi115 bash[46934]: cluster 2022-04-22T21:49:58.216324+0000 mgr.y (mgr.34107) 9006 : cluster [DBG] pgmap v9857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:00 smithi121 bash[36902]: cluster 2022-04-22T21:50:00.000120+0000 mon.a (mon.0) 12736 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:50:01.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:00 smithi121 bash[36902]: audit 2022-04-22T21:50:00.011118+0000 mon.a (mon.0) 12737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:01.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:00 smithi115 bash[45495]: cluster 2022-04-22T21:50:00.000120+0000 mon.a (mon.0) 12736 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:50:01.312 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:00 smithi115 bash[45495]: audit 2022-04-22T21:50:00.011118+0000 mon.a (mon.0) 12737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:01.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:00 smithi115 bash[46934]: cluster 2022-04-22T21:50:00.000120+0000 mon.a (mon.0) 12736 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T21:50:01.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:00 smithi115 bash[46934]: audit 2022-04-22T21:50:00.011118+0000 mon.a (mon.0) 12737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:02 smithi121 bash[36902]: cluster 2022-04-22T21:50:00.217354+0000 mgr.y (mgr.34107) 9007 : cluster [DBG] pgmap v9858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:02 smithi121 bash[36902]: audit 2022-04-22T21:50:01.044236+0000 mon.a (mon.0) 12738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:02.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:01 smithi115 bash[46934]: cluster 2022-04-22T21:50:00.217354+0000 mgr.y (mgr.34107) 9007 : cluster [DBG] pgmap v9858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:02.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:01 smithi115 bash[46934]: audit 2022-04-22T21:50:01.044236+0000 mon.a (mon.0) 12738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:01 smithi115 bash[45495]: cluster 2022-04-22T21:50:00.217354+0000 mgr.y (mgr.34107) 9007 : cluster [DBG] pgmap v9858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:01 smithi115 bash[45495]: audit 2022-04-22T21:50:01.044236+0000 mon.a (mon.0) 12738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:02.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:04 smithi121 bash[36902]: cluster 2022-04-22T21:50:02.218429+0000 mgr.y (mgr.34107) 9008 : cluster [DBG] pgmap v9859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:04 smithi115 bash[45495]: cluster 2022-04-22T21:50:02.218429+0000 mgr.y (mgr.34107) 9008 : cluster [DBG] pgmap v9859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:04 smithi115 bash[46934]: cluster 2022-04-22T21:50:02.218429+0000 mgr.y (mgr.34107) 9008 : cluster [DBG] pgmap v9859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:04.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:06 smithi121 bash[36902]: cluster 2022-04-22T21:50:04.219544+0000 mgr.y (mgr.34107) 9009 : cluster [DBG] pgmap v9860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:06 smithi121 bash[36902]: audit 2022-04-22T21:50:05.013843+0000 mon.a (mon.0) 12739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:06 smithi115 bash[45495]: cluster 2022-04-22T21:50:04.219544+0000 mgr.y (mgr.34107) 9009 : cluster [DBG] pgmap v9860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:06 smithi115 bash[45495]: audit 2022-04-22T21:50:05.013843+0000 mon.a (mon.0) 12739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:06 smithi115 bash[46934]: cluster 2022-04-22T21:50:04.219544+0000 mgr.y (mgr.34107) 9009 : cluster [DBG] pgmap v9860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:06 smithi115 bash[46934]: audit 2022-04-22T21:50:05.013843+0000 mon.a (mon.0) 12739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:07 smithi115 bash[45495]: audit 2022-04-22T21:50:06.048097+0000 mon.a (mon.0) 12740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:07 smithi115 bash[46934]: audit 2022-04-22T21:50:06.048097+0000 mon.a (mon.0) 12740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:07.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:07 smithi121 bash[36902]: audit 2022-04-22T21:50:06.048097+0000 mon.a (mon.0) 12740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:08 smithi115 bash[45495]: cluster 2022-04-22T21:50:06.220531+0000 mgr.y (mgr.34107) 9010 : cluster [DBG] pgmap v9861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:08 smithi115 bash[46934]: cluster 2022-04-22T21:50:06.220531+0000 mgr.y (mgr.34107) 9010 : cluster [DBG] pgmap v9861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:08 smithi121 bash[36902]: cluster 2022-04-22T21:50:06.220531+0000 mgr.y (mgr.34107) 9010 : cluster [DBG] pgmap v9861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:10 smithi115 bash[45495]: cluster 2022-04-22T21:50:08.221681+0000 mgr.y (mgr.34107) 9011 : cluster [DBG] pgmap v9862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:10 smithi115 bash[45495]: audit 2022-04-22T21:50:10.016207+0000 mon.a (mon.0) 12741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:10 smithi115 bash[46934]: cluster 2022-04-22T21:50:08.221681+0000 mgr.y (mgr.34107) 9011 : cluster [DBG] pgmap v9862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:10 smithi115 bash[46934]: audit 2022-04-22T21:50:10.016207+0000 mon.a (mon.0) 12741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:10.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:10 smithi121 bash[36902]: cluster 2022-04-22T21:50:08.221681+0000 mgr.y (mgr.34107) 9011 : cluster [DBG] pgmap v9862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:10.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:10 smithi121 bash[36902]: audit 2022-04-22T21:50:10.016207+0000 mon.a (mon.0) 12741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:11.112 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:50:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[45495]: audit 2022-04-22T21:50:10.360781+0000 mon.a (mon.0) 12742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[45495]: audit 2022-04-22T21:50:10.363719+0000 mon.a (mon.0) 12743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[45495]: audit 2022-04-22T21:50:10.365353+0000 mon.a (mon.0) 12744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[45495]: audit 2022-04-22T21:50:10.726052+0000 mon.a (mon.0) 12745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[46934]: audit 2022-04-22T21:50:10.360781+0000 mon.a (mon.0) 12742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[46934]: audit 2022-04-22T21:50:10.363719+0000 mon.a (mon.0) 12743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[46934]: audit 2022-04-22T21:50:10.365353+0000 mon.a (mon.0) 12744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:50:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:11 smithi115 bash[46934]: audit 2022-04-22T21:50:10.726052+0000 mon.a (mon.0) 12745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:50:11.449 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:50:11.450 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:50:11.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:50:11.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:50:11.451 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:50:11.451 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:11 smithi121 bash[36902]: audit 2022-04-22T21:50:10.360781+0000 mon.a (mon.0) 12742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:11 smithi121 bash[36902]: audit 2022-04-22T21:50:10.363719+0000 mon.a (mon.0) 12743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:50:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:11 smithi121 bash[36902]: audit 2022-04-22T21:50:10.365353+0000 mon.a (mon.0) 12744 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:50:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:11 smithi121 bash[36902]: audit 2022-04-22T21:50:10.726052+0000 mon.a (mon.0) 12745 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:50:11.848 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:50:11.849 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:50:11.849 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:50:11.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:11.849 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:50:11.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:50:11.851 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:50:11.851 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:50:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[45495]: cluster 2022-04-22T21:50:10.222731+0000 mgr.y (mgr.34107) 9012 : cluster [DBG] pgmap v9863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[45495]: audit 2022-04-22T21:50:11.051588+0000 mon.a (mon.0) 12746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[45495]: audit 2022-04-22T21:50:11.091294+0000 mgr.y (mgr.34107) 9013 : audit [DBG] from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[45495]: audit 2022-04-22T21:50:11.845136+0000 mon.a (mon.0) 12747 : audit [DBG] from='client.? 172.21.15.115:0/1959333152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[46934]: cluster 2022-04-22T21:50:10.222731+0000 mgr.y (mgr.34107) 9012 : cluster [DBG] pgmap v9863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[46934]: audit 2022-04-22T21:50:11.051588+0000 mon.a (mon.0) 12746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[46934]: audit 2022-04-22T21:50:11.091294+0000 mgr.y (mgr.34107) 9013 : audit [DBG] from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[46934]: audit 2022-04-22T21:50:11.845136+0000 mon.a (mon.0) 12747 : audit [DBG] from='client.? 172.21.15.115:0/1959333152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:12 smithi121 bash[36902]: cluster 2022-04-22T21:50:10.222731+0000 mgr.y (mgr.34107) 9012 : cluster [DBG] pgmap v9863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:12 smithi121 bash[36902]: audit 2022-04-22T21:50:11.051588+0000 mon.a (mon.0) 12746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:12 smithi121 bash[36902]: audit 2022-04-22T21:50:11.091294+0000 mgr.y (mgr.34107) 9013 : audit [DBG] from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:12 smithi121 bash[36902]: audit 2022-04-22T21:50:11.845136+0000 mon.a (mon.0) 12747 : audit [DBG] from='client.? 172.21.15.115:0/1959333152' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:12.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:13 smithi121 bash[36902]: audit 2022-04-22T21:50:11.427431+0000 mgr.y (mgr.34107) 9014 : audit [DBG] from='client.54649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:13.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:13 smithi115 bash[45495]: audit 2022-04-22T21:50:11.427431+0000 mgr.y (mgr.34107) 9014 : audit [DBG] from='client.54649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:13.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:13 smithi115 bash[46934]: audit 2022-04-22T21:50:11.427431+0000 mgr.y (mgr.34107) 9014 : audit [DBG] from='client.54649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:14 smithi121 bash[36902]: cluster 2022-04-22T21:50:12.223868+0000 mgr.y (mgr.34107) 9015 : cluster [DBG] pgmap v9864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:14.515 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:14 smithi115 bash[45495]: cluster 2022-04-22T21:50:12.223868+0000 mgr.y (mgr.34107) 9015 : cluster [DBG] pgmap v9864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:14.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:14 smithi115 bash[46934]: cluster 2022-04-22T21:50:12.223868+0000 mgr.y (mgr.34107) 9015 : cluster [DBG] pgmap v9864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:14.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:15.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:15 smithi121 bash[36902]: audit 2022-04-22T21:50:15.018709+0000 mon.a (mon.0) 12748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:15 smithi115 bash[45495]: audit 2022-04-22T21:50:15.018709+0000 mon.a (mon.0) 12748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:15 smithi115 bash[46934]: audit 2022-04-22T21:50:15.018709+0000 mon.a (mon.0) 12748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:16 smithi121 bash[36902]: cluster 2022-04-22T21:50:14.224886+0000 mgr.y (mgr.34107) 9016 : cluster [DBG] pgmap v9865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:16 smithi115 bash[45495]: cluster 2022-04-22T21:50:14.224886+0000 mgr.y (mgr.34107) 9016 : cluster [DBG] pgmap v9865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:16 smithi115 bash[46934]: cluster 2022-04-22T21:50:14.224886+0000 mgr.y (mgr.34107) 9016 : cluster [DBG] pgmap v9865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:17 smithi121 bash[36902]: audit 2022-04-22T21:50:16.054708+0000 mon.a (mon.0) 12749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:17 smithi121 bash[36902]: audit 2022-04-22T21:50:16.315251+0000 mon.a (mon.0) 12750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:50:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:17 smithi121 bash[36902]: audit 2022-04-22T21:50:16.315693+0000 mon.a (mon.0) 12751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:50:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[45495]: audit 2022-04-22T21:50:16.054708+0000 mon.a (mon.0) 12749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[45495]: audit 2022-04-22T21:50:16.315251+0000 mon.a (mon.0) 12750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:50:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[45495]: audit 2022-04-22T21:50:16.315693+0000 mon.a (mon.0) 12751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:50:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[46934]: audit 2022-04-22T21:50:16.054708+0000 mon.a (mon.0) 12749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[46934]: audit 2022-04-22T21:50:16.315251+0000 mon.a (mon.0) 12750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:50:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:17 smithi115 bash[46934]: audit 2022-04-22T21:50:16.315693+0000 mon.a (mon.0) 12751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:50:18.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:18 smithi121 bash[36902]: cluster 2022-04-22T21:50:16.226023+0000 mgr.y (mgr.34107) 9017 : cluster [DBG] pgmap v9866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:18 smithi115 bash[45495]: cluster 2022-04-22T21:50:16.226023+0000 mgr.y (mgr.34107) 9017 : cluster [DBG] pgmap v9866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:18 smithi115 bash[46934]: cluster 2022-04-22T21:50:16.226023+0000 mgr.y (mgr.34107) 9017 : cluster [DBG] pgmap v9866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:20 smithi121 bash[36902]: cluster 2022-04-22T21:50:18.227184+0000 mgr.y (mgr.34107) 9018 : cluster [DBG] pgmap v9867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:20 smithi121 bash[36902]: audit 2022-04-22T21:50:20.021942+0000 mon.a (mon.0) 12752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:20 smithi115 bash[45495]: cluster 2022-04-22T21:50:18.227184+0000 mgr.y (mgr.34107) 9018 : cluster [DBG] pgmap v9867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:20.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:20 smithi115 bash[45495]: audit 2022-04-22T21:50:20.021942+0000 mon.a (mon.0) 12752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:20.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:20 smithi115 bash[46934]: cluster 2022-04-22T21:50:18.227184+0000 mgr.y (mgr.34107) 9018 : cluster [DBG] pgmap v9867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:20.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:20 smithi115 bash[46934]: audit 2022-04-22T21:50:20.021942+0000 mon.a (mon.0) 12752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:21 smithi115 bash[45495]: cluster 2022-04-22T21:50:20.228259+0000 mgr.y (mgr.34107) 9019 : cluster [DBG] pgmap v9868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:21 smithi115 bash[45495]: audit 2022-04-22T21:50:21.058647+0000 mon.a (mon.0) 12753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:21 smithi115 bash[46934]: cluster 2022-04-22T21:50:20.228259+0000 mgr.y (mgr.34107) 9019 : cluster [DBG] pgmap v9868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:21 smithi115 bash[46934]: audit 2022-04-22T21:50:21.058647+0000 mon.a (mon.0) 12753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:21 smithi121 bash[36902]: cluster 2022-04-22T21:50:20.228259+0000 mgr.y (mgr.34107) 9019 : cluster [DBG] pgmap v9868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:21 smithi121 bash[36902]: audit 2022-04-22T21:50:21.058647+0000 mon.a (mon.0) 12753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:22.653Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:23 smithi115 bash[45495]: cluster 2022-04-22T21:50:22.229353+0000 mgr.y (mgr.34107) 9020 : cluster [DBG] pgmap v9869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:23 smithi115 bash[46934]: cluster 2022-04-22T21:50:22.229353+0000 mgr.y (mgr.34107) 9020 : cluster [DBG] pgmap v9869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:24.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:23 smithi121 bash[36902]: cluster 2022-04-22T21:50:22.229353+0000 mgr.y (mgr.34107) 9020 : cluster [DBG] pgmap v9869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:24.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:25 smithi115 bash[45495]: cluster 2022-04-22T21:50:24.230456+0000 mgr.y (mgr.34107) 9021 : cluster [DBG] pgmap v9870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:25 smithi115 bash[45495]: audit 2022-04-22T21:50:25.025196+0000 mon.a (mon.0) 12754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:25 smithi115 bash[46934]: cluster 2022-04-22T21:50:24.230456+0000 mgr.y (mgr.34107) 9021 : cluster [DBG] pgmap v9870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:25 smithi115 bash[46934]: audit 2022-04-22T21:50:25.025196+0000 mon.a (mon.0) 12754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:25 smithi121 bash[36902]: cluster 2022-04-22T21:50:24.230456+0000 mgr.y (mgr.34107) 9021 : cluster [DBG] pgmap v9870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:25 smithi121 bash[36902]: audit 2022-04-22T21:50:25.025196+0000 mon.a (mon.0) 12754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:26 smithi115 bash[45495]: audit 2022-04-22T21:50:26.062608+0000 mon.a (mon.0) 12755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:26 smithi115 bash[46934]: audit 2022-04-22T21:50:26.062608+0000 mon.a (mon.0) 12755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:26 smithi121 bash[36902]: audit 2022-04-22T21:50:26.062608+0000 mon.a (mon.0) 12755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:27 smithi115 bash[45495]: cluster 2022-04-22T21:50:26.231632+0000 mgr.y (mgr.34107) 9022 : cluster [DBG] pgmap v9871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:27 smithi115 bash[46934]: cluster 2022-04-22T21:50:26.231632+0000 mgr.y (mgr.34107) 9022 : cluster [DBG] pgmap v9871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:27 smithi121 bash[36902]: cluster 2022-04-22T21:50:26.231632+0000 mgr.y (mgr.34107) 9022 : cluster [DBG] pgmap v9871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:29 smithi121 bash[36902]: cluster 2022-04-22T21:50:28.232775+0000 mgr.y (mgr.34107) 9023 : cluster [DBG] pgmap v9872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:29 smithi115 bash[45495]: cluster 2022-04-22T21:50:28.232775+0000 mgr.y (mgr.34107) 9023 : cluster [DBG] pgmap v9872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:29 smithi115 bash[46934]: cluster 2022-04-22T21:50:28.232775+0000 mgr.y (mgr.34107) 9023 : cluster [DBG] pgmap v9872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:30 smithi121 bash[36902]: audit 2022-04-22T21:50:30.028347+0000 mon.a (mon.0) 12756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:30 smithi115 bash[45495]: audit 2022-04-22T21:50:30.028347+0000 mon.a (mon.0) 12756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:31.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:30 smithi115 bash[46934]: audit 2022-04-22T21:50:30.028347+0000 mon.a (mon.0) 12756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:31 smithi121 bash[36902]: cluster 2022-04-22T21:50:30.233782+0000 mgr.y (mgr.34107) 9024 : cluster [DBG] pgmap v9873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:31 smithi121 bash[36902]: audit 2022-04-22T21:50:31.066225+0000 mon.a (mon.0) 12757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:32.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:31 smithi115 bash[45495]: cluster 2022-04-22T21:50:30.233782+0000 mgr.y (mgr.34107) 9024 : cluster [DBG] pgmap v9873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:32.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:31 smithi115 bash[45495]: audit 2022-04-22T21:50:31.066225+0000 mon.a (mon.0) 12757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:32.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:31 smithi115 bash[46934]: cluster 2022-04-22T21:50:30.233782+0000 mgr.y (mgr.34107) 9024 : cluster [DBG] pgmap v9873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:32.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:31 smithi115 bash[46934]: audit 2022-04-22T21:50:31.066225+0000 mon.a (mon.0) 12757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:32.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:33 smithi121 bash[36902]: cluster 2022-04-22T21:50:32.234813+0000 mgr.y (mgr.34107) 9025 : cluster [DBG] pgmap v9874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:33 smithi115 bash[45495]: cluster 2022-04-22T21:50:32.234813+0000 mgr.y (mgr.34107) 9025 : cluster [DBG] pgmap v9874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:33 smithi115 bash[46934]: cluster 2022-04-22T21:50:32.234813+0000 mgr.y (mgr.34107) 9025 : cluster [DBG] pgmap v9874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:34.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:35 smithi121 bash[36902]: cluster 2022-04-22T21:50:34.235898+0000 mgr.y (mgr.34107) 9026 : cluster [DBG] pgmap v9875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:35 smithi121 bash[36902]: audit 2022-04-22T21:50:35.031792+0000 mon.a (mon.0) 12758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:35 smithi115 bash[45495]: cluster 2022-04-22T21:50:34.235898+0000 mgr.y (mgr.34107) 9026 : cluster [DBG] pgmap v9875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:35 smithi115 bash[45495]: audit 2022-04-22T21:50:35.031792+0000 mon.a (mon.0) 12758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:35 smithi115 bash[46934]: cluster 2022-04-22T21:50:34.235898+0000 mgr.y (mgr.34107) 9026 : cluster [DBG] pgmap v9875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:36.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:35 smithi115 bash[46934]: audit 2022-04-22T21:50:35.031792+0000 mon.a (mon.0) 12758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:36 smithi121 bash[36902]: audit 2022-04-22T21:50:36.069884+0000 mon.a (mon.0) 12759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:36 smithi115 bash[45495]: audit 2022-04-22T21:50:36.069884+0000 mon.a (mon.0) 12759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:36 smithi115 bash[46934]: audit 2022-04-22T21:50:36.069884+0000 mon.a (mon.0) 12759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:37 smithi121 bash[36902]: cluster 2022-04-22T21:50:36.236860+0000 mgr.y (mgr.34107) 9027 : cluster [DBG] pgmap v9876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:37 smithi115 bash[45495]: cluster 2022-04-22T21:50:36.236860+0000 mgr.y (mgr.34107) 9027 : cluster [DBG] pgmap v9876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:37 smithi115 bash[46934]: cluster 2022-04-22T21:50:36.236860+0000 mgr.y (mgr.34107) 9027 : cluster [DBG] pgmap v9876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:40.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:39 smithi121 bash[36902]: cluster 2022-04-22T21:50:38.237931+0000 mgr.y (mgr.34107) 9028 : cluster [DBG] pgmap v9877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:39 smithi115 bash[45495]: cluster 2022-04-22T21:50:38.237931+0000 mgr.y (mgr.34107) 9028 : cluster [DBG] pgmap v9877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:39 smithi115 bash[46934]: cluster 2022-04-22T21:50:38.237931+0000 mgr.y (mgr.34107) 9028 : cluster [DBG] pgmap v9877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:40 smithi121 bash[36902]: audit 2022-04-22T21:50:40.034826+0000 mon.a (mon.0) 12760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:40 smithi115 bash[45495]: audit 2022-04-22T21:50:40.034826+0000 mon.a (mon.0) 12760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:40 smithi115 bash[46934]: audit 2022-04-22T21:50:40.034826+0000 mon.a (mon.0) 12760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:42.223 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:50:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:41 smithi121 bash[36902]: cluster 2022-04-22T21:50:40.239034+0000 mgr.y (mgr.34107) 9029 : cluster [DBG] pgmap v9878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:41 smithi121 bash[36902]: audit 2022-04-22T21:50:41.073435+0000 mon.a (mon.0) 12761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:41 smithi121 bash[36902]: audit 2022-04-22T21:50:41.470804+0000 mon.a (mon.0) 12762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[45495]: cluster 2022-04-22T21:50:40.239034+0000 mgr.y (mgr.34107) 9029 : cluster [DBG] pgmap v9878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[45495]: audit 2022-04-22T21:50:41.073435+0000 mon.a (mon.0) 12761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[45495]: audit 2022-04-22T21:50:41.470804+0000 mon.a (mon.0) 12762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[46934]: cluster 2022-04-22T21:50:40.239034+0000 mgr.y (mgr.34107) 9029 : cluster [DBG] pgmap v9878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[46934]: audit 2022-04-22T21:50:41.073435+0000 mon.a (mon.0) 12761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:41 smithi115 bash[46934]: audit 2022-04-22T21:50:41.470804+0000 mon.a (mon.0) 12762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:42.567 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:50:42.567 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:50:42.567 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:50:42.567 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:50:42.567 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:50:42.568 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:50:42.569 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:50:42.971 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:50:42.971 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:50:42.971 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:50:42.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:50:42.972 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:50:42.973 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:50:42.973 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:50:42.973 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:50:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:42.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:43 smithi121 bash[36902]: audit 2022-04-22T21:50:42.203956+0000 mgr.y (mgr.34107) 9030 : audit [DBG] from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:43 smithi121 bash[36902]: cluster 2022-04-22T21:50:42.240068+0000 mgr.y (mgr.34107) 9031 : cluster [DBG] pgmap v9879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:43 smithi121 bash[36902]: audit 2022-04-22T21:50:42.544228+0000 mgr.y (mgr.34107) 9032 : audit [DBG] from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:43 smithi121 bash[36902]: audit 2022-04-22T21:50:42.967745+0000 mon.c (mon.1) 316 : audit [DBG] from='client.? 172.21.15.115:0/64979386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[45495]: audit 2022-04-22T21:50:42.203956+0000 mgr.y (mgr.34107) 9030 : audit [DBG] from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[45495]: cluster 2022-04-22T21:50:42.240068+0000 mgr.y (mgr.34107) 9031 : cluster [DBG] pgmap v9879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[45495]: audit 2022-04-22T21:50:42.544228+0000 mgr.y (mgr.34107) 9032 : audit [DBG] from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[45495]: audit 2022-04-22T21:50:42.967745+0000 mon.c (mon.1) 316 : audit [DBG] from='client.? 172.21.15.115:0/64979386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[46934]: audit 2022-04-22T21:50:42.203956+0000 mgr.y (mgr.34107) 9030 : audit [DBG] from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[46934]: cluster 2022-04-22T21:50:42.240068+0000 mgr.y (mgr.34107) 9031 : cluster [DBG] pgmap v9879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[46934]: audit 2022-04-22T21:50:42.544228+0000 mgr.y (mgr.34107) 9032 : audit [DBG] from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:50:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:43 smithi115 bash[46934]: audit 2022-04-22T21:50:42.967745+0000 mon.c (mon.1) 316 : audit [DBG] from='client.? 172.21.15.115:0/64979386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:50:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:44.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:45 smithi121 bash[36902]: cluster 2022-04-22T21:50:44.241073+0000 mgr.y (mgr.34107) 9033 : cluster [DBG] pgmap v9880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:45 smithi121 bash[36902]: audit 2022-04-22T21:50:45.038209+0000 mon.a (mon.0) 12763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:45 smithi115 bash[45495]: cluster 2022-04-22T21:50:44.241073+0000 mgr.y (mgr.34107) 9033 : cluster [DBG] pgmap v9880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:45 smithi115 bash[45495]: audit 2022-04-22T21:50:45.038209+0000 mon.a (mon.0) 12763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:45 smithi115 bash[46934]: cluster 2022-04-22T21:50:44.241073+0000 mgr.y (mgr.34107) 9033 : cluster [DBG] pgmap v9880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:45 smithi115 bash[46934]: audit 2022-04-22T21:50:45.038209+0000 mon.a (mon.0) 12763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:46 smithi121 bash[36902]: audit 2022-04-22T21:50:46.077113+0000 mon.a (mon.0) 12764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:46 smithi121 bash[36902]: audit 2022-04-22T21:50:46.473481+0000 mon.a (mon.0) 12765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:46 smithi115 bash[45495]: audit 2022-04-22T21:50:46.077113+0000 mon.a (mon.0) 12764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:46 smithi115 bash[45495]: audit 2022-04-22T21:50:46.473481+0000 mon.a (mon.0) 12765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:46 smithi115 bash[46934]: audit 2022-04-22T21:50:46.077113+0000 mon.a (mon.0) 12764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:46 smithi115 bash[46934]: audit 2022-04-22T21:50:46.473481+0000 mon.a (mon.0) 12765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:47 smithi121 bash[36902]: cluster 2022-04-22T21:50:46.242197+0000 mgr.y (mgr.34107) 9034 : cluster [DBG] pgmap v9881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:47 smithi115 bash[45495]: cluster 2022-04-22T21:50:46.242197+0000 mgr.y (mgr.34107) 9034 : cluster [DBG] pgmap v9881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:48.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:47 smithi115 bash[46934]: cluster 2022-04-22T21:50:46.242197+0000 mgr.y (mgr.34107) 9034 : cluster [DBG] pgmap v9881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:49 smithi121 bash[36902]: cluster 2022-04-22T21:50:48.243330+0000 mgr.y (mgr.34107) 9035 : cluster [DBG] pgmap v9882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:49 smithi115 bash[45495]: cluster 2022-04-22T21:50:48.243330+0000 mgr.y (mgr.34107) 9035 : cluster [DBG] pgmap v9882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:49 smithi115 bash[46934]: cluster 2022-04-22T21:50:48.243330+0000 mgr.y (mgr.34107) 9035 : cluster [DBG] pgmap v9882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:50 smithi121 bash[36902]: audit 2022-04-22T21:50:50.041459+0000 mon.a (mon.0) 12766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:50 smithi115 bash[45495]: audit 2022-04-22T21:50:50.041459+0000 mon.a (mon.0) 12766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:50 smithi115 bash[46934]: audit 2022-04-22T21:50:50.041459+0000 mon.a (mon.0) 12766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:51 smithi121 bash[36902]: cluster 2022-04-22T21:50:50.244367+0000 mgr.y (mgr.34107) 9036 : cluster [DBG] pgmap v9883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:52.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:51 smithi121 bash[36902]: audit 2022-04-22T21:50:51.080839+0000 mon.a (mon.0) 12767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:52.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:51 smithi121 bash[36902]: audit 2022-04-22T21:50:51.476847+0000 mon.a (mon.0) 12768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[46934]: cluster 2022-04-22T21:50:50.244367+0000 mgr.y (mgr.34107) 9036 : cluster [DBG] pgmap v9883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[46934]: audit 2022-04-22T21:50:51.080839+0000 mon.a (mon.0) 12767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[46934]: audit 2022-04-22T21:50:51.476847+0000 mon.a (mon.0) 12768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[45495]: cluster 2022-04-22T21:50:50.244367+0000 mgr.y (mgr.34107) 9036 : cluster [DBG] pgmap v9883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[45495]: audit 2022-04-22T21:50:51.080839+0000 mon.a (mon.0) 12767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:52.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:51 smithi115 bash[45495]: audit 2022-04-22T21:50:51.476847+0000 mon.a (mon.0) 12768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:52.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:53 smithi121 bash[36902]: cluster 2022-04-22T21:50:52.245583+0000 mgr.y (mgr.34107) 9037 : cluster [DBG] pgmap v9884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:54.468 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:53 smithi115 bash[45495]: cluster 2022-04-22T21:50:52.245583+0000 mgr.y (mgr.34107) 9037 : cluster [DBG] pgmap v9884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:54.469 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:53 smithi115 bash[46934]: cluster 2022-04-22T21:50:52.245583+0000 mgr.y (mgr.34107) 9037 : cluster [DBG] pgmap v9884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:50:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:50:54.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:50:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:55 smithi121 bash[36902]: cluster 2022-04-22T21:50:54.246690+0000 mgr.y (mgr.34107) 9038 : cluster [DBG] pgmap v9885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:56.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:55 smithi121 bash[36902]: audit 2022-04-22T21:50:55.044431+0000 mon.a (mon.0) 12769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:55 smithi115 bash[45495]: cluster 2022-04-22T21:50:54.246690+0000 mgr.y (mgr.34107) 9038 : cluster [DBG] pgmap v9885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:55 smithi115 bash[45495]: audit 2022-04-22T21:50:55.044431+0000 mon.a (mon.0) 12769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:55 smithi115 bash[46934]: cluster 2022-04-22T21:50:54.246690+0000 mgr.y (mgr.34107) 9038 : cluster [DBG] pgmap v9885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:55 smithi115 bash[46934]: audit 2022-04-22T21:50:55.044431+0000 mon.a (mon.0) 12769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:50:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:56 smithi121 bash[36902]: audit 2022-04-22T21:50:56.084649+0000 mon.a (mon.0) 12770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:56 smithi121 bash[36902]: audit 2022-04-22T21:50:56.480010+0000 mon.a (mon.0) 12771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:56 smithi115 bash[45495]: audit 2022-04-22T21:50:56.084649+0000 mon.a (mon.0) 12770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:56 smithi115 bash[45495]: audit 2022-04-22T21:50:56.480010+0000 mon.a (mon.0) 12771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:56 smithi115 bash[46934]: audit 2022-04-22T21:50:56.084649+0000 mon.a (mon.0) 12770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:50:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:56 smithi115 bash[46934]: audit 2022-04-22T21:50:56.480010+0000 mon.a (mon.0) 12771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:50:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:57 smithi121 bash[36902]: cluster 2022-04-22T21:50:56.247825+0000 mgr.y (mgr.34107) 9039 : cluster [DBG] pgmap v9886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:57 smithi115 bash[45495]: cluster 2022-04-22T21:50:56.247825+0000 mgr.y (mgr.34107) 9039 : cluster [DBG] pgmap v9886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:50:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:57 smithi115 bash[46934]: cluster 2022-04-22T21:50:56.247825+0000 mgr.y (mgr.34107) 9039 : cluster [DBG] pgmap v9886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:50:59 smithi121 bash[36902]: cluster 2022-04-22T21:50:58.249211+0000 mgr.y (mgr.34107) 9040 : cluster [DBG] pgmap v9887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:50:59 smithi115 bash[45495]: cluster 2022-04-22T21:50:58.249211+0000 mgr.y (mgr.34107) 9040 : cluster [DBG] pgmap v9887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:00.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:50:59 smithi115 bash[46934]: cluster 2022-04-22T21:50:58.249211+0000 mgr.y (mgr.34107) 9040 : cluster [DBG] pgmap v9887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:00 smithi121 bash[36902]: audit 2022-04-22T21:51:00.047723+0000 mon.a (mon.0) 12772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:00 smithi115 bash[45495]: audit 2022-04-22T21:51:00.047723+0000 mon.a (mon.0) 12772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:00 smithi115 bash[46934]: audit 2022-04-22T21:51:00.047723+0000 mon.a (mon.0) 12772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:01 smithi121 bash[36902]: cluster 2022-04-22T21:51:00.250320+0000 mgr.y (mgr.34107) 9041 : cluster [DBG] pgmap v9888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:01 smithi121 bash[36902]: audit 2022-04-22T21:51:01.088489+0000 mon.a (mon.0) 12773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:01 smithi121 bash[36902]: audit 2022-04-22T21:51:01.482886+0000 mon.a (mon.0) 12774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:02.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[46934]: cluster 2022-04-22T21:51:00.250320+0000 mgr.y (mgr.34107) 9041 : cluster [DBG] pgmap v9888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[46934]: audit 2022-04-22T21:51:01.088489+0000 mon.a (mon.0) 12773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[46934]: audit 2022-04-22T21:51:01.482886+0000 mon.a (mon.0) 12774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[45495]: cluster 2022-04-22T21:51:00.250320+0000 mgr.y (mgr.34107) 9041 : cluster [DBG] pgmap v9888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[45495]: audit 2022-04-22T21:51:01.088489+0000 mon.a (mon.0) 12773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:01 smithi115 bash[45495]: audit 2022-04-22T21:51:01.482886+0000 mon.a (mon.0) 12774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:02.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:04.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:03 smithi121 bash[36902]: cluster 2022-04-22T21:51:02.251417+0000 mgr.y (mgr.34107) 9042 : cluster [DBG] pgmap v9889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:03 smithi115 bash[45495]: cluster 2022-04-22T21:51:02.251417+0000 mgr.y (mgr.34107) 9042 : cluster [DBG] pgmap v9889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:04.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:03 smithi115 bash[46934]: cluster 2022-04-22T21:51:02.251417+0000 mgr.y (mgr.34107) 9042 : cluster [DBG] pgmap v9889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:04.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:05 smithi121 bash[36902]: cluster 2022-04-22T21:51:04.252524+0000 mgr.y (mgr.34107) 9043 : cluster [DBG] pgmap v9890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:05 smithi121 bash[36902]: audit 2022-04-22T21:51:05.050747+0000 mon.a (mon.0) 12775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:05 smithi115 bash[46934]: cluster 2022-04-22T21:51:04.252524+0000 mgr.y (mgr.34107) 9043 : cluster [DBG] pgmap v9890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:05 smithi115 bash[46934]: audit 2022-04-22T21:51:05.050747+0000 mon.a (mon.0) 12775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:05 smithi115 bash[45495]: cluster 2022-04-22T21:51:04.252524+0000 mgr.y (mgr.34107) 9043 : cluster [DBG] pgmap v9890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:05 smithi115 bash[45495]: audit 2022-04-22T21:51:05.050747+0000 mon.a (mon.0) 12775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:06 smithi121 bash[36902]: audit 2022-04-22T21:51:06.092198+0000 mon.a (mon.0) 12776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:06 smithi121 bash[36902]: audit 2022-04-22T21:51:06.486025+0000 mon.a (mon.0) 12777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:06 smithi115 bash[45495]: audit 2022-04-22T21:51:06.092198+0000 mon.a (mon.0) 12776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:06 smithi115 bash[45495]: audit 2022-04-22T21:51:06.486025+0000 mon.a (mon.0) 12777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:06 smithi115 bash[46934]: audit 2022-04-22T21:51:06.092198+0000 mon.a (mon.0) 12776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:06 smithi115 bash[46934]: audit 2022-04-22T21:51:06.486025+0000 mon.a (mon.0) 12777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:08.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:07 smithi121 bash[36902]: cluster 2022-04-22T21:51:06.253541+0000 mgr.y (mgr.34107) 9044 : cluster [DBG] pgmap v9891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:07 smithi115 bash[45495]: cluster 2022-04-22T21:51:06.253541+0000 mgr.y (mgr.34107) 9044 : cluster [DBG] pgmap v9891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:07 smithi115 bash[46934]: cluster 2022-04-22T21:51:06.253541+0000 mgr.y (mgr.34107) 9044 : cluster [DBG] pgmap v9891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:10.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:09 smithi121 bash[36902]: cluster 2022-04-22T21:51:08.254691+0000 mgr.y (mgr.34107) 9045 : cluster [DBG] pgmap v9892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:09 smithi115 bash[45495]: cluster 2022-04-22T21:51:08.254691+0000 mgr.y (mgr.34107) 9045 : cluster [DBG] pgmap v9892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:09 smithi115 bash[46934]: cluster 2022-04-22T21:51:08.254691+0000 mgr.y (mgr.34107) 9045 : cluster [DBG] pgmap v9892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[45495]: audit 2022-04-22T21:51:10.053619+0000 mon.a (mon.0) 12778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[45495]: audit 2022-04-22T21:51:10.735209+0000 mon.a (mon.0) 12779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[45495]: audit 2022-04-22T21:51:10.738492+0000 mon.a (mon.0) 12780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[45495]: audit 2022-04-22T21:51:10.740333+0000 mon.a (mon.0) 12781 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[46934]: audit 2022-04-22T21:51:10.053619+0000 mon.a (mon.0) 12778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[46934]: audit 2022-04-22T21:51:10.735209+0000 mon.a (mon.0) 12779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:51:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[46934]: audit 2022-04-22T21:51:10.738492+0000 mon.a (mon.0) 12780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:51:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:10 smithi115 bash[46934]: audit 2022-04-22T21:51:10.740333+0000 mon.a (mon.0) 12781 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:51:11.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:10 smithi121 bash[36902]: audit 2022-04-22T21:51:10.053619+0000 mon.a (mon.0) 12778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:11.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:10 smithi121 bash[36902]: audit 2022-04-22T21:51:10.735209+0000 mon.a (mon.0) 12779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:51:11.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:10 smithi121 bash[36902]: audit 2022-04-22T21:51:10.738492+0000 mon.a (mon.0) 12780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:51:11.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:10 smithi121 bash[36902]: audit 2022-04-22T21:51:10.740333+0000 mon.a (mon.0) 12781 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:51:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:12 smithi121 bash[36902]: cluster 2022-04-22T21:51:10.255769+0000 mgr.y (mgr.34107) 9046 : cluster [DBG] pgmap v9893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:12 smithi121 bash[36902]: audit 2022-04-22T21:51:11.094438+0000 mon.a (mon.0) 12782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:51:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:12 smithi121 bash[36902]: audit 2022-04-22T21:51:11.095387+0000 mon.a (mon.0) 12783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:12 smithi121 bash[36902]: audit 2022-04-22T21:51:11.488951+0000 mon.a (mon.0) 12784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[45495]: cluster 2022-04-22T21:51:10.255769+0000 mgr.y (mgr.34107) 9046 : cluster [DBG] pgmap v9893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[45495]: audit 2022-04-22T21:51:11.094438+0000 mon.a (mon.0) 12782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[45495]: audit 2022-04-22T21:51:11.095387+0000 mon.a (mon.0) 12783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[45495]: audit 2022-04-22T21:51:11.488951+0000 mon.a (mon.0) 12784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[46934]: cluster 2022-04-22T21:51:10.255769+0000 mgr.y (mgr.34107) 9046 : cluster [DBG] pgmap v9893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[46934]: audit 2022-04-22T21:51:11.094438+0000 mon.a (mon.0) 12782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[46934]: audit 2022-04-22T21:51:11.095387+0000 mon.a (mon.0) 12783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[46934]: audit 2022-04-22T21:51:11.488951+0000 mon.a (mon.0) 12784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:12.657Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:13.347 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:51:13.691 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:51:13.692 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:51:13.693 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:51:13.693 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:51:14.098 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:51:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:51:14.100 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:51:14.100 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:51:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:14 smithi121 bash[36902]: cluster 2022-04-22T21:51:12.256845+0000 mgr.y (mgr.34107) 9047 : cluster [DBG] pgmap v9894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:14.519 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:14 smithi115 bash[45495]: cluster 2022-04-22T21:51:12.256845+0000 mgr.y (mgr.34107) 9047 : cluster [DBG] pgmap v9894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:14.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:14 smithi115 bash[46934]: cluster 2022-04-22T21:51:12.256845+0000 mgr.y (mgr.34107) 9047 : cluster [DBG] pgmap v9894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:14.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:15 smithi121 bash[36902]: audit 2022-04-22T21:51:13.329459+0000 mgr.y (mgr.34107) 9048 : audit [DBG] from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:15 smithi121 bash[36902]: audit 2022-04-22T21:51:13.666451+0000 mgr.y (mgr.34107) 9049 : audit [DBG] from='client.44754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:15 smithi121 bash[36902]: audit 2022-04-22T21:51:14.094696+0000 mon.c (mon.1) 317 : audit [DBG] from='client.? 172.21.15.115:0/3337858361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:15 smithi121 bash[36902]: audit 2022-04-22T21:51:15.056624+0000 mon.a (mon.0) 12785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[45495]: audit 2022-04-22T21:51:13.329459+0000 mgr.y (mgr.34107) 9048 : audit [DBG] from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[45495]: audit 2022-04-22T21:51:13.666451+0000 mgr.y (mgr.34107) 9049 : audit [DBG] from='client.44754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[45495]: audit 2022-04-22T21:51:14.094696+0000 mon.c (mon.1) 317 : audit [DBG] from='client.? 172.21.15.115:0/3337858361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:15.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[45495]: audit 2022-04-22T21:51:15.056624+0000 mon.a (mon.0) 12785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[46934]: audit 2022-04-22T21:51:13.329459+0000 mgr.y (mgr.34107) 9048 : audit [DBG] from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[46934]: audit 2022-04-22T21:51:13.666451+0000 mgr.y (mgr.34107) 9049 : audit [DBG] from='client.44754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[46934]: audit 2022-04-22T21:51:14.094696+0000 mon.c (mon.1) 317 : audit [DBG] from='client.? 172.21.15.115:0/3337858361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:15 smithi115 bash[46934]: audit 2022-04-22T21:51:15.056624+0000 mon.a (mon.0) 12785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:16 smithi121 bash[36902]: cluster 2022-04-22T21:51:14.257873+0000 mgr.y (mgr.34107) 9050 : cluster [DBG] pgmap v9895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:16 smithi121 bash[36902]: audit 2022-04-22T21:51:16.103268+0000 mon.a (mon.0) 12786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:16 smithi115 bash[45495]: cluster 2022-04-22T21:51:14.257873+0000 mgr.y (mgr.34107) 9050 : cluster [DBG] pgmap v9895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:16 smithi115 bash[45495]: audit 2022-04-22T21:51:16.103268+0000 mon.a (mon.0) 12786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:16 smithi115 bash[46934]: cluster 2022-04-22T21:51:14.257873+0000 mgr.y (mgr.34107) 9050 : cluster [DBG] pgmap v9895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:16 smithi115 bash[46934]: audit 2022-04-22T21:51:16.103268+0000 mon.a (mon.0) 12786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:17 smithi121 bash[36902]: audit 2022-04-22T21:51:16.322641+0000 mon.a (mon.0) 12787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:17 smithi121 bash[36902]: audit 2022-04-22T21:51:16.324521+0000 mon.a (mon.0) 12788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:17 smithi121 bash[36902]: audit 2022-04-22T21:51:16.491891+0000 mon.a (mon.0) 12789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[45495]: audit 2022-04-22T21:51:16.322641+0000 mon.a (mon.0) 12787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[45495]: audit 2022-04-22T21:51:16.324521+0000 mon.a (mon.0) 12788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[45495]: audit 2022-04-22T21:51:16.491891+0000 mon.a (mon.0) 12789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[46934]: audit 2022-04-22T21:51:16.322641+0000 mon.a (mon.0) 12787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[46934]: audit 2022-04-22T21:51:16.324521+0000 mon.a (mon.0) 12788 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:51:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:17 smithi115 bash[46934]: audit 2022-04-22T21:51:16.491891+0000 mon.a (mon.0) 12789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:18 smithi121 bash[36902]: cluster 2022-04-22T21:51:16.258993+0000 mgr.y (mgr.34107) 9051 : cluster [DBG] pgmap v9896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:18 smithi115 bash[45495]: cluster 2022-04-22T21:51:16.258993+0000 mgr.y (mgr.34107) 9051 : cluster [DBG] pgmap v9896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:18 smithi115 bash[46934]: cluster 2022-04-22T21:51:16.258993+0000 mgr.y (mgr.34107) 9051 : cluster [DBG] pgmap v9896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:20 smithi121 bash[36902]: cluster 2022-04-22T21:51:18.260100+0000 mgr.y (mgr.34107) 9052 : cluster [DBG] pgmap v9897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:20 smithi121 bash[36902]: audit 2022-04-22T21:51:20.059905+0000 mon.a (mon.0) 12790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:20 smithi115 bash[45495]: cluster 2022-04-22T21:51:18.260100+0000 mgr.y (mgr.34107) 9052 : cluster [DBG] pgmap v9897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:20 smithi115 bash[45495]: audit 2022-04-22T21:51:20.059905+0000 mon.a (mon.0) 12790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:20 smithi115 bash[46934]: cluster 2022-04-22T21:51:18.260100+0000 mgr.y (mgr.34107) 9052 : cluster [DBG] pgmap v9897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:20 smithi115 bash[46934]: audit 2022-04-22T21:51:20.059905+0000 mon.a (mon.0) 12790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:21 smithi121 bash[36902]: cluster 2022-04-22T21:51:20.261171+0000 mgr.y (mgr.34107) 9053 : cluster [DBG] pgmap v9898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:21 smithi121 bash[36902]: audit 2022-04-22T21:51:21.106815+0000 mon.a (mon.0) 12791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:21 smithi121 bash[36902]: audit 2022-04-22T21:51:21.494805+0000 mon.a (mon.0) 12792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[45495]: cluster 2022-04-22T21:51:20.261171+0000 mgr.y (mgr.34107) 9053 : cluster [DBG] pgmap v9898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[45495]: audit 2022-04-22T21:51:21.106815+0000 mon.a (mon.0) 12791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[45495]: audit 2022-04-22T21:51:21.494805+0000 mon.a (mon.0) 12792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[46934]: cluster 2022-04-22T21:51:20.261171+0000 mgr.y (mgr.34107) 9053 : cluster [DBG] pgmap v9898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[46934]: audit 2022-04-22T21:51:21.106815+0000 mon.a (mon.0) 12791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:21 smithi115 bash[46934]: audit 2022-04-22T21:51:21.494805+0000 mon.a (mon.0) 12792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:22.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:24.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:23 smithi121 bash[36902]: cluster 2022-04-22T21:51:22.262202+0000 mgr.y (mgr.34107) 9054 : cluster [DBG] pgmap v9899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:24.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:23 smithi115 bash[46934]: cluster 2022-04-22T21:51:22.262202+0000 mgr.y (mgr.34107) 9054 : cluster [DBG] pgmap v9899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:23 smithi115 bash[45495]: cluster 2022-04-22T21:51:22.262202+0000 mgr.y (mgr.34107) 9054 : cluster [DBG] pgmap v9899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:24.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:25 smithi121 bash[36902]: cluster 2022-04-22T21:51:24.263136+0000 mgr.y (mgr.34107) 9055 : cluster [DBG] pgmap v9900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:25 smithi121 bash[36902]: audit 2022-04-22T21:51:25.062896+0000 mon.a (mon.0) 12793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:25 smithi115 bash[45495]: cluster 2022-04-22T21:51:24.263136+0000 mgr.y (mgr.34107) 9055 : cluster [DBG] pgmap v9900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:25 smithi115 bash[45495]: audit 2022-04-22T21:51:25.062896+0000 mon.a (mon.0) 12793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:25 smithi115 bash[46934]: cluster 2022-04-22T21:51:24.263136+0000 mgr.y (mgr.34107) 9055 : cluster [DBG] pgmap v9900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:25 smithi115 bash[46934]: audit 2022-04-22T21:51:25.062896+0000 mon.a (mon.0) 12793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:26 smithi121 bash[36902]: audit 2022-04-22T21:51:26.110357+0000 mon.a (mon.0) 12794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:26 smithi121 bash[36902]: audit 2022-04-22T21:51:26.497892+0000 mon.a (mon.0) 12795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:26 smithi115 bash[45495]: audit 2022-04-22T21:51:26.110357+0000 mon.a (mon.0) 12794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:26 smithi115 bash[45495]: audit 2022-04-22T21:51:26.497892+0000 mon.a (mon.0) 12795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:26 smithi115 bash[46934]: audit 2022-04-22T21:51:26.110357+0000 mon.a (mon.0) 12794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:26 smithi115 bash[46934]: audit 2022-04-22T21:51:26.497892+0000 mon.a (mon.0) 12795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:27 smithi121 bash[36902]: cluster 2022-04-22T21:51:26.264196+0000 mgr.y (mgr.34107) 9056 : cluster [DBG] pgmap v9901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:27 smithi115 bash[45495]: cluster 2022-04-22T21:51:26.264196+0000 mgr.y (mgr.34107) 9056 : cluster [DBG] pgmap v9901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:27 smithi115 bash[46934]: cluster 2022-04-22T21:51:26.264196+0000 mgr.y (mgr.34107) 9056 : cluster [DBG] pgmap v9901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:29 smithi121 bash[36902]: cluster 2022-04-22T21:51:28.265464+0000 mgr.y (mgr.34107) 9057 : cluster [DBG] pgmap v9902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:29 smithi115 bash[45495]: cluster 2022-04-22T21:51:28.265464+0000 mgr.y (mgr.34107) 9057 : cluster [DBG] pgmap v9902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:29 smithi115 bash[46934]: cluster 2022-04-22T21:51:28.265464+0000 mgr.y (mgr.34107) 9057 : cluster [DBG] pgmap v9902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:30 smithi121 bash[36902]: audit 2022-04-22T21:51:30.065991+0000 mon.a (mon.0) 12796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:30 smithi115 bash[45495]: audit 2022-04-22T21:51:30.065991+0000 mon.a (mon.0) 12796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:30 smithi115 bash[46934]: audit 2022-04-22T21:51:30.065991+0000 mon.a (mon.0) 12796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:31 smithi121 bash[36902]: cluster 2022-04-22T21:51:30.266546+0000 mgr.y (mgr.34107) 9058 : cluster [DBG] pgmap v9903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:31 smithi121 bash[36902]: audit 2022-04-22T21:51:31.114204+0000 mon.a (mon.0) 12797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:31 smithi121 bash[36902]: audit 2022-04-22T21:51:31.501192+0000 mon.a (mon.0) 12798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:32.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[45495]: cluster 2022-04-22T21:51:30.266546+0000 mgr.y (mgr.34107) 9058 : cluster [DBG] pgmap v9903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[45495]: audit 2022-04-22T21:51:31.114204+0000 mon.a (mon.0) 12797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[45495]: audit 2022-04-22T21:51:31.501192+0000 mon.a (mon.0) 12798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[46934]: cluster 2022-04-22T21:51:30.266546+0000 mgr.y (mgr.34107) 9058 : cluster [DBG] pgmap v9903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[46934]: audit 2022-04-22T21:51:31.114204+0000 mon.a (mon.0) 12797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:31 smithi115 bash[46934]: audit 2022-04-22T21:51:31.501192+0000 mon.a (mon.0) 12798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:32.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:34.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:33 smithi121 bash[36902]: cluster 2022-04-22T21:51:32.267639+0000 mgr.y (mgr.34107) 9059 : cluster [DBG] pgmap v9904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:33 smithi115 bash[45495]: cluster 2022-04-22T21:51:32.267639+0000 mgr.y (mgr.34107) 9059 : cluster [DBG] pgmap v9904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:33 smithi115 bash[46934]: cluster 2022-04-22T21:51:32.267639+0000 mgr.y (mgr.34107) 9059 : cluster [DBG] pgmap v9904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:34.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:35 smithi121 bash[36902]: cluster 2022-04-22T21:51:34.268616+0000 mgr.y (mgr.34107) 9060 : cluster [DBG] pgmap v9905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:35 smithi121 bash[36902]: audit 2022-04-22T21:51:35.069219+0000 mon.a (mon.0) 12799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:35 smithi115 bash[45495]: cluster 2022-04-22T21:51:34.268616+0000 mgr.y (mgr.34107) 9060 : cluster [DBG] pgmap v9905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:35 smithi115 bash[45495]: audit 2022-04-22T21:51:35.069219+0000 mon.a (mon.0) 12799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:35 smithi115 bash[46934]: cluster 2022-04-22T21:51:34.268616+0000 mgr.y (mgr.34107) 9060 : cluster [DBG] pgmap v9905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:35 smithi115 bash[46934]: audit 2022-04-22T21:51:35.069219+0000 mon.a (mon.0) 12799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:36 smithi121 bash[36902]: audit 2022-04-22T21:51:36.118146+0000 mon.a (mon.0) 12800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:36 smithi121 bash[36902]: audit 2022-04-22T21:51:36.504139+0000 mon.a (mon.0) 12801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:36 smithi115 bash[45495]: audit 2022-04-22T21:51:36.118146+0000 mon.a (mon.0) 12800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:36 smithi115 bash[45495]: audit 2022-04-22T21:51:36.504139+0000 mon.a (mon.0) 12801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:36 smithi115 bash[46934]: audit 2022-04-22T21:51:36.118146+0000 mon.a (mon.0) 12800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:36 smithi115 bash[46934]: audit 2022-04-22T21:51:36.504139+0000 mon.a (mon.0) 12801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:37 smithi121 bash[36902]: cluster 2022-04-22T21:51:36.269595+0000 mgr.y (mgr.34107) 9061 : cluster [DBG] pgmap v9906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:37 smithi115 bash[45495]: cluster 2022-04-22T21:51:36.269595+0000 mgr.y (mgr.34107) 9061 : cluster [DBG] pgmap v9906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:37 smithi115 bash[46934]: cluster 2022-04-22T21:51:36.269595+0000 mgr.y (mgr.34107) 9061 : cluster [DBG] pgmap v9906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:39 smithi121 bash[36902]: cluster 2022-04-22T21:51:38.270727+0000 mgr.y (mgr.34107) 9062 : cluster [DBG] pgmap v9907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:39 smithi115 bash[45495]: cluster 2022-04-22T21:51:38.270727+0000 mgr.y (mgr.34107) 9062 : cluster [DBG] pgmap v9907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:40.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:39 smithi115 bash[46934]: cluster 2022-04-22T21:51:38.270727+0000 mgr.y (mgr.34107) 9062 : cluster [DBG] pgmap v9907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:40 smithi121 bash[36902]: audit 2022-04-22T21:51:40.072241+0000 mon.a (mon.0) 12802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:40 smithi115 bash[45495]: audit 2022-04-22T21:51:40.072241+0000 mon.a (mon.0) 12802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:40 smithi115 bash[46934]: audit 2022-04-22T21:51:40.072241+0000 mon.a (mon.0) 12802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:41 smithi121 bash[36902]: cluster 2022-04-22T21:51:40.271768+0000 mgr.y (mgr.34107) 9063 : cluster [DBG] pgmap v9908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:41 smithi121 bash[36902]: audit 2022-04-22T21:51:41.121955+0000 mon.a (mon.0) 12803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:41 smithi121 bash[36902]: audit 2022-04-22T21:51:41.507271+0000 mon.a (mon.0) 12804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[45495]: cluster 2022-04-22T21:51:40.271768+0000 mgr.y (mgr.34107) 9063 : cluster [DBG] pgmap v9908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[45495]: audit 2022-04-22T21:51:41.121955+0000 mon.a (mon.0) 12803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[45495]: audit 2022-04-22T21:51:41.507271+0000 mon.a (mon.0) 12804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[46934]: cluster 2022-04-22T21:51:40.271768+0000 mgr.y (mgr.34107) 9063 : cluster [DBG] pgmap v9908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[46934]: audit 2022-04-22T21:51:41.121955+0000 mon.a (mon.0) 12803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:41 smithi115 bash[46934]: audit 2022-04-22T21:51:41.507271+0000 mon.a (mon.0) 12804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:43.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:42.660Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:44.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:43 smithi121 bash[36902]: cluster 2022-04-22T21:51:42.272810+0000 mgr.y (mgr.34107) 9064 : cluster [DBG] pgmap v9909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:44.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:43 smithi115 bash[46934]: cluster 2022-04-22T21:51:42.272810+0000 mgr.y (mgr.34107) 9064 : cluster [DBG] pgmap v9909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:43 smithi115 bash[45495]: cluster 2022-04-22T21:51:42.272810+0000 mgr.y (mgr.34107) 9064 : cluster [DBG] pgmap v9909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:44.471 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:51:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:44.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:44.820 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:51:44.821 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:51:44.822 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:51:44.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:51:44.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:51:44.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:51:44.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:51:44.823 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:51:45.228 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:51:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:51:45.231 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:51:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:45 smithi121 bash[36902]: cluster 2022-04-22T21:51:44.273556+0000 mgr.y (mgr.34107) 9065 : cluster [DBG] pgmap v9910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:46.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:45 smithi121 bash[36902]: audit 2022-04-22T21:51:44.450045+0000 mgr.y (mgr.34107) 9066 : audit [DBG] from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:45 smithi121 bash[36902]: audit 2022-04-22T21:51:44.797334+0000 mgr.y (mgr.34107) 9067 : audit [DBG] from='client.54703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:45 smithi121 bash[36902]: audit 2022-04-22T21:51:45.075418+0000 mon.a (mon.0) 12805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:46.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:45 smithi121 bash[36902]: audit 2022-04-22T21:51:45.225280+0000 mon.a (mon.0) 12806 : audit [DBG] from='client.? 172.21.15.115:0/2294837894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[45495]: cluster 2022-04-22T21:51:44.273556+0000 mgr.y (mgr.34107) 9065 : cluster [DBG] pgmap v9910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[45495]: audit 2022-04-22T21:51:44.450045+0000 mgr.y (mgr.34107) 9066 : audit [DBG] from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[45495]: audit 2022-04-22T21:51:44.797334+0000 mgr.y (mgr.34107) 9067 : audit [DBG] from='client.54703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[45495]: audit 2022-04-22T21:51:45.075418+0000 mon.a (mon.0) 12805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:46.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[45495]: audit 2022-04-22T21:51:45.225280+0000 mon.a (mon.0) 12806 : audit [DBG] from='client.? 172.21.15.115:0/2294837894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[46934]: cluster 2022-04-22T21:51:44.273556+0000 mgr.y (mgr.34107) 9065 : cluster [DBG] pgmap v9910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[46934]: audit 2022-04-22T21:51:44.450045+0000 mgr.y (mgr.34107) 9066 : audit [DBG] from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[46934]: audit 2022-04-22T21:51:44.797334+0000 mgr.y (mgr.34107) 9067 : audit [DBG] from='client.54703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:51:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[46934]: audit 2022-04-22T21:51:45.075418+0000 mon.a (mon.0) 12805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:46.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:45 smithi115 bash[46934]: audit 2022-04-22T21:51:45.225280+0000 mon.a (mon.0) 12806 : audit [DBG] from='client.? 172.21.15.115:0/2294837894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:51:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:46 smithi121 bash[36902]: audit 2022-04-22T21:51:46.125660+0000 mon.a (mon.0) 12807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:46 smithi121 bash[36902]: audit 2022-04-22T21:51:46.510295+0000 mon.a (mon.0) 12808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:46 smithi115 bash[45495]: audit 2022-04-22T21:51:46.125660+0000 mon.a (mon.0) 12807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:46 smithi115 bash[45495]: audit 2022-04-22T21:51:46.510295+0000 mon.a (mon.0) 12808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:46 smithi115 bash[46934]: audit 2022-04-22T21:51:46.125660+0000 mon.a (mon.0) 12807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:46 smithi115 bash[46934]: audit 2022-04-22T21:51:46.510295+0000 mon.a (mon.0) 12808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:47 smithi121 bash[36902]: cluster 2022-04-22T21:51:46.274521+0000 mgr.y (mgr.34107) 9068 : cluster [DBG] pgmap v9911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:47 smithi115 bash[45495]: cluster 2022-04-22T21:51:46.274521+0000 mgr.y (mgr.34107) 9068 : cluster [DBG] pgmap v9911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:47 smithi115 bash[46934]: cluster 2022-04-22T21:51:46.274521+0000 mgr.y (mgr.34107) 9068 : cluster [DBG] pgmap v9911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:50.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:49 smithi121 bash[36902]: cluster 2022-04-22T21:51:48.275692+0000 mgr.y (mgr.34107) 9069 : cluster [DBG] pgmap v9912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:49 smithi115 bash[45495]: cluster 2022-04-22T21:51:48.275692+0000 mgr.y (mgr.34107) 9069 : cluster [DBG] pgmap v9912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:49 smithi115 bash[46934]: cluster 2022-04-22T21:51:48.275692+0000 mgr.y (mgr.34107) 9069 : cluster [DBG] pgmap v9912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:51.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:50 smithi121 bash[36902]: audit 2022-04-22T21:51:50.078048+0000 mon.a (mon.0) 12809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:50 smithi115 bash[45495]: audit 2022-04-22T21:51:50.078048+0000 mon.a (mon.0) 12809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:51.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:50 smithi115 bash[46934]: audit 2022-04-22T21:51:50.078048+0000 mon.a (mon.0) 12809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:51 smithi121 bash[36902]: cluster 2022-04-22T21:51:50.276685+0000 mgr.y (mgr.34107) 9070 : cluster [DBG] pgmap v9913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:51 smithi121 bash[36902]: audit 2022-04-22T21:51:51.129323+0000 mon.a (mon.0) 12810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:51 smithi121 bash[36902]: audit 2022-04-22T21:51:51.513319+0000 mon.a (mon.0) 12811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[45495]: cluster 2022-04-22T21:51:50.276685+0000 mgr.y (mgr.34107) 9070 : cluster [DBG] pgmap v9913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[45495]: audit 2022-04-22T21:51:51.129323+0000 mon.a (mon.0) 12810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[45495]: audit 2022-04-22T21:51:51.513319+0000 mon.a (mon.0) 12811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[46934]: cluster 2022-04-22T21:51:50.276685+0000 mgr.y (mgr.34107) 9070 : cluster [DBG] pgmap v9913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[46934]: audit 2022-04-22T21:51:51.129323+0000 mon.a (mon.0) 12810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:51 smithi115 bash[46934]: audit 2022-04-22T21:51:51.513319+0000 mon.a (mon.0) 12811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:52.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:53 smithi121 bash[36902]: cluster 2022-04-22T21:51:52.277772+0000 mgr.y (mgr.34107) 9071 : cluster [DBG] pgmap v9914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:53 smithi115 bash[46934]: cluster 2022-04-22T21:51:52.277772+0000 mgr.y (mgr.34107) 9071 : cluster [DBG] pgmap v9914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:53 smithi115 bash[45495]: cluster 2022-04-22T21:51:52.277772+0000 mgr.y (mgr.34107) 9071 : cluster [DBG] pgmap v9914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:51:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:51:54.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:51:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:55 smithi121 bash[36902]: cluster 2022-04-22T21:51:54.278879+0000 mgr.y (mgr.34107) 9072 : cluster [DBG] pgmap v9915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:55 smithi121 bash[36902]: audit 2022-04-22T21:51:55.081034+0000 mon.a (mon.0) 12812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:55 smithi115 bash[45495]: cluster 2022-04-22T21:51:54.278879+0000 mgr.y (mgr.34107) 9072 : cluster [DBG] pgmap v9915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:55 smithi115 bash[45495]: audit 2022-04-22T21:51:55.081034+0000 mon.a (mon.0) 12812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:55 smithi115 bash[46934]: cluster 2022-04-22T21:51:54.278879+0000 mgr.y (mgr.34107) 9072 : cluster [DBG] pgmap v9915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:55 smithi115 bash[46934]: audit 2022-04-22T21:51:55.081034+0000 mon.a (mon.0) 12812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:51:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:56 smithi121 bash[36902]: audit 2022-04-22T21:51:56.132888+0000 mon.a (mon.0) 12813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:56 smithi121 bash[36902]: audit 2022-04-22T21:51:56.516412+0000 mon.a (mon.0) 12814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:56 smithi115 bash[45495]: audit 2022-04-22T21:51:56.132888+0000 mon.a (mon.0) 12813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:56 smithi115 bash[45495]: audit 2022-04-22T21:51:56.516412+0000 mon.a (mon.0) 12814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:56 smithi115 bash[46934]: audit 2022-04-22T21:51:56.132888+0000 mon.a (mon.0) 12813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:51:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:56 smithi115 bash[46934]: audit 2022-04-22T21:51:56.516412+0000 mon.a (mon.0) 12814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:51:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:57 smithi121 bash[36902]: cluster 2022-04-22T21:51:56.280000+0000 mgr.y (mgr.34107) 9073 : cluster [DBG] pgmap v9916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:57 smithi115 bash[45495]: cluster 2022-04-22T21:51:56.280000+0000 mgr.y (mgr.34107) 9073 : cluster [DBG] pgmap v9916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:51:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:57 smithi115 bash[46934]: cluster 2022-04-22T21:51:56.280000+0000 mgr.y (mgr.34107) 9073 : cluster [DBG] pgmap v9916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:51:59 smithi121 bash[36902]: cluster 2022-04-22T21:51:58.281256+0000 mgr.y (mgr.34107) 9074 : cluster [DBG] pgmap v9917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:51:59 smithi115 bash[45495]: cluster 2022-04-22T21:51:58.281256+0000 mgr.y (mgr.34107) 9074 : cluster [DBG] pgmap v9917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:51:59 smithi115 bash[46934]: cluster 2022-04-22T21:51:58.281256+0000 mgr.y (mgr.34107) 9074 : cluster [DBG] pgmap v9917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:00 smithi121 bash[36902]: audit 2022-04-22T21:52:00.084227+0000 mon.a (mon.0) 12815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:01.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:00 smithi115 bash[45495]: audit 2022-04-22T21:52:00.084227+0000 mon.a (mon.0) 12815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:01.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:00 smithi115 bash[46934]: audit 2022-04-22T21:52:00.084227+0000 mon.a (mon.0) 12815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:01 smithi121 bash[36902]: cluster 2022-04-22T21:52:00.282320+0000 mgr.y (mgr.34107) 9075 : cluster [DBG] pgmap v9918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:01 smithi121 bash[36902]: audit 2022-04-22T21:52:01.136759+0000 mon.a (mon.0) 12816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:01 smithi121 bash[36902]: audit 2022-04-22T21:52:01.519712+0000 mon.a (mon.0) 12817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[45495]: cluster 2022-04-22T21:52:00.282320+0000 mgr.y (mgr.34107) 9075 : cluster [DBG] pgmap v9918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[45495]: audit 2022-04-22T21:52:01.136759+0000 mon.a (mon.0) 12816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[45495]: audit 2022-04-22T21:52:01.519712+0000 mon.a (mon.0) 12817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[46934]: cluster 2022-04-22T21:52:00.282320+0000 mgr.y (mgr.34107) 9075 : cluster [DBG] pgmap v9918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[46934]: audit 2022-04-22T21:52:01.136759+0000 mon.a (mon.0) 12816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:01 smithi115 bash[46934]: audit 2022-04-22T21:52:01.519712+0000 mon.a (mon.0) 12817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:02.661Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:03 smithi121 bash[36902]: cluster 2022-04-22T21:52:02.283571+0000 mgr.y (mgr.34107) 9076 : cluster [DBG] pgmap v9919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:03 smithi115 bash[45495]: cluster 2022-04-22T21:52:02.283571+0000 mgr.y (mgr.34107) 9076 : cluster [DBG] pgmap v9919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:03 smithi115 bash[46934]: cluster 2022-04-22T21:52:02.283571+0000 mgr.y (mgr.34107) 9076 : cluster [DBG] pgmap v9919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:04.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:05 smithi121 bash[36902]: cluster 2022-04-22T21:52:04.284636+0000 mgr.y (mgr.34107) 9077 : cluster [DBG] pgmap v9920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:05 smithi121 bash[36902]: audit 2022-04-22T21:52:05.087618+0000 mon.a (mon.0) 12818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:05 smithi115 bash[46934]: cluster 2022-04-22T21:52:04.284636+0000 mgr.y (mgr.34107) 9077 : cluster [DBG] pgmap v9920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:05 smithi115 bash[46934]: audit 2022-04-22T21:52:05.087618+0000 mon.a (mon.0) 12818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:05 smithi115 bash[45495]: cluster 2022-04-22T21:52:04.284636+0000 mgr.y (mgr.34107) 9077 : cluster [DBG] pgmap v9920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:05 smithi115 bash[45495]: audit 2022-04-22T21:52:05.087618+0000 mon.a (mon.0) 12818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:06 smithi121 bash[36902]: audit 2022-04-22T21:52:06.140399+0000 mon.a (mon.0) 12819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:06 smithi121 bash[36902]: audit 2022-04-22T21:52:06.522800+0000 mon.a (mon.0) 12820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:06 smithi115 bash[45495]: audit 2022-04-22T21:52:06.140399+0000 mon.a (mon.0) 12819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:06 smithi115 bash[45495]: audit 2022-04-22T21:52:06.522800+0000 mon.a (mon.0) 12820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:06 smithi115 bash[46934]: audit 2022-04-22T21:52:06.140399+0000 mon.a (mon.0) 12819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:06 smithi115 bash[46934]: audit 2022-04-22T21:52:06.522800+0000 mon.a (mon.0) 12820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:07 smithi121 bash[36902]: cluster 2022-04-22T21:52:06.285619+0000 mgr.y (mgr.34107) 9078 : cluster [DBG] pgmap v9921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:07 smithi115 bash[45495]: cluster 2022-04-22T21:52:06.285619+0000 mgr.y (mgr.34107) 9078 : cluster [DBG] pgmap v9921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:08.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:07 smithi115 bash[46934]: cluster 2022-04-22T21:52:06.285619+0000 mgr.y (mgr.34107) 9078 : cluster [DBG] pgmap v9921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:09 smithi121 bash[36902]: cluster 2022-04-22T21:52:08.286914+0000 mgr.y (mgr.34107) 9079 : cluster [DBG] pgmap v9922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:09 smithi115 bash[45495]: cluster 2022-04-22T21:52:08.286914+0000 mgr.y (mgr.34107) 9079 : cluster [DBG] pgmap v9922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:09 smithi115 bash[46934]: cluster 2022-04-22T21:52:08.286914+0000 mgr.y (mgr.34107) 9079 : cluster [DBG] pgmap v9922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:10 smithi121 bash[36902]: audit 2022-04-22T21:52:10.090766+0000 mon.a (mon.0) 12821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:10 smithi115 bash[45495]: audit 2022-04-22T21:52:10.090766+0000 mon.a (mon.0) 12821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:11.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:10 smithi115 bash[46934]: audit 2022-04-22T21:52:10.090766+0000 mon.a (mon.0) 12821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: cluster 2022-04-22T21:52:10.288042+0000 mgr.y (mgr.34107) 9080 : cluster [DBG] pgmap v9923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.104570+0000 mon.a (mon.0) 12822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:52:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.107644+0000 mon.a (mon.0) 12823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:52:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.109388+0000 mon.a (mon.0) 12824 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:52:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.143910+0000 mon.a (mon.0) 12825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.466968+0000 mon.a (mon.0) 12826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:52:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:11 smithi121 bash[36902]: audit 2022-04-22T21:52:11.525984+0000 mon.a (mon.0) 12827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: cluster 2022-04-22T21:52:10.288042+0000 mgr.y (mgr.34107) 9080 : cluster [DBG] pgmap v9923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.104570+0000 mon.a (mon.0) 12822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.107644+0000 mon.a (mon.0) 12823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.109388+0000 mon.a (mon.0) 12824 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.143910+0000 mon.a (mon.0) 12825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.466968+0000 mon.a (mon.0) 12826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[45495]: audit 2022-04-22T21:52:11.525984+0000 mon.a (mon.0) 12827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: cluster 2022-04-22T21:52:10.288042+0000 mgr.y (mgr.34107) 9080 : cluster [DBG] pgmap v9923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.104570+0000 mon.a (mon.0) 12822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:52:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.107644+0000 mon.a (mon.0) 12823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:52:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.109388+0000 mon.a (mon.0) 12824 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:52:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.143910+0000 mon.a (mon.0) 12825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.466968+0000 mon.a (mon.0) 12826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:52:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:11 smithi115 bash[46934]: audit 2022-04-22T21:52:11.525984+0000 mon.a (mon.0) 12827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:12.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:14.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:13 smithi121 bash[36902]: cluster 2022-04-22T21:52:12.289105+0000 mgr.y (mgr.34107) 9081 : cluster [DBG] pgmap v9924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:13 smithi115 bash[45495]: cluster 2022-04-22T21:52:12.289105+0000 mgr.y (mgr.34107) 9081 : cluster [DBG] pgmap v9924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:13 smithi115 bash[46934]: cluster 2022-04-22T21:52:12.289105+0000 mgr.y (mgr.34107) 9081 : cluster [DBG] pgmap v9924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:14.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:15.598 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:52:15.949 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:52:15.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:52:15.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:52:15.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:52:15.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:52:15.951 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:52:15.951 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:52:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:15 smithi121 bash[36902]: cluster 2022-04-22T21:52:14.290140+0000 mgr.y (mgr.34107) 9082 : cluster [DBG] pgmap v9925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:15 smithi121 bash[36902]: audit 2022-04-22T21:52:15.093918+0000 mon.a (mon.0) 12828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:15 smithi115 bash[45495]: cluster 2022-04-22T21:52:14.290140+0000 mgr.y (mgr.34107) 9082 : cluster [DBG] pgmap v9925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:15 smithi115 bash[45495]: audit 2022-04-22T21:52:15.093918+0000 mon.a (mon.0) 12828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:15 smithi115 bash[46934]: cluster 2022-04-22T21:52:14.290140+0000 mgr.y (mgr.34107) 9082 : cluster [DBG] pgmap v9925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:15 smithi115 bash[46934]: audit 2022-04-22T21:52:15.093918+0000 mon.a (mon.0) 12828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:16.364 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:52:16.364 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:52:16.364 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:52:16.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:52:16.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:52:16.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:16.366 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:52:16.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:52:16.366 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:52:16.366 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:15.579442+0000 mgr.y (mgr.34107) 9083 : audit [DBG] from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:15.925394+0000 mgr.y (mgr.34107) 9084 : audit [DBG] from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:16.147347+0000 mon.a (mon.0) 12829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:16.335703+0000 mon.a (mon.0) 12830 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:16.338956+0000 mon.a (mon.0) 12831 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:16.360994+0000 mon.a (mon.0) 12832 : audit [DBG] from='client.? 172.21.15.115:0/3973016319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:16 smithi121 bash[36902]: audit 2022-04-22T21:52:16.529141+0000 mon.a (mon.0) 12833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:15.579442+0000 mgr.y (mgr.34107) 9083 : audit [DBG] from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:15.925394+0000 mgr.y (mgr.34107) 9084 : audit [DBG] from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:16.147347+0000 mon.a (mon.0) 12829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:16.335703+0000 mon.a (mon.0) 12830 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:16.338956+0000 mon.a (mon.0) 12831 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:16.360994+0000 mon.a (mon.0) 12832 : audit [DBG] from='client.? 172.21.15.115:0/3973016319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[45495]: audit 2022-04-22T21:52:16.529141+0000 mon.a (mon.0) 12833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:15.579442+0000 mgr.y (mgr.34107) 9083 : audit [DBG] from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:15.925394+0000 mgr.y (mgr.34107) 9084 : audit [DBG] from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:16.147347+0000 mon.a (mon.0) 12829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:16.335703+0000 mon.a (mon.0) 12830 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:16.338956+0000 mon.a (mon.0) 12831 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:16.360994+0000 mon.a (mon.0) 12832 : audit [DBG] from='client.? 172.21.15.115:0/3973016319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:16 smithi115 bash[46934]: audit 2022-04-22T21:52:16.529141+0000 mon.a (mon.0) 12833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:17 smithi121 bash[36902]: cluster 2022-04-22T21:52:16.291052+0000 mgr.y (mgr.34107) 9085 : cluster [DBG] pgmap v9926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:17 smithi115 bash[45495]: cluster 2022-04-22T21:52:16.291052+0000 mgr.y (mgr.34107) 9085 : cluster [DBG] pgmap v9926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:18.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:17 smithi115 bash[46934]: cluster 2022-04-22T21:52:16.291052+0000 mgr.y (mgr.34107) 9085 : cluster [DBG] pgmap v9926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:20 smithi121 bash[36902]: cluster 2022-04-22T21:52:18.292256+0000 mgr.y (mgr.34107) 9086 : cluster [DBG] pgmap v9927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:20.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:20 smithi115 bash[46934]: cluster 2022-04-22T21:52:18.292256+0000 mgr.y (mgr.34107) 9086 : cluster [DBG] pgmap v9927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:19 smithi115 bash[45495]: cluster 2022-04-22T21:52:18.292256+0000 mgr.y (mgr.34107) 9086 : cluster [DBG] pgmap v9927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:21.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:21 smithi121 bash[36902]: audit 2022-04-22T21:52:20.096989+0000 mon.a (mon.0) 12834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:21 smithi115 bash[45495]: audit 2022-04-22T21:52:20.096989+0000 mon.a (mon.0) 12834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:21 smithi115 bash[46934]: audit 2022-04-22T21:52:20.096989+0000 mon.a (mon.0) 12834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[45495]: cluster 2022-04-22T21:52:20.293270+0000 mgr.y (mgr.34107) 9087 : cluster [DBG] pgmap v9928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[45495]: audit 2022-04-22T21:52:21.150808+0000 mon.a (mon.0) 12835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[45495]: audit 2022-04-22T21:52:21.532089+0000 mon.a (mon.0) 12836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[46934]: cluster 2022-04-22T21:52:20.293270+0000 mgr.y (mgr.34107) 9087 : cluster [DBG] pgmap v9928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[46934]: audit 2022-04-22T21:52:21.150808+0000 mon.a (mon.0) 12835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[46934]: audit 2022-04-22T21:52:21.532089+0000 mon.a (mon.0) 12836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:22 smithi121 bash[36902]: cluster 2022-04-22T21:52:20.293270+0000 mgr.y (mgr.34107) 9087 : cluster [DBG] pgmap v9928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:22 smithi121 bash[36902]: audit 2022-04-22T21:52:21.150808+0000 mon.a (mon.0) 12835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:22 smithi121 bash[36902]: audit 2022-04-22T21:52:21.532089+0000 mon.a (mon.0) 12836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:22.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:24 smithi115 bash[45495]: cluster 2022-04-22T21:52:22.294351+0000 mgr.y (mgr.34107) 9088 : cluster [DBG] pgmap v9929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:24 smithi115 bash[46934]: cluster 2022-04-22T21:52:22.294351+0000 mgr.y (mgr.34107) 9088 : cluster [DBG] pgmap v9929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:24 smithi121 bash[36902]: cluster 2022-04-22T21:52:22.294351+0000 mgr.y (mgr.34107) 9088 : cluster [DBG] pgmap v9929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:24.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:26 smithi115 bash[45495]: cluster 2022-04-22T21:52:24.295469+0000 mgr.y (mgr.34107) 9089 : cluster [DBG] pgmap v9930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:26 smithi115 bash[45495]: audit 2022-04-22T21:52:25.100058+0000 mon.a (mon.0) 12837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:26 smithi115 bash[46934]: cluster 2022-04-22T21:52:24.295469+0000 mgr.y (mgr.34107) 9089 : cluster [DBG] pgmap v9930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:26 smithi115 bash[46934]: audit 2022-04-22T21:52:25.100058+0000 mon.a (mon.0) 12837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:26 smithi121 bash[36902]: cluster 2022-04-22T21:52:24.295469+0000 mgr.y (mgr.34107) 9089 : cluster [DBG] pgmap v9930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:26 smithi121 bash[36902]: audit 2022-04-22T21:52:25.100058+0000 mon.a (mon.0) 12837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:27 smithi115 bash[45495]: audit 2022-04-22T21:52:26.154450+0000 mon.a (mon.0) 12838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:27 smithi115 bash[45495]: audit 2022-04-22T21:52:26.535091+0000 mon.a (mon.0) 12839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:27 smithi115 bash[46934]: audit 2022-04-22T21:52:26.154450+0000 mon.a (mon.0) 12838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:27 smithi115 bash[46934]: audit 2022-04-22T21:52:26.535091+0000 mon.a (mon.0) 12839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:27 smithi121 bash[36902]: audit 2022-04-22T21:52:26.154450+0000 mon.a (mon.0) 12838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:27 smithi121 bash[36902]: audit 2022-04-22T21:52:26.535091+0000 mon.a (mon.0) 12839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:28 smithi115 bash[45495]: cluster 2022-04-22T21:52:26.296453+0000 mgr.y (mgr.34107) 9090 : cluster [DBG] pgmap v9931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:28 smithi115 bash[46934]: cluster 2022-04-22T21:52:26.296453+0000 mgr.y (mgr.34107) 9090 : cluster [DBG] pgmap v9931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:28 smithi121 bash[36902]: cluster 2022-04-22T21:52:26.296453+0000 mgr.y (mgr.34107) 9090 : cluster [DBG] pgmap v9931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:30 smithi115 bash[45495]: cluster 2022-04-22T21:52:28.297563+0000 mgr.y (mgr.34107) 9091 : cluster [DBG] pgmap v9932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:30 smithi115 bash[46934]: cluster 2022-04-22T21:52:28.297563+0000 mgr.y (mgr.34107) 9091 : cluster [DBG] pgmap v9932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:30 smithi121 bash[36902]: cluster 2022-04-22T21:52:28.297563+0000 mgr.y (mgr.34107) 9091 : cluster [DBG] pgmap v9932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:31 smithi121 bash[36902]: audit 2022-04-22T21:52:30.103398+0000 mon.a (mon.0) 12840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:31.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:31 smithi115 bash[46934]: audit 2022-04-22T21:52:30.103398+0000 mon.a (mon.0) 12840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:31.576 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:31 smithi115 bash[45495]: audit 2022-04-22T21:52:30.103398+0000 mon.a (mon.0) 12840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:32 smithi121 bash[36902]: cluster 2022-04-22T21:52:30.298505+0000 mgr.y (mgr.34107) 9092 : cluster [DBG] pgmap v9933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:32 smithi121 bash[36902]: audit 2022-04-22T21:52:31.158280+0000 mon.a (mon.0) 12841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:32 smithi121 bash[36902]: audit 2022-04-22T21:52:31.538204+0000 mon.a (mon.0) 12842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[45495]: cluster 2022-04-22T21:52:30.298505+0000 mgr.y (mgr.34107) 9092 : cluster [DBG] pgmap v9933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[45495]: audit 2022-04-22T21:52:31.158280+0000 mon.a (mon.0) 12841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[45495]: audit 2022-04-22T21:52:31.538204+0000 mon.a (mon.0) 12842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[46934]: cluster 2022-04-22T21:52:30.298505+0000 mgr.y (mgr.34107) 9092 : cluster [DBG] pgmap v9933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[46934]: audit 2022-04-22T21:52:31.158280+0000 mon.a (mon.0) 12841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:32.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[46934]: audit 2022-04-22T21:52:31.538204+0000 mon.a (mon.0) 12842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:32.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:34 smithi121 bash[36902]: cluster 2022-04-22T21:52:32.299583+0000 mgr.y (mgr.34107) 9093 : cluster [DBG] pgmap v9934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:34.522 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:34 smithi115 bash[45495]: cluster 2022-04-22T21:52:32.299583+0000 mgr.y (mgr.34107) 9093 : cluster [DBG] pgmap v9934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:34.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:34 smithi115 bash[46934]: cluster 2022-04-22T21:52:32.299583+0000 mgr.y (mgr.34107) 9093 : cluster [DBG] pgmap v9934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:34.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:36 smithi121 bash[36902]: cluster 2022-04-22T21:52:34.300597+0000 mgr.y (mgr.34107) 9094 : cluster [DBG] pgmap v9935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:36 smithi121 bash[36902]: audit 2022-04-22T21:52:35.106333+0000 mon.a (mon.0) 12843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:36 smithi115 bash[45495]: cluster 2022-04-22T21:52:34.300597+0000 mgr.y (mgr.34107) 9094 : cluster [DBG] pgmap v9935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:36 smithi115 bash[45495]: audit 2022-04-22T21:52:35.106333+0000 mon.a (mon.0) 12843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:36 smithi115 bash[46934]: cluster 2022-04-22T21:52:34.300597+0000 mgr.y (mgr.34107) 9094 : cluster [DBG] pgmap v9935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:36 smithi115 bash[46934]: audit 2022-04-22T21:52:35.106333+0000 mon.a (mon.0) 12843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:37 smithi121 bash[36902]: audit 2022-04-22T21:52:36.161875+0000 mon.a (mon.0) 12844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:37.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:37 smithi121 bash[36902]: audit 2022-04-22T21:52:36.541513+0000 mon.a (mon.0) 12845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:37.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:37 smithi115 bash[45495]: audit 2022-04-22T21:52:36.161875+0000 mon.a (mon.0) 12844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:37.555 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:37 smithi115 bash[45495]: audit 2022-04-22T21:52:36.541513+0000 mon.a (mon.0) 12845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:37.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:37 smithi115 bash[46934]: audit 2022-04-22T21:52:36.161875+0000 mon.a (mon.0) 12844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:37.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:37 smithi115 bash[46934]: audit 2022-04-22T21:52:36.541513+0000 mon.a (mon.0) 12845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:38 smithi121 bash[36902]: cluster 2022-04-22T21:52:36.301578+0000 mgr.y (mgr.34107) 9095 : cluster [DBG] pgmap v9936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:38 smithi115 bash[45495]: cluster 2022-04-22T21:52:36.301578+0000 mgr.y (mgr.34107) 9095 : cluster [DBG] pgmap v9936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:38 smithi115 bash[46934]: cluster 2022-04-22T21:52:36.301578+0000 mgr.y (mgr.34107) 9095 : cluster [DBG] pgmap v9936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:40 smithi121 bash[36902]: cluster 2022-04-22T21:52:38.302710+0000 mgr.y (mgr.34107) 9096 : cluster [DBG] pgmap v9937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:40 smithi115 bash[45495]: cluster 2022-04-22T21:52:38.302710+0000 mgr.y (mgr.34107) 9096 : cluster [DBG] pgmap v9937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:40 smithi115 bash[46934]: cluster 2022-04-22T21:52:38.302710+0000 mgr.y (mgr.34107) 9096 : cluster [DBG] pgmap v9937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:41 smithi121 bash[36902]: audit 2022-04-22T21:52:40.109488+0000 mon.a (mon.0) 12846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:41 smithi115 bash[45495]: audit 2022-04-22T21:52:40.109488+0000 mon.a (mon.0) 12846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:41 smithi115 bash[46934]: audit 2022-04-22T21:52:40.109488+0000 mon.a (mon.0) 12846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:42 smithi121 bash[36902]: cluster 2022-04-22T21:52:40.303829+0000 mgr.y (mgr.34107) 9097 : cluster [DBG] pgmap v9938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:42 smithi121 bash[36902]: audit 2022-04-22T21:52:41.165556+0000 mon.a (mon.0) 12847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:42 smithi121 bash[36902]: audit 2022-04-22T21:52:41.544440+0000 mon.a (mon.0) 12848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[45495]: cluster 2022-04-22T21:52:40.303829+0000 mgr.y (mgr.34107) 9097 : cluster [DBG] pgmap v9938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[45495]: audit 2022-04-22T21:52:41.165556+0000 mon.a (mon.0) 12847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[45495]: audit 2022-04-22T21:52:41.544440+0000 mon.a (mon.0) 12848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[46934]: cluster 2022-04-22T21:52:40.303829+0000 mgr.y (mgr.34107) 9097 : cluster [DBG] pgmap v9938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[46934]: audit 2022-04-22T21:52:41.165556+0000 mon.a (mon.0) 12847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[46934]: audit 2022-04-22T21:52:41.544440+0000 mon.a (mon.0) 12848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:42.664Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:44 smithi121 bash[36902]: cluster 2022-04-22T21:52:42.304878+0000 mgr.y (mgr.34107) 9098 : cluster [DBG] pgmap v9939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:44.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:44 smithi115 bash[45495]: cluster 2022-04-22T21:52:42.304878+0000 mgr.y (mgr.34107) 9098 : cluster [DBG] pgmap v9939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:44.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:44 smithi115 bash[46934]: cluster 2022-04-22T21:52:42.304878+0000 mgr.y (mgr.34107) 9098 : cluster [DBG] pgmap v9939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:44.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:46 smithi121 bash[36902]: cluster 2022-04-22T21:52:44.305953+0000 mgr.y (mgr.34107) 9099 : cluster [DBG] pgmap v9940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:46 smithi121 bash[36902]: audit 2022-04-22T21:52:45.112592+0000 mon.a (mon.0) 12849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:46 smithi115 bash[46934]: cluster 2022-04-22T21:52:44.305953+0000 mgr.y (mgr.34107) 9099 : cluster [DBG] pgmap v9940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:46 smithi115 bash[46934]: audit 2022-04-22T21:52:45.112592+0000 mon.a (mon.0) 12849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:46 smithi115 bash[45495]: cluster 2022-04-22T21:52:44.305953+0000 mgr.y (mgr.34107) 9099 : cluster [DBG] pgmap v9940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:46.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:46 smithi115 bash[45495]: audit 2022-04-22T21:52:45.112592+0000 mon.a (mon.0) 12849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:46.738 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:52:47.081 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:52:47.155 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:52:47.156 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:52:47.156 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:52:47.156 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:52:47.157 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:52:47.157 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:52:47.157 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:52:47.157 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:52:47.157 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:52:47.158 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:52:47.158 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:52:47.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:52:47.158 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:52:47.159 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:52:47.159 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:52:47.159 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:52:47.160 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:52:47.160 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:52:47.160 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:52:47.160 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:52:47.484 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:52:47.485 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:52:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:47 smithi121 bash[36902]: audit 2022-04-22T21:52:46.169393+0000 mon.a (mon.0) 12850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:47 smithi121 bash[36902]: audit 2022-04-22T21:52:46.547113+0000 mon.a (mon.0) 12851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:47 smithi115 bash[45495]: audit 2022-04-22T21:52:46.169393+0000 mon.a (mon.0) 12850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:47 smithi115 bash[45495]: audit 2022-04-22T21:52:46.547113+0000 mon.a (mon.0) 12851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:47 smithi115 bash[46934]: audit 2022-04-22T21:52:46.169393+0000 mon.a (mon.0) 12850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:47 smithi115 bash[46934]: audit 2022-04-22T21:52:46.547113+0000 mon.a (mon.0) 12851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:48 smithi121 bash[36902]: cluster 2022-04-22T21:52:46.307032+0000 mgr.y (mgr.34107) 9100 : cluster [DBG] pgmap v9941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:48 smithi121 bash[36902]: audit 2022-04-22T21:52:46.718961+0000 mgr.y (mgr.34107) 9101 : audit [DBG] from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:48 smithi121 bash[36902]: audit 2022-04-22T21:52:47.057853+0000 mgr.y (mgr.34107) 9102 : audit [DBG] from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:48 smithi121 bash[36902]: audit 2022-04-22T21:52:47.480501+0000 mon.a (mon.0) 12852 : audit [DBG] from='client.? 172.21.15.115:0/757430276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[45495]: cluster 2022-04-22T21:52:46.307032+0000 mgr.y (mgr.34107) 9100 : cluster [DBG] pgmap v9941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[45495]: audit 2022-04-22T21:52:46.718961+0000 mgr.y (mgr.34107) 9101 : audit [DBG] from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[45495]: audit 2022-04-22T21:52:47.057853+0000 mgr.y (mgr.34107) 9102 : audit [DBG] from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[45495]: audit 2022-04-22T21:52:47.480501+0000 mon.a (mon.0) 12852 : audit [DBG] from='client.? 172.21.15.115:0/757430276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[46934]: cluster 2022-04-22T21:52:46.307032+0000 mgr.y (mgr.34107) 9100 : cluster [DBG] pgmap v9941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[46934]: audit 2022-04-22T21:52:46.718961+0000 mgr.y (mgr.34107) 9101 : audit [DBG] from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[46934]: audit 2022-04-22T21:52:47.057853+0000 mgr.y (mgr.34107) 9102 : audit [DBG] from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:52:48.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:48 smithi115 bash[46934]: audit 2022-04-22T21:52:47.480501+0000 mon.a (mon.0) 12852 : audit [DBG] from='client.? 172.21.15.115:0/757430276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:52:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:50 smithi121 bash[36902]: cluster 2022-04-22T21:52:48.308199+0000 mgr.y (mgr.34107) 9103 : cluster [DBG] pgmap v9942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:50 smithi115 bash[45495]: cluster 2022-04-22T21:52:48.308199+0000 mgr.y (mgr.34107) 9103 : cluster [DBG] pgmap v9942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:50 smithi115 bash[46934]: cluster 2022-04-22T21:52:48.308199+0000 mgr.y (mgr.34107) 9103 : cluster [DBG] pgmap v9942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:51 smithi121 bash[36902]: audit 2022-04-22T21:52:50.115233+0000 mon.a (mon.0) 12853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:51 smithi115 bash[45495]: audit 2022-04-22T21:52:50.115233+0000 mon.a (mon.0) 12853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:51 smithi115 bash[46934]: audit 2022-04-22T21:52:50.115233+0000 mon.a (mon.0) 12853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:52 smithi121 bash[36902]: cluster 2022-04-22T21:52:50.309269+0000 mgr.y (mgr.34107) 9104 : cluster [DBG] pgmap v9943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:52 smithi121 bash[36902]: audit 2022-04-22T21:52:51.173221+0000 mon.a (mon.0) 12854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:52 smithi121 bash[36902]: audit 2022-04-22T21:52:51.549903+0000 mon.a (mon.0) 12855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[45495]: cluster 2022-04-22T21:52:50.309269+0000 mgr.y (mgr.34107) 9104 : cluster [DBG] pgmap v9943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[45495]: audit 2022-04-22T21:52:51.173221+0000 mon.a (mon.0) 12854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[45495]: audit 2022-04-22T21:52:51.549903+0000 mon.a (mon.0) 12855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[46934]: cluster 2022-04-22T21:52:50.309269+0000 mgr.y (mgr.34107) 9104 : cluster [DBG] pgmap v9943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[46934]: audit 2022-04-22T21:52:51.173221+0000 mon.a (mon.0) 12854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[46934]: audit 2022-04-22T21:52:51.549903+0000 mon.a (mon.0) 12855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:52.664Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:54 smithi121 bash[36902]: cluster 2022-04-22T21:52:52.310352+0000 mgr.y (mgr.34107) 9105 : cluster [DBG] pgmap v9944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:54.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:54 smithi115 bash[45495]: cluster 2022-04-22T21:52:52.310352+0000 mgr.y (mgr.34107) 9105 : cluster [DBG] pgmap v9944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:54.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:54 smithi115 bash[46934]: cluster 2022-04-22T21:52:52.310352+0000 mgr.y (mgr.34107) 9105 : cluster [DBG] pgmap v9944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:52:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:52:54.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:52:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:55 smithi121 bash[36902]: audit 2022-04-22T21:52:55.118147+0000 mon.a (mon.0) 12856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:55 smithi115 bash[45495]: audit 2022-04-22T21:52:55.118147+0000 mon.a (mon.0) 12856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:55 smithi115 bash[46934]: audit 2022-04-22T21:52:55.118147+0000 mon.a (mon.0) 12856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:52:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:56 smithi121 bash[36902]: cluster 2022-04-22T21:52:54.311413+0000 mgr.y (mgr.34107) 9106 : cluster [DBG] pgmap v9945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:56 smithi115 bash[46934]: cluster 2022-04-22T21:52:54.311413+0000 mgr.y (mgr.34107) 9106 : cluster [DBG] pgmap v9945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:56 smithi115 bash[45495]: cluster 2022-04-22T21:52:54.311413+0000 mgr.y (mgr.34107) 9106 : cluster [DBG] pgmap v9945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:57 smithi121 bash[36902]: audit 2022-04-22T21:52:56.177975+0000 mon.a (mon.0) 12857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:57 smithi121 bash[36902]: audit 2022-04-22T21:52:56.552705+0000 mon.a (mon.0) 12858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:57 smithi115 bash[45495]: audit 2022-04-22T21:52:56.177975+0000 mon.a (mon.0) 12857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:57 smithi115 bash[45495]: audit 2022-04-22T21:52:56.552705+0000 mon.a (mon.0) 12858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:57 smithi115 bash[46934]: audit 2022-04-22T21:52:56.177975+0000 mon.a (mon.0) 12857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:52:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:57 smithi115 bash[46934]: audit 2022-04-22T21:52:56.552705+0000 mon.a (mon.0) 12858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:52:58.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:52:58 smithi121 bash[36902]: cluster 2022-04-22T21:52:56.312430+0000 mgr.y (mgr.34107) 9107 : cluster [DBG] pgmap v9946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:52:58 smithi115 bash[45495]: cluster 2022-04-22T21:52:56.312430+0000 mgr.y (mgr.34107) 9107 : cluster [DBG] pgmap v9946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:52:58.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:52:58 smithi115 bash[46934]: cluster 2022-04-22T21:52:56.312430+0000 mgr.y (mgr.34107) 9107 : cluster [DBG] pgmap v9946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:00 smithi121 bash[36902]: cluster 2022-04-22T21:52:58.313552+0000 mgr.y (mgr.34107) 9108 : cluster [DBG] pgmap v9947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:00 smithi121 bash[36902]: audit 2022-04-22T21:53:00.121414+0000 mon.a (mon.0) 12859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:00 smithi115 bash[45495]: cluster 2022-04-22T21:52:58.313552+0000 mgr.y (mgr.34107) 9108 : cluster [DBG] pgmap v9947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:00.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:00 smithi115 bash[45495]: audit 2022-04-22T21:53:00.121414+0000 mon.a (mon.0) 12859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:00 smithi115 bash[46934]: cluster 2022-04-22T21:52:58.313552+0000 mgr.y (mgr.34107) 9108 : cluster [DBG] pgmap v9947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:00.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:00 smithi115 bash[46934]: audit 2022-04-22T21:53:00.121414+0000 mon.a (mon.0) 12859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:01 smithi121 bash[36902]: cluster 2022-04-22T21:53:00.314588+0000 mgr.y (mgr.34107) 9109 : cluster [DBG] pgmap v9948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:01 smithi121 bash[36902]: audit 2022-04-22T21:53:01.181335+0000 mon.a (mon.0) 12860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:01 smithi121 bash[36902]: audit 2022-04-22T21:53:01.555676+0000 mon.a (mon.0) 12861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[45495]: cluster 2022-04-22T21:53:00.314588+0000 mgr.y (mgr.34107) 9109 : cluster [DBG] pgmap v9948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[45495]: audit 2022-04-22T21:53:01.181335+0000 mon.a (mon.0) 12860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[45495]: audit 2022-04-22T21:53:01.555676+0000 mon.a (mon.0) 12861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[46934]: cluster 2022-04-22T21:53:00.314588+0000 mgr.y (mgr.34107) 9109 : cluster [DBG] pgmap v9948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[46934]: audit 2022-04-22T21:53:01.181335+0000 mon.a (mon.0) 12860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:01 smithi115 bash[46934]: audit 2022-04-22T21:53:01.555676+0000 mon.a (mon.0) 12861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:02.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:04.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:03 smithi121 bash[36902]: cluster 2022-04-22T21:53:02.315670+0000 mgr.y (mgr.34107) 9110 : cluster [DBG] pgmap v9949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:03 smithi115 bash[45495]: cluster 2022-04-22T21:53:02.315670+0000 mgr.y (mgr.34107) 9110 : cluster [DBG] pgmap v9949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:03 smithi115 bash[46934]: cluster 2022-04-22T21:53:02.315670+0000 mgr.y (mgr.34107) 9110 : cluster [DBG] pgmap v9949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:04.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:05 smithi121 bash[36902]: cluster 2022-04-22T21:53:04.316714+0000 mgr.y (mgr.34107) 9111 : cluster [DBG] pgmap v9950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:05 smithi121 bash[36902]: audit 2022-04-22T21:53:05.124608+0000 mon.a (mon.0) 12862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:05 smithi115 bash[45495]: cluster 2022-04-22T21:53:04.316714+0000 mgr.y (mgr.34107) 9111 : cluster [DBG] pgmap v9950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:05 smithi115 bash[45495]: audit 2022-04-22T21:53:05.124608+0000 mon.a (mon.0) 12862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:05 smithi115 bash[46934]: cluster 2022-04-22T21:53:04.316714+0000 mgr.y (mgr.34107) 9111 : cluster [DBG] pgmap v9950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:05 smithi115 bash[46934]: audit 2022-04-22T21:53:05.124608+0000 mon.a (mon.0) 12862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:06 smithi121 bash[36902]: audit 2022-04-22T21:53:06.185331+0000 mon.a (mon.0) 12863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:06 smithi121 bash[36902]: audit 2022-04-22T21:53:06.558742+0000 mon.a (mon.0) 12864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:06 smithi115 bash[45495]: audit 2022-04-22T21:53:06.185331+0000 mon.a (mon.0) 12863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:06 smithi115 bash[45495]: audit 2022-04-22T21:53:06.558742+0000 mon.a (mon.0) 12864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:06 smithi115 bash[46934]: audit 2022-04-22T21:53:06.185331+0000 mon.a (mon.0) 12863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:06 smithi115 bash[46934]: audit 2022-04-22T21:53:06.558742+0000 mon.a (mon.0) 12864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:08.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:07 smithi121 bash[36902]: cluster 2022-04-22T21:53:06.317790+0000 mgr.y (mgr.34107) 9112 : cluster [DBG] pgmap v9951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:07 smithi115 bash[45495]: cluster 2022-04-22T21:53:06.317790+0000 mgr.y (mgr.34107) 9112 : cluster [DBG] pgmap v9951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:07 smithi115 bash[46934]: cluster 2022-04-22T21:53:06.317790+0000 mgr.y (mgr.34107) 9112 : cluster [DBG] pgmap v9951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:09 smithi121 bash[36902]: cluster 2022-04-22T21:53:08.318938+0000 mgr.y (mgr.34107) 9113 : cluster [DBG] pgmap v9952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:09 smithi115 bash[45495]: cluster 2022-04-22T21:53:08.318938+0000 mgr.y (mgr.34107) 9113 : cluster [DBG] pgmap v9952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:10.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:09 smithi115 bash[46934]: cluster 2022-04-22T21:53:08.318938+0000 mgr.y (mgr.34107) 9113 : cluster [DBG] pgmap v9952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:10 smithi121 bash[36902]: audit 2022-04-22T21:53:10.127810+0000 mon.a (mon.0) 12865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:10 smithi115 bash[45495]: audit 2022-04-22T21:53:10.127810+0000 mon.a (mon.0) 12865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:10 smithi115 bash[46934]: audit 2022-04-22T21:53:10.127810+0000 mon.a (mon.0) 12865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: cluster 2022-04-22T21:53:10.320010+0000 mgr.y (mgr.34107) 9114 : cluster [DBG] pgmap v9953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.188705+0000 mon.a (mon.0) 12866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.475671+0000 mon.a (mon.0) 12867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.478598+0000 mon.a (mon.0) 12868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.480275+0000 mon.a (mon.0) 12869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:53:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.561769+0000 mon.a (mon.0) 12870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:11 smithi121 bash[36902]: audit 2022-04-22T21:53:11.836052+0000 mon.a (mon.0) 12871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: cluster 2022-04-22T21:53:10.320010+0000 mgr.y (mgr.34107) 9114 : cluster [DBG] pgmap v9953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.188705+0000 mon.a (mon.0) 12866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.475671+0000 mon.a (mon.0) 12867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.478598+0000 mon.a (mon.0) 12868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.480275+0000 mon.a (mon.0) 12869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:53:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.561769+0000 mon.a (mon.0) 12870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[45495]: audit 2022-04-22T21:53:11.836052+0000 mon.a (mon.0) 12871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: cluster 2022-04-22T21:53:10.320010+0000 mgr.y (mgr.34107) 9114 : cluster [DBG] pgmap v9953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.188705+0000 mon.a (mon.0) 12866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.475671+0000 mon.a (mon.0) 12867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.478598+0000 mon.a (mon.0) 12868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.480275+0000 mon.a (mon.0) 12869 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.561769+0000 mon.a (mon.0) 12870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:11 smithi115 bash[46934]: audit 2022-04-22T21:53:11.836052+0000 mon.a (mon.0) 12871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:53:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:12.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:13 smithi121 bash[36902]: cluster 2022-04-22T21:53:12.321131+0000 mgr.y (mgr.34107) 9115 : cluster [DBG] pgmap v9954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:13 smithi115 bash[46934]: cluster 2022-04-22T21:53:12.321131+0000 mgr.y (mgr.34107) 9115 : cluster [DBG] pgmap v9954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:13 smithi115 bash[45495]: cluster 2022-04-22T21:53:12.321131+0000 mgr.y (mgr.34107) 9115 : cluster [DBG] pgmap v9954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:14.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:15 smithi121 bash[36902]: cluster 2022-04-22T21:53:14.322226+0000 mgr.y (mgr.34107) 9116 : cluster [DBG] pgmap v9955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:15 smithi121 bash[36902]: audit 2022-04-22T21:53:15.131038+0000 mon.a (mon.0) 12872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:15 smithi115 bash[45495]: cluster 2022-04-22T21:53:14.322226+0000 mgr.y (mgr.34107) 9116 : cluster [DBG] pgmap v9955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:15 smithi115 bash[45495]: audit 2022-04-22T21:53:15.131038+0000 mon.a (mon.0) 12872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:15 smithi115 bash[46934]: cluster 2022-04-22T21:53:14.322226+0000 mgr.y (mgr.34107) 9116 : cluster [DBG] pgmap v9955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:16.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:15 smithi115 bash[46934]: audit 2022-04-22T21:53:15.131038+0000 mon.a (mon.0) 12872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:16 smithi121 bash[36902]: audit 2022-04-22T21:53:16.192526+0000 mon.a (mon.0) 12873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:16 smithi121 bash[36902]: audit 2022-04-22T21:53:16.347762+0000 mon.a (mon.0) 12874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:53:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:16 smithi121 bash[36902]: audit 2022-04-22T21:53:16.354621+0000 mon.a (mon.0) 12875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:53:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:16 smithi121 bash[36902]: audit 2022-04-22T21:53:16.564552+0000 mon.a (mon.0) 12876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[45495]: audit 2022-04-22T21:53:16.192526+0000 mon.a (mon.0) 12873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[45495]: audit 2022-04-22T21:53:16.347762+0000 mon.a (mon.0) 12874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[45495]: audit 2022-04-22T21:53:16.354621+0000 mon.a (mon.0) 12875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[45495]: audit 2022-04-22T21:53:16.564552+0000 mon.a (mon.0) 12876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[46934]: audit 2022-04-22T21:53:16.192526+0000 mon.a (mon.0) 12873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[46934]: audit 2022-04-22T21:53:16.347762+0000 mon.a (mon.0) 12874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[46934]: audit 2022-04-22T21:53:16.354621+0000 mon.a (mon.0) 12875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:53:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:16 smithi115 bash[46934]: audit 2022-04-22T21:53:16.564552+0000 mon.a (mon.0) 12876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:17.856 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:53:18.202 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:53:18.203 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:53:18.203 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:53:18.203 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:53:18.203 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:53:18.204 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:53:18.204 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:53:18.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:53:18.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:53:18.204 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:53:18.205 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:53:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:53:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:53:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:53:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:53:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:53:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:17 smithi121 bash[36902]: cluster 2022-04-22T21:53:16.323248+0000 mgr.y (mgr.34107) 9117 : cluster [DBG] pgmap v9956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:17 smithi115 bash[46934]: cluster 2022-04-22T21:53:16.323248+0000 mgr.y (mgr.34107) 9117 : cluster [DBG] pgmap v9956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:17 smithi115 bash[45495]: cluster 2022-04-22T21:53:16.323248+0000 mgr.y (mgr.34107) 9117 : cluster [DBG] pgmap v9956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:53:18.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:53:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:53:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:18 smithi121 bash[36902]: audit 2022-04-22T21:53:17.838087+0000 mgr.y (mgr.34107) 9118 : audit [DBG] from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:18 smithi121 bash[36902]: audit 2022-04-22T21:53:18.178114+0000 mgr.y (mgr.34107) 9119 : audit [DBG] from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:18 smithi121 bash[36902]: audit 2022-04-22T21:53:18.598769+0000 mon.a (mon.0) 12877 : audit [DBG] from='client.? 172.21.15.115:0/2271330553' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[45495]: audit 2022-04-22T21:53:17.838087+0000 mgr.y (mgr.34107) 9118 : audit [DBG] from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[45495]: audit 2022-04-22T21:53:18.178114+0000 mgr.y (mgr.34107) 9119 : audit [DBG] from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[45495]: audit 2022-04-22T21:53:18.598769+0000 mon.a (mon.0) 12877 : audit [DBG] from='client.? 172.21.15.115:0/2271330553' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[46934]: audit 2022-04-22T21:53:17.838087+0000 mgr.y (mgr.34107) 9118 : audit [DBG] from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[46934]: audit 2022-04-22T21:53:18.178114+0000 mgr.y (mgr.34107) 9119 : audit [DBG] from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:18 smithi115 bash[46934]: audit 2022-04-22T21:53:18.598769+0000 mon.a (mon.0) 12877 : audit [DBG] from='client.? 172.21.15.115:0/2271330553' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:20.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:19 smithi121 bash[36902]: cluster 2022-04-22T21:53:18.324044+0000 mgr.y (mgr.34107) 9120 : cluster [DBG] pgmap v9957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:20.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:19 smithi115 bash[46934]: cluster 2022-04-22T21:53:18.324044+0000 mgr.y (mgr.34107) 9120 : cluster [DBG] pgmap v9957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:19 smithi115 bash[45495]: cluster 2022-04-22T21:53:18.324044+0000 mgr.y (mgr.34107) 9120 : cluster [DBG] pgmap v9957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:20 smithi121 bash[36902]: audit 2022-04-22T21:53:20.134140+0000 mon.a (mon.0) 12878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:20 smithi115 bash[45495]: audit 2022-04-22T21:53:20.134140+0000 mon.a (mon.0) 12878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:20 smithi115 bash[46934]: audit 2022-04-22T21:53:20.134140+0000 mon.a (mon.0) 12878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:21 smithi121 bash[36902]: cluster 2022-04-22T21:53:20.325050+0000 mgr.y (mgr.34107) 9121 : cluster [DBG] pgmap v9958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:21 smithi121 bash[36902]: audit 2022-04-22T21:53:21.196175+0000 mon.a (mon.0) 12879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:21 smithi121 bash[36902]: audit 2022-04-22T21:53:21.567689+0000 mon.a (mon.0) 12880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[45495]: cluster 2022-04-22T21:53:20.325050+0000 mgr.y (mgr.34107) 9121 : cluster [DBG] pgmap v9958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[45495]: audit 2022-04-22T21:53:21.196175+0000 mon.a (mon.0) 12879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[45495]: audit 2022-04-22T21:53:21.567689+0000 mon.a (mon.0) 12880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[46934]: cluster 2022-04-22T21:53:20.325050+0000 mgr.y (mgr.34107) 9121 : cluster [DBG] pgmap v9958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[46934]: audit 2022-04-22T21:53:21.196175+0000 mon.a (mon.0) 12879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:21 smithi115 bash[46934]: audit 2022-04-22T21:53:21.567689+0000 mon.a (mon.0) 12880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:22.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:23 smithi121 bash[36902]: cluster 2022-04-22T21:53:22.326148+0000 mgr.y (mgr.34107) 9122 : cluster [DBG] pgmap v9959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:23 smithi115 bash[45495]: cluster 2022-04-22T21:53:22.326148+0000 mgr.y (mgr.34107) 9122 : cluster [DBG] pgmap v9959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:23 smithi115 bash[46934]: cluster 2022-04-22T21:53:22.326148+0000 mgr.y (mgr.34107) 9122 : cluster [DBG] pgmap v9959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:24.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:25 smithi121 bash[36902]: cluster 2022-04-22T21:53:24.327219+0000 mgr.y (mgr.34107) 9123 : cluster [DBG] pgmap v9960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:25 smithi121 bash[36902]: audit 2022-04-22T21:53:25.137025+0000 mon.a (mon.0) 12881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:25 smithi115 bash[45495]: cluster 2022-04-22T21:53:24.327219+0000 mgr.y (mgr.34107) 9123 : cluster [DBG] pgmap v9960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:25 smithi115 bash[45495]: audit 2022-04-22T21:53:25.137025+0000 mon.a (mon.0) 12881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:25 smithi115 bash[46934]: cluster 2022-04-22T21:53:24.327219+0000 mgr.y (mgr.34107) 9123 : cluster [DBG] pgmap v9960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:25 smithi115 bash[46934]: audit 2022-04-22T21:53:25.137025+0000 mon.a (mon.0) 12881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:26 smithi121 bash[36902]: audit 2022-04-22T21:53:26.199994+0000 mon.a (mon.0) 12882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:26 smithi121 bash[36902]: audit 2022-04-22T21:53:26.570703+0000 mon.a (mon.0) 12883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:26 smithi115 bash[45495]: audit 2022-04-22T21:53:26.199994+0000 mon.a (mon.0) 12882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:26 smithi115 bash[45495]: audit 2022-04-22T21:53:26.570703+0000 mon.a (mon.0) 12883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:26 smithi115 bash[46934]: audit 2022-04-22T21:53:26.199994+0000 mon.a (mon.0) 12882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:26 smithi115 bash[46934]: audit 2022-04-22T21:53:26.570703+0000 mon.a (mon.0) 12883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:27 smithi121 bash[36902]: cluster 2022-04-22T21:53:26.328284+0000 mgr.y (mgr.34107) 9124 : cluster [DBG] pgmap v9961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:27 smithi115 bash[45495]: cluster 2022-04-22T21:53:26.328284+0000 mgr.y (mgr.34107) 9124 : cluster [DBG] pgmap v9961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:27 smithi115 bash[46934]: cluster 2022-04-22T21:53:26.328284+0000 mgr.y (mgr.34107) 9124 : cluster [DBG] pgmap v9961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:29 smithi121 bash[36902]: cluster 2022-04-22T21:53:28.329431+0000 mgr.y (mgr.34107) 9125 : cluster [DBG] pgmap v9962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:29 smithi115 bash[45495]: cluster 2022-04-22T21:53:28.329431+0000 mgr.y (mgr.34107) 9125 : cluster [DBG] pgmap v9962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:29 smithi115 bash[46934]: cluster 2022-04-22T21:53:28.329431+0000 mgr.y (mgr.34107) 9125 : cluster [DBG] pgmap v9962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:30 smithi121 bash[36902]: audit 2022-04-22T21:53:30.140110+0000 mon.a (mon.0) 12884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:30 smithi115 bash[45495]: audit 2022-04-22T21:53:30.140110+0000 mon.a (mon.0) 12884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:30 smithi115 bash[46934]: audit 2022-04-22T21:53:30.140110+0000 mon.a (mon.0) 12884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:31 smithi121 bash[36902]: cluster 2022-04-22T21:53:30.330482+0000 mgr.y (mgr.34107) 9126 : cluster [DBG] pgmap v9963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:31 smithi121 bash[36902]: audit 2022-04-22T21:53:31.203684+0000 mon.a (mon.0) 12885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:31 smithi121 bash[36902]: audit 2022-04-22T21:53:31.573892+0000 mon.a (mon.0) 12886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:32.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[45495]: cluster 2022-04-22T21:53:30.330482+0000 mgr.y (mgr.34107) 9126 : cluster [DBG] pgmap v9963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[45495]: audit 2022-04-22T21:53:31.203684+0000 mon.a (mon.0) 12885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[45495]: audit 2022-04-22T21:53:31.573892+0000 mon.a (mon.0) 12886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[46934]: cluster 2022-04-22T21:53:30.330482+0000 mgr.y (mgr.34107) 9126 : cluster [DBG] pgmap v9963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[46934]: audit 2022-04-22T21:53:31.203684+0000 mon.a (mon.0) 12885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:31 smithi115 bash[46934]: audit 2022-04-22T21:53:31.573892+0000 mon.a (mon.0) 12886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:32.668Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:34.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:33 smithi121 bash[36902]: cluster 2022-04-22T21:53:32.331724+0000 mgr.y (mgr.34107) 9127 : cluster [DBG] pgmap v9964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:33 smithi115 bash[45495]: cluster 2022-04-22T21:53:32.331724+0000 mgr.y (mgr.34107) 9127 : cluster [DBG] pgmap v9964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:33 smithi115 bash[46934]: cluster 2022-04-22T21:53:32.331724+0000 mgr.y (mgr.34107) 9127 : cluster [DBG] pgmap v9964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:34.524Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:35 smithi121 bash[36902]: cluster 2022-04-22T21:53:34.332835+0000 mgr.y (mgr.34107) 9128 : cluster [DBG] pgmap v9965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:35 smithi121 bash[36902]: audit 2022-04-22T21:53:35.143273+0000 mon.a (mon.0) 12887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:35 smithi115 bash[45495]: cluster 2022-04-22T21:53:34.332835+0000 mgr.y (mgr.34107) 9128 : cluster [DBG] pgmap v9965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:35 smithi115 bash[45495]: audit 2022-04-22T21:53:35.143273+0000 mon.a (mon.0) 12887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:35 smithi115 bash[46934]: cluster 2022-04-22T21:53:34.332835+0000 mgr.y (mgr.34107) 9128 : cluster [DBG] pgmap v9965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:35 smithi115 bash[46934]: audit 2022-04-22T21:53:35.143273+0000 mon.a (mon.0) 12887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:36 smithi121 bash[36902]: audit 2022-04-22T21:53:36.207690+0000 mon.a (mon.0) 12888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:36 smithi121 bash[36902]: audit 2022-04-22T21:53:36.576838+0000 mon.a (mon.0) 12889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:36 smithi115 bash[45495]: audit 2022-04-22T21:53:36.207690+0000 mon.a (mon.0) 12888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:37.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:36 smithi115 bash[45495]: audit 2022-04-22T21:53:36.576838+0000 mon.a (mon.0) 12889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:36 smithi115 bash[46934]: audit 2022-04-22T21:53:36.207690+0000 mon.a (mon.0) 12888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:37.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:36 smithi115 bash[46934]: audit 2022-04-22T21:53:36.576838+0000 mon.a (mon.0) 12889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:37 smithi121 bash[36902]: cluster 2022-04-22T21:53:36.333849+0000 mgr.y (mgr.34107) 9129 : cluster [DBG] pgmap v9966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:37 smithi115 bash[45495]: cluster 2022-04-22T21:53:36.333849+0000 mgr.y (mgr.34107) 9129 : cluster [DBG] pgmap v9966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:37 smithi115 bash[46934]: cluster 2022-04-22T21:53:36.333849+0000 mgr.y (mgr.34107) 9129 : cluster [DBG] pgmap v9966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:40.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:39 smithi121 bash[36902]: cluster 2022-04-22T21:53:38.334931+0000 mgr.y (mgr.34107) 9130 : cluster [DBG] pgmap v9967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:40.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:39 smithi115 bash[45495]: cluster 2022-04-22T21:53:38.334931+0000 mgr.y (mgr.34107) 9130 : cluster [DBG] pgmap v9967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:40.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:39 smithi115 bash[46934]: cluster 2022-04-22T21:53:38.334931+0000 mgr.y (mgr.34107) 9130 : cluster [DBG] pgmap v9967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:41.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:40 smithi121 bash[36902]: audit 2022-04-22T21:53:40.146605+0000 mon.a (mon.0) 12890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:40 smithi115 bash[45495]: audit 2022-04-22T21:53:40.146605+0000 mon.a (mon.0) 12890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:40 smithi115 bash[46934]: audit 2022-04-22T21:53:40.146605+0000 mon.a (mon.0) 12890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:41 smithi121 bash[36902]: cluster 2022-04-22T21:53:40.336016+0000 mgr.y (mgr.34107) 9131 : cluster [DBG] pgmap v9968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:41 smithi121 bash[36902]: audit 2022-04-22T21:53:41.211537+0000 mon.a (mon.0) 12891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:41 smithi121 bash[36902]: audit 2022-04-22T21:53:41.579810+0000 mon.a (mon.0) 12892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[45495]: cluster 2022-04-22T21:53:40.336016+0000 mgr.y (mgr.34107) 9131 : cluster [DBG] pgmap v9968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[45495]: audit 2022-04-22T21:53:41.211537+0000 mon.a (mon.0) 12891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[45495]: audit 2022-04-22T21:53:41.579810+0000 mon.a (mon.0) 12892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[46934]: cluster 2022-04-22T21:53:40.336016+0000 mgr.y (mgr.34107) 9131 : cluster [DBG] pgmap v9968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[46934]: audit 2022-04-22T21:53:41.211537+0000 mon.a (mon.0) 12891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:41 smithi115 bash[46934]: audit 2022-04-22T21:53:41.579810+0000 mon.a (mon.0) 12892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:42.669Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:43 smithi121 bash[36902]: cluster 2022-04-22T21:53:42.337103+0000 mgr.y (mgr.34107) 9132 : cluster [DBG] pgmap v9969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:44.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:43 smithi115 bash[46934]: cluster 2022-04-22T21:53:42.337103+0000 mgr.y (mgr.34107) 9132 : cluster [DBG] pgmap v9969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:43 smithi115 bash[45495]: cluster 2022-04-22T21:53:42.337103+0000 mgr.y (mgr.34107) 9132 : cluster [DBG] pgmap v9969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:44.525Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:45 smithi121 bash[36902]: cluster 2022-04-22T21:53:44.338317+0000 mgr.y (mgr.34107) 9133 : cluster [DBG] pgmap v9970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:45 smithi121 bash[36902]: audit 2022-04-22T21:53:45.149452+0000 mon.a (mon.0) 12893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:45 smithi115 bash[45495]: cluster 2022-04-22T21:53:44.338317+0000 mgr.y (mgr.34107) 9133 : cluster [DBG] pgmap v9970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:45 smithi115 bash[45495]: audit 2022-04-22T21:53:45.149452+0000 mon.a (mon.0) 12893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:45 smithi115 bash[46934]: cluster 2022-04-22T21:53:44.338317+0000 mgr.y (mgr.34107) 9133 : cluster [DBG] pgmap v9970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:45 smithi115 bash[46934]: audit 2022-04-22T21:53:45.149452+0000 mon.a (mon.0) 12893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:46 smithi121 bash[36902]: audit 2022-04-22T21:53:46.215295+0000 mon.a (mon.0) 12894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:47.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:46 smithi121 bash[36902]: audit 2022-04-22T21:53:46.583053+0000 mon.a (mon.0) 12895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:46 smithi115 bash[46934]: audit 2022-04-22T21:53:46.215295+0000 mon.a (mon.0) 12894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:46 smithi115 bash[46934]: audit 2022-04-22T21:53:46.583053+0000 mon.a (mon.0) 12895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:46 smithi115 bash[45495]: audit 2022-04-22T21:53:46.215295+0000 mon.a (mon.0) 12894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:46 smithi115 bash[45495]: audit 2022-04-22T21:53:46.583053+0000 mon.a (mon.0) 12895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:47 smithi121 bash[36902]: cluster 2022-04-22T21:53:46.339427+0000 mgr.y (mgr.34107) 9134 : cluster [DBG] pgmap v9971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:47 smithi115 bash[45495]: cluster 2022-04-22T21:53:46.339427+0000 mgr.y (mgr.34107) 9134 : cluster [DBG] pgmap v9971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:47 smithi115 bash[46934]: cluster 2022-04-22T21:53:46.339427+0000 mgr.y (mgr.34107) 9134 : cluster [DBG] pgmap v9971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:48.976 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:53:49.330 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:53:49.330 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:53:49.330 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:53:49.330 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 258M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.5M - 1dbe0e931976 f4b044642a96 2022-04-22T21:53:49.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 583M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:53:49.332 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:53:49.737 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:53:49.737 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:53:49.738 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:53:49.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:49.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:53:49.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:53:49.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:49.740 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:53:49.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:53:49.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:49.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:53:49.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:53:49.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:53:49.741 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:53:49.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:53:49.741 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:53:49.742 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:53:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:49 smithi121 bash[36902]: cluster 2022-04-22T21:53:48.340594+0000 mgr.y (mgr.34107) 9135 : cluster [DBG] pgmap v9972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:49 smithi121 bash[36902]: audit 2022-04-22T21:53:48.957527+0000 mgr.y (mgr.34107) 9136 : audit [DBG] from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:49 smithi121 bash[36902]: audit 2022-04-22T21:53:49.733777+0000 mon.a (mon.0) 12896 : audit [DBG] from='client.? 172.21.15.115:0/1301849683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[45495]: cluster 2022-04-22T21:53:48.340594+0000 mgr.y (mgr.34107) 9135 : cluster [DBG] pgmap v9972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[45495]: audit 2022-04-22T21:53:48.957527+0000 mgr.y (mgr.34107) 9136 : audit [DBG] from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[45495]: audit 2022-04-22T21:53:49.733777+0000 mon.a (mon.0) 12896 : audit [DBG] from='client.? 172.21.15.115:0/1301849683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[46934]: cluster 2022-04-22T21:53:48.340594+0000 mgr.y (mgr.34107) 9135 : cluster [DBG] pgmap v9972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[46934]: audit 2022-04-22T21:53:48.957527+0000 mgr.y (mgr.34107) 9136 : audit [DBG] from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:50.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:49 smithi115 bash[46934]: audit 2022-04-22T21:53:49.733777+0000 mon.a (mon.0) 12896 : audit [DBG] from='client.? 172.21.15.115:0/1301849683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:53:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:51 smithi121 bash[36902]: audit 2022-04-22T21:53:49.303382+0000 mgr.y (mgr.34107) 9137 : audit [DBG] from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:51 smithi121 bash[36902]: audit 2022-04-22T21:53:50.152491+0000 mon.a (mon.0) 12897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:50 smithi115 bash[45495]: audit 2022-04-22T21:53:49.303382+0000 mgr.y (mgr.34107) 9137 : audit [DBG] from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:51 smithi115 bash[45495]: audit 2022-04-22T21:53:50.152491+0000 mon.a (mon.0) 12897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:51 smithi115 bash[46934]: audit 2022-04-22T21:53:49.303382+0000 mgr.y (mgr.34107) 9137 : audit [DBG] from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:53:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:51 smithi115 bash[46934]: audit 2022-04-22T21:53:50.152491+0000 mon.a (mon.0) 12897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:52.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:52 smithi121 bash[36902]: cluster 2022-04-22T21:53:50.341621+0000 mgr.y (mgr.34107) 9138 : cluster [DBG] pgmap v9973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:52 smithi121 bash[36902]: audit 2022-04-22T21:53:51.219113+0000 mon.a (mon.0) 12898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:52 smithi121 bash[36902]: audit 2022-04-22T21:53:51.586155+0000 mon.a (mon.0) 12899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[45495]: cluster 2022-04-22T21:53:50.341621+0000 mgr.y (mgr.34107) 9138 : cluster [DBG] pgmap v9973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[45495]: audit 2022-04-22T21:53:51.219113+0000 mon.a (mon.0) 12898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[45495]: audit 2022-04-22T21:53:51.586155+0000 mon.a (mon.0) 12899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[46934]: cluster 2022-04-22T21:53:50.341621+0000 mgr.y (mgr.34107) 9138 : cluster [DBG] pgmap v9973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[46934]: audit 2022-04-22T21:53:51.219113+0000 mon.a (mon.0) 12898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[46934]: audit 2022-04-22T21:53:51.586155+0000 mon.a (mon.0) 12899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:52.669Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:54.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:54 smithi121 bash[36902]: cluster 2022-04-22T21:53:52.342693+0000 mgr.y (mgr.34107) 9139 : cluster [DBG] pgmap v9974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:54 smithi115 bash[45495]: cluster 2022-04-22T21:53:52.342693+0000 mgr.y (mgr.34107) 9139 : cluster [DBG] pgmap v9974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:54 smithi115 bash[46934]: cluster 2022-04-22T21:53:52.342693+0000 mgr.y (mgr.34107) 9139 : cluster [DBG] pgmap v9974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:53:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:53:54.526Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:53:56.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:56 smithi115 bash[46934]: cluster 2022-04-22T21:53:54.343837+0000 mgr.y (mgr.34107) 9140 : cluster [DBG] pgmap v9975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:56 smithi115 bash[46934]: audit 2022-04-22T21:53:55.155611+0000 mon.a (mon.0) 12900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:56 smithi115 bash[45495]: cluster 2022-04-22T21:53:54.343837+0000 mgr.y (mgr.34107) 9140 : cluster [DBG] pgmap v9975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:56 smithi115 bash[45495]: audit 2022-04-22T21:53:55.155611+0000 mon.a (mon.0) 12900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:56 smithi121 bash[36902]: cluster 2022-04-22T21:53:54.343837+0000 mgr.y (mgr.34107) 9140 : cluster [DBG] pgmap v9975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:56 smithi121 bash[36902]: audit 2022-04-22T21:53:55.155611+0000 mon.a (mon.0) 12900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:53:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:57 smithi115 bash[45495]: audit 2022-04-22T21:53:56.222706+0000 mon.a (mon.0) 12901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:57 smithi115 bash[45495]: audit 2022-04-22T21:53:56.589182+0000 mon.a (mon.0) 12902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:57 smithi115 bash[46934]: audit 2022-04-22T21:53:56.222706+0000 mon.a (mon.0) 12901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:57 smithi115 bash[46934]: audit 2022-04-22T21:53:56.589182+0000 mon.a (mon.0) 12902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:57 smithi121 bash[36902]: audit 2022-04-22T21:53:56.222706+0000 mon.a (mon.0) 12901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:53:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:57 smithi121 bash[36902]: audit 2022-04-22T21:53:56.589182+0000 mon.a (mon.0) 12902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:53:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:53:58 smithi115 bash[45495]: cluster 2022-04-22T21:53:56.344842+0000 mgr.y (mgr.34107) 9141 : cluster [DBG] pgmap v9976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:53:58 smithi115 bash[46934]: cluster 2022-04-22T21:53:56.344842+0000 mgr.y (mgr.34107) 9141 : cluster [DBG] pgmap v9976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:53:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:53:58 smithi121 bash[36902]: cluster 2022-04-22T21:53:56.344842+0000 mgr.y (mgr.34107) 9141 : cluster [DBG] pgmap v9976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:00 smithi115 bash[45495]: cluster 2022-04-22T21:53:58.345982+0000 mgr.y (mgr.34107) 9142 : cluster [DBG] pgmap v9977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:00 smithi115 bash[46934]: cluster 2022-04-22T21:53:58.345982+0000 mgr.y (mgr.34107) 9142 : cluster [DBG] pgmap v9977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:00 smithi121 bash[36902]: cluster 2022-04-22T21:53:58.345982+0000 mgr.y (mgr.34107) 9142 : cluster [DBG] pgmap v9977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:01 smithi115 bash[45495]: audit 2022-04-22T21:54:00.158725+0000 mon.a (mon.0) 12903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:01 smithi115 bash[46934]: audit 2022-04-22T21:54:00.158725+0000 mon.a (mon.0) 12903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:01 smithi121 bash[36902]: audit 2022-04-22T21:54:00.158725+0000 mon.a (mon.0) 12903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:02 smithi121 bash[36902]: cluster 2022-04-22T21:54:00.347009+0000 mgr.y (mgr.34107) 9143 : cluster [DBG] pgmap v9978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:02 smithi121 bash[36902]: audit 2022-04-22T21:54:01.226645+0000 mon.a (mon.0) 12904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:02 smithi121 bash[36902]: audit 2022-04-22T21:54:01.592374+0000 mon.a (mon.0) 12905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[45495]: cluster 2022-04-22T21:54:00.347009+0000 mgr.y (mgr.34107) 9143 : cluster [DBG] pgmap v9978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:02.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[45495]: audit 2022-04-22T21:54:01.226645+0000 mon.a (mon.0) 12904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:02.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[45495]: audit 2022-04-22T21:54:01.592374+0000 mon.a (mon.0) 12905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:02.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[46934]: cluster 2022-04-22T21:54:00.347009+0000 mgr.y (mgr.34107) 9143 : cluster [DBG] pgmap v9978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:02.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[46934]: audit 2022-04-22T21:54:01.226645+0000 mon.a (mon.0) 12904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:02.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[46934]: audit 2022-04-22T21:54:01.592374+0000 mon.a (mon.0) 12905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:02.671Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:04 smithi121 bash[36902]: cluster 2022-04-22T21:54:02.348134+0000 mgr.y (mgr.34107) 9144 : cluster [DBG] pgmap v9979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:04.636 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:04 smithi115 bash[45495]: cluster 2022-04-22T21:54:02.348134+0000 mgr.y (mgr.34107) 9144 : cluster [DBG] pgmap v9979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:04.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:04 smithi115 bash[46934]: cluster 2022-04-22T21:54:02.348134+0000 mgr.y (mgr.34107) 9144 : cluster [DBG] pgmap v9979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:04.528Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:06.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:06 smithi121 bash[36902]: cluster 2022-04-22T21:54:04.349162+0000 mgr.y (mgr.34107) 9145 : cluster [DBG] pgmap v9980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:06 smithi121 bash[36902]: audit 2022-04-22T21:54:05.161588+0000 mon.a (mon.0) 12906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:06 smithi115 bash[45495]: cluster 2022-04-22T21:54:04.349162+0000 mgr.y (mgr.34107) 9145 : cluster [DBG] pgmap v9980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:06 smithi115 bash[45495]: audit 2022-04-22T21:54:05.161588+0000 mon.a (mon.0) 12906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:06 smithi115 bash[46934]: cluster 2022-04-22T21:54:04.349162+0000 mgr.y (mgr.34107) 9145 : cluster [DBG] pgmap v9980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:06 smithi115 bash[46934]: audit 2022-04-22T21:54:05.161588+0000 mon.a (mon.0) 12906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:07 smithi121 bash[36902]: audit 2022-04-22T21:54:06.230422+0000 mon.a (mon.0) 12907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:07 smithi121 bash[36902]: audit 2022-04-22T21:54:06.595398+0000 mon.a (mon.0) 12908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:07 smithi115 bash[46934]: audit 2022-04-22T21:54:06.230422+0000 mon.a (mon.0) 12907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:07 smithi115 bash[46934]: audit 2022-04-22T21:54:06.595398+0000 mon.a (mon.0) 12908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:07 smithi115 bash[45495]: audit 2022-04-22T21:54:06.230422+0000 mon.a (mon.0) 12907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:07 smithi115 bash[45495]: audit 2022-04-22T21:54:06.595398+0000 mon.a (mon.0) 12908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:08 smithi121 bash[36902]: cluster 2022-04-22T21:54:06.350235+0000 mgr.y (mgr.34107) 9146 : cluster [DBG] pgmap v9981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:08 smithi115 bash[46934]: cluster 2022-04-22T21:54:06.350235+0000 mgr.y (mgr.34107) 9146 : cluster [DBG] pgmap v9981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:08.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:08 smithi115 bash[45495]: cluster 2022-04-22T21:54:06.350235+0000 mgr.y (mgr.34107) 9146 : cluster [DBG] pgmap v9981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:10 smithi121 bash[36902]: cluster 2022-04-22T21:54:08.351374+0000 mgr.y (mgr.34107) 9147 : cluster [DBG] pgmap v9982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:10 smithi115 bash[45495]: cluster 2022-04-22T21:54:08.351374+0000 mgr.y (mgr.34107) 9147 : cluster [DBG] pgmap v9982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:10 smithi115 bash[46934]: cluster 2022-04-22T21:54:08.351374+0000 mgr.y (mgr.34107) 9147 : cluster [DBG] pgmap v9982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:11 smithi121 bash[36902]: audit 2022-04-22T21:54:10.164636+0000 mon.a (mon.0) 12909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:11 smithi115 bash[45495]: audit 2022-04-22T21:54:10.164636+0000 mon.a (mon.0) 12909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:11 smithi115 bash[46934]: audit 2022-04-22T21:54:10.164636+0000 mon.a (mon.0) 12909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:12.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: cluster 2022-04-22T21:54:10.352341+0000 mgr.y (mgr.34107) 9148 : cluster [DBG] pgmap v9983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:12.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: audit 2022-04-22T21:54:11.234357+0000 mon.a (mon.0) 12910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:12.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: audit 2022-04-22T21:54:11.598584+0000 mon.a (mon.0) 12911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:12.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: audit 2022-04-22T21:54:11.846966+0000 mon.a (mon.0) 12912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:54:12.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: audit 2022-04-22T21:54:11.849750+0000 mon.a (mon.0) 12913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[46934]: audit 2022-04-22T21:54:11.851832+0000 mon.a (mon.0) 12914 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: cluster 2022-04-22T21:54:10.352341+0000 mgr.y (mgr.34107) 9148 : cluster [DBG] pgmap v9983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: audit 2022-04-22T21:54:11.234357+0000 mon.a (mon.0) 12910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: audit 2022-04-22T21:54:11.598584+0000 mon.a (mon.0) 12911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: audit 2022-04-22T21:54:11.846966+0000 mon.a (mon.0) 12912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:54:12.200 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: audit 2022-04-22T21:54:11.849750+0000 mon.a (mon.0) 12913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:54:12.201 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[45495]: audit 2022-04-22T21:54:11.851832+0000 mon.a (mon.0) 12914 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:54:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: cluster 2022-04-22T21:54:10.352341+0000 mgr.y (mgr.34107) 9148 : cluster [DBG] pgmap v9983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: audit 2022-04-22T21:54:11.234357+0000 mon.a (mon.0) 12910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: audit 2022-04-22T21:54:11.598584+0000 mon.a (mon.0) 12911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: audit 2022-04-22T21:54:11.846966+0000 mon.a (mon.0) 12912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:54:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: audit 2022-04-22T21:54:11.849750+0000 mon.a (mon.0) 12913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:54:12.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:12 smithi121 bash[36902]: audit 2022-04-22T21:54:11.851832+0000 mon.a (mon.0) 12914 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:54:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:12.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:14 smithi121 bash[36902]: cluster 2022-04-22T21:54:12.353446+0000 mgr.y (mgr.34107) 9149 : cluster [DBG] pgmap v9984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:14.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:14 smithi115 bash[45495]: cluster 2022-04-22T21:54:12.353446+0000 mgr.y (mgr.34107) 9149 : cluster [DBG] pgmap v9984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:14.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:14 smithi115 bash[46934]: cluster 2022-04-22T21:54:12.353446+0000 mgr.y (mgr.34107) 9149 : cluster [DBG] pgmap v9984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:14.529Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:16.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:16 smithi121 bash[36902]: cluster 2022-04-22T21:54:14.354581+0000 mgr.y (mgr.34107) 9150 : cluster [DBG] pgmap v9985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:16.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:16 smithi121 bash[36902]: audit 2022-04-22T21:54:15.167640+0000 mon.a (mon.0) 12915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:16 smithi115 bash[45495]: cluster 2022-04-22T21:54:14.354581+0000 mgr.y (mgr.34107) 9150 : cluster [DBG] pgmap v9985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:16 smithi115 bash[45495]: audit 2022-04-22T21:54:15.167640+0000 mon.a (mon.0) 12915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:16 smithi115 bash[46934]: cluster 2022-04-22T21:54:14.354581+0000 mgr.y (mgr.34107) 9150 : cluster [DBG] pgmap v9985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:16 smithi115 bash[46934]: audit 2022-04-22T21:54:15.167640+0000 mon.a (mon.0) 12915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.238071+0000 mon.a (mon.0) 12916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.328695+0000 mon.a (mon.0) 12917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.362320+0000 mon.a (mon.0) 12918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:54:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.366395+0000 mon.a (mon.0) 12919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:54:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.601626+0000 mon.a (mon.0) 12920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:17 smithi121 bash[36902]: audit 2022-04-22T21:54:16.697786+0000 mon.a (mon.0) 12921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.238071+0000 mon.a (mon.0) 12916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.328695+0000 mon.a (mon.0) 12917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.362320+0000 mon.a (mon.0) 12918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.366395+0000 mon.a (mon.0) 12919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.601626+0000 mon.a (mon.0) 12920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[45495]: audit 2022-04-22T21:54:16.697786+0000 mon.a (mon.0) 12921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.238071+0000 mon.a (mon.0) 12916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.328695+0000 mon.a (mon.0) 12917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.362320+0000 mon.a (mon.0) 12918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.366395+0000 mon.a (mon.0) 12919 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.601626+0000 mon.a (mon.0) 12920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:17 smithi115 bash[46934]: audit 2022-04-22T21:54:16.697786+0000 mon.a (mon.0) 12921 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:54:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:18 smithi121 bash[36902]: cluster 2022-04-22T21:54:16.355450+0000 mgr.y (mgr.34107) 9151 : cluster [DBG] pgmap v9986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:18 smithi115 bash[45495]: cluster 2022-04-22T21:54:16.355450+0000 mgr.y (mgr.34107) 9151 : cluster [DBG] pgmap v9986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:18.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:18 smithi115 bash[46934]: cluster 2022-04-22T21:54:16.355450+0000 mgr.y (mgr.34107) 9151 : cluster [DBG] pgmap v9986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:20.102 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:54:20.432 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:54:20.432 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4s ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:54:20.433 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4s ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4s ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:54:20.434 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:54:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:20 smithi121 bash[36902]: cluster 2022-04-22T21:54:18.356554+0000 mgr.y (mgr.34107) 9152 : cluster [DBG] pgmap v9987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:20 smithi115 bash[45495]: cluster 2022-04-22T21:54:18.356554+0000 mgr.y (mgr.34107) 9152 : cluster [DBG] pgmap v9987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:20 smithi115 bash[46934]: cluster 2022-04-22T21:54:18.356554+0000 mgr.y (mgr.34107) 9152 : cluster [DBG] pgmap v9987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:20.831 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:54:20.831 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:54:20.831 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:20.832 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:54:20.833 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:54:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:21 smithi121 bash[36902]: audit 2022-04-22T21:54:20.083099+0000 mgr.y (mgr.34107) 9153 : audit [DBG] from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:21 smithi121 bash[36902]: audit 2022-04-22T21:54:20.171030+0000 mon.a (mon.0) 12922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:21 smithi121 bash[36902]: audit 2022-04-22T21:54:20.827976+0000 mon.a (mon.0) 12923 : audit [DBG] from='client.? 172.21.15.115:0/3249510289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[45495]: audit 2022-04-22T21:54:20.083099+0000 mgr.y (mgr.34107) 9153 : audit [DBG] from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[45495]: audit 2022-04-22T21:54:20.171030+0000 mon.a (mon.0) 12922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[45495]: audit 2022-04-22T21:54:20.827976+0000 mon.a (mon.0) 12923 : audit [DBG] from='client.? 172.21.15.115:0/3249510289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[46934]: audit 2022-04-22T21:54:20.083099+0000 mgr.y (mgr.34107) 9153 : audit [DBG] from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[46934]: audit 2022-04-22T21:54:20.171030+0000 mon.a (mon.0) 12922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:21 smithi115 bash[46934]: audit 2022-04-22T21:54:20.827976+0000 mon.a (mon.0) 12923 : audit [DBG] from='client.? 172.21.15.115:0/3249510289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:22 smithi121 bash[36902]: cluster 2022-04-22T21:54:20.357581+0000 mgr.y (mgr.34107) 9154 : cluster [DBG] pgmap v9988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:22 smithi121 bash[36902]: audit 2022-04-22T21:54:20.415803+0000 mgr.y (mgr.34107) 9155 : audit [DBG] from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:22 smithi121 bash[36902]: audit 2022-04-22T21:54:21.242032+0000 mon.a (mon.0) 12924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:22 smithi121 bash[36902]: audit 2022-04-22T21:54:21.604432+0000 mon.a (mon.0) 12925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[45495]: cluster 2022-04-22T21:54:20.357581+0000 mgr.y (mgr.34107) 9154 : cluster [DBG] pgmap v9988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[45495]: audit 2022-04-22T21:54:20.415803+0000 mgr.y (mgr.34107) 9155 : audit [DBG] from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[45495]: audit 2022-04-22T21:54:21.242032+0000 mon.a (mon.0) 12924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[45495]: audit 2022-04-22T21:54:21.604432+0000 mon.a (mon.0) 12925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[46934]: cluster 2022-04-22T21:54:20.357581+0000 mgr.y (mgr.34107) 9154 : cluster [DBG] pgmap v9988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[46934]: audit 2022-04-22T21:54:20.415803+0000 mgr.y (mgr.34107) 9155 : audit [DBG] from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[46934]: audit 2022-04-22T21:54:21.242032+0000 mon.a (mon.0) 12924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[46934]: audit 2022-04-22T21:54:21.604432+0000 mon.a (mon.0) 12925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:22.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:24 smithi121 bash[36902]: cluster 2022-04-22T21:54:22.358674+0000 mgr.y (mgr.34107) 9156 : cluster [DBG] pgmap v9989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:24.533 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:24 smithi115 bash[45495]: cluster 2022-04-22T21:54:22.358674+0000 mgr.y (mgr.34107) 9156 : cluster [DBG] pgmap v9989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:24.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:24 smithi115 bash[46934]: cluster 2022-04-22T21:54:22.358674+0000 mgr.y (mgr.34107) 9156 : cluster [DBG] pgmap v9989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:24.530Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:26.430 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:26 smithi121 bash[36902]: cluster 2022-04-22T21:54:24.359896+0000 mgr.y (mgr.34107) 9157 : cluster [DBG] pgmap v9990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:26.431 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:26 smithi121 bash[36902]: audit 2022-04-22T21:54:25.174175+0000 mon.a (mon.0) 12926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:26 smithi115 bash[45495]: cluster 2022-04-22T21:54:24.359896+0000 mgr.y (mgr.34107) 9157 : cluster [DBG] pgmap v9990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:26.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:26 smithi115 bash[45495]: audit 2022-04-22T21:54:25.174175+0000 mon.a (mon.0) 12926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:26 smithi115 bash[46934]: cluster 2022-04-22T21:54:24.359896+0000 mgr.y (mgr.34107) 9157 : cluster [DBG] pgmap v9990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:26.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:26 smithi115 bash[46934]: audit 2022-04-22T21:54:25.174175+0000 mon.a (mon.0) 12926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:27 smithi121 bash[36902]: audit 2022-04-22T21:54:26.245845+0000 mon.a (mon.0) 12927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:27.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:27 smithi121 bash[36902]: audit 2022-04-22T21:54:26.607543+0000 mon.a (mon.0) 12928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:27.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:27 smithi115 bash[45495]: audit 2022-04-22T21:54:26.245845+0000 mon.a (mon.0) 12927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:27.554 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:27 smithi115 bash[45495]: audit 2022-04-22T21:54:26.607543+0000 mon.a (mon.0) 12928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:27.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:27 smithi115 bash[46934]: audit 2022-04-22T21:54:26.245845+0000 mon.a (mon.0) 12927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:27.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:27 smithi115 bash[46934]: audit 2022-04-22T21:54:26.607543+0000 mon.a (mon.0) 12928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:28 smithi121 bash[36902]: cluster 2022-04-22T21:54:26.361087+0000 mgr.y (mgr.34107) 9158 : cluster [DBG] pgmap v9991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:28 smithi115 bash[45495]: cluster 2022-04-22T21:54:26.361087+0000 mgr.y (mgr.34107) 9158 : cluster [DBG] pgmap v9991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:28.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:28 smithi115 bash[46934]: cluster 2022-04-22T21:54:26.361087+0000 mgr.y (mgr.34107) 9158 : cluster [DBG] pgmap v9991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:30 smithi121 bash[36902]: cluster 2022-04-22T21:54:28.362275+0000 mgr.y (mgr.34107) 9159 : cluster [DBG] pgmap v9992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:30 smithi115 bash[45495]: cluster 2022-04-22T21:54:28.362275+0000 mgr.y (mgr.34107) 9159 : cluster [DBG] pgmap v9992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:30 smithi115 bash[46934]: cluster 2022-04-22T21:54:28.362275+0000 mgr.y (mgr.34107) 9159 : cluster [DBG] pgmap v9992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:31 smithi121 bash[36902]: audit 2022-04-22T21:54:30.177339+0000 mon.a (mon.0) 12929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:31 smithi115 bash[45495]: audit 2022-04-22T21:54:30.177339+0000 mon.a (mon.0) 12929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:31 smithi115 bash[46934]: audit 2022-04-22T21:54:30.177339+0000 mon.a (mon.0) 12929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:32 smithi121 bash[36902]: cluster 2022-04-22T21:54:30.363357+0000 mgr.y (mgr.34107) 9160 : cluster [DBG] pgmap v9993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:32 smithi121 bash[36902]: audit 2022-04-22T21:54:31.249856+0000 mon.a (mon.0) 12930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:32 smithi121 bash[36902]: audit 2022-04-22T21:54:31.610597+0000 mon.a (mon.0) 12931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:32.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[46934]: cluster 2022-04-22T21:54:30.363357+0000 mgr.y (mgr.34107) 9160 : cluster [DBG] pgmap v9993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:32.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[46934]: audit 2022-04-22T21:54:31.249856+0000 mon.a (mon.0) 12930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:32.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[46934]: audit 2022-04-22T21:54:31.610597+0000 mon.a (mon.0) 12931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[45495]: cluster 2022-04-22T21:54:30.363357+0000 mgr.y (mgr.34107) 9160 : cluster [DBG] pgmap v9993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[45495]: audit 2022-04-22T21:54:31.249856+0000 mon.a (mon.0) 12930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[45495]: audit 2022-04-22T21:54:31.610597+0000 mon.a (mon.0) 12931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:32.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:34 smithi121 bash[36902]: cluster 2022-04-22T21:54:32.364398+0000 mgr.y (mgr.34107) 9161 : cluster [DBG] pgmap v9994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:34.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:34 smithi115 bash[46934]: cluster 2022-04-22T21:54:32.364398+0000 mgr.y (mgr.34107) 9161 : cluster [DBG] pgmap v9994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:34.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:34 smithi115 bash[45495]: cluster 2022-04-22T21:54:32.364398+0000 mgr.y (mgr.34107) 9161 : cluster [DBG] pgmap v9994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:34.531Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:36 smithi121 bash[36902]: cluster 2022-04-22T21:54:34.365521+0000 mgr.y (mgr.34107) 9162 : cluster [DBG] pgmap v9995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:36 smithi121 bash[36902]: audit 2022-04-22T21:54:35.180347+0000 mon.a (mon.0) 12932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:36 smithi115 bash[46934]: cluster 2022-04-22T21:54:34.365521+0000 mgr.y (mgr.34107) 9162 : cluster [DBG] pgmap v9995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:36 smithi115 bash[46934]: audit 2022-04-22T21:54:35.180347+0000 mon.a (mon.0) 12932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:36 smithi115 bash[45495]: cluster 2022-04-22T21:54:34.365521+0000 mgr.y (mgr.34107) 9162 : cluster [DBG] pgmap v9995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:36 smithi115 bash[45495]: audit 2022-04-22T21:54:35.180347+0000 mon.a (mon.0) 12932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:37 smithi121 bash[36902]: audit 2022-04-22T21:54:36.253355+0000 mon.a (mon.0) 12933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:37 smithi121 bash[36902]: audit 2022-04-22T21:54:36.613762+0000 mon.a (mon.0) 12934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:37 smithi115 bash[45495]: audit 2022-04-22T21:54:36.253355+0000 mon.a (mon.0) 12933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:37 smithi115 bash[45495]: audit 2022-04-22T21:54:36.613762+0000 mon.a (mon.0) 12934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:37 smithi115 bash[46934]: audit 2022-04-22T21:54:36.253355+0000 mon.a (mon.0) 12933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:37 smithi115 bash[46934]: audit 2022-04-22T21:54:36.613762+0000 mon.a (mon.0) 12934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:38 smithi121 bash[36902]: cluster 2022-04-22T21:54:36.366518+0000 mgr.y (mgr.34107) 9163 : cluster [DBG] pgmap v9996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:38.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:38 smithi115 bash[45495]: cluster 2022-04-22T21:54:36.366518+0000 mgr.y (mgr.34107) 9163 : cluster [DBG] pgmap v9996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:38 smithi115 bash[46934]: cluster 2022-04-22T21:54:36.366518+0000 mgr.y (mgr.34107) 9163 : cluster [DBG] pgmap v9996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:40.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:40 smithi121 bash[36902]: cluster 2022-04-22T21:54:38.367589+0000 mgr.y (mgr.34107) 9164 : cluster [DBG] pgmap v9997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:40 smithi115 bash[45495]: cluster 2022-04-22T21:54:38.367589+0000 mgr.y (mgr.34107) 9164 : cluster [DBG] pgmap v9997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:40 smithi115 bash[46934]: cluster 2022-04-22T21:54:38.367589+0000 mgr.y (mgr.34107) 9164 : cluster [DBG] pgmap v9997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:41 smithi121 bash[36902]: audit 2022-04-22T21:54:40.183342+0000 mon.a (mon.0) 12935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:41 smithi115 bash[45495]: audit 2022-04-22T21:54:40.183342+0000 mon.a (mon.0) 12935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:41 smithi115 bash[46934]: audit 2022-04-22T21:54:40.183342+0000 mon.a (mon.0) 12935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:42 smithi121 bash[36902]: cluster 2022-04-22T21:54:40.368470+0000 mgr.y (mgr.34107) 9165 : cluster [DBG] pgmap v9998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:42 smithi121 bash[36902]: audit 2022-04-22T21:54:41.257177+0000 mon.a (mon.0) 12936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:42 smithi121 bash[36902]: audit 2022-04-22T21:54:41.616655+0000 mon.a (mon.0) 12937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[45495]: cluster 2022-04-22T21:54:40.368470+0000 mgr.y (mgr.34107) 9165 : cluster [DBG] pgmap v9998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[45495]: audit 2022-04-22T21:54:41.257177+0000 mon.a (mon.0) 12936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[45495]: audit 2022-04-22T21:54:41.616655+0000 mon.a (mon.0) 12937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[46934]: cluster 2022-04-22T21:54:40.368470+0000 mgr.y (mgr.34107) 9165 : cluster [DBG] pgmap v9998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[46934]: audit 2022-04-22T21:54:41.257177+0000 mon.a (mon.0) 12936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[46934]: audit 2022-04-22T21:54:41.616655+0000 mon.a (mon.0) 12937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:42.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:44 smithi121 bash[36902]: cluster 2022-04-22T21:54:42.369546+0000 mgr.y (mgr.34107) 9166 : cluster [DBG] pgmap v9999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:44.532 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:44 smithi115 bash[45495]: cluster 2022-04-22T21:54:42.369546+0000 mgr.y (mgr.34107) 9166 : cluster [DBG] pgmap v9999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:44.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:44 smithi115 bash[46934]: cluster 2022-04-22T21:54:42.369546+0000 mgr.y (mgr.34107) 9166 : cluster [DBG] pgmap v9999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:44.529Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:46.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:46 smithi121 bash[36902]: cluster 2022-04-22T21:54:44.370754+0000 mgr.y (mgr.34107) 9167 : cluster [DBG] pgmap v10000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:46 smithi121 bash[36902]: audit 2022-04-22T21:54:45.188230+0000 mon.a (mon.0) 12938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:46 smithi115 bash[45495]: cluster 2022-04-22T21:54:44.370754+0000 mgr.y (mgr.34107) 9167 : cluster [DBG] pgmap v10000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:46 smithi115 bash[45495]: audit 2022-04-22T21:54:45.188230+0000 mon.a (mon.0) 12938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:46 smithi115 bash[46934]: cluster 2022-04-22T21:54:44.370754+0000 mgr.y (mgr.34107) 9167 : cluster [DBG] pgmap v10000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:46 smithi115 bash[46934]: audit 2022-04-22T21:54:45.188230+0000 mon.a (mon.0) 12938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:47 smithi121 bash[36902]: audit 2022-04-22T21:54:46.261094+0000 mon.a (mon.0) 12939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:47 smithi121 bash[36902]: audit 2022-04-22T21:54:46.619499+0000 mon.a (mon.0) 12940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:47 smithi115 bash[45495]: audit 2022-04-22T21:54:46.261094+0000 mon.a (mon.0) 12939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:47.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:47 smithi115 bash[45495]: audit 2022-04-22T21:54:46.619499+0000 mon.a (mon.0) 12940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:47 smithi115 bash[46934]: audit 2022-04-22T21:54:46.261094+0000 mon.a (mon.0) 12939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:47.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:47 smithi115 bash[46934]: audit 2022-04-22T21:54:46.619499+0000 mon.a (mon.0) 12940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:48 smithi121 bash[36902]: cluster 2022-04-22T21:54:46.371910+0000 mgr.y (mgr.34107) 9168 : cluster [DBG] pgmap v10001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:48 smithi115 bash[45495]: cluster 2022-04-22T21:54:46.371910+0000 mgr.y (mgr.34107) 9168 : cluster [DBG] pgmap v10001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:48 smithi115 bash[46934]: cluster 2022-04-22T21:54:46.371910+0000 mgr.y (mgr.34107) 9168 : cluster [DBG] pgmap v10001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:50 smithi121 bash[36902]: cluster 2022-04-22T21:54:48.373127+0000 mgr.y (mgr.34107) 9169 : cluster [DBG] pgmap v10002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:50 smithi121 bash[36902]: audit 2022-04-22T21:54:50.190348+0000 mon.a (mon.0) 12941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:50 smithi115 bash[45495]: cluster 2022-04-22T21:54:48.373127+0000 mgr.y (mgr.34107) 9169 : cluster [DBG] pgmap v10002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:50 smithi115 bash[45495]: audit 2022-04-22T21:54:50.190348+0000 mon.a (mon.0) 12941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:50 smithi115 bash[46934]: cluster 2022-04-22T21:54:48.373127+0000 mgr.y (mgr.34107) 9169 : cluster [DBG] pgmap v10002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:50 smithi115 bash[46934]: audit 2022-04-22T21:54:50.190348+0000 mon.a (mon.0) 12941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:51.208 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:54:51.548 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:54:51.558 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 16.0M - ba2b418f427c 9255ba08f72f 2022-04-22T21:54:51.558 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 35s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:54:51.558 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:54:51.559 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 35s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:54:51.559 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 35s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:54:51.559 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 490M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:54:51.559 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 397M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:54:51.560 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 35s ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:54:51.560 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 264M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:54:51.560 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 11.3M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:54:51.560 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 35s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:54:51.560 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 713M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 793M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 537M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 497M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 35s ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 35s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 35s ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 35s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:54:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 35s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:54:51.956 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:54:51.956 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:51.957 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:54:51.958 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:54:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:52 smithi121 bash[36902]: cluster 2022-04-22T21:54:50.374229+0000 mgr.y (mgr.34107) 9170 : cluster [DBG] pgmap v10003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:52 smithi121 bash[36902]: audit 2022-04-22T21:54:51.189166+0000 mgr.y (mgr.34107) 9171 : audit [DBG] from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:52 smithi121 bash[36902]: audit 2022-04-22T21:54:51.264529+0000 mon.a (mon.0) 12942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:52 smithi121 bash[36902]: audit 2022-04-22T21:54:51.622258+0000 mon.a (mon.0) 12943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:52 smithi121 bash[36902]: audit 2022-04-22T21:54:51.953082+0000 mon.c (mon.1) 318 : audit [DBG] from='client.? 172.21.15.115:0/3742534259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[45495]: cluster 2022-04-22T21:54:50.374229+0000 mgr.y (mgr.34107) 9170 : cluster [DBG] pgmap v10003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[45495]: audit 2022-04-22T21:54:51.189166+0000 mgr.y (mgr.34107) 9171 : audit [DBG] from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[45495]: audit 2022-04-22T21:54:51.264529+0000 mon.a (mon.0) 12942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[45495]: audit 2022-04-22T21:54:51.622258+0000 mon.a (mon.0) 12943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[45495]: audit 2022-04-22T21:54:51.953082+0000 mon.c (mon.1) 318 : audit [DBG] from='client.? 172.21.15.115:0/3742534259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[46934]: cluster 2022-04-22T21:54:50.374229+0000 mgr.y (mgr.34107) 9170 : cluster [DBG] pgmap v10003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[46934]: audit 2022-04-22T21:54:51.189166+0000 mgr.y (mgr.34107) 9171 : audit [DBG] from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[46934]: audit 2022-04-22T21:54:51.264529+0000 mon.a (mon.0) 12942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[46934]: audit 2022-04-22T21:54:51.622258+0000 mon.a (mon.0) 12943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:52.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[46934]: audit 2022-04-22T21:54:51.953082+0000 mon.c (mon.1) 318 : audit [DBG] from='client.? 172.21.15.115:0/3742534259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:54:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:52.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:53 smithi121 bash[36902]: audit 2022-04-22T21:54:51.530461+0000 mgr.y (mgr.34107) 9172 : audit [DBG] from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:53.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:53 smithi115 bash[45495]: audit 2022-04-22T21:54:51.530461+0000 mgr.y (mgr.34107) 9172 : audit [DBG] from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:53 smithi115 bash[46934]: audit 2022-04-22T21:54:51.530461+0000 mgr.y (mgr.34107) 9172 : audit [DBG] from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:54:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:54 smithi121 bash[36902]: cluster 2022-04-22T21:54:52.375370+0000 mgr.y (mgr.34107) 9173 : cluster [DBG] pgmap v10004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:54.535 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:54 smithi115 bash[45495]: cluster 2022-04-22T21:54:52.375370+0000 mgr.y (mgr.34107) 9173 : cluster [DBG] pgmap v10004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:54.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:54 smithi115 bash[46934]: cluster 2022-04-22T21:54:52.375370+0000 mgr.y (mgr.34107) 9173 : cluster [DBG] pgmap v10004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:54:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:54:54.531Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:54:55.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:55 smithi121 bash[36902]: audit 2022-04-22T21:54:55.193606+0000 mon.a (mon.0) 12944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:55 smithi115 bash[45495]: audit 2022-04-22T21:54:55.193606+0000 mon.a (mon.0) 12944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:55 smithi115 bash[46934]: audit 2022-04-22T21:54:55.193606+0000 mon.a (mon.0) 12944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:54:56.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:56 smithi121 bash[36902]: cluster 2022-04-22T21:54:54.376446+0000 mgr.y (mgr.34107) 9174 : cluster [DBG] pgmap v10005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:56 smithi115 bash[45495]: cluster 2022-04-22T21:54:54.376446+0000 mgr.y (mgr.34107) 9174 : cluster [DBG] pgmap v10005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:56 smithi115 bash[46934]: cluster 2022-04-22T21:54:54.376446+0000 mgr.y (mgr.34107) 9174 : cluster [DBG] pgmap v10005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:57 smithi121 bash[36902]: audit 2022-04-22T21:54:56.268019+0000 mon.a (mon.0) 12945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:57 smithi121 bash[36902]: audit 2022-04-22T21:54:56.624769+0000 mon.a (mon.0) 12946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:57 smithi115 bash[45495]: audit 2022-04-22T21:54:56.268019+0000 mon.a (mon.0) 12945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:57 smithi115 bash[45495]: audit 2022-04-22T21:54:56.624769+0000 mon.a (mon.0) 12946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:57 smithi115 bash[46934]: audit 2022-04-22T21:54:56.268019+0000 mon.a (mon.0) 12945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:54:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:57 smithi115 bash[46934]: audit 2022-04-22T21:54:56.624769+0000 mon.a (mon.0) 12946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:54:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:54:58 smithi121 bash[36902]: cluster 2022-04-22T21:54:56.377548+0000 mgr.y (mgr.34107) 9175 : cluster [DBG] pgmap v10006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:58.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:54:58 smithi115 bash[46934]: cluster 2022-04-22T21:54:56.377548+0000 mgr.y (mgr.34107) 9175 : cluster [DBG] pgmap v10006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:54:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:54:58 smithi115 bash[45495]: cluster 2022-04-22T21:54:56.377548+0000 mgr.y (mgr.34107) 9175 : cluster [DBG] pgmap v10006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:00 smithi121 bash[36902]: cluster 2022-04-22T21:54:58.378734+0000 mgr.y (mgr.34107) 9176 : cluster [DBG] pgmap v10007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:00 smithi121 bash[36902]: audit 2022-04-22T21:55:00.196769+0000 mon.a (mon.0) 12947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:00 smithi115 bash[45495]: cluster 2022-04-22T21:54:58.378734+0000 mgr.y (mgr.34107) 9176 : cluster [DBG] pgmap v10007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:00 smithi115 bash[45495]: audit 2022-04-22T21:55:00.196769+0000 mon.a (mon.0) 12947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:00 smithi115 bash[46934]: cluster 2022-04-22T21:54:58.378734+0000 mgr.y (mgr.34107) 9176 : cluster [DBG] pgmap v10007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:00 smithi115 bash[46934]: audit 2022-04-22T21:55:00.196769+0000 mon.a (mon.0) 12947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:02 smithi121 bash[36902]: cluster 2022-04-22T21:55:00.379818+0000 mgr.y (mgr.34107) 9177 : cluster [DBG] pgmap v10008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:02 smithi121 bash[36902]: audit 2022-04-22T21:55:01.271653+0000 mon.a (mon.0) 12948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:02 smithi121 bash[36902]: audit 2022-04-22T21:55:01.627672+0000 mon.a (mon.0) 12949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[45495]: cluster 2022-04-22T21:55:00.379818+0000 mgr.y (mgr.34107) 9177 : cluster [DBG] pgmap v10008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[45495]: audit 2022-04-22T21:55:01.271653+0000 mon.a (mon.0) 12948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[45495]: audit 2022-04-22T21:55:01.627672+0000 mon.a (mon.0) 12949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[46934]: cluster 2022-04-22T21:55:00.379818+0000 mgr.y (mgr.34107) 9177 : cluster [DBG] pgmap v10008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[46934]: audit 2022-04-22T21:55:01.271653+0000 mon.a (mon.0) 12948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[46934]: audit 2022-04-22T21:55:01.627672+0000 mon.a (mon.0) 12949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:02.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:03 smithi115 bash[45495]: cluster 2022-04-22T21:55:02.380945+0000 mgr.y (mgr.34107) 9178 : cluster [DBG] pgmap v10009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:03 smithi115 bash[46934]: cluster 2022-04-22T21:55:02.380945+0000 mgr.y (mgr.34107) 9178 : cluster [DBG] pgmap v10009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:03 smithi121 bash[36902]: cluster 2022-04-22T21:55:02.380945+0000 mgr.y (mgr.34107) 9178 : cluster [DBG] pgmap v10009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:04.532Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:05 smithi121 bash[36902]: cluster 2022-04-22T21:55:04.382008+0000 mgr.y (mgr.34107) 9179 : cluster [DBG] pgmap v10010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:05 smithi121 bash[36902]: audit 2022-04-22T21:55:05.199674+0000 mon.a (mon.0) 12950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:05.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:05 smithi115 bash[46934]: cluster 2022-04-22T21:55:04.382008+0000 mgr.y (mgr.34107) 9179 : cluster [DBG] pgmap v10010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:05.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:05 smithi115 bash[46934]: audit 2022-04-22T21:55:05.199674+0000 mon.a (mon.0) 12950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:05 smithi115 bash[45495]: cluster 2022-04-22T21:55:04.382008+0000 mgr.y (mgr.34107) 9179 : cluster [DBG] pgmap v10010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:05 smithi115 bash[45495]: audit 2022-04-22T21:55:05.199674+0000 mon.a (mon.0) 12950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:07.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:06 smithi121 bash[36902]: audit 2022-04-22T21:55:06.275317+0000 mon.a (mon.0) 12951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:06 smithi121 bash[36902]: audit 2022-04-22T21:55:06.630479+0000 mon.a (mon.0) 12952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:07.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:06 smithi115 bash[46934]: audit 2022-04-22T21:55:06.275317+0000 mon.a (mon.0) 12951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:07.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:06 smithi115 bash[46934]: audit 2022-04-22T21:55:06.630479+0000 mon.a (mon.0) 12952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:06 smithi115 bash[45495]: audit 2022-04-22T21:55:06.275317+0000 mon.a (mon.0) 12951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:06 smithi115 bash[45495]: audit 2022-04-22T21:55:06.630479+0000 mon.a (mon.0) 12952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:07 smithi121 bash[36902]: cluster 2022-04-22T21:55:06.383108+0000 mgr.y (mgr.34107) 9180 : cluster [DBG] pgmap v10011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:07 smithi115 bash[45495]: cluster 2022-04-22T21:55:06.383108+0000 mgr.y (mgr.34107) 9180 : cluster [DBG] pgmap v10011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:07 smithi115 bash[46934]: cluster 2022-04-22T21:55:06.383108+0000 mgr.y (mgr.34107) 9180 : cluster [DBG] pgmap v10011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:10.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:09 smithi121 bash[36902]: cluster 2022-04-22T21:55:08.384212+0000 mgr.y (mgr.34107) 9181 : cluster [DBG] pgmap v10012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:10.317 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:09 smithi115 bash[45495]: cluster 2022-04-22T21:55:08.384212+0000 mgr.y (mgr.34107) 9181 : cluster [DBG] pgmap v10012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:10.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:09 smithi115 bash[46934]: cluster 2022-04-22T21:55:08.384212+0000 mgr.y (mgr.34107) 9181 : cluster [DBG] pgmap v10012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:10 smithi121 bash[36902]: audit 2022-04-22T21:55:10.202468+0000 mon.a (mon.0) 12953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:10 smithi115 bash[45495]: audit 2022-04-22T21:55:10.202468+0000 mon.a (mon.0) 12953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:10 smithi115 bash[46934]: audit 2022-04-22T21:55:10.202468+0000 mon.a (mon.0) 12953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:11 smithi121 bash[36902]: cluster 2022-04-22T21:55:10.385349+0000 mgr.y (mgr.34107) 9182 : cluster [DBG] pgmap v10013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:11 smithi121 bash[36902]: audit 2022-04-22T21:55:11.278788+0000 mon.a (mon.0) 12954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:11 smithi121 bash[36902]: audit 2022-04-22T21:55:11.633478+0000 mon.a (mon.0) 12955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[45495]: cluster 2022-04-22T21:55:10.385349+0000 mgr.y (mgr.34107) 9182 : cluster [DBG] pgmap v10013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:12.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[45495]: audit 2022-04-22T21:55:11.278788+0000 mon.a (mon.0) 12954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:12.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[45495]: audit 2022-04-22T21:55:11.633478+0000 mon.a (mon.0) 12955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:12.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[46934]: cluster 2022-04-22T21:55:10.385349+0000 mgr.y (mgr.34107) 9182 : cluster [DBG] pgmap v10013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:12.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[46934]: audit 2022-04-22T21:55:11.278788+0000 mon.a (mon.0) 12954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:12.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:11 smithi115 bash[46934]: audit 2022-04-22T21:55:11.633478+0000 mon.a (mon.0) 12955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:12.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:14.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:13 smithi121 bash[36902]: cluster 2022-04-22T21:55:12.386438+0000 mgr.y (mgr.34107) 9183 : cluster [DBG] pgmap v10014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:13 smithi115 bash[45495]: cluster 2022-04-22T21:55:12.386438+0000 mgr.y (mgr.34107) 9183 : cluster [DBG] pgmap v10014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:13 smithi115 bash[46934]: cluster 2022-04-22T21:55:12.386438+0000 mgr.y (mgr.34107) 9183 : cluster [DBG] pgmap v10014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:14.532Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:15 smithi121 bash[36902]: cluster 2022-04-22T21:55:14.387541+0000 mgr.y (mgr.34107) 9184 : cluster [DBG] pgmap v10015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:15 smithi121 bash[36902]: audit 2022-04-22T21:55:15.205461+0000 mon.a (mon.0) 12956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:15 smithi115 bash[45495]: cluster 2022-04-22T21:55:14.387541+0000 mgr.y (mgr.34107) 9184 : cluster [DBG] pgmap v10015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:15 smithi115 bash[45495]: audit 2022-04-22T21:55:15.205461+0000 mon.a (mon.0) 12956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:15 smithi115 bash[46934]: cluster 2022-04-22T21:55:14.387541+0000 mgr.y (mgr.34107) 9184 : cluster [DBG] pgmap v10015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:15 smithi115 bash[46934]: audit 2022-04-22T21:55:15.205461+0000 mon.a (mon.0) 12956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.282732+0000 mon.a (mon.0) 12957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.377495+0000 mon.a (mon.0) 12958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.378255+0000 mon.a (mon.0) 12959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.636526+0000 mon.a (mon.0) 12960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.707523+0000 mon.a (mon.0) 12961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.710780+0000 mon.a (mon.0) 12962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:55:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:16 smithi121 bash[36902]: audit 2022-04-22T21:55:16.712766+0000 mon.a (mon.0) 12963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.282732+0000 mon.a (mon.0) 12957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.377495+0000 mon.a (mon.0) 12958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.378255+0000 mon.a (mon.0) 12959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.636526+0000 mon.a (mon.0) 12960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.707523+0000 mon.a (mon.0) 12961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.710780+0000 mon.a (mon.0) 12962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:55:17.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[46934]: audit 2022-04-22T21:55:16.712766+0000 mon.a (mon.0) 12963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.282732+0000 mon.a (mon.0) 12957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.377495+0000 mon.a (mon.0) 12958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.378255+0000 mon.a (mon.0) 12959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.636526+0000 mon.a (mon.0) 12960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.707523+0000 mon.a (mon.0) 12961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.710780+0000 mon.a (mon.0) 12962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:55:17.025 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:16 smithi115 bash[45495]: audit 2022-04-22T21:55:16.712766+0000 mon.a (mon.0) 12963 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:55:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[45495]: cluster 2022-04-22T21:55:16.388329+0000 mgr.y (mgr.34107) 9185 : cluster [DBG] pgmap v10016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[45495]: audit 2022-04-22T21:55:17.036635+0000 mon.a (mon.0) 12964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[45495]: audit 2022-04-22T21:55:17.042556+0000 mon.a (mon.0) 12965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[46934]: cluster 2022-04-22T21:55:16.388329+0000 mgr.y (mgr.34107) 9185 : cluster [DBG] pgmap v10016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[46934]: audit 2022-04-22T21:55:17.036635+0000 mon.a (mon.0) 12964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:18 smithi115 bash[46934]: audit 2022-04-22T21:55:17.042556+0000 mon.a (mon.0) 12965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:18 smithi121 bash[36902]: cluster 2022-04-22T21:55:16.388329+0000 mgr.y (mgr.34107) 9185 : cluster [DBG] pgmap v10016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:18 smithi121 bash[36902]: audit 2022-04-22T21:55:17.036635+0000 mon.a (mon.0) 12964 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:18 smithi121 bash[36902]: audit 2022-04-22T21:55:17.042556+0000 mon.a (mon.0) 12965 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:20 smithi121 bash[36902]: cluster 2022-04-22T21:55:18.389443+0000 mgr.y (mgr.34107) 9186 : cluster [DBG] pgmap v10017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:20 smithi115 bash[45495]: cluster 2022-04-22T21:55:18.389443+0000 mgr.y (mgr.34107) 9186 : cluster [DBG] pgmap v10017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:20 smithi115 bash[46934]: cluster 2022-04-22T21:55:18.389443+0000 mgr.y (mgr.34107) 9186 : cluster [DBG] pgmap v10017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:21 smithi121 bash[36902]: audit 2022-04-22T21:55:20.208417+0000 mon.a (mon.0) 12966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:21.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:21 smithi115 bash[46934]: audit 2022-04-22T21:55:20.208417+0000 mon.a (mon.0) 12966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:21 smithi115 bash[45495]: audit 2022-04-22T21:55:20.208417+0000 mon.a (mon.0) 12966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:22.331 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:55:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:22 smithi121 bash[36902]: cluster 2022-04-22T21:55:20.390310+0000 mgr.y (mgr.34107) 9187 : cluster [DBG] pgmap v10018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:22 smithi121 bash[36902]: audit 2022-04-22T21:55:21.213799+0000 mon.a (mon.0) 12967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:22 smithi121 bash[36902]: audit 2022-04-22T21:55:21.286403+0000 mon.a (mon.0) 12968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:22 smithi121 bash[36902]: audit 2022-04-22T21:55:21.562466+0000 mon.a (mon.0) 12969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:22 smithi121 bash[36902]: audit 2022-04-22T21:55:21.639509+0000 mon.a (mon.0) 12970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:22.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[46934]: cluster 2022-04-22T21:55:20.390310+0000 mgr.y (mgr.34107) 9187 : cluster [DBG] pgmap v10018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[46934]: audit 2022-04-22T21:55:21.213799+0000 mon.a (mon.0) 12967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[46934]: audit 2022-04-22T21:55:21.286403+0000 mon.a (mon.0) 12968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[46934]: audit 2022-04-22T21:55:21.562466+0000 mon.a (mon.0) 12969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[46934]: audit 2022-04-22T21:55:21.639509+0000 mon.a (mon.0) 12970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[45495]: cluster 2022-04-22T21:55:20.390310+0000 mgr.y (mgr.34107) 9187 : cluster [DBG] pgmap v10018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[45495]: audit 2022-04-22T21:55:21.213799+0000 mon.a (mon.0) 12967 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[45495]: audit 2022-04-22T21:55:21.286403+0000 mon.a (mon.0) 12968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[45495]: audit 2022-04-22T21:55:21.562466+0000 mon.a (mon.0) 12969 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:55:22.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[45495]: audit 2022-04-22T21:55:21.639509+0000 mon.a (mon.0) 12970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:22.678 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:55:22.678 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 1s ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 66s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 1s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 66s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 66s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 1s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 1s ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 66s ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 1s ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:55:22.679 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 1s ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 66s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 1s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 1s ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 1s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 1s ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 66s ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 66s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 66s ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 66s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:55:22.680 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 66s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:55:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:22.682Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:23.091 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:55:23.091 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:55:23.091 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:55:23.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:23.092 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:55:23.092 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:55:23.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:23.092 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:55:23.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:55:23.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:23.093 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:55:23.093 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:55:23.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:23.094 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:55:23.094 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:55:23.094 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:55:23.094 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:55:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:23 smithi121 bash[36902]: audit 2022-04-22T21:55:23.087399+0000 mon.c (mon.1) 319 : audit [DBG] from='client.? 172.21.15.115:0/1348062791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:23 smithi115 bash[45495]: audit 2022-04-22T21:55:23.087399+0000 mon.c (mon.1) 319 : audit [DBG] from='client.? 172.21.15.115:0/1348062791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:23 smithi115 bash[46934]: audit 2022-04-22T21:55:23.087399+0000 mon.c (mon.1) 319 : audit [DBG] from='client.? 172.21.15.115:0/1348062791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:24 smithi121 bash[36902]: audit 2022-04-22T21:55:22.313007+0000 mgr.y (mgr.34107) 9188 : audit [DBG] from='client.44892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:24 smithi121 bash[36902]: cluster 2022-04-22T21:55:22.391366+0000 mgr.y (mgr.34107) 9189 : cluster [DBG] pgmap v10019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:24 smithi121 bash[36902]: audit 2022-04-22T21:55:22.654230+0000 mgr.y (mgr.34107) 9190 : audit [DBG] from='client.44898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[45495]: audit 2022-04-22T21:55:22.313007+0000 mgr.y (mgr.34107) 9188 : audit [DBG] from='client.44892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[45495]: cluster 2022-04-22T21:55:22.391366+0000 mgr.y (mgr.34107) 9189 : cluster [DBG] pgmap v10019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:24.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[45495]: audit 2022-04-22T21:55:22.654230+0000 mgr.y (mgr.34107) 9190 : audit [DBG] from='client.44898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[46934]: audit 2022-04-22T21:55:22.313007+0000 mgr.y (mgr.34107) 9188 : audit [DBG] from='client.44892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[46934]: cluster 2022-04-22T21:55:22.391366+0000 mgr.y (mgr.34107) 9189 : cluster [DBG] pgmap v10019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[46934]: audit 2022-04-22T21:55:22.654230+0000 mgr.y (mgr.34107) 9190 : audit [DBG] from='client.44898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:24.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:25.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:25 smithi121 bash[36902]: audit 2022-04-22T21:55:25.211749+0000 mon.a (mon.0) 12971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:25 smithi115 bash[45495]: audit 2022-04-22T21:55:25.211749+0000 mon.a (mon.0) 12971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:25.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:25 smithi115 bash[46934]: audit 2022-04-22T21:55:25.211749+0000 mon.a (mon.0) 12971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:26 smithi121 bash[36902]: cluster 2022-04-22T21:55:24.392434+0000 mgr.y (mgr.34107) 9191 : cluster [DBG] pgmap v10020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:26 smithi115 bash[45495]: cluster 2022-04-22T21:55:24.392434+0000 mgr.y (mgr.34107) 9191 : cluster [DBG] pgmap v10020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:26 smithi115 bash[46934]: cluster 2022-04-22T21:55:24.392434+0000 mgr.y (mgr.34107) 9191 : cluster [DBG] pgmap v10020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:27 smithi121 bash[36902]: audit 2022-04-22T21:55:26.290466+0000 mon.a (mon.0) 12972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:27 smithi121 bash[36902]: audit 2022-04-22T21:55:26.642484+0000 mon.a (mon.0) 12973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:27 smithi115 bash[45495]: audit 2022-04-22T21:55:26.290466+0000 mon.a (mon.0) 12972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:27 smithi115 bash[45495]: audit 2022-04-22T21:55:26.642484+0000 mon.a (mon.0) 12973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:27 smithi115 bash[46934]: audit 2022-04-22T21:55:26.290466+0000 mon.a (mon.0) 12972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:27 smithi115 bash[46934]: audit 2022-04-22T21:55:26.642484+0000 mon.a (mon.0) 12973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:28 smithi121 bash[36902]: cluster 2022-04-22T21:55:26.393513+0000 mgr.y (mgr.34107) 9192 : cluster [DBG] pgmap v10021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:28 smithi115 bash[45495]: cluster 2022-04-22T21:55:26.393513+0000 mgr.y (mgr.34107) 9192 : cluster [DBG] pgmap v10021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:28 smithi115 bash[46934]: cluster 2022-04-22T21:55:26.393513+0000 mgr.y (mgr.34107) 9192 : cluster [DBG] pgmap v10021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:30 smithi121 bash[36902]: cluster 2022-04-22T21:55:28.394755+0000 mgr.y (mgr.34107) 9193 : cluster [DBG] pgmap v10022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:30 smithi121 bash[36902]: audit 2022-04-22T21:55:30.214824+0000 mon.a (mon.0) 12974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:30 smithi115 bash[45495]: cluster 2022-04-22T21:55:28.394755+0000 mgr.y (mgr.34107) 9193 : cluster [DBG] pgmap v10022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:30 smithi115 bash[45495]: audit 2022-04-22T21:55:30.214824+0000 mon.a (mon.0) 12974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:30 smithi115 bash[46934]: cluster 2022-04-22T21:55:28.394755+0000 mgr.y (mgr.34107) 9193 : cluster [DBG] pgmap v10022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:30.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:30 smithi115 bash[46934]: audit 2022-04-22T21:55:30.214824+0000 mon.a (mon.0) 12974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:32 smithi121 bash[36902]: cluster 2022-04-22T21:55:30.395888+0000 mgr.y (mgr.34107) 9194 : cluster [DBG] pgmap v10023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:32.536 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:32 smithi121 bash[36902]: audit 2022-04-22T21:55:31.294138+0000 mon.a (mon.0) 12975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:32.536 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:32 smithi121 bash[36902]: audit 2022-04-22T21:55:31.645453+0000 mon.a (mon.0) 12976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[45495]: cluster 2022-04-22T21:55:30.395888+0000 mgr.y (mgr.34107) 9194 : cluster [DBG] pgmap v10023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[45495]: audit 2022-04-22T21:55:31.294138+0000 mon.a (mon.0) 12975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[45495]: audit 2022-04-22T21:55:31.645453+0000 mon.a (mon.0) 12976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[46934]: cluster 2022-04-22T21:55:30.395888+0000 mgr.y (mgr.34107) 9194 : cluster [DBG] pgmap v10023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[46934]: audit 2022-04-22T21:55:31.294138+0000 mon.a (mon.0) 12975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[46934]: audit 2022-04-22T21:55:31.645453+0000 mon.a (mon.0) 12976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:32.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:34 smithi121 bash[36902]: cluster 2022-04-22T21:55:32.397054+0000 mgr.y (mgr.34107) 9195 : cluster [DBG] pgmap v10024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:34.536 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:34 smithi115 bash[45495]: cluster 2022-04-22T21:55:32.397054+0000 mgr.y (mgr.34107) 9195 : cluster [DBG] pgmap v10024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:34.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:34 smithi115 bash[46934]: cluster 2022-04-22T21:55:32.397054+0000 mgr.y (mgr.34107) 9195 : cluster [DBG] pgmap v10024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:34.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:35 smithi115 bash[45495]: audit 2022-04-22T21:55:35.217981+0000 mon.a (mon.0) 12977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:35 smithi115 bash[46934]: audit 2022-04-22T21:55:35.217981+0000 mon.a (mon.0) 12977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:35 smithi121 bash[36902]: audit 2022-04-22T21:55:35.217981+0000 mon.a (mon.0) 12977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:36 smithi115 bash[46934]: cluster 2022-04-22T21:55:34.398092+0000 mgr.y (mgr.34107) 9196 : cluster [DBG] pgmap v10025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:36 smithi115 bash[45495]: cluster 2022-04-22T21:55:34.398092+0000 mgr.y (mgr.34107) 9196 : cluster [DBG] pgmap v10025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:36.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:36 smithi121 bash[36902]: cluster 2022-04-22T21:55:34.398092+0000 mgr.y (mgr.34107) 9196 : cluster [DBG] pgmap v10025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:37.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[46934]: audit 2022-04-22T21:55:36.297752+0000 mon.a (mon.0) 12978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[46934]: cluster 2022-04-22T21:55:36.399182+0000 mgr.y (mgr.34107) 9197 : cluster [DBG] pgmap v10026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[46934]: audit 2022-04-22T21:55:36.648406+0000 mon.a (mon.0) 12979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[45495]: audit 2022-04-22T21:55:36.297752+0000 mon.a (mon.0) 12978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[45495]: cluster 2022-04-22T21:55:36.399182+0000 mgr.y (mgr.34107) 9197 : cluster [DBG] pgmap v10026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:37 smithi115 bash[45495]: audit 2022-04-22T21:55:36.648406+0000 mon.a (mon.0) 12979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:37.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:37 smithi121 bash[36902]: audit 2022-04-22T21:55:36.297752+0000 mon.a (mon.0) 12978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:37 smithi121 bash[36902]: cluster 2022-04-22T21:55:36.399182+0000 mgr.y (mgr.34107) 9197 : cluster [DBG] pgmap v10026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:37 smithi121 bash[36902]: audit 2022-04-22T21:55:36.648406+0000 mon.a (mon.0) 12979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:39 smithi121 bash[36902]: cluster 2022-04-22T21:55:38.400498+0000 mgr.y (mgr.34107) 9198 : cluster [DBG] pgmap v10027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:39 smithi115 bash[45495]: cluster 2022-04-22T21:55:38.400498+0000 mgr.y (mgr.34107) 9198 : cluster [DBG] pgmap v10027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:39 smithi115 bash[46934]: cluster 2022-04-22T21:55:38.400498+0000 mgr.y (mgr.34107) 9198 : cluster [DBG] pgmap v10027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:40.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:40 smithi121 bash[36902]: audit 2022-04-22T21:55:40.220950+0000 mon.a (mon.0) 12980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:40 smithi115 bash[45495]: audit 2022-04-22T21:55:40.220950+0000 mon.a (mon.0) 12980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:40 smithi115 bash[46934]: audit 2022-04-22T21:55:40.220950+0000 mon.a (mon.0) 12980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:41 smithi121 bash[36902]: cluster 2022-04-22T21:55:40.401542+0000 mgr.y (mgr.34107) 9199 : cluster [DBG] pgmap v10028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:41 smithi121 bash[36902]: audit 2022-04-22T21:55:41.301288+0000 mon.a (mon.0) 12981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:41 smithi121 bash[36902]: audit 2022-04-22T21:55:41.651559+0000 mon.a (mon.0) 12982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[45495]: cluster 2022-04-22T21:55:40.401542+0000 mgr.y (mgr.34107) 9199 : cluster [DBG] pgmap v10028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[45495]: audit 2022-04-22T21:55:41.301288+0000 mon.a (mon.0) 12981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[45495]: audit 2022-04-22T21:55:41.651559+0000 mon.a (mon.0) 12982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[46934]: cluster 2022-04-22T21:55:40.401542+0000 mgr.y (mgr.34107) 9199 : cluster [DBG] pgmap v10028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[46934]: audit 2022-04-22T21:55:41.301288+0000 mon.a (mon.0) 12981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:41 smithi115 bash[46934]: audit 2022-04-22T21:55:41.651559+0000 mon.a (mon.0) 12982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:42.676Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:43 smithi121 bash[36902]: cluster 2022-04-22T21:55:42.402647+0000 mgr.y (mgr.34107) 9200 : cluster [DBG] pgmap v10029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:43 smithi115 bash[45495]: cluster 2022-04-22T21:55:42.402647+0000 mgr.y (mgr.34107) 9200 : cluster [DBG] pgmap v10029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:43 smithi115 bash[46934]: cluster 2022-04-22T21:55:42.402647+0000 mgr.y (mgr.34107) 9200 : cluster [DBG] pgmap v10029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:44.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:44.533Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:45 smithi121 bash[36902]: cluster 2022-04-22T21:55:44.403766+0000 mgr.y (mgr.34107) 9201 : cluster [DBG] pgmap v10030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:45 smithi121 bash[36902]: audit 2022-04-22T21:55:45.224212+0000 mon.a (mon.0) 12983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:45 smithi115 bash[45495]: cluster 2022-04-22T21:55:44.403766+0000 mgr.y (mgr.34107) 9201 : cluster [DBG] pgmap v10030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:45 smithi115 bash[45495]: audit 2022-04-22T21:55:45.224212+0000 mon.a (mon.0) 12983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:45 smithi115 bash[46934]: cluster 2022-04-22T21:55:44.403766+0000 mgr.y (mgr.34107) 9201 : cluster [DBG] pgmap v10030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:45 smithi115 bash[46934]: audit 2022-04-22T21:55:45.224212+0000 mon.a (mon.0) 12983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:46 smithi121 bash[36902]: audit 2022-04-22T21:55:46.304858+0000 mon.a (mon.0) 12984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:46 smithi121 bash[36902]: audit 2022-04-22T21:55:46.654422+0000 mon.a (mon.0) 12985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:46 smithi115 bash[45495]: audit 2022-04-22T21:55:46.304858+0000 mon.a (mon.0) 12984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:46 smithi115 bash[45495]: audit 2022-04-22T21:55:46.654422+0000 mon.a (mon.0) 12985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:46 smithi115 bash[46934]: audit 2022-04-22T21:55:46.304858+0000 mon.a (mon.0) 12984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:46 smithi115 bash[46934]: audit 2022-04-22T21:55:46.654422+0000 mon.a (mon.0) 12985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:47 smithi121 bash[36902]: cluster 2022-04-22T21:55:46.404797+0000 mgr.y (mgr.34107) 9202 : cluster [DBG] pgmap v10031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:48.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:47 smithi115 bash[45495]: cluster 2022-04-22T21:55:46.404797+0000 mgr.y (mgr.34107) 9202 : cluster [DBG] pgmap v10031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:48.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:47 smithi115 bash[46934]: cluster 2022-04-22T21:55:46.404797+0000 mgr.y (mgr.34107) 9202 : cluster [DBG] pgmap v10031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:49 smithi121 bash[36902]: cluster 2022-04-22T21:55:48.405894+0000 mgr.y (mgr.34107) 9203 : cluster [DBG] pgmap v10032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:49 smithi115 bash[45495]: cluster 2022-04-22T21:55:48.405894+0000 mgr.y (mgr.34107) 9203 : cluster [DBG] pgmap v10032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:49 smithi115 bash[46934]: cluster 2022-04-22T21:55:48.405894+0000 mgr.y (mgr.34107) 9203 : cluster [DBG] pgmap v10032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:50 smithi121 bash[36902]: audit 2022-04-22T21:55:50.227138+0000 mon.a (mon.0) 12986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:50 smithi115 bash[45495]: audit 2022-04-22T21:55:50.227138+0000 mon.a (mon.0) 12986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:50 smithi115 bash[46934]: audit 2022-04-22T21:55:50.227138+0000 mon.a (mon.0) 12986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:51 smithi121 bash[36902]: cluster 2022-04-22T21:55:50.406944+0000 mgr.y (mgr.34107) 9204 : cluster [DBG] pgmap v10033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:51 smithi121 bash[36902]: audit 2022-04-22T21:55:51.308741+0000 mon.a (mon.0) 12987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:51 smithi121 bash[36902]: audit 2022-04-22T21:55:51.657475+0000 mon.a (mon.0) 12988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[45495]: cluster 2022-04-22T21:55:50.406944+0000 mgr.y (mgr.34107) 9204 : cluster [DBG] pgmap v10033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[45495]: audit 2022-04-22T21:55:51.308741+0000 mon.a (mon.0) 12987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[45495]: audit 2022-04-22T21:55:51.657475+0000 mon.a (mon.0) 12988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[46934]: cluster 2022-04-22T21:55:50.406944+0000 mgr.y (mgr.34107) 9204 : cluster [DBG] pgmap v10033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[46934]: audit 2022-04-22T21:55:51.308741+0000 mon.a (mon.0) 12987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:51 smithi115 bash[46934]: audit 2022-04-22T21:55:51.657475+0000 mon.a (mon.0) 12988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:52.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:53.456 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:55:53.798 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:55:53.798 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 32s ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:55:53.798 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 97s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:55:53.798 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 32s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 97s ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 97s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 32s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 32s ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 97s ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 32s ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 32s ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 97s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:55:53.799 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 32s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 32s ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 32s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 32s ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 97s ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 97s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 97s ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 97s ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:55:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 97s ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:55:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:55:54.197 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:55:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:53 smithi121 bash[36902]: cluster 2022-04-22T21:55:52.408040+0000 mgr.y (mgr.34107) 9205 : cluster [DBG] pgmap v10034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:53 smithi115 bash[45495]: cluster 2022-04-22T21:55:52.408040+0000 mgr.y (mgr.34107) 9205 : cluster [DBG] pgmap v10034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:54.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:53 smithi115 bash[46934]: cluster 2022-04-22T21:55:52.408040+0000 mgr.y (mgr.34107) 9205 : cluster [DBG] pgmap v10034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:55:54.534Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:55:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:54 smithi121 bash[36902]: audit 2022-04-22T21:55:53.437964+0000 mgr.y (mgr.34107) 9206 : audit [DBG] from='client.44910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:54 smithi121 bash[36902]: audit 2022-04-22T21:55:53.776015+0000 mgr.y (mgr.34107) 9207 : audit [DBG] from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:54 smithi121 bash[36902]: audit 2022-04-22T21:55:54.192116+0000 mon.c (mon.1) 320 : audit [DBG] from='client.? 172.21.15.115:0/1711502549' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[45495]: audit 2022-04-22T21:55:53.437964+0000 mgr.y (mgr.34107) 9206 : audit [DBG] from='client.44910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[45495]: audit 2022-04-22T21:55:53.776015+0000 mgr.y (mgr.34107) 9207 : audit [DBG] from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[45495]: audit 2022-04-22T21:55:54.192116+0000 mon.c (mon.1) 320 : audit [DBG] from='client.? 172.21.15.115:0/1711502549' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[46934]: audit 2022-04-22T21:55:53.437964+0000 mgr.y (mgr.34107) 9206 : audit [DBG] from='client.44910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[46934]: audit 2022-04-22T21:55:53.776015+0000 mgr.y (mgr.34107) 9207 : audit [DBG] from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:55:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:54 smithi115 bash[46934]: audit 2022-04-22T21:55:54.192116+0000 mon.c (mon.1) 320 : audit [DBG] from='client.? 172.21.15.115:0/1711502549' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:55:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:55 smithi121 bash[36902]: cluster 2022-04-22T21:55:54.409144+0000 mgr.y (mgr.34107) 9208 : cluster [DBG] pgmap v10035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:55 smithi121 bash[36902]: audit 2022-04-22T21:55:55.230253+0000 mon.a (mon.0) 12989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:55 smithi115 bash[45495]: cluster 2022-04-22T21:55:54.409144+0000 mgr.y (mgr.34107) 9208 : cluster [DBG] pgmap v10035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:55 smithi115 bash[45495]: audit 2022-04-22T21:55:55.230253+0000 mon.a (mon.0) 12989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:55 smithi115 bash[46934]: cluster 2022-04-22T21:55:54.409144+0000 mgr.y (mgr.34107) 9208 : cluster [DBG] pgmap v10035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:55 smithi115 bash[46934]: audit 2022-04-22T21:55:55.230253+0000 mon.a (mon.0) 12989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:55:57.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:56 smithi121 bash[36902]: audit 2022-04-22T21:55:56.312409+0000 mon.a (mon.0) 12990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:57.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:56 smithi121 bash[36902]: audit 2022-04-22T21:55:56.660480+0000 mon.a (mon.0) 12991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:56 smithi115 bash[45495]: audit 2022-04-22T21:55:56.312409+0000 mon.a (mon.0) 12990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:56 smithi115 bash[45495]: audit 2022-04-22T21:55:56.660480+0000 mon.a (mon.0) 12991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:56 smithi115 bash[46934]: audit 2022-04-22T21:55:56.312409+0000 mon.a (mon.0) 12990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:55:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:56 smithi115 bash[46934]: audit 2022-04-22T21:55:56.660480+0000 mon.a (mon.0) 12991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:55:58.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:57 smithi121 bash[36902]: cluster 2022-04-22T21:55:56.410241+0000 mgr.y (mgr.34107) 9209 : cluster [DBG] pgmap v10036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:58.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:57 smithi115 bash[45495]: cluster 2022-04-22T21:55:56.410241+0000 mgr.y (mgr.34107) 9209 : cluster [DBG] pgmap v10036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:55:58.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:57 smithi115 bash[46934]: cluster 2022-04-22T21:55:56.410241+0000 mgr.y (mgr.34107) 9209 : cluster [DBG] pgmap v10036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:00.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:55:59 smithi121 bash[36902]: cluster 2022-04-22T21:55:58.411380+0000 mgr.y (mgr.34107) 9210 : cluster [DBG] pgmap v10037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:55:59 smithi115 bash[45495]: cluster 2022-04-22T21:55:58.411380+0000 mgr.y (mgr.34107) 9210 : cluster [DBG] pgmap v10037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:55:59 smithi115 bash[46934]: cluster 2022-04-22T21:55:58.411380+0000 mgr.y (mgr.34107) 9210 : cluster [DBG] pgmap v10037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:00 smithi121 bash[36902]: audit 2022-04-22T21:56:00.233279+0000 mon.a (mon.0) 12992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:00 smithi115 bash[45495]: audit 2022-04-22T21:56:00.233279+0000 mon.a (mon.0) 12992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:01.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:00 smithi115 bash[46934]: audit 2022-04-22T21:56:00.233279+0000 mon.a (mon.0) 12992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:01 smithi121 bash[36902]: cluster 2022-04-22T21:56:00.412484+0000 mgr.y (mgr.34107) 9211 : cluster [DBG] pgmap v10038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:01 smithi121 bash[36902]: audit 2022-04-22T21:56:01.316069+0000 mon.a (mon.0) 12993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:01 smithi121 bash[36902]: audit 2022-04-22T21:56:01.663309+0000 mon.a (mon.0) 12994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[45495]: cluster 2022-04-22T21:56:00.412484+0000 mgr.y (mgr.34107) 9211 : cluster [DBG] pgmap v10038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[45495]: audit 2022-04-22T21:56:01.316069+0000 mon.a (mon.0) 12993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[45495]: audit 2022-04-22T21:56:01.663309+0000 mon.a (mon.0) 12994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[46934]: cluster 2022-04-22T21:56:00.412484+0000 mgr.y (mgr.34107) 9211 : cluster [DBG] pgmap v10038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[46934]: audit 2022-04-22T21:56:01.316069+0000 mon.a (mon.0) 12993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:01 smithi115 bash[46934]: audit 2022-04-22T21:56:01.663309+0000 mon.a (mon.0) 12994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:02.679Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:03 smithi121 bash[36902]: cluster 2022-04-22T21:56:02.413563+0000 mgr.y (mgr.34107) 9212 : cluster [DBG] pgmap v10039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:03 smithi115 bash[45495]: cluster 2022-04-22T21:56:02.413563+0000 mgr.y (mgr.34107) 9212 : cluster [DBG] pgmap v10039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:03 smithi115 bash[46934]: cluster 2022-04-22T21:56:02.413563+0000 mgr.y (mgr.34107) 9212 : cluster [DBG] pgmap v10039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:04.535Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:05 smithi121 bash[36902]: cluster 2022-04-22T21:56:04.414627+0000 mgr.y (mgr.34107) 9213 : cluster [DBG] pgmap v10040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:05 smithi121 bash[36902]: audit 2022-04-22T21:56:05.236639+0000 mon.a (mon.0) 12995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:05 smithi115 bash[45495]: cluster 2022-04-22T21:56:04.414627+0000 mgr.y (mgr.34107) 9213 : cluster [DBG] pgmap v10040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:05 smithi115 bash[45495]: audit 2022-04-22T21:56:05.236639+0000 mon.a (mon.0) 12995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:05 smithi115 bash[46934]: cluster 2022-04-22T21:56:04.414627+0000 mgr.y (mgr.34107) 9213 : cluster [DBG] pgmap v10040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:05 smithi115 bash[46934]: audit 2022-04-22T21:56:05.236639+0000 mon.a (mon.0) 12995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:06 smithi121 bash[36902]: audit 2022-04-22T21:56:06.319837+0000 mon.a (mon.0) 12996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:07.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:06 smithi121 bash[36902]: audit 2022-04-22T21:56:06.666266+0000 mon.a (mon.0) 12997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:06 smithi115 bash[45495]: audit 2022-04-22T21:56:06.319837+0000 mon.a (mon.0) 12996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:06 smithi115 bash[45495]: audit 2022-04-22T21:56:06.666266+0000 mon.a (mon.0) 12997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:06 smithi115 bash[46934]: audit 2022-04-22T21:56:06.319837+0000 mon.a (mon.0) 12996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:06 smithi115 bash[46934]: audit 2022-04-22T21:56:06.666266+0000 mon.a (mon.0) 12997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:08.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:07 smithi121 bash[36902]: cluster 2022-04-22T21:56:06.415801+0000 mgr.y (mgr.34107) 9214 : cluster [DBG] pgmap v10041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:07 smithi115 bash[45495]: cluster 2022-04-22T21:56:06.415801+0000 mgr.y (mgr.34107) 9214 : cluster [DBG] pgmap v10041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:08.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:07 smithi115 bash[46934]: cluster 2022-04-22T21:56:06.415801+0000 mgr.y (mgr.34107) 9214 : cluster [DBG] pgmap v10041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:09 smithi121 bash[36902]: cluster 2022-04-22T21:56:08.416897+0000 mgr.y (mgr.34107) 9215 : cluster [DBG] pgmap v10042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:09 smithi115 bash[45495]: cluster 2022-04-22T21:56:08.416897+0000 mgr.y (mgr.34107) 9215 : cluster [DBG] pgmap v10042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:09 smithi115 bash[46934]: cluster 2022-04-22T21:56:08.416897+0000 mgr.y (mgr.34107) 9215 : cluster [DBG] pgmap v10042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:11.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:10 smithi121 bash[36902]: audit 2022-04-22T21:56:10.239858+0000 mon.a (mon.0) 12998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:10 smithi115 bash[45495]: audit 2022-04-22T21:56:10.239858+0000 mon.a (mon.0) 12998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:10 smithi115 bash[46934]: audit 2022-04-22T21:56:10.239858+0000 mon.a (mon.0) 12998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:11 smithi121 bash[36902]: cluster 2022-04-22T21:56:10.417915+0000 mgr.y (mgr.34107) 9216 : cluster [DBG] pgmap v10043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:11 smithi121 bash[36902]: audit 2022-04-22T21:56:11.323506+0000 mon.a (mon.0) 12999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:11 smithi121 bash[36902]: audit 2022-04-22T21:56:11.669562+0000 mon.a (mon.0) 13000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:12.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[46934]: cluster 2022-04-22T21:56:10.417915+0000 mgr.y (mgr.34107) 9216 : cluster [DBG] pgmap v10043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[46934]: audit 2022-04-22T21:56:11.323506+0000 mon.a (mon.0) 12999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[46934]: audit 2022-04-22T21:56:11.669562+0000 mon.a (mon.0) 13000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[45495]: cluster 2022-04-22T21:56:10.417915+0000 mgr.y (mgr.34107) 9216 : cluster [DBG] pgmap v10043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[45495]: audit 2022-04-22T21:56:11.323506+0000 mon.a (mon.0) 12999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:11 smithi115 bash[45495]: audit 2022-04-22T21:56:11.669562+0000 mon.a (mon.0) 13000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:12.680Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:13 smithi121 bash[36902]: cluster 2022-04-22T21:56:12.418843+0000 mgr.y (mgr.34107) 9217 : cluster [DBG] pgmap v10044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:13 smithi115 bash[45495]: cluster 2022-04-22T21:56:12.418843+0000 mgr.y (mgr.34107) 9217 : cluster [DBG] pgmap v10044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:13 smithi115 bash[46934]: cluster 2022-04-22T21:56:12.418843+0000 mgr.y (mgr.34107) 9217 : cluster [DBG] pgmap v10044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:14.536Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:15 smithi121 bash[36902]: cluster 2022-04-22T21:56:14.420043+0000 mgr.y (mgr.34107) 9218 : cluster [DBG] pgmap v10045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:15 smithi121 bash[36902]: audit 2022-04-22T21:56:15.242930+0000 mon.a (mon.0) 13001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:15 smithi115 bash[45495]: cluster 2022-04-22T21:56:14.420043+0000 mgr.y (mgr.34107) 9218 : cluster [DBG] pgmap v10045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:15 smithi115 bash[45495]: audit 2022-04-22T21:56:15.242930+0000 mon.a (mon.0) 13001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:15 smithi115 bash[46934]: cluster 2022-04-22T21:56:14.420043+0000 mgr.y (mgr.34107) 9218 : cluster [DBG] pgmap v10045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:15 smithi115 bash[46934]: audit 2022-04-22T21:56:15.242930+0000 mon.a (mon.0) 13001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:17 smithi121 bash[36902]: audit 2022-04-22T21:56:16.327292+0000 mon.a (mon.0) 13002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:17 smithi121 bash[36902]: audit 2022-04-22T21:56:16.387941+0000 mon.a (mon.0) 13003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:56:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:17 smithi121 bash[36902]: audit 2022-04-22T21:56:16.388485+0000 mon.a (mon.0) 13004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:56:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:17 smithi121 bash[36902]: audit 2022-04-22T21:56:16.672568+0000 mon.a (mon.0) 13005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[45495]: audit 2022-04-22T21:56:16.327292+0000 mon.a (mon.0) 13002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[45495]: audit 2022-04-22T21:56:16.387941+0000 mon.a (mon.0) 13003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[45495]: audit 2022-04-22T21:56:16.388485+0000 mon.a (mon.0) 13004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[45495]: audit 2022-04-22T21:56:16.672568+0000 mon.a (mon.0) 13005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[46934]: audit 2022-04-22T21:56:16.327292+0000 mon.a (mon.0) 13002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[46934]: audit 2022-04-22T21:56:16.387941+0000 mon.a (mon.0) 13003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:56:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[46934]: audit 2022-04-22T21:56:16.388485+0000 mon.a (mon.0) 13004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:56:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:17 smithi115 bash[46934]: audit 2022-04-22T21:56:16.672568+0000 mon.a (mon.0) 13005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:18.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:18 smithi121 bash[36902]: cluster 2022-04-22T21:56:16.421075+0000 mgr.y (mgr.34107) 9219 : cluster [DBG] pgmap v10046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:18 smithi115 bash[45495]: cluster 2022-04-22T21:56:16.421075+0000 mgr.y (mgr.34107) 9219 : cluster [DBG] pgmap v10046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:18 smithi115 bash[46934]: cluster 2022-04-22T21:56:16.421075+0000 mgr.y (mgr.34107) 9219 : cluster [DBG] pgmap v10046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:20 smithi121 bash[36902]: cluster 2022-04-22T21:56:18.422241+0000 mgr.y (mgr.34107) 9220 : cluster [DBG] pgmap v10047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:20 smithi115 bash[45495]: cluster 2022-04-22T21:56:18.422241+0000 mgr.y (mgr.34107) 9220 : cluster [DBG] pgmap v10047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:20 smithi115 bash[46934]: cluster 2022-04-22T21:56:18.422241+0000 mgr.y (mgr.34107) 9220 : cluster [DBG] pgmap v10047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:21.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:21 smithi115 bash[46934]: audit 2022-04-22T21:56:20.246129+0000 mon.a (mon.0) 13006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:21 smithi115 bash[45495]: audit 2022-04-22T21:56:20.246129+0000 mon.a (mon.0) 13006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:21 smithi121 bash[36902]: audit 2022-04-22T21:56:20.246129+0000 mon.a (mon.0) 13006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: cluster 2022-04-22T21:56:20.423249+0000 mgr.y (mgr.34107) 9221 : cluster [DBG] pgmap v10048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.331234+0000 mon.a (mon.0) 13007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.570344+0000 mon.a (mon.0) 13008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.573039+0000 mon.a (mon.0) 13009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.574541+0000 mon.a (mon.0) 13010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.675663+0000 mon.a (mon.0) 13011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:22 smithi121 bash[36902]: audit 2022-04-22T21:56:21.939340+0000 mon.a (mon.0) 13012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: cluster 2022-04-22T21:56:20.423249+0000 mgr.y (mgr.34107) 9221 : cluster [DBG] pgmap v10048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.331234+0000 mon.a (mon.0) 13007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.570344+0000 mon.a (mon.0) 13008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.573039+0000 mon.a (mon.0) 13009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.574541+0000 mon.a (mon.0) 13010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.675663+0000 mon.a (mon.0) 13011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[45495]: audit 2022-04-22T21:56:21.939340+0000 mon.a (mon.0) 13012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: cluster 2022-04-22T21:56:20.423249+0000 mgr.y (mgr.34107) 9221 : cluster [DBG] pgmap v10048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.331234+0000 mon.a (mon.0) 13007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.570344+0000 mon.a (mon.0) 13008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.573039+0000 mon.a (mon.0) 13009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.574541+0000 mon.a (mon.0) 13010 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.675663+0000 mon.a (mon.0) 13011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[46934]: audit 2022-04-22T21:56:21.939340+0000 mon.a (mon.0) 13012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:56:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:22.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:24 smithi115 bash[45495]: cluster 2022-04-22T21:56:22.424362+0000 mgr.y (mgr.34107) 9222 : cluster [DBG] pgmap v10049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:24 smithi115 bash[46934]: cluster 2022-04-22T21:56:22.424362+0000 mgr.y (mgr.34107) 9222 : cluster [DBG] pgmap v10049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:24.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:24 smithi121 bash[36902]: cluster 2022-04-22T21:56:22.424362+0000 mgr.y (mgr.34107) 9222 : cluster [DBG] pgmap v10049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:24.560 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:56:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:24.536Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 63s ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 63s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 63s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 63s ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 63s ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 63s ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 63s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 63s ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:56:24.908 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 63s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 63s ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:56:24.909 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:56:25.312 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:56:25.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:56:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:56:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[45495]: cluster 2022-04-22T21:56:24.425402+0000 mgr.y (mgr.34107) 9223 : cluster [DBG] pgmap v10050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[45495]: audit 2022-04-22T21:56:24.541315+0000 mgr.y (mgr.34107) 9224 : audit [DBG] from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[45495]: audit 2022-04-22T21:56:24.883362+0000 mgr.y (mgr.34107) 9225 : audit [DBG] from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[45495]: audit 2022-04-22T21:56:25.249230+0000 mon.a (mon.0) 13013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[45495]: audit 2022-04-22T21:56:25.309271+0000 mon.a (mon.0) 13014 : audit [DBG] from='client.? 172.21.15.115:0/2007971011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[46934]: cluster 2022-04-22T21:56:24.425402+0000 mgr.y (mgr.34107) 9223 : cluster [DBG] pgmap v10050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[46934]: audit 2022-04-22T21:56:24.541315+0000 mgr.y (mgr.34107) 9224 : audit [DBG] from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[46934]: audit 2022-04-22T21:56:24.883362+0000 mgr.y (mgr.34107) 9225 : audit [DBG] from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[46934]: audit 2022-04-22T21:56:25.249230+0000 mon.a (mon.0) 13013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:26 smithi115 bash[46934]: audit 2022-04-22T21:56:25.309271+0000 mon.a (mon.0) 13014 : audit [DBG] from='client.? 172.21.15.115:0/2007971011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:26 smithi121 bash[36902]: cluster 2022-04-22T21:56:24.425402+0000 mgr.y (mgr.34107) 9223 : cluster [DBG] pgmap v10050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:26 smithi121 bash[36902]: audit 2022-04-22T21:56:24.541315+0000 mgr.y (mgr.34107) 9224 : audit [DBG] from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:26 smithi121 bash[36902]: audit 2022-04-22T21:56:24.883362+0000 mgr.y (mgr.34107) 9225 : audit [DBG] from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:26 smithi121 bash[36902]: audit 2022-04-22T21:56:25.249230+0000 mon.a (mon.0) 13013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:26 smithi121 bash[36902]: audit 2022-04-22T21:56:25.309271+0000 mon.a (mon.0) 13014 : audit [DBG] from='client.? 172.21.15.115:0/2007971011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:27.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:27 smithi121 bash[36902]: audit 2022-04-22T21:56:26.334906+0000 mon.a (mon.0) 13015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:27.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:27 smithi121 bash[36902]: audit 2022-04-22T21:56:26.678714+0000 mon.a (mon.0) 13016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:27 smithi115 bash[45495]: audit 2022-04-22T21:56:26.334906+0000 mon.a (mon.0) 13015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:27 smithi115 bash[45495]: audit 2022-04-22T21:56:26.678714+0000 mon.a (mon.0) 13016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:27 smithi115 bash[46934]: audit 2022-04-22T21:56:26.334906+0000 mon.a (mon.0) 13015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:27 smithi115 bash[46934]: audit 2022-04-22T21:56:26.678714+0000 mon.a (mon.0) 13016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:28.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:28 smithi121 bash[36902]: cluster 2022-04-22T21:56:26.426450+0000 mgr.y (mgr.34107) 9226 : cluster [DBG] pgmap v10051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:28 smithi115 bash[45495]: cluster 2022-04-22T21:56:26.426450+0000 mgr.y (mgr.34107) 9226 : cluster [DBG] pgmap v10051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:28 smithi115 bash[46934]: cluster 2022-04-22T21:56:26.426450+0000 mgr.y (mgr.34107) 9226 : cluster [DBG] pgmap v10051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:30.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:30 smithi121 bash[36902]: cluster 2022-04-22T21:56:28.427627+0000 mgr.y (mgr.34107) 9227 : cluster [DBG] pgmap v10052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:30 smithi115 bash[45495]: cluster 2022-04-22T21:56:28.427627+0000 mgr.y (mgr.34107) 9227 : cluster [DBG] pgmap v10052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:30 smithi115 bash[46934]: cluster 2022-04-22T21:56:28.427627+0000 mgr.y (mgr.34107) 9227 : cluster [DBG] pgmap v10052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:31 smithi121 bash[36902]: audit 2022-04-22T21:56:30.252241+0000 mon.a (mon.0) 13017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:31 smithi115 bash[46934]: audit 2022-04-22T21:56:30.252241+0000 mon.a (mon.0) 13017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:31 smithi115 bash[45495]: audit 2022-04-22T21:56:30.252241+0000 mon.a (mon.0) 13017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:32 smithi121 bash[36902]: cluster 2022-04-22T21:56:30.428673+0000 mgr.y (mgr.34107) 9228 : cluster [DBG] pgmap v10053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:32 smithi121 bash[36902]: audit 2022-04-22T21:56:31.338578+0000 mon.a (mon.0) 13018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:32 smithi121 bash[36902]: audit 2022-04-22T21:56:31.681677+0000 mon.a (mon.0) 13019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[46934]: cluster 2022-04-22T21:56:30.428673+0000 mgr.y (mgr.34107) 9228 : cluster [DBG] pgmap v10053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[46934]: audit 2022-04-22T21:56:31.338578+0000 mon.a (mon.0) 13018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[46934]: audit 2022-04-22T21:56:31.681677+0000 mon.a (mon.0) 13019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[45495]: cluster 2022-04-22T21:56:30.428673+0000 mgr.y (mgr.34107) 9228 : cluster [DBG] pgmap v10053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[45495]: audit 2022-04-22T21:56:31.338578+0000 mon.a (mon.0) 13018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:32.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[45495]: audit 2022-04-22T21:56:31.681677+0000 mon.a (mon.0) 13019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:32.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:34 smithi121 bash[36902]: cluster 2022-04-22T21:56:32.429776+0000 mgr.y (mgr.34107) 9229 : cluster [DBG] pgmap v10054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:34 smithi115 bash[46934]: cluster 2022-04-22T21:56:32.429776+0000 mgr.y (mgr.34107) 9229 : cluster [DBG] pgmap v10054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:34 smithi115 bash[45495]: cluster 2022-04-22T21:56:32.429776+0000 mgr.y (mgr.34107) 9229 : cluster [DBG] pgmap v10054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:34.537Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:36 smithi121 bash[36902]: cluster 2022-04-22T21:56:34.430865+0000 mgr.y (mgr.34107) 9230 : cluster [DBG] pgmap v10055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:36 smithi121 bash[36902]: audit 2022-04-22T21:56:35.255135+0000 mon.a (mon.0) 13020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:36 smithi115 bash[45495]: cluster 2022-04-22T21:56:34.430865+0000 mgr.y (mgr.34107) 9230 : cluster [DBG] pgmap v10055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:36 smithi115 bash[45495]: audit 2022-04-22T21:56:35.255135+0000 mon.a (mon.0) 13020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:36 smithi115 bash[46934]: cluster 2022-04-22T21:56:34.430865+0000 mgr.y (mgr.34107) 9230 : cluster [DBG] pgmap v10055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:36 smithi115 bash[46934]: audit 2022-04-22T21:56:35.255135+0000 mon.a (mon.0) 13020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:37 smithi121 bash[36902]: audit 2022-04-22T21:56:36.342226+0000 mon.a (mon.0) 13021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:37 smithi121 bash[36902]: audit 2022-04-22T21:56:36.684843+0000 mon.a (mon.0) 13022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:37 smithi115 bash[45495]: audit 2022-04-22T21:56:36.342226+0000 mon.a (mon.0) 13021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:37 smithi115 bash[45495]: audit 2022-04-22T21:56:36.684843+0000 mon.a (mon.0) 13022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:37 smithi115 bash[46934]: audit 2022-04-22T21:56:36.342226+0000 mon.a (mon.0) 13021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:37 smithi115 bash[46934]: audit 2022-04-22T21:56:36.684843+0000 mon.a (mon.0) 13022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:38.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:38 smithi121 bash[36902]: cluster 2022-04-22T21:56:36.432006+0000 mgr.y (mgr.34107) 9231 : cluster [DBG] pgmap v10056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:38 smithi115 bash[45495]: cluster 2022-04-22T21:56:36.432006+0000 mgr.y (mgr.34107) 9231 : cluster [DBG] pgmap v10056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:38 smithi115 bash[46934]: cluster 2022-04-22T21:56:36.432006+0000 mgr.y (mgr.34107) 9231 : cluster [DBG] pgmap v10056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:40 smithi121 bash[36902]: cluster 2022-04-22T21:56:38.433173+0000 mgr.y (mgr.34107) 9232 : cluster [DBG] pgmap v10057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:40 smithi115 bash[45495]: cluster 2022-04-22T21:56:38.433173+0000 mgr.y (mgr.34107) 9232 : cluster [DBG] pgmap v10057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:40 smithi115 bash[46934]: cluster 2022-04-22T21:56:38.433173+0000 mgr.y (mgr.34107) 9232 : cluster [DBG] pgmap v10057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:41.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:41 smithi121 bash[36902]: audit 2022-04-22T21:56:40.258548+0000 mon.a (mon.0) 13023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:41 smithi115 bash[45495]: audit 2022-04-22T21:56:40.258548+0000 mon.a (mon.0) 13023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:41 smithi115 bash[46934]: audit 2022-04-22T21:56:40.258548+0000 mon.a (mon.0) 13023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:42 smithi121 bash[36902]: cluster 2022-04-22T21:56:40.434220+0000 mgr.y (mgr.34107) 9233 : cluster [DBG] pgmap v10058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:42 smithi121 bash[36902]: audit 2022-04-22T21:56:41.346236+0000 mon.a (mon.0) 13024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:42 smithi121 bash[36902]: audit 2022-04-22T21:56:41.688168+0000 mon.a (mon.0) 13025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[45495]: cluster 2022-04-22T21:56:40.434220+0000 mgr.y (mgr.34107) 9233 : cluster [DBG] pgmap v10058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[45495]: audit 2022-04-22T21:56:41.346236+0000 mon.a (mon.0) 13024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:42.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[45495]: audit 2022-04-22T21:56:41.688168+0000 mon.a (mon.0) 13025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[46934]: cluster 2022-04-22T21:56:40.434220+0000 mgr.y (mgr.34107) 9233 : cluster [DBG] pgmap v10058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[46934]: audit 2022-04-22T21:56:41.346236+0000 mon.a (mon.0) 13024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:42.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[46934]: audit 2022-04-22T21:56:41.688168+0000 mon.a (mon.0) 13025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:42.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:44 smithi121 bash[36902]: cluster 2022-04-22T21:56:42.435287+0000 mgr.y (mgr.34107) 9234 : cluster [DBG] pgmap v10059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:44.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:44 smithi115 bash[45495]: cluster 2022-04-22T21:56:42.435287+0000 mgr.y (mgr.34107) 9234 : cluster [DBG] pgmap v10059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:44 smithi115 bash[46934]: cluster 2022-04-22T21:56:42.435287+0000 mgr.y (mgr.34107) 9234 : cluster [DBG] pgmap v10059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:44.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:46 smithi121 bash[36902]: cluster 2022-04-22T21:56:44.436393+0000 mgr.y (mgr.34107) 9235 : cluster [DBG] pgmap v10060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:46 smithi121 bash[36902]: audit 2022-04-22T21:56:45.261937+0000 mon.a (mon.0) 13026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:46 smithi115 bash[45495]: cluster 2022-04-22T21:56:44.436393+0000 mgr.y (mgr.34107) 9235 : cluster [DBG] pgmap v10060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:46 smithi115 bash[45495]: audit 2022-04-22T21:56:45.261937+0000 mon.a (mon.0) 13026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:46 smithi115 bash[46934]: cluster 2022-04-22T21:56:44.436393+0000 mgr.y (mgr.34107) 9235 : cluster [DBG] pgmap v10060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:46 smithi115 bash[46934]: audit 2022-04-22T21:56:45.261937+0000 mon.a (mon.0) 13026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:47 smithi121 bash[36902]: audit 2022-04-22T21:56:46.349959+0000 mon.a (mon.0) 13027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:47 smithi121 bash[36902]: audit 2022-04-22T21:56:46.691202+0000 mon.a (mon.0) 13028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:47.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:47 smithi115 bash[46934]: audit 2022-04-22T21:56:46.349959+0000 mon.a (mon.0) 13027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:47.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:47 smithi115 bash[46934]: audit 2022-04-22T21:56:46.691202+0000 mon.a (mon.0) 13028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:47.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:47 smithi115 bash[45495]: audit 2022-04-22T21:56:46.349959+0000 mon.a (mon.0) 13027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:47 smithi115 bash[45495]: audit 2022-04-22T21:56:46.691202+0000 mon.a (mon.0) 13028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:48 smithi121 bash[36902]: cluster 2022-04-22T21:56:46.437471+0000 mgr.y (mgr.34107) 9236 : cluster [DBG] pgmap v10061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:48 smithi115 bash[46934]: cluster 2022-04-22T21:56:46.437471+0000 mgr.y (mgr.34107) 9236 : cluster [DBG] pgmap v10061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:48 smithi115 bash[45495]: cluster 2022-04-22T21:56:46.437471+0000 mgr.y (mgr.34107) 9236 : cluster [DBG] pgmap v10061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:50 smithi121 bash[36902]: cluster 2022-04-22T21:56:48.438796+0000 mgr.y (mgr.34107) 9237 : cluster [DBG] pgmap v10062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:50 smithi115 bash[45495]: cluster 2022-04-22T21:56:48.438796+0000 mgr.y (mgr.34107) 9237 : cluster [DBG] pgmap v10062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:50 smithi115 bash[46934]: cluster 2022-04-22T21:56:48.438796+0000 mgr.y (mgr.34107) 9237 : cluster [DBG] pgmap v10062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:51 smithi121 bash[36902]: audit 2022-04-22T21:56:50.265124+0000 mon.a (mon.0) 13029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:51 smithi115 bash[45495]: audit 2022-04-22T21:56:50.265124+0000 mon.a (mon.0) 13029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:51.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:51 smithi115 bash[46934]: audit 2022-04-22T21:56:50.265124+0000 mon.a (mon.0) 13029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:52 smithi121 bash[36902]: cluster 2022-04-22T21:56:50.439909+0000 mgr.y (mgr.34107) 9238 : cluster [DBG] pgmap v10063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:52 smithi121 bash[36902]: audit 2022-04-22T21:56:51.353877+0000 mon.a (mon.0) 13030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:52 smithi121 bash[36902]: audit 2022-04-22T21:56:51.694191+0000 mon.a (mon.0) 13031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[45495]: cluster 2022-04-22T21:56:50.439909+0000 mgr.y (mgr.34107) 9238 : cluster [DBG] pgmap v10063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[45495]: audit 2022-04-22T21:56:51.353877+0000 mon.a (mon.0) 13030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[45495]: audit 2022-04-22T21:56:51.694191+0000 mon.a (mon.0) 13031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[46934]: cluster 2022-04-22T21:56:50.439909+0000 mgr.y (mgr.34107) 9238 : cluster [DBG] pgmap v10063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[46934]: audit 2022-04-22T21:56:51.353877+0000 mon.a (mon.0) 13030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[46934]: audit 2022-04-22T21:56:51.694191+0000 mon.a (mon.0) 13031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:52.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:54 smithi121 bash[36902]: cluster 2022-04-22T21:56:52.441043+0000 mgr.y (mgr.34107) 9239 : cluster [DBG] pgmap v10064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:54 smithi115 bash[45495]: cluster 2022-04-22T21:56:52.441043+0000 mgr.y (mgr.34107) 9239 : cluster [DBG] pgmap v10064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:54 smithi115 bash[46934]: cluster 2022-04-22T21:56:52.441043+0000 mgr.y (mgr.34107) 9239 : cluster [DBG] pgmap v10064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:56:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:56:54.540Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:56:55.684 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:56:56.034 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:56:56.034 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 94s ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:56:56.034 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:56:56.034 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 94s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:56:56.034 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 94s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 94s ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 94s ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 94s ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 94s ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:56:56.035 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 94s ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 94s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 94s ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:56:56.036 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:56:56.445 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:56:56.445 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:56:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:56:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:56 smithi121 bash[36902]: cluster 2022-04-22T21:56:54.442132+0000 mgr.y (mgr.34107) 9240 : cluster [DBG] pgmap v10065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:56 smithi121 bash[36902]: audit 2022-04-22T21:56:55.268378+0000 mon.a (mon.0) 13032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:56 smithi115 bash[46934]: cluster 2022-04-22T21:56:54.442132+0000 mgr.y (mgr.34107) 9240 : cluster [DBG] pgmap v10065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:56 smithi115 bash[46934]: audit 2022-04-22T21:56:55.268378+0000 mon.a (mon.0) 13032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:56 smithi115 bash[45495]: cluster 2022-04-22T21:56:54.442132+0000 mgr.y (mgr.34107) 9240 : cluster [DBG] pgmap v10065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:56 smithi115 bash[45495]: audit 2022-04-22T21:56:55.268378+0000 mon.a (mon.0) 13032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:57 smithi121 bash[36902]: audit 2022-04-22T21:56:55.664157+0000 mgr.y (mgr.34107) 9241 : audit [DBG] from='client.44946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:57 smithi121 bash[36902]: audit 2022-04-22T21:56:56.015965+0000 mgr.y (mgr.34107) 9242 : audit [DBG] from='client.44952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:57 smithi121 bash[36902]: audit 2022-04-22T21:56:56.357547+0000 mon.a (mon.0) 13033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:57 smithi121 bash[36902]: audit 2022-04-22T21:56:56.438638+0000 mon.a (mon.0) 13034 : audit [DBG] from='client.? 172.21.15.115:0/4151945843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:57 smithi121 bash[36902]: audit 2022-04-22T21:56:56.697314+0000 mon.a (mon.0) 13035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[46934]: audit 2022-04-22T21:56:55.664157+0000 mgr.y (mgr.34107) 9241 : audit [DBG] from='client.44946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[46934]: audit 2022-04-22T21:56:56.015965+0000 mgr.y (mgr.34107) 9242 : audit [DBG] from='client.44952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[46934]: audit 2022-04-22T21:56:56.357547+0000 mon.a (mon.0) 13033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[46934]: audit 2022-04-22T21:56:56.438638+0000 mon.a (mon.0) 13034 : audit [DBG] from='client.? 172.21.15.115:0/4151945843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[46934]: audit 2022-04-22T21:56:56.697314+0000 mon.a (mon.0) 13035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[45495]: audit 2022-04-22T21:56:55.664157+0000 mgr.y (mgr.34107) 9241 : audit [DBG] from='client.44946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[45495]: audit 2022-04-22T21:56:56.015965+0000 mgr.y (mgr.34107) 9242 : audit [DBG] from='client.44952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:56:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[45495]: audit 2022-04-22T21:56:56.357547+0000 mon.a (mon.0) 13033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:56:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[45495]: audit 2022-04-22T21:56:56.438638+0000 mon.a (mon.0) 13034 : audit [DBG] from='client.? 172.21.15.115:0/4151945843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:56:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:57 smithi115 bash[45495]: audit 2022-04-22T21:56:56.697314+0000 mon.a (mon.0) 13035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:56:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:56:58 smithi121 bash[36902]: cluster 2022-04-22T21:56:56.443119+0000 mgr.y (mgr.34107) 9243 : cluster [DBG] pgmap v10066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:58.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:56:58 smithi115 bash[46934]: cluster 2022-04-22T21:56:56.443119+0000 mgr.y (mgr.34107) 9243 : cluster [DBG] pgmap v10066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:56:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:56:58 smithi115 bash[45495]: cluster 2022-04-22T21:56:56.443119+0000 mgr.y (mgr.34107) 9243 : cluster [DBG] pgmap v10066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:00 smithi121 bash[36902]: cluster 2022-04-22T21:56:58.444423+0000 mgr.y (mgr.34107) 9244 : cluster [DBG] pgmap v10067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:00 smithi115 bash[45495]: cluster 2022-04-22T21:56:58.444423+0000 mgr.y (mgr.34107) 9244 : cluster [DBG] pgmap v10067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:00 smithi115 bash[46934]: cluster 2022-04-22T21:56:58.444423+0000 mgr.y (mgr.34107) 9244 : cluster [DBG] pgmap v10067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:01.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:01 smithi121 bash[36902]: audit 2022-04-22T21:57:00.271690+0000 mon.a (mon.0) 13036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:01 smithi115 bash[45495]: audit 2022-04-22T21:57:00.271690+0000 mon.a (mon.0) 13036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:01 smithi115 bash[46934]: audit 2022-04-22T21:57:00.271690+0000 mon.a (mon.0) 13036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:02 smithi121 bash[36902]: cluster 2022-04-22T21:57:00.445485+0000 mgr.y (mgr.34107) 9245 : cluster [DBG] pgmap v10068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:02 smithi121 bash[36902]: audit 2022-04-22T21:57:01.361039+0000 mon.a (mon.0) 13037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:02 smithi121 bash[36902]: audit 2022-04-22T21:57:01.700221+0000 mon.a (mon.0) 13038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[45495]: cluster 2022-04-22T21:57:00.445485+0000 mgr.y (mgr.34107) 9245 : cluster [DBG] pgmap v10068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[45495]: audit 2022-04-22T21:57:01.361039+0000 mon.a (mon.0) 13037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[45495]: audit 2022-04-22T21:57:01.700221+0000 mon.a (mon.0) 13038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[46934]: cluster 2022-04-22T21:57:00.445485+0000 mgr.y (mgr.34107) 9245 : cluster [DBG] pgmap v10068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[46934]: audit 2022-04-22T21:57:01.361039+0000 mon.a (mon.0) 13037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[46934]: audit 2022-04-22T21:57:01.700221+0000 mon.a (mon.0) 13038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:02.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:04 smithi121 bash[36902]: cluster 2022-04-22T21:57:02.446593+0000 mgr.y (mgr.34107) 9246 : cluster [DBG] pgmap v10069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:04 smithi115 bash[46934]: cluster 2022-04-22T21:57:02.446593+0000 mgr.y (mgr.34107) 9246 : cluster [DBG] pgmap v10069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:04 smithi115 bash[45495]: cluster 2022-04-22T21:57:02.446593+0000 mgr.y (mgr.34107) 9246 : cluster [DBG] pgmap v10069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:04.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:06 smithi121 bash[36902]: cluster 2022-04-22T21:57:04.447744+0000 mgr.y (mgr.34107) 9247 : cluster [DBG] pgmap v10070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:06 smithi121 bash[36902]: audit 2022-04-22T21:57:05.274831+0000 mon.a (mon.0) 13039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:06 smithi115 bash[45495]: cluster 2022-04-22T21:57:04.447744+0000 mgr.y (mgr.34107) 9247 : cluster [DBG] pgmap v10070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:06 smithi115 bash[45495]: audit 2022-04-22T21:57:05.274831+0000 mon.a (mon.0) 13039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:06 smithi115 bash[46934]: cluster 2022-04-22T21:57:04.447744+0000 mgr.y (mgr.34107) 9247 : cluster [DBG] pgmap v10070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:06 smithi115 bash[46934]: audit 2022-04-22T21:57:05.274831+0000 mon.a (mon.0) 13039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:07 smithi121 bash[36902]: audit 2022-04-22T21:57:06.364907+0000 mon.a (mon.0) 13040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:07 smithi121 bash[36902]: audit 2022-04-22T21:57:06.703582+0000 mon.a (mon.0) 13041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:07 smithi115 bash[45495]: audit 2022-04-22T21:57:06.364907+0000 mon.a (mon.0) 13040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:07 smithi115 bash[45495]: audit 2022-04-22T21:57:06.703582+0000 mon.a (mon.0) 13041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:07 smithi115 bash[46934]: audit 2022-04-22T21:57:06.364907+0000 mon.a (mon.0) 13040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:07 smithi115 bash[46934]: audit 2022-04-22T21:57:06.703582+0000 mon.a (mon.0) 13041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:08.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:08 smithi121 bash[36902]: cluster 2022-04-22T21:57:06.448844+0000 mgr.y (mgr.34107) 9248 : cluster [DBG] pgmap v10071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:08 smithi115 bash[45495]: cluster 2022-04-22T21:57:06.448844+0000 mgr.y (mgr.34107) 9248 : cluster [DBG] pgmap v10071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:08 smithi115 bash[46934]: cluster 2022-04-22T21:57:06.448844+0000 mgr.y (mgr.34107) 9248 : cluster [DBG] pgmap v10071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:10 smithi121 bash[36902]: cluster 2022-04-22T21:57:08.449990+0000 mgr.y (mgr.34107) 9249 : cluster [DBG] pgmap v10072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:10 smithi115 bash[45495]: cluster 2022-04-22T21:57:08.449990+0000 mgr.y (mgr.34107) 9249 : cluster [DBG] pgmap v10072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:10 smithi115 bash[46934]: cluster 2022-04-22T21:57:08.449990+0000 mgr.y (mgr.34107) 9249 : cluster [DBG] pgmap v10072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:11 smithi121 bash[36902]: audit 2022-04-22T21:57:10.278342+0000 mon.a (mon.0) 13042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:11 smithi115 bash[45495]: audit 2022-04-22T21:57:10.278342+0000 mon.a (mon.0) 13042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:11 smithi115 bash[46934]: audit 2022-04-22T21:57:10.278342+0000 mon.a (mon.0) 13042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:12 smithi121 bash[36902]: cluster 2022-04-22T21:57:10.451044+0000 mgr.y (mgr.34107) 9250 : cluster [DBG] pgmap v10073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:12 smithi121 bash[36902]: audit 2022-04-22T21:57:11.368332+0000 mon.a (mon.0) 13043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:12 smithi121 bash[36902]: audit 2022-04-22T21:57:11.706712+0000 mon.a (mon.0) 13044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[45495]: cluster 2022-04-22T21:57:10.451044+0000 mgr.y (mgr.34107) 9250 : cluster [DBG] pgmap v10073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[45495]: audit 2022-04-22T21:57:11.368332+0000 mon.a (mon.0) 13043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[45495]: audit 2022-04-22T21:57:11.706712+0000 mon.a (mon.0) 13044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[46934]: cluster 2022-04-22T21:57:10.451044+0000 mgr.y (mgr.34107) 9250 : cluster [DBG] pgmap v10073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[46934]: audit 2022-04-22T21:57:11.368332+0000 mon.a (mon.0) 13043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[46934]: audit 2022-04-22T21:57:11.706712+0000 mon.a (mon.0) 13044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:12.684Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:14 smithi121 bash[36902]: cluster 2022-04-22T21:57:12.452198+0000 mgr.y (mgr.34107) 9251 : cluster [DBG] pgmap v10074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:14 smithi115 bash[46934]: cluster 2022-04-22T21:57:12.452198+0000 mgr.y (mgr.34107) 9251 : cluster [DBG] pgmap v10074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:14 smithi115 bash[45495]: cluster 2022-04-22T21:57:12.452198+0000 mgr.y (mgr.34107) 9251 : cluster [DBG] pgmap v10074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:14.540Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:16 smithi121 bash[36902]: cluster 2022-04-22T21:57:14.453292+0000 mgr.y (mgr.34107) 9252 : cluster [DBG] pgmap v10075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:16 smithi121 bash[36902]: audit 2022-04-22T21:57:15.281620+0000 mon.a (mon.0) 13045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:16 smithi115 bash[45495]: cluster 2022-04-22T21:57:14.453292+0000 mgr.y (mgr.34107) 9252 : cluster [DBG] pgmap v10075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:16 smithi115 bash[45495]: audit 2022-04-22T21:57:15.281620+0000 mon.a (mon.0) 13045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:16 smithi115 bash[46934]: cluster 2022-04-22T21:57:14.453292+0000 mgr.y (mgr.34107) 9252 : cluster [DBG] pgmap v10075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:16 smithi115 bash[46934]: audit 2022-04-22T21:57:15.281620+0000 mon.a (mon.0) 13045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:17 smithi121 bash[36902]: audit 2022-04-22T21:57:16.372086+0000 mon.a (mon.0) 13046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:17 smithi121 bash[36902]: audit 2022-04-22T21:57:16.398881+0000 mon.a (mon.0) 13047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:57:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:17 smithi121 bash[36902]: audit 2022-04-22T21:57:16.399460+0000 mon.a (mon.0) 13048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:57:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:17 smithi121 bash[36902]: audit 2022-04-22T21:57:16.709798+0000 mon.a (mon.0) 13049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[45495]: audit 2022-04-22T21:57:16.372086+0000 mon.a (mon.0) 13046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[45495]: audit 2022-04-22T21:57:16.398881+0000 mon.a (mon.0) 13047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[45495]: audit 2022-04-22T21:57:16.399460+0000 mon.a (mon.0) 13048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[45495]: audit 2022-04-22T21:57:16.709798+0000 mon.a (mon.0) 13049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[46934]: audit 2022-04-22T21:57:16.372086+0000 mon.a (mon.0) 13046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[46934]: audit 2022-04-22T21:57:16.398881+0000 mon.a (mon.0) 13047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:57:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[46934]: audit 2022-04-22T21:57:16.399460+0000 mon.a (mon.0) 13048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:57:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:17 smithi115 bash[46934]: audit 2022-04-22T21:57:16.709798+0000 mon.a (mon.0) 13049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:18 smithi121 bash[36902]: cluster 2022-04-22T21:57:16.454432+0000 mgr.y (mgr.34107) 9253 : cluster [DBG] pgmap v10076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:18 smithi115 bash[45495]: cluster 2022-04-22T21:57:16.454432+0000 mgr.y (mgr.34107) 9253 : cluster [DBG] pgmap v10076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:18.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:18 smithi115 bash[46934]: cluster 2022-04-22T21:57:16.454432+0000 mgr.y (mgr.34107) 9253 : cluster [DBG] pgmap v10076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:20.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:20 smithi121 bash[36902]: cluster 2022-04-22T21:57:18.455631+0000 mgr.y (mgr.34107) 9254 : cluster [DBG] pgmap v10077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:20 smithi115 bash[45495]: cluster 2022-04-22T21:57:18.455631+0000 mgr.y (mgr.34107) 9254 : cluster [DBG] pgmap v10077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:20.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:20 smithi115 bash[46934]: cluster 2022-04-22T21:57:18.455631+0000 mgr.y (mgr.34107) 9254 : cluster [DBG] pgmap v10077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:21.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:21 smithi121 bash[36902]: audit 2022-04-22T21:57:20.284634+0000 mon.a (mon.0) 13050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:21 smithi115 bash[45495]: audit 2022-04-22T21:57:20.284634+0000 mon.a (mon.0) 13050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:21 smithi115 bash[46934]: audit 2022-04-22T21:57:20.284634+0000 mon.a (mon.0) 13050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: cluster 2022-04-22T21:57:20.456623+0000 mgr.y (mgr.34107) 9255 : cluster [DBG] pgmap v10078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: audit 2022-04-22T21:57:21.375977+0000 mon.a (mon.0) 13051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: audit 2022-04-22T21:57:21.712858+0000 mon.a (mon.0) 13052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: audit 2022-04-22T21:57:21.947773+0000 mon.a (mon.0) 13053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:57:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: audit 2022-04-22T21:57:21.950535+0000 mon.a (mon.0) 13054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:57:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:22 smithi121 bash[36902]: audit 2022-04-22T21:57:21.952230+0000 mon.a (mon.0) 13055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:57:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: cluster 2022-04-22T21:57:20.456623+0000 mgr.y (mgr.34107) 9255 : cluster [DBG] pgmap v10078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: audit 2022-04-22T21:57:21.375977+0000 mon.a (mon.0) 13051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: audit 2022-04-22T21:57:21.712858+0000 mon.a (mon.0) 13052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: audit 2022-04-22T21:57:21.947773+0000 mon.a (mon.0) 13053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:57:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: audit 2022-04-22T21:57:21.950535+0000 mon.a (mon.0) 13054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:57:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[45495]: audit 2022-04-22T21:57:21.952230+0000 mon.a (mon.0) 13055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:57:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: cluster 2022-04-22T21:57:20.456623+0000 mgr.y (mgr.34107) 9255 : cluster [DBG] pgmap v10078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: audit 2022-04-22T21:57:21.375977+0000 mon.a (mon.0) 13051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: audit 2022-04-22T21:57:21.712858+0000 mon.a (mon.0) 13052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:22.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: audit 2022-04-22T21:57:21.947773+0000 mon.a (mon.0) 13053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:57:22.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: audit 2022-04-22T21:57:21.950535+0000 mon.a (mon.0) 13054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:57:22.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[46934]: audit 2022-04-22T21:57:21.952230+0000 mon.a (mon.0) 13055 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:57:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:22.685Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:23.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:23 smithi121 bash[36902]: audit 2022-04-22T21:57:22.335901+0000 mon.a (mon.0) 13056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:57:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:23 smithi121 bash[36902]: cluster 2022-04-22T21:57:22.457847+0000 mgr.y (mgr.34107) 9256 : cluster [DBG] pgmap v10079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:23.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:23 smithi115 bash[46934]: audit 2022-04-22T21:57:22.335901+0000 mon.a (mon.0) 13056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:57:23.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:23 smithi115 bash[46934]: cluster 2022-04-22T21:57:22.457847+0000 mgr.y (mgr.34107) 9256 : cluster [DBG] pgmap v10079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:23 smithi115 bash[45495]: audit 2022-04-22T21:57:22.335901+0000 mon.a (mon.0) 13056 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:57:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:23 smithi115 bash[45495]: cluster 2022-04-22T21:57:22.457847+0000 mgr.y (mgr.34107) 9256 : cluster [DBG] pgmap v10079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:24.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:25.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:25 smithi115 bash[46934]: cluster 2022-04-22T21:57:24.458962+0000 mgr.y (mgr.34107) 9257 : cluster [DBG] pgmap v10080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:25.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:25 smithi115 bash[46934]: audit 2022-04-22T21:57:25.287293+0000 mon.a (mon.0) 13057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:25 smithi115 bash[45495]: cluster 2022-04-22T21:57:24.458962+0000 mgr.y (mgr.34107) 9257 : cluster [DBG] pgmap v10080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:25 smithi115 bash[45495]: audit 2022-04-22T21:57:25.287293+0000 mon.a (mon.0) 13057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:25 smithi121 bash[36902]: cluster 2022-04-22T21:57:24.458962+0000 mgr.y (mgr.34107) 9257 : cluster [DBG] pgmap v10080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:26.036 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:25 smithi121 bash[36902]: audit 2022-04-22T21:57:25.287293+0000 mon.a (mon.0) 13057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:26.816 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:57:27.161 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:57:27.162 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:57:27.163 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:57:27.164 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:57:27.164 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:57:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:26 smithi121 bash[36902]: audit 2022-04-22T21:57:26.379989+0000 mon.a (mon.0) 13058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:26 smithi121 bash[36902]: audit 2022-04-22T21:57:26.715931+0000 mon.a (mon.0) 13059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:26 smithi115 bash[45495]: audit 2022-04-22T21:57:26.379989+0000 mon.a (mon.0) 13058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:26 smithi115 bash[45495]: audit 2022-04-22T21:57:26.715931+0000 mon.a (mon.0) 13059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:26 smithi115 bash[46934]: audit 2022-04-22T21:57:26.379989+0000 mon.a (mon.0) 13058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:26 smithi115 bash[46934]: audit 2022-04-22T21:57:26.715931+0000 mon.a (mon.0) 13059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:27.564 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:57:27.564 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:57:27.564 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:57:27.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:27.565 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:57:27.566 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:57:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:27 smithi121 bash[36902]: cluster 2022-04-22T21:57:26.460052+0000 mgr.y (mgr.34107) 9258 : cluster [DBG] pgmap v10081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:27 smithi121 bash[36902]: audit 2022-04-22T21:57:26.796875+0000 mgr.y (mgr.34107) 9259 : audit [DBG] from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:27 smithi121 bash[36902]: audit 2022-04-22T21:57:27.135731+0000 mgr.y (mgr.34107) 9260 : audit [DBG] from='client.54901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:27 smithi121 bash[36902]: audit 2022-04-22T21:57:27.560357+0000 mon.c (mon.1) 321 : audit [DBG] from='client.? 172.21.15.115:0/1941391338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:57:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[45495]: cluster 2022-04-22T21:57:26.460052+0000 mgr.y (mgr.34107) 9258 : cluster [DBG] pgmap v10081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[45495]: audit 2022-04-22T21:57:26.796875+0000 mgr.y (mgr.34107) 9259 : audit [DBG] from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[45495]: audit 2022-04-22T21:57:27.135731+0000 mgr.y (mgr.34107) 9260 : audit [DBG] from='client.54901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[45495]: audit 2022-04-22T21:57:27.560357+0000 mon.c (mon.1) 321 : audit [DBG] from='client.? 172.21.15.115:0/1941391338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[46934]: cluster 2022-04-22T21:57:26.460052+0000 mgr.y (mgr.34107) 9258 : cluster [DBG] pgmap v10081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[46934]: audit 2022-04-22T21:57:26.796875+0000 mgr.y (mgr.34107) 9259 : audit [DBG] from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[46934]: audit 2022-04-22T21:57:27.135731+0000 mgr.y (mgr.34107) 9260 : audit [DBG] from='client.54901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:27 smithi115 bash[46934]: audit 2022-04-22T21:57:27.560357+0000 mon.c (mon.1) 321 : audit [DBG] from='client.? 172.21.15.115:0/1941391338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:57:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:29 smithi121 bash[36902]: cluster 2022-04-22T21:57:28.461206+0000 mgr.y (mgr.34107) 9261 : cluster [DBG] pgmap v10082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:29 smithi115 bash[45495]: cluster 2022-04-22T21:57:28.461206+0000 mgr.y (mgr.34107) 9261 : cluster [DBG] pgmap v10082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:29 smithi115 bash[46934]: cluster 2022-04-22T21:57:28.461206+0000 mgr.y (mgr.34107) 9261 : cluster [DBG] pgmap v10082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:31.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:30 smithi121 bash[36902]: audit 2022-04-22T21:57:30.290510+0000 mon.a (mon.0) 13060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:30 smithi115 bash[45495]: audit 2022-04-22T21:57:30.290510+0000 mon.a (mon.0) 13060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:30 smithi115 bash[46934]: audit 2022-04-22T21:57:30.290510+0000 mon.a (mon.0) 13060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:31 smithi121 bash[36902]: cluster 2022-04-22T21:57:30.462255+0000 mgr.y (mgr.34107) 9262 : cluster [DBG] pgmap v10083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:31 smithi121 bash[36902]: audit 2022-04-22T21:57:31.383603+0000 mon.a (mon.0) 13061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:31 smithi121 bash[36902]: audit 2022-04-22T21:57:31.718936+0000 mon.a (mon.0) 13062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:32.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[45495]: cluster 2022-04-22T21:57:30.462255+0000 mgr.y (mgr.34107) 9262 : cluster [DBG] pgmap v10083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[45495]: audit 2022-04-22T21:57:31.383603+0000 mon.a (mon.0) 13061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[45495]: audit 2022-04-22T21:57:31.718936+0000 mon.a (mon.0) 13062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[46934]: cluster 2022-04-22T21:57:30.462255+0000 mgr.y (mgr.34107) 9262 : cluster [DBG] pgmap v10083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[46934]: audit 2022-04-22T21:57:31.383603+0000 mon.a (mon.0) 13061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:31 smithi115 bash[46934]: audit 2022-04-22T21:57:31.718936+0000 mon.a (mon.0) 13062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:32.685Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:33 smithi121 bash[36902]: cluster 2022-04-22T21:57:32.463185+0000 mgr.y (mgr.34107) 9263 : cluster [DBG] pgmap v10084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:33 smithi115 bash[45495]: cluster 2022-04-22T21:57:32.463185+0000 mgr.y (mgr.34107) 9263 : cluster [DBG] pgmap v10084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:33 smithi115 bash[46934]: cluster 2022-04-22T21:57:32.463185+0000 mgr.y (mgr.34107) 9263 : cluster [DBG] pgmap v10084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:34.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:35 smithi121 bash[36902]: cluster 2022-04-22T21:57:34.464261+0000 mgr.y (mgr.34107) 9264 : cluster [DBG] pgmap v10085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:35 smithi121 bash[36902]: audit 2022-04-22T21:57:35.293205+0000 mon.a (mon.0) 13063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:36.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:35 smithi115 bash[46934]: cluster 2022-04-22T21:57:34.464261+0000 mgr.y (mgr.34107) 9264 : cluster [DBG] pgmap v10085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:35 smithi115 bash[46934]: audit 2022-04-22T21:57:35.293205+0000 mon.a (mon.0) 13063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:35 smithi115 bash[45495]: cluster 2022-04-22T21:57:34.464261+0000 mgr.y (mgr.34107) 9264 : cluster [DBG] pgmap v10085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:35 smithi115 bash[45495]: audit 2022-04-22T21:57:35.293205+0000 mon.a (mon.0) 13063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:36 smithi121 bash[36902]: audit 2022-04-22T21:57:36.387298+0000 mon.a (mon.0) 13064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:36 smithi121 bash[36902]: audit 2022-04-22T21:57:36.722166+0000 mon.a (mon.0) 13065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:37.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:36 smithi115 bash[46934]: audit 2022-04-22T21:57:36.387298+0000 mon.a (mon.0) 13064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:36 smithi115 bash[46934]: audit 2022-04-22T21:57:36.722166+0000 mon.a (mon.0) 13065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:36 smithi115 bash[45495]: audit 2022-04-22T21:57:36.387298+0000 mon.a (mon.0) 13064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:36 smithi115 bash[45495]: audit 2022-04-22T21:57:36.722166+0000 mon.a (mon.0) 13065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:37 smithi121 bash[36902]: cluster 2022-04-22T21:57:36.465320+0000 mgr.y (mgr.34107) 9265 : cluster [DBG] pgmap v10086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:37 smithi115 bash[45495]: cluster 2022-04-22T21:57:36.465320+0000 mgr.y (mgr.34107) 9265 : cluster [DBG] pgmap v10086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:37 smithi115 bash[46934]: cluster 2022-04-22T21:57:36.465320+0000 mgr.y (mgr.34107) 9265 : cluster [DBG] pgmap v10086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:39 smithi121 bash[36902]: cluster 2022-04-22T21:57:38.466466+0000 mgr.y (mgr.34107) 9266 : cluster [DBG] pgmap v10087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:39 smithi115 bash[45495]: cluster 2022-04-22T21:57:38.466466+0000 mgr.y (mgr.34107) 9266 : cluster [DBG] pgmap v10087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:39 smithi115 bash[46934]: cluster 2022-04-22T21:57:38.466466+0000 mgr.y (mgr.34107) 9266 : cluster [DBG] pgmap v10087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:40 smithi121 bash[36902]: audit 2022-04-22T21:57:40.296372+0000 mon.a (mon.0) 13066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:40 smithi115 bash[45495]: audit 2022-04-22T21:57:40.296372+0000 mon.a (mon.0) 13066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:40 smithi115 bash[46934]: audit 2022-04-22T21:57:40.296372+0000 mon.a (mon.0) 13066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:41 smithi121 bash[36902]: cluster 2022-04-22T21:57:40.467494+0000 mgr.y (mgr.34107) 9267 : cluster [DBG] pgmap v10088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:42.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:41 smithi121 bash[36902]: audit 2022-04-22T21:57:41.390930+0000 mon.a (mon.0) 13067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:42.272 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:41 smithi121 bash[36902]: audit 2022-04-22T21:57:41.725560+0000 mon.a (mon.0) 13068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[45495]: cluster 2022-04-22T21:57:40.467494+0000 mgr.y (mgr.34107) 9267 : cluster [DBG] pgmap v10088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[45495]: audit 2022-04-22T21:57:41.390930+0000 mon.a (mon.0) 13067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[45495]: audit 2022-04-22T21:57:41.725560+0000 mon.a (mon.0) 13068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[46934]: cluster 2022-04-22T21:57:40.467494+0000 mgr.y (mgr.34107) 9267 : cluster [DBG] pgmap v10088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[46934]: audit 2022-04-22T21:57:41.390930+0000 mon.a (mon.0) 13067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:41 smithi115 bash[46934]: audit 2022-04-22T21:57:41.725560+0000 mon.a (mon.0) 13068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:42.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:43 smithi121 bash[36902]: cluster 2022-04-22T21:57:42.468593+0000 mgr.y (mgr.34107) 9268 : cluster [DBG] pgmap v10089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:43 smithi115 bash[45495]: cluster 2022-04-22T21:57:42.468593+0000 mgr.y (mgr.34107) 9268 : cluster [DBG] pgmap v10089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:44.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:43 smithi115 bash[46934]: cluster 2022-04-22T21:57:42.468593+0000 mgr.y (mgr.34107) 9268 : cluster [DBG] pgmap v10089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:44.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:45 smithi121 bash[36902]: cluster 2022-04-22T21:57:44.469688+0000 mgr.y (mgr.34107) 9269 : cluster [DBG] pgmap v10090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:45 smithi121 bash[36902]: audit 2022-04-22T21:57:45.299222+0000 mon.a (mon.0) 13069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:45 smithi115 bash[45495]: cluster 2022-04-22T21:57:44.469688+0000 mgr.y (mgr.34107) 9269 : cluster [DBG] pgmap v10090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:45 smithi115 bash[45495]: audit 2022-04-22T21:57:45.299222+0000 mon.a (mon.0) 13069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:45 smithi115 bash[46934]: cluster 2022-04-22T21:57:44.469688+0000 mgr.y (mgr.34107) 9269 : cluster [DBG] pgmap v10090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:45 smithi115 bash[46934]: audit 2022-04-22T21:57:45.299222+0000 mon.a (mon.0) 13069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:46 smithi121 bash[36902]: audit 2022-04-22T21:57:46.395040+0000 mon.a (mon.0) 13070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:46 smithi121 bash[36902]: audit 2022-04-22T21:57:46.728526+0000 mon.a (mon.0) 13071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:46 smithi115 bash[45495]: audit 2022-04-22T21:57:46.395040+0000 mon.a (mon.0) 13070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:46 smithi115 bash[45495]: audit 2022-04-22T21:57:46.728526+0000 mon.a (mon.0) 13071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:46 smithi115 bash[46934]: audit 2022-04-22T21:57:46.395040+0000 mon.a (mon.0) 13070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:46 smithi115 bash[46934]: audit 2022-04-22T21:57:46.728526+0000 mon.a (mon.0) 13071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:48.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:47 smithi121 bash[36902]: cluster 2022-04-22T21:57:46.470764+0000 mgr.y (mgr.34107) 9270 : cluster [DBG] pgmap v10091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:47 smithi115 bash[45495]: cluster 2022-04-22T21:57:46.470764+0000 mgr.y (mgr.34107) 9270 : cluster [DBG] pgmap v10091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:47 smithi115 bash[46934]: cluster 2022-04-22T21:57:46.470764+0000 mgr.y (mgr.34107) 9270 : cluster [DBG] pgmap v10091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:50 smithi121 bash[36902]: cluster 2022-04-22T21:57:48.471840+0000 mgr.y (mgr.34107) 9271 : cluster [DBG] pgmap v10092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:49 smithi115 bash[45495]: cluster 2022-04-22T21:57:48.471840+0000 mgr.y (mgr.34107) 9271 : cluster [DBG] pgmap v10092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:50 smithi115 bash[46934]: cluster 2022-04-22T21:57:48.471840+0000 mgr.y (mgr.34107) 9271 : cluster [DBG] pgmap v10092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:51 smithi121 bash[36902]: audit 2022-04-22T21:57:50.302244+0000 mon.a (mon.0) 13072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:51 smithi115 bash[46934]: audit 2022-04-22T21:57:50.302244+0000 mon.a (mon.0) 13072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:51.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:51 smithi115 bash[45495]: audit 2022-04-22T21:57:50.302244+0000 mon.a (mon.0) 13072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:52 smithi121 bash[36902]: cluster 2022-04-22T21:57:50.472860+0000 mgr.y (mgr.34107) 9272 : cluster [DBG] pgmap v10093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:52 smithi121 bash[36902]: audit 2022-04-22T21:57:51.398843+0000 mon.a (mon.0) 13073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:52 smithi121 bash[36902]: audit 2022-04-22T21:57:51.731770+0000 mon.a (mon.0) 13074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[45495]: cluster 2022-04-22T21:57:50.472860+0000 mgr.y (mgr.34107) 9272 : cluster [DBG] pgmap v10093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[45495]: audit 2022-04-22T21:57:51.398843+0000 mon.a (mon.0) 13073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[45495]: audit 2022-04-22T21:57:51.731770+0000 mon.a (mon.0) 13074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[46934]: cluster 2022-04-22T21:57:50.472860+0000 mgr.y (mgr.34107) 9272 : cluster [DBG] pgmap v10093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[46934]: audit 2022-04-22T21:57:51.398843+0000 mon.a (mon.0) 13073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[46934]: audit 2022-04-22T21:57:51.731770+0000 mon.a (mon.0) 13074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:52.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:54 smithi115 bash[45495]: cluster 2022-04-22T21:57:52.473928+0000 mgr.y (mgr.34107) 9273 : cluster [DBG] pgmap v10094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:54 smithi115 bash[46934]: cluster 2022-04-22T21:57:52.473928+0000 mgr.y (mgr.34107) 9273 : cluster [DBG] pgmap v10094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:54 smithi121 bash[36902]: cluster 2022-04-22T21:57:52.473928+0000 mgr.y (mgr.34107) 9273 : cluster [DBG] pgmap v10094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:57:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:57:54.542Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:57:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:56 smithi115 bash[45495]: cluster 2022-04-22T21:57:54.475030+0000 mgr.y (mgr.34107) 9274 : cluster [DBG] pgmap v10095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:56 smithi115 bash[45495]: audit 2022-04-22T21:57:55.305521+0000 mon.a (mon.0) 13075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:56 smithi115 bash[46934]: cluster 2022-04-22T21:57:54.475030+0000 mgr.y (mgr.34107) 9274 : cluster [DBG] pgmap v10095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:56 smithi115 bash[46934]: audit 2022-04-22T21:57:55.305521+0000 mon.a (mon.0) 13075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:56 smithi121 bash[36902]: cluster 2022-04-22T21:57:54.475030+0000 mgr.y (mgr.34107) 9274 : cluster [DBG] pgmap v10095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:56 smithi121 bash[36902]: audit 2022-04-22T21:57:55.305521+0000 mon.a (mon.0) 13075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:57:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:57 smithi115 bash[45495]: audit 2022-04-22T21:57:56.402694+0000 mon.a (mon.0) 13076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:57 smithi115 bash[45495]: audit 2022-04-22T21:57:56.734587+0000 mon.a (mon.0) 13077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:57 smithi115 bash[46934]: audit 2022-04-22T21:57:56.402694+0000 mon.a (mon.0) 13076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:57 smithi115 bash[46934]: audit 2022-04-22T21:57:56.734587+0000 mon.a (mon.0) 13077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:57 smithi121 bash[36902]: audit 2022-04-22T21:57:56.402694+0000 mon.a (mon.0) 13076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:57:57.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:57 smithi121 bash[36902]: audit 2022-04-22T21:57:56.734587+0000 mon.a (mon.0) 13077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:57:57.941 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:57:58.280 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:57:58.280 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:57:58.280 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:57:58.281 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:57:58.282 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:57:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:57:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:57:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:57:58.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:58 smithi115 bash[46934]: cluster 2022-04-22T21:57:56.476081+0000 mgr.y (mgr.34107) 9275 : cluster [DBG] pgmap v10096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:58 smithi115 bash[45495]: cluster 2022-04-22T21:57:56.476081+0000 mgr.y (mgr.34107) 9275 : cluster [DBG] pgmap v10096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:58 smithi121 bash[36902]: cluster 2022-04-22T21:57:56.476081+0000 mgr.y (mgr.34107) 9275 : cluster [DBG] pgmap v10096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:57:58.680 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:57:58.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:57:58.682 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:57:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:59 smithi121 bash[36902]: audit 2022-04-22T21:57:57.922169+0000 mgr.y (mgr.34107) 9276 : audit [DBG] from='client.54913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:59 smithi121 bash[36902]: audit 2022-04-22T21:57:58.257334+0000 mgr.y (mgr.34107) 9277 : audit [DBG] from='client.54919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:57:59 smithi121 bash[36902]: audit 2022-04-22T21:57:58.677245+0000 mon.c (mon.1) 322 : audit [DBG] from='client.? 172.21.15.115:0/1399821999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:57:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[45495]: audit 2022-04-22T21:57:57.922169+0000 mgr.y (mgr.34107) 9276 : audit [DBG] from='client.54913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[45495]: audit 2022-04-22T21:57:58.257334+0000 mgr.y (mgr.34107) 9277 : audit [DBG] from='client.54919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[45495]: audit 2022-04-22T21:57:58.677245+0000 mon.c (mon.1) 322 : audit [DBG] from='client.? 172.21.15.115:0/1399821999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:57:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[46934]: audit 2022-04-22T21:57:57.922169+0000 mgr.y (mgr.34107) 9276 : audit [DBG] from='client.54913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[46934]: audit 2022-04-22T21:57:58.257334+0000 mgr.y (mgr.34107) 9277 : audit [DBG] from='client.54919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:57:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:57:59 smithi115 bash[46934]: audit 2022-04-22T21:57:58.677245+0000 mon.c (mon.1) 322 : audit [DBG] from='client.? 172.21.15.115:0/1399821999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:58:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:00 smithi121 bash[36902]: cluster 2022-04-22T21:57:58.477287+0000 mgr.y (mgr.34107) 9278 : cluster [DBG] pgmap v10097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:00 smithi115 bash[45495]: cluster 2022-04-22T21:57:58.477287+0000 mgr.y (mgr.34107) 9278 : cluster [DBG] pgmap v10097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:00 smithi115 bash[46934]: cluster 2022-04-22T21:57:58.477287+0000 mgr.y (mgr.34107) 9278 : cluster [DBG] pgmap v10097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:01.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:01 smithi121 bash[36902]: audit 2022-04-22T21:58:00.308548+0000 mon.a (mon.0) 13078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:01 smithi115 bash[45495]: audit 2022-04-22T21:58:00.308548+0000 mon.a (mon.0) 13078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:01 smithi115 bash[46934]: audit 2022-04-22T21:58:00.308548+0000 mon.a (mon.0) 13078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:02 smithi121 bash[36902]: cluster 2022-04-22T21:58:00.478366+0000 mgr.y (mgr.34107) 9279 : cluster [DBG] pgmap v10098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:02 smithi121 bash[36902]: audit 2022-04-22T21:58:01.406328+0000 mon.a (mon.0) 13079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:02.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:02 smithi121 bash[36902]: audit 2022-04-22T21:58:01.737514+0000 mon.a (mon.0) 13080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[45495]: cluster 2022-04-22T21:58:00.478366+0000 mgr.y (mgr.34107) 9279 : cluster [DBG] pgmap v10098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[45495]: audit 2022-04-22T21:58:01.406328+0000 mon.a (mon.0) 13079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[45495]: audit 2022-04-22T21:58:01.737514+0000 mon.a (mon.0) 13080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[46934]: cluster 2022-04-22T21:58:00.478366+0000 mgr.y (mgr.34107) 9279 : cluster [DBG] pgmap v10098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[46934]: audit 2022-04-22T21:58:01.406328+0000 mon.a (mon.0) 13079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[46934]: audit 2022-04-22T21:58:01.737514+0000 mon.a (mon.0) 13080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:02.687Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:04 smithi121 bash[36902]: cluster 2022-04-22T21:58:02.479459+0000 mgr.y (mgr.34107) 9280 : cluster [DBG] pgmap v10099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:04 smithi115 bash[45495]: cluster 2022-04-22T21:58:02.479459+0000 mgr.y (mgr.34107) 9280 : cluster [DBG] pgmap v10099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:04 smithi115 bash[46934]: cluster 2022-04-22T21:58:02.479459+0000 mgr.y (mgr.34107) 9280 : cluster [DBG] pgmap v10099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:04.543Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:06 smithi121 bash[36902]: cluster 2022-04-22T21:58:04.480523+0000 mgr.y (mgr.34107) 9281 : cluster [DBG] pgmap v10100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:06 smithi121 bash[36902]: audit 2022-04-22T21:58:05.311578+0000 mon.a (mon.0) 13081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:06 smithi115 bash[45495]: cluster 2022-04-22T21:58:04.480523+0000 mgr.y (mgr.34107) 9281 : cluster [DBG] pgmap v10100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:06.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:06 smithi115 bash[45495]: audit 2022-04-22T21:58:05.311578+0000 mon.a (mon.0) 13081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:06.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:06 smithi115 bash[46934]: cluster 2022-04-22T21:58:04.480523+0000 mgr.y (mgr.34107) 9281 : cluster [DBG] pgmap v10100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:06.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:06 smithi115 bash[46934]: audit 2022-04-22T21:58:05.311578+0000 mon.a (mon.0) 13081 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:07.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:07 smithi121 bash[36902]: audit 2022-04-22T21:58:06.410381+0000 mon.a (mon.0) 13082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:07 smithi121 bash[36902]: audit 2022-04-22T21:58:06.740274+0000 mon.a (mon.0) 13083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:07 smithi115 bash[45495]: audit 2022-04-22T21:58:06.410381+0000 mon.a (mon.0) 13082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:07 smithi115 bash[45495]: audit 2022-04-22T21:58:06.740274+0000 mon.a (mon.0) 13083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:07 smithi115 bash[46934]: audit 2022-04-22T21:58:06.410381+0000 mon.a (mon.0) 13082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:07 smithi115 bash[46934]: audit 2022-04-22T21:58:06.740274+0000 mon.a (mon.0) 13083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:08 smithi121 bash[36902]: cluster 2022-04-22T21:58:06.481572+0000 mgr.y (mgr.34107) 9282 : cluster [DBG] pgmap v10101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:08 smithi115 bash[45495]: cluster 2022-04-22T21:58:06.481572+0000 mgr.y (mgr.34107) 9282 : cluster [DBG] pgmap v10101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:08 smithi115 bash[46934]: cluster 2022-04-22T21:58:06.481572+0000 mgr.y (mgr.34107) 9282 : cluster [DBG] pgmap v10101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:10 smithi121 bash[36902]: cluster 2022-04-22T21:58:08.482768+0000 mgr.y (mgr.34107) 9283 : cluster [DBG] pgmap v10102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:10 smithi115 bash[45495]: cluster 2022-04-22T21:58:08.482768+0000 mgr.y (mgr.34107) 9283 : cluster [DBG] pgmap v10102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:10 smithi115 bash[46934]: cluster 2022-04-22T21:58:08.482768+0000 mgr.y (mgr.34107) 9283 : cluster [DBG] pgmap v10102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:11 smithi121 bash[36902]: audit 2022-04-22T21:58:10.314498+0000 mon.a (mon.0) 13084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:11.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:11 smithi115 bash[46934]: audit 2022-04-22T21:58:10.314498+0000 mon.a (mon.0) 13084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:11 smithi115 bash[45495]: audit 2022-04-22T21:58:10.314498+0000 mon.a (mon.0) 13084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:12 smithi121 bash[36902]: cluster 2022-04-22T21:58:10.483857+0000 mgr.y (mgr.34107) 9284 : cluster [DBG] pgmap v10103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:12 smithi121 bash[36902]: audit 2022-04-22T21:58:11.414295+0000 mon.a (mon.0) 13085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:12 smithi121 bash[36902]: audit 2022-04-22T21:58:11.743428+0000 mon.a (mon.0) 13086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:12.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[46934]: cluster 2022-04-22T21:58:10.483857+0000 mgr.y (mgr.34107) 9284 : cluster [DBG] pgmap v10103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:12.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[46934]: audit 2022-04-22T21:58:11.414295+0000 mon.a (mon.0) 13085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[46934]: audit 2022-04-22T21:58:11.743428+0000 mon.a (mon.0) 13086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[45495]: cluster 2022-04-22T21:58:10.483857+0000 mgr.y (mgr.34107) 9284 : cluster [DBG] pgmap v10103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[45495]: audit 2022-04-22T21:58:11.414295+0000 mon.a (mon.0) 13085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[45495]: audit 2022-04-22T21:58:11.743428+0000 mon.a (mon.0) 13086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:12.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:14 smithi121 bash[36902]: cluster 2022-04-22T21:58:12.484944+0000 mgr.y (mgr.34107) 9285 : cluster [DBG] pgmap v10104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:14 smithi115 bash[46934]: cluster 2022-04-22T21:58:12.484944+0000 mgr.y (mgr.34107) 9285 : cluster [DBG] pgmap v10104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:14 smithi115 bash[45495]: cluster 2022-04-22T21:58:12.484944+0000 mgr.y (mgr.34107) 9285 : cluster [DBG] pgmap v10104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:14.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:16 smithi121 bash[36902]: cluster 2022-04-22T21:58:14.486122+0000 mgr.y (mgr.34107) 9286 : cluster [DBG] pgmap v10105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:16.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:16 smithi121 bash[36902]: audit 2022-04-22T21:58:15.317607+0000 mon.a (mon.0) 13087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:16.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:16 smithi115 bash[46934]: cluster 2022-04-22T21:58:14.486122+0000 mgr.y (mgr.34107) 9286 : cluster [DBG] pgmap v10105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:16 smithi115 bash[46934]: audit 2022-04-22T21:58:15.317607+0000 mon.a (mon.0) 13087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:16 smithi115 bash[45495]: cluster 2022-04-22T21:58:14.486122+0000 mgr.y (mgr.34107) 9286 : cluster [DBG] pgmap v10105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:16 smithi115 bash[45495]: audit 2022-04-22T21:58:15.317607+0000 mon.a (mon.0) 13087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:17 smithi121 bash[36902]: audit 2022-04-22T21:58:16.407345+0000 mon.a (mon.0) 13088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:58:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:17 smithi121 bash[36902]: audit 2022-04-22T21:58:16.407807+0000 mon.a (mon.0) 13089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:58:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:17 smithi121 bash[36902]: audit 2022-04-22T21:58:16.417902+0000 mon.a (mon.0) 13090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:17 smithi121 bash[36902]: audit 2022-04-22T21:58:16.746545+0000 mon.a (mon.0) 13091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[45495]: audit 2022-04-22T21:58:16.407345+0000 mon.a (mon.0) 13088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:58:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[45495]: audit 2022-04-22T21:58:16.407807+0000 mon.a (mon.0) 13089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:58:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[45495]: audit 2022-04-22T21:58:16.417902+0000 mon.a (mon.0) 13090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[45495]: audit 2022-04-22T21:58:16.746545+0000 mon.a (mon.0) 13091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[46934]: audit 2022-04-22T21:58:16.407345+0000 mon.a (mon.0) 13088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:58:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[46934]: audit 2022-04-22T21:58:16.407807+0000 mon.a (mon.0) 13089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:58:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[46934]: audit 2022-04-22T21:58:16.417902+0000 mon.a (mon.0) 13090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:17 smithi115 bash[46934]: audit 2022-04-22T21:58:16.746545+0000 mon.a (mon.0) 13091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:18 smithi121 bash[36902]: cluster 2022-04-22T21:58:16.487266+0000 mgr.y (mgr.34107) 9287 : cluster [DBG] pgmap v10106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:18 smithi115 bash[45495]: cluster 2022-04-22T21:58:16.487266+0000 mgr.y (mgr.34107) 9287 : cluster [DBG] pgmap v10106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:18 smithi115 bash[46934]: cluster 2022-04-22T21:58:16.487266+0000 mgr.y (mgr.34107) 9287 : cluster [DBG] pgmap v10106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:20 smithi121 bash[36902]: cluster 2022-04-22T21:58:18.488506+0000 mgr.y (mgr.34107) 9288 : cluster [DBG] pgmap v10107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:20 smithi115 bash[45495]: cluster 2022-04-22T21:58:18.488506+0000 mgr.y (mgr.34107) 9288 : cluster [DBG] pgmap v10107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:20.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:20 smithi115 bash[46934]: cluster 2022-04-22T21:58:18.488506+0000 mgr.y (mgr.34107) 9288 : cluster [DBG] pgmap v10107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:21 smithi121 bash[36902]: audit 2022-04-22T21:58:20.320966+0000 mon.a (mon.0) 13092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:21 smithi115 bash[45495]: audit 2022-04-22T21:58:20.320966+0000 mon.a (mon.0) 13092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:21.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:21 smithi115 bash[46934]: audit 2022-04-22T21:58:20.320966+0000 mon.a (mon.0) 13092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:22.389 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[45495]: cluster 2022-04-22T21:58:20.489560+0000 mgr.y (mgr.34107) 9289 : cluster [DBG] pgmap v10108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:22.389 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[45495]: audit 2022-04-22T21:58:21.421586+0000 mon.a (mon.0) 13093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:22.390 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[45495]: audit 2022-04-22T21:58:21.749707+0000 mon.a (mon.0) 13094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:22.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[46934]: cluster 2022-04-22T21:58:20.489560+0000 mgr.y (mgr.34107) 9289 : cluster [DBG] pgmap v10108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:22.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[46934]: audit 2022-04-22T21:58:21.421586+0000 mon.a (mon.0) 13093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:22.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[46934]: audit 2022-04-22T21:58:21.749707+0000 mon.a (mon.0) 13094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:22.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:22 smithi121 bash[36902]: cluster 2022-04-22T21:58:20.489560+0000 mgr.y (mgr.34107) 9289 : cluster [DBG] pgmap v10108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:22.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:22 smithi121 bash[36902]: audit 2022-04-22T21:58:21.421586+0000 mon.a (mon.0) 13093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:22.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:22 smithi121 bash[36902]: audit 2022-04-22T21:58:21.749707+0000 mon.a (mon.0) 13094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:22.701Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:23 smithi121 bash[36902]: audit 2022-04-22T21:58:22.344743+0000 mon.a (mon.0) 13095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:23 smithi121 bash[36902]: audit 2022-04-22T21:58:22.347657+0000 mon.a (mon.0) 13096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:23 smithi121 bash[36902]: audit 2022-04-22T21:58:22.349427+0000 mon.a (mon.0) 13097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:23 smithi121 bash[36902]: audit 2022-04-22T21:58:22.716995+0000 mon.a (mon.0) 13098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:58:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[45495]: audit 2022-04-22T21:58:22.344743+0000 mon.a (mon.0) 13095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:58:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[45495]: audit 2022-04-22T21:58:22.347657+0000 mon.a (mon.0) 13096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:58:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[45495]: audit 2022-04-22T21:58:22.349427+0000 mon.a (mon.0) 13097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:58:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[45495]: audit 2022-04-22T21:58:22.716995+0000 mon.a (mon.0) 13098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:58:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[46934]: audit 2022-04-22T21:58:22.344743+0000 mon.a (mon.0) 13095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:58:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[46934]: audit 2022-04-22T21:58:22.347657+0000 mon.a (mon.0) 13096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:58:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[46934]: audit 2022-04-22T21:58:22.349427+0000 mon.a (mon.0) 13097 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:58:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:23 smithi115 bash[46934]: audit 2022-04-22T21:58:22.716995+0000 mon.a (mon.0) 13098 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:58:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:24 smithi121 bash[36902]: cluster 2022-04-22T21:58:22.490725+0000 mgr.y (mgr.34107) 9290 : cluster [DBG] pgmap v10109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:24.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:24 smithi115 bash[46934]: cluster 2022-04-22T21:58:22.490725+0000 mgr.y (mgr.34107) 9290 : cluster [DBG] pgmap v10109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:24 smithi115 bash[45495]: cluster 2022-04-22T21:58:22.490725+0000 mgr.y (mgr.34107) 9290 : cluster [DBG] pgmap v10109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:24.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:26 smithi121 bash[36902]: cluster 2022-04-22T21:58:24.491792+0000 mgr.y (mgr.34107) 9291 : cluster [DBG] pgmap v10110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:26 smithi121 bash[36902]: audit 2022-04-22T21:58:25.324035+0000 mon.a (mon.0) 13099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:26 smithi115 bash[45495]: cluster 2022-04-22T21:58:24.491792+0000 mgr.y (mgr.34107) 9291 : cluster [DBG] pgmap v10110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:26 smithi115 bash[45495]: audit 2022-04-22T21:58:25.324035+0000 mon.a (mon.0) 13099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:26 smithi115 bash[46934]: cluster 2022-04-22T21:58:24.491792+0000 mgr.y (mgr.34107) 9291 : cluster [DBG] pgmap v10110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:26 smithi115 bash[46934]: audit 2022-04-22T21:58:25.324035+0000 mon.a (mon.0) 13099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:27 smithi121 bash[36902]: audit 2022-04-22T21:58:26.425433+0000 mon.a (mon.0) 13100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:27 smithi121 bash[36902]: audit 2022-04-22T21:58:26.752207+0000 mon.a (mon.0) 13101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:27 smithi115 bash[45495]: audit 2022-04-22T21:58:26.425433+0000 mon.a (mon.0) 13100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:27.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:27 smithi115 bash[45495]: audit 2022-04-22T21:58:26.752207+0000 mon.a (mon.0) 13101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:27 smithi115 bash[46934]: audit 2022-04-22T21:58:26.425433+0000 mon.a (mon.0) 13100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:27.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:27 smithi115 bash[46934]: audit 2022-04-22T21:58:26.752207+0000 mon.a (mon.0) 13101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:28 smithi121 bash[36902]: cluster 2022-04-22T21:58:26.492935+0000 mgr.y (mgr.34107) 9292 : cluster [DBG] pgmap v10111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:28 smithi115 bash[45495]: cluster 2022-04-22T21:58:26.492935+0000 mgr.y (mgr.34107) 9292 : cluster [DBG] pgmap v10111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:28 smithi115 bash[46934]: cluster 2022-04-22T21:58:26.492935+0000 mgr.y (mgr.34107) 9292 : cluster [DBG] pgmap v10111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:29.043 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:58:29.390 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:58:29.390 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:58:29.390 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:58:29.390 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:58:29.390 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:58:29.391 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:58:29.392 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:58:29.393 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:58:29.393 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:58:29.393 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:58:29.799 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:58:29.799 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:58:29.799 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:58:29.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:58:29.799 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:58:29.800 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:58:29.801 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:58:29.801 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:58:29.801 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:58:30.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:30 smithi121 bash[36902]: cluster 2022-04-22T21:58:28.494239+0000 mgr.y (mgr.34107) 9293 : cluster [DBG] pgmap v10112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:30 smithi121 bash[36902]: audit 2022-04-22T21:58:29.025779+0000 mgr.y (mgr.34107) 9294 : audit [DBG] from='client.45000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:30 smithi121 bash[36902]: audit 2022-04-22T21:58:29.795643+0000 mon.c (mon.1) 323 : audit [DBG] from='client.? 172.21.15.115:0/2785458245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:58:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[45495]: cluster 2022-04-22T21:58:28.494239+0000 mgr.y (mgr.34107) 9293 : cluster [DBG] pgmap v10112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[45495]: audit 2022-04-22T21:58:29.025779+0000 mgr.y (mgr.34107) 9294 : audit [DBG] from='client.45000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[45495]: audit 2022-04-22T21:58:29.795643+0000 mon.c (mon.1) 323 : audit [DBG] from='client.? 172.21.15.115:0/2785458245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:58:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[46934]: cluster 2022-04-22T21:58:28.494239+0000 mgr.y (mgr.34107) 9293 : cluster [DBG] pgmap v10112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[46934]: audit 2022-04-22T21:58:29.025779+0000 mgr.y (mgr.34107) 9294 : audit [DBG] from='client.45000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:30 smithi115 bash[46934]: audit 2022-04-22T21:58:29.795643+0000 mon.c (mon.1) 323 : audit [DBG] from='client.? 172.21.15.115:0/2785458245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:58:31.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:31 smithi121 bash[36902]: audit 2022-04-22T21:58:29.366664+0000 mgr.y (mgr.34107) 9295 : audit [DBG] from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:31 smithi121 bash[36902]: audit 2022-04-22T21:58:30.327013+0000 mon.a (mon.0) 13102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:31 smithi115 bash[45495]: audit 2022-04-22T21:58:29.366664+0000 mgr.y (mgr.34107) 9295 : audit [DBG] from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:31 smithi115 bash[45495]: audit 2022-04-22T21:58:30.327013+0000 mon.a (mon.0) 13102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:31 smithi115 bash[46934]: audit 2022-04-22T21:58:29.366664+0000 mgr.y (mgr.34107) 9295 : audit [DBG] from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:58:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:31 smithi115 bash[46934]: audit 2022-04-22T21:58:30.327013+0000 mon.a (mon.0) 13102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:32 smithi121 bash[36902]: cluster 2022-04-22T21:58:30.495326+0000 mgr.y (mgr.34107) 9296 : cluster [DBG] pgmap v10113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:32 smithi121 bash[36902]: audit 2022-04-22T21:58:31.429135+0000 mon.a (mon.0) 13103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:32 smithi121 bash[36902]: audit 2022-04-22T21:58:31.755191+0000 mon.a (mon.0) 13104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[45495]: cluster 2022-04-22T21:58:30.495326+0000 mgr.y (mgr.34107) 9296 : cluster [DBG] pgmap v10113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[45495]: audit 2022-04-22T21:58:31.429135+0000 mon.a (mon.0) 13103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[45495]: audit 2022-04-22T21:58:31.755191+0000 mon.a (mon.0) 13104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[46934]: cluster 2022-04-22T21:58:30.495326+0000 mgr.y (mgr.34107) 9296 : cluster [DBG] pgmap v10113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[46934]: audit 2022-04-22T21:58:31.429135+0000 mon.a (mon.0) 13103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[46934]: audit 2022-04-22T21:58:31.755191+0000 mon.a (mon.0) 13104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:32.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:34.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:34 smithi121 bash[36902]: cluster 2022-04-22T21:58:32.496384+0000 mgr.y (mgr.34107) 9297 : cluster [DBG] pgmap v10114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:34 smithi115 bash[45495]: cluster 2022-04-22T21:58:32.496384+0000 mgr.y (mgr.34107) 9297 : cluster [DBG] pgmap v10114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:34.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:34 smithi115 bash[46934]: cluster 2022-04-22T21:58:32.496384+0000 mgr.y (mgr.34107) 9297 : cluster [DBG] pgmap v10114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:34.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:36 smithi121 bash[36902]: cluster 2022-04-22T21:58:34.497396+0000 mgr.y (mgr.34107) 9298 : cluster [DBG] pgmap v10115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:36 smithi121 bash[36902]: audit 2022-04-22T21:58:35.330068+0000 mon.a (mon.0) 13105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:36 smithi115 bash[45495]: cluster 2022-04-22T21:58:34.497396+0000 mgr.y (mgr.34107) 9298 : cluster [DBG] pgmap v10115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:36 smithi115 bash[45495]: audit 2022-04-22T21:58:35.330068+0000 mon.a (mon.0) 13105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:36 smithi115 bash[46934]: cluster 2022-04-22T21:58:34.497396+0000 mgr.y (mgr.34107) 9298 : cluster [DBG] pgmap v10115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:36 smithi115 bash[46934]: audit 2022-04-22T21:58:35.330068+0000 mon.a (mon.0) 13105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:37 smithi121 bash[36902]: audit 2022-04-22T21:58:36.432841+0000 mon.a (mon.0) 13106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:37 smithi121 bash[36902]: audit 2022-04-22T21:58:36.758067+0000 mon.a (mon.0) 13107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:37 smithi115 bash[45495]: audit 2022-04-22T21:58:36.432841+0000 mon.a (mon.0) 13106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:37 smithi115 bash[45495]: audit 2022-04-22T21:58:36.758067+0000 mon.a (mon.0) 13107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:37 smithi115 bash[46934]: audit 2022-04-22T21:58:36.432841+0000 mon.a (mon.0) 13106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:37 smithi115 bash[46934]: audit 2022-04-22T21:58:36.758067+0000 mon.a (mon.0) 13107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:38 smithi121 bash[36902]: cluster 2022-04-22T21:58:36.498436+0000 mgr.y (mgr.34107) 9299 : cluster [DBG] pgmap v10116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:38.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:38 smithi115 bash[46934]: cluster 2022-04-22T21:58:36.498436+0000 mgr.y (mgr.34107) 9299 : cluster [DBG] pgmap v10116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:38.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:38 smithi115 bash[45495]: cluster 2022-04-22T21:58:36.498436+0000 mgr.y (mgr.34107) 9299 : cluster [DBG] pgmap v10116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:40 smithi121 bash[36902]: cluster 2022-04-22T21:58:38.499467+0000 mgr.y (mgr.34107) 9300 : cluster [DBG] pgmap v10117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:40 smithi115 bash[45495]: cluster 2022-04-22T21:58:38.499467+0000 mgr.y (mgr.34107) 9300 : cluster [DBG] pgmap v10117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:40 smithi115 bash[46934]: cluster 2022-04-22T21:58:38.499467+0000 mgr.y (mgr.34107) 9300 : cluster [DBG] pgmap v10117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:41 smithi121 bash[36902]: audit 2022-04-22T21:58:40.333177+0000 mon.a (mon.0) 13108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:41 smithi115 bash[46934]: audit 2022-04-22T21:58:40.333177+0000 mon.a (mon.0) 13108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:41 smithi115 bash[45495]: audit 2022-04-22T21:58:40.333177+0000 mon.a (mon.0) 13108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:42 smithi121 bash[36902]: cluster 2022-04-22T21:58:40.500451+0000 mgr.y (mgr.34107) 9301 : cluster [DBG] pgmap v10118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:42 smithi121 bash[36902]: audit 2022-04-22T21:58:41.436547+0000 mon.a (mon.0) 13109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:42 smithi121 bash[36902]: audit 2022-04-22T21:58:41.760765+0000 mon.a (mon.0) 13110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:42.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[46934]: cluster 2022-04-22T21:58:40.500451+0000 mgr.y (mgr.34107) 9301 : cluster [DBG] pgmap v10118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[46934]: audit 2022-04-22T21:58:41.436547+0000 mon.a (mon.0) 13109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[46934]: audit 2022-04-22T21:58:41.760765+0000 mon.a (mon.0) 13110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[45495]: cluster 2022-04-22T21:58:40.500451+0000 mgr.y (mgr.34107) 9301 : cluster [DBG] pgmap v10118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[45495]: audit 2022-04-22T21:58:41.436547+0000 mon.a (mon.0) 13109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[45495]: audit 2022-04-22T21:58:41.760765+0000 mon.a (mon.0) 13110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:42.690Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:44 smithi121 bash[36902]: cluster 2022-04-22T21:58:42.501650+0000 mgr.y (mgr.34107) 9302 : cluster [DBG] pgmap v10119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:44 smithi115 bash[45495]: cluster 2022-04-22T21:58:42.501650+0000 mgr.y (mgr.34107) 9302 : cluster [DBG] pgmap v10119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:44 smithi115 bash[46934]: cluster 2022-04-22T21:58:42.501650+0000 mgr.y (mgr.34107) 9302 : cluster [DBG] pgmap v10119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:44.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:46 smithi121 bash[36902]: cluster 2022-04-22T21:58:44.502603+0000 mgr.y (mgr.34107) 9303 : cluster [DBG] pgmap v10120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:46 smithi121 bash[36902]: audit 2022-04-22T21:58:45.336224+0000 mon.a (mon.0) 13111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:46.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:46 smithi115 bash[46934]: cluster 2022-04-22T21:58:44.502603+0000 mgr.y (mgr.34107) 9303 : cluster [DBG] pgmap v10120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:46.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:46 smithi115 bash[46934]: audit 2022-04-22T21:58:45.336224+0000 mon.a (mon.0) 13111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:46 smithi115 bash[45495]: cluster 2022-04-22T21:58:44.502603+0000 mgr.y (mgr.34107) 9303 : cluster [DBG] pgmap v10120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:46 smithi115 bash[45495]: audit 2022-04-22T21:58:45.336224+0000 mon.a (mon.0) 13111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:47 smithi121 bash[36902]: audit 2022-04-22T21:58:46.439989+0000 mon.a (mon.0) 13112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:47 smithi121 bash[36902]: audit 2022-04-22T21:58:46.763855+0000 mon.a (mon.0) 13113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:47 smithi115 bash[45495]: audit 2022-04-22T21:58:46.439989+0000 mon.a (mon.0) 13112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:47 smithi115 bash[45495]: audit 2022-04-22T21:58:46.763855+0000 mon.a (mon.0) 13113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:47 smithi115 bash[46934]: audit 2022-04-22T21:58:46.439989+0000 mon.a (mon.0) 13112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:47 smithi115 bash[46934]: audit 2022-04-22T21:58:46.763855+0000 mon.a (mon.0) 13113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:48 smithi121 bash[36902]: cluster 2022-04-22T21:58:46.503634+0000 mgr.y (mgr.34107) 9304 : cluster [DBG] pgmap v10121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:48 smithi115 bash[45495]: cluster 2022-04-22T21:58:46.503634+0000 mgr.y (mgr.34107) 9304 : cluster [DBG] pgmap v10121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:48 smithi115 bash[46934]: cluster 2022-04-22T21:58:46.503634+0000 mgr.y (mgr.34107) 9304 : cluster [DBG] pgmap v10121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:50 smithi121 bash[36902]: cluster 2022-04-22T21:58:48.504741+0000 mgr.y (mgr.34107) 9305 : cluster [DBG] pgmap v10122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:50 smithi115 bash[45495]: cluster 2022-04-22T21:58:48.504741+0000 mgr.y (mgr.34107) 9305 : cluster [DBG] pgmap v10122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:50 smithi115 bash[46934]: cluster 2022-04-22T21:58:48.504741+0000 mgr.y (mgr.34107) 9305 : cluster [DBG] pgmap v10122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:51 smithi121 bash[36902]: audit 2022-04-22T21:58:50.339861+0000 mon.a (mon.0) 13114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:51 smithi115 bash[45495]: audit 2022-04-22T21:58:50.339861+0000 mon.a (mon.0) 13114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:51.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:51 smithi115 bash[46934]: audit 2022-04-22T21:58:50.339861+0000 mon.a (mon.0) 13114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:52 smithi121 bash[36902]: cluster 2022-04-22T21:58:50.505720+0000 mgr.y (mgr.34107) 9306 : cluster [DBG] pgmap v10123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:52 smithi121 bash[36902]: audit 2022-04-22T21:58:51.443658+0000 mon.a (mon.0) 13115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:52 smithi121 bash[36902]: audit 2022-04-22T21:58:51.766933+0000 mon.a (mon.0) 13116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[45495]: cluster 2022-04-22T21:58:50.505720+0000 mgr.y (mgr.34107) 9306 : cluster [DBG] pgmap v10123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[45495]: audit 2022-04-22T21:58:51.443658+0000 mon.a (mon.0) 13115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[45495]: audit 2022-04-22T21:58:51.766933+0000 mon.a (mon.0) 13116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[46934]: cluster 2022-04-22T21:58:50.505720+0000 mgr.y (mgr.34107) 9306 : cluster [DBG] pgmap v10123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[46934]: audit 2022-04-22T21:58:51.443658+0000 mon.a (mon.0) 13115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[46934]: audit 2022-04-22T21:58:51.766933+0000 mon.a (mon.0) 13116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:52.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:54 smithi121 bash[36902]: cluster 2022-04-22T21:58:52.506707+0000 mgr.y (mgr.34107) 9307 : cluster [DBG] pgmap v10124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:54.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:54 smithi115 bash[46934]: cluster 2022-04-22T21:58:52.506707+0000 mgr.y (mgr.34107) 9307 : cluster [DBG] pgmap v10124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:54 smithi115 bash[45495]: cluster 2022-04-22T21:58:52.506707+0000 mgr.y (mgr.34107) 9307 : cluster [DBG] pgmap v10124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:58:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:58:54.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:58:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:56 smithi121 bash[36902]: cluster 2022-04-22T21:58:54.507814+0000 mgr.y (mgr.34107) 9308 : cluster [DBG] pgmap v10125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:56 smithi121 bash[36902]: audit 2022-04-22T21:58:55.343184+0000 mon.a (mon.0) 13117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:56 smithi115 bash[45495]: cluster 2022-04-22T21:58:54.507814+0000 mgr.y (mgr.34107) 9308 : cluster [DBG] pgmap v10125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:56 smithi115 bash[45495]: audit 2022-04-22T21:58:55.343184+0000 mon.a (mon.0) 13117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:56 smithi115 bash[46934]: cluster 2022-04-22T21:58:54.507814+0000 mgr.y (mgr.34107) 9308 : cluster [DBG] pgmap v10125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:56 smithi115 bash[46934]: audit 2022-04-22T21:58:55.343184+0000 mon.a (mon.0) 13117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:58:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:57 smithi115 bash[45495]: audit 2022-04-22T21:58:56.447236+0000 mon.a (mon.0) 13118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:57 smithi115 bash[45495]: audit 2022-04-22T21:58:56.770117+0000 mon.a (mon.0) 13119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:57 smithi115 bash[46934]: audit 2022-04-22T21:58:56.447236+0000 mon.a (mon.0) 13118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:57 smithi115 bash[46934]: audit 2022-04-22T21:58:56.770117+0000 mon.a (mon.0) 13119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:57 smithi121 bash[36902]: audit 2022-04-22T21:58:56.447236+0000 mon.a (mon.0) 13118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:58:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:57 smithi121 bash[36902]: audit 2022-04-22T21:58:56.770117+0000 mon.a (mon.0) 13119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:58:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:58:58 smithi115 bash[45495]: cluster 2022-04-22T21:58:56.508735+0000 mgr.y (mgr.34107) 9309 : cluster [DBG] pgmap v10126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:58:58 smithi115 bash[46934]: cluster 2022-04-22T21:58:56.508735+0000 mgr.y (mgr.34107) 9309 : cluster [DBG] pgmap v10126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:58:58.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:58:58 smithi121 bash[36902]: cluster 2022-04-22T21:58:56.508735+0000 mgr.y (mgr.34107) 9309 : cluster [DBG] pgmap v10126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:00.176 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:59:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:59:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:59:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:59:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:00 smithi115 bash[45495]: cluster 2022-04-22T21:58:58.509762+0000 mgr.y (mgr.34107) 9310 : cluster [DBG] pgmap v10127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:00 smithi115 bash[46934]: cluster 2022-04-22T21:58:58.509762+0000 mgr.y (mgr.34107) 9310 : cluster [DBG] pgmap v10127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:00 smithi121 bash[36902]: cluster 2022-04-22T21:58:58.509762+0000 mgr.y (mgr.34107) 9310 : cluster [DBG] pgmap v10127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:59:00.894 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:59:00.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:59:00.896 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:59:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[45495]: audit 2022-04-22T21:59:00.153358+0000 mgr.y (mgr.34107) 9311 : audit [DBG] from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[45495]: audit 2022-04-22T21:59:00.346420+0000 mon.a (mon.0) 13120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[45495]: audit 2022-04-22T21:59:00.890570+0000 mon.a (mon.0) 13121 : audit [DBG] from='client.? 172.21.15.115:0/2409008218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[46934]: audit 2022-04-22T21:59:00.153358+0000 mgr.y (mgr.34107) 9311 : audit [DBG] from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[46934]: audit 2022-04-22T21:59:00.346420+0000 mon.a (mon.0) 13120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:01 smithi115 bash[46934]: audit 2022-04-22T21:59:00.890570+0000 mon.a (mon.0) 13121 : audit [DBG] from='client.? 172.21.15.115:0/2409008218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:01 smithi121 bash[36902]: audit 2022-04-22T21:59:00.153358+0000 mgr.y (mgr.34107) 9311 : audit [DBG] from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:01 smithi121 bash[36902]: audit 2022-04-22T21:59:00.346420+0000 mon.a (mon.0) 13120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:01 smithi121 bash[36902]: audit 2022-04-22T21:59:00.890570+0000 mon.a (mon.0) 13121 : audit [DBG] from='client.? 172.21.15.115:0/2409008218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[45495]: audit 2022-04-22T21:59:00.490831+0000 mgr.y (mgr.34107) 9312 : audit [DBG] from='client.45024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[45495]: cluster 2022-04-22T21:59:00.510527+0000 mgr.y (mgr.34107) 9313 : cluster [DBG] pgmap v10128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[45495]: audit 2022-04-22T21:59:01.450799+0000 mon.a (mon.0) 13122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[45495]: audit 2022-04-22T21:59:01.773051+0000 mon.a (mon.0) 13123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[46934]: audit 2022-04-22T21:59:00.490831+0000 mgr.y (mgr.34107) 9312 : audit [DBG] from='client.45024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[46934]: cluster 2022-04-22T21:59:00.510527+0000 mgr.y (mgr.34107) 9313 : cluster [DBG] pgmap v10128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[46934]: audit 2022-04-22T21:59:01.450799+0000 mon.a (mon.0) 13122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[46934]: audit 2022-04-22T21:59:01.773051+0000 mon.a (mon.0) 13123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:02 smithi121 bash[36902]: audit 2022-04-22T21:59:00.490831+0000 mgr.y (mgr.34107) 9312 : audit [DBG] from='client.45024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:02 smithi121 bash[36902]: cluster 2022-04-22T21:59:00.510527+0000 mgr.y (mgr.34107) 9313 : cluster [DBG] pgmap v10128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:02 smithi121 bash[36902]: audit 2022-04-22T21:59:01.450799+0000 mon.a (mon.0) 13122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:02 smithi121 bash[36902]: audit 2022-04-22T21:59:01.773051+0000 mon.a (mon.0) 13123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:02 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:02.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:04 smithi115 bash[45495]: cluster 2022-04-22T21:59:02.511678+0000 mgr.y (mgr.34107) 9314 : cluster [DBG] pgmap v10129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:04 smithi115 bash[46934]: cluster 2022-04-22T21:59:02.511678+0000 mgr.y (mgr.34107) 9314 : cluster [DBG] pgmap v10129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:04 smithi121 bash[36902]: cluster 2022-04-22T21:59:02.511678+0000 mgr.y (mgr.34107) 9314 : cluster [DBG] pgmap v10129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:04 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:04.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:06.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:06 smithi121 bash[36902]: cluster 2022-04-22T21:59:04.512815+0000 mgr.y (mgr.34107) 9315 : cluster [DBG] pgmap v10130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:06.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:06 smithi121 bash[36902]: audit 2022-04-22T21:59:05.349339+0000 mon.a (mon.0) 13124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:06 smithi115 bash[45495]: cluster 2022-04-22T21:59:04.512815+0000 mgr.y (mgr.34107) 9315 : cluster [DBG] pgmap v10130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:06 smithi115 bash[45495]: audit 2022-04-22T21:59:05.349339+0000 mon.a (mon.0) 13124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:06 smithi115 bash[46934]: cluster 2022-04-22T21:59:04.512815+0000 mgr.y (mgr.34107) 9315 : cluster [DBG] pgmap v10130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:06 smithi115 bash[46934]: audit 2022-04-22T21:59:05.349339+0000 mon.a (mon.0) 13124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:07 smithi121 bash[36902]: audit 2022-04-22T21:59:06.454345+0000 mon.a (mon.0) 13125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:07 smithi121 bash[36902]: audit 2022-04-22T21:59:06.775628+0000 mon.a (mon.0) 13126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:07 smithi115 bash[45495]: audit 2022-04-22T21:59:06.454345+0000 mon.a (mon.0) 13125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:07 smithi115 bash[45495]: audit 2022-04-22T21:59:06.775628+0000 mon.a (mon.0) 13126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:07 smithi115 bash[46934]: audit 2022-04-22T21:59:06.454345+0000 mon.a (mon.0) 13125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:07 smithi115 bash[46934]: audit 2022-04-22T21:59:06.775628+0000 mon.a (mon.0) 13126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:08.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:08 smithi121 bash[36902]: cluster 2022-04-22T21:59:06.513876+0000 mgr.y (mgr.34107) 9316 : cluster [DBG] pgmap v10131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:08 smithi115 bash[45495]: cluster 2022-04-22T21:59:06.513876+0000 mgr.y (mgr.34107) 9316 : cluster [DBG] pgmap v10131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:08 smithi115 bash[46934]: cluster 2022-04-22T21:59:06.513876+0000 mgr.y (mgr.34107) 9316 : cluster [DBG] pgmap v10131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:09 smithi121 bash[36902]: cluster 2022-04-22T21:59:08.515070+0000 mgr.y (mgr.34107) 9317 : cluster [DBG] pgmap v10132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:09 smithi115 bash[45495]: cluster 2022-04-22T21:59:08.515070+0000 mgr.y (mgr.34107) 9317 : cluster [DBG] pgmap v10132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:09 smithi115 bash[46934]: cluster 2022-04-22T21:59:08.515070+0000 mgr.y (mgr.34107) 9317 : cluster [DBG] pgmap v10132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:10.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:10 smithi121 bash[36902]: audit 2022-04-22T21:59:10.352614+0000 mon.a (mon.0) 13127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:10 smithi115 bash[45495]: audit 2022-04-22T21:59:10.352614+0000 mon.a (mon.0) 13127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:10 smithi115 bash[46934]: audit 2022-04-22T21:59:10.352614+0000 mon.a (mon.0) 13127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:11 smithi121 bash[36902]: cluster 2022-04-22T21:59:10.516197+0000 mgr.y (mgr.34107) 9318 : cluster [DBG] pgmap v10133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:11 smithi121 bash[36902]: audit 2022-04-22T21:59:11.458276+0000 mon.a (mon.0) 13128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:11 smithi121 bash[36902]: audit 2022-04-22T21:59:11.778630+0000 mon.a (mon.0) 13129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[45495]: cluster 2022-04-22T21:59:10.516197+0000 mgr.y (mgr.34107) 9318 : cluster [DBG] pgmap v10133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[45495]: audit 2022-04-22T21:59:11.458276+0000 mon.a (mon.0) 13128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[45495]: audit 2022-04-22T21:59:11.778630+0000 mon.a (mon.0) 13129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[46934]: cluster 2022-04-22T21:59:10.516197+0000 mgr.y (mgr.34107) 9318 : cluster [DBG] pgmap v10133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[46934]: audit 2022-04-22T21:59:11.458276+0000 mon.a (mon.0) 13128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:11 smithi115 bash[46934]: audit 2022-04-22T21:59:11.778630+0000 mon.a (mon.0) 13129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:12 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:12.692Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:13 smithi121 bash[36902]: cluster 2022-04-22T21:59:12.517235+0000 mgr.y (mgr.34107) 9319 : cluster [DBG] pgmap v10134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:13 smithi115 bash[45495]: cluster 2022-04-22T21:59:12.517235+0000 mgr.y (mgr.34107) 9319 : cluster [DBG] pgmap v10134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:13 smithi115 bash[46934]: cluster 2022-04-22T21:59:12.517235+0000 mgr.y (mgr.34107) 9319 : cluster [DBG] pgmap v10134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:14 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:14.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:15 smithi121 bash[36902]: cluster 2022-04-22T21:59:14.518338+0000 mgr.y (mgr.34107) 9320 : cluster [DBG] pgmap v10135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:15 smithi121 bash[36902]: audit 2022-04-22T21:59:15.355727+0000 mon.a (mon.0) 13130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:15 smithi115 bash[45495]: cluster 2022-04-22T21:59:14.518338+0000 mgr.y (mgr.34107) 9320 : cluster [DBG] pgmap v10135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:15 smithi115 bash[45495]: audit 2022-04-22T21:59:15.355727+0000 mon.a (mon.0) 13130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:15 smithi115 bash[46934]: cluster 2022-04-22T21:59:14.518338+0000 mgr.y (mgr.34107) 9320 : cluster [DBG] pgmap v10135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:15 smithi115 bash[46934]: audit 2022-04-22T21:59:15.355727+0000 mon.a (mon.0) 13130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:16 smithi121 bash[36902]: audit 2022-04-22T21:59:16.421090+0000 mon.a (mon.0) 13131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:59:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:16 smithi121 bash[36902]: audit 2022-04-22T21:59:16.421530+0000 mon.a (mon.0) 13132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:59:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:16 smithi121 bash[36902]: audit 2022-04-22T21:59:16.461926+0000 mon.a (mon.0) 13133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:16 smithi121 bash[36902]: audit 2022-04-22T21:59:16.781218+0000 mon.a (mon.0) 13134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[45495]: audit 2022-04-22T21:59:16.421090+0000 mon.a (mon.0) 13131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:59:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[45495]: audit 2022-04-22T21:59:16.421530+0000 mon.a (mon.0) 13132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:59:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[45495]: audit 2022-04-22T21:59:16.461926+0000 mon.a (mon.0) 13133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[45495]: audit 2022-04-22T21:59:16.781218+0000 mon.a (mon.0) 13134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[46934]: audit 2022-04-22T21:59:16.421090+0000 mon.a (mon.0) 13131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T21:59:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[46934]: audit 2022-04-22T21:59:16.421530+0000 mon.a (mon.0) 13132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T21:59:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[46934]: audit 2022-04-22T21:59:16.461926+0000 mon.a (mon.0) 13133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:16 smithi115 bash[46934]: audit 2022-04-22T21:59:16.781218+0000 mon.a (mon.0) 13134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:17 smithi121 bash[36902]: cluster 2022-04-22T21:59:16.519343+0000 mgr.y (mgr.34107) 9321 : cluster [DBG] pgmap v10136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:17 smithi115 bash[45495]: cluster 2022-04-22T21:59:16.519343+0000 mgr.y (mgr.34107) 9321 : cluster [DBG] pgmap v10136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:17 smithi115 bash[46934]: cluster 2022-04-22T21:59:16.519343+0000 mgr.y (mgr.34107) 9321 : cluster [DBG] pgmap v10136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:19 smithi121 bash[36902]: cluster 2022-04-22T21:59:18.520444+0000 mgr.y (mgr.34107) 9322 : cluster [DBG] pgmap v10137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:19 smithi115 bash[45495]: cluster 2022-04-22T21:59:18.520444+0000 mgr.y (mgr.34107) 9322 : cluster [DBG] pgmap v10137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:19 smithi115 bash[46934]: cluster 2022-04-22T21:59:18.520444+0000 mgr.y (mgr.34107) 9322 : cluster [DBG] pgmap v10137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:20 smithi121 bash[36902]: audit 2022-04-22T21:59:20.359006+0000 mon.a (mon.0) 13135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:20 smithi115 bash[45495]: audit 2022-04-22T21:59:20.359006+0000 mon.a (mon.0) 13135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:20 smithi115 bash[46934]: audit 2022-04-22T21:59:20.359006+0000 mon.a (mon.0) 13135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:22.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:21 smithi121 bash[36902]: cluster 2022-04-22T21:59:20.521498+0000 mgr.y (mgr.34107) 9323 : cluster [DBG] pgmap v10138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:22.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:21 smithi121 bash[36902]: audit 2022-04-22T21:59:21.465740+0000 mon.a (mon.0) 13136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:22.268 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:21 smithi121 bash[36902]: audit 2022-04-22T21:59:21.784225+0000 mon.a (mon.0) 13137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[45495]: cluster 2022-04-22T21:59:20.521498+0000 mgr.y (mgr.34107) 9323 : cluster [DBG] pgmap v10138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[45495]: audit 2022-04-22T21:59:21.465740+0000 mon.a (mon.0) 13136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[45495]: audit 2022-04-22T21:59:21.784225+0000 mon.a (mon.0) 13137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[46934]: cluster 2022-04-22T21:59:20.521498+0000 mgr.y (mgr.34107) 9323 : cluster [DBG] pgmap v10138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[46934]: audit 2022-04-22T21:59:21.465740+0000 mon.a (mon.0) 13136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:21 smithi115 bash[46934]: audit 2022-04-22T21:59:21.784225+0000 mon.a (mon.0) 13137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:22.997 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:22.694Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:23.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[45495]: audit 2022-04-22T21:59:22.724353+0000 mon.a (mon.0) 13138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:59:23.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[45495]: audit 2022-04-22T21:59:22.727254+0000 mon.a (mon.0) 13139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:59:23.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[45495]: audit 2022-04-22T21:59:22.728985+0000 mon.a (mon.0) 13140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:59:23.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[46934]: audit 2022-04-22T21:59:22.724353+0000 mon.a (mon.0) 13138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:59:23.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[46934]: audit 2022-04-22T21:59:22.727254+0000 mon.a (mon.0) 13139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:59:23.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:22 smithi115 bash[46934]: audit 2022-04-22T21:59:22.728985+0000 mon.a (mon.0) 13140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:59:23.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:22 smithi121 bash[36902]: audit 2022-04-22T21:59:22.724353+0000 mon.a (mon.0) 13138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T21:59:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:22 smithi121 bash[36902]: audit 2022-04-22T21:59:22.727254+0000 mon.a (mon.0) 13139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T21:59:23.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:22 smithi121 bash[36902]: audit 2022-04-22T21:59:22.728985+0000 mon.a (mon.0) 13140 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T21:59:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:24 smithi121 bash[36902]: cluster 2022-04-22T21:59:22.522646+0000 mgr.y (mgr.34107) 9324 : cluster [DBG] pgmap v10139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:24 smithi121 bash[36902]: audit 2022-04-22T21:59:23.090226+0000 mon.a (mon.0) 13141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:59:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:24 smithi115 bash[46934]: cluster 2022-04-22T21:59:22.522646+0000 mgr.y (mgr.34107) 9324 : cluster [DBG] pgmap v10139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:24 smithi115 bash[46934]: audit 2022-04-22T21:59:23.090226+0000 mon.a (mon.0) 13141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:59:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:24 smithi115 bash[45495]: cluster 2022-04-22T21:59:22.522646+0000 mgr.y (mgr.34107) 9324 : cluster [DBG] pgmap v10139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:24 smithi115 bash[45495]: audit 2022-04-22T21:59:23.090226+0000 mon.a (mon.0) 13141 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T21:59:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:24 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:24.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:26 smithi121 bash[36902]: cluster 2022-04-22T21:59:24.523698+0000 mgr.y (mgr.34107) 9325 : cluster [DBG] pgmap v10140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:26 smithi121 bash[36902]: audit 2022-04-22T21:59:25.362303+0000 mon.a (mon.0) 13142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:26 smithi115 bash[45495]: cluster 2022-04-22T21:59:24.523698+0000 mgr.y (mgr.34107) 9325 : cluster [DBG] pgmap v10140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:26 smithi115 bash[45495]: audit 2022-04-22T21:59:25.362303+0000 mon.a (mon.0) 13142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:26 smithi115 bash[46934]: cluster 2022-04-22T21:59:24.523698+0000 mgr.y (mgr.34107) 9325 : cluster [DBG] pgmap v10140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:26 smithi115 bash[46934]: audit 2022-04-22T21:59:25.362303+0000 mon.a (mon.0) 13142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:27 smithi121 bash[36902]: audit 2022-04-22T21:59:26.469724+0000 mon.a (mon.0) 13143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:27 smithi121 bash[36902]: audit 2022-04-22T21:59:26.787242+0000 mon.a (mon.0) 13144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:27 smithi115 bash[45495]: audit 2022-04-22T21:59:26.469724+0000 mon.a (mon.0) 13143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:27 smithi115 bash[45495]: audit 2022-04-22T21:59:26.787242+0000 mon.a (mon.0) 13144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:27 smithi115 bash[46934]: audit 2022-04-22T21:59:26.469724+0000 mon.a (mon.0) 13143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:27 smithi115 bash[46934]: audit 2022-04-22T21:59:26.787242+0000 mon.a (mon.0) 13144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:28 smithi121 bash[36902]: cluster 2022-04-22T21:59:26.524703+0000 mgr.y (mgr.34107) 9326 : cluster [DBG] pgmap v10141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:28.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:28 smithi115 bash[45495]: cluster 2022-04-22T21:59:26.524703+0000 mgr.y (mgr.34107) 9326 : cluster [DBG] pgmap v10141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:28.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:28 smithi115 bash[46934]: cluster 2022-04-22T21:59:26.524703+0000 mgr.y (mgr.34107) 9326 : cluster [DBG] pgmap v10141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:30 smithi121 bash[36902]: cluster 2022-04-22T21:59:28.525869+0000 mgr.y (mgr.34107) 9327 : cluster [DBG] pgmap v10142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:30 smithi115 bash[45495]: cluster 2022-04-22T21:59:28.525869+0000 mgr.y (mgr.34107) 9327 : cluster [DBG] pgmap v10142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:30 smithi115 bash[46934]: cluster 2022-04-22T21:59:28.525869+0000 mgr.y (mgr.34107) 9327 : cluster [DBG] pgmap v10142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:31.275 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T21:59:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:31 smithi121 bash[36902]: audit 2022-04-22T21:59:30.365621+0000 mon.a (mon.0) 13145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:31 smithi115 bash[45495]: audit 2022-04-22T21:59:30.365621+0000 mon.a (mon.0) 13145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:31 smithi115 bash[46934]: audit 2022-04-22T21:59:30.365621+0000 mon.a (mon.0) 13145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:31.627 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T21:59:31.627 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T21:59:31.628 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T21:59:31.629 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T21:59:32.033 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:59:32.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:32.035 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T21:59:32.035 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T21:59:32.035 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:32.035 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T21:59:32.035 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T21:59:32.036 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T21:59:32.036 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T21:59:32.036 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T21:59:32.036 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T21:59:32.037 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T21:59:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:32 smithi121 bash[36902]: cluster 2022-04-22T21:59:30.527046+0000 mgr.y (mgr.34107) 9328 : cluster [DBG] pgmap v10143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:32 smithi121 bash[36902]: audit 2022-04-22T21:59:31.256956+0000 mgr.y (mgr.34107) 9329 : audit [DBG] from='client.45036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:32 smithi121 bash[36902]: audit 2022-04-22T21:59:31.473320+0000 mon.a (mon.0) 13146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:32 smithi121 bash[36902]: audit 2022-04-22T21:59:31.790237+0000 mon.a (mon.0) 13147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:32 smithi121 bash[36902]: audit 2022-04-22T21:59:32.030220+0000 mon.c (mon.1) 324 : audit [DBG] from='client.? 172.21.15.115:0/484859457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[45495]: cluster 2022-04-22T21:59:30.527046+0000 mgr.y (mgr.34107) 9328 : cluster [DBG] pgmap v10143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:32.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[45495]: audit 2022-04-22T21:59:31.256956+0000 mgr.y (mgr.34107) 9329 : audit [DBG] from='client.45036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[45495]: audit 2022-04-22T21:59:31.473320+0000 mon.a (mon.0) 13146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[45495]: audit 2022-04-22T21:59:31.790237+0000 mon.a (mon.0) 13147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:32.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[45495]: audit 2022-04-22T21:59:32.030220+0000 mon.c (mon.1) 324 : audit [DBG] from='client.? 172.21.15.115:0/484859457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[46934]: cluster 2022-04-22T21:59:30.527046+0000 mgr.y (mgr.34107) 9328 : cluster [DBG] pgmap v10143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[46934]: audit 2022-04-22T21:59:31.256956+0000 mgr.y (mgr.34107) 9329 : audit [DBG] from='client.45036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[46934]: audit 2022-04-22T21:59:31.473320+0000 mon.a (mon.0) 13146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[46934]: audit 2022-04-22T21:59:31.790237+0000 mon.a (mon.0) 13147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:32.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[46934]: audit 2022-04-22T21:59:32.030220+0000 mon.c (mon.1) 324 : audit [DBG] from='client.? 172.21.15.115:0/484859457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T21:59:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:32 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:32.693Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:33.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:33 smithi121 bash[36902]: audit 2022-04-22T21:59:31.599195+0000 mgr.y (mgr.34107) 9330 : audit [DBG] from='client.54973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:33 smithi115 bash[45495]: audit 2022-04-22T21:59:31.599195+0000 mgr.y (mgr.34107) 9330 : audit [DBG] from='client.54973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:33 smithi115 bash[46934]: audit 2022-04-22T21:59:31.599195+0000 mgr.y (mgr.34107) 9330 : audit [DBG] from='client.54973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T21:59:34.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:34 smithi121 bash[36902]: cluster 2022-04-22T21:59:32.528170+0000 mgr.y (mgr.34107) 9331 : cluster [DBG] pgmap v10144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:34 smithi115 bash[45495]: cluster 2022-04-22T21:59:32.528170+0000 mgr.y (mgr.34107) 9331 : cluster [DBG] pgmap v10144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:34 smithi115 bash[46934]: cluster 2022-04-22T21:59:32.528170+0000 mgr.y (mgr.34107) 9331 : cluster [DBG] pgmap v10144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:34 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:34.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:36 smithi121 bash[36902]: cluster 2022-04-22T21:59:34.529282+0000 mgr.y (mgr.34107) 9332 : cluster [DBG] pgmap v10145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:36 smithi121 bash[36902]: audit 2022-04-22T21:59:35.368926+0000 mon.a (mon.0) 13148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:36 smithi115 bash[45495]: cluster 2022-04-22T21:59:34.529282+0000 mgr.y (mgr.34107) 9332 : cluster [DBG] pgmap v10145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:36 smithi115 bash[45495]: audit 2022-04-22T21:59:35.368926+0000 mon.a (mon.0) 13148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:36 smithi115 bash[46934]: cluster 2022-04-22T21:59:34.529282+0000 mgr.y (mgr.34107) 9332 : cluster [DBG] pgmap v10145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:36 smithi115 bash[46934]: audit 2022-04-22T21:59:35.368926+0000 mon.a (mon.0) 13148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:37 smithi121 bash[36902]: audit 2022-04-22T21:59:36.476842+0000 mon.a (mon.0) 13149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:37 smithi121 bash[36902]: audit 2022-04-22T21:59:36.793257+0000 mon.a (mon.0) 13150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:37 smithi115 bash[45495]: audit 2022-04-22T21:59:36.476842+0000 mon.a (mon.0) 13149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:37 smithi115 bash[45495]: audit 2022-04-22T21:59:36.793257+0000 mon.a (mon.0) 13150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:37 smithi115 bash[46934]: audit 2022-04-22T21:59:36.476842+0000 mon.a (mon.0) 13149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:37 smithi115 bash[46934]: audit 2022-04-22T21:59:36.793257+0000 mon.a (mon.0) 13150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:38.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:38 smithi121 bash[36902]: cluster 2022-04-22T21:59:36.530459+0000 mgr.y (mgr.34107) 9333 : cluster [DBG] pgmap v10146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:38 smithi115 bash[46934]: cluster 2022-04-22T21:59:36.530459+0000 mgr.y (mgr.34107) 9333 : cluster [DBG] pgmap v10146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:38 smithi115 bash[45495]: cluster 2022-04-22T21:59:36.530459+0000 mgr.y (mgr.34107) 9333 : cluster [DBG] pgmap v10146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:40 smithi121 bash[36902]: cluster 2022-04-22T21:59:38.531415+0000 mgr.y (mgr.34107) 9334 : cluster [DBG] pgmap v10147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:40 smithi115 bash[45495]: cluster 2022-04-22T21:59:38.531415+0000 mgr.y (mgr.34107) 9334 : cluster [DBG] pgmap v10147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:40 smithi115 bash[46934]: cluster 2022-04-22T21:59:38.531415+0000 mgr.y (mgr.34107) 9334 : cluster [DBG] pgmap v10147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:41 smithi121 bash[36902]: audit 2022-04-22T21:59:40.372051+0000 mon.a (mon.0) 13151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:41 smithi115 bash[45495]: audit 2022-04-22T21:59:40.372051+0000 mon.a (mon.0) 13151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:41 smithi115 bash[46934]: audit 2022-04-22T21:59:40.372051+0000 mon.a (mon.0) 13151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:42 smithi121 bash[36902]: cluster 2022-04-22T21:59:40.532408+0000 mgr.y (mgr.34107) 9335 : cluster [DBG] pgmap v10148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:42 smithi121 bash[36902]: audit 2022-04-22T21:59:41.480738+0000 mon.a (mon.0) 13152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:42 smithi121 bash[36902]: audit 2022-04-22T21:59:41.796282+0000 mon.a (mon.0) 13153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[45495]: cluster 2022-04-22T21:59:40.532408+0000 mgr.y (mgr.34107) 9335 : cluster [DBG] pgmap v10148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[45495]: audit 2022-04-22T21:59:41.480738+0000 mon.a (mon.0) 13152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[45495]: audit 2022-04-22T21:59:41.796282+0000 mon.a (mon.0) 13153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[46934]: cluster 2022-04-22T21:59:40.532408+0000 mgr.y (mgr.34107) 9335 : cluster [DBG] pgmap v10148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[46934]: audit 2022-04-22T21:59:41.480738+0000 mon.a (mon.0) 13152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[46934]: audit 2022-04-22T21:59:41.796282+0000 mon.a (mon.0) 13153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:42 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:42.694Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:44 smithi121 bash[36902]: cluster 2022-04-22T21:59:42.533540+0000 mgr.y (mgr.34107) 9336 : cluster [DBG] pgmap v10149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:44 smithi115 bash[45495]: cluster 2022-04-22T21:59:42.533540+0000 mgr.y (mgr.34107) 9336 : cluster [DBG] pgmap v10149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:44 smithi115 bash[46934]: cluster 2022-04-22T21:59:42.533540+0000 mgr.y (mgr.34107) 9336 : cluster [DBG] pgmap v10149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:44 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:44.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:46 smithi121 bash[36902]: cluster 2022-04-22T21:59:44.534637+0000 mgr.y (mgr.34107) 9337 : cluster [DBG] pgmap v10150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:46 smithi121 bash[36902]: audit 2022-04-22T21:59:45.375316+0000 mon.a (mon.0) 13154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:46 smithi115 bash[45495]: cluster 2022-04-22T21:59:44.534637+0000 mgr.y (mgr.34107) 9337 : cluster [DBG] pgmap v10150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:46 smithi115 bash[45495]: audit 2022-04-22T21:59:45.375316+0000 mon.a (mon.0) 13154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:46 smithi115 bash[46934]: cluster 2022-04-22T21:59:44.534637+0000 mgr.y (mgr.34107) 9337 : cluster [DBG] pgmap v10150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:46 smithi115 bash[46934]: audit 2022-04-22T21:59:45.375316+0000 mon.a (mon.0) 13154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:47 smithi121 bash[36902]: audit 2022-04-22T21:59:46.484232+0000 mon.a (mon.0) 13155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:47 smithi121 bash[36902]: audit 2022-04-22T21:59:46.798898+0000 mon.a (mon.0) 13156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:47 smithi115 bash[45495]: audit 2022-04-22T21:59:46.484232+0000 mon.a (mon.0) 13155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:47 smithi115 bash[45495]: audit 2022-04-22T21:59:46.798898+0000 mon.a (mon.0) 13156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:47 smithi115 bash[46934]: audit 2022-04-22T21:59:46.484232+0000 mon.a (mon.0) 13155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:47 smithi115 bash[46934]: audit 2022-04-22T21:59:46.798898+0000 mon.a (mon.0) 13156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:48 smithi121 bash[36902]: cluster 2022-04-22T21:59:46.535901+0000 mgr.y (mgr.34107) 9338 : cluster [DBG] pgmap v10151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:48 smithi115 bash[45495]: cluster 2022-04-22T21:59:46.535901+0000 mgr.y (mgr.34107) 9338 : cluster [DBG] pgmap v10151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:48 smithi115 bash[46934]: cluster 2022-04-22T21:59:46.535901+0000 mgr.y (mgr.34107) 9338 : cluster [DBG] pgmap v10151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:50 smithi121 bash[36902]: cluster 2022-04-22T21:59:48.537131+0000 mgr.y (mgr.34107) 9339 : cluster [DBG] pgmap v10152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:50 smithi115 bash[46934]: cluster 2022-04-22T21:59:48.537131+0000 mgr.y (mgr.34107) 9339 : cluster [DBG] pgmap v10152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:50 smithi115 bash[45495]: cluster 2022-04-22T21:59:48.537131+0000 mgr.y (mgr.34107) 9339 : cluster [DBG] pgmap v10152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:51.519 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:51 smithi121 bash[36902]: audit 2022-04-22T21:59:50.378344+0000 mon.a (mon.0) 13157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:51 smithi115 bash[45495]: audit 2022-04-22T21:59:50.378344+0000 mon.a (mon.0) 13157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:51 smithi115 bash[46934]: audit 2022-04-22T21:59:50.378344+0000 mon.a (mon.0) 13157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:52 smithi121 bash[36902]: cluster 2022-04-22T21:59:50.538361+0000 mgr.y (mgr.34107) 9340 : cluster [DBG] pgmap v10153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:52 smithi121 bash[36902]: audit 2022-04-22T21:59:51.487763+0000 mon.a (mon.0) 13158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:52 smithi121 bash[36902]: audit 2022-04-22T21:59:51.801841+0000 mon.a (mon.0) 13159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[46934]: cluster 2022-04-22T21:59:50.538361+0000 mgr.y (mgr.34107) 9340 : cluster [DBG] pgmap v10153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[46934]: audit 2022-04-22T21:59:51.487763+0000 mon.a (mon.0) 13158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[46934]: audit 2022-04-22T21:59:51.801841+0000 mon.a (mon.0) 13159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[45495]: cluster 2022-04-22T21:59:50.538361+0000 mgr.y (mgr.34107) 9340 : cluster [DBG] pgmap v10153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[45495]: audit 2022-04-22T21:59:51.487763+0000 mon.a (mon.0) 13158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:52.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[45495]: audit 2022-04-22T21:59:51.801841+0000 mon.a (mon.0) 13159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:53.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:52 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:52.695Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:54.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:54 smithi121 bash[36902]: cluster 2022-04-22T21:59:52.539320+0000 mgr.y (mgr.34107) 9341 : cluster [DBG] pgmap v10154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:54 smithi115 bash[45495]: cluster 2022-04-22T21:59:52.539320+0000 mgr.y (mgr.34107) 9341 : cluster [DBG] pgmap v10154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:54 smithi115 bash[46934]: cluster 2022-04-22T21:59:52.539320+0000 mgr.y (mgr.34107) 9341 : cluster [DBG] pgmap v10154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 21:59:54 smithi115 bash[60466]: level=error ts=2022-04-22T21:59:54.549Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T21:59:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:56 smithi121 bash[36902]: cluster 2022-04-22T21:59:54.540488+0000 mgr.y (mgr.34107) 9342 : cluster [DBG] pgmap v10155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:56 smithi121 bash[36902]: audit 2022-04-22T21:59:55.381580+0000 mon.a (mon.0) 13160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:56 smithi115 bash[45495]: cluster 2022-04-22T21:59:54.540488+0000 mgr.y (mgr.34107) 9342 : cluster [DBG] pgmap v10155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:56 smithi115 bash[45495]: audit 2022-04-22T21:59:55.381580+0000 mon.a (mon.0) 13160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:56 smithi115 bash[46934]: cluster 2022-04-22T21:59:54.540488+0000 mgr.y (mgr.34107) 9342 : cluster [DBG] pgmap v10155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:56 smithi115 bash[46934]: audit 2022-04-22T21:59:55.381580+0000 mon.a (mon.0) 13160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T21:59:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:57 smithi121 bash[36902]: audit 2022-04-22T21:59:56.491537+0000 mon.a (mon.0) 13161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:57 smithi121 bash[36902]: audit 2022-04-22T21:59:56.804738+0000 mon.a (mon.0) 13162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:57.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:57 smithi115 bash[45495]: audit 2022-04-22T21:59:56.491537+0000 mon.a (mon.0) 13161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:57.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:57 smithi115 bash[45495]: audit 2022-04-22T21:59:56.804738+0000 mon.a (mon.0) 13162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:57 smithi115 bash[46934]: audit 2022-04-22T21:59:56.491537+0000 mon.a (mon.0) 13161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T21:59:57.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:57 smithi115 bash[46934]: audit 2022-04-22T21:59:56.804738+0000 mon.a (mon.0) 13162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T21:59:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 21:59:58 smithi121 bash[36902]: cluster 2022-04-22T21:59:56.541608+0000 mgr.y (mgr.34107) 9343 : cluster [DBG] pgmap v10156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 21:59:58 smithi115 bash[45495]: cluster 2022-04-22T21:59:56.541608+0000 mgr.y (mgr.34107) 9343 : cluster [DBG] pgmap v10156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T21:59:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 21:59:58 smithi115 bash[46934]: cluster 2022-04-22T21:59:56.541608+0000 mgr.y (mgr.34107) 9343 : cluster [DBG] pgmap v10156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:00 smithi121 bash[36902]: cluster 2022-04-22T21:59:58.542792+0000 mgr.y (mgr.34107) 9344 : cluster [DBG] pgmap v10157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:00 smithi121 bash[36902]: cluster 2022-04-22T22:00:00.000181+0000 mon.a (mon.0) 13163 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:00:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:00 smithi115 bash[46934]: cluster 2022-04-22T21:59:58.542792+0000 mgr.y (mgr.34107) 9344 : cluster [DBG] pgmap v10157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:00 smithi115 bash[46934]: cluster 2022-04-22T22:00:00.000181+0000 mon.a (mon.0) 13163 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:00:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:00 smithi115 bash[45495]: cluster 2022-04-22T21:59:58.542792+0000 mgr.y (mgr.34107) 9344 : cluster [DBG] pgmap v10157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:00 smithi115 bash[45495]: cluster 2022-04-22T22:00:00.000181+0000 mon.a (mon.0) 13163 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:00:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:01 smithi121 bash[36902]: audit 2022-04-22T22:00:00.384851+0000 mon.a (mon.0) 13164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:01 smithi115 bash[45495]: audit 2022-04-22T22:00:00.384851+0000 mon.a (mon.0) 13164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:01 smithi115 bash[46934]: audit 2022-04-22T22:00:00.384851+0000 mon.a (mon.0) 13164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:02.414 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:00:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:02 smithi121 bash[36902]: cluster 2022-04-22T22:00:00.543858+0000 mgr.y (mgr.34107) 9345 : cluster [DBG] pgmap v10158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:02 smithi121 bash[36902]: audit 2022-04-22T22:00:01.495238+0000 mon.a (mon.0) 13165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:02 smithi121 bash[36902]: audit 2022-04-22T22:00:01.807652+0000 mon.a (mon.0) 13166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:02.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[46934]: cluster 2022-04-22T22:00:00.543858+0000 mgr.y (mgr.34107) 9345 : cluster [DBG] pgmap v10158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[46934]: audit 2022-04-22T22:00:01.495238+0000 mon.a (mon.0) 13165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[46934]: audit 2022-04-22T22:00:01.807652+0000 mon.a (mon.0) 13166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[45495]: cluster 2022-04-22T22:00:00.543858+0000 mgr.y (mgr.34107) 9345 : cluster [DBG] pgmap v10158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[45495]: audit 2022-04-22T22:00:01.495238+0000 mon.a (mon.0) 13165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[45495]: audit 2022-04-22T22:00:01.807652+0000 mon.a (mon.0) 13166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:02.762 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:00:02.762 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:00:02.762 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:00:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:00:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:00:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:02.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:03.172 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:00:03.188 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:00:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:00:03.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:03.190 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:00:03.190 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:00:03.190 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:00:03.190 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:00:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:03 smithi121 bash[36902]: audit 2022-04-22T22:00:03.169110+0000 mon.c (mon.1) 325 : audit [DBG] from='client.? 172.21.15.115:0/617140402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:03 smithi115 bash[45495]: audit 2022-04-22T22:00:03.169110+0000 mon.c (mon.1) 325 : audit [DBG] from='client.? 172.21.15.115:0/617140402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:03 smithi115 bash[46934]: audit 2022-04-22T22:00:03.169110+0000 mon.c (mon.1) 325 : audit [DBG] from='client.? 172.21.15.115:0/617140402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:04 smithi121 bash[36902]: audit 2022-04-22T22:00:02.396046+0000 mgr.y (mgr.34107) 9346 : audit [DBG] from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:04 smithi121 bash[36902]: cluster 2022-04-22T22:00:02.544813+0000 mgr.y (mgr.34107) 9347 : cluster [DBG] pgmap v10159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:04 smithi121 bash[36902]: audit 2022-04-22T22:00:02.740411+0000 mgr.y (mgr.34107) 9348 : audit [DBG] from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[45495]: audit 2022-04-22T22:00:02.396046+0000 mgr.y (mgr.34107) 9346 : audit [DBG] from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[45495]: cluster 2022-04-22T22:00:02.544813+0000 mgr.y (mgr.34107) 9347 : cluster [DBG] pgmap v10159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[45495]: audit 2022-04-22T22:00:02.740411+0000 mgr.y (mgr.34107) 9348 : audit [DBG] from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[46934]: audit 2022-04-22T22:00:02.396046+0000 mgr.y (mgr.34107) 9346 : audit [DBG] from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[46934]: cluster 2022-04-22T22:00:02.544813+0000 mgr.y (mgr.34107) 9347 : cluster [DBG] pgmap v10159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[46934]: audit 2022-04-22T22:00:02.740411+0000 mgr.y (mgr.34107) 9348 : audit [DBG] from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:04.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:06 smithi121 bash[36902]: cluster 2022-04-22T22:00:04.545838+0000 mgr.y (mgr.34107) 9349 : cluster [DBG] pgmap v10160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:06 smithi121 bash[36902]: audit 2022-04-22T22:00:05.387606+0000 mon.a (mon.0) 13167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:06 smithi115 bash[45495]: cluster 2022-04-22T22:00:04.545838+0000 mgr.y (mgr.34107) 9349 : cluster [DBG] pgmap v10160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:06 smithi115 bash[45495]: audit 2022-04-22T22:00:05.387606+0000 mon.a (mon.0) 13167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:06 smithi115 bash[46934]: cluster 2022-04-22T22:00:04.545838+0000 mgr.y (mgr.34107) 9349 : cluster [DBG] pgmap v10160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:06 smithi115 bash[46934]: audit 2022-04-22T22:00:05.387606+0000 mon.a (mon.0) 13167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:07 smithi115 bash[45495]: audit 2022-04-22T22:00:06.498742+0000 mon.a (mon.0) 13168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:07 smithi115 bash[45495]: audit 2022-04-22T22:00:06.810774+0000 mon.a (mon.0) 13169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:07 smithi115 bash[46934]: audit 2022-04-22T22:00:06.498742+0000 mon.a (mon.0) 13168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:07 smithi115 bash[46934]: audit 2022-04-22T22:00:06.810774+0000 mon.a (mon.0) 13169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:07 smithi121 bash[36902]: audit 2022-04-22T22:00:06.498742+0000 mon.a (mon.0) 13168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:07 smithi121 bash[36902]: audit 2022-04-22T22:00:06.810774+0000 mon.a (mon.0) 13169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:08 smithi115 bash[45495]: cluster 2022-04-22T22:00:06.546844+0000 mgr.y (mgr.34107) 9350 : cluster [DBG] pgmap v10161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:08 smithi115 bash[46934]: cluster 2022-04-22T22:00:06.546844+0000 mgr.y (mgr.34107) 9350 : cluster [DBG] pgmap v10161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:08 smithi121 bash[36902]: cluster 2022-04-22T22:00:06.546844+0000 mgr.y (mgr.34107) 9350 : cluster [DBG] pgmap v10161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:10 smithi115 bash[45495]: cluster 2022-04-22T22:00:08.547991+0000 mgr.y (mgr.34107) 9351 : cluster [DBG] pgmap v10162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:10 smithi115 bash[46934]: cluster 2022-04-22T22:00:08.547991+0000 mgr.y (mgr.34107) 9351 : cluster [DBG] pgmap v10162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:10 smithi121 bash[36902]: cluster 2022-04-22T22:00:08.547991+0000 mgr.y (mgr.34107) 9351 : cluster [DBG] pgmap v10162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:11 smithi115 bash[45495]: audit 2022-04-22T22:00:10.391111+0000 mon.a (mon.0) 13170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:11 smithi115 bash[46934]: audit 2022-04-22T22:00:10.391111+0000 mon.a (mon.0) 13170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:11 smithi121 bash[36902]: audit 2022-04-22T22:00:10.391111+0000 mon.a (mon.0) 13170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[45495]: cluster 2022-04-22T22:00:10.548962+0000 mgr.y (mgr.34107) 9352 : cluster [DBG] pgmap v10163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[45495]: audit 2022-04-22T22:00:11.502425+0000 mon.a (mon.0) 13171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[45495]: audit 2022-04-22T22:00:11.813854+0000 mon.a (mon.0) 13172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[46934]: cluster 2022-04-22T22:00:10.548962+0000 mgr.y (mgr.34107) 9352 : cluster [DBG] pgmap v10163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[46934]: audit 2022-04-22T22:00:11.502425+0000 mon.a (mon.0) 13171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[46934]: audit 2022-04-22T22:00:11.813854+0000 mon.a (mon.0) 13172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:12 smithi121 bash[36902]: cluster 2022-04-22T22:00:10.548962+0000 mgr.y (mgr.34107) 9352 : cluster [DBG] pgmap v10163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:12 smithi121 bash[36902]: audit 2022-04-22T22:00:11.502425+0000 mon.a (mon.0) 13171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:12 smithi121 bash[36902]: audit 2022-04-22T22:00:11.813854+0000 mon.a (mon.0) 13172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:12.695Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:14 smithi115 bash[45495]: cluster 2022-04-22T22:00:12.550090+0000 mgr.y (mgr.34107) 9353 : cluster [DBG] pgmap v10164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:14 smithi115 bash[46934]: cluster 2022-04-22T22:00:12.550090+0000 mgr.y (mgr.34107) 9353 : cluster [DBG] pgmap v10164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:14 smithi121 bash[36902]: cluster 2022-04-22T22:00:12.550090+0000 mgr.y (mgr.34107) 9353 : cluster [DBG] pgmap v10164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:14.552Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:16 smithi121 bash[36902]: cluster 2022-04-22T22:00:14.551295+0000 mgr.y (mgr.34107) 9354 : cluster [DBG] pgmap v10165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:16 smithi121 bash[36902]: audit 2022-04-22T22:00:15.394487+0000 mon.a (mon.0) 13173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:16 smithi115 bash[45495]: cluster 2022-04-22T22:00:14.551295+0000 mgr.y (mgr.34107) 9354 : cluster [DBG] pgmap v10165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:16 smithi115 bash[45495]: audit 2022-04-22T22:00:15.394487+0000 mon.a (mon.0) 13173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:16 smithi115 bash[46934]: cluster 2022-04-22T22:00:14.551295+0000 mgr.y (mgr.34107) 9354 : cluster [DBG] pgmap v10165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:16 smithi115 bash[46934]: audit 2022-04-22T22:00:15.394487+0000 mon.a (mon.0) 13173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:17 smithi121 bash[36902]: audit 2022-04-22T22:00:16.432693+0000 mon.a (mon.0) 13174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:00:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:17 smithi121 bash[36902]: audit 2022-04-22T22:00:16.433430+0000 mon.a (mon.0) 13175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:00:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:17 smithi121 bash[36902]: audit 2022-04-22T22:00:16.506045+0000 mon.a (mon.0) 13176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:17 smithi121 bash[36902]: audit 2022-04-22T22:00:16.816856+0000 mon.a (mon.0) 13177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[45495]: audit 2022-04-22T22:00:16.432693+0000 mon.a (mon.0) 13174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[45495]: audit 2022-04-22T22:00:16.433430+0000 mon.a (mon.0) 13175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[45495]: audit 2022-04-22T22:00:16.506045+0000 mon.a (mon.0) 13176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[45495]: audit 2022-04-22T22:00:16.816856+0000 mon.a (mon.0) 13177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[46934]: audit 2022-04-22T22:00:16.432693+0000 mon.a (mon.0) 13174 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[46934]: audit 2022-04-22T22:00:16.433430+0000 mon.a (mon.0) 13175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:00:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[46934]: audit 2022-04-22T22:00:16.506045+0000 mon.a (mon.0) 13176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:17 smithi115 bash[46934]: audit 2022-04-22T22:00:16.816856+0000 mon.a (mon.0) 13177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:18 smithi121 bash[36902]: cluster 2022-04-22T22:00:16.552322+0000 mgr.y (mgr.34107) 9355 : cluster [DBG] pgmap v10166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:18 smithi115 bash[45495]: cluster 2022-04-22T22:00:16.552322+0000 mgr.y (mgr.34107) 9355 : cluster [DBG] pgmap v10166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:18 smithi115 bash[46934]: cluster 2022-04-22T22:00:16.552322+0000 mgr.y (mgr.34107) 9355 : cluster [DBG] pgmap v10166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:20 smithi121 bash[36902]: cluster 2022-04-22T22:00:18.553619+0000 mgr.y (mgr.34107) 9356 : cluster [DBG] pgmap v10167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:20.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:20 smithi115 bash[46934]: cluster 2022-04-22T22:00:18.553619+0000 mgr.y (mgr.34107) 9356 : cluster [DBG] pgmap v10167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:20 smithi115 bash[45495]: cluster 2022-04-22T22:00:18.553619+0000 mgr.y (mgr.34107) 9356 : cluster [DBG] pgmap v10167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:21 smithi121 bash[36902]: audit 2022-04-22T22:00:20.397725+0000 mon.a (mon.0) 13178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:21 smithi115 bash[45495]: audit 2022-04-22T22:00:20.397725+0000 mon.a (mon.0) 13178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:21 smithi115 bash[46934]: audit 2022-04-22T22:00:20.397725+0000 mon.a (mon.0) 13178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:22.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[46934]: cluster 2022-04-22T22:00:20.554699+0000 mgr.y (mgr.34107) 9357 : cluster [DBG] pgmap v10168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:22.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[46934]: audit 2022-04-22T22:00:21.509711+0000 mon.a (mon.0) 13179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:22.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[46934]: audit 2022-04-22T22:00:21.819884+0000 mon.a (mon.0) 13180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:22.700 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[45495]: cluster 2022-04-22T22:00:20.554699+0000 mgr.y (mgr.34107) 9357 : cluster [DBG] pgmap v10168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:22.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[45495]: audit 2022-04-22T22:00:21.509711+0000 mon.a (mon.0) 13179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:22.701 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[45495]: audit 2022-04-22T22:00:21.819884+0000 mon.a (mon.0) 13180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:22 smithi121 bash[36902]: cluster 2022-04-22T22:00:20.554699+0000 mgr.y (mgr.34107) 9357 : cluster [DBG] pgmap v10168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:22 smithi121 bash[36902]: audit 2022-04-22T22:00:21.509711+0000 mon.a (mon.0) 13179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:22 smithi121 bash[36902]: audit 2022-04-22T22:00:21.819884+0000 mon.a (mon.0) 13180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:22.697Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:23.426 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[45495]: audit 2022-04-22T22:00:23.098586+0000 mon.a (mon.0) 13181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:00:23.426 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[45495]: audit 2022-04-22T22:00:23.101632+0000 mon.a (mon.0) 13182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:00:23.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[45495]: audit 2022-04-22T22:00:23.103521+0000 mon.a (mon.0) 13183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:00:23.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[46934]: audit 2022-04-22T22:00:23.098586+0000 mon.a (mon.0) 13181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:00:23.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[46934]: audit 2022-04-22T22:00:23.101632+0000 mon.a (mon.0) 13182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:00:23.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:23 smithi115 bash[46934]: audit 2022-04-22T22:00:23.103521+0000 mon.a (mon.0) 13183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:00:23.453 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:23 smithi121 bash[36902]: audit 2022-04-22T22:00:23.098586+0000 mon.a (mon.0) 13181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:00:23.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:23 smithi121 bash[36902]: audit 2022-04-22T22:00:23.101632+0000 mon.a (mon.0) 13182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:00:23.454 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:23 smithi121 bash[36902]: audit 2022-04-22T22:00:23.103521+0000 mon.a (mon.0) 13183 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:00:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:24 smithi121 bash[36902]: cluster 2022-04-22T22:00:22.555807+0000 mgr.y (mgr.34107) 9358 : cluster [DBG] pgmap v10169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:24 smithi121 bash[36902]: audit 2022-04-22T22:00:23.465395+0000 mon.a (mon.0) 13184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:00:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:24.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:24 smithi115 bash[45495]: cluster 2022-04-22T22:00:22.555807+0000 mgr.y (mgr.34107) 9358 : cluster [DBG] pgmap v10169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:24.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:24 smithi115 bash[45495]: audit 2022-04-22T22:00:23.465395+0000 mon.a (mon.0) 13184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:00:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:24 smithi115 bash[46934]: cluster 2022-04-22T22:00:22.555807+0000 mgr.y (mgr.34107) 9358 : cluster [DBG] pgmap v10169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:24 smithi115 bash[46934]: audit 2022-04-22T22:00:23.465395+0000 mon.a (mon.0) 13184 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:00:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:25 smithi121 bash[36902]: cluster 2022-04-22T22:00:24.556695+0000 mgr.y (mgr.34107) 9359 : cluster [DBG] pgmap v10170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:25 smithi115 bash[45495]: cluster 2022-04-22T22:00:24.556695+0000 mgr.y (mgr.34107) 9359 : cluster [DBG] pgmap v10170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:25 smithi115 bash[46934]: cluster 2022-04-22T22:00:24.556695+0000 mgr.y (mgr.34107) 9359 : cluster [DBG] pgmap v10170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:26 smithi121 bash[36902]: audit 2022-04-22T22:00:25.400649+0000 mon.a (mon.0) 13185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:26 smithi115 bash[45495]: audit 2022-04-22T22:00:25.400649+0000 mon.a (mon.0) 13185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:26 smithi115 bash[46934]: audit 2022-04-22T22:00:25.400649+0000 mon.a (mon.0) 13185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:27.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:27 smithi121 bash[36902]: audit 2022-04-22T22:00:26.513469+0000 mon.a (mon.0) 13186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:27 smithi121 bash[36902]: cluster 2022-04-22T22:00:26.557637+0000 mgr.y (mgr.34107) 9360 : cluster [DBG] pgmap v10171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:27 smithi121 bash[36902]: audit 2022-04-22T22:00:26.822931+0000 mon.a (mon.0) 13187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[45495]: audit 2022-04-22T22:00:26.513469+0000 mon.a (mon.0) 13186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[45495]: cluster 2022-04-22T22:00:26.557637+0000 mgr.y (mgr.34107) 9360 : cluster [DBG] pgmap v10171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[45495]: audit 2022-04-22T22:00:26.822931+0000 mon.a (mon.0) 13187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[46934]: audit 2022-04-22T22:00:26.513469+0000 mon.a (mon.0) 13186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[46934]: cluster 2022-04-22T22:00:26.557637+0000 mgr.y (mgr.34107) 9360 : cluster [DBG] pgmap v10171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:27 smithi115 bash[46934]: audit 2022-04-22T22:00:26.822931+0000 mon.a (mon.0) 13187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:29 smithi121 bash[36902]: cluster 2022-04-22T22:00:28.558769+0000 mgr.y (mgr.34107) 9361 : cluster [DBG] pgmap v10172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:29 smithi115 bash[45495]: cluster 2022-04-22T22:00:28.558769+0000 mgr.y (mgr.34107) 9361 : cluster [DBG] pgmap v10172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:29 smithi115 bash[46934]: cluster 2022-04-22T22:00:28.558769+0000 mgr.y (mgr.34107) 9361 : cluster [DBG] pgmap v10172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:30 smithi121 bash[36902]: audit 2022-04-22T22:00:30.403810+0000 mon.a (mon.0) 13188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:30 smithi115 bash[45495]: audit 2022-04-22T22:00:30.403810+0000 mon.a (mon.0) 13188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:31.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:30 smithi115 bash[46934]: audit 2022-04-22T22:00:30.403810+0000 mon.a (mon.0) 13188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:31 smithi121 bash[36902]: cluster 2022-04-22T22:00:30.559819+0000 mgr.y (mgr.34107) 9362 : cluster [DBG] pgmap v10173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:31 smithi121 bash[36902]: audit 2022-04-22T22:00:31.517191+0000 mon.a (mon.0) 13189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:31 smithi121 bash[36902]: audit 2022-04-22T22:00:31.826060+0000 mon.a (mon.0) 13190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[45495]: cluster 2022-04-22T22:00:30.559819+0000 mgr.y (mgr.34107) 9362 : cluster [DBG] pgmap v10173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:32.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[45495]: audit 2022-04-22T22:00:31.517191+0000 mon.a (mon.0) 13189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:32.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[45495]: audit 2022-04-22T22:00:31.826060+0000 mon.a (mon.0) 13190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[46934]: cluster 2022-04-22T22:00:30.559819+0000 mgr.y (mgr.34107) 9362 : cluster [DBG] pgmap v10173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[46934]: audit 2022-04-22T22:00:31.517191+0000 mon.a (mon.0) 13189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:31 smithi115 bash[46934]: audit 2022-04-22T22:00:31.826060+0000 mon.a (mon.0) 13190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:32.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:33.550 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:00:33.899 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:00:33.899 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:00:33.899 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:00:33.899 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:00:33.899 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:00:33.900 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:00:33.900 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:00:33.900 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:00:33.900 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:00:33.900 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:00:33.901 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:00:33.902 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:00:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:33 smithi121 bash[36902]: cluster 2022-04-22T22:00:32.561104+0000 mgr.y (mgr.34107) 9363 : cluster [DBG] pgmap v10174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:33 smithi115 bash[45495]: cluster 2022-04-22T22:00:32.561104+0000 mgr.y (mgr.34107) 9363 : cluster [DBG] pgmap v10174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:33 smithi115 bash[46934]: cluster 2022-04-22T22:00:32.561104+0000 mgr.y (mgr.34107) 9363 : cluster [DBG] pgmap v10174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:34.299 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:00:34.299 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:00:34.299 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:00:34.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:34.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:00:34.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:00:34.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:34.300 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:00:34.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:00:34.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:34.301 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:00:34.301 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:00:34.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:00:34.301 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:00:34.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:00:34.302 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:00:34.302 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:00:35.002 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:34.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:35 smithi121 bash[36902]: audit 2022-04-22T22:00:33.530502+0000 mgr.y (mgr.34107) 9364 : audit [DBG] from='client.55003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:35 smithi121 bash[36902]: audit 2022-04-22T22:00:33.874785+0000 mgr.y (mgr.34107) 9365 : audit [DBG] from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:35 smithi121 bash[36902]: audit 2022-04-22T22:00:34.295350+0000 mon.a (mon.0) 13191 : audit [DBG] from='client.? 172.21.15.115:0/610870853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:34 smithi115 bash[45495]: audit 2022-04-22T22:00:33.530502+0000 mgr.y (mgr.34107) 9364 : audit [DBG] from='client.55003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:34 smithi115 bash[45495]: audit 2022-04-22T22:00:33.874785+0000 mgr.y (mgr.34107) 9365 : audit [DBG] from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:34 smithi115 bash[45495]: audit 2022-04-22T22:00:34.295350+0000 mon.a (mon.0) 13191 : audit [DBG] from='client.? 172.21.15.115:0/610870853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:35 smithi115 bash[46934]: audit 2022-04-22T22:00:33.530502+0000 mgr.y (mgr.34107) 9364 : audit [DBG] from='client.55003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:35 smithi115 bash[46934]: audit 2022-04-22T22:00:33.874785+0000 mgr.y (mgr.34107) 9365 : audit [DBG] from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:00:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:35 smithi115 bash[46934]: audit 2022-04-22T22:00:34.295350+0000 mon.a (mon.0) 13191 : audit [DBG] from='client.? 172.21.15.115:0/610870853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:00:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:36 smithi121 bash[36902]: cluster 2022-04-22T22:00:34.562338+0000 mgr.y (mgr.34107) 9366 : cluster [DBG] pgmap v10175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:36 smithi121 bash[36902]: audit 2022-04-22T22:00:35.407012+0000 mon.a (mon.0) 13192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:36 smithi115 bash[45495]: cluster 2022-04-22T22:00:34.562338+0000 mgr.y (mgr.34107) 9366 : cluster [DBG] pgmap v10175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:36 smithi115 bash[45495]: audit 2022-04-22T22:00:35.407012+0000 mon.a (mon.0) 13192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:36 smithi115 bash[46934]: cluster 2022-04-22T22:00:34.562338+0000 mgr.y (mgr.34107) 9366 : cluster [DBG] pgmap v10175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:36 smithi115 bash[46934]: audit 2022-04-22T22:00:35.407012+0000 mon.a (mon.0) 13192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:37 smithi121 bash[36902]: audit 2022-04-22T22:00:36.520821+0000 mon.a (mon.0) 13193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:37 smithi121 bash[36902]: audit 2022-04-22T22:00:36.828908+0000 mon.a (mon.0) 13194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:37 smithi115 bash[45495]: audit 2022-04-22T22:00:36.520821+0000 mon.a (mon.0) 13193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:37 smithi115 bash[45495]: audit 2022-04-22T22:00:36.828908+0000 mon.a (mon.0) 13194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:37 smithi115 bash[46934]: audit 2022-04-22T22:00:36.520821+0000 mon.a (mon.0) 13193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:37 smithi115 bash[46934]: audit 2022-04-22T22:00:36.828908+0000 mon.a (mon.0) 13194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:38 smithi115 bash[45495]: cluster 2022-04-22T22:00:36.563593+0000 mgr.y (mgr.34107) 9367 : cluster [DBG] pgmap v10176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:38 smithi115 bash[46934]: cluster 2022-04-22T22:00:36.563593+0000 mgr.y (mgr.34107) 9367 : cluster [DBG] pgmap v10176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:38 smithi121 bash[36902]: cluster 2022-04-22T22:00:36.563593+0000 mgr.y (mgr.34107) 9367 : cluster [DBG] pgmap v10176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:40 smithi115 bash[45495]: cluster 2022-04-22T22:00:38.564827+0000 mgr.y (mgr.34107) 9368 : cluster [DBG] pgmap v10177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:40 smithi115 bash[46934]: cluster 2022-04-22T22:00:38.564827+0000 mgr.y (mgr.34107) 9368 : cluster [DBG] pgmap v10177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:40 smithi121 bash[36902]: cluster 2022-04-22T22:00:38.564827+0000 mgr.y (mgr.34107) 9368 : cluster [DBG] pgmap v10177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:41 smithi115 bash[45495]: audit 2022-04-22T22:00:40.410329+0000 mon.a (mon.0) 13195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:41 smithi115 bash[46934]: audit 2022-04-22T22:00:40.410329+0000 mon.a (mon.0) 13195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:41 smithi121 bash[36902]: audit 2022-04-22T22:00:40.410329+0000 mon.a (mon.0) 13195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[45495]: cluster 2022-04-22T22:00:40.565934+0000 mgr.y (mgr.34107) 9369 : cluster [DBG] pgmap v10178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[45495]: audit 2022-04-22T22:00:41.524574+0000 mon.a (mon.0) 13196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[45495]: audit 2022-04-22T22:00:41.831863+0000 mon.a (mon.0) 13197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[46934]: cluster 2022-04-22T22:00:40.565934+0000 mgr.y (mgr.34107) 9369 : cluster [DBG] pgmap v10178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[46934]: audit 2022-04-22T22:00:41.524574+0000 mon.a (mon.0) 13196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[46934]: audit 2022-04-22T22:00:41.831863+0000 mon.a (mon.0) 13197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:42 smithi121 bash[36902]: cluster 2022-04-22T22:00:40.565934+0000 mgr.y (mgr.34107) 9369 : cluster [DBG] pgmap v10178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:42 smithi121 bash[36902]: audit 2022-04-22T22:00:41.524574+0000 mon.a (mon.0) 13196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:42 smithi121 bash[36902]: audit 2022-04-22T22:00:41.831863+0000 mon.a (mon.0) 13197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:42.697Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:44 smithi115 bash[46934]: cluster 2022-04-22T22:00:42.567048+0000 mgr.y (mgr.34107) 9370 : cluster [DBG] pgmap v10179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:44.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:44 smithi115 bash[45495]: cluster 2022-04-22T22:00:42.567048+0000 mgr.y (mgr.34107) 9370 : cluster [DBG] pgmap v10179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:44 smithi121 bash[36902]: cluster 2022-04-22T22:00:42.567048+0000 mgr.y (mgr.34107) 9370 : cluster [DBG] pgmap v10179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:44.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:46 smithi121 bash[36902]: cluster 2022-04-22T22:00:44.568211+0000 mgr.y (mgr.34107) 9371 : cluster [DBG] pgmap v10180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:46 smithi121 bash[36902]: audit 2022-04-22T22:00:45.413646+0000 mon.a (mon.0) 13198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:46 smithi115 bash[45495]: cluster 2022-04-22T22:00:44.568211+0000 mgr.y (mgr.34107) 9371 : cluster [DBG] pgmap v10180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:46 smithi115 bash[45495]: audit 2022-04-22T22:00:45.413646+0000 mon.a (mon.0) 13198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:46 smithi115 bash[46934]: cluster 2022-04-22T22:00:44.568211+0000 mgr.y (mgr.34107) 9371 : cluster [DBG] pgmap v10180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:46.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:46 smithi115 bash[46934]: audit 2022-04-22T22:00:45.413646+0000 mon.a (mon.0) 13198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:47 smithi121 bash[36902]: audit 2022-04-22T22:00:46.528279+0000 mon.a (mon.0) 13199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:47 smithi121 bash[36902]: audit 2022-04-22T22:00:46.834920+0000 mon.a (mon.0) 13200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:47 smithi115 bash[45495]: audit 2022-04-22T22:00:46.528279+0000 mon.a (mon.0) 13199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:47 smithi115 bash[45495]: audit 2022-04-22T22:00:46.834920+0000 mon.a (mon.0) 13200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:47.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:47 smithi115 bash[46934]: audit 2022-04-22T22:00:46.528279+0000 mon.a (mon.0) 13199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:47.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:47 smithi115 bash[46934]: audit 2022-04-22T22:00:46.834920+0000 mon.a (mon.0) 13200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:48 smithi121 bash[36902]: cluster 2022-04-22T22:00:46.569234+0000 mgr.y (mgr.34107) 9372 : cluster [DBG] pgmap v10181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:48 smithi115 bash[45495]: cluster 2022-04-22T22:00:46.569234+0000 mgr.y (mgr.34107) 9372 : cluster [DBG] pgmap v10181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:48 smithi115 bash[46934]: cluster 2022-04-22T22:00:46.569234+0000 mgr.y (mgr.34107) 9372 : cluster [DBG] pgmap v10181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:50.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:50 smithi121 bash[36902]: cluster 2022-04-22T22:00:48.570366+0000 mgr.y (mgr.34107) 9373 : cluster [DBG] pgmap v10182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:50 smithi115 bash[45495]: cluster 2022-04-22T22:00:48.570366+0000 mgr.y (mgr.34107) 9373 : cluster [DBG] pgmap v10182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:50 smithi115 bash[46934]: cluster 2022-04-22T22:00:48.570366+0000 mgr.y (mgr.34107) 9373 : cluster [DBG] pgmap v10182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:51 smithi121 bash[36902]: audit 2022-04-22T22:00:50.417065+0000 mon.a (mon.0) 13201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:51 smithi115 bash[45495]: audit 2022-04-22T22:00:50.417065+0000 mon.a (mon.0) 13201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:51 smithi115 bash[46934]: audit 2022-04-22T22:00:50.417065+0000 mon.a (mon.0) 13201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:52 smithi121 bash[36902]: cluster 2022-04-22T22:00:50.571442+0000 mgr.y (mgr.34107) 9374 : cluster [DBG] pgmap v10183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:52 smithi121 bash[36902]: audit 2022-04-22T22:00:51.532033+0000 mon.a (mon.0) 13202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:52 smithi121 bash[36902]: audit 2022-04-22T22:00:51.837744+0000 mon.a (mon.0) 13203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[45495]: cluster 2022-04-22T22:00:50.571442+0000 mgr.y (mgr.34107) 9374 : cluster [DBG] pgmap v10183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[45495]: audit 2022-04-22T22:00:51.532033+0000 mon.a (mon.0) 13202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[45495]: audit 2022-04-22T22:00:51.837744+0000 mon.a (mon.0) 13203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[46934]: cluster 2022-04-22T22:00:50.571442+0000 mgr.y (mgr.34107) 9374 : cluster [DBG] pgmap v10183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[46934]: audit 2022-04-22T22:00:51.532033+0000 mon.a (mon.0) 13202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[46934]: audit 2022-04-22T22:00:51.837744+0000 mon.a (mon.0) 13203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:52.700Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:54 smithi121 bash[36902]: cluster 2022-04-22T22:00:52.572568+0000 mgr.y (mgr.34107) 9375 : cluster [DBG] pgmap v10184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:54 smithi115 bash[45495]: cluster 2022-04-22T22:00:52.572568+0000 mgr.y (mgr.34107) 9375 : cluster [DBG] pgmap v10184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:54 smithi115 bash[46934]: cluster 2022-04-22T22:00:52.572568+0000 mgr.y (mgr.34107) 9375 : cluster [DBG] pgmap v10184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:00:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:00:54.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:00:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:56 smithi121 bash[36902]: cluster 2022-04-22T22:00:54.573618+0000 mgr.y (mgr.34107) 9376 : cluster [DBG] pgmap v10185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:56 smithi121 bash[36902]: audit 2022-04-22T22:00:55.420318+0000 mon.a (mon.0) 13204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:56 smithi115 bash[45495]: cluster 2022-04-22T22:00:54.573618+0000 mgr.y (mgr.34107) 9376 : cluster [DBG] pgmap v10185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:56 smithi115 bash[45495]: audit 2022-04-22T22:00:55.420318+0000 mon.a (mon.0) 13204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:56 smithi115 bash[46934]: cluster 2022-04-22T22:00:54.573618+0000 mgr.y (mgr.34107) 9376 : cluster [DBG] pgmap v10185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:56 smithi115 bash[46934]: audit 2022-04-22T22:00:55.420318+0000 mon.a (mon.0) 13204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:00:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:57 smithi121 bash[36902]: audit 2022-04-22T22:00:56.535642+0000 mon.a (mon.0) 13205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:57 smithi121 bash[36902]: audit 2022-04-22T22:00:56.840978+0000 mon.a (mon.0) 13206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:57 smithi115 bash[45495]: audit 2022-04-22T22:00:56.535642+0000 mon.a (mon.0) 13205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:57 smithi115 bash[45495]: audit 2022-04-22T22:00:56.840978+0000 mon.a (mon.0) 13206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:57 smithi115 bash[46934]: audit 2022-04-22T22:00:56.535642+0000 mon.a (mon.0) 13205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:00:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:57 smithi115 bash[46934]: audit 2022-04-22T22:00:56.840978+0000 mon.a (mon.0) 13206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:00:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:00:58 smithi121 bash[36902]: cluster 2022-04-22T22:00:56.574602+0000 mgr.y (mgr.34107) 9377 : cluster [DBG] pgmap v10186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:00:58 smithi115 bash[45495]: cluster 2022-04-22T22:00:56.574602+0000 mgr.y (mgr.34107) 9377 : cluster [DBG] pgmap v10186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:00:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:00:58 smithi115 bash[46934]: cluster 2022-04-22T22:00:56.574602+0000 mgr.y (mgr.34107) 9377 : cluster [DBG] pgmap v10186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:00 smithi121 bash[36902]: cluster 2022-04-22T22:00:58.575747+0000 mgr.y (mgr.34107) 9378 : cluster [DBG] pgmap v10187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:00 smithi115 bash[45495]: cluster 2022-04-22T22:00:58.575747+0000 mgr.y (mgr.34107) 9378 : cluster [DBG] pgmap v10187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:00 smithi115 bash[46934]: cluster 2022-04-22T22:00:58.575747+0000 mgr.y (mgr.34107) 9378 : cluster [DBG] pgmap v10187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:01 smithi121 bash[36902]: audit 2022-04-22T22:01:00.423435+0000 mon.a (mon.0) 13207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:01 smithi115 bash[45495]: audit 2022-04-22T22:01:00.423435+0000 mon.a (mon.0) 13207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:01 smithi115 bash[46934]: audit 2022-04-22T22:01:00.423435+0000 mon.a (mon.0) 13207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:02 smithi121 bash[36902]: cluster 2022-04-22T22:01:00.576810+0000 mgr.y (mgr.34107) 9379 : cluster [DBG] pgmap v10188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:02 smithi121 bash[36902]: audit 2022-04-22T22:01:01.539315+0000 mon.a (mon.0) 13208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:02 smithi121 bash[36902]: audit 2022-04-22T22:01:01.843893+0000 mon.a (mon.0) 13209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[45495]: cluster 2022-04-22T22:01:00.576810+0000 mgr.y (mgr.34107) 9379 : cluster [DBG] pgmap v10188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[45495]: audit 2022-04-22T22:01:01.539315+0000 mon.a (mon.0) 13208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[45495]: audit 2022-04-22T22:01:01.843893+0000 mon.a (mon.0) 13209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[46934]: cluster 2022-04-22T22:01:00.576810+0000 mgr.y (mgr.34107) 9379 : cluster [DBG] pgmap v10188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[46934]: audit 2022-04-22T22:01:01.539315+0000 mon.a (mon.0) 13208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[46934]: audit 2022-04-22T22:01:01.843893+0000 mon.a (mon.0) 13209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:02.700Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:04 smithi121 bash[36902]: cluster 2022-04-22T22:01:02.577880+0000 mgr.y (mgr.34107) 9380 : cluster [DBG] pgmap v10189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:04 smithi115 bash[46934]: cluster 2022-04-22T22:01:02.577880+0000 mgr.y (mgr.34107) 9380 : cluster [DBG] pgmap v10189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:04 smithi115 bash[45495]: cluster 2022-04-22T22:01:02.577880+0000 mgr.y (mgr.34107) 9380 : cluster [DBG] pgmap v10189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:04.671 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:01:05.025 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:01:05.026 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:01:05.026 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:01:05.026 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:01:05.026 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:01:05.026 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:01:05.027 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:01:05.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:01:05.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:01:05.027 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:01:05.028 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:01:05.028 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:01:05.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:01:05.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:01:05.028 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:01:05.029 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:01:05.029 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:01:05.029 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:01:05.029 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:01:05.029 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:01:05.030 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:01:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:04.553Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:01:05.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:01:05.431 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:06 smithi121 bash[36902]: cluster 2022-04-22T22:01:04.578876+0000 mgr.y (mgr.34107) 9381 : cluster [DBG] pgmap v10190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:06 smithi121 bash[36902]: audit 2022-04-22T22:01:04.650768+0000 mgr.y (mgr.34107) 9382 : audit [DBG] from='client.55021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:06 smithi121 bash[36902]: audit 2022-04-22T22:01:05.001956+0000 mgr.y (mgr.34107) 9383 : audit [DBG] from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:06 smithi121 bash[36902]: audit 2022-04-22T22:01:05.426072+0000 mon.c (mon.1) 326 : audit [DBG] from='client.? 172.21.15.115:0/1722165204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:06 smithi121 bash[36902]: audit 2022-04-22T22:01:05.426379+0000 mon.a (mon.0) 13210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[45495]: cluster 2022-04-22T22:01:04.578876+0000 mgr.y (mgr.34107) 9381 : cluster [DBG] pgmap v10190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[45495]: audit 2022-04-22T22:01:04.650768+0000 mgr.y (mgr.34107) 9382 : audit [DBG] from='client.55021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[45495]: audit 2022-04-22T22:01:05.001956+0000 mgr.y (mgr.34107) 9383 : audit [DBG] from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[45495]: audit 2022-04-22T22:01:05.426072+0000 mon.c (mon.1) 326 : audit [DBG] from='client.? 172.21.15.115:0/1722165204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[45495]: audit 2022-04-22T22:01:05.426379+0000 mon.a (mon.0) 13210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[46934]: cluster 2022-04-22T22:01:04.578876+0000 mgr.y (mgr.34107) 9381 : cluster [DBG] pgmap v10190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[46934]: audit 2022-04-22T22:01:04.650768+0000 mgr.y (mgr.34107) 9382 : audit [DBG] from='client.55021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[46934]: audit 2022-04-22T22:01:05.001956+0000 mgr.y (mgr.34107) 9383 : audit [DBG] from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[46934]: audit 2022-04-22T22:01:05.426072+0000 mon.c (mon.1) 326 : audit [DBG] from='client.? 172.21.15.115:0/1722165204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:06 smithi115 bash[46934]: audit 2022-04-22T22:01:05.426379+0000 mon.a (mon.0) 13210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:07 smithi121 bash[36902]: audit 2022-04-22T22:01:06.543023+0000 mon.a (mon.0) 13211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:07 smithi121 bash[36902]: audit 2022-04-22T22:01:06.846698+0000 mon.a (mon.0) 13212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:07 smithi115 bash[45495]: audit 2022-04-22T22:01:06.543023+0000 mon.a (mon.0) 13211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:07 smithi115 bash[45495]: audit 2022-04-22T22:01:06.846698+0000 mon.a (mon.0) 13212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:07 smithi115 bash[46934]: audit 2022-04-22T22:01:06.543023+0000 mon.a (mon.0) 13211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:07 smithi115 bash[46934]: audit 2022-04-22T22:01:06.846698+0000 mon.a (mon.0) 13212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:08 smithi121 bash[36902]: cluster 2022-04-22T22:01:06.579964+0000 mgr.y (mgr.34107) 9384 : cluster [DBG] pgmap v10191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:08 smithi115 bash[45495]: cluster 2022-04-22T22:01:06.579964+0000 mgr.y (mgr.34107) 9384 : cluster [DBG] pgmap v10191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:08 smithi115 bash[46934]: cluster 2022-04-22T22:01:06.579964+0000 mgr.y (mgr.34107) 9384 : cluster [DBG] pgmap v10191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:10 smithi121 bash[36902]: cluster 2022-04-22T22:01:08.581222+0000 mgr.y (mgr.34107) 9385 : cluster [DBG] pgmap v10192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:10 smithi115 bash[45495]: cluster 2022-04-22T22:01:08.581222+0000 mgr.y (mgr.34107) 9385 : cluster [DBG] pgmap v10192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:10 smithi115 bash[46934]: cluster 2022-04-22T22:01:08.581222+0000 mgr.y (mgr.34107) 9385 : cluster [DBG] pgmap v10192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:11 smithi121 bash[36902]: audit 2022-04-22T22:01:10.429557+0000 mon.a (mon.0) 13213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:11.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:11 smithi115 bash[46934]: audit 2022-04-22T22:01:10.429557+0000 mon.a (mon.0) 13213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:11 smithi115 bash[45495]: audit 2022-04-22T22:01:10.429557+0000 mon.a (mon.0) 13213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:12 smithi121 bash[36902]: cluster 2022-04-22T22:01:10.582289+0000 mgr.y (mgr.34107) 9386 : cluster [DBG] pgmap v10193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:12 smithi121 bash[36902]: audit 2022-04-22T22:01:11.546593+0000 mon.a (mon.0) 13214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:12 smithi121 bash[36902]: audit 2022-04-22T22:01:11.849232+0000 mon.a (mon.0) 13215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[45495]: cluster 2022-04-22T22:01:10.582289+0000 mgr.y (mgr.34107) 9386 : cluster [DBG] pgmap v10193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[45495]: audit 2022-04-22T22:01:11.546593+0000 mon.a (mon.0) 13214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[45495]: audit 2022-04-22T22:01:11.849232+0000 mon.a (mon.0) 13215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[46934]: cluster 2022-04-22T22:01:10.582289+0000 mgr.y (mgr.34107) 9386 : cluster [DBG] pgmap v10193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[46934]: audit 2022-04-22T22:01:11.546593+0000 mon.a (mon.0) 13214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[46934]: audit 2022-04-22T22:01:11.849232+0000 mon.a (mon.0) 13215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:12.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:14 smithi121 bash[36902]: cluster 2022-04-22T22:01:12.583317+0000 mgr.y (mgr.34107) 9387 : cluster [DBG] pgmap v10194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:14 smithi115 bash[45495]: cluster 2022-04-22T22:01:12.583317+0000 mgr.y (mgr.34107) 9387 : cluster [DBG] pgmap v10194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:14 smithi115 bash[46934]: cluster 2022-04-22T22:01:12.583317+0000 mgr.y (mgr.34107) 9387 : cluster [DBG] pgmap v10194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:14.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:16 smithi121 bash[36902]: cluster 2022-04-22T22:01:14.584392+0000 mgr.y (mgr.34107) 9388 : cluster [DBG] pgmap v10195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:16 smithi121 bash[36902]: audit 2022-04-22T22:01:15.432784+0000 mon.a (mon.0) 13216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:16 smithi115 bash[45495]: cluster 2022-04-22T22:01:14.584392+0000 mgr.y (mgr.34107) 9388 : cluster [DBG] pgmap v10195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:16 smithi115 bash[45495]: audit 2022-04-22T22:01:15.432784+0000 mon.a (mon.0) 13216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:16 smithi115 bash[46934]: cluster 2022-04-22T22:01:14.584392+0000 mgr.y (mgr.34107) 9388 : cluster [DBG] pgmap v10195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:16 smithi115 bash[46934]: audit 2022-04-22T22:01:15.432784+0000 mon.a (mon.0) 13216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:17 smithi121 bash[36902]: audit 2022-04-22T22:01:16.445126+0000 mon.a (mon.0) 13217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:01:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:17 smithi121 bash[36902]: audit 2022-04-22T22:01:16.445681+0000 mon.a (mon.0) 13218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:01:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:17 smithi121 bash[36902]: audit 2022-04-22T22:01:16.550391+0000 mon.a (mon.0) 13219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:17 smithi121 bash[36902]: audit 2022-04-22T22:01:16.851616+0000 mon.a (mon.0) 13220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[45495]: audit 2022-04-22T22:01:16.445126+0000 mon.a (mon.0) 13217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[45495]: audit 2022-04-22T22:01:16.445681+0000 mon.a (mon.0) 13218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[45495]: audit 2022-04-22T22:01:16.550391+0000 mon.a (mon.0) 13219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[45495]: audit 2022-04-22T22:01:16.851616+0000 mon.a (mon.0) 13220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[46934]: audit 2022-04-22T22:01:16.445126+0000 mon.a (mon.0) 13217 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[46934]: audit 2022-04-22T22:01:16.445681+0000 mon.a (mon.0) 13218 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:01:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[46934]: audit 2022-04-22T22:01:16.550391+0000 mon.a (mon.0) 13219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:17.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:17 smithi115 bash[46934]: audit 2022-04-22T22:01:16.851616+0000 mon.a (mon.0) 13220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:18 smithi121 bash[36902]: cluster 2022-04-22T22:01:16.585426+0000 mgr.y (mgr.34107) 9389 : cluster [DBG] pgmap v10196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:18 smithi115 bash[45495]: cluster 2022-04-22T22:01:16.585426+0000 mgr.y (mgr.34107) 9389 : cluster [DBG] pgmap v10196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:18 smithi115 bash[46934]: cluster 2022-04-22T22:01:16.585426+0000 mgr.y (mgr.34107) 9389 : cluster [DBG] pgmap v10196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:20 smithi121 bash[36902]: cluster 2022-04-22T22:01:18.586534+0000 mgr.y (mgr.34107) 9390 : cluster [DBG] pgmap v10197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:20 smithi115 bash[45495]: cluster 2022-04-22T22:01:18.586534+0000 mgr.y (mgr.34107) 9390 : cluster [DBG] pgmap v10197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:20 smithi115 bash[46934]: cluster 2022-04-22T22:01:18.586534+0000 mgr.y (mgr.34107) 9390 : cluster [DBG] pgmap v10197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:21 smithi121 bash[36902]: audit 2022-04-22T22:01:20.435842+0000 mon.a (mon.0) 13221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:21 smithi115 bash[45495]: audit 2022-04-22T22:01:20.435842+0000 mon.a (mon.0) 13221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:21 smithi115 bash[46934]: audit 2022-04-22T22:01:20.435842+0000 mon.a (mon.0) 13221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:22 smithi121 bash[36902]: cluster 2022-04-22T22:01:20.587610+0000 mgr.y (mgr.34107) 9391 : cluster [DBG] pgmap v10198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:22 smithi121 bash[36902]: audit 2022-04-22T22:01:21.553948+0000 mon.a (mon.0) 13222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:22 smithi121 bash[36902]: audit 2022-04-22T22:01:21.854583+0000 mon.a (mon.0) 13223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[45495]: cluster 2022-04-22T22:01:20.587610+0000 mgr.y (mgr.34107) 9391 : cluster [DBG] pgmap v10198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[45495]: audit 2022-04-22T22:01:21.553948+0000 mon.a (mon.0) 13222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[45495]: audit 2022-04-22T22:01:21.854583+0000 mon.a (mon.0) 13223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[46934]: cluster 2022-04-22T22:01:20.587610+0000 mgr.y (mgr.34107) 9391 : cluster [DBG] pgmap v10198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[46934]: audit 2022-04-22T22:01:21.553948+0000 mon.a (mon.0) 13222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[46934]: audit 2022-04-22T22:01:21.854583+0000 mon.a (mon.0) 13223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:22.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:24 smithi121 bash[36902]: cluster 2022-04-22T22:01:22.588699+0000 mgr.y (mgr.34107) 9392 : cluster [DBG] pgmap v10199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:24 smithi121 bash[36902]: audit 2022-04-22T22:01:23.473818+0000 mon.a (mon.0) 13224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:24 smithi121 bash[36902]: audit 2022-04-22T22:01:23.476514+0000 mon.a (mon.0) 13225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:24 smithi121 bash[36902]: audit 2022-04-22T22:01:23.477853+0000 mon.a (mon.0) 13226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:24 smithi121 bash[36902]: audit 2022-04-22T22:01:23.825531+0000 mon.a (mon.0) 13227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:01:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[45495]: cluster 2022-04-22T22:01:22.588699+0000 mgr.y (mgr.34107) 9392 : cluster [DBG] pgmap v10199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[45495]: audit 2022-04-22T22:01:23.473818+0000 mon.a (mon.0) 13224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[45495]: audit 2022-04-22T22:01:23.476514+0000 mon.a (mon.0) 13225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[45495]: audit 2022-04-22T22:01:23.477853+0000 mon.a (mon.0) 13226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[45495]: audit 2022-04-22T22:01:23.825531+0000 mon.a (mon.0) 13227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[46934]: cluster 2022-04-22T22:01:22.588699+0000 mgr.y (mgr.34107) 9392 : cluster [DBG] pgmap v10199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[46934]: audit 2022-04-22T22:01:23.473818+0000 mon.a (mon.0) 13224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:01:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[46934]: audit 2022-04-22T22:01:23.476514+0000 mon.a (mon.0) 13225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:01:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[46934]: audit 2022-04-22T22:01:23.477853+0000 mon.a (mon.0) 13226 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:01:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[46934]: audit 2022-04-22T22:01:23.825531+0000 mon.a (mon.0) 13227 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:01:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:24.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:26 smithi121 bash[36902]: cluster 2022-04-22T22:01:24.589790+0000 mgr.y (mgr.34107) 9393 : cluster [DBG] pgmap v10200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:26 smithi121 bash[36902]: audit 2022-04-22T22:01:25.439023+0000 mon.a (mon.0) 13228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:26 smithi115 bash[45495]: cluster 2022-04-22T22:01:24.589790+0000 mgr.y (mgr.34107) 9393 : cluster [DBG] pgmap v10200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:26 smithi115 bash[45495]: audit 2022-04-22T22:01:25.439023+0000 mon.a (mon.0) 13228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:26 smithi115 bash[46934]: cluster 2022-04-22T22:01:24.589790+0000 mgr.y (mgr.34107) 9393 : cluster [DBG] pgmap v10200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:26 smithi115 bash[46934]: audit 2022-04-22T22:01:25.439023+0000 mon.a (mon.0) 13228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:27 smithi121 bash[36902]: audit 2022-04-22T22:01:26.557692+0000 mon.a (mon.0) 13229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:27.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:27 smithi121 bash[36902]: audit 2022-04-22T22:01:26.857445+0000 mon.a (mon.0) 13230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:27 smithi115 bash[46934]: audit 2022-04-22T22:01:26.557692+0000 mon.a (mon.0) 13229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:27 smithi115 bash[46934]: audit 2022-04-22T22:01:26.857445+0000 mon.a (mon.0) 13230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:27 smithi115 bash[45495]: audit 2022-04-22T22:01:26.557692+0000 mon.a (mon.0) 13229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:27 smithi115 bash[45495]: audit 2022-04-22T22:01:26.857445+0000 mon.a (mon.0) 13230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:28 smithi121 bash[36902]: cluster 2022-04-22T22:01:26.590720+0000 mgr.y (mgr.34107) 9394 : cluster [DBG] pgmap v10201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:28 smithi115 bash[45495]: cluster 2022-04-22T22:01:26.590720+0000 mgr.y (mgr.34107) 9394 : cluster [DBG] pgmap v10201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:28 smithi115 bash[46934]: cluster 2022-04-22T22:01:26.590720+0000 mgr.y (mgr.34107) 9394 : cluster [DBG] pgmap v10201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:30 smithi121 bash[36902]: cluster 2022-04-22T22:01:28.592030+0000 mgr.y (mgr.34107) 9395 : cluster [DBG] pgmap v10202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:30 smithi115 bash[45495]: cluster 2022-04-22T22:01:28.592030+0000 mgr.y (mgr.34107) 9395 : cluster [DBG] pgmap v10202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:30 smithi115 bash[46934]: cluster 2022-04-22T22:01:28.592030+0000 mgr.y (mgr.34107) 9395 : cluster [DBG] pgmap v10202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:31 smithi121 bash[36902]: audit 2022-04-22T22:01:30.442166+0000 mon.a (mon.0) 13231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:31 smithi115 bash[45495]: audit 2022-04-22T22:01:30.442166+0000 mon.a (mon.0) 13231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:31 smithi115 bash[46934]: audit 2022-04-22T22:01:30.442166+0000 mon.a (mon.0) 13231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:32 smithi121 bash[36902]: cluster 2022-04-22T22:01:30.593124+0000 mgr.y (mgr.34107) 9396 : cluster [DBG] pgmap v10203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:32 smithi121 bash[36902]: audit 2022-04-22T22:01:31.561128+0000 mon.a (mon.0) 13232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:32 smithi121 bash[36902]: audit 2022-04-22T22:01:31.860925+0000 mon.a (mon.0) 13233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[45495]: cluster 2022-04-22T22:01:30.593124+0000 mgr.y (mgr.34107) 9396 : cluster [DBG] pgmap v10203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[45495]: audit 2022-04-22T22:01:31.561128+0000 mon.a (mon.0) 13232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:32.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[45495]: audit 2022-04-22T22:01:31.860925+0000 mon.a (mon.0) 13233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[46934]: cluster 2022-04-22T22:01:30.593124+0000 mgr.y (mgr.34107) 9396 : cluster [DBG] pgmap v10203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[46934]: audit 2022-04-22T22:01:31.561128+0000 mon.a (mon.0) 13232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:32.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[46934]: audit 2022-04-22T22:01:31.860925+0000 mon.a (mon.0) 13233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:32.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:34 smithi121 bash[36902]: cluster 2022-04-22T22:01:32.594230+0000 mgr.y (mgr.34107) 9397 : cluster [DBG] pgmap v10204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:34 smithi115 bash[45495]: cluster 2022-04-22T22:01:32.594230+0000 mgr.y (mgr.34107) 9397 : cluster [DBG] pgmap v10204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:34 smithi115 bash[46934]: cluster 2022-04-22T22:01:32.594230+0000 mgr.y (mgr.34107) 9397 : cluster [DBG] pgmap v10204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:34.555Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:35.803 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:01:36.162 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:01:36.184 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:01:36.185 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:01:36.186 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:01:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:36 smithi121 bash[36902]: cluster 2022-04-22T22:01:34.595356+0000 mgr.y (mgr.34107) 9398 : cluster [DBG] pgmap v10205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:36 smithi121 bash[36902]: audit 2022-04-22T22:01:35.445318+0000 mon.a (mon.0) 13234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:36 smithi115 bash[46934]: cluster 2022-04-22T22:01:34.595356+0000 mgr.y (mgr.34107) 9398 : cluster [DBG] pgmap v10205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:36 smithi115 bash[46934]: audit 2022-04-22T22:01:35.445318+0000 mon.a (mon.0) 13234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:36 smithi115 bash[45495]: cluster 2022-04-22T22:01:34.595356+0000 mgr.y (mgr.34107) 9398 : cluster [DBG] pgmap v10205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:36 smithi115 bash[45495]: audit 2022-04-22T22:01:35.445318+0000 mon.a (mon.0) 13234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:36.564 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:01:36.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:01:36.566 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:01:36.567 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:01:36.567 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:01:36.567 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:37 smithi121 bash[36902]: audit 2022-04-22T22:01:35.781688+0000 mgr.y (mgr.34107) 9399 : audit [DBG] from='client.45108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:37 smithi121 bash[36902]: audit 2022-04-22T22:01:36.134185+0000 mgr.y (mgr.34107) 9400 : audit [DBG] from='client.45114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:37 smithi121 bash[36902]: audit 2022-04-22T22:01:36.561142+0000 mon.a (mon.0) 13235 : audit [DBG] from='client.? 172.21.15.115:0/1490739761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:37 smithi121 bash[36902]: audit 2022-04-22T22:01:36.564576+0000 mon.a (mon.0) 13236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:37 smithi121 bash[36902]: audit 2022-04-22T22:01:36.863874+0000 mon.a (mon.0) 13237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[45495]: audit 2022-04-22T22:01:35.781688+0000 mgr.y (mgr.34107) 9399 : audit [DBG] from='client.45108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[45495]: audit 2022-04-22T22:01:36.134185+0000 mgr.y (mgr.34107) 9400 : audit [DBG] from='client.45114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[45495]: audit 2022-04-22T22:01:36.561142+0000 mon.a (mon.0) 13235 : audit [DBG] from='client.? 172.21.15.115:0/1490739761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[45495]: audit 2022-04-22T22:01:36.564576+0000 mon.a (mon.0) 13236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[45495]: audit 2022-04-22T22:01:36.863874+0000 mon.a (mon.0) 13237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[46934]: audit 2022-04-22T22:01:35.781688+0000 mgr.y (mgr.34107) 9399 : audit [DBG] from='client.45108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[46934]: audit 2022-04-22T22:01:36.134185+0000 mgr.y (mgr.34107) 9400 : audit [DBG] from='client.45114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:01:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[46934]: audit 2022-04-22T22:01:36.561142+0000 mon.a (mon.0) 13235 : audit [DBG] from='client.? 172.21.15.115:0/1490739761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:01:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[46934]: audit 2022-04-22T22:01:36.564576+0000 mon.a (mon.0) 13236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:37 smithi115 bash[46934]: audit 2022-04-22T22:01:36.863874+0000 mon.a (mon.0) 13237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:38 smithi121 bash[36902]: cluster 2022-04-22T22:01:36.596381+0000 mgr.y (mgr.34107) 9401 : cluster [DBG] pgmap v10206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:38 smithi115 bash[45495]: cluster 2022-04-22T22:01:36.596381+0000 mgr.y (mgr.34107) 9401 : cluster [DBG] pgmap v10206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:38 smithi115 bash[46934]: cluster 2022-04-22T22:01:36.596381+0000 mgr.y (mgr.34107) 9401 : cluster [DBG] pgmap v10206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:40 smithi121 bash[36902]: cluster 2022-04-22T22:01:38.597583+0000 mgr.y (mgr.34107) 9402 : cluster [DBG] pgmap v10207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:40.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:40 smithi115 bash[46934]: cluster 2022-04-22T22:01:38.597583+0000 mgr.y (mgr.34107) 9402 : cluster [DBG] pgmap v10207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:40 smithi115 bash[45495]: cluster 2022-04-22T22:01:38.597583+0000 mgr.y (mgr.34107) 9402 : cluster [DBG] pgmap v10207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:41 smithi121 bash[36902]: audit 2022-04-22T22:01:40.448976+0000 mon.a (mon.0) 13238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:41 smithi115 bash[45495]: audit 2022-04-22T22:01:40.448976+0000 mon.a (mon.0) 13238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:41 smithi115 bash[46934]: audit 2022-04-22T22:01:40.448976+0000 mon.a (mon.0) 13238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:42 smithi121 bash[36902]: cluster 2022-04-22T22:01:40.598678+0000 mgr.y (mgr.34107) 9403 : cluster [DBG] pgmap v10208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:42 smithi121 bash[36902]: audit 2022-04-22T22:01:41.567853+0000 mon.a (mon.0) 13239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:42 smithi121 bash[36902]: audit 2022-04-22T22:01:41.869008+0000 mon.a (mon.0) 13240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[45495]: cluster 2022-04-22T22:01:40.598678+0000 mgr.y (mgr.34107) 9403 : cluster [DBG] pgmap v10208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[45495]: audit 2022-04-22T22:01:41.567853+0000 mon.a (mon.0) 13239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[45495]: audit 2022-04-22T22:01:41.869008+0000 mon.a (mon.0) 13240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[46934]: cluster 2022-04-22T22:01:40.598678+0000 mgr.y (mgr.34107) 9403 : cluster [DBG] pgmap v10208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[46934]: audit 2022-04-22T22:01:41.567853+0000 mon.a (mon.0) 13239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[46934]: audit 2022-04-22T22:01:41.869008+0000 mon.a (mon.0) 13240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:42.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:44 smithi121 bash[36902]: cluster 2022-04-22T22:01:42.599818+0000 mgr.y (mgr.34107) 9404 : cluster [DBG] pgmap v10209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:44 smithi115 bash[45495]: cluster 2022-04-22T22:01:42.599818+0000 mgr.y (mgr.34107) 9404 : cluster [DBG] pgmap v10209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:44 smithi115 bash[46934]: cluster 2022-04-22T22:01:42.599818+0000 mgr.y (mgr.34107) 9404 : cluster [DBG] pgmap v10209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:44.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:46 smithi121 bash[36902]: cluster 2022-04-22T22:01:44.600880+0000 mgr.y (mgr.34107) 9405 : cluster [DBG] pgmap v10210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:46 smithi121 bash[36902]: audit 2022-04-22T22:01:45.452306+0000 mon.a (mon.0) 13241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:46 smithi115 bash[45495]: cluster 2022-04-22T22:01:44.600880+0000 mgr.y (mgr.34107) 9405 : cluster [DBG] pgmap v10210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:46 smithi115 bash[45495]: audit 2022-04-22T22:01:45.452306+0000 mon.a (mon.0) 13241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:46 smithi115 bash[46934]: cluster 2022-04-22T22:01:44.600880+0000 mgr.y (mgr.34107) 9405 : cluster [DBG] pgmap v10210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:46 smithi115 bash[46934]: audit 2022-04-22T22:01:45.452306+0000 mon.a (mon.0) 13241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:47 smithi121 bash[36902]: audit 2022-04-22T22:01:46.571575+0000 mon.a (mon.0) 13242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:47 smithi121 bash[36902]: audit 2022-04-22T22:01:46.871764+0000 mon.a (mon.0) 13243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:47 smithi115 bash[45495]: audit 2022-04-22T22:01:46.571575+0000 mon.a (mon.0) 13242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:47 smithi115 bash[45495]: audit 2022-04-22T22:01:46.871764+0000 mon.a (mon.0) 13243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:47 smithi115 bash[46934]: audit 2022-04-22T22:01:46.571575+0000 mon.a (mon.0) 13242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:47 smithi115 bash[46934]: audit 2022-04-22T22:01:46.871764+0000 mon.a (mon.0) 13243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:48 smithi121 bash[36902]: cluster 2022-04-22T22:01:46.601950+0000 mgr.y (mgr.34107) 9406 : cluster [DBG] pgmap v10211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:48 smithi115 bash[45495]: cluster 2022-04-22T22:01:46.601950+0000 mgr.y (mgr.34107) 9406 : cluster [DBG] pgmap v10211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:48 smithi115 bash[46934]: cluster 2022-04-22T22:01:46.601950+0000 mgr.y (mgr.34107) 9406 : cluster [DBG] pgmap v10211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:50 smithi115 bash[45495]: cluster 2022-04-22T22:01:48.603116+0000 mgr.y (mgr.34107) 9407 : cluster [DBG] pgmap v10212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:50 smithi115 bash[46934]: cluster 2022-04-22T22:01:48.603116+0000 mgr.y (mgr.34107) 9407 : cluster [DBG] pgmap v10212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:50 smithi121 bash[36902]: cluster 2022-04-22T22:01:48.603116+0000 mgr.y (mgr.34107) 9407 : cluster [DBG] pgmap v10212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:51 smithi115 bash[45495]: audit 2022-04-22T22:01:50.455583+0000 mon.a (mon.0) 13244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:51 smithi115 bash[46934]: audit 2022-04-22T22:01:50.455583+0000 mon.a (mon.0) 13244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:51 smithi121 bash[36902]: audit 2022-04-22T22:01:50.455583+0000 mon.a (mon.0) 13244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[45495]: cluster 2022-04-22T22:01:50.604160+0000 mgr.y (mgr.34107) 9408 : cluster [DBG] pgmap v10213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[45495]: audit 2022-04-22T22:01:51.575328+0000 mon.a (mon.0) 13245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[45495]: audit 2022-04-22T22:01:51.874823+0000 mon.a (mon.0) 13246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[46934]: cluster 2022-04-22T22:01:50.604160+0000 mgr.y (mgr.34107) 9408 : cluster [DBG] pgmap v10213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[46934]: audit 2022-04-22T22:01:51.575328+0000 mon.a (mon.0) 13245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[46934]: audit 2022-04-22T22:01:51.874823+0000 mon.a (mon.0) 13246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:52 smithi121 bash[36902]: cluster 2022-04-22T22:01:50.604160+0000 mgr.y (mgr.34107) 9408 : cluster [DBG] pgmap v10213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:52 smithi121 bash[36902]: audit 2022-04-22T22:01:51.575328+0000 mon.a (mon.0) 13245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:52 smithi121 bash[36902]: audit 2022-04-22T22:01:51.874823+0000 mon.a (mon.0) 13246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:52.705Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:54 smithi115 bash[45495]: cluster 2022-04-22T22:01:52.605328+0000 mgr.y (mgr.34107) 9409 : cluster [DBG] pgmap v10214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:54 smithi115 bash[46934]: cluster 2022-04-22T22:01:52.605328+0000 mgr.y (mgr.34107) 9409 : cluster [DBG] pgmap v10214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:54 smithi121 bash[36902]: cluster 2022-04-22T22:01:52.605328+0000 mgr.y (mgr.34107) 9409 : cluster [DBG] pgmap v10214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:01:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:01:54.556Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:01:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:56 smithi115 bash[45495]: cluster 2022-04-22T22:01:54.606501+0000 mgr.y (mgr.34107) 9410 : cluster [DBG] pgmap v10215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:56 smithi115 bash[45495]: audit 2022-04-22T22:01:55.458786+0000 mon.a (mon.0) 13247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:56 smithi115 bash[46934]: cluster 2022-04-22T22:01:54.606501+0000 mgr.y (mgr.34107) 9410 : cluster [DBG] pgmap v10215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:56 smithi115 bash[46934]: audit 2022-04-22T22:01:55.458786+0000 mon.a (mon.0) 13247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:56 smithi121 bash[36902]: cluster 2022-04-22T22:01:54.606501+0000 mgr.y (mgr.34107) 9410 : cluster [DBG] pgmap v10215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:56 smithi121 bash[36902]: audit 2022-04-22T22:01:55.458786+0000 mon.a (mon.0) 13247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:01:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:57 smithi121 bash[36902]: audit 2022-04-22T22:01:56.578987+0000 mon.a (mon.0) 13248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:57 smithi121 bash[36902]: audit 2022-04-22T22:01:56.877507+0000 mon.a (mon.0) 13249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:57 smithi115 bash[45495]: audit 2022-04-22T22:01:56.578987+0000 mon.a (mon.0) 13248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:57 smithi115 bash[45495]: audit 2022-04-22T22:01:56.877507+0000 mon.a (mon.0) 13249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:57 smithi115 bash[46934]: audit 2022-04-22T22:01:56.578987+0000 mon.a (mon.0) 13248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:01:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:57 smithi115 bash[46934]: audit 2022-04-22T22:01:56.877507+0000 mon.a (mon.0) 13249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:01:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:01:58 smithi121 bash[36902]: cluster 2022-04-22T22:01:56.607594+0000 mgr.y (mgr.34107) 9411 : cluster [DBG] pgmap v10216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:01:58 smithi115 bash[45495]: cluster 2022-04-22T22:01:56.607594+0000 mgr.y (mgr.34107) 9411 : cluster [DBG] pgmap v10216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:01:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:01:58 smithi115 bash[46934]: cluster 2022-04-22T22:01:56.607594+0000 mgr.y (mgr.34107) 9411 : cluster [DBG] pgmap v10216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:00 smithi121 bash[36902]: cluster 2022-04-22T22:01:58.608742+0000 mgr.y (mgr.34107) 9412 : cluster [DBG] pgmap v10217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:00 smithi115 bash[45495]: cluster 2022-04-22T22:01:58.608742+0000 mgr.y (mgr.34107) 9412 : cluster [DBG] pgmap v10217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:00.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:00 smithi115 bash[46934]: cluster 2022-04-22T22:01:58.608742+0000 mgr.y (mgr.34107) 9412 : cluster [DBG] pgmap v10217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:01 smithi121 bash[36902]: audit 2022-04-22T22:02:00.462096+0000 mon.a (mon.0) 13250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:01 smithi115 bash[45495]: audit 2022-04-22T22:02:00.462096+0000 mon.a (mon.0) 13250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:01 smithi115 bash[46934]: audit 2022-04-22T22:02:00.462096+0000 mon.a (mon.0) 13250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:02.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[46934]: cluster 2022-04-22T22:02:00.609829+0000 mgr.y (mgr.34107) 9413 : cluster [DBG] pgmap v10218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:02.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[46934]: audit 2022-04-22T22:02:01.582683+0000 mon.a (mon.0) 13251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:02.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[46934]: audit 2022-04-22T22:02:01.880370+0000 mon.a (mon.0) 13252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:02.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[45495]: cluster 2022-04-22T22:02:00.609829+0000 mgr.y (mgr.34107) 9413 : cluster [DBG] pgmap v10218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:02.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[45495]: audit 2022-04-22T22:02:01.582683+0000 mon.a (mon.0) 13251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:02.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[45495]: audit 2022-04-22T22:02:01.880370+0000 mon.a (mon.0) 13252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:02 smithi121 bash[36902]: cluster 2022-04-22T22:02:00.609829+0000 mgr.y (mgr.34107) 9413 : cluster [DBG] pgmap v10218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:02 smithi121 bash[36902]: audit 2022-04-22T22:02:01.582683+0000 mon.a (mon.0) 13251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:02 smithi121 bash[36902]: audit 2022-04-22T22:02:01.880370+0000 mon.a (mon.0) 13252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:02.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:04 smithi121 bash[36902]: cluster 2022-04-22T22:02:02.611008+0000 mgr.y (mgr.34107) 9414 : cluster [DBG] pgmap v10219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:04 smithi115 bash[45495]: cluster 2022-04-22T22:02:02.611008+0000 mgr.y (mgr.34107) 9414 : cluster [DBG] pgmap v10219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:04.558Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:04 smithi115 bash[46934]: cluster 2022-04-22T22:02:02.611008+0000 mgr.y (mgr.34107) 9414 : cluster [DBG] pgmap v10219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:06 smithi121 bash[36902]: cluster 2022-04-22T22:02:04.612147+0000 mgr.y (mgr.34107) 9415 : cluster [DBG] pgmap v10220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:06 smithi121 bash[36902]: audit 2022-04-22T22:02:05.465456+0000 mon.a (mon.0) 13253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:06 smithi115 bash[45495]: cluster 2022-04-22T22:02:04.612147+0000 mgr.y (mgr.34107) 9415 : cluster [DBG] pgmap v10220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:06 smithi115 bash[45495]: audit 2022-04-22T22:02:05.465456+0000 mon.a (mon.0) 13253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:06 smithi115 bash[46934]: cluster 2022-04-22T22:02:04.612147+0000 mgr.y (mgr.34107) 9415 : cluster [DBG] pgmap v10220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:06 smithi115 bash[46934]: audit 2022-04-22T22:02:05.465456+0000 mon.a (mon.0) 13253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:06.945 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:02:07.287 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:02:07.287 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:02:07.287 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:02:07.288 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:02:07.289 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:02:07.688 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:02:07.688 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:02:07.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:02:07.688 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:02:07.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:07.690 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:02:07.690 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:02:07.690 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:02:07.690 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:02:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:07 smithi121 bash[36902]: audit 2022-04-22T22:02:06.586696+0000 mon.a (mon.0) 13254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:07 smithi121 bash[36902]: audit 2022-04-22T22:02:06.883203+0000 mon.a (mon.0) 13255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:07 smithi115 bash[45495]: audit 2022-04-22T22:02:06.586696+0000 mon.a (mon.0) 13254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:07 smithi115 bash[45495]: audit 2022-04-22T22:02:06.883203+0000 mon.a (mon.0) 13255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:07 smithi115 bash[46934]: audit 2022-04-22T22:02:06.586696+0000 mon.a (mon.0) 13254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:07 smithi115 bash[46934]: audit 2022-04-22T22:02:06.883203+0000 mon.a (mon.0) 13255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:08 smithi121 bash[36902]: cluster 2022-04-22T22:02:06.613211+0000 mgr.y (mgr.34107) 9416 : cluster [DBG] pgmap v10221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:08 smithi121 bash[36902]: audit 2022-04-22T22:02:06.920895+0000 mgr.y (mgr.34107) 9417 : audit [DBG] from='client.55057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:08 smithi121 bash[36902]: audit 2022-04-22T22:02:07.263399+0000 mgr.y (mgr.34107) 9418 : audit [DBG] from='client.45132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:08 smithi121 bash[36902]: audit 2022-04-22T22:02:07.684585+0000 mon.a (mon.0) 13256 : audit [DBG] from='client.? 172.21.15.115:0/1640246486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[45495]: cluster 2022-04-22T22:02:06.613211+0000 mgr.y (mgr.34107) 9416 : cluster [DBG] pgmap v10221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[45495]: audit 2022-04-22T22:02:06.920895+0000 mgr.y (mgr.34107) 9417 : audit [DBG] from='client.55057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[45495]: audit 2022-04-22T22:02:07.263399+0000 mgr.y (mgr.34107) 9418 : audit [DBG] from='client.45132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[45495]: audit 2022-04-22T22:02:07.684585+0000 mon.a (mon.0) 13256 : audit [DBG] from='client.? 172.21.15.115:0/1640246486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[46934]: cluster 2022-04-22T22:02:06.613211+0000 mgr.y (mgr.34107) 9416 : cluster [DBG] pgmap v10221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[46934]: audit 2022-04-22T22:02:06.920895+0000 mgr.y (mgr.34107) 9417 : audit [DBG] from='client.55057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[46934]: audit 2022-04-22T22:02:07.263399+0000 mgr.y (mgr.34107) 9418 : audit [DBG] from='client.45132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:08 smithi115 bash[46934]: audit 2022-04-22T22:02:07.684585+0000 mon.a (mon.0) 13256 : audit [DBG] from='client.? 172.21.15.115:0/1640246486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:10 smithi121 bash[36902]: cluster 2022-04-22T22:02:08.614386+0000 mgr.y (mgr.34107) 9419 : cluster [DBG] pgmap v10222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:10.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:10 smithi115 bash[45495]: cluster 2022-04-22T22:02:08.614386+0000 mgr.y (mgr.34107) 9419 : cluster [DBG] pgmap v10222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:10.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:10 smithi115 bash[46934]: cluster 2022-04-22T22:02:08.614386+0000 mgr.y (mgr.34107) 9419 : cluster [DBG] pgmap v10222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:11 smithi121 bash[36902]: audit 2022-04-22T22:02:10.468797+0000 mon.a (mon.0) 13257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:11 smithi115 bash[45495]: audit 2022-04-22T22:02:10.468797+0000 mon.a (mon.0) 13257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:11 smithi115 bash[46934]: audit 2022-04-22T22:02:10.468797+0000 mon.a (mon.0) 13257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:12.710 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[45495]: cluster 2022-04-22T22:02:10.615438+0000 mgr.y (mgr.34107) 9420 : cluster [DBG] pgmap v10223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:12.711 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[45495]: audit 2022-04-22T22:02:11.590315+0000 mon.a (mon.0) 13258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:12.711 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[45495]: audit 2022-04-22T22:02:11.886123+0000 mon.a (mon.0) 13259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:12.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[46934]: cluster 2022-04-22T22:02:10.615438+0000 mgr.y (mgr.34107) 9420 : cluster [DBG] pgmap v10223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:12.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[46934]: audit 2022-04-22T22:02:11.590315+0000 mon.a (mon.0) 13258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:12.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[46934]: audit 2022-04-22T22:02:11.886123+0000 mon.a (mon.0) 13259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:12 smithi121 bash[36902]: cluster 2022-04-22T22:02:10.615438+0000 mgr.y (mgr.34107) 9420 : cluster [DBG] pgmap v10223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:12 smithi121 bash[36902]: audit 2022-04-22T22:02:11.590315+0000 mon.a (mon.0) 13258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:12 smithi121 bash[36902]: audit 2022-04-22T22:02:11.886123+0000 mon.a (mon.0) 13259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:12.705Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:13 smithi121 bash[36902]: cluster 2022-04-22T22:02:12.616542+0000 mgr.y (mgr.34107) 9421 : cluster [DBG] pgmap v10224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:13 smithi115 bash[45495]: cluster 2022-04-22T22:02:12.616542+0000 mgr.y (mgr.34107) 9421 : cluster [DBG] pgmap v10224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:13 smithi115 bash[46934]: cluster 2022-04-22T22:02:12.616542+0000 mgr.y (mgr.34107) 9421 : cluster [DBG] pgmap v10224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:14.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:15 smithi121 bash[36902]: cluster 2022-04-22T22:02:14.617664+0000 mgr.y (mgr.34107) 9422 : cluster [DBG] pgmap v10225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:15 smithi121 bash[36902]: audit 2022-04-22T22:02:15.471883+0000 mon.a (mon.0) 13260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:15 smithi115 bash[45495]: cluster 2022-04-22T22:02:14.617664+0000 mgr.y (mgr.34107) 9422 : cluster [DBG] pgmap v10225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:15 smithi115 bash[45495]: audit 2022-04-22T22:02:15.471883+0000 mon.a (mon.0) 13260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:15 smithi115 bash[46934]: cluster 2022-04-22T22:02:14.617664+0000 mgr.y (mgr.34107) 9422 : cluster [DBG] pgmap v10225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:15 smithi115 bash[46934]: audit 2022-04-22T22:02:15.471883+0000 mon.a (mon.0) 13260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[45495]: audit 2022-04-22T22:02:16.456704+0000 mon.a (mon.0) 13261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[45495]: audit 2022-04-22T22:02:16.458686+0000 mon.a (mon.0) 13262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[45495]: audit 2022-04-22T22:02:16.594039+0000 mon.a (mon.0) 13263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[45495]: audit 2022-04-22T22:02:16.888778+0000 mon.a (mon.0) 13264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[46934]: audit 2022-04-22T22:02:16.456704+0000 mon.a (mon.0) 13261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:02:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[46934]: audit 2022-04-22T22:02:16.458686+0000 mon.a (mon.0) 13262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:02:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[46934]: audit 2022-04-22T22:02:16.594039+0000 mon.a (mon.0) 13263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:17 smithi115 bash[46934]: audit 2022-04-22T22:02:16.888778+0000 mon.a (mon.0) 13264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:17 smithi121 bash[36902]: audit 2022-04-22T22:02:16.456704+0000 mon.a (mon.0) 13261 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:02:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:17 smithi121 bash[36902]: audit 2022-04-22T22:02:16.458686+0000 mon.a (mon.0) 13262 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:02:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:17 smithi121 bash[36902]: audit 2022-04-22T22:02:16.594039+0000 mon.a (mon.0) 13263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:17 smithi121 bash[36902]: audit 2022-04-22T22:02:16.888778+0000 mon.a (mon.0) 13264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:18 smithi115 bash[46934]: cluster 2022-04-22T22:02:16.618800+0000 mgr.y (mgr.34107) 9423 : cluster [DBG] pgmap v10226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:18 smithi115 bash[45495]: cluster 2022-04-22T22:02:16.618800+0000 mgr.y (mgr.34107) 9423 : cluster [DBG] pgmap v10226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:18 smithi121 bash[36902]: cluster 2022-04-22T22:02:16.618800+0000 mgr.y (mgr.34107) 9423 : cluster [DBG] pgmap v10226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:20 smithi121 bash[36902]: cluster 2022-04-22T22:02:18.619944+0000 mgr.y (mgr.34107) 9424 : cluster [DBG] pgmap v10227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:20 smithi115 bash[45495]: cluster 2022-04-22T22:02:18.619944+0000 mgr.y (mgr.34107) 9424 : cluster [DBG] pgmap v10227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:20 smithi115 bash[46934]: cluster 2022-04-22T22:02:18.619944+0000 mgr.y (mgr.34107) 9424 : cluster [DBG] pgmap v10227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:21 smithi121 bash[36902]: audit 2022-04-22T22:02:20.475105+0000 mon.a (mon.0) 13265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:21 smithi115 bash[45495]: audit 2022-04-22T22:02:20.475105+0000 mon.a (mon.0) 13265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:21 smithi115 bash[46934]: audit 2022-04-22T22:02:20.475105+0000 mon.a (mon.0) 13265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:22 smithi121 bash[36902]: cluster 2022-04-22T22:02:20.621033+0000 mgr.y (mgr.34107) 9425 : cluster [DBG] pgmap v10228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:22 smithi121 bash[36902]: audit 2022-04-22T22:02:21.597787+0000 mon.a (mon.0) 13266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:22 smithi121 bash[36902]: audit 2022-04-22T22:02:21.891840+0000 mon.a (mon.0) 13267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[45495]: cluster 2022-04-22T22:02:20.621033+0000 mgr.y (mgr.34107) 9425 : cluster [DBG] pgmap v10228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[45495]: audit 2022-04-22T22:02:21.597787+0000 mon.a (mon.0) 13266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[45495]: audit 2022-04-22T22:02:21.891840+0000 mon.a (mon.0) 13267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[46934]: cluster 2022-04-22T22:02:20.621033+0000 mgr.y (mgr.34107) 9425 : cluster [DBG] pgmap v10228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[46934]: audit 2022-04-22T22:02:21.597787+0000 mon.a (mon.0) 13266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[46934]: audit 2022-04-22T22:02:21.891840+0000 mon.a (mon.0) 13267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:22.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:24.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[45495]: cluster 2022-04-22T22:02:22.622268+0000 mgr.y (mgr.34107) 9426 : cluster [DBG] pgmap v10229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:24.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[45495]: audit 2022-04-22T22:02:23.833932+0000 mon.a (mon.0) 13268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:02:24.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[45495]: audit 2022-04-22T22:02:23.836981+0000 mon.a (mon.0) 13269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:02:24.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[45495]: audit 2022-04-22T22:02:23.838673+0000 mon.a (mon.0) 13270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:02:24.158 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:24 smithi121 bash[36902]: cluster 2022-04-22T22:02:22.622268+0000 mgr.y (mgr.34107) 9426 : cluster [DBG] pgmap v10229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:24.158 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:24 smithi121 bash[36902]: audit 2022-04-22T22:02:23.833932+0000 mon.a (mon.0) 13268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:02:24.158 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:24 smithi121 bash[36902]: audit 2022-04-22T22:02:23.836981+0000 mon.a (mon.0) 13269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:02:24.158 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:24 smithi121 bash[36902]: audit 2022-04-22T22:02:23.838673+0000 mon.a (mon.0) 13270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:02:24.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[46934]: cluster 2022-04-22T22:02:22.622268+0000 mgr.y (mgr.34107) 9426 : cluster [DBG] pgmap v10229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:24.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[46934]: audit 2022-04-22T22:02:23.833932+0000 mon.a (mon.0) 13268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:02:24.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[46934]: audit 2022-04-22T22:02:23.836981+0000 mon.a (mon.0) 13269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:02:24.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[46934]: audit 2022-04-22T22:02:23.838673+0000 mon.a (mon.0) 13270 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:02:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:24.557Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:25 smithi121 bash[36902]: audit 2022-04-22T22:02:24.189096+0000 mon.a (mon.0) 13271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:02:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:25 smithi115 bash[45495]: audit 2022-04-22T22:02:24.189096+0000 mon.a (mon.0) 13271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:02:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:25 smithi115 bash[46934]: audit 2022-04-22T22:02:24.189096+0000 mon.a (mon.0) 13271 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:02:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:26 smithi121 bash[36902]: cluster 2022-04-22T22:02:24.623409+0000 mgr.y (mgr.34107) 9427 : cluster [DBG] pgmap v10230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:26.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:26 smithi121 bash[36902]: audit 2022-04-22T22:02:25.478224+0000 mon.a (mon.0) 13272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:26 smithi115 bash[45495]: cluster 2022-04-22T22:02:24.623409+0000 mgr.y (mgr.34107) 9427 : cluster [DBG] pgmap v10230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:26.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:26 smithi115 bash[45495]: audit 2022-04-22T22:02:25.478224+0000 mon.a (mon.0) 13272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:26 smithi115 bash[46934]: cluster 2022-04-22T22:02:24.623409+0000 mgr.y (mgr.34107) 9427 : cluster [DBG] pgmap v10230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:26.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:26 smithi115 bash[46934]: audit 2022-04-22T22:02:25.478224+0000 mon.a (mon.0) 13272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:27 smithi121 bash[36902]: audit 2022-04-22T22:02:26.601412+0000 mon.a (mon.0) 13273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:27 smithi121 bash[36902]: audit 2022-04-22T22:02:26.894938+0000 mon.a (mon.0) 13274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:27 smithi115 bash[45495]: audit 2022-04-22T22:02:26.601412+0000 mon.a (mon.0) 13273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:27 smithi115 bash[45495]: audit 2022-04-22T22:02:26.894938+0000 mon.a (mon.0) 13274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:27 smithi115 bash[46934]: audit 2022-04-22T22:02:26.601412+0000 mon.a (mon.0) 13273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:27 smithi115 bash[46934]: audit 2022-04-22T22:02:26.894938+0000 mon.a (mon.0) 13274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:28 smithi121 bash[36902]: cluster 2022-04-22T22:02:26.624437+0000 mgr.y (mgr.34107) 9428 : cluster [DBG] pgmap v10231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:28 smithi115 bash[45495]: cluster 2022-04-22T22:02:26.624437+0000 mgr.y (mgr.34107) 9428 : cluster [DBG] pgmap v10231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:28 smithi115 bash[46934]: cluster 2022-04-22T22:02:26.624437+0000 mgr.y (mgr.34107) 9428 : cluster [DBG] pgmap v10231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:30 smithi121 bash[36902]: cluster 2022-04-22T22:02:28.625595+0000 mgr.y (mgr.34107) 9429 : cluster [DBG] pgmap v10232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:30 smithi115 bash[45495]: cluster 2022-04-22T22:02:28.625595+0000 mgr.y (mgr.34107) 9429 : cluster [DBG] pgmap v10232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:30 smithi115 bash[46934]: cluster 2022-04-22T22:02:28.625595+0000 mgr.y (mgr.34107) 9429 : cluster [DBG] pgmap v10232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:31 smithi121 bash[36902]: audit 2022-04-22T22:02:30.481407+0000 mon.a (mon.0) 13275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:31.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:31 smithi115 bash[45495]: audit 2022-04-22T22:02:30.481407+0000 mon.a (mon.0) 13275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:31.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:31 smithi115 bash[46934]: audit 2022-04-22T22:02:30.481407+0000 mon.a (mon.0) 13275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:32 smithi121 bash[36902]: cluster 2022-04-22T22:02:30.626644+0000 mgr.y (mgr.34107) 9430 : cluster [DBG] pgmap v10233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:32 smithi121 bash[36902]: audit 2022-04-22T22:02:31.605114+0000 mon.a (mon.0) 13276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:32 smithi121 bash[36902]: audit 2022-04-22T22:02:31.897972+0000 mon.a (mon.0) 13277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[45495]: cluster 2022-04-22T22:02:30.626644+0000 mgr.y (mgr.34107) 9430 : cluster [DBG] pgmap v10233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[45495]: audit 2022-04-22T22:02:31.605114+0000 mon.a (mon.0) 13276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[45495]: audit 2022-04-22T22:02:31.897972+0000 mon.a (mon.0) 13277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[46934]: cluster 2022-04-22T22:02:30.626644+0000 mgr.y (mgr.34107) 9430 : cluster [DBG] pgmap v10233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[46934]: audit 2022-04-22T22:02:31.605114+0000 mon.a (mon.0) 13276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[46934]: audit 2022-04-22T22:02:31.897972+0000 mon.a (mon.0) 13277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:32.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:34 smithi121 bash[36902]: cluster 2022-04-22T22:02:32.627780+0000 mgr.y (mgr.34107) 9431 : cluster [DBG] pgmap v10234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:34 smithi115 bash[45495]: cluster 2022-04-22T22:02:32.627780+0000 mgr.y (mgr.34107) 9431 : cluster [DBG] pgmap v10234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:34.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:34 smithi115 bash[46934]: cluster 2022-04-22T22:02:32.627780+0000 mgr.y (mgr.34107) 9431 : cluster [DBG] pgmap v10234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:34.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:36 smithi121 bash[36902]: cluster 2022-04-22T22:02:34.628899+0000 mgr.y (mgr.34107) 9432 : cluster [DBG] pgmap v10235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:36 smithi121 bash[36902]: audit 2022-04-22T22:02:35.484554+0000 mon.a (mon.0) 13278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:36 smithi115 bash[45495]: cluster 2022-04-22T22:02:34.628899+0000 mgr.y (mgr.34107) 9432 : cluster [DBG] pgmap v10235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:36 smithi115 bash[45495]: audit 2022-04-22T22:02:35.484554+0000 mon.a (mon.0) 13278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:36 smithi115 bash[46934]: cluster 2022-04-22T22:02:34.628899+0000 mgr.y (mgr.34107) 9432 : cluster [DBG] pgmap v10235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:36 smithi115 bash[46934]: audit 2022-04-22T22:02:35.484554+0000 mon.a (mon.0) 13278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:37 smithi121 bash[36902]: audit 2022-04-22T22:02:36.608704+0000 mon.a (mon.0) 13279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:37 smithi121 bash[36902]: audit 2022-04-22T22:02:36.901131+0000 mon.a (mon.0) 13280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:37 smithi115 bash[45495]: audit 2022-04-22T22:02:36.608704+0000 mon.a (mon.0) 13279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:37 smithi115 bash[45495]: audit 2022-04-22T22:02:36.901131+0000 mon.a (mon.0) 13280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:37 smithi115 bash[46934]: audit 2022-04-22T22:02:36.608704+0000 mon.a (mon.0) 13279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:37 smithi115 bash[46934]: audit 2022-04-22T22:02:36.901131+0000 mon.a (mon.0) 13280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:38.072 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:02:38.423 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:02:38.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:02:38.425 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:02:38.425 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:02:38.425 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:02:38.425 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:02:38.425 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:02:38.426 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:02:38.426 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:02:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:38 smithi121 bash[36902]: cluster 2022-04-22T22:02:36.630030+0000 mgr.y (mgr.34107) 9433 : cluster [DBG] pgmap v10236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:38 smithi115 bash[46934]: cluster 2022-04-22T22:02:36.630030+0000 mgr.y (mgr.34107) 9433 : cluster [DBG] pgmap v10236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:38 smithi115 bash[45495]: cluster 2022-04-22T22:02:36.630030+0000 mgr.y (mgr.34107) 9433 : cluster [DBG] pgmap v10236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:38.828 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:02:38.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:02:38.830 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:02:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:39 smithi121 bash[36902]: audit 2022-04-22T22:02:38.049500+0000 mgr.y (mgr.34107) 9434 : audit [DBG] from='client.55075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:39 smithi121 bash[36902]: audit 2022-04-22T22:02:38.825270+0000 mon.a (mon.0) 13281 : audit [DBG] from='client.? 172.21.15.115:0/1425247772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:39 smithi115 bash[45495]: audit 2022-04-22T22:02:38.049500+0000 mgr.y (mgr.34107) 9434 : audit [DBG] from='client.55075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:39 smithi115 bash[45495]: audit 2022-04-22T22:02:38.825270+0000 mon.a (mon.0) 13281 : audit [DBG] from='client.? 172.21.15.115:0/1425247772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:39 smithi115 bash[46934]: audit 2022-04-22T22:02:38.049500+0000 mgr.y (mgr.34107) 9434 : audit [DBG] from='client.55075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:39 smithi115 bash[46934]: audit 2022-04-22T22:02:38.825270+0000 mon.a (mon.0) 13281 : audit [DBG] from='client.? 172.21.15.115:0/1425247772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:02:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:40 smithi121 bash[36902]: audit 2022-04-22T22:02:38.401443+0000 mgr.y (mgr.34107) 9435 : audit [DBG] from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:40 smithi121 bash[36902]: cluster 2022-04-22T22:02:38.630857+0000 mgr.y (mgr.34107) 9436 : cluster [DBG] pgmap v10237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:40 smithi115 bash[45495]: audit 2022-04-22T22:02:38.401443+0000 mgr.y (mgr.34107) 9435 : audit [DBG] from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:40 smithi115 bash[45495]: cluster 2022-04-22T22:02:38.630857+0000 mgr.y (mgr.34107) 9436 : cluster [DBG] pgmap v10237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:40 smithi115 bash[46934]: audit 2022-04-22T22:02:38.401443+0000 mgr.y (mgr.34107) 9435 : audit [DBG] from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:02:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:40 smithi115 bash[46934]: cluster 2022-04-22T22:02:38.630857+0000 mgr.y (mgr.34107) 9436 : cluster [DBG] pgmap v10237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:41 smithi115 bash[45495]: audit 2022-04-22T22:02:40.487766+0000 mon.a (mon.0) 13282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:41 smithi115 bash[46934]: audit 2022-04-22T22:02:40.487766+0000 mon.a (mon.0) 13282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:41 smithi121 bash[36902]: audit 2022-04-22T22:02:40.487766+0000 mon.a (mon.0) 13282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:42.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[46934]: cluster 2022-04-22T22:02:40.631890+0000 mgr.y (mgr.34107) 9437 : cluster [DBG] pgmap v10238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[46934]: audit 2022-04-22T22:02:41.612451+0000 mon.a (mon.0) 13283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[46934]: audit 2022-04-22T22:02:41.904157+0000 mon.a (mon.0) 13284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[45495]: cluster 2022-04-22T22:02:40.631890+0000 mgr.y (mgr.34107) 9437 : cluster [DBG] pgmap v10238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[45495]: audit 2022-04-22T22:02:41.612451+0000 mon.a (mon.0) 13283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[45495]: audit 2022-04-22T22:02:41.904157+0000 mon.a (mon.0) 13284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:42 smithi121 bash[36902]: cluster 2022-04-22T22:02:40.631890+0000 mgr.y (mgr.34107) 9437 : cluster [DBG] pgmap v10238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:42 smithi121 bash[36902]: audit 2022-04-22T22:02:41.612451+0000 mon.a (mon.0) 13283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:42 smithi121 bash[36902]: audit 2022-04-22T22:02:41.904157+0000 mon.a (mon.0) 13284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:42.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:44 smithi115 bash[45495]: cluster 2022-04-22T22:02:42.632961+0000 mgr.y (mgr.34107) 9438 : cluster [DBG] pgmap v10239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:44 smithi115 bash[46934]: cluster 2022-04-22T22:02:42.632961+0000 mgr.y (mgr.34107) 9438 : cluster [DBG] pgmap v10239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:44 smithi121 bash[36902]: cluster 2022-04-22T22:02:42.632961+0000 mgr.y (mgr.34107) 9438 : cluster [DBG] pgmap v10239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:44.559Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:46 smithi115 bash[45495]: cluster 2022-04-22T22:02:44.634039+0000 mgr.y (mgr.34107) 9439 : cluster [DBG] pgmap v10240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:46 smithi115 bash[45495]: audit 2022-04-22T22:02:45.490931+0000 mon.a (mon.0) 13285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:46 smithi115 bash[46934]: cluster 2022-04-22T22:02:44.634039+0000 mgr.y (mgr.34107) 9439 : cluster [DBG] pgmap v10240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:46 smithi115 bash[46934]: audit 2022-04-22T22:02:45.490931+0000 mon.a (mon.0) 13285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:46 smithi121 bash[36902]: cluster 2022-04-22T22:02:44.634039+0000 mgr.y (mgr.34107) 9439 : cluster [DBG] pgmap v10240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:46 smithi121 bash[36902]: audit 2022-04-22T22:02:45.490931+0000 mon.a (mon.0) 13285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:47 smithi115 bash[46934]: audit 2022-04-22T22:02:46.615959+0000 mon.a (mon.0) 13286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:47 smithi115 bash[46934]: audit 2022-04-22T22:02:46.907112+0000 mon.a (mon.0) 13287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:47 smithi115 bash[45495]: audit 2022-04-22T22:02:46.615959+0000 mon.a (mon.0) 13286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:47 smithi115 bash[45495]: audit 2022-04-22T22:02:46.907112+0000 mon.a (mon.0) 13287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:47 smithi121 bash[36902]: audit 2022-04-22T22:02:46.615959+0000 mon.a (mon.0) 13286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:47 smithi121 bash[36902]: audit 2022-04-22T22:02:46.907112+0000 mon.a (mon.0) 13287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:48 smithi115 bash[46934]: cluster 2022-04-22T22:02:46.635053+0000 mgr.y (mgr.34107) 9440 : cluster [DBG] pgmap v10241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:48 smithi115 bash[45495]: cluster 2022-04-22T22:02:46.635053+0000 mgr.y (mgr.34107) 9440 : cluster [DBG] pgmap v10241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:48.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:48 smithi121 bash[36902]: cluster 2022-04-22T22:02:46.635053+0000 mgr.y (mgr.34107) 9440 : cluster [DBG] pgmap v10241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:50.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:50 smithi121 bash[36902]: cluster 2022-04-22T22:02:48.636350+0000 mgr.y (mgr.34107) 9441 : cluster [DBG] pgmap v10242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:50.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:50 smithi115 bash[46934]: cluster 2022-04-22T22:02:48.636350+0000 mgr.y (mgr.34107) 9441 : cluster [DBG] pgmap v10242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:50 smithi115 bash[45495]: cluster 2022-04-22T22:02:48.636350+0000 mgr.y (mgr.34107) 9441 : cluster [DBG] pgmap v10242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:51 smithi121 bash[36902]: audit 2022-04-22T22:02:50.494382+0000 mon.a (mon.0) 13288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:51.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:51 smithi115 bash[46934]: audit 2022-04-22T22:02:50.494382+0000 mon.a (mon.0) 13288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:51 smithi115 bash[45495]: audit 2022-04-22T22:02:50.494382+0000 mon.a (mon.0) 13288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:52.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[46934]: cluster 2022-04-22T22:02:50.637365+0000 mgr.y (mgr.34107) 9442 : cluster [DBG] pgmap v10243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:52.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[46934]: audit 2022-04-22T22:02:51.619559+0000 mon.a (mon.0) 13289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:52.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[46934]: audit 2022-04-22T22:02:51.910356+0000 mon.a (mon.0) 13290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:52.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[45495]: cluster 2022-04-22T22:02:50.637365+0000 mgr.y (mgr.34107) 9442 : cluster [DBG] pgmap v10243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:52.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[45495]: audit 2022-04-22T22:02:51.619559+0000 mon.a (mon.0) 13289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:52.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[45495]: audit 2022-04-22T22:02:51.910356+0000 mon.a (mon.0) 13290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:52 smithi121 bash[36902]: cluster 2022-04-22T22:02:50.637365+0000 mgr.y (mgr.34107) 9442 : cluster [DBG] pgmap v10243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:52.827 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:52 smithi121 bash[36902]: audit 2022-04-22T22:02:51.619559+0000 mon.a (mon.0) 13289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:52.827 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:52 smithi121 bash[36902]: audit 2022-04-22T22:02:51.910356+0000 mon.a (mon.0) 13290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:52.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:54.563 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:54 smithi115 bash[45495]: cluster 2022-04-22T22:02:52.638502+0000 mgr.y (mgr.34107) 9443 : cluster [DBG] pgmap v10244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:54.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:54 smithi115 bash[46934]: cluster 2022-04-22T22:02:52.638502+0000 mgr.y (mgr.34107) 9443 : cluster [DBG] pgmap v10244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:54 smithi121 bash[36902]: cluster 2022-04-22T22:02:52.638502+0000 mgr.y (mgr.34107) 9443 : cluster [DBG] pgmap v10244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:02:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:02:54.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:02:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:56 smithi121 bash[36902]: cluster 2022-04-22T22:02:54.639658+0000 mgr.y (mgr.34107) 9444 : cluster [DBG] pgmap v10245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:56 smithi121 bash[36902]: audit 2022-04-22T22:02:55.497523+0000 mon.a (mon.0) 13291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:56 smithi115 bash[45495]: cluster 2022-04-22T22:02:54.639658+0000 mgr.y (mgr.34107) 9444 : cluster [DBG] pgmap v10245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:56 smithi115 bash[45495]: audit 2022-04-22T22:02:55.497523+0000 mon.a (mon.0) 13291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:56 smithi115 bash[46934]: cluster 2022-04-22T22:02:54.639658+0000 mgr.y (mgr.34107) 9444 : cluster [DBG] pgmap v10245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:56 smithi115 bash[46934]: audit 2022-04-22T22:02:55.497523+0000 mon.a (mon.0) 13291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:02:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:57 smithi121 bash[36902]: audit 2022-04-22T22:02:56.623400+0000 mon.a (mon.0) 13292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:57 smithi121 bash[36902]: audit 2022-04-22T22:02:56.913302+0000 mon.a (mon.0) 13293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:57 smithi115 bash[45495]: audit 2022-04-22T22:02:56.623400+0000 mon.a (mon.0) 13292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:57 smithi115 bash[45495]: audit 2022-04-22T22:02:56.913302+0000 mon.a (mon.0) 13293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:57 smithi115 bash[46934]: audit 2022-04-22T22:02:56.623400+0000 mon.a (mon.0) 13292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:02:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:57 smithi115 bash[46934]: audit 2022-04-22T22:02:56.913302+0000 mon.a (mon.0) 13293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:02:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:02:58 smithi121 bash[36902]: cluster 2022-04-22T22:02:56.640648+0000 mgr.y (mgr.34107) 9445 : cluster [DBG] pgmap v10246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:02:58 smithi115 bash[45495]: cluster 2022-04-22T22:02:56.640648+0000 mgr.y (mgr.34107) 9445 : cluster [DBG] pgmap v10246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:02:58.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:02:58 smithi115 bash[46934]: cluster 2022-04-22T22:02:56.640648+0000 mgr.y (mgr.34107) 9445 : cluster [DBG] pgmap v10246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:00 smithi121 bash[36902]: cluster 2022-04-22T22:02:58.641792+0000 mgr.y (mgr.34107) 9446 : cluster [DBG] pgmap v10247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:00 smithi115 bash[45495]: cluster 2022-04-22T22:02:58.641792+0000 mgr.y (mgr.34107) 9446 : cluster [DBG] pgmap v10247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:00 smithi115 bash[46934]: cluster 2022-04-22T22:02:58.641792+0000 mgr.y (mgr.34107) 9446 : cluster [DBG] pgmap v10247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:01 smithi121 bash[36902]: audit 2022-04-22T22:03:00.500745+0000 mon.a (mon.0) 13294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:01 smithi115 bash[45495]: audit 2022-04-22T22:03:00.500745+0000 mon.a (mon.0) 13294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:01 smithi115 bash[46934]: audit 2022-04-22T22:03:00.500745+0000 mon.a (mon.0) 13294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:02.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[45495]: cluster 2022-04-22T22:03:00.642808+0000 mgr.y (mgr.34107) 9447 : cluster [DBG] pgmap v10248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:02.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[45495]: audit 2022-04-22T22:03:01.627149+0000 mon.a (mon.0) 13295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:02.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[45495]: audit 2022-04-22T22:03:01.916435+0000 mon.a (mon.0) 13296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:02.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[46934]: cluster 2022-04-22T22:03:00.642808+0000 mgr.y (mgr.34107) 9447 : cluster [DBG] pgmap v10248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:02.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[46934]: audit 2022-04-22T22:03:01.627149+0000 mon.a (mon.0) 13295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:02.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[46934]: audit 2022-04-22T22:03:01.916435+0000 mon.a (mon.0) 13296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:02 smithi121 bash[36902]: cluster 2022-04-22T22:03:00.642808+0000 mgr.y (mgr.34107) 9447 : cluster [DBG] pgmap v10248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:02 smithi121 bash[36902]: audit 2022-04-22T22:03:01.627149+0000 mon.a (mon.0) 13295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:02 smithi121 bash[36902]: audit 2022-04-22T22:03:01.916435+0000 mon.a (mon.0) 13296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:02.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:04 smithi121 bash[36902]: cluster 2022-04-22T22:03:02.644033+0000 mgr.y (mgr.34107) 9448 : cluster [DBG] pgmap v10249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:04 smithi115 bash[45495]: cluster 2022-04-22T22:03:02.644033+0000 mgr.y (mgr.34107) 9448 : cluster [DBG] pgmap v10249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:04.562Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:04 smithi115 bash[46934]: cluster 2022-04-22T22:03:02.644033+0000 mgr.y (mgr.34107) 9448 : cluster [DBG] pgmap v10249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:06 smithi121 bash[36902]: cluster 2022-04-22T22:03:04.645173+0000 mgr.y (mgr.34107) 9449 : cluster [DBG] pgmap v10250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:06 smithi121 bash[36902]: audit 2022-04-22T22:03:05.503658+0000 mon.a (mon.0) 13297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:06 smithi115 bash[45495]: cluster 2022-04-22T22:03:04.645173+0000 mgr.y (mgr.34107) 9449 : cluster [DBG] pgmap v10250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:06 smithi115 bash[45495]: audit 2022-04-22T22:03:05.503658+0000 mon.a (mon.0) 13297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:06 smithi115 bash[46934]: cluster 2022-04-22T22:03:04.645173+0000 mgr.y (mgr.34107) 9449 : cluster [DBG] pgmap v10250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:06 smithi115 bash[46934]: audit 2022-04-22T22:03:05.503658+0000 mon.a (mon.0) 13297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:07 smithi121 bash[36902]: audit 2022-04-22T22:03:06.630812+0000 mon.a (mon.0) 13298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:07 smithi121 bash[36902]: audit 2022-04-22T22:03:06.919690+0000 mon.a (mon.0) 13299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:07 smithi115 bash[45495]: audit 2022-04-22T22:03:06.630812+0000 mon.a (mon.0) 13298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:07 smithi115 bash[45495]: audit 2022-04-22T22:03:06.919690+0000 mon.a (mon.0) 13299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:07 smithi115 bash[46934]: audit 2022-04-22T22:03:06.630812+0000 mon.a (mon.0) 13298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:07 smithi115 bash[46934]: audit 2022-04-22T22:03:06.919690+0000 mon.a (mon.0) 13299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:08 smithi121 bash[36902]: cluster 2022-04-22T22:03:06.646361+0000 mgr.y (mgr.34107) 9450 : cluster [DBG] pgmap v10251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:08 smithi115 bash[45495]: cluster 2022-04-22T22:03:06.646361+0000 mgr.y (mgr.34107) 9450 : cluster [DBG] pgmap v10251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:08 smithi115 bash[46934]: cluster 2022-04-22T22:03:06.646361+0000 mgr.y (mgr.34107) 9450 : cluster [DBG] pgmap v10251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:09.208 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:03:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:03:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:03:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:03:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:03:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:03:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:03:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:03:09.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:09 smithi121 bash[36902]: cluster 2022-04-22T22:03:08.647621+0000 mgr.y (mgr.34107) 9451 : cluster [DBG] pgmap v10252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:09 smithi121 bash[36902]: audit 2022-04-22T22:03:09.189958+0000 mgr.y (mgr.34107) 9452 : audit [DBG] from='client.55093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:09.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:09 smithi115 bash[46934]: cluster 2022-04-22T22:03:08.647621+0000 mgr.y (mgr.34107) 9451 : cluster [DBG] pgmap v10252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:09 smithi115 bash[46934]: audit 2022-04-22T22:03:09.189958+0000 mgr.y (mgr.34107) 9452 : audit [DBG] from='client.55093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:09 smithi115 bash[45495]: cluster 2022-04-22T22:03:08.647621+0000 mgr.y (mgr.34107) 9451 : cluster [DBG] pgmap v10252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:09 smithi115 bash[45495]: audit 2022-04-22T22:03:09.189958+0000 mgr.y (mgr.34107) 9452 : audit [DBG] from='client.55093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:03:09.965 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:03:09.966 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:03:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:10 smithi121 bash[36902]: audit 2022-04-22T22:03:09.536350+0000 mgr.y (mgr.34107) 9453 : audit [DBG] from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:10 smithi121 bash[36902]: audit 2022-04-22T22:03:09.961146+0000 mon.a (mon.0) 13300 : audit [DBG] from='client.? 172.21.15.115:0/987179372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:10.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:10 smithi115 bash[46934]: audit 2022-04-22T22:03:09.536350+0000 mgr.y (mgr.34107) 9453 : audit [DBG] from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:10.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:10 smithi115 bash[46934]: audit 2022-04-22T22:03:09.961146+0000 mon.a (mon.0) 13300 : audit [DBG] from='client.? 172.21.15.115:0/987179372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:10 smithi115 bash[45495]: audit 2022-04-22T22:03:09.536350+0000 mgr.y (mgr.34107) 9453 : audit [DBG] from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:10 smithi115 bash[45495]: audit 2022-04-22T22:03:09.961146+0000 mon.a (mon.0) 13300 : audit [DBG] from='client.? 172.21.15.115:0/987179372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:11 smithi121 bash[36902]: audit 2022-04-22T22:03:10.506963+0000 mon.a (mon.0) 13301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:11 smithi121 bash[36902]: cluster 2022-04-22T22:03:10.648766+0000 mgr.y (mgr.34107) 9454 : cluster [DBG] pgmap v10253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:11 smithi115 bash[46934]: audit 2022-04-22T22:03:10.506963+0000 mon.a (mon.0) 13301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:11 smithi115 bash[46934]: cluster 2022-04-22T22:03:10.648766+0000 mgr.y (mgr.34107) 9454 : cluster [DBG] pgmap v10253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:11 smithi115 bash[45495]: audit 2022-04-22T22:03:10.506963+0000 mon.a (mon.0) 13301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:11 smithi115 bash[45495]: cluster 2022-04-22T22:03:10.648766+0000 mgr.y (mgr.34107) 9454 : cluster [DBG] pgmap v10253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:12.712 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:12 smithi115 bash[45495]: audit 2022-04-22T22:03:11.634432+0000 mon.a (mon.0) 13302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:12.713 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:12 smithi115 bash[45495]: audit 2022-04-22T22:03:11.922599+0000 mon.a (mon.0) 13303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:12.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:12 smithi115 bash[46934]: audit 2022-04-22T22:03:11.634432+0000 mon.a (mon.0) 13302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:12.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:12 smithi115 bash[46934]: audit 2022-04-22T22:03:11.922599+0000 mon.a (mon.0) 13303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:12 smithi121 bash[36902]: audit 2022-04-22T22:03:11.634432+0000 mon.a (mon.0) 13302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:12 smithi121 bash[36902]: audit 2022-04-22T22:03:11.922599+0000 mon.a (mon.0) 13303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:12.709Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:13 smithi121 bash[36902]: cluster 2022-04-22T22:03:12.649881+0000 mgr.y (mgr.34107) 9455 : cluster [DBG] pgmap v10254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:13 smithi115 bash[45495]: cluster 2022-04-22T22:03:12.649881+0000 mgr.y (mgr.34107) 9455 : cluster [DBG] pgmap v10254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:13 smithi115 bash[46934]: cluster 2022-04-22T22:03:12.649881+0000 mgr.y (mgr.34107) 9455 : cluster [DBG] pgmap v10254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:14.565Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:15 smithi121 bash[36902]: cluster 2022-04-22T22:03:14.651006+0000 mgr.y (mgr.34107) 9456 : cluster [DBG] pgmap v10255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:15 smithi121 bash[36902]: audit 2022-04-22T22:03:15.510297+0000 mon.a (mon.0) 13304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:16.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:15 smithi115 bash[46934]: cluster 2022-04-22T22:03:14.651006+0000 mgr.y (mgr.34107) 9456 : cluster [DBG] pgmap v10255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:16.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:15 smithi115 bash[46934]: audit 2022-04-22T22:03:15.510297+0000 mon.a (mon.0) 13304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:15 smithi115 bash[45495]: cluster 2022-04-22T22:03:14.651006+0000 mgr.y (mgr.34107) 9456 : cluster [DBG] pgmap v10255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:15 smithi115 bash[45495]: audit 2022-04-22T22:03:15.510297+0000 mon.a (mon.0) 13304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:17 smithi121 bash[36902]: audit 2022-04-22T22:03:16.469461+0000 mon.a (mon.0) 13305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:03:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:17 smithi121 bash[36902]: audit 2022-04-22T22:03:16.470571+0000 mon.a (mon.0) 13306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:03:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:17 smithi121 bash[36902]: audit 2022-04-22T22:03:16.637984+0000 mon.a (mon.0) 13307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:17 smithi121 bash[36902]: audit 2022-04-22T22:03:16.925439+0000 mon.a (mon.0) 13308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[45495]: audit 2022-04-22T22:03:16.469461+0000 mon.a (mon.0) 13305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[45495]: audit 2022-04-22T22:03:16.470571+0000 mon.a (mon.0) 13306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[45495]: audit 2022-04-22T22:03:16.637984+0000 mon.a (mon.0) 13307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[45495]: audit 2022-04-22T22:03:16.925439+0000 mon.a (mon.0) 13308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[46934]: audit 2022-04-22T22:03:16.469461+0000 mon.a (mon.0) 13305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[46934]: audit 2022-04-22T22:03:16.470571+0000 mon.a (mon.0) 13306 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[46934]: audit 2022-04-22T22:03:16.637984+0000 mon.a (mon.0) 13307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:17 smithi115 bash[46934]: audit 2022-04-22T22:03:16.925439+0000 mon.a (mon.0) 13308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:18 smithi121 bash[36902]: cluster 2022-04-22T22:03:16.651921+0000 mgr.y (mgr.34107) 9457 : cluster [DBG] pgmap v10256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:18 smithi115 bash[45495]: cluster 2022-04-22T22:03:16.651921+0000 mgr.y (mgr.34107) 9457 : cluster [DBG] pgmap v10256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:18 smithi115 bash[46934]: cluster 2022-04-22T22:03:16.651921+0000 mgr.y (mgr.34107) 9457 : cluster [DBG] pgmap v10256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:20 smithi121 bash[36902]: cluster 2022-04-22T22:03:18.653113+0000 mgr.y (mgr.34107) 9458 : cluster [DBG] pgmap v10257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:20 smithi115 bash[45495]: cluster 2022-04-22T22:03:18.653113+0000 mgr.y (mgr.34107) 9458 : cluster [DBG] pgmap v10257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:20 smithi115 bash[46934]: cluster 2022-04-22T22:03:18.653113+0000 mgr.y (mgr.34107) 9458 : cluster [DBG] pgmap v10257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:21 smithi121 bash[36902]: audit 2022-04-22T22:03:20.513454+0000 mon.a (mon.0) 13309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:21 smithi115 bash[45495]: audit 2022-04-22T22:03:20.513454+0000 mon.a (mon.0) 13309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:21 smithi115 bash[46934]: audit 2022-04-22T22:03:20.513454+0000 mon.a (mon.0) 13309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:22 smithi121 bash[36902]: cluster 2022-04-22T22:03:20.654104+0000 mgr.y (mgr.34107) 9459 : cluster [DBG] pgmap v10258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:22 smithi121 bash[36902]: audit 2022-04-22T22:03:21.642107+0000 mon.a (mon.0) 13310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:22 smithi121 bash[36902]: audit 2022-04-22T22:03:21.928499+0000 mon.a (mon.0) 13311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[45495]: cluster 2022-04-22T22:03:20.654104+0000 mgr.y (mgr.34107) 9459 : cluster [DBG] pgmap v10258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[45495]: audit 2022-04-22T22:03:21.642107+0000 mon.a (mon.0) 13310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[45495]: audit 2022-04-22T22:03:21.928499+0000 mon.a (mon.0) 13311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[46934]: cluster 2022-04-22T22:03:20.654104+0000 mgr.y (mgr.34107) 9459 : cluster [DBG] pgmap v10258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[46934]: audit 2022-04-22T22:03:21.642107+0000 mon.a (mon.0) 13310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[46934]: audit 2022-04-22T22:03:21.928499+0000 mon.a (mon.0) 13311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:22.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:24 smithi121 bash[36902]: cluster 2022-04-22T22:03:22.655206+0000 mgr.y (mgr.34107) 9460 : cluster [DBG] pgmap v10259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:24.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:24 smithi115 bash[46934]: cluster 2022-04-22T22:03:22.655206+0000 mgr.y (mgr.34107) 9460 : cluster [DBG] pgmap v10259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:24 smithi115 bash[45495]: cluster 2022-04-22T22:03:22.655206+0000 mgr.y (mgr.34107) 9460 : cluster [DBG] pgmap v10259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:24.560Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:25 smithi121 bash[36902]: audit 2022-04-22T22:03:24.198058+0000 mon.a (mon.0) 13312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:03:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:25 smithi121 bash[36902]: audit 2022-04-22T22:03:24.200956+0000 mon.a (mon.0) 13313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:03:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:25 smithi121 bash[36902]: audit 2022-04-22T22:03:24.202701+0000 mon.a (mon.0) 13314 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:03:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:25 smithi121 bash[36902]: audit 2022-04-22T22:03:24.572149+0000 mon.a (mon.0) 13315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:03:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[45495]: audit 2022-04-22T22:03:24.198058+0000 mon.a (mon.0) 13312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[45495]: audit 2022-04-22T22:03:24.200956+0000 mon.a (mon.0) 13313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[45495]: audit 2022-04-22T22:03:24.202701+0000 mon.a (mon.0) 13314 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[45495]: audit 2022-04-22T22:03:24.572149+0000 mon.a (mon.0) 13315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[46934]: audit 2022-04-22T22:03:24.198058+0000 mon.a (mon.0) 13312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[46934]: audit 2022-04-22T22:03:24.200956+0000 mon.a (mon.0) 13313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[46934]: audit 2022-04-22T22:03:24.202701+0000 mon.a (mon.0) 13314 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:03:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:25 smithi115 bash[46934]: audit 2022-04-22T22:03:24.572149+0000 mon.a (mon.0) 13315 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:03:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:26 smithi121 bash[36902]: cluster 2022-04-22T22:03:24.655985+0000 mgr.y (mgr.34107) 9461 : cluster [DBG] pgmap v10260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:26 smithi121 bash[36902]: audit 2022-04-22T22:03:25.516674+0000 mon.a (mon.0) 13316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:26 smithi115 bash[45495]: cluster 2022-04-22T22:03:24.655985+0000 mgr.y (mgr.34107) 9461 : cluster [DBG] pgmap v10260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:26 smithi115 bash[45495]: audit 2022-04-22T22:03:25.516674+0000 mon.a (mon.0) 13316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:26 smithi115 bash[46934]: cluster 2022-04-22T22:03:24.655985+0000 mgr.y (mgr.34107) 9461 : cluster [DBG] pgmap v10260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:26 smithi115 bash[46934]: audit 2022-04-22T22:03:25.516674+0000 mon.a (mon.0) 13316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:27 smithi121 bash[36902]: audit 2022-04-22T22:03:26.645679+0000 mon.a (mon.0) 13317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:27 smithi121 bash[36902]: audit 2022-04-22T22:03:26.931410+0000 mon.a (mon.0) 13318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:27 smithi115 bash[45495]: audit 2022-04-22T22:03:26.645679+0000 mon.a (mon.0) 13317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:27 smithi115 bash[45495]: audit 2022-04-22T22:03:26.931410+0000 mon.a (mon.0) 13318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:27 smithi115 bash[46934]: audit 2022-04-22T22:03:26.645679+0000 mon.a (mon.0) 13317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:27 smithi115 bash[46934]: audit 2022-04-22T22:03:26.931410+0000 mon.a (mon.0) 13318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:28 smithi121 bash[36902]: cluster 2022-04-22T22:03:26.656999+0000 mgr.y (mgr.34107) 9462 : cluster [DBG] pgmap v10261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:28 smithi115 bash[45495]: cluster 2022-04-22T22:03:26.656999+0000 mgr.y (mgr.34107) 9462 : cluster [DBG] pgmap v10261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:28 smithi115 bash[46934]: cluster 2022-04-22T22:03:26.656999+0000 mgr.y (mgr.34107) 9462 : cluster [DBG] pgmap v10261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:30 smithi121 bash[36902]: cluster 2022-04-22T22:03:28.658153+0000 mgr.y (mgr.34107) 9463 : cluster [DBG] pgmap v10262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:30 smithi115 bash[45495]: cluster 2022-04-22T22:03:28.658153+0000 mgr.y (mgr.34107) 9463 : cluster [DBG] pgmap v10262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:30 smithi115 bash[46934]: cluster 2022-04-22T22:03:28.658153+0000 mgr.y (mgr.34107) 9463 : cluster [DBG] pgmap v10262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:31 smithi121 bash[36902]: audit 2022-04-22T22:03:30.520094+0000 mon.a (mon.0) 13319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:31 smithi115 bash[45495]: audit 2022-04-22T22:03:30.520094+0000 mon.a (mon.0) 13319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:31 smithi115 bash[46934]: audit 2022-04-22T22:03:30.520094+0000 mon.a (mon.0) 13319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:32 smithi121 bash[36902]: cluster 2022-04-22T22:03:30.659217+0000 mgr.y (mgr.34107) 9464 : cluster [DBG] pgmap v10263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:32 smithi121 bash[36902]: audit 2022-04-22T22:03:31.649607+0000 mon.a (mon.0) 13320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:32 smithi121 bash[36902]: audit 2022-04-22T22:03:31.934336+0000 mon.a (mon.0) 13321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[45495]: cluster 2022-04-22T22:03:30.659217+0000 mgr.y (mgr.34107) 9464 : cluster [DBG] pgmap v10263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[45495]: audit 2022-04-22T22:03:31.649607+0000 mon.a (mon.0) 13320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[45495]: audit 2022-04-22T22:03:31.934336+0000 mon.a (mon.0) 13321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[46934]: cluster 2022-04-22T22:03:30.659217+0000 mgr.y (mgr.34107) 9464 : cluster [DBG] pgmap v10263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[46934]: audit 2022-04-22T22:03:31.649607+0000 mon.a (mon.0) 13320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[46934]: audit 2022-04-22T22:03:31.934336+0000 mon.a (mon.0) 13321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:32.711Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:34 smithi121 bash[36902]: cluster 2022-04-22T22:03:32.660384+0000 mgr.y (mgr.34107) 9465 : cluster [DBG] pgmap v10264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:34 smithi115 bash[45495]: cluster 2022-04-22T22:03:32.660384+0000 mgr.y (mgr.34107) 9465 : cluster [DBG] pgmap v10264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:34 smithi115 bash[46934]: cluster 2022-04-22T22:03:32.660384+0000 mgr.y (mgr.34107) 9465 : cluster [DBG] pgmap v10264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:34.563Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:36 smithi121 bash[36902]: cluster 2022-04-22T22:03:34.661630+0000 mgr.y (mgr.34107) 9466 : cluster [DBG] pgmap v10265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:36 smithi121 bash[36902]: audit 2022-04-22T22:03:35.523196+0000 mon.a (mon.0) 13322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:36.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:36 smithi115 bash[45495]: cluster 2022-04-22T22:03:34.661630+0000 mgr.y (mgr.34107) 9466 : cluster [DBG] pgmap v10265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:36.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:36 smithi115 bash[45495]: audit 2022-04-22T22:03:35.523196+0000 mon.a (mon.0) 13322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:36 smithi115 bash[46934]: cluster 2022-04-22T22:03:34.661630+0000 mgr.y (mgr.34107) 9466 : cluster [DBG] pgmap v10265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:36.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:36 smithi115 bash[46934]: audit 2022-04-22T22:03:35.523196+0000 mon.a (mon.0) 13322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:37 smithi121 bash[36902]: audit 2022-04-22T22:03:36.653647+0000 mon.a (mon.0) 13323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:37 smithi121 bash[36902]: audit 2022-04-22T22:03:36.937367+0000 mon.a (mon.0) 13324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:37.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:37 smithi115 bash[46934]: audit 2022-04-22T22:03:36.653647+0000 mon.a (mon.0) 13323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:37 smithi115 bash[46934]: audit 2022-04-22T22:03:36.937367+0000 mon.a (mon.0) 13324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:37 smithi115 bash[45495]: audit 2022-04-22T22:03:36.653647+0000 mon.a (mon.0) 13323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:37 smithi115 bash[45495]: audit 2022-04-22T22:03:36.937367+0000 mon.a (mon.0) 13324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:38 smithi121 bash[36902]: cluster 2022-04-22T22:03:36.662574+0000 mgr.y (mgr.34107) 9467 : cluster [DBG] pgmap v10266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:38 smithi115 bash[45495]: cluster 2022-04-22T22:03:36.662574+0000 mgr.y (mgr.34107) 9467 : cluster [DBG] pgmap v10266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:38 smithi115 bash[46934]: cluster 2022-04-22T22:03:36.662574+0000 mgr.y (mgr.34107) 9467 : cluster [DBG] pgmap v10266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:40.341 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:03:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:40 smithi121 bash[36902]: cluster 2022-04-22T22:03:38.663763+0000 mgr.y (mgr.34107) 9468 : cluster [DBG] pgmap v10267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:40 smithi115 bash[45495]: cluster 2022-04-22T22:03:38.663763+0000 mgr.y (mgr.34107) 9468 : cluster [DBG] pgmap v10267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:40 smithi115 bash[46934]: cluster 2022-04-22T22:03:38.663763+0000 mgr.y (mgr.34107) 9468 : cluster [DBG] pgmap v10267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:40.684 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:03:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:03:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:03:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:03:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:03:40.685 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:03:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:03:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:03:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:03:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:03:40.686 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:03:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:03:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:03:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:03:40.687 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:03:40.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:03:40.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:03:40.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:03:40.688 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:03:40.689 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:03:40.689 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:03:41.084 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:03:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:03:41.086 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:03:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:41 smithi121 bash[36902]: audit 2022-04-22T22:03:40.322010+0000 mgr.y (mgr.34107) 9469 : audit [DBG] from='client.45180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:41 smithi121 bash[36902]: audit 2022-04-22T22:03:40.526089+0000 mon.a (mon.0) 13325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:41 smithi121 bash[36902]: audit 2022-04-22T22:03:41.080466+0000 mon.c (mon.1) 327 : audit [DBG] from='client.? 172.21.15.115:0/1987672503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[45495]: audit 2022-04-22T22:03:40.322010+0000 mgr.y (mgr.34107) 9469 : audit [DBG] from='client.45180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[45495]: audit 2022-04-22T22:03:40.526089+0000 mon.a (mon.0) 13325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[45495]: audit 2022-04-22T22:03:41.080466+0000 mon.c (mon.1) 327 : audit [DBG] from='client.? 172.21.15.115:0/1987672503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[46934]: audit 2022-04-22T22:03:40.322010+0000 mgr.y (mgr.34107) 9469 : audit [DBG] from='client.45180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[46934]: audit 2022-04-22T22:03:40.526089+0000 mon.a (mon.0) 13325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:41 smithi115 bash[46934]: audit 2022-04-22T22:03:41.080466+0000 mon.c (mon.1) 327 : audit [DBG] from='client.? 172.21.15.115:0/1987672503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:03:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:42 smithi121 bash[36902]: audit 2022-04-22T22:03:40.662954+0000 mgr.y (mgr.34107) 9470 : audit [DBG] from='client.55117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:42 smithi121 bash[36902]: cluster 2022-04-22T22:03:40.664349+0000 mgr.y (mgr.34107) 9471 : cluster [DBG] pgmap v10268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:42 smithi121 bash[36902]: audit 2022-04-22T22:03:41.657538+0000 mon.a (mon.0) 13326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:42 smithi121 bash[36902]: audit 2022-04-22T22:03:41.940311+0000 mon.a (mon.0) 13327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[45495]: audit 2022-04-22T22:03:40.662954+0000 mgr.y (mgr.34107) 9470 : audit [DBG] from='client.55117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[45495]: cluster 2022-04-22T22:03:40.664349+0000 mgr.y (mgr.34107) 9471 : cluster [DBG] pgmap v10268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[45495]: audit 2022-04-22T22:03:41.657538+0000 mon.a (mon.0) 13326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[45495]: audit 2022-04-22T22:03:41.940311+0000 mon.a (mon.0) 13327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[46934]: audit 2022-04-22T22:03:40.662954+0000 mgr.y (mgr.34107) 9470 : audit [DBG] from='client.55117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:03:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[46934]: cluster 2022-04-22T22:03:40.664349+0000 mgr.y (mgr.34107) 9471 : cluster [DBG] pgmap v10268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[46934]: audit 2022-04-22T22:03:41.657538+0000 mon.a (mon.0) 13326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[46934]: audit 2022-04-22T22:03:41.940311+0000 mon.a (mon.0) 13327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:42.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:44 smithi121 bash[36902]: cluster 2022-04-22T22:03:42.665520+0000 mgr.y (mgr.34107) 9472 : cluster [DBG] pgmap v10269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:44 smithi115 bash[45495]: cluster 2022-04-22T22:03:42.665520+0000 mgr.y (mgr.34107) 9472 : cluster [DBG] pgmap v10269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:44 smithi115 bash[46934]: cluster 2022-04-22T22:03:42.665520+0000 mgr.y (mgr.34107) 9472 : cluster [DBG] pgmap v10269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:44.564Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:46 smithi121 bash[36902]: cluster 2022-04-22T22:03:44.666709+0000 mgr.y (mgr.34107) 9473 : cluster [DBG] pgmap v10270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:46 smithi121 bash[36902]: audit 2022-04-22T22:03:45.529148+0000 mon.a (mon.0) 13328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:46 smithi115 bash[45495]: cluster 2022-04-22T22:03:44.666709+0000 mgr.y (mgr.34107) 9473 : cluster [DBG] pgmap v10270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:46 smithi115 bash[45495]: audit 2022-04-22T22:03:45.529148+0000 mon.a (mon.0) 13328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:46 smithi115 bash[46934]: cluster 2022-04-22T22:03:44.666709+0000 mgr.y (mgr.34107) 9473 : cluster [DBG] pgmap v10270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:46 smithi115 bash[46934]: audit 2022-04-22T22:03:45.529148+0000 mon.a (mon.0) 13328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:47 smithi121 bash[36902]: audit 2022-04-22T22:03:46.661193+0000 mon.a (mon.0) 13329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:47 smithi121 bash[36902]: audit 2022-04-22T22:03:46.943213+0000 mon.a (mon.0) 13330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:47 smithi115 bash[46934]: audit 2022-04-22T22:03:46.661193+0000 mon.a (mon.0) 13329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:47 smithi115 bash[46934]: audit 2022-04-22T22:03:46.943213+0000 mon.a (mon.0) 13330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:47 smithi115 bash[45495]: audit 2022-04-22T22:03:46.661193+0000 mon.a (mon.0) 13329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:47 smithi115 bash[45495]: audit 2022-04-22T22:03:46.943213+0000 mon.a (mon.0) 13330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:48 smithi121 bash[36902]: cluster 2022-04-22T22:03:46.667710+0000 mgr.y (mgr.34107) 9474 : cluster [DBG] pgmap v10271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:48 smithi115 bash[46934]: cluster 2022-04-22T22:03:46.667710+0000 mgr.y (mgr.34107) 9474 : cluster [DBG] pgmap v10271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:48 smithi115 bash[45495]: cluster 2022-04-22T22:03:46.667710+0000 mgr.y (mgr.34107) 9474 : cluster [DBG] pgmap v10271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:50 smithi121 bash[36902]: cluster 2022-04-22T22:03:48.668789+0000 mgr.y (mgr.34107) 9475 : cluster [DBG] pgmap v10272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:50 smithi115 bash[45495]: cluster 2022-04-22T22:03:48.668789+0000 mgr.y (mgr.34107) 9475 : cluster [DBG] pgmap v10272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:50 smithi115 bash[46934]: cluster 2022-04-22T22:03:48.668789+0000 mgr.y (mgr.34107) 9475 : cluster [DBG] pgmap v10272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:51 smithi121 bash[36902]: audit 2022-04-22T22:03:50.532249+0000 mon.a (mon.0) 13331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:51 smithi115 bash[45495]: audit 2022-04-22T22:03:50.532249+0000 mon.a (mon.0) 13331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:51 smithi115 bash[46934]: audit 2022-04-22T22:03:50.532249+0000 mon.a (mon.0) 13331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:52 smithi121 bash[36902]: cluster 2022-04-22T22:03:50.669902+0000 mgr.y (mgr.34107) 9476 : cluster [DBG] pgmap v10273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:52 smithi121 bash[36902]: audit 2022-04-22T22:03:51.665036+0000 mon.a (mon.0) 13332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:52 smithi121 bash[36902]: audit 2022-04-22T22:03:51.946236+0000 mon.a (mon.0) 13333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[45495]: cluster 2022-04-22T22:03:50.669902+0000 mgr.y (mgr.34107) 9476 : cluster [DBG] pgmap v10273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[45495]: audit 2022-04-22T22:03:51.665036+0000 mon.a (mon.0) 13332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[45495]: audit 2022-04-22T22:03:51.946236+0000 mon.a (mon.0) 13333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[46934]: cluster 2022-04-22T22:03:50.669902+0000 mgr.y (mgr.34107) 9476 : cluster [DBG] pgmap v10273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[46934]: audit 2022-04-22T22:03:51.665036+0000 mon.a (mon.0) 13332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[46934]: audit 2022-04-22T22:03:51.946236+0000 mon.a (mon.0) 13333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:52.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:54 smithi121 bash[36902]: cluster 2022-04-22T22:03:52.670994+0000 mgr.y (mgr.34107) 9477 : cluster [DBG] pgmap v10274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:54 smithi115 bash[45495]: cluster 2022-04-22T22:03:52.670994+0000 mgr.y (mgr.34107) 9477 : cluster [DBG] pgmap v10274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:54 smithi115 bash[46934]: cluster 2022-04-22T22:03:52.670994+0000 mgr.y (mgr.34107) 9477 : cluster [DBG] pgmap v10274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:03:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:03:54.566Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:03:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:56 smithi121 bash[36902]: cluster 2022-04-22T22:03:54.672069+0000 mgr.y (mgr.34107) 9478 : cluster [DBG] pgmap v10275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:56 smithi121 bash[36902]: audit 2022-04-22T22:03:55.535238+0000 mon.a (mon.0) 13334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:56 smithi115 bash[45495]: cluster 2022-04-22T22:03:54.672069+0000 mgr.y (mgr.34107) 9478 : cluster [DBG] pgmap v10275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:56 smithi115 bash[45495]: audit 2022-04-22T22:03:55.535238+0000 mon.a (mon.0) 13334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:56 smithi115 bash[46934]: cluster 2022-04-22T22:03:54.672069+0000 mgr.y (mgr.34107) 9478 : cluster [DBG] pgmap v10275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:56 smithi115 bash[46934]: audit 2022-04-22T22:03:55.535238+0000 mon.a (mon.0) 13334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:03:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:57 smithi121 bash[36902]: audit 2022-04-22T22:03:56.668857+0000 mon.a (mon.0) 13335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:57 smithi121 bash[36902]: audit 2022-04-22T22:03:56.948865+0000 mon.a (mon.0) 13336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:57 smithi115 bash[45495]: audit 2022-04-22T22:03:56.668857+0000 mon.a (mon.0) 13335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:57 smithi115 bash[45495]: audit 2022-04-22T22:03:56.948865+0000 mon.a (mon.0) 13336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:57 smithi115 bash[46934]: audit 2022-04-22T22:03:56.668857+0000 mon.a (mon.0) 13335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:03:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:57 smithi115 bash[46934]: audit 2022-04-22T22:03:56.948865+0000 mon.a (mon.0) 13336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:03:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:03:58 smithi121 bash[36902]: cluster 2022-04-22T22:03:56.672995+0000 mgr.y (mgr.34107) 9479 : cluster [DBG] pgmap v10276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:03:58 smithi115 bash[45495]: cluster 2022-04-22T22:03:56.672995+0000 mgr.y (mgr.34107) 9479 : cluster [DBG] pgmap v10276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:03:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:03:58 smithi115 bash[46934]: cluster 2022-04-22T22:03:56.672995+0000 mgr.y (mgr.34107) 9479 : cluster [DBG] pgmap v10276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:00 smithi121 bash[36902]: cluster 2022-04-22T22:03:58.674126+0000 mgr.y (mgr.34107) 9480 : cluster [DBG] pgmap v10277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:00 smithi115 bash[45495]: cluster 2022-04-22T22:03:58.674126+0000 mgr.y (mgr.34107) 9480 : cluster [DBG] pgmap v10277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:00 smithi115 bash[46934]: cluster 2022-04-22T22:03:58.674126+0000 mgr.y (mgr.34107) 9480 : cluster [DBG] pgmap v10277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:01 smithi121 bash[36902]: audit 2022-04-22T22:04:00.538527+0000 mon.a (mon.0) 13337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:01 smithi115 bash[45495]: audit 2022-04-22T22:04:00.538527+0000 mon.a (mon.0) 13337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:01 smithi115 bash[46934]: audit 2022-04-22T22:04:00.538527+0000 mon.a (mon.0) 13337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:02 smithi121 bash[36902]: cluster 2022-04-22T22:04:00.675092+0000 mgr.y (mgr.34107) 9481 : cluster [DBG] pgmap v10278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:02 smithi121 bash[36902]: audit 2022-04-22T22:04:01.672636+0000 mon.a (mon.0) 13338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:02 smithi121 bash[36902]: audit 2022-04-22T22:04:01.952167+0000 mon.a (mon.0) 13339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[45495]: cluster 2022-04-22T22:04:00.675092+0000 mgr.y (mgr.34107) 9481 : cluster [DBG] pgmap v10278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[45495]: audit 2022-04-22T22:04:01.672636+0000 mon.a (mon.0) 13338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[45495]: audit 2022-04-22T22:04:01.952167+0000 mon.a (mon.0) 13339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[46934]: cluster 2022-04-22T22:04:00.675092+0000 mgr.y (mgr.34107) 9481 : cluster [DBG] pgmap v10278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[46934]: audit 2022-04-22T22:04:01.672636+0000 mon.a (mon.0) 13338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[46934]: audit 2022-04-22T22:04:01.952167+0000 mon.a (mon.0) 13339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:02.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:04 smithi121 bash[36902]: cluster 2022-04-22T22:04:02.676256+0000 mgr.y (mgr.34107) 9482 : cluster [DBG] pgmap v10279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:04 smithi115 bash[45495]: cluster 2022-04-22T22:04:02.676256+0000 mgr.y (mgr.34107) 9482 : cluster [DBG] pgmap v10279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:04 smithi115 bash[46934]: cluster 2022-04-22T22:04:02.676256+0000 mgr.y (mgr.34107) 9482 : cluster [DBG] pgmap v10279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:04.566Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:06 smithi121 bash[36902]: cluster 2022-04-22T22:04:04.677358+0000 mgr.y (mgr.34107) 9483 : cluster [DBG] pgmap v10280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:06 smithi121 bash[36902]: audit 2022-04-22T22:04:05.541647+0000 mon.a (mon.0) 13340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:06.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:06 smithi115 bash[46934]: cluster 2022-04-22T22:04:04.677358+0000 mgr.y (mgr.34107) 9483 : cluster [DBG] pgmap v10280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:06 smithi115 bash[46934]: audit 2022-04-22T22:04:05.541647+0000 mon.a (mon.0) 13340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:06 smithi115 bash[45495]: cluster 2022-04-22T22:04:04.677358+0000 mgr.y (mgr.34107) 9483 : cluster [DBG] pgmap v10280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:06 smithi115 bash[45495]: audit 2022-04-22T22:04:05.541647+0000 mon.a (mon.0) 13340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:07 smithi121 bash[36902]: audit 2022-04-22T22:04:06.676274+0000 mon.a (mon.0) 13341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:07 smithi121 bash[36902]: audit 2022-04-22T22:04:06.955111+0000 mon.a (mon.0) 13342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:07 smithi115 bash[45495]: audit 2022-04-22T22:04:06.676274+0000 mon.a (mon.0) 13341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:07 smithi115 bash[45495]: audit 2022-04-22T22:04:06.955111+0000 mon.a (mon.0) 13342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:07 smithi115 bash[46934]: audit 2022-04-22T22:04:06.676274+0000 mon.a (mon.0) 13341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:07 smithi115 bash[46934]: audit 2022-04-22T22:04:06.955111+0000 mon.a (mon.0) 13342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:08 smithi121 bash[36902]: cluster 2022-04-22T22:04:06.678426+0000 mgr.y (mgr.34107) 9484 : cluster [DBG] pgmap v10281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:08 smithi115 bash[45495]: cluster 2022-04-22T22:04:06.678426+0000 mgr.y (mgr.34107) 9484 : cluster [DBG] pgmap v10281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:08 smithi115 bash[46934]: cluster 2022-04-22T22:04:06.678426+0000 mgr.y (mgr.34107) 9484 : cluster [DBG] pgmap v10281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:10 smithi121 bash[36902]: cluster 2022-04-22T22:04:08.679752+0000 mgr.y (mgr.34107) 9485 : cluster [DBG] pgmap v10282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:10 smithi115 bash[45495]: cluster 2022-04-22T22:04:08.679752+0000 mgr.y (mgr.34107) 9485 : cluster [DBG] pgmap v10282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:10 smithi115 bash[46934]: cluster 2022-04-22T22:04:08.679752+0000 mgr.y (mgr.34107) 9485 : cluster [DBG] pgmap v10282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:11.457 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:04:11.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:11 smithi121 bash[36902]: audit 2022-04-22T22:04:10.544636+0000 mon.a (mon.0) 13343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:11 smithi115 bash[45495]: audit 2022-04-22T22:04:10.544636+0000 mon.a (mon.0) 13343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:11 smithi115 bash[46934]: audit 2022-04-22T22:04:10.544636+0000 mon.a (mon.0) 13343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:11.793 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 21.9M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 262M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:04:11.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 584M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 646M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:04:11.795 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 65.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:04:12.199 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:04:12.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:04:12.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:04:12.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:12.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:04:12.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:12.201 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:04:12.201 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:04:12.201 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:04:12.201 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:04:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:12 smithi121 bash[36902]: cluster 2022-04-22T22:04:10.680843+0000 mgr.y (mgr.34107) 9486 : cluster [DBG] pgmap v10283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:12 smithi121 bash[36902]: audit 2022-04-22T22:04:11.680253+0000 mon.a (mon.0) 13344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:12 smithi121 bash[36902]: audit 2022-04-22T22:04:11.958030+0000 mon.a (mon.0) 13345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:12 smithi121 bash[36902]: audit 2022-04-22T22:04:12.195838+0000 mon.c (mon.1) 328 : audit [DBG] from='client.? 172.21.15.115:0/3339059018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[45495]: cluster 2022-04-22T22:04:10.680843+0000 mgr.y (mgr.34107) 9486 : cluster [DBG] pgmap v10283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[45495]: audit 2022-04-22T22:04:11.680253+0000 mon.a (mon.0) 13344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[45495]: audit 2022-04-22T22:04:11.958030+0000 mon.a (mon.0) 13345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[45495]: audit 2022-04-22T22:04:12.195838+0000 mon.c (mon.1) 328 : audit [DBG] from='client.? 172.21.15.115:0/3339059018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[46934]: cluster 2022-04-22T22:04:10.680843+0000 mgr.y (mgr.34107) 9486 : cluster [DBG] pgmap v10283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[46934]: audit 2022-04-22T22:04:11.680253+0000 mon.a (mon.0) 13344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[46934]: audit 2022-04-22T22:04:11.958030+0000 mon.a (mon.0) 13345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[46934]: audit 2022-04-22T22:04:12.195838+0000 mon.c (mon.1) 328 : audit [DBG] from='client.? 172.21.15.115:0/3339059018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:12.714Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:13 smithi121 bash[36902]: audit 2022-04-22T22:04:11.438047+0000 mgr.y (mgr.34107) 9487 : audit [DBG] from='client.55129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:13 smithi121 bash[36902]: audit 2022-04-22T22:04:11.776348+0000 mgr.y (mgr.34107) 9488 : audit [DBG] from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:13 smithi115 bash[45495]: audit 2022-04-22T22:04:11.438047+0000 mgr.y (mgr.34107) 9487 : audit [DBG] from='client.55129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:13 smithi115 bash[45495]: audit 2022-04-22T22:04:11.776348+0000 mgr.y (mgr.34107) 9488 : audit [DBG] from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:13 smithi115 bash[46934]: audit 2022-04-22T22:04:11.438047+0000 mgr.y (mgr.34107) 9487 : audit [DBG] from='client.55129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:13 smithi115 bash[46934]: audit 2022-04-22T22:04:11.776348+0000 mgr.y (mgr.34107) 9488 : audit [DBG] from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:14 smithi121 bash[36902]: cluster 2022-04-22T22:04:12.681963+0000 mgr.y (mgr.34107) 9489 : cluster [DBG] pgmap v10284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:14 smithi115 bash[45495]: cluster 2022-04-22T22:04:12.681963+0000 mgr.y (mgr.34107) 9489 : cluster [DBG] pgmap v10284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:14 smithi115 bash[46934]: cluster 2022-04-22T22:04:12.681963+0000 mgr.y (mgr.34107) 9489 : cluster [DBG] pgmap v10284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:14.567Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:16 smithi115 bash[45495]: cluster 2022-04-22T22:04:14.683135+0000 mgr.y (mgr.34107) 9490 : cluster [DBG] pgmap v10285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:16 smithi115 bash[45495]: audit 2022-04-22T22:04:15.547289+0000 mon.a (mon.0) 13346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:16 smithi115 bash[46934]: cluster 2022-04-22T22:04:14.683135+0000 mgr.y (mgr.34107) 9490 : cluster [DBG] pgmap v10285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:16 smithi115 bash[46934]: audit 2022-04-22T22:04:15.547289+0000 mon.a (mon.0) 13346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:16 smithi121 bash[36902]: cluster 2022-04-22T22:04:14.683135+0000 mgr.y (mgr.34107) 9490 : cluster [DBG] pgmap v10285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:16 smithi121 bash[36902]: audit 2022-04-22T22:04:15.547289+0000 mon.a (mon.0) 13346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[45495]: audit 2022-04-22T22:04:16.482794+0000 mon.a (mon.0) 13347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:04:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[45495]: audit 2022-04-22T22:04:16.483310+0000 mon.a (mon.0) 13348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:04:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[45495]: audit 2022-04-22T22:04:16.683814+0000 mon.a (mon.0) 13349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[45495]: audit 2022-04-22T22:04:16.960981+0000 mon.a (mon.0) 13350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[46934]: audit 2022-04-22T22:04:16.482794+0000 mon.a (mon.0) 13347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:04:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[46934]: audit 2022-04-22T22:04:16.483310+0000 mon.a (mon.0) 13348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:04:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[46934]: audit 2022-04-22T22:04:16.683814+0000 mon.a (mon.0) 13349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:17 smithi115 bash[46934]: audit 2022-04-22T22:04:16.960981+0000 mon.a (mon.0) 13350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:17 smithi121 bash[36902]: audit 2022-04-22T22:04:16.482794+0000 mon.a (mon.0) 13347 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:04:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:17 smithi121 bash[36902]: audit 2022-04-22T22:04:16.483310+0000 mon.a (mon.0) 13348 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:04:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:17 smithi121 bash[36902]: audit 2022-04-22T22:04:16.683814+0000 mon.a (mon.0) 13349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:17 smithi121 bash[36902]: audit 2022-04-22T22:04:16.960981+0000 mon.a (mon.0) 13350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:18 smithi115 bash[45495]: cluster 2022-04-22T22:04:16.684427+0000 mgr.y (mgr.34107) 9491 : cluster [DBG] pgmap v10286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:18.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:18 smithi115 bash[46934]: cluster 2022-04-22T22:04:16.684427+0000 mgr.y (mgr.34107) 9491 : cluster [DBG] pgmap v10286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:18 smithi121 bash[36902]: cluster 2022-04-22T22:04:16.684427+0000 mgr.y (mgr.34107) 9491 : cluster [DBG] pgmap v10286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:20 smithi115 bash[45495]: cluster 2022-04-22T22:04:18.685633+0000 mgr.y (mgr.34107) 9492 : cluster [DBG] pgmap v10287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:20 smithi115 bash[46934]: cluster 2022-04-22T22:04:18.685633+0000 mgr.y (mgr.34107) 9492 : cluster [DBG] pgmap v10287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:20 smithi121 bash[36902]: cluster 2022-04-22T22:04:18.685633+0000 mgr.y (mgr.34107) 9492 : cluster [DBG] pgmap v10287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:21 smithi115 bash[45495]: audit 2022-04-22T22:04:20.550285+0000 mon.a (mon.0) 13351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:21 smithi115 bash[46934]: audit 2022-04-22T22:04:20.550285+0000 mon.a (mon.0) 13351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:21 smithi121 bash[36902]: audit 2022-04-22T22:04:20.550285+0000 mon.a (mon.0) 13351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:22.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[45495]: cluster 2022-04-22T22:04:20.686738+0000 mgr.y (mgr.34107) 9493 : cluster [DBG] pgmap v10288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:22.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[45495]: audit 2022-04-22T22:04:21.688764+0000 mon.a (mon.0) 13352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:22.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[45495]: audit 2022-04-22T22:04:21.964058+0000 mon.a (mon.0) 13353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:22.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[46934]: cluster 2022-04-22T22:04:20.686738+0000 mgr.y (mgr.34107) 9493 : cluster [DBG] pgmap v10288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:22.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[46934]: audit 2022-04-22T22:04:21.688764+0000 mon.a (mon.0) 13352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:22.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[46934]: audit 2022-04-22T22:04:21.964058+0000 mon.a (mon.0) 13353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:22 smithi121 bash[36902]: cluster 2022-04-22T22:04:20.686738+0000 mgr.y (mgr.34107) 9493 : cluster [DBG] pgmap v10288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:22 smithi121 bash[36902]: audit 2022-04-22T22:04:21.688764+0000 mon.a (mon.0) 13352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:22 smithi121 bash[36902]: audit 2022-04-22T22:04:21.964058+0000 mon.a (mon.0) 13353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:22.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:24.570 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:24 smithi115 bash[45495]: cluster 2022-04-22T22:04:22.687883+0000 mgr.y (mgr.34107) 9494 : cluster [DBG] pgmap v10289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:24.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:24 smithi115 bash[46934]: cluster 2022-04-22T22:04:22.687883+0000 mgr.y (mgr.34107) 9494 : cluster [DBG] pgmap v10289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:24.612 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:24 smithi121 bash[36902]: cluster 2022-04-22T22:04:22.687883+0000 mgr.y (mgr.34107) 9494 : cluster [DBG] pgmap v10289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:24.941 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:24.567Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:25 smithi121 bash[36902]: audit 2022-04-22T22:04:24.583469+0000 mon.a (mon.0) 13354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:04:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:25 smithi121 bash[36902]: audit 2022-04-22T22:04:24.586396+0000 mon.a (mon.0) 13355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:04:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:25 smithi121 bash[36902]: audit 2022-04-22T22:04:24.588080+0000 mon.a (mon.0) 13356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:04:25.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[46934]: audit 2022-04-22T22:04:24.583469+0000 mon.a (mon.0) 13354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:04:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[46934]: audit 2022-04-22T22:04:24.586396+0000 mon.a (mon.0) 13355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:04:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[46934]: audit 2022-04-22T22:04:24.588080+0000 mon.a (mon.0) 13356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:04:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[45495]: audit 2022-04-22T22:04:24.583469+0000 mon.a (mon.0) 13354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:04:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[45495]: audit 2022-04-22T22:04:24.586396+0000 mon.a (mon.0) 13355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:04:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:25 smithi115 bash[45495]: audit 2022-04-22T22:04:24.588080+0000 mon.a (mon.0) 13356 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:04:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:26 smithi121 bash[36902]: cluster 2022-04-22T22:04:24.688829+0000 mgr.y (mgr.34107) 9495 : cluster [DBG] pgmap v10290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:26 smithi121 bash[36902]: audit 2022-04-22T22:04:25.553535+0000 mon.a (mon.0) 13357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:26 smithi115 bash[45495]: cluster 2022-04-22T22:04:24.688829+0000 mgr.y (mgr.34107) 9495 : cluster [DBG] pgmap v10290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:26 smithi115 bash[45495]: audit 2022-04-22T22:04:25.553535+0000 mon.a (mon.0) 13357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:26 smithi115 bash[46934]: cluster 2022-04-22T22:04:24.688829+0000 mgr.y (mgr.34107) 9495 : cluster [DBG] pgmap v10290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:26 smithi115 bash[46934]: audit 2022-04-22T22:04:25.553535+0000 mon.a (mon.0) 13357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:27 smithi121 bash[36902]: audit 2022-04-22T22:04:26.692381+0000 mon.a (mon.0) 13358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:27.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:27 smithi121 bash[36902]: audit 2022-04-22T22:04:26.966635+0000 mon.a (mon.0) 13359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:27 smithi115 bash[45495]: audit 2022-04-22T22:04:26.692381+0000 mon.a (mon.0) 13358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:27 smithi115 bash[45495]: audit 2022-04-22T22:04:26.966635+0000 mon.a (mon.0) 13359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:27 smithi115 bash[46934]: audit 2022-04-22T22:04:26.692381+0000 mon.a (mon.0) 13358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:27 smithi115 bash[46934]: audit 2022-04-22T22:04:26.966635+0000 mon.a (mon.0) 13359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:28.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:28 smithi121 bash[36902]: cluster 2022-04-22T22:04:26.689879+0000 mgr.y (mgr.34107) 9496 : cluster [DBG] pgmap v10291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:28 smithi115 bash[45495]: cluster 2022-04-22T22:04:26.689879+0000 mgr.y (mgr.34107) 9496 : cluster [DBG] pgmap v10291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:28 smithi115 bash[46934]: cluster 2022-04-22T22:04:26.689879+0000 mgr.y (mgr.34107) 9496 : cluster [DBG] pgmap v10291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:30 smithi121 bash[36902]: cluster 2022-04-22T22:04:28.691112+0000 mgr.y (mgr.34107) 9497 : cluster [DBG] pgmap v10292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:30 smithi121 bash[36902]: audit 2022-04-22T22:04:29.011215+0000 mon.a (mon.0) 13360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:30 smithi121 bash[36902]: audit 2022-04-22T22:04:29.364435+0000 mon.a (mon.0) 13361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[45495]: cluster 2022-04-22T22:04:28.691112+0000 mgr.y (mgr.34107) 9497 : cluster [DBG] pgmap v10292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[45495]: audit 2022-04-22T22:04:29.011215+0000 mon.a (mon.0) 13360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[45495]: audit 2022-04-22T22:04:29.364435+0000 mon.a (mon.0) 13361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[46934]: cluster 2022-04-22T22:04:28.691112+0000 mgr.y (mgr.34107) 9497 : cluster [DBG] pgmap v10292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[46934]: audit 2022-04-22T22:04:29.011215+0000 mon.a (mon.0) 13360 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:30 smithi115 bash[46934]: audit 2022-04-22T22:04:29.364435+0000 mon.a (mon.0) 13361 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:04:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:31 smithi115 bash[45495]: audit 2022-04-22T22:04:30.556520+0000 mon.a (mon.0) 13362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:31 smithi115 bash[46934]: audit 2022-04-22T22:04:30.556520+0000 mon.a (mon.0) 13362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:31 smithi121 bash[36902]: audit 2022-04-22T22:04:30.556520+0000 mon.a (mon.0) 13362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:32 smithi121 bash[36902]: cluster 2022-04-22T22:04:30.692221+0000 mgr.y (mgr.34107) 9498 : cluster [DBG] pgmap v10293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:32 smithi121 bash[36902]: audit 2022-04-22T22:04:31.696263+0000 mon.a (mon.0) 13363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:32 smithi121 bash[36902]: audit 2022-04-22T22:04:31.968966+0000 mon.a (mon.0) 13364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[45495]: cluster 2022-04-22T22:04:30.692221+0000 mgr.y (mgr.34107) 9498 : cluster [DBG] pgmap v10293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[45495]: audit 2022-04-22T22:04:31.696263+0000 mon.a (mon.0) 13363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[45495]: audit 2022-04-22T22:04:31.968966+0000 mon.a (mon.0) 13364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[46934]: cluster 2022-04-22T22:04:30.692221+0000 mgr.y (mgr.34107) 9498 : cluster [DBG] pgmap v10293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[46934]: audit 2022-04-22T22:04:31.696263+0000 mon.a (mon.0) 13363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[46934]: audit 2022-04-22T22:04:31.968966+0000 mon.a (mon.0) 13364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:32.714Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:34 smithi121 bash[36902]: cluster 2022-04-22T22:04:32.693379+0000 mgr.y (mgr.34107) 9499 : cluster [DBG] pgmap v10294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:34.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:34 smithi115 bash[46934]: cluster 2022-04-22T22:04:32.693379+0000 mgr.y (mgr.34107) 9499 : cluster [DBG] pgmap v10294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:34 smithi115 bash[45495]: cluster 2022-04-22T22:04:32.693379+0000 mgr.y (mgr.34107) 9499 : cluster [DBG] pgmap v10294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:34.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:36 smithi121 bash[36902]: cluster 2022-04-22T22:04:34.694505+0000 mgr.y (mgr.34107) 9500 : cluster [DBG] pgmap v10295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:36 smithi121 bash[36902]: audit 2022-04-22T22:04:35.559702+0000 mon.a (mon.0) 13365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:36 smithi115 bash[45495]: cluster 2022-04-22T22:04:34.694505+0000 mgr.y (mgr.34107) 9500 : cluster [DBG] pgmap v10295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:36 smithi115 bash[45495]: audit 2022-04-22T22:04:35.559702+0000 mon.a (mon.0) 13365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:36 smithi115 bash[46934]: cluster 2022-04-22T22:04:34.694505+0000 mgr.y (mgr.34107) 9500 : cluster [DBG] pgmap v10295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:36 smithi115 bash[46934]: audit 2022-04-22T22:04:35.559702+0000 mon.a (mon.0) 13365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:37 smithi121 bash[36902]: audit 2022-04-22T22:04:36.699790+0000 mon.a (mon.0) 13366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:37 smithi121 bash[36902]: audit 2022-04-22T22:04:36.971949+0000 mon.a (mon.0) 13367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:37 smithi115 bash[45495]: audit 2022-04-22T22:04:36.699790+0000 mon.a (mon.0) 13366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:37 smithi115 bash[45495]: audit 2022-04-22T22:04:36.971949+0000 mon.a (mon.0) 13367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:37 smithi115 bash[46934]: audit 2022-04-22T22:04:36.699790+0000 mon.a (mon.0) 13366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:37 smithi115 bash[46934]: audit 2022-04-22T22:04:36.971949+0000 mon.a (mon.0) 13367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:38 smithi121 bash[36902]: cluster 2022-04-22T22:04:36.695643+0000 mgr.y (mgr.34107) 9501 : cluster [DBG] pgmap v10296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:38 smithi115 bash[45495]: cluster 2022-04-22T22:04:36.695643+0000 mgr.y (mgr.34107) 9501 : cluster [DBG] pgmap v10296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:38 smithi115 bash[46934]: cluster 2022-04-22T22:04:36.695643+0000 mgr.y (mgr.34107) 9501 : cluster [DBG] pgmap v10296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:40 smithi121 bash[36902]: cluster 2022-04-22T22:04:38.696764+0000 mgr.y (mgr.34107) 9502 : cluster [DBG] pgmap v10297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:40 smithi115 bash[45495]: cluster 2022-04-22T22:04:38.696764+0000 mgr.y (mgr.34107) 9502 : cluster [DBG] pgmap v10297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:40 smithi115 bash[46934]: cluster 2022-04-22T22:04:38.696764+0000 mgr.y (mgr.34107) 9502 : cluster [DBG] pgmap v10297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:41 smithi121 bash[36902]: audit 2022-04-22T22:04:40.562802+0000 mon.a (mon.0) 13368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:41 smithi115 bash[45495]: audit 2022-04-22T22:04:40.562802+0000 mon.a (mon.0) 13368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:41 smithi115 bash[46934]: audit 2022-04-22T22:04:40.562802+0000 mon.a (mon.0) 13368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:42 smithi121 bash[36902]: cluster 2022-04-22T22:04:40.697829+0000 mgr.y (mgr.34107) 9503 : cluster [DBG] pgmap v10298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:42 smithi121 bash[36902]: audit 2022-04-22T22:04:41.703593+0000 mon.a (mon.0) 13369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:42 smithi121 bash[36902]: audit 2022-04-22T22:04:41.974858+0000 mon.a (mon.0) 13370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[45495]: cluster 2022-04-22T22:04:40.697829+0000 mgr.y (mgr.34107) 9503 : cluster [DBG] pgmap v10298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[45495]: audit 2022-04-22T22:04:41.703593+0000 mon.a (mon.0) 13369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[45495]: audit 2022-04-22T22:04:41.974858+0000 mon.a (mon.0) 13370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[46934]: cluster 2022-04-22T22:04:40.697829+0000 mgr.y (mgr.34107) 9503 : cluster [DBG] pgmap v10298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[46934]: audit 2022-04-22T22:04:41.703593+0000 mon.a (mon.0) 13369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[46934]: audit 2022-04-22T22:04:41.974858+0000 mon.a (mon.0) 13370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:42.581 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:04:42.929 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:04:42.929 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:04:42.929 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 13s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 13s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 13s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 13s ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:04:42.930 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:04:42.931 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:04:42.931 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 13s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:04:42.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:04:42.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:04:42.931 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 13s ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 13s ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 13s ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 13s ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:04:42.932 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 13s ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:04:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:42.711Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:43.333 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:04:43.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:04:43.335 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:04:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:44 smithi121 bash[36902]: audit 2022-04-22T22:04:42.561810+0000 mgr.y (mgr.34107) 9504 : audit [DBG] from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:44 smithi121 bash[36902]: cluster 2022-04-22T22:04:42.698736+0000 mgr.y (mgr.34107) 9505 : cluster [DBG] pgmap v10299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:44 smithi121 bash[36902]: audit 2022-04-22T22:04:42.905539+0000 mgr.y (mgr.34107) 9506 : audit [DBG] from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:44 smithi121 bash[36902]: audit 2022-04-22T22:04:43.330067+0000 mon.c (mon.1) 329 : audit [DBG] from='client.? 172.21.15.115:0/581115933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[45495]: audit 2022-04-22T22:04:42.561810+0000 mgr.y (mgr.34107) 9504 : audit [DBG] from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[45495]: cluster 2022-04-22T22:04:42.698736+0000 mgr.y (mgr.34107) 9505 : cluster [DBG] pgmap v10299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[45495]: audit 2022-04-22T22:04:42.905539+0000 mgr.y (mgr.34107) 9506 : audit [DBG] from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[45495]: audit 2022-04-22T22:04:43.330067+0000 mon.c (mon.1) 329 : audit [DBG] from='client.? 172.21.15.115:0/581115933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[46934]: audit 2022-04-22T22:04:42.561810+0000 mgr.y (mgr.34107) 9504 : audit [DBG] from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[46934]: cluster 2022-04-22T22:04:42.698736+0000 mgr.y (mgr.34107) 9505 : cluster [DBG] pgmap v10299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[46934]: audit 2022-04-22T22:04:42.905539+0000 mgr.y (mgr.34107) 9506 : audit [DBG] from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:04:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[46934]: audit 2022-04-22T22:04:43.330067+0000 mon.c (mon.1) 329 : audit [DBG] from='client.? 172.21.15.115:0/581115933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:04:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:44.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:46 smithi121 bash[36902]: cluster 2022-04-22T22:04:44.699826+0000 mgr.y (mgr.34107) 9507 : cluster [DBG] pgmap v10300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:46 smithi121 bash[36902]: audit 2022-04-22T22:04:45.565979+0000 mon.a (mon.0) 13371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:46.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:46 smithi115 bash[46934]: cluster 2022-04-22T22:04:44.699826+0000 mgr.y (mgr.34107) 9507 : cluster [DBG] pgmap v10300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:46 smithi115 bash[46934]: audit 2022-04-22T22:04:45.565979+0000 mon.a (mon.0) 13371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:46 smithi115 bash[45495]: cluster 2022-04-22T22:04:44.699826+0000 mgr.y (mgr.34107) 9507 : cluster [DBG] pgmap v10300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:46 smithi115 bash[45495]: audit 2022-04-22T22:04:45.565979+0000 mon.a (mon.0) 13371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:47 smithi121 bash[36902]: audit 2022-04-22T22:04:46.707206+0000 mon.a (mon.0) 13372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:47 smithi121 bash[36902]: audit 2022-04-22T22:04:46.977750+0000 mon.a (mon.0) 13373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:47.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:47 smithi115 bash[46934]: audit 2022-04-22T22:04:46.707206+0000 mon.a (mon.0) 13372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:47 smithi115 bash[46934]: audit 2022-04-22T22:04:46.977750+0000 mon.a (mon.0) 13373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:47 smithi115 bash[45495]: audit 2022-04-22T22:04:46.707206+0000 mon.a (mon.0) 13372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:47 smithi115 bash[45495]: audit 2022-04-22T22:04:46.977750+0000 mon.a (mon.0) 13373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:48 smithi121 bash[36902]: cluster 2022-04-22T22:04:46.700940+0000 mgr.y (mgr.34107) 9508 : cluster [DBG] pgmap v10301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:48 smithi115 bash[46934]: cluster 2022-04-22T22:04:46.700940+0000 mgr.y (mgr.34107) 9508 : cluster [DBG] pgmap v10301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:48 smithi115 bash[45495]: cluster 2022-04-22T22:04:46.700940+0000 mgr.y (mgr.34107) 9508 : cluster [DBG] pgmap v10301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:50 smithi121 bash[36902]: cluster 2022-04-22T22:04:48.702159+0000 mgr.y (mgr.34107) 9509 : cluster [DBG] pgmap v10302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:50 smithi115 bash[46934]: cluster 2022-04-22T22:04:48.702159+0000 mgr.y (mgr.34107) 9509 : cluster [DBG] pgmap v10302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:50 smithi115 bash[45495]: cluster 2022-04-22T22:04:48.702159+0000 mgr.y (mgr.34107) 9509 : cluster [DBG] pgmap v10302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:51 smithi121 bash[36902]: audit 2022-04-22T22:04:50.569073+0000 mon.a (mon.0) 13374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:51.569 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:51 smithi115 bash[45495]: audit 2022-04-22T22:04:50.569073+0000 mon.a (mon.0) 13374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:51.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:51 smithi115 bash[46934]: audit 2022-04-22T22:04:50.569073+0000 mon.a (mon.0) 13374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:52 smithi121 bash[36902]: cluster 2022-04-22T22:04:50.703233+0000 mgr.y (mgr.34107) 9510 : cluster [DBG] pgmap v10303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:52 smithi121 bash[36902]: audit 2022-04-22T22:04:51.710943+0000 mon.a (mon.0) 13375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:52 smithi121 bash[36902]: audit 2022-04-22T22:04:51.980881+0000 mon.a (mon.0) 13376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[45495]: cluster 2022-04-22T22:04:50.703233+0000 mgr.y (mgr.34107) 9510 : cluster [DBG] pgmap v10303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[45495]: audit 2022-04-22T22:04:51.710943+0000 mon.a (mon.0) 13375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[45495]: audit 2022-04-22T22:04:51.980881+0000 mon.a (mon.0) 13376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[46934]: cluster 2022-04-22T22:04:50.703233+0000 mgr.y (mgr.34107) 9510 : cluster [DBG] pgmap v10303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[46934]: audit 2022-04-22T22:04:51.710943+0000 mon.a (mon.0) 13375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[46934]: audit 2022-04-22T22:04:51.980881+0000 mon.a (mon.0) 13376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:52.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:54 smithi121 bash[36902]: cluster 2022-04-22T22:04:52.704337+0000 mgr.y (mgr.34107) 9511 : cluster [DBG] pgmap v10304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:54 smithi115 bash[45495]: cluster 2022-04-22T22:04:52.704337+0000 mgr.y (mgr.34107) 9511 : cluster [DBG] pgmap v10304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:54 smithi115 bash[46934]: cluster 2022-04-22T22:04:52.704337+0000 mgr.y (mgr.34107) 9511 : cluster [DBG] pgmap v10304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:04:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:04:54.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:04:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:56 smithi121 bash[36902]: cluster 2022-04-22T22:04:54.705445+0000 mgr.y (mgr.34107) 9512 : cluster [DBG] pgmap v10305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:56 smithi121 bash[36902]: audit 2022-04-22T22:04:55.572080+0000 mon.a (mon.0) 13377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:56 smithi115 bash[45495]: cluster 2022-04-22T22:04:54.705445+0000 mgr.y (mgr.34107) 9512 : cluster [DBG] pgmap v10305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:56 smithi115 bash[45495]: audit 2022-04-22T22:04:55.572080+0000 mon.a (mon.0) 13377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:56 smithi115 bash[46934]: cluster 2022-04-22T22:04:54.705445+0000 mgr.y (mgr.34107) 9512 : cluster [DBG] pgmap v10305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:56 smithi115 bash[46934]: audit 2022-04-22T22:04:55.572080+0000 mon.a (mon.0) 13377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:04:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:57 smithi121 bash[36902]: audit 2022-04-22T22:04:56.714661+0000 mon.a (mon.0) 13378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:57 smithi121 bash[36902]: audit 2022-04-22T22:04:56.983870+0000 mon.a (mon.0) 13379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:57 smithi115 bash[45495]: audit 2022-04-22T22:04:56.714661+0000 mon.a (mon.0) 13378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:57 smithi115 bash[45495]: audit 2022-04-22T22:04:56.983870+0000 mon.a (mon.0) 13379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:57 smithi115 bash[46934]: audit 2022-04-22T22:04:56.714661+0000 mon.a (mon.0) 13378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:04:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:57 smithi115 bash[46934]: audit 2022-04-22T22:04:56.983870+0000 mon.a (mon.0) 13379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:04:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:04:58 smithi121 bash[36902]: cluster 2022-04-22T22:04:56.706549+0000 mgr.y (mgr.34107) 9513 : cluster [DBG] pgmap v10306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:04:58 smithi115 bash[45495]: cluster 2022-04-22T22:04:56.706549+0000 mgr.y (mgr.34107) 9513 : cluster [DBG] pgmap v10306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:04:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:04:58 smithi115 bash[46934]: cluster 2022-04-22T22:04:56.706549+0000 mgr.y (mgr.34107) 9513 : cluster [DBG] pgmap v10306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:00 smithi121 bash[36902]: cluster 2022-04-22T22:04:58.707782+0000 mgr.y (mgr.34107) 9514 : cluster [DBG] pgmap v10307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:00 smithi115 bash[45495]: cluster 2022-04-22T22:04:58.707782+0000 mgr.y (mgr.34107) 9514 : cluster [DBG] pgmap v10307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:00 smithi115 bash[46934]: cluster 2022-04-22T22:04:58.707782+0000 mgr.y (mgr.34107) 9514 : cluster [DBG] pgmap v10307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:01 smithi115 bash[45495]: audit 2022-04-22T22:05:00.575174+0000 mon.a (mon.0) 13380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:01 smithi115 bash[46934]: audit 2022-04-22T22:05:00.575174+0000 mon.a (mon.0) 13380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:01 smithi121 bash[36902]: audit 2022-04-22T22:05:00.575174+0000 mon.a (mon.0) 13380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:02 smithi121 bash[36902]: cluster 2022-04-22T22:05:00.708841+0000 mgr.y (mgr.34107) 9515 : cluster [DBG] pgmap v10308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:02 smithi121 bash[36902]: audit 2022-04-22T22:05:01.718291+0000 mon.a (mon.0) 13381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:02 smithi121 bash[36902]: audit 2022-04-22T22:05:01.987158+0000 mon.a (mon.0) 13382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[45495]: cluster 2022-04-22T22:05:00.708841+0000 mgr.y (mgr.34107) 9515 : cluster [DBG] pgmap v10308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[45495]: audit 2022-04-22T22:05:01.718291+0000 mon.a (mon.0) 13381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[45495]: audit 2022-04-22T22:05:01.987158+0000 mon.a (mon.0) 13382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[46934]: cluster 2022-04-22T22:05:00.708841+0000 mgr.y (mgr.34107) 9515 : cluster [DBG] pgmap v10308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[46934]: audit 2022-04-22T22:05:01.718291+0000 mon.a (mon.0) 13381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[46934]: audit 2022-04-22T22:05:01.987158+0000 mon.a (mon.0) 13382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:02.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:04 smithi121 bash[36902]: cluster 2022-04-22T22:05:02.709912+0000 mgr.y (mgr.34107) 9516 : cluster [DBG] pgmap v10309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:04 smithi115 bash[46934]: cluster 2022-04-22T22:05:02.709912+0000 mgr.y (mgr.34107) 9516 : cluster [DBG] pgmap v10309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:04 smithi115 bash[45495]: cluster 2022-04-22T22:05:02.709912+0000 mgr.y (mgr.34107) 9516 : cluster [DBG] pgmap v10309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:04.569Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:06 smithi121 bash[36902]: cluster 2022-04-22T22:05:04.710930+0000 mgr.y (mgr.34107) 9517 : cluster [DBG] pgmap v10310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:06 smithi121 bash[36902]: audit 2022-04-22T22:05:05.578351+0000 mon.a (mon.0) 13383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:06 smithi115 bash[45495]: cluster 2022-04-22T22:05:04.710930+0000 mgr.y (mgr.34107) 9517 : cluster [DBG] pgmap v10310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:06 smithi115 bash[45495]: audit 2022-04-22T22:05:05.578351+0000 mon.a (mon.0) 13383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:06 smithi115 bash[46934]: cluster 2022-04-22T22:05:04.710930+0000 mgr.y (mgr.34107) 9517 : cluster [DBG] pgmap v10310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:06.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:06 smithi115 bash[46934]: audit 2022-04-22T22:05:05.578351+0000 mon.a (mon.0) 13383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:07 smithi121 bash[36902]: audit 2022-04-22T22:05:06.721936+0000 mon.a (mon.0) 13384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:07 smithi121 bash[36902]: audit 2022-04-22T22:05:06.990187+0000 mon.a (mon.0) 13385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:07.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:07 smithi115 bash[46934]: audit 2022-04-22T22:05:06.721936+0000 mon.a (mon.0) 13384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:07 smithi115 bash[46934]: audit 2022-04-22T22:05:06.990187+0000 mon.a (mon.0) 13385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:07 smithi115 bash[45495]: audit 2022-04-22T22:05:06.721936+0000 mon.a (mon.0) 13384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:07 smithi115 bash[45495]: audit 2022-04-22T22:05:06.990187+0000 mon.a (mon.0) 13385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:08 smithi121 bash[36902]: cluster 2022-04-22T22:05:06.712052+0000 mgr.y (mgr.34107) 9518 : cluster [DBG] pgmap v10311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:08.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:08 smithi115 bash[46934]: cluster 2022-04-22T22:05:06.712052+0000 mgr.y (mgr.34107) 9518 : cluster [DBG] pgmap v10311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:08 smithi115 bash[45495]: cluster 2022-04-22T22:05:06.712052+0000 mgr.y (mgr.34107) 9518 : cluster [DBG] pgmap v10311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:10 smithi121 bash[36902]: cluster 2022-04-22T22:05:08.713187+0000 mgr.y (mgr.34107) 9519 : cluster [DBG] pgmap v10312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:10.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:10 smithi115 bash[46934]: cluster 2022-04-22T22:05:08.713187+0000 mgr.y (mgr.34107) 9519 : cluster [DBG] pgmap v10312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:10 smithi115 bash[45495]: cluster 2022-04-22T22:05:08.713187+0000 mgr.y (mgr.34107) 9519 : cluster [DBG] pgmap v10312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:11 smithi121 bash[36902]: audit 2022-04-22T22:05:10.581564+0000 mon.a (mon.0) 13386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:11.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:11 smithi115 bash[46934]: audit 2022-04-22T22:05:10.581564+0000 mon.a (mon.0) 13386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:11 smithi115 bash[45495]: audit 2022-04-22T22:05:10.581564+0000 mon.a (mon.0) 13386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:12 smithi121 bash[36902]: cluster 2022-04-22T22:05:10.714151+0000 mgr.y (mgr.34107) 9520 : cluster [DBG] pgmap v10313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:12 smithi121 bash[36902]: audit 2022-04-22T22:05:11.725505+0000 mon.a (mon.0) 13387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:12 smithi121 bash[36902]: audit 2022-04-22T22:05:11.992967+0000 mon.a (mon.0) 13388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[45495]: cluster 2022-04-22T22:05:10.714151+0000 mgr.y (mgr.34107) 9520 : cluster [DBG] pgmap v10313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[45495]: audit 2022-04-22T22:05:11.725505+0000 mon.a (mon.0) 13387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[45495]: audit 2022-04-22T22:05:11.992967+0000 mon.a (mon.0) 13388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[46934]: cluster 2022-04-22T22:05:10.714151+0000 mgr.y (mgr.34107) 9520 : cluster [DBG] pgmap v10313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[46934]: audit 2022-04-22T22:05:11.725505+0000 mon.a (mon.0) 13387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[46934]: audit 2022-04-22T22:05:11.992967+0000 mon.a (mon.0) 13388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:12.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:13.718 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:05:14.066 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:05:14.066 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T22:05:14.066 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 45s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 45s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 45s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 407M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 45s ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 269M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 45s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:05:14.067 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 714M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 794M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 498M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 45s ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 45s ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 45s ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 45s ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:05:14.068 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 45s ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:05:14.466 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:05:14.489 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:05:14.490 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:05:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:14 smithi121 bash[36902]: cluster 2022-04-22T22:05:12.715284+0000 mgr.y (mgr.34107) 9521 : cluster [DBG] pgmap v10314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:14 smithi115 bash[46934]: cluster 2022-04-22T22:05:12.715284+0000 mgr.y (mgr.34107) 9521 : cluster [DBG] pgmap v10314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:14 smithi115 bash[45495]: cluster 2022-04-22T22:05:12.715284+0000 mgr.y (mgr.34107) 9521 : cluster [DBG] pgmap v10314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:14.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:15 smithi121 bash[36902]: audit 2022-04-22T22:05:13.697155+0000 mgr.y (mgr.34107) 9522 : audit [DBG] from='client.55165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:15 smithi121 bash[36902]: audit 2022-04-22T22:05:14.044397+0000 mgr.y (mgr.34107) 9523 : audit [DBG] from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:15 smithi121 bash[36902]: audit 2022-04-22T22:05:14.462577+0000 mon.a (mon.0) 13389 : audit [DBG] from='client.? 172.21.15.115:0/681112622' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[45495]: audit 2022-04-22T22:05:13.697155+0000 mgr.y (mgr.34107) 9522 : audit [DBG] from='client.55165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[45495]: audit 2022-04-22T22:05:14.044397+0000 mgr.y (mgr.34107) 9523 : audit [DBG] from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[45495]: audit 2022-04-22T22:05:14.462577+0000 mon.a (mon.0) 13389 : audit [DBG] from='client.? 172.21.15.115:0/681112622' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[46934]: audit 2022-04-22T22:05:13.697155+0000 mgr.y (mgr.34107) 9522 : audit [DBG] from='client.55165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[46934]: audit 2022-04-22T22:05:14.044397+0000 mgr.y (mgr.34107) 9523 : audit [DBG] from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:15 smithi115 bash[46934]: audit 2022-04-22T22:05:14.462577+0000 mon.a (mon.0) 13389 : audit [DBG] from='client.? 172.21.15.115:0/681112622' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:16 smithi121 bash[36902]: cluster 2022-04-22T22:05:14.716414+0000 mgr.y (mgr.34107) 9524 : cluster [DBG] pgmap v10315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:16 smithi121 bash[36902]: audit 2022-04-22T22:05:15.584773+0000 mon.a (mon.0) 13390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:16 smithi115 bash[45495]: cluster 2022-04-22T22:05:14.716414+0000 mgr.y (mgr.34107) 9524 : cluster [DBG] pgmap v10315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:16 smithi115 bash[45495]: audit 2022-04-22T22:05:15.584773+0000 mon.a (mon.0) 13390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:16 smithi115 bash[46934]: cluster 2022-04-22T22:05:14.716414+0000 mgr.y (mgr.34107) 9524 : cluster [DBG] pgmap v10315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:16 smithi115 bash[46934]: audit 2022-04-22T22:05:15.584773+0000 mon.a (mon.0) 13390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:17 smithi121 bash[36902]: audit 2022-04-22T22:05:16.495341+0000 mon.a (mon.0) 13391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:05:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:17 smithi121 bash[36902]: audit 2022-04-22T22:05:16.495899+0000 mon.a (mon.0) 13392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:05:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:17 smithi121 bash[36902]: audit 2022-04-22T22:05:16.728973+0000 mon.a (mon.0) 13393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:17 smithi121 bash[36902]: audit 2022-04-22T22:05:16.996133+0000 mon.a (mon.0) 13394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[45495]: audit 2022-04-22T22:05:16.495341+0000 mon.a (mon.0) 13391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[45495]: audit 2022-04-22T22:05:16.495899+0000 mon.a (mon.0) 13392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[45495]: audit 2022-04-22T22:05:16.728973+0000 mon.a (mon.0) 13393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[45495]: audit 2022-04-22T22:05:16.996133+0000 mon.a (mon.0) 13394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[46934]: audit 2022-04-22T22:05:16.495341+0000 mon.a (mon.0) 13391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[46934]: audit 2022-04-22T22:05:16.495899+0000 mon.a (mon.0) 13392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:05:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[46934]: audit 2022-04-22T22:05:16.728973+0000 mon.a (mon.0) 13393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:17 smithi115 bash[46934]: audit 2022-04-22T22:05:16.996133+0000 mon.a (mon.0) 13394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:18 smithi121 bash[36902]: cluster 2022-04-22T22:05:16.717483+0000 mgr.y (mgr.34107) 9525 : cluster [DBG] pgmap v10316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:18 smithi115 bash[45495]: cluster 2022-04-22T22:05:16.717483+0000 mgr.y (mgr.34107) 9525 : cluster [DBG] pgmap v10316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:18 smithi115 bash[46934]: cluster 2022-04-22T22:05:16.717483+0000 mgr.y (mgr.34107) 9525 : cluster [DBG] pgmap v10316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:20 smithi121 bash[36902]: cluster 2022-04-22T22:05:18.718595+0000 mgr.y (mgr.34107) 9526 : cluster [DBG] pgmap v10317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:20 smithi115 bash[45495]: cluster 2022-04-22T22:05:18.718595+0000 mgr.y (mgr.34107) 9526 : cluster [DBG] pgmap v10317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:20 smithi115 bash[46934]: cluster 2022-04-22T22:05:18.718595+0000 mgr.y (mgr.34107) 9526 : cluster [DBG] pgmap v10317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:21 smithi121 bash[36902]: audit 2022-04-22T22:05:20.587925+0000 mon.a (mon.0) 13395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:21 smithi115 bash[45495]: audit 2022-04-22T22:05:20.587925+0000 mon.a (mon.0) 13395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:21 smithi115 bash[46934]: audit 2022-04-22T22:05:20.587925+0000 mon.a (mon.0) 13395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:22 smithi121 bash[36902]: cluster 2022-04-22T22:05:20.719612+0000 mgr.y (mgr.34107) 9527 : cluster [DBG] pgmap v10318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:22 smithi121 bash[36902]: audit 2022-04-22T22:05:21.732649+0000 mon.a (mon.0) 13396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:22 smithi121 bash[36902]: audit 2022-04-22T22:05:21.998926+0000 mon.a (mon.0) 13397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[45495]: cluster 2022-04-22T22:05:20.719612+0000 mgr.y (mgr.34107) 9527 : cluster [DBG] pgmap v10318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[45495]: audit 2022-04-22T22:05:21.732649+0000 mon.a (mon.0) 13396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[45495]: audit 2022-04-22T22:05:21.998926+0000 mon.a (mon.0) 13397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[46934]: cluster 2022-04-22T22:05:20.719612+0000 mgr.y (mgr.34107) 9527 : cluster [DBG] pgmap v10318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[46934]: audit 2022-04-22T22:05:21.732649+0000 mon.a (mon.0) 13396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[46934]: audit 2022-04-22T22:05:21.998926+0000 mon.a (mon.0) 13397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:22.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:24 smithi121 bash[36902]: cluster 2022-04-22T22:05:22.720704+0000 mgr.y (mgr.34107) 9528 : cluster [DBG] pgmap v10319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:24 smithi115 bash[45495]: cluster 2022-04-22T22:05:22.720704+0000 mgr.y (mgr.34107) 9528 : cluster [DBG] pgmap v10319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:24 smithi115 bash[46934]: cluster 2022-04-22T22:05:22.720704+0000 mgr.y (mgr.34107) 9528 : cluster [DBG] pgmap v10319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:24.570Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:26 smithi121 bash[36902]: cluster 2022-04-22T22:05:24.721820+0000 mgr.y (mgr.34107) 9529 : cluster [DBG] pgmap v10320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:26 smithi121 bash[36902]: audit 2022-04-22T22:05:25.591138+0000 mon.a (mon.0) 13398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:26 smithi115 bash[45495]: cluster 2022-04-22T22:05:24.721820+0000 mgr.y (mgr.34107) 9529 : cluster [DBG] pgmap v10320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:26 smithi115 bash[45495]: audit 2022-04-22T22:05:25.591138+0000 mon.a (mon.0) 13398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:26 smithi115 bash[46934]: cluster 2022-04-22T22:05:24.721820+0000 mgr.y (mgr.34107) 9529 : cluster [DBG] pgmap v10320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:26 smithi115 bash[46934]: audit 2022-04-22T22:05:25.591138+0000 mon.a (mon.0) 13398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:27 smithi115 bash[45495]: audit 2022-04-22T22:05:26.736322+0000 mon.a (mon.0) 13399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:27 smithi115 bash[45495]: audit 2022-04-22T22:05:27.001812+0000 mon.a (mon.0) 13400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:27 smithi115 bash[46934]: audit 2022-04-22T22:05:26.736322+0000 mon.a (mon.0) 13399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:27 smithi115 bash[46934]: audit 2022-04-22T22:05:27.001812+0000 mon.a (mon.0) 13400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:27 smithi121 bash[36902]: audit 2022-04-22T22:05:26.736322+0000 mon.a (mon.0) 13399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:27 smithi121 bash[36902]: audit 2022-04-22T22:05:27.001812+0000 mon.a (mon.0) 13400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:28 smithi115 bash[45495]: cluster 2022-04-22T22:05:26.722917+0000 mgr.y (mgr.34107) 9530 : cluster [DBG] pgmap v10321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:28.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:28 smithi115 bash[46934]: cluster 2022-04-22T22:05:26.722917+0000 mgr.y (mgr.34107) 9530 : cluster [DBG] pgmap v10321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:28 smithi121 bash[36902]: cluster 2022-04-22T22:05:26.722917+0000 mgr.y (mgr.34107) 9530 : cluster [DBG] pgmap v10321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: cluster 2022-04-22T22:05:28.724116+0000 mgr.y (mgr.34107) 9531 : cluster [DBG] pgmap v10322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: audit 2022-04-22T22:05:29.373225+0000 mon.a (mon.0) 13401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:05:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: audit 2022-04-22T22:05:29.376273+0000 mon.a (mon.0) 13402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:05:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: audit 2022-04-22T22:05:29.378148+0000 mon.a (mon.0) 13403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:05:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: audit 2022-04-22T22:05:29.718587+0000 mon.a (mon.0) 13404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:30 smithi121 bash[36902]: audit 2022-04-22T22:05:29.723505+0000 mon.a (mon.0) 13405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:30.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: cluster 2022-04-22T22:05:28.724116+0000 mgr.y (mgr.34107) 9531 : cluster [DBG] pgmap v10322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: audit 2022-04-22T22:05:29.373225+0000 mon.a (mon.0) 13401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: audit 2022-04-22T22:05:29.376273+0000 mon.a (mon.0) 13402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: audit 2022-04-22T22:05:29.378148+0000 mon.a (mon.0) 13403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: audit 2022-04-22T22:05:29.718587+0000 mon.a (mon.0) 13404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[46934]: audit 2022-04-22T22:05:29.723505+0000 mon.a (mon.0) 13405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: cluster 2022-04-22T22:05:28.724116+0000 mgr.y (mgr.34107) 9531 : cluster [DBG] pgmap v10322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: audit 2022-04-22T22:05:29.373225+0000 mon.a (mon.0) 13401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:05:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: audit 2022-04-22T22:05:29.376273+0000 mon.a (mon.0) 13402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:05:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: audit 2022-04-22T22:05:29.378148+0000 mon.a (mon.0) 13403 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:05:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: audit 2022-04-22T22:05:29.718587+0000 mon.a (mon.0) 13404 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:30.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:30 smithi115 bash[45495]: audit 2022-04-22T22:05:29.723505+0000 mon.a (mon.0) 13405 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:31 smithi115 bash[45495]: audit 2022-04-22T22:05:30.594259+0000 mon.a (mon.0) 13406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:31 smithi115 bash[46934]: audit 2022-04-22T22:05:30.594259+0000 mon.a (mon.0) 13406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:31 smithi121 bash[36902]: audit 2022-04-22T22:05:30.594259+0000 mon.a (mon.0) 13406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[45495]: cluster 2022-04-22T22:05:30.725152+0000 mgr.y (mgr.34107) 9532 : cluster [DBG] pgmap v10323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[45495]: audit 2022-04-22T22:05:31.740030+0000 mon.a (mon.0) 13407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[45495]: audit 2022-04-22T22:05:32.004858+0000 mon.a (mon.0) 13408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[46934]: cluster 2022-04-22T22:05:30.725152+0000 mgr.y (mgr.34107) 9532 : cluster [DBG] pgmap v10323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[46934]: audit 2022-04-22T22:05:31.740030+0000 mon.a (mon.0) 13407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[46934]: audit 2022-04-22T22:05:32.004858+0000 mon.a (mon.0) 13408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:32 smithi121 bash[36902]: cluster 2022-04-22T22:05:30.725152+0000 mgr.y (mgr.34107) 9532 : cluster [DBG] pgmap v10323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:32 smithi121 bash[36902]: audit 2022-04-22T22:05:31.740030+0000 mon.a (mon.0) 13407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:32 smithi121 bash[36902]: audit 2022-04-22T22:05:32.004858+0000 mon.a (mon.0) 13408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:32.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[45495]: cluster 2022-04-22T22:05:32.726005+0000 mgr.y (mgr.34107) 9533 : cluster [DBG] pgmap v10324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[45495]: audit 2022-04-22T22:05:33.924269+0000 mon.a (mon.0) 13409 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[45495]: audit 2022-04-22T22:05:34.246010+0000 mon.a (mon.0) 13410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[46934]: cluster 2022-04-22T22:05:32.726005+0000 mgr.y (mgr.34107) 9533 : cluster [DBG] pgmap v10324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[46934]: audit 2022-04-22T22:05:33.924269+0000 mon.a (mon.0) 13409 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:34.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[46934]: audit 2022-04-22T22:05:34.246010+0000 mon.a (mon.0) 13410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:34 smithi121 bash[36902]: cluster 2022-04-22T22:05:32.726005+0000 mgr.y (mgr.34107) 9533 : cluster [DBG] pgmap v10324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:34 smithi121 bash[36902]: audit 2022-04-22T22:05:33.924269+0000 mon.a (mon.0) 13409 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:34 smithi121 bash[36902]: audit 2022-04-22T22:05:34.246010+0000 mon.a (mon.0) 13410 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:05:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:34.571Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:36 smithi121 bash[36902]: cluster 2022-04-22T22:05:34.727088+0000 mgr.y (mgr.34107) 9534 : cluster [DBG] pgmap v10325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:36 smithi121 bash[36902]: audit 2022-04-22T22:05:35.597291+0000 mon.a (mon.0) 13411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:36 smithi115 bash[45495]: cluster 2022-04-22T22:05:34.727088+0000 mgr.y (mgr.34107) 9534 : cluster [DBG] pgmap v10325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:36 smithi115 bash[45495]: audit 2022-04-22T22:05:35.597291+0000 mon.a (mon.0) 13411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:36 smithi115 bash[46934]: cluster 2022-04-22T22:05:34.727088+0000 mgr.y (mgr.34107) 9534 : cluster [DBG] pgmap v10325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:36 smithi115 bash[46934]: audit 2022-04-22T22:05:35.597291+0000 mon.a (mon.0) 13411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:37 smithi121 bash[36902]: audit 2022-04-22T22:05:36.743900+0000 mon.a (mon.0) 13412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:37 smithi121 bash[36902]: audit 2022-04-22T22:05:37.007765+0000 mon.a (mon.0) 13413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:37 smithi115 bash[45495]: audit 2022-04-22T22:05:36.743900+0000 mon.a (mon.0) 13412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:37 smithi115 bash[45495]: audit 2022-04-22T22:05:37.007765+0000 mon.a (mon.0) 13413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:37 smithi115 bash[46934]: audit 2022-04-22T22:05:36.743900+0000 mon.a (mon.0) 13412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:37 smithi115 bash[46934]: audit 2022-04-22T22:05:37.007765+0000 mon.a (mon.0) 13413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:38 smithi121 bash[36902]: cluster 2022-04-22T22:05:36.728195+0000 mgr.y (mgr.34107) 9535 : cluster [DBG] pgmap v10326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:38 smithi115 bash[45495]: cluster 2022-04-22T22:05:36.728195+0000 mgr.y (mgr.34107) 9535 : cluster [DBG] pgmap v10326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:38 smithi115 bash[46934]: cluster 2022-04-22T22:05:36.728195+0000 mgr.y (mgr.34107) 9535 : cluster [DBG] pgmap v10326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:40 smithi121 bash[36902]: cluster 2022-04-22T22:05:38.729406+0000 mgr.y (mgr.34107) 9536 : cluster [DBG] pgmap v10327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:40 smithi115 bash[45495]: cluster 2022-04-22T22:05:38.729406+0000 mgr.y (mgr.34107) 9536 : cluster [DBG] pgmap v10327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:40 smithi115 bash[46934]: cluster 2022-04-22T22:05:38.729406+0000 mgr.y (mgr.34107) 9536 : cluster [DBG] pgmap v10327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:41 smithi121 bash[36902]: audit 2022-04-22T22:05:40.600332+0000 mon.a (mon.0) 13414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:41 smithi115 bash[45495]: audit 2022-04-22T22:05:40.600332+0000 mon.a (mon.0) 13414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:41 smithi115 bash[46934]: audit 2022-04-22T22:05:40.600332+0000 mon.a (mon.0) 13414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:42.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[45495]: cluster 2022-04-22T22:05:40.730411+0000 mgr.y (mgr.34107) 9537 : cluster [DBG] pgmap v10328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:42.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[45495]: audit 2022-04-22T22:05:41.747435+0000 mon.a (mon.0) 13415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:42.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[45495]: audit 2022-04-22T22:05:42.010990+0000 mon.a (mon.0) 13416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:42.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[46934]: cluster 2022-04-22T22:05:40.730411+0000 mgr.y (mgr.34107) 9537 : cluster [DBG] pgmap v10328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:42.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[46934]: audit 2022-04-22T22:05:41.747435+0000 mon.a (mon.0) 13415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:42.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[46934]: audit 2022-04-22T22:05:42.010990+0000 mon.a (mon.0) 13416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:42 smithi121 bash[36902]: cluster 2022-04-22T22:05:40.730411+0000 mgr.y (mgr.34107) 9537 : cluster [DBG] pgmap v10328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:42 smithi121 bash[36902]: audit 2022-04-22T22:05:41.747435+0000 mon.a (mon.0) 13415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:42 smithi121 bash[36902]: audit 2022-04-22T22:05:42.010990+0000 mon.a (mon.0) 13416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:42.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:44 smithi121 bash[36902]: cluster 2022-04-22T22:05:42.731556+0000 mgr.y (mgr.34107) 9538 : cluster [DBG] pgmap v10329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:44.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:44 smithi115 bash[46934]: cluster 2022-04-22T22:05:42.731556+0000 mgr.y (mgr.34107) 9538 : cluster [DBG] pgmap v10329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:44.798 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:44.571Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:44.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:44 smithi115 bash[45495]: cluster 2022-04-22T22:05:42.731556+0000 mgr.y (mgr.34107) 9538 : cluster [DBG] pgmap v10329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:44.844 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:05:45.190 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 11s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 76s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 11s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 76s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 76s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:05:45.191 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 11s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:05:45.192 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 11s ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:05:45.192 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 76s ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:05:45.192 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 11s ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 11s ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 76s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 11s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 11s ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 11s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:05:45.193 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 11s ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:05:45.194 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 76s ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:05:45.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 76s ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:05:45.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 76s ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:05:45.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 76s ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:05:45.195 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 76s ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:05:45.594 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:05:45.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:05:45.597 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:05:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:46 smithi121 bash[36902]: cluster 2022-04-22T22:05:44.732601+0000 mgr.y (mgr.34107) 9539 : cluster [DBG] pgmap v10330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:46 smithi121 bash[36902]: audit 2022-04-22T22:05:44.825132+0000 mgr.y (mgr.34107) 9540 : audit [DBG] from='client.55183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:46 smithi121 bash[36902]: audit 2022-04-22T22:05:45.167745+0000 mgr.y (mgr.34107) 9541 : audit [DBG] from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:46 smithi121 bash[36902]: audit 2022-04-22T22:05:45.588730+0000 mon.a (mon.0) 13417 : audit [DBG] from='client.? 172.21.15.115:0/3649763689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:46 smithi121 bash[36902]: audit 2022-04-22T22:05:45.603200+0000 mon.a (mon.0) 13418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[45495]: cluster 2022-04-22T22:05:44.732601+0000 mgr.y (mgr.34107) 9539 : cluster [DBG] pgmap v10330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[45495]: audit 2022-04-22T22:05:44.825132+0000 mgr.y (mgr.34107) 9540 : audit [DBG] from='client.55183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[45495]: audit 2022-04-22T22:05:45.167745+0000 mgr.y (mgr.34107) 9541 : audit [DBG] from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[45495]: audit 2022-04-22T22:05:45.588730+0000 mon.a (mon.0) 13417 : audit [DBG] from='client.? 172.21.15.115:0/3649763689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:46.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[45495]: audit 2022-04-22T22:05:45.603200+0000 mon.a (mon.0) 13418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[46934]: cluster 2022-04-22T22:05:44.732601+0000 mgr.y (mgr.34107) 9539 : cluster [DBG] pgmap v10330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[46934]: audit 2022-04-22T22:05:44.825132+0000 mgr.y (mgr.34107) 9540 : audit [DBG] from='client.55183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[46934]: audit 2022-04-22T22:05:45.167745+0000 mgr.y (mgr.34107) 9541 : audit [DBG] from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:05:46.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[46934]: audit 2022-04-22T22:05:45.588730+0000 mon.a (mon.0) 13417 : audit [DBG] from='client.? 172.21.15.115:0/3649763689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:05:46.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:46 smithi115 bash[46934]: audit 2022-04-22T22:05:45.603200+0000 mon.a (mon.0) 13418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:47.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:47 smithi121 bash[36902]: audit 2022-04-22T22:05:46.751135+0000 mon.a (mon.0) 13419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:47.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:47 smithi121 bash[36902]: audit 2022-04-22T22:05:47.014059+0000 mon.a (mon.0) 13420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:47.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:47 smithi115 bash[45495]: audit 2022-04-22T22:05:46.751135+0000 mon.a (mon.0) 13419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:47.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:47 smithi115 bash[45495]: audit 2022-04-22T22:05:47.014059+0000 mon.a (mon.0) 13420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:47.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:47 smithi115 bash[46934]: audit 2022-04-22T22:05:46.751135+0000 mon.a (mon.0) 13419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:47.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:47 smithi115 bash[46934]: audit 2022-04-22T22:05:47.014059+0000 mon.a (mon.0) 13420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:48 smithi121 bash[36902]: cluster 2022-04-22T22:05:46.733698+0000 mgr.y (mgr.34107) 9542 : cluster [DBG] pgmap v10331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:48.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:48 smithi115 bash[45495]: cluster 2022-04-22T22:05:46.733698+0000 mgr.y (mgr.34107) 9542 : cluster [DBG] pgmap v10331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:48.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:48 smithi115 bash[46934]: cluster 2022-04-22T22:05:46.733698+0000 mgr.y (mgr.34107) 9542 : cluster [DBG] pgmap v10331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:50 smithi121 bash[36902]: cluster 2022-04-22T22:05:48.734896+0000 mgr.y (mgr.34107) 9543 : cluster [DBG] pgmap v10332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:50 smithi115 bash[45495]: cluster 2022-04-22T22:05:48.734896+0000 mgr.y (mgr.34107) 9543 : cluster [DBG] pgmap v10332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:50 smithi115 bash[46934]: cluster 2022-04-22T22:05:48.734896+0000 mgr.y (mgr.34107) 9543 : cluster [DBG] pgmap v10332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:51 smithi121 bash[36902]: audit 2022-04-22T22:05:50.606069+0000 mon.a (mon.0) 13421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:51 smithi115 bash[45495]: audit 2022-04-22T22:05:50.606069+0000 mon.a (mon.0) 13421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:51 smithi115 bash[46934]: audit 2022-04-22T22:05:50.606069+0000 mon.a (mon.0) 13421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:52.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[45495]: cluster 2022-04-22T22:05:50.735911+0000 mgr.y (mgr.34107) 9544 : cluster [DBG] pgmap v10333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:52.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[45495]: audit 2022-04-22T22:05:51.754719+0000 mon.a (mon.0) 13422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:52.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[45495]: audit 2022-04-22T22:05:52.016899+0000 mon.a (mon.0) 13423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:52.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[46934]: cluster 2022-04-22T22:05:50.735911+0000 mgr.y (mgr.34107) 9544 : cluster [DBG] pgmap v10333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:52.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[46934]: audit 2022-04-22T22:05:51.754719+0000 mon.a (mon.0) 13422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:52.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[46934]: audit 2022-04-22T22:05:52.016899+0000 mon.a (mon.0) 13423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:52 smithi121 bash[36902]: cluster 2022-04-22T22:05:50.735911+0000 mgr.y (mgr.34107) 9544 : cluster [DBG] pgmap v10333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:52 smithi121 bash[36902]: audit 2022-04-22T22:05:51.754719+0000 mon.a (mon.0) 13422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:52 smithi121 bash[36902]: audit 2022-04-22T22:05:52.016899+0000 mon.a (mon.0) 13423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:52.720Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:54 smithi121 bash[36902]: cluster 2022-04-22T22:05:52.737025+0000 mgr.y (mgr.34107) 9545 : cluster [DBG] pgmap v10334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:54.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:54 smithi115 bash[45495]: cluster 2022-04-22T22:05:52.737025+0000 mgr.y (mgr.34107) 9545 : cluster [DBG] pgmap v10334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:54.793 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:05:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:05:54.573Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:05:54.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:54 smithi115 bash[46934]: cluster 2022-04-22T22:05:52.737025+0000 mgr.y (mgr.34107) 9545 : cluster [DBG] pgmap v10334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:56 smithi121 bash[36902]: cluster 2022-04-22T22:05:54.738175+0000 mgr.y (mgr.34107) 9546 : cluster [DBG] pgmap v10335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:56 smithi121 bash[36902]: audit 2022-04-22T22:05:55.609145+0000 mon.a (mon.0) 13424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:56 smithi115 bash[45495]: cluster 2022-04-22T22:05:54.738175+0000 mgr.y (mgr.34107) 9546 : cluster [DBG] pgmap v10335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:56 smithi115 bash[45495]: audit 2022-04-22T22:05:55.609145+0000 mon.a (mon.0) 13424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:56 smithi115 bash[46934]: cluster 2022-04-22T22:05:54.738175+0000 mgr.y (mgr.34107) 9546 : cluster [DBG] pgmap v10335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:56 smithi115 bash[46934]: audit 2022-04-22T22:05:55.609145+0000 mon.a (mon.0) 13424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:05:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:57 smithi121 bash[36902]: audit 2022-04-22T22:05:56.758169+0000 mon.a (mon.0) 13425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:57 smithi121 bash[36902]: audit 2022-04-22T22:05:57.020040+0000 mon.a (mon.0) 13426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:57 smithi115 bash[45495]: audit 2022-04-22T22:05:56.758169+0000 mon.a (mon.0) 13425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:57 smithi115 bash[45495]: audit 2022-04-22T22:05:57.020040+0000 mon.a (mon.0) 13426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:57 smithi115 bash[46934]: audit 2022-04-22T22:05:56.758169+0000 mon.a (mon.0) 13425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:05:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:57 smithi115 bash[46934]: audit 2022-04-22T22:05:57.020040+0000 mon.a (mon.0) 13426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:05:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:05:58 smithi121 bash[36902]: cluster 2022-04-22T22:05:56.739294+0000 mgr.y (mgr.34107) 9547 : cluster [DBG] pgmap v10336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:05:58 smithi115 bash[45495]: cluster 2022-04-22T22:05:56.739294+0000 mgr.y (mgr.34107) 9547 : cluster [DBG] pgmap v10336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:05:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:05:58 smithi115 bash[46934]: cluster 2022-04-22T22:05:56.739294+0000 mgr.y (mgr.34107) 9547 : cluster [DBG] pgmap v10336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:00 smithi121 bash[36902]: cluster 2022-04-22T22:05:58.740423+0000 mgr.y (mgr.34107) 9548 : cluster [DBG] pgmap v10337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:00 smithi115 bash[45495]: cluster 2022-04-22T22:05:58.740423+0000 mgr.y (mgr.34107) 9548 : cluster [DBG] pgmap v10337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:00 smithi115 bash[46934]: cluster 2022-04-22T22:05:58.740423+0000 mgr.y (mgr.34107) 9548 : cluster [DBG] pgmap v10337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:01 smithi121 bash[36902]: audit 2022-04-22T22:06:00.612204+0000 mon.a (mon.0) 13427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:01 smithi115 bash[45495]: audit 2022-04-22T22:06:00.612204+0000 mon.a (mon.0) 13427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:01 smithi115 bash[46934]: audit 2022-04-22T22:06:00.612204+0000 mon.a (mon.0) 13427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:02.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[45495]: cluster 2022-04-22T22:06:00.741528+0000 mgr.y (mgr.34107) 9549 : cluster [DBG] pgmap v10338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:02.725 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[45495]: audit 2022-04-22T22:06:01.762216+0000 mon.a (mon.0) 13428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:02.725 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[45495]: audit 2022-04-22T22:06:02.023058+0000 mon.a (mon.0) 13429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:02.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[46934]: cluster 2022-04-22T22:06:00.741528+0000 mgr.y (mgr.34107) 9549 : cluster [DBG] pgmap v10338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:02.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[46934]: audit 2022-04-22T22:06:01.762216+0000 mon.a (mon.0) 13428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:02.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[46934]: audit 2022-04-22T22:06:02.023058+0000 mon.a (mon.0) 13429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:02 smithi121 bash[36902]: cluster 2022-04-22T22:06:00.741528+0000 mgr.y (mgr.34107) 9549 : cluster [DBG] pgmap v10338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:02 smithi121 bash[36902]: audit 2022-04-22T22:06:01.762216+0000 mon.a (mon.0) 13428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:02 smithi121 bash[36902]: audit 2022-04-22T22:06:02.023058+0000 mon.a (mon.0) 13429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:02.721Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:03 smithi121 bash[36902]: cluster 2022-04-22T22:06:02.742667+0000 mgr.y (mgr.34107) 9550 : cluster [DBG] pgmap v10339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:03.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:03 smithi115 bash[45495]: cluster 2022-04-22T22:06:02.742667+0000 mgr.y (mgr.34107) 9550 : cluster [DBG] pgmap v10339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:03 smithi115 bash[46934]: cluster 2022-04-22T22:06:02.742667+0000 mgr.y (mgr.34107) 9550 : cluster [DBG] pgmap v10339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:04.574Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:05 smithi121 bash[36902]: cluster 2022-04-22T22:06:04.743816+0000 mgr.y (mgr.34107) 9551 : cluster [DBG] pgmap v10340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:05 smithi121 bash[36902]: audit 2022-04-22T22:06:05.615326+0000 mon.a (mon.0) 13430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:05 smithi115 bash[45495]: cluster 2022-04-22T22:06:04.743816+0000 mgr.y (mgr.34107) 9551 : cluster [DBG] pgmap v10340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:05 smithi115 bash[45495]: audit 2022-04-22T22:06:05.615326+0000 mon.a (mon.0) 13430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:05 smithi115 bash[46934]: cluster 2022-04-22T22:06:04.743816+0000 mgr.y (mgr.34107) 9551 : cluster [DBG] pgmap v10340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:05 smithi115 bash[46934]: audit 2022-04-22T22:06:05.615326+0000 mon.a (mon.0) 13430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:07 smithi121 bash[36902]: audit 2022-04-22T22:06:06.765760+0000 mon.a (mon.0) 13431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:07 smithi121 bash[36902]: audit 2022-04-22T22:06:07.026147+0000 mon.a (mon.0) 13432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:07 smithi115 bash[45495]: audit 2022-04-22T22:06:06.765760+0000 mon.a (mon.0) 13431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:07 smithi115 bash[45495]: audit 2022-04-22T22:06:07.026147+0000 mon.a (mon.0) 13432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:07 smithi115 bash[46934]: audit 2022-04-22T22:06:06.765760+0000 mon.a (mon.0) 13431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:07 smithi115 bash[46934]: audit 2022-04-22T22:06:07.026147+0000 mon.a (mon.0) 13432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:08 smithi121 bash[36902]: cluster 2022-04-22T22:06:06.744904+0000 mgr.y (mgr.34107) 9552 : cluster [DBG] pgmap v10341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:08 smithi115 bash[45495]: cluster 2022-04-22T22:06:06.744904+0000 mgr.y (mgr.34107) 9552 : cluster [DBG] pgmap v10341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:08 smithi115 bash[46934]: cluster 2022-04-22T22:06:06.744904+0000 mgr.y (mgr.34107) 9552 : cluster [DBG] pgmap v10341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:10 smithi121 bash[36902]: cluster 2022-04-22T22:06:08.746081+0000 mgr.y (mgr.34107) 9553 : cluster [DBG] pgmap v10342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:10 smithi115 bash[45495]: cluster 2022-04-22T22:06:08.746081+0000 mgr.y (mgr.34107) 9553 : cluster [DBG] pgmap v10342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:10 smithi115 bash[46934]: cluster 2022-04-22T22:06:08.746081+0000 mgr.y (mgr.34107) 9553 : cluster [DBG] pgmap v10342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:11 smithi121 bash[36902]: audit 2022-04-22T22:06:10.618348+0000 mon.a (mon.0) 13433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:11 smithi115 bash[45495]: audit 2022-04-22T22:06:10.618348+0000 mon.a (mon.0) 13433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:11 smithi115 bash[46934]: audit 2022-04-22T22:06:10.618348+0000 mon.a (mon.0) 13433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:12 smithi121 bash[36902]: cluster 2022-04-22T22:06:10.747229+0000 mgr.y (mgr.34107) 9554 : cluster [DBG] pgmap v10343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:12 smithi121 bash[36902]: audit 2022-04-22T22:06:11.769241+0000 mon.a (mon.0) 13434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:12 smithi121 bash[36902]: audit 2022-04-22T22:06:12.029157+0000 mon.a (mon.0) 13435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:12.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[46934]: cluster 2022-04-22T22:06:10.747229+0000 mgr.y (mgr.34107) 9554 : cluster [DBG] pgmap v10343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:12.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[46934]: audit 2022-04-22T22:06:11.769241+0000 mon.a (mon.0) 13434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:12.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[46934]: audit 2022-04-22T22:06:12.029157+0000 mon.a (mon.0) 13435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[45495]: cluster 2022-04-22T22:06:10.747229+0000 mgr.y (mgr.34107) 9554 : cluster [DBG] pgmap v10343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[45495]: audit 2022-04-22T22:06:11.769241+0000 mon.a (mon.0) 13434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[45495]: audit 2022-04-22T22:06:12.029157+0000 mon.a (mon.0) 13435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:12.721Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:14 smithi121 bash[36902]: cluster 2022-04-22T22:06:12.748387+0000 mgr.y (mgr.34107) 9555 : cluster [DBG] pgmap v10344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:14.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:14 smithi115 bash[45495]: cluster 2022-04-22T22:06:12.748387+0000 mgr.y (mgr.34107) 9555 : cluster [DBG] pgmap v10344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:14.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:14 smithi115 bash[46934]: cluster 2022-04-22T22:06:12.748387+0000 mgr.y (mgr.34107) 9555 : cluster [DBG] pgmap v10344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:14.575Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:15.968 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:06:16.326 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:06:16.326 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 42s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:06:16.326 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 107s ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:06:16.326 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 42s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:06:16.326 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 107s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 107s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 42s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 42s ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 107s ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 42s ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 42s ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 107s ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 42s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:06:16.327 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 42s ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 42s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 42s ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 107s ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 107s ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 107s ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 107s ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:06:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 107s ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:16 smithi121 bash[36902]: cluster 2022-04-22T22:06:14.749437+0000 mgr.y (mgr.34107) 9556 : cluster [DBG] pgmap v10345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:16 smithi121 bash[36902]: audit 2022-04-22T22:06:15.621382+0000 mon.a (mon.0) 13436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:16 smithi115 bash[45495]: cluster 2022-04-22T22:06:14.749437+0000 mgr.y (mgr.34107) 9556 : cluster [DBG] pgmap v10345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:16 smithi115 bash[45495]: audit 2022-04-22T22:06:15.621382+0000 mon.a (mon.0) 13436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:16 smithi115 bash[46934]: cluster 2022-04-22T22:06:14.749437+0000 mgr.y (mgr.34107) 9556 : cluster [DBG] pgmap v10345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:16 smithi115 bash[46934]: audit 2022-04-22T22:06:15.621382+0000 mon.a (mon.0) 13436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:06:16.732 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:06:16.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:06:16.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:16.732 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:06:16.732 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:06:16.733 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:06:16.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:16.734 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:06:16.734 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:06:16.734 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:06:16.734 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:06:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:15.948681+0000 mgr.y (mgr.34107) 9557 : audit [DBG] from='client.45270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:16.305647+0000 mgr.y (mgr.34107) 9558 : audit [DBG] from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:16.502783+0000 mon.a (mon.0) 13437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:06:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:16.505570+0000 mon.a (mon.0) 13438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:06:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:16.727810+0000 mon.c (mon.1) 330 : audit [DBG] from='client.? 172.21.15.115:0/3425818351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:16.772950+0000 mon.a (mon.0) 13439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:17 smithi121 bash[36902]: audit 2022-04-22T22:06:17.032212+0000 mon.a (mon.0) 13440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:15.948681+0000 mgr.y (mgr.34107) 9557 : audit [DBG] from='client.45270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:16.305647+0000 mgr.y (mgr.34107) 9558 : audit [DBG] from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:16.502783+0000 mon.a (mon.0) 13437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:06:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:16.505570+0000 mon.a (mon.0) 13438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:06:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:16.727810+0000 mon.c (mon.1) 330 : audit [DBG] from='client.? 172.21.15.115:0/3425818351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:16.772950+0000 mon.a (mon.0) 13439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[45495]: audit 2022-04-22T22:06:17.032212+0000 mon.a (mon.0) 13440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:15.948681+0000 mgr.y (mgr.34107) 9557 : audit [DBG] from='client.45270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:16.305647+0000 mgr.y (mgr.34107) 9558 : audit [DBG] from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:16.502783+0000 mon.a (mon.0) 13437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:16.505570+0000 mon.a (mon.0) 13438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:16.727810+0000 mon.c (mon.1) 330 : audit [DBG] from='client.? 172.21.15.115:0/3425818351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:16.772950+0000 mon.a (mon.0) 13439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:17 smithi115 bash[46934]: audit 2022-04-22T22:06:17.032212+0000 mon.a (mon.0) 13440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:18 smithi121 bash[36902]: cluster 2022-04-22T22:06:16.750482+0000 mgr.y (mgr.34107) 9559 : cluster [DBG] pgmap v10346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:18 smithi115 bash[45495]: cluster 2022-04-22T22:06:16.750482+0000 mgr.y (mgr.34107) 9559 : cluster [DBG] pgmap v10346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:18 smithi115 bash[46934]: cluster 2022-04-22T22:06:16.750482+0000 mgr.y (mgr.34107) 9559 : cluster [DBG] pgmap v10346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:20 smithi121 bash[36902]: cluster 2022-04-22T22:06:18.751782+0000 mgr.y (mgr.34107) 9560 : cluster [DBG] pgmap v10347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:20 smithi115 bash[45495]: cluster 2022-04-22T22:06:18.751782+0000 mgr.y (mgr.34107) 9560 : cluster [DBG] pgmap v10347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:20 smithi115 bash[46934]: cluster 2022-04-22T22:06:18.751782+0000 mgr.y (mgr.34107) 9560 : cluster [DBG] pgmap v10347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:21 smithi121 bash[36902]: audit 2022-04-22T22:06:20.624188+0000 mon.a (mon.0) 13441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:21 smithi115 bash[45495]: audit 2022-04-22T22:06:20.624188+0000 mon.a (mon.0) 13441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:21 smithi115 bash[46934]: audit 2022-04-22T22:06:20.624188+0000 mon.a (mon.0) 13441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:22 smithi121 bash[36902]: cluster 2022-04-22T22:06:20.752802+0000 mgr.y (mgr.34107) 9561 : cluster [DBG] pgmap v10348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:22 smithi121 bash[36902]: audit 2022-04-22T22:06:21.776425+0000 mon.a (mon.0) 13442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:22 smithi121 bash[36902]: audit 2022-04-22T22:06:22.035401+0000 mon.a (mon.0) 13443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[45495]: cluster 2022-04-22T22:06:20.752802+0000 mgr.y (mgr.34107) 9561 : cluster [DBG] pgmap v10348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[45495]: audit 2022-04-22T22:06:21.776425+0000 mon.a (mon.0) 13442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[45495]: audit 2022-04-22T22:06:22.035401+0000 mon.a (mon.0) 13443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[46934]: cluster 2022-04-22T22:06:20.752802+0000 mgr.y (mgr.34107) 9561 : cluster [DBG] pgmap v10348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[46934]: audit 2022-04-22T22:06:21.776425+0000 mon.a (mon.0) 13442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[46934]: audit 2022-04-22T22:06:22.035401+0000 mon.a (mon.0) 13443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:22.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:24 smithi121 bash[36902]: cluster 2022-04-22T22:06:22.753877+0000 mgr.y (mgr.34107) 9562 : cluster [DBG] pgmap v10349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:24 smithi115 bash[45495]: cluster 2022-04-22T22:06:22.753877+0000 mgr.y (mgr.34107) 9562 : cluster [DBG] pgmap v10349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:24 smithi115 bash[46934]: cluster 2022-04-22T22:06:22.753877+0000 mgr.y (mgr.34107) 9562 : cluster [DBG] pgmap v10349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:24.574Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:26 smithi121 bash[36902]: cluster 2022-04-22T22:06:24.754985+0000 mgr.y (mgr.34107) 9563 : cluster [DBG] pgmap v10350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:26.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:26 smithi121 bash[36902]: audit 2022-04-22T22:06:25.627264+0000 mon.a (mon.0) 13444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:26 smithi115 bash[45495]: cluster 2022-04-22T22:06:24.754985+0000 mgr.y (mgr.34107) 9563 : cluster [DBG] pgmap v10350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:26 smithi115 bash[45495]: audit 2022-04-22T22:06:25.627264+0000 mon.a (mon.0) 13444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:26 smithi115 bash[46934]: cluster 2022-04-22T22:06:24.754985+0000 mgr.y (mgr.34107) 9563 : cluster [DBG] pgmap v10350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:26 smithi115 bash[46934]: audit 2022-04-22T22:06:25.627264+0000 mon.a (mon.0) 13444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:27 smithi121 bash[36902]: audit 2022-04-22T22:06:26.779707+0000 mon.a (mon.0) 13445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:27 smithi121 bash[36902]: audit 2022-04-22T22:06:27.038531+0000 mon.a (mon.0) 13446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:27 smithi115 bash[45495]: audit 2022-04-22T22:06:26.779707+0000 mon.a (mon.0) 13445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:27.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:27 smithi115 bash[45495]: audit 2022-04-22T22:06:27.038531+0000 mon.a (mon.0) 13446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:27.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:27 smithi115 bash[46934]: audit 2022-04-22T22:06:26.779707+0000 mon.a (mon.0) 13445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:27.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:27 smithi115 bash[46934]: audit 2022-04-22T22:06:27.038531+0000 mon.a (mon.0) 13446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:28 smithi121 bash[36902]: cluster 2022-04-22T22:06:26.756057+0000 mgr.y (mgr.34107) 9564 : cluster [DBG] pgmap v10351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:28.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:28 smithi115 bash[46934]: cluster 2022-04-22T22:06:26.756057+0000 mgr.y (mgr.34107) 9564 : cluster [DBG] pgmap v10351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:28 smithi115 bash[45495]: cluster 2022-04-22T22:06:26.756057+0000 mgr.y (mgr.34107) 9564 : cluster [DBG] pgmap v10351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:30 smithi121 bash[36902]: cluster 2022-04-22T22:06:28.757191+0000 mgr.y (mgr.34107) 9565 : cluster [DBG] pgmap v10352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:30.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:30 smithi115 bash[46934]: cluster 2022-04-22T22:06:28.757191+0000 mgr.y (mgr.34107) 9565 : cluster [DBG] pgmap v10352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:30 smithi115 bash[45495]: cluster 2022-04-22T22:06:28.757191+0000 mgr.y (mgr.34107) 9565 : cluster [DBG] pgmap v10352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:31 smithi121 bash[36902]: audit 2022-04-22T22:06:30.630565+0000 mon.a (mon.0) 13447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:31.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:31 smithi115 bash[46934]: audit 2022-04-22T22:06:30.630565+0000 mon.a (mon.0) 13447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:31 smithi115 bash[45495]: audit 2022-04-22T22:06:30.630565+0000 mon.a (mon.0) 13447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:32 smithi121 bash[36902]: cluster 2022-04-22T22:06:30.758238+0000 mgr.y (mgr.34107) 9566 : cluster [DBG] pgmap v10353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:32 smithi121 bash[36902]: audit 2022-04-22T22:06:31.783561+0000 mon.a (mon.0) 13448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:32 smithi121 bash[36902]: audit 2022-04-22T22:06:32.041710+0000 mon.a (mon.0) 13449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[45495]: cluster 2022-04-22T22:06:30.758238+0000 mgr.y (mgr.34107) 9566 : cluster [DBG] pgmap v10353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[45495]: audit 2022-04-22T22:06:31.783561+0000 mon.a (mon.0) 13448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[45495]: audit 2022-04-22T22:06:32.041710+0000 mon.a (mon.0) 13449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[46934]: cluster 2022-04-22T22:06:30.758238+0000 mgr.y (mgr.34107) 9566 : cluster [DBG] pgmap v10353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[46934]: audit 2022-04-22T22:06:31.783561+0000 mon.a (mon.0) 13448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[46934]: audit 2022-04-22T22:06:32.041710+0000 mon.a (mon.0) 13449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:32.723Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:34 smithi121 bash[36902]: cluster 2022-04-22T22:06:32.759345+0000 mgr.y (mgr.34107) 9567 : cluster [DBG] pgmap v10354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:34.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:34 smithi115 bash[46934]: cluster 2022-04-22T22:06:32.759345+0000 mgr.y (mgr.34107) 9567 : cluster [DBG] pgmap v10354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:34 smithi115 bash[45495]: cluster 2022-04-22T22:06:32.759345+0000 mgr.y (mgr.34107) 9567 : cluster [DBG] pgmap v10354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:34.575Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:35 smithi121 bash[36902]: audit 2022-04-22T22:06:34.253549+0000 mon.a (mon.0) 13450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:06:35.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:35 smithi121 bash[36902]: audit 2022-04-22T22:06:34.256495+0000 mon.a (mon.0) 13451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:06:35.561 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:35 smithi121 bash[36902]: audit 2022-04-22T22:06:34.258236+0000 mon.a (mon.0) 13452 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:06:35.561 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:35 smithi121 bash[36902]: audit 2022-04-22T22:06:34.635485+0000 mon.a (mon.0) 13453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:06:35.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[46934]: audit 2022-04-22T22:06:34.253549+0000 mon.a (mon.0) 13450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:06:35.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[46934]: audit 2022-04-22T22:06:34.256495+0000 mon.a (mon.0) 13451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[46934]: audit 2022-04-22T22:06:34.258236+0000 mon.a (mon.0) 13452 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[46934]: audit 2022-04-22T22:06:34.635485+0000 mon.a (mon.0) 13453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[45495]: audit 2022-04-22T22:06:34.253549+0000 mon.a (mon.0) 13450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[45495]: audit 2022-04-22T22:06:34.256495+0000 mon.a (mon.0) 13451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[45495]: audit 2022-04-22T22:06:34.258236+0000 mon.a (mon.0) 13452 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:06:35.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:35 smithi115 bash[45495]: audit 2022-04-22T22:06:34.635485+0000 mon.a (mon.0) 13453 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:06:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:36 smithi121 bash[36902]: cluster 2022-04-22T22:06:34.760422+0000 mgr.y (mgr.34107) 9568 : cluster [DBG] pgmap v10355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:36 smithi121 bash[36902]: audit 2022-04-22T22:06:35.633694+0000 mon.a (mon.0) 13454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:36 smithi115 bash[45495]: cluster 2022-04-22T22:06:34.760422+0000 mgr.y (mgr.34107) 9568 : cluster [DBG] pgmap v10355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:36 smithi115 bash[45495]: audit 2022-04-22T22:06:35.633694+0000 mon.a (mon.0) 13454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:36 smithi115 bash[46934]: cluster 2022-04-22T22:06:34.760422+0000 mgr.y (mgr.34107) 9568 : cluster [DBG] pgmap v10355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:36 smithi115 bash[46934]: audit 2022-04-22T22:06:35.633694+0000 mon.a (mon.0) 13454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:37 smithi121 bash[36902]: audit 2022-04-22T22:06:36.787404+0000 mon.a (mon.0) 13455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:37.518 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:37 smithi121 bash[36902]: audit 2022-04-22T22:06:37.044764+0000 mon.a (mon.0) 13456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:37 smithi115 bash[45495]: audit 2022-04-22T22:06:36.787404+0000 mon.a (mon.0) 13455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:37 smithi115 bash[45495]: audit 2022-04-22T22:06:37.044764+0000 mon.a (mon.0) 13456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:37 smithi115 bash[46934]: audit 2022-04-22T22:06:36.787404+0000 mon.a (mon.0) 13455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:37 smithi115 bash[46934]: audit 2022-04-22T22:06:37.044764+0000 mon.a (mon.0) 13456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:38 smithi121 bash[36902]: cluster 2022-04-22T22:06:36.761425+0000 mgr.y (mgr.34107) 9569 : cluster [DBG] pgmap v10356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:38 smithi115 bash[45495]: cluster 2022-04-22T22:06:36.761425+0000 mgr.y (mgr.34107) 9569 : cluster [DBG] pgmap v10356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:38 smithi115 bash[46934]: cluster 2022-04-22T22:06:36.761425+0000 mgr.y (mgr.34107) 9569 : cluster [DBG] pgmap v10356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:40 smithi121 bash[36902]: cluster 2022-04-22T22:06:38.762671+0000 mgr.y (mgr.34107) 9570 : cluster [DBG] pgmap v10357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:40 smithi115 bash[45495]: cluster 2022-04-22T22:06:38.762671+0000 mgr.y (mgr.34107) 9570 : cluster [DBG] pgmap v10357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:40 smithi115 bash[46934]: cluster 2022-04-22T22:06:38.762671+0000 mgr.y (mgr.34107) 9570 : cluster [DBG] pgmap v10357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:41 smithi121 bash[36902]: audit 2022-04-22T22:06:40.637129+0000 mon.a (mon.0) 13457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:41 smithi115 bash[45495]: audit 2022-04-22T22:06:40.637129+0000 mon.a (mon.0) 13457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:41 smithi115 bash[46934]: audit 2022-04-22T22:06:40.637129+0000 mon.a (mon.0) 13457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:42 smithi121 bash[36902]: cluster 2022-04-22T22:06:40.763767+0000 mgr.y (mgr.34107) 9571 : cluster [DBG] pgmap v10358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:42 smithi121 bash[36902]: audit 2022-04-22T22:06:41.791024+0000 mon.a (mon.0) 13458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:42 smithi121 bash[36902]: audit 2022-04-22T22:06:42.047914+0000 mon.a (mon.0) 13459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[45495]: cluster 2022-04-22T22:06:40.763767+0000 mgr.y (mgr.34107) 9571 : cluster [DBG] pgmap v10358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[45495]: audit 2022-04-22T22:06:41.791024+0000 mon.a (mon.0) 13458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[45495]: audit 2022-04-22T22:06:42.047914+0000 mon.a (mon.0) 13459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[46934]: cluster 2022-04-22T22:06:40.763767+0000 mgr.y (mgr.34107) 9571 : cluster [DBG] pgmap v10358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[46934]: audit 2022-04-22T22:06:41.791024+0000 mon.a (mon.0) 13458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[46934]: audit 2022-04-22T22:06:42.047914+0000 mon.a (mon.0) 13459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:42.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:44 smithi121 bash[36902]: cluster 2022-04-22T22:06:42.764817+0000 mgr.y (mgr.34107) 9572 : cluster [DBG] pgmap v10359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:44 smithi115 bash[45495]: cluster 2022-04-22T22:06:42.764817+0000 mgr.y (mgr.34107) 9572 : cluster [DBG] pgmap v10359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:44 smithi115 bash[46934]: cluster 2022-04-22T22:06:42.764817+0000 mgr.y (mgr.34107) 9572 : cluster [DBG] pgmap v10359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:44.576Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:46 smithi121 bash[36902]: cluster 2022-04-22T22:06:44.765837+0000 mgr.y (mgr.34107) 9573 : cluster [DBG] pgmap v10360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:46 smithi121 bash[36902]: audit 2022-04-22T22:06:45.640306+0000 mon.a (mon.0) 13460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:46 smithi115 bash[45495]: cluster 2022-04-22T22:06:44.765837+0000 mgr.y (mgr.34107) 9573 : cluster [DBG] pgmap v10360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:46 smithi115 bash[45495]: audit 2022-04-22T22:06:45.640306+0000 mon.a (mon.0) 13460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:46 smithi115 bash[46934]: cluster 2022-04-22T22:06:44.765837+0000 mgr.y (mgr.34107) 9573 : cluster [DBG] pgmap v10360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:46 smithi115 bash[46934]: audit 2022-04-22T22:06:45.640306+0000 mon.a (mon.0) 13460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:47.107 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:06:47.452 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:06:47.452 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 73s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:06:47.452 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 73s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 73s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 73s ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:06:47.453 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 73s ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 73s ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 73s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 73s ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 73s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 73s ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:06:47.454 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:06:47.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:06:47.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:06:47.455 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:06:47.455 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:06:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:47 smithi121 bash[36902]: audit 2022-04-22T22:06:46.794668+0000 mon.a (mon.0) 13461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:47 smithi121 bash[36902]: audit 2022-04-22T22:06:47.050682+0000 mon.a (mon.0) 13462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:47 smithi115 bash[45495]: audit 2022-04-22T22:06:46.794668+0000 mon.a (mon.0) 13461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:47 smithi115 bash[45495]: audit 2022-04-22T22:06:47.050682+0000 mon.a (mon.0) 13462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:47 smithi115 bash[46934]: audit 2022-04-22T22:06:46.794668+0000 mon.a (mon.0) 13461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:47 smithi115 bash[46934]: audit 2022-04-22T22:06:47.050682+0000 mon.a (mon.0) 13462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:47.857 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:06:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:06:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:06:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:06:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:06:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:06:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:06:47.859 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:06:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:48 smithi121 bash[36902]: cluster 2022-04-22T22:06:46.766855+0000 mgr.y (mgr.34107) 9574 : cluster [DBG] pgmap v10361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:48.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:48 smithi121 bash[36902]: audit 2022-04-22T22:06:47.085759+0000 mgr.y (mgr.34107) 9575 : audit [DBG] from='client.45288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:48.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:48 smithi121 bash[36902]: audit 2022-04-22T22:06:47.853846+0000 mon.c (mon.1) 331 : audit [DBG] from='client.? 172.21.15.115:0/1957382885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[45495]: cluster 2022-04-22T22:06:46.766855+0000 mgr.y (mgr.34107) 9574 : cluster [DBG] pgmap v10361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[45495]: audit 2022-04-22T22:06:47.085759+0000 mgr.y (mgr.34107) 9575 : audit [DBG] from='client.45288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[45495]: audit 2022-04-22T22:06:47.853846+0000 mon.c (mon.1) 331 : audit [DBG] from='client.? 172.21.15.115:0/1957382885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[46934]: cluster 2022-04-22T22:06:46.766855+0000 mgr.y (mgr.34107) 9574 : cluster [DBG] pgmap v10361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[46934]: audit 2022-04-22T22:06:47.085759+0000 mgr.y (mgr.34107) 9575 : audit [DBG] from='client.45288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:48.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:48 smithi115 bash[46934]: audit 2022-04-22T22:06:47.853846+0000 mon.c (mon.1) 331 : audit [DBG] from='client.? 172.21.15.115:0/1957382885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:06:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:49 smithi115 bash[45495]: audit 2022-04-22T22:06:47.428469+0000 mgr.y (mgr.34107) 9576 : audit [DBG] from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:49 smithi115 bash[46934]: audit 2022-04-22T22:06:47.428469+0000 mgr.y (mgr.34107) 9576 : audit [DBG] from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:49 smithi121 bash[36902]: audit 2022-04-22T22:06:47.428469+0000 mgr.y (mgr.34107) 9576 : audit [DBG] from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:06:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:50 smithi115 bash[45495]: cluster 2022-04-22T22:06:48.768089+0000 mgr.y (mgr.34107) 9577 : cluster [DBG] pgmap v10362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:50 smithi115 bash[46934]: cluster 2022-04-22T22:06:48.768089+0000 mgr.y (mgr.34107) 9577 : cluster [DBG] pgmap v10362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:50 smithi121 bash[36902]: cluster 2022-04-22T22:06:48.768089+0000 mgr.y (mgr.34107) 9577 : cluster [DBG] pgmap v10362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:51 smithi115 bash[45495]: audit 2022-04-22T22:06:50.643428+0000 mon.a (mon.0) 13463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:51 smithi115 bash[46934]: audit 2022-04-22T22:06:50.643428+0000 mon.a (mon.0) 13463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:51 smithi121 bash[36902]: audit 2022-04-22T22:06:50.643428+0000 mon.a (mon.0) 13463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[45495]: cluster 2022-04-22T22:06:50.769148+0000 mgr.y (mgr.34107) 9578 : cluster [DBG] pgmap v10363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[45495]: audit 2022-04-22T22:06:51.798191+0000 mon.a (mon.0) 13464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[45495]: audit 2022-04-22T22:06:52.053767+0000 mon.a (mon.0) 13465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[46934]: cluster 2022-04-22T22:06:50.769148+0000 mgr.y (mgr.34107) 9578 : cluster [DBG] pgmap v10363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[46934]: audit 2022-04-22T22:06:51.798191+0000 mon.a (mon.0) 13464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[46934]: audit 2022-04-22T22:06:52.053767+0000 mon.a (mon.0) 13465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:52 smithi121 bash[36902]: cluster 2022-04-22T22:06:50.769148+0000 mgr.y (mgr.34107) 9578 : cluster [DBG] pgmap v10363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:52 smithi121 bash[36902]: audit 2022-04-22T22:06:51.798191+0000 mon.a (mon.0) 13464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:52 smithi121 bash[36902]: audit 2022-04-22T22:06:52.053767+0000 mon.a (mon.0) 13465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:52.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:54 smithi115 bash[45495]: cluster 2022-04-22T22:06:52.770284+0000 mgr.y (mgr.34107) 9579 : cluster [DBG] pgmap v10364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:54 smithi115 bash[46934]: cluster 2022-04-22T22:06:52.770284+0000 mgr.y (mgr.34107) 9579 : cluster [DBG] pgmap v10364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:54 smithi121 bash[36902]: cluster 2022-04-22T22:06:52.770284+0000 mgr.y (mgr.34107) 9579 : cluster [DBG] pgmap v10364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:06:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:06:54.576Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:06:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:56 smithi115 bash[45495]: cluster 2022-04-22T22:06:54.771337+0000 mgr.y (mgr.34107) 9580 : cluster [DBG] pgmap v10365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:56 smithi115 bash[45495]: audit 2022-04-22T22:06:55.646225+0000 mon.a (mon.0) 13466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:56 smithi115 bash[46934]: cluster 2022-04-22T22:06:54.771337+0000 mgr.y (mgr.34107) 9580 : cluster [DBG] pgmap v10365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:56 smithi115 bash[46934]: audit 2022-04-22T22:06:55.646225+0000 mon.a (mon.0) 13466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:56 smithi121 bash[36902]: cluster 2022-04-22T22:06:54.771337+0000 mgr.y (mgr.34107) 9580 : cluster [DBG] pgmap v10365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:56 smithi121 bash[36902]: audit 2022-04-22T22:06:55.646225+0000 mon.a (mon.0) 13466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:06:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:57 smithi121 bash[36902]: audit 2022-04-22T22:06:56.801931+0000 mon.a (mon.0) 13467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:57 smithi121 bash[36902]: audit 2022-04-22T22:06:57.057435+0000 mon.a (mon.0) 13468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:57 smithi115 bash[45495]: audit 2022-04-22T22:06:56.801931+0000 mon.a (mon.0) 13467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:57 smithi115 bash[45495]: audit 2022-04-22T22:06:57.057435+0000 mon.a (mon.0) 13468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:57 smithi115 bash[46934]: audit 2022-04-22T22:06:56.801931+0000 mon.a (mon.0) 13467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:06:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:57 smithi115 bash[46934]: audit 2022-04-22T22:06:57.057435+0000 mon.a (mon.0) 13468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:06:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:06:58 smithi121 bash[36902]: cluster 2022-04-22T22:06:56.772424+0000 mgr.y (mgr.34107) 9581 : cluster [DBG] pgmap v10366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:06:58 smithi115 bash[45495]: cluster 2022-04-22T22:06:56.772424+0000 mgr.y (mgr.34107) 9581 : cluster [DBG] pgmap v10366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:06:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:06:58 smithi115 bash[46934]: cluster 2022-04-22T22:06:56.772424+0000 mgr.y (mgr.34107) 9581 : cluster [DBG] pgmap v10366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:00 smithi121 bash[36902]: cluster 2022-04-22T22:06:58.773649+0000 mgr.y (mgr.34107) 9582 : cluster [DBG] pgmap v10367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:00 smithi115 bash[45495]: cluster 2022-04-22T22:06:58.773649+0000 mgr.y (mgr.34107) 9582 : cluster [DBG] pgmap v10367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:00 smithi115 bash[46934]: cluster 2022-04-22T22:06:58.773649+0000 mgr.y (mgr.34107) 9582 : cluster [DBG] pgmap v10367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:01 smithi121 bash[36902]: audit 2022-04-22T22:07:00.649410+0000 mon.a (mon.0) 13469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:01 smithi115 bash[45495]: audit 2022-04-22T22:07:00.649410+0000 mon.a (mon.0) 13469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:01.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:01 smithi115 bash[46934]: audit 2022-04-22T22:07:00.649410+0000 mon.a (mon.0) 13469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:02.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[45495]: cluster 2022-04-22T22:07:00.774637+0000 mgr.y (mgr.34107) 9583 : cluster [DBG] pgmap v10368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:02.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[45495]: audit 2022-04-22T22:07:01.805592+0000 mon.a (mon.0) 13470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:02.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[45495]: audit 2022-04-22T22:07:02.060578+0000 mon.a (mon.0) 13471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:02.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[46934]: cluster 2022-04-22T22:07:00.774637+0000 mgr.y (mgr.34107) 9583 : cluster [DBG] pgmap v10368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:02.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[46934]: audit 2022-04-22T22:07:01.805592+0000 mon.a (mon.0) 13470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:02.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[46934]: audit 2022-04-22T22:07:02.060578+0000 mon.a (mon.0) 13471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:02 smithi121 bash[36902]: cluster 2022-04-22T22:07:00.774637+0000 mgr.y (mgr.34107) 9583 : cluster [DBG] pgmap v10368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:02 smithi121 bash[36902]: audit 2022-04-22T22:07:01.805592+0000 mon.a (mon.0) 13470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:02 smithi121 bash[36902]: audit 2022-04-22T22:07:02.060578+0000 mon.a (mon.0) 13471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:04.581 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:04 smithi115 bash[45495]: cluster 2022-04-22T22:07:02.775792+0000 mgr.y (mgr.34107) 9584 : cluster [DBG] pgmap v10369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:04.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:04 smithi115 bash[46934]: cluster 2022-04-22T22:07:02.775792+0000 mgr.y (mgr.34107) 9584 : cluster [DBG] pgmap v10369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:04.582 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:02.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:04 smithi121 bash[36902]: cluster 2022-04-22T22:07:02.775792+0000 mgr.y (mgr.34107) 9584 : cluster [DBG] pgmap v10369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:04.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:06 smithi121 bash[36902]: cluster 2022-04-22T22:07:04.776870+0000 mgr.y (mgr.34107) 9585 : cluster [DBG] pgmap v10370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:06 smithi121 bash[36902]: audit 2022-04-22T22:07:05.652652+0000 mon.a (mon.0) 13472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:06 smithi115 bash[45495]: cluster 2022-04-22T22:07:04.776870+0000 mgr.y (mgr.34107) 9585 : cluster [DBG] pgmap v10370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:06 smithi115 bash[45495]: audit 2022-04-22T22:07:05.652652+0000 mon.a (mon.0) 13472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:06 smithi115 bash[46934]: cluster 2022-04-22T22:07:04.776870+0000 mgr.y (mgr.34107) 9585 : cluster [DBG] pgmap v10370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:06 smithi115 bash[46934]: audit 2022-04-22T22:07:05.652652+0000 mon.a (mon.0) 13472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:07 smithi121 bash[36902]: audit 2022-04-22T22:07:06.809466+0000 mon.a (mon.0) 13473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:07 smithi121 bash[36902]: audit 2022-04-22T22:07:07.063311+0000 mon.a (mon.0) 13474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:07 smithi115 bash[45495]: audit 2022-04-22T22:07:06.809466+0000 mon.a (mon.0) 13473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:07 smithi115 bash[45495]: audit 2022-04-22T22:07:07.063311+0000 mon.a (mon.0) 13474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:07.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:07 smithi115 bash[46934]: audit 2022-04-22T22:07:06.809466+0000 mon.a (mon.0) 13473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:07.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:07 smithi115 bash[46934]: audit 2022-04-22T22:07:07.063311+0000 mon.a (mon.0) 13474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:08 smithi121 bash[36902]: cluster 2022-04-22T22:07:06.778036+0000 mgr.y (mgr.34107) 9586 : cluster [DBG] pgmap v10371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:08 smithi115 bash[45495]: cluster 2022-04-22T22:07:06.778036+0000 mgr.y (mgr.34107) 9586 : cluster [DBG] pgmap v10371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:08 smithi115 bash[46934]: cluster 2022-04-22T22:07:06.778036+0000 mgr.y (mgr.34107) 9586 : cluster [DBG] pgmap v10371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:10 smithi121 bash[36902]: cluster 2022-04-22T22:07:08.779214+0000 mgr.y (mgr.34107) 9587 : cluster [DBG] pgmap v10372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:10 smithi115 bash[45495]: cluster 2022-04-22T22:07:08.779214+0000 mgr.y (mgr.34107) 9587 : cluster [DBG] pgmap v10372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:10 smithi115 bash[46934]: cluster 2022-04-22T22:07:08.779214+0000 mgr.y (mgr.34107) 9587 : cluster [DBG] pgmap v10372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:11 smithi121 bash[36902]: audit 2022-04-22T22:07:10.655757+0000 mon.a (mon.0) 13475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:11 smithi115 bash[45495]: audit 2022-04-22T22:07:10.655757+0000 mon.a (mon.0) 13475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:11 smithi115 bash[46934]: audit 2022-04-22T22:07:10.655757+0000 mon.a (mon.0) 13475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:12.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[45495]: cluster 2022-04-22T22:07:10.780213+0000 mgr.y (mgr.34107) 9588 : cluster [DBG] pgmap v10373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:12.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[45495]: audit 2022-04-22T22:07:11.813155+0000 mon.a (mon.0) 13476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:12.729 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[45495]: audit 2022-04-22T22:07:12.066414+0000 mon.a (mon.0) 13477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:12.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[46934]: cluster 2022-04-22T22:07:10.780213+0000 mgr.y (mgr.34107) 9588 : cluster [DBG] pgmap v10373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:12.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[46934]: audit 2022-04-22T22:07:11.813155+0000 mon.a (mon.0) 13476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:12.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[46934]: audit 2022-04-22T22:07:12.066414+0000 mon.a (mon.0) 13477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:12 smithi121 bash[36902]: cluster 2022-04-22T22:07:10.780213+0000 mgr.y (mgr.34107) 9588 : cluster [DBG] pgmap v10373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:12 smithi121 bash[36902]: audit 2022-04-22T22:07:11.813155+0000 mon.a (mon.0) 13476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:12 smithi121 bash[36902]: audit 2022-04-22T22:07:12.066414+0000 mon.a (mon.0) 13477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:13.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:12.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:14 smithi121 bash[36902]: cluster 2022-04-22T22:07:12.781327+0000 mgr.y (mgr.34107) 9589 : cluster [DBG] pgmap v10374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:14.579Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:14 smithi115 bash[45495]: cluster 2022-04-22T22:07:12.781327+0000 mgr.y (mgr.34107) 9589 : cluster [DBG] pgmap v10374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:14 smithi115 bash[46934]: cluster 2022-04-22T22:07:12.781327+0000 mgr.y (mgr.34107) 9589 : cluster [DBG] pgmap v10374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:16 smithi121 bash[36902]: cluster 2022-04-22T22:07:14.782442+0000 mgr.y (mgr.34107) 9590 : cluster [DBG] pgmap v10375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:16 smithi121 bash[36902]: audit 2022-04-22T22:07:15.658739+0000 mon.a (mon.0) 13478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:16 smithi115 bash[45495]: cluster 2022-04-22T22:07:14.782442+0000 mgr.y (mgr.34107) 9590 : cluster [DBG] pgmap v10375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:16 smithi115 bash[45495]: audit 2022-04-22T22:07:15.658739+0000 mon.a (mon.0) 13478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:16 smithi115 bash[46934]: cluster 2022-04-22T22:07:14.782442+0000 mgr.y (mgr.34107) 9590 : cluster [DBG] pgmap v10375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:16 smithi115 bash[46934]: audit 2022-04-22T22:07:15.658739+0000 mon.a (mon.0) 13478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:17 smithi121 bash[36902]: audit 2022-04-22T22:07:16.514272+0000 mon.a (mon.0) 13479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:07:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:17 smithi121 bash[36902]: audit 2022-04-22T22:07:16.515007+0000 mon.a (mon.0) 13480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:07:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:17 smithi121 bash[36902]: audit 2022-04-22T22:07:16.816846+0000 mon.a (mon.0) 13481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:17 smithi121 bash[36902]: audit 2022-04-22T22:07:17.069385+0000 mon.a (mon.0) 13482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[45495]: audit 2022-04-22T22:07:16.514272+0000 mon.a (mon.0) 13479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[45495]: audit 2022-04-22T22:07:16.515007+0000 mon.a (mon.0) 13480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[45495]: audit 2022-04-22T22:07:16.816846+0000 mon.a (mon.0) 13481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[45495]: audit 2022-04-22T22:07:17.069385+0000 mon.a (mon.0) 13482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[46934]: audit 2022-04-22T22:07:16.514272+0000 mon.a (mon.0) 13479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:07:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[46934]: audit 2022-04-22T22:07:16.515007+0000 mon.a (mon.0) 13480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:07:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[46934]: audit 2022-04-22T22:07:16.816846+0000 mon.a (mon.0) 13481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:17 smithi115 bash[46934]: audit 2022-04-22T22:07:17.069385+0000 mon.a (mon.0) 13482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:18.231 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:07:18.569 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:07:18.569 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 104s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:07:18.569 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:07:18.569 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 104s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 104s ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 104s ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 104s ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:07:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 104s ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:07:18.571 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:07:18.571 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 104s ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:07:18.571 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 104s ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:07:18.571 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 104s ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:07:18.571 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 104s ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:07:18.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:07:18.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:07:18.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:07:18.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:07:18.572 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:07:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:18 smithi121 bash[36902]: cluster 2022-04-22T22:07:16.783608+0000 mgr.y (mgr.34107) 9591 : cluster [DBG] pgmap v10376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:18 smithi115 bash[45495]: cluster 2022-04-22T22:07:16.783608+0000 mgr.y (mgr.34107) 9591 : cluster [DBG] pgmap v10376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:18.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:18 smithi115 bash[46934]: cluster 2022-04-22T22:07:16.783608+0000 mgr.y (mgr.34107) 9591 : cluster [DBG] pgmap v10376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:18.979 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:07:18.979 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:07:18.979 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:07:18.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:18.979 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:07:18.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:07:18.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:07:18.981 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:07:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:19 smithi121 bash[36902]: audit 2022-04-22T22:07:18.210930+0000 mgr.y (mgr.34107) 9592 : audit [DBG] from='client.45306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:19 smithi121 bash[36902]: audit 2022-04-22T22:07:18.975631+0000 mon.a (mon.0) 13483 : audit [DBG] from='client.? 172.21.15.115:0/1601260604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:19 smithi115 bash[45495]: audit 2022-04-22T22:07:18.210930+0000 mgr.y (mgr.34107) 9592 : audit [DBG] from='client.45306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:19 smithi115 bash[45495]: audit 2022-04-22T22:07:18.975631+0000 mon.a (mon.0) 13483 : audit [DBG] from='client.? 172.21.15.115:0/1601260604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:19 smithi115 bash[46934]: audit 2022-04-22T22:07:18.210930+0000 mgr.y (mgr.34107) 9592 : audit [DBG] from='client.45306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:19 smithi115 bash[46934]: audit 2022-04-22T22:07:18.975631+0000 mon.a (mon.0) 13483 : audit [DBG] from='client.? 172.21.15.115:0/1601260604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:20 smithi121 bash[36902]: audit 2022-04-22T22:07:18.546536+0000 mgr.y (mgr.34107) 9593 : audit [DBG] from='client.55243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:20 smithi121 bash[36902]: cluster 2022-04-22T22:07:18.784609+0000 mgr.y (mgr.34107) 9594 : cluster [DBG] pgmap v10377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:20.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:20 smithi115 bash[45495]: audit 2022-04-22T22:07:18.546536+0000 mgr.y (mgr.34107) 9593 : audit [DBG] from='client.55243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:20 smithi115 bash[45495]: cluster 2022-04-22T22:07:18.784609+0000 mgr.y (mgr.34107) 9594 : cluster [DBG] pgmap v10377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:20 smithi115 bash[46934]: audit 2022-04-22T22:07:18.546536+0000 mgr.y (mgr.34107) 9593 : audit [DBG] from='client.55243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:20 smithi115 bash[46934]: cluster 2022-04-22T22:07:18.784609+0000 mgr.y (mgr.34107) 9594 : cluster [DBG] pgmap v10377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:21 smithi121 bash[36902]: audit 2022-04-22T22:07:20.661792+0000 mon.a (mon.0) 13484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:21 smithi115 bash[45495]: audit 2022-04-22T22:07:20.661792+0000 mon.a (mon.0) 13484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:21 smithi115 bash[46934]: audit 2022-04-22T22:07:20.661792+0000 mon.a (mon.0) 13484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:22.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[45495]: cluster 2022-04-22T22:07:20.785574+0000 mgr.y (mgr.34107) 9595 : cluster [DBG] pgmap v10378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:22.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[45495]: audit 2022-04-22T22:07:21.820472+0000 mon.a (mon.0) 13485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:22.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[45495]: audit 2022-04-22T22:07:22.072721+0000 mon.a (mon.0) 13486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:22.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[46934]: cluster 2022-04-22T22:07:20.785574+0000 mgr.y (mgr.34107) 9595 : cluster [DBG] pgmap v10378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:22.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[46934]: audit 2022-04-22T22:07:21.820472+0000 mon.a (mon.0) 13485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:22.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[46934]: audit 2022-04-22T22:07:22.072721+0000 mon.a (mon.0) 13486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:22.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:22 smithi121 bash[36902]: cluster 2022-04-22T22:07:20.785574+0000 mgr.y (mgr.34107) 9595 : cluster [DBG] pgmap v10378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:22.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:22 smithi121 bash[36902]: audit 2022-04-22T22:07:21.820472+0000 mon.a (mon.0) 13485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:22.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:22 smithi121 bash[36902]: audit 2022-04-22T22:07:22.072721+0000 mon.a (mon.0) 13486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:22.727Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:24 smithi121 bash[36902]: cluster 2022-04-22T22:07:22.786704+0000 mgr.y (mgr.34107) 9596 : cluster [DBG] pgmap v10379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:24.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:24 smithi115 bash[45495]: cluster 2022-04-22T22:07:22.786704+0000 mgr.y (mgr.34107) 9596 : cluster [DBG] pgmap v10379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:24 smithi115 bash[46934]: cluster 2022-04-22T22:07:22.786704+0000 mgr.y (mgr.34107) 9596 : cluster [DBG] pgmap v10379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:26 smithi121 bash[36902]: cluster 2022-04-22T22:07:24.787931+0000 mgr.y (mgr.34107) 9597 : cluster [DBG] pgmap v10380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:26 smithi121 bash[36902]: audit 2022-04-22T22:07:25.664773+0000 mon.a (mon.0) 13487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:26 smithi115 bash[45495]: cluster 2022-04-22T22:07:24.787931+0000 mgr.y (mgr.34107) 9597 : cluster [DBG] pgmap v10380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:26 smithi115 bash[45495]: audit 2022-04-22T22:07:25.664773+0000 mon.a (mon.0) 13487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:26 smithi115 bash[46934]: cluster 2022-04-22T22:07:24.787931+0000 mgr.y (mgr.34107) 9597 : cluster [DBG] pgmap v10380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:26 smithi115 bash[46934]: audit 2022-04-22T22:07:25.664773+0000 mon.a (mon.0) 13487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:27 smithi121 bash[36902]: audit 2022-04-22T22:07:26.824173+0000 mon.a (mon.0) 13488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:27 smithi121 bash[36902]: audit 2022-04-22T22:07:27.075596+0000 mon.a (mon.0) 13489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:27 smithi115 bash[45495]: audit 2022-04-22T22:07:26.824173+0000 mon.a (mon.0) 13488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:27 smithi115 bash[45495]: audit 2022-04-22T22:07:27.075596+0000 mon.a (mon.0) 13489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:27 smithi115 bash[46934]: audit 2022-04-22T22:07:26.824173+0000 mon.a (mon.0) 13488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:27 smithi115 bash[46934]: audit 2022-04-22T22:07:27.075596+0000 mon.a (mon.0) 13489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:28 smithi121 bash[36902]: cluster 2022-04-22T22:07:26.789036+0000 mgr.y (mgr.34107) 9598 : cluster [DBG] pgmap v10381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:28 smithi115 bash[45495]: cluster 2022-04-22T22:07:26.789036+0000 mgr.y (mgr.34107) 9598 : cluster [DBG] pgmap v10381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:28 smithi115 bash[46934]: cluster 2022-04-22T22:07:26.789036+0000 mgr.y (mgr.34107) 9598 : cluster [DBG] pgmap v10381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:30 smithi121 bash[36902]: cluster 2022-04-22T22:07:28.790229+0000 mgr.y (mgr.34107) 9599 : cluster [DBG] pgmap v10382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:30 smithi115 bash[45495]: cluster 2022-04-22T22:07:28.790229+0000 mgr.y (mgr.34107) 9599 : cluster [DBG] pgmap v10382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:30 smithi115 bash[46934]: cluster 2022-04-22T22:07:28.790229+0000 mgr.y (mgr.34107) 9599 : cluster [DBG] pgmap v10382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:31 smithi121 bash[36902]: audit 2022-04-22T22:07:30.668153+0000 mon.a (mon.0) 13490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:31 smithi115 bash[45495]: audit 2022-04-22T22:07:30.668153+0000 mon.a (mon.0) 13490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:31 smithi115 bash[46934]: audit 2022-04-22T22:07:30.668153+0000 mon.a (mon.0) 13490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:32.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[45495]: cluster 2022-04-22T22:07:30.791308+0000 mgr.y (mgr.34107) 9600 : cluster [DBG] pgmap v10383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:32.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[45495]: audit 2022-04-22T22:07:31.827940+0000 mon.a (mon.0) 13491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:32.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[45495]: audit 2022-04-22T22:07:32.078862+0000 mon.a (mon.0) 13492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:32.730 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[46934]: cluster 2022-04-22T22:07:30.791308+0000 mgr.y (mgr.34107) 9600 : cluster [DBG] pgmap v10383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:32.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[46934]: audit 2022-04-22T22:07:31.827940+0000 mon.a (mon.0) 13491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:32.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[46934]: audit 2022-04-22T22:07:32.078862+0000 mon.a (mon.0) 13492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:32.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:32 smithi121 bash[36902]: cluster 2022-04-22T22:07:30.791308+0000 mgr.y (mgr.34107) 9600 : cluster [DBG] pgmap v10383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:32 smithi121 bash[36902]: audit 2022-04-22T22:07:31.827940+0000 mon.a (mon.0) 13491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:32 smithi121 bash[36902]: audit 2022-04-22T22:07:32.078862+0000 mon.a (mon.0) 13492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:32.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:33 smithi121 bash[36902]: cluster 2022-04-22T22:07:32.792412+0000 mgr.y (mgr.34107) 9601 : cluster [DBG] pgmap v10384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:33 smithi115 bash[45495]: cluster 2022-04-22T22:07:32.792412+0000 mgr.y (mgr.34107) 9601 : cluster [DBG] pgmap v10384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:33 smithi115 bash[46934]: cluster 2022-04-22T22:07:32.792412+0000 mgr.y (mgr.34107) 9601 : cluster [DBG] pgmap v10384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:34.997 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:34.581Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:34.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[46934]: audit 2022-04-22T22:07:34.660854+0000 mon.a (mon.0) 13493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:07:34.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[46934]: audit 2022-04-22T22:07:34.663967+0000 mon.a (mon.0) 13494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:07:34.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[46934]: audit 2022-04-22T22:07:34.665826+0000 mon.a (mon.0) 13495 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:07:34.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[45495]: audit 2022-04-22T22:07:34.660854+0000 mon.a (mon.0) 13493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:07:34.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[45495]: audit 2022-04-22T22:07:34.663967+0000 mon.a (mon.0) 13494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:07:34.999 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:34 smithi115 bash[45495]: audit 2022-04-22T22:07:34.665826+0000 mon.a (mon.0) 13495 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:07:35.013 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:34 smithi121 bash[36902]: audit 2022-04-22T22:07:34.660854+0000 mon.a (mon.0) 13493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:07:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:34 smithi121 bash[36902]: audit 2022-04-22T22:07:34.663967+0000 mon.a (mon.0) 13494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:07:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:34 smithi121 bash[36902]: audit 2022-04-22T22:07:34.665826+0000 mon.a (mon.0) 13495 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:07:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[45495]: cluster 2022-04-22T22:07:34.793436+0000 mgr.y (mgr.34107) 9602 : cluster [DBG] pgmap v10385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[45495]: audit 2022-04-22T22:07:35.025689+0000 mon.a (mon.0) 13496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:07:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[45495]: audit 2022-04-22T22:07:35.671210+0000 mon.a (mon.0) 13497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[46934]: cluster 2022-04-22T22:07:34.793436+0000 mgr.y (mgr.34107) 9602 : cluster [DBG] pgmap v10385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[46934]: audit 2022-04-22T22:07:35.025689+0000 mon.a (mon.0) 13496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:07:36.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:36 smithi115 bash[46934]: audit 2022-04-22T22:07:35.671210+0000 mon.a (mon.0) 13497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:36 smithi121 bash[36902]: cluster 2022-04-22T22:07:34.793436+0000 mgr.y (mgr.34107) 9602 : cluster [DBG] pgmap v10385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:36 smithi121 bash[36902]: audit 2022-04-22T22:07:35.025689+0000 mon.a (mon.0) 13496 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:07:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:36 smithi121 bash[36902]: audit 2022-04-22T22:07:35.671210+0000 mon.a (mon.0) 13497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:37 smithi121 bash[36902]: audit 2022-04-22T22:07:36.831605+0000 mon.a (mon.0) 13498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:37 smithi121 bash[36902]: audit 2022-04-22T22:07:37.081812+0000 mon.a (mon.0) 13499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:37 smithi115 bash[45495]: audit 2022-04-22T22:07:36.831605+0000 mon.a (mon.0) 13498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:37.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:37 smithi115 bash[45495]: audit 2022-04-22T22:07:37.081812+0000 mon.a (mon.0) 13499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:37.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:37 smithi115 bash[46934]: audit 2022-04-22T22:07:36.831605+0000 mon.a (mon.0) 13498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:37.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:37 smithi115 bash[46934]: audit 2022-04-22T22:07:37.081812+0000 mon.a (mon.0) 13499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:38 smithi121 bash[36902]: cluster 2022-04-22T22:07:36.794572+0000 mgr.y (mgr.34107) 9603 : cluster [DBG] pgmap v10386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:38 smithi115 bash[45495]: cluster 2022-04-22T22:07:36.794572+0000 mgr.y (mgr.34107) 9603 : cluster [DBG] pgmap v10386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:38 smithi115 bash[46934]: cluster 2022-04-22T22:07:36.794572+0000 mgr.y (mgr.34107) 9603 : cluster [DBG] pgmap v10386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:40 smithi121 bash[36902]: cluster 2022-04-22T22:07:38.795666+0000 mgr.y (mgr.34107) 9604 : cluster [DBG] pgmap v10387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:40 smithi115 bash[45495]: cluster 2022-04-22T22:07:38.795666+0000 mgr.y (mgr.34107) 9604 : cluster [DBG] pgmap v10387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:40 smithi115 bash[46934]: cluster 2022-04-22T22:07:38.795666+0000 mgr.y (mgr.34107) 9604 : cluster [DBG] pgmap v10387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:41 smithi121 bash[36902]: audit 2022-04-22T22:07:40.674353+0000 mon.a (mon.0) 13500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:41 smithi115 bash[45495]: audit 2022-04-22T22:07:40.674353+0000 mon.a (mon.0) 13500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:41 smithi115 bash[46934]: audit 2022-04-22T22:07:40.674353+0000 mon.a (mon.0) 13500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:42 smithi121 bash[36902]: cluster 2022-04-22T22:07:40.796727+0000 mgr.y (mgr.34107) 9605 : cluster [DBG] pgmap v10388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:42.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:42 smithi121 bash[36902]: audit 2022-04-22T22:07:41.835438+0000 mon.a (mon.0) 13501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:42.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:42 smithi121 bash[36902]: audit 2022-04-22T22:07:42.084659+0000 mon.a (mon.0) 13502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[45495]: cluster 2022-04-22T22:07:40.796727+0000 mgr.y (mgr.34107) 9605 : cluster [DBG] pgmap v10388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[45495]: audit 2022-04-22T22:07:41.835438+0000 mon.a (mon.0) 13501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[45495]: audit 2022-04-22T22:07:42.084659+0000 mon.a (mon.0) 13502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[46934]: cluster 2022-04-22T22:07:40.796727+0000 mgr.y (mgr.34107) 9605 : cluster [DBG] pgmap v10388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[46934]: audit 2022-04-22T22:07:41.835438+0000 mon.a (mon.0) 13501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[46934]: audit 2022-04-22T22:07:42.084659+0000 mon.a (mon.0) 13502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:42.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:44 smithi121 bash[36902]: cluster 2022-04-22T22:07:42.797878+0000 mgr.y (mgr.34107) 9606 : cluster [DBG] pgmap v10389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:44 smithi115 bash[45495]: cluster 2022-04-22T22:07:42.797878+0000 mgr.y (mgr.34107) 9606 : cluster [DBG] pgmap v10389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:44 smithi115 bash[46934]: cluster 2022-04-22T22:07:42.797878+0000 mgr.y (mgr.34107) 9606 : cluster [DBG] pgmap v10389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:44.581Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:46 smithi121 bash[36902]: cluster 2022-04-22T22:07:44.799009+0000 mgr.y (mgr.34107) 9607 : cluster [DBG] pgmap v10390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:46 smithi121 bash[36902]: audit 2022-04-22T22:07:45.677497+0000 mon.a (mon.0) 13503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:46 smithi115 bash[45495]: cluster 2022-04-22T22:07:44.799009+0000 mgr.y (mgr.34107) 9607 : cluster [DBG] pgmap v10390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:46 smithi115 bash[45495]: audit 2022-04-22T22:07:45.677497+0000 mon.a (mon.0) 13503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:46 smithi115 bash[46934]: cluster 2022-04-22T22:07:44.799009+0000 mgr.y (mgr.34107) 9607 : cluster [DBG] pgmap v10390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:46 smithi115 bash[46934]: audit 2022-04-22T22:07:45.677497+0000 mon.a (mon.0) 13503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:47 smithi121 bash[36902]: audit 2022-04-22T22:07:46.838949+0000 mon.a (mon.0) 13504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:47 smithi121 bash[36902]: audit 2022-04-22T22:07:47.087597+0000 mon.a (mon.0) 13505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:47 smithi115 bash[45495]: audit 2022-04-22T22:07:46.838949+0000 mon.a (mon.0) 13504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:47 smithi115 bash[45495]: audit 2022-04-22T22:07:47.087597+0000 mon.a (mon.0) 13505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:47.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:47 smithi115 bash[46934]: audit 2022-04-22T22:07:46.838949+0000 mon.a (mon.0) 13504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:47.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:47 smithi115 bash[46934]: audit 2022-04-22T22:07:47.087597+0000 mon.a (mon.0) 13505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:48 smithi121 bash[36902]: cluster 2022-04-22T22:07:46.800113+0000 mgr.y (mgr.34107) 9608 : cluster [DBG] pgmap v10391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:48 smithi115 bash[45495]: cluster 2022-04-22T22:07:46.800113+0000 mgr.y (mgr.34107) 9608 : cluster [DBG] pgmap v10391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:48 smithi115 bash[46934]: cluster 2022-04-22T22:07:46.800113+0000 mgr.y (mgr.34107) 9608 : cluster [DBG] pgmap v10391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:49.355 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:07:49.698 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:07:49.699 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:07:49.699 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:07:49.699 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:07:49.699 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:07:49.699 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:07:49.700 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:07:49.701 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:07:49.702 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:07:49.702 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:07:50.108 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:07:50.109 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:07:50.110 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:07:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:50 smithi121 bash[36902]: cluster 2022-04-22T22:07:48.801305+0000 mgr.y (mgr.34107) 9609 : cluster [DBG] pgmap v10392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:50 smithi121 bash[36902]: audit 2022-04-22T22:07:49.334280+0000 mgr.y (mgr.34107) 9610 : audit [DBG] from='client.45324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:50 smithi121 bash[36902]: audit 2022-04-22T22:07:50.105008+0000 mon.c (mon.1) 332 : audit [DBG] from='client.? 172.21.15.115:0/4238265076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[45495]: cluster 2022-04-22T22:07:48.801305+0000 mgr.y (mgr.34107) 9609 : cluster [DBG] pgmap v10392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[45495]: audit 2022-04-22T22:07:49.334280+0000 mgr.y (mgr.34107) 9610 : audit [DBG] from='client.45324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[45495]: audit 2022-04-22T22:07:50.105008+0000 mon.c (mon.1) 332 : audit [DBG] from='client.? 172.21.15.115:0/4238265076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[46934]: cluster 2022-04-22T22:07:48.801305+0000 mgr.y (mgr.34107) 9609 : cluster [DBG] pgmap v10392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[46934]: audit 2022-04-22T22:07:49.334280+0000 mgr.y (mgr.34107) 9610 : audit [DBG] from='client.45324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:50.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:50 smithi115 bash[46934]: audit 2022-04-22T22:07:50.105008+0000 mon.c (mon.1) 332 : audit [DBG] from='client.? 172.21.15.115:0/4238265076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:07:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:51 smithi121 bash[36902]: audit 2022-04-22T22:07:49.673137+0000 mgr.y (mgr.34107) 9611 : audit [DBG] from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:51 smithi121 bash[36902]: audit 2022-04-22T22:07:50.680733+0000 mon.a (mon.0) 13506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:51 smithi115 bash[45495]: audit 2022-04-22T22:07:49.673137+0000 mgr.y (mgr.34107) 9611 : audit [DBG] from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:51 smithi115 bash[45495]: audit 2022-04-22T22:07:50.680733+0000 mon.a (mon.0) 13506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:51 smithi115 bash[46934]: audit 2022-04-22T22:07:49.673137+0000 mgr.y (mgr.34107) 9611 : audit [DBG] from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:07:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:51 smithi115 bash[46934]: audit 2022-04-22T22:07:50.680733+0000 mon.a (mon.0) 13506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:52 smithi121 bash[36902]: cluster 2022-04-22T22:07:50.802350+0000 mgr.y (mgr.34107) 9612 : cluster [DBG] pgmap v10393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:52 smithi121 bash[36902]: audit 2022-04-22T22:07:51.842786+0000 mon.a (mon.0) 13507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:52 smithi121 bash[36902]: audit 2022-04-22T22:07:52.090475+0000 mon.a (mon.0) 13508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[45495]: cluster 2022-04-22T22:07:50.802350+0000 mgr.y (mgr.34107) 9612 : cluster [DBG] pgmap v10393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[45495]: audit 2022-04-22T22:07:51.842786+0000 mon.a (mon.0) 13507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[45495]: audit 2022-04-22T22:07:52.090475+0000 mon.a (mon.0) 13508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[46934]: cluster 2022-04-22T22:07:50.802350+0000 mgr.y (mgr.34107) 9612 : cluster [DBG] pgmap v10393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[46934]: audit 2022-04-22T22:07:51.842786+0000 mon.a (mon.0) 13507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[46934]: audit 2022-04-22T22:07:52.090475+0000 mon.a (mon.0) 13508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:52.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:54 smithi121 bash[36902]: cluster 2022-04-22T22:07:52.803534+0000 mgr.y (mgr.34107) 9613 : cluster [DBG] pgmap v10394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:54 smithi115 bash[45495]: cluster 2022-04-22T22:07:52.803534+0000 mgr.y (mgr.34107) 9613 : cluster [DBG] pgmap v10394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:54 smithi115 bash[46934]: cluster 2022-04-22T22:07:52.803534+0000 mgr.y (mgr.34107) 9613 : cluster [DBG] pgmap v10394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:07:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:07:54.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:07:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:56 smithi121 bash[36902]: cluster 2022-04-22T22:07:54.804615+0000 mgr.y (mgr.34107) 9614 : cluster [DBG] pgmap v10395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:56 smithi121 bash[36902]: audit 2022-04-22T22:07:55.683802+0000 mon.a (mon.0) 13509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:56 smithi115 bash[45495]: cluster 2022-04-22T22:07:54.804615+0000 mgr.y (mgr.34107) 9614 : cluster [DBG] pgmap v10395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:56 smithi115 bash[45495]: audit 2022-04-22T22:07:55.683802+0000 mon.a (mon.0) 13509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:56 smithi115 bash[46934]: cluster 2022-04-22T22:07:54.804615+0000 mgr.y (mgr.34107) 9614 : cluster [DBG] pgmap v10395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:56 smithi115 bash[46934]: audit 2022-04-22T22:07:55.683802+0000 mon.a (mon.0) 13509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:07:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:57 smithi121 bash[36902]: audit 2022-04-22T22:07:56.846542+0000 mon.a (mon.0) 13510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:57 smithi121 bash[36902]: audit 2022-04-22T22:07:57.093048+0000 mon.a (mon.0) 13511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:57 smithi115 bash[45495]: audit 2022-04-22T22:07:56.846542+0000 mon.a (mon.0) 13510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:57 smithi115 bash[45495]: audit 2022-04-22T22:07:57.093048+0000 mon.a (mon.0) 13511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:57 smithi115 bash[46934]: audit 2022-04-22T22:07:56.846542+0000 mon.a (mon.0) 13510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:07:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:57 smithi115 bash[46934]: audit 2022-04-22T22:07:57.093048+0000 mon.a (mon.0) 13511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:07:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:07:58 smithi121 bash[36902]: cluster 2022-04-22T22:07:56.805710+0000 mgr.y (mgr.34107) 9615 : cluster [DBG] pgmap v10396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:07:58 smithi115 bash[45495]: cluster 2022-04-22T22:07:56.805710+0000 mgr.y (mgr.34107) 9615 : cluster [DBG] pgmap v10396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:07:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:07:58 smithi115 bash[46934]: cluster 2022-04-22T22:07:56.805710+0000 mgr.y (mgr.34107) 9615 : cluster [DBG] pgmap v10396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:00 smithi121 bash[36902]: cluster 2022-04-22T22:07:58.806901+0000 mgr.y (mgr.34107) 9616 : cluster [DBG] pgmap v10397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:00 smithi115 bash[45495]: cluster 2022-04-22T22:07:58.806901+0000 mgr.y (mgr.34107) 9616 : cluster [DBG] pgmap v10397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:00 smithi115 bash[46934]: cluster 2022-04-22T22:07:58.806901+0000 mgr.y (mgr.34107) 9616 : cluster [DBG] pgmap v10397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:01 smithi121 bash[36902]: audit 2022-04-22T22:08:00.686864+0000 mon.a (mon.0) 13512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:01 smithi115 bash[45495]: audit 2022-04-22T22:08:00.686864+0000 mon.a (mon.0) 13512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:01 smithi115 bash[46934]: audit 2022-04-22T22:08:00.686864+0000 mon.a (mon.0) 13512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:02 smithi121 bash[36902]: cluster 2022-04-22T22:08:00.808029+0000 mgr.y (mgr.34107) 9617 : cluster [DBG] pgmap v10398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:02 smithi121 bash[36902]: audit 2022-04-22T22:08:01.850417+0000 mon.a (mon.0) 13513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:02 smithi121 bash[36902]: audit 2022-04-22T22:08:02.096115+0000 mon.a (mon.0) 13514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[45495]: cluster 2022-04-22T22:08:00.808029+0000 mgr.y (mgr.34107) 9617 : cluster [DBG] pgmap v10398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[45495]: audit 2022-04-22T22:08:01.850417+0000 mon.a (mon.0) 13513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[45495]: audit 2022-04-22T22:08:02.096115+0000 mon.a (mon.0) 13514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[46934]: cluster 2022-04-22T22:08:00.808029+0000 mgr.y (mgr.34107) 9617 : cluster [DBG] pgmap v10398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[46934]: audit 2022-04-22T22:08:01.850417+0000 mon.a (mon.0) 13513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[46934]: audit 2022-04-22T22:08:02.096115+0000 mon.a (mon.0) 13514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:02.729Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:04 smithi121 bash[36902]: cluster 2022-04-22T22:08:02.809147+0000 mgr.y (mgr.34107) 9618 : cluster [DBG] pgmap v10399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:04 smithi115 bash[45495]: cluster 2022-04-22T22:08:02.809147+0000 mgr.y (mgr.34107) 9618 : cluster [DBG] pgmap v10399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:04 smithi115 bash[46934]: cluster 2022-04-22T22:08:02.809147+0000 mgr.y (mgr.34107) 9618 : cluster [DBG] pgmap v10399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:04.583Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:06 smithi121 bash[36902]: cluster 2022-04-22T22:08:04.810250+0000 mgr.y (mgr.34107) 9619 : cluster [DBG] pgmap v10400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:06 smithi121 bash[36902]: audit 2022-04-22T22:08:05.690192+0000 mon.a (mon.0) 13515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:06 smithi115 bash[45495]: cluster 2022-04-22T22:08:04.810250+0000 mgr.y (mgr.34107) 9619 : cluster [DBG] pgmap v10400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:06 smithi115 bash[45495]: audit 2022-04-22T22:08:05.690192+0000 mon.a (mon.0) 13515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:06 smithi115 bash[46934]: cluster 2022-04-22T22:08:04.810250+0000 mgr.y (mgr.34107) 9619 : cluster [DBG] pgmap v10400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:06 smithi115 bash[46934]: audit 2022-04-22T22:08:05.690192+0000 mon.a (mon.0) 13515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:07 smithi121 bash[36902]: audit 2022-04-22T22:08:06.854164+0000 mon.a (mon.0) 13516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:07 smithi121 bash[36902]: audit 2022-04-22T22:08:07.098894+0000 mon.a (mon.0) 13517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:07 smithi115 bash[45495]: audit 2022-04-22T22:08:06.854164+0000 mon.a (mon.0) 13516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:07 smithi115 bash[45495]: audit 2022-04-22T22:08:07.098894+0000 mon.a (mon.0) 13517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:07 smithi115 bash[46934]: audit 2022-04-22T22:08:06.854164+0000 mon.a (mon.0) 13516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:07 smithi115 bash[46934]: audit 2022-04-22T22:08:07.098894+0000 mon.a (mon.0) 13517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:08 smithi121 bash[36902]: cluster 2022-04-22T22:08:06.811339+0000 mgr.y (mgr.34107) 9620 : cluster [DBG] pgmap v10401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:08 smithi115 bash[45495]: cluster 2022-04-22T22:08:06.811339+0000 mgr.y (mgr.34107) 9620 : cluster [DBG] pgmap v10401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:08 smithi115 bash[46934]: cluster 2022-04-22T22:08:06.811339+0000 mgr.y (mgr.34107) 9620 : cluster [DBG] pgmap v10401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:10 smithi121 bash[36902]: cluster 2022-04-22T22:08:08.812601+0000 mgr.y (mgr.34107) 9621 : cluster [DBG] pgmap v10402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:10.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:10 smithi115 bash[45495]: cluster 2022-04-22T22:08:08.812601+0000 mgr.y (mgr.34107) 9621 : cluster [DBG] pgmap v10402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:10.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:10 smithi115 bash[46934]: cluster 2022-04-22T22:08:08.812601+0000 mgr.y (mgr.34107) 9621 : cluster [DBG] pgmap v10402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:11 smithi121 bash[36902]: audit 2022-04-22T22:08:10.693593+0000 mon.a (mon.0) 13518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:11 smithi115 bash[45495]: audit 2022-04-22T22:08:10.693593+0000 mon.a (mon.0) 13518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:11 smithi115 bash[46934]: audit 2022-04-22T22:08:10.693593+0000 mon.a (mon.0) 13518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:12 smithi121 bash[36902]: cluster 2022-04-22T22:08:10.813685+0000 mgr.y (mgr.34107) 9622 : cluster [DBG] pgmap v10403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:12 smithi121 bash[36902]: audit 2022-04-22T22:08:11.857876+0000 mon.a (mon.0) 13519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:12 smithi121 bash[36902]: audit 2022-04-22T22:08:12.101974+0000 mon.a (mon.0) 13520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[45495]: cluster 2022-04-22T22:08:10.813685+0000 mgr.y (mgr.34107) 9622 : cluster [DBG] pgmap v10403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[45495]: audit 2022-04-22T22:08:11.857876+0000 mon.a (mon.0) 13519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[45495]: audit 2022-04-22T22:08:12.101974+0000 mon.a (mon.0) 13520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[46934]: cluster 2022-04-22T22:08:10.813685+0000 mgr.y (mgr.34107) 9622 : cluster [DBG] pgmap v10403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[46934]: audit 2022-04-22T22:08:11.857876+0000 mon.a (mon.0) 13519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[46934]: audit 2022-04-22T22:08:12.101974+0000 mon.a (mon.0) 13520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:12.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:14 smithi121 bash[36902]: cluster 2022-04-22T22:08:12.814794+0000 mgr.y (mgr.34107) 9623 : cluster [DBG] pgmap v10404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:14 smithi115 bash[45495]: cluster 2022-04-22T22:08:12.814794+0000 mgr.y (mgr.34107) 9623 : cluster [DBG] pgmap v10404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:14 smithi115 bash[46934]: cluster 2022-04-22T22:08:12.814794+0000 mgr.y (mgr.34107) 9623 : cluster [DBG] pgmap v10404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:14.584Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:16 smithi121 bash[36902]: cluster 2022-04-22T22:08:14.815917+0000 mgr.y (mgr.34107) 9624 : cluster [DBG] pgmap v10405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:16 smithi121 bash[36902]: audit 2022-04-22T22:08:15.696947+0000 mon.a (mon.0) 13521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:16 smithi115 bash[45495]: cluster 2022-04-22T22:08:14.815917+0000 mgr.y (mgr.34107) 9624 : cluster [DBG] pgmap v10405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:16 smithi115 bash[45495]: audit 2022-04-22T22:08:15.696947+0000 mon.a (mon.0) 13521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:16 smithi115 bash[46934]: cluster 2022-04-22T22:08:14.815917+0000 mgr.y (mgr.34107) 9624 : cluster [DBG] pgmap v10405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:16 smithi115 bash[46934]: audit 2022-04-22T22:08:15.696947+0000 mon.a (mon.0) 13521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[45495]: audit 2022-04-22T22:08:16.526956+0000 mon.a (mon.0) 13522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[45495]: audit 2022-04-22T22:08:16.527383+0000 mon.a (mon.0) 13523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[45495]: audit 2022-04-22T22:08:16.861466+0000 mon.a (mon.0) 13524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[45495]: audit 2022-04-22T22:08:17.105096+0000 mon.a (mon.0) 13525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[46934]: audit 2022-04-22T22:08:16.526956+0000 mon.a (mon.0) 13522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:08:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[46934]: audit 2022-04-22T22:08:16.527383+0000 mon.a (mon.0) 13523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:08:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[46934]: audit 2022-04-22T22:08:16.861466+0000 mon.a (mon.0) 13524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:17 smithi115 bash[46934]: audit 2022-04-22T22:08:17.105096+0000 mon.a (mon.0) 13525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:17 smithi121 bash[36902]: audit 2022-04-22T22:08:16.526956+0000 mon.a (mon.0) 13522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:08:17.801 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:17 smithi121 bash[36902]: audit 2022-04-22T22:08:16.527383+0000 mon.a (mon.0) 13523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:08:17.802 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:17 smithi121 bash[36902]: audit 2022-04-22T22:08:16.861466+0000 mon.a (mon.0) 13524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:17.802 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:17 smithi121 bash[36902]: audit 2022-04-22T22:08:17.105096+0000 mon.a (mon.0) 13525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:18 smithi115 bash[45495]: cluster 2022-04-22T22:08:16.816886+0000 mgr.y (mgr.34107) 9625 : cluster [DBG] pgmap v10406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:18 smithi115 bash[46934]: cluster 2022-04-22T22:08:16.816886+0000 mgr.y (mgr.34107) 9625 : cluster [DBG] pgmap v10406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:18 smithi121 bash[36902]: cluster 2022-04-22T22:08:16.816886+0000 mgr.y (mgr.34107) 9625 : cluster [DBG] pgmap v10406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:20.483 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:08:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:20 smithi115 bash[46934]: cluster 2022-04-22T22:08:18.818061+0000 mgr.y (mgr.34107) 9626 : cluster [DBG] pgmap v10407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:20.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:20 smithi115 bash[45495]: cluster 2022-04-22T22:08:18.818061+0000 mgr.y (mgr.34107) 9626 : cluster [DBG] pgmap v10407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:20 smithi121 bash[36902]: cluster 2022-04-22T22:08:18.818061+0000 mgr.y (mgr.34107) 9626 : cluster [DBG] pgmap v10407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:20.827 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:08:20.827 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:08:20.827 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:08:20.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:08:20.829 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:08:21.236 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:08:21.236 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:08:21.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:08:21.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:08:21.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:21.239 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:08:21.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:08:21.239 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:08:21.239 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:08:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:21 smithi115 bash[45495]: audit 2022-04-22T22:08:20.699932+0000 mon.a (mon.0) 13526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:21 smithi115 bash[45495]: audit 2022-04-22T22:08:21.228144+0000 mon.c (mon.1) 333 : audit [DBG] from='client.? 172.21.15.115:0/2846982724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:21 smithi115 bash[46934]: audit 2022-04-22T22:08:20.699932+0000 mon.a (mon.0) 13526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:21 smithi115 bash[46934]: audit 2022-04-22T22:08:21.228144+0000 mon.c (mon.1) 333 : audit [DBG] from='client.? 172.21.15.115:0/2846982724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:21 smithi121 bash[36902]: audit 2022-04-22T22:08:20.699932+0000 mon.a (mon.0) 13526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:21 smithi121 bash[36902]: audit 2022-04-22T22:08:21.228144+0000 mon.c (mon.1) 333 : audit [DBG] from='client.? 172.21.15.115:0/2846982724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[45495]: audit 2022-04-22T22:08:20.463835+0000 mgr.y (mgr.34107) 9627 : audit [DBG] from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[45495]: audit 2022-04-22T22:08:20.802572+0000 mgr.y (mgr.34107) 9628 : audit [DBG] from='client.45348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[45495]: cluster 2022-04-22T22:08:20.819074+0000 mgr.y (mgr.34107) 9629 : cluster [DBG] pgmap v10408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[45495]: audit 2022-04-22T22:08:21.865124+0000 mon.a (mon.0) 13527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[45495]: audit 2022-04-22T22:08:22.107893+0000 mon.a (mon.0) 13528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[46934]: audit 2022-04-22T22:08:20.463835+0000 mgr.y (mgr.34107) 9627 : audit [DBG] from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[46934]: audit 2022-04-22T22:08:20.802572+0000 mgr.y (mgr.34107) 9628 : audit [DBG] from='client.45348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[46934]: cluster 2022-04-22T22:08:20.819074+0000 mgr.y (mgr.34107) 9629 : cluster [DBG] pgmap v10408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[46934]: audit 2022-04-22T22:08:21.865124+0000 mon.a (mon.0) 13527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[46934]: audit 2022-04-22T22:08:22.107893+0000 mon.a (mon.0) 13528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:22 smithi121 bash[36902]: audit 2022-04-22T22:08:20.463835+0000 mgr.y (mgr.34107) 9627 : audit [DBG] from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:22 smithi121 bash[36902]: audit 2022-04-22T22:08:20.802572+0000 mgr.y (mgr.34107) 9628 : audit [DBG] from='client.45348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:22 smithi121 bash[36902]: cluster 2022-04-22T22:08:20.819074+0000 mgr.y (mgr.34107) 9629 : cluster [DBG] pgmap v10408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:22 smithi121 bash[36902]: audit 2022-04-22T22:08:21.865124+0000 mon.a (mon.0) 13527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:22 smithi121 bash[36902]: audit 2022-04-22T22:08:22.107893+0000 mon.a (mon.0) 13528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:22.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:24.588 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:24 smithi115 bash[45495]: cluster 2022-04-22T22:08:22.820249+0000 mgr.y (mgr.34107) 9630 : cluster [DBG] pgmap v10409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:24.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:24 smithi115 bash[46934]: cluster 2022-04-22T22:08:22.820249+0000 mgr.y (mgr.34107) 9630 : cluster [DBG] pgmap v10409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:24 smithi121 bash[36902]: cluster 2022-04-22T22:08:22.820249+0000 mgr.y (mgr.34107) 9630 : cluster [DBG] pgmap v10409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:24.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:26 smithi121 bash[36902]: cluster 2022-04-22T22:08:24.821300+0000 mgr.y (mgr.34107) 9631 : cluster [DBG] pgmap v10410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:26 smithi121 bash[36902]: audit 2022-04-22T22:08:25.702952+0000 mon.a (mon.0) 13529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:26 smithi115 bash[46934]: cluster 2022-04-22T22:08:24.821300+0000 mgr.y (mgr.34107) 9631 : cluster [DBG] pgmap v10410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:26 smithi115 bash[46934]: audit 2022-04-22T22:08:25.702952+0000 mon.a (mon.0) 13529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:26 smithi115 bash[45495]: cluster 2022-04-22T22:08:24.821300+0000 mgr.y (mgr.34107) 9631 : cluster [DBG] pgmap v10410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:26.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:26 smithi115 bash[45495]: audit 2022-04-22T22:08:25.702952+0000 mon.a (mon.0) 13529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:27 smithi121 bash[36902]: audit 2022-04-22T22:08:26.868521+0000 mon.a (mon.0) 13530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:27 smithi121 bash[36902]: audit 2022-04-22T22:08:27.110819+0000 mon.a (mon.0) 13531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:27 smithi115 bash[46934]: audit 2022-04-22T22:08:26.868521+0000 mon.a (mon.0) 13530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:27 smithi115 bash[46934]: audit 2022-04-22T22:08:27.110819+0000 mon.a (mon.0) 13531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:27 smithi115 bash[45495]: audit 2022-04-22T22:08:26.868521+0000 mon.a (mon.0) 13530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:27 smithi115 bash[45495]: audit 2022-04-22T22:08:27.110819+0000 mon.a (mon.0) 13531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:28 smithi121 bash[36902]: cluster 2022-04-22T22:08:26.822430+0000 mgr.y (mgr.34107) 9632 : cluster [DBG] pgmap v10411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:28 smithi115 bash[45495]: cluster 2022-04-22T22:08:26.822430+0000 mgr.y (mgr.34107) 9632 : cluster [DBG] pgmap v10411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:28 smithi115 bash[46934]: cluster 2022-04-22T22:08:26.822430+0000 mgr.y (mgr.34107) 9632 : cluster [DBG] pgmap v10411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:30 smithi121 bash[36902]: cluster 2022-04-22T22:08:28.823656+0000 mgr.y (mgr.34107) 9633 : cluster [DBG] pgmap v10412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:30 smithi115 bash[45495]: cluster 2022-04-22T22:08:28.823656+0000 mgr.y (mgr.34107) 9633 : cluster [DBG] pgmap v10412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:30 smithi115 bash[46934]: cluster 2022-04-22T22:08:28.823656+0000 mgr.y (mgr.34107) 9633 : cluster [DBG] pgmap v10412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:31 smithi121 bash[36902]: audit 2022-04-22T22:08:30.706130+0000 mon.a (mon.0) 13532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:31.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:31 smithi115 bash[45495]: audit 2022-04-22T22:08:30.706130+0000 mon.a (mon.0) 13532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:31.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:31 smithi115 bash[46934]: audit 2022-04-22T22:08:30.706130+0000 mon.a (mon.0) 13532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:32.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[46934]: cluster 2022-04-22T22:08:30.824632+0000 mgr.y (mgr.34107) 9634 : cluster [DBG] pgmap v10413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:32.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[46934]: audit 2022-04-22T22:08:31.872131+0000 mon.a (mon.0) 13533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:32.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[46934]: audit 2022-04-22T22:08:32.113962+0000 mon.a (mon.0) 13534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:32.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[45495]: cluster 2022-04-22T22:08:30.824632+0000 mgr.y (mgr.34107) 9634 : cluster [DBG] pgmap v10413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:32.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[45495]: audit 2022-04-22T22:08:31.872131+0000 mon.a (mon.0) 13533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:32.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[45495]: audit 2022-04-22T22:08:32.113962+0000 mon.a (mon.0) 13534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:32 smithi121 bash[36902]: cluster 2022-04-22T22:08:30.824632+0000 mgr.y (mgr.34107) 9634 : cluster [DBG] pgmap v10413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:32 smithi121 bash[36902]: audit 2022-04-22T22:08:31.872131+0000 mon.a (mon.0) 13533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:32 smithi121 bash[36902]: audit 2022-04-22T22:08:32.113962+0000 mon.a (mon.0) 13534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:32.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:34.589 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:34 smithi115 bash[45495]: cluster 2022-04-22T22:08:32.825682+0000 mgr.y (mgr.34107) 9635 : cluster [DBG] pgmap v10414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:34.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:34 smithi115 bash[46934]: cluster 2022-04-22T22:08:32.825682+0000 mgr.y (mgr.34107) 9635 : cluster [DBG] pgmap v10414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:34 smithi121 bash[36902]: cluster 2022-04-22T22:08:32.825682+0000 mgr.y (mgr.34107) 9635 : cluster [DBG] pgmap v10414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:34.586Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:35 smithi121 bash[36902]: audit 2022-04-22T22:08:35.033206+0000 mon.a (mon.0) 13535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:08:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:35 smithi121 bash[36902]: audit 2022-04-22T22:08:35.036574+0000 mon.a (mon.0) 13536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:08:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:35 smithi121 bash[36902]: audit 2022-04-22T22:08:35.038461+0000 mon.a (mon.0) 13537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:08:35.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[45495]: audit 2022-04-22T22:08:35.033206+0000 mon.a (mon.0) 13535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:08:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[45495]: audit 2022-04-22T22:08:35.036574+0000 mon.a (mon.0) 13536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:08:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[45495]: audit 2022-04-22T22:08:35.038461+0000 mon.a (mon.0) 13537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:08:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[46934]: audit 2022-04-22T22:08:35.033206+0000 mon.a (mon.0) 13535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:08:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[46934]: audit 2022-04-22T22:08:35.036574+0000 mon.a (mon.0) 13536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:08:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:35 smithi115 bash[46934]: audit 2022-04-22T22:08:35.038461+0000 mon.a (mon.0) 13537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:08:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:36 smithi121 bash[36902]: cluster 2022-04-22T22:08:34.826779+0000 mgr.y (mgr.34107) 9636 : cluster [DBG] pgmap v10415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:36 smithi121 bash[36902]: audit 2022-04-22T22:08:35.406640+0000 mon.a (mon.0) 13538 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:08:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:36 smithi121 bash[36902]: audit 2022-04-22T22:08:35.709275+0000 mon.a (mon.0) 13539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[45495]: cluster 2022-04-22T22:08:34.826779+0000 mgr.y (mgr.34107) 9636 : cluster [DBG] pgmap v10415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[45495]: audit 2022-04-22T22:08:35.406640+0000 mon.a (mon.0) 13538 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:08:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[45495]: audit 2022-04-22T22:08:35.709275+0000 mon.a (mon.0) 13539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[46934]: cluster 2022-04-22T22:08:34.826779+0000 mgr.y (mgr.34107) 9636 : cluster [DBG] pgmap v10415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[46934]: audit 2022-04-22T22:08:35.406640+0000 mon.a (mon.0) 13538 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:08:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:36 smithi115 bash[46934]: audit 2022-04-22T22:08:35.709275+0000 mon.a (mon.0) 13539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:37 smithi121 bash[36902]: audit 2022-04-22T22:08:36.875076+0000 mon.a (mon.0) 13540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:37.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:37 smithi121 bash[36902]: audit 2022-04-22T22:08:37.116976+0000 mon.a (mon.0) 13541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:37.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:37 smithi115 bash[46934]: audit 2022-04-22T22:08:36.875076+0000 mon.a (mon.0) 13540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:37.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:37 smithi115 bash[46934]: audit 2022-04-22T22:08:37.116976+0000 mon.a (mon.0) 13541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:37.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:37 smithi115 bash[45495]: audit 2022-04-22T22:08:36.875076+0000 mon.a (mon.0) 13540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:37.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:37 smithi115 bash[45495]: audit 2022-04-22T22:08:37.116976+0000 mon.a (mon.0) 13541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:38 smithi121 bash[36902]: cluster 2022-04-22T22:08:36.827709+0000 mgr.y (mgr.34107) 9637 : cluster [DBG] pgmap v10416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:38 smithi115 bash[45495]: cluster 2022-04-22T22:08:36.827709+0000 mgr.y (mgr.34107) 9637 : cluster [DBG] pgmap v10416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:38 smithi115 bash[46934]: cluster 2022-04-22T22:08:36.827709+0000 mgr.y (mgr.34107) 9637 : cluster [DBG] pgmap v10416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:40 smithi121 bash[36902]: cluster 2022-04-22T22:08:38.828827+0000 mgr.y (mgr.34107) 9638 : cluster [DBG] pgmap v10417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:40 smithi115 bash[45495]: cluster 2022-04-22T22:08:38.828827+0000 mgr.y (mgr.34107) 9638 : cluster [DBG] pgmap v10417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:40 smithi115 bash[46934]: cluster 2022-04-22T22:08:38.828827+0000 mgr.y (mgr.34107) 9638 : cluster [DBG] pgmap v10417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:41 smithi121 bash[36902]: audit 2022-04-22T22:08:40.712228+0000 mon.a (mon.0) 13542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:41 smithi115 bash[46934]: audit 2022-04-22T22:08:40.712228+0000 mon.a (mon.0) 13542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:41 smithi115 bash[45495]: audit 2022-04-22T22:08:40.712228+0000 mon.a (mon.0) 13542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:42.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[45495]: cluster 2022-04-22T22:08:40.829871+0000 mgr.y (mgr.34107) 9639 : cluster [DBG] pgmap v10418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:42.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[45495]: audit 2022-04-22T22:08:41.878707+0000 mon.a (mon.0) 13543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:42.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[45495]: audit 2022-04-22T22:08:42.120101+0000 mon.a (mon.0) 13544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:42.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[46934]: cluster 2022-04-22T22:08:40.829871+0000 mgr.y (mgr.34107) 9639 : cluster [DBG] pgmap v10418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:42.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[46934]: audit 2022-04-22T22:08:41.878707+0000 mon.a (mon.0) 13543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:42.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[46934]: audit 2022-04-22T22:08:42.120101+0000 mon.a (mon.0) 13544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:42 smithi121 bash[36902]: cluster 2022-04-22T22:08:40.829871+0000 mgr.y (mgr.34107) 9639 : cluster [DBG] pgmap v10418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:42 smithi121 bash[36902]: audit 2022-04-22T22:08:41.878707+0000 mon.a (mon.0) 13543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:42 smithi121 bash[36902]: audit 2022-04-22T22:08:42.120101+0000 mon.a (mon.0) 13544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:42.731Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:43 smithi121 bash[36902]: cluster 2022-04-22T22:08:42.830974+0000 mgr.y (mgr.34107) 9640 : cluster [DBG] pgmap v10419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:43 smithi115 bash[45495]: cluster 2022-04-22T22:08:42.830974+0000 mgr.y (mgr.34107) 9640 : cluster [DBG] pgmap v10419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:43 smithi115 bash[46934]: cluster 2022-04-22T22:08:42.830974+0000 mgr.y (mgr.34107) 9640 : cluster [DBG] pgmap v10419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:44.586Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:45 smithi121 bash[36902]: cluster 2022-04-22T22:08:44.832022+0000 mgr.y (mgr.34107) 9641 : cluster [DBG] pgmap v10420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:45 smithi121 bash[36902]: audit 2022-04-22T22:08:45.715523+0000 mon.a (mon.0) 13545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:45 smithi115 bash[45495]: cluster 2022-04-22T22:08:44.832022+0000 mgr.y (mgr.34107) 9641 : cluster [DBG] pgmap v10420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:45 smithi115 bash[45495]: audit 2022-04-22T22:08:45.715523+0000 mon.a (mon.0) 13545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:45 smithi115 bash[46934]: cluster 2022-04-22T22:08:44.832022+0000 mgr.y (mgr.34107) 9641 : cluster [DBG] pgmap v10420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:45 smithi115 bash[46934]: audit 2022-04-22T22:08:45.715523+0000 mon.a (mon.0) 13545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:47 smithi121 bash[36902]: audit 2022-04-22T22:08:46.882496+0000 mon.a (mon.0) 13546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:47 smithi121 bash[36902]: audit 2022-04-22T22:08:47.123064+0000 mon.a (mon.0) 13547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:47 smithi115 bash[45495]: audit 2022-04-22T22:08:46.882496+0000 mon.a (mon.0) 13546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:47 smithi115 bash[45495]: audit 2022-04-22T22:08:47.123064+0000 mon.a (mon.0) 13547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:47 smithi115 bash[46934]: audit 2022-04-22T22:08:46.882496+0000 mon.a (mon.0) 13546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:47 smithi115 bash[46934]: audit 2022-04-22T22:08:47.123064+0000 mon.a (mon.0) 13547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:48 smithi121 bash[36902]: cluster 2022-04-22T22:08:46.833128+0000 mgr.y (mgr.34107) 9642 : cluster [DBG] pgmap v10421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:48 smithi115 bash[45495]: cluster 2022-04-22T22:08:46.833128+0000 mgr.y (mgr.34107) 9642 : cluster [DBG] pgmap v10421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:48 smithi115 bash[46934]: cluster 2022-04-22T22:08:46.833128+0000 mgr.y (mgr.34107) 9642 : cluster [DBG] pgmap v10421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:50 smithi121 bash[36902]: cluster 2022-04-22T22:08:48.834336+0000 mgr.y (mgr.34107) 9643 : cluster [DBG] pgmap v10422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:50 smithi115 bash[45495]: cluster 2022-04-22T22:08:48.834336+0000 mgr.y (mgr.34107) 9643 : cluster [DBG] pgmap v10422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:50 smithi115 bash[46934]: cluster 2022-04-22T22:08:48.834336+0000 mgr.y (mgr.34107) 9643 : cluster [DBG] pgmap v10422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:51 smithi121 bash[36902]: audit 2022-04-22T22:08:50.718622+0000 mon.a (mon.0) 13548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:51 smithi115 bash[45495]: audit 2022-04-22T22:08:50.718622+0000 mon.a (mon.0) 13548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:51 smithi115 bash[46934]: audit 2022-04-22T22:08:50.718622+0000 mon.a (mon.0) 13548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:51.603 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:08:51.948 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:08:51.948 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:08:51.948 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:08:51.948 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:08:51.948 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:08:51.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:08:51.950 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:08:52.364 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:08:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:08:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:08:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:08:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:08:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:08:52.367 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:08:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:52 smithi121 bash[36902]: cluster 2022-04-22T22:08:50.835379+0000 mgr.y (mgr.34107) 9644 : cluster [DBG] pgmap v10423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:52 smithi121 bash[36902]: audit 2022-04-22T22:08:51.886007+0000 mon.a (mon.0) 13549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:52 smithi121 bash[36902]: audit 2022-04-22T22:08:52.126271+0000 mon.a (mon.0) 13550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[45495]: cluster 2022-04-22T22:08:50.835379+0000 mgr.y (mgr.34107) 9644 : cluster [DBG] pgmap v10423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[45495]: audit 2022-04-22T22:08:51.886007+0000 mon.a (mon.0) 13549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[45495]: audit 2022-04-22T22:08:52.126271+0000 mon.a (mon.0) 13550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[46934]: cluster 2022-04-22T22:08:50.835379+0000 mgr.y (mgr.34107) 9644 : cluster [DBG] pgmap v10423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[46934]: audit 2022-04-22T22:08:51.886007+0000 mon.a (mon.0) 13549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[46934]: audit 2022-04-22T22:08:52.126271+0000 mon.a (mon.0) 13550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:52.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:53 smithi121 bash[36902]: audit 2022-04-22T22:08:51.584966+0000 mgr.y (mgr.34107) 9645 : audit [DBG] from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:53 smithi121 bash[36902]: audit 2022-04-22T22:08:51.925564+0000 mgr.y (mgr.34107) 9646 : audit [DBG] from='client.55297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:53 smithi121 bash[36902]: audit 2022-04-22T22:08:52.360973+0000 mon.c (mon.1) 334 : audit [DBG] from='client.? 172.21.15.115:0/3319943033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[45495]: audit 2022-04-22T22:08:51.584966+0000 mgr.y (mgr.34107) 9645 : audit [DBG] from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[45495]: audit 2022-04-22T22:08:51.925564+0000 mgr.y (mgr.34107) 9646 : audit [DBG] from='client.55297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[45495]: audit 2022-04-22T22:08:52.360973+0000 mon.c (mon.1) 334 : audit [DBG] from='client.? 172.21.15.115:0/3319943033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[46934]: audit 2022-04-22T22:08:51.584966+0000 mgr.y (mgr.34107) 9645 : audit [DBG] from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[46934]: audit 2022-04-22T22:08:51.925564+0000 mgr.y (mgr.34107) 9646 : audit [DBG] from='client.55297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:08:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:53 smithi115 bash[46934]: audit 2022-04-22T22:08:52.360973+0000 mon.c (mon.1) 334 : audit [DBG] from='client.? 172.21.15.115:0/3319943033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:08:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:54 smithi121 bash[36902]: cluster 2022-04-22T22:08:52.836404+0000 mgr.y (mgr.34107) 9647 : cluster [DBG] pgmap v10424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:54 smithi115 bash[45495]: cluster 2022-04-22T22:08:52.836404+0000 mgr.y (mgr.34107) 9647 : cluster [DBG] pgmap v10424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:54 smithi115 bash[46934]: cluster 2022-04-22T22:08:52.836404+0000 mgr.y (mgr.34107) 9647 : cluster [DBG] pgmap v10424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:08:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:08:54.587Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:08:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:56 smithi121 bash[36902]: cluster 2022-04-22T22:08:54.837551+0000 mgr.y (mgr.34107) 9648 : cluster [DBG] pgmap v10425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:56 smithi121 bash[36902]: audit 2022-04-22T22:08:55.721778+0000 mon.a (mon.0) 13551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:56 smithi115 bash[46934]: cluster 2022-04-22T22:08:54.837551+0000 mgr.y (mgr.34107) 9648 : cluster [DBG] pgmap v10425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:56 smithi115 bash[46934]: audit 2022-04-22T22:08:55.721778+0000 mon.a (mon.0) 13551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:56 smithi115 bash[45495]: cluster 2022-04-22T22:08:54.837551+0000 mgr.y (mgr.34107) 9648 : cluster [DBG] pgmap v10425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:56 smithi115 bash[45495]: audit 2022-04-22T22:08:55.721778+0000 mon.a (mon.0) 13551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:08:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:57 smithi121 bash[36902]: audit 2022-04-22T22:08:56.890087+0000 mon.a (mon.0) 13552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:57 smithi121 bash[36902]: audit 2022-04-22T22:08:57.129381+0000 mon.a (mon.0) 13553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:57 smithi115 bash[45495]: audit 2022-04-22T22:08:56.890087+0000 mon.a (mon.0) 13552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:57 smithi115 bash[45495]: audit 2022-04-22T22:08:57.129381+0000 mon.a (mon.0) 13553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:57 smithi115 bash[46934]: audit 2022-04-22T22:08:56.890087+0000 mon.a (mon.0) 13552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:08:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:57 smithi115 bash[46934]: audit 2022-04-22T22:08:57.129381+0000 mon.a (mon.0) 13553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:08:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:08:58 smithi121 bash[36902]: cluster 2022-04-22T22:08:56.838738+0000 mgr.y (mgr.34107) 9649 : cluster [DBG] pgmap v10426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:08:58 smithi115 bash[45495]: cluster 2022-04-22T22:08:56.838738+0000 mgr.y (mgr.34107) 9649 : cluster [DBG] pgmap v10426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:08:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:08:58 smithi115 bash[46934]: cluster 2022-04-22T22:08:56.838738+0000 mgr.y (mgr.34107) 9649 : cluster [DBG] pgmap v10426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:00 smithi121 bash[36902]: cluster 2022-04-22T22:08:58.839994+0000 mgr.y (mgr.34107) 9650 : cluster [DBG] pgmap v10427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:00 smithi115 bash[45495]: cluster 2022-04-22T22:08:58.839994+0000 mgr.y (mgr.34107) 9650 : cluster [DBG] pgmap v10427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:00 smithi115 bash[46934]: cluster 2022-04-22T22:08:58.839994+0000 mgr.y (mgr.34107) 9650 : cluster [DBG] pgmap v10427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:01 smithi121 bash[36902]: audit 2022-04-22T22:09:00.725018+0000 mon.a (mon.0) 13554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:01 smithi115 bash[45495]: audit 2022-04-22T22:09:00.725018+0000 mon.a (mon.0) 13554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:01 smithi115 bash[46934]: audit 2022-04-22T22:09:00.725018+0000 mon.a (mon.0) 13554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:02 smithi121 bash[36902]: cluster 2022-04-22T22:09:00.841055+0000 mgr.y (mgr.34107) 9651 : cluster [DBG] pgmap v10428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:02 smithi121 bash[36902]: audit 2022-04-22T22:09:01.895768+0000 mon.a (mon.0) 13555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:02 smithi121 bash[36902]: audit 2022-04-22T22:09:02.132492+0000 mon.a (mon.0) 13556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[45495]: cluster 2022-04-22T22:09:00.841055+0000 mgr.y (mgr.34107) 9651 : cluster [DBG] pgmap v10428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[45495]: audit 2022-04-22T22:09:01.895768+0000 mon.a (mon.0) 13555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[45495]: audit 2022-04-22T22:09:02.132492+0000 mon.a (mon.0) 13556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[46934]: cluster 2022-04-22T22:09:00.841055+0000 mgr.y (mgr.34107) 9651 : cluster [DBG] pgmap v10428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[46934]: audit 2022-04-22T22:09:01.895768+0000 mon.a (mon.0) 13555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[46934]: audit 2022-04-22T22:09:02.132492+0000 mon.a (mon.0) 13556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:02.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:04 smithi121 bash[36902]: cluster 2022-04-22T22:09:02.842190+0000 mgr.y (mgr.34107) 9652 : cluster [DBG] pgmap v10429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:04 smithi115 bash[45495]: cluster 2022-04-22T22:09:02.842190+0000 mgr.y (mgr.34107) 9652 : cluster [DBG] pgmap v10429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:04 smithi115 bash[46934]: cluster 2022-04-22T22:09:02.842190+0000 mgr.y (mgr.34107) 9652 : cluster [DBG] pgmap v10429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:04.588Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:06 smithi121 bash[36902]: cluster 2022-04-22T22:09:04.843297+0000 mgr.y (mgr.34107) 9653 : cluster [DBG] pgmap v10430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:06.707 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:06 smithi121 bash[36902]: audit 2022-04-22T22:09:05.728257+0000 mon.a (mon.0) 13557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:06.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:06 smithi115 bash[45495]: cluster 2022-04-22T22:09:04.843297+0000 mgr.y (mgr.34107) 9653 : cluster [DBG] pgmap v10430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:06.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:06 smithi115 bash[45495]: audit 2022-04-22T22:09:05.728257+0000 mon.a (mon.0) 13557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:06.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:06 smithi115 bash[46934]: cluster 2022-04-22T22:09:04.843297+0000 mgr.y (mgr.34107) 9653 : cluster [DBG] pgmap v10430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:06.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:06 smithi115 bash[46934]: audit 2022-04-22T22:09:05.728257+0000 mon.a (mon.0) 13557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:07 smithi121 bash[36902]: audit 2022-04-22T22:09:06.899386+0000 mon.a (mon.0) 13558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:07 smithi121 bash[36902]: audit 2022-04-22T22:09:07.135429+0000 mon.a (mon.0) 13559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:07 smithi115 bash[45495]: audit 2022-04-22T22:09:06.899386+0000 mon.a (mon.0) 13558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:07.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:07 smithi115 bash[45495]: audit 2022-04-22T22:09:07.135429+0000 mon.a (mon.0) 13559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:07 smithi115 bash[46934]: audit 2022-04-22T22:09:06.899386+0000 mon.a (mon.0) 13558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:07.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:07 smithi115 bash[46934]: audit 2022-04-22T22:09:07.135429+0000 mon.a (mon.0) 13559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:08 smithi121 bash[36902]: cluster 2022-04-22T22:09:06.844346+0000 mgr.y (mgr.34107) 9654 : cluster [DBG] pgmap v10431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:08 smithi115 bash[45495]: cluster 2022-04-22T22:09:06.844346+0000 mgr.y (mgr.34107) 9654 : cluster [DBG] pgmap v10431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:08 smithi115 bash[46934]: cluster 2022-04-22T22:09:06.844346+0000 mgr.y (mgr.34107) 9654 : cluster [DBG] pgmap v10431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:10 smithi121 bash[36902]: cluster 2022-04-22T22:09:08.845533+0000 mgr.y (mgr.34107) 9655 : cluster [DBG] pgmap v10432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:10 smithi115 bash[46934]: cluster 2022-04-22T22:09:08.845533+0000 mgr.y (mgr.34107) 9655 : cluster [DBG] pgmap v10432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:10 smithi115 bash[45495]: cluster 2022-04-22T22:09:08.845533+0000 mgr.y (mgr.34107) 9655 : cluster [DBG] pgmap v10432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:11 smithi121 bash[36902]: audit 2022-04-22T22:09:10.731420+0000 mon.a (mon.0) 13560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:11 smithi115 bash[45495]: audit 2022-04-22T22:09:10.731420+0000 mon.a (mon.0) 13560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:11 smithi115 bash[46934]: audit 2022-04-22T22:09:10.731420+0000 mon.a (mon.0) 13560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:12 smithi121 bash[36902]: cluster 2022-04-22T22:09:10.846622+0000 mgr.y (mgr.34107) 9656 : cluster [DBG] pgmap v10433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:12 smithi121 bash[36902]: audit 2022-04-22T22:09:11.903242+0000 mon.a (mon.0) 13561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:12 smithi121 bash[36902]: audit 2022-04-22T22:09:12.138568+0000 mon.a (mon.0) 13562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[45495]: cluster 2022-04-22T22:09:10.846622+0000 mgr.y (mgr.34107) 9656 : cluster [DBG] pgmap v10433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[45495]: audit 2022-04-22T22:09:11.903242+0000 mon.a (mon.0) 13561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[45495]: audit 2022-04-22T22:09:12.138568+0000 mon.a (mon.0) 13562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[46934]: cluster 2022-04-22T22:09:10.846622+0000 mgr.y (mgr.34107) 9656 : cluster [DBG] pgmap v10433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[46934]: audit 2022-04-22T22:09:11.903242+0000 mon.a (mon.0) 13561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[46934]: audit 2022-04-22T22:09:12.138568+0000 mon.a (mon.0) 13562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:12.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:14 smithi115 bash[46934]: cluster 2022-04-22T22:09:12.847878+0000 mgr.y (mgr.34107) 9657 : cluster [DBG] pgmap v10434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:14 smithi115 bash[45495]: cluster 2022-04-22T22:09:12.847878+0000 mgr.y (mgr.34107) 9657 : cluster [DBG] pgmap v10434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:14 smithi121 bash[36902]: cluster 2022-04-22T22:09:12.847878+0000 mgr.y (mgr.34107) 9657 : cluster [DBG] pgmap v10434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:14.588Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:16 smithi115 bash[45495]: cluster 2022-04-22T22:09:14.849027+0000 mgr.y (mgr.34107) 9658 : cluster [DBG] pgmap v10435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:16 smithi115 bash[45495]: audit 2022-04-22T22:09:15.734621+0000 mon.a (mon.0) 13563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:16 smithi115 bash[46934]: cluster 2022-04-22T22:09:14.849027+0000 mgr.y (mgr.34107) 9658 : cluster [DBG] pgmap v10435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:16.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:16 smithi115 bash[46934]: audit 2022-04-22T22:09:15.734621+0000 mon.a (mon.0) 13563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:16 smithi121 bash[36902]: cluster 2022-04-22T22:09:14.849027+0000 mgr.y (mgr.34107) 9658 : cluster [DBG] pgmap v10435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:16 smithi121 bash[36902]: audit 2022-04-22T22:09:15.734621+0000 mon.a (mon.0) 13563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[45495]: audit 2022-04-22T22:09:16.537601+0000 mon.a (mon.0) 13564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[45495]: audit 2022-04-22T22:09:16.538394+0000 mon.a (mon.0) 13565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[45495]: audit 2022-04-22T22:09:16.907325+0000 mon.a (mon.0) 13566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[45495]: audit 2022-04-22T22:09:17.141661+0000 mon.a (mon.0) 13567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[46934]: audit 2022-04-22T22:09:16.537601+0000 mon.a (mon.0) 13564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[46934]: audit 2022-04-22T22:09:16.538394+0000 mon.a (mon.0) 13565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:09:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[46934]: audit 2022-04-22T22:09:16.907325+0000 mon.a (mon.0) 13566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:17 smithi115 bash[46934]: audit 2022-04-22T22:09:17.141661+0000 mon.a (mon.0) 13567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:17 smithi121 bash[36902]: audit 2022-04-22T22:09:16.537601+0000 mon.a (mon.0) 13564 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:09:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:17 smithi121 bash[36902]: audit 2022-04-22T22:09:16.538394+0000 mon.a (mon.0) 13565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:09:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:17 smithi121 bash[36902]: audit 2022-04-22T22:09:16.907325+0000 mon.a (mon.0) 13566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:17 smithi121 bash[36902]: audit 2022-04-22T22:09:17.141661+0000 mon.a (mon.0) 13567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:18 smithi115 bash[45495]: cluster 2022-04-22T22:09:16.850200+0000 mgr.y (mgr.34107) 9659 : cluster [DBG] pgmap v10436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:18 smithi115 bash[46934]: cluster 2022-04-22T22:09:16.850200+0000 mgr.y (mgr.34107) 9659 : cluster [DBG] pgmap v10436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:18 smithi121 bash[36902]: cluster 2022-04-22T22:09:16.850200+0000 mgr.y (mgr.34107) 9659 : cluster [DBG] pgmap v10436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:20 smithi115 bash[45495]: cluster 2022-04-22T22:09:18.851435+0000 mgr.y (mgr.34107) 9660 : cluster [DBG] pgmap v10437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:20 smithi115 bash[46934]: cluster 2022-04-22T22:09:18.851435+0000 mgr.y (mgr.34107) 9660 : cluster [DBG] pgmap v10437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:20 smithi121 bash[36902]: cluster 2022-04-22T22:09:18.851435+0000 mgr.y (mgr.34107) 9660 : cluster [DBG] pgmap v10437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:21 smithi121 bash[36902]: audit 2022-04-22T22:09:20.737696+0000 mon.a (mon.0) 13568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:21 smithi115 bash[45495]: audit 2022-04-22T22:09:20.737696+0000 mon.a (mon.0) 13568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:21 smithi115 bash[46934]: audit 2022-04-22T22:09:20.737696+0000 mon.a (mon.0) 13568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:22.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[45495]: cluster 2022-04-22T22:09:20.852402+0000 mgr.y (mgr.34107) 9661 : cluster [DBG] pgmap v10438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:22.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[45495]: audit 2022-04-22T22:09:21.911018+0000 mon.a (mon.0) 13569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:22.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[45495]: audit 2022-04-22T22:09:22.144516+0000 mon.a (mon.0) 13570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:22.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[46934]: cluster 2022-04-22T22:09:20.852402+0000 mgr.y (mgr.34107) 9661 : cluster [DBG] pgmap v10438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:22.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[46934]: audit 2022-04-22T22:09:21.911018+0000 mon.a (mon.0) 13569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:22.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[46934]: audit 2022-04-22T22:09:22.144516+0000 mon.a (mon.0) 13570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:22.738 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:09:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:22 smithi121 bash[36902]: cluster 2022-04-22T22:09:20.852402+0000 mgr.y (mgr.34107) 9661 : cluster [DBG] pgmap v10438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:22 smithi121 bash[36902]: audit 2022-04-22T22:09:21.911018+0000 mon.a (mon.0) 13569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:22 smithi121 bash[36902]: audit 2022-04-22T22:09:22.144516+0000 mon.a (mon.0) 13570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:22.731Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:23.076 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:09:23.077 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:09:23.078 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:09:23.479 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:09:23.479 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:09:23.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:09:23.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:23.480 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:09:23.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:09:23.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:23.484 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:09:23.484 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:09:23.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:09:23.507 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:09:23.507 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:09:24.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[46934]: audit 2022-04-22T22:09:22.718040+0000 mgr.y (mgr.34107) 9662 : audit [DBG] from='client.45378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[46934]: cluster 2022-04-22T22:09:22.853423+0000 mgr.y (mgr.34107) 9663 : cluster [DBG] pgmap v10439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[46934]: audit 2022-04-22T22:09:23.059466+0000 mgr.y (mgr.34107) 9664 : audit [DBG] from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[46934]: audit 2022-04-22T22:09:23.475205+0000 mon.c (mon.1) 335 : audit [DBG] from='client.? 172.21.15.115:0/313648695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[45495]: audit 2022-04-22T22:09:22.718040+0000 mgr.y (mgr.34107) 9662 : audit [DBG] from='client.45378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[45495]: cluster 2022-04-22T22:09:22.853423+0000 mgr.y (mgr.34107) 9663 : cluster [DBG] pgmap v10439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[45495]: audit 2022-04-22T22:09:23.059466+0000 mgr.y (mgr.34107) 9664 : audit [DBG] from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[45495]: audit 2022-04-22T22:09:23.475205+0000 mon.c (mon.1) 335 : audit [DBG] from='client.? 172.21.15.115:0/313648695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:24 smithi121 bash[36902]: audit 2022-04-22T22:09:22.718040+0000 mgr.y (mgr.34107) 9662 : audit [DBG] from='client.45378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.838 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:24 smithi121 bash[36902]: cluster 2022-04-22T22:09:22.853423+0000 mgr.y (mgr.34107) 9663 : cluster [DBG] pgmap v10439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:24.838 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:24 smithi121 bash[36902]: audit 2022-04-22T22:09:23.059466+0000 mgr.y (mgr.34107) 9664 : audit [DBG] from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:24.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:24 smithi121 bash[36902]: audit 2022-04-22T22:09:23.475205+0000 mon.c (mon.1) 335 : audit [DBG] from='client.? 172.21.15.115:0/313648695' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:24.588Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:26 smithi121 bash[36902]: cluster 2022-04-22T22:09:24.854625+0000 mgr.y (mgr.34107) 9665 : cluster [DBG] pgmap v10440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:26 smithi121 bash[36902]: audit 2022-04-22T22:09:25.740785+0000 mon.a (mon.0) 13571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:26 smithi115 bash[45495]: cluster 2022-04-22T22:09:24.854625+0000 mgr.y (mgr.34107) 9665 : cluster [DBG] pgmap v10440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:26 smithi115 bash[45495]: audit 2022-04-22T22:09:25.740785+0000 mon.a (mon.0) 13571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:26 smithi115 bash[46934]: cluster 2022-04-22T22:09:24.854625+0000 mgr.y (mgr.34107) 9665 : cluster [DBG] pgmap v10440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:26 smithi115 bash[46934]: audit 2022-04-22T22:09:25.740785+0000 mon.a (mon.0) 13571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:27 smithi121 bash[36902]: audit 2022-04-22T22:09:26.914500+0000 mon.a (mon.0) 13572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:27 smithi121 bash[36902]: audit 2022-04-22T22:09:27.147551+0000 mon.a (mon.0) 13573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:27 smithi115 bash[45495]: audit 2022-04-22T22:09:26.914500+0000 mon.a (mon.0) 13572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:27 smithi115 bash[45495]: audit 2022-04-22T22:09:27.147551+0000 mon.a (mon.0) 13573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:27 smithi115 bash[46934]: audit 2022-04-22T22:09:26.914500+0000 mon.a (mon.0) 13572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:27.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:27 smithi115 bash[46934]: audit 2022-04-22T22:09:27.147551+0000 mon.a (mon.0) 13573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:28 smithi121 bash[36902]: cluster 2022-04-22T22:09:26.855897+0000 mgr.y (mgr.34107) 9666 : cluster [DBG] pgmap v10441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:28 smithi115 bash[45495]: cluster 2022-04-22T22:09:26.855897+0000 mgr.y (mgr.34107) 9666 : cluster [DBG] pgmap v10441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:28 smithi115 bash[46934]: cluster 2022-04-22T22:09:26.855897+0000 mgr.y (mgr.34107) 9666 : cluster [DBG] pgmap v10441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:30 smithi121 bash[36902]: cluster 2022-04-22T22:09:28.857204+0000 mgr.y (mgr.34107) 9667 : cluster [DBG] pgmap v10442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:30 smithi115 bash[45495]: cluster 2022-04-22T22:09:28.857204+0000 mgr.y (mgr.34107) 9667 : cluster [DBG] pgmap v10442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:30 smithi115 bash[46934]: cluster 2022-04-22T22:09:28.857204+0000 mgr.y (mgr.34107) 9667 : cluster [DBG] pgmap v10442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:31 smithi121 bash[36902]: audit 2022-04-22T22:09:30.743875+0000 mon.a (mon.0) 13574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:31.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:31 smithi115 bash[45495]: audit 2022-04-22T22:09:30.743875+0000 mon.a (mon.0) 13574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:31.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:31 smithi115 bash[46934]: audit 2022-04-22T22:09:30.743875+0000 mon.a (mon.0) 13574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:32.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[45495]: cluster 2022-04-22T22:09:30.858280+0000 mgr.y (mgr.34107) 9668 : cluster [DBG] pgmap v10443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:32.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[45495]: audit 2022-04-22T22:09:31.918401+0000 mon.a (mon.0) 13575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:32.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[45495]: audit 2022-04-22T22:09:32.150650+0000 mon.a (mon.0) 13576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:32.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[46934]: cluster 2022-04-22T22:09:30.858280+0000 mgr.y (mgr.34107) 9668 : cluster [DBG] pgmap v10443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:32.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[46934]: audit 2022-04-22T22:09:31.918401+0000 mon.a (mon.0) 13575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:32.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[46934]: audit 2022-04-22T22:09:32.150650+0000 mon.a (mon.0) 13576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:32 smithi121 bash[36902]: cluster 2022-04-22T22:09:30.858280+0000 mgr.y (mgr.34107) 9668 : cluster [DBG] pgmap v10443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:32 smithi121 bash[36902]: audit 2022-04-22T22:09:31.918401+0000 mon.a (mon.0) 13575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:32 smithi121 bash[36902]: audit 2022-04-22T22:09:32.150650+0000 mon.a (mon.0) 13576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:32.734Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:34.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:34 smithi115 bash[45495]: cluster 2022-04-22T22:09:32.859392+0000 mgr.y (mgr.34107) 9669 : cluster [DBG] pgmap v10444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:34.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:34 smithi115 bash[46934]: cluster 2022-04-22T22:09:32.859392+0000 mgr.y (mgr.34107) 9669 : cluster [DBG] pgmap v10444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:34 smithi121 bash[36902]: cluster 2022-04-22T22:09:32.859392+0000 mgr.y (mgr.34107) 9669 : cluster [DBG] pgmap v10444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:34.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: cluster 2022-04-22T22:09:34.860455+0000 mgr.y (mgr.34107) 9670 : cluster [DBG] pgmap v10445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:36.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: audit 2022-04-22T22:09:35.414255+0000 mon.a (mon.0) 13577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:09:36.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: audit 2022-04-22T22:09:35.417230+0000 mon.a (mon.0) 13578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:09:36.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: audit 2022-04-22T22:09:35.418997+0000 mon.a (mon.0) 13579 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:09:36.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: audit 2022-04-22T22:09:35.746822+0000 mon.a (mon.0) 13580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:36.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:36 smithi121 bash[36902]: audit 2022-04-22T22:09:35.788787+0000 mon.a (mon.0) 13581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: cluster 2022-04-22T22:09:34.860455+0000 mgr.y (mgr.34107) 9670 : cluster [DBG] pgmap v10445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: audit 2022-04-22T22:09:35.414255+0000 mon.a (mon.0) 13577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: audit 2022-04-22T22:09:35.417230+0000 mon.a (mon.0) 13578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: audit 2022-04-22T22:09:35.418997+0000 mon.a (mon.0) 13579 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: audit 2022-04-22T22:09:35.746822+0000 mon.a (mon.0) 13580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:36.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[46934]: audit 2022-04-22T22:09:35.788787+0000 mon.a (mon.0) 13581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: cluster 2022-04-22T22:09:34.860455+0000 mgr.y (mgr.34107) 9670 : cluster [DBG] pgmap v10445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: audit 2022-04-22T22:09:35.414255+0000 mon.a (mon.0) 13577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: audit 2022-04-22T22:09:35.417230+0000 mon.a (mon.0) 13578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: audit 2022-04-22T22:09:35.418997+0000 mon.a (mon.0) 13579 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: audit 2022-04-22T22:09:35.746822+0000 mon.a (mon.0) 13580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:36.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:36 smithi115 bash[45495]: audit 2022-04-22T22:09:35.788787+0000 mon.a (mon.0) 13581 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:09:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:37 smithi121 bash[36902]: audit 2022-04-22T22:09:36.922168+0000 mon.a (mon.0) 13582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:37 smithi121 bash[36902]: audit 2022-04-22T22:09:37.153566+0000 mon.a (mon.0) 13583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:37.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:37 smithi115 bash[46934]: audit 2022-04-22T22:09:36.922168+0000 mon.a (mon.0) 13582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:37.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:37 smithi115 bash[46934]: audit 2022-04-22T22:09:37.153566+0000 mon.a (mon.0) 13583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:37 smithi115 bash[45495]: audit 2022-04-22T22:09:36.922168+0000 mon.a (mon.0) 13582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:37 smithi115 bash[45495]: audit 2022-04-22T22:09:37.153566+0000 mon.a (mon.0) 13583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:38 smithi121 bash[36902]: cluster 2022-04-22T22:09:36.861567+0000 mgr.y (mgr.34107) 9671 : cluster [DBG] pgmap v10446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:38 smithi115 bash[45495]: cluster 2022-04-22T22:09:36.861567+0000 mgr.y (mgr.34107) 9671 : cluster [DBG] pgmap v10446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:38 smithi115 bash[46934]: cluster 2022-04-22T22:09:36.861567+0000 mgr.y (mgr.34107) 9671 : cluster [DBG] pgmap v10446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:40 smithi121 bash[36902]: cluster 2022-04-22T22:09:38.862774+0000 mgr.y (mgr.34107) 9672 : cluster [DBG] pgmap v10447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:40.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:40 smithi115 bash[46934]: cluster 2022-04-22T22:09:38.862774+0000 mgr.y (mgr.34107) 9672 : cluster [DBG] pgmap v10447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:40 smithi115 bash[45495]: cluster 2022-04-22T22:09:38.862774+0000 mgr.y (mgr.34107) 9672 : cluster [DBG] pgmap v10447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:41 smithi121 bash[36902]: audit 2022-04-22T22:09:40.749400+0000 mon.a (mon.0) 13584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:41.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:41 smithi115 bash[46934]: audit 2022-04-22T22:09:40.749400+0000 mon.a (mon.0) 13584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:41.829 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:41 smithi115 bash[45495]: audit 2022-04-22T22:09:40.749400+0000 mon.a (mon.0) 13584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:42.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[45495]: cluster 2022-04-22T22:09:40.863883+0000 mgr.y (mgr.34107) 9673 : cluster [DBG] pgmap v10448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:42.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[45495]: audit 2022-04-22T22:09:41.925770+0000 mon.a (mon.0) 13585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:42.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[45495]: audit 2022-04-22T22:09:42.156512+0000 mon.a (mon.0) 13586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:42.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[46934]: cluster 2022-04-22T22:09:40.863883+0000 mgr.y (mgr.34107) 9673 : cluster [DBG] pgmap v10448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:42.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[46934]: audit 2022-04-22T22:09:41.925770+0000 mon.a (mon.0) 13585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:42.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[46934]: audit 2022-04-22T22:09:42.156512+0000 mon.a (mon.0) 13586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:42 smithi121 bash[36902]: cluster 2022-04-22T22:09:40.863883+0000 mgr.y (mgr.34107) 9673 : cluster [DBG] pgmap v10448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:42 smithi121 bash[36902]: audit 2022-04-22T22:09:41.925770+0000 mon.a (mon.0) 13585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:42 smithi121 bash[36902]: audit 2022-04-22T22:09:42.156512+0000 mon.a (mon.0) 13586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:42.736Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:44 smithi121 bash[36902]: cluster 2022-04-22T22:09:42.865121+0000 mgr.y (mgr.34107) 9674 : cluster [DBG] pgmap v10449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:44 smithi115 bash[45495]: cluster 2022-04-22T22:09:42.865121+0000 mgr.y (mgr.34107) 9674 : cluster [DBG] pgmap v10449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:44.590Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:44 smithi115 bash[46934]: cluster 2022-04-22T22:09:42.865121+0000 mgr.y (mgr.34107) 9674 : cluster [DBG] pgmap v10449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:46 smithi121 bash[36902]: cluster 2022-04-22T22:09:44.866357+0000 mgr.y (mgr.34107) 9675 : cluster [DBG] pgmap v10450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:46 smithi121 bash[36902]: audit 2022-04-22T22:09:45.752737+0000 mon.a (mon.0) 13587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:46 smithi115 bash[45495]: cluster 2022-04-22T22:09:44.866357+0000 mgr.y (mgr.34107) 9675 : cluster [DBG] pgmap v10450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:46 smithi115 bash[45495]: audit 2022-04-22T22:09:45.752737+0000 mon.a (mon.0) 13587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:46 smithi115 bash[46934]: cluster 2022-04-22T22:09:44.866357+0000 mgr.y (mgr.34107) 9675 : cluster [DBG] pgmap v10450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:46 smithi115 bash[46934]: audit 2022-04-22T22:09:45.752737+0000 mon.a (mon.0) 13587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:47 smithi121 bash[36902]: audit 2022-04-22T22:09:46.929406+0000 mon.a (mon.0) 13588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:47 smithi121 bash[36902]: audit 2022-04-22T22:09:47.159649+0000 mon.a (mon.0) 13589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:47 smithi115 bash[45495]: audit 2022-04-22T22:09:46.929406+0000 mon.a (mon.0) 13588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:47 smithi115 bash[45495]: audit 2022-04-22T22:09:47.159649+0000 mon.a (mon.0) 13589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:47 smithi115 bash[46934]: audit 2022-04-22T22:09:46.929406+0000 mon.a (mon.0) 13588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:47 smithi115 bash[46934]: audit 2022-04-22T22:09:47.159649+0000 mon.a (mon.0) 13589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:48 smithi121 bash[36902]: cluster 2022-04-22T22:09:46.867507+0000 mgr.y (mgr.34107) 9676 : cluster [DBG] pgmap v10451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:48 smithi115 bash[45495]: cluster 2022-04-22T22:09:46.867507+0000 mgr.y (mgr.34107) 9676 : cluster [DBG] pgmap v10451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:48 smithi115 bash[46934]: cluster 2022-04-22T22:09:46.867507+0000 mgr.y (mgr.34107) 9676 : cluster [DBG] pgmap v10451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:50 smithi121 bash[36902]: cluster 2022-04-22T22:09:48.868711+0000 mgr.y (mgr.34107) 9677 : cluster [DBG] pgmap v10452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:50.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:50 smithi115 bash[46934]: cluster 2022-04-22T22:09:48.868711+0000 mgr.y (mgr.34107) 9677 : cluster [DBG] pgmap v10452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:50 smithi115 bash[45495]: cluster 2022-04-22T22:09:48.868711+0000 mgr.y (mgr.34107) 9677 : cluster [DBG] pgmap v10452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:51 smithi121 bash[36902]: audit 2022-04-22T22:09:50.755899+0000 mon.a (mon.0) 13590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:51 smithi115 bash[45495]: audit 2022-04-22T22:09:50.755899+0000 mon.a (mon.0) 13590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:51 smithi115 bash[46934]: audit 2022-04-22T22:09:50.755899+0000 mon.a (mon.0) 13590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:52.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[46934]: cluster 2022-04-22T22:09:50.869710+0000 mgr.y (mgr.34107) 9678 : cluster [DBG] pgmap v10453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:52.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[46934]: audit 2022-04-22T22:09:51.933244+0000 mon.a (mon.0) 13591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:52.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[46934]: audit 2022-04-22T22:09:52.162660+0000 mon.a (mon.0) 13592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:52.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[45495]: cluster 2022-04-22T22:09:50.869710+0000 mgr.y (mgr.34107) 9678 : cluster [DBG] pgmap v10453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:52.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[45495]: audit 2022-04-22T22:09:51.933244+0000 mon.a (mon.0) 13591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:52.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[45495]: audit 2022-04-22T22:09:52.162660+0000 mon.a (mon.0) 13592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:52 smithi121 bash[36902]: cluster 2022-04-22T22:09:50.869710+0000 mgr.y (mgr.34107) 9678 : cluster [DBG] pgmap v10453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:52 smithi121 bash[36902]: audit 2022-04-22T22:09:51.933244+0000 mon.a (mon.0) 13591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:52.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:52 smithi121 bash[36902]: audit 2022-04-22T22:09:52.162660+0000 mon.a (mon.0) 13592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:52.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:53.852 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:09:54.193 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:09:54.193 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:09:54.194 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:09:54.195 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:09:54.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:09:54.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:09:54.196 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:09:54.196 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:09:54.592 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:09:54.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:09:54.594 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:09:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:54 smithi121 bash[36902]: cluster 2022-04-22T22:09:52.870807+0000 mgr.y (mgr.34107) 9679 : cluster [DBG] pgmap v10454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:54 smithi115 bash[45495]: cluster 2022-04-22T22:09:52.870807+0000 mgr.y (mgr.34107) 9679 : cluster [DBG] pgmap v10454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:09:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:09:54.589Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:09:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:54 smithi115 bash[46934]: cluster 2022-04-22T22:09:52.870807+0000 mgr.y (mgr.34107) 9679 : cluster [DBG] pgmap v10454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:55 smithi121 bash[36902]: audit 2022-04-22T22:09:53.833193+0000 mgr.y (mgr.34107) 9680 : audit [DBG] from='client.45396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:55 smithi121 bash[36902]: audit 2022-04-22T22:09:54.169875+0000 mgr.y (mgr.34107) 9681 : audit [DBG] from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:55 smithi121 bash[36902]: audit 2022-04-22T22:09:54.589217+0000 mon.c (mon.1) 336 : audit [DBG] from='client.? 172.21.15.115:0/1102550376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[45495]: audit 2022-04-22T22:09:53.833193+0000 mgr.y (mgr.34107) 9680 : audit [DBG] from='client.45396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[45495]: audit 2022-04-22T22:09:54.169875+0000 mgr.y (mgr.34107) 9681 : audit [DBG] from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[45495]: audit 2022-04-22T22:09:54.589217+0000 mon.c (mon.1) 336 : audit [DBG] from='client.? 172.21.15.115:0/1102550376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[46934]: audit 2022-04-22T22:09:53.833193+0000 mgr.y (mgr.34107) 9680 : audit [DBG] from='client.45396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[46934]: audit 2022-04-22T22:09:54.169875+0000 mgr.y (mgr.34107) 9681 : audit [DBG] from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:09:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:55 smithi115 bash[46934]: audit 2022-04-22T22:09:54.589217+0000 mon.c (mon.1) 336 : audit [DBG] from='client.? 172.21.15.115:0/1102550376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:09:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:56 smithi121 bash[36902]: cluster 2022-04-22T22:09:54.871927+0000 mgr.y (mgr.34107) 9682 : cluster [DBG] pgmap v10455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:56 smithi121 bash[36902]: audit 2022-04-22T22:09:55.759037+0000 mon.a (mon.0) 13593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:56 smithi115 bash[45495]: cluster 2022-04-22T22:09:54.871927+0000 mgr.y (mgr.34107) 9682 : cluster [DBG] pgmap v10455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:56 smithi115 bash[45495]: audit 2022-04-22T22:09:55.759037+0000 mon.a (mon.0) 13593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:56 smithi115 bash[46934]: cluster 2022-04-22T22:09:54.871927+0000 mgr.y (mgr.34107) 9682 : cluster [DBG] pgmap v10455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:56 smithi115 bash[46934]: audit 2022-04-22T22:09:55.759037+0000 mon.a (mon.0) 13593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:09:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:57 smithi121 bash[36902]: audit 2022-04-22T22:09:56.937007+0000 mon.a (mon.0) 13594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:57 smithi121 bash[36902]: audit 2022-04-22T22:09:57.165514+0000 mon.a (mon.0) 13595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:57 smithi115 bash[45495]: audit 2022-04-22T22:09:56.937007+0000 mon.a (mon.0) 13594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:57 smithi115 bash[45495]: audit 2022-04-22T22:09:57.165514+0000 mon.a (mon.0) 13595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:57 smithi115 bash[46934]: audit 2022-04-22T22:09:56.937007+0000 mon.a (mon.0) 13594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:09:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:57 smithi115 bash[46934]: audit 2022-04-22T22:09:57.165514+0000 mon.a (mon.0) 13595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:09:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:09:58 smithi121 bash[36902]: cluster 2022-04-22T22:09:56.873094+0000 mgr.y (mgr.34107) 9683 : cluster [DBG] pgmap v10456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:09:58 smithi115 bash[45495]: cluster 2022-04-22T22:09:56.873094+0000 mgr.y (mgr.34107) 9683 : cluster [DBG] pgmap v10456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:09:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:09:58 smithi115 bash[46934]: cluster 2022-04-22T22:09:56.873094+0000 mgr.y (mgr.34107) 9683 : cluster [DBG] pgmap v10456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:00 smithi121 bash[36902]: cluster 2022-04-22T22:09:58.874237+0000 mgr.y (mgr.34107) 9684 : cluster [DBG] pgmap v10457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:00 smithi121 bash[36902]: cluster 2022-04-22T22:10:00.000179+0000 mon.a (mon.0) 13596 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:10:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:00 smithi115 bash[45495]: cluster 2022-04-22T22:09:58.874237+0000 mgr.y (mgr.34107) 9684 : cluster [DBG] pgmap v10457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:00 smithi115 bash[45495]: cluster 2022-04-22T22:10:00.000179+0000 mon.a (mon.0) 13596 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:10:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:00 smithi115 bash[46934]: cluster 2022-04-22T22:09:58.874237+0000 mgr.y (mgr.34107) 9684 : cluster [DBG] pgmap v10457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:00 smithi115 bash[46934]: cluster 2022-04-22T22:10:00.000179+0000 mon.a (mon.0) 13596 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:10:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:01 smithi121 bash[36902]: audit 2022-04-22T22:10:00.762192+0000 mon.a (mon.0) 13597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:01 smithi115 bash[45495]: audit 2022-04-22T22:10:00.762192+0000 mon.a (mon.0) 13597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:01 smithi115 bash[46934]: audit 2022-04-22T22:10:00.762192+0000 mon.a (mon.0) 13597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:02.740 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[45495]: cluster 2022-04-22T22:10:00.875279+0000 mgr.y (mgr.34107) 9685 : cluster [DBG] pgmap v10458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:02.741 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[45495]: audit 2022-04-22T22:10:01.940799+0000 mon.a (mon.0) 13598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:02.741 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[45495]: audit 2022-04-22T22:10:02.168362+0000 mon.a (mon.0) 13599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:02.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[46934]: cluster 2022-04-22T22:10:00.875279+0000 mgr.y (mgr.34107) 9685 : cluster [DBG] pgmap v10458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:02.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[46934]: audit 2022-04-22T22:10:01.940799+0000 mon.a (mon.0) 13598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:02.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[46934]: audit 2022-04-22T22:10:02.168362+0000 mon.a (mon.0) 13599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:02 smithi121 bash[36902]: cluster 2022-04-22T22:10:00.875279+0000 mgr.y (mgr.34107) 9685 : cluster [DBG] pgmap v10458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:02 smithi121 bash[36902]: audit 2022-04-22T22:10:01.940799+0000 mon.a (mon.0) 13598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:02 smithi121 bash[36902]: audit 2022-04-22T22:10:02.168362+0000 mon.a (mon.0) 13599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:02.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:03 smithi121 bash[36902]: cluster 2022-04-22T22:10:02.876384+0000 mgr.y (mgr.34107) 9686 : cluster [DBG] pgmap v10459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:03.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:03 smithi115 bash[46934]: cluster 2022-04-22T22:10:02.876384+0000 mgr.y (mgr.34107) 9686 : cluster [DBG] pgmap v10459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:03 smithi115 bash[45495]: cluster 2022-04-22T22:10:02.876384+0000 mgr.y (mgr.34107) 9686 : cluster [DBG] pgmap v10459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:04.591Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:05 smithi121 bash[36902]: cluster 2022-04-22T22:10:04.877305+0000 mgr.y (mgr.34107) 9687 : cluster [DBG] pgmap v10460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:05 smithi121 bash[36902]: audit 2022-04-22T22:10:05.765390+0000 mon.a (mon.0) 13600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:05 smithi115 bash[45495]: cluster 2022-04-22T22:10:04.877305+0000 mgr.y (mgr.34107) 9687 : cluster [DBG] pgmap v10460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:05 smithi115 bash[45495]: audit 2022-04-22T22:10:05.765390+0000 mon.a (mon.0) 13600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:05 smithi115 bash[46934]: cluster 2022-04-22T22:10:04.877305+0000 mgr.y (mgr.34107) 9687 : cluster [DBG] pgmap v10460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:05 smithi115 bash[46934]: audit 2022-04-22T22:10:05.765390+0000 mon.a (mon.0) 13600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:07 smithi121 bash[36902]: audit 2022-04-22T22:10:06.944359+0000 mon.a (mon.0) 13601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:07 smithi121 bash[36902]: audit 2022-04-22T22:10:07.171242+0000 mon.a (mon.0) 13602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:07 smithi115 bash[45495]: audit 2022-04-22T22:10:06.944359+0000 mon.a (mon.0) 13601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:07 smithi115 bash[45495]: audit 2022-04-22T22:10:07.171242+0000 mon.a (mon.0) 13602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:07 smithi115 bash[46934]: audit 2022-04-22T22:10:06.944359+0000 mon.a (mon.0) 13601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:07 smithi115 bash[46934]: audit 2022-04-22T22:10:07.171242+0000 mon.a (mon.0) 13602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:08 smithi121 bash[36902]: cluster 2022-04-22T22:10:06.878375+0000 mgr.y (mgr.34107) 9688 : cluster [DBG] pgmap v10461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:08 smithi115 bash[45495]: cluster 2022-04-22T22:10:06.878375+0000 mgr.y (mgr.34107) 9688 : cluster [DBG] pgmap v10461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:08 smithi115 bash[46934]: cluster 2022-04-22T22:10:06.878375+0000 mgr.y (mgr.34107) 9688 : cluster [DBG] pgmap v10461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:10 smithi121 bash[36902]: cluster 2022-04-22T22:10:08.879604+0000 mgr.y (mgr.34107) 9689 : cluster [DBG] pgmap v10462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:10 smithi115 bash[45495]: cluster 2022-04-22T22:10:08.879604+0000 mgr.y (mgr.34107) 9689 : cluster [DBG] pgmap v10462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:10.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:10 smithi115 bash[46934]: cluster 2022-04-22T22:10:08.879604+0000 mgr.y (mgr.34107) 9689 : cluster [DBG] pgmap v10462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:11 smithi121 bash[36902]: audit 2022-04-22T22:10:10.768481+0000 mon.a (mon.0) 13603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:11.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:11 smithi115 bash[46934]: audit 2022-04-22T22:10:10.768481+0000 mon.a (mon.0) 13603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:11 smithi115 bash[45495]: audit 2022-04-22T22:10:10.768481+0000 mon.a (mon.0) 13603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:12 smithi121 bash[36902]: cluster 2022-04-22T22:10:10.880586+0000 mgr.y (mgr.34107) 9690 : cluster [DBG] pgmap v10463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:12 smithi121 bash[36902]: audit 2022-04-22T22:10:11.948210+0000 mon.a (mon.0) 13604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:12 smithi121 bash[36902]: audit 2022-04-22T22:10:12.174313+0000 mon.a (mon.0) 13605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[45495]: cluster 2022-04-22T22:10:10.880586+0000 mgr.y (mgr.34107) 9690 : cluster [DBG] pgmap v10463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[45495]: audit 2022-04-22T22:10:11.948210+0000 mon.a (mon.0) 13604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[45495]: audit 2022-04-22T22:10:12.174313+0000 mon.a (mon.0) 13605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[46934]: cluster 2022-04-22T22:10:10.880586+0000 mgr.y (mgr.34107) 9690 : cluster [DBG] pgmap v10463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[46934]: audit 2022-04-22T22:10:11.948210+0000 mon.a (mon.0) 13604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[46934]: audit 2022-04-22T22:10:12.174313+0000 mon.a (mon.0) 13605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:12.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:14 smithi121 bash[36902]: cluster 2022-04-22T22:10:12.881612+0000 mgr.y (mgr.34107) 9691 : cluster [DBG] pgmap v10464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:14 smithi115 bash[45495]: cluster 2022-04-22T22:10:12.881612+0000 mgr.y (mgr.34107) 9691 : cluster [DBG] pgmap v10464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:14 smithi115 bash[46934]: cluster 2022-04-22T22:10:12.881612+0000 mgr.y (mgr.34107) 9691 : cluster [DBG] pgmap v10464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:14.590Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:16 smithi121 bash[36902]: cluster 2022-04-22T22:10:14.882678+0000 mgr.y (mgr.34107) 9692 : cluster [DBG] pgmap v10465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:16 smithi121 bash[36902]: audit 2022-04-22T22:10:15.771957+0000 mon.a (mon.0) 13606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:16 smithi115 bash[46934]: cluster 2022-04-22T22:10:14.882678+0000 mgr.y (mgr.34107) 9692 : cluster [DBG] pgmap v10465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:16 smithi115 bash[46934]: audit 2022-04-22T22:10:15.771957+0000 mon.a (mon.0) 13606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:16 smithi115 bash[45495]: cluster 2022-04-22T22:10:14.882678+0000 mgr.y (mgr.34107) 9692 : cluster [DBG] pgmap v10465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:16.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:16 smithi115 bash[45495]: audit 2022-04-22T22:10:15.771957+0000 mon.a (mon.0) 13606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:17 smithi121 bash[36902]: audit 2022-04-22T22:10:16.549502+0000 mon.a (mon.0) 13607 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:17 smithi121 bash[36902]: audit 2022-04-22T22:10:16.550158+0000 mon.a (mon.0) 13608 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:17 smithi121 bash[36902]: audit 2022-04-22T22:10:16.951816+0000 mon.a (mon.0) 13609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:17 smithi121 bash[36902]: audit 2022-04-22T22:10:17.177191+0000 mon.a (mon.0) 13610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[45495]: audit 2022-04-22T22:10:16.549502+0000 mon.a (mon.0) 13607 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:10:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[45495]: audit 2022-04-22T22:10:16.550158+0000 mon.a (mon.0) 13608 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[45495]: audit 2022-04-22T22:10:16.951816+0000 mon.a (mon.0) 13609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[45495]: audit 2022-04-22T22:10:17.177191+0000 mon.a (mon.0) 13610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[46934]: audit 2022-04-22T22:10:16.549502+0000 mon.a (mon.0) 13607 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[46934]: audit 2022-04-22T22:10:16.550158+0000 mon.a (mon.0) 13608 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[46934]: audit 2022-04-22T22:10:16.951816+0000 mon.a (mon.0) 13609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:17 smithi115 bash[46934]: audit 2022-04-22T22:10:17.177191+0000 mon.a (mon.0) 13610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:18 smithi121 bash[36902]: cluster 2022-04-22T22:10:16.883747+0000 mgr.y (mgr.34107) 9693 : cluster [DBG] pgmap v10466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:18 smithi115 bash[45495]: cluster 2022-04-22T22:10:16.883747+0000 mgr.y (mgr.34107) 9693 : cluster [DBG] pgmap v10466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:18.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:18 smithi115 bash[46934]: cluster 2022-04-22T22:10:16.883747+0000 mgr.y (mgr.34107) 9693 : cluster [DBG] pgmap v10466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:20 smithi121 bash[36902]: cluster 2022-04-22T22:10:18.884793+0000 mgr.y (mgr.34107) 9694 : cluster [DBG] pgmap v10467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:20 smithi115 bash[45495]: cluster 2022-04-22T22:10:18.884793+0000 mgr.y (mgr.34107) 9694 : cluster [DBG] pgmap v10467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:20 smithi115 bash[46934]: cluster 2022-04-22T22:10:18.884793+0000 mgr.y (mgr.34107) 9694 : cluster [DBG] pgmap v10467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:21 smithi121 bash[36902]: audit 2022-04-22T22:10:20.775231+0000 mon.a (mon.0) 13611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:21 smithi115 bash[45495]: audit 2022-04-22T22:10:20.775231+0000 mon.a (mon.0) 13611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:21 smithi115 bash[46934]: audit 2022-04-22T22:10:20.775231+0000 mon.a (mon.0) 13611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:22 smithi121 bash[36902]: cluster 2022-04-22T22:10:20.885787+0000 mgr.y (mgr.34107) 9695 : cluster [DBG] pgmap v10468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:22 smithi121 bash[36902]: audit 2022-04-22T22:10:21.955693+0000 mon.a (mon.0) 13612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:22 smithi121 bash[36902]: audit 2022-04-22T22:10:22.180700+0000 mon.a (mon.0) 13613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[45495]: cluster 2022-04-22T22:10:20.885787+0000 mgr.y (mgr.34107) 9695 : cluster [DBG] pgmap v10468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[45495]: audit 2022-04-22T22:10:21.955693+0000 mon.a (mon.0) 13612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[45495]: audit 2022-04-22T22:10:22.180700+0000 mon.a (mon.0) 13613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[46934]: cluster 2022-04-22T22:10:20.885787+0000 mgr.y (mgr.34107) 9695 : cluster [DBG] pgmap v10468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[46934]: audit 2022-04-22T22:10:21.955693+0000 mon.a (mon.0) 13612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[46934]: audit 2022-04-22T22:10:22.180700+0000 mon.a (mon.0) 13613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:22.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:24 smithi121 bash[36902]: cluster 2022-04-22T22:10:22.886957+0000 mgr.y (mgr.34107) 9696 : cluster [DBG] pgmap v10469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:24 smithi115 bash[45495]: cluster 2022-04-22T22:10:22.886957+0000 mgr.y (mgr.34107) 9696 : cluster [DBG] pgmap v10469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:24 smithi115 bash[46934]: cluster 2022-04-22T22:10:22.886957+0000 mgr.y (mgr.34107) 9696 : cluster [DBG] pgmap v10469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:24.969 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:10:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:24.592Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:10:25.313 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:10:25.314 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:10:25.315 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:10:25.315 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:10:25.718 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:10:25.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:10:25.721 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:10:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:26 smithi121 bash[36902]: cluster 2022-04-22T22:10:24.887958+0000 mgr.y (mgr.34107) 9697 : cluster [DBG] pgmap v10470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:26 smithi121 bash[36902]: audit 2022-04-22T22:10:24.950018+0000 mgr.y (mgr.34107) 9698 : audit [DBG] from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:26 smithi121 bash[36902]: audit 2022-04-22T22:10:25.292389+0000 mgr.y (mgr.34107) 9699 : audit [DBG] from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:26 smithi121 bash[36902]: audit 2022-04-22T22:10:25.715047+0000 mon.a (mon.0) 13614 : audit [DBG] from='client.? 172.21.15.115:0/2614452718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:26 smithi121 bash[36902]: audit 2022-04-22T22:10:25.778178+0000 mon.a (mon.0) 13615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[45495]: cluster 2022-04-22T22:10:24.887958+0000 mgr.y (mgr.34107) 9697 : cluster [DBG] pgmap v10470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[45495]: audit 2022-04-22T22:10:24.950018+0000 mgr.y (mgr.34107) 9698 : audit [DBG] from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[45495]: audit 2022-04-22T22:10:25.292389+0000 mgr.y (mgr.34107) 9699 : audit [DBG] from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[45495]: audit 2022-04-22T22:10:25.715047+0000 mon.a (mon.0) 13614 : audit [DBG] from='client.? 172.21.15.115:0/2614452718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[45495]: audit 2022-04-22T22:10:25.778178+0000 mon.a (mon.0) 13615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[46934]: cluster 2022-04-22T22:10:24.887958+0000 mgr.y (mgr.34107) 9697 : cluster [DBG] pgmap v10470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[46934]: audit 2022-04-22T22:10:24.950018+0000 mgr.y (mgr.34107) 9698 : audit [DBG] from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[46934]: audit 2022-04-22T22:10:25.292389+0000 mgr.y (mgr.34107) 9699 : audit [DBG] from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[46934]: audit 2022-04-22T22:10:25.715047+0000 mon.a (mon.0) 13614 : audit [DBG] from='client.? 172.21.15.115:0/2614452718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:26 smithi115 bash[46934]: audit 2022-04-22T22:10:25.778178+0000 mon.a (mon.0) 13615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:27 smithi121 bash[36902]: audit 2022-04-22T22:10:26.959170+0000 mon.a (mon.0) 13616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:27 smithi121 bash[36902]: audit 2022-04-22T22:10:27.183593+0000 mon.a (mon.0) 13617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:27 smithi115 bash[45495]: audit 2022-04-22T22:10:26.959170+0000 mon.a (mon.0) 13616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:27 smithi115 bash[45495]: audit 2022-04-22T22:10:27.183593+0000 mon.a (mon.0) 13617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:27 smithi115 bash[46934]: audit 2022-04-22T22:10:26.959170+0000 mon.a (mon.0) 13616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:27 smithi115 bash[46934]: audit 2022-04-22T22:10:27.183593+0000 mon.a (mon.0) 13617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:28 smithi121 bash[36902]: cluster 2022-04-22T22:10:26.888999+0000 mgr.y (mgr.34107) 9700 : cluster [DBG] pgmap v10471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:28 smithi115 bash[45495]: cluster 2022-04-22T22:10:26.888999+0000 mgr.y (mgr.34107) 9700 : cluster [DBG] pgmap v10471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:28 smithi115 bash[46934]: cluster 2022-04-22T22:10:26.888999+0000 mgr.y (mgr.34107) 9700 : cluster [DBG] pgmap v10471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:30 smithi121 bash[36902]: cluster 2022-04-22T22:10:28.890062+0000 mgr.y (mgr.34107) 9701 : cluster [DBG] pgmap v10472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:30 smithi115 bash[45495]: cluster 2022-04-22T22:10:28.890062+0000 mgr.y (mgr.34107) 9701 : cluster [DBG] pgmap v10472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:30 smithi115 bash[46934]: cluster 2022-04-22T22:10:28.890062+0000 mgr.y (mgr.34107) 9701 : cluster [DBG] pgmap v10472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:31 smithi115 bash[45495]: audit 2022-04-22T22:10:30.781211+0000 mon.a (mon.0) 13618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:31 smithi115 bash[46934]: audit 2022-04-22T22:10:30.781211+0000 mon.a (mon.0) 13618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:31 smithi121 bash[36902]: audit 2022-04-22T22:10:30.781211+0000 mon.a (mon.0) 13618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[45495]: cluster 2022-04-22T22:10:30.891067+0000 mgr.y (mgr.34107) 9702 : cluster [DBG] pgmap v10473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[45495]: audit 2022-04-22T22:10:31.962748+0000 mon.a (mon.0) 13619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[45495]: audit 2022-04-22T22:10:32.186725+0000 mon.a (mon.0) 13620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[46934]: cluster 2022-04-22T22:10:30.891067+0000 mgr.y (mgr.34107) 9702 : cluster [DBG] pgmap v10473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[46934]: audit 2022-04-22T22:10:31.962748+0000 mon.a (mon.0) 13619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[46934]: audit 2022-04-22T22:10:32.186725+0000 mon.a (mon.0) 13620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:32 smithi121 bash[36902]: cluster 2022-04-22T22:10:30.891067+0000 mgr.y (mgr.34107) 9702 : cluster [DBG] pgmap v10473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:32 smithi121 bash[36902]: audit 2022-04-22T22:10:31.962748+0000 mon.a (mon.0) 13619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:32 smithi121 bash[36902]: audit 2022-04-22T22:10:32.186725+0000 mon.a (mon.0) 13620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:32.738Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:34 smithi115 bash[45495]: cluster 2022-04-22T22:10:32.892171+0000 mgr.y (mgr.34107) 9703 : cluster [DBG] pgmap v10474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:34 smithi115 bash[46934]: cluster 2022-04-22T22:10:32.892171+0000 mgr.y (mgr.34107) 9703 : cluster [DBG] pgmap v10474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:34 smithi121 bash[36902]: cluster 2022-04-22T22:10:32.892171+0000 mgr.y (mgr.34107) 9703 : cluster [DBG] pgmap v10474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:34.592Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: cluster 2022-04-22T22:10:34.893272+0000 mgr.y (mgr.34107) 9704 : cluster [DBG] pgmap v10475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: audit 2022-04-22T22:10:35.784553+0000 mon.a (mon.0) 13621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: audit 2022-04-22T22:10:35.796122+0000 mon.a (mon.0) 13622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:10:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: audit 2022-04-22T22:10:35.798657+0000 mon.a (mon.0) 13623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:10:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: audit 2022-04-22T22:10:35.800318+0000 mon.a (mon.0) 13624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:10:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:36 smithi121 bash[36902]: audit 2022-04-22T22:10:36.129679+0000 mon.a (mon.0) 13625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: cluster 2022-04-22T22:10:34.893272+0000 mgr.y (mgr.34107) 9704 : cluster [DBG] pgmap v10475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: audit 2022-04-22T22:10:35.784553+0000 mon.a (mon.0) 13621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: audit 2022-04-22T22:10:35.796122+0000 mon.a (mon.0) 13622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: audit 2022-04-22T22:10:35.798657+0000 mon.a (mon.0) 13623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: audit 2022-04-22T22:10:35.800318+0000 mon.a (mon.0) 13624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:10:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[45495]: audit 2022-04-22T22:10:36.129679+0000 mon.a (mon.0) 13625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: cluster 2022-04-22T22:10:34.893272+0000 mgr.y (mgr.34107) 9704 : cluster [DBG] pgmap v10475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: audit 2022-04-22T22:10:35.784553+0000 mon.a (mon.0) 13621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: audit 2022-04-22T22:10:35.796122+0000 mon.a (mon.0) 13622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: audit 2022-04-22T22:10:35.798657+0000 mon.a (mon.0) 13623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: audit 2022-04-22T22:10:35.800318+0000 mon.a (mon.0) 13624 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:10:36.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:36 smithi115 bash[46934]: audit 2022-04-22T22:10:36.129679+0000 mon.a (mon.0) 13625 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:10:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:37 smithi121 bash[36902]: audit 2022-04-22T22:10:36.966422+0000 mon.a (mon.0) 13626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:37 smithi121 bash[36902]: audit 2022-04-22T22:10:37.189629+0000 mon.a (mon.0) 13627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:37.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:37 smithi115 bash[46934]: audit 2022-04-22T22:10:36.966422+0000 mon.a (mon.0) 13626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:37 smithi115 bash[46934]: audit 2022-04-22T22:10:37.189629+0000 mon.a (mon.0) 13627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:37 smithi115 bash[45495]: audit 2022-04-22T22:10:36.966422+0000 mon.a (mon.0) 13626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:37 smithi115 bash[45495]: audit 2022-04-22T22:10:37.189629+0000 mon.a (mon.0) 13627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:38 smithi121 bash[36902]: cluster 2022-04-22T22:10:36.894327+0000 mgr.y (mgr.34107) 9705 : cluster [DBG] pgmap v10476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:38 smithi115 bash[45495]: cluster 2022-04-22T22:10:36.894327+0000 mgr.y (mgr.34107) 9705 : cluster [DBG] pgmap v10476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:38.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:38 smithi115 bash[46934]: cluster 2022-04-22T22:10:36.894327+0000 mgr.y (mgr.34107) 9705 : cluster [DBG] pgmap v10476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:40 smithi121 bash[36902]: cluster 2022-04-22T22:10:38.895524+0000 mgr.y (mgr.34107) 9706 : cluster [DBG] pgmap v10477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:40 smithi115 bash[46934]: cluster 2022-04-22T22:10:38.895524+0000 mgr.y (mgr.34107) 9706 : cluster [DBG] pgmap v10477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:40 smithi115 bash[45495]: cluster 2022-04-22T22:10:38.895524+0000 mgr.y (mgr.34107) 9706 : cluster [DBG] pgmap v10477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:41 smithi121 bash[36902]: audit 2022-04-22T22:10:40.787721+0000 mon.a (mon.0) 13628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:41 smithi115 bash[45495]: audit 2022-04-22T22:10:40.787721+0000 mon.a (mon.0) 13628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:41 smithi115 bash[46934]: audit 2022-04-22T22:10:40.787721+0000 mon.a (mon.0) 13628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:42.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[45495]: cluster 2022-04-22T22:10:40.897169+0000 mgr.y (mgr.34107) 9707 : cluster [DBG] pgmap v10478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:42.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[45495]: audit 2022-04-22T22:10:41.970082+0000 mon.a (mon.0) 13629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:42.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[45495]: audit 2022-04-22T22:10:42.192495+0000 mon.a (mon.0) 13630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:42.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[46934]: cluster 2022-04-22T22:10:40.897169+0000 mgr.y (mgr.34107) 9707 : cluster [DBG] pgmap v10478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:42.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[46934]: audit 2022-04-22T22:10:41.970082+0000 mon.a (mon.0) 13629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:42.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[46934]: audit 2022-04-22T22:10:42.192495+0000 mon.a (mon.0) 13630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:42 smithi121 bash[36902]: cluster 2022-04-22T22:10:40.897169+0000 mgr.y (mgr.34107) 9707 : cluster [DBG] pgmap v10478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:42 smithi121 bash[36902]: audit 2022-04-22T22:10:41.970082+0000 mon.a (mon.0) 13629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:42 smithi121 bash[36902]: audit 2022-04-22T22:10:42.192495+0000 mon.a (mon.0) 13630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:42.740Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:44.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:44 smithi115 bash[45495]: cluster 2022-04-22T22:10:42.898256+0000 mgr.y (mgr.34107) 9708 : cluster [DBG] pgmap v10479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:44.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:44 smithi115 bash[46934]: cluster 2022-04-22T22:10:42.898256+0000 mgr.y (mgr.34107) 9708 : cluster [DBG] pgmap v10479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:44 smithi121 bash[36902]: cluster 2022-04-22T22:10:42.898256+0000 mgr.y (mgr.34107) 9708 : cluster [DBG] pgmap v10479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:44.592Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:46 smithi121 bash[36902]: cluster 2022-04-22T22:10:44.899271+0000 mgr.y (mgr.34107) 9709 : cluster [DBG] pgmap v10480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:46 smithi121 bash[36902]: audit 2022-04-22T22:10:45.790601+0000 mon.a (mon.0) 13631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:46 smithi115 bash[45495]: cluster 2022-04-22T22:10:44.899271+0000 mgr.y (mgr.34107) 9709 : cluster [DBG] pgmap v10480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:46 smithi115 bash[45495]: audit 2022-04-22T22:10:45.790601+0000 mon.a (mon.0) 13631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:46 smithi115 bash[46934]: cluster 2022-04-22T22:10:44.899271+0000 mgr.y (mgr.34107) 9709 : cluster [DBG] pgmap v10480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:46 smithi115 bash[46934]: audit 2022-04-22T22:10:45.790601+0000 mon.a (mon.0) 13631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:47 smithi121 bash[36902]: audit 2022-04-22T22:10:46.973772+0000 mon.a (mon.0) 13632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:47 smithi121 bash[36902]: audit 2022-04-22T22:10:47.195502+0000 mon.a (mon.0) 13633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:47 smithi115 bash[45495]: audit 2022-04-22T22:10:46.973772+0000 mon.a (mon.0) 13632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:47 smithi115 bash[45495]: audit 2022-04-22T22:10:47.195502+0000 mon.a (mon.0) 13633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:47 smithi115 bash[46934]: audit 2022-04-22T22:10:46.973772+0000 mon.a (mon.0) 13632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:47 smithi115 bash[46934]: audit 2022-04-22T22:10:47.195502+0000 mon.a (mon.0) 13633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:48 smithi121 bash[36902]: cluster 2022-04-22T22:10:46.900398+0000 mgr.y (mgr.34107) 9710 : cluster [DBG] pgmap v10481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:48 smithi115 bash[45495]: cluster 2022-04-22T22:10:46.900398+0000 mgr.y (mgr.34107) 9710 : cluster [DBG] pgmap v10481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:48 smithi115 bash[46934]: cluster 2022-04-22T22:10:46.900398+0000 mgr.y (mgr.34107) 9710 : cluster [DBG] pgmap v10481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:50 smithi121 bash[36902]: cluster 2022-04-22T22:10:48.901243+0000 mgr.y (mgr.34107) 9711 : cluster [DBG] pgmap v10482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:50 smithi115 bash[45495]: cluster 2022-04-22T22:10:48.901243+0000 mgr.y (mgr.34107) 9711 : cluster [DBG] pgmap v10482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:50 smithi115 bash[46934]: cluster 2022-04-22T22:10:48.901243+0000 mgr.y (mgr.34107) 9711 : cluster [DBG] pgmap v10482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:51 smithi121 bash[36902]: audit 2022-04-22T22:10:50.793360+0000 mon.a (mon.0) 13634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:51 smithi115 bash[45495]: audit 2022-04-22T22:10:50.793360+0000 mon.a (mon.0) 13634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:51 smithi115 bash[46934]: audit 2022-04-22T22:10:50.793360+0000 mon.a (mon.0) 13634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:52.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[45495]: cluster 2022-04-22T22:10:50.902306+0000 mgr.y (mgr.34107) 9712 : cluster [DBG] pgmap v10483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:52.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[45495]: audit 2022-04-22T22:10:51.977401+0000 mon.a (mon.0) 13635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:52.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[45495]: audit 2022-04-22T22:10:52.198457+0000 mon.a (mon.0) 13636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:52.755 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[46934]: cluster 2022-04-22T22:10:50.902306+0000 mgr.y (mgr.34107) 9712 : cluster [DBG] pgmap v10483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:52.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[46934]: audit 2022-04-22T22:10:51.977401+0000 mon.a (mon.0) 13635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:52.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[46934]: audit 2022-04-22T22:10:52.198457+0000 mon.a (mon.0) 13636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:52 smithi121 bash[36902]: cluster 2022-04-22T22:10:50.902306+0000 mgr.y (mgr.34107) 9712 : cluster [DBG] pgmap v10483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:52 smithi121 bash[36902]: audit 2022-04-22T22:10:51.977401+0000 mon.a (mon.0) 13635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:52 smithi121 bash[36902]: audit 2022-04-22T22:10:52.198457+0000 mon.a (mon.0) 13636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:52.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:54.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:54 smithi115 bash[45495]: cluster 2022-04-22T22:10:52.903459+0000 mgr.y (mgr.34107) 9713 : cluster [DBG] pgmap v10484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:54.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:54 smithi115 bash[46934]: cluster 2022-04-22T22:10:52.903459+0000 mgr.y (mgr.34107) 9713 : cluster [DBG] pgmap v10484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:54 smithi121 bash[36902]: cluster 2022-04-22T22:10:52.903459+0000 mgr.y (mgr.34107) 9713 : cluster [DBG] pgmap v10484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:10:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:10:54.593Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:10:56.103 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:10:56.445 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:10:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:10:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:10:56.448 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:10:56.448 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:10:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:56 smithi121 bash[36902]: cluster 2022-04-22T22:10:54.904686+0000 mgr.y (mgr.34107) 9714 : cluster [DBG] pgmap v10485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:56 smithi121 bash[36902]: audit 2022-04-22T22:10:55.796384+0000 mon.a (mon.0) 13637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:56 smithi115 bash[45495]: cluster 2022-04-22T22:10:54.904686+0000 mgr.y (mgr.34107) 9714 : cluster [DBG] pgmap v10485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:56 smithi115 bash[45495]: audit 2022-04-22T22:10:55.796384+0000 mon.a (mon.0) 13637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:56 smithi115 bash[46934]: cluster 2022-04-22T22:10:54.904686+0000 mgr.y (mgr.34107) 9714 : cluster [DBG] pgmap v10485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:56 smithi115 bash[46934]: audit 2022-04-22T22:10:55.796384+0000 mon.a (mon.0) 13637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:10:56.849 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:10:56.849 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:10:56.849 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:10:56.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:10:56.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:10:56.851 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:10:56.851 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:10:56.851 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:10:56.851 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:10:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:57 smithi121 bash[36902]: audit 2022-04-22T22:10:56.082694+0000 mgr.y (mgr.34107) 9715 : audit [DBG] from='client.55363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:57 smithi121 bash[36902]: audit 2022-04-22T22:10:56.426925+0000 mgr.y (mgr.34107) 9716 : audit [DBG] from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:57 smithi121 bash[36902]: audit 2022-04-22T22:10:56.845692+0000 mon.c (mon.1) 337 : audit [DBG] from='client.? 172.21.15.115:0/3543580956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:57 smithi121 bash[36902]: audit 2022-04-22T22:10:56.980968+0000 mon.a (mon.0) 13638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:57 smithi121 bash[36902]: audit 2022-04-22T22:10:57.201513+0000 mon.a (mon.0) 13639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[45495]: audit 2022-04-22T22:10:56.082694+0000 mgr.y (mgr.34107) 9715 : audit [DBG] from='client.55363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[45495]: audit 2022-04-22T22:10:56.426925+0000 mgr.y (mgr.34107) 9716 : audit [DBG] from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[45495]: audit 2022-04-22T22:10:56.845692+0000 mon.c (mon.1) 337 : audit [DBG] from='client.? 172.21.15.115:0/3543580956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[45495]: audit 2022-04-22T22:10:56.980968+0000 mon.a (mon.0) 13638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[45495]: audit 2022-04-22T22:10:57.201513+0000 mon.a (mon.0) 13639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[46934]: audit 2022-04-22T22:10:56.082694+0000 mgr.y (mgr.34107) 9715 : audit [DBG] from='client.55363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[46934]: audit 2022-04-22T22:10:56.426925+0000 mgr.y (mgr.34107) 9716 : audit [DBG] from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[46934]: audit 2022-04-22T22:10:56.845692+0000 mon.c (mon.1) 337 : audit [DBG] from='client.? 172.21.15.115:0/3543580956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:10:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[46934]: audit 2022-04-22T22:10:56.980968+0000 mon.a (mon.0) 13638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:10:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:57 smithi115 bash[46934]: audit 2022-04-22T22:10:57.201513+0000 mon.a (mon.0) 13639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:10:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:10:58 smithi121 bash[36902]: cluster 2022-04-22T22:10:56.905543+0000 mgr.y (mgr.34107) 9717 : cluster [DBG] pgmap v10486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:10:58 smithi115 bash[45495]: cluster 2022-04-22T22:10:56.905543+0000 mgr.y (mgr.34107) 9717 : cluster [DBG] pgmap v10486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:10:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:10:58 smithi115 bash[46934]: cluster 2022-04-22T22:10:56.905543+0000 mgr.y (mgr.34107) 9717 : cluster [DBG] pgmap v10486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:00 smithi121 bash[36902]: cluster 2022-04-22T22:10:58.906588+0000 mgr.y (mgr.34107) 9718 : cluster [DBG] pgmap v10487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:00 smithi115 bash[45495]: cluster 2022-04-22T22:10:58.906588+0000 mgr.y (mgr.34107) 9718 : cluster [DBG] pgmap v10487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:00.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:00 smithi115 bash[46934]: cluster 2022-04-22T22:10:58.906588+0000 mgr.y (mgr.34107) 9718 : cluster [DBG] pgmap v10487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:01 smithi121 bash[36902]: audit 2022-04-22T22:11:00.799424+0000 mon.a (mon.0) 13640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:01 smithi115 bash[45495]: audit 2022-04-22T22:11:00.799424+0000 mon.a (mon.0) 13640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:01 smithi115 bash[46934]: audit 2022-04-22T22:11:00.799424+0000 mon.a (mon.0) 13640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:02.743 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[46934]: cluster 2022-04-22T22:11:00.907690+0000 mgr.y (mgr.34107) 9719 : cluster [DBG] pgmap v10488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:02.743 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[46934]: audit 2022-04-22T22:11:01.984699+0000 mon.a (mon.0) 13641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:02.743 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[46934]: audit 2022-04-22T22:11:02.204327+0000 mon.a (mon.0) 13642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:02.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[45495]: cluster 2022-04-22T22:11:00.907690+0000 mgr.y (mgr.34107) 9719 : cluster [DBG] pgmap v10488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:02.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[45495]: audit 2022-04-22T22:11:01.984699+0000 mon.a (mon.0) 13641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:02.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[45495]: audit 2022-04-22T22:11:02.204327+0000 mon.a (mon.0) 13642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:02 smithi121 bash[36902]: cluster 2022-04-22T22:11:00.907690+0000 mgr.y (mgr.34107) 9719 : cluster [DBG] pgmap v10488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:02 smithi121 bash[36902]: audit 2022-04-22T22:11:01.984699+0000 mon.a (mon.0) 13641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:02 smithi121 bash[36902]: audit 2022-04-22T22:11:02.204327+0000 mon.a (mon.0) 13642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:02.740Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:04 smithi121 bash[36902]: cluster 2022-04-22T22:11:02.908655+0000 mgr.y (mgr.34107) 9720 : cluster [DBG] pgmap v10489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:04.594Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:04 smithi115 bash[45495]: cluster 2022-04-22T22:11:02.908655+0000 mgr.y (mgr.34107) 9720 : cluster [DBG] pgmap v10489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:04 smithi115 bash[46934]: cluster 2022-04-22T22:11:02.908655+0000 mgr.y (mgr.34107) 9720 : cluster [DBG] pgmap v10489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:06 smithi121 bash[36902]: cluster 2022-04-22T22:11:04.909561+0000 mgr.y (mgr.34107) 9721 : cluster [DBG] pgmap v10490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:06 smithi121 bash[36902]: audit 2022-04-22T22:11:05.802333+0000 mon.a (mon.0) 13643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:06 smithi115 bash[45495]: cluster 2022-04-22T22:11:04.909561+0000 mgr.y (mgr.34107) 9721 : cluster [DBG] pgmap v10490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:06 smithi115 bash[45495]: audit 2022-04-22T22:11:05.802333+0000 mon.a (mon.0) 13643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:06 smithi115 bash[46934]: cluster 2022-04-22T22:11:04.909561+0000 mgr.y (mgr.34107) 9721 : cluster [DBG] pgmap v10490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:06 smithi115 bash[46934]: audit 2022-04-22T22:11:05.802333+0000 mon.a (mon.0) 13643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:07 smithi121 bash[36902]: audit 2022-04-22T22:11:06.987741+0000 mon.a (mon.0) 13644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:07.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:07 smithi121 bash[36902]: audit 2022-04-22T22:11:07.207169+0000 mon.a (mon.0) 13645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:07 smithi115 bash[45495]: audit 2022-04-22T22:11:06.987741+0000 mon.a (mon.0) 13644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:07 smithi115 bash[45495]: audit 2022-04-22T22:11:07.207169+0000 mon.a (mon.0) 13645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:07 smithi115 bash[46934]: audit 2022-04-22T22:11:06.987741+0000 mon.a (mon.0) 13644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:07 smithi115 bash[46934]: audit 2022-04-22T22:11:07.207169+0000 mon.a (mon.0) 13645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:08 smithi121 bash[36902]: cluster 2022-04-22T22:11:06.910556+0000 mgr.y (mgr.34107) 9722 : cluster [DBG] pgmap v10491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:08.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:08 smithi115 bash[45495]: cluster 2022-04-22T22:11:06.910556+0000 mgr.y (mgr.34107) 9722 : cluster [DBG] pgmap v10491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:08.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:08 smithi115 bash[46934]: cluster 2022-04-22T22:11:06.910556+0000 mgr.y (mgr.34107) 9722 : cluster [DBG] pgmap v10491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:10 smithi121 bash[36902]: cluster 2022-04-22T22:11:08.911751+0000 mgr.y (mgr.34107) 9723 : cluster [DBG] pgmap v10492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:10 smithi115 bash[45495]: cluster 2022-04-22T22:11:08.911751+0000 mgr.y (mgr.34107) 9723 : cluster [DBG] pgmap v10492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:10 smithi115 bash[46934]: cluster 2022-04-22T22:11:08.911751+0000 mgr.y (mgr.34107) 9723 : cluster [DBG] pgmap v10492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:11 smithi121 bash[36902]: audit 2022-04-22T22:11:10.805666+0000 mon.a (mon.0) 13646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:11 smithi115 bash[45495]: audit 2022-04-22T22:11:10.805666+0000 mon.a (mon.0) 13646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:11 smithi115 bash[46934]: audit 2022-04-22T22:11:10.805666+0000 mon.a (mon.0) 13646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:12.744 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[45495]: cluster 2022-04-22T22:11:10.912852+0000 mgr.y (mgr.34107) 9724 : cluster [DBG] pgmap v10493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:12.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[45495]: audit 2022-04-22T22:11:11.990523+0000 mon.a (mon.0) 13647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:12.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[45495]: audit 2022-04-22T22:11:12.209992+0000 mon.a (mon.0) 13648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:12.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[46934]: cluster 2022-04-22T22:11:10.912852+0000 mgr.y (mgr.34107) 9724 : cluster [DBG] pgmap v10493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:12.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[46934]: audit 2022-04-22T22:11:11.990523+0000 mon.a (mon.0) 13647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:12.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[46934]: audit 2022-04-22T22:11:12.209992+0000 mon.a (mon.0) 13648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:12 smithi121 bash[36902]: cluster 2022-04-22T22:11:10.912852+0000 mgr.y (mgr.34107) 9724 : cluster [DBG] pgmap v10493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:12 smithi121 bash[36902]: audit 2022-04-22T22:11:11.990523+0000 mon.a (mon.0) 13647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:12 smithi121 bash[36902]: audit 2022-04-22T22:11:12.209992+0000 mon.a (mon.0) 13648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:12.741Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:14 smithi121 bash[36902]: cluster 2022-04-22T22:11:12.913985+0000 mgr.y (mgr.34107) 9725 : cluster [DBG] pgmap v10494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:14.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:14 smithi115 bash[45495]: cluster 2022-04-22T22:11:12.913985+0000 mgr.y (mgr.34107) 9725 : cluster [DBG] pgmap v10494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:14 smithi115 bash[46934]: cluster 2022-04-22T22:11:12.913985+0000 mgr.y (mgr.34107) 9725 : cluster [DBG] pgmap v10494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:14.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:14.595Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:16 smithi121 bash[36902]: cluster 2022-04-22T22:11:14.915078+0000 mgr.y (mgr.34107) 9726 : cluster [DBG] pgmap v10495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:16 smithi121 bash[36902]: audit 2022-04-22T22:11:15.808899+0000 mon.a (mon.0) 13649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:16 smithi115 bash[45495]: cluster 2022-04-22T22:11:14.915078+0000 mgr.y (mgr.34107) 9726 : cluster [DBG] pgmap v10495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:16 smithi115 bash[45495]: audit 2022-04-22T22:11:15.808899+0000 mon.a (mon.0) 13649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:16 smithi115 bash[46934]: cluster 2022-04-22T22:11:14.915078+0000 mgr.y (mgr.34107) 9726 : cluster [DBG] pgmap v10495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:16 smithi115 bash[46934]: audit 2022-04-22T22:11:15.808899+0000 mon.a (mon.0) 13649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:17 smithi121 bash[36902]: audit 2022-04-22T22:11:16.560421+0000 mon.a (mon.0) 13650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:11:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:17 smithi121 bash[36902]: audit 2022-04-22T22:11:16.560766+0000 mon.a (mon.0) 13651 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:11:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:17 smithi121 bash[36902]: audit 2022-04-22T22:11:16.994322+0000 mon.a (mon.0) 13652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:17 smithi121 bash[36902]: audit 2022-04-22T22:11:17.212802+0000 mon.a (mon.0) 13653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[46934]: audit 2022-04-22T22:11:16.560421+0000 mon.a (mon.0) 13650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[46934]: audit 2022-04-22T22:11:16.560766+0000 mon.a (mon.0) 13651 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[46934]: audit 2022-04-22T22:11:16.994322+0000 mon.a (mon.0) 13652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[46934]: audit 2022-04-22T22:11:17.212802+0000 mon.a (mon.0) 13653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[45495]: audit 2022-04-22T22:11:16.560421+0000 mon.a (mon.0) 13650 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:11:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[45495]: audit 2022-04-22T22:11:16.560766+0000 mon.a (mon.0) 13651 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:11:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[45495]: audit 2022-04-22T22:11:16.994322+0000 mon.a (mon.0) 13652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:17 smithi115 bash[45495]: audit 2022-04-22T22:11:17.212802+0000 mon.a (mon.0) 13653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:18 smithi121 bash[36902]: cluster 2022-04-22T22:11:16.916112+0000 mgr.y (mgr.34107) 9727 : cluster [DBG] pgmap v10496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:18 smithi115 bash[45495]: cluster 2022-04-22T22:11:16.916112+0000 mgr.y (mgr.34107) 9727 : cluster [DBG] pgmap v10496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:18 smithi115 bash[46934]: cluster 2022-04-22T22:11:16.916112+0000 mgr.y (mgr.34107) 9727 : cluster [DBG] pgmap v10496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:20 smithi121 bash[36902]: cluster 2022-04-22T22:11:18.917302+0000 mgr.y (mgr.34107) 9728 : cluster [DBG] pgmap v10497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:20.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:20 smithi115 bash[46934]: cluster 2022-04-22T22:11:18.917302+0000 mgr.y (mgr.34107) 9728 : cluster [DBG] pgmap v10497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:20 smithi115 bash[45495]: cluster 2022-04-22T22:11:18.917302+0000 mgr.y (mgr.34107) 9728 : cluster [DBG] pgmap v10497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:21 smithi121 bash[36902]: audit 2022-04-22T22:11:20.812063+0000 mon.a (mon.0) 13654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:21 smithi115 bash[45495]: audit 2022-04-22T22:11:20.812063+0000 mon.a (mon.0) 13654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:21 smithi115 bash[46934]: audit 2022-04-22T22:11:20.812063+0000 mon.a (mon.0) 13654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:22.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[46934]: cluster 2022-04-22T22:11:20.918389+0000 mgr.y (mgr.34107) 9729 : cluster [DBG] pgmap v10498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:22.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[46934]: audit 2022-04-22T22:11:21.998062+0000 mon.a (mon.0) 13655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:22.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[46934]: audit 2022-04-22T22:11:22.215777+0000 mon.a (mon.0) 13656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:22.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[45495]: cluster 2022-04-22T22:11:20.918389+0000 mgr.y (mgr.34107) 9729 : cluster [DBG] pgmap v10498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:22.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[45495]: audit 2022-04-22T22:11:21.998062+0000 mon.a (mon.0) 13655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:22.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[45495]: audit 2022-04-22T22:11:22.215777+0000 mon.a (mon.0) 13656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:22 smithi121 bash[36902]: cluster 2022-04-22T22:11:20.918389+0000 mgr.y (mgr.34107) 9729 : cluster [DBG] pgmap v10498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:22 smithi121 bash[36902]: audit 2022-04-22T22:11:21.998062+0000 mon.a (mon.0) 13655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:22 smithi121 bash[36902]: audit 2022-04-22T22:11:22.215777+0000 mon.a (mon.0) 13656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:22.741Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:24 smithi121 bash[36902]: cluster 2022-04-22T22:11:22.919497+0000 mgr.y (mgr.34107) 9730 : cluster [DBG] pgmap v10499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:24.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:24 smithi115 bash[45495]: cluster 2022-04-22T22:11:22.919497+0000 mgr.y (mgr.34107) 9730 : cluster [DBG] pgmap v10499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:24.594Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:24 smithi115 bash[46934]: cluster 2022-04-22T22:11:22.919497+0000 mgr.y (mgr.34107) 9730 : cluster [DBG] pgmap v10499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:26 smithi121 bash[36902]: cluster 2022-04-22T22:11:24.920549+0000 mgr.y (mgr.34107) 9731 : cluster [DBG] pgmap v10500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:26 smithi121 bash[36902]: audit 2022-04-22T22:11:25.815617+0000 mon.a (mon.0) 13657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:26 smithi115 bash[45495]: cluster 2022-04-22T22:11:24.920549+0000 mgr.y (mgr.34107) 9731 : cluster [DBG] pgmap v10500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:26 smithi115 bash[45495]: audit 2022-04-22T22:11:25.815617+0000 mon.a (mon.0) 13657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:26 smithi115 bash[46934]: cluster 2022-04-22T22:11:24.920549+0000 mgr.y (mgr.34107) 9731 : cluster [DBG] pgmap v10500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:26 smithi115 bash[46934]: audit 2022-04-22T22:11:25.815617+0000 mon.a (mon.0) 13657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:27.217 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:11:27.571 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:11:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:11:27.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:11:27.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:11:27.573 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:11:27.573 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:11:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:27 smithi121 bash[36902]: audit 2022-04-22T22:11:27.001643+0000 mon.a (mon.0) 13658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:27 smithi121 bash[36902]: audit 2022-04-22T22:11:27.218368+0000 mon.a (mon.0) 13659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:27 smithi115 bash[45495]: audit 2022-04-22T22:11:27.001643+0000 mon.a (mon.0) 13658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:27 smithi115 bash[45495]: audit 2022-04-22T22:11:27.218368+0000 mon.a (mon.0) 13659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:27 smithi115 bash[46934]: audit 2022-04-22T22:11:27.001643+0000 mon.a (mon.0) 13658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:27 smithi115 bash[46934]: audit 2022-04-22T22:11:27.218368+0000 mon.a (mon.0) 13659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:27.980 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:11:27.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:11:27.982 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:11:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:28 smithi121 bash[36902]: cluster 2022-04-22T22:11:26.921570+0000 mgr.y (mgr.34107) 9732 : cluster [DBG] pgmap v10501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:28 smithi121 bash[36902]: audit 2022-04-22T22:11:27.198943+0000 mgr.y (mgr.34107) 9733 : audit [DBG] from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:28 smithi121 bash[36902]: audit 2022-04-22T22:11:27.549033+0000 mgr.y (mgr.34107) 9734 : audit [DBG] from='client.45456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:28 smithi121 bash[36902]: audit 2022-04-22T22:11:27.977411+0000 mon.a (mon.0) 13660 : audit [DBG] from='client.? 172.21.15.115:0/3815730119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:11:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[45495]: cluster 2022-04-22T22:11:26.921570+0000 mgr.y (mgr.34107) 9732 : cluster [DBG] pgmap v10501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[45495]: audit 2022-04-22T22:11:27.198943+0000 mgr.y (mgr.34107) 9733 : audit [DBG] from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[45495]: audit 2022-04-22T22:11:27.549033+0000 mgr.y (mgr.34107) 9734 : audit [DBG] from='client.45456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[45495]: audit 2022-04-22T22:11:27.977411+0000 mon.a (mon.0) 13660 : audit [DBG] from='client.? 172.21.15.115:0/3815730119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[46934]: cluster 2022-04-22T22:11:26.921570+0000 mgr.y (mgr.34107) 9732 : cluster [DBG] pgmap v10501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[46934]: audit 2022-04-22T22:11:27.198943+0000 mgr.y (mgr.34107) 9733 : audit [DBG] from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[46934]: audit 2022-04-22T22:11:27.549033+0000 mgr.y (mgr.34107) 9734 : audit [DBG] from='client.45456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:28 smithi115 bash[46934]: audit 2022-04-22T22:11:27.977411+0000 mon.a (mon.0) 13660 : audit [DBG] from='client.? 172.21.15.115:0/3815730119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:11:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:29 smithi121 bash[36902]: cluster 2022-04-22T22:11:28.922678+0000 mgr.y (mgr.34107) 9735 : cluster [DBG] pgmap v10502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:29.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:29 smithi115 bash[45495]: cluster 2022-04-22T22:11:28.922678+0000 mgr.y (mgr.34107) 9735 : cluster [DBG] pgmap v10502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:29 smithi115 bash[46934]: cluster 2022-04-22T22:11:28.922678+0000 mgr.y (mgr.34107) 9735 : cluster [DBG] pgmap v10502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:30 smithi121 bash[36902]: audit 2022-04-22T22:11:30.818677+0000 mon.a (mon.0) 13661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:30 smithi115 bash[45495]: audit 2022-04-22T22:11:30.818677+0000 mon.a (mon.0) 13661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:30 smithi115 bash[46934]: audit 2022-04-22T22:11:30.818677+0000 mon.a (mon.0) 13661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:32 smithi121 bash[36902]: cluster 2022-04-22T22:11:30.923780+0000 mgr.y (mgr.34107) 9736 : cluster [DBG] pgmap v10503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:32.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:32 smithi121 bash[36902]: audit 2022-04-22T22:11:32.005081+0000 mon.a (mon.0) 13662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:32 smithi115 bash[45495]: cluster 2022-04-22T22:11:30.923780+0000 mgr.y (mgr.34107) 9736 : cluster [DBG] pgmap v10503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:32 smithi115 bash[45495]: audit 2022-04-22T22:11:32.005081+0000 mon.a (mon.0) 13662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:32 smithi115 bash[46934]: cluster 2022-04-22T22:11:30.923780+0000 mgr.y (mgr.34107) 9736 : cluster [DBG] pgmap v10503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:32 smithi115 bash[46934]: audit 2022-04-22T22:11:32.005081+0000 mon.a (mon.0) 13662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:32.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:33 smithi121 bash[36902]: audit 2022-04-22T22:11:32.220640+0000 mon.a (mon.0) 13663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:33 smithi115 bash[45495]: audit 2022-04-22T22:11:32.220640+0000 mon.a (mon.0) 13663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:33 smithi115 bash[46934]: audit 2022-04-22T22:11:32.220640+0000 mon.a (mon.0) 13663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:34 smithi121 bash[36902]: cluster 2022-04-22T22:11:32.924896+0000 mgr.y (mgr.34107) 9737 : cluster [DBG] pgmap v10504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:34 smithi115 bash[45495]: cluster 2022-04-22T22:11:32.924896+0000 mgr.y (mgr.34107) 9737 : cluster [DBG] pgmap v10504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:34 smithi115 bash[46934]: cluster 2022-04-22T22:11:32.924896+0000 mgr.y (mgr.34107) 9737 : cluster [DBG] pgmap v10504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:34.595Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:36.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[45495]: cluster 2022-04-22T22:11:34.925996+0000 mgr.y (mgr.34107) 9738 : cluster [DBG] pgmap v10505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:36.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[45495]: audit 2022-04-22T22:11:35.821815+0000 mon.a (mon.0) 13664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[45495]: audit 2022-04-22T22:11:36.137487+0000 mon.a (mon.0) 13665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[45495]: audit 2022-04-22T22:11:36.140327+0000 mon.a (mon.0) 13666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[45495]: audit 2022-04-22T22:11:36.141748+0000 mon.a (mon.0) 13667 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[46934]: cluster 2022-04-22T22:11:34.925996+0000 mgr.y (mgr.34107) 9738 : cluster [DBG] pgmap v10505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[46934]: audit 2022-04-22T22:11:35.821815+0000 mon.a (mon.0) 13664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[46934]: audit 2022-04-22T22:11:36.137487+0000 mon.a (mon.0) 13665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:11:36.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[46934]: audit 2022-04-22T22:11:36.140327+0000 mon.a (mon.0) 13666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:11:36.473 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:36 smithi115 bash[46934]: audit 2022-04-22T22:11:36.141748+0000 mon.a (mon.0) 13667 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:11:36.493 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:36 smithi121 bash[36902]: cluster 2022-04-22T22:11:34.925996+0000 mgr.y (mgr.34107) 9738 : cluster [DBG] pgmap v10505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:36.493 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:36 smithi121 bash[36902]: audit 2022-04-22T22:11:35.821815+0000 mon.a (mon.0) 13664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:36.493 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:36 smithi121 bash[36902]: audit 2022-04-22T22:11:36.137487+0000 mon.a (mon.0) 13665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:11:36.493 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:36 smithi121 bash[36902]: audit 2022-04-22T22:11:36.140327+0000 mon.a (mon.0) 13666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:11:36.493 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:36 smithi121 bash[36902]: audit 2022-04-22T22:11:36.141748+0000 mon.a (mon.0) 13667 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:11:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:37 smithi121 bash[36902]: audit 2022-04-22T22:11:36.501835+0000 mon.a (mon.0) 13668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:11:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:37 smithi121 bash[36902]: cluster 2022-04-22T22:11:36.926958+0000 mgr.y (mgr.34107) 9739 : cluster [DBG] pgmap v10506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:37 smithi121 bash[36902]: audit 2022-04-22T22:11:37.008780+0000 mon.a (mon.0) 13669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:37 smithi121 bash[36902]: audit 2022-04-22T22:11:37.223034+0000 mon.a (mon.0) 13670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[45495]: audit 2022-04-22T22:11:36.501835+0000 mon.a (mon.0) 13668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[45495]: cluster 2022-04-22T22:11:36.926958+0000 mgr.y (mgr.34107) 9739 : cluster [DBG] pgmap v10506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[45495]: audit 2022-04-22T22:11:37.008780+0000 mon.a (mon.0) 13669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[45495]: audit 2022-04-22T22:11:37.223034+0000 mon.a (mon.0) 13670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[46934]: audit 2022-04-22T22:11:36.501835+0000 mon.a (mon.0) 13668 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[46934]: cluster 2022-04-22T22:11:36.926958+0000 mgr.y (mgr.34107) 9739 : cluster [DBG] pgmap v10506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[46934]: audit 2022-04-22T22:11:37.008780+0000 mon.a (mon.0) 13669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:37 smithi115 bash[46934]: audit 2022-04-22T22:11:37.223034+0000 mon.a (mon.0) 13670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:39 smithi121 bash[36902]: cluster 2022-04-22T22:11:38.928077+0000 mgr.y (mgr.34107) 9740 : cluster [DBG] pgmap v10507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:39 smithi115 bash[45495]: cluster 2022-04-22T22:11:38.928077+0000 mgr.y (mgr.34107) 9740 : cluster [DBG] pgmap v10507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:39 smithi115 bash[46934]: cluster 2022-04-22T22:11:38.928077+0000 mgr.y (mgr.34107) 9740 : cluster [DBG] pgmap v10507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:40 smithi121 bash[36902]: audit 2022-04-22T22:11:40.825220+0000 mon.a (mon.0) 13671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:40 smithi115 bash[45495]: audit 2022-04-22T22:11:40.825220+0000 mon.a (mon.0) 13671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:40 smithi115 bash[46934]: audit 2022-04-22T22:11:40.825220+0000 mon.a (mon.0) 13671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:42 smithi121 bash[36902]: cluster 2022-04-22T22:11:40.929096+0000 mgr.y (mgr.34107) 9741 : cluster [DBG] pgmap v10508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:42 smithi121 bash[36902]: audit 2022-04-22T22:11:42.012346+0000 mon.a (mon.0) 13672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:42 smithi115 bash[45495]: cluster 2022-04-22T22:11:40.929096+0000 mgr.y (mgr.34107) 9741 : cluster [DBG] pgmap v10508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:42.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:42 smithi115 bash[45495]: audit 2022-04-22T22:11:42.012346+0000 mon.a (mon.0) 13672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:42.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:42 smithi115 bash[46934]: cluster 2022-04-22T22:11:40.929096+0000 mgr.y (mgr.34107) 9741 : cluster [DBG] pgmap v10508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:42.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:42 smithi115 bash[46934]: audit 2022-04-22T22:11:42.012346+0000 mon.a (mon.0) 13672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:42.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:43 smithi121 bash[36902]: audit 2022-04-22T22:11:42.225944+0000 mon.a (mon.0) 13673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:43.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:43 smithi115 bash[46934]: audit 2022-04-22T22:11:42.225944+0000 mon.a (mon.0) 13673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:43 smithi115 bash[45495]: audit 2022-04-22T22:11:42.225944+0000 mon.a (mon.0) 13673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:44 smithi121 bash[36902]: cluster 2022-04-22T22:11:42.930221+0000 mgr.y (mgr.34107) 9742 : cluster [DBG] pgmap v10509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:44 smithi115 bash[45495]: cluster 2022-04-22T22:11:42.930221+0000 mgr.y (mgr.34107) 9742 : cluster [DBG] pgmap v10509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:44 smithi115 bash[46934]: cluster 2022-04-22T22:11:42.930221+0000 mgr.y (mgr.34107) 9742 : cluster [DBG] pgmap v10509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:44.596Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:46 smithi121 bash[36902]: cluster 2022-04-22T22:11:44.931342+0000 mgr.y (mgr.34107) 9743 : cluster [DBG] pgmap v10510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:46 smithi121 bash[36902]: audit 2022-04-22T22:11:45.828337+0000 mon.a (mon.0) 13674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:46 smithi115 bash[45495]: cluster 2022-04-22T22:11:44.931342+0000 mgr.y (mgr.34107) 9743 : cluster [DBG] pgmap v10510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:46 smithi115 bash[45495]: audit 2022-04-22T22:11:45.828337+0000 mon.a (mon.0) 13674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:46 smithi115 bash[46934]: cluster 2022-04-22T22:11:44.931342+0000 mgr.y (mgr.34107) 9743 : cluster [DBG] pgmap v10510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:46 smithi115 bash[46934]: audit 2022-04-22T22:11:45.828337+0000 mon.a (mon.0) 13674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:47 smithi121 bash[36902]: audit 2022-04-22T22:11:47.015800+0000 mon.a (mon.0) 13675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:47 smithi115 bash[45495]: audit 2022-04-22T22:11:47.015800+0000 mon.a (mon.0) 13675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:47 smithi115 bash[46934]: audit 2022-04-22T22:11:47.015800+0000 mon.a (mon.0) 13675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:48 smithi121 bash[36902]: cluster 2022-04-22T22:11:46.932368+0000 mgr.y (mgr.34107) 9744 : cluster [DBG] pgmap v10511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:48 smithi121 bash[36902]: audit 2022-04-22T22:11:47.229758+0000 mon.a (mon.0) 13676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:48 smithi115 bash[46934]: cluster 2022-04-22T22:11:46.932368+0000 mgr.y (mgr.34107) 9744 : cluster [DBG] pgmap v10511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:48.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:48 smithi115 bash[46934]: audit 2022-04-22T22:11:47.229758+0000 mon.a (mon.0) 13676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:48.630 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:48 smithi115 bash[45495]: cluster 2022-04-22T22:11:46.932368+0000 mgr.y (mgr.34107) 9744 : cluster [DBG] pgmap v10511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:48.630 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:48 smithi115 bash[45495]: audit 2022-04-22T22:11:47.229758+0000 mon.a (mon.0) 13676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:50.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:50 smithi121 bash[36902]: cluster 2022-04-22T22:11:48.933479+0000 mgr.y (mgr.34107) 9745 : cluster [DBG] pgmap v10512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:50.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:50 smithi115 bash[45495]: cluster 2022-04-22T22:11:48.933479+0000 mgr.y (mgr.34107) 9745 : cluster [DBG] pgmap v10512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:50 smithi115 bash[46934]: cluster 2022-04-22T22:11:48.933479+0000 mgr.y (mgr.34107) 9745 : cluster [DBG] pgmap v10512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:51 smithi121 bash[36902]: audit 2022-04-22T22:11:50.831421+0000 mon.a (mon.0) 13677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:51.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:51 smithi115 bash[46934]: audit 2022-04-22T22:11:50.831421+0000 mon.a (mon.0) 13677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:51 smithi115 bash[45495]: audit 2022-04-22T22:11:50.831421+0000 mon.a (mon.0) 13677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:52 smithi121 bash[36902]: cluster 2022-04-22T22:11:50.934550+0000 mgr.y (mgr.34107) 9746 : cluster [DBG] pgmap v10513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:52 smithi121 bash[36902]: audit 2022-04-22T22:11:52.019321+0000 mon.a (mon.0) 13678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:52 smithi121 bash[36902]: audit 2022-04-22T22:11:52.231917+0000 mon.a (mon.0) 13679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:52.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[46934]: cluster 2022-04-22T22:11:50.934550+0000 mgr.y (mgr.34107) 9746 : cluster [DBG] pgmap v10513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[46934]: audit 2022-04-22T22:11:52.019321+0000 mon.a (mon.0) 13678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[46934]: audit 2022-04-22T22:11:52.231917+0000 mon.a (mon.0) 13679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[45495]: cluster 2022-04-22T22:11:50.934550+0000 mgr.y (mgr.34107) 9746 : cluster [DBG] pgmap v10513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[45495]: audit 2022-04-22T22:11:52.019321+0000 mon.a (mon.0) 13678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:52.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[45495]: audit 2022-04-22T22:11:52.231917+0000 mon.a (mon.0) 13679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:52.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:54 smithi121 bash[36902]: cluster 2022-04-22T22:11:52.935665+0000 mgr.y (mgr.34107) 9747 : cluster [DBG] pgmap v10514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:54.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:54 smithi115 bash[46934]: cluster 2022-04-22T22:11:52.935665+0000 mgr.y (mgr.34107) 9747 : cluster [DBG] pgmap v10514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:54 smithi115 bash[45495]: cluster 2022-04-22T22:11:52.935665+0000 mgr.y (mgr.34107) 9747 : cluster [DBG] pgmap v10514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:11:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:11:54.596Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:11:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:56 smithi121 bash[36902]: cluster 2022-04-22T22:11:54.936755+0000 mgr.y (mgr.34107) 9748 : cluster [DBG] pgmap v10515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:56 smithi121 bash[36902]: audit 2022-04-22T22:11:55.834316+0000 mon.a (mon.0) 13680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:56 smithi115 bash[45495]: cluster 2022-04-22T22:11:54.936755+0000 mgr.y (mgr.34107) 9748 : cluster [DBG] pgmap v10515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:56 smithi115 bash[45495]: audit 2022-04-22T22:11:55.834316+0000 mon.a (mon.0) 13680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:56 smithi115 bash[46934]: cluster 2022-04-22T22:11:54.936755+0000 mgr.y (mgr.34107) 9748 : cluster [DBG] pgmap v10515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:56 smithi115 bash[46934]: audit 2022-04-22T22:11:55.834316+0000 mon.a (mon.0) 13680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:11:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:57 smithi121 bash[36902]: audit 2022-04-22T22:11:57.022737+0000 mon.a (mon.0) 13681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:57 smithi121 bash[36902]: audit 2022-04-22T22:11:57.235025+0000 mon.a (mon.0) 13682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:57 smithi115 bash[45495]: audit 2022-04-22T22:11:57.022737+0000 mon.a (mon.0) 13681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:57 smithi115 bash[45495]: audit 2022-04-22T22:11:57.235025+0000 mon.a (mon.0) 13682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:57 smithi115 bash[46934]: audit 2022-04-22T22:11:57.022737+0000 mon.a (mon.0) 13681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:11:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:57 smithi115 bash[46934]: audit 2022-04-22T22:11:57.235025+0000 mon.a (mon.0) 13682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:11:58.359 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:11:58.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:58 smithi115 bash[46934]: cluster 2022-04-22T22:11:56.937817+0000 mgr.y (mgr.34107) 9749 : cluster [DBG] pgmap v10516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:58 smithi115 bash[45495]: cluster 2022-04-22T22:11:56.937817+0000 mgr.y (mgr.34107) 9749 : cluster [DBG] pgmap v10516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:11:58.708 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:11:58.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:11:58.710 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:11:58.710 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:11:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:58 smithi121 bash[36902]: cluster 2022-04-22T22:11:56.937817+0000 mgr.y (mgr.34107) 9749 : cluster [DBG] pgmap v10516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:11:59.116 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:11:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:11:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:59 smithi115 bash[45495]: audit 2022-04-22T22:11:58.336580+0000 mgr.y (mgr.34107) 9750 : audit [DBG] from='client.45468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:11:59 smithi115 bash[45495]: audit 2022-04-22T22:11:59.112263+0000 mon.c (mon.1) 338 : audit [DBG] from='client.? 172.21.15.115:0/76065452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:11:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:59 smithi115 bash[46934]: audit 2022-04-22T22:11:58.336580+0000 mgr.y (mgr.34107) 9750 : audit [DBG] from='client.45468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:11:59 smithi115 bash[46934]: audit 2022-04-22T22:11:59.112263+0000 mon.c (mon.1) 338 : audit [DBG] from='client.? 172.21.15.115:0/76065452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:11:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:59 smithi121 bash[36902]: audit 2022-04-22T22:11:58.336580+0000 mgr.y (mgr.34107) 9750 : audit [DBG] from='client.45468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:11:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:11:59 smithi121 bash[36902]: audit 2022-04-22T22:11:59.112263+0000 mon.c (mon.1) 338 : audit [DBG] from='client.? 172.21.15.115:0/76065452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:12:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:00 smithi115 bash[45495]: audit 2022-04-22T22:11:58.685310+0000 mgr.y (mgr.34107) 9751 : audit [DBG] from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:00 smithi115 bash[45495]: cluster 2022-04-22T22:11:58.939041+0000 mgr.y (mgr.34107) 9752 : cluster [DBG] pgmap v10517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:00 smithi115 bash[46934]: audit 2022-04-22T22:11:58.685310+0000 mgr.y (mgr.34107) 9751 : audit [DBG] from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:00 smithi115 bash[46934]: cluster 2022-04-22T22:11:58.939041+0000 mgr.y (mgr.34107) 9752 : cluster [DBG] pgmap v10517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:00 smithi121 bash[36902]: audit 2022-04-22T22:11:58.685310+0000 mgr.y (mgr.34107) 9751 : audit [DBG] from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:00 smithi121 bash[36902]: cluster 2022-04-22T22:11:58.939041+0000 mgr.y (mgr.34107) 9752 : cluster [DBG] pgmap v10517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:01 smithi121 bash[36902]: audit 2022-04-22T22:12:00.837563+0000 mon.a (mon.0) 13683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:01 smithi115 bash[45495]: audit 2022-04-22T22:12:00.837563+0000 mon.a (mon.0) 13683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:01 smithi115 bash[46934]: audit 2022-04-22T22:12:00.837563+0000 mon.a (mon.0) 13683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:02.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[45495]: cluster 2022-04-22T22:12:00.940165+0000 mgr.y (mgr.34107) 9753 : cluster [DBG] pgmap v10518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:02.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[45495]: audit 2022-04-22T22:12:02.026352+0000 mon.a (mon.0) 13684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:02.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[45495]: audit 2022-04-22T22:12:02.238106+0000 mon.a (mon.0) 13685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:02.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[46934]: cluster 2022-04-22T22:12:00.940165+0000 mgr.y (mgr.34107) 9753 : cluster [DBG] pgmap v10518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:02.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[46934]: audit 2022-04-22T22:12:02.026352+0000 mon.a (mon.0) 13684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:02.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[46934]: audit 2022-04-22T22:12:02.238106+0000 mon.a (mon.0) 13685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:02 smithi121 bash[36902]: cluster 2022-04-22T22:12:00.940165+0000 mgr.y (mgr.34107) 9753 : cluster [DBG] pgmap v10518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:02 smithi121 bash[36902]: audit 2022-04-22T22:12:02.026352+0000 mon.a (mon.0) 13684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:02 smithi121 bash[36902]: audit 2022-04-22T22:12:02.238106+0000 mon.a (mon.0) 13685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:02.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:04.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:04 smithi115 bash[45495]: cluster 2022-04-22T22:12:02.941397+0000 mgr.y (mgr.34107) 9754 : cluster [DBG] pgmap v10519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:04.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:04 smithi115 bash[46934]: cluster 2022-04-22T22:12:02.941397+0000 mgr.y (mgr.34107) 9754 : cluster [DBG] pgmap v10519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:04 smithi121 bash[36902]: cluster 2022-04-22T22:12:02.941397+0000 mgr.y (mgr.34107) 9754 : cluster [DBG] pgmap v10519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:04.597Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:06 smithi121 bash[36902]: cluster 2022-04-22T22:12:04.942515+0000 mgr.y (mgr.34107) 9755 : cluster [DBG] pgmap v10520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:06 smithi121 bash[36902]: audit 2022-04-22T22:12:05.840804+0000 mon.a (mon.0) 13686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:06 smithi115 bash[45495]: cluster 2022-04-22T22:12:04.942515+0000 mgr.y (mgr.34107) 9755 : cluster [DBG] pgmap v10520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:06 smithi115 bash[45495]: audit 2022-04-22T22:12:05.840804+0000 mon.a (mon.0) 13686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:06 smithi115 bash[46934]: cluster 2022-04-22T22:12:04.942515+0000 mgr.y (mgr.34107) 9755 : cluster [DBG] pgmap v10520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:06 smithi115 bash[46934]: audit 2022-04-22T22:12:05.840804+0000 mon.a (mon.0) 13686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:07 smithi121 bash[36902]: audit 2022-04-22T22:12:07.030213+0000 mon.a (mon.0) 13687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:07 smithi121 bash[36902]: audit 2022-04-22T22:12:07.241431+0000 mon.a (mon.0) 13688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:07.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:07 smithi115 bash[46934]: audit 2022-04-22T22:12:07.030213+0000 mon.a (mon.0) 13687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:07.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:07 smithi115 bash[46934]: audit 2022-04-22T22:12:07.241431+0000 mon.a (mon.0) 13688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:07 smithi115 bash[45495]: audit 2022-04-22T22:12:07.030213+0000 mon.a (mon.0) 13687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:07 smithi115 bash[45495]: audit 2022-04-22T22:12:07.241431+0000 mon.a (mon.0) 13688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:08 smithi121 bash[36902]: cluster 2022-04-22T22:12:06.943565+0000 mgr.y (mgr.34107) 9756 : cluster [DBG] pgmap v10521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:08.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:08 smithi115 bash[46934]: cluster 2022-04-22T22:12:06.943565+0000 mgr.y (mgr.34107) 9756 : cluster [DBG] pgmap v10521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:08 smithi115 bash[45495]: cluster 2022-04-22T22:12:06.943565+0000 mgr.y (mgr.34107) 9756 : cluster [DBG] pgmap v10521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:10 smithi121 bash[36902]: cluster 2022-04-22T22:12:08.944669+0000 mgr.y (mgr.34107) 9757 : cluster [DBG] pgmap v10522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:10.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:10 smithi115 bash[46934]: cluster 2022-04-22T22:12:08.944669+0000 mgr.y (mgr.34107) 9757 : cluster [DBG] pgmap v10522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:10 smithi115 bash[45495]: cluster 2022-04-22T22:12:08.944669+0000 mgr.y (mgr.34107) 9757 : cluster [DBG] pgmap v10522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:11 smithi121 bash[36902]: audit 2022-04-22T22:12:10.843645+0000 mon.a (mon.0) 13689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:11 smithi115 bash[46934]: audit 2022-04-22T22:12:10.843645+0000 mon.a (mon.0) 13689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:11 smithi115 bash[45495]: audit 2022-04-22T22:12:10.843645+0000 mon.a (mon.0) 13689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:12.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[46934]: cluster 2022-04-22T22:12:10.945749+0000 mgr.y (mgr.34107) 9758 : cluster [DBG] pgmap v10523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:12.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[46934]: audit 2022-04-22T22:12:12.034037+0000 mon.a (mon.0) 13690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:12.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[46934]: audit 2022-04-22T22:12:12.244264+0000 mon.a (mon.0) 13691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:12.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[45495]: cluster 2022-04-22T22:12:10.945749+0000 mgr.y (mgr.34107) 9758 : cluster [DBG] pgmap v10523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:12.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[45495]: audit 2022-04-22T22:12:12.034037+0000 mon.a (mon.0) 13690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:12.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[45495]: audit 2022-04-22T22:12:12.244264+0000 mon.a (mon.0) 13691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:12 smithi121 bash[36902]: cluster 2022-04-22T22:12:10.945749+0000 mgr.y (mgr.34107) 9758 : cluster [DBG] pgmap v10523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:12.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:12 smithi121 bash[36902]: audit 2022-04-22T22:12:12.034037+0000 mon.a (mon.0) 13690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:12.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:12 smithi121 bash[36902]: audit 2022-04-22T22:12:12.244264+0000 mon.a (mon.0) 13691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:12.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:14.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:14 smithi115 bash[45495]: cluster 2022-04-22T22:12:12.946900+0000 mgr.y (mgr.34107) 9759 : cluster [DBG] pgmap v10524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:14.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:14 smithi115 bash[46934]: cluster 2022-04-22T22:12:12.946900+0000 mgr.y (mgr.34107) 9759 : cluster [DBG] pgmap v10524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:14 smithi121 bash[36902]: cluster 2022-04-22T22:12:12.946900+0000 mgr.y (mgr.34107) 9759 : cluster [DBG] pgmap v10524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:14.598Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:16 smithi121 bash[36902]: cluster 2022-04-22T22:12:14.947977+0000 mgr.y (mgr.34107) 9760 : cluster [DBG] pgmap v10525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:16 smithi121 bash[36902]: audit 2022-04-22T22:12:15.846706+0000 mon.a (mon.0) 13692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:16 smithi115 bash[45495]: cluster 2022-04-22T22:12:14.947977+0000 mgr.y (mgr.34107) 9760 : cluster [DBG] pgmap v10525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:16 smithi115 bash[45495]: audit 2022-04-22T22:12:15.846706+0000 mon.a (mon.0) 13692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:16 smithi115 bash[46934]: cluster 2022-04-22T22:12:14.947977+0000 mgr.y (mgr.34107) 9760 : cluster [DBG] pgmap v10525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:16 smithi115 bash[46934]: audit 2022-04-22T22:12:15.846706+0000 mon.a (mon.0) 13692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:17 smithi121 bash[36902]: audit 2022-04-22T22:12:16.571710+0000 mon.a (mon.0) 13693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:12:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:17 smithi121 bash[36902]: audit 2022-04-22T22:12:16.571908+0000 mon.a (mon.0) 13694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:12:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:17 smithi121 bash[36902]: audit 2022-04-22T22:12:17.037710+0000 mon.a (mon.0) 13695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:17 smithi121 bash[36902]: audit 2022-04-22T22:12:17.247263+0000 mon.a (mon.0) 13696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[45495]: audit 2022-04-22T22:12:16.571710+0000 mon.a (mon.0) 13693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[45495]: audit 2022-04-22T22:12:16.571908+0000 mon.a (mon.0) 13694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[45495]: audit 2022-04-22T22:12:17.037710+0000 mon.a (mon.0) 13695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[45495]: audit 2022-04-22T22:12:17.247263+0000 mon.a (mon.0) 13696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[46934]: audit 2022-04-22T22:12:16.571710+0000 mon.a (mon.0) 13693 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:12:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[46934]: audit 2022-04-22T22:12:16.571908+0000 mon.a (mon.0) 13694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:12:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[46934]: audit 2022-04-22T22:12:17.037710+0000 mon.a (mon.0) 13695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:17 smithi115 bash[46934]: audit 2022-04-22T22:12:17.247263+0000 mon.a (mon.0) 13696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:18 smithi121 bash[36902]: cluster 2022-04-22T22:12:16.949046+0000 mgr.y (mgr.34107) 9761 : cluster [DBG] pgmap v10526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:18 smithi115 bash[45495]: cluster 2022-04-22T22:12:16.949046+0000 mgr.y (mgr.34107) 9761 : cluster [DBG] pgmap v10526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:18 smithi115 bash[46934]: cluster 2022-04-22T22:12:16.949046+0000 mgr.y (mgr.34107) 9761 : cluster [DBG] pgmap v10526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:20 smithi121 bash[36902]: cluster 2022-04-22T22:12:18.950305+0000 mgr.y (mgr.34107) 9762 : cluster [DBG] pgmap v10527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:20 smithi115 bash[45495]: cluster 2022-04-22T22:12:18.950305+0000 mgr.y (mgr.34107) 9762 : cluster [DBG] pgmap v10527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:20 smithi115 bash[46934]: cluster 2022-04-22T22:12:18.950305+0000 mgr.y (mgr.34107) 9762 : cluster [DBG] pgmap v10527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:21 smithi121 bash[36902]: audit 2022-04-22T22:12:20.849736+0000 mon.a (mon.0) 13697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:21 smithi115 bash[45495]: audit 2022-04-22T22:12:20.849736+0000 mon.a (mon.0) 13697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:21 smithi115 bash[46934]: audit 2022-04-22T22:12:20.849736+0000 mon.a (mon.0) 13697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[45495]: cluster 2022-04-22T22:12:20.951404+0000 mgr.y (mgr.34107) 9763 : cluster [DBG] pgmap v10528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[45495]: audit 2022-04-22T22:12:22.041448+0000 mon.a (mon.0) 13698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[45495]: audit 2022-04-22T22:12:22.250422+0000 mon.a (mon.0) 13699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[46934]: cluster 2022-04-22T22:12:20.951404+0000 mgr.y (mgr.34107) 9763 : cluster [DBG] pgmap v10528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[46934]: audit 2022-04-22T22:12:22.041448+0000 mon.a (mon.0) 13698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:22.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[46934]: audit 2022-04-22T22:12:22.250422+0000 mon.a (mon.0) 13699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:22 smithi121 bash[36902]: cluster 2022-04-22T22:12:20.951404+0000 mgr.y (mgr.34107) 9763 : cluster [DBG] pgmap v10528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:22 smithi121 bash[36902]: audit 2022-04-22T22:12:22.041448+0000 mon.a (mon.0) 13698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:22 smithi121 bash[36902]: audit 2022-04-22T22:12:22.250422+0000 mon.a (mon.0) 13699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:22.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:24 smithi121 bash[36902]: cluster 2022-04-22T22:12:22.952557+0000 mgr.y (mgr.34107) 9764 : cluster [DBG] pgmap v10529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:24 smithi115 bash[45495]: cluster 2022-04-22T22:12:22.952557+0000 mgr.y (mgr.34107) 9764 : cluster [DBG] pgmap v10529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:24.600Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:24 smithi115 bash[46934]: cluster 2022-04-22T22:12:22.952557+0000 mgr.y (mgr.34107) 9764 : cluster [DBG] pgmap v10529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:26 smithi121 bash[36902]: cluster 2022-04-22T22:12:24.953655+0000 mgr.y (mgr.34107) 9765 : cluster [DBG] pgmap v10530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:26 smithi121 bash[36902]: audit 2022-04-22T22:12:25.852676+0000 mon.a (mon.0) 13700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:26 smithi115 bash[45495]: cluster 2022-04-22T22:12:24.953655+0000 mgr.y (mgr.34107) 9765 : cluster [DBG] pgmap v10530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:26 smithi115 bash[45495]: audit 2022-04-22T22:12:25.852676+0000 mon.a (mon.0) 13700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:26 smithi115 bash[46934]: cluster 2022-04-22T22:12:24.953655+0000 mgr.y (mgr.34107) 9765 : cluster [DBG] pgmap v10530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:26 smithi115 bash[46934]: audit 2022-04-22T22:12:25.852676+0000 mon.a (mon.0) 13700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:27 smithi121 bash[36902]: audit 2022-04-22T22:12:27.045296+0000 mon.a (mon.0) 13701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:27 smithi121 bash[36902]: audit 2022-04-22T22:12:27.253296+0000 mon.a (mon.0) 13702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:27 smithi115 bash[45495]: audit 2022-04-22T22:12:27.045296+0000 mon.a (mon.0) 13701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:27 smithi115 bash[45495]: audit 2022-04-22T22:12:27.253296+0000 mon.a (mon.0) 13702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:27 smithi115 bash[46934]: audit 2022-04-22T22:12:27.045296+0000 mon.a (mon.0) 13701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:27 smithi115 bash[46934]: audit 2022-04-22T22:12:27.253296+0000 mon.a (mon.0) 13702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:28 smithi121 bash[36902]: cluster 2022-04-22T22:12:26.954722+0000 mgr.y (mgr.34107) 9766 : cluster [DBG] pgmap v10531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:28 smithi115 bash[45495]: cluster 2022-04-22T22:12:26.954722+0000 mgr.y (mgr.34107) 9766 : cluster [DBG] pgmap v10531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:28 smithi115 bash[46934]: cluster 2022-04-22T22:12:26.954722+0000 mgr.y (mgr.34107) 9766 : cluster [DBG] pgmap v10531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:12:29.831 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:12:29.831 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:12:29.831 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:12:29.831 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:12:29.832 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:12:29.833 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:12:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:12:30.236 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:12:30.237 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:12:30.238 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:12:30.238 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:12:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:30 smithi121 bash[36902]: cluster 2022-04-22T22:12:28.955861+0000 mgr.y (mgr.34107) 9767 : cluster [DBG] pgmap v10532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:30 smithi121 bash[36902]: audit 2022-04-22T22:12:29.471688+0000 mgr.y (mgr.34107) 9768 : audit [DBG] from='client.55417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:30 smithi121 bash[36902]: audit 2022-04-22T22:12:30.231809+0000 mon.a (mon.0) 13703 : audit [DBG] from='client.? 172.21.15.115:0/2640605886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:12:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[45495]: cluster 2022-04-22T22:12:28.955861+0000 mgr.y (mgr.34107) 9767 : cluster [DBG] pgmap v10532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[45495]: audit 2022-04-22T22:12:29.471688+0000 mgr.y (mgr.34107) 9768 : audit [DBG] from='client.55417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[45495]: audit 2022-04-22T22:12:30.231809+0000 mon.a (mon.0) 13703 : audit [DBG] from='client.? 172.21.15.115:0/2640605886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:12:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[46934]: cluster 2022-04-22T22:12:28.955861+0000 mgr.y (mgr.34107) 9767 : cluster [DBG] pgmap v10532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[46934]: audit 2022-04-22T22:12:29.471688+0000 mgr.y (mgr.34107) 9768 : audit [DBG] from='client.55417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:30 smithi115 bash[46934]: audit 2022-04-22T22:12:30.231809+0000 mon.a (mon.0) 13703 : audit [DBG] from='client.? 172.21.15.115:0/2640605886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:12:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:31 smithi121 bash[36902]: audit 2022-04-22T22:12:29.808776+0000 mgr.y (mgr.34107) 9769 : audit [DBG] from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:31 smithi121 bash[36902]: audit 2022-04-22T22:12:30.855913+0000 mon.a (mon.0) 13704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:31 smithi115 bash[45495]: audit 2022-04-22T22:12:29.808776+0000 mgr.y (mgr.34107) 9769 : audit [DBG] from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:31 smithi115 bash[45495]: audit 2022-04-22T22:12:30.855913+0000 mon.a (mon.0) 13704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:31 smithi115 bash[46934]: audit 2022-04-22T22:12:29.808776+0000 mgr.y (mgr.34107) 9769 : audit [DBG] from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:12:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:31 smithi115 bash[46934]: audit 2022-04-22T22:12:30.855913+0000 mon.a (mon.0) 13704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[45495]: cluster 2022-04-22T22:12:30.957078+0000 mgr.y (mgr.34107) 9770 : cluster [DBG] pgmap v10533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[45495]: audit 2022-04-22T22:12:32.048897+0000 mon.a (mon.0) 13705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[45495]: audit 2022-04-22T22:12:32.256348+0000 mon.a (mon.0) 13706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[46934]: cluster 2022-04-22T22:12:30.957078+0000 mgr.y (mgr.34107) 9770 : cluster [DBG] pgmap v10533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[46934]: audit 2022-04-22T22:12:32.048897+0000 mon.a (mon.0) 13705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:32.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[46934]: audit 2022-04-22T22:12:32.256348+0000 mon.a (mon.0) 13706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:32 smithi121 bash[36902]: cluster 2022-04-22T22:12:30.957078+0000 mgr.y (mgr.34107) 9770 : cluster [DBG] pgmap v10533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:32 smithi121 bash[36902]: audit 2022-04-22T22:12:32.048897+0000 mon.a (mon.0) 13705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:32 smithi121 bash[36902]: audit 2022-04-22T22:12:32.256348+0000 mon.a (mon.0) 13706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:32.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:34 smithi121 bash[36902]: cluster 2022-04-22T22:12:32.958251+0000 mgr.y (mgr.34107) 9771 : cluster [DBG] pgmap v10534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:34 smithi115 bash[45495]: cluster 2022-04-22T22:12:32.958251+0000 mgr.y (mgr.34107) 9771 : cluster [DBG] pgmap v10534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:34.599Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:34 smithi115 bash[46934]: cluster 2022-04-22T22:12:32.958251+0000 mgr.y (mgr.34107) 9771 : cluster [DBG] pgmap v10534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:36 smithi121 bash[36902]: cluster 2022-04-22T22:12:34.959370+0000 mgr.y (mgr.34107) 9772 : cluster [DBG] pgmap v10535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:36 smithi121 bash[36902]: audit 2022-04-22T22:12:35.858962+0000 mon.a (mon.0) 13707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:36 smithi115 bash[45495]: cluster 2022-04-22T22:12:34.959370+0000 mgr.y (mgr.34107) 9772 : cluster [DBG] pgmap v10535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:36 smithi115 bash[45495]: audit 2022-04-22T22:12:35.858962+0000 mon.a (mon.0) 13707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:36 smithi115 bash[46934]: cluster 2022-04-22T22:12:34.959370+0000 mgr.y (mgr.34107) 9772 : cluster [DBG] pgmap v10535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:36 smithi115 bash[46934]: audit 2022-04-22T22:12:35.858962+0000 mon.a (mon.0) 13707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:36.510191+0000 mon.a (mon.0) 13708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:12:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:36.513266+0000 mon.a (mon.0) 13709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:12:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:36.515056+0000 mon.a (mon.0) 13710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:12:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:36.862005+0000 mon.a (mon.0) 13711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:12:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:37.052512+0000 mon.a (mon.0) 13712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:37 smithi121 bash[36902]: audit 2022-04-22T22:12:37.259447+0000 mon.a (mon.0) 13713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:36.510191+0000 mon.a (mon.0) 13708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:36.513266+0000 mon.a (mon.0) 13709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:36.515056+0000 mon.a (mon.0) 13710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:36.862005+0000 mon.a (mon.0) 13711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:37.052512+0000 mon.a (mon.0) 13712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[45495]: audit 2022-04-22T22:12:37.259447+0000 mon.a (mon.0) 13713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:36.510191+0000 mon.a (mon.0) 13708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:36.513266+0000 mon.a (mon.0) 13709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:36.515056+0000 mon.a (mon.0) 13710 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:36.862005+0000 mon.a (mon.0) 13711 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:37.052512+0000 mon.a (mon.0) 13712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:37 smithi115 bash[46934]: audit 2022-04-22T22:12:37.259447+0000 mon.a (mon.0) 13713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:38 smithi121 bash[36902]: cluster 2022-04-22T22:12:36.960434+0000 mgr.y (mgr.34107) 9773 : cluster [DBG] pgmap v10536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:38 smithi115 bash[45495]: cluster 2022-04-22T22:12:36.960434+0000 mgr.y (mgr.34107) 9773 : cluster [DBG] pgmap v10536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:38 smithi115 bash[46934]: cluster 2022-04-22T22:12:36.960434+0000 mgr.y (mgr.34107) 9773 : cluster [DBG] pgmap v10536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:40 smithi121 bash[36902]: cluster 2022-04-22T22:12:38.961596+0000 mgr.y (mgr.34107) 9774 : cluster [DBG] pgmap v10537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:40 smithi115 bash[45495]: cluster 2022-04-22T22:12:38.961596+0000 mgr.y (mgr.34107) 9774 : cluster [DBG] pgmap v10537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:40 smithi115 bash[46934]: cluster 2022-04-22T22:12:38.961596+0000 mgr.y (mgr.34107) 9774 : cluster [DBG] pgmap v10537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:41 smithi121 bash[36902]: audit 2022-04-22T22:12:40.862199+0000 mon.a (mon.0) 13714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:41 smithi115 bash[46934]: audit 2022-04-22T22:12:40.862199+0000 mon.a (mon.0) 13714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:41 smithi115 bash[45495]: audit 2022-04-22T22:12:40.862199+0000 mon.a (mon.0) 13714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:42.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[45495]: cluster 2022-04-22T22:12:40.962636+0000 mgr.y (mgr.34107) 9775 : cluster [DBG] pgmap v10538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:42.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[45495]: audit 2022-04-22T22:12:42.056256+0000 mon.a (mon.0) 13715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:42.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[45495]: audit 2022-04-22T22:12:42.262458+0000 mon.a (mon.0) 13716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:42.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[46934]: cluster 2022-04-22T22:12:40.962636+0000 mgr.y (mgr.34107) 9775 : cluster [DBG] pgmap v10538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:42.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[46934]: audit 2022-04-22T22:12:42.056256+0000 mon.a (mon.0) 13715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:42.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[46934]: audit 2022-04-22T22:12:42.262458+0000 mon.a (mon.0) 13716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:42 smithi121 bash[36902]: cluster 2022-04-22T22:12:40.962636+0000 mgr.y (mgr.34107) 9775 : cluster [DBG] pgmap v10538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:42 smithi121 bash[36902]: audit 2022-04-22T22:12:42.056256+0000 mon.a (mon.0) 13715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:42 smithi121 bash[36902]: audit 2022-04-22T22:12:42.262458+0000 mon.a (mon.0) 13716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:42.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:44 smithi121 bash[36902]: cluster 2022-04-22T22:12:42.963745+0000 mgr.y (mgr.34107) 9776 : cluster [DBG] pgmap v10539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:44 smithi115 bash[45495]: cluster 2022-04-22T22:12:42.963745+0000 mgr.y (mgr.34107) 9776 : cluster [DBG] pgmap v10539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:44.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:44 smithi115 bash[46934]: cluster 2022-04-22T22:12:42.963745+0000 mgr.y (mgr.34107) 9776 : cluster [DBG] pgmap v10539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:46 smithi121 bash[36902]: cluster 2022-04-22T22:12:44.964818+0000 mgr.y (mgr.34107) 9777 : cluster [DBG] pgmap v10540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:46 smithi121 bash[36902]: audit 2022-04-22T22:12:45.865604+0000 mon.a (mon.0) 13717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:46 smithi115 bash[45495]: cluster 2022-04-22T22:12:44.964818+0000 mgr.y (mgr.34107) 9777 : cluster [DBG] pgmap v10540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:46 smithi115 bash[45495]: audit 2022-04-22T22:12:45.865604+0000 mon.a (mon.0) 13717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:46 smithi115 bash[46934]: cluster 2022-04-22T22:12:44.964818+0000 mgr.y (mgr.34107) 9777 : cluster [DBG] pgmap v10540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:46 smithi115 bash[46934]: audit 2022-04-22T22:12:45.865604+0000 mon.a (mon.0) 13717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:47 smithi121 bash[36902]: audit 2022-04-22T22:12:47.059741+0000 mon.a (mon.0) 13718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:47 smithi121 bash[36902]: audit 2022-04-22T22:12:47.265435+0000 mon.a (mon.0) 13719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:47 smithi115 bash[45495]: audit 2022-04-22T22:12:47.059741+0000 mon.a (mon.0) 13718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:47 smithi115 bash[45495]: audit 2022-04-22T22:12:47.265435+0000 mon.a (mon.0) 13719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:47 smithi115 bash[46934]: audit 2022-04-22T22:12:47.059741+0000 mon.a (mon.0) 13718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:47 smithi115 bash[46934]: audit 2022-04-22T22:12:47.265435+0000 mon.a (mon.0) 13719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:48 smithi121 bash[36902]: cluster 2022-04-22T22:12:46.965848+0000 mgr.y (mgr.34107) 9778 : cluster [DBG] pgmap v10541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:48.847 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:48 smithi115 bash[46934]: cluster 2022-04-22T22:12:46.965848+0000 mgr.y (mgr.34107) 9778 : cluster [DBG] pgmap v10541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:48.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:48 smithi115 bash[45495]: cluster 2022-04-22T22:12:46.965848+0000 mgr.y (mgr.34107) 9778 : cluster [DBG] pgmap v10541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:50 smithi121 bash[36902]: cluster 2022-04-22T22:12:48.966985+0000 mgr.y (mgr.34107) 9779 : cluster [DBG] pgmap v10542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:50 smithi115 bash[45495]: cluster 2022-04-22T22:12:48.966985+0000 mgr.y (mgr.34107) 9779 : cluster [DBG] pgmap v10542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:50 smithi115 bash[46934]: cluster 2022-04-22T22:12:48.966985+0000 mgr.y (mgr.34107) 9779 : cluster [DBG] pgmap v10542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:51 smithi121 bash[36902]: audit 2022-04-22T22:12:50.868983+0000 mon.a (mon.0) 13720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:51.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:51 smithi115 bash[46934]: audit 2022-04-22T22:12:50.868983+0000 mon.a (mon.0) 13720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:51.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:51 smithi115 bash[45495]: audit 2022-04-22T22:12:50.868983+0000 mon.a (mon.0) 13720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:52.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[46934]: cluster 2022-04-22T22:12:50.968088+0000 mgr.y (mgr.34107) 9780 : cluster [DBG] pgmap v10543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:52.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[46934]: audit 2022-04-22T22:12:52.063412+0000 mon.a (mon.0) 13721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:52.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[46934]: audit 2022-04-22T22:12:52.268439+0000 mon.a (mon.0) 13722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:52.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[45495]: cluster 2022-04-22T22:12:50.968088+0000 mgr.y (mgr.34107) 9780 : cluster [DBG] pgmap v10543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:52.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[45495]: audit 2022-04-22T22:12:52.063412+0000 mon.a (mon.0) 13721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:52.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[45495]: audit 2022-04-22T22:12:52.268439+0000 mon.a (mon.0) 13722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:52 smithi121 bash[36902]: cluster 2022-04-22T22:12:50.968088+0000 mgr.y (mgr.34107) 9780 : cluster [DBG] pgmap v10543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:52 smithi121 bash[36902]: audit 2022-04-22T22:12:52.063412+0000 mon.a (mon.0) 13721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:52 smithi121 bash[36902]: audit 2022-04-22T22:12:52.268439+0000 mon.a (mon.0) 13722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:52.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:54 smithi121 bash[36902]: cluster 2022-04-22T22:12:52.969198+0000 mgr.y (mgr.34107) 9781 : cluster [DBG] pgmap v10544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:12:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:12:54.601Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:12:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:54 smithi115 bash[45495]: cluster 2022-04-22T22:12:52.969198+0000 mgr.y (mgr.34107) 9781 : cluster [DBG] pgmap v10544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:54 smithi115 bash[46934]: cluster 2022-04-22T22:12:52.969198+0000 mgr.y (mgr.34107) 9781 : cluster [DBG] pgmap v10544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:56 smithi121 bash[36902]: cluster 2022-04-22T22:12:54.970320+0000 mgr.y (mgr.34107) 9782 : cluster [DBG] pgmap v10545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:56 smithi121 bash[36902]: audit 2022-04-22T22:12:55.871826+0000 mon.a (mon.0) 13723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:56 smithi115 bash[45495]: cluster 2022-04-22T22:12:54.970320+0000 mgr.y (mgr.34107) 9782 : cluster [DBG] pgmap v10545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:56 smithi115 bash[45495]: audit 2022-04-22T22:12:55.871826+0000 mon.a (mon.0) 13723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:56 smithi115 bash[46934]: cluster 2022-04-22T22:12:54.970320+0000 mgr.y (mgr.34107) 9782 : cluster [DBG] pgmap v10545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:56 smithi115 bash[46934]: audit 2022-04-22T22:12:55.871826+0000 mon.a (mon.0) 13723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:12:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:57 smithi121 bash[36902]: audit 2022-04-22T22:12:57.067282+0000 mon.a (mon.0) 13724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:57 smithi121 bash[36902]: audit 2022-04-22T22:12:57.271565+0000 mon.a (mon.0) 13725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:57 smithi115 bash[45495]: audit 2022-04-22T22:12:57.067282+0000 mon.a (mon.0) 13724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:57 smithi115 bash[45495]: audit 2022-04-22T22:12:57.271565+0000 mon.a (mon.0) 13725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:57 smithi115 bash[46934]: audit 2022-04-22T22:12:57.067282+0000 mon.a (mon.0) 13724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:12:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:57 smithi115 bash[46934]: audit 2022-04-22T22:12:57.271565+0000 mon.a (mon.0) 13725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:12:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:58 smithi121 bash[36902]: cluster 2022-04-22T22:12:56.971375+0000 mgr.y (mgr.34107) 9783 : cluster [DBG] pgmap v10546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:58 smithi115 bash[45495]: cluster 2022-04-22T22:12:56.971375+0000 mgr.y (mgr.34107) 9783 : cluster [DBG] pgmap v10546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:58 smithi115 bash[46934]: cluster 2022-04-22T22:12:56.971375+0000 mgr.y (mgr.34107) 9783 : cluster [DBG] pgmap v10546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:12:59 smithi121 bash[36902]: cluster 2022-04-22T22:12:58.972527+0000 mgr.y (mgr.34107) 9784 : cluster [DBG] pgmap v10547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:59.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:12:59 smithi115 bash[45495]: cluster 2022-04-22T22:12:58.972527+0000 mgr.y (mgr.34107) 9784 : cluster [DBG] pgmap v10547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:12:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:12:59 smithi115 bash[46934]: cluster 2022-04-22T22:12:58.972527+0000 mgr.y (mgr.34107) 9784 : cluster [DBG] pgmap v10547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:13:00.954 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:13:00.955 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:13:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:13:00.957 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:13:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:00 smithi121 bash[36902]: audit 2022-04-22T22:13:00.874993+0000 mon.a (mon.0) 13726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:00 smithi115 bash[45495]: audit 2022-04-22T22:13:00.874993+0000 mon.a (mon.0) 13726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:00 smithi115 bash[46934]: audit 2022-04-22T22:13:00.874993+0000 mon.a (mon.0) 13726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:01.352 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:13:01.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:13:01.354 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:13:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:02 smithi121 bash[36902]: audit 2022-04-22T22:13:00.593383+0000 mgr.y (mgr.34107) 9785 : audit [DBG] from='client.55435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:02 smithi121 bash[36902]: audit 2022-04-22T22:13:00.932341+0000 mgr.y (mgr.34107) 9786 : audit [DBG] from='client.45510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:02 smithi121 bash[36902]: cluster 2022-04-22T22:13:00.973618+0000 mgr.y (mgr.34107) 9787 : cluster [DBG] pgmap v10548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:02 smithi121 bash[36902]: audit 2022-04-22T22:13:01.349016+0000 mon.c (mon.1) 339 : audit [DBG] from='client.? 172.21.15.115:0/489651440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:02 smithi121 bash[36902]: audit 2022-04-22T22:13:02.071082+0000 mon.a (mon.0) 13727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[45495]: audit 2022-04-22T22:13:00.593383+0000 mgr.y (mgr.34107) 9785 : audit [DBG] from='client.55435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[45495]: audit 2022-04-22T22:13:00.932341+0000 mgr.y (mgr.34107) 9786 : audit [DBG] from='client.45510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[45495]: cluster 2022-04-22T22:13:00.973618+0000 mgr.y (mgr.34107) 9787 : cluster [DBG] pgmap v10548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[45495]: audit 2022-04-22T22:13:01.349016+0000 mon.c (mon.1) 339 : audit [DBG] from='client.? 172.21.15.115:0/489651440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[45495]: audit 2022-04-22T22:13:02.071082+0000 mon.a (mon.0) 13727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[46934]: audit 2022-04-22T22:13:00.593383+0000 mgr.y (mgr.34107) 9785 : audit [DBG] from='client.55435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[46934]: audit 2022-04-22T22:13:00.932341+0000 mgr.y (mgr.34107) 9786 : audit [DBG] from='client.45510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[46934]: cluster 2022-04-22T22:13:00.973618+0000 mgr.y (mgr.34107) 9787 : cluster [DBG] pgmap v10548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[46934]: audit 2022-04-22T22:13:01.349016+0000 mon.c (mon.1) 339 : audit [DBG] from='client.? 172.21.15.115:0/489651440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[46934]: audit 2022-04-22T22:13:02.071082+0000 mon.a (mon.0) 13727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:02.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:03 smithi121 bash[36902]: audit 2022-04-22T22:13:02.274446+0000 mon.a (mon.0) 13728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:03 smithi115 bash[45495]: audit 2022-04-22T22:13:02.274446+0000 mon.a (mon.0) 13728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:03.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:03 smithi115 bash[46934]: audit 2022-04-22T22:13:02.274446+0000 mon.a (mon.0) 13728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:04 smithi121 bash[36902]: cluster 2022-04-22T22:13:02.974773+0000 mgr.y (mgr.34107) 9788 : cluster [DBG] pgmap v10549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:04 smithi115 bash[46934]: cluster 2022-04-22T22:13:02.974773+0000 mgr.y (mgr.34107) 9788 : cluster [DBG] pgmap v10549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:04 smithi115 bash[45495]: cluster 2022-04-22T22:13:02.974773+0000 mgr.y (mgr.34107) 9788 : cluster [DBG] pgmap v10549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:04.600Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:06 smithi121 bash[36902]: cluster 2022-04-22T22:13:04.975943+0000 mgr.y (mgr.34107) 9789 : cluster [DBG] pgmap v10550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:06 smithi121 bash[36902]: audit 2022-04-22T22:13:05.878042+0000 mon.a (mon.0) 13729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:06 smithi115 bash[45495]: cluster 2022-04-22T22:13:04.975943+0000 mgr.y (mgr.34107) 9789 : cluster [DBG] pgmap v10550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:06 smithi115 bash[45495]: audit 2022-04-22T22:13:05.878042+0000 mon.a (mon.0) 13729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:06 smithi115 bash[46934]: cluster 2022-04-22T22:13:04.975943+0000 mgr.y (mgr.34107) 9789 : cluster [DBG] pgmap v10550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:06 smithi115 bash[46934]: audit 2022-04-22T22:13:05.878042+0000 mon.a (mon.0) 13729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:07 smithi121 bash[36902]: audit 2022-04-22T22:13:07.074864+0000 mon.a (mon.0) 13730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:07 smithi115 bash[45495]: audit 2022-04-22T22:13:07.074864+0000 mon.a (mon.0) 13730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:07 smithi115 bash[46934]: audit 2022-04-22T22:13:07.074864+0000 mon.a (mon.0) 13730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:08 smithi115 bash[45495]: cluster 2022-04-22T22:13:06.977029+0000 mgr.y (mgr.34107) 9790 : cluster [DBG] pgmap v10551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:08 smithi115 bash[45495]: audit 2022-04-22T22:13:07.277402+0000 mon.a (mon.0) 13731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:08 smithi115 bash[46934]: cluster 2022-04-22T22:13:06.977029+0000 mgr.y (mgr.34107) 9790 : cluster [DBG] pgmap v10551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:08 smithi115 bash[46934]: audit 2022-04-22T22:13:07.277402+0000 mon.a (mon.0) 13731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:08 smithi121 bash[36902]: cluster 2022-04-22T22:13:06.977029+0000 mgr.y (mgr.34107) 9790 : cluster [DBG] pgmap v10551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:08 smithi121 bash[36902]: audit 2022-04-22T22:13:07.277402+0000 mon.a (mon.0) 13731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:10 smithi115 bash[45495]: cluster 2022-04-22T22:13:08.978194+0000 mgr.y (mgr.34107) 9791 : cluster [DBG] pgmap v10552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:10 smithi115 bash[46934]: cluster 2022-04-22T22:13:08.978194+0000 mgr.y (mgr.34107) 9791 : cluster [DBG] pgmap v10552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:10 smithi121 bash[36902]: cluster 2022-04-22T22:13:08.978194+0000 mgr.y (mgr.34107) 9791 : cluster [DBG] pgmap v10552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:11 smithi115 bash[45495]: audit 2022-04-22T22:13:10.881334+0000 mon.a (mon.0) 13732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:11 smithi115 bash[46934]: audit 2022-04-22T22:13:10.881334+0000 mon.a (mon.0) 13732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:11 smithi121 bash[36902]: audit 2022-04-22T22:13:10.881334+0000 mon.a (mon.0) 13732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:12 smithi115 bash[45495]: cluster 2022-04-22T22:13:10.979269+0000 mgr.y (mgr.34107) 9792 : cluster [DBG] pgmap v10553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:12 smithi115 bash[45495]: audit 2022-04-22T22:13:12.078633+0000 mon.a (mon.0) 13733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:12 smithi115 bash[46934]: cluster 2022-04-22T22:13:10.979269+0000 mgr.y (mgr.34107) 9792 : cluster [DBG] pgmap v10553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:12 smithi115 bash[46934]: audit 2022-04-22T22:13:12.078633+0000 mon.a (mon.0) 13733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:12 smithi121 bash[36902]: cluster 2022-04-22T22:13:10.979269+0000 mgr.y (mgr.34107) 9792 : cluster [DBG] pgmap v10553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:12 smithi121 bash[36902]: audit 2022-04-22T22:13:12.078633+0000 mon.a (mon.0) 13733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:12.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:13 smithi115 bash[45495]: audit 2022-04-22T22:13:12.284083+0000 mon.a (mon.0) 13734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:13 smithi115 bash[46934]: audit 2022-04-22T22:13:12.284083+0000 mon.a (mon.0) 13734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:13 smithi121 bash[36902]: audit 2022-04-22T22:13:12.284083+0000 mon.a (mon.0) 13734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:14 smithi115 bash[45495]: cluster 2022-04-22T22:13:12.980288+0000 mgr.y (mgr.34107) 9793 : cluster [DBG] pgmap v10554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:14 smithi115 bash[46934]: cluster 2022-04-22T22:13:12.980288+0000 mgr.y (mgr.34107) 9793 : cluster [DBG] pgmap v10554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:14 smithi121 bash[36902]: cluster 2022-04-22T22:13:12.980288+0000 mgr.y (mgr.34107) 9793 : cluster [DBG] pgmap v10554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:14.603Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:16 smithi121 bash[36902]: cluster 2022-04-22T22:13:14.981638+0000 mgr.y (mgr.34107) 9794 : cluster [DBG] pgmap v10555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:16 smithi121 bash[36902]: audit 2022-04-22T22:13:15.884492+0000 mon.a (mon.0) 13735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:16 smithi115 bash[45495]: cluster 2022-04-22T22:13:14.981638+0000 mgr.y (mgr.34107) 9794 : cluster [DBG] pgmap v10555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:16 smithi115 bash[45495]: audit 2022-04-22T22:13:15.884492+0000 mon.a (mon.0) 13735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:16 smithi115 bash[46934]: cluster 2022-04-22T22:13:14.981638+0000 mgr.y (mgr.34107) 9794 : cluster [DBG] pgmap v10555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:16 smithi115 bash[46934]: audit 2022-04-22T22:13:15.884492+0000 mon.a (mon.0) 13735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:17 smithi121 bash[36902]: audit 2022-04-22T22:13:16.581976+0000 mon.a (mon.0) 13736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:13:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:17 smithi121 bash[36902]: audit 2022-04-22T22:13:16.582956+0000 mon.a (mon.0) 13737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:13:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:17 smithi121 bash[36902]: audit 2022-04-22T22:13:17.082214+0000 mon.a (mon.0) 13738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:17 smithi121 bash[36902]: audit 2022-04-22T22:13:17.284468+0000 mon.a (mon.0) 13739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[45495]: audit 2022-04-22T22:13:16.581976+0000 mon.a (mon.0) 13736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[45495]: audit 2022-04-22T22:13:16.582956+0000 mon.a (mon.0) 13737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[45495]: audit 2022-04-22T22:13:17.082214+0000 mon.a (mon.0) 13738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[45495]: audit 2022-04-22T22:13:17.284468+0000 mon.a (mon.0) 13739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[46934]: audit 2022-04-22T22:13:16.581976+0000 mon.a (mon.0) 13736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[46934]: audit 2022-04-22T22:13:16.582956+0000 mon.a (mon.0) 13737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[46934]: audit 2022-04-22T22:13:17.082214+0000 mon.a (mon.0) 13738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:17 smithi115 bash[46934]: audit 2022-04-22T22:13:17.284468+0000 mon.a (mon.0) 13739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:18 smithi121 bash[36902]: cluster 2022-04-22T22:13:16.982915+0000 mgr.y (mgr.34107) 9795 : cluster [DBG] pgmap v10556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:18 smithi115 bash[46934]: cluster 2022-04-22T22:13:16.982915+0000 mgr.y (mgr.34107) 9795 : cluster [DBG] pgmap v10556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:18 smithi115 bash[45495]: cluster 2022-04-22T22:13:16.982915+0000 mgr.y (mgr.34107) 9795 : cluster [DBG] pgmap v10556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:20 smithi121 bash[36902]: cluster 2022-04-22T22:13:18.984114+0000 mgr.y (mgr.34107) 9796 : cluster [DBG] pgmap v10557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:20.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:20 smithi115 bash[46934]: cluster 2022-04-22T22:13:18.984114+0000 mgr.y (mgr.34107) 9796 : cluster [DBG] pgmap v10557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:20 smithi115 bash[45495]: cluster 2022-04-22T22:13:18.984114+0000 mgr.y (mgr.34107) 9796 : cluster [DBG] pgmap v10557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:21 smithi121 bash[36902]: audit 2022-04-22T22:13:20.887689+0000 mon.a (mon.0) 13740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:21 smithi115 bash[45495]: audit 2022-04-22T22:13:20.887689+0000 mon.a (mon.0) 13740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:21 smithi115 bash[46934]: audit 2022-04-22T22:13:20.887689+0000 mon.a (mon.0) 13740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:22.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[45495]: cluster 2022-04-22T22:13:20.985161+0000 mgr.y (mgr.34107) 9797 : cluster [DBG] pgmap v10558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:22.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[45495]: audit 2022-04-22T22:13:22.085848+0000 mon.a (mon.0) 13741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:22.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[45495]: audit 2022-04-22T22:13:22.287202+0000 mon.a (mon.0) 13742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:22.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[46934]: cluster 2022-04-22T22:13:20.985161+0000 mgr.y (mgr.34107) 9797 : cluster [DBG] pgmap v10558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:22.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[46934]: audit 2022-04-22T22:13:22.085848+0000 mon.a (mon.0) 13741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:22.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[46934]: audit 2022-04-22T22:13:22.287202+0000 mon.a (mon.0) 13742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:22 smithi121 bash[36902]: cluster 2022-04-22T22:13:20.985161+0000 mgr.y (mgr.34107) 9797 : cluster [DBG] pgmap v10558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:22 smithi121 bash[36902]: audit 2022-04-22T22:13:22.085848+0000 mon.a (mon.0) 13741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:22 smithi121 bash[36902]: audit 2022-04-22T22:13:22.287202+0000 mon.a (mon.0) 13742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:22.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:24 smithi115 bash[46934]: cluster 2022-04-22T22:13:22.986322+0000 mgr.y (mgr.34107) 9798 : cluster [DBG] pgmap v10559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:24.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:24 smithi115 bash[45495]: cluster 2022-04-22T22:13:22.986322+0000 mgr.y (mgr.34107) 9798 : cluster [DBG] pgmap v10559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:24 smithi121 bash[36902]: cluster 2022-04-22T22:13:22.986322+0000 mgr.y (mgr.34107) 9798 : cluster [DBG] pgmap v10559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:24.602Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:26 smithi121 bash[36902]: cluster 2022-04-22T22:13:24.987593+0000 mgr.y (mgr.34107) 9799 : cluster [DBG] pgmap v10560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:26 smithi121 bash[36902]: audit 2022-04-22T22:13:25.891305+0000 mon.a (mon.0) 13743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:26.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:26 smithi115 bash[46934]: cluster 2022-04-22T22:13:24.987593+0000 mgr.y (mgr.34107) 9799 : cluster [DBG] pgmap v10560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:26 smithi115 bash[46934]: audit 2022-04-22T22:13:25.891305+0000 mon.a (mon.0) 13743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:26 smithi115 bash[45495]: cluster 2022-04-22T22:13:24.987593+0000 mgr.y (mgr.34107) 9799 : cluster [DBG] pgmap v10560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:26 smithi115 bash[45495]: audit 2022-04-22T22:13:25.891305+0000 mon.a (mon.0) 13743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:27 smithi121 bash[36902]: audit 2022-04-22T22:13:27.089418+0000 mon.a (mon.0) 13744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:27 smithi121 bash[36902]: audit 2022-04-22T22:13:27.290285+0000 mon.a (mon.0) 13745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:27.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:27 smithi115 bash[46934]: audit 2022-04-22T22:13:27.089418+0000 mon.a (mon.0) 13744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:27.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:27 smithi115 bash[46934]: audit 2022-04-22T22:13:27.290285+0000 mon.a (mon.0) 13745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:27 smithi115 bash[45495]: audit 2022-04-22T22:13:27.089418+0000 mon.a (mon.0) 13744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:27 smithi115 bash[45495]: audit 2022-04-22T22:13:27.290285+0000 mon.a (mon.0) 13745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:28 smithi121 bash[36902]: cluster 2022-04-22T22:13:26.988811+0000 mgr.y (mgr.34107) 9800 : cluster [DBG] pgmap v10561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:28.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:28 smithi115 bash[46934]: cluster 2022-04-22T22:13:26.988811+0000 mgr.y (mgr.34107) 9800 : cluster [DBG] pgmap v10561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:28 smithi115 bash[45495]: cluster 2022-04-22T22:13:26.988811+0000 mgr.y (mgr.34107) 9800 : cluster [DBG] pgmap v10561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:30 smithi121 bash[36902]: cluster 2022-04-22T22:13:28.990016+0000 mgr.y (mgr.34107) 9801 : cluster [DBG] pgmap v10562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:30 smithi115 bash[45495]: cluster 2022-04-22T22:13:28.990016+0000 mgr.y (mgr.34107) 9801 : cluster [DBG] pgmap v10562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:30 smithi115 bash[46934]: cluster 2022-04-22T22:13:28.990016+0000 mgr.y (mgr.34107) 9801 : cluster [DBG] pgmap v10562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:31.730 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:13:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:31 smithi121 bash[36902]: audit 2022-04-22T22:13:30.894407+0000 mon.a (mon.0) 13746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:31 smithi115 bash[45495]: audit 2022-04-22T22:13:30.894407+0000 mon.a (mon.0) 13746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:31 smithi115 bash[46934]: audit 2022-04-22T22:13:30.894407+0000 mon.a (mon.0) 13746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:32.079 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:13:32.079 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:13:32.080 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:13:32.081 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:13:32.480 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:13:32.480 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:13:32.480 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:13:32.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:13:32.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:13:32.482 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:13:32.482 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:13:32.482 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:13:32.482 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:13:32.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[46934]: cluster 2022-04-22T22:13:30.990933+0000 mgr.y (mgr.34107) 9802 : cluster [DBG] pgmap v10563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:32.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[46934]: audit 2022-04-22T22:13:32.092968+0000 mon.a (mon.0) 13747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:32.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[46934]: audit 2022-04-22T22:13:32.293240+0000 mon.a (mon.0) 13748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:32.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[45495]: cluster 2022-04-22T22:13:30.990933+0000 mgr.y (mgr.34107) 9802 : cluster [DBG] pgmap v10563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:32.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[45495]: audit 2022-04-22T22:13:32.092968+0000 mon.a (mon.0) 13747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:32.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[45495]: audit 2022-04-22T22:13:32.293240+0000 mon.a (mon.0) 13748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:32 smithi121 bash[36902]: cluster 2022-04-22T22:13:30.990933+0000 mgr.y (mgr.34107) 9802 : cluster [DBG] pgmap v10563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:32 smithi121 bash[36902]: audit 2022-04-22T22:13:32.092968+0000 mon.a (mon.0) 13747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:32 smithi121 bash[36902]: audit 2022-04-22T22:13:32.293240+0000 mon.a (mon.0) 13748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:32.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:33 smithi121 bash[36902]: audit 2022-04-22T22:13:31.712767+0000 mgr.y (mgr.34107) 9803 : audit [DBG] from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:33 smithi121 bash[36902]: audit 2022-04-22T22:13:32.058480+0000 mgr.y (mgr.34107) 9804 : audit [DBG] from='client.45528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:33 smithi121 bash[36902]: audit 2022-04-22T22:13:32.476996+0000 mon.a (mon.0) 13749 : audit [DBG] from='client.? 172.21.15.115:0/2676794801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[45495]: audit 2022-04-22T22:13:31.712767+0000 mgr.y (mgr.34107) 9803 : audit [DBG] from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[45495]: audit 2022-04-22T22:13:32.058480+0000 mgr.y (mgr.34107) 9804 : audit [DBG] from='client.45528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[45495]: audit 2022-04-22T22:13:32.476996+0000 mon.a (mon.0) 13749 : audit [DBG] from='client.? 172.21.15.115:0/2676794801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[46934]: audit 2022-04-22T22:13:31.712767+0000 mgr.y (mgr.34107) 9803 : audit [DBG] from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[46934]: audit 2022-04-22T22:13:32.058480+0000 mgr.y (mgr.34107) 9804 : audit [DBG] from='client.45528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:13:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:33 smithi115 bash[46934]: audit 2022-04-22T22:13:32.476996+0000 mon.a (mon.0) 13749 : audit [DBG] from='client.? 172.21.15.115:0/2676794801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:13:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:34 smithi121 bash[36902]: cluster 2022-04-22T22:13:32.991895+0000 mgr.y (mgr.34107) 9805 : cluster [DBG] pgmap v10564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:34 smithi115 bash[45495]: cluster 2022-04-22T22:13:32.991895+0000 mgr.y (mgr.34107) 9805 : cluster [DBG] pgmap v10564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:34.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:34 smithi115 bash[46934]: cluster 2022-04-22T22:13:32.991895+0000 mgr.y (mgr.34107) 9805 : cluster [DBG] pgmap v10564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:36 smithi121 bash[36902]: cluster 2022-04-22T22:13:34.992966+0000 mgr.y (mgr.34107) 9806 : cluster [DBG] pgmap v10565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:36 smithi121 bash[36902]: audit 2022-04-22T22:13:35.897517+0000 mon.a (mon.0) 13750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:36 smithi115 bash[45495]: cluster 2022-04-22T22:13:34.992966+0000 mgr.y (mgr.34107) 9806 : cluster [DBG] pgmap v10565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:36 smithi115 bash[45495]: audit 2022-04-22T22:13:35.897517+0000 mon.a (mon.0) 13750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:36 smithi115 bash[46934]: cluster 2022-04-22T22:13:34.992966+0000 mgr.y (mgr.34107) 9806 : cluster [DBG] pgmap v10565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:36 smithi115 bash[46934]: audit 2022-04-22T22:13:35.897517+0000 mon.a (mon.0) 13750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:37.486 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:37 smithi121 bash[36902]: audit 2022-04-22T22:13:36.871248+0000 mon.a (mon.0) 13751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:13:37.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:37 smithi121 bash[36902]: audit 2022-04-22T22:13:36.874121+0000 mon.a (mon.0) 13752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:13:37.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:37 smithi121 bash[36902]: audit 2022-04-22T22:13:36.875826+0000 mon.a (mon.0) 13753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:13:37.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:37 smithi121 bash[36902]: audit 2022-04-22T22:13:37.096395+0000 mon.a (mon.0) 13754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:37.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:37 smithi121 bash[36902]: audit 2022-04-22T22:13:37.296113+0000 mon.a (mon.0) 13755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[45495]: audit 2022-04-22T22:13:36.871248+0000 mon.a (mon.0) 13751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[45495]: audit 2022-04-22T22:13:36.874121+0000 mon.a (mon.0) 13752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[45495]: audit 2022-04-22T22:13:36.875826+0000 mon.a (mon.0) 13753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[45495]: audit 2022-04-22T22:13:37.096395+0000 mon.a (mon.0) 13754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[45495]: audit 2022-04-22T22:13:37.296113+0000 mon.a (mon.0) 13755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[46934]: audit 2022-04-22T22:13:36.871248+0000 mon.a (mon.0) 13751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:13:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[46934]: audit 2022-04-22T22:13:36.874121+0000 mon.a (mon.0) 13752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:13:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[46934]: audit 2022-04-22T22:13:36.875826+0000 mon.a (mon.0) 13753 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:13:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[46934]: audit 2022-04-22T22:13:37.096395+0000 mon.a (mon.0) 13754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:37.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:37 smithi115 bash[46934]: audit 2022-04-22T22:13:37.296113+0000 mon.a (mon.0) 13755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:38 smithi121 bash[36902]: cluster 2022-04-22T22:13:36.994160+0000 mgr.y (mgr.34107) 9807 : cluster [DBG] pgmap v10566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:38 smithi121 bash[36902]: audit 2022-04-22T22:13:37.500554+0000 mon.a (mon.0) 13756 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:38 smithi115 bash[45495]: cluster 2022-04-22T22:13:36.994160+0000 mgr.y (mgr.34107) 9807 : cluster [DBG] pgmap v10566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:38 smithi115 bash[45495]: audit 2022-04-22T22:13:37.500554+0000 mon.a (mon.0) 13756 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:38 smithi115 bash[46934]: cluster 2022-04-22T22:13:36.994160+0000 mgr.y (mgr.34107) 9807 : cluster [DBG] pgmap v10566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:38 smithi115 bash[46934]: audit 2022-04-22T22:13:37.500554+0000 mon.a (mon.0) 13756 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:40 smithi121 bash[36902]: cluster 2022-04-22T22:13:38.994933+0000 mgr.y (mgr.34107) 9808 : cluster [DBG] pgmap v10567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:40 smithi115 bash[45495]: cluster 2022-04-22T22:13:38.994933+0000 mgr.y (mgr.34107) 9808 : cluster [DBG] pgmap v10567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:40 smithi115 bash[46934]: cluster 2022-04-22T22:13:38.994933+0000 mgr.y (mgr.34107) 9808 : cluster [DBG] pgmap v10567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:41.733 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:41 smithi121 bash[36902]: audit 2022-04-22T22:13:40.900262+0000 mon.a (mon.0) 13757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:41 smithi115 bash[45495]: audit 2022-04-22T22:13:40.900262+0000 mon.a (mon.0) 13757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:41 smithi115 bash[46934]: audit 2022-04-22T22:13:40.900262+0000 mon.a (mon.0) 13757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:42.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[45495]: cluster 2022-04-22T22:13:40.996087+0000 mgr.y (mgr.34107) 9809 : cluster [DBG] pgmap v10568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:42.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[45495]: audit 2022-04-22T22:13:42.099686+0000 mon.a (mon.0) 13758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:42.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[45495]: audit 2022-04-22T22:13:42.298596+0000 mon.a (mon.0) 13759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:42.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[46934]: cluster 2022-04-22T22:13:40.996087+0000 mgr.y (mgr.34107) 9809 : cluster [DBG] pgmap v10568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:42.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[46934]: audit 2022-04-22T22:13:42.099686+0000 mon.a (mon.0) 13758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:42.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[46934]: audit 2022-04-22T22:13:42.298596+0000 mon.a (mon.0) 13759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:42 smithi121 bash[36902]: cluster 2022-04-22T22:13:40.996087+0000 mgr.y (mgr.34107) 9809 : cluster [DBG] pgmap v10568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:42 smithi121 bash[36902]: audit 2022-04-22T22:13:42.099686+0000 mon.a (mon.0) 13758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:42 smithi121 bash[36902]: audit 2022-04-22T22:13:42.298596+0000 mon.a (mon.0) 13759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:42.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:44 smithi121 bash[36902]: cluster 2022-04-22T22:13:42.997128+0000 mgr.y (mgr.34107) 9810 : cluster [DBG] pgmap v10569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:44 smithi115 bash[45495]: cluster 2022-04-22T22:13:42.997128+0000 mgr.y (mgr.34107) 9810 : cluster [DBG] pgmap v10569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:44.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:44 smithi115 bash[46934]: cluster 2022-04-22T22:13:42.997128+0000 mgr.y (mgr.34107) 9810 : cluster [DBG] pgmap v10569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:46 smithi121 bash[36902]: cluster 2022-04-22T22:13:44.998264+0000 mgr.y (mgr.34107) 9811 : cluster [DBG] pgmap v10570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:46 smithi121 bash[36902]: audit 2022-04-22T22:13:45.903358+0000 mon.a (mon.0) 13760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:46 smithi115 bash[45495]: cluster 2022-04-22T22:13:44.998264+0000 mgr.y (mgr.34107) 9811 : cluster [DBG] pgmap v10570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:46 smithi115 bash[45495]: audit 2022-04-22T22:13:45.903358+0000 mon.a (mon.0) 13760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:46 smithi115 bash[46934]: cluster 2022-04-22T22:13:44.998264+0000 mgr.y (mgr.34107) 9811 : cluster [DBG] pgmap v10570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:46 smithi115 bash[46934]: audit 2022-04-22T22:13:45.903358+0000 mon.a (mon.0) 13760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:47 smithi121 bash[36902]: cephadm 2022-04-22T22:13:46.426488+0000 mgr.y (mgr.34107) 9812 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:13:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:47 smithi121 bash[36902]: audit 2022-04-22T22:13:46.436522+0000 mon.a (mon.0) 13761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:47 smithi121 bash[36902]: audit 2022-04-22T22:13:46.445067+0000 mon.a (mon.0) 13762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:47 smithi121 bash[36902]: audit 2022-04-22T22:13:47.103174+0000 mon.a (mon.0) 13763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:47 smithi121 bash[36902]: audit 2022-04-22T22:13:47.301792+0000 mon.a (mon.0) 13764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[45495]: cephadm 2022-04-22T22:13:46.426488+0000 mgr.y (mgr.34107) 9812 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:13:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[45495]: audit 2022-04-22T22:13:46.436522+0000 mon.a (mon.0) 13761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[45495]: audit 2022-04-22T22:13:46.445067+0000 mon.a (mon.0) 13762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[45495]: audit 2022-04-22T22:13:47.103174+0000 mon.a (mon.0) 13763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[45495]: audit 2022-04-22T22:13:47.301792+0000 mon.a (mon.0) 13764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[46934]: cephadm 2022-04-22T22:13:46.426488+0000 mgr.y (mgr.34107) 9812 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:13:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[46934]: audit 2022-04-22T22:13:46.436522+0000 mon.a (mon.0) 13761 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[46934]: audit 2022-04-22T22:13:46.445067+0000 mon.a (mon.0) 13762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:13:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[46934]: audit 2022-04-22T22:13:47.103174+0000 mon.a (mon.0) 13763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:47.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:47 smithi115 bash[46934]: audit 2022-04-22T22:13:47.301792+0000 mon.a (mon.0) 13764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:48 smithi121 bash[36902]: cluster 2022-04-22T22:13:46.999325+0000 mgr.y (mgr.34107) 9813 : cluster [DBG] pgmap v10571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:48 smithi115 bash[45495]: cluster 2022-04-22T22:13:46.999325+0000 mgr.y (mgr.34107) 9813 : cluster [DBG] pgmap v10571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:48 smithi115 bash[46934]: cluster 2022-04-22T22:13:46.999325+0000 mgr.y (mgr.34107) 9813 : cluster [DBG] pgmap v10571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:50 smithi121 bash[36902]: cluster 2022-04-22T22:13:49.000519+0000 mgr.y (mgr.34107) 9814 : cluster [DBG] pgmap v10572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:50 smithi115 bash[45495]: cluster 2022-04-22T22:13:49.000519+0000 mgr.y (mgr.34107) 9814 : cluster [DBG] pgmap v10572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:50 smithi115 bash[46934]: cluster 2022-04-22T22:13:49.000519+0000 mgr.y (mgr.34107) 9814 : cluster [DBG] pgmap v10572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:51 smithi121 bash[36902]: audit 2022-04-22T22:13:50.906521+0000 mon.a (mon.0) 13765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:51 smithi115 bash[45495]: audit 2022-04-22T22:13:50.906521+0000 mon.a (mon.0) 13765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:51 smithi115 bash[46934]: audit 2022-04-22T22:13:50.906521+0000 mon.a (mon.0) 13765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:52.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[45495]: cluster 2022-04-22T22:13:51.001548+0000 mgr.y (mgr.34107) 9815 : cluster [DBG] pgmap v10573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:52.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[45495]: audit 2022-04-22T22:13:52.106862+0000 mon.a (mon.0) 13766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:52.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[45495]: audit 2022-04-22T22:13:52.304958+0000 mon.a (mon.0) 13767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:52.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[46934]: cluster 2022-04-22T22:13:51.001548+0000 mgr.y (mgr.34107) 9815 : cluster [DBG] pgmap v10573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:52.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[46934]: audit 2022-04-22T22:13:52.106862+0000 mon.a (mon.0) 13766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:52.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[46934]: audit 2022-04-22T22:13:52.304958+0000 mon.a (mon.0) 13767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:52 smithi121 bash[36902]: cluster 2022-04-22T22:13:51.001548+0000 mgr.y (mgr.34107) 9815 : cluster [DBG] pgmap v10573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:52 smithi121 bash[36902]: audit 2022-04-22T22:13:52.106862+0000 mon.a (mon.0) 13766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:52 smithi121 bash[36902]: audit 2022-04-22T22:13:52.304958+0000 mon.a (mon.0) 13767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:52.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:54 smithi121 bash[36902]: cluster 2022-04-22T22:13:53.002541+0000 mgr.y (mgr.34107) 9816 : cluster [DBG] pgmap v10574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:54 smithi115 bash[45495]: cluster 2022-04-22T22:13:53.002541+0000 mgr.y (mgr.34107) 9816 : cluster [DBG] pgmap v10574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:13:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:13:54.605Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:13:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:54 smithi115 bash[46934]: cluster 2022-04-22T22:13:53.002541+0000 mgr.y (mgr.34107) 9816 : cluster [DBG] pgmap v10574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:56 smithi121 bash[36902]: cluster 2022-04-22T22:13:55.003562+0000 mgr.y (mgr.34107) 9817 : cluster [DBG] pgmap v10575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:56 smithi121 bash[36902]: audit 2022-04-22T22:13:55.909677+0000 mon.a (mon.0) 13768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:56 smithi115 bash[45495]: cluster 2022-04-22T22:13:55.003562+0000 mgr.y (mgr.34107) 9817 : cluster [DBG] pgmap v10575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:56 smithi115 bash[45495]: audit 2022-04-22T22:13:55.909677+0000 mon.a (mon.0) 13768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:56 smithi115 bash[46934]: cluster 2022-04-22T22:13:55.003562+0000 mgr.y (mgr.34107) 9817 : cluster [DBG] pgmap v10575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:56 smithi115 bash[46934]: audit 2022-04-22T22:13:55.909677+0000 mon.a (mon.0) 13768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:13:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:57 smithi121 bash[36902]: audit 2022-04-22T22:13:57.110524+0000 mon.a (mon.0) 13769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:57 smithi121 bash[36902]: audit 2022-04-22T22:13:57.307831+0000 mon.a (mon.0) 13770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:57 smithi115 bash[45495]: audit 2022-04-22T22:13:57.110524+0000 mon.a (mon.0) 13769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:57 smithi115 bash[45495]: audit 2022-04-22T22:13:57.307831+0000 mon.a (mon.0) 13770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:57 smithi115 bash[46934]: audit 2022-04-22T22:13:57.110524+0000 mon.a (mon.0) 13769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:13:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:57 smithi115 bash[46934]: audit 2022-04-22T22:13:57.307831+0000 mon.a (mon.0) 13770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:13:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:13:58 smithi121 bash[36902]: cluster 2022-04-22T22:13:57.004507+0000 mgr.y (mgr.34107) 9818 : cluster [DBG] pgmap v10576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:13:58 smithi115 bash[45495]: cluster 2022-04-22T22:13:57.004507+0000 mgr.y (mgr.34107) 9818 : cluster [DBG] pgmap v10576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:13:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:13:58 smithi115 bash[46934]: cluster 2022-04-22T22:13:57.004507+0000 mgr.y (mgr.34107) 9818 : cluster [DBG] pgmap v10576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:00 smithi121 bash[36902]: cluster 2022-04-22T22:13:59.005793+0000 mgr.y (mgr.34107) 9819 : cluster [DBG] pgmap v10577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:00 smithi115 bash[45495]: cluster 2022-04-22T22:13:59.005793+0000 mgr.y (mgr.34107) 9819 : cluster [DBG] pgmap v10577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:00 smithi115 bash[46934]: cluster 2022-04-22T22:13:59.005793+0000 mgr.y (mgr.34107) 9819 : cluster [DBG] pgmap v10577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:01 smithi121 bash[36902]: audit 2022-04-22T22:14:00.912780+0000 mon.a (mon.0) 13771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:01 smithi115 bash[45495]: audit 2022-04-22T22:14:00.912780+0000 mon.a (mon.0) 13771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:01 smithi115 bash[46934]: audit 2022-04-22T22:14:00.912780+0000 mon.a (mon.0) 13771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:02 smithi121 bash[36902]: cluster 2022-04-22T22:14:01.006851+0000 mgr.y (mgr.34107) 9820 : cluster [DBG] pgmap v10578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:02 smithi121 bash[36902]: audit 2022-04-22T22:14:02.114477+0000 mon.a (mon.0) 13772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:02 smithi121 bash[36902]: audit 2022-04-22T22:14:02.310766+0000 mon.a (mon.0) 13773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:02.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[45495]: cluster 2022-04-22T22:14:01.006851+0000 mgr.y (mgr.34107) 9820 : cluster [DBG] pgmap v10578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[45495]: audit 2022-04-22T22:14:02.114477+0000 mon.a (mon.0) 13772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[45495]: audit 2022-04-22T22:14:02.310766+0000 mon.a (mon.0) 13773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:02.748Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[46934]: cluster 2022-04-22T22:14:01.006851+0000 mgr.y (mgr.34107) 9820 : cluster [DBG] pgmap v10578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[46934]: audit 2022-04-22T22:14:02.114477+0000 mon.a (mon.0) 13772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:02 smithi115 bash[46934]: audit 2022-04-22T22:14:02.310766+0000 mon.a (mon.0) 13773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:02.861 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:14:03.213 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:14:03.213 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 9m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:14:03.214 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:14:03.215 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:14:03.613 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:14:03.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:14:03.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:14:03.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:03.614 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:14:03.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:14:03.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:03.615 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:14:03.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:14:03.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:03.616 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:14:03.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:14:03.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:03.616 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:14:03.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:14:03.617 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:14:03.617 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:14:03.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[45495]: audit 2022-04-22T22:14:02.843173+0000 mgr.y (mgr.34107) 9821 : audit [DBG] from='client.55471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[45495]: cluster 2022-04-22T22:14:03.007814+0000 mgr.y (mgr.34107) 9822 : cluster [DBG] pgmap v10579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[45495]: audit 2022-04-22T22:14:03.185990+0000 mgr.y (mgr.34107) 9823 : audit [DBG] from='client.45546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[46934]: audit 2022-04-22T22:14:02.843173+0000 mgr.y (mgr.34107) 9821 : audit [DBG] from='client.55471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[46934]: cluster 2022-04-22T22:14:03.007814+0000 mgr.y (mgr.34107) 9822 : cluster [DBG] pgmap v10579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:03 smithi115 bash[46934]: audit 2022-04-22T22:14:03.185990+0000 mgr.y (mgr.34107) 9823 : audit [DBG] from='client.45546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:03 smithi121 bash[36902]: audit 2022-04-22T22:14:02.843173+0000 mgr.y (mgr.34107) 9821 : audit [DBG] from='client.55471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:03 smithi121 bash[36902]: cluster 2022-04-22T22:14:03.007814+0000 mgr.y (mgr.34107) 9822 : cluster [DBG] pgmap v10579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:03 smithi121 bash[36902]: audit 2022-04-22T22:14:03.185990+0000 mgr.y (mgr.34107) 9823 : audit [DBG] from='client.45546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:04 smithi115 bash[45495]: audit 2022-04-22T22:14:03.609611+0000 mon.a (mon.0) 13774 : audit [DBG] from='client.? 172.21.15.115:0/2575742409' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:04.604Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:04 smithi115 bash[46934]: audit 2022-04-22T22:14:03.609611+0000 mon.a (mon.0) 13774 : audit [DBG] from='client.? 172.21.15.115:0/2575742409' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:04 smithi121 bash[36902]: audit 2022-04-22T22:14:03.609611+0000 mon.a (mon.0) 13774 : audit [DBG] from='client.? 172.21.15.115:0/2575742409' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:05.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:05 smithi115 bash[46934]: cluster 2022-04-22T22:14:05.008904+0000 mgr.y (mgr.34107) 9824 : cluster [DBG] pgmap v10580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:05 smithi115 bash[45495]: cluster 2022-04-22T22:14:05.008904+0000 mgr.y (mgr.34107) 9824 : cluster [DBG] pgmap v10580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:05 smithi121 bash[36902]: cluster 2022-04-22T22:14:05.008904+0000 mgr.y (mgr.34107) 9824 : cluster [DBG] pgmap v10580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:06 smithi115 bash[45495]: audit 2022-04-22T22:14:05.915710+0000 mon.a (mon.0) 13775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:06 smithi115 bash[46934]: audit 2022-04-22T22:14:05.915710+0000 mon.a (mon.0) 13775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:06 smithi121 bash[36902]: audit 2022-04-22T22:14:05.915710+0000 mon.a (mon.0) 13775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:07 smithi121 bash[36902]: cluster 2022-04-22T22:14:07.009977+0000 mgr.y (mgr.34107) 9825 : cluster [DBG] pgmap v10581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:07 smithi121 bash[36902]: audit 2022-04-22T22:14:07.118303+0000 mon.a (mon.0) 13776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:08.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:07 smithi121 bash[36902]: audit 2022-04-22T22:14:07.313649+0000 mon.a (mon.0) 13777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[45495]: cluster 2022-04-22T22:14:07.009977+0000 mgr.y (mgr.34107) 9825 : cluster [DBG] pgmap v10581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[45495]: audit 2022-04-22T22:14:07.118303+0000 mon.a (mon.0) 13776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[45495]: audit 2022-04-22T22:14:07.313649+0000 mon.a (mon.0) 13777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[46934]: cluster 2022-04-22T22:14:07.009977+0000 mgr.y (mgr.34107) 9825 : cluster [DBG] pgmap v10581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[46934]: audit 2022-04-22T22:14:07.118303+0000 mon.a (mon.0) 13776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:07 smithi115 bash[46934]: audit 2022-04-22T22:14:07.313649+0000 mon.a (mon.0) 13777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:10 smithi121 bash[36902]: cluster 2022-04-22T22:14:09.011153+0000 mgr.y (mgr.34107) 9826 : cluster [DBG] pgmap v10582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:10 smithi115 bash[45495]: cluster 2022-04-22T22:14:09.011153+0000 mgr.y (mgr.34107) 9826 : cluster [DBG] pgmap v10582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:10 smithi115 bash[46934]: cluster 2022-04-22T22:14:09.011153+0000 mgr.y (mgr.34107) 9826 : cluster [DBG] pgmap v10582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:11 smithi121 bash[36902]: audit 2022-04-22T22:14:10.919061+0000 mon.a (mon.0) 13778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:11 smithi115 bash[45495]: audit 2022-04-22T22:14:10.919061+0000 mon.a (mon.0) 13778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:11 smithi115 bash[46934]: audit 2022-04-22T22:14:10.919061+0000 mon.a (mon.0) 13778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:12 smithi121 bash[36902]: cluster 2022-04-22T22:14:11.012204+0000 mgr.y (mgr.34107) 9827 : cluster [DBG] pgmap v10583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:12 smithi121 bash[36902]: audit 2022-04-22T22:14:12.122206+0000 mon.a (mon.0) 13779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:12 smithi115 bash[45495]: cluster 2022-04-22T22:14:11.012204+0000 mgr.y (mgr.34107) 9827 : cluster [DBG] pgmap v10583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:12 smithi115 bash[45495]: audit 2022-04-22T22:14:12.122206+0000 mon.a (mon.0) 13779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:12 smithi115 bash[46934]: cluster 2022-04-22T22:14:11.012204+0000 mgr.y (mgr.34107) 9827 : cluster [DBG] pgmap v10583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:12 smithi115 bash[46934]: audit 2022-04-22T22:14:12.122206+0000 mon.a (mon.0) 13779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:12.751Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:13 smithi115 bash[45495]: audit 2022-04-22T22:14:12.316671+0000 mon.a (mon.0) 13780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:13 smithi115 bash[46934]: audit 2022-04-22T22:14:12.316671+0000 mon.a (mon.0) 13780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:13 smithi121 bash[36902]: audit 2022-04-22T22:14:12.316671+0000 mon.a (mon.0) 13780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:14 smithi115 bash[45495]: cluster 2022-04-22T22:14:13.013311+0000 mgr.y (mgr.34107) 9828 : cluster [DBG] pgmap v10584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:14 smithi115 bash[46934]: cluster 2022-04-22T22:14:13.013311+0000 mgr.y (mgr.34107) 9828 : cluster [DBG] pgmap v10584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:14 smithi121 bash[36902]: cluster 2022-04-22T22:14:13.013311+0000 mgr.y (mgr.34107) 9828 : cluster [DBG] pgmap v10584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:14.605Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:16 smithi115 bash[45495]: cluster 2022-04-22T22:14:15.014586+0000 mgr.y (mgr.34107) 9829 : cluster [DBG] pgmap v10585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:16 smithi115 bash[45495]: audit 2022-04-22T22:14:15.922265+0000 mon.a (mon.0) 13781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:16 smithi115 bash[46934]: cluster 2022-04-22T22:14:15.014586+0000 mgr.y (mgr.34107) 9829 : cluster [DBG] pgmap v10585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:16 smithi115 bash[46934]: audit 2022-04-22T22:14:15.922265+0000 mon.a (mon.0) 13781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:16 smithi121 bash[36902]: cluster 2022-04-22T22:14:15.014586+0000 mgr.y (mgr.34107) 9829 : cluster [DBG] pgmap v10585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:16 smithi121 bash[36902]: audit 2022-04-22T22:14:15.922265+0000 mon.a (mon.0) 13781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[45495]: audit 2022-04-22T22:14:16.594861+0000 mon.a (mon.0) 13782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:14:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[45495]: audit 2022-04-22T22:14:16.595425+0000 mon.a (mon.0) 13783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:14:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[45495]: audit 2022-04-22T22:14:17.126023+0000 mon.a (mon.0) 13784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[46934]: audit 2022-04-22T22:14:16.594861+0000 mon.a (mon.0) 13782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:14:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[46934]: audit 2022-04-22T22:14:16.595425+0000 mon.a (mon.0) 13783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:14:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:17 smithi115 bash[46934]: audit 2022-04-22T22:14:17.126023+0000 mon.a (mon.0) 13784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:17 smithi121 bash[36902]: audit 2022-04-22T22:14:16.594861+0000 mon.a (mon.0) 13782 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:14:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:17 smithi121 bash[36902]: audit 2022-04-22T22:14:16.595425+0000 mon.a (mon.0) 13783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:14:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:17 smithi121 bash[36902]: audit 2022-04-22T22:14:17.126023+0000 mon.a (mon.0) 13784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:18 smithi115 bash[45495]: cluster 2022-04-22T22:14:17.015674+0000 mgr.y (mgr.34107) 9830 : cluster [DBG] pgmap v10586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:18 smithi115 bash[45495]: audit 2022-04-22T22:14:17.319717+0000 mon.a (mon.0) 13785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:18 smithi115 bash[46934]: cluster 2022-04-22T22:14:17.015674+0000 mgr.y (mgr.34107) 9830 : cluster [DBG] pgmap v10586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:18 smithi115 bash[46934]: audit 2022-04-22T22:14:17.319717+0000 mon.a (mon.0) 13785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:18 smithi121 bash[36902]: cluster 2022-04-22T22:14:17.015674+0000 mgr.y (mgr.34107) 9830 : cluster [DBG] pgmap v10586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:18 smithi121 bash[36902]: audit 2022-04-22T22:14:17.319717+0000 mon.a (mon.0) 13785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:20 smithi115 bash[45495]: cluster 2022-04-22T22:14:19.016865+0000 mgr.y (mgr.34107) 9831 : cluster [DBG] pgmap v10587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:20 smithi115 bash[46934]: cluster 2022-04-22T22:14:19.016865+0000 mgr.y (mgr.34107) 9831 : cluster [DBG] pgmap v10587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:20 smithi121 bash[36902]: cluster 2022-04-22T22:14:19.016865+0000 mgr.y (mgr.34107) 9831 : cluster [DBG] pgmap v10587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:21 smithi121 bash[36902]: audit 2022-04-22T22:14:20.925435+0000 mon.a (mon.0) 13786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:21 smithi115 bash[46934]: audit 2022-04-22T22:14:20.925435+0000 mon.a (mon.0) 13786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:21 smithi115 bash[45495]: audit 2022-04-22T22:14:20.925435+0000 mon.a (mon.0) 13786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:22.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:22 smithi115 bash[45495]: cluster 2022-04-22T22:14:21.017909+0000 mgr.y (mgr.34107) 9832 : cluster [DBG] pgmap v10588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:22.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:22 smithi115 bash[45495]: audit 2022-04-22T22:14:22.130038+0000 mon.a (mon.0) 13787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:22.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:22 smithi115 bash[46934]: cluster 2022-04-22T22:14:21.017909+0000 mgr.y (mgr.34107) 9832 : cluster [DBG] pgmap v10588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:22.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:22 smithi115 bash[46934]: audit 2022-04-22T22:14:22.130038+0000 mon.a (mon.0) 13787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:22 smithi121 bash[36902]: cluster 2022-04-22T22:14:21.017909+0000 mgr.y (mgr.34107) 9832 : cluster [DBG] pgmap v10588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:22 smithi121 bash[36902]: audit 2022-04-22T22:14:22.130038+0000 mon.a (mon.0) 13787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:22.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:23 smithi121 bash[36902]: audit 2022-04-22T22:14:22.322578+0000 mon.a (mon.0) 13788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:23 smithi115 bash[45495]: audit 2022-04-22T22:14:22.322578+0000 mon.a (mon.0) 13788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:23 smithi115 bash[46934]: audit 2022-04-22T22:14:22.322578+0000 mon.a (mon.0) 13788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:24.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:24 smithi115 bash[45495]: cluster 2022-04-22T22:14:23.018893+0000 mgr.y (mgr.34107) 9833 : cluster [DBG] pgmap v10589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:24 smithi115 bash[46934]: cluster 2022-04-22T22:14:23.018893+0000 mgr.y (mgr.34107) 9833 : cluster [DBG] pgmap v10589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:24 smithi121 bash[36902]: cluster 2022-04-22T22:14:23.018893+0000 mgr.y (mgr.34107) 9833 : cluster [DBG] pgmap v10589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:24.607Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:26 smithi121 bash[36902]: cluster 2022-04-22T22:14:25.020053+0000 mgr.y (mgr.34107) 9834 : cluster [DBG] pgmap v10590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:26 smithi121 bash[36902]: audit 2022-04-22T22:14:25.928783+0000 mon.a (mon.0) 13789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:26 smithi115 bash[45495]: cluster 2022-04-22T22:14:25.020053+0000 mgr.y (mgr.34107) 9834 : cluster [DBG] pgmap v10590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:26 smithi115 bash[45495]: audit 2022-04-22T22:14:25.928783+0000 mon.a (mon.0) 13789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:26 smithi115 bash[46934]: cluster 2022-04-22T22:14:25.020053+0000 mgr.y (mgr.34107) 9834 : cluster [DBG] pgmap v10590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:26 smithi115 bash[46934]: audit 2022-04-22T22:14:25.928783+0000 mon.a (mon.0) 13789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:27 smithi121 bash[36902]: audit 2022-04-22T22:14:27.133783+0000 mon.a (mon.0) 13790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:27 smithi115 bash[45495]: audit 2022-04-22T22:14:27.133783+0000 mon.a (mon.0) 13790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:27 smithi115 bash[46934]: audit 2022-04-22T22:14:27.133783+0000 mon.a (mon.0) 13790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:28 smithi121 bash[36902]: cluster 2022-04-22T22:14:27.021063+0000 mgr.y (mgr.34107) 9835 : cluster [DBG] pgmap v10591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:28 smithi121 bash[36902]: audit 2022-04-22T22:14:27.325263+0000 mon.a (mon.0) 13791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:28 smithi115 bash[45495]: cluster 2022-04-22T22:14:27.021063+0000 mgr.y (mgr.34107) 9835 : cluster [DBG] pgmap v10591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:28 smithi115 bash[45495]: audit 2022-04-22T22:14:27.325263+0000 mon.a (mon.0) 13791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:28 smithi115 bash[46934]: cluster 2022-04-22T22:14:27.021063+0000 mgr.y (mgr.34107) 9835 : cluster [DBG] pgmap v10591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:28 smithi115 bash[46934]: audit 2022-04-22T22:14:27.325263+0000 mon.a (mon.0) 13791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:30 smithi121 bash[36902]: cluster 2022-04-22T22:14:29.022097+0000 mgr.y (mgr.34107) 9836 : cluster [DBG] pgmap v10592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:30 smithi115 bash[45495]: cluster 2022-04-22T22:14:29.022097+0000 mgr.y (mgr.34107) 9836 : cluster [DBG] pgmap v10592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:30 smithi115 bash[46934]: cluster 2022-04-22T22:14:29.022097+0000 mgr.y (mgr.34107) 9836 : cluster [DBG] pgmap v10592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:31 smithi121 bash[36902]: audit 2022-04-22T22:14:30.932162+0000 mon.a (mon.0) 13792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:31 smithi115 bash[45495]: audit 2022-04-22T22:14:30.932162+0000 mon.a (mon.0) 13792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:31 smithi115 bash[46934]: audit 2022-04-22T22:14:30.932162+0000 mon.a (mon.0) 13792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:32.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[46934]: cluster 2022-04-22T22:14:31.023171+0000 mgr.y (mgr.34107) 9837 : cluster [DBG] pgmap v10593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[46934]: audit 2022-04-22T22:14:32.137267+0000 mon.a (mon.0) 13793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:32.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[46934]: audit 2022-04-22T22:14:32.327673+0000 mon.a (mon.0) 13794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[45495]: cluster 2022-04-22T22:14:31.023171+0000 mgr.y (mgr.34107) 9837 : cluster [DBG] pgmap v10593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[45495]: audit 2022-04-22T22:14:32.137267+0000 mon.a (mon.0) 13793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:32.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[45495]: audit 2022-04-22T22:14:32.327673+0000 mon.a (mon.0) 13794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:32.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:32 smithi121 bash[36902]: cluster 2022-04-22T22:14:31.023171+0000 mgr.y (mgr.34107) 9837 : cluster [DBG] pgmap v10593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:32.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:32 smithi121 bash[36902]: audit 2022-04-22T22:14:32.137267+0000 mon.a (mon.0) 13793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:32.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:32 smithi121 bash[36902]: audit 2022-04-22T22:14:32.327673+0000 mon.a (mon.0) 13794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:32.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:33.990 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:14:34.334 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 10m ago 4h 25.9M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 10m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 10m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:14:34.335 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:14:34.336 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:14:34.336 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 10m ago 4h 270M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:14:34.336 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:14:34.336 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 10m ago 4h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 10m ago 4h 585M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:14:34.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 10m ago 4h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:14:34.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 10m ago 4h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:14:34.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 10m ago 4h 647M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:14:34.338 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 10m ago 4h 66.1M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:14:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:34 smithi115 bash[46934]: cluster 2022-04-22T22:14:33.024195+0000 mgr.y (mgr.34107) 9838 : cluster [DBG] pgmap v10594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:34.611 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:34 smithi115 bash[45495]: cluster 2022-04-22T22:14:33.024195+0000 mgr.y (mgr.34107) 9838 : cluster [DBG] pgmap v10594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:34.739 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:14:34.739 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:14:34.739 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:14:34.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:14:34.740 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:14:34.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:14:34.741 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:14:34.741 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:14:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:34 smithi121 bash[36902]: cluster 2022-04-22T22:14:33.024195+0000 mgr.y (mgr.34107) 9838 : cluster [DBG] pgmap v10594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:34.607Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:35 smithi121 bash[36902]: audit 2022-04-22T22:14:33.971131+0000 mgr.y (mgr.34107) 9839 : audit [DBG] from='client.55489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:35 smithi121 bash[36902]: audit 2022-04-22T22:14:34.311703+0000 mgr.y (mgr.34107) 9840 : audit [DBG] from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:35 smithi121 bash[36902]: audit 2022-04-22T22:14:34.735617+0000 mon.c (mon.1) 340 : audit [DBG] from='client.? 172.21.15.115:0/2990976917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[45495]: audit 2022-04-22T22:14:33.971131+0000 mgr.y (mgr.34107) 9839 : audit [DBG] from='client.55489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[45495]: audit 2022-04-22T22:14:34.311703+0000 mgr.y (mgr.34107) 9840 : audit [DBG] from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[45495]: audit 2022-04-22T22:14:34.735617+0000 mon.c (mon.1) 340 : audit [DBG] from='client.? 172.21.15.115:0/2990976917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[46934]: audit 2022-04-22T22:14:33.971131+0000 mgr.y (mgr.34107) 9839 : audit [DBG] from='client.55489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[46934]: audit 2022-04-22T22:14:34.311703+0000 mgr.y (mgr.34107) 9840 : audit [DBG] from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:14:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:35 smithi115 bash[46934]: audit 2022-04-22T22:14:34.735617+0000 mon.c (mon.1) 340 : audit [DBG] from='client.? 172.21.15.115:0/2990976917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:14:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:36 smithi121 bash[36902]: cluster 2022-04-22T22:14:35.025186+0000 mgr.y (mgr.34107) 9841 : cluster [DBG] pgmap v10595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:36 smithi121 bash[36902]: audit 2022-04-22T22:14:35.935190+0000 mon.a (mon.0) 13795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:36 smithi115 bash[45495]: cluster 2022-04-22T22:14:35.025186+0000 mgr.y (mgr.34107) 9841 : cluster [DBG] pgmap v10595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:36 smithi115 bash[45495]: audit 2022-04-22T22:14:35.935190+0000 mon.a (mon.0) 13795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:36 smithi115 bash[46934]: cluster 2022-04-22T22:14:35.025186+0000 mgr.y (mgr.34107) 9841 : cluster [DBG] pgmap v10595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:36 smithi115 bash[46934]: audit 2022-04-22T22:14:35.935190+0000 mon.a (mon.0) 13795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:37 smithi121 bash[36902]: audit 2022-04-22T22:14:37.141242+0000 mon.a (mon.0) 13796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:37 smithi121 bash[36902]: audit 2022-04-22T22:14:37.330541+0000 mon.a (mon.0) 13797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:37.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:37 smithi115 bash[46934]: audit 2022-04-22T22:14:37.141242+0000 mon.a (mon.0) 13796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:37 smithi115 bash[46934]: audit 2022-04-22T22:14:37.330541+0000 mon.a (mon.0) 13797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:37 smithi115 bash[45495]: audit 2022-04-22T22:14:37.141242+0000 mon.a (mon.0) 13796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:37 smithi115 bash[45495]: audit 2022-04-22T22:14:37.330541+0000 mon.a (mon.0) 13797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:38 smithi121 bash[36902]: cluster 2022-04-22T22:14:37.026143+0000 mgr.y (mgr.34107) 9842 : cluster [DBG] pgmap v10596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:38 smithi115 bash[46934]: cluster 2022-04-22T22:14:37.026143+0000 mgr.y (mgr.34107) 9842 : cluster [DBG] pgmap v10596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:38.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:38 smithi115 bash[45495]: cluster 2022-04-22T22:14:37.026143+0000 mgr.y (mgr.34107) 9842 : cluster [DBG] pgmap v10596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:40 smithi121 bash[36902]: cluster 2022-04-22T22:14:39.027203+0000 mgr.y (mgr.34107) 9843 : cluster [DBG] pgmap v10597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:40.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:40 smithi115 bash[46934]: cluster 2022-04-22T22:14:39.027203+0000 mgr.y (mgr.34107) 9843 : cluster [DBG] pgmap v10597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:40 smithi115 bash[45495]: cluster 2022-04-22T22:14:39.027203+0000 mgr.y (mgr.34107) 9843 : cluster [DBG] pgmap v10597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:41 smithi121 bash[36902]: audit 2022-04-22T22:14:40.938183+0000 mon.a (mon.0) 13798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:41 smithi115 bash[46934]: audit 2022-04-22T22:14:40.938183+0000 mon.a (mon.0) 13798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:41.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:41 smithi115 bash[45495]: audit 2022-04-22T22:14:40.938183+0000 mon.a (mon.0) 13798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:42.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[45495]: cluster 2022-04-22T22:14:41.028249+0000 mgr.y (mgr.34107) 9844 : cluster [DBG] pgmap v10598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:42.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[45495]: audit 2022-04-22T22:14:42.145010+0000 mon.a (mon.0) 13799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:42.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[45495]: audit 2022-04-22T22:14:42.333376+0000 mon.a (mon.0) 13800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:42.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[46934]: cluster 2022-04-22T22:14:41.028249+0000 mgr.y (mgr.34107) 9844 : cluster [DBG] pgmap v10598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:42.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[46934]: audit 2022-04-22T22:14:42.145010+0000 mon.a (mon.0) 13799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:42.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[46934]: audit 2022-04-22T22:14:42.333376+0000 mon.a (mon.0) 13800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:42 smithi121 bash[36902]: cluster 2022-04-22T22:14:41.028249+0000 mgr.y (mgr.34107) 9844 : cluster [DBG] pgmap v10598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:42 smithi121 bash[36902]: audit 2022-04-22T22:14:42.145010+0000 mon.a (mon.0) 13799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:42 smithi121 bash[36902]: audit 2022-04-22T22:14:42.333376+0000 mon.a (mon.0) 13800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:42.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:44 smithi121 bash[36902]: cluster 2022-04-22T22:14:43.028962+0000 mgr.y (mgr.34107) 9845 : cluster [DBG] pgmap v10599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:44.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:44 smithi115 bash[46934]: cluster 2022-04-22T22:14:43.028962+0000 mgr.y (mgr.34107) 9845 : cluster [DBG] pgmap v10599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:44.794 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:44.606Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:44.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:44 smithi115 bash[45495]: cluster 2022-04-22T22:14:43.028962+0000 mgr.y (mgr.34107) 9845 : cluster [DBG] pgmap v10599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:46 smithi121 bash[36902]: cluster 2022-04-22T22:14:45.030026+0000 mgr.y (mgr.34107) 9846 : cluster [DBG] pgmap v10600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:46 smithi121 bash[36902]: audit 2022-04-22T22:14:45.941519+0000 mon.a (mon.0) 13801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:46.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:46 smithi115 bash[45495]: cluster 2022-04-22T22:14:45.030026+0000 mgr.y (mgr.34107) 9846 : cluster [DBG] pgmap v10600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:46.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:46 smithi115 bash[45495]: audit 2022-04-22T22:14:45.941519+0000 mon.a (mon.0) 13801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:46.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:46 smithi115 bash[46934]: cluster 2022-04-22T22:14:45.030026+0000 mgr.y (mgr.34107) 9846 : cluster [DBG] pgmap v10600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:46.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:46 smithi115 bash[46934]: audit 2022-04-22T22:14:45.941519+0000 mon.a (mon.0) 13801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:46.453861+0000 mon.a (mon.0) 13802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:14:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:46.456876+0000 mon.a (mon.0) 13803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:14:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:46.458632+0000 mon.a (mon.0) 13804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:14:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:47.030667+0000 mon.a (mon.0) 13805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:47.148935+0000 mon.a (mon.0) 13806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:47 smithi121 bash[36902]: audit 2022-04-22T22:14:47.336220+0000 mon.a (mon.0) 13807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:46.453861+0000 mon.a (mon.0) 13802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:14:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:46.456876+0000 mon.a (mon.0) 13803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:14:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:46.458632+0000 mon.a (mon.0) 13804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:14:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:47.030667+0000 mon.a (mon.0) 13805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:47.148935+0000 mon.a (mon.0) 13806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[45495]: audit 2022-04-22T22:14:47.336220+0000 mon.a (mon.0) 13807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:46.453861+0000 mon.a (mon.0) 13802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:46.456876+0000 mon.a (mon.0) 13803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:46.458632+0000 mon.a (mon.0) 13804 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:47.030667+0000 mon.a (mon.0) 13805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:47.148935+0000 mon.a (mon.0) 13806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:47 smithi115 bash[46934]: audit 2022-04-22T22:14:47.336220+0000 mon.a (mon.0) 13807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:48 smithi121 bash[36902]: cluster 2022-04-22T22:14:47.031175+0000 mgr.y (mgr.34107) 9847 : cluster [DBG] pgmap v10601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:48 smithi115 bash[45495]: cluster 2022-04-22T22:14:47.031175+0000 mgr.y (mgr.34107) 9847 : cluster [DBG] pgmap v10601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:48 smithi115 bash[46934]: cluster 2022-04-22T22:14:47.031175+0000 mgr.y (mgr.34107) 9847 : cluster [DBG] pgmap v10601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:50 smithi121 bash[36902]: cluster 2022-04-22T22:14:49.032222+0000 mgr.y (mgr.34107) 9848 : cluster [DBG] pgmap v10602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:50 smithi115 bash[45495]: cluster 2022-04-22T22:14:49.032222+0000 mgr.y (mgr.34107) 9848 : cluster [DBG] pgmap v10602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:50 smithi115 bash[46934]: cluster 2022-04-22T22:14:49.032222+0000 mgr.y (mgr.34107) 9848 : cluster [DBG] pgmap v10602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:51 smithi121 bash[36902]: audit 2022-04-22T22:14:50.902706+0000 mon.a (mon.0) 13808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:51 smithi121 bash[36902]: audit 2022-04-22T22:14:50.944740+0000 mon.a (mon.0) 13809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:51 smithi121 bash[36902]: cluster 2022-04-22T22:14:51.033206+0000 mgr.y (mgr.34107) 9849 : cluster [DBG] pgmap v10603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:51 smithi121 bash[36902]: audit 2022-04-22T22:14:51.261916+0000 mon.a (mon.0) 13810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[45495]: audit 2022-04-22T22:14:50.902706+0000 mon.a (mon.0) 13808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[45495]: audit 2022-04-22T22:14:50.944740+0000 mon.a (mon.0) 13809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[45495]: cluster 2022-04-22T22:14:51.033206+0000 mgr.y (mgr.34107) 9849 : cluster [DBG] pgmap v10603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[45495]: audit 2022-04-22T22:14:51.261916+0000 mon.a (mon.0) 13810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[46934]: audit 2022-04-22T22:14:50.902706+0000 mon.a (mon.0) 13808 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[46934]: audit 2022-04-22T22:14:50.944740+0000 mon.a (mon.0) 13809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:52.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[46934]: cluster 2022-04-22T22:14:51.033206+0000 mgr.y (mgr.34107) 9849 : cluster [DBG] pgmap v10603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:52.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:51 smithi115 bash[46934]: audit 2022-04-22T22:14:51.261916+0000 mon.a (mon.0) 13810 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:14:53.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:52 smithi115 bash[45495]: audit 2022-04-22T22:14:52.152584+0000 mon.a (mon.0) 13811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:52 smithi115 bash[45495]: audit 2022-04-22T22:14:52.339393+0000 mon.a (mon.0) 13812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:52.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:52 smithi115 bash[46934]: audit 2022-04-22T22:14:52.152584+0000 mon.a (mon.0) 13811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:52 smithi115 bash[46934]: audit 2022-04-22T22:14:52.339393+0000 mon.a (mon.0) 13812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:52 smithi121 bash[36902]: audit 2022-04-22T22:14:52.152584+0000 mon.a (mon.0) 13811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:53.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:52 smithi121 bash[36902]: audit 2022-04-22T22:14:52.339393+0000 mon.a (mon.0) 13812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:53 smithi121 bash[36902]: cluster 2022-04-22T22:14:53.034189+0000 mgr.y (mgr.34107) 9850 : cluster [DBG] pgmap v10604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:53 smithi115 bash[45495]: cluster 2022-04-22T22:14:53.034189+0000 mgr.y (mgr.34107) 9850 : cluster [DBG] pgmap v10604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:53 smithi115 bash[46934]: cluster 2022-04-22T22:14:53.034189+0000 mgr.y (mgr.34107) 9850 : cluster [DBG] pgmap v10604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:14:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:14:54.608Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:14:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:56 smithi121 bash[36902]: cluster 2022-04-22T22:14:55.035332+0000 mgr.y (mgr.34107) 9851 : cluster [DBG] pgmap v10605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:56 smithi121 bash[36902]: audit 2022-04-22T22:14:55.947622+0000 mon.a (mon.0) 13813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:56 smithi115 bash[45495]: cluster 2022-04-22T22:14:55.035332+0000 mgr.y (mgr.34107) 9851 : cluster [DBG] pgmap v10605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:56 smithi115 bash[45495]: audit 2022-04-22T22:14:55.947622+0000 mon.a (mon.0) 13813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:56 smithi115 bash[46934]: cluster 2022-04-22T22:14:55.035332+0000 mgr.y (mgr.34107) 9851 : cluster [DBG] pgmap v10605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:56 smithi115 bash[46934]: audit 2022-04-22T22:14:55.947622+0000 mon.a (mon.0) 13813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:14:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:57 smithi115 bash[45495]: audit 2022-04-22T22:14:57.156581+0000 mon.a (mon.0) 13814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:57.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:57 smithi115 bash[46934]: audit 2022-04-22T22:14:57.156581+0000 mon.a (mon.0) 13814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:57 smithi121 bash[36902]: audit 2022-04-22T22:14:57.156581+0000 mon.a (mon.0) 13814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:14:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:58 smithi115 bash[45495]: cluster 2022-04-22T22:14:57.036437+0000 mgr.y (mgr.34107) 9852 : cluster [DBG] pgmap v10606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:14:58 smithi115 bash[45495]: audit 2022-04-22T22:14:57.342431+0000 mon.a (mon.0) 13815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:58 smithi115 bash[46934]: cluster 2022-04-22T22:14:57.036437+0000 mgr.y (mgr.34107) 9852 : cluster [DBG] pgmap v10606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:14:58 smithi115 bash[46934]: audit 2022-04-22T22:14:57.342431+0000 mon.a (mon.0) 13815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:14:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:58 smithi121 bash[36902]: cluster 2022-04-22T22:14:57.036437+0000 mgr.y (mgr.34107) 9852 : cluster [DBG] pgmap v10606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:14:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:14:58 smithi121 bash[36902]: audit 2022-04-22T22:14:57.342431+0000 mon.a (mon.0) 13815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:00 smithi115 bash[46934]: cluster 2022-04-22T22:14:59.037588+0000 mgr.y (mgr.34107) 9853 : cluster [DBG] pgmap v10607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:00 smithi115 bash[45495]: cluster 2022-04-22T22:14:59.037588+0000 mgr.y (mgr.34107) 9853 : cluster [DBG] pgmap v10607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:00 smithi121 bash[36902]: cluster 2022-04-22T22:14:59.037588+0000 mgr.y (mgr.34107) 9853 : cluster [DBG] pgmap v10607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:01.762 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:01 smithi121 bash[36902]: audit 2022-04-22T22:15:00.950819+0000 mon.a (mon.0) 13816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:01 smithi115 bash[45495]: audit 2022-04-22T22:15:00.950819+0000 mon.a (mon.0) 13816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:01 smithi115 bash[46934]: audit 2022-04-22T22:15:00.950819+0000 mon.a (mon.0) 13816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:02.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:02 smithi115 bash[46934]: cluster 2022-04-22T22:15:01.038648+0000 mgr.y (mgr.34107) 9854 : cluster [DBG] pgmap v10608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:02.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:02 smithi115 bash[46934]: audit 2022-04-22T22:15:02.160347+0000 mon.a (mon.0) 13817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:02.760 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:02.755Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:02.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:02 smithi115 bash[45495]: cluster 2022-04-22T22:15:01.038648+0000 mgr.y (mgr.34107) 9854 : cluster [DBG] pgmap v10608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:02.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:02 smithi115 bash[45495]: audit 2022-04-22T22:15:02.160347+0000 mon.a (mon.0) 13817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:02 smithi121 bash[36902]: cluster 2022-04-22T22:15:01.038648+0000 mgr.y (mgr.34107) 9854 : cluster [DBG] pgmap v10608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:02 smithi121 bash[36902]: audit 2022-04-22T22:15:02.160347+0000 mon.a (mon.0) 13817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:03 smithi121 bash[36902]: audit 2022-04-22T22:15:02.345610+0000 mon.a (mon.0) 13818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:03.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:03 smithi115 bash[45495]: audit 2022-04-22T22:15:02.345610+0000 mon.a (mon.0) 13818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:03 smithi115 bash[46934]: audit 2022-04-22T22:15:02.345610+0000 mon.a (mon.0) 13818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:04.613 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:04 smithi115 bash[45495]: cluster 2022-04-22T22:15:03.039662+0000 mgr.y (mgr.34107) 9855 : cluster [DBG] pgmap v10609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:04 smithi115 bash[46934]: cluster 2022-04-22T22:15:03.039662+0000 mgr.y (mgr.34107) 9855 : cluster [DBG] pgmap v10609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:04.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:04 smithi121 bash[36902]: cluster 2022-04-22T22:15:03.039662+0000 mgr.y (mgr.34107) 9855 : cluster [DBG] pgmap v10609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:04.609Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:05.114 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 14s ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 14s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:15:05.463 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 14s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 4h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 14s ago 4h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 14s ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:15:05.464 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 14s ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:15:05.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 14s ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:15:05.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 14s ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:15:05.465 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 14s ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:15:05.465 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 14s ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:15:05.872 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:15:05.873 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:15:05.873 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:15:05.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:05.873 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:15:05.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:15:05.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:05.874 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:15:05.874 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:15:05.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:05.875 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:15:05.875 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:15:05.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:05.875 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:15:05.876 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:15:05.876 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:15:05.876 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:15:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:06 smithi121 bash[36902]: cluster 2022-04-22T22:15:05.040658+0000 mgr.y (mgr.34107) 9856 : cluster [DBG] pgmap v10610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:06 smithi121 bash[36902]: audit 2022-04-22T22:15:05.094880+0000 mgr.y (mgr.34107) 9857 : audit [DBG] from='client.55507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:06 smithi121 bash[36902]: audit 2022-04-22T22:15:05.438888+0000 mgr.y (mgr.34107) 9858 : audit [DBG] from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:06 smithi121 bash[36902]: audit 2022-04-22T22:15:05.868640+0000 mon.a (mon.0) 13819 : audit [DBG] from='client.? 172.21.15.115:0/1378485311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:06 smithi121 bash[36902]: audit 2022-04-22T22:15:05.953857+0000 mon.a (mon.0) 13820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[45495]: cluster 2022-04-22T22:15:05.040658+0000 mgr.y (mgr.34107) 9856 : cluster [DBG] pgmap v10610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[45495]: audit 2022-04-22T22:15:05.094880+0000 mgr.y (mgr.34107) 9857 : audit [DBG] from='client.55507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[45495]: audit 2022-04-22T22:15:05.438888+0000 mgr.y (mgr.34107) 9858 : audit [DBG] from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[45495]: audit 2022-04-22T22:15:05.868640+0000 mon.a (mon.0) 13819 : audit [DBG] from='client.? 172.21.15.115:0/1378485311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[45495]: audit 2022-04-22T22:15:05.953857+0000 mon.a (mon.0) 13820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[46934]: cluster 2022-04-22T22:15:05.040658+0000 mgr.y (mgr.34107) 9856 : cluster [DBG] pgmap v10610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[46934]: audit 2022-04-22T22:15:05.094880+0000 mgr.y (mgr.34107) 9857 : audit [DBG] from='client.55507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[46934]: audit 2022-04-22T22:15:05.438888+0000 mgr.y (mgr.34107) 9858 : audit [DBG] from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[46934]: audit 2022-04-22T22:15:05.868640+0000 mon.a (mon.0) 13819 : audit [DBG] from='client.? 172.21.15.115:0/1378485311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:06.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:06 smithi115 bash[46934]: audit 2022-04-22T22:15:05.953857+0000 mon.a (mon.0) 13820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:07 smithi121 bash[36902]: audit 2022-04-22T22:15:07.164258+0000 mon.a (mon.0) 13821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:07 smithi115 bash[45495]: audit 2022-04-22T22:15:07.164258+0000 mon.a (mon.0) 13821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:07 smithi115 bash[46934]: audit 2022-04-22T22:15:07.164258+0000 mon.a (mon.0) 13821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:08 smithi121 bash[36902]: cluster 2022-04-22T22:15:07.041675+0000 mgr.y (mgr.34107) 9859 : cluster [DBG] pgmap v10611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:08.865 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:08 smithi121 bash[36902]: audit 2022-04-22T22:15:07.348712+0000 mon.a (mon.0) 13822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:08.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:08 smithi115 bash[45495]: cluster 2022-04-22T22:15:07.041675+0000 mgr.y (mgr.34107) 9859 : cluster [DBG] pgmap v10611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:08.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:08 smithi115 bash[45495]: audit 2022-04-22T22:15:07.348712+0000 mon.a (mon.0) 13822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:08.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:08 smithi115 bash[46934]: cluster 2022-04-22T22:15:07.041675+0000 mgr.y (mgr.34107) 9859 : cluster [DBG] pgmap v10611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:08.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:08 smithi115 bash[46934]: audit 2022-04-22T22:15:07.348712+0000 mon.a (mon.0) 13822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:10 smithi121 bash[36902]: cluster 2022-04-22T22:15:09.042844+0000 mgr.y (mgr.34107) 9860 : cluster [DBG] pgmap v10612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:10 smithi115 bash[45495]: cluster 2022-04-22T22:15:09.042844+0000 mgr.y (mgr.34107) 9860 : cluster [DBG] pgmap v10612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:10 smithi115 bash[46934]: cluster 2022-04-22T22:15:09.042844+0000 mgr.y (mgr.34107) 9860 : cluster [DBG] pgmap v10612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:11 smithi121 bash[36902]: audit 2022-04-22T22:15:10.957059+0000 mon.a (mon.0) 13823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:11 smithi115 bash[45495]: audit 2022-04-22T22:15:10.957059+0000 mon.a (mon.0) 13823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:11 smithi115 bash[46934]: audit 2022-04-22T22:15:10.957059+0000 mon.a (mon.0) 13823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:12.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:12 smithi115 bash[45495]: cluster 2022-04-22T22:15:11.044031+0000 mgr.y (mgr.34107) 9861 : cluster [DBG] pgmap v10613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:12.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:12 smithi115 bash[45495]: audit 2022-04-22T22:15:12.168069+0000 mon.a (mon.0) 13824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:12 smithi115 bash[46934]: cluster 2022-04-22T22:15:11.044031+0000 mgr.y (mgr.34107) 9861 : cluster [DBG] pgmap v10613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:12 smithi115 bash[46934]: audit 2022-04-22T22:15:12.168069+0000 mon.a (mon.0) 13824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:12 smithi121 bash[36902]: cluster 2022-04-22T22:15:11.044031+0000 mgr.y (mgr.34107) 9861 : cluster [DBG] pgmap v10613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:12 smithi121 bash[36902]: audit 2022-04-22T22:15:12.168069+0000 mon.a (mon.0) 13824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:12.755Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:13 smithi121 bash[36902]: audit 2022-04-22T22:15:12.351525+0000 mon.a (mon.0) 13825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:13 smithi115 bash[45495]: audit 2022-04-22T22:15:12.351525+0000 mon.a (mon.0) 13825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:13 smithi115 bash[46934]: audit 2022-04-22T22:15:12.351525+0000 mon.a (mon.0) 13825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:14.613 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:14 smithi115 bash[45495]: cluster 2022-04-22T22:15:13.045046+0000 mgr.y (mgr.34107) 9862 : cluster [DBG] pgmap v10614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:14 smithi115 bash[46934]: cluster 2022-04-22T22:15:13.045046+0000 mgr.y (mgr.34107) 9862 : cluster [DBG] pgmap v10614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:14 smithi121 bash[36902]: cluster 2022-04-22T22:15:13.045046+0000 mgr.y (mgr.34107) 9862 : cluster [DBG] pgmap v10614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:14.610Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:16 smithi121 bash[36902]: cluster 2022-04-22T22:15:15.046219+0000 mgr.y (mgr.34107) 9863 : cluster [DBG] pgmap v10615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:16 smithi121 bash[36902]: audit 2022-04-22T22:15:15.960335+0000 mon.a (mon.0) 13826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:16 smithi115 bash[45495]: cluster 2022-04-22T22:15:15.046219+0000 mgr.y (mgr.34107) 9863 : cluster [DBG] pgmap v10615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:16 smithi115 bash[45495]: audit 2022-04-22T22:15:15.960335+0000 mon.a (mon.0) 13826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:16 smithi115 bash[46934]: cluster 2022-04-22T22:15:15.046219+0000 mgr.y (mgr.34107) 9863 : cluster [DBG] pgmap v10615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:16 smithi115 bash[46934]: audit 2022-04-22T22:15:15.960335+0000 mon.a (mon.0) 13826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:17 smithi121 bash[36902]: audit 2022-04-22T22:15:16.607004+0000 mon.a (mon.0) 13827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:15:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:17 smithi121 bash[36902]: audit 2022-04-22T22:15:16.607205+0000 mon.a (mon.0) 13828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:15:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:17 smithi121 bash[36902]: audit 2022-04-22T22:15:17.171734+0000 mon.a (mon.0) 13829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:17 smithi121 bash[36902]: audit 2022-04-22T22:15:17.354400+0000 mon.a (mon.0) 13830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[46934]: audit 2022-04-22T22:15:16.607004+0000 mon.a (mon.0) 13827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[46934]: audit 2022-04-22T22:15:16.607205+0000 mon.a (mon.0) 13828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[46934]: audit 2022-04-22T22:15:17.171734+0000 mon.a (mon.0) 13829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[46934]: audit 2022-04-22T22:15:17.354400+0000 mon.a (mon.0) 13830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[45495]: audit 2022-04-22T22:15:16.607004+0000 mon.a (mon.0) 13827 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:15:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[45495]: audit 2022-04-22T22:15:16.607205+0000 mon.a (mon.0) 13828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:15:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[45495]: audit 2022-04-22T22:15:17.171734+0000 mon.a (mon.0) 13829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:17 smithi115 bash[45495]: audit 2022-04-22T22:15:17.354400+0000 mon.a (mon.0) 13830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:18 smithi121 bash[36902]: cluster 2022-04-22T22:15:17.047277+0000 mgr.y (mgr.34107) 9864 : cluster [DBG] pgmap v10616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:18 smithi115 bash[45495]: cluster 2022-04-22T22:15:17.047277+0000 mgr.y (mgr.34107) 9864 : cluster [DBG] pgmap v10616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:18 smithi115 bash[46934]: cluster 2022-04-22T22:15:17.047277+0000 mgr.y (mgr.34107) 9864 : cluster [DBG] pgmap v10616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:20 smithi121 bash[36902]: cluster 2022-04-22T22:15:19.048507+0000 mgr.y (mgr.34107) 9865 : cluster [DBG] pgmap v10617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:21.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:20 smithi115 bash[45495]: cluster 2022-04-22T22:15:19.048507+0000 mgr.y (mgr.34107) 9865 : cluster [DBG] pgmap v10617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:21.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:21 smithi121 bash[36902]: audit 2022-04-22T22:15:20.963527+0000 mon.a (mon.0) 13831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:21.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:20 smithi115 bash[46934]: cluster 2022-04-22T22:15:19.048507+0000 mgr.y (mgr.34107) 9865 : cluster [DBG] pgmap v10617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:21.837 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:21 smithi115 bash[45495]: audit 2022-04-22T22:15:20.963527+0000 mon.a (mon.0) 13831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:21.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:21 smithi115 bash[46934]: audit 2022-04-22T22:15:20.963527+0000 mon.a (mon.0) 13831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:22.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[46934]: cluster 2022-04-22T22:15:21.049665+0000 mgr.y (mgr.34107) 9866 : cluster [DBG] pgmap v10618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:22.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[46934]: audit 2022-04-22T22:15:22.175438+0000 mon.a (mon.0) 13832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:22.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[46934]: audit 2022-04-22T22:15:22.357317+0000 mon.a (mon.0) 13833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:22.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[45495]: cluster 2022-04-22T22:15:21.049665+0000 mgr.y (mgr.34107) 9866 : cluster [DBG] pgmap v10618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:22.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[45495]: audit 2022-04-22T22:15:22.175438+0000 mon.a (mon.0) 13832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:22.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[45495]: audit 2022-04-22T22:15:22.357317+0000 mon.a (mon.0) 13833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:22 smithi121 bash[36902]: cluster 2022-04-22T22:15:21.049665+0000 mgr.y (mgr.34107) 9866 : cluster [DBG] pgmap v10618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:22 smithi121 bash[36902]: audit 2022-04-22T22:15:22.175438+0000 mon.a (mon.0) 13832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:22 smithi121 bash[36902]: audit 2022-04-22T22:15:22.357317+0000 mon.a (mon.0) 13833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:22.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:24 smithi121 bash[36902]: cluster 2022-04-22T22:15:23.050812+0000 mgr.y (mgr.34107) 9867 : cluster [DBG] pgmap v10619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:24.610Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:24 smithi115 bash[45495]: cluster 2022-04-22T22:15:23.050812+0000 mgr.y (mgr.34107) 9867 : cluster [DBG] pgmap v10619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:24 smithi115 bash[46934]: cluster 2022-04-22T22:15:23.050812+0000 mgr.y (mgr.34107) 9867 : cluster [DBG] pgmap v10619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:26 smithi121 bash[36902]: cluster 2022-04-22T22:15:25.051897+0000 mgr.y (mgr.34107) 9868 : cluster [DBG] pgmap v10620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:26 smithi121 bash[36902]: audit 2022-04-22T22:15:25.966931+0000 mon.a (mon.0) 13834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:26.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:26 smithi115 bash[46934]: cluster 2022-04-22T22:15:25.051897+0000 mgr.y (mgr.34107) 9868 : cluster [DBG] pgmap v10620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:26 smithi115 bash[46934]: audit 2022-04-22T22:15:25.966931+0000 mon.a (mon.0) 13834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:26 smithi115 bash[45495]: cluster 2022-04-22T22:15:25.051897+0000 mgr.y (mgr.34107) 9868 : cluster [DBG] pgmap v10620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:26 smithi115 bash[45495]: audit 2022-04-22T22:15:25.966931+0000 mon.a (mon.0) 13834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:27 smithi121 bash[36902]: audit 2022-04-22T22:15:27.179161+0000 mon.a (mon.0) 13835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:27 smithi121 bash[36902]: audit 2022-04-22T22:15:27.360491+0000 mon.a (mon.0) 13836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:27.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:27 smithi115 bash[46934]: audit 2022-04-22T22:15:27.179161+0000 mon.a (mon.0) 13835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:27.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:27 smithi115 bash[46934]: audit 2022-04-22T22:15:27.360491+0000 mon.a (mon.0) 13836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:27 smithi115 bash[45495]: audit 2022-04-22T22:15:27.179161+0000 mon.a (mon.0) 13835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:27 smithi115 bash[45495]: audit 2022-04-22T22:15:27.360491+0000 mon.a (mon.0) 13836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:28 smithi121 bash[36902]: cluster 2022-04-22T22:15:27.052964+0000 mgr.y (mgr.34107) 9869 : cluster [DBG] pgmap v10621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:28 smithi115 bash[45495]: cluster 2022-04-22T22:15:27.052964+0000 mgr.y (mgr.34107) 9869 : cluster [DBG] pgmap v10621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:28 smithi115 bash[46934]: cluster 2022-04-22T22:15:27.052964+0000 mgr.y (mgr.34107) 9869 : cluster [DBG] pgmap v10621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:30 smithi121 bash[36902]: cluster 2022-04-22T22:15:29.054167+0000 mgr.y (mgr.34107) 9870 : cluster [DBG] pgmap v10622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:30 smithi115 bash[45495]: cluster 2022-04-22T22:15:29.054167+0000 mgr.y (mgr.34107) 9870 : cluster [DBG] pgmap v10622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:30 smithi115 bash[46934]: cluster 2022-04-22T22:15:29.054167+0000 mgr.y (mgr.34107) 9870 : cluster [DBG] pgmap v10622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:31 smithi121 bash[36902]: audit 2022-04-22T22:15:30.970226+0000 mon.a (mon.0) 13837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:31 smithi115 bash[45495]: audit 2022-04-22T22:15:30.970226+0000 mon.a (mon.0) 13837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:31 smithi115 bash[46934]: audit 2022-04-22T22:15:30.970226+0000 mon.a (mon.0) 13837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:32.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[46934]: cluster 2022-04-22T22:15:31.055216+0000 mgr.y (mgr.34107) 9871 : cluster [DBG] pgmap v10623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:32.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[46934]: audit 2022-04-22T22:15:32.183117+0000 mon.a (mon.0) 13838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:32.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[46934]: audit 2022-04-22T22:15:32.363851+0000 mon.a (mon.0) 13839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:32.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[45495]: cluster 2022-04-22T22:15:31.055216+0000 mgr.y (mgr.34107) 9871 : cluster [DBG] pgmap v10623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:32.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[45495]: audit 2022-04-22T22:15:32.183117+0000 mon.a (mon.0) 13838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:32.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[45495]: audit 2022-04-22T22:15:32.363851+0000 mon.a (mon.0) 13839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:32 smithi121 bash[36902]: cluster 2022-04-22T22:15:31.055216+0000 mgr.y (mgr.34107) 9871 : cluster [DBG] pgmap v10623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:32 smithi121 bash[36902]: audit 2022-04-22T22:15:32.183117+0000 mon.a (mon.0) 13838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:32 smithi121 bash[36902]: audit 2022-04-22T22:15:32.363851+0000 mon.a (mon.0) 13839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:32.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:34.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:34 smithi121 bash[36902]: cluster 2022-04-22T22:15:33.056312+0000 mgr.y (mgr.34107) 9872 : cluster [DBG] pgmap v10624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:34 smithi115 bash[45495]: cluster 2022-04-22T22:15:33.056312+0000 mgr.y (mgr.34107) 9872 : cluster [DBG] pgmap v10624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:34.610Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:34 smithi115 bash[46934]: cluster 2022-04-22T22:15:33.056312+0000 mgr.y (mgr.34107) 9872 : cluster [DBG] pgmap v10624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:36.239 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:15:36.593 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:15:36.593 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 10m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:15:36.593 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 45s ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 10m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 45s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 45s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 10m ago 4h 491M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 10m ago 5h 414M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 45s ago 4h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 10m ago 4h 276M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 10m ago 4h 10.8M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 45s ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:15:36.594 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 10m ago 4h 715M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 10m ago 4h 795M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 10m ago 4h 538M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 10m ago 4h 499M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 45s ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 45s ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 45s ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 45s ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:15:36.595 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 45s ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:15:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:36 smithi121 bash[36902]: cluster 2022-04-22T22:15:35.057334+0000 mgr.y (mgr.34107) 9873 : cluster [DBG] pgmap v10625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:36 smithi121 bash[36902]: audit 2022-04-22T22:15:35.973363+0000 mon.a (mon.0) 13840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:36.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:36 smithi115 bash[46934]: cluster 2022-04-22T22:15:35.057334+0000 mgr.y (mgr.34107) 9873 : cluster [DBG] pgmap v10625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:36 smithi115 bash[46934]: audit 2022-04-22T22:15:35.973363+0000 mon.a (mon.0) 13840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:36 smithi115 bash[45495]: cluster 2022-04-22T22:15:35.057334+0000 mgr.y (mgr.34107) 9873 : cluster [DBG] pgmap v10625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:36 smithi115 bash[45495]: audit 2022-04-22T22:15:35.973363+0000 mon.a (mon.0) 13840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:37.001 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:15:37.001 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:15:37.001 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:15:37.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:37.002 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:15:37.002 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:15:37.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:37.002 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:15:37.002 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:15:37.003 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:15:37.004 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:15:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:37 smithi121 bash[36902]: audit 2022-04-22T22:15:36.220767+0000 mgr.y (mgr.34107) 9874 : audit [DBG] from='client.55525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:37 smithi121 bash[36902]: audit 2022-04-22T22:15:36.997651+0000 mon.a (mon.0) 13841 : audit [DBG] from='client.? 172.21.15.115:0/3964391520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:37 smithi121 bash[36902]: audit 2022-04-22T22:15:37.186474+0000 mon.a (mon.0) 13842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:37.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:37 smithi121 bash[36902]: audit 2022-04-22T22:15:37.366799+0000 mon.a (mon.0) 13843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[45495]: audit 2022-04-22T22:15:36.220767+0000 mgr.y (mgr.34107) 9874 : audit [DBG] from='client.55525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[45495]: audit 2022-04-22T22:15:36.997651+0000 mon.a (mon.0) 13841 : audit [DBG] from='client.? 172.21.15.115:0/3964391520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[45495]: audit 2022-04-22T22:15:37.186474+0000 mon.a (mon.0) 13842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[45495]: audit 2022-04-22T22:15:37.366799+0000 mon.a (mon.0) 13843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[46934]: audit 2022-04-22T22:15:36.220767+0000 mgr.y (mgr.34107) 9874 : audit [DBG] from='client.55525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[46934]: audit 2022-04-22T22:15:36.997651+0000 mon.a (mon.0) 13841 : audit [DBG] from='client.? 172.21.15.115:0/3964391520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:15:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[46934]: audit 2022-04-22T22:15:37.186474+0000 mon.a (mon.0) 13842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:37 smithi115 bash[46934]: audit 2022-04-22T22:15:37.366799+0000 mon.a (mon.0) 13843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:38 smithi121 bash[36902]: audit 2022-04-22T22:15:36.571312+0000 mgr.y (mgr.34107) 9875 : audit [DBG] from='client.45600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:38 smithi121 bash[36902]: cluster 2022-04-22T22:15:37.058483+0000 mgr.y (mgr.34107) 9876 : cluster [DBG] pgmap v10626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:38 smithi115 bash[45495]: audit 2022-04-22T22:15:36.571312+0000 mgr.y (mgr.34107) 9875 : audit [DBG] from='client.45600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:38 smithi115 bash[45495]: cluster 2022-04-22T22:15:37.058483+0000 mgr.y (mgr.34107) 9876 : cluster [DBG] pgmap v10626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:38 smithi115 bash[46934]: audit 2022-04-22T22:15:36.571312+0000 mgr.y (mgr.34107) 9875 : audit [DBG] from='client.45600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:15:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:38 smithi115 bash[46934]: cluster 2022-04-22T22:15:37.058483+0000 mgr.y (mgr.34107) 9876 : cluster [DBG] pgmap v10626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:40 smithi121 bash[36902]: cluster 2022-04-22T22:15:39.059722+0000 mgr.y (mgr.34107) 9877 : cluster [DBG] pgmap v10627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:40.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:40 smithi115 bash[45495]: cluster 2022-04-22T22:15:39.059722+0000 mgr.y (mgr.34107) 9877 : cluster [DBG] pgmap v10627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:40.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:40 smithi115 bash[46934]: cluster 2022-04-22T22:15:39.059722+0000 mgr.y (mgr.34107) 9877 : cluster [DBG] pgmap v10627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:41.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:41 smithi121 bash[36902]: audit 2022-04-22T22:15:40.976081+0000 mon.a (mon.0) 13844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:41 smithi115 bash[46934]: audit 2022-04-22T22:15:40.976081+0000 mon.a (mon.0) 13844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:41 smithi115 bash[45495]: audit 2022-04-22T22:15:40.976081+0000 mon.a (mon.0) 13844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:42.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[45495]: cluster 2022-04-22T22:15:41.060712+0000 mgr.y (mgr.34107) 9878 : cluster [DBG] pgmap v10628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:42.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[45495]: audit 2022-04-22T22:15:42.190069+0000 mon.a (mon.0) 13845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:42.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[45495]: audit 2022-04-22T22:15:42.369851+0000 mon.a (mon.0) 13846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:42.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[46934]: cluster 2022-04-22T22:15:41.060712+0000 mgr.y (mgr.34107) 9878 : cluster [DBG] pgmap v10628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:42.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[46934]: audit 2022-04-22T22:15:42.190069+0000 mon.a (mon.0) 13845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:42.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[46934]: audit 2022-04-22T22:15:42.369851+0000 mon.a (mon.0) 13846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:42 smithi121 bash[36902]: cluster 2022-04-22T22:15:41.060712+0000 mgr.y (mgr.34107) 9878 : cluster [DBG] pgmap v10628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:42 smithi121 bash[36902]: audit 2022-04-22T22:15:42.190069+0000 mon.a (mon.0) 13845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:42 smithi121 bash[36902]: audit 2022-04-22T22:15:42.369851+0000 mon.a (mon.0) 13846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:42.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:44 smithi121 bash[36902]: cluster 2022-04-22T22:15:43.061784+0000 mgr.y (mgr.34107) 9879 : cluster [DBG] pgmap v10629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:44 smithi115 bash[45495]: cluster 2022-04-22T22:15:43.061784+0000 mgr.y (mgr.34107) 9879 : cluster [DBG] pgmap v10629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:44.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:44 smithi115 bash[46934]: cluster 2022-04-22T22:15:43.061784+0000 mgr.y (mgr.34107) 9879 : cluster [DBG] pgmap v10629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:44.799 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:44.611Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:46 smithi121 bash[36902]: cluster 2022-04-22T22:15:45.062843+0000 mgr.y (mgr.34107) 9880 : cluster [DBG] pgmap v10630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:46 smithi121 bash[36902]: audit 2022-04-22T22:15:45.979542+0000 mon.a (mon.0) 13847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:46 smithi115 bash[45495]: cluster 2022-04-22T22:15:45.062843+0000 mgr.y (mgr.34107) 9880 : cluster [DBG] pgmap v10630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:46 smithi115 bash[45495]: audit 2022-04-22T22:15:45.979542+0000 mon.a (mon.0) 13847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:46 smithi115 bash[46934]: cluster 2022-04-22T22:15:45.062843+0000 mgr.y (mgr.34107) 9880 : cluster [DBG] pgmap v10630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:46 smithi115 bash[46934]: audit 2022-04-22T22:15:45.979542+0000 mon.a (mon.0) 13847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:47 smithi121 bash[36902]: audit 2022-04-22T22:15:47.193762+0000 mon.a (mon.0) 13848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:47 smithi121 bash[36902]: audit 2022-04-22T22:15:47.372793+0000 mon.a (mon.0) 13849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:47 smithi115 bash[45495]: audit 2022-04-22T22:15:47.193762+0000 mon.a (mon.0) 13848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:47 smithi115 bash[45495]: audit 2022-04-22T22:15:47.372793+0000 mon.a (mon.0) 13849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:47 smithi115 bash[46934]: audit 2022-04-22T22:15:47.193762+0000 mon.a (mon.0) 13848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:47 smithi115 bash[46934]: audit 2022-04-22T22:15:47.372793+0000 mon.a (mon.0) 13849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:48 smithi121 bash[36902]: cluster 2022-04-22T22:15:47.063960+0000 mgr.y (mgr.34107) 9881 : cluster [DBG] pgmap v10631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:48 smithi115 bash[45495]: cluster 2022-04-22T22:15:47.063960+0000 mgr.y (mgr.34107) 9881 : cluster [DBG] pgmap v10631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:48 smithi115 bash[46934]: cluster 2022-04-22T22:15:47.063960+0000 mgr.y (mgr.34107) 9881 : cluster [DBG] pgmap v10631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:50 smithi121 bash[36902]: cluster 2022-04-22T22:15:49.065144+0000 mgr.y (mgr.34107) 9882 : cluster [DBG] pgmap v10632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:50 smithi115 bash[45495]: cluster 2022-04-22T22:15:49.065144+0000 mgr.y (mgr.34107) 9882 : cluster [DBG] pgmap v10632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:50 smithi115 bash[46934]: cluster 2022-04-22T22:15:49.065144+0000 mgr.y (mgr.34107) 9882 : cluster [DBG] pgmap v10632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:51.612 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:51 smithi121 bash[36902]: audit 2022-04-22T22:15:50.982710+0000 mon.a (mon.0) 13850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:51.920 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:51 smithi121 bash[36902]: audit 2022-04-22T22:15:51.271882+0000 mon.a (mon.0) 13851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:15:51.925 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:51 smithi121 bash[36902]: audit 2022-04-22T22:15:51.274573+0000 mon.a (mon.0) 13852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:51 smithi121 bash[36902]: audit 2022-04-22T22:15:51.276236+0000 mon.a (mon.0) 13853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[46934]: audit 2022-04-22T22:15:50.982710+0000 mon.a (mon.0) 13850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[46934]: audit 2022-04-22T22:15:51.271882+0000 mon.a (mon.0) 13851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[46934]: audit 2022-04-22T22:15:51.274573+0000 mon.a (mon.0) 13852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[46934]: audit 2022-04-22T22:15:51.276236+0000 mon.a (mon.0) 13853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:15:51.926 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[45495]: audit 2022-04-22T22:15:50.982710+0000 mon.a (mon.0) 13850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:51.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[45495]: audit 2022-04-22T22:15:51.271882+0000 mon.a (mon.0) 13851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:15:51.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[45495]: audit 2022-04-22T22:15:51.274573+0000 mon.a (mon.0) 13852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:15:51.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:51 smithi115 bash[45495]: audit 2022-04-22T22:15:51.276236+0000 mon.a (mon.0) 13853 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:15:52.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[45495]: cluster 2022-04-22T22:15:51.066216+0000 mgr.y (mgr.34107) 9883 : cluster [DBG] pgmap v10633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[45495]: audit 2022-04-22T22:15:51.626080+0000 mon.a (mon.0) 13854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[45495]: audit 2022-04-22T22:15:51.633268+0000 mon.a (mon.0) 13855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[45495]: audit 2022-04-22T22:15:52.197379+0000 mon.a (mon.0) 13856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[45495]: audit 2022-04-22T22:15:52.375919+0000 mon.a (mon.0) 13857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[46934]: cluster 2022-04-22T22:15:51.066216+0000 mgr.y (mgr.34107) 9883 : cluster [DBG] pgmap v10633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[46934]: audit 2022-04-22T22:15:51.626080+0000 mon.a (mon.0) 13854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[46934]: audit 2022-04-22T22:15:51.633268+0000 mon.a (mon.0) 13855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[46934]: audit 2022-04-22T22:15:52.197379+0000 mon.a (mon.0) 13856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:52.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[46934]: audit 2022-04-22T22:15:52.375919+0000 mon.a (mon.0) 13857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:52 smithi121 bash[36902]: cluster 2022-04-22T22:15:51.066216+0000 mgr.y (mgr.34107) 9883 : cluster [DBG] pgmap v10633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:52 smithi121 bash[36902]: audit 2022-04-22T22:15:51.626080+0000 mon.a (mon.0) 13854 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:52 smithi121 bash[36902]: audit 2022-04-22T22:15:51.633268+0000 mon.a (mon.0) 13855 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:52 smithi121 bash[36902]: audit 2022-04-22T22:15:52.197379+0000 mon.a (mon.0) 13856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:52 smithi121 bash[36902]: audit 2022-04-22T22:15:52.375919+0000 mon.a (mon.0) 13857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:52.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:54 smithi121 bash[36902]: cluster 2022-04-22T22:15:53.067350+0000 mgr.y (mgr.34107) 9884 : cluster [DBG] pgmap v10634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:54 smithi115 bash[45495]: cluster 2022-04-22T22:15:53.067350+0000 mgr.y (mgr.34107) 9884 : cluster [DBG] pgmap v10634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:54.792 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:15:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:15:54.612Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:15:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:54 smithi115 bash[46934]: cluster 2022-04-22T22:15:53.067350+0000 mgr.y (mgr.34107) 9884 : cluster [DBG] pgmap v10634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:56 smithi121 bash[36902]: cluster 2022-04-22T22:15:55.068406+0000 mgr.y (mgr.34107) 9885 : cluster [DBG] pgmap v10635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:56 smithi121 bash[36902]: audit 2022-04-22T22:15:55.755443+0000 mon.a (mon.0) 13858 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:56 smithi121 bash[36902]: audit 2022-04-22T22:15:55.986071+0000 mon.a (mon.0) 13859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:56.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[46934]: cluster 2022-04-22T22:15:55.068406+0000 mgr.y (mgr.34107) 9885 : cluster [DBG] pgmap v10635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[46934]: audit 2022-04-22T22:15:55.755443+0000 mon.a (mon.0) 13858 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[46934]: audit 2022-04-22T22:15:55.986071+0000 mon.a (mon.0) 13859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[45495]: cluster 2022-04-22T22:15:55.068406+0000 mgr.y (mgr.34107) 9885 : cluster [DBG] pgmap v10635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[45495]: audit 2022-04-22T22:15:55.755443+0000 mon.a (mon.0) 13858 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:15:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:56 smithi115 bash[45495]: audit 2022-04-22T22:15:55.986071+0000 mon.a (mon.0) 13859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:15:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:57 smithi115 bash[45495]: audit 2022-04-22T22:15:57.200986+0000 mon.a (mon.0) 13860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:57 smithi115 bash[45495]: audit 2022-04-22T22:15:57.379071+0000 mon.a (mon.0) 13861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:57 smithi115 bash[46934]: audit 2022-04-22T22:15:57.200986+0000 mon.a (mon.0) 13860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:57 smithi115 bash[46934]: audit 2022-04-22T22:15:57.379071+0000 mon.a (mon.0) 13861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:57 smithi121 bash[36902]: audit 2022-04-22T22:15:57.200986+0000 mon.a (mon.0) 13860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:15:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:57 smithi121 bash[36902]: audit 2022-04-22T22:15:57.379071+0000 mon.a (mon.0) 13861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:15:58.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:15:58 smithi115 bash[46934]: cluster 2022-04-22T22:15:57.069464+0000 mgr.y (mgr.34107) 9886 : cluster [DBG] pgmap v10636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:15:58 smithi115 bash[45495]: cluster 2022-04-22T22:15:57.069464+0000 mgr.y (mgr.34107) 9886 : cluster [DBG] pgmap v10636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:15:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:15:58 smithi121 bash[36902]: cluster 2022-04-22T22:15:57.069464+0000 mgr.y (mgr.34107) 9886 : cluster [DBG] pgmap v10636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:00 smithi115 bash[45495]: cluster 2022-04-22T22:15:59.070639+0000 mgr.y (mgr.34107) 9887 : cluster [DBG] pgmap v10637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:00 smithi115 bash[46934]: cluster 2022-04-22T22:15:59.070639+0000 mgr.y (mgr.34107) 9887 : cluster [DBG] pgmap v10637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:00 smithi121 bash[36902]: cluster 2022-04-22T22:15:59.070639+0000 mgr.y (mgr.34107) 9887 : cluster [DBG] pgmap v10637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:01 smithi115 bash[45495]: audit 2022-04-22T22:16:00.988752+0000 mon.a (mon.0) 13862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:01 smithi115 bash[46934]: audit 2022-04-22T22:16:00.988752+0000 mon.a (mon.0) 13862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:01 smithi121 bash[36902]: audit 2022-04-22T22:16:00.988752+0000 mon.a (mon.0) 13862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:02.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[45495]: cluster 2022-04-22T22:16:01.071733+0000 mgr.y (mgr.34107) 9888 : cluster [DBG] pgmap v10638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:02.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[45495]: audit 2022-04-22T22:16:02.204815+0000 mon.a (mon.0) 13863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:02.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[45495]: audit 2022-04-22T22:16:02.382341+0000 mon.a (mon.0) 13864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:02.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[46934]: cluster 2022-04-22T22:16:01.071733+0000 mgr.y (mgr.34107) 9888 : cluster [DBG] pgmap v10638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:02.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[46934]: audit 2022-04-22T22:16:02.204815+0000 mon.a (mon.0) 13863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:02.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[46934]: audit 2022-04-22T22:16:02.382341+0000 mon.a (mon.0) 13864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:02 smithi121 bash[36902]: cluster 2022-04-22T22:16:01.071733+0000 mgr.y (mgr.34107) 9888 : cluster [DBG] pgmap v10638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:02 smithi121 bash[36902]: audit 2022-04-22T22:16:02.204815+0000 mon.a (mon.0) 13863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:02 smithi121 bash[36902]: audit 2022-04-22T22:16:02.382341+0000 mon.a (mon.0) 13864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:02.757Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:03.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:03 smithi115 bash[45495]: cluster 2022-04-22T22:16:03.072785+0000 mgr.y (mgr.34107) 9889 : cluster [DBG] pgmap v10639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:03 smithi115 bash[46934]: cluster 2022-04-22T22:16:03.072785+0000 mgr.y (mgr.34107) 9889 : cluster [DBG] pgmap v10639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:03 smithi121 bash[36902]: cluster 2022-04-22T22:16:03.072785+0000 mgr.y (mgr.34107) 9889 : cluster [DBG] pgmap v10639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:04.612Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[45495]: cephadm 2022-04-22T22:16:04.768320+0000 mgr.y (mgr.34107) 9890 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:16:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[45495]: audit 2022-04-22T22:16:04.777138+0000 mon.a (mon.0) 13865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[45495]: audit 2022-04-22T22:16:04.785661+0000 mon.a (mon.0) 13866 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[45495]: cluster 2022-04-22T22:16:05.073875+0000 mgr.y (mgr.34107) 9891 : cluster [DBG] pgmap v10640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[46934]: cephadm 2022-04-22T22:16:04.768320+0000 mgr.y (mgr.34107) 9890 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:16:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[46934]: audit 2022-04-22T22:16:04.777138+0000 mon.a (mon.0) 13865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[46934]: audit 2022-04-22T22:16:04.785661+0000 mon.a (mon.0) 13866 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:05 smithi115 bash[46934]: cluster 2022-04-22T22:16:05.073875+0000 mgr.y (mgr.34107) 9891 : cluster [DBG] pgmap v10640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:05 smithi121 bash[36902]: cephadm 2022-04-22T22:16:04.768320+0000 mgr.y (mgr.34107) 9890 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:16:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:05 smithi121 bash[36902]: audit 2022-04-22T22:16:04.777138+0000 mon.a (mon.0) 13865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:05 smithi121 bash[36902]: audit 2022-04-22T22:16:04.785661+0000 mon.a (mon.0) 13866 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:16:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:05 smithi121 bash[36902]: cluster 2022-04-22T22:16:05.073875+0000 mgr.y (mgr.34107) 9891 : cluster [DBG] pgmap v10640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:06 smithi115 bash[46934]: audit 2022-04-22T22:16:05.991855+0000 mon.a (mon.0) 13867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:06 smithi115 bash[45495]: audit 2022-04-22T22:16:05.991855+0000 mon.a (mon.0) 13867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:06 smithi121 bash[36902]: audit 2022-04-22T22:16:05.991855+0000 mon.a (mon.0) 13867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:07.382 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:16:07.722 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:16:07.722 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 11s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:16:07.722 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 76s ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:16:07.723 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 11s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:16:07.724 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 76s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:16:07.724 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 76s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:16:07.724 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 11s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:16:07.724 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 11s ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:16:07.724 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 76s ago 4h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 11s ago 4h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 11s ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 76s ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 11s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 11s ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 11s ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 11s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:16:07.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 76s ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:16:07.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 76s ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:16:07.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 76s ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:16:07.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 76s ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:16:07.726 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 76s ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:16:08.131 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:08.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:16:08.133 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:16:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:07 smithi121 bash[36902]: cluster 2022-04-22T22:16:07.074942+0000 mgr.y (mgr.34107) 9892 : cluster [DBG] pgmap v10641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:07 smithi121 bash[36902]: audit 2022-04-22T22:16:07.208453+0000 mon.a (mon.0) 13868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:07 smithi121 bash[36902]: audit 2022-04-22T22:16:07.361297+0000 mgr.y (mgr.34107) 9893 : audit [DBG] from='client.55543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:07 smithi121 bash[36902]: audit 2022-04-22T22:16:07.385513+0000 mon.a (mon.0) 13869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[45495]: cluster 2022-04-22T22:16:07.074942+0000 mgr.y (mgr.34107) 9892 : cluster [DBG] pgmap v10641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[45495]: audit 2022-04-22T22:16:07.208453+0000 mon.a (mon.0) 13868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[45495]: audit 2022-04-22T22:16:07.361297+0000 mgr.y (mgr.34107) 9893 : audit [DBG] from='client.55543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[45495]: audit 2022-04-22T22:16:07.385513+0000 mon.a (mon.0) 13869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[46934]: cluster 2022-04-22T22:16:07.074942+0000 mgr.y (mgr.34107) 9892 : cluster [DBG] pgmap v10641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[46934]: audit 2022-04-22T22:16:07.208453+0000 mon.a (mon.0) 13868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[46934]: audit 2022-04-22T22:16:07.361297+0000 mgr.y (mgr.34107) 9893 : audit [DBG] from='client.55543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:07 smithi115 bash[46934]: audit 2022-04-22T22:16:07.385513+0000 mon.a (mon.0) 13869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:08 smithi121 bash[36902]: audit 2022-04-22T22:16:07.704391+0000 mgr.y (mgr.34107) 9894 : audit [DBG] from='client.45618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:08 smithi121 bash[36902]: audit 2022-04-22T22:16:08.128051+0000 mon.c (mon.1) 341 : audit [DBG] from='client.? 172.21.15.115:0/3280251520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:08 smithi115 bash[45495]: audit 2022-04-22T22:16:07.704391+0000 mgr.y (mgr.34107) 9894 : audit [DBG] from='client.45618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:08 smithi115 bash[45495]: audit 2022-04-22T22:16:08.128051+0000 mon.c (mon.1) 341 : audit [DBG] from='client.? 172.21.15.115:0/3280251520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:08 smithi115 bash[46934]: audit 2022-04-22T22:16:07.704391+0000 mgr.y (mgr.34107) 9894 : audit [DBG] from='client.45618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:08 smithi115 bash[46934]: audit 2022-04-22T22:16:08.128051+0000 mon.c (mon.1) 341 : audit [DBG] from='client.? 172.21.15.115:0/3280251520' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:09 smithi121 bash[36902]: cluster 2022-04-22T22:16:09.076133+0000 mgr.y (mgr.34107) 9895 : cluster [DBG] pgmap v10642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:09 smithi115 bash[45495]: cluster 2022-04-22T22:16:09.076133+0000 mgr.y (mgr.34107) 9895 : cluster [DBG] pgmap v10642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:09 smithi115 bash[46934]: cluster 2022-04-22T22:16:09.076133+0000 mgr.y (mgr.34107) 9895 : cluster [DBG] pgmap v10642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:11 smithi121 bash[36902]: audit 2022-04-22T22:16:10.996857+0000 mon.a (mon.0) 13870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:11 smithi115 bash[45495]: audit 2022-04-22T22:16:10.996857+0000 mon.a (mon.0) 13870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:11 smithi115 bash[46934]: audit 2022-04-22T22:16:10.996857+0000 mon.a (mon.0) 13870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:12.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:12 smithi115 bash[45495]: cluster 2022-04-22T22:16:11.077185+0000 mgr.y (mgr.34107) 9896 : cluster [DBG] pgmap v10643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:12.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:12 smithi115 bash[45495]: audit 2022-04-22T22:16:12.211727+0000 mon.a (mon.0) 13871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:12 smithi115 bash[46934]: cluster 2022-04-22T22:16:11.077185+0000 mgr.y (mgr.34107) 9896 : cluster [DBG] pgmap v10643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:12.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:12 smithi115 bash[46934]: audit 2022-04-22T22:16:12.211727+0000 mon.a (mon.0) 13871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:12 smithi121 bash[36902]: cluster 2022-04-22T22:16:11.077185+0000 mgr.y (mgr.34107) 9896 : cluster [DBG] pgmap v10643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:12 smithi121 bash[36902]: audit 2022-04-22T22:16:12.211727+0000 mon.a (mon.0) 13871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:12.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:13 smithi121 bash[36902]: audit 2022-04-22T22:16:12.388297+0000 mon.a (mon.0) 13872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:13 smithi115 bash[46934]: audit 2022-04-22T22:16:12.388297+0000 mon.a (mon.0) 13872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:13 smithi115 bash[45495]: audit 2022-04-22T22:16:12.388297+0000 mon.a (mon.0) 13872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:14.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:14 smithi115 bash[45495]: cluster 2022-04-22T22:16:13.078255+0000 mgr.y (mgr.34107) 9897 : cluster [DBG] pgmap v10644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:14 smithi115 bash[46934]: cluster 2022-04-22T22:16:13.078255+0000 mgr.y (mgr.34107) 9897 : cluster [DBG] pgmap v10644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:14 smithi121 bash[36902]: cluster 2022-04-22T22:16:13.078255+0000 mgr.y (mgr.34107) 9897 : cluster [DBG] pgmap v10644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:14.613Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:16 smithi121 bash[36902]: cluster 2022-04-22T22:16:15.079311+0000 mgr.y (mgr.34107) 9898 : cluster [DBG] pgmap v10645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:16 smithi121 bash[36902]: audit 2022-04-22T22:16:16.000290+0000 mon.a (mon.0) 13873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:16 smithi115 bash[45495]: cluster 2022-04-22T22:16:15.079311+0000 mgr.y (mgr.34107) 9898 : cluster [DBG] pgmap v10645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:16 smithi115 bash[45495]: audit 2022-04-22T22:16:16.000290+0000 mon.a (mon.0) 13873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:16 smithi115 bash[46934]: cluster 2022-04-22T22:16:15.079311+0000 mgr.y (mgr.34107) 9898 : cluster [DBG] pgmap v10645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:16 smithi115 bash[46934]: audit 2022-04-22T22:16:16.000290+0000 mon.a (mon.0) 13873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:17 smithi121 bash[36902]: audit 2022-04-22T22:16:16.618921+0000 mon.a (mon.0) 13874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:16:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:17 smithi121 bash[36902]: audit 2022-04-22T22:16:16.619357+0000 mon.a (mon.0) 13875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:16:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:17 smithi121 bash[36902]: audit 2022-04-22T22:16:17.215271+0000 mon.a (mon.0) 13876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[45495]: audit 2022-04-22T22:16:16.618921+0000 mon.a (mon.0) 13874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:16:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[45495]: audit 2022-04-22T22:16:16.619357+0000 mon.a (mon.0) 13875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:16:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[45495]: audit 2022-04-22T22:16:17.215271+0000 mon.a (mon.0) 13876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[46934]: audit 2022-04-22T22:16:16.618921+0000 mon.a (mon.0) 13874 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:16:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[46934]: audit 2022-04-22T22:16:16.619357+0000 mon.a (mon.0) 13875 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:16:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:17 smithi115 bash[46934]: audit 2022-04-22T22:16:17.215271+0000 mon.a (mon.0) 13876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:18 smithi121 bash[36902]: cluster 2022-04-22T22:16:17.080555+0000 mgr.y (mgr.34107) 9899 : cluster [DBG] pgmap v10646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:18 smithi121 bash[36902]: audit 2022-04-22T22:16:17.391290+0000 mon.a (mon.0) 13877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:18 smithi115 bash[45495]: cluster 2022-04-22T22:16:17.080555+0000 mgr.y (mgr.34107) 9899 : cluster [DBG] pgmap v10646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:18 smithi115 bash[45495]: audit 2022-04-22T22:16:17.391290+0000 mon.a (mon.0) 13877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:18 smithi115 bash[46934]: cluster 2022-04-22T22:16:17.080555+0000 mgr.y (mgr.34107) 9899 : cluster [DBG] pgmap v10646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:18 smithi115 bash[46934]: audit 2022-04-22T22:16:17.391290+0000 mon.a (mon.0) 13877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:20 smithi121 bash[36902]: cluster 2022-04-22T22:16:19.081672+0000 mgr.y (mgr.34107) 9900 : cluster [DBG] pgmap v10647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:20.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:20 smithi115 bash[45495]: cluster 2022-04-22T22:16:19.081672+0000 mgr.y (mgr.34107) 9900 : cluster [DBG] pgmap v10647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:20 smithi115 bash[46934]: cluster 2022-04-22T22:16:19.081672+0000 mgr.y (mgr.34107) 9900 : cluster [DBG] pgmap v10647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:21.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:21 smithi121 bash[36902]: audit 2022-04-22T22:16:21.003126+0000 mon.a (mon.0) 13878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:21 smithi115 bash[45495]: audit 2022-04-22T22:16:21.003126+0000 mon.a (mon.0) 13878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:21 smithi115 bash[46934]: audit 2022-04-22T22:16:21.003126+0000 mon.a (mon.0) 13878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:22.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:22 smithi115 bash[45495]: cluster 2022-04-22T22:16:21.082810+0000 mgr.y (mgr.34107) 9901 : cluster [DBG] pgmap v10648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:22.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:22 smithi115 bash[45495]: audit 2022-04-22T22:16:22.218815+0000 mon.a (mon.0) 13879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:22.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:22 smithi115 bash[46934]: cluster 2022-04-22T22:16:21.082810+0000 mgr.y (mgr.34107) 9901 : cluster [DBG] pgmap v10648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:22.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:22 smithi115 bash[46934]: audit 2022-04-22T22:16:22.218815+0000 mon.a (mon.0) 13879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:22.900 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:22 smithi121 bash[36902]: cluster 2022-04-22T22:16:21.082810+0000 mgr.y (mgr.34107) 9901 : cluster [DBG] pgmap v10648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:22.901 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:22 smithi121 bash[36902]: audit 2022-04-22T22:16:22.218815+0000 mon.a (mon.0) 13879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:23.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:22.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:23 smithi121 bash[36902]: audit 2022-04-22T22:16:22.394482+0000 mon.a (mon.0) 13880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:23 smithi115 bash[46934]: audit 2022-04-22T22:16:22.394482+0000 mon.a (mon.0) 13880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:23 smithi115 bash[45495]: audit 2022-04-22T22:16:22.394482+0000 mon.a (mon.0) 13880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:24 smithi115 bash[46934]: cluster 2022-04-22T22:16:23.083893+0000 mgr.y (mgr.34107) 9902 : cluster [DBG] pgmap v10649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:24.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:24 smithi115 bash[45495]: cluster 2022-04-22T22:16:23.083893+0000 mgr.y (mgr.34107) 9902 : cluster [DBG] pgmap v10649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:24 smithi121 bash[36902]: cluster 2022-04-22T22:16:23.083893+0000 mgr.y (mgr.34107) 9902 : cluster [DBG] pgmap v10649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:24.614Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:26 smithi121 bash[36902]: cluster 2022-04-22T22:16:25.084894+0000 mgr.y (mgr.34107) 9903 : cluster [DBG] pgmap v10650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:26 smithi121 bash[36902]: audit 2022-04-22T22:16:26.005900+0000 mon.a (mon.0) 13881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:26.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:26 smithi115 bash[45495]: cluster 2022-04-22T22:16:25.084894+0000 mgr.y (mgr.34107) 9903 : cluster [DBG] pgmap v10650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:26.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:26 smithi115 bash[45495]: audit 2022-04-22T22:16:26.005900+0000 mon.a (mon.0) 13881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:26 smithi115 bash[46934]: cluster 2022-04-22T22:16:25.084894+0000 mgr.y (mgr.34107) 9903 : cluster [DBG] pgmap v10650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:26.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:26 smithi115 bash[46934]: audit 2022-04-22T22:16:26.005900+0000 mon.a (mon.0) 13881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:27 smithi121 bash[36902]: audit 2022-04-22T22:16:27.222626+0000 mon.a (mon.0) 13882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:27 smithi115 bash[45495]: audit 2022-04-22T22:16:27.222626+0000 mon.a (mon.0) 13882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:27.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:27 smithi115 bash[46934]: audit 2022-04-22T22:16:27.222626+0000 mon.a (mon.0) 13882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:28 smithi121 bash[36902]: cluster 2022-04-22T22:16:27.085966+0000 mgr.y (mgr.34107) 9904 : cluster [DBG] pgmap v10651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:28 smithi121 bash[36902]: audit 2022-04-22T22:16:27.397489+0000 mon.a (mon.0) 13883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:28 smithi115 bash[45495]: cluster 2022-04-22T22:16:27.085966+0000 mgr.y (mgr.34107) 9904 : cluster [DBG] pgmap v10651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:28 smithi115 bash[45495]: audit 2022-04-22T22:16:27.397489+0000 mon.a (mon.0) 13883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:28 smithi115 bash[46934]: cluster 2022-04-22T22:16:27.085966+0000 mgr.y (mgr.34107) 9904 : cluster [DBG] pgmap v10651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:28 smithi115 bash[46934]: audit 2022-04-22T22:16:27.397489+0000 mon.a (mon.0) 13883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:30 smithi121 bash[36902]: cluster 2022-04-22T22:16:29.086949+0000 mgr.y (mgr.34107) 9905 : cluster [DBG] pgmap v10652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:30 smithi115 bash[45495]: cluster 2022-04-22T22:16:29.086949+0000 mgr.y (mgr.34107) 9905 : cluster [DBG] pgmap v10652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:30 smithi115 bash[46934]: cluster 2022-04-22T22:16:29.086949+0000 mgr.y (mgr.34107) 9905 : cluster [DBG] pgmap v10652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:31 smithi121 bash[36902]: audit 2022-04-22T22:16:31.008314+0000 mon.a (mon.0) 13884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:31 smithi115 bash[45495]: audit 2022-04-22T22:16:31.008314+0000 mon.a (mon.0) 13884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:31 smithi115 bash[46934]: audit 2022-04-22T22:16:31.008314+0000 mon.a (mon.0) 13884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:32.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:32 smithi115 bash[45495]: cluster 2022-04-22T22:16:31.088032+0000 mgr.y (mgr.34107) 9906 : cluster [DBG] pgmap v10653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:32.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:32 smithi115 bash[45495]: audit 2022-04-22T22:16:32.226282+0000 mon.a (mon.0) 13885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:32.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:32 smithi115 bash[46934]: cluster 2022-04-22T22:16:31.088032+0000 mgr.y (mgr.34107) 9906 : cluster [DBG] pgmap v10653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:32.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:32 smithi115 bash[46934]: audit 2022-04-22T22:16:32.226282+0000 mon.a (mon.0) 13885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:32 smithi121 bash[36902]: cluster 2022-04-22T22:16:31.088032+0000 mgr.y (mgr.34107) 9906 : cluster [DBG] pgmap v10653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:32 smithi121 bash[36902]: audit 2022-04-22T22:16:32.226282+0000 mon.a (mon.0) 13885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:32.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:33 smithi121 bash[36902]: audit 2022-04-22T22:16:32.400269+0000 mon.a (mon.0) 13886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:33 smithi115 bash[45495]: audit 2022-04-22T22:16:32.400269+0000 mon.a (mon.0) 13886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:33 smithi115 bash[46934]: audit 2022-04-22T22:16:32.400269+0000 mon.a (mon.0) 13886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:34 smithi121 bash[36902]: cluster 2022-04-22T22:16:33.089122+0000 mgr.y (mgr.34107) 9907 : cluster [DBG] pgmap v10654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:34 smithi115 bash[45495]: cluster 2022-04-22T22:16:33.089122+0000 mgr.y (mgr.34107) 9907 : cluster [DBG] pgmap v10654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:34.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:34 smithi115 bash[46934]: cluster 2022-04-22T22:16:33.089122+0000 mgr.y (mgr.34107) 9907 : cluster [DBG] pgmap v10654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:36 smithi121 bash[36902]: cluster 2022-04-22T22:16:35.090275+0000 mgr.y (mgr.34107) 9908 : cluster [DBG] pgmap v10655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:36 smithi121 bash[36902]: audit 2022-04-22T22:16:36.011560+0000 mon.a (mon.0) 13887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:36 smithi115 bash[45495]: cluster 2022-04-22T22:16:35.090275+0000 mgr.y (mgr.34107) 9908 : cluster [DBG] pgmap v10655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:36 smithi115 bash[45495]: audit 2022-04-22T22:16:36.011560+0000 mon.a (mon.0) 13887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:36 smithi115 bash[46934]: cluster 2022-04-22T22:16:35.090275+0000 mgr.y (mgr.34107) 9908 : cluster [DBG] pgmap v10655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:36 smithi115 bash[46934]: audit 2022-04-22T22:16:36.011560+0000 mon.a (mon.0) 13887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:37 smithi121 bash[36902]: audit 2022-04-22T22:16:37.229999+0000 mon.a (mon.0) 13888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:37.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:37 smithi115 bash[45495]: audit 2022-04-22T22:16:37.229999+0000 mon.a (mon.0) 13888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:37 smithi115 bash[46934]: audit 2022-04-22T22:16:37.229999+0000 mon.a (mon.0) 13888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:38.505 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:16:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:38 smithi121 bash[36902]: cluster 2022-04-22T22:16:37.091326+0000 mgr.y (mgr.34107) 9909 : cluster [DBG] pgmap v10656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:38 smithi121 bash[36902]: audit 2022-04-22T22:16:37.403512+0000 mon.a (mon.0) 13889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:38.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:38 smithi115 bash[46934]: cluster 2022-04-22T22:16:37.091326+0000 mgr.y (mgr.34107) 9909 : cluster [DBG] pgmap v10656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:38 smithi115 bash[46934]: audit 2022-04-22T22:16:37.403512+0000 mon.a (mon.0) 13889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:38 smithi115 bash[45495]: cluster 2022-04-22T22:16:37.091326+0000 mgr.y (mgr.34107) 9909 : cluster [DBG] pgmap v10656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:38 smithi115 bash[45495]: audit 2022-04-22T22:16:37.403512+0000 mon.a (mon.0) 13889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 43s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 107s ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 43s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 107s ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 107s ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 43s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:16:38.844 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 43s ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 107s ago 4h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 43s ago 4h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 43s ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 107s ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 43s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 43s ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 43s ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:16:38.845 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 43s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:16:38.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 107s ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:16:38.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 107s ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:16:38.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 107s ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:16:38.846 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 107s ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:16:38.846 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 107s ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:16:39.250 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:16:39.250 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:16:39.250 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:16:39.250 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:16:39.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:16:39.252 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:16:39.252 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:16:39.252 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:16:39.252 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:16:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:39 smithi121 bash[36902]: audit 2022-04-22T22:16:38.485945+0000 mgr.y (mgr.34107) 9910 : audit [DBG] from='client.55561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:39.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:39 smithi121 bash[36902]: audit 2022-04-22T22:16:39.246789+0000 mon.c (mon.1) 342 : audit [DBG] from='client.? 172.21.15.115:0/3241681732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:39.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:39 smithi115 bash[45495]: audit 2022-04-22T22:16:38.485945+0000 mgr.y (mgr.34107) 9910 : audit [DBG] from='client.55561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:39.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:39 smithi115 bash[45495]: audit 2022-04-22T22:16:39.246789+0000 mon.c (mon.1) 342 : audit [DBG] from='client.? 172.21.15.115:0/3241681732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:39.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:39 smithi115 bash[46934]: audit 2022-04-22T22:16:38.485945+0000 mgr.y (mgr.34107) 9910 : audit [DBG] from='client.55561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:39.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:39 smithi115 bash[46934]: audit 2022-04-22T22:16:39.246789+0000 mon.c (mon.1) 342 : audit [DBG] from='client.? 172.21.15.115:0/3241681732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:16:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:40 smithi121 bash[36902]: audit 2022-04-22T22:16:38.820422+0000 mgr.y (mgr.34107) 9911 : audit [DBG] from='client.45636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:40 smithi121 bash[36902]: cluster 2022-04-22T22:16:39.092364+0000 mgr.y (mgr.34107) 9912 : cluster [DBG] pgmap v10657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:40 smithi115 bash[45495]: audit 2022-04-22T22:16:38.820422+0000 mgr.y (mgr.34107) 9911 : audit [DBG] from='client.45636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:40 smithi115 bash[45495]: cluster 2022-04-22T22:16:39.092364+0000 mgr.y (mgr.34107) 9912 : cluster [DBG] pgmap v10657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:40 smithi115 bash[46934]: audit 2022-04-22T22:16:38.820422+0000 mgr.y (mgr.34107) 9911 : audit [DBG] from='client.45636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:16:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:40 smithi115 bash[46934]: cluster 2022-04-22T22:16:39.092364+0000 mgr.y (mgr.34107) 9912 : cluster [DBG] pgmap v10657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:41 smithi121 bash[36902]: audit 2022-04-22T22:16:41.014254+0000 mon.a (mon.0) 13890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:41 smithi115 bash[45495]: audit 2022-04-22T22:16:41.014254+0000 mon.a (mon.0) 13890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:41 smithi115 bash[46934]: audit 2022-04-22T22:16:41.014254+0000 mon.a (mon.0) 13890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:42.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[46934]: cluster 2022-04-22T22:16:41.093405+0000 mgr.y (mgr.34107) 9913 : cluster [DBG] pgmap v10658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:42.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[46934]: audit 2022-04-22T22:16:42.233583+0000 mon.a (mon.0) 13891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:42.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[46934]: audit 2022-04-22T22:16:42.405787+0000 mon.a (mon.0) 13892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:42.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[45495]: cluster 2022-04-22T22:16:41.093405+0000 mgr.y (mgr.34107) 9913 : cluster [DBG] pgmap v10658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:42.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[45495]: audit 2022-04-22T22:16:42.233583+0000 mon.a (mon.0) 13891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:42.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[45495]: audit 2022-04-22T22:16:42.405787+0000 mon.a (mon.0) 13892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:42 smithi121 bash[36902]: cluster 2022-04-22T22:16:41.093405+0000 mgr.y (mgr.34107) 9913 : cluster [DBG] pgmap v10658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:42 smithi121 bash[36902]: audit 2022-04-22T22:16:42.233583+0000 mon.a (mon.0) 13891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:42 smithi121 bash[36902]: audit 2022-04-22T22:16:42.405787+0000 mon.a (mon.0) 13892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:42.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:44 smithi121 bash[36902]: cluster 2022-04-22T22:16:43.094490+0000 mgr.y (mgr.34107) 9914 : cluster [DBG] pgmap v10659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:44 smithi115 bash[45495]: cluster 2022-04-22T22:16:43.094490+0000 mgr.y (mgr.34107) 9914 : cluster [DBG] pgmap v10659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:44.616Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:44 smithi115 bash[46934]: cluster 2022-04-22T22:16:43.094490+0000 mgr.y (mgr.34107) 9914 : cluster [DBG] pgmap v10659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:46 smithi121 bash[36902]: cluster 2022-04-22T22:16:45.095594+0000 mgr.y (mgr.34107) 9915 : cluster [DBG] pgmap v10660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:46 smithi121 bash[36902]: audit 2022-04-22T22:16:46.017329+0000 mon.a (mon.0) 13893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:46 smithi115 bash[45495]: cluster 2022-04-22T22:16:45.095594+0000 mgr.y (mgr.34107) 9915 : cluster [DBG] pgmap v10660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:46 smithi115 bash[45495]: audit 2022-04-22T22:16:46.017329+0000 mon.a (mon.0) 13893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:46 smithi115 bash[46934]: cluster 2022-04-22T22:16:45.095594+0000 mgr.y (mgr.34107) 9915 : cluster [DBG] pgmap v10660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:46 smithi115 bash[46934]: audit 2022-04-22T22:16:46.017329+0000 mon.a (mon.0) 13893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:47 smithi121 bash[36902]: audit 2022-04-22T22:16:47.237330+0000 mon.a (mon.0) 13894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:47 smithi121 bash[36902]: audit 2022-04-22T22:16:47.408744+0000 mon.a (mon.0) 13895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:47 smithi115 bash[45495]: audit 2022-04-22T22:16:47.237330+0000 mon.a (mon.0) 13894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:47 smithi115 bash[45495]: audit 2022-04-22T22:16:47.408744+0000 mon.a (mon.0) 13895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:47 smithi115 bash[46934]: audit 2022-04-22T22:16:47.237330+0000 mon.a (mon.0) 13894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:47 smithi115 bash[46934]: audit 2022-04-22T22:16:47.408744+0000 mon.a (mon.0) 13895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:48 smithi121 bash[36902]: cluster 2022-04-22T22:16:47.096733+0000 mgr.y (mgr.34107) 9916 : cluster [DBG] pgmap v10661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:48 smithi115 bash[45495]: cluster 2022-04-22T22:16:47.096733+0000 mgr.y (mgr.34107) 9916 : cluster [DBG] pgmap v10661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:48 smithi115 bash[46934]: cluster 2022-04-22T22:16:47.096733+0000 mgr.y (mgr.34107) 9916 : cluster [DBG] pgmap v10661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:50 smithi121 bash[36902]: cluster 2022-04-22T22:16:49.097959+0000 mgr.y (mgr.34107) 9917 : cluster [DBG] pgmap v10662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:50 smithi115 bash[45495]: cluster 2022-04-22T22:16:49.097959+0000 mgr.y (mgr.34107) 9917 : cluster [DBG] pgmap v10662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:50 smithi115 bash[46934]: cluster 2022-04-22T22:16:49.097959+0000 mgr.y (mgr.34107) 9917 : cluster [DBG] pgmap v10662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:51 smithi121 bash[36902]: audit 2022-04-22T22:16:51.020083+0000 mon.a (mon.0) 13896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:51 smithi115 bash[45495]: audit 2022-04-22T22:16:51.020083+0000 mon.a (mon.0) 13896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:51 smithi115 bash[46934]: audit 2022-04-22T22:16:51.020083+0000 mon.a (mon.0) 13896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:52 smithi121 bash[36902]: cluster 2022-04-22T22:16:51.099038+0000 mgr.y (mgr.34107) 9918 : cluster [DBG] pgmap v10663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:52 smithi121 bash[36902]: audit 2022-04-22T22:16:52.240985+0000 mon.a (mon.0) 13897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:52 smithi121 bash[36902]: audit 2022-04-22T22:16:52.411979+0000 mon.a (mon.0) 13898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:52.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[45495]: cluster 2022-04-22T22:16:51.099038+0000 mgr.y (mgr.34107) 9918 : cluster [DBG] pgmap v10663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:52.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[45495]: audit 2022-04-22T22:16:52.240985+0000 mon.a (mon.0) 13897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:52.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[45495]: audit 2022-04-22T22:16:52.411979+0000 mon.a (mon.0) 13898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:52.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[46934]: cluster 2022-04-22T22:16:51.099038+0000 mgr.y (mgr.34107) 9918 : cluster [DBG] pgmap v10663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:52.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[46934]: audit 2022-04-22T22:16:52.240985+0000 mon.a (mon.0) 13897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:52.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[46934]: audit 2022-04-22T22:16:52.411979+0000 mon.a (mon.0) 13898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:52.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:54 smithi121 bash[36902]: cluster 2022-04-22T22:16:53.100167+0000 mgr.y (mgr.34107) 9919 : cluster [DBG] pgmap v10664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:54 smithi115 bash[45495]: cluster 2022-04-22T22:16:53.100167+0000 mgr.y (mgr.34107) 9919 : cluster [DBG] pgmap v10664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:16:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:16:54.616Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:16:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:54 smithi115 bash[46934]: cluster 2022-04-22T22:16:53.100167+0000 mgr.y (mgr.34107) 9919 : cluster [DBG] pgmap v10664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:56 smithi121 bash[36902]: cluster 2022-04-22T22:16:55.101167+0000 mgr.y (mgr.34107) 9920 : cluster [DBG] pgmap v10665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:56 smithi121 bash[36902]: audit 2022-04-22T22:16:56.023094+0000 mon.a (mon.0) 13899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:56 smithi115 bash[45495]: cluster 2022-04-22T22:16:55.101167+0000 mgr.y (mgr.34107) 9920 : cluster [DBG] pgmap v10665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:56 smithi115 bash[45495]: audit 2022-04-22T22:16:56.023094+0000 mon.a (mon.0) 13899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:56 smithi115 bash[46934]: cluster 2022-04-22T22:16:55.101167+0000 mgr.y (mgr.34107) 9920 : cluster [DBG] pgmap v10665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:56 smithi115 bash[46934]: audit 2022-04-22T22:16:56.023094+0000 mon.a (mon.0) 13899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:16:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:57 smithi121 bash[36902]: audit 2022-04-22T22:16:57.244704+0000 mon.a (mon.0) 13900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:57 smithi121 bash[36902]: audit 2022-04-22T22:16:57.415016+0000 mon.a (mon.0) 13901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:57.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:57 smithi115 bash[46934]: audit 2022-04-22T22:16:57.244704+0000 mon.a (mon.0) 13900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:57 smithi115 bash[46934]: audit 2022-04-22T22:16:57.415016+0000 mon.a (mon.0) 13901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:57 smithi115 bash[45495]: audit 2022-04-22T22:16:57.244704+0000 mon.a (mon.0) 13900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:16:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:57 smithi115 bash[45495]: audit 2022-04-22T22:16:57.415016+0000 mon.a (mon.0) 13901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:16:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:16:58 smithi121 bash[36902]: cluster 2022-04-22T22:16:57.102400+0000 mgr.y (mgr.34107) 9921 : cluster [DBG] pgmap v10666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:16:58 smithi115 bash[46934]: cluster 2022-04-22T22:16:57.102400+0000 mgr.y (mgr.34107) 9921 : cluster [DBG] pgmap v10666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:16:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:16:58 smithi115 bash[45495]: cluster 2022-04-22T22:16:57.102400+0000 mgr.y (mgr.34107) 9921 : cluster [DBG] pgmap v10666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:00 smithi121 bash[36902]: cluster 2022-04-22T22:16:59.103569+0000 mgr.y (mgr.34107) 9922 : cluster [DBG] pgmap v10667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:00 smithi115 bash[45495]: cluster 2022-04-22T22:16:59.103569+0000 mgr.y (mgr.34107) 9922 : cluster [DBG] pgmap v10667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:00 smithi115 bash[46934]: cluster 2022-04-22T22:16:59.103569+0000 mgr.y (mgr.34107) 9922 : cluster [DBG] pgmap v10667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:01 smithi121 bash[36902]: audit 2022-04-22T22:17:01.026464+0000 mon.a (mon.0) 13902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:01 smithi115 bash[45495]: audit 2022-04-22T22:17:01.026464+0000 mon.a (mon.0) 13902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:01 smithi115 bash[46934]: audit 2022-04-22T22:17:01.026464+0000 mon.a (mon.0) 13902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:02 smithi121 bash[36902]: cluster 2022-04-22T22:17:01.104659+0000 mgr.y (mgr.34107) 9923 : cluster [DBG] pgmap v10668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:02 smithi121 bash[36902]: audit 2022-04-22T22:17:02.248341+0000 mon.a (mon.0) 13903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:02 smithi121 bash[36902]: audit 2022-04-22T22:17:02.418221+0000 mon.a (mon.0) 13904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:02.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[46934]: cluster 2022-04-22T22:17:01.104659+0000 mgr.y (mgr.34107) 9923 : cluster [DBG] pgmap v10668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:02.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[46934]: audit 2022-04-22T22:17:02.248341+0000 mon.a (mon.0) 13903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:02.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[46934]: audit 2022-04-22T22:17:02.418221+0000 mon.a (mon.0) 13904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:02.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[45495]: cluster 2022-04-22T22:17:01.104659+0000 mgr.y (mgr.34107) 9923 : cluster [DBG] pgmap v10668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:02.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[45495]: audit 2022-04-22T22:17:02.248341+0000 mon.a (mon.0) 13903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:02.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[45495]: audit 2022-04-22T22:17:02.418221+0000 mon.a (mon.0) 13904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:02.767 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:02.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:04 smithi121 bash[36902]: cluster 2022-04-22T22:17:03.105745+0000 mgr.y (mgr.34107) 9924 : cluster [DBG] pgmap v10669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:04 smithi115 bash[45495]: cluster 2022-04-22T22:17:03.105745+0000 mgr.y (mgr.34107) 9924 : cluster [DBG] pgmap v10669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:04.615Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:04 smithi115 bash[46934]: cluster 2022-04-22T22:17:03.105745+0000 mgr.y (mgr.34107) 9924 : cluster [DBG] pgmap v10669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[45495]: audit 2022-04-22T22:17:04.794452+0000 mon.a (mon.0) 13905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:17:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[45495]: audit 2022-04-22T22:17:04.797539+0000 mon.a (mon.0) 13906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:17:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[45495]: audit 2022-04-22T22:17:04.799559+0000 mon.a (mon.0) 13907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:17:05.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[45495]: audit 2022-04-22T22:17:05.165365+0000 mon.a (mon.0) 13908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:17:05.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[46934]: audit 2022-04-22T22:17:04.794452+0000 mon.a (mon.0) 13905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:17:05.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[46934]: audit 2022-04-22T22:17:04.797539+0000 mon.a (mon.0) 13906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:17:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[46934]: audit 2022-04-22T22:17:04.799559+0000 mon.a (mon.0) 13907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:17:05.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:05 smithi115 bash[46934]: audit 2022-04-22T22:17:05.165365+0000 mon.a (mon.0) 13908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:17:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:05 smithi121 bash[36902]: audit 2022-04-22T22:17:04.794452+0000 mon.a (mon.0) 13905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:17:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:05 smithi121 bash[36902]: audit 2022-04-22T22:17:04.797539+0000 mon.a (mon.0) 13906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:17:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:05 smithi121 bash[36902]: audit 2022-04-22T22:17:04.799559+0000 mon.a (mon.0) 13907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:17:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:05 smithi121 bash[36902]: audit 2022-04-22T22:17:05.165365+0000 mon.a (mon.0) 13908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:17:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:06 smithi115 bash[45495]: cluster 2022-04-22T22:17:05.106726+0000 mgr.y (mgr.34107) 9925 : cluster [DBG] pgmap v10670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:06 smithi115 bash[45495]: audit 2022-04-22T22:17:06.029753+0000 mon.a (mon.0) 13909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:06.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:06 smithi115 bash[46934]: cluster 2022-04-22T22:17:05.106726+0000 mgr.y (mgr.34107) 9925 : cluster [DBG] pgmap v10670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:06.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:06 smithi115 bash[46934]: audit 2022-04-22T22:17:06.029753+0000 mon.a (mon.0) 13909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:06 smithi121 bash[36902]: cluster 2022-04-22T22:17:05.106726+0000 mgr.y (mgr.34107) 9925 : cluster [DBG] pgmap v10670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:06 smithi121 bash[36902]: audit 2022-04-22T22:17:06.029753+0000 mon.a (mon.0) 13909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:07 smithi115 bash[45495]: audit 2022-04-22T22:17:07.251909+0000 mon.a (mon.0) 13910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:07 smithi115 bash[45495]: audit 2022-04-22T22:17:07.421129+0000 mon.a (mon.0) 13911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:07.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:07 smithi115 bash[46934]: audit 2022-04-22T22:17:07.251909+0000 mon.a (mon.0) 13910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:07.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:07 smithi115 bash[46934]: audit 2022-04-22T22:17:07.421129+0000 mon.a (mon.0) 13911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:07 smithi121 bash[36902]: audit 2022-04-22T22:17:07.251909+0000 mon.a (mon.0) 13910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:07 smithi121 bash[36902]: audit 2022-04-22T22:17:07.421129+0000 mon.a (mon.0) 13911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:08 smithi115 bash[45495]: cluster 2022-04-22T22:17:07.108011+0000 mgr.y (mgr.34107) 9926 : cluster [DBG] pgmap v10671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:08 smithi115 bash[46934]: cluster 2022-04-22T22:17:07.108011+0000 mgr.y (mgr.34107) 9926 : cluster [DBG] pgmap v10671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:08 smithi121 bash[36902]: cluster 2022-04-22T22:17:07.108011+0000 mgr.y (mgr.34107) 9926 : cluster [DBG] pgmap v10671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:09.627 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 74s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 74s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:17:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 74s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 74s ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 4h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 74s ago 4h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 74s ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 74s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 74s ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 74s ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:17:09.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 74s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:17:09.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:17:09.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:17:09.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:17:09.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:17:09.972 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:17:10.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:17:10.372 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:17:10.373 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:17:10.373 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:17:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:10 smithi115 bash[45495]: cluster 2022-04-22T22:17:09.109272+0000 mgr.y (mgr.34107) 9927 : cluster [DBG] pgmap v10672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:10 smithi115 bash[45495]: audit 2022-04-22T22:17:10.367592+0000 mon.a (mon.0) 13912 : audit [DBG] from='client.? 172.21.15.115:0/3965690126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:10 smithi115 bash[46934]: cluster 2022-04-22T22:17:09.109272+0000 mgr.y (mgr.34107) 9927 : cluster [DBG] pgmap v10672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:10 smithi115 bash[46934]: audit 2022-04-22T22:17:10.367592+0000 mon.a (mon.0) 13912 : audit [DBG] from='client.? 172.21.15.115:0/3965690126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:10 smithi121 bash[36902]: cluster 2022-04-22T22:17:09.109272+0000 mgr.y (mgr.34107) 9927 : cluster [DBG] pgmap v10672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:10 smithi121 bash[36902]: audit 2022-04-22T22:17:10.367592+0000 mon.a (mon.0) 13912 : audit [DBG] from='client.? 172.21.15.115:0/3965690126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[45495]: audit 2022-04-22T22:17:09.607342+0000 mgr.y (mgr.34107) 9928 : audit [DBG] from='client.55579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[45495]: audit 2022-04-22T22:17:09.952279+0000 mgr.y (mgr.34107) 9929 : audit [DBG] from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[45495]: audit 2022-04-22T22:17:11.032644+0000 mon.a (mon.0) 13913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[46934]: audit 2022-04-22T22:17:09.607342+0000 mgr.y (mgr.34107) 9928 : audit [DBG] from='client.55579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[46934]: audit 2022-04-22T22:17:09.952279+0000 mgr.y (mgr.34107) 9929 : audit [DBG] from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:11 smithi115 bash[46934]: audit 2022-04-22T22:17:11.032644+0000 mon.a (mon.0) 13913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:11 smithi121 bash[36902]: audit 2022-04-22T22:17:09.607342+0000 mgr.y (mgr.34107) 9928 : audit [DBG] from='client.55579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:11 smithi121 bash[36902]: audit 2022-04-22T22:17:09.952279+0000 mgr.y (mgr.34107) 9929 : audit [DBG] from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:11 smithi121 bash[36902]: audit 2022-04-22T22:17:11.032644+0000 mon.a (mon.0) 13913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:12 smithi121 bash[36902]: cluster 2022-04-22T22:17:11.110370+0000 mgr.y (mgr.34107) 9930 : cluster [DBG] pgmap v10673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:12 smithi121 bash[36902]: audit 2022-04-22T22:17:12.255554+0000 mon.a (mon.0) 13914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:12 smithi121 bash[36902]: audit 2022-04-22T22:17:12.424272+0000 mon.a (mon.0) 13915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:12.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[45495]: cluster 2022-04-22T22:17:11.110370+0000 mgr.y (mgr.34107) 9930 : cluster [DBG] pgmap v10673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[45495]: audit 2022-04-22T22:17:12.255554+0000 mon.a (mon.0) 13914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[45495]: audit 2022-04-22T22:17:12.424272+0000 mon.a (mon.0) 13915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[46934]: cluster 2022-04-22T22:17:11.110370+0000 mgr.y (mgr.34107) 9930 : cluster [DBG] pgmap v10673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[46934]: audit 2022-04-22T22:17:12.255554+0000 mon.a (mon.0) 13914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:12 smithi115 bash[46934]: audit 2022-04-22T22:17:12.424272+0000 mon.a (mon.0) 13915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:13 smithi121 bash[36902]: cluster 2022-04-22T22:17:13.111454+0000 mgr.y (mgr.34107) 9931 : cluster [DBG] pgmap v10674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:13 smithi115 bash[46934]: cluster 2022-04-22T22:17:13.111454+0000 mgr.y (mgr.34107) 9931 : cluster [DBG] pgmap v10674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:13 smithi115 bash[45495]: cluster 2022-04-22T22:17:13.111454+0000 mgr.y (mgr.34107) 9931 : cluster [DBG] pgmap v10674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:14.617Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:16 smithi121 bash[36902]: cluster 2022-04-22T22:17:15.112515+0000 mgr.y (mgr.34107) 9932 : cluster [DBG] pgmap v10675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:16 smithi121 bash[36902]: audit 2022-04-22T22:17:16.035872+0000 mon.a (mon.0) 13916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:16 smithi115 bash[45495]: cluster 2022-04-22T22:17:15.112515+0000 mgr.y (mgr.34107) 9932 : cluster [DBG] pgmap v10675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:16 smithi115 bash[45495]: audit 2022-04-22T22:17:16.035872+0000 mon.a (mon.0) 13916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:16 smithi115 bash[46934]: cluster 2022-04-22T22:17:15.112515+0000 mgr.y (mgr.34107) 9932 : cluster [DBG] pgmap v10675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:16 smithi115 bash[46934]: audit 2022-04-22T22:17:16.035872+0000 mon.a (mon.0) 13916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:17 smithi121 bash[36902]: audit 2022-04-22T22:17:16.629398+0000 mon.a (mon.0) 13917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:17:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:17 smithi121 bash[36902]: audit 2022-04-22T22:17:16.630255+0000 mon.a (mon.0) 13918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:17:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:17 smithi121 bash[36902]: audit 2022-04-22T22:17:17.259164+0000 mon.a (mon.0) 13919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[45495]: audit 2022-04-22T22:17:16.629398+0000 mon.a (mon.0) 13917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:17:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[45495]: audit 2022-04-22T22:17:16.630255+0000 mon.a (mon.0) 13918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:17:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[45495]: audit 2022-04-22T22:17:17.259164+0000 mon.a (mon.0) 13919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[46934]: audit 2022-04-22T22:17:16.629398+0000 mon.a (mon.0) 13917 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:17:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[46934]: audit 2022-04-22T22:17:16.630255+0000 mon.a (mon.0) 13918 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:17:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:17 smithi115 bash[46934]: audit 2022-04-22T22:17:17.259164+0000 mon.a (mon.0) 13919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:18 smithi121 bash[36902]: cluster 2022-04-22T22:17:17.113642+0000 mgr.y (mgr.34107) 9933 : cluster [DBG] pgmap v10676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:18 smithi121 bash[36902]: audit 2022-04-22T22:17:17.427154+0000 mon.a (mon.0) 13920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:18 smithi115 bash[45495]: cluster 2022-04-22T22:17:17.113642+0000 mgr.y (mgr.34107) 9933 : cluster [DBG] pgmap v10676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:18 smithi115 bash[45495]: audit 2022-04-22T22:17:17.427154+0000 mon.a (mon.0) 13920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:18 smithi115 bash[46934]: cluster 2022-04-22T22:17:17.113642+0000 mgr.y (mgr.34107) 9933 : cluster [DBG] pgmap v10676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:18 smithi115 bash[46934]: audit 2022-04-22T22:17:17.427154+0000 mon.a (mon.0) 13920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:20 smithi121 bash[36902]: cluster 2022-04-22T22:17:19.114729+0000 mgr.y (mgr.34107) 9934 : cluster [DBG] pgmap v10677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:20 smithi115 bash[45495]: cluster 2022-04-22T22:17:19.114729+0000 mgr.y (mgr.34107) 9934 : cluster [DBG] pgmap v10677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:20 smithi115 bash[46934]: cluster 2022-04-22T22:17:19.114729+0000 mgr.y (mgr.34107) 9934 : cluster [DBG] pgmap v10677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:21 smithi121 bash[36902]: audit 2022-04-22T22:17:21.038668+0000 mon.a (mon.0) 13921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:21 smithi115 bash[45495]: audit 2022-04-22T22:17:21.038668+0000 mon.a (mon.0) 13921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:21 smithi115 bash[46934]: audit 2022-04-22T22:17:21.038668+0000 mon.a (mon.0) 13921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:22 smithi121 bash[36902]: cluster 2022-04-22T22:17:21.115769+0000 mgr.y (mgr.34107) 9935 : cluster [DBG] pgmap v10678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:22 smithi121 bash[36902]: audit 2022-04-22T22:17:22.263013+0000 mon.a (mon.0) 13922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:22.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:22 smithi115 bash[45495]: cluster 2022-04-22T22:17:21.115769+0000 mgr.y (mgr.34107) 9935 : cluster [DBG] pgmap v10678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:22.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:22 smithi115 bash[45495]: audit 2022-04-22T22:17:22.263013+0000 mon.a (mon.0) 13922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:22.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:22 smithi115 bash[46934]: cluster 2022-04-22T22:17:21.115769+0000 mgr.y (mgr.34107) 9935 : cluster [DBG] pgmap v10678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:22.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:22 smithi115 bash[46934]: audit 2022-04-22T22:17:22.263013+0000 mon.a (mon.0) 13922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:22.762Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:23 smithi121 bash[36902]: audit 2022-04-22T22:17:22.430107+0000 mon.a (mon.0) 13923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:23 smithi115 bash[45495]: audit 2022-04-22T22:17:22.430107+0000 mon.a (mon.0) 13923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:23 smithi115 bash[46934]: audit 2022-04-22T22:17:22.430107+0000 mon.a (mon.0) 13923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:24.621 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:24 smithi115 bash[45495]: cluster 2022-04-22T22:17:23.116981+0000 mgr.y (mgr.34107) 9936 : cluster [DBG] pgmap v10679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:24 smithi115 bash[46934]: cluster 2022-04-22T22:17:23.116981+0000 mgr.y (mgr.34107) 9936 : cluster [DBG] pgmap v10679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:24 smithi121 bash[36902]: cluster 2022-04-22T22:17:23.116981+0000 mgr.y (mgr.34107) 9936 : cluster [DBG] pgmap v10679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:24.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:26 smithi121 bash[36902]: cluster 2022-04-22T22:17:25.118007+0000 mgr.y (mgr.34107) 9937 : cluster [DBG] pgmap v10680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:26 smithi121 bash[36902]: audit 2022-04-22T22:17:26.041769+0000 mon.a (mon.0) 13924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:26 smithi115 bash[45495]: cluster 2022-04-22T22:17:25.118007+0000 mgr.y (mgr.34107) 9937 : cluster [DBG] pgmap v10680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:26 smithi115 bash[45495]: audit 2022-04-22T22:17:26.041769+0000 mon.a (mon.0) 13924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:26 smithi115 bash[46934]: cluster 2022-04-22T22:17:25.118007+0000 mgr.y (mgr.34107) 9937 : cluster [DBG] pgmap v10680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:26 smithi115 bash[46934]: audit 2022-04-22T22:17:26.041769+0000 mon.a (mon.0) 13924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:27 smithi121 bash[36902]: audit 2022-04-22T22:17:27.266849+0000 mon.a (mon.0) 13925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:27 smithi115 bash[45495]: audit 2022-04-22T22:17:27.266849+0000 mon.a (mon.0) 13925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:27 smithi115 bash[46934]: audit 2022-04-22T22:17:27.266849+0000 mon.a (mon.0) 13925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:28 smithi121 bash[36902]: cluster 2022-04-22T22:17:27.119178+0000 mgr.y (mgr.34107) 9938 : cluster [DBG] pgmap v10681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:28 smithi121 bash[36902]: audit 2022-04-22T22:17:27.433105+0000 mon.a (mon.0) 13926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:28 smithi115 bash[45495]: cluster 2022-04-22T22:17:27.119178+0000 mgr.y (mgr.34107) 9938 : cluster [DBG] pgmap v10681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:28 smithi115 bash[45495]: audit 2022-04-22T22:17:27.433105+0000 mon.a (mon.0) 13926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:28 smithi115 bash[46934]: cluster 2022-04-22T22:17:27.119178+0000 mgr.y (mgr.34107) 9938 : cluster [DBG] pgmap v10681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:28 smithi115 bash[46934]: audit 2022-04-22T22:17:27.433105+0000 mon.a (mon.0) 13926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:30.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:30 smithi121 bash[36902]: cluster 2022-04-22T22:17:29.120352+0000 mgr.y (mgr.34107) 9939 : cluster [DBG] pgmap v10682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:30.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:30 smithi115 bash[45495]: cluster 2022-04-22T22:17:29.120352+0000 mgr.y (mgr.34107) 9939 : cluster [DBG] pgmap v10682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:30 smithi115 bash[46934]: cluster 2022-04-22T22:17:29.120352+0000 mgr.y (mgr.34107) 9939 : cluster [DBG] pgmap v10682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:31 smithi121 bash[36902]: audit 2022-04-22T22:17:31.044897+0000 mon.a (mon.0) 13927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:31 smithi115 bash[45495]: audit 2022-04-22T22:17:31.044897+0000 mon.a (mon.0) 13927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:31 smithi115 bash[46934]: audit 2022-04-22T22:17:31.044897+0000 mon.a (mon.0) 13927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:32 smithi121 bash[36902]: cluster 2022-04-22T22:17:31.121390+0000 mgr.y (mgr.34107) 9940 : cluster [DBG] pgmap v10683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:32 smithi121 bash[36902]: audit 2022-04-22T22:17:32.270427+0000 mon.a (mon.0) 13928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:32.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:32 smithi115 bash[45495]: cluster 2022-04-22T22:17:31.121390+0000 mgr.y (mgr.34107) 9940 : cluster [DBG] pgmap v10683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:32.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:32 smithi115 bash[45495]: audit 2022-04-22T22:17:32.270427+0000 mon.a (mon.0) 13928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:32.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:32 smithi115 bash[46934]: cluster 2022-04-22T22:17:31.121390+0000 mgr.y (mgr.34107) 9940 : cluster [DBG] pgmap v10683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:32.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:32 smithi115 bash[46934]: audit 2022-04-22T22:17:32.270427+0000 mon.a (mon.0) 13928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:32.764Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:33 smithi121 bash[36902]: audit 2022-04-22T22:17:32.436148+0000 mon.a (mon.0) 13929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:33 smithi115 bash[45495]: audit 2022-04-22T22:17:32.436148+0000 mon.a (mon.0) 13929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:33 smithi115 bash[46934]: audit 2022-04-22T22:17:32.436148+0000 mon.a (mon.0) 13929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:34 smithi121 bash[36902]: cluster 2022-04-22T22:17:33.122527+0000 mgr.y (mgr.34107) 9941 : cluster [DBG] pgmap v10684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:34 smithi115 bash[45495]: cluster 2022-04-22T22:17:33.122527+0000 mgr.y (mgr.34107) 9941 : cluster [DBG] pgmap v10684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:34 smithi115 bash[46934]: cluster 2022-04-22T22:17:33.122527+0000 mgr.y (mgr.34107) 9941 : cluster [DBG] pgmap v10684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:34.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:36 smithi121 bash[36902]: cluster 2022-04-22T22:17:35.123664+0000 mgr.y (mgr.34107) 9942 : cluster [DBG] pgmap v10685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:36 smithi121 bash[36902]: audit 2022-04-22T22:17:36.048100+0000 mon.a (mon.0) 13930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:36 smithi115 bash[45495]: cluster 2022-04-22T22:17:35.123664+0000 mgr.y (mgr.34107) 9942 : cluster [DBG] pgmap v10685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:36 smithi115 bash[45495]: audit 2022-04-22T22:17:36.048100+0000 mon.a (mon.0) 13930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:36 smithi115 bash[46934]: cluster 2022-04-22T22:17:35.123664+0000 mgr.y (mgr.34107) 9942 : cluster [DBG] pgmap v10685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:36.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:36 smithi115 bash[46934]: audit 2022-04-22T22:17:36.048100+0000 mon.a (mon.0) 13930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:37 smithi121 bash[36902]: audit 2022-04-22T22:17:37.274128+0000 mon.a (mon.0) 13931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:37 smithi115 bash[45495]: audit 2022-04-22T22:17:37.274128+0000 mon.a (mon.0) 13931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:37 smithi115 bash[46934]: audit 2022-04-22T22:17:37.274128+0000 mon.a (mon.0) 13931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:38 smithi121 bash[36902]: cluster 2022-04-22T22:17:37.124787+0000 mgr.y (mgr.34107) 9943 : cluster [DBG] pgmap v10686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:38 smithi121 bash[36902]: audit 2022-04-22T22:17:37.439211+0000 mon.a (mon.0) 13932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:38 smithi115 bash[45495]: cluster 2022-04-22T22:17:37.124787+0000 mgr.y (mgr.34107) 9943 : cluster [DBG] pgmap v10686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:38 smithi115 bash[45495]: audit 2022-04-22T22:17:37.439211+0000 mon.a (mon.0) 13932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:38 smithi115 bash[46934]: cluster 2022-04-22T22:17:37.124787+0000 mgr.y (mgr.34107) 9943 : cluster [DBG] pgmap v10686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:38 smithi115 bash[46934]: audit 2022-04-22T22:17:37.439211+0000 mon.a (mon.0) 13932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:40.752 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:17:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:40 smithi121 bash[36902]: cluster 2022-04-22T22:17:39.125910+0000 mgr.y (mgr.34107) 9944 : cluster [DBG] pgmap v10687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:40.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:40 smithi115 bash[46934]: cluster 2022-04-22T22:17:39.125910+0000 mgr.y (mgr.34107) 9944 : cluster [DBG] pgmap v10687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:40 smithi115 bash[45495]: cluster 2022-04-22T22:17:39.125910+0000 mgr.y (mgr.34107) 9944 : cluster [DBG] pgmap v10687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:41.101 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:17:41.101 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 105s ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:17:41.101 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 2m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 105s ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 4h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 105s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 105s ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 105s ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 105s ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:17:41.102 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 105s ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 105s ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 105s ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 105s ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:17:41.103 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:17:41.104 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:17:41.104 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:17:41.505 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:17:41.505 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:17:41.505 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:17:41.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:41.505 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:17:41.506 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:17:41.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:17:41.507 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:17:41.507 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:17:41.507 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:17:41.507 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:17:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:41 smithi121 bash[36902]: audit 2022-04-22T22:17:41.051222+0000 mon.a (mon.0) 13933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:41 smithi115 bash[45495]: audit 2022-04-22T22:17:41.051222+0000 mon.a (mon.0) 13933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:41 smithi115 bash[46934]: audit 2022-04-22T22:17:41.051222+0000 mon.a (mon.0) 13933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:42 smithi121 bash[36902]: audit 2022-04-22T22:17:40.731978+0000 mgr.y (mgr.34107) 9945 : audit [DBG] from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:42 smithi121 bash[36902]: audit 2022-04-22T22:17:41.077182+0000 mgr.y (mgr.34107) 9946 : audit [DBG] from='client.45672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:42 smithi121 bash[36902]: cluster 2022-04-22T22:17:41.126693+0000 mgr.y (mgr.34107) 9947 : cluster [DBG] pgmap v10688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:42 smithi121 bash[36902]: audit 2022-04-22T22:17:41.499630+0000 mon.c (mon.1) 343 : audit [DBG] from='client.? 172.21.15.115:0/3434048173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:42 smithi121 bash[36902]: audit 2022-04-22T22:17:42.278115+0000 mon.a (mon.0) 13934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:42.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[46934]: audit 2022-04-22T22:17:40.731978+0000 mgr.y (mgr.34107) 9945 : audit [DBG] from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[46934]: audit 2022-04-22T22:17:41.077182+0000 mgr.y (mgr.34107) 9946 : audit [DBG] from='client.45672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[46934]: cluster 2022-04-22T22:17:41.126693+0000 mgr.y (mgr.34107) 9947 : cluster [DBG] pgmap v10688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:42.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[46934]: audit 2022-04-22T22:17:41.499630+0000 mon.c (mon.1) 343 : audit [DBG] from='client.? 172.21.15.115:0/3434048173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:42.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[46934]: audit 2022-04-22T22:17:42.278115+0000 mon.a (mon.0) 13934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:42.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[45495]: audit 2022-04-22T22:17:40.731978+0000 mgr.y (mgr.34107) 9945 : audit [DBG] from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[45495]: audit 2022-04-22T22:17:41.077182+0000 mgr.y (mgr.34107) 9946 : audit [DBG] from='client.45672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:17:42.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[45495]: cluster 2022-04-22T22:17:41.126693+0000 mgr.y (mgr.34107) 9947 : cluster [DBG] pgmap v10688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:42.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[45495]: audit 2022-04-22T22:17:41.499630+0000 mon.c (mon.1) 343 : audit [DBG] from='client.? 172.21.15.115:0/3434048173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:17:42.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[45495]: audit 2022-04-22T22:17:42.278115+0000 mon.a (mon.0) 13934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:43.048 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:42.763Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:43 smithi121 bash[36902]: audit 2022-04-22T22:17:42.442176+0000 mon.a (mon.0) 13935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:43 smithi115 bash[45495]: audit 2022-04-22T22:17:42.442176+0000 mon.a (mon.0) 13935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:43 smithi115 bash[46934]: audit 2022-04-22T22:17:42.442176+0000 mon.a (mon.0) 13935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:44 smithi121 bash[36902]: cluster 2022-04-22T22:17:43.127850+0000 mgr.y (mgr.34107) 9948 : cluster [DBG] pgmap v10689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:44 smithi115 bash[45495]: cluster 2022-04-22T22:17:43.127850+0000 mgr.y (mgr.34107) 9948 : cluster [DBG] pgmap v10689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:44.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:44 smithi115 bash[46934]: cluster 2022-04-22T22:17:43.127850+0000 mgr.y (mgr.34107) 9948 : cluster [DBG] pgmap v10689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:46 smithi121 bash[36902]: cluster 2022-04-22T22:17:45.128807+0000 mgr.y (mgr.34107) 9949 : cluster [DBG] pgmap v10690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:46 smithi121 bash[36902]: audit 2022-04-22T22:17:46.054462+0000 mon.a (mon.0) 13936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:46 smithi115 bash[45495]: cluster 2022-04-22T22:17:45.128807+0000 mgr.y (mgr.34107) 9949 : cluster [DBG] pgmap v10690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:46.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:46 smithi115 bash[45495]: audit 2022-04-22T22:17:46.054462+0000 mon.a (mon.0) 13936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:46 smithi115 bash[46934]: cluster 2022-04-22T22:17:45.128807+0000 mgr.y (mgr.34107) 9949 : cluster [DBG] pgmap v10690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:46.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:46 smithi115 bash[46934]: audit 2022-04-22T22:17:46.054462+0000 mon.a (mon.0) 13936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:47 smithi121 bash[36902]: audit 2022-04-22T22:17:47.281624+0000 mon.a (mon.0) 13937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:47.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:47 smithi121 bash[36902]: audit 2022-04-22T22:17:47.444457+0000 mon.a (mon.0) 13938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:47 smithi115 bash[46934]: audit 2022-04-22T22:17:47.281624+0000 mon.a (mon.0) 13937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:47 smithi115 bash[46934]: audit 2022-04-22T22:17:47.444457+0000 mon.a (mon.0) 13938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:47 smithi115 bash[45495]: audit 2022-04-22T22:17:47.281624+0000 mon.a (mon.0) 13937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:47.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:47 smithi115 bash[45495]: audit 2022-04-22T22:17:47.444457+0000 mon.a (mon.0) 13938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:48 smithi121 bash[36902]: cluster 2022-04-22T22:17:47.129905+0000 mgr.y (mgr.34107) 9950 : cluster [DBG] pgmap v10691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:48 smithi115 bash[46934]: cluster 2022-04-22T22:17:47.129905+0000 mgr.y (mgr.34107) 9950 : cluster [DBG] pgmap v10691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:48.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:48 smithi115 bash[45495]: cluster 2022-04-22T22:17:47.129905+0000 mgr.y (mgr.34107) 9950 : cluster [DBG] pgmap v10691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:50 smithi121 bash[36902]: cluster 2022-04-22T22:17:49.131066+0000 mgr.y (mgr.34107) 9951 : cluster [DBG] pgmap v10692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:50 smithi115 bash[45495]: cluster 2022-04-22T22:17:49.131066+0000 mgr.y (mgr.34107) 9951 : cluster [DBG] pgmap v10692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:50 smithi115 bash[46934]: cluster 2022-04-22T22:17:49.131066+0000 mgr.y (mgr.34107) 9951 : cluster [DBG] pgmap v10692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:51 smithi121 bash[36902]: audit 2022-04-22T22:17:51.057644+0000 mon.a (mon.0) 13939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:51 smithi115 bash[45495]: audit 2022-04-22T22:17:51.057644+0000 mon.a (mon.0) 13939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:51 smithi115 bash[46934]: audit 2022-04-22T22:17:51.057644+0000 mon.a (mon.0) 13939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:52 smithi121 bash[36902]: cluster 2022-04-22T22:17:51.132248+0000 mgr.y (mgr.34107) 9952 : cluster [DBG] pgmap v10693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:52 smithi121 bash[36902]: audit 2022-04-22T22:17:52.285137+0000 mon.a (mon.0) 13940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:52 smithi121 bash[36902]: audit 2022-04-22T22:17:52.447402+0000 mon.a (mon.0) 13941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:52.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[45495]: cluster 2022-04-22T22:17:51.132248+0000 mgr.y (mgr.34107) 9952 : cluster [DBG] pgmap v10693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:52.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[45495]: audit 2022-04-22T22:17:52.285137+0000 mon.a (mon.0) 13940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:52.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[45495]: audit 2022-04-22T22:17:52.447402+0000 mon.a (mon.0) 13941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:52.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[46934]: cluster 2022-04-22T22:17:51.132248+0000 mgr.y (mgr.34107) 9952 : cluster [DBG] pgmap v10693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:52.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[46934]: audit 2022-04-22T22:17:52.285137+0000 mon.a (mon.0) 13940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:52.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[46934]: audit 2022-04-22T22:17:52.447402+0000 mon.a (mon.0) 13941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:52.765Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:54 smithi121 bash[36902]: cluster 2022-04-22T22:17:53.133440+0000 mgr.y (mgr.34107) 9953 : cluster [DBG] pgmap v10694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:54 smithi115 bash[45495]: cluster 2022-04-22T22:17:53.133440+0000 mgr.y (mgr.34107) 9953 : cluster [DBG] pgmap v10694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:54 smithi115 bash[46934]: cluster 2022-04-22T22:17:53.133440+0000 mgr.y (mgr.34107) 9953 : cluster [DBG] pgmap v10694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:17:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:17:54.619Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:17:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:56 smithi121 bash[36902]: cluster 2022-04-22T22:17:55.134421+0000 mgr.y (mgr.34107) 9954 : cluster [DBG] pgmap v10695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:56 smithi121 bash[36902]: audit 2022-04-22T22:17:56.060960+0000 mon.a (mon.0) 13942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:56 smithi115 bash[45495]: cluster 2022-04-22T22:17:55.134421+0000 mgr.y (mgr.34107) 9954 : cluster [DBG] pgmap v10695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:56 smithi115 bash[45495]: audit 2022-04-22T22:17:56.060960+0000 mon.a (mon.0) 13942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:56 smithi115 bash[46934]: cluster 2022-04-22T22:17:55.134421+0000 mgr.y (mgr.34107) 9954 : cluster [DBG] pgmap v10695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:56 smithi115 bash[46934]: audit 2022-04-22T22:17:56.060960+0000 mon.a (mon.0) 13942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:17:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:57 smithi121 bash[36902]: audit 2022-04-22T22:17:57.288879+0000 mon.a (mon.0) 13943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:57 smithi121 bash[36902]: audit 2022-04-22T22:17:57.450377+0000 mon.a (mon.0) 13944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:57 smithi115 bash[45495]: audit 2022-04-22T22:17:57.288879+0000 mon.a (mon.0) 13943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:57 smithi115 bash[45495]: audit 2022-04-22T22:17:57.450377+0000 mon.a (mon.0) 13944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:57 smithi115 bash[46934]: audit 2022-04-22T22:17:57.288879+0000 mon.a (mon.0) 13943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:17:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:57 smithi115 bash[46934]: audit 2022-04-22T22:17:57.450377+0000 mon.a (mon.0) 13944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:17:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:17:58 smithi121 bash[36902]: cluster 2022-04-22T22:17:57.135572+0000 mgr.y (mgr.34107) 9955 : cluster [DBG] pgmap v10696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:17:58 smithi115 bash[45495]: cluster 2022-04-22T22:17:57.135572+0000 mgr.y (mgr.34107) 9955 : cluster [DBG] pgmap v10696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:17:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:17:58 smithi115 bash[46934]: cluster 2022-04-22T22:17:57.135572+0000 mgr.y (mgr.34107) 9955 : cluster [DBG] pgmap v10696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:00 smithi121 bash[36902]: cluster 2022-04-22T22:17:59.136640+0000 mgr.y (mgr.34107) 9956 : cluster [DBG] pgmap v10697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:00.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:00 smithi115 bash[46934]: cluster 2022-04-22T22:17:59.136640+0000 mgr.y (mgr.34107) 9956 : cluster [DBG] pgmap v10697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:00 smithi115 bash[45495]: cluster 2022-04-22T22:17:59.136640+0000 mgr.y (mgr.34107) 9956 : cluster [DBG] pgmap v10697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:01 smithi121 bash[36902]: audit 2022-04-22T22:18:01.064065+0000 mon.a (mon.0) 13945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:01 smithi115 bash[46934]: audit 2022-04-22T22:18:01.064065+0000 mon.a (mon.0) 13945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:01.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:01 smithi115 bash[45495]: audit 2022-04-22T22:18:01.064065+0000 mon.a (mon.0) 13945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:02.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[45495]: cluster 2022-04-22T22:18:01.137675+0000 mgr.y (mgr.34107) 9957 : cluster [DBG] pgmap v10698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:02.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[45495]: audit 2022-04-22T22:18:02.292500+0000 mon.a (mon.0) 13946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:02.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[45495]: audit 2022-04-22T22:18:02.453256+0000 mon.a (mon.0) 13947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:02.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[46934]: cluster 2022-04-22T22:18:01.137675+0000 mgr.y (mgr.34107) 9957 : cluster [DBG] pgmap v10698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:02.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[46934]: audit 2022-04-22T22:18:02.292500+0000 mon.a (mon.0) 13946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:02.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[46934]: audit 2022-04-22T22:18:02.453256+0000 mon.a (mon.0) 13947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:02 smithi121 bash[36902]: cluster 2022-04-22T22:18:01.137675+0000 mgr.y (mgr.34107) 9957 : cluster [DBG] pgmap v10698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:02 smithi121 bash[36902]: audit 2022-04-22T22:18:02.292500+0000 mon.a (mon.0) 13946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:02 smithi121 bash[36902]: audit 2022-04-22T22:18:02.453256+0000 mon.a (mon.0) 13947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:02.765Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:04.618Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:04 smithi115 bash[45495]: cluster 2022-04-22T22:18:03.138830+0000 mgr.y (mgr.34107) 9958 : cluster [DBG] pgmap v10699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:04 smithi115 bash[46934]: cluster 2022-04-22T22:18:03.138830+0000 mgr.y (mgr.34107) 9958 : cluster [DBG] pgmap v10699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:04 smithi121 bash[36902]: cluster 2022-04-22T22:18:03.138830+0000 mgr.y (mgr.34107) 9958 : cluster [DBG] pgmap v10699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[45495]: audit 2022-04-22T22:18:05.173370+0000 mon.a (mon.0) 13948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[45495]: audit 2022-04-22T22:18:05.176246+0000 mon.a (mon.0) 13949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[45495]: audit 2022-04-22T22:18:05.177848+0000 mon.a (mon.0) 13950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[46934]: audit 2022-04-22T22:18:05.173370+0000 mon.a (mon.0) 13948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[46934]: audit 2022-04-22T22:18:05.176246+0000 mon.a (mon.0) 13949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:18:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:05 smithi115 bash[46934]: audit 2022-04-22T22:18:05.177848+0000 mon.a (mon.0) 13950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:18:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:05 smithi121 bash[36902]: audit 2022-04-22T22:18:05.173370+0000 mon.a (mon.0) 13948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:18:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:05 smithi121 bash[36902]: audit 2022-04-22T22:18:05.176246+0000 mon.a (mon.0) 13949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:18:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:05 smithi121 bash[36902]: audit 2022-04-22T22:18:05.177848+0000 mon.a (mon.0) 13950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:18:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[45495]: cluster 2022-04-22T22:18:05.139836+0000 mgr.y (mgr.34107) 9959 : cluster [DBG] pgmap v10700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[45495]: audit 2022-04-22T22:18:05.528800+0000 mon.a (mon.0) 13951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:18:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[45495]: audit 2022-04-22T22:18:06.067364+0000 mon.a (mon.0) 13952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[46934]: cluster 2022-04-22T22:18:05.139836+0000 mgr.y (mgr.34107) 9959 : cluster [DBG] pgmap v10700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[46934]: audit 2022-04-22T22:18:05.528800+0000 mon.a (mon.0) 13951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:18:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:06 smithi115 bash[46934]: audit 2022-04-22T22:18:06.067364+0000 mon.a (mon.0) 13952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:06 smithi121 bash[36902]: cluster 2022-04-22T22:18:05.139836+0000 mgr.y (mgr.34107) 9959 : cluster [DBG] pgmap v10700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:06 smithi121 bash[36902]: audit 2022-04-22T22:18:05.528800+0000 mon.a (mon.0) 13951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:18:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:06 smithi121 bash[36902]: audit 2022-04-22T22:18:06.067364+0000 mon.a (mon.0) 13952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:07 smithi115 bash[46934]: audit 2022-04-22T22:18:07.296222+0000 mon.a (mon.0) 13953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:07 smithi115 bash[46934]: audit 2022-04-22T22:18:07.456329+0000 mon.a (mon.0) 13954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:07 smithi115 bash[45495]: audit 2022-04-22T22:18:07.296222+0000 mon.a (mon.0) 13953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:07 smithi115 bash[45495]: audit 2022-04-22T22:18:07.456329+0000 mon.a (mon.0) 13954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:07 smithi121 bash[36902]: audit 2022-04-22T22:18:07.296222+0000 mon.a (mon.0) 13953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:07 smithi121 bash[36902]: audit 2022-04-22T22:18:07.456329+0000 mon.a (mon.0) 13954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:08 smithi121 bash[36902]: cluster 2022-04-22T22:18:07.141031+0000 mgr.y (mgr.34107) 9960 : cluster [DBG] pgmap v10701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:08 smithi115 bash[45495]: cluster 2022-04-22T22:18:07.141031+0000 mgr.y (mgr.34107) 9960 : cluster [DBG] pgmap v10701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:08 smithi115 bash[46934]: cluster 2022-04-22T22:18:07.141031+0000 mgr.y (mgr.34107) 9960 : cluster [DBG] pgmap v10701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:10 smithi121 bash[36902]: cluster 2022-04-22T22:18:09.142126+0000 mgr.y (mgr.34107) 9961 : cluster [DBG] pgmap v10702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:10 smithi115 bash[45495]: cluster 2022-04-22T22:18:09.142126+0000 mgr.y (mgr.34107) 9961 : cluster [DBG] pgmap v10702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:10 smithi115 bash[46934]: cluster 2022-04-22T22:18:09.142126+0000 mgr.y (mgr.34107) 9961 : cluster [DBG] pgmap v10702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:11.881 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:18:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:11 smithi121 bash[36902]: audit 2022-04-22T22:18:11.070429+0000 mon.a (mon.0) 13955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:11 smithi115 bash[45495]: audit 2022-04-22T22:18:11.070429+0000 mon.a (mon.0) 13955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:11 smithi115 bash[46934]: audit 2022-04-22T22:18:11.070429+0000 mon.a (mon.0) 13955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:18:12.222 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 4h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:18:12.223 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:18:12.224 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:18:12.224 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:18:12.224 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:18:12.224 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:18:12.224 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:18:12.631 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:18:12.631 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:18:12.631 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:18:12.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:12.633 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:18:12.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:18:12.633 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:18:12.633 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:18:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:12 smithi121 bash[36902]: cluster 2022-04-22T22:18:11.143165+0000 mgr.y (mgr.34107) 9962 : cluster [DBG] pgmap v10703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:12 smithi121 bash[36902]: audit 2022-04-22T22:18:11.862089+0000 mgr.y (mgr.34107) 9963 : audit [DBG] from='client.55615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:12 smithi121 bash[36902]: audit 2022-04-22T22:18:12.200590+0000 mgr.y (mgr.34107) 9964 : audit [DBG] from='client.55621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:12 smithi121 bash[36902]: audit 2022-04-22T22:18:12.299833+0000 mon.a (mon.0) 13956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:12 smithi121 bash[36902]: audit 2022-04-22T22:18:12.459540+0000 mon.a (mon.0) 13957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:12.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[45495]: cluster 2022-04-22T22:18:11.143165+0000 mgr.y (mgr.34107) 9962 : cluster [DBG] pgmap v10703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[45495]: audit 2022-04-22T22:18:11.862089+0000 mgr.y (mgr.34107) 9963 : audit [DBG] from='client.55615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[45495]: audit 2022-04-22T22:18:12.200590+0000 mgr.y (mgr.34107) 9964 : audit [DBG] from='client.55621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[45495]: audit 2022-04-22T22:18:12.299833+0000 mon.a (mon.0) 13956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[45495]: audit 2022-04-22T22:18:12.459540+0000 mon.a (mon.0) 13957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[46934]: cluster 2022-04-22T22:18:11.143165+0000 mgr.y (mgr.34107) 9962 : cluster [DBG] pgmap v10703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[46934]: audit 2022-04-22T22:18:11.862089+0000 mgr.y (mgr.34107) 9963 : audit [DBG] from='client.55615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[46934]: audit 2022-04-22T22:18:12.200590+0000 mgr.y (mgr.34107) 9964 : audit [DBG] from='client.55621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[46934]: audit 2022-04-22T22:18:12.299833+0000 mon.a (mon.0) 13956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:12 smithi115 bash[46934]: audit 2022-04-22T22:18:12.459540+0000 mon.a (mon.0) 13957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:13 smithi121 bash[36902]: audit 2022-04-22T22:18:12.627676+0000 mon.c (mon.1) 344 : audit [DBG] from='client.? 172.21.15.115:0/165960989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:13 smithi121 bash[36902]: cluster 2022-04-22T22:18:13.144315+0000 mgr.y (mgr.34107) 9965 : cluster [DBG] pgmap v10704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:13 smithi115 bash[45495]: audit 2022-04-22T22:18:12.627676+0000 mon.c (mon.1) 344 : audit [DBG] from='client.? 172.21.15.115:0/165960989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:13 smithi115 bash[45495]: cluster 2022-04-22T22:18:13.144315+0000 mgr.y (mgr.34107) 9965 : cluster [DBG] pgmap v10704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:13 smithi115 bash[46934]: audit 2022-04-22T22:18:12.627676+0000 mon.c (mon.1) 344 : audit [DBG] from='client.? 172.21.15.115:0/165960989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:13 smithi115 bash[46934]: cluster 2022-04-22T22:18:13.144315+0000 mgr.y (mgr.34107) 9965 : cluster [DBG] pgmap v10704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:14.620Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:16 smithi121 bash[36902]: cluster 2022-04-22T22:18:15.145414+0000 mgr.y (mgr.34107) 9966 : cluster [DBG] pgmap v10705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:16 smithi121 bash[36902]: audit 2022-04-22T22:18:16.073850+0000 mon.a (mon.0) 13958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:16 smithi115 bash[45495]: cluster 2022-04-22T22:18:15.145414+0000 mgr.y (mgr.34107) 9966 : cluster [DBG] pgmap v10705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:16 smithi115 bash[45495]: audit 2022-04-22T22:18:16.073850+0000 mon.a (mon.0) 13958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:16 smithi115 bash[46934]: cluster 2022-04-22T22:18:15.145414+0000 mgr.y (mgr.34107) 9966 : cluster [DBG] pgmap v10705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:16 smithi115 bash[46934]: audit 2022-04-22T22:18:16.073850+0000 mon.a (mon.0) 13958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:17 smithi121 bash[36902]: audit 2022-04-22T22:18:16.640135+0000 mon.a (mon.0) 13959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:18:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:17 smithi121 bash[36902]: audit 2022-04-22T22:18:16.640970+0000 mon.a (mon.0) 13960 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:18:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:17 smithi121 bash[36902]: audit 2022-04-22T22:18:17.303503+0000 mon.a (mon.0) 13961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[45495]: audit 2022-04-22T22:18:16.640135+0000 mon.a (mon.0) 13959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[45495]: audit 2022-04-22T22:18:16.640970+0000 mon.a (mon.0) 13960 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[45495]: audit 2022-04-22T22:18:17.303503+0000 mon.a (mon.0) 13961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[46934]: audit 2022-04-22T22:18:16.640135+0000 mon.a (mon.0) 13959 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[46934]: audit 2022-04-22T22:18:16.640970+0000 mon.a (mon.0) 13960 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:18:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:17 smithi115 bash[46934]: audit 2022-04-22T22:18:17.303503+0000 mon.a (mon.0) 13961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:18 smithi121 bash[36902]: cluster 2022-04-22T22:18:17.146485+0000 mgr.y (mgr.34107) 9967 : cluster [DBG] pgmap v10706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:18 smithi121 bash[36902]: audit 2022-04-22T22:18:17.462449+0000 mon.a (mon.0) 13962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:18 smithi115 bash[45495]: cluster 2022-04-22T22:18:17.146485+0000 mgr.y (mgr.34107) 9967 : cluster [DBG] pgmap v10706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:18 smithi115 bash[45495]: audit 2022-04-22T22:18:17.462449+0000 mon.a (mon.0) 13962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:18 smithi115 bash[46934]: cluster 2022-04-22T22:18:17.146485+0000 mgr.y (mgr.34107) 9967 : cluster [DBG] pgmap v10706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:18 smithi115 bash[46934]: audit 2022-04-22T22:18:17.462449+0000 mon.a (mon.0) 13962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:20 smithi121 bash[36902]: cluster 2022-04-22T22:18:19.147720+0000 mgr.y (mgr.34107) 9968 : cluster [DBG] pgmap v10707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:20.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:20 smithi115 bash[45495]: cluster 2022-04-22T22:18:19.147720+0000 mgr.y (mgr.34107) 9968 : cluster [DBG] pgmap v10707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:20 smithi115 bash[46934]: cluster 2022-04-22T22:18:19.147720+0000 mgr.y (mgr.34107) 9968 : cluster [DBG] pgmap v10707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:21 smithi121 bash[36902]: audit 2022-04-22T22:18:21.077233+0000 mon.a (mon.0) 13963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:21 smithi115 bash[45495]: audit 2022-04-22T22:18:21.077233+0000 mon.a (mon.0) 13963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:21 smithi115 bash[46934]: audit 2022-04-22T22:18:21.077233+0000 mon.a (mon.0) 13963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:22 smithi121 bash[36902]: cluster 2022-04-22T22:18:21.148837+0000 mgr.y (mgr.34107) 9969 : cluster [DBG] pgmap v10708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:22 smithi121 bash[36902]: audit 2022-04-22T22:18:22.307120+0000 mon.a (mon.0) 13964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:22.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:22 smithi115 bash[45495]: cluster 2022-04-22T22:18:21.148837+0000 mgr.y (mgr.34107) 9969 : cluster [DBG] pgmap v10708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:22.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:22 smithi115 bash[45495]: audit 2022-04-22T22:18:22.307120+0000 mon.a (mon.0) 13964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:22.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:22 smithi115 bash[46934]: cluster 2022-04-22T22:18:21.148837+0000 mgr.y (mgr.34107) 9969 : cluster [DBG] pgmap v10708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:22.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:22 smithi115 bash[46934]: audit 2022-04-22T22:18:22.307120+0000 mon.a (mon.0) 13964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:22.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:23 smithi121 bash[36902]: audit 2022-04-22T22:18:22.465614+0000 mon.a (mon.0) 13965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:23 smithi115 bash[45495]: audit 2022-04-22T22:18:22.465614+0000 mon.a (mon.0) 13965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:23 smithi115 bash[46934]: audit 2022-04-22T22:18:22.465614+0000 mon.a (mon.0) 13965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:24 smithi121 bash[36902]: cluster 2022-04-22T22:18:23.149937+0000 mgr.y (mgr.34107) 9970 : cluster [DBG] pgmap v10709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:24.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:24 smithi115 bash[45495]: cluster 2022-04-22T22:18:23.149937+0000 mgr.y (mgr.34107) 9970 : cluster [DBG] pgmap v10709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:24.619Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:24 smithi115 bash[46934]: cluster 2022-04-22T22:18:23.149937+0000 mgr.y (mgr.34107) 9970 : cluster [DBG] pgmap v10709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:26 smithi121 bash[36902]: cluster 2022-04-22T22:18:25.150883+0000 mgr.y (mgr.34107) 9971 : cluster [DBG] pgmap v10710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:26 smithi121 bash[36902]: audit 2022-04-22T22:18:26.080146+0000 mon.a (mon.0) 13966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:26 smithi115 bash[45495]: cluster 2022-04-22T22:18:25.150883+0000 mgr.y (mgr.34107) 9971 : cluster [DBG] pgmap v10710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:26 smithi115 bash[45495]: audit 2022-04-22T22:18:26.080146+0000 mon.a (mon.0) 13966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:26 smithi115 bash[46934]: cluster 2022-04-22T22:18:25.150883+0000 mgr.y (mgr.34107) 9971 : cluster [DBG] pgmap v10710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:26 smithi115 bash[46934]: audit 2022-04-22T22:18:26.080146+0000 mon.a (mon.0) 13966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:27 smithi121 bash[36902]: audit 2022-04-22T22:18:27.310829+0000 mon.a (mon.0) 13967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:27 smithi115 bash[45495]: audit 2022-04-22T22:18:27.310829+0000 mon.a (mon.0) 13967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:27 smithi115 bash[46934]: audit 2022-04-22T22:18:27.310829+0000 mon.a (mon.0) 13967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:28 smithi121 bash[36902]: cluster 2022-04-22T22:18:27.152024+0000 mgr.y (mgr.34107) 9972 : cluster [DBG] pgmap v10711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:28 smithi121 bash[36902]: audit 2022-04-22T22:18:27.468716+0000 mon.a (mon.0) 13968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:28 smithi115 bash[45495]: cluster 2022-04-22T22:18:27.152024+0000 mgr.y (mgr.34107) 9972 : cluster [DBG] pgmap v10711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:28 smithi115 bash[45495]: audit 2022-04-22T22:18:27.468716+0000 mon.a (mon.0) 13968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:28 smithi115 bash[46934]: cluster 2022-04-22T22:18:27.152024+0000 mgr.y (mgr.34107) 9972 : cluster [DBG] pgmap v10711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:28 smithi115 bash[46934]: audit 2022-04-22T22:18:27.468716+0000 mon.a (mon.0) 13968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:30 smithi121 bash[36902]: cluster 2022-04-22T22:18:29.153082+0000 mgr.y (mgr.34107) 9973 : cluster [DBG] pgmap v10712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:30 smithi115 bash[45495]: cluster 2022-04-22T22:18:29.153082+0000 mgr.y (mgr.34107) 9973 : cluster [DBG] pgmap v10712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:30 smithi115 bash[46934]: cluster 2022-04-22T22:18:29.153082+0000 mgr.y (mgr.34107) 9973 : cluster [DBG] pgmap v10712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:31 smithi121 bash[36902]: audit 2022-04-22T22:18:31.083270+0000 mon.a (mon.0) 13969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:31 smithi115 bash[45495]: audit 2022-04-22T22:18:31.083270+0000 mon.a (mon.0) 13969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:31 smithi115 bash[46934]: audit 2022-04-22T22:18:31.083270+0000 mon.a (mon.0) 13969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:32 smithi121 bash[36902]: cluster 2022-04-22T22:18:31.154136+0000 mgr.y (mgr.34107) 9974 : cluster [DBG] pgmap v10713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:32 smithi121 bash[36902]: audit 2022-04-22T22:18:32.314897+0000 mon.a (mon.0) 13970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:32.773 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:32 smithi115 bash[45495]: cluster 2022-04-22T22:18:31.154136+0000 mgr.y (mgr.34107) 9974 : cluster [DBG] pgmap v10713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:32.773 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:32 smithi115 bash[45495]: audit 2022-04-22T22:18:32.314897+0000 mon.a (mon.0) 13970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:32.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:32 smithi115 bash[46934]: cluster 2022-04-22T22:18:31.154136+0000 mgr.y (mgr.34107) 9974 : cluster [DBG] pgmap v10713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:32.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:32 smithi115 bash[46934]: audit 2022-04-22T22:18:32.314897+0000 mon.a (mon.0) 13970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:32.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:33 smithi121 bash[36902]: audit 2022-04-22T22:18:32.471729+0000 mon.a (mon.0) 13971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:33 smithi115 bash[45495]: audit 2022-04-22T22:18:32.471729+0000 mon.a (mon.0) 13971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:33 smithi115 bash[46934]: audit 2022-04-22T22:18:32.471729+0000 mon.a (mon.0) 13971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:34 smithi121 bash[36902]: cluster 2022-04-22T22:18:33.155337+0000 mgr.y (mgr.34107) 9975 : cluster [DBG] pgmap v10714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:34 smithi115 bash[45495]: cluster 2022-04-22T22:18:33.155337+0000 mgr.y (mgr.34107) 9975 : cluster [DBG] pgmap v10714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:34 smithi115 bash[46934]: cluster 2022-04-22T22:18:33.155337+0000 mgr.y (mgr.34107) 9975 : cluster [DBG] pgmap v10714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:34.620Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:36 smithi121 bash[36902]: cluster 2022-04-22T22:18:35.156362+0000 mgr.y (mgr.34107) 9976 : cluster [DBG] pgmap v10715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:36 smithi121 bash[36902]: audit 2022-04-22T22:18:36.086380+0000 mon.a (mon.0) 13972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:36 smithi115 bash[45495]: cluster 2022-04-22T22:18:35.156362+0000 mgr.y (mgr.34107) 9976 : cluster [DBG] pgmap v10715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:36 smithi115 bash[45495]: audit 2022-04-22T22:18:36.086380+0000 mon.a (mon.0) 13972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:36 smithi115 bash[46934]: cluster 2022-04-22T22:18:35.156362+0000 mgr.y (mgr.34107) 9976 : cluster [DBG] pgmap v10715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:36 smithi115 bash[46934]: audit 2022-04-22T22:18:36.086380+0000 mon.a (mon.0) 13972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:37 smithi121 bash[36902]: audit 2022-04-22T22:18:37.318545+0000 mon.a (mon.0) 13973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:37 smithi115 bash[45495]: audit 2022-04-22T22:18:37.318545+0000 mon.a (mon.0) 13973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:37 smithi115 bash[46934]: audit 2022-04-22T22:18:37.318545+0000 mon.a (mon.0) 13973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:38 smithi121 bash[36902]: cluster 2022-04-22T22:18:37.157460+0000 mgr.y (mgr.34107) 9977 : cluster [DBG] pgmap v10716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:38 smithi121 bash[36902]: audit 2022-04-22T22:18:37.475728+0000 mon.a (mon.0) 13974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:38 smithi115 bash[45495]: cluster 2022-04-22T22:18:37.157460+0000 mgr.y (mgr.34107) 9977 : cluster [DBG] pgmap v10716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:38 smithi115 bash[45495]: audit 2022-04-22T22:18:37.475728+0000 mon.a (mon.0) 13974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:38 smithi115 bash[46934]: cluster 2022-04-22T22:18:37.157460+0000 mgr.y (mgr.34107) 9977 : cluster [DBG] pgmap v10716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:38 smithi115 bash[46934]: audit 2022-04-22T22:18:37.475728+0000 mon.a (mon.0) 13974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:40 smithi121 bash[36902]: cluster 2022-04-22T22:18:39.158566+0000 mgr.y (mgr.34107) 9978 : cluster [DBG] pgmap v10717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:40 smithi115 bash[45495]: cluster 2022-04-22T22:18:39.158566+0000 mgr.y (mgr.34107) 9978 : cluster [DBG] pgmap v10717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:40 smithi115 bash[46934]: cluster 2022-04-22T22:18:39.158566+0000 mgr.y (mgr.34107) 9978 : cluster [DBG] pgmap v10717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:41 smithi121 bash[36902]: audit 2022-04-22T22:18:41.089452+0000 mon.a (mon.0) 13975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:41 smithi115 bash[45495]: audit 2022-04-22T22:18:41.089452+0000 mon.a (mon.0) 13975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:41 smithi115 bash[46934]: audit 2022-04-22T22:18:41.089452+0000 mon.a (mon.0) 13975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:42 smithi121 bash[36902]: cluster 2022-04-22T22:18:41.159633+0000 mgr.y (mgr.34107) 9979 : cluster [DBG] pgmap v10718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:42 smithi121 bash[36902]: audit 2022-04-22T22:18:42.322404+0000 mon.a (mon.0) 13976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:42.773 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:42.769Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:42.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:42 smithi115 bash[46934]: cluster 2022-04-22T22:18:41.159633+0000 mgr.y (mgr.34107) 9979 : cluster [DBG] pgmap v10718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:42.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:42 smithi115 bash[46934]: audit 2022-04-22T22:18:42.322404+0000 mon.a (mon.0) 13976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:42.773 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:42 smithi115 bash[45495]: cluster 2022-04-22T22:18:41.159633+0000 mgr.y (mgr.34107) 9979 : cluster [DBG] pgmap v10718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:42.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:42 smithi115 bash[45495]: audit 2022-04-22T22:18:42.322404+0000 mon.a (mon.0) 13976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:43.005 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:18:43.347 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:18:43.347 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:18:43.347 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 3m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:18:43.347 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 2m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 2m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:18:43.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 4h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:18:43.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:18:43.350 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:18:43.750 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:18:43.750 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:18:43.750 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:18:43.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:18:43.752 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:18:43.752 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:18:43.752 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:18:43.752 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:18:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:43 smithi121 bash[36902]: audit 2022-04-22T22:18:42.478618+0000 mon.a (mon.0) 13977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:43 smithi115 bash[45495]: audit 2022-04-22T22:18:42.478618+0000 mon.a (mon.0) 13977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:43 smithi115 bash[46934]: audit 2022-04-22T22:18:42.478618+0000 mon.a (mon.0) 13977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:44 smithi121 bash[36902]: audit 2022-04-22T22:18:42.984672+0000 mgr.y (mgr.34107) 9980 : audit [DBG] from='client.45702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:44 smithi121 bash[36902]: cluster 2022-04-22T22:18:43.160737+0000 mgr.y (mgr.34107) 9981 : cluster [DBG] pgmap v10719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:44 smithi121 bash[36902]: audit 2022-04-22T22:18:43.325644+0000 mgr.y (mgr.34107) 9982 : audit [DBG] from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:44 smithi121 bash[36902]: audit 2022-04-22T22:18:43.746818+0000 mon.a (mon.0) 13978 : audit [DBG] from='client.? 172.21.15.115:0/444087054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:44.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[45495]: audit 2022-04-22T22:18:42.984672+0000 mgr.y (mgr.34107) 9980 : audit [DBG] from='client.45702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[45495]: cluster 2022-04-22T22:18:43.160737+0000 mgr.y (mgr.34107) 9981 : cluster [DBG] pgmap v10719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[45495]: audit 2022-04-22T22:18:43.325644+0000 mgr.y (mgr.34107) 9982 : audit [DBG] from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[45495]: audit 2022-04-22T22:18:43.746818+0000 mon.a (mon.0) 13978 : audit [DBG] from='client.? 172.21.15.115:0/444087054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[46934]: audit 2022-04-22T22:18:42.984672+0000 mgr.y (mgr.34107) 9980 : audit [DBG] from='client.45702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[46934]: cluster 2022-04-22T22:18:43.160737+0000 mgr.y (mgr.34107) 9981 : cluster [DBG] pgmap v10719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[46934]: audit 2022-04-22T22:18:43.325644+0000 mgr.y (mgr.34107) 9982 : audit [DBG] from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:18:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:44 smithi115 bash[46934]: audit 2022-04-22T22:18:43.746818+0000 mon.a (mon.0) 13978 : audit [DBG] from='client.? 172.21.15.115:0/444087054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:18:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:46 smithi121 bash[36902]: cluster 2022-04-22T22:18:45.161719+0000 mgr.y (mgr.34107) 9983 : cluster [DBG] pgmap v10720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:46 smithi121 bash[36902]: audit 2022-04-22T22:18:46.092676+0000 mon.a (mon.0) 13979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:46 smithi115 bash[45495]: cluster 2022-04-22T22:18:45.161719+0000 mgr.y (mgr.34107) 9983 : cluster [DBG] pgmap v10720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:46 smithi115 bash[45495]: audit 2022-04-22T22:18:46.092676+0000 mon.a (mon.0) 13979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:46 smithi115 bash[46934]: cluster 2022-04-22T22:18:45.161719+0000 mgr.y (mgr.34107) 9983 : cluster [DBG] pgmap v10720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:46 smithi115 bash[46934]: audit 2022-04-22T22:18:46.092676+0000 mon.a (mon.0) 13979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:47 smithi121 bash[36902]: audit 2022-04-22T22:18:47.325975+0000 mon.a (mon.0) 13980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:47 smithi115 bash[45495]: audit 2022-04-22T22:18:47.325975+0000 mon.a (mon.0) 13980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:47 smithi115 bash[46934]: audit 2022-04-22T22:18:47.325975+0000 mon.a (mon.0) 13980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:48 smithi121 bash[36902]: cluster 2022-04-22T22:18:47.162800+0000 mgr.y (mgr.34107) 9984 : cluster [DBG] pgmap v10721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:48 smithi121 bash[36902]: audit 2022-04-22T22:18:47.481028+0000 mon.a (mon.0) 13981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:48 smithi115 bash[45495]: cluster 2022-04-22T22:18:47.162800+0000 mgr.y (mgr.34107) 9984 : cluster [DBG] pgmap v10721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:48 smithi115 bash[45495]: audit 2022-04-22T22:18:47.481028+0000 mon.a (mon.0) 13981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:48 smithi115 bash[46934]: cluster 2022-04-22T22:18:47.162800+0000 mgr.y (mgr.34107) 9984 : cluster [DBG] pgmap v10721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:48 smithi115 bash[46934]: audit 2022-04-22T22:18:47.481028+0000 mon.a (mon.0) 13981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:50 smithi121 bash[36902]: cluster 2022-04-22T22:18:49.163875+0000 mgr.y (mgr.34107) 9985 : cluster [DBG] pgmap v10722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:50 smithi115 bash[45495]: cluster 2022-04-22T22:18:49.163875+0000 mgr.y (mgr.34107) 9985 : cluster [DBG] pgmap v10722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:50 smithi115 bash[46934]: cluster 2022-04-22T22:18:49.163875+0000 mgr.y (mgr.34107) 9985 : cluster [DBG] pgmap v10722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:51 smithi121 bash[36902]: audit 2022-04-22T22:18:51.095804+0000 mon.a (mon.0) 13982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:51.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:51 smithi115 bash[46934]: audit 2022-04-22T22:18:51.095804+0000 mon.a (mon.0) 13982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:51 smithi115 bash[45495]: audit 2022-04-22T22:18:51.095804+0000 mon.a (mon.0) 13982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:52 smithi121 bash[36902]: cluster 2022-04-22T22:18:51.164870+0000 mgr.y (mgr.34107) 9986 : cluster [DBG] pgmap v10723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:52 smithi121 bash[36902]: audit 2022-04-22T22:18:52.329253+0000 mon.a (mon.0) 13983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:52 smithi121 bash[36902]: audit 2022-04-22T22:18:52.483663+0000 mon.a (mon.0) 13984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:52.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[45495]: cluster 2022-04-22T22:18:51.164870+0000 mgr.y (mgr.34107) 9986 : cluster [DBG] pgmap v10723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:52.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[45495]: audit 2022-04-22T22:18:52.329253+0000 mon.a (mon.0) 13983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:52.774 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[45495]: audit 2022-04-22T22:18:52.483663+0000 mon.a (mon.0) 13984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:52.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[46934]: cluster 2022-04-22T22:18:51.164870+0000 mgr.y (mgr.34107) 9986 : cluster [DBG] pgmap v10723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:52.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[46934]: audit 2022-04-22T22:18:52.329253+0000 mon.a (mon.0) 13983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:52.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[46934]: audit 2022-04-22T22:18:52.483663+0000 mon.a (mon.0) 13984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:52.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:54 smithi121 bash[36902]: cluster 2022-04-22T22:18:53.165959+0000 mgr.y (mgr.34107) 9987 : cluster [DBG] pgmap v10724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:54 smithi115 bash[45495]: cluster 2022-04-22T22:18:53.165959+0000 mgr.y (mgr.34107) 9987 : cluster [DBG] pgmap v10724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:18:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:18:54.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:18:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:54 smithi115 bash[46934]: cluster 2022-04-22T22:18:53.165959+0000 mgr.y (mgr.34107) 9987 : cluster [DBG] pgmap v10724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:56 smithi121 bash[36902]: cluster 2022-04-22T22:18:55.166964+0000 mgr.y (mgr.34107) 9988 : cluster [DBG] pgmap v10725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:56 smithi121 bash[36902]: audit 2022-04-22T22:18:56.098925+0000 mon.a (mon.0) 13985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:56 smithi115 bash[45495]: cluster 2022-04-22T22:18:55.166964+0000 mgr.y (mgr.34107) 9988 : cluster [DBG] pgmap v10725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:56 smithi115 bash[45495]: audit 2022-04-22T22:18:56.098925+0000 mon.a (mon.0) 13985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:56 smithi115 bash[46934]: cluster 2022-04-22T22:18:55.166964+0000 mgr.y (mgr.34107) 9988 : cluster [DBG] pgmap v10725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:56 smithi115 bash[46934]: audit 2022-04-22T22:18:56.098925+0000 mon.a (mon.0) 13985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:18:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:57 smithi121 bash[36902]: audit 2022-04-22T22:18:57.332974+0000 mon.a (mon.0) 13986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:57 smithi121 bash[36902]: audit 2022-04-22T22:18:57.486586+0000 mon.a (mon.0) 13987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:57 smithi115 bash[45495]: audit 2022-04-22T22:18:57.332974+0000 mon.a (mon.0) 13986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:57 smithi115 bash[45495]: audit 2022-04-22T22:18:57.486586+0000 mon.a (mon.0) 13987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:57 smithi115 bash[46934]: audit 2022-04-22T22:18:57.332974+0000 mon.a (mon.0) 13986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:18:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:57 smithi115 bash[46934]: audit 2022-04-22T22:18:57.486586+0000 mon.a (mon.0) 13987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:18:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:18:58 smithi121 bash[36902]: cluster 2022-04-22T22:18:57.168174+0000 mgr.y (mgr.34107) 9989 : cluster [DBG] pgmap v10726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:18:58 smithi115 bash[45495]: cluster 2022-04-22T22:18:57.168174+0000 mgr.y (mgr.34107) 9989 : cluster [DBG] pgmap v10726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:18:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:18:58 smithi115 bash[46934]: cluster 2022-04-22T22:18:57.168174+0000 mgr.y (mgr.34107) 9989 : cluster [DBG] pgmap v10726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:00 smithi115 bash[45495]: cluster 2022-04-22T22:18:59.169239+0000 mgr.y (mgr.34107) 9990 : cluster [DBG] pgmap v10727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:00 smithi115 bash[46934]: cluster 2022-04-22T22:18:59.169239+0000 mgr.y (mgr.34107) 9990 : cluster [DBG] pgmap v10727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:00 smithi121 bash[36902]: cluster 2022-04-22T22:18:59.169239+0000 mgr.y (mgr.34107) 9990 : cluster [DBG] pgmap v10727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:01 smithi115 bash[45495]: audit 2022-04-22T22:19:01.102010+0000 mon.a (mon.0) 13988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:01 smithi115 bash[46934]: audit 2022-04-22T22:19:01.102010+0000 mon.a (mon.0) 13988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:01 smithi121 bash[36902]: audit 2022-04-22T22:19:01.102010+0000 mon.a (mon.0) 13988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:02.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:02.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[45495]: cluster 2022-04-22T22:19:01.170243+0000 mgr.y (mgr.34107) 9991 : cluster [DBG] pgmap v10728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[45495]: audit 2022-04-22T22:19:02.336803+0000 mon.a (mon.0) 13989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[45495]: audit 2022-04-22T22:19:02.489647+0000 mon.a (mon.0) 13990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[46934]: cluster 2022-04-22T22:19:01.170243+0000 mgr.y (mgr.34107) 9991 : cluster [DBG] pgmap v10728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[46934]: audit 2022-04-22T22:19:02.336803+0000 mon.a (mon.0) 13989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:02 smithi115 bash[46934]: audit 2022-04-22T22:19:02.489647+0000 mon.a (mon.0) 13990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:02 smithi121 bash[36902]: cluster 2022-04-22T22:19:01.170243+0000 mgr.y (mgr.34107) 9991 : cluster [DBG] pgmap v10728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:02 smithi121 bash[36902]: audit 2022-04-22T22:19:02.336803+0000 mon.a (mon.0) 13989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:02 smithi121 bash[36902]: audit 2022-04-22T22:19:02.489647+0000 mon.a (mon.0) 13990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:04 smithi115 bash[45495]: cluster 2022-04-22T22:19:03.171388+0000 mgr.y (mgr.34107) 9992 : cluster [DBG] pgmap v10729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:04.622Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:04 smithi115 bash[46934]: cluster 2022-04-22T22:19:03.171388+0000 mgr.y (mgr.34107) 9992 : cluster [DBG] pgmap v10729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:04 smithi121 bash[36902]: cluster 2022-04-22T22:19:03.171388+0000 mgr.y (mgr.34107) 9992 : cluster [DBG] pgmap v10729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: cluster 2022-04-22T22:19:05.172349+0000 mgr.y (mgr.34107) 9993 : cluster [DBG] pgmap v10730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: audit 2022-04-22T22:19:05.537171+0000 mon.a (mon.0) 13991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: audit 2022-04-22T22:19:05.539665+0000 mon.a (mon.0) 13992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: audit 2022-04-22T22:19:05.541067+0000 mon.a (mon.0) 13993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: audit 2022-04-22T22:19:05.906223+0000 mon.a (mon.0) 13994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:19:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[45495]: audit 2022-04-22T22:19:06.105043+0000 mon.a (mon.0) 13995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: cluster 2022-04-22T22:19:05.172349+0000 mgr.y (mgr.34107) 9993 : cluster [DBG] pgmap v10730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: audit 2022-04-22T22:19:05.537171+0000 mon.a (mon.0) 13991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: audit 2022-04-22T22:19:05.539665+0000 mon.a (mon.0) 13992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: audit 2022-04-22T22:19:05.541067+0000 mon.a (mon.0) 13993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: audit 2022-04-22T22:19:05.906223+0000 mon.a (mon.0) 13994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:19:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:06 smithi115 bash[46934]: audit 2022-04-22T22:19:06.105043+0000 mon.a (mon.0) 13995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: cluster 2022-04-22T22:19:05.172349+0000 mgr.y (mgr.34107) 9993 : cluster [DBG] pgmap v10730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: audit 2022-04-22T22:19:05.537171+0000 mon.a (mon.0) 13991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: audit 2022-04-22T22:19:05.539665+0000 mon.a (mon.0) 13992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: audit 2022-04-22T22:19:05.541067+0000 mon.a (mon.0) 13993 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: audit 2022-04-22T22:19:05.906223+0000 mon.a (mon.0) 13994 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:19:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:06 smithi121 bash[36902]: audit 2022-04-22T22:19:06.105043+0000 mon.a (mon.0) 13995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:07 smithi121 bash[36902]: audit 2022-04-22T22:19:07.340319+0000 mon.a (mon.0) 13996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:07 smithi121 bash[36902]: audit 2022-04-22T22:19:07.492526+0000 mon.a (mon.0) 13997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:07 smithi115 bash[45495]: audit 2022-04-22T22:19:07.340319+0000 mon.a (mon.0) 13996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:07 smithi115 bash[45495]: audit 2022-04-22T22:19:07.492526+0000 mon.a (mon.0) 13997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:07 smithi115 bash[46934]: audit 2022-04-22T22:19:07.340319+0000 mon.a (mon.0) 13996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:07 smithi115 bash[46934]: audit 2022-04-22T22:19:07.492526+0000 mon.a (mon.0) 13997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:08 smithi121 bash[36902]: cluster 2022-04-22T22:19:07.173447+0000 mgr.y (mgr.34107) 9994 : cluster [DBG] pgmap v10731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:08 smithi115 bash[45495]: cluster 2022-04-22T22:19:07.173447+0000 mgr.y (mgr.34107) 9994 : cluster [DBG] pgmap v10731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:08 smithi115 bash[46934]: cluster 2022-04-22T22:19:07.173447+0000 mgr.y (mgr.34107) 9994 : cluster [DBG] pgmap v10731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:10 smithi121 bash[36902]: cluster 2022-04-22T22:19:09.174515+0000 mgr.y (mgr.34107) 9995 : cluster [DBG] pgmap v10732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:10 smithi115 bash[45495]: cluster 2022-04-22T22:19:09.174515+0000 mgr.y (mgr.34107) 9995 : cluster [DBG] pgmap v10732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:10 smithi115 bash[46934]: cluster 2022-04-22T22:19:09.174515+0000 mgr.y (mgr.34107) 9995 : cluster [DBG] pgmap v10732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:11 smithi121 bash[36902]: audit 2022-04-22T22:19:11.108371+0000 mon.a (mon.0) 13998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:11 smithi115 bash[45495]: audit 2022-04-22T22:19:11.108371+0000 mon.a (mon.0) 13998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:11 smithi115 bash[46934]: audit 2022-04-22T22:19:11.108371+0000 mon.a (mon.0) 13998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:12 smithi121 bash[36902]: cluster 2022-04-22T22:19:11.175561+0000 mgr.y (mgr.34107) 9996 : cluster [DBG] pgmap v10733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:12 smithi121 bash[36902]: audit 2022-04-22T22:19:12.343847+0000 mon.a (mon.0) 13999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:12 smithi121 bash[36902]: audit 2022-04-22T22:19:12.495720+0000 mon.a (mon.0) 14000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:12.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[45495]: cluster 2022-04-22T22:19:11.175561+0000 mgr.y (mgr.34107) 9996 : cluster [DBG] pgmap v10733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[45495]: audit 2022-04-22T22:19:12.343847+0000 mon.a (mon.0) 13999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[45495]: audit 2022-04-22T22:19:12.495720+0000 mon.a (mon.0) 14000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[46934]: cluster 2022-04-22T22:19:11.175561+0000 mgr.y (mgr.34107) 9996 : cluster [DBG] pgmap v10733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[46934]: audit 2022-04-22T22:19:12.343847+0000 mon.a (mon.0) 13999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:12 smithi115 bash[46934]: audit 2022-04-22T22:19:12.495720+0000 mon.a (mon.0) 14000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:14.120 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:19:14.474 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:19:14.474 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:19:14.474 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:19:14.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:19:14.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:19:14.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:19:14.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:19:14.475 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:19:14.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:19:14.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:19:14.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:19:14.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:19:14.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:19:14.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:19:14.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 4h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:19:14.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:19:14.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:19:14.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:19:14.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:19:14.478 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:19:14.479 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:19:14.883 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:19:14.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:19:14.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:19:14.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:14.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:14.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:19:14.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:19:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:14 smithi121 bash[36902]: cluster 2022-04-22T22:19:13.176549+0000 mgr.y (mgr.34107) 9997 : cluster [DBG] pgmap v10734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:14 smithi115 bash[45495]: cluster 2022-04-22T22:19:13.176549+0000 mgr.y (mgr.34107) 9997 : cluster [DBG] pgmap v10734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:14 smithi115 bash[46934]: cluster 2022-04-22T22:19:13.176549+0000 mgr.y (mgr.34107) 9997 : cluster [DBG] pgmap v10734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:14.621Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:15 smithi121 bash[36902]: audit 2022-04-22T22:19:14.100996+0000 mgr.y (mgr.34107) 9998 : audit [DBG] from='client.55651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:15 smithi121 bash[36902]: audit 2022-04-22T22:19:14.450191+0000 mgr.y (mgr.34107) 9999 : audit [DBG] from='client.55657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:15 smithi121 bash[36902]: audit 2022-04-22T22:19:14.879394+0000 mon.a (mon.0) 14001 : audit [DBG] from='client.? 172.21.15.115:0/1292278815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[45495]: audit 2022-04-22T22:19:14.100996+0000 mgr.y (mgr.34107) 9998 : audit [DBG] from='client.55651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[45495]: audit 2022-04-22T22:19:14.450191+0000 mgr.y (mgr.34107) 9999 : audit [DBG] from='client.55657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[45495]: audit 2022-04-22T22:19:14.879394+0000 mon.a (mon.0) 14001 : audit [DBG] from='client.? 172.21.15.115:0/1292278815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[46934]: audit 2022-04-22T22:19:14.100996+0000 mgr.y (mgr.34107) 9998 : audit [DBG] from='client.55651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[46934]: audit 2022-04-22T22:19:14.450191+0000 mgr.y (mgr.34107) 9999 : audit [DBG] from='client.55657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:15 smithi115 bash[46934]: audit 2022-04-22T22:19:14.879394+0000 mon.a (mon.0) 14001 : audit [DBG] from='client.? 172.21.15.115:0/1292278815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:16 smithi121 bash[36902]: cluster 2022-04-22T22:19:15.177462+0000 mgr.y (mgr.34107) 10000 : cluster [DBG] pgmap v10735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:16 smithi121 bash[36902]: audit 2022-04-22T22:19:16.111372+0000 mon.a (mon.0) 14002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:16 smithi115 bash[45495]: cluster 2022-04-22T22:19:15.177462+0000 mgr.y (mgr.34107) 10000 : cluster [DBG] pgmap v10735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:16 smithi115 bash[45495]: audit 2022-04-22T22:19:16.111372+0000 mon.a (mon.0) 14002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:16 smithi115 bash[46934]: cluster 2022-04-22T22:19:15.177462+0000 mgr.y (mgr.34107) 10000 : cluster [DBG] pgmap v10735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:16 smithi115 bash[46934]: audit 2022-04-22T22:19:16.111372+0000 mon.a (mon.0) 14002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:17 smithi121 bash[36902]: audit 2022-04-22T22:19:16.649933+0000 mon.a (mon.0) 14003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:19:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:17 smithi121 bash[36902]: audit 2022-04-22T22:19:16.650516+0000 mon.a (mon.0) 14004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:19:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:17 smithi121 bash[36902]: cluster 2022-04-22T22:19:17.178601+0000 mgr.y (mgr.34107) 10001 : cluster [DBG] pgmap v10736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:17 smithi121 bash[36902]: audit 2022-04-22T22:19:17.347690+0000 mon.a (mon.0) 14005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:17 smithi121 bash[36902]: audit 2022-04-22T22:19:17.498680+0000 mon.a (mon.0) 14006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[45495]: audit 2022-04-22T22:19:16.649933+0000 mon.a (mon.0) 14003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:19:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[45495]: audit 2022-04-22T22:19:16.650516+0000 mon.a (mon.0) 14004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:19:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[45495]: cluster 2022-04-22T22:19:17.178601+0000 mgr.y (mgr.34107) 10001 : cluster [DBG] pgmap v10736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[45495]: audit 2022-04-22T22:19:17.347690+0000 mon.a (mon.0) 14005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[45495]: audit 2022-04-22T22:19:17.498680+0000 mon.a (mon.0) 14006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[46934]: audit 2022-04-22T22:19:16.649933+0000 mon.a (mon.0) 14003 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:19:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[46934]: audit 2022-04-22T22:19:16.650516+0000 mon.a (mon.0) 14004 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:19:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[46934]: cluster 2022-04-22T22:19:17.178601+0000 mgr.y (mgr.34107) 10001 : cluster [DBG] pgmap v10736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[46934]: audit 2022-04-22T22:19:17.347690+0000 mon.a (mon.0) 14005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:17 smithi115 bash[46934]: audit 2022-04-22T22:19:17.498680+0000 mon.a (mon.0) 14006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:20.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:20 smithi121 bash[36902]: cluster 2022-04-22T22:19:19.179702+0000 mgr.y (mgr.34107) 10002 : cluster [DBG] pgmap v10737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:20 smithi115 bash[45495]: cluster 2022-04-22T22:19:19.179702+0000 mgr.y (mgr.34107) 10002 : cluster [DBG] pgmap v10737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:20 smithi115 bash[46934]: cluster 2022-04-22T22:19:19.179702+0000 mgr.y (mgr.34107) 10002 : cluster [DBG] pgmap v10737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:21 smithi121 bash[36902]: audit 2022-04-22T22:19:21.114665+0000 mon.a (mon.0) 14007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:21 smithi115 bash[45495]: audit 2022-04-22T22:19:21.114665+0000 mon.a (mon.0) 14007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:21 smithi115 bash[46934]: audit 2022-04-22T22:19:21.114665+0000 mon.a (mon.0) 14007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:22 smithi121 bash[36902]: cluster 2022-04-22T22:19:21.180800+0000 mgr.y (mgr.34107) 10003 : cluster [DBG] pgmap v10738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:22 smithi121 bash[36902]: audit 2022-04-22T22:19:22.351285+0000 mon.a (mon.0) 14008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:22.775 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:22 smithi115 bash[45495]: cluster 2022-04-22T22:19:21.180800+0000 mgr.y (mgr.34107) 10003 : cluster [DBG] pgmap v10738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:22.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:22 smithi115 bash[45495]: audit 2022-04-22T22:19:22.351285+0000 mon.a (mon.0) 14008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:22.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:22 smithi115 bash[46934]: cluster 2022-04-22T22:19:21.180800+0000 mgr.y (mgr.34107) 10003 : cluster [DBG] pgmap v10738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:22.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:22 smithi115 bash[46934]: audit 2022-04-22T22:19:22.351285+0000 mon.a (mon.0) 14008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:22.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:23 smithi121 bash[36902]: audit 2022-04-22T22:19:22.501681+0000 mon.a (mon.0) 14009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:23 smithi115 bash[45495]: audit 2022-04-22T22:19:22.501681+0000 mon.a (mon.0) 14009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:23 smithi115 bash[46934]: audit 2022-04-22T22:19:22.501681+0000 mon.a (mon.0) 14009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:24 smithi121 bash[36902]: cluster 2022-04-22T22:19:23.182015+0000 mgr.y (mgr.34107) 10004 : cluster [DBG] pgmap v10739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:24 smithi115 bash[45495]: cluster 2022-04-22T22:19:23.182015+0000 mgr.y (mgr.34107) 10004 : cluster [DBG] pgmap v10739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:24.623Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:24 smithi115 bash[46934]: cluster 2022-04-22T22:19:23.182015+0000 mgr.y (mgr.34107) 10004 : cluster [DBG] pgmap v10739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:26 smithi121 bash[36902]: cluster 2022-04-22T22:19:25.183015+0000 mgr.y (mgr.34107) 10005 : cluster [DBG] pgmap v10740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:26 smithi121 bash[36902]: audit 2022-04-22T22:19:26.117773+0000 mon.a (mon.0) 14010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:26 smithi115 bash[45495]: cluster 2022-04-22T22:19:25.183015+0000 mgr.y (mgr.34107) 10005 : cluster [DBG] pgmap v10740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:26 smithi115 bash[45495]: audit 2022-04-22T22:19:26.117773+0000 mon.a (mon.0) 14010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:26 smithi115 bash[46934]: cluster 2022-04-22T22:19:25.183015+0000 mgr.y (mgr.34107) 10005 : cluster [DBG] pgmap v10740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:26 smithi115 bash[46934]: audit 2022-04-22T22:19:26.117773+0000 mon.a (mon.0) 14010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:27 smithi121 bash[36902]: audit 2022-04-22T22:19:27.354957+0000 mon.a (mon.0) 14011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:27 smithi115 bash[45495]: audit 2022-04-22T22:19:27.354957+0000 mon.a (mon.0) 14011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:27 smithi115 bash[46934]: audit 2022-04-22T22:19:27.354957+0000 mon.a (mon.0) 14011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:28 smithi121 bash[36902]: cluster 2022-04-22T22:19:27.184237+0000 mgr.y (mgr.34107) 10006 : cluster [DBG] pgmap v10741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:28 smithi121 bash[36902]: audit 2022-04-22T22:19:27.504724+0000 mon.a (mon.0) 14012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:28 smithi115 bash[45495]: cluster 2022-04-22T22:19:27.184237+0000 mgr.y (mgr.34107) 10006 : cluster [DBG] pgmap v10741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:28 smithi115 bash[45495]: audit 2022-04-22T22:19:27.504724+0000 mon.a (mon.0) 14012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:28 smithi115 bash[46934]: cluster 2022-04-22T22:19:27.184237+0000 mgr.y (mgr.34107) 10006 : cluster [DBG] pgmap v10741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:28 smithi115 bash[46934]: audit 2022-04-22T22:19:27.504724+0000 mon.a (mon.0) 14012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:30 smithi121 bash[36902]: cluster 2022-04-22T22:19:29.185329+0000 mgr.y (mgr.34107) 10007 : cluster [DBG] pgmap v10742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:30 smithi115 bash[45495]: cluster 2022-04-22T22:19:29.185329+0000 mgr.y (mgr.34107) 10007 : cluster [DBG] pgmap v10742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:30 smithi115 bash[46934]: cluster 2022-04-22T22:19:29.185329+0000 mgr.y (mgr.34107) 10007 : cluster [DBG] pgmap v10742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:31 smithi121 bash[36902]: audit 2022-04-22T22:19:31.120972+0000 mon.a (mon.0) 14013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:31 smithi115 bash[45495]: audit 2022-04-22T22:19:31.120972+0000 mon.a (mon.0) 14013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:31 smithi115 bash[46934]: audit 2022-04-22T22:19:31.120972+0000 mon.a (mon.0) 14013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:32 smithi121 bash[36902]: cluster 2022-04-22T22:19:31.186380+0000 mgr.y (mgr.34107) 10008 : cluster [DBG] pgmap v10743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:32 smithi121 bash[36902]: audit 2022-04-22T22:19:32.358556+0000 mon.a (mon.0) 14014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:32.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:32 smithi115 bash[45495]: cluster 2022-04-22T22:19:31.186380+0000 mgr.y (mgr.34107) 10008 : cluster [DBG] pgmap v10743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:32.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:32 smithi115 bash[45495]: audit 2022-04-22T22:19:32.358556+0000 mon.a (mon.0) 14014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:32.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:32 smithi115 bash[46934]: cluster 2022-04-22T22:19:31.186380+0000 mgr.y (mgr.34107) 10008 : cluster [DBG] pgmap v10743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:32.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:32 smithi115 bash[46934]: audit 2022-04-22T22:19:32.358556+0000 mon.a (mon.0) 14014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:32.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:33 smithi121 bash[36902]: audit 2022-04-22T22:19:32.507729+0000 mon.a (mon.0) 14015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:33 smithi115 bash[46934]: audit 2022-04-22T22:19:32.507729+0000 mon.a (mon.0) 14015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:33 smithi115 bash[45495]: audit 2022-04-22T22:19:32.507729+0000 mon.a (mon.0) 14015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:34 smithi121 bash[36902]: cluster 2022-04-22T22:19:33.187497+0000 mgr.y (mgr.34107) 10009 : cluster [DBG] pgmap v10744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:34 smithi115 bash[45495]: cluster 2022-04-22T22:19:33.187497+0000 mgr.y (mgr.34107) 10009 : cluster [DBG] pgmap v10744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:34.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:34 smithi115 bash[46934]: cluster 2022-04-22T22:19:33.187497+0000 mgr.y (mgr.34107) 10009 : cluster [DBG] pgmap v10744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:36 smithi121 bash[36902]: cluster 2022-04-22T22:19:35.188546+0000 mgr.y (mgr.34107) 10010 : cluster [DBG] pgmap v10745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:36 smithi121 bash[36902]: audit 2022-04-22T22:19:36.124078+0000 mon.a (mon.0) 14016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:36 smithi115 bash[45495]: cluster 2022-04-22T22:19:35.188546+0000 mgr.y (mgr.34107) 10010 : cluster [DBG] pgmap v10745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:36 smithi115 bash[45495]: audit 2022-04-22T22:19:36.124078+0000 mon.a (mon.0) 14016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:36 smithi115 bash[46934]: cluster 2022-04-22T22:19:35.188546+0000 mgr.y (mgr.34107) 10010 : cluster [DBG] pgmap v10745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:36 smithi115 bash[46934]: audit 2022-04-22T22:19:36.124078+0000 mon.a (mon.0) 14016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:37 smithi121 bash[36902]: audit 2022-04-22T22:19:37.362411+0000 mon.a (mon.0) 14017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:37 smithi115 bash[45495]: audit 2022-04-22T22:19:37.362411+0000 mon.a (mon.0) 14017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:37 smithi115 bash[46934]: audit 2022-04-22T22:19:37.362411+0000 mon.a (mon.0) 14017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:38 smithi121 bash[36902]: cluster 2022-04-22T22:19:37.189537+0000 mgr.y (mgr.34107) 10011 : cluster [DBG] pgmap v10746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:38 smithi121 bash[36902]: audit 2022-04-22T22:19:37.510812+0000 mon.a (mon.0) 14018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:38 smithi115 bash[45495]: cluster 2022-04-22T22:19:37.189537+0000 mgr.y (mgr.34107) 10011 : cluster [DBG] pgmap v10746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:38 smithi115 bash[45495]: audit 2022-04-22T22:19:37.510812+0000 mon.a (mon.0) 14018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:38 smithi115 bash[46934]: cluster 2022-04-22T22:19:37.189537+0000 mgr.y (mgr.34107) 10011 : cluster [DBG] pgmap v10746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:38 smithi115 bash[46934]: audit 2022-04-22T22:19:37.510812+0000 mon.a (mon.0) 14018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:40 smithi121 bash[36902]: cluster 2022-04-22T22:19:39.190614+0000 mgr.y (mgr.34107) 10012 : cluster [DBG] pgmap v10747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:40 smithi115 bash[45495]: cluster 2022-04-22T22:19:39.190614+0000 mgr.y (mgr.34107) 10012 : cluster [DBG] pgmap v10747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:40 smithi115 bash[46934]: cluster 2022-04-22T22:19:39.190614+0000 mgr.y (mgr.34107) 10012 : cluster [DBG] pgmap v10747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:41 smithi121 bash[36902]: audit 2022-04-22T22:19:41.127031+0000 mon.a (mon.0) 14019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:41 smithi115 bash[45495]: audit 2022-04-22T22:19:41.127031+0000 mon.a (mon.0) 14019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:41 smithi115 bash[46934]: audit 2022-04-22T22:19:41.127031+0000 mon.a (mon.0) 14019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:42 smithi121 bash[36902]: cluster 2022-04-22T22:19:41.191641+0000 mgr.y (mgr.34107) 10013 : cluster [DBG] pgmap v10748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:42 smithi121 bash[36902]: audit 2022-04-22T22:19:42.366304+0000 mon.a (mon.0) 14020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:42.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:42 smithi115 bash[45495]: cluster 2022-04-22T22:19:41.191641+0000 mgr.y (mgr.34107) 10013 : cluster [DBG] pgmap v10748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:42.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:42 smithi115 bash[45495]: audit 2022-04-22T22:19:42.366304+0000 mon.a (mon.0) 14020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:42.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:42 smithi115 bash[46934]: cluster 2022-04-22T22:19:41.191641+0000 mgr.y (mgr.34107) 10013 : cluster [DBG] pgmap v10748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:42.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:42 smithi115 bash[46934]: audit 2022-04-22T22:19:42.366304+0000 mon.a (mon.0) 14020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:42.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:43 smithi121 bash[36902]: audit 2022-04-22T22:19:42.513957+0000 mon.a (mon.0) 14021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:43 smithi115 bash[45495]: audit 2022-04-22T22:19:42.513957+0000 mon.a (mon.0) 14021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:43 smithi115 bash[46934]: audit 2022-04-22T22:19:42.513957+0000 mon.a (mon.0) 14021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:44 smithi121 bash[36902]: cluster 2022-04-22T22:19:43.192701+0000 mgr.y (mgr.34107) 10014 : cluster [DBG] pgmap v10749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:44 smithi115 bash[45495]: cluster 2022-04-22T22:19:43.192701+0000 mgr.y (mgr.34107) 10014 : cluster [DBG] pgmap v10749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:44.627Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:44 smithi115 bash[46934]: cluster 2022-04-22T22:19:43.192701+0000 mgr.y (mgr.34107) 10014 : cluster [DBG] pgmap v10749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:45.261 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:19:45.600 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:19:45.600 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:19:45.600 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 4m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:19:45.600 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:19:45.600 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 3m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 3m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 4m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 3m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:19:45.601 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 4h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 4h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:19:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:19:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:19:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:19:46.011 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:19:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:46 smithi121 bash[36902]: cluster 2022-04-22T22:19:45.193751+0000 mgr.y (mgr.34107) 10015 : cluster [DBG] pgmap v10750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:46 smithi121 bash[36902]: audit 2022-04-22T22:19:45.240928+0000 mgr.y (mgr.34107) 10016 : audit [DBG] from='client.45738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:46 smithi121 bash[36902]: audit 2022-04-22T22:19:46.005900+0000 mon.a (mon.0) 14022 : audit [DBG] from='client.? 172.21.15.115:0/40874185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:46 smithi121 bash[36902]: audit 2022-04-22T22:19:46.129957+0000 mon.a (mon.0) 14023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[45495]: cluster 2022-04-22T22:19:45.193751+0000 mgr.y (mgr.34107) 10015 : cluster [DBG] pgmap v10750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[45495]: audit 2022-04-22T22:19:45.240928+0000 mgr.y (mgr.34107) 10016 : audit [DBG] from='client.45738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[45495]: audit 2022-04-22T22:19:46.005900+0000 mon.a (mon.0) 14022 : audit [DBG] from='client.? 172.21.15.115:0/40874185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[45495]: audit 2022-04-22T22:19:46.129957+0000 mon.a (mon.0) 14023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[46934]: cluster 2022-04-22T22:19:45.193751+0000 mgr.y (mgr.34107) 10015 : cluster [DBG] pgmap v10750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[46934]: audit 2022-04-22T22:19:45.240928+0000 mgr.y (mgr.34107) 10016 : audit [DBG] from='client.45738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[46934]: audit 2022-04-22T22:19:46.005900+0000 mon.a (mon.0) 14022 : audit [DBG] from='client.? 172.21.15.115:0/40874185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:46 smithi115 bash[46934]: audit 2022-04-22T22:19:46.129957+0000 mon.a (mon.0) 14023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:47 smithi121 bash[36902]: audit 2022-04-22T22:19:45.582989+0000 mgr.y (mgr.34107) 10017 : audit [DBG] from='client.55675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:47 smithi121 bash[36902]: audit 2022-04-22T22:19:47.370182+0000 mon.a (mon.0) 14024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:47 smithi115 bash[45495]: audit 2022-04-22T22:19:45.582989+0000 mgr.y (mgr.34107) 10017 : audit [DBG] from='client.55675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:47 smithi115 bash[45495]: audit 2022-04-22T22:19:47.370182+0000 mon.a (mon.0) 14024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:47 smithi115 bash[46934]: audit 2022-04-22T22:19:45.582989+0000 mgr.y (mgr.34107) 10017 : audit [DBG] from='client.55675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:19:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:47 smithi115 bash[46934]: audit 2022-04-22T22:19:47.370182+0000 mon.a (mon.0) 14024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:48 smithi121 bash[36902]: cluster 2022-04-22T22:19:47.194888+0000 mgr.y (mgr.34107) 10018 : cluster [DBG] pgmap v10751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:48 smithi121 bash[36902]: audit 2022-04-22T22:19:47.516999+0000 mon.a (mon.0) 14025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:48 smithi115 bash[45495]: cluster 2022-04-22T22:19:47.194888+0000 mgr.y (mgr.34107) 10018 : cluster [DBG] pgmap v10751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:48 smithi115 bash[45495]: audit 2022-04-22T22:19:47.516999+0000 mon.a (mon.0) 14025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:48 smithi115 bash[46934]: cluster 2022-04-22T22:19:47.194888+0000 mgr.y (mgr.34107) 10018 : cluster [DBG] pgmap v10751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:48 smithi115 bash[46934]: audit 2022-04-22T22:19:47.516999+0000 mon.a (mon.0) 14025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:50 smithi121 bash[36902]: cluster 2022-04-22T22:19:49.196015+0000 mgr.y (mgr.34107) 10019 : cluster [DBG] pgmap v10752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:50 smithi115 bash[45495]: cluster 2022-04-22T22:19:49.196015+0000 mgr.y (mgr.34107) 10019 : cluster [DBG] pgmap v10752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:50 smithi115 bash[46934]: cluster 2022-04-22T22:19:49.196015+0000 mgr.y (mgr.34107) 10019 : cluster [DBG] pgmap v10752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:51 smithi121 bash[36902]: audit 2022-04-22T22:19:51.133333+0000 mon.a (mon.0) 14026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:51 smithi115 bash[45495]: audit 2022-04-22T22:19:51.133333+0000 mon.a (mon.0) 14026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:51 smithi115 bash[46934]: audit 2022-04-22T22:19:51.133333+0000 mon.a (mon.0) 14026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:52 smithi121 bash[36902]: cluster 2022-04-22T22:19:51.197084+0000 mgr.y (mgr.34107) 10020 : cluster [DBG] pgmap v10753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:52 smithi121 bash[36902]: audit 2022-04-22T22:19:52.373748+0000 mon.a (mon.0) 14027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:52.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:52 smithi115 bash[46934]: cluster 2022-04-22T22:19:51.197084+0000 mgr.y (mgr.34107) 10020 : cluster [DBG] pgmap v10753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:52.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:52 smithi115 bash[46934]: audit 2022-04-22T22:19:52.373748+0000 mon.a (mon.0) 14027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:52.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:52 smithi115 bash[45495]: cluster 2022-04-22T22:19:51.197084+0000 mgr.y (mgr.34107) 10020 : cluster [DBG] pgmap v10753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:52.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:52 smithi115 bash[45495]: audit 2022-04-22T22:19:52.373748+0000 mon.a (mon.0) 14027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:52.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:53 smithi121 bash[36902]: audit 2022-04-22T22:19:52.519878+0000 mon.a (mon.0) 14028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:53 smithi115 bash[45495]: audit 2022-04-22T22:19:52.519878+0000 mon.a (mon.0) 14028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:53 smithi115 bash[46934]: audit 2022-04-22T22:19:52.519878+0000 mon.a (mon.0) 14028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:54.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:54 smithi121 bash[36902]: cluster 2022-04-22T22:19:53.198220+0000 mgr.y (mgr.34107) 10021 : cluster [DBG] pgmap v10754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:54 smithi115 bash[45495]: cluster 2022-04-22T22:19:53.198220+0000 mgr.y (mgr.34107) 10021 : cluster [DBG] pgmap v10754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:54 smithi115 bash[46934]: cluster 2022-04-22T22:19:53.198220+0000 mgr.y (mgr.34107) 10021 : cluster [DBG] pgmap v10754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:19:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:19:54.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:19:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:56 smithi121 bash[36902]: cluster 2022-04-22T22:19:55.199287+0000 mgr.y (mgr.34107) 10022 : cluster [DBG] pgmap v10755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:56 smithi121 bash[36902]: audit 2022-04-22T22:19:56.136398+0000 mon.a (mon.0) 14029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:56 smithi115 bash[45495]: cluster 2022-04-22T22:19:55.199287+0000 mgr.y (mgr.34107) 10022 : cluster [DBG] pgmap v10755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:56 smithi115 bash[45495]: audit 2022-04-22T22:19:56.136398+0000 mon.a (mon.0) 14029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:56 smithi115 bash[46934]: cluster 2022-04-22T22:19:55.199287+0000 mgr.y (mgr.34107) 10022 : cluster [DBG] pgmap v10755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:56 smithi115 bash[46934]: audit 2022-04-22T22:19:56.136398+0000 mon.a (mon.0) 14029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:19:57.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:57 smithi115 bash[46934]: audit 2022-04-22T22:19:57.377780+0000 mon.a (mon.0) 14030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:57 smithi115 bash[45495]: audit 2022-04-22T22:19:57.377780+0000 mon.a (mon.0) 14030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:57 smithi121 bash[36902]: audit 2022-04-22T22:19:57.377780+0000 mon.a (mon.0) 14030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:19:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:58 smithi115 bash[45495]: cluster 2022-04-22T22:19:57.200511+0000 mgr.y (mgr.34107) 10023 : cluster [DBG] pgmap v10756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:19:58 smithi115 bash[45495]: audit 2022-04-22T22:19:57.522638+0000 mon.a (mon.0) 14031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:58 smithi115 bash[46934]: cluster 2022-04-22T22:19:57.200511+0000 mgr.y (mgr.34107) 10023 : cluster [DBG] pgmap v10756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:19:58 smithi115 bash[46934]: audit 2022-04-22T22:19:57.522638+0000 mon.a (mon.0) 14031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:19:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:58 smithi121 bash[36902]: cluster 2022-04-22T22:19:57.200511+0000 mgr.y (mgr.34107) 10023 : cluster [DBG] pgmap v10756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:19:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:19:58 smithi121 bash[36902]: audit 2022-04-22T22:19:57.522638+0000 mon.a (mon.0) 14031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:00 smithi115 bash[45495]: cluster 2022-04-22T22:19:59.201641+0000 mgr.y (mgr.34107) 10024 : cluster [DBG] pgmap v10757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:00 smithi115 bash[45495]: cluster 2022-04-22T22:20:00.000142+0000 mon.a (mon.0) 14032 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:20:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:00 smithi115 bash[46934]: cluster 2022-04-22T22:19:59.201641+0000 mgr.y (mgr.34107) 10024 : cluster [DBG] pgmap v10757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:00 smithi115 bash[46934]: cluster 2022-04-22T22:20:00.000142+0000 mon.a (mon.0) 14032 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:20:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:00 smithi121 bash[36902]: cluster 2022-04-22T22:19:59.201641+0000 mgr.y (mgr.34107) 10024 : cluster [DBG] pgmap v10757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:00 smithi121 bash[36902]: cluster 2022-04-22T22:20:00.000142+0000 mon.a (mon.0) 14032 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:20:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:01 smithi115 bash[45495]: audit 2022-04-22T22:20:01.139575+0000 mon.a (mon.0) 14033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:01 smithi115 bash[46934]: audit 2022-04-22T22:20:01.139575+0000 mon.a (mon.0) 14033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:01 smithi121 bash[36902]: audit 2022-04-22T22:20:01.139575+0000 mon.a (mon.0) 14033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:02.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[45495]: cluster 2022-04-22T22:20:01.202650+0000 mgr.y (mgr.34107) 10025 : cluster [DBG] pgmap v10758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[45495]: audit 2022-04-22T22:20:02.381321+0000 mon.a (mon.0) 14034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[45495]: audit 2022-04-22T22:20:02.525200+0000 mon.a (mon.0) 14035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:02.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:02.774Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[46934]: cluster 2022-04-22T22:20:01.202650+0000 mgr.y (mgr.34107) 10025 : cluster [DBG] pgmap v10758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[46934]: audit 2022-04-22T22:20:02.381321+0000 mon.a (mon.0) 14034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:02 smithi115 bash[46934]: audit 2022-04-22T22:20:02.525200+0000 mon.a (mon.0) 14035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:02 smithi121 bash[36902]: cluster 2022-04-22T22:20:01.202650+0000 mgr.y (mgr.34107) 10025 : cluster [DBG] pgmap v10758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:02 smithi121 bash[36902]: audit 2022-04-22T22:20:02.381321+0000 mon.a (mon.0) 14034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:02 smithi121 bash[36902]: audit 2022-04-22T22:20:02.525200+0000 mon.a (mon.0) 14035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:04 smithi115 bash[45495]: cluster 2022-04-22T22:20:03.203823+0000 mgr.y (mgr.34107) 10026 : cluster [DBG] pgmap v10759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:04.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:04 smithi115 bash[46934]: cluster 2022-04-22T22:20:03.203823+0000 mgr.y (mgr.34107) 10026 : cluster [DBG] pgmap v10759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:04 smithi121 bash[36902]: cluster 2022-04-22T22:20:03.203823+0000 mgr.y (mgr.34107) 10026 : cluster [DBG] pgmap v10759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: cluster 2022-04-22T22:20:05.204848+0000 mgr.y (mgr.34107) 10027 : cluster [DBG] pgmap v10760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: audit 2022-04-22T22:20:05.913483+0000 mon.a (mon.0) 14036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:20:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: audit 2022-04-22T22:20:05.916470+0000 mon.a (mon.0) 14037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:20:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: audit 2022-04-22T22:20:05.917898+0000 mon.a (mon.0) 14038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:20:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: audit 2022-04-22T22:20:06.142643+0000 mon.a (mon.0) 14039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:06 smithi121 bash[36902]: audit 2022-04-22T22:20:06.291309+0000 mon.a (mon.0) 14040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: cluster 2022-04-22T22:20:05.204848+0000 mgr.y (mgr.34107) 10027 : cluster [DBG] pgmap v10760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: audit 2022-04-22T22:20:05.913483+0000 mon.a (mon.0) 14036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: audit 2022-04-22T22:20:05.916470+0000 mon.a (mon.0) 14037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: audit 2022-04-22T22:20:05.917898+0000 mon.a (mon.0) 14038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: audit 2022-04-22T22:20:06.142643+0000 mon.a (mon.0) 14039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[45495]: audit 2022-04-22T22:20:06.291309+0000 mon.a (mon.0) 14040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:20:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: cluster 2022-04-22T22:20:05.204848+0000 mgr.y (mgr.34107) 10027 : cluster [DBG] pgmap v10760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: audit 2022-04-22T22:20:05.913483+0000 mon.a (mon.0) 14036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:20:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: audit 2022-04-22T22:20:05.916470+0000 mon.a (mon.0) 14037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:20:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: audit 2022-04-22T22:20:05.917898+0000 mon.a (mon.0) 14038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:20:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: audit 2022-04-22T22:20:06.142643+0000 mon.a (mon.0) 14039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:07.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:06 smithi115 bash[46934]: audit 2022-04-22T22:20:06.291309+0000 mon.a (mon.0) 14040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:20:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:07 smithi121 bash[36902]: audit 2022-04-22T22:20:07.384952+0000 mon.a (mon.0) 14041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:07 smithi121 bash[36902]: audit 2022-04-22T22:20:07.528075+0000 mon.a (mon.0) 14042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:07 smithi115 bash[45495]: audit 2022-04-22T22:20:07.384952+0000 mon.a (mon.0) 14041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:07 smithi115 bash[45495]: audit 2022-04-22T22:20:07.528075+0000 mon.a (mon.0) 14042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:07 smithi115 bash[46934]: audit 2022-04-22T22:20:07.384952+0000 mon.a (mon.0) 14041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:07 smithi115 bash[46934]: audit 2022-04-22T22:20:07.528075+0000 mon.a (mon.0) 14042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:08 smithi121 bash[36902]: cluster 2022-04-22T22:20:07.205912+0000 mgr.y (mgr.34107) 10028 : cluster [DBG] pgmap v10761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:08 smithi115 bash[45495]: cluster 2022-04-22T22:20:07.205912+0000 mgr.y (mgr.34107) 10028 : cluster [DBG] pgmap v10761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:08 smithi115 bash[46934]: cluster 2022-04-22T22:20:07.205912+0000 mgr.y (mgr.34107) 10028 : cluster [DBG] pgmap v10761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:10 smithi121 bash[36902]: cluster 2022-04-22T22:20:09.207015+0000 mgr.y (mgr.34107) 10029 : cluster [DBG] pgmap v10762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:10 smithi115 bash[45495]: cluster 2022-04-22T22:20:09.207015+0000 mgr.y (mgr.34107) 10029 : cluster [DBG] pgmap v10762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:10 smithi115 bash[46934]: cluster 2022-04-22T22:20:09.207015+0000 mgr.y (mgr.34107) 10029 : cluster [DBG] pgmap v10762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:11 smithi121 bash[36902]: audit 2022-04-22T22:20:11.145463+0000 mon.a (mon.0) 14043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:11 smithi115 bash[45495]: audit 2022-04-22T22:20:11.145463+0000 mon.a (mon.0) 14043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:11 smithi115 bash[46934]: audit 2022-04-22T22:20:11.145463+0000 mon.a (mon.0) 14043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:12 smithi121 bash[36902]: cluster 2022-04-22T22:20:11.208141+0000 mgr.y (mgr.34107) 10030 : cluster [DBG] pgmap v10763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:12 smithi121 bash[36902]: audit 2022-04-22T22:20:12.388668+0000 mon.a (mon.0) 14044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:12 smithi121 bash[36902]: audit 2022-04-22T22:20:12.531087+0000 mon.a (mon.0) 14045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:12.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[46934]: cluster 2022-04-22T22:20:11.208141+0000 mgr.y (mgr.34107) 10030 : cluster [DBG] pgmap v10763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[46934]: audit 2022-04-22T22:20:12.388668+0000 mon.a (mon.0) 14044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[46934]: audit 2022-04-22T22:20:12.531087+0000 mon.a (mon.0) 14045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[45495]: cluster 2022-04-22T22:20:11.208141+0000 mgr.y (mgr.34107) 10030 : cluster [DBG] pgmap v10763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[45495]: audit 2022-04-22T22:20:12.388668+0000 mon.a (mon.0) 14044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:12 smithi115 bash[45495]: audit 2022-04-22T22:20:12.531087+0000 mon.a (mon.0) 14045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:14 smithi121 bash[36902]: cluster 2022-04-22T22:20:13.209303+0000 mgr.y (mgr.34107) 10031 : cluster [DBG] pgmap v10764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:14 smithi115 bash[45495]: cluster 2022-04-22T22:20:13.209303+0000 mgr.y (mgr.34107) 10031 : cluster [DBG] pgmap v10764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:14.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:14 smithi115 bash[46934]: cluster 2022-04-22T22:20:13.209303+0000 mgr.y (mgr.34107) 10031 : cluster [DBG] pgmap v10764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:16.387 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:20:16.729 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:20:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:20:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 4h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:20:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:20:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:20:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:20:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:16 smithi121 bash[36902]: cluster 2022-04-22T22:20:15.210226+0000 mgr.y (mgr.34107) 10032 : cluster [DBG] pgmap v10765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:16 smithi121 bash[36902]: audit 2022-04-22T22:20:16.148362+0000 mon.a (mon.0) 14046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:17.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:16 smithi115 bash[46934]: cluster 2022-04-22T22:20:15.210226+0000 mgr.y (mgr.34107) 10032 : cluster [DBG] pgmap v10765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:16 smithi115 bash[46934]: audit 2022-04-22T22:20:16.148362+0000 mon.a (mon.0) 14046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:16 smithi115 bash[45495]: cluster 2022-04-22T22:20:15.210226+0000 mgr.y (mgr.34107) 10032 : cluster [DBG] pgmap v10765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:16 smithi115 bash[45495]: audit 2022-04-22T22:20:16.148362+0000 mon.a (mon.0) 14046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:17.133 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:20:17.133 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:20:17.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:20:17.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:17.134 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:20:17.135 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:20:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:16.365692+0000 mgr.y (mgr.34107) 10033 : audit [DBG] from='client.55687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:16.660462+0000 mon.a (mon.0) 14047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:20:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:16.662227+0000 mon.a (mon.0) 14048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:20:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:17.129500+0000 mon.c (mon.1) 345 : audit [DBG] from='client.? 172.21.15.115:0/2699029173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:17.392292+0000 mon.a (mon.0) 14049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:17 smithi121 bash[36902]: audit 2022-04-22T22:20:17.534234+0000 mon.a (mon.0) 14050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:16.365692+0000 mgr.y (mgr.34107) 10033 : audit [DBG] from='client.55687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:16.660462+0000 mon.a (mon.0) 14047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:16.662227+0000 mon.a (mon.0) 14048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:17.129500+0000 mon.c (mon.1) 345 : audit [DBG] from='client.? 172.21.15.115:0/2699029173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:17.392292+0000 mon.a (mon.0) 14049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[45495]: audit 2022-04-22T22:20:17.534234+0000 mon.a (mon.0) 14050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:16.365692+0000 mgr.y (mgr.34107) 10033 : audit [DBG] from='client.55687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:16.660462+0000 mon.a (mon.0) 14047 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:20:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:16.662227+0000 mon.a (mon.0) 14048 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:20:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:17.129500+0000 mon.c (mon.1) 345 : audit [DBG] from='client.? 172.21.15.115:0/2699029173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:17.392292+0000 mon.a (mon.0) 14049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:17 smithi115 bash[46934]: audit 2022-04-22T22:20:17.534234+0000 mon.a (mon.0) 14050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:18 smithi121 bash[36902]: audit 2022-04-22T22:20:16.706218+0000 mgr.y (mgr.34107) 10034 : audit [DBG] from='client.45762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:18 smithi121 bash[36902]: cluster 2022-04-22T22:20:17.211203+0000 mgr.y (mgr.34107) 10035 : cluster [DBG] pgmap v10766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:19.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:18 smithi115 bash[46934]: audit 2022-04-22T22:20:16.706218+0000 mgr.y (mgr.34107) 10034 : audit [DBG] from='client.45762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:19.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:18 smithi115 bash[46934]: cluster 2022-04-22T22:20:17.211203+0000 mgr.y (mgr.34107) 10035 : cluster [DBG] pgmap v10766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:18 smithi115 bash[45495]: audit 2022-04-22T22:20:16.706218+0000 mgr.y (mgr.34107) 10034 : audit [DBG] from='client.45762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:18 smithi115 bash[45495]: cluster 2022-04-22T22:20:17.211203+0000 mgr.y (mgr.34107) 10035 : cluster [DBG] pgmap v10766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:19 smithi121 bash[36902]: cluster 2022-04-22T22:20:19.212256+0000 mgr.y (mgr.34107) 10036 : cluster [DBG] pgmap v10767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:19 smithi115 bash[45495]: cluster 2022-04-22T22:20:19.212256+0000 mgr.y (mgr.34107) 10036 : cluster [DBG] pgmap v10767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:19 smithi115 bash[46934]: cluster 2022-04-22T22:20:19.212256+0000 mgr.y (mgr.34107) 10036 : cluster [DBG] pgmap v10767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:21 smithi121 bash[36902]: audit 2022-04-22T22:20:21.151163+0000 mon.a (mon.0) 14051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:21 smithi115 bash[45495]: audit 2022-04-22T22:20:21.151163+0000 mon.a (mon.0) 14051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:21 smithi115 bash[46934]: audit 2022-04-22T22:20:21.151163+0000 mon.a (mon.0) 14051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:22 smithi121 bash[36902]: cluster 2022-04-22T22:20:21.212947+0000 mgr.y (mgr.34107) 10037 : cluster [DBG] pgmap v10768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:22.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:22 smithi115 bash[45495]: cluster 2022-04-22T22:20:21.212947+0000 mgr.y (mgr.34107) 10037 : cluster [DBG] pgmap v10768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:22.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:22 smithi115 bash[46934]: cluster 2022-04-22T22:20:21.212947+0000 mgr.y (mgr.34107) 10037 : cluster [DBG] pgmap v10768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:22.776Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:23 smithi121 bash[36902]: audit 2022-04-22T22:20:22.396640+0000 mon.a (mon.0) 14052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:23 smithi121 bash[36902]: audit 2022-04-22T22:20:22.537125+0000 mon.a (mon.0) 14053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:23 smithi115 bash[45495]: audit 2022-04-22T22:20:22.396640+0000 mon.a (mon.0) 14052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:23 smithi115 bash[45495]: audit 2022-04-22T22:20:22.537125+0000 mon.a (mon.0) 14053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:23 smithi115 bash[46934]: audit 2022-04-22T22:20:22.396640+0000 mon.a (mon.0) 14052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:23 smithi115 bash[46934]: audit 2022-04-22T22:20:22.537125+0000 mon.a (mon.0) 14053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:24 smithi121 bash[36902]: cluster 2022-04-22T22:20:23.214012+0000 mgr.y (mgr.34107) 10038 : cluster [DBG] pgmap v10769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:24.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:24 smithi115 bash[45495]: cluster 2022-04-22T22:20:23.214012+0000 mgr.y (mgr.34107) 10038 : cluster [DBG] pgmap v10769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:24 smithi115 bash[46934]: cluster 2022-04-22T22:20:23.214012+0000 mgr.y (mgr.34107) 10038 : cluster [DBG] pgmap v10769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:26 smithi121 bash[36902]: cluster 2022-04-22T22:20:25.215182+0000 mgr.y (mgr.34107) 10039 : cluster [DBG] pgmap v10770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:26 smithi121 bash[36902]: audit 2022-04-22T22:20:26.154241+0000 mon.a (mon.0) 14054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:26 smithi115 bash[45495]: cluster 2022-04-22T22:20:25.215182+0000 mgr.y (mgr.34107) 10039 : cluster [DBG] pgmap v10770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:26 smithi115 bash[45495]: audit 2022-04-22T22:20:26.154241+0000 mon.a (mon.0) 14054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:26 smithi115 bash[46934]: cluster 2022-04-22T22:20:25.215182+0000 mgr.y (mgr.34107) 10039 : cluster [DBG] pgmap v10770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:26.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:26 smithi115 bash[46934]: audit 2022-04-22T22:20:26.154241+0000 mon.a (mon.0) 14054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:27 smithi121 bash[36902]: audit 2022-04-22T22:20:27.400507+0000 mon.a (mon.0) 14055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:27 smithi115 bash[45495]: audit 2022-04-22T22:20:27.400507+0000 mon.a (mon.0) 14055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:27 smithi115 bash[46934]: audit 2022-04-22T22:20:27.400507+0000 mon.a (mon.0) 14055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:28 smithi121 bash[36902]: cluster 2022-04-22T22:20:27.216269+0000 mgr.y (mgr.34107) 10040 : cluster [DBG] pgmap v10771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:28 smithi121 bash[36902]: audit 2022-04-22T22:20:27.540149+0000 mon.a (mon.0) 14056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:28 smithi115 bash[45495]: cluster 2022-04-22T22:20:27.216269+0000 mgr.y (mgr.34107) 10040 : cluster [DBG] pgmap v10771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:28 smithi115 bash[45495]: audit 2022-04-22T22:20:27.540149+0000 mon.a (mon.0) 14056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:28.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:28 smithi115 bash[46934]: cluster 2022-04-22T22:20:27.216269+0000 mgr.y (mgr.34107) 10040 : cluster [DBG] pgmap v10771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:28.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:28 smithi115 bash[46934]: audit 2022-04-22T22:20:27.540149+0000 mon.a (mon.0) 14056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:30 smithi121 bash[36902]: cluster 2022-04-22T22:20:29.217406+0000 mgr.y (mgr.34107) 10041 : cluster [DBG] pgmap v10772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:30 smithi115 bash[45495]: cluster 2022-04-22T22:20:29.217406+0000 mgr.y (mgr.34107) 10041 : cluster [DBG] pgmap v10772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:30 smithi115 bash[46934]: cluster 2022-04-22T22:20:29.217406+0000 mgr.y (mgr.34107) 10041 : cluster [DBG] pgmap v10772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:31 smithi121 bash[36902]: audit 2022-04-22T22:20:31.157614+0000 mon.a (mon.0) 14057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:31 smithi115 bash[45495]: audit 2022-04-22T22:20:31.157614+0000 mon.a (mon.0) 14057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:31 smithi115 bash[46934]: audit 2022-04-22T22:20:31.157614+0000 mon.a (mon.0) 14057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:32 smithi121 bash[36902]: cluster 2022-04-22T22:20:31.218554+0000 mgr.y (mgr.34107) 10042 : cluster [DBG] pgmap v10773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:32 smithi121 bash[36902]: audit 2022-04-22T22:20:32.404285+0000 mon.a (mon.0) 14058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:32 smithi115 bash[45495]: cluster 2022-04-22T22:20:31.218554+0000 mgr.y (mgr.34107) 10042 : cluster [DBG] pgmap v10773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:32.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:32 smithi115 bash[45495]: audit 2022-04-22T22:20:32.404285+0000 mon.a (mon.0) 14058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:32 smithi115 bash[46934]: cluster 2022-04-22T22:20:31.218554+0000 mgr.y (mgr.34107) 10042 : cluster [DBG] pgmap v10773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:32 smithi115 bash[46934]: audit 2022-04-22T22:20:32.404285+0000 mon.a (mon.0) 14058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:32.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:33 smithi121 bash[36902]: audit 2022-04-22T22:20:32.543169+0000 mon.a (mon.0) 14059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:33 smithi115 bash[45495]: audit 2022-04-22T22:20:32.543169+0000 mon.a (mon.0) 14059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:33 smithi115 bash[46934]: audit 2022-04-22T22:20:32.543169+0000 mon.a (mon.0) 14059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:34 smithi121 bash[36902]: cluster 2022-04-22T22:20:33.219732+0000 mgr.y (mgr.34107) 10043 : cluster [DBG] pgmap v10774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:34 smithi115 bash[45495]: cluster 2022-04-22T22:20:33.219732+0000 mgr.y (mgr.34107) 10043 : cluster [DBG] pgmap v10774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:34.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:34 smithi115 bash[46934]: cluster 2022-04-22T22:20:33.219732+0000 mgr.y (mgr.34107) 10043 : cluster [DBG] pgmap v10774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:36 smithi121 bash[36902]: cluster 2022-04-22T22:20:35.220738+0000 mgr.y (mgr.34107) 10044 : cluster [DBG] pgmap v10775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:36 smithi121 bash[36902]: audit 2022-04-22T22:20:36.160801+0000 mon.a (mon.0) 14060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:36 smithi115 bash[45495]: cluster 2022-04-22T22:20:35.220738+0000 mgr.y (mgr.34107) 10044 : cluster [DBG] pgmap v10775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:36 smithi115 bash[45495]: audit 2022-04-22T22:20:36.160801+0000 mon.a (mon.0) 14060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:36 smithi115 bash[46934]: cluster 2022-04-22T22:20:35.220738+0000 mgr.y (mgr.34107) 10044 : cluster [DBG] pgmap v10775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:36 smithi115 bash[46934]: audit 2022-04-22T22:20:36.160801+0000 mon.a (mon.0) 14060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:37 smithi121 bash[36902]: audit 2022-04-22T22:20:37.408106+0000 mon.a (mon.0) 14061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:37 smithi115 bash[45495]: audit 2022-04-22T22:20:37.408106+0000 mon.a (mon.0) 14061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:37 smithi115 bash[46934]: audit 2022-04-22T22:20:37.408106+0000 mon.a (mon.0) 14061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:38 smithi121 bash[36902]: cluster 2022-04-22T22:20:37.221968+0000 mgr.y (mgr.34107) 10045 : cluster [DBG] pgmap v10776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:38 smithi121 bash[36902]: audit 2022-04-22T22:20:37.546406+0000 mon.a (mon.0) 14062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:38 smithi115 bash[45495]: cluster 2022-04-22T22:20:37.221968+0000 mgr.y (mgr.34107) 10045 : cluster [DBG] pgmap v10776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:38 smithi115 bash[45495]: audit 2022-04-22T22:20:37.546406+0000 mon.a (mon.0) 14062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:38 smithi115 bash[46934]: cluster 2022-04-22T22:20:37.221968+0000 mgr.y (mgr.34107) 10045 : cluster [DBG] pgmap v10776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:38 smithi115 bash[46934]: audit 2022-04-22T22:20:37.546406+0000 mon.a (mon.0) 14062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:40 smithi121 bash[36902]: cluster 2022-04-22T22:20:39.223175+0000 mgr.y (mgr.34107) 10046 : cluster [DBG] pgmap v10777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:40.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:40 smithi115 bash[45495]: cluster 2022-04-22T22:20:39.223175+0000 mgr.y (mgr.34107) 10046 : cluster [DBG] pgmap v10777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:40 smithi115 bash[46934]: cluster 2022-04-22T22:20:39.223175+0000 mgr.y (mgr.34107) 10046 : cluster [DBG] pgmap v10777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:41 smithi121 bash[36902]: audit 2022-04-22T22:20:41.163854+0000 mon.a (mon.0) 14063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:41 smithi115 bash[45495]: audit 2022-04-22T22:20:41.163854+0000 mon.a (mon.0) 14063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:41 smithi115 bash[46934]: audit 2022-04-22T22:20:41.163854+0000 mon.a (mon.0) 14063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:42 smithi121 bash[36902]: cluster 2022-04-22T22:20:41.224304+0000 mgr.y (mgr.34107) 10047 : cluster [DBG] pgmap v10778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:42 smithi121 bash[36902]: audit 2022-04-22T22:20:42.411746+0000 mon.a (mon.0) 14064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:42.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:42 smithi115 bash[45495]: cluster 2022-04-22T22:20:41.224304+0000 mgr.y (mgr.34107) 10047 : cluster [DBG] pgmap v10778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:42.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:42 smithi115 bash[45495]: audit 2022-04-22T22:20:42.411746+0000 mon.a (mon.0) 14064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:42.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:42 smithi115 bash[46934]: cluster 2022-04-22T22:20:41.224304+0000 mgr.y (mgr.34107) 10047 : cluster [DBG] pgmap v10778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:42.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:42 smithi115 bash[46934]: audit 2022-04-22T22:20:42.411746+0000 mon.a (mon.0) 14064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:42.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:43 smithi121 bash[36902]: audit 2022-04-22T22:20:42.549510+0000 mon.a (mon.0) 14065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:43 smithi115 bash[45495]: audit 2022-04-22T22:20:42.549510+0000 mon.a (mon.0) 14065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:43 smithi115 bash[46934]: audit 2022-04-22T22:20:42.549510+0000 mon.a (mon.0) 14065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:44 smithi121 bash[36902]: cluster 2022-04-22T22:20:43.225438+0000 mgr.y (mgr.34107) 10048 : cluster [DBG] pgmap v10779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:44 smithi115 bash[45495]: cluster 2022-04-22T22:20:43.225438+0000 mgr.y (mgr.34107) 10048 : cluster [DBG] pgmap v10779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:44.629Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:44 smithi115 bash[46934]: cluster 2022-04-22T22:20:43.225438+0000 mgr.y (mgr.34107) 10048 : cluster [DBG] pgmap v10779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:46 smithi121 bash[36902]: cluster 2022-04-22T22:20:45.226639+0000 mgr.y (mgr.34107) 10049 : cluster [DBG] pgmap v10780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:46 smithi121 bash[36902]: audit 2022-04-22T22:20:46.166994+0000 mon.a (mon.0) 14066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:46 smithi115 bash[45495]: cluster 2022-04-22T22:20:45.226639+0000 mgr.y (mgr.34107) 10049 : cluster [DBG] pgmap v10780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:46 smithi115 bash[45495]: audit 2022-04-22T22:20:46.166994+0000 mon.a (mon.0) 14066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:46 smithi115 bash[46934]: cluster 2022-04-22T22:20:45.226639+0000 mgr.y (mgr.34107) 10049 : cluster [DBG] pgmap v10780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:46 smithi115 bash[46934]: audit 2022-04-22T22:20:46.166994+0000 mon.a (mon.0) 14066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:47.505 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:20:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:47 smithi121 bash[36902]: audit 2022-04-22T22:20:47.415181+0000 mon.a (mon.0) 14067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:47.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:47 smithi115 bash[46934]: audit 2022-04-22T22:20:47.415181+0000 mon.a (mon.0) 14067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:47 smithi115 bash[45495]: audit 2022-04-22T22:20:47.415181+0000 mon.a (mon.0) 14067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:47.848 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 5m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 4m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 4m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 5m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 4m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:20:47.849 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 4m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 4h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:20:47.850 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:20:48.262 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:20:48.262 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:48.263 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:20:48.264 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:20:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:48 smithi121 bash[36902]: cluster 2022-04-22T22:20:47.227754+0000 mgr.y (mgr.34107) 10050 : cluster [DBG] pgmap v10781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:48 smithi121 bash[36902]: audit 2022-04-22T22:20:47.485475+0000 mgr.y (mgr.34107) 10051 : audit [DBG] from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:48 smithi121 bash[36902]: audit 2022-04-22T22:20:47.552642+0000 mon.a (mon.0) 14068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:48 smithi121 bash[36902]: audit 2022-04-22T22:20:48.257017+0000 mon.c (mon.1) 346 : audit [DBG] from='client.? 172.21.15.115:0/714142724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[45495]: cluster 2022-04-22T22:20:47.227754+0000 mgr.y (mgr.34107) 10050 : cluster [DBG] pgmap v10781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[45495]: audit 2022-04-22T22:20:47.485475+0000 mgr.y (mgr.34107) 10051 : audit [DBG] from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[45495]: audit 2022-04-22T22:20:47.552642+0000 mon.a (mon.0) 14068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[45495]: audit 2022-04-22T22:20:48.257017+0000 mon.c (mon.1) 346 : audit [DBG] from='client.? 172.21.15.115:0/714142724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[46934]: cluster 2022-04-22T22:20:47.227754+0000 mgr.y (mgr.34107) 10050 : cluster [DBG] pgmap v10781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[46934]: audit 2022-04-22T22:20:47.485475+0000 mgr.y (mgr.34107) 10051 : audit [DBG] from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[46934]: audit 2022-04-22T22:20:47.552642+0000 mon.a (mon.0) 14068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:48 smithi115 bash[46934]: audit 2022-04-22T22:20:48.257017+0000 mon.c (mon.1) 346 : audit [DBG] from='client.? 172.21.15.115:0/714142724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:20:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:49 smithi121 bash[36902]: audit 2022-04-22T22:20:47.825494+0000 mgr.y (mgr.34107) 10052 : audit [DBG] from='client.55711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:49.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:49 smithi115 bash[45495]: audit 2022-04-22T22:20:47.825494+0000 mgr.y (mgr.34107) 10052 : audit [DBG] from='client.55711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:49 smithi115 bash[46934]: audit 2022-04-22T22:20:47.825494+0000 mgr.y (mgr.34107) 10052 : audit [DBG] from='client.55711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:20:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:50 smithi115 bash[45495]: cluster 2022-04-22T22:20:49.228850+0000 mgr.y (mgr.34107) 10053 : cluster [DBG] pgmap v10782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:50 smithi115 bash[46934]: cluster 2022-04-22T22:20:49.228850+0000 mgr.y (mgr.34107) 10053 : cluster [DBG] pgmap v10782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:50 smithi121 bash[36902]: cluster 2022-04-22T22:20:49.228850+0000 mgr.y (mgr.34107) 10053 : cluster [DBG] pgmap v10782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:51 smithi115 bash[45495]: audit 2022-04-22T22:20:51.170160+0000 mon.a (mon.0) 14069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:51 smithi115 bash[46934]: audit 2022-04-22T22:20:51.170160+0000 mon.a (mon.0) 14069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:51 smithi121 bash[36902]: audit 2022-04-22T22:20:51.170160+0000 mon.a (mon.0) 14069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:52.780 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:52 smithi115 bash[45495]: cluster 2022-04-22T22:20:51.229917+0000 mgr.y (mgr.34107) 10054 : cluster [DBG] pgmap v10783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:52.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:52 smithi115 bash[45495]: audit 2022-04-22T22:20:52.418508+0000 mon.a (mon.0) 14070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:52.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:52 smithi115 bash[46934]: cluster 2022-04-22T22:20:51.229917+0000 mgr.y (mgr.34107) 10054 : cluster [DBG] pgmap v10783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:52.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:52 smithi115 bash[46934]: audit 2022-04-22T22:20:52.418508+0000 mon.a (mon.0) 14070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:52 smithi121 bash[36902]: cluster 2022-04-22T22:20:51.229917+0000 mgr.y (mgr.34107) 10054 : cluster [DBG] pgmap v10783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:52 smithi121 bash[36902]: audit 2022-04-22T22:20:52.418508+0000 mon.a (mon.0) 14070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:52.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:53 smithi115 bash[45495]: audit 2022-04-22T22:20:52.555051+0000 mon.a (mon.0) 14071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:53 smithi115 bash[46934]: audit 2022-04-22T22:20:52.555051+0000 mon.a (mon.0) 14071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:53 smithi121 bash[36902]: audit 2022-04-22T22:20:52.555051+0000 mon.a (mon.0) 14071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:54 smithi115 bash[45495]: cluster 2022-04-22T22:20:53.231072+0000 mgr.y (mgr.34107) 10055 : cluster [DBG] pgmap v10784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:20:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:20:54.630Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:20:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:54 smithi115 bash[46934]: cluster 2022-04-22T22:20:53.231072+0000 mgr.y (mgr.34107) 10055 : cluster [DBG] pgmap v10784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:54 smithi121 bash[36902]: cluster 2022-04-22T22:20:53.231072+0000 mgr.y (mgr.34107) 10055 : cluster [DBG] pgmap v10784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:56 smithi115 bash[45495]: cluster 2022-04-22T22:20:55.232214+0000 mgr.y (mgr.34107) 10056 : cluster [DBG] pgmap v10785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:56 smithi115 bash[45495]: audit 2022-04-22T22:20:56.173206+0000 mon.a (mon.0) 14072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:56 smithi115 bash[46934]: cluster 2022-04-22T22:20:55.232214+0000 mgr.y (mgr.34107) 10056 : cluster [DBG] pgmap v10785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:56 smithi115 bash[46934]: audit 2022-04-22T22:20:56.173206+0000 mon.a (mon.0) 14072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:56 smithi121 bash[36902]: cluster 2022-04-22T22:20:55.232214+0000 mgr.y (mgr.34107) 10056 : cluster [DBG] pgmap v10785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:56 smithi121 bash[36902]: audit 2022-04-22T22:20:56.173206+0000 mon.a (mon.0) 14072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:20:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:57 smithi121 bash[36902]: audit 2022-04-22T22:20:57.422172+0000 mon.a (mon.0) 14073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:57 smithi115 bash[45495]: audit 2022-04-22T22:20:57.422172+0000 mon.a (mon.0) 14073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:57 smithi115 bash[46934]: audit 2022-04-22T22:20:57.422172+0000 mon.a (mon.0) 14073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:20:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:58 smithi121 bash[36902]: cluster 2022-04-22T22:20:57.233269+0000 mgr.y (mgr.34107) 10057 : cluster [DBG] pgmap v10786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:20:58 smithi121 bash[36902]: audit 2022-04-22T22:20:57.558040+0000 mon.a (mon.0) 14074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:58 smithi115 bash[45495]: cluster 2022-04-22T22:20:57.233269+0000 mgr.y (mgr.34107) 10057 : cluster [DBG] pgmap v10786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:20:58 smithi115 bash[45495]: audit 2022-04-22T22:20:57.558040+0000 mon.a (mon.0) 14074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:20:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:58 smithi115 bash[46934]: cluster 2022-04-22T22:20:57.233269+0000 mgr.y (mgr.34107) 10057 : cluster [DBG] pgmap v10786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:20:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:20:58 smithi115 bash[46934]: audit 2022-04-22T22:20:57.558040+0000 mon.a (mon.0) 14074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:00 smithi121 bash[36902]: cluster 2022-04-22T22:20:59.234469+0000 mgr.y (mgr.34107) 10058 : cluster [DBG] pgmap v10787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:00 smithi115 bash[45495]: cluster 2022-04-22T22:20:59.234469+0000 mgr.y (mgr.34107) 10058 : cluster [DBG] pgmap v10787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:00 smithi115 bash[46934]: cluster 2022-04-22T22:20:59.234469+0000 mgr.y (mgr.34107) 10058 : cluster [DBG] pgmap v10787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:01 smithi121 bash[36902]: audit 2022-04-22T22:21:01.176501+0000 mon.a (mon.0) 14075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:01 smithi115 bash[45495]: audit 2022-04-22T22:21:01.176501+0000 mon.a (mon.0) 14075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:01 smithi115 bash[46934]: audit 2022-04-22T22:21:01.176501+0000 mon.a (mon.0) 14075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:02 smithi121 bash[36902]: cluster 2022-04-22T22:21:01.235621+0000 mgr.y (mgr.34107) 10059 : cluster [DBG] pgmap v10788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:03.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:02 smithi121 bash[36902]: audit 2022-04-22T22:21:02.425795+0000 mon.a (mon.0) 14076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:02.779Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:02 smithi115 bash[45495]: cluster 2022-04-22T22:21:01.235621+0000 mgr.y (mgr.34107) 10059 : cluster [DBG] pgmap v10788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:02 smithi115 bash[45495]: audit 2022-04-22T22:21:02.425795+0000 mon.a (mon.0) 14076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:03.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:02 smithi115 bash[46934]: cluster 2022-04-22T22:21:01.235621+0000 mgr.y (mgr.34107) 10059 : cluster [DBG] pgmap v10788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:03.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:02 smithi115 bash[46934]: audit 2022-04-22T22:21:02.425795+0000 mon.a (mon.0) 14076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:03 smithi121 bash[36902]: audit 2022-04-22T22:21:02.563171+0000 mon.a (mon.0) 14077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:04.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:03 smithi115 bash[45495]: audit 2022-04-22T22:21:02.563171+0000 mon.a (mon.0) 14077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:03 smithi115 bash[46934]: audit 2022-04-22T22:21:02.563171+0000 mon.a (mon.0) 14077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:04 smithi121 bash[36902]: cluster 2022-04-22T22:21:03.236746+0000 mgr.y (mgr.34107) 10060 : cluster [DBG] pgmap v10789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:04 smithi115 bash[45495]: cluster 2022-04-22T22:21:03.236746+0000 mgr.y (mgr.34107) 10060 : cluster [DBG] pgmap v10789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:04.631Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:04 smithi115 bash[46934]: cluster 2022-04-22T22:21:03.236746+0000 mgr.y (mgr.34107) 10060 : cluster [DBG] pgmap v10789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[46934]: cluster 2022-04-22T22:21:05.237801+0000 mgr.y (mgr.34107) 10061 : cluster [DBG] pgmap v10790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[46934]: audit 2022-04-22T22:21:06.179873+0000 mon.a (mon.0) 14078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[46934]: audit 2022-04-22T22:21:06.300250+0000 mon.a (mon.0) 14079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[46934]: audit 2022-04-22T22:21:06.303125+0000 mon.a (mon.0) 14080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[46934]: audit 2022-04-22T22:21:06.304819+0000 mon.a (mon.0) 14081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:21:06.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[45495]: cluster 2022-04-22T22:21:05.237801+0000 mgr.y (mgr.34107) 10061 : cluster [DBG] pgmap v10790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:06.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[45495]: audit 2022-04-22T22:21:06.179873+0000 mon.a (mon.0) 14078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:06.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[45495]: audit 2022-04-22T22:21:06.300250+0000 mon.a (mon.0) 14079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:21:06.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[45495]: audit 2022-04-22T22:21:06.303125+0000 mon.a (mon.0) 14080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:21:06.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:06 smithi115 bash[45495]: audit 2022-04-22T22:21:06.304819+0000 mon.a (mon.0) 14081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:21:06.622 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:06 smithi121 bash[36902]: cluster 2022-04-22T22:21:05.237801+0000 mgr.y (mgr.34107) 10061 : cluster [DBG] pgmap v10790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:06.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:06 smithi121 bash[36902]: audit 2022-04-22T22:21:06.179873+0000 mon.a (mon.0) 14078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:06.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:06 smithi121 bash[36902]: audit 2022-04-22T22:21:06.300250+0000 mon.a (mon.0) 14079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:21:06.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:06 smithi121 bash[36902]: audit 2022-04-22T22:21:06.303125+0000 mon.a (mon.0) 14080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:21:06.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:06 smithi121 bash[36902]: audit 2022-04-22T22:21:06.304819+0000 mon.a (mon.0) 14081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:21:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:07 smithi121 bash[36902]: audit 2022-04-22T22:21:06.631132+0000 mon.a (mon.0) 14082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:21:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:07 smithi121 bash[36902]: cluster 2022-04-22T22:21:07.238926+0000 mgr.y (mgr.34107) 10062 : cluster [DBG] pgmap v10791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:07 smithi121 bash[36902]: audit 2022-04-22T22:21:07.429560+0000 mon.a (mon.0) 14083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:07 smithi121 bash[36902]: audit 2022-04-22T22:21:07.564655+0000 mon.a (mon.0) 14084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[45495]: audit 2022-04-22T22:21:06.631132+0000 mon.a (mon.0) 14082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:21:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[45495]: cluster 2022-04-22T22:21:07.238926+0000 mgr.y (mgr.34107) 10062 : cluster [DBG] pgmap v10791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[45495]: audit 2022-04-22T22:21:07.429560+0000 mon.a (mon.0) 14083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[45495]: audit 2022-04-22T22:21:07.564655+0000 mon.a (mon.0) 14084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[46934]: audit 2022-04-22T22:21:06.631132+0000 mon.a (mon.0) 14082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:21:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[46934]: cluster 2022-04-22T22:21:07.238926+0000 mgr.y (mgr.34107) 10062 : cluster [DBG] pgmap v10791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[46934]: audit 2022-04-22T22:21:07.429560+0000 mon.a (mon.0) 14083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:07 smithi115 bash[46934]: audit 2022-04-22T22:21:07.564655+0000 mon.a (mon.0) 14084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:10 smithi121 bash[36902]: cluster 2022-04-22T22:21:09.240066+0000 mgr.y (mgr.34107) 10063 : cluster [DBG] pgmap v10792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:10 smithi115 bash[45495]: cluster 2022-04-22T22:21:09.240066+0000 mgr.y (mgr.34107) 10063 : cluster [DBG] pgmap v10792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:10 smithi115 bash[46934]: cluster 2022-04-22T22:21:09.240066+0000 mgr.y (mgr.34107) 10063 : cluster [DBG] pgmap v10792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:11 smithi121 bash[36902]: audit 2022-04-22T22:21:11.182830+0000 mon.a (mon.0) 14085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:11 smithi115 bash[45495]: audit 2022-04-22T22:21:11.182830+0000 mon.a (mon.0) 14085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:11 smithi115 bash[46934]: audit 2022-04-22T22:21:11.182830+0000 mon.a (mon.0) 14085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:12 smithi121 bash[36902]: cluster 2022-04-22T22:21:11.241252+0000 mgr.y (mgr.34107) 10064 : cluster [DBG] pgmap v10793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:12.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:12 smithi115 bash[45495]: cluster 2022-04-22T22:21:11.241252+0000 mgr.y (mgr.34107) 10064 : cluster [DBG] pgmap v10793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:12.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:12 smithi115 bash[46934]: cluster 2022-04-22T22:21:11.241252+0000 mgr.y (mgr.34107) 10064 : cluster [DBG] pgmap v10793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:12.779Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:13 smithi121 bash[36902]: audit 2022-04-22T22:21:12.433356+0000 mon.a (mon.0) 14086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:13 smithi121 bash[36902]: audit 2022-04-22T22:21:12.567701+0000 mon.a (mon.0) 14087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:13 smithi115 bash[45495]: audit 2022-04-22T22:21:12.433356+0000 mon.a (mon.0) 14086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:13 smithi115 bash[45495]: audit 2022-04-22T22:21:12.567701+0000 mon.a (mon.0) 14087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:13 smithi115 bash[46934]: audit 2022-04-22T22:21:12.433356+0000 mon.a (mon.0) 14086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:13 smithi115 bash[46934]: audit 2022-04-22T22:21:12.567701+0000 mon.a (mon.0) 14087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:14 smithi121 bash[36902]: cluster 2022-04-22T22:21:13.242409+0000 mgr.y (mgr.34107) 10065 : cluster [DBG] pgmap v10794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:14.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:14.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:14 smithi115 bash[45495]: cluster 2022-04-22T22:21:13.242409+0000 mgr.y (mgr.34107) 10065 : cluster [DBG] pgmap v10794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:14 smithi115 bash[46934]: cluster 2022-04-22T22:21:13.242409+0000 mgr.y (mgr.34107) 10065 : cluster [DBG] pgmap v10794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:16 smithi121 bash[36902]: cluster 2022-04-22T22:21:15.243500+0000 mgr.y (mgr.34107) 10066 : cluster [DBG] pgmap v10795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:16 smithi121 bash[36902]: audit 2022-04-22T22:21:16.186129+0000 mon.a (mon.0) 14088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:16 smithi115 bash[45495]: cluster 2022-04-22T22:21:15.243500+0000 mgr.y (mgr.34107) 10066 : cluster [DBG] pgmap v10795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:16 smithi115 bash[45495]: audit 2022-04-22T22:21:16.186129+0000 mon.a (mon.0) 14088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:16 smithi115 bash[46934]: cluster 2022-04-22T22:21:15.243500+0000 mgr.y (mgr.34107) 10066 : cluster [DBG] pgmap v10795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:16 smithi115 bash[46934]: audit 2022-04-22T22:21:16.186129+0000 mon.a (mon.0) 14088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:17 smithi121 bash[36902]: audit 2022-04-22T22:21:16.671654+0000 mon.a (mon.0) 14089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:21:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:17 smithi121 bash[36902]: audit 2022-04-22T22:21:16.672357+0000 mon.a (mon.0) 14090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:21:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:17 smithi115 bash[45495]: audit 2022-04-22T22:21:16.671654+0000 mon.a (mon.0) 14089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:21:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:17 smithi115 bash[45495]: audit 2022-04-22T22:21:16.672357+0000 mon.a (mon.0) 14090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:21:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:17 smithi115 bash[46934]: audit 2022-04-22T22:21:16.671654+0000 mon.a (mon.0) 14089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:21:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:17 smithi115 bash[46934]: audit 2022-04-22T22:21:16.672357+0000 mon.a (mon.0) 14090 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:21:18.631 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:21:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:18 smithi121 bash[36902]: cluster 2022-04-22T22:21:17.244576+0000 mgr.y (mgr.34107) 10067 : cluster [DBG] pgmap v10796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:18 smithi121 bash[36902]: audit 2022-04-22T22:21:17.436878+0000 mon.a (mon.0) 14091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:18 smithi121 bash[36902]: audit 2022-04-22T22:21:17.570616+0000 mon.a (mon.0) 14092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[46934]: cluster 2022-04-22T22:21:17.244576+0000 mgr.y (mgr.34107) 10067 : cluster [DBG] pgmap v10796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[46934]: audit 2022-04-22T22:21:17.436878+0000 mon.a (mon.0) 14091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[46934]: audit 2022-04-22T22:21:17.570616+0000 mon.a (mon.0) 14092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[45495]: cluster 2022-04-22T22:21:17.244576+0000 mgr.y (mgr.34107) 10067 : cluster [DBG] pgmap v10796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[45495]: audit 2022-04-22T22:21:17.436878+0000 mon.a (mon.0) 14091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:18 smithi115 bash[45495]: audit 2022-04-22T22:21:17.570616+0000 mon.a (mon.0) 14092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:18.973 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:21:18.974 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:21:18.974 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:21:18.974 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:21:18.974 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:21:18.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:21:18.975 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:21:18.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:21:18.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:21:18.975 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:21:18.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:21:18.976 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:21:18.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:21:18.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:21:18.976 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:21:18.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:21:18.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:21:18.977 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:21:18.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:21:18.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 4h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:21:18.979 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:21:19.379 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:21:19.379 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:21:19.379 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:21:19.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:19.381 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:21:19.381 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:21:19.381 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:21:19.381 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:21:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:19 smithi121 bash[36902]: audit 2022-04-22T22:21:19.375911+0000 mon.a (mon.0) 14093 : audit [DBG] from='client.? 172.21.15.115:0/409532995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:19 smithi115 bash[45495]: audit 2022-04-22T22:21:19.375911+0000 mon.a (mon.0) 14093 : audit [DBG] from='client.? 172.21.15.115:0/409532995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:19 smithi115 bash[46934]: audit 2022-04-22T22:21:19.375911+0000 mon.a (mon.0) 14093 : audit [DBG] from='client.? 172.21.15.115:0/409532995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:20 smithi121 bash[36902]: audit 2022-04-22T22:21:18.611766+0000 mgr.y (mgr.34107) 10068 : audit [DBG] from='client.45792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:20 smithi121 bash[36902]: audit 2022-04-22T22:21:18.951217+0000 mgr.y (mgr.34107) 10069 : audit [DBG] from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:20 smithi121 bash[36902]: cluster 2022-04-22T22:21:19.245626+0000 mgr.y (mgr.34107) 10070 : cluster [DBG] pgmap v10797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[45495]: audit 2022-04-22T22:21:18.611766+0000 mgr.y (mgr.34107) 10068 : audit [DBG] from='client.45792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[45495]: audit 2022-04-22T22:21:18.951217+0000 mgr.y (mgr.34107) 10069 : audit [DBG] from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[45495]: cluster 2022-04-22T22:21:19.245626+0000 mgr.y (mgr.34107) 10070 : cluster [DBG] pgmap v10797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[46934]: audit 2022-04-22T22:21:18.611766+0000 mgr.y (mgr.34107) 10068 : audit [DBG] from='client.45792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[46934]: audit 2022-04-22T22:21:18.951217+0000 mgr.y (mgr.34107) 10069 : audit [DBG] from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:20 smithi115 bash[46934]: cluster 2022-04-22T22:21:19.245626+0000 mgr.y (mgr.34107) 10070 : cluster [DBG] pgmap v10797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:21 smithi121 bash[36902]: audit 2022-04-22T22:21:21.189152+0000 mon.a (mon.0) 14094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:21 smithi115 bash[45495]: audit 2022-04-22T22:21:21.189152+0000 mon.a (mon.0) 14094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:21 smithi115 bash[46934]: audit 2022-04-22T22:21:21.189152+0000 mon.a (mon.0) 14094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:22 smithi121 bash[36902]: cluster 2022-04-22T22:21:21.246730+0000 mgr.y (mgr.34107) 10071 : cluster [DBG] pgmap v10798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:22.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:22 smithi115 bash[45495]: cluster 2022-04-22T22:21:21.246730+0000 mgr.y (mgr.34107) 10071 : cluster [DBG] pgmap v10798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:22.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:22 smithi115 bash[46934]: cluster 2022-04-22T22:21:21.246730+0000 mgr.y (mgr.34107) 10071 : cluster [DBG] pgmap v10798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:22.779Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:23 smithi121 bash[36902]: audit 2022-04-22T22:21:22.440622+0000 mon.a (mon.0) 14095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:23 smithi121 bash[36902]: audit 2022-04-22T22:21:22.573712+0000 mon.a (mon.0) 14096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:23 smithi115 bash[45495]: audit 2022-04-22T22:21:22.440622+0000 mon.a (mon.0) 14095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:23 smithi115 bash[45495]: audit 2022-04-22T22:21:22.573712+0000 mon.a (mon.0) 14096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:23 smithi115 bash[46934]: audit 2022-04-22T22:21:22.440622+0000 mon.a (mon.0) 14095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:23 smithi115 bash[46934]: audit 2022-04-22T22:21:22.573712+0000 mon.a (mon.0) 14096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:24 smithi121 bash[36902]: cluster 2022-04-22T22:21:23.247873+0000 mgr.y (mgr.34107) 10072 : cluster [DBG] pgmap v10799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:24 smithi115 bash[45495]: cluster 2022-04-22T22:21:23.247873+0000 mgr.y (mgr.34107) 10072 : cluster [DBG] pgmap v10799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:24.632Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:24 smithi115 bash[46934]: cluster 2022-04-22T22:21:23.247873+0000 mgr.y (mgr.34107) 10072 : cluster [DBG] pgmap v10799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:26 smithi121 bash[36902]: cluster 2022-04-22T22:21:25.248987+0000 mgr.y (mgr.34107) 10073 : cluster [DBG] pgmap v10800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:26 smithi121 bash[36902]: audit 2022-04-22T22:21:26.192360+0000 mon.a (mon.0) 14097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:26 smithi115 bash[45495]: cluster 2022-04-22T22:21:25.248987+0000 mgr.y (mgr.34107) 10073 : cluster [DBG] pgmap v10800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:26 smithi115 bash[45495]: audit 2022-04-22T22:21:26.192360+0000 mon.a (mon.0) 14097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:26 smithi115 bash[46934]: cluster 2022-04-22T22:21:25.248987+0000 mgr.y (mgr.34107) 10073 : cluster [DBG] pgmap v10800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:26 smithi115 bash[46934]: audit 2022-04-22T22:21:26.192360+0000 mon.a (mon.0) 14097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:27 smithi121 bash[36902]: audit 2022-04-22T22:21:27.444464+0000 mon.a (mon.0) 14098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:27 smithi115 bash[45495]: audit 2022-04-22T22:21:27.444464+0000 mon.a (mon.0) 14098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:27 smithi115 bash[46934]: audit 2022-04-22T22:21:27.444464+0000 mon.a (mon.0) 14098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:28 smithi121 bash[36902]: cluster 2022-04-22T22:21:27.250079+0000 mgr.y (mgr.34107) 10074 : cluster [DBG] pgmap v10801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:28 smithi121 bash[36902]: audit 2022-04-22T22:21:27.576766+0000 mon.a (mon.0) 14099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:28 smithi115 bash[45495]: cluster 2022-04-22T22:21:27.250079+0000 mgr.y (mgr.34107) 10074 : cluster [DBG] pgmap v10801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:28 smithi115 bash[45495]: audit 2022-04-22T22:21:27.576766+0000 mon.a (mon.0) 14099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:28 smithi115 bash[46934]: cluster 2022-04-22T22:21:27.250079+0000 mgr.y (mgr.34107) 10074 : cluster [DBG] pgmap v10801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:28 smithi115 bash[46934]: audit 2022-04-22T22:21:27.576766+0000 mon.a (mon.0) 14099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:30.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:30 smithi121 bash[36902]: cluster 2022-04-22T22:21:29.251191+0000 mgr.y (mgr.34107) 10075 : cluster [DBG] pgmap v10802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:30 smithi115 bash[45495]: cluster 2022-04-22T22:21:29.251191+0000 mgr.y (mgr.34107) 10075 : cluster [DBG] pgmap v10802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:30 smithi115 bash[46934]: cluster 2022-04-22T22:21:29.251191+0000 mgr.y (mgr.34107) 10075 : cluster [DBG] pgmap v10802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:31 smithi121 bash[36902]: audit 2022-04-22T22:21:31.195712+0000 mon.a (mon.0) 14100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:31 smithi115 bash[45495]: audit 2022-04-22T22:21:31.195712+0000 mon.a (mon.0) 14100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:31 smithi115 bash[46934]: audit 2022-04-22T22:21:31.195712+0000 mon.a (mon.0) 14100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:32 smithi121 bash[36902]: cluster 2022-04-22T22:21:31.252375+0000 mgr.y (mgr.34107) 10076 : cluster [DBG] pgmap v10803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:32 smithi121 bash[36902]: audit 2022-04-22T22:21:32.448179+0000 mon.a (mon.0) 14101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:32.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:32 smithi115 bash[45495]: cluster 2022-04-22T22:21:31.252375+0000 mgr.y (mgr.34107) 10076 : cluster [DBG] pgmap v10803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:32.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:32 smithi115 bash[45495]: audit 2022-04-22T22:21:32.448179+0000 mon.a (mon.0) 14101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:32.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:32 smithi115 bash[46934]: cluster 2022-04-22T22:21:31.252375+0000 mgr.y (mgr.34107) 10076 : cluster [DBG] pgmap v10803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:32.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:32 smithi115 bash[46934]: audit 2022-04-22T22:21:32.448179+0000 mon.a (mon.0) 14101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:32.781Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:33 smithi121 bash[36902]: audit 2022-04-22T22:21:32.579837+0000 mon.a (mon.0) 14102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:33 smithi115 bash[45495]: audit 2022-04-22T22:21:32.579837+0000 mon.a (mon.0) 14102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:33 smithi115 bash[46934]: audit 2022-04-22T22:21:32.579837+0000 mon.a (mon.0) 14102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:34 smithi121 bash[36902]: cluster 2022-04-22T22:21:33.253502+0000 mgr.y (mgr.34107) 10077 : cluster [DBG] pgmap v10804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:34 smithi115 bash[45495]: cluster 2022-04-22T22:21:33.253502+0000 mgr.y (mgr.34107) 10077 : cluster [DBG] pgmap v10804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:34.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:34 smithi115 bash[46934]: cluster 2022-04-22T22:21:33.253502+0000 mgr.y (mgr.34107) 10077 : cluster [DBG] pgmap v10804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:36 smithi121 bash[36902]: cluster 2022-04-22T22:21:35.254564+0000 mgr.y (mgr.34107) 10078 : cluster [DBG] pgmap v10805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:36 smithi121 bash[36902]: audit 2022-04-22T22:21:36.199062+0000 mon.a (mon.0) 14103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:36 smithi115 bash[45495]: cluster 2022-04-22T22:21:35.254564+0000 mgr.y (mgr.34107) 10078 : cluster [DBG] pgmap v10805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:36 smithi115 bash[45495]: audit 2022-04-22T22:21:36.199062+0000 mon.a (mon.0) 14103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:36 smithi115 bash[46934]: cluster 2022-04-22T22:21:35.254564+0000 mgr.y (mgr.34107) 10078 : cluster [DBG] pgmap v10805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:36 smithi115 bash[46934]: audit 2022-04-22T22:21:36.199062+0000 mon.a (mon.0) 14103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:37 smithi121 bash[36902]: audit 2022-04-22T22:21:37.451727+0000 mon.a (mon.0) 14104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:37 smithi115 bash[45495]: audit 2022-04-22T22:21:37.451727+0000 mon.a (mon.0) 14104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:37 smithi115 bash[46934]: audit 2022-04-22T22:21:37.451727+0000 mon.a (mon.0) 14104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:38 smithi121 bash[36902]: cluster 2022-04-22T22:21:37.255590+0000 mgr.y (mgr.34107) 10079 : cluster [DBG] pgmap v10806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:38 smithi121 bash[36902]: audit 2022-04-22T22:21:37.582865+0000 mon.a (mon.0) 14105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:38 smithi115 bash[45495]: cluster 2022-04-22T22:21:37.255590+0000 mgr.y (mgr.34107) 10079 : cluster [DBG] pgmap v10806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:38 smithi115 bash[45495]: audit 2022-04-22T22:21:37.582865+0000 mon.a (mon.0) 14105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:38 smithi115 bash[46934]: cluster 2022-04-22T22:21:37.255590+0000 mgr.y (mgr.34107) 10079 : cluster [DBG] pgmap v10806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:38 smithi115 bash[46934]: audit 2022-04-22T22:21:37.582865+0000 mon.a (mon.0) 14105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:40 smithi115 bash[45495]: cluster 2022-04-22T22:21:39.256616+0000 mgr.y (mgr.34107) 10080 : cluster [DBG] pgmap v10807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:40 smithi115 bash[46934]: cluster 2022-04-22T22:21:39.256616+0000 mgr.y (mgr.34107) 10080 : cluster [DBG] pgmap v10807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:40 smithi121 bash[36902]: cluster 2022-04-22T22:21:39.256616+0000 mgr.y (mgr.34107) 10080 : cluster [DBG] pgmap v10807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:41 smithi115 bash[45495]: audit 2022-04-22T22:21:41.202046+0000 mon.a (mon.0) 14106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:41 smithi115 bash[46934]: audit 2022-04-22T22:21:41.202046+0000 mon.a (mon.0) 14106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:41 smithi121 bash[36902]: audit 2022-04-22T22:21:41.202046+0000 mon.a (mon.0) 14106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:42.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:42 smithi115 bash[45495]: cluster 2022-04-22T22:21:41.257794+0000 mgr.y (mgr.34107) 10081 : cluster [DBG] pgmap v10808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:42.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:42 smithi115 bash[45495]: audit 2022-04-22T22:21:42.455276+0000 mon.a (mon.0) 14107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:42.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:42 smithi115 bash[46934]: cluster 2022-04-22T22:21:41.257794+0000 mgr.y (mgr.34107) 10081 : cluster [DBG] pgmap v10808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:42.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:42 smithi115 bash[46934]: audit 2022-04-22T22:21:42.455276+0000 mon.a (mon.0) 14107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:42 smithi121 bash[36902]: cluster 2022-04-22T22:21:41.257794+0000 mgr.y (mgr.34107) 10081 : cluster [DBG] pgmap v10808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:43.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:42 smithi121 bash[36902]: audit 2022-04-22T22:21:42.455276+0000 mon.a (mon.0) 14107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:42.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:43 smithi115 bash[45495]: audit 2022-04-22T22:21:42.585985+0000 mon.a (mon.0) 14108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:43 smithi115 bash[46934]: audit 2022-04-22T22:21:42.585985+0000 mon.a (mon.0) 14108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:43 smithi121 bash[36902]: audit 2022-04-22T22:21:42.585985+0000 mon.a (mon.0) 14108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:44 smithi115 bash[46934]: cluster 2022-04-22T22:21:43.258946+0000 mgr.y (mgr.34107) 10082 : cluster [DBG] pgmap v10809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:44 smithi115 bash[45495]: cluster 2022-04-22T22:21:43.258946+0000 mgr.y (mgr.34107) 10082 : cluster [DBG] pgmap v10809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:44.634Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:44 smithi121 bash[36902]: cluster 2022-04-22T22:21:43.258946+0000 mgr.y (mgr.34107) 10082 : cluster [DBG] pgmap v10809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:46 smithi121 bash[36902]: cluster 2022-04-22T22:21:45.260033+0000 mgr.y (mgr.34107) 10083 : cluster [DBG] pgmap v10810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:46 smithi121 bash[36902]: audit 2022-04-22T22:21:46.205206+0000 mon.a (mon.0) 14109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:47.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:46 smithi115 bash[46934]: cluster 2022-04-22T22:21:45.260033+0000 mgr.y (mgr.34107) 10083 : cluster [DBG] pgmap v10810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:46 smithi115 bash[46934]: audit 2022-04-22T22:21:46.205206+0000 mon.a (mon.0) 14109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:46 smithi115 bash[45495]: cluster 2022-04-22T22:21:45.260033+0000 mgr.y (mgr.34107) 10083 : cluster [DBG] pgmap v10810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:46 smithi115 bash[45495]: audit 2022-04-22T22:21:46.205206+0000 mon.a (mon.0) 14109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:47 smithi121 bash[36902]: audit 2022-04-22T22:21:47.458900+0000 mon.a (mon.0) 14110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:47 smithi115 bash[45495]: audit 2022-04-22T22:21:47.458900+0000 mon.a (mon.0) 14110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:47 smithi115 bash[46934]: audit 2022-04-22T22:21:47.458900+0000 mon.a (mon.0) 14110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:48 smithi121 bash[36902]: cluster 2022-04-22T22:21:47.261115+0000 mgr.y (mgr.34107) 10084 : cluster [DBG] pgmap v10811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:49.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:48 smithi121 bash[36902]: audit 2022-04-22T22:21:47.589216+0000 mon.a (mon.0) 14111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:48 smithi115 bash[45495]: cluster 2022-04-22T22:21:47.261115+0000 mgr.y (mgr.34107) 10084 : cluster [DBG] pgmap v10811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:48 smithi115 bash[45495]: audit 2022-04-22T22:21:47.589216+0000 mon.a (mon.0) 14111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:48 smithi115 bash[46934]: cluster 2022-04-22T22:21:47.261115+0000 mgr.y (mgr.34107) 10084 : cluster [DBG] pgmap v10811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:48 smithi115 bash[46934]: audit 2022-04-22T22:21:47.589216+0000 mon.a (mon.0) 14111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:49.744 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:21:50.085 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:21:50.086 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:21:50.086 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 6m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:21:50.086 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 4h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:21:50.086 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 4h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:21:50.086 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:21:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 5m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:21:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 5m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:21:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 6m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:21:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 5m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:21:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:21:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:21:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 5m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:21:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 5m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:21:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 5m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:21:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:21:50.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:21:50.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:21:50.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:21:50.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:21:50.089 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:21:50.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:21:50.491 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:21:50.492 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:21:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:50 smithi121 bash[36902]: cluster 2022-04-22T22:21:49.262221+0000 mgr.y (mgr.34107) 10085 : cluster [DBG] pgmap v10812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:50 smithi121 bash[36902]: audit 2022-04-22T22:21:50.486166+0000 mon.c (mon.1) 347 : audit [DBG] from='client.? 172.21.15.115:0/3808364747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:50 smithi115 bash[45495]: cluster 2022-04-22T22:21:49.262221+0000 mgr.y (mgr.34107) 10085 : cluster [DBG] pgmap v10812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:50 smithi115 bash[45495]: audit 2022-04-22T22:21:50.486166+0000 mon.c (mon.1) 347 : audit [DBG] from='client.? 172.21.15.115:0/3808364747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:50 smithi115 bash[46934]: cluster 2022-04-22T22:21:49.262221+0000 mgr.y (mgr.34107) 10085 : cluster [DBG] pgmap v10812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:50 smithi115 bash[46934]: audit 2022-04-22T22:21:50.486166+0000 mon.c (mon.1) 347 : audit [DBG] from='client.? 172.21.15.115:0/3808364747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:21:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:51 smithi121 bash[36902]: audit 2022-04-22T22:21:49.726104+0000 mgr.y (mgr.34107) 10086 : audit [DBG] from='client.45810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:51 smithi121 bash[36902]: audit 2022-04-22T22:21:50.068057+0000 mgr.y (mgr.34107) 10087 : audit [DBG] from='client.55747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:51 smithi121 bash[36902]: audit 2022-04-22T22:21:51.208398+0000 mon.a (mon.0) 14112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[45495]: audit 2022-04-22T22:21:49.726104+0000 mgr.y (mgr.34107) 10086 : audit [DBG] from='client.45810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[45495]: audit 2022-04-22T22:21:50.068057+0000 mgr.y (mgr.34107) 10087 : audit [DBG] from='client.55747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[45495]: audit 2022-04-22T22:21:51.208398+0000 mon.a (mon.0) 14112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[46934]: audit 2022-04-22T22:21:49.726104+0000 mgr.y (mgr.34107) 10086 : audit [DBG] from='client.45810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[46934]: audit 2022-04-22T22:21:50.068057+0000 mgr.y (mgr.34107) 10087 : audit [DBG] from='client.55747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:21:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:51 smithi115 bash[46934]: audit 2022-04-22T22:21:51.208398+0000 mon.a (mon.0) 14112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:52 smithi121 bash[36902]: cluster 2022-04-22T22:21:51.263279+0000 mgr.y (mgr.34107) 10088 : cluster [DBG] pgmap v10813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:52 smithi121 bash[36902]: audit 2022-04-22T22:21:52.462740+0000 mon.a (mon.0) 14113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:52 smithi115 bash[45495]: cluster 2022-04-22T22:21:51.263279+0000 mgr.y (mgr.34107) 10088 : cluster [DBG] pgmap v10813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:52 smithi115 bash[45495]: audit 2022-04-22T22:21:52.462740+0000 mon.a (mon.0) 14113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:53.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:52.781Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:53.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:52 smithi115 bash[46934]: cluster 2022-04-22T22:21:51.263279+0000 mgr.y (mgr.34107) 10088 : cluster [DBG] pgmap v10813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:53.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:52 smithi115 bash[46934]: audit 2022-04-22T22:21:52.462740+0000 mon.a (mon.0) 14113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:53 smithi121 bash[36902]: audit 2022-04-22T22:21:52.592351+0000 mon.a (mon.0) 14114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:53 smithi115 bash[45495]: audit 2022-04-22T22:21:52.592351+0000 mon.a (mon.0) 14114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:53 smithi115 bash[46934]: audit 2022-04-22T22:21:52.592351+0000 mon.a (mon.0) 14114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:54 smithi121 bash[36902]: cluster 2022-04-22T22:21:53.264406+0000 mgr.y (mgr.34107) 10089 : cluster [DBG] pgmap v10814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:54 smithi115 bash[46934]: cluster 2022-04-22T22:21:53.264406+0000 mgr.y (mgr.34107) 10089 : cluster [DBG] pgmap v10814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:21:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:21:54.635Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:21:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:54 smithi115 bash[45495]: cluster 2022-04-22T22:21:53.264406+0000 mgr.y (mgr.34107) 10089 : cluster [DBG] pgmap v10814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:56 smithi121 bash[36902]: cluster 2022-04-22T22:21:55.265462+0000 mgr.y (mgr.34107) 10090 : cluster [DBG] pgmap v10815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:56 smithi121 bash[36902]: audit 2022-04-22T22:21:56.211590+0000 mon.a (mon.0) 14115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:56 smithi115 bash[45495]: cluster 2022-04-22T22:21:55.265462+0000 mgr.y (mgr.34107) 10090 : cluster [DBG] pgmap v10815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:56 smithi115 bash[45495]: audit 2022-04-22T22:21:56.211590+0000 mon.a (mon.0) 14115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:56 smithi115 bash[46934]: cluster 2022-04-22T22:21:55.265462+0000 mgr.y (mgr.34107) 10090 : cluster [DBG] pgmap v10815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:56 smithi115 bash[46934]: audit 2022-04-22T22:21:56.211590+0000 mon.a (mon.0) 14115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:21:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:57 smithi121 bash[36902]: audit 2022-04-22T22:21:57.466373+0000 mon.a (mon.0) 14116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:57 smithi115 bash[45495]: audit 2022-04-22T22:21:57.466373+0000 mon.a (mon.0) 14116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:57 smithi115 bash[46934]: audit 2022-04-22T22:21:57.466373+0000 mon.a (mon.0) 14116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:21:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:58 smithi121 bash[36902]: cluster 2022-04-22T22:21:57.266618+0000 mgr.y (mgr.34107) 10091 : cluster [DBG] pgmap v10816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:21:58 smithi121 bash[36902]: audit 2022-04-22T22:21:57.594902+0000 mon.a (mon.0) 14117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:58 smithi115 bash[45495]: cluster 2022-04-22T22:21:57.266618+0000 mgr.y (mgr.34107) 10091 : cluster [DBG] pgmap v10816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:21:58 smithi115 bash[45495]: audit 2022-04-22T22:21:57.594902+0000 mon.a (mon.0) 14117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:21:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:58 smithi115 bash[46934]: cluster 2022-04-22T22:21:57.266618+0000 mgr.y (mgr.34107) 10091 : cluster [DBG] pgmap v10816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:21:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:21:58 smithi115 bash[46934]: audit 2022-04-22T22:21:57.594902+0000 mon.a (mon.0) 14117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:00 smithi121 bash[36902]: cluster 2022-04-22T22:21:59.267851+0000 mgr.y (mgr.34107) 10092 : cluster [DBG] pgmap v10817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:00 smithi115 bash[45495]: cluster 2022-04-22T22:21:59.267851+0000 mgr.y (mgr.34107) 10092 : cluster [DBG] pgmap v10817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:00 smithi115 bash[46934]: cluster 2022-04-22T22:21:59.267851+0000 mgr.y (mgr.34107) 10092 : cluster [DBG] pgmap v10817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:01 smithi121 bash[36902]: audit 2022-04-22T22:22:01.214674+0000 mon.a (mon.0) 14118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:01 smithi115 bash[45495]: audit 2022-04-22T22:22:01.214674+0000 mon.a (mon.0) 14118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:01 smithi115 bash[46934]: audit 2022-04-22T22:22:01.214674+0000 mon.a (mon.0) 14118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:02 smithi121 bash[36902]: cluster 2022-04-22T22:22:01.268761+0000 mgr.y (mgr.34107) 10093 : cluster [DBG] pgmap v10818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:02 smithi121 bash[36902]: audit 2022-04-22T22:22:02.470194+0000 mon.a (mon.0) 14119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:02 smithi121 bash[36902]: audit 2022-04-22T22:22:02.597428+0000 mon.a (mon.0) 14120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:03.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[46934]: cluster 2022-04-22T22:22:01.268761+0000 mgr.y (mgr.34107) 10093 : cluster [DBG] pgmap v10818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[46934]: audit 2022-04-22T22:22:02.470194+0000 mon.a (mon.0) 14119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[46934]: audit 2022-04-22T22:22:02.597428+0000 mon.a (mon.0) 14120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:02.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[45495]: cluster 2022-04-22T22:22:01.268761+0000 mgr.y (mgr.34107) 10093 : cluster [DBG] pgmap v10818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[45495]: audit 2022-04-22T22:22:02.470194+0000 mon.a (mon.0) 14119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:02 smithi115 bash[45495]: audit 2022-04-22T22:22:02.597428+0000 mon.a (mon.0) 14120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:03 smithi121 bash[36902]: cluster 2022-04-22T22:22:03.269806+0000 mgr.y (mgr.34107) 10094 : cluster [DBG] pgmap v10819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:03 smithi115 bash[45495]: cluster 2022-04-22T22:22:03.269806+0000 mgr.y (mgr.34107) 10094 : cluster [DBG] pgmap v10819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:03 smithi115 bash[46934]: cluster 2022-04-22T22:22:03.269806+0000 mgr.y (mgr.34107) 10094 : cluster [DBG] pgmap v10819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:04.636Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:06.664 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:06 smithi115 bash[46934]: cluster 2022-04-22T22:22:05.270701+0000 mgr.y (mgr.34107) 10095 : cluster [DBG] pgmap v10820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:06.664 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:06 smithi115 bash[46934]: audit 2022-04-22T22:22:06.217826+0000 mon.a (mon.0) 14121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:06.665 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:06 smithi115 bash[45495]: cluster 2022-04-22T22:22:05.270701+0000 mgr.y (mgr.34107) 10095 : cluster [DBG] pgmap v10820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:06.665 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:06 smithi115 bash[45495]: audit 2022-04-22T22:22:06.217826+0000 mon.a (mon.0) 14121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:06.667 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:06 smithi121 bash[36902]: cluster 2022-04-22T22:22:05.270701+0000 mgr.y (mgr.34107) 10095 : cluster [DBG] pgmap v10820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:06.667 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:06 smithi121 bash[36902]: audit 2022-04-22T22:22:06.217826+0000 mon.a (mon.0) 14121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:07 smithi121 bash[36902]: audit 2022-04-22T22:22:06.638623+0000 mon.a (mon.0) 14122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:22:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:07 smithi121 bash[36902]: audit 2022-04-22T22:22:06.641930+0000 mon.a (mon.0) 14123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:22:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:07 smithi121 bash[36902]: audit 2022-04-22T22:22:06.643855+0000 mon.a (mon.0) 14124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:22:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:07 smithi121 bash[36902]: audit 2022-04-22T22:22:07.004738+0000 mon.a (mon.0) 14125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:22:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[46934]: audit 2022-04-22T22:22:06.638623+0000 mon.a (mon.0) 14122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:22:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[46934]: audit 2022-04-22T22:22:06.641930+0000 mon.a (mon.0) 14123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:22:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[46934]: audit 2022-04-22T22:22:06.643855+0000 mon.a (mon.0) 14124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:22:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[46934]: audit 2022-04-22T22:22:07.004738+0000 mon.a (mon.0) 14125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:22:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[45495]: audit 2022-04-22T22:22:06.638623+0000 mon.a (mon.0) 14122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:22:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[45495]: audit 2022-04-22T22:22:06.641930+0000 mon.a (mon.0) 14123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:22:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[45495]: audit 2022-04-22T22:22:06.643855+0000 mon.a (mon.0) 14124 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:22:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:07 smithi115 bash[45495]: audit 2022-04-22T22:22:07.004738+0000 mon.a (mon.0) 14125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:08 smithi121 bash[36902]: cluster 2022-04-22T22:22:07.271782+0000 mgr.y (mgr.34107) 10096 : cluster [DBG] pgmap v10821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:08 smithi121 bash[36902]: audit 2022-04-22T22:22:07.473787+0000 mon.a (mon.0) 14126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:08 smithi121 bash[36902]: audit 2022-04-22T22:22:07.600597+0000 mon.a (mon.0) 14127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[45495]: cluster 2022-04-22T22:22:07.271782+0000 mgr.y (mgr.34107) 10096 : cluster [DBG] pgmap v10821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[45495]: audit 2022-04-22T22:22:07.473787+0000 mon.a (mon.0) 14126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[45495]: audit 2022-04-22T22:22:07.600597+0000 mon.a (mon.0) 14127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[46934]: cluster 2022-04-22T22:22:07.271782+0000 mgr.y (mgr.34107) 10096 : cluster [DBG] pgmap v10821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[46934]: audit 2022-04-22T22:22:07.473787+0000 mon.a (mon.0) 14126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:08 smithi115 bash[46934]: audit 2022-04-22T22:22:07.600597+0000 mon.a (mon.0) 14127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:10.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:10 smithi121 bash[36902]: cluster 2022-04-22T22:22:09.272965+0000 mgr.y (mgr.34107) 10097 : cluster [DBG] pgmap v10822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:10 smithi115 bash[45495]: cluster 2022-04-22T22:22:09.272965+0000 mgr.y (mgr.34107) 10097 : cluster [DBG] pgmap v10822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:10 smithi115 bash[46934]: cluster 2022-04-22T22:22:09.272965+0000 mgr.y (mgr.34107) 10097 : cluster [DBG] pgmap v10822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:11 smithi121 bash[36902]: audit 2022-04-22T22:22:11.220955+0000 mon.a (mon.0) 14128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:11 smithi115 bash[45495]: audit 2022-04-22T22:22:11.220955+0000 mon.a (mon.0) 14128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:11 smithi115 bash[46934]: audit 2022-04-22T22:22:11.220955+0000 mon.a (mon.0) 14128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:12 smithi121 bash[36902]: cluster 2022-04-22T22:22:11.274017+0000 mgr.y (mgr.34107) 10098 : cluster [DBG] pgmap v10823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:12.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:12 smithi115 bash[45495]: cluster 2022-04-22T22:22:11.274017+0000 mgr.y (mgr.34107) 10098 : cluster [DBG] pgmap v10823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:12.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:12 smithi115 bash[46934]: cluster 2022-04-22T22:22:11.274017+0000 mgr.y (mgr.34107) 10098 : cluster [DBG] pgmap v10823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:12.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:13 smithi121 bash[36902]: audit 2022-04-22T22:22:12.477455+0000 mon.a (mon.0) 14129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:13 smithi121 bash[36902]: audit 2022-04-22T22:22:12.603334+0000 mon.a (mon.0) 14130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:13 smithi115 bash[45495]: audit 2022-04-22T22:22:12.477455+0000 mon.a (mon.0) 14129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:13 smithi115 bash[45495]: audit 2022-04-22T22:22:12.603334+0000 mon.a (mon.0) 14130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:13 smithi115 bash[46934]: audit 2022-04-22T22:22:12.477455+0000 mon.a (mon.0) 14129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:13 smithi115 bash[46934]: audit 2022-04-22T22:22:12.603334+0000 mon.a (mon.0) 14130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:14 smithi121 bash[36902]: cluster 2022-04-22T22:22:13.274954+0000 mgr.y (mgr.34107) 10099 : cluster [DBG] pgmap v10824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:14 smithi115 bash[45495]: cluster 2022-04-22T22:22:13.274954+0000 mgr.y (mgr.34107) 10099 : cluster [DBG] pgmap v10824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:14 smithi115 bash[46934]: cluster 2022-04-22T22:22:13.274954+0000 mgr.y (mgr.34107) 10099 : cluster [DBG] pgmap v10824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:14.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:16 smithi121 bash[36902]: cluster 2022-04-22T22:22:15.275969+0000 mgr.y (mgr.34107) 10100 : cluster [DBG] pgmap v10825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:16 smithi121 bash[36902]: audit 2022-04-22T22:22:16.223987+0000 mon.a (mon.0) 14131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:16 smithi115 bash[45495]: cluster 2022-04-22T22:22:15.275969+0000 mgr.y (mgr.34107) 10100 : cluster [DBG] pgmap v10825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:16 smithi115 bash[45495]: audit 2022-04-22T22:22:16.223987+0000 mon.a (mon.0) 14131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:16 smithi115 bash[46934]: cluster 2022-04-22T22:22:15.275969+0000 mgr.y (mgr.34107) 10100 : cluster [DBG] pgmap v10825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:16 smithi115 bash[46934]: audit 2022-04-22T22:22:16.223987+0000 mon.a (mon.0) 14131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:17 smithi121 bash[36902]: audit 2022-04-22T22:22:16.684965+0000 mon.a (mon.0) 14132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:22:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:17 smithi121 bash[36902]: audit 2022-04-22T22:22:16.685766+0000 mon.a (mon.0) 14133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:22:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:17 smithi115 bash[45495]: audit 2022-04-22T22:22:16.684965+0000 mon.a (mon.0) 14132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:22:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:17 smithi115 bash[45495]: audit 2022-04-22T22:22:16.685766+0000 mon.a (mon.0) 14133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:22:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:17 smithi115 bash[46934]: audit 2022-04-22T22:22:16.684965+0000 mon.a (mon.0) 14132 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:22:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:17 smithi115 bash[46934]: audit 2022-04-22T22:22:16.685766+0000 mon.a (mon.0) 14133 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:22:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:18 smithi121 bash[36902]: cluster 2022-04-22T22:22:17.276959+0000 mgr.y (mgr.34107) 10101 : cluster [DBG] pgmap v10826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:18 smithi121 bash[36902]: audit 2022-04-22T22:22:17.480913+0000 mon.a (mon.0) 14134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:18 smithi121 bash[36902]: audit 2022-04-22T22:22:17.606116+0000 mon.a (mon.0) 14135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[46934]: cluster 2022-04-22T22:22:17.276959+0000 mgr.y (mgr.34107) 10101 : cluster [DBG] pgmap v10826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[46934]: audit 2022-04-22T22:22:17.480913+0000 mon.a (mon.0) 14134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[46934]: audit 2022-04-22T22:22:17.606116+0000 mon.a (mon.0) 14135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[45495]: cluster 2022-04-22T22:22:17.276959+0000 mgr.y (mgr.34107) 10101 : cluster [DBG] pgmap v10826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[45495]: audit 2022-04-22T22:22:17.480913+0000 mon.a (mon.0) 14134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:18 smithi115 bash[45495]: audit 2022-04-22T22:22:17.606116+0000 mon.a (mon.0) 14135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:20 smithi121 bash[36902]: cluster 2022-04-22T22:22:19.277942+0000 mgr.y (mgr.34107) 10102 : cluster [DBG] pgmap v10827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:20.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:20 smithi115 bash[45495]: cluster 2022-04-22T22:22:19.277942+0000 mgr.y (mgr.34107) 10102 : cluster [DBG] pgmap v10827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:20.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:20 smithi115 bash[46934]: cluster 2022-04-22T22:22:19.277942+0000 mgr.y (mgr.34107) 10102 : cluster [DBG] pgmap v10827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:20.869 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:22:21.213 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:22:21.214 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:22:21.214 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 7m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:22:21.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:22:21.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:22:21.215 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:22:21.215 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:22:21.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:22:21.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 7m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:22:21.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:22:21.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:22:21.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:22:21.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:22:21.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:22:21.217 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:22:21.217 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 4h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:22:21.620 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:22:21.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:22:21.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:22:21.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:21.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:22:21.622 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:22:21.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:22:21.622 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:22:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:21 smithi121 bash[36902]: audit 2022-04-22T22:22:21.226875+0000 mon.a (mon.0) 14136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:21 smithi115 bash[45495]: audit 2022-04-22T22:22:21.226875+0000 mon.a (mon.0) 14136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:21 smithi115 bash[46934]: audit 2022-04-22T22:22:21.226875+0000 mon.a (mon.0) 14136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:22 smithi121 bash[36902]: audit 2022-04-22T22:22:20.851970+0000 mgr.y (mgr.34107) 10103 : audit [DBG] from='client.55759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:22 smithi121 bash[36902]: audit 2022-04-22T22:22:21.188362+0000 mgr.y (mgr.34107) 10104 : audit [DBG] from='client.55765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:22 smithi121 bash[36902]: cluster 2022-04-22T22:22:21.278913+0000 mgr.y (mgr.34107) 10105 : cluster [DBG] pgmap v10828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:22 smithi121 bash[36902]: audit 2022-04-22T22:22:21.616877+0000 mon.c (mon.1) 348 : audit [DBG] from='client.? 172.21.15.115:0/1919295745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:22.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[45495]: audit 2022-04-22T22:22:20.851970+0000 mgr.y (mgr.34107) 10103 : audit [DBG] from='client.55759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[45495]: audit 2022-04-22T22:22:21.188362+0000 mgr.y (mgr.34107) 10104 : audit [DBG] from='client.55765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[45495]: cluster 2022-04-22T22:22:21.278913+0000 mgr.y (mgr.34107) 10105 : cluster [DBG] pgmap v10828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[45495]: audit 2022-04-22T22:22:21.616877+0000 mon.c (mon.1) 348 : audit [DBG] from='client.? 172.21.15.115:0/1919295745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[46934]: audit 2022-04-22T22:22:20.851970+0000 mgr.y (mgr.34107) 10103 : audit [DBG] from='client.55759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[46934]: audit 2022-04-22T22:22:21.188362+0000 mgr.y (mgr.34107) 10104 : audit [DBG] from='client.55765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[46934]: cluster 2022-04-22T22:22:21.278913+0000 mgr.y (mgr.34107) 10105 : cluster [DBG] pgmap v10828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:22.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[46934]: audit 2022-04-22T22:22:21.616877+0000 mon.c (mon.1) 348 : audit [DBG] from='client.? 172.21.15.115:0/1919295745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:22.784Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:23 smithi121 bash[36902]: audit 2022-04-22T22:22:22.484249+0000 mon.a (mon.0) 14137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:23 smithi121 bash[36902]: audit 2022-04-22T22:22:22.609277+0000 mon.a (mon.0) 14138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:23 smithi115 bash[45495]: audit 2022-04-22T22:22:22.484249+0000 mon.a (mon.0) 14137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:23 smithi115 bash[45495]: audit 2022-04-22T22:22:22.609277+0000 mon.a (mon.0) 14138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:23 smithi115 bash[46934]: audit 2022-04-22T22:22:22.484249+0000 mon.a (mon.0) 14137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:23 smithi115 bash[46934]: audit 2022-04-22T22:22:22.609277+0000 mon.a (mon.0) 14138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:24 smithi121 bash[36902]: cluster 2022-04-22T22:22:23.279998+0000 mgr.y (mgr.34107) 10106 : cluster [DBG] pgmap v10829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:24.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:24 smithi115 bash[45495]: cluster 2022-04-22T22:22:23.279998+0000 mgr.y (mgr.34107) 10106 : cluster [DBG] pgmap v10829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:24 smithi115 bash[46934]: cluster 2022-04-22T22:22:23.279998+0000 mgr.y (mgr.34107) 10106 : cluster [DBG] pgmap v10829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:24.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:24.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:26 smithi121 bash[36902]: cluster 2022-04-22T22:22:25.281132+0000 mgr.y (mgr.34107) 10107 : cluster [DBG] pgmap v10830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:26 smithi121 bash[36902]: audit 2022-04-22T22:22:26.229793+0000 mon.a (mon.0) 14139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:26 smithi115 bash[46934]: cluster 2022-04-22T22:22:25.281132+0000 mgr.y (mgr.34107) 10107 : cluster [DBG] pgmap v10830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:26 smithi115 bash[46934]: audit 2022-04-22T22:22:26.229793+0000 mon.a (mon.0) 14139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:26 smithi115 bash[45495]: cluster 2022-04-22T22:22:25.281132+0000 mgr.y (mgr.34107) 10107 : cluster [DBG] pgmap v10830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:26 smithi115 bash[45495]: audit 2022-04-22T22:22:26.229793+0000 mon.a (mon.0) 14139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:27 smithi121 bash[36902]: audit 2022-04-22T22:22:27.487646+0000 mon.a (mon.0) 14140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:27 smithi115 bash[45495]: audit 2022-04-22T22:22:27.487646+0000 mon.a (mon.0) 14140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:27 smithi115 bash[46934]: audit 2022-04-22T22:22:27.487646+0000 mon.a (mon.0) 14140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:28 smithi121 bash[36902]: cluster 2022-04-22T22:22:27.282194+0000 mgr.y (mgr.34107) 10108 : cluster [DBG] pgmap v10831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:28 smithi121 bash[36902]: audit 2022-04-22T22:22:27.612208+0000 mon.a (mon.0) 14141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:28 smithi115 bash[45495]: cluster 2022-04-22T22:22:27.282194+0000 mgr.y (mgr.34107) 10108 : cluster [DBG] pgmap v10831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:28 smithi115 bash[45495]: audit 2022-04-22T22:22:27.612208+0000 mon.a (mon.0) 14141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:28 smithi115 bash[46934]: cluster 2022-04-22T22:22:27.282194+0000 mgr.y (mgr.34107) 10108 : cluster [DBG] pgmap v10831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:29.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:28 smithi115 bash[46934]: audit 2022-04-22T22:22:27.612208+0000 mon.a (mon.0) 14141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:30 smithi121 bash[36902]: cluster 2022-04-22T22:22:29.283259+0000 mgr.y (mgr.34107) 10109 : cluster [DBG] pgmap v10832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:30 smithi115 bash[45495]: cluster 2022-04-22T22:22:29.283259+0000 mgr.y (mgr.34107) 10109 : cluster [DBG] pgmap v10832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:30 smithi115 bash[46934]: cluster 2022-04-22T22:22:29.283259+0000 mgr.y (mgr.34107) 10109 : cluster [DBG] pgmap v10832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:32.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:31 smithi121 bash[36902]: audit 2022-04-22T22:22:31.232893+0000 mon.a (mon.0) 14142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:31 smithi115 bash[45495]: audit 2022-04-22T22:22:31.232893+0000 mon.a (mon.0) 14142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:31 smithi115 bash[46934]: audit 2022-04-22T22:22:31.232893+0000 mon.a (mon.0) 14142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:32 smithi121 bash[36902]: cluster 2022-04-22T22:22:31.284339+0000 mgr.y (mgr.34107) 10110 : cluster [DBG] pgmap v10833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:32 smithi121 bash[36902]: audit 2022-04-22T22:22:32.491629+0000 mon.a (mon.0) 14143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:32.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:32 smithi115 bash[46934]: cluster 2022-04-22T22:22:31.284339+0000 mgr.y (mgr.34107) 10110 : cluster [DBG] pgmap v10833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:32 smithi115 bash[46934]: audit 2022-04-22T22:22:32.491629+0000 mon.a (mon.0) 14143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:32 smithi115 bash[45495]: cluster 2022-04-22T22:22:31.284339+0000 mgr.y (mgr.34107) 10110 : cluster [DBG] pgmap v10833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:32 smithi115 bash[45495]: audit 2022-04-22T22:22:32.491629+0000 mon.a (mon.0) 14143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:33 smithi121 bash[36902]: audit 2022-04-22T22:22:32.615423+0000 mon.a (mon.0) 14144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:33 smithi115 bash[46934]: audit 2022-04-22T22:22:32.615423+0000 mon.a (mon.0) 14144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:33 smithi115 bash[45495]: audit 2022-04-22T22:22:32.615423+0000 mon.a (mon.0) 14144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:34 smithi121 bash[36902]: cluster 2022-04-22T22:22:33.285507+0000 mgr.y (mgr.34107) 10111 : cluster [DBG] pgmap v10834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:34.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:34 smithi115 bash[45495]: cluster 2022-04-22T22:22:33.285507+0000 mgr.y (mgr.34107) 10111 : cluster [DBG] pgmap v10834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:34 smithi115 bash[46934]: cluster 2022-04-22T22:22:33.285507+0000 mgr.y (mgr.34107) 10111 : cluster [DBG] pgmap v10834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:36 smithi121 bash[36902]: cluster 2022-04-22T22:22:35.286711+0000 mgr.y (mgr.34107) 10112 : cluster [DBG] pgmap v10835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:36 smithi121 bash[36902]: audit 2022-04-22T22:22:36.236222+0000 mon.a (mon.0) 14145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:36 smithi115 bash[45495]: cluster 2022-04-22T22:22:35.286711+0000 mgr.y (mgr.34107) 10112 : cluster [DBG] pgmap v10835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:36 smithi115 bash[45495]: audit 2022-04-22T22:22:36.236222+0000 mon.a (mon.0) 14145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:36 smithi115 bash[46934]: cluster 2022-04-22T22:22:35.286711+0000 mgr.y (mgr.34107) 10112 : cluster [DBG] pgmap v10835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:36 smithi115 bash[46934]: audit 2022-04-22T22:22:36.236222+0000 mon.a (mon.0) 14145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:37 smithi121 bash[36902]: audit 2022-04-22T22:22:37.495042+0000 mon.a (mon.0) 14146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:37 smithi115 bash[45495]: audit 2022-04-22T22:22:37.495042+0000 mon.a (mon.0) 14146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:37 smithi115 bash[46934]: audit 2022-04-22T22:22:37.495042+0000 mon.a (mon.0) 14146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:38 smithi121 bash[36902]: cluster 2022-04-22T22:22:37.287838+0000 mgr.y (mgr.34107) 10113 : cluster [DBG] pgmap v10836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:38 smithi121 bash[36902]: audit 2022-04-22T22:22:37.618335+0000 mon.a (mon.0) 14147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:38 smithi115 bash[45495]: cluster 2022-04-22T22:22:37.287838+0000 mgr.y (mgr.34107) 10113 : cluster [DBG] pgmap v10836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:38 smithi115 bash[45495]: audit 2022-04-22T22:22:37.618335+0000 mon.a (mon.0) 14147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:38 smithi115 bash[46934]: cluster 2022-04-22T22:22:37.287838+0000 mgr.y (mgr.34107) 10113 : cluster [DBG] pgmap v10836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:39.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:38 smithi115 bash[46934]: audit 2022-04-22T22:22:37.618335+0000 mon.a (mon.0) 14147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:40 smithi121 bash[36902]: cluster 2022-04-22T22:22:39.288937+0000 mgr.y (mgr.34107) 10114 : cluster [DBG] pgmap v10837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:40 smithi115 bash[45495]: cluster 2022-04-22T22:22:39.288937+0000 mgr.y (mgr.34107) 10114 : cluster [DBG] pgmap v10837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:40 smithi115 bash[46934]: cluster 2022-04-22T22:22:39.288937+0000 mgr.y (mgr.34107) 10114 : cluster [DBG] pgmap v10837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:41 smithi121 bash[36902]: audit 2022-04-22T22:22:41.239814+0000 mon.a (mon.0) 14148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:41 smithi115 bash[45495]: audit 2022-04-22T22:22:41.239814+0000 mon.a (mon.0) 14148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:41 smithi115 bash[46934]: audit 2022-04-22T22:22:41.239814+0000 mon.a (mon.0) 14148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:42 smithi121 bash[36902]: cluster 2022-04-22T22:22:41.290061+0000 mgr.y (mgr.34107) 10115 : cluster [DBG] pgmap v10838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:42 smithi121 bash[36902]: audit 2022-04-22T22:22:42.498892+0000 mon.a (mon.0) 14149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:42.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:42 smithi115 bash[45495]: cluster 2022-04-22T22:22:41.290061+0000 mgr.y (mgr.34107) 10115 : cluster [DBG] pgmap v10838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:42 smithi115 bash[45495]: audit 2022-04-22T22:22:42.498892+0000 mon.a (mon.0) 14149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:42 smithi115 bash[46934]: cluster 2022-04-22T22:22:41.290061+0000 mgr.y (mgr.34107) 10115 : cluster [DBG] pgmap v10838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:42 smithi115 bash[46934]: audit 2022-04-22T22:22:42.498892+0000 mon.a (mon.0) 14149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:43 smithi121 bash[36902]: audit 2022-04-22T22:22:42.621520+0000 mon.a (mon.0) 14150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:44.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:43 smithi115 bash[46934]: audit 2022-04-22T22:22:42.621520+0000 mon.a (mon.0) 14150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:43 smithi115 bash[45495]: audit 2022-04-22T22:22:42.621520+0000 mon.a (mon.0) 14150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:44 smithi121 bash[36902]: cluster 2022-04-22T22:22:43.291136+0000 mgr.y (mgr.34107) 10116 : cluster [DBG] pgmap v10839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:44.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:44 smithi115 bash[45495]: cluster 2022-04-22T22:22:43.291136+0000 mgr.y (mgr.34107) 10116 : cluster [DBG] pgmap v10839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:44 smithi115 bash[46934]: cluster 2022-04-22T22:22:43.291136+0000 mgr.y (mgr.34107) 10116 : cluster [DBG] pgmap v10839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:45 smithi121 bash[36902]: cluster 2022-04-22T22:22:45.292195+0000 mgr.y (mgr.34107) 10117 : cluster [DBG] pgmap v10840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:45 smithi115 bash[45495]: cluster 2022-04-22T22:22:45.292195+0000 mgr.y (mgr.34107) 10117 : cluster [DBG] pgmap v10840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:45 smithi115 bash[46934]: cluster 2022-04-22T22:22:45.292195+0000 mgr.y (mgr.34107) 10117 : cluster [DBG] pgmap v10840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:46 smithi121 bash[36902]: audit 2022-04-22T22:22:46.242984+0000 mon.a (mon.0) 14151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:46 smithi115 bash[45495]: audit 2022-04-22T22:22:46.242984+0000 mon.a (mon.0) 14151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:46 smithi115 bash[46934]: audit 2022-04-22T22:22:46.242984+0000 mon.a (mon.0) 14151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:47 smithi121 bash[36902]: cluster 2022-04-22T22:22:47.293239+0000 mgr.y (mgr.34107) 10118 : cluster [DBG] pgmap v10841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:47 smithi121 bash[36902]: audit 2022-04-22T22:22:47.502530+0000 mon.a (mon.0) 14152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:47 smithi115 bash[45495]: cluster 2022-04-22T22:22:47.293239+0000 mgr.y (mgr.34107) 10118 : cluster [DBG] pgmap v10841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:47 smithi115 bash[45495]: audit 2022-04-22T22:22:47.502530+0000 mon.a (mon.0) 14152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:47 smithi115 bash[46934]: cluster 2022-04-22T22:22:47.293239+0000 mgr.y (mgr.34107) 10118 : cluster [DBG] pgmap v10841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:47 smithi115 bash[46934]: audit 2022-04-22T22:22:47.502530+0000 mon.a (mon.0) 14152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:48 smithi121 bash[36902]: audit 2022-04-22T22:22:47.628605+0000 mon.a (mon.0) 14153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:48 smithi115 bash[45495]: audit 2022-04-22T22:22:47.628605+0000 mon.a (mon.0) 14153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:48 smithi115 bash[46934]: audit 2022-04-22T22:22:47.628605+0000 mon.a (mon.0) 14153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:49 smithi121 bash[36902]: cluster 2022-04-22T22:22:49.294283+0000 mgr.y (mgr.34107) 10119 : cluster [DBG] pgmap v10842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:49 smithi115 bash[46934]: cluster 2022-04-22T22:22:49.294283+0000 mgr.y (mgr.34107) 10119 : cluster [DBG] pgmap v10842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:49 smithi115 bash[45495]: cluster 2022-04-22T22:22:49.294283+0000 mgr.y (mgr.34107) 10119 : cluster [DBG] pgmap v10842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:51 smithi121 bash[36902]: audit 2022-04-22T22:22:51.246115+0000 mon.a (mon.0) 14154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:51.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:51 smithi115 bash[46934]: audit 2022-04-22T22:22:51.246115+0000 mon.a (mon.0) 14154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:51 smithi115 bash[45495]: audit 2022-04-22T22:22:51.246115+0000 mon.a (mon.0) 14154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:51.999 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:22:52.347 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:22:52.347 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 4h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:22:52.347 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:22:52.347 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:22:52.347 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 6m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 6m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 6m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 4h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 4h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 6m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:22:52.348 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 6m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 6m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 6m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:22:52.349 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 5h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:22:52.758 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:22:52.758 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:22:52.759 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:22:52.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:22:52.760 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:22:52.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:22:52.760 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:22:52.760 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:22:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:52 smithi121 bash[36902]: cluster 2022-04-22T22:22:51.295356+0000 mgr.y (mgr.34107) 10120 : cluster [DBG] pgmap v10843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:52.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:52 smithi115 bash[45495]: cluster 2022-04-22T22:22:51.295356+0000 mgr.y (mgr.34107) 10120 : cluster [DBG] pgmap v10843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:52.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:52 smithi115 bash[46934]: cluster 2022-04-22T22:22:51.295356+0000 mgr.y (mgr.34107) 10120 : cluster [DBG] pgmap v10843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:52.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:53 smithi121 bash[36902]: audit 2022-04-22T22:22:51.980352+0000 mgr.y (mgr.34107) 10121 : audit [DBG] from='client.55777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:53 smithi121 bash[36902]: audit 2022-04-22T22:22:52.324925+0000 mgr.y (mgr.34107) 10122 : audit [DBG] from='client.45852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:53 smithi121 bash[36902]: audit 2022-04-22T22:22:52.506171+0000 mon.a (mon.0) 14155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:53 smithi121 bash[36902]: audit 2022-04-22T22:22:52.754969+0000 mon.a (mon.0) 14156 : audit [DBG] from='client.? 172.21.15.115:0/3845340446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[45495]: audit 2022-04-22T22:22:51.980352+0000 mgr.y (mgr.34107) 10121 : audit [DBG] from='client.55777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[45495]: audit 2022-04-22T22:22:52.324925+0000 mgr.y (mgr.34107) 10122 : audit [DBG] from='client.45852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[45495]: audit 2022-04-22T22:22:52.506171+0000 mon.a (mon.0) 14155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:53.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[45495]: audit 2022-04-22T22:22:52.754969+0000 mon.a (mon.0) 14156 : audit [DBG] from='client.? 172.21.15.115:0/3845340446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:53.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[46934]: audit 2022-04-22T22:22:51.980352+0000 mgr.y (mgr.34107) 10121 : audit [DBG] from='client.55777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[46934]: audit 2022-04-22T22:22:52.324925+0000 mgr.y (mgr.34107) 10122 : audit [DBG] from='client.45852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:22:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[46934]: audit 2022-04-22T22:22:52.506171+0000 mon.a (mon.0) 14155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:53.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:53 smithi115 bash[46934]: audit 2022-04-22T22:22:52.754969+0000 mon.a (mon.0) 14156 : audit [DBG] from='client.? 172.21.15.115:0/3845340446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:22:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:54 smithi121 bash[36902]: cluster 2022-04-22T22:22:53.296483+0000 mgr.y (mgr.34107) 10123 : cluster [DBG] pgmap v10844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:54 smithi121 bash[36902]: audit 2022-04-22T22:22:53.627898+0000 mon.a (mon.0) 14157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:54 smithi115 bash[45495]: cluster 2022-04-22T22:22:53.296483+0000 mgr.y (mgr.34107) 10123 : cluster [DBG] pgmap v10844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:54 smithi115 bash[45495]: audit 2022-04-22T22:22:53.627898+0000 mon.a (mon.0) 14157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:54.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:22:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:22:54.638Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:22:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:54 smithi115 bash[46934]: cluster 2022-04-22T22:22:53.296483+0000 mgr.y (mgr.34107) 10123 : cluster [DBG] pgmap v10844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:54 smithi115 bash[46934]: audit 2022-04-22T22:22:53.627898+0000 mon.a (mon.0) 14157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:56 smithi121 bash[36902]: cluster 2022-04-22T22:22:55.297588+0000 mgr.y (mgr.34107) 10124 : cluster [DBG] pgmap v10845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:56 smithi121 bash[36902]: audit 2022-04-22T22:22:56.249592+0000 mon.a (mon.0) 14158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:56 smithi115 bash[45495]: cluster 2022-04-22T22:22:55.297588+0000 mgr.y (mgr.34107) 10124 : cluster [DBG] pgmap v10845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:56 smithi115 bash[45495]: audit 2022-04-22T22:22:56.249592+0000 mon.a (mon.0) 14158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:56 smithi115 bash[46934]: cluster 2022-04-22T22:22:55.297588+0000 mgr.y (mgr.34107) 10124 : cluster [DBG] pgmap v10845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:56 smithi115 bash[46934]: audit 2022-04-22T22:22:56.249592+0000 mon.a (mon.0) 14158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:22:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:58 smithi121 bash[36902]: cluster 2022-04-22T22:22:57.298631+0000 mgr.y (mgr.34107) 10125 : cluster [DBG] pgmap v10846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:58 smithi121 bash[36902]: audit 2022-04-22T22:22:57.509805+0000 mon.a (mon.0) 14159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:58 smithi115 bash[45495]: cluster 2022-04-22T22:22:57.298631+0000 mgr.y (mgr.34107) 10125 : cluster [DBG] pgmap v10846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:58 smithi115 bash[45495]: audit 2022-04-22T22:22:57.509805+0000 mon.a (mon.0) 14159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:58 smithi115 bash[46934]: cluster 2022-04-22T22:22:57.298631+0000 mgr.y (mgr.34107) 10125 : cluster [DBG] pgmap v10846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:22:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:58 smithi115 bash[46934]: audit 2022-04-22T22:22:57.509805+0000 mon.a (mon.0) 14159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:22:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:22:59 smithi121 bash[36902]: audit 2022-04-22T22:22:58.630908+0000 mon.a (mon.0) 14160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:59.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:22:59 smithi115 bash[45495]: audit 2022-04-22T22:22:58.630908+0000 mon.a (mon.0) 14160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:22:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:22:59 smithi115 bash[46934]: audit 2022-04-22T22:22:58.630908+0000 mon.a (mon.0) 14160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:00 smithi121 bash[36902]: cluster 2022-04-22T22:22:59.299729+0000 mgr.y (mgr.34107) 10126 : cluster [DBG] pgmap v10847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:00 smithi115 bash[45495]: cluster 2022-04-22T22:22:59.299729+0000 mgr.y (mgr.34107) 10126 : cluster [DBG] pgmap v10847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:00 smithi115 bash[46934]: cluster 2022-04-22T22:22:59.299729+0000 mgr.y (mgr.34107) 10126 : cluster [DBG] pgmap v10847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:01 smithi121 bash[36902]: audit 2022-04-22T22:23:01.252638+0000 mon.a (mon.0) 14161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:01 smithi115 bash[45495]: audit 2022-04-22T22:23:01.252638+0000 mon.a (mon.0) 14161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:01 smithi115 bash[46934]: audit 2022-04-22T22:23:01.252638+0000 mon.a (mon.0) 14161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:02 smithi121 bash[36902]: cluster 2022-04-22T22:23:01.300799+0000 mgr.y (mgr.34107) 10127 : cluster [DBG] pgmap v10848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:02 smithi115 bash[45495]: cluster 2022-04-22T22:23:01.300799+0000 mgr.y (mgr.34107) 10127 : cluster [DBG] pgmap v10848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:02 smithi115 bash[46934]: cluster 2022-04-22T22:23:01.300799+0000 mgr.y (mgr.34107) 10127 : cluster [DBG] pgmap v10848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:02.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:03 smithi121 bash[36902]: audit 2022-04-22T22:23:02.513481+0000 mon.a (mon.0) 14162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:03 smithi115 bash[45495]: audit 2022-04-22T22:23:02.513481+0000 mon.a (mon.0) 14162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:03 smithi115 bash[46934]: audit 2022-04-22T22:23:02.513481+0000 mon.a (mon.0) 14162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:04 smithi121 bash[36902]: cluster 2022-04-22T22:23:03.301872+0000 mgr.y (mgr.34107) 10128 : cluster [DBG] pgmap v10849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:04 smithi121 bash[36902]: audit 2022-04-22T22:23:03.634147+0000 mon.a (mon.0) 14163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:04.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:04.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:04 smithi115 bash[45495]: cluster 2022-04-22T22:23:03.301872+0000 mgr.y (mgr.34107) 10128 : cluster [DBG] pgmap v10849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:04 smithi115 bash[45495]: audit 2022-04-22T22:23:03.634147+0000 mon.a (mon.0) 14163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:04 smithi115 bash[46934]: cluster 2022-04-22T22:23:03.301872+0000 mgr.y (mgr.34107) 10128 : cluster [DBG] pgmap v10849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:04 smithi115 bash[46934]: audit 2022-04-22T22:23:03.634147+0000 mon.a (mon.0) 14163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:06 smithi121 bash[36902]: cluster 2022-04-22T22:23:05.302923+0000 mgr.y (mgr.34107) 10129 : cluster [DBG] pgmap v10850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:06 smithi121 bash[36902]: audit 2022-04-22T22:23:06.255794+0000 mon.a (mon.0) 14164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:06 smithi115 bash[45495]: cluster 2022-04-22T22:23:05.302923+0000 mgr.y (mgr.34107) 10129 : cluster [DBG] pgmap v10850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:06 smithi115 bash[45495]: audit 2022-04-22T22:23:06.255794+0000 mon.a (mon.0) 14164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:06 smithi115 bash[46934]: cluster 2022-04-22T22:23:05.302923+0000 mgr.y (mgr.34107) 10129 : cluster [DBG] pgmap v10850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:06 smithi115 bash[46934]: audit 2022-04-22T22:23:06.255794+0000 mon.a (mon.0) 14164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:07 smithi121 bash[36902]: audit 2022-04-22T22:23:07.014806+0000 mon.a (mon.0) 14165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:23:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:07 smithi121 bash[36902]: audit 2022-04-22T22:23:07.019962+0000 mon.a (mon.0) 14166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:23:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:07 smithi121 bash[36902]: audit 2022-04-22T22:23:07.021508+0000 mon.a (mon.0) 14167 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:23:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:07 smithi121 bash[36902]: audit 2022-04-22T22:23:07.369339+0000 mon.a (mon.0) 14168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:23:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[45495]: audit 2022-04-22T22:23:07.014806+0000 mon.a (mon.0) 14165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:23:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[45495]: audit 2022-04-22T22:23:07.019962+0000 mon.a (mon.0) 14166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:23:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[45495]: audit 2022-04-22T22:23:07.021508+0000 mon.a (mon.0) 14167 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:23:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[45495]: audit 2022-04-22T22:23:07.369339+0000 mon.a (mon.0) 14168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:23:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[46934]: audit 2022-04-22T22:23:07.014806+0000 mon.a (mon.0) 14165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:23:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[46934]: audit 2022-04-22T22:23:07.019962+0000 mon.a (mon.0) 14166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:23:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[46934]: audit 2022-04-22T22:23:07.021508+0000 mon.a (mon.0) 14167 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:23:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:07 smithi115 bash[46934]: audit 2022-04-22T22:23:07.369339+0000 mon.a (mon.0) 14168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:23:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:08 smithi115 bash[45495]: cluster 2022-04-22T22:23:07.303784+0000 mgr.y (mgr.34107) 10130 : cluster [DBG] pgmap v10851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:08 smithi115 bash[45495]: audit 2022-04-22T22:23:07.519370+0000 mon.a (mon.0) 14169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:08 smithi115 bash[46934]: cluster 2022-04-22T22:23:07.303784+0000 mgr.y (mgr.34107) 10130 : cluster [DBG] pgmap v10851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:08 smithi115 bash[46934]: audit 2022-04-22T22:23:07.519370+0000 mon.a (mon.0) 14169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:08 smithi121 bash[36902]: cluster 2022-04-22T22:23:07.303784+0000 mgr.y (mgr.34107) 10130 : cluster [DBG] pgmap v10851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:08 smithi121 bash[36902]: audit 2022-04-22T22:23:07.519370+0000 mon.a (mon.0) 14169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:09 smithi115 bash[45495]: audit 2022-04-22T22:23:08.637322+0000 mon.a (mon.0) 14170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:09 smithi115 bash[46934]: audit 2022-04-22T22:23:08.637322+0000 mon.a (mon.0) 14170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:09 smithi121 bash[36902]: audit 2022-04-22T22:23:08.637322+0000 mon.a (mon.0) 14170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:10.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:10 smithi115 bash[45495]: cluster 2022-04-22T22:23:09.304750+0000 mgr.y (mgr.34107) 10131 : cluster [DBG] pgmap v10852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:10 smithi115 bash[46934]: cluster 2022-04-22T22:23:09.304750+0000 mgr.y (mgr.34107) 10131 : cluster [DBG] pgmap v10852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:10 smithi121 bash[36902]: cluster 2022-04-22T22:23:09.304750+0000 mgr.y (mgr.34107) 10131 : cluster [DBG] pgmap v10852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:11 smithi115 bash[45495]: audit 2022-04-22T22:23:11.258851+0000 mon.a (mon.0) 14171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:11 smithi115 bash[46934]: audit 2022-04-22T22:23:11.258851+0000 mon.a (mon.0) 14171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:11 smithi121 bash[36902]: audit 2022-04-22T22:23:11.258851+0000 mon.a (mon.0) 14171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:12 smithi115 bash[45495]: cluster 2022-04-22T22:23:11.305692+0000 mgr.y (mgr.34107) 10132 : cluster [DBG] pgmap v10853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:12 smithi115 bash[45495]: audit 2022-04-22T22:23:12.520254+0000 mon.a (mon.0) 14172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:12 smithi115 bash[46934]: cluster 2022-04-22T22:23:11.305692+0000 mgr.y (mgr.34107) 10132 : cluster [DBG] pgmap v10853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:12 smithi115 bash[46934]: audit 2022-04-22T22:23:12.520254+0000 mon.a (mon.0) 14172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:12 smithi121 bash[36902]: cluster 2022-04-22T22:23:11.305692+0000 mgr.y (mgr.34107) 10132 : cluster [DBG] pgmap v10853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:12 smithi121 bash[36902]: audit 2022-04-22T22:23:12.520254+0000 mon.a (mon.0) 14172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:12.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:14 smithi121 bash[36902]: cluster 2022-04-22T22:23:13.306774+0000 mgr.y (mgr.34107) 10133 : cluster [DBG] pgmap v10854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:14 smithi121 bash[36902]: audit 2022-04-22T22:23:13.640322+0000 mon.a (mon.0) 14173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:15.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:14 smithi115 bash[46934]: cluster 2022-04-22T22:23:13.306774+0000 mgr.y (mgr.34107) 10133 : cluster [DBG] pgmap v10854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:15.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:14 smithi115 bash[46934]: audit 2022-04-22T22:23:13.640322+0000 mon.a (mon.0) 14173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:14.640Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:14 smithi115 bash[45495]: cluster 2022-04-22T22:23:13.306774+0000 mgr.y (mgr.34107) 10133 : cluster [DBG] pgmap v10854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:14 smithi115 bash[45495]: audit 2022-04-22T22:23:13.640322+0000 mon.a (mon.0) 14173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:16 smithi121 bash[36902]: cluster 2022-04-22T22:23:15.308167+0000 mgr.y (mgr.34107) 10134 : cluster [DBG] pgmap v10855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:16 smithi121 bash[36902]: audit 2022-04-22T22:23:16.262127+0000 mon.a (mon.0) 14174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:16 smithi115 bash[45495]: cluster 2022-04-22T22:23:15.308167+0000 mgr.y (mgr.34107) 10134 : cluster [DBG] pgmap v10855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:16 smithi115 bash[45495]: audit 2022-04-22T22:23:16.262127+0000 mon.a (mon.0) 14174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:16 smithi115 bash[46934]: cluster 2022-04-22T22:23:15.308167+0000 mgr.y (mgr.34107) 10134 : cluster [DBG] pgmap v10855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:16 smithi115 bash[46934]: audit 2022-04-22T22:23:16.262127+0000 mon.a (mon.0) 14174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:17 smithi121 bash[36902]: audit 2022-04-22T22:23:16.696827+0000 mon.a (mon.0) 14175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:23:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:17 smithi121 bash[36902]: audit 2022-04-22T22:23:16.700276+0000 mon.a (mon.0) 14176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:23:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:17 smithi121 bash[36902]: audit 2022-04-22T22:23:17.523549+0000 mon.a (mon.0) 14177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[45495]: audit 2022-04-22T22:23:16.696827+0000 mon.a (mon.0) 14175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[45495]: audit 2022-04-22T22:23:16.700276+0000 mon.a (mon.0) 14176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[45495]: audit 2022-04-22T22:23:17.523549+0000 mon.a (mon.0) 14177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[46934]: audit 2022-04-22T22:23:16.696827+0000 mon.a (mon.0) 14175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[46934]: audit 2022-04-22T22:23:16.700276+0000 mon.a (mon.0) 14176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:23:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:17 smithi115 bash[46934]: audit 2022-04-22T22:23:17.523549+0000 mon.a (mon.0) 14177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:18 smithi121 bash[36902]: cluster 2022-04-22T22:23:17.309306+0000 mgr.y (mgr.34107) 10135 : cluster [DBG] pgmap v10856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:18 smithi115 bash[45495]: cluster 2022-04-22T22:23:17.309306+0000 mgr.y (mgr.34107) 10135 : cluster [DBG] pgmap v10856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:18 smithi115 bash[46934]: cluster 2022-04-22T22:23:17.309306+0000 mgr.y (mgr.34107) 10135 : cluster [DBG] pgmap v10856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:19 smithi121 bash[36902]: audit 2022-04-22T22:23:18.643137+0000 mon.a (mon.0) 14178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:19 smithi115 bash[45495]: audit 2022-04-22T22:23:18.643137+0000 mon.a (mon.0) 14178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:19 smithi115 bash[46934]: audit 2022-04-22T22:23:18.643137+0000 mon.a (mon.0) 14178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:20 smithi121 bash[36902]: cluster 2022-04-22T22:23:19.310403+0000 mgr.y (mgr.34107) 10136 : cluster [DBG] pgmap v10857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:20 smithi115 bash[45495]: cluster 2022-04-22T22:23:19.310403+0000 mgr.y (mgr.34107) 10136 : cluster [DBG] pgmap v10857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:20 smithi115 bash[46934]: cluster 2022-04-22T22:23:19.310403+0000 mgr.y (mgr.34107) 10136 : cluster [DBG] pgmap v10857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:21 smithi121 bash[36902]: audit 2022-04-22T22:23:21.265445+0000 mon.a (mon.0) 14179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:21 smithi115 bash[45495]: audit 2022-04-22T22:23:21.265445+0000 mon.a (mon.0) 14179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:21 smithi115 bash[46934]: audit 2022-04-22T22:23:21.265445+0000 mon.a (mon.0) 14179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:22 smithi121 bash[36902]: cluster 2022-04-22T22:23:21.311441+0000 mgr.y (mgr.34107) 10137 : cluster [DBG] pgmap v10858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:22 smithi121 bash[36902]: audit 2022-04-22T22:23:22.527282+0000 mon.a (mon.0) 14180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:22.789Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:22 smithi115 bash[45495]: cluster 2022-04-22T22:23:21.311441+0000 mgr.y (mgr.34107) 10137 : cluster [DBG] pgmap v10858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:22 smithi115 bash[45495]: audit 2022-04-22T22:23:22.527282+0000 mon.a (mon.0) 14180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:22 smithi115 bash[46934]: cluster 2022-04-22T22:23:21.311441+0000 mgr.y (mgr.34107) 10137 : cluster [DBG] pgmap v10858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:22 smithi115 bash[46934]: audit 2022-04-22T22:23:22.527282+0000 mon.a (mon.0) 14180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:23.134 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:23:23.482 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:23:23.482 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:23:23.482 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (4h) 8m ago 4h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:23:23.482 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 8m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 5h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:23:23.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:23:23.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:23:23.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:23:23.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 8m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:23:23.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 8m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:23:23.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:23:23.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:23:23.485 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 5h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:23:23.882 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:23:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:23:23.884 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:23:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:24 smithi121 bash[36902]: audit 2022-04-22T22:23:23.115400+0000 mgr.y (mgr.34107) 10138 : audit [DBG] from='client.55795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:24 smithi121 bash[36902]: cluster 2022-04-22T22:23:23.312464+0000 mgr.y (mgr.34107) 10139 : cluster [DBG] pgmap v10859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:24 smithi121 bash[36902]: audit 2022-04-22T22:23:23.460763+0000 mgr.y (mgr.34107) 10140 : audit [DBG] from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:24 smithi121 bash[36902]: audit 2022-04-22T22:23:23.646219+0000 mon.a (mon.0) 14181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:24 smithi121 bash[36902]: audit 2022-04-22T22:23:23.878871+0000 mon.a (mon.0) 14182 : audit [DBG] from='client.? 172.21.15.115:0/2403189714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:24.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[45495]: audit 2022-04-22T22:23:23.115400+0000 mgr.y (mgr.34107) 10138 : audit [DBG] from='client.55795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[45495]: cluster 2022-04-22T22:23:23.312464+0000 mgr.y (mgr.34107) 10139 : cluster [DBG] pgmap v10859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[45495]: audit 2022-04-22T22:23:23.460763+0000 mgr.y (mgr.34107) 10140 : audit [DBG] from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[45495]: audit 2022-04-22T22:23:23.646219+0000 mon.a (mon.0) 14181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[45495]: audit 2022-04-22T22:23:23.878871+0000 mon.a (mon.0) 14182 : audit [DBG] from='client.? 172.21.15.115:0/2403189714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[46934]: audit 2022-04-22T22:23:23.115400+0000 mgr.y (mgr.34107) 10138 : audit [DBG] from='client.55795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[46934]: cluster 2022-04-22T22:23:23.312464+0000 mgr.y (mgr.34107) 10139 : cluster [DBG] pgmap v10859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[46934]: audit 2022-04-22T22:23:23.460763+0000 mgr.y (mgr.34107) 10140 : audit [DBG] from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[46934]: audit 2022-04-22T22:23:23.646219+0000 mon.a (mon.0) 14181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:24 smithi115 bash[46934]: audit 2022-04-22T22:23:23.878871+0000 mon.a (mon.0) 14182 : audit [DBG] from='client.? 172.21.15.115:0/2403189714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:26 smithi121 bash[36902]: cluster 2022-04-22T22:23:25.313575+0000 mgr.y (mgr.34107) 10141 : cluster [DBG] pgmap v10860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:26 smithi121 bash[36902]: audit 2022-04-22T22:23:26.268595+0000 mon.a (mon.0) 14183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:26 smithi115 bash[45495]: cluster 2022-04-22T22:23:25.313575+0000 mgr.y (mgr.34107) 10141 : cluster [DBG] pgmap v10860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:26 smithi115 bash[45495]: audit 2022-04-22T22:23:26.268595+0000 mon.a (mon.0) 14183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:26 smithi115 bash[46934]: cluster 2022-04-22T22:23:25.313575+0000 mgr.y (mgr.34107) 10141 : cluster [DBG] pgmap v10860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:26 smithi115 bash[46934]: audit 2022-04-22T22:23:26.268595+0000 mon.a (mon.0) 14183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:27 smithi121 bash[36902]: audit 2022-04-22T22:23:27.531069+0000 mon.a (mon.0) 14184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:27 smithi115 bash[45495]: audit 2022-04-22T22:23:27.531069+0000 mon.a (mon.0) 14184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:27 smithi115 bash[46934]: audit 2022-04-22T22:23:27.531069+0000 mon.a (mon.0) 14184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:28 smithi121 bash[36902]: cluster 2022-04-22T22:23:27.314607+0000 mgr.y (mgr.34107) 10142 : cluster [DBG] pgmap v10861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:28 smithi115 bash[45495]: cluster 2022-04-22T22:23:27.314607+0000 mgr.y (mgr.34107) 10142 : cluster [DBG] pgmap v10861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:28 smithi115 bash[46934]: cluster 2022-04-22T22:23:27.314607+0000 mgr.y (mgr.34107) 10142 : cluster [DBG] pgmap v10861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:29 smithi121 bash[36902]: audit 2022-04-22T22:23:28.649151+0000 mon.a (mon.0) 14185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:29 smithi115 bash[45495]: audit 2022-04-22T22:23:28.649151+0000 mon.a (mon.0) 14185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:29 smithi115 bash[46934]: audit 2022-04-22T22:23:28.649151+0000 mon.a (mon.0) 14185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:30 smithi121 bash[36902]: cluster 2022-04-22T22:23:29.315779+0000 mgr.y (mgr.34107) 10143 : cluster [DBG] pgmap v10862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:30 smithi115 bash[45495]: cluster 2022-04-22T22:23:29.315779+0000 mgr.y (mgr.34107) 10143 : cluster [DBG] pgmap v10862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:30 smithi115 bash[46934]: cluster 2022-04-22T22:23:29.315779+0000 mgr.y (mgr.34107) 10143 : cluster [DBG] pgmap v10862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:31 smithi121 bash[36902]: audit 2022-04-22T22:23:31.271951+0000 mon.a (mon.0) 14186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:31 smithi115 bash[45495]: audit 2022-04-22T22:23:31.271951+0000 mon.a (mon.0) 14186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:32.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:31 smithi115 bash[46934]: audit 2022-04-22T22:23:31.271951+0000 mon.a (mon.0) 14186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:32 smithi121 bash[36902]: cluster 2022-04-22T22:23:31.316722+0000 mgr.y (mgr.34107) 10144 : cluster [DBG] pgmap v10863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:32 smithi121 bash[36902]: audit 2022-04-22T22:23:32.534595+0000 mon.a (mon.0) 14187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:32.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:32 smithi115 bash[45495]: cluster 2022-04-22T22:23:31.316722+0000 mgr.y (mgr.34107) 10144 : cluster [DBG] pgmap v10863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:32 smithi115 bash[45495]: audit 2022-04-22T22:23:32.534595+0000 mon.a (mon.0) 14187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:32 smithi115 bash[46934]: cluster 2022-04-22T22:23:31.316722+0000 mgr.y (mgr.34107) 10144 : cluster [DBG] pgmap v10863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:32 smithi115 bash[46934]: audit 2022-04-22T22:23:32.534595+0000 mon.a (mon.0) 14187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:33 smithi121 bash[36902]: cluster 2022-04-22T22:23:33.317712+0000 mgr.y (mgr.34107) 10145 : cluster [DBG] pgmap v10864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:33 smithi115 bash[45495]: cluster 2022-04-22T22:23:33.317712+0000 mgr.y (mgr.34107) 10145 : cluster [DBG] pgmap v10864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:33 smithi115 bash[46934]: cluster 2022-04-22T22:23:33.317712+0000 mgr.y (mgr.34107) 10145 : cluster [DBG] pgmap v10864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:34 smithi121 bash[36902]: audit 2022-04-22T22:23:33.651907+0000 mon.a (mon.0) 14188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:34 smithi115 bash[45495]: audit 2022-04-22T22:23:33.651907+0000 mon.a (mon.0) 14188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:34.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:34 smithi115 bash[46934]: audit 2022-04-22T22:23:33.651907+0000 mon.a (mon.0) 14188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:35 smithi121 bash[36902]: cluster 2022-04-22T22:23:35.318742+0000 mgr.y (mgr.34107) 10146 : cluster [DBG] pgmap v10865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:35 smithi115 bash[45495]: cluster 2022-04-22T22:23:35.318742+0000 mgr.y (mgr.34107) 10146 : cluster [DBG] pgmap v10865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:35 smithi115 bash[46934]: cluster 2022-04-22T22:23:35.318742+0000 mgr.y (mgr.34107) 10146 : cluster [DBG] pgmap v10865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:36 smithi121 bash[36902]: audit 2022-04-22T22:23:36.274901+0000 mon.a (mon.0) 14189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:36 smithi115 bash[45495]: audit 2022-04-22T22:23:36.274901+0000 mon.a (mon.0) 14189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:36 smithi115 bash[46934]: audit 2022-04-22T22:23:36.274901+0000 mon.a (mon.0) 14189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:37 smithi121 bash[36902]: cluster 2022-04-22T22:23:37.319852+0000 mgr.y (mgr.34107) 10147 : cluster [DBG] pgmap v10866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:38.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:37 smithi121 bash[36902]: audit 2022-04-22T22:23:37.538306+0000 mon.a (mon.0) 14190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:38.182 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:37 smithi115 bash[45495]: cluster 2022-04-22T22:23:37.319852+0000 mgr.y (mgr.34107) 10147 : cluster [DBG] pgmap v10866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:38.183 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:37 smithi115 bash[45495]: audit 2022-04-22T22:23:37.538306+0000 mon.a (mon.0) 14190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:38.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:37 smithi115 bash[46934]: cluster 2022-04-22T22:23:37.319852+0000 mgr.y (mgr.34107) 10147 : cluster [DBG] pgmap v10866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:38.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:37 smithi115 bash[46934]: audit 2022-04-22T22:23:37.538306+0000 mon.a (mon.0) 14190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:38 smithi121 bash[36902]: audit 2022-04-22T22:23:38.654587+0000 mon.a (mon.0) 14191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:38 smithi115 bash[45495]: audit 2022-04-22T22:23:38.654587+0000 mon.a (mon.0) 14191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:38 smithi115 bash[46934]: audit 2022-04-22T22:23:38.654587+0000 mon.a (mon.0) 14191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:39 smithi121 bash[36902]: cluster 2022-04-22T22:23:39.320910+0000 mgr.y (mgr.34107) 10148 : cluster [DBG] pgmap v10867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:40.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:39 smithi115 bash[46934]: cluster 2022-04-22T22:23:39.320910+0000 mgr.y (mgr.34107) 10148 : cluster [DBG] pgmap v10867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:39 smithi115 bash[45495]: cluster 2022-04-22T22:23:39.320910+0000 mgr.y (mgr.34107) 10148 : cluster [DBG] pgmap v10867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:41 smithi121 bash[36902]: audit 2022-04-22T22:23:41.277917+0000 mon.a (mon.0) 14192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:41 smithi115 bash[45495]: audit 2022-04-22T22:23:41.277917+0000 mon.a (mon.0) 14192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:41 smithi115 bash[46934]: audit 2022-04-22T22:23:41.277917+0000 mon.a (mon.0) 14192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:42.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:42 smithi121 bash[36902]: cluster 2022-04-22T22:23:41.321950+0000 mgr.y (mgr.34107) 10149 : cluster [DBG] pgmap v10868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:42 smithi115 bash[46934]: cluster 2022-04-22T22:23:41.321950+0000 mgr.y (mgr.34107) 10149 : cluster [DBG] pgmap v10868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:42 smithi115 bash[45495]: cluster 2022-04-22T22:23:41.321950+0000 mgr.y (mgr.34107) 10149 : cluster [DBG] pgmap v10868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:42.791Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:43 smithi121 bash[36902]: audit 2022-04-22T22:23:42.542004+0000 mon.a (mon.0) 14193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:43 smithi115 bash[45495]: audit 2022-04-22T22:23:42.542004+0000 mon.a (mon.0) 14193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:43 smithi115 bash[46934]: audit 2022-04-22T22:23:42.542004+0000 mon.a (mon.0) 14193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:44 smithi121 bash[36902]: cluster 2022-04-22T22:23:43.323091+0000 mgr.y (mgr.34107) 10150 : cluster [DBG] pgmap v10869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:44 smithi121 bash[36902]: audit 2022-04-22T22:23:43.657746+0000 mon.a (mon.0) 14194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:44.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:44 smithi115 bash[45495]: cluster 2022-04-22T22:23:43.323091+0000 mgr.y (mgr.34107) 10150 : cluster [DBG] pgmap v10869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:44 smithi115 bash[45495]: audit 2022-04-22T22:23:43.657746+0000 mon.a (mon.0) 14194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:44 smithi115 bash[46934]: cluster 2022-04-22T22:23:43.323091+0000 mgr.y (mgr.34107) 10150 : cluster [DBG] pgmap v10869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:44.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:44 smithi115 bash[46934]: audit 2022-04-22T22:23:43.657746+0000 mon.a (mon.0) 14194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:46 smithi121 bash[36902]: cluster 2022-04-22T22:23:45.324234+0000 mgr.y (mgr.34107) 10151 : cluster [DBG] pgmap v10870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:46 smithi121 bash[36902]: audit 2022-04-22T22:23:46.281114+0000 mon.a (mon.0) 14195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:46.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:46 smithi115 bash[46934]: cluster 2022-04-22T22:23:45.324234+0000 mgr.y (mgr.34107) 10151 : cluster [DBG] pgmap v10870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:46 smithi115 bash[46934]: audit 2022-04-22T22:23:46.281114+0000 mon.a (mon.0) 14195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:46 smithi115 bash[45495]: cluster 2022-04-22T22:23:45.324234+0000 mgr.y (mgr.34107) 10151 : cluster [DBG] pgmap v10870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:46 smithi115 bash[45495]: audit 2022-04-22T22:23:46.281114+0000 mon.a (mon.0) 14195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:48 smithi121 bash[36902]: cluster 2022-04-22T22:23:47.325237+0000 mgr.y (mgr.34107) 10152 : cluster [DBG] pgmap v10871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:48 smithi121 bash[36902]: audit 2022-04-22T22:23:47.545572+0000 mon.a (mon.0) 14196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:48.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:48 smithi115 bash[46934]: cluster 2022-04-22T22:23:47.325237+0000 mgr.y (mgr.34107) 10152 : cluster [DBG] pgmap v10871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:48 smithi115 bash[46934]: audit 2022-04-22T22:23:47.545572+0000 mon.a (mon.0) 14196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:48 smithi115 bash[45495]: cluster 2022-04-22T22:23:47.325237+0000 mgr.y (mgr.34107) 10152 : cluster [DBG] pgmap v10871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:48 smithi115 bash[45495]: audit 2022-04-22T22:23:47.545572+0000 mon.a (mon.0) 14196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:49 smithi121 bash[36902]: audit 2022-04-22T22:23:48.660796+0000 mon.a (mon.0) 14197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:49.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:49 smithi115 bash[46934]: audit 2022-04-22T22:23:48.660796+0000 mon.a (mon.0) 14197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:49 smithi115 bash[45495]: audit 2022-04-22T22:23:48.660796+0000 mon.a (mon.0) 14197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:50 smithi121 bash[36902]: cluster 2022-04-22T22:23:49.326341+0000 mgr.y (mgr.34107) 10153 : cluster [DBG] pgmap v10872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:50 smithi115 bash[45495]: cluster 2022-04-22T22:23:49.326341+0000 mgr.y (mgr.34107) 10153 : cluster [DBG] pgmap v10872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:50 smithi115 bash[46934]: cluster 2022-04-22T22:23:49.326341+0000 mgr.y (mgr.34107) 10153 : cluster [DBG] pgmap v10872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:51 smithi121 bash[36902]: audit 2022-04-22T22:23:51.284200+0000 mon.a (mon.0) 14198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:51 smithi115 bash[45495]: audit 2022-04-22T22:23:51.284200+0000 mon.a (mon.0) 14198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:51 smithi115 bash[46934]: audit 2022-04-22T22:23:51.284200+0000 mon.a (mon.0) 14198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:52 smithi121 bash[36902]: cluster 2022-04-22T22:23:51.327412+0000 mgr.y (mgr.34107) 10154 : cluster [DBG] pgmap v10873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:52 smithi115 bash[45495]: cluster 2022-04-22T22:23:51.327412+0000 mgr.y (mgr.34107) 10154 : cluster [DBG] pgmap v10873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:52 smithi115 bash[46934]: cluster 2022-04-22T22:23:51.327412+0000 mgr.y (mgr.34107) 10154 : cluster [DBG] pgmap v10873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:52.792Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:53 smithi121 bash[36902]: audit 2022-04-22T22:23:52.549558+0000 mon.a (mon.0) 14199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:53 smithi115 bash[45495]: audit 2022-04-22T22:23:52.549558+0000 mon.a (mon.0) 14199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:53 smithi115 bash[46934]: audit 2022-04-22T22:23:52.549558+0000 mon.a (mon.0) 14199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:54.255 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:23:54.608 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 7m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 7m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 7m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 5h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 7m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 7m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 7m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 7m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:23:54.609 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:23:54.610 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:23:54.610 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:23:54.610 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:23:54.610 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 5h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:23:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:54 smithi121 bash[36902]: cluster 2022-04-22T22:23:53.328555+0000 mgr.y (mgr.34107) 10155 : cluster [DBG] pgmap v10874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:54 smithi121 bash[36902]: audit 2022-04-22T22:23:53.664047+0000 mon.a (mon.0) 14200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:23:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:23:54.641Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:23:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:54 smithi115 bash[46934]: cluster 2022-04-22T22:23:53.328555+0000 mgr.y (mgr.34107) 10155 : cluster [DBG] pgmap v10874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:54 smithi115 bash[46934]: audit 2022-04-22T22:23:53.664047+0000 mon.a (mon.0) 14200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:54 smithi115 bash[45495]: cluster 2022-04-22T22:23:53.328555+0000 mgr.y (mgr.34107) 10155 : cluster [DBG] pgmap v10874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:54 smithi115 bash[45495]: audit 2022-04-22T22:23:53.664047+0000 mon.a (mon.0) 14200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:55.015 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:23:55.016 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:23:55.017 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:23:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[45495]: audit 2022-04-22T22:23:54.236313+0000 mgr.y (mgr.34107) 10156 : audit [DBG] from='client.55813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[45495]: audit 2022-04-22T22:23:54.586215+0000 mgr.y (mgr.34107) 10157 : audit [DBG] from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[45495]: audit 2022-04-22T22:23:55.012121+0000 mon.a (mon.0) 14201 : audit [DBG] from='client.? 172.21.15.115:0/3431304838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[46934]: audit 2022-04-22T22:23:54.236313+0000 mgr.y (mgr.34107) 10156 : audit [DBG] from='client.55813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[46934]: audit 2022-04-22T22:23:54.586215+0000 mgr.y (mgr.34107) 10157 : audit [DBG] from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:55 smithi115 bash[46934]: audit 2022-04-22T22:23:55.012121+0000 mon.a (mon.0) 14201 : audit [DBG] from='client.? 172.21.15.115:0/3431304838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:55 smithi121 bash[36902]: audit 2022-04-22T22:23:54.236313+0000 mgr.y (mgr.34107) 10156 : audit [DBG] from='client.55813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:55 smithi121 bash[36902]: audit 2022-04-22T22:23:54.586215+0000 mgr.y (mgr.34107) 10157 : audit [DBG] from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:23:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:55 smithi121 bash[36902]: audit 2022-04-22T22:23:55.012121+0000 mon.a (mon.0) 14201 : audit [DBG] from='client.? 172.21.15.115:0/3431304838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:23:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:56 smithi115 bash[45495]: cluster 2022-04-22T22:23:55.329639+0000 mgr.y (mgr.34107) 10158 : cluster [DBG] pgmap v10875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:56 smithi115 bash[45495]: audit 2022-04-22T22:23:56.287566+0000 mon.a (mon.0) 14202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:56 smithi115 bash[46934]: cluster 2022-04-22T22:23:55.329639+0000 mgr.y (mgr.34107) 10158 : cluster [DBG] pgmap v10875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:56 smithi115 bash[46934]: audit 2022-04-22T22:23:56.287566+0000 mon.a (mon.0) 14202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:56 smithi121 bash[36902]: cluster 2022-04-22T22:23:55.329639+0000 mgr.y (mgr.34107) 10158 : cluster [DBG] pgmap v10875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:56 smithi121 bash[36902]: audit 2022-04-22T22:23:56.287566+0000 mon.a (mon.0) 14202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:23:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:58 smithi115 bash[46934]: cluster 2022-04-22T22:23:57.330632+0000 mgr.y (mgr.34107) 10159 : cluster [DBG] pgmap v10876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:58 smithi115 bash[46934]: audit 2022-04-22T22:23:57.553001+0000 mon.a (mon.0) 14203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:58 smithi115 bash[45495]: cluster 2022-04-22T22:23:57.330632+0000 mgr.y (mgr.34107) 10159 : cluster [DBG] pgmap v10876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:58.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:58 smithi115 bash[45495]: audit 2022-04-22T22:23:57.553001+0000 mon.a (mon.0) 14203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:58 smithi121 bash[36902]: cluster 2022-04-22T22:23:57.330632+0000 mgr.y (mgr.34107) 10159 : cluster [DBG] pgmap v10876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:23:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:58 smithi121 bash[36902]: audit 2022-04-22T22:23:57.553001+0000 mon.a (mon.0) 14203 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:23:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:23:59 smithi115 bash[46934]: audit 2022-04-22T22:23:58.667158+0000 mon.a (mon.0) 14204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:23:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:23:59 smithi115 bash[45495]: audit 2022-04-22T22:23:58.667158+0000 mon.a (mon.0) 14204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:23:59 smithi121 bash[36902]: audit 2022-04-22T22:23:58.667158+0000 mon.a (mon.0) 14204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:00 smithi115 bash[46934]: cluster 2022-04-22T22:23:59.331802+0000 mgr.y (mgr.34107) 10160 : cluster [DBG] pgmap v10877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:00 smithi115 bash[45495]: cluster 2022-04-22T22:23:59.331802+0000 mgr.y (mgr.34107) 10160 : cluster [DBG] pgmap v10877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:00 smithi121 bash[36902]: cluster 2022-04-22T22:23:59.331802+0000 mgr.y (mgr.34107) 10160 : cluster [DBG] pgmap v10877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:01 smithi121 bash[36902]: audit 2022-04-22T22:24:01.290704+0000 mon.a (mon.0) 14205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:02.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:01 smithi115 bash[46934]: audit 2022-04-22T22:24:01.290704+0000 mon.a (mon.0) 14205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:01 smithi115 bash[45495]: audit 2022-04-22T22:24:01.290704+0000 mon.a (mon.0) 14205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:02.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:02 smithi115 bash[45495]: cluster 2022-04-22T22:24:01.332847+0000 mgr.y (mgr.34107) 10161 : cluster [DBG] pgmap v10878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:02.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:02 smithi115 bash[46934]: cluster 2022-04-22T22:24:01.332847+0000 mgr.y (mgr.34107) 10161 : cluster [DBG] pgmap v10878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:02 smithi121 bash[36902]: cluster 2022-04-22T22:24:01.332847+0000 mgr.y (mgr.34107) 10161 : cluster [DBG] pgmap v10878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:02.792Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:03 smithi121 bash[36902]: audit 2022-04-22T22:24:02.556114+0000 mon.a (mon.0) 14206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:03 smithi115 bash[45495]: audit 2022-04-22T22:24:02.556114+0000 mon.a (mon.0) 14206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:03 smithi115 bash[46934]: audit 2022-04-22T22:24:02.556114+0000 mon.a (mon.0) 14206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:04 smithi121 bash[36902]: cluster 2022-04-22T22:24:03.333942+0000 mgr.y (mgr.34107) 10162 : cluster [DBG] pgmap v10879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:04 smithi121 bash[36902]: audit 2022-04-22T22:24:03.670398+0000 mon.a (mon.0) 14207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:04.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:04 smithi115 bash[45495]: cluster 2022-04-22T22:24:03.333942+0000 mgr.y (mgr.34107) 10162 : cluster [DBG] pgmap v10879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:04 smithi115 bash[45495]: audit 2022-04-22T22:24:03.670398+0000 mon.a (mon.0) 14207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:04 smithi115 bash[46934]: cluster 2022-04-22T22:24:03.333942+0000 mgr.y (mgr.34107) 10162 : cluster [DBG] pgmap v10879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:04 smithi115 bash[46934]: audit 2022-04-22T22:24:03.670398+0000 mon.a (mon.0) 14207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:06 smithi121 bash[36902]: cluster 2022-04-22T22:24:05.335008+0000 mgr.y (mgr.34107) 10163 : cluster [DBG] pgmap v10880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:06 smithi121 bash[36902]: audit 2022-04-22T22:24:06.293817+0000 mon.a (mon.0) 14208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:06 smithi115 bash[46934]: cluster 2022-04-22T22:24:05.335008+0000 mgr.y (mgr.34107) 10163 : cluster [DBG] pgmap v10880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:06 smithi115 bash[46934]: audit 2022-04-22T22:24:06.293817+0000 mon.a (mon.0) 14208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:06 smithi115 bash[45495]: cluster 2022-04-22T22:24:05.335008+0000 mgr.y (mgr.34107) 10163 : cluster [DBG] pgmap v10880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:06 smithi115 bash[45495]: audit 2022-04-22T22:24:06.293817+0000 mon.a (mon.0) 14208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:07.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:07 smithi121 bash[36902]: audit 2022-04-22T22:24:07.379054+0000 mon.a (mon.0) 14209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:24:07.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:07 smithi121 bash[36902]: audit 2022-04-22T22:24:07.381888+0000 mon.a (mon.0) 14210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:24:07.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:07 smithi121 bash[36902]: audit 2022-04-22T22:24:07.383723+0000 mon.a (mon.0) 14211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:24:07.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:07 smithi121 bash[36902]: audit 2022-04-22T22:24:07.559455+0000 mon.a (mon.0) 14212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:07.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[45495]: audit 2022-04-22T22:24:07.379054+0000 mon.a (mon.0) 14209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:24:07.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[45495]: audit 2022-04-22T22:24:07.381888+0000 mon.a (mon.0) 14210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:24:07.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[45495]: audit 2022-04-22T22:24:07.383723+0000 mon.a (mon.0) 14211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:24:07.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[45495]: audit 2022-04-22T22:24:07.559455+0000 mon.a (mon.0) 14212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:07.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[46934]: audit 2022-04-22T22:24:07.379054+0000 mon.a (mon.0) 14209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:24:07.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[46934]: audit 2022-04-22T22:24:07.381888+0000 mon.a (mon.0) 14210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:24:07.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[46934]: audit 2022-04-22T22:24:07.383723+0000 mon.a (mon.0) 14211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:24:07.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:07 smithi115 bash[46934]: audit 2022-04-22T22:24:07.559455+0000 mon.a (mon.0) 14212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:08 smithi121 bash[36902]: cluster 2022-04-22T22:24:07.336047+0000 mgr.y (mgr.34107) 10164 : cluster [DBG] pgmap v10881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:08 smithi121 bash[36902]: audit 2022-04-22T22:24:07.762009+0000 mon.a (mon.0) 14213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:24:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:08 smithi115 bash[45495]: cluster 2022-04-22T22:24:07.336047+0000 mgr.y (mgr.34107) 10164 : cluster [DBG] pgmap v10881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:08 smithi115 bash[45495]: audit 2022-04-22T22:24:07.762009+0000 mon.a (mon.0) 14213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:24:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:08 smithi115 bash[46934]: cluster 2022-04-22T22:24:07.336047+0000 mgr.y (mgr.34107) 10164 : cluster [DBG] pgmap v10881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:08 smithi115 bash[46934]: audit 2022-04-22T22:24:07.762009+0000 mon.a (mon.0) 14213 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:24:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:09 smithi121 bash[36902]: audit 2022-04-22T22:24:08.673330+0000 mon.a (mon.0) 14214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:09 smithi115 bash[45495]: audit 2022-04-22T22:24:08.673330+0000 mon.a (mon.0) 14214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:09 smithi115 bash[46934]: audit 2022-04-22T22:24:08.673330+0000 mon.a (mon.0) 14214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:10 smithi121 bash[36902]: cluster 2022-04-22T22:24:09.337134+0000 mgr.y (mgr.34107) 10165 : cluster [DBG] pgmap v10882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:10 smithi115 bash[45495]: cluster 2022-04-22T22:24:09.337134+0000 mgr.y (mgr.34107) 10165 : cluster [DBG] pgmap v10882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:10 smithi115 bash[46934]: cluster 2022-04-22T22:24:09.337134+0000 mgr.y (mgr.34107) 10165 : cluster [DBG] pgmap v10882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:11 smithi121 bash[36902]: audit 2022-04-22T22:24:11.296936+0000 mon.a (mon.0) 14215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:11 smithi115 bash[45495]: audit 2022-04-22T22:24:11.296936+0000 mon.a (mon.0) 14215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:11 smithi115 bash[46934]: audit 2022-04-22T22:24:11.296936+0000 mon.a (mon.0) 14215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:12 smithi121 bash[36902]: cluster 2022-04-22T22:24:11.338216+0000 mgr.y (mgr.34107) 10166 : cluster [DBG] pgmap v10883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:13.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:12 smithi121 bash[36902]: audit 2022-04-22T22:24:12.562723+0000 mon.a (mon.0) 14216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:13.116 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:12.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:12 smithi115 bash[45495]: cluster 2022-04-22T22:24:11.338216+0000 mgr.y (mgr.34107) 10166 : cluster [DBG] pgmap v10883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:12 smithi115 bash[45495]: audit 2022-04-22T22:24:12.562723+0000 mon.a (mon.0) 14216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:12 smithi115 bash[46934]: cluster 2022-04-22T22:24:11.338216+0000 mgr.y (mgr.34107) 10166 : cluster [DBG] pgmap v10883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:12 smithi115 bash[46934]: audit 2022-04-22T22:24:12.562723+0000 mon.a (mon.0) 14216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:14 smithi121 bash[36902]: cluster 2022-04-22T22:24:13.339339+0000 mgr.y (mgr.34107) 10167 : cluster [DBG] pgmap v10884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:14 smithi121 bash[36902]: audit 2022-04-22T22:24:13.676269+0000 mon.a (mon.0) 14217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:14.643Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:14 smithi115 bash[45495]: cluster 2022-04-22T22:24:13.339339+0000 mgr.y (mgr.34107) 10167 : cluster [DBG] pgmap v10884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:14 smithi115 bash[45495]: audit 2022-04-22T22:24:13.676269+0000 mon.a (mon.0) 14217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:14 smithi115 bash[46934]: cluster 2022-04-22T22:24:13.339339+0000 mgr.y (mgr.34107) 10167 : cluster [DBG] pgmap v10884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:14 smithi115 bash[46934]: audit 2022-04-22T22:24:13.676269+0000 mon.a (mon.0) 14217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:16 smithi121 bash[36902]: cluster 2022-04-22T22:24:15.340398+0000 mgr.y (mgr.34107) 10168 : cluster [DBG] pgmap v10885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:16 smithi121 bash[36902]: audit 2022-04-22T22:24:16.299966+0000 mon.a (mon.0) 14218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:16 smithi115 bash[45495]: cluster 2022-04-22T22:24:15.340398+0000 mgr.y (mgr.34107) 10168 : cluster [DBG] pgmap v10885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:16 smithi115 bash[45495]: audit 2022-04-22T22:24:16.299966+0000 mon.a (mon.0) 14218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:16 smithi115 bash[46934]: cluster 2022-04-22T22:24:15.340398+0000 mgr.y (mgr.34107) 10168 : cluster [DBG] pgmap v10885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:16 smithi115 bash[46934]: audit 2022-04-22T22:24:16.299966+0000 mon.a (mon.0) 14218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:17 smithi121 bash[36902]: audit 2022-04-22T22:24:16.708426+0000 mon.a (mon.0) 14219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:24:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:17 smithi121 bash[36902]: audit 2022-04-22T22:24:16.708856+0000 mon.a (mon.0) 14220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:24:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:17 smithi121 bash[36902]: audit 2022-04-22T22:24:17.566457+0000 mon.a (mon.0) 14221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[45495]: audit 2022-04-22T22:24:16.708426+0000 mon.a (mon.0) 14219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:24:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[45495]: audit 2022-04-22T22:24:16.708856+0000 mon.a (mon.0) 14220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:24:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[45495]: audit 2022-04-22T22:24:17.566457+0000 mon.a (mon.0) 14221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[46934]: audit 2022-04-22T22:24:16.708426+0000 mon.a (mon.0) 14219 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:24:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[46934]: audit 2022-04-22T22:24:16.708856+0000 mon.a (mon.0) 14220 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:24:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:17 smithi115 bash[46934]: audit 2022-04-22T22:24:17.566457+0000 mon.a (mon.0) 14221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:18 smithi121 bash[36902]: cluster 2022-04-22T22:24:17.341445+0000 mgr.y (mgr.34107) 10169 : cluster [DBG] pgmap v10886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:18 smithi115 bash[45495]: cluster 2022-04-22T22:24:17.341445+0000 mgr.y (mgr.34107) 10169 : cluster [DBG] pgmap v10886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:18 smithi115 bash[46934]: cluster 2022-04-22T22:24:17.341445+0000 mgr.y (mgr.34107) 10169 : cluster [DBG] pgmap v10886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:19 smithi121 bash[36902]: audit 2022-04-22T22:24:18.679306+0000 mon.a (mon.0) 14222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:19 smithi115 bash[45495]: audit 2022-04-22T22:24:18.679306+0000 mon.a (mon.0) 14222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:19 smithi115 bash[46934]: audit 2022-04-22T22:24:18.679306+0000 mon.a (mon.0) 14222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:20 smithi121 bash[36902]: cluster 2022-04-22T22:24:19.342524+0000 mgr.y (mgr.34107) 10170 : cluster [DBG] pgmap v10887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:20 smithi115 bash[45495]: cluster 2022-04-22T22:24:19.342524+0000 mgr.y (mgr.34107) 10170 : cluster [DBG] pgmap v10887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:20 smithi115 bash[46934]: cluster 2022-04-22T22:24:19.342524+0000 mgr.y (mgr.34107) 10170 : cluster [DBG] pgmap v10887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:21 smithi121 bash[36902]: audit 2022-04-22T22:24:21.302978+0000 mon.a (mon.0) 14223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:21 smithi115 bash[45495]: audit 2022-04-22T22:24:21.302978+0000 mon.a (mon.0) 14223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:21 smithi115 bash[46934]: audit 2022-04-22T22:24:21.302978+0000 mon.a (mon.0) 14223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:22 smithi121 bash[36902]: cluster 2022-04-22T22:24:21.343512+0000 mgr.y (mgr.34107) 10171 : cluster [DBG] pgmap v10888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:22 smithi121 bash[36902]: audit 2022-04-22T22:24:22.570384+0000 mon.a (mon.0) 14224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:22.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:22 smithi115 bash[45495]: cluster 2022-04-22T22:24:21.343512+0000 mgr.y (mgr.34107) 10171 : cluster [DBG] pgmap v10888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:22 smithi115 bash[45495]: audit 2022-04-22T22:24:22.570384+0000 mon.a (mon.0) 14224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:22 smithi115 bash[46934]: cluster 2022-04-22T22:24:21.343512+0000 mgr.y (mgr.34107) 10171 : cluster [DBG] pgmap v10888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:22 smithi115 bash[46934]: audit 2022-04-22T22:24:22.570384+0000 mon.a (mon.0) 14224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:23 smithi121 bash[36902]: cluster 2022-04-22T22:24:23.344553+0000 mgr.y (mgr.34107) 10172 : cluster [DBG] pgmap v10889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:23 smithi115 bash[45495]: cluster 2022-04-22T22:24:23.344553+0000 mgr.y (mgr.34107) 10172 : cluster [DBG] pgmap v10889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:23 smithi115 bash[46934]: cluster 2022-04-22T22:24:23.344553+0000 mgr.y (mgr.34107) 10172 : cluster [DBG] pgmap v10889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:24 smithi121 bash[36902]: audit 2022-04-22T22:24:23.682286+0000 mon.a (mon.0) 14225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:25.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:24 smithi115 bash[46934]: audit 2022-04-22T22:24:23.682286+0000 mon.a (mon.0) 14225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:24 smithi115 bash[45495]: audit 2022-04-22T22:24:23.682286+0000 mon.a (mon.0) 14225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:24.645Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:25.388 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:24:25.736 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:24:25.736 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:24:25.736 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 8m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 8m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 9m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 8m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:24:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 5h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 8m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 8m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 8m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 8m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 9m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 9m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 9m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:24:25.738 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 5h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:24:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:25 smithi121 bash[36902]: cluster 2022-04-22T22:24:25.345600+0000 mgr.y (mgr.34107) 10173 : cluster [DBG] pgmap v10890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:25 smithi121 bash[36902]: audit 2022-04-22T22:24:25.369953+0000 mgr.y (mgr.34107) 10174 : audit [DBG] from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:25 smithi115 bash[45495]: cluster 2022-04-22T22:24:25.345600+0000 mgr.y (mgr.34107) 10173 : cluster [DBG] pgmap v10890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:25 smithi115 bash[45495]: audit 2022-04-22T22:24:25.369953+0000 mgr.y (mgr.34107) 10174 : audit [DBG] from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:25 smithi115 bash[46934]: cluster 2022-04-22T22:24:25.345600+0000 mgr.y (mgr.34107) 10173 : cluster [DBG] pgmap v10890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:26.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:25 smithi115 bash[46934]: audit 2022-04-22T22:24:25.369953+0000 mgr.y (mgr.34107) 10174 : audit [DBG] from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:26.147 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:24:26.148 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:24:26.149 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:24:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:26 smithi121 bash[36902]: audit 2022-04-22T22:24:25.712707+0000 mgr.y (mgr.34107) 10175 : audit [DBG] from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:26 smithi121 bash[36902]: audit 2022-04-22T22:24:26.143988+0000 mon.c (mon.1) 349 : audit [DBG] from='client.? 172.21.15.115:0/1089064416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:26 smithi121 bash[36902]: audit 2022-04-22T22:24:26.305270+0000 mon.a (mon.0) 14226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[45495]: audit 2022-04-22T22:24:25.712707+0000 mgr.y (mgr.34107) 10175 : audit [DBG] from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[45495]: audit 2022-04-22T22:24:26.143988+0000 mon.c (mon.1) 349 : audit [DBG] from='client.? 172.21.15.115:0/1089064416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[45495]: audit 2022-04-22T22:24:26.305270+0000 mon.a (mon.0) 14226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[46934]: audit 2022-04-22T22:24:25.712707+0000 mgr.y (mgr.34107) 10175 : audit [DBG] from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[46934]: audit 2022-04-22T22:24:26.143988+0000 mon.c (mon.1) 349 : audit [DBG] from='client.? 172.21.15.115:0/1089064416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:26 smithi115 bash[46934]: audit 2022-04-22T22:24:26.305270+0000 mon.a (mon.0) 14226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:27 smithi121 bash[36902]: cluster 2022-04-22T22:24:27.346722+0000 mgr.y (mgr.34107) 10176 : cluster [DBG] pgmap v10891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:27 smithi121 bash[36902]: audit 2022-04-22T22:24:27.574200+0000 mon.a (mon.0) 14227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:27 smithi115 bash[45495]: cluster 2022-04-22T22:24:27.346722+0000 mgr.y (mgr.34107) 10176 : cluster [DBG] pgmap v10891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:27 smithi115 bash[45495]: audit 2022-04-22T22:24:27.574200+0000 mon.a (mon.0) 14227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:27 smithi115 bash[46934]: cluster 2022-04-22T22:24:27.346722+0000 mgr.y (mgr.34107) 10176 : cluster [DBG] pgmap v10891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:27 smithi115 bash[46934]: audit 2022-04-22T22:24:27.574200+0000 mon.a (mon.0) 14227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:28 smithi121 bash[36902]: audit 2022-04-22T22:24:28.685201+0000 mon.a (mon.0) 14228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:28 smithi115 bash[45495]: audit 2022-04-22T22:24:28.685201+0000 mon.a (mon.0) 14228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:28 smithi115 bash[46934]: audit 2022-04-22T22:24:28.685201+0000 mon.a (mon.0) 14228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:29 smithi121 bash[36902]: cluster 2022-04-22T22:24:29.347751+0000 mgr.y (mgr.34107) 10177 : cluster [DBG] pgmap v10892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:29 smithi115 bash[45495]: cluster 2022-04-22T22:24:29.347751+0000 mgr.y (mgr.34107) 10177 : cluster [DBG] pgmap v10892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:29 smithi115 bash[46934]: cluster 2022-04-22T22:24:29.347751+0000 mgr.y (mgr.34107) 10177 : cluster [DBG] pgmap v10892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:31 smithi121 bash[36902]: audit 2022-04-22T22:24:31.308404+0000 mon.a (mon.0) 14229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:31 smithi115 bash[45495]: audit 2022-04-22T22:24:31.308404+0000 mon.a (mon.0) 14229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:31 smithi115 bash[46934]: audit 2022-04-22T22:24:31.308404+0000 mon.a (mon.0) 14229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:32 smithi121 bash[36902]: cluster 2022-04-22T22:24:31.348869+0000 mgr.y (mgr.34107) 10178 : cluster [DBG] pgmap v10893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:32 smithi115 bash[46934]: cluster 2022-04-22T22:24:31.348869+0000 mgr.y (mgr.34107) 10178 : cluster [DBG] pgmap v10893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:32 smithi115 bash[45495]: cluster 2022-04-22T22:24:31.348869+0000 mgr.y (mgr.34107) 10178 : cluster [DBG] pgmap v10893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:32.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:33 smithi121 bash[36902]: audit 2022-04-22T22:24:32.577862+0000 mon.a (mon.0) 14230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:33 smithi115 bash[45495]: audit 2022-04-22T22:24:32.577862+0000 mon.a (mon.0) 14230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:33 smithi115 bash[46934]: audit 2022-04-22T22:24:32.577862+0000 mon.a (mon.0) 14230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:34 smithi121 bash[36902]: cluster 2022-04-22T22:24:33.350119+0000 mgr.y (mgr.34107) 10179 : cluster [DBG] pgmap v10894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:34 smithi121 bash[36902]: audit 2022-04-22T22:24:33.688215+0000 mon.a (mon.0) 14231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:34.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:34.646Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:34 smithi115 bash[45495]: cluster 2022-04-22T22:24:33.350119+0000 mgr.y (mgr.34107) 10179 : cluster [DBG] pgmap v10894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:34 smithi115 bash[45495]: audit 2022-04-22T22:24:33.688215+0000 mon.a (mon.0) 14231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:34 smithi115 bash[46934]: cluster 2022-04-22T22:24:33.350119+0000 mgr.y (mgr.34107) 10179 : cluster [DBG] pgmap v10894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:34.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:34 smithi115 bash[46934]: audit 2022-04-22T22:24:33.688215+0000 mon.a (mon.0) 14231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:36 smithi121 bash[36902]: cluster 2022-04-22T22:24:35.351280+0000 mgr.y (mgr.34107) 10180 : cluster [DBG] pgmap v10895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:36 smithi121 bash[36902]: audit 2022-04-22T22:24:36.311295+0000 mon.a (mon.0) 14232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:36 smithi115 bash[45495]: cluster 2022-04-22T22:24:35.351280+0000 mgr.y (mgr.34107) 10180 : cluster [DBG] pgmap v10895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:36 smithi115 bash[45495]: audit 2022-04-22T22:24:36.311295+0000 mon.a (mon.0) 14232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:36 smithi115 bash[46934]: cluster 2022-04-22T22:24:35.351280+0000 mgr.y (mgr.34107) 10180 : cluster [DBG] pgmap v10895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:36 smithi115 bash[46934]: audit 2022-04-22T22:24:36.311295+0000 mon.a (mon.0) 14232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:38 smithi121 bash[36902]: cluster 2022-04-22T22:24:37.352395+0000 mgr.y (mgr.34107) 10181 : cluster [DBG] pgmap v10896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:38 smithi121 bash[36902]: audit 2022-04-22T22:24:37.581451+0000 mon.a (mon.0) 14233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:38 smithi115 bash[45495]: cluster 2022-04-22T22:24:37.352395+0000 mgr.y (mgr.34107) 10181 : cluster [DBG] pgmap v10896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:38 smithi115 bash[45495]: audit 2022-04-22T22:24:37.581451+0000 mon.a (mon.0) 14233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:38 smithi115 bash[46934]: cluster 2022-04-22T22:24:37.352395+0000 mgr.y (mgr.34107) 10181 : cluster [DBG] pgmap v10896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:38 smithi115 bash[46934]: audit 2022-04-22T22:24:37.581451+0000 mon.a (mon.0) 14233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:39 smithi121 bash[36902]: audit 2022-04-22T22:24:38.691148+0000 mon.a (mon.0) 14234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:39 smithi115 bash[45495]: audit 2022-04-22T22:24:38.691148+0000 mon.a (mon.0) 14234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:39 smithi115 bash[46934]: audit 2022-04-22T22:24:38.691148+0000 mon.a (mon.0) 14234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:40 smithi121 bash[36902]: cluster 2022-04-22T22:24:39.353465+0000 mgr.y (mgr.34107) 10182 : cluster [DBG] pgmap v10897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:40 smithi115 bash[45495]: cluster 2022-04-22T22:24:39.353465+0000 mgr.y (mgr.34107) 10182 : cluster [DBG] pgmap v10897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:40 smithi115 bash[46934]: cluster 2022-04-22T22:24:39.353465+0000 mgr.y (mgr.34107) 10182 : cluster [DBG] pgmap v10897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:41 smithi121 bash[36902]: audit 2022-04-22T22:24:41.314258+0000 mon.a (mon.0) 14235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:41 smithi115 bash[45495]: audit 2022-04-22T22:24:41.314258+0000 mon.a (mon.0) 14235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:41 smithi115 bash[46934]: audit 2022-04-22T22:24:41.314258+0000 mon.a (mon.0) 14235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:42.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:42 smithi115 bash[46934]: cluster 2022-04-22T22:24:41.354565+0000 mgr.y (mgr.34107) 10183 : cluster [DBG] pgmap v10898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:42 smithi115 bash[45495]: cluster 2022-04-22T22:24:41.354565+0000 mgr.y (mgr.34107) 10183 : cluster [DBG] pgmap v10898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:42 smithi121 bash[36902]: cluster 2022-04-22T22:24:41.354565+0000 mgr.y (mgr.34107) 10183 : cluster [DBG] pgmap v10898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:42.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:43 smithi115 bash[45495]: audit 2022-04-22T22:24:42.585423+0000 mon.a (mon.0) 14236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:43 smithi115 bash[46934]: audit 2022-04-22T22:24:42.585423+0000 mon.a (mon.0) 14236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:43 smithi121 bash[36902]: audit 2022-04-22T22:24:42.585423+0000 mon.a (mon.0) 14236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:44 smithi115 bash[45495]: cluster 2022-04-22T22:24:43.355751+0000 mgr.y (mgr.34107) 10184 : cluster [DBG] pgmap v10899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:44 smithi115 bash[45495]: audit 2022-04-22T22:24:43.694000+0000 mon.a (mon.0) 14237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:44.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:44.647Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:44 smithi115 bash[46934]: cluster 2022-04-22T22:24:43.355751+0000 mgr.y (mgr.34107) 10184 : cluster [DBG] pgmap v10899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:44 smithi115 bash[46934]: audit 2022-04-22T22:24:43.694000+0000 mon.a (mon.0) 14237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:44 smithi121 bash[36902]: cluster 2022-04-22T22:24:43.355751+0000 mgr.y (mgr.34107) 10184 : cluster [DBG] pgmap v10899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:44 smithi121 bash[36902]: audit 2022-04-22T22:24:43.694000+0000 mon.a (mon.0) 14237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:46 smithi115 bash[45495]: cluster 2022-04-22T22:24:45.356781+0000 mgr.y (mgr.34107) 10185 : cluster [DBG] pgmap v10900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:46 smithi115 bash[45495]: audit 2022-04-22T22:24:46.317461+0000 mon.a (mon.0) 14238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:46 smithi115 bash[46934]: cluster 2022-04-22T22:24:45.356781+0000 mgr.y (mgr.34107) 10185 : cluster [DBG] pgmap v10900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:46 smithi115 bash[46934]: audit 2022-04-22T22:24:46.317461+0000 mon.a (mon.0) 14238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:46 smithi121 bash[36902]: cluster 2022-04-22T22:24:45.356781+0000 mgr.y (mgr.34107) 10185 : cluster [DBG] pgmap v10900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:46 smithi121 bash[36902]: audit 2022-04-22T22:24:46.317461+0000 mon.a (mon.0) 14238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:48 smithi115 bash[45495]: cluster 2022-04-22T22:24:47.357973+0000 mgr.y (mgr.34107) 10186 : cluster [DBG] pgmap v10901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:48 smithi115 bash[45495]: audit 2022-04-22T22:24:47.588978+0000 mon.a (mon.0) 14239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:48 smithi115 bash[46934]: cluster 2022-04-22T22:24:47.357973+0000 mgr.y (mgr.34107) 10186 : cluster [DBG] pgmap v10901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:48 smithi115 bash[46934]: audit 2022-04-22T22:24:47.588978+0000 mon.a (mon.0) 14239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:48 smithi121 bash[36902]: cluster 2022-04-22T22:24:47.357973+0000 mgr.y (mgr.34107) 10186 : cluster [DBG] pgmap v10901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:48 smithi121 bash[36902]: audit 2022-04-22T22:24:47.588978+0000 mon.a (mon.0) 14239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:49 smithi115 bash[45495]: audit 2022-04-22T22:24:48.697207+0000 mon.a (mon.0) 14240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:49 smithi115 bash[46934]: audit 2022-04-22T22:24:48.697207+0000 mon.a (mon.0) 14240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:49 smithi121 bash[36902]: audit 2022-04-22T22:24:48.697207+0000 mon.a (mon.0) 14240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:50 smithi121 bash[36902]: cluster 2022-04-22T22:24:49.359040+0000 mgr.y (mgr.34107) 10187 : cluster [DBG] pgmap v10902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:50 smithi115 bash[45495]: cluster 2022-04-22T22:24:49.359040+0000 mgr.y (mgr.34107) 10187 : cluster [DBG] pgmap v10902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:51.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:50 smithi115 bash[46934]: cluster 2022-04-22T22:24:49.359040+0000 mgr.y (mgr.34107) 10187 : cluster [DBG] pgmap v10902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:51 smithi121 bash[36902]: audit 2022-04-22T22:24:51.320114+0000 mon.a (mon.0) 14241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:51 smithi115 bash[45495]: audit 2022-04-22T22:24:51.320114+0000 mon.a (mon.0) 14241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:51 smithi115 bash[46934]: audit 2022-04-22T22:24:51.320114+0000 mon.a (mon.0) 14241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:52 smithi121 bash[36902]: cluster 2022-04-22T22:24:51.360086+0000 mgr.y (mgr.34107) 10188 : cluster [DBG] pgmap v10903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:52 smithi115 bash[45495]: cluster 2022-04-22T22:24:51.360086+0000 mgr.y (mgr.34107) 10188 : cluster [DBG] pgmap v10903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:52.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:52 smithi115 bash[46934]: cluster 2022-04-22T22:24:51.360086+0000 mgr.y (mgr.34107) 10188 : cluster [DBG] pgmap v10903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:53 smithi121 bash[36902]: audit 2022-04-22T22:24:52.592535+0000 mon.a (mon.0) 14242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:53 smithi115 bash[45495]: audit 2022-04-22T22:24:52.592535+0000 mon.a (mon.0) 14242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:53 smithi115 bash[46934]: audit 2022-04-22T22:24:52.592535+0000 mon.a (mon.0) 14242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:54 smithi121 bash[36902]: cluster 2022-04-22T22:24:53.361192+0000 mgr.y (mgr.34107) 10189 : cluster [DBG] pgmap v10904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:54 smithi121 bash[36902]: audit 2022-04-22T22:24:53.700210+0000 mon.a (mon.0) 14243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:24:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:24:54.647Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:24:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:54 smithi115 bash[45495]: cluster 2022-04-22T22:24:53.361192+0000 mgr.y (mgr.34107) 10189 : cluster [DBG] pgmap v10904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:54 smithi115 bash[45495]: audit 2022-04-22T22:24:53.700210+0000 mon.a (mon.0) 14243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:54 smithi115 bash[46934]: cluster 2022-04-22T22:24:53.361192+0000 mgr.y (mgr.34107) 10189 : cluster [DBG] pgmap v10904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:54 smithi115 bash[46934]: audit 2022-04-22T22:24:53.700210+0000 mon.a (mon.0) 14243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:24:56.525 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:24:56.872 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:24:56.872 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 10m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 10m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 10m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 10m ago 5h 279M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:24:56.873 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 10m ago 5h 12.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 10m ago 5h 586M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 10m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 10m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:24:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 10m ago 5h 648M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:24:56.875 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 10m ago 5h 68.0M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:24:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:56 smithi121 bash[36902]: cluster 2022-04-22T22:24:55.362320+0000 mgr.y (mgr.34107) 10190 : cluster [DBG] pgmap v10905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:56 smithi121 bash[36902]: audit 2022-04-22T22:24:56.323013+0000 mon.a (mon.0) 14244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:56 smithi115 bash[45495]: cluster 2022-04-22T22:24:55.362320+0000 mgr.y (mgr.34107) 10190 : cluster [DBG] pgmap v10905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:56 smithi115 bash[45495]: audit 2022-04-22T22:24:56.323013+0000 mon.a (mon.0) 14244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:56 smithi115 bash[46934]: cluster 2022-04-22T22:24:55.362320+0000 mgr.y (mgr.34107) 10190 : cluster [DBG] pgmap v10905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:56 smithi115 bash[46934]: audit 2022-04-22T22:24:56.323013+0000 mon.a (mon.0) 14244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:24:57.279 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:24:57.279 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:24:57.279 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:24:57.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:24:57.280 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:24:57.281 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:24:57.281 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:24:57.281 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:24:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:57 smithi121 bash[36902]: audit 2022-04-22T22:24:56.506124+0000 mgr.y (mgr.34107) 10191 : audit [DBG] from='client.55849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:57 smithi121 bash[36902]: audit 2022-04-22T22:24:57.275605+0000 mon.a (mon.0) 14245 : audit [DBG] from='client.? 172.21.15.115:0/2689210806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:57 smithi115 bash[45495]: audit 2022-04-22T22:24:56.506124+0000 mgr.y (mgr.34107) 10191 : audit [DBG] from='client.55849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:57 smithi115 bash[45495]: audit 2022-04-22T22:24:57.275605+0000 mon.a (mon.0) 14245 : audit [DBG] from='client.? 172.21.15.115:0/2689210806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:57 smithi115 bash[46934]: audit 2022-04-22T22:24:56.506124+0000 mgr.y (mgr.34107) 10191 : audit [DBG] from='client.55849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:57 smithi115 bash[46934]: audit 2022-04-22T22:24:57.275605+0000 mon.a (mon.0) 14245 : audit [DBG] from='client.? 172.21.15.115:0/2689210806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:24:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:58 smithi121 bash[36902]: audit 2022-04-22T22:24:56.848510+0000 mgr.y (mgr.34107) 10192 : audit [DBG] from='client.55855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:58 smithi121 bash[36902]: cluster 2022-04-22T22:24:57.363452+0000 mgr.y (mgr.34107) 10193 : cluster [DBG] pgmap v10906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:58 smithi121 bash[36902]: audit 2022-04-22T22:24:57.596192+0000 mon.a (mon.0) 14246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[45495]: audit 2022-04-22T22:24:56.848510+0000 mgr.y (mgr.34107) 10192 : audit [DBG] from='client.55855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[45495]: cluster 2022-04-22T22:24:57.363452+0000 mgr.y (mgr.34107) 10193 : cluster [DBG] pgmap v10906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[45495]: audit 2022-04-22T22:24:57.596192+0000 mon.a (mon.0) 14246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:24:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[46934]: audit 2022-04-22T22:24:56.848510+0000 mgr.y (mgr.34107) 10192 : audit [DBG] from='client.55855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:24:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[46934]: cluster 2022-04-22T22:24:57.363452+0000 mgr.y (mgr.34107) 10193 : cluster [DBG] pgmap v10906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:24:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:58 smithi115 bash[46934]: audit 2022-04-22T22:24:57.596192+0000 mon.a (mon.0) 14246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:24:59 smithi121 bash[36902]: audit 2022-04-22T22:24:58.703168+0000 mon.a (mon.0) 14247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:00.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:24:59 smithi115 bash[45495]: audit 2022-04-22T22:24:58.703168+0000 mon.a (mon.0) 14247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:24:59 smithi115 bash[46934]: audit 2022-04-22T22:24:58.703168+0000 mon.a (mon.0) 14247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:00 smithi121 bash[36902]: cluster 2022-04-22T22:24:59.364553+0000 mgr.y (mgr.34107) 10194 : cluster [DBG] pgmap v10907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:00 smithi115 bash[45495]: cluster 2022-04-22T22:24:59.364553+0000 mgr.y (mgr.34107) 10194 : cluster [DBG] pgmap v10907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:00 smithi115 bash[46934]: cluster 2022-04-22T22:24:59.364553+0000 mgr.y (mgr.34107) 10194 : cluster [DBG] pgmap v10907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:01 smithi121 bash[36902]: audit 2022-04-22T22:25:01.325882+0000 mon.a (mon.0) 14248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:01 smithi115 bash[45495]: audit 2022-04-22T22:25:01.325882+0000 mon.a (mon.0) 14248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:01 smithi115 bash[46934]: audit 2022-04-22T22:25:01.325882+0000 mon.a (mon.0) 14248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:02 smithi121 bash[36902]: cluster 2022-04-22T22:25:01.365633+0000 mgr.y (mgr.34107) 10195 : cluster [DBG] pgmap v10908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:03.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:02 smithi115 bash[45495]: cluster 2022-04-22T22:25:01.365633+0000 mgr.y (mgr.34107) 10195 : cluster [DBG] pgmap v10908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:02.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:02 smithi115 bash[46934]: cluster 2022-04-22T22:25:01.365633+0000 mgr.y (mgr.34107) 10195 : cluster [DBG] pgmap v10908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:03 smithi121 bash[36902]: audit 2022-04-22T22:25:02.599695+0000 mon.a (mon.0) 14249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:03 smithi115 bash[45495]: audit 2022-04-22T22:25:02.599695+0000 mon.a (mon.0) 14249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:03 smithi115 bash[46934]: audit 2022-04-22T22:25:02.599695+0000 mon.a (mon.0) 14249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:04 smithi121 bash[36902]: cluster 2022-04-22T22:25:03.366721+0000 mgr.y (mgr.34107) 10196 : cluster [DBG] pgmap v10909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:04 smithi121 bash[36902]: audit 2022-04-22T22:25:03.706360+0000 mon.a (mon.0) 14250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:04.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:04 smithi115 bash[45495]: cluster 2022-04-22T22:25:03.366721+0000 mgr.y (mgr.34107) 10196 : cluster [DBG] pgmap v10909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:04 smithi115 bash[45495]: audit 2022-04-22T22:25:03.706360+0000 mon.a (mon.0) 14250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:04 smithi115 bash[46934]: cluster 2022-04-22T22:25:03.366721+0000 mgr.y (mgr.34107) 10196 : cluster [DBG] pgmap v10909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:04 smithi115 bash[46934]: audit 2022-04-22T22:25:03.706360+0000 mon.a (mon.0) 14250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:06 smithi121 bash[36902]: cluster 2022-04-22T22:25:05.367917+0000 mgr.y (mgr.34107) 10197 : cluster [DBG] pgmap v10910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:06 smithi121 bash[36902]: audit 2022-04-22T22:25:06.328931+0000 mon.a (mon.0) 14251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:06 smithi115 bash[45495]: cluster 2022-04-22T22:25:05.367917+0000 mgr.y (mgr.34107) 10197 : cluster [DBG] pgmap v10910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:06 smithi115 bash[45495]: audit 2022-04-22T22:25:06.328931+0000 mon.a (mon.0) 14251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:06 smithi115 bash[46934]: cluster 2022-04-22T22:25:05.367917+0000 mgr.y (mgr.34107) 10197 : cluster [DBG] pgmap v10910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:06 smithi115 bash[46934]: audit 2022-04-22T22:25:06.328931+0000 mon.a (mon.0) 14251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:07 smithi121 bash[36902]: audit 2022-04-22T22:25:07.603059+0000 mon.a (mon.0) 14252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:07 smithi115 bash[45495]: audit 2022-04-22T22:25:07.603059+0000 mon.a (mon.0) 14252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:07 smithi115 bash[46934]: audit 2022-04-22T22:25:07.603059+0000 mon.a (mon.0) 14252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:08 smithi121 bash[36902]: cluster 2022-04-22T22:25:07.369011+0000 mgr.y (mgr.34107) 10198 : cluster [DBG] pgmap v10911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:08 smithi121 bash[36902]: audit 2022-04-22T22:25:07.769272+0000 mon.a (mon.0) 14253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:25:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:08 smithi121 bash[36902]: audit 2022-04-22T22:25:07.772221+0000 mon.a (mon.0) 14254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:25:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:08 smithi121 bash[36902]: audit 2022-04-22T22:25:07.773755+0000 mon.a (mon.0) 14255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[45495]: cluster 2022-04-22T22:25:07.369011+0000 mgr.y (mgr.34107) 10198 : cluster [DBG] pgmap v10911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[45495]: audit 2022-04-22T22:25:07.769272+0000 mon.a (mon.0) 14253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[45495]: audit 2022-04-22T22:25:07.772221+0000 mon.a (mon.0) 14254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[45495]: audit 2022-04-22T22:25:07.773755+0000 mon.a (mon.0) 14255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[46934]: cluster 2022-04-22T22:25:07.369011+0000 mgr.y (mgr.34107) 10198 : cluster [DBG] pgmap v10911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[46934]: audit 2022-04-22T22:25:07.769272+0000 mon.a (mon.0) 14253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:25:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[46934]: audit 2022-04-22T22:25:07.772221+0000 mon.a (mon.0) 14254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:25:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:08 smithi115 bash[46934]: audit 2022-04-22T22:25:07.773755+0000 mon.a (mon.0) 14255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:25:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:09 smithi121 bash[36902]: audit 2022-04-22T22:25:08.709312+0000 mon.a (mon.0) 14256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:09 smithi115 bash[45495]: audit 2022-04-22T22:25:08.709312+0000 mon.a (mon.0) 14256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:09 smithi115 bash[46934]: audit 2022-04-22T22:25:08.709312+0000 mon.a (mon.0) 14256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:10 smithi121 bash[36902]: cluster 2022-04-22T22:25:09.370054+0000 mgr.y (mgr.34107) 10199 : cluster [DBG] pgmap v10912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:10 smithi115 bash[45495]: cluster 2022-04-22T22:25:09.370054+0000 mgr.y (mgr.34107) 10199 : cluster [DBG] pgmap v10912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:10 smithi115 bash[46934]: cluster 2022-04-22T22:25:09.370054+0000 mgr.y (mgr.34107) 10199 : cluster [DBG] pgmap v10912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:11 smithi121 bash[36902]: audit 2022-04-22T22:25:11.332067+0000 mon.a (mon.0) 14257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:11 smithi115 bash[45495]: audit 2022-04-22T22:25:11.332067+0000 mon.a (mon.0) 14257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:11 smithi115 bash[46934]: audit 2022-04-22T22:25:11.332067+0000 mon.a (mon.0) 14257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:12 smithi121 bash[36902]: cluster 2022-04-22T22:25:11.371140+0000 mgr.y (mgr.34107) 10200 : cluster [DBG] pgmap v10913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:12 smithi121 bash[36902]: audit 2022-04-22T22:25:12.214331+0000 mon.a (mon.0) 14258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:12 smithi121 bash[36902]: audit 2022-04-22T22:25:12.545591+0000 mon.a (mon.0) 14259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:12 smithi121 bash[36902]: audit 2022-04-22T22:25:12.606703+0000 mon.a (mon.0) 14260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:12.799Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[45495]: cluster 2022-04-22T22:25:11.371140+0000 mgr.y (mgr.34107) 10200 : cluster [DBG] pgmap v10913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[45495]: audit 2022-04-22T22:25:12.214331+0000 mon.a (mon.0) 14258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[45495]: audit 2022-04-22T22:25:12.545591+0000 mon.a (mon.0) 14259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[45495]: audit 2022-04-22T22:25:12.606703+0000 mon.a (mon.0) 14260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[46934]: cluster 2022-04-22T22:25:11.371140+0000 mgr.y (mgr.34107) 10200 : cluster [DBG] pgmap v10913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[46934]: audit 2022-04-22T22:25:12.214331+0000 mon.a (mon.0) 14258 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[46934]: audit 2022-04-22T22:25:12.545591+0000 mon.a (mon.0) 14259 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:25:13.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:12 smithi115 bash[46934]: audit 2022-04-22T22:25:12.606703+0000 mon.a (mon.0) 14260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:13 smithi121 bash[36902]: cluster 2022-04-22T22:25:13.372289+0000 mgr.y (mgr.34107) 10201 : cluster [DBG] pgmap v10914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:13 smithi115 bash[45495]: cluster 2022-04-22T22:25:13.372289+0000 mgr.y (mgr.34107) 10201 : cluster [DBG] pgmap v10914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:13 smithi115 bash[46934]: cluster 2022-04-22T22:25:13.372289+0000 mgr.y (mgr.34107) 10201 : cluster [DBG] pgmap v10914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:14 smithi121 bash[36902]: audit 2022-04-22T22:25:13.712069+0000 mon.a (mon.0) 14261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:14 smithi115 bash[45495]: audit 2022-04-22T22:25:13.712069+0000 mon.a (mon.0) 14261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:14.647Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:14 smithi115 bash[46934]: audit 2022-04-22T22:25:13.712069+0000 mon.a (mon.0) 14261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:15 smithi121 bash[36902]: cluster 2022-04-22T22:25:15.373337+0000 mgr.y (mgr.34107) 10202 : cluster [DBG] pgmap v10915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:15 smithi115 bash[45495]: cluster 2022-04-22T22:25:15.373337+0000 mgr.y (mgr.34107) 10202 : cluster [DBG] pgmap v10915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:15 smithi115 bash[46934]: cluster 2022-04-22T22:25:15.373337+0000 mgr.y (mgr.34107) 10202 : cluster [DBG] pgmap v10915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:16 smithi121 bash[36902]: audit 2022-04-22T22:25:16.334865+0000 mon.a (mon.0) 14262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:16 smithi115 bash[45495]: audit 2022-04-22T22:25:16.334865+0000 mon.a (mon.0) 14262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:16 smithi115 bash[46934]: audit 2022-04-22T22:25:16.334865+0000 mon.a (mon.0) 14262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:17 smithi121 bash[36902]: audit 2022-04-22T22:25:16.718164+0000 mon.a (mon.0) 14263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:25:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:17 smithi121 bash[36902]: audit 2022-04-22T22:25:16.718784+0000 mon.a (mon.0) 14264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:25:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:17 smithi121 bash[36902]: cluster 2022-04-22T22:25:17.374428+0000 mgr.y (mgr.34107) 10203 : cluster [DBG] pgmap v10916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:17 smithi121 bash[36902]: audit 2022-04-22T22:25:17.610559+0000 mon.a (mon.0) 14265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[45495]: audit 2022-04-22T22:25:16.718164+0000 mon.a (mon.0) 14263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:25:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[45495]: audit 2022-04-22T22:25:16.718784+0000 mon.a (mon.0) 14264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:25:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[45495]: cluster 2022-04-22T22:25:17.374428+0000 mgr.y (mgr.34107) 10203 : cluster [DBG] pgmap v10916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[45495]: audit 2022-04-22T22:25:17.610559+0000 mon.a (mon.0) 14265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[46934]: audit 2022-04-22T22:25:16.718164+0000 mon.a (mon.0) 14263 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:25:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[46934]: audit 2022-04-22T22:25:16.718784+0000 mon.a (mon.0) 14264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:25:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[46934]: cluster 2022-04-22T22:25:17.374428+0000 mgr.y (mgr.34107) 10203 : cluster [DBG] pgmap v10916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:17 smithi115 bash[46934]: audit 2022-04-22T22:25:17.610559+0000 mon.a (mon.0) 14265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:18 smithi115 bash[45495]: audit 2022-04-22T22:25:18.714930+0000 mon.a (mon.0) 14266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:18 smithi115 bash[46934]: audit 2022-04-22T22:25:18.714930+0000 mon.a (mon.0) 14266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:18 smithi121 bash[36902]: audit 2022-04-22T22:25:18.714930+0000 mon.a (mon.0) 14266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:19 smithi115 bash[45495]: cluster 2022-04-22T22:25:19.375492+0000 mgr.y (mgr.34107) 10204 : cluster [DBG] pgmap v10917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:19 smithi115 bash[46934]: cluster 2022-04-22T22:25:19.375492+0000 mgr.y (mgr.34107) 10204 : cluster [DBG] pgmap v10917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:19 smithi121 bash[36902]: cluster 2022-04-22T22:25:19.375492+0000 mgr.y (mgr.34107) 10204 : cluster [DBG] pgmap v10917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:21 smithi121 bash[36902]: audit 2022-04-22T22:25:21.338063+0000 mon.a (mon.0) 14267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:21 smithi115 bash[45495]: audit 2022-04-22T22:25:21.338063+0000 mon.a (mon.0) 14267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:21 smithi115 bash[46934]: audit 2022-04-22T22:25:21.338063+0000 mon.a (mon.0) 14267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:22.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:22 smithi121 bash[36902]: cluster 2022-04-22T22:25:21.376510+0000 mgr.y (mgr.34107) 10205 : cluster [DBG] pgmap v10918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:22.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:22 smithi115 bash[45495]: cluster 2022-04-22T22:25:21.376510+0000 mgr.y (mgr.34107) 10205 : cluster [DBG] pgmap v10918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:22 smithi115 bash[46934]: cluster 2022-04-22T22:25:21.376510+0000 mgr.y (mgr.34107) 10205 : cluster [DBG] pgmap v10918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:22.799Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:23 smithi121 bash[36902]: audit 2022-04-22T22:25:22.614242+0000 mon.a (mon.0) 14268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:23 smithi115 bash[45495]: audit 2022-04-22T22:25:22.614242+0000 mon.a (mon.0) 14268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:23 smithi115 bash[46934]: audit 2022-04-22T22:25:22.614242+0000 mon.a (mon.0) 14268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:24 smithi121 bash[36902]: cluster 2022-04-22T22:25:23.377759+0000 mgr.y (mgr.34107) 10206 : cluster [DBG] pgmap v10919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:24 smithi121 bash[36902]: audit 2022-04-22T22:25:23.718139+0000 mon.a (mon.0) 14269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:24.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:24.648Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:24 smithi115 bash[45495]: cluster 2022-04-22T22:25:23.377759+0000 mgr.y (mgr.34107) 10206 : cluster [DBG] pgmap v10919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:24 smithi115 bash[45495]: audit 2022-04-22T22:25:23.718139+0000 mon.a (mon.0) 14269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:24 smithi115 bash[46934]: cluster 2022-04-22T22:25:23.377759+0000 mgr.y (mgr.34107) 10206 : cluster [DBG] pgmap v10919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:24 smithi115 bash[46934]: audit 2022-04-22T22:25:23.718139+0000 mon.a (mon.0) 14269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:26 smithi121 bash[36902]: cluster 2022-04-22T22:25:25.378890+0000 mgr.y (mgr.34107) 10207 : cluster [DBG] pgmap v10920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:26 smithi121 bash[36902]: audit 2022-04-22T22:25:26.341404+0000 mon.a (mon.0) 14270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:26 smithi115 bash[45495]: cluster 2022-04-22T22:25:25.378890+0000 mgr.y (mgr.34107) 10207 : cluster [DBG] pgmap v10920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:26 smithi115 bash[45495]: audit 2022-04-22T22:25:26.341404+0000 mon.a (mon.0) 14270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:26 smithi115 bash[46934]: cluster 2022-04-22T22:25:25.378890+0000 mgr.y (mgr.34107) 10207 : cluster [DBG] pgmap v10920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:26 smithi115 bash[46934]: audit 2022-04-22T22:25:26.341404+0000 mon.a (mon.0) 14270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:27.667 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 15s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 9m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 15s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 15s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 9m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:25:28.013 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 9m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 15s ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 9m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 15s ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 9m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 9m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 9m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 9m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:25:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 15s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:25:28.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 15s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:25:28.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 15s ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:25:28.015 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 15s ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:25:28.015 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 15s ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:25:28.418 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:25:28.418 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:25:28.418 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:25:28.419 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:25:28.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:28.420 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:25:28.420 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:25:28.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:28.420 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:25:28.421 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:25:28.421 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:25:28.421 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:25:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:28 smithi121 bash[36902]: cluster 2022-04-22T22:25:27.380021+0000 mgr.y (mgr.34107) 10208 : cluster [DBG] pgmap v10921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:28 smithi121 bash[36902]: audit 2022-04-22T22:25:27.617763+0000 mon.a (mon.0) 14271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:28 smithi121 bash[36902]: audit 2022-04-22T22:25:28.414679+0000 mon.a (mon.0) 14272 : audit [DBG] from='client.? 172.21.15.115:0/3155465234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:25:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[45495]: cluster 2022-04-22T22:25:27.380021+0000 mgr.y (mgr.34107) 10208 : cluster [DBG] pgmap v10921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[45495]: audit 2022-04-22T22:25:27.617763+0000 mon.a (mon.0) 14271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[45495]: audit 2022-04-22T22:25:28.414679+0000 mon.a (mon.0) 14272 : audit [DBG] from='client.? 172.21.15.115:0/3155465234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:25:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[46934]: cluster 2022-04-22T22:25:27.380021+0000 mgr.y (mgr.34107) 10208 : cluster [DBG] pgmap v10921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[46934]: audit 2022-04-22T22:25:27.617763+0000 mon.a (mon.0) 14271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:28 smithi115 bash[46934]: audit 2022-04-22T22:25:28.414679+0000 mon.a (mon.0) 14272 : audit [DBG] from='client.? 172.21.15.115:0/3155465234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[45495]: audit 2022-04-22T22:25:27.647514+0000 mgr.y (mgr.34107) 10209 : audit [DBG] from='client.55867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[45495]: audit 2022-04-22T22:25:27.989495+0000 mgr.y (mgr.34107) 10210 : audit [DBG] from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[45495]: audit 2022-04-22T22:25:28.721318+0000 mon.a (mon.0) 14273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[46934]: audit 2022-04-22T22:25:27.647514+0000 mgr.y (mgr.34107) 10209 : audit [DBG] from='client.55867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[46934]: audit 2022-04-22T22:25:27.989495+0000 mgr.y (mgr.34107) 10210 : audit [DBG] from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:29 smithi115 bash[46934]: audit 2022-04-22T22:25:28.721318+0000 mon.a (mon.0) 14273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:29 smithi121 bash[36902]: audit 2022-04-22T22:25:27.647514+0000 mgr.y (mgr.34107) 10209 : audit [DBG] from='client.55867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:29 smithi121 bash[36902]: audit 2022-04-22T22:25:27.989495+0000 mgr.y (mgr.34107) 10210 : audit [DBG] from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:25:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:29 smithi121 bash[36902]: audit 2022-04-22T22:25:28.721318+0000 mon.a (mon.0) 14273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:30 smithi115 bash[45495]: cluster 2022-04-22T22:25:29.381113+0000 mgr.y (mgr.34107) 10211 : cluster [DBG] pgmap v10922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:30 smithi115 bash[46934]: cluster 2022-04-22T22:25:29.381113+0000 mgr.y (mgr.34107) 10211 : cluster [DBG] pgmap v10922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:30 smithi121 bash[36902]: cluster 2022-04-22T22:25:29.381113+0000 mgr.y (mgr.34107) 10211 : cluster [DBG] pgmap v10922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:31 smithi115 bash[45495]: audit 2022-04-22T22:25:31.344439+0000 mon.a (mon.0) 14274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:31 smithi115 bash[46934]: audit 2022-04-22T22:25:31.344439+0000 mon.a (mon.0) 14274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:31 smithi121 bash[36902]: audit 2022-04-22T22:25:31.344439+0000 mon.a (mon.0) 14274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:32.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:32 smithi115 bash[45495]: cluster 2022-04-22T22:25:31.382147+0000 mgr.y (mgr.34107) 10212 : cluster [DBG] pgmap v10923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:32 smithi115 bash[46934]: cluster 2022-04-22T22:25:31.382147+0000 mgr.y (mgr.34107) 10212 : cluster [DBG] pgmap v10923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:32 smithi121 bash[36902]: cluster 2022-04-22T22:25:31.382147+0000 mgr.y (mgr.34107) 10212 : cluster [DBG] pgmap v10923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:32.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:33 smithi115 bash[46934]: audit 2022-04-22T22:25:32.621608+0000 mon.a (mon.0) 14275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:33 smithi115 bash[45495]: audit 2022-04-22T22:25:32.621608+0000 mon.a (mon.0) 14275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:33 smithi121 bash[36902]: audit 2022-04-22T22:25:32.621608+0000 mon.a (mon.0) 14275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:34.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:34.649Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:34 smithi115 bash[46934]: cluster 2022-04-22T22:25:33.383258+0000 mgr.y (mgr.34107) 10213 : cluster [DBG] pgmap v10924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:34 smithi115 bash[46934]: audit 2022-04-22T22:25:33.724314+0000 mon.a (mon.0) 14276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:34 smithi115 bash[45495]: cluster 2022-04-22T22:25:33.383258+0000 mgr.y (mgr.34107) 10213 : cluster [DBG] pgmap v10924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:34 smithi115 bash[45495]: audit 2022-04-22T22:25:33.724314+0000 mon.a (mon.0) 14276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:34 smithi121 bash[36902]: cluster 2022-04-22T22:25:33.383258+0000 mgr.y (mgr.34107) 10213 : cluster [DBG] pgmap v10924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:34 smithi121 bash[36902]: audit 2022-04-22T22:25:33.724314+0000 mon.a (mon.0) 14276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:36 smithi121 bash[36902]: cluster 2022-04-22T22:25:35.384322+0000 mgr.y (mgr.34107) 10214 : cluster [DBG] pgmap v10925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:36 smithi121 bash[36902]: audit 2022-04-22T22:25:36.347304+0000 mon.a (mon.0) 14277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:37.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:36 smithi115 bash[46934]: cluster 2022-04-22T22:25:35.384322+0000 mgr.y (mgr.34107) 10214 : cluster [DBG] pgmap v10925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:36 smithi115 bash[46934]: audit 2022-04-22T22:25:36.347304+0000 mon.a (mon.0) 14277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:36 smithi115 bash[45495]: cluster 2022-04-22T22:25:35.384322+0000 mgr.y (mgr.34107) 10214 : cluster [DBG] pgmap v10925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:36 smithi115 bash[45495]: audit 2022-04-22T22:25:36.347304+0000 mon.a (mon.0) 14277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:38 smithi121 bash[36902]: cluster 2022-04-22T22:25:37.385431+0000 mgr.y (mgr.34107) 10215 : cluster [DBG] pgmap v10926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:38 smithi121 bash[36902]: audit 2022-04-22T22:25:37.625292+0000 mon.a (mon.0) 14278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:38 smithi115 bash[45495]: cluster 2022-04-22T22:25:37.385431+0000 mgr.y (mgr.34107) 10215 : cluster [DBG] pgmap v10926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:38 smithi115 bash[45495]: audit 2022-04-22T22:25:37.625292+0000 mon.a (mon.0) 14278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:38 smithi115 bash[46934]: cluster 2022-04-22T22:25:37.385431+0000 mgr.y (mgr.34107) 10215 : cluster [DBG] pgmap v10926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:38 smithi115 bash[46934]: audit 2022-04-22T22:25:37.625292+0000 mon.a (mon.0) 14278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:39 smithi121 bash[36902]: audit 2022-04-22T22:25:38.727399+0000 mon.a (mon.0) 14279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:39 smithi115 bash[45495]: audit 2022-04-22T22:25:38.727399+0000 mon.a (mon.0) 14279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:39 smithi115 bash[46934]: audit 2022-04-22T22:25:38.727399+0000 mon.a (mon.0) 14279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:40 smithi121 bash[36902]: cluster 2022-04-22T22:25:39.386575+0000 mgr.y (mgr.34107) 10216 : cluster [DBG] pgmap v10927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:40 smithi115 bash[45495]: cluster 2022-04-22T22:25:39.386575+0000 mgr.y (mgr.34107) 10216 : cluster [DBG] pgmap v10927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:40 smithi115 bash[46934]: cluster 2022-04-22T22:25:39.386575+0000 mgr.y (mgr.34107) 10216 : cluster [DBG] pgmap v10927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:41 smithi121 bash[36902]: audit 2022-04-22T22:25:41.350367+0000 mon.a (mon.0) 14280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:41 smithi115 bash[45495]: audit 2022-04-22T22:25:41.350367+0000 mon.a (mon.0) 14280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:41 smithi115 bash[46934]: audit 2022-04-22T22:25:41.350367+0000 mon.a (mon.0) 14280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:42 smithi121 bash[36902]: cluster 2022-04-22T22:25:41.387790+0000 mgr.y (mgr.34107) 10217 : cluster [DBG] pgmap v10928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:42.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:42 smithi115 bash[45495]: cluster 2022-04-22T22:25:41.387790+0000 mgr.y (mgr.34107) 10217 : cluster [DBG] pgmap v10928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:42 smithi115 bash[46934]: cluster 2022-04-22T22:25:41.387790+0000 mgr.y (mgr.34107) 10217 : cluster [DBG] pgmap v10928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:43 smithi121 bash[36902]: audit 2022-04-22T22:25:42.628720+0000 mon.a (mon.0) 14281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:43 smithi115 bash[46934]: audit 2022-04-22T22:25:42.628720+0000 mon.a (mon.0) 14281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:43 smithi115 bash[45495]: audit 2022-04-22T22:25:42.628720+0000 mon.a (mon.0) 14281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:44 smithi121 bash[36902]: cluster 2022-04-22T22:25:43.389043+0000 mgr.y (mgr.34107) 10218 : cluster [DBG] pgmap v10929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:44 smithi121 bash[36902]: audit 2022-04-22T22:25:43.730610+0000 mon.a (mon.0) 14282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:44 smithi115 bash[45495]: cluster 2022-04-22T22:25:43.389043+0000 mgr.y (mgr.34107) 10218 : cluster [DBG] pgmap v10929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:44 smithi115 bash[45495]: audit 2022-04-22T22:25:43.730610+0000 mon.a (mon.0) 14282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:44.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:44 smithi115 bash[46934]: cluster 2022-04-22T22:25:43.389043+0000 mgr.y (mgr.34107) 10218 : cluster [DBG] pgmap v10929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:44 smithi115 bash[46934]: audit 2022-04-22T22:25:43.730610+0000 mon.a (mon.0) 14282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:46 smithi121 bash[36902]: cluster 2022-04-22T22:25:45.390154+0000 mgr.y (mgr.34107) 10219 : cluster [DBG] pgmap v10930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:46 smithi121 bash[36902]: audit 2022-04-22T22:25:46.353763+0000 mon.a (mon.0) 14283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:46 smithi115 bash[45495]: cluster 2022-04-22T22:25:45.390154+0000 mgr.y (mgr.34107) 10219 : cluster [DBG] pgmap v10930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:46 smithi115 bash[45495]: audit 2022-04-22T22:25:46.353763+0000 mon.a (mon.0) 14283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:46 smithi115 bash[46934]: cluster 2022-04-22T22:25:45.390154+0000 mgr.y (mgr.34107) 10219 : cluster [DBG] pgmap v10930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:46 smithi115 bash[46934]: audit 2022-04-22T22:25:46.353763+0000 mon.a (mon.0) 14283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:48 smithi121 bash[36902]: cluster 2022-04-22T22:25:47.391230+0000 mgr.y (mgr.34107) 10220 : cluster [DBG] pgmap v10931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:48 smithi121 bash[36902]: audit 2022-04-22T22:25:47.631819+0000 mon.a (mon.0) 14284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:48 smithi115 bash[45495]: cluster 2022-04-22T22:25:47.391230+0000 mgr.y (mgr.34107) 10220 : cluster [DBG] pgmap v10931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:48 smithi115 bash[45495]: audit 2022-04-22T22:25:47.631819+0000 mon.a (mon.0) 14284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:48 smithi115 bash[46934]: cluster 2022-04-22T22:25:47.391230+0000 mgr.y (mgr.34107) 10220 : cluster [DBG] pgmap v10931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:48 smithi115 bash[46934]: audit 2022-04-22T22:25:47.631819+0000 mon.a (mon.0) 14284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:49 smithi121 bash[36902]: audit 2022-04-22T22:25:48.733706+0000 mon.a (mon.0) 14285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:49 smithi115 bash[46934]: audit 2022-04-22T22:25:48.733706+0000 mon.a (mon.0) 14285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:49 smithi115 bash[45495]: audit 2022-04-22T22:25:48.733706+0000 mon.a (mon.0) 14285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:50 smithi121 bash[36902]: cluster 2022-04-22T22:25:49.392268+0000 mgr.y (mgr.34107) 10221 : cluster [DBG] pgmap v10932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:50 smithi115 bash[45495]: cluster 2022-04-22T22:25:49.392268+0000 mgr.y (mgr.34107) 10221 : cluster [DBG] pgmap v10932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:50 smithi115 bash[46934]: cluster 2022-04-22T22:25:49.392268+0000 mgr.y (mgr.34107) 10221 : cluster [DBG] pgmap v10932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:51 smithi121 bash[36902]: audit 2022-04-22T22:25:51.356785+0000 mon.a (mon.0) 14286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:51 smithi115 bash[45495]: audit 2022-04-22T22:25:51.356785+0000 mon.a (mon.0) 14286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:51 smithi115 bash[46934]: audit 2022-04-22T22:25:51.356785+0000 mon.a (mon.0) 14286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:52 smithi121 bash[36902]: cluster 2022-04-22T22:25:51.393367+0000 mgr.y (mgr.34107) 10222 : cluster [DBG] pgmap v10933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:53.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:52 smithi115 bash[45495]: cluster 2022-04-22T22:25:51.393367+0000 mgr.y (mgr.34107) 10222 : cluster [DBG] pgmap v10933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:52.801Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:52 smithi115 bash[46934]: cluster 2022-04-22T22:25:51.393367+0000 mgr.y (mgr.34107) 10222 : cluster [DBG] pgmap v10933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:53 smithi121 bash[36902]: audit 2022-04-22T22:25:52.635109+0000 mon.a (mon.0) 14287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:53 smithi115 bash[45495]: audit 2022-04-22T22:25:52.635109+0000 mon.a (mon.0) 14287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:53 smithi115 bash[46934]: audit 2022-04-22T22:25:52.635109+0000 mon.a (mon.0) 14287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:54 smithi121 bash[36902]: cluster 2022-04-22T22:25:53.394626+0000 mgr.y (mgr.34107) 10223 : cluster [DBG] pgmap v10934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:54 smithi121 bash[36902]: audit 2022-04-22T22:25:53.736892+0000 mon.a (mon.0) 14288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:54 smithi115 bash[45495]: cluster 2022-04-22T22:25:53.394626+0000 mgr.y (mgr.34107) 10223 : cluster [DBG] pgmap v10934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:54 smithi115 bash[45495]: audit 2022-04-22T22:25:53.736892+0000 mon.a (mon.0) 14288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:25:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:25:54.647Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:25:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:54 smithi115 bash[46934]: cluster 2022-04-22T22:25:53.394626+0000 mgr.y (mgr.34107) 10223 : cluster [DBG] pgmap v10934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:54 smithi115 bash[46934]: audit 2022-04-22T22:25:53.736892+0000 mon.a (mon.0) 14288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:25:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:56 smithi121 bash[36902]: cluster 2022-04-22T22:25:55.395729+0000 mgr.y (mgr.34107) 10224 : cluster [DBG] pgmap v10935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:56 smithi121 bash[36902]: audit 2022-04-22T22:25:56.359552+0000 mon.a (mon.0) 14289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:56 smithi115 bash[45495]: cluster 2022-04-22T22:25:55.395729+0000 mgr.y (mgr.34107) 10224 : cluster [DBG] pgmap v10935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:56 smithi115 bash[45495]: audit 2022-04-22T22:25:56.359552+0000 mon.a (mon.0) 14289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:56 smithi115 bash[46934]: cluster 2022-04-22T22:25:55.395729+0000 mgr.y (mgr.34107) 10224 : cluster [DBG] pgmap v10935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:56 smithi115 bash[46934]: audit 2022-04-22T22:25:56.359552+0000 mon.a (mon.0) 14289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:25:58.795 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:25:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:58 smithi121 bash[36902]: cluster 2022-04-22T22:25:57.396807+0000 mgr.y (mgr.34107) 10225 : cluster [DBG] pgmap v10936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:58 smithi121 bash[36902]: audit 2022-04-22T22:25:57.638756+0000 mon.a (mon.0) 14290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:59.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:58 smithi115 bash[46934]: cluster 2022-04-22T22:25:57.396807+0000 mgr.y (mgr.34107) 10225 : cluster [DBG] pgmap v10936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:58 smithi115 bash[46934]: audit 2022-04-22T22:25:57.638756+0000 mon.a (mon.0) 14290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:58 smithi115 bash[45495]: cluster 2022-04-22T22:25:57.396807+0000 mgr.y (mgr.34107) 10225 : cluster [DBG] pgmap v10936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:25:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:58 smithi115 bash[45495]: audit 2022-04-22T22:25:57.638756+0000 mon.a (mon.0) 14290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 10m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 46s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 10m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 46s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 46s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:25:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 10m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 10m ago 5h 422M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 46s ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 10m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 10m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 46s ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 10m ago 5h 716M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 10m ago 5h 796M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 10m ago 5h 539M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 10m ago 5h 500M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:25:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 46s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:25:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 46s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:25:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 46s ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:25:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 46s ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:25:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 46s ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:25:59.537 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:25:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:25:59.539 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:26:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:59 smithi121 bash[36902]: audit 2022-04-22T22:25:58.739845+0000 mon.a (mon.0) 14291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:25:59 smithi121 bash[36902]: audit 2022-04-22T22:25:59.534275+0000 mon.a (mon.0) 14292 : audit [DBG] from='client.? 172.21.15.115:0/3236580789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:59 smithi115 bash[45495]: audit 2022-04-22T22:25:58.739845+0000 mon.a (mon.0) 14291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:25:59 smithi115 bash[45495]: audit 2022-04-22T22:25:59.534275+0000 mon.a (mon.0) 14292 : audit [DBG] from='client.? 172.21.15.115:0/3236580789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:59 smithi115 bash[46934]: audit 2022-04-22T22:25:58.739845+0000 mon.a (mon.0) 14291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:25:59 smithi115 bash[46934]: audit 2022-04-22T22:25:59.534275+0000 mon.a (mon.0) 14292 : audit [DBG] from='client.? 172.21.15.115:0/3236580789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:00 smithi121 bash[36902]: audit 2022-04-22T22:25:58.776684+0000 mgr.y (mgr.34107) 10226 : audit [DBG] from='client.55885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:00 smithi121 bash[36902]: audit 2022-04-22T22:25:59.115360+0000 mgr.y (mgr.34107) 10227 : audit [DBG] from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:00 smithi121 bash[36902]: cluster 2022-04-22T22:25:59.397854+0000 mgr.y (mgr.34107) 10228 : cluster [DBG] pgmap v10937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[45495]: audit 2022-04-22T22:25:58.776684+0000 mgr.y (mgr.34107) 10226 : audit [DBG] from='client.55885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[45495]: audit 2022-04-22T22:25:59.115360+0000 mgr.y (mgr.34107) 10227 : audit [DBG] from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[45495]: cluster 2022-04-22T22:25:59.397854+0000 mgr.y (mgr.34107) 10228 : cluster [DBG] pgmap v10937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[46934]: audit 2022-04-22T22:25:58.776684+0000 mgr.y (mgr.34107) 10226 : audit [DBG] from='client.55885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[46934]: audit 2022-04-22T22:25:59.115360+0000 mgr.y (mgr.34107) 10227 : audit [DBG] from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:00 smithi115 bash[46934]: cluster 2022-04-22T22:25:59.397854+0000 mgr.y (mgr.34107) 10228 : cluster [DBG] pgmap v10937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:01 smithi121 bash[36902]: audit 2022-04-22T22:26:01.362765+0000 mon.a (mon.0) 14293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:01 smithi115 bash[45495]: audit 2022-04-22T22:26:01.362765+0000 mon.a (mon.0) 14293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:01 smithi115 bash[46934]: audit 2022-04-22T22:26:01.362765+0000 mon.a (mon.0) 14293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:02 smithi121 bash[36902]: cluster 2022-04-22T22:26:01.398858+0000 mgr.y (mgr.34107) 10229 : cluster [DBG] pgmap v10938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:02 smithi121 bash[36902]: audit 2022-04-22T22:26:02.642640+0000 mon.a (mon.0) 14294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:03.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:02 smithi115 bash[45495]: cluster 2022-04-22T22:26:01.398858+0000 mgr.y (mgr.34107) 10229 : cluster [DBG] pgmap v10938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:02 smithi115 bash[45495]: audit 2022-04-22T22:26:02.642640+0000 mon.a (mon.0) 14294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:02.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:02 smithi115 bash[46934]: cluster 2022-04-22T22:26:01.398858+0000 mgr.y (mgr.34107) 10229 : cluster [DBG] pgmap v10938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:02 smithi115 bash[46934]: audit 2022-04-22T22:26:02.642640+0000 mon.a (mon.0) 14294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:03 smithi121 bash[36902]: cluster 2022-04-22T22:26:03.399912+0000 mgr.y (mgr.34107) 10230 : cluster [DBG] pgmap v10939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:03 smithi115 bash[45495]: cluster 2022-04-22T22:26:03.399912+0000 mgr.y (mgr.34107) 10230 : cluster [DBG] pgmap v10939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:03 smithi115 bash[46934]: cluster 2022-04-22T22:26:03.399912+0000 mgr.y (mgr.34107) 10230 : cluster [DBG] pgmap v10939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:04 smithi121 bash[36902]: audit 2022-04-22T22:26:03.742818+0000 mon.a (mon.0) 14295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:05.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:04 smithi115 bash[46934]: audit 2022-04-22T22:26:03.742818+0000 mon.a (mon.0) 14295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:04.650Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:04 smithi115 bash[45495]: audit 2022-04-22T22:26:03.742818+0000 mon.a (mon.0) 14295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:05 smithi121 bash[36902]: cluster 2022-04-22T22:26:05.400896+0000 mgr.y (mgr.34107) 10231 : cluster [DBG] pgmap v10940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:05 smithi115 bash[45495]: cluster 2022-04-22T22:26:05.400896+0000 mgr.y (mgr.34107) 10231 : cluster [DBG] pgmap v10940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:05 smithi115 bash[46934]: cluster 2022-04-22T22:26:05.400896+0000 mgr.y (mgr.34107) 10231 : cluster [DBG] pgmap v10940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:06 smithi121 bash[36902]: audit 2022-04-22T22:26:06.365938+0000 mon.a (mon.0) 14296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:06 smithi115 bash[45495]: audit 2022-04-22T22:26:06.365938+0000 mon.a (mon.0) 14296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:06 smithi115 bash[46934]: audit 2022-04-22T22:26:06.365938+0000 mon.a (mon.0) 14296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:07 smithi121 bash[36902]: cluster 2022-04-22T22:26:07.401943+0000 mgr.y (mgr.34107) 10232 : cluster [DBG] pgmap v10941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:07 smithi121 bash[36902]: audit 2022-04-22T22:26:07.646258+0000 mon.a (mon.0) 14297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:07 smithi115 bash[45495]: cluster 2022-04-22T22:26:07.401943+0000 mgr.y (mgr.34107) 10232 : cluster [DBG] pgmap v10941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:07 smithi115 bash[45495]: audit 2022-04-22T22:26:07.646258+0000 mon.a (mon.0) 14297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:07 smithi115 bash[46934]: cluster 2022-04-22T22:26:07.401943+0000 mgr.y (mgr.34107) 10232 : cluster [DBG] pgmap v10941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:07 smithi115 bash[46934]: audit 2022-04-22T22:26:07.646258+0000 mon.a (mon.0) 14297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:08 smithi121 bash[36902]: audit 2022-04-22T22:26:08.745881+0000 mon.a (mon.0) 14298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:08 smithi115 bash[45495]: audit 2022-04-22T22:26:08.745881+0000 mon.a (mon.0) 14298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:08 smithi115 bash[46934]: audit 2022-04-22T22:26:08.745881+0000 mon.a (mon.0) 14298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:09 smithi121 bash[36902]: cluster 2022-04-22T22:26:09.403034+0000 mgr.y (mgr.34107) 10233 : cluster [DBG] pgmap v10942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:09 smithi115 bash[45495]: cluster 2022-04-22T22:26:09.403034+0000 mgr.y (mgr.34107) 10233 : cluster [DBG] pgmap v10942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:09 smithi115 bash[46934]: cluster 2022-04-22T22:26:09.403034+0000 mgr.y (mgr.34107) 10233 : cluster [DBG] pgmap v10942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:11 smithi121 bash[36902]: audit 2022-04-22T22:26:11.369031+0000 mon.a (mon.0) 14299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:11 smithi115 bash[45495]: audit 2022-04-22T22:26:11.369031+0000 mon.a (mon.0) 14299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:11 smithi115 bash[46934]: audit 2022-04-22T22:26:11.369031+0000 mon.a (mon.0) 14299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:12.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:12 smithi121 bash[36902]: cluster 2022-04-22T22:26:11.404148+0000 mgr.y (mgr.34107) 10234 : cluster [DBG] pgmap v10943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:12.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:12 smithi115 bash[45495]: cluster 2022-04-22T22:26:11.404148+0000 mgr.y (mgr.34107) 10234 : cluster [DBG] pgmap v10943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:12 smithi115 bash[46934]: cluster 2022-04-22T22:26:11.404148+0000 mgr.y (mgr.34107) 10234 : cluster [DBG] pgmap v10943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:12.801Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.555281+0000 mon.a (mon.0) 14300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:26:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.558316+0000 mon.a (mon.0) 14301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:26:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.559797+0000 mon.a (mon.0) 14302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:26:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.649927+0000 mon.a (mon.0) 14303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.906340+0000 mon.a (mon.0) 14304 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:13 smithi121 bash[36902]: audit 2022-04-22T22:26:12.914460+0000 mon.a (mon.0) 14305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.555281+0000 mon.a (mon.0) 14300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:26:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.558316+0000 mon.a (mon.0) 14301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:26:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.559797+0000 mon.a (mon.0) 14302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:26:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.649927+0000 mon.a (mon.0) 14303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.906340+0000 mon.a (mon.0) 14304 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[45495]: audit 2022-04-22T22:26:12.914460+0000 mon.a (mon.0) 14305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.555281+0000 mon.a (mon.0) 14300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.558316+0000 mon.a (mon.0) 14301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.559797+0000 mon.a (mon.0) 14302 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.649927+0000 mon.a (mon.0) 14303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:13.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.906340+0000 mon.a (mon.0) 14304 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:13.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:13 smithi115 bash[46934]: audit 2022-04-22T22:26:12.914460+0000 mon.a (mon.0) 14305 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:14 smithi121 bash[36902]: cluster 2022-04-22T22:26:13.405402+0000 mgr.y (mgr.34107) 10235 : cluster [DBG] pgmap v10944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:14 smithi121 bash[36902]: audit 2022-04-22T22:26:13.748764+0000 mon.a (mon.0) 14306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:14.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:14.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:14 smithi115 bash[46934]: cluster 2022-04-22T22:26:13.405402+0000 mgr.y (mgr.34107) 10235 : cluster [DBG] pgmap v10944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:14 smithi115 bash[46934]: audit 2022-04-22T22:26:13.748764+0000 mon.a (mon.0) 14306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:14 smithi115 bash[45495]: cluster 2022-04-22T22:26:13.405402+0000 mgr.y (mgr.34107) 10235 : cluster [DBG] pgmap v10944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:14.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:14 smithi115 bash[45495]: audit 2022-04-22T22:26:13.748764+0000 mon.a (mon.0) 14306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:16 smithi115 bash[45495]: cluster 2022-04-22T22:26:15.406497+0000 mgr.y (mgr.34107) 10236 : cluster [DBG] pgmap v10945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:16 smithi115 bash[45495]: audit 2022-04-22T22:26:16.372344+0000 mon.a (mon.0) 14307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:16 smithi115 bash[46934]: cluster 2022-04-22T22:26:15.406497+0000 mgr.y (mgr.34107) 10236 : cluster [DBG] pgmap v10945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:16 smithi115 bash[46934]: audit 2022-04-22T22:26:16.372344+0000 mon.a (mon.0) 14307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:16 smithi121 bash[36902]: cluster 2022-04-22T22:26:15.406497+0000 mgr.y (mgr.34107) 10236 : cluster [DBG] pgmap v10945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:16 smithi121 bash[36902]: audit 2022-04-22T22:26:16.372344+0000 mon.a (mon.0) 14307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[45495]: audit 2022-04-22T22:26:16.726703+0000 mon.a (mon.0) 14308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[45495]: audit 2022-04-22T22:26:16.727183+0000 mon.a (mon.0) 14309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[45495]: audit 2022-04-22T22:26:17.064112+0000 mon.a (mon.0) 14310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[45495]: audit 2022-04-22T22:26:17.414824+0000 mon.a (mon.0) 14311 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[46934]: audit 2022-04-22T22:26:16.726703+0000 mon.a (mon.0) 14308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:26:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[46934]: audit 2022-04-22T22:26:16.727183+0000 mon.a (mon.0) 14309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:26:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[46934]: audit 2022-04-22T22:26:17.064112+0000 mon.a (mon.0) 14310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:17 smithi115 bash[46934]: audit 2022-04-22T22:26:17.414824+0000 mon.a (mon.0) 14311 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:17 smithi121 bash[36902]: audit 2022-04-22T22:26:16.726703+0000 mon.a (mon.0) 14308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:26:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:17 smithi121 bash[36902]: audit 2022-04-22T22:26:16.727183+0000 mon.a (mon.0) 14309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:26:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:17 smithi121 bash[36902]: audit 2022-04-22T22:26:17.064112+0000 mon.a (mon.0) 14310 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:17 smithi121 bash[36902]: audit 2022-04-22T22:26:17.414824+0000 mon.a (mon.0) 14311 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:26:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:18 smithi115 bash[45495]: cluster 2022-04-22T22:26:17.407245+0000 mgr.y (mgr.34107) 10237 : cluster [DBG] pgmap v10946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:18 smithi115 bash[45495]: audit 2022-04-22T22:26:17.653458+0000 mon.a (mon.0) 14312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:18 smithi115 bash[46934]: cluster 2022-04-22T22:26:17.407245+0000 mgr.y (mgr.34107) 10237 : cluster [DBG] pgmap v10946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:18 smithi115 bash[46934]: audit 2022-04-22T22:26:17.653458+0000 mon.a (mon.0) 14312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:18 smithi121 bash[36902]: cluster 2022-04-22T22:26:17.407245+0000 mgr.y (mgr.34107) 10237 : cluster [DBG] pgmap v10946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:18 smithi121 bash[36902]: audit 2022-04-22T22:26:17.653458+0000 mon.a (mon.0) 14312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:19 smithi115 bash[45495]: audit 2022-04-22T22:26:18.751911+0000 mon.a (mon.0) 14313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:19 smithi115 bash[46934]: audit 2022-04-22T22:26:18.751911+0000 mon.a (mon.0) 14313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:19 smithi121 bash[36902]: audit 2022-04-22T22:26:18.751911+0000 mon.a (mon.0) 14313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:20 smithi115 bash[45495]: cluster 2022-04-22T22:26:19.408277+0000 mgr.y (mgr.34107) 10238 : cluster [DBG] pgmap v10947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:20 smithi115 bash[46934]: cluster 2022-04-22T22:26:19.408277+0000 mgr.y (mgr.34107) 10238 : cluster [DBG] pgmap v10947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:20 smithi121 bash[36902]: cluster 2022-04-22T22:26:19.408277+0000 mgr.y (mgr.34107) 10238 : cluster [DBG] pgmap v10947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:21 smithi115 bash[45495]: audit 2022-04-22T22:26:21.374948+0000 mon.a (mon.0) 14314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:21 smithi115 bash[46934]: audit 2022-04-22T22:26:21.374948+0000 mon.a (mon.0) 14314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:21 smithi121 bash[36902]: audit 2022-04-22T22:26:21.374948+0000 mon.a (mon.0) 14314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:22 smithi115 bash[45495]: cluster 2022-04-22T22:26:21.409295+0000 mgr.y (mgr.34107) 10239 : cluster [DBG] pgmap v10948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:22 smithi115 bash[46934]: cluster 2022-04-22T22:26:21.409295+0000 mgr.y (mgr.34107) 10239 : cluster [DBG] pgmap v10948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:22 smithi121 bash[36902]: cluster 2022-04-22T22:26:21.409295+0000 mgr.y (mgr.34107) 10239 : cluster [DBG] pgmap v10948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:22.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:23 smithi121 bash[36902]: audit 2022-04-22T22:26:22.657425+0000 mon.a (mon.0) 14315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:23 smithi115 bash[45495]: audit 2022-04-22T22:26:22.657425+0000 mon.a (mon.0) 14315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:23 smithi115 bash[46934]: audit 2022-04-22T22:26:22.657425+0000 mon.a (mon.0) 14315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:24 smithi121 bash[36902]: cluster 2022-04-22T22:26:23.410377+0000 mgr.y (mgr.34107) 10240 : cluster [DBG] pgmap v10949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:24 smithi121 bash[36902]: audit 2022-04-22T22:26:23.754898+0000 mon.a (mon.0) 14316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:24.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:24 smithi115 bash[45495]: cluster 2022-04-22T22:26:23.410377+0000 mgr.y (mgr.34107) 10240 : cluster [DBG] pgmap v10949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:24 smithi115 bash[45495]: audit 2022-04-22T22:26:23.754898+0000 mon.a (mon.0) 14316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:24 smithi115 bash[46934]: cluster 2022-04-22T22:26:23.410377+0000 mgr.y (mgr.34107) 10240 : cluster [DBG] pgmap v10949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:24 smithi115 bash[46934]: audit 2022-04-22T22:26:23.754898+0000 mon.a (mon.0) 14316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:26 smithi121 bash[36902]: cluster 2022-04-22T22:26:25.411426+0000 mgr.y (mgr.34107) 10241 : cluster [DBG] pgmap v10950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:26 smithi121 bash[36902]: audit 2022-04-22T22:26:26.377974+0000 mon.a (mon.0) 14317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:26 smithi115 bash[45495]: cluster 2022-04-22T22:26:25.411426+0000 mgr.y (mgr.34107) 10241 : cluster [DBG] pgmap v10950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:26 smithi115 bash[45495]: audit 2022-04-22T22:26:26.377974+0000 mon.a (mon.0) 14317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:26 smithi115 bash[46934]: cluster 2022-04-22T22:26:25.411426+0000 mgr.y (mgr.34107) 10241 : cluster [DBG] pgmap v10950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:26 smithi115 bash[46934]: audit 2022-04-22T22:26:26.377974+0000 mon.a (mon.0) 14317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:28 smithi121 bash[36902]: cluster 2022-04-22T22:26:27.412468+0000 mgr.y (mgr.34107) 10242 : cluster [DBG] pgmap v10951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:28 smithi121 bash[36902]: audit 2022-04-22T22:26:27.661056+0000 mon.a (mon.0) 14318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:28 smithi115 bash[45495]: cluster 2022-04-22T22:26:27.412468+0000 mgr.y (mgr.34107) 10242 : cluster [DBG] pgmap v10951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:28 smithi115 bash[45495]: audit 2022-04-22T22:26:27.661056+0000 mon.a (mon.0) 14318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:28 smithi115 bash[46934]: cluster 2022-04-22T22:26:27.412468+0000 mgr.y (mgr.34107) 10242 : cluster [DBG] pgmap v10951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:28 smithi115 bash[46934]: audit 2022-04-22T22:26:27.661056+0000 mon.a (mon.0) 14318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:29.915 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:26:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:29 smithi121 bash[36902]: audit 2022-04-22T22:26:28.758092+0000 mon.a (mon.0) 14319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:30.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:29 smithi115 bash[46934]: audit 2022-04-22T22:26:28.758092+0000 mon.a (mon.0) 14319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:29 smithi115 bash[45495]: audit 2022-04-22T22:26:28.758092+0000 mon.a (mon.0) 14319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 13s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 78s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 13s ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 78s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 78s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:26:30.256 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 13s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 13s ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 78s ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 13s ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 13s ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 78s ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 13s ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 13s ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:26:30.257 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 13s ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 13s ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 78s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 78s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 78s ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 78s ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:26:30.258 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 78s ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:26:30.663 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:26:30.663 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:26:30.663 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:26:30.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:26:30.665 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:26:30.665 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:26:30.665 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:26:30.665 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:26:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:30 smithi121 bash[36902]: cluster 2022-04-22T22:26:29.413525+0000 mgr.y (mgr.34107) 10243 : cluster [DBG] pgmap v10952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:30 smithi115 bash[45495]: cluster 2022-04-22T22:26:29.413525+0000 mgr.y (mgr.34107) 10243 : cluster [DBG] pgmap v10952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:30 smithi115 bash[46934]: cluster 2022-04-22T22:26:29.413525+0000 mgr.y (mgr.34107) 10243 : cluster [DBG] pgmap v10952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:31 smithi121 bash[36902]: audit 2022-04-22T22:26:29.896855+0000 mgr.y (mgr.34107) 10244 : audit [DBG] from='client.55903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:31 smithi121 bash[36902]: audit 2022-04-22T22:26:30.236086+0000 mgr.y (mgr.34107) 10245 : audit [DBG] from='client.55909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:31 smithi121 bash[36902]: audit 2022-04-22T22:26:30.659593+0000 mon.a (mon.0) 14320 : audit [DBG] from='client.? 172.21.15.115:0/4141397451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:31 smithi121 bash[36902]: audit 2022-04-22T22:26:31.381216+0000 mon.a (mon.0) 14321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[45495]: audit 2022-04-22T22:26:29.896855+0000 mgr.y (mgr.34107) 10244 : audit [DBG] from='client.55903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[45495]: audit 2022-04-22T22:26:30.236086+0000 mgr.y (mgr.34107) 10245 : audit [DBG] from='client.55909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[45495]: audit 2022-04-22T22:26:30.659593+0000 mon.a (mon.0) 14320 : audit [DBG] from='client.? 172.21.15.115:0/4141397451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[45495]: audit 2022-04-22T22:26:31.381216+0000 mon.a (mon.0) 14321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[46934]: audit 2022-04-22T22:26:29.896855+0000 mgr.y (mgr.34107) 10244 : audit [DBG] from='client.55903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[46934]: audit 2022-04-22T22:26:30.236086+0000 mgr.y (mgr.34107) 10245 : audit [DBG] from='client.55909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[46934]: audit 2022-04-22T22:26:30.659593+0000 mon.a (mon.0) 14320 : audit [DBG] from='client.? 172.21.15.115:0/4141397451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:26:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:31 smithi115 bash[46934]: audit 2022-04-22T22:26:31.381216+0000 mon.a (mon.0) 14321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:32 smithi121 bash[36902]: cluster 2022-04-22T22:26:31.414778+0000 mgr.y (mgr.34107) 10246 : cluster [DBG] pgmap v10953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:33.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:32 smithi115 bash[45495]: cluster 2022-04-22T22:26:31.414778+0000 mgr.y (mgr.34107) 10246 : cluster [DBG] pgmap v10953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:33.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:32.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:33.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:32 smithi115 bash[46934]: cluster 2022-04-22T22:26:31.414778+0000 mgr.y (mgr.34107) 10246 : cluster [DBG] pgmap v10953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:33 smithi121 bash[36902]: audit 2022-04-22T22:26:32.664785+0000 mon.a (mon.0) 14322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:33 smithi115 bash[45495]: audit 2022-04-22T22:26:32.664785+0000 mon.a (mon.0) 14322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:33 smithi115 bash[46934]: audit 2022-04-22T22:26:32.664785+0000 mon.a (mon.0) 14322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:34 smithi121 bash[36902]: cluster 2022-04-22T22:26:33.416054+0000 mgr.y (mgr.34107) 10247 : cluster [DBG] pgmap v10954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:34 smithi121 bash[36902]: audit 2022-04-22T22:26:33.761053+0000 mon.a (mon.0) 14323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:34 smithi115 bash[45495]: cluster 2022-04-22T22:26:33.416054+0000 mgr.y (mgr.34107) 10247 : cluster [DBG] pgmap v10954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:34 smithi115 bash[45495]: audit 2022-04-22T22:26:33.761053+0000 mon.a (mon.0) 14323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:34.652Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:34 smithi115 bash[46934]: cluster 2022-04-22T22:26:33.416054+0000 mgr.y (mgr.34107) 10247 : cluster [DBG] pgmap v10954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:34 smithi115 bash[46934]: audit 2022-04-22T22:26:33.761053+0000 mon.a (mon.0) 14323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:36 smithi121 bash[36902]: cluster 2022-04-22T22:26:35.417147+0000 mgr.y (mgr.34107) 10248 : cluster [DBG] pgmap v10955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:36 smithi121 bash[36902]: audit 2022-04-22T22:26:36.384533+0000 mon.a (mon.0) 14324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:36 smithi115 bash[45495]: cluster 2022-04-22T22:26:35.417147+0000 mgr.y (mgr.34107) 10248 : cluster [DBG] pgmap v10955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:36 smithi115 bash[45495]: audit 2022-04-22T22:26:36.384533+0000 mon.a (mon.0) 14324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:36 smithi115 bash[46934]: cluster 2022-04-22T22:26:35.417147+0000 mgr.y (mgr.34107) 10248 : cluster [DBG] pgmap v10955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:36 smithi115 bash[46934]: audit 2022-04-22T22:26:36.384533+0000 mon.a (mon.0) 14324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:38 smithi121 bash[36902]: cluster 2022-04-22T22:26:37.418208+0000 mgr.y (mgr.34107) 10249 : cluster [DBG] pgmap v10956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:38 smithi121 bash[36902]: audit 2022-04-22T22:26:37.668226+0000 mon.a (mon.0) 14325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:38 smithi115 bash[45495]: cluster 2022-04-22T22:26:37.418208+0000 mgr.y (mgr.34107) 10249 : cluster [DBG] pgmap v10956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:38 smithi115 bash[45495]: audit 2022-04-22T22:26:37.668226+0000 mon.a (mon.0) 14325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:38 smithi115 bash[46934]: cluster 2022-04-22T22:26:37.418208+0000 mgr.y (mgr.34107) 10249 : cluster [DBG] pgmap v10956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:38 smithi115 bash[46934]: audit 2022-04-22T22:26:37.668226+0000 mon.a (mon.0) 14325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:39 smithi121 bash[36902]: audit 2022-04-22T22:26:38.764280+0000 mon.a (mon.0) 14326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:39 smithi115 bash[45495]: audit 2022-04-22T22:26:38.764280+0000 mon.a (mon.0) 14326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:39 smithi115 bash[46934]: audit 2022-04-22T22:26:38.764280+0000 mon.a (mon.0) 14326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:40 smithi121 bash[36902]: cluster 2022-04-22T22:26:39.419322+0000 mgr.y (mgr.34107) 10250 : cluster [DBG] pgmap v10957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:40 smithi115 bash[45495]: cluster 2022-04-22T22:26:39.419322+0000 mgr.y (mgr.34107) 10250 : cluster [DBG] pgmap v10957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:40 smithi115 bash[46934]: cluster 2022-04-22T22:26:39.419322+0000 mgr.y (mgr.34107) 10250 : cluster [DBG] pgmap v10957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:41 smithi121 bash[36902]: audit 2022-04-22T22:26:41.387643+0000 mon.a (mon.0) 14327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:41 smithi115 bash[45495]: audit 2022-04-22T22:26:41.387643+0000 mon.a (mon.0) 14327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:41 smithi115 bash[46934]: audit 2022-04-22T22:26:41.387643+0000 mon.a (mon.0) 14327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:42 smithi121 bash[36902]: cluster 2022-04-22T22:26:41.420394+0000 mgr.y (mgr.34107) 10251 : cluster [DBG] pgmap v10958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:42.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:42 smithi115 bash[45495]: cluster 2022-04-22T22:26:41.420394+0000 mgr.y (mgr.34107) 10251 : cluster [DBG] pgmap v10958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:42 smithi115 bash[46934]: cluster 2022-04-22T22:26:41.420394+0000 mgr.y (mgr.34107) 10251 : cluster [DBG] pgmap v10958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:43 smithi121 bash[36902]: audit 2022-04-22T22:26:42.672009+0000 mon.a (mon.0) 14328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:43 smithi115 bash[45495]: audit 2022-04-22T22:26:42.672009+0000 mon.a (mon.0) 14328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:43 smithi115 bash[46934]: audit 2022-04-22T22:26:42.672009+0000 mon.a (mon.0) 14328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:44 smithi121 bash[36902]: cluster 2022-04-22T22:26:43.421456+0000 mgr.y (mgr.34107) 10252 : cluster [DBG] pgmap v10959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:44 smithi121 bash[36902]: audit 2022-04-22T22:26:43.767173+0000 mon.a (mon.0) 14329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:44 smithi115 bash[45495]: cluster 2022-04-22T22:26:43.421456+0000 mgr.y (mgr.34107) 10252 : cluster [DBG] pgmap v10959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:44 smithi115 bash[45495]: audit 2022-04-22T22:26:43.767173+0000 mon.a (mon.0) 14329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:44.651Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:44 smithi115 bash[46934]: cluster 2022-04-22T22:26:43.421456+0000 mgr.y (mgr.34107) 10252 : cluster [DBG] pgmap v10959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:44 smithi115 bash[46934]: audit 2022-04-22T22:26:43.767173+0000 mon.a (mon.0) 14329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:46 smithi121 bash[36902]: cluster 2022-04-22T22:26:45.422462+0000 mgr.y (mgr.34107) 10253 : cluster [DBG] pgmap v10960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:46 smithi121 bash[36902]: audit 2022-04-22T22:26:46.390934+0000 mon.a (mon.0) 14330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:46 smithi115 bash[45495]: cluster 2022-04-22T22:26:45.422462+0000 mgr.y (mgr.34107) 10253 : cluster [DBG] pgmap v10960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:46 smithi115 bash[45495]: audit 2022-04-22T22:26:46.390934+0000 mon.a (mon.0) 14330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:46 smithi115 bash[46934]: cluster 2022-04-22T22:26:45.422462+0000 mgr.y (mgr.34107) 10253 : cluster [DBG] pgmap v10960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:46 smithi115 bash[46934]: audit 2022-04-22T22:26:46.390934+0000 mon.a (mon.0) 14330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:48 smithi121 bash[36902]: cluster 2022-04-22T22:26:47.423521+0000 mgr.y (mgr.34107) 10254 : cluster [DBG] pgmap v10961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:48 smithi121 bash[36902]: audit 2022-04-22T22:26:47.675704+0000 mon.a (mon.0) 14331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:48 smithi115 bash[45495]: cluster 2022-04-22T22:26:47.423521+0000 mgr.y (mgr.34107) 10254 : cluster [DBG] pgmap v10961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:48 smithi115 bash[45495]: audit 2022-04-22T22:26:47.675704+0000 mon.a (mon.0) 14331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:48 smithi115 bash[46934]: cluster 2022-04-22T22:26:47.423521+0000 mgr.y (mgr.34107) 10254 : cluster [DBG] pgmap v10961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:48 smithi115 bash[46934]: audit 2022-04-22T22:26:47.675704+0000 mon.a (mon.0) 14331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:49 smithi121 bash[36902]: audit 2022-04-22T22:26:48.770279+0000 mon.a (mon.0) 14332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:49 smithi115 bash[45495]: audit 2022-04-22T22:26:48.770279+0000 mon.a (mon.0) 14332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:49 smithi115 bash[46934]: audit 2022-04-22T22:26:48.770279+0000 mon.a (mon.0) 14332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:50 smithi121 bash[36902]: cluster 2022-04-22T22:26:49.424487+0000 mgr.y (mgr.34107) 10255 : cluster [DBG] pgmap v10962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:50 smithi115 bash[45495]: cluster 2022-04-22T22:26:49.424487+0000 mgr.y (mgr.34107) 10255 : cluster [DBG] pgmap v10962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:50 smithi115 bash[46934]: cluster 2022-04-22T22:26:49.424487+0000 mgr.y (mgr.34107) 10255 : cluster [DBG] pgmap v10962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:51 smithi121 bash[36902]: audit 2022-04-22T22:26:51.394262+0000 mon.a (mon.0) 14333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:52.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:51 smithi115 bash[46934]: audit 2022-04-22T22:26:51.394262+0000 mon.a (mon.0) 14333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:51 smithi115 bash[45495]: audit 2022-04-22T22:26:51.394262+0000 mon.a (mon.0) 14333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:52 smithi121 bash[36902]: cluster 2022-04-22T22:26:51.425631+0000 mgr.y (mgr.34107) 10256 : cluster [DBG] pgmap v10963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:53.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:52 smithi115 bash[45495]: cluster 2022-04-22T22:26:51.425631+0000 mgr.y (mgr.34107) 10256 : cluster [DBG] pgmap v10963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:52.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:52 smithi115 bash[46934]: cluster 2022-04-22T22:26:51.425631+0000 mgr.y (mgr.34107) 10256 : cluster [DBG] pgmap v10963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:53 smithi121 bash[36902]: audit 2022-04-22T22:26:52.679097+0000 mon.a (mon.0) 14334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:53 smithi121 bash[36902]: cluster 2022-04-22T22:26:53.426737+0000 mgr.y (mgr.34107) 10257 : cluster [DBG] pgmap v10964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:53 smithi115 bash[45495]: audit 2022-04-22T22:26:52.679097+0000 mon.a (mon.0) 14334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:53 smithi115 bash[45495]: cluster 2022-04-22T22:26:53.426737+0000 mgr.y (mgr.34107) 10257 : cluster [DBG] pgmap v10964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:53 smithi115 bash[46934]: audit 2022-04-22T22:26:52.679097+0000 mon.a (mon.0) 14334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:53 smithi115 bash[46934]: cluster 2022-04-22T22:26:53.426737+0000 mgr.y (mgr.34107) 10257 : cluster [DBG] pgmap v10964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:54 smithi121 bash[36902]: audit 2022-04-22T22:26:53.773166+0000 mon.a (mon.0) 14335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:55.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:54 smithi115 bash[46934]: audit 2022-04-22T22:26:53.773166+0000 mon.a (mon.0) 14335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:26:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:26:54.654Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:26:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:54 smithi115 bash[45495]: audit 2022-04-22T22:26:53.773166+0000 mon.a (mon.0) 14335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:55 smithi121 bash[36902]: cluster 2022-04-22T22:26:55.427840+0000 mgr.y (mgr.34107) 10258 : cluster [DBG] pgmap v10965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:55 smithi115 bash[45495]: cluster 2022-04-22T22:26:55.427840+0000 mgr.y (mgr.34107) 10258 : cluster [DBG] pgmap v10965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:55 smithi115 bash[46934]: cluster 2022-04-22T22:26:55.427840+0000 mgr.y (mgr.34107) 10258 : cluster [DBG] pgmap v10965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:56 smithi121 bash[36902]: audit 2022-04-22T22:26:56.397221+0000 mon.a (mon.0) 14336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:56 smithi115 bash[45495]: audit 2022-04-22T22:26:56.397221+0000 mon.a (mon.0) 14336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:56 smithi115 bash[46934]: audit 2022-04-22T22:26:56.397221+0000 mon.a (mon.0) 14336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:26:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:57 smithi121 bash[36902]: cluster 2022-04-22T22:26:57.428935+0000 mgr.y (mgr.34107) 10259 : cluster [DBG] pgmap v10966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:57 smithi121 bash[36902]: audit 2022-04-22T22:26:57.682683+0000 mon.a (mon.0) 14337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:57 smithi115 bash[45495]: cluster 2022-04-22T22:26:57.428935+0000 mgr.y (mgr.34107) 10259 : cluster [DBG] pgmap v10966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:57 smithi115 bash[45495]: audit 2022-04-22T22:26:57.682683+0000 mon.a (mon.0) 14337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:57 smithi115 bash[46934]: cluster 2022-04-22T22:26:57.428935+0000 mgr.y (mgr.34107) 10259 : cluster [DBG] pgmap v10966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:26:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:57 smithi115 bash[46934]: audit 2022-04-22T22:26:57.682683+0000 mon.a (mon.0) 14337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:26:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:58 smithi121 bash[36902]: audit 2022-04-22T22:26:58.776325+0000 mon.a (mon.0) 14338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:58 smithi115 bash[45495]: audit 2022-04-22T22:26:58.776325+0000 mon.a (mon.0) 14338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:26:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:58 smithi115 bash[46934]: audit 2022-04-22T22:26:58.776325+0000 mon.a (mon.0) 14338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:26:59 smithi121 bash[36902]: cluster 2022-04-22T22:26:59.430007+0000 mgr.y (mgr.34107) 10260 : cluster [DBG] pgmap v10967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:26:59 smithi115 bash[45495]: cluster 2022-04-22T22:26:59.430007+0000 mgr.y (mgr.34107) 10260 : cluster [DBG] pgmap v10967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:26:59 smithi115 bash[46934]: cluster 2022-04-22T22:26:59.430007+0000 mgr.y (mgr.34107) 10260 : cluster [DBG] pgmap v10967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:01.039 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 44s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 109s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 44s ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 109s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:27:01.381 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 109s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (4h) 44s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 44s ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 109s ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 44s ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 44s ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 109s ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 44s ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 44s ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 44s ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:27:01.382 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 44s ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:27:01.383 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 109s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:27:01.383 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 109s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:27:01.383 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 109s ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:27:01.383 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 109s ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:27:01.383 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 109s ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:27:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:01 smithi121 bash[36902]: audit 2022-04-22T22:27:01.400320+0000 mon.a (mon.0) 14339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:01.786 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:27:01.786 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:27:01.786 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:27:01.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:01.787 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:27:01.787 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:27:01.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:01.787 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:27:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:27:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:27:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:27:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:01.789 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:27:01.789 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:27:01.789 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:27:01.789 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:27:01.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:01 smithi115 bash[46934]: audit 2022-04-22T22:27:01.400320+0000 mon.a (mon.0) 14339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:01 smithi115 bash[45495]: audit 2022-04-22T22:27:01.400320+0000 mon.a (mon.0) 14339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[45495]: audit 2022-04-22T22:27:01.019785+0000 mgr.y (mgr.34107) 10261 : audit [DBG] from='client.45987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[45495]: audit 2022-04-22T22:27:01.358344+0000 mgr.y (mgr.34107) 10262 : audit [DBG] from='client.45993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[45495]: cluster 2022-04-22T22:27:01.431027+0000 mgr.y (mgr.34107) 10263 : cluster [DBG] pgmap v10968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[45495]: audit 2022-04-22T22:27:01.782504+0000 mon.c (mon.1) 350 : audit [DBG] from='client.? 172.21.15.115:0/1986276178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[46934]: audit 2022-04-22T22:27:01.019785+0000 mgr.y (mgr.34107) 10261 : audit [DBG] from='client.45987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[46934]: audit 2022-04-22T22:27:01.358344+0000 mgr.y (mgr.34107) 10262 : audit [DBG] from='client.45993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[46934]: cluster 2022-04-22T22:27:01.431027+0000 mgr.y (mgr.34107) 10263 : cluster [DBG] pgmap v10968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[46934]: audit 2022-04-22T22:27:01.782504+0000 mon.c (mon.1) 350 : audit [DBG] from='client.? 172.21.15.115:0/1986276178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:02 smithi121 bash[36902]: audit 2022-04-22T22:27:01.019785+0000 mgr.y (mgr.34107) 10261 : audit [DBG] from='client.45987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:02 smithi121 bash[36902]: audit 2022-04-22T22:27:01.358344+0000 mgr.y (mgr.34107) 10262 : audit [DBG] from='client.45993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:02 smithi121 bash[36902]: cluster 2022-04-22T22:27:01.431027+0000 mgr.y (mgr.34107) 10263 : cluster [DBG] pgmap v10968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:02 smithi121 bash[36902]: audit 2022-04-22T22:27:01.782504+0000 mon.c (mon.1) 350 : audit [DBG] from='client.? 172.21.15.115:0/1986276178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:02.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:03.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:03 smithi115 bash[45495]: audit 2022-04-22T22:27:02.686483+0000 mon.a (mon.0) 14340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:03 smithi115 bash[46934]: audit 2022-04-22T22:27:02.686483+0000 mon.a (mon.0) 14340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:03 smithi121 bash[36902]: audit 2022-04-22T22:27:02.686483+0000 mon.a (mon.0) 14340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:04.791 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:04.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:04 smithi115 bash[45495]: cluster 2022-04-22T22:27:03.432217+0000 mgr.y (mgr.34107) 10264 : cluster [DBG] pgmap v10969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:04 smithi115 bash[45495]: audit 2022-04-22T22:27:03.779400+0000 mon.a (mon.0) 14341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:04 smithi115 bash[46934]: cluster 2022-04-22T22:27:03.432217+0000 mgr.y (mgr.34107) 10264 : cluster [DBG] pgmap v10969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:04 smithi115 bash[46934]: audit 2022-04-22T22:27:03.779400+0000 mon.a (mon.0) 14341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:04 smithi121 bash[36902]: cluster 2022-04-22T22:27:03.432217+0000 mgr.y (mgr.34107) 10264 : cluster [DBG] pgmap v10969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:04 smithi121 bash[36902]: audit 2022-04-22T22:27:03.779400+0000 mon.a (mon.0) 14341 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:06 smithi115 bash[45495]: cluster 2022-04-22T22:27:05.433345+0000 mgr.y (mgr.34107) 10265 : cluster [DBG] pgmap v10970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:06 smithi115 bash[45495]: audit 2022-04-22T22:27:06.402843+0000 mon.a (mon.0) 14342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:06 smithi115 bash[46934]: cluster 2022-04-22T22:27:05.433345+0000 mgr.y (mgr.34107) 10265 : cluster [DBG] pgmap v10970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:06 smithi115 bash[46934]: audit 2022-04-22T22:27:06.402843+0000 mon.a (mon.0) 14342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:06 smithi121 bash[36902]: cluster 2022-04-22T22:27:05.433345+0000 mgr.y (mgr.34107) 10265 : cluster [DBG] pgmap v10970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:06 smithi121 bash[36902]: audit 2022-04-22T22:27:06.402843+0000 mon.a (mon.0) 14342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:08 smithi121 bash[36902]: cluster 2022-04-22T22:27:07.434461+0000 mgr.y (mgr.34107) 10266 : cluster [DBG] pgmap v10971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:08 smithi121 bash[36902]: audit 2022-04-22T22:27:07.690348+0000 mon.a (mon.0) 14343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:08 smithi115 bash[46934]: cluster 2022-04-22T22:27:07.434461+0000 mgr.y (mgr.34107) 10266 : cluster [DBG] pgmap v10971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:08 smithi115 bash[46934]: audit 2022-04-22T22:27:07.690348+0000 mon.a (mon.0) 14343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:08 smithi115 bash[45495]: cluster 2022-04-22T22:27:07.434461+0000 mgr.y (mgr.34107) 10266 : cluster [DBG] pgmap v10971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:08 smithi115 bash[45495]: audit 2022-04-22T22:27:07.690348+0000 mon.a (mon.0) 14343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:09 smithi121 bash[36902]: audit 2022-04-22T22:27:08.782780+0000 mon.a (mon.0) 14344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:10.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:09 smithi115 bash[46934]: audit 2022-04-22T22:27:08.782780+0000 mon.a (mon.0) 14344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:09 smithi115 bash[45495]: audit 2022-04-22T22:27:08.782780+0000 mon.a (mon.0) 14344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:10 smithi121 bash[36902]: cluster 2022-04-22T22:27:09.435607+0000 mgr.y (mgr.34107) 10267 : cluster [DBG] pgmap v10972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:10 smithi115 bash[45495]: cluster 2022-04-22T22:27:09.435607+0000 mgr.y (mgr.34107) 10267 : cluster [DBG] pgmap v10972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:10 smithi115 bash[46934]: cluster 2022-04-22T22:27:09.435607+0000 mgr.y (mgr.34107) 10267 : cluster [DBG] pgmap v10972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:11 smithi121 bash[36902]: audit 2022-04-22T22:27:11.406075+0000 mon.a (mon.0) 14345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:11 smithi115 bash[45495]: audit 2022-04-22T22:27:11.406075+0000 mon.a (mon.0) 14345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:11 smithi115 bash[46934]: audit 2022-04-22T22:27:11.406075+0000 mon.a (mon.0) 14345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:12 smithi121 bash[36902]: cluster 2022-04-22T22:27:11.436622+0000 mgr.y (mgr.34107) 10268 : cluster [DBG] pgmap v10973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:13.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:12 smithi115 bash[45495]: cluster 2022-04-22T22:27:11.436622+0000 mgr.y (mgr.34107) 10268 : cluster [DBG] pgmap v10973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:13.074 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:12.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:13.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:12 smithi115 bash[46934]: cluster 2022-04-22T22:27:11.436622+0000 mgr.y (mgr.34107) 10268 : cluster [DBG] pgmap v10973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:13 smithi121 bash[36902]: audit 2022-04-22T22:27:12.694183+0000 mon.a (mon.0) 14346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:13 smithi115 bash[45495]: audit 2022-04-22T22:27:12.694183+0000 mon.a (mon.0) 14346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:13 smithi115 bash[46934]: audit 2022-04-22T22:27:12.694183+0000 mon.a (mon.0) 14346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:14 smithi121 bash[36902]: cluster 2022-04-22T22:27:13.437755+0000 mgr.y (mgr.34107) 10269 : cluster [DBG] pgmap v10974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:14 smithi121 bash[36902]: audit 2022-04-22T22:27:13.785951+0000 mon.a (mon.0) 14347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:14.655Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:14 smithi115 bash[45495]: cluster 2022-04-22T22:27:13.437755+0000 mgr.y (mgr.34107) 10269 : cluster [DBG] pgmap v10974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:14 smithi115 bash[45495]: audit 2022-04-22T22:27:13.785951+0000 mon.a (mon.0) 14347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:14 smithi115 bash[46934]: cluster 2022-04-22T22:27:13.437755+0000 mgr.y (mgr.34107) 10269 : cluster [DBG] pgmap v10974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:14 smithi115 bash[46934]: audit 2022-04-22T22:27:13.785951+0000 mon.a (mon.0) 14347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:16 smithi121 bash[36902]: cluster 2022-04-22T22:27:15.438791+0000 mgr.y (mgr.34107) 10270 : cluster [DBG] pgmap v10975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:16 smithi121 bash[36902]: audit 2022-04-22T22:27:16.409492+0000 mon.a (mon.0) 14348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:16 smithi115 bash[45495]: cluster 2022-04-22T22:27:15.438791+0000 mgr.y (mgr.34107) 10270 : cluster [DBG] pgmap v10975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:16 smithi115 bash[45495]: audit 2022-04-22T22:27:16.409492+0000 mon.a (mon.0) 14348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:16 smithi115 bash[46934]: cluster 2022-04-22T22:27:15.438791+0000 mgr.y (mgr.34107) 10270 : cluster [DBG] pgmap v10975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:16 smithi115 bash[46934]: audit 2022-04-22T22:27:16.409492+0000 mon.a (mon.0) 14348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:17 smithi121 bash[36902]: audit 2022-04-22T22:27:16.736223+0000 mon.a (mon.0) 14349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:27:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:17 smithi121 bash[36902]: audit 2022-04-22T22:27:16.741648+0000 mon.a (mon.0) 14350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:27:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:17 smithi121 bash[36902]: audit 2022-04-22T22:27:17.422314+0000 mon.a (mon.0) 14351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:27:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:17 smithi121 bash[36902]: audit 2022-04-22T22:27:17.425441+0000 mon.a (mon.0) 14352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:27:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:17 smithi121 bash[36902]: audit 2022-04-22T22:27:17.427313+0000 mon.a (mon.0) 14353 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[45495]: audit 2022-04-22T22:27:16.736223+0000 mon.a (mon.0) 14349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[45495]: audit 2022-04-22T22:27:16.741648+0000 mon.a (mon.0) 14350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[45495]: audit 2022-04-22T22:27:17.422314+0000 mon.a (mon.0) 14351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[45495]: audit 2022-04-22T22:27:17.425441+0000 mon.a (mon.0) 14352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[45495]: audit 2022-04-22T22:27:17.427313+0000 mon.a (mon.0) 14353 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[46934]: audit 2022-04-22T22:27:16.736223+0000 mon.a (mon.0) 14349 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:27:17.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[46934]: audit 2022-04-22T22:27:16.741648+0000 mon.a (mon.0) 14350 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:27:17.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[46934]: audit 2022-04-22T22:27:17.422314+0000 mon.a (mon.0) 14351 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:27:17.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[46934]: audit 2022-04-22T22:27:17.425441+0000 mon.a (mon.0) 14352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:27:17.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:17 smithi115 bash[46934]: audit 2022-04-22T22:27:17.427313+0000 mon.a (mon.0) 14353 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:27:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:18 smithi121 bash[36902]: cluster 2022-04-22T22:27:17.439661+0000 mgr.y (mgr.34107) 10271 : cluster [DBG] pgmap v10976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:18 smithi121 bash[36902]: audit 2022-04-22T22:27:17.697849+0000 mon.a (mon.0) 14354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:18 smithi121 bash[36902]: audit 2022-04-22T22:27:17.780875+0000 mon.a (mon.0) 14355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:27:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[45495]: cluster 2022-04-22T22:27:17.439661+0000 mgr.y (mgr.34107) 10271 : cluster [DBG] pgmap v10976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[45495]: audit 2022-04-22T22:27:17.697849+0000 mon.a (mon.0) 14354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[45495]: audit 2022-04-22T22:27:17.780875+0000 mon.a (mon.0) 14355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:27:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[46934]: cluster 2022-04-22T22:27:17.439661+0000 mgr.y (mgr.34107) 10271 : cluster [DBG] pgmap v10976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[46934]: audit 2022-04-22T22:27:17.697849+0000 mon.a (mon.0) 14354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:18 smithi115 bash[46934]: audit 2022-04-22T22:27:17.780875+0000 mon.a (mon.0) 14355 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:27:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:19 smithi121 bash[36902]: audit 2022-04-22T22:27:18.788927+0000 mon.a (mon.0) 14356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:19 smithi115 bash[45495]: audit 2022-04-22T22:27:18.788927+0000 mon.a (mon.0) 14356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:19 smithi115 bash[46934]: audit 2022-04-22T22:27:18.788927+0000 mon.a (mon.0) 14356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:20 smithi121 bash[36902]: cluster 2022-04-22T22:27:19.440690+0000 mgr.y (mgr.34107) 10272 : cluster [DBG] pgmap v10977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:20 smithi115 bash[45495]: cluster 2022-04-22T22:27:19.440690+0000 mgr.y (mgr.34107) 10272 : cluster [DBG] pgmap v10977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:20 smithi115 bash[46934]: cluster 2022-04-22T22:27:19.440690+0000 mgr.y (mgr.34107) 10272 : cluster [DBG] pgmap v10977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:21 smithi121 bash[36902]: audit 2022-04-22T22:27:21.412710+0000 mon.a (mon.0) 14357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:21 smithi115 bash[45495]: audit 2022-04-22T22:27:21.412710+0000 mon.a (mon.0) 14357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:21 smithi115 bash[46934]: audit 2022-04-22T22:27:21.412710+0000 mon.a (mon.0) 14357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:22 smithi121 bash[36902]: cluster 2022-04-22T22:27:21.441800+0000 mgr.y (mgr.34107) 10273 : cluster [DBG] pgmap v10978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:23.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:22 smithi115 bash[45495]: cluster 2022-04-22T22:27:21.441800+0000 mgr.y (mgr.34107) 10273 : cluster [DBG] pgmap v10978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:22.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:22 smithi115 bash[46934]: cluster 2022-04-22T22:27:21.441800+0000 mgr.y (mgr.34107) 10273 : cluster [DBG] pgmap v10978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:23 smithi121 bash[36902]: audit 2022-04-22T22:27:22.701376+0000 mon.a (mon.0) 14358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:23 smithi115 bash[45495]: audit 2022-04-22T22:27:22.701376+0000 mon.a (mon.0) 14358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:23 smithi115 bash[46934]: audit 2022-04-22T22:27:22.701376+0000 mon.a (mon.0) 14358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:24 smithi121 bash[36902]: cluster 2022-04-22T22:27:23.442917+0000 mgr.y (mgr.34107) 10274 : cluster [DBG] pgmap v10979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:24 smithi121 bash[36902]: audit 2022-04-22T22:27:23.791706+0000 mon.a (mon.0) 14359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:24.656Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:24 smithi115 bash[45495]: cluster 2022-04-22T22:27:23.442917+0000 mgr.y (mgr.34107) 10274 : cluster [DBG] pgmap v10979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:24 smithi115 bash[45495]: audit 2022-04-22T22:27:23.791706+0000 mon.a (mon.0) 14359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:24 smithi115 bash[46934]: cluster 2022-04-22T22:27:23.442917+0000 mgr.y (mgr.34107) 10274 : cluster [DBG] pgmap v10979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:24 smithi115 bash[46934]: audit 2022-04-22T22:27:23.791706+0000 mon.a (mon.0) 14359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:26 smithi121 bash[36902]: cluster 2022-04-22T22:27:25.444005+0000 mgr.y (mgr.34107) 10275 : cluster [DBG] pgmap v10980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:26 smithi121 bash[36902]: audit 2022-04-22T22:27:26.415960+0000 mon.a (mon.0) 14360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:26 smithi115 bash[45495]: cluster 2022-04-22T22:27:25.444005+0000 mgr.y (mgr.34107) 10275 : cluster [DBG] pgmap v10980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:26 smithi115 bash[45495]: audit 2022-04-22T22:27:26.415960+0000 mon.a (mon.0) 14360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:26 smithi115 bash[46934]: cluster 2022-04-22T22:27:25.444005+0000 mgr.y (mgr.34107) 10275 : cluster [DBG] pgmap v10980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:26 smithi115 bash[46934]: audit 2022-04-22T22:27:26.415960+0000 mon.a (mon.0) 14360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:28 smithi121 bash[36902]: cluster 2022-04-22T22:27:27.445063+0000 mgr.y (mgr.34107) 10276 : cluster [DBG] pgmap v10981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:28 smithi121 bash[36902]: audit 2022-04-22T22:27:27.705217+0000 mon.a (mon.0) 14361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:28 smithi115 bash[45495]: cluster 2022-04-22T22:27:27.445063+0000 mgr.y (mgr.34107) 10276 : cluster [DBG] pgmap v10981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:28 smithi115 bash[45495]: audit 2022-04-22T22:27:27.705217+0000 mon.a (mon.0) 14361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:28 smithi115 bash[46934]: cluster 2022-04-22T22:27:27.445063+0000 mgr.y (mgr.34107) 10276 : cluster [DBG] pgmap v10981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:28 smithi115 bash[46934]: audit 2022-04-22T22:27:27.705217+0000 mon.a (mon.0) 14361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:29 smithi121 bash[36902]: audit 2022-04-22T22:27:28.794870+0000 mon.a (mon.0) 14362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:29 smithi115 bash[45495]: audit 2022-04-22T22:27:28.794870+0000 mon.a (mon.0) 14362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:29 smithi115 bash[46934]: audit 2022-04-22T22:27:28.794870+0000 mon.a (mon.0) 14362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:30 smithi121 bash[36902]: cluster 2022-04-22T22:27:29.446074+0000 mgr.y (mgr.34107) 10277 : cluster [DBG] pgmap v10982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:30 smithi115 bash[45495]: cluster 2022-04-22T22:27:29.446074+0000 mgr.y (mgr.34107) 10277 : cluster [DBG] pgmap v10982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:30 smithi115 bash[46934]: cluster 2022-04-22T22:27:29.446074+0000 mgr.y (mgr.34107) 10277 : cluster [DBG] pgmap v10982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:31 smithi121 bash[36902]: audit 2022-04-22T22:27:31.419152+0000 mon.a (mon.0) 14363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:32.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:31 smithi115 bash[46934]: audit 2022-04-22T22:27:31.419152+0000 mon.a (mon.0) 14363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:31 smithi115 bash[45495]: audit 2022-04-22T22:27:31.419152+0000 mon.a (mon.0) 14363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:32.162 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:27:32.510 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:27:32.510 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 75s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:27:32.510 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:27:32.510 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 75s ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:27:32.510 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (4h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 75s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4h) 75s ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 75s ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:27:32.511 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 75s ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 75s ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 75s ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 75s ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 75s ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:27:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:27:32.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:27:32.513 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:27:32.513 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:27:32.910 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:27:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:27:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:27:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:27:32.913 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:27:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:32 smithi121 bash[36902]: cluster 2022-04-22T22:27:31.447117+0000 mgr.y (mgr.34107) 10278 : cluster [DBG] pgmap v10983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:33.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:32 smithi115 bash[45495]: cluster 2022-04-22T22:27:31.447117+0000 mgr.y (mgr.34107) 10278 : cluster [DBG] pgmap v10983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:32 smithi115 bash[46934]: cluster 2022-04-22T22:27:31.447117+0000 mgr.y (mgr.34107) 10278 : cluster [DBG] pgmap v10983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:32.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:33 smithi121 bash[36902]: audit 2022-04-22T22:27:32.141956+0000 mgr.y (mgr.34107) 10279 : audit [DBG] from='client.55939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:33 smithi121 bash[36902]: audit 2022-04-22T22:27:32.487719+0000 mgr.y (mgr.34107) 10280 : audit [DBG] from='client.46011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:33 smithi121 bash[36902]: audit 2022-04-22T22:27:32.708977+0000 mon.a (mon.0) 14364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:33 smithi121 bash[36902]: audit 2022-04-22T22:27:32.906885+0000 mon.c (mon.1) 351 : audit [DBG] from='client.? 172.21.15.115:0/3936316812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[45495]: audit 2022-04-22T22:27:32.141956+0000 mgr.y (mgr.34107) 10279 : audit [DBG] from='client.55939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[45495]: audit 2022-04-22T22:27:32.487719+0000 mgr.y (mgr.34107) 10280 : audit [DBG] from='client.46011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[45495]: audit 2022-04-22T22:27:32.708977+0000 mon.a (mon.0) 14364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[45495]: audit 2022-04-22T22:27:32.906885+0000 mon.c (mon.1) 351 : audit [DBG] from='client.? 172.21.15.115:0/3936316812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[46934]: audit 2022-04-22T22:27:32.141956+0000 mgr.y (mgr.34107) 10279 : audit [DBG] from='client.55939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[46934]: audit 2022-04-22T22:27:32.487719+0000 mgr.y (mgr.34107) 10280 : audit [DBG] from='client.46011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:27:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[46934]: audit 2022-04-22T22:27:32.708977+0000 mon.a (mon.0) 14364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:33 smithi115 bash[46934]: audit 2022-04-22T22:27:32.906885+0000 mon.c (mon.1) 351 : audit [DBG] from='client.? 172.21.15.115:0/3936316812' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:27:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:34 smithi121 bash[36902]: cluster 2022-04-22T22:27:33.448277+0000 mgr.y (mgr.34107) 10281 : cluster [DBG] pgmap v10984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:34 smithi121 bash[36902]: audit 2022-04-22T22:27:33.797884+0000 mon.a (mon.0) 14365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:34.657Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:34 smithi115 bash[45495]: cluster 2022-04-22T22:27:33.448277+0000 mgr.y (mgr.34107) 10281 : cluster [DBG] pgmap v10984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:34 smithi115 bash[45495]: audit 2022-04-22T22:27:33.797884+0000 mon.a (mon.0) 14365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:34 smithi115 bash[46934]: cluster 2022-04-22T22:27:33.448277+0000 mgr.y (mgr.34107) 10281 : cluster [DBG] pgmap v10984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:34 smithi115 bash[46934]: audit 2022-04-22T22:27:33.797884+0000 mon.a (mon.0) 14365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:36 smithi121 bash[36902]: cluster 2022-04-22T22:27:35.449465+0000 mgr.y (mgr.34107) 10282 : cluster [DBG] pgmap v10985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:36 smithi121 bash[36902]: audit 2022-04-22T22:27:36.422292+0000 mon.a (mon.0) 14366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:36 smithi115 bash[45495]: cluster 2022-04-22T22:27:35.449465+0000 mgr.y (mgr.34107) 10282 : cluster [DBG] pgmap v10985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:36 smithi115 bash[45495]: audit 2022-04-22T22:27:36.422292+0000 mon.a (mon.0) 14366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:36 smithi115 bash[46934]: cluster 2022-04-22T22:27:35.449465+0000 mgr.y (mgr.34107) 10282 : cluster [DBG] pgmap v10985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:36 smithi115 bash[46934]: audit 2022-04-22T22:27:36.422292+0000 mon.a (mon.0) 14366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:38 smithi121 bash[36902]: cluster 2022-04-22T22:27:37.450604+0000 mgr.y (mgr.34107) 10283 : cluster [DBG] pgmap v10986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:38 smithi121 bash[36902]: audit 2022-04-22T22:27:37.713509+0000 mon.a (mon.0) 14367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:38 smithi115 bash[45495]: cluster 2022-04-22T22:27:37.450604+0000 mgr.y (mgr.34107) 10283 : cluster [DBG] pgmap v10986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:38 smithi115 bash[45495]: audit 2022-04-22T22:27:37.713509+0000 mon.a (mon.0) 14367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:38 smithi115 bash[46934]: cluster 2022-04-22T22:27:37.450604+0000 mgr.y (mgr.34107) 10283 : cluster [DBG] pgmap v10986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:38 smithi115 bash[46934]: audit 2022-04-22T22:27:37.713509+0000 mon.a (mon.0) 14367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:39 smithi121 bash[36902]: audit 2022-04-22T22:27:38.801023+0000 mon.a (mon.0) 14368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:39 smithi115 bash[45495]: audit 2022-04-22T22:27:38.801023+0000 mon.a (mon.0) 14368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:39 smithi115 bash[46934]: audit 2022-04-22T22:27:38.801023+0000 mon.a (mon.0) 14368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:40 smithi121 bash[36902]: cluster 2022-04-22T22:27:39.451752+0000 mgr.y (mgr.34107) 10284 : cluster [DBG] pgmap v10987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:40 smithi115 bash[45495]: cluster 2022-04-22T22:27:39.451752+0000 mgr.y (mgr.34107) 10284 : cluster [DBG] pgmap v10987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:40 smithi115 bash[46934]: cluster 2022-04-22T22:27:39.451752+0000 mgr.y (mgr.34107) 10284 : cluster [DBG] pgmap v10987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:41 smithi121 bash[36902]: audit 2022-04-22T22:27:41.425249+0000 mon.a (mon.0) 14369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:41 smithi121 bash[36902]: cluster 2022-04-22T22:27:41.452792+0000 mgr.y (mgr.34107) 10285 : cluster [DBG] pgmap v10988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:41 smithi115 bash[45495]: audit 2022-04-22T22:27:41.425249+0000 mon.a (mon.0) 14369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:41 smithi115 bash[45495]: cluster 2022-04-22T22:27:41.452792+0000 mgr.y (mgr.34107) 10285 : cluster [DBG] pgmap v10988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:41 smithi115 bash[46934]: audit 2022-04-22T22:27:41.425249+0000 mon.a (mon.0) 14369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:41 smithi115 bash[46934]: cluster 2022-04-22T22:27:41.452792+0000 mgr.y (mgr.34107) 10285 : cluster [DBG] pgmap v10988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:42 smithi115 bash[45495]: audit 2022-04-22T22:27:42.717347+0000 mon.a (mon.0) 14370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:42 smithi115 bash[46934]: audit 2022-04-22T22:27:42.717347+0000 mon.a (mon.0) 14370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:42.807Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:42 smithi121 bash[36902]: audit 2022-04-22T22:27:42.717347+0000 mon.a (mon.0) 14370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:43 smithi115 bash[45495]: cluster 2022-04-22T22:27:43.453917+0000 mgr.y (mgr.34107) 10286 : cluster [DBG] pgmap v10989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:43 smithi115 bash[46934]: cluster 2022-04-22T22:27:43.453917+0000 mgr.y (mgr.34107) 10286 : cluster [DBG] pgmap v10989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:43 smithi121 bash[36902]: cluster 2022-04-22T22:27:43.453917+0000 mgr.y (mgr.34107) 10286 : cluster [DBG] pgmap v10989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:44 smithi115 bash[45495]: audit 2022-04-22T22:27:43.803953+0000 mon.a (mon.0) 14371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:44.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:44 smithi115 bash[46934]: audit 2022-04-22T22:27:43.803953+0000 mon.a (mon.0) 14371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:44 smithi121 bash[36902]: audit 2022-04-22T22:27:43.803953+0000 mon.a (mon.0) 14371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:45 smithi121 bash[36902]: cluster 2022-04-22T22:27:45.454968+0000 mgr.y (mgr.34107) 10287 : cluster [DBG] pgmap v10990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:45 smithi115 bash[45495]: cluster 2022-04-22T22:27:45.454968+0000 mgr.y (mgr.34107) 10287 : cluster [DBG] pgmap v10990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:45 smithi115 bash[46934]: cluster 2022-04-22T22:27:45.454968+0000 mgr.y (mgr.34107) 10287 : cluster [DBG] pgmap v10990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:46 smithi121 bash[36902]: audit 2022-04-22T22:27:46.428342+0000 mon.a (mon.0) 14372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:46 smithi115 bash[45495]: audit 2022-04-22T22:27:46.428342+0000 mon.a (mon.0) 14372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:46 smithi115 bash[46934]: audit 2022-04-22T22:27:46.428342+0000 mon.a (mon.0) 14372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:47 smithi121 bash[36902]: cluster 2022-04-22T22:27:47.456112+0000 mgr.y (mgr.34107) 10288 : cluster [DBG] pgmap v10991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:47 smithi121 bash[36902]: audit 2022-04-22T22:27:47.722412+0000 mon.a (mon.0) 14373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:47 smithi115 bash[45495]: cluster 2022-04-22T22:27:47.456112+0000 mgr.y (mgr.34107) 10288 : cluster [DBG] pgmap v10991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:47 smithi115 bash[45495]: audit 2022-04-22T22:27:47.722412+0000 mon.a (mon.0) 14373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:47 smithi115 bash[46934]: cluster 2022-04-22T22:27:47.456112+0000 mgr.y (mgr.34107) 10288 : cluster [DBG] pgmap v10991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:47 smithi115 bash[46934]: audit 2022-04-22T22:27:47.722412+0000 mon.a (mon.0) 14373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:48 smithi121 bash[36902]: audit 2022-04-22T22:27:48.806977+0000 mon.a (mon.0) 14374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:48 smithi115 bash[45495]: audit 2022-04-22T22:27:48.806977+0000 mon.a (mon.0) 14374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:48 smithi115 bash[46934]: audit 2022-04-22T22:27:48.806977+0000 mon.a (mon.0) 14374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:49 smithi121 bash[36902]: cluster 2022-04-22T22:27:49.457312+0000 mgr.y (mgr.34107) 10289 : cluster [DBG] pgmap v10992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:49 smithi115 bash[45495]: cluster 2022-04-22T22:27:49.457312+0000 mgr.y (mgr.34107) 10289 : cluster [DBG] pgmap v10992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:49 smithi115 bash[46934]: cluster 2022-04-22T22:27:49.457312+0000 mgr.y (mgr.34107) 10289 : cluster [DBG] pgmap v10992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:51 smithi121 bash[36902]: audit 2022-04-22T22:27:51.431204+0000 mon.a (mon.0) 14375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:51 smithi115 bash[45495]: audit 2022-04-22T22:27:51.431204+0000 mon.a (mon.0) 14375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:51 smithi115 bash[46934]: audit 2022-04-22T22:27:51.431204+0000 mon.a (mon.0) 14375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:52.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:52 smithi115 bash[45495]: cluster 2022-04-22T22:27:51.458480+0000 mgr.y (mgr.34107) 10290 : cluster [DBG] pgmap v10993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:52 smithi115 bash[46934]: cluster 2022-04-22T22:27:51.458480+0000 mgr.y (mgr.34107) 10290 : cluster [DBG] pgmap v10993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:52 smithi121 bash[36902]: cluster 2022-04-22T22:27:51.458480+0000 mgr.y (mgr.34107) 10290 : cluster [DBG] pgmap v10993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:52.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:53 smithi121 bash[36902]: audit 2022-04-22T22:27:52.727943+0000 mon.a (mon.0) 14376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:53 smithi115 bash[45495]: audit 2022-04-22T22:27:52.727943+0000 mon.a (mon.0) 14376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:53 smithi115 bash[46934]: audit 2022-04-22T22:27:52.727943+0000 mon.a (mon.0) 14376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:54 smithi121 bash[36902]: cluster 2022-04-22T22:27:53.459643+0000 mgr.y (mgr.34107) 10291 : cluster [DBG] pgmap v10994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:54 smithi121 bash[36902]: audit 2022-04-22T22:27:53.810112+0000 mon.a (mon.0) 14377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:27:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:27:54.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:27:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:54 smithi115 bash[46934]: cluster 2022-04-22T22:27:53.459643+0000 mgr.y (mgr.34107) 10291 : cluster [DBG] pgmap v10994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:54 smithi115 bash[46934]: audit 2022-04-22T22:27:53.810112+0000 mon.a (mon.0) 14377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:54 smithi115 bash[45495]: cluster 2022-04-22T22:27:53.459643+0000 mgr.y (mgr.34107) 10291 : cluster [DBG] pgmap v10994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:54 smithi115 bash[45495]: audit 2022-04-22T22:27:53.810112+0000 mon.a (mon.0) 14377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:27:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:56 smithi121 bash[36902]: cluster 2022-04-22T22:27:55.460694+0000 mgr.y (mgr.34107) 10292 : cluster [DBG] pgmap v10995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:56 smithi121 bash[36902]: audit 2022-04-22T22:27:56.434342+0000 mon.a (mon.0) 14378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:56 smithi115 bash[45495]: cluster 2022-04-22T22:27:55.460694+0000 mgr.y (mgr.34107) 10292 : cluster [DBG] pgmap v10995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:56 smithi115 bash[45495]: audit 2022-04-22T22:27:56.434342+0000 mon.a (mon.0) 14378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:56 smithi115 bash[46934]: cluster 2022-04-22T22:27:55.460694+0000 mgr.y (mgr.34107) 10292 : cluster [DBG] pgmap v10995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:56 smithi115 bash[46934]: audit 2022-04-22T22:27:56.434342+0000 mon.a (mon.0) 14378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:27:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:58 smithi121 bash[36902]: cluster 2022-04-22T22:27:57.461760+0000 mgr.y (mgr.34107) 10293 : cluster [DBG] pgmap v10996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:58 smithi121 bash[36902]: audit 2022-04-22T22:27:57.731518+0000 mon.a (mon.0) 14379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:58 smithi115 bash[45495]: cluster 2022-04-22T22:27:57.461760+0000 mgr.y (mgr.34107) 10293 : cluster [DBG] pgmap v10996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:58 smithi115 bash[45495]: audit 2022-04-22T22:27:57.731518+0000 mon.a (mon.0) 14379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:27:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:58 smithi115 bash[46934]: cluster 2022-04-22T22:27:57.461760+0000 mgr.y (mgr.34107) 10293 : cluster [DBG] pgmap v10996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:27:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:58 smithi115 bash[46934]: audit 2022-04-22T22:27:57.731518+0000 mon.a (mon.0) 14379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:27:59 smithi121 bash[36902]: audit 2022-04-22T22:27:58.813159+0000 mon.a (mon.0) 14380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:00.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:27:59 smithi115 bash[46934]: audit 2022-04-22T22:27:58.813159+0000 mon.a (mon.0) 14380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:27:59 smithi115 bash[45495]: audit 2022-04-22T22:27:58.813159+0000 mon.a (mon.0) 14380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:00 smithi121 bash[36902]: cluster 2022-04-22T22:27:59.462842+0000 mgr.y (mgr.34107) 10294 : cluster [DBG] pgmap v10997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:00 smithi115 bash[45495]: cluster 2022-04-22T22:27:59.462842+0000 mgr.y (mgr.34107) 10294 : cluster [DBG] pgmap v10997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:00 smithi115 bash[46934]: cluster 2022-04-22T22:27:59.462842+0000 mgr.y (mgr.34107) 10294 : cluster [DBG] pgmap v10997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:01 smithi121 bash[36902]: audit 2022-04-22T22:28:01.437345+0000 mon.a (mon.0) 14381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:01 smithi115 bash[45495]: audit 2022-04-22T22:28:01.437345+0000 mon.a (mon.0) 14381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:01 smithi115 bash[46934]: audit 2022-04-22T22:28:01.437345+0000 mon.a (mon.0) 14381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:02 smithi121 bash[36902]: cluster 2022-04-22T22:28:01.463959+0000 mgr.y (mgr.34107) 10295 : cluster [DBG] pgmap v10998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:02 smithi115 bash[45495]: cluster 2022-04-22T22:28:01.463959+0000 mgr.y (mgr.34107) 10295 : cluster [DBG] pgmap v10998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:03.043 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:02.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:02 smithi115 bash[46934]: cluster 2022-04-22T22:28:01.463959+0000 mgr.y (mgr.34107) 10295 : cluster [DBG] pgmap v10998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:03.279 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:28:03.617 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:28:03.617 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 106s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 106s ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 106s ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 106s ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 2m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:28:03.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 106s ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 106s ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 2m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 106s ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 106s ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 106s ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 106s ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:28:03.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 2m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:28:03.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 2m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:28:03.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 2m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:28:03.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 2m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:28:03.620 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 2m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:28:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:03 smithi121 bash[36902]: audit 2022-04-22T22:28:02.734512+0000 mon.a (mon.0) 14382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:04.018 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:28:04.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:28:04.018 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:28:04.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:04.019 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:28:04.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:28:04.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:04.019 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:28:04.020 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:28:04.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:04.020 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:28:04.020 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:28:04.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:04.021 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:28:04.021 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:28:04.021 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:28:04.021 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:28:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:03 smithi115 bash[45495]: audit 2022-04-22T22:28:02.734512+0000 mon.a (mon.0) 14382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:03 smithi115 bash[46934]: audit 2022-04-22T22:28:02.734512+0000 mon.a (mon.0) 14382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:04 smithi121 bash[36902]: audit 2022-04-22T22:28:03.260815+0000 mgr.y (mgr.34107) 10296 : audit [DBG] from='client.46023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:04 smithi121 bash[36902]: cluster 2022-04-22T22:28:03.465082+0000 mgr.y (mgr.34107) 10297 : cluster [DBG] pgmap v10999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:04 smithi121 bash[36902]: audit 2022-04-22T22:28:03.599905+0000 mgr.y (mgr.34107) 10298 : audit [DBG] from='client.46029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:04 smithi121 bash[36902]: audit 2022-04-22T22:28:03.816411+0000 mon.a (mon.0) 14383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:04 smithi121 bash[36902]: audit 2022-04-22T22:28:04.014460+0000 mon.c (mon.1) 352 : audit [DBG] from='client.? 172.21.15.115:0/3999149732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:04.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[45495]: audit 2022-04-22T22:28:03.260815+0000 mgr.y (mgr.34107) 10296 : audit [DBG] from='client.46023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[45495]: cluster 2022-04-22T22:28:03.465082+0000 mgr.y (mgr.34107) 10297 : cluster [DBG] pgmap v10999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[45495]: audit 2022-04-22T22:28:03.599905+0000 mgr.y (mgr.34107) 10298 : audit [DBG] from='client.46029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[45495]: audit 2022-04-22T22:28:03.816411+0000 mon.a (mon.0) 14383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[45495]: audit 2022-04-22T22:28:04.014460+0000 mon.c (mon.1) 352 : audit [DBG] from='client.? 172.21.15.115:0/3999149732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[46934]: audit 2022-04-22T22:28:03.260815+0000 mgr.y (mgr.34107) 10296 : audit [DBG] from='client.46023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[46934]: cluster 2022-04-22T22:28:03.465082+0000 mgr.y (mgr.34107) 10297 : cluster [DBG] pgmap v10999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[46934]: audit 2022-04-22T22:28:03.599905+0000 mgr.y (mgr.34107) 10298 : audit [DBG] from='client.46029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[46934]: audit 2022-04-22T22:28:03.816411+0000 mon.a (mon.0) 14383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:05.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:04 smithi115 bash[46934]: audit 2022-04-22T22:28:04.014460+0000 mon.c (mon.1) 352 : audit [DBG] from='client.? 172.21.15.115:0/3999149732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:06 smithi121 bash[36902]: cluster 2022-04-22T22:28:05.466141+0000 mgr.y (mgr.34107) 10299 : cluster [DBG] pgmap v11000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:06 smithi121 bash[36902]: audit 2022-04-22T22:28:06.440424+0000 mon.a (mon.0) 14384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:06 smithi115 bash[45495]: cluster 2022-04-22T22:28:05.466141+0000 mgr.y (mgr.34107) 10299 : cluster [DBG] pgmap v11000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:06 smithi115 bash[45495]: audit 2022-04-22T22:28:06.440424+0000 mon.a (mon.0) 14384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:06 smithi115 bash[46934]: cluster 2022-04-22T22:28:05.466141+0000 mgr.y (mgr.34107) 10299 : cluster [DBG] pgmap v11000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:06 smithi115 bash[46934]: audit 2022-04-22T22:28:06.440424+0000 mon.a (mon.0) 14384 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:08 smithi121 bash[36902]: cluster 2022-04-22T22:28:07.467203+0000 mgr.y (mgr.34107) 10300 : cluster [DBG] pgmap v11001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:08 smithi121 bash[36902]: audit 2022-04-22T22:28:07.738169+0000 mon.a (mon.0) 14385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:08 smithi115 bash[45495]: cluster 2022-04-22T22:28:07.467203+0000 mgr.y (mgr.34107) 10300 : cluster [DBG] pgmap v11001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:08 smithi115 bash[45495]: audit 2022-04-22T22:28:07.738169+0000 mon.a (mon.0) 14385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:08 smithi115 bash[46934]: cluster 2022-04-22T22:28:07.467203+0000 mgr.y (mgr.34107) 10300 : cluster [DBG] pgmap v11001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:08 smithi115 bash[46934]: audit 2022-04-22T22:28:07.738169+0000 mon.a (mon.0) 14385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:09 smithi121 bash[36902]: audit 2022-04-22T22:28:08.819452+0000 mon.a (mon.0) 14386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:09 smithi115 bash[45495]: audit 2022-04-22T22:28:08.819452+0000 mon.a (mon.0) 14386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:09 smithi115 bash[46934]: audit 2022-04-22T22:28:08.819452+0000 mon.a (mon.0) 14386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:10 smithi121 bash[36902]: cluster 2022-04-22T22:28:09.468292+0000 mgr.y (mgr.34107) 10301 : cluster [DBG] pgmap v11002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:10 smithi115 bash[45495]: cluster 2022-04-22T22:28:09.468292+0000 mgr.y (mgr.34107) 10301 : cluster [DBG] pgmap v11002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:10 smithi115 bash[46934]: cluster 2022-04-22T22:28:09.468292+0000 mgr.y (mgr.34107) 10301 : cluster [DBG] pgmap v11002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:11 smithi121 bash[36902]: audit 2022-04-22T22:28:11.443657+0000 mon.a (mon.0) 14387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:11 smithi115 bash[45495]: audit 2022-04-22T22:28:11.443657+0000 mon.a (mon.0) 14387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:11 smithi115 bash[46934]: audit 2022-04-22T22:28:11.443657+0000 mon.a (mon.0) 14387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:12 smithi121 bash[36902]: cluster 2022-04-22T22:28:11.469351+0000 mgr.y (mgr.34107) 10302 : cluster [DBG] pgmap v11003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:12 smithi115 bash[46934]: cluster 2022-04-22T22:28:11.469351+0000 mgr.y (mgr.34107) 10302 : cluster [DBG] pgmap v11003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:12.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:12 smithi115 bash[45495]: cluster 2022-04-22T22:28:11.469351+0000 mgr.y (mgr.34107) 10302 : cluster [DBG] pgmap v11003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:13 smithi121 bash[36902]: audit 2022-04-22T22:28:12.741605+0000 mon.a (mon.0) 14388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:14.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:13 smithi115 bash[46934]: audit 2022-04-22T22:28:12.741605+0000 mon.a (mon.0) 14388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:13 smithi115 bash[45495]: audit 2022-04-22T22:28:12.741605+0000 mon.a (mon.0) 14388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:14 smithi121 bash[36902]: cluster 2022-04-22T22:28:13.470469+0000 mgr.y (mgr.34107) 10303 : cluster [DBG] pgmap v11004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:15.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:14 smithi121 bash[36902]: audit 2022-04-22T22:28:13.822309+0000 mon.a (mon.0) 14389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:14.658Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:14 smithi115 bash[45495]: cluster 2022-04-22T22:28:13.470469+0000 mgr.y (mgr.34107) 10303 : cluster [DBG] pgmap v11004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:14 smithi115 bash[45495]: audit 2022-04-22T22:28:13.822309+0000 mon.a (mon.0) 14389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:14 smithi115 bash[46934]: cluster 2022-04-22T22:28:13.470469+0000 mgr.y (mgr.34107) 10303 : cluster [DBG] pgmap v11004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:14 smithi115 bash[46934]: audit 2022-04-22T22:28:13.822309+0000 mon.a (mon.0) 14389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:16 smithi121 bash[36902]: cluster 2022-04-22T22:28:15.471603+0000 mgr.y (mgr.34107) 10304 : cluster [DBG] pgmap v11005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:16 smithi121 bash[36902]: audit 2022-04-22T22:28:16.446944+0000 mon.a (mon.0) 14390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:16 smithi115 bash[45495]: cluster 2022-04-22T22:28:15.471603+0000 mgr.y (mgr.34107) 10304 : cluster [DBG] pgmap v11005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:16 smithi115 bash[45495]: audit 2022-04-22T22:28:16.446944+0000 mon.a (mon.0) 14390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:16 smithi115 bash[46934]: cluster 2022-04-22T22:28:15.471603+0000 mgr.y (mgr.34107) 10304 : cluster [DBG] pgmap v11005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:16 smithi115 bash[46934]: audit 2022-04-22T22:28:16.446944+0000 mon.a (mon.0) 14390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:17 smithi121 bash[36902]: audit 2022-04-22T22:28:16.751450+0000 mon.a (mon.0) 14391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:28:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:17 smithi121 bash[36902]: audit 2022-04-22T22:28:16.751752+0000 mon.a (mon.0) 14392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:28:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:17 smithi115 bash[45495]: audit 2022-04-22T22:28:16.751450+0000 mon.a (mon.0) 14391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:28:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:17 smithi115 bash[45495]: audit 2022-04-22T22:28:16.751752+0000 mon.a (mon.0) 14392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:28:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:17 smithi115 bash[46934]: audit 2022-04-22T22:28:16.751450+0000 mon.a (mon.0) 14391 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:28:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:17 smithi115 bash[46934]: audit 2022-04-22T22:28:16.751752+0000 mon.a (mon.0) 14392 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:28:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: cluster 2022-04-22T22:28:17.472651+0000 mgr.y (mgr.34107) 10305 : cluster [DBG] pgmap v11006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: audit 2022-04-22T22:28:17.745488+0000 mon.a (mon.0) 14393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: audit 2022-04-22T22:28:17.787795+0000 mon.a (mon.0) 14394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: audit 2022-04-22T22:28:17.790865+0000 mon.a (mon.0) 14395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: audit 2022-04-22T22:28:17.792713+0000 mon.a (mon.0) 14396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:28:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:18 smithi121 bash[36902]: audit 2022-04-22T22:28:18.153400+0000 mon.a (mon.0) 14397 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:28:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: cluster 2022-04-22T22:28:17.472651+0000 mgr.y (mgr.34107) 10305 : cluster [DBG] pgmap v11006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: audit 2022-04-22T22:28:17.745488+0000 mon.a (mon.0) 14393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: audit 2022-04-22T22:28:17.787795+0000 mon.a (mon.0) 14394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: audit 2022-04-22T22:28:17.790865+0000 mon.a (mon.0) 14395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: audit 2022-04-22T22:28:17.792713+0000 mon.a (mon.0) 14396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[45495]: audit 2022-04-22T22:28:18.153400+0000 mon.a (mon.0) 14397 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: cluster 2022-04-22T22:28:17.472651+0000 mgr.y (mgr.34107) 10305 : cluster [DBG] pgmap v11006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: audit 2022-04-22T22:28:17.745488+0000 mon.a (mon.0) 14393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: audit 2022-04-22T22:28:17.787795+0000 mon.a (mon.0) 14394 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:28:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: audit 2022-04-22T22:28:17.790865+0000 mon.a (mon.0) 14395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:28:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: audit 2022-04-22T22:28:17.792713+0000 mon.a (mon.0) 14396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:28:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:18 smithi115 bash[46934]: audit 2022-04-22T22:28:18.153400+0000 mon.a (mon.0) 14397 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:28:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:19 smithi121 bash[36902]: audit 2022-04-22T22:28:18.825417+0000 mon.a (mon.0) 14398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:19 smithi115 bash[45495]: audit 2022-04-22T22:28:18.825417+0000 mon.a (mon.0) 14398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:19 smithi115 bash[46934]: audit 2022-04-22T22:28:18.825417+0000 mon.a (mon.0) 14398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:20 smithi121 bash[36902]: cluster 2022-04-22T22:28:19.473728+0000 mgr.y (mgr.34107) 10306 : cluster [DBG] pgmap v11007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:20 smithi115 bash[45495]: cluster 2022-04-22T22:28:19.473728+0000 mgr.y (mgr.34107) 10306 : cluster [DBG] pgmap v11007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:20 smithi115 bash[46934]: cluster 2022-04-22T22:28:19.473728+0000 mgr.y (mgr.34107) 10306 : cluster [DBG] pgmap v11007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:21 smithi121 bash[36902]: audit 2022-04-22T22:28:21.450162+0000 mon.a (mon.0) 14399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:21 smithi115 bash[45495]: audit 2022-04-22T22:28:21.450162+0000 mon.a (mon.0) 14399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:21 smithi115 bash[46934]: audit 2022-04-22T22:28:21.450162+0000 mon.a (mon.0) 14399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:22 smithi121 bash[36902]: cluster 2022-04-22T22:28:21.474808+0000 mgr.y (mgr.34107) 10307 : cluster [DBG] pgmap v11008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:23.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:22 smithi115 bash[45495]: cluster 2022-04-22T22:28:21.474808+0000 mgr.y (mgr.34107) 10307 : cluster [DBG] pgmap v11008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:22 smithi115 bash[46934]: cluster 2022-04-22T22:28:21.474808+0000 mgr.y (mgr.34107) 10307 : cluster [DBG] pgmap v11008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:22.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:23 smithi121 bash[36902]: audit 2022-04-22T22:28:22.748934+0000 mon.a (mon.0) 14400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:23 smithi115 bash[45495]: audit 2022-04-22T22:28:22.748934+0000 mon.a (mon.0) 14400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:23 smithi115 bash[46934]: audit 2022-04-22T22:28:22.748934+0000 mon.a (mon.0) 14400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:24 smithi121 bash[36902]: cluster 2022-04-22T22:28:23.475964+0000 mgr.y (mgr.34107) 10308 : cluster [DBG] pgmap v11009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:24 smithi121 bash[36902]: audit 2022-04-22T22:28:23.828454+0000 mon.a (mon.0) 14401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:24.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:24 smithi115 bash[45495]: cluster 2022-04-22T22:28:23.475964+0000 mgr.y (mgr.34107) 10308 : cluster [DBG] pgmap v11009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:24 smithi115 bash[45495]: audit 2022-04-22T22:28:23.828454+0000 mon.a (mon.0) 14401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:24 smithi115 bash[46934]: cluster 2022-04-22T22:28:23.475964+0000 mgr.y (mgr.34107) 10308 : cluster [DBG] pgmap v11009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:24 smithi115 bash[46934]: audit 2022-04-22T22:28:23.828454+0000 mon.a (mon.0) 14401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:25 smithi121 bash[36902]: cluster 2022-04-22T22:28:25.477008+0000 mgr.y (mgr.34107) 10309 : cluster [DBG] pgmap v11010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:25 smithi115 bash[45495]: cluster 2022-04-22T22:28:25.477008+0000 mgr.y (mgr.34107) 10309 : cluster [DBG] pgmap v11010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:25 smithi115 bash[46934]: cluster 2022-04-22T22:28:25.477008+0000 mgr.y (mgr.34107) 10309 : cluster [DBG] pgmap v11010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:26 smithi121 bash[36902]: audit 2022-04-22T22:28:26.453158+0000 mon.a (mon.0) 14402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:26 smithi115 bash[45495]: audit 2022-04-22T22:28:26.453158+0000 mon.a (mon.0) 14402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:26 smithi115 bash[46934]: audit 2022-04-22T22:28:26.453158+0000 mon.a (mon.0) 14402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:27 smithi121 bash[36902]: cluster 2022-04-22T22:28:27.478105+0000 mgr.y (mgr.34107) 10310 : cluster [DBG] pgmap v11011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:27 smithi115 bash[45495]: cluster 2022-04-22T22:28:27.478105+0000 mgr.y (mgr.34107) 10310 : cluster [DBG] pgmap v11011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:27 smithi115 bash[46934]: cluster 2022-04-22T22:28:27.478105+0000 mgr.y (mgr.34107) 10310 : cluster [DBG] pgmap v11011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:28 smithi121 bash[36902]: audit 2022-04-22T22:28:27.752470+0000 mon.a (mon.0) 14403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:28 smithi115 bash[45495]: audit 2022-04-22T22:28:27.752470+0000 mon.a (mon.0) 14403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:28 smithi115 bash[46934]: audit 2022-04-22T22:28:27.752470+0000 mon.a (mon.0) 14403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:29 smithi121 bash[36902]: audit 2022-04-22T22:28:28.831376+0000 mon.a (mon.0) 14404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:29 smithi121 bash[36902]: cluster 2022-04-22T22:28:29.479190+0000 mgr.y (mgr.34107) 10311 : cluster [DBG] pgmap v11012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:29 smithi115 bash[45495]: audit 2022-04-22T22:28:28.831376+0000 mon.a (mon.0) 14404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:29 smithi115 bash[45495]: cluster 2022-04-22T22:28:29.479190+0000 mgr.y (mgr.34107) 10311 : cluster [DBG] pgmap v11012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:29 smithi115 bash[46934]: audit 2022-04-22T22:28:28.831376+0000 mon.a (mon.0) 14404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:29 smithi115 bash[46934]: cluster 2022-04-22T22:28:29.479190+0000 mgr.y (mgr.34107) 10311 : cluster [DBG] pgmap v11012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:31 smithi121 bash[36902]: audit 2022-04-22T22:28:31.456585+0000 mon.a (mon.0) 14405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:31 smithi115 bash[45495]: audit 2022-04-22T22:28:31.456585+0000 mon.a (mon.0) 14405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:31 smithi115 bash[46934]: audit 2022-04-22T22:28:31.456585+0000 mon.a (mon.0) 14405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:32.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:32 smithi115 bash[45495]: cluster 2022-04-22T22:28:31.480255+0000 mgr.y (mgr.34107) 10312 : cluster [DBG] pgmap v11013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:32.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:32 smithi115 bash[46934]: cluster 2022-04-22T22:28:31.480255+0000 mgr.y (mgr.34107) 10312 : cluster [DBG] pgmap v11013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:32 smithi121 bash[36902]: cluster 2022-04-22T22:28:31.480255+0000 mgr.y (mgr.34107) 10312 : cluster [DBG] pgmap v11013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:32.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:33 smithi121 bash[36902]: audit 2022-04-22T22:28:32.756225+0000 mon.a (mon.0) 14406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:33 smithi115 bash[45495]: audit 2022-04-22T22:28:32.756225+0000 mon.a (mon.0) 14406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:33 smithi115 bash[46934]: audit 2022-04-22T22:28:32.756225+0000 mon.a (mon.0) 14406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:34.392 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:28:34.734 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:28:34.735 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:28:34.735 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:28:34.735 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:28:34.736 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:28:34.736 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:28:34.737 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:28:34.737 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:28:34.737 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (4h) 3m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:28:34.738 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4h) 2m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:28:34.738 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:28:34.738 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:28:34.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:28:34.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:28:34.739 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:28:34.740 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:28:34.740 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:28:34.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:28:34.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:28:34.742 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:28:34.742 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:28:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:34 smithi121 bash[36902]: cluster 2022-04-22T22:28:33.481397+0000 mgr.y (mgr.34107) 10313 : cluster [DBG] pgmap v11014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:35.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:34 smithi121 bash[36902]: audit 2022-04-22T22:28:33.834433+0000 mon.a (mon.0) 14407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:35.079 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:34.660Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:35.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:34 smithi115 bash[46934]: cluster 2022-04-22T22:28:33.481397+0000 mgr.y (mgr.34107) 10313 : cluster [DBG] pgmap v11014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:35.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:34 smithi115 bash[46934]: audit 2022-04-22T22:28:33.834433+0000 mon.a (mon.0) 14407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:35.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:34 smithi115 bash[45495]: cluster 2022-04-22T22:28:33.481397+0000 mgr.y (mgr.34107) 10313 : cluster [DBG] pgmap v11014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:35.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:34 smithi115 bash[45495]: audit 2022-04-22T22:28:33.834433+0000 mon.a (mon.0) 14407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:35.138 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:28:35.139 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:28:35.140 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:28:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:35 smithi121 bash[36902]: audit 2022-04-22T22:28:34.372753+0000 mgr.y (mgr.34107) 10314 : audit [DBG] from='client.55975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:35 smithi121 bash[36902]: audit 2022-04-22T22:28:35.135145+0000 mon.a (mon.0) 14408 : audit [DBG] from='client.? 172.21.15.115:0/4091360708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:35 smithi115 bash[45495]: audit 2022-04-22T22:28:34.372753+0000 mgr.y (mgr.34107) 10314 : audit [DBG] from='client.55975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:35 smithi115 bash[45495]: audit 2022-04-22T22:28:35.135145+0000 mon.a (mon.0) 14408 : audit [DBG] from='client.? 172.21.15.115:0/4091360708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:35 smithi115 bash[46934]: audit 2022-04-22T22:28:34.372753+0000 mgr.y (mgr.34107) 10314 : audit [DBG] from='client.55975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:35 smithi115 bash[46934]: audit 2022-04-22T22:28:35.135145+0000 mon.a (mon.0) 14408 : audit [DBG] from='client.? 172.21.15.115:0/4091360708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:28:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:36 smithi121 bash[36902]: audit 2022-04-22T22:28:34.708701+0000 mgr.y (mgr.34107) 10315 : audit [DBG] from='client.46047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:36 smithi121 bash[36902]: cluster 2022-04-22T22:28:35.482460+0000 mgr.y (mgr.34107) 10316 : cluster [DBG] pgmap v11015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:36 smithi121 bash[36902]: audit 2022-04-22T22:28:36.459891+0000 mon.a (mon.0) 14409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[45495]: audit 2022-04-22T22:28:34.708701+0000 mgr.y (mgr.34107) 10315 : audit [DBG] from='client.46047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[45495]: cluster 2022-04-22T22:28:35.482460+0000 mgr.y (mgr.34107) 10316 : cluster [DBG] pgmap v11015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[45495]: audit 2022-04-22T22:28:36.459891+0000 mon.a (mon.0) 14409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[46934]: audit 2022-04-22T22:28:34.708701+0000 mgr.y (mgr.34107) 10315 : audit [DBG] from='client.46047 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:28:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[46934]: cluster 2022-04-22T22:28:35.482460+0000 mgr.y (mgr.34107) 10316 : cluster [DBG] pgmap v11015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:37.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:36 smithi115 bash[46934]: audit 2022-04-22T22:28:36.459891+0000 mon.a (mon.0) 14409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:38 smithi121 bash[36902]: cluster 2022-04-22T22:28:37.483459+0000 mgr.y (mgr.34107) 10317 : cluster [DBG] pgmap v11016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:38 smithi121 bash[36902]: audit 2022-04-22T22:28:37.759832+0000 mon.a (mon.0) 14410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:38 smithi115 bash[45495]: cluster 2022-04-22T22:28:37.483459+0000 mgr.y (mgr.34107) 10317 : cluster [DBG] pgmap v11016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:38 smithi115 bash[45495]: audit 2022-04-22T22:28:37.759832+0000 mon.a (mon.0) 14410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:38 smithi115 bash[46934]: cluster 2022-04-22T22:28:37.483459+0000 mgr.y (mgr.34107) 10317 : cluster [DBG] pgmap v11016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:38 smithi115 bash[46934]: audit 2022-04-22T22:28:37.759832+0000 mon.a (mon.0) 14410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:39 smithi121 bash[36902]: audit 2022-04-22T22:28:38.837618+0000 mon.a (mon.0) 14411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:39 smithi115 bash[45495]: audit 2022-04-22T22:28:38.837618+0000 mon.a (mon.0) 14411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:39 smithi115 bash[46934]: audit 2022-04-22T22:28:38.837618+0000 mon.a (mon.0) 14411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:40 smithi121 bash[36902]: cluster 2022-04-22T22:28:39.484556+0000 mgr.y (mgr.34107) 10318 : cluster [DBG] pgmap v11017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:40 smithi115 bash[45495]: cluster 2022-04-22T22:28:39.484556+0000 mgr.y (mgr.34107) 10318 : cluster [DBG] pgmap v11017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:40 smithi115 bash[46934]: cluster 2022-04-22T22:28:39.484556+0000 mgr.y (mgr.34107) 10318 : cluster [DBG] pgmap v11017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:41 smithi121 bash[36902]: audit 2022-04-22T22:28:41.463088+0000 mon.a (mon.0) 14412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:41 smithi115 bash[45495]: audit 2022-04-22T22:28:41.463088+0000 mon.a (mon.0) 14412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:41 smithi115 bash[46934]: audit 2022-04-22T22:28:41.463088+0000 mon.a (mon.0) 14412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:42 smithi121 bash[36902]: cluster 2022-04-22T22:28:41.485594+0000 mgr.y (mgr.34107) 10319 : cluster [DBG] pgmap v11018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:42 smithi115 bash[45495]: cluster 2022-04-22T22:28:41.485594+0000 mgr.y (mgr.34107) 10319 : cluster [DBG] pgmap v11018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:43.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:42.811Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:43.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:42 smithi115 bash[46934]: cluster 2022-04-22T22:28:41.485594+0000 mgr.y (mgr.34107) 10319 : cluster [DBG] pgmap v11018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:43 smithi121 bash[36902]: audit 2022-04-22T22:28:42.763670+0000 mon.a (mon.0) 14413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:43 smithi115 bash[45495]: audit 2022-04-22T22:28:42.763670+0000 mon.a (mon.0) 14413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:43 smithi115 bash[46934]: audit 2022-04-22T22:28:42.763670+0000 mon.a (mon.0) 14413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:44 smithi121 bash[36902]: cluster 2022-04-22T22:28:43.486723+0000 mgr.y (mgr.34107) 10320 : cluster [DBG] pgmap v11019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:44 smithi121 bash[36902]: audit 2022-04-22T22:28:43.840702+0000 mon.a (mon.0) 14414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:44.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:44 smithi115 bash[45495]: cluster 2022-04-22T22:28:43.486723+0000 mgr.y (mgr.34107) 10320 : cluster [DBG] pgmap v11019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:44 smithi115 bash[45495]: audit 2022-04-22T22:28:43.840702+0000 mon.a (mon.0) 14414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:44 smithi115 bash[46934]: cluster 2022-04-22T22:28:43.486723+0000 mgr.y (mgr.34107) 10320 : cluster [DBG] pgmap v11019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:44 smithi115 bash[46934]: audit 2022-04-22T22:28:43.840702+0000 mon.a (mon.0) 14414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:46 smithi121 bash[36902]: cluster 2022-04-22T22:28:45.487859+0000 mgr.y (mgr.34107) 10321 : cluster [DBG] pgmap v11020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:46 smithi121 bash[36902]: audit 2022-04-22T22:28:46.466543+0000 mon.a (mon.0) 14415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:46 smithi115 bash[45495]: cluster 2022-04-22T22:28:45.487859+0000 mgr.y (mgr.34107) 10321 : cluster [DBG] pgmap v11020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:46 smithi115 bash[45495]: audit 2022-04-22T22:28:46.466543+0000 mon.a (mon.0) 14415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:46 smithi115 bash[46934]: cluster 2022-04-22T22:28:45.487859+0000 mgr.y (mgr.34107) 10321 : cluster [DBG] pgmap v11020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:46 smithi115 bash[46934]: audit 2022-04-22T22:28:46.466543+0000 mon.a (mon.0) 14415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:48 smithi121 bash[36902]: cluster 2022-04-22T22:28:47.489003+0000 mgr.y (mgr.34107) 10322 : cluster [DBG] pgmap v11021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:48 smithi121 bash[36902]: audit 2022-04-22T22:28:47.767106+0000 mon.a (mon.0) 14416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:48 smithi115 bash[45495]: cluster 2022-04-22T22:28:47.489003+0000 mgr.y (mgr.34107) 10322 : cluster [DBG] pgmap v11021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:48 smithi115 bash[45495]: audit 2022-04-22T22:28:47.767106+0000 mon.a (mon.0) 14416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:48 smithi115 bash[46934]: cluster 2022-04-22T22:28:47.489003+0000 mgr.y (mgr.34107) 10322 : cluster [DBG] pgmap v11021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:48 smithi115 bash[46934]: audit 2022-04-22T22:28:47.767106+0000 mon.a (mon.0) 14416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:49 smithi121 bash[36902]: audit 2022-04-22T22:28:48.843746+0000 mon.a (mon.0) 14417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:49 smithi115 bash[45495]: audit 2022-04-22T22:28:48.843746+0000 mon.a (mon.0) 14417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:49 smithi115 bash[46934]: audit 2022-04-22T22:28:48.843746+0000 mon.a (mon.0) 14417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:50 smithi121 bash[36902]: cluster 2022-04-22T22:28:49.490238+0000 mgr.y (mgr.34107) 10323 : cluster [DBG] pgmap v11022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:50 smithi115 bash[46934]: cluster 2022-04-22T22:28:49.490238+0000 mgr.y (mgr.34107) 10323 : cluster [DBG] pgmap v11022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:50 smithi115 bash[45495]: cluster 2022-04-22T22:28:49.490238+0000 mgr.y (mgr.34107) 10323 : cluster [DBG] pgmap v11022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:51 smithi121 bash[36902]: audit 2022-04-22T22:28:51.469312+0000 mon.a (mon.0) 14418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:51 smithi115 bash[45495]: audit 2022-04-22T22:28:51.469312+0000 mon.a (mon.0) 14418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:51 smithi115 bash[46934]: audit 2022-04-22T22:28:51.469312+0000 mon.a (mon.0) 14418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:52 smithi121 bash[36902]: cluster 2022-04-22T22:28:51.491314+0000 mgr.y (mgr.34107) 10324 : cluster [DBG] pgmap v11023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:53.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:52.811Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:53.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:52 smithi115 bash[45495]: cluster 2022-04-22T22:28:51.491314+0000 mgr.y (mgr.34107) 10324 : cluster [DBG] pgmap v11023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:53.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:52 smithi115 bash[46934]: cluster 2022-04-22T22:28:51.491314+0000 mgr.y (mgr.34107) 10324 : cluster [DBG] pgmap v11023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:53 smithi121 bash[36902]: audit 2022-04-22T22:28:52.770666+0000 mon.a (mon.0) 14419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:53 smithi115 bash[45495]: audit 2022-04-22T22:28:52.770666+0000 mon.a (mon.0) 14419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:53 smithi115 bash[46934]: audit 2022-04-22T22:28:52.770666+0000 mon.a (mon.0) 14419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:54 smithi121 bash[36902]: cluster 2022-04-22T22:28:53.492434+0000 mgr.y (mgr.34107) 10325 : cluster [DBG] pgmap v11024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:54 smithi121 bash[36902]: audit 2022-04-22T22:28:53.846420+0000 mon.a (mon.0) 14420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:28:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:28:54.659Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:28:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:54 smithi115 bash[45495]: cluster 2022-04-22T22:28:53.492434+0000 mgr.y (mgr.34107) 10325 : cluster [DBG] pgmap v11024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:54 smithi115 bash[45495]: audit 2022-04-22T22:28:53.846420+0000 mon.a (mon.0) 14420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:54 smithi115 bash[46934]: cluster 2022-04-22T22:28:53.492434+0000 mgr.y (mgr.34107) 10325 : cluster [DBG] pgmap v11024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:54 smithi115 bash[46934]: audit 2022-04-22T22:28:53.846420+0000 mon.a (mon.0) 14420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:28:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:56 smithi121 bash[36902]: cluster 2022-04-22T22:28:55.493506+0000 mgr.y (mgr.34107) 10326 : cluster [DBG] pgmap v11025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:56 smithi121 bash[36902]: audit 2022-04-22T22:28:56.472511+0000 mon.a (mon.0) 14421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:56 smithi115 bash[45495]: cluster 2022-04-22T22:28:55.493506+0000 mgr.y (mgr.34107) 10326 : cluster [DBG] pgmap v11025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:56 smithi115 bash[45495]: audit 2022-04-22T22:28:56.472511+0000 mon.a (mon.0) 14421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:56 smithi115 bash[46934]: cluster 2022-04-22T22:28:55.493506+0000 mgr.y (mgr.34107) 10326 : cluster [DBG] pgmap v11025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:56 smithi115 bash[46934]: audit 2022-04-22T22:28:56.472511+0000 mon.a (mon.0) 14421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:28:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:58 smithi121 bash[36902]: cluster 2022-04-22T22:28:57.494590+0000 mgr.y (mgr.34107) 10327 : cluster [DBG] pgmap v11026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:58 smithi121 bash[36902]: audit 2022-04-22T22:28:57.774600+0000 mon.a (mon.0) 14422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:58 smithi115 bash[45495]: cluster 2022-04-22T22:28:57.494590+0000 mgr.y (mgr.34107) 10327 : cluster [DBG] pgmap v11026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:58 smithi115 bash[45495]: audit 2022-04-22T22:28:57.774600+0000 mon.a (mon.0) 14422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:28:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:58 smithi115 bash[46934]: cluster 2022-04-22T22:28:57.494590+0000 mgr.y (mgr.34107) 10327 : cluster [DBG] pgmap v11026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:28:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:58 smithi115 bash[46934]: audit 2022-04-22T22:28:57.774600+0000 mon.a (mon.0) 14422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:28:59 smithi121 bash[36902]: audit 2022-04-22T22:28:58.848949+0000 mon.a (mon.0) 14423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:00.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:28:59 smithi115 bash[45495]: audit 2022-04-22T22:28:58.848949+0000 mon.a (mon.0) 14423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:28:59 smithi115 bash[46934]: audit 2022-04-22T22:28:58.848949+0000 mon.a (mon.0) 14423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:00 smithi121 bash[36902]: cluster 2022-04-22T22:28:59.495728+0000 mgr.y (mgr.34107) 10328 : cluster [DBG] pgmap v11027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:00 smithi115 bash[45495]: cluster 2022-04-22T22:28:59.495728+0000 mgr.y (mgr.34107) 10328 : cluster [DBG] pgmap v11027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:00 smithi115 bash[46934]: cluster 2022-04-22T22:28:59.495728+0000 mgr.y (mgr.34107) 10328 : cluster [DBG] pgmap v11027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:01 smithi121 bash[36902]: audit 2022-04-22T22:29:01.475452+0000 mon.a (mon.0) 14424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:01 smithi115 bash[45495]: audit 2022-04-22T22:29:01.475452+0000 mon.a (mon.0) 14424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:01 smithi115 bash[46934]: audit 2022-04-22T22:29:01.475452+0000 mon.a (mon.0) 14424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:02 smithi121 bash[36902]: cluster 2022-04-22T22:29:01.496736+0000 mgr.y (mgr.34107) 10329 : cluster [DBG] pgmap v11028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:02.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:02 smithi115 bash[45495]: cluster 2022-04-22T22:29:01.496736+0000 mgr.y (mgr.34107) 10329 : cluster [DBG] pgmap v11028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:02 smithi115 bash[46934]: cluster 2022-04-22T22:29:01.496736+0000 mgr.y (mgr.34107) 10329 : cluster [DBG] pgmap v11028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:03 smithi121 bash[36902]: audit 2022-04-22T22:29:02.778454+0000 mon.a (mon.0) 14425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:03 smithi115 bash[45495]: audit 2022-04-22T22:29:02.778454+0000 mon.a (mon.0) 14425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:03 smithi115 bash[46934]: audit 2022-04-22T22:29:02.778454+0000 mon.a (mon.0) 14425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:04 smithi121 bash[36902]: cluster 2022-04-22T22:29:03.497865+0000 mgr.y (mgr.34107) 10330 : cluster [DBG] pgmap v11029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:04 smithi121 bash[36902]: audit 2022-04-22T22:29:03.852085+0000 mon.a (mon.0) 14426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:04.661Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:04 smithi115 bash[45495]: cluster 2022-04-22T22:29:03.497865+0000 mgr.y (mgr.34107) 10330 : cluster [DBG] pgmap v11029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:04 smithi115 bash[45495]: audit 2022-04-22T22:29:03.852085+0000 mon.a (mon.0) 14426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:04 smithi115 bash[46934]: cluster 2022-04-22T22:29:03.497865+0000 mgr.y (mgr.34107) 10330 : cluster [DBG] pgmap v11029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:04 smithi115 bash[46934]: audit 2022-04-22T22:29:03.852085+0000 mon.a (mon.0) 14426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:05.511 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:29:05.860 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:29:05.860 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 2m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:29:05.860 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 2m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:29:05.861 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 2m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 3m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 2m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 2m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 2m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:29:05.862 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 2m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:29:05.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 3m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:29:05.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 3m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:29:05.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 3m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:29:05.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 3m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:29:05.863 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 3m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:29:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:05 smithi121 bash[36902]: audit 2022-04-22T22:29:05.492154+0000 mgr.y (mgr.34107) 10331 : audit [DBG] from='client.55993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:05 smithi121 bash[36902]: cluster 2022-04-22T22:29:05.498899+0000 mgr.y (mgr.34107) 10332 : cluster [DBG] pgmap v11030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:06.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:05 smithi115 bash[46934]: audit 2022-04-22T22:29:05.492154+0000 mgr.y (mgr.34107) 10331 : audit [DBG] from='client.55993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:06.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:05 smithi115 bash[46934]: cluster 2022-04-22T22:29:05.498899+0000 mgr.y (mgr.34107) 10332 : cluster [DBG] pgmap v11030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:05 smithi115 bash[45495]: audit 2022-04-22T22:29:05.492154+0000 mgr.y (mgr.34107) 10331 : audit [DBG] from='client.55993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:05 smithi115 bash[45495]: cluster 2022-04-22T22:29:05.498899+0000 mgr.y (mgr.34107) 10332 : cluster [DBG] pgmap v11030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:06.265 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:29:06.265 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:29:06.266 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:29:06.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:06.266 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:29:06.266 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:29:06.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:06.267 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:29:06.267 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:29:06.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:06.267 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:29:06.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:29:06.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:06.268 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:29:06.268 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:29:06.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:29:06.269 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:06 smithi121 bash[36902]: audit 2022-04-22T22:29:05.837182+0000 mgr.y (mgr.34107) 10333 : audit [DBG] from='client.55999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:06 smithi121 bash[36902]: audit 2022-04-22T22:29:06.261936+0000 mon.c (mon.1) 353 : audit [DBG] from='client.? 172.21.15.115:0/2423240829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:06 smithi121 bash[36902]: audit 2022-04-22T22:29:06.478542+0000 mon.a (mon.0) 14427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[46934]: audit 2022-04-22T22:29:05.837182+0000 mgr.y (mgr.34107) 10333 : audit [DBG] from='client.55999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[46934]: audit 2022-04-22T22:29:06.261936+0000 mon.c (mon.1) 353 : audit [DBG] from='client.? 172.21.15.115:0/2423240829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[46934]: audit 2022-04-22T22:29:06.478542+0000 mon.a (mon.0) 14427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[45495]: audit 2022-04-22T22:29:05.837182+0000 mgr.y (mgr.34107) 10333 : audit [DBG] from='client.55999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[45495]: audit 2022-04-22T22:29:06.261936+0000 mon.c (mon.1) 353 : audit [DBG] from='client.? 172.21.15.115:0/2423240829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:07.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:06 smithi115 bash[45495]: audit 2022-04-22T22:29:06.478542+0000 mon.a (mon.0) 14427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:07 smithi121 bash[36902]: cluster 2022-04-22T22:29:07.500000+0000 mgr.y (mgr.34107) 10334 : cluster [DBG] pgmap v11031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:07 smithi115 bash[45495]: cluster 2022-04-22T22:29:07.500000+0000 mgr.y (mgr.34107) 10334 : cluster [DBG] pgmap v11031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:07 smithi115 bash[46934]: cluster 2022-04-22T22:29:07.500000+0000 mgr.y (mgr.34107) 10334 : cluster [DBG] pgmap v11031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:08 smithi121 bash[36902]: audit 2022-04-22T22:29:07.782379+0000 mon.a (mon.0) 14428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:08 smithi115 bash[45495]: audit 2022-04-22T22:29:07.782379+0000 mon.a (mon.0) 14428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:08 smithi115 bash[46934]: audit 2022-04-22T22:29:07.782379+0000 mon.a (mon.0) 14428 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:09 smithi121 bash[36902]: audit 2022-04-22T22:29:08.854899+0000 mon.a (mon.0) 14429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:09 smithi121 bash[36902]: cluster 2022-04-22T22:29:09.501046+0000 mgr.y (mgr.34107) 10335 : cluster [DBG] pgmap v11032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:09 smithi115 bash[45495]: audit 2022-04-22T22:29:08.854899+0000 mon.a (mon.0) 14429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:09 smithi115 bash[45495]: cluster 2022-04-22T22:29:09.501046+0000 mgr.y (mgr.34107) 10335 : cluster [DBG] pgmap v11032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:09 smithi115 bash[46934]: audit 2022-04-22T22:29:08.854899+0000 mon.a (mon.0) 14429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:09 smithi115 bash[46934]: cluster 2022-04-22T22:29:09.501046+0000 mgr.y (mgr.34107) 10335 : cluster [DBG] pgmap v11032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:11 smithi115 bash[45495]: audit 2022-04-22T22:29:11.481652+0000 mon.a (mon.0) 14430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:11 smithi115 bash[46934]: audit 2022-04-22T22:29:11.481652+0000 mon.a (mon.0) 14430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:11 smithi121 bash[36902]: audit 2022-04-22T22:29:11.481652+0000 mon.a (mon.0) 14430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:12 smithi121 bash[36902]: cluster 2022-04-22T22:29:11.502180+0000 mgr.y (mgr.34107) 10336 : cluster [DBG] pgmap v11033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:12 smithi115 bash[45495]: cluster 2022-04-22T22:29:11.502180+0000 mgr.y (mgr.34107) 10336 : cluster [DBG] pgmap v11033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:12 smithi115 bash[46934]: cluster 2022-04-22T22:29:11.502180+0000 mgr.y (mgr.34107) 10336 : cluster [DBG] pgmap v11033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:12.811Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:13 smithi121 bash[36902]: audit 2022-04-22T22:29:12.786014+0000 mon.a (mon.0) 14431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:13 smithi115 bash[45495]: audit 2022-04-22T22:29:12.786014+0000 mon.a (mon.0) 14431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:13 smithi115 bash[46934]: audit 2022-04-22T22:29:12.786014+0000 mon.a (mon.0) 14431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:14 smithi121 bash[36902]: cluster 2022-04-22T22:29:13.503333+0000 mgr.y (mgr.34107) 10337 : cluster [DBG] pgmap v11034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:14 smithi121 bash[36902]: audit 2022-04-22T22:29:13.858131+0000 mon.a (mon.0) 14432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:14 smithi115 bash[45495]: cluster 2022-04-22T22:29:13.503333+0000 mgr.y (mgr.34107) 10337 : cluster [DBG] pgmap v11034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:14 smithi115 bash[45495]: audit 2022-04-22T22:29:13.858131+0000 mon.a (mon.0) 14432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:14 smithi115 bash[46934]: cluster 2022-04-22T22:29:13.503333+0000 mgr.y (mgr.34107) 10337 : cluster [DBG] pgmap v11034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:14 smithi115 bash[46934]: audit 2022-04-22T22:29:13.858131+0000 mon.a (mon.0) 14432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:14.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:16 smithi121 bash[36902]: cluster 2022-04-22T22:29:15.504430+0000 mgr.y (mgr.34107) 10338 : cluster [DBG] pgmap v11035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:16 smithi121 bash[36902]: audit 2022-04-22T22:29:16.484794+0000 mon.a (mon.0) 14433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:16 smithi115 bash[45495]: cluster 2022-04-22T22:29:15.504430+0000 mgr.y (mgr.34107) 10338 : cluster [DBG] pgmap v11035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:16 smithi115 bash[45495]: audit 2022-04-22T22:29:16.484794+0000 mon.a (mon.0) 14433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:16 smithi115 bash[46934]: cluster 2022-04-22T22:29:15.504430+0000 mgr.y (mgr.34107) 10338 : cluster [DBG] pgmap v11035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:16 smithi115 bash[46934]: audit 2022-04-22T22:29:16.484794+0000 mon.a (mon.0) 14433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:17 smithi121 bash[36902]: audit 2022-04-22T22:29:16.760119+0000 mon.a (mon.0) 14434 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:29:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:17 smithi121 bash[36902]: audit 2022-04-22T22:29:16.760768+0000 mon.a (mon.0) 14435 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:29:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:17 smithi115 bash[45495]: audit 2022-04-22T22:29:16.760119+0000 mon.a (mon.0) 14434 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:29:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:17 smithi115 bash[45495]: audit 2022-04-22T22:29:16.760768+0000 mon.a (mon.0) 14435 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:29:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:17 smithi115 bash[46934]: audit 2022-04-22T22:29:16.760119+0000 mon.a (mon.0) 14434 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:29:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:17 smithi115 bash[46934]: audit 2022-04-22T22:29:16.760768+0000 mon.a (mon.0) 14435 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:29:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: cluster 2022-04-22T22:29:17.505667+0000 mgr.y (mgr.34107) 10339 : cluster [DBG] pgmap v11036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: audit 2022-04-22T22:29:17.789808+0000 mon.a (mon.0) 14436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: audit 2022-04-22T22:29:18.161635+0000 mon.a (mon.0) 14437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: audit 2022-04-22T22:29:18.164547+0000 mon.a (mon.0) 14438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: audit 2022-04-22T22:29:18.166367+0000 mon.a (mon.0) 14439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:18 smithi121 bash[36902]: audit 2022-04-22T22:29:18.537895+0000 mon.a (mon.0) 14440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:29:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: cluster 2022-04-22T22:29:17.505667+0000 mgr.y (mgr.34107) 10339 : cluster [DBG] pgmap v11036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: audit 2022-04-22T22:29:17.789808+0000 mon.a (mon.0) 14436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: audit 2022-04-22T22:29:18.161635+0000 mon.a (mon.0) 14437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: audit 2022-04-22T22:29:18.164547+0000 mon.a (mon.0) 14438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: audit 2022-04-22T22:29:18.166367+0000 mon.a (mon.0) 14439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[45495]: audit 2022-04-22T22:29:18.537895+0000 mon.a (mon.0) 14440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: cluster 2022-04-22T22:29:17.505667+0000 mgr.y (mgr.34107) 10339 : cluster [DBG] pgmap v11036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: audit 2022-04-22T22:29:17.789808+0000 mon.a (mon.0) 14436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: audit 2022-04-22T22:29:18.161635+0000 mon.a (mon.0) 14437 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: audit 2022-04-22T22:29:18.164547+0000 mon.a (mon.0) 14438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: audit 2022-04-22T22:29:18.166367+0000 mon.a (mon.0) 14439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:18 smithi115 bash[46934]: audit 2022-04-22T22:29:18.537895+0000 mon.a (mon.0) 14440 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:29:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:19 smithi121 bash[36902]: audit 2022-04-22T22:29:18.860860+0000 mon.a (mon.0) 14441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:19 smithi115 bash[45495]: audit 2022-04-22T22:29:18.860860+0000 mon.a (mon.0) 14441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:19 smithi115 bash[46934]: audit 2022-04-22T22:29:18.860860+0000 mon.a (mon.0) 14441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:20 smithi121 bash[36902]: cluster 2022-04-22T22:29:19.506747+0000 mgr.y (mgr.34107) 10340 : cluster [DBG] pgmap v11037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:20 smithi115 bash[45495]: cluster 2022-04-22T22:29:19.506747+0000 mgr.y (mgr.34107) 10340 : cluster [DBG] pgmap v11037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:20 smithi115 bash[46934]: cluster 2022-04-22T22:29:19.506747+0000 mgr.y (mgr.34107) 10340 : cluster [DBG] pgmap v11037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:21 smithi121 bash[36902]: audit 2022-04-22T22:29:21.487955+0000 mon.a (mon.0) 14442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:21 smithi115 bash[45495]: audit 2022-04-22T22:29:21.487955+0000 mon.a (mon.0) 14442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:21 smithi115 bash[46934]: audit 2022-04-22T22:29:21.487955+0000 mon.a (mon.0) 14442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:22 smithi121 bash[36902]: cluster 2022-04-22T22:29:21.507780+0000 mgr.y (mgr.34107) 10341 : cluster [DBG] pgmap v11038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:23.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:22 smithi115 bash[45495]: cluster 2022-04-22T22:29:21.507780+0000 mgr.y (mgr.34107) 10341 : cluster [DBG] pgmap v11038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:22.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:22 smithi115 bash[46934]: cluster 2022-04-22T22:29:21.507780+0000 mgr.y (mgr.34107) 10341 : cluster [DBG] pgmap v11038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:23 smithi121 bash[36902]: audit 2022-04-22T22:29:22.793540+0000 mon.a (mon.0) 14443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:23 smithi115 bash[45495]: audit 2022-04-22T22:29:22.793540+0000 mon.a (mon.0) 14443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:23 smithi115 bash[46934]: audit 2022-04-22T22:29:22.793540+0000 mon.a (mon.0) 14443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:24 smithi121 bash[36902]: cluster 2022-04-22T22:29:23.508872+0000 mgr.y (mgr.34107) 10342 : cluster [DBG] pgmap v11039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:25.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:24 smithi121 bash[36902]: audit 2022-04-22T22:29:23.863918+0000 mon.a (mon.0) 14444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:24.662Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:24 smithi115 bash[45495]: cluster 2022-04-22T22:29:23.508872+0000 mgr.y (mgr.34107) 10342 : cluster [DBG] pgmap v11039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:24 smithi115 bash[45495]: audit 2022-04-22T22:29:23.863918+0000 mon.a (mon.0) 14444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:24 smithi115 bash[46934]: cluster 2022-04-22T22:29:23.508872+0000 mgr.y (mgr.34107) 10342 : cluster [DBG] pgmap v11039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:24 smithi115 bash[46934]: audit 2022-04-22T22:29:23.863918+0000 mon.a (mon.0) 14444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:26 smithi121 bash[36902]: cluster 2022-04-22T22:29:25.510047+0000 mgr.y (mgr.34107) 10343 : cluster [DBG] pgmap v11040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:26 smithi121 bash[36902]: audit 2022-04-22T22:29:26.490955+0000 mon.a (mon.0) 14445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:26 smithi115 bash[45495]: cluster 2022-04-22T22:29:25.510047+0000 mgr.y (mgr.34107) 10343 : cluster [DBG] pgmap v11040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:26 smithi115 bash[45495]: audit 2022-04-22T22:29:26.490955+0000 mon.a (mon.0) 14445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:26 smithi115 bash[46934]: cluster 2022-04-22T22:29:25.510047+0000 mgr.y (mgr.34107) 10343 : cluster [DBG] pgmap v11040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:26 smithi115 bash[46934]: audit 2022-04-22T22:29:26.490955+0000 mon.a (mon.0) 14445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:28 smithi121 bash[36902]: cluster 2022-04-22T22:29:27.511160+0000 mgr.y (mgr.34107) 10344 : cluster [DBG] pgmap v11041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:28 smithi121 bash[36902]: audit 2022-04-22T22:29:27.797473+0000 mon.a (mon.0) 14446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:28 smithi115 bash[45495]: cluster 2022-04-22T22:29:27.511160+0000 mgr.y (mgr.34107) 10344 : cluster [DBG] pgmap v11041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:28 smithi115 bash[45495]: audit 2022-04-22T22:29:27.797473+0000 mon.a (mon.0) 14446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:28 smithi115 bash[46934]: cluster 2022-04-22T22:29:27.511160+0000 mgr.y (mgr.34107) 10344 : cluster [DBG] pgmap v11041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:28 smithi115 bash[46934]: audit 2022-04-22T22:29:27.797473+0000 mon.a (mon.0) 14446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:29 smithi121 bash[36902]: audit 2022-04-22T22:29:28.867054+0000 mon.a (mon.0) 14447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:29 smithi115 bash[45495]: audit 2022-04-22T22:29:28.867054+0000 mon.a (mon.0) 14447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:29 smithi115 bash[46934]: audit 2022-04-22T22:29:28.867054+0000 mon.a (mon.0) 14447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:30 smithi121 bash[36902]: cluster 2022-04-22T22:29:29.512266+0000 mgr.y (mgr.34107) 10345 : cluster [DBG] pgmap v11042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:30 smithi115 bash[45495]: cluster 2022-04-22T22:29:29.512266+0000 mgr.y (mgr.34107) 10345 : cluster [DBG] pgmap v11042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:30 smithi115 bash[46934]: cluster 2022-04-22T22:29:29.512266+0000 mgr.y (mgr.34107) 10345 : cluster [DBG] pgmap v11042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:31 smithi121 bash[36902]: audit 2022-04-22T22:29:31.494314+0000 mon.a (mon.0) 14448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:32.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:31 smithi115 bash[46934]: audit 2022-04-22T22:29:31.494314+0000 mon.a (mon.0) 14448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:31 smithi115 bash[45495]: audit 2022-04-22T22:29:31.494314+0000 mon.a (mon.0) 14448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:32 smithi121 bash[36902]: cluster 2022-04-22T22:29:31.513356+0000 mgr.y (mgr.34107) 10346 : cluster [DBG] pgmap v11043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:33.114 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:32.813Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:33.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:32 smithi115 bash[45495]: cluster 2022-04-22T22:29:31.513356+0000 mgr.y (mgr.34107) 10346 : cluster [DBG] pgmap v11043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:32 smithi115 bash[46934]: cluster 2022-04-22T22:29:31.513356+0000 mgr.y (mgr.34107) 10346 : cluster [DBG] pgmap v11043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:33 smithi121 bash[36902]: audit 2022-04-22T22:29:32.801321+0000 mon.a (mon.0) 14449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:33 smithi115 bash[45495]: audit 2022-04-22T22:29:32.801321+0000 mon.a (mon.0) 14449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:33 smithi115 bash[46934]: audit 2022-04-22T22:29:32.801321+0000 mon.a (mon.0) 14449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:34 smithi121 bash[36902]: cluster 2022-04-22T22:29:33.514527+0000 mgr.y (mgr.34107) 10347 : cluster [DBG] pgmap v11044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:34 smithi121 bash[36902]: audit 2022-04-22T22:29:33.870037+0000 mon.a (mon.0) 14450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:34.664Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:34 smithi115 bash[45495]: cluster 2022-04-22T22:29:33.514527+0000 mgr.y (mgr.34107) 10347 : cluster [DBG] pgmap v11044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:34 smithi115 bash[45495]: audit 2022-04-22T22:29:33.870037+0000 mon.a (mon.0) 14450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:34 smithi115 bash[46934]: cluster 2022-04-22T22:29:33.514527+0000 mgr.y (mgr.34107) 10347 : cluster [DBG] pgmap v11044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:34 smithi115 bash[46934]: audit 2022-04-22T22:29:33.870037+0000 mon.a (mon.0) 14450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:36.635 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:29:36.982 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:29:36.982 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:29:36.983 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:29:36.984 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:29:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:36 smithi121 bash[36902]: cluster 2022-04-22T22:29:35.515650+0000 mgr.y (mgr.34107) 10348 : cluster [DBG] pgmap v11045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:36 smithi121 bash[36902]: audit 2022-04-22T22:29:36.497383+0000 mon.a (mon.0) 14451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:36 smithi115 bash[45495]: cluster 2022-04-22T22:29:35.515650+0000 mgr.y (mgr.34107) 10348 : cluster [DBG] pgmap v11045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:36 smithi115 bash[45495]: audit 2022-04-22T22:29:36.497383+0000 mon.a (mon.0) 14451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:36 smithi115 bash[46934]: cluster 2022-04-22T22:29:35.515650+0000 mgr.y (mgr.34107) 10348 : cluster [DBG] pgmap v11045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:36 smithi115 bash[46934]: audit 2022-04-22T22:29:36.497383+0000 mon.a (mon.0) 14451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:37.384 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:29:37.385 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:29:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:37 smithi121 bash[36902]: audit 2022-04-22T22:29:36.616773+0000 mgr.y (mgr.34107) 10349 : audit [DBG] from='client.56011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:37 smithi121 bash[36902]: audit 2022-04-22T22:29:37.380162+0000 mon.a (mon.0) 14452 : audit [DBG] from='client.? 172.21.15.115:0/1131075026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:37 smithi115 bash[45495]: audit 2022-04-22T22:29:36.616773+0000 mgr.y (mgr.34107) 10349 : audit [DBG] from='client.56011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:37 smithi115 bash[45495]: audit 2022-04-22T22:29:37.380162+0000 mon.a (mon.0) 14452 : audit [DBG] from='client.? 172.21.15.115:0/1131075026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:37 smithi115 bash[46934]: audit 2022-04-22T22:29:36.616773+0000 mgr.y (mgr.34107) 10349 : audit [DBG] from='client.56011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:37 smithi115 bash[46934]: audit 2022-04-22T22:29:37.380162+0000 mon.a (mon.0) 14452 : audit [DBG] from='client.? 172.21.15.115:0/1131075026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:29:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:38 smithi121 bash[36902]: audit 2022-04-22T22:29:36.960277+0000 mgr.y (mgr.34107) 10350 : audit [DBG] from='client.56017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:38 smithi121 bash[36902]: cluster 2022-04-22T22:29:37.516758+0000 mgr.y (mgr.34107) 10351 : cluster [DBG] pgmap v11046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:38 smithi121 bash[36902]: audit 2022-04-22T22:29:37.804846+0000 mon.a (mon.0) 14453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[46934]: audit 2022-04-22T22:29:36.960277+0000 mgr.y (mgr.34107) 10350 : audit [DBG] from='client.56017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[46934]: cluster 2022-04-22T22:29:37.516758+0000 mgr.y (mgr.34107) 10351 : cluster [DBG] pgmap v11046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[46934]: audit 2022-04-22T22:29:37.804846+0000 mon.a (mon.0) 14453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[45495]: audit 2022-04-22T22:29:36.960277+0000 mgr.y (mgr.34107) 10350 : audit [DBG] from='client.56017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:29:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[45495]: cluster 2022-04-22T22:29:37.516758+0000 mgr.y (mgr.34107) 10351 : cluster [DBG] pgmap v11046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:39.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:38 smithi115 bash[45495]: audit 2022-04-22T22:29:37.804846+0000 mon.a (mon.0) 14453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:39 smithi121 bash[36902]: audit 2022-04-22T22:29:38.872704+0000 mon.a (mon.0) 14454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:39 smithi115 bash[45495]: audit 2022-04-22T22:29:38.872704+0000 mon.a (mon.0) 14454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:39 smithi115 bash[46934]: audit 2022-04-22T22:29:38.872704+0000 mon.a (mon.0) 14454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:40 smithi121 bash[36902]: cluster 2022-04-22T22:29:39.517974+0000 mgr.y (mgr.34107) 10352 : cluster [DBG] pgmap v11047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:40 smithi115 bash[45495]: cluster 2022-04-22T22:29:39.517974+0000 mgr.y (mgr.34107) 10352 : cluster [DBG] pgmap v11047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:40 smithi115 bash[46934]: cluster 2022-04-22T22:29:39.517974+0000 mgr.y (mgr.34107) 10352 : cluster [DBG] pgmap v11047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:41 smithi121 bash[36902]: audit 2022-04-22T22:29:41.500391+0000 mon.a (mon.0) 14455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:41 smithi121 bash[36902]: cluster 2022-04-22T22:29:41.519071+0000 mgr.y (mgr.34107) 10353 : cluster [DBG] pgmap v11048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:41 smithi115 bash[45495]: audit 2022-04-22T22:29:41.500391+0000 mon.a (mon.0) 14455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:41 smithi115 bash[45495]: cluster 2022-04-22T22:29:41.519071+0000 mgr.y (mgr.34107) 10353 : cluster [DBG] pgmap v11048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:41 smithi115 bash[46934]: audit 2022-04-22T22:29:41.500391+0000 mon.a (mon.0) 14455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:41 smithi115 bash[46934]: cluster 2022-04-22T22:29:41.519071+0000 mgr.y (mgr.34107) 10353 : cluster [DBG] pgmap v11048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:42 smithi121 bash[36902]: audit 2022-04-22T22:29:42.808747+0000 mon.a (mon.0) 14456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:42 smithi115 bash[45495]: audit 2022-04-22T22:29:42.808747+0000 mon.a (mon.0) 14456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:42.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:42 smithi115 bash[46934]: audit 2022-04-22T22:29:42.808747+0000 mon.a (mon.0) 14456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:43 smithi121 bash[36902]: cluster 2022-04-22T22:29:43.520225+0000 mgr.y (mgr.34107) 10354 : cluster [DBG] pgmap v11049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:43 smithi115 bash[45495]: cluster 2022-04-22T22:29:43.520225+0000 mgr.y (mgr.34107) 10354 : cluster [DBG] pgmap v11049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:43 smithi115 bash[46934]: cluster 2022-04-22T22:29:43.520225+0000 mgr.y (mgr.34107) 10354 : cluster [DBG] pgmap v11049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:44 smithi115 bash[45495]: audit 2022-04-22T22:29:43.875402+0000 mon.a (mon.0) 14457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:44.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:44 smithi115 bash[46934]: audit 2022-04-22T22:29:43.875402+0000 mon.a (mon.0) 14457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:44 smithi121 bash[36902]: audit 2022-04-22T22:29:43.875402+0000 mon.a (mon.0) 14457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:45 smithi121 bash[36902]: cluster 2022-04-22T22:29:45.521297+0000 mgr.y (mgr.34107) 10355 : cluster [DBG] pgmap v11050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:45 smithi115 bash[45495]: cluster 2022-04-22T22:29:45.521297+0000 mgr.y (mgr.34107) 10355 : cluster [DBG] pgmap v11050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:45 smithi115 bash[46934]: cluster 2022-04-22T22:29:45.521297+0000 mgr.y (mgr.34107) 10355 : cluster [DBG] pgmap v11050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:46 smithi121 bash[36902]: audit 2022-04-22T22:29:46.503670+0000 mon.a (mon.0) 14458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:47.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:46 smithi115 bash[45495]: audit 2022-04-22T22:29:46.503670+0000 mon.a (mon.0) 14458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:47.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:46 smithi115 bash[46934]: audit 2022-04-22T22:29:46.503670+0000 mon.a (mon.0) 14458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:47 smithi121 bash[36902]: cluster 2022-04-22T22:29:47.522418+0000 mgr.y (mgr.34107) 10356 : cluster [DBG] pgmap v11051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:47 smithi121 bash[36902]: audit 2022-04-22T22:29:47.812617+0000 mon.a (mon.0) 14459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:48.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:47 smithi115 bash[46934]: cluster 2022-04-22T22:29:47.522418+0000 mgr.y (mgr.34107) 10356 : cluster [DBG] pgmap v11051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:48.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:47 smithi115 bash[46934]: audit 2022-04-22T22:29:47.812617+0000 mon.a (mon.0) 14459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:47 smithi115 bash[45495]: cluster 2022-04-22T22:29:47.522418+0000 mgr.y (mgr.34107) 10356 : cluster [DBG] pgmap v11051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:47 smithi115 bash[45495]: audit 2022-04-22T22:29:47.812617+0000 mon.a (mon.0) 14459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:48 smithi121 bash[36902]: audit 2022-04-22T22:29:48.877924+0000 mon.a (mon.0) 14460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:49.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:48 smithi115 bash[46934]: audit 2022-04-22T22:29:48.877924+0000 mon.a (mon.0) 14460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:48 smithi115 bash[45495]: audit 2022-04-22T22:29:48.877924+0000 mon.a (mon.0) 14460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:49 smithi121 bash[36902]: cluster 2022-04-22T22:29:49.523449+0000 mgr.y (mgr.34107) 10357 : cluster [DBG] pgmap v11052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:50.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:49 smithi115 bash[46934]: cluster 2022-04-22T22:29:49.523449+0000 mgr.y (mgr.34107) 10357 : cluster [DBG] pgmap v11052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:49 smithi115 bash[45495]: cluster 2022-04-22T22:29:49.523449+0000 mgr.y (mgr.34107) 10357 : cluster [DBG] pgmap v11052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:51 smithi121 bash[36902]: audit 2022-04-22T22:29:51.506762+0000 mon.a (mon.0) 14461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:51 smithi115 bash[45495]: audit 2022-04-22T22:29:51.506762+0000 mon.a (mon.0) 14461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:51 smithi115 bash[46934]: audit 2022-04-22T22:29:51.506762+0000 mon.a (mon.0) 14461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:52 smithi121 bash[36902]: cluster 2022-04-22T22:29:51.524445+0000 mgr.y (mgr.34107) 10358 : cluster [DBG] pgmap v11053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:53.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:52 smithi115 bash[45495]: cluster 2022-04-22T22:29:51.524445+0000 mgr.y (mgr.34107) 10358 : cluster [DBG] pgmap v11053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:52.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:52 smithi115 bash[46934]: cluster 2022-04-22T22:29:51.524445+0000 mgr.y (mgr.34107) 10358 : cluster [DBG] pgmap v11053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:53 smithi121 bash[36902]: audit 2022-04-22T22:29:52.816139+0000 mon.a (mon.0) 14462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:53 smithi115 bash[45495]: audit 2022-04-22T22:29:52.816139+0000 mon.a (mon.0) 14462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:53 smithi115 bash[46934]: audit 2022-04-22T22:29:52.816139+0000 mon.a (mon.0) 14462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:54 smithi121 bash[36902]: cluster 2022-04-22T22:29:53.525689+0000 mgr.y (mgr.34107) 10359 : cluster [DBG] pgmap v11054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:54 smithi121 bash[36902]: audit 2022-04-22T22:29:53.880776+0000 mon.a (mon.0) 14463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:54 smithi115 bash[45495]: cluster 2022-04-22T22:29:53.525689+0000 mgr.y (mgr.34107) 10359 : cluster [DBG] pgmap v11054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:54 smithi115 bash[45495]: audit 2022-04-22T22:29:53.880776+0000 mon.a (mon.0) 14463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:54 smithi115 bash[46934]: cluster 2022-04-22T22:29:53.525689+0000 mgr.y (mgr.34107) 10359 : cluster [DBG] pgmap v11054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:54 smithi115 bash[46934]: audit 2022-04-22T22:29:53.880776+0000 mon.a (mon.0) 14463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:29:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:29:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:29:54.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:29:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:56 smithi121 bash[36902]: cluster 2022-04-22T22:29:55.526835+0000 mgr.y (mgr.34107) 10360 : cluster [DBG] pgmap v11055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:56 smithi121 bash[36902]: audit 2022-04-22T22:29:56.510012+0000 mon.a (mon.0) 14464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:56 smithi115 bash[45495]: cluster 2022-04-22T22:29:55.526835+0000 mgr.y (mgr.34107) 10360 : cluster [DBG] pgmap v11055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:56 smithi115 bash[45495]: audit 2022-04-22T22:29:56.510012+0000 mon.a (mon.0) 14464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:56 smithi115 bash[46934]: cluster 2022-04-22T22:29:55.526835+0000 mgr.y (mgr.34107) 10360 : cluster [DBG] pgmap v11055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:56 smithi115 bash[46934]: audit 2022-04-22T22:29:56.510012+0000 mon.a (mon.0) 14464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:29:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:58 smithi121 bash[36902]: cluster 2022-04-22T22:29:57.527908+0000 mgr.y (mgr.34107) 10361 : cluster [DBG] pgmap v11056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:58 smithi121 bash[36902]: audit 2022-04-22T22:29:57.819729+0000 mon.a (mon.0) 14465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:58 smithi115 bash[45495]: cluster 2022-04-22T22:29:57.527908+0000 mgr.y (mgr.34107) 10361 : cluster [DBG] pgmap v11056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:58 smithi115 bash[45495]: audit 2022-04-22T22:29:57.819729+0000 mon.a (mon.0) 14465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:29:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:58 smithi115 bash[46934]: cluster 2022-04-22T22:29:57.527908+0000 mgr.y (mgr.34107) 10361 : cluster [DBG] pgmap v11056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:29:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:58 smithi115 bash[46934]: audit 2022-04-22T22:29:57.819729+0000 mon.a (mon.0) 14465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:29:59 smithi121 bash[36902]: audit 2022-04-22T22:29:58.884077+0000 mon.a (mon.0) 14466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:29:59 smithi115 bash[45495]: audit 2022-04-22T22:29:58.884077+0000 mon.a (mon.0) 14466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:29:59 smithi115 bash[46934]: audit 2022-04-22T22:29:58.884077+0000 mon.a (mon.0) 14466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:00 smithi121 bash[36902]: cluster 2022-04-22T22:29:59.529021+0000 mgr.y (mgr.34107) 10362 : cluster [DBG] pgmap v11057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:00 smithi121 bash[36902]: cluster 2022-04-22T22:30:00.000167+0000 mon.a (mon.0) 14467 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:30:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:00 smithi115 bash[45495]: cluster 2022-04-22T22:29:59.529021+0000 mgr.y (mgr.34107) 10362 : cluster [DBG] pgmap v11057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:00 smithi115 bash[45495]: cluster 2022-04-22T22:30:00.000167+0000 mon.a (mon.0) 14467 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:30:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:00 smithi115 bash[46934]: cluster 2022-04-22T22:29:59.529021+0000 mgr.y (mgr.34107) 10362 : cluster [DBG] pgmap v11057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:00 smithi115 bash[46934]: cluster 2022-04-22T22:30:00.000167+0000 mon.a (mon.0) 14467 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:30:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:01 smithi121 bash[36902]: audit 2022-04-22T22:30:01.513088+0000 mon.a (mon.0) 14468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:02.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:01 smithi115 bash[46934]: audit 2022-04-22T22:30:01.513088+0000 mon.a (mon.0) 14468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:01 smithi115 bash[45495]: audit 2022-04-22T22:30:01.513088+0000 mon.a (mon.0) 14468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:02 smithi121 bash[36902]: cluster 2022-04-22T22:30:01.530006+0000 mgr.y (mgr.34107) 10363 : cluster [DBG] pgmap v11058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:02 smithi115 bash[46934]: cluster 2022-04-22T22:30:01.530006+0000 mgr.y (mgr.34107) 10363 : cluster [DBG] pgmap v11058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:02.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:02 smithi115 bash[45495]: cluster 2022-04-22T22:30:01.530006+0000 mgr.y (mgr.34107) 10363 : cluster [DBG] pgmap v11058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:03 smithi121 bash[36902]: audit 2022-04-22T22:30:02.823168+0000 mon.a (mon.0) 14469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:03 smithi115 bash[46934]: audit 2022-04-22T22:30:02.823168+0000 mon.a (mon.0) 14469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:03 smithi115 bash[45495]: audit 2022-04-22T22:30:02.823168+0000 mon.a (mon.0) 14469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:04 smithi121 bash[36902]: cluster 2022-04-22T22:30:03.531294+0000 mgr.y (mgr.34107) 10364 : cluster [DBG] pgmap v11059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:04 smithi121 bash[36902]: audit 2022-04-22T22:30:03.887029+0000 mon.a (mon.0) 14470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:04.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:04 smithi115 bash[46934]: cluster 2022-04-22T22:30:03.531294+0000 mgr.y (mgr.34107) 10364 : cluster [DBG] pgmap v11059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:04 smithi115 bash[46934]: audit 2022-04-22T22:30:03.887029+0000 mon.a (mon.0) 14470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:04 smithi115 bash[45495]: cluster 2022-04-22T22:30:03.531294+0000 mgr.y (mgr.34107) 10364 : cluster [DBG] pgmap v11059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:04 smithi115 bash[45495]: audit 2022-04-22T22:30:03.887029+0000 mon.a (mon.0) 14470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:06 smithi121 bash[36902]: cluster 2022-04-22T22:30:05.532525+0000 mgr.y (mgr.34107) 10365 : cluster [DBG] pgmap v11060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:06 smithi121 bash[36902]: audit 2022-04-22T22:30:06.515992+0000 mon.a (mon.0) 14471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:06 smithi115 bash[46934]: cluster 2022-04-22T22:30:05.532525+0000 mgr.y (mgr.34107) 10365 : cluster [DBG] pgmap v11060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:06 smithi115 bash[46934]: audit 2022-04-22T22:30:06.515992+0000 mon.a (mon.0) 14471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:06 smithi115 bash[45495]: cluster 2022-04-22T22:30:05.532525+0000 mgr.y (mgr.34107) 10365 : cluster [DBG] pgmap v11060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:06 smithi115 bash[45495]: audit 2022-04-22T22:30:06.515992+0000 mon.a (mon.0) 14471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:07.761 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:30:08.107 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:30:08.107 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 3m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:30:08.107 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:30:08.107 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 3m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 3m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 4m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4h) 3m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:30:08.108 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 3m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 3m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 3m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 4m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 4m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 4m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 4m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:30:08.109 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 4m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:30:08.514 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:30:08.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:30:08.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:30:08.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:08.516 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:30:08.516 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:30:08.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:30:08.516 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:30:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:08 smithi121 bash[36902]: cluster 2022-04-22T22:30:07.533461+0000 mgr.y (mgr.34107) 10366 : cluster [DBG] pgmap v11061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:08 smithi121 bash[36902]: audit 2022-04-22T22:30:07.826858+0000 mon.a (mon.0) 14472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:08 smithi121 bash[36902]: audit 2022-04-22T22:30:08.510784+0000 mon.a (mon.0) 14473 : audit [DBG] from='client.? 172.21.15.115:0/3360656918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[45495]: cluster 2022-04-22T22:30:07.533461+0000 mgr.y (mgr.34107) 10366 : cluster [DBG] pgmap v11061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[45495]: audit 2022-04-22T22:30:07.826858+0000 mon.a (mon.0) 14472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:09.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[45495]: audit 2022-04-22T22:30:08.510784+0000 mon.a (mon.0) 14473 : audit [DBG] from='client.? 172.21.15.115:0/3360656918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[46934]: cluster 2022-04-22T22:30:07.533461+0000 mgr.y (mgr.34107) 10366 : cluster [DBG] pgmap v11061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[46934]: audit 2022-04-22T22:30:07.826858+0000 mon.a (mon.0) 14472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:09.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:08 smithi115 bash[46934]: audit 2022-04-22T22:30:08.510784+0000 mon.a (mon.0) 14473 : audit [DBG] from='client.? 172.21.15.115:0/3360656918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:09 smithi121 bash[36902]: audit 2022-04-22T22:30:07.741440+0000 mgr.y (mgr.34107) 10367 : audit [DBG] from='client.56029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:09 smithi121 bash[36902]: audit 2022-04-22T22:30:08.084964+0000 mgr.y (mgr.34107) 10368 : audit [DBG] from='client.56035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:09 smithi121 bash[36902]: audit 2022-04-22T22:30:08.890403+0000 mon.a (mon.0) 14474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[45495]: audit 2022-04-22T22:30:07.741440+0000 mgr.y (mgr.34107) 10367 : audit [DBG] from='client.56029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[45495]: audit 2022-04-22T22:30:08.084964+0000 mgr.y (mgr.34107) 10368 : audit [DBG] from='client.56035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[45495]: audit 2022-04-22T22:30:08.890403+0000 mon.a (mon.0) 14474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[46934]: audit 2022-04-22T22:30:07.741440+0000 mgr.y (mgr.34107) 10367 : audit [DBG] from='client.56029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[46934]: audit 2022-04-22T22:30:08.084964+0000 mgr.y (mgr.34107) 10368 : audit [DBG] from='client.56035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:09 smithi115 bash[46934]: audit 2022-04-22T22:30:08.890403+0000 mon.a (mon.0) 14474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:10 smithi121 bash[36902]: cluster 2022-04-22T22:30:09.534541+0000 mgr.y (mgr.34107) 10369 : cluster [DBG] pgmap v11062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:10 smithi115 bash[45495]: cluster 2022-04-22T22:30:09.534541+0000 mgr.y (mgr.34107) 10369 : cluster [DBG] pgmap v11062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:10 smithi115 bash[46934]: cluster 2022-04-22T22:30:09.534541+0000 mgr.y (mgr.34107) 10369 : cluster [DBG] pgmap v11062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:11 smithi121 bash[36902]: audit 2022-04-22T22:30:11.518990+0000 mon.a (mon.0) 14475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:11 smithi115 bash[45495]: audit 2022-04-22T22:30:11.518990+0000 mon.a (mon.0) 14475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:11 smithi115 bash[46934]: audit 2022-04-22T22:30:11.518990+0000 mon.a (mon.0) 14475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:12 smithi121 bash[36902]: cluster 2022-04-22T22:30:11.535677+0000 mgr.y (mgr.34107) 10370 : cluster [DBG] pgmap v11063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:12 smithi115 bash[45495]: cluster 2022-04-22T22:30:11.535677+0000 mgr.y (mgr.34107) 10370 : cluster [DBG] pgmap v11063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:12.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:12 smithi115 bash[46934]: cluster 2022-04-22T22:30:11.535677+0000 mgr.y (mgr.34107) 10370 : cluster [DBG] pgmap v11063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:13 smithi121 bash[36902]: audit 2022-04-22T22:30:12.830366+0000 mon.a (mon.0) 14476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:13 smithi115 bash[45495]: audit 2022-04-22T22:30:12.830366+0000 mon.a (mon.0) 14476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:13 smithi115 bash[46934]: audit 2022-04-22T22:30:12.830366+0000 mon.a (mon.0) 14476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:14 smithi121 bash[36902]: cluster 2022-04-22T22:30:13.536856+0000 mgr.y (mgr.34107) 10371 : cluster [DBG] pgmap v11064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:14 smithi121 bash[36902]: audit 2022-04-22T22:30:13.893471+0000 mon.a (mon.0) 14477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:14 smithi115 bash[45495]: cluster 2022-04-22T22:30:13.536856+0000 mgr.y (mgr.34107) 10371 : cluster [DBG] pgmap v11064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:14 smithi115 bash[45495]: audit 2022-04-22T22:30:13.893471+0000 mon.a (mon.0) 14477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:14.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:14 smithi115 bash[46934]: cluster 2022-04-22T22:30:13.536856+0000 mgr.y (mgr.34107) 10371 : cluster [DBG] pgmap v11064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:14 smithi115 bash[46934]: audit 2022-04-22T22:30:13.893471+0000 mon.a (mon.0) 14477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:16 smithi121 bash[36902]: cluster 2022-04-22T22:30:15.538070+0000 mgr.y (mgr.34107) 10372 : cluster [DBG] pgmap v11065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:16 smithi121 bash[36902]: audit 2022-04-22T22:30:16.522255+0000 mon.a (mon.0) 14478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:16 smithi115 bash[45495]: cluster 2022-04-22T22:30:15.538070+0000 mgr.y (mgr.34107) 10372 : cluster [DBG] pgmap v11065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:16 smithi115 bash[45495]: audit 2022-04-22T22:30:16.522255+0000 mon.a (mon.0) 14478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:16 smithi115 bash[46934]: cluster 2022-04-22T22:30:15.538070+0000 mgr.y (mgr.34107) 10372 : cluster [DBG] pgmap v11065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:16 smithi115 bash[46934]: audit 2022-04-22T22:30:16.522255+0000 mon.a (mon.0) 14478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:17 smithi121 bash[36902]: audit 2022-04-22T22:30:16.774354+0000 mon.a (mon.0) 14479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:30:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:17 smithi121 bash[36902]: audit 2022-04-22T22:30:16.775018+0000 mon.a (mon.0) 14480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:30:18.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:17 smithi115 bash[46934]: audit 2022-04-22T22:30:16.774354+0000 mon.a (mon.0) 14479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:30:18.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:17 smithi115 bash[46934]: audit 2022-04-22T22:30:16.775018+0000 mon.a (mon.0) 14480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:30:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:17 smithi115 bash[45495]: audit 2022-04-22T22:30:16.774354+0000 mon.a (mon.0) 14479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:30:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:17 smithi115 bash[45495]: audit 2022-04-22T22:30:16.775018+0000 mon.a (mon.0) 14480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:30:18.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[45495]: cluster 2022-04-22T22:30:17.539133+0000 mgr.y (mgr.34107) 10373 : cluster [DBG] pgmap v11066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:18.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[45495]: audit 2022-04-22T22:30:17.834326+0000 mon.a (mon.0) 14481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:18.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[45495]: audit 2022-04-22T22:30:18.546101+0000 mon.a (mon.0) 14482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:30:18.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[45495]: audit 2022-04-22T22:30:18.549243+0000 mon.a (mon.0) 14483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:30:18.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[45495]: audit 2022-04-22T22:30:18.551012+0000 mon.a (mon.0) 14484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:30:18.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[46934]: cluster 2022-04-22T22:30:17.539133+0000 mgr.y (mgr.34107) 10373 : cluster [DBG] pgmap v11066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:18.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[46934]: audit 2022-04-22T22:30:17.834326+0000 mon.a (mon.0) 14481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:18.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[46934]: audit 2022-04-22T22:30:18.546101+0000 mon.a (mon.0) 14482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:30:18.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[46934]: audit 2022-04-22T22:30:18.549243+0000 mon.a (mon.0) 14483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:30:18.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:18 smithi115 bash[46934]: audit 2022-04-22T22:30:18.551012+0000 mon.a (mon.0) 14484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:30:18.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:18 smithi121 bash[36902]: cluster 2022-04-22T22:30:17.539133+0000 mgr.y (mgr.34107) 10373 : cluster [DBG] pgmap v11066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:18.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:18 smithi121 bash[36902]: audit 2022-04-22T22:30:17.834326+0000 mon.a (mon.0) 14481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:18.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:18 smithi121 bash[36902]: audit 2022-04-22T22:30:18.546101+0000 mon.a (mon.0) 14482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:30:18.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:18 smithi121 bash[36902]: audit 2022-04-22T22:30:18.549243+0000 mon.a (mon.0) 14483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:30:18.916 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:18 smithi121 bash[36902]: audit 2022-04-22T22:30:18.551012+0000 mon.a (mon.0) 14484 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:30:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:19 smithi121 bash[36902]: audit 2022-04-22T22:30:18.896689+0000 mon.a (mon.0) 14485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:19 smithi121 bash[36902]: audit 2022-04-22T22:30:18.923220+0000 mon.a (mon.0) 14486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:30:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:19 smithi115 bash[45495]: audit 2022-04-22T22:30:18.896689+0000 mon.a (mon.0) 14485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:19 smithi115 bash[45495]: audit 2022-04-22T22:30:18.923220+0000 mon.a (mon.0) 14486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:30:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:19 smithi115 bash[46934]: audit 2022-04-22T22:30:18.896689+0000 mon.a (mon.0) 14485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:19 smithi115 bash[46934]: audit 2022-04-22T22:30:18.923220+0000 mon.a (mon.0) 14486 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:30:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:20 smithi121 bash[36902]: cluster 2022-04-22T22:30:19.540140+0000 mgr.y (mgr.34107) 10374 : cluster [DBG] pgmap v11067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:20 smithi115 bash[45495]: cluster 2022-04-22T22:30:19.540140+0000 mgr.y (mgr.34107) 10374 : cluster [DBG] pgmap v11067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:20 smithi115 bash[46934]: cluster 2022-04-22T22:30:19.540140+0000 mgr.y (mgr.34107) 10374 : cluster [DBG] pgmap v11067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:21 smithi121 bash[36902]: audit 2022-04-22T22:30:21.525498+0000 mon.a (mon.0) 14487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:21 smithi115 bash[45495]: audit 2022-04-22T22:30:21.525498+0000 mon.a (mon.0) 14487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:22.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:21 smithi115 bash[46934]: audit 2022-04-22T22:30:21.525498+0000 mon.a (mon.0) 14487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:22 smithi121 bash[36902]: cluster 2022-04-22T22:30:21.541251+0000 mgr.y (mgr.34107) 10375 : cluster [DBG] pgmap v11068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:22 smithi115 bash[45495]: cluster 2022-04-22T22:30:21.541251+0000 mgr.y (mgr.34107) 10375 : cluster [DBG] pgmap v11068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:22.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:22 smithi115 bash[46934]: cluster 2022-04-22T22:30:21.541251+0000 mgr.y (mgr.34107) 10375 : cluster [DBG] pgmap v11068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:23 smithi121 bash[36902]: audit 2022-04-22T22:30:22.838444+0000 mon.a (mon.0) 14488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:23 smithi115 bash[45495]: audit 2022-04-22T22:30:22.838444+0000 mon.a (mon.0) 14488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:23 smithi115 bash[46934]: audit 2022-04-22T22:30:22.838444+0000 mon.a (mon.0) 14488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:24 smithi121 bash[36902]: cluster 2022-04-22T22:30:23.542466+0000 mgr.y (mgr.34107) 10376 : cluster [DBG] pgmap v11069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:24 smithi121 bash[36902]: audit 2022-04-22T22:30:23.899380+0000 mon.a (mon.0) 14489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:24.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:24 smithi115 bash[45495]: cluster 2022-04-22T22:30:23.542466+0000 mgr.y (mgr.34107) 10376 : cluster [DBG] pgmap v11069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:24 smithi115 bash[45495]: audit 2022-04-22T22:30:23.899380+0000 mon.a (mon.0) 14489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:24 smithi115 bash[46934]: cluster 2022-04-22T22:30:23.542466+0000 mgr.y (mgr.34107) 10376 : cluster [DBG] pgmap v11069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:24 smithi115 bash[46934]: audit 2022-04-22T22:30:23.899380+0000 mon.a (mon.0) 14489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:25 smithi121 bash[36902]: cluster 2022-04-22T22:30:25.543627+0000 mgr.y (mgr.34107) 10377 : cluster [DBG] pgmap v11070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:25 smithi115 bash[45495]: cluster 2022-04-22T22:30:25.543627+0000 mgr.y (mgr.34107) 10377 : cluster [DBG] pgmap v11070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:25 smithi115 bash[46934]: cluster 2022-04-22T22:30:25.543627+0000 mgr.y (mgr.34107) 10377 : cluster [DBG] pgmap v11070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:26 smithi121 bash[36902]: audit 2022-04-22T22:30:26.528557+0000 mon.a (mon.0) 14490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:26 smithi115 bash[45495]: audit 2022-04-22T22:30:26.528557+0000 mon.a (mon.0) 14490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:26 smithi115 bash[46934]: audit 2022-04-22T22:30:26.528557+0000 mon.a (mon.0) 14490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:27 smithi121 bash[36902]: cluster 2022-04-22T22:30:27.544661+0000 mgr.y (mgr.34107) 10378 : cluster [DBG] pgmap v11071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:27 smithi115 bash[45495]: cluster 2022-04-22T22:30:27.544661+0000 mgr.y (mgr.34107) 10378 : cluster [DBG] pgmap v11071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:27 smithi115 bash[46934]: cluster 2022-04-22T22:30:27.544661+0000 mgr.y (mgr.34107) 10378 : cluster [DBG] pgmap v11071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:28 smithi121 bash[36902]: audit 2022-04-22T22:30:27.841771+0000 mon.a (mon.0) 14491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:28 smithi115 bash[45495]: audit 2022-04-22T22:30:27.841771+0000 mon.a (mon.0) 14491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:28 smithi115 bash[46934]: audit 2022-04-22T22:30:27.841771+0000 mon.a (mon.0) 14491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:29 smithi121 bash[36902]: audit 2022-04-22T22:30:28.902414+0000 mon.a (mon.0) 14492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:29 smithi121 bash[36902]: cluster 2022-04-22T22:30:29.545775+0000 mgr.y (mgr.34107) 10379 : cluster [DBG] pgmap v11072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:29 smithi115 bash[45495]: audit 2022-04-22T22:30:28.902414+0000 mon.a (mon.0) 14492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:29 smithi115 bash[45495]: cluster 2022-04-22T22:30:29.545775+0000 mgr.y (mgr.34107) 10379 : cluster [DBG] pgmap v11072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:29 smithi115 bash[46934]: audit 2022-04-22T22:30:28.902414+0000 mon.a (mon.0) 14492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:29 smithi115 bash[46934]: cluster 2022-04-22T22:30:29.545775+0000 mgr.y (mgr.34107) 10379 : cluster [DBG] pgmap v11072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:31 smithi121 bash[36902]: audit 2022-04-22T22:30:31.531968+0000 mon.a (mon.0) 14493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:31 smithi115 bash[45495]: audit 2022-04-22T22:30:31.531968+0000 mon.a (mon.0) 14493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:31 smithi115 bash[46934]: audit 2022-04-22T22:30:31.531968+0000 mon.a (mon.0) 14493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:32 smithi121 bash[36902]: cluster 2022-04-22T22:30:31.546763+0000 mgr.y (mgr.34107) 10380 : cluster [DBG] pgmap v11073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:32 smithi115 bash[46934]: cluster 2022-04-22T22:30:31.546763+0000 mgr.y (mgr.34107) 10380 : cluster [DBG] pgmap v11073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:33.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:32.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:33.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:32 smithi115 bash[45495]: cluster 2022-04-22T22:30:31.546763+0000 mgr.y (mgr.34107) 10380 : cluster [DBG] pgmap v11073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:33 smithi121 bash[36902]: audit 2022-04-22T22:30:32.845511+0000 mon.a (mon.0) 14494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:33 smithi115 bash[46934]: audit 2022-04-22T22:30:32.845511+0000 mon.a (mon.0) 14494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:33 smithi115 bash[45495]: audit 2022-04-22T22:30:32.845511+0000 mon.a (mon.0) 14494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:34 smithi121 bash[36902]: cluster 2022-04-22T22:30:33.547970+0000 mgr.y (mgr.34107) 10381 : cluster [DBG] pgmap v11074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:34 smithi121 bash[36902]: audit 2022-04-22T22:30:33.905210+0000 mon.a (mon.0) 14495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:34.667Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:34 smithi115 bash[46934]: cluster 2022-04-22T22:30:33.547970+0000 mgr.y (mgr.34107) 10381 : cluster [DBG] pgmap v11074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:34 smithi115 bash[46934]: audit 2022-04-22T22:30:33.905210+0000 mon.a (mon.0) 14495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:34 smithi115 bash[45495]: cluster 2022-04-22T22:30:33.547970+0000 mgr.y (mgr.34107) 10381 : cluster [DBG] pgmap v11074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:35.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:34 smithi115 bash[45495]: audit 2022-04-22T22:30:33.905210+0000 mon.a (mon.0) 14495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:36 smithi121 bash[36902]: cluster 2022-04-22T22:30:35.549089+0000 mgr.y (mgr.34107) 10382 : cluster [DBG] pgmap v11075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:36 smithi121 bash[36902]: audit 2022-04-22T22:30:36.535138+0000 mon.a (mon.0) 14496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:36 smithi115 bash[45495]: cluster 2022-04-22T22:30:35.549089+0000 mgr.y (mgr.34107) 10382 : cluster [DBG] pgmap v11075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:36 smithi115 bash[45495]: audit 2022-04-22T22:30:36.535138+0000 mon.a (mon.0) 14496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:36 smithi115 bash[46934]: cluster 2022-04-22T22:30:35.549089+0000 mgr.y (mgr.34107) 10382 : cluster [DBG] pgmap v11075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:36 smithi115 bash[46934]: audit 2022-04-22T22:30:36.535138+0000 mon.a (mon.0) 14496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:38.877 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:30:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:38 smithi121 bash[36902]: cluster 2022-04-22T22:30:37.550155+0000 mgr.y (mgr.34107) 10383 : cluster [DBG] pgmap v11076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:38 smithi121 bash[36902]: audit 2022-04-22T22:30:37.849139+0000 mon.a (mon.0) 14497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:38 smithi115 bash[46934]: cluster 2022-04-22T22:30:37.550155+0000 mgr.y (mgr.34107) 10383 : cluster [DBG] pgmap v11076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:39.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:38 smithi115 bash[46934]: audit 2022-04-22T22:30:37.849139+0000 mon.a (mon.0) 14497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:38 smithi115 bash[45495]: cluster 2022-04-22T22:30:37.550155+0000 mgr.y (mgr.34107) 10383 : cluster [DBG] pgmap v11076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:38 smithi115 bash[45495]: audit 2022-04-22T22:30:37.849139+0000 mon.a (mon.0) 14497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:30:39.224 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4h) 4m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:30:39.225 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:30:39.226 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:30:39.226 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:30:39.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:30:39.621 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:30:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:39 smithi121 bash[36902]: audit 2022-04-22T22:30:38.908158+0000 mon.a (mon.0) 14498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:39 smithi121 bash[36902]: audit 2022-04-22T22:30:39.616462+0000 mon.a (mon.0) 14499 : audit [DBG] from='client.? 172.21.15.115:0/1236346538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:39 smithi115 bash[46934]: audit 2022-04-22T22:30:38.908158+0000 mon.a (mon.0) 14498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:39 smithi115 bash[46934]: audit 2022-04-22T22:30:39.616462+0000 mon.a (mon.0) 14499 : audit [DBG] from='client.? 172.21.15.115:0/1236346538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:39 smithi115 bash[45495]: audit 2022-04-22T22:30:38.908158+0000 mon.a (mon.0) 14498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:39 smithi115 bash[45495]: audit 2022-04-22T22:30:39.616462+0000 mon.a (mon.0) 14499 : audit [DBG] from='client.? 172.21.15.115:0/1236346538' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:30:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:40 smithi121 bash[36902]: audit 2022-04-22T22:30:38.859617+0000 mgr.y (mgr.34107) 10384 : audit [DBG] from='client.56047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:40 smithi121 bash[36902]: audit 2022-04-22T22:30:39.202187+0000 mgr.y (mgr.34107) 10385 : audit [DBG] from='client.46119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:40 smithi121 bash[36902]: cluster 2022-04-22T22:30:39.551184+0000 mgr.y (mgr.34107) 10386 : cluster [DBG] pgmap v11077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[45495]: audit 2022-04-22T22:30:38.859617+0000 mgr.y (mgr.34107) 10384 : audit [DBG] from='client.56047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[45495]: audit 2022-04-22T22:30:39.202187+0000 mgr.y (mgr.34107) 10385 : audit [DBG] from='client.46119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[45495]: cluster 2022-04-22T22:30:39.551184+0000 mgr.y (mgr.34107) 10386 : cluster [DBG] pgmap v11077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[46934]: audit 2022-04-22T22:30:38.859617+0000 mgr.y (mgr.34107) 10384 : audit [DBG] from='client.56047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[46934]: audit 2022-04-22T22:30:39.202187+0000 mgr.y (mgr.34107) 10385 : audit [DBG] from='client.46119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:30:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:40 smithi115 bash[46934]: cluster 2022-04-22T22:30:39.551184+0000 mgr.y (mgr.34107) 10386 : cluster [DBG] pgmap v11077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:41 smithi121 bash[36902]: audit 2022-04-22T22:30:41.538142+0000 mon.a (mon.0) 14500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:41 smithi115 bash[45495]: audit 2022-04-22T22:30:41.538142+0000 mon.a (mon.0) 14500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:41 smithi115 bash[46934]: audit 2022-04-22T22:30:41.538142+0000 mon.a (mon.0) 14500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:42 smithi121 bash[36902]: cluster 2022-04-22T22:30:41.552168+0000 mgr.y (mgr.34107) 10387 : cluster [DBG] pgmap v11078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:43.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:42 smithi115 bash[45495]: cluster 2022-04-22T22:30:41.552168+0000 mgr.y (mgr.34107) 10387 : cluster [DBG] pgmap v11078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:43.056 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:42.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:43.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:42 smithi115 bash[46934]: cluster 2022-04-22T22:30:41.552168+0000 mgr.y (mgr.34107) 10387 : cluster [DBG] pgmap v11078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:43 smithi121 bash[36902]: audit 2022-04-22T22:30:42.852816+0000 mon.a (mon.0) 14501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:43 smithi115 bash[45495]: audit 2022-04-22T22:30:42.852816+0000 mon.a (mon.0) 14501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:43 smithi115 bash[46934]: audit 2022-04-22T22:30:42.852816+0000 mon.a (mon.0) 14501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:44 smithi121 bash[36902]: cluster 2022-04-22T22:30:43.553295+0000 mgr.y (mgr.34107) 10388 : cluster [DBG] pgmap v11079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:44 smithi121 bash[36902]: audit 2022-04-22T22:30:43.910924+0000 mon.a (mon.0) 14502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:44.665Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:44 smithi115 bash[45495]: cluster 2022-04-22T22:30:43.553295+0000 mgr.y (mgr.34107) 10388 : cluster [DBG] pgmap v11079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:44 smithi115 bash[45495]: audit 2022-04-22T22:30:43.910924+0000 mon.a (mon.0) 14502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:44 smithi115 bash[46934]: cluster 2022-04-22T22:30:43.553295+0000 mgr.y (mgr.34107) 10388 : cluster [DBG] pgmap v11079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:44 smithi115 bash[46934]: audit 2022-04-22T22:30:43.910924+0000 mon.a (mon.0) 14502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:46 smithi121 bash[36902]: cluster 2022-04-22T22:30:45.554373+0000 mgr.y (mgr.34107) 10389 : cluster [DBG] pgmap v11080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:46 smithi121 bash[36902]: audit 2022-04-22T22:30:46.541389+0000 mon.a (mon.0) 14503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:46 smithi115 bash[45495]: cluster 2022-04-22T22:30:45.554373+0000 mgr.y (mgr.34107) 10389 : cluster [DBG] pgmap v11080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:46 smithi115 bash[45495]: audit 2022-04-22T22:30:46.541389+0000 mon.a (mon.0) 14503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:46 smithi115 bash[46934]: cluster 2022-04-22T22:30:45.554373+0000 mgr.y (mgr.34107) 10389 : cluster [DBG] pgmap v11080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:46 smithi115 bash[46934]: audit 2022-04-22T22:30:46.541389+0000 mon.a (mon.0) 14503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:48 smithi121 bash[36902]: cluster 2022-04-22T22:30:47.555438+0000 mgr.y (mgr.34107) 10390 : cluster [DBG] pgmap v11081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:48 smithi121 bash[36902]: audit 2022-04-22T22:30:47.856473+0000 mon.a (mon.0) 14504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:48 smithi115 bash[45495]: cluster 2022-04-22T22:30:47.555438+0000 mgr.y (mgr.34107) 10390 : cluster [DBG] pgmap v11081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:48 smithi115 bash[45495]: audit 2022-04-22T22:30:47.856473+0000 mon.a (mon.0) 14504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:48 smithi115 bash[46934]: cluster 2022-04-22T22:30:47.555438+0000 mgr.y (mgr.34107) 10390 : cluster [DBG] pgmap v11081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:48 smithi115 bash[46934]: audit 2022-04-22T22:30:47.856473+0000 mon.a (mon.0) 14504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:49 smithi121 bash[36902]: audit 2022-04-22T22:30:48.914157+0000 mon.a (mon.0) 14505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:49 smithi115 bash[45495]: audit 2022-04-22T22:30:48.914157+0000 mon.a (mon.0) 14505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:49 smithi115 bash[46934]: audit 2022-04-22T22:30:48.914157+0000 mon.a (mon.0) 14505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:50 smithi121 bash[36902]: cluster 2022-04-22T22:30:49.556478+0000 mgr.y (mgr.34107) 10391 : cluster [DBG] pgmap v11082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:51.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:50 smithi115 bash[46934]: cluster 2022-04-22T22:30:49.556478+0000 mgr.y (mgr.34107) 10391 : cluster [DBG] pgmap v11082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:50 smithi115 bash[45495]: cluster 2022-04-22T22:30:49.556478+0000 mgr.y (mgr.34107) 10391 : cluster [DBG] pgmap v11082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:51 smithi121 bash[36902]: audit 2022-04-22T22:30:51.544626+0000 mon.a (mon.0) 14506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:51 smithi115 bash[45495]: audit 2022-04-22T22:30:51.544626+0000 mon.a (mon.0) 14506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:51 smithi115 bash[46934]: audit 2022-04-22T22:30:51.544626+0000 mon.a (mon.0) 14506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:52 smithi121 bash[36902]: cluster 2022-04-22T22:30:51.557486+0000 mgr.y (mgr.34107) 10392 : cluster [DBG] pgmap v11083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:52 smithi115 bash[45495]: cluster 2022-04-22T22:30:51.557486+0000 mgr.y (mgr.34107) 10392 : cluster [DBG] pgmap v11083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:52.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:52 smithi115 bash[46934]: cluster 2022-04-22T22:30:51.557486+0000 mgr.y (mgr.34107) 10392 : cluster [DBG] pgmap v11083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:53 smithi121 bash[36902]: audit 2022-04-22T22:30:52.860279+0000 mon.a (mon.0) 14507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:53 smithi115 bash[45495]: audit 2022-04-22T22:30:52.860279+0000 mon.a (mon.0) 14507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:53 smithi115 bash[46934]: audit 2022-04-22T22:30:52.860279+0000 mon.a (mon.0) 14507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:54 smithi121 bash[36902]: cluster 2022-04-22T22:30:53.558581+0000 mgr.y (mgr.34107) 10393 : cluster [DBG] pgmap v11084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:54 smithi121 bash[36902]: audit 2022-04-22T22:30:53.917431+0000 mon.a (mon.0) 14508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:30:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:30:54.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:30:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:54 smithi115 bash[45495]: cluster 2022-04-22T22:30:53.558581+0000 mgr.y (mgr.34107) 10393 : cluster [DBG] pgmap v11084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:54 smithi115 bash[45495]: audit 2022-04-22T22:30:53.917431+0000 mon.a (mon.0) 14508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:54 smithi115 bash[46934]: cluster 2022-04-22T22:30:53.558581+0000 mgr.y (mgr.34107) 10393 : cluster [DBG] pgmap v11084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:54 smithi115 bash[46934]: audit 2022-04-22T22:30:53.917431+0000 mon.a (mon.0) 14508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:30:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:56 smithi121 bash[36902]: cluster 2022-04-22T22:30:55.559663+0000 mgr.y (mgr.34107) 10394 : cluster [DBG] pgmap v11085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:56 smithi121 bash[36902]: audit 2022-04-22T22:30:56.548244+0000 mon.a (mon.0) 14509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:56 smithi115 bash[45495]: cluster 2022-04-22T22:30:55.559663+0000 mgr.y (mgr.34107) 10394 : cluster [DBG] pgmap v11085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:56 smithi115 bash[45495]: audit 2022-04-22T22:30:56.548244+0000 mon.a (mon.0) 14509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:56 smithi115 bash[46934]: cluster 2022-04-22T22:30:55.559663+0000 mgr.y (mgr.34107) 10394 : cluster [DBG] pgmap v11085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:56 smithi115 bash[46934]: audit 2022-04-22T22:30:56.548244+0000 mon.a (mon.0) 14509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:30:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:58 smithi121 bash[36902]: cluster 2022-04-22T22:30:57.560714+0000 mgr.y (mgr.34107) 10395 : cluster [DBG] pgmap v11086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:58 smithi121 bash[36902]: audit 2022-04-22T22:30:57.863979+0000 mon.a (mon.0) 14510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:58 smithi115 bash[45495]: cluster 2022-04-22T22:30:57.560714+0000 mgr.y (mgr.34107) 10395 : cluster [DBG] pgmap v11086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:58 smithi115 bash[45495]: audit 2022-04-22T22:30:57.863979+0000 mon.a (mon.0) 14510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:30:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:58 smithi115 bash[46934]: cluster 2022-04-22T22:30:57.560714+0000 mgr.y (mgr.34107) 10395 : cluster [DBG] pgmap v11086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:30:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:58 smithi115 bash[46934]: audit 2022-04-22T22:30:57.863979+0000 mon.a (mon.0) 14510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:30:59 smithi121 bash[36902]: audit 2022-04-22T22:30:58.920650+0000 mon.a (mon.0) 14511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:30:59 smithi115 bash[45495]: audit 2022-04-22T22:30:58.920650+0000 mon.a (mon.0) 14511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:30:59 smithi115 bash[46934]: audit 2022-04-22T22:30:58.920650+0000 mon.a (mon.0) 14511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:00 smithi121 bash[36902]: cluster 2022-04-22T22:30:59.561783+0000 mgr.y (mgr.34107) 10396 : cluster [DBG] pgmap v11087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:00 smithi115 bash[45495]: cluster 2022-04-22T22:30:59.561783+0000 mgr.y (mgr.34107) 10396 : cluster [DBG] pgmap v11087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:00 smithi115 bash[46934]: cluster 2022-04-22T22:30:59.561783+0000 mgr.y (mgr.34107) 10396 : cluster [DBG] pgmap v11087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:01 smithi121 bash[36902]: audit 2022-04-22T22:31:01.552244+0000 mon.a (mon.0) 14512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:01 smithi121 bash[36902]: cluster 2022-04-22T22:31:01.562829+0000 mgr.y (mgr.34107) 10397 : cluster [DBG] pgmap v11088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:01 smithi115 bash[45495]: audit 2022-04-22T22:31:01.552244+0000 mon.a (mon.0) 14512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:01 smithi115 bash[45495]: cluster 2022-04-22T22:31:01.562829+0000 mgr.y (mgr.34107) 10397 : cluster [DBG] pgmap v11088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:01 smithi115 bash[46934]: audit 2022-04-22T22:31:01.552244+0000 mon.a (mon.0) 14512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:01 smithi115 bash[46934]: cluster 2022-04-22T22:31:01.562829+0000 mgr.y (mgr.34107) 10397 : cluster [DBG] pgmap v11088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:02 smithi121 bash[36902]: audit 2022-04-22T22:31:02.867620+0000 mon.a (mon.0) 14513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:02 smithi115 bash[45495]: audit 2022-04-22T22:31:02.867620+0000 mon.a (mon.0) 14513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:02 smithi115 bash[46934]: audit 2022-04-22T22:31:02.867620+0000 mon.a (mon.0) 14513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:02.817Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:03 smithi121 bash[36902]: cluster 2022-04-22T22:31:03.563956+0000 mgr.y (mgr.34107) 10398 : cluster [DBG] pgmap v11089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:03 smithi115 bash[45495]: cluster 2022-04-22T22:31:03.563956+0000 mgr.y (mgr.34107) 10398 : cluster [DBG] pgmap v11089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:03 smithi115 bash[46934]: cluster 2022-04-22T22:31:03.563956+0000 mgr.y (mgr.34107) 10398 : cluster [DBG] pgmap v11089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:04.934 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:04.669Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:04 smithi121 bash[36902]: audit 2022-04-22T22:31:03.928167+0000 mon.a (mon.0) 14514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:04 smithi115 bash[45495]: audit 2022-04-22T22:31:03.928167+0000 mon.a (mon.0) 14514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:04 smithi115 bash[46934]: audit 2022-04-22T22:31:03.928167+0000 mon.a (mon.0) 14514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:05 smithi121 bash[36902]: cluster 2022-04-22T22:31:05.565032+0000 mgr.y (mgr.34107) 10399 : cluster [DBG] pgmap v11090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:05 smithi115 bash[45495]: cluster 2022-04-22T22:31:05.565032+0000 mgr.y (mgr.34107) 10399 : cluster [DBG] pgmap v11090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:05 smithi115 bash[46934]: cluster 2022-04-22T22:31:05.565032+0000 mgr.y (mgr.34107) 10399 : cluster [DBG] pgmap v11090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:06 smithi121 bash[36902]: audit 2022-04-22T22:31:06.555346+0000 mon.a (mon.0) 14515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:06 smithi115 bash[45495]: audit 2022-04-22T22:31:06.555346+0000 mon.a (mon.0) 14515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:07.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:06 smithi115 bash[46934]: audit 2022-04-22T22:31:06.555346+0000 mon.a (mon.0) 14515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:07 smithi121 bash[36902]: cluster 2022-04-22T22:31:07.566146+0000 mgr.y (mgr.34107) 10400 : cluster [DBG] pgmap v11091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:07 smithi121 bash[36902]: audit 2022-04-22T22:31:07.871151+0000 mon.a (mon.0) 14516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:07 smithi115 bash[45495]: cluster 2022-04-22T22:31:07.566146+0000 mgr.y (mgr.34107) 10400 : cluster [DBG] pgmap v11091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:07 smithi115 bash[45495]: audit 2022-04-22T22:31:07.871151+0000 mon.a (mon.0) 14516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:07 smithi115 bash[46934]: cluster 2022-04-22T22:31:07.566146+0000 mgr.y (mgr.34107) 10400 : cluster [DBG] pgmap v11091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:07 smithi115 bash[46934]: audit 2022-04-22T22:31:07.871151+0000 mon.a (mon.0) 14516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:09.993 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:31:10.335 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:31:10.335 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 4m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:31:10.336 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:31:10.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 4m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:31:10.336 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:31:10.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:31:10.336 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 4m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 5m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4h) 4m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:31:10.337 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 4m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:31:10.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 5m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:31:10.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 5m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:31:10.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 5m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:31:10.338 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 5m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:31:10.338 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 5m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:31:10.740 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:31:10.740 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:31:10.740 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:31:10.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:31:10.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:10.742 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:31:10.742 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:31:10.742 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:31:10.742 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:31:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:10 smithi121 bash[36902]: cluster 2022-04-22T22:31:09.567247+0000 mgr.y (mgr.34107) 10401 : cluster [DBG] pgmap v11092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:10 smithi121 bash[36902]: audit 2022-04-22T22:31:09.927048+0000 mon.a (mon.0) 14517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:10 smithi115 bash[45495]: cluster 2022-04-22T22:31:09.567247+0000 mgr.y (mgr.34107) 10401 : cluster [DBG] pgmap v11092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:10 smithi115 bash[45495]: audit 2022-04-22T22:31:09.927048+0000 mon.a (mon.0) 14517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:10 smithi115 bash[46934]: cluster 2022-04-22T22:31:09.567247+0000 mgr.y (mgr.34107) 10401 : cluster [DBG] pgmap v11092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:10 smithi115 bash[46934]: audit 2022-04-22T22:31:09.927048+0000 mon.a (mon.0) 14517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:11 smithi121 bash[36902]: audit 2022-04-22T22:31:09.973368+0000 mgr.y (mgr.34107) 10402 : audit [DBG] from='client.46131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:11 smithi121 bash[36902]: audit 2022-04-22T22:31:10.313006+0000 mgr.y (mgr.34107) 10403 : audit [DBG] from='client.56071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:11 smithi121 bash[36902]: audit 2022-04-22T22:31:10.736627+0000 mon.a (mon.0) 14518 : audit [DBG] from='client.? 172.21.15.115:0/1958950129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:11 smithi121 bash[36902]: audit 2022-04-22T22:31:11.558493+0000 mon.a (mon.0) 14519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[45495]: audit 2022-04-22T22:31:09.973368+0000 mgr.y (mgr.34107) 10402 : audit [DBG] from='client.46131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[45495]: audit 2022-04-22T22:31:10.313006+0000 mgr.y (mgr.34107) 10403 : audit [DBG] from='client.56071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[45495]: audit 2022-04-22T22:31:10.736627+0000 mon.a (mon.0) 14518 : audit [DBG] from='client.? 172.21.15.115:0/1958950129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[45495]: audit 2022-04-22T22:31:11.558493+0000 mon.a (mon.0) 14519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[46934]: audit 2022-04-22T22:31:09.973368+0000 mgr.y (mgr.34107) 10402 : audit [DBG] from='client.46131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[46934]: audit 2022-04-22T22:31:10.313006+0000 mgr.y (mgr.34107) 10403 : audit [DBG] from='client.56071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[46934]: audit 2022-04-22T22:31:10.736627+0000 mon.a (mon.0) 14518 : audit [DBG] from='client.? 172.21.15.115:0/1958950129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:11 smithi115 bash[46934]: audit 2022-04-22T22:31:11.558493+0000 mon.a (mon.0) 14519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:12 smithi121 bash[36902]: cluster 2022-04-22T22:31:11.568227+0000 mgr.y (mgr.34107) 10404 : cluster [DBG] pgmap v11093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:12 smithi115 bash[45495]: cluster 2022-04-22T22:31:11.568227+0000 mgr.y (mgr.34107) 10404 : cluster [DBG] pgmap v11093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:12.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:12 smithi115 bash[46934]: cluster 2022-04-22T22:31:11.568227+0000 mgr.y (mgr.34107) 10404 : cluster [DBG] pgmap v11093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:13 smithi121 bash[36902]: audit 2022-04-22T22:31:12.874884+0000 mon.a (mon.0) 14520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:13 smithi115 bash[45495]: audit 2022-04-22T22:31:12.874884+0000 mon.a (mon.0) 14520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:13 smithi115 bash[46934]: audit 2022-04-22T22:31:12.874884+0000 mon.a (mon.0) 14520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:14 smithi121 bash[36902]: cluster 2022-04-22T22:31:13.569379+0000 mgr.y (mgr.34107) 10405 : cluster [DBG] pgmap v11094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:14 smithi115 bash[45495]: cluster 2022-04-22T22:31:13.569379+0000 mgr.y (mgr.34107) 10405 : cluster [DBG] pgmap v11094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:14.670Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:14 smithi115 bash[46934]: cluster 2022-04-22T22:31:13.569379+0000 mgr.y (mgr.34107) 10405 : cluster [DBG] pgmap v11094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:15 smithi121 bash[36902]: audit 2022-04-22T22:31:14.929989+0000 mon.a (mon.0) 14521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:15 smithi115 bash[45495]: audit 2022-04-22T22:31:14.929989+0000 mon.a (mon.0) 14521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:15 smithi115 bash[46934]: audit 2022-04-22T22:31:14.929989+0000 mon.a (mon.0) 14521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:16 smithi121 bash[36902]: cluster 2022-04-22T22:31:15.570575+0000 mgr.y (mgr.34107) 10406 : cluster [DBG] pgmap v11095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:16 smithi121 bash[36902]: audit 2022-04-22T22:31:16.561394+0000 mon.a (mon.0) 14522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:16 smithi115 bash[45495]: cluster 2022-04-22T22:31:15.570575+0000 mgr.y (mgr.34107) 10406 : cluster [DBG] pgmap v11095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:16 smithi115 bash[45495]: audit 2022-04-22T22:31:16.561394+0000 mon.a (mon.0) 14522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:16 smithi115 bash[46934]: cluster 2022-04-22T22:31:15.570575+0000 mgr.y (mgr.34107) 10406 : cluster [DBG] pgmap v11095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:16 smithi115 bash[46934]: audit 2022-04-22T22:31:16.561394+0000 mon.a (mon.0) 14522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:17 smithi121 bash[36902]: audit 2022-04-22T22:31:16.785387+0000 mon.a (mon.0) 14523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:31:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:17 smithi121 bash[36902]: audit 2022-04-22T22:31:16.786944+0000 mon.a (mon.0) 14524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:31:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:17 smithi115 bash[45495]: audit 2022-04-22T22:31:16.785387+0000 mon.a (mon.0) 14523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:31:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:17 smithi115 bash[45495]: audit 2022-04-22T22:31:16.786944+0000 mon.a (mon.0) 14524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:31:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:17 smithi115 bash[46934]: audit 2022-04-22T22:31:16.785387+0000 mon.a (mon.0) 14523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:31:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:17 smithi115 bash[46934]: audit 2022-04-22T22:31:16.786944+0000 mon.a (mon.0) 14524 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:31:18.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:18 smithi115 bash[46934]: cluster 2022-04-22T22:31:17.571831+0000 mgr.y (mgr.34107) 10407 : cluster [DBG] pgmap v11096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:18.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:18 smithi115 bash[46934]: audit 2022-04-22T22:31:17.878662+0000 mon.a (mon.0) 14525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:18.959 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:18 smithi115 bash[45495]: cluster 2022-04-22T22:31:17.571831+0000 mgr.y (mgr.34107) 10407 : cluster [DBG] pgmap v11096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:18.959 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:18 smithi115 bash[45495]: audit 2022-04-22T22:31:17.878662+0000 mon.a (mon.0) 14525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:18.961 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:18 smithi121 bash[36902]: cluster 2022-04-22T22:31:17.571831+0000 mgr.y (mgr.34107) 10407 : cluster [DBG] pgmap v11096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:18.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:18 smithi121 bash[36902]: audit 2022-04-22T22:31:17.878662+0000 mon.a (mon.0) 14525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:19 smithi121 bash[36902]: audit 2022-04-22T22:31:18.931048+0000 mon.a (mon.0) 14526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:31:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:19 smithi121 bash[36902]: audit 2022-04-22T22:31:18.933916+0000 mon.a (mon.0) 14527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:31:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:19 smithi121 bash[36902]: audit 2022-04-22T22:31:18.935459+0000 mon.a (mon.0) 14528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:31:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:19 smithi121 bash[36902]: audit 2022-04-22T22:31:19.285284+0000 mon.a (mon.0) 14529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:31:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[45495]: audit 2022-04-22T22:31:18.931048+0000 mon.a (mon.0) 14526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[45495]: audit 2022-04-22T22:31:18.933916+0000 mon.a (mon.0) 14527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[45495]: audit 2022-04-22T22:31:18.935459+0000 mon.a (mon.0) 14528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[45495]: audit 2022-04-22T22:31:19.285284+0000 mon.a (mon.0) 14529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[46934]: audit 2022-04-22T22:31:18.931048+0000 mon.a (mon.0) 14526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[46934]: audit 2022-04-22T22:31:18.933916+0000 mon.a (mon.0) 14527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[46934]: audit 2022-04-22T22:31:18.935459+0000 mon.a (mon.0) 14528 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:31:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:19 smithi115 bash[46934]: audit 2022-04-22T22:31:19.285284+0000 mon.a (mon.0) 14529 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:31:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:20 smithi121 bash[36902]: cluster 2022-04-22T22:31:19.572873+0000 mgr.y (mgr.34107) 10408 : cluster [DBG] pgmap v11097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:20 smithi121 bash[36902]: audit 2022-04-22T22:31:19.933176+0000 mon.a (mon.0) 14530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:20 smithi115 bash[45495]: cluster 2022-04-22T22:31:19.572873+0000 mgr.y (mgr.34107) 10408 : cluster [DBG] pgmap v11097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:20 smithi115 bash[45495]: audit 2022-04-22T22:31:19.933176+0000 mon.a (mon.0) 14530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:20 smithi115 bash[46934]: cluster 2022-04-22T22:31:19.572873+0000 mgr.y (mgr.34107) 10408 : cluster [DBG] pgmap v11097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:20 smithi115 bash[46934]: audit 2022-04-22T22:31:19.933176+0000 mon.a (mon.0) 14530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:21 smithi121 bash[36902]: audit 2022-04-22T22:31:21.564635+0000 mon.a (mon.0) 14531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:21 smithi115 bash[45495]: audit 2022-04-22T22:31:21.564635+0000 mon.a (mon.0) 14531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:21 smithi115 bash[46934]: audit 2022-04-22T22:31:21.564635+0000 mon.a (mon.0) 14531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:22 smithi121 bash[36902]: cluster 2022-04-22T22:31:21.573939+0000 mgr.y (mgr.34107) 10409 : cluster [DBG] pgmap v11098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:23.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:22 smithi115 bash[45495]: cluster 2022-04-22T22:31:21.573939+0000 mgr.y (mgr.34107) 10409 : cluster [DBG] pgmap v11098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:22.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:22 smithi115 bash[46934]: cluster 2022-04-22T22:31:21.573939+0000 mgr.y (mgr.34107) 10409 : cluster [DBG] pgmap v11098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:23 smithi121 bash[36902]: audit 2022-04-22T22:31:22.882445+0000 mon.a (mon.0) 14532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:23 smithi115 bash[45495]: audit 2022-04-22T22:31:22.882445+0000 mon.a (mon.0) 14532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:23 smithi115 bash[46934]: audit 2022-04-22T22:31:22.882445+0000 mon.a (mon.0) 14532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:24 smithi121 bash[36902]: cluster 2022-04-22T22:31:23.575108+0000 mgr.y (mgr.34107) 10410 : cluster [DBG] pgmap v11099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:24.671Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:24 smithi115 bash[45495]: cluster 2022-04-22T22:31:23.575108+0000 mgr.y (mgr.34107) 10410 : cluster [DBG] pgmap v11099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:24 smithi115 bash[46934]: cluster 2022-04-22T22:31:23.575108+0000 mgr.y (mgr.34107) 10410 : cluster [DBG] pgmap v11099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:25 smithi121 bash[36902]: audit 2022-04-22T22:31:24.936233+0000 mon.a (mon.0) 14533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:25 smithi115 bash[45495]: audit 2022-04-22T22:31:24.936233+0000 mon.a (mon.0) 14533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:25 smithi115 bash[46934]: audit 2022-04-22T22:31:24.936233+0000 mon.a (mon.0) 14533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:26 smithi121 bash[36902]: cluster 2022-04-22T22:31:25.576204+0000 mgr.y (mgr.34107) 10411 : cluster [DBG] pgmap v11100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:26 smithi121 bash[36902]: audit 2022-04-22T22:31:26.567664+0000 mon.a (mon.0) 14534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:26 smithi115 bash[45495]: cluster 2022-04-22T22:31:25.576204+0000 mgr.y (mgr.34107) 10411 : cluster [DBG] pgmap v11100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:26 smithi115 bash[45495]: audit 2022-04-22T22:31:26.567664+0000 mon.a (mon.0) 14534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:26 smithi115 bash[46934]: cluster 2022-04-22T22:31:25.576204+0000 mgr.y (mgr.34107) 10411 : cluster [DBG] pgmap v11100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:26 smithi115 bash[46934]: audit 2022-04-22T22:31:26.567664+0000 mon.a (mon.0) 14534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:28 smithi121 bash[36902]: cluster 2022-04-22T22:31:27.577329+0000 mgr.y (mgr.34107) 10412 : cluster [DBG] pgmap v11101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:28 smithi121 bash[36902]: audit 2022-04-22T22:31:27.886474+0000 mon.a (mon.0) 14535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:28 smithi115 bash[45495]: cluster 2022-04-22T22:31:27.577329+0000 mgr.y (mgr.34107) 10412 : cluster [DBG] pgmap v11101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:28 smithi115 bash[45495]: audit 2022-04-22T22:31:27.886474+0000 mon.a (mon.0) 14535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:28 smithi115 bash[46934]: cluster 2022-04-22T22:31:27.577329+0000 mgr.y (mgr.34107) 10412 : cluster [DBG] pgmap v11101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:28 smithi115 bash[46934]: audit 2022-04-22T22:31:27.886474+0000 mon.a (mon.0) 14535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:30 smithi121 bash[36902]: cluster 2022-04-22T22:31:29.578434+0000 mgr.y (mgr.34107) 10413 : cluster [DBG] pgmap v11102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:30 smithi121 bash[36902]: audit 2022-04-22T22:31:29.939408+0000 mon.a (mon.0) 14536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:30 smithi115 bash[45495]: cluster 2022-04-22T22:31:29.578434+0000 mgr.y (mgr.34107) 10413 : cluster [DBG] pgmap v11102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:30 smithi115 bash[45495]: audit 2022-04-22T22:31:29.939408+0000 mon.a (mon.0) 14536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:30 smithi115 bash[46934]: cluster 2022-04-22T22:31:29.578434+0000 mgr.y (mgr.34107) 10413 : cluster [DBG] pgmap v11102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:30 smithi115 bash[46934]: audit 2022-04-22T22:31:29.939408+0000 mon.a (mon.0) 14536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:31 smithi121 bash[36902]: audit 2022-04-22T22:31:31.570706+0000 mon.a (mon.0) 14537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:31 smithi115 bash[45495]: audit 2022-04-22T22:31:31.570706+0000 mon.a (mon.0) 14537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:31 smithi115 bash[46934]: audit 2022-04-22T22:31:31.570706+0000 mon.a (mon.0) 14537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:32 smithi121 bash[36902]: cluster 2022-04-22T22:31:31.579402+0000 mgr.y (mgr.34107) 10414 : cluster [DBG] pgmap v11103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:32 smithi115 bash[45495]: cluster 2022-04-22T22:31:31.579402+0000 mgr.y (mgr.34107) 10414 : cluster [DBG] pgmap v11103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:32.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:32 smithi115 bash[46934]: cluster 2022-04-22T22:31:31.579402+0000 mgr.y (mgr.34107) 10414 : cluster [DBG] pgmap v11103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:33 smithi121 bash[36902]: audit 2022-04-22T22:31:32.890533+0000 mon.a (mon.0) 14538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:33 smithi115 bash[45495]: audit 2022-04-22T22:31:32.890533+0000 mon.a (mon.0) 14538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:33 smithi115 bash[46934]: audit 2022-04-22T22:31:32.890533+0000 mon.a (mon.0) 14538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:34 smithi121 bash[36902]: cluster 2022-04-22T22:31:33.580564+0000 mgr.y (mgr.34107) 10415 : cluster [DBG] pgmap v11104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:34 smithi115 bash[45495]: cluster 2022-04-22T22:31:33.580564+0000 mgr.y (mgr.34107) 10415 : cluster [DBG] pgmap v11104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:34 smithi115 bash[46934]: cluster 2022-04-22T22:31:33.580564+0000 mgr.y (mgr.34107) 10415 : cluster [DBG] pgmap v11104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:34.672Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:35 smithi121 bash[36902]: audit 2022-04-22T22:31:34.942531+0000 mon.a (mon.0) 14539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:35 smithi121 bash[36902]: cluster 2022-04-22T22:31:35.581750+0000 mgr.y (mgr.34107) 10416 : cluster [DBG] pgmap v11105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:35 smithi115 bash[45495]: audit 2022-04-22T22:31:34.942531+0000 mon.a (mon.0) 14539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:35 smithi115 bash[45495]: cluster 2022-04-22T22:31:35.581750+0000 mgr.y (mgr.34107) 10416 : cluster [DBG] pgmap v11105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:35 smithi115 bash[46934]: audit 2022-04-22T22:31:34.942531+0000 mon.a (mon.0) 14539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:35 smithi115 bash[46934]: cluster 2022-04-22T22:31:35.581750+0000 mgr.y (mgr.34107) 10416 : cluster [DBG] pgmap v11105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:36 smithi121 bash[36902]: audit 2022-04-22T22:31:36.573895+0000 mon.a (mon.0) 14540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:36 smithi115 bash[45495]: audit 2022-04-22T22:31:36.573895+0000 mon.a (mon.0) 14540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:36 smithi115 bash[46934]: audit 2022-04-22T22:31:36.573895+0000 mon.a (mon.0) 14540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:37 smithi121 bash[36902]: cluster 2022-04-22T22:31:37.582876+0000 mgr.y (mgr.34107) 10417 : cluster [DBG] pgmap v11106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:37 smithi115 bash[45495]: cluster 2022-04-22T22:31:37.582876+0000 mgr.y (mgr.34107) 10417 : cluster [DBG] pgmap v11106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:37 smithi115 bash[46934]: cluster 2022-04-22T22:31:37.582876+0000 mgr.y (mgr.34107) 10417 : cluster [DBG] pgmap v11106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:38 smithi121 bash[36902]: audit 2022-04-22T22:31:37.894285+0000 mon.a (mon.0) 14541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:38 smithi115 bash[45495]: audit 2022-04-22T22:31:37.894285+0000 mon.a (mon.0) 14541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:38 smithi115 bash[46934]: audit 2022-04-22T22:31:37.894285+0000 mon.a (mon.0) 14541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:39 smithi115 bash[45495]: cluster 2022-04-22T22:31:39.584115+0000 mgr.y (mgr.34107) 10418 : cluster [DBG] pgmap v11107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:39 smithi115 bash[46934]: cluster 2022-04-22T22:31:39.584115+0000 mgr.y (mgr.34107) 10418 : cluster [DBG] pgmap v11107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:39 smithi121 bash[36902]: cluster 2022-04-22T22:31:39.584115+0000 mgr.y (mgr.34107) 10418 : cluster [DBG] pgmap v11107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:40 smithi115 bash[45495]: audit 2022-04-22T22:31:39.945688+0000 mon.a (mon.0) 14542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:40 smithi115 bash[46934]: audit 2022-04-22T22:31:39.945688+0000 mon.a (mon.0) 14542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:41.115 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:31:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:40 smithi121 bash[36902]: audit 2022-04-22T22:31:39.945688+0000 mon.a (mon.0) 14542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:41.458 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:31:41.458 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:31:41.458 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:31:41.458 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:31:41.458 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 6m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:31:41.459 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4h) 5m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 6m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 6m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 6m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 6m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:31:41.460 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 6m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:31:41.860 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:31:41.860 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:31:41.861 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:31:41.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:31:41.862 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:31:41.862 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:31:41.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:31:41.862 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:31:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[45495]: audit 2022-04-22T22:31:41.097687+0000 mgr.y (mgr.34107) 10419 : audit [DBG] from='client.46149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[45495]: audit 2022-04-22T22:31:41.434262+0000 mgr.y (mgr.34107) 10420 : audit [DBG] from='client.46155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[45495]: audit 2022-04-22T22:31:41.576759+0000 mon.a (mon.0) 14543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[45495]: cluster 2022-04-22T22:31:41.584915+0000 mgr.y (mgr.34107) 10421 : cluster [DBG] pgmap v11108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[46934]: audit 2022-04-22T22:31:41.097687+0000 mgr.y (mgr.34107) 10419 : audit [DBG] from='client.46149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[46934]: audit 2022-04-22T22:31:41.434262+0000 mgr.y (mgr.34107) 10420 : audit [DBG] from='client.46155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[46934]: audit 2022-04-22T22:31:41.576759+0000 mon.a (mon.0) 14543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:42.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:41 smithi115 bash[46934]: cluster 2022-04-22T22:31:41.584915+0000 mgr.y (mgr.34107) 10421 : cluster [DBG] pgmap v11108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:41 smithi121 bash[36902]: audit 2022-04-22T22:31:41.097687+0000 mgr.y (mgr.34107) 10419 : audit [DBG] from='client.46149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:41 smithi121 bash[36902]: audit 2022-04-22T22:31:41.434262+0000 mgr.y (mgr.34107) 10420 : audit [DBG] from='client.46155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:31:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:41 smithi121 bash[36902]: audit 2022-04-22T22:31:41.576759+0000 mon.a (mon.0) 14543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:41 smithi121 bash[36902]: cluster 2022-04-22T22:31:41.584915+0000 mgr.y (mgr.34107) 10421 : cluster [DBG] pgmap v11108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:42 smithi115 bash[45495]: audit 2022-04-22T22:31:41.856917+0000 mon.a (mon.0) 14544 : audit [DBG] from='client.? 172.21.15.115:0/2110718090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:43.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:42.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:43.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:42 smithi115 bash[46934]: audit 2022-04-22T22:31:41.856917+0000 mon.a (mon.0) 14544 : audit [DBG] from='client.? 172.21.15.115:0/2110718090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:42 smithi121 bash[36902]: audit 2022-04-22T22:31:41.856917+0000 mon.a (mon.0) 14544 : audit [DBG] from='client.? 172.21.15.115:0/2110718090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:31:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:43 smithi115 bash[45495]: audit 2022-04-22T22:31:42.898497+0000 mon.a (mon.0) 14545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:43 smithi115 bash[45495]: cluster 2022-04-22T22:31:43.585952+0000 mgr.y (mgr.34107) 10422 : cluster [DBG] pgmap v11109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:43 smithi115 bash[46934]: audit 2022-04-22T22:31:42.898497+0000 mon.a (mon.0) 14545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:43 smithi115 bash[46934]: cluster 2022-04-22T22:31:43.585952+0000 mgr.y (mgr.34107) 10422 : cluster [DBG] pgmap v11109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:43 smithi121 bash[36902]: audit 2022-04-22T22:31:42.898497+0000 mon.a (mon.0) 14545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:43 smithi121 bash[36902]: cluster 2022-04-22T22:31:43.585952+0000 mgr.y (mgr.34107) 10422 : cluster [DBG] pgmap v11109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:45.004 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:44.673Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:45 smithi121 bash[36902]: audit 2022-04-22T22:31:44.948695+0000 mon.a (mon.0) 14546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:45 smithi115 bash[45495]: audit 2022-04-22T22:31:44.948695+0000 mon.a (mon.0) 14546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:45 smithi115 bash[46934]: audit 2022-04-22T22:31:44.948695+0000 mon.a (mon.0) 14546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:46 smithi115 bash[45495]: cluster 2022-04-22T22:31:45.587001+0000 mgr.y (mgr.34107) 10423 : cluster [DBG] pgmap v11110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:46 smithi115 bash[46934]: cluster 2022-04-22T22:31:45.587001+0000 mgr.y (mgr.34107) 10423 : cluster [DBG] pgmap v11110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:46 smithi121 bash[36902]: cluster 2022-04-22T22:31:45.587001+0000 mgr.y (mgr.34107) 10423 : cluster [DBG] pgmap v11110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:47 smithi115 bash[45495]: audit 2022-04-22T22:31:46.579671+0000 mon.a (mon.0) 14547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:47 smithi115 bash[46934]: audit 2022-04-22T22:31:46.579671+0000 mon.a (mon.0) 14547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:47 smithi121 bash[36902]: audit 2022-04-22T22:31:46.579671+0000 mon.a (mon.0) 14547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:48.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:48 smithi115 bash[46934]: cluster 2022-04-22T22:31:47.588170+0000 mgr.y (mgr.34107) 10424 : cluster [DBG] pgmap v11111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:48 smithi115 bash[46934]: audit 2022-04-22T22:31:47.902099+0000 mon.a (mon.0) 14548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:48 smithi115 bash[45495]: cluster 2022-04-22T22:31:47.588170+0000 mgr.y (mgr.34107) 10424 : cluster [DBG] pgmap v11111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:48 smithi115 bash[45495]: audit 2022-04-22T22:31:47.902099+0000 mon.a (mon.0) 14548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:48 smithi121 bash[36902]: cluster 2022-04-22T22:31:47.588170+0000 mgr.y (mgr.34107) 10424 : cluster [DBG] pgmap v11111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:48 smithi121 bash[36902]: audit 2022-04-22T22:31:47.902099+0000 mon.a (mon.0) 14548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:50 smithi121 bash[36902]: cluster 2022-04-22T22:31:49.589259+0000 mgr.y (mgr.34107) 10425 : cluster [DBG] pgmap v11112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:50 smithi121 bash[36902]: audit 2022-04-22T22:31:49.951609+0000 mon.a (mon.0) 14549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:50 smithi115 bash[45495]: cluster 2022-04-22T22:31:49.589259+0000 mgr.y (mgr.34107) 10425 : cluster [DBG] pgmap v11112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:50 smithi115 bash[45495]: audit 2022-04-22T22:31:49.951609+0000 mon.a (mon.0) 14549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:50 smithi115 bash[46934]: cluster 2022-04-22T22:31:49.589259+0000 mgr.y (mgr.34107) 10425 : cluster [DBG] pgmap v11112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:50 smithi115 bash[46934]: audit 2022-04-22T22:31:49.951609+0000 mon.a (mon.0) 14549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:51 smithi121 bash[36902]: audit 2022-04-22T22:31:51.583058+0000 mon.a (mon.0) 14550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:51 smithi115 bash[45495]: audit 2022-04-22T22:31:51.583058+0000 mon.a (mon.0) 14550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:51 smithi115 bash[46934]: audit 2022-04-22T22:31:51.583058+0000 mon.a (mon.0) 14550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:52 smithi121 bash[36902]: cluster 2022-04-22T22:31:51.590291+0000 mgr.y (mgr.34107) 10426 : cluster [DBG] pgmap v11113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:53.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:52 smithi115 bash[45495]: cluster 2022-04-22T22:31:51.590291+0000 mgr.y (mgr.34107) 10426 : cluster [DBG] pgmap v11113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:52.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:52 smithi115 bash[46934]: cluster 2022-04-22T22:31:51.590291+0000 mgr.y (mgr.34107) 10426 : cluster [DBG] pgmap v11113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:53 smithi121 bash[36902]: audit 2022-04-22T22:31:52.905948+0000 mon.a (mon.0) 14551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:53 smithi115 bash[45495]: audit 2022-04-22T22:31:52.905948+0000 mon.a (mon.0) 14551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:53 smithi115 bash[46934]: audit 2022-04-22T22:31:52.905948+0000 mon.a (mon.0) 14551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:54 smithi121 bash[36902]: cluster 2022-04-22T22:31:53.591444+0000 mgr.y (mgr.34107) 10427 : cluster [DBG] pgmap v11114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:54 smithi115 bash[45495]: cluster 2022-04-22T22:31:53.591444+0000 mgr.y (mgr.34107) 10427 : cluster [DBG] pgmap v11114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:31:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:31:54.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:31:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:54 smithi115 bash[46934]: cluster 2022-04-22T22:31:53.591444+0000 mgr.y (mgr.34107) 10427 : cluster [DBG] pgmap v11114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:55 smithi121 bash[36902]: audit 2022-04-22T22:31:54.954806+0000 mon.a (mon.0) 14552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:55 smithi115 bash[45495]: audit 2022-04-22T22:31:54.954806+0000 mon.a (mon.0) 14552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:55 smithi115 bash[46934]: audit 2022-04-22T22:31:54.954806+0000 mon.a (mon.0) 14552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:31:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:56 smithi121 bash[36902]: cluster 2022-04-22T22:31:55.592488+0000 mgr.y (mgr.34107) 10428 : cluster [DBG] pgmap v11115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:56 smithi121 bash[36902]: audit 2022-04-22T22:31:56.585998+0000 mon.a (mon.0) 14553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:57.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:56 smithi115 bash[46934]: cluster 2022-04-22T22:31:55.592488+0000 mgr.y (mgr.34107) 10428 : cluster [DBG] pgmap v11115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:56 smithi115 bash[46934]: audit 2022-04-22T22:31:56.585998+0000 mon.a (mon.0) 14553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:56 smithi115 bash[45495]: cluster 2022-04-22T22:31:55.592488+0000 mgr.y (mgr.34107) 10428 : cluster [DBG] pgmap v11115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:56 smithi115 bash[45495]: audit 2022-04-22T22:31:56.585998+0000 mon.a (mon.0) 14553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:31:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:58 smithi121 bash[36902]: cluster 2022-04-22T22:31:57.593555+0000 mgr.y (mgr.34107) 10429 : cluster [DBG] pgmap v11116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:31:58 smithi121 bash[36902]: audit 2022-04-22T22:31:57.909728+0000 mon.a (mon.0) 14554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:58 smithi115 bash[45495]: cluster 2022-04-22T22:31:57.593555+0000 mgr.y (mgr.34107) 10429 : cluster [DBG] pgmap v11116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:31:58 smithi115 bash[45495]: audit 2022-04-22T22:31:57.909728+0000 mon.a (mon.0) 14554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:31:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:58 smithi115 bash[46934]: cluster 2022-04-22T22:31:57.593555+0000 mgr.y (mgr.34107) 10429 : cluster [DBG] pgmap v11116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:31:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:31:58 smithi115 bash[46934]: audit 2022-04-22T22:31:57.909728+0000 mon.a (mon.0) 14554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:00 smithi121 bash[36902]: cluster 2022-04-22T22:31:59.594639+0000 mgr.y (mgr.34107) 10430 : cluster [DBG] pgmap v11117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:00 smithi121 bash[36902]: audit 2022-04-22T22:31:59.957366+0000 mon.a (mon.0) 14555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:00 smithi115 bash[45495]: cluster 2022-04-22T22:31:59.594639+0000 mgr.y (mgr.34107) 10430 : cluster [DBG] pgmap v11117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:00 smithi115 bash[45495]: audit 2022-04-22T22:31:59.957366+0000 mon.a (mon.0) 14555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:00 smithi115 bash[46934]: cluster 2022-04-22T22:31:59.594639+0000 mgr.y (mgr.34107) 10430 : cluster [DBG] pgmap v11117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:00 smithi115 bash[46934]: audit 2022-04-22T22:31:59.957366+0000 mon.a (mon.0) 14555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:01 smithi121 bash[36902]: audit 2022-04-22T22:32:01.588938+0000 mon.a (mon.0) 14556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:01 smithi115 bash[45495]: audit 2022-04-22T22:32:01.588938+0000 mon.a (mon.0) 14556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:01 smithi115 bash[46934]: audit 2022-04-22T22:32:01.588938+0000 mon.a (mon.0) 14556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:02 smithi121 bash[36902]: cluster 2022-04-22T22:32:01.595712+0000 mgr.y (mgr.34107) 10431 : cluster [DBG] pgmap v11118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:03.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:02 smithi115 bash[45495]: cluster 2022-04-22T22:32:01.595712+0000 mgr.y (mgr.34107) 10431 : cluster [DBG] pgmap v11118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:02.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:02 smithi115 bash[46934]: cluster 2022-04-22T22:32:01.595712+0000 mgr.y (mgr.34107) 10431 : cluster [DBG] pgmap v11118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:03 smithi121 bash[36902]: audit 2022-04-22T22:32:02.913369+0000 mon.a (mon.0) 14557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:03 smithi115 bash[45495]: audit 2022-04-22T22:32:02.913369+0000 mon.a (mon.0) 14557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:03 smithi115 bash[46934]: audit 2022-04-22T22:32:02.913369+0000 mon.a (mon.0) 14557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:04 smithi121 bash[36902]: cluster 2022-04-22T22:32:03.596899+0000 mgr.y (mgr.34107) 10432 : cluster [DBG] pgmap v11119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:04 smithi115 bash[45495]: cluster 2022-04-22T22:32:03.596899+0000 mgr.y (mgr.34107) 10432 : cluster [DBG] pgmap v11119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:04.675Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:04 smithi115 bash[46934]: cluster 2022-04-22T22:32:03.596899+0000 mgr.y (mgr.34107) 10432 : cluster [DBG] pgmap v11119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:05 smithi121 bash[36902]: audit 2022-04-22T22:32:04.960067+0000 mon.a (mon.0) 14558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:05 smithi115 bash[45495]: audit 2022-04-22T22:32:04.960067+0000 mon.a (mon.0) 14558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:05 smithi115 bash[46934]: audit 2022-04-22T22:32:04.960067+0000 mon.a (mon.0) 14558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:06 smithi121 bash[36902]: cluster 2022-04-22T22:32:05.597929+0000 mgr.y (mgr.34107) 10433 : cluster [DBG] pgmap v11120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:06 smithi121 bash[36902]: audit 2022-04-22T22:32:06.592153+0000 mon.a (mon.0) 14559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:06 smithi115 bash[45495]: cluster 2022-04-22T22:32:05.597929+0000 mgr.y (mgr.34107) 10433 : cluster [DBG] pgmap v11120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:06 smithi115 bash[45495]: audit 2022-04-22T22:32:06.592153+0000 mon.a (mon.0) 14559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:06 smithi115 bash[46934]: cluster 2022-04-22T22:32:05.597929+0000 mgr.y (mgr.34107) 10433 : cluster [DBG] pgmap v11120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:06 smithi115 bash[46934]: audit 2022-04-22T22:32:06.592153+0000 mon.a (mon.0) 14559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:08 smithi121 bash[36902]: cluster 2022-04-22T22:32:07.598959+0000 mgr.y (mgr.34107) 10434 : cluster [DBG] pgmap v11121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:08 smithi121 bash[36902]: audit 2022-04-22T22:32:07.917427+0000 mon.a (mon.0) 14560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:09.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:08 smithi115 bash[46934]: cluster 2022-04-22T22:32:07.598959+0000 mgr.y (mgr.34107) 10434 : cluster [DBG] pgmap v11121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:08 smithi115 bash[46934]: audit 2022-04-22T22:32:07.917427+0000 mon.a (mon.0) 14560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:08 smithi115 bash[45495]: cluster 2022-04-22T22:32:07.598959+0000 mgr.y (mgr.34107) 10434 : cluster [DBG] pgmap v11121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:08 smithi115 bash[45495]: audit 2022-04-22T22:32:07.917427+0000 mon.a (mon.0) 14560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:10 smithi121 bash[36902]: cluster 2022-04-22T22:32:09.600198+0000 mgr.y (mgr.34107) 10435 : cluster [DBG] pgmap v11122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:10 smithi121 bash[36902]: audit 2022-04-22T22:32:09.963122+0000 mon.a (mon.0) 14561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:10 smithi115 bash[46934]: cluster 2022-04-22T22:32:09.600198+0000 mgr.y (mgr.34107) 10435 : cluster [DBG] pgmap v11122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:10 smithi115 bash[46934]: audit 2022-04-22T22:32:09.963122+0000 mon.a (mon.0) 14561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:10 smithi115 bash[45495]: cluster 2022-04-22T22:32:09.600198+0000 mgr.y (mgr.34107) 10435 : cluster [DBG] pgmap v11122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:10 smithi115 bash[45495]: audit 2022-04-22T22:32:09.963122+0000 mon.a (mon.0) 14561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:11 smithi121 bash[36902]: audit 2022-04-22T22:32:11.595189+0000 mon.a (mon.0) 14562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:12.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:11 smithi115 bash[46934]: audit 2022-04-22T22:32:11.595189+0000 mon.a (mon.0) 14562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:11 smithi115 bash[45495]: audit 2022-04-22T22:32:11.595189+0000 mon.a (mon.0) 14562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:12.242 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 5m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 5m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:32:12.579 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 5m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:32:12.580 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:32:12.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (4h) 7m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:32:12.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:32:12.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:32:12.581 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:32:12.581 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:32:12.990 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:32:12.990 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:32:12.990 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:32:12.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:12.990 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:32:12.991 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:32:12.992 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:32:12.992 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:32:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:12 smithi121 bash[36902]: cluster 2022-04-22T22:32:11.601301+0000 mgr.y (mgr.34107) 10436 : cluster [DBG] pgmap v11123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:12 smithi121 bash[36902]: audit 2022-04-22T22:32:12.223954+0000 mgr.y (mgr.34107) 10437 : audit [DBG] from='client.56101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:12 smithi121 bash[36902]: audit 2022-04-22T22:32:12.560529+0000 mgr.y (mgr.34107) 10438 : audit [DBG] from='client.46173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:12.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[45495]: cluster 2022-04-22T22:32:11.601301+0000 mgr.y (mgr.34107) 10436 : cluster [DBG] pgmap v11123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[45495]: audit 2022-04-22T22:32:12.223954+0000 mgr.y (mgr.34107) 10437 : audit [DBG] from='client.56101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[45495]: audit 2022-04-22T22:32:12.560529+0000 mgr.y (mgr.34107) 10438 : audit [DBG] from='client.46173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[46934]: cluster 2022-04-22T22:32:11.601301+0000 mgr.y (mgr.34107) 10436 : cluster [DBG] pgmap v11123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[46934]: audit 2022-04-22T22:32:12.223954+0000 mgr.y (mgr.34107) 10437 : audit [DBG] from='client.56101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:12 smithi115 bash[46934]: audit 2022-04-22T22:32:12.560529+0000 mgr.y (mgr.34107) 10438 : audit [DBG] from='client.46173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:13.750 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:13 smithi121 bash[36902]: audit 2022-04-22T22:32:12.920820+0000 mon.a (mon.0) 14563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:13.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:13 smithi121 bash[36902]: audit 2022-04-22T22:32:12.986372+0000 mon.a (mon.0) 14564 : audit [DBG] from='client.? 172.21.15.115:0/4281744661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:13.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:13 smithi121 bash[36902]: cluster 2022-04-22T22:32:13.602451+0000 mgr.y (mgr.34107) 10439 : cluster [DBG] pgmap v11124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[45495]: audit 2022-04-22T22:32:12.920820+0000 mon.a (mon.0) 14563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[45495]: audit 2022-04-22T22:32:12.986372+0000 mon.a (mon.0) 14564 : audit [DBG] from='client.? 172.21.15.115:0/4281744661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[45495]: cluster 2022-04-22T22:32:13.602451+0000 mgr.y (mgr.34107) 10439 : cluster [DBG] pgmap v11124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[46934]: audit 2022-04-22T22:32:12.920820+0000 mon.a (mon.0) 14563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[46934]: audit 2022-04-22T22:32:12.986372+0000 mon.a (mon.0) 14564 : audit [DBG] from='client.? 172.21.15.115:0/4281744661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:13 smithi115 bash[46934]: cluster 2022-04-22T22:32:13.602451+0000 mgr.y (mgr.34107) 10439 : cluster [DBG] pgmap v11124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:15.022 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:14.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:15 smithi115 bash[45495]: audit 2022-04-22T22:32:14.966460+0000 mon.a (mon.0) 14565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:15 smithi115 bash[46934]: audit 2022-04-22T22:32:14.966460+0000 mon.a (mon.0) 14565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:15 smithi121 bash[36902]: audit 2022-04-22T22:32:14.966460+0000 mon.a (mon.0) 14565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:16 smithi115 bash[45495]: cluster 2022-04-22T22:32:15.603553+0000 mgr.y (mgr.34107) 10440 : cluster [DBG] pgmap v11125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:16 smithi115 bash[46934]: cluster 2022-04-22T22:32:15.603553+0000 mgr.y (mgr.34107) 10440 : cluster [DBG] pgmap v11125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:16 smithi121 bash[36902]: cluster 2022-04-22T22:32:15.603553+0000 mgr.y (mgr.34107) 10440 : cluster [DBG] pgmap v11125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[45495]: audit 2022-04-22T22:32:16.598307+0000 mon.a (mon.0) 14566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[45495]: audit 2022-04-22T22:32:16.797810+0000 mon.a (mon.0) 14567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:32:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[45495]: audit 2022-04-22T22:32:16.801478+0000 mon.a (mon.0) 14568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:32:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[46934]: audit 2022-04-22T22:32:16.598307+0000 mon.a (mon.0) 14566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[46934]: audit 2022-04-22T22:32:16.797810+0000 mon.a (mon.0) 14567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:32:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:17 smithi115 bash[46934]: audit 2022-04-22T22:32:16.801478+0000 mon.a (mon.0) 14568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:32:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:17 smithi121 bash[36902]: audit 2022-04-22T22:32:16.598307+0000 mon.a (mon.0) 14566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:17 smithi121 bash[36902]: audit 2022-04-22T22:32:16.797810+0000 mon.a (mon.0) 14567 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:32:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:17 smithi121 bash[36902]: audit 2022-04-22T22:32:16.801478+0000 mon.a (mon.0) 14568 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:32:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:18 smithi121 bash[36902]: cluster 2022-04-22T22:32:17.604575+0000 mgr.y (mgr.34107) 10441 : cluster [DBG] pgmap v11126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:18 smithi121 bash[36902]: audit 2022-04-22T22:32:17.924242+0000 mon.a (mon.0) 14569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:18 smithi115 bash[45495]: cluster 2022-04-22T22:32:17.604575+0000 mgr.y (mgr.34107) 10441 : cluster [DBG] pgmap v11126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:18 smithi115 bash[45495]: audit 2022-04-22T22:32:17.924242+0000 mon.a (mon.0) 14569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:18 smithi115 bash[46934]: cluster 2022-04-22T22:32:17.604575+0000 mgr.y (mgr.34107) 10441 : cluster [DBG] pgmap v11126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:18 smithi115 bash[46934]: audit 2022-04-22T22:32:17.924242+0000 mon.a (mon.0) 14569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:19 smithi121 bash[36902]: audit 2022-04-22T22:32:19.293560+0000 mon.a (mon.0) 14570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:32:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:19 smithi121 bash[36902]: audit 2022-04-22T22:32:19.296599+0000 mon.a (mon.0) 14571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:32:19.610 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:19 smithi121 bash[36902]: audit 2022-04-22T22:32:19.298206+0000 mon.a (mon.0) 14572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:32:19.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[46934]: audit 2022-04-22T22:32:19.293560+0000 mon.a (mon.0) 14570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:32:19.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[46934]: audit 2022-04-22T22:32:19.296599+0000 mon.a (mon.0) 14571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:32:19.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[46934]: audit 2022-04-22T22:32:19.298206+0000 mon.a (mon.0) 14572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:32:19.627 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[45495]: audit 2022-04-22T22:32:19.293560+0000 mon.a (mon.0) 14570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:32:19.627 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[45495]: audit 2022-04-22T22:32:19.296599+0000 mon.a (mon.0) 14571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:32:19.627 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:19 smithi115 bash[45495]: audit 2022-04-22T22:32:19.298206+0000 mon.a (mon.0) 14572 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:32:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:20 smithi121 bash[36902]: cluster 2022-04-22T22:32:19.605245+0000 mgr.y (mgr.34107) 10442 : cluster [DBG] pgmap v11127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:20 smithi121 bash[36902]: audit 2022-04-22T22:32:19.635221+0000 mon.a (mon.0) 14573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:32:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:20 smithi121 bash[36902]: audit 2022-04-22T22:32:19.969461+0000 mon.a (mon.0) 14574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[45495]: cluster 2022-04-22T22:32:19.605245+0000 mgr.y (mgr.34107) 10442 : cluster [DBG] pgmap v11127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[45495]: audit 2022-04-22T22:32:19.635221+0000 mon.a (mon.0) 14573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:32:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[45495]: audit 2022-04-22T22:32:19.969461+0000 mon.a (mon.0) 14574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[46934]: cluster 2022-04-22T22:32:19.605245+0000 mgr.y (mgr.34107) 10442 : cluster [DBG] pgmap v11127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[46934]: audit 2022-04-22T22:32:19.635221+0000 mon.a (mon.0) 14573 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:32:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:20 smithi115 bash[46934]: audit 2022-04-22T22:32:19.969461+0000 mon.a (mon.0) 14574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:21 smithi121 bash[36902]: audit 2022-04-22T22:32:21.601538+0000 mon.a (mon.0) 14575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:21 smithi115 bash[45495]: audit 2022-04-22T22:32:21.601538+0000 mon.a (mon.0) 14575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:21 smithi115 bash[46934]: audit 2022-04-22T22:32:21.601538+0000 mon.a (mon.0) 14575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:22 smithi121 bash[36902]: cluster 2022-04-22T22:32:21.606232+0000 mgr.y (mgr.34107) 10443 : cluster [DBG] pgmap v11128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:23.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:22 smithi115 bash[45495]: cluster 2022-04-22T22:32:21.606232+0000 mgr.y (mgr.34107) 10443 : cluster [DBG] pgmap v11128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:22 smithi115 bash[46934]: cluster 2022-04-22T22:32:21.606232+0000 mgr.y (mgr.34107) 10443 : cluster [DBG] pgmap v11128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:22.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:23 smithi121 bash[36902]: audit 2022-04-22T22:32:22.927963+0000 mon.a (mon.0) 14576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:23 smithi115 bash[45495]: audit 2022-04-22T22:32:22.927963+0000 mon.a (mon.0) 14576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:23 smithi115 bash[46934]: audit 2022-04-22T22:32:22.927963+0000 mon.a (mon.0) 14576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:24 smithi121 bash[36902]: cluster 2022-04-22T22:32:23.607360+0000 mgr.y (mgr.34107) 10444 : cluster [DBG] pgmap v11129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:24 smithi115 bash[45495]: cluster 2022-04-22T22:32:23.607360+0000 mgr.y (mgr.34107) 10444 : cluster [DBG] pgmap v11129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:24.674Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:24 smithi115 bash[46934]: cluster 2022-04-22T22:32:23.607360+0000 mgr.y (mgr.34107) 10444 : cluster [DBG] pgmap v11129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:25 smithi121 bash[36902]: audit 2022-04-22T22:32:24.972211+0000 mon.a (mon.0) 14577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:25 smithi115 bash[45495]: audit 2022-04-22T22:32:24.972211+0000 mon.a (mon.0) 14577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:25 smithi115 bash[46934]: audit 2022-04-22T22:32:24.972211+0000 mon.a (mon.0) 14577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:26 smithi121 bash[36902]: cluster 2022-04-22T22:32:25.608434+0000 mgr.y (mgr.34107) 10445 : cluster [DBG] pgmap v11130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:26 smithi121 bash[36902]: audit 2022-04-22T22:32:26.604628+0000 mon.a (mon.0) 14578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:26 smithi115 bash[45495]: cluster 2022-04-22T22:32:25.608434+0000 mgr.y (mgr.34107) 10445 : cluster [DBG] pgmap v11130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:26 smithi115 bash[45495]: audit 2022-04-22T22:32:26.604628+0000 mon.a (mon.0) 14578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:26 smithi115 bash[46934]: cluster 2022-04-22T22:32:25.608434+0000 mgr.y (mgr.34107) 10445 : cluster [DBG] pgmap v11130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:26 smithi115 bash[46934]: audit 2022-04-22T22:32:26.604628+0000 mon.a (mon.0) 14578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:28 smithi121 bash[36902]: cluster 2022-04-22T22:32:27.609554+0000 mgr.y (mgr.34107) 10446 : cluster [DBG] pgmap v11131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:28 smithi121 bash[36902]: audit 2022-04-22T22:32:27.931811+0000 mon.a (mon.0) 14579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:28 smithi115 bash[45495]: cluster 2022-04-22T22:32:27.609554+0000 mgr.y (mgr.34107) 10446 : cluster [DBG] pgmap v11131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:28 smithi115 bash[45495]: audit 2022-04-22T22:32:27.931811+0000 mon.a (mon.0) 14579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:28 smithi115 bash[46934]: cluster 2022-04-22T22:32:27.609554+0000 mgr.y (mgr.34107) 10446 : cluster [DBG] pgmap v11131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:28 smithi115 bash[46934]: audit 2022-04-22T22:32:27.931811+0000 mon.a (mon.0) 14579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:30 smithi121 bash[36902]: cluster 2022-04-22T22:32:29.610621+0000 mgr.y (mgr.34107) 10447 : cluster [DBG] pgmap v11132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:30 smithi121 bash[36902]: audit 2022-04-22T22:32:29.975153+0000 mon.a (mon.0) 14580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:30 smithi115 bash[45495]: cluster 2022-04-22T22:32:29.610621+0000 mgr.y (mgr.34107) 10447 : cluster [DBG] pgmap v11132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:30 smithi115 bash[45495]: audit 2022-04-22T22:32:29.975153+0000 mon.a (mon.0) 14580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:30 smithi115 bash[46934]: cluster 2022-04-22T22:32:29.610621+0000 mgr.y (mgr.34107) 10447 : cluster [DBG] pgmap v11132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:30 smithi115 bash[46934]: audit 2022-04-22T22:32:29.975153+0000 mon.a (mon.0) 14580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:31 smithi121 bash[36902]: audit 2022-04-22T22:32:31.607816+0000 mon.a (mon.0) 14581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:31 smithi115 bash[45495]: audit 2022-04-22T22:32:31.607816+0000 mon.a (mon.0) 14581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:31 smithi115 bash[46934]: audit 2022-04-22T22:32:31.607816+0000 mon.a (mon.0) 14581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:32 smithi121 bash[36902]: cluster 2022-04-22T22:32:31.611708+0000 mgr.y (mgr.34107) 10448 : cluster [DBG] pgmap v11133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:32 smithi115 bash[45495]: cluster 2022-04-22T22:32:31.611708+0000 mgr.y (mgr.34107) 10448 : cluster [DBG] pgmap v11133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:32.823Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:32 smithi115 bash[46934]: cluster 2022-04-22T22:32:31.611708+0000 mgr.y (mgr.34107) 10448 : cluster [DBG] pgmap v11133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:33 smithi121 bash[36902]: audit 2022-04-22T22:32:32.935525+0000 mon.a (mon.0) 14582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:33 smithi115 bash[45495]: audit 2022-04-22T22:32:32.935525+0000 mon.a (mon.0) 14582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:33 smithi115 bash[46934]: audit 2022-04-22T22:32:32.935525+0000 mon.a (mon.0) 14582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:34 smithi121 bash[36902]: cluster 2022-04-22T22:32:33.612805+0000 mgr.y (mgr.34107) 10449 : cluster [DBG] pgmap v11134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:34 smithi115 bash[46934]: cluster 2022-04-22T22:32:33.612805+0000 mgr.y (mgr.34107) 10449 : cluster [DBG] pgmap v11134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:34.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:34 smithi115 bash[45495]: cluster 2022-04-22T22:32:33.612805+0000 mgr.y (mgr.34107) 10449 : cluster [DBG] pgmap v11134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:35 smithi121 bash[36902]: audit 2022-04-22T22:32:34.978110+0000 mon.a (mon.0) 14583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:36.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:35 smithi115 bash[46934]: audit 2022-04-22T22:32:34.978110+0000 mon.a (mon.0) 14583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:35 smithi115 bash[45495]: audit 2022-04-22T22:32:34.978110+0000 mon.a (mon.0) 14583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:36 smithi121 bash[36902]: cluster 2022-04-22T22:32:35.613876+0000 mgr.y (mgr.34107) 10450 : cluster [DBG] pgmap v11135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:36 smithi121 bash[36902]: audit 2022-04-22T22:32:36.611155+0000 mon.a (mon.0) 14584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:36 smithi115 bash[45495]: cluster 2022-04-22T22:32:35.613876+0000 mgr.y (mgr.34107) 10450 : cluster [DBG] pgmap v11135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:36 smithi115 bash[45495]: audit 2022-04-22T22:32:36.611155+0000 mon.a (mon.0) 14584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:36 smithi115 bash[46934]: cluster 2022-04-22T22:32:35.613876+0000 mgr.y (mgr.34107) 10450 : cluster [DBG] pgmap v11135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:36 smithi115 bash[46934]: audit 2022-04-22T22:32:36.611155+0000 mon.a (mon.0) 14584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:38 smithi121 bash[36902]: cluster 2022-04-22T22:32:37.614948+0000 mgr.y (mgr.34107) 10451 : cluster [DBG] pgmap v11136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:38 smithi121 bash[36902]: audit 2022-04-22T22:32:37.939244+0000 mon.a (mon.0) 14585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:38 smithi115 bash[46934]: cluster 2022-04-22T22:32:37.614948+0000 mgr.y (mgr.34107) 10451 : cluster [DBG] pgmap v11136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:38 smithi115 bash[46934]: audit 2022-04-22T22:32:37.939244+0000 mon.a (mon.0) 14585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:38 smithi115 bash[45495]: cluster 2022-04-22T22:32:37.614948+0000 mgr.y (mgr.34107) 10451 : cluster [DBG] pgmap v11136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:38 smithi115 bash[45495]: audit 2022-04-22T22:32:37.939244+0000 mon.a (mon.0) 14585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:40 smithi121 bash[36902]: cluster 2022-04-22T22:32:39.616068+0000 mgr.y (mgr.34107) 10452 : cluster [DBG] pgmap v11137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:40 smithi121 bash[36902]: audit 2022-04-22T22:32:39.981025+0000 mon.a (mon.0) 14586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:40 smithi115 bash[45495]: cluster 2022-04-22T22:32:39.616068+0000 mgr.y (mgr.34107) 10452 : cluster [DBG] pgmap v11137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:40 smithi115 bash[45495]: audit 2022-04-22T22:32:39.981025+0000 mon.a (mon.0) 14586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:40 smithi115 bash[46934]: cluster 2022-04-22T22:32:39.616068+0000 mgr.y (mgr.34107) 10452 : cluster [DBG] pgmap v11137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:40 smithi115 bash[46934]: audit 2022-04-22T22:32:39.981025+0000 mon.a (mon.0) 14586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:41 smithi121 bash[36902]: audit 2022-04-22T22:32:41.614465+0000 mon.a (mon.0) 14587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:41 smithi115 bash[45495]: audit 2022-04-22T22:32:41.614465+0000 mon.a (mon.0) 14587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:41 smithi115 bash[46934]: audit 2022-04-22T22:32:41.614465+0000 mon.a (mon.0) 14587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:42 smithi121 bash[36902]: cluster 2022-04-22T22:32:41.617112+0000 mgr.y (mgr.34107) 10453 : cluster [DBG] pgmap v11138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:43.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:42 smithi115 bash[46934]: cluster 2022-04-22T22:32:41.617112+0000 mgr.y (mgr.34107) 10453 : cluster [DBG] pgmap v11138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:42.824Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:42 smithi115 bash[45495]: cluster 2022-04-22T22:32:41.617112+0000 mgr.y (mgr.34107) 10453 : cluster [DBG] pgmap v11138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:43.367 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:32:43.710 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:32:43.711 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 7m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (4h) 7m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 7m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:32:43.712 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 7m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:32:43.713 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 7m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:32:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:43 smithi121 bash[36902]: audit 2022-04-22T22:32:42.942992+0000 mon.a (mon.0) 14588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:43 smithi115 bash[46934]: audit 2022-04-22T22:32:42.942992+0000 mon.a (mon.0) 14588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:43 smithi115 bash[45495]: audit 2022-04-22T22:32:42.942992+0000 mon.a (mon.0) 14588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:44.112 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:32:44.112 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:44.113 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:32:44.114 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:32:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:44 smithi121 bash[36902]: audit 2022-04-22T22:32:43.348364+0000 mgr.y (mgr.34107) 10454 : audit [DBG] from='client.56119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:44 smithi121 bash[36902]: cluster 2022-04-22T22:32:43.618211+0000 mgr.y (mgr.34107) 10455 : cluster [DBG] pgmap v11139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:44 smithi121 bash[36902]: audit 2022-04-22T22:32:43.685600+0000 mgr.y (mgr.34107) 10456 : audit [DBG] from='client.46191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:44 smithi121 bash[36902]: audit 2022-04-22T22:32:44.108897+0000 mon.a (mon.0) 14589 : audit [DBG] from='client.? 172.21.15.115:0/3761964264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:44.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[45495]: audit 2022-04-22T22:32:43.348364+0000 mgr.y (mgr.34107) 10454 : audit [DBG] from='client.56119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[45495]: cluster 2022-04-22T22:32:43.618211+0000 mgr.y (mgr.34107) 10455 : cluster [DBG] pgmap v11139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[45495]: audit 2022-04-22T22:32:43.685600+0000 mgr.y (mgr.34107) 10456 : audit [DBG] from='client.46191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[45495]: audit 2022-04-22T22:32:44.108897+0000 mon.a (mon.0) 14589 : audit [DBG] from='client.? 172.21.15.115:0/3761964264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[46934]: audit 2022-04-22T22:32:43.348364+0000 mgr.y (mgr.34107) 10454 : audit [DBG] from='client.56119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[46934]: cluster 2022-04-22T22:32:43.618211+0000 mgr.y (mgr.34107) 10455 : cluster [DBG] pgmap v11139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[46934]: audit 2022-04-22T22:32:43.685600+0000 mgr.y (mgr.34107) 10456 : audit [DBG] from='client.46191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:32:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:44 smithi115 bash[46934]: audit 2022-04-22T22:32:44.108897+0000 mon.a (mon.0) 14589 : audit [DBG] from='client.? 172.21.15.115:0/3761964264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:32:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:45 smithi121 bash[36902]: audit 2022-04-22T22:32:44.984063+0000 mon.a (mon.0) 14590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:45 smithi121 bash[36902]: cluster 2022-04-22T22:32:45.619354+0000 mgr.y (mgr.34107) 10457 : cluster [DBG] pgmap v11140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:45 smithi115 bash[45495]: audit 2022-04-22T22:32:44.984063+0000 mon.a (mon.0) 14590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:45 smithi115 bash[45495]: cluster 2022-04-22T22:32:45.619354+0000 mgr.y (mgr.34107) 10457 : cluster [DBG] pgmap v11140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:45 smithi115 bash[46934]: audit 2022-04-22T22:32:44.984063+0000 mon.a (mon.0) 14590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:45 smithi115 bash[46934]: cluster 2022-04-22T22:32:45.619354+0000 mgr.y (mgr.34107) 10457 : cluster [DBG] pgmap v11140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:46 smithi121 bash[36902]: audit 2022-04-22T22:32:46.617669+0000 mon.a (mon.0) 14591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:46 smithi115 bash[45495]: audit 2022-04-22T22:32:46.617669+0000 mon.a (mon.0) 14591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:46 smithi115 bash[46934]: audit 2022-04-22T22:32:46.617669+0000 mon.a (mon.0) 14591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:47 smithi121 bash[36902]: cluster 2022-04-22T22:32:47.620415+0000 mgr.y (mgr.34107) 10458 : cluster [DBG] pgmap v11141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:47 smithi115 bash[45495]: cluster 2022-04-22T22:32:47.620415+0000 mgr.y (mgr.34107) 10458 : cluster [DBG] pgmap v11141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:47 smithi115 bash[46934]: cluster 2022-04-22T22:32:47.620415+0000 mgr.y (mgr.34107) 10458 : cluster [DBG] pgmap v11141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:48 smithi121 bash[36902]: audit 2022-04-22T22:32:47.946689+0000 mon.a (mon.0) 14592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:48 smithi115 bash[45495]: audit 2022-04-22T22:32:47.946689+0000 mon.a (mon.0) 14592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:48 smithi115 bash[46934]: audit 2022-04-22T22:32:47.946689+0000 mon.a (mon.0) 14592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:49 smithi115 bash[46934]: cluster 2022-04-22T22:32:49.621569+0000 mgr.y (mgr.34107) 10459 : cluster [DBG] pgmap v11142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:49 smithi115 bash[45495]: cluster 2022-04-22T22:32:49.621569+0000 mgr.y (mgr.34107) 10459 : cluster [DBG] pgmap v11142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:49 smithi121 bash[36902]: cluster 2022-04-22T22:32:49.621569+0000 mgr.y (mgr.34107) 10459 : cluster [DBG] pgmap v11142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:50 smithi115 bash[45495]: audit 2022-04-22T22:32:49.987380+0000 mon.a (mon.0) 14593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:50 smithi115 bash[46934]: audit 2022-04-22T22:32:49.987380+0000 mon.a (mon.0) 14593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:50 smithi121 bash[36902]: audit 2022-04-22T22:32:49.987380+0000 mon.a (mon.0) 14593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:51 smithi115 bash[45495]: audit 2022-04-22T22:32:51.620881+0000 mon.a (mon.0) 14594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:51 smithi115 bash[45495]: cluster 2022-04-22T22:32:51.622571+0000 mgr.y (mgr.34107) 10460 : cluster [DBG] pgmap v11143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:51 smithi115 bash[46934]: audit 2022-04-22T22:32:51.620881+0000 mon.a (mon.0) 14594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:51 smithi115 bash[46934]: cluster 2022-04-22T22:32:51.622571+0000 mgr.y (mgr.34107) 10460 : cluster [DBG] pgmap v11143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:51 smithi121 bash[36902]: audit 2022-04-22T22:32:51.620881+0000 mon.a (mon.0) 14594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:51 smithi121 bash[36902]: cluster 2022-04-22T22:32:51.622571+0000 mgr.y (mgr.34107) 10460 : cluster [DBG] pgmap v11143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:53 smithi121 bash[36902]: audit 2022-04-22T22:32:52.950275+0000 mon.a (mon.0) 14595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:53 smithi115 bash[46934]: audit 2022-04-22T22:32:52.950275+0000 mon.a (mon.0) 14595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:52.825Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:53 smithi115 bash[45495]: audit 2022-04-22T22:32:52.950275+0000 mon.a (mon.0) 14595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:54 smithi115 bash[45495]: cluster 2022-04-22T22:32:53.623747+0000 mgr.y (mgr.34107) 10461 : cluster [DBG] pgmap v11144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:54 smithi115 bash[46934]: cluster 2022-04-22T22:32:53.623747+0000 mgr.y (mgr.34107) 10461 : cluster [DBG] pgmap v11144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:54 smithi121 bash[36902]: cluster 2022-04-22T22:32:53.623747+0000 mgr.y (mgr.34107) 10461 : cluster [DBG] pgmap v11144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:55.018 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:32:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:32:54.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:32:55.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:55 smithi115 bash[46934]: audit 2022-04-22T22:32:54.990568+0000 mon.a (mon.0) 14596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:55.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:55 smithi115 bash[45495]: audit 2022-04-22T22:32:54.990568+0000 mon.a (mon.0) 14596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:55 smithi121 bash[36902]: audit 2022-04-22T22:32:54.990568+0000 mon.a (mon.0) 14596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:32:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:56 smithi115 bash[45495]: cluster 2022-04-22T22:32:55.624788+0000 mgr.y (mgr.34107) 10462 : cluster [DBG] pgmap v11145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:56 smithi115 bash[46934]: cluster 2022-04-22T22:32:55.624788+0000 mgr.y (mgr.34107) 10462 : cluster [DBG] pgmap v11145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:56 smithi121 bash[36902]: cluster 2022-04-22T22:32:55.624788+0000 mgr.y (mgr.34107) 10462 : cluster [DBG] pgmap v11145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:57 smithi115 bash[45495]: audit 2022-04-22T22:32:56.624459+0000 mon.a (mon.0) 14597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:57 smithi115 bash[46934]: audit 2022-04-22T22:32:56.624459+0000 mon.a (mon.0) 14597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:57 smithi121 bash[36902]: audit 2022-04-22T22:32:56.624459+0000 mon.a (mon.0) 14597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:32:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:58 smithi121 bash[36902]: cluster 2022-04-22T22:32:57.625932+0000 mgr.y (mgr.34107) 10463 : cluster [DBG] pgmap v11146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:32:58 smithi121 bash[36902]: audit 2022-04-22T22:32:57.954063+0000 mon.a (mon.0) 14598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:58 smithi115 bash[45495]: cluster 2022-04-22T22:32:57.625932+0000 mgr.y (mgr.34107) 10463 : cluster [DBG] pgmap v11146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:32:58 smithi115 bash[45495]: audit 2022-04-22T22:32:57.954063+0000 mon.a (mon.0) 14598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:32:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:58 smithi115 bash[46934]: cluster 2022-04-22T22:32:57.625932+0000 mgr.y (mgr.34107) 10463 : cluster [DBG] pgmap v11146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:32:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:32:58 smithi115 bash[46934]: audit 2022-04-22T22:32:57.954063+0000 mon.a (mon.0) 14598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:00 smithi121 bash[36902]: cluster 2022-04-22T22:32:59.627079+0000 mgr.y (mgr.34107) 10464 : cluster [DBG] pgmap v11147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:00 smithi121 bash[36902]: audit 2022-04-22T22:32:59.993873+0000 mon.a (mon.0) 14599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:00 smithi115 bash[45495]: cluster 2022-04-22T22:32:59.627079+0000 mgr.y (mgr.34107) 10464 : cluster [DBG] pgmap v11147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:00 smithi115 bash[45495]: audit 2022-04-22T22:32:59.993873+0000 mon.a (mon.0) 14599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:00 smithi115 bash[46934]: cluster 2022-04-22T22:32:59.627079+0000 mgr.y (mgr.34107) 10464 : cluster [DBG] pgmap v11147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:00 smithi115 bash[46934]: audit 2022-04-22T22:32:59.993873+0000 mon.a (mon.0) 14599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:01 smithi121 bash[36902]: audit 2022-04-22T22:33:01.630870+0000 mon.a (mon.0) 14600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:01 smithi115 bash[45495]: audit 2022-04-22T22:33:01.630870+0000 mon.a (mon.0) 14600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:01 smithi115 bash[46934]: audit 2022-04-22T22:33:01.630870+0000 mon.a (mon.0) 14600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:02 smithi121 bash[36902]: cluster 2022-04-22T22:33:01.628140+0000 mgr.y (mgr.34107) 10465 : cluster [DBG] pgmap v11148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:02.826Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:02 smithi115 bash[45495]: cluster 2022-04-22T22:33:01.628140+0000 mgr.y (mgr.34107) 10465 : cluster [DBG] pgmap v11148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:02 smithi115 bash[46934]: cluster 2022-04-22T22:33:01.628140+0000 mgr.y (mgr.34107) 10465 : cluster [DBG] pgmap v11148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:03 smithi121 bash[36902]: audit 2022-04-22T22:33:02.957730+0000 mon.a (mon.0) 14601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:03 smithi115 bash[45495]: audit 2022-04-22T22:33:02.957730+0000 mon.a (mon.0) 14601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:03 smithi115 bash[46934]: audit 2022-04-22T22:33:02.957730+0000 mon.a (mon.0) 14601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:04 smithi121 bash[36902]: cluster 2022-04-22T22:33:03.629261+0000 mgr.y (mgr.34107) 10466 : cluster [DBG] pgmap v11149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:04 smithi115 bash[46934]: cluster 2022-04-22T22:33:03.629261+0000 mgr.y (mgr.34107) 10466 : cluster [DBG] pgmap v11149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:04.680Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:05.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:04 smithi115 bash[45495]: cluster 2022-04-22T22:33:03.629261+0000 mgr.y (mgr.34107) 10466 : cluster [DBG] pgmap v11149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:05 smithi121 bash[36902]: audit 2022-04-22T22:33:04.996768+0000 mon.a (mon.0) 14602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:05 smithi115 bash[45495]: audit 2022-04-22T22:33:04.996768+0000 mon.a (mon.0) 14602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:05 smithi115 bash[46934]: audit 2022-04-22T22:33:04.996768+0000 mon.a (mon.0) 14602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:06 smithi121 bash[36902]: cluster 2022-04-22T22:33:05.630279+0000 mgr.y (mgr.34107) 10467 : cluster [DBG] pgmap v11150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:06 smithi121 bash[36902]: audit 2022-04-22T22:33:06.636730+0000 mon.a (mon.0) 14603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:06 smithi115 bash[45495]: cluster 2022-04-22T22:33:05.630279+0000 mgr.y (mgr.34107) 10467 : cluster [DBG] pgmap v11150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:06 smithi115 bash[45495]: audit 2022-04-22T22:33:06.636730+0000 mon.a (mon.0) 14603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:06 smithi115 bash[46934]: cluster 2022-04-22T22:33:05.630279+0000 mgr.y (mgr.34107) 10467 : cluster [DBG] pgmap v11150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:06 smithi115 bash[46934]: audit 2022-04-22T22:33:06.636730+0000 mon.a (mon.0) 14603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:08 smithi121 bash[36902]: cluster 2022-04-22T22:33:07.631361+0000 mgr.y (mgr.34107) 10468 : cluster [DBG] pgmap v11151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:08 smithi121 bash[36902]: audit 2022-04-22T22:33:07.961473+0000 mon.a (mon.0) 14604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:08 smithi115 bash[45495]: cluster 2022-04-22T22:33:07.631361+0000 mgr.y (mgr.34107) 10468 : cluster [DBG] pgmap v11151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:08 smithi115 bash[45495]: audit 2022-04-22T22:33:07.961473+0000 mon.a (mon.0) 14604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:08 smithi115 bash[46934]: cluster 2022-04-22T22:33:07.631361+0000 mgr.y (mgr.34107) 10468 : cluster [DBG] pgmap v11151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:08 smithi115 bash[46934]: audit 2022-04-22T22:33:07.961473+0000 mon.a (mon.0) 14604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:10 smithi121 bash[36902]: cluster 2022-04-22T22:33:09.632464+0000 mgr.y (mgr.34107) 10469 : cluster [DBG] pgmap v11152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:10 smithi121 bash[36902]: audit 2022-04-22T22:33:09.999468+0000 mon.a (mon.0) 14605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:10 smithi115 bash[45495]: cluster 2022-04-22T22:33:09.632464+0000 mgr.y (mgr.34107) 10469 : cluster [DBG] pgmap v11152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:10 smithi115 bash[45495]: audit 2022-04-22T22:33:09.999468+0000 mon.a (mon.0) 14605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:10 smithi115 bash[46934]: cluster 2022-04-22T22:33:09.632464+0000 mgr.y (mgr.34107) 10469 : cluster [DBG] pgmap v11152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:11.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:10 smithi115 bash[46934]: audit 2022-04-22T22:33:09.999468+0000 mon.a (mon.0) 14605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:12 smithi121 bash[36902]: cluster 2022-04-22T22:33:11.633647+0000 mgr.y (mgr.34107) 10470 : cluster [DBG] pgmap v11153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:12 smithi121 bash[36902]: audit 2022-04-22T22:33:11.639665+0000 mon.a (mon.0) 14606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:12.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:12 smithi115 bash[45495]: cluster 2022-04-22T22:33:11.633647+0000 mgr.y (mgr.34107) 10470 : cluster [DBG] pgmap v11153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:12 smithi115 bash[45495]: audit 2022-04-22T22:33:11.639665+0000 mon.a (mon.0) 14606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:12 smithi115 bash[46934]: cluster 2022-04-22T22:33:11.633647+0000 mgr.y (mgr.34107) 10470 : cluster [DBG] pgmap v11153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:12 smithi115 bash[46934]: audit 2022-04-22T22:33:11.639665+0000 mon.a (mon.0) 14606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:13 smithi121 bash[36902]: audit 2022-04-22T22:33:12.964891+0000 mon.a (mon.0) 14607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:13 smithi115 bash[45495]: audit 2022-04-22T22:33:12.964891+0000 mon.a (mon.0) 14607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:13 smithi115 bash[46934]: audit 2022-04-22T22:33:12.964891+0000 mon.a (mon.0) 14607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:14.492 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 6m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 6m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:33:14.830 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 6m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:33:14.831 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:33:14.832 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:33:14.832 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:33:14.832 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:33:14.832 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:33:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:14 smithi121 bash[36902]: cluster 2022-04-22T22:33:13.634947+0000 mgr.y (mgr.34107) 10471 : cluster [DBG] pgmap v11154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:14.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:14 smithi115 bash[45495]: cluster 2022-04-22T22:33:13.634947+0000 mgr.y (mgr.34107) 10471 : cluster [DBG] pgmap v11154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:14 smithi115 bash[46934]: cluster 2022-04-22T22:33:13.634947+0000 mgr.y (mgr.34107) 10471 : cluster [DBG] pgmap v11154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:15.238 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:33:15.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:33:15.240 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:33:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:15 smithi121 bash[36902]: audit 2022-04-22T22:33:14.469178+0000 mgr.y (mgr.34107) 10472 : audit [DBG] from='client.56137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:15 smithi121 bash[36902]: audit 2022-04-22T22:33:15.002349+0000 mon.a (mon.0) 14608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:15 smithi121 bash[36902]: audit 2022-04-22T22:33:15.235093+0000 mon.a (mon.0) 14609 : audit [DBG] from='client.? 172.21.15.115:0/952443568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[45495]: audit 2022-04-22T22:33:14.469178+0000 mgr.y (mgr.34107) 10472 : audit [DBG] from='client.56137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[45495]: audit 2022-04-22T22:33:15.002349+0000 mon.a (mon.0) 14608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[45495]: audit 2022-04-22T22:33:15.235093+0000 mon.a (mon.0) 14609 : audit [DBG] from='client.? 172.21.15.115:0/952443568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[46934]: audit 2022-04-22T22:33:14.469178+0000 mgr.y (mgr.34107) 10472 : audit [DBG] from='client.56137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[46934]: audit 2022-04-22T22:33:15.002349+0000 mon.a (mon.0) 14608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:15 smithi115 bash[46934]: audit 2022-04-22T22:33:15.235093+0000 mon.a (mon.0) 14609 : audit [DBG] from='client.? 172.21.15.115:0/952443568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:16 smithi121 bash[36902]: audit 2022-04-22T22:33:14.812501+0000 mgr.y (mgr.34107) 10473 : audit [DBG] from='client.46209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:16 smithi121 bash[36902]: cluster 2022-04-22T22:33:15.636054+0000 mgr.y (mgr.34107) 10474 : cluster [DBG] pgmap v11155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:16 smithi121 bash[36902]: audit 2022-04-22T22:33:16.641637+0000 mon.a (mon.0) 14610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[45495]: audit 2022-04-22T22:33:14.812501+0000 mgr.y (mgr.34107) 10473 : audit [DBG] from='client.46209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[45495]: cluster 2022-04-22T22:33:15.636054+0000 mgr.y (mgr.34107) 10474 : cluster [DBG] pgmap v11155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[45495]: audit 2022-04-22T22:33:16.641637+0000 mon.a (mon.0) 14610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[46934]: audit 2022-04-22T22:33:14.812501+0000 mgr.y (mgr.34107) 10473 : audit [DBG] from='client.46209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[46934]: cluster 2022-04-22T22:33:15.636054+0000 mgr.y (mgr.34107) 10474 : cluster [DBG] pgmap v11155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:16 smithi115 bash[46934]: audit 2022-04-22T22:33:16.641637+0000 mon.a (mon.0) 14610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:17 smithi121 bash[36902]: audit 2022-04-22T22:33:16.811560+0000 mon.a (mon.0) 14611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:33:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:17 smithi121 bash[36902]: audit 2022-04-22T22:33:16.812197+0000 mon.a (mon.0) 14612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:33:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:17 smithi115 bash[45495]: audit 2022-04-22T22:33:16.811560+0000 mon.a (mon.0) 14611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:33:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:17 smithi115 bash[45495]: audit 2022-04-22T22:33:16.812197+0000 mon.a (mon.0) 14612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:33:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:17 smithi115 bash[46934]: audit 2022-04-22T22:33:16.811560+0000 mon.a (mon.0) 14611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:33:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:17 smithi115 bash[46934]: audit 2022-04-22T22:33:16.812197+0000 mon.a (mon.0) 14612 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:33:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:18 smithi121 bash[36902]: cluster 2022-04-22T22:33:17.637309+0000 mgr.y (mgr.34107) 10475 : cluster [DBG] pgmap v11156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:18 smithi121 bash[36902]: audit 2022-04-22T22:33:17.968812+0000 mon.a (mon.0) 14613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:19.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:18 smithi115 bash[46934]: cluster 2022-04-22T22:33:17.637309+0000 mgr.y (mgr.34107) 10475 : cluster [DBG] pgmap v11156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:18 smithi115 bash[46934]: audit 2022-04-22T22:33:17.968812+0000 mon.a (mon.0) 14613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:18 smithi115 bash[45495]: cluster 2022-04-22T22:33:17.637309+0000 mgr.y (mgr.34107) 10475 : cluster [DBG] pgmap v11156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:18 smithi115 bash[45495]: audit 2022-04-22T22:33:17.968812+0000 mon.a (mon.0) 14613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:20.007 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[45495]: audit 2022-04-22T22:33:19.644173+0000 mon.a (mon.0) 14614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:33:20.008 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[45495]: audit 2022-04-22T22:33:19.646954+0000 mon.a (mon.0) 14615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:33:20.008 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[45495]: audit 2022-04-22T22:33:19.648674+0000 mon.a (mon.0) 14616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:33:20.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[46934]: audit 2022-04-22T22:33:19.644173+0000 mon.a (mon.0) 14614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:33:20.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[46934]: audit 2022-04-22T22:33:19.646954+0000 mon.a (mon.0) 14615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:33:20.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:19 smithi115 bash[46934]: audit 2022-04-22T22:33:19.648674+0000 mon.a (mon.0) 14616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:33:20.010 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:19 smithi121 bash[36902]: audit 2022-04-22T22:33:19.644173+0000 mon.a (mon.0) 14614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:33:20.010 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:19 smithi121 bash[36902]: audit 2022-04-22T22:33:19.646954+0000 mon.a (mon.0) 14615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:33:20.010 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:19 smithi121 bash[36902]: audit 2022-04-22T22:33:19.648674+0000 mon.a (mon.0) 14616 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:33:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:20 smithi121 bash[36902]: cluster 2022-04-22T22:33:19.638417+0000 mgr.y (mgr.34107) 10476 : cluster [DBG] pgmap v11157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:20 smithi121 bash[36902]: audit 2022-04-22T22:33:20.005170+0000 mon.a (mon.0) 14617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:20 smithi121 bash[36902]: audit 2022-04-22T22:33:20.021782+0000 mon.a (mon.0) 14618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:33:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[45495]: cluster 2022-04-22T22:33:19.638417+0000 mgr.y (mgr.34107) 10476 : cluster [DBG] pgmap v11157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[45495]: audit 2022-04-22T22:33:20.005170+0000 mon.a (mon.0) 14617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[45495]: audit 2022-04-22T22:33:20.021782+0000 mon.a (mon.0) 14618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:33:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[46934]: cluster 2022-04-22T22:33:19.638417+0000 mgr.y (mgr.34107) 10476 : cluster [DBG] pgmap v11157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[46934]: audit 2022-04-22T22:33:20.005170+0000 mon.a (mon.0) 14617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:20 smithi115 bash[46934]: audit 2022-04-22T22:33:20.021782+0000 mon.a (mon.0) 14618 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:33:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:22 smithi121 bash[36902]: cluster 2022-04-22T22:33:21.639704+0000 mgr.y (mgr.34107) 10477 : cluster [DBG] pgmap v11158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:22 smithi121 bash[36902]: audit 2022-04-22T22:33:21.644235+0000 mon.a (mon.0) 14619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:22.827Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:22 smithi115 bash[45495]: cluster 2022-04-22T22:33:21.639704+0000 mgr.y (mgr.34107) 10477 : cluster [DBG] pgmap v11158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:22 smithi115 bash[45495]: audit 2022-04-22T22:33:21.644235+0000 mon.a (mon.0) 14619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:22 smithi115 bash[46934]: cluster 2022-04-22T22:33:21.639704+0000 mgr.y (mgr.34107) 10477 : cluster [DBG] pgmap v11158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:22 smithi115 bash[46934]: audit 2022-04-22T22:33:21.644235+0000 mon.a (mon.0) 14619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:23 smithi121 bash[36902]: audit 2022-04-22T22:33:22.972608+0000 mon.a (mon.0) 14620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:23 smithi115 bash[45495]: audit 2022-04-22T22:33:22.972608+0000 mon.a (mon.0) 14620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:23 smithi115 bash[46934]: audit 2022-04-22T22:33:22.972608+0000 mon.a (mon.0) 14620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:24 smithi121 bash[36902]: cluster 2022-04-22T22:33:23.641107+0000 mgr.y (mgr.34107) 10478 : cluster [DBG] pgmap v11159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:24 smithi115 bash[45495]: cluster 2022-04-22T22:33:23.641107+0000 mgr.y (mgr.34107) 10478 : cluster [DBG] pgmap v11159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:24.678Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:24 smithi115 bash[46934]: cluster 2022-04-22T22:33:23.641107+0000 mgr.y (mgr.34107) 10478 : cluster [DBG] pgmap v11159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:25 smithi121 bash[36902]: audit 2022-04-22T22:33:25.007702+0000 mon.a (mon.0) 14621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:25 smithi115 bash[45495]: audit 2022-04-22T22:33:25.007702+0000 mon.a (mon.0) 14621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:26.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:25 smithi115 bash[46934]: audit 2022-04-22T22:33:25.007702+0000 mon.a (mon.0) 14621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:26 smithi121 bash[36902]: cluster 2022-04-22T22:33:25.642268+0000 mgr.y (mgr.34107) 10479 : cluster [DBG] pgmap v11160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:26 smithi121 bash[36902]: audit 2022-04-22T22:33:26.646983+0000 mon.a (mon.0) 14622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:26 smithi115 bash[45495]: cluster 2022-04-22T22:33:25.642268+0000 mgr.y (mgr.34107) 10479 : cluster [DBG] pgmap v11160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:26 smithi115 bash[45495]: audit 2022-04-22T22:33:26.646983+0000 mon.a (mon.0) 14622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:26 smithi115 bash[46934]: cluster 2022-04-22T22:33:25.642268+0000 mgr.y (mgr.34107) 10479 : cluster [DBG] pgmap v11160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:26 smithi115 bash[46934]: audit 2022-04-22T22:33:26.646983+0000 mon.a (mon.0) 14622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:28 smithi121 bash[36902]: cluster 2022-04-22T22:33:27.643352+0000 mgr.y (mgr.34107) 10480 : cluster [DBG] pgmap v11161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:28 smithi121 bash[36902]: audit 2022-04-22T22:33:27.978150+0000 mon.a (mon.0) 14623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:28 smithi115 bash[45495]: cluster 2022-04-22T22:33:27.643352+0000 mgr.y (mgr.34107) 10480 : cluster [DBG] pgmap v11161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:28 smithi115 bash[45495]: audit 2022-04-22T22:33:27.978150+0000 mon.a (mon.0) 14623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:28 smithi115 bash[46934]: cluster 2022-04-22T22:33:27.643352+0000 mgr.y (mgr.34107) 10480 : cluster [DBG] pgmap v11161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:28 smithi115 bash[46934]: audit 2022-04-22T22:33:27.978150+0000 mon.a (mon.0) 14623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:30 smithi121 bash[36902]: cluster 2022-04-22T22:33:29.644546+0000 mgr.y (mgr.34107) 10481 : cluster [DBG] pgmap v11162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:30 smithi121 bash[36902]: audit 2022-04-22T22:33:30.010874+0000 mon.a (mon.0) 14624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:30 smithi115 bash[45495]: cluster 2022-04-22T22:33:29.644546+0000 mgr.y (mgr.34107) 10481 : cluster [DBG] pgmap v11162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:30 smithi115 bash[45495]: audit 2022-04-22T22:33:30.010874+0000 mon.a (mon.0) 14624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:30 smithi115 bash[46934]: cluster 2022-04-22T22:33:29.644546+0000 mgr.y (mgr.34107) 10481 : cluster [DBG] pgmap v11162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:30 smithi115 bash[46934]: audit 2022-04-22T22:33:30.010874+0000 mon.a (mon.0) 14624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:32 smithi121 bash[36902]: cluster 2022-04-22T22:33:31.645719+0000 mgr.y (mgr.34107) 10482 : cluster [DBG] pgmap v11163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:32 smithi121 bash[36902]: audit 2022-04-22T22:33:31.648984+0000 mon.a (mon.0) 14625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:32.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:32 smithi115 bash[45495]: cluster 2022-04-22T22:33:31.645719+0000 mgr.y (mgr.34107) 10482 : cluster [DBG] pgmap v11163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:33.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:32 smithi115 bash[45495]: audit 2022-04-22T22:33:31.648984+0000 mon.a (mon.0) 14625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:33.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:32 smithi115 bash[46934]: cluster 2022-04-22T22:33:31.645719+0000 mgr.y (mgr.34107) 10482 : cluster [DBG] pgmap v11163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:33.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:32 smithi115 bash[46934]: audit 2022-04-22T22:33:31.648984+0000 mon.a (mon.0) 14625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:33 smithi121 bash[36902]: audit 2022-04-22T22:33:32.981603+0000 mon.a (mon.0) 14626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:33 smithi115 bash[45495]: audit 2022-04-22T22:33:32.981603+0000 mon.a (mon.0) 14626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:33 smithi115 bash[46934]: audit 2022-04-22T22:33:32.981603+0000 mon.a (mon.0) 14626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:34 smithi121 bash[36902]: cluster 2022-04-22T22:33:33.646898+0000 mgr.y (mgr.34107) 10483 : cluster [DBG] pgmap v11164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:34 smithi115 bash[45495]: cluster 2022-04-22T22:33:33.646898+0000 mgr.y (mgr.34107) 10483 : cluster [DBG] pgmap v11164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:34 smithi115 bash[46934]: cluster 2022-04-22T22:33:33.646898+0000 mgr.y (mgr.34107) 10483 : cluster [DBG] pgmap v11164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:34.677Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:35 smithi121 bash[36902]: audit 2022-04-22T22:33:35.013800+0000 mon.a (mon.0) 14627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:35 smithi115 bash[45495]: audit 2022-04-22T22:33:35.013800+0000 mon.a (mon.0) 14627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:35 smithi115 bash[46934]: audit 2022-04-22T22:33:35.013800+0000 mon.a (mon.0) 14627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:36 smithi121 bash[36902]: cluster 2022-04-22T22:33:35.647921+0000 mgr.y (mgr.34107) 10484 : cluster [DBG] pgmap v11165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:37.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:36 smithi121 bash[36902]: audit 2022-04-22T22:33:36.651847+0000 mon.a (mon.0) 14628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:37.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:36 smithi115 bash[45495]: cluster 2022-04-22T22:33:35.647921+0000 mgr.y (mgr.34107) 10484 : cluster [DBG] pgmap v11165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:37.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:36 smithi115 bash[45495]: audit 2022-04-22T22:33:36.651847+0000 mon.a (mon.0) 14628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:37.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:36 smithi115 bash[46934]: cluster 2022-04-22T22:33:35.647921+0000 mgr.y (mgr.34107) 10484 : cluster [DBG] pgmap v11165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:37.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:36 smithi115 bash[46934]: audit 2022-04-22T22:33:36.651847+0000 mon.a (mon.0) 14628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:38 smithi121 bash[36902]: cluster 2022-04-22T22:33:37.649007+0000 mgr.y (mgr.34107) 10485 : cluster [DBG] pgmap v11166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:38 smithi121 bash[36902]: audit 2022-04-22T22:33:37.985236+0000 mon.a (mon.0) 14629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:38 smithi115 bash[45495]: cluster 2022-04-22T22:33:37.649007+0000 mgr.y (mgr.34107) 10485 : cluster [DBG] pgmap v11166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:38 smithi115 bash[45495]: audit 2022-04-22T22:33:37.985236+0000 mon.a (mon.0) 14629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:38 smithi115 bash[46934]: cluster 2022-04-22T22:33:37.649007+0000 mgr.y (mgr.34107) 10485 : cluster [DBG] pgmap v11166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:38 smithi115 bash[46934]: audit 2022-04-22T22:33:37.985236+0000 mon.a (mon.0) 14629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:40 smithi121 bash[36902]: cluster 2022-04-22T22:33:39.650129+0000 mgr.y (mgr.34107) 10486 : cluster [DBG] pgmap v11167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:40 smithi121 bash[36902]: audit 2022-04-22T22:33:40.016834+0000 mon.a (mon.0) 14630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:41.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:40 smithi115 bash[46934]: cluster 2022-04-22T22:33:39.650129+0000 mgr.y (mgr.34107) 10486 : cluster [DBG] pgmap v11167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:40 smithi115 bash[46934]: audit 2022-04-22T22:33:40.016834+0000 mon.a (mon.0) 14630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:40 smithi115 bash[45495]: cluster 2022-04-22T22:33:39.650129+0000 mgr.y (mgr.34107) 10486 : cluster [DBG] pgmap v11167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:40 smithi115 bash[45495]: audit 2022-04-22T22:33:40.016834+0000 mon.a (mon.0) 14630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:42 smithi121 bash[36902]: cluster 2022-04-22T22:33:41.651036+0000 mgr.y (mgr.34107) 10487 : cluster [DBG] pgmap v11168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:42 smithi121 bash[36902]: audit 2022-04-22T22:33:41.654289+0000 mon.a (mon.0) 14631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:42.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:42 smithi115 bash[45495]: cluster 2022-04-22T22:33:41.651036+0000 mgr.y (mgr.34107) 10487 : cluster [DBG] pgmap v11168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:42 smithi115 bash[45495]: audit 2022-04-22T22:33:41.654289+0000 mon.a (mon.0) 14631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:42 smithi115 bash[46934]: cluster 2022-04-22T22:33:41.651036+0000 mgr.y (mgr.34107) 10487 : cluster [DBG] pgmap v11168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:42 smithi115 bash[46934]: audit 2022-04-22T22:33:41.654289+0000 mon.a (mon.0) 14631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:43 smithi121 bash[36902]: audit 2022-04-22T22:33:42.988790+0000 mon.a (mon.0) 14632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:43 smithi115 bash[45495]: audit 2022-04-22T22:33:42.988790+0000 mon.a (mon.0) 14632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:43 smithi115 bash[46934]: audit 2022-04-22T22:33:42.988790+0000 mon.a (mon.0) 14632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:44 smithi121 bash[36902]: cluster 2022-04-22T22:33:43.652319+0000 mgr.y (mgr.34107) 10488 : cluster [DBG] pgmap v11169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:44 smithi115 bash[45495]: cluster 2022-04-22T22:33:43.652319+0000 mgr.y (mgr.34107) 10488 : cluster [DBG] pgmap v11169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:44.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:44 smithi115 bash[46934]: cluster 2022-04-22T22:33:43.652319+0000 mgr.y (mgr.34107) 10488 : cluster [DBG] pgmap v11169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:45.613 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:33:45.956 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 7m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 7m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:33:45.957 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 7m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 8m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (4h) 8m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:33:45.958 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 8m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:33:45.959 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 8m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:33:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:45 smithi121 bash[36902]: audit 2022-04-22T22:33:45.019792+0000 mon.a (mon.0) 14633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:45 smithi115 bash[45495]: audit 2022-04-22T22:33:45.019792+0000 mon.a (mon.0) 14633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:45 smithi115 bash[46934]: audit 2022-04-22T22:33:45.019792+0000 mon.a (mon.0) 14633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:46.369 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:46.370 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:33:46.371 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:33:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:46 smithi121 bash[36902]: audit 2022-04-22T22:33:45.594403+0000 mgr.y (mgr.34107) 10489 : audit [DBG] from='client.56155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:46 smithi121 bash[36902]: cluster 2022-04-22T22:33:45.653328+0000 mgr.y (mgr.34107) 10490 : cluster [DBG] pgmap v11170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:46 smithi121 bash[36902]: audit 2022-04-22T22:33:46.365794+0000 mon.c (mon.1) 354 : audit [DBG] from='client.? 172.21.15.115:0/2649987936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:46 smithi121 bash[36902]: audit 2022-04-22T22:33:46.656825+0000 mon.a (mon.0) 14634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[46934]: audit 2022-04-22T22:33:45.594403+0000 mgr.y (mgr.34107) 10489 : audit [DBG] from='client.56155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[46934]: cluster 2022-04-22T22:33:45.653328+0000 mgr.y (mgr.34107) 10490 : cluster [DBG] pgmap v11170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[46934]: audit 2022-04-22T22:33:46.365794+0000 mon.c (mon.1) 354 : audit [DBG] from='client.? 172.21.15.115:0/2649987936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[46934]: audit 2022-04-22T22:33:46.656825+0000 mon.a (mon.0) 14634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:47.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[45495]: audit 2022-04-22T22:33:45.594403+0000 mgr.y (mgr.34107) 10489 : audit [DBG] from='client.56155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[45495]: cluster 2022-04-22T22:33:45.653328+0000 mgr.y (mgr.34107) 10490 : cluster [DBG] pgmap v11170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[45495]: audit 2022-04-22T22:33:46.365794+0000 mon.c (mon.1) 354 : audit [DBG] from='client.? 172.21.15.115:0/2649987936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:33:47.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:46 smithi115 bash[45495]: audit 2022-04-22T22:33:46.656825+0000 mon.a (mon.0) 14634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:47 smithi121 bash[36902]: audit 2022-04-22T22:33:45.932704+0000 mgr.y (mgr.34107) 10491 : audit [DBG] from='client.46227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:47 smithi115 bash[45495]: audit 2022-04-22T22:33:45.932704+0000 mgr.y (mgr.34107) 10491 : audit [DBG] from='client.46227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:47 smithi115 bash[46934]: audit 2022-04-22T22:33:45.932704+0000 mgr.y (mgr.34107) 10491 : audit [DBG] from='client.46227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:33:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:48 smithi121 bash[36902]: cluster 2022-04-22T22:33:47.654424+0000 mgr.y (mgr.34107) 10492 : cluster [DBG] pgmap v11171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:48 smithi121 bash[36902]: audit 2022-04-22T22:33:47.992484+0000 mon.a (mon.0) 14635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:48 smithi115 bash[45495]: cluster 2022-04-22T22:33:47.654424+0000 mgr.y (mgr.34107) 10492 : cluster [DBG] pgmap v11171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:48 smithi115 bash[45495]: audit 2022-04-22T22:33:47.992484+0000 mon.a (mon.0) 14635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:48 smithi115 bash[46934]: cluster 2022-04-22T22:33:47.654424+0000 mgr.y (mgr.34107) 10492 : cluster [DBG] pgmap v11171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:48 smithi115 bash[46934]: audit 2022-04-22T22:33:47.992484+0000 mon.a (mon.0) 14635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:50 smithi121 bash[36902]: cluster 2022-04-22T22:33:49.655542+0000 mgr.y (mgr.34107) 10493 : cluster [DBG] pgmap v11172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:50 smithi121 bash[36902]: audit 2022-04-22T22:33:50.022924+0000 mon.a (mon.0) 14636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:50 smithi115 bash[45495]: cluster 2022-04-22T22:33:49.655542+0000 mgr.y (mgr.34107) 10493 : cluster [DBG] pgmap v11172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:50 smithi115 bash[45495]: audit 2022-04-22T22:33:50.022924+0000 mon.a (mon.0) 14636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:50 smithi115 bash[46934]: cluster 2022-04-22T22:33:49.655542+0000 mgr.y (mgr.34107) 10493 : cluster [DBG] pgmap v11172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:50 smithi115 bash[46934]: audit 2022-04-22T22:33:50.022924+0000 mon.a (mon.0) 14636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:52 smithi121 bash[36902]: cluster 2022-04-22T22:33:51.656443+0000 mgr.y (mgr.34107) 10494 : cluster [DBG] pgmap v11173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:52 smithi121 bash[36902]: audit 2022-04-22T22:33:51.659366+0000 mon.a (mon.0) 14637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:52.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:52 smithi115 bash[45495]: cluster 2022-04-22T22:33:51.656443+0000 mgr.y (mgr.34107) 10494 : cluster [DBG] pgmap v11173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:52 smithi115 bash[45495]: audit 2022-04-22T22:33:51.659366+0000 mon.a (mon.0) 14637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:52 smithi115 bash[46934]: cluster 2022-04-22T22:33:51.656443+0000 mgr.y (mgr.34107) 10494 : cluster [DBG] pgmap v11173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:52 smithi115 bash[46934]: audit 2022-04-22T22:33:51.659366+0000 mon.a (mon.0) 14637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:53 smithi121 bash[36902]: audit 2022-04-22T22:33:52.996222+0000 mon.a (mon.0) 14638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:53 smithi115 bash[45495]: audit 2022-04-22T22:33:52.996222+0000 mon.a (mon.0) 14638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:53 smithi115 bash[46934]: audit 2022-04-22T22:33:52.996222+0000 mon.a (mon.0) 14638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:54 smithi121 bash[36902]: cluster 2022-04-22T22:33:53.657824+0000 mgr.y (mgr.34107) 10495 : cluster [DBG] pgmap v11174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:54 smithi115 bash[45495]: cluster 2022-04-22T22:33:53.657824+0000 mgr.y (mgr.34107) 10495 : cluster [DBG] pgmap v11174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:54 smithi115 bash[46934]: cluster 2022-04-22T22:33:53.657824+0000 mgr.y (mgr.34107) 10495 : cluster [DBG] pgmap v11174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:55.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:33:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:33:54.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:33:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:55 smithi121 bash[36902]: audit 2022-04-22T22:33:55.026145+0000 mon.a (mon.0) 14639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:55 smithi115 bash[45495]: audit 2022-04-22T22:33:55.026145+0000 mon.a (mon.0) 14639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:55 smithi115 bash[46934]: audit 2022-04-22T22:33:55.026145+0000 mon.a (mon.0) 14639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:33:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:56 smithi121 bash[36902]: cluster 2022-04-22T22:33:55.659040+0000 mgr.y (mgr.34107) 10496 : cluster [DBG] pgmap v11175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:56 smithi121 bash[36902]: audit 2022-04-22T22:33:56.662065+0000 mon.a (mon.0) 14640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:56 smithi115 bash[45495]: cluster 2022-04-22T22:33:55.659040+0000 mgr.y (mgr.34107) 10496 : cluster [DBG] pgmap v11175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:56 smithi115 bash[45495]: audit 2022-04-22T22:33:56.662065+0000 mon.a (mon.0) 14640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:56 smithi115 bash[46934]: cluster 2022-04-22T22:33:55.659040+0000 mgr.y (mgr.34107) 10496 : cluster [DBG] pgmap v11175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:56 smithi115 bash[46934]: audit 2022-04-22T22:33:56.662065+0000 mon.a (mon.0) 14640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:33:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:58 smithi121 bash[36902]: cluster 2022-04-22T22:33:57.660276+0000 mgr.y (mgr.34107) 10497 : cluster [DBG] pgmap v11176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:33:58 smithi121 bash[36902]: audit 2022-04-22T22:33:57.999347+0000 mon.a (mon.0) 14641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:58 smithi115 bash[45495]: cluster 2022-04-22T22:33:57.660276+0000 mgr.y (mgr.34107) 10497 : cluster [DBG] pgmap v11176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:33:58 smithi115 bash[45495]: audit 2022-04-22T22:33:57.999347+0000 mon.a (mon.0) 14641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:33:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:58 smithi115 bash[46934]: cluster 2022-04-22T22:33:57.660276+0000 mgr.y (mgr.34107) 10497 : cluster [DBG] pgmap v11176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:33:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:33:58 smithi115 bash[46934]: audit 2022-04-22T22:33:57.999347+0000 mon.a (mon.0) 14641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:00 smithi121 bash[36902]: cluster 2022-04-22T22:33:59.661485+0000 mgr.y (mgr.34107) 10498 : cluster [DBG] pgmap v11177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:00 smithi121 bash[36902]: audit 2022-04-22T22:34:00.028962+0000 mon.a (mon.0) 14642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:00 smithi115 bash[45495]: cluster 2022-04-22T22:33:59.661485+0000 mgr.y (mgr.34107) 10498 : cluster [DBG] pgmap v11177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:00 smithi115 bash[45495]: audit 2022-04-22T22:34:00.028962+0000 mon.a (mon.0) 14642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:00 smithi115 bash[46934]: cluster 2022-04-22T22:33:59.661485+0000 mgr.y (mgr.34107) 10498 : cluster [DBG] pgmap v11177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:00 smithi115 bash[46934]: audit 2022-04-22T22:34:00.028962+0000 mon.a (mon.0) 14642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:02 smithi121 bash[36902]: cluster 2022-04-22T22:34:01.662510+0000 mgr.y (mgr.34107) 10499 : cluster [DBG] pgmap v11178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:02 smithi121 bash[36902]: audit 2022-04-22T22:34:01.664741+0000 mon.a (mon.0) 14643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:02.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:02 smithi115 bash[45495]: cluster 2022-04-22T22:34:01.662510+0000 mgr.y (mgr.34107) 10499 : cluster [DBG] pgmap v11178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:02 smithi115 bash[45495]: audit 2022-04-22T22:34:01.664741+0000 mon.a (mon.0) 14643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:02 smithi115 bash[46934]: cluster 2022-04-22T22:34:01.662510+0000 mgr.y (mgr.34107) 10499 : cluster [DBG] pgmap v11178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:02 smithi115 bash[46934]: audit 2022-04-22T22:34:01.664741+0000 mon.a (mon.0) 14643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:03 smithi121 bash[36902]: audit 2022-04-22T22:34:03.003032+0000 mon.a (mon.0) 14644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:03 smithi115 bash[45495]: audit 2022-04-22T22:34:03.003032+0000 mon.a (mon.0) 14644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:03 smithi115 bash[46934]: audit 2022-04-22T22:34:03.003032+0000 mon.a (mon.0) 14644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:04 smithi121 bash[36902]: cluster 2022-04-22T22:34:03.663724+0000 mgr.y (mgr.34107) 10500 : cluster [DBG] pgmap v11179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:04.681Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:04 smithi115 bash[45495]: cluster 2022-04-22T22:34:03.663724+0000 mgr.y (mgr.34107) 10500 : cluster [DBG] pgmap v11179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:04 smithi115 bash[46934]: cluster 2022-04-22T22:34:03.663724+0000 mgr.y (mgr.34107) 10500 : cluster [DBG] pgmap v11179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:05 smithi121 bash[36902]: audit 2022-04-22T22:34:05.032099+0000 mon.a (mon.0) 14645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:05 smithi115 bash[46934]: audit 2022-04-22T22:34:05.032099+0000 mon.a (mon.0) 14645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:05 smithi115 bash[45495]: audit 2022-04-22T22:34:05.032099+0000 mon.a (mon.0) 14645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:06 smithi121 bash[36902]: cluster 2022-04-22T22:34:05.664702+0000 mgr.y (mgr.34107) 10501 : cluster [DBG] pgmap v11180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:06 smithi121 bash[36902]: audit 2022-04-22T22:34:06.667407+0000 mon.a (mon.0) 14646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:06 smithi115 bash[45495]: cluster 2022-04-22T22:34:05.664702+0000 mgr.y (mgr.34107) 10501 : cluster [DBG] pgmap v11180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:06 smithi115 bash[45495]: audit 2022-04-22T22:34:06.667407+0000 mon.a (mon.0) 14646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:06 smithi115 bash[46934]: cluster 2022-04-22T22:34:05.664702+0000 mgr.y (mgr.34107) 10501 : cluster [DBG] pgmap v11180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:06 smithi115 bash[46934]: audit 2022-04-22T22:34:06.667407+0000 mon.a (mon.0) 14646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:08 smithi121 bash[36902]: cluster 2022-04-22T22:34:07.665776+0000 mgr.y (mgr.34107) 10502 : cluster [DBG] pgmap v11181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:08 smithi121 bash[36902]: audit 2022-04-22T22:34:08.006713+0000 mon.a (mon.0) 14647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:08 smithi115 bash[45495]: cluster 2022-04-22T22:34:07.665776+0000 mgr.y (mgr.34107) 10502 : cluster [DBG] pgmap v11181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:08 smithi115 bash[45495]: audit 2022-04-22T22:34:08.006713+0000 mon.a (mon.0) 14647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:08 smithi115 bash[46934]: cluster 2022-04-22T22:34:07.665776+0000 mgr.y (mgr.34107) 10502 : cluster [DBG] pgmap v11181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:08 smithi115 bash[46934]: audit 2022-04-22T22:34:08.006713+0000 mon.a (mon.0) 14647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:10 smithi121 bash[36902]: cluster 2022-04-22T22:34:09.666888+0000 mgr.y (mgr.34107) 10503 : cluster [DBG] pgmap v11182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:10 smithi121 bash[36902]: audit 2022-04-22T22:34:10.035224+0000 mon.a (mon.0) 14648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:11.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:10 smithi115 bash[46934]: cluster 2022-04-22T22:34:09.666888+0000 mgr.y (mgr.34107) 10503 : cluster [DBG] pgmap v11182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:11.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:10 smithi115 bash[46934]: audit 2022-04-22T22:34:10.035224+0000 mon.a (mon.0) 14648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:10 smithi115 bash[45495]: cluster 2022-04-22T22:34:09.666888+0000 mgr.y (mgr.34107) 10503 : cluster [DBG] pgmap v11182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:10 smithi115 bash[45495]: audit 2022-04-22T22:34:10.035224+0000 mon.a (mon.0) 14648 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:12 smithi121 bash[36902]: cluster 2022-04-22T22:34:11.667830+0000 mgr.y (mgr.34107) 10504 : cluster [DBG] pgmap v11183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:12 smithi121 bash[36902]: audit 2022-04-22T22:34:11.670456+0000 mon.a (mon.0) 14649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:12.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:12 smithi115 bash[45495]: cluster 2022-04-22T22:34:11.667830+0000 mgr.y (mgr.34107) 10504 : cluster [DBG] pgmap v11183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:12 smithi115 bash[45495]: audit 2022-04-22T22:34:11.670456+0000 mon.a (mon.0) 14649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:12 smithi115 bash[46934]: cluster 2022-04-22T22:34:11.667830+0000 mgr.y (mgr.34107) 10504 : cluster [DBG] pgmap v11183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:12 smithi115 bash[46934]: audit 2022-04-22T22:34:11.670456+0000 mon.a (mon.0) 14649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:13 smithi121 bash[36902]: audit 2022-04-22T22:34:13.010573+0000 mon.a (mon.0) 14650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:13 smithi115 bash[45495]: audit 2022-04-22T22:34:13.010573+0000 mon.a (mon.0) 14650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:13 smithi115 bash[46934]: audit 2022-04-22T22:34:13.010573+0000 mon.a (mon.0) 14650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:14 smithi121 bash[36902]: cluster 2022-04-22T22:34:13.669120+0000 mgr.y (mgr.34107) 10505 : cluster [DBG] pgmap v11184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:14 smithi115 bash[45495]: cluster 2022-04-22T22:34:13.669120+0000 mgr.y (mgr.34107) 10505 : cluster [DBG] pgmap v11184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:14.683Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:14 smithi115 bash[46934]: cluster 2022-04-22T22:34:13.669120+0000 mgr.y (mgr.34107) 10505 : cluster [DBG] pgmap v11184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:15 smithi121 bash[36902]: audit 2022-04-22T22:34:15.038517+0000 mon.a (mon.0) 14651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:15 smithi115 bash[45495]: audit 2022-04-22T22:34:15.038517+0000 mon.a (mon.0) 14651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:16.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:15 smithi115 bash[46934]: audit 2022-04-22T22:34:15.038517+0000 mon.a (mon.0) 14651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:16.744 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:34:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:16 smithi121 bash[36902]: cluster 2022-04-22T22:34:15.670208+0000 mgr.y (mgr.34107) 10506 : cluster [DBG] pgmap v11185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:16 smithi121 bash[36902]: audit 2022-04-22T22:34:16.672826+0000 mon.a (mon.0) 14652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:16 smithi115 bash[45495]: cluster 2022-04-22T22:34:15.670208+0000 mgr.y (mgr.34107) 10506 : cluster [DBG] pgmap v11185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:16 smithi115 bash[45495]: audit 2022-04-22T22:34:16.672826+0000 mon.a (mon.0) 14652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:16 smithi115 bash[46934]: cluster 2022-04-22T22:34:15.670208+0000 mgr.y (mgr.34107) 10506 : cluster [DBG] pgmap v11185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:16 smithi115 bash[46934]: audit 2022-04-22T22:34:16.672826+0000 mon.a (mon.0) 14652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:17.097 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:34:17.097 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:34:17.097 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:34:17.097 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:34:17.097 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:34:17.098 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (4h) 9m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:34:17.099 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:34:17.513 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:34:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:34:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:34:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:34:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:34:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:34:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:34:17.515 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:34:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:17 smithi121 bash[36902]: audit 2022-04-22T22:34:16.724803+0000 mgr.y (mgr.34107) 10507 : audit [DBG] from='client.56173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:17 smithi121 bash[36902]: audit 2022-04-22T22:34:16.822568+0000 mon.a (mon.0) 14653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:34:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:17 smithi121 bash[36902]: audit 2022-04-22T22:34:16.822780+0000 mon.a (mon.0) 14654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:34:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:17 smithi121 bash[36902]: audit 2022-04-22T22:34:17.509393+0000 mon.a (mon.0) 14655 : audit [DBG] from='client.? 172.21.15.115:0/2644164294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[45495]: audit 2022-04-22T22:34:16.724803+0000 mgr.y (mgr.34107) 10507 : audit [DBG] from='client.56173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[45495]: audit 2022-04-22T22:34:16.822568+0000 mon.a (mon.0) 14653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[45495]: audit 2022-04-22T22:34:16.822780+0000 mon.a (mon.0) 14654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[45495]: audit 2022-04-22T22:34:17.509393+0000 mon.a (mon.0) 14655 : audit [DBG] from='client.? 172.21.15.115:0/2644164294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[46934]: audit 2022-04-22T22:34:16.724803+0000 mgr.y (mgr.34107) 10507 : audit [DBG] from='client.56173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[46934]: audit 2022-04-22T22:34:16.822568+0000 mon.a (mon.0) 14653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[46934]: audit 2022-04-22T22:34:16.822780+0000 mon.a (mon.0) 14654 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:34:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:17 smithi115 bash[46934]: audit 2022-04-22T22:34:17.509393+0000 mon.a (mon.0) 14655 : audit [DBG] from='client.? 172.21.15.115:0/2644164294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:18 smithi121 bash[36902]: audit 2022-04-22T22:34:17.072957+0000 mgr.y (mgr.34107) 10508 : audit [DBG] from='client.46245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:18 smithi121 bash[36902]: cluster 2022-04-22T22:34:17.671341+0000 mgr.y (mgr.34107) 10509 : cluster [DBG] pgmap v11186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:18 smithi121 bash[36902]: audit 2022-04-22T22:34:18.014420+0000 mon.a (mon.0) 14656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[46934]: audit 2022-04-22T22:34:17.072957+0000 mgr.y (mgr.34107) 10508 : audit [DBG] from='client.46245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[46934]: cluster 2022-04-22T22:34:17.671341+0000 mgr.y (mgr.34107) 10509 : cluster [DBG] pgmap v11186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[46934]: audit 2022-04-22T22:34:18.014420+0000 mon.a (mon.0) 14656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[45495]: audit 2022-04-22T22:34:17.072957+0000 mgr.y (mgr.34107) 10508 : audit [DBG] from='client.46245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[45495]: cluster 2022-04-22T22:34:17.671341+0000 mgr.y (mgr.34107) 10509 : cluster [DBG] pgmap v11186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:18 smithi115 bash[45495]: audit 2022-04-22T22:34:18.014420+0000 mon.a (mon.0) 14656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: cluster 2022-04-22T22:34:19.672427+0000 mgr.y (mgr.34107) 10510 : cluster [DBG] pgmap v11187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: audit 2022-04-22T22:34:20.031851+0000 mon.a (mon.0) 14657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:34:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: audit 2022-04-22T22:34:20.034653+0000 mon.a (mon.0) 14658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:34:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: audit 2022-04-22T22:34:20.036434+0000 mon.a (mon.0) 14659 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:34:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: audit 2022-04-22T22:34:20.041706+0000 mon.a (mon.0) 14660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:20 smithi121 bash[36902]: audit 2022-04-22T22:34:20.391792+0000 mon.a (mon.0) 14661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: cluster 2022-04-22T22:34:19.672427+0000 mgr.y (mgr.34107) 10510 : cluster [DBG] pgmap v11187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: audit 2022-04-22T22:34:20.031851+0000 mon.a (mon.0) 14657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: audit 2022-04-22T22:34:20.034653+0000 mon.a (mon.0) 14658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: audit 2022-04-22T22:34:20.036434+0000 mon.a (mon.0) 14659 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: audit 2022-04-22T22:34:20.041706+0000 mon.a (mon.0) 14660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[45495]: audit 2022-04-22T22:34:20.391792+0000 mon.a (mon.0) 14661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: cluster 2022-04-22T22:34:19.672427+0000 mgr.y (mgr.34107) 10510 : cluster [DBG] pgmap v11187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: audit 2022-04-22T22:34:20.031851+0000 mon.a (mon.0) 14657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: audit 2022-04-22T22:34:20.034653+0000 mon.a (mon.0) 14658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: audit 2022-04-22T22:34:20.036434+0000 mon.a (mon.0) 14659 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: audit 2022-04-22T22:34:20.041706+0000 mon.a (mon.0) 14660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:20 smithi115 bash[46934]: audit 2022-04-22T22:34:20.391792+0000 mon.a (mon.0) 14661 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:34:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:22 smithi121 bash[36902]: cluster 2022-04-22T22:34:21.673507+0000 mgr.y (mgr.34107) 10511 : cluster [DBG] pgmap v11188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:22 smithi121 bash[36902]: audit 2022-04-22T22:34:21.675404+0000 mon.a (mon.0) 14662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:22.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:22 smithi115 bash[45495]: cluster 2022-04-22T22:34:21.673507+0000 mgr.y (mgr.34107) 10511 : cluster [DBG] pgmap v11188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:22 smithi115 bash[45495]: audit 2022-04-22T22:34:21.675404+0000 mon.a (mon.0) 14662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:22 smithi115 bash[46934]: cluster 2022-04-22T22:34:21.673507+0000 mgr.y (mgr.34107) 10511 : cluster [DBG] pgmap v11188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:22 smithi115 bash[46934]: audit 2022-04-22T22:34:21.675404+0000 mon.a (mon.0) 14662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:23 smithi121 bash[36902]: audit 2022-04-22T22:34:23.018271+0000 mon.a (mon.0) 14663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:23 smithi115 bash[45495]: audit 2022-04-22T22:34:23.018271+0000 mon.a (mon.0) 14663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:23 smithi115 bash[46934]: audit 2022-04-22T22:34:23.018271+0000 mon.a (mon.0) 14663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:24 smithi121 bash[36902]: cluster 2022-04-22T22:34:23.674572+0000 mgr.y (mgr.34107) 10512 : cluster [DBG] pgmap v11189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:24 smithi115 bash[45495]: cluster 2022-04-22T22:34:23.674572+0000 mgr.y (mgr.34107) 10512 : cluster [DBG] pgmap v11189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:24.684Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:24 smithi115 bash[46934]: cluster 2022-04-22T22:34:23.674572+0000 mgr.y (mgr.34107) 10512 : cluster [DBG] pgmap v11189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:25 smithi121 bash[36902]: audit 2022-04-22T22:34:25.044202+0000 mon.a (mon.0) 14664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:25 smithi115 bash[45495]: audit 2022-04-22T22:34:25.044202+0000 mon.a (mon.0) 14664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:25 smithi115 bash[46934]: audit 2022-04-22T22:34:25.044202+0000 mon.a (mon.0) 14664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:26 smithi121 bash[36902]: cluster 2022-04-22T22:34:25.675763+0000 mgr.y (mgr.34107) 10513 : cluster [DBG] pgmap v11190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:26 smithi121 bash[36902]: audit 2022-04-22T22:34:26.677954+0000 mon.a (mon.0) 14665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:26 smithi115 bash[45495]: cluster 2022-04-22T22:34:25.675763+0000 mgr.y (mgr.34107) 10513 : cluster [DBG] pgmap v11190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:26 smithi115 bash[45495]: audit 2022-04-22T22:34:26.677954+0000 mon.a (mon.0) 14665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:26 smithi115 bash[46934]: cluster 2022-04-22T22:34:25.675763+0000 mgr.y (mgr.34107) 10513 : cluster [DBG] pgmap v11190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:26 smithi115 bash[46934]: audit 2022-04-22T22:34:26.677954+0000 mon.a (mon.0) 14665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:28 smithi121 bash[36902]: cluster 2022-04-22T22:34:27.676864+0000 mgr.y (mgr.34107) 10514 : cluster [DBG] pgmap v11191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:28 smithi121 bash[36902]: audit 2022-04-22T22:34:28.022294+0000 mon.a (mon.0) 14666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:28 smithi115 bash[46934]: cluster 2022-04-22T22:34:27.676864+0000 mgr.y (mgr.34107) 10514 : cluster [DBG] pgmap v11191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:28 smithi115 bash[46934]: audit 2022-04-22T22:34:28.022294+0000 mon.a (mon.0) 14666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:28 smithi115 bash[45495]: cluster 2022-04-22T22:34:27.676864+0000 mgr.y (mgr.34107) 10514 : cluster [DBG] pgmap v11191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:28 smithi115 bash[45495]: audit 2022-04-22T22:34:28.022294+0000 mon.a (mon.0) 14666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:30 smithi121 bash[36902]: cluster 2022-04-22T22:34:29.677938+0000 mgr.y (mgr.34107) 10515 : cluster [DBG] pgmap v11192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:30 smithi121 bash[36902]: audit 2022-04-22T22:34:30.047390+0000 mon.a (mon.0) 14667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:30 smithi115 bash[45495]: cluster 2022-04-22T22:34:29.677938+0000 mgr.y (mgr.34107) 10515 : cluster [DBG] pgmap v11192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:30 smithi115 bash[45495]: audit 2022-04-22T22:34:30.047390+0000 mon.a (mon.0) 14667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:30 smithi115 bash[46934]: cluster 2022-04-22T22:34:29.677938+0000 mgr.y (mgr.34107) 10515 : cluster [DBG] pgmap v11192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:30 smithi115 bash[46934]: audit 2022-04-22T22:34:30.047390+0000 mon.a (mon.0) 14667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:32 smithi121 bash[36902]: cluster 2022-04-22T22:34:31.679024+0000 mgr.y (mgr.34107) 10516 : cluster [DBG] pgmap v11193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:32 smithi121 bash[36902]: audit 2022-04-22T22:34:31.680856+0000 mon.a (mon.0) 14668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:32.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:32 smithi115 bash[45495]: cluster 2022-04-22T22:34:31.679024+0000 mgr.y (mgr.34107) 10516 : cluster [DBG] pgmap v11193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:32 smithi115 bash[45495]: audit 2022-04-22T22:34:31.680856+0000 mon.a (mon.0) 14668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:32 smithi115 bash[46934]: cluster 2022-04-22T22:34:31.679024+0000 mgr.y (mgr.34107) 10516 : cluster [DBG] pgmap v11193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:32 smithi115 bash[46934]: audit 2022-04-22T22:34:31.680856+0000 mon.a (mon.0) 14668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:33 smithi121 bash[36902]: audit 2022-04-22T22:34:33.025983+0000 mon.a (mon.0) 14669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:33 smithi115 bash[45495]: audit 2022-04-22T22:34:33.025983+0000 mon.a (mon.0) 14669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:33 smithi115 bash[46934]: audit 2022-04-22T22:34:33.025983+0000 mon.a (mon.0) 14669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:34 smithi121 bash[36902]: cluster 2022-04-22T22:34:33.680123+0000 mgr.y (mgr.34107) 10517 : cluster [DBG] pgmap v11194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:34 smithi115 bash[45495]: cluster 2022-04-22T22:34:33.680123+0000 mgr.y (mgr.34107) 10517 : cluster [DBG] pgmap v11194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:34 smithi115 bash[46934]: cluster 2022-04-22T22:34:33.680123+0000 mgr.y (mgr.34107) 10517 : cluster [DBG] pgmap v11194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:34.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:35 smithi121 bash[36902]: audit 2022-04-22T22:34:35.050425+0000 mon.a (mon.0) 14670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:35 smithi115 bash[45495]: audit 2022-04-22T22:34:35.050425+0000 mon.a (mon.0) 14670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:35 smithi115 bash[46934]: audit 2022-04-22T22:34:35.050425+0000 mon.a (mon.0) 14670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:36 smithi121 bash[36902]: cluster 2022-04-22T22:34:35.681306+0000 mgr.y (mgr.34107) 10518 : cluster [DBG] pgmap v11195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:36 smithi121 bash[36902]: audit 2022-04-22T22:34:36.683243+0000 mon.a (mon.0) 14671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:36 smithi115 bash[45495]: cluster 2022-04-22T22:34:35.681306+0000 mgr.y (mgr.34107) 10518 : cluster [DBG] pgmap v11195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:36 smithi115 bash[45495]: audit 2022-04-22T22:34:36.683243+0000 mon.a (mon.0) 14671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:36 smithi115 bash[46934]: cluster 2022-04-22T22:34:35.681306+0000 mgr.y (mgr.34107) 10518 : cluster [DBG] pgmap v11195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:36 smithi115 bash[46934]: audit 2022-04-22T22:34:36.683243+0000 mon.a (mon.0) 14671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:38 smithi121 bash[36902]: cluster 2022-04-22T22:34:37.682638+0000 mgr.y (mgr.34107) 10519 : cluster [DBG] pgmap v11196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:38 smithi121 bash[36902]: audit 2022-04-22T22:34:38.029465+0000 mon.a (mon.0) 14672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:38 smithi115 bash[45495]: cluster 2022-04-22T22:34:37.682638+0000 mgr.y (mgr.34107) 10519 : cluster [DBG] pgmap v11196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:38 smithi115 bash[45495]: audit 2022-04-22T22:34:38.029465+0000 mon.a (mon.0) 14672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:38 smithi115 bash[46934]: cluster 2022-04-22T22:34:37.682638+0000 mgr.y (mgr.34107) 10519 : cluster [DBG] pgmap v11196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:38 smithi115 bash[46934]: audit 2022-04-22T22:34:38.029465+0000 mon.a (mon.0) 14672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:40 smithi121 bash[36902]: cluster 2022-04-22T22:34:39.683836+0000 mgr.y (mgr.34107) 10520 : cluster [DBG] pgmap v11197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:40 smithi121 bash[36902]: audit 2022-04-22T22:34:40.055023+0000 mon.a (mon.0) 14673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:40 smithi115 bash[45495]: cluster 2022-04-22T22:34:39.683836+0000 mgr.y (mgr.34107) 10520 : cluster [DBG] pgmap v11197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:40 smithi115 bash[45495]: audit 2022-04-22T22:34:40.055023+0000 mon.a (mon.0) 14673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:40 smithi115 bash[46934]: cluster 2022-04-22T22:34:39.683836+0000 mgr.y (mgr.34107) 10520 : cluster [DBG] pgmap v11197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:40 smithi115 bash[46934]: audit 2022-04-22T22:34:40.055023+0000 mon.a (mon.0) 14673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:42 smithi121 bash[36902]: cluster 2022-04-22T22:34:41.684707+0000 mgr.y (mgr.34107) 10521 : cluster [DBG] pgmap v11198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:42 smithi121 bash[36902]: audit 2022-04-22T22:34:41.685732+0000 mon.a (mon.0) 14674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:42.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:42 smithi115 bash[45495]: cluster 2022-04-22T22:34:41.684707+0000 mgr.y (mgr.34107) 10521 : cluster [DBG] pgmap v11198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:42 smithi115 bash[45495]: audit 2022-04-22T22:34:41.685732+0000 mon.a (mon.0) 14674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:42 smithi115 bash[46934]: cluster 2022-04-22T22:34:41.684707+0000 mgr.y (mgr.34107) 10521 : cluster [DBG] pgmap v11198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:42 smithi115 bash[46934]: audit 2022-04-22T22:34:41.685732+0000 mon.a (mon.0) 14674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:43 smithi121 bash[36902]: audit 2022-04-22T22:34:43.032977+0000 mon.a (mon.0) 14675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:43 smithi115 bash[45495]: audit 2022-04-22T22:34:43.032977+0000 mon.a (mon.0) 14675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:43 smithi115 bash[46934]: audit 2022-04-22T22:34:43.032977+0000 mon.a (mon.0) 14675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:44 smithi121 bash[36902]: cluster 2022-04-22T22:34:43.685859+0000 mgr.y (mgr.34107) 10522 : cluster [DBG] pgmap v11199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:44 smithi115 bash[45495]: cluster 2022-04-22T22:34:43.685859+0000 mgr.y (mgr.34107) 10522 : cluster [DBG] pgmap v11199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:44.686Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:44 smithi115 bash[46934]: cluster 2022-04-22T22:34:43.685859+0000 mgr.y (mgr.34107) 10522 : cluster [DBG] pgmap v11199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:45 smithi121 bash[36902]: audit 2022-04-22T22:34:45.058109+0000 mon.a (mon.0) 14676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:45 smithi115 bash[45495]: audit 2022-04-22T22:34:45.058109+0000 mon.a (mon.0) 14676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:45 smithi115 bash[46934]: audit 2022-04-22T22:34:45.058109+0000 mon.a (mon.0) 14676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:46 smithi121 bash[36902]: cluster 2022-04-22T22:34:45.686820+0000 mgr.y (mgr.34107) 10523 : cluster [DBG] pgmap v11200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:46 smithi121 bash[36902]: audit 2022-04-22T22:34:46.688347+0000 mon.a (mon.0) 14677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:46 smithi115 bash[45495]: cluster 2022-04-22T22:34:45.686820+0000 mgr.y (mgr.34107) 10523 : cluster [DBG] pgmap v11200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:46 smithi115 bash[45495]: audit 2022-04-22T22:34:46.688347+0000 mon.a (mon.0) 14677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:46 smithi115 bash[46934]: cluster 2022-04-22T22:34:45.686820+0000 mgr.y (mgr.34107) 10523 : cluster [DBG] pgmap v11200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:47.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:46 smithi115 bash[46934]: audit 2022-04-22T22:34:46.688347+0000 mon.a (mon.0) 14677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:47.888 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:34:48.235 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:34:48.236 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 8m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:34:48.236 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:34:48.236 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (4h) 8m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:34:48.236 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:34:48.237 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:34:48.237 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:34:48.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:34:48.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:34:48.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:34:48.238 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 8m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:34:48.238 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 9m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:34:48.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:34:48.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:34:48.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:34:48.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:34:48.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:34:48.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:34:48.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:34:48.239 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:34:48.240 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 9m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:34:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:34:48.636 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:34:48.636 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:34:48.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:34:48.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:34:48.638 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:34:48.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:34:48.638 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:34:48.638 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:34:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:48 smithi121 bash[36902]: cluster 2022-04-22T22:34:47.687802+0000 mgr.y (mgr.34107) 10524 : cluster [DBG] pgmap v11201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:48 smithi121 bash[36902]: audit 2022-04-22T22:34:48.036338+0000 mon.a (mon.0) 14678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:48 smithi121 bash[36902]: audit 2022-04-22T22:34:48.632748+0000 mon.a (mon.0) 14679 : audit [DBG] from='client.? 172.21.15.115:0/2474347510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[45495]: cluster 2022-04-22T22:34:47.687802+0000 mgr.y (mgr.34107) 10524 : cluster [DBG] pgmap v11201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[45495]: audit 2022-04-22T22:34:48.036338+0000 mon.a (mon.0) 14678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[45495]: audit 2022-04-22T22:34:48.632748+0000 mon.a (mon.0) 14679 : audit [DBG] from='client.? 172.21.15.115:0/2474347510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[46934]: cluster 2022-04-22T22:34:47.687802+0000 mgr.y (mgr.34107) 10524 : cluster [DBG] pgmap v11201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[46934]: audit 2022-04-22T22:34:48.036338+0000 mon.a (mon.0) 14678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:48 smithi115 bash[46934]: audit 2022-04-22T22:34:48.632748+0000 mon.a (mon.0) 14679 : audit [DBG] from='client.? 172.21.15.115:0/2474347510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:34:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:49 smithi115 bash[45495]: audit 2022-04-22T22:34:47.868329+0000 mgr.y (mgr.34107) 10525 : audit [DBG] from='client.46257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:49 smithi115 bash[45495]: audit 2022-04-22T22:34:48.212923+0000 mgr.y (mgr.34107) 10526 : audit [DBG] from='client.56197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:49 smithi115 bash[46934]: audit 2022-04-22T22:34:47.868329+0000 mgr.y (mgr.34107) 10525 : audit [DBG] from='client.46257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:49 smithi115 bash[46934]: audit 2022-04-22T22:34:48.212923+0000 mgr.y (mgr.34107) 10526 : audit [DBG] from='client.56197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:49 smithi121 bash[36902]: audit 2022-04-22T22:34:47.868329+0000 mgr.y (mgr.34107) 10525 : audit [DBG] from='client.46257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:49 smithi121 bash[36902]: audit 2022-04-22T22:34:48.212923+0000 mgr.y (mgr.34107) 10526 : audit [DBG] from='client.56197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:34:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:50 smithi115 bash[45495]: cluster 2022-04-22T22:34:49.688824+0000 mgr.y (mgr.34107) 10527 : cluster [DBG] pgmap v11202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:50 smithi115 bash[45495]: audit 2022-04-22T22:34:50.061310+0000 mon.a (mon.0) 14680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:50 smithi115 bash[46934]: cluster 2022-04-22T22:34:49.688824+0000 mgr.y (mgr.34107) 10527 : cluster [DBG] pgmap v11202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:50 smithi115 bash[46934]: audit 2022-04-22T22:34:50.061310+0000 mon.a (mon.0) 14680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:50 smithi121 bash[36902]: cluster 2022-04-22T22:34:49.688824+0000 mgr.y (mgr.34107) 10527 : cluster [DBG] pgmap v11202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:50 smithi121 bash[36902]: audit 2022-04-22T22:34:50.061310+0000 mon.a (mon.0) 14680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:52 smithi121 bash[36902]: cluster 2022-04-22T22:34:51.689800+0000 mgr.y (mgr.34107) 10528 : cluster [DBG] pgmap v11203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:52 smithi121 bash[36902]: audit 2022-04-22T22:34:51.691590+0000 mon.a (mon.0) 14681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:52.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:52 smithi115 bash[45495]: cluster 2022-04-22T22:34:51.689800+0000 mgr.y (mgr.34107) 10528 : cluster [DBG] pgmap v11203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:52 smithi115 bash[45495]: audit 2022-04-22T22:34:51.691590+0000 mon.a (mon.0) 14681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:52 smithi115 bash[46934]: cluster 2022-04-22T22:34:51.689800+0000 mgr.y (mgr.34107) 10528 : cluster [DBG] pgmap v11203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:52 smithi115 bash[46934]: audit 2022-04-22T22:34:51.691590+0000 mon.a (mon.0) 14681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:53 smithi121 bash[36902]: audit 2022-04-22T22:34:53.039618+0000 mon.a (mon.0) 14682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:53 smithi115 bash[45495]: audit 2022-04-22T22:34:53.039618+0000 mon.a (mon.0) 14682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:53 smithi115 bash[46934]: audit 2022-04-22T22:34:53.039618+0000 mon.a (mon.0) 14682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:54 smithi121 bash[36902]: cluster 2022-04-22T22:34:53.690957+0000 mgr.y (mgr.34107) 10529 : cluster [DBG] pgmap v11204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:54 smithi115 bash[45495]: cluster 2022-04-22T22:34:53.690957+0000 mgr.y (mgr.34107) 10529 : cluster [DBG] pgmap v11204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:34:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:34:54.688Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:34:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:54 smithi115 bash[46934]: cluster 2022-04-22T22:34:53.690957+0000 mgr.y (mgr.34107) 10529 : cluster [DBG] pgmap v11204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:55 smithi121 bash[36902]: audit 2022-04-22T22:34:55.064138+0000 mon.a (mon.0) 14683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:55 smithi115 bash[45495]: audit 2022-04-22T22:34:55.064138+0000 mon.a (mon.0) 14683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:55 smithi115 bash[46934]: audit 2022-04-22T22:34:55.064138+0000 mon.a (mon.0) 14683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:34:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:56 smithi121 bash[36902]: cluster 2022-04-22T22:34:55.692060+0000 mgr.y (mgr.34107) 10530 : cluster [DBG] pgmap v11205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:56 smithi121 bash[36902]: audit 2022-04-22T22:34:56.694059+0000 mon.a (mon.0) 14684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:56 smithi115 bash[45495]: cluster 2022-04-22T22:34:55.692060+0000 mgr.y (mgr.34107) 10530 : cluster [DBG] pgmap v11205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:56 smithi115 bash[45495]: audit 2022-04-22T22:34:56.694059+0000 mon.a (mon.0) 14684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:56 smithi115 bash[46934]: cluster 2022-04-22T22:34:55.692060+0000 mgr.y (mgr.34107) 10530 : cluster [DBG] pgmap v11205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:56 smithi115 bash[46934]: audit 2022-04-22T22:34:56.694059+0000 mon.a (mon.0) 14684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:34:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:58 smithi121 bash[36902]: cluster 2022-04-22T22:34:57.693126+0000 mgr.y (mgr.34107) 10531 : cluster [DBG] pgmap v11206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:34:58 smithi121 bash[36902]: audit 2022-04-22T22:34:58.043423+0000 mon.a (mon.0) 14685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:58 smithi115 bash[45495]: cluster 2022-04-22T22:34:57.693126+0000 mgr.y (mgr.34107) 10531 : cluster [DBG] pgmap v11206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:34:58 smithi115 bash[45495]: audit 2022-04-22T22:34:58.043423+0000 mon.a (mon.0) 14685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:34:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:58 smithi115 bash[46934]: cluster 2022-04-22T22:34:57.693126+0000 mgr.y (mgr.34107) 10531 : cluster [DBG] pgmap v11206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:34:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:34:58 smithi115 bash[46934]: audit 2022-04-22T22:34:58.043423+0000 mon.a (mon.0) 14685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:00 smithi121 bash[36902]: cluster 2022-04-22T22:34:59.694236+0000 mgr.y (mgr.34107) 10532 : cluster [DBG] pgmap v11207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:00 smithi121 bash[36902]: audit 2022-04-22T22:35:00.067104+0000 mon.a (mon.0) 14686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:00 smithi115 bash[45495]: cluster 2022-04-22T22:34:59.694236+0000 mgr.y (mgr.34107) 10532 : cluster [DBG] pgmap v11207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:00 smithi115 bash[45495]: audit 2022-04-22T22:35:00.067104+0000 mon.a (mon.0) 14686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:00 smithi115 bash[46934]: cluster 2022-04-22T22:34:59.694236+0000 mgr.y (mgr.34107) 10532 : cluster [DBG] pgmap v11207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:00 smithi115 bash[46934]: audit 2022-04-22T22:35:00.067104+0000 mon.a (mon.0) 14686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:02 smithi121 bash[36902]: cluster 2022-04-22T22:35:01.695307+0000 mgr.y (mgr.34107) 10533 : cluster [DBG] pgmap v11208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:02 smithi121 bash[36902]: audit 2022-04-22T22:35:01.696707+0000 mon.a (mon.0) 14687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:02.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:02 smithi115 bash[45495]: cluster 2022-04-22T22:35:01.695307+0000 mgr.y (mgr.34107) 10533 : cluster [DBG] pgmap v11208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:02 smithi115 bash[45495]: audit 2022-04-22T22:35:01.696707+0000 mon.a (mon.0) 14687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:02 smithi115 bash[46934]: cluster 2022-04-22T22:35:01.695307+0000 mgr.y (mgr.34107) 10533 : cluster [DBG] pgmap v11208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:02 smithi115 bash[46934]: audit 2022-04-22T22:35:01.696707+0000 mon.a (mon.0) 14687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:03 smithi121 bash[36902]: audit 2022-04-22T22:35:03.047011+0000 mon.a (mon.0) 14688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:03 smithi115 bash[46934]: audit 2022-04-22T22:35:03.047011+0000 mon.a (mon.0) 14688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:03 smithi115 bash[45495]: audit 2022-04-22T22:35:03.047011+0000 mon.a (mon.0) 14688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:04 smithi121 bash[36902]: cluster 2022-04-22T22:35:03.696640+0000 mgr.y (mgr.34107) 10534 : cluster [DBG] pgmap v11209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:04 smithi115 bash[45495]: cluster 2022-04-22T22:35:03.696640+0000 mgr.y (mgr.34107) 10534 : cluster [DBG] pgmap v11209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:04 smithi115 bash[46934]: cluster 2022-04-22T22:35:03.696640+0000 mgr.y (mgr.34107) 10534 : cluster [DBG] pgmap v11209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:04.688Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:05 smithi121 bash[36902]: audit 2022-04-22T22:35:05.070026+0000 mon.a (mon.0) 14689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:05 smithi115 bash[45495]: audit 2022-04-22T22:35:05.070026+0000 mon.a (mon.0) 14689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:05 smithi115 bash[46934]: audit 2022-04-22T22:35:05.070026+0000 mon.a (mon.0) 14689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:06 smithi121 bash[36902]: cluster 2022-04-22T22:35:05.697736+0000 mgr.y (mgr.34107) 10535 : cluster [DBG] pgmap v11210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:06 smithi121 bash[36902]: audit 2022-04-22T22:35:06.699778+0000 mon.a (mon.0) 14690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:06 smithi115 bash[45495]: cluster 2022-04-22T22:35:05.697736+0000 mgr.y (mgr.34107) 10535 : cluster [DBG] pgmap v11210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:06 smithi115 bash[45495]: audit 2022-04-22T22:35:06.699778+0000 mon.a (mon.0) 14690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:06 smithi115 bash[46934]: cluster 2022-04-22T22:35:05.697736+0000 mgr.y (mgr.34107) 10535 : cluster [DBG] pgmap v11210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:06 smithi115 bash[46934]: audit 2022-04-22T22:35:06.699778+0000 mon.a (mon.0) 14690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:08 smithi121 bash[36902]: cluster 2022-04-22T22:35:07.698913+0000 mgr.y (mgr.34107) 10536 : cluster [DBG] pgmap v11211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:08 smithi121 bash[36902]: audit 2022-04-22T22:35:08.050650+0000 mon.a (mon.0) 14691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:08 smithi115 bash[45495]: cluster 2022-04-22T22:35:07.698913+0000 mgr.y (mgr.34107) 10536 : cluster [DBG] pgmap v11211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:08 smithi115 bash[45495]: audit 2022-04-22T22:35:08.050650+0000 mon.a (mon.0) 14691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:08 smithi115 bash[46934]: cluster 2022-04-22T22:35:07.698913+0000 mgr.y (mgr.34107) 10536 : cluster [DBG] pgmap v11211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:08 smithi115 bash[46934]: audit 2022-04-22T22:35:08.050650+0000 mon.a (mon.0) 14691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:10 smithi121 bash[36902]: cluster 2022-04-22T22:35:09.700054+0000 mgr.y (mgr.34107) 10537 : cluster [DBG] pgmap v11212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:11.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:10 smithi121 bash[36902]: audit 2022-04-22T22:35:10.072875+0000 mon.a (mon.0) 14692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:10 smithi115 bash[45495]: cluster 2022-04-22T22:35:09.700054+0000 mgr.y (mgr.34107) 10537 : cluster [DBG] pgmap v11212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:10 smithi115 bash[45495]: audit 2022-04-22T22:35:10.072875+0000 mon.a (mon.0) 14692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:10 smithi115 bash[46934]: cluster 2022-04-22T22:35:09.700054+0000 mgr.y (mgr.34107) 10537 : cluster [DBG] pgmap v11212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:10 smithi115 bash[46934]: audit 2022-04-22T22:35:10.072875+0000 mon.a (mon.0) 14692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:12 smithi121 bash[36902]: cluster 2022-04-22T22:35:11.701151+0000 mgr.y (mgr.34107) 10538 : cluster [DBG] pgmap v11213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:12 smithi121 bash[36902]: audit 2022-04-22T22:35:11.702985+0000 mon.a (mon.0) 14693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:12 smithi115 bash[45495]: cluster 2022-04-22T22:35:11.701151+0000 mgr.y (mgr.34107) 10538 : cluster [DBG] pgmap v11213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:12 smithi115 bash[45495]: audit 2022-04-22T22:35:11.702985+0000 mon.a (mon.0) 14693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:12.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:12 smithi115 bash[46934]: cluster 2022-04-22T22:35:11.701151+0000 mgr.y (mgr.34107) 10538 : cluster [DBG] pgmap v11213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:12 smithi115 bash[46934]: audit 2022-04-22T22:35:11.702985+0000 mon.a (mon.0) 14693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:13 smithi121 bash[36902]: audit 2022-04-22T22:35:13.054543+0000 mon.a (mon.0) 14694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:13 smithi115 bash[45495]: audit 2022-04-22T22:35:13.054543+0000 mon.a (mon.0) 14694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:13 smithi115 bash[46934]: audit 2022-04-22T22:35:13.054543+0000 mon.a (mon.0) 14694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:14 smithi121 bash[36902]: cluster 2022-04-22T22:35:13.702216+0000 mgr.y (mgr.34107) 10539 : cluster [DBG] pgmap v11214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:14 smithi115 bash[45495]: cluster 2022-04-22T22:35:13.702216+0000 mgr.y (mgr.34107) 10539 : cluster [DBG] pgmap v11214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:14.687Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:14 smithi115 bash[46934]: cluster 2022-04-22T22:35:13.702216+0000 mgr.y (mgr.34107) 10539 : cluster [DBG] pgmap v11214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:15 smithi121 bash[36902]: audit 2022-04-22T22:35:15.075809+0000 mon.a (mon.0) 14695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:15 smithi115 bash[45495]: audit 2022-04-22T22:35:15.075809+0000 mon.a (mon.0) 14695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:15 smithi115 bash[46934]: audit 2022-04-22T22:35:15.075809+0000 mon.a (mon.0) 14695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:16 smithi121 bash[36902]: cluster 2022-04-22T22:35:15.703319+0000 mgr.y (mgr.34107) 10540 : cluster [DBG] pgmap v11215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:16 smithi121 bash[36902]: audit 2022-04-22T22:35:16.706112+0000 mon.a (mon.0) 14696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:16 smithi115 bash[45495]: cluster 2022-04-22T22:35:15.703319+0000 mgr.y (mgr.34107) 10540 : cluster [DBG] pgmap v11215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:16 smithi115 bash[45495]: audit 2022-04-22T22:35:16.706112+0000 mon.a (mon.0) 14696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:16 smithi115 bash[46934]: cluster 2022-04-22T22:35:15.703319+0000 mgr.y (mgr.34107) 10540 : cluster [DBG] pgmap v11215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:16 smithi115 bash[46934]: audit 2022-04-22T22:35:16.706112+0000 mon.a (mon.0) 14696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:17 smithi115 bash[46934]: audit 2022-04-22T22:35:16.834645+0000 mon.a (mon.0) 14697 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:35:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:17 smithi115 bash[46934]: audit 2022-04-22T22:35:16.835024+0000 mon.a (mon.0) 14698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:35:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:17 smithi115 bash[45495]: audit 2022-04-22T22:35:16.834645+0000 mon.a (mon.0) 14697 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:35:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:17 smithi115 bash[45495]: audit 2022-04-22T22:35:16.835024+0000 mon.a (mon.0) 14698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:35:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:17 smithi121 bash[36902]: audit 2022-04-22T22:35:16.834645+0000 mon.a (mon.0) 14697 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:35:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:17 smithi121 bash[36902]: audit 2022-04-22T22:35:16.835024+0000 mon.a (mon.0) 14698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:35:19.013 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:35:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:18 smithi115 bash[45495]: cluster 2022-04-22T22:35:17.704484+0000 mgr.y (mgr.34107) 10541 : cluster [DBG] pgmap v11216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:18 smithi115 bash[45495]: audit 2022-04-22T22:35:18.058164+0000 mon.a (mon.0) 14699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:18 smithi115 bash[46934]: cluster 2022-04-22T22:35:17.704484+0000 mgr.y (mgr.34107) 10541 : cluster [DBG] pgmap v11216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:18 smithi115 bash[46934]: audit 2022-04-22T22:35:18.058164+0000 mon.a (mon.0) 14699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:18 smithi121 bash[36902]: cluster 2022-04-22T22:35:17.704484+0000 mgr.y (mgr.34107) 10541 : cluster [DBG] pgmap v11216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:18 smithi121 bash[36902]: audit 2022-04-22T22:35:18.058164+0000 mon.a (mon.0) 14699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:19.359 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:35:19.359 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:35:19.359 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 10m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (4h) 10m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 10m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:35:19.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 10m ago 5h 283M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:35:19.361 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:35:19.361 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:35:19.361 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 10m ago 5h 10.8M - 1dbe0e931976 f4b044642a96 2022-04-22T22:35:19.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:35:19.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:35:19.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:35:19.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:35:19.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 10m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:35:19.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 10m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:35:19.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 10m ago 5h 505M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:35:19.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 10m ago 5h 649M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:35:19.363 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 10m ago 5h 68.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:35:19.760 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:35:19.760 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:35:19.760 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:35:19.761 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:35:19.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:19.762 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:35:19.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:35:19.762 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:35:19.762 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[45495]: audit 2022-04-22T22:35:18.990158+0000 mgr.y (mgr.34107) 10542 : audit [DBG] from='client.56209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[45495]: audit 2022-04-22T22:35:19.335877+0000 mgr.y (mgr.34107) 10543 : audit [DBG] from='client.46281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[45495]: cluster 2022-04-22T22:35:19.705653+0000 mgr.y (mgr.34107) 10544 : cluster [DBG] pgmap v11217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[45495]: audit 2022-04-22T22:35:19.756721+0000 mon.c (mon.1) 355 : audit [DBG] from='client.? 172.21.15.115:0/489624569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[46934]: audit 2022-04-22T22:35:18.990158+0000 mgr.y (mgr.34107) 10542 : audit [DBG] from='client.56209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[46934]: audit 2022-04-22T22:35:19.335877+0000 mgr.y (mgr.34107) 10543 : audit [DBG] from='client.46281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[46934]: cluster 2022-04-22T22:35:19.705653+0000 mgr.y (mgr.34107) 10544 : cluster [DBG] pgmap v11217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:19 smithi115 bash[46934]: audit 2022-04-22T22:35:19.756721+0000 mon.c (mon.1) 355 : audit [DBG] from='client.? 172.21.15.115:0/489624569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:19 smithi121 bash[36902]: audit 2022-04-22T22:35:18.990158+0000 mgr.y (mgr.34107) 10542 : audit [DBG] from='client.56209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:19 smithi121 bash[36902]: audit 2022-04-22T22:35:19.335877+0000 mgr.y (mgr.34107) 10543 : audit [DBG] from='client.46281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:19 smithi121 bash[36902]: cluster 2022-04-22T22:35:19.705653+0000 mgr.y (mgr.34107) 10544 : cluster [DBG] pgmap v11217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:19 smithi121 bash[36902]: audit 2022-04-22T22:35:19.756721+0000 mon.c (mon.1) 355 : audit [DBG] from='client.? 172.21.15.115:0/489624569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[45495]: audit 2022-04-22T22:35:20.078312+0000 mon.a (mon.0) 14700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[45495]: audit 2022-04-22T22:35:20.398681+0000 mon.a (mon.0) 14701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[45495]: audit 2022-04-22T22:35:20.401473+0000 mon.a (mon.0) 14702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[45495]: audit 2022-04-22T22:35:20.403144+0000 mon.a (mon.0) 14703 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[46934]: audit 2022-04-22T22:35:20.078312+0000 mon.a (mon.0) 14700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[46934]: audit 2022-04-22T22:35:20.398681+0000 mon.a (mon.0) 14701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:35:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[46934]: audit 2022-04-22T22:35:20.401473+0000 mon.a (mon.0) 14702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:35:21.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:20 smithi115 bash[46934]: audit 2022-04-22T22:35:20.403144+0000 mon.a (mon.0) 14703 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:20 smithi121 bash[36902]: audit 2022-04-22T22:35:20.078312+0000 mon.a (mon.0) 14700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:20 smithi121 bash[36902]: audit 2022-04-22T22:35:20.398681+0000 mon.a (mon.0) 14701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:20 smithi121 bash[36902]: audit 2022-04-22T22:35:20.401473+0000 mon.a (mon.0) 14702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:35:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:20 smithi121 bash[36902]: audit 2022-04-22T22:35:20.403144+0000 mon.a (mon.0) 14703 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:35:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:22 smithi121 bash[36902]: cluster 2022-04-22T22:35:21.706779+0000 mgr.y (mgr.34107) 10545 : cluster [DBG] pgmap v11218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:22 smithi121 bash[36902]: audit 2022-04-22T22:35:21.709278+0000 mon.a (mon.0) 14704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:23.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:22 smithi115 bash[46934]: cluster 2022-04-22T22:35:21.706779+0000 mgr.y (mgr.34107) 10545 : cluster [DBG] pgmap v11218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:23.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:22 smithi115 bash[46934]: audit 2022-04-22T22:35:21.709278+0000 mon.a (mon.0) 14704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:22 smithi115 bash[45495]: cluster 2022-04-22T22:35:21.706779+0000 mgr.y (mgr.34107) 10545 : cluster [DBG] pgmap v11218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:22 smithi115 bash[45495]: audit 2022-04-22T22:35:21.709278+0000 mon.a (mon.0) 14704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:22.834Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:23 smithi121 bash[36902]: audit 2022-04-22T22:35:23.061516+0000 mon.a (mon.0) 14705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:23 smithi115 bash[45495]: audit 2022-04-22T22:35:23.061516+0000 mon.a (mon.0) 14705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:23 smithi115 bash[46934]: audit 2022-04-22T22:35:23.061516+0000 mon.a (mon.0) 14705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:24 smithi121 bash[36902]: cluster 2022-04-22T22:35:23.707866+0000 mgr.y (mgr.34107) 10546 : cluster [DBG] pgmap v11219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:24 smithi115 bash[45495]: cluster 2022-04-22T22:35:23.707866+0000 mgr.y (mgr.34107) 10546 : cluster [DBG] pgmap v11219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:24.689Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:24 smithi115 bash[46934]: cluster 2022-04-22T22:35:23.707866+0000 mgr.y (mgr.34107) 10546 : cluster [DBG] pgmap v11219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:25 smithi121 bash[36902]: audit 2022-04-22T22:35:24.800895+0000 mon.a (mon.0) 14706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:25 smithi121 bash[36902]: audit 2022-04-22T22:35:25.081143+0000 mon.a (mon.0) 14707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:25 smithi121 bash[36902]: audit 2022-04-22T22:35:25.123076+0000 mon.a (mon.0) 14708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:25 smithi121 bash[36902]: cluster 2022-04-22T22:35:25.708946+0000 mgr.y (mgr.34107) 10547 : cluster [DBG] pgmap v11220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[45495]: audit 2022-04-22T22:35:24.800895+0000 mon.a (mon.0) 14706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[45495]: audit 2022-04-22T22:35:25.081143+0000 mon.a (mon.0) 14707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[45495]: audit 2022-04-22T22:35:25.123076+0000 mon.a (mon.0) 14708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[45495]: cluster 2022-04-22T22:35:25.708946+0000 mgr.y (mgr.34107) 10547 : cluster [DBG] pgmap v11220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[46934]: audit 2022-04-22T22:35:24.800895+0000 mon.a (mon.0) 14706 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[46934]: audit 2022-04-22T22:35:25.081143+0000 mon.a (mon.0) 14707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[46934]: audit 2022-04-22T22:35:25.123076+0000 mon.a (mon.0) 14708 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:35:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:25 smithi115 bash[46934]: cluster 2022-04-22T22:35:25.708946+0000 mgr.y (mgr.34107) 10547 : cluster [DBG] pgmap v11220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:27 smithi121 bash[36902]: audit 2022-04-22T22:35:26.712118+0000 mon.a (mon.0) 14709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:27 smithi115 bash[45495]: audit 2022-04-22T22:35:26.712118+0000 mon.a (mon.0) 14709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:27 smithi115 bash[46934]: audit 2022-04-22T22:35:26.712118+0000 mon.a (mon.0) 14709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:28 smithi121 bash[36902]: cluster 2022-04-22T22:35:27.710006+0000 mgr.y (mgr.34107) 10548 : cluster [DBG] pgmap v11221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:28 smithi121 bash[36902]: audit 2022-04-22T22:35:28.064347+0000 mon.a (mon.0) 14710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:28 smithi115 bash[45495]: cluster 2022-04-22T22:35:27.710006+0000 mgr.y (mgr.34107) 10548 : cluster [DBG] pgmap v11221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:28 smithi115 bash[45495]: audit 2022-04-22T22:35:28.064347+0000 mon.a (mon.0) 14710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:28 smithi115 bash[46934]: cluster 2022-04-22T22:35:27.710006+0000 mgr.y (mgr.34107) 10548 : cluster [DBG] pgmap v11221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:28 smithi115 bash[46934]: audit 2022-04-22T22:35:28.064347+0000 mon.a (mon.0) 14710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:30 smithi115 bash[45495]: cluster 2022-04-22T22:35:29.711135+0000 mgr.y (mgr.34107) 10549 : cluster [DBG] pgmap v11222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:30 smithi115 bash[45495]: audit 2022-04-22T22:35:30.084093+0000 mon.a (mon.0) 14711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:30 smithi115 bash[46934]: cluster 2022-04-22T22:35:29.711135+0000 mgr.y (mgr.34107) 10549 : cluster [DBG] pgmap v11222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:30 smithi115 bash[46934]: audit 2022-04-22T22:35:30.084093+0000 mon.a (mon.0) 14711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:30 smithi121 bash[36902]: cluster 2022-04-22T22:35:29.711135+0000 mgr.y (mgr.34107) 10549 : cluster [DBG] pgmap v11222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:30 smithi121 bash[36902]: audit 2022-04-22T22:35:30.084093+0000 mon.a (mon.0) 14711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:32 smithi121 bash[36902]: cluster 2022-04-22T22:35:31.712255+0000 mgr.y (mgr.34107) 10550 : cluster [DBG] pgmap v11223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:32 smithi121 bash[36902]: audit 2022-04-22T22:35:31.714906+0000 mon.a (mon.0) 14712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:32.834Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:32 smithi115 bash[45495]: cluster 2022-04-22T22:35:31.712255+0000 mgr.y (mgr.34107) 10550 : cluster [DBG] pgmap v11223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:32 smithi115 bash[45495]: audit 2022-04-22T22:35:31.714906+0000 mon.a (mon.0) 14712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:32 smithi115 bash[46934]: cluster 2022-04-22T22:35:31.712255+0000 mgr.y (mgr.34107) 10550 : cluster [DBG] pgmap v11223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:32 smithi115 bash[46934]: audit 2022-04-22T22:35:31.714906+0000 mon.a (mon.0) 14712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:33 smithi121 bash[36902]: audit 2022-04-22T22:35:33.067757+0000 mon.a (mon.0) 14713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:33 smithi115 bash[45495]: audit 2022-04-22T22:35:33.067757+0000 mon.a (mon.0) 14713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:33 smithi115 bash[46934]: audit 2022-04-22T22:35:33.067757+0000 mon.a (mon.0) 14713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:34 smithi121 bash[36902]: cluster 2022-04-22T22:35:33.713314+0000 mgr.y (mgr.34107) 10551 : cluster [DBG] pgmap v11224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:34 smithi115 bash[45495]: cluster 2022-04-22T22:35:33.713314+0000 mgr.y (mgr.34107) 10551 : cluster [DBG] pgmap v11224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:34.690Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:34 smithi115 bash[46934]: cluster 2022-04-22T22:35:33.713314+0000 mgr.y (mgr.34107) 10551 : cluster [DBG] pgmap v11224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:35 smithi121 bash[36902]: audit 2022-04-22T22:35:35.087020+0000 mon.a (mon.0) 14714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:35 smithi115 bash[45495]: audit 2022-04-22T22:35:35.087020+0000 mon.a (mon.0) 14714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:35 smithi115 bash[46934]: audit 2022-04-22T22:35:35.087020+0000 mon.a (mon.0) 14714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:36 smithi121 bash[36902]: cluster 2022-04-22T22:35:35.714379+0000 mgr.y (mgr.34107) 10552 : cluster [DBG] pgmap v11225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:36 smithi115 bash[45495]: cluster 2022-04-22T22:35:35.714379+0000 mgr.y (mgr.34107) 10552 : cluster [DBG] pgmap v11225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:36 smithi115 bash[46934]: cluster 2022-04-22T22:35:35.714379+0000 mgr.y (mgr.34107) 10552 : cluster [DBG] pgmap v11225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:37 smithi121 bash[36902]: audit 2022-04-22T22:35:36.720539+0000 mon.a (mon.0) 14715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:37 smithi115 bash[45495]: audit 2022-04-22T22:35:36.720539+0000 mon.a (mon.0) 14715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:37 smithi115 bash[46934]: audit 2022-04-22T22:35:36.720539+0000 mon.a (mon.0) 14715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:38 smithi121 bash[36902]: cluster 2022-04-22T22:35:37.715620+0000 mgr.y (mgr.34107) 10553 : cluster [DBG] pgmap v11226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:38 smithi121 bash[36902]: audit 2022-04-22T22:35:38.071421+0000 mon.a (mon.0) 14716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:38 smithi115 bash[45495]: cluster 2022-04-22T22:35:37.715620+0000 mgr.y (mgr.34107) 10553 : cluster [DBG] pgmap v11226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:38 smithi115 bash[45495]: audit 2022-04-22T22:35:38.071421+0000 mon.a (mon.0) 14716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:38 smithi115 bash[46934]: cluster 2022-04-22T22:35:37.715620+0000 mgr.y (mgr.34107) 10553 : cluster [DBG] pgmap v11226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:38 smithi115 bash[46934]: audit 2022-04-22T22:35:38.071421+0000 mon.a (mon.0) 14716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:40 smithi121 bash[36902]: cluster 2022-04-22T22:35:39.716651+0000 mgr.y (mgr.34107) 10554 : cluster [DBG] pgmap v11227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:41.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:40 smithi121 bash[36902]: audit 2022-04-22T22:35:40.089843+0000 mon.a (mon.0) 14717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:40 smithi115 bash[45495]: cluster 2022-04-22T22:35:39.716651+0000 mgr.y (mgr.34107) 10554 : cluster [DBG] pgmap v11227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:41.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:40 smithi115 bash[45495]: audit 2022-04-22T22:35:40.089843+0000 mon.a (mon.0) 14717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:40 smithi115 bash[46934]: cluster 2022-04-22T22:35:39.716651+0000 mgr.y (mgr.34107) 10554 : cluster [DBG] pgmap v11227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:40 smithi115 bash[46934]: audit 2022-04-22T22:35:40.089843+0000 mon.a (mon.0) 14717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:42 smithi121 bash[36902]: cluster 2022-04-22T22:35:41.717927+0000 mgr.y (mgr.34107) 10555 : cluster [DBG] pgmap v11228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:42 smithi121 bash[36902]: audit 2022-04-22T22:35:41.720846+0000 mon.a (mon.0) 14718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:43.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:42.836Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:42 smithi115 bash[45495]: cluster 2022-04-22T22:35:41.717927+0000 mgr.y (mgr.34107) 10555 : cluster [DBG] pgmap v11228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:42 smithi115 bash[45495]: audit 2022-04-22T22:35:41.720846+0000 mon.a (mon.0) 14718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:43.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:42 smithi115 bash[46934]: cluster 2022-04-22T22:35:41.717927+0000 mgr.y (mgr.34107) 10555 : cluster [DBG] pgmap v11228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:43.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:42 smithi115 bash[46934]: audit 2022-04-22T22:35:41.720846+0000 mon.a (mon.0) 14718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:43 smithi121 bash[36902]: audit 2022-04-22T22:35:43.075127+0000 mon.a (mon.0) 14719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:44.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:43 smithi115 bash[46934]: audit 2022-04-22T22:35:43.075127+0000 mon.a (mon.0) 14719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:43 smithi115 bash[45495]: audit 2022-04-22T22:35:43.075127+0000 mon.a (mon.0) 14719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:44 smithi121 bash[36902]: cluster 2022-04-22T22:35:43.719137+0000 mgr.y (mgr.34107) 10556 : cluster [DBG] pgmap v11229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:44 smithi115 bash[45495]: cluster 2022-04-22T22:35:43.719137+0000 mgr.y (mgr.34107) 10556 : cluster [DBG] pgmap v11229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:44.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:44 smithi115 bash[46934]: cluster 2022-04-22T22:35:43.719137+0000 mgr.y (mgr.34107) 10556 : cluster [DBG] pgmap v11229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:45 smithi121 bash[36902]: audit 2022-04-22T22:35:45.092913+0000 mon.a (mon.0) 14720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:45 smithi115 bash[45495]: audit 2022-04-22T22:35:45.092913+0000 mon.a (mon.0) 14720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:45 smithi115 bash[46934]: audit 2022-04-22T22:35:45.092913+0000 mon.a (mon.0) 14720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:46 smithi121 bash[36902]: cluster 2022-04-22T22:35:45.720046+0000 mgr.y (mgr.34107) 10557 : cluster [DBG] pgmap v11230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:46 smithi115 bash[45495]: cluster 2022-04-22T22:35:45.720046+0000 mgr.y (mgr.34107) 10557 : cluster [DBG] pgmap v11230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:46 smithi115 bash[46934]: cluster 2022-04-22T22:35:45.720046+0000 mgr.y (mgr.34107) 10557 : cluster [DBG] pgmap v11230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:47 smithi121 bash[36902]: audit 2022-04-22T22:35:46.727137+0000 mon.a (mon.0) 14721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:47 smithi115 bash[45495]: audit 2022-04-22T22:35:46.727137+0000 mon.a (mon.0) 14721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:47 smithi115 bash[46934]: audit 2022-04-22T22:35:46.727137+0000 mon.a (mon.0) 14721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:48 smithi121 bash[36902]: cluster 2022-04-22T22:35:47.721273+0000 mgr.y (mgr.34107) 10558 : cluster [DBG] pgmap v11231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:48 smithi121 bash[36902]: audit 2022-04-22T22:35:48.078895+0000 mon.a (mon.0) 14722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:48 smithi115 bash[45495]: cluster 2022-04-22T22:35:47.721273+0000 mgr.y (mgr.34107) 10558 : cluster [DBG] pgmap v11231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:48 smithi115 bash[45495]: audit 2022-04-22T22:35:48.078895+0000 mon.a (mon.0) 14722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:48 smithi115 bash[46934]: cluster 2022-04-22T22:35:47.721273+0000 mgr.y (mgr.34107) 10558 : cluster [DBG] pgmap v11231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:48 smithi115 bash[46934]: audit 2022-04-22T22:35:48.078895+0000 mon.a (mon.0) 14722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:50.137 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:35:50.485 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 9m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 25s ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 25s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 25s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 25s ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:35:50.486 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4h) 9m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (4h) 25s ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 25s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 25s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 25s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 25s ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:35:50.487 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (4h) 25s ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:35:50.883 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:35:50.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:35:50.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:35:50.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:35:50.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:35:50.885 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:35:50.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:35:50.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:35:50.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:35:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:50 smithi121 bash[36902]: cluster 2022-04-22T22:35:49.722331+0000 mgr.y (mgr.34107) 10559 : cluster [DBG] pgmap v11232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:51.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:50 smithi121 bash[36902]: audit 2022-04-22T22:35:50.095659+0000 mon.a (mon.0) 14723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:50 smithi115 bash[45495]: cluster 2022-04-22T22:35:49.722331+0000 mgr.y (mgr.34107) 10559 : cluster [DBG] pgmap v11232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:50 smithi115 bash[45495]: audit 2022-04-22T22:35:50.095659+0000 mon.a (mon.0) 14723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:50 smithi115 bash[46934]: cluster 2022-04-22T22:35:49.722331+0000 mgr.y (mgr.34107) 10559 : cluster [DBG] pgmap v11232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:50 smithi115 bash[46934]: audit 2022-04-22T22:35:50.095659+0000 mon.a (mon.0) 14723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:51 smithi121 bash[36902]: audit 2022-04-22T22:35:50.118002+0000 mgr.y (mgr.34107) 10560 : audit [DBG] from='client.46293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:51 smithi121 bash[36902]: audit 2022-04-22T22:35:50.461684+0000 mgr.y (mgr.34107) 10561 : audit [DBG] from='client.46299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:51 smithi121 bash[36902]: audit 2022-04-22T22:35:50.880011+0000 mon.c (mon.1) 356 : audit [DBG] from='client.? 172.21.15.115:0/1695038972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:51 smithi121 bash[36902]: audit 2022-04-22T22:35:51.727257+0000 mon.a (mon.0) 14724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[45495]: audit 2022-04-22T22:35:50.118002+0000 mgr.y (mgr.34107) 10560 : audit [DBG] from='client.46293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[45495]: audit 2022-04-22T22:35:50.461684+0000 mgr.y (mgr.34107) 10561 : audit [DBG] from='client.46299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[45495]: audit 2022-04-22T22:35:50.880011+0000 mon.c (mon.1) 356 : audit [DBG] from='client.? 172.21.15.115:0/1695038972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[45495]: audit 2022-04-22T22:35:51.727257+0000 mon.a (mon.0) 14724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[46934]: audit 2022-04-22T22:35:50.118002+0000 mgr.y (mgr.34107) 10560 : audit [DBG] from='client.46293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[46934]: audit 2022-04-22T22:35:50.461684+0000 mgr.y (mgr.34107) 10561 : audit [DBG] from='client.46299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:35:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[46934]: audit 2022-04-22T22:35:50.880011+0000 mon.c (mon.1) 356 : audit [DBG] from='client.? 172.21.15.115:0/1695038972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:35:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:51 smithi115 bash[46934]: audit 2022-04-22T22:35:51.727257+0000 mon.a (mon.0) 14724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:52 smithi121 bash[36902]: cluster 2022-04-22T22:35:51.723391+0000 mgr.y (mgr.34107) 10562 : cluster [DBG] pgmap v11233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:53.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:52 smithi115 bash[46934]: cluster 2022-04-22T22:35:51.723391+0000 mgr.y (mgr.34107) 10562 : cluster [DBG] pgmap v11233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:52 smithi115 bash[45495]: cluster 2022-04-22T22:35:51.723391+0000 mgr.y (mgr.34107) 10562 : cluster [DBG] pgmap v11233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:52.836Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:53 smithi121 bash[36902]: audit 2022-04-22T22:35:53.082379+0000 mon.a (mon.0) 14725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:53 smithi115 bash[45495]: audit 2022-04-22T22:35:53.082379+0000 mon.a (mon.0) 14725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:53 smithi115 bash[46934]: audit 2022-04-22T22:35:53.082379+0000 mon.a (mon.0) 14725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:54 smithi121 bash[36902]: cluster 2022-04-22T22:35:53.724528+0000 mgr.y (mgr.34107) 10563 : cluster [DBG] pgmap v11234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:54 smithi115 bash[45495]: cluster 2022-04-22T22:35:53.724528+0000 mgr.y (mgr.34107) 10563 : cluster [DBG] pgmap v11234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:54 smithi115 bash[46934]: cluster 2022-04-22T22:35:53.724528+0000 mgr.y (mgr.34107) 10563 : cluster [DBG] pgmap v11234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:35:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:35:54.691Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:35:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:55 smithi121 bash[36902]: audit 2022-04-22T22:35:55.098423+0000 mon.a (mon.0) 14726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:55 smithi115 bash[45495]: audit 2022-04-22T22:35:55.098423+0000 mon.a (mon.0) 14726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:55 smithi115 bash[46934]: audit 2022-04-22T22:35:55.098423+0000 mon.a (mon.0) 14726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:35:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:56 smithi121 bash[36902]: cluster 2022-04-22T22:35:55.725644+0000 mgr.y (mgr.34107) 10564 : cluster [DBG] pgmap v11235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:56 smithi121 bash[36902]: audit 2022-04-22T22:35:56.730037+0000 mon.a (mon.0) 14727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:56 smithi115 bash[45495]: cluster 2022-04-22T22:35:55.725644+0000 mgr.y (mgr.34107) 10564 : cluster [DBG] pgmap v11235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:56 smithi115 bash[45495]: audit 2022-04-22T22:35:56.730037+0000 mon.a (mon.0) 14727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:56 smithi115 bash[46934]: cluster 2022-04-22T22:35:55.725644+0000 mgr.y (mgr.34107) 10564 : cluster [DBG] pgmap v11235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:56 smithi115 bash[46934]: audit 2022-04-22T22:35:56.730037+0000 mon.a (mon.0) 14727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:35:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:58 smithi115 bash[45495]: cluster 2022-04-22T22:35:57.726719+0000 mgr.y (mgr.34107) 10565 : cluster [DBG] pgmap v11236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:35:58 smithi115 bash[45495]: audit 2022-04-22T22:35:58.086368+0000 mon.a (mon.0) 14728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:58 smithi115 bash[46934]: cluster 2022-04-22T22:35:57.726719+0000 mgr.y (mgr.34107) 10565 : cluster [DBG] pgmap v11236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:35:58 smithi115 bash[46934]: audit 2022-04-22T22:35:58.086368+0000 mon.a (mon.0) 14728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:35:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:58 smithi121 bash[36902]: cluster 2022-04-22T22:35:57.726719+0000 mgr.y (mgr.34107) 10565 : cluster [DBG] pgmap v11236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:35:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:35:58 smithi121 bash[36902]: audit 2022-04-22T22:35:58.086368+0000 mon.a (mon.0) 14728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:00 smithi115 bash[45495]: cluster 2022-04-22T22:35:59.727928+0000 mgr.y (mgr.34107) 10566 : cluster [DBG] pgmap v11237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:00 smithi115 bash[45495]: audit 2022-04-22T22:36:00.101312+0000 mon.a (mon.0) 14729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:00 smithi115 bash[46934]: cluster 2022-04-22T22:35:59.727928+0000 mgr.y (mgr.34107) 10566 : cluster [DBG] pgmap v11237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:00 smithi115 bash[46934]: audit 2022-04-22T22:36:00.101312+0000 mon.a (mon.0) 14729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:00 smithi121 bash[36902]: cluster 2022-04-22T22:35:59.727928+0000 mgr.y (mgr.34107) 10566 : cluster [DBG] pgmap v11237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:00 smithi121 bash[36902]: audit 2022-04-22T22:36:00.101312+0000 mon.a (mon.0) 14729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:02 smithi121 bash[36902]: cluster 2022-04-22T22:36:01.729111+0000 mgr.y (mgr.34107) 10567 : cluster [DBG] pgmap v11238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:02 smithi121 bash[36902]: audit 2022-04-22T22:36:01.732863+0000 mon.a (mon.0) 14730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:02.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:02 smithi115 bash[45495]: cluster 2022-04-22T22:36:01.729111+0000 mgr.y (mgr.34107) 10567 : cluster [DBG] pgmap v11238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:02 smithi115 bash[45495]: audit 2022-04-22T22:36:01.732863+0000 mon.a (mon.0) 14730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:03.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:02 smithi115 bash[46934]: cluster 2022-04-22T22:36:01.729111+0000 mgr.y (mgr.34107) 10567 : cluster [DBG] pgmap v11238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:03.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:02 smithi115 bash[46934]: audit 2022-04-22T22:36:01.732863+0000 mon.a (mon.0) 14730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:03 smithi121 bash[36902]: audit 2022-04-22T22:36:03.089965+0000 mon.a (mon.0) 14731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:03 smithi115 bash[45495]: audit 2022-04-22T22:36:03.089965+0000 mon.a (mon.0) 14731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:03 smithi115 bash[46934]: audit 2022-04-22T22:36:03.089965+0000 mon.a (mon.0) 14731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:04 smithi121 bash[36902]: cluster 2022-04-22T22:36:03.730397+0000 mgr.y (mgr.34107) 10568 : cluster [DBG] pgmap v11239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:04 smithi115 bash[45495]: cluster 2022-04-22T22:36:03.730397+0000 mgr.y (mgr.34107) 10568 : cluster [DBG] pgmap v11239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:04.693Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:04 smithi115 bash[46934]: cluster 2022-04-22T22:36:03.730397+0000 mgr.y (mgr.34107) 10568 : cluster [DBG] pgmap v11239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:05 smithi121 bash[36902]: audit 2022-04-22T22:36:05.104544+0000 mon.a (mon.0) 14732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:05 smithi115 bash[45495]: audit 2022-04-22T22:36:05.104544+0000 mon.a (mon.0) 14732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:05 smithi115 bash[46934]: audit 2022-04-22T22:36:05.104544+0000 mon.a (mon.0) 14732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:06 smithi121 bash[36902]: cluster 2022-04-22T22:36:05.731203+0000 mgr.y (mgr.34107) 10569 : cluster [DBG] pgmap v11240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:06 smithi115 bash[45495]: cluster 2022-04-22T22:36:05.731203+0000 mgr.y (mgr.34107) 10569 : cluster [DBG] pgmap v11240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:06 smithi115 bash[46934]: cluster 2022-04-22T22:36:05.731203+0000 mgr.y (mgr.34107) 10569 : cluster [DBG] pgmap v11240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:07 smithi121 bash[36902]: audit 2022-04-22T22:36:06.735529+0000 mon.a (mon.0) 14733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:07 smithi115 bash[45495]: audit 2022-04-22T22:36:06.735529+0000 mon.a (mon.0) 14733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:07 smithi115 bash[46934]: audit 2022-04-22T22:36:06.735529+0000 mon.a (mon.0) 14733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:08 smithi121 bash[36902]: cluster 2022-04-22T22:36:07.732587+0000 mgr.y (mgr.34107) 10570 : cluster [DBG] pgmap v11241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:08 smithi121 bash[36902]: audit 2022-04-22T22:36:08.093648+0000 mon.a (mon.0) 14734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:08 smithi115 bash[45495]: cluster 2022-04-22T22:36:07.732587+0000 mgr.y (mgr.34107) 10570 : cluster [DBG] pgmap v11241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:09.080 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:08 smithi115 bash[45495]: audit 2022-04-22T22:36:08.093648+0000 mon.a (mon.0) 14734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:08 smithi115 bash[46934]: cluster 2022-04-22T22:36:07.732587+0000 mgr.y (mgr.34107) 10570 : cluster [DBG] pgmap v11241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:08 smithi115 bash[46934]: audit 2022-04-22T22:36:08.093648+0000 mon.a (mon.0) 14734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:10 smithi121 bash[36902]: cluster 2022-04-22T22:36:09.733721+0000 mgr.y (mgr.34107) 10571 : cluster [DBG] pgmap v11242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:11.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:10 smithi121 bash[36902]: audit 2022-04-22T22:36:10.107550+0000 mon.a (mon.0) 14735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:10 smithi115 bash[45495]: cluster 2022-04-22T22:36:09.733721+0000 mgr.y (mgr.34107) 10571 : cluster [DBG] pgmap v11242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:10 smithi115 bash[45495]: audit 2022-04-22T22:36:10.107550+0000 mon.a (mon.0) 14735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:10 smithi115 bash[46934]: cluster 2022-04-22T22:36:09.733721+0000 mgr.y (mgr.34107) 10571 : cluster [DBG] pgmap v11242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:10 smithi115 bash[46934]: audit 2022-04-22T22:36:10.107550+0000 mon.a (mon.0) 14735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:12 smithi121 bash[36902]: cluster 2022-04-22T22:36:11.734824+0000 mgr.y (mgr.34107) 10572 : cluster [DBG] pgmap v11243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:12 smithi121 bash[36902]: audit 2022-04-22T22:36:11.738037+0000 mon.a (mon.0) 14736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:12.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:12 smithi115 bash[45495]: cluster 2022-04-22T22:36:11.734824+0000 mgr.y (mgr.34107) 10572 : cluster [DBG] pgmap v11243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:12 smithi115 bash[45495]: audit 2022-04-22T22:36:11.738037+0000 mon.a (mon.0) 14736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:12 smithi115 bash[46934]: cluster 2022-04-22T22:36:11.734824+0000 mgr.y (mgr.34107) 10572 : cluster [DBG] pgmap v11243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:12 smithi115 bash[46934]: audit 2022-04-22T22:36:11.738037+0000 mon.a (mon.0) 14736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:13 smithi121 bash[36902]: audit 2022-04-22T22:36:13.097378+0000 mon.a (mon.0) 14737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:13 smithi115 bash[45495]: audit 2022-04-22T22:36:13.097378+0000 mon.a (mon.0) 14737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:13 smithi115 bash[46934]: audit 2022-04-22T22:36:13.097378+0000 mon.a (mon.0) 14737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:14 smithi121 bash[36902]: cluster 2022-04-22T22:36:13.736067+0000 mgr.y (mgr.34107) 10573 : cluster [DBG] pgmap v11244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:14 smithi115 bash[45495]: cluster 2022-04-22T22:36:13.736067+0000 mgr.y (mgr.34107) 10573 : cluster [DBG] pgmap v11244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:14.693Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:14 smithi115 bash[46934]: cluster 2022-04-22T22:36:13.736067+0000 mgr.y (mgr.34107) 10573 : cluster [DBG] pgmap v11244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:15 smithi121 bash[36902]: audit 2022-04-22T22:36:15.110764+0000 mon.a (mon.0) 14738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:15 smithi115 bash[45495]: audit 2022-04-22T22:36:15.110764+0000 mon.a (mon.0) 14738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:15 smithi115 bash[46934]: audit 2022-04-22T22:36:15.110764+0000 mon.a (mon.0) 14738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:16 smithi121 bash[36902]: cluster 2022-04-22T22:36:15.737067+0000 mgr.y (mgr.34107) 10574 : cluster [DBG] pgmap v11245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:16 smithi115 bash[45495]: cluster 2022-04-22T22:36:15.737067+0000 mgr.y (mgr.34107) 10574 : cluster [DBG] pgmap v11245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:16 smithi115 bash[46934]: cluster 2022-04-22T22:36:15.737067+0000 mgr.y (mgr.34107) 10574 : cluster [DBG] pgmap v11245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:17 smithi121 bash[36902]: audit 2022-04-22T22:36:16.740657+0000 mon.a (mon.0) 14739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:17 smithi121 bash[36902]: audit 2022-04-22T22:36:16.846961+0000 mon.a (mon.0) 14740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:36:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:17 smithi121 bash[36902]: audit 2022-04-22T22:36:16.847738+0000 mon.a (mon.0) 14741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:36:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[45495]: audit 2022-04-22T22:36:16.740657+0000 mon.a (mon.0) 14739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[45495]: audit 2022-04-22T22:36:16.846961+0000 mon.a (mon.0) 14740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:36:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[45495]: audit 2022-04-22T22:36:16.847738+0000 mon.a (mon.0) 14741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:36:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[46934]: audit 2022-04-22T22:36:16.740657+0000 mon.a (mon.0) 14739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[46934]: audit 2022-04-22T22:36:16.846961+0000 mon.a (mon.0) 14740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:36:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:17 smithi115 bash[46934]: audit 2022-04-22T22:36:16.847738+0000 mon.a (mon.0) 14741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:36:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:18 smithi121 bash[36902]: cluster 2022-04-22T22:36:17.737965+0000 mgr.y (mgr.34107) 10575 : cluster [DBG] pgmap v11246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:18 smithi121 bash[36902]: audit 2022-04-22T22:36:18.101109+0000 mon.a (mon.0) 14742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:19.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:18 smithi115 bash[46934]: cluster 2022-04-22T22:36:17.737965+0000 mgr.y (mgr.34107) 10575 : cluster [DBG] pgmap v11246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:18 smithi115 bash[46934]: audit 2022-04-22T22:36:18.101109+0000 mon.a (mon.0) 14742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:18 smithi115 bash[45495]: cluster 2022-04-22T22:36:17.737965+0000 mgr.y (mgr.34107) 10575 : cluster [DBG] pgmap v11246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:18 smithi115 bash[45495]: audit 2022-04-22T22:36:18.101109+0000 mon.a (mon.0) 14742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:20 smithi121 bash[36902]: cluster 2022-04-22T22:36:19.738964+0000 mgr.y (mgr.34107) 10576 : cluster [DBG] pgmap v11247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:21.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:20 smithi121 bash[36902]: audit 2022-04-22T22:36:20.113832+0000 mon.a (mon.0) 14743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:20 smithi115 bash[45495]: cluster 2022-04-22T22:36:19.738964+0000 mgr.y (mgr.34107) 10576 : cluster [DBG] pgmap v11247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:20 smithi115 bash[45495]: audit 2022-04-22T22:36:20.113832+0000 mon.a (mon.0) 14743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:20 smithi115 bash[46934]: cluster 2022-04-22T22:36:19.738964+0000 mgr.y (mgr.34107) 10576 : cluster [DBG] pgmap v11247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:20 smithi115 bash[46934]: audit 2022-04-22T22:36:20.113832+0000 mon.a (mon.0) 14743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:21.266 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:36:21.614 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 10m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 56s ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 10m ago 5h 23.7M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 56s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 56s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 10m ago 5h 492M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 10m ago 5h 431M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 56s ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:36:21.615 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 10m ago 5h 290M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 10m ago 5h 10.2M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 56s ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 10m ago 5h 717M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 10m ago 5h 797M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 10m ago 5h 540M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 10m ago 5h 501M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 56s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 56s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 56s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 56s ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:36:21.616 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 56s ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:36:22.023 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:36:22.024 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:36:22.025 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:22.025 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:36:22.025 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:36:22.025 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:36:22.025 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:36:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:22 smithi121 bash[36902]: audit 2022-04-22T22:36:21.247307+0000 mgr.y (mgr.34107) 10577 : audit [DBG] from='client.56245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:22 smithi121 bash[36902]: audit 2022-04-22T22:36:21.590003+0000 mgr.y (mgr.34107) 10578 : audit [DBG] from='client.56251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:22 smithi121 bash[36902]: cluster 2022-04-22T22:36:21.739913+0000 mgr.y (mgr.34107) 10579 : cluster [DBG] pgmap v11248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:22 smithi121 bash[36902]: audit 2022-04-22T22:36:21.742923+0000 mon.a (mon.0) 14744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:23.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:22 smithi121 bash[36902]: audit 2022-04-22T22:36:22.020078+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.115:0/3394608608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:23.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:22.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[45495]: audit 2022-04-22T22:36:21.247307+0000 mgr.y (mgr.34107) 10577 : audit [DBG] from='client.56245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[45495]: audit 2022-04-22T22:36:21.590003+0000 mgr.y (mgr.34107) 10578 : audit [DBG] from='client.56251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[45495]: cluster 2022-04-22T22:36:21.739913+0000 mgr.y (mgr.34107) 10579 : cluster [DBG] pgmap v11248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[45495]: audit 2022-04-22T22:36:21.742923+0000 mon.a (mon.0) 14744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[45495]: audit 2022-04-22T22:36:22.020078+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.115:0/3394608608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[46934]: audit 2022-04-22T22:36:21.247307+0000 mgr.y (mgr.34107) 10577 : audit [DBG] from='client.56245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[46934]: audit 2022-04-22T22:36:21.590003+0000 mgr.y (mgr.34107) 10578 : audit [DBG] from='client.56251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:23.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[46934]: cluster 2022-04-22T22:36:21.739913+0000 mgr.y (mgr.34107) 10579 : cluster [DBG] pgmap v11248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[46934]: audit 2022-04-22T22:36:21.742923+0000 mon.a (mon.0) 14744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:22 smithi115 bash[46934]: audit 2022-04-22T22:36:22.020078+0000 mon.c (mon.1) 357 : audit [DBG] from='client.? 172.21.15.115:0/3394608608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:23 smithi121 bash[36902]: audit 2022-04-22T22:36:23.104822+0000 mon.a (mon.0) 14745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:23 smithi115 bash[45495]: audit 2022-04-22T22:36:23.104822+0000 mon.a (mon.0) 14745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:23 smithi115 bash[46934]: audit 2022-04-22T22:36:23.104822+0000 mon.a (mon.0) 14745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:24 smithi121 bash[36902]: cluster 2022-04-22T22:36:23.740968+0000 mgr.y (mgr.34107) 10580 : cluster [DBG] pgmap v11249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:24 smithi115 bash[45495]: cluster 2022-04-22T22:36:23.740968+0000 mgr.y (mgr.34107) 10580 : cluster [DBG] pgmap v11249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:24.693Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:24 smithi115 bash[46934]: cluster 2022-04-22T22:36:23.740968+0000 mgr.y (mgr.34107) 10580 : cluster [DBG] pgmap v11249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.116969+0000 mon.a (mon.0) 14746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:26.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.132058+0000 mon.a (mon.0) 14747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:36:26.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.134730+0000 mon.a (mon.0) 14748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:36:26.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.136494+0000 mon.a (mon.0) 14749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:36:26.055 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.464397+0000 mon.a (mon.0) 14750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:26.055 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:25 smithi121 bash[36902]: audit 2022-04-22T22:36:25.481992+0000 mon.a (mon.0) 14751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:26.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.116969+0000 mon.a (mon.0) 14746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:26.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.132058+0000 mon.a (mon.0) 14747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:36:26.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.134730+0000 mon.a (mon.0) 14748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:36:26.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.136494+0000 mon.a (mon.0) 14749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:36:26.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.464397+0000 mon.a (mon.0) 14750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:26.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[45495]: audit 2022-04-22T22:36:25.481992+0000 mon.a (mon.0) 14751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:26.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.116969+0000 mon.a (mon.0) 14746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:26.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.132058+0000 mon.a (mon.0) 14747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:36:26.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.134730+0000 mon.a (mon.0) 14748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:36:26.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.136494+0000 mon.a (mon.0) 14749 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:36:26.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.464397+0000 mon.a (mon.0) 14750 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:26.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:25 smithi115 bash[46934]: audit 2022-04-22T22:36:25.481992+0000 mon.a (mon.0) 14751 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:26 smithi121 bash[36902]: cluster 2022-04-22T22:36:25.741902+0000 mgr.y (mgr.34107) 10581 : cluster [DBG] pgmap v11250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:26 smithi115 bash[45495]: cluster 2022-04-22T22:36:25.741902+0000 mgr.y (mgr.34107) 10581 : cluster [DBG] pgmap v11250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:26 smithi115 bash[46934]: cluster 2022-04-22T22:36:25.741902+0000 mgr.y (mgr.34107) 10581 : cluster [DBG] pgmap v11250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:27 smithi121 bash[36902]: audit 2022-04-22T22:36:26.745496+0000 mon.a (mon.0) 14752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:28.051 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:27 smithi115 bash[45495]: audit 2022-04-22T22:36:26.745496+0000 mon.a (mon.0) 14752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:28.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:27 smithi115 bash[46934]: audit 2022-04-22T22:36:26.745496+0000 mon.a (mon.0) 14752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:28 smithi121 bash[36902]: cluster 2022-04-22T22:36:27.743161+0000 mgr.y (mgr.34107) 10582 : cluster [DBG] pgmap v11251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:28 smithi121 bash[36902]: audit 2022-04-22T22:36:28.108680+0000 mon.a (mon.0) 14753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:29.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:28 smithi115 bash[46934]: cluster 2022-04-22T22:36:27.743161+0000 mgr.y (mgr.34107) 10582 : cluster [DBG] pgmap v11251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:28 smithi115 bash[46934]: audit 2022-04-22T22:36:28.108680+0000 mon.a (mon.0) 14753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:28 smithi115 bash[45495]: cluster 2022-04-22T22:36:27.743161+0000 mgr.y (mgr.34107) 10582 : cluster [DBG] pgmap v11251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:28 smithi115 bash[45495]: audit 2022-04-22T22:36:28.108680+0000 mon.a (mon.0) 14753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:30 smithi121 bash[36902]: audit 2022-04-22T22:36:29.618614+0000 mon.a (mon.0) 14754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:30 smithi121 bash[36902]: cluster 2022-04-22T22:36:29.744266+0000 mgr.y (mgr.34107) 10583 : cluster [DBG] pgmap v11252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:30 smithi121 bash[36902]: audit 2022-04-22T22:36:29.954607+0000 mon.a (mon.0) 14755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:30 smithi121 bash[36902]: audit 2022-04-22T22:36:30.119828+0000 mon.a (mon.0) 14756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[45495]: audit 2022-04-22T22:36:29.618614+0000 mon.a (mon.0) 14754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[45495]: cluster 2022-04-22T22:36:29.744266+0000 mgr.y (mgr.34107) 10583 : cluster [DBG] pgmap v11252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[45495]: audit 2022-04-22T22:36:29.954607+0000 mon.a (mon.0) 14755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[45495]: audit 2022-04-22T22:36:30.119828+0000 mon.a (mon.0) 14756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[46934]: audit 2022-04-22T22:36:29.618614+0000 mon.a (mon.0) 14754 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[46934]: cluster 2022-04-22T22:36:29.744266+0000 mgr.y (mgr.34107) 10583 : cluster [DBG] pgmap v11252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[46934]: audit 2022-04-22T22:36:29.954607+0000 mon.a (mon.0) 14755 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:36:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:30 smithi115 bash[46934]: audit 2022-04-22T22:36:30.119828+0000 mon.a (mon.0) 14756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:32 smithi121 bash[36902]: cluster 2022-04-22T22:36:31.745377+0000 mgr.y (mgr.34107) 10584 : cluster [DBG] pgmap v11253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:32 smithi121 bash[36902]: audit 2022-04-22T22:36:31.748106+0000 mon.a (mon.0) 14757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:33.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:32.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:32 smithi115 bash[45495]: cluster 2022-04-22T22:36:31.745377+0000 mgr.y (mgr.34107) 10584 : cluster [DBG] pgmap v11253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:32 smithi115 bash[45495]: audit 2022-04-22T22:36:31.748106+0000 mon.a (mon.0) 14757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:32 smithi115 bash[46934]: cluster 2022-04-22T22:36:31.745377+0000 mgr.y (mgr.34107) 10584 : cluster [DBG] pgmap v11253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:32 smithi115 bash[46934]: audit 2022-04-22T22:36:31.748106+0000 mon.a (mon.0) 14757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:33 smithi121 bash[36902]: audit 2022-04-22T22:36:33.112081+0000 mon.a (mon.0) 14758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:33 smithi115 bash[45495]: audit 2022-04-22T22:36:33.112081+0000 mon.a (mon.0) 14758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:34.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:33 smithi115 bash[46934]: audit 2022-04-22T22:36:33.112081+0000 mon.a (mon.0) 14758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:34 smithi121 bash[36902]: cluster 2022-04-22T22:36:33.746673+0000 mgr.y (mgr.34107) 10585 : cluster [DBG] pgmap v11254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:34 smithi115 bash[45495]: cluster 2022-04-22T22:36:33.746673+0000 mgr.y (mgr.34107) 10585 : cluster [DBG] pgmap v11254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:34.695Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:34 smithi115 bash[46934]: cluster 2022-04-22T22:36:33.746673+0000 mgr.y (mgr.34107) 10585 : cluster [DBG] pgmap v11254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:35 smithi121 bash[36902]: audit 2022-04-22T22:36:35.122770+0000 mon.a (mon.0) 14759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:35 smithi115 bash[45495]: audit 2022-04-22T22:36:35.122770+0000 mon.a (mon.0) 14759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:35 smithi115 bash[46934]: audit 2022-04-22T22:36:35.122770+0000 mon.a (mon.0) 14759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:36 smithi121 bash[36902]: cluster 2022-04-22T22:36:35.747850+0000 mgr.y (mgr.34107) 10586 : cluster [DBG] pgmap v11255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:36 smithi115 bash[45495]: cluster 2022-04-22T22:36:35.747850+0000 mgr.y (mgr.34107) 10586 : cluster [DBG] pgmap v11255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:36 smithi115 bash[46934]: cluster 2022-04-22T22:36:35.747850+0000 mgr.y (mgr.34107) 10586 : cluster [DBG] pgmap v11255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:37 smithi121 bash[36902]: audit 2022-04-22T22:36:36.751161+0000 mon.a (mon.0) 14760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:37 smithi115 bash[45495]: audit 2022-04-22T22:36:36.751161+0000 mon.a (mon.0) 14760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:37 smithi115 bash[46934]: audit 2022-04-22T22:36:36.751161+0000 mon.a (mon.0) 14760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:38 smithi121 bash[36902]: cluster 2022-04-22T22:36:37.748572+0000 mgr.y (mgr.34107) 10587 : cluster [DBG] pgmap v11256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:38 smithi121 bash[36902]: audit 2022-04-22T22:36:38.115956+0000 mon.a (mon.0) 14761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:38 smithi115 bash[45495]: cluster 2022-04-22T22:36:37.748572+0000 mgr.y (mgr.34107) 10587 : cluster [DBG] pgmap v11256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:38 smithi115 bash[45495]: audit 2022-04-22T22:36:38.115956+0000 mon.a (mon.0) 14761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:38 smithi115 bash[46934]: cluster 2022-04-22T22:36:37.748572+0000 mgr.y (mgr.34107) 10587 : cluster [DBG] pgmap v11256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:38 smithi115 bash[46934]: audit 2022-04-22T22:36:38.115956+0000 mon.a (mon.0) 14761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:39 smithi121 bash[36902]: cluster 2022-04-22T22:36:39.749724+0000 mgr.y (mgr.34107) 10588 : cluster [DBG] pgmap v11257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:39 smithi115 bash[45495]: cluster 2022-04-22T22:36:39.749724+0000 mgr.y (mgr.34107) 10588 : cluster [DBG] pgmap v11257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:39 smithi115 bash[46934]: cluster 2022-04-22T22:36:39.749724+0000 mgr.y (mgr.34107) 10588 : cluster [DBG] pgmap v11257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:40 smithi121 bash[36902]: audit 2022-04-22T22:36:40.125843+0000 mon.a (mon.0) 14762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:40 smithi115 bash[45495]: audit 2022-04-22T22:36:40.125843+0000 mon.a (mon.0) 14762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:40 smithi115 bash[46934]: audit 2022-04-22T22:36:40.125843+0000 mon.a (mon.0) 14762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:42 smithi121 bash[36902]: cluster 2022-04-22T22:36:41.750809+0000 mgr.y (mgr.34107) 10589 : cluster [DBG] pgmap v11258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:42 smithi121 bash[36902]: audit 2022-04-22T22:36:41.754024+0000 mon.a (mon.0) 14763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:43.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:42 smithi115 bash[45495]: cluster 2022-04-22T22:36:41.750809+0000 mgr.y (mgr.34107) 10589 : cluster [DBG] pgmap v11258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:42 smithi115 bash[45495]: audit 2022-04-22T22:36:41.754024+0000 mon.a (mon.0) 14763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:42.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:42 smithi115 bash[46934]: cluster 2022-04-22T22:36:41.750809+0000 mgr.y (mgr.34107) 10589 : cluster [DBG] pgmap v11258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:42 smithi115 bash[46934]: audit 2022-04-22T22:36:41.754024+0000 mon.a (mon.0) 14763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:43 smithi121 bash[36902]: audit 2022-04-22T22:36:43.119974+0000 mon.a (mon.0) 14764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:43 smithi115 bash[45495]: audit 2022-04-22T22:36:43.119974+0000 mon.a (mon.0) 14764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:43 smithi115 bash[46934]: audit 2022-04-22T22:36:43.119974+0000 mon.a (mon.0) 14764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:44 smithi121 bash[36902]: cluster 2022-04-22T22:36:43.752024+0000 mgr.y (mgr.34107) 10590 : cluster [DBG] pgmap v11259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:44 smithi115 bash[45495]: cluster 2022-04-22T22:36:43.752024+0000 mgr.y (mgr.34107) 10590 : cluster [DBG] pgmap v11259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:44.694Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:44 smithi115 bash[46934]: cluster 2022-04-22T22:36:43.752024+0000 mgr.y (mgr.34107) 10590 : cluster [DBG] pgmap v11259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:45 smithi121 bash[36902]: audit 2022-04-22T22:36:45.128842+0000 mon.a (mon.0) 14765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:45 smithi115 bash[45495]: audit 2022-04-22T22:36:45.128842+0000 mon.a (mon.0) 14765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:46.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:45 smithi115 bash[46934]: audit 2022-04-22T22:36:45.128842+0000 mon.a (mon.0) 14765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:46 smithi121 bash[36902]: cluster 2022-04-22T22:36:45.753113+0000 mgr.y (mgr.34107) 10591 : cluster [DBG] pgmap v11260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:46 smithi115 bash[45495]: cluster 2022-04-22T22:36:45.753113+0000 mgr.y (mgr.34107) 10591 : cluster [DBG] pgmap v11260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:46 smithi115 bash[46934]: cluster 2022-04-22T22:36:45.753113+0000 mgr.y (mgr.34107) 10591 : cluster [DBG] pgmap v11260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:47 smithi121 bash[36902]: audit 2022-04-22T22:36:46.756928+0000 mon.a (mon.0) 14766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:47 smithi115 bash[45495]: audit 2022-04-22T22:36:46.756928+0000 mon.a (mon.0) 14766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:47 smithi115 bash[46934]: audit 2022-04-22T22:36:46.756928+0000 mon.a (mon.0) 14766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:48 smithi121 bash[36902]: cluster 2022-04-22T22:36:47.754177+0000 mgr.y (mgr.34107) 10592 : cluster [DBG] pgmap v11261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:48 smithi121 bash[36902]: audit 2022-04-22T22:36:48.123812+0000 mon.a (mon.0) 14767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:48 smithi115 bash[45495]: cluster 2022-04-22T22:36:47.754177+0000 mgr.y (mgr.34107) 10592 : cluster [DBG] pgmap v11261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:48 smithi115 bash[45495]: audit 2022-04-22T22:36:48.123812+0000 mon.a (mon.0) 14767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:48 smithi115 bash[46934]: cluster 2022-04-22T22:36:47.754177+0000 mgr.y (mgr.34107) 10592 : cluster [DBG] pgmap v11261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:48 smithi115 bash[46934]: audit 2022-04-22T22:36:48.123812+0000 mon.a (mon.0) 14767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:49 smithi121 bash[36902]: cluster 2022-04-22T22:36:49.755312+0000 mgr.y (mgr.34107) 10593 : cluster [DBG] pgmap v11262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:49 smithi115 bash[45495]: cluster 2022-04-22T22:36:49.755312+0000 mgr.y (mgr.34107) 10593 : cluster [DBG] pgmap v11262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:49 smithi115 bash[46934]: cluster 2022-04-22T22:36:49.755312+0000 mgr.y (mgr.34107) 10593 : cluster [DBG] pgmap v11262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:50 smithi121 bash[36902]: audit 2022-04-22T22:36:50.131864+0000 mon.a (mon.0) 14768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:51.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:50 smithi115 bash[46934]: audit 2022-04-22T22:36:50.131864+0000 mon.a (mon.0) 14768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:50 smithi115 bash[45495]: audit 2022-04-22T22:36:50.131864+0000 mon.a (mon.0) 14768 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:52.400 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:36:52.744 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:36:52.744 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4h) 23s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:36:52.744 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 87s ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:36:52.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 23s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:36:52.744 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 87s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 87s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 23s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 23s ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 87s ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 23s ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 23s ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 87s ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 23s ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 23s ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:36:52.745 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 23s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 23s ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 87s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 87s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 87s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 87s ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:36:52.746 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 87s ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:36:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:52 smithi121 bash[36902]: cluster 2022-04-22T22:36:51.756398+0000 mgr.y (mgr.34107) 10594 : cluster [DBG] pgmap v11263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:52 smithi121 bash[36902]: audit 2022-04-22T22:36:51.759620+0000 mon.a (mon.0) 14769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:52.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:52 smithi115 bash[45495]: cluster 2022-04-22T22:36:51.756398+0000 mgr.y (mgr.34107) 10594 : cluster [DBG] pgmap v11263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:52 smithi115 bash[45495]: audit 2022-04-22T22:36:51.759620+0000 mon.a (mon.0) 14769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:52 smithi115 bash[46934]: cluster 2022-04-22T22:36:51.756398+0000 mgr.y (mgr.34107) 10594 : cluster [DBG] pgmap v11263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:52 smithi115 bash[46934]: audit 2022-04-22T22:36:51.759620+0000 mon.a (mon.0) 14769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:53.154 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:53.155 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:36:53.156 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:36:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:53 smithi121 bash[36902]: audit 2022-04-22T22:36:52.382444+0000 mgr.y (mgr.34107) 10595 : audit [DBG] from='client.56263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:53 smithi121 bash[36902]: audit 2022-04-22T22:36:52.724465+0000 mgr.y (mgr.34107) 10596 : audit [DBG] from='client.56269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:53 smithi121 bash[36902]: audit 2022-04-22T22:36:53.127533+0000 mon.a (mon.0) 14770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:53 smithi121 bash[36902]: audit 2022-04-22T22:36:53.151177+0000 mon.a (mon.0) 14771 : audit [DBG] from='client.? 172.21.15.115:0/254294550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[45495]: audit 2022-04-22T22:36:52.382444+0000 mgr.y (mgr.34107) 10595 : audit [DBG] from='client.56263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[45495]: audit 2022-04-22T22:36:52.724465+0000 mgr.y (mgr.34107) 10596 : audit [DBG] from='client.56269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[45495]: audit 2022-04-22T22:36:53.127533+0000 mon.a (mon.0) 14770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[45495]: audit 2022-04-22T22:36:53.151177+0000 mon.a (mon.0) 14771 : audit [DBG] from='client.? 172.21.15.115:0/254294550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[46934]: audit 2022-04-22T22:36:52.382444+0000 mgr.y (mgr.34107) 10595 : audit [DBG] from='client.56263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[46934]: audit 2022-04-22T22:36:52.724465+0000 mgr.y (mgr.34107) 10596 : audit [DBG] from='client.56269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:36:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[46934]: audit 2022-04-22T22:36:53.127533+0000 mon.a (mon.0) 14770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:54.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:53 smithi115 bash[46934]: audit 2022-04-22T22:36:53.151177+0000 mon.a (mon.0) 14771 : audit [DBG] from='client.? 172.21.15.115:0/254294550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:36:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:54 smithi121 bash[36902]: cluster 2022-04-22T22:36:53.757573+0000 mgr.y (mgr.34107) 10597 : cluster [DBG] pgmap v11264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:54 smithi115 bash[45495]: cluster 2022-04-22T22:36:53.757573+0000 mgr.y (mgr.34107) 10597 : cluster [DBG] pgmap v11264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:36:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:36:54.695Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:36:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:54 smithi115 bash[46934]: cluster 2022-04-22T22:36:53.757573+0000 mgr.y (mgr.34107) 10597 : cluster [DBG] pgmap v11264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:55 smithi121 bash[36902]: audit 2022-04-22T22:36:55.134764+0000 mon.a (mon.0) 14772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:55 smithi115 bash[45495]: audit 2022-04-22T22:36:55.134764+0000 mon.a (mon.0) 14772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:55 smithi115 bash[46934]: audit 2022-04-22T22:36:55.134764+0000 mon.a (mon.0) 14772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:36:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:56 smithi121 bash[36902]: cluster 2022-04-22T22:36:55.758671+0000 mgr.y (mgr.34107) 10598 : cluster [DBG] pgmap v11265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:56 smithi115 bash[45495]: cluster 2022-04-22T22:36:55.758671+0000 mgr.y (mgr.34107) 10598 : cluster [DBG] pgmap v11265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:56 smithi115 bash[46934]: cluster 2022-04-22T22:36:55.758671+0000 mgr.y (mgr.34107) 10598 : cluster [DBG] pgmap v11265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:57 smithi121 bash[36902]: audit 2022-04-22T22:36:56.762657+0000 mon.a (mon.0) 14773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:57 smithi115 bash[45495]: audit 2022-04-22T22:36:56.762657+0000 mon.a (mon.0) 14773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:57 smithi115 bash[46934]: audit 2022-04-22T22:36:56.762657+0000 mon.a (mon.0) 14773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:36:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:58 smithi121 bash[36902]: cluster 2022-04-22T22:36:57.759773+0000 mgr.y (mgr.34107) 10599 : cluster [DBG] pgmap v11266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:58 smithi121 bash[36902]: audit 2022-04-22T22:36:58.131002+0000 mon.a (mon.0) 14774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:58 smithi115 bash[45495]: cluster 2022-04-22T22:36:57.759773+0000 mgr.y (mgr.34107) 10599 : cluster [DBG] pgmap v11266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:58 smithi115 bash[45495]: audit 2022-04-22T22:36:58.131002+0000 mon.a (mon.0) 14774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:36:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:58 smithi115 bash[46934]: cluster 2022-04-22T22:36:57.759773+0000 mgr.y (mgr.34107) 10599 : cluster [DBG] pgmap v11266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:36:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:58 smithi115 bash[46934]: audit 2022-04-22T22:36:58.131002+0000 mon.a (mon.0) 14774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:36:59 smithi121 bash[36902]: cluster 2022-04-22T22:36:59.760893+0000 mgr.y (mgr.34107) 10600 : cluster [DBG] pgmap v11267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:36:59 smithi115 bash[45495]: cluster 2022-04-22T22:36:59.760893+0000 mgr.y (mgr.34107) 10600 : cluster [DBG] pgmap v11267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:36:59 smithi115 bash[46934]: cluster 2022-04-22T22:36:59.760893+0000 mgr.y (mgr.34107) 10600 : cluster [DBG] pgmap v11267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:00 smithi121 bash[36902]: audit 2022-04-22T22:37:00.137950+0000 mon.a (mon.0) 14775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:00 smithi115 bash[45495]: audit 2022-04-22T22:37:00.137950+0000 mon.a (mon.0) 14775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:00 smithi115 bash[46934]: audit 2022-04-22T22:37:00.137950+0000 mon.a (mon.0) 14775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:02 smithi121 bash[36902]: cluster 2022-04-22T22:37:01.762116+0000 mgr.y (mgr.34107) 10601 : cluster [DBG] pgmap v11268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:02 smithi121 bash[36902]: audit 2022-04-22T22:37:01.765622+0000 mon.a (mon.0) 14776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:02.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:02 smithi115 bash[46934]: cluster 2022-04-22T22:37:01.762116+0000 mgr.y (mgr.34107) 10601 : cluster [DBG] pgmap v11268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:03.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:02 smithi115 bash[46934]: audit 2022-04-22T22:37:01.765622+0000 mon.a (mon.0) 14776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:03.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:02 smithi115 bash[45495]: cluster 2022-04-22T22:37:01.762116+0000 mgr.y (mgr.34107) 10601 : cluster [DBG] pgmap v11268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:02 smithi115 bash[45495]: audit 2022-04-22T22:37:01.765622+0000 mon.a (mon.0) 14776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:03 smithi121 bash[36902]: audit 2022-04-22T22:37:03.134760+0000 mon.a (mon.0) 14777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:03 smithi115 bash[45495]: audit 2022-04-22T22:37:03.134760+0000 mon.a (mon.0) 14777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:03 smithi115 bash[46934]: audit 2022-04-22T22:37:03.134760+0000 mon.a (mon.0) 14777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:04 smithi121 bash[36902]: cluster 2022-04-22T22:37:03.763320+0000 mgr.y (mgr.34107) 10602 : cluster [DBG] pgmap v11269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:04.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:04 smithi115 bash[45495]: cluster 2022-04-22T22:37:03.763320+0000 mgr.y (mgr.34107) 10602 : cluster [DBG] pgmap v11269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:04 smithi115 bash[46934]: cluster 2022-04-22T22:37:03.763320+0000 mgr.y (mgr.34107) 10602 : cluster [DBG] pgmap v11269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:05 smithi121 bash[36902]: audit 2022-04-22T22:37:05.140879+0000 mon.a (mon.0) 14778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:05 smithi115 bash[45495]: audit 2022-04-22T22:37:05.140879+0000 mon.a (mon.0) 14778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:05 smithi115 bash[46934]: audit 2022-04-22T22:37:05.140879+0000 mon.a (mon.0) 14778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:06 smithi121 bash[36902]: cluster 2022-04-22T22:37:05.764457+0000 mgr.y (mgr.34107) 10603 : cluster [DBG] pgmap v11270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:06 smithi115 bash[45495]: cluster 2022-04-22T22:37:05.764457+0000 mgr.y (mgr.34107) 10603 : cluster [DBG] pgmap v11270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:06 smithi115 bash[46934]: cluster 2022-04-22T22:37:05.764457+0000 mgr.y (mgr.34107) 10603 : cluster [DBG] pgmap v11270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:07 smithi121 bash[36902]: audit 2022-04-22T22:37:06.768571+0000 mon.a (mon.0) 14779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:07 smithi115 bash[45495]: audit 2022-04-22T22:37:06.768571+0000 mon.a (mon.0) 14779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:07 smithi115 bash[46934]: audit 2022-04-22T22:37:06.768571+0000 mon.a (mon.0) 14779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:08 smithi121 bash[36902]: cluster 2022-04-22T22:37:07.765579+0000 mgr.y (mgr.34107) 10604 : cluster [DBG] pgmap v11271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:08 smithi121 bash[36902]: audit 2022-04-22T22:37:08.138568+0000 mon.a (mon.0) 14780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:08 smithi115 bash[45495]: cluster 2022-04-22T22:37:07.765579+0000 mgr.y (mgr.34107) 10604 : cluster [DBG] pgmap v11271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:08 smithi115 bash[45495]: audit 2022-04-22T22:37:08.138568+0000 mon.a (mon.0) 14780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:08 smithi115 bash[46934]: cluster 2022-04-22T22:37:07.765579+0000 mgr.y (mgr.34107) 10604 : cluster [DBG] pgmap v11271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:08 smithi115 bash[46934]: audit 2022-04-22T22:37:08.138568+0000 mon.a (mon.0) 14780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:09 smithi121 bash[36902]: cluster 2022-04-22T22:37:09.766606+0000 mgr.y (mgr.34107) 10605 : cluster [DBG] pgmap v11272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:09 smithi115 bash[45495]: cluster 2022-04-22T22:37:09.766606+0000 mgr.y (mgr.34107) 10605 : cluster [DBG] pgmap v11272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:09 smithi115 bash[46934]: cluster 2022-04-22T22:37:09.766606+0000 mgr.y (mgr.34107) 10605 : cluster [DBG] pgmap v11272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:10 smithi121 bash[36902]: audit 2022-04-22T22:37:10.144128+0000 mon.a (mon.0) 14781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:10 smithi115 bash[45495]: audit 2022-04-22T22:37:10.144128+0000 mon.a (mon.0) 14781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:10 smithi115 bash[46934]: audit 2022-04-22T22:37:10.144128+0000 mon.a (mon.0) 14781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:12 smithi121 bash[36902]: cluster 2022-04-22T22:37:11.767727+0000 mgr.y (mgr.34107) 10606 : cluster [DBG] pgmap v11273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:12 smithi121 bash[36902]: audit 2022-04-22T22:37:11.771617+0000 mon.a (mon.0) 14782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:13.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:12.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:12 smithi115 bash[45495]: cluster 2022-04-22T22:37:11.767727+0000 mgr.y (mgr.34107) 10606 : cluster [DBG] pgmap v11273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:13.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:12 smithi115 bash[45495]: audit 2022-04-22T22:37:11.771617+0000 mon.a (mon.0) 14782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:12 smithi115 bash[46934]: cluster 2022-04-22T22:37:11.767727+0000 mgr.y (mgr.34107) 10606 : cluster [DBG] pgmap v11273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:12 smithi115 bash[46934]: audit 2022-04-22T22:37:11.771617+0000 mon.a (mon.0) 14782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:13 smithi121 bash[36902]: audit 2022-04-22T22:37:13.142410+0000 mon.a (mon.0) 14783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:13 smithi115 bash[45495]: audit 2022-04-22T22:37:13.142410+0000 mon.a (mon.0) 14783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:13 smithi115 bash[46934]: audit 2022-04-22T22:37:13.142410+0000 mon.a (mon.0) 14783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:14 smithi121 bash[36902]: cluster 2022-04-22T22:37:13.768850+0000 mgr.y (mgr.34107) 10607 : cluster [DBG] pgmap v11274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:14.696Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:14 smithi115 bash[45495]: cluster 2022-04-22T22:37:13.768850+0000 mgr.y (mgr.34107) 10607 : cluster [DBG] pgmap v11274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:14 smithi115 bash[46934]: cluster 2022-04-22T22:37:13.768850+0000 mgr.y (mgr.34107) 10607 : cluster [DBG] pgmap v11274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:15 smithi121 bash[36902]: audit 2022-04-22T22:37:15.147193+0000 mon.a (mon.0) 14784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:16.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:15 smithi115 bash[46934]: audit 2022-04-22T22:37:15.147193+0000 mon.a (mon.0) 14784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:15 smithi115 bash[45495]: audit 2022-04-22T22:37:15.147193+0000 mon.a (mon.0) 14784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:16 smithi121 bash[36902]: cluster 2022-04-22T22:37:15.769921+0000 mgr.y (mgr.34107) 10608 : cluster [DBG] pgmap v11275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:16 smithi115 bash[45495]: cluster 2022-04-22T22:37:15.769921+0000 mgr.y (mgr.34107) 10608 : cluster [DBG] pgmap v11275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:16 smithi115 bash[46934]: cluster 2022-04-22T22:37:15.769921+0000 mgr.y (mgr.34107) 10608 : cluster [DBG] pgmap v11275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:17 smithi121 bash[36902]: audit 2022-04-22T22:37:16.774615+0000 mon.a (mon.0) 14785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:17 smithi121 bash[36902]: audit 2022-04-22T22:37:16.859585+0000 mon.a (mon.0) 14786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:37:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:17 smithi121 bash[36902]: audit 2022-04-22T22:37:16.860050+0000 mon.a (mon.0) 14787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:37:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[45495]: audit 2022-04-22T22:37:16.774615+0000 mon.a (mon.0) 14785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[45495]: audit 2022-04-22T22:37:16.859585+0000 mon.a (mon.0) 14786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:37:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[45495]: audit 2022-04-22T22:37:16.860050+0000 mon.a (mon.0) 14787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:37:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[46934]: audit 2022-04-22T22:37:16.774615+0000 mon.a (mon.0) 14785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[46934]: audit 2022-04-22T22:37:16.859585+0000 mon.a (mon.0) 14786 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:37:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:17 smithi115 bash[46934]: audit 2022-04-22T22:37:16.860050+0000 mon.a (mon.0) 14787 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:37:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:18 smithi121 bash[36902]: cluster 2022-04-22T22:37:17.770915+0000 mgr.y (mgr.34107) 10609 : cluster [DBG] pgmap v11276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:18 smithi121 bash[36902]: audit 2022-04-22T22:37:18.146393+0000 mon.a (mon.0) 14788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:18 smithi115 bash[45495]: cluster 2022-04-22T22:37:17.770915+0000 mgr.y (mgr.34107) 10609 : cluster [DBG] pgmap v11276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:18 smithi115 bash[45495]: audit 2022-04-22T22:37:18.146393+0000 mon.a (mon.0) 14788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:18 smithi115 bash[46934]: cluster 2022-04-22T22:37:17.770915+0000 mgr.y (mgr.34107) 10609 : cluster [DBG] pgmap v11276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:18 smithi115 bash[46934]: audit 2022-04-22T22:37:18.146393+0000 mon.a (mon.0) 14788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:19 smithi121 bash[36902]: cluster 2022-04-22T22:37:19.772042+0000 mgr.y (mgr.34107) 10610 : cluster [DBG] pgmap v11277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:19 smithi115 bash[45495]: cluster 2022-04-22T22:37:19.772042+0000 mgr.y (mgr.34107) 10610 : cluster [DBG] pgmap v11277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:19 smithi115 bash[46934]: cluster 2022-04-22T22:37:19.772042+0000 mgr.y (mgr.34107) 10610 : cluster [DBG] pgmap v11277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:20 smithi121 bash[36902]: audit 2022-04-22T22:37:20.150177+0000 mon.a (mon.0) 14789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:20 smithi115 bash[45495]: audit 2022-04-22T22:37:20.150177+0000 mon.a (mon.0) 14789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:20 smithi115 bash[46934]: audit 2022-04-22T22:37:20.150177+0000 mon.a (mon.0) 14789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:22 smithi121 bash[36902]: cluster 2022-04-22T22:37:21.773131+0000 mgr.y (mgr.34107) 10611 : cluster [DBG] pgmap v11278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:23.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:22 smithi121 bash[36902]: audit 2022-04-22T22:37:21.777898+0000 mon.a (mon.0) 14790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:23.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:22.841Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:23.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:22 smithi115 bash[45495]: cluster 2022-04-22T22:37:21.773131+0000 mgr.y (mgr.34107) 10611 : cluster [DBG] pgmap v11278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:23.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:22 smithi115 bash[45495]: audit 2022-04-22T22:37:21.777898+0000 mon.a (mon.0) 14790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:22 smithi115 bash[46934]: cluster 2022-04-22T22:37:21.773131+0000 mgr.y (mgr.34107) 10611 : cluster [DBG] pgmap v11278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:23.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:22 smithi115 bash[46934]: audit 2022-04-22T22:37:21.777898+0000 mon.a (mon.0) 14790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:23.539 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:37:23.891 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:37:23.891 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 54s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 119s ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 54s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 119s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 119s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 54s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 54s ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 119s ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 54s ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 54s ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:37:23.892 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 119s ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 54s ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 54s ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 54s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 54s ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 119s ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 119s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 119s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 119s ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:37:23.893 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 119s ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:37:24.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:23 smithi121 bash[36902]: audit 2022-04-22T22:37:23.149839+0000 mon.a (mon.0) 14791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:24.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:23 smithi115 bash[45495]: audit 2022-04-22T22:37:23.149839+0000 mon.a (mon.0) 14791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:23 smithi115 bash[46934]: audit 2022-04-22T22:37:23.149839+0000 mon.a (mon.0) 14791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:24.299 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:37:24.299 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:37:24.299 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:37:24.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:24.300 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:37:24.301 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:37:24.301 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:37:24.301 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:37:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:24 smithi121 bash[36902]: audit 2022-04-22T22:37:23.518913+0000 mgr.y (mgr.34107) 10612 : audit [DBG] from='client.46347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:24 smithi121 bash[36902]: cluster 2022-04-22T22:37:23.774217+0000 mgr.y (mgr.34107) 10613 : cluster [DBG] pgmap v11279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:24 smithi121 bash[36902]: audit 2022-04-22T22:37:24.295549+0000 mon.c (mon.1) 358 : audit [DBG] from='client.? 172.21.15.115:0/3897483885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:24.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[45495]: audit 2022-04-22T22:37:23.518913+0000 mgr.y (mgr.34107) 10612 : audit [DBG] from='client.46347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[45495]: cluster 2022-04-22T22:37:23.774217+0000 mgr.y (mgr.34107) 10613 : cluster [DBG] pgmap v11279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[45495]: audit 2022-04-22T22:37:24.295549+0000 mon.c (mon.1) 358 : audit [DBG] from='client.? 172.21.15.115:0/3897483885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[46934]: audit 2022-04-22T22:37:23.518913+0000 mgr.y (mgr.34107) 10612 : audit [DBG] from='client.46347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[46934]: cluster 2022-04-22T22:37:23.774217+0000 mgr.y (mgr.34107) 10613 : cluster [DBG] pgmap v11279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:24 smithi115 bash[46934]: audit 2022-04-22T22:37:24.295549+0000 mon.c (mon.1) 358 : audit [DBG] from='client.? 172.21.15.115:0/3897483885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:25 smithi121 bash[36902]: audit 2022-04-22T22:37:23.869946+0000 mgr.y (mgr.34107) 10614 : audit [DBG] from='client.56287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:25 smithi121 bash[36902]: audit 2022-04-22T22:37:25.153219+0000 mon.a (mon.0) 14792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:26.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:25 smithi115 bash[46934]: audit 2022-04-22T22:37:23.869946+0000 mgr.y (mgr.34107) 10614 : audit [DBG] from='client.56287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:25 smithi115 bash[46934]: audit 2022-04-22T22:37:25.153219+0000 mon.a (mon.0) 14792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:25 smithi115 bash[45495]: audit 2022-04-22T22:37:23.869946+0000 mgr.y (mgr.34107) 10614 : audit [DBG] from='client.56287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:25 smithi115 bash[45495]: audit 2022-04-22T22:37:25.153219+0000 mon.a (mon.0) 14792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:26 smithi121 bash[36902]: cluster 2022-04-22T22:37:25.775303+0000 mgr.y (mgr.34107) 10615 : cluster [DBG] pgmap v11280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:27.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:26 smithi115 bash[46934]: cluster 2022-04-22T22:37:25.775303+0000 mgr.y (mgr.34107) 10615 : cluster [DBG] pgmap v11280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:26 smithi115 bash[45495]: cluster 2022-04-22T22:37:25.775303+0000 mgr.y (mgr.34107) 10615 : cluster [DBG] pgmap v11280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:27 smithi121 bash[36902]: audit 2022-04-22T22:37:26.780948+0000 mon.a (mon.0) 14793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:27 smithi115 bash[45495]: audit 2022-04-22T22:37:26.780948+0000 mon.a (mon.0) 14793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:27 smithi115 bash[46934]: audit 2022-04-22T22:37:26.780948+0000 mon.a (mon.0) 14793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:28 smithi115 bash[45495]: cluster 2022-04-22T22:37:27.776395+0000 mgr.y (mgr.34107) 10616 : cluster [DBG] pgmap v11281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:28 smithi115 bash[45495]: audit 2022-04-22T22:37:28.153721+0000 mon.a (mon.0) 14794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:28 smithi115 bash[46934]: cluster 2022-04-22T22:37:27.776395+0000 mgr.y (mgr.34107) 10616 : cluster [DBG] pgmap v11281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:28 smithi115 bash[46934]: audit 2022-04-22T22:37:28.153721+0000 mon.a (mon.0) 14794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:28 smithi121 bash[36902]: cluster 2022-04-22T22:37:27.776395+0000 mgr.y (mgr.34107) 10616 : cluster [DBG] pgmap v11281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:28 smithi121 bash[36902]: audit 2022-04-22T22:37:28.153721+0000 mon.a (mon.0) 14794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:29 smithi121 bash[36902]: cluster 2022-04-22T22:37:29.777446+0000 mgr.y (mgr.34107) 10617 : cluster [DBG] pgmap v11282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:29 smithi115 bash[45495]: cluster 2022-04-22T22:37:29.777446+0000 mgr.y (mgr.34107) 10617 : cluster [DBG] pgmap v11282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:29 smithi115 bash[46934]: cluster 2022-04-22T22:37:29.777446+0000 mgr.y (mgr.34107) 10617 : cluster [DBG] pgmap v11282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:30 smithi121 bash[36902]: audit 2022-04-22T22:37:29.964224+0000 mon.a (mon.0) 14795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:37:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:30 smithi121 bash[36902]: audit 2022-04-22T22:37:29.967237+0000 mon.a (mon.0) 14796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:37:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:30 smithi121 bash[36902]: audit 2022-04-22T22:37:29.969168+0000 mon.a (mon.0) 14797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:37:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:30 smithi121 bash[36902]: audit 2022-04-22T22:37:30.156355+0000 mon.a (mon.0) 14798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:30 smithi121 bash[36902]: audit 2022-04-22T22:37:30.333252+0000 mon.a (mon.0) 14799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[45495]: audit 2022-04-22T22:37:29.964224+0000 mon.a (mon.0) 14795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[45495]: audit 2022-04-22T22:37:29.967237+0000 mon.a (mon.0) 14796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[45495]: audit 2022-04-22T22:37:29.969168+0000 mon.a (mon.0) 14797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[45495]: audit 2022-04-22T22:37:30.156355+0000 mon.a (mon.0) 14798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[45495]: audit 2022-04-22T22:37:30.333252+0000 mon.a (mon.0) 14799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:37:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[46934]: audit 2022-04-22T22:37:29.964224+0000 mon.a (mon.0) 14795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:37:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[46934]: audit 2022-04-22T22:37:29.967237+0000 mon.a (mon.0) 14796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:37:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[46934]: audit 2022-04-22T22:37:29.969168+0000 mon.a (mon.0) 14797 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:37:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[46934]: audit 2022-04-22T22:37:30.156355+0000 mon.a (mon.0) 14798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:31.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:30 smithi115 bash[46934]: audit 2022-04-22T22:37:30.333252+0000 mon.a (mon.0) 14799 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:37:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:32 smithi121 bash[36902]: cluster 2022-04-22T22:37:31.778592+0000 mgr.y (mgr.34107) 10618 : cluster [DBG] pgmap v11283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:32 smithi121 bash[36902]: audit 2022-04-22T22:37:31.783993+0000 mon.a (mon.0) 14800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:33.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:32 smithi115 bash[45495]: cluster 2022-04-22T22:37:31.778592+0000 mgr.y (mgr.34107) 10618 : cluster [DBG] pgmap v11283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:32 smithi115 bash[45495]: audit 2022-04-22T22:37:31.783993+0000 mon.a (mon.0) 14800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:32.842Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:32 smithi115 bash[46934]: cluster 2022-04-22T22:37:31.778592+0000 mgr.y (mgr.34107) 10618 : cluster [DBG] pgmap v11283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:32 smithi115 bash[46934]: audit 2022-04-22T22:37:31.783993+0000 mon.a (mon.0) 14800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:33 smithi121 bash[36902]: audit 2022-04-22T22:37:33.157417+0000 mon.a (mon.0) 14801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:33 smithi115 bash[45495]: audit 2022-04-22T22:37:33.157417+0000 mon.a (mon.0) 14801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:33 smithi115 bash[46934]: audit 2022-04-22T22:37:33.157417+0000 mon.a (mon.0) 14801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:34 smithi121 bash[36902]: cluster 2022-04-22T22:37:33.779756+0000 mgr.y (mgr.34107) 10619 : cluster [DBG] pgmap v11284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:34 smithi115 bash[45495]: cluster 2022-04-22T22:37:33.779756+0000 mgr.y (mgr.34107) 10619 : cluster [DBG] pgmap v11284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:34.698Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:34 smithi115 bash[46934]: cluster 2022-04-22T22:37:33.779756+0000 mgr.y (mgr.34107) 10619 : cluster [DBG] pgmap v11284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:35 smithi121 bash[36902]: audit 2022-04-22T22:37:35.159199+0000 mon.a (mon.0) 14802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:35 smithi115 bash[45495]: audit 2022-04-22T22:37:35.159199+0000 mon.a (mon.0) 14802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:35 smithi115 bash[46934]: audit 2022-04-22T22:37:35.159199+0000 mon.a (mon.0) 14802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:36 smithi115 bash[45495]: cluster 2022-04-22T22:37:35.780794+0000 mgr.y (mgr.34107) 10620 : cluster [DBG] pgmap v11285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:36 smithi115 bash[46934]: cluster 2022-04-22T22:37:35.780794+0000 mgr.y (mgr.34107) 10620 : cluster [DBG] pgmap v11285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:36 smithi121 bash[36902]: cluster 2022-04-22T22:37:35.780794+0000 mgr.y (mgr.34107) 10620 : cluster [DBG] pgmap v11285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:37 smithi115 bash[45495]: audit 2022-04-22T22:37:36.786870+0000 mon.a (mon.0) 14803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:37 smithi115 bash[46934]: audit 2022-04-22T22:37:36.786870+0000 mon.a (mon.0) 14803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:37 smithi121 bash[36902]: audit 2022-04-22T22:37:36.786870+0000 mon.a (mon.0) 14803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:38 smithi115 bash[45495]: cluster 2022-04-22T22:37:37.781877+0000 mgr.y (mgr.34107) 10621 : cluster [DBG] pgmap v11286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:38 smithi115 bash[45495]: audit 2022-04-22T22:37:38.161106+0000 mon.a (mon.0) 14804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:38 smithi115 bash[46934]: cluster 2022-04-22T22:37:37.781877+0000 mgr.y (mgr.34107) 10621 : cluster [DBG] pgmap v11286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:38 smithi115 bash[46934]: audit 2022-04-22T22:37:38.161106+0000 mon.a (mon.0) 14804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:38 smithi121 bash[36902]: cluster 2022-04-22T22:37:37.781877+0000 mgr.y (mgr.34107) 10621 : cluster [DBG] pgmap v11286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:38 smithi121 bash[36902]: audit 2022-04-22T22:37:38.161106+0000 mon.a (mon.0) 14804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:39 smithi121 bash[36902]: cluster 2022-04-22T22:37:39.782974+0000 mgr.y (mgr.34107) 10622 : cluster [DBG] pgmap v11287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:39 smithi115 bash[45495]: cluster 2022-04-22T22:37:39.782974+0000 mgr.y (mgr.34107) 10622 : cluster [DBG] pgmap v11287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:39 smithi115 bash[46934]: cluster 2022-04-22T22:37:39.782974+0000 mgr.y (mgr.34107) 10622 : cluster [DBG] pgmap v11287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:40 smithi121 bash[36902]: audit 2022-04-22T22:37:40.162474+0000 mon.a (mon.0) 14805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:41.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:40 smithi115 bash[46934]: audit 2022-04-22T22:37:40.162474+0000 mon.a (mon.0) 14805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:40 smithi115 bash[45495]: audit 2022-04-22T22:37:40.162474+0000 mon.a (mon.0) 14805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:42 smithi121 bash[36902]: cluster 2022-04-22T22:37:41.784098+0000 mgr.y (mgr.34107) 10623 : cluster [DBG] pgmap v11288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:42 smithi121 bash[36902]: audit 2022-04-22T22:37:41.789848+0000 mon.a (mon.0) 14806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:42 smithi115 bash[45495]: cluster 2022-04-22T22:37:41.784098+0000 mgr.y (mgr.34107) 10623 : cluster [DBG] pgmap v11288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:43.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:42 smithi115 bash[45495]: audit 2022-04-22T22:37:41.789848+0000 mon.a (mon.0) 14806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:43.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:42.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:43.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:42 smithi115 bash[46934]: cluster 2022-04-22T22:37:41.784098+0000 mgr.y (mgr.34107) 10623 : cluster [DBG] pgmap v11288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:43.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:42 smithi115 bash[46934]: audit 2022-04-22T22:37:41.789848+0000 mon.a (mon.0) 14806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:44.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:43 smithi121 bash[36902]: audit 2022-04-22T22:37:43.164710+0000 mon.a (mon.0) 14807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:44.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:43 smithi115 bash[45495]: audit 2022-04-22T22:37:43.164710+0000 mon.a (mon.0) 14807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:43 smithi115 bash[46934]: audit 2022-04-22T22:37:43.164710+0000 mon.a (mon.0) 14807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:44 smithi121 bash[36902]: cluster 2022-04-22T22:37:43.785282+0000 mgr.y (mgr.34107) 10624 : cluster [DBG] pgmap v11289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:44 smithi115 bash[45495]: cluster 2022-04-22T22:37:43.785282+0000 mgr.y (mgr.34107) 10624 : cluster [DBG] pgmap v11289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:44.697Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:44 smithi115 bash[46934]: cluster 2022-04-22T22:37:43.785282+0000 mgr.y (mgr.34107) 10624 : cluster [DBG] pgmap v11289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:45 smithi121 bash[36902]: audit 2022-04-22T22:37:45.165612+0000 mon.a (mon.0) 14808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:45 smithi115 bash[45495]: audit 2022-04-22T22:37:45.165612+0000 mon.a (mon.0) 14808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:45 smithi115 bash[46934]: audit 2022-04-22T22:37:45.165612+0000 mon.a (mon.0) 14808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:46 smithi121 bash[36902]: cluster 2022-04-22T22:37:45.786378+0000 mgr.y (mgr.34107) 10625 : cluster [DBG] pgmap v11290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:46 smithi115 bash[45495]: cluster 2022-04-22T22:37:45.786378+0000 mgr.y (mgr.34107) 10625 : cluster [DBG] pgmap v11290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:46 smithi115 bash[46934]: cluster 2022-04-22T22:37:45.786378+0000 mgr.y (mgr.34107) 10625 : cluster [DBG] pgmap v11290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:47 smithi121 bash[36902]: audit 2022-04-22T22:37:46.792908+0000 mon.a (mon.0) 14809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:47 smithi115 bash[45495]: audit 2022-04-22T22:37:46.792908+0000 mon.a (mon.0) 14809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:47 smithi115 bash[46934]: audit 2022-04-22T22:37:46.792908+0000 mon.a (mon.0) 14809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:48 smithi115 bash[45495]: cluster 2022-04-22T22:37:47.787585+0000 mgr.y (mgr.34107) 10626 : cluster [DBG] pgmap v11291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:48 smithi115 bash[45495]: audit 2022-04-22T22:37:48.168460+0000 mon.a (mon.0) 14810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:48 smithi115 bash[46934]: cluster 2022-04-22T22:37:47.787585+0000 mgr.y (mgr.34107) 10626 : cluster [DBG] pgmap v11291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:48 smithi115 bash[46934]: audit 2022-04-22T22:37:48.168460+0000 mon.a (mon.0) 14810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:48 smithi121 bash[36902]: cluster 2022-04-22T22:37:47.787585+0000 mgr.y (mgr.34107) 10626 : cluster [DBG] pgmap v11291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:48 smithi121 bash[36902]: audit 2022-04-22T22:37:48.168460+0000 mon.a (mon.0) 14810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:49 smithi121 bash[36902]: cluster 2022-04-22T22:37:49.788663+0000 mgr.y (mgr.34107) 10627 : cluster [DBG] pgmap v11292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:49 smithi115 bash[45495]: cluster 2022-04-22T22:37:49.788663+0000 mgr.y (mgr.34107) 10627 : cluster [DBG] pgmap v11292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:49 smithi115 bash[46934]: cluster 2022-04-22T22:37:49.788663+0000 mgr.y (mgr.34107) 10627 : cluster [DBG] pgmap v11292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:50 smithi121 bash[36902]: audit 2022-04-22T22:37:50.169032+0000 mon.a (mon.0) 14811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:50 smithi115 bash[45495]: audit 2022-04-22T22:37:50.169032+0000 mon.a (mon.0) 14811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:50 smithi115 bash[46934]: audit 2022-04-22T22:37:50.169032+0000 mon.a (mon.0) 14811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:53.022 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:52 smithi121 bash[36902]: cluster 2022-04-22T22:37:51.789721+0000 mgr.y (mgr.34107) 10628 : cluster [DBG] pgmap v11293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:53.023 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:52 smithi121 bash[36902]: audit 2022-04-22T22:37:51.796102+0000 mon.a (mon.0) 14812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:52.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:52 smithi115 bash[45495]: cluster 2022-04-22T22:37:51.789721+0000 mgr.y (mgr.34107) 10628 : cluster [DBG] pgmap v11293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:52 smithi115 bash[45495]: audit 2022-04-22T22:37:51.796102+0000 mon.a (mon.0) 14812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:52 smithi115 bash[46934]: cluster 2022-04-22T22:37:51.789721+0000 mgr.y (mgr.34107) 10628 : cluster [DBG] pgmap v11293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:53.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:52 smithi115 bash[46934]: audit 2022-04-22T22:37:51.796102+0000 mon.a (mon.0) 14812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:53 smithi121 bash[36902]: audit 2022-04-22T22:37:53.171960+0000 mon.a (mon.0) 14813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:53 smithi115 bash[45495]: audit 2022-04-22T22:37:53.171960+0000 mon.a (mon.0) 14813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:53 smithi115 bash[46934]: audit 2022-04-22T22:37:53.171960+0000 mon.a (mon.0) 14813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:54.676 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:37:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:54 smithi121 bash[36902]: cluster 2022-04-22T22:37:53.790928+0000 mgr.y (mgr.34107) 10629 : cluster [DBG] pgmap v11294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:55.020 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 85s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 85s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 85s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 85s ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:37:55.021 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 85s ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 85s ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 85s ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 85s ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 85s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 85s ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:37:55.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:37:55.023 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:37:55.023 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:37:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:54 smithi115 bash[45495]: cluster 2022-04-22T22:37:53.790928+0000 mgr.y (mgr.34107) 10629 : cluster [DBG] pgmap v11294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:37:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:37:54.697Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:37:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:54 smithi115 bash[46934]: cluster 2022-04-22T22:37:53.790928+0000 mgr.y (mgr.34107) 10629 : cluster [DBG] pgmap v11294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:55.430 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:37:55.430 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:37:55.430 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:37:55.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:37:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:37:55.432 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:37:55.432 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:37:55.432 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:37:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:55 smithi121 bash[36902]: audit 2022-04-22T22:37:54.657508+0000 mgr.y (mgr.34107) 10630 : audit [DBG] from='client.56299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:55 smithi121 bash[36902]: audit 2022-04-22T22:37:55.171859+0000 mon.a (mon.0) 14814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:56.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:55 smithi121 bash[36902]: audit 2022-04-22T22:37:55.426704+0000 mon.c (mon.1) 359 : audit [DBG] from='client.? 172.21.15.115:0/1359785094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[45495]: audit 2022-04-22T22:37:54.657508+0000 mgr.y (mgr.34107) 10630 : audit [DBG] from='client.56299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[45495]: audit 2022-04-22T22:37:55.171859+0000 mon.a (mon.0) 14814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[45495]: audit 2022-04-22T22:37:55.426704+0000 mon.c (mon.1) 359 : audit [DBG] from='client.? 172.21.15.115:0/1359785094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[46934]: audit 2022-04-22T22:37:54.657508+0000 mgr.y (mgr.34107) 10630 : audit [DBG] from='client.56299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[46934]: audit 2022-04-22T22:37:55.171859+0000 mon.a (mon.0) 14814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:37:56.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:55 smithi115 bash[46934]: audit 2022-04-22T22:37:55.426704+0000 mon.c (mon.1) 359 : audit [DBG] from='client.? 172.21.15.115:0/1359785094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:37:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:56 smithi121 bash[36902]: audit 2022-04-22T22:37:54.998633+0000 mgr.y (mgr.34107) 10631 : audit [DBG] from='client.46371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:56 smithi121 bash[36902]: cluster 2022-04-22T22:37:55.792057+0000 mgr.y (mgr.34107) 10632 : cluster [DBG] pgmap v11295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:56 smithi115 bash[45495]: audit 2022-04-22T22:37:54.998633+0000 mgr.y (mgr.34107) 10631 : audit [DBG] from='client.46371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:56 smithi115 bash[45495]: cluster 2022-04-22T22:37:55.792057+0000 mgr.y (mgr.34107) 10632 : cluster [DBG] pgmap v11295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:56 smithi115 bash[46934]: audit 2022-04-22T22:37:54.998633+0000 mgr.y (mgr.34107) 10631 : audit [DBG] from='client.46371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:37:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:56 smithi115 bash[46934]: cluster 2022-04-22T22:37:55.792057+0000 mgr.y (mgr.34107) 10632 : cluster [DBG] pgmap v11295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:57 smithi121 bash[36902]: audit 2022-04-22T22:37:56.799078+0000 mon.a (mon.0) 14815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:57 smithi115 bash[45495]: audit 2022-04-22T22:37:56.799078+0000 mon.a (mon.0) 14815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:57 smithi115 bash[46934]: audit 2022-04-22T22:37:56.799078+0000 mon.a (mon.0) 14815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:37:59.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:58 smithi115 bash[45495]: cluster 2022-04-22T22:37:57.793440+0000 mgr.y (mgr.34107) 10633 : cluster [DBG] pgmap v11296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:59.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:58 smithi115 bash[45495]: audit 2022-04-22T22:37:58.175757+0000 mon.a (mon.0) 14816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:58 smithi115 bash[46934]: cluster 2022-04-22T22:37:57.793440+0000 mgr.y (mgr.34107) 10633 : cluster [DBG] pgmap v11296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:59.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:58 smithi115 bash[46934]: audit 2022-04-22T22:37:58.175757+0000 mon.a (mon.0) 14816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:37:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:58 smithi121 bash[36902]: cluster 2022-04-22T22:37:57.793440+0000 mgr.y (mgr.34107) 10633 : cluster [DBG] pgmap v11296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:37:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:58 smithi121 bash[36902]: audit 2022-04-22T22:37:58.175757+0000 mon.a (mon.0) 14816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:37:59 smithi121 bash[36902]: cluster 2022-04-22T22:37:59.794578+0000 mgr.y (mgr.34107) 10634 : cluster [DBG] pgmap v11297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:37:59 smithi115 bash[45495]: cluster 2022-04-22T22:37:59.794578+0000 mgr.y (mgr.34107) 10634 : cluster [DBG] pgmap v11297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:37:59 smithi115 bash[46934]: cluster 2022-04-22T22:37:59.794578+0000 mgr.y (mgr.34107) 10634 : cluster [DBG] pgmap v11297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:00 smithi121 bash[36902]: audit 2022-04-22T22:38:00.174571+0000 mon.a (mon.0) 14817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:00 smithi115 bash[45495]: audit 2022-04-22T22:38:00.174571+0000 mon.a (mon.0) 14817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:00 smithi115 bash[46934]: audit 2022-04-22T22:38:00.174571+0000 mon.a (mon.0) 14817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:02 smithi121 bash[36902]: cluster 2022-04-22T22:38:01.795696+0000 mgr.y (mgr.34107) 10635 : cluster [DBG] pgmap v11298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:03.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:02 smithi121 bash[36902]: audit 2022-04-22T22:38:01.802068+0000 mon.a (mon.0) 14818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:03.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:02.844Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:02 smithi115 bash[45495]: cluster 2022-04-22T22:38:01.795696+0000 mgr.y (mgr.34107) 10635 : cluster [DBG] pgmap v11298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:03.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:02 smithi115 bash[45495]: audit 2022-04-22T22:38:01.802068+0000 mon.a (mon.0) 14818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:02 smithi115 bash[46934]: cluster 2022-04-22T22:38:01.795696+0000 mgr.y (mgr.34107) 10635 : cluster [DBG] pgmap v11298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:03.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:02 smithi115 bash[46934]: audit 2022-04-22T22:38:01.802068+0000 mon.a (mon.0) 14818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:03 smithi121 bash[36902]: audit 2022-04-22T22:38:03.179692+0000 mon.a (mon.0) 14819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:03 smithi115 bash[45495]: audit 2022-04-22T22:38:03.179692+0000 mon.a (mon.0) 14819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:03 smithi115 bash[46934]: audit 2022-04-22T22:38:03.179692+0000 mon.a (mon.0) 14819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:04 smithi121 bash[36902]: cluster 2022-04-22T22:38:03.796762+0000 mgr.y (mgr.34107) 10636 : cluster [DBG] pgmap v11299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:05.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:04 smithi115 bash[46934]: cluster 2022-04-22T22:38:03.796762+0000 mgr.y (mgr.34107) 10636 : cluster [DBG] pgmap v11299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:04.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:04 smithi115 bash[45495]: cluster 2022-04-22T22:38:03.796762+0000 mgr.y (mgr.34107) 10636 : cluster [DBG] pgmap v11299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:05 smithi121 bash[36902]: audit 2022-04-22T22:38:05.177791+0000 mon.a (mon.0) 14820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:05 smithi115 bash[45495]: audit 2022-04-22T22:38:05.177791+0000 mon.a (mon.0) 14820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:05 smithi115 bash[46934]: audit 2022-04-22T22:38:05.177791+0000 mon.a (mon.0) 14820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:06 smithi115 bash[45495]: cluster 2022-04-22T22:38:05.797850+0000 mgr.y (mgr.34107) 10637 : cluster [DBG] pgmap v11300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:06 smithi115 bash[46934]: cluster 2022-04-22T22:38:05.797850+0000 mgr.y (mgr.34107) 10637 : cluster [DBG] pgmap v11300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:06 smithi121 bash[36902]: cluster 2022-04-22T22:38:05.797850+0000 mgr.y (mgr.34107) 10637 : cluster [DBG] pgmap v11300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:07 smithi115 bash[45495]: audit 2022-04-22T22:38:06.805036+0000 mon.a (mon.0) 14821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:07 smithi115 bash[46934]: audit 2022-04-22T22:38:06.805036+0000 mon.a (mon.0) 14821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:08.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:07 smithi121 bash[36902]: audit 2022-04-22T22:38:06.805036+0000 mon.a (mon.0) 14821 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:09.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:08 smithi115 bash[46934]: cluster 2022-04-22T22:38:07.798957+0000 mgr.y (mgr.34107) 10638 : cluster [DBG] pgmap v11301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:08 smithi115 bash[46934]: audit 2022-04-22T22:38:08.183538+0000 mon.a (mon.0) 14822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:08 smithi115 bash[45495]: cluster 2022-04-22T22:38:07.798957+0000 mgr.y (mgr.34107) 10638 : cluster [DBG] pgmap v11301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:08 smithi115 bash[45495]: audit 2022-04-22T22:38:08.183538+0000 mon.a (mon.0) 14822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:08 smithi121 bash[36902]: cluster 2022-04-22T22:38:07.798957+0000 mgr.y (mgr.34107) 10638 : cluster [DBG] pgmap v11301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:08 smithi121 bash[36902]: audit 2022-04-22T22:38:08.183538+0000 mon.a (mon.0) 14822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:09 smithi121 bash[36902]: cluster 2022-04-22T22:38:09.800065+0000 mgr.y (mgr.34107) 10639 : cluster [DBG] pgmap v11302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:09 smithi115 bash[45495]: cluster 2022-04-22T22:38:09.800065+0000 mgr.y (mgr.34107) 10639 : cluster [DBG] pgmap v11302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:09 smithi115 bash[46934]: cluster 2022-04-22T22:38:09.800065+0000 mgr.y (mgr.34107) 10639 : cluster [DBG] pgmap v11302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:10 smithi121 bash[36902]: audit 2022-04-22T22:38:10.180943+0000 mon.a (mon.0) 14823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:10 smithi115 bash[45495]: audit 2022-04-22T22:38:10.180943+0000 mon.a (mon.0) 14823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:10 smithi115 bash[46934]: audit 2022-04-22T22:38:10.180943+0000 mon.a (mon.0) 14823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:13.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:12 smithi121 bash[36902]: audit 2022-04-22T22:38:11.808142+0000 mon.a (mon.0) 14824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:13.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:12 smithi115 bash[45495]: audit 2022-04-22T22:38:11.808142+0000 mon.a (mon.0) 14824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:12 smithi115 bash[46934]: audit 2022-04-22T22:38:11.808142+0000 mon.a (mon.0) 14824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:13.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:12.843Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:13 smithi121 bash[36902]: cluster 2022-04-22T22:38:11.801239+0000 mgr.y (mgr.34107) 10640 : cluster [DBG] pgmap v11303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:14.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:13 smithi121 bash[36902]: audit 2022-04-22T22:38:13.187020+0000 mon.a (mon.0) 14825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:13 smithi115 bash[45495]: cluster 2022-04-22T22:38:11.801239+0000 mgr.y (mgr.34107) 10640 : cluster [DBG] pgmap v11303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:13 smithi115 bash[45495]: audit 2022-04-22T22:38:13.187020+0000 mon.a (mon.0) 14825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:13 smithi115 bash[46934]: cluster 2022-04-22T22:38:11.801239+0000 mgr.y (mgr.34107) 10640 : cluster [DBG] pgmap v11303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:13 smithi115 bash[46934]: audit 2022-04-22T22:38:13.187020+0000 mon.a (mon.0) 14825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:14.699Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:15 smithi121 bash[36902]: cluster 2022-04-22T22:38:13.802328+0000 mgr.y (mgr.34107) 10641 : cluster [DBG] pgmap v11304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:15 smithi121 bash[36902]: audit 2022-04-22T22:38:15.183999+0000 mon.a (mon.0) 14826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:15 smithi115 bash[45495]: cluster 2022-04-22T22:38:13.802328+0000 mgr.y (mgr.34107) 10641 : cluster [DBG] pgmap v11304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:15 smithi115 bash[45495]: audit 2022-04-22T22:38:15.183999+0000 mon.a (mon.0) 14826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:15 smithi115 bash[46934]: cluster 2022-04-22T22:38:13.802328+0000 mgr.y (mgr.34107) 10641 : cluster [DBG] pgmap v11304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:15 smithi115 bash[46934]: audit 2022-04-22T22:38:15.183999+0000 mon.a (mon.0) 14826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:17 smithi121 bash[36902]: cluster 2022-04-22T22:38:15.803444+0000 mgr.y (mgr.34107) 10642 : cluster [DBG] pgmap v11305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:17 smithi121 bash[36902]: audit 2022-04-22T22:38:16.810996+0000 mon.a (mon.0) 14827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:17 smithi121 bash[36902]: audit 2022-04-22T22:38:16.869785+0000 mon.a (mon.0) 14828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:38:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:17 smithi121 bash[36902]: audit 2022-04-22T22:38:16.870236+0000 mon.a (mon.0) 14829 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:38:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[45495]: cluster 2022-04-22T22:38:15.803444+0000 mgr.y (mgr.34107) 10642 : cluster [DBG] pgmap v11305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[45495]: audit 2022-04-22T22:38:16.810996+0000 mon.a (mon.0) 14827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[45495]: audit 2022-04-22T22:38:16.869785+0000 mon.a (mon.0) 14828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[45495]: audit 2022-04-22T22:38:16.870236+0000 mon.a (mon.0) 14829 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[46934]: cluster 2022-04-22T22:38:15.803444+0000 mgr.y (mgr.34107) 10642 : cluster [DBG] pgmap v11305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[46934]: audit 2022-04-22T22:38:16.810996+0000 mon.a (mon.0) 14827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[46934]: audit 2022-04-22T22:38:16.869785+0000 mon.a (mon.0) 14828 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:38:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:17 smithi115 bash[46934]: audit 2022-04-22T22:38:16.870236+0000 mon.a (mon.0) 14829 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:38:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:18 smithi115 bash[45495]: audit 2022-04-22T22:38:18.190689+0000 mon.a (mon.0) 14830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:18 smithi115 bash[46934]: audit 2022-04-22T22:38:18.190689+0000 mon.a (mon.0) 14830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:18 smithi121 bash[36902]: audit 2022-04-22T22:38:18.190689+0000 mon.a (mon.0) 14830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:19 smithi115 bash[45495]: cluster 2022-04-22T22:38:17.804512+0000 mgr.y (mgr.34107) 10643 : cluster [DBG] pgmap v11306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:19 smithi115 bash[46934]: cluster 2022-04-22T22:38:17.804512+0000 mgr.y (mgr.34107) 10643 : cluster [DBG] pgmap v11306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:19 smithi121 bash[36902]: cluster 2022-04-22T22:38:17.804512+0000 mgr.y (mgr.34107) 10643 : cluster [DBG] pgmap v11306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:21.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:20 smithi115 bash[45495]: audit 2022-04-22T22:38:20.186899+0000 mon.a (mon.0) 14831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:21.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:20 smithi115 bash[46934]: audit 2022-04-22T22:38:20.186899+0000 mon.a (mon.0) 14831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:20 smithi121 bash[36902]: audit 2022-04-22T22:38:20.186899+0000 mon.a (mon.0) 14831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:21 smithi115 bash[45495]: cluster 2022-04-22T22:38:19.805591+0000 mgr.y (mgr.34107) 10644 : cluster [DBG] pgmap v11307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:21 smithi115 bash[46934]: cluster 2022-04-22T22:38:19.805591+0000 mgr.y (mgr.34107) 10644 : cluster [DBG] pgmap v11307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:21 smithi121 bash[36902]: cluster 2022-04-22T22:38:19.805591+0000 mgr.y (mgr.34107) 10644 : cluster [DBG] pgmap v11307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:22 smithi121 bash[36902]: audit 2022-04-22T22:38:21.813875+0000 mon.a (mon.0) 14832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:22 smithi115 bash[45495]: audit 2022-04-22T22:38:21.813875+0000 mon.a (mon.0) 14832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:22 smithi115 bash[46934]: audit 2022-04-22T22:38:21.813875+0000 mon.a (mon.0) 14832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:22.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:23 smithi121 bash[36902]: cluster 2022-04-22T22:38:21.806624+0000 mgr.y (mgr.34107) 10645 : cluster [DBG] pgmap v11308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:23 smithi121 bash[36902]: audit 2022-04-22T22:38:23.194599+0000 mon.a (mon.0) 14833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:23 smithi115 bash[45495]: cluster 2022-04-22T22:38:21.806624+0000 mgr.y (mgr.34107) 10645 : cluster [DBG] pgmap v11308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:23 smithi115 bash[45495]: audit 2022-04-22T22:38:23.194599+0000 mon.a (mon.0) 14833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:23 smithi115 bash[46934]: cluster 2022-04-22T22:38:21.806624+0000 mgr.y (mgr.34107) 10645 : cluster [DBG] pgmap v11308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:23 smithi115 bash[46934]: audit 2022-04-22T22:38:23.194599+0000 mon.a (mon.0) 14833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:24.700Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:25.798 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 116s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 116s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:38:26.152 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 116s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 116s ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 116s ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 116s ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 116s ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 116s ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 116s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 116s ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:38:26.153 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:38:26.154 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:38:26.154 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:38:26.154 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:38:26.154 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:38:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:25 smithi121 bash[36902]: cluster 2022-04-22T22:38:23.807366+0000 mgr.y (mgr.34107) 10646 : cluster [DBG] pgmap v11309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:25 smithi121 bash[36902]: audit 2022-04-22T22:38:25.190105+0000 mon.a (mon.0) 14834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:25 smithi115 bash[45495]: cluster 2022-04-22T22:38:23.807366+0000 mgr.y (mgr.34107) 10646 : cluster [DBG] pgmap v11309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:25 smithi115 bash[45495]: audit 2022-04-22T22:38:25.190105+0000 mon.a (mon.0) 14834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:25 smithi115 bash[46934]: cluster 2022-04-22T22:38:23.807366+0000 mgr.y (mgr.34107) 10646 : cluster [DBG] pgmap v11309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:25 smithi115 bash[46934]: audit 2022-04-22T22:38:25.190105+0000 mon.a (mon.0) 14834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:26.560 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:38:26.561 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:38:26.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:38:26.563 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:38:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:26 smithi121 bash[36902]: audit 2022-04-22T22:38:25.779863+0000 mgr.y (mgr.34107) 10647 : audit [DBG] from='client.46383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:26 smithi121 bash[36902]: audit 2022-04-22T22:38:26.557183+0000 mon.c (mon.1) 360 : audit [DBG] from='client.? 172.21.15.115:0/3203132147' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:26 smithi115 bash[45495]: audit 2022-04-22T22:38:25.779863+0000 mgr.y (mgr.34107) 10647 : audit [DBG] from='client.46383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:26 smithi115 bash[45495]: audit 2022-04-22T22:38:26.557183+0000 mon.c (mon.1) 360 : audit [DBG] from='client.? 172.21.15.115:0/3203132147' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:26 smithi115 bash[46934]: audit 2022-04-22T22:38:25.779863+0000 mgr.y (mgr.34107) 10647 : audit [DBG] from='client.46383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:26 smithi115 bash[46934]: audit 2022-04-22T22:38:26.557183+0000 mon.c (mon.1) 360 : audit [DBG] from='client.? 172.21.15.115:0/3203132147' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:27 smithi121 bash[36902]: cluster 2022-04-22T22:38:25.807931+0000 mgr.y (mgr.34107) 10648 : cluster [DBG] pgmap v11310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:27 smithi121 bash[36902]: audit 2022-04-22T22:38:26.128352+0000 mgr.y (mgr.34107) 10649 : audit [DBG] from='client.46389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:27 smithi121 bash[36902]: audit 2022-04-22T22:38:26.816520+0000 mon.a (mon.0) 14835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[45495]: cluster 2022-04-22T22:38:25.807931+0000 mgr.y (mgr.34107) 10648 : cluster [DBG] pgmap v11310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[45495]: audit 2022-04-22T22:38:26.128352+0000 mgr.y (mgr.34107) 10649 : audit [DBG] from='client.46389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[45495]: audit 2022-04-22T22:38:26.816520+0000 mon.a (mon.0) 14835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[46934]: cluster 2022-04-22T22:38:25.807931+0000 mgr.y (mgr.34107) 10648 : cluster [DBG] pgmap v11310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[46934]: audit 2022-04-22T22:38:26.128352+0000 mgr.y (mgr.34107) 10649 : audit [DBG] from='client.46389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:28.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:27 smithi115 bash[46934]: audit 2022-04-22T22:38:26.816520+0000 mon.a (mon.0) 14835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:28 smithi121 bash[36902]: cluster 2022-04-22T22:38:27.808948+0000 mgr.y (mgr.34107) 10650 : cluster [DBG] pgmap v11311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:28 smithi121 bash[36902]: audit 2022-04-22T22:38:28.198385+0000 mon.a (mon.0) 14836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:28 smithi115 bash[45495]: cluster 2022-04-22T22:38:27.808948+0000 mgr.y (mgr.34107) 10650 : cluster [DBG] pgmap v11311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:28 smithi115 bash[45495]: audit 2022-04-22T22:38:28.198385+0000 mon.a (mon.0) 14836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:28 smithi115 bash[46934]: cluster 2022-04-22T22:38:27.808948+0000 mgr.y (mgr.34107) 10650 : cluster [DBG] pgmap v11311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:28 smithi115 bash[46934]: audit 2022-04-22T22:38:28.198385+0000 mon.a (mon.0) 14836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: cluster 2022-04-22T22:38:29.809994+0000 mgr.y (mgr.34107) 10651 : cluster [DBG] pgmap v11312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: audit 2022-04-22T22:38:30.193457+0000 mon.a (mon.0) 14837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: audit 2022-04-22T22:38:30.342081+0000 mon.a (mon.0) 14838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:38:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: audit 2022-04-22T22:38:30.345207+0000 mon.a (mon.0) 14839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:38:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: audit 2022-04-22T22:38:30.347191+0000 mon.a (mon.0) 14840 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:38:31.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:30 smithi121 bash[36902]: audit 2022-04-22T22:38:30.699285+0000 mon.a (mon.0) 14841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:38:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: cluster 2022-04-22T22:38:29.809994+0000 mgr.y (mgr.34107) 10651 : cluster [DBG] pgmap v11312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: audit 2022-04-22T22:38:30.193457+0000 mon.a (mon.0) 14837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: audit 2022-04-22T22:38:30.342081+0000 mon.a (mon.0) 14838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:38:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: audit 2022-04-22T22:38:30.345207+0000 mon.a (mon.0) 14839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:38:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: audit 2022-04-22T22:38:30.347191+0000 mon.a (mon.0) 14840 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:38:31.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[45495]: audit 2022-04-22T22:38:30.699285+0000 mon.a (mon.0) 14841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:38:31.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: cluster 2022-04-22T22:38:29.809994+0000 mgr.y (mgr.34107) 10651 : cluster [DBG] pgmap v11312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: audit 2022-04-22T22:38:30.193457+0000 mon.a (mon.0) 14837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: audit 2022-04-22T22:38:30.342081+0000 mon.a (mon.0) 14838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:38:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: audit 2022-04-22T22:38:30.345207+0000 mon.a (mon.0) 14839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:38:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: audit 2022-04-22T22:38:30.347191+0000 mon.a (mon.0) 14840 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:38:31.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:30 smithi115 bash[46934]: audit 2022-04-22T22:38:30.699285+0000 mon.a (mon.0) 14841 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:38:33.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:32 smithi121 bash[36902]: audit 2022-04-22T22:38:31.819019+0000 mon.a (mon.0) 14842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:33.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:32 smithi115 bash[46934]: audit 2022-04-22T22:38:31.819019+0000 mon.a (mon.0) 14842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:33.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:32 smithi115 bash[45495]: audit 2022-04-22T22:38:31.819019+0000 mon.a (mon.0) 14842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:33.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:32.844Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:34.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:33 smithi121 bash[36902]: cluster 2022-04-22T22:38:31.811043+0000 mgr.y (mgr.34107) 10652 : cluster [DBG] pgmap v11313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:33 smithi121 bash[36902]: audit 2022-04-22T22:38:33.202329+0000 mon.a (mon.0) 14843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:33 smithi115 bash[45495]: cluster 2022-04-22T22:38:31.811043+0000 mgr.y (mgr.34107) 10652 : cluster [DBG] pgmap v11313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:33 smithi115 bash[45495]: audit 2022-04-22T22:38:33.202329+0000 mon.a (mon.0) 14843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:33 smithi115 bash[46934]: cluster 2022-04-22T22:38:31.811043+0000 mgr.y (mgr.34107) 10652 : cluster [DBG] pgmap v11313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:33 smithi115 bash[46934]: audit 2022-04-22T22:38:33.202329+0000 mon.a (mon.0) 14843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:34.700Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:35 smithi121 bash[36902]: cluster 2022-04-22T22:38:33.812163+0000 mgr.y (mgr.34107) 10653 : cluster [DBG] pgmap v11314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:35 smithi121 bash[36902]: audit 2022-04-22T22:38:35.196762+0000 mon.a (mon.0) 14844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:35 smithi115 bash[45495]: cluster 2022-04-22T22:38:33.812163+0000 mgr.y (mgr.34107) 10653 : cluster [DBG] pgmap v11314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:35 smithi115 bash[45495]: audit 2022-04-22T22:38:35.196762+0000 mon.a (mon.0) 14844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:35 smithi115 bash[46934]: cluster 2022-04-22T22:38:33.812163+0000 mgr.y (mgr.34107) 10653 : cluster [DBG] pgmap v11314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:35 smithi115 bash[46934]: audit 2022-04-22T22:38:35.196762+0000 mon.a (mon.0) 14844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:37 smithi115 bash[45495]: cluster 2022-04-22T22:38:35.813236+0000 mgr.y (mgr.34107) 10654 : cluster [DBG] pgmap v11315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:37 smithi115 bash[45495]: audit 2022-04-22T22:38:36.821714+0000 mon.a (mon.0) 14845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:37 smithi115 bash[46934]: cluster 2022-04-22T22:38:35.813236+0000 mgr.y (mgr.34107) 10654 : cluster [DBG] pgmap v11315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:37 smithi115 bash[46934]: audit 2022-04-22T22:38:36.821714+0000 mon.a (mon.0) 14845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:37 smithi121 bash[36902]: cluster 2022-04-22T22:38:35.813236+0000 mgr.y (mgr.34107) 10654 : cluster [DBG] pgmap v11315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:37 smithi121 bash[36902]: audit 2022-04-22T22:38:36.821714+0000 mon.a (mon.0) 14845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:38 smithi115 bash[45495]: audit 2022-04-22T22:38:38.206360+0000 mon.a (mon.0) 14846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:38 smithi115 bash[46934]: audit 2022-04-22T22:38:38.206360+0000 mon.a (mon.0) 14846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:38 smithi121 bash[36902]: audit 2022-04-22T22:38:38.206360+0000 mon.a (mon.0) 14846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:40.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:39 smithi115 bash[45495]: cluster 2022-04-22T22:38:37.814329+0000 mgr.y (mgr.34107) 10655 : cluster [DBG] pgmap v11316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:39 smithi115 bash[46934]: cluster 2022-04-22T22:38:37.814329+0000 mgr.y (mgr.34107) 10655 : cluster [DBG] pgmap v11316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:39 smithi121 bash[36902]: cluster 2022-04-22T22:38:37.814329+0000 mgr.y (mgr.34107) 10655 : cluster [DBG] pgmap v11316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:41.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:40 smithi115 bash[45495]: audit 2022-04-22T22:38:40.199878+0000 mon.a (mon.0) 14847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:41.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:40 smithi115 bash[46934]: audit 2022-04-22T22:38:40.199878+0000 mon.a (mon.0) 14847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:40 smithi121 bash[36902]: audit 2022-04-22T22:38:40.199878+0000 mon.a (mon.0) 14847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:41 smithi115 bash[45495]: cluster 2022-04-22T22:38:39.815422+0000 mgr.y (mgr.34107) 10656 : cluster [DBG] pgmap v11317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:41 smithi115 bash[46934]: cluster 2022-04-22T22:38:39.815422+0000 mgr.y (mgr.34107) 10656 : cluster [DBG] pgmap v11317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:41 smithi121 bash[36902]: cluster 2022-04-22T22:38:39.815422+0000 mgr.y (mgr.34107) 10656 : cluster [DBG] pgmap v11317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:42 smithi121 bash[36902]: audit 2022-04-22T22:38:41.824761+0000 mon.a (mon.0) 14848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:42 smithi115 bash[45495]: audit 2022-04-22T22:38:41.824761+0000 mon.a (mon.0) 14848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:43.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:42.844Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:42 smithi115 bash[46934]: audit 2022-04-22T22:38:41.824761+0000 mon.a (mon.0) 14848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:43 smithi121 bash[36902]: cluster 2022-04-22T22:38:41.816437+0000 mgr.y (mgr.34107) 10657 : cluster [DBG] pgmap v11318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:43 smithi121 bash[36902]: audit 2022-04-22T22:38:43.209927+0000 mon.a (mon.0) 14849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:43 smithi115 bash[45495]: cluster 2022-04-22T22:38:41.816437+0000 mgr.y (mgr.34107) 10657 : cluster [DBG] pgmap v11318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:43 smithi115 bash[45495]: audit 2022-04-22T22:38:43.209927+0000 mon.a (mon.0) 14849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:43 smithi115 bash[46934]: cluster 2022-04-22T22:38:41.816437+0000 mgr.y (mgr.34107) 10657 : cluster [DBG] pgmap v11318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:43 smithi115 bash[46934]: audit 2022-04-22T22:38:43.209927+0000 mon.a (mon.0) 14849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:44.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:45 smithi121 bash[36902]: cluster 2022-04-22T22:38:43.817564+0000 mgr.y (mgr.34107) 10658 : cluster [DBG] pgmap v11319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:45 smithi121 bash[36902]: audit 2022-04-22T22:38:45.202884+0000 mon.a (mon.0) 14850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:45 smithi115 bash[45495]: cluster 2022-04-22T22:38:43.817564+0000 mgr.y (mgr.34107) 10658 : cluster [DBG] pgmap v11319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:45 smithi115 bash[45495]: audit 2022-04-22T22:38:45.202884+0000 mon.a (mon.0) 14850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:45 smithi115 bash[46934]: cluster 2022-04-22T22:38:43.817564+0000 mgr.y (mgr.34107) 10658 : cluster [DBG] pgmap v11319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:45 smithi115 bash[46934]: audit 2022-04-22T22:38:45.202884+0000 mon.a (mon.0) 14850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:47 smithi121 bash[36902]: cluster 2022-04-22T22:38:45.818577+0000 mgr.y (mgr.34107) 10659 : cluster [DBG] pgmap v11320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:47 smithi121 bash[36902]: audit 2022-04-22T22:38:46.827988+0000 mon.a (mon.0) 14851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:47 smithi115 bash[45495]: cluster 2022-04-22T22:38:45.818577+0000 mgr.y (mgr.34107) 10659 : cluster [DBG] pgmap v11320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:47 smithi115 bash[45495]: audit 2022-04-22T22:38:46.827988+0000 mon.a (mon.0) 14851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:47 smithi115 bash[46934]: cluster 2022-04-22T22:38:45.818577+0000 mgr.y (mgr.34107) 10659 : cluster [DBG] pgmap v11320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:47 smithi115 bash[46934]: audit 2022-04-22T22:38:46.827988+0000 mon.a (mon.0) 14851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:48 smithi121 bash[36902]: audit 2022-04-22T22:38:48.213524+0000 mon.a (mon.0) 14852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:48 smithi115 bash[45495]: audit 2022-04-22T22:38:48.213524+0000 mon.a (mon.0) 14852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:48 smithi115 bash[46934]: audit 2022-04-22T22:38:48.213524+0000 mon.a (mon.0) 14852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:49 smithi121 bash[36902]: cluster 2022-04-22T22:38:47.819506+0000 mgr.y (mgr.34107) 10660 : cluster [DBG] pgmap v11321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:49 smithi115 bash[45495]: cluster 2022-04-22T22:38:47.819506+0000 mgr.y (mgr.34107) 10660 : cluster [DBG] pgmap v11321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:49 smithi115 bash[46934]: cluster 2022-04-22T22:38:47.819506+0000 mgr.y (mgr.34107) 10660 : cluster [DBG] pgmap v11321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:50 smithi121 bash[36902]: cluster 2022-04-22T22:38:49.820489+0000 mgr.y (mgr.34107) 10661 : cluster [DBG] pgmap v11322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:50 smithi121 bash[36902]: audit 2022-04-22T22:38:50.205913+0000 mon.a (mon.0) 14853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:50 smithi115 bash[45495]: cluster 2022-04-22T22:38:49.820489+0000 mgr.y (mgr.34107) 10661 : cluster [DBG] pgmap v11322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:50 smithi115 bash[45495]: audit 2022-04-22T22:38:50.205913+0000 mon.a (mon.0) 14853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:50 smithi115 bash[46934]: cluster 2022-04-22T22:38:49.820489+0000 mgr.y (mgr.34107) 10661 : cluster [DBG] pgmap v11322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:50 smithi115 bash[46934]: audit 2022-04-22T22:38:50.205913+0000 mon.a (mon.0) 14853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:52 smithi115 bash[45495]: audit 2022-04-22T22:38:51.831057+0000 mon.a (mon.0) 14854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:53.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:52.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:52 smithi115 bash[46934]: audit 2022-04-22T22:38:51.831057+0000 mon.a (mon.0) 14854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:52 smithi121 bash[36902]: audit 2022-04-22T22:38:51.831057+0000 mon.a (mon.0) 14854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:53 smithi115 bash[45495]: cluster 2022-04-22T22:38:51.821529+0000 mgr.y (mgr.34107) 10662 : cluster [DBG] pgmap v11323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:53 smithi115 bash[45495]: audit 2022-04-22T22:38:53.217339+0000 mon.a (mon.0) 14855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:53 smithi115 bash[46934]: cluster 2022-04-22T22:38:51.821529+0000 mgr.y (mgr.34107) 10662 : cluster [DBG] pgmap v11323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:53 smithi115 bash[46934]: audit 2022-04-22T22:38:53.217339+0000 mon.a (mon.0) 14855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:53 smithi121 bash[36902]: cluster 2022-04-22T22:38:51.821529+0000 mgr.y (mgr.34107) 10662 : cluster [DBG] pgmap v11323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:53 smithi121 bash[36902]: audit 2022-04-22T22:38:53.217339+0000 mon.a (mon.0) 14855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:38:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:38:54.702Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:38:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:55 smithi115 bash[45495]: cluster 2022-04-22T22:38:53.822675+0000 mgr.y (mgr.34107) 10663 : cluster [DBG] pgmap v11324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:55 smithi115 bash[45495]: audit 2022-04-22T22:38:55.209174+0000 mon.a (mon.0) 14856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:55 smithi115 bash[46934]: cluster 2022-04-22T22:38:53.822675+0000 mgr.y (mgr.34107) 10663 : cluster [DBG] pgmap v11324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:55 smithi115 bash[46934]: audit 2022-04-22T22:38:55.209174+0000 mon.a (mon.0) 14856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:55 smithi121 bash[36902]: cluster 2022-04-22T22:38:53.822675+0000 mgr.y (mgr.34107) 10663 : cluster [DBG] pgmap v11324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:55 smithi121 bash[36902]: audit 2022-04-22T22:38:55.209174+0000 mon.a (mon.0) 14856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:38:56.940 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:38:57.290 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:38:57.290 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:38:57.290 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:38:57.290 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:38:57.291 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:38:57.292 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:38:57.293 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:38:57.698 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:38:57.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:38:57.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:38:57.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:57.699 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:38:57.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:38:57.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:38:57.700 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:38:57.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:38:57.701 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:38:57.701 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:38:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[46934]: cluster 2022-04-22T22:38:55.823784+0000 mgr.y (mgr.34107) 10664 : cluster [DBG] pgmap v11325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[46934]: audit 2022-04-22T22:38:56.833713+0000 mon.a (mon.0) 14857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[46934]: audit 2022-04-22T22:38:57.694706+0000 mon.a (mon.0) 14858 : audit [DBG] from='client.? 172.21.15.115:0/2365414073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:58.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[45495]: cluster 2022-04-22T22:38:55.823784+0000 mgr.y (mgr.34107) 10664 : cluster [DBG] pgmap v11325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:58.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[45495]: audit 2022-04-22T22:38:56.833713+0000 mon.a (mon.0) 14857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:58.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:57 smithi115 bash[45495]: audit 2022-04-22T22:38:57.694706+0000 mon.a (mon.0) 14858 : audit [DBG] from='client.? 172.21.15.115:0/2365414073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:57 smithi121 bash[36902]: cluster 2022-04-22T22:38:55.823784+0000 mgr.y (mgr.34107) 10664 : cluster [DBG] pgmap v11325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:38:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:57 smithi121 bash[36902]: audit 2022-04-22T22:38:56.833713+0000 mon.a (mon.0) 14857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:38:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:57 smithi121 bash[36902]: audit 2022-04-22T22:38:57.694706+0000 mon.a (mon.0) 14858 : audit [DBG] from='client.? 172.21.15.115:0/2365414073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:38:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[45495]: audit 2022-04-22T22:38:56.921715+0000 mgr.y (mgr.34107) 10665 : audit [DBG] from='client.56335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[45495]: audit 2022-04-22T22:38:57.265533+0000 mgr.y (mgr.34107) 10666 : audit [DBG] from='client.56341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[45495]: audit 2022-04-22T22:38:58.221006+0000 mon.a (mon.0) 14859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[46934]: audit 2022-04-22T22:38:56.921715+0000 mgr.y (mgr.34107) 10665 : audit [DBG] from='client.56335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[46934]: audit 2022-04-22T22:38:57.265533+0000 mgr.y (mgr.34107) 10666 : audit [DBG] from='client.56341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:58 smithi115 bash[46934]: audit 2022-04-22T22:38:58.221006+0000 mon.a (mon.0) 14859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:38:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:58 smithi121 bash[36902]: audit 2022-04-22T22:38:56.921715+0000 mgr.y (mgr.34107) 10665 : audit [DBG] from='client.56335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:58 smithi121 bash[36902]: audit 2022-04-22T22:38:57.265533+0000 mgr.y (mgr.34107) 10666 : audit [DBG] from='client.56341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:38:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:58 smithi121 bash[36902]: audit 2022-04-22T22:38:58.221006+0000 mon.a (mon.0) 14859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:38:59 smithi115 bash[45495]: cluster 2022-04-22T22:38:57.824831+0000 mgr.y (mgr.34107) 10667 : cluster [DBG] pgmap v11326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:38:59 smithi115 bash[46934]: cluster 2022-04-22T22:38:57.824831+0000 mgr.y (mgr.34107) 10667 : cluster [DBG] pgmap v11326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:38:59 smithi121 bash[36902]: cluster 2022-04-22T22:38:57.824831+0000 mgr.y (mgr.34107) 10667 : cluster [DBG] pgmap v11326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:00 smithi121 bash[36902]: audit 2022-04-22T22:39:00.212132+0000 mon.a (mon.0) 14860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:01.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:00 smithi115 bash[46934]: audit 2022-04-22T22:39:00.212132+0000 mon.a (mon.0) 14860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:00 smithi115 bash[45495]: audit 2022-04-22T22:39:00.212132+0000 mon.a (mon.0) 14860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:01 smithi121 bash[36902]: cluster 2022-04-22T22:38:59.826016+0000 mgr.y (mgr.34107) 10668 : cluster [DBG] pgmap v11327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:01 smithi115 bash[45495]: cluster 2022-04-22T22:38:59.826016+0000 mgr.y (mgr.34107) 10668 : cluster [DBG] pgmap v11327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:01 smithi115 bash[46934]: cluster 2022-04-22T22:38:59.826016+0000 mgr.y (mgr.34107) 10668 : cluster [DBG] pgmap v11327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:02 smithi121 bash[36902]: audit 2022-04-22T22:39:01.836658+0000 mon.a (mon.0) 14861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:02 smithi115 bash[45495]: audit 2022-04-22T22:39:01.836658+0000 mon.a (mon.0) 14861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:02.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:02 smithi115 bash[46934]: audit 2022-04-22T22:39:01.836658+0000 mon.a (mon.0) 14861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:03 smithi121 bash[36902]: cluster 2022-04-22T22:39:01.827092+0000 mgr.y (mgr.34107) 10669 : cluster [DBG] pgmap v11328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:03 smithi121 bash[36902]: audit 2022-04-22T22:39:03.224448+0000 mon.a (mon.0) 14862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:03 smithi115 bash[45495]: cluster 2022-04-22T22:39:01.827092+0000 mgr.y (mgr.34107) 10669 : cluster [DBG] pgmap v11328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:03 smithi115 bash[45495]: audit 2022-04-22T22:39:03.224448+0000 mon.a (mon.0) 14862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:03 smithi115 bash[46934]: cluster 2022-04-22T22:39:01.827092+0000 mgr.y (mgr.34107) 10669 : cluster [DBG] pgmap v11328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:03 smithi115 bash[46934]: audit 2022-04-22T22:39:03.224448+0000 mon.a (mon.0) 14862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:04 smithi115 bash[45495]: cluster 2022-04-22T22:39:03.828099+0000 mgr.y (mgr.34107) 10670 : cluster [DBG] pgmap v11329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:04.703Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:04 smithi115 bash[46934]: cluster 2022-04-22T22:39:03.828099+0000 mgr.y (mgr.34107) 10670 : cluster [DBG] pgmap v11329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:04 smithi121 bash[36902]: cluster 2022-04-22T22:39:03.828099+0000 mgr.y (mgr.34107) 10670 : cluster [DBG] pgmap v11329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:05 smithi121 bash[36902]: audit 2022-04-22T22:39:05.215418+0000 mon.a (mon.0) 14863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:05 smithi115 bash[45495]: audit 2022-04-22T22:39:05.215418+0000 mon.a (mon.0) 14863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:05 smithi115 bash[46934]: audit 2022-04-22T22:39:05.215418+0000 mon.a (mon.0) 14863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:07 smithi121 bash[36902]: cluster 2022-04-22T22:39:05.829119+0000 mgr.y (mgr.34107) 10671 : cluster [DBG] pgmap v11330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:07 smithi121 bash[36902]: audit 2022-04-22T22:39:06.839399+0000 mon.a (mon.0) 14864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:07 smithi115 bash[45495]: cluster 2022-04-22T22:39:05.829119+0000 mgr.y (mgr.34107) 10671 : cluster [DBG] pgmap v11330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:07 smithi115 bash[45495]: audit 2022-04-22T22:39:06.839399+0000 mon.a (mon.0) 14864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:07 smithi115 bash[46934]: cluster 2022-04-22T22:39:05.829119+0000 mgr.y (mgr.34107) 10671 : cluster [DBG] pgmap v11330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:07 smithi115 bash[46934]: audit 2022-04-22T22:39:06.839399+0000 mon.a (mon.0) 14864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:08 smithi115 bash[45495]: audit 2022-04-22T22:39:08.228298+0000 mon.a (mon.0) 14865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:08 smithi115 bash[46934]: audit 2022-04-22T22:39:08.228298+0000 mon.a (mon.0) 14865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:08 smithi121 bash[36902]: audit 2022-04-22T22:39:08.228298+0000 mon.a (mon.0) 14865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:09 smithi115 bash[45495]: cluster 2022-04-22T22:39:07.830287+0000 mgr.y (mgr.34107) 10672 : cluster [DBG] pgmap v11331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:09 smithi115 bash[46934]: cluster 2022-04-22T22:39:07.830287+0000 mgr.y (mgr.34107) 10672 : cluster [DBG] pgmap v11331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:09 smithi121 bash[36902]: cluster 2022-04-22T22:39:07.830287+0000 mgr.y (mgr.34107) 10672 : cluster [DBG] pgmap v11331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:11.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:10 smithi115 bash[45495]: audit 2022-04-22T22:39:10.218552+0000 mon.a (mon.0) 14866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:10 smithi115 bash[46934]: audit 2022-04-22T22:39:10.218552+0000 mon.a (mon.0) 14866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:10 smithi121 bash[36902]: audit 2022-04-22T22:39:10.218552+0000 mon.a (mon.0) 14866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:11 smithi115 bash[45495]: cluster 2022-04-22T22:39:09.831355+0000 mgr.y (mgr.34107) 10673 : cluster [DBG] pgmap v11332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:11 smithi115 bash[46934]: cluster 2022-04-22T22:39:09.831355+0000 mgr.y (mgr.34107) 10673 : cluster [DBG] pgmap v11332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:11 smithi121 bash[36902]: cluster 2022-04-22T22:39:09.831355+0000 mgr.y (mgr.34107) 10673 : cluster [DBG] pgmap v11332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:12 smithi121 bash[36902]: audit 2022-04-22T22:39:11.841918+0000 mon.a (mon.0) 14867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:12 smithi115 bash[46934]: audit 2022-04-22T22:39:11.841918+0000 mon.a (mon.0) 14867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:12.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:13.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:12 smithi115 bash[45495]: audit 2022-04-22T22:39:11.841918+0000 mon.a (mon.0) 14867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:13 smithi121 bash[36902]: cluster 2022-04-22T22:39:11.832337+0000 mgr.y (mgr.34107) 10674 : cluster [DBG] pgmap v11333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:13 smithi121 bash[36902]: audit 2022-04-22T22:39:13.232017+0000 mon.a (mon.0) 14868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:13 smithi115 bash[46934]: cluster 2022-04-22T22:39:11.832337+0000 mgr.y (mgr.34107) 10674 : cluster [DBG] pgmap v11333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:13 smithi115 bash[46934]: audit 2022-04-22T22:39:13.232017+0000 mon.a (mon.0) 14868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:13 smithi115 bash[45495]: cluster 2022-04-22T22:39:11.832337+0000 mgr.y (mgr.34107) 10674 : cluster [DBG] pgmap v11333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:13 smithi115 bash[45495]: audit 2022-04-22T22:39:13.232017+0000 mon.a (mon.0) 14868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:14.704Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:15 smithi121 bash[36902]: cluster 2022-04-22T22:39:13.833527+0000 mgr.y (mgr.34107) 10675 : cluster [DBG] pgmap v11334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:15 smithi121 bash[36902]: audit 2022-04-22T22:39:15.221464+0000 mon.a (mon.0) 14869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:15 smithi115 bash[46934]: cluster 2022-04-22T22:39:13.833527+0000 mgr.y (mgr.34107) 10675 : cluster [DBG] pgmap v11334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:15 smithi115 bash[46934]: audit 2022-04-22T22:39:15.221464+0000 mon.a (mon.0) 14869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:15 smithi115 bash[45495]: cluster 2022-04-22T22:39:13.833527+0000 mgr.y (mgr.34107) 10675 : cluster [DBG] pgmap v11334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:15 smithi115 bash[45495]: audit 2022-04-22T22:39:15.221464+0000 mon.a (mon.0) 14869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:17 smithi121 bash[36902]: cluster 2022-04-22T22:39:15.834586+0000 mgr.y (mgr.34107) 10676 : cluster [DBG] pgmap v11335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:17 smithi121 bash[36902]: audit 2022-04-22T22:39:16.844740+0000 mon.a (mon.0) 14870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:17 smithi121 bash[36902]: audit 2022-04-22T22:39:16.882031+0000 mon.a (mon.0) 14871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:39:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:17 smithi121 bash[36902]: audit 2022-04-22T22:39:16.882249+0000 mon.a (mon.0) 14872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:39:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[45495]: cluster 2022-04-22T22:39:15.834586+0000 mgr.y (mgr.34107) 10676 : cluster [DBG] pgmap v11335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[45495]: audit 2022-04-22T22:39:16.844740+0000 mon.a (mon.0) 14870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[45495]: audit 2022-04-22T22:39:16.882031+0000 mon.a (mon.0) 14871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:39:18.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[45495]: audit 2022-04-22T22:39:16.882249+0000 mon.a (mon.0) 14872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:39:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[46934]: cluster 2022-04-22T22:39:15.834586+0000 mgr.y (mgr.34107) 10676 : cluster [DBG] pgmap v11335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[46934]: audit 2022-04-22T22:39:16.844740+0000 mon.a (mon.0) 14870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:18.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[46934]: audit 2022-04-22T22:39:16.882031+0000 mon.a (mon.0) 14871 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:39:18.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:17 smithi115 bash[46934]: audit 2022-04-22T22:39:16.882249+0000 mon.a (mon.0) 14872 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:39:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:18 smithi121 bash[36902]: audit 2022-04-22T22:39:18.235564+0000 mon.a (mon.0) 14873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:18 smithi115 bash[45495]: audit 2022-04-22T22:39:18.235564+0000 mon.a (mon.0) 14873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:18 smithi115 bash[46934]: audit 2022-04-22T22:39:18.235564+0000 mon.a (mon.0) 14873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:20.269 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:19 smithi121 bash[36902]: cluster 2022-04-22T22:39:17.835680+0000 mgr.y (mgr.34107) 10677 : cluster [DBG] pgmap v11336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:19 smithi115 bash[45495]: cluster 2022-04-22T22:39:17.835680+0000 mgr.y (mgr.34107) 10677 : cluster [DBG] pgmap v11336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:19 smithi115 bash[46934]: cluster 2022-04-22T22:39:17.835680+0000 mgr.y (mgr.34107) 10677 : cluster [DBG] pgmap v11336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:20 smithi121 bash[36902]: audit 2022-04-22T22:39:20.224450+0000 mon.a (mon.0) 14874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:20 smithi115 bash[45495]: audit 2022-04-22T22:39:20.224450+0000 mon.a (mon.0) 14874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:20 smithi115 bash[46934]: audit 2022-04-22T22:39:20.224450+0000 mon.a (mon.0) 14874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:21 smithi121 bash[36902]: cluster 2022-04-22T22:39:19.836577+0000 mgr.y (mgr.34107) 10678 : cluster [DBG] pgmap v11337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:21 smithi115 bash[45495]: cluster 2022-04-22T22:39:19.836577+0000 mgr.y (mgr.34107) 10678 : cluster [DBG] pgmap v11337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:21 smithi115 bash[46934]: cluster 2022-04-22T22:39:19.836577+0000 mgr.y (mgr.34107) 10678 : cluster [DBG] pgmap v11337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:22 smithi121 bash[36902]: cluster 2022-04-22T22:39:21.837291+0000 mgr.y (mgr.34107) 10679 : cluster [DBG] pgmap v11338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:22 smithi121 bash[36902]: audit 2022-04-22T22:39:21.847660+0000 mon.a (mon.0) 14875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:22.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:22 smithi115 bash[45495]: cluster 2022-04-22T22:39:21.837291+0000 mgr.y (mgr.34107) 10679 : cluster [DBG] pgmap v11338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:22 smithi115 bash[45495]: audit 2022-04-22T22:39:21.847660+0000 mon.a (mon.0) 14875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:22 smithi115 bash[46934]: cluster 2022-04-22T22:39:21.837291+0000 mgr.y (mgr.34107) 10679 : cluster [DBG] pgmap v11338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:22 smithi115 bash[46934]: audit 2022-04-22T22:39:21.847660+0000 mon.a (mon.0) 14875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:23 smithi121 bash[36902]: audit 2022-04-22T22:39:23.239402+0000 mon.a (mon.0) 14876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:23 smithi115 bash[45495]: audit 2022-04-22T22:39:23.239402+0000 mon.a (mon.0) 14876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:23 smithi115 bash[46934]: audit 2022-04-22T22:39:23.239402+0000 mon.a (mon.0) 14876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:24 smithi115 bash[45495]: cluster 2022-04-22T22:39:23.838345+0000 mgr.y (mgr.34107) 10680 : cluster [DBG] pgmap v11339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:24.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:24 smithi115 bash[46934]: cluster 2022-04-22T22:39:23.838345+0000 mgr.y (mgr.34107) 10680 : cluster [DBG] pgmap v11339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:24 smithi121 bash[36902]: cluster 2022-04-22T22:39:23.838345+0000 mgr.y (mgr.34107) 10680 : cluster [DBG] pgmap v11339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:25 smithi121 bash[36902]: audit 2022-04-22T22:39:25.227406+0000 mon.a (mon.0) 14877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:25 smithi115 bash[45495]: audit 2022-04-22T22:39:25.227406+0000 mon.a (mon.0) 14877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:25 smithi115 bash[46934]: audit 2022-04-22T22:39:25.227406+0000 mon.a (mon.0) 14877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:28.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:27 smithi115 bash[46934]: cluster 2022-04-22T22:39:25.839490+0000 mgr.y (mgr.34107) 10681 : cluster [DBG] pgmap v11340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:27 smithi115 bash[46934]: audit 2022-04-22T22:39:26.850475+0000 mon.a (mon.0) 14878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:27 smithi115 bash[45495]: cluster 2022-04-22T22:39:25.839490+0000 mgr.y (mgr.34107) 10681 : cluster [DBG] pgmap v11340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:27 smithi115 bash[45495]: audit 2022-04-22T22:39:26.850475+0000 mon.a (mon.0) 14878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:28.081 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:39:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:27 smithi121 bash[36902]: cluster 2022-04-22T22:39:25.839490+0000 mgr.y (mgr.34107) 10681 : cluster [DBG] pgmap v11340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:28.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:27 smithi121 bash[36902]: audit 2022-04-22T22:39:26.850475+0000 mon.a (mon.0) 14878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:28.430 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:39:28.430 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:39:28.431 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:39:28.432 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:39:28.433 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:39:28.433 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:39:28.837 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:39:28.837 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:39:28.837 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:39:28.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:39:28.838 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:39:28.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:28.839 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:39:28.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:39:28.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:39:28.839 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:39:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:28 smithi115 bash[45495]: audit 2022-04-22T22:39:28.242888+0000 mon.a (mon.0) 14879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:28 smithi115 bash[46934]: audit 2022-04-22T22:39:28.242888+0000 mon.a (mon.0) 14879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:28 smithi121 bash[36902]: audit 2022-04-22T22:39:28.242888+0000 mon.a (mon.0) 14879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[45495]: cluster 2022-04-22T22:39:27.840139+0000 mgr.y (mgr.34107) 10682 : cluster [DBG] pgmap v11341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[45495]: audit 2022-04-22T22:39:28.058595+0000 mgr.y (mgr.34107) 10683 : audit [DBG] from='client.56353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[45495]: audit 2022-04-22T22:39:28.406658+0000 mgr.y (mgr.34107) 10684 : audit [DBG] from='client.46425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[45495]: audit 2022-04-22T22:39:28.833832+0000 mon.a (mon.0) 14880 : audit [DBG] from='client.? 172.21.15.115:0/1206400642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:39:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[46934]: cluster 2022-04-22T22:39:27.840139+0000 mgr.y (mgr.34107) 10682 : cluster [DBG] pgmap v11341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[46934]: audit 2022-04-22T22:39:28.058595+0000 mgr.y (mgr.34107) 10683 : audit [DBG] from='client.56353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[46934]: audit 2022-04-22T22:39:28.406658+0000 mgr.y (mgr.34107) 10684 : audit [DBG] from='client.46425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:29 smithi115 bash[46934]: audit 2022-04-22T22:39:28.833832+0000 mon.a (mon.0) 14880 : audit [DBG] from='client.? 172.21.15.115:0/1206400642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:39:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:29 smithi121 bash[36902]: cluster 2022-04-22T22:39:27.840139+0000 mgr.y (mgr.34107) 10682 : cluster [DBG] pgmap v11341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:29 smithi121 bash[36902]: audit 2022-04-22T22:39:28.058595+0000 mgr.y (mgr.34107) 10683 : audit [DBG] from='client.56353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:29 smithi121 bash[36902]: audit 2022-04-22T22:39:28.406658+0000 mgr.y (mgr.34107) 10684 : audit [DBG] from='client.46425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:39:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:29 smithi121 bash[36902]: audit 2022-04-22T22:39:28.833832+0000 mon.a (mon.0) 14880 : audit [DBG] from='client.? 172.21.15.115:0/1206400642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:39:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:30 smithi121 bash[36902]: audit 2022-04-22T22:39:30.230635+0000 mon.a (mon.0) 14881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:30 smithi121 bash[36902]: audit 2022-04-22T22:39:30.706317+0000 mon.a (mon.0) 14882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:39:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:30 smithi121 bash[36902]: audit 2022-04-22T22:39:30.709115+0000 mon.a (mon.0) 14883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:39:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:30 smithi121 bash[36902]: audit 2022-04-22T22:39:30.710776+0000 mon.a (mon.0) 14884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:39:31.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[45495]: audit 2022-04-22T22:39:30.230635+0000 mon.a (mon.0) 14881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[45495]: audit 2022-04-22T22:39:30.706317+0000 mon.a (mon.0) 14882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[45495]: audit 2022-04-22T22:39:30.709115+0000 mon.a (mon.0) 14883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[45495]: audit 2022-04-22T22:39:30.710776+0000 mon.a (mon.0) 14884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[46934]: audit 2022-04-22T22:39:30.230635+0000 mon.a (mon.0) 14881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[46934]: audit 2022-04-22T22:39:30.706317+0000 mon.a (mon.0) 14882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[46934]: audit 2022-04-22T22:39:30.709115+0000 mon.a (mon.0) 14883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:39:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:30 smithi115 bash[46934]: audit 2022-04-22T22:39:30.710776+0000 mon.a (mon.0) 14884 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:39:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:31 smithi115 bash[45495]: cluster 2022-04-22T22:39:29.841172+0000 mgr.y (mgr.34107) 10685 : cluster [DBG] pgmap v11342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:31 smithi115 bash[45495]: audit 2022-04-22T22:39:31.081560+0000 mon.a (mon.0) 14885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:39:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:31 smithi115 bash[46934]: cluster 2022-04-22T22:39:29.841172+0000 mgr.y (mgr.34107) 10685 : cluster [DBG] pgmap v11342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:31 smithi115 bash[46934]: audit 2022-04-22T22:39:31.081560+0000 mon.a (mon.0) 14885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:39:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:31 smithi121 bash[36902]: cluster 2022-04-22T22:39:29.841172+0000 mgr.y (mgr.34107) 10685 : cluster [DBG] pgmap v11342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:31 smithi121 bash[36902]: audit 2022-04-22T22:39:31.081560+0000 mon.a (mon.0) 14885 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:39:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:32 smithi121 bash[36902]: audit 2022-04-22T22:39:31.853606+0000 mon.a (mon.0) 14886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:32 smithi115 bash[45495]: audit 2022-04-22T22:39:31.853606+0000 mon.a (mon.0) 14886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:32 smithi115 bash[46934]: audit 2022-04-22T22:39:31.853606+0000 mon.a (mon.0) 14886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:32.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:33 smithi121 bash[36902]: cluster 2022-04-22T22:39:31.842248+0000 mgr.y (mgr.34107) 10686 : cluster [DBG] pgmap v11343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:33 smithi121 bash[36902]: audit 2022-04-22T22:39:33.246505+0000 mon.a (mon.0) 14887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:33 smithi115 bash[45495]: cluster 2022-04-22T22:39:31.842248+0000 mgr.y (mgr.34107) 10686 : cluster [DBG] pgmap v11343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:33 smithi115 bash[45495]: audit 2022-04-22T22:39:33.246505+0000 mon.a (mon.0) 14887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:33 smithi115 bash[46934]: cluster 2022-04-22T22:39:31.842248+0000 mgr.y (mgr.34107) 10686 : cluster [DBG] pgmap v11343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:33 smithi115 bash[46934]: audit 2022-04-22T22:39:33.246505+0000 mon.a (mon.0) 14887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:34.706Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:35 smithi121 bash[36902]: cluster 2022-04-22T22:39:33.843543+0000 mgr.y (mgr.34107) 10687 : cluster [DBG] pgmap v11344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:35 smithi121 bash[36902]: audit 2022-04-22T22:39:35.233703+0000 mon.a (mon.0) 14888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:35 smithi115 bash[45495]: cluster 2022-04-22T22:39:33.843543+0000 mgr.y (mgr.34107) 10687 : cluster [DBG] pgmap v11344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:35 smithi115 bash[45495]: audit 2022-04-22T22:39:35.233703+0000 mon.a (mon.0) 14888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:35 smithi115 bash[46934]: cluster 2022-04-22T22:39:33.843543+0000 mgr.y (mgr.34107) 10687 : cluster [DBG] pgmap v11344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:35 smithi115 bash[46934]: audit 2022-04-22T22:39:35.233703+0000 mon.a (mon.0) 14888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:37 smithi115 bash[45495]: cluster 2022-04-22T22:39:35.844585+0000 mgr.y (mgr.34107) 10688 : cluster [DBG] pgmap v11345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:37 smithi115 bash[45495]: audit 2022-04-22T22:39:36.856552+0000 mon.a (mon.0) 14889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:37 smithi115 bash[46934]: cluster 2022-04-22T22:39:35.844585+0000 mgr.y (mgr.34107) 10688 : cluster [DBG] pgmap v11345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:37 smithi115 bash[46934]: audit 2022-04-22T22:39:36.856552+0000 mon.a (mon.0) 14889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:37 smithi121 bash[36902]: cluster 2022-04-22T22:39:35.844585+0000 mgr.y (mgr.34107) 10688 : cluster [DBG] pgmap v11345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:37 smithi121 bash[36902]: audit 2022-04-22T22:39:36.856552+0000 mon.a (mon.0) 14889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:38 smithi121 bash[36902]: audit 2022-04-22T22:39:38.250354+0000 mon.a (mon.0) 14890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:38 smithi115 bash[45495]: audit 2022-04-22T22:39:38.250354+0000 mon.a (mon.0) 14890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:38 smithi115 bash[46934]: audit 2022-04-22T22:39:38.250354+0000 mon.a (mon.0) 14890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:39 smithi121 bash[36902]: cluster 2022-04-22T22:39:37.845692+0000 mgr.y (mgr.34107) 10689 : cluster [DBG] pgmap v11346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:40.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:39 smithi115 bash[45495]: cluster 2022-04-22T22:39:37.845692+0000 mgr.y (mgr.34107) 10689 : cluster [DBG] pgmap v11346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:40.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:39 smithi115 bash[46934]: cluster 2022-04-22T22:39:37.845692+0000 mgr.y (mgr.34107) 10689 : cluster [DBG] pgmap v11346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:40 smithi121 bash[36902]: audit 2022-04-22T22:39:40.236603+0000 mon.a (mon.0) 14891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:40 smithi115 bash[45495]: audit 2022-04-22T22:39:40.236603+0000 mon.a (mon.0) 14891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:40 smithi115 bash[46934]: audit 2022-04-22T22:39:40.236603+0000 mon.a (mon.0) 14891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:41 smithi121 bash[36902]: cluster 2022-04-22T22:39:39.846782+0000 mgr.y (mgr.34107) 10690 : cluster [DBG] pgmap v11347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:41 smithi115 bash[45495]: cluster 2022-04-22T22:39:39.846782+0000 mgr.y (mgr.34107) 10690 : cluster [DBG] pgmap v11347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:41 smithi115 bash[46934]: cluster 2022-04-22T22:39:39.846782+0000 mgr.y (mgr.34107) 10690 : cluster [DBG] pgmap v11347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:42 smithi121 bash[36902]: audit 2022-04-22T22:39:41.859415+0000 mon.a (mon.0) 14892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:42 smithi115 bash[45495]: audit 2022-04-22T22:39:41.859415+0000 mon.a (mon.0) 14892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:42.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:42 smithi115 bash[46934]: audit 2022-04-22T22:39:41.859415+0000 mon.a (mon.0) 14892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:43 smithi121 bash[36902]: cluster 2022-04-22T22:39:41.847893+0000 mgr.y (mgr.34107) 10691 : cluster [DBG] pgmap v11348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:43 smithi121 bash[36902]: audit 2022-04-22T22:39:43.254348+0000 mon.a (mon.0) 14893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:43 smithi115 bash[45495]: cluster 2022-04-22T22:39:41.847893+0000 mgr.y (mgr.34107) 10691 : cluster [DBG] pgmap v11348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:43 smithi115 bash[45495]: audit 2022-04-22T22:39:43.254348+0000 mon.a (mon.0) 14893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:43 smithi115 bash[46934]: cluster 2022-04-22T22:39:41.847893+0000 mgr.y (mgr.34107) 10691 : cluster [DBG] pgmap v11348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:43 smithi115 bash[46934]: audit 2022-04-22T22:39:43.254348+0000 mon.a (mon.0) 14893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:44.707Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:45 smithi121 bash[36902]: cluster 2022-04-22T22:39:43.849106+0000 mgr.y (mgr.34107) 10692 : cluster [DBG] pgmap v11349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:45 smithi121 bash[36902]: audit 2022-04-22T22:39:45.239662+0000 mon.a (mon.0) 14894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:45 smithi115 bash[45495]: cluster 2022-04-22T22:39:43.849106+0000 mgr.y (mgr.34107) 10692 : cluster [DBG] pgmap v11349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:45 smithi115 bash[45495]: audit 2022-04-22T22:39:45.239662+0000 mon.a (mon.0) 14894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:45 smithi115 bash[46934]: cluster 2022-04-22T22:39:43.849106+0000 mgr.y (mgr.34107) 10692 : cluster [DBG] pgmap v11349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:45 smithi115 bash[46934]: audit 2022-04-22T22:39:45.239662+0000 mon.a (mon.0) 14894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:47 smithi115 bash[45495]: cluster 2022-04-22T22:39:45.850108+0000 mgr.y (mgr.34107) 10693 : cluster [DBG] pgmap v11350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:47 smithi115 bash[45495]: audit 2022-04-22T22:39:46.862487+0000 mon.a (mon.0) 14895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:47 smithi115 bash[46934]: cluster 2022-04-22T22:39:45.850108+0000 mgr.y (mgr.34107) 10693 : cluster [DBG] pgmap v11350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:47 smithi115 bash[46934]: audit 2022-04-22T22:39:46.862487+0000 mon.a (mon.0) 14895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:47 smithi121 bash[36902]: cluster 2022-04-22T22:39:45.850108+0000 mgr.y (mgr.34107) 10693 : cluster [DBG] pgmap v11350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:47 smithi121 bash[36902]: audit 2022-04-22T22:39:46.862487+0000 mon.a (mon.0) 14895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:48 smithi121 bash[36902]: audit 2022-04-22T22:39:48.258186+0000 mon.a (mon.0) 14896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:48 smithi115 bash[45495]: audit 2022-04-22T22:39:48.258186+0000 mon.a (mon.0) 14896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:48 smithi115 bash[46934]: audit 2022-04-22T22:39:48.258186+0000 mon.a (mon.0) 14896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:49 smithi121 bash[36902]: cluster 2022-04-22T22:39:47.851301+0000 mgr.y (mgr.34107) 10694 : cluster [DBG] pgmap v11351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:49 smithi115 bash[45495]: cluster 2022-04-22T22:39:47.851301+0000 mgr.y (mgr.34107) 10694 : cluster [DBG] pgmap v11351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:49 smithi115 bash[46934]: cluster 2022-04-22T22:39:47.851301+0000 mgr.y (mgr.34107) 10694 : cluster [DBG] pgmap v11351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:50 smithi121 bash[36902]: audit 2022-04-22T22:39:50.242830+0000 mon.a (mon.0) 14897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:50 smithi115 bash[45495]: audit 2022-04-22T22:39:50.242830+0000 mon.a (mon.0) 14897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:51.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:50 smithi115 bash[46934]: audit 2022-04-22T22:39:50.242830+0000 mon.a (mon.0) 14897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:51 smithi121 bash[36902]: cluster 2022-04-22T22:39:49.852374+0000 mgr.y (mgr.34107) 10695 : cluster [DBG] pgmap v11352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:51 smithi115 bash[45495]: cluster 2022-04-22T22:39:49.852374+0000 mgr.y (mgr.34107) 10695 : cluster [DBG] pgmap v11352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:51 smithi115 bash[46934]: cluster 2022-04-22T22:39:49.852374+0000 mgr.y (mgr.34107) 10695 : cluster [DBG] pgmap v11352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:52 smithi121 bash[36902]: audit 2022-04-22T22:39:51.865806+0000 mon.a (mon.0) 14898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:52 smithi115 bash[45495]: audit 2022-04-22T22:39:51.865806+0000 mon.a (mon.0) 14898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:52.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:52 smithi115 bash[46934]: audit 2022-04-22T22:39:51.865806+0000 mon.a (mon.0) 14898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:53 smithi121 bash[36902]: cluster 2022-04-22T22:39:51.853441+0000 mgr.y (mgr.34107) 10696 : cluster [DBG] pgmap v11353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:53 smithi121 bash[36902]: audit 2022-04-22T22:39:53.261966+0000 mon.a (mon.0) 14899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:53 smithi115 bash[45495]: cluster 2022-04-22T22:39:51.853441+0000 mgr.y (mgr.34107) 10696 : cluster [DBG] pgmap v11353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:53 smithi115 bash[45495]: audit 2022-04-22T22:39:53.261966+0000 mon.a (mon.0) 14899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:53 smithi115 bash[46934]: cluster 2022-04-22T22:39:51.853441+0000 mgr.y (mgr.34107) 10696 : cluster [DBG] pgmap v11353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:53 smithi115 bash[46934]: audit 2022-04-22T22:39:53.261966+0000 mon.a (mon.0) 14899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:54 smithi115 bash[45495]: cluster 2022-04-22T22:39:53.854558+0000 mgr.y (mgr.34107) 10697 : cluster [DBG] pgmap v11354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:39:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:39:54.708Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:39:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:54 smithi115 bash[46934]: cluster 2022-04-22T22:39:53.854558+0000 mgr.y (mgr.34107) 10697 : cluster [DBG] pgmap v11354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:54 smithi121 bash[36902]: cluster 2022-04-22T22:39:53.854558+0000 mgr.y (mgr.34107) 10697 : cluster [DBG] pgmap v11354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:55 smithi121 bash[36902]: audit 2022-04-22T22:39:55.245864+0000 mon.a (mon.0) 14900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:55 smithi115 bash[45495]: audit 2022-04-22T22:39:55.245864+0000 mon.a (mon.0) 14900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:55 smithi115 bash[46934]: audit 2022-04-22T22:39:55.245864+0000 mon.a (mon.0) 14900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:39:58.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:57 smithi115 bash[46934]: cluster 2022-04-22T22:39:55.855714+0000 mgr.y (mgr.34107) 10698 : cluster [DBG] pgmap v11355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:57 smithi115 bash[46934]: audit 2022-04-22T22:39:56.868834+0000 mon.a (mon.0) 14901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:57 smithi115 bash[45495]: cluster 2022-04-22T22:39:55.855714+0000 mgr.y (mgr.34107) 10698 : cluster [DBG] pgmap v11355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:57 smithi115 bash[45495]: audit 2022-04-22T22:39:56.868834+0000 mon.a (mon.0) 14901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:57 smithi121 bash[36902]: cluster 2022-04-22T22:39:55.855714+0000 mgr.y (mgr.34107) 10698 : cluster [DBG] pgmap v11355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:39:59.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:57 smithi121 bash[36902]: audit 2022-04-22T22:39:56.868834+0000 mon.a (mon.0) 14901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:39:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:39:59.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:58 smithi121 bash[36902]: audit 2022-04-22T22:39:58.265352+0000 mon.a (mon.0) 14902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:59.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:58 smithi115 bash[45495]: audit 2022-04-22T22:39:58.265352+0000 mon.a (mon.0) 14902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:59.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:58 smithi115 bash[46934]: audit 2022-04-22T22:39:58.265352+0000 mon.a (mon.0) 14902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:39:59.559 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:39:59.560 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:39:59.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:39:59.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:39:59.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:39:59.561 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:39:59.561 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:39:59.960 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:39:59.960 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:39:59.960 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:39:59.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:39:59.961 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:39:59.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:39:59.962 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:39:59.962 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:39:59.962 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:39:59.962 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:40:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:39:59 smithi121 bash[36902]: cluster 2022-04-22T22:39:57.856878+0000 mgr.y (mgr.34107) 10699 : cluster [DBG] pgmap v11356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:39:59 smithi115 bash[45495]: cluster 2022-04-22T22:39:57.856878+0000 mgr.y (mgr.34107) 10699 : cluster [DBG] pgmap v11356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:39:59 smithi115 bash[46934]: cluster 2022-04-22T22:39:57.856878+0000 mgr.y (mgr.34107) 10699 : cluster [DBG] pgmap v11356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:00 smithi121 bash[36902]: audit 2022-04-22T22:39:59.185024+0000 mgr.y (mgr.34107) 10700 : audit [DBG] from='client.46437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:00 smithi121 bash[36902]: audit 2022-04-22T22:39:59.532792+0000 mgr.y (mgr.34107) 10701 : audit [DBG] from='client.46443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:00 smithi121 bash[36902]: audit 2022-04-22T22:39:59.956644+0000 mon.c (mon.1) 361 : audit [DBG] from='client.? 172.21.15.115:0/613492464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:00 smithi121 bash[36902]: cluster 2022-04-22T22:40:00.000150+0000 mon.a (mon.0) 14903 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:00 smithi121 bash[36902]: audit 2022-04-22T22:40:00.248819+0000 mon.a (mon.0) 14904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[45495]: audit 2022-04-22T22:39:59.185024+0000 mgr.y (mgr.34107) 10700 : audit [DBG] from='client.46437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[45495]: audit 2022-04-22T22:39:59.532792+0000 mgr.y (mgr.34107) 10701 : audit [DBG] from='client.46443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[45495]: audit 2022-04-22T22:39:59.956644+0000 mon.c (mon.1) 361 : audit [DBG] from='client.? 172.21.15.115:0/613492464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[45495]: cluster 2022-04-22T22:40:00.000150+0000 mon.a (mon.0) 14903 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:40:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[45495]: audit 2022-04-22T22:40:00.248819+0000 mon.a (mon.0) 14904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[46934]: audit 2022-04-22T22:39:59.185024+0000 mgr.y (mgr.34107) 10700 : audit [DBG] from='client.46437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[46934]: audit 2022-04-22T22:39:59.532792+0000 mgr.y (mgr.34107) 10701 : audit [DBG] from='client.46443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[46934]: audit 2022-04-22T22:39:59.956644+0000 mon.c (mon.1) 361 : audit [DBG] from='client.? 172.21.15.115:0/613492464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[46934]: cluster 2022-04-22T22:40:00.000150+0000 mon.a (mon.0) 14903 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:40:01.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:00 smithi115 bash[46934]: audit 2022-04-22T22:40:00.248819+0000 mon.a (mon.0) 14904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:01 smithi121 bash[36902]: cluster 2022-04-22T22:39:59.857813+0000 mgr.y (mgr.34107) 10702 : cluster [DBG] pgmap v11357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:01 smithi115 bash[45495]: cluster 2022-04-22T22:39:59.857813+0000 mgr.y (mgr.34107) 10702 : cluster [DBG] pgmap v11357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:01 smithi115 bash[46934]: cluster 2022-04-22T22:39:59.857813+0000 mgr.y (mgr.34107) 10702 : cluster [DBG] pgmap v11357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:02 smithi121 bash[36902]: audit 2022-04-22T22:40:01.871830+0000 mon.a (mon.0) 14905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:02 smithi115 bash[45495]: audit 2022-04-22T22:40:01.871830+0000 mon.a (mon.0) 14905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:02.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:02 smithi115 bash[46934]: audit 2022-04-22T22:40:01.871830+0000 mon.a (mon.0) 14905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:03 smithi121 bash[36902]: cluster 2022-04-22T22:40:01.858911+0000 mgr.y (mgr.34107) 10703 : cluster [DBG] pgmap v11358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:03 smithi121 bash[36902]: audit 2022-04-22T22:40:03.269285+0000 mon.a (mon.0) 14906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:03 smithi115 bash[45495]: cluster 2022-04-22T22:40:01.858911+0000 mgr.y (mgr.34107) 10703 : cluster [DBG] pgmap v11358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:03 smithi115 bash[45495]: audit 2022-04-22T22:40:03.269285+0000 mon.a (mon.0) 14906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:03 smithi115 bash[46934]: cluster 2022-04-22T22:40:01.858911+0000 mgr.y (mgr.34107) 10703 : cluster [DBG] pgmap v11358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:03 smithi115 bash[46934]: audit 2022-04-22T22:40:03.269285+0000 mon.a (mon.0) 14906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:04.707Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:05 smithi121 bash[36902]: cluster 2022-04-22T22:40:03.860023+0000 mgr.y (mgr.34107) 10704 : cluster [DBG] pgmap v11359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:05 smithi121 bash[36902]: audit 2022-04-22T22:40:05.252021+0000 mon.a (mon.0) 14907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:05 smithi115 bash[45495]: cluster 2022-04-22T22:40:03.860023+0000 mgr.y (mgr.34107) 10704 : cluster [DBG] pgmap v11359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:05 smithi115 bash[45495]: audit 2022-04-22T22:40:05.252021+0000 mon.a (mon.0) 14907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:05 smithi115 bash[46934]: cluster 2022-04-22T22:40:03.860023+0000 mgr.y (mgr.34107) 10704 : cluster [DBG] pgmap v11359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:05 smithi115 bash[46934]: audit 2022-04-22T22:40:05.252021+0000 mon.a (mon.0) 14907 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:07 smithi121 bash[36902]: cluster 2022-04-22T22:40:05.861120+0000 mgr.y (mgr.34107) 10705 : cluster [DBG] pgmap v11360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:07 smithi121 bash[36902]: audit 2022-04-22T22:40:06.874764+0000 mon.a (mon.0) 14908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:08.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:07 smithi115 bash[46934]: cluster 2022-04-22T22:40:05.861120+0000 mgr.y (mgr.34107) 10705 : cluster [DBG] pgmap v11360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:07 smithi115 bash[46934]: audit 2022-04-22T22:40:06.874764+0000 mon.a (mon.0) 14908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:07 smithi115 bash[45495]: cluster 2022-04-22T22:40:05.861120+0000 mgr.y (mgr.34107) 10705 : cluster [DBG] pgmap v11360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:07 smithi115 bash[45495]: audit 2022-04-22T22:40:06.874764+0000 mon.a (mon.0) 14908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:08 smithi121 bash[36902]: audit 2022-04-22T22:40:08.272950+0000 mon.a (mon.0) 14909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:08 smithi115 bash[46934]: audit 2022-04-22T22:40:08.272950+0000 mon.a (mon.0) 14909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:08 smithi115 bash[45495]: audit 2022-04-22T22:40:08.272950+0000 mon.a (mon.0) 14909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:09 smithi121 bash[36902]: cluster 2022-04-22T22:40:07.862217+0000 mgr.y (mgr.34107) 10706 : cluster [DBG] pgmap v11361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:10.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:09 smithi115 bash[46934]: cluster 2022-04-22T22:40:07.862217+0000 mgr.y (mgr.34107) 10706 : cluster [DBG] pgmap v11361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:09 smithi115 bash[45495]: cluster 2022-04-22T22:40:07.862217+0000 mgr.y (mgr.34107) 10706 : cluster [DBG] pgmap v11361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:10 smithi121 bash[36902]: cluster 2022-04-22T22:40:09.863218+0000 mgr.y (mgr.34107) 10707 : cluster [DBG] pgmap v11362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:10 smithi121 bash[36902]: audit 2022-04-22T22:40:10.255061+0000 mon.a (mon.0) 14910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:10 smithi115 bash[45495]: cluster 2022-04-22T22:40:09.863218+0000 mgr.y (mgr.34107) 10707 : cluster [DBG] pgmap v11362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:10 smithi115 bash[45495]: audit 2022-04-22T22:40:10.255061+0000 mon.a (mon.0) 14910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:10 smithi115 bash[46934]: cluster 2022-04-22T22:40:09.863218+0000 mgr.y (mgr.34107) 10707 : cluster [DBG] pgmap v11362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:11.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:10 smithi115 bash[46934]: audit 2022-04-22T22:40:10.255061+0000 mon.a (mon.0) 14910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:12 smithi121 bash[36902]: audit 2022-04-22T22:40:11.877760+0000 mon.a (mon.0) 14911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:12 smithi115 bash[45495]: audit 2022-04-22T22:40:11.877760+0000 mon.a (mon.0) 14911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:12.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:12 smithi115 bash[46934]: audit 2022-04-22T22:40:11.877760+0000 mon.a (mon.0) 14911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:13 smithi121 bash[36902]: cluster 2022-04-22T22:40:11.864307+0000 mgr.y (mgr.34107) 10708 : cluster [DBG] pgmap v11363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:13 smithi121 bash[36902]: audit 2022-04-22T22:40:13.276420+0000 mon.a (mon.0) 14912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:13 smithi115 bash[45495]: cluster 2022-04-22T22:40:11.864307+0000 mgr.y (mgr.34107) 10708 : cluster [DBG] pgmap v11363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:13 smithi115 bash[45495]: audit 2022-04-22T22:40:13.276420+0000 mon.a (mon.0) 14912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:13 smithi115 bash[46934]: cluster 2022-04-22T22:40:11.864307+0000 mgr.y (mgr.34107) 10708 : cluster [DBG] pgmap v11363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:13 smithi115 bash[46934]: audit 2022-04-22T22:40:13.276420+0000 mon.a (mon.0) 14912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:14.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:15 smithi121 bash[36902]: cluster 2022-04-22T22:40:13.865509+0000 mgr.y (mgr.34107) 10709 : cluster [DBG] pgmap v11364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:15 smithi121 bash[36902]: audit 2022-04-22T22:40:15.258186+0000 mon.a (mon.0) 14913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:15 smithi115 bash[45495]: cluster 2022-04-22T22:40:13.865509+0000 mgr.y (mgr.34107) 10709 : cluster [DBG] pgmap v11364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:15 smithi115 bash[45495]: audit 2022-04-22T22:40:15.258186+0000 mon.a (mon.0) 14913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:15 smithi115 bash[46934]: cluster 2022-04-22T22:40:13.865509+0000 mgr.y (mgr.34107) 10709 : cluster [DBG] pgmap v11364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:15 smithi115 bash[46934]: audit 2022-04-22T22:40:15.258186+0000 mon.a (mon.0) 14913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:17 smithi121 bash[36902]: cluster 2022-04-22T22:40:15.866533+0000 mgr.y (mgr.34107) 10710 : cluster [DBG] pgmap v11365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:17 smithi121 bash[36902]: audit 2022-04-22T22:40:16.880717+0000 mon.a (mon.0) 14914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:17 smithi121 bash[36902]: audit 2022-04-22T22:40:16.892791+0000 mon.a (mon.0) 14915 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:40:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:17 smithi121 bash[36902]: audit 2022-04-22T22:40:16.893305+0000 mon.a (mon.0) 14916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:40:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[45495]: cluster 2022-04-22T22:40:15.866533+0000 mgr.y (mgr.34107) 10710 : cluster [DBG] pgmap v11365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[45495]: audit 2022-04-22T22:40:16.880717+0000 mon.a (mon.0) 14914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[45495]: audit 2022-04-22T22:40:16.892791+0000 mon.a (mon.0) 14915 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:40:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[45495]: audit 2022-04-22T22:40:16.893305+0000 mon.a (mon.0) 14916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:40:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[46934]: cluster 2022-04-22T22:40:15.866533+0000 mgr.y (mgr.34107) 10710 : cluster [DBG] pgmap v11365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[46934]: audit 2022-04-22T22:40:16.880717+0000 mon.a (mon.0) 14914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[46934]: audit 2022-04-22T22:40:16.892791+0000 mon.a (mon.0) 14915 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:40:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:17 smithi115 bash[46934]: audit 2022-04-22T22:40:16.893305+0000 mon.a (mon.0) 14916 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:40:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:18 smithi121 bash[36902]: audit 2022-04-22T22:40:18.279970+0000 mon.a (mon.0) 14917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:19.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:18 smithi115 bash[46934]: audit 2022-04-22T22:40:18.279970+0000 mon.a (mon.0) 14917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:18 smithi115 bash[45495]: audit 2022-04-22T22:40:18.279970+0000 mon.a (mon.0) 14917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:19 smithi121 bash[36902]: cluster 2022-04-22T22:40:17.867556+0000 mgr.y (mgr.34107) 10711 : cluster [DBG] pgmap v11366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:19 smithi115 bash[45495]: cluster 2022-04-22T22:40:17.867556+0000 mgr.y (mgr.34107) 10711 : cluster [DBG] pgmap v11366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:19 smithi115 bash[46934]: cluster 2022-04-22T22:40:17.867556+0000 mgr.y (mgr.34107) 10711 : cluster [DBG] pgmap v11366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:20 smithi121 bash[36902]: audit 2022-04-22T22:40:20.261364+0000 mon.a (mon.0) 14918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:20 smithi115 bash[45495]: audit 2022-04-22T22:40:20.261364+0000 mon.a (mon.0) 14918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:20 smithi115 bash[46934]: audit 2022-04-22T22:40:20.261364+0000 mon.a (mon.0) 14918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:21 smithi121 bash[36902]: cluster 2022-04-22T22:40:19.868580+0000 mgr.y (mgr.34107) 10712 : cluster [DBG] pgmap v11367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:21 smithi115 bash[45495]: cluster 2022-04-22T22:40:19.868580+0000 mgr.y (mgr.34107) 10712 : cluster [DBG] pgmap v11367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:21 smithi115 bash[46934]: cluster 2022-04-22T22:40:19.868580+0000 mgr.y (mgr.34107) 10712 : cluster [DBG] pgmap v11367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:22 smithi121 bash[36902]: audit 2022-04-22T22:40:21.883513+0000 mon.a (mon.0) 14919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:22 smithi115 bash[45495]: audit 2022-04-22T22:40:21.883513+0000 mon.a (mon.0) 14919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:22 smithi115 bash[46934]: audit 2022-04-22T22:40:21.883513+0000 mon.a (mon.0) 14919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:22.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:23 smithi121 bash[36902]: cluster 2022-04-22T22:40:21.870126+0000 mgr.y (mgr.34107) 10713 : cluster [DBG] pgmap v11368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:23 smithi121 bash[36902]: audit 2022-04-22T22:40:23.283681+0000 mon.a (mon.0) 14920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:23 smithi115 bash[45495]: cluster 2022-04-22T22:40:21.870126+0000 mgr.y (mgr.34107) 10713 : cluster [DBG] pgmap v11368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:23 smithi115 bash[45495]: audit 2022-04-22T22:40:23.283681+0000 mon.a (mon.0) 14920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:23 smithi115 bash[46934]: cluster 2022-04-22T22:40:21.870126+0000 mgr.y (mgr.34107) 10713 : cluster [DBG] pgmap v11368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:23 smithi115 bash[46934]: audit 2022-04-22T22:40:23.283681+0000 mon.a (mon.0) 14920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:24 smithi115 bash[45495]: cluster 2022-04-22T22:40:23.871344+0000 mgr.y (mgr.34107) 10714 : cluster [DBG] pgmap v11369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:24.710Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:24 smithi115 bash[46934]: cluster 2022-04-22T22:40:23.871344+0000 mgr.y (mgr.34107) 10714 : cluster [DBG] pgmap v11369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:24 smithi121 bash[36902]: cluster 2022-04-22T22:40:23.871344+0000 mgr.y (mgr.34107) 10714 : cluster [DBG] pgmap v11369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:25 smithi121 bash[36902]: audit 2022-04-22T22:40:25.264364+0000 mon.a (mon.0) 14921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:25 smithi115 bash[45495]: audit 2022-04-22T22:40:25.264364+0000 mon.a (mon.0) 14921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:26.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:25 smithi115 bash[46934]: audit 2022-04-22T22:40:25.264364+0000 mon.a (mon.0) 14921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:27 smithi121 bash[36902]: cluster 2022-04-22T22:40:25.872427+0000 mgr.y (mgr.34107) 10715 : cluster [DBG] pgmap v11370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:27 smithi121 bash[36902]: audit 2022-04-22T22:40:26.885849+0000 mon.a (mon.0) 14922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:28.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:27 smithi115 bash[46934]: cluster 2022-04-22T22:40:25.872427+0000 mgr.y (mgr.34107) 10715 : cluster [DBG] pgmap v11370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:27 smithi115 bash[46934]: audit 2022-04-22T22:40:26.885849+0000 mon.a (mon.0) 14922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:27 smithi115 bash[45495]: cluster 2022-04-22T22:40:25.872427+0000 mgr.y (mgr.34107) 10715 : cluster [DBG] pgmap v11370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:27 smithi115 bash[45495]: audit 2022-04-22T22:40:26.885849+0000 mon.a (mon.0) 14922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:28 smithi121 bash[36902]: audit 2022-04-22T22:40:28.287429+0000 mon.a (mon.0) 14923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:29.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:28 smithi115 bash[46934]: audit 2022-04-22T22:40:28.287429+0000 mon.a (mon.0) 14923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:29.304 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:28 smithi115 bash[45495]: audit 2022-04-22T22:40:28.287429+0000 mon.a (mon.0) 14923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:29 smithi121 bash[36902]: cluster 2022-04-22T22:40:27.873539+0000 mgr.y (mgr.34107) 10716 : cluster [DBG] pgmap v11371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:29 smithi115 bash[45495]: cluster 2022-04-22T22:40:27.873539+0000 mgr.y (mgr.34107) 10716 : cluster [DBG] pgmap v11371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:29 smithi115 bash[46934]: cluster 2022-04-22T22:40:27.873539+0000 mgr.y (mgr.34107) 10716 : cluster [DBG] pgmap v11371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:30.335 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:40:30.679 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:40:30.680 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:40:30.681 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:40:30.682 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:40:31.086 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:40:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:40:31.088 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:40:31.088 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:40:31.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:30 smithi115 bash[45495]: audit 2022-04-22T22:40:30.267204+0000 mon.a (mon.0) 14924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:30 smithi115 bash[46934]: audit 2022-04-22T22:40:30.267204+0000 mon.a (mon.0) 14924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:31.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:30 smithi121 bash[36902]: audit 2022-04-22T22:40:30.267204+0000 mon.a (mon.0) 14924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: cluster 2022-04-22T22:40:29.874575+0000 mgr.y (mgr.34107) 10717 : cluster [DBG] pgmap v11372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:30.314133+0000 mgr.y (mgr.34107) 10718 : audit [DBG] from='client.56389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:30.657412+0000 mgr.y (mgr.34107) 10719 : audit [DBG] from='client.46461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:31.082354+0000 mon.a (mon.0) 14925 : audit [DBG] from='client.? 172.21.15.115:0/3297264405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:31.088556+0000 mon.a (mon.0) 14926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:31.091265+0000 mon.a (mon.0) 14927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:31.092433+0000 mon.a (mon.0) 14928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:40:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:31 smithi121 bash[36902]: audit 2022-04-22T22:40:31.440024+0000 mon.a (mon.0) 14929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:40:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: cluster 2022-04-22T22:40:29.874575+0000 mgr.y (mgr.34107) 10717 : cluster [DBG] pgmap v11372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:30.314133+0000 mgr.y (mgr.34107) 10718 : audit [DBG] from='client.56389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:30.657412+0000 mgr.y (mgr.34107) 10719 : audit [DBG] from='client.46461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:31.082354+0000 mon.a (mon.0) 14925 : audit [DBG] from='client.? 172.21.15.115:0/3297264405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:31.088556+0000 mon.a (mon.0) 14926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:40:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:31.091265+0000 mon.a (mon.0) 14927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:40:32.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:31.092433+0000 mon.a (mon.0) 14928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:40:32.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[45495]: audit 2022-04-22T22:40:31.440024+0000 mon.a (mon.0) 14929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:40:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: cluster 2022-04-22T22:40:29.874575+0000 mgr.y (mgr.34107) 10717 : cluster [DBG] pgmap v11372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:32.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:30.314133+0000 mgr.y (mgr.34107) 10718 : audit [DBG] from='client.56389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:30.657412+0000 mgr.y (mgr.34107) 10719 : audit [DBG] from='client.46461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:40:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:31.082354+0000 mon.a (mon.0) 14925 : audit [DBG] from='client.? 172.21.15.115:0/3297264405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:40:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:31.088556+0000 mon.a (mon.0) 14926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:40:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:31.091265+0000 mon.a (mon.0) 14927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:40:32.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:31.092433+0000 mon.a (mon.0) 14928 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:40:32.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:31 smithi115 bash[46934]: audit 2022-04-22T22:40:31.440024+0000 mon.a (mon.0) 14929 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:40:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:32 smithi121 bash[36902]: audit 2022-04-22T22:40:31.888775+0000 mon.a (mon.0) 14930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:32 smithi115 bash[45495]: audit 2022-04-22T22:40:31.888775+0000 mon.a (mon.0) 14930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:32 smithi115 bash[46934]: audit 2022-04-22T22:40:31.888775+0000 mon.a (mon.0) 14930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:32.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:33 smithi121 bash[36902]: cluster 2022-04-22T22:40:31.875672+0000 mgr.y (mgr.34107) 10720 : cluster [DBG] pgmap v11373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:33 smithi121 bash[36902]: audit 2022-04-22T22:40:33.291233+0000 mon.a (mon.0) 14931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:33 smithi115 bash[45495]: cluster 2022-04-22T22:40:31.875672+0000 mgr.y (mgr.34107) 10720 : cluster [DBG] pgmap v11373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:33 smithi115 bash[45495]: audit 2022-04-22T22:40:33.291233+0000 mon.a (mon.0) 14931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:33 smithi115 bash[46934]: cluster 2022-04-22T22:40:31.875672+0000 mgr.y (mgr.34107) 10720 : cluster [DBG] pgmap v11373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:33 smithi115 bash[46934]: audit 2022-04-22T22:40:33.291233+0000 mon.a (mon.0) 14931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:34.711Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:35 smithi121 bash[36902]: cluster 2022-04-22T22:40:33.876847+0000 mgr.y (mgr.34107) 10721 : cluster [DBG] pgmap v11374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:35 smithi121 bash[36902]: audit 2022-04-22T22:40:35.270274+0000 mon.a (mon.0) 14932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:35 smithi115 bash[45495]: cluster 2022-04-22T22:40:33.876847+0000 mgr.y (mgr.34107) 10721 : cluster [DBG] pgmap v11374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:35 smithi115 bash[45495]: audit 2022-04-22T22:40:35.270274+0000 mon.a (mon.0) 14932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:35 smithi115 bash[46934]: cluster 2022-04-22T22:40:33.876847+0000 mgr.y (mgr.34107) 10721 : cluster [DBG] pgmap v11374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:35 smithi115 bash[46934]: audit 2022-04-22T22:40:35.270274+0000 mon.a (mon.0) 14932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:37 smithi121 bash[36902]: cluster 2022-04-22T22:40:35.877761+0000 mgr.y (mgr.34107) 10722 : cluster [DBG] pgmap v11375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:37 smithi121 bash[36902]: audit 2022-04-22T22:40:36.891759+0000 mon.a (mon.0) 14933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:37 smithi115 bash[46934]: cluster 2022-04-22T22:40:35.877761+0000 mgr.y (mgr.34107) 10722 : cluster [DBG] pgmap v11375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:37 smithi115 bash[46934]: audit 2022-04-22T22:40:36.891759+0000 mon.a (mon.0) 14933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:37 smithi115 bash[45495]: cluster 2022-04-22T22:40:35.877761+0000 mgr.y (mgr.34107) 10722 : cluster [DBG] pgmap v11375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:38.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:37 smithi115 bash[45495]: audit 2022-04-22T22:40:36.891759+0000 mon.a (mon.0) 14933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:38 smithi121 bash[36902]: cluster 2022-04-22T22:40:37.878617+0000 mgr.y (mgr.34107) 10723 : cluster [DBG] pgmap v11376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:38 smithi121 bash[36902]: audit 2022-04-22T22:40:38.295396+0000 mon.a (mon.0) 14934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:38 smithi115 bash[45495]: cluster 2022-04-22T22:40:37.878617+0000 mgr.y (mgr.34107) 10723 : cluster [DBG] pgmap v11376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:38 smithi115 bash[45495]: audit 2022-04-22T22:40:38.295396+0000 mon.a (mon.0) 14934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:38 smithi115 bash[46934]: cluster 2022-04-22T22:40:37.878617+0000 mgr.y (mgr.34107) 10723 : cluster [DBG] pgmap v11376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:38 smithi115 bash[46934]: audit 2022-04-22T22:40:38.295396+0000 mon.a (mon.0) 14934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:40 smithi121 bash[36902]: cluster 2022-04-22T22:40:39.879679+0000 mgr.y (mgr.34107) 10724 : cluster [DBG] pgmap v11377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:40 smithi121 bash[36902]: audit 2022-04-22T22:40:40.273242+0000 mon.a (mon.0) 14935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:40 smithi115 bash[45495]: cluster 2022-04-22T22:40:39.879679+0000 mgr.y (mgr.34107) 10724 : cluster [DBG] pgmap v11377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:40 smithi115 bash[45495]: audit 2022-04-22T22:40:40.273242+0000 mon.a (mon.0) 14935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:40 smithi115 bash[46934]: cluster 2022-04-22T22:40:39.879679+0000 mgr.y (mgr.34107) 10724 : cluster [DBG] pgmap v11377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:40 smithi115 bash[46934]: audit 2022-04-22T22:40:40.273242+0000 mon.a (mon.0) 14935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:42 smithi121 bash[36902]: audit 2022-04-22T22:40:41.894662+0000 mon.a (mon.0) 14936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:42 smithi115 bash[45495]: audit 2022-04-22T22:40:41.894662+0000 mon.a (mon.0) 14936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:43.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:42.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:43.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:42 smithi115 bash[46934]: audit 2022-04-22T22:40:41.894662+0000 mon.a (mon.0) 14936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:43 smithi121 bash[36902]: cluster 2022-04-22T22:40:41.880690+0000 mgr.y (mgr.34107) 10725 : cluster [DBG] pgmap v11378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:43 smithi121 bash[36902]: audit 2022-04-22T22:40:43.299022+0000 mon.a (mon.0) 14937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:43 smithi115 bash[45495]: cluster 2022-04-22T22:40:41.880690+0000 mgr.y (mgr.34107) 10725 : cluster [DBG] pgmap v11378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:43 smithi115 bash[45495]: audit 2022-04-22T22:40:43.299022+0000 mon.a (mon.0) 14937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:43 smithi115 bash[46934]: cluster 2022-04-22T22:40:41.880690+0000 mgr.y (mgr.34107) 10725 : cluster [DBG] pgmap v11378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:43 smithi115 bash[46934]: audit 2022-04-22T22:40:43.299022+0000 mon.a (mon.0) 14937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:44.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:45 smithi121 bash[36902]: cluster 2022-04-22T22:40:43.881847+0000 mgr.y (mgr.34107) 10726 : cluster [DBG] pgmap v11379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:45 smithi121 bash[36902]: audit 2022-04-22T22:40:45.276355+0000 mon.a (mon.0) 14938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:45 smithi115 bash[45495]: cluster 2022-04-22T22:40:43.881847+0000 mgr.y (mgr.34107) 10726 : cluster [DBG] pgmap v11379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:45 smithi115 bash[45495]: audit 2022-04-22T22:40:45.276355+0000 mon.a (mon.0) 14938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:45 smithi115 bash[46934]: cluster 2022-04-22T22:40:43.881847+0000 mgr.y (mgr.34107) 10726 : cluster [DBG] pgmap v11379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:45 smithi115 bash[46934]: audit 2022-04-22T22:40:45.276355+0000 mon.a (mon.0) 14938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:47 smithi121 bash[36902]: cluster 2022-04-22T22:40:45.882726+0000 mgr.y (mgr.34107) 10727 : cluster [DBG] pgmap v11380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:47 smithi121 bash[36902]: audit 2022-04-22T22:40:46.897375+0000 mon.a (mon.0) 14939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:47 smithi115 bash[45495]: cluster 2022-04-22T22:40:45.882726+0000 mgr.y (mgr.34107) 10727 : cluster [DBG] pgmap v11380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:47 smithi115 bash[45495]: audit 2022-04-22T22:40:46.897375+0000 mon.a (mon.0) 14939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:47 smithi115 bash[46934]: cluster 2022-04-22T22:40:45.882726+0000 mgr.y (mgr.34107) 10727 : cluster [DBG] pgmap v11380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:48.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:47 smithi115 bash[46934]: audit 2022-04-22T22:40:46.897375+0000 mon.a (mon.0) 14939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:48 smithi121 bash[36902]: audit 2022-04-22T22:40:48.302651+0000 mon.a (mon.0) 14940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:48 smithi115 bash[45495]: audit 2022-04-22T22:40:48.302651+0000 mon.a (mon.0) 14940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:48 smithi115 bash[46934]: audit 2022-04-22T22:40:48.302651+0000 mon.a (mon.0) 14940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:49 smithi121 bash[36902]: cluster 2022-04-22T22:40:47.883805+0000 mgr.y (mgr.34107) 10728 : cluster [DBG] pgmap v11381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:49 smithi115 bash[45495]: cluster 2022-04-22T22:40:47.883805+0000 mgr.y (mgr.34107) 10728 : cluster [DBG] pgmap v11381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:49 smithi115 bash[46934]: cluster 2022-04-22T22:40:47.883805+0000 mgr.y (mgr.34107) 10728 : cluster [DBG] pgmap v11381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:50 smithi121 bash[36902]: audit 2022-04-22T22:40:50.279312+0000 mon.a (mon.0) 14941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:50 smithi115 bash[45495]: audit 2022-04-22T22:40:50.279312+0000 mon.a (mon.0) 14941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:50 smithi115 bash[46934]: audit 2022-04-22T22:40:50.279312+0000 mon.a (mon.0) 14941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:51 smithi121 bash[36902]: cluster 2022-04-22T22:40:49.884980+0000 mgr.y (mgr.34107) 10729 : cluster [DBG] pgmap v11382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:51 smithi115 bash[45495]: cluster 2022-04-22T22:40:49.884980+0000 mgr.y (mgr.34107) 10729 : cluster [DBG] pgmap v11382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:52.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:51 smithi115 bash[46934]: cluster 2022-04-22T22:40:49.884980+0000 mgr.y (mgr.34107) 10729 : cluster [DBG] pgmap v11382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:52 smithi121 bash[36902]: cluster 2022-04-22T22:40:51.886076+0000 mgr.y (mgr.34107) 10730 : cluster [DBG] pgmap v11383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:52 smithi121 bash[36902]: audit 2022-04-22T22:40:51.900663+0000 mon.a (mon.0) 14942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:52.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:52 smithi115 bash[45495]: cluster 2022-04-22T22:40:51.886076+0000 mgr.y (mgr.34107) 10730 : cluster [DBG] pgmap v11383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:52 smithi115 bash[45495]: audit 2022-04-22T22:40:51.900663+0000 mon.a (mon.0) 14942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:52 smithi115 bash[46934]: cluster 2022-04-22T22:40:51.886076+0000 mgr.y (mgr.34107) 10730 : cluster [DBG] pgmap v11383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:53.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:52 smithi115 bash[46934]: audit 2022-04-22T22:40:51.900663+0000 mon.a (mon.0) 14942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:53 smithi121 bash[36902]: audit 2022-04-22T22:40:53.306204+0000 mon.a (mon.0) 14943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:53 smithi115 bash[45495]: audit 2022-04-22T22:40:53.306204+0000 mon.a (mon.0) 14943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:53 smithi115 bash[46934]: audit 2022-04-22T22:40:53.306204+0000 mon.a (mon.0) 14943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:54 smithi115 bash[45495]: cluster 2022-04-22T22:40:53.887016+0000 mgr.y (mgr.34107) 10731 : cluster [DBG] pgmap v11384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:40:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:40:54.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:40:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:54 smithi115 bash[46934]: cluster 2022-04-22T22:40:53.887016+0000 mgr.y (mgr.34107) 10731 : cluster [DBG] pgmap v11384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:54 smithi121 bash[36902]: cluster 2022-04-22T22:40:53.887016+0000 mgr.y (mgr.34107) 10731 : cluster [DBG] pgmap v11384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:55 smithi121 bash[36902]: audit 2022-04-22T22:40:55.282628+0000 mon.a (mon.0) 14944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:55 smithi115 bash[45495]: audit 2022-04-22T22:40:55.282628+0000 mon.a (mon.0) 14944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:55 smithi115 bash[46934]: audit 2022-04-22T22:40:55.282628+0000 mon.a (mon.0) 14944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:40:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:57 smithi121 bash[36902]: cluster 2022-04-22T22:40:55.888109+0000 mgr.y (mgr.34107) 10732 : cluster [DBG] pgmap v11385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:57 smithi121 bash[36902]: audit 2022-04-22T22:40:56.903882+0000 mon.a (mon.0) 14945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:57 smithi115 bash[45495]: cluster 2022-04-22T22:40:55.888109+0000 mgr.y (mgr.34107) 10732 : cluster [DBG] pgmap v11385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:57 smithi115 bash[45495]: audit 2022-04-22T22:40:56.903882+0000 mon.a (mon.0) 14945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:57 smithi115 bash[46934]: cluster 2022-04-22T22:40:55.888109+0000 mgr.y (mgr.34107) 10732 : cluster [DBG] pgmap v11385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:40:58.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:57 smithi115 bash[46934]: audit 2022-04-22T22:40:56.903882+0000 mon.a (mon.0) 14945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:40:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:58 smithi121 bash[36902]: audit 2022-04-22T22:40:58.309781+0000 mon.a (mon.0) 14946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:58 smithi115 bash[45495]: audit 2022-04-22T22:40:58.309781+0000 mon.a (mon.0) 14946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:40:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:58 smithi115 bash[46934]: audit 2022-04-22T22:40:58.309781+0000 mon.a (mon.0) 14946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:40:59 smithi121 bash[36902]: cluster 2022-04-22T22:40:57.889124+0000 mgr.y (mgr.34107) 10733 : cluster [DBG] pgmap v11386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:40:59 smithi115 bash[45495]: cluster 2022-04-22T22:40:57.889124+0000 mgr.y (mgr.34107) 10733 : cluster [DBG] pgmap v11386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:40:59 smithi115 bash[46934]: cluster 2022-04-22T22:40:57.889124+0000 mgr.y (mgr.34107) 10733 : cluster [DBG] pgmap v11386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:00 smithi121 bash[36902]: audit 2022-04-22T22:41:00.285805+0000 mon.a (mon.0) 14947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:01.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:00 smithi115 bash[46934]: audit 2022-04-22T22:41:00.285805+0000 mon.a (mon.0) 14947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:00 smithi115 bash[45495]: audit 2022-04-22T22:41:00.285805+0000 mon.a (mon.0) 14947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:01.456 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:41:01.803 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:41:01.803 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:41:01.803 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:41:01.803 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:41:01.803 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:41:01.804 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:41:01.805 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:41:01.805 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:41:01.805 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:41:01.805 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:41:01.805 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:41:02.212 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:41:02.212 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:41:02.212 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:41:02.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:02.213 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:41:02.214 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:41:02.214 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:41:02.214 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:41:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:01 smithi121 bash[36902]: cluster 2022-04-22T22:40:59.890204+0000 mgr.y (mgr.34107) 10734 : cluster [DBG] pgmap v11387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:01 smithi115 bash[45495]: cluster 2022-04-22T22:40:59.890204+0000 mgr.y (mgr.34107) 10734 : cluster [DBG] pgmap v11387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:01 smithi115 bash[46934]: cluster 2022-04-22T22:40:59.890204+0000 mgr.y (mgr.34107) 10734 : cluster [DBG] pgmap v11387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:02 smithi121 bash[36902]: audit 2022-04-22T22:41:01.439285+0000 mgr.y (mgr.34107) 10735 : audit [DBG] from='client.56407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:02 smithi121 bash[36902]: audit 2022-04-22T22:41:01.779871+0000 mgr.y (mgr.34107) 10736 : audit [DBG] from='client.56413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:02 smithi121 bash[36902]: audit 2022-04-22T22:41:01.906824+0000 mon.a (mon.0) 14948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:02 smithi121 bash[36902]: audit 2022-04-22T22:41:02.208643+0000 mon.c (mon.1) 362 : audit [DBG] from='client.? 172.21.15.115:0/740749259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:02.853Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[45495]: audit 2022-04-22T22:41:01.439285+0000 mgr.y (mgr.34107) 10735 : audit [DBG] from='client.56407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[45495]: audit 2022-04-22T22:41:01.779871+0000 mgr.y (mgr.34107) 10736 : audit [DBG] from='client.56413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[45495]: audit 2022-04-22T22:41:01.906824+0000 mon.a (mon.0) 14948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[45495]: audit 2022-04-22T22:41:02.208643+0000 mon.c (mon.1) 362 : audit [DBG] from='client.? 172.21.15.115:0/740749259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[46934]: audit 2022-04-22T22:41:01.439285+0000 mgr.y (mgr.34107) 10735 : audit [DBG] from='client.56407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[46934]: audit 2022-04-22T22:41:01.779871+0000 mgr.y (mgr.34107) 10736 : audit [DBG] from='client.56413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[46934]: audit 2022-04-22T22:41:01.906824+0000 mon.a (mon.0) 14948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:02 smithi115 bash[46934]: audit 2022-04-22T22:41:02.208643+0000 mon.c (mon.1) 362 : audit [DBG] from='client.? 172.21.15.115:0/740749259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:03 smithi121 bash[36902]: cluster 2022-04-22T22:41:01.890952+0000 mgr.y (mgr.34107) 10737 : cluster [DBG] pgmap v11388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:03 smithi121 bash[36902]: audit 2022-04-22T22:41:03.313124+0000 mon.a (mon.0) 14949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:03 smithi115 bash[45495]: cluster 2022-04-22T22:41:01.890952+0000 mgr.y (mgr.34107) 10737 : cluster [DBG] pgmap v11388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:03 smithi115 bash[45495]: audit 2022-04-22T22:41:03.313124+0000 mon.a (mon.0) 14949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:03 smithi115 bash[46934]: cluster 2022-04-22T22:41:01.890952+0000 mgr.y (mgr.34107) 10737 : cluster [DBG] pgmap v11388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:03 smithi115 bash[46934]: audit 2022-04-22T22:41:03.313124+0000 mon.a (mon.0) 14949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:04.713Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:05 smithi121 bash[36902]: cluster 2022-04-22T22:41:03.892023+0000 mgr.y (mgr.34107) 10738 : cluster [DBG] pgmap v11389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:05 smithi121 bash[36902]: audit 2022-04-22T22:41:05.288816+0000 mon.a (mon.0) 14950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:05 smithi115 bash[45495]: cluster 2022-04-22T22:41:03.892023+0000 mgr.y (mgr.34107) 10738 : cluster [DBG] pgmap v11389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:05 smithi115 bash[45495]: audit 2022-04-22T22:41:05.288816+0000 mon.a (mon.0) 14950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:05 smithi115 bash[46934]: cluster 2022-04-22T22:41:03.892023+0000 mgr.y (mgr.34107) 10738 : cluster [DBG] pgmap v11389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:05 smithi115 bash[46934]: audit 2022-04-22T22:41:05.288816+0000 mon.a (mon.0) 14950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:07 smithi121 bash[36902]: cluster 2022-04-22T22:41:05.892978+0000 mgr.y (mgr.34107) 10739 : cluster [DBG] pgmap v11390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:07 smithi121 bash[36902]: audit 2022-04-22T22:41:06.909764+0000 mon.a (mon.0) 14951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:08.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:07 smithi115 bash[46934]: cluster 2022-04-22T22:41:05.892978+0000 mgr.y (mgr.34107) 10739 : cluster [DBG] pgmap v11390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:07 smithi115 bash[46934]: audit 2022-04-22T22:41:06.909764+0000 mon.a (mon.0) 14951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:07 smithi115 bash[45495]: cluster 2022-04-22T22:41:05.892978+0000 mgr.y (mgr.34107) 10739 : cluster [DBG] pgmap v11390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:07 smithi115 bash[45495]: audit 2022-04-22T22:41:06.909764+0000 mon.a (mon.0) 14951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:08 smithi121 bash[36902]: cluster 2022-04-22T22:41:07.894070+0000 mgr.y (mgr.34107) 10740 : cluster [DBG] pgmap v11391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:08 smithi121 bash[36902]: audit 2022-04-22T22:41:08.316924+0000 mon.a (mon.0) 14952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:09.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:08 smithi115 bash[46934]: cluster 2022-04-22T22:41:07.894070+0000 mgr.y (mgr.34107) 10740 : cluster [DBG] pgmap v11391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:08 smithi115 bash[46934]: audit 2022-04-22T22:41:08.316924+0000 mon.a (mon.0) 14952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:08 smithi115 bash[45495]: cluster 2022-04-22T22:41:07.894070+0000 mgr.y (mgr.34107) 10740 : cluster [DBG] pgmap v11391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:08 smithi115 bash[45495]: audit 2022-04-22T22:41:08.316924+0000 mon.a (mon.0) 14952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:10 smithi121 bash[36902]: cluster 2022-04-22T22:41:09.895167+0000 mgr.y (mgr.34107) 10741 : cluster [DBG] pgmap v11392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:10 smithi121 bash[36902]: audit 2022-04-22T22:41:10.291689+0000 mon.a (mon.0) 14953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:10 smithi115 bash[46934]: cluster 2022-04-22T22:41:09.895167+0000 mgr.y (mgr.34107) 10741 : cluster [DBG] pgmap v11392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:10 smithi115 bash[46934]: audit 2022-04-22T22:41:10.291689+0000 mon.a (mon.0) 14953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:10 smithi115 bash[45495]: cluster 2022-04-22T22:41:09.895167+0000 mgr.y (mgr.34107) 10741 : cluster [DBG] pgmap v11392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:11.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:10 smithi115 bash[45495]: audit 2022-04-22T22:41:10.291689+0000 mon.a (mon.0) 14953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:12 smithi121 bash[36902]: audit 2022-04-22T22:41:11.912609+0000 mon.a (mon.0) 14954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:12 smithi115 bash[45495]: audit 2022-04-22T22:41:11.912609+0000 mon.a (mon.0) 14954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:12.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:12 smithi115 bash[46934]: audit 2022-04-22T22:41:11.912609+0000 mon.a (mon.0) 14954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:13 smithi121 bash[36902]: cluster 2022-04-22T22:41:11.896295+0000 mgr.y (mgr.34107) 10742 : cluster [DBG] pgmap v11393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:13 smithi121 bash[36902]: audit 2022-04-22T22:41:13.320740+0000 mon.a (mon.0) 14955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:13 smithi115 bash[45495]: cluster 2022-04-22T22:41:11.896295+0000 mgr.y (mgr.34107) 10742 : cluster [DBG] pgmap v11393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:13 smithi115 bash[45495]: audit 2022-04-22T22:41:13.320740+0000 mon.a (mon.0) 14955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:13 smithi115 bash[46934]: cluster 2022-04-22T22:41:11.896295+0000 mgr.y (mgr.34107) 10742 : cluster [DBG] pgmap v11393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:14.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:13 smithi115 bash[46934]: audit 2022-04-22T22:41:13.320740+0000 mon.a (mon.0) 14955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:14.712Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:15 smithi121 bash[36902]: cluster 2022-04-22T22:41:13.897403+0000 mgr.y (mgr.34107) 10743 : cluster [DBG] pgmap v11394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:15 smithi121 bash[36902]: audit 2022-04-22T22:41:15.294615+0000 mon.a (mon.0) 14956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:16.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:15 smithi115 bash[46934]: cluster 2022-04-22T22:41:13.897403+0000 mgr.y (mgr.34107) 10743 : cluster [DBG] pgmap v11394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:15 smithi115 bash[46934]: audit 2022-04-22T22:41:15.294615+0000 mon.a (mon.0) 14956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:15 smithi115 bash[45495]: cluster 2022-04-22T22:41:13.897403+0000 mgr.y (mgr.34107) 10743 : cluster [DBG] pgmap v11394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:15 smithi115 bash[45495]: audit 2022-04-22T22:41:15.294615+0000 mon.a (mon.0) 14956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:17 smithi121 bash[36902]: cluster 2022-04-22T22:41:15.898502+0000 mgr.y (mgr.34107) 10744 : cluster [DBG] pgmap v11395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:17 smithi121 bash[36902]: audit 2022-04-22T22:41:16.905038+0000 mon.a (mon.0) 14957 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:41:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:17 smithi121 bash[36902]: audit 2022-04-22T22:41:16.905509+0000 mon.a (mon.0) 14958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:41:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:17 smithi121 bash[36902]: audit 2022-04-22T22:41:16.915427+0000 mon.a (mon.0) 14959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[46934]: cluster 2022-04-22T22:41:15.898502+0000 mgr.y (mgr.34107) 10744 : cluster [DBG] pgmap v11395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[46934]: audit 2022-04-22T22:41:16.905038+0000 mon.a (mon.0) 14957 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[46934]: audit 2022-04-22T22:41:16.905509+0000 mon.a (mon.0) 14958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[46934]: audit 2022-04-22T22:41:16.915427+0000 mon.a (mon.0) 14959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[45495]: cluster 2022-04-22T22:41:15.898502+0000 mgr.y (mgr.34107) 10744 : cluster [DBG] pgmap v11395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[45495]: audit 2022-04-22T22:41:16.905038+0000 mon.a (mon.0) 14957 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:41:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[45495]: audit 2022-04-22T22:41:16.905509+0000 mon.a (mon.0) 14958 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:41:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:17 smithi115 bash[45495]: audit 2022-04-22T22:41:16.915427+0000 mon.a (mon.0) 14959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:18 smithi121 bash[36902]: audit 2022-04-22T22:41:18.324342+0000 mon.a (mon.0) 14960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:19.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:18 smithi115 bash[46934]: audit 2022-04-22T22:41:18.324342+0000 mon.a (mon.0) 14960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:19.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:18 smithi115 bash[45495]: audit 2022-04-22T22:41:18.324342+0000 mon.a (mon.0) 14960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:19 smithi121 bash[36902]: cluster 2022-04-22T22:41:17.899390+0000 mgr.y (mgr.34107) 10745 : cluster [DBG] pgmap v11396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:19 smithi115 bash[45495]: cluster 2022-04-22T22:41:17.899390+0000 mgr.y (mgr.34107) 10745 : cluster [DBG] pgmap v11396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:19 smithi115 bash[46934]: cluster 2022-04-22T22:41:17.899390+0000 mgr.y (mgr.34107) 10745 : cluster [DBG] pgmap v11396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:20 smithi121 bash[36902]: audit 2022-04-22T22:41:20.297697+0000 mon.a (mon.0) 14961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:20 smithi115 bash[45495]: audit 2022-04-22T22:41:20.297697+0000 mon.a (mon.0) 14961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:20 smithi115 bash[46934]: audit 2022-04-22T22:41:20.297697+0000 mon.a (mon.0) 14961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:21 smithi121 bash[36902]: cluster 2022-04-22T22:41:19.900447+0000 mgr.y (mgr.34107) 10746 : cluster [DBG] pgmap v11397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:21 smithi115 bash[45495]: cluster 2022-04-22T22:41:19.900447+0000 mgr.y (mgr.34107) 10746 : cluster [DBG] pgmap v11397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:21 smithi115 bash[46934]: cluster 2022-04-22T22:41:19.900447+0000 mgr.y (mgr.34107) 10746 : cluster [DBG] pgmap v11397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:22 smithi121 bash[36902]: audit 2022-04-22T22:41:21.917957+0000 mon.a (mon.0) 14962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:22 smithi115 bash[45495]: audit 2022-04-22T22:41:21.917957+0000 mon.a (mon.0) 14962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:22.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:22 smithi115 bash[46934]: audit 2022-04-22T22:41:21.917957+0000 mon.a (mon.0) 14962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:23 smithi121 bash[36902]: cluster 2022-04-22T22:41:21.901571+0000 mgr.y (mgr.34107) 10747 : cluster [DBG] pgmap v11398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:23 smithi121 bash[36902]: audit 2022-04-22T22:41:23.328063+0000 mon.a (mon.0) 14963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:24.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:23 smithi115 bash[46934]: cluster 2022-04-22T22:41:21.901571+0000 mgr.y (mgr.34107) 10747 : cluster [DBG] pgmap v11398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:23 smithi115 bash[46934]: audit 2022-04-22T22:41:23.328063+0000 mon.a (mon.0) 14963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:23 smithi115 bash[45495]: cluster 2022-04-22T22:41:21.901571+0000 mgr.y (mgr.34107) 10747 : cluster [DBG] pgmap v11398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:23 smithi115 bash[45495]: audit 2022-04-22T22:41:23.328063+0000 mon.a (mon.0) 14963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:24.716Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:24 smithi115 bash[45495]: cluster 2022-04-22T22:41:23.902725+0000 mgr.y (mgr.34107) 10748 : cluster [DBG] pgmap v11399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:24 smithi115 bash[46934]: cluster 2022-04-22T22:41:23.902725+0000 mgr.y (mgr.34107) 10748 : cluster [DBG] pgmap v11399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:24 smithi121 bash[36902]: cluster 2022-04-22T22:41:23.902725+0000 mgr.y (mgr.34107) 10748 : cluster [DBG] pgmap v11399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:25 smithi121 bash[36902]: audit 2022-04-22T22:41:25.300605+0000 mon.a (mon.0) 14964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:25 smithi115 bash[45495]: audit 2022-04-22T22:41:25.300605+0000 mon.a (mon.0) 14964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:25 smithi115 bash[46934]: audit 2022-04-22T22:41:25.300605+0000 mon.a (mon.0) 14964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:27 smithi121 bash[36902]: cluster 2022-04-22T22:41:25.903627+0000 mgr.y (mgr.34107) 10749 : cluster [DBG] pgmap v11400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:27 smithi121 bash[36902]: audit 2022-04-22T22:41:26.921017+0000 mon.a (mon.0) 14965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:27 smithi115 bash[45495]: cluster 2022-04-22T22:41:25.903627+0000 mgr.y (mgr.34107) 10749 : cluster [DBG] pgmap v11400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:27 smithi115 bash[45495]: audit 2022-04-22T22:41:26.921017+0000 mon.a (mon.0) 14965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:27 smithi115 bash[46934]: cluster 2022-04-22T22:41:25.903627+0000 mgr.y (mgr.34107) 10749 : cluster [DBG] pgmap v11400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:27 smithi115 bash[46934]: audit 2022-04-22T22:41:26.921017+0000 mon.a (mon.0) 14965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:28 smithi121 bash[36902]: audit 2022-04-22T22:41:28.331770+0000 mon.a (mon.0) 14966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:29.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:28 smithi115 bash[46934]: audit 2022-04-22T22:41:28.331770+0000 mon.a (mon.0) 14966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:28 smithi115 bash[45495]: audit 2022-04-22T22:41:28.331770+0000 mon.a (mon.0) 14966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:29 smithi121 bash[36902]: cluster 2022-04-22T22:41:27.904736+0000 mgr.y (mgr.34107) 10750 : cluster [DBG] pgmap v11401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:30.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:29 smithi115 bash[46934]: cluster 2022-04-22T22:41:27.904736+0000 mgr.y (mgr.34107) 10750 : cluster [DBG] pgmap v11401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:29 smithi115 bash[45495]: cluster 2022-04-22T22:41:27.904736+0000 mgr.y (mgr.34107) 10750 : cluster [DBG] pgmap v11401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:31.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:30 smithi121 bash[36902]: audit 2022-04-22T22:41:30.303699+0000 mon.a (mon.0) 14967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:30 smithi115 bash[45495]: audit 2022-04-22T22:41:30.303699+0000 mon.a (mon.0) 14967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:30 smithi115 bash[46934]: audit 2022-04-22T22:41:30.303699+0000 mon.a (mon.0) 14967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[45495]: cluster 2022-04-22T22:41:29.905808+0000 mgr.y (mgr.34107) 10751 : cluster [DBG] pgmap v11402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[45495]: audit 2022-04-22T22:41:31.447767+0000 mon.a (mon.0) 14968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:41:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[45495]: audit 2022-04-22T22:41:31.450837+0000 mon.a (mon.0) 14969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:41:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[45495]: audit 2022-04-22T22:41:31.452822+0000 mon.a (mon.0) 14970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[45495]: audit 2022-04-22T22:41:31.816872+0000 mon.a (mon.0) 14971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[46934]: cluster 2022-04-22T22:41:29.905808+0000 mgr.y (mgr.34107) 10751 : cluster [DBG] pgmap v11402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[46934]: audit 2022-04-22T22:41:31.447767+0000 mon.a (mon.0) 14968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[46934]: audit 2022-04-22T22:41:31.450837+0000 mon.a (mon.0) 14969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[46934]: audit 2022-04-22T22:41:31.452822+0000 mon.a (mon.0) 14970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:41:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:31 smithi115 bash[46934]: audit 2022-04-22T22:41:31.816872+0000 mon.a (mon.0) 14971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:41:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:31 smithi121 bash[36902]: cluster 2022-04-22T22:41:29.905808+0000 mgr.y (mgr.34107) 10751 : cluster [DBG] pgmap v11402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:31 smithi121 bash[36902]: audit 2022-04-22T22:41:31.447767+0000 mon.a (mon.0) 14968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:41:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:31 smithi121 bash[36902]: audit 2022-04-22T22:41:31.450837+0000 mon.a (mon.0) 14969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:41:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:31 smithi121 bash[36902]: audit 2022-04-22T22:41:31.452822+0000 mon.a (mon.0) 14970 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:41:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:31 smithi121 bash[36902]: audit 2022-04-22T22:41:31.816872+0000 mon.a (mon.0) 14971 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:41:32.591 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:41:32.935 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:41:32.935 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:41:32.936 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:41:32.937 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:41:32.938 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:41:32.938 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:41:32.938 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:41:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:32 smithi121 bash[36902]: audit 2022-04-22T22:41:31.924085+0000 mon.a (mon.0) 14972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:32 smithi115 bash[45495]: audit 2022-04-22T22:41:31.924085+0000 mon.a (mon.0) 14972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:32 smithi115 bash[46934]: audit 2022-04-22T22:41:31.924085+0000 mon.a (mon.0) 14972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:32.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:41:33.342 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:41:33.343 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:41:33.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:41:33.344 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:41:33.344 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:41:33.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:41:33.345 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:41:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:33 smithi121 bash[36902]: cluster 2022-04-22T22:41:31.906998+0000 mgr.y (mgr.34107) 10752 : cluster [DBG] pgmap v11403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:33 smithi121 bash[36902]: audit 2022-04-22T22:41:32.570275+0000 mgr.y (mgr.34107) 10753 : audit [DBG] from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:33 smithi121 bash[36902]: audit 2022-04-22T22:41:32.910512+0000 mgr.y (mgr.34107) 10754 : audit [DBG] from='client.56431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:33 smithi121 bash[36902]: audit 2022-04-22T22:41:33.335416+0000 mon.a (mon.0) 14973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:33 smithi121 bash[36902]: audit 2022-04-22T22:41:33.338124+0000 mon.c (mon.1) 363 : audit [DBG] from='client.? 172.21.15.115:0/1797497861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[45495]: cluster 2022-04-22T22:41:31.906998+0000 mgr.y (mgr.34107) 10752 : cluster [DBG] pgmap v11403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[45495]: audit 2022-04-22T22:41:32.570275+0000 mgr.y (mgr.34107) 10753 : audit [DBG] from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[45495]: audit 2022-04-22T22:41:32.910512+0000 mgr.y (mgr.34107) 10754 : audit [DBG] from='client.56431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[45495]: audit 2022-04-22T22:41:33.335416+0000 mon.a (mon.0) 14973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:34.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[45495]: audit 2022-04-22T22:41:33.338124+0000 mon.c (mon.1) 363 : audit [DBG] from='client.? 172.21.15.115:0/1797497861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[46934]: cluster 2022-04-22T22:41:31.906998+0000 mgr.y (mgr.34107) 10752 : cluster [DBG] pgmap v11403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[46934]: audit 2022-04-22T22:41:32.570275+0000 mgr.y (mgr.34107) 10753 : audit [DBG] from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[46934]: audit 2022-04-22T22:41:32.910512+0000 mgr.y (mgr.34107) 10754 : audit [DBG] from='client.56431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:41:34.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[46934]: audit 2022-04-22T22:41:33.335416+0000 mon.a (mon.0) 14973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:34.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:33 smithi115 bash[46934]: audit 2022-04-22T22:41:33.338124+0000 mon.c (mon.1) 363 : audit [DBG] from='client.? 172.21.15.115:0/1797497861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:41:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:34 smithi115 bash[45495]: cluster 2022-04-22T22:41:33.908130+0000 mgr.y (mgr.34107) 10755 : cluster [DBG] pgmap v11404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:34.715Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:34 smithi115 bash[46934]: cluster 2022-04-22T22:41:33.908130+0000 mgr.y (mgr.34107) 10755 : cluster [DBG] pgmap v11404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:34 smithi121 bash[36902]: cluster 2022-04-22T22:41:33.908130+0000 mgr.y (mgr.34107) 10755 : cluster [DBG] pgmap v11404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:35 smithi121 bash[36902]: audit 2022-04-22T22:41:35.306805+0000 mon.a (mon.0) 14974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:35 smithi115 bash[45495]: audit 2022-04-22T22:41:35.306805+0000 mon.a (mon.0) 14974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:35 smithi115 bash[46934]: audit 2022-04-22T22:41:35.306805+0000 mon.a (mon.0) 14974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:37 smithi121 bash[36902]: cluster 2022-04-22T22:41:35.909345+0000 mgr.y (mgr.34107) 10756 : cluster [DBG] pgmap v11405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:37 smithi121 bash[36902]: audit 2022-04-22T22:41:36.927036+0000 mon.a (mon.0) 14975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:37 smithi115 bash[45495]: cluster 2022-04-22T22:41:35.909345+0000 mgr.y (mgr.34107) 10756 : cluster [DBG] pgmap v11405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:37 smithi115 bash[45495]: audit 2022-04-22T22:41:36.927036+0000 mon.a (mon.0) 14975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:37 smithi115 bash[46934]: cluster 2022-04-22T22:41:35.909345+0000 mgr.y (mgr.34107) 10756 : cluster [DBG] pgmap v11405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:37 smithi115 bash[46934]: audit 2022-04-22T22:41:36.927036+0000 mon.a (mon.0) 14975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:38 smithi121 bash[36902]: audit 2022-04-22T22:41:38.338914+0000 mon.a (mon.0) 14976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:38 smithi115 bash[45495]: audit 2022-04-22T22:41:38.338914+0000 mon.a (mon.0) 14976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:38 smithi115 bash[46934]: audit 2022-04-22T22:41:38.338914+0000 mon.a (mon.0) 14976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:39 smithi121 bash[36902]: cluster 2022-04-22T22:41:37.910357+0000 mgr.y (mgr.34107) 10757 : cluster [DBG] pgmap v11406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:39 smithi115 bash[45495]: cluster 2022-04-22T22:41:37.910357+0000 mgr.y (mgr.34107) 10757 : cluster [DBG] pgmap v11406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:39 smithi115 bash[46934]: cluster 2022-04-22T22:41:37.910357+0000 mgr.y (mgr.34107) 10757 : cluster [DBG] pgmap v11406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:40 smithi121 bash[36902]: audit 2022-04-22T22:41:40.309937+0000 mon.a (mon.0) 14977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:40 smithi115 bash[45495]: audit 2022-04-22T22:41:40.309937+0000 mon.a (mon.0) 14977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:40 smithi115 bash[46934]: audit 2022-04-22T22:41:40.309937+0000 mon.a (mon.0) 14977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:41 smithi121 bash[36902]: cluster 2022-04-22T22:41:39.911403+0000 mgr.y (mgr.34107) 10758 : cluster [DBG] pgmap v11407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:41 smithi115 bash[45495]: cluster 2022-04-22T22:41:39.911403+0000 mgr.y (mgr.34107) 10758 : cluster [DBG] pgmap v11407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:41 smithi115 bash[46934]: cluster 2022-04-22T22:41:39.911403+0000 mgr.y (mgr.34107) 10758 : cluster [DBG] pgmap v11407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:42 smithi121 bash[36902]: audit 2022-04-22T22:41:41.930239+0000 mon.a (mon.0) 14978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:42 smithi115 bash[45495]: audit 2022-04-22T22:41:41.930239+0000 mon.a (mon.0) 14978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:42.852Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:42 smithi115 bash[46934]: audit 2022-04-22T22:41:41.930239+0000 mon.a (mon.0) 14978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:43 smithi121 bash[36902]: cluster 2022-04-22T22:41:41.912457+0000 mgr.y (mgr.34107) 10759 : cluster [DBG] pgmap v11408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:43 smithi121 bash[36902]: audit 2022-04-22T22:41:43.342807+0000 mon.a (mon.0) 14979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:43 smithi115 bash[45495]: cluster 2022-04-22T22:41:41.912457+0000 mgr.y (mgr.34107) 10759 : cluster [DBG] pgmap v11408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:43 smithi115 bash[45495]: audit 2022-04-22T22:41:43.342807+0000 mon.a (mon.0) 14979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:43 smithi115 bash[46934]: cluster 2022-04-22T22:41:41.912457+0000 mgr.y (mgr.34107) 10759 : cluster [DBG] pgmap v11408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:43 smithi115 bash[46934]: audit 2022-04-22T22:41:43.342807+0000 mon.a (mon.0) 14979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:44 smithi115 bash[45495]: cluster 2022-04-22T22:41:43.913609+0000 mgr.y (mgr.34107) 10760 : cluster [DBG] pgmap v11409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:44.718Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:44 smithi115 bash[46934]: cluster 2022-04-22T22:41:43.913609+0000 mgr.y (mgr.34107) 10760 : cluster [DBG] pgmap v11409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:44 smithi121 bash[36902]: cluster 2022-04-22T22:41:43.913609+0000 mgr.y (mgr.34107) 10760 : cluster [DBG] pgmap v11409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:45 smithi121 bash[36902]: audit 2022-04-22T22:41:45.312912+0000 mon.a (mon.0) 14980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:45 smithi115 bash[45495]: audit 2022-04-22T22:41:45.312912+0000 mon.a (mon.0) 14980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:45 smithi115 bash[46934]: audit 2022-04-22T22:41:45.312912+0000 mon.a (mon.0) 14980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:47 smithi121 bash[36902]: cluster 2022-04-22T22:41:45.914503+0000 mgr.y (mgr.34107) 10761 : cluster [DBG] pgmap v11410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:47 smithi121 bash[36902]: audit 2022-04-22T22:41:46.933186+0000 mon.a (mon.0) 14981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:47 smithi115 bash[45495]: cluster 2022-04-22T22:41:45.914503+0000 mgr.y (mgr.34107) 10761 : cluster [DBG] pgmap v11410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:47 smithi115 bash[45495]: audit 2022-04-22T22:41:46.933186+0000 mon.a (mon.0) 14981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:47 smithi115 bash[46934]: cluster 2022-04-22T22:41:45.914503+0000 mgr.y (mgr.34107) 10761 : cluster [DBG] pgmap v11410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:47 smithi115 bash[46934]: audit 2022-04-22T22:41:46.933186+0000 mon.a (mon.0) 14981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:48 smithi121 bash[36902]: audit 2022-04-22T22:41:48.346448+0000 mon.a (mon.0) 14982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:48 smithi115 bash[45495]: audit 2022-04-22T22:41:48.346448+0000 mon.a (mon.0) 14982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:48 smithi115 bash[46934]: audit 2022-04-22T22:41:48.346448+0000 mon.a (mon.0) 14982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:49 smithi121 bash[36902]: cluster 2022-04-22T22:41:47.915638+0000 mgr.y (mgr.34107) 10762 : cluster [DBG] pgmap v11411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:49 smithi115 bash[45495]: cluster 2022-04-22T22:41:47.915638+0000 mgr.y (mgr.34107) 10762 : cluster [DBG] pgmap v11411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:49 smithi115 bash[46934]: cluster 2022-04-22T22:41:47.915638+0000 mgr.y (mgr.34107) 10762 : cluster [DBG] pgmap v11411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:50 smithi121 bash[36902]: audit 2022-04-22T22:41:50.316002+0000 mon.a (mon.0) 14983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:50 smithi115 bash[45495]: audit 2022-04-22T22:41:50.316002+0000 mon.a (mon.0) 14983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:50 smithi115 bash[46934]: audit 2022-04-22T22:41:50.316002+0000 mon.a (mon.0) 14983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:51 smithi121 bash[36902]: cluster 2022-04-22T22:41:49.916764+0000 mgr.y (mgr.34107) 10763 : cluster [DBG] pgmap v11412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:51 smithi115 bash[45495]: cluster 2022-04-22T22:41:49.916764+0000 mgr.y (mgr.34107) 10763 : cluster [DBG] pgmap v11412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:51 smithi115 bash[46934]: cluster 2022-04-22T22:41:49.916764+0000 mgr.y (mgr.34107) 10763 : cluster [DBG] pgmap v11412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:52 smithi121 bash[36902]: audit 2022-04-22T22:41:51.936018+0000 mon.a (mon.0) 14984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:52 smithi115 bash[45495]: audit 2022-04-22T22:41:51.936018+0000 mon.a (mon.0) 14984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:52.855Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:52 smithi115 bash[46934]: audit 2022-04-22T22:41:51.936018+0000 mon.a (mon.0) 14984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:53 smithi121 bash[36902]: cluster 2022-04-22T22:41:51.917826+0000 mgr.y (mgr.34107) 10764 : cluster [DBG] pgmap v11413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:53 smithi121 bash[36902]: audit 2022-04-22T22:41:53.350478+0000 mon.a (mon.0) 14985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:53 smithi115 bash[45495]: cluster 2022-04-22T22:41:51.917826+0000 mgr.y (mgr.34107) 10764 : cluster [DBG] pgmap v11413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:53 smithi115 bash[45495]: audit 2022-04-22T22:41:53.350478+0000 mon.a (mon.0) 14985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:53 smithi115 bash[46934]: cluster 2022-04-22T22:41:51.917826+0000 mgr.y (mgr.34107) 10764 : cluster [DBG] pgmap v11413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:53 smithi115 bash[46934]: audit 2022-04-22T22:41:53.350478+0000 mon.a (mon.0) 14985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:54 smithi115 bash[45495]: cluster 2022-04-22T22:41:53.919043+0000 mgr.y (mgr.34107) 10765 : cluster [DBG] pgmap v11414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:41:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:41:54.717Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:41:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:54 smithi115 bash[46934]: cluster 2022-04-22T22:41:53.919043+0000 mgr.y (mgr.34107) 10765 : cluster [DBG] pgmap v11414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:54 smithi121 bash[36902]: cluster 2022-04-22T22:41:53.919043+0000 mgr.y (mgr.34107) 10765 : cluster [DBG] pgmap v11414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:55 smithi121 bash[36902]: audit 2022-04-22T22:41:55.319261+0000 mon.a (mon.0) 14986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:55 smithi115 bash[46934]: audit 2022-04-22T22:41:55.319261+0000 mon.a (mon.0) 14986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:55 smithi115 bash[45495]: audit 2022-04-22T22:41:55.319261+0000 mon.a (mon.0) 14986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:41:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:57 smithi121 bash[36902]: cluster 2022-04-22T22:41:55.920199+0000 mgr.y (mgr.34107) 10766 : cluster [DBG] pgmap v11415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:57 smithi121 bash[36902]: audit 2022-04-22T22:41:56.939069+0000 mon.a (mon.0) 14987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:57 smithi115 bash[45495]: cluster 2022-04-22T22:41:55.920199+0000 mgr.y (mgr.34107) 10766 : cluster [DBG] pgmap v11415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:57 smithi115 bash[45495]: audit 2022-04-22T22:41:56.939069+0000 mon.a (mon.0) 14987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:57 smithi115 bash[46934]: cluster 2022-04-22T22:41:55.920199+0000 mgr.y (mgr.34107) 10766 : cluster [DBG] pgmap v11415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:41:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:57 smithi115 bash[46934]: audit 2022-04-22T22:41:56.939069+0000 mon.a (mon.0) 14987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:41:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:58 smithi121 bash[36902]: audit 2022-04-22T22:41:58.354260+0000 mon.a (mon.0) 14988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:58 smithi115 bash[45495]: audit 2022-04-22T22:41:58.354260+0000 mon.a (mon.0) 14988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:41:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:58 smithi115 bash[46934]: audit 2022-04-22T22:41:58.354260+0000 mon.a (mon.0) 14988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:41:59 smithi121 bash[36902]: cluster 2022-04-22T22:41:57.921256+0000 mgr.y (mgr.34107) 10767 : cluster [DBG] pgmap v11416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:41:59 smithi115 bash[45495]: cluster 2022-04-22T22:41:57.921256+0000 mgr.y (mgr.34107) 10767 : cluster [DBG] pgmap v11416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:41:59 smithi115 bash[46934]: cluster 2022-04-22T22:41:57.921256+0000 mgr.y (mgr.34107) 10767 : cluster [DBG] pgmap v11416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:00 smithi121 bash[36902]: audit 2022-04-22T22:42:00.322084+0000 mon.a (mon.0) 14989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:01.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:00 smithi115 bash[45495]: audit 2022-04-22T22:42:00.322084+0000 mon.a (mon.0) 14989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:00 smithi115 bash[46934]: audit 2022-04-22T22:42:00.322084+0000 mon.a (mon.0) 14989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:01 smithi121 bash[36902]: cluster 2022-04-22T22:41:59.922365+0000 mgr.y (mgr.34107) 10768 : cluster [DBG] pgmap v11417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:02.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:01 smithi115 bash[45495]: cluster 2022-04-22T22:41:59.922365+0000 mgr.y (mgr.34107) 10768 : cluster [DBG] pgmap v11417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:02.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:01 smithi115 bash[46934]: cluster 2022-04-22T22:41:59.922365+0000 mgr.y (mgr.34107) 10768 : cluster [DBG] pgmap v11417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:02 smithi121 bash[36902]: audit 2022-04-22T22:42:01.942028+0000 mon.a (mon.0) 14990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:02 smithi115 bash[45495]: audit 2022-04-22T22:42:01.942028+0000 mon.a (mon.0) 14990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:02.855Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:02 smithi115 bash[46934]: audit 2022-04-22T22:42:01.942028+0000 mon.a (mon.0) 14990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:03.722 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:42:04.071 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:42:04.071 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:42:04.071 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:42:04.071 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:42:04.072 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:42:04.073 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:42:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:03 smithi121 bash[36902]: cluster 2022-04-22T22:42:01.923469+0000 mgr.y (mgr.34107) 10769 : cluster [DBG] pgmap v11418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:03 smithi121 bash[36902]: audit 2022-04-22T22:42:03.358075+0000 mon.a (mon.0) 14991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:03 smithi121 bash[36902]: audit 2022-04-22T22:42:03.701366+0000 mgr.y (mgr.34107) 10770 : audit [DBG] from='client.46506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[45495]: cluster 2022-04-22T22:42:01.923469+0000 mgr.y (mgr.34107) 10769 : cluster [DBG] pgmap v11418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[45495]: audit 2022-04-22T22:42:03.358075+0000 mon.a (mon.0) 14991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[45495]: audit 2022-04-22T22:42:03.701366+0000 mgr.y (mgr.34107) 10770 : audit [DBG] from='client.46506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[46934]: cluster 2022-04-22T22:42:01.923469+0000 mgr.y (mgr.34107) 10769 : cluster [DBG] pgmap v11418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[46934]: audit 2022-04-22T22:42:03.358075+0000 mon.a (mon.0) 14991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:03 smithi115 bash[46934]: audit 2022-04-22T22:42:03.701366+0000 mgr.y (mgr.34107) 10770 : audit [DBG] from='client.46506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:04.472 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:42:04.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:42:04.474 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:42:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:04.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[45495]: cluster 2022-04-22T22:42:03.924542+0000 mgr.y (mgr.34107) 10771 : cluster [DBG] pgmap v11419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[45495]: audit 2022-04-22T22:42:04.048575+0000 mgr.y (mgr.34107) 10772 : audit [DBG] from='client.56449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[45495]: audit 2022-04-22T22:42:04.469358+0000 mon.c (mon.1) 364 : audit [DBG] from='client.? 172.21.15.115:0/1582389115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[46934]: cluster 2022-04-22T22:42:03.924542+0000 mgr.y (mgr.34107) 10771 : cluster [DBG] pgmap v11419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[46934]: audit 2022-04-22T22:42:04.048575+0000 mgr.y (mgr.34107) 10772 : audit [DBG] from='client.56449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:04 smithi115 bash[46934]: audit 2022-04-22T22:42:04.469358+0000 mon.c (mon.1) 364 : audit [DBG] from='client.? 172.21.15.115:0/1582389115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:04 smithi121 bash[36902]: cluster 2022-04-22T22:42:03.924542+0000 mgr.y (mgr.34107) 10771 : cluster [DBG] pgmap v11419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:04 smithi121 bash[36902]: audit 2022-04-22T22:42:04.048575+0000 mgr.y (mgr.34107) 10772 : audit [DBG] from='client.56449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:04 smithi121 bash[36902]: audit 2022-04-22T22:42:04.469358+0000 mon.c (mon.1) 364 : audit [DBG] from='client.? 172.21.15.115:0/1582389115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:05 smithi121 bash[36902]: audit 2022-04-22T22:42:05.325064+0000 mon.a (mon.0) 14992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:05 smithi115 bash[46934]: audit 2022-04-22T22:42:05.325064+0000 mon.a (mon.0) 14992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:05 smithi115 bash[45495]: audit 2022-04-22T22:42:05.325064+0000 mon.a (mon.0) 14992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:07 smithi121 bash[36902]: cluster 2022-04-22T22:42:05.925531+0000 mgr.y (mgr.34107) 10773 : cluster [DBG] pgmap v11420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:07 smithi121 bash[36902]: audit 2022-04-22T22:42:06.944996+0000 mon.a (mon.0) 14993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:07 smithi115 bash[45495]: cluster 2022-04-22T22:42:05.925531+0000 mgr.y (mgr.34107) 10773 : cluster [DBG] pgmap v11420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:07 smithi115 bash[45495]: audit 2022-04-22T22:42:06.944996+0000 mon.a (mon.0) 14993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:07 smithi115 bash[46934]: cluster 2022-04-22T22:42:05.925531+0000 mgr.y (mgr.34107) 10773 : cluster [DBG] pgmap v11420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:07 smithi115 bash[46934]: audit 2022-04-22T22:42:06.944996+0000 mon.a (mon.0) 14993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:08 smithi121 bash[36902]: audit 2022-04-22T22:42:08.362016+0000 mon.a (mon.0) 14994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:08 smithi115 bash[45495]: audit 2022-04-22T22:42:08.362016+0000 mon.a (mon.0) 14994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:08 smithi115 bash[46934]: audit 2022-04-22T22:42:08.362016+0000 mon.a (mon.0) 14994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:09 smithi121 bash[36902]: cluster 2022-04-22T22:42:07.926650+0000 mgr.y (mgr.34107) 10774 : cluster [DBG] pgmap v11421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:10.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:09 smithi115 bash[45495]: cluster 2022-04-22T22:42:07.926650+0000 mgr.y (mgr.34107) 10774 : cluster [DBG] pgmap v11421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:10.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:09 smithi115 bash[46934]: cluster 2022-04-22T22:42:07.926650+0000 mgr.y (mgr.34107) 10774 : cluster [DBG] pgmap v11421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:10 smithi121 bash[36902]: audit 2022-04-22T22:42:10.328134+0000 mon.a (mon.0) 14995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:11.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:10 smithi115 bash[45495]: audit 2022-04-22T22:42:10.328134+0000 mon.a (mon.0) 14995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:11.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:10 smithi115 bash[46934]: audit 2022-04-22T22:42:10.328134+0000 mon.a (mon.0) 14995 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:11 smithi121 bash[36902]: cluster 2022-04-22T22:42:09.927778+0000 mgr.y (mgr.34107) 10775 : cluster [DBG] pgmap v11422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:11 smithi115 bash[45495]: cluster 2022-04-22T22:42:09.927778+0000 mgr.y (mgr.34107) 10775 : cluster [DBG] pgmap v11422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:11 smithi115 bash[46934]: cluster 2022-04-22T22:42:09.927778+0000 mgr.y (mgr.34107) 10775 : cluster [DBG] pgmap v11422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:12 smithi121 bash[36902]: audit 2022-04-22T22:42:11.948274+0000 mon.a (mon.0) 14996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:12 smithi115 bash[45495]: audit 2022-04-22T22:42:11.948274+0000 mon.a (mon.0) 14996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:12.856Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:12 smithi115 bash[46934]: audit 2022-04-22T22:42:11.948274+0000 mon.a (mon.0) 14996 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:13 smithi121 bash[36902]: cluster 2022-04-22T22:42:11.928792+0000 mgr.y (mgr.34107) 10776 : cluster [DBG] pgmap v11423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:13 smithi121 bash[36902]: audit 2022-04-22T22:42:13.366076+0000 mon.a (mon.0) 14997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:13 smithi115 bash[46934]: cluster 2022-04-22T22:42:11.928792+0000 mgr.y (mgr.34107) 10776 : cluster [DBG] pgmap v11423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:13 smithi115 bash[46934]: audit 2022-04-22T22:42:13.366076+0000 mon.a (mon.0) 14997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:13 smithi115 bash[45495]: cluster 2022-04-22T22:42:11.928792+0000 mgr.y (mgr.34107) 10776 : cluster [DBG] pgmap v11423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:13 smithi115 bash[45495]: audit 2022-04-22T22:42:13.366076+0000 mon.a (mon.0) 14997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:14.719Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:15 smithi121 bash[36902]: cluster 2022-04-22T22:42:13.929928+0000 mgr.y (mgr.34107) 10777 : cluster [DBG] pgmap v11424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:15 smithi121 bash[36902]: audit 2022-04-22T22:42:15.331215+0000 mon.a (mon.0) 14998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:15 smithi115 bash[45495]: cluster 2022-04-22T22:42:13.929928+0000 mgr.y (mgr.34107) 10777 : cluster [DBG] pgmap v11424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:15 smithi115 bash[45495]: audit 2022-04-22T22:42:15.331215+0000 mon.a (mon.0) 14998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:15 smithi115 bash[46934]: cluster 2022-04-22T22:42:13.929928+0000 mgr.y (mgr.34107) 10777 : cluster [DBG] pgmap v11424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:15 smithi115 bash[46934]: audit 2022-04-22T22:42:15.331215+0000 mon.a (mon.0) 14998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:17 smithi121 bash[36902]: cluster 2022-04-22T22:42:15.930999+0000 mgr.y (mgr.34107) 10778 : cluster [DBG] pgmap v11425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:17 smithi121 bash[36902]: audit 2022-04-22T22:42:16.915628+0000 mon.a (mon.0) 14999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:42:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:17 smithi121 bash[36902]: audit 2022-04-22T22:42:16.916209+0000 mon.a (mon.0) 15000 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:42:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:17 smithi121 bash[36902]: audit 2022-04-22T22:42:16.951210+0000 mon.a (mon.0) 15001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[45495]: cluster 2022-04-22T22:42:15.930999+0000 mgr.y (mgr.34107) 10778 : cluster [DBG] pgmap v11425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[45495]: audit 2022-04-22T22:42:16.915628+0000 mon.a (mon.0) 14999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:42:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[45495]: audit 2022-04-22T22:42:16.916209+0000 mon.a (mon.0) 15000 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:42:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[45495]: audit 2022-04-22T22:42:16.951210+0000 mon.a (mon.0) 15001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[46934]: cluster 2022-04-22T22:42:15.930999+0000 mgr.y (mgr.34107) 10778 : cluster [DBG] pgmap v11425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[46934]: audit 2022-04-22T22:42:16.915628+0000 mon.a (mon.0) 14999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:42:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[46934]: audit 2022-04-22T22:42:16.916209+0000 mon.a (mon.0) 15000 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:42:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:17 smithi115 bash[46934]: audit 2022-04-22T22:42:16.951210+0000 mon.a (mon.0) 15001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:18 smithi121 bash[36902]: audit 2022-04-22T22:42:18.369810+0000 mon.a (mon.0) 15002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:18 smithi115 bash[45495]: audit 2022-04-22T22:42:18.369810+0000 mon.a (mon.0) 15002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:18 smithi115 bash[46934]: audit 2022-04-22T22:42:18.369810+0000 mon.a (mon.0) 15002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:19 smithi121 bash[36902]: cluster 2022-04-22T22:42:17.932202+0000 mgr.y (mgr.34107) 10779 : cluster [DBG] pgmap v11426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:19 smithi115 bash[45495]: cluster 2022-04-22T22:42:17.932202+0000 mgr.y (mgr.34107) 10779 : cluster [DBG] pgmap v11426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:19 smithi115 bash[46934]: cluster 2022-04-22T22:42:17.932202+0000 mgr.y (mgr.34107) 10779 : cluster [DBG] pgmap v11426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:20 smithi121 bash[36902]: audit 2022-04-22T22:42:20.334164+0000 mon.a (mon.0) 15003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:20 smithi115 bash[45495]: audit 2022-04-22T22:42:20.334164+0000 mon.a (mon.0) 15003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:20 smithi115 bash[46934]: audit 2022-04-22T22:42:20.334164+0000 mon.a (mon.0) 15003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:21 smithi121 bash[36902]: cluster 2022-04-22T22:42:19.933350+0000 mgr.y (mgr.34107) 10780 : cluster [DBG] pgmap v11427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:21 smithi115 bash[45495]: cluster 2022-04-22T22:42:19.933350+0000 mgr.y (mgr.34107) 10780 : cluster [DBG] pgmap v11427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:21 smithi115 bash[46934]: cluster 2022-04-22T22:42:19.933350+0000 mgr.y (mgr.34107) 10780 : cluster [DBG] pgmap v11427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:22 smithi121 bash[36902]: audit 2022-04-22T22:42:21.953970+0000 mon.a (mon.0) 15004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:22 smithi115 bash[45495]: audit 2022-04-22T22:42:21.953970+0000 mon.a (mon.0) 15004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:22.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:22 smithi115 bash[46934]: audit 2022-04-22T22:42:21.953970+0000 mon.a (mon.0) 15004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:23 smithi121 bash[36902]: cluster 2022-04-22T22:42:21.934540+0000 mgr.y (mgr.34107) 10781 : cluster [DBG] pgmap v11428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:23 smithi121 bash[36902]: audit 2022-04-22T22:42:23.373437+0000 mon.a (mon.0) 15005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:23 smithi115 bash[46934]: cluster 2022-04-22T22:42:21.934540+0000 mgr.y (mgr.34107) 10781 : cluster [DBG] pgmap v11428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:23 smithi115 bash[46934]: audit 2022-04-22T22:42:23.373437+0000 mon.a (mon.0) 15005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:23 smithi115 bash[45495]: cluster 2022-04-22T22:42:21.934540+0000 mgr.y (mgr.34107) 10781 : cluster [DBG] pgmap v11428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:23 smithi115 bash[45495]: audit 2022-04-22T22:42:23.373437+0000 mon.a (mon.0) 15005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:24.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:24 smithi115 bash[45495]: cluster 2022-04-22T22:42:23.935720+0000 mgr.y (mgr.34107) 10782 : cluster [DBG] pgmap v11429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:24 smithi115 bash[46934]: cluster 2022-04-22T22:42:23.935720+0000 mgr.y (mgr.34107) 10782 : cluster [DBG] pgmap v11429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:24 smithi121 bash[36902]: cluster 2022-04-22T22:42:23.935720+0000 mgr.y (mgr.34107) 10782 : cluster [DBG] pgmap v11429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:25 smithi121 bash[36902]: audit 2022-04-22T22:42:25.337309+0000 mon.a (mon.0) 15006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:26.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:25 smithi115 bash[46934]: audit 2022-04-22T22:42:25.337309+0000 mon.a (mon.0) 15006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:25 smithi115 bash[45495]: audit 2022-04-22T22:42:25.337309+0000 mon.a (mon.0) 15006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:27 smithi121 bash[36902]: cluster 2022-04-22T22:42:25.936737+0000 mgr.y (mgr.34107) 10783 : cluster [DBG] pgmap v11430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:27 smithi121 bash[36902]: audit 2022-04-22T22:42:26.956838+0000 mon.a (mon.0) 15007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:27 smithi115 bash[45495]: cluster 2022-04-22T22:42:25.936737+0000 mgr.y (mgr.34107) 10783 : cluster [DBG] pgmap v11430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:27 smithi115 bash[45495]: audit 2022-04-22T22:42:26.956838+0000 mon.a (mon.0) 15007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:27 smithi115 bash[46934]: cluster 2022-04-22T22:42:25.936737+0000 mgr.y (mgr.34107) 10783 : cluster [DBG] pgmap v11430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:27 smithi115 bash[46934]: audit 2022-04-22T22:42:26.956838+0000 mon.a (mon.0) 15007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:28 smithi121 bash[36902]: audit 2022-04-22T22:42:28.377063+0000 mon.a (mon.0) 15008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:28 smithi115 bash[45495]: audit 2022-04-22T22:42:28.377063+0000 mon.a (mon.0) 15008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:28 smithi115 bash[46934]: audit 2022-04-22T22:42:28.377063+0000 mon.a (mon.0) 15008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:29 smithi121 bash[36902]: cluster 2022-04-22T22:42:27.937808+0000 mgr.y (mgr.34107) 10784 : cluster [DBG] pgmap v11431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:29 smithi115 bash[45495]: cluster 2022-04-22T22:42:27.937808+0000 mgr.y (mgr.34107) 10784 : cluster [DBG] pgmap v11431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:29 smithi115 bash[46934]: cluster 2022-04-22T22:42:27.937808+0000 mgr.y (mgr.34107) 10784 : cluster [DBG] pgmap v11431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:30 smithi121 bash[36902]: audit 2022-04-22T22:42:30.340227+0000 mon.a (mon.0) 15009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:30 smithi115 bash[45495]: audit 2022-04-22T22:42:30.340227+0000 mon.a (mon.0) 15009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:30 smithi115 bash[46934]: audit 2022-04-22T22:42:30.340227+0000 mon.a (mon.0) 15009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[45495]: cluster 2022-04-22T22:42:29.938820+0000 mgr.y (mgr.34107) 10785 : cluster [DBG] pgmap v11432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[45495]: audit 2022-04-22T22:42:31.825021+0000 mon.a (mon.0) 15010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[45495]: audit 2022-04-22T22:42:31.832236+0000 mon.a (mon.0) 15011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[45495]: audit 2022-04-22T22:42:31.833405+0000 mon.a (mon.0) 15012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[46934]: cluster 2022-04-22T22:42:29.938820+0000 mgr.y (mgr.34107) 10785 : cluster [DBG] pgmap v11432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:32.191 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[46934]: audit 2022-04-22T22:42:31.825021+0000 mon.a (mon.0) 15010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[46934]: audit 2022-04-22T22:42:31.832236+0000 mon.a (mon.0) 15011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:31 smithi115 bash[46934]: audit 2022-04-22T22:42:31.833405+0000 mon.a (mon.0) 15012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:31 smithi121 bash[36902]: cluster 2022-04-22T22:42:29.938820+0000 mgr.y (mgr.34107) 10785 : cluster [DBG] pgmap v11432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:31 smithi121 bash[36902]: audit 2022-04-22T22:42:31.825021+0000 mon.a (mon.0) 15010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:31 smithi121 bash[36902]: audit 2022-04-22T22:42:31.832236+0000 mon.a (mon.0) 15011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:42:32.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:31 smithi121 bash[36902]: audit 2022-04-22T22:42:31.833405+0000 mon.a (mon.0) 15012 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:42:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:32 smithi121 bash[36902]: audit 2022-04-22T22:42:31.960209+0000 mon.a (mon.0) 15013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:32 smithi121 bash[36902]: audit 2022-04-22T22:42:32.203459+0000 mon.a (mon.0) 15014 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:42:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:32 smithi115 bash[45495]: audit 2022-04-22T22:42:31.960209+0000 mon.a (mon.0) 15013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:32 smithi115 bash[45495]: audit 2022-04-22T22:42:32.203459+0000 mon.a (mon.0) 15014 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:42:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:32.860Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:32 smithi115 bash[46934]: audit 2022-04-22T22:42:31.960209+0000 mon.a (mon.0) 15013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:32 smithi115 bash[46934]: audit 2022-04-22T22:42:32.203459+0000 mon.a (mon.0) 15014 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:42:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:33 smithi121 bash[36902]: cluster 2022-04-22T22:42:31.940116+0000 mgr.y (mgr.34107) 10786 : cluster [DBG] pgmap v11433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:33 smithi121 bash[36902]: audit 2022-04-22T22:42:33.381078+0000 mon.a (mon.0) 15015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:33 smithi115 bash[45495]: cluster 2022-04-22T22:42:31.940116+0000 mgr.y (mgr.34107) 10786 : cluster [DBG] pgmap v11433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:33 smithi115 bash[45495]: audit 2022-04-22T22:42:33.381078+0000 mon.a (mon.0) 15015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:33 smithi115 bash[46934]: cluster 2022-04-22T22:42:31.940116+0000 mgr.y (mgr.34107) 10786 : cluster [DBG] pgmap v11433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:33 smithi115 bash[46934]: audit 2022-04-22T22:42:33.381078+0000 mon.a (mon.0) 15015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:34.855 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:42:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:34.721Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:34 smithi115 bash[45495]: cluster 2022-04-22T22:42:33.941272+0000 mgr.y (mgr.34107) 10787 : cluster [DBG] pgmap v11434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:34 smithi115 bash[45495]: audit 2022-04-22T22:42:34.833021+0000 mgr.y (mgr.34107) 10788 : audit [DBG] from='client.46524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:34 smithi115 bash[46934]: cluster 2022-04-22T22:42:33.941272+0000 mgr.y (mgr.34107) 10787 : cluster [DBG] pgmap v11434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:34 smithi115 bash[46934]: audit 2022-04-22T22:42:34.833021+0000 mgr.y (mgr.34107) 10788 : audit [DBG] from='client.46524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:35.204 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:42:35.236 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:42:35.236 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:42:35.236 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:42:35.236 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:42:35.236 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:42:35.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:42:35.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:42:35.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:42:35.238 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:42:35.238 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:42:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:34 smithi121 bash[36902]: cluster 2022-04-22T22:42:33.941272+0000 mgr.y (mgr.34107) 10787 : cluster [DBG] pgmap v11434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:34 smithi121 bash[36902]: audit 2022-04-22T22:42:34.833021+0000 mgr.y (mgr.34107) 10788 : audit [DBG] from='client.46524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:35.612 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:42:35.612 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:42:35.612 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:42:35.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:42:35.613 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:42:35.614 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:42:35.614 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:42:35.614 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:42:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:35 smithi121 bash[36902]: audit 2022-04-22T22:42:35.182467+0000 mgr.y (mgr.34107) 10789 : audit [DBG] from='client.46530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:35 smithi121 bash[36902]: audit 2022-04-22T22:42:35.347033+0000 mon.a (mon.0) 15016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:35 smithi121 bash[36902]: audit 2022-04-22T22:42:35.608632+0000 mon.a (mon.0) 15017 : audit [DBG] from='client.? 172.21.15.115:0/2281177228' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:36.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[46934]: audit 2022-04-22T22:42:35.182467+0000 mgr.y (mgr.34107) 10789 : audit [DBG] from='client.46530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[46934]: audit 2022-04-22T22:42:35.347033+0000 mon.a (mon.0) 15016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[46934]: audit 2022-04-22T22:42:35.608632+0000 mon.a (mon.0) 15017 : audit [DBG] from='client.? 172.21.15.115:0/2281177228' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[45495]: audit 2022-04-22T22:42:35.182467+0000 mgr.y (mgr.34107) 10789 : audit [DBG] from='client.46530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:42:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[45495]: audit 2022-04-22T22:42:35.347033+0000 mon.a (mon.0) 15016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:35 smithi115 bash[45495]: audit 2022-04-22T22:42:35.608632+0000 mon.a (mon.0) 15017 : audit [DBG] from='client.? 172.21.15.115:0/2281177228' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:42:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:37 smithi121 bash[36902]: cluster 2022-04-22T22:42:35.942385+0000 mgr.y (mgr.34107) 10790 : cluster [DBG] pgmap v11435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:37 smithi121 bash[36902]: audit 2022-04-22T22:42:36.963146+0000 mon.a (mon.0) 15018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:37 smithi115 bash[45495]: cluster 2022-04-22T22:42:35.942385+0000 mgr.y (mgr.34107) 10790 : cluster [DBG] pgmap v11435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:37 smithi115 bash[45495]: audit 2022-04-22T22:42:36.963146+0000 mon.a (mon.0) 15018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:37 smithi115 bash[46934]: cluster 2022-04-22T22:42:35.942385+0000 mgr.y (mgr.34107) 10790 : cluster [DBG] pgmap v11435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:37 smithi115 bash[46934]: audit 2022-04-22T22:42:36.963146+0000 mon.a (mon.0) 15018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:38 smithi121 bash[36902]: audit 2022-04-22T22:42:38.384919+0000 mon.a (mon.0) 15019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:38 smithi115 bash[45495]: audit 2022-04-22T22:42:38.384919+0000 mon.a (mon.0) 15019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:38 smithi115 bash[46934]: audit 2022-04-22T22:42:38.384919+0000 mon.a (mon.0) 15019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:39 smithi121 bash[36902]: cluster 2022-04-22T22:42:37.943418+0000 mgr.y (mgr.34107) 10791 : cluster [DBG] pgmap v11436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:39 smithi115 bash[45495]: cluster 2022-04-22T22:42:37.943418+0000 mgr.y (mgr.34107) 10791 : cluster [DBG] pgmap v11436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:39 smithi115 bash[46934]: cluster 2022-04-22T22:42:37.943418+0000 mgr.y (mgr.34107) 10791 : cluster [DBG] pgmap v11436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:40 smithi121 bash[36902]: audit 2022-04-22T22:42:40.345117+0000 mon.a (mon.0) 15020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:40 smithi115 bash[45495]: audit 2022-04-22T22:42:40.345117+0000 mon.a (mon.0) 15020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:40 smithi115 bash[46934]: audit 2022-04-22T22:42:40.345117+0000 mon.a (mon.0) 15020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:41 smithi121 bash[36902]: cluster 2022-04-22T22:42:39.944481+0000 mgr.y (mgr.34107) 10792 : cluster [DBG] pgmap v11437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:42.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:41 smithi115 bash[46934]: cluster 2022-04-22T22:42:39.944481+0000 mgr.y (mgr.34107) 10792 : cluster [DBG] pgmap v11437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:41 smithi115 bash[45495]: cluster 2022-04-22T22:42:39.944481+0000 mgr.y (mgr.34107) 10792 : cluster [DBG] pgmap v11437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:42 smithi121 bash[36902]: audit 2022-04-22T22:42:41.966114+0000 mon.a (mon.0) 15021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:43.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:42 smithi115 bash[46934]: audit 2022-04-22T22:42:41.966114+0000 mon.a (mon.0) 15021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:42.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:42 smithi115 bash[45495]: audit 2022-04-22T22:42:41.966114+0000 mon.a (mon.0) 15021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:43 smithi121 bash[36902]: cluster 2022-04-22T22:42:41.945527+0000 mgr.y (mgr.34107) 10793 : cluster [DBG] pgmap v11438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:43 smithi121 bash[36902]: audit 2022-04-22T22:42:43.388670+0000 mon.a (mon.0) 15022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:44.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:43 smithi115 bash[46934]: cluster 2022-04-22T22:42:41.945527+0000 mgr.y (mgr.34107) 10793 : cluster [DBG] pgmap v11438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:43 smithi115 bash[46934]: audit 2022-04-22T22:42:43.388670+0000 mon.a (mon.0) 15022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:43 smithi115 bash[45495]: cluster 2022-04-22T22:42:41.945527+0000 mgr.y (mgr.34107) 10793 : cluster [DBG] pgmap v11438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:43 smithi115 bash[45495]: audit 2022-04-22T22:42:43.388670+0000 mon.a (mon.0) 15022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:44 smithi115 bash[45495]: cluster 2022-04-22T22:42:43.946788+0000 mgr.y (mgr.34107) 10794 : cluster [DBG] pgmap v11439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:44.723Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:44 smithi115 bash[46934]: cluster 2022-04-22T22:42:43.946788+0000 mgr.y (mgr.34107) 10794 : cluster [DBG] pgmap v11439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:44 smithi121 bash[36902]: cluster 2022-04-22T22:42:43.946788+0000 mgr.y (mgr.34107) 10794 : cluster [DBG] pgmap v11439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:45 smithi121 bash[36902]: audit 2022-04-22T22:42:45.348064+0000 mon.a (mon.0) 15023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:45 smithi115 bash[45495]: audit 2022-04-22T22:42:45.348064+0000 mon.a (mon.0) 15023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:45 smithi115 bash[46934]: audit 2022-04-22T22:42:45.348064+0000 mon.a (mon.0) 15023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:47 smithi121 bash[36902]: cluster 2022-04-22T22:42:45.947897+0000 mgr.y (mgr.34107) 10795 : cluster [DBG] pgmap v11440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:47 smithi121 bash[36902]: audit 2022-04-22T22:42:46.969064+0000 mon.a (mon.0) 15024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:48.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:47 smithi115 bash[46934]: cluster 2022-04-22T22:42:45.947897+0000 mgr.y (mgr.34107) 10795 : cluster [DBG] pgmap v11440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:47 smithi115 bash[46934]: audit 2022-04-22T22:42:46.969064+0000 mon.a (mon.0) 15024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:47 smithi115 bash[45495]: cluster 2022-04-22T22:42:45.947897+0000 mgr.y (mgr.34107) 10795 : cluster [DBG] pgmap v11440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:47 smithi115 bash[45495]: audit 2022-04-22T22:42:46.969064+0000 mon.a (mon.0) 15024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:48 smithi121 bash[36902]: audit 2022-04-22T22:42:48.392243+0000 mon.a (mon.0) 15025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:49.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:48 smithi115 bash[46934]: audit 2022-04-22T22:42:48.392243+0000 mon.a (mon.0) 15025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:48 smithi115 bash[45495]: audit 2022-04-22T22:42:48.392243+0000 mon.a (mon.0) 15025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:49 smithi121 bash[36902]: cluster 2022-04-22T22:42:47.948968+0000 mgr.y (mgr.34107) 10796 : cluster [DBG] pgmap v11441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:49 smithi115 bash[45495]: cluster 2022-04-22T22:42:47.948968+0000 mgr.y (mgr.34107) 10796 : cluster [DBG] pgmap v11441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:49 smithi115 bash[46934]: cluster 2022-04-22T22:42:47.948968+0000 mgr.y (mgr.34107) 10796 : cluster [DBG] pgmap v11441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:50 smithi121 bash[36902]: audit 2022-04-22T22:42:50.351133+0000 mon.a (mon.0) 15026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:50 smithi115 bash[45495]: audit 2022-04-22T22:42:50.351133+0000 mon.a (mon.0) 15026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:50 smithi115 bash[46934]: audit 2022-04-22T22:42:50.351133+0000 mon.a (mon.0) 15026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:51 smithi121 bash[36902]: cluster 2022-04-22T22:42:49.950041+0000 mgr.y (mgr.34107) 10797 : cluster [DBG] pgmap v11442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:51 smithi115 bash[45495]: cluster 2022-04-22T22:42:49.950041+0000 mgr.y (mgr.34107) 10797 : cluster [DBG] pgmap v11442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:51 smithi115 bash[46934]: cluster 2022-04-22T22:42:49.950041+0000 mgr.y (mgr.34107) 10797 : cluster [DBG] pgmap v11442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:52 smithi121 bash[36902]: audit 2022-04-22T22:42:51.972197+0000 mon.a (mon.0) 15027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:52 smithi115 bash[45495]: audit 2022-04-22T22:42:51.972197+0000 mon.a (mon.0) 15027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:53.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:52.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:52 smithi115 bash[46934]: audit 2022-04-22T22:42:51.972197+0000 mon.a (mon.0) 15027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:53 smithi121 bash[36902]: cluster 2022-04-22T22:42:51.951082+0000 mgr.y (mgr.34107) 10798 : cluster [DBG] pgmap v11443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:53 smithi121 bash[36902]: audit 2022-04-22T22:42:53.395758+0000 mon.a (mon.0) 15028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:53 smithi115 bash[45495]: cluster 2022-04-22T22:42:51.951082+0000 mgr.y (mgr.34107) 10798 : cluster [DBG] pgmap v11443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:53 smithi115 bash[45495]: audit 2022-04-22T22:42:53.395758+0000 mon.a (mon.0) 15028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:53 smithi115 bash[46934]: cluster 2022-04-22T22:42:51.951082+0000 mgr.y (mgr.34107) 10798 : cluster [DBG] pgmap v11443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:53 smithi115 bash[46934]: audit 2022-04-22T22:42:53.395758+0000 mon.a (mon.0) 15028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:54 smithi115 bash[45495]: cluster 2022-04-22T22:42:53.952207+0000 mgr.y (mgr.34107) 10799 : cluster [DBG] pgmap v11444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:42:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:42:54.722Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:42:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:54 smithi115 bash[46934]: cluster 2022-04-22T22:42:53.952207+0000 mgr.y (mgr.34107) 10799 : cluster [DBG] pgmap v11444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:54 smithi121 bash[36902]: cluster 2022-04-22T22:42:53.952207+0000 mgr.y (mgr.34107) 10799 : cluster [DBG] pgmap v11444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:55 smithi121 bash[36902]: audit 2022-04-22T22:42:55.354453+0000 mon.a (mon.0) 15029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:55 smithi115 bash[45495]: audit 2022-04-22T22:42:55.354453+0000 mon.a (mon.0) 15029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:55 smithi115 bash[46934]: audit 2022-04-22T22:42:55.354453+0000 mon.a (mon.0) 15029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:42:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:57 smithi121 bash[36902]: cluster 2022-04-22T22:42:55.953369+0000 mgr.y (mgr.34107) 10800 : cluster [DBG] pgmap v11445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:57 smithi121 bash[36902]: audit 2022-04-22T22:42:56.975442+0000 mon.a (mon.0) 15030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:57 smithi115 bash[45495]: cluster 2022-04-22T22:42:55.953369+0000 mgr.y (mgr.34107) 10800 : cluster [DBG] pgmap v11445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:57 smithi115 bash[45495]: audit 2022-04-22T22:42:56.975442+0000 mon.a (mon.0) 15030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:57 smithi115 bash[46934]: cluster 2022-04-22T22:42:55.953369+0000 mgr.y (mgr.34107) 10800 : cluster [DBG] pgmap v11445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:42:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:57 smithi115 bash[46934]: audit 2022-04-22T22:42:56.975442+0000 mon.a (mon.0) 15030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:42:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:58 smithi121 bash[36902]: audit 2022-04-22T22:42:58.399672+0000 mon.a (mon.0) 15031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:58 smithi115 bash[45495]: audit 2022-04-22T22:42:58.399672+0000 mon.a (mon.0) 15031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:42:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:58 smithi115 bash[46934]: audit 2022-04-22T22:42:58.399672+0000 mon.a (mon.0) 15031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:42:59 smithi121 bash[36902]: cluster 2022-04-22T22:42:57.954457+0000 mgr.y (mgr.34107) 10801 : cluster [DBG] pgmap v11446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:42:59 smithi115 bash[45495]: cluster 2022-04-22T22:42:57.954457+0000 mgr.y (mgr.34107) 10801 : cluster [DBG] pgmap v11446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:42:59 smithi115 bash[46934]: cluster 2022-04-22T22:42:57.954457+0000 mgr.y (mgr.34107) 10801 : cluster [DBG] pgmap v11446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:00 smithi121 bash[36902]: audit 2022-04-22T22:43:00.357388+0000 mon.a (mon.0) 15032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:00 smithi115 bash[45495]: audit 2022-04-22T22:43:00.357388+0000 mon.a (mon.0) 15032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:00 smithi115 bash[46934]: audit 2022-04-22T22:43:00.357388+0000 mon.a (mon.0) 15032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:01 smithi121 bash[36902]: cluster 2022-04-22T22:42:59.955611+0000 mgr.y (mgr.34107) 10802 : cluster [DBG] pgmap v11447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:02.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:01 smithi115 bash[46934]: cluster 2022-04-22T22:42:59.955611+0000 mgr.y (mgr.34107) 10802 : cluster [DBG] pgmap v11447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:01 smithi115 bash[45495]: cluster 2022-04-22T22:42:59.955611+0000 mgr.y (mgr.34107) 10802 : cluster [DBG] pgmap v11447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:02 smithi121 bash[36902]: cluster 2022-04-22T22:43:01.956653+0000 mgr.y (mgr.34107) 10803 : cluster [DBG] pgmap v11448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:02 smithi121 bash[36902]: audit 2022-04-22T22:43:01.978629+0000 mon.a (mon.0) 15033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:02 smithi115 bash[45495]: cluster 2022-04-22T22:43:01.956653+0000 mgr.y (mgr.34107) 10803 : cluster [DBG] pgmap v11448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:02 smithi115 bash[45495]: audit 2022-04-22T22:43:01.978629+0000 mon.a (mon.0) 15033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:02.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:02 smithi115 bash[46934]: cluster 2022-04-22T22:43:01.956653+0000 mgr.y (mgr.34107) 10803 : cluster [DBG] pgmap v11448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:02 smithi115 bash[46934]: audit 2022-04-22T22:43:01.978629+0000 mon.a (mon.0) 15033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:03 smithi121 bash[36902]: audit 2022-04-22T22:43:03.403337+0000 mon.a (mon.0) 15034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:03 smithi115 bash[45495]: audit 2022-04-22T22:43:03.403337+0000 mon.a (mon.0) 15034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:03 smithi115 bash[46934]: audit 2022-04-22T22:43:03.403337+0000 mon.a (mon.0) 15034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:04 smithi115 bash[45495]: cluster 2022-04-22T22:43:03.957773+0000 mgr.y (mgr.34107) 10804 : cluster [DBG] pgmap v11449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:04.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:05.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:04 smithi115 bash[46934]: cluster 2022-04-22T22:43:03.957773+0000 mgr.y (mgr.34107) 10804 : cluster [DBG] pgmap v11449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:04 smithi121 bash[36902]: cluster 2022-04-22T22:43:03.957773+0000 mgr.y (mgr.34107) 10804 : cluster [DBG] pgmap v11449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:05.984 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:43:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:05 smithi121 bash[36902]: audit 2022-04-22T22:43:05.360339+0000 mon.a (mon.0) 15035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:05 smithi115 bash[46934]: audit 2022-04-22T22:43:05.360339+0000 mon.a (mon.0) 15035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:05 smithi115 bash[45495]: audit 2022-04-22T22:43:05.360339+0000 mon.a (mon.0) 15035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:43:06.334 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:43:06.335 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:43:06.336 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:43:06.336 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:43:06.336 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:43:06.336 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:43:06.336 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:43:06.747 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:06.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:43:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:43:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:06 smithi121 bash[36902]: cluster 2022-04-22T22:43:05.958916+0000 mgr.y (mgr.34107) 10805 : cluster [DBG] pgmap v11450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:06 smithi121 bash[36902]: audit 2022-04-22T22:43:05.965055+0000 mgr.y (mgr.34107) 10806 : audit [DBG] from='client.56479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:06 smithi121 bash[36902]: audit 2022-04-22T22:43:06.311791+0000 mgr.y (mgr.34107) 10807 : audit [DBG] from='client.56485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:06 smithi121 bash[36902]: audit 2022-04-22T22:43:06.744011+0000 mon.a (mon.0) 15036 : audit [DBG] from='client.? 172.21.15.115:0/149956875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[45495]: cluster 2022-04-22T22:43:05.958916+0000 mgr.y (mgr.34107) 10805 : cluster [DBG] pgmap v11450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[45495]: audit 2022-04-22T22:43:05.965055+0000 mgr.y (mgr.34107) 10806 : audit [DBG] from='client.56479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[45495]: audit 2022-04-22T22:43:06.311791+0000 mgr.y (mgr.34107) 10807 : audit [DBG] from='client.56485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[45495]: audit 2022-04-22T22:43:06.744011+0000 mon.a (mon.0) 15036 : audit [DBG] from='client.? 172.21.15.115:0/149956875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[46934]: cluster 2022-04-22T22:43:05.958916+0000 mgr.y (mgr.34107) 10805 : cluster [DBG] pgmap v11450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:07.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[46934]: audit 2022-04-22T22:43:05.965055+0000 mgr.y (mgr.34107) 10806 : audit [DBG] from='client.56479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[46934]: audit 2022-04-22T22:43:06.311791+0000 mgr.y (mgr.34107) 10807 : audit [DBG] from='client.56485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:07.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:06 smithi115 bash[46934]: audit 2022-04-22T22:43:06.744011+0000 mon.a (mon.0) 15036 : audit [DBG] from='client.? 172.21.15.115:0/149956875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:07 smithi121 bash[36902]: audit 2022-04-22T22:43:06.981831+0000 mon.a (mon.0) 15037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:07 smithi115 bash[45495]: audit 2022-04-22T22:43:06.981831+0000 mon.a (mon.0) 15037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:07 smithi115 bash[46934]: audit 2022-04-22T22:43:06.981831+0000 mon.a (mon.0) 15037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:08 smithi121 bash[36902]: cluster 2022-04-22T22:43:07.960104+0000 mgr.y (mgr.34107) 10808 : cluster [DBG] pgmap v11451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:08 smithi121 bash[36902]: audit 2022-04-22T22:43:08.406998+0000 mon.a (mon.0) 15038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:08 smithi115 bash[45495]: cluster 2022-04-22T22:43:07.960104+0000 mgr.y (mgr.34107) 10808 : cluster [DBG] pgmap v11451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:08 smithi115 bash[45495]: audit 2022-04-22T22:43:08.406998+0000 mon.a (mon.0) 15038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:08 smithi115 bash[46934]: cluster 2022-04-22T22:43:07.960104+0000 mgr.y (mgr.34107) 10808 : cluster [DBG] pgmap v11451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:08 smithi115 bash[46934]: audit 2022-04-22T22:43:08.406998+0000 mon.a (mon.0) 15038 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:11.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:11 smithi115 bash[45495]: cluster 2022-04-22T22:43:09.961206+0000 mgr.y (mgr.34107) 10809 : cluster [DBG] pgmap v11452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:11 smithi115 bash[45495]: audit 2022-04-22T22:43:10.363230+0000 mon.a (mon.0) 15039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:11 smithi115 bash[46934]: cluster 2022-04-22T22:43:09.961206+0000 mgr.y (mgr.34107) 10809 : cluster [DBG] pgmap v11452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:11 smithi115 bash[46934]: audit 2022-04-22T22:43:10.363230+0000 mon.a (mon.0) 15039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:11 smithi121 bash[36902]: cluster 2022-04-22T22:43:09.961206+0000 mgr.y (mgr.34107) 10809 : cluster [DBG] pgmap v11452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:11 smithi121 bash[36902]: audit 2022-04-22T22:43:10.363230+0000 mon.a (mon.0) 15039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:12 smithi121 bash[36902]: audit 2022-04-22T22:43:11.984733+0000 mon.a (mon.0) 15040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:12.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:12 smithi115 bash[45495]: audit 2022-04-22T22:43:11.984733+0000 mon.a (mon.0) 15040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:12 smithi115 bash[46934]: audit 2022-04-22T22:43:11.984733+0000 mon.a (mon.0) 15040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:13 smithi121 bash[36902]: cluster 2022-04-22T22:43:11.962312+0000 mgr.y (mgr.34107) 10810 : cluster [DBG] pgmap v11453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:13 smithi121 bash[36902]: audit 2022-04-22T22:43:13.410660+0000 mon.a (mon.0) 15041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:13 smithi115 bash[45495]: cluster 2022-04-22T22:43:11.962312+0000 mgr.y (mgr.34107) 10810 : cluster [DBG] pgmap v11453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:13 smithi115 bash[45495]: audit 2022-04-22T22:43:13.410660+0000 mon.a (mon.0) 15041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:13 smithi115 bash[46934]: cluster 2022-04-22T22:43:11.962312+0000 mgr.y (mgr.34107) 10810 : cluster [DBG] pgmap v11453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:13 smithi115 bash[46934]: audit 2022-04-22T22:43:13.410660+0000 mon.a (mon.0) 15041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:14.724Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:15 smithi121 bash[36902]: cluster 2022-04-22T22:43:13.963522+0000 mgr.y (mgr.34107) 10811 : cluster [DBG] pgmap v11454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:15 smithi121 bash[36902]: audit 2022-04-22T22:43:15.366382+0000 mon.a (mon.0) 15042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:15 smithi115 bash[45495]: cluster 2022-04-22T22:43:13.963522+0000 mgr.y (mgr.34107) 10811 : cluster [DBG] pgmap v11454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:15 smithi115 bash[45495]: audit 2022-04-22T22:43:15.366382+0000 mon.a (mon.0) 15042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:15 smithi115 bash[46934]: cluster 2022-04-22T22:43:13.963522+0000 mgr.y (mgr.34107) 10811 : cluster [DBG] pgmap v11454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:15 smithi115 bash[46934]: audit 2022-04-22T22:43:15.366382+0000 mon.a (mon.0) 15042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:17 smithi121 bash[36902]: cluster 2022-04-22T22:43:15.964682+0000 mgr.y (mgr.34107) 10812 : cluster [DBG] pgmap v11455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:17 smithi121 bash[36902]: audit 2022-04-22T22:43:16.925216+0000 mon.a (mon.0) 15043 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:43:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:17 smithi121 bash[36902]: audit 2022-04-22T22:43:16.925848+0000 mon.a (mon.0) 15044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:43:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:17 smithi121 bash[36902]: audit 2022-04-22T22:43:16.987209+0000 mon.a (mon.0) 15045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[45495]: cluster 2022-04-22T22:43:15.964682+0000 mgr.y (mgr.34107) 10812 : cluster [DBG] pgmap v11455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[45495]: audit 2022-04-22T22:43:16.925216+0000 mon.a (mon.0) 15043 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:43:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[45495]: audit 2022-04-22T22:43:16.925848+0000 mon.a (mon.0) 15044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:43:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[45495]: audit 2022-04-22T22:43:16.987209+0000 mon.a (mon.0) 15045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[46934]: cluster 2022-04-22T22:43:15.964682+0000 mgr.y (mgr.34107) 10812 : cluster [DBG] pgmap v11455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[46934]: audit 2022-04-22T22:43:16.925216+0000 mon.a (mon.0) 15043 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:43:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[46934]: audit 2022-04-22T22:43:16.925848+0000 mon.a (mon.0) 15044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:43:18.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:17 smithi115 bash[46934]: audit 2022-04-22T22:43:16.987209+0000 mon.a (mon.0) 15045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:18 smithi121 bash[36902]: cluster 2022-04-22T22:43:17.965825+0000 mgr.y (mgr.34107) 10813 : cluster [DBG] pgmap v11456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:18 smithi121 bash[36902]: audit 2022-04-22T22:43:18.414475+0000 mon.a (mon.0) 15046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:18 smithi115 bash[45495]: cluster 2022-04-22T22:43:17.965825+0000 mgr.y (mgr.34107) 10813 : cluster [DBG] pgmap v11456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:18 smithi115 bash[45495]: audit 2022-04-22T22:43:18.414475+0000 mon.a (mon.0) 15046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:18 smithi115 bash[46934]: cluster 2022-04-22T22:43:17.965825+0000 mgr.y (mgr.34107) 10813 : cluster [DBG] pgmap v11456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:18 smithi115 bash[46934]: audit 2022-04-22T22:43:18.414475+0000 mon.a (mon.0) 15046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:21 smithi115 bash[45495]: cluster 2022-04-22T22:43:19.966926+0000 mgr.y (mgr.34107) 10814 : cluster [DBG] pgmap v11457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:21 smithi115 bash[45495]: audit 2022-04-22T22:43:20.369281+0000 mon.a (mon.0) 15047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:21 smithi115 bash[46934]: cluster 2022-04-22T22:43:19.966926+0000 mgr.y (mgr.34107) 10814 : cluster [DBG] pgmap v11457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:21 smithi115 bash[46934]: audit 2022-04-22T22:43:20.369281+0000 mon.a (mon.0) 15047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:21 smithi121 bash[36902]: cluster 2022-04-22T22:43:19.966926+0000 mgr.y (mgr.34107) 10814 : cluster [DBG] pgmap v11457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:21 smithi121 bash[36902]: audit 2022-04-22T22:43:20.369281+0000 mon.a (mon.0) 15047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:22 smithi121 bash[36902]: audit 2022-04-22T22:43:21.990303+0000 mon.a (mon.0) 15048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:22 smithi115 bash[45495]: audit 2022-04-22T22:43:21.990303+0000 mon.a (mon.0) 15048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:22.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:22 smithi115 bash[46934]: audit 2022-04-22T22:43:21.990303+0000 mon.a (mon.0) 15048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:23 smithi121 bash[36902]: cluster 2022-04-22T22:43:21.968071+0000 mgr.y (mgr.34107) 10815 : cluster [DBG] pgmap v11458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:23 smithi121 bash[36902]: audit 2022-04-22T22:43:23.418422+0000 mon.a (mon.0) 15049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:23 smithi115 bash[45495]: cluster 2022-04-22T22:43:21.968071+0000 mgr.y (mgr.34107) 10815 : cluster [DBG] pgmap v11458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:23 smithi115 bash[45495]: audit 2022-04-22T22:43:23.418422+0000 mon.a (mon.0) 15049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:23 smithi115 bash[46934]: cluster 2022-04-22T22:43:21.968071+0000 mgr.y (mgr.34107) 10815 : cluster [DBG] pgmap v11458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:23 smithi115 bash[46934]: audit 2022-04-22T22:43:23.418422+0000 mon.a (mon.0) 15049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:24 smithi115 bash[45495]: cluster 2022-04-22T22:43:23.969124+0000 mgr.y (mgr.34107) 10816 : cluster [DBG] pgmap v11459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:24.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:24 smithi115 bash[46934]: cluster 2022-04-22T22:43:23.969124+0000 mgr.y (mgr.34107) 10816 : cluster [DBG] pgmap v11459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:24 smithi121 bash[36902]: cluster 2022-04-22T22:43:23.969124+0000 mgr.y (mgr.34107) 10816 : cluster [DBG] pgmap v11459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:25 smithi121 bash[36902]: audit 2022-04-22T22:43:25.372350+0000 mon.a (mon.0) 15050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:25 smithi115 bash[45495]: audit 2022-04-22T22:43:25.372350+0000 mon.a (mon.0) 15050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:25 smithi115 bash[46934]: audit 2022-04-22T22:43:25.372350+0000 mon.a (mon.0) 15050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:27 smithi121 bash[36902]: cluster 2022-04-22T22:43:25.970131+0000 mgr.y (mgr.34107) 10817 : cluster [DBG] pgmap v11460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:27 smithi121 bash[36902]: audit 2022-04-22T22:43:26.993571+0000 mon.a (mon.0) 15051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:27 smithi115 bash[45495]: cluster 2022-04-22T22:43:25.970131+0000 mgr.y (mgr.34107) 10817 : cluster [DBG] pgmap v11460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:27 smithi115 bash[45495]: audit 2022-04-22T22:43:26.993571+0000 mon.a (mon.0) 15051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:27 smithi115 bash[46934]: cluster 2022-04-22T22:43:25.970131+0000 mgr.y (mgr.34107) 10817 : cluster [DBG] pgmap v11460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:27 smithi115 bash[46934]: audit 2022-04-22T22:43:26.993571+0000 mon.a (mon.0) 15051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:28 smithi121 bash[36902]: audit 2022-04-22T22:43:28.421982+0000 mon.a (mon.0) 15052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:28 smithi115 bash[45495]: audit 2022-04-22T22:43:28.421982+0000 mon.a (mon.0) 15052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:28 smithi115 bash[46934]: audit 2022-04-22T22:43:28.421982+0000 mon.a (mon.0) 15052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:29 smithi121 bash[36902]: cluster 2022-04-22T22:43:27.971236+0000 mgr.y (mgr.34107) 10818 : cluster [DBG] pgmap v11461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:29 smithi115 bash[45495]: cluster 2022-04-22T22:43:27.971236+0000 mgr.y (mgr.34107) 10818 : cluster [DBG] pgmap v11461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:30.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:29 smithi115 bash[46934]: cluster 2022-04-22T22:43:27.971236+0000 mgr.y (mgr.34107) 10818 : cluster [DBG] pgmap v11461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:30 smithi121 bash[36902]: audit 2022-04-22T22:43:30.375516+0000 mon.a (mon.0) 15053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:30 smithi115 bash[45495]: audit 2022-04-22T22:43:30.375516+0000 mon.a (mon.0) 15053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:30 smithi115 bash[46934]: audit 2022-04-22T22:43:30.375516+0000 mon.a (mon.0) 15053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:32.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:31 smithi115 bash[46934]: cluster 2022-04-22T22:43:29.972368+0000 mgr.y (mgr.34107) 10819 : cluster [DBG] pgmap v11462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:32.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:31 smithi115 bash[45495]: cluster 2022-04-22T22:43:29.972368+0000 mgr.y (mgr.34107) 10819 : cluster [DBG] pgmap v11462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:32.243 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:31 smithi121 bash[36902]: cluster 2022-04-22T22:43:29.972368+0000 mgr.y (mgr.34107) 10819 : cluster [DBG] pgmap v11462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: cluster 2022-04-22T22:43:31.973487+0000 mgr.y (mgr.34107) 10820 : cluster [DBG] pgmap v11463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: audit 2022-04-22T22:43:31.996585+0000 mon.a (mon.0) 15054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: audit 2022-04-22T22:43:32.212968+0000 mon.a (mon.0) 15055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:43:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: audit 2022-04-22T22:43:32.216205+0000 mon.a (mon.0) 15056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:43:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: audit 2022-04-22T22:43:32.218145+0000 mon.a (mon.0) 15057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:43:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:32 smithi121 bash[36902]: audit 2022-04-22T22:43:32.597143+0000 mon.a (mon.0) 15058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:43:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:32.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: cluster 2022-04-22T22:43:31.973487+0000 mgr.y (mgr.34107) 10820 : cluster [DBG] pgmap v11463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: audit 2022-04-22T22:43:31.996585+0000 mon.a (mon.0) 15054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: audit 2022-04-22T22:43:32.212968+0000 mon.a (mon.0) 15055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: audit 2022-04-22T22:43:32.216205+0000 mon.a (mon.0) 15056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: audit 2022-04-22T22:43:32.218145+0000 mon.a (mon.0) 15057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:43:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[45495]: audit 2022-04-22T22:43:32.597143+0000 mon.a (mon.0) 15058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: cluster 2022-04-22T22:43:31.973487+0000 mgr.y (mgr.34107) 10820 : cluster [DBG] pgmap v11463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: audit 2022-04-22T22:43:31.996585+0000 mon.a (mon.0) 15054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: audit 2022-04-22T22:43:32.212968+0000 mon.a (mon.0) 15055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: audit 2022-04-22T22:43:32.216205+0000 mon.a (mon.0) 15056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: audit 2022-04-22T22:43:32.218145+0000 mon.a (mon.0) 15057 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:43:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:32 smithi115 bash[46934]: audit 2022-04-22T22:43:32.597143+0000 mon.a (mon.0) 15058 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:43:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:33 smithi121 bash[36902]: audit 2022-04-22T22:43:33.425505+0000 mon.a (mon.0) 15059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:33 smithi115 bash[45495]: audit 2022-04-22T22:43:33.425505+0000 mon.a (mon.0) 15059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:33 smithi115 bash[46934]: audit 2022-04-22T22:43:33.425505+0000 mon.a (mon.0) 15059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:34 smithi115 bash[45495]: cluster 2022-04-22T22:43:33.974581+0000 mgr.y (mgr.34107) 10821 : cluster [DBG] pgmap v11464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:34.725Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:34 smithi115 bash[46934]: cluster 2022-04-22T22:43:33.974581+0000 mgr.y (mgr.34107) 10821 : cluster [DBG] pgmap v11464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:34 smithi121 bash[36902]: cluster 2022-04-22T22:43:33.974581+0000 mgr.y (mgr.34107) 10821 : cluster [DBG] pgmap v11464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:35 smithi121 bash[36902]: audit 2022-04-22T22:43:35.378655+0000 mon.a (mon.0) 15060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:36.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:35 smithi115 bash[45495]: audit 2022-04-22T22:43:35.378655+0000 mon.a (mon.0) 15060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:36.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:35 smithi115 bash[46934]: audit 2022-04-22T22:43:35.378655+0000 mon.a (mon.0) 15060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:37.128 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:43:37.484 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:43:37.484 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:43:37.484 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:43:37.484 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:43:37.484 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:43:37.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:43:37.486 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:43:37.891 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:37.892 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:43:37.893 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:43:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:37 smithi121 bash[36902]: cluster 2022-04-22T22:43:35.975693+0000 mgr.y (mgr.34107) 10822 : cluster [DBG] pgmap v11465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:37 smithi121 bash[36902]: audit 2022-04-22T22:43:36.999447+0000 mon.a (mon.0) 15061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:37 smithi115 bash[45495]: cluster 2022-04-22T22:43:35.975693+0000 mgr.y (mgr.34107) 10822 : cluster [DBG] pgmap v11465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:37 smithi115 bash[45495]: audit 2022-04-22T22:43:36.999447+0000 mon.a (mon.0) 15061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:37 smithi115 bash[46934]: cluster 2022-04-22T22:43:35.975693+0000 mgr.y (mgr.34107) 10822 : cluster [DBG] pgmap v11465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:37 smithi115 bash[46934]: audit 2022-04-22T22:43:36.999447+0000 mon.a (mon.0) 15061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:38 smithi121 bash[36902]: audit 2022-04-22T22:43:37.109305+0000 mgr.y (mgr.34107) 10823 : audit [DBG] from='client.56497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:38 smithi121 bash[36902]: audit 2022-04-22T22:43:37.460325+0000 mgr.y (mgr.34107) 10824 : audit [DBG] from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:38 smithi121 bash[36902]: audit 2022-04-22T22:43:37.888003+0000 mon.c (mon.1) 365 : audit [DBG] from='client.? 172.21.15.115:0/883582342' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:38 smithi121 bash[36902]: audit 2022-04-22T22:43:38.429386+0000 mon.a (mon.0) 15062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[45495]: audit 2022-04-22T22:43:37.109305+0000 mgr.y (mgr.34107) 10823 : audit [DBG] from='client.56497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[45495]: audit 2022-04-22T22:43:37.460325+0000 mgr.y (mgr.34107) 10824 : audit [DBG] from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[45495]: audit 2022-04-22T22:43:37.888003+0000 mon.c (mon.1) 365 : audit [DBG] from='client.? 172.21.15.115:0/883582342' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[45495]: audit 2022-04-22T22:43:38.429386+0000 mon.a (mon.0) 15062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[46934]: audit 2022-04-22T22:43:37.109305+0000 mgr.y (mgr.34107) 10823 : audit [DBG] from='client.56497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[46934]: audit 2022-04-22T22:43:37.460325+0000 mgr.y (mgr.34107) 10824 : audit [DBG] from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:43:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[46934]: audit 2022-04-22T22:43:37.888003+0000 mon.c (mon.1) 365 : audit [DBG] from='client.? 172.21.15.115:0/883582342' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:43:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:38 smithi115 bash[46934]: audit 2022-04-22T22:43:38.429386+0000 mon.a (mon.0) 15062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:39 smithi121 bash[36902]: cluster 2022-04-22T22:43:37.976676+0000 mgr.y (mgr.34107) 10825 : cluster [DBG] pgmap v11466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:39 smithi115 bash[45495]: cluster 2022-04-22T22:43:37.976676+0000 mgr.y (mgr.34107) 10825 : cluster [DBG] pgmap v11466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:39 smithi115 bash[46934]: cluster 2022-04-22T22:43:37.976676+0000 mgr.y (mgr.34107) 10825 : cluster [DBG] pgmap v11466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:40 smithi121 bash[36902]: audit 2022-04-22T22:43:40.381753+0000 mon.a (mon.0) 15063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:40 smithi115 bash[45495]: audit 2022-04-22T22:43:40.381753+0000 mon.a (mon.0) 15063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:41.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:40 smithi115 bash[46934]: audit 2022-04-22T22:43:40.381753+0000 mon.a (mon.0) 15063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:41 smithi121 bash[36902]: cluster 2022-04-22T22:43:39.977880+0000 mgr.y (mgr.34107) 10826 : cluster [DBG] pgmap v11467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:41 smithi115 bash[45495]: cluster 2022-04-22T22:43:39.977880+0000 mgr.y (mgr.34107) 10826 : cluster [DBG] pgmap v11467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:41 smithi115 bash[46934]: cluster 2022-04-22T22:43:39.977880+0000 mgr.y (mgr.34107) 10826 : cluster [DBG] pgmap v11467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:42 smithi121 bash[36902]: cluster 2022-04-22T22:43:41.979033+0000 mgr.y (mgr.34107) 10827 : cluster [DBG] pgmap v11468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:42 smithi121 bash[36902]: audit 2022-04-22T22:43:42.002396+0000 mon.a (mon.0) 15064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:42 smithi115 bash[45495]: cluster 2022-04-22T22:43:41.979033+0000 mgr.y (mgr.34107) 10827 : cluster [DBG] pgmap v11468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:43.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:42 smithi115 bash[45495]: audit 2022-04-22T22:43:42.002396+0000 mon.a (mon.0) 15064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:42.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:42 smithi115 bash[46934]: cluster 2022-04-22T22:43:41.979033+0000 mgr.y (mgr.34107) 10827 : cluster [DBG] pgmap v11468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:42 smithi115 bash[46934]: audit 2022-04-22T22:43:42.002396+0000 mon.a (mon.0) 15064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:43 smithi121 bash[36902]: audit 2022-04-22T22:43:43.432990+0000 mon.a (mon.0) 15065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:43 smithi115 bash[45495]: audit 2022-04-22T22:43:43.432990+0000 mon.a (mon.0) 15065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:43 smithi115 bash[46934]: audit 2022-04-22T22:43:43.432990+0000 mon.a (mon.0) 15065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:44 smithi115 bash[45495]: cluster 2022-04-22T22:43:43.980630+0000 mgr.y (mgr.34107) 10828 : cluster [DBG] pgmap v11469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:44.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:44 smithi115 bash[46934]: cluster 2022-04-22T22:43:43.980630+0000 mgr.y (mgr.34107) 10828 : cluster [DBG] pgmap v11469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:44 smithi121 bash[36902]: cluster 2022-04-22T22:43:43.980630+0000 mgr.y (mgr.34107) 10828 : cluster [DBG] pgmap v11469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:45 smithi121 bash[36902]: audit 2022-04-22T22:43:45.384882+0000 mon.a (mon.0) 15066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:45 smithi115 bash[45495]: audit 2022-04-22T22:43:45.384882+0000 mon.a (mon.0) 15066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:45 smithi115 bash[46934]: audit 2022-04-22T22:43:45.384882+0000 mon.a (mon.0) 15066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:47 smithi121 bash[36902]: cluster 2022-04-22T22:43:45.981579+0000 mgr.y (mgr.34107) 10829 : cluster [DBG] pgmap v11470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:47 smithi121 bash[36902]: audit 2022-04-22T22:43:47.005583+0000 mon.a (mon.0) 15067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:47 smithi115 bash[45495]: cluster 2022-04-22T22:43:45.981579+0000 mgr.y (mgr.34107) 10829 : cluster [DBG] pgmap v11470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:47 smithi115 bash[45495]: audit 2022-04-22T22:43:47.005583+0000 mon.a (mon.0) 15067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:47 smithi115 bash[46934]: cluster 2022-04-22T22:43:45.981579+0000 mgr.y (mgr.34107) 10829 : cluster [DBG] pgmap v11470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:47 smithi115 bash[46934]: audit 2022-04-22T22:43:47.005583+0000 mon.a (mon.0) 15067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:48 smithi121 bash[36902]: audit 2022-04-22T22:43:48.436584+0000 mon.a (mon.0) 15068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:48 smithi115 bash[45495]: audit 2022-04-22T22:43:48.436584+0000 mon.a (mon.0) 15068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:49.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:48 smithi115 bash[46934]: audit 2022-04-22T22:43:48.436584+0000 mon.a (mon.0) 15068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:49 smithi121 bash[36902]: cluster 2022-04-22T22:43:47.982826+0000 mgr.y (mgr.34107) 10830 : cluster [DBG] pgmap v11471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:49 smithi115 bash[45495]: cluster 2022-04-22T22:43:47.982826+0000 mgr.y (mgr.34107) 10830 : cluster [DBG] pgmap v11471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:49 smithi115 bash[46934]: cluster 2022-04-22T22:43:47.982826+0000 mgr.y (mgr.34107) 10830 : cluster [DBG] pgmap v11471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:50 smithi121 bash[36902]: cluster 2022-04-22T22:43:49.984025+0000 mgr.y (mgr.34107) 10831 : cluster [DBG] pgmap v11472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:50 smithi121 bash[36902]: audit 2022-04-22T22:43:50.388294+0000 mon.a (mon.0) 15069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:50 smithi115 bash[45495]: cluster 2022-04-22T22:43:49.984025+0000 mgr.y (mgr.34107) 10831 : cluster [DBG] pgmap v11472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:50 smithi115 bash[45495]: audit 2022-04-22T22:43:50.388294+0000 mon.a (mon.0) 15069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:50 smithi115 bash[46934]: cluster 2022-04-22T22:43:49.984025+0000 mgr.y (mgr.34107) 10831 : cluster [DBG] pgmap v11472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:50 smithi115 bash[46934]: audit 2022-04-22T22:43:50.388294+0000 mon.a (mon.0) 15069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:52 smithi121 bash[36902]: audit 2022-04-22T22:43:52.008674+0000 mon.a (mon.0) 15070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:52 smithi115 bash[45495]: audit 2022-04-22T22:43:52.008674+0000 mon.a (mon.0) 15070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:52.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:52 smithi115 bash[46934]: audit 2022-04-22T22:43:52.008674+0000 mon.a (mon.0) 15070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:53 smithi121 bash[36902]: cluster 2022-04-22T22:43:51.985120+0000 mgr.y (mgr.34107) 10832 : cluster [DBG] pgmap v11473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:53 smithi121 bash[36902]: audit 2022-04-22T22:43:53.440299+0000 mon.a (mon.0) 15071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:53 smithi115 bash[45495]: cluster 2022-04-22T22:43:51.985120+0000 mgr.y (mgr.34107) 10832 : cluster [DBG] pgmap v11473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:53 smithi115 bash[45495]: audit 2022-04-22T22:43:53.440299+0000 mon.a (mon.0) 15071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:53 smithi115 bash[46934]: cluster 2022-04-22T22:43:51.985120+0000 mgr.y (mgr.34107) 10832 : cluster [DBG] pgmap v11473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:53 smithi115 bash[46934]: audit 2022-04-22T22:43:53.440299+0000 mon.a (mon.0) 15071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:54 smithi115 bash[45495]: cluster 2022-04-22T22:43:53.986275+0000 mgr.y (mgr.34107) 10833 : cluster [DBG] pgmap v11474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:43:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:43:54.726Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:43:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:54 smithi115 bash[46934]: cluster 2022-04-22T22:43:53.986275+0000 mgr.y (mgr.34107) 10833 : cluster [DBG] pgmap v11474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:54 smithi121 bash[36902]: cluster 2022-04-22T22:43:53.986275+0000 mgr.y (mgr.34107) 10833 : cluster [DBG] pgmap v11474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:55 smithi121 bash[36902]: audit 2022-04-22T22:43:55.391454+0000 mon.a (mon.0) 15072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:55 smithi115 bash[45495]: audit 2022-04-22T22:43:55.391454+0000 mon.a (mon.0) 15072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:56.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:55 smithi115 bash[46934]: audit 2022-04-22T22:43:55.391454+0000 mon.a (mon.0) 15072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:43:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:57 smithi121 bash[36902]: cluster 2022-04-22T22:43:55.987395+0000 mgr.y (mgr.34107) 10834 : cluster [DBG] pgmap v11475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:57 smithi121 bash[36902]: audit 2022-04-22T22:43:57.011656+0000 mon.a (mon.0) 15073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:57 smithi115 bash[45495]: cluster 2022-04-22T22:43:55.987395+0000 mgr.y (mgr.34107) 10834 : cluster [DBG] pgmap v11475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:57 smithi115 bash[45495]: audit 2022-04-22T22:43:57.011656+0000 mon.a (mon.0) 15073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:57 smithi115 bash[46934]: cluster 2022-04-22T22:43:55.987395+0000 mgr.y (mgr.34107) 10834 : cluster [DBG] pgmap v11475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:43:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:57 smithi115 bash[46934]: audit 2022-04-22T22:43:57.011656+0000 mon.a (mon.0) 15073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:43:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:58 smithi121 bash[36902]: audit 2022-04-22T22:43:58.443860+0000 mon.a (mon.0) 15074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:58 smithi115 bash[45495]: audit 2022-04-22T22:43:58.443860+0000 mon.a (mon.0) 15074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:43:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:58 smithi115 bash[46934]: audit 2022-04-22T22:43:58.443860+0000 mon.a (mon.0) 15074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:43:59 smithi121 bash[36902]: cluster 2022-04-22T22:43:57.988637+0000 mgr.y (mgr.34107) 10835 : cluster [DBG] pgmap v11476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:43:59 smithi115 bash[45495]: cluster 2022-04-22T22:43:57.988637+0000 mgr.y (mgr.34107) 10835 : cluster [DBG] pgmap v11476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:43:59 smithi115 bash[46934]: cluster 2022-04-22T22:43:57.988637+0000 mgr.y (mgr.34107) 10835 : cluster [DBG] pgmap v11476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:00 smithi121 bash[36902]: cluster 2022-04-22T22:43:59.989419+0000 mgr.y (mgr.34107) 10836 : cluster [DBG] pgmap v11477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:01.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:00 smithi121 bash[36902]: audit 2022-04-22T22:44:00.394559+0000 mon.a (mon.0) 15075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:00 smithi115 bash[45495]: cluster 2022-04-22T22:43:59.989419+0000 mgr.y (mgr.34107) 10836 : cluster [DBG] pgmap v11477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:00 smithi115 bash[45495]: audit 2022-04-22T22:44:00.394559+0000 mon.a (mon.0) 15075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:00 smithi115 bash[46934]: cluster 2022-04-22T22:43:59.989419+0000 mgr.y (mgr.34107) 10836 : cluster [DBG] pgmap v11477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:00 smithi115 bash[46934]: audit 2022-04-22T22:44:00.394559+0000 mon.a (mon.0) 15075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:02 smithi121 bash[36902]: audit 2022-04-22T22:44:02.014934+0000 mon.a (mon.0) 15076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:03.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:02 smithi115 bash[45495]: audit 2022-04-22T22:44:02.014934+0000 mon.a (mon.0) 15076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:02.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:02 smithi115 bash[46934]: audit 2022-04-22T22:44:02.014934+0000 mon.a (mon.0) 15076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:03 smithi121 bash[36902]: cluster 2022-04-22T22:44:01.990454+0000 mgr.y (mgr.34107) 10837 : cluster [DBG] pgmap v11478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:03 smithi121 bash[36902]: audit 2022-04-22T22:44:03.447802+0000 mon.a (mon.0) 15077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:03 smithi115 bash[45495]: cluster 2022-04-22T22:44:01.990454+0000 mgr.y (mgr.34107) 10837 : cluster [DBG] pgmap v11478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:03 smithi115 bash[45495]: audit 2022-04-22T22:44:03.447802+0000 mon.a (mon.0) 15077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:03 smithi115 bash[46934]: cluster 2022-04-22T22:44:01.990454+0000 mgr.y (mgr.34107) 10837 : cluster [DBG] pgmap v11478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:03 smithi115 bash[46934]: audit 2022-04-22T22:44:03.447802+0000 mon.a (mon.0) 15077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:04 smithi115 bash[45495]: cluster 2022-04-22T22:44:03.991645+0000 mgr.y (mgr.34107) 10838 : cluster [DBG] pgmap v11479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:05.090 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:04.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:05.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:04 smithi115 bash[46934]: cluster 2022-04-22T22:44:03.991645+0000 mgr.y (mgr.34107) 10838 : cluster [DBG] pgmap v11479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:04 smithi121 bash[36902]: cluster 2022-04-22T22:44:03.991645+0000 mgr.y (mgr.34107) 10838 : cluster [DBG] pgmap v11479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:05 smithi121 bash[36902]: audit 2022-04-22T22:44:05.397588+0000 mon.a (mon.0) 15078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:05 smithi115 bash[45495]: audit 2022-04-22T22:44:05.397588+0000 mon.a (mon.0) 15078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:06.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:05 smithi115 bash[46934]: audit 2022-04-22T22:44:05.397588+0000 mon.a (mon.0) 15078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:07 smithi121 bash[36902]: cluster 2022-04-22T22:44:05.992947+0000 mgr.y (mgr.34107) 10839 : cluster [DBG] pgmap v11480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:07 smithi121 bash[36902]: audit 2022-04-22T22:44:07.018031+0000 mon.a (mon.0) 15079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:08.265 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:44:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:07 smithi115 bash[45495]: cluster 2022-04-22T22:44:05.992947+0000 mgr.y (mgr.34107) 10839 : cluster [DBG] pgmap v11480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:07 smithi115 bash[45495]: audit 2022-04-22T22:44:07.018031+0000 mon.a (mon.0) 15079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:07 smithi115 bash[46934]: cluster 2022-04-22T22:44:05.992947+0000 mgr.y (mgr.34107) 10839 : cluster [DBG] pgmap v11480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:07 smithi115 bash[46934]: audit 2022-04-22T22:44:07.018031+0000 mon.a (mon.0) 15079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:44:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:44:08.612 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:44:08.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:44:08.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:44:08.613 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:44:08.613 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:44:09.012 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:44:09.012 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:44:09.012 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:44:09.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:09.012 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:44:09.013 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:44:09.014 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:44:09.014 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:44:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:08 smithi121 bash[36902]: audit 2022-04-22T22:44:08.451378+0000 mon.a (mon.0) 15080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:08 smithi115 bash[45495]: audit 2022-04-22T22:44:08.451378+0000 mon.a (mon.0) 15080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:09.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:08 smithi115 bash[46934]: audit 2022-04-22T22:44:08.451378+0000 mon.a (mon.0) 15080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:09 smithi121 bash[36902]: cluster 2022-04-22T22:44:07.993802+0000 mgr.y (mgr.34107) 10840 : cluster [DBG] pgmap v11481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:09 smithi121 bash[36902]: audit 2022-04-22T22:44:08.245133+0000 mgr.y (mgr.34107) 10841 : audit [DBG] from='client.56515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:09 smithi121 bash[36902]: audit 2022-04-22T22:44:08.588106+0000 mgr.y (mgr.34107) 10842 : audit [DBG] from='client.46584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:09 smithi121 bash[36902]: audit 2022-04-22T22:44:09.008562+0000 mon.a (mon.0) 15081 : audit [DBG] from='client.? 172.21.15.115:0/864775252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[45495]: cluster 2022-04-22T22:44:07.993802+0000 mgr.y (mgr.34107) 10840 : cluster [DBG] pgmap v11481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[45495]: audit 2022-04-22T22:44:08.245133+0000 mgr.y (mgr.34107) 10841 : audit [DBG] from='client.56515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[45495]: audit 2022-04-22T22:44:08.588106+0000 mgr.y (mgr.34107) 10842 : audit [DBG] from='client.46584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[45495]: audit 2022-04-22T22:44:09.008562+0000 mon.a (mon.0) 15081 : audit [DBG] from='client.? 172.21.15.115:0/864775252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[46934]: cluster 2022-04-22T22:44:07.993802+0000 mgr.y (mgr.34107) 10840 : cluster [DBG] pgmap v11481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[46934]: audit 2022-04-22T22:44:08.245133+0000 mgr.y (mgr.34107) 10841 : audit [DBG] from='client.56515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[46934]: audit 2022-04-22T22:44:08.588106+0000 mgr.y (mgr.34107) 10842 : audit [DBG] from='client.46584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:09 smithi115 bash[46934]: audit 2022-04-22T22:44:09.008562+0000 mon.a (mon.0) 15081 : audit [DBG] from='client.? 172.21.15.115:0/864775252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:10 smithi121 bash[36902]: cluster 2022-04-22T22:44:09.995001+0000 mgr.y (mgr.34107) 10843 : cluster [DBG] pgmap v11482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:10 smithi121 bash[36902]: audit 2022-04-22T22:44:10.400659+0000 mon.a (mon.0) 15082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:11.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:10 smithi115 bash[46934]: cluster 2022-04-22T22:44:09.995001+0000 mgr.y (mgr.34107) 10843 : cluster [DBG] pgmap v11482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:10 smithi115 bash[46934]: audit 2022-04-22T22:44:10.400659+0000 mon.a (mon.0) 15082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:10 smithi115 bash[45495]: cluster 2022-04-22T22:44:09.995001+0000 mgr.y (mgr.34107) 10843 : cluster [DBG] pgmap v11482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:10 smithi115 bash[45495]: audit 2022-04-22T22:44:10.400659+0000 mon.a (mon.0) 15082 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:12 smithi121 bash[36902]: audit 2022-04-22T22:44:12.021211+0000 mon.a (mon.0) 15083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:12 smithi115 bash[45495]: audit 2022-04-22T22:44:12.021211+0000 mon.a (mon.0) 15083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:12.865Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:12 smithi115 bash[46934]: audit 2022-04-22T22:44:12.021211+0000 mon.a (mon.0) 15083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:13 smithi121 bash[36902]: cluster 2022-04-22T22:44:11.996082+0000 mgr.y (mgr.34107) 10844 : cluster [DBG] pgmap v11483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:13 smithi121 bash[36902]: audit 2022-04-22T22:44:13.455093+0000 mon.a (mon.0) 15084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:13 smithi115 bash[45495]: cluster 2022-04-22T22:44:11.996082+0000 mgr.y (mgr.34107) 10844 : cluster [DBG] pgmap v11483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:13 smithi115 bash[45495]: audit 2022-04-22T22:44:13.455093+0000 mon.a (mon.0) 15084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:13 smithi115 bash[46934]: cluster 2022-04-22T22:44:11.996082+0000 mgr.y (mgr.34107) 10844 : cluster [DBG] pgmap v11483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:13 smithi115 bash[46934]: audit 2022-04-22T22:44:13.455093+0000 mon.a (mon.0) 15084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:14 smithi115 bash[45495]: cluster 2022-04-22T22:44:13.997310+0000 mgr.y (mgr.34107) 10845 : cluster [DBG] pgmap v11484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:14.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:14 smithi115 bash[46934]: cluster 2022-04-22T22:44:13.997310+0000 mgr.y (mgr.34107) 10845 : cluster [DBG] pgmap v11484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:14 smithi121 bash[36902]: cluster 2022-04-22T22:44:13.997310+0000 mgr.y (mgr.34107) 10845 : cluster [DBG] pgmap v11484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:15 smithi121 bash[36902]: audit 2022-04-22T22:44:15.403756+0000 mon.a (mon.0) 15085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:15 smithi115 bash[45495]: audit 2022-04-22T22:44:15.403756+0000 mon.a (mon.0) 15085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:15 smithi115 bash[46934]: audit 2022-04-22T22:44:15.403756+0000 mon.a (mon.0) 15085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:17 smithi121 bash[36902]: cluster 2022-04-22T22:44:15.998649+0000 mgr.y (mgr.34107) 10846 : cluster [DBG] pgmap v11485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:17 smithi121 bash[36902]: audit 2022-04-22T22:44:16.935073+0000 mon.a (mon.0) 15086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:44:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:17 smithi121 bash[36902]: audit 2022-04-22T22:44:16.935851+0000 mon.a (mon.0) 15087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:44:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:17 smithi121 bash[36902]: audit 2022-04-22T22:44:17.023905+0000 mon.a (mon.0) 15088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[46934]: cluster 2022-04-22T22:44:15.998649+0000 mgr.y (mgr.34107) 10846 : cluster [DBG] pgmap v11485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[46934]: audit 2022-04-22T22:44:16.935073+0000 mon.a (mon.0) 15086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:44:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[46934]: audit 2022-04-22T22:44:16.935851+0000 mon.a (mon.0) 15087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:44:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[46934]: audit 2022-04-22T22:44:17.023905+0000 mon.a (mon.0) 15088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[45495]: cluster 2022-04-22T22:44:15.998649+0000 mgr.y (mgr.34107) 10846 : cluster [DBG] pgmap v11485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[45495]: audit 2022-04-22T22:44:16.935073+0000 mon.a (mon.0) 15086 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:44:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[45495]: audit 2022-04-22T22:44:16.935851+0000 mon.a (mon.0) 15087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:44:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:17 smithi115 bash[45495]: audit 2022-04-22T22:44:17.023905+0000 mon.a (mon.0) 15088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:18 smithi121 bash[36902]: audit 2022-04-22T22:44:18.458848+0000 mon.a (mon.0) 15089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:18 smithi115 bash[45495]: audit 2022-04-22T22:44:18.458848+0000 mon.a (mon.0) 15089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:18 smithi115 bash[46934]: audit 2022-04-22T22:44:18.458848+0000 mon.a (mon.0) 15089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:19 smithi121 bash[36902]: cluster 2022-04-22T22:44:17.999665+0000 mgr.y (mgr.34107) 10847 : cluster [DBG] pgmap v11486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:19 smithi115 bash[45495]: cluster 2022-04-22T22:44:17.999665+0000 mgr.y (mgr.34107) 10847 : cluster [DBG] pgmap v11486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:19 smithi115 bash[46934]: cluster 2022-04-22T22:44:17.999665+0000 mgr.y (mgr.34107) 10847 : cluster [DBG] pgmap v11486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:20 smithi121 bash[36902]: cluster 2022-04-22T22:44:20.000752+0000 mgr.y (mgr.34107) 10848 : cluster [DBG] pgmap v11487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:21.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:20 smithi121 bash[36902]: audit 2022-04-22T22:44:20.406978+0000 mon.a (mon.0) 15090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:20 smithi115 bash[45495]: cluster 2022-04-22T22:44:20.000752+0000 mgr.y (mgr.34107) 10848 : cluster [DBG] pgmap v11487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:20 smithi115 bash[45495]: audit 2022-04-22T22:44:20.406978+0000 mon.a (mon.0) 15090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:20 smithi115 bash[46934]: cluster 2022-04-22T22:44:20.000752+0000 mgr.y (mgr.34107) 10848 : cluster [DBG] pgmap v11487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:20 smithi115 bash[46934]: audit 2022-04-22T22:44:20.406978+0000 mon.a (mon.0) 15090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:22 smithi121 bash[36902]: audit 2022-04-22T22:44:22.026574+0000 mon.a (mon.0) 15091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:22 smithi115 bash[45495]: audit 2022-04-22T22:44:22.026574+0000 mon.a (mon.0) 15091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:22.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:22 smithi115 bash[46934]: audit 2022-04-22T22:44:22.026574+0000 mon.a (mon.0) 15091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:23 smithi121 bash[36902]: cluster 2022-04-22T22:44:22.001740+0000 mgr.y (mgr.34107) 10849 : cluster [DBG] pgmap v11488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:23 smithi121 bash[36902]: audit 2022-04-22T22:44:23.462897+0000 mon.a (mon.0) 15092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:23 smithi115 bash[45495]: cluster 2022-04-22T22:44:22.001740+0000 mgr.y (mgr.34107) 10849 : cluster [DBG] pgmap v11488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:23 smithi115 bash[45495]: audit 2022-04-22T22:44:23.462897+0000 mon.a (mon.0) 15092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:23 smithi115 bash[46934]: cluster 2022-04-22T22:44:22.001740+0000 mgr.y (mgr.34107) 10849 : cluster [DBG] pgmap v11488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:23 smithi115 bash[46934]: audit 2022-04-22T22:44:23.462897+0000 mon.a (mon.0) 15092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:24 smithi115 bash[45495]: cluster 2022-04-22T22:44:24.002946+0000 mgr.y (mgr.34107) 10850 : cluster [DBG] pgmap v11489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:24.728Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:24 smithi115 bash[46934]: cluster 2022-04-22T22:44:24.002946+0000 mgr.y (mgr.34107) 10850 : cluster [DBG] pgmap v11489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:24 smithi121 bash[36902]: cluster 2022-04-22T22:44:24.002946+0000 mgr.y (mgr.34107) 10850 : cluster [DBG] pgmap v11489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:25 smithi121 bash[36902]: audit 2022-04-22T22:44:25.410175+0000 mon.a (mon.0) 15093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:25 smithi115 bash[46934]: audit 2022-04-22T22:44:25.410175+0000 mon.a (mon.0) 15093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:25 smithi115 bash[45495]: audit 2022-04-22T22:44:25.410175+0000 mon.a (mon.0) 15093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:27 smithi121 bash[36902]: cluster 2022-04-22T22:44:26.003609+0000 mgr.y (mgr.34107) 10851 : cluster [DBG] pgmap v11490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:27 smithi121 bash[36902]: audit 2022-04-22T22:44:27.029740+0000 mon.a (mon.0) 15094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:27 smithi115 bash[46934]: cluster 2022-04-22T22:44:26.003609+0000 mgr.y (mgr.34107) 10851 : cluster [DBG] pgmap v11490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:27 smithi115 bash[46934]: audit 2022-04-22T22:44:27.029740+0000 mon.a (mon.0) 15094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:27 smithi115 bash[45495]: cluster 2022-04-22T22:44:26.003609+0000 mgr.y (mgr.34107) 10851 : cluster [DBG] pgmap v11490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:28.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:27 smithi115 bash[45495]: audit 2022-04-22T22:44:27.029740+0000 mon.a (mon.0) 15094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:28 smithi121 bash[36902]: audit 2022-04-22T22:44:28.466647+0000 mon.a (mon.0) 15095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:28 smithi115 bash[45495]: audit 2022-04-22T22:44:28.466647+0000 mon.a (mon.0) 15095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:28 smithi115 bash[46934]: audit 2022-04-22T22:44:28.466647+0000 mon.a (mon.0) 15095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:29 smithi121 bash[36902]: cluster 2022-04-22T22:44:28.004532+0000 mgr.y (mgr.34107) 10852 : cluster [DBG] pgmap v11491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:29 smithi115 bash[45495]: cluster 2022-04-22T22:44:28.004532+0000 mgr.y (mgr.34107) 10852 : cluster [DBG] pgmap v11491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:29 smithi115 bash[46934]: cluster 2022-04-22T22:44:28.004532+0000 mgr.y (mgr.34107) 10852 : cluster [DBG] pgmap v11491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:30 smithi121 bash[36902]: cluster 2022-04-22T22:44:30.005634+0000 mgr.y (mgr.34107) 10853 : cluster [DBG] pgmap v11492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:31.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:30 smithi121 bash[36902]: audit 2022-04-22T22:44:30.413492+0000 mon.a (mon.0) 15096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:31.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:30 smithi115 bash[45495]: cluster 2022-04-22T22:44:30.005634+0000 mgr.y (mgr.34107) 10853 : cluster [DBG] pgmap v11492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:30 smithi115 bash[45495]: audit 2022-04-22T22:44:30.413492+0000 mon.a (mon.0) 15096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:30 smithi115 bash[46934]: cluster 2022-04-22T22:44:30.005634+0000 mgr.y (mgr.34107) 10853 : cluster [DBG] pgmap v11492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:30 smithi115 bash[46934]: audit 2022-04-22T22:44:30.413492+0000 mon.a (mon.0) 15096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:32.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:32 smithi121 bash[36902]: audit 2022-04-22T22:44:32.032796+0000 mon.a (mon.0) 15097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:32.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:32 smithi121 bash[36902]: audit 2022-04-22T22:44:32.606129+0000 mon.a (mon.0) 15098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:44:32.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:32 smithi121 bash[36902]: audit 2022-04-22T22:44:32.609073+0000 mon.a (mon.0) 15099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:44:32.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:32 smithi121 bash[36902]: audit 2022-04-22T22:44:32.610820+0000 mon.a (mon.0) 15100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:44:33.217 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:32.867Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:32.032796+0000 mon.a (mon.0) 15097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:32.606129+0000 mon.a (mon.0) 15098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:44:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:32.609073+0000 mon.a (mon.0) 15099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:44:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:32.610820+0000 mon.a (mon.0) 15100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:44:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:32.032796+0000 mon.a (mon.0) 15097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:32.606129+0000 mon.a (mon.0) 15098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:44:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:32.609073+0000 mon.a (mon.0) 15099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:44:33.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:32.610820+0000 mon.a (mon.0) 15100 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:44:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:33 smithi121 bash[36902]: cluster 2022-04-22T22:44:32.006703+0000 mgr.y (mgr.34107) 10854 : cluster [DBG] pgmap v11493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:33 smithi121 bash[36902]: audit 2022-04-22T22:44:33.222111+0000 mon.a (mon.0) 15101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:33 smithi121 bash[36902]: audit 2022-04-22T22:44:33.470439+0000 mon.a (mon.0) 15102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: cluster 2022-04-22T22:44:32.006703+0000 mgr.y (mgr.34107) 10854 : cluster [DBG] pgmap v11493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:33.222111+0000 mon.a (mon.0) 15101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[45495]: audit 2022-04-22T22:44:33.470439+0000 mon.a (mon.0) 15102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: cluster 2022-04-22T22:44:32.006703+0000 mgr.y (mgr.34107) 10854 : cluster [DBG] pgmap v11493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:33.222111+0000 mon.a (mon.0) 15101 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:33 smithi115 bash[46934]: audit 2022-04-22T22:44:33.470439+0000 mon.a (mon.0) 15102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:34 smithi115 bash[45495]: cluster 2022-04-22T22:44:34.007914+0000 mgr.y (mgr.34107) 10855 : cluster [DBG] pgmap v11494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:34.729Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:34 smithi115 bash[46934]: cluster 2022-04-22T22:44:34.007914+0000 mgr.y (mgr.34107) 10855 : cluster [DBG] pgmap v11494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:34 smithi121 bash[36902]: cluster 2022-04-22T22:44:34.007914+0000 mgr.y (mgr.34107) 10855 : cluster [DBG] pgmap v11494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:35 smithi115 bash[45495]: audit 2022-04-22T22:44:35.416668+0000 mon.a (mon.0) 15103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:35 smithi115 bash[46934]: audit 2022-04-22T22:44:35.416668+0000 mon.a (mon.0) 15103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:36.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:35 smithi121 bash[36902]: audit 2022-04-22T22:44:35.416668+0000 mon.a (mon.0) 15103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:37 smithi121 bash[36902]: cluster 2022-04-22T22:44:36.008607+0000 mgr.y (mgr.34107) 10856 : cluster [DBG] pgmap v11495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:37 smithi121 bash[36902]: audit 2022-04-22T22:44:37.035841+0000 mon.a (mon.0) 15104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:37 smithi115 bash[45495]: cluster 2022-04-22T22:44:36.008607+0000 mgr.y (mgr.34107) 10856 : cluster [DBG] pgmap v11495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:37 smithi115 bash[45495]: audit 2022-04-22T22:44:37.035841+0000 mon.a (mon.0) 15104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:37 smithi115 bash[46934]: cluster 2022-04-22T22:44:36.008607+0000 mgr.y (mgr.34107) 10856 : cluster [DBG] pgmap v11495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:37 smithi115 bash[46934]: audit 2022-04-22T22:44:37.035841+0000 mon.a (mon.0) 15104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:38 smithi121 bash[36902]: audit 2022-04-22T22:44:38.474103+0000 mon.a (mon.0) 15105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:38 smithi115 bash[46934]: audit 2022-04-22T22:44:38.474103+0000 mon.a (mon.0) 15105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:38 smithi115 bash[45495]: audit 2022-04-22T22:44:38.474103+0000 mon.a (mon.0) 15105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:39.384 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:44:39.725 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:44:39.725 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:44:39.725 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:44:39.725 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:44:39.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:44:39.727 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:44:40.132 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:44:40.132 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:44:40.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:44:40.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:44:40.134 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:44:40.134 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:44:40.134 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:44:40.134 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:44:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:39 smithi121 bash[36902]: cluster 2022-04-22T22:44:38.009594+0000 mgr.y (mgr.34107) 10857 : cluster [DBG] pgmap v11496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:39 smithi121 bash[36902]: audit 2022-04-22T22:44:39.363004+0000 mgr.y (mgr.34107) 10858 : audit [DBG] from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:39 smithi121 bash[36902]: audit 2022-04-22T22:44:39.703095+0000 mgr.y (mgr.34107) 10859 : audit [DBG] from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[45495]: cluster 2022-04-22T22:44:38.009594+0000 mgr.y (mgr.34107) 10857 : cluster [DBG] pgmap v11496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[45495]: audit 2022-04-22T22:44:39.363004+0000 mgr.y (mgr.34107) 10858 : audit [DBG] from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[45495]: audit 2022-04-22T22:44:39.703095+0000 mgr.y (mgr.34107) 10859 : audit [DBG] from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[46934]: cluster 2022-04-22T22:44:38.009594+0000 mgr.y (mgr.34107) 10857 : cluster [DBG] pgmap v11496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[46934]: audit 2022-04-22T22:44:39.363004+0000 mgr.y (mgr.34107) 10858 : audit [DBG] from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:39 smithi115 bash[46934]: audit 2022-04-22T22:44:39.703095+0000 mgr.y (mgr.34107) 10859 : audit [DBG] from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:44:41.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:40 smithi121 bash[36902]: cluster 2022-04-22T22:44:40.010309+0000 mgr.y (mgr.34107) 10860 : cluster [DBG] pgmap v11497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:41.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:40 smithi121 bash[36902]: audit 2022-04-22T22:44:40.128858+0000 mon.c (mon.1) 366 : audit [DBG] from='client.? 172.21.15.115:0/484958274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:41.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:40 smithi121 bash[36902]: audit 2022-04-22T22:44:40.419645+0000 mon.a (mon.0) 15106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[45495]: cluster 2022-04-22T22:44:40.010309+0000 mgr.y (mgr.34107) 10860 : cluster [DBG] pgmap v11497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[45495]: audit 2022-04-22T22:44:40.128858+0000 mon.c (mon.1) 366 : audit [DBG] from='client.? 172.21.15.115:0/484958274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[45495]: audit 2022-04-22T22:44:40.419645+0000 mon.a (mon.0) 15106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[46934]: cluster 2022-04-22T22:44:40.010309+0000 mgr.y (mgr.34107) 10860 : cluster [DBG] pgmap v11497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[46934]: audit 2022-04-22T22:44:40.128858+0000 mon.c (mon.1) 366 : audit [DBG] from='client.? 172.21.15.115:0/484958274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:44:41.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:40 smithi115 bash[46934]: audit 2022-04-22T22:44:40.419645+0000 mon.a (mon.0) 15106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:43 smithi121 bash[36902]: cephadm 2022-04-22T22:44:41.991743+0000 mgr.y (mgr.34107) 10861 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:44:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:43 smithi121 bash[36902]: audit 2022-04-22T22:44:42.003855+0000 mon.a (mon.0) 15107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:43 smithi121 bash[36902]: audit 2022-04-22T22:44:42.010258+0000 mon.a (mon.0) 15108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:43 smithi121 bash[36902]: cluster 2022-04-22T22:44:42.011178+0000 mgr.y (mgr.34107) 10862 : cluster [DBG] pgmap v11498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:43 smithi121 bash[36902]: audit 2022-04-22T22:44:42.038773+0000 mon.a (mon.0) 15109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:42.867Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[45495]: cephadm 2022-04-22T22:44:41.991743+0000 mgr.y (mgr.34107) 10861 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:44:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[45495]: audit 2022-04-22T22:44:42.003855+0000 mon.a (mon.0) 15107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[45495]: audit 2022-04-22T22:44:42.010258+0000 mon.a (mon.0) 15108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[45495]: cluster 2022-04-22T22:44:42.011178+0000 mgr.y (mgr.34107) 10862 : cluster [DBG] pgmap v11498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[45495]: audit 2022-04-22T22:44:42.038773+0000 mon.a (mon.0) 15109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[46934]: cephadm 2022-04-22T22:44:41.991743+0000 mgr.y (mgr.34107) 10861 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T22:44:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[46934]: audit 2022-04-22T22:44:42.003855+0000 mon.a (mon.0) 15107 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[46934]: audit 2022-04-22T22:44:42.010258+0000 mon.a (mon.0) 15108 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:44:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[46934]: cluster 2022-04-22T22:44:42.011178+0000 mgr.y (mgr.34107) 10862 : cluster [DBG] pgmap v11498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:43 smithi115 bash[46934]: audit 2022-04-22T22:44:42.038773+0000 mon.a (mon.0) 15109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:44 smithi115 bash[45495]: audit 2022-04-22T22:44:43.477730+0000 mon.a (mon.0) 15110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:44.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:44 smithi115 bash[46934]: audit 2022-04-22T22:44:43.477730+0000 mon.a (mon.0) 15110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:44 smithi121 bash[36902]: audit 2022-04-22T22:44:43.477730+0000 mon.a (mon.0) 15110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:44.730Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:45 smithi121 bash[36902]: cluster 2022-04-22T22:44:44.012403+0000 mgr.y (mgr.34107) 10863 : cluster [DBG] pgmap v11499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:45 smithi115 bash[45495]: cluster 2022-04-22T22:44:44.012403+0000 mgr.y (mgr.34107) 10863 : cluster [DBG] pgmap v11499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:45 smithi115 bash[46934]: cluster 2022-04-22T22:44:44.012403+0000 mgr.y (mgr.34107) 10863 : cluster [DBG] pgmap v11499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:46 smithi121 bash[36902]: audit 2022-04-22T22:44:45.422845+0000 mon.a (mon.0) 15111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:46 smithi115 bash[45495]: audit 2022-04-22T22:44:45.422845+0000 mon.a (mon.0) 15111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:46 smithi115 bash[46934]: audit 2022-04-22T22:44:45.422845+0000 mon.a (mon.0) 15111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:47 smithi121 bash[36902]: cluster 2022-04-22T22:44:46.013218+0000 mgr.y (mgr.34107) 10864 : cluster [DBG] pgmap v11500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:47 smithi121 bash[36902]: audit 2022-04-22T22:44:47.041976+0000 mon.a (mon.0) 15112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:47 smithi115 bash[45495]: cluster 2022-04-22T22:44:46.013218+0000 mgr.y (mgr.34107) 10864 : cluster [DBG] pgmap v11500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:47 smithi115 bash[45495]: audit 2022-04-22T22:44:47.041976+0000 mon.a (mon.0) 15112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:47 smithi115 bash[46934]: cluster 2022-04-22T22:44:46.013218+0000 mgr.y (mgr.34107) 10864 : cluster [DBG] pgmap v11500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:47 smithi115 bash[46934]: audit 2022-04-22T22:44:47.041976+0000 mon.a (mon.0) 15112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:49 smithi121 bash[36902]: cluster 2022-04-22T22:44:48.014197+0000 mgr.y (mgr.34107) 10865 : cluster [DBG] pgmap v11501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:49 smithi121 bash[36902]: audit 2022-04-22T22:44:48.481512+0000 mon.a (mon.0) 15113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:49.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:49 smithi115 bash[46934]: cluster 2022-04-22T22:44:48.014197+0000 mgr.y (mgr.34107) 10865 : cluster [DBG] pgmap v11501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:49 smithi115 bash[46934]: audit 2022-04-22T22:44:48.481512+0000 mon.a (mon.0) 15113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:49 smithi115 bash[45495]: cluster 2022-04-22T22:44:48.014197+0000 mgr.y (mgr.34107) 10865 : cluster [DBG] pgmap v11501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:49 smithi115 bash[45495]: audit 2022-04-22T22:44:48.481512+0000 mon.a (mon.0) 15113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:51 smithi121 bash[36902]: cluster 2022-04-22T22:44:50.015576+0000 mgr.y (mgr.34107) 10866 : cluster [DBG] pgmap v11502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:51 smithi121 bash[36902]: audit 2022-04-22T22:44:50.425900+0000 mon.a (mon.0) 15114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:51 smithi115 bash[45495]: cluster 2022-04-22T22:44:50.015576+0000 mgr.y (mgr.34107) 10866 : cluster [DBG] pgmap v11502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:51 smithi115 bash[45495]: audit 2022-04-22T22:44:50.425900+0000 mon.a (mon.0) 15114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:51 smithi115 bash[46934]: cluster 2022-04-22T22:44:50.015576+0000 mgr.y (mgr.34107) 10866 : cluster [DBG] pgmap v11502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:51 smithi115 bash[46934]: audit 2022-04-22T22:44:50.425900+0000 mon.a (mon.0) 15114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:52 smithi121 bash[36902]: cluster 2022-04-22T22:44:52.016699+0000 mgr.y (mgr.34107) 10867 : cluster [DBG] pgmap v11503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:52 smithi121 bash[36902]: audit 2022-04-22T22:44:52.045017+0000 mon.a (mon.0) 15115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:52.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:52 smithi115 bash[46934]: cluster 2022-04-22T22:44:52.016699+0000 mgr.y (mgr.34107) 10867 : cluster [DBG] pgmap v11503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:52 smithi115 bash[46934]: audit 2022-04-22T22:44:52.045017+0000 mon.a (mon.0) 15115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:52 smithi115 bash[45495]: cluster 2022-04-22T22:44:52.016699+0000 mgr.y (mgr.34107) 10867 : cluster [DBG] pgmap v11503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:52 smithi115 bash[45495]: audit 2022-04-22T22:44:52.045017+0000 mon.a (mon.0) 15115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:53 smithi121 bash[36902]: audit 2022-04-22T22:44:53.484957+0000 mon.a (mon.0) 15116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:54.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:53 smithi115 bash[46934]: audit 2022-04-22T22:44:53.484957+0000 mon.a (mon.0) 15116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:53 smithi115 bash[45495]: audit 2022-04-22T22:44:53.484957+0000 mon.a (mon.0) 15116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:55.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:54 smithi115 bash[46934]: cluster 2022-04-22T22:44:54.017849+0000 mgr.y (mgr.34107) 10868 : cluster [DBG] pgmap v11504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:44:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:44:54.731Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:44:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:54 smithi115 bash[45495]: cluster 2022-04-22T22:44:54.017849+0000 mgr.y (mgr.34107) 10868 : cluster [DBG] pgmap v11504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:54 smithi121 bash[36902]: cluster 2022-04-22T22:44:54.017849+0000 mgr.y (mgr.34107) 10868 : cluster [DBG] pgmap v11504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:55 smithi121 bash[36902]: audit 2022-04-22T22:44:55.428856+0000 mon.a (mon.0) 15117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:55 smithi115 bash[45495]: audit 2022-04-22T22:44:55.428856+0000 mon.a (mon.0) 15117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:56.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:55 smithi115 bash[46934]: audit 2022-04-22T22:44:55.428856+0000 mon.a (mon.0) 15117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:44:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:56 smithi121 bash[36902]: cluster 2022-04-22T22:44:56.019129+0000 mgr.y (mgr.34107) 10869 : cluster [DBG] pgmap v11505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:57.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:56 smithi115 bash[45495]: cluster 2022-04-22T22:44:56.019129+0000 mgr.y (mgr.34107) 10869 : cluster [DBG] pgmap v11505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:57.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:56 smithi115 bash[46934]: cluster 2022-04-22T22:44:56.019129+0000 mgr.y (mgr.34107) 10869 : cluster [DBG] pgmap v11505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:57 smithi121 bash[36902]: audit 2022-04-22T22:44:57.048139+0000 mon.a (mon.0) 15118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:57 smithi115 bash[45495]: audit 2022-04-22T22:44:57.048139+0000 mon.a (mon.0) 15118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:57 smithi115 bash[46934]: audit 2022-04-22T22:44:57.048139+0000 mon.a (mon.0) 15118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:44:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:58 smithi121 bash[36902]: cluster 2022-04-22T22:44:58.020227+0000 mgr.y (mgr.34107) 10870 : cluster [DBG] pgmap v11506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:44:58 smithi121 bash[36902]: audit 2022-04-22T22:44:58.488936+0000 mon.a (mon.0) 15119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:58 smithi115 bash[45495]: cluster 2022-04-22T22:44:58.020227+0000 mgr.y (mgr.34107) 10870 : cluster [DBG] pgmap v11506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:44:58 smithi115 bash[45495]: audit 2022-04-22T22:44:58.488936+0000 mon.a (mon.0) 15119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:44:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:58 smithi115 bash[46934]: cluster 2022-04-22T22:44:58.020227+0000 mgr.y (mgr.34107) 10870 : cluster [DBG] pgmap v11506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:44:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:44:58 smithi115 bash[46934]: audit 2022-04-22T22:44:58.488936+0000 mon.a (mon.0) 15119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:01 smithi121 bash[36902]: cluster 2022-04-22T22:45:00.021324+0000 mgr.y (mgr.34107) 10871 : cluster [DBG] pgmap v11507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:01 smithi121 bash[36902]: audit 2022-04-22T22:45:00.431892+0000 mon.a (mon.0) 15120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:01 smithi115 bash[46934]: cluster 2022-04-22T22:45:00.021324+0000 mgr.y (mgr.34107) 10871 : cluster [DBG] pgmap v11507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:01 smithi115 bash[46934]: audit 2022-04-22T22:45:00.431892+0000 mon.a (mon.0) 15120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:01 smithi115 bash[45495]: cluster 2022-04-22T22:45:00.021324+0000 mgr.y (mgr.34107) 10871 : cluster [DBG] pgmap v11507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:01 smithi115 bash[45495]: audit 2022-04-22T22:45:00.431892+0000 mon.a (mon.0) 15120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:02 smithi121 bash[36902]: cluster 2022-04-22T22:45:02.022371+0000 mgr.y (mgr.34107) 10872 : cluster [DBG] pgmap v11508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:02 smithi121 bash[36902]: audit 2022-04-22T22:45:02.051306+0000 mon.a (mon.0) 15121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:03.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:02 smithi115 bash[46934]: cluster 2022-04-22T22:45:02.022371+0000 mgr.y (mgr.34107) 10872 : cluster [DBG] pgmap v11508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:03.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:02 smithi115 bash[46934]: audit 2022-04-22T22:45:02.051306+0000 mon.a (mon.0) 15121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:02.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:02 smithi115 bash[45495]: cluster 2022-04-22T22:45:02.022371+0000 mgr.y (mgr.34107) 10872 : cluster [DBG] pgmap v11508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:03.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:02 smithi115 bash[45495]: audit 2022-04-22T22:45:02.051306+0000 mon.a (mon.0) 15121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:03 smithi121 bash[36902]: audit 2022-04-22T22:45:03.492627+0000 mon.a (mon.0) 15122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:03 smithi115 bash[45495]: audit 2022-04-22T22:45:03.492627+0000 mon.a (mon.0) 15122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:04.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:03 smithi115 bash[46934]: audit 2022-04-22T22:45:03.492627+0000 mon.a (mon.0) 15122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:04 smithi115 bash[45495]: cluster 2022-04-22T22:45:04.023542+0000 mgr.y (mgr.34107) 10873 : cluster [DBG] pgmap v11509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:04.731Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:04 smithi115 bash[46934]: cluster 2022-04-22T22:45:04.023542+0000 mgr.y (mgr.34107) 10873 : cluster [DBG] pgmap v11509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:04 smithi121 bash[36902]: cluster 2022-04-22T22:45:04.023542+0000 mgr.y (mgr.34107) 10873 : cluster [DBG] pgmap v11509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:05 smithi121 bash[36902]: audit 2022-04-22T22:45:05.435003+0000 mon.a (mon.0) 15123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:05 smithi115 bash[45495]: audit 2022-04-22T22:45:05.435003+0000 mon.a (mon.0) 15123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:06.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:05 smithi115 bash[46934]: audit 2022-04-22T22:45:05.435003+0000 mon.a (mon.0) 15123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:06 smithi121 bash[36902]: cluster 2022-04-22T22:45:06.024486+0000 mgr.y (mgr.34107) 10874 : cluster [DBG] pgmap v11510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:06 smithi115 bash[45495]: cluster 2022-04-22T22:45:06.024486+0000 mgr.y (mgr.34107) 10874 : cluster [DBG] pgmap v11510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:06 smithi115 bash[46934]: cluster 2022-04-22T22:45:06.024486+0000 mgr.y (mgr.34107) 10874 : cluster [DBG] pgmap v11510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:07 smithi121 bash[36902]: audit 2022-04-22T22:45:07.054522+0000 mon.a (mon.0) 15124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:07 smithi115 bash[45495]: audit 2022-04-22T22:45:07.054522+0000 mon.a (mon.0) 15124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:08.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:07 smithi115 bash[46934]: audit 2022-04-22T22:45:07.054522+0000 mon.a (mon.0) 15124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:08 smithi121 bash[36902]: cluster 2022-04-22T22:45:08.025477+0000 mgr.y (mgr.34107) 10875 : cluster [DBG] pgmap v11511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:08 smithi121 bash[36902]: audit 2022-04-22T22:45:08.496180+0000 mon.a (mon.0) 15125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:08 smithi115 bash[45495]: cluster 2022-04-22T22:45:08.025477+0000 mgr.y (mgr.34107) 10875 : cluster [DBG] pgmap v11511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:08 smithi115 bash[45495]: audit 2022-04-22T22:45:08.496180+0000 mon.a (mon.0) 15125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:08 smithi115 bash[46934]: cluster 2022-04-22T22:45:08.025477+0000 mgr.y (mgr.34107) 10875 : cluster [DBG] pgmap v11511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:08 smithi115 bash[46934]: audit 2022-04-22T22:45:08.496180+0000 mon.a (mon.0) 15125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:10.514 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:45:10.859 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:45:10.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:45:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:45:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:45:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:45:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:45:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:45:11.260 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:45:11.260 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:45:11.261 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:45:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:45:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:45:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:45:11.262 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:45:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:11 smithi121 bash[36902]: cluster 2022-04-22T22:45:10.026609+0000 mgr.y (mgr.34107) 10876 : cluster [DBG] pgmap v11512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:11 smithi121 bash[36902]: audit 2022-04-22T22:45:10.438099+0000 mon.a (mon.0) 15126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:11 smithi121 bash[36902]: audit 2022-04-22T22:45:10.494419+0000 mgr.y (mgr.34107) 10877 : audit [DBG] from='client.46614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:11 smithi121 bash[36902]: audit 2022-04-22T22:45:10.834523+0000 mgr.y (mgr.34107) 10878 : audit [DBG] from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[45495]: cluster 2022-04-22T22:45:10.026609+0000 mgr.y (mgr.34107) 10876 : cluster [DBG] pgmap v11512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[45495]: audit 2022-04-22T22:45:10.438099+0000 mon.a (mon.0) 15126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[45495]: audit 2022-04-22T22:45:10.494419+0000 mgr.y (mgr.34107) 10877 : audit [DBG] from='client.46614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[45495]: audit 2022-04-22T22:45:10.834523+0000 mgr.y (mgr.34107) 10878 : audit [DBG] from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[46934]: cluster 2022-04-22T22:45:10.026609+0000 mgr.y (mgr.34107) 10876 : cluster [DBG] pgmap v11512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[46934]: audit 2022-04-22T22:45:10.438099+0000 mon.a (mon.0) 15126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[46934]: audit 2022-04-22T22:45:10.494419+0000 mgr.y (mgr.34107) 10877 : audit [DBG] from='client.46614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:11 smithi115 bash[46934]: audit 2022-04-22T22:45:10.834523+0000 mgr.y (mgr.34107) 10878 : audit [DBG] from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:12 smithi121 bash[36902]: audit 2022-04-22T22:45:11.256787+0000 mon.c (mon.1) 367 : audit [DBG] from='client.? 172.21.15.115:0/4195035867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:12.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:12 smithi121 bash[36902]: audit 2022-04-22T22:45:12.057431+0000 mon.a (mon.0) 15127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:12 smithi115 bash[45495]: audit 2022-04-22T22:45:11.256787+0000 mon.c (mon.1) 367 : audit [DBG] from='client.? 172.21.15.115:0/4195035867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:12 smithi115 bash[45495]: audit 2022-04-22T22:45:12.057431+0000 mon.a (mon.0) 15127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:12 smithi115 bash[46934]: audit 2022-04-22T22:45:11.256787+0000 mon.c (mon.1) 367 : audit [DBG] from='client.? 172.21.15.115:0/4195035867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:12 smithi115 bash[46934]: audit 2022-04-22T22:45:12.057431+0000 mon.a (mon.0) 15127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:13.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:13 smithi115 bash[46934]: cluster 2022-04-22T22:45:12.027614+0000 mgr.y (mgr.34107) 10879 : cluster [DBG] pgmap v11513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:12.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:13 smithi115 bash[45495]: cluster 2022-04-22T22:45:12.027614+0000 mgr.y (mgr.34107) 10879 : cluster [DBG] pgmap v11513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:13 smithi121 bash[36902]: cluster 2022-04-22T22:45:12.027614+0000 mgr.y (mgr.34107) 10879 : cluster [DBG] pgmap v11513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:14 smithi121 bash[36902]: audit 2022-04-22T22:45:13.499961+0000 mon.a (mon.0) 15128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:14 smithi115 bash[45495]: audit 2022-04-22T22:45:13.499961+0000 mon.a (mon.0) 15128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:14 smithi115 bash[46934]: audit 2022-04-22T22:45:13.499961+0000 mon.a (mon.0) 15128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:14.732Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:15 smithi121 bash[36902]: cluster 2022-04-22T22:45:14.028743+0000 mgr.y (mgr.34107) 10880 : cluster [DBG] pgmap v11514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:15 smithi115 bash[45495]: cluster 2022-04-22T22:45:14.028743+0000 mgr.y (mgr.34107) 10880 : cluster [DBG] pgmap v11514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:15 smithi115 bash[46934]: cluster 2022-04-22T22:45:14.028743+0000 mgr.y (mgr.34107) 10880 : cluster [DBG] pgmap v11514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:16 smithi121 bash[36902]: audit 2022-04-22T22:45:15.441050+0000 mon.a (mon.0) 15129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:16.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:16 smithi115 bash[46934]: audit 2022-04-22T22:45:15.441050+0000 mon.a (mon.0) 15129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:16 smithi115 bash[45495]: audit 2022-04-22T22:45:15.441050+0000 mon.a (mon.0) 15129 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:17 smithi121 bash[36902]: cluster 2022-04-22T22:45:16.029584+0000 mgr.y (mgr.34107) 10881 : cluster [DBG] pgmap v11515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:17 smithi121 bash[36902]: audit 2022-04-22T22:45:16.947597+0000 mon.a (mon.0) 15130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:45:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:17 smithi121 bash[36902]: audit 2022-04-22T22:45:16.948282+0000 mon.a (mon.0) 15131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:45:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:17 smithi121 bash[36902]: audit 2022-04-22T22:45:17.060282+0000 mon.a (mon.0) 15132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[45495]: cluster 2022-04-22T22:45:16.029584+0000 mgr.y (mgr.34107) 10881 : cluster [DBG] pgmap v11515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[45495]: audit 2022-04-22T22:45:16.947597+0000 mon.a (mon.0) 15130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:45:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[45495]: audit 2022-04-22T22:45:16.948282+0000 mon.a (mon.0) 15131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:45:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[45495]: audit 2022-04-22T22:45:17.060282+0000 mon.a (mon.0) 15132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[46934]: cluster 2022-04-22T22:45:16.029584+0000 mgr.y (mgr.34107) 10881 : cluster [DBG] pgmap v11515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[46934]: audit 2022-04-22T22:45:16.947597+0000 mon.a (mon.0) 15130 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:45:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[46934]: audit 2022-04-22T22:45:16.948282+0000 mon.a (mon.0) 15131 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:45:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:17 smithi115 bash[46934]: audit 2022-04-22T22:45:17.060282+0000 mon.a (mon.0) 15132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:19 smithi121 bash[36902]: cluster 2022-04-22T22:45:18.030347+0000 mgr.y (mgr.34107) 10882 : cluster [DBG] pgmap v11516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:19 smithi121 bash[36902]: audit 2022-04-22T22:45:18.503874+0000 mon.a (mon.0) 15133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:19 smithi115 bash[45495]: cluster 2022-04-22T22:45:18.030347+0000 mgr.y (mgr.34107) 10882 : cluster [DBG] pgmap v11516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:19 smithi115 bash[45495]: audit 2022-04-22T22:45:18.503874+0000 mon.a (mon.0) 15133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:19 smithi115 bash[46934]: cluster 2022-04-22T22:45:18.030347+0000 mgr.y (mgr.34107) 10882 : cluster [DBG] pgmap v11516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:19 smithi115 bash[46934]: audit 2022-04-22T22:45:18.503874+0000 mon.a (mon.0) 15133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:21 smithi121 bash[36902]: cluster 2022-04-22T22:45:20.031165+0000 mgr.y (mgr.34107) 10883 : cluster [DBG] pgmap v11517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:21 smithi121 bash[36902]: audit 2022-04-22T22:45:20.444161+0000 mon.a (mon.0) 15134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:21 smithi115 bash[45495]: cluster 2022-04-22T22:45:20.031165+0000 mgr.y (mgr.34107) 10883 : cluster [DBG] pgmap v11517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:21 smithi115 bash[45495]: audit 2022-04-22T22:45:20.444161+0000 mon.a (mon.0) 15134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:21 smithi115 bash[46934]: cluster 2022-04-22T22:45:20.031165+0000 mgr.y (mgr.34107) 10883 : cluster [DBG] pgmap v11517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:21 smithi115 bash[46934]: audit 2022-04-22T22:45:20.444161+0000 mon.a (mon.0) 15134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:22 smithi121 bash[36902]: cluster 2022-04-22T22:45:22.032222+0000 mgr.y (mgr.34107) 10884 : cluster [DBG] pgmap v11518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:22 smithi121 bash[36902]: audit 2022-04-22T22:45:22.063178+0000 mon.a (mon.0) 15135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:22.869Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:22 smithi115 bash[45495]: cluster 2022-04-22T22:45:22.032222+0000 mgr.y (mgr.34107) 10884 : cluster [DBG] pgmap v11518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:22 smithi115 bash[45495]: audit 2022-04-22T22:45:22.063178+0000 mon.a (mon.0) 15135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:22 smithi115 bash[46934]: cluster 2022-04-22T22:45:22.032222+0000 mgr.y (mgr.34107) 10884 : cluster [DBG] pgmap v11518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:22 smithi115 bash[46934]: audit 2022-04-22T22:45:22.063178+0000 mon.a (mon.0) 15135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:23 smithi121 bash[36902]: audit 2022-04-22T22:45:23.507571+0000 mon.a (mon.0) 15136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:23 smithi115 bash[45495]: audit 2022-04-22T22:45:23.507571+0000 mon.a (mon.0) 15136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:23 smithi115 bash[46934]: audit 2022-04-22T22:45:23.507571+0000 mon.a (mon.0) 15136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:24 smithi115 bash[45495]: cluster 2022-04-22T22:45:24.033289+0000 mgr.y (mgr.34107) 10885 : cluster [DBG] pgmap v11519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:24.732Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:24 smithi115 bash[46934]: cluster 2022-04-22T22:45:24.033289+0000 mgr.y (mgr.34107) 10885 : cluster [DBG] pgmap v11519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:24 smithi121 bash[36902]: cluster 2022-04-22T22:45:24.033289+0000 mgr.y (mgr.34107) 10885 : cluster [DBG] pgmap v11519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:25 smithi121 bash[36902]: audit 2022-04-22T22:45:25.447333+0000 mon.a (mon.0) 15137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:25 smithi115 bash[45495]: audit 2022-04-22T22:45:25.447333+0000 mon.a (mon.0) 15137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:25 smithi115 bash[46934]: audit 2022-04-22T22:45:25.447333+0000 mon.a (mon.0) 15137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:26 smithi121 bash[36902]: cluster 2022-04-22T22:45:26.034333+0000 mgr.y (mgr.34107) 10886 : cluster [DBG] pgmap v11520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:26 smithi115 bash[45495]: cluster 2022-04-22T22:45:26.034333+0000 mgr.y (mgr.34107) 10886 : cluster [DBG] pgmap v11520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:26 smithi115 bash[46934]: cluster 2022-04-22T22:45:26.034333+0000 mgr.y (mgr.34107) 10886 : cluster [DBG] pgmap v11520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:27 smithi121 bash[36902]: audit 2022-04-22T22:45:27.066247+0000 mon.a (mon.0) 15138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:27 smithi115 bash[45495]: audit 2022-04-22T22:45:27.066247+0000 mon.a (mon.0) 15138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:27 smithi115 bash[46934]: audit 2022-04-22T22:45:27.066247+0000 mon.a (mon.0) 15138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:28 smithi121 bash[36902]: cluster 2022-04-22T22:45:28.035507+0000 mgr.y (mgr.34107) 10887 : cluster [DBG] pgmap v11521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:28 smithi121 bash[36902]: audit 2022-04-22T22:45:28.511516+0000 mon.a (mon.0) 15139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:28 smithi115 bash[45495]: cluster 2022-04-22T22:45:28.035507+0000 mgr.y (mgr.34107) 10887 : cluster [DBG] pgmap v11521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:28 smithi115 bash[45495]: audit 2022-04-22T22:45:28.511516+0000 mon.a (mon.0) 15139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:28 smithi115 bash[46934]: cluster 2022-04-22T22:45:28.035507+0000 mgr.y (mgr.34107) 10887 : cluster [DBG] pgmap v11521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:28 smithi115 bash[46934]: audit 2022-04-22T22:45:28.511516+0000 mon.a (mon.0) 15139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:31 smithi121 bash[36902]: cluster 2022-04-22T22:45:30.036525+0000 mgr.y (mgr.34107) 10888 : cluster [DBG] pgmap v11522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:31 smithi121 bash[36902]: audit 2022-04-22T22:45:30.450290+0000 mon.a (mon.0) 15140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:31 smithi115 bash[45495]: cluster 2022-04-22T22:45:30.036525+0000 mgr.y (mgr.34107) 10888 : cluster [DBG] pgmap v11522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:31 smithi115 bash[45495]: audit 2022-04-22T22:45:30.450290+0000 mon.a (mon.0) 15140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:31 smithi115 bash[46934]: cluster 2022-04-22T22:45:30.036525+0000 mgr.y (mgr.34107) 10888 : cluster [DBG] pgmap v11522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:31 smithi115 bash[46934]: audit 2022-04-22T22:45:30.450290+0000 mon.a (mon.0) 15140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:32 smithi121 bash[36902]: cluster 2022-04-22T22:45:32.037586+0000 mgr.y (mgr.34107) 10889 : cluster [DBG] pgmap v11523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:32 smithi121 bash[36902]: audit 2022-04-22T22:45:32.069510+0000 mon.a (mon.0) 15141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:32.871Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:32 smithi115 bash[45495]: cluster 2022-04-22T22:45:32.037586+0000 mgr.y (mgr.34107) 10889 : cluster [DBG] pgmap v11523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:32 smithi115 bash[45495]: audit 2022-04-22T22:45:32.069510+0000 mon.a (mon.0) 15141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:32 smithi115 bash[46934]: cluster 2022-04-22T22:45:32.037586+0000 mgr.y (mgr.34107) 10889 : cluster [DBG] pgmap v11523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:32 smithi115 bash[46934]: audit 2022-04-22T22:45:32.069510+0000 mon.a (mon.0) 15141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:33 smithi121 bash[36902]: audit 2022-04-22T22:45:33.515505+0000 mon.a (mon.0) 15142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:33 smithi115 bash[45495]: audit 2022-04-22T22:45:33.515505+0000 mon.a (mon.0) 15142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:33 smithi115 bash[46934]: audit 2022-04-22T22:45:33.515505+0000 mon.a (mon.0) 15142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:34 smithi115 bash[46934]: cluster 2022-04-22T22:45:34.038755+0000 mgr.y (mgr.34107) 10890 : cluster [DBG] pgmap v11524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:34.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:34 smithi115 bash[45495]: cluster 2022-04-22T22:45:34.038755+0000 mgr.y (mgr.34107) 10890 : cluster [DBG] pgmap v11524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:34 smithi121 bash[36902]: cluster 2022-04-22T22:45:34.038755+0000 mgr.y (mgr.34107) 10890 : cluster [DBG] pgmap v11524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:35 smithi121 bash[36902]: audit 2022-04-22T22:45:35.453360+0000 mon.a (mon.0) 15143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:35 smithi115 bash[45495]: audit 2022-04-22T22:45:35.453360+0000 mon.a (mon.0) 15143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:35 smithi115 bash[46934]: audit 2022-04-22T22:45:35.453360+0000 mon.a (mon.0) 15143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:36 smithi121 bash[36902]: cluster 2022-04-22T22:45:36.039829+0000 mgr.y (mgr.34107) 10891 : cluster [DBG] pgmap v11525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:36 smithi115 bash[45495]: cluster 2022-04-22T22:45:36.039829+0000 mgr.y (mgr.34107) 10891 : cluster [DBG] pgmap v11525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:36 smithi115 bash[46934]: cluster 2022-04-22T22:45:36.039829+0000 mgr.y (mgr.34107) 10891 : cluster [DBG] pgmap v11525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:37 smithi121 bash[36902]: audit 2022-04-22T22:45:37.072605+0000 mon.a (mon.0) 15144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:37 smithi115 bash[45495]: audit 2022-04-22T22:45:37.072605+0000 mon.a (mon.0) 15144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:37 smithi115 bash[46934]: audit 2022-04-22T22:45:37.072605+0000 mon.a (mon.0) 15144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:38 smithi121 bash[36902]: cluster 2022-04-22T22:45:38.040922+0000 mgr.y (mgr.34107) 10892 : cluster [DBG] pgmap v11526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:38 smithi121 bash[36902]: audit 2022-04-22T22:45:38.518874+0000 mon.a (mon.0) 15145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:38 smithi115 bash[46934]: cluster 2022-04-22T22:45:38.040922+0000 mgr.y (mgr.34107) 10892 : cluster [DBG] pgmap v11526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:38 smithi115 bash[46934]: audit 2022-04-22T22:45:38.518874+0000 mon.a (mon.0) 15145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:38 smithi115 bash[45495]: cluster 2022-04-22T22:45:38.040922+0000 mgr.y (mgr.34107) 10892 : cluster [DBG] pgmap v11526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:38 smithi115 bash[45495]: audit 2022-04-22T22:45:38.518874+0000 mon.a (mon.0) 15145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:41 smithi121 bash[36902]: cluster 2022-04-22T22:45:40.041968+0000 mgr.y (mgr.34107) 10893 : cluster [DBG] pgmap v11527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:41 smithi121 bash[36902]: audit 2022-04-22T22:45:40.456400+0000 mon.a (mon.0) 15146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:41 smithi115 bash[45495]: cluster 2022-04-22T22:45:40.041968+0000 mgr.y (mgr.34107) 10893 : cluster [DBG] pgmap v11527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:41 smithi115 bash[45495]: audit 2022-04-22T22:45:40.456400+0000 mon.a (mon.0) 15146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:41 smithi115 bash[46934]: cluster 2022-04-22T22:45:40.041968+0000 mgr.y (mgr.34107) 10893 : cluster [DBG] pgmap v11527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:41 smithi115 bash[46934]: audit 2022-04-22T22:45:40.456400+0000 mon.a (mon.0) 15146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:41.634 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:45:41.990 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:45:41.990 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 10m ago 5h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 10m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 10m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 10m ago 5h 288M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:45:41.991 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 10m ago 5h 10.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 10m ago 5h 587M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 10m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 10m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 10m ago 5h 650M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:45:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 10m ago 5h 68.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:45:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[46934]: audit 2022-04-22T22:45:41.615188+0000 mgr.y (mgr.34107) 10894 : audit [DBG] from='client.56569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[46934]: audit 2022-04-22T22:45:42.019589+0000 mon.a (mon.0) 15147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[46934]: audit 2022-04-22T22:45:42.022505+0000 mon.a (mon.0) 15148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[46934]: audit 2022-04-22T22:45:42.024205+0000 mon.a (mon.0) 15149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[46934]: audit 2022-04-22T22:45:42.075383+0000 mon.a (mon.0) 15150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[45495]: audit 2022-04-22T22:45:41.615188+0000 mgr.y (mgr.34107) 10894 : audit [DBG] from='client.56569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[45495]: audit 2022-04-22T22:45:42.019589+0000 mon.a (mon.0) 15147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[45495]: audit 2022-04-22T22:45:42.022505+0000 mon.a (mon.0) 15148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:45:42.370 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[45495]: audit 2022-04-22T22:45:42.024205+0000 mon.a (mon.0) 15149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:45:42.371 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[45495]: audit 2022-04-22T22:45:42.075383+0000 mon.a (mon.0) 15150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:42.388 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:45:42.389 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:45:42.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:42 smithi121 bash[36902]: audit 2022-04-22T22:45:41.615188+0000 mgr.y (mgr.34107) 10894 : audit [DBG] from='client.56569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:42 smithi121 bash[36902]: audit 2022-04-22T22:45:42.019589+0000 mon.a (mon.0) 15147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:45:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:42 smithi121 bash[36902]: audit 2022-04-22T22:45:42.022505+0000 mon.a (mon.0) 15148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:45:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:42 smithi121 bash[36902]: audit 2022-04-22T22:45:42.024205+0000 mon.a (mon.0) 15149 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:45:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:42 smithi121 bash[36902]: audit 2022-04-22T22:45:42.075383+0000 mon.a (mon.0) 15150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:43.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:42.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[46934]: audit 2022-04-22T22:45:41.966306+0000 mgr.y (mgr.34107) 10895 : audit [DBG] from='client.46638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:43.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[46934]: cluster 2022-04-22T22:45:42.042660+0000 mgr.y (mgr.34107) 10896 : cluster [DBG] pgmap v11528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[46934]: audit 2022-04-22T22:45:42.384723+0000 mon.a (mon.0) 15151 : audit [DBG] from='client.? 172.21.15.115:0/21888033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[46934]: audit 2022-04-22T22:45:42.627312+0000 mon.a (mon.0) 15152 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[45495]: audit 2022-04-22T22:45:41.966306+0000 mgr.y (mgr.34107) 10895 : audit [DBG] from='client.46638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[45495]: cluster 2022-04-22T22:45:42.042660+0000 mgr.y (mgr.34107) 10896 : cluster [DBG] pgmap v11528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[45495]: audit 2022-04-22T22:45:42.384723+0000 mon.a (mon.0) 15151 : audit [DBG] from='client.? 172.21.15.115:0/21888033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:43.296 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:43 smithi115 bash[45495]: audit 2022-04-22T22:45:42.627312+0000 mon.a (mon.0) 15152 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:43 smithi121 bash[36902]: audit 2022-04-22T22:45:41.966306+0000 mgr.y (mgr.34107) 10895 : audit [DBG] from='client.46638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:45:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:43 smithi121 bash[36902]: cluster 2022-04-22T22:45:42.042660+0000 mgr.y (mgr.34107) 10896 : cluster [DBG] pgmap v11528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:43 smithi121 bash[36902]: audit 2022-04-22T22:45:42.384723+0000 mon.a (mon.0) 15151 : audit [DBG] from='client.? 172.21.15.115:0/21888033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:45:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:43 smithi121 bash[36902]: audit 2022-04-22T22:45:42.627312+0000 mon.a (mon.0) 15152 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:44 smithi121 bash[36902]: audit 2022-04-22T22:45:43.522502+0000 mon.a (mon.0) 15153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:44.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:44 smithi115 bash[46934]: audit 2022-04-22T22:45:43.522502+0000 mon.a (mon.0) 15153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:44 smithi115 bash[45495]: audit 2022-04-22T22:45:43.522502+0000 mon.a (mon.0) 15153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:44.733Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:45 smithi121 bash[36902]: cluster 2022-04-22T22:45:44.043876+0000 mgr.y (mgr.34107) 10897 : cluster [DBG] pgmap v11529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:45 smithi115 bash[45495]: cluster 2022-04-22T22:45:44.043876+0000 mgr.y (mgr.34107) 10897 : cluster [DBG] pgmap v11529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:45 smithi115 bash[46934]: cluster 2022-04-22T22:45:44.043876+0000 mgr.y (mgr.34107) 10897 : cluster [DBG] pgmap v11529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:46.452 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:46 smithi121 bash[36902]: audit 2022-04-22T22:45:45.459180+0000 mon.a (mon.0) 15154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:46 smithi115 bash[45495]: audit 2022-04-22T22:45:45.459180+0000 mon.a (mon.0) 15154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:46 smithi115 bash[46934]: audit 2022-04-22T22:45:45.459180+0000 mon.a (mon.0) 15154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:47 smithi121 bash[36902]: cluster 2022-04-22T22:45:46.044922+0000 mgr.y (mgr.34107) 10898 : cluster [DBG] pgmap v11530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:47 smithi121 bash[36902]: audit 2022-04-22T22:45:46.467423+0000 mon.a (mon.0) 15155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:47 smithi121 bash[36902]: audit 2022-04-22T22:45:46.791837+0000 mon.a (mon.0) 15156 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:47 smithi121 bash[36902]: audit 2022-04-22T22:45:47.077704+0000 mon.a (mon.0) 15157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[45495]: cluster 2022-04-22T22:45:46.044922+0000 mgr.y (mgr.34107) 10898 : cluster [DBG] pgmap v11530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[45495]: audit 2022-04-22T22:45:46.467423+0000 mon.a (mon.0) 15155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[45495]: audit 2022-04-22T22:45:46.791837+0000 mon.a (mon.0) 15156 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[45495]: audit 2022-04-22T22:45:47.077704+0000 mon.a (mon.0) 15157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[46934]: cluster 2022-04-22T22:45:46.044922+0000 mgr.y (mgr.34107) 10898 : cluster [DBG] pgmap v11530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[46934]: audit 2022-04-22T22:45:46.467423+0000 mon.a (mon.0) 15155 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[46934]: audit 2022-04-22T22:45:46.791837+0000 mon.a (mon.0) 15156 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:45:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:47 smithi115 bash[46934]: audit 2022-04-22T22:45:47.077704+0000 mon.a (mon.0) 15157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:49 smithi121 bash[36902]: cluster 2022-04-22T22:45:48.045990+0000 mgr.y (mgr.34107) 10899 : cluster [DBG] pgmap v11531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:49 smithi121 bash[36902]: audit 2022-04-22T22:45:48.526218+0000 mon.a (mon.0) 15158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:49.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:49 smithi115 bash[45495]: cluster 2022-04-22T22:45:48.045990+0000 mgr.y (mgr.34107) 10899 : cluster [DBG] pgmap v11531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:49 smithi115 bash[45495]: audit 2022-04-22T22:45:48.526218+0000 mon.a (mon.0) 15158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:49 smithi115 bash[46934]: cluster 2022-04-22T22:45:48.045990+0000 mgr.y (mgr.34107) 10899 : cluster [DBG] pgmap v11531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:49 smithi115 bash[46934]: audit 2022-04-22T22:45:48.526218+0000 mon.a (mon.0) 15158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:50 smithi121 bash[36902]: audit 2022-04-22T22:45:50.464189+0000 mon.a (mon.0) 15159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:50.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:50 smithi115 bash[46934]: audit 2022-04-22T22:45:50.464189+0000 mon.a (mon.0) 15159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:50 smithi115 bash[45495]: audit 2022-04-22T22:45:50.464189+0000 mon.a (mon.0) 15159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:51 smithi121 bash[36902]: cluster 2022-04-22T22:45:50.046968+0000 mgr.y (mgr.34107) 10900 : cluster [DBG] pgmap v11532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:51.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:51 smithi115 bash[46934]: cluster 2022-04-22T22:45:50.046968+0000 mgr.y (mgr.34107) 10900 : cluster [DBG] pgmap v11532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:51 smithi115 bash[45495]: cluster 2022-04-22T22:45:50.046968+0000 mgr.y (mgr.34107) 10900 : cluster [DBG] pgmap v11532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:52 smithi121 bash[36902]: cluster 2022-04-22T22:45:52.048089+0000 mgr.y (mgr.34107) 10901 : cluster [DBG] pgmap v11533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:52 smithi121 bash[36902]: audit 2022-04-22T22:45:52.080803+0000 mon.a (mon.0) 15160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:52.871Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:52 smithi115 bash[45495]: cluster 2022-04-22T22:45:52.048089+0000 mgr.y (mgr.34107) 10901 : cluster [DBG] pgmap v11533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:52 smithi115 bash[45495]: audit 2022-04-22T22:45:52.080803+0000 mon.a (mon.0) 15160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:52 smithi115 bash[46934]: cluster 2022-04-22T22:45:52.048089+0000 mgr.y (mgr.34107) 10901 : cluster [DBG] pgmap v11533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:52 smithi115 bash[46934]: audit 2022-04-22T22:45:52.080803+0000 mon.a (mon.0) 15160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:53 smithi121 bash[36902]: audit 2022-04-22T22:45:53.529944+0000 mon.a (mon.0) 15161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:53 smithi115 bash[45495]: audit 2022-04-22T22:45:53.529944+0000 mon.a (mon.0) 15161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:53 smithi115 bash[46934]: audit 2022-04-22T22:45:53.529944+0000 mon.a (mon.0) 15161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:54 smithi115 bash[45495]: cluster 2022-04-22T22:45:54.049335+0000 mgr.y (mgr.34107) 10902 : cluster [DBG] pgmap v11534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:45:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:45:54.734Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:45:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:54 smithi115 bash[46934]: cluster 2022-04-22T22:45:54.049335+0000 mgr.y (mgr.34107) 10902 : cluster [DBG] pgmap v11534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:54 smithi121 bash[36902]: cluster 2022-04-22T22:45:54.049335+0000 mgr.y (mgr.34107) 10902 : cluster [DBG] pgmap v11534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:55 smithi121 bash[36902]: audit 2022-04-22T22:45:55.467344+0000 mon.a (mon.0) 15162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:55 smithi115 bash[45495]: audit 2022-04-22T22:45:55.467344+0000 mon.a (mon.0) 15162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:55 smithi115 bash[46934]: audit 2022-04-22T22:45:55.467344+0000 mon.a (mon.0) 15162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:45:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:56 smithi121 bash[36902]: cluster 2022-04-22T22:45:56.050404+0000 mgr.y (mgr.34107) 10903 : cluster [DBG] pgmap v11535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:56 smithi115 bash[45495]: cluster 2022-04-22T22:45:56.050404+0000 mgr.y (mgr.34107) 10903 : cluster [DBG] pgmap v11535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:56 smithi115 bash[46934]: cluster 2022-04-22T22:45:56.050404+0000 mgr.y (mgr.34107) 10903 : cluster [DBG] pgmap v11535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:57 smithi121 bash[36902]: audit 2022-04-22T22:45:57.083942+0000 mon.a (mon.0) 15163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:57 smithi115 bash[45495]: audit 2022-04-22T22:45:57.083942+0000 mon.a (mon.0) 15163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:57 smithi115 bash[46934]: audit 2022-04-22T22:45:57.083942+0000 mon.a (mon.0) 15163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:45:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:58 smithi121 bash[36902]: cluster 2022-04-22T22:45:58.051469+0000 mgr.y (mgr.34107) 10904 : cluster [DBG] pgmap v11536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:45:58 smithi121 bash[36902]: audit 2022-04-22T22:45:58.533420+0000 mon.a (mon.0) 15164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:58 smithi115 bash[45495]: cluster 2022-04-22T22:45:58.051469+0000 mgr.y (mgr.34107) 10904 : cluster [DBG] pgmap v11536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:45:58 smithi115 bash[45495]: audit 2022-04-22T22:45:58.533420+0000 mon.a (mon.0) 15164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:45:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:58 smithi115 bash[46934]: cluster 2022-04-22T22:45:58.051469+0000 mgr.y (mgr.34107) 10904 : cluster [DBG] pgmap v11536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:45:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:45:58 smithi115 bash[46934]: audit 2022-04-22T22:45:58.533420+0000 mon.a (mon.0) 15164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:01 smithi121 bash[36902]: cluster 2022-04-22T22:46:00.052446+0000 mgr.y (mgr.34107) 10905 : cluster [DBG] pgmap v11537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:01 smithi121 bash[36902]: audit 2022-04-22T22:46:00.470377+0000 mon.a (mon.0) 15165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:01 smithi115 bash[46934]: cluster 2022-04-22T22:46:00.052446+0000 mgr.y (mgr.34107) 10905 : cluster [DBG] pgmap v11537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:01 smithi115 bash[46934]: audit 2022-04-22T22:46:00.470377+0000 mon.a (mon.0) 15165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:01 smithi115 bash[45495]: cluster 2022-04-22T22:46:00.052446+0000 mgr.y (mgr.34107) 10905 : cluster [DBG] pgmap v11537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:01 smithi115 bash[45495]: audit 2022-04-22T22:46:00.470377+0000 mon.a (mon.0) 15165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:02 smithi121 bash[36902]: cluster 2022-04-22T22:46:02.053473+0000 mgr.y (mgr.34107) 10906 : cluster [DBG] pgmap v11538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:02 smithi121 bash[36902]: audit 2022-04-22T22:46:02.087244+0000 mon.a (mon.0) 15166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:02.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:02 smithi115 bash[45495]: cluster 2022-04-22T22:46:02.053473+0000 mgr.y (mgr.34107) 10906 : cluster [DBG] pgmap v11538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:02 smithi115 bash[45495]: audit 2022-04-22T22:46:02.087244+0000 mon.a (mon.0) 15166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:02 smithi115 bash[46934]: cluster 2022-04-22T22:46:02.053473+0000 mgr.y (mgr.34107) 10906 : cluster [DBG] pgmap v11538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:02 smithi115 bash[46934]: audit 2022-04-22T22:46:02.087244+0000 mon.a (mon.0) 15166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:03 smithi121 bash[36902]: audit 2022-04-22T22:46:03.537629+0000 mon.a (mon.0) 15167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:03 smithi115 bash[45495]: audit 2022-04-22T22:46:03.537629+0000 mon.a (mon.0) 15167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:03 smithi115 bash[46934]: audit 2022-04-22T22:46:03.537629+0000 mon.a (mon.0) 15167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:04 smithi115 bash[45495]: cluster 2022-04-22T22:46:04.054644+0000 mgr.y (mgr.34107) 10907 : cluster [DBG] pgmap v11539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:04 smithi115 bash[46934]: cluster 2022-04-22T22:46:04.054644+0000 mgr.y (mgr.34107) 10907 : cluster [DBG] pgmap v11539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:04.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:04 smithi121 bash[36902]: cluster 2022-04-22T22:46:04.054644+0000 mgr.y (mgr.34107) 10907 : cluster [DBG] pgmap v11539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:05 smithi121 bash[36902]: audit 2022-04-22T22:46:05.473618+0000 mon.a (mon.0) 15168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:05 smithi115 bash[45495]: audit 2022-04-22T22:46:05.473618+0000 mon.a (mon.0) 15168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:05 smithi115 bash[46934]: audit 2022-04-22T22:46:05.473618+0000 mon.a (mon.0) 15168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:06 smithi121 bash[36902]: cluster 2022-04-22T22:46:06.055880+0000 mgr.y (mgr.34107) 10908 : cluster [DBG] pgmap v11540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:06 smithi115 bash[45495]: cluster 2022-04-22T22:46:06.055880+0000 mgr.y (mgr.34107) 10908 : cluster [DBG] pgmap v11540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:06 smithi115 bash[46934]: cluster 2022-04-22T22:46:06.055880+0000 mgr.y (mgr.34107) 10908 : cluster [DBG] pgmap v11540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:07 smithi121 bash[36902]: audit 2022-04-22T22:46:07.090328+0000 mon.a (mon.0) 15169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:07 smithi115 bash[45495]: audit 2022-04-22T22:46:07.090328+0000 mon.a (mon.0) 15169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:07 smithi115 bash[46934]: audit 2022-04-22T22:46:07.090328+0000 mon.a (mon.0) 15169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:08 smithi121 bash[36902]: cluster 2022-04-22T22:46:08.056992+0000 mgr.y (mgr.34107) 10909 : cluster [DBG] pgmap v11541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:08 smithi121 bash[36902]: audit 2022-04-22T22:46:08.541452+0000 mon.a (mon.0) 15170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:08 smithi115 bash[45495]: cluster 2022-04-22T22:46:08.056992+0000 mgr.y (mgr.34107) 10909 : cluster [DBG] pgmap v11541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:08 smithi115 bash[45495]: audit 2022-04-22T22:46:08.541452+0000 mon.a (mon.0) 15170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:08 smithi115 bash[46934]: cluster 2022-04-22T22:46:08.056992+0000 mgr.y (mgr.34107) 10909 : cluster [DBG] pgmap v11541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:08 smithi115 bash[46934]: audit 2022-04-22T22:46:08.541452+0000 mon.a (mon.0) 15170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:11 smithi121 bash[36902]: cluster 2022-04-22T22:46:10.058003+0000 mgr.y (mgr.34107) 10910 : cluster [DBG] pgmap v11542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:11 smithi121 bash[36902]: audit 2022-04-22T22:46:10.476878+0000 mon.a (mon.0) 15171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:11 smithi115 bash[45495]: cluster 2022-04-22T22:46:10.058003+0000 mgr.y (mgr.34107) 10910 : cluster [DBG] pgmap v11542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:11 smithi115 bash[45495]: audit 2022-04-22T22:46:10.476878+0000 mon.a (mon.0) 15171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:11 smithi115 bash[46934]: cluster 2022-04-22T22:46:10.058003+0000 mgr.y (mgr.34107) 10910 : cluster [DBG] pgmap v11542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:11 smithi115 bash[46934]: audit 2022-04-22T22:46:10.476878+0000 mon.a (mon.0) 15171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:12.763 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 26s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 26s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 26s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:46:13.108 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 26s ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 26s ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:46:13.109 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 26s ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:46:13.110 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 26s ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:46:13.110 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 26s ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:46:13.110 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 26s ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:46:13.110 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 26s ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:46:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:12 smithi121 bash[36902]: cluster 2022-04-22T22:46:12.059079+0000 mgr.y (mgr.34107) 10911 : cluster [DBG] pgmap v11543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:12 smithi121 bash[36902]: audit 2022-04-22T22:46:12.093511+0000 mon.a (mon.0) 15172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:12 smithi121 bash[36902]: audit 2022-04-22T22:46:12.743617+0000 mgr.y (mgr.34107) 10912 : audit [DBG] from='client.46650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:13.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[46934]: cluster 2022-04-22T22:46:12.059079+0000 mgr.y (mgr.34107) 10911 : cluster [DBG] pgmap v11543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[46934]: audit 2022-04-22T22:46:12.093511+0000 mon.a (mon.0) 15172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[46934]: audit 2022-04-22T22:46:12.743617+0000 mgr.y (mgr.34107) 10912 : audit [DBG] from='client.46650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:13.294 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:12.873Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:13.294 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[45495]: cluster 2022-04-22T22:46:12.059079+0000 mgr.y (mgr.34107) 10911 : cluster [DBG] pgmap v11543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[45495]: audit 2022-04-22T22:46:12.093511+0000 mon.a (mon.0) 15172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:13.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:12 smithi115 bash[45495]: audit 2022-04-22T22:46:12.743617+0000 mgr.y (mgr.34107) 10912 : audit [DBG] from='client.46650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:46:13.515 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:46:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:13 smithi121 bash[36902]: audit 2022-04-22T22:46:13.084622+0000 mgr.y (mgr.34107) 10913 : audit [DBG] from='client.46656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:13 smithi121 bash[36902]: audit 2022-04-22T22:46:13.510233+0000 mon.c (mon.1) 368 : audit [DBG] from='client.? 172.21.15.115:0/3634696589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:13 smithi121 bash[36902]: audit 2022-04-22T22:46:13.544969+0000 mon.a (mon.0) 15173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[46934]: audit 2022-04-22T22:46:13.084622+0000 mgr.y (mgr.34107) 10913 : audit [DBG] from='client.46656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[46934]: audit 2022-04-22T22:46:13.510233+0000 mon.c (mon.1) 368 : audit [DBG] from='client.? 172.21.15.115:0/3634696589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[46934]: audit 2022-04-22T22:46:13.544969+0000 mon.a (mon.0) 15173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[45495]: audit 2022-04-22T22:46:13.084622+0000 mgr.y (mgr.34107) 10913 : audit [DBG] from='client.46656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[45495]: audit 2022-04-22T22:46:13.510233+0000 mon.c (mon.1) 368 : audit [DBG] from='client.? 172.21.15.115:0/3634696589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:13 smithi115 bash[45495]: audit 2022-04-22T22:46:13.544969+0000 mon.a (mon.0) 15173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:14 smithi115 bash[45495]: cluster 2022-04-22T22:46:14.060305+0000 mgr.y (mgr.34107) 10914 : cluster [DBG] pgmap v11544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:14 smithi115 bash[46934]: cluster 2022-04-22T22:46:14.060305+0000 mgr.y (mgr.34107) 10914 : cluster [DBG] pgmap v11544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:14.736Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:14 smithi121 bash[36902]: cluster 2022-04-22T22:46:14.060305+0000 mgr.y (mgr.34107) 10914 : cluster [DBG] pgmap v11544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:15 smithi121 bash[36902]: audit 2022-04-22T22:46:15.479909+0000 mon.a (mon.0) 15174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:15 smithi115 bash[45495]: audit 2022-04-22T22:46:15.479909+0000 mon.a (mon.0) 15174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:15 smithi115 bash[46934]: audit 2022-04-22T22:46:15.479909+0000 mon.a (mon.0) 15174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:16 smithi121 bash[36902]: cluster 2022-04-22T22:46:16.061453+0000 mgr.y (mgr.34107) 10915 : cluster [DBG] pgmap v11545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:16 smithi115 bash[45495]: cluster 2022-04-22T22:46:16.061453+0000 mgr.y (mgr.34107) 10915 : cluster [DBG] pgmap v11545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:16 smithi115 bash[46934]: cluster 2022-04-22T22:46:16.061453+0000 mgr.y (mgr.34107) 10915 : cluster [DBG] pgmap v11545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:17 smithi121 bash[36902]: audit 2022-04-22T22:46:16.955891+0000 mon.a (mon.0) 15175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:46:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:17 smithi121 bash[36902]: audit 2022-04-22T22:46:16.960013+0000 mon.a (mon.0) 15176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:46:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:17 smithi121 bash[36902]: audit 2022-04-22T22:46:17.096524+0000 mon.a (mon.0) 15177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:18.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[46934]: audit 2022-04-22T22:46:16.955891+0000 mon.a (mon.0) 15175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:46:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[46934]: audit 2022-04-22T22:46:16.960013+0000 mon.a (mon.0) 15176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:46:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[46934]: audit 2022-04-22T22:46:17.096524+0000 mon.a (mon.0) 15177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[45495]: audit 2022-04-22T22:46:16.955891+0000 mon.a (mon.0) 15175 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:46:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[45495]: audit 2022-04-22T22:46:16.960013+0000 mon.a (mon.0) 15176 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:46:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:17 smithi115 bash[45495]: audit 2022-04-22T22:46:17.096524+0000 mon.a (mon.0) 15177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:18 smithi121 bash[36902]: cluster 2022-04-22T22:46:18.062500+0000 mgr.y (mgr.34107) 10916 : cluster [DBG] pgmap v11546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:18 smithi121 bash[36902]: audit 2022-04-22T22:46:18.548528+0000 mon.a (mon.0) 15178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:18 smithi115 bash[45495]: cluster 2022-04-22T22:46:18.062500+0000 mgr.y (mgr.34107) 10916 : cluster [DBG] pgmap v11546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:18 smithi115 bash[45495]: audit 2022-04-22T22:46:18.548528+0000 mon.a (mon.0) 15178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:18 smithi115 bash[46934]: cluster 2022-04-22T22:46:18.062500+0000 mgr.y (mgr.34107) 10916 : cluster [DBG] pgmap v11546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:18 smithi115 bash[46934]: audit 2022-04-22T22:46:18.548528+0000 mon.a (mon.0) 15178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:21 smithi121 bash[36902]: cluster 2022-04-22T22:46:20.063598+0000 mgr.y (mgr.34107) 10917 : cluster [DBG] pgmap v11547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:21 smithi121 bash[36902]: audit 2022-04-22T22:46:20.482756+0000 mon.a (mon.0) 15179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:21 smithi115 bash[45495]: cluster 2022-04-22T22:46:20.063598+0000 mgr.y (mgr.34107) 10917 : cluster [DBG] pgmap v11547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:21 smithi115 bash[45495]: audit 2022-04-22T22:46:20.482756+0000 mon.a (mon.0) 15179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:21 smithi115 bash[46934]: cluster 2022-04-22T22:46:20.063598+0000 mgr.y (mgr.34107) 10917 : cluster [DBG] pgmap v11547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:21 smithi115 bash[46934]: audit 2022-04-22T22:46:20.482756+0000 mon.a (mon.0) 15179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:22 smithi121 bash[36902]: cluster 2022-04-22T22:46:22.064690+0000 mgr.y (mgr.34107) 10918 : cluster [DBG] pgmap v11548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:22 smithi121 bash[36902]: audit 2022-04-22T22:46:22.099435+0000 mon.a (mon.0) 15180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:22.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:22 smithi115 bash[45495]: cluster 2022-04-22T22:46:22.064690+0000 mgr.y (mgr.34107) 10918 : cluster [DBG] pgmap v11548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:22 smithi115 bash[45495]: audit 2022-04-22T22:46:22.099435+0000 mon.a (mon.0) 15180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:22 smithi115 bash[46934]: cluster 2022-04-22T22:46:22.064690+0000 mgr.y (mgr.34107) 10918 : cluster [DBG] pgmap v11548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:22 smithi115 bash[46934]: audit 2022-04-22T22:46:22.099435+0000 mon.a (mon.0) 15180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:23 smithi121 bash[36902]: audit 2022-04-22T22:46:23.552310+0000 mon.a (mon.0) 15181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:23 smithi115 bash[45495]: audit 2022-04-22T22:46:23.552310+0000 mon.a (mon.0) 15181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:23 smithi115 bash[46934]: audit 2022-04-22T22:46:23.552310+0000 mon.a (mon.0) 15181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:24 smithi115 bash[45495]: cluster 2022-04-22T22:46:24.065859+0000 mgr.y (mgr.34107) 10919 : cluster [DBG] pgmap v11549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:24.735Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:24 smithi115 bash[46934]: cluster 2022-04-22T22:46:24.065859+0000 mgr.y (mgr.34107) 10919 : cluster [DBG] pgmap v11549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:24 smithi121 bash[36902]: cluster 2022-04-22T22:46:24.065859+0000 mgr.y (mgr.34107) 10919 : cluster [DBG] pgmap v11549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:25 smithi121 bash[36902]: audit 2022-04-22T22:46:25.485778+0000 mon.a (mon.0) 15182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:25 smithi115 bash[45495]: audit 2022-04-22T22:46:25.485778+0000 mon.a (mon.0) 15182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:25 smithi115 bash[46934]: audit 2022-04-22T22:46:25.485778+0000 mon.a (mon.0) 15182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:26 smithi121 bash[36902]: cluster 2022-04-22T22:46:26.067010+0000 mgr.y (mgr.34107) 10920 : cluster [DBG] pgmap v11550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:26 smithi115 bash[45495]: cluster 2022-04-22T22:46:26.067010+0000 mgr.y (mgr.34107) 10920 : cluster [DBG] pgmap v11550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:26 smithi115 bash[46934]: cluster 2022-04-22T22:46:26.067010+0000 mgr.y (mgr.34107) 10920 : cluster [DBG] pgmap v11550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:27 smithi121 bash[36902]: audit 2022-04-22T22:46:27.101948+0000 mon.a (mon.0) 15183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:27 smithi115 bash[45495]: audit 2022-04-22T22:46:27.101948+0000 mon.a (mon.0) 15183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:27 smithi115 bash[46934]: audit 2022-04-22T22:46:27.101948+0000 mon.a (mon.0) 15183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:28 smithi121 bash[36902]: cluster 2022-04-22T22:46:28.068117+0000 mgr.y (mgr.34107) 10921 : cluster [DBG] pgmap v11551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:28 smithi121 bash[36902]: audit 2022-04-22T22:46:28.556002+0000 mon.a (mon.0) 15184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:28 smithi115 bash[45495]: cluster 2022-04-22T22:46:28.068117+0000 mgr.y (mgr.34107) 10921 : cluster [DBG] pgmap v11551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:28 smithi115 bash[45495]: audit 2022-04-22T22:46:28.556002+0000 mon.a (mon.0) 15184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:28 smithi115 bash[46934]: cluster 2022-04-22T22:46:28.068117+0000 mgr.y (mgr.34107) 10921 : cluster [DBG] pgmap v11551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:28 smithi115 bash[46934]: audit 2022-04-22T22:46:28.556002+0000 mon.a (mon.0) 15184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:31 smithi121 bash[36902]: cluster 2022-04-22T22:46:30.069286+0000 mgr.y (mgr.34107) 10922 : cluster [DBG] pgmap v11552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:31 smithi121 bash[36902]: audit 2022-04-22T22:46:30.489224+0000 mon.a (mon.0) 15185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:31 smithi115 bash[45495]: cluster 2022-04-22T22:46:30.069286+0000 mgr.y (mgr.34107) 10922 : cluster [DBG] pgmap v11552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:31 smithi115 bash[45495]: audit 2022-04-22T22:46:30.489224+0000 mon.a (mon.0) 15185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:31 smithi115 bash[46934]: cluster 2022-04-22T22:46:30.069286+0000 mgr.y (mgr.34107) 10922 : cluster [DBG] pgmap v11552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:31 smithi115 bash[46934]: audit 2022-04-22T22:46:30.489224+0000 mon.a (mon.0) 15185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:32 smithi121 bash[36902]: cluster 2022-04-22T22:46:32.070389+0000 mgr.y (mgr.34107) 10923 : cluster [DBG] pgmap v11553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:32 smithi121 bash[36902]: audit 2022-04-22T22:46:32.104512+0000 mon.a (mon.0) 15186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:32.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:32 smithi115 bash[45495]: cluster 2022-04-22T22:46:32.070389+0000 mgr.y (mgr.34107) 10923 : cluster [DBG] pgmap v11553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:32 smithi115 bash[45495]: audit 2022-04-22T22:46:32.104512+0000 mon.a (mon.0) 15186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:32 smithi115 bash[46934]: cluster 2022-04-22T22:46:32.070389+0000 mgr.y (mgr.34107) 10923 : cluster [DBG] pgmap v11553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:32 smithi115 bash[46934]: audit 2022-04-22T22:46:32.104512+0000 mon.a (mon.0) 15186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:33 smithi121 bash[36902]: audit 2022-04-22T22:46:33.559749+0000 mon.a (mon.0) 15187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:34.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:33 smithi115 bash[45495]: audit 2022-04-22T22:46:33.559749+0000 mon.a (mon.0) 15187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:34.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:33 smithi115 bash[46934]: audit 2022-04-22T22:46:33.559749+0000 mon.a (mon.0) 15187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:34.736Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:34 smithi115 bash[45495]: cluster 2022-04-22T22:46:34.071577+0000 mgr.y (mgr.34107) 10924 : cluster [DBG] pgmap v11554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:34 smithi115 bash[46934]: cluster 2022-04-22T22:46:34.071577+0000 mgr.y (mgr.34107) 10924 : cluster [DBG] pgmap v11554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:34 smithi121 bash[36902]: cluster 2022-04-22T22:46:34.071577+0000 mgr.y (mgr.34107) 10924 : cluster [DBG] pgmap v11554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:35 smithi121 bash[36902]: audit 2022-04-22T22:46:35.492487+0000 mon.a (mon.0) 15188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:35 smithi115 bash[45495]: audit 2022-04-22T22:46:35.492487+0000 mon.a (mon.0) 15188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:35 smithi115 bash[46934]: audit 2022-04-22T22:46:35.492487+0000 mon.a (mon.0) 15188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:36 smithi121 bash[36902]: cluster 2022-04-22T22:46:36.072645+0000 mgr.y (mgr.34107) 10925 : cluster [DBG] pgmap v11555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:36 smithi115 bash[45495]: cluster 2022-04-22T22:46:36.072645+0000 mgr.y (mgr.34107) 10925 : cluster [DBG] pgmap v11555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:36 smithi115 bash[46934]: cluster 2022-04-22T22:46:36.072645+0000 mgr.y (mgr.34107) 10925 : cluster [DBG] pgmap v11555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:37 smithi121 bash[36902]: audit 2022-04-22T22:46:37.107675+0000 mon.a (mon.0) 15189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:37 smithi115 bash[45495]: audit 2022-04-22T22:46:37.107675+0000 mon.a (mon.0) 15189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:37 smithi115 bash[46934]: audit 2022-04-22T22:46:37.107675+0000 mon.a (mon.0) 15189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:38 smithi121 bash[36902]: cluster 2022-04-22T22:46:38.073822+0000 mgr.y (mgr.34107) 10926 : cluster [DBG] pgmap v11556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:38 smithi121 bash[36902]: audit 2022-04-22T22:46:38.563550+0000 mon.a (mon.0) 15190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:38 smithi115 bash[45495]: cluster 2022-04-22T22:46:38.073822+0000 mgr.y (mgr.34107) 10926 : cluster [DBG] pgmap v11556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:38 smithi115 bash[45495]: audit 2022-04-22T22:46:38.563550+0000 mon.a (mon.0) 15190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:38 smithi115 bash[46934]: cluster 2022-04-22T22:46:38.073822+0000 mgr.y (mgr.34107) 10926 : cluster [DBG] pgmap v11556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:38 smithi115 bash[46934]: audit 2022-04-22T22:46:38.563550+0000 mon.a (mon.0) 15190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:41 smithi121 bash[36902]: cluster 2022-04-22T22:46:40.075003+0000 mgr.y (mgr.34107) 10927 : cluster [DBG] pgmap v11557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:41.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:41 smithi121 bash[36902]: audit 2022-04-22T22:46:40.495282+0000 mon.a (mon.0) 15191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:41 smithi115 bash[45495]: cluster 2022-04-22T22:46:40.075003+0000 mgr.y (mgr.34107) 10927 : cluster [DBG] pgmap v11557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:41 smithi115 bash[45495]: audit 2022-04-22T22:46:40.495282+0000 mon.a (mon.0) 15191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:41 smithi115 bash[46934]: cluster 2022-04-22T22:46:40.075003+0000 mgr.y (mgr.34107) 10927 : cluster [DBG] pgmap v11557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:41 smithi115 bash[46934]: audit 2022-04-22T22:46:40.495282+0000 mon.a (mon.0) 15191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:42 smithi121 bash[36902]: cluster 2022-04-22T22:46:42.076171+0000 mgr.y (mgr.34107) 10928 : cluster [DBG] pgmap v11558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:42 smithi121 bash[36902]: audit 2022-04-22T22:46:42.110630+0000 mon.a (mon.0) 15192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:42.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:42 smithi115 bash[45495]: cluster 2022-04-22T22:46:42.076171+0000 mgr.y (mgr.34107) 10928 : cluster [DBG] pgmap v11558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:42 smithi115 bash[45495]: audit 2022-04-22T22:46:42.110630+0000 mon.a (mon.0) 15192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:42 smithi115 bash[46934]: cluster 2022-04-22T22:46:42.076171+0000 mgr.y (mgr.34107) 10928 : cluster [DBG] pgmap v11558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:42 smithi115 bash[46934]: audit 2022-04-22T22:46:42.110630+0000 mon.a (mon.0) 15192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:43.884 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 10m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 57s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 10m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 57s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 57s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 10m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:46:44.235 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 10m ago 5h 438M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 57s ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 10m ago 5h 298M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 10m ago 5h 10.6M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 57s ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 10m ago 5h 718M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 10m ago 5h 798M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 10m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 10m ago 5h 502M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:46:44.236 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 57s ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:46:44.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 57s ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:46:44.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 57s ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:46:44.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 57s ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:46:44.237 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 57s ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:46:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:43 smithi121 bash[36902]: audit 2022-04-22T22:46:43.567378+0000 mon.a (mon.0) 15193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:43 smithi121 bash[36902]: audit 2022-04-22T22:46:43.865854+0000 mgr.y (mgr.34107) 10929 : audit [DBG] from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:43 smithi115 bash[45495]: audit 2022-04-22T22:46:43.567378+0000 mon.a (mon.0) 15193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:43 smithi115 bash[45495]: audit 2022-04-22T22:46:43.865854+0000 mgr.y (mgr.34107) 10929 : audit [DBG] from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:43 smithi115 bash[46934]: audit 2022-04-22T22:46:43.567378+0000 mon.a (mon.0) 15193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:43 smithi115 bash[46934]: audit 2022-04-22T22:46:43.865854+0000 mgr.y (mgr.34107) 10929 : audit [DBG] from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:44.641 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:46:44.641 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:46:44.641 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:46:44.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:44.642 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:46:44.642 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:46:44.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:44.642 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:46:44.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:46:44.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:44.643 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:46:44.643 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:46:44.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:46:44.644 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:46:44.644 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:46:44.644 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:46:44.644 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:46:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:44.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[45495]: cluster 2022-04-22T22:46:44.077372+0000 mgr.y (mgr.34107) 10930 : cluster [DBG] pgmap v11559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[45495]: audit 2022-04-22T22:46:44.212073+0000 mgr.y (mgr.34107) 10931 : audit [DBG] from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[45495]: audit 2022-04-22T22:46:44.637094+0000 mon.c (mon.1) 369 : audit [DBG] from='client.? 172.21.15.115:0/1078405190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[46934]: cluster 2022-04-22T22:46:44.077372+0000 mgr.y (mgr.34107) 10930 : cluster [DBG] pgmap v11559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[46934]: audit 2022-04-22T22:46:44.212073+0000 mgr.y (mgr.34107) 10931 : audit [DBG] from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:44 smithi115 bash[46934]: audit 2022-04-22T22:46:44.637094+0000 mon.c (mon.1) 369 : audit [DBG] from='client.? 172.21.15.115:0/1078405190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:44 smithi121 bash[36902]: cluster 2022-04-22T22:46:44.077372+0000 mgr.y (mgr.34107) 10930 : cluster [DBG] pgmap v11559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:44 smithi121 bash[36902]: audit 2022-04-22T22:46:44.212073+0000 mgr.y (mgr.34107) 10931 : audit [DBG] from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:46:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:44 smithi121 bash[36902]: audit 2022-04-22T22:46:44.637094+0000 mon.c (mon.1) 369 : audit [DBG] from='client.? 172.21.15.115:0/1078405190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:46:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:45 smithi121 bash[36902]: audit 2022-04-22T22:46:45.498456+0000 mon.a (mon.0) 15194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:45 smithi115 bash[45495]: audit 2022-04-22T22:46:45.498456+0000 mon.a (mon.0) 15194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:45 smithi115 bash[46934]: audit 2022-04-22T22:46:45.498456+0000 mon.a (mon.0) 15194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:47.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[46934]: cluster 2022-04-22T22:46:46.078442+0000 mgr.y (mgr.34107) 10932 : cluster [DBG] pgmap v11560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:47.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[46934]: audit 2022-04-22T22:46:46.799892+0000 mon.a (mon.0) 15195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:46:47.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[46934]: audit 2022-04-22T22:46:46.802971+0000 mon.a (mon.0) 15196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:46:47.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[46934]: audit 2022-04-22T22:46:46.804798+0000 mon.a (mon.0) 15197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:46:47.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[45495]: cluster 2022-04-22T22:46:46.078442+0000 mgr.y (mgr.34107) 10932 : cluster [DBG] pgmap v11560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:47.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[45495]: audit 2022-04-22T22:46:46.799892+0000 mon.a (mon.0) 15195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:46:47.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[45495]: audit 2022-04-22T22:46:46.802971+0000 mon.a (mon.0) 15196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:46:47.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:46 smithi115 bash[45495]: audit 2022-04-22T22:46:46.804798+0000 mon.a (mon.0) 15197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:46:47.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:46 smithi121 bash[36902]: cluster 2022-04-22T22:46:46.078442+0000 mgr.y (mgr.34107) 10932 : cluster [DBG] pgmap v11560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:47.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:46 smithi121 bash[36902]: audit 2022-04-22T22:46:46.799892+0000 mon.a (mon.0) 15195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:46:47.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:46 smithi121 bash[36902]: audit 2022-04-22T22:46:46.802971+0000 mon.a (mon.0) 15196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:46:47.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:46 smithi121 bash[36902]: audit 2022-04-22T22:46:46.804798+0000 mon.a (mon.0) 15197 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:46:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:47 smithi121 bash[36902]: audit 2022-04-22T22:46:47.113397+0000 mon.a (mon.0) 15198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:47 smithi121 bash[36902]: audit 2022-04-22T22:46:47.152048+0000 mon.a (mon.0) 15199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:47 smithi121 bash[36902]: audit 2022-04-22T22:46:47.161646+0000 mon.a (mon.0) 15200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[45495]: audit 2022-04-22T22:46:47.113397+0000 mon.a (mon.0) 15198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[45495]: audit 2022-04-22T22:46:47.152048+0000 mon.a (mon.0) 15199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[45495]: audit 2022-04-22T22:46:47.161646+0000 mon.a (mon.0) 15200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[46934]: audit 2022-04-22T22:46:47.113397+0000 mon.a (mon.0) 15198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[46934]: audit 2022-04-22T22:46:47.152048+0000 mon.a (mon.0) 15199 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:47 smithi115 bash[46934]: audit 2022-04-22T22:46:47.161646+0000 mon.a (mon.0) 15200 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:48 smithi121 bash[36902]: cluster 2022-04-22T22:46:48.079546+0000 mgr.y (mgr.34107) 10933 : cluster [DBG] pgmap v11561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:48 smithi121 bash[36902]: audit 2022-04-22T22:46:48.571051+0000 mon.a (mon.0) 15201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:48 smithi115 bash[45495]: cluster 2022-04-22T22:46:48.079546+0000 mgr.y (mgr.34107) 10933 : cluster [DBG] pgmap v11561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:48 smithi115 bash[45495]: audit 2022-04-22T22:46:48.571051+0000 mon.a (mon.0) 15201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:48 smithi115 bash[46934]: cluster 2022-04-22T22:46:48.079546+0000 mgr.y (mgr.34107) 10933 : cluster [DBG] pgmap v11561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:48 smithi115 bash[46934]: audit 2022-04-22T22:46:48.571051+0000 mon.a (mon.0) 15201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:51 smithi121 bash[36902]: cluster 2022-04-22T22:46:50.080395+0000 mgr.y (mgr.34107) 10934 : cluster [DBG] pgmap v11562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:51 smithi121 bash[36902]: audit 2022-04-22T22:46:50.500955+0000 mon.a (mon.0) 15202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:51 smithi115 bash[46934]: cluster 2022-04-22T22:46:50.080395+0000 mgr.y (mgr.34107) 10934 : cluster [DBG] pgmap v11562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:51 smithi115 bash[46934]: audit 2022-04-22T22:46:50.500955+0000 mon.a (mon.0) 15202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:51 smithi115 bash[45495]: cluster 2022-04-22T22:46:50.080395+0000 mgr.y (mgr.34107) 10934 : cluster [DBG] pgmap v11562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:51.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:51 smithi115 bash[45495]: audit 2022-04-22T22:46:50.500955+0000 mon.a (mon.0) 15202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:52 smithi121 bash[36902]: audit 2022-04-22T22:46:51.287583+0000 mon.a (mon.0) 15203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:52.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:52 smithi121 bash[36902]: audit 2022-04-22T22:46:52.115862+0000 mon.a (mon.0) 15204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:52.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:52 smithi115 bash[46934]: audit 2022-04-22T22:46:51.287583+0000 mon.a (mon.0) 15203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:52.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:52 smithi115 bash[46934]: audit 2022-04-22T22:46:52.115862+0000 mon.a (mon.0) 15204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:52 smithi115 bash[45495]: audit 2022-04-22T22:46:51.287583+0000 mon.a (mon.0) 15203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:46:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:52 smithi115 bash[45495]: audit 2022-04-22T22:46:52.115862+0000 mon.a (mon.0) 15204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:52.877Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:53 smithi121 bash[36902]: cluster 2022-04-22T22:46:52.081451+0000 mgr.y (mgr.34107) 10935 : cluster [DBG] pgmap v11563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:53.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:53 smithi115 bash[45495]: cluster 2022-04-22T22:46:52.081451+0000 mgr.y (mgr.34107) 10935 : cluster [DBG] pgmap v11563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:53.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:53 smithi115 bash[46934]: cluster 2022-04-22T22:46:52.081451+0000 mgr.y (mgr.34107) 10935 : cluster [DBG] pgmap v11563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:54.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:54 smithi115 bash[46934]: audit 2022-04-22T22:46:53.574822+0000 mon.a (mon.0) 15205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:54.684 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:54 smithi115 bash[45495]: audit 2022-04-22T22:46:53.574822+0000 mon.a (mon.0) 15205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:54 smithi121 bash[36902]: audit 2022-04-22T22:46:53.574822+0000 mon.a (mon.0) 15205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:54.790 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:46:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:46:54.737Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:46:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:55 smithi121 bash[36902]: cluster 2022-04-22T22:46:54.082626+0000 mgr.y (mgr.34107) 10936 : cluster [DBG] pgmap v11564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:55 smithi115 bash[45495]: cluster 2022-04-22T22:46:54.082626+0000 mgr.y (mgr.34107) 10936 : cluster [DBG] pgmap v11564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:55 smithi115 bash[46934]: cluster 2022-04-22T22:46:54.082626+0000 mgr.y (mgr.34107) 10936 : cluster [DBG] pgmap v11564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:56.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:56 smithi115 bash[45495]: audit 2022-04-22T22:46:55.503654+0000 mon.a (mon.0) 15206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:56 smithi121 bash[36902]: audit 2022-04-22T22:46:55.503654+0000 mon.a (mon.0) 15206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:56.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:56 smithi115 bash[46934]: audit 2022-04-22T22:46:55.503654+0000 mon.a (mon.0) 15206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:46:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:57 smithi121 bash[36902]: cluster 2022-04-22T22:46:56.083778+0000 mgr.y (mgr.34107) 10937 : cluster [DBG] pgmap v11565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:57 smithi121 bash[36902]: audit 2022-04-22T22:46:57.118881+0000 mon.a (mon.0) 15207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:57.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:57 smithi115 bash[46934]: cluster 2022-04-22T22:46:56.083778+0000 mgr.y (mgr.34107) 10937 : cluster [DBG] pgmap v11565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:57.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:57 smithi115 bash[46934]: audit 2022-04-22T22:46:57.118881+0000 mon.a (mon.0) 15207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:57 smithi115 bash[45495]: cluster 2022-04-22T22:46:56.083778+0000 mgr.y (mgr.34107) 10937 : cluster [DBG] pgmap v11565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:57 smithi115 bash[45495]: audit 2022-04-22T22:46:57.118881+0000 mon.a (mon.0) 15207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:46:59.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:59 smithi115 bash[45495]: cluster 2022-04-22T22:46:58.084950+0000 mgr.y (mgr.34107) 10938 : cluster [DBG] pgmap v11566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:59.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:46:59 smithi115 bash[45495]: audit 2022-04-22T22:46:58.578690+0000 mon.a (mon.0) 15208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:59.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:59 smithi115 bash[46934]: cluster 2022-04-22T22:46:58.084950+0000 mgr.y (mgr.34107) 10938 : cluster [DBG] pgmap v11566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:59.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:46:59 smithi115 bash[46934]: audit 2022-04-22T22:46:58.578690+0000 mon.a (mon.0) 15208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:46:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:59 smithi121 bash[36902]: cluster 2022-04-22T22:46:58.084950+0000 mgr.y (mgr.34107) 10938 : cluster [DBG] pgmap v11566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:46:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:46:59 smithi121 bash[36902]: audit 2022-04-22T22:46:58.578690+0000 mon.a (mon.0) 15208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:01 smithi121 bash[36902]: cluster 2022-04-22T22:47:00.085758+0000 mgr.y (mgr.34107) 10939 : cluster [DBG] pgmap v11567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:01 smithi121 bash[36902]: cephadm 2022-04-22T22:47:00.459390+0000 mgr.y (mgr.34107) 10940 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:47:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:01 smithi121 bash[36902]: audit 2022-04-22T22:47:00.470401+0000 mon.a (mon.0) 15209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:01 smithi121 bash[36902]: audit 2022-04-22T22:47:00.478660+0000 mon.a (mon.0) 15210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:01 smithi121 bash[36902]: audit 2022-04-22T22:47:00.506623+0000 mon.a (mon.0) 15211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[45495]: cluster 2022-04-22T22:47:00.085758+0000 mgr.y (mgr.34107) 10939 : cluster [DBG] pgmap v11567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[45495]: cephadm 2022-04-22T22:47:00.459390+0000 mgr.y (mgr.34107) 10940 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:47:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[45495]: audit 2022-04-22T22:47:00.470401+0000 mon.a (mon.0) 15209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[45495]: audit 2022-04-22T22:47:00.478660+0000 mon.a (mon.0) 15210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[45495]: audit 2022-04-22T22:47:00.506623+0000 mon.a (mon.0) 15211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[46934]: cluster 2022-04-22T22:47:00.085758+0000 mgr.y (mgr.34107) 10939 : cluster [DBG] pgmap v11567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[46934]: cephadm 2022-04-22T22:47:00.459390+0000 mgr.y (mgr.34107) 10940 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T22:47:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[46934]: audit 2022-04-22T22:47:00.470401+0000 mon.a (mon.0) 15209 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[46934]: audit 2022-04-22T22:47:00.478660+0000 mon.a (mon.0) 15210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:47:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:01 smithi115 bash[46934]: audit 2022-04-22T22:47:00.506623+0000 mon.a (mon.0) 15211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:02 smithi121 bash[36902]: cluster 2022-04-22T22:47:02.087010+0000 mgr.y (mgr.34107) 10941 : cluster [DBG] pgmap v11568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:02 smithi121 bash[36902]: audit 2022-04-22T22:47:02.121260+0000 mon.a (mon.0) 15212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:03.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:02 smithi115 bash[46934]: cluster 2022-04-22T22:47:02.087010+0000 mgr.y (mgr.34107) 10941 : cluster [DBG] pgmap v11568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:02 smithi115 bash[46934]: audit 2022-04-22T22:47:02.121260+0000 mon.a (mon.0) 15212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:02.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:02 smithi115 bash[45495]: cluster 2022-04-22T22:47:02.087010+0000 mgr.y (mgr.34107) 10941 : cluster [DBG] pgmap v11568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:02 smithi115 bash[45495]: audit 2022-04-22T22:47:02.121260+0000 mon.a (mon.0) 15212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:03 smithi121 bash[36902]: audit 2022-04-22T22:47:03.582391+0000 mon.a (mon.0) 15213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:04.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:03 smithi115 bash[45495]: audit 2022-04-22T22:47:03.582391+0000 mon.a (mon.0) 15213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:03 smithi115 bash[46934]: audit 2022-04-22T22:47:03.582391+0000 mon.a (mon.0) 15213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:04 smithi115 bash[45495]: cluster 2022-04-22T22:47:04.088142+0000 mgr.y (mgr.34107) 10942 : cluster [DBG] pgmap v11569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:05.060 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:04.739Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:05.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:04 smithi115 bash[46934]: cluster 2022-04-22T22:47:04.088142+0000 mgr.y (mgr.34107) 10942 : cluster [DBG] pgmap v11569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:04 smithi121 bash[36902]: cluster 2022-04-22T22:47:04.088142+0000 mgr.y (mgr.34107) 10942 : cluster [DBG] pgmap v11569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:05 smithi121 bash[36902]: audit 2022-04-22T22:47:05.509551+0000 mon.a (mon.0) 15214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:05 smithi115 bash[45495]: audit 2022-04-22T22:47:05.509551+0000 mon.a (mon.0) 15214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:05 smithi115 bash[46934]: audit 2022-04-22T22:47:05.509551+0000 mon.a (mon.0) 15214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:06 smithi121 bash[36902]: cluster 2022-04-22T22:47:06.089206+0000 mgr.y (mgr.34107) 10943 : cluster [DBG] pgmap v11570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:06 smithi115 bash[45495]: cluster 2022-04-22T22:47:06.089206+0000 mgr.y (mgr.34107) 10943 : cluster [DBG] pgmap v11570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:06 smithi115 bash[46934]: cluster 2022-04-22T22:47:06.089206+0000 mgr.y (mgr.34107) 10943 : cluster [DBG] pgmap v11570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:07 smithi121 bash[36902]: audit 2022-04-22T22:47:07.124410+0000 mon.a (mon.0) 15215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:07 smithi115 bash[45495]: audit 2022-04-22T22:47:07.124410+0000 mon.a (mon.0) 15215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:07 smithi115 bash[46934]: audit 2022-04-22T22:47:07.124410+0000 mon.a (mon.0) 15215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:08 smithi121 bash[36902]: cluster 2022-04-22T22:47:08.090271+0000 mgr.y (mgr.34107) 10944 : cluster [DBG] pgmap v11571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:08 smithi121 bash[36902]: audit 2022-04-22T22:47:08.586133+0000 mon.a (mon.0) 15216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:09.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:08 smithi115 bash[46934]: cluster 2022-04-22T22:47:08.090271+0000 mgr.y (mgr.34107) 10944 : cluster [DBG] pgmap v11571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:08 smithi115 bash[46934]: audit 2022-04-22T22:47:08.586133+0000 mon.a (mon.0) 15216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:08 smithi115 bash[45495]: cluster 2022-04-22T22:47:08.090271+0000 mgr.y (mgr.34107) 10944 : cluster [DBG] pgmap v11571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:08 smithi115 bash[45495]: audit 2022-04-22T22:47:08.586133+0000 mon.a (mon.0) 15216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:11 smithi121 bash[36902]: cluster 2022-04-22T22:47:10.091306+0000 mgr.y (mgr.34107) 10945 : cluster [DBG] pgmap v11572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:11 smithi121 bash[36902]: audit 2022-04-22T22:47:10.512541+0000 mon.a (mon.0) 15217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:11 smithi115 bash[45495]: cluster 2022-04-22T22:47:10.091306+0000 mgr.y (mgr.34107) 10945 : cluster [DBG] pgmap v11572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:11 smithi115 bash[45495]: audit 2022-04-22T22:47:10.512541+0000 mon.a (mon.0) 15217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:11 smithi115 bash[46934]: cluster 2022-04-22T22:47:10.091306+0000 mgr.y (mgr.34107) 10945 : cluster [DBG] pgmap v11572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:11.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:11 smithi115 bash[46934]: audit 2022-04-22T22:47:10.512541+0000 mon.a (mon.0) 15217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:12 smithi121 bash[36902]: cluster 2022-04-22T22:47:12.092452+0000 mgr.y (mgr.34107) 10946 : cluster [DBG] pgmap v11573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:12 smithi121 bash[36902]: audit 2022-04-22T22:47:12.127047+0000 mon.a (mon.0) 15218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:13.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:12 smithi115 bash[46934]: cluster 2022-04-22T22:47:12.092452+0000 mgr.y (mgr.34107) 10946 : cluster [DBG] pgmap v11573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:13.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:12 smithi115 bash[46934]: audit 2022-04-22T22:47:12.127047+0000 mon.a (mon.0) 15218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:12.880Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:12 smithi115 bash[45495]: cluster 2022-04-22T22:47:12.092452+0000 mgr.y (mgr.34107) 10946 : cluster [DBG] pgmap v11573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:12 smithi115 bash[45495]: audit 2022-04-22T22:47:12.127047+0000 mon.a (mon.0) 15218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:13 smithi121 bash[36902]: audit 2022-04-22T22:47:13.589683+0000 mon.a (mon.0) 15219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:13 smithi115 bash[45495]: audit 2022-04-22T22:47:13.589683+0000 mon.a (mon.0) 15219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:13 smithi115 bash[46934]: audit 2022-04-22T22:47:13.589683+0000 mon.a (mon.0) 15219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:15.013 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:47:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:14.738Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:14 smithi115 bash[46934]: cluster 2022-04-22T22:47:14.093604+0000 mgr.y (mgr.34107) 10947 : cluster [DBG] pgmap v11574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:14 smithi115 bash[45495]: cluster 2022-04-22T22:47:14.093604+0000 mgr.y (mgr.34107) 10947 : cluster [DBG] pgmap v11574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:14 smithi121 bash[36902]: cluster 2022-04-22T22:47:14.093604+0000 mgr.y (mgr.34107) 10947 : cluster [DBG] pgmap v11574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:15.356 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:47:15.356 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 24s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:47:15.356 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 88s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:47:15.356 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 24s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 88s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 88s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 24s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 24s ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 88s ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 24s ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 24s ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 88s ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 24s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:47:15.357 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 24s ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 24s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 24s ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 88s ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 88s ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 88s ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 88s ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:47:15.358 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 88s ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:47:15.767 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:47:15.767 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:47:15.767 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:47:15.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:47:15.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:15.769 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:47:15.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:47:15.769 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:47:15.769 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:47:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:15 smithi121 bash[36902]: audit 2022-04-22T22:47:14.994023+0000 mgr.y (mgr.34107) 10948 : audit [DBG] from='client.46686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:15 smithi121 bash[36902]: audit 2022-04-22T22:47:15.333941+0000 mgr.y (mgr.34107) 10949 : audit [DBG] from='client.56629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:15 smithi121 bash[36902]: audit 2022-04-22T22:47:15.515395+0000 mon.a (mon.0) 15220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:15 smithi121 bash[36902]: audit 2022-04-22T22:47:15.763863+0000 mon.c (mon.1) 370 : audit [DBG] from='client.? 172.21.15.115:0/1749541094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[45495]: audit 2022-04-22T22:47:14.994023+0000 mgr.y (mgr.34107) 10948 : audit [DBG] from='client.46686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[45495]: audit 2022-04-22T22:47:15.333941+0000 mgr.y (mgr.34107) 10949 : audit [DBG] from='client.56629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[45495]: audit 2022-04-22T22:47:15.515395+0000 mon.a (mon.0) 15220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[45495]: audit 2022-04-22T22:47:15.763863+0000 mon.c (mon.1) 370 : audit [DBG] from='client.? 172.21.15.115:0/1749541094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[46934]: audit 2022-04-22T22:47:14.994023+0000 mgr.y (mgr.34107) 10948 : audit [DBG] from='client.46686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[46934]: audit 2022-04-22T22:47:15.333941+0000 mgr.y (mgr.34107) 10949 : audit [DBG] from='client.56629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[46934]: audit 2022-04-22T22:47:15.515395+0000 mon.a (mon.0) 15220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:15 smithi115 bash[46934]: audit 2022-04-22T22:47:15.763863+0000 mon.c (mon.1) 370 : audit [DBG] from='client.? 172.21.15.115:0/1749541094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:16 smithi121 bash[36902]: cluster 2022-04-22T22:47:16.094736+0000 mgr.y (mgr.34107) 10950 : cluster [DBG] pgmap v11575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:16 smithi115 bash[45495]: cluster 2022-04-22T22:47:16.094736+0000 mgr.y (mgr.34107) 10950 : cluster [DBG] pgmap v11575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:16 smithi115 bash[46934]: cluster 2022-04-22T22:47:16.094736+0000 mgr.y (mgr.34107) 10950 : cluster [DBG] pgmap v11575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:17 smithi121 bash[36902]: audit 2022-04-22T22:47:16.966145+0000 mon.a (mon.0) 15221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:47:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:17 smithi121 bash[36902]: audit 2022-04-22T22:47:16.966583+0000 mon.a (mon.0) 15222 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:47:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:17 smithi121 bash[36902]: audit 2022-04-22T22:47:17.129940+0000 mon.a (mon.0) 15223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[45495]: audit 2022-04-22T22:47:16.966145+0000 mon.a (mon.0) 15221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:47:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[45495]: audit 2022-04-22T22:47:16.966583+0000 mon.a (mon.0) 15222 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:47:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[45495]: audit 2022-04-22T22:47:17.129940+0000 mon.a (mon.0) 15223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[46934]: audit 2022-04-22T22:47:16.966145+0000 mon.a (mon.0) 15221 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:47:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[46934]: audit 2022-04-22T22:47:16.966583+0000 mon.a (mon.0) 15222 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:47:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:17 smithi115 bash[46934]: audit 2022-04-22T22:47:17.129940+0000 mon.a (mon.0) 15223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:18 smithi121 bash[36902]: cluster 2022-04-22T22:47:18.095826+0000 mgr.y (mgr.34107) 10951 : cluster [DBG] pgmap v11576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:18 smithi121 bash[36902]: audit 2022-04-22T22:47:18.593801+0000 mon.a (mon.0) 15224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:18 smithi115 bash[45495]: cluster 2022-04-22T22:47:18.095826+0000 mgr.y (mgr.34107) 10951 : cluster [DBG] pgmap v11576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:18 smithi115 bash[45495]: audit 2022-04-22T22:47:18.593801+0000 mon.a (mon.0) 15224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:18 smithi115 bash[46934]: cluster 2022-04-22T22:47:18.095826+0000 mgr.y (mgr.34107) 10951 : cluster [DBG] pgmap v11576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:18 smithi115 bash[46934]: audit 2022-04-22T22:47:18.593801+0000 mon.a (mon.0) 15224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:21 smithi121 bash[36902]: cluster 2022-04-22T22:47:20.096883+0000 mgr.y (mgr.34107) 10952 : cluster [DBG] pgmap v11577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:21 smithi121 bash[36902]: audit 2022-04-22T22:47:20.517807+0000 mon.a (mon.0) 15225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:21 smithi115 bash[45495]: cluster 2022-04-22T22:47:20.096883+0000 mgr.y (mgr.34107) 10952 : cluster [DBG] pgmap v11577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:21 smithi115 bash[45495]: audit 2022-04-22T22:47:20.517807+0000 mon.a (mon.0) 15225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:21 smithi115 bash[46934]: cluster 2022-04-22T22:47:20.096883+0000 mgr.y (mgr.34107) 10952 : cluster [DBG] pgmap v11577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:21 smithi115 bash[46934]: audit 2022-04-22T22:47:20.517807+0000 mon.a (mon.0) 15225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:22 smithi121 bash[36902]: cluster 2022-04-22T22:47:22.098045+0000 mgr.y (mgr.34107) 10953 : cluster [DBG] pgmap v11578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:22 smithi121 bash[36902]: audit 2022-04-22T22:47:22.133127+0000 mon.a (mon.0) 15226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:22.880Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:22 smithi115 bash[45495]: cluster 2022-04-22T22:47:22.098045+0000 mgr.y (mgr.34107) 10953 : cluster [DBG] pgmap v11578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:22 smithi115 bash[45495]: audit 2022-04-22T22:47:22.133127+0000 mon.a (mon.0) 15226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:22 smithi115 bash[46934]: cluster 2022-04-22T22:47:22.098045+0000 mgr.y (mgr.34107) 10953 : cluster [DBG] pgmap v11578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:22 smithi115 bash[46934]: audit 2022-04-22T22:47:22.133127+0000 mon.a (mon.0) 15226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:23 smithi121 bash[36902]: audit 2022-04-22T22:47:23.597167+0000 mon.a (mon.0) 15227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:23 smithi115 bash[45495]: audit 2022-04-22T22:47:23.597167+0000 mon.a (mon.0) 15227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:23 smithi115 bash[46934]: audit 2022-04-22T22:47:23.597167+0000 mon.a (mon.0) 15227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:24 smithi115 bash[45495]: cluster 2022-04-22T22:47:24.099151+0000 mgr.y (mgr.34107) 10954 : cluster [DBG] pgmap v11579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:24.741Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:24 smithi115 bash[46934]: cluster 2022-04-22T22:47:24.099151+0000 mgr.y (mgr.34107) 10954 : cluster [DBG] pgmap v11579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:24 smithi121 bash[36902]: cluster 2022-04-22T22:47:24.099151+0000 mgr.y (mgr.34107) 10954 : cluster [DBG] pgmap v11579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:25 smithi121 bash[36902]: audit 2022-04-22T22:47:25.520928+0000 mon.a (mon.0) 15228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:25 smithi115 bash[45495]: audit 2022-04-22T22:47:25.520928+0000 mon.a (mon.0) 15228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:26.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:25 smithi115 bash[46934]: audit 2022-04-22T22:47:25.520928+0000 mon.a (mon.0) 15228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:26 smithi121 bash[36902]: cluster 2022-04-22T22:47:26.100306+0000 mgr.y (mgr.34107) 10955 : cluster [DBG] pgmap v11580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:26 smithi115 bash[45495]: cluster 2022-04-22T22:47:26.100306+0000 mgr.y (mgr.34107) 10955 : cluster [DBG] pgmap v11580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:26 smithi115 bash[46934]: cluster 2022-04-22T22:47:26.100306+0000 mgr.y (mgr.34107) 10955 : cluster [DBG] pgmap v11580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:27 smithi121 bash[36902]: audit 2022-04-22T22:47:27.136363+0000 mon.a (mon.0) 15229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:27 smithi115 bash[45495]: audit 2022-04-22T22:47:27.136363+0000 mon.a (mon.0) 15229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:27 smithi115 bash[46934]: audit 2022-04-22T22:47:27.136363+0000 mon.a (mon.0) 15229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:28 smithi121 bash[36902]: cluster 2022-04-22T22:47:28.101414+0000 mgr.y (mgr.34107) 10956 : cluster [DBG] pgmap v11581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:28 smithi121 bash[36902]: audit 2022-04-22T22:47:28.600745+0000 mon.a (mon.0) 15230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:28 smithi115 bash[45495]: cluster 2022-04-22T22:47:28.101414+0000 mgr.y (mgr.34107) 10956 : cluster [DBG] pgmap v11581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:28 smithi115 bash[45495]: audit 2022-04-22T22:47:28.600745+0000 mon.a (mon.0) 15230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:28 smithi115 bash[46934]: cluster 2022-04-22T22:47:28.101414+0000 mgr.y (mgr.34107) 10956 : cluster [DBG] pgmap v11581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:28 smithi115 bash[46934]: audit 2022-04-22T22:47:28.600745+0000 mon.a (mon.0) 15230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:31 smithi121 bash[36902]: cluster 2022-04-22T22:47:30.102462+0000 mgr.y (mgr.34107) 10957 : cluster [DBG] pgmap v11582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:31 smithi121 bash[36902]: audit 2022-04-22T22:47:30.523817+0000 mon.a (mon.0) 15231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:31 smithi115 bash[45495]: cluster 2022-04-22T22:47:30.102462+0000 mgr.y (mgr.34107) 10957 : cluster [DBG] pgmap v11582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:31 smithi115 bash[45495]: audit 2022-04-22T22:47:30.523817+0000 mon.a (mon.0) 15231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:31 smithi115 bash[46934]: cluster 2022-04-22T22:47:30.102462+0000 mgr.y (mgr.34107) 10957 : cluster [DBG] pgmap v11582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:31 smithi115 bash[46934]: audit 2022-04-22T22:47:30.523817+0000 mon.a (mon.0) 15231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:32 smithi121 bash[36902]: cluster 2022-04-22T22:47:32.103701+0000 mgr.y (mgr.34107) 10958 : cluster [DBG] pgmap v11583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:32 smithi121 bash[36902]: audit 2022-04-22T22:47:32.139400+0000 mon.a (mon.0) 15232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:32.880Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:32 smithi115 bash[45495]: cluster 2022-04-22T22:47:32.103701+0000 mgr.y (mgr.34107) 10958 : cluster [DBG] pgmap v11583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:32 smithi115 bash[45495]: audit 2022-04-22T22:47:32.139400+0000 mon.a (mon.0) 15232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:32 smithi115 bash[46934]: cluster 2022-04-22T22:47:32.103701+0000 mgr.y (mgr.34107) 10958 : cluster [DBG] pgmap v11583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:32 smithi115 bash[46934]: audit 2022-04-22T22:47:32.139400+0000 mon.a (mon.0) 15232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:33 smithi121 bash[36902]: audit 2022-04-22T22:47:33.604399+0000 mon.a (mon.0) 15233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:33 smithi115 bash[45495]: audit 2022-04-22T22:47:33.604399+0000 mon.a (mon.0) 15233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:33 smithi115 bash[46934]: audit 2022-04-22T22:47:33.604399+0000 mon.a (mon.0) 15233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:34 smithi115 bash[45495]: cluster 2022-04-22T22:47:34.104819+0000 mgr.y (mgr.34107) 10959 : cluster [DBG] pgmap v11584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:35.044 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:34.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:35.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:34 smithi115 bash[46934]: cluster 2022-04-22T22:47:34.104819+0000 mgr.y (mgr.34107) 10959 : cluster [DBG] pgmap v11584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:34 smithi121 bash[36902]: cluster 2022-04-22T22:47:34.104819+0000 mgr.y (mgr.34107) 10959 : cluster [DBG] pgmap v11584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:35 smithi121 bash[36902]: audit 2022-04-22T22:47:35.526780+0000 mon.a (mon.0) 15234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:35 smithi115 bash[45495]: audit 2022-04-22T22:47:35.526780+0000 mon.a (mon.0) 15234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:35 smithi115 bash[46934]: audit 2022-04-22T22:47:35.526780+0000 mon.a (mon.0) 15234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:36 smithi121 bash[36902]: cluster 2022-04-22T22:47:36.106030+0000 mgr.y (mgr.34107) 10960 : cluster [DBG] pgmap v11585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:36 smithi115 bash[45495]: cluster 2022-04-22T22:47:36.106030+0000 mgr.y (mgr.34107) 10960 : cluster [DBG] pgmap v11585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:36 smithi115 bash[46934]: cluster 2022-04-22T22:47:36.106030+0000 mgr.y (mgr.34107) 10960 : cluster [DBG] pgmap v11585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:37 smithi121 bash[36902]: audit 2022-04-22T22:47:37.142467+0000 mon.a (mon.0) 15235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:37 smithi115 bash[45495]: audit 2022-04-22T22:47:37.142467+0000 mon.a (mon.0) 15235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:37 smithi115 bash[46934]: audit 2022-04-22T22:47:37.142467+0000 mon.a (mon.0) 15235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:38 smithi121 bash[36902]: cluster 2022-04-22T22:47:38.107126+0000 mgr.y (mgr.34107) 10961 : cluster [DBG] pgmap v11586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:38 smithi121 bash[36902]: audit 2022-04-22T22:47:38.608101+0000 mon.a (mon.0) 15236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:38 smithi115 bash[46934]: cluster 2022-04-22T22:47:38.107126+0000 mgr.y (mgr.34107) 10961 : cluster [DBG] pgmap v11586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:38 smithi115 bash[46934]: audit 2022-04-22T22:47:38.608101+0000 mon.a (mon.0) 15236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:38 smithi115 bash[45495]: cluster 2022-04-22T22:47:38.107126+0000 mgr.y (mgr.34107) 10961 : cluster [DBG] pgmap v11586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:38 smithi115 bash[45495]: audit 2022-04-22T22:47:38.608101+0000 mon.a (mon.0) 15236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:41 smithi121 bash[36902]: cluster 2022-04-22T22:47:40.108191+0000 mgr.y (mgr.34107) 10962 : cluster [DBG] pgmap v11587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:41 smithi121 bash[36902]: audit 2022-04-22T22:47:40.529700+0000 mon.a (mon.0) 15237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:41 smithi115 bash[45495]: cluster 2022-04-22T22:47:40.108191+0000 mgr.y (mgr.34107) 10962 : cluster [DBG] pgmap v11587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:41 smithi115 bash[45495]: audit 2022-04-22T22:47:40.529700+0000 mon.a (mon.0) 15237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:41 smithi115 bash[46934]: cluster 2022-04-22T22:47:40.108191+0000 mgr.y (mgr.34107) 10962 : cluster [DBG] pgmap v11587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:41 smithi115 bash[46934]: audit 2022-04-22T22:47:40.529700+0000 mon.a (mon.0) 15237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:42 smithi121 bash[36902]: cluster 2022-04-22T22:47:42.109289+0000 mgr.y (mgr.34107) 10963 : cluster [DBG] pgmap v11588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:42 smithi121 bash[36902]: audit 2022-04-22T22:47:42.145549+0000 mon.a (mon.0) 15238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:43.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:42 smithi115 bash[46934]: cluster 2022-04-22T22:47:42.109289+0000 mgr.y (mgr.34107) 10963 : cluster [DBG] pgmap v11588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:42 smithi115 bash[46934]: audit 2022-04-22T22:47:42.145549+0000 mon.a (mon.0) 15238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:42.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:42 smithi115 bash[45495]: cluster 2022-04-22T22:47:42.109289+0000 mgr.y (mgr.34107) 10963 : cluster [DBG] pgmap v11588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:42 smithi115 bash[45495]: audit 2022-04-22T22:47:42.145549+0000 mon.a (mon.0) 15238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:43 smithi121 bash[36902]: audit 2022-04-22T22:47:43.611800+0000 mon.a (mon.0) 15239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:43 smithi115 bash[45495]: audit 2022-04-22T22:47:43.611800+0000 mon.a (mon.0) 15239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:43 smithi115 bash[46934]: audit 2022-04-22T22:47:43.611800+0000 mon.a (mon.0) 15239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:44.742Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:44 smithi115 bash[45495]: cluster 2022-04-22T22:47:44.110430+0000 mgr.y (mgr.34107) 10964 : cluster [DBG] pgmap v11589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:44 smithi115 bash[46934]: cluster 2022-04-22T22:47:44.110430+0000 mgr.y (mgr.34107) 10964 : cluster [DBG] pgmap v11589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:44 smithi121 bash[36902]: cluster 2022-04-22T22:47:44.110430+0000 mgr.y (mgr.34107) 10964 : cluster [DBG] pgmap v11589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:46.151 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:47:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:45 smithi121 bash[36902]: audit 2022-04-22T22:47:45.532741+0000 mon.a (mon.0) 15240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:45 smithi115 bash[46934]: audit 2022-04-22T22:47:45.532741+0000 mon.a (mon.0) 15240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:46.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:45 smithi115 bash[45495]: audit 2022-04-22T22:47:45.532741+0000 mon.a (mon.0) 15240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:47:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 55s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:47:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:47:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 55s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:47:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 55s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 55s ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 55s ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 55s ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 55s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:47:46.505 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 55s ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 55s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 55s ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:47:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:47:46.919 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:47:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:47:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:47:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:47:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:47:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:47:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:47:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:47:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:47:46.922 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:47:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:46 smithi121 bash[36902]: cluster 2022-04-22T22:47:46.111639+0000 mgr.y (mgr.34107) 10965 : cluster [DBG] pgmap v11590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:46 smithi121 bash[36902]: audit 2022-04-22T22:47:46.130793+0000 mgr.y (mgr.34107) 10966 : audit [DBG] from='client.46704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:46 smithi121 bash[36902]: audit 2022-04-22T22:47:46.482184+0000 mgr.y (mgr.34107) 10967 : audit [DBG] from='client.56647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:46 smithi121 bash[36902]: audit 2022-04-22T22:47:46.915918+0000 mon.a (mon.0) 15241 : audit [DBG] from='client.? 172.21.15.115:0/3368691980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[45495]: cluster 2022-04-22T22:47:46.111639+0000 mgr.y (mgr.34107) 10965 : cluster [DBG] pgmap v11590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[45495]: audit 2022-04-22T22:47:46.130793+0000 mgr.y (mgr.34107) 10966 : audit [DBG] from='client.46704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[45495]: audit 2022-04-22T22:47:46.482184+0000 mgr.y (mgr.34107) 10967 : audit [DBG] from='client.56647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[45495]: audit 2022-04-22T22:47:46.915918+0000 mon.a (mon.0) 15241 : audit [DBG] from='client.? 172.21.15.115:0/3368691980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[46934]: cluster 2022-04-22T22:47:46.111639+0000 mgr.y (mgr.34107) 10965 : cluster [DBG] pgmap v11590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[46934]: audit 2022-04-22T22:47:46.130793+0000 mgr.y (mgr.34107) 10966 : audit [DBG] from='client.46704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[46934]: audit 2022-04-22T22:47:46.482184+0000 mgr.y (mgr.34107) 10967 : audit [DBG] from='client.56647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:47:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:46 smithi115 bash[46934]: audit 2022-04-22T22:47:46.915918+0000 mon.a (mon.0) 15241 : audit [DBG] from='client.? 172.21.15.115:0/3368691980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:47:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:47 smithi121 bash[36902]: audit 2022-04-22T22:47:47.148474+0000 mon.a (mon.0) 15242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:47 smithi115 bash[45495]: audit 2022-04-22T22:47:47.148474+0000 mon.a (mon.0) 15242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:47 smithi115 bash[46934]: audit 2022-04-22T22:47:47.148474+0000 mon.a (mon.0) 15242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:48 smithi121 bash[36902]: cluster 2022-04-22T22:47:48.112784+0000 mgr.y (mgr.34107) 10968 : cluster [DBG] pgmap v11591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:48 smithi121 bash[36902]: audit 2022-04-22T22:47:48.615335+0000 mon.a (mon.0) 15243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:48 smithi115 bash[45495]: cluster 2022-04-22T22:47:48.112784+0000 mgr.y (mgr.34107) 10968 : cluster [DBG] pgmap v11591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:48 smithi115 bash[45495]: audit 2022-04-22T22:47:48.615335+0000 mon.a (mon.0) 15243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:48 smithi115 bash[46934]: cluster 2022-04-22T22:47:48.112784+0000 mgr.y (mgr.34107) 10968 : cluster [DBG] pgmap v11591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:48 smithi115 bash[46934]: audit 2022-04-22T22:47:48.615335+0000 mon.a (mon.0) 15243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:51 smithi121 bash[36902]: cluster 2022-04-22T22:47:50.113845+0000 mgr.y (mgr.34107) 10969 : cluster [DBG] pgmap v11592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:51 smithi121 bash[36902]: audit 2022-04-22T22:47:50.535824+0000 mon.a (mon.0) 15244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:51.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:51 smithi115 bash[46934]: cluster 2022-04-22T22:47:50.113845+0000 mgr.y (mgr.34107) 10969 : cluster [DBG] pgmap v11592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:51 smithi115 bash[46934]: audit 2022-04-22T22:47:50.535824+0000 mon.a (mon.0) 15244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:51 smithi115 bash[45495]: cluster 2022-04-22T22:47:50.113845+0000 mgr.y (mgr.34107) 10969 : cluster [DBG] pgmap v11592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:51 smithi115 bash[45495]: audit 2022-04-22T22:47:50.535824+0000 mon.a (mon.0) 15244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:52 smithi121 bash[36902]: cluster 2022-04-22T22:47:52.115000+0000 mgr.y (mgr.34107) 10970 : cluster [DBG] pgmap v11593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:52 smithi121 bash[36902]: audit 2022-04-22T22:47:52.151615+0000 mon.a (mon.0) 15245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:52.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:52 smithi115 bash[45495]: cluster 2022-04-22T22:47:52.115000+0000 mgr.y (mgr.34107) 10970 : cluster [DBG] pgmap v11593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:52 smithi115 bash[45495]: audit 2022-04-22T22:47:52.151615+0000 mon.a (mon.0) 15245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:52 smithi115 bash[46934]: cluster 2022-04-22T22:47:52.115000+0000 mgr.y (mgr.34107) 10970 : cluster [DBG] pgmap v11593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:52 smithi115 bash[46934]: audit 2022-04-22T22:47:52.151615+0000 mon.a (mon.0) 15245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:53 smithi121 bash[36902]: audit 2022-04-22T22:47:53.619020+0000 mon.a (mon.0) 15246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:53 smithi115 bash[45495]: audit 2022-04-22T22:47:53.619020+0000 mon.a (mon.0) 15246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:53 smithi115 bash[46934]: audit 2022-04-22T22:47:53.619020+0000 mon.a (mon.0) 15246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:54 smithi115 bash[45495]: cluster 2022-04-22T22:47:54.116178+0000 mgr.y (mgr.34107) 10971 : cluster [DBG] pgmap v11594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:47:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:47:54.743Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:47:55.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:54 smithi115 bash[46934]: cluster 2022-04-22T22:47:54.116178+0000 mgr.y (mgr.34107) 10971 : cluster [DBG] pgmap v11594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:54 smithi121 bash[36902]: cluster 2022-04-22T22:47:54.116178+0000 mgr.y (mgr.34107) 10971 : cluster [DBG] pgmap v11594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:55 smithi121 bash[36902]: audit 2022-04-22T22:47:55.538704+0000 mon.a (mon.0) 15247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:55 smithi115 bash[45495]: audit 2022-04-22T22:47:55.538704+0000 mon.a (mon.0) 15247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:55 smithi115 bash[46934]: audit 2022-04-22T22:47:55.538704+0000 mon.a (mon.0) 15247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:47:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:56 smithi121 bash[36902]: cluster 2022-04-22T22:47:56.117317+0000 mgr.y (mgr.34107) 10972 : cluster [DBG] pgmap v11595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:56 smithi115 bash[45495]: cluster 2022-04-22T22:47:56.117317+0000 mgr.y (mgr.34107) 10972 : cluster [DBG] pgmap v11595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:56 smithi115 bash[46934]: cluster 2022-04-22T22:47:56.117317+0000 mgr.y (mgr.34107) 10972 : cluster [DBG] pgmap v11595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:57 smithi121 bash[36902]: audit 2022-04-22T22:47:57.154861+0000 mon.a (mon.0) 15248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:57 smithi115 bash[45495]: audit 2022-04-22T22:47:57.154861+0000 mon.a (mon.0) 15248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:57 smithi115 bash[46934]: audit 2022-04-22T22:47:57.154861+0000 mon.a (mon.0) 15248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:47:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:58 smithi121 bash[36902]: cluster 2022-04-22T22:47:58.118442+0000 mgr.y (mgr.34107) 10973 : cluster [DBG] pgmap v11596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:47:58 smithi121 bash[36902]: audit 2022-04-22T22:47:58.622694+0000 mon.a (mon.0) 15249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:58 smithi115 bash[45495]: cluster 2022-04-22T22:47:58.118442+0000 mgr.y (mgr.34107) 10973 : cluster [DBG] pgmap v11596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:47:58 smithi115 bash[45495]: audit 2022-04-22T22:47:58.622694+0000 mon.a (mon.0) 15249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:47:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:58 smithi115 bash[46934]: cluster 2022-04-22T22:47:58.118442+0000 mgr.y (mgr.34107) 10973 : cluster [DBG] pgmap v11596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:47:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:47:58 smithi115 bash[46934]: audit 2022-04-22T22:47:58.622694+0000 mon.a (mon.0) 15249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: cluster 2022-04-22T22:48:00.119542+0000 mgr.y (mgr.34107) 10974 : cluster [DBG] pgmap v11597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: audit 2022-04-22T22:48:00.486449+0000 mon.a (mon.0) 15250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:48:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: audit 2022-04-22T22:48:00.489497+0000 mon.a (mon.0) 15251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:48:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: audit 2022-04-22T22:48:00.491455+0000 mon.a (mon.0) 15252 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:48:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: audit 2022-04-22T22:48:00.541926+0000 mon.a (mon.0) 15253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:01 smithi121 bash[36902]: audit 2022-04-22T22:48:00.836026+0000 mon.a (mon.0) 15254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:48:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: cluster 2022-04-22T22:48:00.119542+0000 mgr.y (mgr.34107) 10974 : cluster [DBG] pgmap v11597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: audit 2022-04-22T22:48:00.486449+0000 mon.a (mon.0) 15250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: audit 2022-04-22T22:48:00.489497+0000 mon.a (mon.0) 15251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: audit 2022-04-22T22:48:00.491455+0000 mon.a (mon.0) 15252 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: audit 2022-04-22T22:48:00.541926+0000 mon.a (mon.0) 15253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[45495]: audit 2022-04-22T22:48:00.836026+0000 mon.a (mon.0) 15254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: cluster 2022-04-22T22:48:00.119542+0000 mgr.y (mgr.34107) 10974 : cluster [DBG] pgmap v11597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: audit 2022-04-22T22:48:00.486449+0000 mon.a (mon.0) 15250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:48:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: audit 2022-04-22T22:48:00.489497+0000 mon.a (mon.0) 15251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:48:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: audit 2022-04-22T22:48:00.491455+0000 mon.a (mon.0) 15252 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:48:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: audit 2022-04-22T22:48:00.541926+0000 mon.a (mon.0) 15253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:01 smithi115 bash[46934]: audit 2022-04-22T22:48:00.836026+0000 mon.a (mon.0) 15254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:48:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:02 smithi121 bash[36902]: cluster 2022-04-22T22:48:02.120640+0000 mgr.y (mgr.34107) 10975 : cluster [DBG] pgmap v11598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:02 smithi121 bash[36902]: audit 2022-04-22T22:48:02.158038+0000 mon.a (mon.0) 15255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:02 smithi115 bash[45495]: cluster 2022-04-22T22:48:02.120640+0000 mgr.y (mgr.34107) 10975 : cluster [DBG] pgmap v11598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:02 smithi115 bash[45495]: audit 2022-04-22T22:48:02.158038+0000 mon.a (mon.0) 15255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:02 smithi115 bash[46934]: cluster 2022-04-22T22:48:02.120640+0000 mgr.y (mgr.34107) 10975 : cluster [DBG] pgmap v11598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:02 smithi115 bash[46934]: audit 2022-04-22T22:48:02.158038+0000 mon.a (mon.0) 15255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:03.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:02.883Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:03 smithi121 bash[36902]: audit 2022-04-22T22:48:03.626456+0000 mon.a (mon.0) 15256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:04.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:03 smithi115 bash[46934]: audit 2022-04-22T22:48:03.626456+0000 mon.a (mon.0) 15256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:03 smithi115 bash[45495]: audit 2022-04-22T22:48:03.626456+0000 mon.a (mon.0) 15256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:04 smithi115 bash[45495]: cluster 2022-04-22T22:48:04.121779+0000 mgr.y (mgr.34107) 10976 : cluster [DBG] pgmap v11599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:04.744Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:04 smithi115 bash[46934]: cluster 2022-04-22T22:48:04.121779+0000 mgr.y (mgr.34107) 10976 : cluster [DBG] pgmap v11599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:04 smithi121 bash[36902]: cluster 2022-04-22T22:48:04.121779+0000 mgr.y (mgr.34107) 10976 : cluster [DBG] pgmap v11599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:05 smithi121 bash[36902]: audit 2022-04-22T22:48:05.544541+0000 mon.a (mon.0) 15257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:05 smithi115 bash[45495]: audit 2022-04-22T22:48:05.544541+0000 mon.a (mon.0) 15257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:05 smithi115 bash[46934]: audit 2022-04-22T22:48:05.544541+0000 mon.a (mon.0) 15257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:06 smithi121 bash[36902]: cluster 2022-04-22T22:48:06.122859+0000 mgr.y (mgr.34107) 10977 : cluster [DBG] pgmap v11600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:06 smithi115 bash[45495]: cluster 2022-04-22T22:48:06.122859+0000 mgr.y (mgr.34107) 10977 : cluster [DBG] pgmap v11600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:06 smithi115 bash[46934]: cluster 2022-04-22T22:48:06.122859+0000 mgr.y (mgr.34107) 10977 : cluster [DBG] pgmap v11600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:07 smithi121 bash[36902]: audit 2022-04-22T22:48:07.160882+0000 mon.a (mon.0) 15258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:07 smithi115 bash[45495]: audit 2022-04-22T22:48:07.160882+0000 mon.a (mon.0) 15258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:07 smithi115 bash[46934]: audit 2022-04-22T22:48:07.160882+0000 mon.a (mon.0) 15258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:08 smithi121 bash[36902]: cluster 2022-04-22T22:48:08.124015+0000 mgr.y (mgr.34107) 10978 : cluster [DBG] pgmap v11601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:08 smithi121 bash[36902]: audit 2022-04-22T22:48:08.630281+0000 mon.a (mon.0) 15259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:08 smithi115 bash[45495]: cluster 2022-04-22T22:48:08.124015+0000 mgr.y (mgr.34107) 10978 : cluster [DBG] pgmap v11601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:08 smithi115 bash[45495]: audit 2022-04-22T22:48:08.630281+0000 mon.a (mon.0) 15259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:08 smithi115 bash[46934]: cluster 2022-04-22T22:48:08.124015+0000 mgr.y (mgr.34107) 10978 : cluster [DBG] pgmap v11601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:08 smithi115 bash[46934]: audit 2022-04-22T22:48:08.630281+0000 mon.a (mon.0) 15259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:11 smithi121 bash[36902]: cluster 2022-04-22T22:48:10.124904+0000 mgr.y (mgr.34107) 10979 : cluster [DBG] pgmap v11602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:11 smithi121 bash[36902]: audit 2022-04-22T22:48:10.547551+0000 mon.a (mon.0) 15260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:11 smithi115 bash[45495]: cluster 2022-04-22T22:48:10.124904+0000 mgr.y (mgr.34107) 10979 : cluster [DBG] pgmap v11602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:11 smithi115 bash[45495]: audit 2022-04-22T22:48:10.547551+0000 mon.a (mon.0) 15260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:11 smithi115 bash[46934]: cluster 2022-04-22T22:48:10.124904+0000 mgr.y (mgr.34107) 10979 : cluster [DBG] pgmap v11602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:11 smithi115 bash[46934]: audit 2022-04-22T22:48:10.547551+0000 mon.a (mon.0) 15260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:12 smithi121 bash[36902]: cluster 2022-04-22T22:48:12.126028+0000 mgr.y (mgr.34107) 10980 : cluster [DBG] pgmap v11603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:12 smithi121 bash[36902]: audit 2022-04-22T22:48:12.164150+0000 mon.a (mon.0) 15261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:12 smithi115 bash[45495]: cluster 2022-04-22T22:48:12.126028+0000 mgr.y (mgr.34107) 10980 : cluster [DBG] pgmap v11603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:12 smithi115 bash[45495]: audit 2022-04-22T22:48:12.164150+0000 mon.a (mon.0) 15261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:12 smithi115 bash[46934]: cluster 2022-04-22T22:48:12.126028+0000 mgr.y (mgr.34107) 10980 : cluster [DBG] pgmap v11603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:12 smithi115 bash[46934]: audit 2022-04-22T22:48:12.164150+0000 mon.a (mon.0) 15261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:13.293 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:12.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:13 smithi121 bash[36902]: audit 2022-04-22T22:48:13.634140+0000 mon.a (mon.0) 15262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:14.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:13 smithi115 bash[46934]: audit 2022-04-22T22:48:13.634140+0000 mon.a (mon.0) 15262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:13 smithi115 bash[45495]: audit 2022-04-22T22:48:13.634140+0000 mon.a (mon.0) 15262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:14 smithi115 bash[45495]: cluster 2022-04-22T22:48:14.127263+0000 mgr.y (mgr.34107) 10981 : cluster [DBG] pgmap v11604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:14.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:14 smithi115 bash[46934]: cluster 2022-04-22T22:48:14.127263+0000 mgr.y (mgr.34107) 10981 : cluster [DBG] pgmap v11604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:14 smithi121 bash[36902]: cluster 2022-04-22T22:48:14.127263+0000 mgr.y (mgr.34107) 10981 : cluster [DBG] pgmap v11604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:15 smithi121 bash[36902]: audit 2022-04-22T22:48:15.550294+0000 mon.a (mon.0) 15263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:15 smithi115 bash[45495]: audit 2022-04-22T22:48:15.550294+0000 mon.a (mon.0) 15263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:15 smithi115 bash[46934]: audit 2022-04-22T22:48:15.550294+0000 mon.a (mon.0) 15263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:16 smithi121 bash[36902]: cluster 2022-04-22T22:48:16.128366+0000 mgr.y (mgr.34107) 10982 : cluster [DBG] pgmap v11605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:16 smithi115 bash[45495]: cluster 2022-04-22T22:48:16.128366+0000 mgr.y (mgr.34107) 10982 : cluster [DBG] pgmap v11605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:16 smithi115 bash[46934]: cluster 2022-04-22T22:48:16.128366+0000 mgr.y (mgr.34107) 10982 : cluster [DBG] pgmap v11605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:17.294 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:48:17.636 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:48:17.636 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 86s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:48:17.636 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:48:17.636 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 86s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:48:17.636 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:48:17.637 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:48:17.637 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 86s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:48:17.638 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 86s ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:48:17.638 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:48:17.638 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 86s ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:48:17.638 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 86s ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 86s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 86s ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 86s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 86s ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:48:17.639 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:48:18.053 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:48:18.053 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:48:18.053 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:48:18.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:48:18.054 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:48:18.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:18.055 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:48:18.055 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:48:18.055 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:48:18.055 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:48:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:17 smithi121 bash[36902]: audit 2022-04-22T22:48:16.977230+0000 mon.a (mon.0) 15264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:48:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:17 smithi121 bash[36902]: audit 2022-04-22T22:48:16.977535+0000 mon.a (mon.0) 15265 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:48:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:17 smithi121 bash[36902]: audit 2022-04-22T22:48:17.167155+0000 mon.a (mon.0) 15266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:17 smithi121 bash[36902]: audit 2022-04-22T22:48:17.276334+0000 mgr.y (mgr.34107) 10983 : audit [DBG] from='client.46722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:17 smithi121 bash[36902]: audit 2022-04-22T22:48:17.613852+0000 mgr.y (mgr.34107) 10984 : audit [DBG] from='client.56665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[45495]: audit 2022-04-22T22:48:16.977230+0000 mon.a (mon.0) 15264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:48:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[45495]: audit 2022-04-22T22:48:16.977535+0000 mon.a (mon.0) 15265 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:48:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[45495]: audit 2022-04-22T22:48:17.167155+0000 mon.a (mon.0) 15266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[45495]: audit 2022-04-22T22:48:17.276334+0000 mgr.y (mgr.34107) 10983 : audit [DBG] from='client.46722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[45495]: audit 2022-04-22T22:48:17.613852+0000 mgr.y (mgr.34107) 10984 : audit [DBG] from='client.56665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[46934]: audit 2022-04-22T22:48:16.977230+0000 mon.a (mon.0) 15264 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:48:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[46934]: audit 2022-04-22T22:48:16.977535+0000 mon.a (mon.0) 15265 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:48:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[46934]: audit 2022-04-22T22:48:17.167155+0000 mon.a (mon.0) 15266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[46934]: audit 2022-04-22T22:48:17.276334+0000 mgr.y (mgr.34107) 10983 : audit [DBG] from='client.46722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:18.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:17 smithi115 bash[46934]: audit 2022-04-22T22:48:17.613852+0000 mgr.y (mgr.34107) 10984 : audit [DBG] from='client.56665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:18 smithi121 bash[36902]: audit 2022-04-22T22:48:18.049746+0000 mon.a (mon.0) 15267 : audit [DBG] from='client.? 172.21.15.115:0/162720999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:18 smithi121 bash[36902]: cluster 2022-04-22T22:48:18.129440+0000 mgr.y (mgr.34107) 10985 : cluster [DBG] pgmap v11606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:18 smithi121 bash[36902]: audit 2022-04-22T22:48:18.637588+0000 mon.a (mon.0) 15268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[45495]: audit 2022-04-22T22:48:18.049746+0000 mon.a (mon.0) 15267 : audit [DBG] from='client.? 172.21.15.115:0/162720999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[45495]: cluster 2022-04-22T22:48:18.129440+0000 mgr.y (mgr.34107) 10985 : cluster [DBG] pgmap v11606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[45495]: audit 2022-04-22T22:48:18.637588+0000 mon.a (mon.0) 15268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[46934]: audit 2022-04-22T22:48:18.049746+0000 mon.a (mon.0) 15267 : audit [DBG] from='client.? 172.21.15.115:0/162720999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:19.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[46934]: cluster 2022-04-22T22:48:18.129440+0000 mgr.y (mgr.34107) 10985 : cluster [DBG] pgmap v11606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:19.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:18 smithi115 bash[46934]: audit 2022-04-22T22:48:18.637588+0000 mon.a (mon.0) 15268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:21 smithi121 bash[36902]: cluster 2022-04-22T22:48:20.130490+0000 mgr.y (mgr.34107) 10986 : cluster [DBG] pgmap v11607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:21 smithi121 bash[36902]: audit 2022-04-22T22:48:20.553166+0000 mon.a (mon.0) 15269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:21 smithi115 bash[45495]: cluster 2022-04-22T22:48:20.130490+0000 mgr.y (mgr.34107) 10986 : cluster [DBG] pgmap v11607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:21 smithi115 bash[45495]: audit 2022-04-22T22:48:20.553166+0000 mon.a (mon.0) 15269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:21 smithi115 bash[46934]: cluster 2022-04-22T22:48:20.130490+0000 mgr.y (mgr.34107) 10986 : cluster [DBG] pgmap v11607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:21 smithi115 bash[46934]: audit 2022-04-22T22:48:20.553166+0000 mon.a (mon.0) 15269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:22 smithi121 bash[36902]: cluster 2022-04-22T22:48:22.131731+0000 mgr.y (mgr.34107) 10987 : cluster [DBG] pgmap v11608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:22 smithi121 bash[36902]: audit 2022-04-22T22:48:22.170194+0000 mon.a (mon.0) 15270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:22 smithi115 bash[45495]: cluster 2022-04-22T22:48:22.131731+0000 mgr.y (mgr.34107) 10987 : cluster [DBG] pgmap v11608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:22 smithi115 bash[45495]: audit 2022-04-22T22:48:22.170194+0000 mon.a (mon.0) 15270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:22 smithi115 bash[46934]: cluster 2022-04-22T22:48:22.131731+0000 mgr.y (mgr.34107) 10987 : cluster [DBG] pgmap v11608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:22 smithi115 bash[46934]: audit 2022-04-22T22:48:22.170194+0000 mon.a (mon.0) 15270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:23.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:22.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:23 smithi121 bash[36902]: audit 2022-04-22T22:48:23.641372+0000 mon.a (mon.0) 15271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:23 smithi115 bash[45495]: audit 2022-04-22T22:48:23.641372+0000 mon.a (mon.0) 15271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:23 smithi115 bash[46934]: audit 2022-04-22T22:48:23.641372+0000 mon.a (mon.0) 15271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:25.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:24 smithi115 bash[45495]: cluster 2022-04-22T22:48:24.132798+0000 mgr.y (mgr.34107) 10988 : cluster [DBG] pgmap v11609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:24 smithi115 bash[46934]: cluster 2022-04-22T22:48:24.132798+0000 mgr.y (mgr.34107) 10988 : cluster [DBG] pgmap v11609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:25.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:24.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:24 smithi121 bash[36902]: cluster 2022-04-22T22:48:24.132798+0000 mgr.y (mgr.34107) 10988 : cluster [DBG] pgmap v11609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:25 smithi121 bash[36902]: audit 2022-04-22T22:48:25.556172+0000 mon.a (mon.0) 15272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:25 smithi115 bash[45495]: audit 2022-04-22T22:48:25.556172+0000 mon.a (mon.0) 15272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:25 smithi115 bash[46934]: audit 2022-04-22T22:48:25.556172+0000 mon.a (mon.0) 15272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:26 smithi121 bash[36902]: cluster 2022-04-22T22:48:26.133901+0000 mgr.y (mgr.34107) 10989 : cluster [DBG] pgmap v11610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:26 smithi115 bash[45495]: cluster 2022-04-22T22:48:26.133901+0000 mgr.y (mgr.34107) 10989 : cluster [DBG] pgmap v11610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:26 smithi115 bash[46934]: cluster 2022-04-22T22:48:26.133901+0000 mgr.y (mgr.34107) 10989 : cluster [DBG] pgmap v11610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:27 smithi121 bash[36902]: audit 2022-04-22T22:48:27.173175+0000 mon.a (mon.0) 15273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:27 smithi115 bash[45495]: audit 2022-04-22T22:48:27.173175+0000 mon.a (mon.0) 15273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:27 smithi115 bash[46934]: audit 2022-04-22T22:48:27.173175+0000 mon.a (mon.0) 15273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:28 smithi121 bash[36902]: cluster 2022-04-22T22:48:28.135166+0000 mgr.y (mgr.34107) 10990 : cluster [DBG] pgmap v11611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:28 smithi121 bash[36902]: audit 2022-04-22T22:48:28.645128+0000 mon.a (mon.0) 15274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:28 smithi115 bash[45495]: cluster 2022-04-22T22:48:28.135166+0000 mgr.y (mgr.34107) 10990 : cluster [DBG] pgmap v11611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:28 smithi115 bash[45495]: audit 2022-04-22T22:48:28.645128+0000 mon.a (mon.0) 15274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:28 smithi115 bash[46934]: cluster 2022-04-22T22:48:28.135166+0000 mgr.y (mgr.34107) 10990 : cluster [DBG] pgmap v11611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:28 smithi115 bash[46934]: audit 2022-04-22T22:48:28.645128+0000 mon.a (mon.0) 15274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:31 smithi121 bash[36902]: cluster 2022-04-22T22:48:30.136166+0000 mgr.y (mgr.34107) 10991 : cluster [DBG] pgmap v11612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:31 smithi121 bash[36902]: audit 2022-04-22T22:48:30.559294+0000 mon.a (mon.0) 15275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:31 smithi115 bash[45495]: cluster 2022-04-22T22:48:30.136166+0000 mgr.y (mgr.34107) 10991 : cluster [DBG] pgmap v11612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:31 smithi115 bash[45495]: audit 2022-04-22T22:48:30.559294+0000 mon.a (mon.0) 15275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:31 smithi115 bash[46934]: cluster 2022-04-22T22:48:30.136166+0000 mgr.y (mgr.34107) 10991 : cluster [DBG] pgmap v11612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:31 smithi115 bash[46934]: audit 2022-04-22T22:48:30.559294+0000 mon.a (mon.0) 15275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:32 smithi121 bash[36902]: cluster 2022-04-22T22:48:32.137348+0000 mgr.y (mgr.34107) 10992 : cluster [DBG] pgmap v11613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:33.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:32 smithi121 bash[36902]: audit 2022-04-22T22:48:32.176374+0000 mon.a (mon.0) 15276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:33.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:32 smithi115 bash[45495]: cluster 2022-04-22T22:48:32.137348+0000 mgr.y (mgr.34107) 10992 : cluster [DBG] pgmap v11613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:33.308 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:32 smithi115 bash[45495]: audit 2022-04-22T22:48:32.176374+0000 mon.a (mon.0) 15276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:33.309 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:32.885Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:33.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:32 smithi115 bash[46934]: cluster 2022-04-22T22:48:32.137348+0000 mgr.y (mgr.34107) 10992 : cluster [DBG] pgmap v11613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:33.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:32 smithi115 bash[46934]: audit 2022-04-22T22:48:32.176374+0000 mon.a (mon.0) 15276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:33 smithi121 bash[36902]: audit 2022-04-22T22:48:33.648783+0000 mon.a (mon.0) 15277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:33 smithi115 bash[45495]: audit 2022-04-22T22:48:33.648783+0000 mon.a (mon.0) 15277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:33 smithi115 bash[46934]: audit 2022-04-22T22:48:33.648783+0000 mon.a (mon.0) 15277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:34 smithi115 bash[45495]: cluster 2022-04-22T22:48:34.138487+0000 mgr.y (mgr.34107) 10993 : cluster [DBG] pgmap v11614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:34 smithi115 bash[46934]: cluster 2022-04-22T22:48:34.138487+0000 mgr.y (mgr.34107) 10993 : cluster [DBG] pgmap v11614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:34.745Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:34 smithi121 bash[36902]: cluster 2022-04-22T22:48:34.138487+0000 mgr.y (mgr.34107) 10993 : cluster [DBG] pgmap v11614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:35 smithi121 bash[36902]: audit 2022-04-22T22:48:35.562240+0000 mon.a (mon.0) 15278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:35 smithi115 bash[45495]: audit 2022-04-22T22:48:35.562240+0000 mon.a (mon.0) 15278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:35 smithi115 bash[46934]: audit 2022-04-22T22:48:35.562240+0000 mon.a (mon.0) 15278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:36 smithi121 bash[36902]: cluster 2022-04-22T22:48:36.139706+0000 mgr.y (mgr.34107) 10994 : cluster [DBG] pgmap v11615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:36 smithi115 bash[45495]: cluster 2022-04-22T22:48:36.139706+0000 mgr.y (mgr.34107) 10994 : cluster [DBG] pgmap v11615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:36 smithi115 bash[46934]: cluster 2022-04-22T22:48:36.139706+0000 mgr.y (mgr.34107) 10994 : cluster [DBG] pgmap v11615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:37 smithi121 bash[36902]: audit 2022-04-22T22:48:37.179446+0000 mon.a (mon.0) 15279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:37 smithi115 bash[45495]: audit 2022-04-22T22:48:37.179446+0000 mon.a (mon.0) 15279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:37 smithi115 bash[46934]: audit 2022-04-22T22:48:37.179446+0000 mon.a (mon.0) 15279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:38 smithi121 bash[36902]: cluster 2022-04-22T22:48:38.140715+0000 mgr.y (mgr.34107) 10995 : cluster [DBG] pgmap v11616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:38 smithi121 bash[36902]: audit 2022-04-22T22:48:38.652455+0000 mon.a (mon.0) 15280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:38 smithi115 bash[45495]: cluster 2022-04-22T22:48:38.140715+0000 mgr.y (mgr.34107) 10995 : cluster [DBG] pgmap v11616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:38 smithi115 bash[45495]: audit 2022-04-22T22:48:38.652455+0000 mon.a (mon.0) 15280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:38 smithi115 bash[46934]: cluster 2022-04-22T22:48:38.140715+0000 mgr.y (mgr.34107) 10995 : cluster [DBG] pgmap v11616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:38 smithi115 bash[46934]: audit 2022-04-22T22:48:38.652455+0000 mon.a (mon.0) 15280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:41 smithi121 bash[36902]: cluster 2022-04-22T22:48:40.141715+0000 mgr.y (mgr.34107) 10996 : cluster [DBG] pgmap v11617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:41 smithi121 bash[36902]: audit 2022-04-22T22:48:40.565187+0000 mon.a (mon.0) 15281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:41 smithi115 bash[46934]: cluster 2022-04-22T22:48:40.141715+0000 mgr.y (mgr.34107) 10996 : cluster [DBG] pgmap v11617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:41 smithi115 bash[46934]: audit 2022-04-22T22:48:40.565187+0000 mon.a (mon.0) 15281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:41 smithi115 bash[45495]: cluster 2022-04-22T22:48:40.141715+0000 mgr.y (mgr.34107) 10996 : cluster [DBG] pgmap v11617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:41 smithi115 bash[45495]: audit 2022-04-22T22:48:40.565187+0000 mon.a (mon.0) 15281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:42 smithi121 bash[36902]: cluster 2022-04-22T22:48:42.142778+0000 mgr.y (mgr.34107) 10997 : cluster [DBG] pgmap v11618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:42 smithi121 bash[36902]: audit 2022-04-22T22:48:42.182445+0000 mon.a (mon.0) 15282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:42.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:42 smithi115 bash[45495]: cluster 2022-04-22T22:48:42.142778+0000 mgr.y (mgr.34107) 10997 : cluster [DBG] pgmap v11618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:42 smithi115 bash[45495]: audit 2022-04-22T22:48:42.182445+0000 mon.a (mon.0) 15282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:42 smithi115 bash[46934]: cluster 2022-04-22T22:48:42.142778+0000 mgr.y (mgr.34107) 10997 : cluster [DBG] pgmap v11618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:42 smithi115 bash[46934]: audit 2022-04-22T22:48:42.182445+0000 mon.a (mon.0) 15282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:43 smithi121 bash[36902]: audit 2022-04-22T22:48:43.656306+0000 mon.a (mon.0) 15283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:44.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:43 smithi115 bash[46934]: audit 2022-04-22T22:48:43.656306+0000 mon.a (mon.0) 15283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:43 smithi115 bash[45495]: audit 2022-04-22T22:48:43.656306+0000 mon.a (mon.0) 15283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:44.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:44 smithi115 bash[45495]: cluster 2022-04-22T22:48:44.143932+0000 mgr.y (mgr.34107) 10998 : cluster [DBG] pgmap v11619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:44 smithi115 bash[46934]: cluster 2022-04-22T22:48:44.143932+0000 mgr.y (mgr.34107) 10998 : cluster [DBG] pgmap v11619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:44 smithi121 bash[36902]: cluster 2022-04-22T22:48:44.143932+0000 mgr.y (mgr.34107) 10998 : cluster [DBG] pgmap v11619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:45 smithi121 bash[36902]: audit 2022-04-22T22:48:45.568173+0000 mon.a (mon.0) 15284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:45 smithi115 bash[45495]: audit 2022-04-22T22:48:45.568173+0000 mon.a (mon.0) 15284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:45 smithi115 bash[46934]: audit 2022-04-22T22:48:45.568173+0000 mon.a (mon.0) 15284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:46 smithi121 bash[36902]: cluster 2022-04-22T22:48:46.145070+0000 mgr.y (mgr.34107) 10999 : cluster [DBG] pgmap v11620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:46 smithi115 bash[45495]: cluster 2022-04-22T22:48:46.145070+0000 mgr.y (mgr.34107) 10999 : cluster [DBG] pgmap v11620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:46 smithi115 bash[46934]: cluster 2022-04-22T22:48:46.145070+0000 mgr.y (mgr.34107) 10999 : cluster [DBG] pgmap v11620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:47 smithi121 bash[36902]: audit 2022-04-22T22:48:47.185394+0000 mon.a (mon.0) 15285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:47 smithi115 bash[45495]: audit 2022-04-22T22:48:47.185394+0000 mon.a (mon.0) 15285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:47 smithi115 bash[46934]: audit 2022-04-22T22:48:47.185394+0000 mon.a (mon.0) 15285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:48.430 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:48:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:48:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 117s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:48:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:48:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 117s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:48:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 117s ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 117s ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 117s ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 117s ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 117s ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 117s ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:48:48.771 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 117s ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 117s ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:48:48.772 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:48:49.169 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:48:49.169 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:49.170 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:48:49.171 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:48:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:48 smithi121 bash[36902]: cluster 2022-04-22T22:48:48.146353+0000 mgr.y (mgr.34107) 11000 : cluster [DBG] pgmap v11621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:48 smithi121 bash[36902]: audit 2022-04-22T22:48:48.412611+0000 mgr.y (mgr.34107) 11001 : audit [DBG] from='client.56677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:48 smithi121 bash[36902]: audit 2022-04-22T22:48:48.659911+0000 mon.a (mon.0) 15286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:48 smithi121 bash[36902]: audit 2022-04-22T22:48:48.752833+0000 mgr.y (mgr.34107) 11002 : audit [DBG] from='client.56683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[45495]: cluster 2022-04-22T22:48:48.146353+0000 mgr.y (mgr.34107) 11000 : cluster [DBG] pgmap v11621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[45495]: audit 2022-04-22T22:48:48.412611+0000 mgr.y (mgr.34107) 11001 : audit [DBG] from='client.56677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[45495]: audit 2022-04-22T22:48:48.659911+0000 mon.a (mon.0) 15286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[45495]: audit 2022-04-22T22:48:48.752833+0000 mgr.y (mgr.34107) 11002 : audit [DBG] from='client.56683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[46934]: cluster 2022-04-22T22:48:48.146353+0000 mgr.y (mgr.34107) 11000 : cluster [DBG] pgmap v11621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[46934]: audit 2022-04-22T22:48:48.412611+0000 mgr.y (mgr.34107) 11001 : audit [DBG] from='client.56677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[46934]: audit 2022-04-22T22:48:48.659911+0000 mon.a (mon.0) 15286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:48 smithi115 bash[46934]: audit 2022-04-22T22:48:48.752833+0000 mgr.y (mgr.34107) 11002 : audit [DBG] from='client.56683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:48:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:49 smithi121 bash[36902]: audit 2022-04-22T22:48:49.165854+0000 mon.a (mon.0) 15287 : audit [DBG] from='client.? 172.21.15.115:0/3409280437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:49 smithi115 bash[45495]: audit 2022-04-22T22:48:49.165854+0000 mon.a (mon.0) 15287 : audit [DBG] from='client.? 172.21.15.115:0/3409280437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:50.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:49 smithi115 bash[46934]: audit 2022-04-22T22:48:49.165854+0000 mon.a (mon.0) 15287 : audit [DBG] from='client.? 172.21.15.115:0/3409280437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:48:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:51 smithi121 bash[36902]: cluster 2022-04-22T22:48:50.147315+0000 mgr.y (mgr.34107) 11003 : cluster [DBG] pgmap v11622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:51.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:51 smithi121 bash[36902]: audit 2022-04-22T22:48:50.571227+0000 mon.a (mon.0) 15288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:51.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:51 smithi115 bash[45495]: cluster 2022-04-22T22:48:50.147315+0000 mgr.y (mgr.34107) 11003 : cluster [DBG] pgmap v11622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:51 smithi115 bash[45495]: audit 2022-04-22T22:48:50.571227+0000 mon.a (mon.0) 15288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:51 smithi115 bash[46934]: cluster 2022-04-22T22:48:50.147315+0000 mgr.y (mgr.34107) 11003 : cluster [DBG] pgmap v11622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:51 smithi115 bash[46934]: audit 2022-04-22T22:48:50.571227+0000 mon.a (mon.0) 15288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:52 smithi121 bash[36902]: cluster 2022-04-22T22:48:52.148378+0000 mgr.y (mgr.34107) 11004 : cluster [DBG] pgmap v11623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:52 smithi121 bash[36902]: audit 2022-04-22T22:48:52.188537+0000 mon.a (mon.0) 15289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:53.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:52 smithi115 bash[46934]: cluster 2022-04-22T22:48:52.148378+0000 mgr.y (mgr.34107) 11004 : cluster [DBG] pgmap v11623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:52 smithi115 bash[46934]: audit 2022-04-22T22:48:52.188537+0000 mon.a (mon.0) 15289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:52.886Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:52 smithi115 bash[45495]: cluster 2022-04-22T22:48:52.148378+0000 mgr.y (mgr.34107) 11004 : cluster [DBG] pgmap v11623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:52 smithi115 bash[45495]: audit 2022-04-22T22:48:52.188537+0000 mon.a (mon.0) 15289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:53 smithi121 bash[36902]: audit 2022-04-22T22:48:53.663419+0000 mon.a (mon.0) 15290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:53 smithi115 bash[45495]: audit 2022-04-22T22:48:53.663419+0000 mon.a (mon.0) 15290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:53 smithi115 bash[46934]: audit 2022-04-22T22:48:53.663419+0000 mon.a (mon.0) 15290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:54 smithi115 bash[45495]: cluster 2022-04-22T22:48:54.149443+0000 mgr.y (mgr.34107) 11005 : cluster [DBG] pgmap v11624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:54 smithi115 bash[46934]: cluster 2022-04-22T22:48:54.149443+0000 mgr.y (mgr.34107) 11005 : cluster [DBG] pgmap v11624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:48:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:48:54.746Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:48:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:54 smithi121 bash[36902]: cluster 2022-04-22T22:48:54.149443+0000 mgr.y (mgr.34107) 11005 : cluster [DBG] pgmap v11624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:55 smithi121 bash[36902]: audit 2022-04-22T22:48:55.574360+0000 mon.a (mon.0) 15291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:55 smithi115 bash[45495]: audit 2022-04-22T22:48:55.574360+0000 mon.a (mon.0) 15291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:55 smithi115 bash[46934]: audit 2022-04-22T22:48:55.574360+0000 mon.a (mon.0) 15291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:48:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:56 smithi121 bash[36902]: cluster 2022-04-22T22:48:56.150571+0000 mgr.y (mgr.34107) 11006 : cluster [DBG] pgmap v11625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:56 smithi115 bash[45495]: cluster 2022-04-22T22:48:56.150571+0000 mgr.y (mgr.34107) 11006 : cluster [DBG] pgmap v11625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:56 smithi115 bash[46934]: cluster 2022-04-22T22:48:56.150571+0000 mgr.y (mgr.34107) 11006 : cluster [DBG] pgmap v11625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:57 smithi121 bash[36902]: audit 2022-04-22T22:48:57.191555+0000 mon.a (mon.0) 15292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:57 smithi115 bash[45495]: audit 2022-04-22T22:48:57.191555+0000 mon.a (mon.0) 15292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:57 smithi115 bash[46934]: audit 2022-04-22T22:48:57.191555+0000 mon.a (mon.0) 15292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:48:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:59 smithi121 bash[36902]: cluster 2022-04-22T22:48:58.151738+0000 mgr.y (mgr.34107) 11007 : cluster [DBG] pgmap v11626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:48:59 smithi121 bash[36902]: audit 2022-04-22T22:48:58.667266+0000 mon.a (mon.0) 15293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:58 smithi115 bash[45495]: cluster 2022-04-22T22:48:58.151738+0000 mgr.y (mgr.34107) 11007 : cluster [DBG] pgmap v11626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:48:58 smithi115 bash[45495]: audit 2022-04-22T22:48:58.667266+0000 mon.a (mon.0) 15293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:48:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:59 smithi115 bash[46934]: cluster 2022-04-22T22:48:58.151738+0000 mgr.y (mgr.34107) 11007 : cluster [DBG] pgmap v11626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:48:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:48:59 smithi115 bash[46934]: audit 2022-04-22T22:48:58.667266+0000 mon.a (mon.0) 15293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: cluster 2022-04-22T22:49:00.152696+0000 mgr.y (mgr.34107) 11008 : cluster [DBG] pgmap v11627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: audit 2022-04-22T22:49:00.577175+0000 mon.a (mon.0) 15294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: audit 2022-04-22T22:49:00.846034+0000 mon.a (mon.0) 15295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:49:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: audit 2022-04-22T22:49:00.848968+0000 mon.a (mon.0) 15296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:49:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: audit 2022-04-22T22:49:00.850794+0000 mon.a (mon.0) 15297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:49:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:01 smithi121 bash[36902]: audit 2022-04-22T22:49:01.199654+0000 mon.a (mon.0) 15298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:49:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: cluster 2022-04-22T22:49:00.152696+0000 mgr.y (mgr.34107) 11008 : cluster [DBG] pgmap v11627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: audit 2022-04-22T22:49:00.577175+0000 mon.a (mon.0) 15294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: audit 2022-04-22T22:49:00.846034+0000 mon.a (mon.0) 15295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: audit 2022-04-22T22:49:00.848968+0000 mon.a (mon.0) 15296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: audit 2022-04-22T22:49:00.850794+0000 mon.a (mon.0) 15297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[45495]: audit 2022-04-22T22:49:01.199654+0000 mon.a (mon.0) 15298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: cluster 2022-04-22T22:49:00.152696+0000 mgr.y (mgr.34107) 11008 : cluster [DBG] pgmap v11627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: audit 2022-04-22T22:49:00.577175+0000 mon.a (mon.0) 15294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: audit 2022-04-22T22:49:00.846034+0000 mon.a (mon.0) 15295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:49:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: audit 2022-04-22T22:49:00.848968+0000 mon.a (mon.0) 15296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:49:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: audit 2022-04-22T22:49:00.850794+0000 mon.a (mon.0) 15297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:49:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:01 smithi115 bash[46934]: audit 2022-04-22T22:49:01.199654+0000 mon.a (mon.0) 15298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:49:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:02 smithi121 bash[36902]: cluster 2022-04-22T22:49:02.153829+0000 mgr.y (mgr.34107) 11009 : cluster [DBG] pgmap v11628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:03.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:02 smithi121 bash[36902]: audit 2022-04-22T22:49:02.194582+0000 mon.a (mon.0) 15299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:02.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:02 smithi115 bash[45495]: cluster 2022-04-22T22:49:02.153829+0000 mgr.y (mgr.34107) 11009 : cluster [DBG] pgmap v11628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:02 smithi115 bash[45495]: audit 2022-04-22T22:49:02.194582+0000 mon.a (mon.0) 15299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:02 smithi115 bash[46934]: cluster 2022-04-22T22:49:02.153829+0000 mgr.y (mgr.34107) 11009 : cluster [DBG] pgmap v11628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:02 smithi115 bash[46934]: audit 2022-04-22T22:49:02.194582+0000 mon.a (mon.0) 15299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:04.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:03 smithi121 bash[36902]: audit 2022-04-22T22:49:03.670836+0000 mon.a (mon.0) 15300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:04.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:03 smithi115 bash[46934]: audit 2022-04-22T22:49:03.670836+0000 mon.a (mon.0) 15300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:03 smithi115 bash[45495]: audit 2022-04-22T22:49:03.670836+0000 mon.a (mon.0) 15300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:04 smithi115 bash[45495]: cluster 2022-04-22T22:49:04.154932+0000 mgr.y (mgr.34107) 11010 : cluster [DBG] pgmap v11629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:04 smithi115 bash[46934]: cluster 2022-04-22T22:49:04.154932+0000 mgr.y (mgr.34107) 11010 : cluster [DBG] pgmap v11629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:04.747Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:04 smithi121 bash[36902]: cluster 2022-04-22T22:49:04.154932+0000 mgr.y (mgr.34107) 11010 : cluster [DBG] pgmap v11629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:06.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:05 smithi121 bash[36902]: audit 2022-04-22T22:49:05.580082+0000 mon.a (mon.0) 15301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:05 smithi115 bash[45495]: audit 2022-04-22T22:49:05.580082+0000 mon.a (mon.0) 15301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:05 smithi115 bash[46934]: audit 2022-04-22T22:49:05.580082+0000 mon.a (mon.0) 15301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:06 smithi121 bash[36902]: cluster 2022-04-22T22:49:06.156042+0000 mgr.y (mgr.34107) 11011 : cluster [DBG] pgmap v11630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:06 smithi115 bash[45495]: cluster 2022-04-22T22:49:06.156042+0000 mgr.y (mgr.34107) 11011 : cluster [DBG] pgmap v11630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:06 smithi115 bash[46934]: cluster 2022-04-22T22:49:06.156042+0000 mgr.y (mgr.34107) 11011 : cluster [DBG] pgmap v11630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:07 smithi121 bash[36902]: audit 2022-04-22T22:49:07.197882+0000 mon.a (mon.0) 15302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:07 smithi115 bash[45495]: audit 2022-04-22T22:49:07.197882+0000 mon.a (mon.0) 15302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:07 smithi115 bash[46934]: audit 2022-04-22T22:49:07.197882+0000 mon.a (mon.0) 15302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:09 smithi121 bash[36902]: cluster 2022-04-22T22:49:08.157196+0000 mgr.y (mgr.34107) 11012 : cluster [DBG] pgmap v11631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:09 smithi121 bash[36902]: audit 2022-04-22T22:49:08.674652+0000 mon.a (mon.0) 15303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:09.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:09 smithi115 bash[46934]: cluster 2022-04-22T22:49:08.157196+0000 mgr.y (mgr.34107) 11012 : cluster [DBG] pgmap v11631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:09 smithi115 bash[46934]: audit 2022-04-22T22:49:08.674652+0000 mon.a (mon.0) 15303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:09 smithi115 bash[45495]: cluster 2022-04-22T22:49:08.157196+0000 mgr.y (mgr.34107) 11012 : cluster [DBG] pgmap v11631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:09 smithi115 bash[45495]: audit 2022-04-22T22:49:08.674652+0000 mon.a (mon.0) 15303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:11 smithi121 bash[36902]: cluster 2022-04-22T22:49:10.158167+0000 mgr.y (mgr.34107) 11013 : cluster [DBG] pgmap v11632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:11 smithi121 bash[36902]: audit 2022-04-22T22:49:10.583036+0000 mon.a (mon.0) 15304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:11.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:11 smithi115 bash[45495]: cluster 2022-04-22T22:49:10.158167+0000 mgr.y (mgr.34107) 11013 : cluster [DBG] pgmap v11632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:11 smithi115 bash[45495]: audit 2022-04-22T22:49:10.583036+0000 mon.a (mon.0) 15304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:11 smithi115 bash[46934]: cluster 2022-04-22T22:49:10.158167+0000 mgr.y (mgr.34107) 11013 : cluster [DBG] pgmap v11632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:11 smithi115 bash[46934]: audit 2022-04-22T22:49:10.583036+0000 mon.a (mon.0) 15304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:12 smithi121 bash[36902]: cluster 2022-04-22T22:49:12.159352+0000 mgr.y (mgr.34107) 11014 : cluster [DBG] pgmap v11633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:12 smithi121 bash[36902]: audit 2022-04-22T22:49:12.200915+0000 mon.a (mon.0) 15305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:12.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:12 smithi115 bash[45495]: cluster 2022-04-22T22:49:12.159352+0000 mgr.y (mgr.34107) 11014 : cluster [DBG] pgmap v11633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:12 smithi115 bash[45495]: audit 2022-04-22T22:49:12.200915+0000 mon.a (mon.0) 15305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:12 smithi115 bash[46934]: cluster 2022-04-22T22:49:12.159352+0000 mgr.y (mgr.34107) 11014 : cluster [DBG] pgmap v11633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:12 smithi115 bash[46934]: audit 2022-04-22T22:49:12.200915+0000 mon.a (mon.0) 15305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:13 smithi121 bash[36902]: audit 2022-04-22T22:49:13.678252+0000 mon.a (mon.0) 15306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:13 smithi115 bash[45495]: audit 2022-04-22T22:49:13.678252+0000 mon.a (mon.0) 15306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:13 smithi115 bash[46934]: audit 2022-04-22T22:49:13.678252+0000 mon.a (mon.0) 15306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:14 smithi115 bash[45495]: cluster 2022-04-22T22:49:14.160415+0000 mgr.y (mgr.34107) 11015 : cluster [DBG] pgmap v11634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:15.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:14.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:15.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:14 smithi115 bash[46934]: cluster 2022-04-22T22:49:14.160415+0000 mgr.y (mgr.34107) 11015 : cluster [DBG] pgmap v11634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:14 smithi121 bash[36902]: cluster 2022-04-22T22:49:14.160415+0000 mgr.y (mgr.34107) 11015 : cluster [DBG] pgmap v11634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:15 smithi121 bash[36902]: audit 2022-04-22T22:49:15.585995+0000 mon.a (mon.0) 15307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:15 smithi115 bash[45495]: audit 2022-04-22T22:49:15.585995+0000 mon.a (mon.0) 15307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:15 smithi115 bash[46934]: audit 2022-04-22T22:49:15.585995+0000 mon.a (mon.0) 15307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:16 smithi121 bash[36902]: cluster 2022-04-22T22:49:16.161519+0000 mgr.y (mgr.34107) 11016 : cluster [DBG] pgmap v11635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:16 smithi115 bash[45495]: cluster 2022-04-22T22:49:16.161519+0000 mgr.y (mgr.34107) 11016 : cluster [DBG] pgmap v11635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:16 smithi115 bash[46934]: cluster 2022-04-22T22:49:16.161519+0000 mgr.y (mgr.34107) 11016 : cluster [DBG] pgmap v11635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:17 smithi121 bash[36902]: audit 2022-04-22T22:49:16.990756+0000 mon.a (mon.0) 15308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:49:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:17 smithi121 bash[36902]: audit 2022-04-22T22:49:16.991258+0000 mon.a (mon.0) 15309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:49:18.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:17 smithi121 bash[36902]: audit 2022-04-22T22:49:17.204202+0000 mon.a (mon.0) 15310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[45495]: audit 2022-04-22T22:49:16.990756+0000 mon.a (mon.0) 15308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:49:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[45495]: audit 2022-04-22T22:49:16.991258+0000 mon.a (mon.0) 15309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:49:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[45495]: audit 2022-04-22T22:49:17.204202+0000 mon.a (mon.0) 15310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[46934]: audit 2022-04-22T22:49:16.990756+0000 mon.a (mon.0) 15308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:49:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[46934]: audit 2022-04-22T22:49:16.991258+0000 mon.a (mon.0) 15309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:49:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:17 smithi115 bash[46934]: audit 2022-04-22T22:49:17.204202+0000 mon.a (mon.0) 15310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:19 smithi121 bash[36902]: cluster 2022-04-22T22:49:18.162659+0000 mgr.y (mgr.34107) 11017 : cluster [DBG] pgmap v11636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:19 smithi121 bash[36902]: audit 2022-04-22T22:49:18.681740+0000 mon.a (mon.0) 15311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:19.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:19 smithi115 bash[46934]: cluster 2022-04-22T22:49:18.162659+0000 mgr.y (mgr.34107) 11017 : cluster [DBG] pgmap v11636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:19.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:19 smithi115 bash[46934]: audit 2022-04-22T22:49:18.681740+0000 mon.a (mon.0) 15311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:19 smithi115 bash[45495]: cluster 2022-04-22T22:49:18.162659+0000 mgr.y (mgr.34107) 11017 : cluster [DBG] pgmap v11636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:19 smithi115 bash[45495]: audit 2022-04-22T22:49:18.681740+0000 mon.a (mon.0) 15311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:19.542 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:49:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:49:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:49:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:49:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:49:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:49:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:49:19.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:49:19.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:49:19.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:49:19.887 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:49:19.887 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:49:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:20 smithi121 bash[36902]: audit 2022-04-22T22:49:19.523887+0000 mgr.y (mgr.34107) 11018 : audit [DBG] from='client.46758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:20.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:20 smithi121 bash[36902]: audit 2022-04-22T22:49:19.862089+0000 mgr.y (mgr.34107) 11019 : audit [DBG] from='client.56701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:20.283 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:49:20.283 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:20.284 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:49:20.285 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:49:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:20 smithi115 bash[45495]: audit 2022-04-22T22:49:19.523887+0000 mgr.y (mgr.34107) 11018 : audit [DBG] from='client.46758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:20 smithi115 bash[45495]: audit 2022-04-22T22:49:19.862089+0000 mgr.y (mgr.34107) 11019 : audit [DBG] from='client.56701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:20.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:20 smithi115 bash[46934]: audit 2022-04-22T22:49:19.523887+0000 mgr.y (mgr.34107) 11018 : audit [DBG] from='client.46758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:20 smithi115 bash[46934]: audit 2022-04-22T22:49:19.862089+0000 mgr.y (mgr.34107) 11019 : audit [DBG] from='client.56701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:21.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[46934]: cluster 2022-04-22T22:49:20.163611+0000 mgr.y (mgr.34107) 11020 : cluster [DBG] pgmap v11637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[46934]: audit 2022-04-22T22:49:20.280128+0000 mon.c (mon.1) 371 : audit [DBG] from='client.? 172.21.15.115:0/3497112837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[46934]: audit 2022-04-22T22:49:20.588861+0000 mon.a (mon.0) 15312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[45495]: cluster 2022-04-22T22:49:20.163611+0000 mgr.y (mgr.34107) 11020 : cluster [DBG] pgmap v11637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[45495]: audit 2022-04-22T22:49:20.280128+0000 mon.c (mon.1) 371 : audit [DBG] from='client.? 172.21.15.115:0/3497112837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:21 smithi115 bash[45495]: audit 2022-04-22T22:49:20.588861+0000 mon.a (mon.0) 15312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:21 smithi121 bash[36902]: cluster 2022-04-22T22:49:20.163611+0000 mgr.y (mgr.34107) 11020 : cluster [DBG] pgmap v11637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:21 smithi121 bash[36902]: audit 2022-04-22T22:49:20.280128+0000 mon.c (mon.1) 371 : audit [DBG] from='client.? 172.21.15.115:0/3497112837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:21 smithi121 bash[36902]: audit 2022-04-22T22:49:20.588861+0000 mon.a (mon.0) 15312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:22 smithi121 bash[36902]: cluster 2022-04-22T22:49:22.164723+0000 mgr.y (mgr.34107) 11021 : cluster [DBG] pgmap v11638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:23.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:22 smithi121 bash[36902]: audit 2022-04-22T22:49:22.207206+0000 mon.a (mon.0) 15313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:22.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:22 smithi115 bash[46934]: cluster 2022-04-22T22:49:22.164723+0000 mgr.y (mgr.34107) 11021 : cluster [DBG] pgmap v11638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:22 smithi115 bash[46934]: audit 2022-04-22T22:49:22.207206+0000 mon.a (mon.0) 15313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:23.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:22 smithi115 bash[45495]: cluster 2022-04-22T22:49:22.164723+0000 mgr.y (mgr.34107) 11021 : cluster [DBG] pgmap v11638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:23.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:22 smithi115 bash[45495]: audit 2022-04-22T22:49:22.207206+0000 mon.a (mon.0) 15313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:24.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:23 smithi121 bash[36902]: audit 2022-04-22T22:49:23.685510+0000 mon.a (mon.0) 15314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:24.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:23 smithi115 bash[46934]: audit 2022-04-22T22:49:23.685510+0000 mon.a (mon.0) 15314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:23 smithi115 bash[45495]: audit 2022-04-22T22:49:23.685510+0000 mon.a (mon.0) 15314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:24 smithi115 bash[45495]: cluster 2022-04-22T22:49:24.165821+0000 mgr.y (mgr.34107) 11022 : cluster [DBG] pgmap v11639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:24.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:24 smithi115 bash[46934]: cluster 2022-04-22T22:49:24.165821+0000 mgr.y (mgr.34107) 11022 : cluster [DBG] pgmap v11639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:24 smithi121 bash[36902]: cluster 2022-04-22T22:49:24.165821+0000 mgr.y (mgr.34107) 11022 : cluster [DBG] pgmap v11639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:25 smithi121 bash[36902]: audit 2022-04-22T22:49:25.592117+0000 mon.a (mon.0) 15315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:25 smithi115 bash[45495]: audit 2022-04-22T22:49:25.592117+0000 mon.a (mon.0) 15315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:25 smithi115 bash[46934]: audit 2022-04-22T22:49:25.592117+0000 mon.a (mon.0) 15315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:26 smithi121 bash[36902]: cluster 2022-04-22T22:49:26.166846+0000 mgr.y (mgr.34107) 11023 : cluster [DBG] pgmap v11640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:26 smithi115 bash[45495]: cluster 2022-04-22T22:49:26.166846+0000 mgr.y (mgr.34107) 11023 : cluster [DBG] pgmap v11640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:26 smithi115 bash[46934]: cluster 2022-04-22T22:49:26.166846+0000 mgr.y (mgr.34107) 11023 : cluster [DBG] pgmap v11640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:28.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:28 smithi121 bash[36902]: audit 2022-04-22T22:49:27.210066+0000 mon.a (mon.0) 15316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:27 smithi115 bash[45495]: audit 2022-04-22T22:49:27.210066+0000 mon.a (mon.0) 15316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:28 smithi115 bash[46934]: audit 2022-04-22T22:49:27.210066+0000 mon.a (mon.0) 15316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:29 smithi121 bash[36902]: cluster 2022-04-22T22:49:28.168069+0000 mgr.y (mgr.34107) 11024 : cluster [DBG] pgmap v11641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:29 smithi121 bash[36902]: audit 2022-04-22T22:49:28.689244+0000 mon.a (mon.0) 15317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:29 smithi115 bash[45495]: cluster 2022-04-22T22:49:28.168069+0000 mgr.y (mgr.34107) 11024 : cluster [DBG] pgmap v11641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:29 smithi115 bash[45495]: audit 2022-04-22T22:49:28.689244+0000 mon.a (mon.0) 15317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:29 smithi115 bash[46934]: cluster 2022-04-22T22:49:28.168069+0000 mgr.y (mgr.34107) 11024 : cluster [DBG] pgmap v11641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:29 smithi115 bash[46934]: audit 2022-04-22T22:49:28.689244+0000 mon.a (mon.0) 15317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:31 smithi121 bash[36902]: cluster 2022-04-22T22:49:30.169093+0000 mgr.y (mgr.34107) 11025 : cluster [DBG] pgmap v11642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:31 smithi121 bash[36902]: audit 2022-04-22T22:49:30.595097+0000 mon.a (mon.0) 15318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:31 smithi115 bash[45495]: cluster 2022-04-22T22:49:30.169093+0000 mgr.y (mgr.34107) 11025 : cluster [DBG] pgmap v11642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:31 smithi115 bash[45495]: audit 2022-04-22T22:49:30.595097+0000 mon.a (mon.0) 15318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:31 smithi115 bash[46934]: cluster 2022-04-22T22:49:30.169093+0000 mgr.y (mgr.34107) 11025 : cluster [DBG] pgmap v11642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:31 smithi115 bash[46934]: audit 2022-04-22T22:49:30.595097+0000 mon.a (mon.0) 15318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:33.180 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:32.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:32 smithi121 bash[36902]: cluster 2022-04-22T22:49:32.170334+0000 mgr.y (mgr.34107) 11026 : cluster [DBG] pgmap v11643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:32 smithi121 bash[36902]: audit 2022-04-22T22:49:32.213217+0000 mon.a (mon.0) 15319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[45495]: cluster 2022-04-22T22:49:32.170334+0000 mgr.y (mgr.34107) 11026 : cluster [DBG] pgmap v11643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[45495]: audit 2022-04-22T22:49:32.213217+0000 mon.a (mon.0) 15319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[46934]: cluster 2022-04-22T22:49:32.170334+0000 mgr.y (mgr.34107) 11026 : cluster [DBG] pgmap v11643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[46934]: audit 2022-04-22T22:49:32.213217+0000 mon.a (mon.0) 15319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:33 smithi121 bash[36902]: audit 2022-04-22T22:49:33.692808+0000 mon.a (mon.0) 15320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[45495]: audit 2022-04-22T22:49:33.692808+0000 mon.a (mon.0) 15320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:33 smithi115 bash[46934]: audit 2022-04-22T22:49:33.692808+0000 mon.a (mon.0) 15320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:34 smithi115 bash[45495]: cluster 2022-04-22T22:49:34.171452+0000 mgr.y (mgr.34107) 11027 : cluster [DBG] pgmap v11644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:34.749Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:34 smithi115 bash[46934]: cluster 2022-04-22T22:49:34.171452+0000 mgr.y (mgr.34107) 11027 : cluster [DBG] pgmap v11644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:34 smithi121 bash[36902]: cluster 2022-04-22T22:49:34.171452+0000 mgr.y (mgr.34107) 11027 : cluster [DBG] pgmap v11644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:35 smithi121 bash[36902]: audit 2022-04-22T22:49:35.598164+0000 mon.a (mon.0) 15321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:35 smithi115 bash[45495]: audit 2022-04-22T22:49:35.598164+0000 mon.a (mon.0) 15321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:35 smithi115 bash[46934]: audit 2022-04-22T22:49:35.598164+0000 mon.a (mon.0) 15321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:37 smithi121 bash[36902]: cluster 2022-04-22T22:49:36.172508+0000 mgr.y (mgr.34107) 11028 : cluster [DBG] pgmap v11645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:37 smithi115 bash[45495]: cluster 2022-04-22T22:49:36.172508+0000 mgr.y (mgr.34107) 11028 : cluster [DBG] pgmap v11645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:37 smithi115 bash[46934]: cluster 2022-04-22T22:49:36.172508+0000 mgr.y (mgr.34107) 11028 : cluster [DBG] pgmap v11645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:38 smithi121 bash[36902]: audit 2022-04-22T22:49:37.216157+0000 mon.a (mon.0) 15322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:38 smithi115 bash[45495]: audit 2022-04-22T22:49:37.216157+0000 mon.a (mon.0) 15322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:38.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:38 smithi115 bash[46934]: audit 2022-04-22T22:49:37.216157+0000 mon.a (mon.0) 15322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:39 smithi115 bash[45495]: cluster 2022-04-22T22:49:38.173785+0000 mgr.y (mgr.34107) 11029 : cluster [DBG] pgmap v11646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:39 smithi115 bash[45495]: audit 2022-04-22T22:49:38.696312+0000 mon.a (mon.0) 15323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:39 smithi115 bash[46934]: cluster 2022-04-22T22:49:38.173785+0000 mgr.y (mgr.34107) 11029 : cluster [DBG] pgmap v11646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:39 smithi115 bash[46934]: audit 2022-04-22T22:49:38.696312+0000 mon.a (mon.0) 15323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:39 smithi121 bash[36902]: cluster 2022-04-22T22:49:38.173785+0000 mgr.y (mgr.34107) 11029 : cluster [DBG] pgmap v11646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:39 smithi121 bash[36902]: audit 2022-04-22T22:49:38.696312+0000 mon.a (mon.0) 15323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:41 smithi121 bash[36902]: cluster 2022-04-22T22:49:40.174931+0000 mgr.y (mgr.34107) 11030 : cluster [DBG] pgmap v11647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:41 smithi121 bash[36902]: audit 2022-04-22T22:49:40.601380+0000 mon.a (mon.0) 15324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:41 smithi115 bash[45495]: cluster 2022-04-22T22:49:40.174931+0000 mgr.y (mgr.34107) 11030 : cluster [DBG] pgmap v11647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:41.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:41 smithi115 bash[45495]: audit 2022-04-22T22:49:40.601380+0000 mon.a (mon.0) 15324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:41 smithi115 bash[46934]: cluster 2022-04-22T22:49:40.174931+0000 mgr.y (mgr.34107) 11030 : cluster [DBG] pgmap v11647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:41 smithi115 bash[46934]: audit 2022-04-22T22:49:40.601380+0000 mon.a (mon.0) 15324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:42 smithi121 bash[36902]: cluster 2022-04-22T22:49:42.175997+0000 mgr.y (mgr.34107) 11031 : cluster [DBG] pgmap v11648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:42 smithi121 bash[36902]: audit 2022-04-22T22:49:42.219277+0000 mon.a (mon.0) 15325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:42.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:42 smithi115 bash[45495]: cluster 2022-04-22T22:49:42.175997+0000 mgr.y (mgr.34107) 11031 : cluster [DBG] pgmap v11648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:42 smithi115 bash[45495]: audit 2022-04-22T22:49:42.219277+0000 mon.a (mon.0) 15325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:42 smithi115 bash[46934]: cluster 2022-04-22T22:49:42.175997+0000 mgr.y (mgr.34107) 11031 : cluster [DBG] pgmap v11648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:42 smithi115 bash[46934]: audit 2022-04-22T22:49:42.219277+0000 mon.a (mon.0) 15325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:43 smithi121 bash[36902]: audit 2022-04-22T22:49:43.699973+0000 mon.a (mon.0) 15326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:43 smithi115 bash[45495]: audit 2022-04-22T22:49:43.699973+0000 mon.a (mon.0) 15326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:43 smithi115 bash[46934]: audit 2022-04-22T22:49:43.699973+0000 mon.a (mon.0) 15326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:44 smithi115 bash[45495]: cluster 2022-04-22T22:49:44.177094+0000 mgr.y (mgr.34107) 11032 : cluster [DBG] pgmap v11649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:44.751Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:44 smithi115 bash[46934]: cluster 2022-04-22T22:49:44.177094+0000 mgr.y (mgr.34107) 11032 : cluster [DBG] pgmap v11649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:44 smithi121 bash[36902]: cluster 2022-04-22T22:49:44.177094+0000 mgr.y (mgr.34107) 11032 : cluster [DBG] pgmap v11649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:45 smithi121 bash[36902]: audit 2022-04-22T22:49:45.604607+0000 mon.a (mon.0) 15327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:46.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:46 smithi115 bash[46934]: audit 2022-04-22T22:49:45.604607+0000 mon.a (mon.0) 15327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:45 smithi115 bash[45495]: audit 2022-04-22T22:49:45.604607+0000 mon.a (mon.0) 15327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:47 smithi121 bash[36902]: cluster 2022-04-22T22:49:46.178165+0000 mgr.y (mgr.34107) 11033 : cluster [DBG] pgmap v11650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:47 smithi115 bash[45495]: cluster 2022-04-22T22:49:46.178165+0000 mgr.y (mgr.34107) 11033 : cluster [DBG] pgmap v11650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:47 smithi115 bash[46934]: cluster 2022-04-22T22:49:46.178165+0000 mgr.y (mgr.34107) 11033 : cluster [DBG] pgmap v11650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:48 smithi121 bash[36902]: audit 2022-04-22T22:49:47.222415+0000 mon.a (mon.0) 15328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:48 smithi115 bash[45495]: audit 2022-04-22T22:49:47.222415+0000 mon.a (mon.0) 15328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:48 smithi115 bash[46934]: audit 2022-04-22T22:49:47.222415+0000 mon.a (mon.0) 15328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:49 smithi115 bash[45495]: cluster 2022-04-22T22:49:48.179182+0000 mgr.y (mgr.34107) 11034 : cluster [DBG] pgmap v11651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:49 smithi115 bash[45495]: audit 2022-04-22T22:49:48.703774+0000 mon.a (mon.0) 15329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:49 smithi115 bash[46934]: cluster 2022-04-22T22:49:48.179182+0000 mgr.y (mgr.34107) 11034 : cluster [DBG] pgmap v11651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:49 smithi115 bash[46934]: audit 2022-04-22T22:49:48.703774+0000 mon.a (mon.0) 15329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:49 smithi121 bash[36902]: cluster 2022-04-22T22:49:48.179182+0000 mgr.y (mgr.34107) 11034 : cluster [DBG] pgmap v11651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:49 smithi121 bash[36902]: audit 2022-04-22T22:49:48.703774+0000 mon.a (mon.0) 15329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:50.658 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:49:51.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:49:51.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:49:51.003 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:49:51.411 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:49:51.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:49:51.411 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:49:51.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:51.411 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:49:51.412 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:49:51.413 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:49:51.413 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:49:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:51 smithi121 bash[36902]: cluster 2022-04-22T22:49:50.180303+0000 mgr.y (mgr.34107) 11035 : cluster [DBG] pgmap v11652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:51 smithi121 bash[36902]: audit 2022-04-22T22:49:50.607613+0000 mon.a (mon.0) 15330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:51.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:51 smithi121 bash[36902]: audit 2022-04-22T22:49:50.639046+0000 mgr.y (mgr.34107) 11036 : audit [DBG] from='client.46776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[45495]: cluster 2022-04-22T22:49:50.180303+0000 mgr.y (mgr.34107) 11035 : cluster [DBG] pgmap v11652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[45495]: audit 2022-04-22T22:49:50.607613+0000 mon.a (mon.0) 15330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[45495]: audit 2022-04-22T22:49:50.639046+0000 mgr.y (mgr.34107) 11036 : audit [DBG] from='client.46776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[46934]: cluster 2022-04-22T22:49:50.180303+0000 mgr.y (mgr.34107) 11035 : cluster [DBG] pgmap v11652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[46934]: audit 2022-04-22T22:49:50.607613+0000 mon.a (mon.0) 15330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:51 smithi115 bash[46934]: audit 2022-04-22T22:49:50.639046+0000 mgr.y (mgr.34107) 11036 : audit [DBG] from='client.46776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:52 smithi121 bash[36902]: audit 2022-04-22T22:49:50.978117+0000 mgr.y (mgr.34107) 11037 : audit [DBG] from='client.56719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:52 smithi121 bash[36902]: audit 2022-04-22T22:49:51.407655+0000 mon.c (mon.1) 372 : audit [DBG] from='client.? 172.21.15.115:0/2892617556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:52 smithi121 bash[36902]: audit 2022-04-22T22:49:52.225817+0000 mon.a (mon.0) 15331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[45495]: audit 2022-04-22T22:49:50.978117+0000 mgr.y (mgr.34107) 11037 : audit [DBG] from='client.56719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[45495]: audit 2022-04-22T22:49:51.407655+0000 mon.c (mon.1) 372 : audit [DBG] from='client.? 172.21.15.115:0/2892617556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[45495]: audit 2022-04-22T22:49:52.225817+0000 mon.a (mon.0) 15331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[46934]: audit 2022-04-22T22:49:50.978117+0000 mgr.y (mgr.34107) 11037 : audit [DBG] from='client.56719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[46934]: audit 2022-04-22T22:49:51.407655+0000 mon.c (mon.1) 372 : audit [DBG] from='client.? 172.21.15.115:0/2892617556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:49:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[46934]: audit 2022-04-22T22:49:52.225817+0000 mon.a (mon.0) 15331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:53.243 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:52.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:53 smithi121 bash[36902]: cluster 2022-04-22T22:49:52.181386+0000 mgr.y (mgr.34107) 11038 : cluster [DBG] pgmap v11653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:53.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:53 smithi115 bash[45495]: cluster 2022-04-22T22:49:52.181386+0000 mgr.y (mgr.34107) 11038 : cluster [DBG] pgmap v11653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:53 smithi115 bash[46934]: cluster 2022-04-22T22:49:52.181386+0000 mgr.y (mgr.34107) 11038 : cluster [DBG] pgmap v11653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:54 smithi121 bash[36902]: audit 2022-04-22T22:49:53.707305+0000 mon.a (mon.0) 15332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:54 smithi115 bash[45495]: audit 2022-04-22T22:49:53.707305+0000 mon.a (mon.0) 15332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:54 smithi115 bash[46934]: audit 2022-04-22T22:49:53.707305+0000 mon.a (mon.0) 15332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:49:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:49:54.750Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:49:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:55 smithi121 bash[36902]: cluster 2022-04-22T22:49:54.182478+0000 mgr.y (mgr.34107) 11039 : cluster [DBG] pgmap v11654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:55 smithi115 bash[45495]: cluster 2022-04-22T22:49:54.182478+0000 mgr.y (mgr.34107) 11039 : cluster [DBG] pgmap v11654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:55 smithi115 bash[46934]: cluster 2022-04-22T22:49:54.182478+0000 mgr.y (mgr.34107) 11039 : cluster [DBG] pgmap v11654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:56 smithi121 bash[36902]: audit 2022-04-22T22:49:55.610438+0000 mon.a (mon.0) 15333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:56 smithi115 bash[46934]: audit 2022-04-22T22:49:55.610438+0000 mon.a (mon.0) 15333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:56 smithi115 bash[45495]: audit 2022-04-22T22:49:55.610438+0000 mon.a (mon.0) 15333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:49:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:57 smithi115 bash[45495]: cluster 2022-04-22T22:49:56.183596+0000 mgr.y (mgr.34107) 11040 : cluster [DBG] pgmap v11655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:57 smithi115 bash[45495]: audit 2022-04-22T22:49:57.228779+0000 mon.a (mon.0) 15334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:57 smithi115 bash[46934]: cluster 2022-04-22T22:49:56.183596+0000 mgr.y (mgr.34107) 11040 : cluster [DBG] pgmap v11655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:57 smithi115 bash[46934]: audit 2022-04-22T22:49:57.228779+0000 mon.a (mon.0) 15334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:57 smithi121 bash[36902]: cluster 2022-04-22T22:49:56.183596+0000 mgr.y (mgr.34107) 11040 : cluster [DBG] pgmap v11655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:57 smithi121 bash[36902]: audit 2022-04-22T22:49:57.228779+0000 mon.a (mon.0) 15334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:49:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:59 smithi115 bash[46934]: cluster 2022-04-22T22:49:58.184601+0000 mgr.y (mgr.34107) 11041 : cluster [DBG] pgmap v11656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:49:59 smithi115 bash[46934]: audit 2022-04-22T22:49:58.711010+0000 mon.a (mon.0) 15335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:59 smithi115 bash[45495]: cluster 2022-04-22T22:49:58.184601+0000 mgr.y (mgr.34107) 11041 : cluster [DBG] pgmap v11656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:49:59 smithi115 bash[45495]: audit 2022-04-22T22:49:58.711010+0000 mon.a (mon.0) 15335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:49:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:59 smithi121 bash[36902]: cluster 2022-04-22T22:49:58.184601+0000 mgr.y (mgr.34107) 11041 : cluster [DBG] pgmap v11656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:49:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:49:59 smithi121 bash[36902]: audit 2022-04-22T22:49:58.711010+0000 mon.a (mon.0) 15335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:00 smithi115 bash[45495]: cluster 2022-04-22T22:50:00.000161+0000 mon.a (mon.0) 15336 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:50:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:00 smithi115 bash[46934]: cluster 2022-04-22T22:50:00.000161+0000 mon.a (mon.0) 15336 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:50:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:00 smithi121 bash[36902]: cluster 2022-04-22T22:50:00.000161+0000 mon.a (mon.0) 15336 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T22:50:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:01 smithi121 bash[36902]: cluster 2022-04-22T22:50:00.185613+0000 mgr.y (mgr.34107) 11042 : cluster [DBG] pgmap v11657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:01 smithi121 bash[36902]: audit 2022-04-22T22:50:00.613392+0000 mon.a (mon.0) 15337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:01 smithi121 bash[36902]: audit 2022-04-22T22:50:01.210196+0000 mon.a (mon.0) 15338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:50:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:01 smithi121 bash[36902]: audit 2022-04-22T22:50:01.213187+0000 mon.a (mon.0) 15339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:50:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:01 smithi121 bash[36902]: audit 2022-04-22T22:50:01.214926+0000 mon.a (mon.0) 15340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[46934]: cluster 2022-04-22T22:50:00.185613+0000 mgr.y (mgr.34107) 11042 : cluster [DBG] pgmap v11657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[46934]: audit 2022-04-22T22:50:00.613392+0000 mon.a (mon.0) 15337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[46934]: audit 2022-04-22T22:50:01.210196+0000 mon.a (mon.0) 15338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[46934]: audit 2022-04-22T22:50:01.213187+0000 mon.a (mon.0) 15339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[46934]: audit 2022-04-22T22:50:01.214926+0000 mon.a (mon.0) 15340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[45495]: cluster 2022-04-22T22:50:00.185613+0000 mgr.y (mgr.34107) 11042 : cluster [DBG] pgmap v11657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[45495]: audit 2022-04-22T22:50:00.613392+0000 mon.a (mon.0) 15337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[45495]: audit 2022-04-22T22:50:01.210196+0000 mon.a (mon.0) 15338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:50:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[45495]: audit 2022-04-22T22:50:01.213187+0000 mon.a (mon.0) 15339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:50:01.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:01 smithi115 bash[45495]: audit 2022-04-22T22:50:01.214926+0000 mon.a (mon.0) 15340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:50:02.893 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:02 smithi115 bash[45495]: audit 2022-04-22T22:50:01.576288+0000 mon.a (mon.0) 15341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:50:02.893 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:02 smithi115 bash[45495]: audit 2022-04-22T22:50:02.232018+0000 mon.a (mon.0) 15342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:02.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:02 smithi115 bash[46934]: audit 2022-04-22T22:50:01.576288+0000 mon.a (mon.0) 15341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:50:02.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:02 smithi115 bash[46934]: audit 2022-04-22T22:50:02.232018+0000 mon.a (mon.0) 15342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:02 smithi121 bash[36902]: audit 2022-04-22T22:50:01.576288+0000 mon.a (mon.0) 15341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:50:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:02 smithi121 bash[36902]: audit 2022-04-22T22:50:02.232018+0000 mon.a (mon.0) 15342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:02.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:04.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:03 smithi121 bash[36902]: cluster 2022-04-22T22:50:02.186713+0000 mgr.y (mgr.34107) 11043 : cluster [DBG] pgmap v11658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:03 smithi115 bash[46934]: cluster 2022-04-22T22:50:02.186713+0000 mgr.y (mgr.34107) 11043 : cluster [DBG] pgmap v11658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:04.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:03 smithi115 bash[45495]: cluster 2022-04-22T22:50:02.186713+0000 mgr.y (mgr.34107) 11043 : cluster [DBG] pgmap v11658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:04 smithi121 bash[36902]: audit 2022-04-22T22:50:03.714787+0000 mon.a (mon.0) 15343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:04 smithi115 bash[45495]: audit 2022-04-22T22:50:03.714787+0000 mon.a (mon.0) 15343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:04.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:04 smithi115 bash[46934]: audit 2022-04-22T22:50:03.714787+0000 mon.a (mon.0) 15343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:05 smithi121 bash[36902]: cluster 2022-04-22T22:50:04.187853+0000 mgr.y (mgr.34107) 11044 : cluster [DBG] pgmap v11659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:05 smithi115 bash[45495]: cluster 2022-04-22T22:50:04.187853+0000 mgr.y (mgr.34107) 11044 : cluster [DBG] pgmap v11659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:06.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:05 smithi115 bash[46934]: cluster 2022-04-22T22:50:04.187853+0000 mgr.y (mgr.34107) 11044 : cluster [DBG] pgmap v11659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:06 smithi121 bash[36902]: audit 2022-04-22T22:50:05.616580+0000 mon.a (mon.0) 15344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:06 smithi115 bash[45495]: audit 2022-04-22T22:50:05.616580+0000 mon.a (mon.0) 15344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:06 smithi115 bash[46934]: audit 2022-04-22T22:50:05.616580+0000 mon.a (mon.0) 15344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:07 smithi121 bash[36902]: cluster 2022-04-22T22:50:06.188947+0000 mgr.y (mgr.34107) 11045 : cluster [DBG] pgmap v11660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:07 smithi121 bash[36902]: audit 2022-04-22T22:50:07.235043+0000 mon.a (mon.0) 15345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:07 smithi115 bash[45495]: cluster 2022-04-22T22:50:06.188947+0000 mgr.y (mgr.34107) 11045 : cluster [DBG] pgmap v11660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:07 smithi115 bash[45495]: audit 2022-04-22T22:50:07.235043+0000 mon.a (mon.0) 15345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:07 smithi115 bash[46934]: cluster 2022-04-22T22:50:06.188947+0000 mgr.y (mgr.34107) 11045 : cluster [DBG] pgmap v11660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:07 smithi115 bash[46934]: audit 2022-04-22T22:50:07.235043+0000 mon.a (mon.0) 15345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:09 smithi121 bash[36902]: cluster 2022-04-22T22:50:08.190156+0000 mgr.y (mgr.34107) 11046 : cluster [DBG] pgmap v11661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:09 smithi121 bash[36902]: audit 2022-04-22T22:50:08.718460+0000 mon.a (mon.0) 15346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:09 smithi115 bash[45495]: cluster 2022-04-22T22:50:08.190156+0000 mgr.y (mgr.34107) 11046 : cluster [DBG] pgmap v11661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:09 smithi115 bash[45495]: audit 2022-04-22T22:50:08.718460+0000 mon.a (mon.0) 15346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:09 smithi115 bash[46934]: cluster 2022-04-22T22:50:08.190156+0000 mgr.y (mgr.34107) 11046 : cluster [DBG] pgmap v11661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:09 smithi115 bash[46934]: audit 2022-04-22T22:50:08.718460+0000 mon.a (mon.0) 15346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:11 smithi121 bash[36902]: cluster 2022-04-22T22:50:10.191379+0000 mgr.y (mgr.34107) 11047 : cluster [DBG] pgmap v11662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:11 smithi121 bash[36902]: audit 2022-04-22T22:50:10.619242+0000 mon.a (mon.0) 15347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:11 smithi115 bash[45495]: cluster 2022-04-22T22:50:10.191379+0000 mgr.y (mgr.34107) 11047 : cluster [DBG] pgmap v11662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:11 smithi115 bash[45495]: audit 2022-04-22T22:50:10.619242+0000 mon.a (mon.0) 15347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:11 smithi115 bash[46934]: cluster 2022-04-22T22:50:10.191379+0000 mgr.y (mgr.34107) 11047 : cluster [DBG] pgmap v11662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:11 smithi115 bash[46934]: audit 2022-04-22T22:50:10.619242+0000 mon.a (mon.0) 15347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:12 smithi121 bash[36902]: cluster 2022-04-22T22:50:12.192567+0000 mgr.y (mgr.34107) 11048 : cluster [DBG] pgmap v11663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:13.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:12 smithi121 bash[36902]: audit 2022-04-22T22:50:12.238128+0000 mon.a (mon.0) 15348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:12.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:12 smithi115 bash[45495]: cluster 2022-04-22T22:50:12.192567+0000 mgr.y (mgr.34107) 11048 : cluster [DBG] pgmap v11663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:13.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:12 smithi115 bash[45495]: audit 2022-04-22T22:50:12.238128+0000 mon.a (mon.0) 15348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:12 smithi115 bash[46934]: cluster 2022-04-22T22:50:12.192567+0000 mgr.y (mgr.34107) 11048 : cluster [DBG] pgmap v11663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:12 smithi115 bash[46934]: audit 2022-04-22T22:50:12.238128+0000 mon.a (mon.0) 15348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:13 smithi121 bash[36902]: audit 2022-04-22T22:50:13.722074+0000 mon.a (mon.0) 15349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:13 smithi115 bash[45495]: audit 2022-04-22T22:50:13.722074+0000 mon.a (mon.0) 15349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:13 smithi115 bash[46934]: audit 2022-04-22T22:50:13.722074+0000 mon.a (mon.0) 15349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:15.008 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:14.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:15.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:15 smithi121 bash[36902]: cluster 2022-04-22T22:50:14.193788+0000 mgr.y (mgr.34107) 11049 : cluster [DBG] pgmap v11664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:15 smithi115 bash[45495]: cluster 2022-04-22T22:50:14.193788+0000 mgr.y (mgr.34107) 11049 : cluster [DBG] pgmap v11664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:15 smithi115 bash[46934]: cluster 2022-04-22T22:50:14.193788+0000 mgr.y (mgr.34107) 11049 : cluster [DBG] pgmap v11664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:16 smithi121 bash[36902]: audit 2022-04-22T22:50:15.622045+0000 mon.a (mon.0) 15350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:16 smithi115 bash[45495]: audit 2022-04-22T22:50:15.622045+0000 mon.a (mon.0) 15350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:16 smithi115 bash[46934]: audit 2022-04-22T22:50:15.622045+0000 mon.a (mon.0) 15350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[45495]: cluster 2022-04-22T22:50:16.194926+0000 mgr.y (mgr.34107) 11050 : cluster [DBG] pgmap v11665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[45495]: audit 2022-04-22T22:50:16.997812+0000 mon.a (mon.0) 15351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:50:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[45495]: audit 2022-04-22T22:50:17.001252+0000 mon.a (mon.0) 15352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:50:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[46934]: cluster 2022-04-22T22:50:16.194926+0000 mgr.y (mgr.34107) 11050 : cluster [DBG] pgmap v11665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[46934]: audit 2022-04-22T22:50:16.997812+0000 mon.a (mon.0) 15351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:50:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:17 smithi115 bash[46934]: audit 2022-04-22T22:50:17.001252+0000 mon.a (mon.0) 15352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:50:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:17 smithi121 bash[36902]: cluster 2022-04-22T22:50:16.194926+0000 mgr.y (mgr.34107) 11050 : cluster [DBG] pgmap v11665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:17 smithi121 bash[36902]: audit 2022-04-22T22:50:16.997812+0000 mon.a (mon.0) 15351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:50:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:17 smithi121 bash[36902]: audit 2022-04-22T22:50:17.001252+0000 mon.a (mon.0) 15352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:50:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:18 smithi115 bash[45495]: audit 2022-04-22T22:50:17.241313+0000 mon.a (mon.0) 15353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:18 smithi115 bash[46934]: audit 2022-04-22T22:50:17.241313+0000 mon.a (mon.0) 15353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:18 smithi121 bash[36902]: audit 2022-04-22T22:50:17.241313+0000 mon.a (mon.0) 15353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:19 smithi115 bash[45495]: cluster 2022-04-22T22:50:18.196098+0000 mgr.y (mgr.34107) 11051 : cluster [DBG] pgmap v11666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:19 smithi115 bash[45495]: audit 2022-04-22T22:50:18.725655+0000 mon.a (mon.0) 15354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:19 smithi115 bash[46934]: cluster 2022-04-22T22:50:18.196098+0000 mgr.y (mgr.34107) 11051 : cluster [DBG] pgmap v11666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:19 smithi115 bash[46934]: audit 2022-04-22T22:50:18.725655+0000 mon.a (mon.0) 15354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:19 smithi121 bash[36902]: cluster 2022-04-22T22:50:18.196098+0000 mgr.y (mgr.34107) 11051 : cluster [DBG] pgmap v11666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:19 smithi121 bash[36902]: audit 2022-04-22T22:50:18.725655+0000 mon.a (mon.0) 15354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:21 smithi121 bash[36902]: cluster 2022-04-22T22:50:20.197220+0000 mgr.y (mgr.34107) 11052 : cluster [DBG] pgmap v11667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:21.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:21 smithi121 bash[36902]: audit 2022-04-22T22:50:20.625160+0000 mon.a (mon.0) 15355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:21 smithi115 bash[45495]: cluster 2022-04-22T22:50:20.197220+0000 mgr.y (mgr.34107) 11052 : cluster [DBG] pgmap v11667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:21 smithi115 bash[45495]: audit 2022-04-22T22:50:20.625160+0000 mon.a (mon.0) 15355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:21 smithi115 bash[46934]: cluster 2022-04-22T22:50:20.197220+0000 mgr.y (mgr.34107) 11052 : cluster [DBG] pgmap v11667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:21 smithi115 bash[46934]: audit 2022-04-22T22:50:20.625160+0000 mon.a (mon.0) 15355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:21.784 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:50:22.130 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:50:22.131 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:50:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:50:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:50:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:50:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:50:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:50:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:22 smithi121 bash[36902]: audit 2022-04-22T22:50:21.765634+0000 mgr.y (mgr.34107) 11053 : audit [DBG] from='client.56731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:22.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:22 smithi121 bash[36902]: audit 2022-04-22T22:50:22.244483+0000 mon.a (mon.0) 15356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:50:22.536 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:50:22.537 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:50:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:50:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:22 smithi115 bash[45495]: audit 2022-04-22T22:50:21.765634+0000 mgr.y (mgr.34107) 11053 : audit [DBG] from='client.56731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:22 smithi115 bash[45495]: audit 2022-04-22T22:50:22.244483+0000 mon.a (mon.0) 15356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:22.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:22 smithi115 bash[46934]: audit 2022-04-22T22:50:21.765634+0000 mgr.y (mgr.34107) 11053 : audit [DBG] from='client.56731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:22 smithi115 bash[46934]: audit 2022-04-22T22:50:22.244483+0000 mon.a (mon.0) 15356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:23.261 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:22.890Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:23 smithi121 bash[36902]: audit 2022-04-22T22:50:22.108578+0000 mgr.y (mgr.34107) 11054 : audit [DBG] from='client.56737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:23 smithi121 bash[36902]: cluster 2022-04-22T22:50:22.198077+0000 mgr.y (mgr.34107) 11055 : cluster [DBG] pgmap v11668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:23 smithi121 bash[36902]: audit 2022-04-22T22:50:22.532597+0000 mon.c (mon.1) 373 : audit [DBG] from='client.? 172.21.15.115:0/543915493' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[45495]: audit 2022-04-22T22:50:22.108578+0000 mgr.y (mgr.34107) 11054 : audit [DBG] from='client.56737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[45495]: cluster 2022-04-22T22:50:22.198077+0000 mgr.y (mgr.34107) 11055 : cluster [DBG] pgmap v11668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[45495]: audit 2022-04-22T22:50:22.532597+0000 mon.c (mon.1) 373 : audit [DBG] from='client.? 172.21.15.115:0/543915493' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[46934]: audit 2022-04-22T22:50:22.108578+0000 mgr.y (mgr.34107) 11054 : audit [DBG] from='client.56737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[46934]: cluster 2022-04-22T22:50:22.198077+0000 mgr.y (mgr.34107) 11055 : cluster [DBG] pgmap v11668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:23 smithi115 bash[46934]: audit 2022-04-22T22:50:22.532597+0000 mon.c (mon.1) 373 : audit [DBG] from='client.? 172.21.15.115:0/543915493' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:24 smithi115 bash[45495]: audit 2022-04-22T22:50:23.729361+0000 mon.a (mon.0) 15357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:24 smithi115 bash[46934]: audit 2022-04-22T22:50:23.729361+0000 mon.a (mon.0) 15357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:24 smithi121 bash[36902]: audit 2022-04-22T22:50:23.729361+0000 mon.a (mon.0) 15357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:24.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:25 smithi115 bash[45495]: cluster 2022-04-22T22:50:24.199210+0000 mgr.y (mgr.34107) 11056 : cluster [DBG] pgmap v11669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:25 smithi115 bash[46934]: cluster 2022-04-22T22:50:24.199210+0000 mgr.y (mgr.34107) 11056 : cluster [DBG] pgmap v11669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:25 smithi121 bash[36902]: cluster 2022-04-22T22:50:24.199210+0000 mgr.y (mgr.34107) 11056 : cluster [DBG] pgmap v11669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:26 smithi115 bash[46934]: audit 2022-04-22T22:50:25.628190+0000 mon.a (mon.0) 15358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:26 smithi115 bash[45495]: audit 2022-04-22T22:50:25.628190+0000 mon.a (mon.0) 15358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:26 smithi121 bash[36902]: audit 2022-04-22T22:50:25.628190+0000 mon.a (mon.0) 15358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:27 smithi115 bash[45495]: cluster 2022-04-22T22:50:26.200373+0000 mgr.y (mgr.34107) 11057 : cluster [DBG] pgmap v11670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:27 smithi115 bash[45495]: audit 2022-04-22T22:50:27.247083+0000 mon.a (mon.0) 15359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:27 smithi115 bash[46934]: cluster 2022-04-22T22:50:26.200373+0000 mgr.y (mgr.34107) 11057 : cluster [DBG] pgmap v11670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:27 smithi115 bash[46934]: audit 2022-04-22T22:50:27.247083+0000 mon.a (mon.0) 15359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:27 smithi121 bash[36902]: cluster 2022-04-22T22:50:26.200373+0000 mgr.y (mgr.34107) 11057 : cluster [DBG] pgmap v11670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:27 smithi121 bash[36902]: audit 2022-04-22T22:50:27.247083+0000 mon.a (mon.0) 15359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:29 smithi115 bash[45495]: cluster 2022-04-22T22:50:28.201490+0000 mgr.y (mgr.34107) 11058 : cluster [DBG] pgmap v11671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:29 smithi115 bash[45495]: audit 2022-04-22T22:50:28.732857+0000 mon.a (mon.0) 15360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:29 smithi115 bash[46934]: cluster 2022-04-22T22:50:28.201490+0000 mgr.y (mgr.34107) 11058 : cluster [DBG] pgmap v11671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:29 smithi115 bash[46934]: audit 2022-04-22T22:50:28.732857+0000 mon.a (mon.0) 15360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:29 smithi121 bash[36902]: cluster 2022-04-22T22:50:28.201490+0000 mgr.y (mgr.34107) 11058 : cluster [DBG] pgmap v11671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:29 smithi121 bash[36902]: audit 2022-04-22T22:50:28.732857+0000 mon.a (mon.0) 15360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:31.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:31 smithi115 bash[45495]: cluster 2022-04-22T22:50:30.202674+0000 mgr.y (mgr.34107) 11059 : cluster [DBG] pgmap v11672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:31 smithi115 bash[45495]: audit 2022-04-22T22:50:30.631264+0000 mon.a (mon.0) 15361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:31 smithi115 bash[46934]: cluster 2022-04-22T22:50:30.202674+0000 mgr.y (mgr.34107) 11059 : cluster [DBG] pgmap v11672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:31 smithi115 bash[46934]: audit 2022-04-22T22:50:30.631264+0000 mon.a (mon.0) 15361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:31 smithi121 bash[36902]: cluster 2022-04-22T22:50:30.202674+0000 mgr.y (mgr.34107) 11059 : cluster [DBG] pgmap v11672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:31 smithi121 bash[36902]: audit 2022-04-22T22:50:30.631264+0000 mon.a (mon.0) 15361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:33 smithi121 bash[36902]: cluster 2022-04-22T22:50:32.203827+0000 mgr.y (mgr.34107) 11060 : cluster [DBG] pgmap v11673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:33 smithi121 bash[36902]: audit 2022-04-22T22:50:32.250237+0000 mon.a (mon.0) 15362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:33.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:33 smithi115 bash[45495]: cluster 2022-04-22T22:50:32.203827+0000 mgr.y (mgr.34107) 11060 : cluster [DBG] pgmap v11673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:33 smithi115 bash[45495]: audit 2022-04-22T22:50:32.250237+0000 mon.a (mon.0) 15362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:32.890Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:33 smithi115 bash[46934]: cluster 2022-04-22T22:50:32.203827+0000 mgr.y (mgr.34107) 11060 : cluster [DBG] pgmap v11673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:33 smithi115 bash[46934]: audit 2022-04-22T22:50:32.250237+0000 mon.a (mon.0) 15362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:34 smithi121 bash[36902]: audit 2022-04-22T22:50:33.736494+0000 mon.a (mon.0) 15363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:34 smithi115 bash[45495]: audit 2022-04-22T22:50:33.736494+0000 mon.a (mon.0) 15363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:34 smithi115 bash[46934]: audit 2022-04-22T22:50:33.736494+0000 mon.a (mon.0) 15363 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:35.015 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:34.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:35 smithi115 bash[45495]: cluster 2022-04-22T22:50:34.204916+0000 mgr.y (mgr.34107) 11061 : cluster [DBG] pgmap v11674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:35 smithi115 bash[46934]: cluster 2022-04-22T22:50:34.204916+0000 mgr.y (mgr.34107) 11061 : cluster [DBG] pgmap v11674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:35 smithi121 bash[36902]: cluster 2022-04-22T22:50:34.204916+0000 mgr.y (mgr.34107) 11061 : cluster [DBG] pgmap v11674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:36.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:36 smithi115 bash[45495]: audit 2022-04-22T22:50:35.634548+0000 mon.a (mon.0) 15364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:36 smithi115 bash[46934]: audit 2022-04-22T22:50:35.634548+0000 mon.a (mon.0) 15364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:36 smithi121 bash[36902]: audit 2022-04-22T22:50:35.634548+0000 mon.a (mon.0) 15364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:37 smithi115 bash[45495]: cluster 2022-04-22T22:50:36.205963+0000 mgr.y (mgr.34107) 11062 : cluster [DBG] pgmap v11675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:37 smithi115 bash[46934]: cluster 2022-04-22T22:50:36.205963+0000 mgr.y (mgr.34107) 11062 : cluster [DBG] pgmap v11675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:37 smithi121 bash[36902]: cluster 2022-04-22T22:50:36.205963+0000 mgr.y (mgr.34107) 11062 : cluster [DBG] pgmap v11675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:38 smithi115 bash[45495]: audit 2022-04-22T22:50:37.253438+0000 mon.a (mon.0) 15365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:38 smithi115 bash[46934]: audit 2022-04-22T22:50:37.253438+0000 mon.a (mon.0) 15365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:38 smithi121 bash[36902]: audit 2022-04-22T22:50:37.253438+0000 mon.a (mon.0) 15365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:39 smithi115 bash[45495]: cluster 2022-04-22T22:50:38.207076+0000 mgr.y (mgr.34107) 11063 : cluster [DBG] pgmap v11676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:39 smithi115 bash[45495]: audit 2022-04-22T22:50:38.740101+0000 mon.a (mon.0) 15366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:39 smithi115 bash[46934]: cluster 2022-04-22T22:50:38.207076+0000 mgr.y (mgr.34107) 11063 : cluster [DBG] pgmap v11676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:39 smithi115 bash[46934]: audit 2022-04-22T22:50:38.740101+0000 mon.a (mon.0) 15366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:39 smithi121 bash[36902]: cluster 2022-04-22T22:50:38.207076+0000 mgr.y (mgr.34107) 11063 : cluster [DBG] pgmap v11676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:39.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:39 smithi121 bash[36902]: audit 2022-04-22T22:50:38.740101+0000 mon.a (mon.0) 15366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:41 smithi115 bash[45495]: cluster 2022-04-22T22:50:40.208210+0000 mgr.y (mgr.34107) 11064 : cluster [DBG] pgmap v11677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:41 smithi115 bash[45495]: audit 2022-04-22T22:50:40.637903+0000 mon.a (mon.0) 15367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:41 smithi115 bash[46934]: cluster 2022-04-22T22:50:40.208210+0000 mgr.y (mgr.34107) 11064 : cluster [DBG] pgmap v11677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:41 smithi115 bash[46934]: audit 2022-04-22T22:50:40.637903+0000 mon.a (mon.0) 15367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:41 smithi121 bash[36902]: cluster 2022-04-22T22:50:40.208210+0000 mgr.y (mgr.34107) 11064 : cluster [DBG] pgmap v11677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:41 smithi121 bash[36902]: audit 2022-04-22T22:50:40.637903+0000 mon.a (mon.0) 15367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:43 smithi121 bash[36902]: cluster 2022-04-22T22:50:42.209254+0000 mgr.y (mgr.34107) 11065 : cluster [DBG] pgmap v11678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:43.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:43 smithi121 bash[36902]: audit 2022-04-22T22:50:42.256544+0000 mon.a (mon.0) 15368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:42.892Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:43 smithi115 bash[45495]: cluster 2022-04-22T22:50:42.209254+0000 mgr.y (mgr.34107) 11065 : cluster [DBG] pgmap v11678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:43 smithi115 bash[45495]: audit 2022-04-22T22:50:42.256544+0000 mon.a (mon.0) 15368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:43 smithi115 bash[46934]: cluster 2022-04-22T22:50:42.209254+0000 mgr.y (mgr.34107) 11065 : cluster [DBG] pgmap v11678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:43 smithi115 bash[46934]: audit 2022-04-22T22:50:42.256544+0000 mon.a (mon.0) 15368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:44.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:44 smithi121 bash[36902]: audit 2022-04-22T22:50:43.743723+0000 mon.a (mon.0) 15369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:44.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:44 smithi115 bash[45495]: audit 2022-04-22T22:50:43.743723+0000 mon.a (mon.0) 15369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:44 smithi115 bash[46934]: audit 2022-04-22T22:50:43.743723+0000 mon.a (mon.0) 15369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:45.015 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:44.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:45.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:45 smithi115 bash[45495]: cluster 2022-04-22T22:50:44.210367+0000 mgr.y (mgr.34107) 11066 : cluster [DBG] pgmap v11679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:45.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:45 smithi115 bash[46934]: cluster 2022-04-22T22:50:44.210367+0000 mgr.y (mgr.34107) 11066 : cluster [DBG] pgmap v11679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:45 smithi121 bash[36902]: cluster 2022-04-22T22:50:44.210367+0000 mgr.y (mgr.34107) 11066 : cluster [DBG] pgmap v11679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:46 smithi115 bash[45495]: audit 2022-04-22T22:50:45.640576+0000 mon.a (mon.0) 15370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:46 smithi115 bash[46934]: audit 2022-04-22T22:50:45.640576+0000 mon.a (mon.0) 15370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:46 smithi121 bash[36902]: audit 2022-04-22T22:50:45.640576+0000 mon.a (mon.0) 15370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:47 smithi115 bash[45495]: cluster 2022-04-22T22:50:46.211660+0000 mgr.y (mgr.34107) 11067 : cluster [DBG] pgmap v11680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:47 smithi115 bash[46934]: cluster 2022-04-22T22:50:46.211660+0000 mgr.y (mgr.34107) 11067 : cluster [DBG] pgmap v11680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:47 smithi121 bash[36902]: cluster 2022-04-22T22:50:46.211660+0000 mgr.y (mgr.34107) 11067 : cluster [DBG] pgmap v11680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:48 smithi115 bash[45495]: audit 2022-04-22T22:50:47.259353+0000 mon.a (mon.0) 15371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:48 smithi115 bash[46934]: audit 2022-04-22T22:50:47.259353+0000 mon.a (mon.0) 15371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:48 smithi121 bash[36902]: audit 2022-04-22T22:50:47.259353+0000 mon.a (mon.0) 15371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:49 smithi115 bash[45495]: cluster 2022-04-22T22:50:48.212901+0000 mgr.y (mgr.34107) 11068 : cluster [DBG] pgmap v11681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:49 smithi115 bash[45495]: audit 2022-04-22T22:50:48.747087+0000 mon.a (mon.0) 15372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:49 smithi115 bash[46934]: cluster 2022-04-22T22:50:48.212901+0000 mgr.y (mgr.34107) 11068 : cluster [DBG] pgmap v11681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:49 smithi115 bash[46934]: audit 2022-04-22T22:50:48.747087+0000 mon.a (mon.0) 15372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:49 smithi121 bash[36902]: cluster 2022-04-22T22:50:48.212901+0000 mgr.y (mgr.34107) 11068 : cluster [DBG] pgmap v11681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:49 smithi121 bash[36902]: audit 2022-04-22T22:50:48.747087+0000 mon.a (mon.0) 15372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:51.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:51 smithi115 bash[45495]: cluster 2022-04-22T22:50:50.213978+0000 mgr.y (mgr.34107) 11069 : cluster [DBG] pgmap v11682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:51 smithi115 bash[45495]: audit 2022-04-22T22:50:50.643423+0000 mon.a (mon.0) 15373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:51 smithi115 bash[46934]: cluster 2022-04-22T22:50:50.213978+0000 mgr.y (mgr.34107) 11069 : cluster [DBG] pgmap v11682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:51 smithi115 bash[46934]: audit 2022-04-22T22:50:50.643423+0000 mon.a (mon.0) 15373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:51 smithi121 bash[36902]: cluster 2022-04-22T22:50:50.213978+0000 mgr.y (mgr.34107) 11069 : cluster [DBG] pgmap v11682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:51 smithi121 bash[36902]: audit 2022-04-22T22:50:50.643423+0000 mon.a (mon.0) 15373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:52.915 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:50:53.257 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:50:53.257 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:50:53.257 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:50:53.257 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:50:53.258 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:50:53.259 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:50:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:53 smithi121 bash[36902]: cluster 2022-04-22T22:50:52.215017+0000 mgr.y (mgr.34107) 11070 : cluster [DBG] pgmap v11683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:53 smithi121 bash[36902]: audit 2022-04-22T22:50:52.262685+0000 mon.a (mon.0) 15374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:53.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:53 smithi121 bash[36902]: audit 2022-04-22T22:50:52.897439+0000 mgr.y (mgr.34107) 11071 : audit [DBG] from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[45495]: cluster 2022-04-22T22:50:52.215017+0000 mgr.y (mgr.34107) 11070 : cluster [DBG] pgmap v11683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[45495]: audit 2022-04-22T22:50:52.262685+0000 mon.a (mon.0) 15374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[45495]: audit 2022-04-22T22:50:52.897439+0000 mgr.y (mgr.34107) 11071 : audit [DBG] from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:52.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[46934]: cluster 2022-04-22T22:50:52.215017+0000 mgr.y (mgr.34107) 11070 : cluster [DBG] pgmap v11683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[46934]: audit 2022-04-22T22:50:52.262685+0000 mon.a (mon.0) 15374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:53 smithi115 bash[46934]: audit 2022-04-22T22:50:52.897439+0000 mgr.y (mgr.34107) 11071 : audit [DBG] from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:50:53.658 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:50:53.658 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:50:53.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:53.658 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:50:53.659 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:50:53.660 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:50:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:54 smithi121 bash[36902]: audit 2022-04-22T22:50:53.240079+0000 mgr.y (mgr.34107) 11072 : audit [DBG] from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:54.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:54 smithi121 bash[36902]: audit 2022-04-22T22:50:53.654539+0000 mon.a (mon.0) 15375 : audit [DBG] from='client.? 172.21.15.115:0/2614588771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:54 smithi121 bash[36902]: audit 2022-04-22T22:50:53.750834+0000 mon.a (mon.0) 15376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[46934]: audit 2022-04-22T22:50:53.240079+0000 mgr.y (mgr.34107) 11072 : audit [DBG] from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[46934]: audit 2022-04-22T22:50:53.654539+0000 mon.a (mon.0) 15375 : audit [DBG] from='client.? 172.21.15.115:0/2614588771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[46934]: audit 2022-04-22T22:50:53.750834+0000 mon.a (mon.0) 15376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[45495]: audit 2022-04-22T22:50:53.240079+0000 mgr.y (mgr.34107) 11072 : audit [DBG] from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:50:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[45495]: audit 2022-04-22T22:50:53.654539+0000 mon.a (mon.0) 15375 : audit [DBG] from='client.? 172.21.15.115:0/2614588771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:50:54.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[45495]: audit 2022-04-22T22:50:53.750834+0000 mon.a (mon.0) 15376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:55.012 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:50:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:50:54.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:50:55.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:55 smithi115 bash[45495]: cluster 2022-04-22T22:50:54.216100+0000 mgr.y (mgr.34107) 11073 : cluster [DBG] pgmap v11684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:55.014 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:55 smithi115 bash[46934]: cluster 2022-04-22T22:50:54.216100+0000 mgr.y (mgr.34107) 11073 : cluster [DBG] pgmap v11684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:55 smithi121 bash[36902]: cluster 2022-04-22T22:50:54.216100+0000 mgr.y (mgr.34107) 11073 : cluster [DBG] pgmap v11684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:56 smithi115 bash[45495]: audit 2022-04-22T22:50:55.646454+0000 mon.a (mon.0) 15377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:56 smithi115 bash[46934]: audit 2022-04-22T22:50:55.646454+0000 mon.a (mon.0) 15377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:56 smithi121 bash[36902]: audit 2022-04-22T22:50:55.646454+0000 mon.a (mon.0) 15377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:50:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:57 smithi115 bash[45495]: cluster 2022-04-22T22:50:56.217236+0000 mgr.y (mgr.34107) 11074 : cluster [DBG] pgmap v11685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:57 smithi115 bash[46934]: cluster 2022-04-22T22:50:56.217236+0000 mgr.y (mgr.34107) 11074 : cluster [DBG] pgmap v11685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:57 smithi121 bash[36902]: cluster 2022-04-22T22:50:56.217236+0000 mgr.y (mgr.34107) 11074 : cluster [DBG] pgmap v11685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:58 smithi115 bash[45495]: audit 2022-04-22T22:50:57.265776+0000 mon.a (mon.0) 15378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:58 smithi115 bash[46934]: audit 2022-04-22T22:50:57.265776+0000 mon.a (mon.0) 15378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:58 smithi121 bash[36902]: audit 2022-04-22T22:50:57.265776+0000 mon.a (mon.0) 15378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:50:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:59 smithi121 bash[36902]: cluster 2022-04-22T22:50:58.218425+0000 mgr.y (mgr.34107) 11075 : cluster [DBG] pgmap v11686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:50:59 smithi121 bash[36902]: audit 2022-04-22T22:50:58.754610+0000 mon.a (mon.0) 15379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:59 smithi115 bash[46934]: cluster 2022-04-22T22:50:58.218425+0000 mgr.y (mgr.34107) 11075 : cluster [DBG] pgmap v11686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:50:59 smithi115 bash[46934]: audit 2022-04-22T22:50:58.754610+0000 mon.a (mon.0) 15379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:50:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:59 smithi115 bash[45495]: cluster 2022-04-22T22:50:58.218425+0000 mgr.y (mgr.34107) 11075 : cluster [DBG] pgmap v11686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:50:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:50:59 smithi115 bash[45495]: audit 2022-04-22T22:50:58.754610+0000 mon.a (mon.0) 15379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:01.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:01 smithi115 bash[45495]: cluster 2022-04-22T22:51:00.219667+0000 mgr.y (mgr.34107) 11076 : cluster [DBG] pgmap v11687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:01 smithi115 bash[45495]: audit 2022-04-22T22:51:00.649491+0000 mon.a (mon.0) 15380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:01 smithi115 bash[46934]: cluster 2022-04-22T22:51:00.219667+0000 mgr.y (mgr.34107) 11076 : cluster [DBG] pgmap v11687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:01 smithi115 bash[46934]: audit 2022-04-22T22:51:00.649491+0000 mon.a (mon.0) 15380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:01.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:01 smithi121 bash[36902]: cluster 2022-04-22T22:51:00.219667+0000 mgr.y (mgr.34107) 11076 : cluster [DBG] pgmap v11687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:01.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:01 smithi121 bash[36902]: audit 2022-04-22T22:51:00.649491+0000 mon.a (mon.0) 15380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[45495]: audit 2022-04-22T22:51:01.585530+0000 mon.a (mon.0) 15381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[45495]: audit 2022-04-22T22:51:01.588513+0000 mon.a (mon.0) 15382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[45495]: audit 2022-04-22T22:51:01.590081+0000 mon.a (mon.0) 15383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[45495]: audit 2022-04-22T22:51:01.941444+0000 mon.a (mon.0) 15384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[45495]: audit 2022-04-22T22:51:02.269098+0000 mon.a (mon.0) 15385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[46934]: audit 2022-04-22T22:51:01.585530+0000 mon.a (mon.0) 15381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:51:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[46934]: audit 2022-04-22T22:51:01.588513+0000 mon.a (mon.0) 15382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:51:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[46934]: audit 2022-04-22T22:51:01.590081+0000 mon.a (mon.0) 15383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:51:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[46934]: audit 2022-04-22T22:51:01.941444+0000 mon.a (mon.0) 15384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:51:02.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[46934]: audit 2022-04-22T22:51:02.269098+0000 mon.a (mon.0) 15385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:02 smithi121 bash[36902]: audit 2022-04-22T22:51:01.585530+0000 mon.a (mon.0) 15381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:51:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:02 smithi121 bash[36902]: audit 2022-04-22T22:51:01.588513+0000 mon.a (mon.0) 15382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:51:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:02 smithi121 bash[36902]: audit 2022-04-22T22:51:01.590081+0000 mon.a (mon.0) 15383 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:51:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:02 smithi121 bash[36902]: audit 2022-04-22T22:51:01.941444+0000 mon.a (mon.0) 15384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:51:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:02 smithi121 bash[36902]: audit 2022-04-22T22:51:02.269098+0000 mon.a (mon.0) 15385 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:03.286 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:02.894Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:03 smithi115 bash[45495]: cluster 2022-04-22T22:51:02.220715+0000 mgr.y (mgr.34107) 11077 : cluster [DBG] pgmap v11688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:03 smithi115 bash[46934]: cluster 2022-04-22T22:51:02.220715+0000 mgr.y (mgr.34107) 11077 : cluster [DBG] pgmap v11688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:03 smithi121 bash[36902]: cluster 2022-04-22T22:51:02.220715+0000 mgr.y (mgr.34107) 11077 : cluster [DBG] pgmap v11688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:04.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:04 smithi115 bash[45495]: audit 2022-04-22T22:51:03.758361+0000 mon.a (mon.0) 15386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:04.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:04 smithi115 bash[46934]: audit 2022-04-22T22:51:03.758361+0000 mon.a (mon.0) 15386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:04 smithi121 bash[36902]: audit 2022-04-22T22:51:03.758361+0000 mon.a (mon.0) 15386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:04.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:05 smithi121 bash[36902]: cluster 2022-04-22T22:51:04.221798+0000 mgr.y (mgr.34107) 11078 : cluster [DBG] pgmap v11689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:05 smithi115 bash[45495]: cluster 2022-04-22T22:51:04.221798+0000 mgr.y (mgr.34107) 11078 : cluster [DBG] pgmap v11689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:05 smithi115 bash[46934]: cluster 2022-04-22T22:51:04.221798+0000 mgr.y (mgr.34107) 11078 : cluster [DBG] pgmap v11689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:06 smithi121 bash[36902]: audit 2022-04-22T22:51:05.652584+0000 mon.a (mon.0) 15387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:06 smithi115 bash[45495]: audit 2022-04-22T22:51:05.652584+0000 mon.a (mon.0) 15387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:06 smithi115 bash[46934]: audit 2022-04-22T22:51:05.652584+0000 mon.a (mon.0) 15387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:07 smithi121 bash[36902]: cluster 2022-04-22T22:51:06.222935+0000 mgr.y (mgr.34107) 11079 : cluster [DBG] pgmap v11690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:07 smithi121 bash[36902]: audit 2022-04-22T22:51:07.272174+0000 mon.a (mon.0) 15388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:07.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:07 smithi115 bash[46934]: cluster 2022-04-22T22:51:06.222935+0000 mgr.y (mgr.34107) 11079 : cluster [DBG] pgmap v11690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:07 smithi115 bash[46934]: audit 2022-04-22T22:51:07.272174+0000 mon.a (mon.0) 15388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:07 smithi115 bash[45495]: cluster 2022-04-22T22:51:06.222935+0000 mgr.y (mgr.34107) 11079 : cluster [DBG] pgmap v11690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:07 smithi115 bash[45495]: audit 2022-04-22T22:51:07.272174+0000 mon.a (mon.0) 15388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:09 smithi121 bash[36902]: cluster 2022-04-22T22:51:08.224141+0000 mgr.y (mgr.34107) 11080 : cluster [DBG] pgmap v11691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:09 smithi121 bash[36902]: audit 2022-04-22T22:51:08.762154+0000 mon.a (mon.0) 15389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:09 smithi115 bash[46934]: cluster 2022-04-22T22:51:08.224141+0000 mgr.y (mgr.34107) 11080 : cluster [DBG] pgmap v11691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:09 smithi115 bash[46934]: audit 2022-04-22T22:51:08.762154+0000 mon.a (mon.0) 15389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:09 smithi115 bash[45495]: cluster 2022-04-22T22:51:08.224141+0000 mgr.y (mgr.34107) 11080 : cluster [DBG] pgmap v11691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:09 smithi115 bash[45495]: audit 2022-04-22T22:51:08.762154+0000 mon.a (mon.0) 15389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:11 smithi121 bash[36902]: cluster 2022-04-22T22:51:10.225260+0000 mgr.y (mgr.34107) 11081 : cluster [DBG] pgmap v11692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:11 smithi121 bash[36902]: audit 2022-04-22T22:51:10.655603+0000 mon.a (mon.0) 15390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:11 smithi115 bash[46934]: cluster 2022-04-22T22:51:10.225260+0000 mgr.y (mgr.34107) 11081 : cluster [DBG] pgmap v11692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:11 smithi115 bash[46934]: audit 2022-04-22T22:51:10.655603+0000 mon.a (mon.0) 15390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:11 smithi115 bash[45495]: cluster 2022-04-22T22:51:10.225260+0000 mgr.y (mgr.34107) 11081 : cluster [DBG] pgmap v11692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:11 smithi115 bash[45495]: audit 2022-04-22T22:51:10.655603+0000 mon.a (mon.0) 15390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:12.893Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:13 smithi115 bash[45495]: cluster 2022-04-22T22:51:12.226359+0000 mgr.y (mgr.34107) 11082 : cluster [DBG] pgmap v11693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:13 smithi115 bash[45495]: audit 2022-04-22T22:51:12.275501+0000 mon.a (mon.0) 15391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:13 smithi115 bash[46934]: cluster 2022-04-22T22:51:12.226359+0000 mgr.y (mgr.34107) 11082 : cluster [DBG] pgmap v11693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:13 smithi115 bash[46934]: audit 2022-04-22T22:51:12.275501+0000 mon.a (mon.0) 15391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:13 smithi121 bash[36902]: cluster 2022-04-22T22:51:12.226359+0000 mgr.y (mgr.34107) 11082 : cluster [DBG] pgmap v11693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:13 smithi121 bash[36902]: audit 2022-04-22T22:51:12.275501+0000 mon.a (mon.0) 15391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:14 smithi115 bash[45495]: audit 2022-04-22T22:51:13.765862+0000 mon.a (mon.0) 15392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:14 smithi115 bash[46934]: audit 2022-04-22T22:51:13.765862+0000 mon.a (mon.0) 15392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:14 smithi121 bash[36902]: audit 2022-04-22T22:51:13.765862+0000 mon.a (mon.0) 15392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:15.029 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:14.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:15.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:15 smithi115 bash[45495]: cluster 2022-04-22T22:51:14.227452+0000 mgr.y (mgr.34107) 11083 : cluster [DBG] pgmap v11694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:15.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:15 smithi115 bash[46934]: cluster 2022-04-22T22:51:14.227452+0000 mgr.y (mgr.34107) 11083 : cluster [DBG] pgmap v11694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:15 smithi121 bash[36902]: cluster 2022-04-22T22:51:14.227452+0000 mgr.y (mgr.34107) 11083 : cluster [DBG] pgmap v11694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:16 smithi115 bash[45495]: audit 2022-04-22T22:51:15.658578+0000 mon.a (mon.0) 15393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:16 smithi115 bash[46934]: audit 2022-04-22T22:51:15.658578+0000 mon.a (mon.0) 15393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:16 smithi121 bash[36902]: audit 2022-04-22T22:51:15.658578+0000 mon.a (mon.0) 15393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[45495]: cluster 2022-04-22T22:51:16.228554+0000 mgr.y (mgr.34107) 11084 : cluster [DBG] pgmap v11695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[45495]: audit 2022-04-22T22:51:17.011420+0000 mon.a (mon.0) 15394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:51:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[45495]: audit 2022-04-22T22:51:17.011844+0000 mon.a (mon.0) 15395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:51:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[46934]: cluster 2022-04-22T22:51:16.228554+0000 mgr.y (mgr.34107) 11084 : cluster [DBG] pgmap v11695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[46934]: audit 2022-04-22T22:51:17.011420+0000 mon.a (mon.0) 15394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:51:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:17 smithi115 bash[46934]: audit 2022-04-22T22:51:17.011844+0000 mon.a (mon.0) 15395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:17 smithi121 bash[36902]: cluster 2022-04-22T22:51:16.228554+0000 mgr.y (mgr.34107) 11084 : cluster [DBG] pgmap v11695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:17 smithi121 bash[36902]: audit 2022-04-22T22:51:17.011420+0000 mon.a (mon.0) 15394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:51:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:17 smithi121 bash[36902]: audit 2022-04-22T22:51:17.011844+0000 mon.a (mon.0) 15395 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:51:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:18 smithi115 bash[45495]: audit 2022-04-22T22:51:17.278702+0000 mon.a (mon.0) 15396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:18 smithi115 bash[46934]: audit 2022-04-22T22:51:17.278702+0000 mon.a (mon.0) 15396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:18 smithi121 bash[36902]: audit 2022-04-22T22:51:17.278702+0000 mon.a (mon.0) 15396 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:19 smithi121 bash[36902]: cluster 2022-04-22T22:51:18.229727+0000 mgr.y (mgr.34107) 11085 : cluster [DBG] pgmap v11696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:19 smithi121 bash[36902]: audit 2022-04-22T22:51:18.769307+0000 mon.a (mon.0) 15397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:19 smithi115 bash[45495]: cluster 2022-04-22T22:51:18.229727+0000 mgr.y (mgr.34107) 11085 : cluster [DBG] pgmap v11696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:19 smithi115 bash[45495]: audit 2022-04-22T22:51:18.769307+0000 mon.a (mon.0) 15397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:19 smithi115 bash[46934]: cluster 2022-04-22T22:51:18.229727+0000 mgr.y (mgr.34107) 11085 : cluster [DBG] pgmap v11696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:19 smithi115 bash[46934]: audit 2022-04-22T22:51:18.769307+0000 mon.a (mon.0) 15397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:21 smithi115 bash[45495]: cluster 2022-04-22T22:51:20.230829+0000 mgr.y (mgr.34107) 11086 : cluster [DBG] pgmap v11697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:21 smithi115 bash[45495]: audit 2022-04-22T22:51:20.661504+0000 mon.a (mon.0) 15398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:21 smithi115 bash[46934]: cluster 2022-04-22T22:51:20.230829+0000 mgr.y (mgr.34107) 11086 : cluster [DBG] pgmap v11697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:21 smithi115 bash[46934]: audit 2022-04-22T22:51:20.661504+0000 mon.a (mon.0) 15398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:21 smithi121 bash[36902]: cluster 2022-04-22T22:51:20.230829+0000 mgr.y (mgr.34107) 11086 : cluster [DBG] pgmap v11697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:21 smithi121 bash[36902]: audit 2022-04-22T22:51:20.661504+0000 mon.a (mon.0) 15398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:22.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:23 smithi115 bash[45495]: cluster 2022-04-22T22:51:22.231939+0000 mgr.y (mgr.34107) 11087 : cluster [DBG] pgmap v11698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:23 smithi115 bash[45495]: audit 2022-04-22T22:51:22.281909+0000 mon.a (mon.0) 15399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:23 smithi115 bash[46934]: cluster 2022-04-22T22:51:22.231939+0000 mgr.y (mgr.34107) 11087 : cluster [DBG] pgmap v11698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:23 smithi115 bash[46934]: audit 2022-04-22T22:51:22.281909+0000 mon.a (mon.0) 15399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:23 smithi121 bash[36902]: cluster 2022-04-22T22:51:22.231939+0000 mgr.y (mgr.34107) 11087 : cluster [DBG] pgmap v11698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:23 smithi121 bash[36902]: audit 2022-04-22T22:51:22.281909+0000 mon.a (mon.0) 15399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:24.028 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:51:24.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:24 smithi115 bash[45495]: audit 2022-04-22T22:51:23.773026+0000 mon.a (mon.0) 15400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:24 smithi115 bash[46934]: audit 2022-04-22T22:51:23.773026+0000 mon.a (mon.0) 15400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:24.376 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:51:24.377 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:51:24.377 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:51:24.377 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:51:24.377 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:51:24.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:51:24.378 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:51:24.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:51:24.378 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:51:24.379 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:51:24.379 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:51:24.379 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:51:24.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:51:24.379 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:51:24.380 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:51:24.381 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:51:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:24 smithi121 bash[36902]: audit 2022-04-22T22:51:23.773026+0000 mon.a (mon.0) 15400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:24.779 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:51:24.780 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:51:25.038 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:24.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:25.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[46934]: audit 2022-04-22T22:51:24.010942+0000 mgr.y (mgr.34107) 11088 : audit [DBG] from='client.56767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[46934]: cluster 2022-04-22T22:51:24.233048+0000 mgr.y (mgr.34107) 11089 : cluster [DBG] pgmap v11699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[46934]: audit 2022-04-22T22:51:24.353285+0000 mgr.y (mgr.34107) 11090 : audit [DBG] from='client.56773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[46934]: audit 2022-04-22T22:51:24.775211+0000 mon.a (mon.0) 15401 : audit [DBG] from='client.? 172.21.15.115:0/325177702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[45495]: audit 2022-04-22T22:51:24.010942+0000 mgr.y (mgr.34107) 11088 : audit [DBG] from='client.56767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[45495]: cluster 2022-04-22T22:51:24.233048+0000 mgr.y (mgr.34107) 11089 : cluster [DBG] pgmap v11699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:25.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[45495]: audit 2022-04-22T22:51:24.353285+0000 mgr.y (mgr.34107) 11090 : audit [DBG] from='client.56773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:25 smithi115 bash[45495]: audit 2022-04-22T22:51:24.775211+0000 mon.a (mon.0) 15401 : audit [DBG] from='client.? 172.21.15.115:0/325177702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:25 smithi121 bash[36902]: audit 2022-04-22T22:51:24.010942+0000 mgr.y (mgr.34107) 11088 : audit [DBG] from='client.56767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:25 smithi121 bash[36902]: cluster 2022-04-22T22:51:24.233048+0000 mgr.y (mgr.34107) 11089 : cluster [DBG] pgmap v11699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:25 smithi121 bash[36902]: audit 2022-04-22T22:51:24.353285+0000 mgr.y (mgr.34107) 11090 : audit [DBG] from='client.56773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:25 smithi121 bash[36902]: audit 2022-04-22T22:51:24.775211+0000 mon.a (mon.0) 15401 : audit [DBG] from='client.? 172.21.15.115:0/325177702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:26 smithi121 bash[36902]: audit 2022-04-22T22:51:25.664446+0000 mon.a (mon.0) 15402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:26 smithi115 bash[46934]: audit 2022-04-22T22:51:25.664446+0000 mon.a (mon.0) 15402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:26 smithi115 bash[45495]: audit 2022-04-22T22:51:25.664446+0000 mon.a (mon.0) 15402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:27 smithi121 bash[36902]: cluster 2022-04-22T22:51:26.234075+0000 mgr.y (mgr.34107) 11091 : cluster [DBG] pgmap v11700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:27 smithi115 bash[45495]: cluster 2022-04-22T22:51:26.234075+0000 mgr.y (mgr.34107) 11091 : cluster [DBG] pgmap v11700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:27 smithi115 bash[46934]: cluster 2022-04-22T22:51:26.234075+0000 mgr.y (mgr.34107) 11091 : cluster [DBG] pgmap v11700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:28 smithi121 bash[36902]: audit 2022-04-22T22:51:27.284953+0000 mon.a (mon.0) 15403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:28 smithi115 bash[45495]: audit 2022-04-22T22:51:27.284953+0000 mon.a (mon.0) 15403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:28 smithi115 bash[46934]: audit 2022-04-22T22:51:27.284953+0000 mon.a (mon.0) 15403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:29 smithi121 bash[36902]: cluster 2022-04-22T22:51:28.235274+0000 mgr.y (mgr.34107) 11092 : cluster [DBG] pgmap v11701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:29 smithi121 bash[36902]: audit 2022-04-22T22:51:28.776624+0000 mon.a (mon.0) 15404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:29 smithi115 bash[45495]: cluster 2022-04-22T22:51:28.235274+0000 mgr.y (mgr.34107) 11092 : cluster [DBG] pgmap v11701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:29 smithi115 bash[45495]: audit 2022-04-22T22:51:28.776624+0000 mon.a (mon.0) 15404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:29 smithi115 bash[46934]: cluster 2022-04-22T22:51:28.235274+0000 mgr.y (mgr.34107) 11092 : cluster [DBG] pgmap v11701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:29 smithi115 bash[46934]: audit 2022-04-22T22:51:28.776624+0000 mon.a (mon.0) 15404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:31 smithi121 bash[36902]: cluster 2022-04-22T22:51:30.236392+0000 mgr.y (mgr.34107) 11093 : cluster [DBG] pgmap v11702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:31 smithi121 bash[36902]: audit 2022-04-22T22:51:30.667310+0000 mon.a (mon.0) 15405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:31 smithi115 bash[45495]: cluster 2022-04-22T22:51:30.236392+0000 mgr.y (mgr.34107) 11093 : cluster [DBG] pgmap v11702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:31 smithi115 bash[45495]: audit 2022-04-22T22:51:30.667310+0000 mon.a (mon.0) 15405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:31 smithi115 bash[46934]: cluster 2022-04-22T22:51:30.236392+0000 mgr.y (mgr.34107) 11093 : cluster [DBG] pgmap v11702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:31 smithi115 bash[46934]: audit 2022-04-22T22:51:30.667310+0000 mon.a (mon.0) 15405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:32.895Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:33 smithi115 bash[45495]: cluster 2022-04-22T22:51:32.237429+0000 mgr.y (mgr.34107) 11094 : cluster [DBG] pgmap v11703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:33 smithi115 bash[45495]: audit 2022-04-22T22:51:32.288207+0000 mon.a (mon.0) 15406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:33 smithi115 bash[46934]: cluster 2022-04-22T22:51:32.237429+0000 mgr.y (mgr.34107) 11094 : cluster [DBG] pgmap v11703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:33 smithi115 bash[46934]: audit 2022-04-22T22:51:32.288207+0000 mon.a (mon.0) 15406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:33 smithi121 bash[36902]: cluster 2022-04-22T22:51:32.237429+0000 mgr.y (mgr.34107) 11094 : cluster [DBG] pgmap v11703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:33 smithi121 bash[36902]: audit 2022-04-22T22:51:32.288207+0000 mon.a (mon.0) 15406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:34 smithi115 bash[45495]: audit 2022-04-22T22:51:33.780373+0000 mon.a (mon.0) 15407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:34 smithi115 bash[46934]: audit 2022-04-22T22:51:33.780373+0000 mon.a (mon.0) 15407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:34 smithi121 bash[36902]: audit 2022-04-22T22:51:33.780373+0000 mon.a (mon.0) 15407 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:35.036 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:34.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:35.037 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:35 smithi115 bash[45495]: cluster 2022-04-22T22:51:34.238449+0000 mgr.y (mgr.34107) 11095 : cluster [DBG] pgmap v11704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:35.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:35 smithi115 bash[46934]: cluster 2022-04-22T22:51:34.238449+0000 mgr.y (mgr.34107) 11095 : cluster [DBG] pgmap v11704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:35 smithi121 bash[36902]: cluster 2022-04-22T22:51:34.238449+0000 mgr.y (mgr.34107) 11095 : cluster [DBG] pgmap v11704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:36.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:36 smithi115 bash[46934]: audit 2022-04-22T22:51:35.670474+0000 mon.a (mon.0) 15408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:36 smithi115 bash[45495]: audit 2022-04-22T22:51:35.670474+0000 mon.a (mon.0) 15408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:36 smithi121 bash[36902]: audit 2022-04-22T22:51:35.670474+0000 mon.a (mon.0) 15408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:37 smithi121 bash[36902]: cluster 2022-04-22T22:51:36.239537+0000 mgr.y (mgr.34107) 11096 : cluster [DBG] pgmap v11705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:37 smithi115 bash[46934]: cluster 2022-04-22T22:51:36.239537+0000 mgr.y (mgr.34107) 11096 : cluster [DBG] pgmap v11705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:37 smithi115 bash[45495]: cluster 2022-04-22T22:51:36.239537+0000 mgr.y (mgr.34107) 11096 : cluster [DBG] pgmap v11705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:38 smithi121 bash[36902]: audit 2022-04-22T22:51:37.291030+0000 mon.a (mon.0) 15409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:38 smithi115 bash[46934]: audit 2022-04-22T22:51:37.291030+0000 mon.a (mon.0) 15409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:38 smithi115 bash[45495]: audit 2022-04-22T22:51:37.291030+0000 mon.a (mon.0) 15409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:39 smithi121 bash[36902]: cluster 2022-04-22T22:51:38.240645+0000 mgr.y (mgr.34107) 11097 : cluster [DBG] pgmap v11706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:39 smithi121 bash[36902]: audit 2022-04-22T22:51:38.784022+0000 mon.a (mon.0) 15410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:39 smithi115 bash[45495]: cluster 2022-04-22T22:51:38.240645+0000 mgr.y (mgr.34107) 11097 : cluster [DBG] pgmap v11706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:39 smithi115 bash[45495]: audit 2022-04-22T22:51:38.784022+0000 mon.a (mon.0) 15410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:39 smithi115 bash[46934]: cluster 2022-04-22T22:51:38.240645+0000 mgr.y (mgr.34107) 11097 : cluster [DBG] pgmap v11706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:39 smithi115 bash[46934]: audit 2022-04-22T22:51:38.784022+0000 mon.a (mon.0) 15410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:41 smithi121 bash[36902]: cluster 2022-04-22T22:51:40.241717+0000 mgr.y (mgr.34107) 11098 : cluster [DBG] pgmap v11707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:41 smithi121 bash[36902]: audit 2022-04-22T22:51:40.673514+0000 mon.a (mon.0) 15411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:41 smithi115 bash[46934]: cluster 2022-04-22T22:51:40.241717+0000 mgr.y (mgr.34107) 11098 : cluster [DBG] pgmap v11707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:41 smithi115 bash[46934]: audit 2022-04-22T22:51:40.673514+0000 mon.a (mon.0) 15411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:41 smithi115 bash[45495]: cluster 2022-04-22T22:51:40.241717+0000 mgr.y (mgr.34107) 11098 : cluster [DBG] pgmap v11707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:41 smithi115 bash[45495]: audit 2022-04-22T22:51:40.673514+0000 mon.a (mon.0) 15411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:42.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:43 smithi115 bash[46934]: cluster 2022-04-22T22:51:42.242773+0000 mgr.y (mgr.34107) 11099 : cluster [DBG] pgmap v11708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:43 smithi115 bash[46934]: audit 2022-04-22T22:51:42.294468+0000 mon.a (mon.0) 15412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:43 smithi115 bash[45495]: cluster 2022-04-22T22:51:42.242773+0000 mgr.y (mgr.34107) 11099 : cluster [DBG] pgmap v11708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:43 smithi115 bash[45495]: audit 2022-04-22T22:51:42.294468+0000 mon.a (mon.0) 15412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:43 smithi121 bash[36902]: cluster 2022-04-22T22:51:42.242773+0000 mgr.y (mgr.34107) 11099 : cluster [DBG] pgmap v11708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:43 smithi121 bash[36902]: audit 2022-04-22T22:51:42.294468+0000 mon.a (mon.0) 15412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:44 smithi115 bash[45495]: audit 2022-04-22T22:51:43.787811+0000 mon.a (mon.0) 15413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:44 smithi115 bash[46934]: audit 2022-04-22T22:51:43.787811+0000 mon.a (mon.0) 15413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:44 smithi121 bash[36902]: audit 2022-04-22T22:51:43.787811+0000 mon.a (mon.0) 15413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:45.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:44.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:45 smithi121 bash[36902]: cluster 2022-04-22T22:51:44.243845+0000 mgr.y (mgr.34107) 11100 : cluster [DBG] pgmap v11709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:45.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:45 smithi115 bash[46934]: cluster 2022-04-22T22:51:44.243845+0000 mgr.y (mgr.34107) 11100 : cluster [DBG] pgmap v11709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:45 smithi115 bash[45495]: cluster 2022-04-22T22:51:44.243845+0000 mgr.y (mgr.34107) 11100 : cluster [DBG] pgmap v11709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:46 smithi121 bash[36902]: audit 2022-04-22T22:51:45.676726+0000 mon.a (mon.0) 15414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:46.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:46 smithi115 bash[46934]: audit 2022-04-22T22:51:45.676726+0000 mon.a (mon.0) 15414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:46 smithi115 bash[45495]: audit 2022-04-22T22:51:45.676726+0000 mon.a (mon.0) 15414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:47 smithi121 bash[36902]: cluster 2022-04-22T22:51:46.244890+0000 mgr.y (mgr.34107) 11101 : cluster [DBG] pgmap v11710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:47.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:47 smithi115 bash[46934]: cluster 2022-04-22T22:51:46.244890+0000 mgr.y (mgr.34107) 11101 : cluster [DBG] pgmap v11710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:47 smithi115 bash[45495]: cluster 2022-04-22T22:51:46.244890+0000 mgr.y (mgr.34107) 11101 : cluster [DBG] pgmap v11710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:48 smithi121 bash[36902]: audit 2022-04-22T22:51:47.297579+0000 mon.a (mon.0) 15415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:48 smithi115 bash[46934]: audit 2022-04-22T22:51:47.297579+0000 mon.a (mon.0) 15415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:48 smithi115 bash[45495]: audit 2022-04-22T22:51:47.297579+0000 mon.a (mon.0) 15415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:49 smithi121 bash[36902]: cluster 2022-04-22T22:51:48.246082+0000 mgr.y (mgr.34107) 11102 : cluster [DBG] pgmap v11711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:49 smithi121 bash[36902]: audit 2022-04-22T22:51:48.791749+0000 mon.a (mon.0) 15416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:49.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:49 smithi115 bash[46934]: cluster 2022-04-22T22:51:48.246082+0000 mgr.y (mgr.34107) 11102 : cluster [DBG] pgmap v11711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:49.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:49 smithi115 bash[46934]: audit 2022-04-22T22:51:48.791749+0000 mon.a (mon.0) 15416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:49 smithi115 bash[45495]: cluster 2022-04-22T22:51:48.246082+0000 mgr.y (mgr.34107) 11102 : cluster [DBG] pgmap v11711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:49 smithi115 bash[45495]: audit 2022-04-22T22:51:48.791749+0000 mon.a (mon.0) 15416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:51 smithi121 bash[36902]: cluster 2022-04-22T22:51:50.247133+0000 mgr.y (mgr.34107) 11103 : cluster [DBG] pgmap v11712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:51 smithi121 bash[36902]: audit 2022-04-22T22:51:50.679865+0000 mon.a (mon.0) 15417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:51 smithi115 bash[46934]: cluster 2022-04-22T22:51:50.247133+0000 mgr.y (mgr.34107) 11103 : cluster [DBG] pgmap v11712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:51 smithi115 bash[46934]: audit 2022-04-22T22:51:50.679865+0000 mon.a (mon.0) 15417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:51 smithi115 bash[45495]: cluster 2022-04-22T22:51:50.247133+0000 mgr.y (mgr.34107) 11103 : cluster [DBG] pgmap v11712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:51 smithi115 bash[45495]: audit 2022-04-22T22:51:50.679865+0000 mon.a (mon.0) 15417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:53.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:53 smithi115 bash[46934]: cluster 2022-04-22T22:51:52.248230+0000 mgr.y (mgr.34107) 11104 : cluster [DBG] pgmap v11713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:53 smithi115 bash[46934]: audit 2022-04-22T22:51:52.300791+0000 mon.a (mon.0) 15418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:52.895Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:53 smithi115 bash[45495]: cluster 2022-04-22T22:51:52.248230+0000 mgr.y (mgr.34107) 11104 : cluster [DBG] pgmap v11713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:53 smithi115 bash[45495]: audit 2022-04-22T22:51:52.300791+0000 mon.a (mon.0) 15418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:53 smithi121 bash[36902]: cluster 2022-04-22T22:51:52.248230+0000 mgr.y (mgr.34107) 11104 : cluster [DBG] pgmap v11713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:53 smithi121 bash[36902]: audit 2022-04-22T22:51:52.300791+0000 mon.a (mon.0) 15418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:54 smithi115 bash[46934]: audit 2022-04-22T22:51:53.795278+0000 mon.a (mon.0) 15419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:54 smithi115 bash[45495]: audit 2022-04-22T22:51:53.795278+0000 mon.a (mon.0) 15419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:54 smithi121 bash[36902]: audit 2022-04-22T22:51:53.795278+0000 mon.a (mon.0) 15419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:51:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:51:54.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:51:55.169 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:51:55.513 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:51:55.513 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:51:55.514 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:51:55.515 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:51:55.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:55 smithi121 bash[36902]: cluster 2022-04-22T22:51:54.249347+0000 mgr.y (mgr.34107) 11105 : cluster [DBG] pgmap v11714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:55.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:55 smithi115 bash[46934]: cluster 2022-04-22T22:51:54.249347+0000 mgr.y (mgr.34107) 11105 : cluster [DBG] pgmap v11714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:55 smithi115 bash[45495]: cluster 2022-04-22T22:51:54.249347+0000 mgr.y (mgr.34107) 11105 : cluster [DBG] pgmap v11714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:51:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:51:55.909 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:51:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:56 smithi121 bash[36902]: audit 2022-04-22T22:51:55.149648+0000 mgr.y (mgr.34107) 11106 : audit [DBG] from='client.46848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:56 smithi121 bash[36902]: audit 2022-04-22T22:51:55.491042+0000 mgr.y (mgr.34107) 11107 : audit [DBG] from='client.56791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:56 smithi121 bash[36902]: audit 2022-04-22T22:51:55.682692+0000 mon.a (mon.0) 15420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:56 smithi121 bash[36902]: audit 2022-04-22T22:51:55.904266+0000 mon.c (mon.1) 374 : audit [DBG] from='client.? 172.21.15.115:0/2984427588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[45495]: audit 2022-04-22T22:51:55.149648+0000 mgr.y (mgr.34107) 11106 : audit [DBG] from='client.46848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[45495]: audit 2022-04-22T22:51:55.491042+0000 mgr.y (mgr.34107) 11107 : audit [DBG] from='client.56791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[45495]: audit 2022-04-22T22:51:55.682692+0000 mon.a (mon.0) 15420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[45495]: audit 2022-04-22T22:51:55.904266+0000 mon.c (mon.1) 374 : audit [DBG] from='client.? 172.21.15.115:0/2984427588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[46934]: audit 2022-04-22T22:51:55.149648+0000 mgr.y (mgr.34107) 11106 : audit [DBG] from='client.46848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[46934]: audit 2022-04-22T22:51:55.491042+0000 mgr.y (mgr.34107) 11107 : audit [DBG] from='client.56791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:51:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[46934]: audit 2022-04-22T22:51:55.682692+0000 mon.a (mon.0) 15420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:51:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:56 smithi115 bash[46934]: audit 2022-04-22T22:51:55.904266+0000 mon.c (mon.1) 374 : audit [DBG] from='client.? 172.21.15.115:0/2984427588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:51:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:57 smithi121 bash[36902]: cluster 2022-04-22T22:51:56.250580+0000 mgr.y (mgr.34107) 11108 : cluster [DBG] pgmap v11715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:57 smithi115 bash[45495]: cluster 2022-04-22T22:51:56.250580+0000 mgr.y (mgr.34107) 11108 : cluster [DBG] pgmap v11715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:57 smithi115 bash[46934]: cluster 2022-04-22T22:51:56.250580+0000 mgr.y (mgr.34107) 11108 : cluster [DBG] pgmap v11715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:58 smithi121 bash[36902]: audit 2022-04-22T22:51:57.303955+0000 mon.a (mon.0) 15421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:58 smithi115 bash[45495]: audit 2022-04-22T22:51:57.303955+0000 mon.a (mon.0) 15421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:58 smithi115 bash[46934]: audit 2022-04-22T22:51:57.303955+0000 mon.a (mon.0) 15421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:51:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:59 smithi121 bash[36902]: cluster 2022-04-22T22:51:58.251765+0000 mgr.y (mgr.34107) 11109 : cluster [DBG] pgmap v11716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:51:59 smithi121 bash[36902]: audit 2022-04-22T22:51:58.798916+0000 mon.a (mon.0) 15422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:59 smithi115 bash[45495]: cluster 2022-04-22T22:51:58.251765+0000 mgr.y (mgr.34107) 11109 : cluster [DBG] pgmap v11716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:51:59 smithi115 bash[45495]: audit 2022-04-22T22:51:58.798916+0000 mon.a (mon.0) 15422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:51:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:59 smithi115 bash[46934]: cluster 2022-04-22T22:51:58.251765+0000 mgr.y (mgr.34107) 11109 : cluster [DBG] pgmap v11716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:51:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:51:59 smithi115 bash[46934]: audit 2022-04-22T22:51:58.798916+0000 mon.a (mon.0) 15422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:01 smithi121 bash[36902]: cluster 2022-04-22T22:52:00.252834+0000 mgr.y (mgr.34107) 11110 : cluster [DBG] pgmap v11717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:01 smithi121 bash[36902]: audit 2022-04-22T22:52:00.685676+0000 mon.a (mon.0) 15423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:01 smithi115 bash[45495]: cluster 2022-04-22T22:52:00.252834+0000 mgr.y (mgr.34107) 11110 : cluster [DBG] pgmap v11717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:01 smithi115 bash[45495]: audit 2022-04-22T22:52:00.685676+0000 mon.a (mon.0) 15423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:01 smithi115 bash[46934]: cluster 2022-04-22T22:52:00.252834+0000 mgr.y (mgr.34107) 11110 : cluster [DBG] pgmap v11717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:01 smithi115 bash[46934]: audit 2022-04-22T22:52:00.685676+0000 mon.a (mon.0) 15423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:02 smithi121 bash[36902]: audit 2022-04-22T22:52:01.949004+0000 mon.a (mon.0) 15424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:52:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:02 smithi121 bash[36902]: audit 2022-04-22T22:52:01.951901+0000 mon.a (mon.0) 15425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:52:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:02 smithi121 bash[36902]: audit 2022-04-22T22:52:01.953557+0000 mon.a (mon.0) 15426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:52:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:02 smithi121 bash[36902]: audit 2022-04-22T22:52:02.306684+0000 mon.a (mon.0) 15427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[45495]: audit 2022-04-22T22:52:01.949004+0000 mon.a (mon.0) 15424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:52:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[45495]: audit 2022-04-22T22:52:01.951901+0000 mon.a (mon.0) 15425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:52:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[45495]: audit 2022-04-22T22:52:01.953557+0000 mon.a (mon.0) 15426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:52:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[45495]: audit 2022-04-22T22:52:02.306684+0000 mon.a (mon.0) 15427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[46934]: audit 2022-04-22T22:52:01.949004+0000 mon.a (mon.0) 15424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:52:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[46934]: audit 2022-04-22T22:52:01.951901+0000 mon.a (mon.0) 15425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:52:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[46934]: audit 2022-04-22T22:52:01.953557+0000 mon.a (mon.0) 15426 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:52:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[46934]: audit 2022-04-22T22:52:02.306684+0000 mon.a (mon.0) 15427 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:02.895Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:03 smithi121 bash[36902]: cluster 2022-04-22T22:52:02.253544+0000 mgr.y (mgr.34107) 11111 : cluster [DBG] pgmap v11718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:03 smithi121 bash[36902]: audit 2022-04-22T22:52:02.320585+0000 mon.a (mon.0) 15428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:52:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:03 smithi115 bash[46934]: cluster 2022-04-22T22:52:02.253544+0000 mgr.y (mgr.34107) 11111 : cluster [DBG] pgmap v11718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:03 smithi115 bash[46934]: audit 2022-04-22T22:52:02.320585+0000 mon.a (mon.0) 15428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:52:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:03 smithi115 bash[45495]: cluster 2022-04-22T22:52:02.253544+0000 mgr.y (mgr.34107) 11111 : cluster [DBG] pgmap v11718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:03 smithi115 bash[45495]: audit 2022-04-22T22:52:02.320585+0000 mon.a (mon.0) 15428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:52:04.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:04 smithi115 bash[45495]: audit 2022-04-22T22:52:03.802792+0000 mon.a (mon.0) 15429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:04.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:04 smithi115 bash[46934]: audit 2022-04-22T22:52:03.802792+0000 mon.a (mon.0) 15429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:04 smithi121 bash[36902]: audit 2022-04-22T22:52:03.802792+0000 mon.a (mon.0) 15429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:04.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:05 smithi121 bash[36902]: cluster 2022-04-22T22:52:04.254585+0000 mgr.y (mgr.34107) 11112 : cluster [DBG] pgmap v11719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:05 smithi115 bash[46934]: cluster 2022-04-22T22:52:04.254585+0000 mgr.y (mgr.34107) 11112 : cluster [DBG] pgmap v11719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:05 smithi115 bash[45495]: cluster 2022-04-22T22:52:04.254585+0000 mgr.y (mgr.34107) 11112 : cluster [DBG] pgmap v11719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:06 smithi121 bash[36902]: audit 2022-04-22T22:52:05.688705+0000 mon.a (mon.0) 15430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:06 smithi115 bash[45495]: audit 2022-04-22T22:52:05.688705+0000 mon.a (mon.0) 15430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:06 smithi115 bash[46934]: audit 2022-04-22T22:52:05.688705+0000 mon.a (mon.0) 15430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:07 smithi121 bash[36902]: cluster 2022-04-22T22:52:06.255701+0000 mgr.y (mgr.34107) 11113 : cluster [DBG] pgmap v11720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:07 smithi121 bash[36902]: audit 2022-04-22T22:52:07.308967+0000 mon.a (mon.0) 15431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:07.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:07 smithi115 bash[46934]: cluster 2022-04-22T22:52:06.255701+0000 mgr.y (mgr.34107) 11113 : cluster [DBG] pgmap v11720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:07 smithi115 bash[46934]: audit 2022-04-22T22:52:07.308967+0000 mon.a (mon.0) 15431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:07 smithi115 bash[45495]: cluster 2022-04-22T22:52:06.255701+0000 mgr.y (mgr.34107) 11113 : cluster [DBG] pgmap v11720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:07 smithi115 bash[45495]: audit 2022-04-22T22:52:07.308967+0000 mon.a (mon.0) 15431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:09 smithi121 bash[36902]: cluster 2022-04-22T22:52:08.256808+0000 mgr.y (mgr.34107) 11114 : cluster [DBG] pgmap v11721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:09 smithi121 bash[36902]: audit 2022-04-22T22:52:08.806547+0000 mon.a (mon.0) 15432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:09 smithi115 bash[45495]: cluster 2022-04-22T22:52:08.256808+0000 mgr.y (mgr.34107) 11114 : cluster [DBG] pgmap v11721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:09 smithi115 bash[45495]: audit 2022-04-22T22:52:08.806547+0000 mon.a (mon.0) 15432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:09 smithi115 bash[46934]: cluster 2022-04-22T22:52:08.256808+0000 mgr.y (mgr.34107) 11114 : cluster [DBG] pgmap v11721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:09 smithi115 bash[46934]: audit 2022-04-22T22:52:08.806547+0000 mon.a (mon.0) 15432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:11 smithi121 bash[36902]: cluster 2022-04-22T22:52:10.257860+0000 mgr.y (mgr.34107) 11115 : cluster [DBG] pgmap v11722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:11 smithi121 bash[36902]: audit 2022-04-22T22:52:10.691882+0000 mon.a (mon.0) 15433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:11 smithi115 bash[46934]: cluster 2022-04-22T22:52:10.257860+0000 mgr.y (mgr.34107) 11115 : cluster [DBG] pgmap v11722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:11 smithi115 bash[46934]: audit 2022-04-22T22:52:10.691882+0000 mon.a (mon.0) 15433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:11 smithi115 bash[45495]: cluster 2022-04-22T22:52:10.257860+0000 mgr.y (mgr.34107) 11115 : cluster [DBG] pgmap v11722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:11 smithi115 bash[45495]: audit 2022-04-22T22:52:10.691882+0000 mon.a (mon.0) 15433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:12.896Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:13 smithi115 bash[45495]: cluster 2022-04-22T22:52:12.258873+0000 mgr.y (mgr.34107) 11116 : cluster [DBG] pgmap v11723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:13 smithi115 bash[45495]: audit 2022-04-22T22:52:12.311999+0000 mon.a (mon.0) 15434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:13 smithi115 bash[46934]: cluster 2022-04-22T22:52:12.258873+0000 mgr.y (mgr.34107) 11116 : cluster [DBG] pgmap v11723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:13 smithi115 bash[46934]: audit 2022-04-22T22:52:12.311999+0000 mon.a (mon.0) 15434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:13 smithi121 bash[36902]: cluster 2022-04-22T22:52:12.258873+0000 mgr.y (mgr.34107) 11116 : cluster [DBG] pgmap v11723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:13 smithi121 bash[36902]: audit 2022-04-22T22:52:12.311999+0000 mon.a (mon.0) 15434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:14 smithi115 bash[45495]: audit 2022-04-22T22:52:13.810415+0000 mon.a (mon.0) 15435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:14 smithi115 bash[46934]: audit 2022-04-22T22:52:13.810415+0000 mon.a (mon.0) 15435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:14 smithi121 bash[36902]: audit 2022-04-22T22:52:13.810415+0000 mon.a (mon.0) 15435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:14.758Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:15 smithi121 bash[36902]: cluster 2022-04-22T22:52:14.260033+0000 mgr.y (mgr.34107) 11117 : cluster [DBG] pgmap v11724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:15 smithi115 bash[45495]: cluster 2022-04-22T22:52:14.260033+0000 mgr.y (mgr.34107) 11117 : cluster [DBG] pgmap v11724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:15 smithi115 bash[46934]: cluster 2022-04-22T22:52:14.260033+0000 mgr.y (mgr.34107) 11117 : cluster [DBG] pgmap v11724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:16 smithi121 bash[36902]: audit 2022-04-22T22:52:15.695048+0000 mon.a (mon.0) 15436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:16 smithi115 bash[45495]: audit 2022-04-22T22:52:15.695048+0000 mon.a (mon.0) 15436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:16 smithi115 bash[46934]: audit 2022-04-22T22:52:15.695048+0000 mon.a (mon.0) 15436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:17 smithi121 bash[36902]: cluster 2022-04-22T22:52:16.261059+0000 mgr.y (mgr.34107) 11118 : cluster [DBG] pgmap v11725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:17 smithi121 bash[36902]: audit 2022-04-22T22:52:17.022777+0000 mon.a (mon.0) 15437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:52:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:17 smithi121 bash[36902]: audit 2022-04-22T22:52:17.023024+0000 mon.a (mon.0) 15438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:52:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[45495]: cluster 2022-04-22T22:52:16.261059+0000 mgr.y (mgr.34107) 11118 : cluster [DBG] pgmap v11725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[45495]: audit 2022-04-22T22:52:17.022777+0000 mon.a (mon.0) 15437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:52:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[45495]: audit 2022-04-22T22:52:17.023024+0000 mon.a (mon.0) 15438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:52:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[46934]: cluster 2022-04-22T22:52:16.261059+0000 mgr.y (mgr.34107) 11118 : cluster [DBG] pgmap v11725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[46934]: audit 2022-04-22T22:52:17.022777+0000 mon.a (mon.0) 15437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:52:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:17 smithi115 bash[46934]: audit 2022-04-22T22:52:17.023024+0000 mon.a (mon.0) 15438 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:52:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:18 smithi121 bash[36902]: audit 2022-04-22T22:52:17.315173+0000 mon.a (mon.0) 15439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:18 smithi115 bash[45495]: audit 2022-04-22T22:52:17.315173+0000 mon.a (mon.0) 15439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:18 smithi115 bash[46934]: audit 2022-04-22T22:52:17.315173+0000 mon.a (mon.0) 15439 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:19 smithi121 bash[36902]: cluster 2022-04-22T22:52:18.262201+0000 mgr.y (mgr.34107) 11119 : cluster [DBG] pgmap v11726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:19 smithi121 bash[36902]: audit 2022-04-22T22:52:18.814258+0000 mon.a (mon.0) 15440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:19 smithi115 bash[45495]: cluster 2022-04-22T22:52:18.262201+0000 mgr.y (mgr.34107) 11119 : cluster [DBG] pgmap v11726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:19 smithi115 bash[45495]: audit 2022-04-22T22:52:18.814258+0000 mon.a (mon.0) 15440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:19 smithi115 bash[46934]: cluster 2022-04-22T22:52:18.262201+0000 mgr.y (mgr.34107) 11119 : cluster [DBG] pgmap v11726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:19 smithi115 bash[46934]: audit 2022-04-22T22:52:18.814258+0000 mon.a (mon.0) 15440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:21 smithi121 bash[36902]: cluster 2022-04-22T22:52:20.263262+0000 mgr.y (mgr.34107) 11120 : cluster [DBG] pgmap v11727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:21 smithi121 bash[36902]: audit 2022-04-22T22:52:20.698334+0000 mon.a (mon.0) 15441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:21 smithi115 bash[45495]: cluster 2022-04-22T22:52:20.263262+0000 mgr.y (mgr.34107) 11120 : cluster [DBG] pgmap v11727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:21 smithi115 bash[45495]: audit 2022-04-22T22:52:20.698334+0000 mon.a (mon.0) 15441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:21 smithi115 bash[46934]: cluster 2022-04-22T22:52:20.263262+0000 mgr.y (mgr.34107) 11120 : cluster [DBG] pgmap v11727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:21 smithi115 bash[46934]: audit 2022-04-22T22:52:20.698334+0000 mon.a (mon.0) 15441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:22.898Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:23 smithi115 bash[45495]: cluster 2022-04-22T22:52:22.264368+0000 mgr.y (mgr.34107) 11121 : cluster [DBG] pgmap v11728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:23 smithi115 bash[45495]: audit 2022-04-22T22:52:22.318290+0000 mon.a (mon.0) 15442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:23 smithi115 bash[46934]: cluster 2022-04-22T22:52:22.264368+0000 mgr.y (mgr.34107) 11121 : cluster [DBG] pgmap v11728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:23 smithi115 bash[46934]: audit 2022-04-22T22:52:22.318290+0000 mon.a (mon.0) 15442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:23 smithi121 bash[36902]: cluster 2022-04-22T22:52:22.264368+0000 mgr.y (mgr.34107) 11121 : cluster [DBG] pgmap v11728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:23 smithi121 bash[36902]: audit 2022-04-22T22:52:22.318290+0000 mon.a (mon.0) 15442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:24 smithi121 bash[36902]: audit 2022-04-22T22:52:23.817980+0000 mon.a (mon.0) 15443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:24 smithi115 bash[45495]: audit 2022-04-22T22:52:23.817980+0000 mon.a (mon.0) 15443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:24 smithi115 bash[46934]: audit 2022-04-22T22:52:23.817980+0000 mon.a (mon.0) 15443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:24.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:25 smithi121 bash[36902]: cluster 2022-04-22T22:52:24.265403+0000 mgr.y (mgr.34107) 11122 : cluster [DBG] pgmap v11729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:25 smithi115 bash[45495]: cluster 2022-04-22T22:52:24.265403+0000 mgr.y (mgr.34107) 11122 : cluster [DBG] pgmap v11729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:25 smithi115 bash[46934]: cluster 2022-04-22T22:52:24.265403+0000 mgr.y (mgr.34107) 11122 : cluster [DBG] pgmap v11729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:26.279 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:52:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:26 smithi121 bash[36902]: audit 2022-04-22T22:52:25.701446+0000 mon.a (mon.0) 15444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:26 smithi115 bash[45495]: audit 2022-04-22T22:52:25.701446+0000 mon.a (mon.0) 15444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:26 smithi115 bash[46934]: audit 2022-04-22T22:52:25.701446+0000 mon.a (mon.0) 15444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:52:26.619 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:52:26.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:52:26.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:52:26.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:52:26.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:52:26.621 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:52:27.025 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:52:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:52:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:52:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:52:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:52:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:52:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:52:27.028 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:52:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:27 smithi121 bash[36902]: audit 2022-04-22T22:52:26.260521+0000 mgr.y (mgr.34107) 11123 : audit [DBG] from='client.46866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:27 smithi121 bash[36902]: cluster 2022-04-22T22:52:26.266305+0000 mgr.y (mgr.34107) 11124 : cluster [DBG] pgmap v11730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:27 smithi121 bash[36902]: audit 2022-04-22T22:52:26.601023+0000 mgr.y (mgr.34107) 11125 : audit [DBG] from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:27 smithi121 bash[36902]: audit 2022-04-22T22:52:27.021892+0000 mon.c (mon.1) 375 : audit [DBG] from='client.? 172.21.15.115:0/877864964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[45495]: audit 2022-04-22T22:52:26.260521+0000 mgr.y (mgr.34107) 11123 : audit [DBG] from='client.46866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[45495]: cluster 2022-04-22T22:52:26.266305+0000 mgr.y (mgr.34107) 11124 : cluster [DBG] pgmap v11730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[45495]: audit 2022-04-22T22:52:26.601023+0000 mgr.y (mgr.34107) 11125 : audit [DBG] from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[45495]: audit 2022-04-22T22:52:27.021892+0000 mon.c (mon.1) 375 : audit [DBG] from='client.? 172.21.15.115:0/877864964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[46934]: audit 2022-04-22T22:52:26.260521+0000 mgr.y (mgr.34107) 11123 : audit [DBG] from='client.46866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[46934]: cluster 2022-04-22T22:52:26.266305+0000 mgr.y (mgr.34107) 11124 : cluster [DBG] pgmap v11730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[46934]: audit 2022-04-22T22:52:26.601023+0000 mgr.y (mgr.34107) 11125 : audit [DBG] from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:27 smithi115 bash[46934]: audit 2022-04-22T22:52:27.021892+0000 mon.c (mon.1) 375 : audit [DBG] from='client.? 172.21.15.115:0/877864964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:28 smithi121 bash[36902]: audit 2022-04-22T22:52:27.321552+0000 mon.a (mon.0) 15445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:28 smithi115 bash[45495]: audit 2022-04-22T22:52:27.321552+0000 mon.a (mon.0) 15445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:28 smithi115 bash[46934]: audit 2022-04-22T22:52:27.321552+0000 mon.a (mon.0) 15445 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:29 smithi121 bash[36902]: cluster 2022-04-22T22:52:28.267469+0000 mgr.y (mgr.34107) 11126 : cluster [DBG] pgmap v11731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:29 smithi121 bash[36902]: audit 2022-04-22T22:52:28.821704+0000 mon.a (mon.0) 15446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:29 smithi115 bash[45495]: cluster 2022-04-22T22:52:28.267469+0000 mgr.y (mgr.34107) 11126 : cluster [DBG] pgmap v11731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:29 smithi115 bash[45495]: audit 2022-04-22T22:52:28.821704+0000 mon.a (mon.0) 15446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:29 smithi115 bash[46934]: cluster 2022-04-22T22:52:28.267469+0000 mgr.y (mgr.34107) 11126 : cluster [DBG] pgmap v11731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:29 smithi115 bash[46934]: audit 2022-04-22T22:52:28.821704+0000 mon.a (mon.0) 15446 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:31 smithi121 bash[36902]: cluster 2022-04-22T22:52:30.268540+0000 mgr.y (mgr.34107) 11127 : cluster [DBG] pgmap v11732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:31 smithi121 bash[36902]: audit 2022-04-22T22:52:30.704410+0000 mon.a (mon.0) 15447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:31 smithi115 bash[45495]: cluster 2022-04-22T22:52:30.268540+0000 mgr.y (mgr.34107) 11127 : cluster [DBG] pgmap v11732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:31 smithi115 bash[45495]: audit 2022-04-22T22:52:30.704410+0000 mon.a (mon.0) 15447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:31 smithi115 bash[46934]: cluster 2022-04-22T22:52:30.268540+0000 mgr.y (mgr.34107) 11127 : cluster [DBG] pgmap v11732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:31 smithi115 bash[46934]: audit 2022-04-22T22:52:30.704410+0000 mon.a (mon.0) 15447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:32.898Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:33 smithi115 bash[45495]: cluster 2022-04-22T22:52:32.269604+0000 mgr.y (mgr.34107) 11128 : cluster [DBG] pgmap v11733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:33 smithi115 bash[45495]: audit 2022-04-22T22:52:32.324754+0000 mon.a (mon.0) 15448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:33 smithi115 bash[46934]: cluster 2022-04-22T22:52:32.269604+0000 mgr.y (mgr.34107) 11128 : cluster [DBG] pgmap v11733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:33 smithi115 bash[46934]: audit 2022-04-22T22:52:32.324754+0000 mon.a (mon.0) 15448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:33 smithi121 bash[36902]: cluster 2022-04-22T22:52:32.269604+0000 mgr.y (mgr.34107) 11128 : cluster [DBG] pgmap v11733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:33 smithi121 bash[36902]: audit 2022-04-22T22:52:32.324754+0000 mon.a (mon.0) 15448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:34 smithi121 bash[36902]: audit 2022-04-22T22:52:33.825581+0000 mon.a (mon.0) 15449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:34 smithi115 bash[45495]: audit 2022-04-22T22:52:33.825581+0000 mon.a (mon.0) 15449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:34 smithi115 bash[46934]: audit 2022-04-22T22:52:33.825581+0000 mon.a (mon.0) 15449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:34.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:35 smithi121 bash[36902]: cluster 2022-04-22T22:52:34.270679+0000 mgr.y (mgr.34107) 11129 : cluster [DBG] pgmap v11734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:35 smithi115 bash[45495]: cluster 2022-04-22T22:52:34.270679+0000 mgr.y (mgr.34107) 11129 : cluster [DBG] pgmap v11734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:35 smithi115 bash[46934]: cluster 2022-04-22T22:52:34.270679+0000 mgr.y (mgr.34107) 11129 : cluster [DBG] pgmap v11734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:36 smithi121 bash[36902]: audit 2022-04-22T22:52:35.707402+0000 mon.a (mon.0) 15450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:36 smithi115 bash[45495]: audit 2022-04-22T22:52:35.707402+0000 mon.a (mon.0) 15450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:36 smithi115 bash[46934]: audit 2022-04-22T22:52:35.707402+0000 mon.a (mon.0) 15450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:37 smithi121 bash[36902]: cluster 2022-04-22T22:52:36.271825+0000 mgr.y (mgr.34107) 11130 : cluster [DBG] pgmap v11735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:37 smithi115 bash[45495]: cluster 2022-04-22T22:52:36.271825+0000 mgr.y (mgr.34107) 11130 : cluster [DBG] pgmap v11735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:37 smithi115 bash[46934]: cluster 2022-04-22T22:52:36.271825+0000 mgr.y (mgr.34107) 11130 : cluster [DBG] pgmap v11735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:38 smithi121 bash[36902]: audit 2022-04-22T22:52:37.328059+0000 mon.a (mon.0) 15451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:38 smithi115 bash[45495]: audit 2022-04-22T22:52:37.328059+0000 mon.a (mon.0) 15451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:38 smithi115 bash[46934]: audit 2022-04-22T22:52:37.328059+0000 mon.a (mon.0) 15451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:39 smithi121 bash[36902]: cluster 2022-04-22T22:52:38.273005+0000 mgr.y (mgr.34107) 11131 : cluster [DBG] pgmap v11736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:39 smithi121 bash[36902]: audit 2022-04-22T22:52:38.829418+0000 mon.a (mon.0) 15452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:39.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:39 smithi115 bash[46934]: cluster 2022-04-22T22:52:38.273005+0000 mgr.y (mgr.34107) 11131 : cluster [DBG] pgmap v11736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:39 smithi115 bash[46934]: audit 2022-04-22T22:52:38.829418+0000 mon.a (mon.0) 15452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:39 smithi115 bash[45495]: cluster 2022-04-22T22:52:38.273005+0000 mgr.y (mgr.34107) 11131 : cluster [DBG] pgmap v11736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:39 smithi115 bash[45495]: audit 2022-04-22T22:52:38.829418+0000 mon.a (mon.0) 15452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:41 smithi121 bash[36902]: cluster 2022-04-22T22:52:40.274109+0000 mgr.y (mgr.34107) 11132 : cluster [DBG] pgmap v11737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:41 smithi121 bash[36902]: audit 2022-04-22T22:52:40.710423+0000 mon.a (mon.0) 15453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:41 smithi115 bash[45495]: cluster 2022-04-22T22:52:40.274109+0000 mgr.y (mgr.34107) 11132 : cluster [DBG] pgmap v11737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:41 smithi115 bash[45495]: audit 2022-04-22T22:52:40.710423+0000 mon.a (mon.0) 15453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:41 smithi115 bash[46934]: cluster 2022-04-22T22:52:40.274109+0000 mgr.y (mgr.34107) 11132 : cluster [DBG] pgmap v11737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:41 smithi115 bash[46934]: audit 2022-04-22T22:52:40.710423+0000 mon.a (mon.0) 15453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:42.899Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:43 smithi115 bash[46934]: cluster 2022-04-22T22:52:42.275206+0000 mgr.y (mgr.34107) 11133 : cluster [DBG] pgmap v11738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:43 smithi115 bash[46934]: audit 2022-04-22T22:52:42.331218+0000 mon.a (mon.0) 15454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:43 smithi115 bash[45495]: cluster 2022-04-22T22:52:42.275206+0000 mgr.y (mgr.34107) 11133 : cluster [DBG] pgmap v11738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:43 smithi115 bash[45495]: audit 2022-04-22T22:52:42.331218+0000 mon.a (mon.0) 15454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:43 smithi121 bash[36902]: cluster 2022-04-22T22:52:42.275206+0000 mgr.y (mgr.34107) 11133 : cluster [DBG] pgmap v11738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:43 smithi121 bash[36902]: audit 2022-04-22T22:52:42.331218+0000 mon.a (mon.0) 15454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:44 smithi121 bash[36902]: audit 2022-04-22T22:52:43.833181+0000 mon.a (mon.0) 15455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:44 smithi115 bash[45495]: audit 2022-04-22T22:52:43.833181+0000 mon.a (mon.0) 15455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:44 smithi115 bash[46934]: audit 2022-04-22T22:52:43.833181+0000 mon.a (mon.0) 15455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:44.759Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:45 smithi121 bash[36902]: cluster 2022-04-22T22:52:44.276327+0000 mgr.y (mgr.34107) 11134 : cluster [DBG] pgmap v11739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:45.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:45 smithi115 bash[46934]: cluster 2022-04-22T22:52:44.276327+0000 mgr.y (mgr.34107) 11134 : cluster [DBG] pgmap v11739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:45 smithi115 bash[45495]: cluster 2022-04-22T22:52:44.276327+0000 mgr.y (mgr.34107) 11134 : cluster [DBG] pgmap v11739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:46 smithi121 bash[36902]: audit 2022-04-22T22:52:45.713599+0000 mon.a (mon.0) 15456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:46 smithi115 bash[45495]: audit 2022-04-22T22:52:45.713599+0000 mon.a (mon.0) 15456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:46 smithi115 bash[46934]: audit 2022-04-22T22:52:45.713599+0000 mon.a (mon.0) 15456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:47 smithi121 bash[36902]: cluster 2022-04-22T22:52:46.277444+0000 mgr.y (mgr.34107) 11135 : cluster [DBG] pgmap v11740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:47.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:47 smithi115 bash[45495]: cluster 2022-04-22T22:52:46.277444+0000 mgr.y (mgr.34107) 11135 : cluster [DBG] pgmap v11740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:47.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:47 smithi115 bash[46934]: cluster 2022-04-22T22:52:46.277444+0000 mgr.y (mgr.34107) 11135 : cluster [DBG] pgmap v11740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:48 smithi121 bash[36902]: audit 2022-04-22T22:52:47.334198+0000 mon.a (mon.0) 15457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:48 smithi115 bash[45495]: audit 2022-04-22T22:52:47.334198+0000 mon.a (mon.0) 15457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:48 smithi115 bash[46934]: audit 2022-04-22T22:52:47.334198+0000 mon.a (mon.0) 15457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:49 smithi121 bash[36902]: cluster 2022-04-22T22:52:48.278550+0000 mgr.y (mgr.34107) 11136 : cluster [DBG] pgmap v11741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:49 smithi121 bash[36902]: audit 2022-04-22T22:52:48.836987+0000 mon.a (mon.0) 15458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:49 smithi115 bash[45495]: cluster 2022-04-22T22:52:48.278550+0000 mgr.y (mgr.34107) 11136 : cluster [DBG] pgmap v11741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:49 smithi115 bash[45495]: audit 2022-04-22T22:52:48.836987+0000 mon.a (mon.0) 15458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:49 smithi115 bash[46934]: cluster 2022-04-22T22:52:48.278550+0000 mgr.y (mgr.34107) 11136 : cluster [DBG] pgmap v11741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:49 smithi115 bash[46934]: audit 2022-04-22T22:52:48.836987+0000 mon.a (mon.0) 15458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:51 smithi121 bash[36902]: cluster 2022-04-22T22:52:50.279646+0000 mgr.y (mgr.34107) 11137 : cluster [DBG] pgmap v11742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:51 smithi121 bash[36902]: audit 2022-04-22T22:52:50.716652+0000 mon.a (mon.0) 15459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:51 smithi115 bash[45495]: cluster 2022-04-22T22:52:50.279646+0000 mgr.y (mgr.34107) 11137 : cluster [DBG] pgmap v11742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:51 smithi115 bash[45495]: audit 2022-04-22T22:52:50.716652+0000 mon.a (mon.0) 15459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:51 smithi115 bash[46934]: cluster 2022-04-22T22:52:50.279646+0000 mgr.y (mgr.34107) 11137 : cluster [DBG] pgmap v11742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:51 smithi115 bash[46934]: audit 2022-04-22T22:52:50.716652+0000 mon.a (mon.0) 15459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:52.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:53 smithi115 bash[46934]: cluster 2022-04-22T22:52:52.280648+0000 mgr.y (mgr.34107) 11138 : cluster [DBG] pgmap v11743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:53 smithi115 bash[46934]: audit 2022-04-22T22:52:52.337585+0000 mon.a (mon.0) 15460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:53 smithi115 bash[45495]: cluster 2022-04-22T22:52:52.280648+0000 mgr.y (mgr.34107) 11138 : cluster [DBG] pgmap v11743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:53 smithi115 bash[45495]: audit 2022-04-22T22:52:52.337585+0000 mon.a (mon.0) 15460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:53 smithi121 bash[36902]: cluster 2022-04-22T22:52:52.280648+0000 mgr.y (mgr.34107) 11138 : cluster [DBG] pgmap v11743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:53 smithi121 bash[36902]: audit 2022-04-22T22:52:52.337585+0000 mon.a (mon.0) 15460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:54 smithi121 bash[36902]: audit 2022-04-22T22:52:53.840914+0000 mon.a (mon.0) 15461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:54 smithi115 bash[45495]: audit 2022-04-22T22:52:53.840914+0000 mon.a (mon.0) 15461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:54 smithi115 bash[46934]: audit 2022-04-22T22:52:53.840914+0000 mon.a (mon.0) 15461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:52:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:52:54.760Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:52:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:55 smithi121 bash[36902]: cluster 2022-04-22T22:52:54.281680+0000 mgr.y (mgr.34107) 11139 : cluster [DBG] pgmap v11744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:55 smithi115 bash[45495]: cluster 2022-04-22T22:52:54.281680+0000 mgr.y (mgr.34107) 11139 : cluster [DBG] pgmap v11744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:55 smithi115 bash[46934]: cluster 2022-04-22T22:52:54.281680+0000 mgr.y (mgr.34107) 11139 : cluster [DBG] pgmap v11744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:56 smithi121 bash[36902]: audit 2022-04-22T22:52:55.719790+0000 mon.a (mon.0) 15462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:56 smithi115 bash[45495]: audit 2022-04-22T22:52:55.719790+0000 mon.a (mon.0) 15462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:56 smithi115 bash[46934]: audit 2022-04-22T22:52:55.719790+0000 mon.a (mon.0) 15462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:52:57.401 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:52:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:57 smithi121 bash[36902]: cluster 2022-04-22T22:52:56.282685+0000 mgr.y (mgr.34107) 11140 : cluster [DBG] pgmap v11745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:57.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:57 smithi115 bash[46934]: cluster 2022-04-22T22:52:56.282685+0000 mgr.y (mgr.34107) 11140 : cluster [DBG] pgmap v11745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:57 smithi115 bash[45495]: cluster 2022-04-22T22:52:56.282685+0000 mgr.y (mgr.34107) 11140 : cluster [DBG] pgmap v11745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:57.739 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:52:57.739 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:52:57.739 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:52:57.740 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:52:57.741 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:52:58.144 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:52:58.144 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:52:58.144 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:52:58.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:58.144 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:52:58.145 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:52:58.146 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:52:58.146 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:52:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:58 smithi121 bash[36902]: audit 2022-04-22T22:52:57.340383+0000 mon.a (mon.0) 15463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:58 smithi121 bash[36902]: audit 2022-04-22T22:52:57.382656+0000 mgr.y (mgr.34107) 11141 : audit [DBG] from='client.46884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:58 smithi121 bash[36902]: audit 2022-04-22T22:52:57.716879+0000 mgr.y (mgr.34107) 11142 : audit [DBG] from='client.56827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[45495]: audit 2022-04-22T22:52:57.340383+0000 mon.a (mon.0) 15463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[45495]: audit 2022-04-22T22:52:57.382656+0000 mgr.y (mgr.34107) 11141 : audit [DBG] from='client.46884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[45495]: audit 2022-04-22T22:52:57.716879+0000 mgr.y (mgr.34107) 11142 : audit [DBG] from='client.56827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[46934]: audit 2022-04-22T22:52:57.340383+0000 mon.a (mon.0) 15463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:52:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[46934]: audit 2022-04-22T22:52:57.382656+0000 mgr.y (mgr.34107) 11141 : audit [DBG] from='client.46884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:58 smithi115 bash[46934]: audit 2022-04-22T22:52:57.716879+0000 mgr.y (mgr.34107) 11142 : audit [DBG] from='client.56827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:52:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:59 smithi121 bash[36902]: audit 2022-04-22T22:52:58.140709+0000 mon.c (mon.1) 376 : audit [DBG] from='client.? 172.21.15.115:0/212215055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:59 smithi121 bash[36902]: cluster 2022-04-22T22:52:58.283943+0000 mgr.y (mgr.34107) 11143 : cluster [DBG] pgmap v11746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:52:59 smithi121 bash[36902]: audit 2022-04-22T22:52:58.844484+0000 mon.a (mon.0) 15464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[45495]: audit 2022-04-22T22:52:58.140709+0000 mon.c (mon.1) 376 : audit [DBG] from='client.? 172.21.15.115:0/212215055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[45495]: cluster 2022-04-22T22:52:58.283943+0000 mgr.y (mgr.34107) 11143 : cluster [DBG] pgmap v11746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[45495]: audit 2022-04-22T22:52:58.844484+0000 mon.a (mon.0) 15464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:52:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[46934]: audit 2022-04-22T22:52:58.140709+0000 mon.c (mon.1) 376 : audit [DBG] from='client.? 172.21.15.115:0/212215055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:52:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[46934]: cluster 2022-04-22T22:52:58.283943+0000 mgr.y (mgr.34107) 11143 : cluster [DBG] pgmap v11746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:52:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:52:59 smithi115 bash[46934]: audit 2022-04-22T22:52:58.844484+0000 mon.a (mon.0) 15464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:01 smithi121 bash[36902]: cluster 2022-04-22T22:53:00.284983+0000 mgr.y (mgr.34107) 11144 : cluster [DBG] pgmap v11747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:01 smithi121 bash[36902]: audit 2022-04-22T22:53:00.722920+0000 mon.a (mon.0) 15465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:01 smithi115 bash[45495]: cluster 2022-04-22T22:53:00.284983+0000 mgr.y (mgr.34107) 11144 : cluster [DBG] pgmap v11747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:01 smithi115 bash[45495]: audit 2022-04-22T22:53:00.722920+0000 mon.a (mon.0) 15465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:01 smithi115 bash[46934]: cluster 2022-04-22T22:53:00.284983+0000 mgr.y (mgr.34107) 11144 : cluster [DBG] pgmap v11747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:01 smithi115 bash[46934]: audit 2022-04-22T22:53:00.722920+0000 mon.a (mon.0) 15465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:03.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:02.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: cluster 2022-04-22T22:53:02.285997+0000 mgr.y (mgr.34107) 11145 : cluster [DBG] pgmap v11748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: audit 2022-04-22T22:53:02.329001+0000 mon.a (mon.0) 15466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:53:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: audit 2022-04-22T22:53:02.332097+0000 mon.a (mon.0) 15467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:53:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: audit 2022-04-22T22:53:02.333817+0000 mon.a (mon.0) 15468 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:53:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: audit 2022-04-22T22:53:02.343187+0000 mon.a (mon.0) 15469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:03 smithi121 bash[36902]: audit 2022-04-22T22:53:02.676899+0000 mon.a (mon.0) 15470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:53:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: cluster 2022-04-22T22:53:02.285997+0000 mgr.y (mgr.34107) 11145 : cluster [DBG] pgmap v11748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: audit 2022-04-22T22:53:02.329001+0000 mon.a (mon.0) 15466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: audit 2022-04-22T22:53:02.332097+0000 mon.a (mon.0) 15467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: audit 2022-04-22T22:53:02.333817+0000 mon.a (mon.0) 15468 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: audit 2022-04-22T22:53:02.343187+0000 mon.a (mon.0) 15469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[45495]: audit 2022-04-22T22:53:02.676899+0000 mon.a (mon.0) 15470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: cluster 2022-04-22T22:53:02.285997+0000 mgr.y (mgr.34107) 11145 : cluster [DBG] pgmap v11748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: audit 2022-04-22T22:53:02.329001+0000 mon.a (mon.0) 15466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:53:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: audit 2022-04-22T22:53:02.332097+0000 mon.a (mon.0) 15467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:53:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: audit 2022-04-22T22:53:02.333817+0000 mon.a (mon.0) 15468 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:53:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: audit 2022-04-22T22:53:02.343187+0000 mon.a (mon.0) 15469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:03 smithi115 bash[46934]: audit 2022-04-22T22:53:02.676899+0000 mon.a (mon.0) 15470 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:53:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:04 smithi121 bash[36902]: audit 2022-04-22T22:53:03.848014+0000 mon.a (mon.0) 15471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:04 smithi115 bash[45495]: audit 2022-04-22T22:53:03.848014+0000 mon.a (mon.0) 15471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:04 smithi115 bash[46934]: audit 2022-04-22T22:53:03.848014+0000 mon.a (mon.0) 15471 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:04.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:05 smithi121 bash[36902]: cluster 2022-04-22T22:53:04.287064+0000 mgr.y (mgr.34107) 11146 : cluster [DBG] pgmap v11749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:05 smithi115 bash[45495]: cluster 2022-04-22T22:53:04.287064+0000 mgr.y (mgr.34107) 11146 : cluster [DBG] pgmap v11749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:05 smithi115 bash[46934]: cluster 2022-04-22T22:53:04.287064+0000 mgr.y (mgr.34107) 11146 : cluster [DBG] pgmap v11749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:06 smithi121 bash[36902]: audit 2022-04-22T22:53:05.726041+0000 mon.a (mon.0) 15472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:06 smithi115 bash[45495]: audit 2022-04-22T22:53:05.726041+0000 mon.a (mon.0) 15472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:06 smithi115 bash[46934]: audit 2022-04-22T22:53:05.726041+0000 mon.a (mon.0) 15472 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:07 smithi121 bash[36902]: cluster 2022-04-22T22:53:06.288233+0000 mgr.y (mgr.34107) 11147 : cluster [DBG] pgmap v11750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:07 smithi115 bash[45495]: cluster 2022-04-22T22:53:06.288233+0000 mgr.y (mgr.34107) 11147 : cluster [DBG] pgmap v11750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:07 smithi115 bash[46934]: cluster 2022-04-22T22:53:06.288233+0000 mgr.y (mgr.34107) 11147 : cluster [DBG] pgmap v11750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:08 smithi121 bash[36902]: audit 2022-04-22T22:53:07.346287+0000 mon.a (mon.0) 15473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:08 smithi115 bash[45495]: audit 2022-04-22T22:53:07.346287+0000 mon.a (mon.0) 15473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:08 smithi115 bash[46934]: audit 2022-04-22T22:53:07.346287+0000 mon.a (mon.0) 15473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:09 smithi121 bash[36902]: cluster 2022-04-22T22:53:08.289357+0000 mgr.y (mgr.34107) 11148 : cluster [DBG] pgmap v11751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:09 smithi121 bash[36902]: audit 2022-04-22T22:53:08.851773+0000 mon.a (mon.0) 15474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:09 smithi115 bash[45495]: cluster 2022-04-22T22:53:08.289357+0000 mgr.y (mgr.34107) 11148 : cluster [DBG] pgmap v11751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:09 smithi115 bash[45495]: audit 2022-04-22T22:53:08.851773+0000 mon.a (mon.0) 15474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:09 smithi115 bash[46934]: cluster 2022-04-22T22:53:08.289357+0000 mgr.y (mgr.34107) 11148 : cluster [DBG] pgmap v11751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:09 smithi115 bash[46934]: audit 2022-04-22T22:53:08.851773+0000 mon.a (mon.0) 15474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:11 smithi121 bash[36902]: cluster 2022-04-22T22:53:10.290397+0000 mgr.y (mgr.34107) 11149 : cluster [DBG] pgmap v11752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:11 smithi121 bash[36902]: audit 2022-04-22T22:53:10.729176+0000 mon.a (mon.0) 15475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:11 smithi115 bash[45495]: cluster 2022-04-22T22:53:10.290397+0000 mgr.y (mgr.34107) 11149 : cluster [DBG] pgmap v11752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:11 smithi115 bash[45495]: audit 2022-04-22T22:53:10.729176+0000 mon.a (mon.0) 15475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:11 smithi115 bash[46934]: cluster 2022-04-22T22:53:10.290397+0000 mgr.y (mgr.34107) 11149 : cluster [DBG] pgmap v11752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:11 smithi115 bash[46934]: audit 2022-04-22T22:53:10.729176+0000 mon.a (mon.0) 15475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:12.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:13 smithi115 bash[45495]: cluster 2022-04-22T22:53:12.291306+0000 mgr.y (mgr.34107) 11150 : cluster [DBG] pgmap v11753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:13 smithi115 bash[45495]: audit 2022-04-22T22:53:12.349283+0000 mon.a (mon.0) 15476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:13 smithi115 bash[46934]: cluster 2022-04-22T22:53:12.291306+0000 mgr.y (mgr.34107) 11150 : cluster [DBG] pgmap v11753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:13 smithi115 bash[46934]: audit 2022-04-22T22:53:12.349283+0000 mon.a (mon.0) 15476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:13 smithi121 bash[36902]: cluster 2022-04-22T22:53:12.291306+0000 mgr.y (mgr.34107) 11150 : cluster [DBG] pgmap v11753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:13 smithi121 bash[36902]: audit 2022-04-22T22:53:12.349283+0000 mon.a (mon.0) 15476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:14 smithi121 bash[36902]: audit 2022-04-22T22:53:13.855644+0000 mon.a (mon.0) 15477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:14 smithi115 bash[45495]: audit 2022-04-22T22:53:13.855644+0000 mon.a (mon.0) 15477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:14 smithi115 bash[46934]: audit 2022-04-22T22:53:13.855644+0000 mon.a (mon.0) 15477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:14.762Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:15 smithi121 bash[36902]: cluster 2022-04-22T22:53:14.292435+0000 mgr.y (mgr.34107) 11151 : cluster [DBG] pgmap v11754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:15 smithi115 bash[45495]: cluster 2022-04-22T22:53:14.292435+0000 mgr.y (mgr.34107) 11151 : cluster [DBG] pgmap v11754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:15 smithi115 bash[46934]: cluster 2022-04-22T22:53:14.292435+0000 mgr.y (mgr.34107) 11151 : cluster [DBG] pgmap v11754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:16 smithi121 bash[36902]: audit 2022-04-22T22:53:15.732319+0000 mon.a (mon.0) 15478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:16 smithi115 bash[45495]: audit 2022-04-22T22:53:15.732319+0000 mon.a (mon.0) 15478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:16 smithi115 bash[46934]: audit 2022-04-22T22:53:15.732319+0000 mon.a (mon.0) 15478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:17 smithi121 bash[36902]: cluster 2022-04-22T22:53:16.293540+0000 mgr.y (mgr.34107) 11152 : cluster [DBG] pgmap v11755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:17 smithi121 bash[36902]: audit 2022-04-22T22:53:17.034927+0000 mon.a (mon.0) 15479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:53:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:17 smithi121 bash[36902]: audit 2022-04-22T22:53:17.035179+0000 mon.a (mon.0) 15480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:53:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[45495]: cluster 2022-04-22T22:53:16.293540+0000 mgr.y (mgr.34107) 11152 : cluster [DBG] pgmap v11755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[45495]: audit 2022-04-22T22:53:17.034927+0000 mon.a (mon.0) 15479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:53:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[45495]: audit 2022-04-22T22:53:17.035179+0000 mon.a (mon.0) 15480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:53:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[46934]: cluster 2022-04-22T22:53:16.293540+0000 mgr.y (mgr.34107) 11152 : cluster [DBG] pgmap v11755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[46934]: audit 2022-04-22T22:53:17.034927+0000 mon.a (mon.0) 15479 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:53:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:17 smithi115 bash[46934]: audit 2022-04-22T22:53:17.035179+0000 mon.a (mon.0) 15480 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:53:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:18 smithi121 bash[36902]: audit 2022-04-22T22:53:17.352252+0000 mon.a (mon.0) 15481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:18 smithi115 bash[45495]: audit 2022-04-22T22:53:17.352252+0000 mon.a (mon.0) 15481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:18 smithi115 bash[46934]: audit 2022-04-22T22:53:17.352252+0000 mon.a (mon.0) 15481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:19 smithi121 bash[36902]: cluster 2022-04-22T22:53:18.294714+0000 mgr.y (mgr.34107) 11153 : cluster [DBG] pgmap v11756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:19 smithi121 bash[36902]: audit 2022-04-22T22:53:18.859415+0000 mon.a (mon.0) 15482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:19 smithi115 bash[45495]: cluster 2022-04-22T22:53:18.294714+0000 mgr.y (mgr.34107) 11153 : cluster [DBG] pgmap v11756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:19 smithi115 bash[45495]: audit 2022-04-22T22:53:18.859415+0000 mon.a (mon.0) 15482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:19 smithi115 bash[46934]: cluster 2022-04-22T22:53:18.294714+0000 mgr.y (mgr.34107) 11153 : cluster [DBG] pgmap v11756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:19 smithi115 bash[46934]: audit 2022-04-22T22:53:18.859415+0000 mon.a (mon.0) 15482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:21 smithi121 bash[36902]: cluster 2022-04-22T22:53:20.295824+0000 mgr.y (mgr.34107) 11154 : cluster [DBG] pgmap v11757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:21 smithi121 bash[36902]: audit 2022-04-22T22:53:20.735517+0000 mon.a (mon.0) 15483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:21 smithi115 bash[45495]: cluster 2022-04-22T22:53:20.295824+0000 mgr.y (mgr.34107) 11154 : cluster [DBG] pgmap v11757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:21 smithi115 bash[45495]: audit 2022-04-22T22:53:20.735517+0000 mon.a (mon.0) 15483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:21 smithi115 bash[46934]: cluster 2022-04-22T22:53:20.295824+0000 mgr.y (mgr.34107) 11154 : cluster [DBG] pgmap v11757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:21 smithi115 bash[46934]: audit 2022-04-22T22:53:20.735517+0000 mon.a (mon.0) 15483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:22.902Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:23 smithi115 bash[45495]: cluster 2022-04-22T22:53:22.296964+0000 mgr.y (mgr.34107) 11155 : cluster [DBG] pgmap v11758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:23 smithi115 bash[45495]: audit 2022-04-22T22:53:22.355387+0000 mon.a (mon.0) 15484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:23 smithi115 bash[46934]: cluster 2022-04-22T22:53:22.296964+0000 mgr.y (mgr.34107) 11155 : cluster [DBG] pgmap v11758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:23 smithi115 bash[46934]: audit 2022-04-22T22:53:22.355387+0000 mon.a (mon.0) 15484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:23 smithi121 bash[36902]: cluster 2022-04-22T22:53:22.296964+0000 mgr.y (mgr.34107) 11155 : cluster [DBG] pgmap v11758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:23 smithi121 bash[36902]: audit 2022-04-22T22:53:22.355387+0000 mon.a (mon.0) 15484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:24 smithi121 bash[36902]: audit 2022-04-22T22:53:23.863044+0000 mon.a (mon.0) 15485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:24 smithi115 bash[45495]: audit 2022-04-22T22:53:23.863044+0000 mon.a (mon.0) 15485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:24 smithi115 bash[46934]: audit 2022-04-22T22:53:23.863044+0000 mon.a (mon.0) 15485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:24.761Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:25 smithi121 bash[36902]: cluster 2022-04-22T22:53:24.298100+0000 mgr.y (mgr.34107) 11156 : cluster [DBG] pgmap v11759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:25 smithi115 bash[45495]: cluster 2022-04-22T22:53:24.298100+0000 mgr.y (mgr.34107) 11156 : cluster [DBG] pgmap v11759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:25 smithi115 bash[46934]: cluster 2022-04-22T22:53:24.298100+0000 mgr.y (mgr.34107) 11156 : cluster [DBG] pgmap v11759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:26 smithi121 bash[36902]: audit 2022-04-22T22:53:25.738570+0000 mon.a (mon.0) 15486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:26 smithi115 bash[45495]: audit 2022-04-22T22:53:25.738570+0000 mon.a (mon.0) 15486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:26 smithi115 bash[46934]: audit 2022-04-22T22:53:25.738570+0000 mon.a (mon.0) 15486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:27 smithi121 bash[36902]: cluster 2022-04-22T22:53:26.299112+0000 mgr.y (mgr.34107) 11157 : cluster [DBG] pgmap v11760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:27 smithi115 bash[45495]: cluster 2022-04-22T22:53:26.299112+0000 mgr.y (mgr.34107) 11157 : cluster [DBG] pgmap v11760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:27 smithi115 bash[46934]: cluster 2022-04-22T22:53:26.299112+0000 mgr.y (mgr.34107) 11157 : cluster [DBG] pgmap v11760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:28 smithi121 bash[36902]: audit 2022-04-22T22:53:27.359027+0000 mon.a (mon.0) 15487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:28.516 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:53:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:28 smithi115 bash[45495]: audit 2022-04-22T22:53:27.359027+0000 mon.a (mon.0) 15487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:28 smithi115 bash[46934]: audit 2022-04-22T22:53:27.359027+0000 mon.a (mon.0) 15487 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:53:28.857 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:53:28.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:53:28.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:53:28.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:53:28.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:53:28.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:53:28.859 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:53:29.271 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:53:29.272 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:53:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:29 smithi121 bash[36902]: cluster 2022-04-22T22:53:28.300119+0000 mgr.y (mgr.34107) 11158 : cluster [DBG] pgmap v11761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:29 smithi121 bash[36902]: audit 2022-04-22T22:53:28.498758+0000 mgr.y (mgr.34107) 11159 : audit [DBG] from='client.46902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:29 smithi121 bash[36902]: audit 2022-04-22T22:53:28.839420+0000 mgr.y (mgr.34107) 11160 : audit [DBG] from='client.56845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:29 smithi121 bash[36902]: audit 2022-04-22T22:53:28.866844+0000 mon.a (mon.0) 15488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[45495]: cluster 2022-04-22T22:53:28.300119+0000 mgr.y (mgr.34107) 11158 : cluster [DBG] pgmap v11761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[45495]: audit 2022-04-22T22:53:28.498758+0000 mgr.y (mgr.34107) 11159 : audit [DBG] from='client.46902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[45495]: audit 2022-04-22T22:53:28.839420+0000 mgr.y (mgr.34107) 11160 : audit [DBG] from='client.56845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[45495]: audit 2022-04-22T22:53:28.866844+0000 mon.a (mon.0) 15488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[46934]: cluster 2022-04-22T22:53:28.300119+0000 mgr.y (mgr.34107) 11158 : cluster [DBG] pgmap v11761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[46934]: audit 2022-04-22T22:53:28.498758+0000 mgr.y (mgr.34107) 11159 : audit [DBG] from='client.46902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[46934]: audit 2022-04-22T22:53:28.839420+0000 mgr.y (mgr.34107) 11160 : audit [DBG] from='client.56845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:53:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:29 smithi115 bash[46934]: audit 2022-04-22T22:53:28.866844+0000 mon.a (mon.0) 15488 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:30.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:30 smithi121 bash[36902]: audit 2022-04-22T22:53:29.267235+0000 mon.a (mon.0) 15489 : audit [DBG] from='client.? 172.21.15.115:0/1848563708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:53:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:30 smithi115 bash[45495]: audit 2022-04-22T22:53:29.267235+0000 mon.a (mon.0) 15489 : audit [DBG] from='client.? 172.21.15.115:0/1848563708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:53:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:30 smithi115 bash[46934]: audit 2022-04-22T22:53:29.267235+0000 mon.a (mon.0) 15489 : audit [DBG] from='client.? 172.21.15.115:0/1848563708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:53:31.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:31 smithi121 bash[36902]: cluster 2022-04-22T22:53:30.301333+0000 mgr.y (mgr.34107) 11161 : cluster [DBG] pgmap v11762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:31 smithi121 bash[36902]: audit 2022-04-22T22:53:30.741500+0000 mon.a (mon.0) 15490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:31 smithi115 bash[45495]: cluster 2022-04-22T22:53:30.301333+0000 mgr.y (mgr.34107) 11161 : cluster [DBG] pgmap v11762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:31 smithi115 bash[45495]: audit 2022-04-22T22:53:30.741500+0000 mon.a (mon.0) 15490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:31 smithi115 bash[46934]: cluster 2022-04-22T22:53:30.301333+0000 mgr.y (mgr.34107) 11161 : cluster [DBG] pgmap v11762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:31 smithi115 bash[46934]: audit 2022-04-22T22:53:30.741500+0000 mon.a (mon.0) 15490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:32.902Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:33 smithi115 bash[45495]: cluster 2022-04-22T22:53:32.302412+0000 mgr.y (mgr.34107) 11162 : cluster [DBG] pgmap v11763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:33 smithi115 bash[45495]: audit 2022-04-22T22:53:32.361936+0000 mon.a (mon.0) 15491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:33 smithi115 bash[46934]: cluster 2022-04-22T22:53:32.302412+0000 mgr.y (mgr.34107) 11162 : cluster [DBG] pgmap v11763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:33 smithi115 bash[46934]: audit 2022-04-22T22:53:32.361936+0000 mon.a (mon.0) 15491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:33 smithi121 bash[36902]: cluster 2022-04-22T22:53:32.302412+0000 mgr.y (mgr.34107) 11162 : cluster [DBG] pgmap v11763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:33 smithi121 bash[36902]: audit 2022-04-22T22:53:32.361936+0000 mon.a (mon.0) 15491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:34 smithi121 bash[36902]: audit 2022-04-22T22:53:33.870302+0000 mon.a (mon.0) 15492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:34 smithi115 bash[45495]: audit 2022-04-22T22:53:33.870302+0000 mon.a (mon.0) 15492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:34 smithi115 bash[46934]: audit 2022-04-22T22:53:33.870302+0000 mon.a (mon.0) 15492 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:34.762Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:35 smithi121 bash[36902]: cluster 2022-04-22T22:53:34.303520+0000 mgr.y (mgr.34107) 11163 : cluster [DBG] pgmap v11764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:35 smithi115 bash[45495]: cluster 2022-04-22T22:53:34.303520+0000 mgr.y (mgr.34107) 11163 : cluster [DBG] pgmap v11764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:35 smithi115 bash[46934]: cluster 2022-04-22T22:53:34.303520+0000 mgr.y (mgr.34107) 11163 : cluster [DBG] pgmap v11764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:36 smithi121 bash[36902]: audit 2022-04-22T22:53:35.744685+0000 mon.a (mon.0) 15493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:36 smithi115 bash[45495]: audit 2022-04-22T22:53:35.744685+0000 mon.a (mon.0) 15493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:36 smithi115 bash[46934]: audit 2022-04-22T22:53:35.744685+0000 mon.a (mon.0) 15493 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:37 smithi121 bash[36902]: cluster 2022-04-22T22:53:36.304563+0000 mgr.y (mgr.34107) 11164 : cluster [DBG] pgmap v11765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:37 smithi115 bash[45495]: cluster 2022-04-22T22:53:36.304563+0000 mgr.y (mgr.34107) 11164 : cluster [DBG] pgmap v11765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:37 smithi115 bash[46934]: cluster 2022-04-22T22:53:36.304563+0000 mgr.y (mgr.34107) 11164 : cluster [DBG] pgmap v11765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:38 smithi121 bash[36902]: audit 2022-04-22T22:53:37.364956+0000 mon.a (mon.0) 15494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:38 smithi115 bash[45495]: audit 2022-04-22T22:53:37.364956+0000 mon.a (mon.0) 15494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:38 smithi115 bash[46934]: audit 2022-04-22T22:53:37.364956+0000 mon.a (mon.0) 15494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:39 smithi121 bash[36902]: cluster 2022-04-22T22:53:38.305669+0000 mgr.y (mgr.34107) 11165 : cluster [DBG] pgmap v11766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:39 smithi121 bash[36902]: audit 2022-04-22T22:53:38.874082+0000 mon.a (mon.0) 15495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:39 smithi115 bash[45495]: cluster 2022-04-22T22:53:38.305669+0000 mgr.y (mgr.34107) 11165 : cluster [DBG] pgmap v11766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:39 smithi115 bash[45495]: audit 2022-04-22T22:53:38.874082+0000 mon.a (mon.0) 15495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:39 smithi115 bash[46934]: cluster 2022-04-22T22:53:38.305669+0000 mgr.y (mgr.34107) 11165 : cluster [DBG] pgmap v11766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:39 smithi115 bash[46934]: audit 2022-04-22T22:53:38.874082+0000 mon.a (mon.0) 15495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:41 smithi121 bash[36902]: cluster 2022-04-22T22:53:40.306725+0000 mgr.y (mgr.34107) 11166 : cluster [DBG] pgmap v11767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:41 smithi121 bash[36902]: audit 2022-04-22T22:53:40.747654+0000 mon.a (mon.0) 15496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:41 smithi115 bash[45495]: cluster 2022-04-22T22:53:40.306725+0000 mgr.y (mgr.34107) 11166 : cluster [DBG] pgmap v11767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:41 smithi115 bash[45495]: audit 2022-04-22T22:53:40.747654+0000 mon.a (mon.0) 15496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:41 smithi115 bash[46934]: cluster 2022-04-22T22:53:40.306725+0000 mgr.y (mgr.34107) 11166 : cluster [DBG] pgmap v11767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:41 smithi115 bash[46934]: audit 2022-04-22T22:53:40.747654+0000 mon.a (mon.0) 15496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:42.902Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:43 smithi115 bash[45495]: cluster 2022-04-22T22:53:42.307814+0000 mgr.y (mgr.34107) 11167 : cluster [DBG] pgmap v11768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:43 smithi115 bash[45495]: audit 2022-04-22T22:53:42.368171+0000 mon.a (mon.0) 15497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:43 smithi115 bash[46934]: cluster 2022-04-22T22:53:42.307814+0000 mgr.y (mgr.34107) 11167 : cluster [DBG] pgmap v11768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:43 smithi115 bash[46934]: audit 2022-04-22T22:53:42.368171+0000 mon.a (mon.0) 15497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:43 smithi121 bash[36902]: cluster 2022-04-22T22:53:42.307814+0000 mgr.y (mgr.34107) 11167 : cluster [DBG] pgmap v11768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:43 smithi121 bash[36902]: audit 2022-04-22T22:53:42.368171+0000 mon.a (mon.0) 15497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:44 smithi121 bash[36902]: audit 2022-04-22T22:53:43.877881+0000 mon.a (mon.0) 15498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:44 smithi115 bash[45495]: audit 2022-04-22T22:53:43.877881+0000 mon.a (mon.0) 15498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:44 smithi115 bash[46934]: audit 2022-04-22T22:53:43.877881+0000 mon.a (mon.0) 15498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:44.762Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:45 smithi121 bash[36902]: cluster 2022-04-22T22:53:44.308974+0000 mgr.y (mgr.34107) 11168 : cluster [DBG] pgmap v11769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:45 smithi115 bash[45495]: cluster 2022-04-22T22:53:44.308974+0000 mgr.y (mgr.34107) 11168 : cluster [DBG] pgmap v11769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:45 smithi115 bash[46934]: cluster 2022-04-22T22:53:44.308974+0000 mgr.y (mgr.34107) 11168 : cluster [DBG] pgmap v11769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:46 smithi121 bash[36902]: audit 2022-04-22T22:53:45.750606+0000 mon.a (mon.0) 15499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:46 smithi115 bash[45495]: audit 2022-04-22T22:53:45.750606+0000 mon.a (mon.0) 15499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:46 smithi115 bash[46934]: audit 2022-04-22T22:53:45.750606+0000 mon.a (mon.0) 15499 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:47 smithi121 bash[36902]: cluster 2022-04-22T22:53:46.310074+0000 mgr.y (mgr.34107) 11169 : cluster [DBG] pgmap v11770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:47 smithi115 bash[45495]: cluster 2022-04-22T22:53:46.310074+0000 mgr.y (mgr.34107) 11169 : cluster [DBG] pgmap v11770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:47 smithi115 bash[46934]: cluster 2022-04-22T22:53:46.310074+0000 mgr.y (mgr.34107) 11169 : cluster [DBG] pgmap v11770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:48 smithi121 bash[36902]: audit 2022-04-22T22:53:47.371092+0000 mon.a (mon.0) 15500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:48 smithi115 bash[45495]: audit 2022-04-22T22:53:47.371092+0000 mon.a (mon.0) 15500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:48 smithi115 bash[46934]: audit 2022-04-22T22:53:47.371092+0000 mon.a (mon.0) 15500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:49 smithi121 bash[36902]: cluster 2022-04-22T22:53:48.311216+0000 mgr.y (mgr.34107) 11170 : cluster [DBG] pgmap v11771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:49 smithi121 bash[36902]: audit 2022-04-22T22:53:48.881421+0000 mon.a (mon.0) 15501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:49 smithi115 bash[45495]: cluster 2022-04-22T22:53:48.311216+0000 mgr.y (mgr.34107) 11170 : cluster [DBG] pgmap v11771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:49 smithi115 bash[45495]: audit 2022-04-22T22:53:48.881421+0000 mon.a (mon.0) 15501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:49 smithi115 bash[46934]: cluster 2022-04-22T22:53:48.311216+0000 mgr.y (mgr.34107) 11170 : cluster [DBG] pgmap v11771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:49 smithi115 bash[46934]: audit 2022-04-22T22:53:48.881421+0000 mon.a (mon.0) 15501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:51 smithi121 bash[36902]: cluster 2022-04-22T22:53:50.312343+0000 mgr.y (mgr.34107) 11171 : cluster [DBG] pgmap v11772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:51 smithi121 bash[36902]: audit 2022-04-22T22:53:50.753479+0000 mon.a (mon.0) 15502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:51 smithi115 bash[45495]: cluster 2022-04-22T22:53:50.312343+0000 mgr.y (mgr.34107) 11171 : cluster [DBG] pgmap v11772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:51 smithi115 bash[45495]: audit 2022-04-22T22:53:50.753479+0000 mon.a (mon.0) 15502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:51 smithi115 bash[46934]: cluster 2022-04-22T22:53:50.312343+0000 mgr.y (mgr.34107) 11171 : cluster [DBG] pgmap v11772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:51 smithi115 bash[46934]: audit 2022-04-22T22:53:50.753479+0000 mon.a (mon.0) 15502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:53 smithi115 bash[45495]: cluster 2022-04-22T22:53:52.313394+0000 mgr.y (mgr.34107) 11172 : cluster [DBG] pgmap v11773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:53 smithi115 bash[45495]: audit 2022-04-22T22:53:52.374023+0000 mon.a (mon.0) 15503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:52.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:53 smithi115 bash[46934]: cluster 2022-04-22T22:53:52.313394+0000 mgr.y (mgr.34107) 11172 : cluster [DBG] pgmap v11773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:53 smithi115 bash[46934]: audit 2022-04-22T22:53:52.374023+0000 mon.a (mon.0) 15503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:53 smithi121 bash[36902]: cluster 2022-04-22T22:53:52.313394+0000 mgr.y (mgr.34107) 11172 : cluster [DBG] pgmap v11773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:53 smithi121 bash[36902]: audit 2022-04-22T22:53:52.374023+0000 mon.a (mon.0) 15503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:54 smithi121 bash[36902]: audit 2022-04-22T22:53:53.884708+0000 mon.a (mon.0) 15504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:54 smithi115 bash[45495]: audit 2022-04-22T22:53:53.884708+0000 mon.a (mon.0) 15504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:54 smithi115 bash[46934]: audit 2022-04-22T22:53:53.884708+0000 mon.a (mon.0) 15504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:53:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:53:54.764Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:53:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:55 smithi121 bash[36902]: cluster 2022-04-22T22:53:54.314456+0000 mgr.y (mgr.34107) 11173 : cluster [DBG] pgmap v11774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:55 smithi115 bash[45495]: cluster 2022-04-22T22:53:54.314456+0000 mgr.y (mgr.34107) 11173 : cluster [DBG] pgmap v11774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:55 smithi115 bash[46934]: cluster 2022-04-22T22:53:54.314456+0000 mgr.y (mgr.34107) 11173 : cluster [DBG] pgmap v11774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:56 smithi121 bash[36902]: audit 2022-04-22T22:53:55.756705+0000 mon.a (mon.0) 15505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:56 smithi115 bash[45495]: audit 2022-04-22T22:53:55.756705+0000 mon.a (mon.0) 15505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:56 smithi115 bash[46934]: audit 2022-04-22T22:53:55.756705+0000 mon.a (mon.0) 15505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:53:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:57 smithi121 bash[36902]: cluster 2022-04-22T22:53:56.315592+0000 mgr.y (mgr.34107) 11174 : cluster [DBG] pgmap v11775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:57 smithi115 bash[45495]: cluster 2022-04-22T22:53:56.315592+0000 mgr.y (mgr.34107) 11174 : cluster [DBG] pgmap v11775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:57 smithi115 bash[46934]: cluster 2022-04-22T22:53:56.315592+0000 mgr.y (mgr.34107) 11174 : cluster [DBG] pgmap v11775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:58 smithi121 bash[36902]: audit 2022-04-22T22:53:57.376987+0000 mon.a (mon.0) 15506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:58 smithi115 bash[45495]: audit 2022-04-22T22:53:57.376987+0000 mon.a (mon.0) 15506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:58 smithi115 bash[46934]: audit 2022-04-22T22:53:57.376987+0000 mon.a (mon.0) 15506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:53:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:59 smithi121 bash[36902]: cluster 2022-04-22T22:53:58.316742+0000 mgr.y (mgr.34107) 11175 : cluster [DBG] pgmap v11776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:53:59 smithi121 bash[36902]: audit 2022-04-22T22:53:58.888512+0000 mon.a (mon.0) 15507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:59 smithi115 bash[45495]: cluster 2022-04-22T22:53:58.316742+0000 mgr.y (mgr.34107) 11175 : cluster [DBG] pgmap v11776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:53:59 smithi115 bash[45495]: audit 2022-04-22T22:53:58.888512+0000 mon.a (mon.0) 15507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:59 smithi115 bash[46934]: cluster 2022-04-22T22:53:58.316742+0000 mgr.y (mgr.34107) 11175 : cluster [DBG] pgmap v11776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:53:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:53:59 smithi115 bash[46934]: audit 2022-04-22T22:53:58.888512+0000 mon.a (mon.0) 15507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:53:59.651 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:54:00.268 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:54:00.269 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:54:00.270 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:54:00.270 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:54:00.270 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:54:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:54:00.403 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:54:00.403 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:54:00.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:00.403 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:54:00.404 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:54:00.405 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:00.405 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:54:00.405 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:54:00.405 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:54:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:54:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:00 smithi121 bash[36902]: audit 2022-04-22T22:53:59.630629+0000 mgr.y (mgr.34107) 11176 : audit [DBG] from='client.56857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:00.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:00 smithi121 bash[36902]: audit 2022-04-22T22:53:59.973309+0000 mgr.y (mgr.34107) 11177 : audit [DBG] from='client.46926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:00 smithi115 bash[45495]: audit 2022-04-22T22:53:59.630629+0000 mgr.y (mgr.34107) 11176 : audit [DBG] from='client.56857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:00.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:00 smithi115 bash[45495]: audit 2022-04-22T22:53:59.973309+0000 mgr.y (mgr.34107) 11177 : audit [DBG] from='client.46926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:00 smithi115 bash[46934]: audit 2022-04-22T22:53:59.630629+0000 mgr.y (mgr.34107) 11176 : audit [DBG] from='client.56857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:00 smithi115 bash[46934]: audit 2022-04-22T22:53:59.973309+0000 mgr.y (mgr.34107) 11177 : audit [DBG] from='client.46926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:01 smithi121 bash[36902]: cluster 2022-04-22T22:54:00.317794+0000 mgr.y (mgr.34107) 11178 : cluster [DBG] pgmap v11777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:01 smithi121 bash[36902]: audit 2022-04-22T22:54:00.399540+0000 mon.c (mon.1) 377 : audit [DBG] from='client.? 172.21.15.115:0/2192492438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:01 smithi121 bash[36902]: audit 2022-04-22T22:54:00.759843+0000 mon.a (mon.0) 15508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[45495]: cluster 2022-04-22T22:54:00.317794+0000 mgr.y (mgr.34107) 11178 : cluster [DBG] pgmap v11777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[45495]: audit 2022-04-22T22:54:00.399540+0000 mon.c (mon.1) 377 : audit [DBG] from='client.? 172.21.15.115:0/2192492438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[45495]: audit 2022-04-22T22:54:00.759843+0000 mon.a (mon.0) 15508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[46934]: cluster 2022-04-22T22:54:00.317794+0000 mgr.y (mgr.34107) 11178 : cluster [DBG] pgmap v11777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[46934]: audit 2022-04-22T22:54:00.399540+0000 mon.c (mon.1) 377 : audit [DBG] from='client.? 172.21.15.115:0/2192492438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:01 smithi115 bash[46934]: audit 2022-04-22T22:54:00.759843+0000 mon.a (mon.0) 15508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:03.034 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:02.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: cluster 2022-04-22T22:54:02.318960+0000 mgr.y (mgr.34107) 11179 : cluster [DBG] pgmap v11778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: audit 2022-04-22T22:54:02.380257+0000 mon.a (mon.0) 15509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: audit 2022-04-22T22:54:02.684198+0000 mon.a (mon.0) 15510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: audit 2022-04-22T22:54:02.686989+0000 mon.a (mon.0) 15511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: audit 2022-04-22T22:54:02.688607+0000 mon.a (mon.0) 15512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:54:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[45495]: audit 2022-04-22T22:54:03.039926+0000 mon.a (mon.0) 15513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: cluster 2022-04-22T22:54:02.318960+0000 mgr.y (mgr.34107) 11179 : cluster [DBG] pgmap v11778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: audit 2022-04-22T22:54:02.380257+0000 mon.a (mon.0) 15509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: audit 2022-04-22T22:54:02.684198+0000 mon.a (mon.0) 15510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: audit 2022-04-22T22:54:02.686989+0000 mon.a (mon.0) 15511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: audit 2022-04-22T22:54:02.688607+0000 mon.a (mon.0) 15512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:54:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:03 smithi115 bash[46934]: audit 2022-04-22T22:54:03.039926+0000 mon.a (mon.0) 15513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:54:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: cluster 2022-04-22T22:54:02.318960+0000 mgr.y (mgr.34107) 11179 : cluster [DBG] pgmap v11778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: audit 2022-04-22T22:54:02.380257+0000 mon.a (mon.0) 15509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: audit 2022-04-22T22:54:02.684198+0000 mon.a (mon.0) 15510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:54:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: audit 2022-04-22T22:54:02.686989+0000 mon.a (mon.0) 15511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:54:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: audit 2022-04-22T22:54:02.688607+0000 mon.a (mon.0) 15512 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:54:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:03 smithi121 bash[36902]: audit 2022-04-22T22:54:03.039926+0000 mon.a (mon.0) 15513 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:54:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:04 smithi121 bash[36902]: audit 2022-04-22T22:54:03.892205+0000 mon.a (mon.0) 15514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:04 smithi115 bash[45495]: audit 2022-04-22T22:54:03.892205+0000 mon.a (mon.0) 15514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:04 smithi115 bash[46934]: audit 2022-04-22T22:54:03.892205+0000 mon.a (mon.0) 15514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:04.764Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:05 smithi121 bash[36902]: cluster 2022-04-22T22:54:04.320103+0000 mgr.y (mgr.34107) 11180 : cluster [DBG] pgmap v11779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:05 smithi115 bash[45495]: cluster 2022-04-22T22:54:04.320103+0000 mgr.y (mgr.34107) 11180 : cluster [DBG] pgmap v11779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:05 smithi115 bash[46934]: cluster 2022-04-22T22:54:04.320103+0000 mgr.y (mgr.34107) 11180 : cluster [DBG] pgmap v11779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:06 smithi121 bash[36902]: audit 2022-04-22T22:54:05.762725+0000 mon.a (mon.0) 15515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:06 smithi115 bash[45495]: audit 2022-04-22T22:54:05.762725+0000 mon.a (mon.0) 15515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:06 smithi115 bash[46934]: audit 2022-04-22T22:54:05.762725+0000 mon.a (mon.0) 15515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:07 smithi121 bash[36902]: cluster 2022-04-22T22:54:06.321227+0000 mgr.y (mgr.34107) 11181 : cluster [DBG] pgmap v11780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:07 smithi115 bash[45495]: cluster 2022-04-22T22:54:06.321227+0000 mgr.y (mgr.34107) 11181 : cluster [DBG] pgmap v11780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:07 smithi115 bash[46934]: cluster 2022-04-22T22:54:06.321227+0000 mgr.y (mgr.34107) 11181 : cluster [DBG] pgmap v11780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:08 smithi121 bash[36902]: audit 2022-04-22T22:54:07.383345+0000 mon.a (mon.0) 15516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:08 smithi115 bash[45495]: audit 2022-04-22T22:54:07.383345+0000 mon.a (mon.0) 15516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:08 smithi115 bash[46934]: audit 2022-04-22T22:54:07.383345+0000 mon.a (mon.0) 15516 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:09 smithi121 bash[36902]: cluster 2022-04-22T22:54:08.322345+0000 mgr.y (mgr.34107) 11182 : cluster [DBG] pgmap v11781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:09 smithi121 bash[36902]: audit 2022-04-22T22:54:08.895901+0000 mon.a (mon.0) 15517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:09 smithi115 bash[45495]: cluster 2022-04-22T22:54:08.322345+0000 mgr.y (mgr.34107) 11182 : cluster [DBG] pgmap v11781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:09 smithi115 bash[45495]: audit 2022-04-22T22:54:08.895901+0000 mon.a (mon.0) 15517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:09 smithi115 bash[46934]: cluster 2022-04-22T22:54:08.322345+0000 mgr.y (mgr.34107) 11182 : cluster [DBG] pgmap v11781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:09 smithi115 bash[46934]: audit 2022-04-22T22:54:08.895901+0000 mon.a (mon.0) 15517 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:11 smithi121 bash[36902]: cluster 2022-04-22T22:54:10.323443+0000 mgr.y (mgr.34107) 11183 : cluster [DBG] pgmap v11782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:11 smithi121 bash[36902]: audit 2022-04-22T22:54:10.765738+0000 mon.a (mon.0) 15518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:11 smithi115 bash[45495]: cluster 2022-04-22T22:54:10.323443+0000 mgr.y (mgr.34107) 11183 : cluster [DBG] pgmap v11782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:11 smithi115 bash[45495]: audit 2022-04-22T22:54:10.765738+0000 mon.a (mon.0) 15518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:11 smithi115 bash[46934]: cluster 2022-04-22T22:54:10.323443+0000 mgr.y (mgr.34107) 11183 : cluster [DBG] pgmap v11782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:11 smithi115 bash[46934]: audit 2022-04-22T22:54:10.765738+0000 mon.a (mon.0) 15518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:13 smithi115 bash[45495]: cluster 2022-04-22T22:54:12.324512+0000 mgr.y (mgr.34107) 11184 : cluster [DBG] pgmap v11783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:13 smithi115 bash[45495]: audit 2022-04-22T22:54:12.386462+0000 mon.a (mon.0) 15519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:12.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:13 smithi115 bash[46934]: cluster 2022-04-22T22:54:12.324512+0000 mgr.y (mgr.34107) 11184 : cluster [DBG] pgmap v11783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:13 smithi115 bash[46934]: audit 2022-04-22T22:54:12.386462+0000 mon.a (mon.0) 15519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:13 smithi121 bash[36902]: cluster 2022-04-22T22:54:12.324512+0000 mgr.y (mgr.34107) 11184 : cluster [DBG] pgmap v11783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:13 smithi121 bash[36902]: audit 2022-04-22T22:54:12.386462+0000 mon.a (mon.0) 15519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:14 smithi121 bash[36902]: audit 2022-04-22T22:54:13.899815+0000 mon.a (mon.0) 15520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:14 smithi115 bash[45495]: audit 2022-04-22T22:54:13.899815+0000 mon.a (mon.0) 15520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:14 smithi115 bash[46934]: audit 2022-04-22T22:54:13.899815+0000 mon.a (mon.0) 15520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:14.765Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:15 smithi121 bash[36902]: cluster 2022-04-22T22:54:14.325603+0000 mgr.y (mgr.34107) 11185 : cluster [DBG] pgmap v11784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:15 smithi115 bash[45495]: cluster 2022-04-22T22:54:14.325603+0000 mgr.y (mgr.34107) 11185 : cluster [DBG] pgmap v11784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:15 smithi115 bash[46934]: cluster 2022-04-22T22:54:14.325603+0000 mgr.y (mgr.34107) 11185 : cluster [DBG] pgmap v11784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:16 smithi121 bash[36902]: audit 2022-04-22T22:54:15.768927+0000 mon.a (mon.0) 15521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:16 smithi115 bash[45495]: audit 2022-04-22T22:54:15.768927+0000 mon.a (mon.0) 15521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:16 smithi115 bash[46934]: audit 2022-04-22T22:54:15.768927+0000 mon.a (mon.0) 15521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:17 smithi121 bash[36902]: cluster 2022-04-22T22:54:16.326742+0000 mgr.y (mgr.34107) 11186 : cluster [DBG] pgmap v11785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:17 smithi121 bash[36902]: audit 2022-04-22T22:54:17.046784+0000 mon.a (mon.0) 15522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:54:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:17 smithi121 bash[36902]: audit 2022-04-22T22:54:17.047163+0000 mon.a (mon.0) 15523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:54:17.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[46934]: cluster 2022-04-22T22:54:16.326742+0000 mgr.y (mgr.34107) 11186 : cluster [DBG] pgmap v11785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[46934]: audit 2022-04-22T22:54:17.046784+0000 mon.a (mon.0) 15522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:54:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[46934]: audit 2022-04-22T22:54:17.047163+0000 mon.a (mon.0) 15523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[45495]: cluster 2022-04-22T22:54:16.326742+0000 mgr.y (mgr.34107) 11186 : cluster [DBG] pgmap v11785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[45495]: audit 2022-04-22T22:54:17.046784+0000 mon.a (mon.0) 15522 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:54:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:17 smithi115 bash[45495]: audit 2022-04-22T22:54:17.047163+0000 mon.a (mon.0) 15523 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:54:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:18 smithi121 bash[36902]: audit 2022-04-22T22:54:17.389446+0000 mon.a (mon.0) 15524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:18.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:18 smithi115 bash[46934]: audit 2022-04-22T22:54:17.389446+0000 mon.a (mon.0) 15524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:18 smithi115 bash[45495]: audit 2022-04-22T22:54:17.389446+0000 mon.a (mon.0) 15524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:19 smithi121 bash[36902]: cluster 2022-04-22T22:54:18.327984+0000 mgr.y (mgr.34107) 11187 : cluster [DBG] pgmap v11786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:19 smithi121 bash[36902]: audit 2022-04-22T22:54:18.903895+0000 mon.a (mon.0) 15525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:19 smithi115 bash[45495]: cluster 2022-04-22T22:54:18.327984+0000 mgr.y (mgr.34107) 11187 : cluster [DBG] pgmap v11786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:19 smithi115 bash[45495]: audit 2022-04-22T22:54:18.903895+0000 mon.a (mon.0) 15525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:19 smithi115 bash[46934]: cluster 2022-04-22T22:54:18.327984+0000 mgr.y (mgr.34107) 11187 : cluster [DBG] pgmap v11786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:19 smithi115 bash[46934]: audit 2022-04-22T22:54:18.903895+0000 mon.a (mon.0) 15525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:21 smithi121 bash[36902]: cluster 2022-04-22T22:54:20.328991+0000 mgr.y (mgr.34107) 11188 : cluster [DBG] pgmap v11787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:21 smithi121 bash[36902]: audit 2022-04-22T22:54:20.771955+0000 mon.a (mon.0) 15526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:21 smithi115 bash[45495]: cluster 2022-04-22T22:54:20.328991+0000 mgr.y (mgr.34107) 11188 : cluster [DBG] pgmap v11787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:21 smithi115 bash[45495]: audit 2022-04-22T22:54:20.771955+0000 mon.a (mon.0) 15526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:21 smithi115 bash[46934]: cluster 2022-04-22T22:54:20.328991+0000 mgr.y (mgr.34107) 11188 : cluster [DBG] pgmap v11787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:21 smithi115 bash[46934]: audit 2022-04-22T22:54:20.771955+0000 mon.a (mon.0) 15526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:22.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:23 smithi115 bash[46934]: cluster 2022-04-22T22:54:22.330055+0000 mgr.y (mgr.34107) 11189 : cluster [DBG] pgmap v11788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:23 smithi115 bash[46934]: audit 2022-04-22T22:54:22.392542+0000 mon.a (mon.0) 15527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:23.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:23 smithi115 bash[45495]: cluster 2022-04-22T22:54:22.330055+0000 mgr.y (mgr.34107) 11189 : cluster [DBG] pgmap v11788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:23.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:23 smithi115 bash[45495]: audit 2022-04-22T22:54:22.392542+0000 mon.a (mon.0) 15527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:23 smithi121 bash[36902]: cluster 2022-04-22T22:54:22.330055+0000 mgr.y (mgr.34107) 11189 : cluster [DBG] pgmap v11788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:23 smithi121 bash[36902]: audit 2022-04-22T22:54:22.392542+0000 mon.a (mon.0) 15527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:24 smithi121 bash[36902]: audit 2022-04-22T22:54:23.907599+0000 mon.a (mon.0) 15528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:24 smithi115 bash[46934]: audit 2022-04-22T22:54:23.907599+0000 mon.a (mon.0) 15528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:24 smithi115 bash[45495]: audit 2022-04-22T22:54:23.907599+0000 mon.a (mon.0) 15528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:24.766Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:25 smithi121 bash[36902]: cluster 2022-04-22T22:54:24.331110+0000 mgr.y (mgr.34107) 11190 : cluster [DBG] pgmap v11789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:25 smithi115 bash[45495]: cluster 2022-04-22T22:54:24.331110+0000 mgr.y (mgr.34107) 11190 : cluster [DBG] pgmap v11789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:25 smithi115 bash[46934]: cluster 2022-04-22T22:54:24.331110+0000 mgr.y (mgr.34107) 11190 : cluster [DBG] pgmap v11789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:26 smithi121 bash[36902]: audit 2022-04-22T22:54:25.774966+0000 mon.a (mon.0) 15529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:26 smithi115 bash[45495]: audit 2022-04-22T22:54:25.774966+0000 mon.a (mon.0) 15529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:26 smithi115 bash[46934]: audit 2022-04-22T22:54:25.774966+0000 mon.a (mon.0) 15529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:27 smithi121 bash[36902]: cluster 2022-04-22T22:54:26.332286+0000 mgr.y (mgr.34107) 11191 : cluster [DBG] pgmap v11790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:27 smithi115 bash[45495]: cluster 2022-04-22T22:54:26.332286+0000 mgr.y (mgr.34107) 11191 : cluster [DBG] pgmap v11790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:27 smithi115 bash[46934]: cluster 2022-04-22T22:54:26.332286+0000 mgr.y (mgr.34107) 11191 : cluster [DBG] pgmap v11790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:28 smithi121 bash[36902]: audit 2022-04-22T22:54:27.395836+0000 mon.a (mon.0) 15530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:28 smithi115 bash[45495]: audit 2022-04-22T22:54:27.395836+0000 mon.a (mon.0) 15530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:28 smithi115 bash[46934]: audit 2022-04-22T22:54:27.395836+0000 mon.a (mon.0) 15530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:29 smithi121 bash[36902]: cluster 2022-04-22T22:54:28.333572+0000 mgr.y (mgr.34107) 11192 : cluster [DBG] pgmap v11791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:29 smithi121 bash[36902]: audit 2022-04-22T22:54:28.911176+0000 mon.a (mon.0) 15531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:29 smithi115 bash[45495]: cluster 2022-04-22T22:54:28.333572+0000 mgr.y (mgr.34107) 11192 : cluster [DBG] pgmap v11791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:29 smithi115 bash[45495]: audit 2022-04-22T22:54:28.911176+0000 mon.a (mon.0) 15531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:29 smithi115 bash[46934]: cluster 2022-04-22T22:54:28.333572+0000 mgr.y (mgr.34107) 11192 : cluster [DBG] pgmap v11791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:29 smithi115 bash[46934]: audit 2022-04-22T22:54:28.911176+0000 mon.a (mon.0) 15531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:30.772 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:54:31.112 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:54:31.112 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:54:31.112 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:54:31.112 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:54:31.112 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:54:31.113 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:54:31.113 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:54:31.113 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:54:31.113 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:54:31.114 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:54:31.114 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:54:31.114 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:54:31.114 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:54:31.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:54:31.116 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:54:31.116 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:54:31.522 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:31.523 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:54:31.524 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:54:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:31 smithi121 bash[36902]: cluster 2022-04-22T22:54:30.334659+0000 mgr.y (mgr.34107) 11193 : cluster [DBG] pgmap v11792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:31 smithi121 bash[36902]: audit 2022-04-22T22:54:30.754004+0000 mgr.y (mgr.34107) 11194 : audit [DBG] from='client.56875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:31 smithi121 bash[36902]: audit 2022-04-22T22:54:30.777767+0000 mon.a (mon.0) 15532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[46934]: cluster 2022-04-22T22:54:30.334659+0000 mgr.y (mgr.34107) 11193 : cluster [DBG] pgmap v11792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[46934]: audit 2022-04-22T22:54:30.754004+0000 mgr.y (mgr.34107) 11194 : audit [DBG] from='client.56875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[46934]: audit 2022-04-22T22:54:30.777767+0000 mon.a (mon.0) 15532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[45495]: cluster 2022-04-22T22:54:30.334659+0000 mgr.y (mgr.34107) 11193 : cluster [DBG] pgmap v11792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[45495]: audit 2022-04-22T22:54:30.754004+0000 mgr.y (mgr.34107) 11194 : audit [DBG] from='client.56875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:31 smithi115 bash[45495]: audit 2022-04-22T22:54:30.777767+0000 mon.a (mon.0) 15532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:32 smithi121 bash[36902]: audit 2022-04-22T22:54:31.093944+0000 mgr.y (mgr.34107) 11195 : audit [DBG] from='client.56881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:32.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:32 smithi121 bash[36902]: audit 2022-04-22T22:54:31.519108+0000 mon.c (mon.1) 378 : audit [DBG] from='client.? 172.21.15.115:0/1987011171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:32.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:32 smithi115 bash[46934]: audit 2022-04-22T22:54:31.093944+0000 mgr.y (mgr.34107) 11195 : audit [DBG] from='client.56881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:32 smithi115 bash[46934]: audit 2022-04-22T22:54:31.519108+0000 mon.c (mon.1) 378 : audit [DBG] from='client.? 172.21.15.115:0/1987011171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:32 smithi115 bash[45495]: audit 2022-04-22T22:54:31.093944+0000 mgr.y (mgr.34107) 11195 : audit [DBG] from='client.56881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:54:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:32 smithi115 bash[45495]: audit 2022-04-22T22:54:31.519108+0000 mon.c (mon.1) 378 : audit [DBG] from='client.? 172.21.15.115:0/1987011171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:54:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:32.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:33 smithi121 bash[36902]: cluster 2022-04-22T22:54:32.335673+0000 mgr.y (mgr.34107) 11196 : cluster [DBG] pgmap v11793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:33 smithi121 bash[36902]: audit 2022-04-22T22:54:32.398768+0000 mon.a (mon.0) 15533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:33 smithi115 bash[45495]: cluster 2022-04-22T22:54:32.335673+0000 mgr.y (mgr.34107) 11196 : cluster [DBG] pgmap v11793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:33 smithi115 bash[45495]: audit 2022-04-22T22:54:32.398768+0000 mon.a (mon.0) 15533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:33 smithi115 bash[46934]: cluster 2022-04-22T22:54:32.335673+0000 mgr.y (mgr.34107) 11196 : cluster [DBG] pgmap v11793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:33 smithi115 bash[46934]: audit 2022-04-22T22:54:32.398768+0000 mon.a (mon.0) 15533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:34 smithi121 bash[36902]: audit 2022-04-22T22:54:33.914868+0000 mon.a (mon.0) 15534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:34.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:34 smithi115 bash[45495]: audit 2022-04-22T22:54:33.914868+0000 mon.a (mon.0) 15534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:34.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:34 smithi115 bash[46934]: audit 2022-04-22T22:54:33.914868+0000 mon.a (mon.0) 15534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:34.766Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:35 smithi121 bash[36902]: cluster 2022-04-22T22:54:34.336749+0000 mgr.y (mgr.34107) 11197 : cluster [DBG] pgmap v11794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:35 smithi115 bash[45495]: cluster 2022-04-22T22:54:34.336749+0000 mgr.y (mgr.34107) 11197 : cluster [DBG] pgmap v11794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:35 smithi115 bash[46934]: cluster 2022-04-22T22:54:34.336749+0000 mgr.y (mgr.34107) 11197 : cluster [DBG] pgmap v11794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:36 smithi121 bash[36902]: audit 2022-04-22T22:54:35.780415+0000 mon.a (mon.0) 15535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:36 smithi115 bash[45495]: audit 2022-04-22T22:54:35.780415+0000 mon.a (mon.0) 15535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:36 smithi115 bash[46934]: audit 2022-04-22T22:54:35.780415+0000 mon.a (mon.0) 15535 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:37 smithi121 bash[36902]: cluster 2022-04-22T22:54:36.337817+0000 mgr.y (mgr.34107) 11198 : cluster [DBG] pgmap v11795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:37 smithi121 bash[36902]: audit 2022-04-22T22:54:37.401655+0000 mon.a (mon.0) 15536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:37 smithi115 bash[45495]: cluster 2022-04-22T22:54:36.337817+0000 mgr.y (mgr.34107) 11198 : cluster [DBG] pgmap v11795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:37 smithi115 bash[45495]: audit 2022-04-22T22:54:37.401655+0000 mon.a (mon.0) 15536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:37 smithi115 bash[46934]: cluster 2022-04-22T22:54:36.337817+0000 mgr.y (mgr.34107) 11198 : cluster [DBG] pgmap v11795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:37 smithi115 bash[46934]: audit 2022-04-22T22:54:37.401655+0000 mon.a (mon.0) 15536 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:39 smithi121 bash[36902]: cluster 2022-04-22T22:54:38.338912+0000 mgr.y (mgr.34107) 11199 : cluster [DBG] pgmap v11796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:39 smithi121 bash[36902]: audit 2022-04-22T22:54:38.918695+0000 mon.a (mon.0) 15537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:39 smithi115 bash[45495]: cluster 2022-04-22T22:54:38.338912+0000 mgr.y (mgr.34107) 11199 : cluster [DBG] pgmap v11796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:39 smithi115 bash[45495]: audit 2022-04-22T22:54:38.918695+0000 mon.a (mon.0) 15537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:39 smithi115 bash[46934]: cluster 2022-04-22T22:54:38.338912+0000 mgr.y (mgr.34107) 11199 : cluster [DBG] pgmap v11796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:39 smithi115 bash[46934]: audit 2022-04-22T22:54:38.918695+0000 mon.a (mon.0) 15537 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:41 smithi121 bash[36902]: cluster 2022-04-22T22:54:40.339928+0000 mgr.y (mgr.34107) 11200 : cluster [DBG] pgmap v11797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:41 smithi121 bash[36902]: audit 2022-04-22T22:54:40.783356+0000 mon.a (mon.0) 15538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:41 smithi115 bash[45495]: cluster 2022-04-22T22:54:40.339928+0000 mgr.y (mgr.34107) 11200 : cluster [DBG] pgmap v11797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:41 smithi115 bash[45495]: audit 2022-04-22T22:54:40.783356+0000 mon.a (mon.0) 15538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:41 smithi115 bash[46934]: cluster 2022-04-22T22:54:40.339928+0000 mgr.y (mgr.34107) 11200 : cluster [DBG] pgmap v11797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:41 smithi115 bash[46934]: audit 2022-04-22T22:54:40.783356+0000 mon.a (mon.0) 15538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:42.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:43 smithi115 bash[45495]: cluster 2022-04-22T22:54:42.340971+0000 mgr.y (mgr.34107) 11201 : cluster [DBG] pgmap v11798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:43 smithi115 bash[45495]: audit 2022-04-22T22:54:42.404869+0000 mon.a (mon.0) 15539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:43 smithi115 bash[46934]: cluster 2022-04-22T22:54:42.340971+0000 mgr.y (mgr.34107) 11201 : cluster [DBG] pgmap v11798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:43 smithi115 bash[46934]: audit 2022-04-22T22:54:42.404869+0000 mon.a (mon.0) 15539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:43 smithi121 bash[36902]: cluster 2022-04-22T22:54:42.340971+0000 mgr.y (mgr.34107) 11201 : cluster [DBG] pgmap v11798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:43 smithi121 bash[36902]: audit 2022-04-22T22:54:42.404869+0000 mon.a (mon.0) 15539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:44 smithi121 bash[36902]: audit 2022-04-22T22:54:43.922375+0000 mon.a (mon.0) 15540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:44 smithi115 bash[45495]: audit 2022-04-22T22:54:43.922375+0000 mon.a (mon.0) 15540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:44 smithi115 bash[46934]: audit 2022-04-22T22:54:43.922375+0000 mon.a (mon.0) 15540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:44.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:45 smithi121 bash[36902]: cluster 2022-04-22T22:54:44.342073+0000 mgr.y (mgr.34107) 11202 : cluster [DBG] pgmap v11799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:45 smithi115 bash[45495]: cluster 2022-04-22T22:54:44.342073+0000 mgr.y (mgr.34107) 11202 : cluster [DBG] pgmap v11799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:45 smithi115 bash[46934]: cluster 2022-04-22T22:54:44.342073+0000 mgr.y (mgr.34107) 11202 : cluster [DBG] pgmap v11799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:46 smithi121 bash[36902]: audit 2022-04-22T22:54:45.786497+0000 mon.a (mon.0) 15541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:46 smithi115 bash[45495]: audit 2022-04-22T22:54:45.786497+0000 mon.a (mon.0) 15541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:46 smithi115 bash[46934]: audit 2022-04-22T22:54:45.786497+0000 mon.a (mon.0) 15541 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:47 smithi121 bash[36902]: cluster 2022-04-22T22:54:46.343158+0000 mgr.y (mgr.34107) 11203 : cluster [DBG] pgmap v11800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:47 smithi115 bash[45495]: cluster 2022-04-22T22:54:46.343158+0000 mgr.y (mgr.34107) 11203 : cluster [DBG] pgmap v11800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:47 smithi115 bash[46934]: cluster 2022-04-22T22:54:46.343158+0000 mgr.y (mgr.34107) 11203 : cluster [DBG] pgmap v11800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:48 smithi121 bash[36902]: audit 2022-04-22T22:54:47.408197+0000 mon.a (mon.0) 15542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:48 smithi115 bash[45495]: audit 2022-04-22T22:54:47.408197+0000 mon.a (mon.0) 15542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:48 smithi115 bash[46934]: audit 2022-04-22T22:54:47.408197+0000 mon.a (mon.0) 15542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:49 smithi121 bash[36902]: cluster 2022-04-22T22:54:48.344300+0000 mgr.y (mgr.34107) 11204 : cluster [DBG] pgmap v11801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:49 smithi121 bash[36902]: audit 2022-04-22T22:54:48.926342+0000 mon.a (mon.0) 15543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:49 smithi115 bash[45495]: cluster 2022-04-22T22:54:48.344300+0000 mgr.y (mgr.34107) 11204 : cluster [DBG] pgmap v11801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:49 smithi115 bash[45495]: audit 2022-04-22T22:54:48.926342+0000 mon.a (mon.0) 15543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:49 smithi115 bash[46934]: cluster 2022-04-22T22:54:48.344300+0000 mgr.y (mgr.34107) 11204 : cluster [DBG] pgmap v11801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:49 smithi115 bash[46934]: audit 2022-04-22T22:54:48.926342+0000 mon.a (mon.0) 15543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:51 smithi121 bash[36902]: cluster 2022-04-22T22:54:50.345362+0000 mgr.y (mgr.34107) 11205 : cluster [DBG] pgmap v11802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:51 smithi121 bash[36902]: audit 2022-04-22T22:54:50.789285+0000 mon.a (mon.0) 15544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:51 smithi115 bash[45495]: cluster 2022-04-22T22:54:50.345362+0000 mgr.y (mgr.34107) 11205 : cluster [DBG] pgmap v11802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:51 smithi115 bash[45495]: audit 2022-04-22T22:54:50.789285+0000 mon.a (mon.0) 15544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:51 smithi115 bash[46934]: cluster 2022-04-22T22:54:50.345362+0000 mgr.y (mgr.34107) 11205 : cluster [DBG] pgmap v11802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:51 smithi115 bash[46934]: audit 2022-04-22T22:54:50.789285+0000 mon.a (mon.0) 15544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:53.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:53 smithi115 bash[45495]: cluster 2022-04-22T22:54:52.346439+0000 mgr.y (mgr.34107) 11206 : cluster [DBG] pgmap v11803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:53 smithi115 bash[45495]: audit 2022-04-22T22:54:52.411234+0000 mon.a (mon.0) 15545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:52.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:53 smithi115 bash[46934]: cluster 2022-04-22T22:54:52.346439+0000 mgr.y (mgr.34107) 11206 : cluster [DBG] pgmap v11803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:53 smithi115 bash[46934]: audit 2022-04-22T22:54:52.411234+0000 mon.a (mon.0) 15545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:53 smithi121 bash[36902]: cluster 2022-04-22T22:54:52.346439+0000 mgr.y (mgr.34107) 11206 : cluster [DBG] pgmap v11803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:53 smithi121 bash[36902]: audit 2022-04-22T22:54:52.411234+0000 mon.a (mon.0) 15545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:54 smithi121 bash[36902]: audit 2022-04-22T22:54:53.929853+0000 mon.a (mon.0) 15546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:54 smithi115 bash[45495]: audit 2022-04-22T22:54:53.929853+0000 mon.a (mon.0) 15546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:54 smithi115 bash[46934]: audit 2022-04-22T22:54:53.929853+0000 mon.a (mon.0) 15546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:54:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:54:54.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:54:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:55 smithi121 bash[36902]: cluster 2022-04-22T22:54:54.347546+0000 mgr.y (mgr.34107) 11207 : cluster [DBG] pgmap v11804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:55.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:55 smithi115 bash[46934]: cluster 2022-04-22T22:54:54.347546+0000 mgr.y (mgr.34107) 11207 : cluster [DBG] pgmap v11804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:55 smithi115 bash[45495]: cluster 2022-04-22T22:54:54.347546+0000 mgr.y (mgr.34107) 11207 : cluster [DBG] pgmap v11804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:56 smithi121 bash[36902]: audit 2022-04-22T22:54:55.792345+0000 mon.a (mon.0) 15547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:56 smithi115 bash[45495]: audit 2022-04-22T22:54:55.792345+0000 mon.a (mon.0) 15547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:56 smithi115 bash[46934]: audit 2022-04-22T22:54:55.792345+0000 mon.a (mon.0) 15547 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:54:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:57 smithi121 bash[36902]: cluster 2022-04-22T22:54:56.348410+0000 mgr.y (mgr.34107) 11208 : cluster [DBG] pgmap v11805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:57.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:57 smithi115 bash[46934]: cluster 2022-04-22T22:54:56.348410+0000 mgr.y (mgr.34107) 11208 : cluster [DBG] pgmap v11805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:57 smithi115 bash[45495]: cluster 2022-04-22T22:54:56.348410+0000 mgr.y (mgr.34107) 11208 : cluster [DBG] pgmap v11805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:58 smithi121 bash[36902]: audit 2022-04-22T22:54:57.414351+0000 mon.a (mon.0) 15548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:58 smithi115 bash[45495]: audit 2022-04-22T22:54:57.414351+0000 mon.a (mon.0) 15548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:58 smithi115 bash[46934]: audit 2022-04-22T22:54:57.414351+0000 mon.a (mon.0) 15548 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:54:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:59 smithi121 bash[36902]: cluster 2022-04-22T22:54:58.349411+0000 mgr.y (mgr.34107) 11209 : cluster [DBG] pgmap v11806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:54:59 smithi121 bash[36902]: audit 2022-04-22T22:54:58.933439+0000 mon.a (mon.0) 15549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:59 smithi115 bash[45495]: cluster 2022-04-22T22:54:58.349411+0000 mgr.y (mgr.34107) 11209 : cluster [DBG] pgmap v11806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:54:59 smithi115 bash[45495]: audit 2022-04-22T22:54:58.933439+0000 mon.a (mon.0) 15549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:54:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:59 smithi115 bash[46934]: cluster 2022-04-22T22:54:58.349411+0000 mgr.y (mgr.34107) 11209 : cluster [DBG] pgmap v11806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:54:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:54:59 smithi115 bash[46934]: audit 2022-04-22T22:54:58.933439+0000 mon.a (mon.0) 15549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:01 smithi121 bash[36902]: cluster 2022-04-22T22:55:00.350510+0000 mgr.y (mgr.34107) 11210 : cluster [DBG] pgmap v11807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:01 smithi121 bash[36902]: audit 2022-04-22T22:55:00.795630+0000 mon.a (mon.0) 15550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:01.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:01 smithi115 bash[46934]: cluster 2022-04-22T22:55:00.350510+0000 mgr.y (mgr.34107) 11210 : cluster [DBG] pgmap v11807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:01 smithi115 bash[46934]: audit 2022-04-22T22:55:00.795630+0000 mon.a (mon.0) 15550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:01 smithi115 bash[45495]: cluster 2022-04-22T22:55:00.350510+0000 mgr.y (mgr.34107) 11210 : cluster [DBG] pgmap v11807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:01 smithi115 bash[45495]: audit 2022-04-22T22:55:00.795630+0000 mon.a (mon.0) 15550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:01.900 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:55:02.248 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:55:02.248 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:55:02.248 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:55:02.248 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:55:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:55:02.250 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:55:02.645 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:55:02.646 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:55:02.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:02 smithi121 bash[36902]: audit 2022-04-22T22:55:01.879906+0000 mgr.y (mgr.34107) 11211 : audit [DBG] from='client.56893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:02.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:02 smithi115 bash[45495]: audit 2022-04-22T22:55:01.879906+0000 mgr.y (mgr.34107) 11211 : audit [DBG] from='client.56893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:02 smithi115 bash[46934]: audit 2022-04-22T22:55:01.879906+0000 mgr.y (mgr.34107) 11211 : audit [DBG] from='client.56893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:02.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:02.226260+0000 mgr.y (mgr.34107) 11212 : audit [DBG] from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: cluster 2022-04-22T22:55:02.351705+0000 mgr.y (mgr.34107) 11213 : cluster [DBG] pgmap v11808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:02.416863+0000 mon.a (mon.0) 15551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:02.641268+0000 mon.a (mon.0) 15552 : audit [DBG] from='client.? 172.21.15.115:0/2861516677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:03.048339+0000 mon.a (mon.0) 15553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:03.051368+0000 mon.a (mon.0) 15554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:03.053056+0000 mon.a (mon.0) 15555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:55:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:03 smithi121 bash[36902]: audit 2022-04-22T22:55:03.403803+0000 mon.a (mon.0) 15556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:02.226260+0000 mgr.y (mgr.34107) 11212 : audit [DBG] from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: cluster 2022-04-22T22:55:02.351705+0000 mgr.y (mgr.34107) 11213 : cluster [DBG] pgmap v11808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:02.416863+0000 mon.a (mon.0) 15551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:02.641268+0000 mon.a (mon.0) 15552 : audit [DBG] from='client.? 172.21.15.115:0/2861516677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:03.048339+0000 mon.a (mon.0) 15553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:55:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:03.051368+0000 mon.a (mon.0) 15554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:03.053056+0000 mon.a (mon.0) 15555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[45495]: audit 2022-04-22T22:55:03.403803+0000 mon.a (mon.0) 15556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:02.226260+0000 mgr.y (mgr.34107) 11212 : audit [DBG] from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: cluster 2022-04-22T22:55:02.351705+0000 mgr.y (mgr.34107) 11213 : cluster [DBG] pgmap v11808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:02.416863+0000 mon.a (mon.0) 15551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:02.641268+0000 mon.a (mon.0) 15552 : audit [DBG] from='client.? 172.21.15.115:0/2861516677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:03.048339+0000 mon.a (mon.0) 15553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:55:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:03.051368+0000 mon.a (mon.0) 15554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:55:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:03.053056+0000 mon.a (mon.0) 15555 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:55:03.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:03 smithi115 bash[46934]: audit 2022-04-22T22:55:03.403803+0000 mon.a (mon.0) 15556 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:55:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:04 smithi121 bash[36902]: audit 2022-04-22T22:55:03.936984+0000 mon.a (mon.0) 15557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:04.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:04 smithi115 bash[45495]: audit 2022-04-22T22:55:03.936984+0000 mon.a (mon.0) 15557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:04.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:04 smithi115 bash[46934]: audit 2022-04-22T22:55:03.936984+0000 mon.a (mon.0) 15557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:04.767Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:05 smithi121 bash[36902]: cluster 2022-04-22T22:55:04.352745+0000 mgr.y (mgr.34107) 11214 : cluster [DBG] pgmap v11809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:05 smithi115 bash[45495]: cluster 2022-04-22T22:55:04.352745+0000 mgr.y (mgr.34107) 11214 : cluster [DBG] pgmap v11809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:05 smithi115 bash[46934]: cluster 2022-04-22T22:55:04.352745+0000 mgr.y (mgr.34107) 11214 : cluster [DBG] pgmap v11809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:06 smithi121 bash[36902]: audit 2022-04-22T22:55:05.798759+0000 mon.a (mon.0) 15558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:06 smithi115 bash[45495]: audit 2022-04-22T22:55:05.798759+0000 mon.a (mon.0) 15558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:06 smithi115 bash[46934]: audit 2022-04-22T22:55:05.798759+0000 mon.a (mon.0) 15558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:07 smithi121 bash[36902]: cluster 2022-04-22T22:55:06.353787+0000 mgr.y (mgr.34107) 11215 : cluster [DBG] pgmap v11810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:07 smithi121 bash[36902]: audit 2022-04-22T22:55:07.419547+0000 mon.a (mon.0) 15559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:07 smithi115 bash[45495]: cluster 2022-04-22T22:55:06.353787+0000 mgr.y (mgr.34107) 11215 : cluster [DBG] pgmap v11810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:07 smithi115 bash[45495]: audit 2022-04-22T22:55:07.419547+0000 mon.a (mon.0) 15559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:07 smithi115 bash[46934]: cluster 2022-04-22T22:55:06.353787+0000 mgr.y (mgr.34107) 11215 : cluster [DBG] pgmap v11810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:07 smithi115 bash[46934]: audit 2022-04-22T22:55:07.419547+0000 mon.a (mon.0) 15559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:09 smithi121 bash[36902]: cluster 2022-04-22T22:55:08.354935+0000 mgr.y (mgr.34107) 11216 : cluster [DBG] pgmap v11811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:09 smithi121 bash[36902]: audit 2022-04-22T22:55:08.940671+0000 mon.a (mon.0) 15560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:09.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:09 smithi115 bash[46934]: cluster 2022-04-22T22:55:08.354935+0000 mgr.y (mgr.34107) 11216 : cluster [DBG] pgmap v11811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:09 smithi115 bash[46934]: audit 2022-04-22T22:55:08.940671+0000 mon.a (mon.0) 15560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:09 smithi115 bash[45495]: cluster 2022-04-22T22:55:08.354935+0000 mgr.y (mgr.34107) 11216 : cluster [DBG] pgmap v11811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:09 smithi115 bash[45495]: audit 2022-04-22T22:55:08.940671+0000 mon.a (mon.0) 15560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:11 smithi121 bash[36902]: cluster 2022-04-22T22:55:10.356009+0000 mgr.y (mgr.34107) 11217 : cluster [DBG] pgmap v11812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:11 smithi121 bash[36902]: audit 2022-04-22T22:55:10.801637+0000 mon.a (mon.0) 15561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:11 smithi115 bash[46934]: cluster 2022-04-22T22:55:10.356009+0000 mgr.y (mgr.34107) 11217 : cluster [DBG] pgmap v11812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:11 smithi115 bash[46934]: audit 2022-04-22T22:55:10.801637+0000 mon.a (mon.0) 15561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:11 smithi115 bash[45495]: cluster 2022-04-22T22:55:10.356009+0000 mgr.y (mgr.34107) 11217 : cluster [DBG] pgmap v11812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:11 smithi115 bash[45495]: audit 2022-04-22T22:55:10.801637+0000 mon.a (mon.0) 15561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:13.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:13 smithi115 bash[45495]: cluster 2022-04-22T22:55:12.357125+0000 mgr.y (mgr.34107) 11218 : cluster [DBG] pgmap v11813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:13 smithi115 bash[45495]: audit 2022-04-22T22:55:12.422639+0000 mon.a (mon.0) 15562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:12.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:13 smithi115 bash[46934]: cluster 2022-04-22T22:55:12.357125+0000 mgr.y (mgr.34107) 11218 : cluster [DBG] pgmap v11813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:13 smithi115 bash[46934]: audit 2022-04-22T22:55:12.422639+0000 mon.a (mon.0) 15562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:13 smithi121 bash[36902]: cluster 2022-04-22T22:55:12.357125+0000 mgr.y (mgr.34107) 11218 : cluster [DBG] pgmap v11813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:13 smithi121 bash[36902]: audit 2022-04-22T22:55:12.422639+0000 mon.a (mon.0) 15562 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:14 smithi121 bash[36902]: audit 2022-04-22T22:55:13.944359+0000 mon.a (mon.0) 15563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:14 smithi115 bash[45495]: audit 2022-04-22T22:55:13.944359+0000 mon.a (mon.0) 15563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:14 smithi115 bash[46934]: audit 2022-04-22T22:55:13.944359+0000 mon.a (mon.0) 15563 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:14.768Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:15 smithi121 bash[36902]: cluster 2022-04-22T22:55:14.358225+0000 mgr.y (mgr.34107) 11219 : cluster [DBG] pgmap v11814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:15 smithi115 bash[45495]: cluster 2022-04-22T22:55:14.358225+0000 mgr.y (mgr.34107) 11219 : cluster [DBG] pgmap v11814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:15 smithi115 bash[46934]: cluster 2022-04-22T22:55:14.358225+0000 mgr.y (mgr.34107) 11219 : cluster [DBG] pgmap v11814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:16 smithi121 bash[36902]: audit 2022-04-22T22:55:15.804706+0000 mon.a (mon.0) 15564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:16 smithi115 bash[45495]: audit 2022-04-22T22:55:15.804706+0000 mon.a (mon.0) 15564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:16 smithi115 bash[46934]: audit 2022-04-22T22:55:15.804706+0000 mon.a (mon.0) 15564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:17 smithi121 bash[36902]: cluster 2022-04-22T22:55:16.359411+0000 mgr.y (mgr.34107) 11220 : cluster [DBG] pgmap v11815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:17 smithi121 bash[36902]: audit 2022-04-22T22:55:17.058214+0000 mon.a (mon.0) 15565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:55:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:17 smithi121 bash[36902]: audit 2022-04-22T22:55:17.061113+0000 mon.a (mon.0) 15566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[45495]: cluster 2022-04-22T22:55:16.359411+0000 mgr.y (mgr.34107) 11220 : cluster [DBG] pgmap v11815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[45495]: audit 2022-04-22T22:55:17.058214+0000 mon.a (mon.0) 15565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[45495]: audit 2022-04-22T22:55:17.061113+0000 mon.a (mon.0) 15566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[46934]: cluster 2022-04-22T22:55:16.359411+0000 mgr.y (mgr.34107) 11220 : cluster [DBG] pgmap v11815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[46934]: audit 2022-04-22T22:55:17.058214+0000 mon.a (mon.0) 15565 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:55:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:17 smithi115 bash[46934]: audit 2022-04-22T22:55:17.061113+0000 mon.a (mon.0) 15566 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:55:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:18 smithi121 bash[36902]: audit 2022-04-22T22:55:17.425876+0000 mon.a (mon.0) 15567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:18 smithi115 bash[45495]: audit 2022-04-22T22:55:17.425876+0000 mon.a (mon.0) 15567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:18 smithi115 bash[46934]: audit 2022-04-22T22:55:17.425876+0000 mon.a (mon.0) 15567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:19 smithi121 bash[36902]: cluster 2022-04-22T22:55:18.360591+0000 mgr.y (mgr.34107) 11221 : cluster [DBG] pgmap v11816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:19 smithi121 bash[36902]: audit 2022-04-22T22:55:18.947925+0000 mon.a (mon.0) 15568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:19 smithi115 bash[45495]: cluster 2022-04-22T22:55:18.360591+0000 mgr.y (mgr.34107) 11221 : cluster [DBG] pgmap v11816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:19 smithi115 bash[45495]: audit 2022-04-22T22:55:18.947925+0000 mon.a (mon.0) 15568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:19 smithi115 bash[46934]: cluster 2022-04-22T22:55:18.360591+0000 mgr.y (mgr.34107) 11221 : cluster [DBG] pgmap v11816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:19 smithi115 bash[46934]: audit 2022-04-22T22:55:18.947925+0000 mon.a (mon.0) 15568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:21 smithi121 bash[36902]: cluster 2022-04-22T22:55:20.361666+0000 mgr.y (mgr.34107) 11222 : cluster [DBG] pgmap v11817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:21 smithi121 bash[36902]: audit 2022-04-22T22:55:20.807850+0000 mon.a (mon.0) 15569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:21 smithi115 bash[45495]: cluster 2022-04-22T22:55:20.361666+0000 mgr.y (mgr.34107) 11222 : cluster [DBG] pgmap v11817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:21 smithi115 bash[45495]: audit 2022-04-22T22:55:20.807850+0000 mon.a (mon.0) 15569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:21 smithi115 bash[46934]: cluster 2022-04-22T22:55:20.361666+0000 mgr.y (mgr.34107) 11222 : cluster [DBG] pgmap v11817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:21 smithi115 bash[46934]: audit 2022-04-22T22:55:20.807850+0000 mon.a (mon.0) 15569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:22.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:23 smithi115 bash[45495]: cluster 2022-04-22T22:55:22.362733+0000 mgr.y (mgr.34107) 11223 : cluster [DBG] pgmap v11818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:23.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:23 smithi115 bash[45495]: audit 2022-04-22T22:55:22.429116+0000 mon.a (mon.0) 15570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:23 smithi115 bash[46934]: cluster 2022-04-22T22:55:22.362733+0000 mgr.y (mgr.34107) 11223 : cluster [DBG] pgmap v11818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:23 smithi115 bash[46934]: audit 2022-04-22T22:55:22.429116+0000 mon.a (mon.0) 15570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:23 smithi121 bash[36902]: cluster 2022-04-22T22:55:22.362733+0000 mgr.y (mgr.34107) 11223 : cluster [DBG] pgmap v11818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:23 smithi121 bash[36902]: audit 2022-04-22T22:55:22.429116+0000 mon.a (mon.0) 15570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:24 smithi121 bash[36902]: audit 2022-04-22T22:55:23.952043+0000 mon.a (mon.0) 15571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:24 smithi115 bash[45495]: audit 2022-04-22T22:55:23.952043+0000 mon.a (mon.0) 15571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:24 smithi115 bash[46934]: audit 2022-04-22T22:55:23.952043+0000 mon.a (mon.0) 15571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:24.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:25 smithi121 bash[36902]: cluster 2022-04-22T22:55:24.363782+0000 mgr.y (mgr.34107) 11224 : cluster [DBG] pgmap v11819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:25 smithi115 bash[45495]: cluster 2022-04-22T22:55:24.363782+0000 mgr.y (mgr.34107) 11224 : cluster [DBG] pgmap v11819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:25 smithi115 bash[46934]: cluster 2022-04-22T22:55:24.363782+0000 mgr.y (mgr.34107) 11224 : cluster [DBG] pgmap v11819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:26 smithi121 bash[36902]: audit 2022-04-22T22:55:25.810862+0000 mon.a (mon.0) 15572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:26 smithi115 bash[45495]: audit 2022-04-22T22:55:25.810862+0000 mon.a (mon.0) 15572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:26 smithi115 bash[46934]: audit 2022-04-22T22:55:25.810862+0000 mon.a (mon.0) 15572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:27 smithi121 bash[36902]: cluster 2022-04-22T22:55:26.364813+0000 mgr.y (mgr.34107) 11225 : cluster [DBG] pgmap v11820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:27 smithi115 bash[45495]: cluster 2022-04-22T22:55:26.364813+0000 mgr.y (mgr.34107) 11225 : cluster [DBG] pgmap v11820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:27 smithi115 bash[46934]: cluster 2022-04-22T22:55:26.364813+0000 mgr.y (mgr.34107) 11225 : cluster [DBG] pgmap v11820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:28 smithi121 bash[36902]: audit 2022-04-22T22:55:27.432253+0000 mon.a (mon.0) 15573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:28 smithi115 bash[46934]: audit 2022-04-22T22:55:27.432253+0000 mon.a (mon.0) 15573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:28 smithi115 bash[45495]: audit 2022-04-22T22:55:27.432253+0000 mon.a (mon.0) 15573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:29 smithi121 bash[36902]: cluster 2022-04-22T22:55:28.366108+0000 mgr.y (mgr.34107) 11226 : cluster [DBG] pgmap v11821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:29 smithi121 bash[36902]: audit 2022-04-22T22:55:28.955714+0000 mon.a (mon.0) 15574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:29 smithi115 bash[45495]: cluster 2022-04-22T22:55:28.366108+0000 mgr.y (mgr.34107) 11226 : cluster [DBG] pgmap v11821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:29 smithi115 bash[45495]: audit 2022-04-22T22:55:28.955714+0000 mon.a (mon.0) 15574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:29 smithi115 bash[46934]: cluster 2022-04-22T22:55:28.366108+0000 mgr.y (mgr.34107) 11226 : cluster [DBG] pgmap v11821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:29 smithi115 bash[46934]: audit 2022-04-22T22:55:28.955714+0000 mon.a (mon.0) 15574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:31.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:31 smithi121 bash[36902]: cluster 2022-04-22T22:55:30.367222+0000 mgr.y (mgr.34107) 11227 : cluster [DBG] pgmap v11822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:31.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:31 smithi121 bash[36902]: audit 2022-04-22T22:55:30.813809+0000 mon.a (mon.0) 15575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:31 smithi115 bash[45495]: cluster 2022-04-22T22:55:30.367222+0000 mgr.y (mgr.34107) 11227 : cluster [DBG] pgmap v11822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:31 smithi115 bash[45495]: audit 2022-04-22T22:55:30.813809+0000 mon.a (mon.0) 15575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:31 smithi115 bash[46934]: cluster 2022-04-22T22:55:30.367222+0000 mgr.y (mgr.34107) 11227 : cluster [DBG] pgmap v11822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:31 smithi115 bash[46934]: audit 2022-04-22T22:55:30.813809+0000 mon.a (mon.0) 15575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:33.022 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:55:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:32.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[45495]: cluster 2022-04-22T22:55:32.368311+0000 mgr.y (mgr.34107) 11228 : cluster [DBG] pgmap v11823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[45495]: audit 2022-04-22T22:55:32.435313+0000 mon.a (mon.0) 15576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[45495]: audit 2022-04-22T22:55:33.003466+0000 mgr.y (mgr.34107) 11229 : audit [DBG] from='client.46974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[46934]: cluster 2022-04-22T22:55:32.368311+0000 mgr.y (mgr.34107) 11228 : cluster [DBG] pgmap v11823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[46934]: audit 2022-04-22T22:55:32.435313+0000 mon.a (mon.0) 15576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:33 smithi115 bash[46934]: audit 2022-04-22T22:55:33.003466+0000 mgr.y (mgr.34107) 11229 : audit [DBG] from='client.46974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:33.370 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:55:33.370 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:55:33.370 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:55:33.370 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:55:33.371 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:55:33.371 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:55:33.371 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:55:33.371 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:55:33.371 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:55:33.372 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:55:33.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:55:33.373 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:55:33.373 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:55:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:33 smithi121 bash[36902]: cluster 2022-04-22T22:55:32.368311+0000 mgr.y (mgr.34107) 11228 : cluster [DBG] pgmap v11823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:33 smithi121 bash[36902]: audit 2022-04-22T22:55:32.435313+0000 mon.a (mon.0) 15576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:33 smithi121 bash[36902]: audit 2022-04-22T22:55:33.003466+0000 mgr.y (mgr.34107) 11229 : audit [DBG] from='client.46974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:33.768 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:55:33.769 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:55:33.770 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:55:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:34 smithi121 bash[36902]: audit 2022-04-22T22:55:33.345794+0000 mgr.y (mgr.34107) 11230 : audit [DBG] from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:34 smithi121 bash[36902]: audit 2022-04-22T22:55:33.765091+0000 mon.a (mon.0) 15577 : audit [DBG] from='client.? 172.21.15.115:0/1070535270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:34 smithi121 bash[36902]: audit 2022-04-22T22:55:33.959202+0000 mon.a (mon.0) 15578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[45495]: audit 2022-04-22T22:55:33.345794+0000 mgr.y (mgr.34107) 11230 : audit [DBG] from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[45495]: audit 2022-04-22T22:55:33.765091+0000 mon.a (mon.0) 15577 : audit [DBG] from='client.? 172.21.15.115:0/1070535270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[45495]: audit 2022-04-22T22:55:33.959202+0000 mon.a (mon.0) 15578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[46934]: audit 2022-04-22T22:55:33.345794+0000 mgr.y (mgr.34107) 11230 : audit [DBG] from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:55:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[46934]: audit 2022-04-22T22:55:33.765091+0000 mon.a (mon.0) 15577 : audit [DBG] from='client.? 172.21.15.115:0/1070535270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:55:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[46934]: audit 2022-04-22T22:55:33.959202+0000 mon.a (mon.0) 15578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:34.770Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:35 smithi121 bash[36902]: cluster 2022-04-22T22:55:34.369440+0000 mgr.y (mgr.34107) 11231 : cluster [DBG] pgmap v11824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:35 smithi115 bash[45495]: cluster 2022-04-22T22:55:34.369440+0000 mgr.y (mgr.34107) 11231 : cluster [DBG] pgmap v11824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:35 smithi115 bash[46934]: cluster 2022-04-22T22:55:34.369440+0000 mgr.y (mgr.34107) 11231 : cluster [DBG] pgmap v11824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:36 smithi121 bash[36902]: audit 2022-04-22T22:55:35.816981+0000 mon.a (mon.0) 15579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:36 smithi115 bash[45495]: audit 2022-04-22T22:55:35.816981+0000 mon.a (mon.0) 15579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:36 smithi115 bash[46934]: audit 2022-04-22T22:55:35.816981+0000 mon.a (mon.0) 15579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:37 smithi121 bash[36902]: cluster 2022-04-22T22:55:36.370567+0000 mgr.y (mgr.34107) 11232 : cluster [DBG] pgmap v11825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:37 smithi115 bash[45495]: cluster 2022-04-22T22:55:36.370567+0000 mgr.y (mgr.34107) 11232 : cluster [DBG] pgmap v11825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:37 smithi115 bash[46934]: cluster 2022-04-22T22:55:36.370567+0000 mgr.y (mgr.34107) 11232 : cluster [DBG] pgmap v11825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:38 smithi121 bash[36902]: audit 2022-04-22T22:55:37.438415+0000 mon.a (mon.0) 15580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:38 smithi115 bash[46934]: audit 2022-04-22T22:55:37.438415+0000 mon.a (mon.0) 15580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:38 smithi115 bash[45495]: audit 2022-04-22T22:55:37.438415+0000 mon.a (mon.0) 15580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:39 smithi121 bash[36902]: cluster 2022-04-22T22:55:38.371747+0000 mgr.y (mgr.34107) 11233 : cluster [DBG] pgmap v11826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:39 smithi121 bash[36902]: audit 2022-04-22T22:55:38.962896+0000 mon.a (mon.0) 15581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:39.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:39 smithi115 bash[46934]: cluster 2022-04-22T22:55:38.371747+0000 mgr.y (mgr.34107) 11233 : cluster [DBG] pgmap v11826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:39 smithi115 bash[46934]: audit 2022-04-22T22:55:38.962896+0000 mon.a (mon.0) 15581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:39 smithi115 bash[45495]: cluster 2022-04-22T22:55:38.371747+0000 mgr.y (mgr.34107) 11233 : cluster [DBG] pgmap v11826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:39 smithi115 bash[45495]: audit 2022-04-22T22:55:38.962896+0000 mon.a (mon.0) 15581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:41 smithi121 bash[36902]: cluster 2022-04-22T22:55:40.372846+0000 mgr.y (mgr.34107) 11234 : cluster [DBG] pgmap v11827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:41 smithi121 bash[36902]: audit 2022-04-22T22:55:40.820154+0000 mon.a (mon.0) 15582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:41.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:41 smithi115 bash[45495]: cluster 2022-04-22T22:55:40.372846+0000 mgr.y (mgr.34107) 11234 : cluster [DBG] pgmap v11827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:41 smithi115 bash[45495]: audit 2022-04-22T22:55:40.820154+0000 mon.a (mon.0) 15582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:41.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:41 smithi115 bash[46934]: cluster 2022-04-22T22:55:40.372846+0000 mgr.y (mgr.34107) 11234 : cluster [DBG] pgmap v11827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:41.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:41 smithi115 bash[46934]: audit 2022-04-22T22:55:40.820154+0000 mon.a (mon.0) 15582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:42.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:43 smithi115 bash[45495]: cluster 2022-04-22T22:55:42.373948+0000 mgr.y (mgr.34107) 11235 : cluster [DBG] pgmap v11828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:43 smithi115 bash[45495]: audit 2022-04-22T22:55:42.441625+0000 mon.a (mon.0) 15583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:43 smithi115 bash[46934]: cluster 2022-04-22T22:55:42.373948+0000 mgr.y (mgr.34107) 11235 : cluster [DBG] pgmap v11828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:43 smithi115 bash[46934]: audit 2022-04-22T22:55:42.441625+0000 mon.a (mon.0) 15583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:43 smithi121 bash[36902]: cluster 2022-04-22T22:55:42.373948+0000 mgr.y (mgr.34107) 11235 : cluster [DBG] pgmap v11828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:43 smithi121 bash[36902]: audit 2022-04-22T22:55:42.441625+0000 mon.a (mon.0) 15583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:44 smithi121 bash[36902]: audit 2022-04-22T22:55:43.966689+0000 mon.a (mon.0) 15584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:44 smithi115 bash[45495]: audit 2022-04-22T22:55:43.966689+0000 mon.a (mon.0) 15584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:44 smithi115 bash[46934]: audit 2022-04-22T22:55:43.966689+0000 mon.a (mon.0) 15584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:44.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:45 smithi121 bash[36902]: cluster 2022-04-22T22:55:44.375175+0000 mgr.y (mgr.34107) 11236 : cluster [DBG] pgmap v11829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:45.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:45 smithi115 bash[46934]: cluster 2022-04-22T22:55:44.375175+0000 mgr.y (mgr.34107) 11236 : cluster [DBG] pgmap v11829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:45 smithi115 bash[45495]: cluster 2022-04-22T22:55:44.375175+0000 mgr.y (mgr.34107) 11236 : cluster [DBG] pgmap v11829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:46 smithi121 bash[36902]: audit 2022-04-22T22:55:45.823199+0000 mon.a (mon.0) 15585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:46 smithi115 bash[45495]: audit 2022-04-22T22:55:45.823199+0000 mon.a (mon.0) 15585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:46 smithi115 bash[46934]: audit 2022-04-22T22:55:45.823199+0000 mon.a (mon.0) 15585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:47 smithi121 bash[36902]: cluster 2022-04-22T22:55:46.376208+0000 mgr.y (mgr.34107) 11237 : cluster [DBG] pgmap v11830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:47 smithi115 bash[45495]: cluster 2022-04-22T22:55:46.376208+0000 mgr.y (mgr.34107) 11237 : cluster [DBG] pgmap v11830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:47 smithi115 bash[46934]: cluster 2022-04-22T22:55:46.376208+0000 mgr.y (mgr.34107) 11237 : cluster [DBG] pgmap v11830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:48 smithi121 bash[36902]: audit 2022-04-22T22:55:47.444667+0000 mon.a (mon.0) 15586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:48 smithi115 bash[45495]: audit 2022-04-22T22:55:47.444667+0000 mon.a (mon.0) 15586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:48 smithi115 bash[46934]: audit 2022-04-22T22:55:47.444667+0000 mon.a (mon.0) 15586 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:49 smithi121 bash[36902]: cluster 2022-04-22T22:55:48.377343+0000 mgr.y (mgr.34107) 11238 : cluster [DBG] pgmap v11831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:49 smithi121 bash[36902]: audit 2022-04-22T22:55:48.970550+0000 mon.a (mon.0) 15587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:49 smithi115 bash[45495]: cluster 2022-04-22T22:55:48.377343+0000 mgr.y (mgr.34107) 11238 : cluster [DBG] pgmap v11831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:49 smithi115 bash[45495]: audit 2022-04-22T22:55:48.970550+0000 mon.a (mon.0) 15587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:49 smithi115 bash[46934]: cluster 2022-04-22T22:55:48.377343+0000 mgr.y (mgr.34107) 11238 : cluster [DBG] pgmap v11831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:49 smithi115 bash[46934]: audit 2022-04-22T22:55:48.970550+0000 mon.a (mon.0) 15587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:51 smithi121 bash[36902]: cluster 2022-04-22T22:55:50.378424+0000 mgr.y (mgr.34107) 11239 : cluster [DBG] pgmap v11832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:51 smithi121 bash[36902]: audit 2022-04-22T22:55:50.826185+0000 mon.a (mon.0) 15588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:51 smithi115 bash[45495]: cluster 2022-04-22T22:55:50.378424+0000 mgr.y (mgr.34107) 11239 : cluster [DBG] pgmap v11832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:51 smithi115 bash[45495]: audit 2022-04-22T22:55:50.826185+0000 mon.a (mon.0) 15588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:51 smithi115 bash[46934]: cluster 2022-04-22T22:55:50.378424+0000 mgr.y (mgr.34107) 11239 : cluster [DBG] pgmap v11832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:51 smithi115 bash[46934]: audit 2022-04-22T22:55:50.826185+0000 mon.a (mon.0) 15588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:52.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:53 smithi115 bash[45495]: cluster 2022-04-22T22:55:52.379539+0000 mgr.y (mgr.34107) 11240 : cluster [DBG] pgmap v11833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:53 smithi115 bash[45495]: audit 2022-04-22T22:55:52.448322+0000 mon.a (mon.0) 15589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:53 smithi115 bash[46934]: cluster 2022-04-22T22:55:52.379539+0000 mgr.y (mgr.34107) 11240 : cluster [DBG] pgmap v11833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:53 smithi115 bash[46934]: audit 2022-04-22T22:55:52.448322+0000 mon.a (mon.0) 15589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:53 smithi121 bash[36902]: cluster 2022-04-22T22:55:52.379539+0000 mgr.y (mgr.34107) 11240 : cluster [DBG] pgmap v11833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:53 smithi121 bash[36902]: audit 2022-04-22T22:55:52.448322+0000 mon.a (mon.0) 15589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:54 smithi121 bash[36902]: audit 2022-04-22T22:55:53.974124+0000 mon.a (mon.0) 15590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:54 smithi115 bash[46934]: audit 2022-04-22T22:55:53.974124+0000 mon.a (mon.0) 15590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:54 smithi115 bash[45495]: audit 2022-04-22T22:55:53.974124+0000 mon.a (mon.0) 15590 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:55:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:55:54.772Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:55:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:55 smithi121 bash[36902]: cluster 2022-04-22T22:55:54.380591+0000 mgr.y (mgr.34107) 11241 : cluster [DBG] pgmap v11834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:55 smithi115 bash[45495]: cluster 2022-04-22T22:55:54.380591+0000 mgr.y (mgr.34107) 11241 : cluster [DBG] pgmap v11834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:55 smithi115 bash[46934]: cluster 2022-04-22T22:55:54.380591+0000 mgr.y (mgr.34107) 11241 : cluster [DBG] pgmap v11834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:56 smithi121 bash[36902]: audit 2022-04-22T22:55:55.829472+0000 mon.a (mon.0) 15591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:56 smithi115 bash[45495]: audit 2022-04-22T22:55:55.829472+0000 mon.a (mon.0) 15591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:56 smithi115 bash[46934]: audit 2022-04-22T22:55:55.829472+0000 mon.a (mon.0) 15591 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:55:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:57 smithi121 bash[36902]: cluster 2022-04-22T22:55:56.381681+0000 mgr.y (mgr.34107) 11242 : cluster [DBG] pgmap v11835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:57 smithi115 bash[45495]: cluster 2022-04-22T22:55:56.381681+0000 mgr.y (mgr.34107) 11242 : cluster [DBG] pgmap v11835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:57 smithi115 bash[46934]: cluster 2022-04-22T22:55:56.381681+0000 mgr.y (mgr.34107) 11242 : cluster [DBG] pgmap v11835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:58 smithi121 bash[36902]: audit 2022-04-22T22:55:57.451341+0000 mon.a (mon.0) 15592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:58.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:58 smithi115 bash[46934]: audit 2022-04-22T22:55:57.451341+0000 mon.a (mon.0) 15592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:58 smithi115 bash[45495]: audit 2022-04-22T22:55:57.451341+0000 mon.a (mon.0) 15592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:55:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:59 smithi121 bash[36902]: cluster 2022-04-22T22:55:58.382922+0000 mgr.y (mgr.34107) 11243 : cluster [DBG] pgmap v11836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:55:59 smithi121 bash[36902]: audit 2022-04-22T22:55:58.977715+0000 mon.a (mon.0) 15593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:59 smithi115 bash[45495]: cluster 2022-04-22T22:55:58.382922+0000 mgr.y (mgr.34107) 11243 : cluster [DBG] pgmap v11836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:55:59 smithi115 bash[45495]: audit 2022-04-22T22:55:58.977715+0000 mon.a (mon.0) 15593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:55:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:59 smithi115 bash[46934]: cluster 2022-04-22T22:55:58.382922+0000 mgr.y (mgr.34107) 11243 : cluster [DBG] pgmap v11836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:55:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:55:59 smithi115 bash[46934]: audit 2022-04-22T22:55:58.977715+0000 mon.a (mon.0) 15593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:01 smithi121 bash[36902]: cluster 2022-04-22T22:56:00.384189+0000 mgr.y (mgr.34107) 11244 : cluster [DBG] pgmap v11837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:01 smithi121 bash[36902]: audit 2022-04-22T22:56:00.832882+0000 mon.a (mon.0) 15594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:01 smithi115 bash[45495]: cluster 2022-04-22T22:56:00.384189+0000 mgr.y (mgr.34107) 11244 : cluster [DBG] pgmap v11837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:01 smithi115 bash[45495]: audit 2022-04-22T22:56:00.832882+0000 mon.a (mon.0) 15594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:01 smithi115 bash[46934]: cluster 2022-04-22T22:56:00.384189+0000 mgr.y (mgr.34107) 11244 : cluster [DBG] pgmap v11837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:01 smithi115 bash[46934]: audit 2022-04-22T22:56:00.832882+0000 mon.a (mon.0) 15594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:03.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:02.913Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:03.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:03 smithi115 bash[45495]: cluster 2022-04-22T22:56:02.385318+0000 mgr.y (mgr.34107) 11245 : cluster [DBG] pgmap v11838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:03.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:03 smithi115 bash[45495]: audit 2022-04-22T22:56:02.454551+0000 mon.a (mon.0) 15595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:03 smithi115 bash[46934]: cluster 2022-04-22T22:56:02.385318+0000 mgr.y (mgr.34107) 11245 : cluster [DBG] pgmap v11838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:03.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:03 smithi115 bash[46934]: audit 2022-04-22T22:56:02.454551+0000 mon.a (mon.0) 15595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:03.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:03 smithi121 bash[36902]: cluster 2022-04-22T22:56:02.385318+0000 mgr.y (mgr.34107) 11245 : cluster [DBG] pgmap v11838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:03.441 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:03 smithi121 bash[36902]: audit 2022-04-22T22:56:02.454551+0000 mon.a (mon.0) 15595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:04.139 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 10m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 10m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:56:04.482 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 10m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 10m ago 5h 296M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 10m ago 5h 10.7M - 1dbe0e931976 f4b044642a96 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:56:04.483 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 10m ago 5h 588M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:56:04.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 10m ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:56:04.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 10m ago 5h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:56:04.484 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 10m ago 5h 651M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:56:04.484 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 10m ago 5h 70.5M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:56:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:04 smithi121 bash[36902]: audit 2022-04-22T22:56:03.412200+0000 mon.a (mon.0) 15596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:56:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:04 smithi121 bash[36902]: audit 2022-04-22T22:56:03.415268+0000 mon.a (mon.0) 15597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:56:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:04 smithi121 bash[36902]: audit 2022-04-22T22:56:03.417090+0000 mon.a (mon.0) 15598 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:56:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:04 smithi121 bash[36902]: audit 2022-04-22T22:56:03.981117+0000 mon.a (mon.0) 15599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[46934]: audit 2022-04-22T22:56:03.412200+0000 mon.a (mon.0) 15596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:56:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[46934]: audit 2022-04-22T22:56:03.415268+0000 mon.a (mon.0) 15597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:56:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[46934]: audit 2022-04-22T22:56:03.417090+0000 mon.a (mon.0) 15598 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:56:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[46934]: audit 2022-04-22T22:56:03.981117+0000 mon.a (mon.0) 15599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[45495]: audit 2022-04-22T22:56:03.412200+0000 mon.a (mon.0) 15596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:56:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[45495]: audit 2022-04-22T22:56:03.415268+0000 mon.a (mon.0) 15597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:56:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[45495]: audit 2022-04-22T22:56:03.417090+0000 mon.a (mon.0) 15598 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:56:04.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[45495]: audit 2022-04-22T22:56:03.981117+0000 mon.a (mon.0) 15599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:04.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:56:04.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:56:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:04.771Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:05 smithi121 bash[36902]: audit 2022-04-22T22:56:04.121888+0000 mgr.y (mgr.34107) 11246 : audit [DBG] from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:05 smithi121 bash[36902]: cluster 2022-04-22T22:56:04.386057+0000 mgr.y (mgr.34107) 11247 : cluster [DBG] pgmap v11839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:05 smithi121 bash[36902]: audit 2022-04-22T22:56:04.463911+0000 mgr.y (mgr.34107) 11248 : audit [DBG] from='client.56935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:05 smithi121 bash[36902]: audit 2022-04-22T22:56:04.880393+0000 mon.a (mon.0) 15600 : audit [DBG] from='client.? 172.21.15.115:0/2787592739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[45495]: audit 2022-04-22T22:56:04.121888+0000 mgr.y (mgr.34107) 11246 : audit [DBG] from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[45495]: cluster 2022-04-22T22:56:04.386057+0000 mgr.y (mgr.34107) 11247 : cluster [DBG] pgmap v11839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[45495]: audit 2022-04-22T22:56:04.463911+0000 mgr.y (mgr.34107) 11248 : audit [DBG] from='client.56935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[45495]: audit 2022-04-22T22:56:04.880393+0000 mon.a (mon.0) 15600 : audit [DBG] from='client.? 172.21.15.115:0/2787592739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[46934]: audit 2022-04-22T22:56:04.121888+0000 mgr.y (mgr.34107) 11246 : audit [DBG] from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[46934]: cluster 2022-04-22T22:56:04.386057+0000 mgr.y (mgr.34107) 11247 : cluster [DBG] pgmap v11839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[46934]: audit 2022-04-22T22:56:04.463911+0000 mgr.y (mgr.34107) 11248 : audit [DBG] from='client.56935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:05 smithi115 bash[46934]: audit 2022-04-22T22:56:04.880393+0000 mon.a (mon.0) 15600 : audit [DBG] from='client.? 172.21.15.115:0/2787592739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:06 smithi121 bash[36902]: audit 2022-04-22T22:56:05.836142+0000 mon.a (mon.0) 15601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:06 smithi115 bash[45495]: audit 2022-04-22T22:56:05.836142+0000 mon.a (mon.0) 15601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:06 smithi115 bash[46934]: audit 2022-04-22T22:56:05.836142+0000 mon.a (mon.0) 15601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:07 smithi121 bash[36902]: cluster 2022-04-22T22:56:06.387097+0000 mgr.y (mgr.34107) 11249 : cluster [DBG] pgmap v11840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:07 smithi115 bash[45495]: cluster 2022-04-22T22:56:06.387097+0000 mgr.y (mgr.34107) 11249 : cluster [DBG] pgmap v11840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:07 smithi115 bash[46934]: cluster 2022-04-22T22:56:06.387097+0000 mgr.y (mgr.34107) 11249 : cluster [DBG] pgmap v11840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:08 smithi121 bash[36902]: audit 2022-04-22T22:56:07.457647+0000 mon.a (mon.0) 15602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:08 smithi121 bash[36902]: audit 2022-04-22T22:56:07.880669+0000 mon.a (mon.0) 15603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:08 smithi115 bash[45495]: audit 2022-04-22T22:56:07.457647+0000 mon.a (mon.0) 15602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:08 smithi115 bash[45495]: audit 2022-04-22T22:56:07.880669+0000 mon.a (mon.0) 15603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:08 smithi115 bash[46934]: audit 2022-04-22T22:56:07.457647+0000 mon.a (mon.0) 15602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:08 smithi115 bash[46934]: audit 2022-04-22T22:56:07.880669+0000 mon.a (mon.0) 15603 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:09 smithi121 bash[36902]: audit 2022-04-22T22:56:08.223325+0000 mon.a (mon.0) 15604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:09 smithi121 bash[36902]: cluster 2022-04-22T22:56:08.388256+0000 mgr.y (mgr.34107) 11250 : cluster [DBG] pgmap v11841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:09 smithi121 bash[36902]: audit 2022-04-22T22:56:08.984648+0000 mon.a (mon.0) 15605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[45495]: audit 2022-04-22T22:56:08.223325+0000 mon.a (mon.0) 15604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[45495]: cluster 2022-04-22T22:56:08.388256+0000 mgr.y (mgr.34107) 11250 : cluster [DBG] pgmap v11841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[45495]: audit 2022-04-22T22:56:08.984648+0000 mon.a (mon.0) 15605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[46934]: audit 2022-04-22T22:56:08.223325+0000 mon.a (mon.0) 15604 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:56:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[46934]: cluster 2022-04-22T22:56:08.388256+0000 mgr.y (mgr.34107) 11250 : cluster [DBG] pgmap v11841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:09 smithi115 bash[46934]: audit 2022-04-22T22:56:08.984648+0000 mon.a (mon.0) 15605 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:11 smithi121 bash[36902]: cluster 2022-04-22T22:56:10.389345+0000 mgr.y (mgr.34107) 11251 : cluster [DBG] pgmap v11842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:11 smithi121 bash[36902]: audit 2022-04-22T22:56:10.839255+0000 mon.a (mon.0) 15606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:11 smithi115 bash[45495]: cluster 2022-04-22T22:56:10.389345+0000 mgr.y (mgr.34107) 11251 : cluster [DBG] pgmap v11842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:11 smithi115 bash[45495]: audit 2022-04-22T22:56:10.839255+0000 mon.a (mon.0) 15606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:11 smithi115 bash[46934]: cluster 2022-04-22T22:56:10.389345+0000 mgr.y (mgr.34107) 11251 : cluster [DBG] pgmap v11842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:11 smithi115 bash[46934]: audit 2022-04-22T22:56:10.839255+0000 mon.a (mon.0) 15606 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:12.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:13 smithi115 bash[45495]: cluster 2022-04-22T22:56:12.390578+0000 mgr.y (mgr.34107) 11252 : cluster [DBG] pgmap v11843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:13 smithi115 bash[45495]: audit 2022-04-22T22:56:12.460667+0000 mon.a (mon.0) 15607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:13 smithi115 bash[46934]: cluster 2022-04-22T22:56:12.390578+0000 mgr.y (mgr.34107) 11252 : cluster [DBG] pgmap v11843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:13.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:13 smithi115 bash[46934]: audit 2022-04-22T22:56:12.460667+0000 mon.a (mon.0) 15607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:13 smithi121 bash[36902]: cluster 2022-04-22T22:56:12.390578+0000 mgr.y (mgr.34107) 11252 : cluster [DBG] pgmap v11843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:13 smithi121 bash[36902]: audit 2022-04-22T22:56:12.460667+0000 mon.a (mon.0) 15607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:14 smithi121 bash[36902]: audit 2022-04-22T22:56:13.988467+0000 mon.a (mon.0) 15608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:14 smithi115 bash[45495]: audit 2022-04-22T22:56:13.988467+0000 mon.a (mon.0) 15608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:14 smithi115 bash[46934]: audit 2022-04-22T22:56:13.988467+0000 mon.a (mon.0) 15608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:14.773Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:15 smithi121 bash[36902]: cluster 2022-04-22T22:56:14.391812+0000 mgr.y (mgr.34107) 11253 : cluster [DBG] pgmap v11844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:15 smithi115 bash[45495]: cluster 2022-04-22T22:56:14.391812+0000 mgr.y (mgr.34107) 11253 : cluster [DBG] pgmap v11844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:15 smithi115 bash[46934]: cluster 2022-04-22T22:56:14.391812+0000 mgr.y (mgr.34107) 11253 : cluster [DBG] pgmap v11844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:16 smithi121 bash[36902]: audit 2022-04-22T22:56:15.842287+0000 mon.a (mon.0) 15609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:16 smithi115 bash[45495]: audit 2022-04-22T22:56:15.842287+0000 mon.a (mon.0) 15609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:16 smithi115 bash[46934]: audit 2022-04-22T22:56:15.842287+0000 mon.a (mon.0) 15609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:17 smithi121 bash[36902]: cluster 2022-04-22T22:56:16.392887+0000 mgr.y (mgr.34107) 11254 : cluster [DBG] pgmap v11845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:17 smithi121 bash[36902]: audit 2022-04-22T22:56:17.068777+0000 mon.a (mon.0) 15610 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:56:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:17 smithi121 bash[36902]: audit 2022-04-22T22:56:17.069660+0000 mon.a (mon.0) 15611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:56:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[45495]: cluster 2022-04-22T22:56:16.392887+0000 mgr.y (mgr.34107) 11254 : cluster [DBG] pgmap v11845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[45495]: audit 2022-04-22T22:56:17.068777+0000 mon.a (mon.0) 15610 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:56:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[45495]: audit 2022-04-22T22:56:17.069660+0000 mon.a (mon.0) 15611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:56:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[46934]: cluster 2022-04-22T22:56:16.392887+0000 mgr.y (mgr.34107) 11254 : cluster [DBG] pgmap v11845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[46934]: audit 2022-04-22T22:56:17.068777+0000 mon.a (mon.0) 15610 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:56:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:17 smithi115 bash[46934]: audit 2022-04-22T22:56:17.069660+0000 mon.a (mon.0) 15611 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:56:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:18 smithi121 bash[36902]: audit 2022-04-22T22:56:17.463917+0000 mon.a (mon.0) 15612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:18.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:18 smithi115 bash[45495]: audit 2022-04-22T22:56:17.463917+0000 mon.a (mon.0) 15612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:18.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:18 smithi115 bash[46934]: audit 2022-04-22T22:56:17.463917+0000 mon.a (mon.0) 15612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:19 smithi121 bash[36902]: cluster 2022-04-22T22:56:18.394099+0000 mgr.y (mgr.34107) 11255 : cluster [DBG] pgmap v11846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:19 smithi121 bash[36902]: audit 2022-04-22T22:56:18.992068+0000 mon.a (mon.0) 15613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:19 smithi115 bash[46934]: cluster 2022-04-22T22:56:18.394099+0000 mgr.y (mgr.34107) 11255 : cluster [DBG] pgmap v11846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:19 smithi115 bash[46934]: audit 2022-04-22T22:56:18.992068+0000 mon.a (mon.0) 15613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:19.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:19 smithi115 bash[45495]: cluster 2022-04-22T22:56:18.394099+0000 mgr.y (mgr.34107) 11255 : cluster [DBG] pgmap v11846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:19.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:19 smithi115 bash[45495]: audit 2022-04-22T22:56:18.992068+0000 mon.a (mon.0) 15613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:21 smithi121 bash[36902]: cluster 2022-04-22T22:56:20.395184+0000 mgr.y (mgr.34107) 11256 : cluster [DBG] pgmap v11847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:21 smithi121 bash[36902]: audit 2022-04-22T22:56:20.845485+0000 mon.a (mon.0) 15614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:21 smithi115 bash[45495]: cluster 2022-04-22T22:56:20.395184+0000 mgr.y (mgr.34107) 11256 : cluster [DBG] pgmap v11847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:21 smithi115 bash[45495]: audit 2022-04-22T22:56:20.845485+0000 mon.a (mon.0) 15614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:21 smithi115 bash[46934]: cluster 2022-04-22T22:56:20.395184+0000 mgr.y (mgr.34107) 11256 : cluster [DBG] pgmap v11847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:21 smithi115 bash[46934]: audit 2022-04-22T22:56:20.845485+0000 mon.a (mon.0) 15614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:22.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:23 smithi115 bash[45495]: cluster 2022-04-22T22:56:22.396335+0000 mgr.y (mgr.34107) 11257 : cluster [DBG] pgmap v11848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:23 smithi115 bash[45495]: audit 2022-04-22T22:56:22.467064+0000 mon.a (mon.0) 15615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:23 smithi115 bash[46934]: cluster 2022-04-22T22:56:22.396335+0000 mgr.y (mgr.34107) 11257 : cluster [DBG] pgmap v11848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:23 smithi115 bash[46934]: audit 2022-04-22T22:56:22.467064+0000 mon.a (mon.0) 15615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:23 smithi121 bash[36902]: cluster 2022-04-22T22:56:22.396335+0000 mgr.y (mgr.34107) 11257 : cluster [DBG] pgmap v11848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:23 smithi121 bash[36902]: audit 2022-04-22T22:56:22.467064+0000 mon.a (mon.0) 15615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:24 smithi121 bash[36902]: audit 2022-04-22T22:56:23.995919+0000 mon.a (mon.0) 15616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:24 smithi115 bash[45495]: audit 2022-04-22T22:56:23.995919+0000 mon.a (mon.0) 15616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:24 smithi115 bash[46934]: audit 2022-04-22T22:56:23.995919+0000 mon.a (mon.0) 15616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:24.774Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:25 smithi121 bash[36902]: cluster 2022-04-22T22:56:24.397442+0000 mgr.y (mgr.34107) 11258 : cluster [DBG] pgmap v11849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:25 smithi115 bash[45495]: cluster 2022-04-22T22:56:24.397442+0000 mgr.y (mgr.34107) 11258 : cluster [DBG] pgmap v11849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:25 smithi115 bash[46934]: cluster 2022-04-22T22:56:24.397442+0000 mgr.y (mgr.34107) 11258 : cluster [DBG] pgmap v11849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:26 smithi121 bash[36902]: audit 2022-04-22T22:56:25.848530+0000 mon.a (mon.0) 15617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:26.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:26 smithi115 bash[46934]: audit 2022-04-22T22:56:25.848530+0000 mon.a (mon.0) 15617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:26 smithi115 bash[45495]: audit 2022-04-22T22:56:25.848530+0000 mon.a (mon.0) 15617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:27 smithi121 bash[36902]: cluster 2022-04-22T22:56:26.398451+0000 mgr.y (mgr.34107) 11259 : cluster [DBG] pgmap v11850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:27 smithi115 bash[46934]: cluster 2022-04-22T22:56:26.398451+0000 mgr.y (mgr.34107) 11259 : cluster [DBG] pgmap v11850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:27 smithi115 bash[45495]: cluster 2022-04-22T22:56:26.398451+0000 mgr.y (mgr.34107) 11259 : cluster [DBG] pgmap v11850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:28 smithi121 bash[36902]: audit 2022-04-22T22:56:27.470250+0000 mon.a (mon.0) 15618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:28.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:28 smithi115 bash[46934]: audit 2022-04-22T22:56:27.470250+0000 mon.a (mon.0) 15618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:28 smithi115 bash[45495]: audit 2022-04-22T22:56:27.470250+0000 mon.a (mon.0) 15618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:29 smithi121 bash[36902]: cluster 2022-04-22T22:56:28.399926+0000 mgr.y (mgr.34107) 11260 : cluster [DBG] pgmap v11851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:29 smithi121 bash[36902]: audit 2022-04-22T22:56:28.999800+0000 mon.a (mon.0) 15619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:29 smithi115 bash[45495]: cluster 2022-04-22T22:56:28.399926+0000 mgr.y (mgr.34107) 11260 : cluster [DBG] pgmap v11851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:29 smithi115 bash[45495]: audit 2022-04-22T22:56:28.999800+0000 mon.a (mon.0) 15619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:29 smithi115 bash[46934]: cluster 2022-04-22T22:56:28.399926+0000 mgr.y (mgr.34107) 11260 : cluster [DBG] pgmap v11851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:29 smithi115 bash[46934]: audit 2022-04-22T22:56:28.999800+0000 mon.a (mon.0) 15619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:31 smithi121 bash[36902]: cluster 2022-04-22T22:56:30.401162+0000 mgr.y (mgr.34107) 11261 : cluster [DBG] pgmap v11852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:31 smithi121 bash[36902]: audit 2022-04-22T22:56:30.851307+0000 mon.a (mon.0) 15620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:31.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:31 smithi115 bash[46934]: cluster 2022-04-22T22:56:30.401162+0000 mgr.y (mgr.34107) 11261 : cluster [DBG] pgmap v11852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:31 smithi115 bash[46934]: audit 2022-04-22T22:56:30.851307+0000 mon.a (mon.0) 15620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:31 smithi115 bash[45495]: cluster 2022-04-22T22:56:30.401162+0000 mgr.y (mgr.34107) 11261 : cluster [DBG] pgmap v11852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:31 smithi115 bash[45495]: audit 2022-04-22T22:56:30.851307+0000 mon.a (mon.0) 15620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:32.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:33 smithi115 bash[45495]: cluster 2022-04-22T22:56:32.402329+0000 mgr.y (mgr.34107) 11262 : cluster [DBG] pgmap v11853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:33 smithi115 bash[45495]: audit 2022-04-22T22:56:32.473155+0000 mon.a (mon.0) 15621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:33 smithi115 bash[46934]: cluster 2022-04-22T22:56:32.402329+0000 mgr.y (mgr.34107) 11262 : cluster [DBG] pgmap v11853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:33 smithi115 bash[46934]: audit 2022-04-22T22:56:32.473155+0000 mon.a (mon.0) 15621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:33 smithi121 bash[36902]: cluster 2022-04-22T22:56:32.402329+0000 mgr.y (mgr.34107) 11262 : cluster [DBG] pgmap v11853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:33 smithi121 bash[36902]: audit 2022-04-22T22:56:32.473155+0000 mon.a (mon.0) 15621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:34 smithi121 bash[36902]: audit 2022-04-22T22:56:34.003296+0000 mon.a (mon.0) 15622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:34 smithi115 bash[45495]: audit 2022-04-22T22:56:34.003296+0000 mon.a (mon.0) 15622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:34 smithi115 bash[46934]: audit 2022-04-22T22:56:34.003296+0000 mon.a (mon.0) 15622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:34.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:35.276 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:56:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:35 smithi121 bash[36902]: cluster 2022-04-22T22:56:34.403416+0000 mgr.y (mgr.34107) 11263 : cluster [DBG] pgmap v11854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:35 smithi115 bash[45495]: cluster 2022-04-22T22:56:34.403416+0000 mgr.y (mgr.34107) 11263 : cluster [DBG] pgmap v11854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:35 smithi115 bash[46934]: cluster 2022-04-22T22:56:34.403416+0000 mgr.y (mgr.34107) 11263 : cluster [DBG] pgmap v11854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 27s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 27s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 27s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:56:35.619 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 27s ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 27s ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:56:35.620 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 27s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:56:35.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 27s ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:56:35.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 27s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:56:35.621 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 27s ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:56:35.621 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 27s ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:56:36.030 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:56:36.030 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:56:36.030 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:56:36.030 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:36.030 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:56:36.031 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:56:36.032 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:56:36.032 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:56:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:36 smithi121 bash[36902]: audit 2022-04-22T22:56:35.254549+0000 mgr.y (mgr.34107) 11264 : audit [DBG] from='client.56947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:36 smithi121 bash[36902]: audit 2022-04-22T22:56:35.595249+0000 mgr.y (mgr.34107) 11265 : audit [DBG] from='client.47016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:36 smithi121 bash[36902]: audit 2022-04-22T22:56:35.854300+0000 mon.a (mon.0) 15623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:36 smithi121 bash[36902]: audit 2022-04-22T22:56:36.026947+0000 mon.c (mon.1) 379 : audit [DBG] from='client.? 172.21.15.115:0/2339883412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[46934]: audit 2022-04-22T22:56:35.254549+0000 mgr.y (mgr.34107) 11264 : audit [DBG] from='client.56947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[46934]: audit 2022-04-22T22:56:35.595249+0000 mgr.y (mgr.34107) 11265 : audit [DBG] from='client.47016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[46934]: audit 2022-04-22T22:56:35.854300+0000 mon.a (mon.0) 15623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[46934]: audit 2022-04-22T22:56:36.026947+0000 mon.c (mon.1) 379 : audit [DBG] from='client.? 172.21.15.115:0/2339883412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[45495]: audit 2022-04-22T22:56:35.254549+0000 mgr.y (mgr.34107) 11264 : audit [DBG] from='client.56947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[45495]: audit 2022-04-22T22:56:35.595249+0000 mgr.y (mgr.34107) 11265 : audit [DBG] from='client.47016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[45495]: audit 2022-04-22T22:56:35.854300+0000 mon.a (mon.0) 15623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:36 smithi115 bash[45495]: audit 2022-04-22T22:56:36.026947+0000 mon.c (mon.1) 379 : audit [DBG] from='client.? 172.21.15.115:0/2339883412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:56:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:37 smithi121 bash[36902]: cluster 2022-04-22T22:56:36.404465+0000 mgr.y (mgr.34107) 11266 : cluster [DBG] pgmap v11855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:37.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:37 smithi115 bash[46934]: cluster 2022-04-22T22:56:36.404465+0000 mgr.y (mgr.34107) 11266 : cluster [DBG] pgmap v11855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:37.549 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:37 smithi115 bash[45495]: cluster 2022-04-22T22:56:36.404465+0000 mgr.y (mgr.34107) 11266 : cluster [DBG] pgmap v11855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:38 smithi121 bash[36902]: audit 2022-04-22T22:56:37.476575+0000 mon.a (mon.0) 15624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:38 smithi115 bash[46934]: audit 2022-04-22T22:56:37.476575+0000 mon.a (mon.0) 15624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:38 smithi115 bash[45495]: audit 2022-04-22T22:56:37.476575+0000 mon.a (mon.0) 15624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:39 smithi121 bash[36902]: cluster 2022-04-22T22:56:38.405622+0000 mgr.y (mgr.34107) 11267 : cluster [DBG] pgmap v11856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:39 smithi121 bash[36902]: audit 2022-04-22T22:56:39.006921+0000 mon.a (mon.0) 15625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:39.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:39 smithi115 bash[46934]: cluster 2022-04-22T22:56:38.405622+0000 mgr.y (mgr.34107) 11267 : cluster [DBG] pgmap v11856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:39 smithi115 bash[46934]: audit 2022-04-22T22:56:39.006921+0000 mon.a (mon.0) 15625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:39 smithi115 bash[45495]: cluster 2022-04-22T22:56:38.405622+0000 mgr.y (mgr.34107) 11267 : cluster [DBG] pgmap v11856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:39 smithi115 bash[45495]: audit 2022-04-22T22:56:39.006921+0000 mon.a (mon.0) 15625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:41 smithi121 bash[36902]: cluster 2022-04-22T22:56:40.406724+0000 mgr.y (mgr.34107) 11268 : cluster [DBG] pgmap v11857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:41 smithi121 bash[36902]: audit 2022-04-22T22:56:40.857149+0000 mon.a (mon.0) 15626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:41 smithi115 bash[46934]: cluster 2022-04-22T22:56:40.406724+0000 mgr.y (mgr.34107) 11268 : cluster [DBG] pgmap v11857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:41 smithi115 bash[46934]: audit 2022-04-22T22:56:40.857149+0000 mon.a (mon.0) 15626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:41 smithi115 bash[45495]: cluster 2022-04-22T22:56:40.406724+0000 mgr.y (mgr.34107) 11268 : cluster [DBG] pgmap v11857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:41 smithi115 bash[45495]: audit 2022-04-22T22:56:40.857149+0000 mon.a (mon.0) 15626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:42.916Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:43 smithi115 bash[45495]: cluster 2022-04-22T22:56:42.407785+0000 mgr.y (mgr.34107) 11269 : cluster [DBG] pgmap v11858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:43 smithi115 bash[45495]: audit 2022-04-22T22:56:42.479898+0000 mon.a (mon.0) 15627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:43 smithi115 bash[46934]: cluster 2022-04-22T22:56:42.407785+0000 mgr.y (mgr.34107) 11269 : cluster [DBG] pgmap v11858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:43 smithi115 bash[46934]: audit 2022-04-22T22:56:42.479898+0000 mon.a (mon.0) 15627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:43 smithi121 bash[36902]: cluster 2022-04-22T22:56:42.407785+0000 mgr.y (mgr.34107) 11269 : cluster [DBG] pgmap v11858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:43 smithi121 bash[36902]: audit 2022-04-22T22:56:42.479898+0000 mon.a (mon.0) 15627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:44 smithi121 bash[36902]: audit 2022-04-22T22:56:44.010577+0000 mon.a (mon.0) 15628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:44 smithi115 bash[45495]: audit 2022-04-22T22:56:44.010577+0000 mon.a (mon.0) 15628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:44 smithi115 bash[46934]: audit 2022-04-22T22:56:44.010577+0000 mon.a (mon.0) 15628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:44.775Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:45 smithi121 bash[36902]: cluster 2022-04-22T22:56:44.408867+0000 mgr.y (mgr.34107) 11270 : cluster [DBG] pgmap v11859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:45 smithi115 bash[45495]: cluster 2022-04-22T22:56:44.408867+0000 mgr.y (mgr.34107) 11270 : cluster [DBG] pgmap v11859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:45 smithi115 bash[46934]: cluster 2022-04-22T22:56:44.408867+0000 mgr.y (mgr.34107) 11270 : cluster [DBG] pgmap v11859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:46 smithi121 bash[36902]: audit 2022-04-22T22:56:45.860291+0000 mon.a (mon.0) 15629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:46 smithi115 bash[45495]: audit 2022-04-22T22:56:45.860291+0000 mon.a (mon.0) 15629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:46 smithi115 bash[46934]: audit 2022-04-22T22:56:45.860291+0000 mon.a (mon.0) 15629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:47 smithi121 bash[36902]: cluster 2022-04-22T22:56:46.409960+0000 mgr.y (mgr.34107) 11271 : cluster [DBG] pgmap v11860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:47 smithi115 bash[45495]: cluster 2022-04-22T22:56:46.409960+0000 mgr.y (mgr.34107) 11271 : cluster [DBG] pgmap v11860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:47 smithi115 bash[46934]: cluster 2022-04-22T22:56:46.409960+0000 mgr.y (mgr.34107) 11271 : cluster [DBG] pgmap v11860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:48 smithi121 bash[36902]: audit 2022-04-22T22:56:47.482986+0000 mon.a (mon.0) 15630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:48 smithi115 bash[45495]: audit 2022-04-22T22:56:47.482986+0000 mon.a (mon.0) 15630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:48 smithi115 bash[46934]: audit 2022-04-22T22:56:47.482986+0000 mon.a (mon.0) 15630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:49 smithi121 bash[36902]: cluster 2022-04-22T22:56:48.411116+0000 mgr.y (mgr.34107) 11272 : cluster [DBG] pgmap v11861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:49 smithi121 bash[36902]: audit 2022-04-22T22:56:49.014516+0000 mon.a (mon.0) 15631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:49 smithi115 bash[45495]: cluster 2022-04-22T22:56:48.411116+0000 mgr.y (mgr.34107) 11272 : cluster [DBG] pgmap v11861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:49 smithi115 bash[45495]: audit 2022-04-22T22:56:49.014516+0000 mon.a (mon.0) 15631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:49 smithi115 bash[46934]: cluster 2022-04-22T22:56:48.411116+0000 mgr.y (mgr.34107) 11272 : cluster [DBG] pgmap v11861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:49 smithi115 bash[46934]: audit 2022-04-22T22:56:49.014516+0000 mon.a (mon.0) 15631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:51 smithi121 bash[36902]: cluster 2022-04-22T22:56:50.412345+0000 mgr.y (mgr.34107) 11273 : cluster [DBG] pgmap v11862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:51 smithi121 bash[36902]: audit 2022-04-22T22:56:50.863133+0000 mon.a (mon.0) 15632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:51 smithi115 bash[45495]: cluster 2022-04-22T22:56:50.412345+0000 mgr.y (mgr.34107) 11273 : cluster [DBG] pgmap v11862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:51 smithi115 bash[45495]: audit 2022-04-22T22:56:50.863133+0000 mon.a (mon.0) 15632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:51 smithi115 bash[46934]: cluster 2022-04-22T22:56:50.412345+0000 mgr.y (mgr.34107) 11273 : cluster [DBG] pgmap v11862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:51 smithi115 bash[46934]: audit 2022-04-22T22:56:50.863133+0000 mon.a (mon.0) 15632 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:52.917Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:53 smithi115 bash[45495]: cluster 2022-04-22T22:56:52.413433+0000 mgr.y (mgr.34107) 11274 : cluster [DBG] pgmap v11863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:53 smithi115 bash[45495]: audit 2022-04-22T22:56:52.485996+0000 mon.a (mon.0) 15633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:53 smithi115 bash[46934]: cluster 2022-04-22T22:56:52.413433+0000 mgr.y (mgr.34107) 11274 : cluster [DBG] pgmap v11863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:53 smithi115 bash[46934]: audit 2022-04-22T22:56:52.485996+0000 mon.a (mon.0) 15633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:53 smithi121 bash[36902]: cluster 2022-04-22T22:56:52.413433+0000 mgr.y (mgr.34107) 11274 : cluster [DBG] pgmap v11863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:53 smithi121 bash[36902]: audit 2022-04-22T22:56:52.485996+0000 mon.a (mon.0) 15633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:54 smithi121 bash[36902]: audit 2022-04-22T22:56:54.018421+0000 mon.a (mon.0) 15634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:54 smithi115 bash[45495]: audit 2022-04-22T22:56:54.018421+0000 mon.a (mon.0) 15634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:54 smithi115 bash[46934]: audit 2022-04-22T22:56:54.018421+0000 mon.a (mon.0) 15634 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:56:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:56:54.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:56:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:55 smithi121 bash[36902]: cluster 2022-04-22T22:56:54.414522+0000 mgr.y (mgr.34107) 11275 : cluster [DBG] pgmap v11864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:55 smithi115 bash[45495]: cluster 2022-04-22T22:56:54.414522+0000 mgr.y (mgr.34107) 11275 : cluster [DBG] pgmap v11864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:55 smithi115 bash[46934]: cluster 2022-04-22T22:56:54.414522+0000 mgr.y (mgr.34107) 11275 : cluster [DBG] pgmap v11864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:56 smithi121 bash[36902]: audit 2022-04-22T22:56:55.866332+0000 mon.a (mon.0) 15635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:56 smithi115 bash[46934]: audit 2022-04-22T22:56:55.866332+0000 mon.a (mon.0) 15635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:56 smithi115 bash[45495]: audit 2022-04-22T22:56:55.866332+0000 mon.a (mon.0) 15635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:56:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:57 smithi121 bash[36902]: cluster 2022-04-22T22:56:56.415615+0000 mgr.y (mgr.34107) 11276 : cluster [DBG] pgmap v11865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:57 smithi115 bash[45495]: cluster 2022-04-22T22:56:56.415615+0000 mgr.y (mgr.34107) 11276 : cluster [DBG] pgmap v11865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:57 smithi115 bash[46934]: cluster 2022-04-22T22:56:56.415615+0000 mgr.y (mgr.34107) 11276 : cluster [DBG] pgmap v11865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:58 smithi121 bash[36902]: audit 2022-04-22T22:56:57.489068+0000 mon.a (mon.0) 15636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:58 smithi115 bash[45495]: audit 2022-04-22T22:56:57.489068+0000 mon.a (mon.0) 15636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:58 smithi115 bash[46934]: audit 2022-04-22T22:56:57.489068+0000 mon.a (mon.0) 15636 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:56:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:59 smithi121 bash[36902]: cluster 2022-04-22T22:56:58.416881+0000 mgr.y (mgr.34107) 11277 : cluster [DBG] pgmap v11866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:56:59 smithi121 bash[36902]: audit 2022-04-22T22:56:59.022045+0000 mon.a (mon.0) 15637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:59 smithi115 bash[45495]: cluster 2022-04-22T22:56:58.416881+0000 mgr.y (mgr.34107) 11277 : cluster [DBG] pgmap v11866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:56:59 smithi115 bash[45495]: audit 2022-04-22T22:56:59.022045+0000 mon.a (mon.0) 15637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:56:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:59 smithi115 bash[46934]: cluster 2022-04-22T22:56:58.416881+0000 mgr.y (mgr.34107) 11277 : cluster [DBG] pgmap v11866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:56:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:56:59 smithi115 bash[46934]: audit 2022-04-22T22:56:59.022045+0000 mon.a (mon.0) 15637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:01 smithi121 bash[36902]: cluster 2022-04-22T22:57:00.417983+0000 mgr.y (mgr.34107) 11278 : cluster [DBG] pgmap v11867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:01 smithi121 bash[36902]: audit 2022-04-22T22:57:00.869597+0000 mon.a (mon.0) 15638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:01 smithi115 bash[45495]: cluster 2022-04-22T22:57:00.417983+0000 mgr.y (mgr.34107) 11278 : cluster [DBG] pgmap v11867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:01 smithi115 bash[45495]: audit 2022-04-22T22:57:00.869597+0000 mon.a (mon.0) 15638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:01 smithi115 bash[46934]: cluster 2022-04-22T22:57:00.417983+0000 mgr.y (mgr.34107) 11278 : cluster [DBG] pgmap v11867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:01 smithi115 bash[46934]: audit 2022-04-22T22:57:00.869597+0000 mon.a (mon.0) 15638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:03.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:03 smithi115 bash[46934]: cluster 2022-04-22T22:57:02.419165+0000 mgr.y (mgr.34107) 11279 : cluster [DBG] pgmap v11868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:03 smithi115 bash[46934]: audit 2022-04-22T22:57:02.492131+0000 mon.a (mon.0) 15639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:02.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:03 smithi115 bash[45495]: cluster 2022-04-22T22:57:02.419165+0000 mgr.y (mgr.34107) 11279 : cluster [DBG] pgmap v11868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:03.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:03 smithi115 bash[45495]: audit 2022-04-22T22:57:02.492131+0000 mon.a (mon.0) 15639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:03 smithi121 bash[36902]: cluster 2022-04-22T22:57:02.419165+0000 mgr.y (mgr.34107) 11279 : cluster [DBG] pgmap v11868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:03 smithi121 bash[36902]: audit 2022-04-22T22:57:02.492131+0000 mon.a (mon.0) 15639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:04 smithi121 bash[36902]: audit 2022-04-22T22:57:04.025582+0000 mon.a (mon.0) 15640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:04 smithi115 bash[46934]: audit 2022-04-22T22:57:04.025582+0000 mon.a (mon.0) 15640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:04 smithi115 bash[45495]: audit 2022-04-22T22:57:04.025582+0000 mon.a (mon.0) 15640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:04.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:05 smithi121 bash[36902]: cluster 2022-04-22T22:57:04.420300+0000 mgr.y (mgr.34107) 11280 : cluster [DBG] pgmap v11869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:05 smithi115 bash[45495]: cluster 2022-04-22T22:57:04.420300+0000 mgr.y (mgr.34107) 11280 : cluster [DBG] pgmap v11869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:05 smithi115 bash[46934]: cluster 2022-04-22T22:57:04.420300+0000 mgr.y (mgr.34107) 11280 : cluster [DBG] pgmap v11869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:06.399 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:57:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:06 smithi121 bash[36902]: audit 2022-04-22T22:57:05.872639+0000 mon.a (mon.0) 15641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:06.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:06 smithi115 bash[46934]: audit 2022-04-22T22:57:05.872639+0000 mon.a (mon.0) 15641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:06 smithi115 bash[45495]: audit 2022-04-22T22:57:05.872639+0000 mon.a (mon.0) 15641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:06.747 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:57:06.747 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 10m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T22:57:06.747 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 58s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:57:06.747 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 10m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 58s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 58s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 10m ago 5h 493M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 10m ago 5h 447M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 58s ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 10m ago 5h 303M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 10m ago 5h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 58s ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 10m ago 5h 719M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:57:06.748 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 10m ago 5h 799M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 10m ago 5h 541M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 10m ago 5h 503M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 58s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 58s ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 58s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 58s ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:57:06.749 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 58s ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:57:07.141 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:57:07.141 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:57:07.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:57:07.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:07.142 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:57:07.143 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:57:07.143 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:57:07.143 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:57:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:07 smithi121 bash[36902]: audit 2022-04-22T22:57:06.378231+0000 mgr.y (mgr.34107) 11281 : audit [DBG] from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:07 smithi121 bash[36902]: cluster 2022-04-22T22:57:06.421373+0000 mgr.y (mgr.34107) 11282 : cluster [DBG] pgmap v11870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:07 smithi121 bash[36902]: audit 2022-04-22T22:57:06.723656+0000 mgr.y (mgr.34107) 11283 : audit [DBG] from='client.47034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:07 smithi121 bash[36902]: audit 2022-04-22T22:57:07.138117+0000 mon.a (mon.0) 15642 : audit [DBG] from='client.? 172.21.15.115:0/3841482070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[45495]: audit 2022-04-22T22:57:06.378231+0000 mgr.y (mgr.34107) 11281 : audit [DBG] from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[45495]: cluster 2022-04-22T22:57:06.421373+0000 mgr.y (mgr.34107) 11282 : cluster [DBG] pgmap v11870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[45495]: audit 2022-04-22T22:57:06.723656+0000 mgr.y (mgr.34107) 11283 : audit [DBG] from='client.47034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[45495]: audit 2022-04-22T22:57:07.138117+0000 mon.a (mon.0) 15642 : audit [DBG] from='client.? 172.21.15.115:0/3841482070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[46934]: audit 2022-04-22T22:57:06.378231+0000 mgr.y (mgr.34107) 11281 : audit [DBG] from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[46934]: cluster 2022-04-22T22:57:06.421373+0000 mgr.y (mgr.34107) 11282 : cluster [DBG] pgmap v11870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[46934]: audit 2022-04-22T22:57:06.723656+0000 mgr.y (mgr.34107) 11283 : audit [DBG] from='client.47034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:07 smithi115 bash[46934]: audit 2022-04-22T22:57:07.138117+0000 mon.a (mon.0) 15642 : audit [DBG] from='client.? 172.21.15.115:0/3841482070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:08 smithi121 bash[36902]: audit 2022-04-22T22:57:07.495124+0000 mon.a (mon.0) 15643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:08 smithi115 bash[45495]: audit 2022-04-22T22:57:07.495124+0000 mon.a (mon.0) 15643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:08 smithi115 bash[46934]: audit 2022-04-22T22:57:07.495124+0000 mon.a (mon.0) 15643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:08.230836+0000 mon.a (mon.0) 15644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:57:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:08.233579+0000 mon.a (mon.0) 15645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:57:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:08.235143+0000 mon.a (mon.0) 15646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:57:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: cluster 2022-04-22T22:57:08.422547+0000 mgr.y (mgr.34107) 11284 : cluster [DBG] pgmap v11871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:08.554024+0000 mon.a (mon.0) 15647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:08.573237+0000 mon.a (mon.0) 15648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:09 smithi121 bash[36902]: audit 2022-04-22T22:57:09.029597+0000 mon.a (mon.0) 15649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:08.230836+0000 mon.a (mon.0) 15644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:08.233579+0000 mon.a (mon.0) 15645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:08.235143+0000 mon.a (mon.0) 15646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: cluster 2022-04-22T22:57:08.422547+0000 mgr.y (mgr.34107) 11284 : cluster [DBG] pgmap v11871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:08.554024+0000 mon.a (mon.0) 15647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:08.573237+0000 mon.a (mon.0) 15648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[46934]: audit 2022-04-22T22:57:09.029597+0000 mon.a (mon.0) 15649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:08.230836+0000 mon.a (mon.0) 15644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:08.233579+0000 mon.a (mon.0) 15645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:08.235143+0000 mon.a (mon.0) 15646 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: cluster 2022-04-22T22:57:08.422547+0000 mgr.y (mgr.34107) 11284 : cluster [DBG] pgmap v11871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:08.554024+0000 mon.a (mon.0) 15647 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:08.573237+0000 mon.a (mon.0) 15648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:09.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:09 smithi115 bash[45495]: audit 2022-04-22T22:57:09.029597+0000 mon.a (mon.0) 15649 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:11 smithi121 bash[36902]: cluster 2022-04-22T22:57:10.423688+0000 mgr.y (mgr.34107) 11285 : cluster [DBG] pgmap v11872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:11.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:11 smithi121 bash[36902]: audit 2022-04-22T22:57:10.875421+0000 mon.a (mon.0) 15650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:11 smithi115 bash[46934]: cluster 2022-04-22T22:57:10.423688+0000 mgr.y (mgr.34107) 11285 : cluster [DBG] pgmap v11872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:11.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:11 smithi115 bash[46934]: audit 2022-04-22T22:57:10.875421+0000 mon.a (mon.0) 15650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:11 smithi115 bash[45495]: cluster 2022-04-22T22:57:10.423688+0000 mgr.y (mgr.34107) 11285 : cluster [DBG] pgmap v11872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:11 smithi115 bash[45495]: audit 2022-04-22T22:57:10.875421+0000 mon.a (mon.0) 15650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:12.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[45495]: cluster 2022-04-22T22:57:12.424619+0000 mgr.y (mgr.34107) 11286 : cluster [DBG] pgmap v11873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[45495]: audit 2022-04-22T22:57:12.497871+0000 mon.a (mon.0) 15651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[45495]: audit 2022-04-22T22:57:12.667449+0000 mon.a (mon.0) 15652 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[45495]: audit 2022-04-22T22:57:12.991910+0000 mon.a (mon.0) 15653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[46934]: cluster 2022-04-22T22:57:12.424619+0000 mgr.y (mgr.34107) 11286 : cluster [DBG] pgmap v11873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[46934]: audit 2022-04-22T22:57:12.497871+0000 mon.a (mon.0) 15651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[46934]: audit 2022-04-22T22:57:12.667449+0000 mon.a (mon.0) 15652 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:13 smithi115 bash[46934]: audit 2022-04-22T22:57:12.991910+0000 mon.a (mon.0) 15653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:13 smithi121 bash[36902]: cluster 2022-04-22T22:57:12.424619+0000 mgr.y (mgr.34107) 11286 : cluster [DBG] pgmap v11873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:13 smithi121 bash[36902]: audit 2022-04-22T22:57:12.497871+0000 mon.a (mon.0) 15651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:13 smithi121 bash[36902]: audit 2022-04-22T22:57:12.667449+0000 mon.a (mon.0) 15652 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:13 smithi121 bash[36902]: audit 2022-04-22T22:57:12.991910+0000 mon.a (mon.0) 15653 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:57:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:14 smithi121 bash[36902]: audit 2022-04-22T22:57:14.033087+0000 mon.a (mon.0) 15654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:14 smithi115 bash[45495]: audit 2022-04-22T22:57:14.033087+0000 mon.a (mon.0) 15654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:14 smithi115 bash[46934]: audit 2022-04-22T22:57:14.033087+0000 mon.a (mon.0) 15654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:14.777Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:15 smithi121 bash[36902]: cluster 2022-04-22T22:57:14.425712+0000 mgr.y (mgr.34107) 11287 : cluster [DBG] pgmap v11874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:15 smithi115 bash[45495]: cluster 2022-04-22T22:57:14.425712+0000 mgr.y (mgr.34107) 11287 : cluster [DBG] pgmap v11874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:15 smithi115 bash[46934]: cluster 2022-04-22T22:57:14.425712+0000 mgr.y (mgr.34107) 11287 : cluster [DBG] pgmap v11874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:16 smithi121 bash[36902]: audit 2022-04-22T22:57:15.877909+0000 mon.a (mon.0) 15655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:16 smithi115 bash[45495]: audit 2022-04-22T22:57:15.877909+0000 mon.a (mon.0) 15655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:16 smithi115 bash[46934]: audit 2022-04-22T22:57:15.877909+0000 mon.a (mon.0) 15655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:17 smithi121 bash[36902]: cluster 2022-04-22T22:57:16.426786+0000 mgr.y (mgr.34107) 11288 : cluster [DBG] pgmap v11875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:17 smithi121 bash[36902]: audit 2022-04-22T22:57:17.079049+0000 mon.a (mon.0) 15656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:57:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:17 smithi121 bash[36902]: audit 2022-04-22T22:57:17.079602+0000 mon.a (mon.0) 15657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:57:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[45495]: cluster 2022-04-22T22:57:16.426786+0000 mgr.y (mgr.34107) 11288 : cluster [DBG] pgmap v11875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[45495]: audit 2022-04-22T22:57:17.079049+0000 mon.a (mon.0) 15656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:57:17.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[45495]: audit 2022-04-22T22:57:17.079602+0000 mon.a (mon.0) 15657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:57:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[46934]: cluster 2022-04-22T22:57:16.426786+0000 mgr.y (mgr.34107) 11288 : cluster [DBG] pgmap v11875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:17.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[46934]: audit 2022-04-22T22:57:17.079049+0000 mon.a (mon.0) 15656 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:57:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:17 smithi115 bash[46934]: audit 2022-04-22T22:57:17.079602+0000 mon.a (mon.0) 15657 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:57:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:18 smithi121 bash[36902]: audit 2022-04-22T22:57:17.500736+0000 mon.a (mon.0) 15658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:18.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:18 smithi115 bash[45495]: audit 2022-04-22T22:57:17.500736+0000 mon.a (mon.0) 15658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:18 smithi115 bash[46934]: audit 2022-04-22T22:57:17.500736+0000 mon.a (mon.0) 15658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:19 smithi121 bash[36902]: cluster 2022-04-22T22:57:18.427945+0000 mgr.y (mgr.34107) 11289 : cluster [DBG] pgmap v11876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:19 smithi121 bash[36902]: audit 2022-04-22T22:57:19.036471+0000 mon.a (mon.0) 15659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:19 smithi115 bash[45495]: cluster 2022-04-22T22:57:18.427945+0000 mgr.y (mgr.34107) 11289 : cluster [DBG] pgmap v11876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:19 smithi115 bash[45495]: audit 2022-04-22T22:57:19.036471+0000 mon.a (mon.0) 15659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:19 smithi115 bash[46934]: cluster 2022-04-22T22:57:18.427945+0000 mgr.y (mgr.34107) 11289 : cluster [DBG] pgmap v11876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:19 smithi115 bash[46934]: audit 2022-04-22T22:57:19.036471+0000 mon.a (mon.0) 15659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:21 smithi121 bash[36902]: cluster 2022-04-22T22:57:20.428946+0000 mgr.y (mgr.34107) 11290 : cluster [DBG] pgmap v11877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:21 smithi121 bash[36902]: audit 2022-04-22T22:57:20.881038+0000 mon.a (mon.0) 15660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:21.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:21 smithi115 bash[46934]: cluster 2022-04-22T22:57:20.428946+0000 mgr.y (mgr.34107) 11290 : cluster [DBG] pgmap v11877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:21 smithi115 bash[46934]: audit 2022-04-22T22:57:20.881038+0000 mon.a (mon.0) 15660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:21 smithi115 bash[45495]: cluster 2022-04-22T22:57:20.428946+0000 mgr.y (mgr.34107) 11290 : cluster [DBG] pgmap v11877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:21 smithi115 bash[45495]: audit 2022-04-22T22:57:20.881038+0000 mon.a (mon.0) 15660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:22.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:23 smithi115 bash[45495]: cluster 2022-04-22T22:57:22.429960+0000 mgr.y (mgr.34107) 11291 : cluster [DBG] pgmap v11878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:23 smithi115 bash[45495]: audit 2022-04-22T22:57:22.503787+0000 mon.a (mon.0) 15661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:23 smithi115 bash[46934]: cluster 2022-04-22T22:57:22.429960+0000 mgr.y (mgr.34107) 11291 : cluster [DBG] pgmap v11878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:23 smithi115 bash[46934]: audit 2022-04-22T22:57:22.503787+0000 mon.a (mon.0) 15661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:23 smithi121 bash[36902]: cluster 2022-04-22T22:57:22.429960+0000 mgr.y (mgr.34107) 11291 : cluster [DBG] pgmap v11878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:23 smithi121 bash[36902]: audit 2022-04-22T22:57:22.503787+0000 mon.a (mon.0) 15661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:24 smithi121 bash[36902]: audit 2022-04-22T22:57:24.040197+0000 mon.a (mon.0) 15662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:24 smithi115 bash[45495]: audit 2022-04-22T22:57:24.040197+0000 mon.a (mon.0) 15662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:24 smithi115 bash[46934]: audit 2022-04-22T22:57:24.040197+0000 mon.a (mon.0) 15662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:24.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:25 smithi121 bash[36902]: cluster 2022-04-22T22:57:24.431076+0000 mgr.y (mgr.34107) 11292 : cluster [DBG] pgmap v11879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:25 smithi115 bash[45495]: cluster 2022-04-22T22:57:24.431076+0000 mgr.y (mgr.34107) 11292 : cluster [DBG] pgmap v11879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:25 smithi115 bash[46934]: cluster 2022-04-22T22:57:24.431076+0000 mgr.y (mgr.34107) 11292 : cluster [DBG] pgmap v11879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:26 smithi121 bash[36902]: audit 2022-04-22T22:57:25.884190+0000 mon.a (mon.0) 15663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:26 smithi115 bash[45495]: audit 2022-04-22T22:57:25.884190+0000 mon.a (mon.0) 15663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:26 smithi115 bash[46934]: audit 2022-04-22T22:57:25.884190+0000 mon.a (mon.0) 15663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:27 smithi121 bash[36902]: cluster 2022-04-22T22:57:26.432170+0000 mgr.y (mgr.34107) 11293 : cluster [DBG] pgmap v11880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:27 smithi115 bash[45495]: cluster 2022-04-22T22:57:26.432170+0000 mgr.y (mgr.34107) 11293 : cluster [DBG] pgmap v11880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:27 smithi115 bash[46934]: cluster 2022-04-22T22:57:26.432170+0000 mgr.y (mgr.34107) 11293 : cluster [DBG] pgmap v11880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:28 smithi121 bash[36902]: audit 2022-04-22T22:57:27.506886+0000 mon.a (mon.0) 15664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:28 smithi115 bash[45495]: audit 2022-04-22T22:57:27.506886+0000 mon.a (mon.0) 15664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:28 smithi115 bash[46934]: audit 2022-04-22T22:57:27.506886+0000 mon.a (mon.0) 15664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:29 smithi121 bash[36902]: cluster 2022-04-22T22:57:28.433353+0000 mgr.y (mgr.34107) 11294 : cluster [DBG] pgmap v11881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:29 smithi121 bash[36902]: audit 2022-04-22T22:57:29.043855+0000 mon.a (mon.0) 15665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:29 smithi115 bash[45495]: cluster 2022-04-22T22:57:28.433353+0000 mgr.y (mgr.34107) 11294 : cluster [DBG] pgmap v11881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:29 smithi115 bash[45495]: audit 2022-04-22T22:57:29.043855+0000 mon.a (mon.0) 15665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:29 smithi115 bash[46934]: cluster 2022-04-22T22:57:28.433353+0000 mgr.y (mgr.34107) 11294 : cluster [DBG] pgmap v11881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:29 smithi115 bash[46934]: audit 2022-04-22T22:57:29.043855+0000 mon.a (mon.0) 15665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:31 smithi121 bash[36902]: cluster 2022-04-22T22:57:30.434430+0000 mgr.y (mgr.34107) 11295 : cluster [DBG] pgmap v11882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:31.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:31 smithi121 bash[36902]: audit 2022-04-22T22:57:30.887015+0000 mon.a (mon.0) 15666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:31 smithi115 bash[45495]: cluster 2022-04-22T22:57:30.434430+0000 mgr.y (mgr.34107) 11295 : cluster [DBG] pgmap v11882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:31 smithi115 bash[45495]: audit 2022-04-22T22:57:30.887015+0000 mon.a (mon.0) 15666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:31 smithi115 bash[46934]: cluster 2022-04-22T22:57:30.434430+0000 mgr.y (mgr.34107) 11295 : cluster [DBG] pgmap v11882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:31 smithi115 bash[46934]: audit 2022-04-22T22:57:30.887015+0000 mon.a (mon.0) 15666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:32.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:33 smithi115 bash[45495]: cluster 2022-04-22T22:57:32.435528+0000 mgr.y (mgr.34107) 11296 : cluster [DBG] pgmap v11883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:33 smithi115 bash[45495]: audit 2022-04-22T22:57:32.510010+0000 mon.a (mon.0) 15667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:33 smithi115 bash[46934]: cluster 2022-04-22T22:57:32.435528+0000 mgr.y (mgr.34107) 11296 : cluster [DBG] pgmap v11883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:33 smithi115 bash[46934]: audit 2022-04-22T22:57:32.510010+0000 mon.a (mon.0) 15667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:33 smithi121 bash[36902]: cluster 2022-04-22T22:57:32.435528+0000 mgr.y (mgr.34107) 11296 : cluster [DBG] pgmap v11883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:33 smithi121 bash[36902]: audit 2022-04-22T22:57:32.510010+0000 mon.a (mon.0) 15667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:34 smithi121 bash[36902]: audit 2022-04-22T22:57:34.047929+0000 mon.a (mon.0) 15668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:34 smithi115 bash[45495]: audit 2022-04-22T22:57:34.047929+0000 mon.a (mon.0) 15668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:34 smithi115 bash[46934]: audit 2022-04-22T22:57:34.047929+0000 mon.a (mon.0) 15668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:34.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:35 smithi121 bash[36902]: cluster 2022-04-22T22:57:34.436633+0000 mgr.y (mgr.34107) 11297 : cluster [DBG] pgmap v11884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:35.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:35 smithi115 bash[46934]: cluster 2022-04-22T22:57:34.436633+0000 mgr.y (mgr.34107) 11297 : cluster [DBG] pgmap v11884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:35 smithi115 bash[45495]: cluster 2022-04-22T22:57:34.436633+0000 mgr.y (mgr.34107) 11297 : cluster [DBG] pgmap v11884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:36 smithi121 bash[36902]: audit 2022-04-22T22:57:35.890040+0000 mon.a (mon.0) 15669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:36 smithi115 bash[45495]: audit 2022-04-22T22:57:35.890040+0000 mon.a (mon.0) 15669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:36 smithi115 bash[46934]: audit 2022-04-22T22:57:35.890040+0000 mon.a (mon.0) 15669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:37.511 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:57:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:37 smithi121 bash[36902]: cluster 2022-04-22T22:57:36.437678+0000 mgr.y (mgr.34107) 11298 : cluster [DBG] pgmap v11885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:37.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:37 smithi115 bash[46934]: cluster 2022-04-22T22:57:36.437678+0000 mgr.y (mgr.34107) 11298 : cluster [DBG] pgmap v11885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:37 smithi115 bash[45495]: cluster 2022-04-22T22:57:36.437678+0000 mgr.y (mgr.34107) 11298 : cluster [DBG] pgmap v11885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:37.857 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 25s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 89s ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 25s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 89s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 89s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:57:37.858 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 25s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 25s ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 89s ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 25s ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 25s ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 89s ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 25s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 25s ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:57:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 25s ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 25s ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 89s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 89s ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 89s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 89s ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:57:37.860 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 89s ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:57:38.263 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:57:38.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:57:38.265 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:57:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:38 smithi121 bash[36902]: audit 2022-04-22T22:57:37.493992+0000 mgr.y (mgr.34107) 11299 : audit [DBG] from='client.47046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:38 smithi121 bash[36902]: audit 2022-04-22T22:57:37.512737+0000 mon.a (mon.0) 15670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:38 smithi121 bash[36902]: audit 2022-04-22T22:57:37.835678+0000 mgr.y (mgr.34107) 11300 : audit [DBG] from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[45495]: audit 2022-04-22T22:57:37.493992+0000 mgr.y (mgr.34107) 11299 : audit [DBG] from='client.47046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[45495]: audit 2022-04-22T22:57:37.512737+0000 mon.a (mon.0) 15670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[45495]: audit 2022-04-22T22:57:37.835678+0000 mgr.y (mgr.34107) 11300 : audit [DBG] from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[46934]: audit 2022-04-22T22:57:37.493992+0000 mgr.y (mgr.34107) 11299 : audit [DBG] from='client.47046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[46934]: audit 2022-04-22T22:57:37.512737+0000 mon.a (mon.0) 15670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:38 smithi115 bash[46934]: audit 2022-04-22T22:57:37.835678+0000 mgr.y (mgr.34107) 11300 : audit [DBG] from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:57:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:39 smithi121 bash[36902]: audit 2022-04-22T22:57:38.260269+0000 mon.c (mon.1) 380 : audit [DBG] from='client.? 172.21.15.115:0/1696937325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:39 smithi121 bash[36902]: cluster 2022-04-22T22:57:38.438858+0000 mgr.y (mgr.34107) 11301 : cluster [DBG] pgmap v11886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:39 smithi121 bash[36902]: audit 2022-04-22T22:57:39.051644+0000 mon.a (mon.0) 15671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:39.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[46934]: audit 2022-04-22T22:57:38.260269+0000 mon.c (mon.1) 380 : audit [DBG] from='client.? 172.21.15.115:0/1696937325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[46934]: cluster 2022-04-22T22:57:38.438858+0000 mgr.y (mgr.34107) 11301 : cluster [DBG] pgmap v11886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[46934]: audit 2022-04-22T22:57:39.051644+0000 mon.a (mon.0) 15671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[45495]: audit 2022-04-22T22:57:38.260269+0000 mon.c (mon.1) 380 : audit [DBG] from='client.? 172.21.15.115:0/1696937325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:57:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[45495]: cluster 2022-04-22T22:57:38.438858+0000 mgr.y (mgr.34107) 11301 : cluster [DBG] pgmap v11886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:39 smithi115 bash[45495]: audit 2022-04-22T22:57:39.051644+0000 mon.a (mon.0) 15671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:41 smithi121 bash[36902]: cluster 2022-04-22T22:57:40.439957+0000 mgr.y (mgr.34107) 11302 : cluster [DBG] pgmap v11887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:41 smithi121 bash[36902]: audit 2022-04-22T22:57:40.893174+0000 mon.a (mon.0) 15672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:41 smithi115 bash[46934]: cluster 2022-04-22T22:57:40.439957+0000 mgr.y (mgr.34107) 11302 : cluster [DBG] pgmap v11887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:41.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:41 smithi115 bash[46934]: audit 2022-04-22T22:57:40.893174+0000 mon.a (mon.0) 15672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:41 smithi115 bash[45495]: cluster 2022-04-22T22:57:40.439957+0000 mgr.y (mgr.34107) 11302 : cluster [DBG] pgmap v11887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:41 smithi115 bash[45495]: audit 2022-04-22T22:57:40.893174+0000 mon.a (mon.0) 15672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:42.921Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:43 smithi115 bash[45495]: cluster 2022-04-22T22:57:42.441113+0000 mgr.y (mgr.34107) 11303 : cluster [DBG] pgmap v11888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:43 smithi115 bash[45495]: audit 2022-04-22T22:57:42.515792+0000 mon.a (mon.0) 15673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:43 smithi115 bash[46934]: cluster 2022-04-22T22:57:42.441113+0000 mgr.y (mgr.34107) 11303 : cluster [DBG] pgmap v11888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:43 smithi115 bash[46934]: audit 2022-04-22T22:57:42.515792+0000 mon.a (mon.0) 15673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:43 smithi121 bash[36902]: cluster 2022-04-22T22:57:42.441113+0000 mgr.y (mgr.34107) 11303 : cluster [DBG] pgmap v11888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:43 smithi121 bash[36902]: audit 2022-04-22T22:57:42.515792+0000 mon.a (mon.0) 15673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:44 smithi121 bash[36902]: audit 2022-04-22T22:57:44.055269+0000 mon.a (mon.0) 15674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:44 smithi115 bash[45495]: audit 2022-04-22T22:57:44.055269+0000 mon.a (mon.0) 15674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:44.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:44 smithi115 bash[46934]: audit 2022-04-22T22:57:44.055269+0000 mon.a (mon.0) 15674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:44.778Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:45 smithi121 bash[36902]: cluster 2022-04-22T22:57:44.442231+0000 mgr.y (mgr.34107) 11304 : cluster [DBG] pgmap v11889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:45 smithi115 bash[45495]: cluster 2022-04-22T22:57:44.442231+0000 mgr.y (mgr.34107) 11304 : cluster [DBG] pgmap v11889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:45 smithi115 bash[46934]: cluster 2022-04-22T22:57:44.442231+0000 mgr.y (mgr.34107) 11304 : cluster [DBG] pgmap v11889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:46 smithi121 bash[36902]: audit 2022-04-22T22:57:45.896053+0000 mon.a (mon.0) 15675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:46 smithi115 bash[45495]: audit 2022-04-22T22:57:45.896053+0000 mon.a (mon.0) 15675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:46 smithi115 bash[46934]: audit 2022-04-22T22:57:45.896053+0000 mon.a (mon.0) 15675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:47 smithi121 bash[36902]: cluster 2022-04-22T22:57:46.443314+0000 mgr.y (mgr.34107) 11305 : cluster [DBG] pgmap v11890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:47 smithi115 bash[45495]: cluster 2022-04-22T22:57:46.443314+0000 mgr.y (mgr.34107) 11305 : cluster [DBG] pgmap v11890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:47 smithi115 bash[46934]: cluster 2022-04-22T22:57:46.443314+0000 mgr.y (mgr.34107) 11305 : cluster [DBG] pgmap v11890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:48 smithi121 bash[36902]: audit 2022-04-22T22:57:47.518918+0000 mon.a (mon.0) 15676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:48 smithi115 bash[45495]: audit 2022-04-22T22:57:47.518918+0000 mon.a (mon.0) 15676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:48 smithi115 bash[46934]: audit 2022-04-22T22:57:47.518918+0000 mon.a (mon.0) 15676 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:49 smithi121 bash[36902]: cluster 2022-04-22T22:57:48.444519+0000 mgr.y (mgr.34107) 11306 : cluster [DBG] pgmap v11891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:49 smithi121 bash[36902]: audit 2022-04-22T22:57:49.058627+0000 mon.a (mon.0) 15677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:49 smithi115 bash[45495]: cluster 2022-04-22T22:57:48.444519+0000 mgr.y (mgr.34107) 11306 : cluster [DBG] pgmap v11891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:49 smithi115 bash[45495]: audit 2022-04-22T22:57:49.058627+0000 mon.a (mon.0) 15677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:49 smithi115 bash[46934]: cluster 2022-04-22T22:57:48.444519+0000 mgr.y (mgr.34107) 11306 : cluster [DBG] pgmap v11891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:49 smithi115 bash[46934]: audit 2022-04-22T22:57:49.058627+0000 mon.a (mon.0) 15677 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:51 smithi121 bash[36902]: cluster 2022-04-22T22:57:50.445612+0000 mgr.y (mgr.34107) 11307 : cluster [DBG] pgmap v11892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:51 smithi121 bash[36902]: audit 2022-04-22T22:57:50.899241+0000 mon.a (mon.0) 15678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:51 smithi115 bash[45495]: cluster 2022-04-22T22:57:50.445612+0000 mgr.y (mgr.34107) 11307 : cluster [DBG] pgmap v11892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:51 smithi115 bash[45495]: audit 2022-04-22T22:57:50.899241+0000 mon.a (mon.0) 15678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:51 smithi115 bash[46934]: cluster 2022-04-22T22:57:50.445612+0000 mgr.y (mgr.34107) 11307 : cluster [DBG] pgmap v11892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:51 smithi115 bash[46934]: audit 2022-04-22T22:57:50.899241+0000 mon.a (mon.0) 15678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:52.922Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:53 smithi115 bash[45495]: cluster 2022-04-22T22:57:52.446701+0000 mgr.y (mgr.34107) 11308 : cluster [DBG] pgmap v11893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:53 smithi115 bash[45495]: audit 2022-04-22T22:57:52.521990+0000 mon.a (mon.0) 15679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:53 smithi115 bash[46934]: cluster 2022-04-22T22:57:52.446701+0000 mgr.y (mgr.34107) 11308 : cluster [DBG] pgmap v11893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:53 smithi115 bash[46934]: audit 2022-04-22T22:57:52.521990+0000 mon.a (mon.0) 15679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:53 smithi121 bash[36902]: cluster 2022-04-22T22:57:52.446701+0000 mgr.y (mgr.34107) 11308 : cluster [DBG] pgmap v11893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:53 smithi121 bash[36902]: audit 2022-04-22T22:57:52.521990+0000 mon.a (mon.0) 15679 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:54 smithi121 bash[36902]: audit 2022-04-22T22:57:54.062405+0000 mon.a (mon.0) 15680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:54 smithi115 bash[45495]: audit 2022-04-22T22:57:54.062405+0000 mon.a (mon.0) 15680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:54 smithi115 bash[46934]: audit 2022-04-22T22:57:54.062405+0000 mon.a (mon.0) 15680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:57:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:57:54.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:57:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:55 smithi121 bash[36902]: cluster 2022-04-22T22:57:54.447832+0000 mgr.y (mgr.34107) 11309 : cluster [DBG] pgmap v11894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:55 smithi115 bash[45495]: cluster 2022-04-22T22:57:54.447832+0000 mgr.y (mgr.34107) 11309 : cluster [DBG] pgmap v11894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:55 smithi115 bash[46934]: cluster 2022-04-22T22:57:54.447832+0000 mgr.y (mgr.34107) 11309 : cluster [DBG] pgmap v11894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:56 smithi121 bash[36902]: audit 2022-04-22T22:57:55.902406+0000 mon.a (mon.0) 15681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:56 smithi115 bash[46934]: audit 2022-04-22T22:57:55.902406+0000 mon.a (mon.0) 15681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:56 smithi115 bash[45495]: audit 2022-04-22T22:57:55.902406+0000 mon.a (mon.0) 15681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:57:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:57 smithi121 bash[36902]: cluster 2022-04-22T22:57:56.448957+0000 mgr.y (mgr.34107) 11310 : cluster [DBG] pgmap v11895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:57 smithi115 bash[45495]: cluster 2022-04-22T22:57:56.448957+0000 mgr.y (mgr.34107) 11310 : cluster [DBG] pgmap v11895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:57 smithi115 bash[46934]: cluster 2022-04-22T22:57:56.448957+0000 mgr.y (mgr.34107) 11310 : cluster [DBG] pgmap v11895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:58 smithi121 bash[36902]: audit 2022-04-22T22:57:57.525309+0000 mon.a (mon.0) 15682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:58 smithi115 bash[45495]: audit 2022-04-22T22:57:57.525309+0000 mon.a (mon.0) 15682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:58 smithi115 bash[46934]: audit 2022-04-22T22:57:57.525309+0000 mon.a (mon.0) 15682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:57:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:59 smithi121 bash[36902]: cluster 2022-04-22T22:57:58.450098+0000 mgr.y (mgr.34107) 11311 : cluster [DBG] pgmap v11896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:57:59 smithi121 bash[36902]: audit 2022-04-22T22:57:59.066138+0000 mon.a (mon.0) 15683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:59 smithi115 bash[45495]: cluster 2022-04-22T22:57:58.450098+0000 mgr.y (mgr.34107) 11311 : cluster [DBG] pgmap v11896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:57:59 smithi115 bash[45495]: audit 2022-04-22T22:57:59.066138+0000 mon.a (mon.0) 15683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:57:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:59 smithi115 bash[46934]: cluster 2022-04-22T22:57:58.450098+0000 mgr.y (mgr.34107) 11311 : cluster [DBG] pgmap v11896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:57:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:57:59 smithi115 bash[46934]: audit 2022-04-22T22:57:59.066138+0000 mon.a (mon.0) 15683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:01 smithi121 bash[36902]: cluster 2022-04-22T22:58:00.451108+0000 mgr.y (mgr.34107) 11312 : cluster [DBG] pgmap v11897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:01 smithi121 bash[36902]: audit 2022-04-22T22:58:00.905612+0000 mon.a (mon.0) 15684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:01.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:01 smithi115 bash[46934]: cluster 2022-04-22T22:58:00.451108+0000 mgr.y (mgr.34107) 11312 : cluster [DBG] pgmap v11897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:01 smithi115 bash[46934]: audit 2022-04-22T22:58:00.905612+0000 mon.a (mon.0) 15684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:01 smithi115 bash[45495]: cluster 2022-04-22T22:58:00.451108+0000 mgr.y (mgr.34107) 11312 : cluster [DBG] pgmap v11897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:01 smithi115 bash[45495]: audit 2022-04-22T22:58:00.905612+0000 mon.a (mon.0) 15684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:02.924Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:03 smithi115 bash[45495]: cluster 2022-04-22T22:58:02.452154+0000 mgr.y (mgr.34107) 11313 : cluster [DBG] pgmap v11898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:03 smithi115 bash[45495]: audit 2022-04-22T22:58:02.528605+0000 mon.a (mon.0) 15685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:03 smithi115 bash[46934]: cluster 2022-04-22T22:58:02.452154+0000 mgr.y (mgr.34107) 11313 : cluster [DBG] pgmap v11898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:03 smithi115 bash[46934]: audit 2022-04-22T22:58:02.528605+0000 mon.a (mon.0) 15685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:03 smithi121 bash[36902]: cluster 2022-04-22T22:58:02.452154+0000 mgr.y (mgr.34107) 11313 : cluster [DBG] pgmap v11898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:03 smithi121 bash[36902]: audit 2022-04-22T22:58:02.528605+0000 mon.a (mon.0) 15685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:04 smithi121 bash[36902]: audit 2022-04-22T22:58:04.069945+0000 mon.a (mon.0) 15686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:04 smithi115 bash[45495]: audit 2022-04-22T22:58:04.069945+0000 mon.a (mon.0) 15686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:04.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:04 smithi115 bash[46934]: audit 2022-04-22T22:58:04.069945+0000 mon.a (mon.0) 15686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:04.780Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:05 smithi121 bash[36902]: cluster 2022-04-22T22:58:04.453216+0000 mgr.y (mgr.34107) 11314 : cluster [DBG] pgmap v11899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:05.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:05 smithi115 bash[46934]: cluster 2022-04-22T22:58:04.453216+0000 mgr.y (mgr.34107) 11314 : cluster [DBG] pgmap v11899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:05 smithi115 bash[45495]: cluster 2022-04-22T22:58:04.453216+0000 mgr.y (mgr.34107) 11314 : cluster [DBG] pgmap v11899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:06 smithi121 bash[36902]: audit 2022-04-22T22:58:05.908752+0000 mon.a (mon.0) 15687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:06 smithi115 bash[45495]: audit 2022-04-22T22:58:05.908752+0000 mon.a (mon.0) 15687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:06 smithi115 bash[46934]: audit 2022-04-22T22:58:05.908752+0000 mon.a (mon.0) 15687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:07 smithi121 bash[36902]: cluster 2022-04-22T22:58:06.454265+0000 mgr.y (mgr.34107) 11315 : cluster [DBG] pgmap v11900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:07 smithi115 bash[45495]: cluster 2022-04-22T22:58:06.454265+0000 mgr.y (mgr.34107) 11315 : cluster [DBG] pgmap v11900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:07 smithi115 bash[46934]: cluster 2022-04-22T22:58:06.454265+0000 mgr.y (mgr.34107) 11315 : cluster [DBG] pgmap v11900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:08 smithi121 bash[36902]: audit 2022-04-22T22:58:07.531617+0000 mon.a (mon.0) 15688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:08.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:08 smithi115 bash[46934]: audit 2022-04-22T22:58:07.531617+0000 mon.a (mon.0) 15688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:08.617 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:08 smithi115 bash[45495]: audit 2022-04-22T22:58:07.531617+0000 mon.a (mon.0) 15688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:08.635 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 56s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 56s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:58:08.978 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 56s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 56s ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 56s ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 56s ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 56s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 56s ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 56s ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:58:08.979 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 56s ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:58:08.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:58:08.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:58:08.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:58:08.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:58:08.980 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:58:09.392 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:58:09.392 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:58:09.392 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:58:09.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:58:09.393 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:58:09.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:09.394 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:58:09.394 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:58:09.394 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:58:09.394 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:58:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:09 smithi121 bash[36902]: cluster 2022-04-22T22:58:08.455290+0000 mgr.y (mgr.34107) 11316 : cluster [DBG] pgmap v11901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:09 smithi121 bash[36902]: audit 2022-04-22T22:58:08.616486+0000 mgr.y (mgr.34107) 11317 : audit [DBG] from='client.47064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:09 smithi121 bash[36902]: audit 2022-04-22T22:58:08.952969+0000 mgr.y (mgr.34107) 11318 : audit [DBG] from='client.57007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:09 smithi121 bash[36902]: audit 2022-04-22T22:58:09.073532+0000 mon.a (mon.0) 15689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[45495]: cluster 2022-04-22T22:58:08.455290+0000 mgr.y (mgr.34107) 11316 : cluster [DBG] pgmap v11901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[45495]: audit 2022-04-22T22:58:08.616486+0000 mgr.y (mgr.34107) 11317 : audit [DBG] from='client.47064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[45495]: audit 2022-04-22T22:58:08.952969+0000 mgr.y (mgr.34107) 11318 : audit [DBG] from='client.57007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[45495]: audit 2022-04-22T22:58:09.073532+0000 mon.a (mon.0) 15689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[46934]: cluster 2022-04-22T22:58:08.455290+0000 mgr.y (mgr.34107) 11316 : cluster [DBG] pgmap v11901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[46934]: audit 2022-04-22T22:58:08.616486+0000 mgr.y (mgr.34107) 11317 : audit [DBG] from='client.47064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[46934]: audit 2022-04-22T22:58:08.952969+0000 mgr.y (mgr.34107) 11318 : audit [DBG] from='client.57007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:09 smithi115 bash[46934]: audit 2022-04-22T22:58:09.073532+0000 mon.a (mon.0) 15689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:10.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:10 smithi121 bash[36902]: audit 2022-04-22T22:58:09.388378+0000 mon.c (mon.1) 381 : audit [DBG] from='client.? 172.21.15.115:0/2690837648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:10 smithi115 bash[45495]: audit 2022-04-22T22:58:09.388378+0000 mon.c (mon.1) 381 : audit [DBG] from='client.? 172.21.15.115:0/2690837648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:10 smithi115 bash[46934]: audit 2022-04-22T22:58:09.388378+0000 mon.c (mon.1) 381 : audit [DBG] from='client.? 172.21.15.115:0/2690837648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:11 smithi121 bash[36902]: cluster 2022-04-22T22:58:10.456364+0000 mgr.y (mgr.34107) 11319 : cluster [DBG] pgmap v11902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:11.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:11 smithi121 bash[36902]: audit 2022-04-22T22:58:10.911910+0000 mon.a (mon.0) 15690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:11 smithi115 bash[45495]: cluster 2022-04-22T22:58:10.456364+0000 mgr.y (mgr.34107) 11319 : cluster [DBG] pgmap v11902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:11 smithi115 bash[45495]: audit 2022-04-22T22:58:10.911910+0000 mon.a (mon.0) 15690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:11 smithi115 bash[46934]: cluster 2022-04-22T22:58:10.456364+0000 mgr.y (mgr.34107) 11319 : cluster [DBG] pgmap v11902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:11 smithi115 bash[46934]: audit 2022-04-22T22:58:10.911910+0000 mon.a (mon.0) 15690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:13.159 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:12.924Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:13.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[46934]: cluster 2022-04-22T22:58:12.457444+0000 mgr.y (mgr.34107) 11320 : cluster [DBG] pgmap v11903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:13.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[46934]: audit 2022-04-22T22:58:12.534868+0000 mon.a (mon.0) 15691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:13.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[46934]: audit 2022-04-22T22:58:13.001026+0000 mon.a (mon.0) 15692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[46934]: audit 2022-04-22T22:58:13.003775+0000 mon.a (mon.0) 15693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[46934]: audit 2022-04-22T22:58:13.005316+0000 mon.a (mon.0) 15694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[45495]: cluster 2022-04-22T22:58:12.457444+0000 mgr.y (mgr.34107) 11320 : cluster [DBG] pgmap v11903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[45495]: audit 2022-04-22T22:58:12.534868+0000 mon.a (mon.0) 15691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[45495]: audit 2022-04-22T22:58:13.001026+0000 mon.a (mon.0) 15692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[45495]: audit 2022-04-22T22:58:13.003775+0000 mon.a (mon.0) 15693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:58:13.160 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:13 smithi115 bash[45495]: audit 2022-04-22T22:58:13.005316+0000 mon.a (mon.0) 15694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:58:13.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:13 smithi121 bash[36902]: cluster 2022-04-22T22:58:12.457444+0000 mgr.y (mgr.34107) 11320 : cluster [DBG] pgmap v11903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:13.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:13 smithi121 bash[36902]: audit 2022-04-22T22:58:12.534868+0000 mon.a (mon.0) 15691 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:13.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:13 smithi121 bash[36902]: audit 2022-04-22T22:58:13.001026+0000 mon.a (mon.0) 15692 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:58:13.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:13 smithi121 bash[36902]: audit 2022-04-22T22:58:13.003775+0000 mon.a (mon.0) 15693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:58:13.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:13 smithi121 bash[36902]: audit 2022-04-22T22:58:13.005316+0000 mon.a (mon.0) 15694 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:58:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:14 smithi121 bash[36902]: audit 2022-04-22T22:58:13.359551+0000 mon.a (mon.0) 15695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:58:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:14 smithi121 bash[36902]: audit 2022-04-22T22:58:14.076585+0000 mon.a (mon.0) 15696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:14.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:14 smithi115 bash[46934]: audit 2022-04-22T22:58:13.359551+0000 mon.a (mon.0) 15695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:58:14.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:14 smithi115 bash[46934]: audit 2022-04-22T22:58:14.076585+0000 mon.a (mon.0) 15696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:14.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:14 smithi115 bash[45495]: audit 2022-04-22T22:58:13.359551+0000 mon.a (mon.0) 15695 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:58:14.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:14 smithi115 bash[45495]: audit 2022-04-22T22:58:14.076585+0000 mon.a (mon.0) 15696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:14.781Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:15 smithi121 bash[36902]: cluster 2022-04-22T22:58:14.458554+0000 mgr.y (mgr.34107) 11321 : cluster [DBG] pgmap v11904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:15 smithi115 bash[45495]: cluster 2022-04-22T22:58:14.458554+0000 mgr.y (mgr.34107) 11321 : cluster [DBG] pgmap v11904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:15 smithi115 bash[46934]: cluster 2022-04-22T22:58:14.458554+0000 mgr.y (mgr.34107) 11321 : cluster [DBG] pgmap v11904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:16 smithi121 bash[36902]: audit 2022-04-22T22:58:15.914789+0000 mon.a (mon.0) 15697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:16 smithi115 bash[45495]: audit 2022-04-22T22:58:15.914789+0000 mon.a (mon.0) 15697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:16 smithi115 bash[46934]: audit 2022-04-22T22:58:15.914789+0000 mon.a (mon.0) 15697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:17 smithi121 bash[36902]: cluster 2022-04-22T22:58:16.459570+0000 mgr.y (mgr.34107) 11322 : cluster [DBG] pgmap v11905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:17 smithi121 bash[36902]: audit 2022-04-22T22:58:17.089209+0000 mon.a (mon.0) 15698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:58:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:17 smithi121 bash[36902]: audit 2022-04-22T22:58:17.089725+0000 mon.a (mon.0) 15699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:58:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[45495]: cluster 2022-04-22T22:58:16.459570+0000 mgr.y (mgr.34107) 11322 : cluster [DBG] pgmap v11905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[45495]: audit 2022-04-22T22:58:17.089209+0000 mon.a (mon.0) 15698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:58:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[45495]: audit 2022-04-22T22:58:17.089725+0000 mon.a (mon.0) 15699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:58:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[46934]: cluster 2022-04-22T22:58:16.459570+0000 mgr.y (mgr.34107) 11322 : cluster [DBG] pgmap v11905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[46934]: audit 2022-04-22T22:58:17.089209+0000 mon.a (mon.0) 15698 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:58:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:17 smithi115 bash[46934]: audit 2022-04-22T22:58:17.089725+0000 mon.a (mon.0) 15699 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:58:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:18 smithi121 bash[36902]: audit 2022-04-22T22:58:17.537986+0000 mon.a (mon.0) 15700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:18 smithi115 bash[45495]: audit 2022-04-22T22:58:17.537986+0000 mon.a (mon.0) 15700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:18 smithi115 bash[46934]: audit 2022-04-22T22:58:17.537986+0000 mon.a (mon.0) 15700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:19 smithi121 bash[36902]: cluster 2022-04-22T22:58:18.460656+0000 mgr.y (mgr.34107) 11323 : cluster [DBG] pgmap v11906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:19 smithi121 bash[36902]: audit 2022-04-22T22:58:19.080150+0000 mon.a (mon.0) 15701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:19 smithi115 bash[45495]: cluster 2022-04-22T22:58:18.460656+0000 mgr.y (mgr.34107) 11323 : cluster [DBG] pgmap v11906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:19 smithi115 bash[45495]: audit 2022-04-22T22:58:19.080150+0000 mon.a (mon.0) 15701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:19 smithi115 bash[46934]: cluster 2022-04-22T22:58:18.460656+0000 mgr.y (mgr.34107) 11323 : cluster [DBG] pgmap v11906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:19 smithi115 bash[46934]: audit 2022-04-22T22:58:19.080150+0000 mon.a (mon.0) 15701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:21 smithi115 bash[45495]: cluster 2022-04-22T22:58:20.461706+0000 mgr.y (mgr.34107) 11324 : cluster [DBG] pgmap v11907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:21 smithi115 bash[45495]: audit 2022-04-22T22:58:20.917966+0000 mon.a (mon.0) 15702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:21 smithi115 bash[46934]: cluster 2022-04-22T22:58:20.461706+0000 mgr.y (mgr.34107) 11324 : cluster [DBG] pgmap v11907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:21 smithi115 bash[46934]: audit 2022-04-22T22:58:20.917966+0000 mon.a (mon.0) 15702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:21 smithi121 bash[36902]: cluster 2022-04-22T22:58:20.461706+0000 mgr.y (mgr.34107) 11324 : cluster [DBG] pgmap v11907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:21 smithi121 bash[36902]: audit 2022-04-22T22:58:20.917966+0000 mon.a (mon.0) 15702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:22.924Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:23 smithi115 bash[45495]: cluster 2022-04-22T22:58:22.462773+0000 mgr.y (mgr.34107) 11325 : cluster [DBG] pgmap v11908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:23 smithi115 bash[45495]: audit 2022-04-22T22:58:22.540757+0000 mon.a (mon.0) 15703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:23 smithi115 bash[46934]: cluster 2022-04-22T22:58:22.462773+0000 mgr.y (mgr.34107) 11325 : cluster [DBG] pgmap v11908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:23 smithi115 bash[46934]: audit 2022-04-22T22:58:22.540757+0000 mon.a (mon.0) 15703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:23 smithi121 bash[36902]: cluster 2022-04-22T22:58:22.462773+0000 mgr.y (mgr.34107) 11325 : cluster [DBG] pgmap v11908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:23 smithi121 bash[36902]: audit 2022-04-22T22:58:22.540757+0000 mon.a (mon.0) 15703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:24 smithi121 bash[36902]: audit 2022-04-22T22:58:24.083865+0000 mon.a (mon.0) 15704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:24 smithi115 bash[45495]: audit 2022-04-22T22:58:24.083865+0000 mon.a (mon.0) 15704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:24 smithi115 bash[46934]: audit 2022-04-22T22:58:24.083865+0000 mon.a (mon.0) 15704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:24.781Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:25 smithi121 bash[36902]: cluster 2022-04-22T22:58:24.464009+0000 mgr.y (mgr.34107) 11326 : cluster [DBG] pgmap v11909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:25 smithi115 bash[45495]: cluster 2022-04-22T22:58:24.464009+0000 mgr.y (mgr.34107) 11326 : cluster [DBG] pgmap v11909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:25 smithi115 bash[46934]: cluster 2022-04-22T22:58:24.464009+0000 mgr.y (mgr.34107) 11326 : cluster [DBG] pgmap v11909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:26 smithi121 bash[36902]: audit 2022-04-22T22:58:25.921063+0000 mon.a (mon.0) 15705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:26 smithi115 bash[45495]: audit 2022-04-22T22:58:25.921063+0000 mon.a (mon.0) 15705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:26 smithi115 bash[46934]: audit 2022-04-22T22:58:25.921063+0000 mon.a (mon.0) 15705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:27 smithi121 bash[36902]: cluster 2022-04-22T22:58:26.465123+0000 mgr.y (mgr.34107) 11327 : cluster [DBG] pgmap v11910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:27 smithi115 bash[45495]: cluster 2022-04-22T22:58:26.465123+0000 mgr.y (mgr.34107) 11327 : cluster [DBG] pgmap v11910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:27 smithi115 bash[46934]: cluster 2022-04-22T22:58:26.465123+0000 mgr.y (mgr.34107) 11327 : cluster [DBG] pgmap v11910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:28 smithi121 bash[36902]: audit 2022-04-22T22:58:27.543946+0000 mon.a (mon.0) 15706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:28 smithi115 bash[45495]: audit 2022-04-22T22:58:27.543946+0000 mon.a (mon.0) 15706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:28 smithi115 bash[46934]: audit 2022-04-22T22:58:27.543946+0000 mon.a (mon.0) 15706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:29 smithi121 bash[36902]: cluster 2022-04-22T22:58:28.466306+0000 mgr.y (mgr.34107) 11328 : cluster [DBG] pgmap v11911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:29 smithi121 bash[36902]: audit 2022-04-22T22:58:29.087679+0000 mon.a (mon.0) 15707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:29 smithi115 bash[45495]: cluster 2022-04-22T22:58:28.466306+0000 mgr.y (mgr.34107) 11328 : cluster [DBG] pgmap v11911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:29 smithi115 bash[45495]: audit 2022-04-22T22:58:29.087679+0000 mon.a (mon.0) 15707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:29 smithi115 bash[46934]: cluster 2022-04-22T22:58:28.466306+0000 mgr.y (mgr.34107) 11328 : cluster [DBG] pgmap v11911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:29 smithi115 bash[46934]: audit 2022-04-22T22:58:29.087679+0000 mon.a (mon.0) 15707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:31.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:31 smithi115 bash[45495]: cluster 2022-04-22T22:58:30.467363+0000 mgr.y (mgr.34107) 11329 : cluster [DBG] pgmap v11912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:31 smithi115 bash[45495]: audit 2022-04-22T22:58:30.924158+0000 mon.a (mon.0) 15708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:31 smithi115 bash[46934]: cluster 2022-04-22T22:58:30.467363+0000 mgr.y (mgr.34107) 11329 : cluster [DBG] pgmap v11912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:31 smithi115 bash[46934]: audit 2022-04-22T22:58:30.924158+0000 mon.a (mon.0) 15708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:31 smithi121 bash[36902]: cluster 2022-04-22T22:58:30.467363+0000 mgr.y (mgr.34107) 11329 : cluster [DBG] pgmap v11912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:31 smithi121 bash[36902]: audit 2022-04-22T22:58:30.924158+0000 mon.a (mon.0) 15708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:33 smithi115 bash[45495]: cluster 2022-04-22T22:58:32.468556+0000 mgr.y (mgr.34107) 11330 : cluster [DBG] pgmap v11913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:33 smithi115 bash[45495]: audit 2022-04-22T22:58:32.546858+0000 mon.a (mon.0) 15709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:32.926Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:33 smithi115 bash[46934]: cluster 2022-04-22T22:58:32.468556+0000 mgr.y (mgr.34107) 11330 : cluster [DBG] pgmap v11913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:33 smithi115 bash[46934]: audit 2022-04-22T22:58:32.546858+0000 mon.a (mon.0) 15709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:33 smithi121 bash[36902]: cluster 2022-04-22T22:58:32.468556+0000 mgr.y (mgr.34107) 11330 : cluster [DBG] pgmap v11913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:33 smithi121 bash[36902]: audit 2022-04-22T22:58:32.546858+0000 mon.a (mon.0) 15709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:34 smithi121 bash[36902]: audit 2022-04-22T22:58:34.091154+0000 mon.a (mon.0) 15710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:34 smithi115 bash[45495]: audit 2022-04-22T22:58:34.091154+0000 mon.a (mon.0) 15710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:34 smithi115 bash[46934]: audit 2022-04-22T22:58:34.091154+0000 mon.a (mon.0) 15710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:34.782Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:35 smithi121 bash[36902]: cluster 2022-04-22T22:58:34.469671+0000 mgr.y (mgr.34107) 11331 : cluster [DBG] pgmap v11914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:35 smithi115 bash[45495]: cluster 2022-04-22T22:58:34.469671+0000 mgr.y (mgr.34107) 11331 : cluster [DBG] pgmap v11914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:35 smithi115 bash[46934]: cluster 2022-04-22T22:58:34.469671+0000 mgr.y (mgr.34107) 11331 : cluster [DBG] pgmap v11914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:36 smithi121 bash[36902]: audit 2022-04-22T22:58:35.927264+0000 mon.a (mon.0) 15711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:36 smithi115 bash[45495]: audit 2022-04-22T22:58:35.927264+0000 mon.a (mon.0) 15711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:36 smithi115 bash[46934]: audit 2022-04-22T22:58:35.927264+0000 mon.a (mon.0) 15711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:37 smithi121 bash[36902]: cluster 2022-04-22T22:58:36.470643+0000 mgr.y (mgr.34107) 11332 : cluster [DBG] pgmap v11915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:37 smithi115 bash[45495]: cluster 2022-04-22T22:58:36.470643+0000 mgr.y (mgr.34107) 11332 : cluster [DBG] pgmap v11915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:37 smithi115 bash[46934]: cluster 2022-04-22T22:58:36.470643+0000 mgr.y (mgr.34107) 11332 : cluster [DBG] pgmap v11915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:38 smithi121 bash[36902]: audit 2022-04-22T22:58:37.549946+0000 mon.a (mon.0) 15712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:38 smithi115 bash[45495]: audit 2022-04-22T22:58:37.549946+0000 mon.a (mon.0) 15712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:38 smithi115 bash[46934]: audit 2022-04-22T22:58:37.549946+0000 mon.a (mon.0) 15712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:39 smithi121 bash[36902]: cluster 2022-04-22T22:58:38.471866+0000 mgr.y (mgr.34107) 11333 : cluster [DBG] pgmap v11916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:39 smithi121 bash[36902]: audit 2022-04-22T22:58:39.094996+0000 mon.a (mon.0) 15713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:39.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:39 smithi115 bash[46934]: cluster 2022-04-22T22:58:38.471866+0000 mgr.y (mgr.34107) 11333 : cluster [DBG] pgmap v11916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:39 smithi115 bash[46934]: audit 2022-04-22T22:58:39.094996+0000 mon.a (mon.0) 15713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:39 smithi115 bash[45495]: cluster 2022-04-22T22:58:38.471866+0000 mgr.y (mgr.34107) 11333 : cluster [DBG] pgmap v11916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:39 smithi115 bash[45495]: audit 2022-04-22T22:58:39.094996+0000 mon.a (mon.0) 15713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:39.766 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:58:40.113 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 87s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 87s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 87s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 87s ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:58:40.114 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 87s ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 87s ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 87s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 87s ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 87s ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 87s ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:58:40.115 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:58:40.503 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:58:40.503 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:58:40.504 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:58:40.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:58:40.505 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:58:40.505 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:58:40.505 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:58:40.505 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:58:40.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:40 smithi121 bash[36902]: audit 2022-04-22T22:58:39.746953+0000 mgr.y (mgr.34107) 11334 : audit [DBG] from='client.47082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:40 smithi115 bash[45495]: audit 2022-04-22T22:58:39.746953+0000 mgr.y (mgr.34107) 11334 : audit [DBG] from='client.47082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:40 smithi115 bash[46934]: audit 2022-04-22T22:58:39.746953+0000 mgr.y (mgr.34107) 11334 : audit [DBG] from='client.47082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:41 smithi121 bash[36902]: audit 2022-04-22T22:58:40.089319+0000 mgr.y (mgr.34107) 11335 : audit [DBG] from='client.47088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:41.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:41 smithi121 bash[36902]: cluster 2022-04-22T22:58:40.472932+0000 mgr.y (mgr.34107) 11336 : cluster [DBG] pgmap v11917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:41 smithi121 bash[36902]: audit 2022-04-22T22:58:40.499783+0000 mon.c (mon.1) 382 : audit [DBG] from='client.? 172.21.15.115:0/2775199604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:41 smithi121 bash[36902]: audit 2022-04-22T22:58:40.930443+0000 mon.a (mon.0) 15714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:41.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[46934]: audit 2022-04-22T22:58:40.089319+0000 mgr.y (mgr.34107) 11335 : audit [DBG] from='client.47088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[46934]: cluster 2022-04-22T22:58:40.472932+0000 mgr.y (mgr.34107) 11336 : cluster [DBG] pgmap v11917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[46934]: audit 2022-04-22T22:58:40.499783+0000 mon.c (mon.1) 382 : audit [DBG] from='client.? 172.21.15.115:0/2775199604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[46934]: audit 2022-04-22T22:58:40.930443+0000 mon.a (mon.0) 15714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[45495]: audit 2022-04-22T22:58:40.089319+0000 mgr.y (mgr.34107) 11335 : audit [DBG] from='client.47088 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:58:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[45495]: cluster 2022-04-22T22:58:40.472932+0000 mgr.y (mgr.34107) 11336 : cluster [DBG] pgmap v11917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[45495]: audit 2022-04-22T22:58:40.499783+0000 mon.c (mon.1) 382 : audit [DBG] from='client.? 172.21.15.115:0/2775199604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:58:41.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:41 smithi115 bash[45495]: audit 2022-04-22T22:58:40.930443+0000 mon.a (mon.0) 15714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:42.927Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:43 smithi115 bash[45495]: cluster 2022-04-22T22:58:42.474002+0000 mgr.y (mgr.34107) 11337 : cluster [DBG] pgmap v11918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:43 smithi115 bash[45495]: audit 2022-04-22T22:58:42.552922+0000 mon.a (mon.0) 15715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:43 smithi115 bash[46934]: cluster 2022-04-22T22:58:42.474002+0000 mgr.y (mgr.34107) 11337 : cluster [DBG] pgmap v11918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:43 smithi115 bash[46934]: audit 2022-04-22T22:58:42.552922+0000 mon.a (mon.0) 15715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:43.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:43 smithi121 bash[36902]: cluster 2022-04-22T22:58:42.474002+0000 mgr.y (mgr.34107) 11337 : cluster [DBG] pgmap v11918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:43.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:43 smithi121 bash[36902]: audit 2022-04-22T22:58:42.552922+0000 mon.a (mon.0) 15715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:44 smithi121 bash[36902]: audit 2022-04-22T22:58:44.098922+0000 mon.a (mon.0) 15716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:44 smithi115 bash[45495]: audit 2022-04-22T22:58:44.098922+0000 mon.a (mon.0) 15716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:44 smithi115 bash[46934]: audit 2022-04-22T22:58:44.098922+0000 mon.a (mon.0) 15716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:44.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:45 smithi121 bash[36902]: cluster 2022-04-22T22:58:44.475115+0000 mgr.y (mgr.34107) 11338 : cluster [DBG] pgmap v11919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:45 smithi115 bash[45495]: cluster 2022-04-22T22:58:44.475115+0000 mgr.y (mgr.34107) 11338 : cluster [DBG] pgmap v11919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:45 smithi115 bash[46934]: cluster 2022-04-22T22:58:44.475115+0000 mgr.y (mgr.34107) 11338 : cluster [DBG] pgmap v11919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:46 smithi121 bash[36902]: audit 2022-04-22T22:58:45.933526+0000 mon.a (mon.0) 15717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:46 smithi115 bash[45495]: audit 2022-04-22T22:58:45.933526+0000 mon.a (mon.0) 15717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:46 smithi115 bash[46934]: audit 2022-04-22T22:58:45.933526+0000 mon.a (mon.0) 15717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:47 smithi121 bash[36902]: cluster 2022-04-22T22:58:46.476226+0000 mgr.y (mgr.34107) 11339 : cluster [DBG] pgmap v11920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:47 smithi115 bash[45495]: cluster 2022-04-22T22:58:46.476226+0000 mgr.y (mgr.34107) 11339 : cluster [DBG] pgmap v11920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:47 smithi115 bash[46934]: cluster 2022-04-22T22:58:46.476226+0000 mgr.y (mgr.34107) 11339 : cluster [DBG] pgmap v11920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:48 smithi121 bash[36902]: audit 2022-04-22T22:58:47.555881+0000 mon.a (mon.0) 15718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:48 smithi115 bash[45495]: audit 2022-04-22T22:58:47.555881+0000 mon.a (mon.0) 15718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:48 smithi115 bash[46934]: audit 2022-04-22T22:58:47.555881+0000 mon.a (mon.0) 15718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:49 smithi121 bash[36902]: cluster 2022-04-22T22:58:48.477370+0000 mgr.y (mgr.34107) 11340 : cluster [DBG] pgmap v11921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:49 smithi121 bash[36902]: audit 2022-04-22T22:58:49.102492+0000 mon.a (mon.0) 15719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:49 smithi115 bash[45495]: cluster 2022-04-22T22:58:48.477370+0000 mgr.y (mgr.34107) 11340 : cluster [DBG] pgmap v11921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:49 smithi115 bash[45495]: audit 2022-04-22T22:58:49.102492+0000 mon.a (mon.0) 15719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:49 smithi115 bash[46934]: cluster 2022-04-22T22:58:48.477370+0000 mgr.y (mgr.34107) 11340 : cluster [DBG] pgmap v11921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:49 smithi115 bash[46934]: audit 2022-04-22T22:58:49.102492+0000 mon.a (mon.0) 15719 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:51 smithi115 bash[45495]: cluster 2022-04-22T22:58:50.478444+0000 mgr.y (mgr.34107) 11341 : cluster [DBG] pgmap v11922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:51.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:51 smithi115 bash[45495]: audit 2022-04-22T22:58:50.936170+0000 mon.a (mon.0) 15720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:51 smithi115 bash[46934]: cluster 2022-04-22T22:58:50.478444+0000 mgr.y (mgr.34107) 11341 : cluster [DBG] pgmap v11922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:51.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:51 smithi115 bash[46934]: audit 2022-04-22T22:58:50.936170+0000 mon.a (mon.0) 15720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:51 smithi121 bash[36902]: cluster 2022-04-22T22:58:50.478444+0000 mgr.y (mgr.34107) 11341 : cluster [DBG] pgmap v11922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:51 smithi121 bash[36902]: audit 2022-04-22T22:58:50.936170+0000 mon.a (mon.0) 15720 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:53 smithi115 bash[45495]: cluster 2022-04-22T22:58:52.479591+0000 mgr.y (mgr.34107) 11342 : cluster [DBG] pgmap v11923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:53 smithi115 bash[45495]: audit 2022-04-22T22:58:52.558993+0000 mon.a (mon.0) 15721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:52.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:53 smithi115 bash[46934]: cluster 2022-04-22T22:58:52.479591+0000 mgr.y (mgr.34107) 11342 : cluster [DBG] pgmap v11923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:53 smithi115 bash[46934]: audit 2022-04-22T22:58:52.558993+0000 mon.a (mon.0) 15721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:53 smithi121 bash[36902]: cluster 2022-04-22T22:58:52.479591+0000 mgr.y (mgr.34107) 11342 : cluster [DBG] pgmap v11923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:53 smithi121 bash[36902]: audit 2022-04-22T22:58:52.558993+0000 mon.a (mon.0) 15721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:54 smithi121 bash[36902]: audit 2022-04-22T22:58:54.106368+0000 mon.a (mon.0) 15722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:54 smithi115 bash[45495]: audit 2022-04-22T22:58:54.106368+0000 mon.a (mon.0) 15722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:54 smithi115 bash[46934]: audit 2022-04-22T22:58:54.106368+0000 mon.a (mon.0) 15722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:58:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:58:54.783Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:58:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:55 smithi121 bash[36902]: cluster 2022-04-22T22:58:54.480726+0000 mgr.y (mgr.34107) 11343 : cluster [DBG] pgmap v11924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:55.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:55 smithi115 bash[46934]: cluster 2022-04-22T22:58:54.480726+0000 mgr.y (mgr.34107) 11343 : cluster [DBG] pgmap v11924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:55 smithi115 bash[45495]: cluster 2022-04-22T22:58:54.480726+0000 mgr.y (mgr.34107) 11343 : cluster [DBG] pgmap v11924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:56 smithi121 bash[36902]: audit 2022-04-22T22:58:55.939385+0000 mon.a (mon.0) 15723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:56 smithi115 bash[45495]: audit 2022-04-22T22:58:55.939385+0000 mon.a (mon.0) 15723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:56 smithi115 bash[46934]: audit 2022-04-22T22:58:55.939385+0000 mon.a (mon.0) 15723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:58:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:57 smithi121 bash[36902]: cluster 2022-04-22T22:58:56.481825+0000 mgr.y (mgr.34107) 11344 : cluster [DBG] pgmap v11925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:57.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:57 smithi115 bash[46934]: cluster 2022-04-22T22:58:56.481825+0000 mgr.y (mgr.34107) 11344 : cluster [DBG] pgmap v11925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:57 smithi115 bash[45495]: cluster 2022-04-22T22:58:56.481825+0000 mgr.y (mgr.34107) 11344 : cluster [DBG] pgmap v11925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:58 smithi121 bash[36902]: audit 2022-04-22T22:58:57.562187+0000 mon.a (mon.0) 15724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:58 smithi115 bash[45495]: audit 2022-04-22T22:58:57.562187+0000 mon.a (mon.0) 15724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:58 smithi115 bash[46934]: audit 2022-04-22T22:58:57.562187+0000 mon.a (mon.0) 15724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:58:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:59 smithi121 bash[36902]: cluster 2022-04-22T22:58:58.482976+0000 mgr.y (mgr.34107) 11345 : cluster [DBG] pgmap v11926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:58:59 smithi121 bash[36902]: audit 2022-04-22T22:58:59.110205+0000 mon.a (mon.0) 15725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:59 smithi115 bash[45495]: cluster 2022-04-22T22:58:58.482976+0000 mgr.y (mgr.34107) 11345 : cluster [DBG] pgmap v11926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:58:59 smithi115 bash[45495]: audit 2022-04-22T22:58:59.110205+0000 mon.a (mon.0) 15725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:58:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:59 smithi115 bash[46934]: cluster 2022-04-22T22:58:58.482976+0000 mgr.y (mgr.34107) 11345 : cluster [DBG] pgmap v11926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:58:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:58:59 smithi115 bash[46934]: audit 2022-04-22T22:58:59.110205+0000 mon.a (mon.0) 15725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:01 smithi121 bash[36902]: cluster 2022-04-22T22:59:00.484053+0000 mgr.y (mgr.34107) 11346 : cluster [DBG] pgmap v11927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:01 smithi121 bash[36902]: audit 2022-04-22T22:59:00.942614+0000 mon.a (mon.0) 15726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:01 smithi115 bash[45495]: cluster 2022-04-22T22:59:00.484053+0000 mgr.y (mgr.34107) 11346 : cluster [DBG] pgmap v11927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:01 smithi115 bash[45495]: audit 2022-04-22T22:59:00.942614+0000 mon.a (mon.0) 15726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:01 smithi115 bash[46934]: cluster 2022-04-22T22:59:00.484053+0000 mgr.y (mgr.34107) 11346 : cluster [DBG] pgmap v11927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:01 smithi115 bash[46934]: audit 2022-04-22T22:59:00.942614+0000 mon.a (mon.0) 15726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:02 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:02.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:03 smithi115 bash[45495]: cluster 2022-04-22T22:59:02.485157+0000 mgr.y (mgr.34107) 11347 : cluster [DBG] pgmap v11928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:03 smithi115 bash[45495]: audit 2022-04-22T22:59:02.565663+0000 mon.a (mon.0) 15727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:03 smithi115 bash[46934]: cluster 2022-04-22T22:59:02.485157+0000 mgr.y (mgr.34107) 11347 : cluster [DBG] pgmap v11928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:03 smithi115 bash[46934]: audit 2022-04-22T22:59:02.565663+0000 mon.a (mon.0) 15727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:03 smithi121 bash[36902]: cluster 2022-04-22T22:59:02.485157+0000 mgr.y (mgr.34107) 11347 : cluster [DBG] pgmap v11928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:03 smithi121 bash[36902]: audit 2022-04-22T22:59:02.565663+0000 mon.a (mon.0) 15727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:04 smithi121 bash[36902]: audit 2022-04-22T22:59:04.114091+0000 mon.a (mon.0) 15728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:04 smithi115 bash[45495]: audit 2022-04-22T22:59:04.114091+0000 mon.a (mon.0) 15728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:04 smithi115 bash[46934]: audit 2022-04-22T22:59:04.114091+0000 mon.a (mon.0) 15728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:05.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:04 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:04.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:05 smithi121 bash[36902]: cluster 2022-04-22T22:59:04.486223+0000 mgr.y (mgr.34107) 11348 : cluster [DBG] pgmap v11929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:05 smithi115 bash[45495]: cluster 2022-04-22T22:59:04.486223+0000 mgr.y (mgr.34107) 11348 : cluster [DBG] pgmap v11929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:05 smithi115 bash[46934]: cluster 2022-04-22T22:59:04.486223+0000 mgr.y (mgr.34107) 11348 : cluster [DBG] pgmap v11929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:06 smithi121 bash[36902]: audit 2022-04-22T22:59:05.945556+0000 mon.a (mon.0) 15729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:06 smithi115 bash[45495]: audit 2022-04-22T22:59:05.945556+0000 mon.a (mon.0) 15729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:06 smithi115 bash[46934]: audit 2022-04-22T22:59:05.945556+0000 mon.a (mon.0) 15729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:07 smithi121 bash[36902]: cluster 2022-04-22T22:59:06.487299+0000 mgr.y (mgr.34107) 11349 : cluster [DBG] pgmap v11930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:07 smithi115 bash[45495]: cluster 2022-04-22T22:59:06.487299+0000 mgr.y (mgr.34107) 11349 : cluster [DBG] pgmap v11930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:07 smithi115 bash[46934]: cluster 2022-04-22T22:59:06.487299+0000 mgr.y (mgr.34107) 11349 : cluster [DBG] pgmap v11930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:08 smithi121 bash[36902]: audit 2022-04-22T22:59:07.568883+0000 mon.a (mon.0) 15730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:08 smithi115 bash[45495]: audit 2022-04-22T22:59:07.568883+0000 mon.a (mon.0) 15730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:08 smithi115 bash[46934]: audit 2022-04-22T22:59:07.568883+0000 mon.a (mon.0) 15730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:09 smithi121 bash[36902]: cluster 2022-04-22T22:59:08.488495+0000 mgr.y (mgr.34107) 11350 : cluster [DBG] pgmap v11931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:09 smithi121 bash[36902]: audit 2022-04-22T22:59:09.117656+0000 mon.a (mon.0) 15731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:09 smithi115 bash[45495]: cluster 2022-04-22T22:59:08.488495+0000 mgr.y (mgr.34107) 11350 : cluster [DBG] pgmap v11931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:09 smithi115 bash[45495]: audit 2022-04-22T22:59:09.117656+0000 mon.a (mon.0) 15731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:09 smithi115 bash[46934]: cluster 2022-04-22T22:59:08.488495+0000 mgr.y (mgr.34107) 11350 : cluster [DBG] pgmap v11931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:09 smithi115 bash[46934]: audit 2022-04-22T22:59:09.117656+0000 mon.a (mon.0) 15731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:10.879 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:59:11.218 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 118s ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 118s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 118s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 118s ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:59:11.219 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 118s ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 118s ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 118s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 118s ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 118s ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 118s ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:59:11.220 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:59:11.221 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:59:11.619 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:59:11.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:59:11.621 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:59:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:11 smithi121 bash[36902]: cluster 2022-04-22T22:59:10.489564+0000 mgr.y (mgr.34107) 11351 : cluster [DBG] pgmap v11932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:11 smithi121 bash[36902]: audit 2022-04-22T22:59:10.859084+0000 mgr.y (mgr.34107) 11352 : audit [DBG] from='client.57037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:11 smithi121 bash[36902]: audit 2022-04-22T22:59:10.948462+0000 mon.a (mon.0) 15732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[45495]: cluster 2022-04-22T22:59:10.489564+0000 mgr.y (mgr.34107) 11351 : cluster [DBG] pgmap v11932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[45495]: audit 2022-04-22T22:59:10.859084+0000 mgr.y (mgr.34107) 11352 : audit [DBG] from='client.57037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[45495]: audit 2022-04-22T22:59:10.948462+0000 mon.a (mon.0) 15732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[46934]: cluster 2022-04-22T22:59:10.489564+0000 mgr.y (mgr.34107) 11351 : cluster [DBG] pgmap v11932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[46934]: audit 2022-04-22T22:59:10.859084+0000 mgr.y (mgr.34107) 11352 : audit [DBG] from='client.57037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:11 smithi115 bash[46934]: audit 2022-04-22T22:59:10.948462+0000 mon.a (mon.0) 15732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:12.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:12 smithi115 bash[46934]: audit 2022-04-22T22:59:11.195742+0000 mgr.y (mgr.34107) 11353 : audit [DBG] from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:12.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:12 smithi115 bash[46934]: audit 2022-04-22T22:59:11.616019+0000 mon.c (mon.1) 383 : audit [DBG] from='client.? 172.21.15.115:0/2031958864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:12.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:12 smithi115 bash[45495]: audit 2022-04-22T22:59:11.195742+0000 mgr.y (mgr.34107) 11353 : audit [DBG] from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:12.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:12 smithi115 bash[45495]: audit 2022-04-22T22:59:11.616019+0000 mon.c (mon.1) 383 : audit [DBG] from='client.? 172.21.15.115:0/2031958864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:12 smithi121 bash[36902]: audit 2022-04-22T22:59:11.195742+0000 mgr.y (mgr.34107) 11353 : audit [DBG] from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:13.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:12 smithi121 bash[36902]: audit 2022-04-22T22:59:11.616019+0000 mon.c (mon.1) 383 : audit [DBG] from='client.? 172.21.15.115:0/2031958864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:12 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:12.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[46934]: cluster 2022-04-22T22:59:12.490655+0000 mgr.y (mgr.34107) 11354 : cluster [DBG] pgmap v11933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[46934]: audit 2022-04-22T22:59:12.571907+0000 mon.a (mon.0) 15733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[46934]: audit 2022-04-22T22:59:13.367572+0000 mon.a (mon.0) 15734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[46934]: audit 2022-04-22T22:59:13.371397+0000 mon.a (mon.0) 15735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[46934]: audit 2022-04-22T22:59:13.373271+0000 mon.a (mon.0) 15736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[45495]: cluster 2022-04-22T22:59:12.490655+0000 mgr.y (mgr.34107) 11354 : cluster [DBG] pgmap v11933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:13.707 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[45495]: audit 2022-04-22T22:59:12.571907+0000 mon.a (mon.0) 15733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:13.708 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[45495]: audit 2022-04-22T22:59:13.367572+0000 mon.a (mon.0) 15734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:59:13.708 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[45495]: audit 2022-04-22T22:59:13.371397+0000 mon.a (mon.0) 15735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:59:13.708 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:13 smithi115 bash[45495]: audit 2022-04-22T22:59:13.373271+0000 mon.a (mon.0) 15736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:59:13.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:13 smithi121 bash[36902]: cluster 2022-04-22T22:59:12.490655+0000 mgr.y (mgr.34107) 11354 : cluster [DBG] pgmap v11933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:13.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:13 smithi121 bash[36902]: audit 2022-04-22T22:59:12.571907+0000 mon.a (mon.0) 15733 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:13.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:13 smithi121 bash[36902]: audit 2022-04-22T22:59:13.367572+0000 mon.a (mon.0) 15734 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T22:59:13.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:13 smithi121 bash[36902]: audit 2022-04-22T22:59:13.371397+0000 mon.a (mon.0) 15735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T22:59:13.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:13 smithi121 bash[36902]: audit 2022-04-22T22:59:13.373271+0000 mon.a (mon.0) 15736 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T22:59:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:14 smithi121 bash[36902]: audit 2022-04-22T22:59:13.733638+0000 mon.a (mon.0) 15737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:59:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:14 smithi121 bash[36902]: audit 2022-04-22T22:59:14.121523+0000 mon.a (mon.0) 15738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:14 smithi115 bash[45495]: audit 2022-04-22T22:59:13.733638+0000 mon.a (mon.0) 15737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:59:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:14 smithi115 bash[45495]: audit 2022-04-22T22:59:14.121523+0000 mon.a (mon.0) 15738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:14 smithi115 bash[46934]: audit 2022-04-22T22:59:13.733638+0000 mon.a (mon.0) 15737 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T22:59:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:14 smithi115 bash[46934]: audit 2022-04-22T22:59:14.121523+0000 mon.a (mon.0) 15738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:15.042 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:14 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:14.785Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:15 smithi121 bash[36902]: cluster 2022-04-22T22:59:14.491778+0000 mgr.y (mgr.34107) 11355 : cluster [DBG] pgmap v11934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:15 smithi115 bash[45495]: cluster 2022-04-22T22:59:14.491778+0000 mgr.y (mgr.34107) 11355 : cluster [DBG] pgmap v11934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:15 smithi115 bash[46934]: cluster 2022-04-22T22:59:14.491778+0000 mgr.y (mgr.34107) 11355 : cluster [DBG] pgmap v11934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:16 smithi121 bash[36902]: audit 2022-04-22T22:59:15.951154+0000 mon.a (mon.0) 15739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:16 smithi115 bash[45495]: audit 2022-04-22T22:59:15.951154+0000 mon.a (mon.0) 15739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:16 smithi115 bash[46934]: audit 2022-04-22T22:59:15.951154+0000 mon.a (mon.0) 15739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:17 smithi121 bash[36902]: cluster 2022-04-22T22:59:16.492882+0000 mgr.y (mgr.34107) 11356 : cluster [DBG] pgmap v11935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:17 smithi121 bash[36902]: audit 2022-04-22T22:59:17.100579+0000 mon.a (mon.0) 15740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:59:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:17 smithi121 bash[36902]: audit 2022-04-22T22:59:17.101773+0000 mon.a (mon.0) 15741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:59:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:17 smithi121 bash[36902]: audit 2022-04-22T22:59:17.574949+0000 mon.a (mon.0) 15742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[45495]: cluster 2022-04-22T22:59:16.492882+0000 mgr.y (mgr.34107) 11356 : cluster [DBG] pgmap v11935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[45495]: audit 2022-04-22T22:59:17.100579+0000 mon.a (mon.0) 15740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:59:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[45495]: audit 2022-04-22T22:59:17.101773+0000 mon.a (mon.0) 15741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:59:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[45495]: audit 2022-04-22T22:59:17.574949+0000 mon.a (mon.0) 15742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[46934]: cluster 2022-04-22T22:59:16.492882+0000 mgr.y (mgr.34107) 11356 : cluster [DBG] pgmap v11935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[46934]: audit 2022-04-22T22:59:17.100579+0000 mon.a (mon.0) 15740 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T22:59:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[46934]: audit 2022-04-22T22:59:17.101773+0000 mon.a (mon.0) 15741 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T22:59:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:17 smithi115 bash[46934]: audit 2022-04-22T22:59:17.574949+0000 mon.a (mon.0) 15742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:19 smithi121 bash[36902]: cluster 2022-04-22T22:59:18.494043+0000 mgr.y (mgr.34107) 11357 : cluster [DBG] pgmap v11936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:19 smithi121 bash[36902]: audit 2022-04-22T22:59:19.125423+0000 mon.a (mon.0) 15743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:19 smithi115 bash[45495]: cluster 2022-04-22T22:59:18.494043+0000 mgr.y (mgr.34107) 11357 : cluster [DBG] pgmap v11936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:19 smithi115 bash[45495]: audit 2022-04-22T22:59:19.125423+0000 mon.a (mon.0) 15743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:19 smithi115 bash[46934]: cluster 2022-04-22T22:59:18.494043+0000 mgr.y (mgr.34107) 11357 : cluster [DBG] pgmap v11936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:19 smithi115 bash[46934]: audit 2022-04-22T22:59:19.125423+0000 mon.a (mon.0) 15743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:21 smithi115 bash[45495]: cluster 2022-04-22T22:59:20.495116+0000 mgr.y (mgr.34107) 11358 : cluster [DBG] pgmap v11937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:21 smithi115 bash[45495]: audit 2022-04-22T22:59:20.954380+0000 mon.a (mon.0) 15744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:21 smithi115 bash[46934]: cluster 2022-04-22T22:59:20.495116+0000 mgr.y (mgr.34107) 11358 : cluster [DBG] pgmap v11937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:21 smithi115 bash[46934]: audit 2022-04-22T22:59:20.954380+0000 mon.a (mon.0) 15744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:21 smithi121 bash[36902]: cluster 2022-04-22T22:59:20.495116+0000 mgr.y (mgr.34107) 11358 : cluster [DBG] pgmap v11937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:21 smithi121 bash[36902]: audit 2022-04-22T22:59:20.954380+0000 mon.a (mon.0) 15744 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:23 smithi115 bash[45495]: cluster 2022-04-22T22:59:22.496216+0000 mgr.y (mgr.34107) 11359 : cluster [DBG] pgmap v11938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:23 smithi115 bash[45495]: audit 2022-04-22T22:59:22.578138+0000 mon.a (mon.0) 15745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:22 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:22.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:23 smithi115 bash[46934]: cluster 2022-04-22T22:59:22.496216+0000 mgr.y (mgr.34107) 11359 : cluster [DBG] pgmap v11938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:23 smithi115 bash[46934]: audit 2022-04-22T22:59:22.578138+0000 mon.a (mon.0) 15745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:23 smithi121 bash[36902]: cluster 2022-04-22T22:59:22.496216+0000 mgr.y (mgr.34107) 11359 : cluster [DBG] pgmap v11938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:23 smithi121 bash[36902]: audit 2022-04-22T22:59:22.578138+0000 mon.a (mon.0) 15745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:24 smithi121 bash[36902]: audit 2022-04-22T22:59:24.129150+0000 mon.a (mon.0) 15746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:24.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:24 smithi115 bash[45495]: audit 2022-04-22T22:59:24.129150+0000 mon.a (mon.0) 15746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:24 smithi115 bash[46934]: audit 2022-04-22T22:59:24.129150+0000 mon.a (mon.0) 15746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:25.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:24 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:24.786Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:25 smithi121 bash[36902]: cluster 2022-04-22T22:59:24.497310+0000 mgr.y (mgr.34107) 11360 : cluster [DBG] pgmap v11939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:25 smithi115 bash[45495]: cluster 2022-04-22T22:59:24.497310+0000 mgr.y (mgr.34107) 11360 : cluster [DBG] pgmap v11939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:25.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:25 smithi115 bash[46934]: cluster 2022-04-22T22:59:24.497310+0000 mgr.y (mgr.34107) 11360 : cluster [DBG] pgmap v11939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:26 smithi121 bash[36902]: audit 2022-04-22T22:59:25.957305+0000 mon.a (mon.0) 15747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:26 smithi115 bash[45495]: audit 2022-04-22T22:59:25.957305+0000 mon.a (mon.0) 15747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:26 smithi115 bash[46934]: audit 2022-04-22T22:59:25.957305+0000 mon.a (mon.0) 15747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:27 smithi121 bash[36902]: cluster 2022-04-22T22:59:26.498438+0000 mgr.y (mgr.34107) 11361 : cluster [DBG] pgmap v11940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:27 smithi115 bash[45495]: cluster 2022-04-22T22:59:26.498438+0000 mgr.y (mgr.34107) 11361 : cluster [DBG] pgmap v11940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:27 smithi115 bash[46934]: cluster 2022-04-22T22:59:26.498438+0000 mgr.y (mgr.34107) 11361 : cluster [DBG] pgmap v11940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:28 smithi121 bash[36902]: audit 2022-04-22T22:59:27.581176+0000 mon.a (mon.0) 15748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:28 smithi115 bash[45495]: audit 2022-04-22T22:59:27.581176+0000 mon.a (mon.0) 15748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:28 smithi115 bash[46934]: audit 2022-04-22T22:59:27.581176+0000 mon.a (mon.0) 15748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:29 smithi121 bash[36902]: cluster 2022-04-22T22:59:28.499648+0000 mgr.y (mgr.34107) 11362 : cluster [DBG] pgmap v11941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:29 smithi121 bash[36902]: audit 2022-04-22T22:59:29.132904+0000 mon.a (mon.0) 15749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:29 smithi115 bash[45495]: cluster 2022-04-22T22:59:28.499648+0000 mgr.y (mgr.34107) 11362 : cluster [DBG] pgmap v11941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:29 smithi115 bash[45495]: audit 2022-04-22T22:59:29.132904+0000 mon.a (mon.0) 15749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:29 smithi115 bash[46934]: cluster 2022-04-22T22:59:28.499648+0000 mgr.y (mgr.34107) 11362 : cluster [DBG] pgmap v11941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:29 smithi115 bash[46934]: audit 2022-04-22T22:59:29.132904+0000 mon.a (mon.0) 15749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:31 smithi121 bash[36902]: cluster 2022-04-22T22:59:30.500688+0000 mgr.y (mgr.34107) 11363 : cluster [DBG] pgmap v11942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:31 smithi121 bash[36902]: audit 2022-04-22T22:59:30.959771+0000 mon.a (mon.0) 15750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:31 smithi115 bash[45495]: cluster 2022-04-22T22:59:30.500688+0000 mgr.y (mgr.34107) 11363 : cluster [DBG] pgmap v11942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:31 smithi115 bash[45495]: audit 2022-04-22T22:59:30.959771+0000 mon.a (mon.0) 15750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:31 smithi115 bash[46934]: cluster 2022-04-22T22:59:30.500688+0000 mgr.y (mgr.34107) 11363 : cluster [DBG] pgmap v11942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:31 smithi115 bash[46934]: audit 2022-04-22T22:59:30.959771+0000 mon.a (mon.0) 15750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:32 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:32.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:33 smithi115 bash[45495]: cluster 2022-04-22T22:59:32.501754+0000 mgr.y (mgr.34107) 11364 : cluster [DBG] pgmap v11943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:33 smithi115 bash[45495]: audit 2022-04-22T22:59:32.584266+0000 mon.a (mon.0) 15751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:33 smithi115 bash[46934]: cluster 2022-04-22T22:59:32.501754+0000 mgr.y (mgr.34107) 11364 : cluster [DBG] pgmap v11943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:33 smithi115 bash[46934]: audit 2022-04-22T22:59:32.584266+0000 mon.a (mon.0) 15751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:33 smithi121 bash[36902]: cluster 2022-04-22T22:59:32.501754+0000 mgr.y (mgr.34107) 11364 : cluster [DBG] pgmap v11943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:33 smithi121 bash[36902]: audit 2022-04-22T22:59:32.584266+0000 mon.a (mon.0) 15751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:34 smithi121 bash[36902]: audit 2022-04-22T22:59:34.136770+0000 mon.a (mon.0) 15752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:34 smithi115 bash[45495]: audit 2022-04-22T22:59:34.136770+0000 mon.a (mon.0) 15752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:34 smithi115 bash[46934]: audit 2022-04-22T22:59:34.136770+0000 mon.a (mon.0) 15752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:35.196 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:34 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:34.787Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:35.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:35 smithi115 bash[46934]: cluster 2022-04-22T22:59:34.502882+0000 mgr.y (mgr.34107) 11365 : cluster [DBG] pgmap v11944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:35 smithi121 bash[36902]: cluster 2022-04-22T22:59:34.502882+0000 mgr.y (mgr.34107) 11365 : cluster [DBG] pgmap v11944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:35 smithi115 bash[45495]: cluster 2022-04-22T22:59:34.502882+0000 mgr.y (mgr.34107) 11365 : cluster [DBG] pgmap v11944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:36 smithi121 bash[36902]: audit 2022-04-22T22:59:35.963059+0000 mon.a (mon.0) 15753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:36 smithi115 bash[45495]: audit 2022-04-22T22:59:35.963059+0000 mon.a (mon.0) 15753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:36 smithi115 bash[46934]: audit 2022-04-22T22:59:35.963059+0000 mon.a (mon.0) 15753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:37 smithi121 bash[36902]: cluster 2022-04-22T22:59:36.503938+0000 mgr.y (mgr.34107) 11366 : cluster [DBG] pgmap v11945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:37.686 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:37 smithi115 bash[45495]: cluster 2022-04-22T22:59:36.503938+0000 mgr.y (mgr.34107) 11366 : cluster [DBG] pgmap v11945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:37.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:37 smithi115 bash[46934]: cluster 2022-04-22T22:59:36.503938+0000 mgr.y (mgr.34107) 11366 : cluster [DBG] pgmap v11945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:38 smithi121 bash[36902]: audit 2022-04-22T22:59:37.587446+0000 mon.a (mon.0) 15754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:38.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:38 smithi115 bash[45495]: audit 2022-04-22T22:59:37.587446+0000 mon.a (mon.0) 15754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:38 smithi115 bash[46934]: audit 2022-04-22T22:59:37.587446+0000 mon.a (mon.0) 15754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:39 smithi121 bash[36902]: cluster 2022-04-22T22:59:38.505091+0000 mgr.y (mgr.34107) 11367 : cluster [DBG] pgmap v11946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:39 smithi121 bash[36902]: audit 2022-04-22T22:59:39.141033+0000 mon.a (mon.0) 15755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:39 smithi115 bash[45495]: cluster 2022-04-22T22:59:38.505091+0000 mgr.y (mgr.34107) 11367 : cluster [DBG] pgmap v11946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:39 smithi115 bash[45495]: audit 2022-04-22T22:59:39.141033+0000 mon.a (mon.0) 15755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:39 smithi115 bash[46934]: cluster 2022-04-22T22:59:38.505091+0000 mgr.y (mgr.34107) 11367 : cluster [DBG] pgmap v11946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:39 smithi115 bash[46934]: audit 2022-04-22T22:59:39.141033+0000 mon.a (mon.0) 15755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:41.992 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T22:59:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:41 smithi121 bash[36902]: cluster 2022-04-22T22:59:40.506152+0000 mgr.y (mgr.34107) 11368 : cluster [DBG] pgmap v11947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:41 smithi121 bash[36902]: audit 2022-04-22T22:59:40.966107+0000 mon.a (mon.0) 15756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:41 smithi115 bash[45495]: cluster 2022-04-22T22:59:40.506152+0000 mgr.y (mgr.34107) 11368 : cluster [DBG] pgmap v11947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:41 smithi115 bash[45495]: audit 2022-04-22T22:59:40.966107+0000 mon.a (mon.0) 15756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:41 smithi115 bash[46934]: cluster 2022-04-22T22:59:40.506152+0000 mgr.y (mgr.34107) 11368 : cluster [DBG] pgmap v11947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:41 smithi115 bash[46934]: audit 2022-04-22T22:59:40.966107+0000 mon.a (mon.0) 15756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:42.341 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T22:59:42.344 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T22:59:42.344 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T22:59:42.344 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T22:59:42.345 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T22:59:42.345 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T22:59:42.345 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T22:59:42.346 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T22:59:42.347 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T22:59:42.348 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:42.748 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T22:59:42.749 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T22:59:42.934 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:42 smithi115 bash[45495]: audit 2022-04-22T22:59:41.973214+0000 mgr.y (mgr.34107) 11369 : audit [DBG] from='client.57055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:42.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:42 smithi115 bash[46934]: audit 2022-04-22T22:59:41.973214+0000 mgr.y (mgr.34107) 11369 : audit [DBG] from='client.57055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:43.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:42 smithi121 bash[36902]: audit 2022-04-22T22:59:41.973214+0000 mgr.y (mgr.34107) 11369 : audit [DBG] from='client.57055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:42 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:42.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:43 smithi121 bash[36902]: audit 2022-04-22T22:59:42.318275+0000 mgr.y (mgr.34107) 11370 : audit [DBG] from='client.47124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:43 smithi121 bash[36902]: cluster 2022-04-22T22:59:42.507174+0000 mgr.y (mgr.34107) 11371 : cluster [DBG] pgmap v11948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:43 smithi121 bash[36902]: audit 2022-04-22T22:59:42.590578+0000 mon.a (mon.0) 15757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:43 smithi121 bash[36902]: audit 2022-04-22T22:59:42.744075+0000 mon.c (mon.1) 384 : audit [DBG] from='client.? 172.21.15.115:0/2825919384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[45495]: audit 2022-04-22T22:59:42.318275+0000 mgr.y (mgr.34107) 11370 : audit [DBG] from='client.47124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[45495]: cluster 2022-04-22T22:59:42.507174+0000 mgr.y (mgr.34107) 11371 : cluster [DBG] pgmap v11948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[45495]: audit 2022-04-22T22:59:42.590578+0000 mon.a (mon.0) 15757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[45495]: audit 2022-04-22T22:59:42.744075+0000 mon.c (mon.1) 384 : audit [DBG] from='client.? 172.21.15.115:0/2825919384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[46934]: audit 2022-04-22T22:59:42.318275+0000 mgr.y (mgr.34107) 11370 : audit [DBG] from='client.47124 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T22:59:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[46934]: cluster 2022-04-22T22:59:42.507174+0000 mgr.y (mgr.34107) 11371 : cluster [DBG] pgmap v11948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:44.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[46934]: audit 2022-04-22T22:59:42.590578+0000 mon.a (mon.0) 15757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:44.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:43 smithi115 bash[46934]: audit 2022-04-22T22:59:42.744075+0000 mon.c (mon.1) 384 : audit [DBG] from='client.? 172.21.15.115:0/2825919384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T22:59:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:44 smithi121 bash[36902]: audit 2022-04-22T22:59:44.146363+0000 mon.a (mon.0) 15758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:44 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:44.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:44 smithi115 bash[45495]: audit 2022-04-22T22:59:44.146363+0000 mon.a (mon.0) 15758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:44 smithi115 bash[46934]: audit 2022-04-22T22:59:44.146363+0000 mon.a (mon.0) 15758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:45 smithi121 bash[36902]: cluster 2022-04-22T22:59:44.508293+0000 mgr.y (mgr.34107) 11372 : cluster [DBG] pgmap v11949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:45 smithi115 bash[45495]: cluster 2022-04-22T22:59:44.508293+0000 mgr.y (mgr.34107) 11372 : cluster [DBG] pgmap v11949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:45 smithi115 bash[46934]: cluster 2022-04-22T22:59:44.508293+0000 mgr.y (mgr.34107) 11372 : cluster [DBG] pgmap v11949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:46 smithi121 bash[36902]: audit 2022-04-22T22:59:45.969458+0000 mon.a (mon.0) 15759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:46 smithi115 bash[45495]: audit 2022-04-22T22:59:45.969458+0000 mon.a (mon.0) 15759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:46 smithi115 bash[46934]: audit 2022-04-22T22:59:45.969458+0000 mon.a (mon.0) 15759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:47 smithi121 bash[36902]: cluster 2022-04-22T22:59:46.509505+0000 mgr.y (mgr.34107) 11373 : cluster [DBG] pgmap v11950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:47 smithi115 bash[45495]: cluster 2022-04-22T22:59:46.509505+0000 mgr.y (mgr.34107) 11373 : cluster [DBG] pgmap v11950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:47 smithi115 bash[46934]: cluster 2022-04-22T22:59:46.509505+0000 mgr.y (mgr.34107) 11373 : cluster [DBG] pgmap v11950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:48 smithi121 bash[36902]: audit 2022-04-22T22:59:47.595063+0000 mon.a (mon.0) 15760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:48 smithi115 bash[45495]: audit 2022-04-22T22:59:47.595063+0000 mon.a (mon.0) 15760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:48 smithi115 bash[46934]: audit 2022-04-22T22:59:47.595063+0000 mon.a (mon.0) 15760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:49 smithi121 bash[36902]: cluster 2022-04-22T22:59:48.510697+0000 mgr.y (mgr.34107) 11374 : cluster [DBG] pgmap v11951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:49 smithi121 bash[36902]: audit 2022-04-22T22:59:49.149986+0000 mon.a (mon.0) 15761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:49 smithi115 bash[45495]: cluster 2022-04-22T22:59:48.510697+0000 mgr.y (mgr.34107) 11374 : cluster [DBG] pgmap v11951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:49 smithi115 bash[45495]: audit 2022-04-22T22:59:49.149986+0000 mon.a (mon.0) 15761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:49 smithi115 bash[46934]: cluster 2022-04-22T22:59:48.510697+0000 mgr.y (mgr.34107) 11374 : cluster [DBG] pgmap v11951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:50.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:49 smithi115 bash[46934]: audit 2022-04-22T22:59:49.149986+0000 mon.a (mon.0) 15761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:51 smithi121 bash[36902]: cluster 2022-04-22T22:59:50.511803+0000 mgr.y (mgr.34107) 11375 : cluster [DBG] pgmap v11952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:51 smithi121 bash[36902]: audit 2022-04-22T22:59:50.972511+0000 mon.a (mon.0) 15762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:51 smithi115 bash[45495]: cluster 2022-04-22T22:59:50.511803+0000 mgr.y (mgr.34107) 11375 : cluster [DBG] pgmap v11952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:51 smithi115 bash[45495]: audit 2022-04-22T22:59:50.972511+0000 mon.a (mon.0) 15762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:51 smithi115 bash[46934]: cluster 2022-04-22T22:59:50.511803+0000 mgr.y (mgr.34107) 11375 : cluster [DBG] pgmap v11952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:51 smithi115 bash[46934]: audit 2022-04-22T22:59:50.972511+0000 mon.a (mon.0) 15762 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:53.196 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:52 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:52.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:53 smithi121 bash[36902]: cluster 2022-04-22T22:59:52.512850+0000 mgr.y (mgr.34107) 11376 : cluster [DBG] pgmap v11953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:53 smithi121 bash[36902]: audit 2022-04-22T22:59:52.597747+0000 mon.a (mon.0) 15763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:53.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:53 smithi115 bash[45495]: cluster 2022-04-22T22:59:52.512850+0000 mgr.y (mgr.34107) 11376 : cluster [DBG] pgmap v11953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:53.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:53 smithi115 bash[45495]: audit 2022-04-22T22:59:52.597747+0000 mon.a (mon.0) 15763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:53 smithi115 bash[46934]: cluster 2022-04-22T22:59:52.512850+0000 mgr.y (mgr.34107) 11376 : cluster [DBG] pgmap v11953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:53 smithi115 bash[46934]: audit 2022-04-22T22:59:52.597747+0000 mon.a (mon.0) 15763 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:54 smithi121 bash[36902]: audit 2022-04-22T22:59:54.153762+0000 mon.a (mon.0) 15764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:54 smithi115 bash[45495]: audit 2022-04-22T22:59:54.153762+0000 mon.a (mon.0) 15764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:54 smithi115 bash[46934]: audit 2022-04-22T22:59:54.153762+0000 mon.a (mon.0) 15764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:55.206 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 22:59:54 smithi115 bash[60466]: level=error ts=2022-04-22T22:59:54.789Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T22:59:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:55 smithi121 bash[36902]: cluster 2022-04-22T22:59:54.513967+0000 mgr.y (mgr.34107) 11377 : cluster [DBG] pgmap v11954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:55 smithi115 bash[45495]: cluster 2022-04-22T22:59:54.513967+0000 mgr.y (mgr.34107) 11377 : cluster [DBG] pgmap v11954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:55 smithi115 bash[46934]: cluster 2022-04-22T22:59:54.513967+0000 mgr.y (mgr.34107) 11377 : cluster [DBG] pgmap v11954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:56 smithi121 bash[36902]: audit 2022-04-22T22:59:55.975800+0000 mon.a (mon.0) 15765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:56 smithi115 bash[46934]: audit 2022-04-22T22:59:55.975800+0000 mon.a (mon.0) 15765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:56 smithi115 bash[45495]: audit 2022-04-22T22:59:55.975800+0000 mon.a (mon.0) 15765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T22:59:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:57 smithi121 bash[36902]: cluster 2022-04-22T22:59:56.515098+0000 mgr.y (mgr.34107) 11378 : cluster [DBG] pgmap v11955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:57 smithi115 bash[45495]: cluster 2022-04-22T22:59:56.515098+0000 mgr.y (mgr.34107) 11378 : cluster [DBG] pgmap v11955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:57 smithi115 bash[46934]: cluster 2022-04-22T22:59:56.515098+0000 mgr.y (mgr.34107) 11378 : cluster [DBG] pgmap v11955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:58 smithi121 bash[36902]: audit 2022-04-22T22:59:57.600897+0000 mon.a (mon.0) 15766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:58 smithi115 bash[46934]: audit 2022-04-22T22:59:57.600897+0000 mon.a (mon.0) 15766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:58 smithi115 bash[45495]: audit 2022-04-22T22:59:57.600897+0000 mon.a (mon.0) 15766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T22:59:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:59 smithi121 bash[36902]: cluster 2022-04-22T22:59:58.516465+0000 mgr.y (mgr.34107) 11379 : cluster [DBG] pgmap v11956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 22:59:59 smithi121 bash[36902]: audit 2022-04-22T22:59:59.157540+0000 mon.a (mon.0) 15767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:59 smithi115 bash[46934]: cluster 2022-04-22T22:59:58.516465+0000 mgr.y (mgr.34107) 11379 : cluster [DBG] pgmap v11956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 22:59:59 smithi115 bash[46934]: audit 2022-04-22T22:59:59.157540+0000 mon.a (mon.0) 15767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T22:59:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:59 smithi115 bash[45495]: cluster 2022-04-22T22:59:58.516465+0000 mgr.y (mgr.34107) 11379 : cluster [DBG] pgmap v11956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T22:59:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 22:59:59 smithi115 bash[45495]: audit 2022-04-22T22:59:59.157540+0000 mon.a (mon.0) 15767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:00 smithi121 bash[36902]: cluster 2022-04-22T23:00:00.000169+0000 mon.a (mon.0) 15768 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:00:00.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:00 smithi115 bash[46934]: cluster 2022-04-22T23:00:00.000169+0000 mon.a (mon.0) 15768 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:00:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:00 smithi115 bash[45495]: cluster 2022-04-22T23:00:00.000169+0000 mon.a (mon.0) 15768 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:00:01.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[36902]: cluster 2022-04-22T23:00:00.517572+0000 mgr.y (mgr.34107) 11380 : cluster [DBG] pgmap v11957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[36902]: audit 2022-04-22T23:00:00.978635+0000 mon.a (mon.0) 15769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:01 smithi115 bash[45495]: cluster 2022-04-22T23:00:00.517572+0000 mgr.y (mgr.34107) 11380 : cluster [DBG] pgmap v11957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:01 smithi115 bash[45495]: audit 2022-04-22T23:00:00.978635+0000 mon.a (mon.0) 15769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:01 smithi115 bash[46934]: cluster 2022-04-22T23:00:00.517572+0000 mgr.y (mgr.34107) 11380 : cluster [DBG] pgmap v11957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:01 smithi115 bash[46934]: audit 2022-04-22T23:00:00.978635+0000 mon.a (mon.0) 15769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:02.014 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[49697]: ts=2022-04-22T23:00:01.667Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1650657601220 maxt=1650664800000 ulid=01G19R1WYRYJMZG5CTV839X7ZW duration=43.246884ms 2022-04-22T23:00:02.015 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[49697]: ts=2022-04-22T23:00:01.669Z caller=head.go:827 level=info component=tsdb msg="Head GC completed" duration=1.912857ms 2022-04-22T23:00:02.015 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[49697]: ts=2022-04-22T23:00:01.670Z caller=checkpoint.go:98 level=info component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 mint=1650664800000 2022-04-22T23:00:02.015 INFO:journalctl@ceph.prometheus.a.smithi121.stdout:Apr 22 23:00:01 smithi121 bash[49697]: ts=2022-04-22T23:00:01.734Z caller=head.go:996 level=info component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=64.591243ms 2022-04-22T23:00:03.196 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:02.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:03 smithi121 bash[36902]: cluster 2022-04-22T23:00:02.518654+0000 mgr.y (mgr.34107) 11381 : cluster [DBG] pgmap v11958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:03 smithi121 bash[36902]: audit 2022-04-22T23:00:02.603933+0000 mon.a (mon.0) 15770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:03 smithi115 bash[46934]: cluster 2022-04-22T23:00:02.518654+0000 mgr.y (mgr.34107) 11381 : cluster [DBG] pgmap v11958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:03 smithi115 bash[46934]: audit 2022-04-22T23:00:02.603933+0000 mon.a (mon.0) 15770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:03 smithi115 bash[45495]: cluster 2022-04-22T23:00:02.518654+0000 mgr.y (mgr.34107) 11381 : cluster [DBG] pgmap v11958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:03 smithi115 bash[45495]: audit 2022-04-22T23:00:02.603933+0000 mon.a (mon.0) 15770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:04 smithi121 bash[36902]: audit 2022-04-22T23:00:04.161470+0000 mon.a (mon.0) 15771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:04 smithi115 bash[45495]: audit 2022-04-22T23:00:04.161470+0000 mon.a (mon.0) 15771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:04 smithi115 bash[46934]: audit 2022-04-22T23:00:04.161470+0000 mon.a (mon.0) 15771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:05.206 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:04.788Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:05 smithi121 bash[36902]: cluster 2022-04-22T23:00:04.519784+0000 mgr.y (mgr.34107) 11382 : cluster [DBG] pgmap v11959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:05 smithi115 bash[45495]: cluster 2022-04-22T23:00:04.519784+0000 mgr.y (mgr.34107) 11382 : cluster [DBG] pgmap v11959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:05.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:05 smithi115 bash[46934]: cluster 2022-04-22T23:00:04.519784+0000 mgr.y (mgr.34107) 11382 : cluster [DBG] pgmap v11959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:06 smithi121 bash[36902]: audit 2022-04-22T23:00:05.981732+0000 mon.a (mon.0) 15772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:06 smithi115 bash[45495]: audit 2022-04-22T23:00:05.981732+0000 mon.a (mon.0) 15772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:06 smithi115 bash[46934]: audit 2022-04-22T23:00:05.981732+0000 mon.a (mon.0) 15772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:07 smithi121 bash[36902]: cluster 2022-04-22T23:00:06.520842+0000 mgr.y (mgr.34107) 11383 : cluster [DBG] pgmap v11960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:07 smithi115 bash[45495]: cluster 2022-04-22T23:00:06.520842+0000 mgr.y (mgr.34107) 11383 : cluster [DBG] pgmap v11960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:07 smithi115 bash[46934]: cluster 2022-04-22T23:00:06.520842+0000 mgr.y (mgr.34107) 11383 : cluster [DBG] pgmap v11960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:08 smithi121 bash[36902]: audit 2022-04-22T23:00:07.607023+0000 mon.a (mon.0) 15773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:08 smithi115 bash[45495]: audit 2022-04-22T23:00:07.607023+0000 mon.a (mon.0) 15773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:08 smithi115 bash[46934]: audit 2022-04-22T23:00:07.607023+0000 mon.a (mon.0) 15773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:09 smithi121 bash[36902]: cluster 2022-04-22T23:00:08.522048+0000 mgr.y (mgr.34107) 11384 : cluster [DBG] pgmap v11961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:09 smithi121 bash[36902]: audit 2022-04-22T23:00:09.165308+0000 mon.a (mon.0) 15774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:09 smithi115 bash[45495]: cluster 2022-04-22T23:00:08.522048+0000 mgr.y (mgr.34107) 11384 : cluster [DBG] pgmap v11961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:09 smithi115 bash[45495]: audit 2022-04-22T23:00:09.165308+0000 mon.a (mon.0) 15774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:09 smithi115 bash[46934]: cluster 2022-04-22T23:00:08.522048+0000 mgr.y (mgr.34107) 11384 : cluster [DBG] pgmap v11961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:09 smithi115 bash[46934]: audit 2022-04-22T23:00:09.165308+0000 mon.a (mon.0) 15774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:11 smithi121 bash[36902]: cluster 2022-04-22T23:00:10.523233+0000 mgr.y (mgr.34107) 11385 : cluster [DBG] pgmap v11962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:11 smithi121 bash[36902]: audit 2022-04-22T23:00:10.984610+0000 mon.a (mon.0) 15775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:11 smithi115 bash[45495]: cluster 2022-04-22T23:00:10.523233+0000 mgr.y (mgr.34107) 11385 : cluster [DBG] pgmap v11962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:11 smithi115 bash[45495]: audit 2022-04-22T23:00:10.984610+0000 mon.a (mon.0) 15775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:11 smithi115 bash[46934]: cluster 2022-04-22T23:00:10.523233+0000 mgr.y (mgr.34107) 11385 : cluster [DBG] pgmap v11962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:11 smithi115 bash[46934]: audit 2022-04-22T23:00:10.984610+0000 mon.a (mon.0) 15775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:13.135 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:00:13.209 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:12.933Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:13.483 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:00:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:00:13.485 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:00:13.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:00:13.486 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:00:13.486 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:00:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:13 smithi121 bash[36902]: cluster 2022-04-22T23:00:12.524350+0000 mgr.y (mgr.34107) 11386 : cluster [DBG] pgmap v11963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:13 smithi121 bash[36902]: audit 2022-04-22T23:00:12.610179+0000 mon.a (mon.0) 15776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:13.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:13 smithi115 bash[46934]: cluster 2022-04-22T23:00:12.524350+0000 mgr.y (mgr.34107) 11386 : cluster [DBG] pgmap v11963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:13.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:13 smithi115 bash[46934]: audit 2022-04-22T23:00:12.610179+0000 mon.a (mon.0) 15776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:13 smithi115 bash[45495]: cluster 2022-04-22T23:00:12.524350+0000 mgr.y (mgr.34107) 11386 : cluster [DBG] pgmap v11963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:13 smithi115 bash[45495]: audit 2022-04-22T23:00:12.610179+0000 mon.a (mon.0) 15776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:13.898 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:00:13.899 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:00:13.900 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:00:14.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.114911+0000 mgr.y (mgr.34107) 11387 : audit [DBG] from='client.57073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.453845+0000 mgr.y (mgr.34107) 11388 : audit [DBG] from='client.47142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.742939+0000 mon.a (mon.0) 15777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.745566+0000 mon.a (mon.0) 15778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.747195+0000 mon.a (mon.0) 15779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:13.894795+0000 mon.c (mon.1) 385 : audit [DBG] from='client.? 172.21.15.115:0/291220005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:14.097297+0000 mon.a (mon.0) 15780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:00:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:14 smithi121 bash[36902]: audit 2022-04-22T23:00:14.168738+0000 mon.a (mon.0) 15781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.114911+0000 mgr.y (mgr.34107) 11387 : audit [DBG] from='client.57073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.453845+0000 mgr.y (mgr.34107) 11388 : audit [DBG] from='client.47142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.742939+0000 mon.a (mon.0) 15777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.745566+0000 mon.a (mon.0) 15778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.747195+0000 mon.a (mon.0) 15779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:13.894795+0000 mon.c (mon.1) 385 : audit [DBG] from='client.? 172.21.15.115:0/291220005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:14.097297+0000 mon.a (mon.0) 15780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:00:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[46934]: audit 2022-04-22T23:00:14.168738+0000 mon.a (mon.0) 15781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.114911+0000 mgr.y (mgr.34107) 11387 : audit [DBG] from='client.57073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.453845+0000 mgr.y (mgr.34107) 11388 : audit [DBG] from='client.47142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.742939+0000 mon.a (mon.0) 15777 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.745566+0000 mon.a (mon.0) 15778 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.747195+0000 mon.a (mon.0) 15779 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:13.894795+0000 mon.c (mon.1) 385 : audit [DBG] from='client.? 172.21.15.115:0/291220005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:14.097297+0000 mon.a (mon.0) 15780 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:00:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[45495]: audit 2022-04-22T23:00:14.168738+0000 mon.a (mon.0) 15781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:15.220 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:14.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:15.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:15 smithi115 bash[46934]: cluster 2022-04-22T23:00:14.525477+0000 mgr.y (mgr.34107) 11389 : cluster [DBG] pgmap v11964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:15 smithi121 bash[36902]: cluster 2022-04-22T23:00:14.525477+0000 mgr.y (mgr.34107) 11389 : cluster [DBG] pgmap v11964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:15 smithi115 bash[45495]: cluster 2022-04-22T23:00:14.525477+0000 mgr.y (mgr.34107) 11389 : cluster [DBG] pgmap v11964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:16 smithi121 bash[36902]: audit 2022-04-22T23:00:15.987578+0000 mon.a (mon.0) 15782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:16 smithi115 bash[45495]: audit 2022-04-22T23:00:15.987578+0000 mon.a (mon.0) 15782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:16 smithi115 bash[46934]: audit 2022-04-22T23:00:15.987578+0000 mon.a (mon.0) 15782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:17 smithi121 bash[36902]: cluster 2022-04-22T23:00:16.526574+0000 mgr.y (mgr.34107) 11390 : cluster [DBG] pgmap v11965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:17.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:17 smithi121 bash[36902]: audit 2022-04-22T23:00:17.113297+0000 mon.a (mon.0) 15783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:00:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:17 smithi121 bash[36902]: audit 2022-04-22T23:00:17.116568+0000 mon.a (mon.0) 15784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[45495]: cluster 2022-04-22T23:00:16.526574+0000 mgr.y (mgr.34107) 11390 : cluster [DBG] pgmap v11965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[45495]: audit 2022-04-22T23:00:17.113297+0000 mon.a (mon.0) 15783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[45495]: audit 2022-04-22T23:00:17.116568+0000 mon.a (mon.0) 15784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[46934]: cluster 2022-04-22T23:00:16.526574+0000 mgr.y (mgr.34107) 11390 : cluster [DBG] pgmap v11965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[46934]: audit 2022-04-22T23:00:17.113297+0000 mon.a (mon.0) 15783 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:00:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:17 smithi115 bash[46934]: audit 2022-04-22T23:00:17.116568+0000 mon.a (mon.0) 15784 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:00:18.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:18 smithi121 bash[36902]: audit 2022-04-22T23:00:17.613572+0000 mon.a (mon.0) 15785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:18 smithi115 bash[45495]: audit 2022-04-22T23:00:17.613572+0000 mon.a (mon.0) 15785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:18 smithi115 bash[46934]: audit 2022-04-22T23:00:17.613572+0000 mon.a (mon.0) 15785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:19 smithi121 bash[36902]: cluster 2022-04-22T23:00:18.527777+0000 mgr.y (mgr.34107) 11391 : cluster [DBG] pgmap v11966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:19 smithi121 bash[36902]: audit 2022-04-22T23:00:19.172449+0000 mon.a (mon.0) 15786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:19 smithi115 bash[45495]: cluster 2022-04-22T23:00:18.527777+0000 mgr.y (mgr.34107) 11391 : cluster [DBG] pgmap v11966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:19 smithi115 bash[45495]: audit 2022-04-22T23:00:19.172449+0000 mon.a (mon.0) 15786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:19 smithi115 bash[46934]: cluster 2022-04-22T23:00:18.527777+0000 mgr.y (mgr.34107) 11391 : cluster [DBG] pgmap v11966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:19 smithi115 bash[46934]: audit 2022-04-22T23:00:19.172449+0000 mon.a (mon.0) 15786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:21 smithi121 bash[36902]: cluster 2022-04-22T23:00:20.528787+0000 mgr.y (mgr.34107) 11392 : cluster [DBG] pgmap v11967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:21 smithi121 bash[36902]: audit 2022-04-22T23:00:20.990609+0000 mon.a (mon.0) 15787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:21 smithi115 bash[45495]: cluster 2022-04-22T23:00:20.528787+0000 mgr.y (mgr.34107) 11392 : cluster [DBG] pgmap v11967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:21 smithi115 bash[45495]: audit 2022-04-22T23:00:20.990609+0000 mon.a (mon.0) 15787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:21 smithi115 bash[46934]: cluster 2022-04-22T23:00:20.528787+0000 mgr.y (mgr.34107) 11392 : cluster [DBG] pgmap v11967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:21 smithi115 bash[46934]: audit 2022-04-22T23:00:20.990609+0000 mon.a (mon.0) 15787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:23.201 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:22.933Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:23 smithi121 bash[36902]: cluster 2022-04-22T23:00:22.529893+0000 mgr.y (mgr.34107) 11393 : cluster [DBG] pgmap v11968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:23 smithi121 bash[36902]: audit 2022-04-22T23:00:22.616837+0000 mon.a (mon.0) 15788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:23 smithi115 bash[45495]: cluster 2022-04-22T23:00:22.529893+0000 mgr.y (mgr.34107) 11393 : cluster [DBG] pgmap v11968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:23 smithi115 bash[45495]: audit 2022-04-22T23:00:22.616837+0000 mon.a (mon.0) 15788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:23 smithi115 bash[46934]: cluster 2022-04-22T23:00:22.529893+0000 mgr.y (mgr.34107) 11393 : cluster [DBG] pgmap v11968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:23 smithi115 bash[46934]: audit 2022-04-22T23:00:22.616837+0000 mon.a (mon.0) 15788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:24 smithi121 bash[36902]: audit 2022-04-22T23:00:24.175824+0000 mon.a (mon.0) 15789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:24.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:24 smithi115 bash[45495]: audit 2022-04-22T23:00:24.175824+0000 mon.a (mon.0) 15789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:24 smithi115 bash[46934]: audit 2022-04-22T23:00:24.175824+0000 mon.a (mon.0) 15789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:25.222 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:24.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:25 smithi121 bash[36902]: cluster 2022-04-22T23:00:24.530979+0000 mgr.y (mgr.34107) 11394 : cluster [DBG] pgmap v11969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:25 smithi115 bash[45495]: cluster 2022-04-22T23:00:24.530979+0000 mgr.y (mgr.34107) 11394 : cluster [DBG] pgmap v11969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:25 smithi115 bash[46934]: cluster 2022-04-22T23:00:24.530979+0000 mgr.y (mgr.34107) 11394 : cluster [DBG] pgmap v11969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:26 smithi121 bash[36902]: audit 2022-04-22T23:00:25.995057+0000 mon.a (mon.0) 15790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:26 smithi115 bash[45495]: audit 2022-04-22T23:00:25.995057+0000 mon.a (mon.0) 15790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:26 smithi115 bash[46934]: audit 2022-04-22T23:00:25.995057+0000 mon.a (mon.0) 15790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:27 smithi121 bash[36902]: cluster 2022-04-22T23:00:26.532160+0000 mgr.y (mgr.34107) 11395 : cluster [DBG] pgmap v11970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:27 smithi115 bash[45495]: cluster 2022-04-22T23:00:26.532160+0000 mgr.y (mgr.34107) 11395 : cluster [DBG] pgmap v11970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:27 smithi115 bash[46934]: cluster 2022-04-22T23:00:26.532160+0000 mgr.y (mgr.34107) 11395 : cluster [DBG] pgmap v11970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:28 smithi121 bash[36902]: audit 2022-04-22T23:00:27.619841+0000 mon.a (mon.0) 15791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:28 smithi115 bash[45495]: audit 2022-04-22T23:00:27.619841+0000 mon.a (mon.0) 15791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:28 smithi115 bash[46934]: audit 2022-04-22T23:00:27.619841+0000 mon.a (mon.0) 15791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:29 smithi121 bash[36902]: cluster 2022-04-22T23:00:28.533259+0000 mgr.y (mgr.34107) 11396 : cluster [DBG] pgmap v11971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:29 smithi121 bash[36902]: audit 2022-04-22T23:00:29.179828+0000 mon.a (mon.0) 15792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:29 smithi115 bash[45495]: cluster 2022-04-22T23:00:28.533259+0000 mgr.y (mgr.34107) 11396 : cluster [DBG] pgmap v11971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:29 smithi115 bash[45495]: audit 2022-04-22T23:00:29.179828+0000 mon.a (mon.0) 15792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:29 smithi115 bash[46934]: cluster 2022-04-22T23:00:28.533259+0000 mgr.y (mgr.34107) 11396 : cluster [DBG] pgmap v11971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:29 smithi115 bash[46934]: audit 2022-04-22T23:00:29.179828+0000 mon.a (mon.0) 15792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:31 smithi121 bash[36902]: cluster 2022-04-22T23:00:30.534313+0000 mgr.y (mgr.34107) 11397 : cluster [DBG] pgmap v11972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:31 smithi121 bash[36902]: audit 2022-04-22T23:00:30.999706+0000 mon.a (mon.0) 15793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:31 smithi115 bash[45495]: cluster 2022-04-22T23:00:30.534313+0000 mgr.y (mgr.34107) 11397 : cluster [DBG] pgmap v11972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:31 smithi115 bash[45495]: audit 2022-04-22T23:00:30.999706+0000 mon.a (mon.0) 15793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:31 smithi115 bash[46934]: cluster 2022-04-22T23:00:30.534313+0000 mgr.y (mgr.34107) 11397 : cluster [DBG] pgmap v11972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:31 smithi115 bash[46934]: audit 2022-04-22T23:00:30.999706+0000 mon.a (mon.0) 15793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:33.211 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:32.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:33 smithi121 bash[36902]: cluster 2022-04-22T23:00:32.535411+0000 mgr.y (mgr.34107) 11398 : cluster [DBG] pgmap v11973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:33 smithi121 bash[36902]: audit 2022-04-22T23:00:32.622942+0000 mon.a (mon.0) 15794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:33 smithi115 bash[45495]: cluster 2022-04-22T23:00:32.535411+0000 mgr.y (mgr.34107) 11398 : cluster [DBG] pgmap v11973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:33 smithi115 bash[45495]: audit 2022-04-22T23:00:32.622942+0000 mon.a (mon.0) 15794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:33 smithi115 bash[46934]: cluster 2022-04-22T23:00:32.535411+0000 mgr.y (mgr.34107) 11398 : cluster [DBG] pgmap v11973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:33 smithi115 bash[46934]: audit 2022-04-22T23:00:32.622942+0000 mon.a (mon.0) 15794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:34 smithi121 bash[36902]: audit 2022-04-22T23:00:34.183613+0000 mon.a (mon.0) 15795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:34 smithi115 bash[45495]: audit 2022-04-22T23:00:34.183613+0000 mon.a (mon.0) 15795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:34 smithi115 bash[46934]: audit 2022-04-22T23:00:34.183613+0000 mon.a (mon.0) 15795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:35.233 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:34.790Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:35 smithi121 bash[36902]: cluster 2022-04-22T23:00:34.536543+0000 mgr.y (mgr.34107) 11399 : cluster [DBG] pgmap v11974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:35 smithi115 bash[45495]: cluster 2022-04-22T23:00:34.536543+0000 mgr.y (mgr.34107) 11399 : cluster [DBG] pgmap v11974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:35 smithi115 bash[46934]: cluster 2022-04-22T23:00:34.536543+0000 mgr.y (mgr.34107) 11399 : cluster [DBG] pgmap v11974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:36 smithi121 bash[36902]: audit 2022-04-22T23:00:36.002687+0000 mon.a (mon.0) 15796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:36 smithi115 bash[45495]: audit 2022-04-22T23:00:36.002687+0000 mon.a (mon.0) 15796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:36 smithi115 bash[46934]: audit 2022-04-22T23:00:36.002687+0000 mon.a (mon.0) 15796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:37 smithi121 bash[36902]: cluster 2022-04-22T23:00:36.537632+0000 mgr.y (mgr.34107) 11400 : cluster [DBG] pgmap v11975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:37 smithi115 bash[45495]: cluster 2022-04-22T23:00:36.537632+0000 mgr.y (mgr.34107) 11400 : cluster [DBG] pgmap v11975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:37 smithi115 bash[46934]: cluster 2022-04-22T23:00:36.537632+0000 mgr.y (mgr.34107) 11400 : cluster [DBG] pgmap v11975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:38 smithi121 bash[36902]: audit 2022-04-22T23:00:37.626175+0000 mon.a (mon.0) 15797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:38 smithi115 bash[45495]: audit 2022-04-22T23:00:37.626175+0000 mon.a (mon.0) 15797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:38 smithi115 bash[46934]: audit 2022-04-22T23:00:37.626175+0000 mon.a (mon.0) 15797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:39 smithi121 bash[36902]: cluster 2022-04-22T23:00:38.538732+0000 mgr.y (mgr.34107) 11401 : cluster [DBG] pgmap v11976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:39.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:39 smithi121 bash[36902]: audit 2022-04-22T23:00:39.187557+0000 mon.a (mon.0) 15798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:39 smithi115 bash[45495]: cluster 2022-04-22T23:00:38.538732+0000 mgr.y (mgr.34107) 11401 : cluster [DBG] pgmap v11976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:39 smithi115 bash[45495]: audit 2022-04-22T23:00:39.187557+0000 mon.a (mon.0) 15798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:39 smithi115 bash[46934]: cluster 2022-04-22T23:00:38.538732+0000 mgr.y (mgr.34107) 11401 : cluster [DBG] pgmap v11976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:39 smithi115 bash[46934]: audit 2022-04-22T23:00:39.187557+0000 mon.a (mon.0) 15798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:41 smithi121 bash[36902]: cluster 2022-04-22T23:00:40.539870+0000 mgr.y (mgr.34107) 11402 : cluster [DBG] pgmap v11977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:41 smithi121 bash[36902]: audit 2022-04-22T23:00:41.005050+0000 mon.a (mon.0) 15799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:41 smithi115 bash[45495]: cluster 2022-04-22T23:00:40.539870+0000 mgr.y (mgr.34107) 11402 : cluster [DBG] pgmap v11977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:41 smithi115 bash[45495]: audit 2022-04-22T23:00:41.005050+0000 mon.a (mon.0) 15799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:41 smithi115 bash[46934]: cluster 2022-04-22T23:00:40.539870+0000 mgr.y (mgr.34107) 11402 : cluster [DBG] pgmap v11977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:41 smithi115 bash[46934]: audit 2022-04-22T23:00:41.005050+0000 mon.a (mon.0) 15799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:43.212 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:42.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:43 smithi121 bash[36902]: cluster 2022-04-22T23:00:42.540937+0000 mgr.y (mgr.34107) 11403 : cluster [DBG] pgmap v11978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:43 smithi121 bash[36902]: audit 2022-04-22T23:00:42.629228+0000 mon.a (mon.0) 15800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:43 smithi115 bash[45495]: cluster 2022-04-22T23:00:42.540937+0000 mgr.y (mgr.34107) 11403 : cluster [DBG] pgmap v11978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:43 smithi115 bash[45495]: audit 2022-04-22T23:00:42.629228+0000 mon.a (mon.0) 15800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:43 smithi115 bash[46934]: cluster 2022-04-22T23:00:42.540937+0000 mgr.y (mgr.34107) 11403 : cluster [DBG] pgmap v11978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:43 smithi115 bash[46934]: audit 2022-04-22T23:00:42.629228+0000 mon.a (mon.0) 15800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:44.272 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:00:44.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:44 smithi121 bash[36902]: audit 2022-04-22T23:00:44.190922+0000 mon.a (mon.0) 15801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:44.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:44 smithi115 bash[45495]: audit 2022-04-22T23:00:44.190922+0000 mon.a (mon.0) 15801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:44 smithi115 bash[46934]: audit 2022-04-22T23:00:44.190922+0000 mon.a (mon.0) 15801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:44.617 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:00:44.617 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:00:44.617 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:00:44.617 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:00:44.617 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:00:44.618 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:00:44.619 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:00:45.018 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:00:45.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:00:45.018 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:00:45.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:45.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:00:45.019 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:00:45.020 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:00:45.020 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:00:45.225 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:44.791Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:45.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:45 smithi121 bash[36902]: audit 2022-04-22T23:00:44.253158+0000 mgr.y (mgr.34107) 11404 : audit [DBG] from='client.47154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:45 smithi121 bash[36902]: cluster 2022-04-22T23:00:44.541987+0000 mgr.y (mgr.34107) 11405 : cluster [DBG] pgmap v11979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:45 smithi121 bash[36902]: audit 2022-04-22T23:00:44.599413+0000 mgr.y (mgr.34107) 11406 : audit [DBG] from='client.57097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:45 smithi121 bash[36902]: audit 2022-04-22T23:00:45.014660+0000 mon.c (mon.1) 386 : audit [DBG] from='client.? 172.21.15.115:0/1373182203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[45495]: audit 2022-04-22T23:00:44.253158+0000 mgr.y (mgr.34107) 11404 : audit [DBG] from='client.47154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[45495]: cluster 2022-04-22T23:00:44.541987+0000 mgr.y (mgr.34107) 11405 : cluster [DBG] pgmap v11979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[45495]: audit 2022-04-22T23:00:44.599413+0000 mgr.y (mgr.34107) 11406 : audit [DBG] from='client.57097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[45495]: audit 2022-04-22T23:00:45.014660+0000 mon.c (mon.1) 386 : audit [DBG] from='client.? 172.21.15.115:0/1373182203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[46934]: audit 2022-04-22T23:00:44.253158+0000 mgr.y (mgr.34107) 11404 : audit [DBG] from='client.47154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[46934]: cluster 2022-04-22T23:00:44.541987+0000 mgr.y (mgr.34107) 11405 : cluster [DBG] pgmap v11979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[46934]: audit 2022-04-22T23:00:44.599413+0000 mgr.y (mgr.34107) 11406 : audit [DBG] from='client.57097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:00:45.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:45 smithi115 bash[46934]: audit 2022-04-22T23:00:45.014660+0000 mon.c (mon.1) 386 : audit [DBG] from='client.? 172.21.15.115:0/1373182203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:00:46.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:46 smithi121 bash[36902]: audit 2022-04-22T23:00:46.007988+0000 mon.a (mon.0) 15802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:46 smithi115 bash[45495]: audit 2022-04-22T23:00:46.007988+0000 mon.a (mon.0) 15802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:46 smithi115 bash[46934]: audit 2022-04-22T23:00:46.007988+0000 mon.a (mon.0) 15802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:47.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:47 smithi121 bash[36902]: cluster 2022-04-22T23:00:46.543087+0000 mgr.y (mgr.34107) 11407 : cluster [DBG] pgmap v11980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:47 smithi115 bash[45495]: cluster 2022-04-22T23:00:46.543087+0000 mgr.y (mgr.34107) 11407 : cluster [DBG] pgmap v11980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:47 smithi115 bash[46934]: cluster 2022-04-22T23:00:46.543087+0000 mgr.y (mgr.34107) 11407 : cluster [DBG] pgmap v11980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:48.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:48 smithi121 bash[36902]: audit 2022-04-22T23:00:47.632453+0000 mon.a (mon.0) 15803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:48 smithi115 bash[45495]: audit 2022-04-22T23:00:47.632453+0000 mon.a (mon.0) 15803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:48 smithi115 bash[46934]: audit 2022-04-22T23:00:47.632453+0000 mon.a (mon.0) 15803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:49.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:49 smithi121 bash[36902]: cluster 2022-04-22T23:00:48.544214+0000 mgr.y (mgr.34107) 11408 : cluster [DBG] pgmap v11981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:49 smithi121 bash[36902]: audit 2022-04-22T23:00:49.194514+0000 mon.a (mon.0) 15804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:49 smithi115 bash[45495]: cluster 2022-04-22T23:00:48.544214+0000 mgr.y (mgr.34107) 11408 : cluster [DBG] pgmap v11981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:49 smithi115 bash[45495]: audit 2022-04-22T23:00:49.194514+0000 mon.a (mon.0) 15804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:49 smithi115 bash[46934]: cluster 2022-04-22T23:00:48.544214+0000 mgr.y (mgr.34107) 11408 : cluster [DBG] pgmap v11981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:49 smithi115 bash[46934]: audit 2022-04-22T23:00:49.194514+0000 mon.a (mon.0) 15804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:51 smithi121 bash[36902]: cluster 2022-04-22T23:00:50.545258+0000 mgr.y (mgr.34107) 11409 : cluster [DBG] pgmap v11982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:51 smithi121 bash[36902]: audit 2022-04-22T23:00:51.010332+0000 mon.a (mon.0) 15805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:51 smithi115 bash[45495]: cluster 2022-04-22T23:00:50.545258+0000 mgr.y (mgr.34107) 11409 : cluster [DBG] pgmap v11982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:52.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:51 smithi115 bash[45495]: audit 2022-04-22T23:00:51.010332+0000 mon.a (mon.0) 15805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:51 smithi115 bash[46934]: cluster 2022-04-22T23:00:50.545258+0000 mgr.y (mgr.34107) 11409 : cluster [DBG] pgmap v11982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:51 smithi115 bash[46934]: audit 2022-04-22T23:00:51.010332+0000 mon.a (mon.0) 15805 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:53.219 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:52.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:53.220 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:53 smithi115 bash[45495]: cluster 2022-04-22T23:00:52.546340+0000 mgr.y (mgr.34107) 11410 : cluster [DBG] pgmap v11983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:53.221 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:53 smithi115 bash[45495]: audit 2022-04-22T23:00:52.635614+0000 mon.a (mon.0) 15806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:53.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:53 smithi115 bash[46934]: cluster 2022-04-22T23:00:52.546340+0000 mgr.y (mgr.34107) 11410 : cluster [DBG] pgmap v11983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:53.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:53 smithi115 bash[46934]: audit 2022-04-22T23:00:52.635614+0000 mon.a (mon.0) 15806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:53 smithi121 bash[36902]: cluster 2022-04-22T23:00:52.546340+0000 mgr.y (mgr.34107) 11410 : cluster [DBG] pgmap v11983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:53 smithi121 bash[36902]: audit 2022-04-22T23:00:52.635614+0000 mon.a (mon.0) 15806 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:54.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:54 smithi121 bash[36902]: audit 2022-04-22T23:00:54.198304+0000 mon.a (mon.0) 15807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:54.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:54 smithi115 bash[45495]: audit 2022-04-22T23:00:54.198304+0000 mon.a (mon.0) 15807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:54 smithi115 bash[46934]: audit 2022-04-22T23:00:54.198304+0000 mon.a (mon.0) 15807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:55.234 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:00:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:00:54.791Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:00:55.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:55 smithi121 bash[36902]: cluster 2022-04-22T23:00:54.547370+0000 mgr.y (mgr.34107) 11411 : cluster [DBG] pgmap v11984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:55.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:55 smithi115 bash[45495]: cluster 2022-04-22T23:00:54.547370+0000 mgr.y (mgr.34107) 11411 : cluster [DBG] pgmap v11984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:55 smithi115 bash[46934]: cluster 2022-04-22T23:00:54.547370+0000 mgr.y (mgr.34107) 11411 : cluster [DBG] pgmap v11984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:56.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:56 smithi121 bash[36902]: audit 2022-04-22T23:00:56.013075+0000 mon.a (mon.0) 15808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:56 smithi115 bash[45495]: audit 2022-04-22T23:00:56.013075+0000 mon.a (mon.0) 15808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:56 smithi115 bash[46934]: audit 2022-04-22T23:00:56.013075+0000 mon.a (mon.0) 15808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:00:57.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:57 smithi121 bash[36902]: cluster 2022-04-22T23:00:56.548396+0000 mgr.y (mgr.34107) 11412 : cluster [DBG] pgmap v11985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:57 smithi115 bash[45495]: cluster 2022-04-22T23:00:56.548396+0000 mgr.y (mgr.34107) 11412 : cluster [DBG] pgmap v11985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:57 smithi115 bash[46934]: cluster 2022-04-22T23:00:56.548396+0000 mgr.y (mgr.34107) 11412 : cluster [DBG] pgmap v11985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:58.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:58 smithi121 bash[36902]: audit 2022-04-22T23:00:57.638799+0000 mon.a (mon.0) 15809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:58 smithi115 bash[45495]: audit 2022-04-22T23:00:57.638799+0000 mon.a (mon.0) 15809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:58 smithi115 bash[46934]: audit 2022-04-22T23:00:57.638799+0000 mon.a (mon.0) 15809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:00:59.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:59 smithi121 bash[36902]: cluster 2022-04-22T23:00:58.549512+0000 mgr.y (mgr.34107) 11413 : cluster [DBG] pgmap v11986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:00:59 smithi121 bash[36902]: audit 2022-04-22T23:00:59.202333+0000 mon.a (mon.0) 15810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:59 smithi115 bash[45495]: cluster 2022-04-22T23:00:58.549512+0000 mgr.y (mgr.34107) 11413 : cluster [DBG] pgmap v11986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:00:59 smithi115 bash[45495]: audit 2022-04-22T23:00:59.202333+0000 mon.a (mon.0) 15810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:00:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:59 smithi115 bash[46934]: cluster 2022-04-22T23:00:58.549512+0000 mgr.y (mgr.34107) 11413 : cluster [DBG] pgmap v11986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:00:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:00:59 smithi115 bash[46934]: audit 2022-04-22T23:00:59.202333+0000 mon.a (mon.0) 15810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:01 smithi121 bash[36902]: cluster 2022-04-22T23:01:00.550558+0000 mgr.y (mgr.34107) 11414 : cluster [DBG] pgmap v11987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:01 smithi121 bash[36902]: audit 2022-04-22T23:01:01.015629+0000 mon.a (mon.0) 15811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:01 smithi115 bash[45495]: cluster 2022-04-22T23:01:00.550558+0000 mgr.y (mgr.34107) 11414 : cluster [DBG] pgmap v11987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:01 smithi115 bash[45495]: audit 2022-04-22T23:01:01.015629+0000 mon.a (mon.0) 15811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:01 smithi115 bash[46934]: cluster 2022-04-22T23:01:00.550558+0000 mgr.y (mgr.34107) 11414 : cluster [DBG] pgmap v11987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:01 smithi115 bash[46934]: audit 2022-04-22T23:01:01.015629+0000 mon.a (mon.0) 15811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:03.225 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:02.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:03 smithi121 bash[36902]: cluster 2022-04-22T23:01:02.551734+0000 mgr.y (mgr.34107) 11415 : cluster [DBG] pgmap v11988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:03 smithi121 bash[36902]: audit 2022-04-22T23:01:02.642100+0000 mon.a (mon.0) 15812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:03 smithi115 bash[45495]: cluster 2022-04-22T23:01:02.551734+0000 mgr.y (mgr.34107) 11415 : cluster [DBG] pgmap v11988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:03 smithi115 bash[45495]: audit 2022-04-22T23:01:02.642100+0000 mon.a (mon.0) 15812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:03 smithi115 bash[46934]: cluster 2022-04-22T23:01:02.551734+0000 mgr.y (mgr.34107) 11415 : cluster [DBG] pgmap v11988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:03 smithi115 bash[46934]: audit 2022-04-22T23:01:02.642100+0000 mon.a (mon.0) 15812 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:04.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:04 smithi121 bash[36902]: audit 2022-04-22T23:01:04.206003+0000 mon.a (mon.0) 15813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:04 smithi115 bash[45495]: audit 2022-04-22T23:01:04.206003+0000 mon.a (mon.0) 15813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:04 smithi115 bash[46934]: audit 2022-04-22T23:01:04.206003+0000 mon.a (mon.0) 15813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:05.227 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:04.793Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:05.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:05 smithi121 bash[36902]: cluster 2022-04-22T23:01:04.552832+0000 mgr.y (mgr.34107) 11416 : cluster [DBG] pgmap v11989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:05 smithi115 bash[45495]: cluster 2022-04-22T23:01:04.552832+0000 mgr.y (mgr.34107) 11416 : cluster [DBG] pgmap v11989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:05 smithi115 bash[46934]: cluster 2022-04-22T23:01:04.552832+0000 mgr.y (mgr.34107) 11416 : cluster [DBG] pgmap v11989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:06.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:06 smithi121 bash[36902]: audit 2022-04-22T23:01:06.018583+0000 mon.a (mon.0) 15814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:06 smithi115 bash[45495]: audit 2022-04-22T23:01:06.018583+0000 mon.a (mon.0) 15814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:06 smithi115 bash[46934]: audit 2022-04-22T23:01:06.018583+0000 mon.a (mon.0) 15814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:07.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:07 smithi121 bash[36902]: cluster 2022-04-22T23:01:06.553971+0000 mgr.y (mgr.34107) 11417 : cluster [DBG] pgmap v11990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:07 smithi115 bash[45495]: cluster 2022-04-22T23:01:06.553971+0000 mgr.y (mgr.34107) 11417 : cluster [DBG] pgmap v11990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:07 smithi115 bash[46934]: cluster 2022-04-22T23:01:06.553971+0000 mgr.y (mgr.34107) 11417 : cluster [DBG] pgmap v11990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:08.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:08 smithi121 bash[36902]: audit 2022-04-22T23:01:07.645048+0000 mon.a (mon.0) 15815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:08.574 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:08 smithi115 bash[45495]: audit 2022-04-22T23:01:07.645048+0000 mon.a (mon.0) 15815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:08.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:08 smithi115 bash[46934]: audit 2022-04-22T23:01:07.645048+0000 mon.a (mon.0) 15815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:09 smithi121 bash[36902]: cluster 2022-04-22T23:01:08.555164+0000 mgr.y (mgr.34107) 11418 : cluster [DBG] pgmap v11991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:09.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:09 smithi121 bash[36902]: audit 2022-04-22T23:01:09.209659+0000 mon.a (mon.0) 15816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:09 smithi115 bash[45495]: cluster 2022-04-22T23:01:08.555164+0000 mgr.y (mgr.34107) 11418 : cluster [DBG] pgmap v11991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:09 smithi115 bash[45495]: audit 2022-04-22T23:01:09.209659+0000 mon.a (mon.0) 15816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:09 smithi115 bash[46934]: cluster 2022-04-22T23:01:08.555164+0000 mgr.y (mgr.34107) 11418 : cluster [DBG] pgmap v11991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:09 smithi115 bash[46934]: audit 2022-04-22T23:01:09.209659+0000 mon.a (mon.0) 15816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:11 smithi121 bash[36902]: cluster 2022-04-22T23:01:10.556241+0000 mgr.y (mgr.34107) 11419 : cluster [DBG] pgmap v11992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:11 smithi121 bash[36902]: audit 2022-04-22T23:01:11.021726+0000 mon.a (mon.0) 15817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:11 smithi115 bash[45495]: cluster 2022-04-22T23:01:10.556241+0000 mgr.y (mgr.34107) 11419 : cluster [DBG] pgmap v11992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:11 smithi115 bash[45495]: audit 2022-04-22T23:01:11.021726+0000 mon.a (mon.0) 15817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:11 smithi115 bash[46934]: cluster 2022-04-22T23:01:10.556241+0000 mgr.y (mgr.34107) 11419 : cluster [DBG] pgmap v11992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:11 smithi115 bash[46934]: audit 2022-04-22T23:01:11.021726+0000 mon.a (mon.0) 15817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:13.222 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:12.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:13 smithi121 bash[36902]: cluster 2022-04-22T23:01:12.557336+0000 mgr.y (mgr.34107) 11420 : cluster [DBG] pgmap v11993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:13 smithi121 bash[36902]: audit 2022-04-22T23:01:12.648118+0000 mon.a (mon.0) 15818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:13 smithi115 bash[45495]: cluster 2022-04-22T23:01:12.557336+0000 mgr.y (mgr.34107) 11420 : cluster [DBG] pgmap v11993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:13 smithi115 bash[45495]: audit 2022-04-22T23:01:12.648118+0000 mon.a (mon.0) 15818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:13 smithi115 bash[46934]: cluster 2022-04-22T23:01:12.557336+0000 mgr.y (mgr.34107) 11420 : cluster [DBG] pgmap v11993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:13 smithi115 bash[46934]: audit 2022-04-22T23:01:12.648118+0000 mon.a (mon.0) 15818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:14.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[46934]: audit 2022-04-22T23:01:14.106420+0000 mon.a (mon.0) 15819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:01:14.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[46934]: audit 2022-04-22T23:01:14.109503+0000 mon.a (mon.0) 15820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:01:14.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[46934]: audit 2022-04-22T23:01:14.111238+0000 mon.a (mon.0) 15821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:01:14.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[46934]: audit 2022-04-22T23:01:14.213521+0000 mon.a (mon.0) 15822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:14.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[45495]: audit 2022-04-22T23:01:14.106420+0000 mon.a (mon.0) 15819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:01:14.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[45495]: audit 2022-04-22T23:01:14.109503+0000 mon.a (mon.0) 15820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[45495]: audit 2022-04-22T23:01:14.111238+0000 mon.a (mon.0) 15821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[45495]: audit 2022-04-22T23:01:14.213521+0000 mon.a (mon.0) 15822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:14 smithi121 bash[36902]: audit 2022-04-22T23:01:14.106420+0000 mon.a (mon.0) 15819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:14 smithi121 bash[36902]: audit 2022-04-22T23:01:14.109503+0000 mon.a (mon.0) 15820 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:14 smithi121 bash[36902]: audit 2022-04-22T23:01:14.111238+0000 mon.a (mon.0) 15821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:01:14.463 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:14 smithi121 bash[36902]: audit 2022-04-22T23:01:14.213521+0000 mon.a (mon.0) 15822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:14.792Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:15.390 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:01:15.724 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:01:15.724 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:01:15.724 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:01:15.724 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:01:15.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:01:15.726 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:01:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:15 smithi121 bash[36902]: audit 2022-04-22T23:01:14.469887+0000 mon.a (mon.0) 15823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:01:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:15 smithi121 bash[36902]: cluster 2022-04-22T23:01:14.558452+0000 mgr.y (mgr.34107) 11421 : cluster [DBG] pgmap v11994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:15 smithi115 bash[45495]: audit 2022-04-22T23:01:14.469887+0000 mon.a (mon.0) 15823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:01:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:15 smithi115 bash[45495]: cluster 2022-04-22T23:01:14.558452+0000 mgr.y (mgr.34107) 11421 : cluster [DBG] pgmap v11994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:15 smithi115 bash[46934]: audit 2022-04-22T23:01:14.469887+0000 mon.a (mon.0) 15823 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:01:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:15 smithi115 bash[46934]: cluster 2022-04-22T23:01:14.558452+0000 mgr.y (mgr.34107) 11421 : cluster [DBG] pgmap v11994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:16.135 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:01:16.135 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:01:16.135 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:01:16.136 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:01:16.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:16.137 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:01:16.137 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:01:16.137 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:01:16.137 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:01:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:16 smithi121 bash[36902]: audit 2022-04-22T23:01:15.372497+0000 mgr.y (mgr.34107) 11422 : audit [DBG] from='client.57109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:16 smithi121 bash[36902]: audit 2022-04-22T23:01:15.707504+0000 mgr.y (mgr.34107) 11423 : audit [DBG] from='client.47178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:16 smithi121 bash[36902]: audit 2022-04-22T23:01:16.024647+0000 mon.a (mon.0) 15824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:16 smithi121 bash[36902]: audit 2022-04-22T23:01:16.131800+0000 mon.c (mon.1) 387 : audit [DBG] from='client.? 172.21.15.115:0/3188311573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[45495]: audit 2022-04-22T23:01:15.372497+0000 mgr.y (mgr.34107) 11422 : audit [DBG] from='client.57109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[45495]: audit 2022-04-22T23:01:15.707504+0000 mgr.y (mgr.34107) 11423 : audit [DBG] from='client.47178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[45495]: audit 2022-04-22T23:01:16.024647+0000 mon.a (mon.0) 15824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[45495]: audit 2022-04-22T23:01:16.131800+0000 mon.c (mon.1) 387 : audit [DBG] from='client.? 172.21.15.115:0/3188311573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[46934]: audit 2022-04-22T23:01:15.372497+0000 mgr.y (mgr.34107) 11422 : audit [DBG] from='client.57109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[46934]: audit 2022-04-22T23:01:15.707504+0000 mgr.y (mgr.34107) 11423 : audit [DBG] from='client.47178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[46934]: audit 2022-04-22T23:01:16.024647+0000 mon.a (mon.0) 15824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:16 smithi115 bash[46934]: audit 2022-04-22T23:01:16.131800+0000 mon.c (mon.1) 387 : audit [DBG] from='client.? 172.21.15.115:0/3188311573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:17 smithi121 bash[36902]: cluster 2022-04-22T23:01:16.559554+0000 mgr.y (mgr.34107) 11424 : cluster [DBG] pgmap v11995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:17 smithi121 bash[36902]: audit 2022-04-22T23:01:17.125379+0000 mon.a (mon.0) 15825 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:01:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:17 smithi121 bash[36902]: audit 2022-04-22T23:01:17.125910+0000 mon.a (mon.0) 15826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:01:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[45495]: cluster 2022-04-22T23:01:16.559554+0000 mgr.y (mgr.34107) 11424 : cluster [DBG] pgmap v11995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[45495]: audit 2022-04-22T23:01:17.125379+0000 mon.a (mon.0) 15825 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:01:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[45495]: audit 2022-04-22T23:01:17.125910+0000 mon.a (mon.0) 15826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:01:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[46934]: cluster 2022-04-22T23:01:16.559554+0000 mgr.y (mgr.34107) 11424 : cluster [DBG] pgmap v11995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[46934]: audit 2022-04-22T23:01:17.125379+0000 mon.a (mon.0) 15825 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:01:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:17 smithi115 bash[46934]: audit 2022-04-22T23:01:17.125910+0000 mon.a (mon.0) 15826 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:01:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:18 smithi121 bash[36902]: audit 2022-04-22T23:01:17.651530+0000 mon.a (mon.0) 15827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:18 smithi115 bash[45495]: audit 2022-04-22T23:01:17.651530+0000 mon.a (mon.0) 15827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:18 smithi115 bash[46934]: audit 2022-04-22T23:01:17.651530+0000 mon.a (mon.0) 15827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:19 smithi121 bash[36902]: cluster 2022-04-22T23:01:18.560605+0000 mgr.y (mgr.34107) 11425 : cluster [DBG] pgmap v11996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:19 smithi121 bash[36902]: audit 2022-04-22T23:01:19.216806+0000 mon.a (mon.0) 15828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:19 smithi115 bash[45495]: cluster 2022-04-22T23:01:18.560605+0000 mgr.y (mgr.34107) 11425 : cluster [DBG] pgmap v11996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:19 smithi115 bash[45495]: audit 2022-04-22T23:01:19.216806+0000 mon.a (mon.0) 15828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:19 smithi115 bash[46934]: cluster 2022-04-22T23:01:18.560605+0000 mgr.y (mgr.34107) 11425 : cluster [DBG] pgmap v11996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:19.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:19 smithi115 bash[46934]: audit 2022-04-22T23:01:19.216806+0000 mon.a (mon.0) 15828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:21 smithi121 bash[36902]: cluster 2022-04-22T23:01:20.561655+0000 mgr.y (mgr.34107) 11426 : cluster [DBG] pgmap v11997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:21 smithi121 bash[36902]: audit 2022-04-22T23:01:21.027555+0000 mon.a (mon.0) 15829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:21 smithi115 bash[46934]: cluster 2022-04-22T23:01:20.561655+0000 mgr.y (mgr.34107) 11426 : cluster [DBG] pgmap v11997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:21 smithi115 bash[46934]: audit 2022-04-22T23:01:21.027555+0000 mon.a (mon.0) 15829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:21 smithi115 bash[45495]: cluster 2022-04-22T23:01:20.561655+0000 mgr.y (mgr.34107) 11426 : cluster [DBG] pgmap v11997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:21 smithi115 bash[45495]: audit 2022-04-22T23:01:21.027555+0000 mon.a (mon.0) 15829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:23.225 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:22.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:23 smithi121 bash[36902]: cluster 2022-04-22T23:01:22.562713+0000 mgr.y (mgr.34107) 11427 : cluster [DBG] pgmap v11998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:23 smithi121 bash[36902]: audit 2022-04-22T23:01:22.654636+0000 mon.a (mon.0) 15830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:23 smithi115 bash[45495]: cluster 2022-04-22T23:01:22.562713+0000 mgr.y (mgr.34107) 11427 : cluster [DBG] pgmap v11998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:23 smithi115 bash[45495]: audit 2022-04-22T23:01:22.654636+0000 mon.a (mon.0) 15830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:23 smithi115 bash[46934]: cluster 2022-04-22T23:01:22.562713+0000 mgr.y (mgr.34107) 11427 : cluster [DBG] pgmap v11998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:23 smithi115 bash[46934]: audit 2022-04-22T23:01:22.654636+0000 mon.a (mon.0) 15830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:24.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:24 smithi121 bash[36902]: audit 2022-04-22T23:01:24.220440+0000 mon.a (mon.0) 15831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:24 smithi115 bash[45495]: audit 2022-04-22T23:01:24.220440+0000 mon.a (mon.0) 15831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:24 smithi115 bash[46934]: audit 2022-04-22T23:01:24.220440+0000 mon.a (mon.0) 15831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:25.236 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:24.792Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:25.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:25 smithi121 bash[36902]: cluster 2022-04-22T23:01:24.563797+0000 mgr.y (mgr.34107) 11428 : cluster [DBG] pgmap v11999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:25 smithi115 bash[45495]: cluster 2022-04-22T23:01:24.563797+0000 mgr.y (mgr.34107) 11428 : cluster [DBG] pgmap v11999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:25 smithi115 bash[46934]: cluster 2022-04-22T23:01:24.563797+0000 mgr.y (mgr.34107) 11428 : cluster [DBG] pgmap v11999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:26.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:26 smithi121 bash[36902]: audit 2022-04-22T23:01:26.030532+0000 mon.a (mon.0) 15832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:26 smithi115 bash[45495]: audit 2022-04-22T23:01:26.030532+0000 mon.a (mon.0) 15832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:26 smithi115 bash[46934]: audit 2022-04-22T23:01:26.030532+0000 mon.a (mon.0) 15832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:27.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:27 smithi121 bash[36902]: cluster 2022-04-22T23:01:26.564926+0000 mgr.y (mgr.34107) 11429 : cluster [DBG] pgmap v12000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:27 smithi115 bash[45495]: cluster 2022-04-22T23:01:26.564926+0000 mgr.y (mgr.34107) 11429 : cluster [DBG] pgmap v12000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:27 smithi115 bash[46934]: cluster 2022-04-22T23:01:26.564926+0000 mgr.y (mgr.34107) 11429 : cluster [DBG] pgmap v12000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:28.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:28 smithi121 bash[36902]: audit 2022-04-22T23:01:27.657695+0000 mon.a (mon.0) 15833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:28.551 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:28 smithi115 bash[45495]: audit 2022-04-22T23:01:27.657695+0000 mon.a (mon.0) 15833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:28.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:28 smithi115 bash[46934]: audit 2022-04-22T23:01:27.657695+0000 mon.a (mon.0) 15833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:29 smithi121 bash[36902]: cluster 2022-04-22T23:01:28.566185+0000 mgr.y (mgr.34107) 11430 : cluster [DBG] pgmap v12001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:29 smithi121 bash[36902]: audit 2022-04-22T23:01:29.224054+0000 mon.a (mon.0) 15834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:29 smithi115 bash[45495]: cluster 2022-04-22T23:01:28.566185+0000 mgr.y (mgr.34107) 11430 : cluster [DBG] pgmap v12001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:29 smithi115 bash[45495]: audit 2022-04-22T23:01:29.224054+0000 mon.a (mon.0) 15834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:29 smithi115 bash[46934]: cluster 2022-04-22T23:01:28.566185+0000 mgr.y (mgr.34107) 11430 : cluster [DBG] pgmap v12001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:29 smithi115 bash[46934]: audit 2022-04-22T23:01:29.224054+0000 mon.a (mon.0) 15834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:31 smithi121 bash[36902]: cluster 2022-04-22T23:01:30.567411+0000 mgr.y (mgr.34107) 11431 : cluster [DBG] pgmap v12002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:31 smithi121 bash[36902]: audit 2022-04-22T23:01:31.033557+0000 mon.a (mon.0) 15835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:31 smithi115 bash[45495]: cluster 2022-04-22T23:01:30.567411+0000 mgr.y (mgr.34107) 11431 : cluster [DBG] pgmap v12002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:31 smithi115 bash[45495]: audit 2022-04-22T23:01:31.033557+0000 mon.a (mon.0) 15835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:31 smithi115 bash[46934]: cluster 2022-04-22T23:01:30.567411+0000 mgr.y (mgr.34107) 11431 : cluster [DBG] pgmap v12002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:31 smithi115 bash[46934]: audit 2022-04-22T23:01:31.033557+0000 mon.a (mon.0) 15835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:33.233 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:32.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:33 smithi121 bash[36902]: cluster 2022-04-22T23:01:32.568668+0000 mgr.y (mgr.34107) 11432 : cluster [DBG] pgmap v12003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:33 smithi121 bash[36902]: audit 2022-04-22T23:01:32.660930+0000 mon.a (mon.0) 15836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:33 smithi115 bash[45495]: cluster 2022-04-22T23:01:32.568668+0000 mgr.y (mgr.34107) 11432 : cluster [DBG] pgmap v12003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:33 smithi115 bash[45495]: audit 2022-04-22T23:01:32.660930+0000 mon.a (mon.0) 15836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:33 smithi115 bash[46934]: cluster 2022-04-22T23:01:32.568668+0000 mgr.y (mgr.34107) 11432 : cluster [DBG] pgmap v12003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:33 smithi115 bash[46934]: audit 2022-04-22T23:01:32.660930+0000 mon.a (mon.0) 15836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:34.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:34 smithi121 bash[36902]: audit 2022-04-22T23:01:34.227792+0000 mon.a (mon.0) 15837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:34.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:34 smithi115 bash[45495]: audit 2022-04-22T23:01:34.227792+0000 mon.a (mon.0) 15837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:34 smithi115 bash[46934]: audit 2022-04-22T23:01:34.227792+0000 mon.a (mon.0) 15837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:35.245 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:34.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:35.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:35 smithi121 bash[36902]: cluster 2022-04-22T23:01:34.569823+0000 mgr.y (mgr.34107) 11433 : cluster [DBG] pgmap v12004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:35.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:35 smithi115 bash[46934]: cluster 2022-04-22T23:01:34.569823+0000 mgr.y (mgr.34107) 11433 : cluster [DBG] pgmap v12004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:35 smithi115 bash[45495]: cluster 2022-04-22T23:01:34.569823+0000 mgr.y (mgr.34107) 11433 : cluster [DBG] pgmap v12004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:36 smithi121 bash[36902]: audit 2022-04-22T23:01:36.036836+0000 mon.a (mon.0) 15838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:36 smithi115 bash[46934]: audit 2022-04-22T23:01:36.036836+0000 mon.a (mon.0) 15838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:36 smithi115 bash[45495]: audit 2022-04-22T23:01:36.036836+0000 mon.a (mon.0) 15838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:37.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:37 smithi121 bash[36902]: cluster 2022-04-22T23:01:36.570918+0000 mgr.y (mgr.34107) 11434 : cluster [DBG] pgmap v12005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:37 smithi115 bash[45495]: cluster 2022-04-22T23:01:36.570918+0000 mgr.y (mgr.34107) 11434 : cluster [DBG] pgmap v12005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:37 smithi115 bash[46934]: cluster 2022-04-22T23:01:36.570918+0000 mgr.y (mgr.34107) 11434 : cluster [DBG] pgmap v12005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:38.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:38 smithi121 bash[36902]: audit 2022-04-22T23:01:37.663952+0000 mon.a (mon.0) 15839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:38 smithi115 bash[46934]: audit 2022-04-22T23:01:37.663952+0000 mon.a (mon.0) 15839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:38 smithi115 bash[45495]: audit 2022-04-22T23:01:37.663952+0000 mon.a (mon.0) 15839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:39 smithi115 bash[45495]: cluster 2022-04-22T23:01:38.572022+0000 mgr.y (mgr.34107) 11435 : cluster [DBG] pgmap v12006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:39 smithi115 bash[45495]: audit 2022-04-22T23:01:39.231414+0000 mon.a (mon.0) 15840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:39 smithi115 bash[46934]: cluster 2022-04-22T23:01:38.572022+0000 mgr.y (mgr.34107) 11435 : cluster [DBG] pgmap v12006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:39 smithi115 bash[46934]: audit 2022-04-22T23:01:39.231414+0000 mon.a (mon.0) 15840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:39 smithi121 bash[36902]: cluster 2022-04-22T23:01:38.572022+0000 mgr.y (mgr.34107) 11435 : cluster [DBG] pgmap v12006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:39 smithi121 bash[36902]: audit 2022-04-22T23:01:39.231414+0000 mon.a (mon.0) 15840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:41 smithi121 bash[36902]: cluster 2022-04-22T23:01:40.573105+0000 mgr.y (mgr.34107) 11436 : cluster [DBG] pgmap v12007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:41 smithi121 bash[36902]: audit 2022-04-22T23:01:41.041866+0000 mon.a (mon.0) 15841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:41 smithi115 bash[45495]: cluster 2022-04-22T23:01:40.573105+0000 mgr.y (mgr.34107) 11436 : cluster [DBG] pgmap v12007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:41 smithi115 bash[45495]: audit 2022-04-22T23:01:41.041866+0000 mon.a (mon.0) 15841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:41 smithi115 bash[46934]: cluster 2022-04-22T23:01:40.573105+0000 mgr.y (mgr.34107) 11436 : cluster [DBG] pgmap v12007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:41 smithi115 bash[46934]: audit 2022-04-22T23:01:41.041866+0000 mon.a (mon.0) 15841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:43.236 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:42.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:43.236 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:43 smithi115 bash[45495]: cluster 2022-04-22T23:01:42.574216+0000 mgr.y (mgr.34107) 11437 : cluster [DBG] pgmap v12008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:43.237 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:43 smithi115 bash[45495]: audit 2022-04-22T23:01:42.667101+0000 mon.a (mon.0) 15842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:43 smithi121 bash[36902]: cluster 2022-04-22T23:01:42.574216+0000 mgr.y (mgr.34107) 11437 : cluster [DBG] pgmap v12008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:43.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:43 smithi121 bash[36902]: audit 2022-04-22T23:01:42.667101+0000 mon.a (mon.0) 15842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:43.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:43 smithi115 bash[46934]: cluster 2022-04-22T23:01:42.574216+0000 mgr.y (mgr.34107) 11437 : cluster [DBG] pgmap v12008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:43 smithi115 bash[46934]: audit 2022-04-22T23:01:42.667101+0000 mon.a (mon.0) 15842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:44 smithi121 bash[36902]: audit 2022-04-22T23:01:44.235218+0000 mon.a (mon.0) 15843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:44 smithi115 bash[46934]: audit 2022-04-22T23:01:44.235218+0000 mon.a (mon.0) 15843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:44.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:44 smithi115 bash[45495]: audit 2022-04-22T23:01:44.235218+0000 mon.a (mon.0) 15843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:44.794Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:45 smithi121 bash[36902]: cluster 2022-04-22T23:01:44.575374+0000 mgr.y (mgr.34107) 11438 : cluster [DBG] pgmap v12009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:45.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:45 smithi115 bash[46934]: cluster 2022-04-22T23:01:44.575374+0000 mgr.y (mgr.34107) 11438 : cluster [DBG] pgmap v12009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:45 smithi115 bash[45495]: cluster 2022-04-22T23:01:44.575374+0000 mgr.y (mgr.34107) 11438 : cluster [DBG] pgmap v12009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:46.506 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:01:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:46 smithi121 bash[36902]: audit 2022-04-22T23:01:46.044759+0000 mon.a (mon.0) 15844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:46 smithi115 bash[45495]: audit 2022-04-22T23:01:46.044759+0000 mon.a (mon.0) 15844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:46 smithi115 bash[46934]: audit 2022-04-22T23:01:46.044759+0000 mon.a (mon.0) 15844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:01:46.856 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:01:46.857 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:01:46.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:01:46.858 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:01:46.858 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:47.263 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:01:47.264 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:01:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:47 smithi121 bash[36902]: audit 2022-04-22T23:01:46.487996+0000 mgr.y (mgr.34107) 11439 : audit [DBG] from='client.57127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:47 smithi121 bash[36902]: cluster 2022-04-22T23:01:46.576410+0000 mgr.y (mgr.34107) 11440 : cluster [DBG] pgmap v12010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:47 smithi121 bash[36902]: audit 2022-04-22T23:01:46.830959+0000 mgr.y (mgr.34107) 11441 : audit [DBG] from='client.57133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:47 smithi121 bash[36902]: audit 2022-04-22T23:01:47.259769+0000 mon.c (mon.1) 388 : audit [DBG] from='client.? 172.21.15.115:0/3475684393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[45495]: audit 2022-04-22T23:01:46.487996+0000 mgr.y (mgr.34107) 11439 : audit [DBG] from='client.57127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[45495]: cluster 2022-04-22T23:01:46.576410+0000 mgr.y (mgr.34107) 11440 : cluster [DBG] pgmap v12010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[45495]: audit 2022-04-22T23:01:46.830959+0000 mgr.y (mgr.34107) 11441 : audit [DBG] from='client.57133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[45495]: audit 2022-04-22T23:01:47.259769+0000 mon.c (mon.1) 388 : audit [DBG] from='client.? 172.21.15.115:0/3475684393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[46934]: audit 2022-04-22T23:01:46.487996+0000 mgr.y (mgr.34107) 11439 : audit [DBG] from='client.57127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[46934]: cluster 2022-04-22T23:01:46.576410+0000 mgr.y (mgr.34107) 11440 : cluster [DBG] pgmap v12010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[46934]: audit 2022-04-22T23:01:46.830959+0000 mgr.y (mgr.34107) 11441 : audit [DBG] from='client.57133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:01:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:47 smithi115 bash[46934]: audit 2022-04-22T23:01:47.259769+0000 mon.c (mon.1) 388 : audit [DBG] from='client.? 172.21.15.115:0/3475684393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:01:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:48 smithi121 bash[36902]: audit 2022-04-22T23:01:47.670191+0000 mon.a (mon.0) 15845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:48 smithi115 bash[45495]: audit 2022-04-22T23:01:47.670191+0000 mon.a (mon.0) 15845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:48 smithi115 bash[46934]: audit 2022-04-22T23:01:47.670191+0000 mon.a (mon.0) 15845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:49 smithi121 bash[36902]: cluster 2022-04-22T23:01:48.577574+0000 mgr.y (mgr.34107) 11442 : cluster [DBG] pgmap v12011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:49 smithi121 bash[36902]: audit 2022-04-22T23:01:49.238851+0000 mon.a (mon.0) 15846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:49.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:49 smithi115 bash[45495]: cluster 2022-04-22T23:01:48.577574+0000 mgr.y (mgr.34107) 11442 : cluster [DBG] pgmap v12011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:49 smithi115 bash[45495]: audit 2022-04-22T23:01:49.238851+0000 mon.a (mon.0) 15846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:49 smithi115 bash[46934]: cluster 2022-04-22T23:01:48.577574+0000 mgr.y (mgr.34107) 11442 : cluster [DBG] pgmap v12011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:49 smithi115 bash[46934]: audit 2022-04-22T23:01:49.238851+0000 mon.a (mon.0) 15846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:51 smithi121 bash[36902]: cluster 2022-04-22T23:01:50.578656+0000 mgr.y (mgr.34107) 11443 : cluster [DBG] pgmap v12012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:51 smithi121 bash[36902]: audit 2022-04-22T23:01:51.047780+0000 mon.a (mon.0) 15847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:51 smithi115 bash[46934]: cluster 2022-04-22T23:01:50.578656+0000 mgr.y (mgr.34107) 11443 : cluster [DBG] pgmap v12012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:51 smithi115 bash[46934]: audit 2022-04-22T23:01:51.047780+0000 mon.a (mon.0) 15847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:51 smithi115 bash[45495]: cluster 2022-04-22T23:01:50.578656+0000 mgr.y (mgr.34107) 11443 : cluster [DBG] pgmap v12012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:51 smithi115 bash[45495]: audit 2022-04-22T23:01:51.047780+0000 mon.a (mon.0) 15847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:53.235 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:52.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:53 smithi115 bash[46934]: cluster 2022-04-22T23:01:52.579766+0000 mgr.y (mgr.34107) 11444 : cluster [DBG] pgmap v12013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:53 smithi115 bash[46934]: audit 2022-04-22T23:01:52.673451+0000 mon.a (mon.0) 15848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:53.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:53 smithi115 bash[45495]: cluster 2022-04-22T23:01:52.579766+0000 mgr.y (mgr.34107) 11444 : cluster [DBG] pgmap v12013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:53.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:53 smithi115 bash[45495]: audit 2022-04-22T23:01:52.673451+0000 mon.a (mon.0) 15848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:53 smithi121 bash[36902]: cluster 2022-04-22T23:01:52.579766+0000 mgr.y (mgr.34107) 11444 : cluster [DBG] pgmap v12013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:53 smithi121 bash[36902]: audit 2022-04-22T23:01:52.673451+0000 mon.a (mon.0) 15848 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:54 smithi121 bash[36902]: audit 2022-04-22T23:01:54.242734+0000 mon.a (mon.0) 15849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:54.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:54 smithi115 bash[46934]: audit 2022-04-22T23:01:54.242734+0000 mon.a (mon.0) 15849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:54 smithi115 bash[45495]: audit 2022-04-22T23:01:54.242734+0000 mon.a (mon.0) 15849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:01:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:01:54.795Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:01:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:55 smithi121 bash[36902]: cluster 2022-04-22T23:01:54.580719+0000 mgr.y (mgr.34107) 11445 : cluster [DBG] pgmap v12014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:55 smithi115 bash[45495]: cluster 2022-04-22T23:01:54.580719+0000 mgr.y (mgr.34107) 11445 : cluster [DBG] pgmap v12014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:55 smithi115 bash[46934]: cluster 2022-04-22T23:01:54.580719+0000 mgr.y (mgr.34107) 11445 : cluster [DBG] pgmap v12014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:56 smithi121 bash[36902]: audit 2022-04-22T23:01:56.050861+0000 mon.a (mon.0) 15850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:56 smithi115 bash[45495]: audit 2022-04-22T23:01:56.050861+0000 mon.a (mon.0) 15850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:56 smithi115 bash[46934]: audit 2022-04-22T23:01:56.050861+0000 mon.a (mon.0) 15850 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:01:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:57 smithi121 bash[36902]: cluster 2022-04-22T23:01:56.581783+0000 mgr.y (mgr.34107) 11446 : cluster [DBG] pgmap v12015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:57 smithi115 bash[45495]: cluster 2022-04-22T23:01:56.581783+0000 mgr.y (mgr.34107) 11446 : cluster [DBG] pgmap v12015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:57.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:57 smithi115 bash[46934]: cluster 2022-04-22T23:01:56.581783+0000 mgr.y (mgr.34107) 11446 : cluster [DBG] pgmap v12015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:58 smithi121 bash[36902]: audit 2022-04-22T23:01:57.676633+0000 mon.a (mon.0) 15851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:58 smithi115 bash[45495]: audit 2022-04-22T23:01:57.676633+0000 mon.a (mon.0) 15851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:58 smithi115 bash[46934]: audit 2022-04-22T23:01:57.676633+0000 mon.a (mon.0) 15851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:01:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:59 smithi121 bash[36902]: cluster 2022-04-22T23:01:58.582920+0000 mgr.y (mgr.34107) 11447 : cluster [DBG] pgmap v12016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:01:59 smithi121 bash[36902]: audit 2022-04-22T23:01:59.246775+0000 mon.a (mon.0) 15852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:59 smithi115 bash[45495]: cluster 2022-04-22T23:01:58.582920+0000 mgr.y (mgr.34107) 11447 : cluster [DBG] pgmap v12016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:01:59 smithi115 bash[45495]: audit 2022-04-22T23:01:59.246775+0000 mon.a (mon.0) 15852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:01:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:59 smithi115 bash[46934]: cluster 2022-04-22T23:01:58.582920+0000 mgr.y (mgr.34107) 11447 : cluster [DBG] pgmap v12016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:01:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:01:59 smithi115 bash[46934]: audit 2022-04-22T23:01:59.246775+0000 mon.a (mon.0) 15852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:01 smithi121 bash[36902]: cluster 2022-04-22T23:02:00.584023+0000 mgr.y (mgr.34107) 11448 : cluster [DBG] pgmap v12017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:01 smithi121 bash[36902]: audit 2022-04-22T23:02:01.054049+0000 mon.a (mon.0) 15853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:01 smithi115 bash[46934]: cluster 2022-04-22T23:02:00.584023+0000 mgr.y (mgr.34107) 11448 : cluster [DBG] pgmap v12017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:01 smithi115 bash[46934]: audit 2022-04-22T23:02:01.054049+0000 mon.a (mon.0) 15853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:01 smithi115 bash[45495]: cluster 2022-04-22T23:02:00.584023+0000 mgr.y (mgr.34107) 11448 : cluster [DBG] pgmap v12017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:01 smithi115 bash[45495]: audit 2022-04-22T23:02:01.054049+0000 mon.a (mon.0) 15853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:03.238 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:02.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:03 smithi121 bash[36902]: cluster 2022-04-22T23:02:02.585058+0000 mgr.y (mgr.34107) 11449 : cluster [DBG] pgmap v12018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:03 smithi121 bash[36902]: audit 2022-04-22T23:02:02.679719+0000 mon.a (mon.0) 15854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:03 smithi115 bash[45495]: cluster 2022-04-22T23:02:02.585058+0000 mgr.y (mgr.34107) 11449 : cluster [DBG] pgmap v12018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:03 smithi115 bash[45495]: audit 2022-04-22T23:02:02.679719+0000 mon.a (mon.0) 15854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:03 smithi115 bash[46934]: cluster 2022-04-22T23:02:02.585058+0000 mgr.y (mgr.34107) 11449 : cluster [DBG] pgmap v12018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:03 smithi115 bash[46934]: audit 2022-04-22T23:02:02.679719+0000 mon.a (mon.0) 15854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:04 smithi121 bash[36902]: audit 2022-04-22T23:02:04.250454+0000 mon.a (mon.0) 15855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:04 smithi115 bash[45495]: audit 2022-04-22T23:02:04.250454+0000 mon.a (mon.0) 15855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:04 smithi115 bash[46934]: audit 2022-04-22T23:02:04.250454+0000 mon.a (mon.0) 15855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:04.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:05 smithi121 bash[36902]: cluster 2022-04-22T23:02:04.586195+0000 mgr.y (mgr.34107) 11450 : cluster [DBG] pgmap v12019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:05 smithi115 bash[45495]: cluster 2022-04-22T23:02:04.586195+0000 mgr.y (mgr.34107) 11450 : cluster [DBG] pgmap v12019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:05 smithi115 bash[46934]: cluster 2022-04-22T23:02:04.586195+0000 mgr.y (mgr.34107) 11450 : cluster [DBG] pgmap v12019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:06 smithi121 bash[36902]: audit 2022-04-22T23:02:06.057332+0000 mon.a (mon.0) 15856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:06 smithi115 bash[45495]: audit 2022-04-22T23:02:06.057332+0000 mon.a (mon.0) 15856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:06 smithi115 bash[46934]: audit 2022-04-22T23:02:06.057332+0000 mon.a (mon.0) 15856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:07 smithi121 bash[36902]: cluster 2022-04-22T23:02:06.587275+0000 mgr.y (mgr.34107) 11451 : cluster [DBG] pgmap v12020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:07 smithi115 bash[45495]: cluster 2022-04-22T23:02:06.587275+0000 mgr.y (mgr.34107) 11451 : cluster [DBG] pgmap v12020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:07 smithi115 bash[46934]: cluster 2022-04-22T23:02:06.587275+0000 mgr.y (mgr.34107) 11451 : cluster [DBG] pgmap v12020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:08 smithi121 bash[36902]: audit 2022-04-22T23:02:07.683041+0000 mon.a (mon.0) 15857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:08 smithi115 bash[45495]: audit 2022-04-22T23:02:07.683041+0000 mon.a (mon.0) 15857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:08 smithi115 bash[46934]: audit 2022-04-22T23:02:07.683041+0000 mon.a (mon.0) 15857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:09 smithi121 bash[36902]: cluster 2022-04-22T23:02:08.588456+0000 mgr.y (mgr.34107) 11452 : cluster [DBG] pgmap v12021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:09 smithi121 bash[36902]: audit 2022-04-22T23:02:09.254266+0000 mon.a (mon.0) 15858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:09 smithi115 bash[46934]: cluster 2022-04-22T23:02:08.588456+0000 mgr.y (mgr.34107) 11452 : cluster [DBG] pgmap v12021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:09 smithi115 bash[46934]: audit 2022-04-22T23:02:09.254266+0000 mon.a (mon.0) 15858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:09 smithi115 bash[45495]: cluster 2022-04-22T23:02:08.588456+0000 mgr.y (mgr.34107) 11452 : cluster [DBG] pgmap v12021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:09 smithi115 bash[45495]: audit 2022-04-22T23:02:09.254266+0000 mon.a (mon.0) 15858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:11 smithi121 bash[36902]: cluster 2022-04-22T23:02:10.589550+0000 mgr.y (mgr.34107) 11453 : cluster [DBG] pgmap v12022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:12.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:11 smithi121 bash[36902]: audit 2022-04-22T23:02:11.060155+0000 mon.a (mon.0) 15859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:11 smithi115 bash[45495]: cluster 2022-04-22T23:02:10.589550+0000 mgr.y (mgr.34107) 11453 : cluster [DBG] pgmap v12022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:11 smithi115 bash[45495]: audit 2022-04-22T23:02:11.060155+0000 mon.a (mon.0) 15859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:11 smithi115 bash[46934]: cluster 2022-04-22T23:02:10.589550+0000 mgr.y (mgr.34107) 11453 : cluster [DBG] pgmap v12022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:12.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:11 smithi115 bash[46934]: audit 2022-04-22T23:02:11.060155+0000 mon.a (mon.0) 15859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:13.245 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:12.940Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:13 smithi121 bash[36902]: cluster 2022-04-22T23:02:12.590708+0000 mgr.y (mgr.34107) 11454 : cluster [DBG] pgmap v12023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:13 smithi121 bash[36902]: audit 2022-04-22T23:02:12.686285+0000 mon.a (mon.0) 15860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:13.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:13 smithi115 bash[45495]: cluster 2022-04-22T23:02:12.590708+0000 mgr.y (mgr.34107) 11454 : cluster [DBG] pgmap v12023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:13 smithi115 bash[45495]: audit 2022-04-22T23:02:12.686285+0000 mon.a (mon.0) 15860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:13 smithi115 bash[46934]: cluster 2022-04-22T23:02:12.590708+0000 mgr.y (mgr.34107) 11454 : cluster [DBG] pgmap v12023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:13 smithi115 bash[46934]: audit 2022-04-22T23:02:12.686285+0000 mon.a (mon.0) 15860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:14.565 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:14 smithi121 bash[36902]: audit 2022-04-22T23:02:14.257986+0000 mon.a (mon.0) 15861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:14.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:14 smithi115 bash[45495]: audit 2022-04-22T23:02:14.257986+0000 mon.a (mon.0) 15861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:14 smithi115 bash[46934]: audit 2022-04-22T23:02:14.257986+0000 mon.a (mon.0) 15861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:14.796Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:15 smithi121 bash[36902]: audit 2022-04-22T23:02:14.476935+0000 mon.a (mon.0) 15862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:02:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:15 smithi121 bash[36902]: audit 2022-04-22T23:02:14.479973+0000 mon.a (mon.0) 15863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:02:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:15 smithi121 bash[36902]: audit 2022-04-22T23:02:14.481711+0000 mon.a (mon.0) 15864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:02:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:15 smithi121 bash[36902]: cluster 2022-04-22T23:02:14.591858+0000 mgr.y (mgr.34107) 11455 : cluster [DBG] pgmap v12024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:15 smithi121 bash[36902]: audit 2022-04-22T23:02:14.846409+0000 mon.a (mon.0) 15865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:02:15.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[46934]: audit 2022-04-22T23:02:14.476935+0000 mon.a (mon.0) 15862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[46934]: audit 2022-04-22T23:02:14.479973+0000 mon.a (mon.0) 15863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[46934]: audit 2022-04-22T23:02:14.481711+0000 mon.a (mon.0) 15864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[46934]: cluster 2022-04-22T23:02:14.591858+0000 mgr.y (mgr.34107) 11455 : cluster [DBG] pgmap v12024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[46934]: audit 2022-04-22T23:02:14.846409+0000 mon.a (mon.0) 15865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[45495]: audit 2022-04-22T23:02:14.476935+0000 mon.a (mon.0) 15862 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:02:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[45495]: audit 2022-04-22T23:02:14.479973+0000 mon.a (mon.0) 15863 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:02:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[45495]: audit 2022-04-22T23:02:14.481711+0000 mon.a (mon.0) 15864 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:02:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[45495]: cluster 2022-04-22T23:02:14.591858+0000 mgr.y (mgr.34107) 11455 : cluster [DBG] pgmap v12024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:15 smithi115 bash[45495]: audit 2022-04-22T23:02:14.846409+0000 mon.a (mon.0) 15865 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:02:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:16 smithi121 bash[36902]: audit 2022-04-22T23:02:16.063294+0000 mon.a (mon.0) 15866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:16 smithi115 bash[45495]: audit 2022-04-22T23:02:16.063294+0000 mon.a (mon.0) 15866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:16 smithi115 bash[46934]: audit 2022-04-22T23:02:16.063294+0000 mon.a (mon.0) 15866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:17.642 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:02:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:17 smithi121 bash[36902]: cluster 2022-04-22T23:02:16.592914+0000 mgr.y (mgr.34107) 11456 : cluster [DBG] pgmap v12025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:17 smithi121 bash[36902]: audit 2022-04-22T23:02:17.139519+0000 mon.a (mon.0) 15867 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:02:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:17 smithi121 bash[36902]: audit 2022-04-22T23:02:17.140036+0000 mon.a (mon.0) 15868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:02:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[46934]: cluster 2022-04-22T23:02:16.592914+0000 mgr.y (mgr.34107) 11456 : cluster [DBG] pgmap v12025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[46934]: audit 2022-04-22T23:02:17.139519+0000 mon.a (mon.0) 15867 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:02:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[46934]: audit 2022-04-22T23:02:17.140036+0000 mon.a (mon.0) 15868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:02:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[45495]: cluster 2022-04-22T23:02:16.592914+0000 mgr.y (mgr.34107) 11456 : cluster [DBG] pgmap v12025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[45495]: audit 2022-04-22T23:02:17.139519+0000 mon.a (mon.0) 15867 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:02:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:17 smithi115 bash[45495]: audit 2022-04-22T23:02:17.140036+0000 mon.a (mon.0) 15868 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:02:17.979 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:02:17.979 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:02:17.979 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:02:17.979 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:02:17.980 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:02:17.981 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:02:18.392 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:18.439 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:02:18.440 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:02:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:18 smithi121 bash[36902]: audit 2022-04-22T23:02:17.623111+0000 mgr.y (mgr.34107) 11457 : audit [DBG] from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:18 smithi121 bash[36902]: audit 2022-04-22T23:02:17.689286+0000 mon.a (mon.0) 15869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:18 smithi121 bash[36902]: audit 2022-04-22T23:02:17.962345+0000 mgr.y (mgr.34107) 11458 : audit [DBG] from='client.57151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[45495]: audit 2022-04-22T23:02:17.623111+0000 mgr.y (mgr.34107) 11457 : audit [DBG] from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[45495]: audit 2022-04-22T23:02:17.689286+0000 mon.a (mon.0) 15869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[45495]: audit 2022-04-22T23:02:17.962345+0000 mgr.y (mgr.34107) 11458 : audit [DBG] from='client.57151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[46934]: audit 2022-04-22T23:02:17.623111+0000 mgr.y (mgr.34107) 11457 : audit [DBG] from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[46934]: audit 2022-04-22T23:02:17.689286+0000 mon.a (mon.0) 15869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:18 smithi115 bash[46934]: audit 2022-04-22T23:02:17.962345+0000 mgr.y (mgr.34107) 11458 : audit [DBG] from='client.57151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:19 smithi121 bash[36902]: audit 2022-04-22T23:02:18.389133+0000 mon.a (mon.0) 15870 : audit [DBG] from='client.? 172.21.15.115:0/1672576814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:19 smithi121 bash[36902]: cluster 2022-04-22T23:02:18.594054+0000 mgr.y (mgr.34107) 11459 : cluster [DBG] pgmap v12026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:19 smithi121 bash[36902]: audit 2022-04-22T23:02:19.261560+0000 mon.a (mon.0) 15871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[45495]: audit 2022-04-22T23:02:18.389133+0000 mon.a (mon.0) 15870 : audit [DBG] from='client.? 172.21.15.115:0/1672576814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[45495]: cluster 2022-04-22T23:02:18.594054+0000 mgr.y (mgr.34107) 11459 : cluster [DBG] pgmap v12026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[45495]: audit 2022-04-22T23:02:19.261560+0000 mon.a (mon.0) 15871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[46934]: audit 2022-04-22T23:02:18.389133+0000 mon.a (mon.0) 15870 : audit [DBG] from='client.? 172.21.15.115:0/1672576814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[46934]: cluster 2022-04-22T23:02:18.594054+0000 mgr.y (mgr.34107) 11459 : cluster [DBG] pgmap v12026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:19 smithi115 bash[46934]: audit 2022-04-22T23:02:19.261560+0000 mon.a (mon.0) 15871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:21 smithi121 bash[36902]: cluster 2022-04-22T23:02:20.595132+0000 mgr.y (mgr.34107) 11460 : cluster [DBG] pgmap v12027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:21 smithi121 bash[36902]: audit 2022-04-22T23:02:21.066248+0000 mon.a (mon.0) 15872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:21 smithi115 bash[45495]: cluster 2022-04-22T23:02:20.595132+0000 mgr.y (mgr.34107) 11460 : cluster [DBG] pgmap v12027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:21 smithi115 bash[45495]: audit 2022-04-22T23:02:21.066248+0000 mon.a (mon.0) 15872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:21 smithi115 bash[46934]: cluster 2022-04-22T23:02:20.595132+0000 mgr.y (mgr.34107) 11460 : cluster [DBG] pgmap v12027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:21 smithi115 bash[46934]: audit 2022-04-22T23:02:21.066248+0000 mon.a (mon.0) 15872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:23.247 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:22.941Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:23.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:23 smithi115 bash[46934]: cluster 2022-04-22T23:02:22.596246+0000 mgr.y (mgr.34107) 11461 : cluster [DBG] pgmap v12028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:23.249 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:23 smithi115 bash[46934]: audit 2022-04-22T23:02:22.692173+0000 mon.a (mon.0) 15873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:23 smithi121 bash[36902]: cluster 2022-04-22T23:02:22.596246+0000 mgr.y (mgr.34107) 11461 : cluster [DBG] pgmap v12028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:23.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:23 smithi121 bash[36902]: audit 2022-04-22T23:02:22.692173+0000 mon.a (mon.0) 15873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:23 smithi115 bash[45495]: cluster 2022-04-22T23:02:22.596246+0000 mgr.y (mgr.34107) 11461 : cluster [DBG] pgmap v12028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:23 smithi115 bash[45495]: audit 2022-04-22T23:02:22.692173+0000 mon.a (mon.0) 15873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:24 smithi121 bash[36902]: audit 2022-04-22T23:02:24.265019+0000 mon.a (mon.0) 15874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:24 smithi115 bash[45495]: audit 2022-04-22T23:02:24.265019+0000 mon.a (mon.0) 15874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:24 smithi115 bash[46934]: audit 2022-04-22T23:02:24.265019+0000 mon.a (mon.0) 15874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:24.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:25 smithi121 bash[36902]: cluster 2022-04-22T23:02:24.597406+0000 mgr.y (mgr.34107) 11462 : cluster [DBG] pgmap v12029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:25.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:25 smithi115 bash[45495]: cluster 2022-04-22T23:02:24.597406+0000 mgr.y (mgr.34107) 11462 : cluster [DBG] pgmap v12029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:25 smithi115 bash[46934]: cluster 2022-04-22T23:02:24.597406+0000 mgr.y (mgr.34107) 11462 : cluster [DBG] pgmap v12029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:26 smithi121 bash[36902]: audit 2022-04-22T23:02:26.069238+0000 mon.a (mon.0) 15875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:26 smithi115 bash[45495]: audit 2022-04-22T23:02:26.069238+0000 mon.a (mon.0) 15875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:26 smithi115 bash[46934]: audit 2022-04-22T23:02:26.069238+0000 mon.a (mon.0) 15875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:27 smithi121 bash[36902]: cluster 2022-04-22T23:02:26.598519+0000 mgr.y (mgr.34107) 11463 : cluster [DBG] pgmap v12030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:27.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:27 smithi115 bash[45495]: cluster 2022-04-22T23:02:26.598519+0000 mgr.y (mgr.34107) 11463 : cluster [DBG] pgmap v12030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:27.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:27 smithi115 bash[46934]: cluster 2022-04-22T23:02:26.598519+0000 mgr.y (mgr.34107) 11463 : cluster [DBG] pgmap v12030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:28 smithi121 bash[36902]: audit 2022-04-22T23:02:27.695519+0000 mon.a (mon.0) 15876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:28 smithi115 bash[45495]: audit 2022-04-22T23:02:27.695519+0000 mon.a (mon.0) 15876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:28 smithi115 bash[46934]: audit 2022-04-22T23:02:27.695519+0000 mon.a (mon.0) 15876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:29 smithi121 bash[36902]: cluster 2022-04-22T23:02:28.599731+0000 mgr.y (mgr.34107) 11464 : cluster [DBG] pgmap v12031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:29 smithi121 bash[36902]: audit 2022-04-22T23:02:29.268573+0000 mon.a (mon.0) 15877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:29 smithi115 bash[46934]: cluster 2022-04-22T23:02:28.599731+0000 mgr.y (mgr.34107) 11464 : cluster [DBG] pgmap v12031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:29 smithi115 bash[46934]: audit 2022-04-22T23:02:29.268573+0000 mon.a (mon.0) 15877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:29 smithi115 bash[45495]: cluster 2022-04-22T23:02:28.599731+0000 mgr.y (mgr.34107) 11464 : cluster [DBG] pgmap v12031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:29 smithi115 bash[45495]: audit 2022-04-22T23:02:29.268573+0000 mon.a (mon.0) 15877 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:31 smithi121 bash[36902]: cluster 2022-04-22T23:02:30.600724+0000 mgr.y (mgr.34107) 11465 : cluster [DBG] pgmap v12032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:31 smithi121 bash[36902]: audit 2022-04-22T23:02:31.072347+0000 mon.a (mon.0) 15878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:31 smithi115 bash[45495]: cluster 2022-04-22T23:02:30.600724+0000 mgr.y (mgr.34107) 11465 : cluster [DBG] pgmap v12032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:31 smithi115 bash[45495]: audit 2022-04-22T23:02:31.072347+0000 mon.a (mon.0) 15878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:31 smithi115 bash[46934]: cluster 2022-04-22T23:02:30.600724+0000 mgr.y (mgr.34107) 11465 : cluster [DBG] pgmap v12032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:31 smithi115 bash[46934]: audit 2022-04-22T23:02:31.072347+0000 mon.a (mon.0) 15878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:33.248 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:32.942Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:33.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:33 smithi115 bash[46934]: cluster 2022-04-22T23:02:32.601730+0000 mgr.y (mgr.34107) 11466 : cluster [DBG] pgmap v12033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:33.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:33 smithi115 bash[46934]: audit 2022-04-22T23:02:32.698677+0000 mon.a (mon.0) 15879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:33 smithi121 bash[36902]: cluster 2022-04-22T23:02:32.601730+0000 mgr.y (mgr.34107) 11466 : cluster [DBG] pgmap v12033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:33.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:33 smithi121 bash[36902]: audit 2022-04-22T23:02:32.698677+0000 mon.a (mon.0) 15879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:33 smithi115 bash[45495]: cluster 2022-04-22T23:02:32.601730+0000 mgr.y (mgr.34107) 11466 : cluster [DBG] pgmap v12033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:33 smithi115 bash[45495]: audit 2022-04-22T23:02:32.698677+0000 mon.a (mon.0) 15879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:34 smithi121 bash[36902]: audit 2022-04-22T23:02:34.272093+0000 mon.a (mon.0) 15880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:34 smithi115 bash[45495]: audit 2022-04-22T23:02:34.272093+0000 mon.a (mon.0) 15880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:34 smithi115 bash[46934]: audit 2022-04-22T23:02:34.272093+0000 mon.a (mon.0) 15880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:34.798Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:35 smithi121 bash[36902]: cluster 2022-04-22T23:02:34.602771+0000 mgr.y (mgr.34107) 11467 : cluster [DBG] pgmap v12034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:35 smithi115 bash[45495]: cluster 2022-04-22T23:02:34.602771+0000 mgr.y (mgr.34107) 11467 : cluster [DBG] pgmap v12034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:35 smithi115 bash[46934]: cluster 2022-04-22T23:02:34.602771+0000 mgr.y (mgr.34107) 11467 : cluster [DBG] pgmap v12034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:36 smithi121 bash[36902]: audit 2022-04-22T23:02:36.075583+0000 mon.a (mon.0) 15881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:36 smithi115 bash[45495]: audit 2022-04-22T23:02:36.075583+0000 mon.a (mon.0) 15881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:36 smithi115 bash[46934]: audit 2022-04-22T23:02:36.075583+0000 mon.a (mon.0) 15881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:37 smithi121 bash[36902]: cluster 2022-04-22T23:02:36.603711+0000 mgr.y (mgr.34107) 11468 : cluster [DBG] pgmap v12035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:37 smithi115 bash[45495]: cluster 2022-04-22T23:02:36.603711+0000 mgr.y (mgr.34107) 11468 : cluster [DBG] pgmap v12035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:37 smithi115 bash[46934]: cluster 2022-04-22T23:02:36.603711+0000 mgr.y (mgr.34107) 11468 : cluster [DBG] pgmap v12035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:38 smithi121 bash[36902]: audit 2022-04-22T23:02:37.701848+0000 mon.a (mon.0) 15882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:38 smithi115 bash[45495]: audit 2022-04-22T23:02:37.701848+0000 mon.a (mon.0) 15882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:38 smithi115 bash[46934]: audit 2022-04-22T23:02:37.701848+0000 mon.a (mon.0) 15882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:39.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:39 smithi121 bash[36902]: cluster 2022-04-22T23:02:38.604835+0000 mgr.y (mgr.34107) 11469 : cluster [DBG] pgmap v12036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:39.802 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:39 smithi121 bash[36902]: audit 2022-04-22T23:02:39.275857+0000 mon.a (mon.0) 15883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:39.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:39 smithi115 bash[45495]: cluster 2022-04-22T23:02:38.604835+0000 mgr.y (mgr.34107) 11469 : cluster [DBG] pgmap v12036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:39.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:39 smithi115 bash[45495]: audit 2022-04-22T23:02:39.275857+0000 mon.a (mon.0) 15883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:39.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:39 smithi115 bash[46934]: cluster 2022-04-22T23:02:38.604835+0000 mgr.y (mgr.34107) 11469 : cluster [DBG] pgmap v12036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:39.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:39 smithi115 bash[46934]: audit 2022-04-22T23:02:39.275857+0000 mon.a (mon.0) 15883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:41 smithi121 bash[36902]: cluster 2022-04-22T23:02:40.605926+0000 mgr.y (mgr.34107) 11470 : cluster [DBG] pgmap v12037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:42.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:41 smithi121 bash[36902]: audit 2022-04-22T23:02:41.078086+0000 mon.a (mon.0) 15884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:41 smithi115 bash[45495]: cluster 2022-04-22T23:02:40.605926+0000 mgr.y (mgr.34107) 11470 : cluster [DBG] pgmap v12037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:41 smithi115 bash[45495]: audit 2022-04-22T23:02:41.078086+0000 mon.a (mon.0) 15884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:41 smithi115 bash[46934]: cluster 2022-04-22T23:02:40.605926+0000 mgr.y (mgr.34107) 11470 : cluster [DBG] pgmap v12037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:41 smithi115 bash[46934]: audit 2022-04-22T23:02:41.078086+0000 mon.a (mon.0) 15884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:43.253 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:42.942Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:43.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:43 smithi115 bash[46934]: cluster 2022-04-22T23:02:42.607019+0000 mgr.y (mgr.34107) 11471 : cluster [DBG] pgmap v12038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:43.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:43 smithi115 bash[46934]: audit 2022-04-22T23:02:42.705116+0000 mon.a (mon.0) 15885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:43.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:43 smithi115 bash[45495]: cluster 2022-04-22T23:02:42.607019+0000 mgr.y (mgr.34107) 11471 : cluster [DBG] pgmap v12038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:43.254 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:43 smithi115 bash[45495]: audit 2022-04-22T23:02:42.705116+0000 mon.a (mon.0) 15885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:43 smithi121 bash[36902]: cluster 2022-04-22T23:02:42.607019+0000 mgr.y (mgr.34107) 11471 : cluster [DBG] pgmap v12038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:43.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:43 smithi121 bash[36902]: audit 2022-04-22T23:02:42.705116+0000 mon.a (mon.0) 15885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:44 smithi121 bash[36902]: audit 2022-04-22T23:02:44.279577+0000 mon.a (mon.0) 15886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:44 smithi115 bash[45495]: audit 2022-04-22T23:02:44.279577+0000 mon.a (mon.0) 15886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:44 smithi115 bash[46934]: audit 2022-04-22T23:02:44.279577+0000 mon.a (mon.0) 15886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:44.797Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:45 smithi121 bash[36902]: cluster 2022-04-22T23:02:44.608139+0000 mgr.y (mgr.34107) 11472 : cluster [DBG] pgmap v12039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:45 smithi115 bash[45495]: cluster 2022-04-22T23:02:44.608139+0000 mgr.y (mgr.34107) 11472 : cluster [DBG] pgmap v12039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:45 smithi115 bash[46934]: cluster 2022-04-22T23:02:44.608139+0000 mgr.y (mgr.34107) 11472 : cluster [DBG] pgmap v12039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:46 smithi121 bash[36902]: audit 2022-04-22T23:02:46.080740+0000 mon.a (mon.0) 15887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:46 smithi115 bash[45495]: audit 2022-04-22T23:02:46.080740+0000 mon.a (mon.0) 15887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:46 smithi115 bash[46934]: audit 2022-04-22T23:02:46.080740+0000 mon.a (mon.0) 15887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:47 smithi121 bash[36902]: cluster 2022-04-22T23:02:46.609372+0000 mgr.y (mgr.34107) 11473 : cluster [DBG] pgmap v12040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:47 smithi115 bash[45495]: cluster 2022-04-22T23:02:46.609372+0000 mgr.y (mgr.34107) 11473 : cluster [DBG] pgmap v12040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:47 smithi115 bash[46934]: cluster 2022-04-22T23:02:46.609372+0000 mgr.y (mgr.34107) 11473 : cluster [DBG] pgmap v12040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:48.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:48 smithi121 bash[36902]: audit 2022-04-22T23:02:47.708109+0000 mon.a (mon.0) 15888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:48.770 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:02:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:48 smithi115 bash[46934]: audit 2022-04-22T23:02:47.708109+0000 mon.a (mon.0) 15888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:48 smithi115 bash[45495]: audit 2022-04-22T23:02:47.708109+0000 mon.a (mon.0) 15888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:02:49.122 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:02:49.123 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:02:49.124 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:02:49.124 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:02:49.532 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:02:49.532 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:02:49.533 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:02:49.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:02:49.534 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:02:49.534 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:02:49.534 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:02:49.534 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:02:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:49 smithi121 bash[36902]: cluster 2022-04-22T23:02:48.610207+0000 mgr.y (mgr.34107) 11474 : cluster [DBG] pgmap v12041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:49 smithi121 bash[36902]: audit 2022-04-22T23:02:48.752204+0000 mgr.y (mgr.34107) 11475 : audit [DBG] from='client.47226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:49 smithi121 bash[36902]: audit 2022-04-22T23:02:49.283135+0000 mon.a (mon.0) 15889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[45495]: cluster 2022-04-22T23:02:48.610207+0000 mgr.y (mgr.34107) 11474 : cluster [DBG] pgmap v12041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[45495]: audit 2022-04-22T23:02:48.752204+0000 mgr.y (mgr.34107) 11475 : audit [DBG] from='client.47226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[45495]: audit 2022-04-22T23:02:49.283135+0000 mon.a (mon.0) 15889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[46934]: cluster 2022-04-22T23:02:48.610207+0000 mgr.y (mgr.34107) 11474 : cluster [DBG] pgmap v12041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[46934]: audit 2022-04-22T23:02:48.752204+0000 mgr.y (mgr.34107) 11475 : audit [DBG] from='client.47226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:49 smithi115 bash[46934]: audit 2022-04-22T23:02:49.283135+0000 mon.a (mon.0) 15889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:50 smithi121 bash[36902]: audit 2022-04-22T23:02:49.099254+0000 mgr.y (mgr.34107) 11476 : audit [DBG] from='client.47232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:50.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:50 smithi121 bash[36902]: audit 2022-04-22T23:02:49.529126+0000 mon.c (mon.1) 389 : audit [DBG] from='client.? 172.21.15.115:0/1567388313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:50 smithi115 bash[45495]: audit 2022-04-22T23:02:49.099254+0000 mgr.y (mgr.34107) 11476 : audit [DBG] from='client.47232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:50 smithi115 bash[45495]: audit 2022-04-22T23:02:49.529126+0000 mon.c (mon.1) 389 : audit [DBG] from='client.? 172.21.15.115:0/1567388313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:50 smithi115 bash[46934]: audit 2022-04-22T23:02:49.099254+0000 mgr.y (mgr.34107) 11476 : audit [DBG] from='client.47232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:02:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:50 smithi115 bash[46934]: audit 2022-04-22T23:02:49.529126+0000 mon.c (mon.1) 389 : audit [DBG] from='client.? 172.21.15.115:0/1567388313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:02:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:51 smithi121 bash[36902]: cluster 2022-04-22T23:02:50.611185+0000 mgr.y (mgr.34107) 11477 : cluster [DBG] pgmap v12042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:51.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:51 smithi121 bash[36902]: audit 2022-04-22T23:02:51.083954+0000 mon.a (mon.0) 15890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:51.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:51 smithi115 bash[45495]: cluster 2022-04-22T23:02:50.611185+0000 mgr.y (mgr.34107) 11477 : cluster [DBG] pgmap v12042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:51 smithi115 bash[45495]: audit 2022-04-22T23:02:51.083954+0000 mon.a (mon.0) 15890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:51 smithi115 bash[46934]: cluster 2022-04-22T23:02:50.611185+0000 mgr.y (mgr.34107) 11477 : cluster [DBG] pgmap v12042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:51 smithi115 bash[46934]: audit 2022-04-22T23:02:51.083954+0000 mon.a (mon.0) 15890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:53.257 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:52.943Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:53 smithi121 bash[36902]: cluster 2022-04-22T23:02:52.612246+0000 mgr.y (mgr.34107) 11478 : cluster [DBG] pgmap v12043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:53.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:53 smithi121 bash[36902]: audit 2022-04-22T23:02:52.711196+0000 mon.a (mon.0) 15891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:53 smithi115 bash[45495]: cluster 2022-04-22T23:02:52.612246+0000 mgr.y (mgr.34107) 11478 : cluster [DBG] pgmap v12043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:53 smithi115 bash[45495]: audit 2022-04-22T23:02:52.711196+0000 mon.a (mon.0) 15891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:53 smithi115 bash[46934]: cluster 2022-04-22T23:02:52.612246+0000 mgr.y (mgr.34107) 11478 : cluster [DBG] pgmap v12043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:53 smithi115 bash[46934]: audit 2022-04-22T23:02:52.711196+0000 mon.a (mon.0) 15891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:54 smithi121 bash[36902]: audit 2022-04-22T23:02:54.286400+0000 mon.a (mon.0) 15892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:54 smithi115 bash[45495]: audit 2022-04-22T23:02:54.286400+0000 mon.a (mon.0) 15892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:54 smithi115 bash[46934]: audit 2022-04-22T23:02:54.286400+0000 mon.a (mon.0) 15892 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:02:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:02:54.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:02:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:55 smithi121 bash[36902]: cluster 2022-04-22T23:02:54.613309+0000 mgr.y (mgr.34107) 11479 : cluster [DBG] pgmap v12044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:55 smithi115 bash[45495]: cluster 2022-04-22T23:02:54.613309+0000 mgr.y (mgr.34107) 11479 : cluster [DBG] pgmap v12044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:55 smithi115 bash[46934]: cluster 2022-04-22T23:02:54.613309+0000 mgr.y (mgr.34107) 11479 : cluster [DBG] pgmap v12044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:56 smithi121 bash[36902]: audit 2022-04-22T23:02:56.087279+0000 mon.a (mon.0) 15893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:56 smithi115 bash[45495]: audit 2022-04-22T23:02:56.087279+0000 mon.a (mon.0) 15893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:56 smithi115 bash[46934]: audit 2022-04-22T23:02:56.087279+0000 mon.a (mon.0) 15893 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:02:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:57 smithi121 bash[36902]: cluster 2022-04-22T23:02:56.614294+0000 mgr.y (mgr.34107) 11480 : cluster [DBG] pgmap v12045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:57 smithi115 bash[45495]: cluster 2022-04-22T23:02:56.614294+0000 mgr.y (mgr.34107) 11480 : cluster [DBG] pgmap v12045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:57 smithi115 bash[46934]: cluster 2022-04-22T23:02:56.614294+0000 mgr.y (mgr.34107) 11480 : cluster [DBG] pgmap v12045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:58.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:58 smithi121 bash[36902]: audit 2022-04-22T23:02:57.714433+0000 mon.a (mon.0) 15894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:58 smithi115 bash[45495]: audit 2022-04-22T23:02:57.714433+0000 mon.a (mon.0) 15894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:58 smithi115 bash[46934]: audit 2022-04-22T23:02:57.714433+0000 mon.a (mon.0) 15894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:02:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:59 smithi121 bash[36902]: cluster 2022-04-22T23:02:58.615438+0000 mgr.y (mgr.34107) 11481 : cluster [DBG] pgmap v12046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:02:59 smithi121 bash[36902]: audit 2022-04-22T23:02:59.290395+0000 mon.a (mon.0) 15895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:59 smithi115 bash[45495]: cluster 2022-04-22T23:02:58.615438+0000 mgr.y (mgr.34107) 11481 : cluster [DBG] pgmap v12046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:02:59 smithi115 bash[45495]: audit 2022-04-22T23:02:59.290395+0000 mon.a (mon.0) 15895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:02:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:59 smithi115 bash[46934]: cluster 2022-04-22T23:02:58.615438+0000 mgr.y (mgr.34107) 11481 : cluster [DBG] pgmap v12046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:02:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:02:59 smithi115 bash[46934]: audit 2022-04-22T23:02:59.290395+0000 mon.a (mon.0) 15895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:01 smithi121 bash[36902]: cluster 2022-04-22T23:03:00.616483+0000 mgr.y (mgr.34107) 11482 : cluster [DBG] pgmap v12047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:01 smithi121 bash[36902]: audit 2022-04-22T23:03:01.090289+0000 mon.a (mon.0) 15896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:01 smithi115 bash[46934]: cluster 2022-04-22T23:03:00.616483+0000 mgr.y (mgr.34107) 11482 : cluster [DBG] pgmap v12047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:01 smithi115 bash[46934]: audit 2022-04-22T23:03:01.090289+0000 mon.a (mon.0) 15896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:01 smithi115 bash[45495]: cluster 2022-04-22T23:03:00.616483+0000 mgr.y (mgr.34107) 11482 : cluster [DBG] pgmap v12047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:01 smithi115 bash[45495]: audit 2022-04-22T23:03:01.090289+0000 mon.a (mon.0) 15896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:03.260 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:02.942Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:03.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:03 smithi115 bash[45495]: cluster 2022-04-22T23:03:02.617541+0000 mgr.y (mgr.34107) 11483 : cluster [DBG] pgmap v12048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:03.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:03 smithi115 bash[45495]: audit 2022-04-22T23:03:02.717781+0000 mon.a (mon.0) 15897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:03 smithi121 bash[36902]: cluster 2022-04-22T23:03:02.617541+0000 mgr.y (mgr.34107) 11483 : cluster [DBG] pgmap v12048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:03 smithi121 bash[36902]: audit 2022-04-22T23:03:02.717781+0000 mon.a (mon.0) 15897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:03.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:03 smithi115 bash[46934]: cluster 2022-04-22T23:03:02.617541+0000 mgr.y (mgr.34107) 11483 : cluster [DBG] pgmap v12048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:03 smithi115 bash[46934]: audit 2022-04-22T23:03:02.717781+0000 mon.a (mon.0) 15897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:04 smithi121 bash[36902]: audit 2022-04-22T23:03:04.294400+0000 mon.a (mon.0) 15898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:04 smithi115 bash[45495]: audit 2022-04-22T23:03:04.294400+0000 mon.a (mon.0) 15898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:04 smithi115 bash[46934]: audit 2022-04-22T23:03:04.294400+0000 mon.a (mon.0) 15898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:04.798Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:05 smithi121 bash[36902]: cluster 2022-04-22T23:03:04.618656+0000 mgr.y (mgr.34107) 11484 : cluster [DBG] pgmap v12049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:05 smithi115 bash[45495]: cluster 2022-04-22T23:03:04.618656+0000 mgr.y (mgr.34107) 11484 : cluster [DBG] pgmap v12049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:05 smithi115 bash[46934]: cluster 2022-04-22T23:03:04.618656+0000 mgr.y (mgr.34107) 11484 : cluster [DBG] pgmap v12049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:06 smithi121 bash[36902]: audit 2022-04-22T23:03:06.093829+0000 mon.a (mon.0) 15899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:06 smithi115 bash[45495]: audit 2022-04-22T23:03:06.093829+0000 mon.a (mon.0) 15899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:06 smithi115 bash[46934]: audit 2022-04-22T23:03:06.093829+0000 mon.a (mon.0) 15899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:07 smithi121 bash[36902]: cluster 2022-04-22T23:03:06.619819+0000 mgr.y (mgr.34107) 11485 : cluster [DBG] pgmap v12050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:07 smithi115 bash[45495]: cluster 2022-04-22T23:03:06.619819+0000 mgr.y (mgr.34107) 11485 : cluster [DBG] pgmap v12050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:07 smithi115 bash[46934]: cluster 2022-04-22T23:03:06.619819+0000 mgr.y (mgr.34107) 11485 : cluster [DBG] pgmap v12050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:08 smithi121 bash[36902]: audit 2022-04-22T23:03:07.720633+0000 mon.a (mon.0) 15900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:08 smithi115 bash[45495]: audit 2022-04-22T23:03:07.720633+0000 mon.a (mon.0) 15900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:08 smithi115 bash[46934]: audit 2022-04-22T23:03:07.720633+0000 mon.a (mon.0) 15900 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:09 smithi121 bash[36902]: cluster 2022-04-22T23:03:08.621003+0000 mgr.y (mgr.34107) 11486 : cluster [DBG] pgmap v12051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:09 smithi121 bash[36902]: audit 2022-04-22T23:03:09.297988+0000 mon.a (mon.0) 15901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:09 smithi115 bash[45495]: cluster 2022-04-22T23:03:08.621003+0000 mgr.y (mgr.34107) 11486 : cluster [DBG] pgmap v12051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:09 smithi115 bash[45495]: audit 2022-04-22T23:03:09.297988+0000 mon.a (mon.0) 15901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:09 smithi115 bash[46934]: cluster 2022-04-22T23:03:08.621003+0000 mgr.y (mgr.34107) 11486 : cluster [DBG] pgmap v12051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:09 smithi115 bash[46934]: audit 2022-04-22T23:03:09.297988+0000 mon.a (mon.0) 15901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:11 smithi121 bash[36902]: cluster 2022-04-22T23:03:10.622062+0000 mgr.y (mgr.34107) 11487 : cluster [DBG] pgmap v12052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:11 smithi121 bash[36902]: audit 2022-04-22T23:03:11.096883+0000 mon.a (mon.0) 15902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:11 smithi115 bash[45495]: cluster 2022-04-22T23:03:10.622062+0000 mgr.y (mgr.34107) 11487 : cluster [DBG] pgmap v12052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:11 smithi115 bash[45495]: audit 2022-04-22T23:03:11.096883+0000 mon.a (mon.0) 15902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:11 smithi115 bash[46934]: cluster 2022-04-22T23:03:10.622062+0000 mgr.y (mgr.34107) 11487 : cluster [DBG] pgmap v12052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:11 smithi115 bash[46934]: audit 2022-04-22T23:03:11.096883+0000 mon.a (mon.0) 15902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:13.263 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:12.945Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:13 smithi121 bash[36902]: cluster 2022-04-22T23:03:12.623155+0000 mgr.y (mgr.34107) 11488 : cluster [DBG] pgmap v12053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:13.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:13 smithi121 bash[36902]: audit 2022-04-22T23:03:12.723716+0000 mon.a (mon.0) 15903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:13 smithi115 bash[45495]: cluster 2022-04-22T23:03:12.623155+0000 mgr.y (mgr.34107) 11488 : cluster [DBG] pgmap v12053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:13 smithi115 bash[45495]: audit 2022-04-22T23:03:12.723716+0000 mon.a (mon.0) 15903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:13 smithi115 bash[46934]: cluster 2022-04-22T23:03:12.623155+0000 mgr.y (mgr.34107) 11488 : cluster [DBG] pgmap v12053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:13 smithi115 bash[46934]: audit 2022-04-22T23:03:12.723716+0000 mon.a (mon.0) 15903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:14 smithi121 bash[36902]: audit 2022-04-22T23:03:14.301745+0000 mon.a (mon.0) 15904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:14.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:14 smithi115 bash[45495]: audit 2022-04-22T23:03:14.301745+0000 mon.a (mon.0) 15904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:14 smithi115 bash[46934]: audit 2022-04-22T23:03:14.301745+0000 mon.a (mon.0) 15904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:15.217 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:14.799Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:15 smithi121 bash[36902]: cluster 2022-04-22T23:03:14.624482+0000 mgr.y (mgr.34107) 11489 : cluster [DBG] pgmap v12054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:15 smithi121 bash[36902]: audit 2022-04-22T23:03:14.856129+0000 mon.a (mon.0) 15905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:03:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:15 smithi121 bash[36902]: audit 2022-04-22T23:03:14.859245+0000 mon.a (mon.0) 15906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:03:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:15 smithi121 bash[36902]: audit 2022-04-22T23:03:14.860916+0000 mon.a (mon.0) 15907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:03:15.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:15 smithi121 bash[36902]: audit 2022-04-22T23:03:15.225332+0000 mon.a (mon.0) 15908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:03:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[45495]: cluster 2022-04-22T23:03:14.624482+0000 mgr.y (mgr.34107) 11489 : cluster [DBG] pgmap v12054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[45495]: audit 2022-04-22T23:03:14.856129+0000 mon.a (mon.0) 15905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:03:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[45495]: audit 2022-04-22T23:03:14.859245+0000 mon.a (mon.0) 15906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:03:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[45495]: audit 2022-04-22T23:03:14.860916+0000 mon.a (mon.0) 15907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:03:15.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[45495]: audit 2022-04-22T23:03:15.225332+0000 mon.a (mon.0) 15908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:03:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[46934]: cluster 2022-04-22T23:03:14.624482+0000 mgr.y (mgr.34107) 11489 : cluster [DBG] pgmap v12054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[46934]: audit 2022-04-22T23:03:14.856129+0000 mon.a (mon.0) 15905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:03:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[46934]: audit 2022-04-22T23:03:14.859245+0000 mon.a (mon.0) 15906 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:03:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[46934]: audit 2022-04-22T23:03:14.860916+0000 mon.a (mon.0) 15907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:03:15.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:15 smithi115 bash[46934]: audit 2022-04-22T23:03:15.225332+0000 mon.a (mon.0) 15908 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:03:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:16 smithi121 bash[36902]: audit 2022-04-22T23:03:16.100158+0000 mon.a (mon.0) 15909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:16 smithi115 bash[45495]: audit 2022-04-22T23:03:16.100158+0000 mon.a (mon.0) 15909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:16 smithi115 bash[46934]: audit 2022-04-22T23:03:16.100158+0000 mon.a (mon.0) 15909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:17 smithi121 bash[36902]: cluster 2022-04-22T23:03:16.625619+0000 mgr.y (mgr.34107) 11490 : cluster [DBG] pgmap v12055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:17 smithi121 bash[36902]: audit 2022-04-22T23:03:17.151368+0000 mon.a (mon.0) 15910 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:03:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:17 smithi121 bash[36902]: audit 2022-04-22T23:03:17.151923+0000 mon.a (mon.0) 15911 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[45495]: cluster 2022-04-22T23:03:16.625619+0000 mgr.y (mgr.34107) 11490 : cluster [DBG] pgmap v12055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[45495]: audit 2022-04-22T23:03:17.151368+0000 mon.a (mon.0) 15910 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[45495]: audit 2022-04-22T23:03:17.151923+0000 mon.a (mon.0) 15911 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[46934]: cluster 2022-04-22T23:03:16.625619+0000 mgr.y (mgr.34107) 11490 : cluster [DBG] pgmap v12055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[46934]: audit 2022-04-22T23:03:17.151368+0000 mon.a (mon.0) 15910 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:03:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:17 smithi115 bash[46934]: audit 2022-04-22T23:03:17.151923+0000 mon.a (mon.0) 15911 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:03:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:18 smithi121 bash[36902]: audit 2022-04-22T23:03:17.726661+0000 mon.a (mon.0) 15912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:18 smithi115 bash[45495]: audit 2022-04-22T23:03:17.726661+0000 mon.a (mon.0) 15912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:18 smithi115 bash[46934]: audit 2022-04-22T23:03:17.726661+0000 mon.a (mon.0) 15912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:19 smithi121 bash[36902]: cluster 2022-04-22T23:03:18.626788+0000 mgr.y (mgr.34107) 11491 : cluster [DBG] pgmap v12056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:19 smithi121 bash[36902]: audit 2022-04-22T23:03:19.305491+0000 mon.a (mon.0) 15913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:19 smithi115 bash[45495]: cluster 2022-04-22T23:03:18.626788+0000 mgr.y (mgr.34107) 11491 : cluster [DBG] pgmap v12056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:19 smithi115 bash[45495]: audit 2022-04-22T23:03:19.305491+0000 mon.a (mon.0) 15913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:19 smithi115 bash[46934]: cluster 2022-04-22T23:03:18.626788+0000 mgr.y (mgr.34107) 11491 : cluster [DBG] pgmap v12056: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:19 smithi115 bash[46934]: audit 2022-04-22T23:03:19.305491+0000 mon.a (mon.0) 15913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:19.908 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:03:20.258 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:03:20.259 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:03:20.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:03:20.261 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:03:20.261 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:03:20.669 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:03:20.669 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:03:20.669 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:03:20.670 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:03:20.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:20.671 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:03:20.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:03:20.671 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:03:20.671 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:03:20.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:20 smithi121 bash[36902]: audit 2022-04-22T23:03:19.887859+0000 mgr.y (mgr.34107) 11492 : audit [DBG] from='client.57181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:20 smithi115 bash[45495]: audit 2022-04-22T23:03:19.887859+0000 mgr.y (mgr.34107) 11492 : audit [DBG] from='client.57181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:20 smithi115 bash[46934]: audit 2022-04-22T23:03:19.887859+0000 mgr.y (mgr.34107) 11492 : audit [DBG] from='client.57181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:21.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:21 smithi121 bash[36902]: audit 2022-04-22T23:03:20.234125+0000 mgr.y (mgr.34107) 11493 : audit [DBG] from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:21 smithi121 bash[36902]: cluster 2022-04-22T23:03:20.627861+0000 mgr.y (mgr.34107) 11494 : cluster [DBG] pgmap v12057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:21 smithi121 bash[36902]: audit 2022-04-22T23:03:20.665905+0000 mon.a (mon.0) 15914 : audit [DBG] from='client.? 172.21.15.115:0/3347031456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:21 smithi121 bash[36902]: audit 2022-04-22T23:03:21.103165+0000 mon.a (mon.0) 15915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[45495]: audit 2022-04-22T23:03:20.234125+0000 mgr.y (mgr.34107) 11493 : audit [DBG] from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[45495]: cluster 2022-04-22T23:03:20.627861+0000 mgr.y (mgr.34107) 11494 : cluster [DBG] pgmap v12057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[45495]: audit 2022-04-22T23:03:20.665905+0000 mon.a (mon.0) 15914 : audit [DBG] from='client.? 172.21.15.115:0/3347031456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[45495]: audit 2022-04-22T23:03:21.103165+0000 mon.a (mon.0) 15915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[46934]: audit 2022-04-22T23:03:20.234125+0000 mgr.y (mgr.34107) 11493 : audit [DBG] from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[46934]: cluster 2022-04-22T23:03:20.627861+0000 mgr.y (mgr.34107) 11494 : cluster [DBG] pgmap v12057: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[46934]: audit 2022-04-22T23:03:20.665905+0000 mon.a (mon.0) 15914 : audit [DBG] from='client.? 172.21.15.115:0/3347031456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:21 smithi115 bash[46934]: audit 2022-04-22T23:03:21.103165+0000 mon.a (mon.0) 15915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:23.267 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:22.945Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:23 smithi115 bash[45495]: cluster 2022-04-22T23:03:22.628928+0000 mgr.y (mgr.34107) 11495 : cluster [DBG] pgmap v12058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:23 smithi115 bash[45495]: audit 2022-04-22T23:03:22.729896+0000 mon.a (mon.0) 15916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:23 smithi115 bash[46934]: cluster 2022-04-22T23:03:22.628928+0000 mgr.y (mgr.34107) 11495 : cluster [DBG] pgmap v12058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:23.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:23 smithi115 bash[46934]: audit 2022-04-22T23:03:22.729896+0000 mon.a (mon.0) 15916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:23 smithi121 bash[36902]: cluster 2022-04-22T23:03:22.628928+0000 mgr.y (mgr.34107) 11495 : cluster [DBG] pgmap v12058: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:23 smithi121 bash[36902]: audit 2022-04-22T23:03:22.729896+0000 mon.a (mon.0) 15916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:24 smithi121 bash[36902]: audit 2022-04-22T23:03:24.308953+0000 mon.a (mon.0) 15917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:24 smithi115 bash[45495]: audit 2022-04-22T23:03:24.308953+0000 mon.a (mon.0) 15917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:24.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:24 smithi115 bash[46934]: audit 2022-04-22T23:03:24.308953+0000 mon.a (mon.0) 15917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:24.800Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:25 smithi121 bash[36902]: cluster 2022-04-22T23:03:24.629911+0000 mgr.y (mgr.34107) 11496 : cluster [DBG] pgmap v12059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:25.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:25 smithi115 bash[45495]: cluster 2022-04-22T23:03:24.629911+0000 mgr.y (mgr.34107) 11496 : cluster [DBG] pgmap v12059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:25 smithi115 bash[46934]: cluster 2022-04-22T23:03:24.629911+0000 mgr.y (mgr.34107) 11496 : cluster [DBG] pgmap v12059: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:26 smithi121 bash[36902]: audit 2022-04-22T23:03:26.106321+0000 mon.a (mon.0) 15918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:26.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:26 smithi115 bash[46934]: audit 2022-04-22T23:03:26.106321+0000 mon.a (mon.0) 15918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:26 smithi115 bash[45495]: audit 2022-04-22T23:03:26.106321+0000 mon.a (mon.0) 15918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:27 smithi121 bash[36902]: cluster 2022-04-22T23:03:26.631082+0000 mgr.y (mgr.34107) 11497 : cluster [DBG] pgmap v12060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:27 smithi115 bash[45495]: cluster 2022-04-22T23:03:26.631082+0000 mgr.y (mgr.34107) 11497 : cluster [DBG] pgmap v12060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:27 smithi115 bash[46934]: cluster 2022-04-22T23:03:26.631082+0000 mgr.y (mgr.34107) 11497 : cluster [DBG] pgmap v12060: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:28 smithi121 bash[36902]: audit 2022-04-22T23:03:27.732973+0000 mon.a (mon.0) 15919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:28 smithi115 bash[45495]: audit 2022-04-22T23:03:27.732973+0000 mon.a (mon.0) 15919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:28 smithi115 bash[46934]: audit 2022-04-22T23:03:27.732973+0000 mon.a (mon.0) 15919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:29 smithi121 bash[36902]: cluster 2022-04-22T23:03:28.632264+0000 mgr.y (mgr.34107) 11498 : cluster [DBG] pgmap v12061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:29 smithi121 bash[36902]: audit 2022-04-22T23:03:29.312501+0000 mon.a (mon.0) 15920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:29.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:29 smithi115 bash[45495]: cluster 2022-04-22T23:03:28.632264+0000 mgr.y (mgr.34107) 11498 : cluster [DBG] pgmap v12061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:29 smithi115 bash[45495]: audit 2022-04-22T23:03:29.312501+0000 mon.a (mon.0) 15920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:29 smithi115 bash[46934]: cluster 2022-04-22T23:03:28.632264+0000 mgr.y (mgr.34107) 11498 : cluster [DBG] pgmap v12061: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:29 smithi115 bash[46934]: audit 2022-04-22T23:03:29.312501+0000 mon.a (mon.0) 15920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:31 smithi121 bash[36902]: cluster 2022-04-22T23:03:30.633472+0000 mgr.y (mgr.34107) 11499 : cluster [DBG] pgmap v12062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:31 smithi121 bash[36902]: audit 2022-04-22T23:03:31.109221+0000 mon.a (mon.0) 15921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:32.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:31 smithi115 bash[45495]: cluster 2022-04-22T23:03:30.633472+0000 mgr.y (mgr.34107) 11499 : cluster [DBG] pgmap v12062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:31 smithi115 bash[45495]: audit 2022-04-22T23:03:31.109221+0000 mon.a (mon.0) 15921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:31 smithi115 bash[46934]: cluster 2022-04-22T23:03:30.633472+0000 mgr.y (mgr.34107) 11499 : cluster [DBG] pgmap v12062: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:31 smithi115 bash[46934]: audit 2022-04-22T23:03:31.109221+0000 mon.a (mon.0) 15921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:33.270 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:32.946Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:33 smithi115 bash[45495]: cluster 2022-04-22T23:03:32.634503+0000 mgr.y (mgr.34107) 11500 : cluster [DBG] pgmap v12063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:33 smithi115 bash[45495]: audit 2022-04-22T23:03:32.736090+0000 mon.a (mon.0) 15922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:33 smithi115 bash[46934]: cluster 2022-04-22T23:03:32.634503+0000 mgr.y (mgr.34107) 11500 : cluster [DBG] pgmap v12063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:33 smithi115 bash[46934]: audit 2022-04-22T23:03:32.736090+0000 mon.a (mon.0) 15922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:33 smithi121 bash[36902]: cluster 2022-04-22T23:03:32.634503+0000 mgr.y (mgr.34107) 11500 : cluster [DBG] pgmap v12063: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:33 smithi121 bash[36902]: audit 2022-04-22T23:03:32.736090+0000 mon.a (mon.0) 15922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:34 smithi121 bash[36902]: audit 2022-04-22T23:03:34.316184+0000 mon.a (mon.0) 15923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:34 smithi115 bash[45495]: audit 2022-04-22T23:03:34.316184+0000 mon.a (mon.0) 15923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:34 smithi115 bash[46934]: audit 2022-04-22T23:03:34.316184+0000 mon.a (mon.0) 15923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:34.801Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:35.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:35 smithi121 bash[36902]: cluster 2022-04-22T23:03:34.635636+0000 mgr.y (mgr.34107) 11501 : cluster [DBG] pgmap v12064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:35 smithi115 bash[45495]: cluster 2022-04-22T23:03:34.635636+0000 mgr.y (mgr.34107) 11501 : cluster [DBG] pgmap v12064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:35 smithi115 bash[46934]: cluster 2022-04-22T23:03:34.635636+0000 mgr.y (mgr.34107) 11501 : cluster [DBG] pgmap v12064: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:36 smithi121 bash[36902]: audit 2022-04-22T23:03:36.112292+0000 mon.a (mon.0) 15924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:36 smithi115 bash[45495]: audit 2022-04-22T23:03:36.112292+0000 mon.a (mon.0) 15924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:36 smithi115 bash[46934]: audit 2022-04-22T23:03:36.112292+0000 mon.a (mon.0) 15924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:37 smithi121 bash[36902]: cluster 2022-04-22T23:03:36.636675+0000 mgr.y (mgr.34107) 11502 : cluster [DBG] pgmap v12065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:37 smithi115 bash[45495]: cluster 2022-04-22T23:03:36.636675+0000 mgr.y (mgr.34107) 11502 : cluster [DBG] pgmap v12065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:37 smithi115 bash[46934]: cluster 2022-04-22T23:03:36.636675+0000 mgr.y (mgr.34107) 11502 : cluster [DBG] pgmap v12065: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:38 smithi121 bash[36902]: audit 2022-04-22T23:03:37.739599+0000 mon.a (mon.0) 15925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:38 smithi115 bash[45495]: audit 2022-04-22T23:03:37.739599+0000 mon.a (mon.0) 15925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:38 smithi115 bash[46934]: audit 2022-04-22T23:03:37.739599+0000 mon.a (mon.0) 15925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:39 smithi121 bash[36902]: cluster 2022-04-22T23:03:38.637789+0000 mgr.y (mgr.34107) 11503 : cluster [DBG] pgmap v12066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:39 smithi121 bash[36902]: audit 2022-04-22T23:03:39.319881+0000 mon.a (mon.0) 15926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:39 smithi115 bash[45495]: cluster 2022-04-22T23:03:38.637789+0000 mgr.y (mgr.34107) 11503 : cluster [DBG] pgmap v12066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:39 smithi115 bash[45495]: audit 2022-04-22T23:03:39.319881+0000 mon.a (mon.0) 15926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:39 smithi115 bash[46934]: cluster 2022-04-22T23:03:38.637789+0000 mgr.y (mgr.34107) 11503 : cluster [DBG] pgmap v12066: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:39 smithi115 bash[46934]: audit 2022-04-22T23:03:39.319881+0000 mon.a (mon.0) 15926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:41 smithi121 bash[36902]: cluster 2022-04-22T23:03:40.638867+0000 mgr.y (mgr.34107) 11504 : cluster [DBG] pgmap v12067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:41 smithi121 bash[36902]: audit 2022-04-22T23:03:41.115547+0000 mon.a (mon.0) 15927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:41 smithi115 bash[45495]: cluster 2022-04-22T23:03:40.638867+0000 mgr.y (mgr.34107) 11504 : cluster [DBG] pgmap v12067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:41 smithi115 bash[45495]: audit 2022-04-22T23:03:41.115547+0000 mon.a (mon.0) 15927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:41 smithi115 bash[46934]: cluster 2022-04-22T23:03:40.638867+0000 mgr.y (mgr.34107) 11504 : cluster [DBG] pgmap v12067: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:41 smithi115 bash[46934]: audit 2022-04-22T23:03:41.115547+0000 mon.a (mon.0) 15927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:43.275 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:42.948Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:43.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:43 smithi115 bash[45495]: cluster 2022-04-22T23:03:42.639958+0000 mgr.y (mgr.34107) 11505 : cluster [DBG] pgmap v12068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:43 smithi115 bash[45495]: audit 2022-04-22T23:03:42.742781+0000 mon.a (mon.0) 15928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:43 smithi115 bash[46934]: cluster 2022-04-22T23:03:42.639958+0000 mgr.y (mgr.34107) 11505 : cluster [DBG] pgmap v12068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:43 smithi115 bash[46934]: audit 2022-04-22T23:03:42.742781+0000 mon.a (mon.0) 15928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:43 smithi121 bash[36902]: cluster 2022-04-22T23:03:42.639958+0000 mgr.y (mgr.34107) 11505 : cluster [DBG] pgmap v12068: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:43 smithi121 bash[36902]: audit 2022-04-22T23:03:42.742781+0000 mon.a (mon.0) 15928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:44 smithi121 bash[36902]: audit 2022-04-22T23:03:44.323522+0000 mon.a (mon.0) 15929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:44 smithi115 bash[45495]: audit 2022-04-22T23:03:44.323522+0000 mon.a (mon.0) 15929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:44 smithi115 bash[46934]: audit 2022-04-22T23:03:44.323522+0000 mon.a (mon.0) 15929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:44.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:45 smithi121 bash[36902]: cluster 2022-04-22T23:03:44.641070+0000 mgr.y (mgr.34107) 11506 : cluster [DBG] pgmap v12069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:45.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:45 smithi115 bash[45495]: cluster 2022-04-22T23:03:44.641070+0000 mgr.y (mgr.34107) 11506 : cluster [DBG] pgmap v12069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:45 smithi115 bash[46934]: cluster 2022-04-22T23:03:44.641070+0000 mgr.y (mgr.34107) 11506 : cluster [DBG] pgmap v12069: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:46 smithi121 bash[36902]: audit 2022-04-22T23:03:46.118564+0000 mon.a (mon.0) 15930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:46 smithi115 bash[45495]: audit 2022-04-22T23:03:46.118564+0000 mon.a (mon.0) 15930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:46 smithi115 bash[46934]: audit 2022-04-22T23:03:46.118564+0000 mon.a (mon.0) 15930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:47 smithi121 bash[36902]: cluster 2022-04-22T23:03:46.642151+0000 mgr.y (mgr.34107) 11507 : cluster [DBG] pgmap v12070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:47 smithi115 bash[45495]: cluster 2022-04-22T23:03:46.642151+0000 mgr.y (mgr.34107) 11507 : cluster [DBG] pgmap v12070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:47 smithi115 bash[46934]: cluster 2022-04-22T23:03:46.642151+0000 mgr.y (mgr.34107) 11507 : cluster [DBG] pgmap v12070: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:48 smithi121 bash[36902]: audit 2022-04-22T23:03:47.746176+0000 mon.a (mon.0) 15931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:48.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:48 smithi115 bash[45495]: audit 2022-04-22T23:03:47.746176+0000 mon.a (mon.0) 15931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:48 smithi115 bash[46934]: audit 2022-04-22T23:03:47.746176+0000 mon.a (mon.0) 15931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:49 smithi121 bash[36902]: cluster 2022-04-22T23:03:48.643246+0000 mgr.y (mgr.34107) 11508 : cluster [DBG] pgmap v12071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:49 smithi121 bash[36902]: audit 2022-04-22T23:03:49.327153+0000 mon.a (mon.0) 15932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:49 smithi115 bash[46934]: cluster 2022-04-22T23:03:48.643246+0000 mgr.y (mgr.34107) 11508 : cluster [DBG] pgmap v12071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:49 smithi115 bash[46934]: audit 2022-04-22T23:03:49.327153+0000 mon.a (mon.0) 15932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:49 smithi115 bash[45495]: cluster 2022-04-22T23:03:48.643246+0000 mgr.y (mgr.34107) 11508 : cluster [DBG] pgmap v12071: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:49 smithi115 bash[45495]: audit 2022-04-22T23:03:49.327153+0000 mon.a (mon.0) 15932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:51.053 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:03:51.402 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:03:51.403 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:03:51.404 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:03:51.404 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:03:51.404 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:03:51.404 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:03:51.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:51.805 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:03:51.805 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:03:51.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:51.805 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:03:51.806 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:51 smithi121 bash[36902]: cluster 2022-04-22T23:03:50.644398+0000 mgr.y (mgr.34107) 11509 : cluster [DBG] pgmap v12072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:51 smithi121 bash[36902]: audit 2022-04-22T23:03:51.032364+0000 mgr.y (mgr.34107) 11510 : audit [DBG] from='client.47262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:52.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:51 smithi121 bash[36902]: audit 2022-04-22T23:03:51.121557+0000 mon.a (mon.0) 15933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[45495]: cluster 2022-04-22T23:03:50.644398+0000 mgr.y (mgr.34107) 11509 : cluster [DBG] pgmap v12072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:52.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[45495]: audit 2022-04-22T23:03:51.032364+0000 mgr.y (mgr.34107) 11510 : audit [DBG] from='client.47262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:52.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[45495]: audit 2022-04-22T23:03:51.121557+0000 mon.a (mon.0) 15933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:52.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[46934]: cluster 2022-04-22T23:03:50.644398+0000 mgr.y (mgr.34107) 11509 : cluster [DBG] pgmap v12072: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[46934]: audit 2022-04-22T23:03:51.032364+0000 mgr.y (mgr.34107) 11510 : audit [DBG] from='client.47262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:52.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:51 smithi115 bash[46934]: audit 2022-04-22T23:03:51.121557+0000 mon.a (mon.0) 15933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:53.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:52 smithi121 bash[36902]: audit 2022-04-22T23:03:51.375330+0000 mgr.y (mgr.34107) 11511 : audit [DBG] from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:52 smithi121 bash[36902]: audit 2022-04-22T23:03:51.800269+0000 mon.a (mon.0) 15934 : audit [DBG] from='client.? 172.21.15.115:0/2198908213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:53.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:52.949Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:52 smithi115 bash[45495]: audit 2022-04-22T23:03:51.375330+0000 mgr.y (mgr.34107) 11511 : audit [DBG] from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:52 smithi115 bash[45495]: audit 2022-04-22T23:03:51.800269+0000 mon.a (mon.0) 15934 : audit [DBG] from='client.? 172.21.15.115:0/2198908213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:52 smithi115 bash[46934]: audit 2022-04-22T23:03:51.375330+0000 mgr.y (mgr.34107) 11511 : audit [DBG] from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:03:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:52 smithi115 bash[46934]: audit 2022-04-22T23:03:51.800269+0000 mon.a (mon.0) 15934 : audit [DBG] from='client.? 172.21.15.115:0/2198908213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:03:54.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:53 smithi121 bash[36902]: cluster 2022-04-22T23:03:52.645508+0000 mgr.y (mgr.34107) 11512 : cluster [DBG] pgmap v12073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:53 smithi121 bash[36902]: audit 2022-04-22T23:03:52.749169+0000 mon.a (mon.0) 15935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:53 smithi115 bash[46934]: cluster 2022-04-22T23:03:52.645508+0000 mgr.y (mgr.34107) 11512 : cluster [DBG] pgmap v12073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:53 smithi115 bash[46934]: audit 2022-04-22T23:03:52.749169+0000 mon.a (mon.0) 15935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:54.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:53 smithi115 bash[45495]: cluster 2022-04-22T23:03:52.645508+0000 mgr.y (mgr.34107) 11512 : cluster [DBG] pgmap v12073: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:54.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:53 smithi115 bash[45495]: audit 2022-04-22T23:03:52.749169+0000 mon.a (mon.0) 15935 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:54 smithi121 bash[36902]: audit 2022-04-22T23:03:54.330918+0000 mon.a (mon.0) 15936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:55.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:03:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:03:54.802Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:03:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:54 smithi115 bash[45495]: audit 2022-04-22T23:03:54.330918+0000 mon.a (mon.0) 15936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:54 smithi115 bash[46934]: audit 2022-04-22T23:03:54.330918+0000 mon.a (mon.0) 15936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:03:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:55 smithi121 bash[36902]: cluster 2022-04-22T23:03:54.646600+0000 mgr.y (mgr.34107) 11513 : cluster [DBG] pgmap v12074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:56.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:55 smithi115 bash[46934]: cluster 2022-04-22T23:03:54.646600+0000 mgr.y (mgr.34107) 11513 : cluster [DBG] pgmap v12074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:55 smithi115 bash[45495]: cluster 2022-04-22T23:03:54.646600+0000 mgr.y (mgr.34107) 11513 : cluster [DBG] pgmap v12074: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:56 smithi121 bash[36902]: audit 2022-04-22T23:03:56.124660+0000 mon.a (mon.0) 15937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:56 smithi115 bash[45495]: audit 2022-04-22T23:03:56.124660+0000 mon.a (mon.0) 15937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:56 smithi115 bash[46934]: audit 2022-04-22T23:03:56.124660+0000 mon.a (mon.0) 15937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:03:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:57 smithi121 bash[36902]: cluster 2022-04-22T23:03:56.647697+0000 mgr.y (mgr.34107) 11514 : cluster [DBG] pgmap v12075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:57 smithi115 bash[45495]: cluster 2022-04-22T23:03:56.647697+0000 mgr.y (mgr.34107) 11514 : cluster [DBG] pgmap v12075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:57 smithi115 bash[46934]: cluster 2022-04-22T23:03:56.647697+0000 mgr.y (mgr.34107) 11514 : cluster [DBG] pgmap v12075: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:03:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:58 smithi121 bash[36902]: audit 2022-04-22T23:03:57.752123+0000 mon.a (mon.0) 15938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:58 smithi115 bash[45495]: audit 2022-04-22T23:03:57.752123+0000 mon.a (mon.0) 15938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:03:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:58 smithi115 bash[46934]: audit 2022-04-22T23:03:57.752123+0000 mon.a (mon.0) 15938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:59 smithi121 bash[36902]: cluster 2022-04-22T23:03:58.648847+0000 mgr.y (mgr.34107) 11515 : cluster [DBG] pgmap v12076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:03:59 smithi121 bash[36902]: audit 2022-04-22T23:03:59.334852+0000 mon.a (mon.0) 15939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:00.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:59 smithi115 bash[45495]: cluster 2022-04-22T23:03:58.648847+0000 mgr.y (mgr.34107) 11515 : cluster [DBG] pgmap v12076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:03:59 smithi115 bash[45495]: audit 2022-04-22T23:03:59.334852+0000 mon.a (mon.0) 15939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:59 smithi115 bash[46934]: cluster 2022-04-22T23:03:58.648847+0000 mgr.y (mgr.34107) 11515 : cluster [DBG] pgmap v12076: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:03:59 smithi115 bash[46934]: audit 2022-04-22T23:03:59.334852+0000 mon.a (mon.0) 15939 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:01 smithi121 bash[36902]: cluster 2022-04-22T23:04:00.649944+0000 mgr.y (mgr.34107) 11516 : cluster [DBG] pgmap v12077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:01 smithi121 bash[36902]: audit 2022-04-22T23:04:01.127870+0000 mon.a (mon.0) 15940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:01 smithi115 bash[45495]: cluster 2022-04-22T23:04:00.649944+0000 mgr.y (mgr.34107) 11516 : cluster [DBG] pgmap v12077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:01 smithi115 bash[45495]: audit 2022-04-22T23:04:01.127870+0000 mon.a (mon.0) 15940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:01 smithi115 bash[46934]: cluster 2022-04-22T23:04:00.649944+0000 mgr.y (mgr.34107) 11516 : cluster [DBG] pgmap v12077: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:01 smithi115 bash[46934]: audit 2022-04-22T23:04:01.127870+0000 mon.a (mon.0) 15940 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:03.279 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:02.949Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:03.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:03 smithi115 bash[46934]: cluster 2022-04-22T23:04:02.650972+0000 mgr.y (mgr.34107) 11517 : cluster [DBG] pgmap v12078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:03.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:03 smithi115 bash[46934]: audit 2022-04-22T23:04:02.755334+0000 mon.a (mon.0) 15941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:03.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:03 smithi115 bash[45495]: cluster 2022-04-22T23:04:02.650972+0000 mgr.y (mgr.34107) 11517 : cluster [DBG] pgmap v12078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:03 smithi115 bash[45495]: audit 2022-04-22T23:04:02.755334+0000 mon.a (mon.0) 15941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:03 smithi121 bash[36902]: cluster 2022-04-22T23:04:02.650972+0000 mgr.y (mgr.34107) 11517 : cluster [DBG] pgmap v12078: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:03 smithi121 bash[36902]: audit 2022-04-22T23:04:02.755334+0000 mon.a (mon.0) 15941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:04 smithi121 bash[36902]: audit 2022-04-22T23:04:04.338783+0000 mon.a (mon.0) 15942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:04 smithi115 bash[45495]: audit 2022-04-22T23:04:04.338783+0000 mon.a (mon.0) 15942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:04 smithi115 bash[46934]: audit 2022-04-22T23:04:04.338783+0000 mon.a (mon.0) 15942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:04.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:05 smithi121 bash[36902]: cluster 2022-04-22T23:04:04.652121+0000 mgr.y (mgr.34107) 11518 : cluster [DBG] pgmap v12079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:05 smithi115 bash[45495]: cluster 2022-04-22T23:04:04.652121+0000 mgr.y (mgr.34107) 11518 : cluster [DBG] pgmap v12079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:05 smithi115 bash[46934]: cluster 2022-04-22T23:04:04.652121+0000 mgr.y (mgr.34107) 11518 : cluster [DBG] pgmap v12079: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:06 smithi121 bash[36902]: audit 2022-04-22T23:04:06.131064+0000 mon.a (mon.0) 15943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:06 smithi115 bash[45495]: audit 2022-04-22T23:04:06.131064+0000 mon.a (mon.0) 15943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:06 smithi115 bash[46934]: audit 2022-04-22T23:04:06.131064+0000 mon.a (mon.0) 15943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:07 smithi121 bash[36902]: cluster 2022-04-22T23:04:06.653147+0000 mgr.y (mgr.34107) 11519 : cluster [DBG] pgmap v12080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:07 smithi115 bash[45495]: cluster 2022-04-22T23:04:06.653147+0000 mgr.y (mgr.34107) 11519 : cluster [DBG] pgmap v12080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:07 smithi115 bash[46934]: cluster 2022-04-22T23:04:06.653147+0000 mgr.y (mgr.34107) 11519 : cluster [DBG] pgmap v12080: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:08 smithi121 bash[36902]: audit 2022-04-22T23:04:07.758582+0000 mon.a (mon.0) 15944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:08 smithi115 bash[45495]: audit 2022-04-22T23:04:07.758582+0000 mon.a (mon.0) 15944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:08.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:08 smithi115 bash[46934]: audit 2022-04-22T23:04:07.758582+0000 mon.a (mon.0) 15944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:09 smithi121 bash[36902]: cluster 2022-04-22T23:04:08.654355+0000 mgr.y (mgr.34107) 11520 : cluster [DBG] pgmap v12081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:09 smithi121 bash[36902]: audit 2022-04-22T23:04:09.342456+0000 mon.a (mon.0) 15945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:09 smithi115 bash[45495]: cluster 2022-04-22T23:04:08.654355+0000 mgr.y (mgr.34107) 11520 : cluster [DBG] pgmap v12081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:09 smithi115 bash[45495]: audit 2022-04-22T23:04:09.342456+0000 mon.a (mon.0) 15945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:09 smithi115 bash[46934]: cluster 2022-04-22T23:04:08.654355+0000 mgr.y (mgr.34107) 11520 : cluster [DBG] pgmap v12081: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:09 smithi115 bash[46934]: audit 2022-04-22T23:04:09.342456+0000 mon.a (mon.0) 15945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:11 smithi121 bash[36902]: cluster 2022-04-22T23:04:10.655399+0000 mgr.y (mgr.34107) 11521 : cluster [DBG] pgmap v12082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:11 smithi121 bash[36902]: audit 2022-04-22T23:04:11.134151+0000 mon.a (mon.0) 15946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:11 smithi115 bash[46934]: cluster 2022-04-22T23:04:10.655399+0000 mgr.y (mgr.34107) 11521 : cluster [DBG] pgmap v12082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:11 smithi115 bash[46934]: audit 2022-04-22T23:04:11.134151+0000 mon.a (mon.0) 15946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:11 smithi115 bash[45495]: cluster 2022-04-22T23:04:10.655399+0000 mgr.y (mgr.34107) 11521 : cluster [DBG] pgmap v12082: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:11 smithi115 bash[45495]: audit 2022-04-22T23:04:11.134151+0000 mon.a (mon.0) 15946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:13.282 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:12.950Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:13 smithi115 bash[45495]: cluster 2022-04-22T23:04:12.656476+0000 mgr.y (mgr.34107) 11522 : cluster [DBG] pgmap v12083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:13 smithi115 bash[45495]: audit 2022-04-22T23:04:12.761743+0000 mon.a (mon.0) 15947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:13 smithi115 bash[46934]: cluster 2022-04-22T23:04:12.656476+0000 mgr.y (mgr.34107) 11522 : cluster [DBG] pgmap v12083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:13 smithi115 bash[46934]: audit 2022-04-22T23:04:12.761743+0000 mon.a (mon.0) 15947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:13 smithi121 bash[36902]: cluster 2022-04-22T23:04:12.656476+0000 mgr.y (mgr.34107) 11522 : cluster [DBG] pgmap v12083: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:13 smithi121 bash[36902]: audit 2022-04-22T23:04:12.761743+0000 mon.a (mon.0) 15947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:14 smithi121 bash[36902]: audit 2022-04-22T23:04:14.346556+0000 mon.a (mon.0) 15948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:14 smithi115 bash[45495]: audit 2022-04-22T23:04:14.346556+0000 mon.a (mon.0) 15948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:14 smithi115 bash[46934]: audit 2022-04-22T23:04:14.346556+0000 mon.a (mon.0) 15948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:15.262 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:14.803Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:15 smithi121 bash[36902]: cluster 2022-04-22T23:04:14.657653+0000 mgr.y (mgr.34107) 11523 : cluster [DBG] pgmap v12084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:15 smithi121 bash[36902]: audit 2022-04-22T23:04:15.232306+0000 mon.a (mon.0) 15949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:04:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:15 smithi121 bash[36902]: audit 2022-04-22T23:04:15.235178+0000 mon.a (mon.0) 15950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:04:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:15 smithi121 bash[36902]: audit 2022-04-22T23:04:15.237037+0000 mon.a (mon.0) 15951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:04:15.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[45495]: cluster 2022-04-22T23:04:14.657653+0000 mgr.y (mgr.34107) 11523 : cluster [DBG] pgmap v12084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[45495]: audit 2022-04-22T23:04:15.232306+0000 mon.a (mon.0) 15949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[45495]: audit 2022-04-22T23:04:15.235178+0000 mon.a (mon.0) 15950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[45495]: audit 2022-04-22T23:04:15.237037+0000 mon.a (mon.0) 15951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[46934]: cluster 2022-04-22T23:04:14.657653+0000 mgr.y (mgr.34107) 11523 : cluster [DBG] pgmap v12084: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[46934]: audit 2022-04-22T23:04:15.232306+0000 mon.a (mon.0) 15949 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:04:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[46934]: audit 2022-04-22T23:04:15.235178+0000 mon.a (mon.0) 15950 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:04:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:15 smithi115 bash[46934]: audit 2022-04-22T23:04:15.237037+0000 mon.a (mon.0) 15951 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:04:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:16 smithi121 bash[36902]: audit 2022-04-22T23:04:15.612484+0000 mon.a (mon.0) 15952 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:04:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:16 smithi121 bash[36902]: audit 2022-04-22T23:04:16.137402+0000 mon.a (mon.0) 15953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:16 smithi115 bash[45495]: audit 2022-04-22T23:04:15.612484+0000 mon.a (mon.0) 15952 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:04:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:16 smithi115 bash[45495]: audit 2022-04-22T23:04:16.137402+0000 mon.a (mon.0) 15953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:17.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:16 smithi115 bash[46934]: audit 2022-04-22T23:04:15.612484+0000 mon.a (mon.0) 15952 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:04:17.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:16 smithi115 bash[46934]: audit 2022-04-22T23:04:16.137402+0000 mon.a (mon.0) 15953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:17 smithi121 bash[36902]: cluster 2022-04-22T23:04:16.658742+0000 mgr.y (mgr.34107) 11524 : cluster [DBG] pgmap v12085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:17 smithi121 bash[36902]: audit 2022-04-22T23:04:17.163258+0000 mon.a (mon.0) 15954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:04:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:17 smithi121 bash[36902]: audit 2022-04-22T23:04:17.163796+0000 mon.a (mon.0) 15955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:04:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[45495]: cluster 2022-04-22T23:04:16.658742+0000 mgr.y (mgr.34107) 11524 : cluster [DBG] pgmap v12085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[45495]: audit 2022-04-22T23:04:17.163258+0000 mon.a (mon.0) 15954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:04:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[45495]: audit 2022-04-22T23:04:17.163796+0000 mon.a (mon.0) 15955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:04:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[46934]: cluster 2022-04-22T23:04:16.658742+0000 mgr.y (mgr.34107) 11524 : cluster [DBG] pgmap v12085: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[46934]: audit 2022-04-22T23:04:17.163258+0000 mon.a (mon.0) 15954 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:04:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:17 smithi115 bash[46934]: audit 2022-04-22T23:04:17.163796+0000 mon.a (mon.0) 15955 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:04:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:18 smithi121 bash[36902]: audit 2022-04-22T23:04:17.764833+0000 mon.a (mon.0) 15956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:18 smithi115 bash[45495]: audit 2022-04-22T23:04:17.764833+0000 mon.a (mon.0) 15956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:18 smithi115 bash[46934]: audit 2022-04-22T23:04:17.764833+0000 mon.a (mon.0) 15956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:19 smithi121 bash[36902]: cluster 2022-04-22T23:04:18.659955+0000 mgr.y (mgr.34107) 11525 : cluster [DBG] pgmap v12086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:19 smithi121 bash[36902]: audit 2022-04-22T23:04:19.350125+0000 mon.a (mon.0) 15957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:19 smithi115 bash[46934]: cluster 2022-04-22T23:04:18.659955+0000 mgr.y (mgr.34107) 11525 : cluster [DBG] pgmap v12086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:19 smithi115 bash[46934]: audit 2022-04-22T23:04:19.350125+0000 mon.a (mon.0) 15957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:19 smithi115 bash[45495]: cluster 2022-04-22T23:04:18.659955+0000 mgr.y (mgr.34107) 11525 : cluster [DBG] pgmap v12086: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:19 smithi115 bash[45495]: audit 2022-04-22T23:04:19.350125+0000 mon.a (mon.0) 15957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:21 smithi121 bash[36902]: cluster 2022-04-22T23:04:20.661005+0000 mgr.y (mgr.34107) 11526 : cluster [DBG] pgmap v12087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:21 smithi121 bash[36902]: audit 2022-04-22T23:04:21.140277+0000 mon.a (mon.0) 15958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:21 smithi115 bash[46934]: cluster 2022-04-22T23:04:20.661005+0000 mgr.y (mgr.34107) 11526 : cluster [DBG] pgmap v12087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:21 smithi115 bash[46934]: audit 2022-04-22T23:04:21.140277+0000 mon.a (mon.0) 15958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:21 smithi115 bash[45495]: cluster 2022-04-22T23:04:20.661005+0000 mgr.y (mgr.34107) 11526 : cluster [DBG] pgmap v12087: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:21 smithi115 bash[45495]: audit 2022-04-22T23:04:21.140277+0000 mon.a (mon.0) 15958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:22.188 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:04:22.537 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:04:22.538 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:04:22.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:04:22.539 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:04:22.539 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:04:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:04:22.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:04:22.942 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:04:22.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:22.942 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:04:22.943 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:04:22.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:22.944 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:04:22.944 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:04:22.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:22.945 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:04:22.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:04:22.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:22.945 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:04:22.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:04:22.946 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:04:22.946 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:04:23.286 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:22.948Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:23.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[46934]: audit 2022-04-22T23:04:22.165916+0000 mgr.y (mgr.34107) 11527 : audit [DBG] from='client.47277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[46934]: audit 2022-04-22T23:04:22.509512+0000 mgr.y (mgr.34107) 11528 : audit [DBG] from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[46934]: cluster 2022-04-22T23:04:22.661841+0000 mgr.y (mgr.34107) 11529 : cluster [DBG] pgmap v12088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:23.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[46934]: audit 2022-04-22T23:04:22.768020+0000 mon.a (mon.0) 15959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:23.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[46934]: audit 2022-04-22T23:04:22.937872+0000 mon.a (mon.0) 15960 : audit [DBG] from='client.? 172.21.15.115:0/977091501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:23.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[45495]: audit 2022-04-22T23:04:22.165916+0000 mgr.y (mgr.34107) 11527 : audit [DBG] from='client.47277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.289 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[45495]: audit 2022-04-22T23:04:22.509512+0000 mgr.y (mgr.34107) 11528 : audit [DBG] from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[45495]: cluster 2022-04-22T23:04:22.661841+0000 mgr.y (mgr.34107) 11529 : cluster [DBG] pgmap v12088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[45495]: audit 2022-04-22T23:04:22.768020+0000 mon.a (mon.0) 15959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:23.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:23 smithi115 bash[45495]: audit 2022-04-22T23:04:22.937872+0000 mon.a (mon.0) 15960 : audit [DBG] from='client.? 172.21.15.115:0/977091501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:23 smithi121 bash[36902]: audit 2022-04-22T23:04:22.165916+0000 mgr.y (mgr.34107) 11527 : audit [DBG] from='client.47277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:23 smithi121 bash[36902]: audit 2022-04-22T23:04:22.509512+0000 mgr.y (mgr.34107) 11528 : audit [DBG] from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:23 smithi121 bash[36902]: cluster 2022-04-22T23:04:22.661841+0000 mgr.y (mgr.34107) 11529 : cluster [DBG] pgmap v12088: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:23 smithi121 bash[36902]: audit 2022-04-22T23:04:22.768020+0000 mon.a (mon.0) 15959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:23 smithi121 bash[36902]: audit 2022-04-22T23:04:22.937872+0000 mon.a (mon.0) 15960 : audit [DBG] from='client.? 172.21.15.115:0/977091501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:24 smithi121 bash[36902]: audit 2022-04-22T23:04:24.353883+0000 mon.a (mon.0) 15961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:24 smithi115 bash[45495]: audit 2022-04-22T23:04:24.353883+0000 mon.a (mon.0) 15961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:24 smithi115 bash[46934]: audit 2022-04-22T23:04:24.353883+0000 mon.a (mon.0) 15961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:24.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:25 smithi121 bash[36902]: cluster 2022-04-22T23:04:24.662903+0000 mgr.y (mgr.34107) 11530 : cluster [DBG] pgmap v12089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:25.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:25 smithi115 bash[45495]: cluster 2022-04-22T23:04:24.662903+0000 mgr.y (mgr.34107) 11530 : cluster [DBG] pgmap v12089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:25 smithi115 bash[46934]: cluster 2022-04-22T23:04:24.662903+0000 mgr.y (mgr.34107) 11530 : cluster [DBG] pgmap v12089: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:26 smithi121 bash[36902]: audit 2022-04-22T23:04:26.143251+0000 mon.a (mon.0) 15962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:26 smithi115 bash[45495]: audit 2022-04-22T23:04:26.143251+0000 mon.a (mon.0) 15962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:26 smithi115 bash[46934]: audit 2022-04-22T23:04:26.143251+0000 mon.a (mon.0) 15962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:27 smithi121 bash[36902]: cluster 2022-04-22T23:04:26.664047+0000 mgr.y (mgr.34107) 11531 : cluster [DBG] pgmap v12090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:27 smithi115 bash[45495]: cluster 2022-04-22T23:04:26.664047+0000 mgr.y (mgr.34107) 11531 : cluster [DBG] pgmap v12090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:27.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:27 smithi115 bash[46934]: cluster 2022-04-22T23:04:26.664047+0000 mgr.y (mgr.34107) 11531 : cluster [DBG] pgmap v12090: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:28 smithi121 bash[36902]: audit 2022-04-22T23:04:27.771146+0000 mon.a (mon.0) 15963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:28 smithi115 bash[45495]: audit 2022-04-22T23:04:27.771146+0000 mon.a (mon.0) 15963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:28 smithi115 bash[46934]: audit 2022-04-22T23:04:27.771146+0000 mon.a (mon.0) 15963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:29 smithi121 bash[36902]: cluster 2022-04-22T23:04:28.665233+0000 mgr.y (mgr.34107) 11532 : cluster [DBG] pgmap v12091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:29 smithi121 bash[36902]: audit 2022-04-22T23:04:29.357613+0000 mon.a (mon.0) 15964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:29 smithi115 bash[45495]: cluster 2022-04-22T23:04:28.665233+0000 mgr.y (mgr.34107) 11532 : cluster [DBG] pgmap v12091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:29 smithi115 bash[45495]: audit 2022-04-22T23:04:29.357613+0000 mon.a (mon.0) 15964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:29 smithi115 bash[46934]: cluster 2022-04-22T23:04:28.665233+0000 mgr.y (mgr.34107) 11532 : cluster [DBG] pgmap v12091: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:29 smithi115 bash[46934]: audit 2022-04-22T23:04:29.357613+0000 mon.a (mon.0) 15964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:31 smithi121 bash[36902]: cluster 2022-04-22T23:04:30.666321+0000 mgr.y (mgr.34107) 11533 : cluster [DBG] pgmap v12092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:31 smithi121 bash[36902]: audit 2022-04-22T23:04:31.146247+0000 mon.a (mon.0) 15965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:31 smithi115 bash[45495]: cluster 2022-04-22T23:04:30.666321+0000 mgr.y (mgr.34107) 11533 : cluster [DBG] pgmap v12092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:31 smithi115 bash[45495]: audit 2022-04-22T23:04:31.146247+0000 mon.a (mon.0) 15965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:31 smithi115 bash[46934]: cluster 2022-04-22T23:04:30.666321+0000 mgr.y (mgr.34107) 11533 : cluster [DBG] pgmap v12092: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:31 smithi115 bash[46934]: audit 2022-04-22T23:04:31.146247+0000 mon.a (mon.0) 15965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:33.288 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:32.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:33.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:33 smithi115 bash[46934]: cluster 2022-04-22T23:04:32.667462+0000 mgr.y (mgr.34107) 11534 : cluster [DBG] pgmap v12093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:33.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:33 smithi115 bash[46934]: audit 2022-04-22T23:04:32.774278+0000 mon.a (mon.0) 15966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:33.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:33 smithi115 bash[45495]: cluster 2022-04-22T23:04:32.667462+0000 mgr.y (mgr.34107) 11534 : cluster [DBG] pgmap v12093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:33 smithi115 bash[45495]: audit 2022-04-22T23:04:32.774278+0000 mon.a (mon.0) 15966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:33 smithi121 bash[36902]: cluster 2022-04-22T23:04:32.667462+0000 mgr.y (mgr.34107) 11534 : cluster [DBG] pgmap v12093: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:33 smithi121 bash[36902]: audit 2022-04-22T23:04:32.774278+0000 mon.a (mon.0) 15966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:34 smithi121 bash[36902]: audit 2022-04-22T23:04:34.361604+0000 mon.a (mon.0) 15967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:34 smithi115 bash[45495]: audit 2022-04-22T23:04:34.361604+0000 mon.a (mon.0) 15967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:34 smithi115 bash[46934]: audit 2022-04-22T23:04:34.361604+0000 mon.a (mon.0) 15967 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:34.804Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:35 smithi121 bash[36902]: cluster 2022-04-22T23:04:34.668577+0000 mgr.y (mgr.34107) 11535 : cluster [DBG] pgmap v12094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:35 smithi115 bash[45495]: cluster 2022-04-22T23:04:34.668577+0000 mgr.y (mgr.34107) 11535 : cluster [DBG] pgmap v12094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:35 smithi115 bash[46934]: cluster 2022-04-22T23:04:34.668577+0000 mgr.y (mgr.34107) 11535 : cluster [DBG] pgmap v12094: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:36 smithi121 bash[36902]: audit 2022-04-22T23:04:36.149225+0000 mon.a (mon.0) 15968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:36 smithi115 bash[45495]: audit 2022-04-22T23:04:36.149225+0000 mon.a (mon.0) 15968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:36 smithi115 bash[46934]: audit 2022-04-22T23:04:36.149225+0000 mon.a (mon.0) 15968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:37 smithi121 bash[36902]: cluster 2022-04-22T23:04:36.669696+0000 mgr.y (mgr.34107) 11536 : cluster [DBG] pgmap v12095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:37 smithi115 bash[45495]: cluster 2022-04-22T23:04:36.669696+0000 mgr.y (mgr.34107) 11536 : cluster [DBG] pgmap v12095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:37 smithi115 bash[46934]: cluster 2022-04-22T23:04:36.669696+0000 mgr.y (mgr.34107) 11536 : cluster [DBG] pgmap v12095: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:38 smithi121 bash[36902]: audit 2022-04-22T23:04:37.777481+0000 mon.a (mon.0) 15969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:38 smithi115 bash[45495]: audit 2022-04-22T23:04:37.777481+0000 mon.a (mon.0) 15969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:38 smithi115 bash[46934]: audit 2022-04-22T23:04:37.777481+0000 mon.a (mon.0) 15969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:39 smithi121 bash[36902]: cluster 2022-04-22T23:04:38.670867+0000 mgr.y (mgr.34107) 11537 : cluster [DBG] pgmap v12096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:39 smithi121 bash[36902]: audit 2022-04-22T23:04:39.365280+0000 mon.a (mon.0) 15970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:39 smithi115 bash[45495]: cluster 2022-04-22T23:04:38.670867+0000 mgr.y (mgr.34107) 11537 : cluster [DBG] pgmap v12096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:39 smithi115 bash[45495]: audit 2022-04-22T23:04:39.365280+0000 mon.a (mon.0) 15970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:39 smithi115 bash[46934]: cluster 2022-04-22T23:04:38.670867+0000 mgr.y (mgr.34107) 11537 : cluster [DBG] pgmap v12096: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:39 smithi115 bash[46934]: audit 2022-04-22T23:04:39.365280+0000 mon.a (mon.0) 15970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:41 smithi121 bash[36902]: cluster 2022-04-22T23:04:40.671917+0000 mgr.y (mgr.34107) 11538 : cluster [DBG] pgmap v12097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:42.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:41 smithi121 bash[36902]: audit 2022-04-22T23:04:41.152296+0000 mon.a (mon.0) 15971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:42.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:41 smithi115 bash[45495]: cluster 2022-04-22T23:04:40.671917+0000 mgr.y (mgr.34107) 11538 : cluster [DBG] pgmap v12097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:41 smithi115 bash[45495]: audit 2022-04-22T23:04:41.152296+0000 mon.a (mon.0) 15971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:41 smithi115 bash[46934]: cluster 2022-04-22T23:04:40.671917+0000 mgr.y (mgr.34107) 11538 : cluster [DBG] pgmap v12097: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:41 smithi115 bash[46934]: audit 2022-04-22T23:04:41.152296+0000 mon.a (mon.0) 15971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:42.951Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:43 smithi121 bash[36902]: cluster 2022-04-22T23:04:42.672928+0000 mgr.y (mgr.34107) 11539 : cluster [DBG] pgmap v12098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:43 smithi121 bash[36902]: audit 2022-04-22T23:04:42.780831+0000 mon.a (mon.0) 15972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:43 smithi115 bash[45495]: cluster 2022-04-22T23:04:42.672928+0000 mgr.y (mgr.34107) 11539 : cluster [DBG] pgmap v12098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:43 smithi115 bash[45495]: audit 2022-04-22T23:04:42.780831+0000 mon.a (mon.0) 15972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:43 smithi115 bash[46934]: cluster 2022-04-22T23:04:42.672928+0000 mgr.y (mgr.34107) 11539 : cluster [DBG] pgmap v12098: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:43 smithi115 bash[46934]: audit 2022-04-22T23:04:42.780831+0000 mon.a (mon.0) 15972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:44 smithi121 bash[36902]: audit 2022-04-22T23:04:44.369068+0000 mon.a (mon.0) 15973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:44 smithi115 bash[45495]: audit 2022-04-22T23:04:44.369068+0000 mon.a (mon.0) 15973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:44 smithi115 bash[46934]: audit 2022-04-22T23:04:44.369068+0000 mon.a (mon.0) 15973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:44.805Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:45 smithi121 bash[36902]: cluster 2022-04-22T23:04:44.673974+0000 mgr.y (mgr.34107) 11540 : cluster [DBG] pgmap v12099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:45.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:45 smithi115 bash[45495]: cluster 2022-04-22T23:04:44.673974+0000 mgr.y (mgr.34107) 11540 : cluster [DBG] pgmap v12099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:45.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:45 smithi115 bash[46934]: cluster 2022-04-22T23:04:44.673974+0000 mgr.y (mgr.34107) 11540 : cluster [DBG] pgmap v12099: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:46 smithi121 bash[36902]: audit 2022-04-22T23:04:46.155313+0000 mon.a (mon.0) 15974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:46 smithi115 bash[45495]: audit 2022-04-22T23:04:46.155313+0000 mon.a (mon.0) 15974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:46 smithi115 bash[46934]: audit 2022-04-22T23:04:46.155313+0000 mon.a (mon.0) 15974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:47 smithi121 bash[36902]: cluster 2022-04-22T23:04:46.675027+0000 mgr.y (mgr.34107) 11541 : cluster [DBG] pgmap v12100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:47 smithi115 bash[45495]: cluster 2022-04-22T23:04:46.675027+0000 mgr.y (mgr.34107) 11541 : cluster [DBG] pgmap v12100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:47 smithi115 bash[46934]: cluster 2022-04-22T23:04:46.675027+0000 mgr.y (mgr.34107) 11541 : cluster [DBG] pgmap v12100: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:48 smithi121 bash[36902]: audit 2022-04-22T23:04:47.783998+0000 mon.a (mon.0) 15975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:48 smithi115 bash[45495]: audit 2022-04-22T23:04:47.783998+0000 mon.a (mon.0) 15975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:48 smithi115 bash[46934]: audit 2022-04-22T23:04:47.783998+0000 mon.a (mon.0) 15975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:49 smithi121 bash[36902]: cluster 2022-04-22T23:04:48.676300+0000 mgr.y (mgr.34107) 11542 : cluster [DBG] pgmap v12101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:49 smithi121 bash[36902]: audit 2022-04-22T23:04:49.372830+0000 mon.a (mon.0) 15976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:49 smithi115 bash[45495]: cluster 2022-04-22T23:04:48.676300+0000 mgr.y (mgr.34107) 11542 : cluster [DBG] pgmap v12101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:49 smithi115 bash[45495]: audit 2022-04-22T23:04:49.372830+0000 mon.a (mon.0) 15976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:49 smithi115 bash[46934]: cluster 2022-04-22T23:04:48.676300+0000 mgr.y (mgr.34107) 11542 : cluster [DBG] pgmap v12101: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:49 smithi115 bash[46934]: audit 2022-04-22T23:04:49.372830+0000 mon.a (mon.0) 15976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:51 smithi121 bash[36902]: cluster 2022-04-22T23:04:50.677412+0000 mgr.y (mgr.34107) 11543 : cluster [DBG] pgmap v12102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:52.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:51 smithi121 bash[36902]: audit 2022-04-22T23:04:51.158705+0000 mon.a (mon.0) 15977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:51 smithi115 bash[45495]: cluster 2022-04-22T23:04:50.677412+0000 mgr.y (mgr.34107) 11543 : cluster [DBG] pgmap v12102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:51 smithi115 bash[45495]: audit 2022-04-22T23:04:51.158705+0000 mon.a (mon.0) 15977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:51 smithi115 bash[46934]: cluster 2022-04-22T23:04:50.677412+0000 mgr.y (mgr.34107) 11543 : cluster [DBG] pgmap v12102: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:51 smithi115 bash[46934]: audit 2022-04-22T23:04:51.158705+0000 mon.a (mon.0) 15977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:52.950Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:53.315 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:04:53.656 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:04:53.656 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:04:53.656 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:04:53.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:04:53.658 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:04:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:53 smithi121 bash[36902]: cluster 2022-04-22T23:04:52.678588+0000 mgr.y (mgr.34107) 11544 : cluster [DBG] pgmap v12103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:53 smithi121 bash[36902]: audit 2022-04-22T23:04:52.787375+0000 mon.a (mon.0) 15978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:53 smithi115 bash[45495]: cluster 2022-04-22T23:04:52.678588+0000 mgr.y (mgr.34107) 11544 : cluster [DBG] pgmap v12103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:53 smithi115 bash[45495]: audit 2022-04-22T23:04:52.787375+0000 mon.a (mon.0) 15978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:53 smithi115 bash[46934]: cluster 2022-04-22T23:04:52.678588+0000 mgr.y (mgr.34107) 11544 : cluster [DBG] pgmap v12103: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:53 smithi115 bash[46934]: audit 2022-04-22T23:04:52.787375+0000 mon.a (mon.0) 15978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:54.064 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:04:54.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:54.066 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:04:54.066 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:04:54.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:54.066 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:04:54.067 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:04:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:54 smithi121 bash[36902]: audit 2022-04-22T23:04:53.297640+0000 mgr.y (mgr.34107) 11545 : audit [DBG] from='client.57235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:54 smithi121 bash[36902]: audit 2022-04-22T23:04:53.633914+0000 mgr.y (mgr.34107) 11546 : audit [DBG] from='client.57241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:54 smithi121 bash[36902]: audit 2022-04-22T23:04:54.060360+0000 mon.a (mon.0) 15979 : audit [DBG] from='client.? 172.21.15.115:0/2321688337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[45495]: audit 2022-04-22T23:04:53.297640+0000 mgr.y (mgr.34107) 11545 : audit [DBG] from='client.57235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[45495]: audit 2022-04-22T23:04:53.633914+0000 mgr.y (mgr.34107) 11546 : audit [DBG] from='client.57241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[45495]: audit 2022-04-22T23:04:54.060360+0000 mon.a (mon.0) 15979 : audit [DBG] from='client.? 172.21.15.115:0/2321688337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[46934]: audit 2022-04-22T23:04:53.297640+0000 mgr.y (mgr.34107) 11545 : audit [DBG] from='client.57235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[46934]: audit 2022-04-22T23:04:53.633914+0000 mgr.y (mgr.34107) 11546 : audit [DBG] from='client.57241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:04:54.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[46934]: audit 2022-04-22T23:04:54.060360+0000 mon.a (mon.0) 15979 : audit [DBG] from='client.? 172.21.15.115:0/2321688337' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:04:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:04:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:04:54.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:04:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:55 smithi121 bash[36902]: audit 2022-04-22T23:04:54.376439+0000 mon.a (mon.0) 15980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:55 smithi121 bash[36902]: cluster 2022-04-22T23:04:54.679852+0000 mgr.y (mgr.34107) 11547 : cluster [DBG] pgmap v12104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:55.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:55 smithi115 bash[45495]: audit 2022-04-22T23:04:54.376439+0000 mon.a (mon.0) 15980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:55 smithi115 bash[45495]: cluster 2022-04-22T23:04:54.679852+0000 mgr.y (mgr.34107) 11547 : cluster [DBG] pgmap v12104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:55 smithi115 bash[46934]: audit 2022-04-22T23:04:54.376439+0000 mon.a (mon.0) 15980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:04:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:55 smithi115 bash[46934]: cluster 2022-04-22T23:04:54.679852+0000 mgr.y (mgr.34107) 11547 : cluster [DBG] pgmap v12104: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:56.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:56 smithi121 bash[36902]: audit 2022-04-22T23:04:56.161918+0000 mon.a (mon.0) 15981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:56.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:56 smithi115 bash[45495]: audit 2022-04-22T23:04:56.161918+0000 mon.a (mon.0) 15981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:56 smithi115 bash[46934]: audit 2022-04-22T23:04:56.161918+0000 mon.a (mon.0) 15981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:04:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:57 smithi121 bash[36902]: cluster 2022-04-22T23:04:56.680955+0000 mgr.y (mgr.34107) 11548 : cluster [DBG] pgmap v12105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:57.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:57 smithi115 bash[45495]: cluster 2022-04-22T23:04:56.680955+0000 mgr.y (mgr.34107) 11548 : cluster [DBG] pgmap v12105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:57 smithi115 bash[46934]: cluster 2022-04-22T23:04:56.680955+0000 mgr.y (mgr.34107) 11548 : cluster [DBG] pgmap v12105: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:58.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:58 smithi121 bash[36902]: audit 2022-04-22T23:04:57.790568+0000 mon.a (mon.0) 15982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:58 smithi115 bash[45495]: audit 2022-04-22T23:04:57.790568+0000 mon.a (mon.0) 15982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:58 smithi115 bash[46934]: audit 2022-04-22T23:04:57.790568+0000 mon.a (mon.0) 15982 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:04:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:04:59 smithi121 bash[36902]: cluster 2022-04-22T23:04:58.682103+0000 mgr.y (mgr.34107) 11549 : cluster [DBG] pgmap v12106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:59.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:04:59 smithi115 bash[45495]: cluster 2022-04-22T23:04:58.682103+0000 mgr.y (mgr.34107) 11549 : cluster [DBG] pgmap v12106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:04:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:04:59 smithi115 bash[46934]: cluster 2022-04-22T23:04:58.682103+0000 mgr.y (mgr.34107) 11549 : cluster [DBG] pgmap v12106: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:00.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:00 smithi121 bash[36902]: audit 2022-04-22T23:04:59.379780+0000 mon.a (mon.0) 15983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:00.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:00 smithi115 bash[45495]: audit 2022-04-22T23:04:59.379780+0000 mon.a (mon.0) 15983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:00 smithi115 bash[46934]: audit 2022-04-22T23:04:59.379780+0000 mon.a (mon.0) 15983 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:01 smithi115 bash[45495]: cluster 2022-04-22T23:05:00.683206+0000 mgr.y (mgr.34107) 11550 : cluster [DBG] pgmap v12107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:01 smithi115 bash[45495]: audit 2022-04-22T23:05:01.165137+0000 mon.a (mon.0) 15984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:01 smithi115 bash[46934]: cluster 2022-04-22T23:05:00.683206+0000 mgr.y (mgr.34107) 11550 : cluster [DBG] pgmap v12107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:01 smithi115 bash[46934]: audit 2022-04-22T23:05:01.165137+0000 mon.a (mon.0) 15984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:01 smithi121 bash[36902]: cluster 2022-04-22T23:05:00.683206+0000 mgr.y (mgr.34107) 11550 : cluster [DBG] pgmap v12107: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:01.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:01 smithi121 bash[36902]: audit 2022-04-22T23:05:01.165137+0000 mon.a (mon.0) 15984 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:02.952Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:03 smithi121 bash[36902]: cluster 2022-04-22T23:05:02.684345+0000 mgr.y (mgr.34107) 11551 : cluster [DBG] pgmap v12108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:03 smithi121 bash[36902]: audit 2022-04-22T23:05:02.793772+0000 mon.a (mon.0) 15985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:03 smithi115 bash[45495]: cluster 2022-04-22T23:05:02.684345+0000 mgr.y (mgr.34107) 11551 : cluster [DBG] pgmap v12108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:03 smithi115 bash[45495]: audit 2022-04-22T23:05:02.793772+0000 mon.a (mon.0) 15985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:03 smithi115 bash[46934]: cluster 2022-04-22T23:05:02.684345+0000 mgr.y (mgr.34107) 11551 : cluster [DBG] pgmap v12108: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:03 smithi115 bash[46934]: audit 2022-04-22T23:05:02.793772+0000 mon.a (mon.0) 15985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:04 smithi121 bash[36902]: audit 2022-04-22T23:05:04.383360+0000 mon.a (mon.0) 15986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:04 smithi115 bash[45495]: audit 2022-04-22T23:05:04.383360+0000 mon.a (mon.0) 15986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:04 smithi115 bash[46934]: audit 2022-04-22T23:05:04.383360+0000 mon.a (mon.0) 15986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:04.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:05 smithi121 bash[36902]: cluster 2022-04-22T23:05:04.685140+0000 mgr.y (mgr.34107) 11552 : cluster [DBG] pgmap v12109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:05 smithi115 bash[45495]: cluster 2022-04-22T23:05:04.685140+0000 mgr.y (mgr.34107) 11552 : cluster [DBG] pgmap v12109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:05 smithi115 bash[46934]: cluster 2022-04-22T23:05:04.685140+0000 mgr.y (mgr.34107) 11552 : cluster [DBG] pgmap v12109: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:06.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:06 smithi121 bash[36902]: audit 2022-04-22T23:05:06.168139+0000 mon.a (mon.0) 15987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:06 smithi115 bash[45495]: audit 2022-04-22T23:05:06.168139+0000 mon.a (mon.0) 15987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:06 smithi115 bash[46934]: audit 2022-04-22T23:05:06.168139+0000 mon.a (mon.0) 15987 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:07 smithi121 bash[36902]: cluster 2022-04-22T23:05:06.686217+0000 mgr.y (mgr.34107) 11553 : cluster [DBG] pgmap v12110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:07 smithi115 bash[45495]: cluster 2022-04-22T23:05:06.686217+0000 mgr.y (mgr.34107) 11553 : cluster [DBG] pgmap v12110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:07 smithi115 bash[46934]: cluster 2022-04-22T23:05:06.686217+0000 mgr.y (mgr.34107) 11553 : cluster [DBG] pgmap v12110: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:08 smithi121 bash[36902]: audit 2022-04-22T23:05:07.796956+0000 mon.a (mon.0) 15988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:08 smithi115 bash[45495]: audit 2022-04-22T23:05:07.796956+0000 mon.a (mon.0) 15988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:08 smithi115 bash[46934]: audit 2022-04-22T23:05:07.796956+0000 mon.a (mon.0) 15988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:09 smithi121 bash[36902]: cluster 2022-04-22T23:05:08.687436+0000 mgr.y (mgr.34107) 11554 : cluster [DBG] pgmap v12111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:09.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:09 smithi121 bash[36902]: audit 2022-04-22T23:05:09.387269+0000 mon.a (mon.0) 15989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:09 smithi115 bash[45495]: cluster 2022-04-22T23:05:08.687436+0000 mgr.y (mgr.34107) 11554 : cluster [DBG] pgmap v12111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:09 smithi115 bash[45495]: audit 2022-04-22T23:05:09.387269+0000 mon.a (mon.0) 15989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:09 smithi115 bash[46934]: cluster 2022-04-22T23:05:08.687436+0000 mgr.y (mgr.34107) 11554 : cluster [DBG] pgmap v12111: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:09 smithi115 bash[46934]: audit 2022-04-22T23:05:09.387269+0000 mon.a (mon.0) 15989 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:11 smithi121 bash[36902]: cluster 2022-04-22T23:05:10.688626+0000 mgr.y (mgr.34107) 11555 : cluster [DBG] pgmap v12112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:12.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:11 smithi121 bash[36902]: audit 2022-04-22T23:05:11.171103+0000 mon.a (mon.0) 15990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:12.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:11 smithi115 bash[45495]: cluster 2022-04-22T23:05:10.688626+0000 mgr.y (mgr.34107) 11555 : cluster [DBG] pgmap v12112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:11 smithi115 bash[45495]: audit 2022-04-22T23:05:11.171103+0000 mon.a (mon.0) 15990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:11 smithi115 bash[46934]: cluster 2022-04-22T23:05:10.688626+0000 mgr.y (mgr.34107) 11555 : cluster [DBG] pgmap v12112: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:11 smithi115 bash[46934]: audit 2022-04-22T23:05:11.171103+0000 mon.a (mon.0) 15990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:12.952Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:13 smithi121 bash[36902]: cluster 2022-04-22T23:05:12.689791+0000 mgr.y (mgr.34107) 11556 : cluster [DBG] pgmap v12113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:13 smithi121 bash[36902]: audit 2022-04-22T23:05:12.799952+0000 mon.a (mon.0) 15991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:13.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:13 smithi115 bash[46934]: cluster 2022-04-22T23:05:12.689791+0000 mgr.y (mgr.34107) 11556 : cluster [DBG] pgmap v12113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:13 smithi115 bash[46934]: audit 2022-04-22T23:05:12.799952+0000 mon.a (mon.0) 15991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:13 smithi115 bash[45495]: cluster 2022-04-22T23:05:12.689791+0000 mgr.y (mgr.34107) 11556 : cluster [DBG] pgmap v12113: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:13 smithi115 bash[45495]: audit 2022-04-22T23:05:12.799952+0000 mon.a (mon.0) 15991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:14 smithi121 bash[36902]: audit 2022-04-22T23:05:14.390840+0000 mon.a (mon.0) 15992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:14 smithi115 bash[46934]: audit 2022-04-22T23:05:14.390840+0000 mon.a (mon.0) 15992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:14 smithi115 bash[45495]: audit 2022-04-22T23:05:14.390840+0000 mon.a (mon.0) 15992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:14.806Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:15 smithi121 bash[36902]: cluster 2022-04-22T23:05:14.690931+0000 mgr.y (mgr.34107) 11557 : cluster [DBG] pgmap v12114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:15 smithi115 bash[45495]: cluster 2022-04-22T23:05:14.690931+0000 mgr.y (mgr.34107) 11557 : cluster [DBG] pgmap v12114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:15 smithi115 bash[46934]: cluster 2022-04-22T23:05:14.690931+0000 mgr.y (mgr.34107) 11557 : cluster [DBG] pgmap v12114: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:16 smithi121 bash[36902]: audit 2022-04-22T23:05:15.620069+0000 mon.a (mon.0) 15993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:05:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:16 smithi121 bash[36902]: audit 2022-04-22T23:05:15.623077+0000 mon.a (mon.0) 15994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:05:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:16 smithi121 bash[36902]: audit 2022-04-22T23:05:15.624859+0000 mon.a (mon.0) 15995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:05:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:16 smithi121 bash[36902]: audit 2022-04-22T23:05:15.977890+0000 mon.a (mon.0) 15996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:05:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:16 smithi121 bash[36902]: audit 2022-04-22T23:05:16.174379+0000 mon.a (mon.0) 15997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[45495]: audit 2022-04-22T23:05:15.620069+0000 mon.a (mon.0) 15993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:05:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[45495]: audit 2022-04-22T23:05:15.623077+0000 mon.a (mon.0) 15994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:05:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[45495]: audit 2022-04-22T23:05:15.624859+0000 mon.a (mon.0) 15995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:05:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[45495]: audit 2022-04-22T23:05:15.977890+0000 mon.a (mon.0) 15996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:05:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[45495]: audit 2022-04-22T23:05:16.174379+0000 mon.a (mon.0) 15997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[46934]: audit 2022-04-22T23:05:15.620069+0000 mon.a (mon.0) 15993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:05:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[46934]: audit 2022-04-22T23:05:15.623077+0000 mon.a (mon.0) 15994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:05:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[46934]: audit 2022-04-22T23:05:15.624859+0000 mon.a (mon.0) 15995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:05:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[46934]: audit 2022-04-22T23:05:15.977890+0000 mon.a (mon.0) 15996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:05:16.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:16 smithi115 bash[46934]: audit 2022-04-22T23:05:16.174379+0000 mon.a (mon.0) 15997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:17 smithi121 bash[36902]: cluster 2022-04-22T23:05:16.692056+0000 mgr.y (mgr.34107) 11558 : cluster [DBG] pgmap v12115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:17 smithi121 bash[36902]: audit 2022-04-22T23:05:17.175549+0000 mon.a (mon.0) 15998 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:05:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:17 smithi121 bash[36902]: audit 2022-04-22T23:05:17.175759+0000 mon.a (mon.0) 15999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[46934]: cluster 2022-04-22T23:05:16.692056+0000 mgr.y (mgr.34107) 11558 : cluster [DBG] pgmap v12115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[46934]: audit 2022-04-22T23:05:17.175549+0000 mon.a (mon.0) 15998 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[46934]: audit 2022-04-22T23:05:17.175759+0000 mon.a (mon.0) 15999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[45495]: cluster 2022-04-22T23:05:16.692056+0000 mgr.y (mgr.34107) 11558 : cluster [DBG] pgmap v12115: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[45495]: audit 2022-04-22T23:05:17.175549+0000 mon.a (mon.0) 15998 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:05:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:17 smithi115 bash[45495]: audit 2022-04-22T23:05:17.175759+0000 mon.a (mon.0) 15999 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:05:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:18 smithi121 bash[36902]: audit 2022-04-22T23:05:17.803163+0000 mon.a (mon.0) 16000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:18 smithi115 bash[45495]: audit 2022-04-22T23:05:17.803163+0000 mon.a (mon.0) 16000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:18 smithi115 bash[46934]: audit 2022-04-22T23:05:17.803163+0000 mon.a (mon.0) 16000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:19 smithi121 bash[36902]: cluster 2022-04-22T23:05:18.693236+0000 mgr.y (mgr.34107) 11559 : cluster [DBG] pgmap v12116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:19.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:19 smithi121 bash[36902]: audit 2022-04-22T23:05:19.394755+0000 mon.a (mon.0) 16001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:19 smithi115 bash[45495]: cluster 2022-04-22T23:05:18.693236+0000 mgr.y (mgr.34107) 11559 : cluster [DBG] pgmap v12116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:19 smithi115 bash[45495]: audit 2022-04-22T23:05:19.394755+0000 mon.a (mon.0) 16001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:19 smithi115 bash[46934]: cluster 2022-04-22T23:05:18.693236+0000 mgr.y (mgr.34107) 11559 : cluster [DBG] pgmap v12116: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:19 smithi115 bash[46934]: audit 2022-04-22T23:05:19.394755+0000 mon.a (mon.0) 16001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:21 smithi121 bash[36902]: cluster 2022-04-22T23:05:20.694365+0000 mgr.y (mgr.34107) 11560 : cluster [DBG] pgmap v12117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:22.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:21 smithi121 bash[36902]: audit 2022-04-22T23:05:21.177453+0000 mon.a (mon.0) 16002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:22.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:21 smithi115 bash[45495]: cluster 2022-04-22T23:05:20.694365+0000 mgr.y (mgr.34107) 11560 : cluster [DBG] pgmap v12117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:21 smithi115 bash[45495]: audit 2022-04-22T23:05:21.177453+0000 mon.a (mon.0) 16002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:21 smithi115 bash[46934]: cluster 2022-04-22T23:05:20.694365+0000 mgr.y (mgr.34107) 11560 : cluster [DBG] pgmap v12117: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:21 smithi115 bash[46934]: audit 2022-04-22T23:05:21.177453+0000 mon.a (mon.0) 16002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:22.953Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:23 smithi121 bash[36902]: cluster 2022-04-22T23:05:22.695414+0000 mgr.y (mgr.34107) 11561 : cluster [DBG] pgmap v12118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:23 smithi121 bash[36902]: audit 2022-04-22T23:05:22.806364+0000 mon.a (mon.0) 16003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:23 smithi115 bash[45495]: cluster 2022-04-22T23:05:22.695414+0000 mgr.y (mgr.34107) 11561 : cluster [DBG] pgmap v12118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:23.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:23 smithi115 bash[45495]: audit 2022-04-22T23:05:22.806364+0000 mon.a (mon.0) 16003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:23.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:23 smithi115 bash[46934]: cluster 2022-04-22T23:05:22.695414+0000 mgr.y (mgr.34107) 11561 : cluster [DBG] pgmap v12118: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:23.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:23 smithi115 bash[46934]: audit 2022-04-22T23:05:22.806364+0000 mon.a (mon.0) 16003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:24.443 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:05:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:24 smithi121 bash[36902]: audit 2022-04-22T23:05:24.398401+0000 mon.a (mon.0) 16004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:24 smithi115 bash[45495]: audit 2022-04-22T23:05:24.398401+0000 mon.a (mon.0) 16004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:24 smithi115 bash[46934]: audit 2022-04-22T23:05:24.398401+0000 mon.a (mon.0) 16004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:24.791 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:05:24.791 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:05:24.792 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:05:24.793 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:05:25.201 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:05:25.202 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:05:25.203 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:05:25.203 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:05:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:24.807Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:25 smithi121 bash[36902]: audit 2022-04-22T23:05:24.424950+0000 mgr.y (mgr.34107) 11562 : audit [DBG] from='client.57253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:25 smithi121 bash[36902]: cluster 2022-04-22T23:05:24.696399+0000 mgr.y (mgr.34107) 11563 : cluster [DBG] pgmap v12119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:25 smithi121 bash[36902]: audit 2022-04-22T23:05:24.766864+0000 mgr.y (mgr.34107) 11564 : audit [DBG] from='client.57259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:25 smithi121 bash[36902]: audit 2022-04-22T23:05:25.197239+0000 mon.a (mon.0) 16005 : audit [DBG] from='client.? 172.21.15.115:0/4020446079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[45495]: audit 2022-04-22T23:05:24.424950+0000 mgr.y (mgr.34107) 11562 : audit [DBG] from='client.57253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[45495]: cluster 2022-04-22T23:05:24.696399+0000 mgr.y (mgr.34107) 11563 : cluster [DBG] pgmap v12119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[45495]: audit 2022-04-22T23:05:24.766864+0000 mgr.y (mgr.34107) 11564 : audit [DBG] from='client.57259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[45495]: audit 2022-04-22T23:05:25.197239+0000 mon.a (mon.0) 16005 : audit [DBG] from='client.? 172.21.15.115:0/4020446079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[46934]: audit 2022-04-22T23:05:24.424950+0000 mgr.y (mgr.34107) 11562 : audit [DBG] from='client.57253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[46934]: cluster 2022-04-22T23:05:24.696399+0000 mgr.y (mgr.34107) 11563 : cluster [DBG] pgmap v12119: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[46934]: audit 2022-04-22T23:05:24.766864+0000 mgr.y (mgr.34107) 11564 : audit [DBG] from='client.57259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:25 smithi115 bash[46934]: audit 2022-04-22T23:05:25.197239+0000 mon.a (mon.0) 16005 : audit [DBG] from='client.? 172.21.15.115:0/4020446079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:26 smithi121 bash[36902]: audit 2022-04-22T23:05:26.180337+0000 mon.a (mon.0) 16006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:26 smithi115 bash[45495]: audit 2022-04-22T23:05:26.180337+0000 mon.a (mon.0) 16006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:26 smithi115 bash[46934]: audit 2022-04-22T23:05:26.180337+0000 mon.a (mon.0) 16006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:27.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:27 smithi121 bash[36902]: cluster 2022-04-22T23:05:26.697502+0000 mgr.y (mgr.34107) 11565 : cluster [DBG] pgmap v12120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:27.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:27 smithi115 bash[45495]: cluster 2022-04-22T23:05:26.697502+0000 mgr.y (mgr.34107) 11565 : cluster [DBG] pgmap v12120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:27 smithi115 bash[46934]: cluster 2022-04-22T23:05:26.697502+0000 mgr.y (mgr.34107) 11565 : cluster [DBG] pgmap v12120: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:28.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:28 smithi121 bash[36902]: audit 2022-04-22T23:05:27.809603+0000 mon.a (mon.0) 16007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:28.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:28 smithi115 bash[46934]: audit 2022-04-22T23:05:27.809603+0000 mon.a (mon.0) 16007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:28 smithi115 bash[45495]: audit 2022-04-22T23:05:27.809603+0000 mon.a (mon.0) 16007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:29 smithi121 bash[36902]: cluster 2022-04-22T23:05:28.698637+0000 mgr.y (mgr.34107) 11566 : cluster [DBG] pgmap v12121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:29.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:29 smithi121 bash[36902]: audit 2022-04-22T23:05:29.402004+0000 mon.a (mon.0) 16008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:29 smithi115 bash[45495]: cluster 2022-04-22T23:05:28.698637+0000 mgr.y (mgr.34107) 11566 : cluster [DBG] pgmap v12121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:29 smithi115 bash[45495]: audit 2022-04-22T23:05:29.402004+0000 mon.a (mon.0) 16008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:29 smithi115 bash[46934]: cluster 2022-04-22T23:05:28.698637+0000 mgr.y (mgr.34107) 11566 : cluster [DBG] pgmap v12121: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:29 smithi115 bash[46934]: audit 2022-04-22T23:05:29.402004+0000 mon.a (mon.0) 16008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:31 smithi121 bash[36902]: cluster 2022-04-22T23:05:30.699734+0000 mgr.y (mgr.34107) 11567 : cluster [DBG] pgmap v12122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:31 smithi121 bash[36902]: audit 2022-04-22T23:05:31.183343+0000 mon.a (mon.0) 16009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:31 smithi115 bash[45495]: cluster 2022-04-22T23:05:30.699734+0000 mgr.y (mgr.34107) 11567 : cluster [DBG] pgmap v12122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:31 smithi115 bash[45495]: audit 2022-04-22T23:05:31.183343+0000 mon.a (mon.0) 16009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:31 smithi115 bash[46934]: cluster 2022-04-22T23:05:30.699734+0000 mgr.y (mgr.34107) 11567 : cluster [DBG] pgmap v12122: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:32.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:31 smithi115 bash[46934]: audit 2022-04-22T23:05:31.183343+0000 mon.a (mon.0) 16009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:32.954Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:33 smithi121 bash[36902]: cluster 2022-04-22T23:05:32.700846+0000 mgr.y (mgr.34107) 11568 : cluster [DBG] pgmap v12123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:33 smithi121 bash[36902]: audit 2022-04-22T23:05:32.812680+0000 mon.a (mon.0) 16010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:33 smithi115 bash[45495]: cluster 2022-04-22T23:05:32.700846+0000 mgr.y (mgr.34107) 11568 : cluster [DBG] pgmap v12123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:33 smithi115 bash[45495]: audit 2022-04-22T23:05:32.812680+0000 mon.a (mon.0) 16010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:33 smithi115 bash[46934]: cluster 2022-04-22T23:05:32.700846+0000 mgr.y (mgr.34107) 11568 : cluster [DBG] pgmap v12123: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:33 smithi115 bash[46934]: audit 2022-04-22T23:05:32.812680+0000 mon.a (mon.0) 16010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:34 smithi121 bash[36902]: audit 2022-04-22T23:05:34.405788+0000 mon.a (mon.0) 16011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:34 smithi115 bash[45495]: audit 2022-04-22T23:05:34.405788+0000 mon.a (mon.0) 16011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:34 smithi115 bash[46934]: audit 2022-04-22T23:05:34.405788+0000 mon.a (mon.0) 16011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:34.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:35.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:35 smithi121 bash[36902]: cluster 2022-04-22T23:05:34.701949+0000 mgr.y (mgr.34107) 11569 : cluster [DBG] pgmap v12124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:35 smithi115 bash[45495]: cluster 2022-04-22T23:05:34.701949+0000 mgr.y (mgr.34107) 11569 : cluster [DBG] pgmap v12124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:35 smithi115 bash[46934]: cluster 2022-04-22T23:05:34.701949+0000 mgr.y (mgr.34107) 11569 : cluster [DBG] pgmap v12124: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:36 smithi121 bash[36902]: audit 2022-04-22T23:05:36.186296+0000 mon.a (mon.0) 16012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:36 smithi115 bash[45495]: audit 2022-04-22T23:05:36.186296+0000 mon.a (mon.0) 16012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:36 smithi115 bash[46934]: audit 2022-04-22T23:05:36.186296+0000 mon.a (mon.0) 16012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:37 smithi121 bash[36902]: cluster 2022-04-22T23:05:36.703031+0000 mgr.y (mgr.34107) 11570 : cluster [DBG] pgmap v12125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:37 smithi115 bash[45495]: cluster 2022-04-22T23:05:36.703031+0000 mgr.y (mgr.34107) 11570 : cluster [DBG] pgmap v12125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:37 smithi115 bash[46934]: cluster 2022-04-22T23:05:36.703031+0000 mgr.y (mgr.34107) 11570 : cluster [DBG] pgmap v12125: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:38 smithi121 bash[36902]: audit 2022-04-22T23:05:37.815732+0000 mon.a (mon.0) 16013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:38 smithi115 bash[45495]: audit 2022-04-22T23:05:37.815732+0000 mon.a (mon.0) 16013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:38 smithi115 bash[46934]: audit 2022-04-22T23:05:37.815732+0000 mon.a (mon.0) 16013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:39 smithi121 bash[36902]: cluster 2022-04-22T23:05:38.704239+0000 mgr.y (mgr.34107) 11571 : cluster [DBG] pgmap v12126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:39 smithi121 bash[36902]: audit 2022-04-22T23:05:39.409504+0000 mon.a (mon.0) 16014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:39 smithi115 bash[45495]: cluster 2022-04-22T23:05:38.704239+0000 mgr.y (mgr.34107) 11571 : cluster [DBG] pgmap v12126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:39 smithi115 bash[45495]: audit 2022-04-22T23:05:39.409504+0000 mon.a (mon.0) 16014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:39 smithi115 bash[46934]: cluster 2022-04-22T23:05:38.704239+0000 mgr.y (mgr.34107) 11571 : cluster [DBG] pgmap v12126: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:39 smithi115 bash[46934]: audit 2022-04-22T23:05:39.409504+0000 mon.a (mon.0) 16014 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:41 smithi115 bash[45495]: cluster 2022-04-22T23:05:40.705294+0000 mgr.y (mgr.34107) 11572 : cluster [DBG] pgmap v12127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:42.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:41 smithi115 bash[45495]: audit 2022-04-22T23:05:41.189257+0000 mon.a (mon.0) 16015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:41 smithi115 bash[46934]: cluster 2022-04-22T23:05:40.705294+0000 mgr.y (mgr.34107) 11572 : cluster [DBG] pgmap v12127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:42.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:41 smithi115 bash[46934]: audit 2022-04-22T23:05:41.189257+0000 mon.a (mon.0) 16015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:41 smithi121 bash[36902]: cluster 2022-04-22T23:05:40.705294+0000 mgr.y (mgr.34107) 11572 : cluster [DBG] pgmap v12127: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:41 smithi121 bash[36902]: audit 2022-04-22T23:05:41.189257+0000 mon.a (mon.0) 16015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:42.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:43 smithi121 bash[36902]: cluster 2022-04-22T23:05:42.706415+0000 mgr.y (mgr.34107) 11573 : cluster [DBG] pgmap v12128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:43 smithi121 bash[36902]: audit 2022-04-22T23:05:42.818885+0000 mon.a (mon.0) 16016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:43 smithi115 bash[45495]: cluster 2022-04-22T23:05:42.706415+0000 mgr.y (mgr.34107) 11573 : cluster [DBG] pgmap v12128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:43 smithi115 bash[45495]: audit 2022-04-22T23:05:42.818885+0000 mon.a (mon.0) 16016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:43 smithi115 bash[46934]: cluster 2022-04-22T23:05:42.706415+0000 mgr.y (mgr.34107) 11573 : cluster [DBG] pgmap v12128: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:43 smithi115 bash[46934]: audit 2022-04-22T23:05:42.818885+0000 mon.a (mon.0) 16016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:44.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:44 smithi121 bash[36902]: audit 2022-04-22T23:05:44.413342+0000 mon.a (mon.0) 16017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:44 smithi115 bash[45495]: audit 2022-04-22T23:05:44.413342+0000 mon.a (mon.0) 16017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:44 smithi115 bash[46934]: audit 2022-04-22T23:05:44.413342+0000 mon.a (mon.0) 16017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:44.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:45.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:45 smithi121 bash[36902]: cluster 2022-04-22T23:05:44.707549+0000 mgr.y (mgr.34107) 11574 : cluster [DBG] pgmap v12129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:45.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:45 smithi115 bash[46934]: cluster 2022-04-22T23:05:44.707549+0000 mgr.y (mgr.34107) 11574 : cluster [DBG] pgmap v12129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:45 smithi115 bash[45495]: cluster 2022-04-22T23:05:44.707549+0000 mgr.y (mgr.34107) 11574 : cluster [DBG] pgmap v12129: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:46.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:46 smithi121 bash[36902]: audit 2022-04-22T23:05:46.192330+0000 mon.a (mon.0) 16018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:46 smithi115 bash[45495]: audit 2022-04-22T23:05:46.192330+0000 mon.a (mon.0) 16018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:46 smithi115 bash[46934]: audit 2022-04-22T23:05:46.192330+0000 mon.a (mon.0) 16018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:47.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:47 smithi121 bash[36902]: cluster 2022-04-22T23:05:46.708702+0000 mgr.y (mgr.34107) 11575 : cluster [DBG] pgmap v12130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:47.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:47 smithi115 bash[45495]: cluster 2022-04-22T23:05:46.708702+0000 mgr.y (mgr.34107) 11575 : cluster [DBG] pgmap v12130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:47 smithi115 bash[46934]: cluster 2022-04-22T23:05:46.708702+0000 mgr.y (mgr.34107) 11575 : cluster [DBG] pgmap v12130: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:48.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:48 smithi121 bash[36902]: audit 2022-04-22T23:05:47.822017+0000 mon.a (mon.0) 16019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:48 smithi115 bash[45495]: audit 2022-04-22T23:05:47.822017+0000 mon.a (mon.0) 16019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:48.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:48 smithi115 bash[46934]: audit 2022-04-22T23:05:47.822017+0000 mon.a (mon.0) 16019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:49.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:49 smithi121 bash[36902]: cluster 2022-04-22T23:05:48.709894+0000 mgr.y (mgr.34107) 11576 : cluster [DBG] pgmap v12131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:49 smithi121 bash[36902]: audit 2022-04-22T23:05:49.417090+0000 mon.a (mon.0) 16020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:49 smithi115 bash[45495]: cluster 2022-04-22T23:05:48.709894+0000 mgr.y (mgr.34107) 11576 : cluster [DBG] pgmap v12131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:49 smithi115 bash[45495]: audit 2022-04-22T23:05:49.417090+0000 mon.a (mon.0) 16020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:49 smithi115 bash[46934]: cluster 2022-04-22T23:05:48.709894+0000 mgr.y (mgr.34107) 11576 : cluster [DBG] pgmap v12131: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:49 smithi115 bash[46934]: audit 2022-04-22T23:05:49.417090+0000 mon.a (mon.0) 16020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:51 smithi115 bash[46934]: cluster 2022-04-22T23:05:50.710940+0000 mgr.y (mgr.34107) 11577 : cluster [DBG] pgmap v12132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:51 smithi115 bash[46934]: audit 2022-04-22T23:05:51.195345+0000 mon.a (mon.0) 16021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:51 smithi115 bash[45495]: cluster 2022-04-22T23:05:50.710940+0000 mgr.y (mgr.34107) 11577 : cluster [DBG] pgmap v12132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:52.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:51 smithi115 bash[45495]: audit 2022-04-22T23:05:51.195345+0000 mon.a (mon.0) 16021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:51 smithi121 bash[36902]: cluster 2022-04-22T23:05:50.710940+0000 mgr.y (mgr.34107) 11577 : cluster [DBG] pgmap v12132: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:51 smithi121 bash[36902]: audit 2022-04-22T23:05:51.195345+0000 mon.a (mon.0) 16021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:52.956Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:53 smithi121 bash[36902]: cluster 2022-04-22T23:05:52.712103+0000 mgr.y (mgr.34107) 11578 : cluster [DBG] pgmap v12133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:53 smithi121 bash[36902]: audit 2022-04-22T23:05:52.825320+0000 mon.a (mon.0) 16022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:53 smithi115 bash[45495]: cluster 2022-04-22T23:05:52.712103+0000 mgr.y (mgr.34107) 11578 : cluster [DBG] pgmap v12133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:53 smithi115 bash[45495]: audit 2022-04-22T23:05:52.825320+0000 mon.a (mon.0) 16022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:53 smithi115 bash[46934]: cluster 2022-04-22T23:05:52.712103+0000 mgr.y (mgr.34107) 11578 : cluster [DBG] pgmap v12133: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:53 smithi115 bash[46934]: audit 2022-04-22T23:05:52.825320+0000 mon.a (mon.0) 16022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:54.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:54 smithi121 bash[36902]: audit 2022-04-22T23:05:54.420837+0000 mon.a (mon.0) 16023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:54 smithi115 bash[45495]: audit 2022-04-22T23:05:54.420837+0000 mon.a (mon.0) 16023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:54 smithi115 bash[46934]: audit 2022-04-22T23:05:54.420837+0000 mon.a (mon.0) 16023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:05:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:05:54.808Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:05:55.581 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:05:55.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:55 smithi121 bash[36902]: cluster 2022-04-22T23:05:54.713165+0000 mgr.y (mgr.34107) 11579 : cluster [DBG] pgmap v12134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:55 smithi115 bash[45495]: cluster 2022-04-22T23:05:54.713165+0000 mgr.y (mgr.34107) 11579 : cluster [DBG] pgmap v12134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:55 smithi115 bash[46934]: cluster 2022-04-22T23:05:54.713165+0000 mgr.y (mgr.34107) 11579 : cluster [DBG] pgmap v12134: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.2M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:05:55.918 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 301M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 11.2M - 1dbe0e931976 f4b044642a96 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:05:55.919 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 652M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:05:55.920 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 71.3M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:05:56.324 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:05:56.324 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:05:56.324 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:05:56.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:05:56.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:05:56.326 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:05:56.326 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:05:56.326 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:05:56.326 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:05:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:56 smithi121 bash[36902]: audit 2022-04-22T23:05:55.561956+0000 mgr.y (mgr.34107) 11580 : audit [DBG] from='client.47331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:56 smithi121 bash[36902]: audit 2022-04-22T23:05:55.901136+0000 mgr.y (mgr.34107) 11581 : audit [DBG] from='client.57277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:56 smithi121 bash[36902]: audit 2022-04-22T23:05:56.197981+0000 mon.a (mon.0) 16024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:56 smithi121 bash[36902]: audit 2022-04-22T23:05:56.320806+0000 mon.c (mon.1) 390 : audit [DBG] from='client.? 172.21.15.115:0/1006047541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[45495]: audit 2022-04-22T23:05:55.561956+0000 mgr.y (mgr.34107) 11580 : audit [DBG] from='client.47331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[45495]: audit 2022-04-22T23:05:55.901136+0000 mgr.y (mgr.34107) 11581 : audit [DBG] from='client.57277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[45495]: audit 2022-04-22T23:05:56.197981+0000 mon.a (mon.0) 16024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[45495]: audit 2022-04-22T23:05:56.320806+0000 mon.c (mon.1) 390 : audit [DBG] from='client.? 172.21.15.115:0/1006047541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[46934]: audit 2022-04-22T23:05:55.561956+0000 mgr.y (mgr.34107) 11580 : audit [DBG] from='client.47331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[46934]: audit 2022-04-22T23:05:55.901136+0000 mgr.y (mgr.34107) 11581 : audit [DBG] from='client.57277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:05:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[46934]: audit 2022-04-22T23:05:56.197981+0000 mon.a (mon.0) 16024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:05:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:56 smithi115 bash[46934]: audit 2022-04-22T23:05:56.320806+0000 mon.c (mon.1) 390 : audit [DBG] from='client.? 172.21.15.115:0/1006047541' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:05:57.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:57 smithi121 bash[36902]: cluster 2022-04-22T23:05:56.714387+0000 mgr.y (mgr.34107) 11582 : cluster [DBG] pgmap v12135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:57 smithi115 bash[45495]: cluster 2022-04-22T23:05:56.714387+0000 mgr.y (mgr.34107) 11582 : cluster [DBG] pgmap v12135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:57 smithi115 bash[46934]: cluster 2022-04-22T23:05:56.714387+0000 mgr.y (mgr.34107) 11582 : cluster [DBG] pgmap v12135: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:58.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:58 smithi121 bash[36902]: audit 2022-04-22T23:05:57.828673+0000 mon.a (mon.0) 16025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:58 smithi115 bash[45495]: audit 2022-04-22T23:05:57.828673+0000 mon.a (mon.0) 16025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:58 smithi115 bash[46934]: audit 2022-04-22T23:05:57.828673+0000 mon.a (mon.0) 16025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:05:59.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:59 smithi121 bash[36902]: cluster 2022-04-22T23:05:58.715652+0000 mgr.y (mgr.34107) 11583 : cluster [DBG] pgmap v12136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:05:59 smithi121 bash[36902]: audit 2022-04-22T23:05:59.424469+0000 mon.a (mon.0) 16026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:59 smithi115 bash[45495]: cluster 2022-04-22T23:05:58.715652+0000 mgr.y (mgr.34107) 11583 : cluster [DBG] pgmap v12136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:05:59 smithi115 bash[45495]: audit 2022-04-22T23:05:59.424469+0000 mon.a (mon.0) 16026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:05:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:59 smithi115 bash[46934]: cluster 2022-04-22T23:05:58.715652+0000 mgr.y (mgr.34107) 11583 : cluster [DBG] pgmap v12136: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:05:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:05:59 smithi115 bash[46934]: audit 2022-04-22T23:05:59.424469+0000 mon.a (mon.0) 16026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:01 smithi115 bash[45495]: cluster 2022-04-22T23:06:00.716628+0000 mgr.y (mgr.34107) 11584 : cluster [DBG] pgmap v12137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:01 smithi115 bash[45495]: audit 2022-04-22T23:06:01.200411+0000 mon.a (mon.0) 16027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:01 smithi115 bash[46934]: cluster 2022-04-22T23:06:00.716628+0000 mgr.y (mgr.34107) 11584 : cluster [DBG] pgmap v12137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:01 smithi115 bash[46934]: audit 2022-04-22T23:06:01.200411+0000 mon.a (mon.0) 16027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:01 smithi121 bash[36902]: cluster 2022-04-22T23:06:00.716628+0000 mgr.y (mgr.34107) 11584 : cluster [DBG] pgmap v12137: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:01 smithi121 bash[36902]: audit 2022-04-22T23:06:01.200411+0000 mon.a (mon.0) 16027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:02.955Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:03 smithi121 bash[36902]: cluster 2022-04-22T23:06:02.717733+0000 mgr.y (mgr.34107) 11585 : cluster [DBG] pgmap v12138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:03 smithi121 bash[36902]: audit 2022-04-22T23:06:02.832004+0000 mon.a (mon.0) 16028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:03 smithi115 bash[45495]: cluster 2022-04-22T23:06:02.717733+0000 mgr.y (mgr.34107) 11585 : cluster [DBG] pgmap v12138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:03 smithi115 bash[45495]: audit 2022-04-22T23:06:02.832004+0000 mon.a (mon.0) 16028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:03 smithi115 bash[46934]: cluster 2022-04-22T23:06:02.717733+0000 mgr.y (mgr.34107) 11585 : cluster [DBG] pgmap v12138: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:03 smithi115 bash[46934]: audit 2022-04-22T23:06:02.832004+0000 mon.a (mon.0) 16028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:04.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:04 smithi121 bash[36902]: audit 2022-04-22T23:06:04.427753+0000 mon.a (mon.0) 16029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:04 smithi115 bash[45495]: audit 2022-04-22T23:06:04.427753+0000 mon.a (mon.0) 16029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:04 smithi115 bash[46934]: audit 2022-04-22T23:06:04.427753+0000 mon.a (mon.0) 16029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:04.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:05.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:05 smithi121 bash[36902]: cluster 2022-04-22T23:06:04.718852+0000 mgr.y (mgr.34107) 11586 : cluster [DBG] pgmap v12139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:05 smithi115 bash[45495]: cluster 2022-04-22T23:06:04.718852+0000 mgr.y (mgr.34107) 11586 : cluster [DBG] pgmap v12139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:05 smithi115 bash[46934]: cluster 2022-04-22T23:06:04.718852+0000 mgr.y (mgr.34107) 11586 : cluster [DBG] pgmap v12139: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:06.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:06 smithi121 bash[36902]: audit 2022-04-22T23:06:06.203447+0000 mon.a (mon.0) 16030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:06.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:06 smithi115 bash[45495]: audit 2022-04-22T23:06:06.203447+0000 mon.a (mon.0) 16030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:06 smithi115 bash[46934]: audit 2022-04-22T23:06:06.203447+0000 mon.a (mon.0) 16030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:07.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:07 smithi121 bash[36902]: cluster 2022-04-22T23:06:06.720066+0000 mgr.y (mgr.34107) 11587 : cluster [DBG] pgmap v12140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:07 smithi115 bash[45495]: cluster 2022-04-22T23:06:06.720066+0000 mgr.y (mgr.34107) 11587 : cluster [DBG] pgmap v12140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:07 smithi115 bash[46934]: cluster 2022-04-22T23:06:06.720066+0000 mgr.y (mgr.34107) 11587 : cluster [DBG] pgmap v12140: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:08.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:08 smithi121 bash[36902]: audit 2022-04-22T23:06:07.835284+0000 mon.a (mon.0) 16031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:08.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:08 smithi115 bash[45495]: audit 2022-04-22T23:06:07.835284+0000 mon.a (mon.0) 16031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:08 smithi115 bash[46934]: audit 2022-04-22T23:06:07.835284+0000 mon.a (mon.0) 16031 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:09.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:09 smithi115 bash[45495]: cluster 2022-04-22T23:06:08.721256+0000 mgr.y (mgr.34107) 11588 : cluster [DBG] pgmap v12141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:09 smithi115 bash[45495]: audit 2022-04-22T23:06:09.431336+0000 mon.a (mon.0) 16032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:09 smithi115 bash[46934]: cluster 2022-04-22T23:06:08.721256+0000 mgr.y (mgr.34107) 11588 : cluster [DBG] pgmap v12141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:09 smithi115 bash[46934]: audit 2022-04-22T23:06:09.431336+0000 mon.a (mon.0) 16032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:09 smithi121 bash[36902]: cluster 2022-04-22T23:06:08.721256+0000 mgr.y (mgr.34107) 11588 : cluster [DBG] pgmap v12141: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:09 smithi121 bash[36902]: audit 2022-04-22T23:06:09.431336+0000 mon.a (mon.0) 16032 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:12.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:11 smithi115 bash[46934]: cluster 2022-04-22T23:06:10.722363+0000 mgr.y (mgr.34107) 11589 : cluster [DBG] pgmap v12142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:12.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:11 smithi115 bash[46934]: audit 2022-04-22T23:06:11.206307+0000 mon.a (mon.0) 16033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:11 smithi115 bash[45495]: cluster 2022-04-22T23:06:10.722363+0000 mgr.y (mgr.34107) 11589 : cluster [DBG] pgmap v12142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:12.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:11 smithi115 bash[45495]: audit 2022-04-22T23:06:11.206307+0000 mon.a (mon.0) 16033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:11 smithi121 bash[36902]: cluster 2022-04-22T23:06:10.722363+0000 mgr.y (mgr.34107) 11589 : cluster [DBG] pgmap v12142: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:11 smithi121 bash[36902]: audit 2022-04-22T23:06:11.206307+0000 mon.a (mon.0) 16033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:12.957Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:13 smithi121 bash[36902]: cluster 2022-04-22T23:06:12.723387+0000 mgr.y (mgr.34107) 11590 : cluster [DBG] pgmap v12143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:13 smithi121 bash[36902]: audit 2022-04-22T23:06:12.838426+0000 mon.a (mon.0) 16034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:13 smithi115 bash[45495]: cluster 2022-04-22T23:06:12.723387+0000 mgr.y (mgr.34107) 11590 : cluster [DBG] pgmap v12143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:13 smithi115 bash[45495]: audit 2022-04-22T23:06:12.838426+0000 mon.a (mon.0) 16034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:13 smithi115 bash[46934]: cluster 2022-04-22T23:06:12.723387+0000 mgr.y (mgr.34107) 11590 : cluster [DBG] pgmap v12143: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:13 smithi115 bash[46934]: audit 2022-04-22T23:06:12.838426+0000 mon.a (mon.0) 16034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:14.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:14 smithi121 bash[36902]: audit 2022-04-22T23:06:14.434881+0000 mon.a (mon.0) 16035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:14 smithi115 bash[45495]: audit 2022-04-22T23:06:14.434881+0000 mon.a (mon.0) 16035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:14 smithi115 bash[46934]: audit 2022-04-22T23:06:14.434881+0000 mon.a (mon.0) 16035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:14.809Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:15 smithi121 bash[36902]: cluster 2022-04-22T23:06:14.724694+0000 mgr.y (mgr.34107) 11591 : cluster [DBG] pgmap v12144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:15 smithi115 bash[45495]: cluster 2022-04-22T23:06:14.724694+0000 mgr.y (mgr.34107) 11591 : cluster [DBG] pgmap v12144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:15 smithi115 bash[46934]: cluster 2022-04-22T23:06:14.724694+0000 mgr.y (mgr.34107) 11591 : cluster [DBG] pgmap v12144: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:16.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:16 smithi121 bash[36902]: audit 2022-04-22T23:06:15.987173+0000 mon.a (mon.0) 16036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:06:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:16 smithi121 bash[36902]: audit 2022-04-22T23:06:15.990203+0000 mon.a (mon.0) 16037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:06:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:16 smithi121 bash[36902]: audit 2022-04-22T23:06:15.993603+0000 mon.a (mon.0) 16038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:06:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:16 smithi121 bash[36902]: audit 2022-04-22T23:06:16.209447+0000 mon.a (mon.0) 16039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[45495]: audit 2022-04-22T23:06:15.987173+0000 mon.a (mon.0) 16036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[45495]: audit 2022-04-22T23:06:15.990203+0000 mon.a (mon.0) 16037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[45495]: audit 2022-04-22T23:06:15.993603+0000 mon.a (mon.0) 16038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[45495]: audit 2022-04-22T23:06:16.209447+0000 mon.a (mon.0) 16039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[46934]: audit 2022-04-22T23:06:15.987173+0000 mon.a (mon.0) 16036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[46934]: audit 2022-04-22T23:06:15.990203+0000 mon.a (mon.0) 16037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[46934]: audit 2022-04-22T23:06:15.993603+0000 mon.a (mon.0) 16038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:06:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:16 smithi115 bash[46934]: audit 2022-04-22T23:06:16.209447+0000 mon.a (mon.0) 16039 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:17.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:17 smithi121 bash[36902]: cluster 2022-04-22T23:06:16.725781+0000 mgr.y (mgr.34107) 11592 : cluster [DBG] pgmap v12145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:17 smithi121 bash[36902]: audit 2022-04-22T23:06:17.186417+0000 mon.a (mon.0) 16040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:06:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:17 smithi121 bash[36902]: audit 2022-04-22T23:06:17.186661+0000 mon.a (mon.0) 16041 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:06:17.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[45495]: cluster 2022-04-22T23:06:16.725781+0000 mgr.y (mgr.34107) 11592 : cluster [DBG] pgmap v12145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[45495]: audit 2022-04-22T23:06:17.186417+0000 mon.a (mon.0) 16040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:06:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[45495]: audit 2022-04-22T23:06:17.186661+0000 mon.a (mon.0) 16041 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:06:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[46934]: cluster 2022-04-22T23:06:16.725781+0000 mgr.y (mgr.34107) 11592 : cluster [DBG] pgmap v12145: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[46934]: audit 2022-04-22T23:06:17.186417+0000 mon.a (mon.0) 16040 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:06:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:17 smithi115 bash[46934]: audit 2022-04-22T23:06:17.186661+0000 mon.a (mon.0) 16041 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:06:18.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:18 smithi121 bash[36902]: audit 2022-04-22T23:06:17.841634+0000 mon.a (mon.0) 16042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:18 smithi115 bash[45495]: audit 2022-04-22T23:06:17.841634+0000 mon.a (mon.0) 16042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:18 smithi115 bash[46934]: audit 2022-04-22T23:06:17.841634+0000 mon.a (mon.0) 16042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:19 smithi115 bash[46934]: cluster 2022-04-22T23:06:18.726923+0000 mgr.y (mgr.34107) 11593 : cluster [DBG] pgmap v12146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:19 smithi115 bash[46934]: audit 2022-04-22T23:06:19.438925+0000 mon.a (mon.0) 16043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:19 smithi115 bash[45495]: cluster 2022-04-22T23:06:18.726923+0000 mgr.y (mgr.34107) 11593 : cluster [DBG] pgmap v12146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:19 smithi115 bash[45495]: audit 2022-04-22T23:06:19.438925+0000 mon.a (mon.0) 16043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:19 smithi121 bash[36902]: cluster 2022-04-22T23:06:18.726923+0000 mgr.y (mgr.34107) 11593 : cluster [DBG] pgmap v12146: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:19 smithi121 bash[36902]: audit 2022-04-22T23:06:19.438925+0000 mon.a (mon.0) 16043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:21 smithi121 bash[36902]: audit 2022-04-22T23:06:20.390898+0000 mon.a (mon.0) 16044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:21 smithi121 bash[36902]: cluster 2022-04-22T23:06:20.728196+0000 mgr.y (mgr.34107) 11594 : cluster [DBG] pgmap v12147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:21 smithi121 bash[36902]: audit 2022-04-22T23:06:20.753546+0000 mon.a (mon.0) 16045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:21 smithi121 bash[36902]: audit 2022-04-22T23:06:21.212272+0000 mon.a (mon.0) 16046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[45495]: audit 2022-04-22T23:06:20.390898+0000 mon.a (mon.0) 16044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[45495]: cluster 2022-04-22T23:06:20.728196+0000 mgr.y (mgr.34107) 11594 : cluster [DBG] pgmap v12147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[45495]: audit 2022-04-22T23:06:20.753546+0000 mon.a (mon.0) 16045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[45495]: audit 2022-04-22T23:06:21.212272+0000 mon.a (mon.0) 16046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[46934]: audit 2022-04-22T23:06:20.390898+0000 mon.a (mon.0) 16044 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[46934]: cluster 2022-04-22T23:06:20.728196+0000 mgr.y (mgr.34107) 11594 : cluster [DBG] pgmap v12147: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[46934]: audit 2022-04-22T23:06:20.753546+0000 mon.a (mon.0) 16045 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:06:21.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:21 smithi115 bash[46934]: audit 2022-04-22T23:06:21.212272+0000 mon.a (mon.0) 16046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:22.957Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:23 smithi121 bash[36902]: cluster 2022-04-22T23:06:22.729218+0000 mgr.y (mgr.34107) 11595 : cluster [DBG] pgmap v12148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:23 smithi121 bash[36902]: audit 2022-04-22T23:06:22.844409+0000 mon.a (mon.0) 16047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:23.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:23 smithi115 bash[46934]: cluster 2022-04-22T23:06:22.729218+0000 mgr.y (mgr.34107) 11595 : cluster [DBG] pgmap v12148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:23 smithi115 bash[46934]: audit 2022-04-22T23:06:22.844409+0000 mon.a (mon.0) 16047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:23 smithi115 bash[45495]: cluster 2022-04-22T23:06:22.729218+0000 mgr.y (mgr.34107) 11595 : cluster [DBG] pgmap v12148: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:23 smithi115 bash[45495]: audit 2022-04-22T23:06:22.844409+0000 mon.a (mon.0) 16047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:24.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:24 smithi121 bash[36902]: audit 2022-04-22T23:06:24.442631+0000 mon.a (mon.0) 16048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:24.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:24 smithi115 bash[45495]: audit 2022-04-22T23:06:24.442631+0000 mon.a (mon.0) 16048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:24 smithi115 bash[46934]: audit 2022-04-22T23:06:24.442631+0000 mon.a (mon.0) 16048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:24.810Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:25.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:25 smithi121 bash[36902]: cluster 2022-04-22T23:06:24.730308+0000 mgr.y (mgr.34107) 11596 : cluster [DBG] pgmap v12149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:25 smithi115 bash[45495]: cluster 2022-04-22T23:06:24.730308+0000 mgr.y (mgr.34107) 11596 : cluster [DBG] pgmap v12149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:25 smithi115 bash[46934]: cluster 2022-04-22T23:06:24.730308+0000 mgr.y (mgr.34107) 11596 : cluster [DBG] pgmap v12149: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:26.703 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:06:26.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:26 smithi121 bash[36902]: audit 2022-04-22T23:06:26.215753+0000 mon.a (mon.0) 16049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:26.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:26 smithi115 bash[45495]: audit 2022-04-22T23:06:26.215753+0000 mon.a (mon.0) 16049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:26.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:26 smithi115 bash[46934]: audit 2022-04-22T23:06:26.215753+0000 mon.a (mon.0) 16049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:27.046 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6s ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:06:27.047 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6s ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:06:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6s ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:06:27.049 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6s ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:06:27.450 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:06:27.450 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:06:27.450 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:06:27.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:27.452 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:06:27.452 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:06:27.452 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:06:27.452 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:06:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[46934]: audit 2022-04-22T23:06:26.684594+0000 mgr.y (mgr.34107) 11597 : audit [DBG] from='client.47349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[46934]: cluster 2022-04-22T23:06:26.731435+0000 mgr.y (mgr.34107) 11598 : cluster [DBG] pgmap v12150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[46934]: audit 2022-04-22T23:06:27.022605+0000 mgr.y (mgr.34107) 11599 : audit [DBG] from='client.57295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[46934]: audit 2022-04-22T23:06:27.446611+0000 mon.c (mon.1) 391 : audit [DBG] from='client.? 172.21.15.115:0/405711387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:06:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[45495]: audit 2022-04-22T23:06:26.684594+0000 mgr.y (mgr.34107) 11597 : audit [DBG] from='client.47349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[45495]: cluster 2022-04-22T23:06:26.731435+0000 mgr.y (mgr.34107) 11598 : cluster [DBG] pgmap v12150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[45495]: audit 2022-04-22T23:06:27.022605+0000 mgr.y (mgr.34107) 11599 : audit [DBG] from='client.57295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:27 smithi115 bash[45495]: audit 2022-04-22T23:06:27.446611+0000 mon.c (mon.1) 391 : audit [DBG] from='client.? 172.21.15.115:0/405711387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:06:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:27 smithi121 bash[36902]: audit 2022-04-22T23:06:26.684594+0000 mgr.y (mgr.34107) 11597 : audit [DBG] from='client.47349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:27 smithi121 bash[36902]: cluster 2022-04-22T23:06:26.731435+0000 mgr.y (mgr.34107) 11598 : cluster [DBG] pgmap v12150: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:27 smithi121 bash[36902]: audit 2022-04-22T23:06:27.022605+0000 mgr.y (mgr.34107) 11599 : audit [DBG] from='client.57295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:27 smithi121 bash[36902]: audit 2022-04-22T23:06:27.446611+0000 mon.c (mon.1) 391 : audit [DBG] from='client.? 172.21.15.115:0/405711387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:06:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:28 smithi115 bash[45495]: audit 2022-04-22T23:06:27.847869+0000 mon.a (mon.0) 16050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:28 smithi115 bash[46934]: audit 2022-04-22T23:06:27.847869+0000 mon.a (mon.0) 16050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:28 smithi121 bash[36902]: audit 2022-04-22T23:06:27.847869+0000 mon.a (mon.0) 16050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:29 smithi115 bash[45495]: cluster 2022-04-22T23:06:28.732559+0000 mgr.y (mgr.34107) 11600 : cluster [DBG] pgmap v12151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:29 smithi115 bash[45495]: audit 2022-04-22T23:06:29.446515+0000 mon.a (mon.0) 16051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:29 smithi115 bash[46934]: cluster 2022-04-22T23:06:28.732559+0000 mgr.y (mgr.34107) 11600 : cluster [DBG] pgmap v12151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:29 smithi115 bash[46934]: audit 2022-04-22T23:06:29.446515+0000 mon.a (mon.0) 16051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:29 smithi121 bash[36902]: cluster 2022-04-22T23:06:28.732559+0000 mgr.y (mgr.34107) 11600 : cluster [DBG] pgmap v12151: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:29 smithi121 bash[36902]: audit 2022-04-22T23:06:29.446515+0000 mon.a (mon.0) 16051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:31 smithi121 bash[36902]: cluster 2022-04-22T23:06:30.733787+0000 mgr.y (mgr.34107) 11601 : cluster [DBG] pgmap v12152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:31 smithi121 bash[36902]: audit 2022-04-22T23:06:31.218659+0000 mon.a (mon.0) 16052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:31 smithi115 bash[45495]: cluster 2022-04-22T23:06:30.733787+0000 mgr.y (mgr.34107) 11601 : cluster [DBG] pgmap v12152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:31 smithi115 bash[45495]: audit 2022-04-22T23:06:31.218659+0000 mon.a (mon.0) 16052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:31 smithi115 bash[46934]: cluster 2022-04-22T23:06:30.733787+0000 mgr.y (mgr.34107) 11601 : cluster [DBG] pgmap v12152: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:31 smithi115 bash[46934]: audit 2022-04-22T23:06:31.218659+0000 mon.a (mon.0) 16052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:32.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:33 smithi121 bash[36902]: cluster 2022-04-22T23:06:32.734892+0000 mgr.y (mgr.34107) 11602 : cluster [DBG] pgmap v12153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:33 smithi121 bash[36902]: audit 2022-04-22T23:06:32.850841+0000 mon.a (mon.0) 16053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:33 smithi115 bash[45495]: cluster 2022-04-22T23:06:32.734892+0000 mgr.y (mgr.34107) 11602 : cluster [DBG] pgmap v12153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:33 smithi115 bash[45495]: audit 2022-04-22T23:06:32.850841+0000 mon.a (mon.0) 16053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:33 smithi115 bash[46934]: cluster 2022-04-22T23:06:32.734892+0000 mgr.y (mgr.34107) 11602 : cluster [DBG] pgmap v12153: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:33 smithi115 bash[46934]: audit 2022-04-22T23:06:32.850841+0000 mon.a (mon.0) 16053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:34 smithi121 bash[36902]: audit 2022-04-22T23:06:34.450227+0000 mon.a (mon.0) 16054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:34 smithi115 bash[45495]: audit 2022-04-22T23:06:34.450227+0000 mon.a (mon.0) 16054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:34 smithi115 bash[46934]: audit 2022-04-22T23:06:34.450227+0000 mon.a (mon.0) 16054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:34.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:35 smithi115 bash[45495]: cluster 2022-04-22T23:06:34.735945+0000 mgr.y (mgr.34107) 11603 : cluster [DBG] pgmap v12154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:35 smithi115 bash[46934]: cluster 2022-04-22T23:06:34.735945+0000 mgr.y (mgr.34107) 11603 : cluster [DBG] pgmap v12154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:35 smithi121 bash[36902]: cluster 2022-04-22T23:06:34.735945+0000 mgr.y (mgr.34107) 11603 : cluster [DBG] pgmap v12154: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:36 smithi115 bash[45495]: audit 2022-04-22T23:06:36.221925+0000 mon.a (mon.0) 16055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:36 smithi115 bash[46934]: audit 2022-04-22T23:06:36.221925+0000 mon.a (mon.0) 16055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:36 smithi121 bash[36902]: audit 2022-04-22T23:06:36.221925+0000 mon.a (mon.0) 16055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:37 smithi115 bash[46934]: cluster 2022-04-22T23:06:36.737021+0000 mgr.y (mgr.34107) 11604 : cluster [DBG] pgmap v12155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:37 smithi115 bash[45495]: cluster 2022-04-22T23:06:36.737021+0000 mgr.y (mgr.34107) 11604 : cluster [DBG] pgmap v12155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:37 smithi121 bash[36902]: cluster 2022-04-22T23:06:36.737021+0000 mgr.y (mgr.34107) 11604 : cluster [DBG] pgmap v12155: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:38 smithi115 bash[46934]: audit 2022-04-22T23:06:37.853927+0000 mon.a (mon.0) 16056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:38 smithi115 bash[45495]: audit 2022-04-22T23:06:37.853927+0000 mon.a (mon.0) 16056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:38 smithi121 bash[36902]: audit 2022-04-22T23:06:37.853927+0000 mon.a (mon.0) 16056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:39 smithi115 bash[45495]: cluster 2022-04-22T23:06:38.738273+0000 mgr.y (mgr.34107) 11605 : cluster [DBG] pgmap v12156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:39 smithi115 bash[45495]: audit 2022-04-22T23:06:39.454188+0000 mon.a (mon.0) 16057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:39 smithi115 bash[46934]: cluster 2022-04-22T23:06:38.738273+0000 mgr.y (mgr.34107) 11605 : cluster [DBG] pgmap v12156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:39 smithi115 bash[46934]: audit 2022-04-22T23:06:39.454188+0000 mon.a (mon.0) 16057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:39 smithi121 bash[36902]: cluster 2022-04-22T23:06:38.738273+0000 mgr.y (mgr.34107) 11605 : cluster [DBG] pgmap v12156: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:39 smithi121 bash[36902]: audit 2022-04-22T23:06:39.454188+0000 mon.a (mon.0) 16057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:41 smithi121 bash[36902]: cluster 2022-04-22T23:06:40.739305+0000 mgr.y (mgr.34107) 11606 : cluster [DBG] pgmap v12157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:41 smithi121 bash[36902]: audit 2022-04-22T23:06:41.224714+0000 mon.a (mon.0) 16058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:41 smithi115 bash[45495]: cluster 2022-04-22T23:06:40.739305+0000 mgr.y (mgr.34107) 11606 : cluster [DBG] pgmap v12157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:41 smithi115 bash[45495]: audit 2022-04-22T23:06:41.224714+0000 mon.a (mon.0) 16058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:41 smithi115 bash[46934]: cluster 2022-04-22T23:06:40.739305+0000 mgr.y (mgr.34107) 11606 : cluster [DBG] pgmap v12157: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:41 smithi115 bash[46934]: audit 2022-04-22T23:06:41.224714+0000 mon.a (mon.0) 16058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:42.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:43 smithi121 bash[36902]: cluster 2022-04-22T23:06:42.740511+0000 mgr.y (mgr.34107) 11607 : cluster [DBG] pgmap v12158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:43 smithi121 bash[36902]: audit 2022-04-22T23:06:42.856866+0000 mon.a (mon.0) 16059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:43 smithi115 bash[45495]: cluster 2022-04-22T23:06:42.740511+0000 mgr.y (mgr.34107) 11607 : cluster [DBG] pgmap v12158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:43 smithi115 bash[45495]: audit 2022-04-22T23:06:42.856866+0000 mon.a (mon.0) 16059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:43 smithi115 bash[46934]: cluster 2022-04-22T23:06:42.740511+0000 mgr.y (mgr.34107) 11607 : cluster [DBG] pgmap v12158: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:43 smithi115 bash[46934]: audit 2022-04-22T23:06:42.856866+0000 mon.a (mon.0) 16059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:44.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:44 smithi115 bash[45495]: audit 2022-04-22T23:06:44.457854+0000 mon.a (mon.0) 16060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:44 smithi115 bash[46934]: audit 2022-04-22T23:06:44.457854+0000 mon.a (mon.0) 16060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:44 smithi121 bash[36902]: audit 2022-04-22T23:06:44.457854+0000 mon.a (mon.0) 16060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:44.812Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:45 smithi115 bash[45495]: cluster 2022-04-22T23:06:44.741758+0000 mgr.y (mgr.34107) 11608 : cluster [DBG] pgmap v12159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:45 smithi115 bash[46934]: cluster 2022-04-22T23:06:44.741758+0000 mgr.y (mgr.34107) 11608 : cluster [DBG] pgmap v12159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:45 smithi121 bash[36902]: cluster 2022-04-22T23:06:44.741758+0000 mgr.y (mgr.34107) 11608 : cluster [DBG] pgmap v12159: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:46 smithi115 bash[45495]: audit 2022-04-22T23:06:46.227663+0000 mon.a (mon.0) 16061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:46 smithi115 bash[46934]: audit 2022-04-22T23:06:46.227663+0000 mon.a (mon.0) 16061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:46 smithi121 bash[36902]: audit 2022-04-22T23:06:46.227663+0000 mon.a (mon.0) 16061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:47 smithi115 bash[45495]: cluster 2022-04-22T23:06:46.742987+0000 mgr.y (mgr.34107) 11609 : cluster [DBG] pgmap v12160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:47 smithi115 bash[46934]: cluster 2022-04-22T23:06:46.742987+0000 mgr.y (mgr.34107) 11609 : cluster [DBG] pgmap v12160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:47 smithi121 bash[36902]: cluster 2022-04-22T23:06:46.742987+0000 mgr.y (mgr.34107) 11609 : cluster [DBG] pgmap v12160: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:48 smithi115 bash[45495]: audit 2022-04-22T23:06:47.860083+0000 mon.a (mon.0) 16062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:48 smithi115 bash[46934]: audit 2022-04-22T23:06:47.860083+0000 mon.a (mon.0) 16062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:48 smithi121 bash[36902]: audit 2022-04-22T23:06:47.860083+0000 mon.a (mon.0) 16062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:49 smithi121 bash[36902]: cluster 2022-04-22T23:06:48.744187+0000 mgr.y (mgr.34107) 11610 : cluster [DBG] pgmap v12161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:49 smithi121 bash[36902]: audit 2022-04-22T23:06:49.461439+0000 mon.a (mon.0) 16063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:49 smithi115 bash[45495]: cluster 2022-04-22T23:06:48.744187+0000 mgr.y (mgr.34107) 11610 : cluster [DBG] pgmap v12161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:49 smithi115 bash[45495]: audit 2022-04-22T23:06:49.461439+0000 mon.a (mon.0) 16063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:49 smithi115 bash[46934]: cluster 2022-04-22T23:06:48.744187+0000 mgr.y (mgr.34107) 11610 : cluster [DBG] pgmap v12161: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:49 smithi115 bash[46934]: audit 2022-04-22T23:06:49.461439+0000 mon.a (mon.0) 16063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:51 smithi121 bash[36902]: cluster 2022-04-22T23:06:50.744964+0000 mgr.y (mgr.34107) 11611 : cluster [DBG] pgmap v12162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:51 smithi121 bash[36902]: audit 2022-04-22T23:06:51.230748+0000 mon.a (mon.0) 16064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:51 smithi115 bash[45495]: cluster 2022-04-22T23:06:50.744964+0000 mgr.y (mgr.34107) 11611 : cluster [DBG] pgmap v12162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:51 smithi115 bash[45495]: audit 2022-04-22T23:06:51.230748+0000 mon.a (mon.0) 16064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:51 smithi115 bash[46934]: cluster 2022-04-22T23:06:50.744964+0000 mgr.y (mgr.34107) 11611 : cluster [DBG] pgmap v12162: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:51 smithi115 bash[46934]: audit 2022-04-22T23:06:51.230748+0000 mon.a (mon.0) 16064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:52.959Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:53 smithi121 bash[36902]: cluster 2022-04-22T23:06:52.745948+0000 mgr.y (mgr.34107) 11612 : cluster [DBG] pgmap v12163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:53 smithi121 bash[36902]: audit 2022-04-22T23:06:52.863448+0000 mon.a (mon.0) 16065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:53 smithi115 bash[45495]: cluster 2022-04-22T23:06:52.745948+0000 mgr.y (mgr.34107) 11612 : cluster [DBG] pgmap v12163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:53 smithi115 bash[45495]: audit 2022-04-22T23:06:52.863448+0000 mon.a (mon.0) 16065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:53 smithi115 bash[46934]: cluster 2022-04-22T23:06:52.745948+0000 mgr.y (mgr.34107) 11612 : cluster [DBG] pgmap v12163: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:53 smithi115 bash[46934]: audit 2022-04-22T23:06:52.863448+0000 mon.a (mon.0) 16065 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:54.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:54 smithi115 bash[45495]: audit 2022-04-22T23:06:54.465279+0000 mon.a (mon.0) 16066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:54 smithi115 bash[46934]: audit 2022-04-22T23:06:54.465279+0000 mon.a (mon.0) 16066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:54 smithi121 bash[36902]: audit 2022-04-22T23:06:54.465279+0000 mon.a (mon.0) 16066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:06:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:06:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:06:54.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:06:55.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:55 smithi115 bash[46934]: cluster 2022-04-22T23:06:54.747026+0000 mgr.y (mgr.34107) 11613 : cluster [DBG] pgmap v12164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:55 smithi115 bash[45495]: cluster 2022-04-22T23:06:54.747026+0000 mgr.y (mgr.34107) 11613 : cluster [DBG] pgmap v12164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:55 smithi121 bash[36902]: cluster 2022-04-22T23:06:54.747026+0000 mgr.y (mgr.34107) 11613 : cluster [DBG] pgmap v12164: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:56 smithi115 bash[45495]: audit 2022-04-22T23:06:56.233708+0000 mon.a (mon.0) 16067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:56 smithi115 bash[46934]: audit 2022-04-22T23:06:56.233708+0000 mon.a (mon.0) 16067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:57.058 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:56 smithi121 bash[36902]: audit 2022-04-22T23:06:56.233708+0000 mon.a (mon.0) 16067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:06:57.822 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:06:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:57 smithi121 bash[36902]: cluster 2022-04-22T23:06:56.748148+0000 mgr.y (mgr.34107) 11614 : cluster [DBG] pgmap v12165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:58.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:57 smithi115 bash[46934]: cluster 2022-04-22T23:06:56.748148+0000 mgr.y (mgr.34107) 11614 : cluster [DBG] pgmap v12165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:57 smithi115 bash[45495]: cluster 2022-04-22T23:06:56.748148+0000 mgr.y (mgr.34107) 11614 : cluster [DBG] pgmap v12165: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:06:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:06:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:06:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 37s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:06:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:06:58.171 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 37s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 37s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 5h 453M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 37s ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 309M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:06:58.172 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 11.0M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:06:58.173 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 37s ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:06:58.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:06:58.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 800M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:06:58.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 542M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:06:58.173 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 504M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:06:58.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 37s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:06:58.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 37s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:06:58.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 37s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:06:58.174 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 37s ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:06:58.174 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 37s ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:06:58.573 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:06:58.574 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:06:58.575 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:06:58.575 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:06:58.575 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:06:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:58 smithi121 bash[36902]: audit 2022-04-22T23:06:57.802277+0000 mgr.y (mgr.34107) 11615 : audit [DBG] from='client.57307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:58 smithi121 bash[36902]: audit 2022-04-22T23:06:57.866612+0000 mon.a (mon.0) 16068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:58 smithi115 bash[45495]: audit 2022-04-22T23:06:57.802277+0000 mgr.y (mgr.34107) 11615 : audit [DBG] from='client.57307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:58 smithi115 bash[45495]: audit 2022-04-22T23:06:57.866612+0000 mon.a (mon.0) 16068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:06:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:58 smithi115 bash[46934]: audit 2022-04-22T23:06:57.802277+0000 mgr.y (mgr.34107) 11615 : audit [DBG] from='client.57307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:06:59.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:58 smithi115 bash[46934]: audit 2022-04-22T23:06:57.866612+0000 mon.a (mon.0) 16068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:59 smithi121 bash[36902]: audit 2022-04-22T23:06:58.148093+0000 mgr.y (mgr.34107) 11616 : audit [DBG] from='client.47373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:59 smithi121 bash[36902]: audit 2022-04-22T23:06:58.569007+0000 mon.a (mon.0) 16069 : audit [DBG] from='client.? 172.21.15.115:0/3553230735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:59 smithi121 bash[36902]: cluster 2022-04-22T23:06:58.749324+0000 mgr.y (mgr.34107) 11617 : cluster [DBG] pgmap v12166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:06:59 smithi121 bash[36902]: audit 2022-04-22T23:06:59.468953+0000 mon.a (mon.0) 16070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[45495]: audit 2022-04-22T23:06:58.148093+0000 mgr.y (mgr.34107) 11616 : audit [DBG] from='client.47373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[45495]: audit 2022-04-22T23:06:58.569007+0000 mon.a (mon.0) 16069 : audit [DBG] from='client.? 172.21.15.115:0/3553230735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[45495]: cluster 2022-04-22T23:06:58.749324+0000 mgr.y (mgr.34107) 11617 : cluster [DBG] pgmap v12166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[45495]: audit 2022-04-22T23:06:59.468953+0000 mon.a (mon.0) 16070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[46934]: audit 2022-04-22T23:06:58.148093+0000 mgr.y (mgr.34107) 11616 : audit [DBG] from='client.47373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[46934]: audit 2022-04-22T23:06:58.569007+0000 mon.a (mon.0) 16069 : audit [DBG] from='client.? 172.21.15.115:0/3553230735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[46934]: cluster 2022-04-22T23:06:58.749324+0000 mgr.y (mgr.34107) 11617 : cluster [DBG] pgmap v12166: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:06:59 smithi115 bash[46934]: audit 2022-04-22T23:06:59.468953+0000 mon.a (mon.0) 16070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:01 smithi121 bash[36902]: cluster 2022-04-22T23:07:00.750378+0000 mgr.y (mgr.34107) 11618 : cluster [DBG] pgmap v12167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:01 smithi121 bash[36902]: audit 2022-04-22T23:07:01.236835+0000 mon.a (mon.0) 16071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:01 smithi115 bash[45495]: cluster 2022-04-22T23:07:00.750378+0000 mgr.y (mgr.34107) 11618 : cluster [DBG] pgmap v12167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:01 smithi115 bash[45495]: audit 2022-04-22T23:07:01.236835+0000 mon.a (mon.0) 16071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:01 smithi115 bash[46934]: cluster 2022-04-22T23:07:00.750378+0000 mgr.y (mgr.34107) 11618 : cluster [DBG] pgmap v12167: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:01 smithi115 bash[46934]: audit 2022-04-22T23:07:01.236835+0000 mon.a (mon.0) 16071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:02.960Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:03 smithi121 bash[36902]: cluster 2022-04-22T23:07:02.751496+0000 mgr.y (mgr.34107) 11619 : cluster [DBG] pgmap v12168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:03 smithi121 bash[36902]: audit 2022-04-22T23:07:02.869436+0000 mon.a (mon.0) 16072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:03 smithi115 bash[45495]: cluster 2022-04-22T23:07:02.751496+0000 mgr.y (mgr.34107) 11619 : cluster [DBG] pgmap v12168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:03 smithi115 bash[45495]: audit 2022-04-22T23:07:02.869436+0000 mon.a (mon.0) 16072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:03 smithi115 bash[46934]: cluster 2022-04-22T23:07:02.751496+0000 mgr.y (mgr.34107) 11619 : cluster [DBG] pgmap v12168: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:03 smithi115 bash[46934]: audit 2022-04-22T23:07:02.869436+0000 mon.a (mon.0) 16072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:04 smithi115 bash[45495]: audit 2022-04-22T23:07:04.472562+0000 mon.a (mon.0) 16073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:04.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:04 smithi115 bash[46934]: audit 2022-04-22T23:07:04.472562+0000 mon.a (mon.0) 16073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:04 smithi121 bash[36902]: audit 2022-04-22T23:07:04.472562+0000 mon.a (mon.0) 16073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:04.814Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:05.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:05 smithi115 bash[45495]: cluster 2022-04-22T23:07:04.752559+0000 mgr.y (mgr.34107) 11620 : cluster [DBG] pgmap v12169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:05 smithi115 bash[46934]: cluster 2022-04-22T23:07:04.752559+0000 mgr.y (mgr.34107) 11620 : cluster [DBG] pgmap v12169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:05 smithi121 bash[36902]: cluster 2022-04-22T23:07:04.752559+0000 mgr.y (mgr.34107) 11620 : cluster [DBG] pgmap v12169: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:06 smithi121 bash[36902]: audit 2022-04-22T23:07:06.239822+0000 mon.a (mon.0) 16074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:06 smithi115 bash[45495]: audit 2022-04-22T23:07:06.239822+0000 mon.a (mon.0) 16074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:06 smithi115 bash[46934]: audit 2022-04-22T23:07:06.239822+0000 mon.a (mon.0) 16074 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:07 smithi121 bash[36902]: cluster 2022-04-22T23:07:06.753610+0000 mgr.y (mgr.34107) 11621 : cluster [DBG] pgmap v12170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:07 smithi115 bash[45495]: cluster 2022-04-22T23:07:06.753610+0000 mgr.y (mgr.34107) 11621 : cluster [DBG] pgmap v12170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:07 smithi115 bash[46934]: cluster 2022-04-22T23:07:06.753610+0000 mgr.y (mgr.34107) 11621 : cluster [DBG] pgmap v12170: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:08 smithi121 bash[36902]: audit 2022-04-22T23:07:07.872636+0000 mon.a (mon.0) 16075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:09.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:08 smithi115 bash[45495]: audit 2022-04-22T23:07:07.872636+0000 mon.a (mon.0) 16075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:08 smithi115 bash[46934]: audit 2022-04-22T23:07:07.872636+0000 mon.a (mon.0) 16075 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:09 smithi121 bash[36902]: cluster 2022-04-22T23:07:08.754891+0000 mgr.y (mgr.34107) 11622 : cluster [DBG] pgmap v12171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:09 smithi121 bash[36902]: audit 2022-04-22T23:07:09.476247+0000 mon.a (mon.0) 16076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:09 smithi115 bash[45495]: cluster 2022-04-22T23:07:08.754891+0000 mgr.y (mgr.34107) 11622 : cluster [DBG] pgmap v12171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:09 smithi115 bash[45495]: audit 2022-04-22T23:07:09.476247+0000 mon.a (mon.0) 16076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:09 smithi115 bash[46934]: cluster 2022-04-22T23:07:08.754891+0000 mgr.y (mgr.34107) 11622 : cluster [DBG] pgmap v12171: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:09 smithi115 bash[46934]: audit 2022-04-22T23:07:09.476247+0000 mon.a (mon.0) 16076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:11 smithi121 bash[36902]: cluster 2022-04-22T23:07:10.756033+0000 mgr.y (mgr.34107) 11623 : cluster [DBG] pgmap v12172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:11 smithi121 bash[36902]: audit 2022-04-22T23:07:11.242668+0000 mon.a (mon.0) 16077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:12.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:11 smithi115 bash[45495]: cluster 2022-04-22T23:07:10.756033+0000 mgr.y (mgr.34107) 11623 : cluster [DBG] pgmap v12172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:11 smithi115 bash[45495]: audit 2022-04-22T23:07:11.242668+0000 mon.a (mon.0) 16077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:11 smithi115 bash[46934]: cluster 2022-04-22T23:07:10.756033+0000 mgr.y (mgr.34107) 11623 : cluster [DBG] pgmap v12172: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:11 smithi115 bash[46934]: audit 2022-04-22T23:07:11.242668+0000 mon.a (mon.0) 16077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:12.963Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:13 smithi121 bash[36902]: cluster 2022-04-22T23:07:12.757059+0000 mgr.y (mgr.34107) 11624 : cluster [DBG] pgmap v12173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:13 smithi121 bash[36902]: audit 2022-04-22T23:07:12.875816+0000 mon.a (mon.0) 16078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:13 smithi115 bash[45495]: cluster 2022-04-22T23:07:12.757059+0000 mgr.y (mgr.34107) 11624 : cluster [DBG] pgmap v12173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:13 smithi115 bash[45495]: audit 2022-04-22T23:07:12.875816+0000 mon.a (mon.0) 16078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:13 smithi115 bash[46934]: cluster 2022-04-22T23:07:12.757059+0000 mgr.y (mgr.34107) 11624 : cluster [DBG] pgmap v12173: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:13 smithi115 bash[46934]: audit 2022-04-22T23:07:12.875816+0000 mon.a (mon.0) 16078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:14.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:14 smithi115 bash[46934]: audit 2022-04-22T23:07:14.479868+0000 mon.a (mon.0) 16079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:14 smithi115 bash[45495]: audit 2022-04-22T23:07:14.479868+0000 mon.a (mon.0) 16079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:14 smithi121 bash[36902]: audit 2022-04-22T23:07:14.479868+0000 mon.a (mon.0) 16079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:14.816Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:15 smithi121 bash[36902]: cluster 2022-04-22T23:07:14.758163+0000 mgr.y (mgr.34107) 11625 : cluster [DBG] pgmap v12174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:15 smithi115 bash[45495]: cluster 2022-04-22T23:07:14.758163+0000 mgr.y (mgr.34107) 11625 : cluster [DBG] pgmap v12174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:15 smithi115 bash[46934]: cluster 2022-04-22T23:07:14.758163+0000 mgr.y (mgr.34107) 11625 : cluster [DBG] pgmap v12174: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:16 smithi121 bash[36902]: audit 2022-04-22T23:07:16.245425+0000 mon.a (mon.0) 16080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:16 smithi115 bash[45495]: audit 2022-04-22T23:07:16.245425+0000 mon.a (mon.0) 16080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:16 smithi115 bash[46934]: audit 2022-04-22T23:07:16.245425+0000 mon.a (mon.0) 16080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:17 smithi121 bash[36902]: cluster 2022-04-22T23:07:16.759263+0000 mgr.y (mgr.34107) 11626 : cluster [DBG] pgmap v12175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:17 smithi121 bash[36902]: audit 2022-04-22T23:07:17.199467+0000 mon.a (mon.0) 16081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:07:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:17 smithi121 bash[36902]: audit 2022-04-22T23:07:17.199898+0000 mon.a (mon.0) 16082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:07:18.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[46934]: cluster 2022-04-22T23:07:16.759263+0000 mgr.y (mgr.34107) 11626 : cluster [DBG] pgmap v12175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[46934]: audit 2022-04-22T23:07:17.199467+0000 mon.a (mon.0) 16081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:07:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[46934]: audit 2022-04-22T23:07:17.199898+0000 mon.a (mon.0) 16082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:07:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[45495]: cluster 2022-04-22T23:07:16.759263+0000 mgr.y (mgr.34107) 11626 : cluster [DBG] pgmap v12175: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[45495]: audit 2022-04-22T23:07:17.199467+0000 mon.a (mon.0) 16081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:07:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:17 smithi115 bash[45495]: audit 2022-04-22T23:07:17.199898+0000 mon.a (mon.0) 16082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:07:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:18 smithi121 bash[36902]: audit 2022-04-22T23:07:17.878806+0000 mon.a (mon.0) 16083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:18 smithi115 bash[45495]: audit 2022-04-22T23:07:17.878806+0000 mon.a (mon.0) 16083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:18 smithi115 bash[46934]: audit 2022-04-22T23:07:17.878806+0000 mon.a (mon.0) 16083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:19 smithi121 bash[36902]: cluster 2022-04-22T23:07:18.760347+0000 mgr.y (mgr.34107) 11627 : cluster [DBG] pgmap v12176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:19 smithi121 bash[36902]: audit 2022-04-22T23:07:19.483670+0000 mon.a (mon.0) 16084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:19 smithi115 bash[45495]: cluster 2022-04-22T23:07:18.760347+0000 mgr.y (mgr.34107) 11627 : cluster [DBG] pgmap v12176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:19 smithi115 bash[45495]: audit 2022-04-22T23:07:19.483670+0000 mon.a (mon.0) 16084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:19 smithi115 bash[46934]: cluster 2022-04-22T23:07:18.760347+0000 mgr.y (mgr.34107) 11627 : cluster [DBG] pgmap v12176: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:19 smithi115 bash[46934]: audit 2022-04-22T23:07:19.483670+0000 mon.a (mon.0) 16084 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:21.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[45495]: audit 2022-04-22T23:07:20.763736+0000 mon.a (mon.0) 16085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:07:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[45495]: audit 2022-04-22T23:07:20.766399+0000 mon.a (mon.0) 16086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:07:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[45495]: audit 2022-04-22T23:07:20.767939+0000 mon.a (mon.0) 16087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:07:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[46934]: audit 2022-04-22T23:07:20.763736+0000 mon.a (mon.0) 16085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:07:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[46934]: audit 2022-04-22T23:07:20.766399+0000 mon.a (mon.0) 16086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:07:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:20 smithi115 bash[46934]: audit 2022-04-22T23:07:20.767939+0000 mon.a (mon.0) 16087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:07:21.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:20 smithi121 bash[36902]: audit 2022-04-22T23:07:20.763736+0000 mon.a (mon.0) 16085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:07:21.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:20 smithi121 bash[36902]: audit 2022-04-22T23:07:20.766399+0000 mon.a (mon.0) 16086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:07:21.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:20 smithi121 bash[36902]: audit 2022-04-22T23:07:20.767939+0000 mon.a (mon.0) 16087 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:07:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:22 smithi121 bash[36902]: cluster 2022-04-22T23:07:20.761399+0000 mgr.y (mgr.34107) 11628 : cluster [DBG] pgmap v12177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:22 smithi121 bash[36902]: audit 2022-04-22T23:07:21.109713+0000 mon.a (mon.0) 16088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:22 smithi121 bash[36902]: audit 2022-04-22T23:07:21.117323+0000 mon.a (mon.0) 16089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:22 smithi121 bash[36902]: audit 2022-04-22T23:07:21.248337+0000 mon.a (mon.0) 16090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[45495]: cluster 2022-04-22T23:07:20.761399+0000 mgr.y (mgr.34107) 11628 : cluster [DBG] pgmap v12177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[45495]: audit 2022-04-22T23:07:21.109713+0000 mon.a (mon.0) 16088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[45495]: audit 2022-04-22T23:07:21.117323+0000 mon.a (mon.0) 16089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[45495]: audit 2022-04-22T23:07:21.248337+0000 mon.a (mon.0) 16090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[46934]: cluster 2022-04-22T23:07:20.761399+0000 mgr.y (mgr.34107) 11628 : cluster [DBG] pgmap v12177: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[46934]: audit 2022-04-22T23:07:21.109713+0000 mon.a (mon.0) 16088 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[46934]: audit 2022-04-22T23:07:21.117323+0000 mon.a (mon.0) 16089 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[46934]: audit 2022-04-22T23:07:21.248337+0000 mon.a (mon.0) 16090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:22.963Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:23 smithi121 bash[36902]: cluster 2022-04-22T23:07:22.762528+0000 mgr.y (mgr.34107) 11629 : cluster [DBG] pgmap v12178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:23 smithi121 bash[36902]: audit 2022-04-22T23:07:22.881751+0000 mon.a (mon.0) 16091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:23 smithi115 bash[45495]: cluster 2022-04-22T23:07:22.762528+0000 mgr.y (mgr.34107) 11629 : cluster [DBG] pgmap v12178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:23 smithi115 bash[45495]: audit 2022-04-22T23:07:22.881751+0000 mon.a (mon.0) 16091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:23 smithi115 bash[46934]: cluster 2022-04-22T23:07:22.762528+0000 mgr.y (mgr.34107) 11629 : cluster [DBG] pgmap v12178: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:23 smithi115 bash[46934]: audit 2022-04-22T23:07:22.881751+0000 mon.a (mon.0) 16091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:24.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:24 smithi115 bash[45495]: audit 2022-04-22T23:07:24.487530+0000 mon.a (mon.0) 16092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:24.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:24 smithi115 bash[46934]: audit 2022-04-22T23:07:24.487530+0000 mon.a (mon.0) 16092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:24 smithi121 bash[36902]: audit 2022-04-22T23:07:24.487530+0000 mon.a (mon.0) 16092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:25.243 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:24.815Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:25 smithi121 bash[36902]: cluster 2022-04-22T23:07:24.763456+0000 mgr.y (mgr.34107) 11630 : cluster [DBG] pgmap v12179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:26.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:25 smithi121 bash[36902]: audit 2022-04-22T23:07:25.252488+0000 mon.a (mon.0) 16093 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:25 smithi115 bash[45495]: cluster 2022-04-22T23:07:24.763456+0000 mgr.y (mgr.34107) 11630 : cluster [DBG] pgmap v12179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:25 smithi115 bash[45495]: audit 2022-04-22T23:07:25.252488+0000 mon.a (mon.0) 16093 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:25 smithi115 bash[46934]: cluster 2022-04-22T23:07:24.763456+0000 mgr.y (mgr.34107) 11630 : cluster [DBG] pgmap v12179: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:25 smithi115 bash[46934]: audit 2022-04-22T23:07:25.252488+0000 mon.a (mon.0) 16093 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:26 smithi121 bash[36902]: audit 2022-04-22T23:07:25.602564+0000 mon.a (mon.0) 16094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:26 smithi121 bash[36902]: audit 2022-04-22T23:07:26.251142+0000 mon.a (mon.0) 16095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:26 smithi115 bash[45495]: audit 2022-04-22T23:07:25.602564+0000 mon.a (mon.0) 16094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:26 smithi115 bash[45495]: audit 2022-04-22T23:07:26.251142+0000 mon.a (mon.0) 16095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:26 smithi115 bash[46934]: audit 2022-04-22T23:07:25.602564+0000 mon.a (mon.0) 16094 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:07:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:26 smithi115 bash[46934]: audit 2022-04-22T23:07:26.251142+0000 mon.a (mon.0) 16095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:27 smithi121 bash[36902]: cluster 2022-04-22T23:07:26.764579+0000 mgr.y (mgr.34107) 11631 : cluster [DBG] pgmap v12180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:27 smithi115 bash[45495]: cluster 2022-04-22T23:07:26.764579+0000 mgr.y (mgr.34107) 11631 : cluster [DBG] pgmap v12180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:27 smithi115 bash[46934]: cluster 2022-04-22T23:07:26.764579+0000 mgr.y (mgr.34107) 11631 : cluster [DBG] pgmap v12180: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:28.947 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:07:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:28 smithi121 bash[36902]: audit 2022-04-22T23:07:27.884898+0000 mon.a (mon.0) 16096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:29.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:28 smithi115 bash[46934]: audit 2022-04-22T23:07:27.884898+0000 mon.a (mon.0) 16096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:28 smithi115 bash[45495]: audit 2022-04-22T23:07:27.884898+0000 mon.a (mon.0) 16096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:29.295 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4s ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 68s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 68s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 68s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4s ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 68s ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:07:29.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4s ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4s ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 68s ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4s ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 68s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 68s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 68s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:07:29.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 68s ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:07:29.298 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 68s ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:07:29.700 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:07:29.700 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:07:29.700 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:07:29.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:07:29.702 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:07:29.702 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:07:29.702 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:07:29.702 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:07:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:29 smithi121 bash[36902]: cluster 2022-04-22T23:07:28.765683+0000 mgr.y (mgr.34107) 11632 : cluster [DBG] pgmap v12181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:29 smithi121 bash[36902]: audit 2022-04-22T23:07:28.926554+0000 mgr.y (mgr.34107) 11633 : audit [DBG] from='client.47385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:29 smithi121 bash[36902]: audit 2022-04-22T23:07:29.490888+0000 mon.a (mon.0) 16097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[45495]: cluster 2022-04-22T23:07:28.765683+0000 mgr.y (mgr.34107) 11632 : cluster [DBG] pgmap v12181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[45495]: audit 2022-04-22T23:07:28.926554+0000 mgr.y (mgr.34107) 11633 : audit [DBG] from='client.47385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[45495]: audit 2022-04-22T23:07:29.490888+0000 mon.a (mon.0) 16097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[46934]: cluster 2022-04-22T23:07:28.765683+0000 mgr.y (mgr.34107) 11632 : cluster [DBG] pgmap v12181: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[46934]: audit 2022-04-22T23:07:28.926554+0000 mgr.y (mgr.34107) 11633 : audit [DBG] from='client.47385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:29 smithi115 bash[46934]: audit 2022-04-22T23:07:29.490888+0000 mon.a (mon.0) 16097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:31.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:30 smithi121 bash[36902]: audit 2022-04-22T23:07:29.271086+0000 mgr.y (mgr.34107) 11634 : audit [DBG] from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:31.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:30 smithi121 bash[36902]: audit 2022-04-22T23:07:29.696710+0000 mon.a (mon.0) 16098 : audit [DBG] from='client.? 172.21.15.115:0/3782234446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:30 smithi115 bash[45495]: audit 2022-04-22T23:07:29.271086+0000 mgr.y (mgr.34107) 11634 : audit [DBG] from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:31.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:30 smithi115 bash[45495]: audit 2022-04-22T23:07:29.696710+0000 mon.a (mon.0) 16098 : audit [DBG] from='client.? 172.21.15.115:0/3782234446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:30 smithi115 bash[46934]: audit 2022-04-22T23:07:29.271086+0000 mgr.y (mgr.34107) 11634 : audit [DBG] from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:07:31.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:30 smithi115 bash[46934]: audit 2022-04-22T23:07:29.696710+0000 mon.a (mon.0) 16098 : audit [DBG] from='client.? 172.21.15.115:0/3782234446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:07:32.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:31 smithi121 bash[36902]: cluster 2022-04-22T23:07:30.766846+0000 mgr.y (mgr.34107) 11635 : cluster [DBG] pgmap v12182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:32.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:31 smithi121 bash[36902]: audit 2022-04-22T23:07:31.254120+0000 mon.a (mon.0) 16099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:31 smithi115 bash[45495]: cluster 2022-04-22T23:07:30.766846+0000 mgr.y (mgr.34107) 11635 : cluster [DBG] pgmap v12182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:32.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:31 smithi115 bash[45495]: audit 2022-04-22T23:07:31.254120+0000 mon.a (mon.0) 16099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:31 smithi115 bash[46934]: cluster 2022-04-22T23:07:30.766846+0000 mgr.y (mgr.34107) 11635 : cluster [DBG] pgmap v12182: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:32.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:31 smithi115 bash[46934]: audit 2022-04-22T23:07:31.254120+0000 mon.a (mon.0) 16099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:32.964Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:33 smithi121 bash[36902]: cluster 2022-04-22T23:07:32.767925+0000 mgr.y (mgr.34107) 11636 : cluster [DBG] pgmap v12183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:33 smithi121 bash[36902]: audit 2022-04-22T23:07:32.888036+0000 mon.a (mon.0) 16100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:33 smithi115 bash[46934]: cluster 2022-04-22T23:07:32.767925+0000 mgr.y (mgr.34107) 11636 : cluster [DBG] pgmap v12183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:33 smithi115 bash[46934]: audit 2022-04-22T23:07:32.888036+0000 mon.a (mon.0) 16100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:33 smithi115 bash[45495]: cluster 2022-04-22T23:07:32.767925+0000 mgr.y (mgr.34107) 11636 : cluster [DBG] pgmap v12183: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:33 smithi115 bash[45495]: audit 2022-04-22T23:07:32.888036+0000 mon.a (mon.0) 16100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:34.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:34 smithi115 bash[45495]: audit 2022-04-22T23:07:34.494463+0000 mon.a (mon.0) 16101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:34.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:34 smithi115 bash[46934]: audit 2022-04-22T23:07:34.494463+0000 mon.a (mon.0) 16101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:34 smithi121 bash[36902]: audit 2022-04-22T23:07:34.494463+0000 mon.a (mon.0) 16101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:34.816Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:35 smithi121 bash[36902]: cluster 2022-04-22T23:07:34.768944+0000 mgr.y (mgr.34107) 11637 : cluster [DBG] pgmap v12184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:35 smithi115 bash[45495]: cluster 2022-04-22T23:07:34.768944+0000 mgr.y (mgr.34107) 11637 : cluster [DBG] pgmap v12184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:35 smithi115 bash[46934]: cluster 2022-04-22T23:07:34.768944+0000 mgr.y (mgr.34107) 11637 : cluster [DBG] pgmap v12184: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:36 smithi121 bash[36902]: audit 2022-04-22T23:07:36.257207+0000 mon.a (mon.0) 16102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:37.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:36 smithi115 bash[45495]: audit 2022-04-22T23:07:36.257207+0000 mon.a (mon.0) 16102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:36 smithi115 bash[46934]: audit 2022-04-22T23:07:36.257207+0000 mon.a (mon.0) 16102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:37 smithi121 bash[36902]: cluster 2022-04-22T23:07:36.769977+0000 mgr.y (mgr.34107) 11638 : cluster [DBG] pgmap v12185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:37 smithi115 bash[45495]: cluster 2022-04-22T23:07:36.769977+0000 mgr.y (mgr.34107) 11638 : cluster [DBG] pgmap v12185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:37 smithi115 bash[46934]: cluster 2022-04-22T23:07:36.769977+0000 mgr.y (mgr.34107) 11638 : cluster [DBG] pgmap v12185: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:38 smithi121 bash[36902]: audit 2022-04-22T23:07:37.891247+0000 mon.a (mon.0) 16103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:38 smithi115 bash[45495]: audit 2022-04-22T23:07:37.891247+0000 mon.a (mon.0) 16103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:38 smithi115 bash[46934]: audit 2022-04-22T23:07:37.891247+0000 mon.a (mon.0) 16103 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:39 smithi121 bash[36902]: cluster 2022-04-22T23:07:38.771237+0000 mgr.y (mgr.34107) 11639 : cluster [DBG] pgmap v12186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:39 smithi121 bash[36902]: audit 2022-04-22T23:07:39.498033+0000 mon.a (mon.0) 16104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:39 smithi115 bash[45495]: cluster 2022-04-22T23:07:38.771237+0000 mgr.y (mgr.34107) 11639 : cluster [DBG] pgmap v12186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:39 smithi115 bash[45495]: audit 2022-04-22T23:07:39.498033+0000 mon.a (mon.0) 16104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:39 smithi115 bash[46934]: cluster 2022-04-22T23:07:38.771237+0000 mgr.y (mgr.34107) 11639 : cluster [DBG] pgmap v12186: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:39 smithi115 bash[46934]: audit 2022-04-22T23:07:39.498033+0000 mon.a (mon.0) 16104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:41 smithi121 bash[36902]: cluster 2022-04-22T23:07:40.772372+0000 mgr.y (mgr.34107) 11640 : cluster [DBG] pgmap v12187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:41 smithi121 bash[36902]: audit 2022-04-22T23:07:41.260223+0000 mon.a (mon.0) 16105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:41 smithi115 bash[45495]: cluster 2022-04-22T23:07:40.772372+0000 mgr.y (mgr.34107) 11640 : cluster [DBG] pgmap v12187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:41 smithi115 bash[45495]: audit 2022-04-22T23:07:41.260223+0000 mon.a (mon.0) 16105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:41 smithi115 bash[46934]: cluster 2022-04-22T23:07:40.772372+0000 mgr.y (mgr.34107) 11640 : cluster [DBG] pgmap v12187: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:41 smithi115 bash[46934]: audit 2022-04-22T23:07:41.260223+0000 mon.a (mon.0) 16105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:42.964Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:43 smithi121 bash[36902]: cluster 2022-04-22T23:07:42.773453+0000 mgr.y (mgr.34107) 11641 : cluster [DBG] pgmap v12188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:43 smithi121 bash[36902]: audit 2022-04-22T23:07:42.894369+0000 mon.a (mon.0) 16106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:43 smithi115 bash[45495]: cluster 2022-04-22T23:07:42.773453+0000 mgr.y (mgr.34107) 11641 : cluster [DBG] pgmap v12188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:43 smithi115 bash[45495]: audit 2022-04-22T23:07:42.894369+0000 mon.a (mon.0) 16106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:43 smithi115 bash[46934]: cluster 2022-04-22T23:07:42.773453+0000 mgr.y (mgr.34107) 11641 : cluster [DBG] pgmap v12188: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:43 smithi115 bash[46934]: audit 2022-04-22T23:07:42.894369+0000 mon.a (mon.0) 16106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:44.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:44 smithi115 bash[45495]: audit 2022-04-22T23:07:44.501883+0000 mon.a (mon.0) 16107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:44.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:44 smithi115 bash[46934]: audit 2022-04-22T23:07:44.501883+0000 mon.a (mon.0) 16107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:44 smithi121 bash[36902]: audit 2022-04-22T23:07:44.501883+0000 mon.a (mon.0) 16107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:44.816Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:45 smithi121 bash[36902]: cluster 2022-04-22T23:07:44.774552+0000 mgr.y (mgr.34107) 11642 : cluster [DBG] pgmap v12189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:45 smithi115 bash[45495]: cluster 2022-04-22T23:07:44.774552+0000 mgr.y (mgr.34107) 11642 : cluster [DBG] pgmap v12189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:45 smithi115 bash[46934]: cluster 2022-04-22T23:07:44.774552+0000 mgr.y (mgr.34107) 11642 : cluster [DBG] pgmap v12189: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:46 smithi121 bash[36902]: audit 2022-04-22T23:07:46.263350+0000 mon.a (mon.0) 16108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:47.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:46 smithi115 bash[46934]: audit 2022-04-22T23:07:46.263350+0000 mon.a (mon.0) 16108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:46 smithi115 bash[45495]: audit 2022-04-22T23:07:46.263350+0000 mon.a (mon.0) 16108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:47 smithi121 bash[36902]: cluster 2022-04-22T23:07:46.775712+0000 mgr.y (mgr.34107) 11643 : cluster [DBG] pgmap v12190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:47 smithi115 bash[45495]: cluster 2022-04-22T23:07:46.775712+0000 mgr.y (mgr.34107) 11643 : cluster [DBG] pgmap v12190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:47 smithi115 bash[46934]: cluster 2022-04-22T23:07:46.775712+0000 mgr.y (mgr.34107) 11643 : cluster [DBG] pgmap v12190: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:48 smithi121 bash[36902]: audit 2022-04-22T23:07:47.897593+0000 mon.a (mon.0) 16109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:48 smithi115 bash[45495]: audit 2022-04-22T23:07:47.897593+0000 mon.a (mon.0) 16109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:48 smithi115 bash[46934]: audit 2022-04-22T23:07:47.897593+0000 mon.a (mon.0) 16109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:49 smithi121 bash[36902]: cluster 2022-04-22T23:07:48.776908+0000 mgr.y (mgr.34107) 11644 : cluster [DBG] pgmap v12191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:49 smithi121 bash[36902]: audit 2022-04-22T23:07:49.505932+0000 mon.a (mon.0) 16110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:49 smithi115 bash[46934]: cluster 2022-04-22T23:07:48.776908+0000 mgr.y (mgr.34107) 11644 : cluster [DBG] pgmap v12191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:49 smithi115 bash[46934]: audit 2022-04-22T23:07:49.505932+0000 mon.a (mon.0) 16110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:49 smithi115 bash[45495]: cluster 2022-04-22T23:07:48.776908+0000 mgr.y (mgr.34107) 11644 : cluster [DBG] pgmap v12191: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:50.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:49 smithi115 bash[45495]: audit 2022-04-22T23:07:49.505932+0000 mon.a (mon.0) 16110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:51 smithi121 bash[36902]: cluster 2022-04-22T23:07:50.778014+0000 mgr.y (mgr.34107) 11645 : cluster [DBG] pgmap v12192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:51 smithi121 bash[36902]: audit 2022-04-22T23:07:51.266323+0000 mon.a (mon.0) 16111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:51 smithi115 bash[45495]: cluster 2022-04-22T23:07:50.778014+0000 mgr.y (mgr.34107) 11645 : cluster [DBG] pgmap v12192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:51 smithi115 bash[45495]: audit 2022-04-22T23:07:51.266323+0000 mon.a (mon.0) 16111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:51 smithi115 bash[46934]: cluster 2022-04-22T23:07:50.778014+0000 mgr.y (mgr.34107) 11645 : cluster [DBG] pgmap v12192: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:51 smithi115 bash[46934]: audit 2022-04-22T23:07:51.266323+0000 mon.a (mon.0) 16111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:52.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:53 smithi121 bash[36902]: cluster 2022-04-22T23:07:52.779086+0000 mgr.y (mgr.34107) 11646 : cluster [DBG] pgmap v12193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:53 smithi121 bash[36902]: audit 2022-04-22T23:07:52.900682+0000 mon.a (mon.0) 16112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:53 smithi115 bash[45495]: cluster 2022-04-22T23:07:52.779086+0000 mgr.y (mgr.34107) 11646 : cluster [DBG] pgmap v12193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:53 smithi115 bash[45495]: audit 2022-04-22T23:07:52.900682+0000 mon.a (mon.0) 16112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:53 smithi115 bash[46934]: cluster 2022-04-22T23:07:52.779086+0000 mgr.y (mgr.34107) 11646 : cluster [DBG] pgmap v12193: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:53 smithi115 bash[46934]: audit 2022-04-22T23:07:52.900682+0000 mon.a (mon.0) 16112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:54.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:54 smithi115 bash[45495]: audit 2022-04-22T23:07:54.509389+0000 mon.a (mon.0) 16113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:54.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:54 smithi115 bash[46934]: audit 2022-04-22T23:07:54.509389+0000 mon.a (mon.0) 16113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:54 smithi121 bash[36902]: audit 2022-04-22T23:07:54.509389+0000 mon.a (mon.0) 16113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:07:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:07:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:07:54.818Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:07:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:55 smithi121 bash[36902]: cluster 2022-04-22T23:07:54.780261+0000 mgr.y (mgr.34107) 11647 : cluster [DBG] pgmap v12194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:55 smithi115 bash[45495]: cluster 2022-04-22T23:07:54.780261+0000 mgr.y (mgr.34107) 11647 : cluster [DBG] pgmap v12194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:55 smithi115 bash[46934]: cluster 2022-04-22T23:07:54.780261+0000 mgr.y (mgr.34107) 11647 : cluster [DBG] pgmap v12194: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:56 smithi121 bash[36902]: audit 2022-04-22T23:07:56.269299+0000 mon.a (mon.0) 16114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:56 smithi115 bash[45495]: audit 2022-04-22T23:07:56.269299+0000 mon.a (mon.0) 16114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:56 smithi115 bash[46934]: audit 2022-04-22T23:07:56.269299+0000 mon.a (mon.0) 16114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:07:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:57 smithi121 bash[36902]: cluster 2022-04-22T23:07:56.781399+0000 mgr.y (mgr.34107) 11648 : cluster [DBG] pgmap v12195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:57 smithi115 bash[45495]: cluster 2022-04-22T23:07:56.781399+0000 mgr.y (mgr.34107) 11648 : cluster [DBG] pgmap v12195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:57 smithi115 bash[46934]: cluster 2022-04-22T23:07:56.781399+0000 mgr.y (mgr.34107) 11648 : cluster [DBG] pgmap v12195: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:07:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:58 smithi121 bash[36902]: audit 2022-04-22T23:07:57.903903+0000 mon.a (mon.0) 16115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:58 smithi115 bash[45495]: audit 2022-04-22T23:07:57.903903+0000 mon.a (mon.0) 16115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:07:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:58 smithi115 bash[46934]: audit 2022-04-22T23:07:57.903903+0000 mon.a (mon.0) 16115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:59 smithi121 bash[36902]: cluster 2022-04-22T23:07:58.782534+0000 mgr.y (mgr.34107) 11649 : cluster [DBG] pgmap v12196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:07:59 smithi121 bash[36902]: audit 2022-04-22T23:07:59.513240+0000 mon.a (mon.0) 16116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:00.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:59 smithi115 bash[46934]: cluster 2022-04-22T23:07:58.782534+0000 mgr.y (mgr.34107) 11649 : cluster [DBG] pgmap v12196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:07:59 smithi115 bash[46934]: audit 2022-04-22T23:07:59.513240+0000 mon.a (mon.0) 16116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:59 smithi115 bash[45495]: cluster 2022-04-22T23:07:58.782534+0000 mgr.y (mgr.34107) 11649 : cluster [DBG] pgmap v12196: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:07:59 smithi115 bash[45495]: audit 2022-04-22T23:07:59.513240+0000 mon.a (mon.0) 16116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:00.063 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 35s ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 100s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 35s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 100s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 100s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 35s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:08:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 35s ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 100s ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 35s ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 35s ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 100s ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 35s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 35s ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 35s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 35s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 100s ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:08:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 100s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:08:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 100s ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:08:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 100s ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:08:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 100s ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:08:00.811 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:08:00.811 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:08:00.812 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:08:00.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:00.813 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:08:00.813 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:08:00.813 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:08:00.813 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:08:01.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:00 smithi121 bash[36902]: audit 2022-04-22T23:08:00.044288+0000 mgr.y (mgr.34107) 11650 : audit [DBG] from='client.57343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:00 smithi115 bash[45495]: audit 2022-04-22T23:08:00.044288+0000 mgr.y (mgr.34107) 11650 : audit [DBG] from='client.57343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:00 smithi115 bash[46934]: audit 2022-04-22T23:08:00.044288+0000 mgr.y (mgr.34107) 11650 : audit [DBG] from='client.57343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:01 smithi121 bash[36902]: audit 2022-04-22T23:08:00.381380+0000 mgr.y (mgr.34107) 11651 : audit [DBG] from='client.47409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:01 smithi121 bash[36902]: cluster 2022-04-22T23:08:00.783601+0000 mgr.y (mgr.34107) 11652 : cluster [DBG] pgmap v12197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:01 smithi121 bash[36902]: audit 2022-04-22T23:08:00.807665+0000 mon.a (mon.0) 16117 : audit [DBG] from='client.? 172.21.15.115:0/2621550511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:01 smithi121 bash[36902]: audit 2022-04-22T23:08:01.272457+0000 mon.a (mon.0) 16118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:02.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[45495]: audit 2022-04-22T23:08:00.381380+0000 mgr.y (mgr.34107) 11651 : audit [DBG] from='client.47409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[45495]: cluster 2022-04-22T23:08:00.783601+0000 mgr.y (mgr.34107) 11652 : cluster [DBG] pgmap v12197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[45495]: audit 2022-04-22T23:08:00.807665+0000 mon.a (mon.0) 16117 : audit [DBG] from='client.? 172.21.15.115:0/2621550511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[45495]: audit 2022-04-22T23:08:01.272457+0000 mon.a (mon.0) 16118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[46934]: audit 2022-04-22T23:08:00.381380+0000 mgr.y (mgr.34107) 11651 : audit [DBG] from='client.47409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[46934]: cluster 2022-04-22T23:08:00.783601+0000 mgr.y (mgr.34107) 11652 : cluster [DBG] pgmap v12197: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[46934]: audit 2022-04-22T23:08:00.807665+0000 mon.a (mon.0) 16117 : audit [DBG] from='client.? 172.21.15.115:0/2621550511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:02.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:01 smithi115 bash[46934]: audit 2022-04-22T23:08:01.272457+0000 mon.a (mon.0) 16118 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:02.965Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:03 smithi121 bash[36902]: cluster 2022-04-22T23:08:02.784711+0000 mgr.y (mgr.34107) 11653 : cluster [DBG] pgmap v12198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:03 smithi121 bash[36902]: audit 2022-04-22T23:08:02.906862+0000 mon.a (mon.0) 16119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:03 smithi115 bash[45495]: cluster 2022-04-22T23:08:02.784711+0000 mgr.y (mgr.34107) 11653 : cluster [DBG] pgmap v12198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:03.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:03 smithi115 bash[45495]: audit 2022-04-22T23:08:02.906862+0000 mon.a (mon.0) 16119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:03 smithi115 bash[46934]: cluster 2022-04-22T23:08:02.784711+0000 mgr.y (mgr.34107) 11653 : cluster [DBG] pgmap v12198: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:03.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:03 smithi115 bash[46934]: audit 2022-04-22T23:08:02.906862+0000 mon.a (mon.0) 16119 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:04.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:04 smithi115 bash[46934]: audit 2022-04-22T23:08:04.516985+0000 mon.a (mon.0) 16120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:04.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:04 smithi115 bash[45495]: audit 2022-04-22T23:08:04.516985+0000 mon.a (mon.0) 16120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:04 smithi121 bash[36902]: audit 2022-04-22T23:08:04.516985+0000 mon.a (mon.0) 16120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:04.820Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:05 smithi121 bash[36902]: cluster 2022-04-22T23:08:04.785870+0000 mgr.y (mgr.34107) 11654 : cluster [DBG] pgmap v12199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:05 smithi115 bash[45495]: cluster 2022-04-22T23:08:04.785870+0000 mgr.y (mgr.34107) 11654 : cluster [DBG] pgmap v12199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:05 smithi115 bash[46934]: cluster 2022-04-22T23:08:04.785870+0000 mgr.y (mgr.34107) 11654 : cluster [DBG] pgmap v12199: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:06 smithi121 bash[36902]: audit 2022-04-22T23:08:06.275613+0000 mon.a (mon.0) 16121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:06 smithi115 bash[45495]: audit 2022-04-22T23:08:06.275613+0000 mon.a (mon.0) 16121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:06 smithi115 bash[46934]: audit 2022-04-22T23:08:06.275613+0000 mon.a (mon.0) 16121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:07 smithi121 bash[36902]: cluster 2022-04-22T23:08:06.786993+0000 mgr.y (mgr.34107) 11655 : cluster [DBG] pgmap v12200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:07 smithi115 bash[45495]: cluster 2022-04-22T23:08:06.786993+0000 mgr.y (mgr.34107) 11655 : cluster [DBG] pgmap v12200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:07 smithi115 bash[46934]: cluster 2022-04-22T23:08:06.786993+0000 mgr.y (mgr.34107) 11655 : cluster [DBG] pgmap v12200: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:08 smithi121 bash[36902]: audit 2022-04-22T23:08:07.909750+0000 mon.a (mon.0) 16122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:08 smithi115 bash[46934]: audit 2022-04-22T23:08:07.909750+0000 mon.a (mon.0) 16122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:09.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:08 smithi115 bash[45495]: audit 2022-04-22T23:08:07.909750+0000 mon.a (mon.0) 16122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:09 smithi121 bash[36902]: cluster 2022-04-22T23:08:08.788192+0000 mgr.y (mgr.34107) 11656 : cluster [DBG] pgmap v12201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:09 smithi121 bash[36902]: audit 2022-04-22T23:08:09.520552+0000 mon.a (mon.0) 16123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:09 smithi115 bash[46934]: cluster 2022-04-22T23:08:08.788192+0000 mgr.y (mgr.34107) 11656 : cluster [DBG] pgmap v12201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:09 smithi115 bash[46934]: audit 2022-04-22T23:08:09.520552+0000 mon.a (mon.0) 16123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:09 smithi115 bash[45495]: cluster 2022-04-22T23:08:08.788192+0000 mgr.y (mgr.34107) 11656 : cluster [DBG] pgmap v12201: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:09 smithi115 bash[45495]: audit 2022-04-22T23:08:09.520552+0000 mon.a (mon.0) 16123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:11 smithi121 bash[36902]: cluster 2022-04-22T23:08:10.789285+0000 mgr.y (mgr.34107) 11657 : cluster [DBG] pgmap v12202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:11 smithi121 bash[36902]: audit 2022-04-22T23:08:11.278524+0000 mon.a (mon.0) 16124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:11 smithi115 bash[45495]: cluster 2022-04-22T23:08:10.789285+0000 mgr.y (mgr.34107) 11657 : cluster [DBG] pgmap v12202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:11 smithi115 bash[45495]: audit 2022-04-22T23:08:11.278524+0000 mon.a (mon.0) 16124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:11 smithi115 bash[46934]: cluster 2022-04-22T23:08:10.789285+0000 mgr.y (mgr.34107) 11657 : cluster [DBG] pgmap v12202: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:11 smithi115 bash[46934]: audit 2022-04-22T23:08:11.278524+0000 mon.a (mon.0) 16124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:12.966Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:13 smithi121 bash[36902]: cluster 2022-04-22T23:08:12.790387+0000 mgr.y (mgr.34107) 11658 : cluster [DBG] pgmap v12203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:13 smithi121 bash[36902]: audit 2022-04-22T23:08:12.912968+0000 mon.a (mon.0) 16125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:13 smithi115 bash[45495]: cluster 2022-04-22T23:08:12.790387+0000 mgr.y (mgr.34107) 11658 : cluster [DBG] pgmap v12203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:13 smithi115 bash[45495]: audit 2022-04-22T23:08:12.912968+0000 mon.a (mon.0) 16125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:13 smithi115 bash[46934]: cluster 2022-04-22T23:08:12.790387+0000 mgr.y (mgr.34107) 11658 : cluster [DBG] pgmap v12203: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:13 smithi115 bash[46934]: audit 2022-04-22T23:08:12.912968+0000 mon.a (mon.0) 16125 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:14 smithi121 bash[36902]: audit 2022-04-22T23:08:14.524236+0000 mon.a (mon.0) 16126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:14 smithi115 bash[45495]: audit 2022-04-22T23:08:14.524236+0000 mon.a (mon.0) 16126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:14.819Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:14 smithi115 bash[46934]: audit 2022-04-22T23:08:14.524236+0000 mon.a (mon.0) 16126 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:15 smithi121 bash[36902]: cluster 2022-04-22T23:08:14.791565+0000 mgr.y (mgr.34107) 11659 : cluster [DBG] pgmap v12204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:15 smithi115 bash[45495]: cluster 2022-04-22T23:08:14.791565+0000 mgr.y (mgr.34107) 11659 : cluster [DBG] pgmap v12204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:15 smithi115 bash[46934]: cluster 2022-04-22T23:08:14.791565+0000 mgr.y (mgr.34107) 11659 : cluster [DBG] pgmap v12204: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:16 smithi121 bash[36902]: audit 2022-04-22T23:08:16.281616+0000 mon.a (mon.0) 16127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:16 smithi115 bash[45495]: audit 2022-04-22T23:08:16.281616+0000 mon.a (mon.0) 16127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:16 smithi115 bash[46934]: audit 2022-04-22T23:08:16.281616+0000 mon.a (mon.0) 16127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:18.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:17 smithi121 bash[36902]: cluster 2022-04-22T23:08:16.792609+0000 mgr.y (mgr.34107) 11660 : cluster [DBG] pgmap v12205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:17 smithi121 bash[36902]: audit 2022-04-22T23:08:17.208923+0000 mon.a (mon.0) 16128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:08:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:17 smithi121 bash[36902]: audit 2022-04-22T23:08:17.209535+0000 mon.a (mon.0) 16129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[45495]: cluster 2022-04-22T23:08:16.792609+0000 mgr.y (mgr.34107) 11660 : cluster [DBG] pgmap v12205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[45495]: audit 2022-04-22T23:08:17.208923+0000 mon.a (mon.0) 16128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[45495]: audit 2022-04-22T23:08:17.209535+0000 mon.a (mon.0) 16129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[46934]: cluster 2022-04-22T23:08:16.792609+0000 mgr.y (mgr.34107) 11660 : cluster [DBG] pgmap v12205: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[46934]: audit 2022-04-22T23:08:17.208923+0000 mon.a (mon.0) 16128 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:08:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:17 smithi115 bash[46934]: audit 2022-04-22T23:08:17.209535+0000 mon.a (mon.0) 16129 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:08:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:18 smithi121 bash[36902]: audit 2022-04-22T23:08:17.916027+0000 mon.a (mon.0) 16130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:18 smithi115 bash[45495]: audit 2022-04-22T23:08:17.916027+0000 mon.a (mon.0) 16130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:18 smithi115 bash[46934]: audit 2022-04-22T23:08:17.916027+0000 mon.a (mon.0) 16130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:20.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:19 smithi121 bash[36902]: cluster 2022-04-22T23:08:18.793781+0000 mgr.y (mgr.34107) 11661 : cluster [DBG] pgmap v12206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:19 smithi121 bash[36902]: audit 2022-04-22T23:08:19.527905+0000 mon.a (mon.0) 16131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:20.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:19 smithi115 bash[45495]: cluster 2022-04-22T23:08:18.793781+0000 mgr.y (mgr.34107) 11661 : cluster [DBG] pgmap v12206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:19 smithi115 bash[45495]: audit 2022-04-22T23:08:19.527905+0000 mon.a (mon.0) 16131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:19 smithi115 bash[46934]: cluster 2022-04-22T23:08:18.793781+0000 mgr.y (mgr.34107) 11661 : cluster [DBG] pgmap v12206: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:19 smithi115 bash[46934]: audit 2022-04-22T23:08:19.527905+0000 mon.a (mon.0) 16131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:21 smithi121 bash[36902]: cluster 2022-04-22T23:08:20.794874+0000 mgr.y (mgr.34107) 11662 : cluster [DBG] pgmap v12207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:21 smithi121 bash[36902]: audit 2022-04-22T23:08:21.284664+0000 mon.a (mon.0) 16132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:21 smithi115 bash[46934]: cluster 2022-04-22T23:08:20.794874+0000 mgr.y (mgr.34107) 11662 : cluster [DBG] pgmap v12207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:21 smithi115 bash[46934]: audit 2022-04-22T23:08:21.284664+0000 mon.a (mon.0) 16132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:21 smithi115 bash[45495]: cluster 2022-04-22T23:08:20.794874+0000 mgr.y (mgr.34107) 11662 : cluster [DBG] pgmap v12207: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:21 smithi115 bash[45495]: audit 2022-04-22T23:08:21.284664+0000 mon.a (mon.0) 16132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:22.967Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:23 smithi121 bash[36902]: cluster 2022-04-22T23:08:22.796003+0000 mgr.y (mgr.34107) 11663 : cluster [DBG] pgmap v12208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:23 smithi121 bash[36902]: audit 2022-04-22T23:08:22.919039+0000 mon.a (mon.0) 16133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:23 smithi115 bash[45495]: cluster 2022-04-22T23:08:22.796003+0000 mgr.y (mgr.34107) 11663 : cluster [DBG] pgmap v12208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:23 smithi115 bash[45495]: audit 2022-04-22T23:08:22.919039+0000 mon.a (mon.0) 16133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:23 smithi115 bash[46934]: cluster 2022-04-22T23:08:22.796003+0000 mgr.y (mgr.34107) 11663 : cluster [DBG] pgmap v12208: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:23 smithi115 bash[46934]: audit 2022-04-22T23:08:22.919039+0000 mon.a (mon.0) 16133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:24 smithi121 bash[36902]: audit 2022-04-22T23:08:24.531743+0000 mon.a (mon.0) 16134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:24 smithi115 bash[45495]: audit 2022-04-22T23:08:24.531743+0000 mon.a (mon.0) 16134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:24.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:24 smithi115 bash[46934]: audit 2022-04-22T23:08:24.531743+0000 mon.a (mon.0) 16134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:25.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:25 smithi115 bash[46934]: cluster 2022-04-22T23:08:24.797091+0000 mgr.y (mgr.34107) 11664 : cluster [DBG] pgmap v12209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:25.972 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:25 smithi115 bash[45495]: cluster 2022-04-22T23:08:24.797091+0000 mgr.y (mgr.34107) 11664 : cluster [DBG] pgmap v12209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:25.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:25 smithi121 bash[36902]: cluster 2022-04-22T23:08:24.797091+0000 mgr.y (mgr.34107) 11664 : cluster [DBG] pgmap v12209: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:26 smithi121 bash[36902]: audit 2022-04-22T23:08:25.610337+0000 mon.a (mon.0) 16135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:08:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:26 smithi121 bash[36902]: audit 2022-04-22T23:08:25.613076+0000 mon.a (mon.0) 16136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:08:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:26 smithi121 bash[36902]: audit 2022-04-22T23:08:25.614754+0000 mon.a (mon.0) 16137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:08:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:26 smithi121 bash[36902]: audit 2022-04-22T23:08:25.988378+0000 mon.a (mon.0) 16138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:08:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:26 smithi121 bash[36902]: audit 2022-04-22T23:08:26.287995+0000 mon.a (mon.0) 16139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[45495]: audit 2022-04-22T23:08:25.610337+0000 mon.a (mon.0) 16135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:08:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[45495]: audit 2022-04-22T23:08:25.613076+0000 mon.a (mon.0) 16136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:08:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[45495]: audit 2022-04-22T23:08:25.614754+0000 mon.a (mon.0) 16137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:08:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[45495]: audit 2022-04-22T23:08:25.988378+0000 mon.a (mon.0) 16138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:08:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[45495]: audit 2022-04-22T23:08:26.287995+0000 mon.a (mon.0) 16139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[46934]: audit 2022-04-22T23:08:25.610337+0000 mon.a (mon.0) 16135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:08:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[46934]: audit 2022-04-22T23:08:25.613076+0000 mon.a (mon.0) 16136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:08:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[46934]: audit 2022-04-22T23:08:25.614754+0000 mon.a (mon.0) 16137 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:08:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[46934]: audit 2022-04-22T23:08:25.988378+0000 mon.a (mon.0) 16138 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:08:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:26 smithi115 bash[46934]: audit 2022-04-22T23:08:26.287995+0000 mon.a (mon.0) 16139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:27 smithi121 bash[36902]: cluster 2022-04-22T23:08:26.798306+0000 mgr.y (mgr.34107) 11665 : cluster [DBG] pgmap v12210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:27 smithi115 bash[45495]: cluster 2022-04-22T23:08:26.798306+0000 mgr.y (mgr.34107) 11665 : cluster [DBG] pgmap v12210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:27 smithi115 bash[46934]: cluster 2022-04-22T23:08:26.798306+0000 mgr.y (mgr.34107) 11665 : cluster [DBG] pgmap v12210: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:28 smithi121 bash[36902]: audit 2022-04-22T23:08:27.921957+0000 mon.a (mon.0) 16140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:28 smithi115 bash[45495]: audit 2022-04-22T23:08:27.921957+0000 mon.a (mon.0) 16140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:28 smithi115 bash[46934]: audit 2022-04-22T23:08:27.921957+0000 mon.a (mon.0) 16140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:29 smithi121 bash[36902]: cluster 2022-04-22T23:08:28.799435+0000 mgr.y (mgr.34107) 11666 : cluster [DBG] pgmap v12211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:29 smithi121 bash[36902]: audit 2022-04-22T23:08:29.535348+0000 mon.a (mon.0) 16141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:29 smithi115 bash[46934]: cluster 2022-04-22T23:08:28.799435+0000 mgr.y (mgr.34107) 11666 : cluster [DBG] pgmap v12211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:29 smithi115 bash[46934]: audit 2022-04-22T23:08:29.535348+0000 mon.a (mon.0) 16141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:29 smithi115 bash[45495]: cluster 2022-04-22T23:08:28.799435+0000 mgr.y (mgr.34107) 11666 : cluster [DBG] pgmap v12211: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:29 smithi115 bash[45495]: audit 2022-04-22T23:08:29.535348+0000 mon.a (mon.0) 16141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:31.185 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:08:31.530 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 66s ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 66s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 66s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 66s ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:08:31.531 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 66s ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 66s ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 66s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 66s ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 66s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 66s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:08:31.532 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:08:31.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:08:31.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:08:31.533 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:08:31.533 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:08:31.931 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:08:31.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:08:31.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:08:31.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:31.932 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:08:31.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:08:31.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:31.932 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:08:31.933 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:08:31.934 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:08:31.934 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:08:31.934 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:08:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:31 smithi121 bash[36902]: cluster 2022-04-22T23:08:30.800549+0000 mgr.y (mgr.34107) 11667 : cluster [DBG] pgmap v12212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:31 smithi121 bash[36902]: audit 2022-04-22T23:08:31.291173+0000 mon.a (mon.0) 16142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:31 smithi115 bash[45495]: cluster 2022-04-22T23:08:30.800549+0000 mgr.y (mgr.34107) 11667 : cluster [DBG] pgmap v12212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:31 smithi115 bash[45495]: audit 2022-04-22T23:08:31.291173+0000 mon.a (mon.0) 16142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:31 smithi115 bash[46934]: cluster 2022-04-22T23:08:30.800549+0000 mgr.y (mgr.34107) 11667 : cluster [DBG] pgmap v12212: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:31 smithi115 bash[46934]: audit 2022-04-22T23:08:31.291173+0000 mon.a (mon.0) 16142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:33.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:32 smithi121 bash[36902]: audit 2022-04-22T23:08:31.166958+0000 mgr.y (mgr.34107) 11668 : audit [DBG] from='client.57361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:32 smithi121 bash[36902]: audit 2022-04-22T23:08:31.508008+0000 mgr.y (mgr.34107) 11669 : audit [DBG] from='client.47427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:32 smithi121 bash[36902]: audit 2022-04-22T23:08:31.927544+0000 mon.c (mon.1) 392 : audit [DBG] from='client.? 172.21.15.115:0/652556939' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:32.968Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[45495]: audit 2022-04-22T23:08:31.166958+0000 mgr.y (mgr.34107) 11668 : audit [DBG] from='client.57361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[45495]: audit 2022-04-22T23:08:31.508008+0000 mgr.y (mgr.34107) 11669 : audit [DBG] from='client.47427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[45495]: audit 2022-04-22T23:08:31.927544+0000 mon.c (mon.1) 392 : audit [DBG] from='client.? 172.21.15.115:0/652556939' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[46934]: audit 2022-04-22T23:08:31.166958+0000 mgr.y (mgr.34107) 11668 : audit [DBG] from='client.57361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[46934]: audit 2022-04-22T23:08:31.508008+0000 mgr.y (mgr.34107) 11669 : audit [DBG] from='client.47427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:08:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:32 smithi115 bash[46934]: audit 2022-04-22T23:08:31.927544+0000 mon.c (mon.1) 392 : audit [DBG] from='client.? 172.21.15.115:0/652556939' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:08:34.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:33 smithi121 bash[36902]: cluster 2022-04-22T23:08:32.801767+0000 mgr.y (mgr.34107) 11670 : cluster [DBG] pgmap v12213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:33 smithi121 bash[36902]: audit 2022-04-22T23:08:32.924971+0000 mon.a (mon.0) 16143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:33 smithi115 bash[45495]: cluster 2022-04-22T23:08:32.801767+0000 mgr.y (mgr.34107) 11670 : cluster [DBG] pgmap v12213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:33 smithi115 bash[45495]: audit 2022-04-22T23:08:32.924971+0000 mon.a (mon.0) 16143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:33 smithi115 bash[46934]: cluster 2022-04-22T23:08:32.801767+0000 mgr.y (mgr.34107) 11670 : cluster [DBG] pgmap v12213: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:33 smithi115 bash[46934]: audit 2022-04-22T23:08:32.924971+0000 mon.a (mon.0) 16143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:34 smithi121 bash[36902]: audit 2022-04-22T23:08:34.538952+0000 mon.a (mon.0) 16144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:34 smithi115 bash[45495]: audit 2022-04-22T23:08:34.538952+0000 mon.a (mon.0) 16144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:34.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:34 smithi115 bash[46934]: audit 2022-04-22T23:08:34.538952+0000 mon.a (mon.0) 16144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:35 smithi121 bash[36902]: cluster 2022-04-22T23:08:34.802943+0000 mgr.y (mgr.34107) 11671 : cluster [DBG] pgmap v12214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:35 smithi115 bash[45495]: cluster 2022-04-22T23:08:34.802943+0000 mgr.y (mgr.34107) 11671 : cluster [DBG] pgmap v12214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:35 smithi115 bash[46934]: cluster 2022-04-22T23:08:34.802943+0000 mgr.y (mgr.34107) 11671 : cluster [DBG] pgmap v12214: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:36 smithi121 bash[36902]: audit 2022-04-22T23:08:36.293853+0000 mon.a (mon.0) 16145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:36 smithi115 bash[45495]: audit 2022-04-22T23:08:36.293853+0000 mon.a (mon.0) 16145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:36 smithi115 bash[46934]: audit 2022-04-22T23:08:36.293853+0000 mon.a (mon.0) 16145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:37 smithi121 bash[36902]: cluster 2022-04-22T23:08:36.804097+0000 mgr.y (mgr.34107) 11672 : cluster [DBG] pgmap v12215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:37 smithi115 bash[45495]: cluster 2022-04-22T23:08:36.804097+0000 mgr.y (mgr.34107) 11672 : cluster [DBG] pgmap v12215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:37 smithi115 bash[46934]: cluster 2022-04-22T23:08:36.804097+0000 mgr.y (mgr.34107) 11672 : cluster [DBG] pgmap v12215: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:38 smithi121 bash[36902]: audit 2022-04-22T23:08:37.928053+0000 mon.a (mon.0) 16146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:38 smithi115 bash[45495]: audit 2022-04-22T23:08:37.928053+0000 mon.a (mon.0) 16146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:38 smithi115 bash[46934]: audit 2022-04-22T23:08:37.928053+0000 mon.a (mon.0) 16146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:40.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:39 smithi121 bash[36902]: cluster 2022-04-22T23:08:38.805251+0000 mgr.y (mgr.34107) 11673 : cluster [DBG] pgmap v12216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:39 smithi121 bash[36902]: audit 2022-04-22T23:08:39.542624+0000 mon.a (mon.0) 16147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:39 smithi115 bash[46934]: cluster 2022-04-22T23:08:38.805251+0000 mgr.y (mgr.34107) 11673 : cluster [DBG] pgmap v12216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:39 smithi115 bash[46934]: audit 2022-04-22T23:08:39.542624+0000 mon.a (mon.0) 16147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:39 smithi115 bash[45495]: cluster 2022-04-22T23:08:38.805251+0000 mgr.y (mgr.34107) 11673 : cluster [DBG] pgmap v12216: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:39 smithi115 bash[45495]: audit 2022-04-22T23:08:39.542624+0000 mon.a (mon.0) 16147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:41 smithi121 bash[36902]: cluster 2022-04-22T23:08:40.806333+0000 mgr.y (mgr.34107) 11674 : cluster [DBG] pgmap v12217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:41 smithi121 bash[36902]: audit 2022-04-22T23:08:41.296843+0000 mon.a (mon.0) 16148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:41 smithi115 bash[46934]: cluster 2022-04-22T23:08:40.806333+0000 mgr.y (mgr.34107) 11674 : cluster [DBG] pgmap v12217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:41 smithi115 bash[46934]: audit 2022-04-22T23:08:41.296843+0000 mon.a (mon.0) 16148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:41 smithi115 bash[45495]: cluster 2022-04-22T23:08:40.806333+0000 mgr.y (mgr.34107) 11674 : cluster [DBG] pgmap v12217: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:41 smithi115 bash[45495]: audit 2022-04-22T23:08:41.296843+0000 mon.a (mon.0) 16148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:42.968Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:43 smithi121 bash[36902]: cluster 2022-04-22T23:08:42.807421+0000 mgr.y (mgr.34107) 11675 : cluster [DBG] pgmap v12218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:43 smithi121 bash[36902]: audit 2022-04-22T23:08:42.930906+0000 mon.a (mon.0) 16149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:43 smithi115 bash[45495]: cluster 2022-04-22T23:08:42.807421+0000 mgr.y (mgr.34107) 11675 : cluster [DBG] pgmap v12218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:43 smithi115 bash[45495]: audit 2022-04-22T23:08:42.930906+0000 mon.a (mon.0) 16149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:43 smithi115 bash[46934]: cluster 2022-04-22T23:08:42.807421+0000 mgr.y (mgr.34107) 11675 : cluster [DBG] pgmap v12218: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:43 smithi115 bash[46934]: audit 2022-04-22T23:08:42.930906+0000 mon.a (mon.0) 16149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:44 smithi121 bash[36902]: audit 2022-04-22T23:08:44.546407+0000 mon.a (mon.0) 16150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:44 smithi115 bash[45495]: audit 2022-04-22T23:08:44.546407+0000 mon.a (mon.0) 16150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:44.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:44 smithi115 bash[46934]: audit 2022-04-22T23:08:44.546407+0000 mon.a (mon.0) 16150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:46.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:45 smithi121 bash[36902]: cluster 2022-04-22T23:08:44.808507+0000 mgr.y (mgr.34107) 11676 : cluster [DBG] pgmap v12219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:45 smithi115 bash[45495]: cluster 2022-04-22T23:08:44.808507+0000 mgr.y (mgr.34107) 11676 : cluster [DBG] pgmap v12219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:45 smithi115 bash[46934]: cluster 2022-04-22T23:08:44.808507+0000 mgr.y (mgr.34107) 11676 : cluster [DBG] pgmap v12219: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:46 smithi121 bash[36902]: audit 2022-04-22T23:08:46.300169+0000 mon.a (mon.0) 16151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:46 smithi115 bash[45495]: audit 2022-04-22T23:08:46.300169+0000 mon.a (mon.0) 16151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:46 smithi115 bash[46934]: audit 2022-04-22T23:08:46.300169+0000 mon.a (mon.0) 16151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:47 smithi121 bash[36902]: cluster 2022-04-22T23:08:46.809571+0000 mgr.y (mgr.34107) 11677 : cluster [DBG] pgmap v12220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:48.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:47 smithi115 bash[45495]: cluster 2022-04-22T23:08:46.809571+0000 mgr.y (mgr.34107) 11677 : cluster [DBG] pgmap v12220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:47 smithi115 bash[46934]: cluster 2022-04-22T23:08:46.809571+0000 mgr.y (mgr.34107) 11677 : cluster [DBG] pgmap v12220: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:48 smithi121 bash[36902]: audit 2022-04-22T23:08:47.934000+0000 mon.a (mon.0) 16152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:48 smithi115 bash[45495]: audit 2022-04-22T23:08:47.934000+0000 mon.a (mon.0) 16152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:48 smithi115 bash[46934]: audit 2022-04-22T23:08:47.934000+0000 mon.a (mon.0) 16152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:49 smithi121 bash[36902]: cluster 2022-04-22T23:08:48.810746+0000 mgr.y (mgr.34107) 11678 : cluster [DBG] pgmap v12221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:50.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:49 smithi121 bash[36902]: audit 2022-04-22T23:08:49.550183+0000 mon.a (mon.0) 16153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:49 smithi115 bash[46934]: cluster 2022-04-22T23:08:48.810746+0000 mgr.y (mgr.34107) 11678 : cluster [DBG] pgmap v12221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:49 smithi115 bash[46934]: audit 2022-04-22T23:08:49.550183+0000 mon.a (mon.0) 16153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:49 smithi115 bash[45495]: cluster 2022-04-22T23:08:48.810746+0000 mgr.y (mgr.34107) 11678 : cluster [DBG] pgmap v12221: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:49 smithi115 bash[45495]: audit 2022-04-22T23:08:49.550183+0000 mon.a (mon.0) 16153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:51 smithi121 bash[36902]: cluster 2022-04-22T23:08:50.811825+0000 mgr.y (mgr.34107) 11679 : cluster [DBG] pgmap v12222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:51 smithi121 bash[36902]: audit 2022-04-22T23:08:51.303197+0000 mon.a (mon.0) 16154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:51 smithi115 bash[45495]: cluster 2022-04-22T23:08:50.811825+0000 mgr.y (mgr.34107) 11679 : cluster [DBG] pgmap v12222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:51 smithi115 bash[45495]: audit 2022-04-22T23:08:51.303197+0000 mon.a (mon.0) 16154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:51 smithi115 bash[46934]: cluster 2022-04-22T23:08:50.811825+0000 mgr.y (mgr.34107) 11679 : cluster [DBG] pgmap v12222: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:51 smithi115 bash[46934]: audit 2022-04-22T23:08:51.303197+0000 mon.a (mon.0) 16154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:52.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:53 smithi121 bash[36902]: cluster 2022-04-22T23:08:52.812981+0000 mgr.y (mgr.34107) 11680 : cluster [DBG] pgmap v12223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:53 smithi121 bash[36902]: audit 2022-04-22T23:08:52.936934+0000 mon.a (mon.0) 16155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:53 smithi115 bash[45495]: cluster 2022-04-22T23:08:52.812981+0000 mgr.y (mgr.34107) 11680 : cluster [DBG] pgmap v12223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:53 smithi115 bash[45495]: audit 2022-04-22T23:08:52.936934+0000 mon.a (mon.0) 16155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:53 smithi115 bash[46934]: cluster 2022-04-22T23:08:52.812981+0000 mgr.y (mgr.34107) 11680 : cluster [DBG] pgmap v12223: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:53 smithi115 bash[46934]: audit 2022-04-22T23:08:52.936934+0000 mon.a (mon.0) 16155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:54 smithi121 bash[36902]: audit 2022-04-22T23:08:54.553795+0000 mon.a (mon.0) 16156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:54 smithi115 bash[45495]: audit 2022-04-22T23:08:54.553795+0000 mon.a (mon.0) 16156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:08:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:08:54.821Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:08:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:54 smithi115 bash[46934]: audit 2022-04-22T23:08:54.553795+0000 mon.a (mon.0) 16156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:08:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:55 smithi121 bash[36902]: cluster 2022-04-22T23:08:54.814384+0000 mgr.y (mgr.34107) 11681 : cluster [DBG] pgmap v12224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:56.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:55 smithi115 bash[46934]: cluster 2022-04-22T23:08:54.814384+0000 mgr.y (mgr.34107) 11681 : cluster [DBG] pgmap v12224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:55 smithi115 bash[45495]: cluster 2022-04-22T23:08:54.814384+0000 mgr.y (mgr.34107) 11681 : cluster [DBG] pgmap v12224: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:56 smithi121 bash[36902]: audit 2022-04-22T23:08:56.306324+0000 mon.a (mon.0) 16157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:56 smithi115 bash[46934]: audit 2022-04-22T23:08:56.306324+0000 mon.a (mon.0) 16157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:56 smithi115 bash[45495]: audit 2022-04-22T23:08:56.306324+0000 mon.a (mon.0) 16157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:08:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:57 smithi121 bash[36902]: cluster 2022-04-22T23:08:56.815521+0000 mgr.y (mgr.34107) 11682 : cluster [DBG] pgmap v12225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:57 smithi115 bash[45495]: cluster 2022-04-22T23:08:56.815521+0000 mgr.y (mgr.34107) 11682 : cluster [DBG] pgmap v12225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:57 smithi115 bash[46934]: cluster 2022-04-22T23:08:56.815521+0000 mgr.y (mgr.34107) 11682 : cluster [DBG] pgmap v12225: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:08:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:58 smithi121 bash[36902]: audit 2022-04-22T23:08:57.940271+0000 mon.a (mon.0) 16158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:58 smithi115 bash[45495]: audit 2022-04-22T23:08:57.940271+0000 mon.a (mon.0) 16158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:08:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:58 smithi115 bash[46934]: audit 2022-04-22T23:08:57.940271+0000 mon.a (mon.0) 16158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:00.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:59 smithi121 bash[36902]: cluster 2022-04-22T23:08:58.816715+0000 mgr.y (mgr.34107) 11683 : cluster [DBG] pgmap v12226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:08:59 smithi121 bash[36902]: audit 2022-04-22T23:08:59.557755+0000 mon.a (mon.0) 16159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:59 smithi115 bash[45495]: cluster 2022-04-22T23:08:58.816715+0000 mgr.y (mgr.34107) 11683 : cluster [DBG] pgmap v12226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:08:59 smithi115 bash[45495]: audit 2022-04-22T23:08:59.557755+0000 mon.a (mon.0) 16159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:59 smithi115 bash[46934]: cluster 2022-04-22T23:08:58.816715+0000 mgr.y (mgr.34107) 11683 : cluster [DBG] pgmap v12226: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:08:59 smithi115 bash[46934]: audit 2022-04-22T23:08:59.557755+0000 mon.a (mon.0) 16159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:01 smithi121 bash[36902]: cluster 2022-04-22T23:09:00.817805+0000 mgr.y (mgr.34107) 11684 : cluster [DBG] pgmap v12227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:01 smithi121 bash[36902]: audit 2022-04-22T23:09:01.309520+0000 mon.a (mon.0) 16160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:02.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:01 smithi115 bash[45495]: cluster 2022-04-22T23:09:00.817805+0000 mgr.y (mgr.34107) 11684 : cluster [DBG] pgmap v12227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:01 smithi115 bash[45495]: audit 2022-04-22T23:09:01.309520+0000 mon.a (mon.0) 16160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:01 smithi115 bash[46934]: cluster 2022-04-22T23:09:00.817805+0000 mgr.y (mgr.34107) 11684 : cluster [DBG] pgmap v12227: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:01 smithi115 bash[46934]: audit 2022-04-22T23:09:01.309520+0000 mon.a (mon.0) 16160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:02.317 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:09:02.663 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:09:02.663 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 97s ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:09:02.663 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 97s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 97s ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 97s ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:09:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 97s ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:09:02.666 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 97s ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 97s ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 97s ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 97s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 97s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:09:02.667 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:09:02.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:09:02.668 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:09:02.668 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:09:03.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:09:03.068 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:09:03.069 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:09:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:02.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:03 smithi121 bash[36902]: audit 2022-04-22T23:09:02.295708+0000 mgr.y (mgr.34107) 11685 : audit [DBG] from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:03 smithi121 bash[36902]: audit 2022-04-22T23:09:02.640364+0000 mgr.y (mgr.34107) 11686 : audit [DBG] from='client.47445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:03 smithi121 bash[36902]: cluster 2022-04-22T23:09:02.818843+0000 mgr.y (mgr.34107) 11687 : cluster [DBG] pgmap v12228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:03 smithi121 bash[36902]: audit 2022-04-22T23:09:02.943150+0000 mon.a (mon.0) 16161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:03 smithi121 bash[36902]: audit 2022-04-22T23:09:03.063343+0000 mon.c (mon.1) 393 : audit [DBG] from='client.? 172.21.15.115:0/444653479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[46934]: audit 2022-04-22T23:09:02.295708+0000 mgr.y (mgr.34107) 11685 : audit [DBG] from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[46934]: audit 2022-04-22T23:09:02.640364+0000 mgr.y (mgr.34107) 11686 : audit [DBG] from='client.47445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[46934]: cluster 2022-04-22T23:09:02.818843+0000 mgr.y (mgr.34107) 11687 : cluster [DBG] pgmap v12228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[46934]: audit 2022-04-22T23:09:02.943150+0000 mon.a (mon.0) 16161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[46934]: audit 2022-04-22T23:09:03.063343+0000 mon.c (mon.1) 393 : audit [DBG] from='client.? 172.21.15.115:0/444653479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[45495]: audit 2022-04-22T23:09:02.295708+0000 mgr.y (mgr.34107) 11685 : audit [DBG] from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[45495]: audit 2022-04-22T23:09:02.640364+0000 mgr.y (mgr.34107) 11686 : audit [DBG] from='client.47445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[45495]: cluster 2022-04-22T23:09:02.818843+0000 mgr.y (mgr.34107) 11687 : cluster [DBG] pgmap v12228: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[45495]: audit 2022-04-22T23:09:02.943150+0000 mon.a (mon.0) 16161 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:03.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:03 smithi115 bash[45495]: audit 2022-04-22T23:09:03.063343+0000 mon.c (mon.1) 393 : audit [DBG] from='client.? 172.21.15.115:0/444653479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:04 smithi121 bash[36902]: audit 2022-04-22T23:09:04.561349+0000 mon.a (mon.0) 16162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:04.823Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:04 smithi115 bash[45495]: audit 2022-04-22T23:09:04.561349+0000 mon.a (mon.0) 16162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:04 smithi115 bash[46934]: audit 2022-04-22T23:09:04.561349+0000 mon.a (mon.0) 16162 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:05 smithi121 bash[36902]: cluster 2022-04-22T23:09:04.820275+0000 mgr.y (mgr.34107) 11688 : cluster [DBG] pgmap v12229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:05 smithi115 bash[45495]: cluster 2022-04-22T23:09:04.820275+0000 mgr.y (mgr.34107) 11688 : cluster [DBG] pgmap v12229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:05 smithi115 bash[46934]: cluster 2022-04-22T23:09:04.820275+0000 mgr.y (mgr.34107) 11688 : cluster [DBG] pgmap v12229: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:06 smithi121 bash[36902]: audit 2022-04-22T23:09:06.312963+0000 mon.a (mon.0) 16163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:07.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:06 smithi115 bash[45495]: audit 2022-04-22T23:09:06.312963+0000 mon.a (mon.0) 16163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:06 smithi115 bash[46934]: audit 2022-04-22T23:09:06.312963+0000 mon.a (mon.0) 16163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:08.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:07 smithi121 bash[36902]: cluster 2022-04-22T23:09:06.821443+0000 mgr.y (mgr.34107) 11689 : cluster [DBG] pgmap v12230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:08.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:07 smithi115 bash[45495]: cluster 2022-04-22T23:09:06.821443+0000 mgr.y (mgr.34107) 11689 : cluster [DBG] pgmap v12230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:07 smithi115 bash[46934]: cluster 2022-04-22T23:09:06.821443+0000 mgr.y (mgr.34107) 11689 : cluster [DBG] pgmap v12230: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:08 smithi121 bash[36902]: audit 2022-04-22T23:09:07.946051+0000 mon.a (mon.0) 16164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:08 smithi115 bash[45495]: audit 2022-04-22T23:09:07.946051+0000 mon.a (mon.0) 16164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:08 smithi115 bash[46934]: audit 2022-04-22T23:09:07.946051+0000 mon.a (mon.0) 16164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:09 smithi121 bash[36902]: cluster 2022-04-22T23:09:08.822638+0000 mgr.y (mgr.34107) 11690 : cluster [DBG] pgmap v12231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:09 smithi121 bash[36902]: audit 2022-04-22T23:09:09.565115+0000 mon.a (mon.0) 16165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:10.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:09 smithi115 bash[45495]: cluster 2022-04-22T23:09:08.822638+0000 mgr.y (mgr.34107) 11690 : cluster [DBG] pgmap v12231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:09 smithi115 bash[45495]: audit 2022-04-22T23:09:09.565115+0000 mon.a (mon.0) 16165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:09 smithi115 bash[46934]: cluster 2022-04-22T23:09:08.822638+0000 mgr.y (mgr.34107) 11690 : cluster [DBG] pgmap v12231: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:09 smithi115 bash[46934]: audit 2022-04-22T23:09:09.565115+0000 mon.a (mon.0) 16165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:11 smithi121 bash[36902]: cluster 2022-04-22T23:09:10.823772+0000 mgr.y (mgr.34107) 11691 : cluster [DBG] pgmap v12232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:11 smithi121 bash[36902]: audit 2022-04-22T23:09:11.315900+0000 mon.a (mon.0) 16166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:11 smithi115 bash[45495]: cluster 2022-04-22T23:09:10.823772+0000 mgr.y (mgr.34107) 11691 : cluster [DBG] pgmap v12232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:11 smithi115 bash[45495]: audit 2022-04-22T23:09:11.315900+0000 mon.a (mon.0) 16166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:11 smithi115 bash[46934]: cluster 2022-04-22T23:09:10.823772+0000 mgr.y (mgr.34107) 11691 : cluster [DBG] pgmap v12232: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:11 smithi115 bash[46934]: audit 2022-04-22T23:09:11.315900+0000 mon.a (mon.0) 16166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:12.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:13 smithi121 bash[36902]: cluster 2022-04-22T23:09:12.824883+0000 mgr.y (mgr.34107) 11692 : cluster [DBG] pgmap v12233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:13 smithi121 bash[36902]: audit 2022-04-22T23:09:12.949036+0000 mon.a (mon.0) 16167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:13 smithi115 bash[45495]: cluster 2022-04-22T23:09:12.824883+0000 mgr.y (mgr.34107) 11692 : cluster [DBG] pgmap v12233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:13 smithi115 bash[45495]: audit 2022-04-22T23:09:12.949036+0000 mon.a (mon.0) 16167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:13 smithi115 bash[46934]: cluster 2022-04-22T23:09:12.824883+0000 mgr.y (mgr.34107) 11692 : cluster [DBG] pgmap v12233: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:13 smithi115 bash[46934]: audit 2022-04-22T23:09:12.949036+0000 mon.a (mon.0) 16167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:14 smithi121 bash[36902]: audit 2022-04-22T23:09:14.568911+0000 mon.a (mon.0) 16168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:14.825Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:14 smithi115 bash[46934]: audit 2022-04-22T23:09:14.568911+0000 mon.a (mon.0) 16168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:14 smithi115 bash[45495]: audit 2022-04-22T23:09:14.568911+0000 mon.a (mon.0) 16168 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:16.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:15 smithi121 bash[36902]: cluster 2022-04-22T23:09:14.826055+0000 mgr.y (mgr.34107) 11693 : cluster [DBG] pgmap v12234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:15 smithi115 bash[45495]: cluster 2022-04-22T23:09:14.826055+0000 mgr.y (mgr.34107) 11693 : cluster [DBG] pgmap v12234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:15 smithi115 bash[46934]: cluster 2022-04-22T23:09:14.826055+0000 mgr.y (mgr.34107) 11693 : cluster [DBG] pgmap v12234: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:16 smithi121 bash[36902]: audit 2022-04-22T23:09:16.319094+0000 mon.a (mon.0) 16169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:16 smithi115 bash[45495]: audit 2022-04-22T23:09:16.319094+0000 mon.a (mon.0) 16169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:16 smithi115 bash[46934]: audit 2022-04-22T23:09:16.319094+0000 mon.a (mon.0) 16169 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:17 smithi121 bash[36902]: cluster 2022-04-22T23:09:16.827176+0000 mgr.y (mgr.34107) 11694 : cluster [DBG] pgmap v12235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:17 smithi121 bash[36902]: audit 2022-04-22T23:09:17.219445+0000 mon.a (mon.0) 16170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:09:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:17 smithi121 bash[36902]: audit 2022-04-22T23:09:17.220097+0000 mon.a (mon.0) 16171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[45495]: cluster 2022-04-22T23:09:16.827176+0000 mgr.y (mgr.34107) 11694 : cluster [DBG] pgmap v12235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[45495]: audit 2022-04-22T23:09:17.219445+0000 mon.a (mon.0) 16170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[45495]: audit 2022-04-22T23:09:17.220097+0000 mon.a (mon.0) 16171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[46934]: cluster 2022-04-22T23:09:16.827176+0000 mgr.y (mgr.34107) 11694 : cluster [DBG] pgmap v12235: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[46934]: audit 2022-04-22T23:09:17.219445+0000 mon.a (mon.0) 16170 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:09:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:17 smithi115 bash[46934]: audit 2022-04-22T23:09:17.220097+0000 mon.a (mon.0) 16171 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:09:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:18 smithi121 bash[36902]: audit 2022-04-22T23:09:17.952173+0000 mon.a (mon.0) 16172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:18 smithi115 bash[45495]: audit 2022-04-22T23:09:17.952173+0000 mon.a (mon.0) 16172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:18 smithi115 bash[46934]: audit 2022-04-22T23:09:17.952173+0000 mon.a (mon.0) 16172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:19 smithi121 bash[36902]: cluster 2022-04-22T23:09:18.828374+0000 mgr.y (mgr.34107) 11695 : cluster [DBG] pgmap v12236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:19 smithi121 bash[36902]: audit 2022-04-22T23:09:19.572511+0000 mon.a (mon.0) 16173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:19 smithi115 bash[45495]: cluster 2022-04-22T23:09:18.828374+0000 mgr.y (mgr.34107) 11695 : cluster [DBG] pgmap v12236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:19 smithi115 bash[45495]: audit 2022-04-22T23:09:19.572511+0000 mon.a (mon.0) 16173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:19 smithi115 bash[46934]: cluster 2022-04-22T23:09:18.828374+0000 mgr.y (mgr.34107) 11695 : cluster [DBG] pgmap v12236: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:20.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:19 smithi115 bash[46934]: audit 2022-04-22T23:09:19.572511+0000 mon.a (mon.0) 16173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:21 smithi121 bash[36902]: cluster 2022-04-22T23:09:20.829499+0000 mgr.y (mgr.34107) 11696 : cluster [DBG] pgmap v12237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:21 smithi121 bash[36902]: audit 2022-04-22T23:09:21.321861+0000 mon.a (mon.0) 16174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:22.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:21 smithi115 bash[45495]: cluster 2022-04-22T23:09:20.829499+0000 mgr.y (mgr.34107) 11696 : cluster [DBG] pgmap v12237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:21 smithi115 bash[45495]: audit 2022-04-22T23:09:21.321861+0000 mon.a (mon.0) 16174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:21 smithi115 bash[46934]: cluster 2022-04-22T23:09:20.829499+0000 mgr.y (mgr.34107) 11696 : cluster [DBG] pgmap v12237: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:21 smithi115 bash[46934]: audit 2022-04-22T23:09:21.321861+0000 mon.a (mon.0) 16174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:22.969Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:23 smithi121 bash[36902]: cluster 2022-04-22T23:09:22.830586+0000 mgr.y (mgr.34107) 11697 : cluster [DBG] pgmap v12238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:23.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:23 smithi121 bash[36902]: audit 2022-04-22T23:09:22.955081+0000 mon.a (mon.0) 16175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:23 smithi115 bash[46934]: cluster 2022-04-22T23:09:22.830586+0000 mgr.y (mgr.34107) 11697 : cluster [DBG] pgmap v12238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:23 smithi115 bash[46934]: audit 2022-04-22T23:09:22.955081+0000 mon.a (mon.0) 16175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:23 smithi115 bash[45495]: cluster 2022-04-22T23:09:22.830586+0000 mgr.y (mgr.34107) 11697 : cluster [DBG] pgmap v12238: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:23 smithi115 bash[45495]: audit 2022-04-22T23:09:22.955081+0000 mon.a (mon.0) 16175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:24 smithi121 bash[36902]: audit 2022-04-22T23:09:24.576362+0000 mon.a (mon.0) 16176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:24 smithi115 bash[45495]: audit 2022-04-22T23:09:24.576362+0000 mon.a (mon.0) 16176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:24.822Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:24 smithi115 bash[46934]: audit 2022-04-22T23:09:24.576362+0000 mon.a (mon.0) 16176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:25 smithi121 bash[36902]: cluster 2022-04-22T23:09:24.831714+0000 mgr.y (mgr.34107) 11698 : cluster [DBG] pgmap v12239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:26.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:25 smithi115 bash[46934]: cluster 2022-04-22T23:09:24.831714+0000 mgr.y (mgr.34107) 11698 : cluster [DBG] pgmap v12239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:26.024 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:25 smithi115 bash[45495]: cluster 2022-04-22T23:09:24.831714+0000 mgr.y (mgr.34107) 11698 : cluster [DBG] pgmap v12239: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:26 smithi121 bash[36902]: audit 2022-04-22T23:09:26.002827+0000 mon.a (mon.0) 16177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:09:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:26 smithi121 bash[36902]: audit 2022-04-22T23:09:26.004790+0000 mon.a (mon.0) 16178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:09:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:26 smithi121 bash[36902]: audit 2022-04-22T23:09:26.005780+0000 mon.a (mon.0) 16179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:09:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:26 smithi121 bash[36902]: audit 2022-04-22T23:09:26.324487+0000 mon.a (mon.0) 16180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:26 smithi121 bash[36902]: audit 2022-04-22T23:09:26.359408+0000 mon.a (mon.0) 16181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:09:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[45495]: audit 2022-04-22T23:09:26.002827+0000 mon.a (mon.0) 16177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:09:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[45495]: audit 2022-04-22T23:09:26.004790+0000 mon.a (mon.0) 16178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:09:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[45495]: audit 2022-04-22T23:09:26.005780+0000 mon.a (mon.0) 16179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:09:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[45495]: audit 2022-04-22T23:09:26.324487+0000 mon.a (mon.0) 16180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[45495]: audit 2022-04-22T23:09:26.359408+0000 mon.a (mon.0) 16181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:09:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[46934]: audit 2022-04-22T23:09:26.002827+0000 mon.a (mon.0) 16177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:09:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[46934]: audit 2022-04-22T23:09:26.004790+0000 mon.a (mon.0) 16178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:09:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[46934]: audit 2022-04-22T23:09:26.005780+0000 mon.a (mon.0) 16179 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:09:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[46934]: audit 2022-04-22T23:09:26.324487+0000 mon.a (mon.0) 16180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:26 smithi115 bash[46934]: audit 2022-04-22T23:09:26.359408+0000 mon.a (mon.0) 16181 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:09:28.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:27 smithi121 bash[36902]: cluster 2022-04-22T23:09:26.832778+0000 mgr.y (mgr.34107) 11699 : cluster [DBG] pgmap v12240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:28.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:27 smithi115 bash[45495]: cluster 2022-04-22T23:09:26.832778+0000 mgr.y (mgr.34107) 11699 : cluster [DBG] pgmap v12240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:27 smithi115 bash[46934]: cluster 2022-04-22T23:09:26.832778+0000 mgr.y (mgr.34107) 11699 : cluster [DBG] pgmap v12240: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:28 smithi121 bash[36902]: audit 2022-04-22T23:09:27.958222+0000 mon.a (mon.0) 16182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:28 smithi115 bash[45495]: audit 2022-04-22T23:09:27.958222+0000 mon.a (mon.0) 16182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:28 smithi115 bash[46934]: audit 2022-04-22T23:09:27.958222+0000 mon.a (mon.0) 16182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:29 smithi121 bash[36902]: cluster 2022-04-22T23:09:28.833976+0000 mgr.y (mgr.34107) 11700 : cluster [DBG] pgmap v12241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:29 smithi121 bash[36902]: audit 2022-04-22T23:09:29.579994+0000 mon.a (mon.0) 16183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:30.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:29 smithi115 bash[46934]: cluster 2022-04-22T23:09:28.833976+0000 mgr.y (mgr.34107) 11700 : cluster [DBG] pgmap v12241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:29 smithi115 bash[46934]: audit 2022-04-22T23:09:29.579994+0000 mon.a (mon.0) 16183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:29 smithi115 bash[45495]: cluster 2022-04-22T23:09:28.833976+0000 mgr.y (mgr.34107) 11700 : cluster [DBG] pgmap v12241: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:29 smithi115 bash[45495]: audit 2022-04-22T23:09:29.579994+0000 mon.a (mon.0) 16183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:31 smithi121 bash[36902]: cluster 2022-04-22T23:09:30.835100+0000 mgr.y (mgr.34107) 11701 : cluster [DBG] pgmap v12242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:31 smithi121 bash[36902]: audit 2022-04-22T23:09:31.327150+0000 mon.a (mon.0) 16184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:32.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:31 smithi115 bash[46934]: cluster 2022-04-22T23:09:30.835100+0000 mgr.y (mgr.34107) 11701 : cluster [DBG] pgmap v12242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:31 smithi115 bash[46934]: audit 2022-04-22T23:09:31.327150+0000 mon.a (mon.0) 16184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:31 smithi115 bash[45495]: cluster 2022-04-22T23:09:30.835100+0000 mgr.y (mgr.34107) 11701 : cluster [DBG] pgmap v12242: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:31 smithi115 bash[45495]: audit 2022-04-22T23:09:31.327150+0000 mon.a (mon.0) 16184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:32.971Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:33.441 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:09:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:33 smithi121 bash[36902]: cluster 2022-04-22T23:09:32.836215+0000 mgr.y (mgr.34107) 11702 : cluster [DBG] pgmap v12243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:33 smithi121 bash[36902]: audit 2022-04-22T23:09:32.961233+0000 mon.a (mon.0) 16185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:33 smithi115 bash[45495]: cluster 2022-04-22T23:09:32.836215+0000 mgr.y (mgr.34107) 11702 : cluster [DBG] pgmap v12243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:33 smithi115 bash[45495]: audit 2022-04-22T23:09:32.961233+0000 mon.a (mon.0) 16185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:33 smithi115 bash[46934]: cluster 2022-04-22T23:09:32.836215+0000 mgr.y (mgr.34107) 11702 : cluster [DBG] pgmap v12243: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:33 smithi115 bash[46934]: audit 2022-04-22T23:09:32.961233+0000 mon.a (mon.0) 16185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:33.793 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:09:33.793 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:09:33.794 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:09:33.795 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:09:33.796 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:09:33.796 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:09:34.196 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:09:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:09:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:09:34.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:34 smithi121 bash[36902]: audit 2022-04-22T23:09:33.420633+0000 mgr.y (mgr.34107) 11703 : audit [DBG] from='client.47457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:34 smithi121 bash[36902]: audit 2022-04-22T23:09:33.764354+0000 mgr.y (mgr.34107) 11704 : audit [DBG] from='client.57403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:34 smithi121 bash[36902]: audit 2022-04-22T23:09:34.193233+0000 mon.a (mon.0) 16186 : audit [DBG] from='client.? 172.21.15.115:0/4288024381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[45495]: audit 2022-04-22T23:09:33.420633+0000 mgr.y (mgr.34107) 11703 : audit [DBG] from='client.47457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[45495]: audit 2022-04-22T23:09:33.764354+0000 mgr.y (mgr.34107) 11704 : audit [DBG] from='client.57403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[45495]: audit 2022-04-22T23:09:34.193233+0000 mon.a (mon.0) 16186 : audit [DBG] from='client.? 172.21.15.115:0/4288024381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[46934]: audit 2022-04-22T23:09:33.420633+0000 mgr.y (mgr.34107) 11703 : audit [DBG] from='client.47457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[46934]: audit 2022-04-22T23:09:33.764354+0000 mgr.y (mgr.34107) 11704 : audit [DBG] from='client.57403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:09:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[46934]: audit 2022-04-22T23:09:34.193233+0000 mon.a (mon.0) 16186 : audit [DBG] from='client.? 172.21.15.115:0/4288024381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:09:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:34.826Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:35 smithi121 bash[36902]: audit 2022-04-22T23:09:34.583903+0000 mon.a (mon.0) 16187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:35 smithi121 bash[36902]: cluster 2022-04-22T23:09:34.837317+0000 mgr.y (mgr.34107) 11705 : cluster [DBG] pgmap v12244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:35.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:35 smithi115 bash[45495]: audit 2022-04-22T23:09:34.583903+0000 mon.a (mon.0) 16187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:35 smithi115 bash[45495]: cluster 2022-04-22T23:09:34.837317+0000 mgr.y (mgr.34107) 11705 : cluster [DBG] pgmap v12244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:35 smithi115 bash[46934]: audit 2022-04-22T23:09:34.583903+0000 mon.a (mon.0) 16187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:35 smithi115 bash[46934]: cluster 2022-04-22T23:09:34.837317+0000 mgr.y (mgr.34107) 11705 : cluster [DBG] pgmap v12244: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:36.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:36 smithi121 bash[36902]: audit 2022-04-22T23:09:36.330403+0000 mon.a (mon.0) 16188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:36 smithi115 bash[45495]: audit 2022-04-22T23:09:36.330403+0000 mon.a (mon.0) 16188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:36 smithi115 bash[46934]: audit 2022-04-22T23:09:36.330403+0000 mon.a (mon.0) 16188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:37.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:37 smithi121 bash[36902]: cluster 2022-04-22T23:09:36.838561+0000 mgr.y (mgr.34107) 11706 : cluster [DBG] pgmap v12245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:37 smithi115 bash[45495]: cluster 2022-04-22T23:09:36.838561+0000 mgr.y (mgr.34107) 11706 : cluster [DBG] pgmap v12245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:37 smithi115 bash[46934]: cluster 2022-04-22T23:09:36.838561+0000 mgr.y (mgr.34107) 11706 : cluster [DBG] pgmap v12245: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:38.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:38 smithi121 bash[36902]: audit 2022-04-22T23:09:37.964546+0000 mon.a (mon.0) 16189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:38 smithi115 bash[46934]: audit 2022-04-22T23:09:37.964546+0000 mon.a (mon.0) 16189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:38.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:38 smithi115 bash[45495]: audit 2022-04-22T23:09:37.964546+0000 mon.a (mon.0) 16189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:39.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:39 smithi121 bash[36902]: cluster 2022-04-22T23:09:38.839787+0000 mgr.y (mgr.34107) 11707 : cluster [DBG] pgmap v12246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:39 smithi115 bash[46934]: cluster 2022-04-22T23:09:38.839787+0000 mgr.y (mgr.34107) 11707 : cluster [DBG] pgmap v12246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:39 smithi115 bash[45495]: cluster 2022-04-22T23:09:38.839787+0000 mgr.y (mgr.34107) 11707 : cluster [DBG] pgmap v12246: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:40.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:40 smithi121 bash[36902]: audit 2022-04-22T23:09:39.587611+0000 mon.a (mon.0) 16190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:40 smithi115 bash[45495]: audit 2022-04-22T23:09:39.587611+0000 mon.a (mon.0) 16190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:40 smithi115 bash[46934]: audit 2022-04-22T23:09:39.587611+0000 mon.a (mon.0) 16190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:41.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:41 smithi121 bash[36902]: cluster 2022-04-22T23:09:40.840905+0000 mgr.y (mgr.34107) 11708 : cluster [DBG] pgmap v12247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:41 smithi121 bash[36902]: audit 2022-04-22T23:09:41.333343+0000 mon.a (mon.0) 16191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:41 smithi115 bash[46934]: cluster 2022-04-22T23:09:40.840905+0000 mgr.y (mgr.34107) 11708 : cluster [DBG] pgmap v12247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:41 smithi115 bash[46934]: audit 2022-04-22T23:09:41.333343+0000 mon.a (mon.0) 16191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:41 smithi115 bash[45495]: cluster 2022-04-22T23:09:40.840905+0000 mgr.y (mgr.34107) 11708 : cluster [DBG] pgmap v12247: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:41 smithi115 bash[45495]: audit 2022-04-22T23:09:41.333343+0000 mon.a (mon.0) 16191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:42.973Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:43 smithi121 bash[36902]: cluster 2022-04-22T23:09:42.842015+0000 mgr.y (mgr.34107) 11709 : cluster [DBG] pgmap v12248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:43 smithi121 bash[36902]: audit 2022-04-22T23:09:42.967689+0000 mon.a (mon.0) 16192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:43 smithi115 bash[45495]: cluster 2022-04-22T23:09:42.842015+0000 mgr.y (mgr.34107) 11709 : cluster [DBG] pgmap v12248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:43 smithi115 bash[45495]: audit 2022-04-22T23:09:42.967689+0000 mon.a (mon.0) 16192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:43 smithi115 bash[46934]: cluster 2022-04-22T23:09:42.842015+0000 mgr.y (mgr.34107) 11709 : cluster [DBG] pgmap v12248: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:43 smithi115 bash[46934]: audit 2022-04-22T23:09:42.967689+0000 mon.a (mon.0) 16192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:44 smithi121 bash[36902]: audit 2022-04-22T23:09:44.590927+0000 mon.a (mon.0) 16193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:44 smithi115 bash[45495]: audit 2022-04-22T23:09:44.590927+0000 mon.a (mon.0) 16193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:44.826Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:44 smithi115 bash[46934]: audit 2022-04-22T23:09:44.590927+0000 mon.a (mon.0) 16193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:45 smithi121 bash[36902]: cluster 2022-04-22T23:09:44.842840+0000 mgr.y (mgr.34107) 11710 : cluster [DBG] pgmap v12249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:45 smithi115 bash[45495]: cluster 2022-04-22T23:09:44.842840+0000 mgr.y (mgr.34107) 11710 : cluster [DBG] pgmap v12249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:45 smithi115 bash[46934]: cluster 2022-04-22T23:09:44.842840+0000 mgr.y (mgr.34107) 11710 : cluster [DBG] pgmap v12249: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:47.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:46 smithi121 bash[36902]: audit 2022-04-22T23:09:46.336599+0000 mon.a (mon.0) 16194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:46 smithi115 bash[45495]: audit 2022-04-22T23:09:46.336599+0000 mon.a (mon.0) 16194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:46 smithi115 bash[46934]: audit 2022-04-22T23:09:46.336599+0000 mon.a (mon.0) 16194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:48.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:47 smithi121 bash[36902]: cluster 2022-04-22T23:09:46.843963+0000 mgr.y (mgr.34107) 11711 : cluster [DBG] pgmap v12250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:47 smithi115 bash[45495]: cluster 2022-04-22T23:09:46.843963+0000 mgr.y (mgr.34107) 11711 : cluster [DBG] pgmap v12250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:47 smithi115 bash[46934]: cluster 2022-04-22T23:09:46.843963+0000 mgr.y (mgr.34107) 11711 : cluster [DBG] pgmap v12250: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:48 smithi121 bash[36902]: audit 2022-04-22T23:09:47.970762+0000 mon.a (mon.0) 16195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:48 smithi115 bash[45495]: audit 2022-04-22T23:09:47.970762+0000 mon.a (mon.0) 16195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:48 smithi115 bash[46934]: audit 2022-04-22T23:09:47.970762+0000 mon.a (mon.0) 16195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:49 smithi121 bash[36902]: cluster 2022-04-22T23:09:48.845097+0000 mgr.y (mgr.34107) 11712 : cluster [DBG] pgmap v12251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:49 smithi121 bash[36902]: audit 2022-04-22T23:09:49.594823+0000 mon.a (mon.0) 16196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:50.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:49 smithi115 bash[45495]: cluster 2022-04-22T23:09:48.845097+0000 mgr.y (mgr.34107) 11712 : cluster [DBG] pgmap v12251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:49 smithi115 bash[45495]: audit 2022-04-22T23:09:49.594823+0000 mon.a (mon.0) 16196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:49 smithi115 bash[46934]: cluster 2022-04-22T23:09:48.845097+0000 mgr.y (mgr.34107) 11712 : cluster [DBG] pgmap v12251: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:49 smithi115 bash[46934]: audit 2022-04-22T23:09:49.594823+0000 mon.a (mon.0) 16196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:51 smithi121 bash[36902]: cluster 2022-04-22T23:09:50.846218+0000 mgr.y (mgr.34107) 11713 : cluster [DBG] pgmap v12252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:51 smithi121 bash[36902]: audit 2022-04-22T23:09:51.339608+0000 mon.a (mon.0) 16197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:52.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:51 smithi115 bash[45495]: cluster 2022-04-22T23:09:50.846218+0000 mgr.y (mgr.34107) 11713 : cluster [DBG] pgmap v12252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:51 smithi115 bash[45495]: audit 2022-04-22T23:09:51.339608+0000 mon.a (mon.0) 16197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:51 smithi115 bash[46934]: cluster 2022-04-22T23:09:50.846218+0000 mgr.y (mgr.34107) 11713 : cluster [DBG] pgmap v12252: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:51 smithi115 bash[46934]: audit 2022-04-22T23:09:51.339608+0000 mon.a (mon.0) 16197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:52.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:53 smithi121 bash[36902]: cluster 2022-04-22T23:09:52.847378+0000 mgr.y (mgr.34107) 11714 : cluster [DBG] pgmap v12253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:53 smithi121 bash[36902]: audit 2022-04-22T23:09:52.973926+0000 mon.a (mon.0) 16198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:53 smithi115 bash[45495]: cluster 2022-04-22T23:09:52.847378+0000 mgr.y (mgr.34107) 11714 : cluster [DBG] pgmap v12253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:53 smithi115 bash[45495]: audit 2022-04-22T23:09:52.973926+0000 mon.a (mon.0) 16198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:53 smithi115 bash[46934]: cluster 2022-04-22T23:09:52.847378+0000 mgr.y (mgr.34107) 11714 : cluster [DBG] pgmap v12253: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:53 smithi115 bash[46934]: audit 2022-04-22T23:09:52.973926+0000 mon.a (mon.0) 16198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:54 smithi121 bash[36902]: audit 2022-04-22T23:09:54.598838+0000 mon.a (mon.0) 16199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:54 smithi115 bash[45495]: audit 2022-04-22T23:09:54.598838+0000 mon.a (mon.0) 16199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:09:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:09:54.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:09:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:54 smithi115 bash[46934]: audit 2022-04-22T23:09:54.598838+0000 mon.a (mon.0) 16199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:09:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:55 smithi121 bash[36902]: cluster 2022-04-22T23:09:54.848419+0000 mgr.y (mgr.34107) 11715 : cluster [DBG] pgmap v12254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:55 smithi115 bash[45495]: cluster 2022-04-22T23:09:54.848419+0000 mgr.y (mgr.34107) 11715 : cluster [DBG] pgmap v12254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:55 smithi115 bash[46934]: cluster 2022-04-22T23:09:54.848419+0000 mgr.y (mgr.34107) 11715 : cluster [DBG] pgmap v12254: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:57.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:56 smithi121 bash[36902]: audit 2022-04-22T23:09:56.342649+0000 mon.a (mon.0) 16200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:57.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:56 smithi115 bash[45495]: audit 2022-04-22T23:09:56.342649+0000 mon.a (mon.0) 16200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:56 smithi115 bash[46934]: audit 2022-04-22T23:09:56.342649+0000 mon.a (mon.0) 16200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:09:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:57 smithi121 bash[36902]: cluster 2022-04-22T23:09:56.849551+0000 mgr.y (mgr.34107) 11716 : cluster [DBG] pgmap v12255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:57 smithi115 bash[45495]: cluster 2022-04-22T23:09:56.849551+0000 mgr.y (mgr.34107) 11716 : cluster [DBG] pgmap v12255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:57 smithi115 bash[46934]: cluster 2022-04-22T23:09:56.849551+0000 mgr.y (mgr.34107) 11716 : cluster [DBG] pgmap v12255: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:09:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:58 smithi121 bash[36902]: audit 2022-04-22T23:09:57.977255+0000 mon.a (mon.0) 16201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:58 smithi115 bash[45495]: audit 2022-04-22T23:09:57.977255+0000 mon.a (mon.0) 16201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:09:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:58 smithi115 bash[46934]: audit 2022-04-22T23:09:57.977255+0000 mon.a (mon.0) 16201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:59 smithi121 bash[36902]: cluster 2022-04-22T23:09:58.850754+0000 mgr.y (mgr.34107) 11717 : cluster [DBG] pgmap v12256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:09:59 smithi121 bash[36902]: audit 2022-04-22T23:09:59.602463+0000 mon.a (mon.0) 16202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:59 smithi115 bash[45495]: cluster 2022-04-22T23:09:58.850754+0000 mgr.y (mgr.34107) 11717 : cluster [DBG] pgmap v12256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:09:59 smithi115 bash[45495]: audit 2022-04-22T23:09:59.602463+0000 mon.a (mon.0) 16202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:59 smithi115 bash[46934]: cluster 2022-04-22T23:09:58.850754+0000 mgr.y (mgr.34107) 11717 : cluster [DBG] pgmap v12256: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:09:59 smithi115 bash[46934]: audit 2022-04-22T23:09:59.602463+0000 mon.a (mon.0) 16202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:01.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:00 smithi121 bash[36902]: cluster 2022-04-22T23:10:00.000164+0000 mon.a (mon.0) 16203 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:10:01.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:00 smithi115 bash[45495]: cluster 2022-04-22T23:10:00.000164+0000 mon.a (mon.0) 16203 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:10:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:00 smithi115 bash[46934]: cluster 2022-04-22T23:10:00.000164+0000 mon.a (mon.0) 16203 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:10:02.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:01 smithi121 bash[36902]: cluster 2022-04-22T23:10:00.851922+0000 mgr.y (mgr.34107) 11718 : cluster [DBG] pgmap v12257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:01 smithi121 bash[36902]: audit 2022-04-22T23:10:01.345454+0000 mon.a (mon.0) 16204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:01 smithi115 bash[45495]: cluster 2022-04-22T23:10:00.851922+0000 mgr.y (mgr.34107) 11718 : cluster [DBG] pgmap v12257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:01 smithi115 bash[45495]: audit 2022-04-22T23:10:01.345454+0000 mon.a (mon.0) 16204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:01 smithi115 bash[46934]: cluster 2022-04-22T23:10:00.851922+0000 mgr.y (mgr.34107) 11718 : cluster [DBG] pgmap v12257: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:01 smithi115 bash[46934]: audit 2022-04-22T23:10:01.345454+0000 mon.a (mon.0) 16204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:02.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:03 smithi121 bash[36902]: cluster 2022-04-22T23:10:02.853027+0000 mgr.y (mgr.34107) 11719 : cluster [DBG] pgmap v12258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:03 smithi121 bash[36902]: audit 2022-04-22T23:10:02.980106+0000 mon.a (mon.0) 16205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:03 smithi115 bash[45495]: cluster 2022-04-22T23:10:02.853027+0000 mgr.y (mgr.34107) 11719 : cluster [DBG] pgmap v12258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:03 smithi115 bash[45495]: audit 2022-04-22T23:10:02.980106+0000 mon.a (mon.0) 16205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:03 smithi115 bash[46934]: cluster 2022-04-22T23:10:02.853027+0000 mgr.y (mgr.34107) 11719 : cluster [DBG] pgmap v12258: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:03 smithi115 bash[46934]: audit 2022-04-22T23:10:02.980106+0000 mon.a (mon.0) 16205 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:04.567 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:10:04.913 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:10:04.913 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:10:04.914 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:10:04.915 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:10:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:04 smithi121 bash[36902]: audit 2022-04-22T23:10:04.606241+0000 mon.a (mon.0) 16206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:04 smithi115 bash[45495]: audit 2022-04-22T23:10:04.606241+0000 mon.a (mon.0) 16206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:04 smithi115 bash[46934]: audit 2022-04-22T23:10:04.606241+0000 mon.a (mon.0) 16206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:04.824Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:05.316 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:10:05.317 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:10:05.317 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:10:05.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:05.317 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:10:05.317 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:10:05.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:05.318 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:10:05.318 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:10:05.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:05.319 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:10:05.319 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:10:05.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:05.320 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:10:05.320 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:10:05.320 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:10:05.320 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:10:06.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:05 smithi121 bash[36902]: audit 2022-04-22T23:10:04.547668+0000 mgr.y (mgr.34107) 11720 : audit [DBG] from='client.57415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:05 smithi121 bash[36902]: cluster 2022-04-22T23:10:04.854068+0000 mgr.y (mgr.34107) 11721 : cluster [DBG] pgmap v12259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:05 smithi121 bash[36902]: audit 2022-04-22T23:10:04.889153+0000 mgr.y (mgr.34107) 11722 : audit [DBG] from='client.47481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:05 smithi121 bash[36902]: audit 2022-04-22T23:10:05.312430+0000 mon.c (mon.1) 394 : audit [DBG] from='client.? 172.21.15.115:0/1834972013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[45495]: audit 2022-04-22T23:10:04.547668+0000 mgr.y (mgr.34107) 11720 : audit [DBG] from='client.57415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[45495]: cluster 2022-04-22T23:10:04.854068+0000 mgr.y (mgr.34107) 11721 : cluster [DBG] pgmap v12259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[45495]: audit 2022-04-22T23:10:04.889153+0000 mgr.y (mgr.34107) 11722 : audit [DBG] from='client.47481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[45495]: audit 2022-04-22T23:10:05.312430+0000 mon.c (mon.1) 394 : audit [DBG] from='client.? 172.21.15.115:0/1834972013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[46934]: audit 2022-04-22T23:10:04.547668+0000 mgr.y (mgr.34107) 11720 : audit [DBG] from='client.57415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[46934]: cluster 2022-04-22T23:10:04.854068+0000 mgr.y (mgr.34107) 11721 : cluster [DBG] pgmap v12259: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:06.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[46934]: audit 2022-04-22T23:10:04.889153+0000 mgr.y (mgr.34107) 11722 : audit [DBG] from='client.47481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:06.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:05 smithi115 bash[46934]: audit 2022-04-22T23:10:05.312430+0000 mon.c (mon.1) 394 : audit [DBG] from='client.? 172.21.15.115:0/1834972013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:07.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:06 smithi121 bash[36902]: audit 2022-04-22T23:10:06.348832+0000 mon.a (mon.0) 16207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:06 smithi115 bash[45495]: audit 2022-04-22T23:10:06.348832+0000 mon.a (mon.0) 16207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:07.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:06 smithi115 bash[46934]: audit 2022-04-22T23:10:06.348832+0000 mon.a (mon.0) 16207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:07 smithi121 bash[36902]: cluster 2022-04-22T23:10:06.855178+0000 mgr.y (mgr.34107) 11723 : cluster [DBG] pgmap v12260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:07 smithi115 bash[45495]: cluster 2022-04-22T23:10:06.855178+0000 mgr.y (mgr.34107) 11723 : cluster [DBG] pgmap v12260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:07 smithi115 bash[46934]: cluster 2022-04-22T23:10:06.855178+0000 mgr.y (mgr.34107) 11723 : cluster [DBG] pgmap v12260: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:09.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:08 smithi121 bash[36902]: audit 2022-04-22T23:10:07.982728+0000 mon.a (mon.0) 16208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:08 smithi115 bash[45495]: audit 2022-04-22T23:10:07.982728+0000 mon.a (mon.0) 16208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:08 smithi115 bash[46934]: audit 2022-04-22T23:10:07.982728+0000 mon.a (mon.0) 16208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:10.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:09 smithi121 bash[36902]: cluster 2022-04-22T23:10:08.856431+0000 mgr.y (mgr.34107) 11724 : cluster [DBG] pgmap v12261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:09 smithi121 bash[36902]: audit 2022-04-22T23:10:09.609834+0000 mon.a (mon.0) 16209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:09 smithi115 bash[45495]: cluster 2022-04-22T23:10:08.856431+0000 mgr.y (mgr.34107) 11724 : cluster [DBG] pgmap v12261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:09 smithi115 bash[45495]: audit 2022-04-22T23:10:09.609834+0000 mon.a (mon.0) 16209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:09 smithi115 bash[46934]: cluster 2022-04-22T23:10:08.856431+0000 mgr.y (mgr.34107) 11724 : cluster [DBG] pgmap v12261: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:09 smithi115 bash[46934]: audit 2022-04-22T23:10:09.609834+0000 mon.a (mon.0) 16209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:11 smithi121 bash[36902]: cluster 2022-04-22T23:10:10.857559+0000 mgr.y (mgr.34107) 11725 : cluster [DBG] pgmap v12262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:11 smithi121 bash[36902]: audit 2022-04-22T23:10:11.351759+0000 mon.a (mon.0) 16210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:11 smithi115 bash[45495]: cluster 2022-04-22T23:10:10.857559+0000 mgr.y (mgr.34107) 11725 : cluster [DBG] pgmap v12262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:11 smithi115 bash[45495]: audit 2022-04-22T23:10:11.351759+0000 mon.a (mon.0) 16210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:11 smithi115 bash[46934]: cluster 2022-04-22T23:10:10.857559+0000 mgr.y (mgr.34107) 11725 : cluster [DBG] pgmap v12262: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:11 smithi115 bash[46934]: audit 2022-04-22T23:10:11.351759+0000 mon.a (mon.0) 16210 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:12.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:13 smithi121 bash[36902]: cluster 2022-04-22T23:10:12.858664+0000 mgr.y (mgr.34107) 11726 : cluster [DBG] pgmap v12263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:13 smithi121 bash[36902]: audit 2022-04-22T23:10:12.985770+0000 mon.a (mon.0) 16211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:13 smithi115 bash[45495]: cluster 2022-04-22T23:10:12.858664+0000 mgr.y (mgr.34107) 11726 : cluster [DBG] pgmap v12263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:13 smithi115 bash[45495]: audit 2022-04-22T23:10:12.985770+0000 mon.a (mon.0) 16211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:13 smithi115 bash[46934]: cluster 2022-04-22T23:10:12.858664+0000 mgr.y (mgr.34107) 11726 : cluster [DBG] pgmap v12263: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:13 smithi115 bash[46934]: audit 2022-04-22T23:10:12.985770+0000 mon.a (mon.0) 16211 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:14 smithi121 bash[36902]: audit 2022-04-22T23:10:14.613454+0000 mon.a (mon.0) 16212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:15.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:14 smithi115 bash[45495]: audit 2022-04-22T23:10:14.613454+0000 mon.a (mon.0) 16212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:15.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:14.828Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:14 smithi115 bash[46934]: audit 2022-04-22T23:10:14.613454+0000 mon.a (mon.0) 16212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:15 smithi121 bash[36902]: cluster 2022-04-22T23:10:14.859824+0000 mgr.y (mgr.34107) 11727 : cluster [DBG] pgmap v12264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:16.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:15 smithi115 bash[45495]: cluster 2022-04-22T23:10:14.859824+0000 mgr.y (mgr.34107) 11727 : cluster [DBG] pgmap v12264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:15 smithi115 bash[46934]: cluster 2022-04-22T23:10:14.859824+0000 mgr.y (mgr.34107) 11727 : cluster [DBG] pgmap v12264: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:17.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:16 smithi121 bash[36902]: audit 2022-04-22T23:10:16.354851+0000 mon.a (mon.0) 16213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:16 smithi115 bash[45495]: audit 2022-04-22T23:10:16.354851+0000 mon.a (mon.0) 16213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:16 smithi115 bash[46934]: audit 2022-04-22T23:10:16.354851+0000 mon.a (mon.0) 16213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:17 smithi121 bash[36902]: cluster 2022-04-22T23:10:16.860915+0000 mgr.y (mgr.34107) 11728 : cluster [DBG] pgmap v12265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:17 smithi121 bash[36902]: audit 2022-04-22T23:10:17.230024+0000 mon.a (mon.0) 16214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:10:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:17 smithi121 bash[36902]: audit 2022-04-22T23:10:17.230786+0000 mon.a (mon.0) 16215 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[45495]: cluster 2022-04-22T23:10:16.860915+0000 mgr.y (mgr.34107) 11728 : cluster [DBG] pgmap v12265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[45495]: audit 2022-04-22T23:10:17.230024+0000 mon.a (mon.0) 16214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[45495]: audit 2022-04-22T23:10:17.230786+0000 mon.a (mon.0) 16215 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[46934]: cluster 2022-04-22T23:10:16.860915+0000 mgr.y (mgr.34107) 11728 : cluster [DBG] pgmap v12265: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[46934]: audit 2022-04-22T23:10:17.230024+0000 mon.a (mon.0) 16214 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:10:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:17 smithi115 bash[46934]: audit 2022-04-22T23:10:17.230786+0000 mon.a (mon.0) 16215 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:10:19.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:18 smithi121 bash[36902]: audit 2022-04-22T23:10:17.988628+0000 mon.a (mon.0) 16216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:19.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:18 smithi115 bash[45495]: audit 2022-04-22T23:10:17.988628+0000 mon.a (mon.0) 16216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:18 smithi115 bash[46934]: audit 2022-04-22T23:10:17.988628+0000 mon.a (mon.0) 16216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:19 smithi121 bash[36902]: cluster 2022-04-22T23:10:18.862123+0000 mgr.y (mgr.34107) 11729 : cluster [DBG] pgmap v12266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:19 smithi121 bash[36902]: audit 2022-04-22T23:10:19.617110+0000 mon.a (mon.0) 16217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:19 smithi115 bash[45495]: cluster 2022-04-22T23:10:18.862123+0000 mgr.y (mgr.34107) 11729 : cluster [DBG] pgmap v12266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:19 smithi115 bash[45495]: audit 2022-04-22T23:10:19.617110+0000 mon.a (mon.0) 16217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:19 smithi115 bash[46934]: cluster 2022-04-22T23:10:18.862123+0000 mgr.y (mgr.34107) 11729 : cluster [DBG] pgmap v12266: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:19 smithi115 bash[46934]: audit 2022-04-22T23:10:19.617110+0000 mon.a (mon.0) 16217 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:21 smithi121 bash[36902]: cluster 2022-04-22T23:10:20.863237+0000 mgr.y (mgr.34107) 11730 : cluster [DBG] pgmap v12267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:21 smithi121 bash[36902]: audit 2022-04-22T23:10:21.357949+0000 mon.a (mon.0) 16218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:21 smithi115 bash[46934]: cluster 2022-04-22T23:10:20.863237+0000 mgr.y (mgr.34107) 11730 : cluster [DBG] pgmap v12267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:21 smithi115 bash[46934]: audit 2022-04-22T23:10:21.357949+0000 mon.a (mon.0) 16218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:21 smithi115 bash[45495]: cluster 2022-04-22T23:10:20.863237+0000 mgr.y (mgr.34107) 11730 : cluster [DBG] pgmap v12267: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:21 smithi115 bash[45495]: audit 2022-04-22T23:10:21.357949+0000 mon.a (mon.0) 16218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:22.976Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:23 smithi121 bash[36902]: cluster 2022-04-22T23:10:22.864389+0000 mgr.y (mgr.34107) 11731 : cluster [DBG] pgmap v12268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:23 smithi121 bash[36902]: audit 2022-04-22T23:10:22.991804+0000 mon.a (mon.0) 16219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:23 smithi115 bash[46934]: cluster 2022-04-22T23:10:22.864389+0000 mgr.y (mgr.34107) 11731 : cluster [DBG] pgmap v12268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:23 smithi115 bash[46934]: audit 2022-04-22T23:10:22.991804+0000 mon.a (mon.0) 16219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:23 smithi115 bash[45495]: cluster 2022-04-22T23:10:22.864389+0000 mgr.y (mgr.34107) 11731 : cluster [DBG] pgmap v12268: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:23 smithi115 bash[45495]: audit 2022-04-22T23:10:22.991804+0000 mon.a (mon.0) 16219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:24 smithi121 bash[36902]: audit 2022-04-22T23:10:24.620852+0000 mon.a (mon.0) 16220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:25.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:24 smithi115 bash[46934]: audit 2022-04-22T23:10:24.620852+0000 mon.a (mon.0) 16220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:24.829Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:24 smithi115 bash[45495]: audit 2022-04-22T23:10:24.620852+0000 mon.a (mon.0) 16220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:25 smithi121 bash[36902]: cluster 2022-04-22T23:10:24.865485+0000 mgr.y (mgr.34107) 11732 : cluster [DBG] pgmap v12269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:25 smithi115 bash[45495]: cluster 2022-04-22T23:10:24.865485+0000 mgr.y (mgr.34107) 11732 : cluster [DBG] pgmap v12269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:25 smithi115 bash[46934]: cluster 2022-04-22T23:10:24.865485+0000 mgr.y (mgr.34107) 11732 : cluster [DBG] pgmap v12269: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:26 smithi121 bash[36902]: audit 2022-04-22T23:10:26.361008+0000 mon.a (mon.0) 16221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:26 smithi121 bash[36902]: audit 2022-04-22T23:10:26.367620+0000 mon.a (mon.0) 16222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:10:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:26 smithi121 bash[36902]: audit 2022-04-22T23:10:26.370343+0000 mon.a (mon.0) 16223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:10:27.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:26 smithi121 bash[36902]: audit 2022-04-22T23:10:26.371956+0000 mon.a (mon.0) 16224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:10:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[46934]: audit 2022-04-22T23:10:26.361008+0000 mon.a (mon.0) 16221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[46934]: audit 2022-04-22T23:10:26.367620+0000 mon.a (mon.0) 16222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:10:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[46934]: audit 2022-04-22T23:10:26.370343+0000 mon.a (mon.0) 16223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:10:27.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[46934]: audit 2022-04-22T23:10:26.371956+0000 mon.a (mon.0) 16224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:10:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[45495]: audit 2022-04-22T23:10:26.361008+0000 mon.a (mon.0) 16221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:27.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[45495]: audit 2022-04-22T23:10:26.367620+0000 mon.a (mon.0) 16222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:10:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[45495]: audit 2022-04-22T23:10:26.370343+0000 mon.a (mon.0) 16223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:10:27.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:26 smithi115 bash[45495]: audit 2022-04-22T23:10:26.371956+0000 mon.a (mon.0) 16224 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:10:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:27 smithi121 bash[36902]: audit 2022-04-22T23:10:26.733234+0000 mon.a (mon.0) 16225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:10:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:27 smithi121 bash[36902]: cluster 2022-04-22T23:10:26.866585+0000 mgr.y (mgr.34107) 11733 : cluster [DBG] pgmap v12270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:27 smithi115 bash[46934]: audit 2022-04-22T23:10:26.733234+0000 mon.a (mon.0) 16225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:10:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:27 smithi115 bash[46934]: cluster 2022-04-22T23:10:26.866585+0000 mgr.y (mgr.34107) 11733 : cluster [DBG] pgmap v12270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:27 smithi115 bash[45495]: audit 2022-04-22T23:10:26.733234+0000 mon.a (mon.0) 16225 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:10:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:27 smithi115 bash[45495]: cluster 2022-04-22T23:10:26.866585+0000 mgr.y (mgr.34107) 11733 : cluster [DBG] pgmap v12270: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:28 smithi121 bash[36902]: audit 2022-04-22T23:10:27.994629+0000 mon.a (mon.0) 16226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:28 smithi115 bash[45495]: audit 2022-04-22T23:10:27.994629+0000 mon.a (mon.0) 16226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:28 smithi115 bash[46934]: audit 2022-04-22T23:10:27.994629+0000 mon.a (mon.0) 16226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:30.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:29 smithi121 bash[36902]: cluster 2022-04-22T23:10:28.867777+0000 mgr.y (mgr.34107) 11734 : cluster [DBG] pgmap v12271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:29 smithi121 bash[36902]: audit 2022-04-22T23:10:29.624606+0000 mon.a (mon.0) 16227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:29 smithi115 bash[46934]: cluster 2022-04-22T23:10:28.867777+0000 mgr.y (mgr.34107) 11734 : cluster [DBG] pgmap v12271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:29 smithi115 bash[46934]: audit 2022-04-22T23:10:29.624606+0000 mon.a (mon.0) 16227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:29 smithi115 bash[45495]: cluster 2022-04-22T23:10:28.867777+0000 mgr.y (mgr.34107) 11734 : cluster [DBG] pgmap v12271: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:30.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:29 smithi115 bash[45495]: audit 2022-04-22T23:10:29.624606+0000 mon.a (mon.0) 16227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:31 smithi121 bash[36902]: cluster 2022-04-22T23:10:30.868789+0000 mgr.y (mgr.34107) 11735 : cluster [DBG] pgmap v12272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:31 smithi121 bash[36902]: audit 2022-04-22T23:10:31.363965+0000 mon.a (mon.0) 16228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:31 smithi115 bash[45495]: cluster 2022-04-22T23:10:30.868789+0000 mgr.y (mgr.34107) 11735 : cluster [DBG] pgmap v12272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:31 smithi115 bash[45495]: audit 2022-04-22T23:10:31.363965+0000 mon.a (mon.0) 16228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:31 smithi115 bash[46934]: cluster 2022-04-22T23:10:30.868789+0000 mgr.y (mgr.34107) 11735 : cluster [DBG] pgmap v12272: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:31 smithi115 bash[46934]: audit 2022-04-22T23:10:31.363965+0000 mon.a (mon.0) 16228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:32.975Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:33 smithi121 bash[36902]: cluster 2022-04-22T23:10:32.869798+0000 mgr.y (mgr.34107) 11736 : cluster [DBG] pgmap v12273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:33 smithi121 bash[36902]: audit 2022-04-22T23:10:32.997594+0000 mon.a (mon.0) 16229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:33 smithi115 bash[45495]: cluster 2022-04-22T23:10:32.869798+0000 mgr.y (mgr.34107) 11736 : cluster [DBG] pgmap v12273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:33 smithi115 bash[45495]: audit 2022-04-22T23:10:32.997594+0000 mon.a (mon.0) 16229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:33 smithi115 bash[46934]: cluster 2022-04-22T23:10:32.869798+0000 mgr.y (mgr.34107) 11736 : cluster [DBG] pgmap v12273: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:33 smithi115 bash[46934]: audit 2022-04-22T23:10:32.997594+0000 mon.a (mon.0) 16229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:34 smithi121 bash[36902]: audit 2022-04-22T23:10:34.628190+0000 mon.a (mon.0) 16230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:34 smithi115 bash[45495]: audit 2022-04-22T23:10:34.628190+0000 mon.a (mon.0) 16230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:34.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:34 smithi115 bash[46934]: audit 2022-04-22T23:10:34.628190+0000 mon.a (mon.0) 16230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:35.686 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:10:36.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:35 smithi121 bash[36902]: cluster 2022-04-22T23:10:34.870875+0000 mgr.y (mgr.34107) 11737 : cluster [DBG] pgmap v12274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:36.020 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:10:36.020 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:10:36.020 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:10:36.020 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:10:36.020 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:10:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:10:36.022 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:10:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:35 smithi115 bash[45495]: cluster 2022-04-22T23:10:34.870875+0000 mgr.y (mgr.34107) 11737 : cluster [DBG] pgmap v12274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:35 smithi115 bash[46934]: cluster 2022-04-22T23:10:34.870875+0000 mgr.y (mgr.34107) 11737 : cluster [DBG] pgmap v12274: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:10:36.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:10:36.456 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:10:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:36 smithi121 bash[36902]: audit 2022-04-22T23:10:35.667565+0000 mgr.y (mgr.34107) 11738 : audit [DBG] from='client.47493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:36 smithi121 bash[36902]: audit 2022-04-22T23:10:36.003121+0000 mgr.y (mgr.34107) 11739 : audit [DBG] from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:36 smithi121 bash[36902]: audit 2022-04-22T23:10:36.367039+0000 mon.a (mon.0) 16231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:37.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:36 smithi121 bash[36902]: audit 2022-04-22T23:10:36.451089+0000 mon.c (mon.1) 395 : audit [DBG] from='client.? 172.21.15.115:0/1396439988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[46934]: audit 2022-04-22T23:10:35.667565+0000 mgr.y (mgr.34107) 11738 : audit [DBG] from='client.47493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[46934]: audit 2022-04-22T23:10:36.003121+0000 mgr.y (mgr.34107) 11739 : audit [DBG] from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[46934]: audit 2022-04-22T23:10:36.367039+0000 mon.a (mon.0) 16231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[46934]: audit 2022-04-22T23:10:36.451089+0000 mon.c (mon.1) 395 : audit [DBG] from='client.? 172.21.15.115:0/1396439988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[45495]: audit 2022-04-22T23:10:35.667565+0000 mgr.y (mgr.34107) 11738 : audit [DBG] from='client.47493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[45495]: audit 2022-04-22T23:10:36.003121+0000 mgr.y (mgr.34107) 11739 : audit [DBG] from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:10:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[45495]: audit 2022-04-22T23:10:36.367039+0000 mon.a (mon.0) 16231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:37.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:36 smithi115 bash[45495]: audit 2022-04-22T23:10:36.451089+0000 mon.c (mon.1) 395 : audit [DBG] from='client.? 172.21.15.115:0/1396439988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:10:38.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:37 smithi121 bash[36902]: cluster 2022-04-22T23:10:36.871994+0000 mgr.y (mgr.34107) 11740 : cluster [DBG] pgmap v12275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:38.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:37 smithi115 bash[45495]: cluster 2022-04-22T23:10:36.871994+0000 mgr.y (mgr.34107) 11740 : cluster [DBG] pgmap v12275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:37 smithi115 bash[46934]: cluster 2022-04-22T23:10:36.871994+0000 mgr.y (mgr.34107) 11740 : cluster [DBG] pgmap v12275: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:39.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:38 smithi121 bash[36902]: audit 2022-04-22T23:10:38.000533+0000 mon.a (mon.0) 16232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:39.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:38 smithi115 bash[45495]: audit 2022-04-22T23:10:38.000533+0000 mon.a (mon.0) 16232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:38 smithi115 bash[46934]: audit 2022-04-22T23:10:38.000533+0000 mon.a (mon.0) 16232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:40.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:39 smithi121 bash[36902]: cluster 2022-04-22T23:10:38.873103+0000 mgr.y (mgr.34107) 11741 : cluster [DBG] pgmap v12276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:39 smithi121 bash[36902]: audit 2022-04-22T23:10:39.631882+0000 mon.a (mon.0) 16233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:39 smithi115 bash[45495]: cluster 2022-04-22T23:10:38.873103+0000 mgr.y (mgr.34107) 11741 : cluster [DBG] pgmap v12276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:39 smithi115 bash[45495]: audit 2022-04-22T23:10:39.631882+0000 mon.a (mon.0) 16233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:39 smithi115 bash[46934]: cluster 2022-04-22T23:10:38.873103+0000 mgr.y (mgr.34107) 11741 : cluster [DBG] pgmap v12276: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:40.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:39 smithi115 bash[46934]: audit 2022-04-22T23:10:39.631882+0000 mon.a (mon.0) 16233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:41 smithi121 bash[36902]: cluster 2022-04-22T23:10:40.874185+0000 mgr.y (mgr.34107) 11742 : cluster [DBG] pgmap v12277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:41 smithi121 bash[36902]: audit 2022-04-22T23:10:41.369880+0000 mon.a (mon.0) 16234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:42.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:41 smithi115 bash[45495]: cluster 2022-04-22T23:10:40.874185+0000 mgr.y (mgr.34107) 11742 : cluster [DBG] pgmap v12277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:41 smithi115 bash[45495]: audit 2022-04-22T23:10:41.369880+0000 mon.a (mon.0) 16234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:41 smithi115 bash[46934]: cluster 2022-04-22T23:10:40.874185+0000 mgr.y (mgr.34107) 11742 : cluster [DBG] pgmap v12277: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:41 smithi115 bash[46934]: audit 2022-04-22T23:10:41.369880+0000 mon.a (mon.0) 16234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:42.976Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:43.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:43 smithi121 bash[36902]: cluster 2022-04-22T23:10:42.875273+0000 mgr.y (mgr.34107) 11743 : cluster [DBG] pgmap v12278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:43 smithi121 bash[36902]: audit 2022-04-22T23:10:43.003433+0000 mon.a (mon.0) 16235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:43 smithi115 bash[45495]: cluster 2022-04-22T23:10:42.875273+0000 mgr.y (mgr.34107) 11743 : cluster [DBG] pgmap v12278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:43 smithi115 bash[45495]: audit 2022-04-22T23:10:43.003433+0000 mon.a (mon.0) 16235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:43 smithi115 bash[46934]: cluster 2022-04-22T23:10:42.875273+0000 mgr.y (mgr.34107) 11743 : cluster [DBG] pgmap v12278: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:43 smithi115 bash[46934]: audit 2022-04-22T23:10:43.003433+0000 mon.a (mon.0) 16235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:44 smithi121 bash[36902]: audit 2022-04-22T23:10:44.635338+0000 mon.a (mon.0) 16236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:44 smithi115 bash[45495]: audit 2022-04-22T23:10:44.635338+0000 mon.a (mon.0) 16236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:44.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:45.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:44 smithi115 bash[46934]: audit 2022-04-22T23:10:44.635338+0000 mon.a (mon.0) 16236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:45 smithi121 bash[36902]: cluster 2022-04-22T23:10:44.876387+0000 mgr.y (mgr.34107) 11744 : cluster [DBG] pgmap v12279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:46.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:45 smithi115 bash[45495]: cluster 2022-04-22T23:10:44.876387+0000 mgr.y (mgr.34107) 11744 : cluster [DBG] pgmap v12279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:45 smithi115 bash[46934]: cluster 2022-04-22T23:10:44.876387+0000 mgr.y (mgr.34107) 11744 : cluster [DBG] pgmap v12279: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:46 smithi121 bash[36902]: audit 2022-04-22T23:10:46.372927+0000 mon.a (mon.0) 16237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:47.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:46 smithi115 bash[45495]: audit 2022-04-22T23:10:46.372927+0000 mon.a (mon.0) 16237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:46 smithi115 bash[46934]: audit 2022-04-22T23:10:46.372927+0000 mon.a (mon.0) 16237 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:47 smithi121 bash[36902]: cluster 2022-04-22T23:10:46.877495+0000 mgr.y (mgr.34107) 11745 : cluster [DBG] pgmap v12280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:47 smithi115 bash[45495]: cluster 2022-04-22T23:10:46.877495+0000 mgr.y (mgr.34107) 11745 : cluster [DBG] pgmap v12280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:47 smithi115 bash[46934]: cluster 2022-04-22T23:10:46.877495+0000 mgr.y (mgr.34107) 11745 : cluster [DBG] pgmap v12280: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:49.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:48 smithi121 bash[36902]: audit 2022-04-22T23:10:48.006508+0000 mon.a (mon.0) 16238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:48 smithi115 bash[45495]: audit 2022-04-22T23:10:48.006508+0000 mon.a (mon.0) 16238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:48 smithi115 bash[46934]: audit 2022-04-22T23:10:48.006508+0000 mon.a (mon.0) 16238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:50.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:49 smithi121 bash[36902]: cluster 2022-04-22T23:10:48.878633+0000 mgr.y (mgr.34107) 11746 : cluster [DBG] pgmap v12281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:50.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:49 smithi121 bash[36902]: audit 2022-04-22T23:10:49.638967+0000 mon.a (mon.0) 16239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:49 smithi115 bash[46934]: cluster 2022-04-22T23:10:48.878633+0000 mgr.y (mgr.34107) 11746 : cluster [DBG] pgmap v12281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:50.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:49 smithi115 bash[46934]: audit 2022-04-22T23:10:49.638967+0000 mon.a (mon.0) 16239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:49 smithi115 bash[45495]: cluster 2022-04-22T23:10:48.878633+0000 mgr.y (mgr.34107) 11746 : cluster [DBG] pgmap v12281: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:50.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:49 smithi115 bash[45495]: audit 2022-04-22T23:10:49.638967+0000 mon.a (mon.0) 16239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:51 smithi121 bash[36902]: cluster 2022-04-22T23:10:50.880139+0000 mgr.y (mgr.34107) 11747 : cluster [DBG] pgmap v12282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:51 smithi121 bash[36902]: audit 2022-04-22T23:10:51.375951+0000 mon.a (mon.0) 16240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:51 smithi115 bash[45495]: cluster 2022-04-22T23:10:50.880139+0000 mgr.y (mgr.34107) 11747 : cluster [DBG] pgmap v12282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:51 smithi115 bash[45495]: audit 2022-04-22T23:10:51.375951+0000 mon.a (mon.0) 16240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:51 smithi115 bash[46934]: cluster 2022-04-22T23:10:50.880139+0000 mgr.y (mgr.34107) 11747 : cluster [DBG] pgmap v12282: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:51 smithi115 bash[46934]: audit 2022-04-22T23:10:51.375951+0000 mon.a (mon.0) 16240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:52.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:53.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:53 smithi121 bash[36902]: cluster 2022-04-22T23:10:52.881170+0000 mgr.y (mgr.34107) 11748 : cluster [DBG] pgmap v12283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:53 smithi121 bash[36902]: audit 2022-04-22T23:10:53.009650+0000 mon.a (mon.0) 16241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:53 smithi115 bash[45495]: cluster 2022-04-22T23:10:52.881170+0000 mgr.y (mgr.34107) 11748 : cluster [DBG] pgmap v12283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:53 smithi115 bash[45495]: audit 2022-04-22T23:10:53.009650+0000 mon.a (mon.0) 16241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:53 smithi115 bash[46934]: cluster 2022-04-22T23:10:52.881170+0000 mgr.y (mgr.34107) 11748 : cluster [DBG] pgmap v12283: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:53 smithi115 bash[46934]: audit 2022-04-22T23:10:53.009650+0000 mon.a (mon.0) 16241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:54 smithi121 bash[36902]: audit 2022-04-22T23:10:54.642684+0000 mon.a (mon.0) 16242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:54 smithi115 bash[45495]: audit 2022-04-22T23:10:54.642684+0000 mon.a (mon.0) 16242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:10:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:10:54.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:10:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:54 smithi115 bash[46934]: audit 2022-04-22T23:10:54.642684+0000 mon.a (mon.0) 16242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:10:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:55 smithi121 bash[36902]: cluster 2022-04-22T23:10:54.882246+0000 mgr.y (mgr.34107) 11749 : cluster [DBG] pgmap v12284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:55 smithi115 bash[45495]: cluster 2022-04-22T23:10:54.882246+0000 mgr.y (mgr.34107) 11749 : cluster [DBG] pgmap v12284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:55 smithi115 bash[46934]: cluster 2022-04-22T23:10:54.882246+0000 mgr.y (mgr.34107) 11749 : cluster [DBG] pgmap v12284: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:56 smithi121 bash[36902]: audit 2022-04-22T23:10:56.379153+0000 mon.a (mon.0) 16243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:56 smithi115 bash[45495]: audit 2022-04-22T23:10:56.379153+0000 mon.a (mon.0) 16243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:56 smithi115 bash[46934]: audit 2022-04-22T23:10:56.379153+0000 mon.a (mon.0) 16243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:10:58.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:57 smithi121 bash[36902]: cluster 2022-04-22T23:10:56.883234+0000 mgr.y (mgr.34107) 11750 : cluster [DBG] pgmap v12285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:57 smithi115 bash[45495]: cluster 2022-04-22T23:10:56.883234+0000 mgr.y (mgr.34107) 11750 : cluster [DBG] pgmap v12285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:57 smithi115 bash[46934]: cluster 2022-04-22T23:10:56.883234+0000 mgr.y (mgr.34107) 11750 : cluster [DBG] pgmap v12285: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:10:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:58 smithi121 bash[36902]: audit 2022-04-22T23:10:58.012457+0000 mon.a (mon.0) 16244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:59.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:58 smithi115 bash[45495]: audit 2022-04-22T23:10:58.012457+0000 mon.a (mon.0) 16244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:10:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:58 smithi115 bash[46934]: audit 2022-04-22T23:10:58.012457+0000 mon.a (mon.0) 16244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:59 smithi121 bash[36902]: cluster 2022-04-22T23:10:58.884342+0000 mgr.y (mgr.34107) 11751 : cluster [DBG] pgmap v12286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:10:59 smithi121 bash[36902]: audit 2022-04-22T23:10:59.646331+0000 mon.a (mon.0) 16245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:59 smithi115 bash[45495]: cluster 2022-04-22T23:10:58.884342+0000 mgr.y (mgr.34107) 11751 : cluster [DBG] pgmap v12286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:10:59 smithi115 bash[45495]: audit 2022-04-22T23:10:59.646331+0000 mon.a (mon.0) 16245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:59 smithi115 bash[46934]: cluster 2022-04-22T23:10:58.884342+0000 mgr.y (mgr.34107) 11751 : cluster [DBG] pgmap v12286: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:10:59 smithi115 bash[46934]: audit 2022-04-22T23:10:59.646331+0000 mon.a (mon.0) 16245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:01 smithi121 bash[36902]: cluster 2022-04-22T23:11:00.885376+0000 mgr.y (mgr.34107) 11752 : cluster [DBG] pgmap v12287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:01 smithi121 bash[36902]: audit 2022-04-22T23:11:01.382247+0000 mon.a (mon.0) 16246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:01 smithi115 bash[45495]: cluster 2022-04-22T23:11:00.885376+0000 mgr.y (mgr.34107) 11752 : cluster [DBG] pgmap v12287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:01 smithi115 bash[45495]: audit 2022-04-22T23:11:01.382247+0000 mon.a (mon.0) 16246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:01 smithi115 bash[46934]: cluster 2022-04-22T23:11:00.885376+0000 mgr.y (mgr.34107) 11752 : cluster [DBG] pgmap v12287: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:02.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:01 smithi115 bash[46934]: audit 2022-04-22T23:11:01.382247+0000 mon.a (mon.0) 16246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:02.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:03.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:03 smithi121 bash[36902]: cluster 2022-04-22T23:11:02.886559+0000 mgr.y (mgr.34107) 11753 : cluster [DBG] pgmap v12288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:03 smithi121 bash[36902]: audit 2022-04-22T23:11:03.015335+0000 mon.a (mon.0) 16247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:03 smithi115 bash[45495]: cluster 2022-04-22T23:11:02.886559+0000 mgr.y (mgr.34107) 11753 : cluster [DBG] pgmap v12288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:03 smithi115 bash[45495]: audit 2022-04-22T23:11:03.015335+0000 mon.a (mon.0) 16247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:03 smithi115 bash[46934]: cluster 2022-04-22T23:11:02.886559+0000 mgr.y (mgr.34107) 11753 : cluster [DBG] pgmap v12288: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:03 smithi115 bash[46934]: audit 2022-04-22T23:11:03.015335+0000 mon.a (mon.0) 16247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:05.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:04 smithi121 bash[36902]: audit 2022-04-22T23:11:04.649887+0000 mon.a (mon.0) 16248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:05.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:04 smithi115 bash[45495]: audit 2022-04-22T23:11:04.649887+0000 mon.a (mon.0) 16248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:04.832Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:04 smithi115 bash[46934]: audit 2022-04-22T23:11:04.649887+0000 mon.a (mon.0) 16248 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:05 smithi121 bash[36902]: cluster 2022-04-22T23:11:04.887663+0000 mgr.y (mgr.34107) 11754 : cluster [DBG] pgmap v12289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:06.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:05 smithi115 bash[45495]: cluster 2022-04-22T23:11:04.887663+0000 mgr.y (mgr.34107) 11754 : cluster [DBG] pgmap v12289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:05 smithi115 bash[46934]: cluster 2022-04-22T23:11:04.887663+0000 mgr.y (mgr.34107) 11754 : cluster [DBG] pgmap v12289: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:06.833 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:11:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:06 smithi121 bash[36902]: audit 2022-04-22T23:11:06.385372+0000 mon.a (mon.0) 16249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:07.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:06 smithi115 bash[46934]: audit 2022-04-22T23:11:06.385372+0000 mon.a (mon.0) 16249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:06 smithi115 bash[45495]: audit 2022-04-22T23:11:06.385372+0000 mon.a (mon.0) 16249 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:07.175 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:11:07.175 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:11:07.175 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:11:07.175 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:11:07.176 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:11:07.177 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:11:07.179 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:11:07.181 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:11:07.182 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:11:07.183 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:11:07.183 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:11:07.183 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:11:07.183 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:11:07.586 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:11:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:11:07.588 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:11:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:07 smithi121 bash[36902]: audit 2022-04-22T23:11:06.814843+0000 mgr.y (mgr.34107) 11755 : audit [DBG] from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:07 smithi121 bash[36902]: cluster 2022-04-22T23:11:06.888739+0000 mgr.y (mgr.34107) 11756 : cluster [DBG] pgmap v12290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:07 smithi121 bash[36902]: audit 2022-04-22T23:11:07.152062+0000 mgr.y (mgr.34107) 11757 : audit [DBG] from='client.47517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:07 smithi121 bash[36902]: audit 2022-04-22T23:11:07.582453+0000 mon.c (mon.1) 396 : audit [DBG] from='client.? 172.21.15.115:0/2485602504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[45495]: audit 2022-04-22T23:11:06.814843+0000 mgr.y (mgr.34107) 11755 : audit [DBG] from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[45495]: cluster 2022-04-22T23:11:06.888739+0000 mgr.y (mgr.34107) 11756 : cluster [DBG] pgmap v12290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[45495]: audit 2022-04-22T23:11:07.152062+0000 mgr.y (mgr.34107) 11757 : audit [DBG] from='client.47517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[45495]: audit 2022-04-22T23:11:07.582453+0000 mon.c (mon.1) 396 : audit [DBG] from='client.? 172.21.15.115:0/2485602504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[46934]: audit 2022-04-22T23:11:06.814843+0000 mgr.y (mgr.34107) 11755 : audit [DBG] from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[46934]: cluster 2022-04-22T23:11:06.888739+0000 mgr.y (mgr.34107) 11756 : cluster [DBG] pgmap v12290: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[46934]: audit 2022-04-22T23:11:07.152062+0000 mgr.y (mgr.34107) 11757 : audit [DBG] from='client.47517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:08.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:07 smithi115 bash[46934]: audit 2022-04-22T23:11:07.582453+0000 mon.c (mon.1) 396 : audit [DBG] from='client.? 172.21.15.115:0/2485602504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:09.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:08 smithi121 bash[36902]: audit 2022-04-22T23:11:08.018670+0000 mon.a (mon.0) 16250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:09.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:08 smithi115 bash[45495]: audit 2022-04-22T23:11:08.018670+0000 mon.a (mon.0) 16250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:09.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:08 smithi115 bash[46934]: audit 2022-04-22T23:11:08.018670+0000 mon.a (mon.0) 16250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:09 smithi121 bash[36902]: cluster 2022-04-22T23:11:08.889935+0000 mgr.y (mgr.34107) 11758 : cluster [DBG] pgmap v12291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:10.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:09 smithi121 bash[36902]: audit 2022-04-22T23:11:09.653668+0000 mon.a (mon.0) 16251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:09 smithi115 bash[45495]: cluster 2022-04-22T23:11:08.889935+0000 mgr.y (mgr.34107) 11758 : cluster [DBG] pgmap v12291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:09 smithi115 bash[45495]: audit 2022-04-22T23:11:09.653668+0000 mon.a (mon.0) 16251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:09 smithi115 bash[46934]: cluster 2022-04-22T23:11:08.889935+0000 mgr.y (mgr.34107) 11758 : cluster [DBG] pgmap v12291: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:09 smithi115 bash[46934]: audit 2022-04-22T23:11:09.653668+0000 mon.a (mon.0) 16251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:11 smithi121 bash[36902]: cluster 2022-04-22T23:11:10.891168+0000 mgr.y (mgr.34107) 11759 : cluster [DBG] pgmap v12292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:11 smithi121 bash[36902]: audit 2022-04-22T23:11:11.388434+0000 mon.a (mon.0) 16252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:11 smithi115 bash[45495]: cluster 2022-04-22T23:11:10.891168+0000 mgr.y (mgr.34107) 11759 : cluster [DBG] pgmap v12292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:11 smithi115 bash[45495]: audit 2022-04-22T23:11:11.388434+0000 mon.a (mon.0) 16252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:11 smithi115 bash[46934]: cluster 2022-04-22T23:11:10.891168+0000 mgr.y (mgr.34107) 11759 : cluster [DBG] pgmap v12292: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:11 smithi115 bash[46934]: audit 2022-04-22T23:11:11.388434+0000 mon.a (mon.0) 16252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:12.978Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:13.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:13 smithi121 bash[36902]: cluster 2022-04-22T23:11:12.892255+0000 mgr.y (mgr.34107) 11760 : cluster [DBG] pgmap v12293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:13 smithi121 bash[36902]: audit 2022-04-22T23:11:13.021773+0000 mon.a (mon.0) 16253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:13 smithi115 bash[45495]: cluster 2022-04-22T23:11:12.892255+0000 mgr.y (mgr.34107) 11760 : cluster [DBG] pgmap v12293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:13 smithi115 bash[45495]: audit 2022-04-22T23:11:13.021773+0000 mon.a (mon.0) 16253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:13 smithi115 bash[46934]: cluster 2022-04-22T23:11:12.892255+0000 mgr.y (mgr.34107) 11760 : cluster [DBG] pgmap v12293: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:13 smithi115 bash[46934]: audit 2022-04-22T23:11:13.021773+0000 mon.a (mon.0) 16253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:15.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:14 smithi121 bash[36902]: audit 2022-04-22T23:11:14.657460+0000 mon.a (mon.0) 16254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:15.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:14 smithi115 bash[46934]: audit 2022-04-22T23:11:14.657460+0000 mon.a (mon.0) 16254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:15.090 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:14.833Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:15.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:14 smithi115 bash[45495]: audit 2022-04-22T23:11:14.657460+0000 mon.a (mon.0) 16254 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:16.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:15 smithi121 bash[36902]: cluster 2022-04-22T23:11:14.893292+0000 mgr.y (mgr.34107) 11761 : cluster [DBG] pgmap v12294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:16.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:15 smithi115 bash[46934]: cluster 2022-04-22T23:11:14.893292+0000 mgr.y (mgr.34107) 11761 : cluster [DBG] pgmap v12294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:15 smithi115 bash[45495]: cluster 2022-04-22T23:11:14.893292+0000 mgr.y (mgr.34107) 11761 : cluster [DBG] pgmap v12294: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:16 smithi121 bash[36902]: audit 2022-04-22T23:11:16.391734+0000 mon.a (mon.0) 16255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:17.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:16 smithi115 bash[45495]: audit 2022-04-22T23:11:16.391734+0000 mon.a (mon.0) 16255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:16 smithi115 bash[46934]: audit 2022-04-22T23:11:16.391734+0000 mon.a (mon.0) 16255 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:17 smithi121 bash[36902]: cluster 2022-04-22T23:11:16.894511+0000 mgr.y (mgr.34107) 11762 : cluster [DBG] pgmap v12295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:17 smithi121 bash[36902]: audit 2022-04-22T23:11:17.238883+0000 mon.a (mon.0) 16256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:11:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:17 smithi121 bash[36902]: audit 2022-04-22T23:11:17.240931+0000 mon.a (mon.0) 16257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:11:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[45495]: cluster 2022-04-22T23:11:16.894511+0000 mgr.y (mgr.34107) 11762 : cluster [DBG] pgmap v12295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[45495]: audit 2022-04-22T23:11:17.238883+0000 mon.a (mon.0) 16256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:11:18.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[45495]: audit 2022-04-22T23:11:17.240931+0000 mon.a (mon.0) 16257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:11:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[46934]: cluster 2022-04-22T23:11:16.894511+0000 mgr.y (mgr.34107) 11762 : cluster [DBG] pgmap v12295: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[46934]: audit 2022-04-22T23:11:17.238883+0000 mon.a (mon.0) 16256 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:11:18.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:17 smithi115 bash[46934]: audit 2022-04-22T23:11:17.240931+0000 mon.a (mon.0) 16257 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:11:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:18 smithi121 bash[36902]: audit 2022-04-22T23:11:18.024830+0000 mon.a (mon.0) 16258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:18 smithi115 bash[45495]: audit 2022-04-22T23:11:18.024830+0000 mon.a (mon.0) 16258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:18 smithi115 bash[46934]: audit 2022-04-22T23:11:18.024830+0000 mon.a (mon.0) 16258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:19 smithi121 bash[36902]: cluster 2022-04-22T23:11:18.895729+0000 mgr.y (mgr.34107) 11763 : cluster [DBG] pgmap v12296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:19 smithi121 bash[36902]: audit 2022-04-22T23:11:19.661221+0000 mon.a (mon.0) 16259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:19 smithi115 bash[45495]: cluster 2022-04-22T23:11:18.895729+0000 mgr.y (mgr.34107) 11763 : cluster [DBG] pgmap v12296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:19 smithi115 bash[45495]: audit 2022-04-22T23:11:19.661221+0000 mon.a (mon.0) 16259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:19 smithi115 bash[46934]: cluster 2022-04-22T23:11:18.895729+0000 mgr.y (mgr.34107) 11763 : cluster [DBG] pgmap v12296: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:19 smithi115 bash[46934]: audit 2022-04-22T23:11:19.661221+0000 mon.a (mon.0) 16259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:22.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:21 smithi121 bash[36902]: cluster 2022-04-22T23:11:20.896980+0000 mgr.y (mgr.34107) 11764 : cluster [DBG] pgmap v12297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:21 smithi121 bash[36902]: audit 2022-04-22T23:11:21.394730+0000 mon.a (mon.0) 16260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:21 smithi115 bash[45495]: cluster 2022-04-22T23:11:20.896980+0000 mgr.y (mgr.34107) 11764 : cluster [DBG] pgmap v12297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:21 smithi115 bash[45495]: audit 2022-04-22T23:11:21.394730+0000 mon.a (mon.0) 16260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:21 smithi115 bash[46934]: cluster 2022-04-22T23:11:20.896980+0000 mgr.y (mgr.34107) 11764 : cluster [DBG] pgmap v12297: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:22.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:21 smithi115 bash[46934]: audit 2022-04-22T23:11:21.394730+0000 mon.a (mon.0) 16260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:22.979Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:23 smithi121 bash[36902]: cluster 2022-04-22T23:11:22.898103+0000 mgr.y (mgr.34107) 11765 : cluster [DBG] pgmap v12298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:23 smithi121 bash[36902]: audit 2022-04-22T23:11:23.027914+0000 mon.a (mon.0) 16261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:23 smithi115 bash[45495]: cluster 2022-04-22T23:11:22.898103+0000 mgr.y (mgr.34107) 11765 : cluster [DBG] pgmap v12298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:23 smithi115 bash[45495]: audit 2022-04-22T23:11:23.027914+0000 mon.a (mon.0) 16261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:23 smithi115 bash[46934]: cluster 2022-04-22T23:11:22.898103+0000 mgr.y (mgr.34107) 11765 : cluster [DBG] pgmap v12298: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:23 smithi115 bash[46934]: audit 2022-04-22T23:11:23.027914+0000 mon.a (mon.0) 16261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:25.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:24 smithi121 bash[36902]: audit 2022-04-22T23:11:24.664963+0000 mon.a (mon.0) 16262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:25.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:24 smithi115 bash[45495]: audit 2022-04-22T23:11:24.664963+0000 mon.a (mon.0) 16262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:24.834Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:25.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:24 smithi115 bash[46934]: audit 2022-04-22T23:11:24.664963+0000 mon.a (mon.0) 16262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:26.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:25 smithi121 bash[36902]: cluster 2022-04-22T23:11:24.899222+0000 mgr.y (mgr.34107) 11766 : cluster [DBG] pgmap v12299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:25 smithi115 bash[45495]: cluster 2022-04-22T23:11:24.899222+0000 mgr.y (mgr.34107) 11766 : cluster [DBG] pgmap v12299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:25 smithi115 bash[46934]: cluster 2022-04-22T23:11:24.899222+0000 mgr.y (mgr.34107) 11766 : cluster [DBG] pgmap v12299: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:27.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:26 smithi121 bash[36902]: audit 2022-04-22T23:11:26.397751+0000 mon.a (mon.0) 16263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:26 smithi115 bash[46934]: audit 2022-04-22T23:11:26.397751+0000 mon.a (mon.0) 16263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:27.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:26 smithi115 bash[45495]: audit 2022-04-22T23:11:26.397751+0000 mon.a (mon.0) 16263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:27 smithi121 bash[36902]: audit 2022-04-22T23:11:26.742212+0000 mon.a (mon.0) 16264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:11:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:27 smithi121 bash[36902]: audit 2022-04-22T23:11:26.744079+0000 mon.a (mon.0) 16265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:11:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:27 smithi121 bash[36902]: audit 2022-04-22T23:11:26.745323+0000 mon.a (mon.0) 16266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:11:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:27 smithi121 bash[36902]: cluster 2022-04-22T23:11:26.900285+0000 mgr.y (mgr.34107) 11767 : cluster [DBG] pgmap v12300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:28.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:27 smithi121 bash[36902]: audit 2022-04-22T23:11:27.052936+0000 mon.a (mon.0) 16267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:11:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[46934]: audit 2022-04-22T23:11:26.742212+0000 mon.a (mon.0) 16264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:11:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[46934]: audit 2022-04-22T23:11:26.744079+0000 mon.a (mon.0) 16265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:11:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[46934]: audit 2022-04-22T23:11:26.745323+0000 mon.a (mon.0) 16266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:11:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[46934]: cluster 2022-04-22T23:11:26.900285+0000 mgr.y (mgr.34107) 11767 : cluster [DBG] pgmap v12300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:28.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[46934]: audit 2022-04-22T23:11:27.052936+0000 mon.a (mon.0) 16267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:11:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[45495]: audit 2022-04-22T23:11:26.742212+0000 mon.a (mon.0) 16264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:11:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[45495]: audit 2022-04-22T23:11:26.744079+0000 mon.a (mon.0) 16265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:11:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[45495]: audit 2022-04-22T23:11:26.745323+0000 mon.a (mon.0) 16266 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:11:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[45495]: cluster 2022-04-22T23:11:26.900285+0000 mgr.y (mgr.34107) 11767 : cluster [DBG] pgmap v12300: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:28.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:27 smithi115 bash[45495]: audit 2022-04-22T23:11:27.052936+0000 mon.a (mon.0) 16267 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:11:29.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:28 smithi121 bash[36902]: audit 2022-04-22T23:11:28.031316+0000 mon.a (mon.0) 16268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:29.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:28 smithi115 bash[45495]: audit 2022-04-22T23:11:28.031316+0000 mon.a (mon.0) 16268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:29.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:28 smithi115 bash[46934]: audit 2022-04-22T23:11:28.031316+0000 mon.a (mon.0) 16268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:29 smithi121 bash[36902]: cluster 2022-04-22T23:11:28.901461+0000 mgr.y (mgr.34107) 11768 : cluster [DBG] pgmap v12301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:30.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:29 smithi121 bash[36902]: audit 2022-04-22T23:11:29.668466+0000 mon.a (mon.0) 16269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:30.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:29 smithi115 bash[45495]: cluster 2022-04-22T23:11:28.901461+0000 mgr.y (mgr.34107) 11768 : cluster [DBG] pgmap v12301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:30.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:29 smithi115 bash[45495]: audit 2022-04-22T23:11:29.668466+0000 mon.a (mon.0) 16269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:29 smithi115 bash[46934]: cluster 2022-04-22T23:11:28.901461+0000 mgr.y (mgr.34107) 11768 : cluster [DBG] pgmap v12301: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:30.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:29 smithi115 bash[46934]: audit 2022-04-22T23:11:29.668466+0000 mon.a (mon.0) 16269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:31 smithi121 bash[36902]: cluster 2022-04-22T23:11:30.902558+0000 mgr.y (mgr.34107) 11769 : cluster [DBG] pgmap v12302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:31 smithi121 bash[36902]: audit 2022-04-22T23:11:31.400587+0000 mon.a (mon.0) 16270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:31 smithi115 bash[46934]: cluster 2022-04-22T23:11:30.902558+0000 mgr.y (mgr.34107) 11769 : cluster [DBG] pgmap v12302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:31 smithi115 bash[46934]: audit 2022-04-22T23:11:31.400587+0000 mon.a (mon.0) 16270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:31 smithi115 bash[45495]: cluster 2022-04-22T23:11:30.902558+0000 mgr.y (mgr.34107) 11769 : cluster [DBG] pgmap v12302: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:32.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:31 smithi115 bash[45495]: audit 2022-04-22T23:11:31.400587+0000 mon.a (mon.0) 16270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:32.979Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:33 smithi121 bash[36902]: cluster 2022-04-22T23:11:32.903668+0000 mgr.y (mgr.34107) 11770 : cluster [DBG] pgmap v12303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:33 smithi121 bash[36902]: audit 2022-04-22T23:11:33.034613+0000 mon.a (mon.0) 16271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:33 smithi115 bash[45495]: cluster 2022-04-22T23:11:32.903668+0000 mgr.y (mgr.34107) 11770 : cluster [DBG] pgmap v12303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:33 smithi115 bash[45495]: audit 2022-04-22T23:11:33.034613+0000 mon.a (mon.0) 16271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:33 smithi115 bash[46934]: cluster 2022-04-22T23:11:32.903668+0000 mgr.y (mgr.34107) 11770 : cluster [DBG] pgmap v12303: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:33 smithi115 bash[46934]: audit 2022-04-22T23:11:33.034613+0000 mon.a (mon.0) 16271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:35.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:34 smithi121 bash[36902]: audit 2022-04-22T23:11:34.672074+0000 mon.a (mon.0) 16272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:35.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:34 smithi115 bash[45495]: audit 2022-04-22T23:11:34.672074+0000 mon.a (mon.0) 16272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:34.836Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:34 smithi115 bash[46934]: audit 2022-04-22T23:11:34.672074+0000 mon.a (mon.0) 16272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:35 smithi121 bash[36902]: cluster 2022-04-22T23:11:34.904817+0000 mgr.y (mgr.34107) 11771 : cluster [DBG] pgmap v12304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:36.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:35 smithi115 bash[45495]: cluster 2022-04-22T23:11:34.904817+0000 mgr.y (mgr.34107) 11771 : cluster [DBG] pgmap v12304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:35 smithi115 bash[46934]: cluster 2022-04-22T23:11:34.904817+0000 mgr.y (mgr.34107) 11771 : cluster [DBG] pgmap v12304: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:36 smithi121 bash[36902]: audit 2022-04-22T23:11:36.403600+0000 mon.a (mon.0) 16273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:36 smithi115 bash[45495]: audit 2022-04-22T23:11:36.403600+0000 mon.a (mon.0) 16273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:36 smithi115 bash[46934]: audit 2022-04-22T23:11:36.403600+0000 mon.a (mon.0) 16273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:37.959 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:11:38.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:37 smithi121 bash[36902]: cluster 2022-04-22T23:11:36.905917+0000 mgr.y (mgr.34107) 11772 : cluster [DBG] pgmap v12305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:38.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:37 smithi115 bash[46934]: cluster 2022-04-22T23:11:36.905917+0000 mgr.y (mgr.34107) 11772 : cluster [DBG] pgmap v12305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:38.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:37 smithi115 bash[45495]: cluster 2022-04-22T23:11:36.905917+0000 mgr.y (mgr.34107) 11772 : cluster [DBG] pgmap v12305: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:38.298 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:11:38.298 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:11:38.298 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:11:38.299 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:11:38.299 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:11:38.299 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:11:38.299 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:11:38.299 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:11:38.300 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:11:38.301 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:11:38.301 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:11:38.301 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:11:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:11:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:11:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:11:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:11:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:11:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:11:38.707 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:11:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:38 smithi121 bash[36902]: audit 2022-04-22T23:11:37.938814+0000 mgr.y (mgr.34107) 11773 : audit [DBG] from='client.57469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:38 smithi121 bash[36902]: audit 2022-04-22T23:11:38.037695+0000 mon.a (mon.0) 16274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:39.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:38 smithi121 bash[36902]: audit 2022-04-22T23:11:38.701178+0000 mon.c (mon.1) 397 : audit [DBG] from='client.? 172.21.15.115:0/2650407098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[45495]: audit 2022-04-22T23:11:37.938814+0000 mgr.y (mgr.34107) 11773 : audit [DBG] from='client.57469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[45495]: audit 2022-04-22T23:11:38.037695+0000 mon.a (mon.0) 16274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:39.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[45495]: audit 2022-04-22T23:11:38.701178+0000 mon.c (mon.1) 397 : audit [DBG] from='client.? 172.21.15.115:0/2650407098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[46934]: audit 2022-04-22T23:11:37.938814+0000 mgr.y (mgr.34107) 11773 : audit [DBG] from='client.57469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:39.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[46934]: audit 2022-04-22T23:11:38.037695+0000 mon.a (mon.0) 16274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:39.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:38 smithi115 bash[46934]: audit 2022-04-22T23:11:38.701178+0000 mon.c (mon.1) 397 : audit [DBG] from='client.? 172.21.15.115:0/2650407098' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:11:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:39 smithi121 bash[36902]: audit 2022-04-22T23:11:38.280437+0000 mgr.y (mgr.34107) 11774 : audit [DBG] from='client.57475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:39 smithi121 bash[36902]: cluster 2022-04-22T23:11:38.907037+0000 mgr.y (mgr.34107) 11775 : cluster [DBG] pgmap v12306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:40.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:39 smithi121 bash[36902]: audit 2022-04-22T23:11:39.675711+0000 mon.a (mon.0) 16275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[45495]: audit 2022-04-22T23:11:38.280437+0000 mgr.y (mgr.34107) 11774 : audit [DBG] from='client.57475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[45495]: cluster 2022-04-22T23:11:38.907037+0000 mgr.y (mgr.34107) 11775 : cluster [DBG] pgmap v12306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[45495]: audit 2022-04-22T23:11:39.675711+0000 mon.a (mon.0) 16275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[46934]: audit 2022-04-22T23:11:38.280437+0000 mgr.y (mgr.34107) 11774 : audit [DBG] from='client.57475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[46934]: cluster 2022-04-22T23:11:38.907037+0000 mgr.y (mgr.34107) 11775 : cluster [DBG] pgmap v12306: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:40.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:39 smithi115 bash[46934]: audit 2022-04-22T23:11:39.675711+0000 mon.a (mon.0) 16275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:42.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:41 smithi121 bash[36902]: cluster 2022-04-22T23:11:40.908177+0000 mgr.y (mgr.34107) 11776 : cluster [DBG] pgmap v12307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:41 smithi121 bash[36902]: audit 2022-04-22T23:11:41.406568+0000 mon.a (mon.0) 16276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:41 smithi115 bash[45495]: cluster 2022-04-22T23:11:40.908177+0000 mgr.y (mgr.34107) 11776 : cluster [DBG] pgmap v12307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:41 smithi115 bash[45495]: audit 2022-04-22T23:11:41.406568+0000 mon.a (mon.0) 16276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:41 smithi115 bash[46934]: cluster 2022-04-22T23:11:40.908177+0000 mgr.y (mgr.34107) 11776 : cluster [DBG] pgmap v12307: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:41 smithi115 bash[46934]: audit 2022-04-22T23:11:41.406568+0000 mon.a (mon.0) 16276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:42.981Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:43 smithi121 bash[36902]: cluster 2022-04-22T23:11:42.909266+0000 mgr.y (mgr.34107) 11777 : cluster [DBG] pgmap v12308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:43 smithi121 bash[36902]: audit 2022-04-22T23:11:43.040591+0000 mon.a (mon.0) 16277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:43 smithi115 bash[46934]: cluster 2022-04-22T23:11:42.909266+0000 mgr.y (mgr.34107) 11777 : cluster [DBG] pgmap v12308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:43 smithi115 bash[46934]: audit 2022-04-22T23:11:43.040591+0000 mon.a (mon.0) 16277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:43 smithi115 bash[45495]: cluster 2022-04-22T23:11:42.909266+0000 mgr.y (mgr.34107) 11777 : cluster [DBG] pgmap v12308: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:43 smithi115 bash[45495]: audit 2022-04-22T23:11:43.040591+0000 mon.a (mon.0) 16277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:45.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:44 smithi121 bash[36902]: audit 2022-04-22T23:11:44.679373+0000 mon.a (mon.0) 16278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:45.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:44.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:45.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:44 smithi115 bash[45495]: audit 2022-04-22T23:11:44.679373+0000 mon.a (mon.0) 16278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:44 smithi115 bash[46934]: audit 2022-04-22T23:11:44.679373+0000 mon.a (mon.0) 16278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:46.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:45 smithi121 bash[36902]: cluster 2022-04-22T23:11:44.910374+0000 mgr.y (mgr.34107) 11778 : cluster [DBG] pgmap v12309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:45 smithi115 bash[45495]: cluster 2022-04-22T23:11:44.910374+0000 mgr.y (mgr.34107) 11778 : cluster [DBG] pgmap v12309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:45 smithi115 bash[46934]: cluster 2022-04-22T23:11:44.910374+0000 mgr.y (mgr.34107) 11778 : cluster [DBG] pgmap v12309: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:47.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:46 smithi121 bash[36902]: audit 2022-04-22T23:11:46.409587+0000 mon.a (mon.0) 16279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:47.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:46 smithi115 bash[45495]: audit 2022-04-22T23:11:46.409587+0000 mon.a (mon.0) 16279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:47.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:46 smithi115 bash[46934]: audit 2022-04-22T23:11:46.409587+0000 mon.a (mon.0) 16279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:48.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:47 smithi121 bash[36902]: cluster 2022-04-22T23:11:46.911468+0000 mgr.y (mgr.34107) 11779 : cluster [DBG] pgmap v12310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:48.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:47 smithi115 bash[45495]: cluster 2022-04-22T23:11:46.911468+0000 mgr.y (mgr.34107) 11779 : cluster [DBG] pgmap v12310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:47 smithi115 bash[46934]: cluster 2022-04-22T23:11:46.911468+0000 mgr.y (mgr.34107) 11779 : cluster [DBG] pgmap v12310: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:48 smithi121 bash[36902]: audit 2022-04-22T23:11:48.043444+0000 mon.a (mon.0) 16280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:49.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:48 smithi115 bash[45495]: audit 2022-04-22T23:11:48.043444+0000 mon.a (mon.0) 16280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:48 smithi115 bash[46934]: audit 2022-04-22T23:11:48.043444+0000 mon.a (mon.0) 16280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:50.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:49 smithi115 bash[45495]: cluster 2022-04-22T23:11:48.912605+0000 mgr.y (mgr.34107) 11780 : cluster [DBG] pgmap v12311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:50.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:49 smithi115 bash[45495]: audit 2022-04-22T23:11:49.683175+0000 mon.a (mon.0) 16281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:49 smithi115 bash[46934]: cluster 2022-04-22T23:11:48.912605+0000 mgr.y (mgr.34107) 11780 : cluster [DBG] pgmap v12311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:50.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:49 smithi115 bash[46934]: audit 2022-04-22T23:11:49.683175+0000 mon.a (mon.0) 16281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:50.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:49 smithi121 bash[36902]: cluster 2022-04-22T23:11:48.912605+0000 mgr.y (mgr.34107) 11780 : cluster [DBG] pgmap v12311: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:49 smithi121 bash[36902]: audit 2022-04-22T23:11:49.683175+0000 mon.a (mon.0) 16281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:51 smithi121 bash[36902]: cluster 2022-04-22T23:11:50.913752+0000 mgr.y (mgr.34107) 11781 : cluster [DBG] pgmap v12312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:51 smithi121 bash[36902]: audit 2022-04-22T23:11:51.412680+0000 mon.a (mon.0) 16282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:51 smithi115 bash[45495]: cluster 2022-04-22T23:11:50.913752+0000 mgr.y (mgr.34107) 11781 : cluster [DBG] pgmap v12312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:51 smithi115 bash[45495]: audit 2022-04-22T23:11:51.412680+0000 mon.a (mon.0) 16282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:51 smithi115 bash[46934]: cluster 2022-04-22T23:11:50.913752+0000 mgr.y (mgr.34107) 11781 : cluster [DBG] pgmap v12312: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:51 smithi115 bash[46934]: audit 2022-04-22T23:11:51.412680+0000 mon.a (mon.0) 16282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:52.981Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:53.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:53 smithi121 bash[36902]: cluster 2022-04-22T23:11:52.914834+0000 mgr.y (mgr.34107) 11782 : cluster [DBG] pgmap v12313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:53.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:53 smithi121 bash[36902]: audit 2022-04-22T23:11:53.046399+0000 mon.a (mon.0) 16283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:53 smithi115 bash[45495]: cluster 2022-04-22T23:11:52.914834+0000 mgr.y (mgr.34107) 11782 : cluster [DBG] pgmap v12313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:53 smithi115 bash[45495]: audit 2022-04-22T23:11:53.046399+0000 mon.a (mon.0) 16283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:53 smithi115 bash[46934]: cluster 2022-04-22T23:11:52.914834+0000 mgr.y (mgr.34107) 11782 : cluster [DBG] pgmap v12313: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:53 smithi115 bash[46934]: audit 2022-04-22T23:11:53.046399+0000 mon.a (mon.0) 16283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:55.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:54 smithi121 bash[36902]: audit 2022-04-22T23:11:54.687085+0000 mon.a (mon.0) 16284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:55.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:54 smithi115 bash[46934]: audit 2022-04-22T23:11:54.687085+0000 mon.a (mon.0) 16284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:11:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:11:54.836Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:11:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:54 smithi115 bash[45495]: audit 2022-04-22T23:11:54.687085+0000 mon.a (mon.0) 16284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:11:56.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:55 smithi121 bash[36902]: cluster 2022-04-22T23:11:54.915981+0000 mgr.y (mgr.34107) 11783 : cluster [DBG] pgmap v12314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:56.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:55 smithi115 bash[45495]: cluster 2022-04-22T23:11:54.915981+0000 mgr.y (mgr.34107) 11783 : cluster [DBG] pgmap v12314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:55 smithi115 bash[46934]: cluster 2022-04-22T23:11:54.915981+0000 mgr.y (mgr.34107) 11783 : cluster [DBG] pgmap v12314: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:56 smithi121 bash[36902]: audit 2022-04-22T23:11:56.415799+0000 mon.a (mon.0) 16285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:56 smithi115 bash[46934]: audit 2022-04-22T23:11:56.415799+0000 mon.a (mon.0) 16285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:56 smithi115 bash[45495]: audit 2022-04-22T23:11:56.415799+0000 mon.a (mon.0) 16285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:11:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:57 smithi121 bash[36902]: cluster 2022-04-22T23:11:56.917086+0000 mgr.y (mgr.34107) 11784 : cluster [DBG] pgmap v12315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:57 smithi115 bash[46934]: cluster 2022-04-22T23:11:56.917086+0000 mgr.y (mgr.34107) 11784 : cluster [DBG] pgmap v12315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:57 smithi115 bash[45495]: cluster 2022-04-22T23:11:56.917086+0000 mgr.y (mgr.34107) 11784 : cluster [DBG] pgmap v12315: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:11:59.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:58 smithi121 bash[36902]: audit 2022-04-22T23:11:58.049534+0000 mon.a (mon.0) 16286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:58 smithi115 bash[46934]: audit 2022-04-22T23:11:58.049534+0000 mon.a (mon.0) 16286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:11:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:58 smithi115 bash[45495]: audit 2022-04-22T23:11:58.049534+0000 mon.a (mon.0) 16286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:59 smithi115 bash[45495]: cluster 2022-04-22T23:11:58.918215+0000 mgr.y (mgr.34107) 11785 : cluster [DBG] pgmap v12316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:11:59 smithi115 bash[45495]: audit 2022-04-22T23:11:59.690837+0000 mon.a (mon.0) 16287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:59 smithi115 bash[46934]: cluster 2022-04-22T23:11:58.918215+0000 mgr.y (mgr.34107) 11785 : cluster [DBG] pgmap v12316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:11:59 smithi115 bash[46934]: audit 2022-04-22T23:11:59.690837+0000 mon.a (mon.0) 16287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:59 smithi121 bash[36902]: cluster 2022-04-22T23:11:58.918215+0000 mgr.y (mgr.34107) 11785 : cluster [DBG] pgmap v12316: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:11:59 smithi121 bash[36902]: audit 2022-04-22T23:11:59.690837+0000 mon.a (mon.0) 16287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:02.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:01 smithi121 bash[36902]: cluster 2022-04-22T23:12:00.919354+0000 mgr.y (mgr.34107) 11786 : cluster [DBG] pgmap v12317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:01 smithi121 bash[36902]: audit 2022-04-22T23:12:01.418767+0000 mon.a (mon.0) 16288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:01 smithi115 bash[45495]: cluster 2022-04-22T23:12:00.919354+0000 mgr.y (mgr.34107) 11786 : cluster [DBG] pgmap v12317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:01 smithi115 bash[45495]: audit 2022-04-22T23:12:01.418767+0000 mon.a (mon.0) 16288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:01 smithi115 bash[46934]: cluster 2022-04-22T23:12:00.919354+0000 mgr.y (mgr.34107) 11786 : cluster [DBG] pgmap v12317: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:01 smithi115 bash[46934]: audit 2022-04-22T23:12:01.418767+0000 mon.a (mon.0) 16288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:02.983Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:03 smithi121 bash[36902]: cluster 2022-04-22T23:12:02.920539+0000 mgr.y (mgr.34107) 11787 : cluster [DBG] pgmap v12318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:03 smithi121 bash[36902]: audit 2022-04-22T23:12:03.052754+0000 mon.a (mon.0) 16289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:03 smithi115 bash[45495]: cluster 2022-04-22T23:12:02.920539+0000 mgr.y (mgr.34107) 11787 : cluster [DBG] pgmap v12318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:03 smithi115 bash[45495]: audit 2022-04-22T23:12:03.052754+0000 mon.a (mon.0) 16289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:03 smithi115 bash[46934]: cluster 2022-04-22T23:12:02.920539+0000 mgr.y (mgr.34107) 11787 : cluster [DBG] pgmap v12318: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:03.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:03 smithi115 bash[46934]: audit 2022-04-22T23:12:03.052754+0000 mon.a (mon.0) 16289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:05.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:04 smithi121 bash[36902]: audit 2022-04-22T23:12:04.695014+0000 mon.a (mon.0) 16290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:05.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:04 smithi115 bash[45495]: audit 2022-04-22T23:12:04.695014+0000 mon.a (mon.0) 16290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:05.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:04.835Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:05.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:04 smithi115 bash[46934]: audit 2022-04-22T23:12:04.695014+0000 mon.a (mon.0) 16290 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:06.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:05 smithi121 bash[36902]: cluster 2022-04-22T23:12:04.921655+0000 mgr.y (mgr.34107) 11788 : cluster [DBG] pgmap v12319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:06.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:05 smithi115 bash[46934]: cluster 2022-04-22T23:12:04.921655+0000 mgr.y (mgr.34107) 11788 : cluster [DBG] pgmap v12319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:06.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:05 smithi115 bash[45495]: cluster 2022-04-22T23:12:04.921655+0000 mgr.y (mgr.34107) 11788 : cluster [DBG] pgmap v12319: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:07.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:06 smithi121 bash[36902]: audit 2022-04-22T23:12:06.422039+0000 mon.a (mon.0) 16291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:07.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:06 smithi115 bash[45495]: audit 2022-04-22T23:12:06.422039+0000 mon.a (mon.0) 16291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:07.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:06 smithi115 bash[46934]: audit 2022-04-22T23:12:06.422039+0000 mon.a (mon.0) 16291 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:08.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:07 smithi115 bash[45495]: cluster 2022-04-22T23:12:06.922758+0000 mgr.y (mgr.34107) 11789 : cluster [DBG] pgmap v12320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:08.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:07 smithi115 bash[46934]: cluster 2022-04-22T23:12:06.922758+0000 mgr.y (mgr.34107) 11789 : cluster [DBG] pgmap v12320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:07 smithi121 bash[36902]: cluster 2022-04-22T23:12:06.922758+0000 mgr.y (mgr.34107) 11789 : cluster [DBG] pgmap v12320: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:08 smithi115 bash[45495]: audit 2022-04-22T23:12:08.055811+0000 mon.a (mon.0) 16292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:08 smithi115 bash[46934]: audit 2022-04-22T23:12:08.055811+0000 mon.a (mon.0) 16292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:09.075 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:12:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:08 smithi121 bash[36902]: audit 2022-04-22T23:12:08.055811+0000 mon.a (mon.0) 16292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:09.426 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:12:09.426 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:12:09.426 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:12:09.427 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:12:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:12:09.838 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:12:09.838 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:12:09.839 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:12:09.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:09.840 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:12:09.840 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:12:09.840 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:12:09.840 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:12:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[45495]: cluster 2022-04-22T23:12:08.923970+0000 mgr.y (mgr.34107) 11790 : cluster [DBG] pgmap v12321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[45495]: audit 2022-04-22T23:12:09.056081+0000 mgr.y (mgr.34107) 11791 : audit [DBG] from='client.57487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:10.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[45495]: audit 2022-04-22T23:12:09.699024+0000 mon.a (mon.0) 16293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:10.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[46934]: cluster 2022-04-22T23:12:08.923970+0000 mgr.y (mgr.34107) 11790 : cluster [DBG] pgmap v12321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[46934]: audit 2022-04-22T23:12:09.056081+0000 mgr.y (mgr.34107) 11791 : audit [DBG] from='client.57487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:10.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:09 smithi115 bash[46934]: audit 2022-04-22T23:12:09.699024+0000 mon.a (mon.0) 16293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:09 smithi121 bash[36902]: cluster 2022-04-22T23:12:08.923970+0000 mgr.y (mgr.34107) 11790 : cluster [DBG] pgmap v12321: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:09 smithi121 bash[36902]: audit 2022-04-22T23:12:09.056081+0000 mgr.y (mgr.34107) 11791 : audit [DBG] from='client.57487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:09 smithi121 bash[36902]: audit 2022-04-22T23:12:09.699024+0000 mon.a (mon.0) 16293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:10 smithi115 bash[45495]: audit 2022-04-22T23:12:09.404508+0000 mgr.y (mgr.34107) 11792 : audit [DBG] from='client.47553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:11.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:10 smithi115 bash[45495]: audit 2022-04-22T23:12:09.834786+0000 mon.c (mon.1) 398 : audit [DBG] from='client.? 172.21.15.115:0/907131865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:10 smithi115 bash[46934]: audit 2022-04-22T23:12:09.404508+0000 mgr.y (mgr.34107) 11792 : audit [DBG] from='client.47553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:11.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:10 smithi115 bash[46934]: audit 2022-04-22T23:12:09.834786+0000 mon.c (mon.1) 398 : audit [DBG] from='client.? 172.21.15.115:0/907131865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:11.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:10 smithi121 bash[36902]: audit 2022-04-22T23:12:09.404508+0000 mgr.y (mgr.34107) 11792 : audit [DBG] from='client.47553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:10 smithi121 bash[36902]: audit 2022-04-22T23:12:09.834786+0000 mon.c (mon.1) 398 : audit [DBG] from='client.? 172.21.15.115:0/907131865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:12.044 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:11 smithi115 bash[45495]: cluster 2022-04-22T23:12:10.925147+0000 mgr.y (mgr.34107) 11793 : cluster [DBG] pgmap v12322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:12.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:11 smithi115 bash[45495]: audit 2022-04-22T23:12:11.424958+0000 mon.a (mon.0) 16294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:11 smithi115 bash[46934]: cluster 2022-04-22T23:12:10.925147+0000 mgr.y (mgr.34107) 11793 : cluster [DBG] pgmap v12322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:12.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:11 smithi115 bash[46934]: audit 2022-04-22T23:12:11.424958+0000 mon.a (mon.0) 16294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:12.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:11 smithi121 bash[36902]: cluster 2022-04-22T23:12:10.925147+0000 mgr.y (mgr.34107) 11793 : cluster [DBG] pgmap v12322: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:12.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:11 smithi121 bash[36902]: audit 2022-04-22T23:12:11.424958+0000 mon.a (mon.0) 16294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:12.982Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:13 smithi121 bash[36902]: cluster 2022-04-22T23:12:12.926258+0000 mgr.y (mgr.34107) 11794 : cluster [DBG] pgmap v12323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:13 smithi121 bash[36902]: audit 2022-04-22T23:12:13.059010+0000 mon.a (mon.0) 16295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:13 smithi115 bash[46934]: cluster 2022-04-22T23:12:12.926258+0000 mgr.y (mgr.34107) 11794 : cluster [DBG] pgmap v12323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:13 smithi115 bash[46934]: audit 2022-04-22T23:12:13.059010+0000 mon.a (mon.0) 16295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:13 smithi115 bash[45495]: cluster 2022-04-22T23:12:12.926258+0000 mgr.y (mgr.34107) 11794 : cluster [DBG] pgmap v12323: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:13 smithi115 bash[45495]: audit 2022-04-22T23:12:13.059010+0000 mon.a (mon.0) 16295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:15.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:14 smithi121 bash[36902]: audit 2022-04-22T23:12:14.702758+0000 mon.a (mon.0) 16296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:15.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:14.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:15.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:14 smithi115 bash[45495]: audit 2022-04-22T23:12:14.702758+0000 mon.a (mon.0) 16296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:15.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:14 smithi115 bash[46934]: audit 2022-04-22T23:12:14.702758+0000 mon.a (mon.0) 16296 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:16.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:15 smithi115 bash[45495]: cluster 2022-04-22T23:12:14.927362+0000 mgr.y (mgr.34107) 11795 : cluster [DBG] pgmap v12324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:16.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:15 smithi115 bash[46934]: cluster 2022-04-22T23:12:14.927362+0000 mgr.y (mgr.34107) 11795 : cluster [DBG] pgmap v12324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:15 smithi121 bash[36902]: cluster 2022-04-22T23:12:14.927362+0000 mgr.y (mgr.34107) 11795 : cluster [DBG] pgmap v12324: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:17.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:16 smithi115 bash[46934]: audit 2022-04-22T23:12:16.428024+0000 mon.a (mon.0) 16297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:16 smithi115 bash[45495]: audit 2022-04-22T23:12:16.428024+0000 mon.a (mon.0) 16297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:17.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:16 smithi121 bash[36902]: audit 2022-04-22T23:12:16.428024+0000 mon.a (mon.0) 16297 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[45495]: cluster 2022-04-22T23:12:16.928548+0000 mgr.y (mgr.34107) 11796 : cluster [DBG] pgmap v12325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[45495]: audit 2022-04-22T23:12:17.249274+0000 mon.a (mon.0) 16298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:12:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[45495]: audit 2022-04-22T23:12:17.257763+0000 mon.a (mon.0) 16299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:12:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[46934]: cluster 2022-04-22T23:12:16.928548+0000 mgr.y (mgr.34107) 11796 : cluster [DBG] pgmap v12325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[46934]: audit 2022-04-22T23:12:17.249274+0000 mon.a (mon.0) 16298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:12:18.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:17 smithi115 bash[46934]: audit 2022-04-22T23:12:17.257763+0000 mon.a (mon.0) 16299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:12:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:17 smithi121 bash[36902]: cluster 2022-04-22T23:12:16.928548+0000 mgr.y (mgr.34107) 11796 : cluster [DBG] pgmap v12325: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:17 smithi121 bash[36902]: audit 2022-04-22T23:12:17.249274+0000 mon.a (mon.0) 16298 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:12:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:17 smithi121 bash[36902]: audit 2022-04-22T23:12:17.257763+0000 mon.a (mon.0) 16299 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:12:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:18 smithi115 bash[45495]: audit 2022-04-22T23:12:18.062096+0000 mon.a (mon.0) 16300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:18 smithi115 bash[46934]: audit 2022-04-22T23:12:18.062096+0000 mon.a (mon.0) 16300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:19.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:18 smithi121 bash[36902]: audit 2022-04-22T23:12:18.062096+0000 mon.a (mon.0) 16300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:19 smithi115 bash[45495]: cluster 2022-04-22T23:12:18.929809+0000 mgr.y (mgr.34107) 11797 : cluster [DBG] pgmap v12326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:20.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:19 smithi115 bash[45495]: audit 2022-04-22T23:12:19.706546+0000 mon.a (mon.0) 16301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:19 smithi115 bash[46934]: cluster 2022-04-22T23:12:18.929809+0000 mgr.y (mgr.34107) 11797 : cluster [DBG] pgmap v12326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:20.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:19 smithi115 bash[46934]: audit 2022-04-22T23:12:19.706546+0000 mon.a (mon.0) 16301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:19 smithi121 bash[36902]: cluster 2022-04-22T23:12:18.929809+0000 mgr.y (mgr.34107) 11797 : cluster [DBG] pgmap v12326: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:19 smithi121 bash[36902]: audit 2022-04-22T23:12:19.706546+0000 mon.a (mon.0) 16301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:21 smithi121 bash[36902]: cluster 2022-04-22T23:12:20.930954+0000 mgr.y (mgr.34107) 11798 : cluster [DBG] pgmap v12327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:22.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:21 smithi121 bash[36902]: audit 2022-04-22T23:12:21.430864+0000 mon.a (mon.0) 16302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:21 smithi115 bash[45495]: cluster 2022-04-22T23:12:20.930954+0000 mgr.y (mgr.34107) 11798 : cluster [DBG] pgmap v12327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:22.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:21 smithi115 bash[45495]: audit 2022-04-22T23:12:21.430864+0000 mon.a (mon.0) 16302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:21 smithi115 bash[46934]: cluster 2022-04-22T23:12:20.930954+0000 mgr.y (mgr.34107) 11798 : cluster [DBG] pgmap v12327: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:22.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:21 smithi115 bash[46934]: audit 2022-04-22T23:12:21.430864+0000 mon.a (mon.0) 16302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:22.982Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:23 smithi121 bash[36902]: cluster 2022-04-22T23:12:22.932034+0000 mgr.y (mgr.34107) 11799 : cluster [DBG] pgmap v12328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:23 smithi121 bash[36902]: audit 2022-04-22T23:12:23.065325+0000 mon.a (mon.0) 16303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:23 smithi115 bash[45495]: cluster 2022-04-22T23:12:22.932034+0000 mgr.y (mgr.34107) 11799 : cluster [DBG] pgmap v12328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:23 smithi115 bash[45495]: audit 2022-04-22T23:12:23.065325+0000 mon.a (mon.0) 16303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:23 smithi115 bash[46934]: cluster 2022-04-22T23:12:22.932034+0000 mgr.y (mgr.34107) 11799 : cluster [DBG] pgmap v12328: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:23 smithi115 bash[46934]: audit 2022-04-22T23:12:23.065325+0000 mon.a (mon.0) 16303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:24 smithi115 bash[45495]: audit 2022-04-22T23:12:24.710414+0000 mon.a (mon.0) 16304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:24.837Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:24 smithi115 bash[46934]: audit 2022-04-22T23:12:24.710414+0000 mon.a (mon.0) 16304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:25.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:24 smithi121 bash[36902]: audit 2022-04-22T23:12:24.710414+0000 mon.a (mon.0) 16304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:25 smithi115 bash[45495]: cluster 2022-04-22T23:12:24.933051+0000 mgr.y (mgr.34107) 11800 : cluster [DBG] pgmap v12329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:26.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:25 smithi115 bash[46934]: cluster 2022-04-22T23:12:24.933051+0000 mgr.y (mgr.34107) 11800 : cluster [DBG] pgmap v12329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:25 smithi121 bash[36902]: cluster 2022-04-22T23:12:24.933051+0000 mgr.y (mgr.34107) 11800 : cluster [DBG] pgmap v12329: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:27.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:26 smithi115 bash[45495]: audit 2022-04-22T23:12:26.433973+0000 mon.a (mon.0) 16305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:27.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:26 smithi115 bash[46934]: audit 2022-04-22T23:12:26.433973+0000 mon.a (mon.0) 16305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:27.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:26 smithi121 bash[36902]: audit 2022-04-22T23:12:26.433973+0000 mon.a (mon.0) 16305 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[45495]: cluster 2022-04-22T23:12:26.934171+0000 mgr.y (mgr.34107) 11801 : cluster [DBG] pgmap v12330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[45495]: audit 2022-04-22T23:12:27.060302+0000 mon.a (mon.0) 16306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[45495]: audit 2022-04-22T23:12:27.064104+0000 mon.a (mon.0) 16307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[45495]: audit 2022-04-22T23:12:27.065773+0000 mon.a (mon.0) 16308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[45495]: audit 2022-04-22T23:12:27.421575+0000 mon.a (mon.0) 16309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:12:28.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[46934]: cluster 2022-04-22T23:12:26.934171+0000 mgr.y (mgr.34107) 11801 : cluster [DBG] pgmap v12330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[46934]: audit 2022-04-22T23:12:27.060302+0000 mon.a (mon.0) 16306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:12:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[46934]: audit 2022-04-22T23:12:27.064104+0000 mon.a (mon.0) 16307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:12:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[46934]: audit 2022-04-22T23:12:27.065773+0000 mon.a (mon.0) 16308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:12:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:27 smithi115 bash[46934]: audit 2022-04-22T23:12:27.421575+0000 mon.a (mon.0) 16309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:12:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:27 smithi121 bash[36902]: cluster 2022-04-22T23:12:26.934171+0000 mgr.y (mgr.34107) 11801 : cluster [DBG] pgmap v12330: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:27 smithi121 bash[36902]: audit 2022-04-22T23:12:27.060302+0000 mon.a (mon.0) 16306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:12:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:27 smithi121 bash[36902]: audit 2022-04-22T23:12:27.064104+0000 mon.a (mon.0) 16307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:12:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:27 smithi121 bash[36902]: audit 2022-04-22T23:12:27.065773+0000 mon.a (mon.0) 16308 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:12:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:27 smithi121 bash[36902]: audit 2022-04-22T23:12:27.421575+0000 mon.a (mon.0) 16309 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:12:29.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:28 smithi115 bash[45495]: audit 2022-04-22T23:12:28.068206+0000 mon.a (mon.0) 16310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:29.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:28 smithi115 bash[46934]: audit 2022-04-22T23:12:28.068206+0000 mon.a (mon.0) 16310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:28 smithi121 bash[36902]: audit 2022-04-22T23:12:28.068206+0000 mon.a (mon.0) 16310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:29 smithi121 bash[36902]: cluster 2022-04-22T23:12:28.935349+0000 mgr.y (mgr.34107) 11802 : cluster [DBG] pgmap v12331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:29 smithi121 bash[36902]: audit 2022-04-22T23:12:29.714025+0000 mon.a (mon.0) 16311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:29 smithi115 bash[45495]: cluster 2022-04-22T23:12:28.935349+0000 mgr.y (mgr.34107) 11802 : cluster [DBG] pgmap v12331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:29 smithi115 bash[45495]: audit 2022-04-22T23:12:29.714025+0000 mon.a (mon.0) 16311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:29 smithi115 bash[46934]: cluster 2022-04-22T23:12:28.935349+0000 mgr.y (mgr.34107) 11802 : cluster [DBG] pgmap v12331: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:29 smithi115 bash[46934]: audit 2022-04-22T23:12:29.714025+0000 mon.a (mon.0) 16311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:32.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:31 smithi121 bash[36902]: cluster 2022-04-22T23:12:30.936442+0000 mgr.y (mgr.34107) 11803 : cluster [DBG] pgmap v12332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:32.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:31 smithi121 bash[36902]: audit 2022-04-22T23:12:31.437007+0000 mon.a (mon.0) 16312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:31 smithi115 bash[45495]: cluster 2022-04-22T23:12:30.936442+0000 mgr.y (mgr.34107) 11803 : cluster [DBG] pgmap v12332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:31 smithi115 bash[45495]: audit 2022-04-22T23:12:31.437007+0000 mon.a (mon.0) 16312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:31 smithi115 bash[46934]: cluster 2022-04-22T23:12:30.936442+0000 mgr.y (mgr.34107) 11803 : cluster [DBG] pgmap v12332: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:31 smithi115 bash[46934]: audit 2022-04-22T23:12:31.437007+0000 mon.a (mon.0) 16312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:32.984Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:33.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:33 smithi121 bash[36902]: cluster 2022-04-22T23:12:32.937512+0000 mgr.y (mgr.34107) 11804 : cluster [DBG] pgmap v12333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:33 smithi121 bash[36902]: audit 2022-04-22T23:12:33.071215+0000 mon.a (mon.0) 16313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:33 smithi115 bash[45495]: cluster 2022-04-22T23:12:32.937512+0000 mgr.y (mgr.34107) 11804 : cluster [DBG] pgmap v12333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:33 smithi115 bash[45495]: audit 2022-04-22T23:12:33.071215+0000 mon.a (mon.0) 16313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:33 smithi115 bash[46934]: cluster 2022-04-22T23:12:32.937512+0000 mgr.y (mgr.34107) 11804 : cluster [DBG] pgmap v12333: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:33 smithi115 bash[46934]: audit 2022-04-22T23:12:33.071215+0000 mon.a (mon.0) 16313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:34 smithi115 bash[45495]: audit 2022-04-22T23:12:34.717654+0000 mon.a (mon.0) 16314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:35.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:34.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:34 smithi115 bash[46934]: audit 2022-04-22T23:12:34.717654+0000 mon.a (mon.0) 16314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:34 smithi121 bash[36902]: audit 2022-04-22T23:12:34.717654+0000 mon.a (mon.0) 16314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:35 smithi115 bash[45495]: cluster 2022-04-22T23:12:34.938762+0000 mgr.y (mgr.34107) 11805 : cluster [DBG] pgmap v12334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:35 smithi115 bash[46934]: cluster 2022-04-22T23:12:34.938762+0000 mgr.y (mgr.34107) 11805 : cluster [DBG] pgmap v12334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:36.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:35 smithi121 bash[36902]: cluster 2022-04-22T23:12:34.938762+0000 mgr.y (mgr.34107) 11805 : cluster [DBG] pgmap v12334: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:37.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:36 smithi115 bash[45495]: audit 2022-04-22T23:12:36.440462+0000 mon.a (mon.0) 16315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:37.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:36 smithi115 bash[46934]: audit 2022-04-22T23:12:36.440462+0000 mon.a (mon.0) 16315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:36 smithi121 bash[36902]: audit 2022-04-22T23:12:36.440462+0000 mon.a (mon.0) 16315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:38.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:37 smithi121 bash[36902]: cluster 2022-04-22T23:12:36.939947+0000 mgr.y (mgr.34107) 11806 : cluster [DBG] pgmap v12335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:37 smithi115 bash[45495]: cluster 2022-04-22T23:12:36.939947+0000 mgr.y (mgr.34107) 11806 : cluster [DBG] pgmap v12335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:37 smithi115 bash[46934]: cluster 2022-04-22T23:12:36.939947+0000 mgr.y (mgr.34107) 11806 : cluster [DBG] pgmap v12335: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:38 smithi121 bash[36902]: audit 2022-04-22T23:12:38.074245+0000 mon.a (mon.0) 16316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:39.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:38 smithi115 bash[46934]: audit 2022-04-22T23:12:38.074245+0000 mon.a (mon.0) 16316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:38 smithi115 bash[45495]: audit 2022-04-22T23:12:38.074245+0000 mon.a (mon.0) 16316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:40.208 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:12:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:39 smithi121 bash[36902]: cluster 2022-04-22T23:12:38.941211+0000 mgr.y (mgr.34107) 11807 : cluster [DBG] pgmap v12336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:39 smithi121 bash[36902]: audit 2022-04-22T23:12:39.721371+0000 mon.a (mon.0) 16317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:39 smithi115 bash[45495]: cluster 2022-04-22T23:12:38.941211+0000 mgr.y (mgr.34107) 11807 : cluster [DBG] pgmap v12336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:39 smithi115 bash[45495]: audit 2022-04-22T23:12:39.721371+0000 mon.a (mon.0) 16317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:39 smithi115 bash[46934]: cluster 2022-04-22T23:12:38.941211+0000 mgr.y (mgr.34107) 11807 : cluster [DBG] pgmap v12336: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:39 smithi115 bash[46934]: audit 2022-04-22T23:12:39.721371+0000 mon.a (mon.0) 16317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:40.547 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:12:40.548 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:12:40.549 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:12:40.550 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:12:40.550 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:12:40.550 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:12:40.550 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:12:40.954 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:12:40.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:12:40.956 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:12:41.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:40 smithi121 bash[36902]: audit 2022-04-22T23:12:40.187388+0000 mgr.y (mgr.34107) 11808 : audit [DBG] from='client.57505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:40 smithi115 bash[45495]: audit 2022-04-22T23:12:40.187388+0000 mgr.y (mgr.34107) 11808 : audit [DBG] from='client.57505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:40 smithi115 bash[46934]: audit 2022-04-22T23:12:40.187388+0000 mgr.y (mgr.34107) 11808 : audit [DBG] from='client.57505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:41 smithi121 bash[36902]: audit 2022-04-22T23:12:40.525660+0000 mgr.y (mgr.34107) 11809 : audit [DBG] from='client.47571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:41 smithi121 bash[36902]: cluster 2022-04-22T23:12:40.941986+0000 mgr.y (mgr.34107) 11810 : cluster [DBG] pgmap v12337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:41 smithi121 bash[36902]: audit 2022-04-22T23:12:40.951063+0000 mon.a (mon.0) 16318 : audit [DBG] from='client.? 172.21.15.115:0/562900950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:42.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:41 smithi121 bash[36902]: audit 2022-04-22T23:12:41.443585+0000 mon.a (mon.0) 16319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[45495]: audit 2022-04-22T23:12:40.525660+0000 mgr.y (mgr.34107) 11809 : audit [DBG] from='client.47571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[45495]: cluster 2022-04-22T23:12:40.941986+0000 mgr.y (mgr.34107) 11810 : cluster [DBG] pgmap v12337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[45495]: audit 2022-04-22T23:12:40.951063+0000 mon.a (mon.0) 16318 : audit [DBG] from='client.? 172.21.15.115:0/562900950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[45495]: audit 2022-04-22T23:12:41.443585+0000 mon.a (mon.0) 16319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[46934]: audit 2022-04-22T23:12:40.525660+0000 mgr.y (mgr.34107) 11809 : audit [DBG] from='client.47571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:12:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[46934]: cluster 2022-04-22T23:12:40.941986+0000 mgr.y (mgr.34107) 11810 : cluster [DBG] pgmap v12337: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[46934]: audit 2022-04-22T23:12:40.951063+0000 mon.a (mon.0) 16318 : audit [DBG] from='client.? 172.21.15.115:0/562900950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:12:42.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:41 smithi115 bash[46934]: audit 2022-04-22T23:12:41.443585+0000 mon.a (mon.0) 16319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:42.985Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:43 smithi121 bash[36902]: cluster 2022-04-22T23:12:42.943034+0000 mgr.y (mgr.34107) 11811 : cluster [DBG] pgmap v12338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:43 smithi121 bash[36902]: audit 2022-04-22T23:12:43.077075+0000 mon.a (mon.0) 16320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:43 smithi115 bash[45495]: cluster 2022-04-22T23:12:42.943034+0000 mgr.y (mgr.34107) 11811 : cluster [DBG] pgmap v12338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:43 smithi115 bash[45495]: audit 2022-04-22T23:12:43.077075+0000 mon.a (mon.0) 16320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:43 smithi115 bash[46934]: cluster 2022-04-22T23:12:42.943034+0000 mgr.y (mgr.34107) 11811 : cluster [DBG] pgmap v12338: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:43 smithi115 bash[46934]: audit 2022-04-22T23:12:43.077075+0000 mon.a (mon.0) 16320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:45.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:44 smithi115 bash[45495]: audit 2022-04-22T23:12:44.724967+0000 mon.a (mon.0) 16321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:45.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:44 smithi115 bash[46934]: audit 2022-04-22T23:12:44.724967+0000 mon.a (mon.0) 16321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:45.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:44.838Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:44 smithi121 bash[36902]: audit 2022-04-22T23:12:44.724967+0000 mon.a (mon.0) 16321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:46.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:45 smithi115 bash[45495]: cluster 2022-04-22T23:12:44.944175+0000 mgr.y (mgr.34107) 11812 : cluster [DBG] pgmap v12339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:45 smithi115 bash[46934]: cluster 2022-04-22T23:12:44.944175+0000 mgr.y (mgr.34107) 11812 : cluster [DBG] pgmap v12339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:46.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:45 smithi121 bash[36902]: cluster 2022-04-22T23:12:44.944175+0000 mgr.y (mgr.34107) 11812 : cluster [DBG] pgmap v12339: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:47.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:46 smithi121 bash[36902]: audit 2022-04-22T23:12:46.446623+0000 mon.a (mon.0) 16322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:46 smithi115 bash[45495]: audit 2022-04-22T23:12:46.446623+0000 mon.a (mon.0) 16322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:46 smithi115 bash[46934]: audit 2022-04-22T23:12:46.446623+0000 mon.a (mon.0) 16322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:47 smithi121 bash[36902]: cluster 2022-04-22T23:12:46.945302+0000 mgr.y (mgr.34107) 11813 : cluster [DBG] pgmap v12340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:47 smithi115 bash[45495]: cluster 2022-04-22T23:12:46.945302+0000 mgr.y (mgr.34107) 11813 : cluster [DBG] pgmap v12340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:47 smithi115 bash[46934]: cluster 2022-04-22T23:12:46.945302+0000 mgr.y (mgr.34107) 11813 : cluster [DBG] pgmap v12340: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:48 smithi121 bash[36902]: audit 2022-04-22T23:12:48.079911+0000 mon.a (mon.0) 16323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:48 smithi115 bash[45495]: audit 2022-04-22T23:12:48.079911+0000 mon.a (mon.0) 16323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:48 smithi115 bash[46934]: audit 2022-04-22T23:12:48.079911+0000 mon.a (mon.0) 16323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:49 smithi121 bash[36902]: cluster 2022-04-22T23:12:48.946533+0000 mgr.y (mgr.34107) 11814 : cluster [DBG] pgmap v12341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:49 smithi121 bash[36902]: audit 2022-04-22T23:12:49.728740+0000 mon.a (mon.0) 16324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:49 smithi115 bash[45495]: cluster 2022-04-22T23:12:48.946533+0000 mgr.y (mgr.34107) 11814 : cluster [DBG] pgmap v12341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:49 smithi115 bash[45495]: audit 2022-04-22T23:12:49.728740+0000 mon.a (mon.0) 16324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:49 smithi115 bash[46934]: cluster 2022-04-22T23:12:48.946533+0000 mgr.y (mgr.34107) 11814 : cluster [DBG] pgmap v12341: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:49 smithi115 bash[46934]: audit 2022-04-22T23:12:49.728740+0000 mon.a (mon.0) 16324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:52.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:52 smithi121 bash[36902]: cluster 2022-04-22T23:12:50.947719+0000 mgr.y (mgr.34107) 11815 : cluster [DBG] pgmap v12342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:52.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:52 smithi121 bash[36902]: audit 2022-04-22T23:12:51.449850+0000 mon.a (mon.0) 16325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:52 smithi115 bash[45495]: cluster 2022-04-22T23:12:50.947719+0000 mgr.y (mgr.34107) 11815 : cluster [DBG] pgmap v12342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:52.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:52 smithi115 bash[45495]: audit 2022-04-22T23:12:51.449850+0000 mon.a (mon.0) 16325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:52 smithi115 bash[46934]: cluster 2022-04-22T23:12:50.947719+0000 mgr.y (mgr.34107) 11815 : cluster [DBG] pgmap v12342: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:52.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:52 smithi115 bash[46934]: audit 2022-04-22T23:12:51.449850+0000 mon.a (mon.0) 16325 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:52.984Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:53 smithi121 bash[36902]: cluster 2022-04-22T23:12:52.948919+0000 mgr.y (mgr.34107) 11816 : cluster [DBG] pgmap v12343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:53 smithi121 bash[36902]: audit 2022-04-22T23:12:53.082526+0000 mon.a (mon.0) 16326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:53 smithi115 bash[45495]: cluster 2022-04-22T23:12:52.948919+0000 mgr.y (mgr.34107) 11816 : cluster [DBG] pgmap v12343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:53 smithi115 bash[45495]: audit 2022-04-22T23:12:53.082526+0000 mon.a (mon.0) 16326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:53 smithi115 bash[46934]: cluster 2022-04-22T23:12:52.948919+0000 mgr.y (mgr.34107) 11816 : cluster [DBG] pgmap v12343: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:53 smithi115 bash[46934]: audit 2022-04-22T23:12:53.082526+0000 mon.a (mon.0) 16326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:54 smithi115 bash[45495]: audit 2022-04-22T23:12:54.732381+0000 mon.a (mon.0) 16327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:12:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:12:54.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:12:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:54 smithi115 bash[46934]: audit 2022-04-22T23:12:54.732381+0000 mon.a (mon.0) 16327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:55.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:54 smithi121 bash[36902]: audit 2022-04-22T23:12:54.732381+0000 mon.a (mon.0) 16327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:12:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:55 smithi121 bash[36902]: cluster 2022-04-22T23:12:54.950088+0000 mgr.y (mgr.34107) 11817 : cluster [DBG] pgmap v12344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:55 smithi115 bash[45495]: cluster 2022-04-22T23:12:54.950088+0000 mgr.y (mgr.34107) 11817 : cluster [DBG] pgmap v12344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:55 smithi115 bash[46934]: cluster 2022-04-22T23:12:54.950088+0000 mgr.y (mgr.34107) 11817 : cluster [DBG] pgmap v12344: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:57.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:56 smithi121 bash[36902]: audit 2022-04-22T23:12:56.453019+0000 mon.a (mon.0) 16328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:56 smithi115 bash[45495]: audit 2022-04-22T23:12:56.453019+0000 mon.a (mon.0) 16328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:56 smithi115 bash[46934]: audit 2022-04-22T23:12:56.453019+0000 mon.a (mon.0) 16328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:12:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:57 smithi121 bash[36902]: cluster 2022-04-22T23:12:56.951174+0000 mgr.y (mgr.34107) 11818 : cluster [DBG] pgmap v12345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:57 smithi115 bash[45495]: cluster 2022-04-22T23:12:56.951174+0000 mgr.y (mgr.34107) 11818 : cluster [DBG] pgmap v12345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:57 smithi115 bash[46934]: cluster 2022-04-22T23:12:56.951174+0000 mgr.y (mgr.34107) 11818 : cluster [DBG] pgmap v12345: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:12:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:58 smithi121 bash[36902]: audit 2022-04-22T23:12:58.085517+0000 mon.a (mon.0) 16329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:58 smithi115 bash[45495]: audit 2022-04-22T23:12:58.085517+0000 mon.a (mon.0) 16329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:12:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:58 smithi115 bash[46934]: audit 2022-04-22T23:12:58.085517+0000 mon.a (mon.0) 16329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:59 smithi121 bash[36902]: cluster 2022-04-22T23:12:58.952370+0000 mgr.y (mgr.34107) 11819 : cluster [DBG] pgmap v12346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:12:59 smithi121 bash[36902]: audit 2022-04-22T23:12:59.736061+0000 mon.a (mon.0) 16330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:59 smithi115 bash[45495]: cluster 2022-04-22T23:12:58.952370+0000 mgr.y (mgr.34107) 11819 : cluster [DBG] pgmap v12346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:12:59 smithi115 bash[45495]: audit 2022-04-22T23:12:59.736061+0000 mon.a (mon.0) 16330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:59 smithi115 bash[46934]: cluster 2022-04-22T23:12:58.952370+0000 mgr.y (mgr.34107) 11819 : cluster [DBG] pgmap v12346: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:12:59 smithi115 bash[46934]: audit 2022-04-22T23:12:59.736061+0000 mon.a (mon.0) 16330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:02 smithi121 bash[36902]: cluster 2022-04-22T23:13:00.953599+0000 mgr.y (mgr.34107) 11820 : cluster [DBG] pgmap v12347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:02 smithi121 bash[36902]: audit 2022-04-22T23:13:01.456003+0000 mon.a (mon.0) 16331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:02 smithi115 bash[45495]: cluster 2022-04-22T23:13:00.953599+0000 mgr.y (mgr.34107) 11820 : cluster [DBG] pgmap v12347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:02.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:02 smithi115 bash[45495]: audit 2022-04-22T23:13:01.456003+0000 mon.a (mon.0) 16331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:02 smithi115 bash[46934]: cluster 2022-04-22T23:13:00.953599+0000 mgr.y (mgr.34107) 11820 : cluster [DBG] pgmap v12347: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:02.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:02 smithi115 bash[46934]: audit 2022-04-22T23:13:01.456003+0000 mon.a (mon.0) 16331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:02.986Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:03 smithi121 bash[36902]: cluster 2022-04-22T23:13:02.954873+0000 mgr.y (mgr.34107) 11821 : cluster [DBG] pgmap v12348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:03 smithi121 bash[36902]: audit 2022-04-22T23:13:03.088780+0000 mon.a (mon.0) 16332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:03 smithi115 bash[45495]: cluster 2022-04-22T23:13:02.954873+0000 mgr.y (mgr.34107) 11821 : cluster [DBG] pgmap v12348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:03 smithi115 bash[45495]: audit 2022-04-22T23:13:03.088780+0000 mon.a (mon.0) 16332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:03 smithi115 bash[46934]: cluster 2022-04-22T23:13:02.954873+0000 mgr.y (mgr.34107) 11821 : cluster [DBG] pgmap v12348: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:03 smithi115 bash[46934]: audit 2022-04-22T23:13:03.088780+0000 mon.a (mon.0) 16332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:04 smithi121 bash[36902]: audit 2022-04-22T23:13:04.739734+0000 mon.a (mon.0) 16333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:04.841Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:04 smithi115 bash[45495]: audit 2022-04-22T23:13:04.739734+0000 mon.a (mon.0) 16333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:04 smithi115 bash[46934]: audit 2022-04-22T23:13:04.739734+0000 mon.a (mon.0) 16333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:05 smithi121 bash[36902]: cluster 2022-04-22T23:13:04.956104+0000 mgr.y (mgr.34107) 11822 : cluster [DBG] pgmap v12349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:05 smithi115 bash[45495]: cluster 2022-04-22T23:13:04.956104+0000 mgr.y (mgr.34107) 11822 : cluster [DBG] pgmap v12349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:05 smithi115 bash[46934]: cluster 2022-04-22T23:13:04.956104+0000 mgr.y (mgr.34107) 11822 : cluster [DBG] pgmap v12349: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:06 smithi121 bash[36902]: audit 2022-04-22T23:13:06.459160+0000 mon.a (mon.0) 16334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:06 smithi115 bash[45495]: audit 2022-04-22T23:13:06.459160+0000 mon.a (mon.0) 16334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:06 smithi115 bash[46934]: audit 2022-04-22T23:13:06.459160+0000 mon.a (mon.0) 16334 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:07 smithi121 bash[36902]: cluster 2022-04-22T23:13:06.957256+0000 mgr.y (mgr.34107) 11823 : cluster [DBG] pgmap v12350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:07 smithi115 bash[45495]: cluster 2022-04-22T23:13:06.957256+0000 mgr.y (mgr.34107) 11823 : cluster [DBG] pgmap v12350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:07 smithi115 bash[46934]: cluster 2022-04-22T23:13:06.957256+0000 mgr.y (mgr.34107) 11823 : cluster [DBG] pgmap v12350: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:08 smithi121 bash[36902]: audit 2022-04-22T23:13:08.091745+0000 mon.a (mon.0) 16335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:08 smithi115 bash[45495]: audit 2022-04-22T23:13:08.091745+0000 mon.a (mon.0) 16335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:08 smithi115 bash[46934]: audit 2022-04-22T23:13:08.091745+0000 mon.a (mon.0) 16335 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:09 smithi121 bash[36902]: cluster 2022-04-22T23:13:08.958420+0000 mgr.y (mgr.34107) 11824 : cluster [DBG] pgmap v12351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:09 smithi121 bash[36902]: audit 2022-04-22T23:13:09.743618+0000 mon.a (mon.0) 16336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:09 smithi115 bash[45495]: cluster 2022-04-22T23:13:08.958420+0000 mgr.y (mgr.34107) 11824 : cluster [DBG] pgmap v12351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:09 smithi115 bash[45495]: audit 2022-04-22T23:13:09.743618+0000 mon.a (mon.0) 16336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:09 smithi115 bash[46934]: cluster 2022-04-22T23:13:08.958420+0000 mgr.y (mgr.34107) 11824 : cluster [DBG] pgmap v12351: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:09 smithi115 bash[46934]: audit 2022-04-22T23:13:09.743618+0000 mon.a (mon.0) 16336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:11.324 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:13:11.662 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:13:11.662 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:13:11.662 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:13:11.663 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:13:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:12.067 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:13:12.068 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:13:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:12 smithi115 bash[45495]: cluster 2022-04-22T23:13:10.959541+0000 mgr.y (mgr.34107) 11825 : cluster [DBG] pgmap v12352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:12 smithi115 bash[45495]: audit 2022-04-22T23:13:11.461984+0000 mon.a (mon.0) 16337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:12 smithi115 bash[46934]: cluster 2022-04-22T23:13:10.959541+0000 mgr.y (mgr.34107) 11825 : cluster [DBG] pgmap v12352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:12.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:12 smithi115 bash[46934]: audit 2022-04-22T23:13:11.461984+0000 mon.a (mon.0) 16337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:12 smithi121 bash[36902]: cluster 2022-04-22T23:13:10.959541+0000 mgr.y (mgr.34107) 11825 : cluster [DBG] pgmap v12352: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:12 smithi121 bash[36902]: audit 2022-04-22T23:13:11.461984+0000 mon.a (mon.0) 16337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:12.985Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[45495]: audit 2022-04-22T23:13:11.305074+0000 mgr.y (mgr.34107) 11826 : audit [DBG] from='client.57523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[45495]: audit 2022-04-22T23:13:11.644234+0000 mgr.y (mgr.34107) 11827 : audit [DBG] from='client.57529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[45495]: audit 2022-04-22T23:13:12.063360+0000 mon.c (mon.1) 399 : audit [DBG] from='client.? 172.21.15.115:0/246894610' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[46934]: audit 2022-04-22T23:13:11.305074+0000 mgr.y (mgr.34107) 11826 : audit [DBG] from='client.57523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[46934]: audit 2022-04-22T23:13:11.644234+0000 mgr.y (mgr.34107) 11827 : audit [DBG] from='client.57529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:13 smithi115 bash[46934]: audit 2022-04-22T23:13:12.063360+0000 mon.c (mon.1) 399 : audit [DBG] from='client.? 172.21.15.115:0/246894610' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:13 smithi121 bash[36902]: audit 2022-04-22T23:13:11.305074+0000 mgr.y (mgr.34107) 11826 : audit [DBG] from='client.57523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:13 smithi121 bash[36902]: audit 2022-04-22T23:13:11.644234+0000 mgr.y (mgr.34107) 11827 : audit [DBG] from='client.57529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:13 smithi121 bash[36902]: audit 2022-04-22T23:13:12.063360+0000 mon.c (mon.1) 399 : audit [DBG] from='client.? 172.21.15.115:0/246894610' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:14.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:14 smithi115 bash[45495]: cluster 2022-04-22T23:13:12.960668+0000 mgr.y (mgr.34107) 11828 : cluster [DBG] pgmap v12353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:14 smithi115 bash[45495]: audit 2022-04-22T23:13:13.094881+0000 mon.a (mon.0) 16338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:14 smithi115 bash[46934]: cluster 2022-04-22T23:13:12.960668+0000 mgr.y (mgr.34107) 11828 : cluster [DBG] pgmap v12353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:14 smithi115 bash[46934]: audit 2022-04-22T23:13:13.094881+0000 mon.a (mon.0) 16338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:14 smithi121 bash[36902]: cluster 2022-04-22T23:13:12.960668+0000 mgr.y (mgr.34107) 11828 : cluster [DBG] pgmap v12353: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:14 smithi121 bash[36902]: audit 2022-04-22T23:13:13.094881+0000 mon.a (mon.0) 16338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:15 smithi115 bash[46934]: audit 2022-04-22T23:13:14.747413+0000 mon.a (mon.0) 16339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:14.841Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:15 smithi115 bash[45495]: audit 2022-04-22T23:13:14.747413+0000 mon.a (mon.0) 16339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:15 smithi121 bash[36902]: audit 2022-04-22T23:13:14.747413+0000 mon.a (mon.0) 16339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:16 smithi115 bash[45495]: cluster 2022-04-22T23:13:14.961818+0000 mgr.y (mgr.34107) 11829 : cluster [DBG] pgmap v12354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:16 smithi115 bash[46934]: cluster 2022-04-22T23:13:14.961818+0000 mgr.y (mgr.34107) 11829 : cluster [DBG] pgmap v12354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:16.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:16 smithi121 bash[36902]: cluster 2022-04-22T23:13:14.961818+0000 mgr.y (mgr.34107) 11829 : cluster [DBG] pgmap v12354: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:17 smithi121 bash[36902]: audit 2022-04-22T23:13:16.465218+0000 mon.a (mon.0) 16340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:17 smithi115 bash[45495]: audit 2022-04-22T23:13:16.465218+0000 mon.a (mon.0) 16340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:17 smithi115 bash[46934]: audit 2022-04-22T23:13:16.465218+0000 mon.a (mon.0) 16340 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:18 smithi121 bash[36902]: cluster 2022-04-22T23:13:16.962840+0000 mgr.y (mgr.34107) 11830 : cluster [DBG] pgmap v12355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:18 smithi121 bash[36902]: audit 2022-04-22T23:13:17.267730+0000 mon.a (mon.0) 16341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:13:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:18 smithi121 bash[36902]: audit 2022-04-22T23:13:17.268507+0000 mon.a (mon.0) 16342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:13:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[45495]: cluster 2022-04-22T23:13:16.962840+0000 mgr.y (mgr.34107) 11830 : cluster [DBG] pgmap v12355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:18.741 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[45495]: audit 2022-04-22T23:13:17.267730+0000 mon.a (mon.0) 16341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:13:18.742 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[45495]: audit 2022-04-22T23:13:17.268507+0000 mon.a (mon.0) 16342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:13:18.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[46934]: cluster 2022-04-22T23:13:16.962840+0000 mgr.y (mgr.34107) 11830 : cluster [DBG] pgmap v12355: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:18.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[46934]: audit 2022-04-22T23:13:17.267730+0000 mon.a (mon.0) 16341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:13:18.743 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:18 smithi115 bash[46934]: audit 2022-04-22T23:13:17.268507+0000 mon.a (mon.0) 16342 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:13:19.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:19 smithi121 bash[36902]: audit 2022-04-22T23:13:18.098080+0000 mon.a (mon.0) 16343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:19 smithi115 bash[45495]: audit 2022-04-22T23:13:18.098080+0000 mon.a (mon.0) 16343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:19 smithi115 bash[46934]: audit 2022-04-22T23:13:18.098080+0000 mon.a (mon.0) 16343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:20 smithi121 bash[36902]: cluster 2022-04-22T23:13:18.963957+0000 mgr.y (mgr.34107) 11831 : cluster [DBG] pgmap v12356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:20 smithi121 bash[36902]: audit 2022-04-22T23:13:19.751018+0000 mon.a (mon.0) 16344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:20 smithi115 bash[45495]: cluster 2022-04-22T23:13:18.963957+0000 mgr.y (mgr.34107) 11831 : cluster [DBG] pgmap v12356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:20 smithi115 bash[45495]: audit 2022-04-22T23:13:19.751018+0000 mon.a (mon.0) 16344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:20 smithi115 bash[46934]: cluster 2022-04-22T23:13:18.963957+0000 mgr.y (mgr.34107) 11831 : cluster [DBG] pgmap v12356: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:20 smithi115 bash[46934]: audit 2022-04-22T23:13:19.751018+0000 mon.a (mon.0) 16344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:22 smithi121 bash[36902]: cluster 2022-04-22T23:13:20.965029+0000 mgr.y (mgr.34107) 11832 : cluster [DBG] pgmap v12357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:22 smithi121 bash[36902]: audit 2022-04-22T23:13:21.468223+0000 mon.a (mon.0) 16345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:22 smithi115 bash[45495]: cluster 2022-04-22T23:13:20.965029+0000 mgr.y (mgr.34107) 11832 : cluster [DBG] pgmap v12357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:22 smithi115 bash[45495]: audit 2022-04-22T23:13:21.468223+0000 mon.a (mon.0) 16345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:22 smithi115 bash[46934]: cluster 2022-04-22T23:13:20.965029+0000 mgr.y (mgr.34107) 11832 : cluster [DBG] pgmap v12357: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:22 smithi115 bash[46934]: audit 2022-04-22T23:13:21.468223+0000 mon.a (mon.0) 16345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:22.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:23 smithi121 bash[36902]: cluster 2022-04-22T23:13:22.966229+0000 mgr.y (mgr.34107) 11833 : cluster [DBG] pgmap v12358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:23 smithi121 bash[36902]: audit 2022-04-22T23:13:23.100900+0000 mon.a (mon.0) 16346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:23 smithi115 bash[45495]: cluster 2022-04-22T23:13:22.966229+0000 mgr.y (mgr.34107) 11833 : cluster [DBG] pgmap v12358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:23 smithi115 bash[45495]: audit 2022-04-22T23:13:23.100900+0000 mon.a (mon.0) 16346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:23 smithi115 bash[46934]: cluster 2022-04-22T23:13:22.966229+0000 mgr.y (mgr.34107) 11833 : cluster [DBG] pgmap v12358: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:23 smithi115 bash[46934]: audit 2022-04-22T23:13:23.100900+0000 mon.a (mon.0) 16346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:24 smithi121 bash[36902]: audit 2022-04-22T23:13:24.754788+0000 mon.a (mon.0) 16347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:24 smithi115 bash[45495]: audit 2022-04-22T23:13:24.754788+0000 mon.a (mon.0) 16347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:24.842Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:24 smithi115 bash[46934]: audit 2022-04-22T23:13:24.754788+0000 mon.a (mon.0) 16347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:26.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:25 smithi121 bash[36902]: cluster 2022-04-22T23:13:24.967373+0000 mgr.y (mgr.34107) 11834 : cluster [DBG] pgmap v12359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:25 smithi115 bash[45495]: cluster 2022-04-22T23:13:24.967373+0000 mgr.y (mgr.34107) 11834 : cluster [DBG] pgmap v12359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:25 smithi115 bash[46934]: cluster 2022-04-22T23:13:24.967373+0000 mgr.y (mgr.34107) 11834 : cluster [DBG] pgmap v12359: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:26 smithi121 bash[36902]: audit 2022-04-22T23:13:26.471335+0000 mon.a (mon.0) 16348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:26 smithi115 bash[45495]: audit 2022-04-22T23:13:26.471335+0000 mon.a (mon.0) 16348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:26 smithi115 bash[46934]: audit 2022-04-22T23:13:26.471335+0000 mon.a (mon.0) 16348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[45495]: cluster 2022-04-22T23:13:26.968603+0000 mgr.y (mgr.34107) 11835 : cluster [DBG] pgmap v12360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[45495]: audit 2022-04-22T23:13:27.430774+0000 mon.a (mon.0) 16349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:13:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[45495]: audit 2022-04-22T23:13:27.433623+0000 mon.a (mon.0) 16350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:13:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[45495]: audit 2022-04-22T23:13:27.435375+0000 mon.a (mon.0) 16351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:13:28.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[45495]: audit 2022-04-22T23:13:27.786488+0000 mon.a (mon.0) 16352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:13:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[46934]: cluster 2022-04-22T23:13:26.968603+0000 mgr.y (mgr.34107) 11835 : cluster [DBG] pgmap v12360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[46934]: audit 2022-04-22T23:13:27.430774+0000 mon.a (mon.0) 16349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:13:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[46934]: audit 2022-04-22T23:13:27.433623+0000 mon.a (mon.0) 16350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:13:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[46934]: audit 2022-04-22T23:13:27.435375+0000 mon.a (mon.0) 16351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:13:28.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:27 smithi115 bash[46934]: audit 2022-04-22T23:13:27.786488+0000 mon.a (mon.0) 16352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:13:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:27 smithi121 bash[36902]: cluster 2022-04-22T23:13:26.968603+0000 mgr.y (mgr.34107) 11835 : cluster [DBG] pgmap v12360: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:27 smithi121 bash[36902]: audit 2022-04-22T23:13:27.430774+0000 mon.a (mon.0) 16349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:13:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:27 smithi121 bash[36902]: audit 2022-04-22T23:13:27.433623+0000 mon.a (mon.0) 16350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:13:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:27 smithi121 bash[36902]: audit 2022-04-22T23:13:27.435375+0000 mon.a (mon.0) 16351 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:13:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:27 smithi121 bash[36902]: audit 2022-04-22T23:13:27.786488+0000 mon.a (mon.0) 16352 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:13:29.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:28 smithi121 bash[36902]: audit 2022-04-22T23:13:28.103711+0000 mon.a (mon.0) 16353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:28 smithi115 bash[45495]: audit 2022-04-22T23:13:28.103711+0000 mon.a (mon.0) 16353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:28 smithi115 bash[46934]: audit 2022-04-22T23:13:28.103711+0000 mon.a (mon.0) 16353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:30.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:29 smithi121 bash[36902]: cluster 2022-04-22T23:13:28.969814+0000 mgr.y (mgr.34107) 11836 : cluster [DBG] pgmap v12361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:29 smithi121 bash[36902]: audit 2022-04-22T23:13:29.758666+0000 mon.a (mon.0) 16354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:29 smithi115 bash[45495]: cluster 2022-04-22T23:13:28.969814+0000 mgr.y (mgr.34107) 11836 : cluster [DBG] pgmap v12361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:29 smithi115 bash[45495]: audit 2022-04-22T23:13:29.758666+0000 mon.a (mon.0) 16354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:29 smithi115 bash[46934]: cluster 2022-04-22T23:13:28.969814+0000 mgr.y (mgr.34107) 11836 : cluster [DBG] pgmap v12361: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:29 smithi115 bash[46934]: audit 2022-04-22T23:13:29.758666+0000 mon.a (mon.0) 16354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:32 smithi115 bash[45495]: cluster 2022-04-22T23:13:30.971022+0000 mgr.y (mgr.34107) 11837 : cluster [DBG] pgmap v12362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:32.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:32 smithi115 bash[45495]: audit 2022-04-22T23:13:31.474478+0000 mon.a (mon.0) 16355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:32.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:32 smithi115 bash[46934]: cluster 2022-04-22T23:13:30.971022+0000 mgr.y (mgr.34107) 11837 : cluster [DBG] pgmap v12362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:32.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:32 smithi115 bash[46934]: audit 2022-04-22T23:13:31.474478+0000 mon.a (mon.0) 16355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:32 smithi121 bash[36902]: cluster 2022-04-22T23:13:30.971022+0000 mgr.y (mgr.34107) 11837 : cluster [DBG] pgmap v12362: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:32 smithi121 bash[36902]: audit 2022-04-22T23:13:31.474478+0000 mon.a (mon.0) 16355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:32.987Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:33 smithi121 bash[36902]: cluster 2022-04-22T23:13:32.972138+0000 mgr.y (mgr.34107) 11838 : cluster [DBG] pgmap v12363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:33 smithi121 bash[36902]: audit 2022-04-22T23:13:33.106554+0000 mon.a (mon.0) 16356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:33 smithi115 bash[45495]: cluster 2022-04-22T23:13:32.972138+0000 mgr.y (mgr.34107) 11838 : cluster [DBG] pgmap v12363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:33 smithi115 bash[45495]: audit 2022-04-22T23:13:33.106554+0000 mon.a (mon.0) 16356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:33 smithi115 bash[46934]: cluster 2022-04-22T23:13:32.972138+0000 mgr.y (mgr.34107) 11838 : cluster [DBG] pgmap v12363: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:33 smithi115 bash[46934]: audit 2022-04-22T23:13:33.106554+0000 mon.a (mon.0) 16356 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:34 smithi121 bash[36902]: audit 2022-04-22T23:13:34.762374+0000 mon.a (mon.0) 16357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:34 smithi115 bash[45495]: audit 2022-04-22T23:13:34.762374+0000 mon.a (mon.0) 16357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:34 smithi115 bash[46934]: audit 2022-04-22T23:13:34.762374+0000 mon.a (mon.0) 16357 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:34.842Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:35 smithi121 bash[36902]: cluster 2022-04-22T23:13:34.973268+0000 mgr.y (mgr.34107) 11839 : cluster [DBG] pgmap v12364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:35 smithi115 bash[45495]: cluster 2022-04-22T23:13:34.973268+0000 mgr.y (mgr.34107) 11839 : cluster [DBG] pgmap v12364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:35 smithi115 bash[46934]: cluster 2022-04-22T23:13:34.973268+0000 mgr.y (mgr.34107) 11839 : cluster [DBG] pgmap v12364: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:37.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:36 smithi121 bash[36902]: audit 2022-04-22T23:13:36.477646+0000 mon.a (mon.0) 16358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:36 smithi115 bash[45495]: audit 2022-04-22T23:13:36.477646+0000 mon.a (mon.0) 16358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:36 smithi115 bash[46934]: audit 2022-04-22T23:13:36.477646+0000 mon.a (mon.0) 16358 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:37 smithi121 bash[36902]: cluster 2022-04-22T23:13:36.974428+0000 mgr.y (mgr.34107) 11840 : cluster [DBG] pgmap v12365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:37 smithi115 bash[45495]: cluster 2022-04-22T23:13:36.974428+0000 mgr.y (mgr.34107) 11840 : cluster [DBG] pgmap v12365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:37 smithi115 bash[46934]: cluster 2022-04-22T23:13:36.974428+0000 mgr.y (mgr.34107) 11840 : cluster [DBG] pgmap v12365: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:39.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:38 smithi121 bash[36902]: audit 2022-04-22T23:13:38.109635+0000 mon.a (mon.0) 16359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:38 smithi115 bash[45495]: audit 2022-04-22T23:13:38.109635+0000 mon.a (mon.0) 16359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:38 smithi115 bash[46934]: audit 2022-04-22T23:13:38.109635+0000 mon.a (mon.0) 16359 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:39 smithi121 bash[36902]: cluster 2022-04-22T23:13:38.975524+0000 mgr.y (mgr.34107) 11841 : cluster [DBG] pgmap v12366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:39 smithi121 bash[36902]: audit 2022-04-22T23:13:39.766042+0000 mon.a (mon.0) 16360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:40.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:39 smithi115 bash[45495]: cluster 2022-04-22T23:13:38.975524+0000 mgr.y (mgr.34107) 11841 : cluster [DBG] pgmap v12366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:39 smithi115 bash[45495]: audit 2022-04-22T23:13:39.766042+0000 mon.a (mon.0) 16360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:39 smithi115 bash[46934]: cluster 2022-04-22T23:13:38.975524+0000 mgr.y (mgr.34107) 11841 : cluster [DBG] pgmap v12366: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:39 smithi115 bash[46934]: audit 2022-04-22T23:13:39.766042+0000 mon.a (mon.0) 16360 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:42 smithi115 bash[46934]: cluster 2022-04-22T23:13:40.976646+0000 mgr.y (mgr.34107) 11842 : cluster [DBG] pgmap v12367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:42 smithi115 bash[46934]: audit 2022-04-22T23:13:41.480835+0000 mon.a (mon.0) 16361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:42 smithi115 bash[45495]: cluster 2022-04-22T23:13:40.976646+0000 mgr.y (mgr.34107) 11842 : cluster [DBG] pgmap v12367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:42 smithi115 bash[45495]: audit 2022-04-22T23:13:41.480835+0000 mon.a (mon.0) 16361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:42.443 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:13:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:42 smithi121 bash[36902]: cluster 2022-04-22T23:13:40.976646+0000 mgr.y (mgr.34107) 11842 : cluster [DBG] pgmap v12367: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:42 smithi121 bash[36902]: audit 2022-04-22T23:13:41.480835+0000 mon.a (mon.0) 16361 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:42.791 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:13:42.791 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:13:42.791 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:13:42.791 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:13:42.792 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:13:42.792 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:13:42.792 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:13:42.792 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:13:42.793 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:13:42.794 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:13:42.794 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:13:42.794 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:13:42.794 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:13:42.794 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:13:43.193 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:13:43.193 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:13:43.193 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:13:43.194 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:13:43.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:13:43.195 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:13:43.195 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:13:43.195 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:13:43.195 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:13:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:42.988Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:43 smithi121 bash[36902]: audit 2022-04-22T23:13:42.425023+0000 mgr.y (mgr.34107) 11843 : audit [DBG] from='client.47601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:43 smithi121 bash[36902]: audit 2022-04-22T23:13:42.769230+0000 mgr.y (mgr.34107) 11844 : audit [DBG] from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:43 smithi121 bash[36902]: cluster 2022-04-22T23:13:42.977739+0000 mgr.y (mgr.34107) 11845 : cluster [DBG] pgmap v12368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:43 smithi121 bash[36902]: audit 2022-04-22T23:13:43.112334+0000 mon.a (mon.0) 16362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:43 smithi121 bash[36902]: audit 2022-04-22T23:13:43.189855+0000 mon.a (mon.0) 16363 : audit [DBG] from='client.? 172.21.15.115:0/2469557217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[45495]: audit 2022-04-22T23:13:42.425023+0000 mgr.y (mgr.34107) 11843 : audit [DBG] from='client.47601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[45495]: audit 2022-04-22T23:13:42.769230+0000 mgr.y (mgr.34107) 11844 : audit [DBG] from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[45495]: cluster 2022-04-22T23:13:42.977739+0000 mgr.y (mgr.34107) 11845 : cluster [DBG] pgmap v12368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[45495]: audit 2022-04-22T23:13:43.112334+0000 mon.a (mon.0) 16362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[45495]: audit 2022-04-22T23:13:43.189855+0000 mon.a (mon.0) 16363 : audit [DBG] from='client.? 172.21.15.115:0/2469557217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[46934]: audit 2022-04-22T23:13:42.425023+0000 mgr.y (mgr.34107) 11843 : audit [DBG] from='client.47601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[46934]: audit 2022-04-22T23:13:42.769230+0000 mgr.y (mgr.34107) 11844 : audit [DBG] from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:13:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[46934]: cluster 2022-04-22T23:13:42.977739+0000 mgr.y (mgr.34107) 11845 : cluster [DBG] pgmap v12368: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[46934]: audit 2022-04-22T23:13:43.112334+0000 mon.a (mon.0) 16362 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:43 smithi115 bash[46934]: audit 2022-04-22T23:13:43.189855+0000 mon.a (mon.0) 16363 : audit [DBG] from='client.? 172.21.15.115:0/2469557217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:13:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:44 smithi121 bash[36902]: audit 2022-04-22T23:13:44.769958+0000 mon.a (mon.0) 16364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:44 smithi115 bash[45495]: audit 2022-04-22T23:13:44.769958+0000 mon.a (mon.0) 16364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:44 smithi115 bash[46934]: audit 2022-04-22T23:13:44.769958+0000 mon.a (mon.0) 16364 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:44.840Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:45 smithi121 bash[36902]: cluster 2022-04-22T23:13:44.978917+0000 mgr.y (mgr.34107) 11846 : cluster [DBG] pgmap v12369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:45 smithi115 bash[45495]: cluster 2022-04-22T23:13:44.978917+0000 mgr.y (mgr.34107) 11846 : cluster [DBG] pgmap v12369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:45 smithi115 bash[46934]: cluster 2022-04-22T23:13:44.978917+0000 mgr.y (mgr.34107) 11846 : cluster [DBG] pgmap v12369: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:46 smithi121 bash[36902]: audit 2022-04-22T23:13:46.483629+0000 mon.a (mon.0) 16365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:46 smithi115 bash[45495]: audit 2022-04-22T23:13:46.483629+0000 mon.a (mon.0) 16365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:46 smithi115 bash[46934]: audit 2022-04-22T23:13:46.483629+0000 mon.a (mon.0) 16365 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:48.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:47 smithi121 bash[36902]: cluster 2022-04-22T23:13:46.980100+0000 mgr.y (mgr.34107) 11847 : cluster [DBG] pgmap v12370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:47 smithi115 bash[45495]: cluster 2022-04-22T23:13:46.980100+0000 mgr.y (mgr.34107) 11847 : cluster [DBG] pgmap v12370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:47 smithi115 bash[46934]: cluster 2022-04-22T23:13:46.980100+0000 mgr.y (mgr.34107) 11847 : cluster [DBG] pgmap v12370: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:49.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:48 smithi121 bash[36902]: audit 2022-04-22T23:13:48.114953+0000 mon.a (mon.0) 16366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:49.366 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:48 smithi115 bash[45495]: audit 2022-04-22T23:13:48.114953+0000 mon.a (mon.0) 16366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:48 smithi115 bash[46934]: audit 2022-04-22T23:13:48.114953+0000 mon.a (mon.0) 16366 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:49 smithi121 bash[36902]: cluster 2022-04-22T23:13:48.981199+0000 mgr.y (mgr.34107) 11848 : cluster [DBG] pgmap v12371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:49 smithi121 bash[36902]: audit 2022-04-22T23:13:49.773566+0000 mon.a (mon.0) 16367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:49 smithi115 bash[45495]: cluster 2022-04-22T23:13:48.981199+0000 mgr.y (mgr.34107) 11848 : cluster [DBG] pgmap v12371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:49 smithi115 bash[45495]: audit 2022-04-22T23:13:49.773566+0000 mon.a (mon.0) 16367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:49 smithi115 bash[46934]: cluster 2022-04-22T23:13:48.981199+0000 mgr.y (mgr.34107) 11848 : cluster [DBG] pgmap v12371: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:49 smithi115 bash[46934]: audit 2022-04-22T23:13:49.773566+0000 mon.a (mon.0) 16367 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:52 smithi121 bash[36902]: cluster 2022-04-22T23:13:50.982467+0000 mgr.y (mgr.34107) 11849 : cluster [DBG] pgmap v12372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:52 smithi121 bash[36902]: audit 2022-04-22T23:13:51.486504+0000 mon.a (mon.0) 16368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:52 smithi115 bash[45495]: cluster 2022-04-22T23:13:50.982467+0000 mgr.y (mgr.34107) 11849 : cluster [DBG] pgmap v12372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:52 smithi115 bash[45495]: audit 2022-04-22T23:13:51.486504+0000 mon.a (mon.0) 16368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:52 smithi115 bash[46934]: cluster 2022-04-22T23:13:50.982467+0000 mgr.y (mgr.34107) 11849 : cluster [DBG] pgmap v12372: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:52.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:52 smithi115 bash[46934]: audit 2022-04-22T23:13:51.486504+0000 mon.a (mon.0) 16368 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:52.990Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:53 smithi121 bash[36902]: cluster 2022-04-22T23:13:52.983521+0000 mgr.y (mgr.34107) 11850 : cluster [DBG] pgmap v12373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:53 smithi121 bash[36902]: audit 2022-04-22T23:13:53.117546+0000 mon.a (mon.0) 16369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:53.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:53 smithi115 bash[45495]: cluster 2022-04-22T23:13:52.983521+0000 mgr.y (mgr.34107) 11850 : cluster [DBG] pgmap v12373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:53 smithi115 bash[45495]: audit 2022-04-22T23:13:53.117546+0000 mon.a (mon.0) 16369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:53 smithi115 bash[46934]: cluster 2022-04-22T23:13:52.983521+0000 mgr.y (mgr.34107) 11850 : cluster [DBG] pgmap v12373: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:53 smithi115 bash[46934]: audit 2022-04-22T23:13:53.117546+0000 mon.a (mon.0) 16369 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:54 smithi121 bash[36902]: audit 2022-04-22T23:13:54.777237+0000 mon.a (mon.0) 16370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:54 smithi115 bash[45495]: audit 2022-04-22T23:13:54.777237+0000 mon.a (mon.0) 16370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:13:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:13:54.842Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:13:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:54 smithi115 bash[46934]: audit 2022-04-22T23:13:54.777237+0000 mon.a (mon.0) 16370 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:13:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:55 smithi121 bash[36902]: cluster 2022-04-22T23:13:54.984596+0000 mgr.y (mgr.34107) 11851 : cluster [DBG] pgmap v12374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:55 smithi115 bash[45495]: cluster 2022-04-22T23:13:54.984596+0000 mgr.y (mgr.34107) 11851 : cluster [DBG] pgmap v12374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:55 smithi115 bash[46934]: cluster 2022-04-22T23:13:54.984596+0000 mgr.y (mgr.34107) 11851 : cluster [DBG] pgmap v12374: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:56 smithi121 bash[36902]: audit 2022-04-22T23:13:56.489997+0000 mon.a (mon.0) 16371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:56 smithi115 bash[45495]: audit 2022-04-22T23:13:56.489997+0000 mon.a (mon.0) 16371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:56 smithi115 bash[46934]: audit 2022-04-22T23:13:56.489997+0000 mon.a (mon.0) 16371 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:13:58.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:57 smithi121 bash[36902]: cluster 2022-04-22T23:13:56.985721+0000 mgr.y (mgr.34107) 11852 : cluster [DBG] pgmap v12375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:57 smithi115 bash[45495]: cluster 2022-04-22T23:13:56.985721+0000 mgr.y (mgr.34107) 11852 : cluster [DBG] pgmap v12375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:57 smithi115 bash[46934]: cluster 2022-04-22T23:13:56.985721+0000 mgr.y (mgr.34107) 11852 : cluster [DBG] pgmap v12375: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:13:59.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:58 smithi121 bash[36902]: audit 2022-04-22T23:13:58.119992+0000 mon.a (mon.0) 16372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:58 smithi115 bash[45495]: audit 2022-04-22T23:13:58.119992+0000 mon.a (mon.0) 16372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:13:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:58 smithi115 bash[46934]: audit 2022-04-22T23:13:58.119992+0000 mon.a (mon.0) 16372 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:00.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:59 smithi121 bash[36902]: cluster 2022-04-22T23:13:58.986826+0000 mgr.y (mgr.34107) 11853 : cluster [DBG] pgmap v12376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:13:59 smithi121 bash[36902]: audit 2022-04-22T23:13:59.781104+0000 mon.a (mon.0) 16373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:59 smithi115 bash[45495]: cluster 2022-04-22T23:13:58.986826+0000 mgr.y (mgr.34107) 11853 : cluster [DBG] pgmap v12376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:13:59 smithi115 bash[45495]: audit 2022-04-22T23:13:59.781104+0000 mon.a (mon.0) 16373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:59 smithi115 bash[46934]: cluster 2022-04-22T23:13:58.986826+0000 mgr.y (mgr.34107) 11853 : cluster [DBG] pgmap v12376: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:13:59 smithi115 bash[46934]: audit 2022-04-22T23:13:59.781104+0000 mon.a (mon.0) 16373 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:02 smithi121 bash[36902]: cluster 2022-04-22T23:14:00.987953+0000 mgr.y (mgr.34107) 11854 : cluster [DBG] pgmap v12377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:02 smithi121 bash[36902]: audit 2022-04-22T23:14:01.493060+0000 mon.a (mon.0) 16374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:02.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:02 smithi115 bash[45495]: cluster 2022-04-22T23:14:00.987953+0000 mgr.y (mgr.34107) 11854 : cluster [DBG] pgmap v12377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:02 smithi115 bash[45495]: audit 2022-04-22T23:14:01.493060+0000 mon.a (mon.0) 16374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:02 smithi115 bash[46934]: cluster 2022-04-22T23:14:00.987953+0000 mgr.y (mgr.34107) 11854 : cluster [DBG] pgmap v12377: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:02 smithi115 bash[46934]: audit 2022-04-22T23:14:01.493060+0000 mon.a (mon.0) 16374 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:02.991Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:03 smithi121 bash[36902]: cluster 2022-04-22T23:14:02.989269+0000 mgr.y (mgr.34107) 11855 : cluster [DBG] pgmap v12378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:03 smithi121 bash[36902]: audit 2022-04-22T23:14:03.122546+0000 mon.a (mon.0) 16375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:03 smithi115 bash[45495]: cluster 2022-04-22T23:14:02.989269+0000 mgr.y (mgr.34107) 11855 : cluster [DBG] pgmap v12378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:03 smithi115 bash[45495]: audit 2022-04-22T23:14:03.122546+0000 mon.a (mon.0) 16375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:03 smithi115 bash[46934]: cluster 2022-04-22T23:14:02.989269+0000 mgr.y (mgr.34107) 11855 : cluster [DBG] pgmap v12378: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:03 smithi115 bash[46934]: audit 2022-04-22T23:14:03.122546+0000 mon.a (mon.0) 16375 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:05.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:04 smithi121 bash[36902]: audit 2022-04-22T23:14:04.785008+0000 mon.a (mon.0) 16376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:04 smithi115 bash[45495]: audit 2022-04-22T23:14:04.785008+0000 mon.a (mon.0) 16376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:04.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:04 smithi115 bash[46934]: audit 2022-04-22T23:14:04.785008+0000 mon.a (mon.0) 16376 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:05 smithi121 bash[36902]: cluster 2022-04-22T23:14:04.990450+0000 mgr.y (mgr.34107) 11856 : cluster [DBG] pgmap v12379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:05 smithi115 bash[45495]: cluster 2022-04-22T23:14:04.990450+0000 mgr.y (mgr.34107) 11856 : cluster [DBG] pgmap v12379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:05 smithi115 bash[46934]: cluster 2022-04-22T23:14:04.990450+0000 mgr.y (mgr.34107) 11856 : cluster [DBG] pgmap v12379: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:06 smithi121 bash[36902]: audit 2022-04-22T23:14:06.496189+0000 mon.a (mon.0) 16377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:06 smithi115 bash[45495]: audit 2022-04-22T23:14:06.496189+0000 mon.a (mon.0) 16377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:06 smithi115 bash[46934]: audit 2022-04-22T23:14:06.496189+0000 mon.a (mon.0) 16377 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:08.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:07 smithi121 bash[36902]: cluster 2022-04-22T23:14:06.991671+0000 mgr.y (mgr.34107) 11857 : cluster [DBG] pgmap v12380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:07 smithi115 bash[45495]: cluster 2022-04-22T23:14:06.991671+0000 mgr.y (mgr.34107) 11857 : cluster [DBG] pgmap v12380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:07 smithi115 bash[46934]: cluster 2022-04-22T23:14:06.991671+0000 mgr.y (mgr.34107) 11857 : cluster [DBG] pgmap v12380: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:08 smithi121 bash[36902]: audit 2022-04-22T23:14:08.125441+0000 mon.a (mon.0) 16378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:08 smithi115 bash[45495]: audit 2022-04-22T23:14:08.125441+0000 mon.a (mon.0) 16378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:08 smithi115 bash[46934]: audit 2022-04-22T23:14:08.125441+0000 mon.a (mon.0) 16378 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:09 smithi121 bash[36902]: cluster 2022-04-22T23:14:08.992815+0000 mgr.y (mgr.34107) 11858 : cluster [DBG] pgmap v12381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:09 smithi121 bash[36902]: audit 2022-04-22T23:14:09.788745+0000 mon.a (mon.0) 16379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:09 smithi115 bash[45495]: cluster 2022-04-22T23:14:08.992815+0000 mgr.y (mgr.34107) 11858 : cluster [DBG] pgmap v12381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:09 smithi115 bash[45495]: audit 2022-04-22T23:14:09.788745+0000 mon.a (mon.0) 16379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:09 smithi115 bash[46934]: cluster 2022-04-22T23:14:08.992815+0000 mgr.y (mgr.34107) 11858 : cluster [DBG] pgmap v12381: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:09 smithi115 bash[46934]: audit 2022-04-22T23:14:09.788745+0000 mon.a (mon.0) 16379 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:12 smithi121 bash[36902]: cluster 2022-04-22T23:14:10.993894+0000 mgr.y (mgr.34107) 11859 : cluster [DBG] pgmap v12382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:12.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:12 smithi121 bash[36902]: audit 2022-04-22T23:14:11.499047+0000 mon.a (mon.0) 16380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:12.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:12 smithi115 bash[45495]: cluster 2022-04-22T23:14:10.993894+0000 mgr.y (mgr.34107) 11859 : cluster [DBG] pgmap v12382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:12.581 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:12 smithi115 bash[45495]: audit 2022-04-22T23:14:11.499047+0000 mon.a (mon.0) 16380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:12.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:12 smithi115 bash[46934]: cluster 2022-04-22T23:14:10.993894+0000 mgr.y (mgr.34107) 11859 : cluster [DBG] pgmap v12382: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:12.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:12 smithi115 bash[46934]: audit 2022-04-22T23:14:11.499047+0000 mon.a (mon.0) 16380 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:12.991Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:13.564 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:14:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:13 smithi121 bash[36902]: cluster 2022-04-22T23:14:12.994652+0000 mgr.y (mgr.34107) 11860 : cluster [DBG] pgmap v12383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:13 smithi121 bash[36902]: audit 2022-04-22T23:14:13.128123+0000 mon.a (mon.0) 16381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:13.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:13 smithi115 bash[46934]: cluster 2022-04-22T23:14:12.994652+0000 mgr.y (mgr.34107) 11860 : cluster [DBG] pgmap v12383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:13 smithi115 bash[46934]: audit 2022-04-22T23:14:13.128123+0000 mon.a (mon.0) 16381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:13 smithi115 bash[45495]: cluster 2022-04-22T23:14:12.994652+0000 mgr.y (mgr.34107) 11860 : cluster [DBG] pgmap v12383: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:13 smithi115 bash[45495]: audit 2022-04-22T23:14:13.128123+0000 mon.a (mon.0) 16381 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:14:13.916 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:14:13.917 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:14:13.918 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:14:13.918 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:14:13.918 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:14:13.918 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:14:13.918 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:14:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:14:14.330 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:14.331 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:14:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:14:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:14 smithi121 bash[36902]: audit 2022-04-22T23:14:13.545596+0000 mgr.y (mgr.34107) 11861 : audit [DBG] from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:14 smithi121 bash[36902]: audit 2022-04-22T23:14:13.893779+0000 mgr.y (mgr.34107) 11862 : audit [DBG] from='client.47625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:14 smithi121 bash[36902]: audit 2022-04-22T23:14:14.326693+0000 mon.c (mon.1) 400 : audit [DBG] from='client.? 172.21.15.115:0/2890402501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[45495]: audit 2022-04-22T23:14:13.545596+0000 mgr.y (mgr.34107) 11861 : audit [DBG] from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[45495]: audit 2022-04-22T23:14:13.893779+0000 mgr.y (mgr.34107) 11862 : audit [DBG] from='client.47625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[45495]: audit 2022-04-22T23:14:14.326693+0000 mon.c (mon.1) 400 : audit [DBG] from='client.? 172.21.15.115:0/2890402501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[46934]: audit 2022-04-22T23:14:13.545596+0000 mgr.y (mgr.34107) 11861 : audit [DBG] from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[46934]: audit 2022-04-22T23:14:13.893779+0000 mgr.y (mgr.34107) 11862 : audit [DBG] from='client.47625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[46934]: audit 2022-04-22T23:14:14.326693+0000 mon.c (mon.1) 400 : audit [DBG] from='client.? 172.21.15.115:0/2890402501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:14.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:15.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:15 smithi121 bash[36902]: audit 2022-04-22T23:14:14.792302+0000 mon.a (mon.0) 16382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:15 smithi121 bash[36902]: cluster 2022-04-22T23:14:14.995784+0000 mgr.y (mgr.34107) 11863 : cluster [DBG] pgmap v12384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:15.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:15 smithi115 bash[45495]: audit 2022-04-22T23:14:14.792302+0000 mon.a (mon.0) 16382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:15 smithi115 bash[45495]: cluster 2022-04-22T23:14:14.995784+0000 mgr.y (mgr.34107) 11863 : cluster [DBG] pgmap v12384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:15 smithi115 bash[46934]: audit 2022-04-22T23:14:14.792302+0000 mon.a (mon.0) 16382 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:15 smithi115 bash[46934]: cluster 2022-04-22T23:14:14.995784+0000 mgr.y (mgr.34107) 11863 : cluster [DBG] pgmap v12384: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:17.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:16 smithi121 bash[36902]: audit 2022-04-22T23:14:16.502105+0000 mon.a (mon.0) 16383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:17.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:16 smithi115 bash[46934]: audit 2022-04-22T23:14:16.502105+0000 mon.a (mon.0) 16383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:17.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:16 smithi115 bash[45495]: audit 2022-04-22T23:14:16.502105+0000 mon.a (mon.0) 16383 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:17 smithi121 bash[36902]: cluster 2022-04-22T23:14:16.996862+0000 mgr.y (mgr.34107) 11864 : cluster [DBG] pgmap v12385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:17 smithi121 bash[36902]: audit 2022-04-22T23:14:17.280008+0000 mon.a (mon.0) 16384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:14:18.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:17 smithi121 bash[36902]: audit 2022-04-22T23:14:17.281692+0000 mon.a (mon.0) 16385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:14:18.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[45495]: cluster 2022-04-22T23:14:16.996862+0000 mgr.y (mgr.34107) 11864 : cluster [DBG] pgmap v12385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[45495]: audit 2022-04-22T23:14:17.280008+0000 mon.a (mon.0) 16384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:14:18.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[45495]: audit 2022-04-22T23:14:17.281692+0000 mon.a (mon.0) 16385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:14:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[46934]: cluster 2022-04-22T23:14:16.996862+0000 mgr.y (mgr.34107) 11864 : cluster [DBG] pgmap v12385: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[46934]: audit 2022-04-22T23:14:17.280008+0000 mon.a (mon.0) 16384 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:14:18.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:17 smithi115 bash[46934]: audit 2022-04-22T23:14:17.281692+0000 mon.a (mon.0) 16385 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:14:19.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:18 smithi121 bash[36902]: audit 2022-04-22T23:14:18.131098+0000 mon.a (mon.0) 16386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:19.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:18 smithi115 bash[45495]: audit 2022-04-22T23:14:18.131098+0000 mon.a (mon.0) 16386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:18 smithi115 bash[46934]: audit 2022-04-22T23:14:18.131098+0000 mon.a (mon.0) 16386 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:20.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:19 smithi121 bash[36902]: cluster 2022-04-22T23:14:18.997951+0000 mgr.y (mgr.34107) 11865 : cluster [DBG] pgmap v12386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:20.145 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:19 smithi115 bash[45495]: cluster 2022-04-22T23:14:18.997951+0000 mgr.y (mgr.34107) 11865 : cluster [DBG] pgmap v12386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:20.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:19 smithi115 bash[46934]: cluster 2022-04-22T23:14:18.997951+0000 mgr.y (mgr.34107) 11865 : cluster [DBG] pgmap v12386: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:21.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:20 smithi121 bash[36902]: audit 2022-04-22T23:14:19.795889+0000 mon.a (mon.0) 16387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:21.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:20 smithi115 bash[46934]: audit 2022-04-22T23:14:19.795889+0000 mon.a (mon.0) 16387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:21.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:20 smithi115 bash[45495]: audit 2022-04-22T23:14:19.795889+0000 mon.a (mon.0) 16387 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:21 smithi121 bash[36902]: cluster 2022-04-22T23:14:20.998935+0000 mgr.y (mgr.34107) 11866 : cluster [DBG] pgmap v12387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:22.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:21 smithi121 bash[36902]: audit 2022-04-22T23:14:21.504960+0000 mon.a (mon.0) 16388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:21 smithi115 bash[45495]: cluster 2022-04-22T23:14:20.998935+0000 mgr.y (mgr.34107) 11866 : cluster [DBG] pgmap v12387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:22.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:21 smithi115 bash[45495]: audit 2022-04-22T23:14:21.504960+0000 mon.a (mon.0) 16388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:21 smithi115 bash[46934]: cluster 2022-04-22T23:14:20.998935+0000 mgr.y (mgr.34107) 11866 : cluster [DBG] pgmap v12387: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:22.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:21 smithi115 bash[46934]: audit 2022-04-22T23:14:21.504960+0000 mon.a (mon.0) 16388 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:22.992Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:23.764 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:23 smithi121 bash[36902]: cluster 2022-04-22T23:14:22.999996+0000 mgr.y (mgr.34107) 11867 : cluster [DBG] pgmap v12388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:23 smithi121 bash[36902]: audit 2022-04-22T23:14:23.134266+0000 mon.a (mon.0) 16389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:23 smithi115 bash[45495]: cluster 2022-04-22T23:14:22.999996+0000 mgr.y (mgr.34107) 11867 : cluster [DBG] pgmap v12388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:23 smithi115 bash[45495]: audit 2022-04-22T23:14:23.134266+0000 mon.a (mon.0) 16389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:23 smithi115 bash[46934]: cluster 2022-04-22T23:14:22.999996+0000 mgr.y (mgr.34107) 11867 : cluster [DBG] pgmap v12388: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:23 smithi115 bash[46934]: audit 2022-04-22T23:14:23.134266+0000 mon.a (mon.0) 16389 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:24 smithi121 bash[36902]: audit 2022-04-22T23:14:24.799647+0000 mon.a (mon.0) 16390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:24 smithi115 bash[45495]: audit 2022-04-22T23:14:24.799647+0000 mon.a (mon.0) 16390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:24 smithi115 bash[46934]: audit 2022-04-22T23:14:24.799647+0000 mon.a (mon.0) 16390 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:24.846Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:25 smithi121 bash[36902]: cluster 2022-04-22T23:14:25.001123+0000 mgr.y (mgr.34107) 11868 : cluster [DBG] pgmap v12389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:25 smithi115 bash[45495]: cluster 2022-04-22T23:14:25.001123+0000 mgr.y (mgr.34107) 11868 : cluster [DBG] pgmap v12389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:25 smithi115 bash[46934]: cluster 2022-04-22T23:14:25.001123+0000 mgr.y (mgr.34107) 11868 : cluster [DBG] pgmap v12389: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:26 smithi121 bash[36902]: audit 2022-04-22T23:14:26.507819+0000 mon.a (mon.0) 16391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:26 smithi115 bash[45495]: audit 2022-04-22T23:14:26.507819+0000 mon.a (mon.0) 16391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:26 smithi115 bash[46934]: audit 2022-04-22T23:14:26.507819+0000 mon.a (mon.0) 16391 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:28.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:27 smithi121 bash[36902]: cluster 2022-04-22T23:14:27.002170+0000 mgr.y (mgr.34107) 11869 : cluster [DBG] pgmap v12390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:28.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:27 smithi121 bash[36902]: audit 2022-04-22T23:14:27.795452+0000 mon.a (mon.0) 16392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:14:28.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:27 smithi121 bash[36902]: audit 2022-04-22T23:14:27.798968+0000 mon.a (mon.0) 16393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:14:28.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:27 smithi121 bash[36902]: audit 2022-04-22T23:14:27.800970+0000 mon.a (mon.0) 16394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:14:28.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[46934]: cluster 2022-04-22T23:14:27.002170+0000 mgr.y (mgr.34107) 11869 : cluster [DBG] pgmap v12390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[46934]: audit 2022-04-22T23:14:27.795452+0000 mon.a (mon.0) 16392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[46934]: audit 2022-04-22T23:14:27.798968+0000 mon.a (mon.0) 16393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[46934]: audit 2022-04-22T23:14:27.800970+0000 mon.a (mon.0) 16394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[45495]: cluster 2022-04-22T23:14:27.002170+0000 mgr.y (mgr.34107) 11869 : cluster [DBG] pgmap v12390: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[45495]: audit 2022-04-22T23:14:27.795452+0000 mon.a (mon.0) 16392 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:14:28.150 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[45495]: audit 2022-04-22T23:14:27.798968+0000 mon.a (mon.0) 16393 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:14:28.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:27 smithi115 bash[45495]: audit 2022-04-22T23:14:27.800970+0000 mon.a (mon.0) 16394 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:14:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:28 smithi121 bash[36902]: audit 2022-04-22T23:14:28.137286+0000 mon.a (mon.0) 16395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:28 smithi121 bash[36902]: audit 2022-04-22T23:14:28.158387+0000 mon.a (mon.0) 16396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:14:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:28 smithi115 bash[46934]: audit 2022-04-22T23:14:28.137286+0000 mon.a (mon.0) 16395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:28 smithi115 bash[46934]: audit 2022-04-22T23:14:28.158387+0000 mon.a (mon.0) 16396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:14:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:28 smithi115 bash[45495]: audit 2022-04-22T23:14:28.137286+0000 mon.a (mon.0) 16395 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:28 smithi115 bash[45495]: audit 2022-04-22T23:14:28.158387+0000 mon.a (mon.0) 16396 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:14:30.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:29 smithi121 bash[36902]: cluster 2022-04-22T23:14:29.003238+0000 mgr.y (mgr.34107) 11870 : cluster [DBG] pgmap v12391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:30.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:29 smithi121 bash[36902]: audit 2022-04-22T23:14:29.803408+0000 mon.a (mon.0) 16397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:30.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:29 smithi115 bash[45495]: cluster 2022-04-22T23:14:29.003238+0000 mgr.y (mgr.34107) 11870 : cluster [DBG] pgmap v12391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:30.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:29 smithi115 bash[45495]: audit 2022-04-22T23:14:29.803408+0000 mon.a (mon.0) 16397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:30.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:29 smithi115 bash[46934]: cluster 2022-04-22T23:14:29.003238+0000 mgr.y (mgr.34107) 11870 : cluster [DBG] pgmap v12391: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:30.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:29 smithi115 bash[46934]: audit 2022-04-22T23:14:29.803408+0000 mon.a (mon.0) 16397 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:32 smithi121 bash[36902]: cluster 2022-04-22T23:14:31.004310+0000 mgr.y (mgr.34107) 11871 : cluster [DBG] pgmap v12392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:32 smithi121 bash[36902]: audit 2022-04-22T23:14:31.510729+0000 mon.a (mon.0) 16398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:32 smithi115 bash[45495]: cluster 2022-04-22T23:14:31.004310+0000 mgr.y (mgr.34107) 11871 : cluster [DBG] pgmap v12392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:32 smithi115 bash[45495]: audit 2022-04-22T23:14:31.510729+0000 mon.a (mon.0) 16398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:32 smithi115 bash[46934]: cluster 2022-04-22T23:14:31.004310+0000 mgr.y (mgr.34107) 11871 : cluster [DBG] pgmap v12392: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:32 smithi115 bash[46934]: audit 2022-04-22T23:14:31.510729+0000 mon.a (mon.0) 16398 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:33.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:32.992Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:33 smithi121 bash[36902]: cluster 2022-04-22T23:14:33.005181+0000 mgr.y (mgr.34107) 11872 : cluster [DBG] pgmap v12393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:33 smithi121 bash[36902]: audit 2022-04-22T23:14:33.140021+0000 mon.a (mon.0) 16399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:33.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:33 smithi115 bash[45495]: cluster 2022-04-22T23:14:33.005181+0000 mgr.y (mgr.34107) 11872 : cluster [DBG] pgmap v12393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:33 smithi115 bash[45495]: audit 2022-04-22T23:14:33.140021+0000 mon.a (mon.0) 16399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:33 smithi115 bash[46934]: cluster 2022-04-22T23:14:33.005181+0000 mgr.y (mgr.34107) 11872 : cluster [DBG] pgmap v12393: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:33 smithi115 bash[46934]: audit 2022-04-22T23:14:33.140021+0000 mon.a (mon.0) 16399 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:34 smithi121 bash[36902]: audit 2022-04-22T23:14:34.807147+0000 mon.a (mon.0) 16400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:34 smithi115 bash[45495]: audit 2022-04-22T23:14:34.807147+0000 mon.a (mon.0) 16400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:34 smithi115 bash[46934]: audit 2022-04-22T23:14:34.807147+0000 mon.a (mon.0) 16400 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:34.847Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:35 smithi121 bash[36902]: cluster 2022-04-22T23:14:35.006301+0000 mgr.y (mgr.34107) 11873 : cluster [DBG] pgmap v12394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:35 smithi115 bash[45495]: cluster 2022-04-22T23:14:35.006301+0000 mgr.y (mgr.34107) 11873 : cluster [DBG] pgmap v12394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:35 smithi115 bash[46934]: cluster 2022-04-22T23:14:35.006301+0000 mgr.y (mgr.34107) 11873 : cluster [DBG] pgmap v12394: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:36 smithi121 bash[36902]: audit 2022-04-22T23:14:36.513706+0000 mon.a (mon.0) 16401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:36 smithi115 bash[45495]: audit 2022-04-22T23:14:36.513706+0000 mon.a (mon.0) 16401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:36 smithi115 bash[46934]: audit 2022-04-22T23:14:36.513706+0000 mon.a (mon.0) 16401 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:37 smithi121 bash[36902]: cluster 2022-04-22T23:14:37.007388+0000 mgr.y (mgr.34107) 11874 : cluster [DBG] pgmap v12395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:37 smithi115 bash[45495]: cluster 2022-04-22T23:14:37.007388+0000 mgr.y (mgr.34107) 11874 : cluster [DBG] pgmap v12395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:37 smithi115 bash[46934]: cluster 2022-04-22T23:14:37.007388+0000 mgr.y (mgr.34107) 11874 : cluster [DBG] pgmap v12395: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:38 smithi121 bash[36902]: audit 2022-04-22T23:14:38.142853+0000 mon.a (mon.0) 16402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:39.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:38 smithi115 bash[45495]: audit 2022-04-22T23:14:38.142853+0000 mon.a (mon.0) 16402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:38 smithi115 bash[46934]: audit 2022-04-22T23:14:38.142853+0000 mon.a (mon.0) 16402 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:39 smithi121 bash[36902]: cluster 2022-04-22T23:14:39.008501+0000 mgr.y (mgr.34107) 11875 : cluster [DBG] pgmap v12396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:39 smithi121 bash[36902]: audit 2022-04-22T23:14:39.810937+0000 mon.a (mon.0) 16403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:39 smithi115 bash[45495]: cluster 2022-04-22T23:14:39.008501+0000 mgr.y (mgr.34107) 11875 : cluster [DBG] pgmap v12396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:39 smithi115 bash[45495]: audit 2022-04-22T23:14:39.810937+0000 mon.a (mon.0) 16403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:39 smithi115 bash[46934]: cluster 2022-04-22T23:14:39.008501+0000 mgr.y (mgr.34107) 11875 : cluster [DBG] pgmap v12396: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:39 smithi115 bash[46934]: audit 2022-04-22T23:14:39.810937+0000 mon.a (mon.0) 16403 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:42 smithi121 bash[36902]: cluster 2022-04-22T23:14:41.009510+0000 mgr.y (mgr.34107) 11876 : cluster [DBG] pgmap v12397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:42 smithi121 bash[36902]: audit 2022-04-22T23:14:41.516670+0000 mon.a (mon.0) 16404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:42 smithi115 bash[45495]: cluster 2022-04-22T23:14:41.009510+0000 mgr.y (mgr.34107) 11876 : cluster [DBG] pgmap v12397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:42 smithi115 bash[45495]: audit 2022-04-22T23:14:41.516670+0000 mon.a (mon.0) 16404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:42 smithi115 bash[46934]: cluster 2022-04-22T23:14:41.009510+0000 mgr.y (mgr.34107) 11876 : cluster [DBG] pgmap v12397: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:42 smithi115 bash[46934]: audit 2022-04-22T23:14:41.516670+0000 mon.a (mon.0) 16404 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:42.992Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:43 smithi121 bash[36902]: cluster 2022-04-22T23:14:43.010329+0000 mgr.y (mgr.34107) 11877 : cluster [DBG] pgmap v12398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:43.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:43 smithi121 bash[36902]: audit 2022-04-22T23:14:43.145746+0000 mon.a (mon.0) 16405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:43 smithi115 bash[45495]: cluster 2022-04-22T23:14:43.010329+0000 mgr.y (mgr.34107) 11877 : cluster [DBG] pgmap v12398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:43 smithi115 bash[45495]: audit 2022-04-22T23:14:43.145746+0000 mon.a (mon.0) 16405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:43 smithi115 bash[46934]: cluster 2022-04-22T23:14:43.010329+0000 mgr.y (mgr.34107) 11877 : cluster [DBG] pgmap v12398: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:43 smithi115 bash[46934]: audit 2022-04-22T23:14:43.145746+0000 mon.a (mon.0) 16405 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:44.700 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:14:45.052 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:14:45.052 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:14:45.052 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:14:45.053 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:14:45.054 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:14:45.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:44 smithi121 bash[36902]: audit 2022-04-22T23:14:44.814303+0000 mon.a (mon.0) 16406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:45.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:44 smithi115 bash[46934]: audit 2022-04-22T23:14:44.814303+0000 mon.a (mon.0) 16406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:44.845Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:44 smithi115 bash[45495]: audit 2022-04-22T23:14:44.814303+0000 mon.a (mon.0) 16406 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:14:45.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:14:45.456 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:14:45.457 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:14:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:45 smithi121 bash[36902]: audit 2022-04-22T23:14:44.681133+0000 mgr.y (mgr.34107) 11878 : audit [DBG] from='client.47637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:45 smithi121 bash[36902]: cluster 2022-04-22T23:14:45.011414+0000 mgr.y (mgr.34107) 11879 : cluster [DBG] pgmap v12399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:45 smithi121 bash[36902]: audit 2022-04-22T23:14:45.027968+0000 mgr.y (mgr.34107) 11880 : audit [DBG] from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:45 smithi121 bash[36902]: audit 2022-04-22T23:14:45.451183+0000 mon.a (mon.0) 16407 : audit [DBG] from='client.? 172.21.15.115:0/1611651025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[45495]: audit 2022-04-22T23:14:44.681133+0000 mgr.y (mgr.34107) 11878 : audit [DBG] from='client.47637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[45495]: cluster 2022-04-22T23:14:45.011414+0000 mgr.y (mgr.34107) 11879 : cluster [DBG] pgmap v12399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[45495]: audit 2022-04-22T23:14:45.027968+0000 mgr.y (mgr.34107) 11880 : audit [DBG] from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[45495]: audit 2022-04-22T23:14:45.451183+0000 mon.a (mon.0) 16407 : audit [DBG] from='client.? 172.21.15.115:0/1611651025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[46934]: audit 2022-04-22T23:14:44.681133+0000 mgr.y (mgr.34107) 11878 : audit [DBG] from='client.47637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[46934]: cluster 2022-04-22T23:14:45.011414+0000 mgr.y (mgr.34107) 11879 : cluster [DBG] pgmap v12399: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[46934]: audit 2022-04-22T23:14:45.027968+0000 mgr.y (mgr.34107) 11880 : audit [DBG] from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:14:46.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:45 smithi115 bash[46934]: audit 2022-04-22T23:14:45.451183+0000 mon.a (mon.0) 16407 : audit [DBG] from='client.? 172.21.15.115:0/1611651025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:14:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:46 smithi121 bash[36902]: audit 2022-04-22T23:14:46.519728+0000 mon.a (mon.0) 16408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:46 smithi115 bash[45495]: audit 2022-04-22T23:14:46.519728+0000 mon.a (mon.0) 16408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:47.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:46 smithi115 bash[46934]: audit 2022-04-22T23:14:46.519728+0000 mon.a (mon.0) 16408 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:47 smithi121 bash[36902]: cluster 2022-04-22T23:14:47.012498+0000 mgr.y (mgr.34107) 11881 : cluster [DBG] pgmap v12400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:48.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:47 smithi115 bash[45495]: cluster 2022-04-22T23:14:47.012498+0000 mgr.y (mgr.34107) 11881 : cluster [DBG] pgmap v12400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:47 smithi115 bash[46934]: cluster 2022-04-22T23:14:47.012498+0000 mgr.y (mgr.34107) 11881 : cluster [DBG] pgmap v12400: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:48 smithi121 bash[36902]: audit 2022-04-22T23:14:48.148840+0000 mon.a (mon.0) 16409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:49.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:48 smithi115 bash[45495]: audit 2022-04-22T23:14:48.148840+0000 mon.a (mon.0) 16409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:48 smithi115 bash[46934]: audit 2022-04-22T23:14:48.148840+0000 mon.a (mon.0) 16409 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:49 smithi121 bash[36902]: cluster 2022-04-22T23:14:49.013629+0000 mgr.y (mgr.34107) 11882 : cluster [DBG] pgmap v12401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:49 smithi121 bash[36902]: audit 2022-04-22T23:14:49.817768+0000 mon.a (mon.0) 16410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:49 smithi115 bash[45495]: cluster 2022-04-22T23:14:49.013629+0000 mgr.y (mgr.34107) 11882 : cluster [DBG] pgmap v12401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:49 smithi115 bash[45495]: audit 2022-04-22T23:14:49.817768+0000 mon.a (mon.0) 16410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:49 smithi115 bash[46934]: cluster 2022-04-22T23:14:49.013629+0000 mgr.y (mgr.34107) 11882 : cluster [DBG] pgmap v12401: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:49 smithi115 bash[46934]: audit 2022-04-22T23:14:49.817768+0000 mon.a (mon.0) 16410 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:52.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:52 smithi121 bash[36902]: cluster 2022-04-22T23:14:51.014742+0000 mgr.y (mgr.34107) 11883 : cluster [DBG] pgmap v12402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:52 smithi121 bash[36902]: audit 2022-04-22T23:14:51.522659+0000 mon.a (mon.0) 16411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:52 smithi115 bash[46934]: cluster 2022-04-22T23:14:51.014742+0000 mgr.y (mgr.34107) 11883 : cluster [DBG] pgmap v12402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:52 smithi115 bash[46934]: audit 2022-04-22T23:14:51.522659+0000 mon.a (mon.0) 16411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:52 smithi115 bash[45495]: cluster 2022-04-22T23:14:51.014742+0000 mgr.y (mgr.34107) 11883 : cluster [DBG] pgmap v12402: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:52 smithi115 bash[45495]: audit 2022-04-22T23:14:51.522659+0000 mon.a (mon.0) 16411 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:52.993Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:53 smithi121 bash[36902]: cluster 2022-04-22T23:14:53.015714+0000 mgr.y (mgr.34107) 11884 : cluster [DBG] pgmap v12403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:53 smithi121 bash[36902]: audit 2022-04-22T23:14:53.151933+0000 mon.a (mon.0) 16412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:53 smithi115 bash[45495]: cluster 2022-04-22T23:14:53.015714+0000 mgr.y (mgr.34107) 11884 : cluster [DBG] pgmap v12403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:53 smithi115 bash[45495]: audit 2022-04-22T23:14:53.151933+0000 mon.a (mon.0) 16412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:53 smithi115 bash[46934]: cluster 2022-04-22T23:14:53.015714+0000 mgr.y (mgr.34107) 11884 : cluster [DBG] pgmap v12403: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:53 smithi115 bash[46934]: audit 2022-04-22T23:14:53.151933+0000 mon.a (mon.0) 16412 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:54 smithi121 bash[36902]: audit 2022-04-22T23:14:54.821508+0000 mon.a (mon.0) 16413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:54 smithi115 bash[45495]: audit 2022-04-22T23:14:54.821508+0000 mon.a (mon.0) 16413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:14:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:14:54.848Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:14:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:54 smithi115 bash[46934]: audit 2022-04-22T23:14:54.821508+0000 mon.a (mon.0) 16413 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:14:56.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:55 smithi121 bash[36902]: cluster 2022-04-22T23:14:55.016854+0000 mgr.y (mgr.34107) 11885 : cluster [DBG] pgmap v12404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:55 smithi115 bash[45495]: cluster 2022-04-22T23:14:55.016854+0000 mgr.y (mgr.34107) 11885 : cluster [DBG] pgmap v12404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:55 smithi115 bash[46934]: cluster 2022-04-22T23:14:55.016854+0000 mgr.y (mgr.34107) 11885 : cluster [DBG] pgmap v12404: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:56 smithi121 bash[36902]: audit 2022-04-22T23:14:56.525703+0000 mon.a (mon.0) 16414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:56 smithi115 bash[45495]: audit 2022-04-22T23:14:56.525703+0000 mon.a (mon.0) 16414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:56 smithi115 bash[46934]: audit 2022-04-22T23:14:56.525703+0000 mon.a (mon.0) 16414 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:14:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:57 smithi121 bash[36902]: cluster 2022-04-22T23:14:57.017945+0000 mgr.y (mgr.34107) 11886 : cluster [DBG] pgmap v12405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:57 smithi115 bash[45495]: cluster 2022-04-22T23:14:57.017945+0000 mgr.y (mgr.34107) 11886 : cluster [DBG] pgmap v12405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:57 smithi115 bash[46934]: cluster 2022-04-22T23:14:57.017945+0000 mgr.y (mgr.34107) 11886 : cluster [DBG] pgmap v12405: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:14:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:58 smithi121 bash[36902]: audit 2022-04-22T23:14:58.154979+0000 mon.a (mon.0) 16415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:58 smithi115 bash[45495]: audit 2022-04-22T23:14:58.154979+0000 mon.a (mon.0) 16415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:14:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:58 smithi115 bash[46934]: audit 2022-04-22T23:14:58.154979+0000 mon.a (mon.0) 16415 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:59 smithi121 bash[36902]: cluster 2022-04-22T23:14:59.019085+0000 mgr.y (mgr.34107) 11887 : cluster [DBG] pgmap v12406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:14:59 smithi121 bash[36902]: audit 2022-04-22T23:14:59.825275+0000 mon.a (mon.0) 16416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:59 smithi115 bash[45495]: cluster 2022-04-22T23:14:59.019085+0000 mgr.y (mgr.34107) 11887 : cluster [DBG] pgmap v12406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:14:59 smithi115 bash[45495]: audit 2022-04-22T23:14:59.825275+0000 mon.a (mon.0) 16416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:59 smithi115 bash[46934]: cluster 2022-04-22T23:14:59.019085+0000 mgr.y (mgr.34107) 11887 : cluster [DBG] pgmap v12406: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:14:59 smithi115 bash[46934]: audit 2022-04-22T23:14:59.825275+0000 mon.a (mon.0) 16416 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:02 smithi121 bash[36902]: cluster 2022-04-22T23:15:01.020129+0000 mgr.y (mgr.34107) 11888 : cluster [DBG] pgmap v12407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:02.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:02 smithi121 bash[36902]: audit 2022-04-22T23:15:01.528819+0000 mon.a (mon.0) 16417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:02 smithi115 bash[45495]: cluster 2022-04-22T23:15:01.020129+0000 mgr.y (mgr.34107) 11888 : cluster [DBG] pgmap v12407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:02 smithi115 bash[45495]: audit 2022-04-22T23:15:01.528819+0000 mon.a (mon.0) 16417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:02 smithi115 bash[46934]: cluster 2022-04-22T23:15:01.020129+0000 mgr.y (mgr.34107) 11888 : cluster [DBG] pgmap v12407: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:02 smithi115 bash[46934]: audit 2022-04-22T23:15:01.528819+0000 mon.a (mon.0) 16417 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:02.994Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:03 smithi121 bash[36902]: cluster 2022-04-22T23:15:03.021069+0000 mgr.y (mgr.34107) 11889 : cluster [DBG] pgmap v12408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:03 smithi121 bash[36902]: audit 2022-04-22T23:15:03.158223+0000 mon.a (mon.0) 16418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:03 smithi115 bash[46934]: cluster 2022-04-22T23:15:03.021069+0000 mgr.y (mgr.34107) 11889 : cluster [DBG] pgmap v12408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:03 smithi115 bash[46934]: audit 2022-04-22T23:15:03.158223+0000 mon.a (mon.0) 16418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:03 smithi115 bash[45495]: cluster 2022-04-22T23:15:03.021069+0000 mgr.y (mgr.34107) 11889 : cluster [DBG] pgmap v12408: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:03 smithi115 bash[45495]: audit 2022-04-22T23:15:03.158223+0000 mon.a (mon.0) 16418 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:04 smithi121 bash[36902]: audit 2022-04-22T23:15:04.828822+0000 mon.a (mon.0) 16419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:05.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:04 smithi115 bash[46934]: audit 2022-04-22T23:15:04.828822+0000 mon.a (mon.0) 16419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:04.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:04 smithi115 bash[45495]: audit 2022-04-22T23:15:04.828822+0000 mon.a (mon.0) 16419 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:05 smithi121 bash[36902]: cluster 2022-04-22T23:15:05.022178+0000 mgr.y (mgr.34107) 11890 : cluster [DBG] pgmap v12409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:05 smithi115 bash[45495]: cluster 2022-04-22T23:15:05.022178+0000 mgr.y (mgr.34107) 11890 : cluster [DBG] pgmap v12409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:05 smithi115 bash[46934]: cluster 2022-04-22T23:15:05.022178+0000 mgr.y (mgr.34107) 11890 : cluster [DBG] pgmap v12409: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:06 smithi121 bash[36902]: audit 2022-04-22T23:15:06.531946+0000 mon.a (mon.0) 16420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:06 smithi115 bash[45495]: audit 2022-04-22T23:15:06.531946+0000 mon.a (mon.0) 16420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:06 smithi115 bash[46934]: audit 2022-04-22T23:15:06.531946+0000 mon.a (mon.0) 16420 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:07 smithi121 bash[36902]: cluster 2022-04-22T23:15:07.023246+0000 mgr.y (mgr.34107) 11891 : cluster [DBG] pgmap v12410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:07 smithi115 bash[45495]: cluster 2022-04-22T23:15:07.023246+0000 mgr.y (mgr.34107) 11891 : cluster [DBG] pgmap v12410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:07 smithi115 bash[46934]: cluster 2022-04-22T23:15:07.023246+0000 mgr.y (mgr.34107) 11891 : cluster [DBG] pgmap v12410: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:09.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:08 smithi121 bash[36902]: audit 2022-04-22T23:15:08.160932+0000 mon.a (mon.0) 16421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:09.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:08 smithi115 bash[45495]: audit 2022-04-22T23:15:08.160932+0000 mon.a (mon.0) 16421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:08 smithi115 bash[46934]: audit 2022-04-22T23:15:08.160932+0000 mon.a (mon.0) 16421 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:09 smithi121 bash[36902]: cluster 2022-04-22T23:15:09.024450+0000 mgr.y (mgr.34107) 11892 : cluster [DBG] pgmap v12411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:09 smithi121 bash[36902]: audit 2022-04-22T23:15:09.832658+0000 mon.a (mon.0) 16422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:10.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:09 smithi115 bash[45495]: cluster 2022-04-22T23:15:09.024450+0000 mgr.y (mgr.34107) 11892 : cluster [DBG] pgmap v12411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:09 smithi115 bash[45495]: audit 2022-04-22T23:15:09.832658+0000 mon.a (mon.0) 16422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:09 smithi115 bash[46934]: cluster 2022-04-22T23:15:09.024450+0000 mgr.y (mgr.34107) 11892 : cluster [DBG] pgmap v12411: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:09 smithi115 bash[46934]: audit 2022-04-22T23:15:09.832658+0000 mon.a (mon.0) 16422 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:12 smithi121 bash[36902]: cluster 2022-04-22T23:15:11.025550+0000 mgr.y (mgr.34107) 11893 : cluster [DBG] pgmap v12412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:12 smithi121 bash[36902]: audit 2022-04-22T23:15:11.534954+0000 mon.a (mon.0) 16423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:12 smithi115 bash[45495]: cluster 2022-04-22T23:15:11.025550+0000 mgr.y (mgr.34107) 11893 : cluster [DBG] pgmap v12412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:12.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:12 smithi115 bash[45495]: audit 2022-04-22T23:15:11.534954+0000 mon.a (mon.0) 16423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:12 smithi115 bash[46934]: cluster 2022-04-22T23:15:11.025550+0000 mgr.y (mgr.34107) 11893 : cluster [DBG] pgmap v12412: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:12 smithi115 bash[46934]: audit 2022-04-22T23:15:11.534954+0000 mon.a (mon.0) 16423 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:12.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:13 smithi121 bash[36902]: cluster 2022-04-22T23:15:13.026343+0000 mgr.y (mgr.34107) 11894 : cluster [DBG] pgmap v12413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:13 smithi121 bash[36902]: audit 2022-04-22T23:15:13.163997+0000 mon.a (mon.0) 16424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:13 smithi115 bash[45495]: cluster 2022-04-22T23:15:13.026343+0000 mgr.y (mgr.34107) 11894 : cluster [DBG] pgmap v12413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:13 smithi115 bash[45495]: audit 2022-04-22T23:15:13.163997+0000 mon.a (mon.0) 16424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:13 smithi115 bash[46934]: cluster 2022-04-22T23:15:13.026343+0000 mgr.y (mgr.34107) 11894 : cluster [DBG] pgmap v12413: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:13 smithi115 bash[46934]: audit 2022-04-22T23:15:13.163997+0000 mon.a (mon.0) 16424 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:14 smithi121 bash[36902]: audit 2022-04-22T23:15:14.836374+0000 mon.a (mon.0) 16425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:14 smithi115 bash[45495]: audit 2022-04-22T23:15:14.836374+0000 mon.a (mon.0) 16425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:14.849Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:14 smithi115 bash[46934]: audit 2022-04-22T23:15:14.836374+0000 mon.a (mon.0) 16425 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:15.825 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:15:16.186 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 8m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 5h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 5h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:15:16.187 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:15:16.188 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:15:16.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:15 smithi121 bash[36902]: cluster 2022-04-22T23:15:15.027398+0000 mgr.y (mgr.34107) 11895 : cluster [DBG] pgmap v12414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:15 smithi115 bash[45495]: cluster 2022-04-22T23:15:15.027398+0000 mgr.y (mgr.34107) 11895 : cluster [DBG] pgmap v12414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:15 smithi115 bash[46934]: cluster 2022-04-22T23:15:15.027398+0000 mgr.y (mgr.34107) 11895 : cluster [DBG] pgmap v12414: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:16.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:15:16.591 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:15:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:16 smithi121 bash[36902]: audit 2022-04-22T23:15:15.805561+0000 mgr.y (mgr.34107) 11896 : audit [DBG] from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:16 smithi121 bash[36902]: audit 2022-04-22T23:15:16.162190+0000 mgr.y (mgr.34107) 11897 : audit [DBG] from='client.57601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:16 smithi121 bash[36902]: audit 2022-04-22T23:15:16.537801+0000 mon.a (mon.0) 16426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:16 smithi121 bash[36902]: audit 2022-04-22T23:15:16.586049+0000 mon.c (mon.1) 401 : audit [DBG] from='client.? 172.21.15.115:0/1207331200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[45495]: audit 2022-04-22T23:15:15.805561+0000 mgr.y (mgr.34107) 11896 : audit [DBG] from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[45495]: audit 2022-04-22T23:15:16.162190+0000 mgr.y (mgr.34107) 11897 : audit [DBG] from='client.57601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[45495]: audit 2022-04-22T23:15:16.537801+0000 mon.a (mon.0) 16426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[45495]: audit 2022-04-22T23:15:16.586049+0000 mon.c (mon.1) 401 : audit [DBG] from='client.? 172.21.15.115:0/1207331200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[46934]: audit 2022-04-22T23:15:15.805561+0000 mgr.y (mgr.34107) 11896 : audit [DBG] from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[46934]: audit 2022-04-22T23:15:16.162190+0000 mgr.y (mgr.34107) 11897 : audit [DBG] from='client.57601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[46934]: audit 2022-04-22T23:15:16.537801+0000 mon.a (mon.0) 16426 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:17.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:16 smithi115 bash[46934]: audit 2022-04-22T23:15:16.586049+0000 mon.c (mon.1) 401 : audit [DBG] from='client.? 172.21.15.115:0/1207331200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:17 smithi121 bash[36902]: cluster 2022-04-22T23:15:17.028410+0000 mgr.y (mgr.34107) 11898 : cluster [DBG] pgmap v12415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:17 smithi121 bash[36902]: audit 2022-04-22T23:15:17.294248+0000 mon.a (mon.0) 16427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:15:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:17 smithi121 bash[36902]: audit 2022-04-22T23:15:17.295366+0000 mon.a (mon.0) 16428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[45495]: cluster 2022-04-22T23:15:17.028410+0000 mgr.y (mgr.34107) 11898 : cluster [DBG] pgmap v12415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[45495]: audit 2022-04-22T23:15:17.294248+0000 mon.a (mon.0) 16427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[45495]: audit 2022-04-22T23:15:17.295366+0000 mon.a (mon.0) 16428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[46934]: cluster 2022-04-22T23:15:17.028410+0000 mgr.y (mgr.34107) 11898 : cluster [DBG] pgmap v12415: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[46934]: audit 2022-04-22T23:15:17.294248+0000 mon.a (mon.0) 16427 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:15:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:17 smithi115 bash[46934]: audit 2022-04-22T23:15:17.295366+0000 mon.a (mon.0) 16428 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:15:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:18 smithi121 bash[36902]: audit 2022-04-22T23:15:18.167091+0000 mon.a (mon.0) 16429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:18 smithi115 bash[45495]: audit 2022-04-22T23:15:18.167091+0000 mon.a (mon.0) 16429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:18 smithi115 bash[46934]: audit 2022-04-22T23:15:18.167091+0000 mon.a (mon.0) 16429 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:19 smithi121 bash[36902]: cluster 2022-04-22T23:15:19.029527+0000 mgr.y (mgr.34107) 11899 : cluster [DBG] pgmap v12416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:19 smithi121 bash[36902]: audit 2022-04-22T23:15:19.840074+0000 mon.a (mon.0) 16430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:19 smithi115 bash[45495]: cluster 2022-04-22T23:15:19.029527+0000 mgr.y (mgr.34107) 11899 : cluster [DBG] pgmap v12416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:19 smithi115 bash[45495]: audit 2022-04-22T23:15:19.840074+0000 mon.a (mon.0) 16430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:19 smithi115 bash[46934]: cluster 2022-04-22T23:15:19.029527+0000 mgr.y (mgr.34107) 11899 : cluster [DBG] pgmap v12416: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:19 smithi115 bash[46934]: audit 2022-04-22T23:15:19.840074+0000 mon.a (mon.0) 16430 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:22 smithi121 bash[36902]: cluster 2022-04-22T23:15:21.030704+0000 mgr.y (mgr.34107) 11900 : cluster [DBG] pgmap v12417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:22 smithi121 bash[36902]: audit 2022-04-22T23:15:21.540626+0000 mon.a (mon.0) 16431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:22 smithi115 bash[45495]: cluster 2022-04-22T23:15:21.030704+0000 mgr.y (mgr.34107) 11900 : cluster [DBG] pgmap v12417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:22 smithi115 bash[45495]: audit 2022-04-22T23:15:21.540626+0000 mon.a (mon.0) 16431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:22 smithi115 bash[46934]: cluster 2022-04-22T23:15:21.030704+0000 mgr.y (mgr.34107) 11900 : cluster [DBG] pgmap v12417: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:22 smithi115 bash[46934]: audit 2022-04-22T23:15:21.540626+0000 mon.a (mon.0) 16431 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:22.995Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:23 smithi121 bash[36902]: cluster 2022-04-22T23:15:23.031863+0000 mgr.y (mgr.34107) 11901 : cluster [DBG] pgmap v12418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:23 smithi121 bash[36902]: audit 2022-04-22T23:15:23.170320+0000 mon.a (mon.0) 16432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:23 smithi115 bash[45495]: cluster 2022-04-22T23:15:23.031863+0000 mgr.y (mgr.34107) 11901 : cluster [DBG] pgmap v12418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:23 smithi115 bash[45495]: audit 2022-04-22T23:15:23.170320+0000 mon.a (mon.0) 16432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:23 smithi115 bash[46934]: cluster 2022-04-22T23:15:23.031863+0000 mgr.y (mgr.34107) 11901 : cluster [DBG] pgmap v12418: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:23 smithi115 bash[46934]: audit 2022-04-22T23:15:23.170320+0000 mon.a (mon.0) 16432 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:24 smithi121 bash[36902]: audit 2022-04-22T23:15:24.843760+0000 mon.a (mon.0) 16433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:24 smithi115 bash[45495]: audit 2022-04-22T23:15:24.843760+0000 mon.a (mon.0) 16433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:24.850Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:24 smithi115 bash[46934]: audit 2022-04-22T23:15:24.843760+0000 mon.a (mon.0) 16433 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:25 smithi121 bash[36902]: cluster 2022-04-22T23:15:25.032895+0000 mgr.y (mgr.34107) 11902 : cluster [DBG] pgmap v12419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:25 smithi115 bash[45495]: cluster 2022-04-22T23:15:25.032895+0000 mgr.y (mgr.34107) 11902 : cluster [DBG] pgmap v12419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:25 smithi115 bash[46934]: cluster 2022-04-22T23:15:25.032895+0000 mgr.y (mgr.34107) 11902 : cluster [DBG] pgmap v12419: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:26 smithi121 bash[36902]: audit 2022-04-22T23:15:26.543693+0000 mon.a (mon.0) 16434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:26 smithi115 bash[45495]: audit 2022-04-22T23:15:26.543693+0000 mon.a (mon.0) 16434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:26 smithi115 bash[46934]: audit 2022-04-22T23:15:26.543693+0000 mon.a (mon.0) 16434 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:28.192 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:27 smithi115 bash[45495]: cluster 2022-04-22T23:15:27.033967+0000 mgr.y (mgr.34107) 11903 : cluster [DBG] pgmap v12420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:28.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:27 smithi115 bash[46934]: cluster 2022-04-22T23:15:27.033967+0000 mgr.y (mgr.34107) 11903 : cluster [DBG] pgmap v12420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:28.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:27 smithi121 bash[36902]: cluster 2022-04-22T23:15:27.033967+0000 mgr.y (mgr.34107) 11903 : cluster [DBG] pgmap v12420: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:28 smithi121 bash[36902]: audit 2022-04-22T23:15:28.167771+0000 mon.a (mon.0) 16435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:15:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:28 smithi121 bash[36902]: audit 2022-04-22T23:15:28.170496+0000 mon.a (mon.0) 16436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:15:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:28 smithi121 bash[36902]: audit 2022-04-22T23:15:28.172231+0000 mon.a (mon.0) 16437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:15:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:28 smithi121 bash[36902]: audit 2022-04-22T23:15:28.173422+0000 mon.a (mon.0) 16438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:28 smithi121 bash[36902]: audit 2022-04-22T23:15:28.747642+0000 mon.a (mon.0) 16439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[45495]: audit 2022-04-22T23:15:28.167771+0000 mon.a (mon.0) 16435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[45495]: audit 2022-04-22T23:15:28.170496+0000 mon.a (mon.0) 16436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[45495]: audit 2022-04-22T23:15:28.172231+0000 mon.a (mon.0) 16437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[45495]: audit 2022-04-22T23:15:28.173422+0000 mon.a (mon.0) 16438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[45495]: audit 2022-04-22T23:15:28.747642+0000 mon.a (mon.0) 16439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[46934]: audit 2022-04-22T23:15:28.167771+0000 mon.a (mon.0) 16435 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:15:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[46934]: audit 2022-04-22T23:15:28.170496+0000 mon.a (mon.0) 16436 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:15:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[46934]: audit 2022-04-22T23:15:28.172231+0000 mon.a (mon.0) 16437 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:15:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[46934]: audit 2022-04-22T23:15:28.173422+0000 mon.a (mon.0) 16438 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:28 smithi115 bash[46934]: audit 2022-04-22T23:15:28.747642+0000 mon.a (mon.0) 16439 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:29 smithi121 bash[36902]: cluster 2022-04-22T23:15:29.035177+0000 mgr.y (mgr.34107) 11904 : cluster [DBG] pgmap v12421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:29 smithi121 bash[36902]: audit 2022-04-22T23:15:29.847415+0000 mon.a (mon.0) 16440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:30.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:29 smithi115 bash[45495]: cluster 2022-04-22T23:15:29.035177+0000 mgr.y (mgr.34107) 11904 : cluster [DBG] pgmap v12421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:29 smithi115 bash[45495]: audit 2022-04-22T23:15:29.847415+0000 mon.a (mon.0) 16440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:29 smithi115 bash[46934]: cluster 2022-04-22T23:15:29.035177+0000 mgr.y (mgr.34107) 11904 : cluster [DBG] pgmap v12421: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:29 smithi115 bash[46934]: audit 2022-04-22T23:15:29.847415+0000 mon.a (mon.0) 16440 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:32 smithi121 bash[36902]: cluster 2022-04-22T23:15:31.036347+0000 mgr.y (mgr.34107) 11905 : cluster [DBG] pgmap v12422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:32 smithi121 bash[36902]: audit 2022-04-22T23:15:31.546554+0000 mon.a (mon.0) 16441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:32 smithi115 bash[45495]: cluster 2022-04-22T23:15:31.036347+0000 mgr.y (mgr.34107) 11905 : cluster [DBG] pgmap v12422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:32 smithi115 bash[45495]: audit 2022-04-22T23:15:31.546554+0000 mon.a (mon.0) 16441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:32 smithi115 bash[46934]: cluster 2022-04-22T23:15:31.036347+0000 mgr.y (mgr.34107) 11905 : cluster [DBG] pgmap v12422: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:32 smithi115 bash[46934]: audit 2022-04-22T23:15:31.546554+0000 mon.a (mon.0) 16441 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:32 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:32.996Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:33 smithi121 bash[36902]: cluster 2022-04-22T23:15:33.037474+0000 mgr.y (mgr.34107) 11906 : cluster [DBG] pgmap v12423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:33 smithi121 bash[36902]: audit 2022-04-22T23:15:33.176788+0000 mon.a (mon.0) 16442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:33 smithi115 bash[45495]: cluster 2022-04-22T23:15:33.037474+0000 mgr.y (mgr.34107) 11906 : cluster [DBG] pgmap v12423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:33 smithi115 bash[45495]: audit 2022-04-22T23:15:33.176788+0000 mon.a (mon.0) 16442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:33 smithi115 bash[46934]: cluster 2022-04-22T23:15:33.037474+0000 mgr.y (mgr.34107) 11906 : cluster [DBG] pgmap v12423: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:33 smithi115 bash[46934]: audit 2022-04-22T23:15:33.176788+0000 mon.a (mon.0) 16442 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:35.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:34 smithi121 bash[36902]: audit 2022-04-22T23:15:34.851025+0000 mon.a (mon.0) 16443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:34 smithi115 bash[45495]: audit 2022-04-22T23:15:34.851025+0000 mon.a (mon.0) 16443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:34.853Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:34 smithi115 bash[46934]: audit 2022-04-22T23:15:34.851025+0000 mon.a (mon.0) 16443 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:35 smithi121 bash[36902]: cluster 2022-04-22T23:15:35.038584+0000 mgr.y (mgr.34107) 11907 : cluster [DBG] pgmap v12424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:35 smithi115 bash[45495]: cluster 2022-04-22T23:15:35.038584+0000 mgr.y (mgr.34107) 11907 : cluster [DBG] pgmap v12424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:35 smithi115 bash[46934]: cluster 2022-04-22T23:15:35.038584+0000 mgr.y (mgr.34107) 11907 : cluster [DBG] pgmap v12424: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:37.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:36 smithi121 bash[36902]: audit 2022-04-22T23:15:36.549677+0000 mon.a (mon.0) 16444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:36 smithi115 bash[45495]: audit 2022-04-22T23:15:36.549677+0000 mon.a (mon.0) 16444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:36 smithi115 bash[46934]: audit 2022-04-22T23:15:36.549677+0000 mon.a (mon.0) 16444 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:37 smithi121 bash[36902]: cluster 2022-04-22T23:15:37.039707+0000 mgr.y (mgr.34107) 11908 : cluster [DBG] pgmap v12425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:37 smithi121 bash[36902]: audit 2022-04-22T23:15:37.422172+0000 mon.a (mon.0) 16445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:37 smithi121 bash[36902]: audit 2022-04-22T23:15:37.428615+0000 mon.a (mon.0) 16446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[45495]: cluster 2022-04-22T23:15:37.039707+0000 mgr.y (mgr.34107) 11908 : cluster [DBG] pgmap v12425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[45495]: audit 2022-04-22T23:15:37.422172+0000 mon.a (mon.0) 16445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[45495]: audit 2022-04-22T23:15:37.428615+0000 mon.a (mon.0) 16446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[46934]: cluster 2022-04-22T23:15:37.039707+0000 mgr.y (mgr.34107) 11908 : cluster [DBG] pgmap v12425: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[46934]: audit 2022-04-22T23:15:37.422172+0000 mon.a (mon.0) 16445 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:37 smithi115 bash[46934]: audit 2022-04-22T23:15:37.428615+0000 mon.a (mon.0) 16446 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:15:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:38 smithi121 bash[36902]: cephadm 2022-04-22T23:15:37.410161+0000 mgr.y (mgr.34107) 11909 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T23:15:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:38 smithi121 bash[36902]: audit 2022-04-22T23:15:38.179894+0000 mon.a (mon.0) 16447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:38 smithi115 bash[45495]: cephadm 2022-04-22T23:15:37.410161+0000 mgr.y (mgr.34107) 11909 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T23:15:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:38 smithi115 bash[45495]: audit 2022-04-22T23:15:38.179894+0000 mon.a (mon.0) 16447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:38 smithi115 bash[46934]: cephadm 2022-04-22T23:15:37.410161+0000 mgr.y (mgr.34107) 11909 : cephadm [INF] Detected new or changed devices on smithi121 2022-04-22T23:15:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:38 smithi115 bash[46934]: audit 2022-04-22T23:15:38.179894+0000 mon.a (mon.0) 16447 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:39 smithi121 bash[36902]: cluster 2022-04-22T23:15:39.040977+0000 mgr.y (mgr.34107) 11910 : cluster [DBG] pgmap v12426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:39 smithi121 bash[36902]: audit 2022-04-22T23:15:39.854022+0000 mon.a (mon.0) 16448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:39 smithi115 bash[45495]: cluster 2022-04-22T23:15:39.040977+0000 mgr.y (mgr.34107) 11910 : cluster [DBG] pgmap v12426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:39 smithi115 bash[45495]: audit 2022-04-22T23:15:39.854022+0000 mon.a (mon.0) 16448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:39 smithi115 bash[46934]: cluster 2022-04-22T23:15:39.040977+0000 mgr.y (mgr.34107) 11910 : cluster [DBG] pgmap v12426: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:39 smithi115 bash[46934]: audit 2022-04-22T23:15:39.854022+0000 mon.a (mon.0) 16448 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:42 smithi121 bash[36902]: cluster 2022-04-22T23:15:41.042201+0000 mgr.y (mgr.34107) 11911 : cluster [DBG] pgmap v12427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:42 smithi121 bash[36902]: audit 2022-04-22T23:15:41.552758+0000 mon.a (mon.0) 16449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:42.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:42 smithi115 bash[45495]: cluster 2022-04-22T23:15:41.042201+0000 mgr.y (mgr.34107) 11911 : cluster [DBG] pgmap v12427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:42 smithi115 bash[45495]: audit 2022-04-22T23:15:41.552758+0000 mon.a (mon.0) 16449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:42 smithi115 bash[46934]: cluster 2022-04-22T23:15:41.042201+0000 mgr.y (mgr.34107) 11911 : cluster [DBG] pgmap v12427: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:42 smithi115 bash[46934]: audit 2022-04-22T23:15:41.552758+0000 mon.a (mon.0) 16449 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:42 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:42.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:43 smithi121 bash[36902]: cluster 2022-04-22T23:15:43.043295+0000 mgr.y (mgr.34107) 11912 : cluster [DBG] pgmap v12428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:43 smithi121 bash[36902]: audit 2022-04-22T23:15:43.182986+0000 mon.a (mon.0) 16450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:43.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:43 smithi115 bash[45495]: cluster 2022-04-22T23:15:43.043295+0000 mgr.y (mgr.34107) 11912 : cluster [DBG] pgmap v12428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:43 smithi115 bash[45495]: audit 2022-04-22T23:15:43.182986+0000 mon.a (mon.0) 16450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:43 smithi115 bash[46934]: cluster 2022-04-22T23:15:43.043295+0000 mgr.y (mgr.34107) 11912 : cluster [DBG] pgmap v12428: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:43 smithi115 bash[46934]: audit 2022-04-22T23:15:43.182986+0000 mon.a (mon.0) 16450 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:44 smithi121 bash[36902]: audit 2022-04-22T23:15:44.857900+0000 mon.a (mon.0) 16451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:45.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:44 smithi115 bash[45495]: audit 2022-04-22T23:15:44.857900+0000 mon.a (mon.0) 16451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:44 smithi115 bash[46934]: audit 2022-04-22T23:15:44.857900+0000 mon.a (mon.0) 16451 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:44.852Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:45 smithi121 bash[36902]: cluster 2022-04-22T23:15:45.044449+0000 mgr.y (mgr.34107) 11913 : cluster [DBG] pgmap v12429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:45 smithi115 bash[45495]: cluster 2022-04-22T23:15:45.044449+0000 mgr.y (mgr.34107) 11913 : cluster [DBG] pgmap v12429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:45 smithi115 bash[46934]: cluster 2022-04-22T23:15:45.044449+0000 mgr.y (mgr.34107) 11913 : cluster [DBG] pgmap v12429: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:46.966 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:15:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:46 smithi121 bash[36902]: audit 2022-04-22T23:15:46.555855+0000 mon.a (mon.0) 16452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:46 smithi115 bash[45495]: audit 2022-04-22T23:15:46.555855+0000 mon.a (mon.0) 16452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:46 smithi115 bash[46934]: audit 2022-04-22T23:15:46.555855+0000 mon.a (mon.0) 16452 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 6h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:15:47.304 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:15:47.305 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:15:47.306 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:15:47.713 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:15:47.713 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:15:47.713 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:15:47.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:47.713 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:15:47.714 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:15:47.715 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:15:47.715 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:15:47.715 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:15:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:47 smithi121 bash[36902]: audit 2022-04-22T23:15:46.947020+0000 mgr.y (mgr.34107) 11914 : audit [DBG] from='client.57613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:47 smithi121 bash[36902]: cluster 2022-04-22T23:15:47.045495+0000 mgr.y (mgr.34107) 11915 : cluster [DBG] pgmap v12430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:47 smithi121 bash[36902]: audit 2022-04-22T23:15:47.709591+0000 mon.c (mon.1) 402 : audit [DBG] from='client.? 172.21.15.115:0/2114202662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[45495]: audit 2022-04-22T23:15:46.947020+0000 mgr.y (mgr.34107) 11914 : audit [DBG] from='client.57613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[45495]: cluster 2022-04-22T23:15:47.045495+0000 mgr.y (mgr.34107) 11915 : cluster [DBG] pgmap v12430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[45495]: audit 2022-04-22T23:15:47.709591+0000 mon.c (mon.1) 402 : audit [DBG] from='client.? 172.21.15.115:0/2114202662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[46934]: audit 2022-04-22T23:15:46.947020+0000 mgr.y (mgr.34107) 11914 : audit [DBG] from='client.57613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[46934]: cluster 2022-04-22T23:15:47.045495+0000 mgr.y (mgr.34107) 11915 : cluster [DBG] pgmap v12430: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:47 smithi115 bash[46934]: audit 2022-04-22T23:15:47.709591+0000 mon.c (mon.1) 402 : audit [DBG] from='client.? 172.21.15.115:0/2114202662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:15:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:48 smithi121 bash[36902]: audit 2022-04-22T23:15:47.281892+0000 mgr.y (mgr.34107) 11916 : audit [DBG] from='client.57619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:48 smithi121 bash[36902]: audit 2022-04-22T23:15:48.186057+0000 mon.a (mon.0) 16453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:48 smithi115 bash[45495]: audit 2022-04-22T23:15:47.281892+0000 mgr.y (mgr.34107) 11916 : audit [DBG] from='client.57619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:48 smithi115 bash[45495]: audit 2022-04-22T23:15:48.186057+0000 mon.a (mon.0) 16453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:48 smithi115 bash[46934]: audit 2022-04-22T23:15:47.281892+0000 mgr.y (mgr.34107) 11916 : audit [DBG] from='client.57619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:15:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:48 smithi115 bash[46934]: audit 2022-04-22T23:15:48.186057+0000 mon.a (mon.0) 16453 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:49 smithi121 bash[36902]: cluster 2022-04-22T23:15:49.047022+0000 mgr.y (mgr.34107) 11917 : cluster [DBG] pgmap v12431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:50.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:49 smithi121 bash[36902]: audit 2022-04-22T23:15:49.861686+0000 mon.a (mon.0) 16454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:50.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:49 smithi115 bash[45495]: cluster 2022-04-22T23:15:49.047022+0000 mgr.y (mgr.34107) 11917 : cluster [DBG] pgmap v12431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:49 smithi115 bash[45495]: audit 2022-04-22T23:15:49.861686+0000 mon.a (mon.0) 16454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:49 smithi115 bash[46934]: cluster 2022-04-22T23:15:49.047022+0000 mgr.y (mgr.34107) 11917 : cluster [DBG] pgmap v12431: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:49 smithi115 bash[46934]: audit 2022-04-22T23:15:49.861686+0000 mon.a (mon.0) 16454 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:52 smithi121 bash[36902]: cluster 2022-04-22T23:15:51.048009+0000 mgr.y (mgr.34107) 11918 : cluster [DBG] pgmap v12432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:52 smithi121 bash[36902]: audit 2022-04-22T23:15:51.558942+0000 mon.a (mon.0) 16455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:52 smithi115 bash[45495]: cluster 2022-04-22T23:15:51.048009+0000 mgr.y (mgr.34107) 11918 : cluster [DBG] pgmap v12432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:52 smithi115 bash[45495]: audit 2022-04-22T23:15:51.558942+0000 mon.a (mon.0) 16455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:52 smithi115 bash[46934]: cluster 2022-04-22T23:15:51.048009+0000 mgr.y (mgr.34107) 11918 : cluster [DBG] pgmap v12432: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:52 smithi115 bash[46934]: audit 2022-04-22T23:15:51.558942+0000 mon.a (mon.0) 16455 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:52.996Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:53 smithi115 bash[45495]: cluster 2022-04-22T23:15:53.049006+0000 mgr.y (mgr.34107) 11919 : cluster [DBG] pgmap v12433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:53 smithi115 bash[45495]: audit 2022-04-22T23:15:53.188945+0000 mon.a (mon.0) 16456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:53 smithi115 bash[46934]: cluster 2022-04-22T23:15:53.049006+0000 mgr.y (mgr.34107) 11919 : cluster [DBG] pgmap v12433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:53 smithi115 bash[46934]: audit 2022-04-22T23:15:53.188945+0000 mon.a (mon.0) 16456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:53 smithi121 bash[36902]: cluster 2022-04-22T23:15:53.049006+0000 mgr.y (mgr.34107) 11919 : cluster [DBG] pgmap v12433: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:53 smithi121 bash[36902]: audit 2022-04-22T23:15:53.188945+0000 mon.a (mon.0) 16456 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:54 smithi121 bash[36902]: audit 2022-04-22T23:15:54.865570+0000 mon.a (mon.0) 16457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:54 smithi115 bash[45495]: audit 2022-04-22T23:15:54.865570+0000 mon.a (mon.0) 16457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:15:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:15:54.852Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:15:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:54 smithi115 bash[46934]: audit 2022-04-22T23:15:54.865570+0000 mon.a (mon.0) 16457 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:15:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:55 smithi121 bash[36902]: cluster 2022-04-22T23:15:55.050001+0000 mgr.y (mgr.34107) 11920 : cluster [DBG] pgmap v12434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:55 smithi115 bash[45495]: cluster 2022-04-22T23:15:55.050001+0000 mgr.y (mgr.34107) 11920 : cluster [DBG] pgmap v12434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:55 smithi115 bash[46934]: cluster 2022-04-22T23:15:55.050001+0000 mgr.y (mgr.34107) 11920 : cluster [DBG] pgmap v12434: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:56 smithi121 bash[36902]: audit 2022-04-22T23:15:56.562108+0000 mon.a (mon.0) 16458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:56 smithi115 bash[45495]: audit 2022-04-22T23:15:56.562108+0000 mon.a (mon.0) 16458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:56 smithi115 bash[46934]: audit 2022-04-22T23:15:56.562108+0000 mon.a (mon.0) 16458 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:15:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:57 smithi121 bash[36902]: cluster 2022-04-22T23:15:57.051125+0000 mgr.y (mgr.34107) 11921 : cluster [DBG] pgmap v12435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:57 smithi115 bash[45495]: cluster 2022-04-22T23:15:57.051125+0000 mgr.y (mgr.34107) 11921 : cluster [DBG] pgmap v12435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:57 smithi115 bash[46934]: cluster 2022-04-22T23:15:57.051125+0000 mgr.y (mgr.34107) 11921 : cluster [DBG] pgmap v12435: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:15:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:58 smithi121 bash[36902]: audit 2022-04-22T23:15:58.191743+0000 mon.a (mon.0) 16459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:59.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:58 smithi115 bash[45495]: audit 2022-04-22T23:15:58.191743+0000 mon.a (mon.0) 16459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:15:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:58 smithi115 bash[46934]: audit 2022-04-22T23:15:58.191743+0000 mon.a (mon.0) 16459 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:59 smithi121 bash[36902]: cluster 2022-04-22T23:15:59.052371+0000 mgr.y (mgr.34107) 11922 : cluster [DBG] pgmap v12436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:15:59 smithi121 bash[36902]: audit 2022-04-22T23:15:59.869303+0000 mon.a (mon.0) 16460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:00.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:59 smithi115 bash[45495]: cluster 2022-04-22T23:15:59.052371+0000 mgr.y (mgr.34107) 11922 : cluster [DBG] pgmap v12436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:15:59 smithi115 bash[45495]: audit 2022-04-22T23:15:59.869303+0000 mon.a (mon.0) 16460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:59 smithi115 bash[46934]: cluster 2022-04-22T23:15:59.052371+0000 mgr.y (mgr.34107) 11922 : cluster [DBG] pgmap v12436: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:15:59 smithi115 bash[46934]: audit 2022-04-22T23:15:59.869303+0000 mon.a (mon.0) 16460 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:02 smithi121 bash[36902]: cluster 2022-04-22T23:16:01.053477+0000 mgr.y (mgr.34107) 11923 : cluster [DBG] pgmap v12437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:02.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:02 smithi121 bash[36902]: audit 2022-04-22T23:16:01.565149+0000 mon.a (mon.0) 16461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:02 smithi115 bash[45495]: cluster 2022-04-22T23:16:01.053477+0000 mgr.y (mgr.34107) 11923 : cluster [DBG] pgmap v12437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:02 smithi115 bash[45495]: audit 2022-04-22T23:16:01.565149+0000 mon.a (mon.0) 16461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:02 smithi115 bash[46934]: cluster 2022-04-22T23:16:01.053477+0000 mgr.y (mgr.34107) 11923 : cluster [DBG] pgmap v12437: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:02 smithi115 bash[46934]: audit 2022-04-22T23:16:01.565149+0000 mon.a (mon.0) 16461 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:02 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:02.997Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:03 smithi115 bash[45495]: cluster 2022-04-22T23:16:03.054469+0000 mgr.y (mgr.34107) 11924 : cluster [DBG] pgmap v12438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:03 smithi115 bash[45495]: audit 2022-04-22T23:16:03.194328+0000 mon.a (mon.0) 16462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:03 smithi115 bash[46934]: cluster 2022-04-22T23:16:03.054469+0000 mgr.y (mgr.34107) 11924 : cluster [DBG] pgmap v12438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:03 smithi115 bash[46934]: audit 2022-04-22T23:16:03.194328+0000 mon.a (mon.0) 16462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:03 smithi121 bash[36902]: cluster 2022-04-22T23:16:03.054469+0000 mgr.y (mgr.34107) 11924 : cluster [DBG] pgmap v12438: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:03 smithi121 bash[36902]: audit 2022-04-22T23:16:03.194328+0000 mon.a (mon.0) 16462 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:04 smithi121 bash[36902]: audit 2022-04-22T23:16:04.873161+0000 mon.a (mon.0) 16463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:04 smithi115 bash[46934]: audit 2022-04-22T23:16:04.873161+0000 mon.a (mon.0) 16463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:05.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:04.851Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:05.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:04 smithi115 bash[45495]: audit 2022-04-22T23:16:04.873161+0000 mon.a (mon.0) 16463 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:05 smithi121 bash[36902]: cluster 2022-04-22T23:16:05.055439+0000 mgr.y (mgr.34107) 11925 : cluster [DBG] pgmap v12439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:06.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:05 smithi115 bash[45495]: cluster 2022-04-22T23:16:05.055439+0000 mgr.y (mgr.34107) 11925 : cluster [DBG] pgmap v12439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:05 smithi115 bash[46934]: cluster 2022-04-22T23:16:05.055439+0000 mgr.y (mgr.34107) 11925 : cluster [DBG] pgmap v12439: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:06 smithi121 bash[36902]: audit 2022-04-22T23:16:06.568330+0000 mon.a (mon.0) 16464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:06 smithi115 bash[45495]: audit 2022-04-22T23:16:06.568330+0000 mon.a (mon.0) 16464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:06 smithi115 bash[46934]: audit 2022-04-22T23:16:06.568330+0000 mon.a (mon.0) 16464 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:07 smithi121 bash[36902]: cluster 2022-04-22T23:16:07.056530+0000 mgr.y (mgr.34107) 11926 : cluster [DBG] pgmap v12440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:08.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:07 smithi115 bash[45495]: cluster 2022-04-22T23:16:07.056530+0000 mgr.y (mgr.34107) 11926 : cluster [DBG] pgmap v12440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:07 smithi115 bash[46934]: cluster 2022-04-22T23:16:07.056530+0000 mgr.y (mgr.34107) 11926 : cluster [DBG] pgmap v12440: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:08 smithi121 bash[36902]: audit 2022-04-22T23:16:08.197830+0000 mon.a (mon.0) 16465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:08 smithi115 bash[45495]: audit 2022-04-22T23:16:08.197830+0000 mon.a (mon.0) 16465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:08 smithi115 bash[46934]: audit 2022-04-22T23:16:08.197830+0000 mon.a (mon.0) 16465 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:09 smithi121 bash[36902]: cluster 2022-04-22T23:16:09.057635+0000 mgr.y (mgr.34107) 11927 : cluster [DBG] pgmap v12441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:09 smithi121 bash[36902]: audit 2022-04-22T23:16:09.876604+0000 mon.a (mon.0) 16466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:09 smithi115 bash[46934]: cluster 2022-04-22T23:16:09.057635+0000 mgr.y (mgr.34107) 11927 : cluster [DBG] pgmap v12441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:09 smithi115 bash[46934]: audit 2022-04-22T23:16:09.876604+0000 mon.a (mon.0) 16466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:09 smithi115 bash[45495]: cluster 2022-04-22T23:16:09.057635+0000 mgr.y (mgr.34107) 11927 : cluster [DBG] pgmap v12441: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:09 smithi115 bash[45495]: audit 2022-04-22T23:16:09.876604+0000 mon.a (mon.0) 16466 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:12 smithi121 bash[36902]: cluster 2022-04-22T23:16:11.058724+0000 mgr.y (mgr.34107) 11928 : cluster [DBG] pgmap v12442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:12 smithi121 bash[36902]: audit 2022-04-22T23:16:11.571287+0000 mon.a (mon.0) 16467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:12 smithi115 bash[45495]: cluster 2022-04-22T23:16:11.058724+0000 mgr.y (mgr.34107) 11928 : cluster [DBG] pgmap v12442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:12 smithi115 bash[45495]: audit 2022-04-22T23:16:11.571287+0000 mon.a (mon.0) 16467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:12 smithi115 bash[46934]: cluster 2022-04-22T23:16:11.058724+0000 mgr.y (mgr.34107) 11928 : cluster [DBG] pgmap v12442: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:12 smithi115 bash[46934]: audit 2022-04-22T23:16:11.571287+0000 mon.a (mon.0) 16467 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:12 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:12.998Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:13 smithi115 bash[45495]: cluster 2022-04-22T23:16:13.059566+0000 mgr.y (mgr.34107) 11929 : cluster [DBG] pgmap v12443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:13 smithi115 bash[45495]: audit 2022-04-22T23:16:13.200779+0000 mon.a (mon.0) 16468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:13 smithi115 bash[46934]: cluster 2022-04-22T23:16:13.059566+0000 mgr.y (mgr.34107) 11929 : cluster [DBG] pgmap v12443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:13 smithi115 bash[46934]: audit 2022-04-22T23:16:13.200779+0000 mon.a (mon.0) 16468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:13 smithi121 bash[36902]: cluster 2022-04-22T23:16:13.059566+0000 mgr.y (mgr.34107) 11929 : cluster [DBG] pgmap v12443: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:13 smithi121 bash[36902]: audit 2022-04-22T23:16:13.200779+0000 mon.a (mon.0) 16468 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:14 smithi121 bash[36902]: audit 2022-04-22T23:16:14.880262+0000 mon.a (mon.0) 16469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:14 smithi115 bash[45495]: audit 2022-04-22T23:16:14.880262+0000 mon.a (mon.0) 16469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:14.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:14 smithi115 bash[46934]: audit 2022-04-22T23:16:14.880262+0000 mon.a (mon.0) 16469 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:15 smithi121 bash[36902]: cluster 2022-04-22T23:16:15.060530+0000 mgr.y (mgr.34107) 11930 : cluster [DBG] pgmap v12444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:15 smithi115 bash[45495]: cluster 2022-04-22T23:16:15.060530+0000 mgr.y (mgr.34107) 11930 : cluster [DBG] pgmap v12444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:15 smithi115 bash[46934]: cluster 2022-04-22T23:16:15.060530+0000 mgr.y (mgr.34107) 11930 : cluster [DBG] pgmap v12444: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:16 smithi121 bash[36902]: audit 2022-04-22T23:16:16.574327+0000 mon.a (mon.0) 16470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:16 smithi115 bash[45495]: audit 2022-04-22T23:16:16.574327+0000 mon.a (mon.0) 16470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:16 smithi115 bash[46934]: audit 2022-04-22T23:16:16.574327+0000 mon.a (mon.0) 16470 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:18.096 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:16:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:17 smithi121 bash[36902]: cluster 2022-04-22T23:16:17.061580+0000 mgr.y (mgr.34107) 11931 : cluster [DBG] pgmap v12445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:17 smithi121 bash[36902]: audit 2022-04-22T23:16:17.305557+0000 mon.a (mon.0) 16471 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:16:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:17 smithi121 bash[36902]: audit 2022-04-22T23:16:17.306101+0000 mon.a (mon.0) 16472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:16:18.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[45495]: cluster 2022-04-22T23:16:17.061580+0000 mgr.y (mgr.34107) 11931 : cluster [DBG] pgmap v12445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[45495]: audit 2022-04-22T23:16:17.305557+0000 mon.a (mon.0) 16471 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:16:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[45495]: audit 2022-04-22T23:16:17.306101+0000 mon.a (mon.0) 16472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:16:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[46934]: cluster 2022-04-22T23:16:17.061580+0000 mgr.y (mgr.34107) 11931 : cluster [DBG] pgmap v12445: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[46934]: audit 2022-04-22T23:16:17.305557+0000 mon.a (mon.0) 16471 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:16:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:17 smithi115 bash[46934]: audit 2022-04-22T23:16:17.306101+0000 mon.a (mon.0) 16472 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:16:18.443 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 9m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 6h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 5h 307M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:16:18.444 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 5h 10.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 5h 589M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 5h 508M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:16:18.445 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 5h 653M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:16:18.446 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 5h 64.7M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:18.851 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:16:18.852 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:16:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:18 smithi121 bash[36902]: audit 2022-04-22T23:16:18.075700+0000 mgr.y (mgr.34107) 11932 : audit [DBG] from='client.47691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:18 smithi121 bash[36902]: audit 2022-04-22T23:16:18.203698+0000 mon.a (mon.0) 16473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:18 smithi121 bash[36902]: audit 2022-04-22T23:16:18.847081+0000 mon.c (mon.1) 403 : audit [DBG] from='client.? 172.21.15.115:0/900585605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[45495]: audit 2022-04-22T23:16:18.075700+0000 mgr.y (mgr.34107) 11932 : audit [DBG] from='client.47691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[45495]: audit 2022-04-22T23:16:18.203698+0000 mon.a (mon.0) 16473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[45495]: audit 2022-04-22T23:16:18.847081+0000 mon.c (mon.1) 403 : audit [DBG] from='client.? 172.21.15.115:0/900585605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[46934]: audit 2022-04-22T23:16:18.075700+0000 mgr.y (mgr.34107) 11932 : audit [DBG] from='client.47691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:19.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[46934]: audit 2022-04-22T23:16:18.203698+0000 mon.a (mon.0) 16473 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:19.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:18 smithi115 bash[46934]: audit 2022-04-22T23:16:18.847081+0000 mon.c (mon.1) 403 : audit [DBG] from='client.? 172.21.15.115:0/900585605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:19 smithi121 bash[36902]: audit 2022-04-22T23:16:18.418999+0000 mgr.y (mgr.34107) 11933 : audit [DBG] from='client.47697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:19 smithi121 bash[36902]: cluster 2022-04-22T23:16:19.062712+0000 mgr.y (mgr.34107) 11934 : cluster [DBG] pgmap v12446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:19 smithi121 bash[36902]: audit 2022-04-22T23:16:19.884029+0000 mon.a (mon.0) 16474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[46934]: audit 2022-04-22T23:16:18.418999+0000 mgr.y (mgr.34107) 11933 : audit [DBG] from='client.47697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[46934]: cluster 2022-04-22T23:16:19.062712+0000 mgr.y (mgr.34107) 11934 : cluster [DBG] pgmap v12446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[46934]: audit 2022-04-22T23:16:19.884029+0000 mon.a (mon.0) 16474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[45495]: audit 2022-04-22T23:16:18.418999+0000 mgr.y (mgr.34107) 11933 : audit [DBG] from='client.47697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[45495]: cluster 2022-04-22T23:16:19.062712+0000 mgr.y (mgr.34107) 11934 : cluster [DBG] pgmap v12446: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:19 smithi115 bash[45495]: audit 2022-04-22T23:16:19.884029+0000 mon.a (mon.0) 16474 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:22 smithi121 bash[36902]: cluster 2022-04-22T23:16:21.063713+0000 mgr.y (mgr.34107) 11935 : cluster [DBG] pgmap v12447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:22 smithi121 bash[36902]: audit 2022-04-22T23:16:21.577230+0000 mon.a (mon.0) 16475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:22 smithi115 bash[45495]: cluster 2022-04-22T23:16:21.063713+0000 mgr.y (mgr.34107) 11935 : cluster [DBG] pgmap v12447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:22 smithi115 bash[45495]: audit 2022-04-22T23:16:21.577230+0000 mon.a (mon.0) 16475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:22 smithi115 bash[46934]: cluster 2022-04-22T23:16:21.063713+0000 mgr.y (mgr.34107) 11935 : cluster [DBG] pgmap v12447: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:22 smithi115 bash[46934]: audit 2022-04-22T23:16:21.577230+0000 mon.a (mon.0) 16475 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:22 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:22.998Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:23 smithi121 bash[36902]: cluster 2022-04-22T23:16:23.064734+0000 mgr.y (mgr.34107) 11936 : cluster [DBG] pgmap v12448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:23 smithi121 bash[36902]: audit 2022-04-22T23:16:23.207291+0000 mon.a (mon.0) 16476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:23 smithi115 bash[45495]: cluster 2022-04-22T23:16:23.064734+0000 mgr.y (mgr.34107) 11936 : cluster [DBG] pgmap v12448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:23 smithi115 bash[45495]: audit 2022-04-22T23:16:23.207291+0000 mon.a (mon.0) 16476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:23 smithi115 bash[46934]: cluster 2022-04-22T23:16:23.064734+0000 mgr.y (mgr.34107) 11936 : cluster [DBG] pgmap v12448: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:23 smithi115 bash[46934]: audit 2022-04-22T23:16:23.207291+0000 mon.a (mon.0) 16476 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:24 smithi121 bash[36902]: audit 2022-04-22T23:16:24.887821+0000 mon.a (mon.0) 16477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:25.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:24 smithi115 bash[45495]: audit 2022-04-22T23:16:24.887821+0000 mon.a (mon.0) 16477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:24.854Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:24 smithi115 bash[46934]: audit 2022-04-22T23:16:24.887821+0000 mon.a (mon.0) 16477 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:25 smithi121 bash[36902]: cluster 2022-04-22T23:16:25.065748+0000 mgr.y (mgr.34107) 11937 : cluster [DBG] pgmap v12449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:25 smithi115 bash[45495]: cluster 2022-04-22T23:16:25.065748+0000 mgr.y (mgr.34107) 11937 : cluster [DBG] pgmap v12449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:25 smithi115 bash[46934]: cluster 2022-04-22T23:16:25.065748+0000 mgr.y (mgr.34107) 11937 : cluster [DBG] pgmap v12449: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:26 smithi121 bash[36902]: audit 2022-04-22T23:16:26.580678+0000 mon.a (mon.0) 16478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:26 smithi115 bash[45495]: audit 2022-04-22T23:16:26.580678+0000 mon.a (mon.0) 16478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:26 smithi115 bash[46934]: audit 2022-04-22T23:16:26.580678+0000 mon.a (mon.0) 16478 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:27 smithi121 bash[36902]: cluster 2022-04-22T23:16:27.066897+0000 mgr.y (mgr.34107) 11938 : cluster [DBG] pgmap v12450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:27 smithi115 bash[46934]: cluster 2022-04-22T23:16:27.066897+0000 mgr.y (mgr.34107) 11938 : cluster [DBG] pgmap v12450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:27 smithi115 bash[45495]: cluster 2022-04-22T23:16:27.066897+0000 mgr.y (mgr.34107) 11938 : cluster [DBG] pgmap v12450: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:28 smithi121 bash[36902]: audit 2022-04-22T23:16:28.210599+0000 mon.a (mon.0) 16479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:28 smithi115 bash[45495]: audit 2022-04-22T23:16:28.210599+0000 mon.a (mon.0) 16479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:28 smithi115 bash[46934]: audit 2022-04-22T23:16:28.210599+0000 mon.a (mon.0) 16479 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:29 smithi121 bash[36902]: cluster 2022-04-22T23:16:29.068026+0000 mgr.y (mgr.34107) 11939 : cluster [DBG] pgmap v12451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:30.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:29 smithi121 bash[36902]: audit 2022-04-22T23:16:29.891324+0000 mon.a (mon.0) 16480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:30.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:29 smithi115 bash[46934]: cluster 2022-04-22T23:16:29.068026+0000 mgr.y (mgr.34107) 11939 : cluster [DBG] pgmap v12451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:29 smithi115 bash[46934]: audit 2022-04-22T23:16:29.891324+0000 mon.a (mon.0) 16480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:29 smithi115 bash[45495]: cluster 2022-04-22T23:16:29.068026+0000 mgr.y (mgr.34107) 11939 : cluster [DBG] pgmap v12451: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:29 smithi115 bash[45495]: audit 2022-04-22T23:16:29.891324+0000 mon.a (mon.0) 16480 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:32 smithi121 bash[36902]: cluster 2022-04-22T23:16:31.069105+0000 mgr.y (mgr.34107) 11940 : cluster [DBG] pgmap v12452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:32 smithi121 bash[36902]: audit 2022-04-22T23:16:31.583588+0000 mon.a (mon.0) 16481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:32.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:32 smithi115 bash[46934]: cluster 2022-04-22T23:16:31.069105+0000 mgr.y (mgr.34107) 11940 : cluster [DBG] pgmap v12452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:32 smithi115 bash[46934]: audit 2022-04-22T23:16:31.583588+0000 mon.a (mon.0) 16481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:32 smithi115 bash[45495]: cluster 2022-04-22T23:16:31.069105+0000 mgr.y (mgr.34107) 11940 : cluster [DBG] pgmap v12452: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:32 smithi115 bash[45495]: audit 2022-04-22T23:16:31.583588+0000 mon.a (mon.0) 16481 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:32.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:33 smithi115 bash[45495]: cluster 2022-04-22T23:16:33.070215+0000 mgr.y (mgr.34107) 11941 : cluster [DBG] pgmap v12453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:33 smithi115 bash[45495]: audit 2022-04-22T23:16:33.213533+0000 mon.a (mon.0) 16482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:33 smithi115 bash[46934]: cluster 2022-04-22T23:16:33.070215+0000 mgr.y (mgr.34107) 11941 : cluster [DBG] pgmap v12453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:33 smithi115 bash[46934]: audit 2022-04-22T23:16:33.213533+0000 mon.a (mon.0) 16482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:33 smithi121 bash[36902]: cluster 2022-04-22T23:16:33.070215+0000 mgr.y (mgr.34107) 11941 : cluster [DBG] pgmap v12453: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:33 smithi121 bash[36902]: audit 2022-04-22T23:16:33.213533+0000 mon.a (mon.0) 16482 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:34 smithi121 bash[36902]: audit 2022-04-22T23:16:34.895109+0000 mon.a (mon.0) 16483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:34.856Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:34 smithi115 bash[45495]: audit 2022-04-22T23:16:34.895109+0000 mon.a (mon.0) 16483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:34 smithi115 bash[46934]: audit 2022-04-22T23:16:34.895109+0000 mon.a (mon.0) 16483 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:35 smithi121 bash[36902]: cluster 2022-04-22T23:16:35.071350+0000 mgr.y (mgr.34107) 11942 : cluster [DBG] pgmap v12454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:35 smithi115 bash[46934]: cluster 2022-04-22T23:16:35.071350+0000 mgr.y (mgr.34107) 11942 : cluster [DBG] pgmap v12454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:35 smithi115 bash[45495]: cluster 2022-04-22T23:16:35.071350+0000 mgr.y (mgr.34107) 11942 : cluster [DBG] pgmap v12454: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:36 smithi121 bash[36902]: audit 2022-04-22T23:16:36.586568+0000 mon.a (mon.0) 16484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:36 smithi115 bash[45495]: audit 2022-04-22T23:16:36.586568+0000 mon.a (mon.0) 16484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:36 smithi115 bash[46934]: audit 2022-04-22T23:16:36.586568+0000 mon.a (mon.0) 16484 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:38.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[46934]: cluster 2022-04-22T23:16:37.072467+0000 mgr.y (mgr.34107) 11943 : cluster [DBG] pgmap v12455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:38.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[46934]: audit 2022-04-22T23:16:37.438116+0000 mon.a (mon.0) 16485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:16:38.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[46934]: audit 2022-04-22T23:16:37.441147+0000 mon.a (mon.0) 16486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:16:38.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[46934]: audit 2022-04-22T23:16:37.442924+0000 mon.a (mon.0) 16487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:16:38.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[45495]: cluster 2022-04-22T23:16:37.072467+0000 mgr.y (mgr.34107) 11943 : cluster [DBG] pgmap v12455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:38.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[45495]: audit 2022-04-22T23:16:37.438116+0000 mon.a (mon.0) 16485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:16:38.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[45495]: audit 2022-04-22T23:16:37.441147+0000 mon.a (mon.0) 16486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:16:38.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:37 smithi115 bash[45495]: audit 2022-04-22T23:16:37.442924+0000 mon.a (mon.0) 16487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:16:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:37 smithi121 bash[36902]: cluster 2022-04-22T23:16:37.072467+0000 mgr.y (mgr.34107) 11943 : cluster [DBG] pgmap v12455: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:37 smithi121 bash[36902]: audit 2022-04-22T23:16:37.438116+0000 mon.a (mon.0) 16485 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:16:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:37 smithi121 bash[36902]: audit 2022-04-22T23:16:37.441147+0000 mon.a (mon.0) 16486 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:16:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:37 smithi121 bash[36902]: audit 2022-04-22T23:16:37.442924+0000 mon.a (mon.0) 16487 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:39 smithi121 bash[36902]: audit 2022-04-22T23:16:38.072536+0000 mon.a (mon.0) 16488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:39 smithi121 bash[36902]: audit 2022-04-22T23:16:38.216949+0000 mon.a (mon.0) 16489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:39 smithi115 bash[46934]: audit 2022-04-22T23:16:38.072536+0000 mon.a (mon.0) 16488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:39 smithi115 bash[46934]: audit 2022-04-22T23:16:38.216949+0000 mon.a (mon.0) 16489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:39 smithi115 bash[45495]: audit 2022-04-22T23:16:38.072536+0000 mon.a (mon.0) 16488 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:39 smithi115 bash[45495]: audit 2022-04-22T23:16:38.216949+0000 mon.a (mon.0) 16489 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:40 smithi121 bash[36902]: cluster 2022-04-22T23:16:39.073700+0000 mgr.y (mgr.34107) 11944 : cluster [DBG] pgmap v12456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:40 smithi121 bash[36902]: audit 2022-04-22T23:16:39.898782+0000 mon.a (mon.0) 16490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:40.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:40 smithi115 bash[45495]: cluster 2022-04-22T23:16:39.073700+0000 mgr.y (mgr.34107) 11944 : cluster [DBG] pgmap v12456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:40 smithi115 bash[45495]: audit 2022-04-22T23:16:39.898782+0000 mon.a (mon.0) 16490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:40 smithi115 bash[46934]: cluster 2022-04-22T23:16:39.073700+0000 mgr.y (mgr.34107) 11944 : cluster [DBG] pgmap v12456: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:40 smithi115 bash[46934]: audit 2022-04-22T23:16:39.898782+0000 mon.a (mon.0) 16490 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:42.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:42 smithi121 bash[36902]: cluster 2022-04-22T23:16:41.074858+0000 mgr.y (mgr.34107) 11945 : cluster [DBG] pgmap v12457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:42.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:42 smithi121 bash[36902]: audit 2022-04-22T23:16:41.589640+0000 mon.a (mon.0) 16491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:42.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:42 smithi121 bash[36902]: audit 2022-04-22T23:16:41.874988+0000 mon.a (mon.0) 16492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[45495]: cluster 2022-04-22T23:16:41.074858+0000 mgr.y (mgr.34107) 11945 : cluster [DBG] pgmap v12457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[45495]: audit 2022-04-22T23:16:41.589640+0000 mon.a (mon.0) 16491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[45495]: audit 2022-04-22T23:16:41.874988+0000 mon.a (mon.0) 16492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[46934]: cluster 2022-04-22T23:16:41.074858+0000 mgr.y (mgr.34107) 11945 : cluster [DBG] pgmap v12457: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[46934]: audit 2022-04-22T23:16:41.589640+0000 mon.a (mon.0) 16491 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:42.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:42 smithi115 bash[46934]: audit 2022-04-22T23:16:41.874988+0000 mon.a (mon.0) 16492 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:43 smithi115 bash[45495]: audit 2022-04-22T23:16:42.221454+0000 mon.a (mon.0) 16493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:43 smithi115 bash[46934]: audit 2022-04-22T23:16:42.221454+0000 mon.a (mon.0) 16493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:43.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:43 smithi121 bash[36902]: audit 2022-04-22T23:16:42.221454+0000 mon.a (mon.0) 16493 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:16:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:44 smithi121 bash[36902]: cluster 2022-04-22T23:16:43.075870+0000 mgr.y (mgr.34107) 11946 : cluster [DBG] pgmap v12458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:44 smithi121 bash[36902]: audit 2022-04-22T23:16:43.226897+0000 mon.a (mon.0) 16494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:44 smithi115 bash[45495]: cluster 2022-04-22T23:16:43.075870+0000 mgr.y (mgr.34107) 11946 : cluster [DBG] pgmap v12458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:44 smithi115 bash[45495]: audit 2022-04-22T23:16:43.226897+0000 mon.a (mon.0) 16494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:44 smithi115 bash[46934]: cluster 2022-04-22T23:16:43.075870+0000 mgr.y (mgr.34107) 11946 : cluster [DBG] pgmap v12458: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:44 smithi115 bash[46934]: audit 2022-04-22T23:16:43.226897+0000 mon.a (mon.0) 16494 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:45.232 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:44.857Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:45 smithi121 bash[36902]: audit 2022-04-22T23:16:44.902621+0000 mon.a (mon.0) 16495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:45 smithi115 bash[45495]: audit 2022-04-22T23:16:44.902621+0000 mon.a (mon.0) 16495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:45 smithi115 bash[46934]: audit 2022-04-22T23:16:44.902621+0000 mon.a (mon.0) 16495 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:46 smithi121 bash[36902]: cluster 2022-04-22T23:16:45.076881+0000 mgr.y (mgr.34107) 11947 : cluster [DBG] pgmap v12459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:46 smithi115 bash[45495]: cluster 2022-04-22T23:16:45.076881+0000 mgr.y (mgr.34107) 11947 : cluster [DBG] pgmap v12459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:46 smithi115 bash[46934]: cluster 2022-04-22T23:16:45.076881+0000 mgr.y (mgr.34107) 11947 : cluster [DBG] pgmap v12459: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:47 smithi121 bash[36902]: audit 2022-04-22T23:16:46.592796+0000 mon.a (mon.0) 16496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:47 smithi115 bash[45495]: audit 2022-04-22T23:16:46.592796+0000 mon.a (mon.0) 16496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:47 smithi115 bash[46934]: audit 2022-04-22T23:16:46.592796+0000 mon.a (mon.0) 16496 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:48 smithi121 bash[36902]: cluster 2022-04-22T23:16:47.077958+0000 mgr.y (mgr.34107) 11948 : cluster [DBG] pgmap v12460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:48 smithi121 bash[36902]: audit 2022-04-22T23:16:48.222994+0000 mon.a (mon.0) 16497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:48 smithi115 bash[45495]: cluster 2022-04-22T23:16:47.077958+0000 mgr.y (mgr.34107) 11948 : cluster [DBG] pgmap v12460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:48 smithi115 bash[45495]: audit 2022-04-22T23:16:48.222994+0000 mon.a (mon.0) 16497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:48 smithi115 bash[46934]: cluster 2022-04-22T23:16:47.077958+0000 mgr.y (mgr.34107) 11948 : cluster [DBG] pgmap v12460: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:48 smithi115 bash[46934]: audit 2022-04-22T23:16:48.222994+0000 mon.a (mon.0) 16497 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:49.236 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:16:49.583 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:16:49.584 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:16:49.584 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 7s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:16:49.584 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:16:49.584 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:16:49.585 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:16:49.585 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:16:49.585 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 6h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:16:49.585 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7s ago 5h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:16:49.586 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:16:49.586 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:16:49.586 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7s ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:16:49.586 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:16:49.587 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:16:49.587 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:16:49.587 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:16:49.587 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7s ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:16:49.587 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:16:49.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7s ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:16:49.588 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7s ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:16:49.588 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7s ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:16:49.991 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:16:49.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:16:49.993 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:50 smithi121 bash[36902]: cluster 2022-04-22T23:16:49.079045+0000 mgr.y (mgr.34107) 11949 : cluster [DBG] pgmap v12461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:50 smithi121 bash[36902]: audit 2022-04-22T23:16:49.213888+0000 mgr.y (mgr.34107) 11950 : audit [DBG] from='client.47709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:50 smithi121 bash[36902]: audit 2022-04-22T23:16:49.906264+0000 mon.a (mon.0) 16498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:50 smithi121 bash[36902]: audit 2022-04-22T23:16:49.987938+0000 mon.a (mon.0) 16499 : audit [DBG] from='client.? 172.21.15.115:0/1635887269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[45495]: cluster 2022-04-22T23:16:49.079045+0000 mgr.y (mgr.34107) 11949 : cluster [DBG] pgmap v12461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[45495]: audit 2022-04-22T23:16:49.213888+0000 mgr.y (mgr.34107) 11950 : audit [DBG] from='client.47709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[45495]: audit 2022-04-22T23:16:49.906264+0000 mon.a (mon.0) 16498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[45495]: audit 2022-04-22T23:16:49.987938+0000 mon.a (mon.0) 16499 : audit [DBG] from='client.? 172.21.15.115:0/1635887269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[46934]: cluster 2022-04-22T23:16:49.079045+0000 mgr.y (mgr.34107) 11949 : cluster [DBG] pgmap v12461: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[46934]: audit 2022-04-22T23:16:49.213888+0000 mgr.y (mgr.34107) 11950 : audit [DBG] from='client.47709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[46934]: audit 2022-04-22T23:16:49.906264+0000 mon.a (mon.0) 16498 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:50 smithi115 bash[46934]: audit 2022-04-22T23:16:49.987938+0000 mon.a (mon.0) 16499 : audit [DBG] from='client.? 172.21.15.115:0/1635887269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:16:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:51 smithi121 bash[36902]: audit 2022-04-22T23:16:49.560232+0000 mgr.y (mgr.34107) 11951 : audit [DBG] from='client.47715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:51 smithi115 bash[45495]: audit 2022-04-22T23:16:49.560232+0000 mgr.y (mgr.34107) 11951 : audit [DBG] from='client.47715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:51 smithi115 bash[46934]: audit 2022-04-22T23:16:49.560232+0000 mgr.y (mgr.34107) 11951 : audit [DBG] from='client.47715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:16:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:52 smithi121 bash[36902]: cluster 2022-04-22T23:16:51.080153+0000 mgr.y (mgr.34107) 11952 : cluster [DBG] pgmap v12462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:52 smithi121 bash[36902]: audit 2022-04-22T23:16:51.595851+0000 mon.a (mon.0) 16500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:52 smithi115 bash[45495]: cluster 2022-04-22T23:16:51.080153+0000 mgr.y (mgr.34107) 11952 : cluster [DBG] pgmap v12462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:52 smithi115 bash[45495]: audit 2022-04-22T23:16:51.595851+0000 mon.a (mon.0) 16500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:52 smithi115 bash[46934]: cluster 2022-04-22T23:16:51.080153+0000 mgr.y (mgr.34107) 11952 : cluster [DBG] pgmap v12462: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:52 smithi115 bash[46934]: audit 2022-04-22T23:16:51.595851+0000 mon.a (mon.0) 16500 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:52 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:52.999Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:53 smithi121 bash[36902]: cluster 2022-04-22T23:16:53.081206+0000 mgr.y (mgr.34107) 11953 : cluster [DBG] pgmap v12463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:53 smithi121 bash[36902]: audit 2022-04-22T23:16:53.226029+0000 mon.a (mon.0) 16501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:53 smithi115 bash[46934]: cluster 2022-04-22T23:16:53.081206+0000 mgr.y (mgr.34107) 11953 : cluster [DBG] pgmap v12463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:53 smithi115 bash[46934]: audit 2022-04-22T23:16:53.226029+0000 mon.a (mon.0) 16501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:53 smithi115 bash[45495]: cluster 2022-04-22T23:16:53.081206+0000 mgr.y (mgr.34107) 11953 : cluster [DBG] pgmap v12463: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:53 smithi115 bash[45495]: audit 2022-04-22T23:16:53.226029+0000 mon.a (mon.0) 16501 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:54 smithi121 bash[36902]: audit 2022-04-22T23:16:54.909820+0000 mon.a (mon.0) 16502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:54 smithi115 bash[45495]: audit 2022-04-22T23:16:54.909820+0000 mon.a (mon.0) 16502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:16:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:16:54.858Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:16:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:54 smithi115 bash[46934]: audit 2022-04-22T23:16:54.909820+0000 mon.a (mon.0) 16502 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:16:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:55 smithi121 bash[36902]: cluster 2022-04-22T23:16:55.082315+0000 mgr.y (mgr.34107) 11954 : cluster [DBG] pgmap v12464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:55 smithi115 bash[45495]: cluster 2022-04-22T23:16:55.082315+0000 mgr.y (mgr.34107) 11954 : cluster [DBG] pgmap v12464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:55 smithi115 bash[46934]: cluster 2022-04-22T23:16:55.082315+0000 mgr.y (mgr.34107) 11954 : cluster [DBG] pgmap v12464: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:56 smithi121 bash[36902]: audit 2022-04-22T23:16:56.599010+0000 mon.a (mon.0) 16503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:56 smithi115 bash[45495]: audit 2022-04-22T23:16:56.599010+0000 mon.a (mon.0) 16503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:56 smithi115 bash[46934]: audit 2022-04-22T23:16:56.599010+0000 mon.a (mon.0) 16503 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:16:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:57 smithi121 bash[36902]: cluster 2022-04-22T23:16:57.083461+0000 mgr.y (mgr.34107) 11955 : cluster [DBG] pgmap v12465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:58.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:57 smithi115 bash[45495]: cluster 2022-04-22T23:16:57.083461+0000 mgr.y (mgr.34107) 11955 : cluster [DBG] pgmap v12465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:57 smithi115 bash[46934]: cluster 2022-04-22T23:16:57.083461+0000 mgr.y (mgr.34107) 11955 : cluster [DBG] pgmap v12465: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:16:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:16:58 smithi121 bash[36902]: audit 2022-04-22T23:16:58.229274+0000 mon.a (mon.0) 16504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:58 smithi115 bash[45495]: audit 2022-04-22T23:16:58.229274+0000 mon.a (mon.0) 16504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:16:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:58 smithi115 bash[46934]: audit 2022-04-22T23:16:58.229274+0000 mon.a (mon.0) 16504 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:00 smithi121 bash[36902]: cluster 2022-04-22T23:16:59.084496+0000 mgr.y (mgr.34107) 11956 : cluster [DBG] pgmap v12466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:00 smithi121 bash[36902]: audit 2022-04-22T23:16:59.913576+0000 mon.a (mon.0) 16505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:59 smithi115 bash[45495]: cluster 2022-04-22T23:16:59.084496+0000 mgr.y (mgr.34107) 11956 : cluster [DBG] pgmap v12466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:16:59 smithi115 bash[45495]: audit 2022-04-22T23:16:59.913576+0000 mon.a (mon.0) 16505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:59 smithi115 bash[46934]: cluster 2022-04-22T23:16:59.084496+0000 mgr.y (mgr.34107) 11956 : cluster [DBG] pgmap v12466: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:16:59 smithi115 bash[46934]: audit 2022-04-22T23:16:59.913576+0000 mon.a (mon.0) 16505 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:02 smithi121 bash[36902]: cluster 2022-04-22T23:17:01.085561+0000 mgr.y (mgr.34107) 11957 : cluster [DBG] pgmap v12467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:02 smithi121 bash[36902]: audit 2022-04-22T23:17:01.602032+0000 mon.a (mon.0) 16506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:02 smithi115 bash[45495]: cluster 2022-04-22T23:17:01.085561+0000 mgr.y (mgr.34107) 11957 : cluster [DBG] pgmap v12467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:02 smithi115 bash[45495]: audit 2022-04-22T23:17:01.602032+0000 mon.a (mon.0) 16506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:02 smithi115 bash[46934]: cluster 2022-04-22T23:17:01.085561+0000 mgr.y (mgr.34107) 11957 : cluster [DBG] pgmap v12467: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:02 smithi115 bash[46934]: audit 2022-04-22T23:17:01.602032+0000 mon.a (mon.0) 16506 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:03.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:03 smithi121 bash[36902]: cluster 2022-04-22T23:17:03.086669+0000 mgr.y (mgr.34107) 11958 : cluster [DBG] pgmap v12468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:04.045 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:03 smithi121 bash[36902]: audit 2022-04-22T23:17:03.232298+0000 mon.a (mon.0) 16507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:03 smithi115 bash[45495]: cluster 2022-04-22T23:17:03.086669+0000 mgr.y (mgr.34107) 11958 : cluster [DBG] pgmap v12468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:04.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:03 smithi115 bash[45495]: audit 2022-04-22T23:17:03.232298+0000 mon.a (mon.0) 16507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:03 smithi115 bash[46934]: cluster 2022-04-22T23:17:03.086669+0000 mgr.y (mgr.34107) 11958 : cluster [DBG] pgmap v12468: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:03 smithi115 bash[46934]: audit 2022-04-22T23:17:03.232298+0000 mon.a (mon.0) 16507 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:04 smithi121 bash[36902]: audit 2022-04-22T23:17:04.917643+0000 mon.a (mon.0) 16508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:05.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:04 smithi115 bash[45495]: audit 2022-04-22T23:17:04.917643+0000 mon.a (mon.0) 16508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:04 smithi115 bash[46934]: audit 2022-04-22T23:17:04.917643+0000 mon.a (mon.0) 16508 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:04.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:05 smithi121 bash[36902]: cluster 2022-04-22T23:17:05.087770+0000 mgr.y (mgr.34107) 11959 : cluster [DBG] pgmap v12469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:05 smithi115 bash[46934]: cluster 2022-04-22T23:17:05.087770+0000 mgr.y (mgr.34107) 11959 : cluster [DBG] pgmap v12469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:05 smithi115 bash[45495]: cluster 2022-04-22T23:17:05.087770+0000 mgr.y (mgr.34107) 11959 : cluster [DBG] pgmap v12469: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:07.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:06 smithi121 bash[36902]: audit 2022-04-22T23:17:06.604935+0000 mon.a (mon.0) 16509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:06 smithi115 bash[45495]: audit 2022-04-22T23:17:06.604935+0000 mon.a (mon.0) 16509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:06 smithi115 bash[46934]: audit 2022-04-22T23:17:06.604935+0000 mon.a (mon.0) 16509 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:07 smithi121 bash[36902]: cluster 2022-04-22T23:17:07.088822+0000 mgr.y (mgr.34107) 11960 : cluster [DBG] pgmap v12470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:07 smithi115 bash[45495]: cluster 2022-04-22T23:17:07.088822+0000 mgr.y (mgr.34107) 11960 : cluster [DBG] pgmap v12470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:07 smithi115 bash[46934]: cluster 2022-04-22T23:17:07.088822+0000 mgr.y (mgr.34107) 11960 : cluster [DBG] pgmap v12470: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:08 smithi121 bash[36902]: audit 2022-04-22T23:17:08.235218+0000 mon.a (mon.0) 16510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:08 smithi115 bash[45495]: audit 2022-04-22T23:17:08.235218+0000 mon.a (mon.0) 16510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:08 smithi115 bash[46934]: audit 2022-04-22T23:17:08.235218+0000 mon.a (mon.0) 16510 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:10 smithi121 bash[36902]: cluster 2022-04-22T23:17:09.090052+0000 mgr.y (mgr.34107) 11961 : cluster [DBG] pgmap v12471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:10 smithi121 bash[36902]: audit 2022-04-22T23:17:09.921287+0000 mon.a (mon.0) 16511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:10 smithi115 bash[45495]: cluster 2022-04-22T23:17:09.090052+0000 mgr.y (mgr.34107) 11961 : cluster [DBG] pgmap v12471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:10 smithi115 bash[45495]: audit 2022-04-22T23:17:09.921287+0000 mon.a (mon.0) 16511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:10 smithi115 bash[46934]: cluster 2022-04-22T23:17:09.090052+0000 mgr.y (mgr.34107) 11961 : cluster [DBG] pgmap v12471: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:10 smithi115 bash[46934]: audit 2022-04-22T23:17:09.921287+0000 mon.a (mon.0) 16511 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:12 smithi121 bash[36902]: cluster 2022-04-22T23:17:11.091139+0000 mgr.y (mgr.34107) 11962 : cluster [DBG] pgmap v12472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:12 smithi121 bash[36902]: audit 2022-04-22T23:17:11.607867+0000 mon.a (mon.0) 16512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:12 smithi115 bash[46934]: cluster 2022-04-22T23:17:11.091139+0000 mgr.y (mgr.34107) 11962 : cluster [DBG] pgmap v12472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:12 smithi115 bash[46934]: audit 2022-04-22T23:17:11.607867+0000 mon.a (mon.0) 16512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:12 smithi115 bash[45495]: cluster 2022-04-22T23:17:11.091139+0000 mgr.y (mgr.34107) 11962 : cluster [DBG] pgmap v12472: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:12 smithi115 bash[45495]: audit 2022-04-22T23:17:11.607867+0000 mon.a (mon.0) 16512 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:13.000Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:13 smithi121 bash[36902]: cluster 2022-04-22T23:17:13.092222+0000 mgr.y (mgr.34107) 11963 : cluster [DBG] pgmap v12473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:13 smithi121 bash[36902]: audit 2022-04-22T23:17:13.238301+0000 mon.a (mon.0) 16513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:13 smithi115 bash[45495]: cluster 2022-04-22T23:17:13.092222+0000 mgr.y (mgr.34107) 11963 : cluster [DBG] pgmap v12473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:13 smithi115 bash[45495]: audit 2022-04-22T23:17:13.238301+0000 mon.a (mon.0) 16513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:13 smithi115 bash[46934]: cluster 2022-04-22T23:17:13.092222+0000 mgr.y (mgr.34107) 11963 : cluster [DBG] pgmap v12473: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:13 smithi115 bash[46934]: audit 2022-04-22T23:17:13.238301+0000 mon.a (mon.0) 16513 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:14 smithi121 bash[36902]: audit 2022-04-22T23:17:14.925144+0000 mon.a (mon.0) 16514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:14 smithi115 bash[45495]: audit 2022-04-22T23:17:14.925144+0000 mon.a (mon.0) 16514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:14 smithi115 bash[46934]: audit 2022-04-22T23:17:14.925144+0000 mon.a (mon.0) 16514 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:14.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:15 smithi121 bash[36902]: cluster 2022-04-22T23:17:15.093224+0000 mgr.y (mgr.34107) 11964 : cluster [DBG] pgmap v12474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:16.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:15 smithi115 bash[45495]: cluster 2022-04-22T23:17:15.093224+0000 mgr.y (mgr.34107) 11964 : cluster [DBG] pgmap v12474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:15 smithi115 bash[46934]: cluster 2022-04-22T23:17:15.093224+0000 mgr.y (mgr.34107) 11964 : cluster [DBG] pgmap v12474: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:16 smithi121 bash[36902]: audit 2022-04-22T23:17:16.610786+0000 mon.a (mon.0) 16515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:17.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:16 smithi115 bash[45495]: audit 2022-04-22T23:17:16.610786+0000 mon.a (mon.0) 16515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:16 smithi115 bash[46934]: audit 2022-04-22T23:17:16.610786+0000 mon.a (mon.0) 16515 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:17 smithi121 bash[36902]: cluster 2022-04-22T23:17:17.094374+0000 mgr.y (mgr.34107) 11965 : cluster [DBG] pgmap v12475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:17 smithi121 bash[36902]: audit 2022-04-22T23:17:17.315230+0000 mon.a (mon.0) 16516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:17:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:18 smithi121 bash[36902]: audit 2022-04-22T23:17:17.315918+0000 mon.a (mon.0) 16517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:17 smithi115 bash[45495]: cluster 2022-04-22T23:17:17.094374+0000 mgr.y (mgr.34107) 11965 : cluster [DBG] pgmap v12475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:17 smithi115 bash[45495]: audit 2022-04-22T23:17:17.315230+0000 mon.a (mon.0) 16516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:17 smithi115 bash[45495]: audit 2022-04-22T23:17:17.315918+0000 mon.a (mon.0) 16517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:17 smithi115 bash[46934]: cluster 2022-04-22T23:17:17.094374+0000 mgr.y (mgr.34107) 11965 : cluster [DBG] pgmap v12475: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:18 smithi115 bash[46934]: audit 2022-04-22T23:17:17.315230+0000 mon.a (mon.0) 16516 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:17:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:18 smithi115 bash[46934]: audit 2022-04-22T23:17:17.315918+0000 mon.a (mon.0) 16517 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:17:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:19 smithi121 bash[36902]: audit 2022-04-22T23:17:18.241330+0000 mon.a (mon.0) 16518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:19.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:18 smithi115 bash[45495]: audit 2022-04-22T23:17:18.241330+0000 mon.a (mon.0) 16518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:19.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:19 smithi115 bash[46934]: audit 2022-04-22T23:17:18.241330+0000 mon.a (mon.0) 16518 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:20 smithi121 bash[36902]: cluster 2022-04-22T23:17:19.095606+0000 mgr.y (mgr.34107) 11966 : cluster [DBG] pgmap v12476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:20 smithi121 bash[36902]: audit 2022-04-22T23:17:19.928724+0000 mon.a (mon.0) 16519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:20 smithi115 bash[45495]: cluster 2022-04-22T23:17:19.095606+0000 mgr.y (mgr.34107) 11966 : cluster [DBG] pgmap v12476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:20 smithi115 bash[45495]: audit 2022-04-22T23:17:19.928724+0000 mon.a (mon.0) 16519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:20 smithi115 bash[46934]: cluster 2022-04-22T23:17:19.095606+0000 mgr.y (mgr.34107) 11966 : cluster [DBG] pgmap v12476: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:20 smithi115 bash[46934]: audit 2022-04-22T23:17:19.928724+0000 mon.a (mon.0) 16519 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:20.361 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:17:20.705 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:17:20.705 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 5h 15.5M - ba2b418f427c 9255ba08f72f 2022-04-22T23:17:20.706 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 38s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:17:20.706 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:17:20.706 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 38s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:17:20.706 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 38s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:17:20.706 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 9m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:17:20.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 6h 461M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:17:20.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 38s ago 5h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:17:20.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 5h 316M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:17:20.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 5h 10.4M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:17:20.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 38s ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 5h 720M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 5h 801M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 38s ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:17:20.708 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 38s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:17:20.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 38s ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:17:20.709 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 38s ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:17:20.709 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 38s ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:17:21.126 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:17:21.126 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:17:21.126 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:17:21.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:21.128 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:17:21.128 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:17:21.128 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:17:21.128 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:17:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:21 smithi121 bash[36902]: audit 2022-04-22T23:17:21.122667+0000 mon.c (mon.1) 404 : audit [DBG] from='client.? 172.21.15.115:0/653021394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:21 smithi115 bash[45495]: audit 2022-04-22T23:17:21.122667+0000 mon.c (mon.1) 404 : audit [DBG] from='client.? 172.21.15.115:0/653021394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:21 smithi115 bash[46934]: audit 2022-04-22T23:17:21.122667+0000 mon.c (mon.1) 404 : audit [DBG] from='client.? 172.21.15.115:0/653021394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:22.186 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[46934]: audit 2022-04-22T23:17:20.343944+0000 mgr.y (mgr.34107) 11967 : audit [DBG] from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[46934]: audit 2022-04-22T23:17:20.680636+0000 mgr.y (mgr.34107) 11968 : audit [DBG] from='client.47733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[46934]: cluster 2022-04-22T23:17:21.096665+0000 mgr.y (mgr.34107) 11969 : cluster [DBG] pgmap v12477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[46934]: audit 2022-04-22T23:17:21.613965+0000 mon.a (mon.0) 16520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[45495]: audit 2022-04-22T23:17:20.343944+0000 mgr.y (mgr.34107) 11967 : audit [DBG] from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[45495]: audit 2022-04-22T23:17:20.680636+0000 mgr.y (mgr.34107) 11968 : audit [DBG] from='client.47733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[45495]: cluster 2022-04-22T23:17:21.096665+0000 mgr.y (mgr.34107) 11969 : cluster [DBG] pgmap v12477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:22.188 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:22 smithi115 bash[45495]: audit 2022-04-22T23:17:21.613965+0000 mon.a (mon.0) 16520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:22 smithi121 bash[36902]: audit 2022-04-22T23:17:20.343944+0000 mgr.y (mgr.34107) 11967 : audit [DBG] from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:22 smithi121 bash[36902]: audit 2022-04-22T23:17:20.680636+0000 mgr.y (mgr.34107) 11968 : audit [DBG] from='client.47733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:22 smithi121 bash[36902]: cluster 2022-04-22T23:17:21.096665+0000 mgr.y (mgr.34107) 11969 : cluster [DBG] pgmap v12477: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:22 smithi121 bash[36902]: audit 2022-04-22T23:17:21.613965+0000 mon.a (mon.0) 16520 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:23.001Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:23 smithi121 bash[36902]: cluster 2022-04-22T23:17:23.097695+0000 mgr.y (mgr.34107) 11970 : cluster [DBG] pgmap v12478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:23 smithi121 bash[36902]: audit 2022-04-22T23:17:23.244455+0000 mon.a (mon.0) 16521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:23 smithi115 bash[45495]: cluster 2022-04-22T23:17:23.097695+0000 mgr.y (mgr.34107) 11970 : cluster [DBG] pgmap v12478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:23 smithi115 bash[45495]: audit 2022-04-22T23:17:23.244455+0000 mon.a (mon.0) 16521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:23 smithi115 bash[46934]: cluster 2022-04-22T23:17:23.097695+0000 mgr.y (mgr.34107) 11970 : cluster [DBG] pgmap v12478: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:23 smithi115 bash[46934]: audit 2022-04-22T23:17:23.244455+0000 mon.a (mon.0) 16521 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:24 smithi121 bash[36902]: audit 2022-04-22T23:17:24.932464+0000 mon.a (mon.0) 16522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:24.859Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:24 smithi115 bash[45495]: audit 2022-04-22T23:17:24.932464+0000 mon.a (mon.0) 16522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:24 smithi115 bash[46934]: audit 2022-04-22T23:17:24.932464+0000 mon.a (mon.0) 16522 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:26 smithi121 bash[36902]: cluster 2022-04-22T23:17:25.098772+0000 mgr.y (mgr.34107) 11971 : cluster [DBG] pgmap v12479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:25 smithi115 bash[45495]: cluster 2022-04-22T23:17:25.098772+0000 mgr.y (mgr.34107) 11971 : cluster [DBG] pgmap v12479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:26 smithi115 bash[46934]: cluster 2022-04-22T23:17:25.098772+0000 mgr.y (mgr.34107) 11971 : cluster [DBG] pgmap v12479: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:27.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:27 smithi121 bash[36902]: audit 2022-04-22T23:17:26.617048+0000 mon.a (mon.0) 16523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:27 smithi115 bash[45495]: audit 2022-04-22T23:17:26.617048+0000 mon.a (mon.0) 16523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:27 smithi115 bash[46934]: audit 2022-04-22T23:17:26.617048+0000 mon.a (mon.0) 16523 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:28 smithi115 bash[45495]: cluster 2022-04-22T23:17:27.100067+0000 mgr.y (mgr.34107) 11972 : cluster [DBG] pgmap v12480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:28 smithi115 bash[46934]: cluster 2022-04-22T23:17:27.100067+0000 mgr.y (mgr.34107) 11972 : cluster [DBG] pgmap v12480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:28 smithi121 bash[36902]: cluster 2022-04-22T23:17:27.100067+0000 mgr.y (mgr.34107) 11972 : cluster [DBG] pgmap v12480: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:29 smithi115 bash[45495]: audit 2022-04-22T23:17:28.246833+0000 mon.a (mon.0) 16524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:29 smithi115 bash[46934]: audit 2022-04-22T23:17:28.246833+0000 mon.a (mon.0) 16524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:29 smithi121 bash[36902]: audit 2022-04-22T23:17:28.246833+0000 mon.a (mon.0) 16524 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:30 smithi115 bash[45495]: cluster 2022-04-22T23:17:29.101125+0000 mgr.y (mgr.34107) 11973 : cluster [DBG] pgmap v12481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:30.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:30 smithi115 bash[45495]: audit 2022-04-22T23:17:29.936313+0000 mon.a (mon.0) 16525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:30 smithi115 bash[46934]: cluster 2022-04-22T23:17:29.101125+0000 mgr.y (mgr.34107) 11973 : cluster [DBG] pgmap v12481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:30.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:30 smithi115 bash[46934]: audit 2022-04-22T23:17:29.936313+0000 mon.a (mon.0) 16525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:30 smithi121 bash[36902]: cluster 2022-04-22T23:17:29.101125+0000 mgr.y (mgr.34107) 11973 : cluster [DBG] pgmap v12481: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:30 smithi121 bash[36902]: audit 2022-04-22T23:17:29.936313+0000 mon.a (mon.0) 16525 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:32 smithi121 bash[36902]: cluster 2022-04-22T23:17:31.102218+0000 mgr.y (mgr.34107) 11974 : cluster [DBG] pgmap v12482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:32 smithi121 bash[36902]: audit 2022-04-22T23:17:31.620111+0000 mon.a (mon.0) 16526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:32 smithi115 bash[45495]: cluster 2022-04-22T23:17:31.102218+0000 mgr.y (mgr.34107) 11974 : cluster [DBG] pgmap v12482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:32 smithi115 bash[45495]: audit 2022-04-22T23:17:31.620111+0000 mon.a (mon.0) 16526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:32 smithi115 bash[46934]: cluster 2022-04-22T23:17:31.102218+0000 mgr.y (mgr.34107) 11974 : cluster [DBG] pgmap v12482: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:32 smithi115 bash[46934]: audit 2022-04-22T23:17:31.620111+0000 mon.a (mon.0) 16526 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:33.002Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:33 smithi121 bash[36902]: cluster 2022-04-22T23:17:33.103268+0000 mgr.y (mgr.34107) 11975 : cluster [DBG] pgmap v12483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:33 smithi121 bash[36902]: audit 2022-04-22T23:17:33.249997+0000 mon.a (mon.0) 16527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:33 smithi115 bash[45495]: cluster 2022-04-22T23:17:33.103268+0000 mgr.y (mgr.34107) 11975 : cluster [DBG] pgmap v12483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:33 smithi115 bash[45495]: audit 2022-04-22T23:17:33.249997+0000 mon.a (mon.0) 16527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:33 smithi115 bash[46934]: cluster 2022-04-22T23:17:33.103268+0000 mgr.y (mgr.34107) 11975 : cluster [DBG] pgmap v12483: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:33 smithi115 bash[46934]: audit 2022-04-22T23:17:33.249997+0000 mon.a (mon.0) 16527 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:35 smithi121 bash[36902]: audit 2022-04-22T23:17:34.940065+0000 mon.a (mon.0) 16528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:34.860Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:34 smithi115 bash[45495]: audit 2022-04-22T23:17:34.940065+0000 mon.a (mon.0) 16528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:34 smithi115 bash[46934]: audit 2022-04-22T23:17:34.940065+0000 mon.a (mon.0) 16528 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:36 smithi121 bash[36902]: cluster 2022-04-22T23:17:35.104334+0000 mgr.y (mgr.34107) 11976 : cluster [DBG] pgmap v12484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:36 smithi115 bash[46934]: cluster 2022-04-22T23:17:35.104334+0000 mgr.y (mgr.34107) 11976 : cluster [DBG] pgmap v12484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:36 smithi115 bash[45495]: cluster 2022-04-22T23:17:35.104334+0000 mgr.y (mgr.34107) 11976 : cluster [DBG] pgmap v12484: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:37 smithi121 bash[36902]: audit 2022-04-22T23:17:36.623852+0000 mon.a (mon.0) 16529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:37 smithi115 bash[46934]: audit 2022-04-22T23:17:36.623852+0000 mon.a (mon.0) 16529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:37 smithi115 bash[45495]: audit 2022-04-22T23:17:36.623852+0000 mon.a (mon.0) 16529 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:38 smithi115 bash[46934]: cluster 2022-04-22T23:17:37.105604+0000 mgr.y (mgr.34107) 11977 : cluster [DBG] pgmap v12485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:38 smithi115 bash[45495]: cluster 2022-04-22T23:17:37.105604+0000 mgr.y (mgr.34107) 11977 : cluster [DBG] pgmap v12485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:38 smithi121 bash[36902]: cluster 2022-04-22T23:17:37.105604+0000 mgr.y (mgr.34107) 11977 : cluster [DBG] pgmap v12485: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:39 smithi115 bash[45495]: audit 2022-04-22T23:17:38.252869+0000 mon.a (mon.0) 16530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:39 smithi115 bash[46934]: audit 2022-04-22T23:17:38.252869+0000 mon.a (mon.0) 16530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:39 smithi121 bash[36902]: audit 2022-04-22T23:17:38.252869+0000 mon.a (mon.0) 16530 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:40 smithi115 bash[45495]: cluster 2022-04-22T23:17:39.106759+0000 mgr.y (mgr.34107) 11978 : cluster [DBG] pgmap v12486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:40 smithi115 bash[45495]: audit 2022-04-22T23:17:39.943844+0000 mon.a (mon.0) 16531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:40 smithi115 bash[46934]: cluster 2022-04-22T23:17:39.106759+0000 mgr.y (mgr.34107) 11978 : cluster [DBG] pgmap v12486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:40 smithi115 bash[46934]: audit 2022-04-22T23:17:39.943844+0000 mon.a (mon.0) 16531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:40 smithi121 bash[36902]: cluster 2022-04-22T23:17:39.106759+0000 mgr.y (mgr.34107) 11978 : cluster [DBG] pgmap v12486: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:40 smithi121 bash[36902]: audit 2022-04-22T23:17:39.943844+0000 mon.a (mon.0) 16531 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:42 smithi121 bash[36902]: cluster 2022-04-22T23:17:41.107709+0000 mgr.y (mgr.34107) 11979 : cluster [DBG] pgmap v12487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:42 smithi121 bash[36902]: audit 2022-04-22T23:17:41.626939+0000 mon.a (mon.0) 16532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:42.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:42 smithi115 bash[46934]: cluster 2022-04-22T23:17:41.107709+0000 mgr.y (mgr.34107) 11979 : cluster [DBG] pgmap v12487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:42.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:42 smithi115 bash[46934]: audit 2022-04-22T23:17:41.626939+0000 mon.a (mon.0) 16532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:42 smithi115 bash[45495]: cluster 2022-04-22T23:17:41.107709+0000 mgr.y (mgr.34107) 11979 : cluster [DBG] pgmap v12487: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:42 smithi115 bash[45495]: audit 2022-04-22T23:17:41.626939+0000 mon.a (mon.0) 16532 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:43.004Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[45495]: audit 2022-04-22T23:17:42.230854+0000 mon.a (mon.0) 16533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:17:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[45495]: audit 2022-04-22T23:17:42.234009+0000 mon.a (mon.0) 16534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:17:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[45495]: audit 2022-04-22T23:17:42.236155+0000 mon.a (mon.0) 16535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[45495]: audit 2022-04-22T23:17:42.575843+0000 mon.a (mon.0) 16536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[45495]: audit 2022-04-22T23:17:42.580252+0000 mon.a (mon.0) 16537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[46934]: audit 2022-04-22T23:17:42.230854+0000 mon.a (mon.0) 16533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[46934]: audit 2022-04-22T23:17:42.234009+0000 mon.a (mon.0) 16534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[46934]: audit 2022-04-22T23:17:42.236155+0000 mon.a (mon.0) 16535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[46934]: audit 2022-04-22T23:17:42.575843+0000 mon.a (mon.0) 16536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:43 smithi115 bash[46934]: audit 2022-04-22T23:17:42.580252+0000 mon.a (mon.0) 16537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:43 smithi121 bash[36902]: audit 2022-04-22T23:17:42.230854+0000 mon.a (mon.0) 16533 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:17:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:43 smithi121 bash[36902]: audit 2022-04-22T23:17:42.234009+0000 mon.a (mon.0) 16534 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:17:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:43 smithi121 bash[36902]: audit 2022-04-22T23:17:42.236155+0000 mon.a (mon.0) 16535 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:17:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:43 smithi121 bash[36902]: audit 2022-04-22T23:17:42.575843+0000 mon.a (mon.0) 16536 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:43 smithi121 bash[36902]: audit 2022-04-22T23:17:42.580252+0000 mon.a (mon.0) 16537 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:44 smithi121 bash[36902]: cluster 2022-04-22T23:17:43.108732+0000 mgr.y (mgr.34107) 11980 : cluster [DBG] pgmap v12488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:44 smithi121 bash[36902]: audit 2022-04-22T23:17:43.256035+0000 mon.a (mon.0) 16538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:44 smithi115 bash[45495]: cluster 2022-04-22T23:17:43.108732+0000 mgr.y (mgr.34107) 11980 : cluster [DBG] pgmap v12488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:44 smithi115 bash[45495]: audit 2022-04-22T23:17:43.256035+0000 mon.a (mon.0) 16538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:44 smithi115 bash[46934]: cluster 2022-04-22T23:17:43.108732+0000 mgr.y (mgr.34107) 11980 : cluster [DBG] pgmap v12488: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:44 smithi115 bash[46934]: audit 2022-04-22T23:17:43.256035+0000 mon.a (mon.0) 16538 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:45.180 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:44.862Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:45 smithi121 bash[36902]: audit 2022-04-22T23:17:44.947396+0000 mon.a (mon.0) 16539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:45 smithi115 bash[45495]: audit 2022-04-22T23:17:44.947396+0000 mon.a (mon.0) 16539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:45 smithi115 bash[46934]: audit 2022-04-22T23:17:44.947396+0000 mon.a (mon.0) 16539 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:46 smithi121 bash[36902]: cluster 2022-04-22T23:17:45.109820+0000 mgr.y (mgr.34107) 11981 : cluster [DBG] pgmap v12489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:46.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:46 smithi115 bash[46934]: cluster 2022-04-22T23:17:45.109820+0000 mgr.y (mgr.34107) 11981 : cluster [DBG] pgmap v12489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:46 smithi115 bash[45495]: cluster 2022-04-22T23:17:45.109820+0000 mgr.y (mgr.34107) 11981 : cluster [DBG] pgmap v12489: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:47 smithi121 bash[36902]: audit 2022-04-22T23:17:46.629872+0000 mon.a (mon.0) 16540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:47 smithi121 bash[36902]: audit 2022-04-22T23:17:46.736055+0000 mon.a (mon.0) 16541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:47.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:47 smithi115 bash[45495]: audit 2022-04-22T23:17:46.629872+0000 mon.a (mon.0) 16540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:47 smithi115 bash[45495]: audit 2022-04-22T23:17:46.736055+0000 mon.a (mon.0) 16541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:47 smithi115 bash[46934]: audit 2022-04-22T23:17:46.629872+0000 mon.a (mon.0) 16540 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:47 smithi115 bash[46934]: audit 2022-04-22T23:17:46.736055+0000 mon.a (mon.0) 16541 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:48 smithi121 bash[36902]: cluster 2022-04-22T23:17:47.110697+0000 mgr.y (mgr.34107) 11982 : cluster [DBG] pgmap v12490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:48 smithi115 bash[45495]: cluster 2022-04-22T23:17:47.110697+0000 mgr.y (mgr.34107) 11982 : cluster [DBG] pgmap v12490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:48 smithi115 bash[46934]: cluster 2022-04-22T23:17:47.110697+0000 mgr.y (mgr.34107) 11982 : cluster [DBG] pgmap v12490: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:49 smithi121 bash[36902]: audit 2022-04-22T23:17:48.259288+0000 mon.a (mon.0) 16542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:49 smithi115 bash[45495]: audit 2022-04-22T23:17:48.259288+0000 mon.a (mon.0) 16542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:49 smithi115 bash[46934]: audit 2022-04-22T23:17:48.259288+0000 mon.a (mon.0) 16542 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:50.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:50 smithi115 bash[46934]: cluster 2022-04-22T23:17:49.111737+0000 mgr.y (mgr.34107) 11983 : cluster [DBG] pgmap v12491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:50.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:50 smithi115 bash[46934]: audit 2022-04-22T23:17:49.951085+0000 mon.a (mon.0) 16543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:50 smithi115 bash[45495]: cluster 2022-04-22T23:17:49.111737+0000 mgr.y (mgr.34107) 11983 : cluster [DBG] pgmap v12491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:50.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:50 smithi115 bash[45495]: audit 2022-04-22T23:17:49.951085+0000 mon.a (mon.0) 16543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:50 smithi121 bash[36902]: cluster 2022-04-22T23:17:49.111737+0000 mgr.y (mgr.34107) 11983 : cluster [DBG] pgmap v12491: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:50 smithi121 bash[36902]: audit 2022-04-22T23:17:49.951085+0000 mon.a (mon.0) 16543 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:51.490 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 69s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 69s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 69s ago 5h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:17:51.827 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5s ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5s ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 69s ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5s ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 69s ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5s ago 5h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:17:51.828 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5s ago 5h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:17:51.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:17:51.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:17:51.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 69s ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:17:51.829 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 69s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:17:51.830 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 69s ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:17:51.830 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 69s ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:17:51.830 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 69s ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:17:52.227 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:17:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:17:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:17:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:17:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:17:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:17:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:17:52.229 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:17:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:52 smithi121 bash[36902]: cluster 2022-04-22T23:17:51.112751+0000 mgr.y (mgr.34107) 11984 : cluster [DBG] pgmap v12492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:52 smithi121 bash[36902]: audit 2022-04-22T23:17:51.632316+0000 mon.a (mon.0) 16544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:52 smithi115 bash[45495]: cluster 2022-04-22T23:17:51.112751+0000 mgr.y (mgr.34107) 11984 : cluster [DBG] pgmap v12492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:52 smithi115 bash[45495]: audit 2022-04-22T23:17:51.632316+0000 mon.a (mon.0) 16544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:52 smithi115 bash[46934]: cluster 2022-04-22T23:17:51.112751+0000 mgr.y (mgr.34107) 11984 : cluster [DBG] pgmap v12492: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:52 smithi115 bash[46934]: audit 2022-04-22T23:17:51.632316+0000 mon.a (mon.0) 16544 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[46934]: audit 2022-04-22T23:17:51.469131+0000 mgr.y (mgr.34107) 11985 : audit [DBG] from='client.47745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[46934]: audit 2022-04-22T23:17:51.803788+0000 mgr.y (mgr.34107) 11986 : audit [DBG] from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[46934]: audit 2022-04-22T23:17:52.223670+0000 mon.c (mon.1) 405 : audit [DBG] from='client.? 172.21.15.115:0/10717448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:53.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[45495]: audit 2022-04-22T23:17:51.469131+0000 mgr.y (mgr.34107) 11985 : audit [DBG] from='client.47745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[45495]: audit 2022-04-22T23:17:51.803788+0000 mgr.y (mgr.34107) 11986 : audit [DBG] from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:53 smithi115 bash[45495]: audit 2022-04-22T23:17:52.223670+0000 mon.c (mon.1) 405 : audit [DBG] from='client.? 172.21.15.115:0/10717448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:53 smithi121 bash[36902]: audit 2022-04-22T23:17:51.469131+0000 mgr.y (mgr.34107) 11985 : audit [DBG] from='client.47745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:53 smithi121 bash[36902]: audit 2022-04-22T23:17:51.803788+0000 mgr.y (mgr.34107) 11986 : audit [DBG] from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:17:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:53 smithi121 bash[36902]: audit 2022-04-22T23:17:52.223670+0000 mon.c (mon.1) 405 : audit [DBG] from='client.? 172.21.15.115:0/10717448' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:17:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:54 smithi121 bash[36902]: cluster 2022-04-22T23:17:53.113824+0000 mgr.y (mgr.34107) 11987 : cluster [DBG] pgmap v12493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:54 smithi121 bash[36902]: audit 2022-04-22T23:17:53.262338+0000 mon.a (mon.0) 16545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:54 smithi115 bash[45495]: cluster 2022-04-22T23:17:53.113824+0000 mgr.y (mgr.34107) 11987 : cluster [DBG] pgmap v12493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:54 smithi115 bash[45495]: audit 2022-04-22T23:17:53.262338+0000 mon.a (mon.0) 16545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:54 smithi115 bash[46934]: cluster 2022-04-22T23:17:53.113824+0000 mgr.y (mgr.34107) 11987 : cluster [DBG] pgmap v12493: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:54 smithi115 bash[46934]: audit 2022-04-22T23:17:53.262338+0000 mon.a (mon.0) 16545 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:55.218 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:17:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:17:54.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:17:55.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:55 smithi115 bash[46934]: audit 2022-04-22T23:17:54.954832+0000 mon.a (mon.0) 16546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:55.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:55 smithi115 bash[45495]: audit 2022-04-22T23:17:54.954832+0000 mon.a (mon.0) 16546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:55 smithi121 bash[36902]: audit 2022-04-22T23:17:54.954832+0000 mon.a (mon.0) 16546 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:17:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:56 smithi121 bash[36902]: cluster 2022-04-22T23:17:55.114854+0000 mgr.y (mgr.34107) 11988 : cluster [DBG] pgmap v12494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:56 smithi121 bash[36902]: audit 2022-04-22T23:17:56.211393+0000 mon.a (mon.0) 16547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:56 smithi115 bash[45495]: cluster 2022-04-22T23:17:55.114854+0000 mgr.y (mgr.34107) 11988 : cluster [DBG] pgmap v12494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:56 smithi115 bash[45495]: audit 2022-04-22T23:17:56.211393+0000 mon.a (mon.0) 16547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:56 smithi115 bash[46934]: cluster 2022-04-22T23:17:55.114854+0000 mgr.y (mgr.34107) 11988 : cluster [DBG] pgmap v12494: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:56 smithi115 bash[46934]: audit 2022-04-22T23:17:56.211393+0000 mon.a (mon.0) 16547 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:57 smithi121 bash[36902]: cephadm 2022-04-22T23:17:56.201765+0000 mgr.y (mgr.34107) 11989 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T23:17:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:57 smithi121 bash[36902]: audit 2022-04-22T23:17:56.217778+0000 mon.a (mon.0) 16548 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:57 smithi121 bash[36902]: audit 2022-04-22T23:17:56.634969+0000 mon.a (mon.0) 16549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[45495]: cephadm 2022-04-22T23:17:56.201765+0000 mgr.y (mgr.34107) 11989 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T23:17:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[45495]: audit 2022-04-22T23:17:56.217778+0000 mon.a (mon.0) 16548 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[45495]: audit 2022-04-22T23:17:56.634969+0000 mon.a (mon.0) 16549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[46934]: cephadm 2022-04-22T23:17:56.201765+0000 mgr.y (mgr.34107) 11989 : cephadm [INF] Detected new or changed devices on smithi115 2022-04-22T23:17:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[46934]: audit 2022-04-22T23:17:56.217778+0000 mon.a (mon.0) 16548 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:17:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:57 smithi115 bash[46934]: audit 2022-04-22T23:17:56.634969+0000 mon.a (mon.0) 16549 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:17:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:58 smithi121 bash[36902]: cluster 2022-04-22T23:17:57.116025+0000 mgr.y (mgr.34107) 11990 : cluster [DBG] pgmap v12495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:58 smithi115 bash[45495]: cluster 2022-04-22T23:17:57.116025+0000 mgr.y (mgr.34107) 11990 : cluster [DBG] pgmap v12495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:58 smithi115 bash[46934]: cluster 2022-04-22T23:17:57.116025+0000 mgr.y (mgr.34107) 11990 : cluster [DBG] pgmap v12495: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:17:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:17:59 smithi121 bash[36902]: audit 2022-04-22T23:17:58.265207+0000 mon.a (mon.0) 16550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:17:59 smithi115 bash[45495]: audit 2022-04-22T23:17:58.265207+0000 mon.a (mon.0) 16550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:17:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:17:59 smithi115 bash[46934]: audit 2022-04-22T23:17:58.265207+0000 mon.a (mon.0) 16550 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:00 smithi121 bash[36902]: cluster 2022-04-22T23:17:59.117096+0000 mgr.y (mgr.34107) 11991 : cluster [DBG] pgmap v12496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:00 smithi121 bash[36902]: audit 2022-04-22T23:17:59.958527+0000 mon.a (mon.0) 16551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:00 smithi115 bash[45495]: cluster 2022-04-22T23:17:59.117096+0000 mgr.y (mgr.34107) 11991 : cluster [DBG] pgmap v12496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:00 smithi115 bash[45495]: audit 2022-04-22T23:17:59.958527+0000 mon.a (mon.0) 16551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:00 smithi115 bash[46934]: cluster 2022-04-22T23:17:59.117096+0000 mgr.y (mgr.34107) 11991 : cluster [DBG] pgmap v12496: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:00 smithi115 bash[46934]: audit 2022-04-22T23:17:59.958527+0000 mon.a (mon.0) 16551 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:02 smithi121 bash[36902]: cluster 2022-04-22T23:18:01.118165+0000 mgr.y (mgr.34107) 11992 : cluster [DBG] pgmap v12497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:02 smithi121 bash[36902]: audit 2022-04-22T23:18:01.638069+0000 mon.a (mon.0) 16552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:02.545 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:02 smithi115 bash[45495]: cluster 2022-04-22T23:18:01.118165+0000 mgr.y (mgr.34107) 11992 : cluster [DBG] pgmap v12497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:02.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:02 smithi115 bash[45495]: audit 2022-04-22T23:18:01.638069+0000 mon.a (mon.0) 16552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:02 smithi115 bash[46934]: cluster 2022-04-22T23:18:01.118165+0000 mgr.y (mgr.34107) 11992 : cluster [DBG] pgmap v12497: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:02.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:02 smithi115 bash[46934]: audit 2022-04-22T23:18:01.638069+0000 mon.a (mon.0) 16552 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:03.296 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:03.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:03 smithi121 bash[36902]: cluster 2022-04-22T23:18:03.119189+0000 mgr.y (mgr.34107) 11993 : cluster [DBG] pgmap v12498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:03 smithi121 bash[36902]: audit 2022-04-22T23:18:03.268256+0000 mon.a (mon.0) 16553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:03 smithi115 bash[45495]: cluster 2022-04-22T23:18:03.119189+0000 mgr.y (mgr.34107) 11993 : cluster [DBG] pgmap v12498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:03 smithi115 bash[45495]: audit 2022-04-22T23:18:03.268256+0000 mon.a (mon.0) 16553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:03 smithi115 bash[46934]: cluster 2022-04-22T23:18:03.119189+0000 mgr.y (mgr.34107) 11993 : cluster [DBG] pgmap v12498: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:03 smithi115 bash[46934]: audit 2022-04-22T23:18:03.268256+0000 mon.a (mon.0) 16553 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:05 smithi115 bash[46934]: audit 2022-04-22T23:18:04.962241+0000 mon.a (mon.0) 16554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:05.561 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:04.861Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:05.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:05 smithi115 bash[45495]: audit 2022-04-22T23:18:04.962241+0000 mon.a (mon.0) 16554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:05.562 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:05 smithi121 bash[36902]: audit 2022-04-22T23:18:04.962241+0000 mon.a (mon.0) 16554 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:06 smithi115 bash[45495]: cluster 2022-04-22T23:18:05.120287+0000 mgr.y (mgr.34107) 11994 : cluster [DBG] pgmap v12499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:06 smithi115 bash[46934]: cluster 2022-04-22T23:18:05.120287+0000 mgr.y (mgr.34107) 11994 : cluster [DBG] pgmap v12499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:06 smithi121 bash[36902]: cluster 2022-04-22T23:18:05.120287+0000 mgr.y (mgr.34107) 11994 : cluster [DBG] pgmap v12499: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:07 smithi115 bash[45495]: audit 2022-04-22T23:18:06.641004+0000 mon.a (mon.0) 16555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:07 smithi115 bash[46934]: audit 2022-04-22T23:18:06.641004+0000 mon.a (mon.0) 16555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:07 smithi121 bash[36902]: audit 2022-04-22T23:18:06.641004+0000 mon.a (mon.0) 16555 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:08 smithi115 bash[45495]: cluster 2022-04-22T23:18:07.121479+0000 mgr.y (mgr.34107) 11995 : cluster [DBG] pgmap v12500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:08 smithi115 bash[46934]: cluster 2022-04-22T23:18:07.121479+0000 mgr.y (mgr.34107) 11995 : cluster [DBG] pgmap v12500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:08 smithi121 bash[36902]: cluster 2022-04-22T23:18:07.121479+0000 mgr.y (mgr.34107) 11995 : cluster [DBG] pgmap v12500: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:09 smithi115 bash[45495]: audit 2022-04-22T23:18:08.271216+0000 mon.a (mon.0) 16556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:09 smithi115 bash[46934]: audit 2022-04-22T23:18:08.271216+0000 mon.a (mon.0) 16556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:09 smithi121 bash[36902]: audit 2022-04-22T23:18:08.271216+0000 mon.a (mon.0) 16556 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:10 smithi121 bash[36902]: cluster 2022-04-22T23:18:09.122925+0000 mgr.y (mgr.34107) 11996 : cluster [DBG] pgmap v12501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:10 smithi121 bash[36902]: audit 2022-04-22T23:18:09.965873+0000 mon.a (mon.0) 16557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:10 smithi115 bash[46934]: cluster 2022-04-22T23:18:09.122925+0000 mgr.y (mgr.34107) 11996 : cluster [DBG] pgmap v12501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:10 smithi115 bash[46934]: audit 2022-04-22T23:18:09.965873+0000 mon.a (mon.0) 16557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:10 smithi115 bash[45495]: cluster 2022-04-22T23:18:09.122925+0000 mgr.y (mgr.34107) 11996 : cluster [DBG] pgmap v12501: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:10 smithi115 bash[45495]: audit 2022-04-22T23:18:09.965873+0000 mon.a (mon.0) 16557 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:12 smithi121 bash[36902]: cluster 2022-04-22T23:18:11.124138+0000 mgr.y (mgr.34107) 11997 : cluster [DBG] pgmap v12502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:12 smithi121 bash[36902]: audit 2022-04-22T23:18:11.644081+0000 mon.a (mon.0) 16558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:12 smithi115 bash[46934]: cluster 2022-04-22T23:18:11.124138+0000 mgr.y (mgr.34107) 11997 : cluster [DBG] pgmap v12502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:12 smithi115 bash[46934]: audit 2022-04-22T23:18:11.644081+0000 mon.a (mon.0) 16558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:12 smithi115 bash[45495]: cluster 2022-04-22T23:18:11.124138+0000 mgr.y (mgr.34107) 11997 : cluster [DBG] pgmap v12502: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:12 smithi115 bash[45495]: audit 2022-04-22T23:18:11.644081+0000 mon.a (mon.0) 16558 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:13.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:13 smithi121 bash[36902]: cluster 2022-04-22T23:18:13.125299+0000 mgr.y (mgr.34107) 11998 : cluster [DBG] pgmap v12503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:13 smithi121 bash[36902]: audit 2022-04-22T23:18:13.274182+0000 mon.a (mon.0) 16559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:13 smithi115 bash[45495]: cluster 2022-04-22T23:18:13.125299+0000 mgr.y (mgr.34107) 11998 : cluster [DBG] pgmap v12503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:13 smithi115 bash[45495]: audit 2022-04-22T23:18:13.274182+0000 mon.a (mon.0) 16559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:13 smithi115 bash[46934]: cluster 2022-04-22T23:18:13.125299+0000 mgr.y (mgr.34107) 11998 : cluster [DBG] pgmap v12503: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:13 smithi115 bash[46934]: audit 2022-04-22T23:18:13.274182+0000 mon.a (mon.0) 16559 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:15 smithi115 bash[45495]: audit 2022-04-22T23:18:14.969627+0000 mon.a (mon.0) 16560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:14.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:15 smithi115 bash[46934]: audit 2022-04-22T23:18:14.969627+0000 mon.a (mon.0) 16560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:15.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:15 smithi121 bash[36902]: audit 2022-04-22T23:18:14.969627+0000 mon.a (mon.0) 16560 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:16 smithi115 bash[45495]: cluster 2022-04-22T23:18:15.126458+0000 mgr.y (mgr.34107) 11999 : cluster [DBG] pgmap v12504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:16 smithi115 bash[46934]: cluster 2022-04-22T23:18:15.126458+0000 mgr.y (mgr.34107) 11999 : cluster [DBG] pgmap v12504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:16 smithi121 bash[36902]: cluster 2022-04-22T23:18:15.126458+0000 mgr.y (mgr.34107) 11999 : cluster [DBG] pgmap v12504: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:17 smithi115 bash[46934]: audit 2022-04-22T23:18:16.647271+0000 mon.a (mon.0) 16561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:17 smithi115 bash[45495]: audit 2022-04-22T23:18:16.647271+0000 mon.a (mon.0) 16561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:17 smithi121 bash[36902]: audit 2022-04-22T23:18:16.647271+0000 mon.a (mon.0) 16561 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:18 smithi121 bash[36902]: cluster 2022-04-22T23:18:17.127433+0000 mgr.y (mgr.34107) 12000 : cluster [DBG] pgmap v12505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:18 smithi121 bash[36902]: audit 2022-04-22T23:18:17.324588+0000 mon.a (mon.0) 16562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:18:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:18 smithi121 bash[36902]: audit 2022-04-22T23:18:17.329358+0000 mon.a (mon.0) 16563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[46934]: cluster 2022-04-22T23:18:17.127433+0000 mgr.y (mgr.34107) 12000 : cluster [DBG] pgmap v12505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[46934]: audit 2022-04-22T23:18:17.324588+0000 mon.a (mon.0) 16562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[46934]: audit 2022-04-22T23:18:17.329358+0000 mon.a (mon.0) 16563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[45495]: cluster 2022-04-22T23:18:17.127433+0000 mgr.y (mgr.34107) 12000 : cluster [DBG] pgmap v12505: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[45495]: audit 2022-04-22T23:18:17.324588+0000 mon.a (mon.0) 16562 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:18:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:18 smithi115 bash[45495]: audit 2022-04-22T23:18:17.329358+0000 mon.a (mon.0) 16563 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:18:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:19 smithi121 bash[36902]: audit 2022-04-22T23:18:18.277371+0000 mon.a (mon.0) 16564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:19.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:19 smithi115 bash[46934]: audit 2022-04-22T23:18:18.277371+0000 mon.a (mon.0) 16564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:19 smithi115 bash[45495]: audit 2022-04-22T23:18:18.277371+0000 mon.a (mon.0) 16564 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:20 smithi121 bash[36902]: cluster 2022-04-22T23:18:19.128495+0000 mgr.y (mgr.34107) 12001 : cluster [DBG] pgmap v12506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:20 smithi121 bash[36902]: audit 2022-04-22T23:18:19.973268+0000 mon.a (mon.0) 16565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:20 smithi115 bash[45495]: cluster 2022-04-22T23:18:19.128495+0000 mgr.y (mgr.34107) 12001 : cluster [DBG] pgmap v12506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:20 smithi115 bash[45495]: audit 2022-04-22T23:18:19.973268+0000 mon.a (mon.0) 16565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:20 smithi115 bash[46934]: cluster 2022-04-22T23:18:19.128495+0000 mgr.y (mgr.34107) 12001 : cluster [DBG] pgmap v12506: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:20 smithi115 bash[46934]: audit 2022-04-22T23:18:19.973268+0000 mon.a (mon.0) 16565 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:22 smithi121 bash[36902]: cluster 2022-04-22T23:18:21.129508+0000 mgr.y (mgr.34107) 12002 : cluster [DBG] pgmap v12507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:22 smithi121 bash[36902]: audit 2022-04-22T23:18:21.650534+0000 mon.a (mon.0) 16566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:22 smithi115 bash[45495]: cluster 2022-04-22T23:18:21.129508+0000 mgr.y (mgr.34107) 12002 : cluster [DBG] pgmap v12507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:22.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:22 smithi115 bash[45495]: audit 2022-04-22T23:18:21.650534+0000 mon.a (mon.0) 16566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:22 smithi115 bash[46934]: cluster 2022-04-22T23:18:21.129508+0000 mgr.y (mgr.34107) 12002 : cluster [DBG] pgmap v12507: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:22 smithi115 bash[46934]: audit 2022-04-22T23:18:21.650534+0000 mon.a (mon.0) 16566 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:22.599 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:18:22.940 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:18:22.940 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 36s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:18:22.940 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 101s ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 36s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 101s ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 101s ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 36s ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 36s ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 101s ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 36s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 36s ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 101s ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 36s ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:18:22.941 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 36s ago 5h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 36s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 36s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 101s ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 101s ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 101s ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 101s ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:18:22.942 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 101s ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:18:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:23.003Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:23.350 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:18:23.351 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:23 smithi121 bash[36902]: audit 2022-04-22T23:18:22.580481+0000 mgr.y (mgr.34107) 12003 : audit [DBG] from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:23 smithi121 bash[36902]: audit 2022-04-22T23:18:22.918650+0000 mgr.y (mgr.34107) 12004 : audit [DBG] from='client.57709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:23 smithi121 bash[36902]: cluster 2022-04-22T23:18:23.130534+0000 mgr.y (mgr.34107) 12005 : cluster [DBG] pgmap v12508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:23 smithi121 bash[36902]: audit 2022-04-22T23:18:23.280417+0000 mon.a (mon.0) 16567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:23 smithi121 bash[36902]: audit 2022-04-22T23:18:23.346210+0000 mon.c (mon.1) 406 : audit [DBG] from='client.? 172.21.15.115:0/3154453625' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[45495]: audit 2022-04-22T23:18:22.580481+0000 mgr.y (mgr.34107) 12003 : audit [DBG] from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[45495]: audit 2022-04-22T23:18:22.918650+0000 mgr.y (mgr.34107) 12004 : audit [DBG] from='client.57709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[45495]: cluster 2022-04-22T23:18:23.130534+0000 mgr.y (mgr.34107) 12005 : cluster [DBG] pgmap v12508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[45495]: audit 2022-04-22T23:18:23.280417+0000 mon.a (mon.0) 16567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[45495]: audit 2022-04-22T23:18:23.346210+0000 mon.c (mon.1) 406 : audit [DBG] from='client.? 172.21.15.115:0/3154453625' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[46934]: audit 2022-04-22T23:18:22.580481+0000 mgr.y (mgr.34107) 12003 : audit [DBG] from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[46934]: audit 2022-04-22T23:18:22.918650+0000 mgr.y (mgr.34107) 12004 : audit [DBG] from='client.57709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[46934]: cluster 2022-04-22T23:18:23.130534+0000 mgr.y (mgr.34107) 12005 : cluster [DBG] pgmap v12508: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[46934]: audit 2022-04-22T23:18:23.280417+0000 mon.a (mon.0) 16567 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:24.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:23 smithi115 bash[46934]: audit 2022-04-22T23:18:23.346210+0000 mon.c (mon.1) 406 : audit [DBG] from='client.? 172.21.15.115:0/3154453625' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:25 smithi115 bash[45495]: audit 2022-04-22T23:18:24.976856+0000 mon.a (mon.0) 16568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:24.863Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:25 smithi115 bash[46934]: audit 2022-04-22T23:18:24.976856+0000 mon.a (mon.0) 16568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:25 smithi121 bash[36902]: audit 2022-04-22T23:18:24.976856+0000 mon.a (mon.0) 16568 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:26 smithi115 bash[45495]: cluster 2022-04-22T23:18:25.131427+0000 mgr.y (mgr.34107) 12006 : cluster [DBG] pgmap v12509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:26 smithi115 bash[46934]: cluster 2022-04-22T23:18:25.131427+0000 mgr.y (mgr.34107) 12006 : cluster [DBG] pgmap v12509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:26 smithi121 bash[36902]: cluster 2022-04-22T23:18:25.131427+0000 mgr.y (mgr.34107) 12006 : cluster [DBG] pgmap v12509: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:27 smithi121 bash[36902]: audit 2022-04-22T23:18:26.653694+0000 mon.a (mon.0) 16569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:27 smithi115 bash[45495]: audit 2022-04-22T23:18:26.653694+0000 mon.a (mon.0) 16569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:27 smithi115 bash[46934]: audit 2022-04-22T23:18:26.653694+0000 mon.a (mon.0) 16569 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:28 smithi121 bash[36902]: cluster 2022-04-22T23:18:27.132507+0000 mgr.y (mgr.34107) 12007 : cluster [DBG] pgmap v12510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:28 smithi115 bash[45495]: cluster 2022-04-22T23:18:27.132507+0000 mgr.y (mgr.34107) 12007 : cluster [DBG] pgmap v12510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:28 smithi115 bash[46934]: cluster 2022-04-22T23:18:27.132507+0000 mgr.y (mgr.34107) 12007 : cluster [DBG] pgmap v12510: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:29 smithi121 bash[36902]: audit 2022-04-22T23:18:28.283404+0000 mon.a (mon.0) 16570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:29 smithi115 bash[45495]: audit 2022-04-22T23:18:28.283404+0000 mon.a (mon.0) 16570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:29 smithi115 bash[46934]: audit 2022-04-22T23:18:28.283404+0000 mon.a (mon.0) 16570 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:30 smithi121 bash[36902]: cluster 2022-04-22T23:18:29.133732+0000 mgr.y (mgr.34107) 12008 : cluster [DBG] pgmap v12511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:30 smithi121 bash[36902]: audit 2022-04-22T23:18:29.980674+0000 mon.a (mon.0) 16571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:30 smithi115 bash[45495]: cluster 2022-04-22T23:18:29.133732+0000 mgr.y (mgr.34107) 12008 : cluster [DBG] pgmap v12511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:30 smithi115 bash[45495]: audit 2022-04-22T23:18:29.980674+0000 mon.a (mon.0) 16571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:30 smithi115 bash[46934]: cluster 2022-04-22T23:18:29.133732+0000 mgr.y (mgr.34107) 12008 : cluster [DBG] pgmap v12511: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:30 smithi115 bash[46934]: audit 2022-04-22T23:18:29.980674+0000 mon.a (mon.0) 16571 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:32 smithi121 bash[36902]: cluster 2022-04-22T23:18:31.134846+0000 mgr.y (mgr.34107) 12009 : cluster [DBG] pgmap v12512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:32 smithi121 bash[36902]: audit 2022-04-22T23:18:31.656548+0000 mon.a (mon.0) 16572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:32 smithi115 bash[46934]: cluster 2022-04-22T23:18:31.134846+0000 mgr.y (mgr.34107) 12009 : cluster [DBG] pgmap v12512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:32 smithi115 bash[46934]: audit 2022-04-22T23:18:31.656548+0000 mon.a (mon.0) 16572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:32 smithi115 bash[45495]: cluster 2022-04-22T23:18:31.134846+0000 mgr.y (mgr.34107) 12009 : cluster [DBG] pgmap v12512: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:32 smithi115 bash[45495]: audit 2022-04-22T23:18:31.656548+0000 mon.a (mon.0) 16572 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:33.007Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:33 smithi121 bash[36902]: cluster 2022-04-22T23:18:33.135711+0000 mgr.y (mgr.34107) 12010 : cluster [DBG] pgmap v12513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:33 smithi121 bash[36902]: audit 2022-04-22T23:18:33.286504+0000 mon.a (mon.0) 16573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:33 smithi115 bash[45495]: cluster 2022-04-22T23:18:33.135711+0000 mgr.y (mgr.34107) 12010 : cluster [DBG] pgmap v12513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:33 smithi115 bash[45495]: audit 2022-04-22T23:18:33.286504+0000 mon.a (mon.0) 16573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:33 smithi115 bash[46934]: cluster 2022-04-22T23:18:33.135711+0000 mgr.y (mgr.34107) 12010 : cluster [DBG] pgmap v12513: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:33 smithi115 bash[46934]: audit 2022-04-22T23:18:33.286504+0000 mon.a (mon.0) 16573 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:35 smithi115 bash[45495]: audit 2022-04-22T23:18:34.984239+0000 mon.a (mon.0) 16574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:34.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:35 smithi115 bash[46934]: audit 2022-04-22T23:18:34.984239+0000 mon.a (mon.0) 16574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:35 smithi121 bash[36902]: audit 2022-04-22T23:18:34.984239+0000 mon.a (mon.0) 16574 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:36 smithi121 bash[36902]: cluster 2022-04-22T23:18:35.136798+0000 mgr.y (mgr.34107) 12011 : cluster [DBG] pgmap v12514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:36 smithi115 bash[45495]: cluster 2022-04-22T23:18:35.136798+0000 mgr.y (mgr.34107) 12011 : cluster [DBG] pgmap v12514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:36 smithi115 bash[46934]: cluster 2022-04-22T23:18:35.136798+0000 mgr.y (mgr.34107) 12011 : cluster [DBG] pgmap v12514: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:37 smithi121 bash[36902]: audit 2022-04-22T23:18:36.659451+0000 mon.a (mon.0) 16575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:37 smithi115 bash[46934]: audit 2022-04-22T23:18:36.659451+0000 mon.a (mon.0) 16575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:37.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:37 smithi115 bash[45495]: audit 2022-04-22T23:18:36.659451+0000 mon.a (mon.0) 16575 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:38 smithi121 bash[36902]: cluster 2022-04-22T23:18:37.137990+0000 mgr.y (mgr.34107) 12012 : cluster [DBG] pgmap v12515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:38 smithi115 bash[46934]: cluster 2022-04-22T23:18:37.137990+0000 mgr.y (mgr.34107) 12012 : cluster [DBG] pgmap v12515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:38 smithi115 bash[45495]: cluster 2022-04-22T23:18:37.137990+0000 mgr.y (mgr.34107) 12012 : cluster [DBG] pgmap v12515: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:39 smithi121 bash[36902]: audit 2022-04-22T23:18:38.289956+0000 mon.a (mon.0) 16576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:39 smithi115 bash[46934]: audit 2022-04-22T23:18:38.289956+0000 mon.a (mon.0) 16576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:39 smithi115 bash[45495]: audit 2022-04-22T23:18:38.289956+0000 mon.a (mon.0) 16576 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:40 smithi121 bash[36902]: cluster 2022-04-22T23:18:39.139074+0000 mgr.y (mgr.34107) 12013 : cluster [DBG] pgmap v12516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:40 smithi121 bash[36902]: audit 2022-04-22T23:18:39.987623+0000 mon.a (mon.0) 16577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:40 smithi115 bash[46934]: cluster 2022-04-22T23:18:39.139074+0000 mgr.y (mgr.34107) 12013 : cluster [DBG] pgmap v12516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:40 smithi115 bash[46934]: audit 2022-04-22T23:18:39.987623+0000 mon.a (mon.0) 16577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:40 smithi115 bash[45495]: cluster 2022-04-22T23:18:39.139074+0000 mgr.y (mgr.34107) 12013 : cluster [DBG] pgmap v12516: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:40 smithi115 bash[45495]: audit 2022-04-22T23:18:39.987623+0000 mon.a (mon.0) 16577 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:42 smithi121 bash[36902]: cluster 2022-04-22T23:18:41.140064+0000 mgr.y (mgr.34107) 12014 : cluster [DBG] pgmap v12517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:42 smithi121 bash[36902]: audit 2022-04-22T23:18:41.662429+0000 mon.a (mon.0) 16578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:42 smithi115 bash[45495]: cluster 2022-04-22T23:18:41.140064+0000 mgr.y (mgr.34107) 12014 : cluster [DBG] pgmap v12517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:42 smithi115 bash[45495]: audit 2022-04-22T23:18:41.662429+0000 mon.a (mon.0) 16578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:42 smithi115 bash[46934]: cluster 2022-04-22T23:18:41.140064+0000 mgr.y (mgr.34107) 12014 : cluster [DBG] pgmap v12517: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:42 smithi115 bash[46934]: audit 2022-04-22T23:18:41.662429+0000 mon.a (mon.0) 16578 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:43.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:43 smithi121 bash[36902]: cluster 2022-04-22T23:18:43.141049+0000 mgr.y (mgr.34107) 12015 : cluster [DBG] pgmap v12518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:43 smithi121 bash[36902]: audit 2022-04-22T23:18:43.293003+0000 mon.a (mon.0) 16579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:43 smithi115 bash[45495]: cluster 2022-04-22T23:18:43.141049+0000 mgr.y (mgr.34107) 12015 : cluster [DBG] pgmap v12518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:43 smithi115 bash[45495]: audit 2022-04-22T23:18:43.293003+0000 mon.a (mon.0) 16579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:43 smithi115 bash[46934]: cluster 2022-04-22T23:18:43.141049+0000 mgr.y (mgr.34107) 12015 : cluster [DBG] pgmap v12518: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:43 smithi115 bash[46934]: audit 2022-04-22T23:18:43.293003+0000 mon.a (mon.0) 16579 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:45 smithi115 bash[45495]: audit 2022-04-22T23:18:44.991413+0000 mon.a (mon.0) 16580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:44.864Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:45 smithi115 bash[46934]: audit 2022-04-22T23:18:44.991413+0000 mon.a (mon.0) 16580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:45 smithi121 bash[36902]: audit 2022-04-22T23:18:44.991413+0000 mon.a (mon.0) 16580 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:46 smithi121 bash[36902]: cluster 2022-04-22T23:18:45.142141+0000 mgr.y (mgr.34107) 12016 : cluster [DBG] pgmap v12519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:46 smithi115 bash[45495]: cluster 2022-04-22T23:18:45.142141+0000 mgr.y (mgr.34107) 12016 : cluster [DBG] pgmap v12519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:46 smithi115 bash[46934]: cluster 2022-04-22T23:18:45.142141+0000 mgr.y (mgr.34107) 12016 : cluster [DBG] pgmap v12519: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:47 smithi121 bash[36902]: audit 2022-04-22T23:18:46.665458+0000 mon.a (mon.0) 16581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:47 smithi115 bash[45495]: audit 2022-04-22T23:18:46.665458+0000 mon.a (mon.0) 16581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:47 smithi115 bash[46934]: audit 2022-04-22T23:18:46.665458+0000 mon.a (mon.0) 16581 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:48 smithi121 bash[36902]: cluster 2022-04-22T23:18:47.143303+0000 mgr.y (mgr.34107) 12017 : cluster [DBG] pgmap v12520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:48 smithi115 bash[45495]: cluster 2022-04-22T23:18:47.143303+0000 mgr.y (mgr.34107) 12017 : cluster [DBG] pgmap v12520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:48 smithi115 bash[46934]: cluster 2022-04-22T23:18:47.143303+0000 mgr.y (mgr.34107) 12017 : cluster [DBG] pgmap v12520: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:49 smithi121 bash[36902]: audit 2022-04-22T23:18:48.295917+0000 mon.a (mon.0) 16582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:49 smithi115 bash[45495]: audit 2022-04-22T23:18:48.295917+0000 mon.a (mon.0) 16582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:49 smithi115 bash[46934]: audit 2022-04-22T23:18:48.295917+0000 mon.a (mon.0) 16582 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:50 smithi121 bash[36902]: cluster 2022-04-22T23:18:49.144442+0000 mgr.y (mgr.34107) 12018 : cluster [DBG] pgmap v12521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:50 smithi121 bash[36902]: audit 2022-04-22T23:18:49.994949+0000 mon.a (mon.0) 16583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:50 smithi115 bash[45495]: cluster 2022-04-22T23:18:49.144442+0000 mgr.y (mgr.34107) 12018 : cluster [DBG] pgmap v12521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:50 smithi115 bash[45495]: audit 2022-04-22T23:18:49.994949+0000 mon.a (mon.0) 16583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:50 smithi115 bash[46934]: cluster 2022-04-22T23:18:49.144442+0000 mgr.y (mgr.34107) 12018 : cluster [DBG] pgmap v12521: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:50 smithi115 bash[46934]: audit 2022-04-22T23:18:49.994949+0000 mon.a (mon.0) 16583 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:52 smithi121 bash[36902]: cluster 2022-04-22T23:18:51.145536+0000 mgr.y (mgr.34107) 12019 : cluster [DBG] pgmap v12522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:52 smithi121 bash[36902]: audit 2022-04-22T23:18:51.668444+0000 mon.a (mon.0) 16584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:52 smithi115 bash[45495]: cluster 2022-04-22T23:18:51.145536+0000 mgr.y (mgr.34107) 12019 : cluster [DBG] pgmap v12522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:52 smithi115 bash[45495]: audit 2022-04-22T23:18:51.668444+0000 mon.a (mon.0) 16584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:52 smithi115 bash[46934]: cluster 2022-04-22T23:18:51.145536+0000 mgr.y (mgr.34107) 12019 : cluster [DBG] pgmap v12522: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:52 smithi115 bash[46934]: audit 2022-04-22T23:18:51.668444+0000 mon.a (mon.0) 16584 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:53.006Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:53.732 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:18:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:53 smithi121 bash[36902]: cluster 2022-04-22T23:18:53.146568+0000 mgr.y (mgr.34107) 12020 : cluster [DBG] pgmap v12523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:53 smithi121 bash[36902]: audit 2022-04-22T23:18:53.298864+0000 mon.a (mon.0) 16585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:54.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:53 smithi115 bash[46934]: cluster 2022-04-22T23:18:53.146568+0000 mgr.y (mgr.34107) 12020 : cluster [DBG] pgmap v12523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:54.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:53 smithi115 bash[46934]: audit 2022-04-22T23:18:53.298864+0000 mon.a (mon.0) 16585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:53 smithi115 bash[45495]: cluster 2022-04-22T23:18:53.146568+0000 mgr.y (mgr.34107) 12020 : cluster [DBG] pgmap v12523: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:53 smithi115 bash[45495]: audit 2022-04-22T23:18:53.298864+0000 mon.a (mon.0) 16585 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 67s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 67s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:18:54.080 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 67s ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 67s ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 67s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 67s ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 67s ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 67s ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 67s ago 5h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:18:54.081 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 67s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:18:54.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:18:54.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:18:54.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:18:54.082 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:18:54.082 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:18:54.491 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:18:54.491 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:18:54.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:18:54.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:54.492 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:18:54.492 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:18:54.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:54.493 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:18:54.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:18:54.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:54.493 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:18:54.493 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:18:54.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:18:54.494 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:18:54.494 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:18:54.494 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:18:54.495 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:18:54.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[45495]: audit 2022-04-22T23:18:53.713327+0000 mgr.y (mgr.34107) 12021 : audit [DBG] from='client.57721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:54.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[45495]: audit 2022-04-22T23:18:54.056379+0000 mgr.y (mgr.34107) 12022 : audit [DBG] from='client.57727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:54.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[45495]: audit 2022-04-22T23:18:54.486956+0000 mon.a (mon.0) 16586 : audit [DBG] from='client.? 172.21.15.115:0/4206171986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[46934]: audit 2022-04-22T23:18:53.713327+0000 mgr.y (mgr.34107) 12021 : audit [DBG] from='client.57721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[46934]: audit 2022-04-22T23:18:54.056379+0000 mgr.y (mgr.34107) 12022 : audit [DBG] from='client.57727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[46934]: audit 2022-04-22T23:18:54.486956+0000 mon.a (mon.0) 16586 : audit [DBG] from='client.? 172.21.15.115:0/4206171986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:54 smithi121 bash[36902]: audit 2022-04-22T23:18:53.713327+0000 mgr.y (mgr.34107) 12021 : audit [DBG] from='client.57721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:54 smithi121 bash[36902]: audit 2022-04-22T23:18:54.056379+0000 mgr.y (mgr.34107) 12022 : audit [DBG] from='client.57727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:18:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:54 smithi121 bash[36902]: audit 2022-04-22T23:18:54.486956+0000 mon.a (mon.0) 16586 : audit [DBG] from='client.? 172.21.15.115:0/4206171986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:18:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:18:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:18:54.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:18:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:55 smithi121 bash[36902]: audit 2022-04-22T23:18:54.998922+0000 mon.a (mon.0) 16587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:55 smithi121 bash[36902]: cluster 2022-04-22T23:18:55.147659+0000 mgr.y (mgr.34107) 12023 : cluster [DBG] pgmap v12524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:55 smithi115 bash[45495]: audit 2022-04-22T23:18:54.998922+0000 mon.a (mon.0) 16587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:55 smithi115 bash[45495]: cluster 2022-04-22T23:18:55.147659+0000 mgr.y (mgr.34107) 12023 : cluster [DBG] pgmap v12524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:55 smithi115 bash[46934]: audit 2022-04-22T23:18:54.998922+0000 mon.a (mon.0) 16587 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:18:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:55 smithi115 bash[46934]: cluster 2022-04-22T23:18:55.147659+0000 mgr.y (mgr.34107) 12023 : cluster [DBG] pgmap v12524: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:56 smithi121 bash[36902]: audit 2022-04-22T23:18:56.227923+0000 mon.a (mon.0) 16588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:18:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:56 smithi121 bash[36902]: audit 2022-04-22T23:18:56.230698+0000 mon.a (mon.0) 16589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:18:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:56 smithi121 bash[36902]: audit 2022-04-22T23:18:56.232268+0000 mon.a (mon.0) 16590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:18:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[45495]: audit 2022-04-22T23:18:56.227923+0000 mon.a (mon.0) 16588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:18:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[45495]: audit 2022-04-22T23:18:56.230698+0000 mon.a (mon.0) 16589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:18:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[45495]: audit 2022-04-22T23:18:56.232268+0000 mon.a (mon.0) 16590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:18:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[46934]: audit 2022-04-22T23:18:56.227923+0000 mon.a (mon.0) 16588 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:18:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[46934]: audit 2022-04-22T23:18:56.230698+0000 mon.a (mon.0) 16589 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:18:57.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:56 smithi115 bash[46934]: audit 2022-04-22T23:18:56.232268+0000 mon.a (mon.0) 16590 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:18:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:57 smithi121 bash[36902]: audit 2022-04-22T23:18:56.597628+0000 mon.a (mon.0) 16591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:18:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:57 smithi121 bash[36902]: audit 2022-04-22T23:18:56.671443+0000 mon.a (mon.0) 16592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:57 smithi121 bash[36902]: cluster 2022-04-22T23:18:57.148732+0000 mgr.y (mgr.34107) 12024 : cluster [DBG] pgmap v12525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[45495]: audit 2022-04-22T23:18:56.597628+0000 mon.a (mon.0) 16591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[45495]: audit 2022-04-22T23:18:56.671443+0000 mon.a (mon.0) 16592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[45495]: cluster 2022-04-22T23:18:57.148732+0000 mgr.y (mgr.34107) 12024 : cluster [DBG] pgmap v12525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[46934]: audit 2022-04-22T23:18:56.597628+0000 mon.a (mon.0) 16591 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[46934]: audit 2022-04-22T23:18:56.671443+0000 mon.a (mon.0) 16592 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:18:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:57 smithi115 bash[46934]: cluster 2022-04-22T23:18:57.148732+0000 mgr.y (mgr.34107) 12024 : cluster [DBG] pgmap v12525: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:18:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:58 smithi121 bash[36902]: audit 2022-04-22T23:18:58.301940+0000 mon.a (mon.0) 16593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:58 smithi115 bash[45495]: audit 2022-04-22T23:18:58.301940+0000 mon.a (mon.0) 16593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:18:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:58 smithi115 bash[46934]: audit 2022-04-22T23:18:58.301940+0000 mon.a (mon.0) 16593 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:18:59 smithi121 bash[36902]: cluster 2022-04-22T23:18:59.149794+0000 mgr.y (mgr.34107) 12025 : cluster [DBG] pgmap v12526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:00.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:18:59 smithi115 bash[45495]: cluster 2022-04-22T23:18:59.149794+0000 mgr.y (mgr.34107) 12025 : cluster [DBG] pgmap v12526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:18:59 smithi115 bash[46934]: cluster 2022-04-22T23:18:59.149794+0000 mgr.y (mgr.34107) 12025 : cluster [DBG] pgmap v12526: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:01.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:00 smithi121 bash[36902]: audit 2022-04-22T23:19:00.002717+0000 mon.a (mon.0) 16594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:01.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:00 smithi115 bash[45495]: audit 2022-04-22T23:19:00.002717+0000 mon.a (mon.0) 16594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:01.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:00 smithi115 bash[46934]: audit 2022-04-22T23:19:00.002717+0000 mon.a (mon.0) 16594 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:02.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:01 smithi121 bash[36902]: cluster 2022-04-22T23:19:01.150881+0000 mgr.y (mgr.34107) 12026 : cluster [DBG] pgmap v12527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:02.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:01 smithi115 bash[45495]: cluster 2022-04-22T23:19:01.150881+0000 mgr.y (mgr.34107) 12026 : cluster [DBG] pgmap v12527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:02.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:01 smithi115 bash[46934]: cluster 2022-04-22T23:19:01.150881+0000 mgr.y (mgr.34107) 12026 : cluster [DBG] pgmap v12527: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:03.013 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:02 smithi115 bash[45495]: audit 2022-04-22T23:19:01.674331+0000 mon.a (mon.0) 16595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:03.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:02 smithi115 bash[46934]: audit 2022-04-22T23:19:01.674331+0000 mon.a (mon.0) 16595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:03.014 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:02 smithi121 bash[36902]: audit 2022-04-22T23:19:01.674331+0000 mon.a (mon.0) 16595 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:03.008Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:03 smithi121 bash[36902]: cluster 2022-04-22T23:19:03.152041+0000 mgr.y (mgr.34107) 12027 : cluster [DBG] pgmap v12528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:03 smithi121 bash[36902]: audit 2022-04-22T23:19:03.304939+0000 mon.a (mon.0) 16596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:03 smithi115 bash[45495]: cluster 2022-04-22T23:19:03.152041+0000 mgr.y (mgr.34107) 12027 : cluster [DBG] pgmap v12528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:03 smithi115 bash[45495]: audit 2022-04-22T23:19:03.304939+0000 mon.a (mon.0) 16596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:03 smithi115 bash[46934]: cluster 2022-04-22T23:19:03.152041+0000 mgr.y (mgr.34107) 12027 : cluster [DBG] pgmap v12528: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:03 smithi115 bash[46934]: audit 2022-04-22T23:19:03.304939+0000 mon.a (mon.0) 16596 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:05 smithi115 bash[46934]: audit 2022-04-22T23:19:05.006828+0000 mon.a (mon.0) 16597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:04.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:05 smithi115 bash[45495]: audit 2022-04-22T23:19:05.006828+0000 mon.a (mon.0) 16597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:05 smithi121 bash[36902]: audit 2022-04-22T23:19:05.006828+0000 mon.a (mon.0) 16597 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:06 smithi121 bash[36902]: cluster 2022-04-22T23:19:05.153027+0000 mgr.y (mgr.34107) 12028 : cluster [DBG] pgmap v12529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:06 smithi115 bash[45495]: cluster 2022-04-22T23:19:05.153027+0000 mgr.y (mgr.34107) 12028 : cluster [DBG] pgmap v12529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:06 smithi115 bash[46934]: cluster 2022-04-22T23:19:05.153027+0000 mgr.y (mgr.34107) 12028 : cluster [DBG] pgmap v12529: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:07 smithi121 bash[36902]: audit 2022-04-22T23:19:06.677295+0000 mon.a (mon.0) 16598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:07 smithi115 bash[45495]: audit 2022-04-22T23:19:06.677295+0000 mon.a (mon.0) 16598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:07 smithi115 bash[46934]: audit 2022-04-22T23:19:06.677295+0000 mon.a (mon.0) 16598 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:08 smithi121 bash[36902]: cluster 2022-04-22T23:19:07.154209+0000 mgr.y (mgr.34107) 12029 : cluster [DBG] pgmap v12530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:08 smithi115 bash[45495]: cluster 2022-04-22T23:19:07.154209+0000 mgr.y (mgr.34107) 12029 : cluster [DBG] pgmap v12530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:08 smithi115 bash[46934]: cluster 2022-04-22T23:19:07.154209+0000 mgr.y (mgr.34107) 12029 : cluster [DBG] pgmap v12530: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:09 smithi121 bash[36902]: audit 2022-04-22T23:19:08.308104+0000 mon.a (mon.0) 16599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:09 smithi115 bash[46934]: audit 2022-04-22T23:19:08.308104+0000 mon.a (mon.0) 16599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:09 smithi115 bash[45495]: audit 2022-04-22T23:19:08.308104+0000 mon.a (mon.0) 16599 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:10 smithi121 bash[36902]: cluster 2022-04-22T23:19:09.155341+0000 mgr.y (mgr.34107) 12030 : cluster [DBG] pgmap v12531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:10 smithi121 bash[36902]: audit 2022-04-22T23:19:10.010216+0000 mon.a (mon.0) 16600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:10.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:10 smithi115 bash[45495]: cluster 2022-04-22T23:19:09.155341+0000 mgr.y (mgr.34107) 12030 : cluster [DBG] pgmap v12531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:10 smithi115 bash[45495]: audit 2022-04-22T23:19:10.010216+0000 mon.a (mon.0) 16600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:10 smithi115 bash[46934]: cluster 2022-04-22T23:19:09.155341+0000 mgr.y (mgr.34107) 12030 : cluster [DBG] pgmap v12531: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:10 smithi115 bash[46934]: audit 2022-04-22T23:19:10.010216+0000 mon.a (mon.0) 16600 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:12 smithi121 bash[36902]: cluster 2022-04-22T23:19:11.156428+0000 mgr.y (mgr.34107) 12031 : cluster [DBG] pgmap v12532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:12 smithi121 bash[36902]: audit 2022-04-22T23:19:11.680451+0000 mon.a (mon.0) 16601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:12 smithi115 bash[45495]: cluster 2022-04-22T23:19:11.156428+0000 mgr.y (mgr.34107) 12031 : cluster [DBG] pgmap v12532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:12 smithi115 bash[45495]: audit 2022-04-22T23:19:11.680451+0000 mon.a (mon.0) 16601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:12 smithi115 bash[46934]: cluster 2022-04-22T23:19:11.156428+0000 mgr.y (mgr.34107) 12031 : cluster [DBG] pgmap v12532: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:12 smithi115 bash[46934]: audit 2022-04-22T23:19:11.680451+0000 mon.a (mon.0) 16601 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:13.009Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:13 smithi121 bash[36902]: cluster 2022-04-22T23:19:13.157522+0000 mgr.y (mgr.34107) 12032 : cluster [DBG] pgmap v12533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:13 smithi121 bash[36902]: audit 2022-04-22T23:19:13.311620+0000 mon.a (mon.0) 16602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:13 smithi115 bash[45495]: cluster 2022-04-22T23:19:13.157522+0000 mgr.y (mgr.34107) 12032 : cluster [DBG] pgmap v12533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:13 smithi115 bash[45495]: audit 2022-04-22T23:19:13.311620+0000 mon.a (mon.0) 16602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:13 smithi115 bash[46934]: cluster 2022-04-22T23:19:13.157522+0000 mgr.y (mgr.34107) 12032 : cluster [DBG] pgmap v12533: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:13 smithi115 bash[46934]: audit 2022-04-22T23:19:13.311620+0000 mon.a (mon.0) 16602 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:15 smithi115 bash[45495]: audit 2022-04-22T23:19:15.013706+0000 mon.a (mon.0) 16603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:14.866Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:15 smithi115 bash[46934]: audit 2022-04-22T23:19:15.013706+0000 mon.a (mon.0) 16603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:15 smithi121 bash[36902]: audit 2022-04-22T23:19:15.013706+0000 mon.a (mon.0) 16603 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:16 smithi121 bash[36902]: cluster 2022-04-22T23:19:15.158629+0000 mgr.y (mgr.34107) 12033 : cluster [DBG] pgmap v12534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:16 smithi115 bash[45495]: cluster 2022-04-22T23:19:15.158629+0000 mgr.y (mgr.34107) 12033 : cluster [DBG] pgmap v12534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:16 smithi115 bash[46934]: cluster 2022-04-22T23:19:15.158629+0000 mgr.y (mgr.34107) 12033 : cluster [DBG] pgmap v12534: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:17 smithi121 bash[36902]: audit 2022-04-22T23:19:16.683692+0000 mon.a (mon.0) 16604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:17 smithi115 bash[45495]: audit 2022-04-22T23:19:16.683692+0000 mon.a (mon.0) 16604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:17 smithi115 bash[46934]: audit 2022-04-22T23:19:16.683692+0000 mon.a (mon.0) 16604 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:18 smithi121 bash[36902]: cluster 2022-04-22T23:19:17.159882+0000 mgr.y (mgr.34107) 12034 : cluster [DBG] pgmap v12535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:18 smithi121 bash[36902]: audit 2022-04-22T23:19:17.334848+0000 mon.a (mon.0) 16605 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:19:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:18 smithi121 bash[36902]: audit 2022-04-22T23:19:17.340919+0000 mon.a (mon.0) 16606 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:19:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[45495]: cluster 2022-04-22T23:19:17.159882+0000 mgr.y (mgr.34107) 12034 : cluster [DBG] pgmap v12535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[45495]: audit 2022-04-22T23:19:17.334848+0000 mon.a (mon.0) 16605 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:19:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[45495]: audit 2022-04-22T23:19:17.340919+0000 mon.a (mon.0) 16606 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:19:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[46934]: cluster 2022-04-22T23:19:17.159882+0000 mgr.y (mgr.34107) 12034 : cluster [DBG] pgmap v12535: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[46934]: audit 2022-04-22T23:19:17.334848+0000 mon.a (mon.0) 16605 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:19:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:18 smithi115 bash[46934]: audit 2022-04-22T23:19:17.340919+0000 mon.a (mon.0) 16606 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:19:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:19 smithi121 bash[36902]: audit 2022-04-22T23:19:18.314650+0000 mon.a (mon.0) 16607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:19 smithi115 bash[45495]: audit 2022-04-22T23:19:18.314650+0000 mon.a (mon.0) 16607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:19 smithi115 bash[46934]: audit 2022-04-22T23:19:18.314650+0000 mon.a (mon.0) 16607 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:20 smithi121 bash[36902]: cluster 2022-04-22T23:19:19.160949+0000 mgr.y (mgr.34107) 12035 : cluster [DBG] pgmap v12536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:20 smithi121 bash[36902]: audit 2022-04-22T23:19:20.017350+0000 mon.a (mon.0) 16608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:20 smithi115 bash[45495]: cluster 2022-04-22T23:19:19.160949+0000 mgr.y (mgr.34107) 12035 : cluster [DBG] pgmap v12536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:20 smithi115 bash[45495]: audit 2022-04-22T23:19:20.017350+0000 mon.a (mon.0) 16608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:20 smithi115 bash[46934]: cluster 2022-04-22T23:19:19.160949+0000 mgr.y (mgr.34107) 12035 : cluster [DBG] pgmap v12536: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:20.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:20 smithi115 bash[46934]: audit 2022-04-22T23:19:20.017350+0000 mon.a (mon.0) 16608 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:22 smithi121 bash[36902]: cluster 2022-04-22T23:19:21.161971+0000 mgr.y (mgr.34107) 12036 : cluster [DBG] pgmap v12537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:22 smithi121 bash[36902]: audit 2022-04-22T23:19:21.686685+0000 mon.a (mon.0) 16609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:22 smithi115 bash[45495]: cluster 2022-04-22T23:19:21.161971+0000 mgr.y (mgr.34107) 12036 : cluster [DBG] pgmap v12537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:22 smithi115 bash[45495]: audit 2022-04-22T23:19:21.686685+0000 mon.a (mon.0) 16609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:22 smithi115 bash[46934]: cluster 2022-04-22T23:19:21.161971+0000 mgr.y (mgr.34107) 12036 : cluster [DBG] pgmap v12537: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:22 smithi115 bash[46934]: audit 2022-04-22T23:19:21.686685+0000 mon.a (mon.0) 16609 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:23.010Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:23 smithi121 bash[36902]: cluster 2022-04-22T23:19:23.163034+0000 mgr.y (mgr.34107) 12037 : cluster [DBG] pgmap v12538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:23 smithi121 bash[36902]: audit 2022-04-22T23:19:23.318448+0000 mon.a (mon.0) 16610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:23 smithi115 bash[45495]: cluster 2022-04-22T23:19:23.163034+0000 mgr.y (mgr.34107) 12037 : cluster [DBG] pgmap v12538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:23 smithi115 bash[45495]: audit 2022-04-22T23:19:23.318448+0000 mon.a (mon.0) 16610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:23 smithi115 bash[46934]: cluster 2022-04-22T23:19:23.163034+0000 mgr.y (mgr.34107) 12037 : cluster [DBG] pgmap v12538: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:23 smithi115 bash[46934]: audit 2022-04-22T23:19:23.318448+0000 mon.a (mon.0) 16610 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:24.864 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:19:25.213 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:19:25.213 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 98s ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 2m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 98s ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 2m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 98s ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 98s ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:19:25.214 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 2m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 98s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 98s ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 98s ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 98s ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 98s ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 98s ago 5h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:19:25.215 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:19:25.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:19:25.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:19:25.216 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:19:25.216 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:19:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:25 smithi115 bash[46934]: audit 2022-04-22T23:19:25.021051+0000 mon.a (mon.0) 16611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:25 smithi115 bash[45495]: audit 2022-04-22T23:19:25.021051+0000 mon.a (mon.0) 16611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:24.868Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:25 smithi121 bash[36902]: audit 2022-04-22T23:19:25.021051+0000 mon.a (mon.0) 16611 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:25.619 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:19:25.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:19:25.621 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:19:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:26 smithi121 bash[36902]: audit 2022-04-22T23:19:24.844946+0000 mgr.y (mgr.34107) 12038 : audit [DBG] from='client.57739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:26 smithi121 bash[36902]: cluster 2022-04-22T23:19:25.164187+0000 mgr.y (mgr.34107) 12039 : cluster [DBG] pgmap v12539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:26 smithi121 bash[36902]: audit 2022-04-22T23:19:25.191012+0000 mgr.y (mgr.34107) 12040 : audit [DBG] from='client.47805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:26 smithi121 bash[36902]: audit 2022-04-22T23:19:25.615933+0000 mon.c (mon.1) 407 : audit [DBG] from='client.? 172.21.15.115:0/2731543431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[45495]: audit 2022-04-22T23:19:24.844946+0000 mgr.y (mgr.34107) 12038 : audit [DBG] from='client.57739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[45495]: cluster 2022-04-22T23:19:25.164187+0000 mgr.y (mgr.34107) 12039 : cluster [DBG] pgmap v12539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[45495]: audit 2022-04-22T23:19:25.191012+0000 mgr.y (mgr.34107) 12040 : audit [DBG] from='client.47805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[45495]: audit 2022-04-22T23:19:25.615933+0000 mon.c (mon.1) 407 : audit [DBG] from='client.? 172.21.15.115:0/2731543431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[46934]: audit 2022-04-22T23:19:24.844946+0000 mgr.y (mgr.34107) 12038 : audit [DBG] from='client.57739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[46934]: cluster 2022-04-22T23:19:25.164187+0000 mgr.y (mgr.34107) 12039 : cluster [DBG] pgmap v12539: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[46934]: audit 2022-04-22T23:19:25.191012+0000 mgr.y (mgr.34107) 12040 : audit [DBG] from='client.47805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:26 smithi115 bash[46934]: audit 2022-04-22T23:19:25.615933+0000 mon.c (mon.1) 407 : audit [DBG] from='client.? 172.21.15.115:0/2731543431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:27 smithi121 bash[36902]: audit 2022-04-22T23:19:26.689771+0000 mon.a (mon.0) 16612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:27 smithi115 bash[45495]: audit 2022-04-22T23:19:26.689771+0000 mon.a (mon.0) 16612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:27 smithi115 bash[46934]: audit 2022-04-22T23:19:26.689771+0000 mon.a (mon.0) 16612 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:28 smithi121 bash[36902]: cluster 2022-04-22T23:19:27.165546+0000 mgr.y (mgr.34107) 12041 : cluster [DBG] pgmap v12540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:28.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:28 smithi115 bash[45495]: cluster 2022-04-22T23:19:27.165546+0000 mgr.y (mgr.34107) 12041 : cluster [DBG] pgmap v12540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:28 smithi115 bash[46934]: cluster 2022-04-22T23:19:27.165546+0000 mgr.y (mgr.34107) 12041 : cluster [DBG] pgmap v12540: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:29 smithi121 bash[36902]: audit 2022-04-22T23:19:28.321624+0000 mon.a (mon.0) 16613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:29 smithi115 bash[45495]: audit 2022-04-22T23:19:28.321624+0000 mon.a (mon.0) 16613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:29 smithi115 bash[46934]: audit 2022-04-22T23:19:28.321624+0000 mon.a (mon.0) 16613 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:30 smithi121 bash[36902]: cluster 2022-04-22T23:19:29.166763+0000 mgr.y (mgr.34107) 12042 : cluster [DBG] pgmap v12541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:30 smithi121 bash[36902]: audit 2022-04-22T23:19:30.024285+0000 mon.a (mon.0) 16614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:30 smithi115 bash[45495]: cluster 2022-04-22T23:19:29.166763+0000 mgr.y (mgr.34107) 12042 : cluster [DBG] pgmap v12541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:30 smithi115 bash[45495]: audit 2022-04-22T23:19:30.024285+0000 mon.a (mon.0) 16614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:30 smithi115 bash[46934]: cluster 2022-04-22T23:19:29.166763+0000 mgr.y (mgr.34107) 12042 : cluster [DBG] pgmap v12541: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:30 smithi115 bash[46934]: audit 2022-04-22T23:19:30.024285+0000 mon.a (mon.0) 16614 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:32 smithi121 bash[36902]: cluster 2022-04-22T23:19:31.167934+0000 mgr.y (mgr.34107) 12043 : cluster [DBG] pgmap v12542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:32 smithi121 bash[36902]: audit 2022-04-22T23:19:31.692698+0000 mon.a (mon.0) 16615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:32 smithi115 bash[45495]: cluster 2022-04-22T23:19:31.167934+0000 mgr.y (mgr.34107) 12043 : cluster [DBG] pgmap v12542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:32 smithi115 bash[45495]: audit 2022-04-22T23:19:31.692698+0000 mon.a (mon.0) 16615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:32 smithi115 bash[46934]: cluster 2022-04-22T23:19:31.167934+0000 mgr.y (mgr.34107) 12043 : cluster [DBG] pgmap v12542: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:32 smithi115 bash[46934]: audit 2022-04-22T23:19:31.692698+0000 mon.a (mon.0) 16615 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:33.345 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:33.010Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:33 smithi121 bash[36902]: cluster 2022-04-22T23:19:33.168991+0000 mgr.y (mgr.34107) 12044 : cluster [DBG] pgmap v12543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:33 smithi121 bash[36902]: audit 2022-04-22T23:19:33.324591+0000 mon.a (mon.0) 16616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:33 smithi115 bash[45495]: cluster 2022-04-22T23:19:33.168991+0000 mgr.y (mgr.34107) 12044 : cluster [DBG] pgmap v12543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:33 smithi115 bash[45495]: audit 2022-04-22T23:19:33.324591+0000 mon.a (mon.0) 16616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:33 smithi115 bash[46934]: cluster 2022-04-22T23:19:33.168991+0000 mgr.y (mgr.34107) 12044 : cluster [DBG] pgmap v12543: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:33 smithi115 bash[46934]: audit 2022-04-22T23:19:33.324591+0000 mon.a (mon.0) 16616 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:35 smithi115 bash[45495]: audit 2022-04-22T23:19:35.028126+0000 mon.a (mon.0) 16617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:34.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:35 smithi115 bash[46934]: audit 2022-04-22T23:19:35.028126+0000 mon.a (mon.0) 16617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:35 smithi121 bash[36902]: audit 2022-04-22T23:19:35.028126+0000 mon.a (mon.0) 16617 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:36 smithi121 bash[36902]: cluster 2022-04-22T23:19:35.169933+0000 mgr.y (mgr.34107) 12045 : cluster [DBG] pgmap v12544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:36 smithi115 bash[45495]: cluster 2022-04-22T23:19:35.169933+0000 mgr.y (mgr.34107) 12045 : cluster [DBG] pgmap v12544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:36 smithi115 bash[46934]: cluster 2022-04-22T23:19:35.169933+0000 mgr.y (mgr.34107) 12045 : cluster [DBG] pgmap v12544: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:37 smithi121 bash[36902]: audit 2022-04-22T23:19:36.695717+0000 mon.a (mon.0) 16618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:37.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:37 smithi115 bash[45495]: audit 2022-04-22T23:19:36.695717+0000 mon.a (mon.0) 16618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:37 smithi115 bash[46934]: audit 2022-04-22T23:19:36.695717+0000 mon.a (mon.0) 16618 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:38 smithi121 bash[36902]: cluster 2022-04-22T23:19:37.171077+0000 mgr.y (mgr.34107) 12046 : cluster [DBG] pgmap v12545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:38 smithi115 bash[45495]: cluster 2022-04-22T23:19:37.171077+0000 mgr.y (mgr.34107) 12046 : cluster [DBG] pgmap v12545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:38 smithi115 bash[46934]: cluster 2022-04-22T23:19:37.171077+0000 mgr.y (mgr.34107) 12046 : cluster [DBG] pgmap v12545: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:39 smithi121 bash[36902]: audit 2022-04-22T23:19:38.327557+0000 mon.a (mon.0) 16619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:39 smithi115 bash[45495]: audit 2022-04-22T23:19:38.327557+0000 mon.a (mon.0) 16619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:39 smithi115 bash[46934]: audit 2022-04-22T23:19:38.327557+0000 mon.a (mon.0) 16619 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:40 smithi121 bash[36902]: cluster 2022-04-22T23:19:39.172170+0000 mgr.y (mgr.34107) 12047 : cluster [DBG] pgmap v12546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:40 smithi121 bash[36902]: audit 2022-04-22T23:19:40.031551+0000 mon.a (mon.0) 16620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:40 smithi115 bash[45495]: cluster 2022-04-22T23:19:39.172170+0000 mgr.y (mgr.34107) 12047 : cluster [DBG] pgmap v12546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:40 smithi115 bash[45495]: audit 2022-04-22T23:19:40.031551+0000 mon.a (mon.0) 16620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:40 smithi115 bash[46934]: cluster 2022-04-22T23:19:39.172170+0000 mgr.y (mgr.34107) 12047 : cluster [DBG] pgmap v12546: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:40 smithi115 bash[46934]: audit 2022-04-22T23:19:40.031551+0000 mon.a (mon.0) 16620 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:42 smithi121 bash[36902]: cluster 2022-04-22T23:19:41.173193+0000 mgr.y (mgr.34107) 12048 : cluster [DBG] pgmap v12547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:42 smithi121 bash[36902]: audit 2022-04-22T23:19:41.698658+0000 mon.a (mon.0) 16621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:42 smithi115 bash[45495]: cluster 2022-04-22T23:19:41.173193+0000 mgr.y (mgr.34107) 12048 : cluster [DBG] pgmap v12547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:42 smithi115 bash[45495]: audit 2022-04-22T23:19:41.698658+0000 mon.a (mon.0) 16621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:42 smithi115 bash[46934]: cluster 2022-04-22T23:19:41.173193+0000 mgr.y (mgr.34107) 12048 : cluster [DBG] pgmap v12547: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:42 smithi115 bash[46934]: audit 2022-04-22T23:19:41.698658+0000 mon.a (mon.0) 16621 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:43.013Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:43 smithi121 bash[36902]: cluster 2022-04-22T23:19:43.174179+0000 mgr.y (mgr.34107) 12049 : cluster [DBG] pgmap v12548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:43 smithi121 bash[36902]: audit 2022-04-22T23:19:43.330383+0000 mon.a (mon.0) 16622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:43 smithi115 bash[45495]: cluster 2022-04-22T23:19:43.174179+0000 mgr.y (mgr.34107) 12049 : cluster [DBG] pgmap v12548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:43 smithi115 bash[45495]: audit 2022-04-22T23:19:43.330383+0000 mon.a (mon.0) 16622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:43 smithi115 bash[46934]: cluster 2022-04-22T23:19:43.174179+0000 mgr.y (mgr.34107) 12049 : cluster [DBG] pgmap v12548: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:43 smithi115 bash[46934]: audit 2022-04-22T23:19:43.330383+0000 mon.a (mon.0) 16622 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:45 smithi115 bash[45495]: audit 2022-04-22T23:19:45.035584+0000 mon.a (mon.0) 16623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:44.869Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:45 smithi115 bash[46934]: audit 2022-04-22T23:19:45.035584+0000 mon.a (mon.0) 16623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:45 smithi121 bash[36902]: audit 2022-04-22T23:19:45.035584+0000 mon.a (mon.0) 16623 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:46 smithi121 bash[36902]: cluster 2022-04-22T23:19:45.175113+0000 mgr.y (mgr.34107) 12050 : cluster [DBG] pgmap v12549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:46 smithi115 bash[45495]: cluster 2022-04-22T23:19:45.175113+0000 mgr.y (mgr.34107) 12050 : cluster [DBG] pgmap v12549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:46 smithi115 bash[46934]: cluster 2022-04-22T23:19:45.175113+0000 mgr.y (mgr.34107) 12050 : cluster [DBG] pgmap v12549: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:47 smithi121 bash[36902]: audit 2022-04-22T23:19:46.701796+0000 mon.a (mon.0) 16624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:47 smithi115 bash[45495]: audit 2022-04-22T23:19:46.701796+0000 mon.a (mon.0) 16624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:47 smithi115 bash[46934]: audit 2022-04-22T23:19:46.701796+0000 mon.a (mon.0) 16624 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:48 smithi121 bash[36902]: cluster 2022-04-22T23:19:47.176313+0000 mgr.y (mgr.34107) 12051 : cluster [DBG] pgmap v12550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:48 smithi115 bash[45495]: cluster 2022-04-22T23:19:47.176313+0000 mgr.y (mgr.34107) 12051 : cluster [DBG] pgmap v12550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:48 smithi115 bash[46934]: cluster 2022-04-22T23:19:47.176313+0000 mgr.y (mgr.34107) 12051 : cluster [DBG] pgmap v12550: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:49 smithi121 bash[36902]: audit 2022-04-22T23:19:48.333475+0000 mon.a (mon.0) 16625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:49 smithi115 bash[45495]: audit 2022-04-22T23:19:48.333475+0000 mon.a (mon.0) 16625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:49 smithi115 bash[46934]: audit 2022-04-22T23:19:48.333475+0000 mon.a (mon.0) 16625 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:50 smithi121 bash[36902]: cluster 2022-04-22T23:19:49.177412+0000 mgr.y (mgr.34107) 12052 : cluster [DBG] pgmap v12551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:50 smithi121 bash[36902]: audit 2022-04-22T23:19:50.039240+0000 mon.a (mon.0) 16626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:50 smithi115 bash[45495]: cluster 2022-04-22T23:19:49.177412+0000 mgr.y (mgr.34107) 12052 : cluster [DBG] pgmap v12551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:50 smithi115 bash[45495]: audit 2022-04-22T23:19:50.039240+0000 mon.a (mon.0) 16626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:50 smithi115 bash[46934]: cluster 2022-04-22T23:19:49.177412+0000 mgr.y (mgr.34107) 12052 : cluster [DBG] pgmap v12551: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:50 smithi115 bash[46934]: audit 2022-04-22T23:19:50.039240+0000 mon.a (mon.0) 16626 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:52 smithi121 bash[36902]: cluster 2022-04-22T23:19:51.178479+0000 mgr.y (mgr.34107) 12053 : cluster [DBG] pgmap v12552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:52.551 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:52 smithi121 bash[36902]: audit 2022-04-22T23:19:51.704802+0000 mon.a (mon.0) 16627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:52.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:52 smithi115 bash[45495]: cluster 2022-04-22T23:19:51.178479+0000 mgr.y (mgr.34107) 12053 : cluster [DBG] pgmap v12552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:52.552 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:52 smithi115 bash[45495]: audit 2022-04-22T23:19:51.704802+0000 mon.a (mon.0) 16627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:52.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:52 smithi115 bash[46934]: cluster 2022-04-22T23:19:51.178479+0000 mgr.y (mgr.34107) 12053 : cluster [DBG] pgmap v12552: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:52.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:52 smithi115 bash[46934]: audit 2022-04-22T23:19:51.704802+0000 mon.a (mon.0) 16627 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:53.013Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:53 smithi121 bash[36902]: cluster 2022-04-22T23:19:53.179541+0000 mgr.y (mgr.34107) 12054 : cluster [DBG] pgmap v12553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:53 smithi121 bash[36902]: audit 2022-04-22T23:19:53.336580+0000 mon.a (mon.0) 16628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:53 smithi115 bash[45495]: cluster 2022-04-22T23:19:53.179541+0000 mgr.y (mgr.34107) 12054 : cluster [DBG] pgmap v12553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:53 smithi115 bash[45495]: audit 2022-04-22T23:19:53.336580+0000 mon.a (mon.0) 16628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:53 smithi115 bash[46934]: cluster 2022-04-22T23:19:53.179541+0000 mgr.y (mgr.34107) 12054 : cluster [DBG] pgmap v12553: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:53 smithi115 bash[46934]: audit 2022-04-22T23:19:53.336580+0000 mon.a (mon.0) 16628 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:55 smithi115 bash[45495]: audit 2022-04-22T23:19:55.042821+0000 mon.a (mon.0) 16629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:55 smithi115 bash[46934]: audit 2022-04-22T23:19:55.042821+0000 mon.a (mon.0) 16629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:55.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:19:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:19:54.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:19:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:55 smithi121 bash[36902]: audit 2022-04-22T23:19:55.042821+0000 mon.a (mon.0) 16629 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:19:55.987 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:19:56.346 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:19:56.375 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 5h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:19:56.376 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:19:56.377 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:19:56.377 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:19:56.377 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:19:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:56 smithi121 bash[36902]: cluster 2022-04-22T23:19:55.180436+0000 mgr.y (mgr.34107) 12055 : cluster [DBG] pgmap v12554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:56 smithi115 bash[46934]: cluster 2022-04-22T23:19:55.180436+0000 mgr.y (mgr.34107) 12055 : cluster [DBG] pgmap v12554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:56 smithi115 bash[45495]: cluster 2022-04-22T23:19:55.180436+0000 mgr.y (mgr.34107) 12055 : cluster [DBG] pgmap v12554: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:56.753 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:56.754 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:19:56.755 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:19:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:55.967540+0000 mgr.y (mgr.34107) 12056 : audit [DBG] from='client.47817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.607315+0000 mon.a (mon.0) 16630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:19:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.609811+0000 mon.a (mon.0) 16631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:19:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.611284+0000 mon.a (mon.0) 16632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:19:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.707648+0000 mon.a (mon.0) 16633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.750137+0000 mon.c (mon.1) 408 : audit [DBG] from='client.? 172.21.15.115:0/136909917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:57.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:57 smithi121 bash[36902]: audit 2022-04-22T23:19:56.964115+0000 mon.a (mon.0) 16634 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:19:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:55.967540+0000 mgr.y (mgr.34107) 12056 : audit [DBG] from='client.47817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.607315+0000 mon.a (mon.0) 16630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:19:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.609811+0000 mon.a (mon.0) 16631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:19:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.611284+0000 mon.a (mon.0) 16632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:19:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.707648+0000 mon.a (mon.0) 16633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.750137+0000 mon.c (mon.1) 408 : audit [DBG] from='client.? 172.21.15.115:0/136909917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[45495]: audit 2022-04-22T23:19:56.964115+0000 mon.a (mon.0) 16634 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:55.967540+0000 mgr.y (mgr.34107) 12056 : audit [DBG] from='client.47817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.607315+0000 mon.a (mon.0) 16630 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.609811+0000 mon.a (mon.0) 16631 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.611284+0000 mon.a (mon.0) 16632 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.707648+0000 mon.a (mon.0) 16633 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:19:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.750137+0000 mon.c (mon.1) 408 : audit [DBG] from='client.? 172.21.15.115:0/136909917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:19:57.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:57 smithi115 bash[46934]: audit 2022-04-22T23:19:56.964115+0000 mon.a (mon.0) 16634 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:19:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:58 smithi121 bash[36902]: audit 2022-04-22T23:19:56.323451+0000 mgr.y (mgr.34107) 12057 : audit [DBG] from='client.57763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:58 smithi121 bash[36902]: cluster 2022-04-22T23:19:57.181560+0000 mgr.y (mgr.34107) 12058 : cluster [DBG] pgmap v12555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:58.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:58 smithi115 bash[45495]: audit 2022-04-22T23:19:56.323451+0000 mgr.y (mgr.34107) 12057 : audit [DBG] from='client.57763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:58 smithi115 bash[45495]: cluster 2022-04-22T23:19:57.181560+0000 mgr.y (mgr.34107) 12058 : cluster [DBG] pgmap v12555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:58 smithi115 bash[46934]: audit 2022-04-22T23:19:56.323451+0000 mgr.y (mgr.34107) 12057 : audit [DBG] from='client.57763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:19:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:58 smithi115 bash[46934]: cluster 2022-04-22T23:19:57.181560+0000 mgr.y (mgr.34107) 12058 : cluster [DBG] pgmap v12555: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:19:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:19:59 smithi121 bash[36902]: audit 2022-04-22T23:19:58.339883+0000 mon.a (mon.0) 16635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:19:59 smithi115 bash[45495]: audit 2022-04-22T23:19:58.339883+0000 mon.a (mon.0) 16635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:19:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:19:59 smithi115 bash[46934]: audit 2022-04-22T23:19:58.339883+0000 mon.a (mon.0) 16635 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:00 smithi121 bash[36902]: cluster 2022-04-22T23:19:59.182628+0000 mgr.y (mgr.34107) 12059 : cluster [DBG] pgmap v12556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:00 smithi121 bash[36902]: cluster 2022-04-22T23:20:00.000179+0000 mon.a (mon.0) 16636 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:20:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:00 smithi121 bash[36902]: audit 2022-04-22T23:20:00.046587+0000 mon.a (mon.0) 16637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[45495]: cluster 2022-04-22T23:19:59.182628+0000 mgr.y (mgr.34107) 12059 : cluster [DBG] pgmap v12556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[45495]: cluster 2022-04-22T23:20:00.000179+0000 mon.a (mon.0) 16636 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:20:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[45495]: audit 2022-04-22T23:20:00.046587+0000 mon.a (mon.0) 16637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[46934]: cluster 2022-04-22T23:19:59.182628+0000 mgr.y (mgr.34107) 12059 : cluster [DBG] pgmap v12556: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[46934]: cluster 2022-04-22T23:20:00.000179+0000 mon.a (mon.0) 16636 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:20:00.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:00 smithi115 bash[46934]: audit 2022-04-22T23:20:00.046587+0000 mon.a (mon.0) 16637 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:02 smithi121 bash[36902]: cluster 2022-04-22T23:20:01.183834+0000 mgr.y (mgr.34107) 12060 : cluster [DBG] pgmap v12557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:02 smithi121 bash[36902]: audit 2022-04-22T23:20:01.710403+0000 mon.a (mon.0) 16638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:02 smithi115 bash[45495]: cluster 2022-04-22T23:20:01.183834+0000 mgr.y (mgr.34107) 12060 : cluster [DBG] pgmap v12557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:02 smithi115 bash[45495]: audit 2022-04-22T23:20:01.710403+0000 mon.a (mon.0) 16638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:02 smithi115 bash[46934]: cluster 2022-04-22T23:20:01.183834+0000 mgr.y (mgr.34107) 12060 : cluster [DBG] pgmap v12557: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:02 smithi115 bash[46934]: audit 2022-04-22T23:20:01.710403+0000 mon.a (mon.0) 16638 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:03.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:03 smithi121 bash[36902]: cluster 2022-04-22T23:20:03.184875+0000 mgr.y (mgr.34107) 12061 : cluster [DBG] pgmap v12558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:03 smithi121 bash[36902]: audit 2022-04-22T23:20:03.342949+0000 mon.a (mon.0) 16639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:03 smithi115 bash[45495]: cluster 2022-04-22T23:20:03.184875+0000 mgr.y (mgr.34107) 12061 : cluster [DBG] pgmap v12558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:03 smithi115 bash[45495]: audit 2022-04-22T23:20:03.342949+0000 mon.a (mon.0) 16639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:03 smithi115 bash[46934]: cluster 2022-04-22T23:20:03.184875+0000 mgr.y (mgr.34107) 12061 : cluster [DBG] pgmap v12558: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:03 smithi115 bash[46934]: audit 2022-04-22T23:20:03.342949+0000 mon.a (mon.0) 16639 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:05 smithi115 bash[45495]: audit 2022-04-22T23:20:05.050394+0000 mon.a (mon.0) 16640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:05.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:04.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:05 smithi115 bash[46934]: audit 2022-04-22T23:20:05.050394+0000 mon.a (mon.0) 16640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:05.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:05 smithi121 bash[36902]: audit 2022-04-22T23:20:05.050394+0000 mon.a (mon.0) 16640 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:06 smithi121 bash[36902]: cluster 2022-04-22T23:20:05.185861+0000 mgr.y (mgr.34107) 12062 : cluster [DBG] pgmap v12559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:06 smithi115 bash[45495]: cluster 2022-04-22T23:20:05.185861+0000 mgr.y (mgr.34107) 12062 : cluster [DBG] pgmap v12559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:06 smithi115 bash[46934]: cluster 2022-04-22T23:20:05.185861+0000 mgr.y (mgr.34107) 12062 : cluster [DBG] pgmap v12559: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:07 smithi121 bash[36902]: audit 2022-04-22T23:20:06.713829+0000 mon.a (mon.0) 16641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:07 smithi115 bash[45495]: audit 2022-04-22T23:20:06.713829+0000 mon.a (mon.0) 16641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:07 smithi115 bash[46934]: audit 2022-04-22T23:20:06.713829+0000 mon.a (mon.0) 16641 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:08 smithi121 bash[36902]: cluster 2022-04-22T23:20:07.187164+0000 mgr.y (mgr.34107) 12063 : cluster [DBG] pgmap v12560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:08 smithi115 bash[45495]: cluster 2022-04-22T23:20:07.187164+0000 mgr.y (mgr.34107) 12063 : cluster [DBG] pgmap v12560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:08 smithi115 bash[46934]: cluster 2022-04-22T23:20:07.187164+0000 mgr.y (mgr.34107) 12063 : cluster [DBG] pgmap v12560: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:09 smithi121 bash[36902]: audit 2022-04-22T23:20:08.346386+0000 mon.a (mon.0) 16642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:09 smithi115 bash[45495]: audit 2022-04-22T23:20:08.346386+0000 mon.a (mon.0) 16642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:09 smithi115 bash[46934]: audit 2022-04-22T23:20:08.346386+0000 mon.a (mon.0) 16642 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:10 smithi121 bash[36902]: cluster 2022-04-22T23:20:09.188245+0000 mgr.y (mgr.34107) 12064 : cluster [DBG] pgmap v12561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:10 smithi121 bash[36902]: audit 2022-04-22T23:20:10.054068+0000 mon.a (mon.0) 16643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:10 smithi115 bash[45495]: cluster 2022-04-22T23:20:09.188245+0000 mgr.y (mgr.34107) 12064 : cluster [DBG] pgmap v12561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:10 smithi115 bash[45495]: audit 2022-04-22T23:20:10.054068+0000 mon.a (mon.0) 16643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:10 smithi115 bash[46934]: cluster 2022-04-22T23:20:09.188245+0000 mgr.y (mgr.34107) 12064 : cluster [DBG] pgmap v12561: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:10 smithi115 bash[46934]: audit 2022-04-22T23:20:10.054068+0000 mon.a (mon.0) 16643 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:12 smithi121 bash[36902]: cluster 2022-04-22T23:20:11.189454+0000 mgr.y (mgr.34107) 12065 : cluster [DBG] pgmap v12562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:12 smithi121 bash[36902]: audit 2022-04-22T23:20:11.716972+0000 mon.a (mon.0) 16644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:12 smithi115 bash[45495]: cluster 2022-04-22T23:20:11.189454+0000 mgr.y (mgr.34107) 12065 : cluster [DBG] pgmap v12562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:12 smithi115 bash[45495]: audit 2022-04-22T23:20:11.716972+0000 mon.a (mon.0) 16644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:12 smithi115 bash[46934]: cluster 2022-04-22T23:20:11.189454+0000 mgr.y (mgr.34107) 12065 : cluster [DBG] pgmap v12562: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:12 smithi115 bash[46934]: audit 2022-04-22T23:20:11.716972+0000 mon.a (mon.0) 16644 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:13.013Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:13 smithi121 bash[36902]: cluster 2022-04-22T23:20:13.190595+0000 mgr.y (mgr.34107) 12066 : cluster [DBG] pgmap v12563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:13 smithi121 bash[36902]: audit 2022-04-22T23:20:13.349603+0000 mon.a (mon.0) 16645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:14.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:13 smithi115 bash[45495]: cluster 2022-04-22T23:20:13.190595+0000 mgr.y (mgr.34107) 12066 : cluster [DBG] pgmap v12563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:13 smithi115 bash[45495]: audit 2022-04-22T23:20:13.349603+0000 mon.a (mon.0) 16645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:13 smithi115 bash[46934]: cluster 2022-04-22T23:20:13.190595+0000 mgr.y (mgr.34107) 12066 : cluster [DBG] pgmap v12563: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:13 smithi115 bash[46934]: audit 2022-04-22T23:20:13.349603+0000 mon.a (mon.0) 16645 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:15 smithi115 bash[45495]: audit 2022-04-22T23:20:15.057693+0000 mon.a (mon.0) 16646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:14.870Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:15.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:15 smithi115 bash[46934]: audit 2022-04-22T23:20:15.057693+0000 mon.a (mon.0) 16646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:15 smithi121 bash[36902]: audit 2022-04-22T23:20:15.057693+0000 mon.a (mon.0) 16646 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:16 smithi121 bash[36902]: cluster 2022-04-22T23:20:15.191648+0000 mgr.y (mgr.34107) 12067 : cluster [DBG] pgmap v12564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:16 smithi115 bash[45495]: cluster 2022-04-22T23:20:15.191648+0000 mgr.y (mgr.34107) 12067 : cluster [DBG] pgmap v12564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:16 smithi115 bash[46934]: cluster 2022-04-22T23:20:15.191648+0000 mgr.y (mgr.34107) 12067 : cluster [DBG] pgmap v12564: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:17 smithi121 bash[36902]: audit 2022-04-22T23:20:16.720102+0000 mon.a (mon.0) 16647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:17 smithi115 bash[45495]: audit 2022-04-22T23:20:16.720102+0000 mon.a (mon.0) 16647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:17 smithi115 bash[46934]: audit 2022-04-22T23:20:16.720102+0000 mon.a (mon.0) 16647 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:18 smithi121 bash[36902]: cluster 2022-04-22T23:20:17.192802+0000 mgr.y (mgr.34107) 12068 : cluster [DBG] pgmap v12565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:18 smithi121 bash[36902]: audit 2022-04-22T23:20:17.351052+0000 mon.a (mon.0) 16648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:20:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:18 smithi121 bash[36902]: audit 2022-04-22T23:20:17.351645+0000 mon.a (mon.0) 16649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:20:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[45495]: cluster 2022-04-22T23:20:17.192802+0000 mgr.y (mgr.34107) 12068 : cluster [DBG] pgmap v12565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[45495]: audit 2022-04-22T23:20:17.351052+0000 mon.a (mon.0) 16648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:20:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[45495]: audit 2022-04-22T23:20:17.351645+0000 mon.a (mon.0) 16649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:20:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[46934]: cluster 2022-04-22T23:20:17.192802+0000 mgr.y (mgr.34107) 12068 : cluster [DBG] pgmap v12565: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[46934]: audit 2022-04-22T23:20:17.351052+0000 mon.a (mon.0) 16648 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:20:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:18 smithi115 bash[46934]: audit 2022-04-22T23:20:17.351645+0000 mon.a (mon.0) 16649 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:20:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:19 smithi121 bash[36902]: audit 2022-04-22T23:20:18.352877+0000 mon.a (mon.0) 16650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:19 smithi115 bash[45495]: audit 2022-04-22T23:20:18.352877+0000 mon.a (mon.0) 16650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:19 smithi115 bash[46934]: audit 2022-04-22T23:20:18.352877+0000 mon.a (mon.0) 16650 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:20 smithi121 bash[36902]: cluster 2022-04-22T23:20:19.193799+0000 mgr.y (mgr.34107) 12069 : cluster [DBG] pgmap v12566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:20 smithi121 bash[36902]: audit 2022-04-22T23:20:20.061514+0000 mon.a (mon.0) 16651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:20 smithi115 bash[45495]: cluster 2022-04-22T23:20:19.193799+0000 mgr.y (mgr.34107) 12069 : cluster [DBG] pgmap v12566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:20 smithi115 bash[45495]: audit 2022-04-22T23:20:20.061514+0000 mon.a (mon.0) 16651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:20 smithi115 bash[46934]: cluster 2022-04-22T23:20:19.193799+0000 mgr.y (mgr.34107) 12069 : cluster [DBG] pgmap v12566: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:20 smithi115 bash[46934]: audit 2022-04-22T23:20:20.061514+0000 mon.a (mon.0) 16651 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:22 smithi121 bash[36902]: cluster 2022-04-22T23:20:21.194852+0000 mgr.y (mgr.34107) 12070 : cluster [DBG] pgmap v12567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:22 smithi121 bash[36902]: audit 2022-04-22T23:20:21.722828+0000 mon.a (mon.0) 16652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:22 smithi115 bash[45495]: cluster 2022-04-22T23:20:21.194852+0000 mgr.y (mgr.34107) 12070 : cluster [DBG] pgmap v12567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:22 smithi115 bash[45495]: audit 2022-04-22T23:20:21.722828+0000 mon.a (mon.0) 16652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:22 smithi115 bash[46934]: cluster 2022-04-22T23:20:21.194852+0000 mgr.y (mgr.34107) 12070 : cluster [DBG] pgmap v12567: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:22 smithi115 bash[46934]: audit 2022-04-22T23:20:21.722828+0000 mon.a (mon.0) 16652 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:23.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:23 smithi121 bash[36902]: cluster 2022-04-22T23:20:23.195944+0000 mgr.y (mgr.34107) 12071 : cluster [DBG] pgmap v12568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:23 smithi121 bash[36902]: audit 2022-04-22T23:20:23.355963+0000 mon.a (mon.0) 16653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:23 smithi115 bash[45495]: cluster 2022-04-22T23:20:23.195944+0000 mgr.y (mgr.34107) 12071 : cluster [DBG] pgmap v12568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:23 smithi115 bash[45495]: audit 2022-04-22T23:20:23.355963+0000 mon.a (mon.0) 16653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:23 smithi115 bash[46934]: cluster 2022-04-22T23:20:23.195944+0000 mgr.y (mgr.34107) 12071 : cluster [DBG] pgmap v12568: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:23 smithi115 bash[46934]: audit 2022-04-22T23:20:23.355963+0000 mon.a (mon.0) 16653 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:25 smithi115 bash[45495]: audit 2022-04-22T23:20:25.065444+0000 mon.a (mon.0) 16654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:24.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:25.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:25 smithi115 bash[46934]: audit 2022-04-22T23:20:25.065444+0000 mon.a (mon.0) 16654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:25 smithi121 bash[36902]: audit 2022-04-22T23:20:25.065444+0000 mon.a (mon.0) 16654 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:26 smithi121 bash[36902]: cluster 2022-04-22T23:20:25.197030+0000 mgr.y (mgr.34107) 12072 : cluster [DBG] pgmap v12569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:26 smithi115 bash[45495]: cluster 2022-04-22T23:20:25.197030+0000 mgr.y (mgr.34107) 12072 : cluster [DBG] pgmap v12569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:26 smithi115 bash[46934]: cluster 2022-04-22T23:20:25.197030+0000 mgr.y (mgr.34107) 12072 : cluster [DBG] pgmap v12569: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:27.119 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 3m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 3m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:20:27.470 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 2m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 2m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 3m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 2m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 2m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:20:27.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:20:27.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:20:27.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:20:27.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 5h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:20:27.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:20:27.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:20:27.473 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:20:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:27 smithi121 bash[36902]: audit 2022-04-22T23:20:26.725921+0000 mon.a (mon.0) 16655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:27 smithi115 bash[45495]: audit 2022-04-22T23:20:26.725921+0000 mon.a (mon.0) 16655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:27 smithi115 bash[46934]: audit 2022-04-22T23:20:26.725921+0000 mon.a (mon.0) 16655 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:27.870 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:20:27.871 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:20:27.872 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:20:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:28 smithi121 bash[36902]: audit 2022-04-22T23:20:27.099512+0000 mgr.y (mgr.34107) 12073 : audit [DBG] from='client.57775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:28 smithi121 bash[36902]: cluster 2022-04-22T23:20:27.198158+0000 mgr.y (mgr.34107) 12074 : cluster [DBG] pgmap v12570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:28 smithi121 bash[36902]: audit 2022-04-22T23:20:27.866956+0000 mon.c (mon.1) 409 : audit [DBG] from='client.? 172.21.15.115:0/2683539230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[45495]: audit 2022-04-22T23:20:27.099512+0000 mgr.y (mgr.34107) 12073 : audit [DBG] from='client.57775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[45495]: cluster 2022-04-22T23:20:27.198158+0000 mgr.y (mgr.34107) 12074 : cluster [DBG] pgmap v12570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[45495]: audit 2022-04-22T23:20:27.866956+0000 mon.c (mon.1) 409 : audit [DBG] from='client.? 172.21.15.115:0/2683539230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[46934]: audit 2022-04-22T23:20:27.099512+0000 mgr.y (mgr.34107) 12073 : audit [DBG] from='client.57775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[46934]: cluster 2022-04-22T23:20:27.198158+0000 mgr.y (mgr.34107) 12074 : cluster [DBG] pgmap v12570: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:28.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:28 smithi115 bash[46934]: audit 2022-04-22T23:20:27.866956+0000 mon.c (mon.1) 409 : audit [DBG] from='client.? 172.21.15.115:0/2683539230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:29 smithi121 bash[36902]: audit 2022-04-22T23:20:27.442884+0000 mgr.y (mgr.34107) 12075 : audit [DBG] from='client.47841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:29 smithi121 bash[36902]: audit 2022-04-22T23:20:28.359044+0000 mon.a (mon.0) 16656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:29 smithi115 bash[45495]: audit 2022-04-22T23:20:27.442884+0000 mgr.y (mgr.34107) 12075 : audit [DBG] from='client.47841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:29 smithi115 bash[45495]: audit 2022-04-22T23:20:28.359044+0000 mon.a (mon.0) 16656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:29 smithi115 bash[46934]: audit 2022-04-22T23:20:27.442884+0000 mgr.y (mgr.34107) 12075 : audit [DBG] from='client.47841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:29 smithi115 bash[46934]: audit 2022-04-22T23:20:28.359044+0000 mon.a (mon.0) 16656 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:30 smithi121 bash[36902]: cluster 2022-04-22T23:20:29.199296+0000 mgr.y (mgr.34107) 12076 : cluster [DBG] pgmap v12571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:30 smithi121 bash[36902]: audit 2022-04-22T23:20:30.069070+0000 mon.a (mon.0) 16657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:30 smithi115 bash[45495]: cluster 2022-04-22T23:20:29.199296+0000 mgr.y (mgr.34107) 12076 : cluster [DBG] pgmap v12571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:30 smithi115 bash[45495]: audit 2022-04-22T23:20:30.069070+0000 mon.a (mon.0) 16657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:30 smithi115 bash[46934]: cluster 2022-04-22T23:20:29.199296+0000 mgr.y (mgr.34107) 12076 : cluster [DBG] pgmap v12571: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:30 smithi115 bash[46934]: audit 2022-04-22T23:20:30.069070+0000 mon.a (mon.0) 16657 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:32 smithi121 bash[36902]: cluster 2022-04-22T23:20:31.200454+0000 mgr.y (mgr.34107) 12077 : cluster [DBG] pgmap v12572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:32 smithi121 bash[36902]: audit 2022-04-22T23:20:31.728962+0000 mon.a (mon.0) 16658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:32 smithi115 bash[45495]: cluster 2022-04-22T23:20:31.200454+0000 mgr.y (mgr.34107) 12077 : cluster [DBG] pgmap v12572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:32 smithi115 bash[45495]: audit 2022-04-22T23:20:31.728962+0000 mon.a (mon.0) 16658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:32 smithi115 bash[46934]: cluster 2022-04-22T23:20:31.200454+0000 mgr.y (mgr.34107) 12077 : cluster [DBG] pgmap v12572: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:32 smithi115 bash[46934]: audit 2022-04-22T23:20:31.728962+0000 mon.a (mon.0) 16658 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:33.014Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:33 smithi121 bash[36902]: cluster 2022-04-22T23:20:33.201588+0000 mgr.y (mgr.34107) 12078 : cluster [DBG] pgmap v12573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:33 smithi121 bash[36902]: audit 2022-04-22T23:20:33.362086+0000 mon.a (mon.0) 16659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:33 smithi115 bash[45495]: cluster 2022-04-22T23:20:33.201588+0000 mgr.y (mgr.34107) 12078 : cluster [DBG] pgmap v12573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:33 smithi115 bash[45495]: audit 2022-04-22T23:20:33.362086+0000 mon.a (mon.0) 16659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:33 smithi115 bash[46934]: cluster 2022-04-22T23:20:33.201588+0000 mgr.y (mgr.34107) 12078 : cluster [DBG] pgmap v12573: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:33 smithi115 bash[46934]: audit 2022-04-22T23:20:33.362086+0000 mon.a (mon.0) 16659 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:35.129 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:34.872Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:35 smithi121 bash[36902]: audit 2022-04-22T23:20:35.072882+0000 mon.a (mon.0) 16660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:35 smithi115 bash[45495]: audit 2022-04-22T23:20:35.072882+0000 mon.a (mon.0) 16660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:35 smithi115 bash[46934]: audit 2022-04-22T23:20:35.072882+0000 mon.a (mon.0) 16660 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:36 smithi121 bash[36902]: cluster 2022-04-22T23:20:35.202662+0000 mgr.y (mgr.34107) 12079 : cluster [DBG] pgmap v12574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:36 smithi115 bash[45495]: cluster 2022-04-22T23:20:35.202662+0000 mgr.y (mgr.34107) 12079 : cluster [DBG] pgmap v12574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:36 smithi115 bash[46934]: cluster 2022-04-22T23:20:35.202662+0000 mgr.y (mgr.34107) 12079 : cluster [DBG] pgmap v12574: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:37 smithi121 bash[36902]: audit 2022-04-22T23:20:36.732150+0000 mon.a (mon.0) 16661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:37 smithi115 bash[45495]: audit 2022-04-22T23:20:36.732150+0000 mon.a (mon.0) 16661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:37 smithi115 bash[46934]: audit 2022-04-22T23:20:36.732150+0000 mon.a (mon.0) 16661 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:38.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:38 smithi121 bash[36902]: cluster 2022-04-22T23:20:37.203815+0000 mgr.y (mgr.34107) 12080 : cluster [DBG] pgmap v12575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:38 smithi115 bash[45495]: cluster 2022-04-22T23:20:37.203815+0000 mgr.y (mgr.34107) 12080 : cluster [DBG] pgmap v12575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:38 smithi115 bash[46934]: cluster 2022-04-22T23:20:37.203815+0000 mgr.y (mgr.34107) 12080 : cluster [DBG] pgmap v12575: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:39 smithi121 bash[36902]: audit 2022-04-22T23:20:38.365169+0000 mon.a (mon.0) 16662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:39.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:39 smithi115 bash[45495]: audit 2022-04-22T23:20:38.365169+0000 mon.a (mon.0) 16662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:39 smithi115 bash[46934]: audit 2022-04-22T23:20:38.365169+0000 mon.a (mon.0) 16662 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:40 smithi121 bash[36902]: cluster 2022-04-22T23:20:39.204878+0000 mgr.y (mgr.34107) 12081 : cluster [DBG] pgmap v12576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:40 smithi121 bash[36902]: audit 2022-04-22T23:20:40.076435+0000 mon.a (mon.0) 16663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:40 smithi115 bash[45495]: cluster 2022-04-22T23:20:39.204878+0000 mgr.y (mgr.34107) 12081 : cluster [DBG] pgmap v12576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:40 smithi115 bash[45495]: audit 2022-04-22T23:20:40.076435+0000 mon.a (mon.0) 16663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:40 smithi115 bash[46934]: cluster 2022-04-22T23:20:39.204878+0000 mgr.y (mgr.34107) 12081 : cluster [DBG] pgmap v12576: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:40 smithi115 bash[46934]: audit 2022-04-22T23:20:40.076435+0000 mon.a (mon.0) 16663 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:42 smithi115 bash[45495]: cluster 2022-04-22T23:20:41.205934+0000 mgr.y (mgr.34107) 12082 : cluster [DBG] pgmap v12577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:42 smithi115 bash[45495]: audit 2022-04-22T23:20:41.735143+0000 mon.a (mon.0) 16664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:42 smithi115 bash[46934]: cluster 2022-04-22T23:20:41.205934+0000 mgr.y (mgr.34107) 12082 : cluster [DBG] pgmap v12577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:42 smithi115 bash[46934]: audit 2022-04-22T23:20:41.735143+0000 mon.a (mon.0) 16664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:42 smithi121 bash[36902]: cluster 2022-04-22T23:20:41.205934+0000 mgr.y (mgr.34107) 12082 : cluster [DBG] pgmap v12577: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:42 smithi121 bash[36902]: audit 2022-04-22T23:20:41.735143+0000 mon.a (mon.0) 16664 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:43.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:43 smithi121 bash[36902]: cluster 2022-04-22T23:20:43.207364+0000 mgr.y (mgr.34107) 12083 : cluster [DBG] pgmap v12578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:43 smithi121 bash[36902]: audit 2022-04-22T23:20:43.368604+0000 mon.a (mon.0) 16665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:43 smithi115 bash[45495]: cluster 2022-04-22T23:20:43.207364+0000 mgr.y (mgr.34107) 12083 : cluster [DBG] pgmap v12578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:43 smithi115 bash[45495]: audit 2022-04-22T23:20:43.368604+0000 mon.a (mon.0) 16665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:43 smithi115 bash[46934]: cluster 2022-04-22T23:20:43.207364+0000 mgr.y (mgr.34107) 12083 : cluster [DBG] pgmap v12578: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:43 smithi115 bash[46934]: audit 2022-04-22T23:20:43.368604+0000 mon.a (mon.0) 16665 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:45.135 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:44.873Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:45 smithi121 bash[36902]: audit 2022-04-22T23:20:45.079733+0000 mon.a (mon.0) 16666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:45 smithi115 bash[45495]: audit 2022-04-22T23:20:45.079733+0000 mon.a (mon.0) 16666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:45 smithi115 bash[46934]: audit 2022-04-22T23:20:45.079733+0000 mon.a (mon.0) 16666 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:46 smithi121 bash[36902]: cluster 2022-04-22T23:20:45.208350+0000 mgr.y (mgr.34107) 12084 : cluster [DBG] pgmap v12579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:46 smithi115 bash[46934]: cluster 2022-04-22T23:20:45.208350+0000 mgr.y (mgr.34107) 12084 : cluster [DBG] pgmap v12579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:46 smithi115 bash[45495]: cluster 2022-04-22T23:20:45.208350+0000 mgr.y (mgr.34107) 12084 : cluster [DBG] pgmap v12579: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:47 smithi121 bash[36902]: audit 2022-04-22T23:20:46.738199+0000 mon.a (mon.0) 16667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:47 smithi115 bash[46934]: audit 2022-04-22T23:20:46.738199+0000 mon.a (mon.0) 16667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:47 smithi115 bash[45495]: audit 2022-04-22T23:20:46.738199+0000 mon.a (mon.0) 16667 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:48 smithi121 bash[36902]: cluster 2022-04-22T23:20:47.209460+0000 mgr.y (mgr.34107) 12085 : cluster [DBG] pgmap v12580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:48 smithi115 bash[45495]: cluster 2022-04-22T23:20:47.209460+0000 mgr.y (mgr.34107) 12085 : cluster [DBG] pgmap v12580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:48 smithi115 bash[46934]: cluster 2022-04-22T23:20:47.209460+0000 mgr.y (mgr.34107) 12085 : cluster [DBG] pgmap v12580: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:49 smithi121 bash[36902]: audit 2022-04-22T23:20:48.371847+0000 mon.a (mon.0) 16668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:49 smithi115 bash[45495]: audit 2022-04-22T23:20:48.371847+0000 mon.a (mon.0) 16668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:49 smithi115 bash[46934]: audit 2022-04-22T23:20:48.371847+0000 mon.a (mon.0) 16668 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:50 smithi121 bash[36902]: cluster 2022-04-22T23:20:49.210589+0000 mgr.y (mgr.34107) 12086 : cluster [DBG] pgmap v12581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:50 smithi121 bash[36902]: audit 2022-04-22T23:20:50.083227+0000 mon.a (mon.0) 16669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:50 smithi115 bash[45495]: cluster 2022-04-22T23:20:49.210589+0000 mgr.y (mgr.34107) 12086 : cluster [DBG] pgmap v12581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:50 smithi115 bash[45495]: audit 2022-04-22T23:20:50.083227+0000 mon.a (mon.0) 16669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:50 smithi115 bash[46934]: cluster 2022-04-22T23:20:49.210589+0000 mgr.y (mgr.34107) 12086 : cluster [DBG] pgmap v12581: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:50 smithi115 bash[46934]: audit 2022-04-22T23:20:50.083227+0000 mon.a (mon.0) 16669 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:52 smithi115 bash[46934]: cluster 2022-04-22T23:20:51.212206+0000 mgr.y (mgr.34107) 12087 : cluster [DBG] pgmap v12582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:52 smithi115 bash[46934]: audit 2022-04-22T23:20:51.741081+0000 mon.a (mon.0) 16670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:52 smithi115 bash[45495]: cluster 2022-04-22T23:20:51.212206+0000 mgr.y (mgr.34107) 12087 : cluster [DBG] pgmap v12582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:52 smithi115 bash[45495]: audit 2022-04-22T23:20:51.741081+0000 mon.a (mon.0) 16670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:52 smithi121 bash[36902]: cluster 2022-04-22T23:20:51.212206+0000 mgr.y (mgr.34107) 12087 : cluster [DBG] pgmap v12582: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:52 smithi121 bash[36902]: audit 2022-04-22T23:20:51.741081+0000 mon.a (mon.0) 16670 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:53.016Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:53 smithi121 bash[36902]: cluster 2022-04-22T23:20:53.213300+0000 mgr.y (mgr.34107) 12088 : cluster [DBG] pgmap v12583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:53 smithi121 bash[36902]: audit 2022-04-22T23:20:53.374779+0000 mon.a (mon.0) 16671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:54.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:53 smithi115 bash[45495]: cluster 2022-04-22T23:20:53.213300+0000 mgr.y (mgr.34107) 12088 : cluster [DBG] pgmap v12583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:53 smithi115 bash[45495]: audit 2022-04-22T23:20:53.374779+0000 mon.a (mon.0) 16671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:53 smithi115 bash[46934]: cluster 2022-04-22T23:20:53.213300+0000 mgr.y (mgr.34107) 12088 : cluster [DBG] pgmap v12583: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:53 smithi115 bash[46934]: audit 2022-04-22T23:20:53.374779+0000 mon.a (mon.0) 16671 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:55.143 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:20:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:20:54.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:20:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:55 smithi121 bash[36902]: audit 2022-04-22T23:20:55.086939+0000 mon.a (mon.0) 16672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:55 smithi115 bash[45495]: audit 2022-04-22T23:20:55.086939+0000 mon.a (mon.0) 16672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:55 smithi115 bash[46934]: audit 2022-04-22T23:20:55.086939+0000 mon.a (mon.0) 16672 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:20:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:56 smithi121 bash[36902]: cluster 2022-04-22T23:20:55.214485+0000 mgr.y (mgr.34107) 12089 : cluster [DBG] pgmap v12584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:56 smithi115 bash[45495]: cluster 2022-04-22T23:20:55.214485+0000 mgr.y (mgr.34107) 12089 : cluster [DBG] pgmap v12584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:56 smithi115 bash[46934]: cluster 2022-04-22T23:20:55.214485+0000 mgr.y (mgr.34107) 12089 : cluster [DBG] pgmap v12584: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:57 smithi121 bash[36902]: audit 2022-04-22T23:20:56.744195+0000 mon.a (mon.0) 16673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:57 smithi121 bash[36902]: audit 2022-04-22T23:20:56.970710+0000 mon.a (mon.0) 16674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:20:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:57 smithi121 bash[36902]: audit 2022-04-22T23:20:56.973846+0000 mon.a (mon.0) 16675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:20:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:57 smithi121 bash[36902]: audit 2022-04-22T23:20:56.975902+0000 mon.a (mon.0) 16676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:20:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[45495]: audit 2022-04-22T23:20:56.744195+0000 mon.a (mon.0) 16673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[45495]: audit 2022-04-22T23:20:56.970710+0000 mon.a (mon.0) 16674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:20:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[45495]: audit 2022-04-22T23:20:56.973846+0000 mon.a (mon.0) 16675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:20:57.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[45495]: audit 2022-04-22T23:20:56.975902+0000 mon.a (mon.0) 16676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:20:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[46934]: audit 2022-04-22T23:20:56.744195+0000 mon.a (mon.0) 16673 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:20:57.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[46934]: audit 2022-04-22T23:20:56.970710+0000 mon.a (mon.0) 16674 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:20:57.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[46934]: audit 2022-04-22T23:20:56.973846+0000 mon.a (mon.0) 16675 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:20:57.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:57 smithi115 bash[46934]: audit 2022-04-22T23:20:56.975902+0000 mon.a (mon.0) 16676 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:20:58.245 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:20:58.593 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:20:58.594 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 5h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:20:58.595 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:20:58.596 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:20:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:58 smithi121 bash[36902]: cluster 2022-04-22T23:20:57.215577+0000 mgr.y (mgr.34107) 12090 : cluster [DBG] pgmap v12585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:58 smithi121 bash[36902]: audit 2022-04-22T23:20:57.318461+0000 mon.a (mon.0) 16677 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:20:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:58 smithi115 bash[45495]: cluster 2022-04-22T23:20:57.215577+0000 mgr.y (mgr.34107) 12090 : cluster [DBG] pgmap v12585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:58 smithi115 bash[45495]: audit 2022-04-22T23:20:57.318461+0000 mon.a (mon.0) 16677 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:20:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:58 smithi115 bash[46934]: cluster 2022-04-22T23:20:57.215577+0000 mgr.y (mgr.34107) 12090 : cluster [DBG] pgmap v12585: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:58.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:58 smithi115 bash[46934]: audit 2022-04-22T23:20:57.318461+0000 mon.a (mon.0) 16677 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:20:58.995 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:20:58.995 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:58.996 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:20:58.997 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:59 smithi121 bash[36902]: audit 2022-04-22T23:20:58.227388+0000 mgr.y (mgr.34107) 12091 : audit [DBG] from='client.47853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:59 smithi121 bash[36902]: audit 2022-04-22T23:20:58.377742+0000 mon.a (mon.0) 16678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:59 smithi121 bash[36902]: audit 2022-04-22T23:20:58.571208+0000 mgr.y (mgr.34107) 12092 : audit [DBG] from='client.57799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:59 smithi121 bash[36902]: audit 2022-04-22T23:20:58.991746+0000 mon.a (mon.0) 16679 : audit [DBG] from='client.? 172.21.15.115:0/3631060011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:20:59 smithi121 bash[36902]: cluster 2022-04-22T23:20:59.216247+0000 mgr.y (mgr.34107) 12093 : cluster [DBG] pgmap v12586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[45495]: audit 2022-04-22T23:20:58.227388+0000 mgr.y (mgr.34107) 12091 : audit [DBG] from='client.47853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[45495]: audit 2022-04-22T23:20:58.377742+0000 mon.a (mon.0) 16678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[45495]: audit 2022-04-22T23:20:58.571208+0000 mgr.y (mgr.34107) 12092 : audit [DBG] from='client.57799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[45495]: audit 2022-04-22T23:20:58.991746+0000 mon.a (mon.0) 16679 : audit [DBG] from='client.? 172.21.15.115:0/3631060011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[45495]: cluster 2022-04-22T23:20:59.216247+0000 mgr.y (mgr.34107) 12093 : cluster [DBG] pgmap v12586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:20:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[46934]: audit 2022-04-22T23:20:58.227388+0000 mgr.y (mgr.34107) 12091 : audit [DBG] from='client.47853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[46934]: audit 2022-04-22T23:20:58.377742+0000 mon.a (mon.0) 16678 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:20:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[46934]: audit 2022-04-22T23:20:58.571208+0000 mgr.y (mgr.34107) 12092 : audit [DBG] from='client.57799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:20:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[46934]: audit 2022-04-22T23:20:58.991746+0000 mon.a (mon.0) 16679 : audit [DBG] from='client.? 172.21.15.115:0/3631060011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:20:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:20:59 smithi115 bash[46934]: cluster 2022-04-22T23:20:59.216247+0000 mgr.y (mgr.34107) 12093 : cluster [DBG] pgmap v12586: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:00 smithi121 bash[36902]: audit 2022-04-22T23:21:00.090809+0000 mon.a (mon.0) 16680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:00 smithi115 bash[45495]: audit 2022-04-22T23:21:00.090809+0000 mon.a (mon.0) 16680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:00 smithi115 bash[46934]: audit 2022-04-22T23:21:00.090809+0000 mon.a (mon.0) 16680 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:01 smithi121 bash[36902]: cluster 2022-04-22T23:21:01.217256+0000 mgr.y (mgr.34107) 12094 : cluster [DBG] pgmap v12587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:01 smithi115 bash[45495]: cluster 2022-04-22T23:21:01.217256+0000 mgr.y (mgr.34107) 12094 : cluster [DBG] pgmap v12587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:01 smithi115 bash[46934]: cluster 2022-04-22T23:21:01.217256+0000 mgr.y (mgr.34107) 12094 : cluster [DBG] pgmap v12587: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:02 smithi121 bash[36902]: audit 2022-04-22T23:21:01.747166+0000 mon.a (mon.0) 16681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:02 smithi115 bash[45495]: audit 2022-04-22T23:21:01.747166+0000 mon.a (mon.0) 16681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:02 smithi115 bash[46934]: audit 2022-04-22T23:21:01.747166+0000 mon.a (mon.0) 16681 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:03.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:03 smithi121 bash[36902]: cluster 2022-04-22T23:21:03.218416+0000 mgr.y (mgr.34107) 12095 : cluster [DBG] pgmap v12588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:03 smithi121 bash[36902]: audit 2022-04-22T23:21:03.380617+0000 mon.a (mon.0) 16682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:03 smithi115 bash[45495]: cluster 2022-04-22T23:21:03.218416+0000 mgr.y (mgr.34107) 12095 : cluster [DBG] pgmap v12588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:03 smithi115 bash[45495]: audit 2022-04-22T23:21:03.380617+0000 mon.a (mon.0) 16682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:03 smithi115 bash[46934]: cluster 2022-04-22T23:21:03.218416+0000 mgr.y (mgr.34107) 12095 : cluster [DBG] pgmap v12588: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:03 smithi115 bash[46934]: audit 2022-04-22T23:21:03.380617+0000 mon.a (mon.0) 16682 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:05.150 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:04.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:05 smithi121 bash[36902]: audit 2022-04-22T23:21:05.094318+0000 mon.a (mon.0) 16683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:05.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:05 smithi115 bash[45495]: audit 2022-04-22T23:21:05.094318+0000 mon.a (mon.0) 16683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:05 smithi115 bash[46934]: audit 2022-04-22T23:21:05.094318+0000 mon.a (mon.0) 16683 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:06 smithi121 bash[36902]: cluster 2022-04-22T23:21:05.219553+0000 mgr.y (mgr.34107) 12096 : cluster [DBG] pgmap v12589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:06 smithi115 bash[45495]: cluster 2022-04-22T23:21:05.219553+0000 mgr.y (mgr.34107) 12096 : cluster [DBG] pgmap v12589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:06 smithi115 bash[46934]: cluster 2022-04-22T23:21:05.219553+0000 mgr.y (mgr.34107) 12096 : cluster [DBG] pgmap v12589: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:07 smithi121 bash[36902]: audit 2022-04-22T23:21:06.750459+0000 mon.a (mon.0) 16684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:07 smithi115 bash[45495]: audit 2022-04-22T23:21:06.750459+0000 mon.a (mon.0) 16684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:07 smithi115 bash[46934]: audit 2022-04-22T23:21:06.750459+0000 mon.a (mon.0) 16684 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:08 smithi121 bash[36902]: cluster 2022-04-22T23:21:07.220290+0000 mgr.y (mgr.34107) 12097 : cluster [DBG] pgmap v12590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:08 smithi115 bash[45495]: cluster 2022-04-22T23:21:07.220290+0000 mgr.y (mgr.34107) 12097 : cluster [DBG] pgmap v12590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:08 smithi115 bash[46934]: cluster 2022-04-22T23:21:07.220290+0000 mgr.y (mgr.34107) 12097 : cluster [DBG] pgmap v12590: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:09 smithi121 bash[36902]: audit 2022-04-22T23:21:08.383823+0000 mon.a (mon.0) 16685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:09 smithi115 bash[45495]: audit 2022-04-22T23:21:08.383823+0000 mon.a (mon.0) 16685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:09 smithi115 bash[46934]: audit 2022-04-22T23:21:08.383823+0000 mon.a (mon.0) 16685 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:10 smithi121 bash[36902]: cluster 2022-04-22T23:21:09.221463+0000 mgr.y (mgr.34107) 12098 : cluster [DBG] pgmap v12591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:10 smithi121 bash[36902]: audit 2022-04-22T23:21:10.097927+0000 mon.a (mon.0) 16686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:10 smithi115 bash[45495]: cluster 2022-04-22T23:21:09.221463+0000 mgr.y (mgr.34107) 12098 : cluster [DBG] pgmap v12591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:10 smithi115 bash[45495]: audit 2022-04-22T23:21:10.097927+0000 mon.a (mon.0) 16686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:10 smithi115 bash[46934]: cluster 2022-04-22T23:21:09.221463+0000 mgr.y (mgr.34107) 12098 : cluster [DBG] pgmap v12591: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:10 smithi115 bash[46934]: audit 2022-04-22T23:21:10.097927+0000 mon.a (mon.0) 16686 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:12 smithi115 bash[45495]: cluster 2022-04-22T23:21:11.222562+0000 mgr.y (mgr.34107) 12099 : cluster [DBG] pgmap v12592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:12 smithi115 bash[45495]: audit 2022-04-22T23:21:11.753416+0000 mon.a (mon.0) 16687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:12 smithi115 bash[46934]: cluster 2022-04-22T23:21:11.222562+0000 mgr.y (mgr.34107) 12099 : cluster [DBG] pgmap v12592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:12 smithi115 bash[46934]: audit 2022-04-22T23:21:11.753416+0000 mon.a (mon.0) 16687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:12 smithi121 bash[36902]: cluster 2022-04-22T23:21:11.222562+0000 mgr.y (mgr.34107) 12099 : cluster [DBG] pgmap v12592: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:12 smithi121 bash[36902]: audit 2022-04-22T23:21:11.753416+0000 mon.a (mon.0) 16687 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:13.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:13 smithi121 bash[36902]: cluster 2022-04-22T23:21:13.223743+0000 mgr.y (mgr.34107) 12100 : cluster [DBG] pgmap v12593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:13 smithi121 bash[36902]: audit 2022-04-22T23:21:13.386780+0000 mon.a (mon.0) 16688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:13 smithi115 bash[45495]: cluster 2022-04-22T23:21:13.223743+0000 mgr.y (mgr.34107) 12100 : cluster [DBG] pgmap v12593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:13 smithi115 bash[45495]: audit 2022-04-22T23:21:13.386780+0000 mon.a (mon.0) 16688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:13 smithi115 bash[46934]: cluster 2022-04-22T23:21:13.223743+0000 mgr.y (mgr.34107) 12100 : cluster [DBG] pgmap v12593: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:13 smithi115 bash[46934]: audit 2022-04-22T23:21:13.386780+0000 mon.a (mon.0) 16688 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:15.157 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:14.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:15 smithi121 bash[36902]: audit 2022-04-22T23:21:15.101679+0000 mon.a (mon.0) 16689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:15 smithi115 bash[45495]: audit 2022-04-22T23:21:15.101679+0000 mon.a (mon.0) 16689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:15 smithi115 bash[46934]: audit 2022-04-22T23:21:15.101679+0000 mon.a (mon.0) 16689 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:16 smithi121 bash[36902]: cluster 2022-04-22T23:21:15.224533+0000 mgr.y (mgr.34107) 12101 : cluster [DBG] pgmap v12594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:16 smithi115 bash[45495]: cluster 2022-04-22T23:21:15.224533+0000 mgr.y (mgr.34107) 12101 : cluster [DBG] pgmap v12594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:16 smithi115 bash[46934]: cluster 2022-04-22T23:21:15.224533+0000 mgr.y (mgr.34107) 12101 : cluster [DBG] pgmap v12594: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:17 smithi121 bash[36902]: audit 2022-04-22T23:21:16.756626+0000 mon.a (mon.0) 16690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:17 smithi115 bash[45495]: audit 2022-04-22T23:21:16.756626+0000 mon.a (mon.0) 16690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:17 smithi115 bash[46934]: audit 2022-04-22T23:21:16.756626+0000 mon.a (mon.0) 16690 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:18 smithi121 bash[36902]: cluster 2022-04-22T23:21:17.225617+0000 mgr.y (mgr.34107) 12102 : cluster [DBG] pgmap v12595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:18 smithi121 bash[36902]: audit 2022-04-22T23:21:17.361935+0000 mon.a (mon.0) 16691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:21:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:18 smithi121 bash[36902]: audit 2022-04-22T23:21:17.364599+0000 mon.a (mon.0) 16692 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[45495]: cluster 2022-04-22T23:21:17.225617+0000 mgr.y (mgr.34107) 12102 : cluster [DBG] pgmap v12595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[45495]: audit 2022-04-22T23:21:17.361935+0000 mon.a (mon.0) 16691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[45495]: audit 2022-04-22T23:21:17.364599+0000 mon.a (mon.0) 16692 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[46934]: cluster 2022-04-22T23:21:17.225617+0000 mgr.y (mgr.34107) 12102 : cluster [DBG] pgmap v12595: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[46934]: audit 2022-04-22T23:21:17.361935+0000 mon.a (mon.0) 16691 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:21:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:18 smithi115 bash[46934]: audit 2022-04-22T23:21:17.364599+0000 mon.a (mon.0) 16692 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:21:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:19 smithi121 bash[36902]: audit 2022-04-22T23:21:18.389895+0000 mon.a (mon.0) 16693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:19 smithi115 bash[45495]: audit 2022-04-22T23:21:18.389895+0000 mon.a (mon.0) 16693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:19 smithi115 bash[46934]: audit 2022-04-22T23:21:18.389895+0000 mon.a (mon.0) 16693 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:20 smithi121 bash[36902]: cluster 2022-04-22T23:21:19.226729+0000 mgr.y (mgr.34107) 12103 : cluster [DBG] pgmap v12596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:20 smithi121 bash[36902]: audit 2022-04-22T23:21:20.105310+0000 mon.a (mon.0) 16694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:20.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:20 smithi115 bash[45495]: cluster 2022-04-22T23:21:19.226729+0000 mgr.y (mgr.34107) 12103 : cluster [DBG] pgmap v12596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:20 smithi115 bash[45495]: audit 2022-04-22T23:21:20.105310+0000 mon.a (mon.0) 16694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:20 smithi115 bash[46934]: cluster 2022-04-22T23:21:19.226729+0000 mgr.y (mgr.34107) 12103 : cluster [DBG] pgmap v12596: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:20 smithi115 bash[46934]: audit 2022-04-22T23:21:20.105310+0000 mon.a (mon.0) 16694 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:22 smithi115 bash[45495]: cluster 2022-04-22T23:21:21.227818+0000 mgr.y (mgr.34107) 12104 : cluster [DBG] pgmap v12597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:22 smithi115 bash[45495]: audit 2022-04-22T23:21:21.759762+0000 mon.a (mon.0) 16695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:22 smithi115 bash[46934]: cluster 2022-04-22T23:21:21.227818+0000 mgr.y (mgr.34107) 12104 : cluster [DBG] pgmap v12597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:22 smithi115 bash[46934]: audit 2022-04-22T23:21:21.759762+0000 mon.a (mon.0) 16695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:22 smithi121 bash[36902]: cluster 2022-04-22T23:21:21.227818+0000 mgr.y (mgr.34107) 12104 : cluster [DBG] pgmap v12597: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:22 smithi121 bash[36902]: audit 2022-04-22T23:21:21.759762+0000 mon.a (mon.0) 16695 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:23.017Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:23 smithi121 bash[36902]: cluster 2022-04-22T23:21:23.229004+0000 mgr.y (mgr.34107) 12105 : cluster [DBG] pgmap v12598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:24.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:23 smithi121 bash[36902]: audit 2022-04-22T23:21:23.393090+0000 mon.a (mon.0) 16696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:23 smithi115 bash[45495]: cluster 2022-04-22T23:21:23.229004+0000 mgr.y (mgr.34107) 12105 : cluster [DBG] pgmap v12598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:23 smithi115 bash[45495]: audit 2022-04-22T23:21:23.393090+0000 mon.a (mon.0) 16696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:23 smithi115 bash[46934]: cluster 2022-04-22T23:21:23.229004+0000 mgr.y (mgr.34107) 12105 : cluster [DBG] pgmap v12598: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:23 smithi115 bash[46934]: audit 2022-04-22T23:21:23.393090+0000 mon.a (mon.0) 16696 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:25.165 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:24.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:25 smithi121 bash[36902]: audit 2022-04-22T23:21:25.108823+0000 mon.a (mon.0) 16697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:25 smithi115 bash[45495]: audit 2022-04-22T23:21:25.108823+0000 mon.a (mon.0) 16697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:25 smithi115 bash[46934]: audit 2022-04-22T23:21:25.108823+0000 mon.a (mon.0) 16697 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:26 smithi121 bash[36902]: cluster 2022-04-22T23:21:25.230151+0000 mgr.y (mgr.34107) 12106 : cluster [DBG] pgmap v12599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:26 smithi115 bash[45495]: cluster 2022-04-22T23:21:25.230151+0000 mgr.y (mgr.34107) 12106 : cluster [DBG] pgmap v12599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:26 smithi115 bash[46934]: cluster 2022-04-22T23:21:25.230151+0000 mgr.y (mgr.34107) 12106 : cluster [DBG] pgmap v12599: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:27 smithi121 bash[36902]: audit 2022-04-22T23:21:26.763108+0000 mon.a (mon.0) 16698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:27 smithi115 bash[45495]: audit 2022-04-22T23:21:26.763108+0000 mon.a (mon.0) 16698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:27 smithi115 bash[46934]: audit 2022-04-22T23:21:26.763108+0000 mon.a (mon.0) 16698 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:28 smithi121 bash[36902]: cluster 2022-04-22T23:21:27.231290+0000 mgr.y (mgr.34107) 12107 : cluster [DBG] pgmap v12600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:28 smithi115 bash[45495]: cluster 2022-04-22T23:21:27.231290+0000 mgr.y (mgr.34107) 12107 : cluster [DBG] pgmap v12600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:28 smithi115 bash[46934]: cluster 2022-04-22T23:21:27.231290+0000 mgr.y (mgr.34107) 12107 : cluster [DBG] pgmap v12600: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:29.375 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:21:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:29 smithi121 bash[36902]: audit 2022-04-22T23:21:28.396077+0000 mon.a (mon.0) 16699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:29.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:29 smithi115 bash[46934]: audit 2022-04-22T23:21:28.396077+0000 mon.a (mon.0) 16699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:29 smithi115 bash[45495]: audit 2022-04-22T23:21:28.396077+0000 mon.a (mon.0) 16699 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:29.722 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:21:29.722 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:21:29.722 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 4m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:21:29.722 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:21:29.723 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:21:29.723 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 4m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:21:29.723 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 3m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:21:29.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 3m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:21:29.723 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 4m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 3m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 3m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 3m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:21:29.724 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 3m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 5h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:21:29.725 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:21:30.130 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:21:30.130 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:21:30.130 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:21:30.131 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:21:30.132 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:21:30.132 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:21:30.132 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:21:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:30 smithi121 bash[36902]: cluster 2022-04-22T23:21:29.232385+0000 mgr.y (mgr.34107) 12108 : cluster [DBG] pgmap v12601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:30 smithi121 bash[36902]: audit 2022-04-22T23:21:30.112297+0000 mon.a (mon.0) 16700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:30 smithi121 bash[36902]: audit 2022-04-22T23:21:30.126853+0000 mon.c (mon.1) 410 : audit [DBG] from='client.? 172.21.15.115:0/3968621922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:21:30.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[45495]: cluster 2022-04-22T23:21:29.232385+0000 mgr.y (mgr.34107) 12108 : cluster [DBG] pgmap v12601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[45495]: audit 2022-04-22T23:21:30.112297+0000 mon.a (mon.0) 16700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[45495]: audit 2022-04-22T23:21:30.126853+0000 mon.c (mon.1) 410 : audit [DBG] from='client.? 172.21.15.115:0/3968621922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:21:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[46934]: cluster 2022-04-22T23:21:29.232385+0000 mgr.y (mgr.34107) 12108 : cluster [DBG] pgmap v12601: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[46934]: audit 2022-04-22T23:21:30.112297+0000 mon.a (mon.0) 16700 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:30 smithi115 bash[46934]: audit 2022-04-22T23:21:30.126853+0000 mon.c (mon.1) 410 : audit [DBG] from='client.? 172.21.15.115:0/3968621922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:21:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:31 smithi121 bash[36902]: audit 2022-04-22T23:21:29.354423+0000 mgr.y (mgr.34107) 12109 : audit [DBG] from='client.57811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:31 smithi121 bash[36902]: audit 2022-04-22T23:21:29.699409+0000 mgr.y (mgr.34107) 12110 : audit [DBG] from='client.47877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:31 smithi115 bash[45495]: audit 2022-04-22T23:21:29.354423+0000 mgr.y (mgr.34107) 12109 : audit [DBG] from='client.57811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:31 smithi115 bash[45495]: audit 2022-04-22T23:21:29.699409+0000 mgr.y (mgr.34107) 12110 : audit [DBG] from='client.47877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:31 smithi115 bash[46934]: audit 2022-04-22T23:21:29.354423+0000 mgr.y (mgr.34107) 12109 : audit [DBG] from='client.57811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:31 smithi115 bash[46934]: audit 2022-04-22T23:21:29.699409+0000 mgr.y (mgr.34107) 12110 : audit [DBG] from='client.47877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:21:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:32 smithi121 bash[36902]: cluster 2022-04-22T23:21:31.233490+0000 mgr.y (mgr.34107) 12111 : cluster [DBG] pgmap v12602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:32 smithi121 bash[36902]: audit 2022-04-22T23:21:31.766229+0000 mon.a (mon.0) 16701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:32 smithi115 bash[45495]: cluster 2022-04-22T23:21:31.233490+0000 mgr.y (mgr.34107) 12111 : cluster [DBG] pgmap v12602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:32 smithi115 bash[45495]: audit 2022-04-22T23:21:31.766229+0000 mon.a (mon.0) 16701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:32 smithi115 bash[46934]: cluster 2022-04-22T23:21:31.233490+0000 mgr.y (mgr.34107) 12111 : cluster [DBG] pgmap v12602: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:32 smithi115 bash[46934]: audit 2022-04-22T23:21:31.766229+0000 mon.a (mon.0) 16701 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:33.018Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:33 smithi121 bash[36902]: cluster 2022-04-22T23:21:33.234241+0000 mgr.y (mgr.34107) 12112 : cluster [DBG] pgmap v12603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:33 smithi121 bash[36902]: audit 2022-04-22T23:21:33.399339+0000 mon.a (mon.0) 16702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:33 smithi115 bash[45495]: cluster 2022-04-22T23:21:33.234241+0000 mgr.y (mgr.34107) 12112 : cluster [DBG] pgmap v12603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:33 smithi115 bash[45495]: audit 2022-04-22T23:21:33.399339+0000 mon.a (mon.0) 16702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:33 smithi115 bash[46934]: cluster 2022-04-22T23:21:33.234241+0000 mgr.y (mgr.34107) 12112 : cluster [DBG] pgmap v12603: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:33 smithi115 bash[46934]: audit 2022-04-22T23:21:33.399339+0000 mon.a (mon.0) 16702 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:35.171 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:34.874Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:35 smithi121 bash[36902]: audit 2022-04-22T23:21:35.115875+0000 mon.a (mon.0) 16703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:35.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:35 smithi115 bash[45495]: audit 2022-04-22T23:21:35.115875+0000 mon.a (mon.0) 16703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:35 smithi115 bash[46934]: audit 2022-04-22T23:21:35.115875+0000 mon.a (mon.0) 16703 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:36 smithi121 bash[36902]: cluster 2022-04-22T23:21:35.235253+0000 mgr.y (mgr.34107) 12113 : cluster [DBG] pgmap v12604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:36 smithi115 bash[45495]: cluster 2022-04-22T23:21:35.235253+0000 mgr.y (mgr.34107) 12113 : cluster [DBG] pgmap v12604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:36 smithi115 bash[46934]: cluster 2022-04-22T23:21:35.235253+0000 mgr.y (mgr.34107) 12113 : cluster [DBG] pgmap v12604: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:37 smithi121 bash[36902]: audit 2022-04-22T23:21:36.769251+0000 mon.a (mon.0) 16704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:37 smithi115 bash[45495]: audit 2022-04-22T23:21:36.769251+0000 mon.a (mon.0) 16704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:37 smithi115 bash[46934]: audit 2022-04-22T23:21:36.769251+0000 mon.a (mon.0) 16704 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:38 smithi121 bash[36902]: cluster 2022-04-22T23:21:37.236463+0000 mgr.y (mgr.34107) 12114 : cluster [DBG] pgmap v12605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:38 smithi115 bash[45495]: cluster 2022-04-22T23:21:37.236463+0000 mgr.y (mgr.34107) 12114 : cluster [DBG] pgmap v12605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:38 smithi115 bash[46934]: cluster 2022-04-22T23:21:37.236463+0000 mgr.y (mgr.34107) 12114 : cluster [DBG] pgmap v12605: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:39 smithi121 bash[36902]: audit 2022-04-22T23:21:38.402405+0000 mon.a (mon.0) 16705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:39 smithi115 bash[45495]: audit 2022-04-22T23:21:38.402405+0000 mon.a (mon.0) 16705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:39 smithi115 bash[46934]: audit 2022-04-22T23:21:38.402405+0000 mon.a (mon.0) 16705 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:40 smithi121 bash[36902]: cluster 2022-04-22T23:21:39.237564+0000 mgr.y (mgr.34107) 12115 : cluster [DBG] pgmap v12606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:40.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:40 smithi121 bash[36902]: audit 2022-04-22T23:21:40.119704+0000 mon.a (mon.0) 16706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:40 smithi115 bash[46934]: cluster 2022-04-22T23:21:39.237564+0000 mgr.y (mgr.34107) 12115 : cluster [DBG] pgmap v12606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:40.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:40 smithi115 bash[46934]: audit 2022-04-22T23:21:40.119704+0000 mon.a (mon.0) 16706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:40 smithi115 bash[45495]: cluster 2022-04-22T23:21:39.237564+0000 mgr.y (mgr.34107) 12115 : cluster [DBG] pgmap v12606: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:40.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:40 smithi115 bash[45495]: audit 2022-04-22T23:21:40.119704+0000 mon.a (mon.0) 16706 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:42 smithi121 bash[36902]: cluster 2022-04-22T23:21:41.238607+0000 mgr.y (mgr.34107) 12116 : cluster [DBG] pgmap v12607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:42 smithi121 bash[36902]: audit 2022-04-22T23:21:41.772343+0000 mon.a (mon.0) 16707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:42 smithi115 bash[45495]: cluster 2022-04-22T23:21:41.238607+0000 mgr.y (mgr.34107) 12116 : cluster [DBG] pgmap v12607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:42 smithi115 bash[45495]: audit 2022-04-22T23:21:41.772343+0000 mon.a (mon.0) 16707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:42 smithi115 bash[46934]: cluster 2022-04-22T23:21:41.238607+0000 mgr.y (mgr.34107) 12116 : cluster [DBG] pgmap v12607: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:42 smithi115 bash[46934]: audit 2022-04-22T23:21:41.772343+0000 mon.a (mon.0) 16707 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:43.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:43.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:43 smithi121 bash[36902]: cluster 2022-04-22T23:21:43.239350+0000 mgr.y (mgr.34107) 12117 : cluster [DBG] pgmap v12608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:43 smithi121 bash[36902]: audit 2022-04-22T23:21:43.405728+0000 mon.a (mon.0) 16708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:43 smithi115 bash[45495]: cluster 2022-04-22T23:21:43.239350+0000 mgr.y (mgr.34107) 12117 : cluster [DBG] pgmap v12608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:43 smithi115 bash[45495]: audit 2022-04-22T23:21:43.405728+0000 mon.a (mon.0) 16708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:43 smithi115 bash[46934]: cluster 2022-04-22T23:21:43.239350+0000 mgr.y (mgr.34107) 12117 : cluster [DBG] pgmap v12608: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:43 smithi115 bash[46934]: audit 2022-04-22T23:21:43.405728+0000 mon.a (mon.0) 16708 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:45.179 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:44.876Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:45 smithi121 bash[36902]: audit 2022-04-22T23:21:45.123293+0000 mon.a (mon.0) 16709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:45 smithi115 bash[45495]: audit 2022-04-22T23:21:45.123293+0000 mon.a (mon.0) 16709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:45 smithi115 bash[46934]: audit 2022-04-22T23:21:45.123293+0000 mon.a (mon.0) 16709 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:46 smithi121 bash[36902]: cluster 2022-04-22T23:21:45.240349+0000 mgr.y (mgr.34107) 12118 : cluster [DBG] pgmap v12609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:46 smithi115 bash[45495]: cluster 2022-04-22T23:21:45.240349+0000 mgr.y (mgr.34107) 12118 : cluster [DBG] pgmap v12609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:46 smithi115 bash[46934]: cluster 2022-04-22T23:21:45.240349+0000 mgr.y (mgr.34107) 12118 : cluster [DBG] pgmap v12609: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:47 smithi121 bash[36902]: audit 2022-04-22T23:21:46.775517+0000 mon.a (mon.0) 16710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:47 smithi115 bash[45495]: audit 2022-04-22T23:21:46.775517+0000 mon.a (mon.0) 16710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:47 smithi115 bash[46934]: audit 2022-04-22T23:21:46.775517+0000 mon.a (mon.0) 16710 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:48.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:48 smithi121 bash[36902]: cluster 2022-04-22T23:21:47.241483+0000 mgr.y (mgr.34107) 12119 : cluster [DBG] pgmap v12610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:48 smithi115 bash[45495]: cluster 2022-04-22T23:21:47.241483+0000 mgr.y (mgr.34107) 12119 : cluster [DBG] pgmap v12610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:48 smithi115 bash[46934]: cluster 2022-04-22T23:21:47.241483+0000 mgr.y (mgr.34107) 12119 : cluster [DBG] pgmap v12610: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:49 smithi121 bash[36902]: audit 2022-04-22T23:21:48.408888+0000 mon.a (mon.0) 16711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:49 smithi115 bash[45495]: audit 2022-04-22T23:21:48.408888+0000 mon.a (mon.0) 16711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:49 smithi115 bash[46934]: audit 2022-04-22T23:21:48.408888+0000 mon.a (mon.0) 16711 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:50 smithi121 bash[36902]: cluster 2022-04-22T23:21:49.242494+0000 mgr.y (mgr.34107) 12120 : cluster [DBG] pgmap v12611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:50 smithi121 bash[36902]: audit 2022-04-22T23:21:50.127145+0000 mon.a (mon.0) 16712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:50 smithi115 bash[45495]: cluster 2022-04-22T23:21:49.242494+0000 mgr.y (mgr.34107) 12120 : cluster [DBG] pgmap v12611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:50 smithi115 bash[45495]: audit 2022-04-22T23:21:50.127145+0000 mon.a (mon.0) 16712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:50 smithi115 bash[46934]: cluster 2022-04-22T23:21:49.242494+0000 mgr.y (mgr.34107) 12120 : cluster [DBG] pgmap v12611: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:50 smithi115 bash[46934]: audit 2022-04-22T23:21:50.127145+0000 mon.a (mon.0) 16712 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:52 smithi121 bash[36902]: cluster 2022-04-22T23:21:51.243632+0000 mgr.y (mgr.34107) 12121 : cluster [DBG] pgmap v12612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:52 smithi121 bash[36902]: audit 2022-04-22T23:21:51.778536+0000 mon.a (mon.0) 16713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:52.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:52 smithi115 bash[45495]: cluster 2022-04-22T23:21:51.243632+0000 mgr.y (mgr.34107) 12121 : cluster [DBG] pgmap v12612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:52 smithi115 bash[45495]: audit 2022-04-22T23:21:51.778536+0000 mon.a (mon.0) 16713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:52 smithi115 bash[46934]: cluster 2022-04-22T23:21:51.243632+0000 mgr.y (mgr.34107) 12121 : cluster [DBG] pgmap v12612: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:52 smithi115 bash[46934]: audit 2022-04-22T23:21:51.778536+0000 mon.a (mon.0) 16713 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:53.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:53 smithi121 bash[36902]: cluster 2022-04-22T23:21:53.244816+0000 mgr.y (mgr.34107) 12122 : cluster [DBG] pgmap v12613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:53 smithi121 bash[36902]: audit 2022-04-22T23:21:53.411860+0000 mon.a (mon.0) 16714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:53 smithi115 bash[45495]: cluster 2022-04-22T23:21:53.244816+0000 mgr.y (mgr.34107) 12122 : cluster [DBG] pgmap v12613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:53 smithi115 bash[45495]: audit 2022-04-22T23:21:53.411860+0000 mon.a (mon.0) 16714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:53 smithi115 bash[46934]: cluster 2022-04-22T23:21:53.244816+0000 mgr.y (mgr.34107) 12122 : cluster [DBG] pgmap v12613: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:53 smithi115 bash[46934]: audit 2022-04-22T23:21:53.411860+0000 mon.a (mon.0) 16714 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:55.186 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:21:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:21:54.875Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:21:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:55 smithi121 bash[36902]: audit 2022-04-22T23:21:55.130665+0000 mon.a (mon.0) 16715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:55 smithi115 bash[45495]: audit 2022-04-22T23:21:55.130665+0000 mon.a (mon.0) 16715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:55 smithi115 bash[46934]: audit 2022-04-22T23:21:55.130665+0000 mon.a (mon.0) 16715 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:21:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:56 smithi121 bash[36902]: cluster 2022-04-22T23:21:55.246046+0000 mgr.y (mgr.34107) 12123 : cluster [DBG] pgmap v12614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:56 smithi115 bash[45495]: cluster 2022-04-22T23:21:55.246046+0000 mgr.y (mgr.34107) 12123 : cluster [DBG] pgmap v12614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:56 smithi115 bash[46934]: cluster 2022-04-22T23:21:55.246046+0000 mgr.y (mgr.34107) 12123 : cluster [DBG] pgmap v12614: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:57 smithi121 bash[36902]: audit 2022-04-22T23:21:56.781679+0000 mon.a (mon.0) 16716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:57.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:57 smithi115 bash[46934]: audit 2022-04-22T23:21:56.781679+0000 mon.a (mon.0) 16716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:57 smithi115 bash[45495]: audit 2022-04-22T23:21:56.781679+0000 mon.a (mon.0) 16716 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:21:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:58 smithi121 bash[36902]: cluster 2022-04-22T23:21:57.247115+0000 mgr.y (mgr.34107) 12124 : cluster [DBG] pgmap v12615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:58 smithi121 bash[36902]: audit 2022-04-22T23:21:57.327916+0000 mon.a (mon.0) 16717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:21:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:58 smithi121 bash[36902]: audit 2022-04-22T23:21:57.330719+0000 mon.a (mon.0) 16718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:21:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:58 smithi121 bash[36902]: audit 2022-04-22T23:21:57.332376+0000 mon.a (mon.0) 16719 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:21:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:58 smithi121 bash[36902]: audit 2022-04-22T23:21:57.707385+0000 mon.a (mon.0) 16720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:21:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[45495]: cluster 2022-04-22T23:21:57.247115+0000 mgr.y (mgr.34107) 12124 : cluster [DBG] pgmap v12615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[45495]: audit 2022-04-22T23:21:57.327916+0000 mon.a (mon.0) 16717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:21:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[45495]: audit 2022-04-22T23:21:57.330719+0000 mon.a (mon.0) 16718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:21:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[45495]: audit 2022-04-22T23:21:57.332376+0000 mon.a (mon.0) 16719 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:21:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[45495]: audit 2022-04-22T23:21:57.707385+0000 mon.a (mon.0) 16720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:21:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[46934]: cluster 2022-04-22T23:21:57.247115+0000 mgr.y (mgr.34107) 12124 : cluster [DBG] pgmap v12615: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:21:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[46934]: audit 2022-04-22T23:21:57.327916+0000 mon.a (mon.0) 16717 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:21:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[46934]: audit 2022-04-22T23:21:57.330719+0000 mon.a (mon.0) 16718 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:21:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[46934]: audit 2022-04-22T23:21:57.332376+0000 mon.a (mon.0) 16719 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:21:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:58 smithi115 bash[46934]: audit 2022-04-22T23:21:57.707385+0000 mon.a (mon.0) 16720 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:21:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:21:59 smithi121 bash[36902]: audit 2022-04-22T23:21:58.414990+0000 mon.a (mon.0) 16721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:59.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:21:59 smithi115 bash[45495]: audit 2022-04-22T23:21:58.414990+0000 mon.a (mon.0) 16721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:21:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:21:59 smithi115 bash[46934]: audit 2022-04-22T23:21:58.414990+0000 mon.a (mon.0) 16721 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:22:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:00 smithi121 bash[36902]: cluster 2022-04-22T23:21:59.248074+0000 mgr.y (mgr.34107) 12125 : cluster [DBG] pgmap v12616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:00 smithi121 bash[36902]: audit 2022-04-22T23:22:00.134335+0000 mon.a (mon.0) 16722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:00 smithi115 bash[46934]: cluster 2022-04-22T23:21:59.248074+0000 mgr.y (mgr.34107) 12125 : cluster [DBG] pgmap v12616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:00 smithi115 bash[46934]: audit 2022-04-22T23:22:00.134335+0000 mon.a (mon.0) 16722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:00 smithi115 bash[45495]: cluster 2022-04-22T23:21:59.248074+0000 mgr.y (mgr.34107) 12125 : cluster [DBG] pgmap v12616: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:00.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:00 smithi115 bash[45495]: audit 2022-04-22T23:22:00.134335+0000 mon.a (mon.0) 16722 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:00.861 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:22:00.861 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:22:00.861 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 5h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 5h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:22:00.862 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:22:00.863 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:22:00.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:22:00.864 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:22:00.864 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:22:01.275 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:22:01.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:22:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:22:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:01 smithi121 bash[36902]: audit 2022-04-22T23:22:01.271960+0000 mon.c (mon.1) 411 : audit [DBG] from='client.? 172.21.15.115:0/2934569675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:01 smithi115 bash[46934]: audit 2022-04-22T23:22:01.271960+0000 mon.c (mon.1) 411 : audit [DBG] from='client.? 172.21.15.115:0/2934569675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:01 smithi115 bash[45495]: audit 2022-04-22T23:22:01.271960+0000 mon.c (mon.1) 411 : audit [DBG] from='client.? 172.21.15.115:0/2934569675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:02 smithi121 bash[36902]: audit 2022-04-22T23:22:00.492186+0000 mgr.y (mgr.34107) 12126 : audit [DBG] from='client.47889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:02 smithi121 bash[36902]: audit 2022-04-22T23:22:00.836645+0000 mgr.y (mgr.34107) 12127 : audit [DBG] from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:02 smithi121 bash[36902]: cluster 2022-04-22T23:22:01.249107+0000 mgr.y (mgr.34107) 12128 : cluster [DBG] pgmap v12617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:02 smithi121 bash[36902]: audit 2022-04-22T23:22:01.784771+0000 mon.a (mon.0) 16723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[45495]: audit 2022-04-22T23:22:00.492186+0000 mgr.y (mgr.34107) 12126 : audit [DBG] from='client.47889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[45495]: audit 2022-04-22T23:22:00.836645+0000 mgr.y (mgr.34107) 12127 : audit [DBG] from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[45495]: cluster 2022-04-22T23:22:01.249107+0000 mgr.y (mgr.34107) 12128 : cluster [DBG] pgmap v12617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[45495]: audit 2022-04-22T23:22:01.784771+0000 mon.a (mon.0) 16723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[46934]: audit 2022-04-22T23:22:00.492186+0000 mgr.y (mgr.34107) 12126 : audit [DBG] from='client.47889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[46934]: audit 2022-04-22T23:22:00.836645+0000 mgr.y (mgr.34107) 12127 : audit [DBG] from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[46934]: cluster 2022-04-22T23:22:01.249107+0000 mgr.y (mgr.34107) 12128 : cluster [DBG] pgmap v12617: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:02 smithi115 bash[46934]: audit 2022-04-22T23:22:01.784771+0000 mon.a (mon.0) 16723 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:03.020Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:03 smithi121 bash[36902]: cluster 2022-04-22T23:22:03.250227+0000 mgr.y (mgr.34107) 12129 : cluster [DBG] pgmap v12618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:03 smithi121 bash[36902]: audit 2022-04-22T23:22:03.418192+0000 mon.a (mon.0) 16724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:03 smithi115 bash[45495]: cluster 2022-04-22T23:22:03.250227+0000 mgr.y (mgr.34107) 12129 : cluster [DBG] pgmap v12618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:03 smithi115 bash[45495]: audit 2022-04-22T23:22:03.418192+0000 mon.a (mon.0) 16724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:03 smithi115 bash[46934]: cluster 2022-04-22T23:22:03.250227+0000 mgr.y (mgr.34107) 12129 : cluster [DBG] pgmap v12618: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:03 smithi115 bash[46934]: audit 2022-04-22T23:22:03.418192+0000 mon.a (mon.0) 16724 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:05.194 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:04.877Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:05 smithi121 bash[36902]: audit 2022-04-22T23:22:05.137834+0000 mon.a (mon.0) 16725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:05 smithi115 bash[45495]: audit 2022-04-22T23:22:05.137834+0000 mon.a (mon.0) 16725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:05 smithi115 bash[46934]: audit 2022-04-22T23:22:05.137834+0000 mon.a (mon.0) 16725 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:06 smithi121 bash[36902]: cluster 2022-04-22T23:22:05.251325+0000 mgr.y (mgr.34107) 12130 : cluster [DBG] pgmap v12619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:06 smithi115 bash[45495]: cluster 2022-04-22T23:22:05.251325+0000 mgr.y (mgr.34107) 12130 : cluster [DBG] pgmap v12619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:06 smithi115 bash[46934]: cluster 2022-04-22T23:22:05.251325+0000 mgr.y (mgr.34107) 12130 : cluster [DBG] pgmap v12619: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:07 smithi121 bash[36902]: audit 2022-04-22T23:22:06.787981+0000 mon.a (mon.0) 16726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:07 smithi115 bash[45495]: audit 2022-04-22T23:22:06.787981+0000 mon.a (mon.0) 16726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:07 smithi115 bash[46934]: audit 2022-04-22T23:22:06.787981+0000 mon.a (mon.0) 16726 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:08 smithi121 bash[36902]: cluster 2022-04-22T23:22:07.252212+0000 mgr.y (mgr.34107) 12131 : cluster [DBG] pgmap v12620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:08 smithi115 bash[45495]: cluster 2022-04-22T23:22:07.252212+0000 mgr.y (mgr.34107) 12131 : cluster [DBG] pgmap v12620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:08 smithi115 bash[46934]: cluster 2022-04-22T23:22:07.252212+0000 mgr.y (mgr.34107) 12131 : cluster [DBG] pgmap v12620: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:09 smithi121 bash[36902]: audit 2022-04-22T23:22:08.421121+0000 mon.a (mon.0) 16727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:09 smithi115 bash[45495]: audit 2022-04-22T23:22:08.421121+0000 mon.a (mon.0) 16727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:09 smithi115 bash[46934]: audit 2022-04-22T23:22:08.421121+0000 mon.a (mon.0) 16727 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:10 smithi121 bash[36902]: cluster 2022-04-22T23:22:09.253204+0000 mgr.y (mgr.34107) 12132 : cluster [DBG] pgmap v12621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:10.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:10 smithi121 bash[36902]: audit 2022-04-22T23:22:10.141635+0000 mon.a (mon.0) 16728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:10 smithi115 bash[45495]: cluster 2022-04-22T23:22:09.253204+0000 mgr.y (mgr.34107) 12132 : cluster [DBG] pgmap v12621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:10 smithi115 bash[45495]: audit 2022-04-22T23:22:10.141635+0000 mon.a (mon.0) 16728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:10 smithi115 bash[46934]: cluster 2022-04-22T23:22:09.253204+0000 mgr.y (mgr.34107) 12132 : cluster [DBG] pgmap v12621: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:10 smithi115 bash[46934]: audit 2022-04-22T23:22:10.141635+0000 mon.a (mon.0) 16728 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:12 smithi121 bash[36902]: cluster 2022-04-22T23:22:11.254312+0000 mgr.y (mgr.34107) 12133 : cluster [DBG] pgmap v12622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:12 smithi121 bash[36902]: audit 2022-04-22T23:22:11.790847+0000 mon.a (mon.0) 16729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:12 smithi115 bash[46934]: cluster 2022-04-22T23:22:11.254312+0000 mgr.y (mgr.34107) 12133 : cluster [DBG] pgmap v12622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:12 smithi115 bash[46934]: audit 2022-04-22T23:22:11.790847+0000 mon.a (mon.0) 16729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:12 smithi115 bash[45495]: cluster 2022-04-22T23:22:11.254312+0000 mgr.y (mgr.34107) 12133 : cluster [DBG] pgmap v12622: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:12 smithi115 bash[45495]: audit 2022-04-22T23:22:11.790847+0000 mon.a (mon.0) 16729 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:13.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:13 smithi121 bash[36902]: cluster 2022-04-22T23:22:13.255459+0000 mgr.y (mgr.34107) 12134 : cluster [DBG] pgmap v12623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:14.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:13 smithi121 bash[36902]: audit 2022-04-22T23:22:13.424249+0000 mon.a (mon.0) 16730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:14.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:13 smithi115 bash[46934]: cluster 2022-04-22T23:22:13.255459+0000 mgr.y (mgr.34107) 12134 : cluster [DBG] pgmap v12623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:14.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:13 smithi115 bash[46934]: audit 2022-04-22T23:22:13.424249+0000 mon.a (mon.0) 16730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:14.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:13 smithi115 bash[45495]: cluster 2022-04-22T23:22:13.255459+0000 mgr.y (mgr.34107) 12134 : cluster [DBG] pgmap v12623: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:14.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:13 smithi115 bash[45495]: audit 2022-04-22T23:22:13.424249+0000 mon.a (mon.0) 16730 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:15.201 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:14.878Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:15 smithi121 bash[36902]: audit 2022-04-22T23:22:15.145449+0000 mon.a (mon.0) 16731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:15 smithi115 bash[45495]: audit 2022-04-22T23:22:15.145449+0000 mon.a (mon.0) 16731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:15 smithi115 bash[46934]: audit 2022-04-22T23:22:15.145449+0000 mon.a (mon.0) 16731 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:16 smithi121 bash[36902]: cluster 2022-04-22T23:22:15.256502+0000 mgr.y (mgr.34107) 12135 : cluster [DBG] pgmap v12624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:16 smithi115 bash[45495]: cluster 2022-04-22T23:22:15.256502+0000 mgr.y (mgr.34107) 12135 : cluster [DBG] pgmap v12624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:16 smithi115 bash[46934]: cluster 2022-04-22T23:22:15.256502+0000 mgr.y (mgr.34107) 12135 : cluster [DBG] pgmap v12624: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:17 smithi121 bash[36902]: audit 2022-04-22T23:22:16.793898+0000 mon.a (mon.0) 16732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:17 smithi115 bash[45495]: audit 2022-04-22T23:22:16.793898+0000 mon.a (mon.0) 16732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:17 smithi115 bash[46934]: audit 2022-04-22T23:22:16.793898+0000 mon.a (mon.0) 16732 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:18 smithi121 bash[36902]: cluster 2022-04-22T23:22:17.257361+0000 mgr.y (mgr.34107) 12136 : cluster [DBG] pgmap v12625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:18 smithi121 bash[36902]: audit 2022-04-22T23:22:17.373612+0000 mon.a (mon.0) 16733 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:22:18.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:18 smithi121 bash[36902]: audit 2022-04-22T23:22:17.374106+0000 mon.a (mon.0) 16734 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:22:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[45495]: cluster 2022-04-22T23:22:17.257361+0000 mgr.y (mgr.34107) 12136 : cluster [DBG] pgmap v12625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[45495]: audit 2022-04-22T23:22:17.373612+0000 mon.a (mon.0) 16733 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:22:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[45495]: audit 2022-04-22T23:22:17.374106+0000 mon.a (mon.0) 16734 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:22:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[46934]: cluster 2022-04-22T23:22:17.257361+0000 mgr.y (mgr.34107) 12136 : cluster [DBG] pgmap v12625: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[46934]: audit 2022-04-22T23:22:17.373612+0000 mon.a (mon.0) 16733 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:22:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:18 smithi115 bash[46934]: audit 2022-04-22T23:22:17.374106+0000 mon.a (mon.0) 16734 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:22:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:19 smithi121 bash[36902]: audit 2022-04-22T23:22:18.427522+0000 mon.a (mon.0) 16735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:19.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:19 smithi115 bash[45495]: audit 2022-04-22T23:22:18.427522+0000 mon.a (mon.0) 16735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:19 smithi115 bash[46934]: audit 2022-04-22T23:22:18.427522+0000 mon.a (mon.0) 16735 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:20 smithi121 bash[36902]: cluster 2022-04-22T23:22:19.258320+0000 mgr.y (mgr.34107) 12137 : cluster [DBG] pgmap v12626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:20.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:20 smithi121 bash[36902]: audit 2022-04-22T23:22:20.149429+0000 mon.a (mon.0) 16736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:20 smithi115 bash[45495]: cluster 2022-04-22T23:22:19.258320+0000 mgr.y (mgr.34107) 12137 : cluster [DBG] pgmap v12626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:20 smithi115 bash[45495]: audit 2022-04-22T23:22:20.149429+0000 mon.a (mon.0) 16736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:20 smithi115 bash[46934]: cluster 2022-04-22T23:22:19.258320+0000 mgr.y (mgr.34107) 12137 : cluster [DBG] pgmap v12626: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:20 smithi115 bash[46934]: audit 2022-04-22T23:22:20.149429+0000 mon.a (mon.0) 16736 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:22 smithi121 bash[36902]: cluster 2022-04-22T23:22:21.259409+0000 mgr.y (mgr.34107) 12138 : cluster [DBG] pgmap v12627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:22 smithi121 bash[36902]: audit 2022-04-22T23:22:21.796945+0000 mon.a (mon.0) 16737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:22 smithi115 bash[46934]: cluster 2022-04-22T23:22:21.259409+0000 mgr.y (mgr.34107) 12138 : cluster [DBG] pgmap v12627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:22 smithi115 bash[46934]: audit 2022-04-22T23:22:21.796945+0000 mon.a (mon.0) 16737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:22 smithi115 bash[45495]: cluster 2022-04-22T23:22:21.259409+0000 mgr.y (mgr.34107) 12138 : cluster [DBG] pgmap v12627: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:22 smithi115 bash[45495]: audit 2022-04-22T23:22:21.796945+0000 mon.a (mon.0) 16737 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:23.022Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:23 smithi121 bash[36902]: cluster 2022-04-22T23:22:23.260503+0000 mgr.y (mgr.34107) 12139 : cluster [DBG] pgmap v12628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:23 smithi121 bash[36902]: audit 2022-04-22T23:22:23.430623+0000 mon.a (mon.0) 16738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:23 smithi115 bash[45495]: cluster 2022-04-22T23:22:23.260503+0000 mgr.y (mgr.34107) 12139 : cluster [DBG] pgmap v12628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:23 smithi115 bash[45495]: audit 2022-04-22T23:22:23.430623+0000 mon.a (mon.0) 16738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:23 smithi115 bash[46934]: cluster 2022-04-22T23:22:23.260503+0000 mgr.y (mgr.34107) 12139 : cluster [DBG] pgmap v12628: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:23 smithi115 bash[46934]: audit 2022-04-22T23:22:23.430623+0000 mon.a (mon.0) 16738 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:25.209 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:24.879Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:25 smithi121 bash[36902]: audit 2022-04-22T23:22:25.153014+0000 mon.a (mon.0) 16739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:25.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:25 smithi115 bash[45495]: audit 2022-04-22T23:22:25.153014+0000 mon.a (mon.0) 16739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:25 smithi115 bash[46934]: audit 2022-04-22T23:22:25.153014+0000 mon.a (mon.0) 16739 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:26 smithi121 bash[36902]: cluster 2022-04-22T23:22:25.261477+0000 mgr.y (mgr.34107) 12140 : cluster [DBG] pgmap v12629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:26 smithi115 bash[45495]: cluster 2022-04-22T23:22:25.261477+0000 mgr.y (mgr.34107) 12140 : cluster [DBG] pgmap v12629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:26 smithi115 bash[46934]: cluster 2022-04-22T23:22:25.261477+0000 mgr.y (mgr.34107) 12140 : cluster [DBG] pgmap v12629: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:27 smithi121 bash[36902]: audit 2022-04-22T23:22:26.800308+0000 mon.a (mon.0) 16740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:27 smithi115 bash[45495]: audit 2022-04-22T23:22:26.800308+0000 mon.a (mon.0) 16740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:27 smithi115 bash[46934]: audit 2022-04-22T23:22:26.800308+0000 mon.a (mon.0) 16740 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:28 smithi121 bash[36902]: cluster 2022-04-22T23:22:27.262558+0000 mgr.y (mgr.34107) 12141 : cluster [DBG] pgmap v12630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:28 smithi115 bash[45495]: cluster 2022-04-22T23:22:27.262558+0000 mgr.y (mgr.34107) 12141 : cluster [DBG] pgmap v12630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:28 smithi115 bash[46934]: cluster 2022-04-22T23:22:27.262558+0000 mgr.y (mgr.34107) 12141 : cluster [DBG] pgmap v12630: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:29 smithi121 bash[36902]: audit 2022-04-22T23:22:28.433679+0000 mon.a (mon.0) 16741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:29.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:29 smithi115 bash[45495]: audit 2022-04-22T23:22:28.433679+0000 mon.a (mon.0) 16741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:29 smithi115 bash[46934]: audit 2022-04-22T23:22:28.433679+0000 mon.a (mon.0) 16741 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:30 smithi121 bash[36902]: cluster 2022-04-22T23:22:29.263657+0000 mgr.y (mgr.34107) 12142 : cluster [DBG] pgmap v12631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:30.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:30 smithi121 bash[36902]: audit 2022-04-22T23:22:30.156447+0000 mon.a (mon.0) 16742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:30 smithi115 bash[45495]: cluster 2022-04-22T23:22:29.263657+0000 mgr.y (mgr.34107) 12142 : cluster [DBG] pgmap v12631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:30 smithi115 bash[45495]: audit 2022-04-22T23:22:30.156447+0000 mon.a (mon.0) 16742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:30 smithi115 bash[46934]: cluster 2022-04-22T23:22:29.263657+0000 mgr.y (mgr.34107) 12142 : cluster [DBG] pgmap v12631: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:30 smithi115 bash[46934]: audit 2022-04-22T23:22:30.156447+0000 mon.a (mon.0) 16742 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:22:32.000 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:22:32.000 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 5m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 5m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 4m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 4m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 5m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 4m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 5h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:22:32.001 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 5h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 4m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 4m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 4m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 4m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:22:32.002 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 5h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:22:32.408 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:22:32.409 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:22:32.410 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:22:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:32 smithi121 bash[36902]: cluster 2022-04-22T23:22:31.264715+0000 mgr.y (mgr.34107) 12143 : cluster [DBG] pgmap v12632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:32 smithi121 bash[36902]: audit 2022-04-22T23:22:31.803343+0000 mon.a (mon.0) 16743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:32 smithi115 bash[45495]: cluster 2022-04-22T23:22:31.264715+0000 mgr.y (mgr.34107) 12143 : cluster [DBG] pgmap v12632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:32 smithi115 bash[45495]: audit 2022-04-22T23:22:31.803343+0000 mon.a (mon.0) 16743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:32 smithi115 bash[46934]: cluster 2022-04-22T23:22:31.264715+0000 mgr.y (mgr.34107) 12143 : cluster [DBG] pgmap v12632: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:32 smithi115 bash[46934]: audit 2022-04-22T23:22:31.803343+0000 mon.a (mon.0) 16743 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:33.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:33 smithi121 bash[36902]: audit 2022-04-22T23:22:31.634503+0000 mgr.y (mgr.34107) 12144 : audit [DBG] from='client.57847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:33 smithi121 bash[36902]: audit 2022-04-22T23:22:31.977968+0000 mgr.y (mgr.34107) 12145 : audit [DBG] from='client.47913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:33 smithi121 bash[36902]: audit 2022-04-22T23:22:32.404273+0000 mon.a (mon.0) 16744 : audit [DBG] from='client.? 172.21.15.115:0/4100932949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[45495]: audit 2022-04-22T23:22:31.634503+0000 mgr.y (mgr.34107) 12144 : audit [DBG] from='client.57847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[45495]: audit 2022-04-22T23:22:31.977968+0000 mgr.y (mgr.34107) 12145 : audit [DBG] from='client.47913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[45495]: audit 2022-04-22T23:22:32.404273+0000 mon.a (mon.0) 16744 : audit [DBG] from='client.? 172.21.15.115:0/4100932949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[46934]: audit 2022-04-22T23:22:31.634503+0000 mgr.y (mgr.34107) 12144 : audit [DBG] from='client.57847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[46934]: audit 2022-04-22T23:22:31.977968+0000 mgr.y (mgr.34107) 12145 : audit [DBG] from='client.47913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:22:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:33 smithi115 bash[46934]: audit 2022-04-22T23:22:32.404273+0000 mon.a (mon.0) 16744 : audit [DBG] from='client.? 172.21.15.115:0/4100932949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:22:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:34 smithi121 bash[36902]: cluster 2022-04-22T23:22:33.265898+0000 mgr.y (mgr.34107) 12146 : cluster [DBG] pgmap v12633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:34 smithi121 bash[36902]: audit 2022-04-22T23:22:33.437039+0000 mon.a (mon.0) 16745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:34 smithi115 bash[46934]: cluster 2022-04-22T23:22:33.265898+0000 mgr.y (mgr.34107) 12146 : cluster [DBG] pgmap v12633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:34 smithi115 bash[46934]: audit 2022-04-22T23:22:33.437039+0000 mon.a (mon.0) 16745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:34 smithi115 bash[45495]: cluster 2022-04-22T23:22:33.265898+0000 mgr.y (mgr.34107) 12146 : cluster [DBG] pgmap v12633: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:34.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:34 smithi115 bash[45495]: audit 2022-04-22T23:22:33.437039+0000 mon.a (mon.0) 16745 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:34.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:35 smithi121 bash[36902]: audit 2022-04-22T23:22:35.160126+0000 mon.a (mon.0) 16746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:35 smithi115 bash[46934]: audit 2022-04-22T23:22:35.160126+0000 mon.a (mon.0) 16746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:35 smithi115 bash[45495]: audit 2022-04-22T23:22:35.160126+0000 mon.a (mon.0) 16746 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:36 smithi121 bash[36902]: cluster 2022-04-22T23:22:35.266921+0000 mgr.y (mgr.34107) 12147 : cluster [DBG] pgmap v12634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:36 smithi115 bash[46934]: cluster 2022-04-22T23:22:35.266921+0000 mgr.y (mgr.34107) 12147 : cluster [DBG] pgmap v12634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:36 smithi115 bash[45495]: cluster 2022-04-22T23:22:35.266921+0000 mgr.y (mgr.34107) 12147 : cluster [DBG] pgmap v12634: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:37 smithi121 bash[36902]: audit 2022-04-22T23:22:36.806392+0000 mon.a (mon.0) 16747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:37 smithi115 bash[45495]: audit 2022-04-22T23:22:36.806392+0000 mon.a (mon.0) 16747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:37 smithi115 bash[46934]: audit 2022-04-22T23:22:36.806392+0000 mon.a (mon.0) 16747 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:38 smithi121 bash[36902]: cluster 2022-04-22T23:22:37.268119+0000 mgr.y (mgr.34107) 12148 : cluster [DBG] pgmap v12635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:38.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:38 smithi115 bash[45495]: cluster 2022-04-22T23:22:37.268119+0000 mgr.y (mgr.34107) 12148 : cluster [DBG] pgmap v12635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:38 smithi115 bash[46934]: cluster 2022-04-22T23:22:37.268119+0000 mgr.y (mgr.34107) 12148 : cluster [DBG] pgmap v12635: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:39 smithi121 bash[36902]: audit 2022-04-22T23:22:38.440223+0000 mon.a (mon.0) 16748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:39 smithi121 bash[36902]: cluster 2022-04-22T23:22:39.269202+0000 mgr.y (mgr.34107) 12149 : cluster [DBG] pgmap v12636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:39 smithi115 bash[45495]: audit 2022-04-22T23:22:38.440223+0000 mon.a (mon.0) 16748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:39 smithi115 bash[45495]: cluster 2022-04-22T23:22:39.269202+0000 mgr.y (mgr.34107) 12149 : cluster [DBG] pgmap v12636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:39 smithi115 bash[46934]: audit 2022-04-22T23:22:38.440223+0000 mon.a (mon.0) 16748 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:39 smithi115 bash[46934]: cluster 2022-04-22T23:22:39.269202+0000 mgr.y (mgr.34107) 12149 : cluster [DBG] pgmap v12636: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:40 smithi121 bash[36902]: audit 2022-04-22T23:22:40.163932+0000 mon.a (mon.0) 16749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:40 smithi115 bash[45495]: audit 2022-04-22T23:22:40.163932+0000 mon.a (mon.0) 16749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:40 smithi115 bash[46934]: audit 2022-04-22T23:22:40.163932+0000 mon.a (mon.0) 16749 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:41 smithi121 bash[36902]: cluster 2022-04-22T23:22:41.270184+0000 mgr.y (mgr.34107) 12150 : cluster [DBG] pgmap v12637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:41 smithi115 bash[45495]: cluster 2022-04-22T23:22:41.270184+0000 mgr.y (mgr.34107) 12150 : cluster [DBG] pgmap v12637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:41 smithi115 bash[46934]: cluster 2022-04-22T23:22:41.270184+0000 mgr.y (mgr.34107) 12150 : cluster [DBG] pgmap v12637: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:42.772 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:42 smithi121 bash[36902]: audit 2022-04-22T23:22:41.809467+0000 mon.a (mon.0) 16750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:42.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:42 smithi115 bash[45495]: audit 2022-04-22T23:22:41.809467+0000 mon.a (mon.0) 16750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:42 smithi115 bash[46934]: audit 2022-04-22T23:22:41.809467+0000 mon.a (mon.0) 16750 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:43.024Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:43 smithi121 bash[36902]: cluster 2022-04-22T23:22:43.271334+0000 mgr.y (mgr.34107) 12151 : cluster [DBG] pgmap v12638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:43 smithi121 bash[36902]: audit 2022-04-22T23:22:43.443224+0000 mon.a (mon.0) 16751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:43 smithi115 bash[45495]: cluster 2022-04-22T23:22:43.271334+0000 mgr.y (mgr.34107) 12151 : cluster [DBG] pgmap v12638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:43 smithi115 bash[45495]: audit 2022-04-22T23:22:43.443224+0000 mon.a (mon.0) 16751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:43 smithi115 bash[46934]: cluster 2022-04-22T23:22:43.271334+0000 mgr.y (mgr.34107) 12151 : cluster [DBG] pgmap v12638: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:44.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:43 smithi115 bash[46934]: audit 2022-04-22T23:22:43.443224+0000 mon.a (mon.0) 16751 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:45.223 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:44.881Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:45 smithi121 bash[36902]: audit 2022-04-22T23:22:45.167580+0000 mon.a (mon.0) 16752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:45.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:45 smithi115 bash[45495]: audit 2022-04-22T23:22:45.167580+0000 mon.a (mon.0) 16752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:45 smithi115 bash[46934]: audit 2022-04-22T23:22:45.167580+0000 mon.a (mon.0) 16752 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:46 smithi121 bash[36902]: cluster 2022-04-22T23:22:45.272390+0000 mgr.y (mgr.34107) 12152 : cluster [DBG] pgmap v12639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:46 smithi115 bash[45495]: cluster 2022-04-22T23:22:45.272390+0000 mgr.y (mgr.34107) 12152 : cluster [DBG] pgmap v12639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:46 smithi115 bash[46934]: cluster 2022-04-22T23:22:45.272390+0000 mgr.y (mgr.34107) 12152 : cluster [DBG] pgmap v12639: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:47 smithi121 bash[36902]: audit 2022-04-22T23:22:46.812532+0000 mon.a (mon.0) 16753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:47 smithi115 bash[45495]: audit 2022-04-22T23:22:46.812532+0000 mon.a (mon.0) 16753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:47 smithi115 bash[46934]: audit 2022-04-22T23:22:46.812532+0000 mon.a (mon.0) 16753 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:48 smithi121 bash[36902]: cluster 2022-04-22T23:22:47.273442+0000 mgr.y (mgr.34107) 12153 : cluster [DBG] pgmap v12640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:48.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:48 smithi115 bash[46934]: cluster 2022-04-22T23:22:47.273442+0000 mgr.y (mgr.34107) 12153 : cluster [DBG] pgmap v12640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:48 smithi115 bash[45495]: cluster 2022-04-22T23:22:47.273442+0000 mgr.y (mgr.34107) 12153 : cluster [DBG] pgmap v12640: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:49 smithi121 bash[36902]: audit 2022-04-22T23:22:48.446482+0000 mon.a (mon.0) 16754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:49 smithi115 bash[45495]: audit 2022-04-22T23:22:48.446482+0000 mon.a (mon.0) 16754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:49 smithi115 bash[46934]: audit 2022-04-22T23:22:48.446482+0000 mon.a (mon.0) 16754 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:50 smithi121 bash[36902]: cluster 2022-04-22T23:22:49.274535+0000 mgr.y (mgr.34107) 12154 : cluster [DBG] pgmap v12641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:50.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:50 smithi121 bash[36902]: audit 2022-04-22T23:22:50.171073+0000 mon.a (mon.0) 16755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:50 smithi115 bash[45495]: cluster 2022-04-22T23:22:49.274535+0000 mgr.y (mgr.34107) 12154 : cluster [DBG] pgmap v12641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:50.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:50 smithi115 bash[45495]: audit 2022-04-22T23:22:50.171073+0000 mon.a (mon.0) 16755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:50 smithi115 bash[46934]: cluster 2022-04-22T23:22:49.274535+0000 mgr.y (mgr.34107) 12154 : cluster [DBG] pgmap v12641: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:50 smithi115 bash[46934]: audit 2022-04-22T23:22:50.171073+0000 mon.a (mon.0) 16755 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:52 smithi121 bash[36902]: cluster 2022-04-22T23:22:51.275665+0000 mgr.y (mgr.34107) 12155 : cluster [DBG] pgmap v12642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:52 smithi121 bash[36902]: audit 2022-04-22T23:22:51.815660+0000 mon.a (mon.0) 16756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:52 smithi115 bash[45495]: cluster 2022-04-22T23:22:51.275665+0000 mgr.y (mgr.34107) 12155 : cluster [DBG] pgmap v12642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:52 smithi115 bash[45495]: audit 2022-04-22T23:22:51.815660+0000 mon.a (mon.0) 16756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:52 smithi115 bash[46934]: cluster 2022-04-22T23:22:51.275665+0000 mgr.y (mgr.34107) 12155 : cluster [DBG] pgmap v12642: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:52 smithi115 bash[46934]: audit 2022-04-22T23:22:51.815660+0000 mon.a (mon.0) 16756 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:53.025Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:53 smithi121 bash[36902]: cluster 2022-04-22T23:22:53.276720+0000 mgr.y (mgr.34107) 12156 : cluster [DBG] pgmap v12643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:53 smithi121 bash[36902]: audit 2022-04-22T23:22:53.449924+0000 mon.a (mon.0) 16757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:54.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:53 smithi115 bash[46934]: cluster 2022-04-22T23:22:53.276720+0000 mgr.y (mgr.34107) 12156 : cluster [DBG] pgmap v12643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:53 smithi115 bash[46934]: audit 2022-04-22T23:22:53.449924+0000 mon.a (mon.0) 16757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:53 smithi115 bash[45495]: cluster 2022-04-22T23:22:53.276720+0000 mgr.y (mgr.34107) 12156 : cluster [DBG] pgmap v12643: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:53 smithi115 bash[45495]: audit 2022-04-22T23:22:53.449924+0000 mon.a (mon.0) 16757 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:55.230 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:22:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:22:54.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:22:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:55 smithi121 bash[36902]: audit 2022-04-22T23:22:55.174383+0000 mon.a (mon.0) 16758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:55 smithi115 bash[45495]: audit 2022-04-22T23:22:55.174383+0000 mon.a (mon.0) 16758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:55 smithi115 bash[46934]: audit 2022-04-22T23:22:55.174383+0000 mon.a (mon.0) 16758 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:22:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:56 smithi121 bash[36902]: cluster 2022-04-22T23:22:55.277921+0000 mgr.y (mgr.34107) 12157 : cluster [DBG] pgmap v12644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:56 smithi115 bash[45495]: cluster 2022-04-22T23:22:55.277921+0000 mgr.y (mgr.34107) 12157 : cluster [DBG] pgmap v12644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:56 smithi115 bash[46934]: cluster 2022-04-22T23:22:55.277921+0000 mgr.y (mgr.34107) 12157 : cluster [DBG] pgmap v12644: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:57 smithi121 bash[36902]: audit 2022-04-22T23:22:56.818770+0000 mon.a (mon.0) 16759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:57.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:57 smithi115 bash[45495]: audit 2022-04-22T23:22:56.818770+0000 mon.a (mon.0) 16759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:57 smithi115 bash[46934]: audit 2022-04-22T23:22:56.818770+0000 mon.a (mon.0) 16759 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:58 smithi121 bash[36902]: cluster 2022-04-22T23:22:57.279062+0000 mgr.y (mgr.34107) 12158 : cluster [DBG] pgmap v12645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:58 smithi121 bash[36902]: audit 2022-04-22T23:22:57.716923+0000 mon.a (mon.0) 16760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:58 smithi121 bash[36902]: audit 2022-04-22T23:22:57.719958+0000 mon.a (mon.0) 16761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:58 smithi121 bash[36902]: audit 2022-04-22T23:22:57.721782+0000 mon.a (mon.0) 16762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:22:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:58 smithi121 bash[36902]: audit 2022-04-22T23:22:58.063717+0000 mon.a (mon.0) 16763 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:22:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[46934]: cluster 2022-04-22T23:22:57.279062+0000 mgr.y (mgr.34107) 12158 : cluster [DBG] pgmap v12645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[46934]: audit 2022-04-22T23:22:57.716923+0000 mon.a (mon.0) 16760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:22:58.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[46934]: audit 2022-04-22T23:22:57.719958+0000 mon.a (mon.0) 16761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:22:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[46934]: audit 2022-04-22T23:22:57.721782+0000 mon.a (mon.0) 16762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:22:58.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[46934]: audit 2022-04-22T23:22:58.063717+0000 mon.a (mon.0) 16763 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:22:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[45495]: cluster 2022-04-22T23:22:57.279062+0000 mgr.y (mgr.34107) 12158 : cluster [DBG] pgmap v12645: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:22:58.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[45495]: audit 2022-04-22T23:22:57.716923+0000 mon.a (mon.0) 16760 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:22:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[45495]: audit 2022-04-22T23:22:57.719958+0000 mon.a (mon.0) 16761 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:22:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[45495]: audit 2022-04-22T23:22:57.721782+0000 mon.a (mon.0) 16762 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:22:58.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:58 smithi115 bash[45495]: audit 2022-04-22T23:22:58.063717+0000 mon.a (mon.0) 16763 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:22:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:22:59 smithi121 bash[36902]: audit 2022-04-22T23:22:58.452962+0000 mon.a (mon.0) 16764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:22:59 smithi115 bash[46934]: audit 2022-04-22T23:22:58.452962+0000 mon.a (mon.0) 16764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:22:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:22:59 smithi115 bash[45495]: audit 2022-04-22T23:22:58.452962+0000 mon.a (mon.0) 16764 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:00 smithi115 bash[45495]: cluster 2022-04-22T23:22:59.280262+0000 mgr.y (mgr.34107) 12159 : cluster [DBG] pgmap v12646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:00.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:00 smithi115 bash[45495]: audit 2022-04-22T23:23:00.177986+0000 mon.a (mon.0) 16765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:00 smithi115 bash[46934]: cluster 2022-04-22T23:22:59.280262+0000 mgr.y (mgr.34107) 12159 : cluster [DBG] pgmap v12646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:00.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:00 smithi115 bash[46934]: audit 2022-04-22T23:23:00.177986+0000 mon.a (mon.0) 16765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:00 smithi121 bash[36902]: cluster 2022-04-22T23:22:59.280262+0000 mgr.y (mgr.34107) 12159 : cluster [DBG] pgmap v12646: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:00 smithi121 bash[36902]: audit 2022-04-22T23:23:00.177986+0000 mon.a (mon.0) 16765 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:02 smithi121 bash[36902]: cluster 2022-04-22T23:23:01.281320+0000 mgr.y (mgr.34107) 12160 : cluster [DBG] pgmap v12647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:02 smithi121 bash[36902]: audit 2022-04-22T23:23:01.822070+0000 mon.a (mon.0) 16766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:02.782 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:23:02.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:02 smithi115 bash[45495]: cluster 2022-04-22T23:23:01.281320+0000 mgr.y (mgr.34107) 12160 : cluster [DBG] pgmap v12647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:02 smithi115 bash[45495]: audit 2022-04-22T23:23:01.822070+0000 mon.a (mon.0) 16766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:02 smithi115 bash[46934]: cluster 2022-04-22T23:23:01.281320+0000 mgr.y (mgr.34107) 12160 : cluster [DBG] pgmap v12647: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:02 smithi115 bash[46934]: audit 2022-04-22T23:23:01.822070+0000 mon.a (mon.0) 16766 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:03.131 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:23:03.131 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 5h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:23:03.131 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (5h) 6m ago 5h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:23:03.132 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:23:03.133 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:23:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:03.023Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:03.547 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:23:03.547 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:23:03.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:03.548 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:23:03.549 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:23:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:03 smithi121 bash[36902]: audit 2022-04-22T23:23:02.762541+0000 mgr.y (mgr.34107) 12161 : audit [DBG] from='client.47925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:03 smithi121 bash[36902]: audit 2022-04-22T23:23:03.107399+0000 mgr.y (mgr.34107) 12162 : audit [DBG] from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:03 smithi121 bash[36902]: cluster 2022-04-22T23:23:03.282311+0000 mgr.y (mgr.34107) 12163 : cluster [DBG] pgmap v12648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:03 smithi121 bash[36902]: audit 2022-04-22T23:23:03.456239+0000 mon.a (mon.0) 16767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:04.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:03 smithi121 bash[36902]: audit 2022-04-22T23:23:03.540656+0000 mon.a (mon.0) 16768 : audit [DBG] from='client.? 172.21.15.115:0/762661560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[45495]: audit 2022-04-22T23:23:02.762541+0000 mgr.y (mgr.34107) 12161 : audit [DBG] from='client.47925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[45495]: audit 2022-04-22T23:23:03.107399+0000 mgr.y (mgr.34107) 12162 : audit [DBG] from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[45495]: cluster 2022-04-22T23:23:03.282311+0000 mgr.y (mgr.34107) 12163 : cluster [DBG] pgmap v12648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[45495]: audit 2022-04-22T23:23:03.456239+0000 mon.a (mon.0) 16767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[45495]: audit 2022-04-22T23:23:03.540656+0000 mon.a (mon.0) 16768 : audit [DBG] from='client.? 172.21.15.115:0/762661560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[46934]: audit 2022-04-22T23:23:02.762541+0000 mgr.y (mgr.34107) 12161 : audit [DBG] from='client.47925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[46934]: audit 2022-04-22T23:23:03.107399+0000 mgr.y (mgr.34107) 12162 : audit [DBG] from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[46934]: cluster 2022-04-22T23:23:03.282311+0000 mgr.y (mgr.34107) 12163 : cluster [DBG] pgmap v12648: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[46934]: audit 2022-04-22T23:23:03.456239+0000 mon.a (mon.0) 16767 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:03 smithi115 bash[46934]: audit 2022-04-22T23:23:03.540656+0000 mon.a (mon.0) 16768 : audit [DBG] from='client.? 172.21.15.115:0/762661560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:05.237 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:04.882Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:05 smithi121 bash[36902]: audit 2022-04-22T23:23:05.181654+0000 mon.a (mon.0) 16769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:05 smithi115 bash[45495]: audit 2022-04-22T23:23:05.181654+0000 mon.a (mon.0) 16769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:05 smithi115 bash[46934]: audit 2022-04-22T23:23:05.181654+0000 mon.a (mon.0) 16769 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:06 smithi121 bash[36902]: cluster 2022-04-22T23:23:05.283367+0000 mgr.y (mgr.34107) 12164 : cluster [DBG] pgmap v12649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:06 smithi115 bash[46934]: cluster 2022-04-22T23:23:05.283367+0000 mgr.y (mgr.34107) 12164 : cluster [DBG] pgmap v12649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:06.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:06 smithi115 bash[45495]: cluster 2022-04-22T23:23:05.283367+0000 mgr.y (mgr.34107) 12164 : cluster [DBG] pgmap v12649: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:07 smithi121 bash[36902]: audit 2022-04-22T23:23:06.825141+0000 mon.a (mon.0) 16770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:07 smithi115 bash[45495]: audit 2022-04-22T23:23:06.825141+0000 mon.a (mon.0) 16770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:07 smithi115 bash[46934]: audit 2022-04-22T23:23:06.825141+0000 mon.a (mon.0) 16770 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:08 smithi121 bash[36902]: cluster 2022-04-22T23:23:07.284504+0000 mgr.y (mgr.34107) 12165 : cluster [DBG] pgmap v12650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:08 smithi115 bash[45495]: cluster 2022-04-22T23:23:07.284504+0000 mgr.y (mgr.34107) 12165 : cluster [DBG] pgmap v12650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:08 smithi115 bash[46934]: cluster 2022-04-22T23:23:07.284504+0000 mgr.y (mgr.34107) 12165 : cluster [DBG] pgmap v12650: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:09 smithi121 bash[36902]: audit 2022-04-22T23:23:08.459071+0000 mon.a (mon.0) 16771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:09 smithi115 bash[45495]: audit 2022-04-22T23:23:08.459071+0000 mon.a (mon.0) 16771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:09 smithi115 bash[46934]: audit 2022-04-22T23:23:08.459071+0000 mon.a (mon.0) 16771 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:10 smithi115 bash[45495]: cluster 2022-04-22T23:23:09.285627+0000 mgr.y (mgr.34107) 12166 : cluster [DBG] pgmap v12651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:10 smithi115 bash[45495]: audit 2022-04-22T23:23:10.185426+0000 mon.a (mon.0) 16772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:10 smithi115 bash[46934]: cluster 2022-04-22T23:23:09.285627+0000 mgr.y (mgr.34107) 12166 : cluster [DBG] pgmap v12651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:10 smithi115 bash[46934]: audit 2022-04-22T23:23:10.185426+0000 mon.a (mon.0) 16772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:10 smithi121 bash[36902]: cluster 2022-04-22T23:23:09.285627+0000 mgr.y (mgr.34107) 12166 : cluster [DBG] pgmap v12651: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:10 smithi121 bash[36902]: audit 2022-04-22T23:23:10.185426+0000 mon.a (mon.0) 16772 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:12 smithi121 bash[36902]: cluster 2022-04-22T23:23:11.286746+0000 mgr.y (mgr.34107) 12167 : cluster [DBG] pgmap v12652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:12 smithi121 bash[36902]: audit 2022-04-22T23:23:11.828275+0000 mon.a (mon.0) 16773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:12 smithi115 bash[45495]: cluster 2022-04-22T23:23:11.286746+0000 mgr.y (mgr.34107) 12167 : cluster [DBG] pgmap v12652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:12 smithi115 bash[45495]: audit 2022-04-22T23:23:11.828275+0000 mon.a (mon.0) 16773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:12 smithi115 bash[46934]: cluster 2022-04-22T23:23:11.286746+0000 mgr.y (mgr.34107) 12167 : cluster [DBG] pgmap v12652: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:12 smithi115 bash[46934]: audit 2022-04-22T23:23:11.828275+0000 mon.a (mon.0) 16773 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:13.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:13 smithi121 bash[36902]: cluster 2022-04-22T23:23:13.287955+0000 mgr.y (mgr.34107) 12168 : cluster [DBG] pgmap v12653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:13 smithi121 bash[36902]: audit 2022-04-22T23:23:13.462389+0000 mon.a (mon.0) 16774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:13 smithi115 bash[45495]: cluster 2022-04-22T23:23:13.287955+0000 mgr.y (mgr.34107) 12168 : cluster [DBG] pgmap v12653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:14.042 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:13 smithi115 bash[45495]: audit 2022-04-22T23:23:13.462389+0000 mon.a (mon.0) 16774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:14.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:13 smithi115 bash[46934]: cluster 2022-04-22T23:23:13.287955+0000 mgr.y (mgr.34107) 12168 : cluster [DBG] pgmap v12653: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:14.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:13 smithi115 bash[46934]: audit 2022-04-22T23:23:13.462389+0000 mon.a (mon.0) 16774 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:15.245 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:14.883Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:15 smithi121 bash[36902]: audit 2022-04-22T23:23:15.189234+0000 mon.a (mon.0) 16775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:15 smithi115 bash[45495]: audit 2022-04-22T23:23:15.189234+0000 mon.a (mon.0) 16775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:15 smithi115 bash[46934]: audit 2022-04-22T23:23:15.189234+0000 mon.a (mon.0) 16775 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:16 smithi121 bash[36902]: cluster 2022-04-22T23:23:15.289129+0000 mgr.y (mgr.34107) 12169 : cluster [DBG] pgmap v12654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:16 smithi115 bash[45495]: cluster 2022-04-22T23:23:15.289129+0000 mgr.y (mgr.34107) 12169 : cluster [DBG] pgmap v12654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:16 smithi115 bash[46934]: cluster 2022-04-22T23:23:15.289129+0000 mgr.y (mgr.34107) 12169 : cluster [DBG] pgmap v12654: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:17 smithi121 bash[36902]: audit 2022-04-22T23:23:16.831500+0000 mon.a (mon.0) 16776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:17 smithi115 bash[45495]: audit 2022-04-22T23:23:16.831500+0000 mon.a (mon.0) 16776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:17 smithi115 bash[46934]: audit 2022-04-22T23:23:16.831500+0000 mon.a (mon.0) 16776 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[45495]: cluster 2022-04-22T23:23:17.290219+0000 mgr.y (mgr.34107) 12170 : cluster [DBG] pgmap v12655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[45495]: audit 2022-04-22T23:23:17.383695+0000 mon.a (mon.0) 16777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:23:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[45495]: audit 2022-04-22T23:23:17.387505+0000 mon.a (mon.0) 16778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:23:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[46934]: cluster 2022-04-22T23:23:17.290219+0000 mgr.y (mgr.34107) 12170 : cluster [DBG] pgmap v12655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[46934]: audit 2022-04-22T23:23:17.383695+0000 mon.a (mon.0) 16777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:23:18.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:18 smithi115 bash[46934]: audit 2022-04-22T23:23:17.387505+0000 mon.a (mon.0) 16778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:23:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:18 smithi121 bash[36902]: cluster 2022-04-22T23:23:17.290219+0000 mgr.y (mgr.34107) 12170 : cluster [DBG] pgmap v12655: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:18 smithi121 bash[36902]: audit 2022-04-22T23:23:17.383695+0000 mon.a (mon.0) 16777 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:23:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:18 smithi121 bash[36902]: audit 2022-04-22T23:23:17.387505+0000 mon.a (mon.0) 16778 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:23:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:19 smithi115 bash[45495]: audit 2022-04-22T23:23:18.465430+0000 mon.a (mon.0) 16779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:19 smithi115 bash[46934]: audit 2022-04-22T23:23:18.465430+0000 mon.a (mon.0) 16779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:19 smithi121 bash[36902]: audit 2022-04-22T23:23:18.465430+0000 mon.a (mon.0) 16779 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:20 smithi115 bash[46934]: cluster 2022-04-22T23:23:19.291332+0000 mgr.y (mgr.34107) 12171 : cluster [DBG] pgmap v12656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:20.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:20 smithi115 bash[46934]: audit 2022-04-22T23:23:20.192938+0000 mon.a (mon.0) 16780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:20 smithi115 bash[45495]: cluster 2022-04-22T23:23:19.291332+0000 mgr.y (mgr.34107) 12171 : cluster [DBG] pgmap v12656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:20.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:20 smithi115 bash[45495]: audit 2022-04-22T23:23:20.192938+0000 mon.a (mon.0) 16780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:20 smithi121 bash[36902]: cluster 2022-04-22T23:23:19.291332+0000 mgr.y (mgr.34107) 12171 : cluster [DBG] pgmap v12656: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:20 smithi121 bash[36902]: audit 2022-04-22T23:23:20.192938+0000 mon.a (mon.0) 16780 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:21 smithi121 bash[36902]: cluster 2022-04-22T23:23:21.292502+0000 mgr.y (mgr.34107) 12172 : cluster [DBG] pgmap v12657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:21.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:21 smithi115 bash[45495]: cluster 2022-04-22T23:23:21.292502+0000 mgr.y (mgr.34107) 12172 : cluster [DBG] pgmap v12657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:21.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:21 smithi115 bash[46934]: cluster 2022-04-22T23:23:21.292502+0000 mgr.y (mgr.34107) 12172 : cluster [DBG] pgmap v12657: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:22 smithi121 bash[36902]: audit 2022-04-22T23:23:21.834554+0000 mon.a (mon.0) 16781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:22 smithi115 bash[46934]: audit 2022-04-22T23:23:21.834554+0000 mon.a (mon.0) 16781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:22 smithi115 bash[45495]: audit 2022-04-22T23:23:21.834554+0000 mon.a (mon.0) 16781 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:23.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:23 smithi121 bash[36902]: cluster 2022-04-22T23:23:23.293649+0000 mgr.y (mgr.34107) 12173 : cluster [DBG] pgmap v12658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:23 smithi121 bash[36902]: audit 2022-04-22T23:23:23.468524+0000 mon.a (mon.0) 16782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:23 smithi115 bash[46934]: cluster 2022-04-22T23:23:23.293649+0000 mgr.y (mgr.34107) 12173 : cluster [DBG] pgmap v12658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:23 smithi115 bash[46934]: audit 2022-04-22T23:23:23.468524+0000 mon.a (mon.0) 16782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:23 smithi115 bash[45495]: cluster 2022-04-22T23:23:23.293649+0000 mgr.y (mgr.34107) 12173 : cluster [DBG] pgmap v12658: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:23 smithi115 bash[45495]: audit 2022-04-22T23:23:23.468524+0000 mon.a (mon.0) 16782 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:25.253 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:24.884Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:25 smithi121 bash[36902]: audit 2022-04-22T23:23:25.196799+0000 mon.a (mon.0) 16783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:25.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:25 smithi115 bash[46934]: audit 2022-04-22T23:23:25.196799+0000 mon.a (mon.0) 16783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:25 smithi115 bash[45495]: audit 2022-04-22T23:23:25.196799+0000 mon.a (mon.0) 16783 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:26 smithi121 bash[36902]: cluster 2022-04-22T23:23:25.294860+0000 mgr.y (mgr.34107) 12174 : cluster [DBG] pgmap v12659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:26 smithi115 bash[45495]: cluster 2022-04-22T23:23:25.294860+0000 mgr.y (mgr.34107) 12174 : cluster [DBG] pgmap v12659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:26 smithi115 bash[46934]: cluster 2022-04-22T23:23:25.294860+0000 mgr.y (mgr.34107) 12174 : cluster [DBG] pgmap v12659: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:27 smithi115 bash[45495]: audit 2022-04-22T23:23:26.837734+0000 mon.a (mon.0) 16784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:27 smithi115 bash[46934]: audit 2022-04-22T23:23:26.837734+0000 mon.a (mon.0) 16784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:27 smithi121 bash[36902]: audit 2022-04-22T23:23:26.837734+0000 mon.a (mon.0) 16784 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:28 smithi115 bash[45495]: cluster 2022-04-22T23:23:27.296022+0000 mgr.y (mgr.34107) 12175 : cluster [DBG] pgmap v12660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:28 smithi115 bash[46934]: cluster 2022-04-22T23:23:27.296022+0000 mgr.y (mgr.34107) 12175 : cluster [DBG] pgmap v12660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:28 smithi121 bash[36902]: cluster 2022-04-22T23:23:27.296022+0000 mgr.y (mgr.34107) 12175 : cluster [DBG] pgmap v12660: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:29 smithi115 bash[45495]: audit 2022-04-22T23:23:28.471596+0000 mon.a (mon.0) 16785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:29 smithi115 bash[46934]: audit 2022-04-22T23:23:28.471596+0000 mon.a (mon.0) 16785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:29 smithi121 bash[36902]: audit 2022-04-22T23:23:28.471596+0000 mon.a (mon.0) 16785 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:30 smithi115 bash[45495]: cluster 2022-04-22T23:23:29.297124+0000 mgr.y (mgr.34107) 12176 : cluster [DBG] pgmap v12661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:30.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:30 smithi115 bash[45495]: audit 2022-04-22T23:23:30.200568+0000 mon.a (mon.0) 16786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:30 smithi115 bash[46934]: cluster 2022-04-22T23:23:29.297124+0000 mgr.y (mgr.34107) 12176 : cluster [DBG] pgmap v12661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:30.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:30 smithi115 bash[46934]: audit 2022-04-22T23:23:30.200568+0000 mon.a (mon.0) 16786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:30 smithi121 bash[36902]: cluster 2022-04-22T23:23:29.297124+0000 mgr.y (mgr.34107) 12176 : cluster [DBG] pgmap v12661: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:30 smithi121 bash[36902]: audit 2022-04-22T23:23:30.200568+0000 mon.a (mon.0) 16786 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:31 smithi121 bash[36902]: cluster 2022-04-22T23:23:31.298248+0000 mgr.y (mgr.34107) 12177 : cluster [DBG] pgmap v12662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:31 smithi115 bash[46934]: cluster 2022-04-22T23:23:31.298248+0000 mgr.y (mgr.34107) 12177 : cluster [DBG] pgmap v12662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:31 smithi115 bash[45495]: cluster 2022-04-22T23:23:31.298248+0000 mgr.y (mgr.34107) 12177 : cluster [DBG] pgmap v12662: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:32 smithi121 bash[36902]: audit 2022-04-22T23:23:31.840991+0000 mon.a (mon.0) 16787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:32 smithi115 bash[45495]: audit 2022-04-22T23:23:31.840991+0000 mon.a (mon.0) 16787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:32 smithi115 bash[46934]: audit 2022-04-22T23:23:31.840991+0000 mon.a (mon.0) 16787 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:33.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:33.028Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:33.911 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:23:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:33 smithi121 bash[36902]: cluster 2022-04-22T23:23:33.299396+0000 mgr.y (mgr.34107) 12178 : cluster [DBG] pgmap v12663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:33 smithi121 bash[36902]: audit 2022-04-22T23:23:33.474767+0000 mon.a (mon.0) 16788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:34.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:33 smithi115 bash[46934]: cluster 2022-04-22T23:23:33.299396+0000 mgr.y (mgr.34107) 12178 : cluster [DBG] pgmap v12663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:33 smithi115 bash[46934]: audit 2022-04-22T23:23:33.474767+0000 mon.a (mon.0) 16788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:33 smithi115 bash[45495]: cluster 2022-04-22T23:23:33.299396+0000 mgr.y (mgr.34107) 12178 : cluster [DBG] pgmap v12663: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:33 smithi115 bash[45495]: audit 2022-04-22T23:23:33.474767+0000 mon.a (mon.0) 16788 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 6m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 6m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:23:34.259 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 5m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 5m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 6m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:23:34.260 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 6m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:23:34.261 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 6m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:23:34.261 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:23:34.261 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:23:34.261 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:23:34.665 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:23:34.666 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:23:34.667 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:23:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:34 smithi121 bash[36902]: audit 2022-04-22T23:23:33.893484+0000 mgr.y (mgr.34107) 12179 : audit [DBG] from='client.47943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:34 smithi121 bash[36902]: audit 2022-04-22T23:23:34.231847+0000 mgr.y (mgr.34107) 12180 : audit [DBG] from='client.57889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:34 smithi121 bash[36902]: audit 2022-04-22T23:23:34.661924+0000 mon.c (mon.1) 412 : audit [DBG] from='client.? 172.21.15.115:0/3557363013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:35.040 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:34.886Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[45495]: audit 2022-04-22T23:23:33.893484+0000 mgr.y (mgr.34107) 12179 : audit [DBG] from='client.47943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[45495]: audit 2022-04-22T23:23:34.231847+0000 mgr.y (mgr.34107) 12180 : audit [DBG] from='client.57889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[45495]: audit 2022-04-22T23:23:34.661924+0000 mon.c (mon.1) 412 : audit [DBG] from='client.? 172.21.15.115:0/3557363013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[46934]: audit 2022-04-22T23:23:33.893484+0000 mgr.y (mgr.34107) 12179 : audit [DBG] from='client.47943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[46934]: audit 2022-04-22T23:23:34.231847+0000 mgr.y (mgr.34107) 12180 : audit [DBG] from='client.57889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:23:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:34 smithi115 bash[46934]: audit 2022-04-22T23:23:34.661924+0000 mon.c (mon.1) 412 : audit [DBG] from='client.? 172.21.15.115:0/3557363013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:23:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:35 smithi121 bash[36902]: audit 2022-04-22T23:23:35.204332+0000 mon.a (mon.0) 16789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:36.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:35 smithi121 bash[36902]: cluster 2022-04-22T23:23:35.300443+0000 mgr.y (mgr.34107) 12181 : cluster [DBG] pgmap v12664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:35 smithi115 bash[45495]: audit 2022-04-22T23:23:35.204332+0000 mon.a (mon.0) 16789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:36.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:35 smithi115 bash[45495]: cluster 2022-04-22T23:23:35.300443+0000 mgr.y (mgr.34107) 12181 : cluster [DBG] pgmap v12664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:35 smithi115 bash[46934]: audit 2022-04-22T23:23:35.204332+0000 mon.a (mon.0) 16789 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:36.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:35 smithi115 bash[46934]: cluster 2022-04-22T23:23:35.300443+0000 mgr.y (mgr.34107) 12181 : cluster [DBG] pgmap v12664: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:36 smithi121 bash[36902]: audit 2022-04-22T23:23:36.843862+0000 mon.a (mon.0) 16790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:37.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:36 smithi115 bash[45495]: audit 2022-04-22T23:23:36.843862+0000 mon.a (mon.0) 16790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:36 smithi115 bash[46934]: audit 2022-04-22T23:23:36.843862+0000 mon.a (mon.0) 16790 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:37 smithi121 bash[36902]: cluster 2022-04-22T23:23:37.301479+0000 mgr.y (mgr.34107) 12182 : cluster [DBG] pgmap v12665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:37 smithi115 bash[45495]: cluster 2022-04-22T23:23:37.301479+0000 mgr.y (mgr.34107) 12182 : cluster [DBG] pgmap v12665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:37 smithi115 bash[46934]: cluster 2022-04-22T23:23:37.301479+0000 mgr.y (mgr.34107) 12182 : cluster [DBG] pgmap v12665: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:38 smithi121 bash[36902]: audit 2022-04-22T23:23:38.477942+0000 mon.a (mon.0) 16791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:38 smithi115 bash[45495]: audit 2022-04-22T23:23:38.477942+0000 mon.a (mon.0) 16791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:38 smithi115 bash[46934]: audit 2022-04-22T23:23:38.477942+0000 mon.a (mon.0) 16791 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:40.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:39 smithi121 bash[36902]: cluster 2022-04-22T23:23:39.302666+0000 mgr.y (mgr.34107) 12183 : cluster [DBG] pgmap v12666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:39 smithi115 bash[45495]: cluster 2022-04-22T23:23:39.302666+0000 mgr.y (mgr.34107) 12183 : cluster [DBG] pgmap v12666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:40.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:39 smithi115 bash[46934]: cluster 2022-04-22T23:23:39.302666+0000 mgr.y (mgr.34107) 12183 : cluster [DBG] pgmap v12666: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:40 smithi121 bash[36902]: audit 2022-04-22T23:23:40.208187+0000 mon.a (mon.0) 16792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:40 smithi115 bash[45495]: audit 2022-04-22T23:23:40.208187+0000 mon.a (mon.0) 16792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:40 smithi115 bash[46934]: audit 2022-04-22T23:23:40.208187+0000 mon.a (mon.0) 16792 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:41 smithi121 bash[36902]: cluster 2022-04-22T23:23:41.303864+0000 mgr.y (mgr.34107) 12184 : cluster [DBG] pgmap v12667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:42.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:41 smithi121 bash[36902]: audit 2022-04-22T23:23:41.846903+0000 mon.a (mon.0) 16793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:41 smithi115 bash[45495]: cluster 2022-04-22T23:23:41.303864+0000 mgr.y (mgr.34107) 12184 : cluster [DBG] pgmap v12667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:42.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:41 smithi115 bash[45495]: audit 2022-04-22T23:23:41.846903+0000 mon.a (mon.0) 16793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:41 smithi115 bash[46934]: cluster 2022-04-22T23:23:41.303864+0000 mgr.y (mgr.34107) 12184 : cluster [DBG] pgmap v12667: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:42.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:41 smithi115 bash[46934]: audit 2022-04-22T23:23:41.846903+0000 mon.a (mon.0) 16793 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:43.027Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:43 smithi121 bash[36902]: cluster 2022-04-22T23:23:43.304974+0000 mgr.y (mgr.34107) 12185 : cluster [DBG] pgmap v12668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:43 smithi121 bash[36902]: audit 2022-04-22T23:23:43.481104+0000 mon.a (mon.0) 16794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:43 smithi115 bash[45495]: cluster 2022-04-22T23:23:43.304974+0000 mgr.y (mgr.34107) 12185 : cluster [DBG] pgmap v12668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:43 smithi115 bash[45495]: audit 2022-04-22T23:23:43.481104+0000 mon.a (mon.0) 16794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:43 smithi115 bash[46934]: cluster 2022-04-22T23:23:43.304974+0000 mgr.y (mgr.34107) 12185 : cluster [DBG] pgmap v12668: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:43 smithi115 bash[46934]: audit 2022-04-22T23:23:43.481104+0000 mon.a (mon.0) 16794 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:45.268 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:44.885Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:45 smithi115 bash[45495]: audit 2022-04-22T23:23:45.212047+0000 mon.a (mon.0) 16795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:45 smithi115 bash[46934]: audit 2022-04-22T23:23:45.212047+0000 mon.a (mon.0) 16795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:45 smithi121 bash[36902]: audit 2022-04-22T23:23:45.212047+0000 mon.a (mon.0) 16795 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:46 smithi115 bash[46934]: cluster 2022-04-22T23:23:45.306207+0000 mgr.y (mgr.34107) 12186 : cluster [DBG] pgmap v12669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:46 smithi115 bash[45495]: cluster 2022-04-22T23:23:45.306207+0000 mgr.y (mgr.34107) 12186 : cluster [DBG] pgmap v12669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:46 smithi121 bash[36902]: cluster 2022-04-22T23:23:45.306207+0000 mgr.y (mgr.34107) 12186 : cluster [DBG] pgmap v12669: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:47 smithi115 bash[46934]: audit 2022-04-22T23:23:46.849921+0000 mon.a (mon.0) 16796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:47 smithi115 bash[45495]: audit 2022-04-22T23:23:46.849921+0000 mon.a (mon.0) 16796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:47 smithi121 bash[36902]: audit 2022-04-22T23:23:46.849921+0000 mon.a (mon.0) 16796 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:48 smithi115 bash[45495]: cluster 2022-04-22T23:23:47.307319+0000 mgr.y (mgr.34107) 12187 : cluster [DBG] pgmap v12670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:48 smithi115 bash[46934]: cluster 2022-04-22T23:23:47.307319+0000 mgr.y (mgr.34107) 12187 : cluster [DBG] pgmap v12670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:48 smithi121 bash[36902]: cluster 2022-04-22T23:23:47.307319+0000 mgr.y (mgr.34107) 12187 : cluster [DBG] pgmap v12670: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:49 smithi115 bash[45495]: audit 2022-04-22T23:23:48.484542+0000 mon.a (mon.0) 16797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:49 smithi115 bash[46934]: audit 2022-04-22T23:23:48.484542+0000 mon.a (mon.0) 16797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:49 smithi121 bash[36902]: audit 2022-04-22T23:23:48.484542+0000 mon.a (mon.0) 16797 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:50 smithi121 bash[36902]: cluster 2022-04-22T23:23:49.308398+0000 mgr.y (mgr.34107) 12188 : cluster [DBG] pgmap v12671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:50 smithi121 bash[36902]: audit 2022-04-22T23:23:50.215774+0000 mon.a (mon.0) 16798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:50 smithi115 bash[45495]: cluster 2022-04-22T23:23:49.308398+0000 mgr.y (mgr.34107) 12188 : cluster [DBG] pgmap v12671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:50 smithi115 bash[45495]: audit 2022-04-22T23:23:50.215774+0000 mon.a (mon.0) 16798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:50 smithi115 bash[46934]: cluster 2022-04-22T23:23:49.308398+0000 mgr.y (mgr.34107) 12188 : cluster [DBG] pgmap v12671: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:50 smithi115 bash[46934]: audit 2022-04-22T23:23:50.215774+0000 mon.a (mon.0) 16798 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:51 smithi121 bash[36902]: cluster 2022-04-22T23:23:51.309495+0000 mgr.y (mgr.34107) 12189 : cluster [DBG] pgmap v12672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:51 smithi115 bash[45495]: cluster 2022-04-22T23:23:51.309495+0000 mgr.y (mgr.34107) 12189 : cluster [DBG] pgmap v12672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:51 smithi115 bash[46934]: cluster 2022-04-22T23:23:51.309495+0000 mgr.y (mgr.34107) 12189 : cluster [DBG] pgmap v12672: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:52 smithi121 bash[36902]: audit 2022-04-22T23:23:51.852987+0000 mon.a (mon.0) 16799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:52 smithi115 bash[45495]: audit 2022-04-22T23:23:51.852987+0000 mon.a (mon.0) 16799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:52 smithi115 bash[46934]: audit 2022-04-22T23:23:51.852987+0000 mon.a (mon.0) 16799 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:53.029Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:54.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:53 smithi121 bash[36902]: cluster 2022-04-22T23:23:53.310640+0000 mgr.y (mgr.34107) 12190 : cluster [DBG] pgmap v12673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:54.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:53 smithi121 bash[36902]: audit 2022-04-22T23:23:53.487860+0000 mon.a (mon.0) 16800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:53 smithi115 bash[45495]: cluster 2022-04-22T23:23:53.310640+0000 mgr.y (mgr.34107) 12190 : cluster [DBG] pgmap v12673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:53 smithi115 bash[45495]: audit 2022-04-22T23:23:53.487860+0000 mon.a (mon.0) 16800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:53 smithi115 bash[46934]: cluster 2022-04-22T23:23:53.310640+0000 mgr.y (mgr.34107) 12190 : cluster [DBG] pgmap v12673: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:53 smithi115 bash[46934]: audit 2022-04-22T23:23:53.487860+0000 mon.a (mon.0) 16800 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:55.276 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:23:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:23:54.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:23:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:55 smithi115 bash[45495]: audit 2022-04-22T23:23:55.220448+0000 mon.a (mon.0) 16801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:55 smithi115 bash[46934]: audit 2022-04-22T23:23:55.220448+0000 mon.a (mon.0) 16801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:55 smithi121 bash[36902]: audit 2022-04-22T23:23:55.220448+0000 mon.a (mon.0) 16801 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:23:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:56 smithi115 bash[45495]: cluster 2022-04-22T23:23:55.311733+0000 mgr.y (mgr.34107) 12191 : cluster [DBG] pgmap v12674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:56 smithi115 bash[46934]: cluster 2022-04-22T23:23:55.311733+0000 mgr.y (mgr.34107) 12191 : cluster [DBG] pgmap v12674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:56 smithi121 bash[36902]: cluster 2022-04-22T23:23:55.311733+0000 mgr.y (mgr.34107) 12191 : cluster [DBG] pgmap v12674: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:57 smithi115 bash[45495]: audit 2022-04-22T23:23:56.855992+0000 mon.a (mon.0) 16802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:57 smithi115 bash[46934]: audit 2022-04-22T23:23:56.855992+0000 mon.a (mon.0) 16802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:57 smithi121 bash[36902]: audit 2022-04-22T23:23:56.855992+0000 mon.a (mon.0) 16802 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:23:58.414 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:58 smithi121 bash[36902]: cluster 2022-04-22T23:23:57.312813+0000 mgr.y (mgr.34107) 12192 : cluster [DBG] pgmap v12675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:58.414 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:58 smithi121 bash[36902]: audit 2022-04-22T23:23:58.073877+0000 mon.a (mon.0) 16803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:23:58.414 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:58 smithi121 bash[36902]: audit 2022-04-22T23:23:58.076916+0000 mon.a (mon.0) 16804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:23:58.414 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:58 smithi121 bash[36902]: audit 2022-04-22T23:23:58.078776+0000 mon.a (mon.0) 16805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:23:58.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[45495]: cluster 2022-04-22T23:23:57.312813+0000 mgr.y (mgr.34107) 12192 : cluster [DBG] pgmap v12675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:58.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[45495]: audit 2022-04-22T23:23:58.073877+0000 mon.a (mon.0) 16803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:23:58.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[45495]: audit 2022-04-22T23:23:58.076916+0000 mon.a (mon.0) 16804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:23:58.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[45495]: audit 2022-04-22T23:23:58.078776+0000 mon.a (mon.0) 16805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:23:58.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[46934]: cluster 2022-04-22T23:23:57.312813+0000 mgr.y (mgr.34107) 12192 : cluster [DBG] pgmap v12675: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:58.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[46934]: audit 2022-04-22T23:23:58.073877+0000 mon.a (mon.0) 16803 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:23:58.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[46934]: audit 2022-04-22T23:23:58.076916+0000 mon.a (mon.0) 16804 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:23:58.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:58 smithi115 bash[46934]: audit 2022-04-22T23:23:58.078776+0000 mon.a (mon.0) 16805 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:23:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:59 smithi121 bash[36902]: audit 2022-04-22T23:23:58.456766+0000 mon.a (mon.0) 16806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:23:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:59 smithi121 bash[36902]: audit 2022-04-22T23:23:58.491131+0000 mon.a (mon.0) 16807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:23:59 smithi121 bash[36902]: cluster 2022-04-22T23:23:59.314019+0000 mgr.y (mgr.34107) 12193 : cluster [DBG] pgmap v12676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[46934]: audit 2022-04-22T23:23:58.456766+0000 mon.a (mon.0) 16806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:23:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[46934]: audit 2022-04-22T23:23:58.491131+0000 mon.a (mon.0) 16807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[46934]: cluster 2022-04-22T23:23:59.314019+0000 mgr.y (mgr.34107) 12193 : cluster [DBG] pgmap v12676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:23:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[45495]: audit 2022-04-22T23:23:58.456766+0000 mon.a (mon.0) 16806 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:23:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[45495]: audit 2022-04-22T23:23:58.491131+0000 mon.a (mon.0) 16807 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:23:59.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:23:59 smithi115 bash[45495]: cluster 2022-04-22T23:23:59.314019+0000 mgr.y (mgr.34107) 12193 : cluster [DBG] pgmap v12676: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:00 smithi121 bash[36902]: audit 2022-04-22T23:24:00.224665+0000 mon.a (mon.0) 16808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:00 smithi115 bash[45495]: audit 2022-04-22T23:24:00.224665+0000 mon.a (mon.0) 16808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:00 smithi115 bash[46934]: audit 2022-04-22T23:24:00.224665+0000 mon.a (mon.0) 16808 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:01 smithi121 bash[36902]: cluster 2022-04-22T23:24:01.315152+0000 mgr.y (mgr.34107) 12194 : cluster [DBG] pgmap v12677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:01 smithi115 bash[45495]: cluster 2022-04-22T23:24:01.315152+0000 mgr.y (mgr.34107) 12194 : cluster [DBG] pgmap v12677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:01 smithi115 bash[46934]: cluster 2022-04-22T23:24:01.315152+0000 mgr.y (mgr.34107) 12194 : cluster [DBG] pgmap v12677: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:02 smithi121 bash[36902]: audit 2022-04-22T23:24:01.858903+0000 mon.a (mon.0) 16809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:02 smithi115 bash[45495]: audit 2022-04-22T23:24:01.858903+0000 mon.a (mon.0) 16809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:02.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:02 smithi115 bash[46934]: audit 2022-04-22T23:24:01.858903+0000 mon.a (mon.0) 16809 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:03.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:03 smithi121 bash[36902]: cluster 2022-04-22T23:24:03.316305+0000 mgr.y (mgr.34107) 12195 : cluster [DBG] pgmap v12678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:03 smithi121 bash[36902]: audit 2022-04-22T23:24:03.494439+0000 mon.a (mon.0) 16810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:03 smithi115 bash[45495]: cluster 2022-04-22T23:24:03.316305+0000 mgr.y (mgr.34107) 12195 : cluster [DBG] pgmap v12678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:03 smithi115 bash[45495]: audit 2022-04-22T23:24:03.494439+0000 mon.a (mon.0) 16810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:03 smithi115 bash[46934]: cluster 2022-04-22T23:24:03.316305+0000 mgr.y (mgr.34107) 12195 : cluster [DBG] pgmap v12678: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:03 smithi115 bash[46934]: audit 2022-04-22T23:24:03.494439+0000 mon.a (mon.0) 16810 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:05.046 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:24:05.284 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:04.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:05.394 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 7m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:24:05.395 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:24:05.396 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:24:05.397 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:24:05.397 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:24:05.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:05 smithi115 bash[45495]: audit 2022-04-22T23:24:05.228141+0000 mon.a (mon.0) 16811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:05.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:05 smithi115 bash[46934]: audit 2022-04-22T23:24:05.228141+0000 mon.a (mon.0) 16811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:05 smithi121 bash[36902]: audit 2022-04-22T23:24:05.228141+0000 mon.a (mon.0) 16811 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:05.799 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:24:05.799 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:24:05.799 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:24:05.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:05.801 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:24:05.801 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:24:05.801 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:24:05.801 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:24:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:06 smithi121 bash[36902]: audit 2022-04-22T23:24:05.028462+0000 mgr.y (mgr.34107) 12196 : audit [DBG] from='client.57901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:06 smithi121 bash[36902]: cluster 2022-04-22T23:24:05.317349+0000 mgr.y (mgr.34107) 12197 : cluster [DBG] pgmap v12679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:06.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:06 smithi121 bash[36902]: audit 2022-04-22T23:24:05.795529+0000 mon.a (mon.0) 16812 : audit [DBG] from='client.? 172.21.15.115:0/2972365559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[45495]: audit 2022-04-22T23:24:05.028462+0000 mgr.y (mgr.34107) 12196 : audit [DBG] from='client.57901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[45495]: cluster 2022-04-22T23:24:05.317349+0000 mgr.y (mgr.34107) 12197 : cluster [DBG] pgmap v12679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[45495]: audit 2022-04-22T23:24:05.795529+0000 mon.a (mon.0) 16812 : audit [DBG] from='client.? 172.21.15.115:0/2972365559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[46934]: audit 2022-04-22T23:24:05.028462+0000 mgr.y (mgr.34107) 12196 : audit [DBG] from='client.57901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[46934]: cluster 2022-04-22T23:24:05.317349+0000 mgr.y (mgr.34107) 12197 : cluster [DBG] pgmap v12679: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:06.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:06 smithi115 bash[46934]: audit 2022-04-22T23:24:05.795529+0000 mon.a (mon.0) 16812 : audit [DBG] from='client.? 172.21.15.115:0/2972365559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:07 smithi121 bash[36902]: audit 2022-04-22T23:24:05.370959+0000 mgr.y (mgr.34107) 12198 : audit [DBG] from='client.57907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:07 smithi121 bash[36902]: audit 2022-04-22T23:24:06.861847+0000 mon.a (mon.0) 16813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:07 smithi115 bash[45495]: audit 2022-04-22T23:24:05.370959+0000 mgr.y (mgr.34107) 12198 : audit [DBG] from='client.57907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:07 smithi115 bash[45495]: audit 2022-04-22T23:24:06.861847+0000 mon.a (mon.0) 16813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:07 smithi115 bash[46934]: audit 2022-04-22T23:24:05.370959+0000 mgr.y (mgr.34107) 12198 : audit [DBG] from='client.57907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:07 smithi115 bash[46934]: audit 2022-04-22T23:24:06.861847+0000 mon.a (mon.0) 16813 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:08 smithi121 bash[36902]: cluster 2022-04-22T23:24:07.318429+0000 mgr.y (mgr.34107) 12199 : cluster [DBG] pgmap v12680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:08.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:08 smithi115 bash[45495]: cluster 2022-04-22T23:24:07.318429+0000 mgr.y (mgr.34107) 12199 : cluster [DBG] pgmap v12680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:08.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:08 smithi115 bash[46934]: cluster 2022-04-22T23:24:07.318429+0000 mgr.y (mgr.34107) 12199 : cluster [DBG] pgmap v12680: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:09 smithi121 bash[36902]: audit 2022-04-22T23:24:08.497511+0000 mon.a (mon.0) 16814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:09 smithi115 bash[45495]: audit 2022-04-22T23:24:08.497511+0000 mon.a (mon.0) 16814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:09 smithi115 bash[46934]: audit 2022-04-22T23:24:08.497511+0000 mon.a (mon.0) 16814 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:10 smithi121 bash[36902]: cluster 2022-04-22T23:24:09.319453+0000 mgr.y (mgr.34107) 12200 : cluster [DBG] pgmap v12681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:10 smithi121 bash[36902]: audit 2022-04-22T23:24:10.231962+0000 mon.a (mon.0) 16815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:10 smithi115 bash[45495]: cluster 2022-04-22T23:24:09.319453+0000 mgr.y (mgr.34107) 12200 : cluster [DBG] pgmap v12681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:10 smithi115 bash[45495]: audit 2022-04-22T23:24:10.231962+0000 mon.a (mon.0) 16815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:10 smithi115 bash[46934]: cluster 2022-04-22T23:24:09.319453+0000 mgr.y (mgr.34107) 12200 : cluster [DBG] pgmap v12681: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:10 smithi115 bash[46934]: audit 2022-04-22T23:24:10.231962+0000 mon.a (mon.0) 16815 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:11 smithi121 bash[36902]: cluster 2022-04-22T23:24:11.320552+0000 mgr.y (mgr.34107) 12201 : cluster [DBG] pgmap v12682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:11 smithi115 bash[45495]: cluster 2022-04-22T23:24:11.320552+0000 mgr.y (mgr.34107) 12201 : cluster [DBG] pgmap v12682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:11 smithi115 bash[46934]: cluster 2022-04-22T23:24:11.320552+0000 mgr.y (mgr.34107) 12201 : cluster [DBG] pgmap v12682: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:12 smithi121 bash[36902]: audit 2022-04-22T23:24:11.864804+0000 mon.a (mon.0) 16816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:12 smithi115 bash[45495]: audit 2022-04-22T23:24:11.864804+0000 mon.a (mon.0) 16816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:12 smithi115 bash[46934]: audit 2022-04-22T23:24:11.864804+0000 mon.a (mon.0) 16816 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:13.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:13 smithi121 bash[36902]: cluster 2022-04-22T23:24:13.321702+0000 mgr.y (mgr.34107) 12202 : cluster [DBG] pgmap v12683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:13 smithi121 bash[36902]: audit 2022-04-22T23:24:13.500480+0000 mon.a (mon.0) 16817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:13 smithi115 bash[45495]: cluster 2022-04-22T23:24:13.321702+0000 mgr.y (mgr.34107) 12202 : cluster [DBG] pgmap v12683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:13 smithi115 bash[45495]: audit 2022-04-22T23:24:13.500480+0000 mon.a (mon.0) 16817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:13 smithi115 bash[46934]: cluster 2022-04-22T23:24:13.321702+0000 mgr.y (mgr.34107) 12202 : cluster [DBG] pgmap v12683: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:13 smithi115 bash[46934]: audit 2022-04-22T23:24:13.500480+0000 mon.a (mon.0) 16817 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:14.887Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:15 smithi121 bash[36902]: audit 2022-04-22T23:24:15.235740+0000 mon.a (mon.0) 16818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:15 smithi115 bash[45495]: audit 2022-04-22T23:24:15.235740+0000 mon.a (mon.0) 16818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:15 smithi115 bash[46934]: audit 2022-04-22T23:24:15.235740+0000 mon.a (mon.0) 16818 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:16 smithi121 bash[36902]: cluster 2022-04-22T23:24:15.322841+0000 mgr.y (mgr.34107) 12203 : cluster [DBG] pgmap v12684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:16 smithi115 bash[45495]: cluster 2022-04-22T23:24:15.322841+0000 mgr.y (mgr.34107) 12203 : cluster [DBG] pgmap v12684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:16 smithi115 bash[46934]: cluster 2022-04-22T23:24:15.322841+0000 mgr.y (mgr.34107) 12203 : cluster [DBG] pgmap v12684: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:17 smithi121 bash[36902]: audit 2022-04-22T23:24:16.867892+0000 mon.a (mon.0) 16819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:17 smithi115 bash[45495]: audit 2022-04-22T23:24:16.867892+0000 mon.a (mon.0) 16819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:17.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:17 smithi115 bash[46934]: audit 2022-04-22T23:24:16.867892+0000 mon.a (mon.0) 16819 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:18 smithi121 bash[36902]: cluster 2022-04-22T23:24:17.324012+0000 mgr.y (mgr.34107) 12204 : cluster [DBG] pgmap v12685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:18 smithi121 bash[36902]: audit 2022-04-22T23:24:17.398283+0000 mon.a (mon.0) 16820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:24:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:18 smithi121 bash[36902]: audit 2022-04-22T23:24:17.398788+0000 mon.a (mon.0) 16821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:24:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[46934]: cluster 2022-04-22T23:24:17.324012+0000 mgr.y (mgr.34107) 12204 : cluster [DBG] pgmap v12685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[46934]: audit 2022-04-22T23:24:17.398283+0000 mon.a (mon.0) 16820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:24:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[46934]: audit 2022-04-22T23:24:17.398788+0000 mon.a (mon.0) 16821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:24:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[45495]: cluster 2022-04-22T23:24:17.324012+0000 mgr.y (mgr.34107) 12204 : cluster [DBG] pgmap v12685: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[45495]: audit 2022-04-22T23:24:17.398283+0000 mon.a (mon.0) 16820 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:24:18.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:18 smithi115 bash[45495]: audit 2022-04-22T23:24:17.398788+0000 mon.a (mon.0) 16821 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:24:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:19 smithi121 bash[36902]: audit 2022-04-22T23:24:18.503749+0000 mon.a (mon.0) 16822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:19 smithi115 bash[45495]: audit 2022-04-22T23:24:18.503749+0000 mon.a (mon.0) 16822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:19 smithi115 bash[46934]: audit 2022-04-22T23:24:18.503749+0000 mon.a (mon.0) 16822 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:20 smithi121 bash[36902]: cluster 2022-04-22T23:24:19.325168+0000 mgr.y (mgr.34107) 12205 : cluster [DBG] pgmap v12686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:20 smithi121 bash[36902]: audit 2022-04-22T23:24:20.239334+0000 mon.a (mon.0) 16823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:20 smithi115 bash[45495]: cluster 2022-04-22T23:24:19.325168+0000 mgr.y (mgr.34107) 12205 : cluster [DBG] pgmap v12686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:20 smithi115 bash[45495]: audit 2022-04-22T23:24:20.239334+0000 mon.a (mon.0) 16823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:20 smithi115 bash[46934]: cluster 2022-04-22T23:24:19.325168+0000 mgr.y (mgr.34107) 12205 : cluster [DBG] pgmap v12686: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:20 smithi115 bash[46934]: audit 2022-04-22T23:24:20.239334+0000 mon.a (mon.0) 16823 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:21 smithi121 bash[36902]: cluster 2022-04-22T23:24:21.326312+0000 mgr.y (mgr.34107) 12206 : cluster [DBG] pgmap v12687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:21 smithi115 bash[45495]: cluster 2022-04-22T23:24:21.326312+0000 mgr.y (mgr.34107) 12206 : cluster [DBG] pgmap v12687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:21 smithi115 bash[46934]: cluster 2022-04-22T23:24:21.326312+0000 mgr.y (mgr.34107) 12206 : cluster [DBG] pgmap v12687: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:22 smithi121 bash[36902]: audit 2022-04-22T23:24:21.871070+0000 mon.a (mon.0) 16824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:22 smithi115 bash[45495]: audit 2022-04-22T23:24:21.871070+0000 mon.a (mon.0) 16824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:22 smithi115 bash[46934]: audit 2022-04-22T23:24:21.871070+0000 mon.a (mon.0) 16824 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:23.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:23.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:23 smithi121 bash[36902]: cluster 2022-04-22T23:24:23.327403+0000 mgr.y (mgr.34107) 12207 : cluster [DBG] pgmap v12688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:23 smithi121 bash[36902]: audit 2022-04-22T23:24:23.506835+0000 mon.a (mon.0) 16825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:23 smithi115 bash[45495]: cluster 2022-04-22T23:24:23.327403+0000 mgr.y (mgr.34107) 12207 : cluster [DBG] pgmap v12688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:23 smithi115 bash[45495]: audit 2022-04-22T23:24:23.506835+0000 mon.a (mon.0) 16825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:23 smithi115 bash[46934]: cluster 2022-04-22T23:24:23.327403+0000 mgr.y (mgr.34107) 12207 : cluster [DBG] pgmap v12688: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:23 smithi115 bash[46934]: audit 2022-04-22T23:24:23.506835+0000 mon.a (mon.0) 16825 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:24.888Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:25 smithi121 bash[36902]: audit 2022-04-22T23:24:25.242951+0000 mon.a (mon.0) 16826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:25 smithi115 bash[46934]: audit 2022-04-22T23:24:25.242951+0000 mon.a (mon.0) 16826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:25 smithi115 bash[45495]: audit 2022-04-22T23:24:25.242951+0000 mon.a (mon.0) 16826 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:26 smithi121 bash[36902]: cluster 2022-04-22T23:24:25.328415+0000 mgr.y (mgr.34107) 12208 : cluster [DBG] pgmap v12689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:26 smithi115 bash[45495]: cluster 2022-04-22T23:24:25.328415+0000 mgr.y (mgr.34107) 12208 : cluster [DBG] pgmap v12689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:26 smithi115 bash[46934]: cluster 2022-04-22T23:24:25.328415+0000 mgr.y (mgr.34107) 12208 : cluster [DBG] pgmap v12689: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:27 smithi121 bash[36902]: audit 2022-04-22T23:24:26.874261+0000 mon.a (mon.0) 16827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:27 smithi115 bash[45495]: audit 2022-04-22T23:24:26.874261+0000 mon.a (mon.0) 16827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:27 smithi115 bash[46934]: audit 2022-04-22T23:24:26.874261+0000 mon.a (mon.0) 16827 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:28 smithi121 bash[36902]: cluster 2022-04-22T23:24:27.329549+0000 mgr.y (mgr.34107) 12209 : cluster [DBG] pgmap v12690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:28.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:28 smithi115 bash[45495]: cluster 2022-04-22T23:24:27.329549+0000 mgr.y (mgr.34107) 12209 : cluster [DBG] pgmap v12690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:28 smithi115 bash[46934]: cluster 2022-04-22T23:24:27.329549+0000 mgr.y (mgr.34107) 12209 : cluster [DBG] pgmap v12690: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:29 smithi121 bash[36902]: audit 2022-04-22T23:24:28.510092+0000 mon.a (mon.0) 16828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:29 smithi115 bash[46934]: audit 2022-04-22T23:24:28.510092+0000 mon.a (mon.0) 16828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:29 smithi115 bash[45495]: audit 2022-04-22T23:24:28.510092+0000 mon.a (mon.0) 16828 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:30 smithi121 bash[36902]: cluster 2022-04-22T23:24:29.330391+0000 mgr.y (mgr.34107) 12210 : cluster [DBG] pgmap v12691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:30 smithi121 bash[36902]: audit 2022-04-22T23:24:30.246778+0000 mon.a (mon.0) 16829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:30 smithi115 bash[45495]: cluster 2022-04-22T23:24:29.330391+0000 mgr.y (mgr.34107) 12210 : cluster [DBG] pgmap v12691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:30 smithi115 bash[45495]: audit 2022-04-22T23:24:30.246778+0000 mon.a (mon.0) 16829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:30 smithi115 bash[46934]: cluster 2022-04-22T23:24:29.330391+0000 mgr.y (mgr.34107) 12210 : cluster [DBG] pgmap v12691: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:30.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:30 smithi115 bash[46934]: audit 2022-04-22T23:24:30.246778+0000 mon.a (mon.0) 16829 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:31 smithi121 bash[36902]: cluster 2022-04-22T23:24:31.331416+0000 mgr.y (mgr.34107) 12211 : cluster [DBG] pgmap v12692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:31 smithi115 bash[45495]: cluster 2022-04-22T23:24:31.331416+0000 mgr.y (mgr.34107) 12211 : cluster [DBG] pgmap v12692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:31 smithi115 bash[46934]: cluster 2022-04-22T23:24:31.331416+0000 mgr.y (mgr.34107) 12211 : cluster [DBG] pgmap v12692: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:32 smithi121 bash[36902]: audit 2022-04-22T23:24:31.877197+0000 mon.a (mon.0) 16830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:32.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:32 smithi115 bash[45495]: audit 2022-04-22T23:24:31.877197+0000 mon.a (mon.0) 16830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:32 smithi115 bash[46934]: audit 2022-04-22T23:24:31.877197+0000 mon.a (mon.0) 16830 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:33.031Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:33 smithi121 bash[36902]: cluster 2022-04-22T23:24:33.332522+0000 mgr.y (mgr.34107) 12212 : cluster [DBG] pgmap v12693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:33 smithi121 bash[36902]: audit 2022-04-22T23:24:33.513283+0000 mon.a (mon.0) 16831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:34.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:33 smithi115 bash[45495]: cluster 2022-04-22T23:24:33.332522+0000 mgr.y (mgr.34107) 12212 : cluster [DBG] pgmap v12693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:33 smithi115 bash[45495]: audit 2022-04-22T23:24:33.513283+0000 mon.a (mon.0) 16831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:33 smithi115 bash[46934]: cluster 2022-04-22T23:24:33.332522+0000 mgr.y (mgr.34107) 12212 : cluster [DBG] pgmap v12693: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:33 smithi115 bash[46934]: audit 2022-04-22T23:24:33.513283+0000 mon.a (mon.0) 16831 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:34.889Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:35 smithi121 bash[36902]: audit 2022-04-22T23:24:35.250739+0000 mon.a (mon.0) 16832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:35 smithi115 bash[45495]: audit 2022-04-22T23:24:35.250739+0000 mon.a (mon.0) 16832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:35 smithi115 bash[46934]: audit 2022-04-22T23:24:35.250739+0000 mon.a (mon.0) 16832 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:36.175 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 7m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 7m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:24:36.521 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 6m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 6m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 7m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 6m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 6m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 6m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 6m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 6m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:24:36.522 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 7m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:24:36.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 7m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:24:36.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 7m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:24:36.523 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:24:36.523 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:24:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:36 smithi121 bash[36902]: cluster 2022-04-22T23:24:35.333585+0000 mgr.y (mgr.34107) 12213 : cluster [DBG] pgmap v12694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:36 smithi115 bash[45495]: cluster 2022-04-22T23:24:35.333585+0000 mgr.y (mgr.34107) 12213 : cluster [DBG] pgmap v12694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:36 smithi115 bash[46934]: cluster 2022-04-22T23:24:35.333585+0000 mgr.y (mgr.34107) 12213 : cluster [DBG] pgmap v12694: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:36.930 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:24:36.931 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:24:36.932 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:24:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:37 smithi121 bash[36902]: audit 2022-04-22T23:24:36.158220+0000 mgr.y (mgr.34107) 12214 : audit [DBG] from='client.47979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:37 smithi121 bash[36902]: audit 2022-04-22T23:24:36.880305+0000 mon.a (mon.0) 16833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:37 smithi121 bash[36902]: audit 2022-04-22T23:24:36.926964+0000 mon.c (mon.1) 413 : audit [DBG] from='client.? 172.21.15.115:0/2036178771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[45495]: audit 2022-04-22T23:24:36.158220+0000 mgr.y (mgr.34107) 12214 : audit [DBG] from='client.47979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[45495]: audit 2022-04-22T23:24:36.880305+0000 mon.a (mon.0) 16833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[45495]: audit 2022-04-22T23:24:36.926964+0000 mon.c (mon.1) 413 : audit [DBG] from='client.? 172.21.15.115:0/2036178771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[46934]: audit 2022-04-22T23:24:36.158220+0000 mgr.y (mgr.34107) 12214 : audit [DBG] from='client.47979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[46934]: audit 2022-04-22T23:24:36.880305+0000 mon.a (mon.0) 16833 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:37 smithi115 bash[46934]: audit 2022-04-22T23:24:36.926964+0000 mon.c (mon.1) 413 : audit [DBG] from='client.? 172.21.15.115:0/2036178771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:24:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:38 smithi121 bash[36902]: audit 2022-04-22T23:24:36.498592+0000 mgr.y (mgr.34107) 12215 : audit [DBG] from='client.47985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:38 smithi121 bash[36902]: cluster 2022-04-22T23:24:37.334716+0000 mgr.y (mgr.34107) 12216 : cluster [DBG] pgmap v12695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:38 smithi115 bash[45495]: audit 2022-04-22T23:24:36.498592+0000 mgr.y (mgr.34107) 12215 : audit [DBG] from='client.47985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:38 smithi115 bash[45495]: cluster 2022-04-22T23:24:37.334716+0000 mgr.y (mgr.34107) 12216 : cluster [DBG] pgmap v12695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:38 smithi115 bash[46934]: audit 2022-04-22T23:24:36.498592+0000 mgr.y (mgr.34107) 12215 : audit [DBG] from='client.47985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:24:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:38 smithi115 bash[46934]: cluster 2022-04-22T23:24:37.334716+0000 mgr.y (mgr.34107) 12216 : cluster [DBG] pgmap v12695: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:39 smithi121 bash[36902]: audit 2022-04-22T23:24:38.516684+0000 mon.a (mon.0) 16834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:39.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:39 smithi115 bash[45495]: audit 2022-04-22T23:24:38.516684+0000 mon.a (mon.0) 16834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:39 smithi115 bash[46934]: audit 2022-04-22T23:24:38.516684+0000 mon.a (mon.0) 16834 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:40 smithi121 bash[36902]: cluster 2022-04-22T23:24:39.335641+0000 mgr.y (mgr.34107) 12217 : cluster [DBG] pgmap v12696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:40 smithi121 bash[36902]: audit 2022-04-22T23:24:40.254018+0000 mon.a (mon.0) 16835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:40 smithi115 bash[45495]: cluster 2022-04-22T23:24:39.335641+0000 mgr.y (mgr.34107) 12217 : cluster [DBG] pgmap v12696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:40 smithi115 bash[45495]: audit 2022-04-22T23:24:40.254018+0000 mon.a (mon.0) 16835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:40 smithi115 bash[46934]: cluster 2022-04-22T23:24:39.335641+0000 mgr.y (mgr.34107) 12217 : cluster [DBG] pgmap v12696: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:40 smithi115 bash[46934]: audit 2022-04-22T23:24:40.254018+0000 mon.a (mon.0) 16835 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:42 smithi121 bash[36902]: cluster 2022-04-22T23:24:41.336620+0000 mgr.y (mgr.34107) 12218 : cluster [DBG] pgmap v12697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:42 smithi121 bash[36902]: audit 2022-04-22T23:24:41.882880+0000 mon.a (mon.0) 16836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:42 smithi115 bash[45495]: cluster 2022-04-22T23:24:41.336620+0000 mgr.y (mgr.34107) 12218 : cluster [DBG] pgmap v12697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:42 smithi115 bash[45495]: audit 2022-04-22T23:24:41.882880+0000 mon.a (mon.0) 16836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:42 smithi115 bash[46934]: cluster 2022-04-22T23:24:41.336620+0000 mgr.y (mgr.34107) 12218 : cluster [DBG] pgmap v12697: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:42 smithi115 bash[46934]: audit 2022-04-22T23:24:41.882880+0000 mon.a (mon.0) 16836 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:43.032Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:43 smithi121 bash[36902]: cluster 2022-04-22T23:24:43.337672+0000 mgr.y (mgr.34107) 12219 : cluster [DBG] pgmap v12698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:44.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:43 smithi121 bash[36902]: audit 2022-04-22T23:24:43.519949+0000 mon.a (mon.0) 16837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:43 smithi115 bash[45495]: cluster 2022-04-22T23:24:43.337672+0000 mgr.y (mgr.34107) 12219 : cluster [DBG] pgmap v12698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:43 smithi115 bash[45495]: audit 2022-04-22T23:24:43.519949+0000 mon.a (mon.0) 16837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:43 smithi115 bash[46934]: cluster 2022-04-22T23:24:43.337672+0000 mgr.y (mgr.34107) 12219 : cluster [DBG] pgmap v12698: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:43 smithi115 bash[46934]: audit 2022-04-22T23:24:43.519949+0000 mon.a (mon.0) 16837 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:44.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:45 smithi121 bash[36902]: audit 2022-04-22T23:24:45.257702+0000 mon.a (mon.0) 16838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:45 smithi115 bash[45495]: audit 2022-04-22T23:24:45.257702+0000 mon.a (mon.0) 16838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:45 smithi115 bash[46934]: audit 2022-04-22T23:24:45.257702+0000 mon.a (mon.0) 16838 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:46 smithi121 bash[36902]: cluster 2022-04-22T23:24:45.338780+0000 mgr.y (mgr.34107) 12220 : cluster [DBG] pgmap v12699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:46 smithi115 bash[45495]: cluster 2022-04-22T23:24:45.338780+0000 mgr.y (mgr.34107) 12220 : cluster [DBG] pgmap v12699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:46 smithi115 bash[46934]: cluster 2022-04-22T23:24:45.338780+0000 mgr.y (mgr.34107) 12220 : cluster [DBG] pgmap v12699: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:47 smithi121 bash[36902]: audit 2022-04-22T23:24:46.885927+0000 mon.a (mon.0) 16839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:47 smithi115 bash[45495]: audit 2022-04-22T23:24:46.885927+0000 mon.a (mon.0) 16839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:47 smithi115 bash[46934]: audit 2022-04-22T23:24:46.885927+0000 mon.a (mon.0) 16839 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:48 smithi121 bash[36902]: cluster 2022-04-22T23:24:47.339868+0000 mgr.y (mgr.34107) 12221 : cluster [DBG] pgmap v12700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:48 smithi115 bash[45495]: cluster 2022-04-22T23:24:47.339868+0000 mgr.y (mgr.34107) 12221 : cluster [DBG] pgmap v12700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:48 smithi115 bash[46934]: cluster 2022-04-22T23:24:47.339868+0000 mgr.y (mgr.34107) 12221 : cluster [DBG] pgmap v12700: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:49.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:49 smithi121 bash[36902]: audit 2022-04-22T23:24:48.523152+0000 mon.a (mon.0) 16840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:49.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:49 smithi115 bash[45495]: audit 2022-04-22T23:24:48.523152+0000 mon.a (mon.0) 16840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:49.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:49 smithi115 bash[46934]: audit 2022-04-22T23:24:48.523152+0000 mon.a (mon.0) 16840 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:50 smithi121 bash[36902]: cluster 2022-04-22T23:24:49.340795+0000 mgr.y (mgr.34107) 12222 : cluster [DBG] pgmap v12701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:50 smithi121 bash[36902]: audit 2022-04-22T23:24:50.261196+0000 mon.a (mon.0) 16841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:50 smithi115 bash[45495]: cluster 2022-04-22T23:24:49.340795+0000 mgr.y (mgr.34107) 12222 : cluster [DBG] pgmap v12701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:50 smithi115 bash[45495]: audit 2022-04-22T23:24:50.261196+0000 mon.a (mon.0) 16841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:50 smithi115 bash[46934]: cluster 2022-04-22T23:24:49.340795+0000 mgr.y (mgr.34107) 12222 : cluster [DBG] pgmap v12701: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:50 smithi115 bash[46934]: audit 2022-04-22T23:24:50.261196+0000 mon.a (mon.0) 16841 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:51 smithi121 bash[36902]: cluster 2022-04-22T23:24:51.341862+0000 mgr.y (mgr.34107) 12223 : cluster [DBG] pgmap v12702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:51.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:51 smithi115 bash[45495]: cluster 2022-04-22T23:24:51.341862+0000 mgr.y (mgr.34107) 12223 : cluster [DBG] pgmap v12702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:51.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:51 smithi115 bash[46934]: cluster 2022-04-22T23:24:51.341862+0000 mgr.y (mgr.34107) 12223 : cluster [DBG] pgmap v12702: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:52 smithi121 bash[36902]: audit 2022-04-22T23:24:51.889013+0000 mon.a (mon.0) 16842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:52.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:52 smithi115 bash[45495]: audit 2022-04-22T23:24:51.889013+0000 mon.a (mon.0) 16842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:52 smithi115 bash[46934]: audit 2022-04-22T23:24:51.889013+0000 mon.a (mon.0) 16842 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:53.032Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:53 smithi121 bash[36902]: cluster 2022-04-22T23:24:53.342956+0000 mgr.y (mgr.34107) 12224 : cluster [DBG] pgmap v12703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:53 smithi121 bash[36902]: audit 2022-04-22T23:24:53.526191+0000 mon.a (mon.0) 16843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:53 smithi115 bash[45495]: cluster 2022-04-22T23:24:53.342956+0000 mgr.y (mgr.34107) 12224 : cluster [DBG] pgmap v12703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:53 smithi115 bash[45495]: audit 2022-04-22T23:24:53.526191+0000 mon.a (mon.0) 16843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:53 smithi115 bash[46934]: cluster 2022-04-22T23:24:53.342956+0000 mgr.y (mgr.34107) 12224 : cluster [DBG] pgmap v12703: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:53 smithi115 bash[46934]: audit 2022-04-22T23:24:53.526191+0000 mon.a (mon.0) 16843 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:24:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:24:54.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:24:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:55 smithi121 bash[36902]: audit 2022-04-22T23:24:55.264847+0000 mon.a (mon.0) 16844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:55 smithi115 bash[45495]: audit 2022-04-22T23:24:55.264847+0000 mon.a (mon.0) 16844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:55 smithi115 bash[46934]: audit 2022-04-22T23:24:55.264847+0000 mon.a (mon.0) 16844 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:24:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:56 smithi121 bash[36902]: cluster 2022-04-22T23:24:55.344122+0000 mgr.y (mgr.34107) 12225 : cluster [DBG] pgmap v12704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:56 smithi115 bash[45495]: cluster 2022-04-22T23:24:55.344122+0000 mgr.y (mgr.34107) 12225 : cluster [DBG] pgmap v12704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:56 smithi115 bash[46934]: cluster 2022-04-22T23:24:55.344122+0000 mgr.y (mgr.34107) 12225 : cluster [DBG] pgmap v12704: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:57 smithi121 bash[36902]: audit 2022-04-22T23:24:56.892023+0000 mon.a (mon.0) 16845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:57 smithi115 bash[45495]: audit 2022-04-22T23:24:56.892023+0000 mon.a (mon.0) 16845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:57 smithi115 bash[46934]: audit 2022-04-22T23:24:56.892023+0000 mon.a (mon.0) 16845 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:24:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:58 smithi121 bash[36902]: cluster 2022-04-22T23:24:57.345212+0000 mgr.y (mgr.34107) 12226 : cluster [DBG] pgmap v12705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:58.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:58 smithi115 bash[45495]: cluster 2022-04-22T23:24:57.345212+0000 mgr.y (mgr.34107) 12226 : cluster [DBG] pgmap v12705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:58 smithi115 bash[46934]: cluster 2022-04-22T23:24:57.345212+0000 mgr.y (mgr.34107) 12226 : cluster [DBG] pgmap v12705: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:24:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:59 smithi121 bash[36902]: audit 2022-04-22T23:24:58.464506+0000 mon.a (mon.0) 16846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:24:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:59 smithi121 bash[36902]: audit 2022-04-22T23:24:58.467716+0000 mon.a (mon.0) 16847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:24:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:59 smithi121 bash[36902]: audit 2022-04-22T23:24:58.469527+0000 mon.a (mon.0) 16848 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:24:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:59 smithi121 bash[36902]: audit 2022-04-22T23:24:58.528898+0000 mon.a (mon.0) 16849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:24:59 smithi121 bash[36902]: audit 2022-04-22T23:24:58.805134+0000 mon.a (mon.0) 16850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:24:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[45495]: audit 2022-04-22T23:24:58.464506+0000 mon.a (mon.0) 16846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:24:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[45495]: audit 2022-04-22T23:24:58.467716+0000 mon.a (mon.0) 16847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:24:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[45495]: audit 2022-04-22T23:24:58.469527+0000 mon.a (mon.0) 16848 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:24:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[45495]: audit 2022-04-22T23:24:58.528898+0000 mon.a (mon.0) 16849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[45495]: audit 2022-04-22T23:24:58.805134+0000 mon.a (mon.0) 16850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:24:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[46934]: audit 2022-04-22T23:24:58.464506+0000 mon.a (mon.0) 16846 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:24:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[46934]: audit 2022-04-22T23:24:58.467716+0000 mon.a (mon.0) 16847 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:24:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[46934]: audit 2022-04-22T23:24:58.469527+0000 mon.a (mon.0) 16848 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:24:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[46934]: audit 2022-04-22T23:24:58.528898+0000 mon.a (mon.0) 16849 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:24:59.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:24:59 smithi115 bash[46934]: audit 2022-04-22T23:24:58.805134+0000 mon.a (mon.0) 16850 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:25:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:00 smithi121 bash[36902]: cluster 2022-04-22T23:24:59.346131+0000 mgr.y (mgr.34107) 12227 : cluster [DBG] pgmap v12706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:00.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:00 smithi121 bash[36902]: audit 2022-04-22T23:25:00.268362+0000 mon.a (mon.0) 16851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:00 smithi115 bash[45495]: cluster 2022-04-22T23:24:59.346131+0000 mgr.y (mgr.34107) 12227 : cluster [DBG] pgmap v12706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:00.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:00 smithi115 bash[45495]: audit 2022-04-22T23:25:00.268362+0000 mon.a (mon.0) 16851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:00 smithi115 bash[46934]: cluster 2022-04-22T23:24:59.346131+0000 mgr.y (mgr.34107) 12227 : cluster [DBG] pgmap v12706: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:00.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:00 smithi115 bash[46934]: audit 2022-04-22T23:25:00.268362+0000 mon.a (mon.0) 16851 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:01 smithi121 bash[36902]: cluster 2022-04-22T23:25:01.347212+0000 mgr.y (mgr.34107) 12228 : cluster [DBG] pgmap v12707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:01 smithi115 bash[45495]: cluster 2022-04-22T23:25:01.347212+0000 mgr.y (mgr.34107) 12228 : cluster [DBG] pgmap v12707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:01 smithi115 bash[46934]: cluster 2022-04-22T23:25:01.347212+0000 mgr.y (mgr.34107) 12228 : cluster [DBG] pgmap v12707: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:02 smithi121 bash[36902]: audit 2022-04-22T23:25:01.894962+0000 mon.a (mon.0) 16852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:02 smithi115 bash[46934]: audit 2022-04-22T23:25:01.894962+0000 mon.a (mon.0) 16852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:02 smithi115 bash[45495]: audit 2022-04-22T23:25:01.894962+0000 mon.a (mon.0) 16852 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:03.033Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:03 smithi121 bash[36902]: cluster 2022-04-22T23:25:03.348326+0000 mgr.y (mgr.34107) 12229 : cluster [DBG] pgmap v12708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:04.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:03 smithi121 bash[36902]: audit 2022-04-22T23:25:03.531533+0000 mon.a (mon.0) 16853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:03 smithi115 bash[45495]: cluster 2022-04-22T23:25:03.348326+0000 mgr.y (mgr.34107) 12229 : cluster [DBG] pgmap v12708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:04.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:03 smithi115 bash[45495]: audit 2022-04-22T23:25:03.531533+0000 mon.a (mon.0) 16853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:03 smithi115 bash[46934]: cluster 2022-04-22T23:25:03.348326+0000 mgr.y (mgr.34107) 12229 : cluster [DBG] pgmap v12708: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:03 smithi115 bash[46934]: audit 2022-04-22T23:25:03.531533+0000 mon.a (mon.0) 16853 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:05.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:04.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:05 smithi121 bash[36902]: audit 2022-04-22T23:25:05.272125+0000 mon.a (mon.0) 16854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:05 smithi115 bash[45495]: audit 2022-04-22T23:25:05.272125+0000 mon.a (mon.0) 16854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:05 smithi115 bash[46934]: audit 2022-04-22T23:25:05.272125+0000 mon.a (mon.0) 16854 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:06 smithi121 bash[36902]: cluster 2022-04-22T23:25:05.349360+0000 mgr.y (mgr.34107) 12230 : cluster [DBG] pgmap v12709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:06.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:06 smithi115 bash[46934]: cluster 2022-04-22T23:25:05.349360+0000 mgr.y (mgr.34107) 12230 : cluster [DBG] pgmap v12709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:06 smithi115 bash[45495]: cluster 2022-04-22T23:25:05.349360+0000 mgr.y (mgr.34107) 12230 : cluster [DBG] pgmap v12709: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:07.311 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:25:07.655 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:25:07.655 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:25:07.655 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 8m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:25:07.655 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:25:07.656 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:25:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:25:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:07 smithi121 bash[36902]: audit 2022-04-22T23:25:06.898156+0000 mon.a (mon.0) 16855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:07 smithi115 bash[46934]: audit 2022-04-22T23:25:06.898156+0000 mon.a (mon.0) 16855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:07 smithi115 bash[45495]: audit 2022-04-22T23:25:06.898156+0000 mon.a (mon.0) 16855 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:08.059 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:25:08.060 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:25:08.061 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:25:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:08 smithi121 bash[36902]: audit 2022-04-22T23:25:07.289380+0000 mgr.y (mgr.34107) 12231 : audit [DBG] from='client.47997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:08 smithi121 bash[36902]: cluster 2022-04-22T23:25:07.350248+0000 mgr.y (mgr.34107) 12232 : cluster [DBG] pgmap v12710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:08 smithi121 bash[36902]: audit 2022-04-22T23:25:08.055864+0000 mon.c (mon.1) 414 : audit [DBG] from='client.? 172.21.15.115:0/2921159977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[46934]: audit 2022-04-22T23:25:07.289380+0000 mgr.y (mgr.34107) 12231 : audit [DBG] from='client.47997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[46934]: cluster 2022-04-22T23:25:07.350248+0000 mgr.y (mgr.34107) 12232 : cluster [DBG] pgmap v12710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[46934]: audit 2022-04-22T23:25:08.055864+0000 mon.c (mon.1) 414 : audit [DBG] from='client.? 172.21.15.115:0/2921159977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[45495]: audit 2022-04-22T23:25:07.289380+0000 mgr.y (mgr.34107) 12231 : audit [DBG] from='client.47997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[45495]: cluster 2022-04-22T23:25:07.350248+0000 mgr.y (mgr.34107) 12232 : cluster [DBG] pgmap v12710: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:08.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:08 smithi115 bash[45495]: audit 2022-04-22T23:25:08.055864+0000 mon.c (mon.1) 414 : audit [DBG] from='client.? 172.21.15.115:0/2921159977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:09 smithi121 bash[36902]: audit 2022-04-22T23:25:07.633257+0000 mgr.y (mgr.34107) 12233 : audit [DBG] from='client.57943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:09 smithi121 bash[36902]: audit 2022-04-22T23:25:08.534705+0000 mon.a (mon.0) 16856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:09 smithi115 bash[45495]: audit 2022-04-22T23:25:07.633257+0000 mgr.y (mgr.34107) 12233 : audit [DBG] from='client.57943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:09 smithi115 bash[45495]: audit 2022-04-22T23:25:08.534705+0000 mon.a (mon.0) 16856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:09 smithi115 bash[46934]: audit 2022-04-22T23:25:07.633257+0000 mgr.y (mgr.34107) 12233 : audit [DBG] from='client.57943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:09 smithi115 bash[46934]: audit 2022-04-22T23:25:08.534705+0000 mon.a (mon.0) 16856 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:10 smithi121 bash[36902]: cluster 2022-04-22T23:25:09.351177+0000 mgr.y (mgr.34107) 12234 : cluster [DBG] pgmap v12711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:10 smithi121 bash[36902]: audit 2022-04-22T23:25:10.275781+0000 mon.a (mon.0) 16857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:10 smithi115 bash[45495]: cluster 2022-04-22T23:25:09.351177+0000 mgr.y (mgr.34107) 12234 : cluster [DBG] pgmap v12711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:10 smithi115 bash[45495]: audit 2022-04-22T23:25:10.275781+0000 mon.a (mon.0) 16857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:10 smithi115 bash[46934]: cluster 2022-04-22T23:25:09.351177+0000 mgr.y (mgr.34107) 12234 : cluster [DBG] pgmap v12711: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:10 smithi115 bash[46934]: audit 2022-04-22T23:25:10.275781+0000 mon.a (mon.0) 16857 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:12 smithi121 bash[36902]: cluster 2022-04-22T23:25:11.352340+0000 mgr.y (mgr.34107) 12235 : cluster [DBG] pgmap v12712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:12 smithi121 bash[36902]: audit 2022-04-22T23:25:11.901367+0000 mon.a (mon.0) 16858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:12 smithi115 bash[45495]: cluster 2022-04-22T23:25:11.352340+0000 mgr.y (mgr.34107) 12235 : cluster [DBG] pgmap v12712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:12 smithi115 bash[45495]: audit 2022-04-22T23:25:11.901367+0000 mon.a (mon.0) 16858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:12 smithi115 bash[46934]: cluster 2022-04-22T23:25:11.352340+0000 mgr.y (mgr.34107) 12235 : cluster [DBG] pgmap v12712: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:12 smithi115 bash[46934]: audit 2022-04-22T23:25:11.901367+0000 mon.a (mon.0) 16858 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:13.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:13.034Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:13 smithi121 bash[36902]: cluster 2022-04-22T23:25:13.353463+0000 mgr.y (mgr.34107) 12236 : cluster [DBG] pgmap v12713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:14.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:13 smithi121 bash[36902]: audit 2022-04-22T23:25:13.537981+0000 mon.a (mon.0) 16859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:13 smithi115 bash[45495]: cluster 2022-04-22T23:25:13.353463+0000 mgr.y (mgr.34107) 12236 : cluster [DBG] pgmap v12713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:14.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:13 smithi115 bash[45495]: audit 2022-04-22T23:25:13.537981+0000 mon.a (mon.0) 16859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:13 smithi115 bash[46934]: cluster 2022-04-22T23:25:13.353463+0000 mgr.y (mgr.34107) 12236 : cluster [DBG] pgmap v12713: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:14.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:13 smithi115 bash[46934]: audit 2022-04-22T23:25:13.537981+0000 mon.a (mon.0) 16859 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:14.891Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:15 smithi121 bash[36902]: audit 2022-04-22T23:25:15.279414+0000 mon.a (mon.0) 16860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:15 smithi115 bash[45495]: audit 2022-04-22T23:25:15.279414+0000 mon.a (mon.0) 16860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:15 smithi115 bash[46934]: audit 2022-04-22T23:25:15.279414+0000 mon.a (mon.0) 16860 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:16 smithi121 bash[36902]: cluster 2022-04-22T23:25:15.354657+0000 mgr.y (mgr.34107) 12237 : cluster [DBG] pgmap v12714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:16.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:16 smithi115 bash[45495]: cluster 2022-04-22T23:25:15.354657+0000 mgr.y (mgr.34107) 12237 : cluster [DBG] pgmap v12714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:16 smithi115 bash[46934]: cluster 2022-04-22T23:25:15.354657+0000 mgr.y (mgr.34107) 12237 : cluster [DBG] pgmap v12714: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:17 smithi121 bash[36902]: audit 2022-04-22T23:25:16.904363+0000 mon.a (mon.0) 16861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:17 smithi115 bash[45495]: audit 2022-04-22T23:25:16.904363+0000 mon.a (mon.0) 16861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:17 smithi115 bash[46934]: audit 2022-04-22T23:25:16.904363+0000 mon.a (mon.0) 16861 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:18 smithi121 bash[36902]: cluster 2022-04-22T23:25:17.355749+0000 mgr.y (mgr.34107) 12238 : cluster [DBG] pgmap v12715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:18 smithi121 bash[36902]: audit 2022-04-22T23:25:17.406387+0000 mon.a (mon.0) 16862 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:25:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:18 smithi121 bash[36902]: audit 2022-04-22T23:25:17.410801+0000 mon.a (mon.0) 16863 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:25:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[45495]: cluster 2022-04-22T23:25:17.355749+0000 mgr.y (mgr.34107) 12238 : cluster [DBG] pgmap v12715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[45495]: audit 2022-04-22T23:25:17.406387+0000 mon.a (mon.0) 16862 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:25:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[45495]: audit 2022-04-22T23:25:17.410801+0000 mon.a (mon.0) 16863 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:25:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[46934]: cluster 2022-04-22T23:25:17.355749+0000 mgr.y (mgr.34107) 12238 : cluster [DBG] pgmap v12715: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[46934]: audit 2022-04-22T23:25:17.406387+0000 mon.a (mon.0) 16862 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:25:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:18 smithi115 bash[46934]: audit 2022-04-22T23:25:17.410801+0000 mon.a (mon.0) 16863 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:25:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:19 smithi121 bash[36902]: audit 2022-04-22T23:25:18.540856+0000 mon.a (mon.0) 16864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:19 smithi115 bash[45495]: audit 2022-04-22T23:25:18.540856+0000 mon.a (mon.0) 16864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:19 smithi115 bash[46934]: audit 2022-04-22T23:25:18.540856+0000 mon.a (mon.0) 16864 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:20 smithi121 bash[36902]: cluster 2022-04-22T23:25:19.356584+0000 mgr.y (mgr.34107) 12239 : cluster [DBG] pgmap v12716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:20 smithi121 bash[36902]: audit 2022-04-22T23:25:20.282758+0000 mon.a (mon.0) 16865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:20 smithi115 bash[45495]: cluster 2022-04-22T23:25:19.356584+0000 mgr.y (mgr.34107) 12239 : cluster [DBG] pgmap v12716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:20 smithi115 bash[45495]: audit 2022-04-22T23:25:20.282758+0000 mon.a (mon.0) 16865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:20 smithi115 bash[46934]: cluster 2022-04-22T23:25:19.356584+0000 mgr.y (mgr.34107) 12239 : cluster [DBG] pgmap v12716: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:20.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:20 smithi115 bash[46934]: audit 2022-04-22T23:25:20.282758+0000 mon.a (mon.0) 16865 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:22 smithi121 bash[36902]: cluster 2022-04-22T23:25:21.357637+0000 mgr.y (mgr.34107) 12240 : cluster [DBG] pgmap v12717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:22 smithi121 bash[36902]: audit 2022-04-22T23:25:21.907591+0000 mon.a (mon.0) 16866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:22 smithi115 bash[46934]: cluster 2022-04-22T23:25:21.357637+0000 mgr.y (mgr.34107) 12240 : cluster [DBG] pgmap v12717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:22 smithi115 bash[46934]: audit 2022-04-22T23:25:21.907591+0000 mon.a (mon.0) 16866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:22 smithi115 bash[45495]: cluster 2022-04-22T23:25:21.357637+0000 mgr.y (mgr.34107) 12240 : cluster [DBG] pgmap v12717: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:22.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:22 smithi115 bash[45495]: audit 2022-04-22T23:25:21.907591+0000 mon.a (mon.0) 16866 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:23.035Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:23 smithi121 bash[36902]: cluster 2022-04-22T23:25:23.358940+0000 mgr.y (mgr.34107) 12241 : cluster [DBG] pgmap v12718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:24.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:23 smithi121 bash[36902]: audit 2022-04-22T23:25:23.544302+0000 mon.a (mon.0) 16867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:24.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:23 smithi115 bash[46934]: cluster 2022-04-22T23:25:23.358940+0000 mgr.y (mgr.34107) 12241 : cluster [DBG] pgmap v12718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:23 smithi115 bash[46934]: audit 2022-04-22T23:25:23.544302+0000 mon.a (mon.0) 16867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:23 smithi115 bash[45495]: cluster 2022-04-22T23:25:23.358940+0000 mgr.y (mgr.34107) 12241 : cluster [DBG] pgmap v12718: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:23 smithi115 bash[45495]: audit 2022-04-22T23:25:23.544302+0000 mon.a (mon.0) 16867 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:24.893Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:25 smithi121 bash[36902]: audit 2022-04-22T23:25:25.286385+0000 mon.a (mon.0) 16868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:25 smithi115 bash[45495]: audit 2022-04-22T23:25:25.286385+0000 mon.a (mon.0) 16868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:25 smithi115 bash[46934]: audit 2022-04-22T23:25:25.286385+0000 mon.a (mon.0) 16868 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:26 smithi121 bash[36902]: cluster 2022-04-22T23:25:25.360036+0000 mgr.y (mgr.34107) 12242 : cluster [DBG] pgmap v12719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:26 smithi115 bash[45495]: cluster 2022-04-22T23:25:25.360036+0000 mgr.y (mgr.34107) 12242 : cluster [DBG] pgmap v12719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:26 smithi115 bash[46934]: cluster 2022-04-22T23:25:25.360036+0000 mgr.y (mgr.34107) 12242 : cluster [DBG] pgmap v12719: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:27 smithi121 bash[36902]: audit 2022-04-22T23:25:26.910655+0000 mon.a (mon.0) 16869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:27 smithi115 bash[45495]: audit 2022-04-22T23:25:26.910655+0000 mon.a (mon.0) 16869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:27 smithi115 bash[46934]: audit 2022-04-22T23:25:26.910655+0000 mon.a (mon.0) 16869 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:28.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:28 smithi121 bash[36902]: cluster 2022-04-22T23:25:27.361149+0000 mgr.y (mgr.34107) 12243 : cluster [DBG] pgmap v12720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:28.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:28 smithi115 bash[45495]: cluster 2022-04-22T23:25:27.361149+0000 mgr.y (mgr.34107) 12243 : cluster [DBG] pgmap v12720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:28.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:28 smithi115 bash[46934]: cluster 2022-04-22T23:25:27.361149+0000 mgr.y (mgr.34107) 12243 : cluster [DBG] pgmap v12720: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:29 smithi121 bash[36902]: audit 2022-04-22T23:25:28.547302+0000 mon.a (mon.0) 16870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:29.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:29 smithi115 bash[45495]: audit 2022-04-22T23:25:28.547302+0000 mon.a (mon.0) 16870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:29 smithi115 bash[46934]: audit 2022-04-22T23:25:28.547302+0000 mon.a (mon.0) 16870 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:30 smithi121 bash[36902]: cluster 2022-04-22T23:25:29.362349+0000 mgr.y (mgr.34107) 12244 : cluster [DBG] pgmap v12721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:30.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:30 smithi121 bash[36902]: audit 2022-04-22T23:25:30.290184+0000 mon.a (mon.0) 16871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:30.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:30 smithi115 bash[45495]: cluster 2022-04-22T23:25:29.362349+0000 mgr.y (mgr.34107) 12244 : cluster [DBG] pgmap v12721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:30.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:30 smithi115 bash[45495]: audit 2022-04-22T23:25:30.290184+0000 mon.a (mon.0) 16871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:30.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:30 smithi115 bash[46934]: cluster 2022-04-22T23:25:29.362349+0000 mgr.y (mgr.34107) 12244 : cluster [DBG] pgmap v12721: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:30.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:30 smithi115 bash[46934]: audit 2022-04-22T23:25:30.290184+0000 mon.a (mon.0) 16871 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:32 smithi121 bash[36902]: cluster 2022-04-22T23:25:31.363621+0000 mgr.y (mgr.34107) 12245 : cluster [DBG] pgmap v12722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:32 smithi121 bash[36902]: audit 2022-04-22T23:25:31.913824+0000 mon.a (mon.0) 16872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:32 smithi115 bash[45495]: cluster 2022-04-22T23:25:31.363621+0000 mgr.y (mgr.34107) 12245 : cluster [DBG] pgmap v12722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:32 smithi115 bash[45495]: audit 2022-04-22T23:25:31.913824+0000 mon.a (mon.0) 16872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:32 smithi115 bash[46934]: cluster 2022-04-22T23:25:31.363621+0000 mgr.y (mgr.34107) 12245 : cluster [DBG] pgmap v12722: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:32 smithi115 bash[46934]: audit 2022-04-22T23:25:31.913824+0000 mon.a (mon.0) 16872 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:33.035Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:33 smithi121 bash[36902]: cluster 2022-04-22T23:25:33.364797+0000 mgr.y (mgr.34107) 12246 : cluster [DBG] pgmap v12723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:34.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:33 smithi121 bash[36902]: audit 2022-04-22T23:25:33.550586+0000 mon.a (mon.0) 16873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:33 smithi115 bash[46934]: cluster 2022-04-22T23:25:33.364797+0000 mgr.y (mgr.34107) 12246 : cluster [DBG] pgmap v12723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:34.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:33 smithi115 bash[46934]: audit 2022-04-22T23:25:33.550586+0000 mon.a (mon.0) 16873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:33 smithi115 bash[45495]: cluster 2022-04-22T23:25:33.364797+0000 mgr.y (mgr.34107) 12246 : cluster [DBG] pgmap v12723: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:34.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:33 smithi115 bash[45495]: audit 2022-04-22T23:25:33.550586+0000 mon.a (mon.0) 16873 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:34.893Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:35 smithi121 bash[36902]: audit 2022-04-22T23:25:35.294010+0000 mon.a (mon.0) 16874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:35 smithi115 bash[45495]: audit 2022-04-22T23:25:35.294010+0000 mon.a (mon.0) 16874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:35 smithi115 bash[46934]: audit 2022-04-22T23:25:35.294010+0000 mon.a (mon.0) 16874 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:36 smithi121 bash[36902]: cluster 2022-04-22T23:25:35.365780+0000 mgr.y (mgr.34107) 12247 : cluster [DBG] pgmap v12724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:36 smithi115 bash[45495]: cluster 2022-04-22T23:25:35.365780+0000 mgr.y (mgr.34107) 12247 : cluster [DBG] pgmap v12724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:36 smithi115 bash[46934]: cluster 2022-04-22T23:25:35.365780+0000 mgr.y (mgr.34107) 12247 : cluster [DBG] pgmap v12724: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:37 smithi121 bash[36902]: audit 2022-04-22T23:25:36.916941+0000 mon.a (mon.0) 16875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:37.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:37 smithi115 bash[45495]: audit 2022-04-22T23:25:36.916941+0000 mon.a (mon.0) 16875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:37 smithi115 bash[46934]: audit 2022-04-22T23:25:36.916941+0000 mon.a (mon.0) 16875 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:38.441 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:25:38.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:38 smithi121 bash[36902]: cluster 2022-04-22T23:25:37.366794+0000 mgr.y (mgr.34107) 12248 : cluster [DBG] pgmap v12725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:38.780 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:25:38.780 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:25:38.780 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 8m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 7m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 7m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 7m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 7m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:25:38.781 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 7m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 7m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 7m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 7m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:25:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:25:38.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:38 smithi115 bash[45495]: cluster 2022-04-22T23:25:37.366794+0000 mgr.y (mgr.34107) 12248 : cluster [DBG] pgmap v12725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:38 smithi115 bash[46934]: cluster 2022-04-22T23:25:37.366794+0000 mgr.y (mgr.34107) 12248 : cluster [DBG] pgmap v12725: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:39.187 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:25:39.187 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:25:39.187 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:25:39.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:39.187 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:25:39.188 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:25:39.189 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:25:39.189 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:25:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:39 smithi121 bash[36902]: audit 2022-04-22T23:25:38.553168+0000 mon.a (mon.0) 16876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:39 smithi121 bash[36902]: audit 2022-04-22T23:25:39.183562+0000 mon.a (mon.0) 16877 : audit [DBG] from='client.? 172.21.15.115:0/1034886569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:39 smithi115 bash[45495]: audit 2022-04-22T23:25:38.553168+0000 mon.a (mon.0) 16876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:39 smithi115 bash[45495]: audit 2022-04-22T23:25:39.183562+0000 mon.a (mon.0) 16877 : audit [DBG] from='client.? 172.21.15.115:0/1034886569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:39 smithi115 bash[46934]: audit 2022-04-22T23:25:38.553168+0000 mon.a (mon.0) 16876 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:39 smithi115 bash[46934]: audit 2022-04-22T23:25:39.183562+0000 mon.a (mon.0) 16877 : audit [DBG] from='client.? 172.21.15.115:0/1034886569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:25:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:40 smithi121 bash[36902]: audit 2022-04-22T23:25:38.422983+0000 mgr.y (mgr.34107) 12249 : audit [DBG] from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:40 smithi121 bash[36902]: audit 2022-04-22T23:25:38.762962+0000 mgr.y (mgr.34107) 12250 : audit [DBG] from='client.57961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:40 smithi121 bash[36902]: cluster 2022-04-22T23:25:39.367903+0000 mgr.y (mgr.34107) 12251 : cluster [DBG] pgmap v12726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:40 smithi121 bash[36902]: audit 2022-04-22T23:25:40.297803+0000 mon.a (mon.0) 16878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[45495]: audit 2022-04-22T23:25:38.422983+0000 mgr.y (mgr.34107) 12249 : audit [DBG] from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[45495]: audit 2022-04-22T23:25:38.762962+0000 mgr.y (mgr.34107) 12250 : audit [DBG] from='client.57961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[45495]: cluster 2022-04-22T23:25:39.367903+0000 mgr.y (mgr.34107) 12251 : cluster [DBG] pgmap v12726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[45495]: audit 2022-04-22T23:25:40.297803+0000 mon.a (mon.0) 16878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[46934]: audit 2022-04-22T23:25:38.422983+0000 mgr.y (mgr.34107) 12249 : audit [DBG] from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[46934]: audit 2022-04-22T23:25:38.762962+0000 mgr.y (mgr.34107) 12250 : audit [DBG] from='client.57961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[46934]: cluster 2022-04-22T23:25:39.367903+0000 mgr.y (mgr.34107) 12251 : cluster [DBG] pgmap v12726: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:40.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:40 smithi115 bash[46934]: audit 2022-04-22T23:25:40.297803+0000 mon.a (mon.0) 16878 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:41 smithi121 bash[36902]: cluster 2022-04-22T23:25:41.369028+0000 mgr.y (mgr.34107) 12252 : cluster [DBG] pgmap v12727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:41 smithi115 bash[45495]: cluster 2022-04-22T23:25:41.369028+0000 mgr.y (mgr.34107) 12252 : cluster [DBG] pgmap v12727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:41 smithi115 bash[46934]: cluster 2022-04-22T23:25:41.369028+0000 mgr.y (mgr.34107) 12252 : cluster [DBG] pgmap v12727: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:42 smithi121 bash[36902]: audit 2022-04-22T23:25:41.920062+0000 mon.a (mon.0) 16879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:42 smithi115 bash[45495]: audit 2022-04-22T23:25:41.920062+0000 mon.a (mon.0) 16879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:42 smithi115 bash[46934]: audit 2022-04-22T23:25:41.920062+0000 mon.a (mon.0) 16879 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:43.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:43 smithi121 bash[36902]: cluster 2022-04-22T23:25:43.370155+0000 mgr.y (mgr.34107) 12253 : cluster [DBG] pgmap v12728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:44.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:43 smithi121 bash[36902]: audit 2022-04-22T23:25:43.555877+0000 mon.a (mon.0) 16880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:43 smithi115 bash[46934]: cluster 2022-04-22T23:25:43.370155+0000 mgr.y (mgr.34107) 12253 : cluster [DBG] pgmap v12728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:44.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:43 smithi115 bash[46934]: audit 2022-04-22T23:25:43.555877+0000 mon.a (mon.0) 16880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:43 smithi115 bash[45495]: cluster 2022-04-22T23:25:43.370155+0000 mgr.y (mgr.34107) 12253 : cluster [DBG] pgmap v12728: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:44.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:43 smithi115 bash[45495]: audit 2022-04-22T23:25:43.555877+0000 mon.a (mon.0) 16880 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:44.894Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:45 smithi121 bash[36902]: audit 2022-04-22T23:25:45.301473+0000 mon.a (mon.0) 16881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:45.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:45 smithi115 bash[45495]: audit 2022-04-22T23:25:45.301473+0000 mon.a (mon.0) 16881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:45.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:45 smithi115 bash[46934]: audit 2022-04-22T23:25:45.301473+0000 mon.a (mon.0) 16881 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:46 smithi121 bash[36902]: cluster 2022-04-22T23:25:45.371150+0000 mgr.y (mgr.34107) 12254 : cluster [DBG] pgmap v12729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:46.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:46 smithi115 bash[45495]: cluster 2022-04-22T23:25:45.371150+0000 mgr.y (mgr.34107) 12254 : cluster [DBG] pgmap v12729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:46 smithi115 bash[46934]: cluster 2022-04-22T23:25:45.371150+0000 mgr.y (mgr.34107) 12254 : cluster [DBG] pgmap v12729: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:47 smithi121 bash[36902]: audit 2022-04-22T23:25:46.923269+0000 mon.a (mon.0) 16882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:47 smithi115 bash[45495]: audit 2022-04-22T23:25:46.923269+0000 mon.a (mon.0) 16882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:47 smithi115 bash[46934]: audit 2022-04-22T23:25:46.923269+0000 mon.a (mon.0) 16882 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:48.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:48 smithi121 bash[36902]: cluster 2022-04-22T23:25:47.372150+0000 mgr.y (mgr.34107) 12255 : cluster [DBG] pgmap v12730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:48.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:48 smithi115 bash[45495]: cluster 2022-04-22T23:25:47.372150+0000 mgr.y (mgr.34107) 12255 : cluster [DBG] pgmap v12730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:48.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:48 smithi115 bash[46934]: cluster 2022-04-22T23:25:47.372150+0000 mgr.y (mgr.34107) 12255 : cluster [DBG] pgmap v12730: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:49 smithi121 bash[36902]: audit 2022-04-22T23:25:48.558898+0000 mon.a (mon.0) 16883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:49 smithi115 bash[45495]: audit 2022-04-22T23:25:48.558898+0000 mon.a (mon.0) 16883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:49 smithi115 bash[46934]: audit 2022-04-22T23:25:48.558898+0000 mon.a (mon.0) 16883 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:50 smithi121 bash[36902]: cluster 2022-04-22T23:25:49.373204+0000 mgr.y (mgr.34107) 12256 : cluster [DBG] pgmap v12731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:50.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:50 smithi121 bash[36902]: audit 2022-04-22T23:25:50.304932+0000 mon.a (mon.0) 16884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:50 smithi115 bash[45495]: cluster 2022-04-22T23:25:49.373204+0000 mgr.y (mgr.34107) 12256 : cluster [DBG] pgmap v12731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:50.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:50 smithi115 bash[45495]: audit 2022-04-22T23:25:50.304932+0000 mon.a (mon.0) 16884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:50 smithi115 bash[46934]: cluster 2022-04-22T23:25:49.373204+0000 mgr.y (mgr.34107) 12256 : cluster [DBG] pgmap v12731: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:50.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:50 smithi115 bash[46934]: audit 2022-04-22T23:25:50.304932+0000 mon.a (mon.0) 16884 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:52 smithi121 bash[36902]: cluster 2022-04-22T23:25:51.374267+0000 mgr.y (mgr.34107) 12257 : cluster [DBG] pgmap v12732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:52 smithi121 bash[36902]: audit 2022-04-22T23:25:51.926304+0000 mon.a (mon.0) 16885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:52 smithi115 bash[46934]: cluster 2022-04-22T23:25:51.374267+0000 mgr.y (mgr.34107) 12257 : cluster [DBG] pgmap v12732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:52 smithi115 bash[46934]: audit 2022-04-22T23:25:51.926304+0000 mon.a (mon.0) 16885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:52 smithi115 bash[45495]: cluster 2022-04-22T23:25:51.374267+0000 mgr.y (mgr.34107) 12257 : cluster [DBG] pgmap v12732: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:52.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:52 smithi115 bash[45495]: audit 2022-04-22T23:25:51.926304+0000 mon.a (mon.0) 16885 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:53.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:53.035Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:53 smithi121 bash[36902]: audit 2022-04-22T23:25:53.561835+0000 mon.a (mon.0) 16886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:53 smithi115 bash[45495]: audit 2022-04-22T23:25:53.561835+0000 mon.a (mon.0) 16886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:53 smithi115 bash[46934]: audit 2022-04-22T23:25:53.561835+0000 mon.a (mon.0) 16886 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:54 smithi121 bash[36902]: cluster 2022-04-22T23:25:53.375365+0000 mgr.y (mgr.34107) 12258 : cluster [DBG] pgmap v12733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:55.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:54 smithi115 bash[45495]: cluster 2022-04-22T23:25:53.375365+0000 mgr.y (mgr.34107) 12258 : cluster [DBG] pgmap v12733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:25:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:25:54.895Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:25:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:54 smithi115 bash[46934]: cluster 2022-04-22T23:25:53.375365+0000 mgr.y (mgr.34107) 12258 : cluster [DBG] pgmap v12733: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:56.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:55 smithi121 bash[36902]: audit 2022-04-22T23:25:55.308642+0000 mon.a (mon.0) 16887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:55 smithi115 bash[45495]: audit 2022-04-22T23:25:55.308642+0000 mon.a (mon.0) 16887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:55 smithi115 bash[46934]: audit 2022-04-22T23:25:55.308642+0000 mon.a (mon.0) 16887 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:25:57.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:56 smithi121 bash[36902]: cluster 2022-04-22T23:25:55.376376+0000 mgr.y (mgr.34107) 12259 : cluster [DBG] pgmap v12734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:56 smithi115 bash[45495]: cluster 2022-04-22T23:25:55.376376+0000 mgr.y (mgr.34107) 12259 : cluster [DBG] pgmap v12734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:56 smithi115 bash[46934]: cluster 2022-04-22T23:25:55.376376+0000 mgr.y (mgr.34107) 12259 : cluster [DBG] pgmap v12734: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:58.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:57 smithi121 bash[36902]: audit 2022-04-22T23:25:56.929457+0000 mon.a (mon.0) 16888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:58.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:57 smithi115 bash[45495]: audit 2022-04-22T23:25:56.929457+0000 mon.a (mon.0) 16888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:57 smithi115 bash[46934]: audit 2022-04-22T23:25:56.929457+0000 mon.a (mon.0) 16888 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:25:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:58 smithi121 bash[36902]: cluster 2022-04-22T23:25:57.377464+0000 mgr.y (mgr.34107) 12260 : cluster [DBG] pgmap v12735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:59.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:58 smithi121 bash[36902]: audit 2022-04-22T23:25:58.565059+0000 mon.a (mon.0) 16889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:59.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:58 smithi115 bash[46934]: cluster 2022-04-22T23:25:57.377464+0000 mgr.y (mgr.34107) 12260 : cluster [DBG] pgmap v12735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:58 smithi115 bash[46934]: audit 2022-04-22T23:25:58.565059+0000 mon.a (mon.0) 16889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:25:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:58 smithi115 bash[45495]: cluster 2022-04-22T23:25:57.377464+0000 mgr.y (mgr.34107) 12260 : cluster [DBG] pgmap v12735: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:25:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:58 smithi115 bash[45495]: audit 2022-04-22T23:25:58.565059+0000 mon.a (mon.0) 16889 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:59 smithi121 bash[36902]: audit 2022-04-22T23:25:58.811770+0000 mon.a (mon.0) 16890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:26:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:59 smithi121 bash[36902]: audit 2022-04-22T23:25:58.814688+0000 mon.a (mon.0) 16891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:26:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:59 smithi121 bash[36902]: audit 2022-04-22T23:25:58.816523+0000 mon.a (mon.0) 16892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:26:00.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:25:59 smithi121 bash[36902]: audit 2022-04-22T23:25:59.187957+0000 mon.a (mon.0) 16893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[45495]: audit 2022-04-22T23:25:58.811770+0000 mon.a (mon.0) 16890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[45495]: audit 2022-04-22T23:25:58.814688+0000 mon.a (mon.0) 16891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[45495]: audit 2022-04-22T23:25:58.816523+0000 mon.a (mon.0) 16892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[45495]: audit 2022-04-22T23:25:59.187957+0000 mon.a (mon.0) 16893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[46934]: audit 2022-04-22T23:25:58.811770+0000 mon.a (mon.0) 16890 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:26:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[46934]: audit 2022-04-22T23:25:58.814688+0000 mon.a (mon.0) 16891 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:26:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[46934]: audit 2022-04-22T23:25:58.816523+0000 mon.a (mon.0) 16892 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:26:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:25:59 smithi115 bash[46934]: audit 2022-04-22T23:25:59.187957+0000 mon.a (mon.0) 16893 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:26:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:01 smithi121 bash[36902]: cluster 2022-04-22T23:25:59.378534+0000 mgr.y (mgr.34107) 12261 : cluster [DBG] pgmap v12736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:01 smithi121 bash[36902]: audit 2022-04-22T23:26:00.312526+0000 mon.a (mon.0) 16894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:01 smithi115 bash[45495]: cluster 2022-04-22T23:25:59.378534+0000 mgr.y (mgr.34107) 12261 : cluster [DBG] pgmap v12736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:01 smithi115 bash[45495]: audit 2022-04-22T23:26:00.312526+0000 mon.a (mon.0) 16894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:01 smithi115 bash[46934]: cluster 2022-04-22T23:25:59.378534+0000 mgr.y (mgr.34107) 12261 : cluster [DBG] pgmap v12736: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:01 smithi115 bash[46934]: audit 2022-04-22T23:26:00.312526+0000 mon.a (mon.0) 16894 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:02 smithi121 bash[36902]: audit 2022-04-22T23:26:01.932245+0000 mon.a (mon.0) 16895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:02 smithi115 bash[45495]: audit 2022-04-22T23:26:01.932245+0000 mon.a (mon.0) 16895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:02 smithi115 bash[46934]: audit 2022-04-22T23:26:01.932245+0000 mon.a (mon.0) 16895 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:03 smithi115 bash[45495]: cluster 2022-04-22T23:26:01.379664+0000 mgr.y (mgr.34107) 12262 : cluster [DBG] pgmap v12737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:03.036Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:03 smithi115 bash[46934]: cluster 2022-04-22T23:26:01.379664+0000 mgr.y (mgr.34107) 12262 : cluster [DBG] pgmap v12737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:03 smithi121 bash[36902]: cluster 2022-04-22T23:26:01.379664+0000 mgr.y (mgr.34107) 12262 : cluster [DBG] pgmap v12737: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:04 smithi121 bash[36902]: audit 2022-04-22T23:26:03.567978+0000 mon.a (mon.0) 16896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:04.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:04 smithi115 bash[45495]: audit 2022-04-22T23:26:03.567978+0000 mon.a (mon.0) 16896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:04 smithi115 bash[46934]: audit 2022-04-22T23:26:03.567978+0000 mon.a (mon.0) 16896 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:05.214 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:04.894Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:05.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:05 smithi115 bash[45495]: cluster 2022-04-22T23:26:03.380802+0000 mgr.y (mgr.34107) 12263 : cluster [DBG] pgmap v12738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:05.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:05 smithi115 bash[46934]: cluster 2022-04-22T23:26:03.380802+0000 mgr.y (mgr.34107) 12263 : cluster [DBG] pgmap v12738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:05 smithi121 bash[36902]: cluster 2022-04-22T23:26:03.380802+0000 mgr.y (mgr.34107) 12263 : cluster [DBG] pgmap v12738: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:06 smithi121 bash[36902]: audit 2022-04-22T23:26:05.316289+0000 mon.a (mon.0) 16897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:06 smithi115 bash[45495]: audit 2022-04-22T23:26:05.316289+0000 mon.a (mon.0) 16897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:06 smithi115 bash[46934]: audit 2022-04-22T23:26:05.316289+0000 mon.a (mon.0) 16897 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:07 smithi121 bash[36902]: cluster 2022-04-22T23:26:05.381816+0000 mgr.y (mgr.34107) 12264 : cluster [DBG] pgmap v12739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:07 smithi121 bash[36902]: audit 2022-04-22T23:26:06.935531+0000 mon.a (mon.0) 16898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:07 smithi115 bash[45495]: cluster 2022-04-22T23:26:05.381816+0000 mgr.y (mgr.34107) 12264 : cluster [DBG] pgmap v12739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:07 smithi115 bash[45495]: audit 2022-04-22T23:26:06.935531+0000 mon.a (mon.0) 16898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:07 smithi115 bash[46934]: cluster 2022-04-22T23:26:05.381816+0000 mgr.y (mgr.34107) 12264 : cluster [DBG] pgmap v12739: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:07 smithi115 bash[46934]: audit 2022-04-22T23:26:06.935531+0000 mon.a (mon.0) 16898 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:09 smithi121 bash[36902]: cluster 2022-04-22T23:26:07.382943+0000 mgr.y (mgr.34107) 12265 : cluster [DBG] pgmap v12740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:09 smithi121 bash[36902]: audit 2022-04-22T23:26:08.571114+0000 mon.a (mon.0) 16899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:09 smithi115 bash[45495]: cluster 2022-04-22T23:26:07.382943+0000 mgr.y (mgr.34107) 12265 : cluster [DBG] pgmap v12740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:09 smithi115 bash[45495]: audit 2022-04-22T23:26:08.571114+0000 mon.a (mon.0) 16899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:09 smithi115 bash[46934]: cluster 2022-04-22T23:26:07.382943+0000 mgr.y (mgr.34107) 12265 : cluster [DBG] pgmap v12740: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:09 smithi115 bash[46934]: audit 2022-04-22T23:26:08.571114+0000 mon.a (mon.0) 16899 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:09.564 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:26:09.911 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:26:09.911 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:26:09.911 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 9m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:26:09.911 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:26:09.911 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:26:09.912 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:26:09.913 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:26:10.315 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:26:10.399 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:26:10.400 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:26:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:11 smithi121 bash[36902]: cluster 2022-04-22T23:26:09.384022+0000 mgr.y (mgr.34107) 12266 : cluster [DBG] pgmap v12741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:11 smithi121 bash[36902]: audit 2022-04-22T23:26:09.544900+0000 mgr.y (mgr.34107) 12267 : audit [DBG] from='client.57973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:11 smithi121 bash[36902]: audit 2022-04-22T23:26:09.886695+0000 mgr.y (mgr.34107) 12268 : audit [DBG] from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:11 smithi121 bash[36902]: audit 2022-04-22T23:26:10.311667+0000 mon.a (mon.0) 16900 : audit [DBG] from='client.? 172.21.15.115:0/26456860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:11 smithi121 bash[36902]: audit 2022-04-22T23:26:10.319651+0000 mon.a (mon.0) 16901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[45495]: cluster 2022-04-22T23:26:09.384022+0000 mgr.y (mgr.34107) 12266 : cluster [DBG] pgmap v12741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:11.543 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[45495]: audit 2022-04-22T23:26:09.544900+0000 mgr.y (mgr.34107) 12267 : audit [DBG] from='client.57973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[45495]: audit 2022-04-22T23:26:09.886695+0000 mgr.y (mgr.34107) 12268 : audit [DBG] from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[45495]: audit 2022-04-22T23:26:10.311667+0000 mon.a (mon.0) 16900 : audit [DBG] from='client.? 172.21.15.115:0/26456860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[45495]: audit 2022-04-22T23:26:10.319651+0000 mon.a (mon.0) 16901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[46934]: cluster 2022-04-22T23:26:09.384022+0000 mgr.y (mgr.34107) 12266 : cluster [DBG] pgmap v12741: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[46934]: audit 2022-04-22T23:26:09.544900+0000 mgr.y (mgr.34107) 12267 : audit [DBG] from='client.57973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[46934]: audit 2022-04-22T23:26:09.886695+0000 mgr.y (mgr.34107) 12268 : audit [DBG] from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[46934]: audit 2022-04-22T23:26:10.311667+0000 mon.a (mon.0) 16900 : audit [DBG] from='client.? 172.21.15.115:0/26456860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:11 smithi115 bash[46934]: audit 2022-04-22T23:26:10.319651+0000 mon.a (mon.0) 16901 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:12 smithi121 bash[36902]: audit 2022-04-22T23:26:11.938820+0000 mon.a (mon.0) 16902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:12 smithi115 bash[45495]: audit 2022-04-22T23:26:11.938820+0000 mon.a (mon.0) 16902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:12 smithi115 bash[46934]: audit 2022-04-22T23:26:11.938820+0000 mon.a (mon.0) 16902 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:13 smithi121 bash[36902]: cluster 2022-04-22T23:26:11.385184+0000 mgr.y (mgr.34107) 12269 : cluster [DBG] pgmap v12742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:13 smithi115 bash[45495]: cluster 2022-04-22T23:26:11.385184+0000 mgr.y (mgr.34107) 12269 : cluster [DBG] pgmap v12742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:13.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:13.037Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:13 smithi115 bash[46934]: cluster 2022-04-22T23:26:11.385184+0000 mgr.y (mgr.34107) 12269 : cluster [DBG] pgmap v12742: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:14 smithi121 bash[36902]: audit 2022-04-22T23:26:13.574561+0000 mon.a (mon.0) 16903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:14 smithi115 bash[45495]: audit 2022-04-22T23:26:13.574561+0000 mon.a (mon.0) 16903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:14 smithi115 bash[46934]: audit 2022-04-22T23:26:13.574561+0000 mon.a (mon.0) 16903 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:15.256 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:14.895Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:15 smithi121 bash[36902]: cluster 2022-04-22T23:26:13.386324+0000 mgr.y (mgr.34107) 12270 : cluster [DBG] pgmap v12743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:15 smithi115 bash[45495]: cluster 2022-04-22T23:26:13.386324+0000 mgr.y (mgr.34107) 12270 : cluster [DBG] pgmap v12743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:15 smithi115 bash[46934]: cluster 2022-04-22T23:26:13.386324+0000 mgr.y (mgr.34107) 12270 : cluster [DBG] pgmap v12743: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:16 smithi115 bash[45495]: audit 2022-04-22T23:26:15.322914+0000 mon.a (mon.0) 16904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:16 smithi115 bash[46934]: audit 2022-04-22T23:26:15.322914+0000 mon.a (mon.0) 16904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:16 smithi121 bash[36902]: audit 2022-04-22T23:26:15.322914+0000 mon.a (mon.0) 16904 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:17 smithi115 bash[46934]: cluster 2022-04-22T23:26:15.387433+0000 mgr.y (mgr.34107) 12271 : cluster [DBG] pgmap v12744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:17 smithi115 bash[46934]: audit 2022-04-22T23:26:16.941769+0000 mon.a (mon.0) 16905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:17 smithi115 bash[45495]: cluster 2022-04-22T23:26:15.387433+0000 mgr.y (mgr.34107) 12271 : cluster [DBG] pgmap v12744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:17 smithi115 bash[45495]: audit 2022-04-22T23:26:16.941769+0000 mon.a (mon.0) 16905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:17 smithi121 bash[36902]: cluster 2022-04-22T23:26:15.387433+0000 mgr.y (mgr.34107) 12271 : cluster [DBG] pgmap v12744: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:17 smithi121 bash[36902]: audit 2022-04-22T23:26:16.941769+0000 mon.a (mon.0) 16905 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:18 smithi115 bash[45495]: audit 2022-04-22T23:26:17.421744+0000 mon.a (mon.0) 16906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:26:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:18 smithi115 bash[45495]: audit 2022-04-22T23:26:17.422605+0000 mon.a (mon.0) 16907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:26:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:18 smithi115 bash[46934]: audit 2022-04-22T23:26:17.421744+0000 mon.a (mon.0) 16906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:26:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:18 smithi115 bash[46934]: audit 2022-04-22T23:26:17.422605+0000 mon.a (mon.0) 16907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:26:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:18 smithi121 bash[36902]: audit 2022-04-22T23:26:17.421744+0000 mon.a (mon.0) 16906 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:26:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:18 smithi121 bash[36902]: audit 2022-04-22T23:26:17.422605+0000 mon.a (mon.0) 16907 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:26:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:19 smithi115 bash[45495]: cluster 2022-04-22T23:26:17.388500+0000 mgr.y (mgr.34107) 12272 : cluster [DBG] pgmap v12745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:19 smithi115 bash[45495]: audit 2022-04-22T23:26:18.577939+0000 mon.a (mon.0) 16908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:19 smithi115 bash[46934]: cluster 2022-04-22T23:26:17.388500+0000 mgr.y (mgr.34107) 12272 : cluster [DBG] pgmap v12745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:19.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:19 smithi115 bash[46934]: audit 2022-04-22T23:26:18.577939+0000 mon.a (mon.0) 16908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:19 smithi121 bash[36902]: cluster 2022-04-22T23:26:17.388500+0000 mgr.y (mgr.34107) 12272 : cluster [DBG] pgmap v12745: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:19 smithi121 bash[36902]: audit 2022-04-22T23:26:18.577939+0000 mon.a (mon.0) 16908 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:21 smithi115 bash[45495]: cluster 2022-04-22T23:26:19.389578+0000 mgr.y (mgr.34107) 12273 : cluster [DBG] pgmap v12746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:21 smithi115 bash[45495]: audit 2022-04-22T23:26:20.326827+0000 mon.a (mon.0) 16909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:21 smithi115 bash[46934]: cluster 2022-04-22T23:26:19.389578+0000 mgr.y (mgr.34107) 12273 : cluster [DBG] pgmap v12746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:21 smithi115 bash[46934]: audit 2022-04-22T23:26:20.326827+0000 mon.a (mon.0) 16909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:21 smithi121 bash[36902]: cluster 2022-04-22T23:26:19.389578+0000 mgr.y (mgr.34107) 12273 : cluster [DBG] pgmap v12746: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:21 smithi121 bash[36902]: audit 2022-04-22T23:26:20.326827+0000 mon.a (mon.0) 16909 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:22 smithi121 bash[36902]: audit 2022-04-22T23:26:21.944843+0000 mon.a (mon.0) 16910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:22 smithi115 bash[45495]: audit 2022-04-22T23:26:21.944843+0000 mon.a (mon.0) 16910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:22 smithi115 bash[46934]: audit 2022-04-22T23:26:21.944843+0000 mon.a (mon.0) 16910 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:23.295 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:23.038Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:23 smithi121 bash[36902]: cluster 2022-04-22T23:26:21.390688+0000 mgr.y (mgr.34107) 12274 : cluster [DBG] pgmap v12747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:23 smithi115 bash[45495]: cluster 2022-04-22T23:26:21.390688+0000 mgr.y (mgr.34107) 12274 : cluster [DBG] pgmap v12747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:23 smithi115 bash[46934]: cluster 2022-04-22T23:26:21.390688+0000 mgr.y (mgr.34107) 12274 : cluster [DBG] pgmap v12747: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:24 smithi121 bash[36902]: audit 2022-04-22T23:26:23.580974+0000 mon.a (mon.0) 16911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:24 smithi115 bash[45495]: audit 2022-04-22T23:26:23.580974+0000 mon.a (mon.0) 16911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:24 smithi115 bash[46934]: audit 2022-04-22T23:26:23.580974+0000 mon.a (mon.0) 16911 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:25.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:24.897Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:25 smithi121 bash[36902]: cluster 2022-04-22T23:26:23.391842+0000 mgr.y (mgr.34107) 12275 : cluster [DBG] pgmap v12748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:25 smithi115 bash[45495]: cluster 2022-04-22T23:26:23.391842+0000 mgr.y (mgr.34107) 12275 : cluster [DBG] pgmap v12748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:25 smithi115 bash[46934]: cluster 2022-04-22T23:26:23.391842+0000 mgr.y (mgr.34107) 12275 : cluster [DBG] pgmap v12748: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:26 smithi121 bash[36902]: audit 2022-04-22T23:26:25.330639+0000 mon.a (mon.0) 16912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:26 smithi115 bash[45495]: audit 2022-04-22T23:26:25.330639+0000 mon.a (mon.0) 16912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:26 smithi115 bash[46934]: audit 2022-04-22T23:26:25.330639+0000 mon.a (mon.0) 16912 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:27 smithi121 bash[36902]: cluster 2022-04-22T23:26:25.392854+0000 mgr.y (mgr.34107) 12276 : cluster [DBG] pgmap v12749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:27 smithi121 bash[36902]: audit 2022-04-22T23:26:26.948223+0000 mon.a (mon.0) 16913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:27 smithi115 bash[45495]: cluster 2022-04-22T23:26:25.392854+0000 mgr.y (mgr.34107) 12276 : cluster [DBG] pgmap v12749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:27 smithi115 bash[45495]: audit 2022-04-22T23:26:26.948223+0000 mon.a (mon.0) 16913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:27 smithi115 bash[46934]: cluster 2022-04-22T23:26:25.392854+0000 mgr.y (mgr.34107) 12276 : cluster [DBG] pgmap v12749: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:27 smithi115 bash[46934]: audit 2022-04-22T23:26:26.948223+0000 mon.a (mon.0) 16913 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:29 smithi121 bash[36902]: cluster 2022-04-22T23:26:27.393940+0000 mgr.y (mgr.34107) 12277 : cluster [DBG] pgmap v12750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:29 smithi121 bash[36902]: audit 2022-04-22T23:26:28.584247+0000 mon.a (mon.0) 16914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:29 smithi115 bash[45495]: cluster 2022-04-22T23:26:27.393940+0000 mgr.y (mgr.34107) 12277 : cluster [DBG] pgmap v12750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:29 smithi115 bash[45495]: audit 2022-04-22T23:26:28.584247+0000 mon.a (mon.0) 16914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:29 smithi115 bash[46934]: cluster 2022-04-22T23:26:27.393940+0000 mgr.y (mgr.34107) 12277 : cluster [DBG] pgmap v12750: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:29.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:29 smithi115 bash[46934]: audit 2022-04-22T23:26:28.584247+0000 mon.a (mon.0) 16914 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:31 smithi121 bash[36902]: cluster 2022-04-22T23:26:29.395090+0000 mgr.y (mgr.34107) 12278 : cluster [DBG] pgmap v12751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:31 smithi121 bash[36902]: audit 2022-04-22T23:26:30.334361+0000 mon.a (mon.0) 16915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:31 smithi115 bash[45495]: cluster 2022-04-22T23:26:29.395090+0000 mgr.y (mgr.34107) 12278 : cluster [DBG] pgmap v12751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:31 smithi115 bash[45495]: audit 2022-04-22T23:26:30.334361+0000 mon.a (mon.0) 16915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:31 smithi115 bash[46934]: cluster 2022-04-22T23:26:29.395090+0000 mgr.y (mgr.34107) 12278 : cluster [DBG] pgmap v12751: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:31.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:31 smithi115 bash[46934]: audit 2022-04-22T23:26:30.334361+0000 mon.a (mon.0) 16915 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:32 smithi121 bash[36902]: audit 2022-04-22T23:26:31.951536+0000 mon.a (mon.0) 16916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:32 smithi115 bash[45495]: audit 2022-04-22T23:26:31.951536+0000 mon.a (mon.0) 16916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:32 smithi115 bash[46934]: audit 2022-04-22T23:26:31.951536+0000 mon.a (mon.0) 16916 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:33.338 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:33.039Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:33.340 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:33 smithi115 bash[45495]: cluster 2022-04-22T23:26:31.396149+0000 mgr.y (mgr.34107) 12279 : cluster [DBG] pgmap v12752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:33.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:33 smithi115 bash[46934]: cluster 2022-04-22T23:26:31.396149+0000 mgr.y (mgr.34107) 12279 : cluster [DBG] pgmap v12752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:33 smithi121 bash[36902]: cluster 2022-04-22T23:26:31.396149+0000 mgr.y (mgr.34107) 12279 : cluster [DBG] pgmap v12752: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:34 smithi121 bash[36902]: audit 2022-04-22T23:26:33.587630+0000 mon.a (mon.0) 16917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:34 smithi115 bash[45495]: audit 2022-04-22T23:26:33.587630+0000 mon.a (mon.0) 16917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:34 smithi115 bash[46934]: audit 2022-04-22T23:26:33.587630+0000 mon.a (mon.0) 16917 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:35.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:34.897Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:35 smithi121 bash[36902]: cluster 2022-04-22T23:26:33.397243+0000 mgr.y (mgr.34107) 12280 : cluster [DBG] pgmap v12753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:35 smithi121 bash[36902]: audit 2022-04-22T23:26:35.338277+0000 mon.a (mon.0) 16918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:35 smithi115 bash[46934]: cluster 2022-04-22T23:26:33.397243+0000 mgr.y (mgr.34107) 12280 : cluster [DBG] pgmap v12753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:35 smithi115 bash[46934]: audit 2022-04-22T23:26:35.338277+0000 mon.a (mon.0) 16918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:35 smithi115 bash[45495]: cluster 2022-04-22T23:26:33.397243+0000 mgr.y (mgr.34107) 12280 : cluster [DBG] pgmap v12753: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:35.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:35 smithi115 bash[45495]: audit 2022-04-22T23:26:35.338277+0000 mon.a (mon.0) 16918 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:37 smithi121 bash[36902]: cluster 2022-04-22T23:26:35.398439+0000 mgr.y (mgr.34107) 12281 : cluster [DBG] pgmap v12754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:37 smithi121 bash[36902]: audit 2022-04-22T23:26:36.954783+0000 mon.a (mon.0) 16919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:37 smithi115 bash[46934]: cluster 2022-04-22T23:26:35.398439+0000 mgr.y (mgr.34107) 12281 : cluster [DBG] pgmap v12754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:37 smithi115 bash[46934]: audit 2022-04-22T23:26:36.954783+0000 mon.a (mon.0) 16919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:37 smithi115 bash[45495]: cluster 2022-04-22T23:26:35.398439+0000 mgr.y (mgr.34107) 12281 : cluster [DBG] pgmap v12754: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:37 smithi115 bash[45495]: audit 2022-04-22T23:26:36.954783+0000 mon.a (mon.0) 16919 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:39 smithi121 bash[36902]: cluster 2022-04-22T23:26:37.399587+0000 mgr.y (mgr.34107) 12282 : cluster [DBG] pgmap v12755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:39 smithi121 bash[36902]: audit 2022-04-22T23:26:38.590692+0000 mon.a (mon.0) 16920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:39 smithi115 bash[45495]: cluster 2022-04-22T23:26:37.399587+0000 mgr.y (mgr.34107) 12282 : cluster [DBG] pgmap v12755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:39 smithi115 bash[45495]: audit 2022-04-22T23:26:38.590692+0000 mon.a (mon.0) 16920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:39 smithi115 bash[46934]: cluster 2022-04-22T23:26:37.399587+0000 mgr.y (mgr.34107) 12282 : cluster [DBG] pgmap v12755: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:39 smithi115 bash[46934]: audit 2022-04-22T23:26:38.590692+0000 mon.a (mon.0) 16920 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:40.695 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:26:40.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:40 smithi121 bash[36902]: audit 2022-04-22T23:26:40.341863+0000 mon.a (mon.0) 16921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:40.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:40 smithi115 bash[46934]: audit 2022-04-22T23:26:40.341863+0000 mon.a (mon.0) 16921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:40.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:40 smithi115 bash[45495]: audit 2022-04-22T23:26:40.341863+0000 mon.a (mon.0) 16921 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:41.040 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:26:41.041 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:26:41.041 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 9m ago 6h 26.3M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:26:41.041 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 8m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:26:41.041 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 9m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:26:41.041 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 9m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (5h) 8m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 8m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 9m ago 6h 314M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 8m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 8m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 9m ago 6h 12.6M - 1dbe0e931976 f4b044642a96 2022-04-22T23:26:41.042 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 8m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 8m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 8m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 8m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 9m ago 6h 590M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 9m ago 6h 723M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 9m ago 6h 509M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:26:41.043 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 9m ago 6h 654M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:26:41.044 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 9m ago 6h 66.6M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:26:41.446 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:26:41.446 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:26:41.446 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:41.447 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:26:41.448 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:26:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:41 smithi121 bash[36902]: cluster 2022-04-22T23:26:39.400592+0000 mgr.y (mgr.34107) 12283 : cluster [DBG] pgmap v12756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:41 smithi115 bash[46934]: cluster 2022-04-22T23:26:39.400592+0000 mgr.y (mgr.34107) 12283 : cluster [DBG] pgmap v12756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:41 smithi115 bash[45495]: cluster 2022-04-22T23:26:39.400592+0000 mgr.y (mgr.34107) 12283 : cluster [DBG] pgmap v12756: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:42 smithi121 bash[36902]: audit 2022-04-22T23:26:40.672446+0000 mgr.y (mgr.34107) 12284 : audit [DBG] from='client.57991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:42 smithi121 bash[36902]: audit 2022-04-22T23:26:41.016983+0000 mgr.y (mgr.34107) 12285 : audit [DBG] from='client.57997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:42 smithi121 bash[36902]: audit 2022-04-22T23:26:41.442457+0000 mon.c (mon.1) 415 : audit [DBG] from='client.? 172.21.15.115:0/3721106876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:42.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:42 smithi121 bash[36902]: audit 2022-04-22T23:26:41.957802+0000 mon.a (mon.0) 16922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[45495]: audit 2022-04-22T23:26:40.672446+0000 mgr.y (mgr.34107) 12284 : audit [DBG] from='client.57991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[45495]: audit 2022-04-22T23:26:41.016983+0000 mgr.y (mgr.34107) 12285 : audit [DBG] from='client.57997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[45495]: audit 2022-04-22T23:26:41.442457+0000 mon.c (mon.1) 415 : audit [DBG] from='client.? 172.21.15.115:0/3721106876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[45495]: audit 2022-04-22T23:26:41.957802+0000 mon.a (mon.0) 16922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[46934]: audit 2022-04-22T23:26:40.672446+0000 mgr.y (mgr.34107) 12284 : audit [DBG] from='client.57991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[46934]: audit 2022-04-22T23:26:41.016983+0000 mgr.y (mgr.34107) 12285 : audit [DBG] from='client.57997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:26:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[46934]: audit 2022-04-22T23:26:41.442457+0000 mon.c (mon.1) 415 : audit [DBG] from='client.? 172.21.15.115:0/3721106876' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:26:42.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:42 smithi115 bash[46934]: audit 2022-04-22T23:26:41.957802+0000 mon.a (mon.0) 16922 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:43.376 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:43.040Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:43 smithi121 bash[36902]: cluster 2022-04-22T23:26:41.401638+0000 mgr.y (mgr.34107) 12286 : cluster [DBG] pgmap v12757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:43 smithi115 bash[45495]: cluster 2022-04-22T23:26:41.401638+0000 mgr.y (mgr.34107) 12286 : cluster [DBG] pgmap v12757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:43 smithi115 bash[46934]: cluster 2022-04-22T23:26:41.401638+0000 mgr.y (mgr.34107) 12286 : cluster [DBG] pgmap v12757: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:44 smithi121 bash[36902]: audit 2022-04-22T23:26:43.593965+0000 mon.a (mon.0) 16923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:44 smithi115 bash[45495]: audit 2022-04-22T23:26:43.593965+0000 mon.a (mon.0) 16923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:44 smithi115 bash[46934]: audit 2022-04-22T23:26:43.593965+0000 mon.a (mon.0) 16923 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:44.898Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:45 smithi121 bash[36902]: cluster 2022-04-22T23:26:43.402793+0000 mgr.y (mgr.34107) 12287 : cluster [DBG] pgmap v12758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:45 smithi121 bash[36902]: audit 2022-04-22T23:26:45.345795+0000 mon.a (mon.0) 16924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:45 smithi115 bash[45495]: cluster 2022-04-22T23:26:43.402793+0000 mgr.y (mgr.34107) 12287 : cluster [DBG] pgmap v12758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:45 smithi115 bash[45495]: audit 2022-04-22T23:26:45.345795+0000 mon.a (mon.0) 16924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:45 smithi115 bash[46934]: cluster 2022-04-22T23:26:43.402793+0000 mgr.y (mgr.34107) 12287 : cluster [DBG] pgmap v12758: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:45 smithi115 bash[46934]: audit 2022-04-22T23:26:45.345795+0000 mon.a (mon.0) 16924 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:47 smithi121 bash[36902]: cluster 2022-04-22T23:26:45.404039+0000 mgr.y (mgr.34107) 12288 : cluster [DBG] pgmap v12759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:47 smithi121 bash[36902]: audit 2022-04-22T23:26:46.960754+0000 mon.a (mon.0) 16925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:47 smithi115 bash[45495]: cluster 2022-04-22T23:26:45.404039+0000 mgr.y (mgr.34107) 12288 : cluster [DBG] pgmap v12759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:47 smithi115 bash[45495]: audit 2022-04-22T23:26:46.960754+0000 mon.a (mon.0) 16925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:47 smithi115 bash[46934]: cluster 2022-04-22T23:26:45.404039+0000 mgr.y (mgr.34107) 12288 : cluster [DBG] pgmap v12759: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:47 smithi115 bash[46934]: audit 2022-04-22T23:26:46.960754+0000 mon.a (mon.0) 16925 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:48 smithi121 bash[36902]: cluster 2022-04-22T23:26:47.405137+0000 mgr.y (mgr.34107) 12289 : cluster [DBG] pgmap v12760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:49.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:48 smithi121 bash[36902]: audit 2022-04-22T23:26:48.597337+0000 mon.a (mon.0) 16926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:48 smithi115 bash[45495]: cluster 2022-04-22T23:26:47.405137+0000 mgr.y (mgr.34107) 12289 : cluster [DBG] pgmap v12760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:49.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:48 smithi115 bash[45495]: audit 2022-04-22T23:26:48.597337+0000 mon.a (mon.0) 16926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:48 smithi115 bash[46934]: cluster 2022-04-22T23:26:47.405137+0000 mgr.y (mgr.34107) 12289 : cluster [DBG] pgmap v12760: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:49.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:48 smithi115 bash[46934]: audit 2022-04-22T23:26:48.597337+0000 mon.a (mon.0) 16926 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:50 smithi121 bash[36902]: cluster 2022-04-22T23:26:49.406272+0000 mgr.y (mgr.34107) 12290 : cluster [DBG] pgmap v12761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:51.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:50 smithi121 bash[36902]: audit 2022-04-22T23:26:50.349652+0000 mon.a (mon.0) 16927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:50 smithi115 bash[45495]: cluster 2022-04-22T23:26:49.406272+0000 mgr.y (mgr.34107) 12290 : cluster [DBG] pgmap v12761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:51.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:50 smithi115 bash[45495]: audit 2022-04-22T23:26:50.349652+0000 mon.a (mon.0) 16927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:50 smithi115 bash[46934]: cluster 2022-04-22T23:26:49.406272+0000 mgr.y (mgr.34107) 12290 : cluster [DBG] pgmap v12761: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:51.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:50 smithi115 bash[46934]: audit 2022-04-22T23:26:50.349652+0000 mon.a (mon.0) 16927 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:52 smithi121 bash[36902]: cluster 2022-04-22T23:26:51.407455+0000 mgr.y (mgr.34107) 12291 : cluster [DBG] pgmap v12762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:53.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:52 smithi121 bash[36902]: audit 2022-04-22T23:26:51.963611+0000 mon.a (mon.0) 16928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:52 smithi115 bash[45495]: cluster 2022-04-22T23:26:51.407455+0000 mgr.y (mgr.34107) 12291 : cluster [DBG] pgmap v12762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:53.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:52 smithi115 bash[45495]: audit 2022-04-22T23:26:51.963611+0000 mon.a (mon.0) 16928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:52 smithi115 bash[46934]: cluster 2022-04-22T23:26:51.407455+0000 mgr.y (mgr.34107) 12291 : cluster [DBG] pgmap v12762: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:53.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:52 smithi115 bash[46934]: audit 2022-04-22T23:26:51.963611+0000 mon.a (mon.0) 16928 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:53.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:53.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:54.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:53 smithi121 bash[36902]: audit 2022-04-22T23:26:53.600579+0000 mon.a (mon.0) 16929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:54.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:53 smithi115 bash[45495]: audit 2022-04-22T23:26:53.600579+0000 mon.a (mon.0) 16929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:54.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:53 smithi115 bash[46934]: audit 2022-04-22T23:26:53.600579+0000 mon.a (mon.0) 16929 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:55.015 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:54 smithi121 bash[36902]: cluster 2022-04-22T23:26:53.408571+0000 mgr.y (mgr.34107) 12292 : cluster [DBG] pgmap v12763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:55.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:54 smithi115 bash[45495]: cluster 2022-04-22T23:26:53.408571+0000 mgr.y (mgr.34107) 12292 : cluster [DBG] pgmap v12763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:55.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:26:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:26:54.899Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:26:55.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:54 smithi115 bash[46934]: cluster 2022-04-22T23:26:53.408571+0000 mgr.y (mgr.34107) 12292 : cluster [DBG] pgmap v12763: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:56.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:55 smithi115 bash[45495]: audit 2022-04-22T23:26:55.353406+0000 mon.a (mon.0) 16930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:56.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:55 smithi115 bash[46934]: audit 2022-04-22T23:26:55.353406+0000 mon.a (mon.0) 16930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:55 smithi121 bash[36902]: audit 2022-04-22T23:26:55.353406+0000 mon.a (mon.0) 16930 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:26:57.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:56 smithi115 bash[45495]: cluster 2022-04-22T23:26:55.409624+0000 mgr.y (mgr.34107) 12293 : cluster [DBG] pgmap v12764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:57.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:56 smithi115 bash[46934]: cluster 2022-04-22T23:26:55.409624+0000 mgr.y (mgr.34107) 12293 : cluster [DBG] pgmap v12764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:56 smithi121 bash[36902]: cluster 2022-04-22T23:26:55.409624+0000 mgr.y (mgr.34107) 12293 : cluster [DBG] pgmap v12764: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:58.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:57 smithi115 bash[45495]: audit 2022-04-22T23:26:56.966490+0000 mon.a (mon.0) 16931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:58.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:57 smithi115 bash[46934]: audit 2022-04-22T23:26:56.966490+0000 mon.a (mon.0) 16931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:57 smithi121 bash[36902]: audit 2022-04-22T23:26:56.966490+0000 mon.a (mon.0) 16931 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:26:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:58 smithi115 bash[45495]: cluster 2022-04-22T23:26:57.410678+0000 mgr.y (mgr.34107) 12294 : cluster [DBG] pgmap v12765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:59.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:58 smithi115 bash[45495]: audit 2022-04-22T23:26:58.603752+0000 mon.a (mon.0) 16932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:58 smithi115 bash[46934]: cluster 2022-04-22T23:26:57.410678+0000 mgr.y (mgr.34107) 12294 : cluster [DBG] pgmap v12765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:59.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:58 smithi115 bash[46934]: audit 2022-04-22T23:26:58.603752+0000 mon.a (mon.0) 16932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:26:59.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:58 smithi121 bash[36902]: cluster 2022-04-22T23:26:57.410678+0000 mgr.y (mgr.34107) 12294 : cluster [DBG] pgmap v12765: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:26:59.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:58 smithi121 bash[36902]: audit 2022-04-22T23:26:58.603752+0000 mon.a (mon.0) 16932 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:59 smithi121 bash[36902]: audit 2022-04-22T23:26:59.196106+0000 mon.a (mon.0) 16933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:27:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:59 smithi121 bash[36902]: audit 2022-04-22T23:26:59.198961+0000 mon.a (mon.0) 16934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:27:00.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:26:59 smithi121 bash[36902]: audit 2022-04-22T23:26:59.200603+0000 mon.a (mon.0) 16935 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:27:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[45495]: audit 2022-04-22T23:26:59.196106+0000 mon.a (mon.0) 16933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:27:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[45495]: audit 2022-04-22T23:26:59.198961+0000 mon.a (mon.0) 16934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:27:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[45495]: audit 2022-04-22T23:26:59.200603+0000 mon.a (mon.0) 16935 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:27:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[46934]: audit 2022-04-22T23:26:59.196106+0000 mon.a (mon.0) 16933 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:27:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[46934]: audit 2022-04-22T23:26:59.198961+0000 mon.a (mon.0) 16934 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:27:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:26:59 smithi115 bash[46934]: audit 2022-04-22T23:26:59.200603+0000 mon.a (mon.0) 16935 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:00 smithi121 bash[36902]: cluster 2022-04-22T23:26:59.411787+0000 mgr.y (mgr.34107) 12295 : cluster [DBG] pgmap v12766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:00 smithi121 bash[36902]: audit 2022-04-22T23:27:00.356957+0000 mon.a (mon.0) 16936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:01.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:00 smithi115 bash[45495]: cluster 2022-04-22T23:26:59.411787+0000 mgr.y (mgr.34107) 12295 : cluster [DBG] pgmap v12766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:00 smithi115 bash[45495]: audit 2022-04-22T23:27:00.356957+0000 mon.a (mon.0) 16936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:00 smithi115 bash[46934]: cluster 2022-04-22T23:26:59.411787+0000 mgr.y (mgr.34107) 12295 : cluster [DBG] pgmap v12766: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:00 smithi115 bash[46934]: audit 2022-04-22T23:27:00.356957+0000 mon.a (mon.0) 16936 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:02 smithi121 bash[36902]: cluster 2022-04-22T23:27:01.412769+0000 mgr.y (mgr.34107) 12296 : cluster [DBG] pgmap v12767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:02 smithi121 bash[36902]: audit 2022-04-22T23:27:01.969496+0000 mon.a (mon.0) 16937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:03.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:03.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:02 smithi115 bash[45495]: cluster 2022-04-22T23:27:01.412769+0000 mgr.y (mgr.34107) 12296 : cluster [DBG] pgmap v12767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:02 smithi115 bash[45495]: audit 2022-04-22T23:27:01.969496+0000 mon.a (mon.0) 16937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:02 smithi115 bash[46934]: cluster 2022-04-22T23:27:01.412769+0000 mgr.y (mgr.34107) 12296 : cluster [DBG] pgmap v12767: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:02 smithi115 bash[46934]: audit 2022-04-22T23:27:01.969496+0000 mon.a (mon.0) 16937 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:03.987 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:03 smithi121 bash[36902]: audit 2022-04-22T23:27:03.606827+0000 mon.a (mon.0) 16938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:03.987 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:03 smithi121 bash[36902]: audit 2022-04-22T23:27:03.668466+0000 mon.a (mon.0) 16939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:03 smithi115 bash[45495]: audit 2022-04-22T23:27:03.606827+0000 mon.a (mon.0) 16938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:03 smithi115 bash[45495]: audit 2022-04-22T23:27:03.668466+0000 mon.a (mon.0) 16939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:03 smithi115 bash[46934]: audit 2022-04-22T23:27:03.606827+0000 mon.a (mon.0) 16938 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:03 smithi115 bash[46934]: audit 2022-04-22T23:27:03.668466+0000 mon.a (mon.0) 16939 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:04 smithi121 bash[36902]: cluster 2022-04-22T23:27:03.413871+0000 mgr.y (mgr.34107) 12297 : cluster [DBG] pgmap v12768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:04 smithi121 bash[36902]: audit 2022-04-22T23:27:03.995697+0000 mon.a (mon.0) 16940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:04.899Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:04 smithi115 bash[45495]: cluster 2022-04-22T23:27:03.413871+0000 mgr.y (mgr.34107) 12297 : cluster [DBG] pgmap v12768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:04 smithi115 bash[45495]: audit 2022-04-22T23:27:03.995697+0000 mon.a (mon.0) 16940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:04 smithi115 bash[46934]: cluster 2022-04-22T23:27:03.413871+0000 mgr.y (mgr.34107) 12297 : cluster [DBG] pgmap v12768: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:05.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:04 smithi115 bash[46934]: audit 2022-04-22T23:27:03.995697+0000 mon.a (mon.0) 16940 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:27:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:06 smithi121 bash[36902]: audit 2022-04-22T23:27:05.360762+0000 mon.a (mon.0) 16941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:06 smithi115 bash[45495]: audit 2022-04-22T23:27:05.360762+0000 mon.a (mon.0) 16941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:06 smithi115 bash[46934]: audit 2022-04-22T23:27:05.360762+0000 mon.a (mon.0) 16941 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:07 smithi121 bash[36902]: cluster 2022-04-22T23:27:05.414870+0000 mgr.y (mgr.34107) 12298 : cluster [DBG] pgmap v12769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:07 smithi121 bash[36902]: audit 2022-04-22T23:27:06.972569+0000 mon.a (mon.0) 16942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:07 smithi115 bash[46934]: cluster 2022-04-22T23:27:05.414870+0000 mgr.y (mgr.34107) 12298 : cluster [DBG] pgmap v12769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:07 smithi115 bash[46934]: audit 2022-04-22T23:27:06.972569+0000 mon.a (mon.0) 16942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:07 smithi115 bash[45495]: cluster 2022-04-22T23:27:05.414870+0000 mgr.y (mgr.34107) 12298 : cluster [DBG] pgmap v12769: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:07 smithi115 bash[45495]: audit 2022-04-22T23:27:06.972569+0000 mon.a (mon.0) 16942 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:09 smithi115 bash[45495]: cluster 2022-04-22T23:27:07.416039+0000 mgr.y (mgr.34107) 12299 : cluster [DBG] pgmap v12770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:09 smithi115 bash[45495]: audit 2022-04-22T23:27:08.609861+0000 mon.a (mon.0) 16943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:09 smithi115 bash[46934]: cluster 2022-04-22T23:27:07.416039+0000 mgr.y (mgr.34107) 12299 : cluster [DBG] pgmap v12770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:09 smithi115 bash[46934]: audit 2022-04-22T23:27:08.609861+0000 mon.a (mon.0) 16943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:09 smithi121 bash[36902]: cluster 2022-04-22T23:27:07.416039+0000 mgr.y (mgr.34107) 12299 : cluster [DBG] pgmap v12770: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:09 smithi121 bash[36902]: audit 2022-04-22T23:27:08.609861+0000 mon.a (mon.0) 16943 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:11 smithi115 bash[45495]: cluster 2022-04-22T23:27:09.417115+0000 mgr.y (mgr.34107) 12300 : cluster [DBG] pgmap v12771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:11 smithi115 bash[45495]: audit 2022-04-22T23:27:10.364594+0000 mon.a (mon.0) 16944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:11 smithi115 bash[46934]: cluster 2022-04-22T23:27:09.417115+0000 mgr.y (mgr.34107) 12300 : cluster [DBG] pgmap v12771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:11 smithi115 bash[46934]: audit 2022-04-22T23:27:10.364594+0000 mon.a (mon.0) 16944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:11 smithi121 bash[36902]: cluster 2022-04-22T23:27:09.417115+0000 mgr.y (mgr.34107) 12300 : cluster [DBG] pgmap v12771: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:11 smithi121 bash[36902]: audit 2022-04-22T23:27:10.364594+0000 mon.a (mon.0) 16944 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:11.821 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:27:12.164 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:27:12.164 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:27:12.164 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 8s ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 8s ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (5h) 8s ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 9m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 8s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8s ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:27:12.165 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 8s ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 8s ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 8s ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 8s ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:27:12.166 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8s ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:27:12.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:12 smithi115 bash[46934]: audit 2022-04-22T23:27:11.975554+0000 mon.a (mon.0) 16945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:12.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:12 smithi115 bash[45495]: audit 2022-04-22T23:27:11.975554+0000 mon.a (mon.0) 16945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:12 smithi121 bash[36902]: audit 2022-04-22T23:27:11.975554+0000 mon.a (mon.0) 16945 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:12.570 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:27:12.570 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:27:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:27:12.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:12.572 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:27:12.572 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:27:12.572 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:27:12.572 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:27:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:13.040Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[45495]: cluster 2022-04-22T23:27:11.418339+0000 mgr.y (mgr.34107) 12301 : cluster [DBG] pgmap v12772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[45495]: audit 2022-04-22T23:27:11.801899+0000 mgr.y (mgr.34107) 12302 : audit [DBG] from='client.58009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[45495]: audit 2022-04-22T23:27:12.142164+0000 mgr.y (mgr.34107) 12303 : audit [DBG] from='client.58015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[45495]: audit 2022-04-22T23:27:12.566667+0000 mon.c (mon.1) 416 : audit [DBG] from='client.? 172.21.15.115:0/2883811515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[46934]: cluster 2022-04-22T23:27:11.418339+0000 mgr.y (mgr.34107) 12301 : cluster [DBG] pgmap v12772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[46934]: audit 2022-04-22T23:27:11.801899+0000 mgr.y (mgr.34107) 12302 : audit [DBG] from='client.58009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[46934]: audit 2022-04-22T23:27:12.142164+0000 mgr.y (mgr.34107) 12303 : audit [DBG] from='client.58015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:13 smithi115 bash[46934]: audit 2022-04-22T23:27:12.566667+0000 mon.c (mon.1) 416 : audit [DBG] from='client.? 172.21.15.115:0/2883811515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:13 smithi121 bash[36902]: cluster 2022-04-22T23:27:11.418339+0000 mgr.y (mgr.34107) 12301 : cluster [DBG] pgmap v12772: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:13 smithi121 bash[36902]: audit 2022-04-22T23:27:11.801899+0000 mgr.y (mgr.34107) 12302 : audit [DBG] from='client.58009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:13 smithi121 bash[36902]: audit 2022-04-22T23:27:12.142164+0000 mgr.y (mgr.34107) 12303 : audit [DBG] from='client.58015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:13 smithi121 bash[36902]: audit 2022-04-22T23:27:12.566667+0000 mon.c (mon.1) 416 : audit [DBG] from='client.? 172.21.15.115:0/2883811515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:14 smithi121 bash[36902]: audit 2022-04-22T23:27:13.612985+0000 mon.a (mon.0) 16946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:14.548 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:14 smithi115 bash[45495]: audit 2022-04-22T23:27:13.612985+0000 mon.a (mon.0) 16946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:14.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:14 smithi115 bash[46934]: audit 2022-04-22T23:27:13.612985+0000 mon.a (mon.0) 16946 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:15 smithi115 bash[45495]: cluster 2022-04-22T23:27:13.419548+0000 mgr.y (mgr.34107) 12304 : cluster [DBG] pgmap v12773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:14.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:15 smithi115 bash[46934]: cluster 2022-04-22T23:27:13.419548+0000 mgr.y (mgr.34107) 12304 : cluster [DBG] pgmap v12773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:15 smithi121 bash[36902]: cluster 2022-04-22T23:27:13.419548+0000 mgr.y (mgr.34107) 12304 : cluster [DBG] pgmap v12773: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:16 smithi121 bash[36902]: audit 2022-04-22T23:27:15.368080+0000 mon.a (mon.0) 16947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:16 smithi115 bash[45495]: audit 2022-04-22T23:27:15.368080+0000 mon.a (mon.0) 16947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:16 smithi115 bash[46934]: audit 2022-04-22T23:27:15.368080+0000 mon.a (mon.0) 16947 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:17 smithi121 bash[36902]: cluster 2022-04-22T23:27:15.420551+0000 mgr.y (mgr.34107) 12305 : cluster [DBG] pgmap v12774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:17 smithi121 bash[36902]: audit 2022-04-22T23:27:16.978523+0000 mon.a (mon.0) 16948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:17 smithi115 bash[45495]: cluster 2022-04-22T23:27:15.420551+0000 mgr.y (mgr.34107) 12305 : cluster [DBG] pgmap v12774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:17 smithi115 bash[45495]: audit 2022-04-22T23:27:16.978523+0000 mon.a (mon.0) 16948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:17 smithi115 bash[46934]: cluster 2022-04-22T23:27:15.420551+0000 mgr.y (mgr.34107) 12305 : cluster [DBG] pgmap v12774: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:17 smithi115 bash[46934]: audit 2022-04-22T23:27:16.978523+0000 mon.a (mon.0) 16948 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:18 smithi121 bash[36902]: audit 2022-04-22T23:27:17.430945+0000 mon.a (mon.0) 16949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:27:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:18 smithi121 bash[36902]: audit 2022-04-22T23:27:17.435753+0000 mon.a (mon.0) 16950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:27:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:18 smithi115 bash[45495]: audit 2022-04-22T23:27:17.430945+0000 mon.a (mon.0) 16949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:27:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:18 smithi115 bash[45495]: audit 2022-04-22T23:27:17.435753+0000 mon.a (mon.0) 16950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:27:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:18 smithi115 bash[46934]: audit 2022-04-22T23:27:17.430945+0000 mon.a (mon.0) 16949 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:27:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:18 smithi115 bash[46934]: audit 2022-04-22T23:27:17.435753+0000 mon.a (mon.0) 16950 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:27:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:19 smithi121 bash[36902]: cluster 2022-04-22T23:27:17.421550+0000 mgr.y (mgr.34107) 12306 : cluster [DBG] pgmap v12775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:19 smithi121 bash[36902]: audit 2022-04-22T23:27:18.616291+0000 mon.a (mon.0) 16951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:19 smithi115 bash[45495]: cluster 2022-04-22T23:27:17.421550+0000 mgr.y (mgr.34107) 12306 : cluster [DBG] pgmap v12775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:19 smithi115 bash[45495]: audit 2022-04-22T23:27:18.616291+0000 mon.a (mon.0) 16951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:19 smithi115 bash[46934]: cluster 2022-04-22T23:27:17.421550+0000 mgr.y (mgr.34107) 12306 : cluster [DBG] pgmap v12775: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:19 smithi115 bash[46934]: audit 2022-04-22T23:27:18.616291+0000 mon.a (mon.0) 16951 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:21 smithi121 bash[36902]: cluster 2022-04-22T23:27:19.422643+0000 mgr.y (mgr.34107) 12307 : cluster [DBG] pgmap v12776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:21 smithi121 bash[36902]: audit 2022-04-22T23:27:20.372016+0000 mon.a (mon.0) 16952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:21.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:21 smithi115 bash[45495]: cluster 2022-04-22T23:27:19.422643+0000 mgr.y (mgr.34107) 12307 : cluster [DBG] pgmap v12776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:21 smithi115 bash[45495]: audit 2022-04-22T23:27:20.372016+0000 mon.a (mon.0) 16952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:21 smithi115 bash[46934]: cluster 2022-04-22T23:27:19.422643+0000 mgr.y (mgr.34107) 12307 : cluster [DBG] pgmap v12776: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:21 smithi115 bash[46934]: audit 2022-04-22T23:27:20.372016+0000 mon.a (mon.0) 16952 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:22 smithi121 bash[36902]: audit 2022-04-22T23:27:21.981484+0000 mon.a (mon.0) 16953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:22 smithi115 bash[45495]: audit 2022-04-22T23:27:21.981484+0000 mon.a (mon.0) 16953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:22 smithi115 bash[46934]: audit 2022-04-22T23:27:21.981484+0000 mon.a (mon.0) 16953 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:23 smithi121 bash[36902]: cluster 2022-04-22T23:27:21.423756+0000 mgr.y (mgr.34107) 12308 : cluster [DBG] pgmap v12777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:23.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:23 smithi115 bash[45495]: cluster 2022-04-22T23:27:21.423756+0000 mgr.y (mgr.34107) 12308 : cluster [DBG] pgmap v12777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:23 smithi115 bash[46934]: cluster 2022-04-22T23:27:21.423756+0000 mgr.y (mgr.34107) 12308 : cluster [DBG] pgmap v12777: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:23.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:23.042Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:24 smithi121 bash[36902]: audit 2022-04-22T23:27:23.619391+0000 mon.a (mon.0) 16954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:24 smithi115 bash[45495]: audit 2022-04-22T23:27:23.619391+0000 mon.a (mon.0) 16954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:24 smithi115 bash[46934]: audit 2022-04-22T23:27:23.619391+0000 mon.a (mon.0) 16954 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:25 smithi115 bash[45495]: cluster 2022-04-22T23:27:23.424910+0000 mgr.y (mgr.34107) 12309 : cluster [DBG] pgmap v12778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:24.900Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:25 smithi115 bash[46934]: cluster 2022-04-22T23:27:23.424910+0000 mgr.y (mgr.34107) 12309 : cluster [DBG] pgmap v12778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:25 smithi121 bash[36902]: cluster 2022-04-22T23:27:23.424910+0000 mgr.y (mgr.34107) 12309 : cluster [DBG] pgmap v12778: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:26 smithi121 bash[36902]: audit 2022-04-22T23:27:25.375816+0000 mon.a (mon.0) 16955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:26 smithi115 bash[45495]: audit 2022-04-22T23:27:25.375816+0000 mon.a (mon.0) 16955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:26 smithi115 bash[46934]: audit 2022-04-22T23:27:25.375816+0000 mon.a (mon.0) 16955 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:27 smithi121 bash[36902]: cluster 2022-04-22T23:27:25.425961+0000 mgr.y (mgr.34107) 12310 : cluster [DBG] pgmap v12779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:27 smithi121 bash[36902]: audit 2022-04-22T23:27:26.984682+0000 mon.a (mon.0) 16956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:27 smithi115 bash[45495]: cluster 2022-04-22T23:27:25.425961+0000 mgr.y (mgr.34107) 12310 : cluster [DBG] pgmap v12779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:27 smithi115 bash[45495]: audit 2022-04-22T23:27:26.984682+0000 mon.a (mon.0) 16956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:27 smithi115 bash[46934]: cluster 2022-04-22T23:27:25.425961+0000 mgr.y (mgr.34107) 12310 : cluster [DBG] pgmap v12779: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:27 smithi115 bash[46934]: audit 2022-04-22T23:27:26.984682+0000 mon.a (mon.0) 16956 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:29 smithi121 bash[36902]: cluster 2022-04-22T23:27:27.426965+0000 mgr.y (mgr.34107) 12311 : cluster [DBG] pgmap v12780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:29 smithi121 bash[36902]: audit 2022-04-22T23:27:28.622605+0000 mon.a (mon.0) 16957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:29 smithi115 bash[45495]: cluster 2022-04-22T23:27:27.426965+0000 mgr.y (mgr.34107) 12311 : cluster [DBG] pgmap v12780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:29 smithi115 bash[45495]: audit 2022-04-22T23:27:28.622605+0000 mon.a (mon.0) 16957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:29 smithi115 bash[46934]: cluster 2022-04-22T23:27:27.426965+0000 mgr.y (mgr.34107) 12311 : cluster [DBG] pgmap v12780: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:29 smithi115 bash[46934]: audit 2022-04-22T23:27:28.622605+0000 mon.a (mon.0) 16957 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:31 smithi121 bash[36902]: cluster 2022-04-22T23:27:29.428177+0000 mgr.y (mgr.34107) 12312 : cluster [DBG] pgmap v12781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:31 smithi121 bash[36902]: audit 2022-04-22T23:27:30.379376+0000 mon.a (mon.0) 16958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:31 smithi115 bash[45495]: cluster 2022-04-22T23:27:29.428177+0000 mgr.y (mgr.34107) 12312 : cluster [DBG] pgmap v12781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:31 smithi115 bash[45495]: audit 2022-04-22T23:27:30.379376+0000 mon.a (mon.0) 16958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:31 smithi115 bash[46934]: cluster 2022-04-22T23:27:29.428177+0000 mgr.y (mgr.34107) 12312 : cluster [DBG] pgmap v12781: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:31 smithi115 bash[46934]: audit 2022-04-22T23:27:30.379376+0000 mon.a (mon.0) 16958 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:32.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:32 smithi121 bash[36902]: audit 2022-04-22T23:27:31.987553+0000 mon.a (mon.0) 16959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:32.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:32 smithi115 bash[45495]: audit 2022-04-22T23:27:31.987553+0000 mon.a (mon.0) 16959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:32 smithi115 bash[46934]: audit 2022-04-22T23:27:31.987553+0000 mon.a (mon.0) 16959 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:33 smithi121 bash[36902]: cluster 2022-04-22T23:27:31.429334+0000 mgr.y (mgr.34107) 12313 : cluster [DBG] pgmap v12782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:33 smithi115 bash[45495]: cluster 2022-04-22T23:27:31.429334+0000 mgr.y (mgr.34107) 12313 : cluster [DBG] pgmap v12782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:33.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:33.042Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:33 smithi115 bash[46934]: cluster 2022-04-22T23:27:31.429334+0000 mgr.y (mgr.34107) 12313 : cluster [DBG] pgmap v12782: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:34 smithi121 bash[36902]: audit 2022-04-22T23:27:33.625911+0000 mon.a (mon.0) 16960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:34 smithi115 bash[45495]: audit 2022-04-22T23:27:33.625911+0000 mon.a (mon.0) 16960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:34 smithi115 bash[46934]: audit 2022-04-22T23:27:33.625911+0000 mon.a (mon.0) 16960 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:35.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:34.901Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:35 smithi115 bash[46934]: cluster 2022-04-22T23:27:33.430473+0000 mgr.y (mgr.34107) 12314 : cluster [DBG] pgmap v12783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:35 smithi115 bash[45495]: cluster 2022-04-22T23:27:33.430473+0000 mgr.y (mgr.34107) 12314 : cluster [DBG] pgmap v12783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:35 smithi121 bash[36902]: cluster 2022-04-22T23:27:33.430473+0000 mgr.y (mgr.34107) 12314 : cluster [DBG] pgmap v12783: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:36 smithi121 bash[36902]: audit 2022-04-22T23:27:35.383076+0000 mon.a (mon.0) 16961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:36.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:36 smithi115 bash[45495]: audit 2022-04-22T23:27:35.383076+0000 mon.a (mon.0) 16961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:36 smithi115 bash[46934]: audit 2022-04-22T23:27:35.383076+0000 mon.a (mon.0) 16961 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:37 smithi121 bash[36902]: cluster 2022-04-22T23:27:35.431587+0000 mgr.y (mgr.34107) 12315 : cluster [DBG] pgmap v12784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:37 smithi121 bash[36902]: audit 2022-04-22T23:27:36.990563+0000 mon.a (mon.0) 16962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:37 smithi115 bash[45495]: cluster 2022-04-22T23:27:35.431587+0000 mgr.y (mgr.34107) 12315 : cluster [DBG] pgmap v12784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:37 smithi115 bash[45495]: audit 2022-04-22T23:27:36.990563+0000 mon.a (mon.0) 16962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:37 smithi115 bash[46934]: cluster 2022-04-22T23:27:35.431587+0000 mgr.y (mgr.34107) 12315 : cluster [DBG] pgmap v12784: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:37 smithi115 bash[46934]: audit 2022-04-22T23:27:36.990563+0000 mon.a (mon.0) 16962 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:39 smithi121 bash[36902]: cluster 2022-04-22T23:27:37.432612+0000 mgr.y (mgr.34107) 12316 : cluster [DBG] pgmap v12785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:39 smithi121 bash[36902]: audit 2022-04-22T23:27:38.629094+0000 mon.a (mon.0) 16963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:39 smithi115 bash[45495]: cluster 2022-04-22T23:27:37.432612+0000 mgr.y (mgr.34107) 12316 : cluster [DBG] pgmap v12785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:39 smithi115 bash[45495]: audit 2022-04-22T23:27:38.629094+0000 mon.a (mon.0) 16963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:39 smithi115 bash[46934]: cluster 2022-04-22T23:27:37.432612+0000 mgr.y (mgr.34107) 12316 : cluster [DBG] pgmap v12785: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:39 smithi115 bash[46934]: audit 2022-04-22T23:27:38.629094+0000 mon.a (mon.0) 16963 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:41 smithi121 bash[36902]: cluster 2022-04-22T23:27:39.433669+0000 mgr.y (mgr.34107) 12317 : cluster [DBG] pgmap v12786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:41 smithi121 bash[36902]: audit 2022-04-22T23:27:40.386861+0000 mon.a (mon.0) 16964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:41 smithi115 bash[46934]: cluster 2022-04-22T23:27:39.433669+0000 mgr.y (mgr.34107) 12317 : cluster [DBG] pgmap v12786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:41.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:41 smithi115 bash[46934]: audit 2022-04-22T23:27:40.386861+0000 mon.a (mon.0) 16964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:41 smithi115 bash[45495]: cluster 2022-04-22T23:27:39.433669+0000 mgr.y (mgr.34107) 12317 : cluster [DBG] pgmap v12786: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:41 smithi115 bash[45495]: audit 2022-04-22T23:27:40.386861+0000 mon.a (mon.0) 16964 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:42.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:42 smithi121 bash[36902]: audit 2022-04-22T23:27:41.993842+0000 mon.a (mon.0) 16965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:42.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:42 smithi115 bash[45495]: audit 2022-04-22T23:27:41.993842+0000 mon.a (mon.0) 16965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:42.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:42 smithi115 bash[46934]: audit 2022-04-22T23:27:41.993842+0000 mon.a (mon.0) 16965 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:42.945 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:27:43.295 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:27:43.295 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 9m ago 6h 15.7M - ba2b418f427c 9255ba08f72f 2022-04-22T23:27:43.295 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 39s ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:27:43.295 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 9m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 39s ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 39s ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 9m ago 6h 494M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5h) 9m ago 6h 470M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 39s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 9m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 9m ago 6h 10.7M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 39s ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:27:43.296 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 9m ago 6h 721M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 9m ago 6h 802M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 9m ago 6h 543M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 9m ago 6h 506M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 39s ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 39s ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 39s ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 39s ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:27:43.297 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 39s ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:27:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:43 smithi121 bash[36902]: cluster 2022-04-22T23:27:41.434788+0000 mgr.y (mgr.34107) 12318 : cluster [DBG] pgmap v12787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:43 smithi115 bash[46934]: cluster 2022-04-22T23:27:41.434788+0000 mgr.y (mgr.34107) 12318 : cluster [DBG] pgmap v12787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:43.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:43.042Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:43 smithi115 bash[45495]: cluster 2022-04-22T23:27:41.434788+0000 mgr.y (mgr.34107) 12318 : cluster [DBG] pgmap v12787: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:27:43.697 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:27:43.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:27:43.699 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:27:43.699 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:27:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:44 smithi121 bash[36902]: audit 2022-04-22T23:27:42.924282+0000 mgr.y (mgr.34107) 12319 : audit [DBG] from='client.58027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:44 smithi121 bash[36902]: audit 2022-04-22T23:27:43.272118+0000 mgr.y (mgr.34107) 12320 : audit [DBG] from='client.48093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:44 smithi121 bash[36902]: audit 2022-04-22T23:27:43.632146+0000 mon.a (mon.0) 16966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:44.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:44 smithi121 bash[36902]: audit 2022-04-22T23:27:43.693340+0000 mon.a (mon.0) 16967 : audit [DBG] from='client.? 172.21.15.115:0/367969786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[45495]: audit 2022-04-22T23:27:42.924282+0000 mgr.y (mgr.34107) 12319 : audit [DBG] from='client.58027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[45495]: audit 2022-04-22T23:27:43.272118+0000 mgr.y (mgr.34107) 12320 : audit [DBG] from='client.48093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[45495]: audit 2022-04-22T23:27:43.632146+0000 mon.a (mon.0) 16966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[45495]: audit 2022-04-22T23:27:43.693340+0000 mon.a (mon.0) 16967 : audit [DBG] from='client.? 172.21.15.115:0/367969786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[46934]: audit 2022-04-22T23:27:42.924282+0000 mgr.y (mgr.34107) 12319 : audit [DBG] from='client.58027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[46934]: audit 2022-04-22T23:27:43.272118+0000 mgr.y (mgr.34107) 12320 : audit [DBG] from='client.48093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:27:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[46934]: audit 2022-04-22T23:27:43.632146+0000 mon.a (mon.0) 16966 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:44.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[46934]: audit 2022-04-22T23:27:43.693340+0000 mon.a (mon.0) 16967 : audit [DBG] from='client.? 172.21.15.115:0/367969786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:27:45.193 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:44.902Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:45 smithi121 bash[36902]: cluster 2022-04-22T23:27:43.435679+0000 mgr.y (mgr.34107) 12321 : cluster [DBG] pgmap v12788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:45.685 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:45 smithi115 bash[45495]: cluster 2022-04-22T23:27:43.435679+0000 mgr.y (mgr.34107) 12321 : cluster [DBG] pgmap v12788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:45.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:45 smithi115 bash[46934]: cluster 2022-04-22T23:27:43.435679+0000 mgr.y (mgr.34107) 12321 : cluster [DBG] pgmap v12788: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:46 smithi121 bash[36902]: audit 2022-04-22T23:27:45.390465+0000 mon.a (mon.0) 16968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:46 smithi115 bash[45495]: audit 2022-04-22T23:27:45.390465+0000 mon.a (mon.0) 16968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:46 smithi115 bash[46934]: audit 2022-04-22T23:27:45.390465+0000 mon.a (mon.0) 16968 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:47 smithi121 bash[36902]: cluster 2022-04-22T23:27:45.436646+0000 mgr.y (mgr.34107) 12322 : cluster [DBG] pgmap v12789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:47 smithi121 bash[36902]: audit 2022-04-22T23:27:46.996798+0000 mon.a (mon.0) 16969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:47 smithi115 bash[46934]: cluster 2022-04-22T23:27:45.436646+0000 mgr.y (mgr.34107) 12322 : cluster [DBG] pgmap v12789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:47 smithi115 bash[46934]: audit 2022-04-22T23:27:46.996798+0000 mon.a (mon.0) 16969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:47 smithi115 bash[45495]: cluster 2022-04-22T23:27:45.436646+0000 mgr.y (mgr.34107) 12322 : cluster [DBG] pgmap v12789: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:47.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:47 smithi115 bash[45495]: audit 2022-04-22T23:27:46.996798+0000 mon.a (mon.0) 16969 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:49 smithi121 bash[36902]: cluster 2022-04-22T23:27:47.437748+0000 mgr.y (mgr.34107) 12323 : cluster [DBG] pgmap v12790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:49 smithi121 bash[36902]: audit 2022-04-22T23:27:48.635033+0000 mon.a (mon.0) 16970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:49 smithi115 bash[45495]: cluster 2022-04-22T23:27:47.437748+0000 mgr.y (mgr.34107) 12323 : cluster [DBG] pgmap v12790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:49 smithi115 bash[45495]: audit 2022-04-22T23:27:48.635033+0000 mon.a (mon.0) 16970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:49 smithi115 bash[46934]: cluster 2022-04-22T23:27:47.437748+0000 mgr.y (mgr.34107) 12323 : cluster [DBG] pgmap v12790: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:49 smithi115 bash[46934]: audit 2022-04-22T23:27:48.635033+0000 mon.a (mon.0) 16970 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:51 smithi121 bash[36902]: cluster 2022-04-22T23:27:49.438832+0000 mgr.y (mgr.34107) 12324 : cluster [DBG] pgmap v12791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:51 smithi121 bash[36902]: audit 2022-04-22T23:27:50.394164+0000 mon.a (mon.0) 16971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:51 smithi115 bash[45495]: cluster 2022-04-22T23:27:49.438832+0000 mgr.y (mgr.34107) 12324 : cluster [DBG] pgmap v12791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:51 smithi115 bash[45495]: audit 2022-04-22T23:27:50.394164+0000 mon.a (mon.0) 16971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:51 smithi115 bash[46934]: cluster 2022-04-22T23:27:49.438832+0000 mgr.y (mgr.34107) 12324 : cluster [DBG] pgmap v12791: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:51 smithi115 bash[46934]: audit 2022-04-22T23:27:50.394164+0000 mon.a (mon.0) 16971 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:52 smithi121 bash[36902]: audit 2022-04-22T23:27:51.999838+0000 mon.a (mon.0) 16972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:52.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:52 smithi115 bash[45495]: audit 2022-04-22T23:27:51.999838+0000 mon.a (mon.0) 16972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:52 smithi115 bash[46934]: audit 2022-04-22T23:27:51.999838+0000 mon.a (mon.0) 16972 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:53 smithi121 bash[36902]: cluster 2022-04-22T23:27:51.439952+0000 mgr.y (mgr.34107) 12325 : cluster [DBG] pgmap v12792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:53.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:53.041Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:53 smithi115 bash[45495]: cluster 2022-04-22T23:27:51.439952+0000 mgr.y (mgr.34107) 12325 : cluster [DBG] pgmap v12792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:53 smithi115 bash[46934]: cluster 2022-04-22T23:27:51.439952+0000 mgr.y (mgr.34107) 12325 : cluster [DBG] pgmap v12792: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:54 smithi121 bash[36902]: audit 2022-04-22T23:27:53.638106+0000 mon.a (mon.0) 16973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:54 smithi115 bash[45495]: audit 2022-04-22T23:27:53.638106+0000 mon.a (mon.0) 16973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:54 smithi115 bash[46934]: audit 2022-04-22T23:27:53.638106+0000 mon.a (mon.0) 16973 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:55.238 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:27:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:27:54.903Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:27:55.238 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:55 smithi115 bash[45495]: cluster 2022-04-22T23:27:53.441094+0000 mgr.y (mgr.34107) 12326 : cluster [DBG] pgmap v12793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:55 smithi121 bash[36902]: cluster 2022-04-22T23:27:53.441094+0000 mgr.y (mgr.34107) 12326 : cluster [DBG] pgmap v12793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:55.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:55 smithi115 bash[46934]: cluster 2022-04-22T23:27:53.441094+0000 mgr.y (mgr.34107) 12326 : cluster [DBG] pgmap v12793: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:56 smithi121 bash[36902]: audit 2022-04-22T23:27:55.397914+0000 mon.a (mon.0) 16974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:56.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:56 smithi115 bash[45495]: audit 2022-04-22T23:27:55.397914+0000 mon.a (mon.0) 16974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:56 smithi115 bash[46934]: audit 2022-04-22T23:27:55.397914+0000 mon.a (mon.0) 16974 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:27:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:57 smithi121 bash[36902]: cluster 2022-04-22T23:27:55.442100+0000 mgr.y (mgr.34107) 12327 : cluster [DBG] pgmap v12794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:57 smithi121 bash[36902]: audit 2022-04-22T23:27:57.002818+0000 mon.a (mon.0) 16975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:57 smithi115 bash[45495]: cluster 2022-04-22T23:27:55.442100+0000 mgr.y (mgr.34107) 12327 : cluster [DBG] pgmap v12794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:57 smithi115 bash[45495]: audit 2022-04-22T23:27:57.002818+0000 mon.a (mon.0) 16975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:57 smithi115 bash[46934]: cluster 2022-04-22T23:27:55.442100+0000 mgr.y (mgr.34107) 12327 : cluster [DBG] pgmap v12794: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:57 smithi115 bash[46934]: audit 2022-04-22T23:27:57.002818+0000 mon.a (mon.0) 16975 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:27:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:59 smithi121 bash[36902]: cluster 2022-04-22T23:27:57.443173+0000 mgr.y (mgr.34107) 12328 : cluster [DBG] pgmap v12795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:27:59 smithi121 bash[36902]: audit 2022-04-22T23:27:58.641171+0000 mon.a (mon.0) 16976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:59 smithi115 bash[45495]: cluster 2022-04-22T23:27:57.443173+0000 mgr.y (mgr.34107) 12328 : cluster [DBG] pgmap v12795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:27:59 smithi115 bash[45495]: audit 2022-04-22T23:27:58.641171+0000 mon.a (mon.0) 16976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:27:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:59 smithi115 bash[46934]: cluster 2022-04-22T23:27:57.443173+0000 mgr.y (mgr.34107) 12328 : cluster [DBG] pgmap v12795: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:27:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:27:59 smithi115 bash[46934]: audit 2022-04-22T23:27:58.641171+0000 mon.a (mon.0) 16976 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:01 smithi121 bash[36902]: cluster 2022-04-22T23:27:59.444289+0000 mgr.y (mgr.34107) 12329 : cluster [DBG] pgmap v12796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:01 smithi121 bash[36902]: audit 2022-04-22T23:28:00.401668+0000 mon.a (mon.0) 16977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:01 smithi115 bash[46934]: cluster 2022-04-22T23:27:59.444289+0000 mgr.y (mgr.34107) 12329 : cluster [DBG] pgmap v12796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:01 smithi115 bash[46934]: audit 2022-04-22T23:28:00.401668+0000 mon.a (mon.0) 16977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:01 smithi115 bash[45495]: cluster 2022-04-22T23:27:59.444289+0000 mgr.y (mgr.34107) 12329 : cluster [DBG] pgmap v12796: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:01 smithi115 bash[45495]: audit 2022-04-22T23:28:00.401668+0000 mon.a (mon.0) 16977 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:02 smithi115 bash[46934]: audit 2022-04-22T23:28:02.005825+0000 mon.a (mon.0) 16978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:02 smithi115 bash[45495]: audit 2022-04-22T23:28:02.005825+0000 mon.a (mon.0) 16978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:02 smithi121 bash[36902]: audit 2022-04-22T23:28:02.005825+0000 mon.a (mon.0) 16978 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:03.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:03.044Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:03 smithi115 bash[45495]: cluster 2022-04-22T23:28:01.445402+0000 mgr.y (mgr.34107) 12330 : cluster [DBG] pgmap v12797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:03 smithi115 bash[46934]: cluster 2022-04-22T23:28:01.445402+0000 mgr.y (mgr.34107) 12330 : cluster [DBG] pgmap v12797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:03 smithi121 bash[36902]: cluster 2022-04-22T23:28:01.445402+0000 mgr.y (mgr.34107) 12330 : cluster [DBG] pgmap v12797: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[45495]: audit 2022-04-22T23:28:03.644182+0000 mon.a (mon.0) 16979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[45495]: audit 2022-04-22T23:28:04.004147+0000 mon.a (mon.0) 16980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[45495]: audit 2022-04-22T23:28:04.006973+0000 mon.a (mon.0) 16981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[45495]: audit 2022-04-22T23:28:04.008685+0000 mon.a (mon.0) 16982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[46934]: audit 2022-04-22T23:28:03.644182+0000 mon.a (mon.0) 16979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[46934]: audit 2022-04-22T23:28:04.004147+0000 mon.a (mon.0) 16980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:28:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[46934]: audit 2022-04-22T23:28:04.006973+0000 mon.a (mon.0) 16981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:28:04.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[46934]: audit 2022-04-22T23:28:04.008685+0000 mon.a (mon.0) 16982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:28:04.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:04 smithi121 bash[36902]: audit 2022-04-22T23:28:03.644182+0000 mon.a (mon.0) 16979 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:04.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:04 smithi121 bash[36902]: audit 2022-04-22T23:28:04.004147+0000 mon.a (mon.0) 16980 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:28:04.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:04 smithi121 bash[36902]: audit 2022-04-22T23:28:04.006973+0000 mon.a (mon.0) 16981 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:28:04.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:04 smithi121 bash[36902]: audit 2022-04-22T23:28:04.008685+0000 mon.a (mon.0) 16982 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:28:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:04.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:05.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:05 smithi121 bash[36902]: cluster 2022-04-22T23:28:03.446568+0000 mgr.y (mgr.34107) 12331 : cluster [DBG] pgmap v12798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:05.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:05 smithi121 bash[36902]: audit 2022-04-22T23:28:04.342278+0000 mon.a (mon.0) 16983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:05.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:05 smithi121 bash[36902]: audit 2022-04-22T23:28:04.347102+0000 mon.a (mon.0) 16984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[45495]: cluster 2022-04-22T23:28:03.446568+0000 mgr.y (mgr.34107) 12331 : cluster [DBG] pgmap v12798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[45495]: audit 2022-04-22T23:28:04.342278+0000 mon.a (mon.0) 16983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[45495]: audit 2022-04-22T23:28:04.347102+0000 mon.a (mon.0) 16984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[46934]: cluster 2022-04-22T23:28:03.446568+0000 mgr.y (mgr.34107) 12331 : cluster [DBG] pgmap v12798: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[46934]: audit 2022-04-22T23:28:04.342278+0000 mon.a (mon.0) 16983 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:05.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:05 smithi115 bash[46934]: audit 2022-04-22T23:28:04.347102+0000 mon.a (mon.0) 16984 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:06 smithi121 bash[36902]: audit 2022-04-22T23:28:05.405148+0000 mon.a (mon.0) 16985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:06 smithi115 bash[45495]: audit 2022-04-22T23:28:05.405148+0000 mon.a (mon.0) 16985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:06 smithi115 bash[46934]: audit 2022-04-22T23:28:05.405148+0000 mon.a (mon.0) 16985 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:07 smithi121 bash[36902]: cluster 2022-04-22T23:28:05.447655+0000 mgr.y (mgr.34107) 12332 : cluster [DBG] pgmap v12799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:07 smithi121 bash[36902]: audit 2022-04-22T23:28:07.008752+0000 mon.a (mon.0) 16986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:07 smithi115 bash[45495]: cluster 2022-04-22T23:28:05.447655+0000 mgr.y (mgr.34107) 12332 : cluster [DBG] pgmap v12799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:07 smithi115 bash[45495]: audit 2022-04-22T23:28:07.008752+0000 mon.a (mon.0) 16986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:07 smithi115 bash[46934]: cluster 2022-04-22T23:28:05.447655+0000 mgr.y (mgr.34107) 12332 : cluster [DBG] pgmap v12799: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:07 smithi115 bash[46934]: audit 2022-04-22T23:28:07.008752+0000 mon.a (mon.0) 16986 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:09 smithi121 bash[36902]: cluster 2022-04-22T23:28:07.448624+0000 mgr.y (mgr.34107) 12333 : cluster [DBG] pgmap v12800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:09 smithi121 bash[36902]: audit 2022-04-22T23:28:08.530124+0000 mon.a (mon.0) 16987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:09 smithi121 bash[36902]: audit 2022-04-22T23:28:08.647317+0000 mon.a (mon.0) 16988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:09.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:09 smithi121 bash[36902]: audit 2022-04-22T23:28:08.879746+0000 mon.a (mon.0) 16989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[45495]: cluster 2022-04-22T23:28:07.448624+0000 mgr.y (mgr.34107) 12333 : cluster [DBG] pgmap v12800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[45495]: audit 2022-04-22T23:28:08.530124+0000 mon.a (mon.0) 16987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[45495]: audit 2022-04-22T23:28:08.647317+0000 mon.a (mon.0) 16988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[45495]: audit 2022-04-22T23:28:08.879746+0000 mon.a (mon.0) 16989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[46934]: cluster 2022-04-22T23:28:07.448624+0000 mgr.y (mgr.34107) 12333 : cluster [DBG] pgmap v12800: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[46934]: audit 2022-04-22T23:28:08.530124+0000 mon.a (mon.0) 16987 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[46934]: audit 2022-04-22T23:28:08.647317+0000 mon.a (mon.0) 16988 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:09 smithi115 bash[46934]: audit 2022-04-22T23:28:08.879746+0000 mon.a (mon.0) 16989 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:28:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:11 smithi121 bash[36902]: cluster 2022-04-22T23:28:09.449830+0000 mgr.y (mgr.34107) 12334 : cluster [DBG] pgmap v12801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:11 smithi121 bash[36902]: audit 2022-04-22T23:28:10.408925+0000 mon.a (mon.0) 16990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:11 smithi115 bash[45495]: cluster 2022-04-22T23:28:09.449830+0000 mgr.y (mgr.34107) 12334 : cluster [DBG] pgmap v12801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:11 smithi115 bash[45495]: audit 2022-04-22T23:28:10.408925+0000 mon.a (mon.0) 16990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:11 smithi115 bash[46934]: cluster 2022-04-22T23:28:09.449830+0000 mgr.y (mgr.34107) 12334 : cluster [DBG] pgmap v12801: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:11.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:11 smithi115 bash[46934]: audit 2022-04-22T23:28:10.408925+0000 mon.a (mon.0) 16990 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:12 smithi121 bash[36902]: audit 2022-04-22T23:28:12.011682+0000 mon.a (mon.0) 16991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:12 smithi115 bash[45495]: audit 2022-04-22T23:28:12.011682+0000 mon.a (mon.0) 16991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:12 smithi115 bash[46934]: audit 2022-04-22T23:28:12.011682+0000 mon.a (mon.0) 16991 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:13.375 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:13.043Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:13 smithi121 bash[36902]: cluster 2022-04-22T23:28:11.450951+0000 mgr.y (mgr.34107) 12335 : cluster [DBG] pgmap v12802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:13 smithi115 bash[45495]: cluster 2022-04-22T23:28:11.450951+0000 mgr.y (mgr.34107) 12335 : cluster [DBG] pgmap v12802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:13 smithi115 bash[46934]: cluster 2022-04-22T23:28:11.450951+0000 mgr.y (mgr.34107) 12335 : cluster [DBG] pgmap v12802: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:14.074 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:28:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:28:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5s ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 70s ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5s ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 70s ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 70s ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 5s ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 5s ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 70s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5h) 5s ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5s ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:28:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 70s ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 5s ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 5s ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 5s ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 5s ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 70s ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 70s ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 70s ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 70s ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:28:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 70s ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:28:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:14 smithi121 bash[36902]: audit 2022-04-22T23:28:13.650464+0000 mon.a (mon.0) 16992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:14 smithi115 bash[45495]: audit 2022-04-22T23:28:13.650464+0000 mon.a (mon.0) 16992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:14 smithi115 bash[46934]: audit 2022-04-22T23:28:13.650464+0000 mon.a (mon.0) 16992 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:14.818 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:28:14.818 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:28:14.818 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:28:14.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:14.818 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:28:14.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:28:14.820 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:28:14.820 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:28:15.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:14.902Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:15 smithi121 bash[36902]: cluster 2022-04-22T23:28:13.452130+0000 mgr.y (mgr.34107) 12336 : cluster [DBG] pgmap v12803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:15 smithi121 bash[36902]: audit 2022-04-22T23:28:14.055869+0000 mgr.y (mgr.34107) 12337 : audit [DBG] from='client.48105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:15 smithi121 bash[36902]: audit 2022-04-22T23:28:14.399127+0000 mgr.y (mgr.34107) 12338 : audit [DBG] from='client.58051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:15 smithi121 bash[36902]: audit 2022-04-22T23:28:14.814392+0000 mon.c (mon.1) 417 : audit [DBG] from='client.? 172.21.15.115:0/981951225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[45495]: cluster 2022-04-22T23:28:13.452130+0000 mgr.y (mgr.34107) 12336 : cluster [DBG] pgmap v12803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[45495]: audit 2022-04-22T23:28:14.055869+0000 mgr.y (mgr.34107) 12337 : audit [DBG] from='client.48105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[45495]: audit 2022-04-22T23:28:14.399127+0000 mgr.y (mgr.34107) 12338 : audit [DBG] from='client.58051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[45495]: audit 2022-04-22T23:28:14.814392+0000 mon.c (mon.1) 417 : audit [DBG] from='client.? 172.21.15.115:0/981951225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[46934]: cluster 2022-04-22T23:28:13.452130+0000 mgr.y (mgr.34107) 12336 : cluster [DBG] pgmap v12803: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[46934]: audit 2022-04-22T23:28:14.055869+0000 mgr.y (mgr.34107) 12337 : audit [DBG] from='client.48105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[46934]: audit 2022-04-22T23:28:14.399127+0000 mgr.y (mgr.34107) 12338 : audit [DBG] from='client.58051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:15 smithi115 bash[46934]: audit 2022-04-22T23:28:14.814392+0000 mon.c (mon.1) 417 : audit [DBG] from='client.? 172.21.15.115:0/981951225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:16 smithi121 bash[36902]: audit 2022-04-22T23:28:15.412483+0000 mon.a (mon.0) 16993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:16 smithi115 bash[45495]: audit 2022-04-22T23:28:15.412483+0000 mon.a (mon.0) 16993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:16 smithi115 bash[46934]: audit 2022-04-22T23:28:15.412483+0000 mon.a (mon.0) 16993 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:17 smithi121 bash[36902]: cluster 2022-04-22T23:28:15.453328+0000 mgr.y (mgr.34107) 12339 : cluster [DBG] pgmap v12804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:18.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:17 smithi121 bash[36902]: audit 2022-04-22T23:28:17.014569+0000 mon.a (mon.0) 16994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:18.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:17 smithi115 bash[45495]: cluster 2022-04-22T23:28:15.453328+0000 mgr.y (mgr.34107) 12339 : cluster [DBG] pgmap v12804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:18.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:17 smithi115 bash[45495]: audit 2022-04-22T23:28:17.014569+0000 mon.a (mon.0) 16994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:18.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:17 smithi115 bash[46934]: cluster 2022-04-22T23:28:15.453328+0000 mgr.y (mgr.34107) 12339 : cluster [DBG] pgmap v12804: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:18.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:17 smithi115 bash[46934]: audit 2022-04-22T23:28:17.014569+0000 mon.a (mon.0) 16994 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:18 smithi121 bash[36902]: audit 2022-04-22T23:28:17.442402+0000 mon.a (mon.0) 16995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:28:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:18 smithi121 bash[36902]: audit 2022-04-22T23:28:17.451045+0000 mon.a (mon.0) 16996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:28:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:18 smithi115 bash[45495]: audit 2022-04-22T23:28:17.442402+0000 mon.a (mon.0) 16995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:28:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:18 smithi115 bash[45495]: audit 2022-04-22T23:28:17.451045+0000 mon.a (mon.0) 16996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:28:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:18 smithi115 bash[46934]: audit 2022-04-22T23:28:17.442402+0000 mon.a (mon.0) 16995 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:28:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:18 smithi115 bash[46934]: audit 2022-04-22T23:28:17.451045+0000 mon.a (mon.0) 16996 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:28:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:19 smithi121 bash[36902]: cluster 2022-04-22T23:28:17.454336+0000 mgr.y (mgr.34107) 12340 : cluster [DBG] pgmap v12805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:19 smithi121 bash[36902]: audit 2022-04-22T23:28:18.653646+0000 mon.a (mon.0) 16997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:19 smithi115 bash[45495]: cluster 2022-04-22T23:28:17.454336+0000 mgr.y (mgr.34107) 12340 : cluster [DBG] pgmap v12805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:19 smithi115 bash[45495]: audit 2022-04-22T23:28:18.653646+0000 mon.a (mon.0) 16997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:19 smithi115 bash[46934]: cluster 2022-04-22T23:28:17.454336+0000 mgr.y (mgr.34107) 12340 : cluster [DBG] pgmap v12805: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:19 smithi115 bash[46934]: audit 2022-04-22T23:28:18.653646+0000 mon.a (mon.0) 16997 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:20 smithi121 bash[36902]: cluster 2022-04-22T23:28:19.455430+0000 mgr.y (mgr.34107) 12341 : cluster [DBG] pgmap v12806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:20.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:20 smithi115 bash[45495]: cluster 2022-04-22T23:28:19.455430+0000 mgr.y (mgr.34107) 12341 : cluster [DBG] pgmap v12806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:20 smithi115 bash[46934]: cluster 2022-04-22T23:28:19.455430+0000 mgr.y (mgr.34107) 12341 : cluster [DBG] pgmap v12806: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:21 smithi121 bash[36902]: audit 2022-04-22T23:28:20.419950+0000 mon.a (mon.0) 16998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:21 smithi115 bash[45495]: audit 2022-04-22T23:28:20.419950+0000 mon.a (mon.0) 16998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:21 smithi115 bash[46934]: audit 2022-04-22T23:28:20.419950+0000 mon.a (mon.0) 16998 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:22 smithi121 bash[36902]: cluster 2022-04-22T23:28:21.456685+0000 mgr.y (mgr.34107) 12342 : cluster [DBG] pgmap v12807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:22 smithi121 bash[36902]: audit 2022-04-22T23:28:22.017721+0000 mon.a (mon.0) 16999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:22 smithi115 bash[45495]: cluster 2022-04-22T23:28:21.456685+0000 mgr.y (mgr.34107) 12342 : cluster [DBG] pgmap v12807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:22 smithi115 bash[45495]: audit 2022-04-22T23:28:22.017721+0000 mon.a (mon.0) 16999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:22 smithi115 bash[46934]: cluster 2022-04-22T23:28:21.456685+0000 mgr.y (mgr.34107) 12342 : cluster [DBG] pgmap v12807: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:22 smithi115 bash[46934]: audit 2022-04-22T23:28:22.017721+0000 mon.a (mon.0) 16999 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:23.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:23.045Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:24.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:23 smithi115 bash[45495]: audit 2022-04-22T23:28:23.656974+0000 mon.a (mon.0) 17000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:24.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:23 smithi115 bash[46934]: audit 2022-04-22T23:28:23.656974+0000 mon.a (mon.0) 17000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:23 smithi121 bash[36902]: audit 2022-04-22T23:28:23.656974+0000 mon.a (mon.0) 17000 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:25.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:24 smithi115 bash[45495]: cluster 2022-04-22T23:28:23.457993+0000 mgr.y (mgr.34107) 12343 : cluster [DBG] pgmap v12808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:25.041 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:24.904Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:25.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:24 smithi115 bash[46934]: cluster 2022-04-22T23:28:23.457993+0000 mgr.y (mgr.34107) 12343 : cluster [DBG] pgmap v12808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:24 smithi121 bash[36902]: cluster 2022-04-22T23:28:23.457993+0000 mgr.y (mgr.34107) 12343 : cluster [DBG] pgmap v12808: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:25 smithi121 bash[36902]: audit 2022-04-22T23:28:25.420428+0000 mon.a (mon.0) 17001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:25 smithi115 bash[45495]: audit 2022-04-22T23:28:25.420428+0000 mon.a (mon.0) 17001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:25 smithi115 bash[46934]: audit 2022-04-22T23:28:25.420428+0000 mon.a (mon.0) 17001 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:26 smithi121 bash[36902]: cluster 2022-04-22T23:28:25.459188+0000 mgr.y (mgr.34107) 12344 : cluster [DBG] pgmap v12809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:27.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:26 smithi115 bash[45495]: cluster 2022-04-22T23:28:25.459188+0000 mgr.y (mgr.34107) 12344 : cluster [DBG] pgmap v12809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:26 smithi115 bash[46934]: cluster 2022-04-22T23:28:25.459188+0000 mgr.y (mgr.34107) 12344 : cluster [DBG] pgmap v12809: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:27 smithi121 bash[36902]: audit 2022-04-22T23:28:27.020858+0000 mon.a (mon.0) 17002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:27 smithi115 bash[45495]: audit 2022-04-22T23:28:27.020858+0000 mon.a (mon.0) 17002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:27 smithi115 bash[46934]: audit 2022-04-22T23:28:27.020858+0000 mon.a (mon.0) 17002 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:28 smithi121 bash[36902]: cluster 2022-04-22T23:28:27.460405+0000 mgr.y (mgr.34107) 12345 : cluster [DBG] pgmap v12810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:28 smithi121 bash[36902]: audit 2022-04-22T23:28:28.660296+0000 mon.a (mon.0) 17003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:28 smithi115 bash[45495]: cluster 2022-04-22T23:28:27.460405+0000 mgr.y (mgr.34107) 12345 : cluster [DBG] pgmap v12810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:28 smithi115 bash[45495]: audit 2022-04-22T23:28:28.660296+0000 mon.a (mon.0) 17003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:28 smithi115 bash[46934]: cluster 2022-04-22T23:28:27.460405+0000 mgr.y (mgr.34107) 12345 : cluster [DBG] pgmap v12810: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:28 smithi115 bash[46934]: audit 2022-04-22T23:28:28.660296+0000 mon.a (mon.0) 17003 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:30 smithi121 bash[36902]: cluster 2022-04-22T23:28:29.461564+0000 mgr.y (mgr.34107) 12346 : cluster [DBG] pgmap v12811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:30 smithi121 bash[36902]: audit 2022-04-22T23:28:30.424247+0000 mon.a (mon.0) 17004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:30 smithi115 bash[45495]: cluster 2022-04-22T23:28:29.461564+0000 mgr.y (mgr.34107) 12346 : cluster [DBG] pgmap v12811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:30 smithi115 bash[45495]: audit 2022-04-22T23:28:30.424247+0000 mon.a (mon.0) 17004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:30 smithi115 bash[46934]: cluster 2022-04-22T23:28:29.461564+0000 mgr.y (mgr.34107) 12346 : cluster [DBG] pgmap v12811: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:30 smithi115 bash[46934]: audit 2022-04-22T23:28:30.424247+0000 mon.a (mon.0) 17004 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:32 smithi121 bash[36902]: cluster 2022-04-22T23:28:31.462594+0000 mgr.y (mgr.34107) 12347 : cluster [DBG] pgmap v12812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:32 smithi121 bash[36902]: audit 2022-04-22T23:28:32.023908+0000 mon.a (mon.0) 17005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:32 smithi115 bash[46934]: cluster 2022-04-22T23:28:31.462594+0000 mgr.y (mgr.34107) 12347 : cluster [DBG] pgmap v12812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:32 smithi115 bash[46934]: audit 2022-04-22T23:28:32.023908+0000 mon.a (mon.0) 17005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:33.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:33.045Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:32 smithi115 bash[45495]: cluster 2022-04-22T23:28:31.462594+0000 mgr.y (mgr.34107) 12347 : cluster [DBG] pgmap v12812: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:33.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:32 smithi115 bash[45495]: audit 2022-04-22T23:28:32.023908+0000 mon.a (mon.0) 17005 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:33 smithi121 bash[36902]: audit 2022-04-22T23:28:33.663523+0000 mon.a (mon.0) 17006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:33 smithi115 bash[45495]: audit 2022-04-22T23:28:33.663523+0000 mon.a (mon.0) 17006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:34.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:33 smithi115 bash[46934]: audit 2022-04-22T23:28:33.663523+0000 mon.a (mon.0) 17006 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:34 smithi121 bash[36902]: cluster 2022-04-22T23:28:33.463781+0000 mgr.y (mgr.34107) 12348 : cluster [DBG] pgmap v12813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:34.906Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:34 smithi115 bash[45495]: cluster 2022-04-22T23:28:33.463781+0000 mgr.y (mgr.34107) 12348 : cluster [DBG] pgmap v12813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:34 smithi115 bash[46934]: cluster 2022-04-22T23:28:33.463781+0000 mgr.y (mgr.34107) 12348 : cluster [DBG] pgmap v12813: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:35 smithi121 bash[36902]: audit 2022-04-22T23:28:35.427964+0000 mon.a (mon.0) 17007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:35 smithi115 bash[45495]: audit 2022-04-22T23:28:35.427964+0000 mon.a (mon.0) 17007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:35 smithi115 bash[46934]: audit 2022-04-22T23:28:35.427964+0000 mon.a (mon.0) 17007 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:36 smithi121 bash[36902]: cluster 2022-04-22T23:28:35.464861+0000 mgr.y (mgr.34107) 12349 : cluster [DBG] pgmap v12814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:36 smithi115 bash[45495]: cluster 2022-04-22T23:28:35.464861+0000 mgr.y (mgr.34107) 12349 : cluster [DBG] pgmap v12814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:36 smithi115 bash[46934]: cluster 2022-04-22T23:28:35.464861+0000 mgr.y (mgr.34107) 12349 : cluster [DBG] pgmap v12814: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:37 smithi121 bash[36902]: audit 2022-04-22T23:28:37.027021+0000 mon.a (mon.0) 17008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:38.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:37 smithi115 bash[45495]: audit 2022-04-22T23:28:37.027021+0000 mon.a (mon.0) 17008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:37 smithi115 bash[46934]: audit 2022-04-22T23:28:37.027021+0000 mon.a (mon.0) 17008 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:38 smithi121 bash[36902]: cluster 2022-04-22T23:28:37.466091+0000 mgr.y (mgr.34107) 12350 : cluster [DBG] pgmap v12815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:38 smithi121 bash[36902]: audit 2022-04-22T23:28:38.666681+0000 mon.a (mon.0) 17009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:38 smithi115 bash[45495]: cluster 2022-04-22T23:28:37.466091+0000 mgr.y (mgr.34107) 12350 : cluster [DBG] pgmap v12815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:38 smithi115 bash[45495]: audit 2022-04-22T23:28:38.666681+0000 mon.a (mon.0) 17009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:38 smithi115 bash[46934]: cluster 2022-04-22T23:28:37.466091+0000 mgr.y (mgr.34107) 12350 : cluster [DBG] pgmap v12815: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:38 smithi115 bash[46934]: audit 2022-04-22T23:28:38.666681+0000 mon.a (mon.0) 17009 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:40 smithi121 bash[36902]: cluster 2022-04-22T23:28:39.467206+0000 mgr.y (mgr.34107) 12351 : cluster [DBG] pgmap v12816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:40 smithi121 bash[36902]: audit 2022-04-22T23:28:40.431746+0000 mon.a (mon.0) 17010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:40 smithi115 bash[45495]: cluster 2022-04-22T23:28:39.467206+0000 mgr.y (mgr.34107) 12351 : cluster [DBG] pgmap v12816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:40 smithi115 bash[45495]: audit 2022-04-22T23:28:40.431746+0000 mon.a (mon.0) 17010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:40 smithi115 bash[46934]: cluster 2022-04-22T23:28:39.467206+0000 mgr.y (mgr.34107) 12351 : cluster [DBG] pgmap v12816: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:40 smithi115 bash[46934]: audit 2022-04-22T23:28:40.431746+0000 mon.a (mon.0) 17010 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:42 smithi121 bash[36902]: cluster 2022-04-22T23:28:41.468327+0000 mgr.y (mgr.34107) 12352 : cluster [DBG] pgmap v12817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:43.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:42 smithi121 bash[36902]: audit 2022-04-22T23:28:42.030164+0000 mon.a (mon.0) 17011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:43.046Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:42 smithi115 bash[45495]: cluster 2022-04-22T23:28:41.468327+0000 mgr.y (mgr.34107) 12352 : cluster [DBG] pgmap v12817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:42 smithi115 bash[45495]: audit 2022-04-22T23:28:42.030164+0000 mon.a (mon.0) 17011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:42 smithi115 bash[46934]: cluster 2022-04-22T23:28:41.468327+0000 mgr.y (mgr.34107) 12352 : cluster [DBG] pgmap v12817: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:42 smithi115 bash[46934]: audit 2022-04-22T23:28:42.030164+0000 mon.a (mon.0) 17011 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:43 smithi121 bash[36902]: audit 2022-04-22T23:28:43.669923+0000 mon.a (mon.0) 17012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:43 smithi115 bash[45495]: audit 2022-04-22T23:28:43.669923+0000 mon.a (mon.0) 17012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:43 smithi115 bash[46934]: audit 2022-04-22T23:28:43.669923+0000 mon.a (mon.0) 17012 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:45.195 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:28:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:44 smithi121 bash[36902]: cluster 2022-04-22T23:28:43.469489+0000 mgr.y (mgr.34107) 12353 : cluster [DBG] pgmap v12818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:45.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:44.905Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:44 smithi115 bash[45495]: cluster 2022-04-22T23:28:43.469489+0000 mgr.y (mgr.34107) 12353 : cluster [DBG] pgmap v12818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:44 smithi115 bash[46934]: cluster 2022-04-22T23:28:43.469489+0000 mgr.y (mgr.34107) 12353 : cluster [DBG] pgmap v12818: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:45.534 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 37s ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 101s ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 37s ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 101s ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 101s ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 37s ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 37s ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (5h) 101s ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:28:45.535 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 37s ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 37s ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 101s ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 37s ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 37s ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 37s ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 37s ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 101s ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 101s ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 101s ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:28:45.536 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 101s ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:28:45.537 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 101s ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:28:45.946 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:28:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:28:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:28:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:28:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:28:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:28:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:28:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:28:45.948 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:28:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:45 smithi121 bash[36902]: audit 2022-04-22T23:28:45.175862+0000 mgr.y (mgr.34107) 12354 : audit [DBG] from='client.58063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:45 smithi121 bash[36902]: audit 2022-04-22T23:28:45.435196+0000 mon.a (mon.0) 17013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:45 smithi115 bash[45495]: audit 2022-04-22T23:28:45.175862+0000 mgr.y (mgr.34107) 12354 : audit [DBG] from='client.58063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:45 smithi115 bash[45495]: audit 2022-04-22T23:28:45.435196+0000 mon.a (mon.0) 17013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:45 smithi115 bash[46934]: audit 2022-04-22T23:28:45.175862+0000 mgr.y (mgr.34107) 12354 : audit [DBG] from='client.58063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:45 smithi115 bash[46934]: audit 2022-04-22T23:28:45.435196+0000 mon.a (mon.0) 17013 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:46 smithi121 bash[36902]: cluster 2022-04-22T23:28:45.470370+0000 mgr.y (mgr.34107) 12355 : cluster [DBG] pgmap v12819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:46 smithi121 bash[36902]: audit 2022-04-22T23:28:45.517647+0000 mgr.y (mgr.34107) 12356 : audit [DBG] from='client.48129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:46 smithi121 bash[36902]: audit 2022-04-22T23:28:45.942537+0000 mon.a (mon.0) 17014 : audit [DBG] from='client.? 172.21.15.115:0/1129025577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[45495]: cluster 2022-04-22T23:28:45.470370+0000 mgr.y (mgr.34107) 12355 : cluster [DBG] pgmap v12819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[45495]: audit 2022-04-22T23:28:45.517647+0000 mgr.y (mgr.34107) 12356 : audit [DBG] from='client.48129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[45495]: audit 2022-04-22T23:28:45.942537+0000 mon.a (mon.0) 17014 : audit [DBG] from='client.? 172.21.15.115:0/1129025577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[46934]: cluster 2022-04-22T23:28:45.470370+0000 mgr.y (mgr.34107) 12355 : cluster [DBG] pgmap v12819: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[46934]: audit 2022-04-22T23:28:45.517647+0000 mgr.y (mgr.34107) 12356 : audit [DBG] from='client.48129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:28:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:46 smithi115 bash[46934]: audit 2022-04-22T23:28:45.942537+0000 mon.a (mon.0) 17014 : audit [DBG] from='client.? 172.21.15.115:0/1129025577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:28:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:47 smithi121 bash[36902]: audit 2022-04-22T23:28:47.033408+0000 mon.a (mon.0) 17015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:47 smithi115 bash[45495]: audit 2022-04-22T23:28:47.033408+0000 mon.a (mon.0) 17015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:47 smithi115 bash[46934]: audit 2022-04-22T23:28:47.033408+0000 mon.a (mon.0) 17015 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:48 smithi121 bash[36902]: cluster 2022-04-22T23:28:47.471428+0000 mgr.y (mgr.34107) 12357 : cluster [DBG] pgmap v12820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:48 smithi121 bash[36902]: audit 2022-04-22T23:28:48.673048+0000 mon.a (mon.0) 17016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:48 smithi115 bash[45495]: cluster 2022-04-22T23:28:47.471428+0000 mgr.y (mgr.34107) 12357 : cluster [DBG] pgmap v12820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:48 smithi115 bash[45495]: audit 2022-04-22T23:28:48.673048+0000 mon.a (mon.0) 17016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:48 smithi115 bash[46934]: cluster 2022-04-22T23:28:47.471428+0000 mgr.y (mgr.34107) 12357 : cluster [DBG] pgmap v12820: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:48 smithi115 bash[46934]: audit 2022-04-22T23:28:48.673048+0000 mon.a (mon.0) 17016 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:50 smithi121 bash[36902]: cluster 2022-04-22T23:28:49.472454+0000 mgr.y (mgr.34107) 12358 : cluster [DBG] pgmap v12821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:50 smithi121 bash[36902]: audit 2022-04-22T23:28:50.438823+0000 mon.a (mon.0) 17017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:50 smithi115 bash[45495]: cluster 2022-04-22T23:28:49.472454+0000 mgr.y (mgr.34107) 12358 : cluster [DBG] pgmap v12821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:50 smithi115 bash[45495]: audit 2022-04-22T23:28:50.438823+0000 mon.a (mon.0) 17017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:50 smithi115 bash[46934]: cluster 2022-04-22T23:28:49.472454+0000 mgr.y (mgr.34107) 12358 : cluster [DBG] pgmap v12821: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:50 smithi115 bash[46934]: audit 2022-04-22T23:28:50.438823+0000 mon.a (mon.0) 17017 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:52 smithi121 bash[36902]: cluster 2022-04-22T23:28:51.473676+0000 mgr.y (mgr.34107) 12359 : cluster [DBG] pgmap v12822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:52 smithi121 bash[36902]: audit 2022-04-22T23:28:52.036375+0000 mon.a (mon.0) 17018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:53.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:52 smithi115 bash[45495]: cluster 2022-04-22T23:28:51.473676+0000 mgr.y (mgr.34107) 12359 : cluster [DBG] pgmap v12822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:53.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:52 smithi115 bash[45495]: audit 2022-04-22T23:28:52.036375+0000 mon.a (mon.0) 17018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:52 smithi115 bash[46934]: cluster 2022-04-22T23:28:51.473676+0000 mgr.y (mgr.34107) 12359 : cluster [DBG] pgmap v12822: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:52 smithi115 bash[46934]: audit 2022-04-22T23:28:52.036375+0000 mon.a (mon.0) 17018 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:53 smithi121 bash[36902]: audit 2022-04-22T23:28:53.676085+0000 mon.a (mon.0) 17019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:53 smithi115 bash[45495]: audit 2022-04-22T23:28:53.676085+0000 mon.a (mon.0) 17019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:53 smithi115 bash[46934]: audit 2022-04-22T23:28:53.676085+0000 mon.a (mon.0) 17019 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:54 smithi121 bash[36902]: cluster 2022-04-22T23:28:53.474873+0000 mgr.y (mgr.34107) 12360 : cluster [DBG] pgmap v12823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:54 smithi115 bash[45495]: cluster 2022-04-22T23:28:53.474873+0000 mgr.y (mgr.34107) 12360 : cluster [DBG] pgmap v12823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:28:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:28:54.908Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:28:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:54 smithi115 bash[46934]: cluster 2022-04-22T23:28:53.474873+0000 mgr.y (mgr.34107) 12360 : cluster [DBG] pgmap v12823: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:55 smithi121 bash[36902]: audit 2022-04-22T23:28:55.442630+0000 mon.a (mon.0) 17020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:55 smithi115 bash[45495]: audit 2022-04-22T23:28:55.442630+0000 mon.a (mon.0) 17020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:55 smithi115 bash[46934]: audit 2022-04-22T23:28:55.442630+0000 mon.a (mon.0) 17020 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:28:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:56 smithi121 bash[36902]: cluster 2022-04-22T23:28:55.475969+0000 mgr.y (mgr.34107) 12361 : cluster [DBG] pgmap v12824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:56 smithi115 bash[45495]: cluster 2022-04-22T23:28:55.475969+0000 mgr.y (mgr.34107) 12361 : cluster [DBG] pgmap v12824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:56 smithi115 bash[46934]: cluster 2022-04-22T23:28:55.475969+0000 mgr.y (mgr.34107) 12361 : cluster [DBG] pgmap v12824: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:57 smithi121 bash[36902]: audit 2022-04-22T23:28:57.039422+0000 mon.a (mon.0) 17021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:57 smithi115 bash[45495]: audit 2022-04-22T23:28:57.039422+0000 mon.a (mon.0) 17021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:57 smithi115 bash[46934]: audit 2022-04-22T23:28:57.039422+0000 mon.a (mon.0) 17021 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:28:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:58 smithi121 bash[36902]: cluster 2022-04-22T23:28:57.477032+0000 mgr.y (mgr.34107) 12362 : cluster [DBG] pgmap v12825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:28:58 smithi121 bash[36902]: audit 2022-04-22T23:28:58.679457+0000 mon.a (mon.0) 17022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:58 smithi115 bash[45495]: cluster 2022-04-22T23:28:57.477032+0000 mgr.y (mgr.34107) 12362 : cluster [DBG] pgmap v12825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:28:58 smithi115 bash[45495]: audit 2022-04-22T23:28:58.679457+0000 mon.a (mon.0) 17022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:28:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:58 smithi115 bash[46934]: cluster 2022-04-22T23:28:57.477032+0000 mgr.y (mgr.34107) 12362 : cluster [DBG] pgmap v12825: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:28:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:28:58 smithi115 bash[46934]: audit 2022-04-22T23:28:58.679457+0000 mon.a (mon.0) 17022 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:00 smithi121 bash[36902]: cluster 2022-04-22T23:28:59.478051+0000 mgr.y (mgr.34107) 12363 : cluster [DBG] pgmap v12826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:00 smithi121 bash[36902]: audit 2022-04-22T23:29:00.446049+0000 mon.a (mon.0) 17023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:00 smithi115 bash[45495]: cluster 2022-04-22T23:28:59.478051+0000 mgr.y (mgr.34107) 12363 : cluster [DBG] pgmap v12826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:00 smithi115 bash[45495]: audit 2022-04-22T23:29:00.446049+0000 mon.a (mon.0) 17023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:00 smithi115 bash[46934]: cluster 2022-04-22T23:28:59.478051+0000 mgr.y (mgr.34107) 12363 : cluster [DBG] pgmap v12826: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:00 smithi115 bash[46934]: audit 2022-04-22T23:29:00.446049+0000 mon.a (mon.0) 17023 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:02 smithi121 bash[36902]: cluster 2022-04-22T23:29:01.479159+0000 mgr.y (mgr.34107) 12364 : cluster [DBG] pgmap v12827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:02 smithi121 bash[36902]: audit 2022-04-22T23:29:02.042455+0000 mon.a (mon.0) 17024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:03.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:02 smithi115 bash[45495]: cluster 2022-04-22T23:29:01.479159+0000 mgr.y (mgr.34107) 12364 : cluster [DBG] pgmap v12827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:02 smithi115 bash[45495]: audit 2022-04-22T23:29:02.042455+0000 mon.a (mon.0) 17024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:02 smithi115 bash[46934]: cluster 2022-04-22T23:29:01.479159+0000 mgr.y (mgr.34107) 12364 : cluster [DBG] pgmap v12827: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:02 smithi115 bash[46934]: audit 2022-04-22T23:29:02.042455+0000 mon.a (mon.0) 17024 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:03 smithi121 bash[36902]: audit 2022-04-22T23:29:03.682509+0000 mon.a (mon.0) 17025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:03 smithi115 bash[45495]: audit 2022-04-22T23:29:03.682509+0000 mon.a (mon.0) 17025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:03 smithi115 bash[46934]: audit 2022-04-22T23:29:03.682509+0000 mon.a (mon.0) 17025 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:04 smithi121 bash[36902]: cluster 2022-04-22T23:29:03.480335+0000 mgr.y (mgr.34107) 12365 : cluster [DBG] pgmap v12828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:04 smithi115 bash[45495]: cluster 2022-04-22T23:29:03.480335+0000 mgr.y (mgr.34107) 12365 : cluster [DBG] pgmap v12828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:04.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:04 smithi115 bash[46934]: cluster 2022-04-22T23:29:03.480335+0000 mgr.y (mgr.34107) 12365 : cluster [DBG] pgmap v12828: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:05 smithi121 bash[36902]: audit 2022-04-22T23:29:05.449623+0000 mon.a (mon.0) 17026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:05 smithi115 bash[45495]: audit 2022-04-22T23:29:05.449623+0000 mon.a (mon.0) 17026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:05 smithi115 bash[46934]: audit 2022-04-22T23:29:05.449623+0000 mon.a (mon.0) 17026 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:06 smithi121 bash[36902]: cluster 2022-04-22T23:29:05.481408+0000 mgr.y (mgr.34107) 12366 : cluster [DBG] pgmap v12829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:07.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:06 smithi115 bash[45495]: cluster 2022-04-22T23:29:05.481408+0000 mgr.y (mgr.34107) 12366 : cluster [DBG] pgmap v12829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:06 smithi115 bash[46934]: cluster 2022-04-22T23:29:05.481408+0000 mgr.y (mgr.34107) 12366 : cluster [DBG] pgmap v12829: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:07 smithi121 bash[36902]: audit 2022-04-22T23:29:07.045430+0000 mon.a (mon.0) 17027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:07 smithi115 bash[45495]: audit 2022-04-22T23:29:07.045430+0000 mon.a (mon.0) 17027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:07 smithi115 bash[46934]: audit 2022-04-22T23:29:07.045430+0000 mon.a (mon.0) 17027 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:09.216 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[45495]: cluster 2022-04-22T23:29:07.482498+0000 mgr.y (mgr.34107) 12367 : cluster [DBG] pgmap v12830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[45495]: audit 2022-04-22T23:29:08.685698+0000 mon.a (mon.0) 17028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[45495]: audit 2022-04-22T23:29:08.887306+0000 mon.a (mon.0) 17029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[45495]: audit 2022-04-22T23:29:08.890401+0000 mon.a (mon.0) 17030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[45495]: audit 2022-04-22T23:29:08.892234+0000 mon.a (mon.0) 17031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[46934]: cluster 2022-04-22T23:29:07.482498+0000 mgr.y (mgr.34107) 12367 : cluster [DBG] pgmap v12830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:09.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[46934]: audit 2022-04-22T23:29:08.685698+0000 mon.a (mon.0) 17028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:09.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[46934]: audit 2022-04-22T23:29:08.887306+0000 mon.a (mon.0) 17029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:29:09.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[46934]: audit 2022-04-22T23:29:08.890401+0000 mon.a (mon.0) 17030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:29:09.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:08 smithi115 bash[46934]: audit 2022-04-22T23:29:08.892234+0000 mon.a (mon.0) 17031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:29:09.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:08 smithi121 bash[36902]: cluster 2022-04-22T23:29:07.482498+0000 mgr.y (mgr.34107) 12367 : cluster [DBG] pgmap v12830: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:09.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:08 smithi121 bash[36902]: audit 2022-04-22T23:29:08.685698+0000 mon.a (mon.0) 17028 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:09.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:08 smithi121 bash[36902]: audit 2022-04-22T23:29:08.887306+0000 mon.a (mon.0) 17029 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:29:09.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:08 smithi121 bash[36902]: audit 2022-04-22T23:29:08.890401+0000 mon.a (mon.0) 17030 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:29:09.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:08 smithi121 bash[36902]: audit 2022-04-22T23:29:08.892234+0000 mon.a (mon.0) 17031 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:29:10.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:10 smithi115 bash[45495]: audit 2022-04-22T23:29:09.257297+0000 mon.a (mon.0) 17032 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:29:10.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:10 smithi115 bash[46934]: audit 2022-04-22T23:29:09.257297+0000 mon.a (mon.0) 17032 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:29:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:10 smithi121 bash[36902]: audit 2022-04-22T23:29:09.257297+0000 mon.a (mon.0) 17032 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:29:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:11 smithi115 bash[45495]: cluster 2022-04-22T23:29:09.483728+0000 mgr.y (mgr.34107) 12368 : cluster [DBG] pgmap v12831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:11 smithi115 bash[45495]: audit 2022-04-22T23:29:10.452931+0000 mon.a (mon.0) 17033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:11 smithi115 bash[46934]: cluster 2022-04-22T23:29:09.483728+0000 mgr.y (mgr.34107) 12368 : cluster [DBG] pgmap v12831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:11 smithi115 bash[46934]: audit 2022-04-22T23:29:10.452931+0000 mon.a (mon.0) 17033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:11 smithi121 bash[36902]: cluster 2022-04-22T23:29:09.483728+0000 mgr.y (mgr.34107) 12368 : cluster [DBG] pgmap v12831: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:11 smithi121 bash[36902]: audit 2022-04-22T23:29:10.452931+0000 mon.a (mon.0) 17033 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:12 smithi115 bash[45495]: audit 2022-04-22T23:29:12.048479+0000 mon.a (mon.0) 17034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:12 smithi115 bash[46934]: audit 2022-04-22T23:29:12.048479+0000 mon.a (mon.0) 17034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:12 smithi121 bash[36902]: audit 2022-04-22T23:29:12.048479+0000 mon.a (mon.0) 17034 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:13 smithi115 bash[45495]: cluster 2022-04-22T23:29:11.484862+0000 mgr.y (mgr.34107) 12369 : cluster [DBG] pgmap v12832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:13.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:13.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:13 smithi115 bash[46934]: cluster 2022-04-22T23:29:11.484862+0000 mgr.y (mgr.34107) 12369 : cluster [DBG] pgmap v12832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:13 smithi121 bash[36902]: cluster 2022-04-22T23:29:11.484862+0000 mgr.y (mgr.34107) 12369 : cluster [DBG] pgmap v12832: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:14 smithi115 bash[45495]: audit 2022-04-22T23:29:13.688723+0000 mon.a (mon.0) 17035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:14 smithi115 bash[46934]: audit 2022-04-22T23:29:13.688723+0000 mon.a (mon.0) 17035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:14 smithi121 bash[36902]: audit 2022-04-22T23:29:13.688723+0000 mon.a (mon.0) 17035 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:14.907Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:15 smithi121 bash[36902]: cluster 2022-04-22T23:29:13.486002+0000 mgr.y (mgr.34107) 12370 : cluster [DBG] pgmap v12833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:15 smithi115 bash[45495]: cluster 2022-04-22T23:29:13.486002+0000 mgr.y (mgr.34107) 12370 : cluster [DBG] pgmap v12833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:15 smithi115 bash[46934]: cluster 2022-04-22T23:29:13.486002+0000 mgr.y (mgr.34107) 12370 : cluster [DBG] pgmap v12833: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:16.328 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 68s ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 2m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 68s ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:29:16.674 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 2m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 68s ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 68s ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 2m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 68s ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 68s ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 68s ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 68s ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 68s ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:29:16.675 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 68s ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:29:16.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:29:16.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:29:16.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:29:16.676 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:29:16.676 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:29:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:16 smithi121 bash[36902]: audit 2022-04-22T23:29:15.456615+0000 mon.a (mon.0) 17036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:16 smithi115 bash[46934]: audit 2022-04-22T23:29:15.456615+0000 mon.a (mon.0) 17036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:16 smithi115 bash[45495]: audit 2022-04-22T23:29:15.456615+0000 mon.a (mon.0) 17036 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:17.071 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:29:17.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:29:17.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:29:17.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:17.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:29:17.072 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:29:17.073 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:29:17.073 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:29:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:17 smithi121 bash[36902]: cluster 2022-04-22T23:29:15.487034+0000 mgr.y (mgr.34107) 12371 : cluster [DBG] pgmap v12834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:17 smithi121 bash[36902]: audit 2022-04-22T23:29:16.310572+0000 mgr.y (mgr.34107) 12372 : audit [DBG] from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:17 smithi121 bash[36902]: audit 2022-04-22T23:29:17.051569+0000 mon.a (mon.0) 17037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:17.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:17 smithi121 bash[36902]: audit 2022-04-22T23:29:17.066921+0000 mon.c (mon.1) 418 : audit [DBG] from='client.? 172.21.15.115:0/3112360846' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[46934]: cluster 2022-04-22T23:29:15.487034+0000 mgr.y (mgr.34107) 12371 : cluster [DBG] pgmap v12834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[46934]: audit 2022-04-22T23:29:16.310572+0000 mgr.y (mgr.34107) 12372 : audit [DBG] from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[46934]: audit 2022-04-22T23:29:17.051569+0000 mon.a (mon.0) 17037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:17.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[46934]: audit 2022-04-22T23:29:17.066921+0000 mon.c (mon.1) 418 : audit [DBG] from='client.? 172.21.15.115:0/3112360846' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[45495]: cluster 2022-04-22T23:29:15.487034+0000 mgr.y (mgr.34107) 12371 : cluster [DBG] pgmap v12834: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:17.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[45495]: audit 2022-04-22T23:29:16.310572+0000 mgr.y (mgr.34107) 12372 : audit [DBG] from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[45495]: audit 2022-04-22T23:29:17.051569+0000 mon.a (mon.0) 17037 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:17.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:17 smithi115 bash[45495]: audit 2022-04-22T23:29:17.066921+0000 mon.c (mon.1) 418 : audit [DBG] from='client.? 172.21.15.115:0/3112360846' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:18 smithi121 bash[36902]: audit 2022-04-22T23:29:16.650913+0000 mgr.y (mgr.34107) 12373 : audit [DBG] from='client.58087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:18 smithi121 bash[36902]: audit 2022-04-22T23:29:17.461115+0000 mon.a (mon.0) 17038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:29:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:18 smithi121 bash[36902]: audit 2022-04-22T23:29:17.461728+0000 mon.a (mon.0) 17039 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[45495]: audit 2022-04-22T23:29:16.650913+0000 mgr.y (mgr.34107) 12373 : audit [DBG] from='client.58087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[45495]: audit 2022-04-22T23:29:17.461115+0000 mon.a (mon.0) 17038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:29:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[45495]: audit 2022-04-22T23:29:17.461728+0000 mon.a (mon.0) 17039 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:29:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[46934]: audit 2022-04-22T23:29:16.650913+0000 mgr.y (mgr.34107) 12373 : audit [DBG] from='client.58087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[46934]: audit 2022-04-22T23:29:17.461115+0000 mon.a (mon.0) 17038 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:29:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:18 smithi115 bash[46934]: audit 2022-04-22T23:29:17.461728+0000 mon.a (mon.0) 17039 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:29:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:19 smithi121 bash[36902]: cluster 2022-04-22T23:29:17.488137+0000 mgr.y (mgr.34107) 12374 : cluster [DBG] pgmap v12835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:19 smithi121 bash[36902]: audit 2022-04-22T23:29:18.691894+0000 mon.a (mon.0) 17040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:19 smithi115 bash[45495]: cluster 2022-04-22T23:29:17.488137+0000 mgr.y (mgr.34107) 12374 : cluster [DBG] pgmap v12835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:19 smithi115 bash[45495]: audit 2022-04-22T23:29:18.691894+0000 mon.a (mon.0) 17040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:19 smithi115 bash[46934]: cluster 2022-04-22T23:29:17.488137+0000 mgr.y (mgr.34107) 12374 : cluster [DBG] pgmap v12835: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:19 smithi115 bash[46934]: audit 2022-04-22T23:29:18.691894+0000 mon.a (mon.0) 17040 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:21 smithi121 bash[36902]: cluster 2022-04-22T23:29:19.489238+0000 mgr.y (mgr.34107) 12375 : cluster [DBG] pgmap v12836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:21 smithi121 bash[36902]: audit 2022-04-22T23:29:20.460219+0000 mon.a (mon.0) 17041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:21 smithi115 bash[45495]: cluster 2022-04-22T23:29:19.489238+0000 mgr.y (mgr.34107) 12375 : cluster [DBG] pgmap v12836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:21 smithi115 bash[45495]: audit 2022-04-22T23:29:20.460219+0000 mon.a (mon.0) 17041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:21 smithi115 bash[46934]: cluster 2022-04-22T23:29:19.489238+0000 mgr.y (mgr.34107) 12375 : cluster [DBG] pgmap v12836: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:21 smithi115 bash[46934]: audit 2022-04-22T23:29:20.460219+0000 mon.a (mon.0) 17041 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:22 smithi121 bash[36902]: audit 2022-04-22T23:29:22.054372+0000 mon.a (mon.0) 17042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:22 smithi115 bash[46934]: audit 2022-04-22T23:29:22.054372+0000 mon.a (mon.0) 17042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:22 smithi115 bash[45495]: audit 2022-04-22T23:29:22.054372+0000 mon.a (mon.0) 17042 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:23.327 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:23.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:23 smithi121 bash[36902]: cluster 2022-04-22T23:29:21.490393+0000 mgr.y (mgr.34107) 12376 : cluster [DBG] pgmap v12837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:23 smithi115 bash[45495]: cluster 2022-04-22T23:29:21.490393+0000 mgr.y (mgr.34107) 12376 : cluster [DBG] pgmap v12837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:23 smithi115 bash[46934]: cluster 2022-04-22T23:29:21.490393+0000 mgr.y (mgr.34107) 12376 : cluster [DBG] pgmap v12837: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:24.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:24 smithi121 bash[36902]: audit 2022-04-22T23:29:23.694961+0000 mon.a (mon.0) 17043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:24 smithi115 bash[45495]: audit 2022-04-22T23:29:23.694961+0000 mon.a (mon.0) 17043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:24 smithi115 bash[46934]: audit 2022-04-22T23:29:23.694961+0000 mon.a (mon.0) 17043 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:24.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:25 smithi121 bash[36902]: cluster 2022-04-22T23:29:23.491587+0000 mgr.y (mgr.34107) 12377 : cluster [DBG] pgmap v12838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:25 smithi115 bash[45495]: cluster 2022-04-22T23:29:23.491587+0000 mgr.y (mgr.34107) 12377 : cluster [DBG] pgmap v12838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:25 smithi115 bash[46934]: cluster 2022-04-22T23:29:23.491587+0000 mgr.y (mgr.34107) 12377 : cluster [DBG] pgmap v12838: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:26 smithi121 bash[36902]: audit 2022-04-22T23:29:25.464022+0000 mon.a (mon.0) 17044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:26 smithi115 bash[45495]: audit 2022-04-22T23:29:25.464022+0000 mon.a (mon.0) 17044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:26 smithi115 bash[46934]: audit 2022-04-22T23:29:25.464022+0000 mon.a (mon.0) 17044 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:27 smithi121 bash[36902]: cluster 2022-04-22T23:29:25.492570+0000 mgr.y (mgr.34107) 12378 : cluster [DBG] pgmap v12839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:27 smithi121 bash[36902]: audit 2022-04-22T23:29:27.057362+0000 mon.a (mon.0) 17045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:27 smithi115 bash[46934]: cluster 2022-04-22T23:29:25.492570+0000 mgr.y (mgr.34107) 12378 : cluster [DBG] pgmap v12839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:27 smithi115 bash[46934]: audit 2022-04-22T23:29:27.057362+0000 mon.a (mon.0) 17045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:27 smithi115 bash[45495]: cluster 2022-04-22T23:29:25.492570+0000 mgr.y (mgr.34107) 12378 : cluster [DBG] pgmap v12839: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:27.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:27 smithi115 bash[45495]: audit 2022-04-22T23:29:27.057362+0000 mon.a (mon.0) 17045 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:29 smithi121 bash[36902]: cluster 2022-04-22T23:29:27.493781+0000 mgr.y (mgr.34107) 12379 : cluster [DBG] pgmap v12840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:29.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:29 smithi121 bash[36902]: audit 2022-04-22T23:29:28.698289+0000 mon.a (mon.0) 17046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:29 smithi115 bash[45495]: cluster 2022-04-22T23:29:27.493781+0000 mgr.y (mgr.34107) 12379 : cluster [DBG] pgmap v12840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:29 smithi115 bash[45495]: audit 2022-04-22T23:29:28.698289+0000 mon.a (mon.0) 17046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:29 smithi115 bash[46934]: cluster 2022-04-22T23:29:27.493781+0000 mgr.y (mgr.34107) 12379 : cluster [DBG] pgmap v12840: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:29 smithi115 bash[46934]: audit 2022-04-22T23:29:28.698289+0000 mon.a (mon.0) 17046 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:31 smithi121 bash[36902]: cluster 2022-04-22T23:29:29.494978+0000 mgr.y (mgr.34107) 12380 : cluster [DBG] pgmap v12841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:31 smithi121 bash[36902]: audit 2022-04-22T23:29:30.467809+0000 mon.a (mon.0) 17047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:31 smithi115 bash[45495]: cluster 2022-04-22T23:29:29.494978+0000 mgr.y (mgr.34107) 12380 : cluster [DBG] pgmap v12841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:31 smithi115 bash[45495]: audit 2022-04-22T23:29:30.467809+0000 mon.a (mon.0) 17047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:31 smithi115 bash[46934]: cluster 2022-04-22T23:29:29.494978+0000 mgr.y (mgr.34107) 12380 : cluster [DBG] pgmap v12841: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:31 smithi115 bash[46934]: audit 2022-04-22T23:29:30.467809+0000 mon.a (mon.0) 17047 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:32 smithi121 bash[36902]: audit 2022-04-22T23:29:32.060460+0000 mon.a (mon.0) 17048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:32 smithi115 bash[45495]: audit 2022-04-22T23:29:32.060460+0000 mon.a (mon.0) 17048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:32 smithi115 bash[46934]: audit 2022-04-22T23:29:32.060460+0000 mon.a (mon.0) 17048 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:33.372 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:33.048Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:33 smithi121 bash[36902]: cluster 2022-04-22T23:29:31.496178+0000 mgr.y (mgr.34107) 12381 : cluster [DBG] pgmap v12842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:33 smithi115 bash[45495]: cluster 2022-04-22T23:29:31.496178+0000 mgr.y (mgr.34107) 12381 : cluster [DBG] pgmap v12842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:33 smithi115 bash[46934]: cluster 2022-04-22T23:29:31.496178+0000 mgr.y (mgr.34107) 12381 : cluster [DBG] pgmap v12842: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:34 smithi121 bash[36902]: audit 2022-04-22T23:29:33.701529+0000 mon.a (mon.0) 17049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:34.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:34 smithi115 bash[45495]: audit 2022-04-22T23:29:33.701529+0000 mon.a (mon.0) 17049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:34 smithi115 bash[46934]: audit 2022-04-22T23:29:33.701529+0000 mon.a (mon.0) 17049 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:34.909Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:35 smithi121 bash[36902]: cluster 2022-04-22T23:29:33.497374+0000 mgr.y (mgr.34107) 12382 : cluster [DBG] pgmap v12843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:35 smithi115 bash[45495]: cluster 2022-04-22T23:29:33.497374+0000 mgr.y (mgr.34107) 12382 : cluster [DBG] pgmap v12843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:35 smithi115 bash[46934]: cluster 2022-04-22T23:29:33.497374+0000 mgr.y (mgr.34107) 12382 : cluster [DBG] pgmap v12843: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:36 smithi121 bash[36902]: audit 2022-04-22T23:29:35.471658+0000 mon.a (mon.0) 17050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:36 smithi115 bash[45495]: audit 2022-04-22T23:29:35.471658+0000 mon.a (mon.0) 17050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:36 smithi115 bash[46934]: audit 2022-04-22T23:29:35.471658+0000 mon.a (mon.0) 17050 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:37 smithi121 bash[36902]: cluster 2022-04-22T23:29:35.498448+0000 mgr.y (mgr.34107) 12383 : cluster [DBG] pgmap v12844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:37 smithi121 bash[36902]: audit 2022-04-22T23:29:37.063623+0000 mon.a (mon.0) 17051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:37 smithi115 bash[46934]: cluster 2022-04-22T23:29:35.498448+0000 mgr.y (mgr.34107) 12383 : cluster [DBG] pgmap v12844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:37 smithi115 bash[46934]: audit 2022-04-22T23:29:37.063623+0000 mon.a (mon.0) 17051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:37 smithi115 bash[45495]: cluster 2022-04-22T23:29:35.498448+0000 mgr.y (mgr.34107) 12383 : cluster [DBG] pgmap v12844: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:37 smithi115 bash[45495]: audit 2022-04-22T23:29:37.063623+0000 mon.a (mon.0) 17051 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:39 smithi121 bash[36902]: cluster 2022-04-22T23:29:37.499569+0000 mgr.y (mgr.34107) 12384 : cluster [DBG] pgmap v12845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:39 smithi121 bash[36902]: audit 2022-04-22T23:29:38.704713+0000 mon.a (mon.0) 17052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:39 smithi115 bash[45495]: cluster 2022-04-22T23:29:37.499569+0000 mgr.y (mgr.34107) 12384 : cluster [DBG] pgmap v12845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:39 smithi115 bash[45495]: audit 2022-04-22T23:29:38.704713+0000 mon.a (mon.0) 17052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:39 smithi115 bash[46934]: cluster 2022-04-22T23:29:37.499569+0000 mgr.y (mgr.34107) 12384 : cluster [DBG] pgmap v12845: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:39.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:39 smithi115 bash[46934]: audit 2022-04-22T23:29:38.704713+0000 mon.a (mon.0) 17052 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:41 smithi121 bash[36902]: cluster 2022-04-22T23:29:39.500687+0000 mgr.y (mgr.34107) 12385 : cluster [DBG] pgmap v12846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:41 smithi121 bash[36902]: audit 2022-04-22T23:29:40.475288+0000 mon.a (mon.0) 17053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:41 smithi115 bash[45495]: cluster 2022-04-22T23:29:39.500687+0000 mgr.y (mgr.34107) 12385 : cluster [DBG] pgmap v12846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:41 smithi115 bash[45495]: audit 2022-04-22T23:29:40.475288+0000 mon.a (mon.0) 17053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:41 smithi115 bash[46934]: cluster 2022-04-22T23:29:39.500687+0000 mgr.y (mgr.34107) 12385 : cluster [DBG] pgmap v12846: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:41.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:41 smithi115 bash[46934]: audit 2022-04-22T23:29:40.475288+0000 mon.a (mon.0) 17053 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:42 smithi121 bash[36902]: audit 2022-04-22T23:29:42.066637+0000 mon.a (mon.0) 17054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:42 smithi115 bash[45495]: audit 2022-04-22T23:29:42.066637+0000 mon.a (mon.0) 17054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:42 smithi115 bash[46934]: audit 2022-04-22T23:29:42.066637+0000 mon.a (mon.0) 17054 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:43.411 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:43.049Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:43 smithi121 bash[36902]: cluster 2022-04-22T23:29:41.501771+0000 mgr.y (mgr.34107) 12386 : cluster [DBG] pgmap v12847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:43 smithi115 bash[45495]: cluster 2022-04-22T23:29:41.501771+0000 mgr.y (mgr.34107) 12386 : cluster [DBG] pgmap v12847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:43 smithi115 bash[46934]: cluster 2022-04-22T23:29:41.501771+0000 mgr.y (mgr.34107) 12386 : cluster [DBG] pgmap v12847: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:44 smithi121 bash[36902]: audit 2022-04-22T23:29:43.708151+0000 mon.a (mon.0) 17055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:44 smithi115 bash[45495]: audit 2022-04-22T23:29:43.708151+0000 mon.a (mon.0) 17055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:44 smithi115 bash[46934]: audit 2022-04-22T23:29:43.708151+0000 mon.a (mon.0) 17055 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:44.910Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:45 smithi121 bash[36902]: cluster 2022-04-22T23:29:43.503029+0000 mgr.y (mgr.34107) 12387 : cluster [DBG] pgmap v12848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:45.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:45 smithi115 bash[45495]: cluster 2022-04-22T23:29:43.503029+0000 mgr.y (mgr.34107) 12387 : cluster [DBG] pgmap v12848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:45 smithi115 bash[46934]: cluster 2022-04-22T23:29:43.503029+0000 mgr.y (mgr.34107) 12387 : cluster [DBG] pgmap v12848: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:46 smithi121 bash[36902]: audit 2022-04-22T23:29:45.479182+0000 mon.a (mon.0) 17056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:46 smithi115 bash[45495]: audit 2022-04-22T23:29:45.479182+0000 mon.a (mon.0) 17056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:46 smithi115 bash[46934]: audit 2022-04-22T23:29:45.479182+0000 mon.a (mon.0) 17056 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:47.452 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:29:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:47 smithi121 bash[36902]: cluster 2022-04-22T23:29:45.504134+0000 mgr.y (mgr.34107) 12388 : cluster [DBG] pgmap v12849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:47 smithi121 bash[36902]: audit 2022-04-22T23:29:47.069645+0000 mon.a (mon.0) 17057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:47 smithi115 bash[45495]: cluster 2022-04-22T23:29:45.504134+0000 mgr.y (mgr.34107) 12388 : cluster [DBG] pgmap v12849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:47 smithi115 bash[45495]: audit 2022-04-22T23:29:47.069645+0000 mon.a (mon.0) 17057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:47 smithi115 bash[46934]: cluster 2022-04-22T23:29:45.504134+0000 mgr.y (mgr.34107) 12388 : cluster [DBG] pgmap v12849: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:47.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:47 smithi115 bash[46934]: audit 2022-04-22T23:29:47.069645+0000 mon.a (mon.0) 17057 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:47.796 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:29:47.796 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 99s ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:29:47.796 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 2m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 99s ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 2m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 2m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 99s ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 99s ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 2m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 99s ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 99s ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:29:47.797 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 2m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5h) 99s ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 99s ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 99s ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 99s ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 2m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 2m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 2m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 2m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:29:47.798 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 2m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:29:48.197 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:29:48.198 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:29:48.199 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:29:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:48 smithi121 bash[36902]: audit 2022-04-22T23:29:47.430901+0000 mgr.y (mgr.34107) 12389 : audit [DBG] from='client.58099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:48 smithi121 bash[36902]: cluster 2022-04-22T23:29:47.505236+0000 mgr.y (mgr.34107) 12390 : cluster [DBG] pgmap v12850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:48 smithi121 bash[36902]: audit 2022-04-22T23:29:47.772222+0000 mgr.y (mgr.34107) 12391 : audit [DBG] from='client.48165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:48 smithi121 bash[36902]: audit 2022-04-22T23:29:48.193712+0000 mon.c (mon.1) 419 : audit [DBG] from='client.? 172.21.15.115:0/1690785362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:49.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:48 smithi121 bash[36902]: audit 2022-04-22T23:29:48.711274+0000 mon.a (mon.0) 17058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[45495]: audit 2022-04-22T23:29:47.430901+0000 mgr.y (mgr.34107) 12389 : audit [DBG] from='client.58099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.342 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[45495]: cluster 2022-04-22T23:29:47.505236+0000 mgr.y (mgr.34107) 12390 : cluster [DBG] pgmap v12850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:49.342 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[45495]: audit 2022-04-22T23:29:47.772222+0000 mgr.y (mgr.34107) 12391 : audit [DBG] from='client.48165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.342 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[45495]: audit 2022-04-22T23:29:48.193712+0000 mon.c (mon.1) 419 : audit [DBG] from='client.? 172.21.15.115:0/1690785362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:49.342 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[45495]: audit 2022-04-22T23:29:48.711274+0000 mon.a (mon.0) 17058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:49.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[46934]: audit 2022-04-22T23:29:47.430901+0000 mgr.y (mgr.34107) 12389 : audit [DBG] from='client.58099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[46934]: cluster 2022-04-22T23:29:47.505236+0000 mgr.y (mgr.34107) 12390 : cluster [DBG] pgmap v12850: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:49.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[46934]: audit 2022-04-22T23:29:47.772222+0000 mgr.y (mgr.34107) 12391 : audit [DBG] from='client.48165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:29:49.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[46934]: audit 2022-04-22T23:29:48.193712+0000 mon.c (mon.1) 419 : audit [DBG] from='client.? 172.21.15.115:0/1690785362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:29:49.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:48 smithi115 bash[46934]: audit 2022-04-22T23:29:48.711274+0000 mon.a (mon.0) 17058 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:50 smithi121 bash[36902]: cluster 2022-04-22T23:29:49.506376+0000 mgr.y (mgr.34107) 12392 : cluster [DBG] pgmap v12851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:50 smithi121 bash[36902]: audit 2022-04-22T23:29:50.482925+0000 mon.a (mon.0) 17059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:50 smithi115 bash[45495]: cluster 2022-04-22T23:29:49.506376+0000 mgr.y (mgr.34107) 12392 : cluster [DBG] pgmap v12851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:50 smithi115 bash[45495]: audit 2022-04-22T23:29:50.482925+0000 mon.a (mon.0) 17059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:50 smithi115 bash[46934]: cluster 2022-04-22T23:29:49.506376+0000 mgr.y (mgr.34107) 12392 : cluster [DBG] pgmap v12851: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:50 smithi115 bash[46934]: audit 2022-04-22T23:29:50.482925+0000 mon.a (mon.0) 17059 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:52 smithi121 bash[36902]: cluster 2022-04-22T23:29:51.507450+0000 mgr.y (mgr.34107) 12393 : cluster [DBG] pgmap v12852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:52 smithi121 bash[36902]: audit 2022-04-22T23:29:52.072753+0000 mon.a (mon.0) 17060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:53.049Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:52 smithi115 bash[45495]: cluster 2022-04-22T23:29:51.507450+0000 mgr.y (mgr.34107) 12393 : cluster [DBG] pgmap v12852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:52 smithi115 bash[45495]: audit 2022-04-22T23:29:52.072753+0000 mon.a (mon.0) 17060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:52 smithi115 bash[46934]: cluster 2022-04-22T23:29:51.507450+0000 mgr.y (mgr.34107) 12393 : cluster [DBG] pgmap v12852: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:52 smithi115 bash[46934]: audit 2022-04-22T23:29:52.072753+0000 mon.a (mon.0) 17060 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:53 smithi121 bash[36902]: audit 2022-04-22T23:29:53.714486+0000 mon.a (mon.0) 17061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:53 smithi115 bash[45495]: audit 2022-04-22T23:29:53.714486+0000 mon.a (mon.0) 17061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:53 smithi115 bash[46934]: audit 2022-04-22T23:29:53.714486+0000 mon.a (mon.0) 17061 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:54 smithi121 bash[36902]: cluster 2022-04-22T23:29:53.508553+0000 mgr.y (mgr.34107) 12394 : cluster [DBG] pgmap v12853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:54 smithi115 bash[46934]: cluster 2022-04-22T23:29:53.508553+0000 mgr.y (mgr.34107) 12394 : cluster [DBG] pgmap v12853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:29:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:29:54.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:29:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:54 smithi115 bash[45495]: cluster 2022-04-22T23:29:53.508553+0000 mgr.y (mgr.34107) 12394 : cluster [DBG] pgmap v12853: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:55 smithi121 bash[36902]: audit 2022-04-22T23:29:55.486882+0000 mon.a (mon.0) 17062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:55 smithi115 bash[45495]: audit 2022-04-22T23:29:55.486882+0000 mon.a (mon.0) 17062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:55 smithi115 bash[46934]: audit 2022-04-22T23:29:55.486882+0000 mon.a (mon.0) 17062 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:29:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:56 smithi121 bash[36902]: cluster 2022-04-22T23:29:55.509726+0000 mgr.y (mgr.34107) 12395 : cluster [DBG] pgmap v12854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:56 smithi115 bash[45495]: cluster 2022-04-22T23:29:55.509726+0000 mgr.y (mgr.34107) 12395 : cluster [DBG] pgmap v12854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:56 smithi115 bash[46934]: cluster 2022-04-22T23:29:55.509726+0000 mgr.y (mgr.34107) 12395 : cluster [DBG] pgmap v12854: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:57 smithi121 bash[36902]: audit 2022-04-22T23:29:57.075711+0000 mon.a (mon.0) 17063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:57 smithi115 bash[46934]: audit 2022-04-22T23:29:57.075711+0000 mon.a (mon.0) 17063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:57 smithi115 bash[45495]: audit 2022-04-22T23:29:57.075711+0000 mon.a (mon.0) 17063 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:29:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:58 smithi121 bash[36902]: cluster 2022-04-22T23:29:57.510845+0000 mgr.y (mgr.34107) 12396 : cluster [DBG] pgmap v12855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:59.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:29:58 smithi121 bash[36902]: audit 2022-04-22T23:29:58.717727+0000 mon.a (mon.0) 17064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:58 smithi115 bash[45495]: cluster 2022-04-22T23:29:57.510845+0000 mgr.y (mgr.34107) 12396 : cluster [DBG] pgmap v12855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:29:58 smithi115 bash[45495]: audit 2022-04-22T23:29:58.717727+0000 mon.a (mon.0) 17064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:29:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:58 smithi115 bash[46934]: cluster 2022-04-22T23:29:57.510845+0000 mgr.y (mgr.34107) 12396 : cluster [DBG] pgmap v12855: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:29:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:29:58 smithi115 bash[46934]: audit 2022-04-22T23:29:58.717727+0000 mon.a (mon.0) 17064 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:00 smithi121 bash[36902]: cluster 2022-04-22T23:29:59.511968+0000 mgr.y (mgr.34107) 12397 : cluster [DBG] pgmap v12856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:00 smithi121 bash[36902]: cluster 2022-04-22T23:30:00.000167+0000 mon.a (mon.0) 17065 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:30:01.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:00 smithi121 bash[36902]: audit 2022-04-22T23:30:00.490651+0000 mon.a (mon.0) 17066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[45495]: cluster 2022-04-22T23:29:59.511968+0000 mgr.y (mgr.34107) 12397 : cluster [DBG] pgmap v12856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[45495]: cluster 2022-04-22T23:30:00.000167+0000 mon.a (mon.0) 17065 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:30:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[45495]: audit 2022-04-22T23:30:00.490651+0000 mon.a (mon.0) 17066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[46934]: cluster 2022-04-22T23:29:59.511968+0000 mgr.y (mgr.34107) 12397 : cluster [DBG] pgmap v12856: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[46934]: cluster 2022-04-22T23:30:00.000167+0000 mon.a (mon.0) 17065 : cluster [WRN] overall HEALTH_WARN Redeploying daemon grafana.a on host smithi121 failed. 2022-04-22T23:30:01.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:00 smithi115 bash[46934]: audit 2022-04-22T23:30:00.490651+0000 mon.a (mon.0) 17066 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:02 smithi121 bash[36902]: cluster 2022-04-22T23:30:01.513163+0000 mgr.y (mgr.34107) 12398 : cluster [DBG] pgmap v12857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:03.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:02 smithi121 bash[36902]: audit 2022-04-22T23:30:02.078680+0000 mon.a (mon.0) 17067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:02 smithi115 bash[46934]: cluster 2022-04-22T23:30:01.513163+0000 mgr.y (mgr.34107) 12398 : cluster [DBG] pgmap v12857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:02 smithi115 bash[46934]: audit 2022-04-22T23:30:02.078680+0000 mon.a (mon.0) 17067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:03.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:03.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:02 smithi115 bash[45495]: cluster 2022-04-22T23:30:01.513163+0000 mgr.y (mgr.34107) 12398 : cluster [DBG] pgmap v12857: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:03.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:02 smithi115 bash[45495]: audit 2022-04-22T23:30:02.078680+0000 mon.a (mon.0) 17067 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:04.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:03 smithi121 bash[36902]: audit 2022-04-22T23:30:03.720803+0000 mon.a (mon.0) 17068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:03 smithi115 bash[45495]: audit 2022-04-22T23:30:03.720803+0000 mon.a (mon.0) 17068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:03 smithi115 bash[46934]: audit 2022-04-22T23:30:03.720803+0000 mon.a (mon.0) 17068 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:05.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:04 smithi121 bash[36902]: cluster 2022-04-22T23:30:03.514454+0000 mgr.y (mgr.34107) 12399 : cluster [DBG] pgmap v12858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:04 smithi115 bash[45495]: cluster 2022-04-22T23:30:03.514454+0000 mgr.y (mgr.34107) 12399 : cluster [DBG] pgmap v12858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:04.912Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:04 smithi115 bash[46934]: cluster 2022-04-22T23:30:03.514454+0000 mgr.y (mgr.34107) 12399 : cluster [DBG] pgmap v12858: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:05 smithi121 bash[36902]: audit 2022-04-22T23:30:05.494603+0000 mon.a (mon.0) 17069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:06.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:05 smithi115 bash[45495]: audit 2022-04-22T23:30:05.494603+0000 mon.a (mon.0) 17069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:06.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:05 smithi115 bash[46934]: audit 2022-04-22T23:30:05.494603+0000 mon.a (mon.0) 17069 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:07.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:06 smithi121 bash[36902]: cluster 2022-04-22T23:30:05.515548+0000 mgr.y (mgr.34107) 12400 : cluster [DBG] pgmap v12859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:07.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:06 smithi115 bash[45495]: cluster 2022-04-22T23:30:05.515548+0000 mgr.y (mgr.34107) 12400 : cluster [DBG] pgmap v12859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:07.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:06 smithi115 bash[46934]: cluster 2022-04-22T23:30:05.515548+0000 mgr.y (mgr.34107) 12400 : cluster [DBG] pgmap v12859: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:08.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:07 smithi121 bash[36902]: audit 2022-04-22T23:30:07.081775+0000 mon.a (mon.0) 17070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:07 smithi115 bash[45495]: audit 2022-04-22T23:30:07.081775+0000 mon.a (mon.0) 17070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:07 smithi115 bash[46934]: audit 2022-04-22T23:30:07.081775+0000 mon.a (mon.0) 17070 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:08 smithi121 bash[36902]: cluster 2022-04-22T23:30:07.516571+0000 mgr.y (mgr.34107) 12401 : cluster [DBG] pgmap v12860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:09.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:08 smithi121 bash[36902]: audit 2022-04-22T23:30:08.724096+0000 mon.a (mon.0) 17071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:08 smithi115 bash[46934]: cluster 2022-04-22T23:30:07.516571+0000 mgr.y (mgr.34107) 12401 : cluster [DBG] pgmap v12860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:09.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:08 smithi115 bash[46934]: audit 2022-04-22T23:30:08.724096+0000 mon.a (mon.0) 17071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:08 smithi115 bash[45495]: cluster 2022-04-22T23:30:07.516571+0000 mgr.y (mgr.34107) 12401 : cluster [DBG] pgmap v12860: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:09.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:08 smithi115 bash[45495]: audit 2022-04-22T23:30:08.724096+0000 mon.a (mon.0) 17071 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:09.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[45495]: audit 2022-04-22T23:30:09.265111+0000 mon.a (mon.0) 17072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:30:09.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[45495]: audit 2022-04-22T23:30:09.267977+0000 mon.a (mon.0) 17073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:30:09.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[45495]: audit 2022-04-22T23:30:09.269510+0000 mon.a (mon.0) 17074 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:30:09.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[45495]: audit 2022-04-22T23:30:09.633814+0000 mon.a (mon.0) 17075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:30:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:09 smithi121 bash[36902]: audit 2022-04-22T23:30:09.265111+0000 mon.a (mon.0) 17072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:30:10.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:09 smithi121 bash[36902]: audit 2022-04-22T23:30:09.267977+0000 mon.a (mon.0) 17073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:30:10.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:09 smithi121 bash[36902]: audit 2022-04-22T23:30:09.269510+0000 mon.a (mon.0) 17074 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:30:10.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:09 smithi121 bash[36902]: audit 2022-04-22T23:30:09.633814+0000 mon.a (mon.0) 17075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:30:10.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[46934]: audit 2022-04-22T23:30:09.265111+0000 mon.a (mon.0) 17072 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:30:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[46934]: audit 2022-04-22T23:30:09.267977+0000 mon.a (mon.0) 17073 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:30:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[46934]: audit 2022-04-22T23:30:09.269510+0000 mon.a (mon.0) 17074 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:30:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:09 smithi115 bash[46934]: audit 2022-04-22T23:30:09.633814+0000 mon.a (mon.0) 17075 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:30:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:10 smithi121 bash[36902]: cluster 2022-04-22T23:30:09.517631+0000 mgr.y (mgr.34107) 12402 : cluster [DBG] pgmap v12861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:11.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:10 smithi121 bash[36902]: audit 2022-04-22T23:30:10.498374+0000 mon.a (mon.0) 17076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:10 smithi115 bash[45495]: cluster 2022-04-22T23:30:09.517631+0000 mgr.y (mgr.34107) 12402 : cluster [DBG] pgmap v12861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:11.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:10 smithi115 bash[45495]: audit 2022-04-22T23:30:10.498374+0000 mon.a (mon.0) 17076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:10 smithi115 bash[46934]: cluster 2022-04-22T23:30:09.517631+0000 mgr.y (mgr.34107) 12402 : cluster [DBG] pgmap v12861: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:11.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:10 smithi115 bash[46934]: audit 2022-04-22T23:30:10.498374+0000 mon.a (mon.0) 17076 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:12 smithi121 bash[36902]: cluster 2022-04-22T23:30:11.518711+0000 mgr.y (mgr.34107) 12403 : cluster [DBG] pgmap v12862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:13.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:12 smithi121 bash[36902]: audit 2022-04-22T23:30:12.084887+0000 mon.a (mon.0) 17077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:12 smithi115 bash[45495]: cluster 2022-04-22T23:30:11.518711+0000 mgr.y (mgr.34107) 12403 : cluster [DBG] pgmap v12862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:13.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:12 smithi115 bash[45495]: audit 2022-04-22T23:30:12.084887+0000 mon.a (mon.0) 17077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:13.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:13.051Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:12 smithi115 bash[46934]: cluster 2022-04-22T23:30:11.518711+0000 mgr.y (mgr.34107) 12403 : cluster [DBG] pgmap v12862: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:13.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:12 smithi115 bash[46934]: audit 2022-04-22T23:30:12.084887+0000 mon.a (mon.0) 17077 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:14.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:13 smithi121 bash[36902]: audit 2022-04-22T23:30:13.727323+0000 mon.a (mon.0) 17078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:14.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:13 smithi115 bash[45495]: audit 2022-04-22T23:30:13.727323+0000 mon.a (mon.0) 17078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:14.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:13 smithi115 bash[46934]: audit 2022-04-22T23:30:13.727323+0000 mon.a (mon.0) 17078 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:15.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:14 smithi121 bash[36902]: cluster 2022-04-22T23:30:13.519867+0000 mgr.y (mgr.34107) 12404 : cluster [DBG] pgmap v12863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:15.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:14 smithi115 bash[45495]: cluster 2022-04-22T23:30:13.519867+0000 mgr.y (mgr.34107) 12404 : cluster [DBG] pgmap v12863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:14.914Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:14 smithi115 bash[46934]: cluster 2022-04-22T23:30:13.519867+0000 mgr.y (mgr.34107) 12404 : cluster [DBG] pgmap v12863: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:16.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:15 smithi121 bash[36902]: audit 2022-04-22T23:30:15.501976+0000 mon.a (mon.0) 17079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:16.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:15 smithi115 bash[45495]: audit 2022-04-22T23:30:15.501976+0000 mon.a (mon.0) 17079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:16.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:15 smithi115 bash[46934]: audit 2022-04-22T23:30:15.501976+0000 mon.a (mon.0) 17079 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:16 smithi121 bash[36902]: cluster 2022-04-22T23:30:15.520870+0000 mgr.y (mgr.34107) 12405 : cluster [DBG] pgmap v12864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:17.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:16 smithi115 bash[45495]: cluster 2022-04-22T23:30:15.520870+0000 mgr.y (mgr.34107) 12405 : cluster [DBG] pgmap v12864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:17.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:16 smithi115 bash[46934]: cluster 2022-04-22T23:30:15.520870+0000 mgr.y (mgr.34107) 12405 : cluster [DBG] pgmap v12864: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:17 smithi121 bash[36902]: audit 2022-04-22T23:30:17.088075+0000 mon.a (mon.0) 17080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:17 smithi121 bash[36902]: audit 2022-04-22T23:30:17.474479+0000 mon.a (mon.0) 17081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:30:18.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:17 smithi121 bash[36902]: audit 2022-04-22T23:30:17.474953+0000 mon.a (mon.0) 17082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:30:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[46934]: audit 2022-04-22T23:30:17.088075+0000 mon.a (mon.0) 17080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[46934]: audit 2022-04-22T23:30:17.474479+0000 mon.a (mon.0) 17081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:30:18.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[46934]: audit 2022-04-22T23:30:17.474953+0000 mon.a (mon.0) 17082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:30:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[45495]: audit 2022-04-22T23:30:17.088075+0000 mon.a (mon.0) 17080 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:18.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[45495]: audit 2022-04-22T23:30:17.474479+0000 mon.a (mon.0) 17081 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:30:18.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:17 smithi115 bash[45495]: audit 2022-04-22T23:30:17.474953+0000 mon.a (mon.0) 17082 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:30:18.570 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:30:18.920 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 3m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 3m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 2m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 2m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 3m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:30:18.921 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 2m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 2m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:30:18.922 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:30:18.923 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:30:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:18 smithi121 bash[36902]: cluster 2022-04-22T23:30:17.521966+0000 mgr.y (mgr.34107) 12406 : cluster [DBG] pgmap v12865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:19.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:18 smithi121 bash[36902]: audit 2022-04-22T23:30:18.730364+0000 mon.a (mon.0) 17083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:19.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:18 smithi115 bash[45495]: cluster 2022-04-22T23:30:17.521966+0000 mgr.y (mgr.34107) 12406 : cluster [DBG] pgmap v12865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:19.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:18 smithi115 bash[45495]: audit 2022-04-22T23:30:18.730364+0000 mon.a (mon.0) 17083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:19.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:18 smithi115 bash[46934]: cluster 2022-04-22T23:30:17.521966+0000 mgr.y (mgr.34107) 12406 : cluster [DBG] pgmap v12865: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:19.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:18 smithi115 bash[46934]: audit 2022-04-22T23:30:18.730364+0000 mon.a (mon.0) 17083 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:30:19.325 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:30:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:30:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:30:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:30:19.327 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:30:19.327 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:30:19.327 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:30:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:19 smithi121 bash[36902]: audit 2022-04-22T23:30:18.551362+0000 mgr.y (mgr.34107) 12407 : audit [DBG] from='client.48177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:19 smithi121 bash[36902]: audit 2022-04-22T23:30:18.895192+0000 mgr.y (mgr.34107) 12408 : audit [DBG] from='client.58123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:19 smithi121 bash[36902]: audit 2022-04-22T23:30:19.321510+0000 mon.a (mon.0) 17084 : audit [DBG] from='client.? 172.21.15.115:0/1642511234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:20.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[45495]: audit 2022-04-22T23:30:18.551362+0000 mgr.y (mgr.34107) 12407 : audit [DBG] from='client.48177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[45495]: audit 2022-04-22T23:30:18.895192+0000 mgr.y (mgr.34107) 12408 : audit [DBG] from='client.58123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[45495]: audit 2022-04-22T23:30:19.321510+0000 mon.a (mon.0) 17084 : audit [DBG] from='client.? 172.21.15.115:0/1642511234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[46934]: audit 2022-04-22T23:30:18.551362+0000 mgr.y (mgr.34107) 12407 : audit [DBG] from='client.48177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[46934]: audit 2022-04-22T23:30:18.895192+0000 mgr.y (mgr.34107) 12408 : audit [DBG] from='client.58123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:20.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:19 smithi115 bash[46934]: audit 2022-04-22T23:30:19.321510+0000 mon.a (mon.0) 17084 : audit [DBG] from='client.? 172.21.15.115:0/1642511234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:20 smithi121 bash[36902]: cluster 2022-04-22T23:30:19.523056+0000 mgr.y (mgr.34107) 12409 : cluster [DBG] pgmap v12866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:21.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:20 smithi121 bash[36902]: audit 2022-04-22T23:30:20.505782+0000 mon.a (mon.0) 17085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:21.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:20 smithi115 bash[45495]: cluster 2022-04-22T23:30:19.523056+0000 mgr.y (mgr.34107) 12409 : cluster [DBG] pgmap v12866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:21.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:20 smithi115 bash[45495]: audit 2022-04-22T23:30:20.505782+0000 mon.a (mon.0) 17085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:20 smithi115 bash[46934]: cluster 2022-04-22T23:30:19.523056+0000 mgr.y (mgr.34107) 12409 : cluster [DBG] pgmap v12866: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:21.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:20 smithi115 bash[46934]: audit 2022-04-22T23:30:20.505782+0000 mon.a (mon.0) 17085 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:22 smithi121 bash[36902]: cluster 2022-04-22T23:30:21.524169+0000 mgr.y (mgr.34107) 12410 : cluster [DBG] pgmap v12867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:23.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:22 smithi121 bash[36902]: audit 2022-04-22T23:30:22.090948+0000 mon.a (mon.0) 17086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:23.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:23.050Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:22 smithi115 bash[45495]: cluster 2022-04-22T23:30:21.524169+0000 mgr.y (mgr.34107) 12410 : cluster [DBG] pgmap v12867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:23.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:22 smithi115 bash[45495]: audit 2022-04-22T23:30:22.090948+0000 mon.a (mon.0) 17086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:23.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:22 smithi115 bash[46934]: cluster 2022-04-22T23:30:21.524169+0000 mgr.y (mgr.34107) 12410 : cluster [DBG] pgmap v12867: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:23.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:22 smithi115 bash[46934]: audit 2022-04-22T23:30:22.090948+0000 mon.a (mon.0) 17086 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:23 smithi121 bash[36902]: audit 2022-04-22T23:30:23.733302+0000 mon.a (mon.0) 17087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:23 smithi115 bash[45495]: audit 2022-04-22T23:30:23.733302+0000 mon.a (mon.0) 17087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:24.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:23 smithi115 bash[46934]: audit 2022-04-22T23:30:23.733302+0000 mon.a (mon.0) 17087 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:24 smithi121 bash[36902]: cluster 2022-04-22T23:30:23.525263+0000 mgr.y (mgr.34107) 12411 : cluster [DBG] pgmap v12868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:24 smithi115 bash[45495]: cluster 2022-04-22T23:30:23.525263+0000 mgr.y (mgr.34107) 12411 : cluster [DBG] pgmap v12868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:24.913Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:24 smithi115 bash[46934]: cluster 2022-04-22T23:30:23.525263+0000 mgr.y (mgr.34107) 12411 : cluster [DBG] pgmap v12868: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:25 smithi121 bash[36902]: audit 2022-04-22T23:30:25.509358+0000 mon.a (mon.0) 17088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:26.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:25 smithi115 bash[46934]: audit 2022-04-22T23:30:25.509358+0000 mon.a (mon.0) 17088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:26.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:25 smithi115 bash[45495]: audit 2022-04-22T23:30:25.509358+0000 mon.a (mon.0) 17088 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:26 smithi121 bash[36902]: cluster 2022-04-22T23:30:25.526327+0000 mgr.y (mgr.34107) 12412 : cluster [DBG] pgmap v12869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:27.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:26 smithi115 bash[46934]: cluster 2022-04-22T23:30:25.526327+0000 mgr.y (mgr.34107) 12412 : cluster [DBG] pgmap v12869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:26 smithi115 bash[45495]: cluster 2022-04-22T23:30:25.526327+0000 mgr.y (mgr.34107) 12412 : cluster [DBG] pgmap v12869: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:27 smithi121 bash[36902]: audit 2022-04-22T23:30:27.094139+0000 mon.a (mon.0) 17089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:28.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:27 smithi115 bash[45495]: audit 2022-04-22T23:30:27.094139+0000 mon.a (mon.0) 17089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:27 smithi115 bash[46934]: audit 2022-04-22T23:30:27.094139+0000 mon.a (mon.0) 17089 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:28 smithi121 bash[36902]: cluster 2022-04-22T23:30:27.527443+0000 mgr.y (mgr.34107) 12413 : cluster [DBG] pgmap v12870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:28 smithi121 bash[36902]: audit 2022-04-22T23:30:28.736657+0000 mon.a (mon.0) 17090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:29.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:28 smithi115 bash[45495]: cluster 2022-04-22T23:30:27.527443+0000 mgr.y (mgr.34107) 12413 : cluster [DBG] pgmap v12870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:28 smithi115 bash[45495]: audit 2022-04-22T23:30:28.736657+0000 mon.a (mon.0) 17090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:28 smithi115 bash[46934]: cluster 2022-04-22T23:30:27.527443+0000 mgr.y (mgr.34107) 12413 : cluster [DBG] pgmap v12870: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:28 smithi115 bash[46934]: audit 2022-04-22T23:30:28.736657+0000 mon.a (mon.0) 17090 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:30 smithi121 bash[36902]: cluster 2022-04-22T23:30:29.528441+0000 mgr.y (mgr.34107) 12414 : cluster [DBG] pgmap v12871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:30 smithi121 bash[36902]: audit 2022-04-22T23:30:30.512893+0000 mon.a (mon.0) 17091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:30 smithi115 bash[45495]: cluster 2022-04-22T23:30:29.528441+0000 mgr.y (mgr.34107) 12414 : cluster [DBG] pgmap v12871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:30 smithi115 bash[45495]: audit 2022-04-22T23:30:30.512893+0000 mon.a (mon.0) 17091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:30 smithi115 bash[46934]: cluster 2022-04-22T23:30:29.528441+0000 mgr.y (mgr.34107) 12414 : cluster [DBG] pgmap v12871: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:30 smithi115 bash[46934]: audit 2022-04-22T23:30:30.512893+0000 mon.a (mon.0) 17091 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:33 smithi121 bash[36902]: cluster 2022-04-22T23:30:31.529513+0000 mgr.y (mgr.34107) 12415 : cluster [DBG] pgmap v12872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:33 smithi121 bash[36902]: audit 2022-04-22T23:30:32.097080+0000 mon.a (mon.0) 17092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:33.053Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:32 smithi115 bash[45495]: cluster 2022-04-22T23:30:31.529513+0000 mgr.y (mgr.34107) 12415 : cluster [DBG] pgmap v12872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:32 smithi115 bash[45495]: audit 2022-04-22T23:30:32.097080+0000 mon.a (mon.0) 17092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:32 smithi115 bash[46934]: cluster 2022-04-22T23:30:31.529513+0000 mgr.y (mgr.34107) 12415 : cluster [DBG] pgmap v12872: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:32 smithi115 bash[46934]: audit 2022-04-22T23:30:32.097080+0000 mon.a (mon.0) 17092 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:34 smithi121 bash[36902]: audit 2022-04-22T23:30:33.739759+0000 mon.a (mon.0) 17093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:34 smithi115 bash[45495]: audit 2022-04-22T23:30:33.739759+0000 mon.a (mon.0) 17093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:34 smithi115 bash[46934]: audit 2022-04-22T23:30:33.739759+0000 mon.a (mon.0) 17093 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:35 smithi121 bash[36902]: cluster 2022-04-22T23:30:33.530623+0000 mgr.y (mgr.34107) 12416 : cluster [DBG] pgmap v12873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:35 smithi115 bash[45495]: cluster 2022-04-22T23:30:33.530623+0000 mgr.y (mgr.34107) 12416 : cluster [DBG] pgmap v12873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:35 smithi115 bash[46934]: cluster 2022-04-22T23:30:33.530623+0000 mgr.y (mgr.34107) 12416 : cluster [DBG] pgmap v12873: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:34.913Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:36 smithi115 bash[45495]: audit 2022-04-22T23:30:35.516415+0000 mon.a (mon.0) 17094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:36 smithi115 bash[46934]: audit 2022-04-22T23:30:35.516415+0000 mon.a (mon.0) 17094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:36 smithi121 bash[36902]: audit 2022-04-22T23:30:35.516415+0000 mon.a (mon.0) 17094 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:37 smithi115 bash[45495]: cluster 2022-04-22T23:30:35.531706+0000 mgr.y (mgr.34107) 12417 : cluster [DBG] pgmap v12874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:37 smithi115 bash[46934]: cluster 2022-04-22T23:30:35.531706+0000 mgr.y (mgr.34107) 12417 : cluster [DBG] pgmap v12874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:37 smithi121 bash[36902]: cluster 2022-04-22T23:30:35.531706+0000 mgr.y (mgr.34107) 12417 : cluster [DBG] pgmap v12874: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:38 smithi115 bash[45495]: audit 2022-04-22T23:30:37.100095+0000 mon.a (mon.0) 17095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:38 smithi115 bash[46934]: audit 2022-04-22T23:30:37.100095+0000 mon.a (mon.0) 17095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:38 smithi121 bash[36902]: audit 2022-04-22T23:30:37.100095+0000 mon.a (mon.0) 17095 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:39 smithi115 bash[45495]: cluster 2022-04-22T23:30:37.532851+0000 mgr.y (mgr.34107) 12418 : cluster [DBG] pgmap v12875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:39 smithi115 bash[45495]: audit 2022-04-22T23:30:38.742859+0000 mon.a (mon.0) 17096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:39 smithi115 bash[46934]: cluster 2022-04-22T23:30:37.532851+0000 mgr.y (mgr.34107) 12418 : cluster [DBG] pgmap v12875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:39 smithi115 bash[46934]: audit 2022-04-22T23:30:38.742859+0000 mon.a (mon.0) 17096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:39 smithi121 bash[36902]: cluster 2022-04-22T23:30:37.532851+0000 mgr.y (mgr.34107) 12418 : cluster [DBG] pgmap v12875: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:39 smithi121 bash[36902]: audit 2022-04-22T23:30:38.742859+0000 mon.a (mon.0) 17096 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:41 smithi115 bash[45495]: cluster 2022-04-22T23:30:39.534048+0000 mgr.y (mgr.34107) 12419 : cluster [DBG] pgmap v12876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:41 smithi115 bash[45495]: audit 2022-04-22T23:30:40.520184+0000 mon.a (mon.0) 17097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:41 smithi115 bash[46934]: cluster 2022-04-22T23:30:39.534048+0000 mgr.y (mgr.34107) 12419 : cluster [DBG] pgmap v12876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:41 smithi115 bash[46934]: audit 2022-04-22T23:30:40.520184+0000 mon.a (mon.0) 17097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:41 smithi121 bash[36902]: cluster 2022-04-22T23:30:39.534048+0000 mgr.y (mgr.34107) 12419 : cluster [DBG] pgmap v12876: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:41 smithi121 bash[36902]: audit 2022-04-22T23:30:40.520184+0000 mon.a (mon.0) 17097 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:43 smithi121 bash[36902]: cluster 2022-04-22T23:30:41.535178+0000 mgr.y (mgr.34107) 12420 : cluster [DBG] pgmap v12877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:43 smithi121 bash[36902]: audit 2022-04-22T23:30:42.102742+0000 mon.a (mon.0) 17098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:43.540 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:43.050Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:43 smithi115 bash[45495]: cluster 2022-04-22T23:30:41.535178+0000 mgr.y (mgr.34107) 12420 : cluster [DBG] pgmap v12877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:43 smithi115 bash[45495]: audit 2022-04-22T23:30:42.102742+0000 mon.a (mon.0) 17098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:43 smithi115 bash[46934]: cluster 2022-04-22T23:30:41.535178+0000 mgr.y (mgr.34107) 12420 : cluster [DBG] pgmap v12877: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:43 smithi115 bash[46934]: audit 2022-04-22T23:30:42.102742+0000 mon.a (mon.0) 17098 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:44 smithi121 bash[36902]: audit 2022-04-22T23:30:43.746246+0000 mon.a (mon.0) 17099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:44 smithi115 bash[45495]: audit 2022-04-22T23:30:43.746246+0000 mon.a (mon.0) 17099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:44 smithi115 bash[46934]: audit 2022-04-22T23:30:43.746246+0000 mon.a (mon.0) 17099 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:45 smithi115 bash[45495]: cluster 2022-04-22T23:30:43.536281+0000 mgr.y (mgr.34107) 12421 : cluster [DBG] pgmap v12878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:44.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:45 smithi115 bash[46934]: cluster 2022-04-22T23:30:43.536281+0000 mgr.y (mgr.34107) 12421 : cluster [DBG] pgmap v12878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:45 smithi121 bash[36902]: cluster 2022-04-22T23:30:43.536281+0000 mgr.y (mgr.34107) 12421 : cluster [DBG] pgmap v12878: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:46 smithi121 bash[36902]: audit 2022-04-22T23:30:45.523983+0000 mon.a (mon.0) 17100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:46 smithi115 bash[45495]: audit 2022-04-22T23:30:45.523983+0000 mon.a (mon.0) 17100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:46 smithi115 bash[46934]: audit 2022-04-22T23:30:45.523983+0000 mon.a (mon.0) 17100 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:47 smithi121 bash[36902]: cluster 2022-04-22T23:30:45.537256+0000 mgr.y (mgr.34107) 12422 : cluster [DBG] pgmap v12879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:47 smithi115 bash[45495]: cluster 2022-04-22T23:30:45.537256+0000 mgr.y (mgr.34107) 12422 : cluster [DBG] pgmap v12879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:47 smithi115 bash[46934]: cluster 2022-04-22T23:30:45.537256+0000 mgr.y (mgr.34107) 12422 : cluster [DBG] pgmap v12879: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:48 smithi121 bash[36902]: audit 2022-04-22T23:30:47.106018+0000 mon.a (mon.0) 17101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:48.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:48 smithi115 bash[45495]: audit 2022-04-22T23:30:47.106018+0000 mon.a (mon.0) 17101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:48 smithi115 bash[46934]: audit 2022-04-22T23:30:47.106018+0000 mon.a (mon.0) 17101 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:49 smithi121 bash[36902]: cluster 2022-04-22T23:30:47.538343+0000 mgr.y (mgr.34107) 12423 : cluster [DBG] pgmap v12880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:49 smithi121 bash[36902]: audit 2022-04-22T23:30:48.749509+0000 mon.a (mon.0) 17102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:49 smithi115 bash[46934]: cluster 2022-04-22T23:30:47.538343+0000 mgr.y (mgr.34107) 12423 : cluster [DBG] pgmap v12880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:49 smithi115 bash[46934]: audit 2022-04-22T23:30:48.749509+0000 mon.a (mon.0) 17102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:49 smithi115 bash[45495]: cluster 2022-04-22T23:30:47.538343+0000 mgr.y (mgr.34107) 12423 : cluster [DBG] pgmap v12880: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:49 smithi115 bash[45495]: audit 2022-04-22T23:30:48.749509+0000 mon.a (mon.0) 17102 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:49.707 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 2m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 3m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 2m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 3m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 3m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 2m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:30:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 2m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 3m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 2m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 2m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 3m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 2m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (5h) 2m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5h) 2m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 2m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 3m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:30:50.050 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 3m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:30:50.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 3m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:30:50.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 3m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:30:50.051 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 3m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:30:50.452 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:30:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:30:50.454 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:30:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:51 smithi121 bash[36902]: cluster 2022-04-22T23:30:49.539359+0000 mgr.y (mgr.34107) 12424 : cluster [DBG] pgmap v12881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:51 smithi121 bash[36902]: audit 2022-04-22T23:30:49.687595+0000 mgr.y (mgr.34107) 12425 : audit [DBG] from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:51 smithi121 bash[36902]: audit 2022-04-22T23:30:50.030953+0000 mgr.y (mgr.34107) 12426 : audit [DBG] from='client.48201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:51 smithi121 bash[36902]: audit 2022-04-22T23:30:50.448910+0000 mon.a (mon.0) 17103 : audit [DBG] from='client.? 172.21.15.115:0/2349195140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:51.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:51 smithi121 bash[36902]: audit 2022-04-22T23:30:50.527673+0000 mon.a (mon.0) 17104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[45495]: cluster 2022-04-22T23:30:49.539359+0000 mgr.y (mgr.34107) 12424 : cluster [DBG] pgmap v12881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[45495]: audit 2022-04-22T23:30:49.687595+0000 mgr.y (mgr.34107) 12425 : audit [DBG] from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[45495]: audit 2022-04-22T23:30:50.030953+0000 mgr.y (mgr.34107) 12426 : audit [DBG] from='client.48201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[45495]: audit 2022-04-22T23:30:50.448910+0000 mon.a (mon.0) 17103 : audit [DBG] from='client.? 172.21.15.115:0/2349195140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[45495]: audit 2022-04-22T23:30:50.527673+0000 mon.a (mon.0) 17104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[46934]: cluster 2022-04-22T23:30:49.539359+0000 mgr.y (mgr.34107) 12424 : cluster [DBG] pgmap v12881: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[46934]: audit 2022-04-22T23:30:49.687595+0000 mgr.y (mgr.34107) 12425 : audit [DBG] from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[46934]: audit 2022-04-22T23:30:50.030953+0000 mgr.y (mgr.34107) 12426 : audit [DBG] from='client.48201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:30:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[46934]: audit 2022-04-22T23:30:50.448910+0000 mon.a (mon.0) 17103 : audit [DBG] from='client.? 172.21.15.115:0/2349195140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:30:51.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:51 smithi115 bash[46934]: audit 2022-04-22T23:30:50.527673+0000 mon.a (mon.0) 17104 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:53 smithi121 bash[36902]: cluster 2022-04-22T23:30:51.540414+0000 mgr.y (mgr.34107) 12427 : cluster [DBG] pgmap v12882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:53 smithi121 bash[36902]: audit 2022-04-22T23:30:52.108767+0000 mon.a (mon.0) 17105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:53.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:53.053Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:53 smithi115 bash[45495]: cluster 2022-04-22T23:30:51.540414+0000 mgr.y (mgr.34107) 12427 : cluster [DBG] pgmap v12882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:53 smithi115 bash[45495]: audit 2022-04-22T23:30:52.108767+0000 mon.a (mon.0) 17105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:53 smithi115 bash[46934]: cluster 2022-04-22T23:30:51.540414+0000 mgr.y (mgr.34107) 12427 : cluster [DBG] pgmap v12882: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:53 smithi115 bash[46934]: audit 2022-04-22T23:30:52.108767+0000 mon.a (mon.0) 17105 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:54 smithi121 bash[36902]: audit 2022-04-22T23:30:53.752640+0000 mon.a (mon.0) 17106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:54 smithi115 bash[45495]: audit 2022-04-22T23:30:53.752640+0000 mon.a (mon.0) 17106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:54 smithi115 bash[46934]: audit 2022-04-22T23:30:53.752640+0000 mon.a (mon.0) 17106 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:55 smithi115 bash[45495]: cluster 2022-04-22T23:30:53.541506+0000 mgr.y (mgr.34107) 12428 : cluster [DBG] pgmap v12883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:30:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:30:54.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:30:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:55 smithi115 bash[46934]: cluster 2022-04-22T23:30:53.541506+0000 mgr.y (mgr.34107) 12428 : cluster [DBG] pgmap v12883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:55 smithi121 bash[36902]: cluster 2022-04-22T23:30:53.541506+0000 mgr.y (mgr.34107) 12428 : cluster [DBG] pgmap v12883: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:56 smithi121 bash[36902]: audit 2022-04-22T23:30:55.531144+0000 mon.a (mon.0) 17107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:56.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:56 smithi115 bash[46934]: audit 2022-04-22T23:30:55.531144+0000 mon.a (mon.0) 17107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:56 smithi115 bash[45495]: audit 2022-04-22T23:30:55.531144+0000 mon.a (mon.0) 17107 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:30:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:57 smithi121 bash[36902]: cluster 2022-04-22T23:30:55.542555+0000 mgr.y (mgr.34107) 12429 : cluster [DBG] pgmap v12884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:57 smithi115 bash[46934]: cluster 2022-04-22T23:30:55.542555+0000 mgr.y (mgr.34107) 12429 : cluster [DBG] pgmap v12884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:57 smithi115 bash[45495]: cluster 2022-04-22T23:30:55.542555+0000 mgr.y (mgr.34107) 12429 : cluster [DBG] pgmap v12884: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:58 smithi121 bash[36902]: audit 2022-04-22T23:30:57.111438+0000 mon.a (mon.0) 17108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:58 smithi115 bash[46934]: audit 2022-04-22T23:30:57.111438+0000 mon.a (mon.0) 17108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:58 smithi115 bash[45495]: audit 2022-04-22T23:30:57.111438+0000 mon.a (mon.0) 17108 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:30:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:59 smithi121 bash[36902]: cluster 2022-04-22T23:30:57.543608+0000 mgr.y (mgr.34107) 12430 : cluster [DBG] pgmap v12885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:30:59 smithi121 bash[36902]: audit 2022-04-22T23:30:58.755948+0000 mon.a (mon.0) 17109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:59 smithi115 bash[45495]: cluster 2022-04-22T23:30:57.543608+0000 mgr.y (mgr.34107) 12430 : cluster [DBG] pgmap v12885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:59.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:30:59 smithi115 bash[45495]: audit 2022-04-22T23:30:58.755948+0000 mon.a (mon.0) 17109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:30:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:59 smithi115 bash[46934]: cluster 2022-04-22T23:30:57.543608+0000 mgr.y (mgr.34107) 12430 : cluster [DBG] pgmap v12885: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:30:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:30:59 smithi115 bash[46934]: audit 2022-04-22T23:30:58.755948+0000 mon.a (mon.0) 17109 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:01 smithi121 bash[36902]: cluster 2022-04-22T23:30:59.544558+0000 mgr.y (mgr.34107) 12431 : cluster [DBG] pgmap v12886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:01 smithi121 bash[36902]: audit 2022-04-22T23:31:00.534801+0000 mon.a (mon.0) 17110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:01 smithi115 bash[45495]: cluster 2022-04-22T23:30:59.544558+0000 mgr.y (mgr.34107) 12431 : cluster [DBG] pgmap v12886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:01 smithi115 bash[45495]: audit 2022-04-22T23:31:00.534801+0000 mon.a (mon.0) 17110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:01 smithi115 bash[46934]: cluster 2022-04-22T23:30:59.544558+0000 mgr.y (mgr.34107) 12431 : cluster [DBG] pgmap v12886: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:01.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:01 smithi115 bash[46934]: audit 2022-04-22T23:31:00.534801+0000 mon.a (mon.0) 17110 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:02.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:02 smithi121 bash[36902]: audit 2022-04-22T23:31:02.114059+0000 mon.a (mon.0) 17111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:02 smithi115 bash[45495]: audit 2022-04-22T23:31:02.114059+0000 mon.a (mon.0) 17111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:02 smithi115 bash[46934]: audit 2022-04-22T23:31:02.114059+0000 mon.a (mon.0) 17111 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:03 smithi121 bash[36902]: cluster 2022-04-22T23:31:01.545747+0000 mgr.y (mgr.34107) 12432 : cluster [DBG] pgmap v12887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:03 smithi115 bash[46934]: cluster 2022-04-22T23:31:01.545747+0000 mgr.y (mgr.34107) 12432 : cluster [DBG] pgmap v12887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:03.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:03.055Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:03.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:03 smithi115 bash[45495]: cluster 2022-04-22T23:31:01.545747+0000 mgr.y (mgr.34107) 12432 : cluster [DBG] pgmap v12887: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:04 smithi121 bash[36902]: audit 2022-04-22T23:31:03.759129+0000 mon.a (mon.0) 17112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:04 smithi115 bash[46934]: audit 2022-04-22T23:31:03.759129+0000 mon.a (mon.0) 17112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:04 smithi115 bash[45495]: audit 2022-04-22T23:31:03.759129+0000 mon.a (mon.0) 17112 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:05 smithi115 bash[45495]: cluster 2022-04-22T23:31:03.546870+0000 mgr.y (mgr.34107) 12433 : cluster [DBG] pgmap v12888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:04.916Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:05 smithi115 bash[46934]: cluster 2022-04-22T23:31:03.546870+0000 mgr.y (mgr.34107) 12433 : cluster [DBG] pgmap v12888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:05 smithi121 bash[36902]: cluster 2022-04-22T23:31:03.546870+0000 mgr.y (mgr.34107) 12433 : cluster [DBG] pgmap v12888: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:06 smithi121 bash[36902]: audit 2022-04-22T23:31:05.538383+0000 mon.a (mon.0) 17113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:06 smithi115 bash[45495]: audit 2022-04-22T23:31:05.538383+0000 mon.a (mon.0) 17113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:06 smithi115 bash[46934]: audit 2022-04-22T23:31:05.538383+0000 mon.a (mon.0) 17113 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:07.157 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:07 smithi121 bash[36902]: cluster 2022-04-22T23:31:05.547909+0000 mgr.y (mgr.34107) 12434 : cluster [DBG] pgmap v12889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:07 smithi121 bash[36902]: audit 2022-04-22T23:31:07.117376+0000 mon.a (mon.0) 17114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:07 smithi115 bash[45495]: cluster 2022-04-22T23:31:05.547909+0000 mgr.y (mgr.34107) 12434 : cluster [DBG] pgmap v12889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:07 smithi115 bash[45495]: audit 2022-04-22T23:31:07.117376+0000 mon.a (mon.0) 17114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:07 smithi115 bash[46934]: cluster 2022-04-22T23:31:05.547909+0000 mgr.y (mgr.34107) 12434 : cluster [DBG] pgmap v12889: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:07.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:07 smithi115 bash[46934]: audit 2022-04-22T23:31:07.117376+0000 mon.a (mon.0) 17114 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:09 smithi121 bash[36902]: cluster 2022-04-22T23:31:07.548936+0000 mgr.y (mgr.34107) 12435 : cluster [DBG] pgmap v12890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:09 smithi121 bash[36902]: audit 2022-04-22T23:31:08.762375+0000 mon.a (mon.0) 17115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:09 smithi115 bash[45495]: cluster 2022-04-22T23:31:07.548936+0000 mgr.y (mgr.34107) 12435 : cluster [DBG] pgmap v12890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:09 smithi115 bash[45495]: audit 2022-04-22T23:31:08.762375+0000 mon.a (mon.0) 17115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:09 smithi115 bash[46934]: cluster 2022-04-22T23:31:07.548936+0000 mgr.y (mgr.34107) 12435 : cluster [DBG] pgmap v12890: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:09 smithi115 bash[46934]: audit 2022-04-22T23:31:08.762375+0000 mon.a (mon.0) 17115 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:10 smithi121 bash[36902]: audit 2022-04-22T23:31:09.642866+0000 mon.a (mon.0) 17116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:31:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:10 smithi121 bash[36902]: audit 2022-04-22T23:31:09.645936+0000 mon.a (mon.0) 17117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:31:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:10 smithi121 bash[36902]: audit 2022-04-22T23:31:09.647771+0000 mon.a (mon.0) 17118 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:31:10.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:10 smithi121 bash[36902]: audit 2022-04-22T23:31:09.993626+0000 mon.a (mon.0) 17119 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:31:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[46934]: audit 2022-04-22T23:31:09.642866+0000 mon.a (mon.0) 17116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:31:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[46934]: audit 2022-04-22T23:31:09.645936+0000 mon.a (mon.0) 17117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:31:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[46934]: audit 2022-04-22T23:31:09.647771+0000 mon.a (mon.0) 17118 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:31:10.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[46934]: audit 2022-04-22T23:31:09.993626+0000 mon.a (mon.0) 17119 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:31:10.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[45495]: audit 2022-04-22T23:31:09.642866+0000 mon.a (mon.0) 17116 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:31:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[45495]: audit 2022-04-22T23:31:09.645936+0000 mon.a (mon.0) 17117 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:31:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[45495]: audit 2022-04-22T23:31:09.647771+0000 mon.a (mon.0) 17118 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:31:10.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:10 smithi115 bash[45495]: audit 2022-04-22T23:31:09.993626+0000 mon.a (mon.0) 17119 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:31:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:11 smithi121 bash[36902]: cluster 2022-04-22T23:31:09.550199+0000 mgr.y (mgr.34107) 12436 : cluster [DBG] pgmap v12891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:11 smithi121 bash[36902]: audit 2022-04-22T23:31:10.542040+0000 mon.a (mon.0) 17120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:11 smithi115 bash[45495]: cluster 2022-04-22T23:31:09.550199+0000 mgr.y (mgr.34107) 12436 : cluster [DBG] pgmap v12891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:11 smithi115 bash[45495]: audit 2022-04-22T23:31:10.542040+0000 mon.a (mon.0) 17120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:11 smithi115 bash[46934]: cluster 2022-04-22T23:31:09.550199+0000 mgr.y (mgr.34107) 12436 : cluster [DBG] pgmap v12891: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:11.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:11 smithi115 bash[46934]: audit 2022-04-22T23:31:10.542040+0000 mon.a (mon.0) 17120 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:12 smithi121 bash[36902]: audit 2022-04-22T23:31:12.120411+0000 mon.a (mon.0) 17121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:12 smithi115 bash[45495]: audit 2022-04-22T23:31:12.120411+0000 mon.a (mon.0) 17121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:12 smithi115 bash[46934]: audit 2022-04-22T23:31:12.120411+0000 mon.a (mon.0) 17121 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:13 smithi121 bash[36902]: cluster 2022-04-22T23:31:11.551246+0000 mgr.y (mgr.34107) 12437 : cluster [DBG] pgmap v12892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:13 smithi115 bash[45495]: cluster 2022-04-22T23:31:11.551246+0000 mgr.y (mgr.34107) 12437 : cluster [DBG] pgmap v12892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:13 smithi115 bash[46934]: cluster 2022-04-22T23:31:11.551246+0000 mgr.y (mgr.34107) 12437 : cluster [DBG] pgmap v12892: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:13.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:13.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:14 smithi121 bash[36902]: audit 2022-04-22T23:31:13.765711+0000 mon.a (mon.0) 17122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:14 smithi115 bash[45495]: audit 2022-04-22T23:31:13.765711+0000 mon.a (mon.0) 17122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:14 smithi115 bash[46934]: audit 2022-04-22T23:31:13.765711+0000 mon.a (mon.0) 17122 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:15.191 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:14.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:15 smithi121 bash[36902]: cluster 2022-04-22T23:31:13.552383+0000 mgr.y (mgr.34107) 12438 : cluster [DBG] pgmap v12893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:15.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:15 smithi115 bash[45495]: cluster 2022-04-22T23:31:13.552383+0000 mgr.y (mgr.34107) 12438 : cluster [DBG] pgmap v12893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:15.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:15 smithi115 bash[46934]: cluster 2022-04-22T23:31:13.552383+0000 mgr.y (mgr.34107) 12438 : cluster [DBG] pgmap v12893: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:16 smithi121 bash[36902]: audit 2022-04-22T23:31:15.545933+0000 mon.a (mon.0) 17123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:16 smithi115 bash[45495]: audit 2022-04-22T23:31:15.545933+0000 mon.a (mon.0) 17123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:16 smithi115 bash[46934]: audit 2022-04-22T23:31:15.545933+0000 mon.a (mon.0) 17123 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:17 smithi121 bash[36902]: cluster 2022-04-22T23:31:15.553421+0000 mgr.y (mgr.34107) 12439 : cluster [DBG] pgmap v12894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:17 smithi121 bash[36902]: audit 2022-04-22T23:31:17.123615+0000 mon.a (mon.0) 17124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:17 smithi115 bash[45495]: cluster 2022-04-22T23:31:15.553421+0000 mgr.y (mgr.34107) 12439 : cluster [DBG] pgmap v12894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:17 smithi115 bash[45495]: audit 2022-04-22T23:31:17.123615+0000 mon.a (mon.0) 17124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:17 smithi115 bash[46934]: cluster 2022-04-22T23:31:15.553421+0000 mgr.y (mgr.34107) 12439 : cluster [DBG] pgmap v12894: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:17 smithi115 bash[46934]: audit 2022-04-22T23:31:17.123615+0000 mon.a (mon.0) 17124 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:18 smithi121 bash[36902]: audit 2022-04-22T23:31:17.485696+0000 mon.a (mon.0) 17125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:31:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:18 smithi121 bash[36902]: audit 2022-04-22T23:31:17.487297+0000 mon.a (mon.0) 17126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:31:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:18 smithi115 bash[45495]: audit 2022-04-22T23:31:17.485696+0000 mon.a (mon.0) 17125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:31:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:18 smithi115 bash[45495]: audit 2022-04-22T23:31:17.487297+0000 mon.a (mon.0) 17126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:31:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:18 smithi115 bash[46934]: audit 2022-04-22T23:31:17.485696+0000 mon.a (mon.0) 17125 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:31:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:18 smithi115 bash[46934]: audit 2022-04-22T23:31:17.487297+0000 mon.a (mon.0) 17126 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:31:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:19 smithi121 bash[36902]: cluster 2022-04-22T23:31:17.554489+0000 mgr.y (mgr.34107) 12440 : cluster [DBG] pgmap v12895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:19 smithi121 bash[36902]: audit 2022-04-22T23:31:18.768778+0000 mon.a (mon.0) 17127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:19 smithi115 bash[45495]: cluster 2022-04-22T23:31:17.554489+0000 mgr.y (mgr.34107) 12440 : cluster [DBG] pgmap v12895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:19 smithi115 bash[45495]: audit 2022-04-22T23:31:18.768778+0000 mon.a (mon.0) 17127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:19 smithi115 bash[46934]: cluster 2022-04-22T23:31:17.554489+0000 mgr.y (mgr.34107) 12440 : cluster [DBG] pgmap v12895: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:19 smithi115 bash[46934]: audit 2022-04-22T23:31:18.768778+0000 mon.a (mon.0) 17127 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:20.831 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:31:21.177 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:31:21.177 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:31:21.177 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 4m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:31:21.177 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 4m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 3m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 3m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 4m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 3m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:31:21.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 3m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 3m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 3m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (5h) 3m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:31:21.179 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:31:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:21 smithi121 bash[36902]: cluster 2022-04-22T23:31:19.555538+0000 mgr.y (mgr.34107) 12441 : cluster [DBG] pgmap v12896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:21 smithi121 bash[36902]: audit 2022-04-22T23:31:20.549578+0000 mon.a (mon.0) 17128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:21 smithi115 bash[46934]: cluster 2022-04-22T23:31:19.555538+0000 mgr.y (mgr.34107) 12441 : cluster [DBG] pgmap v12896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:21 smithi115 bash[46934]: audit 2022-04-22T23:31:20.549578+0000 mon.a (mon.0) 17128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:21 smithi115 bash[45495]: cluster 2022-04-22T23:31:19.555538+0000 mgr.y (mgr.34107) 12441 : cluster [DBG] pgmap v12896: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:21.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:21 smithi115 bash[45495]: audit 2022-04-22T23:31:20.549578+0000 mon.a (mon.0) 17128 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:21.589 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:31:21.589 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:21.590 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:31:21.591 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:31:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:22 smithi121 bash[36902]: audit 2022-04-22T23:31:20.811394+0000 mgr.y (mgr.34107) 12442 : audit [DBG] from='client.48213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:22 smithi121 bash[36902]: audit 2022-04-22T23:31:21.153869+0000 mgr.y (mgr.34107) 12443 : audit [DBG] from='client.48219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:22 smithi121 bash[36902]: audit 2022-04-22T23:31:21.585663+0000 mon.a (mon.0) 17129 : audit [DBG] from='client.? 172.21.15.115:0/1106043559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:22.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:22 smithi121 bash[36902]: audit 2022-04-22T23:31:22.126424+0000 mon.a (mon.0) 17130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[45495]: audit 2022-04-22T23:31:20.811394+0000 mgr.y (mgr.34107) 12442 : audit [DBG] from='client.48213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[45495]: audit 2022-04-22T23:31:21.153869+0000 mgr.y (mgr.34107) 12443 : audit [DBG] from='client.48219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[45495]: audit 2022-04-22T23:31:21.585663+0000 mon.a (mon.0) 17129 : audit [DBG] from='client.? 172.21.15.115:0/1106043559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[45495]: audit 2022-04-22T23:31:22.126424+0000 mon.a (mon.0) 17130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[46934]: audit 2022-04-22T23:31:20.811394+0000 mgr.y (mgr.34107) 12442 : audit [DBG] from='client.48213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[46934]: audit 2022-04-22T23:31:21.153869+0000 mgr.y (mgr.34107) 12443 : audit [DBG] from='client.48219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:22.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[46934]: audit 2022-04-22T23:31:21.585663+0000 mon.a (mon.0) 17129 : audit [DBG] from='client.? 172.21.15.115:0/1106043559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:22.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:22 smithi115 bash[46934]: audit 2022-04-22T23:31:22.126424+0000 mon.a (mon.0) 17130 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:23 smithi121 bash[36902]: cluster 2022-04-22T23:31:21.556526+0000 mgr.y (mgr.34107) 12444 : cluster [DBG] pgmap v12897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:23 smithi115 bash[45495]: cluster 2022-04-22T23:31:21.556526+0000 mgr.y (mgr.34107) 12444 : cluster [DBG] pgmap v12897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:23.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:23.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:23 smithi115 bash[46934]: cluster 2022-04-22T23:31:21.556526+0000 mgr.y (mgr.34107) 12444 : cluster [DBG] pgmap v12897: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:24 smithi121 bash[36902]: audit 2022-04-22T23:31:23.771831+0000 mon.a (mon.0) 17131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:24 smithi115 bash[45495]: audit 2022-04-22T23:31:23.771831+0000 mon.a (mon.0) 17131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:24 smithi115 bash[46934]: audit 2022-04-22T23:31:23.771831+0000 mon.a (mon.0) 17131 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:25.245 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:24.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:25 smithi121 bash[36902]: cluster 2022-04-22T23:31:23.557738+0000 mgr.y (mgr.34107) 12445 : cluster [DBG] pgmap v12898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:25.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:25 smithi115 bash[45495]: cluster 2022-04-22T23:31:23.557738+0000 mgr.y (mgr.34107) 12445 : cluster [DBG] pgmap v12898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:25.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:25 smithi115 bash[46934]: cluster 2022-04-22T23:31:23.557738+0000 mgr.y (mgr.34107) 12445 : cluster [DBG] pgmap v12898: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:26 smithi121 bash[36902]: audit 2022-04-22T23:31:25.553325+0000 mon.a (mon.0) 17132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:26 smithi115 bash[46934]: audit 2022-04-22T23:31:25.553325+0000 mon.a (mon.0) 17132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:26 smithi115 bash[45495]: audit 2022-04-22T23:31:25.553325+0000 mon.a (mon.0) 17132 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:27 smithi121 bash[36902]: cluster 2022-04-22T23:31:25.558772+0000 mgr.y (mgr.34107) 12446 : cluster [DBG] pgmap v12899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:27 smithi121 bash[36902]: audit 2022-04-22T23:31:27.129461+0000 mon.a (mon.0) 17133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:27 smithi115 bash[45495]: cluster 2022-04-22T23:31:25.558772+0000 mgr.y (mgr.34107) 12446 : cluster [DBG] pgmap v12899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:27 smithi115 bash[45495]: audit 2022-04-22T23:31:27.129461+0000 mon.a (mon.0) 17133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:27 smithi115 bash[46934]: cluster 2022-04-22T23:31:25.558772+0000 mgr.y (mgr.34107) 12446 : cluster [DBG] pgmap v12899: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:27 smithi115 bash[46934]: audit 2022-04-22T23:31:27.129461+0000 mon.a (mon.0) 17133 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:29 smithi121 bash[36902]: cluster 2022-04-22T23:31:27.559879+0000 mgr.y (mgr.34107) 12447 : cluster [DBG] pgmap v12900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:29 smithi121 bash[36902]: audit 2022-04-22T23:31:28.774922+0000 mon.a (mon.0) 17134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:29 smithi115 bash[45495]: cluster 2022-04-22T23:31:27.559879+0000 mgr.y (mgr.34107) 12447 : cluster [DBG] pgmap v12900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:29 smithi115 bash[45495]: audit 2022-04-22T23:31:28.774922+0000 mon.a (mon.0) 17134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:29 smithi115 bash[46934]: cluster 2022-04-22T23:31:27.559879+0000 mgr.y (mgr.34107) 12447 : cluster [DBG] pgmap v12900: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:29 smithi115 bash[46934]: audit 2022-04-22T23:31:28.774922+0000 mon.a (mon.0) 17134 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:31 smithi115 bash[45495]: cluster 2022-04-22T23:31:29.560968+0000 mgr.y (mgr.34107) 12448 : cluster [DBG] pgmap v12901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:31 smithi115 bash[45495]: audit 2022-04-22T23:31:30.556758+0000 mon.a (mon.0) 17135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:31 smithi115 bash[46934]: cluster 2022-04-22T23:31:29.560968+0000 mgr.y (mgr.34107) 12448 : cluster [DBG] pgmap v12901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:31 smithi115 bash[46934]: audit 2022-04-22T23:31:30.556758+0000 mon.a (mon.0) 17135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:31 smithi121 bash[36902]: cluster 2022-04-22T23:31:29.560968+0000 mgr.y (mgr.34107) 12448 : cluster [DBG] pgmap v12901: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:31 smithi121 bash[36902]: audit 2022-04-22T23:31:30.556758+0000 mon.a (mon.0) 17135 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:32.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:32 smithi115 bash[45495]: audit 2022-04-22T23:31:32.132691+0000 mon.a (mon.0) 17136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:32 smithi115 bash[46934]: audit 2022-04-22T23:31:32.132691+0000 mon.a (mon.0) 17136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:32 smithi121 bash[36902]: audit 2022-04-22T23:31:32.132691+0000 mon.a (mon.0) 17136 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:33.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:33.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:33 smithi115 bash[45495]: cluster 2022-04-22T23:31:31.562071+0000 mgr.y (mgr.34107) 12449 : cluster [DBG] pgmap v12902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:33.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:33 smithi115 bash[46934]: cluster 2022-04-22T23:31:31.562071+0000 mgr.y (mgr.34107) 12449 : cluster [DBG] pgmap v12902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:33 smithi121 bash[36902]: cluster 2022-04-22T23:31:31.562071+0000 mgr.y (mgr.34107) 12449 : cluster [DBG] pgmap v12902: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:34 smithi115 bash[45495]: audit 2022-04-22T23:31:33.777961+0000 mon.a (mon.0) 17137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:34 smithi115 bash[46934]: audit 2022-04-22T23:31:33.777961+0000 mon.a (mon.0) 17137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:34 smithi121 bash[36902]: audit 2022-04-22T23:31:33.777961+0000 mon.a (mon.0) 17137 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:35.289 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:34.915Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:35 smithi115 bash[45495]: cluster 2022-04-22T23:31:33.563235+0000 mgr.y (mgr.34107) 12450 : cluster [DBG] pgmap v12903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:35 smithi115 bash[46934]: cluster 2022-04-22T23:31:33.563235+0000 mgr.y (mgr.34107) 12450 : cluster [DBG] pgmap v12903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:35 smithi121 bash[36902]: cluster 2022-04-22T23:31:33.563235+0000 mgr.y (mgr.34107) 12450 : cluster [DBG] pgmap v12903: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:36 smithi121 bash[36902]: audit 2022-04-22T23:31:35.560217+0000 mon.a (mon.0) 17138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:36.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:36 smithi115 bash[45495]: audit 2022-04-22T23:31:35.560217+0000 mon.a (mon.0) 17138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:36 smithi115 bash[46934]: audit 2022-04-22T23:31:35.560217+0000 mon.a (mon.0) 17138 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:37 smithi121 bash[36902]: cluster 2022-04-22T23:31:35.564272+0000 mgr.y (mgr.34107) 12451 : cluster [DBG] pgmap v12904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:37 smithi121 bash[36902]: audit 2022-04-22T23:31:37.135526+0000 mon.a (mon.0) 17139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:37 smithi115 bash[45495]: cluster 2022-04-22T23:31:35.564272+0000 mgr.y (mgr.34107) 12451 : cluster [DBG] pgmap v12904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:37 smithi115 bash[45495]: audit 2022-04-22T23:31:37.135526+0000 mon.a (mon.0) 17139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:37 smithi115 bash[46934]: cluster 2022-04-22T23:31:35.564272+0000 mgr.y (mgr.34107) 12451 : cluster [DBG] pgmap v12904: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:37 smithi115 bash[46934]: audit 2022-04-22T23:31:37.135526+0000 mon.a (mon.0) 17139 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:39 smithi121 bash[36902]: cluster 2022-04-22T23:31:37.565339+0000 mgr.y (mgr.34107) 12452 : cluster [DBG] pgmap v12905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:39 smithi121 bash[36902]: audit 2022-04-22T23:31:38.780910+0000 mon.a (mon.0) 17140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:39 smithi115 bash[45495]: cluster 2022-04-22T23:31:37.565339+0000 mgr.y (mgr.34107) 12452 : cluster [DBG] pgmap v12905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:39 smithi115 bash[45495]: audit 2022-04-22T23:31:38.780910+0000 mon.a (mon.0) 17140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:39 smithi115 bash[46934]: cluster 2022-04-22T23:31:37.565339+0000 mgr.y (mgr.34107) 12452 : cluster [DBG] pgmap v12905: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:39 smithi115 bash[46934]: audit 2022-04-22T23:31:38.780910+0000 mon.a (mon.0) 17140 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:41 smithi121 bash[36902]: cluster 2022-04-22T23:31:39.566437+0000 mgr.y (mgr.34107) 12453 : cluster [DBG] pgmap v12906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:41.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:41 smithi121 bash[36902]: audit 2022-04-22T23:31:40.564071+0000 mon.a (mon.0) 17141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:41 smithi115 bash[45495]: cluster 2022-04-22T23:31:39.566437+0000 mgr.y (mgr.34107) 12453 : cluster [DBG] pgmap v12906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:41.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:41 smithi115 bash[45495]: audit 2022-04-22T23:31:40.564071+0000 mon.a (mon.0) 17141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:41 smithi115 bash[46934]: cluster 2022-04-22T23:31:39.566437+0000 mgr.y (mgr.34107) 12453 : cluster [DBG] pgmap v12906: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:41.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:41 smithi115 bash[46934]: audit 2022-04-22T23:31:40.564071+0000 mon.a (mon.0) 17141 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:42.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:42 smithi121 bash[36902]: audit 2022-04-22T23:31:42.138873+0000 mon.a (mon.0) 17142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:42.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:42 smithi115 bash[45495]: audit 2022-04-22T23:31:42.138873+0000 mon.a (mon.0) 17142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:42.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:42 smithi115 bash[46934]: audit 2022-04-22T23:31:42.138873+0000 mon.a (mon.0) 17142 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:43.319 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:43.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:43.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:43 smithi121 bash[36902]: cluster 2022-04-22T23:31:41.567496+0000 mgr.y (mgr.34107) 12454 : cluster [DBG] pgmap v12907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:43.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:43 smithi115 bash[46934]: cluster 2022-04-22T23:31:41.567496+0000 mgr.y (mgr.34107) 12454 : cluster [DBG] pgmap v12907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:43.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:43 smithi115 bash[45495]: cluster 2022-04-22T23:31:41.567496+0000 mgr.y (mgr.34107) 12454 : cluster [DBG] pgmap v12907: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:44.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:44 smithi121 bash[36902]: audit 2022-04-22T23:31:43.783985+0000 mon.a (mon.0) 17143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:44.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:44 smithi115 bash[45495]: audit 2022-04-22T23:31:43.783985+0000 mon.a (mon.0) 17143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:44.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:44 smithi115 bash[46934]: audit 2022-04-22T23:31:43.783985+0000 mon.a (mon.0) 17143 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:44.917Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:45.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:45 smithi121 bash[36902]: cluster 2022-04-22T23:31:43.568701+0000 mgr.y (mgr.34107) 12455 : cluster [DBG] pgmap v12908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:45.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:45 smithi115 bash[45495]: cluster 2022-04-22T23:31:43.568701+0000 mgr.y (mgr.34107) 12455 : cluster [DBG] pgmap v12908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:45.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:45 smithi115 bash[46934]: cluster 2022-04-22T23:31:43.568701+0000 mgr.y (mgr.34107) 12455 : cluster [DBG] pgmap v12908: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:46.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:46 smithi121 bash[36902]: audit 2022-04-22T23:31:45.567856+0000 mon.a (mon.0) 17144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:46.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:46 smithi115 bash[45495]: audit 2022-04-22T23:31:45.567856+0000 mon.a (mon.0) 17144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:46.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:46 smithi115 bash[46934]: audit 2022-04-22T23:31:45.567856+0000 mon.a (mon.0) 17144 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:47 smithi121 bash[36902]: cluster 2022-04-22T23:31:45.569880+0000 mgr.y (mgr.34107) 12456 : cluster [DBG] pgmap v12909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:47.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:47 smithi121 bash[36902]: audit 2022-04-22T23:31:47.142260+0000 mon.a (mon.0) 17145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:47 smithi115 bash[45495]: cluster 2022-04-22T23:31:45.569880+0000 mgr.y (mgr.34107) 12456 : cluster [DBG] pgmap v12909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:47.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:47 smithi115 bash[45495]: audit 2022-04-22T23:31:47.142260+0000 mon.a (mon.0) 17145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:47 smithi115 bash[46934]: cluster 2022-04-22T23:31:45.569880+0000 mgr.y (mgr.34107) 12456 : cluster [DBG] pgmap v12909: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:47.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:47 smithi115 bash[46934]: audit 2022-04-22T23:31:47.142260+0000 mon.a (mon.0) 17145 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:49 smithi121 bash[36902]: cluster 2022-04-22T23:31:47.570972+0000 mgr.y (mgr.34107) 12457 : cluster [DBG] pgmap v12910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:49.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:49 smithi121 bash[36902]: audit 2022-04-22T23:31:48.787360+0000 mon.a (mon.0) 17146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:49 smithi115 bash[45495]: cluster 2022-04-22T23:31:47.570972+0000 mgr.y (mgr.34107) 12457 : cluster [DBG] pgmap v12910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:49.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:49 smithi115 bash[45495]: audit 2022-04-22T23:31:48.787360+0000 mon.a (mon.0) 17146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:49 smithi115 bash[46934]: cluster 2022-04-22T23:31:47.570972+0000 mgr.y (mgr.34107) 12457 : cluster [DBG] pgmap v12910: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:49.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:49 smithi115 bash[46934]: audit 2022-04-22T23:31:48.787360+0000 mon.a (mon.0) 17146 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:51 smithi121 bash[36902]: cluster 2022-04-22T23:31:49.572090+0000 mgr.y (mgr.34107) 12458 : cluster [DBG] pgmap v12911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:51 smithi121 bash[36902]: audit 2022-04-22T23:31:50.571761+0000 mon.a (mon.0) 17147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:51 smithi115 bash[46934]: cluster 2022-04-22T23:31:49.572090+0000 mgr.y (mgr.34107) 12458 : cluster [DBG] pgmap v12911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:51.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:51 smithi115 bash[46934]: audit 2022-04-22T23:31:50.571761+0000 mon.a (mon.0) 17147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:51 smithi115 bash[45495]: cluster 2022-04-22T23:31:49.572090+0000 mgr.y (mgr.34107) 12458 : cluster [DBG] pgmap v12911: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:51.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:51 smithi115 bash[45495]: audit 2022-04-22T23:31:50.571761+0000 mon.a (mon.0) 17147 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:51.969 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:31:52.328 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:31:52.328 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 3m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:31:52.328 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 4m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:31:52.328 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 3m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:31:52.329 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 4m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:31:52.329 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 4m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:31:52.329 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 3m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:31:52.329 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 3m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:31:52.329 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 4m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:31:52.330 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 3m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:31:52.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 3m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:31:52.330 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 4m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:31:52.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 3m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:31:52.330 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 3m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:31:52.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 3m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:31:52.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 3m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:31:52.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 4m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:31:52.331 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 4m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:31:52.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 4m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:31:52.332 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 4m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:31:52.332 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 4m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:31:52.730 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:31:52.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:31:52.732 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:31:52.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:52 smithi121 bash[36902]: audit 2022-04-22T23:31:52.145041+0000 mon.a (mon.0) 17148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:52.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:52 smithi115 bash[45495]: audit 2022-04-22T23:31:52.145041+0000 mon.a (mon.0) 17148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:52 smithi115 bash[46934]: audit 2022-04-22T23:31:52.145041+0000 mon.a (mon.0) 17148 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:53.357 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:53.056Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:53 smithi121 bash[36902]: cluster 2022-04-22T23:31:51.573168+0000 mgr.y (mgr.34107) 12459 : cluster [DBG] pgmap v12912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:53 smithi121 bash[36902]: audit 2022-04-22T23:31:51.947656+0000 mgr.y (mgr.34107) 12460 : audit [DBG] from='client.48231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:53 smithi121 bash[36902]: audit 2022-04-22T23:31:52.300794+0000 mgr.y (mgr.34107) 12461 : audit [DBG] from='client.58177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:53 smithi121 bash[36902]: audit 2022-04-22T23:31:52.726215+0000 mon.c (mon.1) 420 : audit [DBG] from='client.? 172.21.15.115:0/2132917599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[45495]: cluster 2022-04-22T23:31:51.573168+0000 mgr.y (mgr.34107) 12459 : cluster [DBG] pgmap v12912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[45495]: audit 2022-04-22T23:31:51.947656+0000 mgr.y (mgr.34107) 12460 : audit [DBG] from='client.48231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[45495]: audit 2022-04-22T23:31:52.300794+0000 mgr.y (mgr.34107) 12461 : audit [DBG] from='client.58177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[45495]: audit 2022-04-22T23:31:52.726215+0000 mon.c (mon.1) 420 : audit [DBG] from='client.? 172.21.15.115:0/2132917599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[46934]: cluster 2022-04-22T23:31:51.573168+0000 mgr.y (mgr.34107) 12459 : cluster [DBG] pgmap v12912: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:53.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[46934]: audit 2022-04-22T23:31:51.947656+0000 mgr.y (mgr.34107) 12460 : audit [DBG] from='client.48231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[46934]: audit 2022-04-22T23:31:52.300794+0000 mgr.y (mgr.34107) 12461 : audit [DBG] from='client.58177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:31:53.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:53 smithi115 bash[46934]: audit 2022-04-22T23:31:52.726215+0000 mon.c (mon.1) 420 : audit [DBG] from='client.? 172.21.15.115:0/2132917599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:31:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:54 smithi121 bash[36902]: audit 2022-04-22T23:31:53.790376+0000 mon.a (mon.0) 17149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:54.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:54 smithi115 bash[45495]: audit 2022-04-22T23:31:53.790376+0000 mon.a (mon.0) 17149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:54.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:54 smithi115 bash[46934]: audit 2022-04-22T23:31:53.790376+0000 mon.a (mon.0) 17149 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:55.290 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:31:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:31:54.919Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:31:55.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:55 smithi121 bash[36902]: cluster 2022-04-22T23:31:53.574254+0000 mgr.y (mgr.34107) 12462 : cluster [DBG] pgmap v12913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:55.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:55 smithi115 bash[45495]: cluster 2022-04-22T23:31:53.574254+0000 mgr.y (mgr.34107) 12462 : cluster [DBG] pgmap v12913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:55.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:55 smithi115 bash[46934]: cluster 2022-04-22T23:31:53.574254+0000 mgr.y (mgr.34107) 12462 : cluster [DBG] pgmap v12913: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:56 smithi121 bash[36902]: audit 2022-04-22T23:31:55.576849+0000 mon.a (mon.0) 17150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:56.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:56 smithi115 bash[45495]: audit 2022-04-22T23:31:55.576849+0000 mon.a (mon.0) 17150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:56.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:56 smithi115 bash[46934]: audit 2022-04-22T23:31:55.576849+0000 mon.a (mon.0) 17150 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:31:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:57 smithi121 bash[36902]: cluster 2022-04-22T23:31:55.575336+0000 mgr.y (mgr.34107) 12463 : cluster [DBG] pgmap v12914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:57 smithi121 bash[36902]: audit 2022-04-22T23:31:57.147623+0000 mon.a (mon.0) 17151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:57 smithi115 bash[46934]: cluster 2022-04-22T23:31:55.575336+0000 mgr.y (mgr.34107) 12463 : cluster [DBG] pgmap v12914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:57.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:57 smithi115 bash[46934]: audit 2022-04-22T23:31:57.147623+0000 mon.a (mon.0) 17151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:57 smithi115 bash[45495]: cluster 2022-04-22T23:31:55.575336+0000 mgr.y (mgr.34107) 12463 : cluster [DBG] pgmap v12914: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:57.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:57 smithi115 bash[45495]: audit 2022-04-22T23:31:57.147623+0000 mon.a (mon.0) 17151 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:31:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:59 smithi121 bash[36902]: cluster 2022-04-22T23:31:57.576434+0000 mgr.y (mgr.34107) 12464 : cluster [DBG] pgmap v12915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:31:59 smithi121 bash[36902]: audit 2022-04-22T23:31:58.793661+0000 mon.a (mon.0) 17152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:59 smithi115 bash[45495]: cluster 2022-04-22T23:31:57.576434+0000 mgr.y (mgr.34107) 12464 : cluster [DBG] pgmap v12915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:31:59 smithi115 bash[45495]: audit 2022-04-22T23:31:58.793661+0000 mon.a (mon.0) 17152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:31:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:59 smithi115 bash[46934]: cluster 2022-04-22T23:31:57.576434+0000 mgr.y (mgr.34107) 12464 : cluster [DBG] pgmap v12915: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:31:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:31:59 smithi115 bash[46934]: audit 2022-04-22T23:31:58.793661+0000 mon.a (mon.0) 17152 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:01 smithi121 bash[36902]: cluster 2022-04-22T23:31:59.577550+0000 mgr.y (mgr.34107) 12465 : cluster [DBG] pgmap v12916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:01 smithi121 bash[36902]: audit 2022-04-22T23:32:00.579684+0000 mon.a (mon.0) 17153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:01 smithi115 bash[45495]: cluster 2022-04-22T23:31:59.577550+0000 mgr.y (mgr.34107) 12465 : cluster [DBG] pgmap v12916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:01 smithi115 bash[45495]: audit 2022-04-22T23:32:00.579684+0000 mon.a (mon.0) 17153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:01 smithi115 bash[46934]: cluster 2022-04-22T23:31:59.577550+0000 mgr.y (mgr.34107) 12465 : cluster [DBG] pgmap v12916: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:01.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:01 smithi115 bash[46934]: audit 2022-04-22T23:32:00.579684+0000 mon.a (mon.0) 17153 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:02 smithi121 bash[36902]: audit 2022-04-22T23:32:02.150696+0000 mon.a (mon.0) 17154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:02 smithi115 bash[46934]: audit 2022-04-22T23:32:02.150696+0000 mon.a (mon.0) 17154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:02.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:02 smithi115 bash[45495]: audit 2022-04-22T23:32:02.150696+0000 mon.a (mon.0) 17154 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:03.395 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:03.057Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:03 smithi121 bash[36902]: cluster 2022-04-22T23:32:01.578674+0000 mgr.y (mgr.34107) 12466 : cluster [DBG] pgmap v12917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:03.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:03 smithi115 bash[45495]: cluster 2022-04-22T23:32:01.578674+0000 mgr.y (mgr.34107) 12466 : cluster [DBG] pgmap v12917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:03.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:03 smithi115 bash[46934]: cluster 2022-04-22T23:32:01.578674+0000 mgr.y (mgr.34107) 12466 : cluster [DBG] pgmap v12917: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:04 smithi121 bash[36902]: audit 2022-04-22T23:32:03.796855+0000 mon.a (mon.0) 17155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:04 smithi115 bash[45495]: audit 2022-04-22T23:32:03.796855+0000 mon.a (mon.0) 17155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:04 smithi115 bash[46934]: audit 2022-04-22T23:32:03.796855+0000 mon.a (mon.0) 17155 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:04.918Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:05 smithi121 bash[36902]: cluster 2022-04-22T23:32:03.579789+0000 mgr.y (mgr.34107) 12467 : cluster [DBG] pgmap v12918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:05 smithi115 bash[45495]: cluster 2022-04-22T23:32:03.579789+0000 mgr.y (mgr.34107) 12467 : cluster [DBG] pgmap v12918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:05 smithi115 bash[46934]: cluster 2022-04-22T23:32:03.579789+0000 mgr.y (mgr.34107) 12467 : cluster [DBG] pgmap v12918: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:06 smithi121 bash[36902]: audit 2022-04-22T23:32:05.583185+0000 mon.a (mon.0) 17156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:06 smithi115 bash[45495]: audit 2022-04-22T23:32:05.583185+0000 mon.a (mon.0) 17156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:06 smithi115 bash[46934]: audit 2022-04-22T23:32:05.583185+0000 mon.a (mon.0) 17156 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:07 smithi121 bash[36902]: cluster 2022-04-22T23:32:05.580928+0000 mgr.y (mgr.34107) 12468 : cluster [DBG] pgmap v12919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:07 smithi121 bash[36902]: audit 2022-04-22T23:32:07.153574+0000 mon.a (mon.0) 17157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:07 smithi115 bash[45495]: cluster 2022-04-22T23:32:05.580928+0000 mgr.y (mgr.34107) 12468 : cluster [DBG] pgmap v12919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:07 smithi115 bash[45495]: audit 2022-04-22T23:32:07.153574+0000 mon.a (mon.0) 17157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:07 smithi115 bash[46934]: cluster 2022-04-22T23:32:05.580928+0000 mgr.y (mgr.34107) 12468 : cluster [DBG] pgmap v12919: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:07 smithi115 bash[46934]: audit 2022-04-22T23:32:07.153574+0000 mon.a (mon.0) 17157 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:09 smithi121 bash[36902]: cluster 2022-04-22T23:32:07.582174+0000 mgr.y (mgr.34107) 12469 : cluster [DBG] pgmap v12920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:09 smithi121 bash[36902]: audit 2022-04-22T23:32:08.799783+0000 mon.a (mon.0) 17158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:09 smithi115 bash[45495]: cluster 2022-04-22T23:32:07.582174+0000 mgr.y (mgr.34107) 12469 : cluster [DBG] pgmap v12920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:09 smithi115 bash[45495]: audit 2022-04-22T23:32:08.799783+0000 mon.a (mon.0) 17158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:09 smithi115 bash[46934]: cluster 2022-04-22T23:32:07.582174+0000 mgr.y (mgr.34107) 12469 : cluster [DBG] pgmap v12920: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:09 smithi115 bash[46934]: audit 2022-04-22T23:32:08.799783+0000 mon.a (mon.0) 17158 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:10 smithi121 bash[36902]: audit 2022-04-22T23:32:10.001111+0000 mon.a (mon.0) 17159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:32:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:10 smithi121 bash[36902]: audit 2022-04-22T23:32:10.004187+0000 mon.a (mon.0) 17160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:32:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:10 smithi121 bash[36902]: audit 2022-04-22T23:32:10.006000+0000 mon.a (mon.0) 17161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:32:10.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:10 smithi121 bash[36902]: audit 2022-04-22T23:32:10.364861+0000 mon.a (mon.0) 17162 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:32:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[45495]: audit 2022-04-22T23:32:10.001111+0000 mon.a (mon.0) 17159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:32:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[45495]: audit 2022-04-22T23:32:10.004187+0000 mon.a (mon.0) 17160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:32:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[45495]: audit 2022-04-22T23:32:10.006000+0000 mon.a (mon.0) 17161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:32:10.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[45495]: audit 2022-04-22T23:32:10.364861+0000 mon.a (mon.0) 17162 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:32:10.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[46934]: audit 2022-04-22T23:32:10.001111+0000 mon.a (mon.0) 17159 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:32:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[46934]: audit 2022-04-22T23:32:10.004187+0000 mon.a (mon.0) 17160 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:32:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[46934]: audit 2022-04-22T23:32:10.006000+0000 mon.a (mon.0) 17161 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:32:10.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:10 smithi115 bash[46934]: audit 2022-04-22T23:32:10.364861+0000 mon.a (mon.0) 17162 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:32:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:11 smithi121 bash[36902]: cluster 2022-04-22T23:32:09.583370+0000 mgr.y (mgr.34107) 12470 : cluster [DBG] pgmap v12921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:11.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:11 smithi121 bash[36902]: audit 2022-04-22T23:32:10.586981+0000 mon.a (mon.0) 17163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:11.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:11 smithi115 bash[45495]: cluster 2022-04-22T23:32:09.583370+0000 mgr.y (mgr.34107) 12470 : cluster [DBG] pgmap v12921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:11.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:11 smithi115 bash[45495]: audit 2022-04-22T23:32:10.586981+0000 mon.a (mon.0) 17163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:11 smithi115 bash[46934]: cluster 2022-04-22T23:32:09.583370+0000 mgr.y (mgr.34107) 12470 : cluster [DBG] pgmap v12921: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:11.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:11 smithi115 bash[46934]: audit 2022-04-22T23:32:10.586981+0000 mon.a (mon.0) 17163 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:12 smithi121 bash[36902]: audit 2022-04-22T23:32:12.156527+0000 mon.a (mon.0) 17164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:12 smithi115 bash[45495]: audit 2022-04-22T23:32:12.156527+0000 mon.a (mon.0) 17164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:12 smithi115 bash[46934]: audit 2022-04-22T23:32:12.156527+0000 mon.a (mon.0) 17164 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:13.440 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:13.060Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:13 smithi121 bash[36902]: cluster 2022-04-22T23:32:11.584551+0000 mgr.y (mgr.34107) 12471 : cluster [DBG] pgmap v12922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:13 smithi115 bash[45495]: cluster 2022-04-22T23:32:11.584551+0000 mgr.y (mgr.34107) 12471 : cluster [DBG] pgmap v12922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:13 smithi115 bash[46934]: cluster 2022-04-22T23:32:11.584551+0000 mgr.y (mgr.34107) 12471 : cluster [DBG] pgmap v12922: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:14 smithi121 bash[36902]: audit 2022-04-22T23:32:13.802972+0000 mon.a (mon.0) 17165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:14 smithi115 bash[45495]: audit 2022-04-22T23:32:13.802972+0000 mon.a (mon.0) 17165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:14 smithi115 bash[46934]: audit 2022-04-22T23:32:13.802972+0000 mon.a (mon.0) 17165 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:14.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:15 smithi121 bash[36902]: cluster 2022-04-22T23:32:13.585711+0000 mgr.y (mgr.34107) 12472 : cluster [DBG] pgmap v12923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:15 smithi115 bash[45495]: cluster 2022-04-22T23:32:13.585711+0000 mgr.y (mgr.34107) 12472 : cluster [DBG] pgmap v12923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:15 smithi115 bash[46934]: cluster 2022-04-22T23:32:13.585711+0000 mgr.y (mgr.34107) 12472 : cluster [DBG] pgmap v12923: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:16 smithi121 bash[36902]: audit 2022-04-22T23:32:15.590691+0000 mon.a (mon.0) 17166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:16 smithi115 bash[45495]: audit 2022-04-22T23:32:15.590691+0000 mon.a (mon.0) 17166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:16 smithi115 bash[46934]: audit 2022-04-22T23:32:15.590691+0000 mon.a (mon.0) 17166 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:17 smithi121 bash[36902]: cluster 2022-04-22T23:32:15.586801+0000 mgr.y (mgr.34107) 12473 : cluster [DBG] pgmap v12924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:17 smithi121 bash[36902]: audit 2022-04-22T23:32:17.159351+0000 mon.a (mon.0) 17167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:17 smithi115 bash[45495]: cluster 2022-04-22T23:32:15.586801+0000 mgr.y (mgr.34107) 12473 : cluster [DBG] pgmap v12924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:17 smithi115 bash[45495]: audit 2022-04-22T23:32:17.159351+0000 mon.a (mon.0) 17167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:17 smithi115 bash[46934]: cluster 2022-04-22T23:32:15.586801+0000 mgr.y (mgr.34107) 12473 : cluster [DBG] pgmap v12924: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:17 smithi115 bash[46934]: audit 2022-04-22T23:32:17.159351+0000 mon.a (mon.0) 17167 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:18 smithi121 bash[36902]: audit 2022-04-22T23:32:17.497582+0000 mon.a (mon.0) 17168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:32:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:18 smithi121 bash[36902]: audit 2022-04-22T23:32:17.498038+0000 mon.a (mon.0) 17169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:32:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:18 smithi121 bash[36902]: cluster 2022-04-22T23:32:17.587912+0000 mgr.y (mgr.34107) 12474 : cluster [DBG] pgmap v12925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[45495]: audit 2022-04-22T23:32:17.497582+0000 mon.a (mon.0) 17168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:32:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[45495]: audit 2022-04-22T23:32:17.498038+0000 mon.a (mon.0) 17169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:32:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[45495]: cluster 2022-04-22T23:32:17.587912+0000 mgr.y (mgr.34107) 12474 : cluster [DBG] pgmap v12925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[46934]: audit 2022-04-22T23:32:17.497582+0000 mon.a (mon.0) 17168 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:32:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[46934]: audit 2022-04-22T23:32:17.498038+0000 mon.a (mon.0) 17169 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:32:18.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:18 smithi115 bash[46934]: cluster 2022-04-22T23:32:17.587912+0000 mgr.y (mgr.34107) 12474 : cluster [DBG] pgmap v12925: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:19 smithi121 bash[36902]: audit 2022-04-22T23:32:18.806030+0000 mon.a (mon.0) 17170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:19 smithi115 bash[45495]: audit 2022-04-22T23:32:18.806030+0000 mon.a (mon.0) 17170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:19 smithi115 bash[46934]: audit 2022-04-22T23:32:18.806030+0000 mon.a (mon.0) 17170 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:20.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:20 smithi121 bash[36902]: cluster 2022-04-22T23:32:19.588943+0000 mgr.y (mgr.34107) 12475 : cluster [DBG] pgmap v12926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:20.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:20 smithi115 bash[45495]: cluster 2022-04-22T23:32:19.588943+0000 mgr.y (mgr.34107) 12475 : cluster [DBG] pgmap v12926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:20 smithi115 bash[46934]: cluster 2022-04-22T23:32:19.588943+0000 mgr.y (mgr.34107) 12475 : cluster [DBG] pgmap v12926: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:21 smithi121 bash[36902]: audit 2022-04-22T23:32:20.594438+0000 mon.a (mon.0) 17171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:21 smithi115 bash[45495]: audit 2022-04-22T23:32:20.594438+0000 mon.a (mon.0) 17171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:21 smithi115 bash[46934]: audit 2022-04-22T23:32:20.594438+0000 mon.a (mon.0) 17171 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:22 smithi121 bash[36902]: cluster 2022-04-22T23:32:21.590166+0000 mgr.y (mgr.34107) 12476 : cluster [DBG] pgmap v12927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:22 smithi121 bash[36902]: audit 2022-04-22T23:32:22.162506+0000 mon.a (mon.0) 17172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:22.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:22 smithi115 bash[45495]: cluster 2022-04-22T23:32:21.590166+0000 mgr.y (mgr.34107) 12476 : cluster [DBG] pgmap v12927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:22 smithi115 bash[45495]: audit 2022-04-22T23:32:22.162506+0000 mon.a (mon.0) 17172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:22 smithi115 bash[46934]: cluster 2022-04-22T23:32:21.590166+0000 mgr.y (mgr.34107) 12476 : cluster [DBG] pgmap v12927: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:22 smithi115 bash[46934]: audit 2022-04-22T23:32:22.162506+0000 mon.a (mon.0) 17172 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:23.101 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 5m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 5m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:32:23.441 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 4m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 4m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 5m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 4m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 4m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 4m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 4m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:32:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 4m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:32:23.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (5h) 5m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:32:23.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:32:23.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:32:23.443 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:32:23.443 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:32:23.540 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:23.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:23.849 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:32:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:32:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:32:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:32:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:32:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:32:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:32:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:32:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:32:23.854 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:32:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:23 smithi121 bash[36902]: audit 2022-04-22T23:32:23.083962+0000 mgr.y (mgr.34107) 12477 : audit [DBG] from='client.58189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:23 smithi121 bash[36902]: audit 2022-04-22T23:32:23.418461+0000 mgr.y (mgr.34107) 12478 : audit [DBG] from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:23 smithi121 bash[36902]: audit 2022-04-22T23:32:23.809225+0000 mon.a (mon.0) 17173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:24.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:23 smithi121 bash[36902]: audit 2022-04-22T23:32:23.845606+0000 mon.c (mon.1) 421 : audit [DBG] from='client.? 172.21.15.115:0/3442273724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[45495]: audit 2022-04-22T23:32:23.083962+0000 mgr.y (mgr.34107) 12477 : audit [DBG] from='client.58189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[45495]: audit 2022-04-22T23:32:23.418461+0000 mgr.y (mgr.34107) 12478 : audit [DBG] from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[45495]: audit 2022-04-22T23:32:23.809225+0000 mon.a (mon.0) 17173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:24.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[45495]: audit 2022-04-22T23:32:23.845606+0000 mon.c (mon.1) 421 : audit [DBG] from='client.? 172.21.15.115:0/3442273724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[46934]: audit 2022-04-22T23:32:23.083962+0000 mgr.y (mgr.34107) 12477 : audit [DBG] from='client.58189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[46934]: audit 2022-04-22T23:32:23.418461+0000 mgr.y (mgr.34107) 12478 : audit [DBG] from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[46934]: audit 2022-04-22T23:32:23.809225+0000 mon.a (mon.0) 17173 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:24.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:23 smithi115 bash[46934]: audit 2022-04-22T23:32:23.845606+0000 mon.c (mon.1) 421 : audit [DBG] from='client.? 172.21.15.115:0/3442273724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:25.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:24 smithi121 bash[36902]: cluster 2022-04-22T23:32:23.591233+0000 mgr.y (mgr.34107) 12479 : cluster [DBG] pgmap v12928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:24 smithi115 bash[45495]: cluster 2022-04-22T23:32:23.591233+0000 mgr.y (mgr.34107) 12479 : cluster [DBG] pgmap v12928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:24.922Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:24 smithi115 bash[46934]: cluster 2022-04-22T23:32:23.591233+0000 mgr.y (mgr.34107) 12479 : cluster [DBG] pgmap v12928: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:26.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:25 smithi121 bash[36902]: audit 2022-04-22T23:32:25.597976+0000 mon.a (mon.0) 17174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:26.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:25 smithi115 bash[45495]: audit 2022-04-22T23:32:25.597976+0000 mon.a (mon.0) 17174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:26.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:25 smithi115 bash[46934]: audit 2022-04-22T23:32:25.597976+0000 mon.a (mon.0) 17174 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:27.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:26 smithi121 bash[36902]: cluster 2022-04-22T23:32:25.592310+0000 mgr.y (mgr.34107) 12480 : cluster [DBG] pgmap v12929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:27.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:26 smithi115 bash[45495]: cluster 2022-04-22T23:32:25.592310+0000 mgr.y (mgr.34107) 12480 : cluster [DBG] pgmap v12929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:27.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:26 smithi115 bash[46934]: cluster 2022-04-22T23:32:25.592310+0000 mgr.y (mgr.34107) 12480 : cluster [DBG] pgmap v12929: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:28.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:27 smithi121 bash[36902]: audit 2022-04-22T23:32:27.165552+0000 mon.a (mon.0) 17175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:28.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:27 smithi115 bash[45495]: audit 2022-04-22T23:32:27.165552+0000 mon.a (mon.0) 17175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:28.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:27 smithi115 bash[46934]: audit 2022-04-22T23:32:27.165552+0000 mon.a (mon.0) 17175 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:28 smithi121 bash[36902]: cluster 2022-04-22T23:32:27.593400+0000 mgr.y (mgr.34107) 12481 : cluster [DBG] pgmap v12930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:28 smithi121 bash[36902]: audit 2022-04-22T23:32:28.812283+0000 mon.a (mon.0) 17176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:28 smithi115 bash[45495]: cluster 2022-04-22T23:32:27.593400+0000 mgr.y (mgr.34107) 12481 : cluster [DBG] pgmap v12930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:28 smithi115 bash[45495]: audit 2022-04-22T23:32:28.812283+0000 mon.a (mon.0) 17176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:29.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:28 smithi115 bash[46934]: cluster 2022-04-22T23:32:27.593400+0000 mgr.y (mgr.34107) 12481 : cluster [DBG] pgmap v12930: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:28 smithi115 bash[46934]: audit 2022-04-22T23:32:28.812283+0000 mon.a (mon.0) 17176 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:30 smithi121 bash[36902]: cluster 2022-04-22T23:32:29.594532+0000 mgr.y (mgr.34107) 12482 : cluster [DBG] pgmap v12931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:30 smithi121 bash[36902]: audit 2022-04-22T23:32:30.601566+0000 mon.a (mon.0) 17177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:30 smithi115 bash[45495]: cluster 2022-04-22T23:32:29.594532+0000 mgr.y (mgr.34107) 12482 : cluster [DBG] pgmap v12931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:30 smithi115 bash[45495]: audit 2022-04-22T23:32:30.601566+0000 mon.a (mon.0) 17177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:30 smithi115 bash[46934]: cluster 2022-04-22T23:32:29.594532+0000 mgr.y (mgr.34107) 12482 : cluster [DBG] pgmap v12931: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:30 smithi115 bash[46934]: audit 2022-04-22T23:32:30.601566+0000 mon.a (mon.0) 17177 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:32 smithi121 bash[36902]: cluster 2022-04-22T23:32:31.595681+0000 mgr.y (mgr.34107) 12483 : cluster [DBG] pgmap v12932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:32 smithi121 bash[36902]: audit 2022-04-22T23:32:32.168461+0000 mon.a (mon.0) 17178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:33.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:33.059Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:32 smithi115 bash[45495]: cluster 2022-04-22T23:32:31.595681+0000 mgr.y (mgr.34107) 12483 : cluster [DBG] pgmap v12932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:33.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:32 smithi115 bash[45495]: audit 2022-04-22T23:32:32.168461+0000 mon.a (mon.0) 17178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:32 smithi115 bash[46934]: cluster 2022-04-22T23:32:31.595681+0000 mgr.y (mgr.34107) 12483 : cluster [DBG] pgmap v12932: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:33.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:32 smithi115 bash[46934]: audit 2022-04-22T23:32:32.168461+0000 mon.a (mon.0) 17178 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:33 smithi121 bash[36902]: audit 2022-04-22T23:32:33.815169+0000 mon.a (mon.0) 17179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:34.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:33 smithi115 bash[45495]: audit 2022-04-22T23:32:33.815169+0000 mon.a (mon.0) 17179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:33 smithi115 bash[46934]: audit 2022-04-22T23:32:33.815169+0000 mon.a (mon.0) 17179 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:34 smithi121 bash[36902]: cluster 2022-04-22T23:32:33.596819+0000 mgr.y (mgr.34107) 12484 : cluster [DBG] pgmap v12933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:35.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:34 smithi115 bash[45495]: cluster 2022-04-22T23:32:33.596819+0000 mgr.y (mgr.34107) 12484 : cluster [DBG] pgmap v12933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:34.920Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:34 smithi115 bash[46934]: cluster 2022-04-22T23:32:33.596819+0000 mgr.y (mgr.34107) 12484 : cluster [DBG] pgmap v12933: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:35 smithi121 bash[36902]: audit 2022-04-22T23:32:35.605313+0000 mon.a (mon.0) 17180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:35 smithi115 bash[45495]: audit 2022-04-22T23:32:35.605313+0000 mon.a (mon.0) 17180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:35 smithi115 bash[46934]: audit 2022-04-22T23:32:35.605313+0000 mon.a (mon.0) 17180 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:36 smithi121 bash[36902]: cluster 2022-04-22T23:32:35.597896+0000 mgr.y (mgr.34107) 12485 : cluster [DBG] pgmap v12934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:36 smithi115 bash[45495]: cluster 2022-04-22T23:32:35.597896+0000 mgr.y (mgr.34107) 12485 : cluster [DBG] pgmap v12934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:36 smithi115 bash[46934]: cluster 2022-04-22T23:32:35.597896+0000 mgr.y (mgr.34107) 12485 : cluster [DBG] pgmap v12934: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:37 smithi121 bash[36902]: audit 2022-04-22T23:32:37.171165+0000 mon.a (mon.0) 17181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:37 smithi115 bash[46934]: audit 2022-04-22T23:32:37.171165+0000 mon.a (mon.0) 17181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:37 smithi115 bash[45495]: audit 2022-04-22T23:32:37.171165+0000 mon.a (mon.0) 17181 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:38 smithi121 bash[36902]: cluster 2022-04-22T23:32:37.598982+0000 mgr.y (mgr.34107) 12486 : cluster [DBG] pgmap v12935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:38 smithi121 bash[36902]: audit 2022-04-22T23:32:38.818330+0000 mon.a (mon.0) 17182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:38 smithi115 bash[45495]: cluster 2022-04-22T23:32:37.598982+0000 mgr.y (mgr.34107) 12486 : cluster [DBG] pgmap v12935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:38 smithi115 bash[45495]: audit 2022-04-22T23:32:38.818330+0000 mon.a (mon.0) 17182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:38 smithi115 bash[46934]: cluster 2022-04-22T23:32:37.598982+0000 mgr.y (mgr.34107) 12486 : cluster [DBG] pgmap v12935: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:39.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:38 smithi115 bash[46934]: audit 2022-04-22T23:32:38.818330+0000 mon.a (mon.0) 17182 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:40 smithi121 bash[36902]: cluster 2022-04-22T23:32:39.600139+0000 mgr.y (mgr.34107) 12487 : cluster [DBG] pgmap v12936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:40 smithi121 bash[36902]: audit 2022-04-22T23:32:40.609201+0000 mon.a (mon.0) 17183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:41.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:40 smithi115 bash[45495]: cluster 2022-04-22T23:32:39.600139+0000 mgr.y (mgr.34107) 12487 : cluster [DBG] pgmap v12936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:40 smithi115 bash[45495]: audit 2022-04-22T23:32:40.609201+0000 mon.a (mon.0) 17183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:40 smithi115 bash[46934]: cluster 2022-04-22T23:32:39.600139+0000 mgr.y (mgr.34107) 12487 : cluster [DBG] pgmap v12936: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:40 smithi115 bash[46934]: audit 2022-04-22T23:32:40.609201+0000 mon.a (mon.0) 17183 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:42 smithi121 bash[36902]: cluster 2022-04-22T23:32:41.601214+0000 mgr.y (mgr.34107) 12488 : cluster [DBG] pgmap v12937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:42 smithi121 bash[36902]: audit 2022-04-22T23:32:42.174403+0000 mon.a (mon.0) 17184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:43.060Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:42 smithi115 bash[45495]: cluster 2022-04-22T23:32:41.601214+0000 mgr.y (mgr.34107) 12488 : cluster [DBG] pgmap v12937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:42 smithi115 bash[45495]: audit 2022-04-22T23:32:42.174403+0000 mon.a (mon.0) 17184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:42 smithi115 bash[46934]: cluster 2022-04-22T23:32:41.601214+0000 mgr.y (mgr.34107) 12488 : cluster [DBG] pgmap v12937: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:43.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:42 smithi115 bash[46934]: audit 2022-04-22T23:32:42.174403+0000 mon.a (mon.0) 17184 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:43 smithi121 bash[36902]: audit 2022-04-22T23:32:43.821348+0000 mon.a (mon.0) 17185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:43 smithi115 bash[45495]: audit 2022-04-22T23:32:43.821348+0000 mon.a (mon.0) 17185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:43 smithi115 bash[46934]: audit 2022-04-22T23:32:43.821348+0000 mon.a (mon.0) 17185 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:44 smithi121 bash[36902]: cluster 2022-04-22T23:32:43.602215+0000 mgr.y (mgr.34107) 12489 : cluster [DBG] pgmap v12938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:44 smithi115 bash[45495]: cluster 2022-04-22T23:32:43.602215+0000 mgr.y (mgr.34107) 12489 : cluster [DBG] pgmap v12938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:44 smithi115 bash[46934]: cluster 2022-04-22T23:32:43.602215+0000 mgr.y (mgr.34107) 12489 : cluster [DBG] pgmap v12938: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:45.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:44.922Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:45 smithi121 bash[36902]: audit 2022-04-22T23:32:45.612951+0000 mon.a (mon.0) 17186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:46.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:45 smithi115 bash[45495]: audit 2022-04-22T23:32:45.612951+0000 mon.a (mon.0) 17186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:45 smithi115 bash[46934]: audit 2022-04-22T23:32:45.612951+0000 mon.a (mon.0) 17186 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:46 smithi121 bash[36902]: cluster 2022-04-22T23:32:45.603247+0000 mgr.y (mgr.34107) 12490 : cluster [DBG] pgmap v12939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:47.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:46 smithi115 bash[45495]: cluster 2022-04-22T23:32:45.603247+0000 mgr.y (mgr.34107) 12490 : cluster [DBG] pgmap v12939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:46 smithi115 bash[46934]: cluster 2022-04-22T23:32:45.603247+0000 mgr.y (mgr.34107) 12490 : cluster [DBG] pgmap v12939: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:47 smithi121 bash[36902]: audit 2022-04-22T23:32:47.177772+0000 mon.a (mon.0) 17187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:47 smithi115 bash[45495]: audit 2022-04-22T23:32:47.177772+0000 mon.a (mon.0) 17187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:47 smithi115 bash[46934]: audit 2022-04-22T23:32:47.177772+0000 mon.a (mon.0) 17187 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:48 smithi121 bash[36902]: cluster 2022-04-22T23:32:47.604368+0000 mgr.y (mgr.34107) 12491 : cluster [DBG] pgmap v12940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:48 smithi121 bash[36902]: audit 2022-04-22T23:32:48.824523+0000 mon.a (mon.0) 17188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:48 smithi115 bash[45495]: cluster 2022-04-22T23:32:47.604368+0000 mgr.y (mgr.34107) 12491 : cluster [DBG] pgmap v12940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:48 smithi115 bash[45495]: audit 2022-04-22T23:32:48.824523+0000 mon.a (mon.0) 17188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:48 smithi115 bash[46934]: cluster 2022-04-22T23:32:47.604368+0000 mgr.y (mgr.34107) 12491 : cluster [DBG] pgmap v12940: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:49.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:48 smithi115 bash[46934]: audit 2022-04-22T23:32:48.824523+0000 mon.a (mon.0) 17188 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:50 smithi121 bash[36902]: cluster 2022-04-22T23:32:49.605377+0000 mgr.y (mgr.34107) 12492 : cluster [DBG] pgmap v12941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:50 smithi121 bash[36902]: audit 2022-04-22T23:32:50.616612+0000 mon.a (mon.0) 17189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:50 smithi115 bash[46934]: cluster 2022-04-22T23:32:49.605377+0000 mgr.y (mgr.34107) 12492 : cluster [DBG] pgmap v12941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:50 smithi115 bash[46934]: audit 2022-04-22T23:32:50.616612+0000 mon.a (mon.0) 17189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:50 smithi115 bash[45495]: cluster 2022-04-22T23:32:49.605377+0000 mgr.y (mgr.34107) 12492 : cluster [DBG] pgmap v12941: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:50 smithi115 bash[45495]: audit 2022-04-22T23:32:50.616612+0000 mon.a (mon.0) 17189 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:52 smithi121 bash[36902]: cluster 2022-04-22T23:32:51.606480+0000 mgr.y (mgr.34107) 12493 : cluster [DBG] pgmap v12942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:52 smithi121 bash[36902]: audit 2022-04-22T23:32:52.180923+0000 mon.a (mon.0) 17190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:53.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:53.061Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:52 smithi115 bash[45495]: cluster 2022-04-22T23:32:51.606480+0000 mgr.y (mgr.34107) 12493 : cluster [DBG] pgmap v12942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:52 smithi115 bash[45495]: audit 2022-04-22T23:32:52.180923+0000 mon.a (mon.0) 17190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:52 smithi115 bash[46934]: cluster 2022-04-22T23:32:51.606480+0000 mgr.y (mgr.34107) 12493 : cluster [DBG] pgmap v12942: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:53.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:52 smithi115 bash[46934]: audit 2022-04-22T23:32:52.180923+0000 mon.a (mon.0) 17190 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:54.227 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:32:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:53 smithi121 bash[36902]: audit 2022-04-22T23:32:53.827762+0000 mon.a (mon.0) 17191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:54.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:53 smithi115 bash[45495]: audit 2022-04-22T23:32:53.827762+0000 mon.a (mon.0) 17191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:53 smithi115 bash[46934]: audit 2022-04-22T23:32:53.827762+0000 mon.a (mon.0) 17191 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:54.572 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:32:54.572 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 4m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:32:54.572 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 5m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:32:54.572 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 4m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 5m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 5m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 4m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 4m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 5m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 4m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:32:54.573 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 4m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 5m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 4m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 4m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 4m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 4m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 5m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:32:54.574 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (5h) 5m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:32:54.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 5m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:32:54.575 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 5m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:32:54.575 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 5m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:32:54.980 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:32:54.981 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:32:54.982 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:32:54.982 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:32:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:54 smithi121 bash[36902]: cluster 2022-04-22T23:32:53.607619+0000 mgr.y (mgr.34107) 12494 : cluster [DBG] pgmap v12943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:54 smithi121 bash[36902]: audit 2022-04-22T23:32:54.204858+0000 mgr.y (mgr.34107) 12495 : audit [DBG] from='client.58207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:54 smithi121 bash[36902]: audit 2022-04-22T23:32:54.976066+0000 mon.c (mon.1) 422 : audit [DBG] from='client.? 172.21.15.115:0/527079974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:32:54.936Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:32:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[45495]: cluster 2022-04-22T23:32:53.607619+0000 mgr.y (mgr.34107) 12494 : cluster [DBG] pgmap v12943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[45495]: audit 2022-04-22T23:32:54.204858+0000 mgr.y (mgr.34107) 12495 : audit [DBG] from='client.58207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[45495]: audit 2022-04-22T23:32:54.976066+0000 mon.c (mon.1) 422 : audit [DBG] from='client.? 172.21.15.115:0/527079974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[46934]: cluster 2022-04-22T23:32:53.607619+0000 mgr.y (mgr.34107) 12494 : cluster [DBG] pgmap v12943: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[46934]: audit 2022-04-22T23:32:54.204858+0000 mgr.y (mgr.34107) 12495 : audit [DBG] from='client.58207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:55.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:54 smithi115 bash[46934]: audit 2022-04-22T23:32:54.976066+0000 mon.c (mon.1) 422 : audit [DBG] from='client.? 172.21.15.115:0/527079974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:32:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:56 smithi121 bash[36902]: audit 2022-04-22T23:32:54.548834+0000 mgr.y (mgr.34107) 12496 : audit [DBG] from='client.48273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:56 smithi121 bash[36902]: audit 2022-04-22T23:32:55.620322+0000 mon.a (mon.0) 17192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:56.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:55 smithi115 bash[45495]: audit 2022-04-22T23:32:54.548834+0000 mgr.y (mgr.34107) 12496 : audit [DBG] from='client.48273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:55 smithi115 bash[45495]: audit 2022-04-22T23:32:55.620322+0000 mon.a (mon.0) 17192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:55 smithi115 bash[46934]: audit 2022-04-22T23:32:54.548834+0000 mgr.y (mgr.34107) 12496 : audit [DBG] from='client.48273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:32:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:55 smithi115 bash[46934]: audit 2022-04-22T23:32:55.620322+0000 mon.a (mon.0) 17192 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:32:57.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:57 smithi121 bash[36902]: cluster 2022-04-22T23:32:55.608684+0000 mgr.y (mgr.34107) 12497 : cluster [DBG] pgmap v12944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:57.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:56 smithi115 bash[45495]: cluster 2022-04-22T23:32:55.608684+0000 mgr.y (mgr.34107) 12497 : cluster [DBG] pgmap v12944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:57 smithi115 bash[46934]: cluster 2022-04-22T23:32:55.608684+0000 mgr.y (mgr.34107) 12497 : cluster [DBG] pgmap v12944: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:58.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:58 smithi121 bash[36902]: audit 2022-04-22T23:32:57.183762+0000 mon.a (mon.0) 17193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:58 smithi115 bash[45495]: audit 2022-04-22T23:32:57.183762+0000 mon.a (mon.0) 17193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:58 smithi115 bash[46934]: audit 2022-04-22T23:32:57.183762+0000 mon.a (mon.0) 17193 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:32:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:59 smithi115 bash[45495]: cluster 2022-04-22T23:32:57.609766+0000 mgr.y (mgr.34107) 12498 : cluster [DBG] pgmap v12945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:32:59 smithi115 bash[45495]: audit 2022-04-22T23:32:58.830971+0000 mon.a (mon.0) 17194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:59 smithi115 bash[46934]: cluster 2022-04-22T23:32:57.609766+0000 mgr.y (mgr.34107) 12498 : cluster [DBG] pgmap v12945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:32:59 smithi115 bash[46934]: audit 2022-04-22T23:32:58.830971+0000 mon.a (mon.0) 17194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:32:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:59 smithi121 bash[36902]: cluster 2022-04-22T23:32:57.609766+0000 mgr.y (mgr.34107) 12498 : cluster [DBG] pgmap v12945: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:32:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:32:59 smithi121 bash[36902]: audit 2022-04-22T23:32:58.830971+0000 mon.a (mon.0) 17194 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:01 smithi115 bash[45495]: cluster 2022-04-22T23:32:59.610961+0000 mgr.y (mgr.34107) 12499 : cluster [DBG] pgmap v12946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:01 smithi115 bash[45495]: audit 2022-04-22T23:33:00.623813+0000 mon.a (mon.0) 17195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:01 smithi115 bash[46934]: cluster 2022-04-22T23:32:59.610961+0000 mgr.y (mgr.34107) 12499 : cluster [DBG] pgmap v12946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:01.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:01 smithi115 bash[46934]: audit 2022-04-22T23:33:00.623813+0000 mon.a (mon.0) 17195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:01 smithi121 bash[36902]: cluster 2022-04-22T23:32:59.610961+0000 mgr.y (mgr.34107) 12499 : cluster [DBG] pgmap v12946: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:01 smithi121 bash[36902]: audit 2022-04-22T23:33:00.623813+0000 mon.a (mon.0) 17195 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:03 smithi115 bash[45495]: cluster 2022-04-22T23:33:01.612125+0000 mgr.y (mgr.34107) 12500 : cluster [DBG] pgmap v12947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:03.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:03 smithi115 bash[45495]: audit 2022-04-22T23:33:02.186152+0000 mon.a (mon.0) 17196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:03.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:03.062Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:03.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:03 smithi115 bash[46934]: cluster 2022-04-22T23:33:01.612125+0000 mgr.y (mgr.34107) 12500 : cluster [DBG] pgmap v12947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:03.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:03 smithi115 bash[46934]: audit 2022-04-22T23:33:02.186152+0000 mon.a (mon.0) 17196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:03 smithi121 bash[36902]: cluster 2022-04-22T23:33:01.612125+0000 mgr.y (mgr.34107) 12500 : cluster [DBG] pgmap v12947: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:03.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:03 smithi121 bash[36902]: audit 2022-04-22T23:33:02.186152+0000 mon.a (mon.0) 17196 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:04.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:04 smithi115 bash[45495]: audit 2022-04-22T23:33:03.833941+0000 mon.a (mon.0) 17197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:04.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:04 smithi115 bash[46934]: audit 2022-04-22T23:33:03.833941+0000 mon.a (mon.0) 17197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:04.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:04 smithi121 bash[36902]: audit 2022-04-22T23:33:03.833941+0000 mon.a (mon.0) 17197 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:05 smithi115 bash[45495]: cluster 2022-04-22T23:33:03.613238+0000 mgr.y (mgr.34107) 12501 : cluster [DBG] pgmap v12948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:05 smithi115 bash[46934]: cluster 2022-04-22T23:33:03.613238+0000 mgr.y (mgr.34107) 12501 : cluster [DBG] pgmap v12948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:04.923Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:05 smithi121 bash[36902]: cluster 2022-04-22T23:33:03.613238+0000 mgr.y (mgr.34107) 12501 : cluster [DBG] pgmap v12948: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:06 smithi121 bash[36902]: audit 2022-04-22T23:33:05.627671+0000 mon.a (mon.0) 17198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:06.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:06 smithi115 bash[45495]: audit 2022-04-22T23:33:05.627671+0000 mon.a (mon.0) 17198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:06 smithi115 bash[46934]: audit 2022-04-22T23:33:05.627671+0000 mon.a (mon.0) 17198 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:07 smithi121 bash[36902]: cluster 2022-04-22T23:33:05.614324+0000 mgr.y (mgr.34107) 12502 : cluster [DBG] pgmap v12949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:07.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:07 smithi115 bash[45495]: cluster 2022-04-22T23:33:05.614324+0000 mgr.y (mgr.34107) 12502 : cluster [DBG] pgmap v12949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:07 smithi115 bash[46934]: cluster 2022-04-22T23:33:05.614324+0000 mgr.y (mgr.34107) 12502 : cluster [DBG] pgmap v12949: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:08 smithi121 bash[36902]: audit 2022-04-22T23:33:07.189187+0000 mon.a (mon.0) 17199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:08.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:08 smithi115 bash[45495]: audit 2022-04-22T23:33:07.189187+0000 mon.a (mon.0) 17199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:08 smithi115 bash[46934]: audit 2022-04-22T23:33:07.189187+0000 mon.a (mon.0) 17199 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:09 smithi121 bash[36902]: cluster 2022-04-22T23:33:07.615452+0000 mgr.y (mgr.34107) 12503 : cluster [DBG] pgmap v12950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:09 smithi121 bash[36902]: audit 2022-04-22T23:33:08.837259+0000 mon.a (mon.0) 17200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:09 smithi115 bash[45495]: cluster 2022-04-22T23:33:07.615452+0000 mgr.y (mgr.34107) 12503 : cluster [DBG] pgmap v12950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:09 smithi115 bash[45495]: audit 2022-04-22T23:33:08.837259+0000 mon.a (mon.0) 17200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:09 smithi115 bash[46934]: cluster 2022-04-22T23:33:07.615452+0000 mgr.y (mgr.34107) 12503 : cluster [DBG] pgmap v12950: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:09 smithi115 bash[46934]: audit 2022-04-22T23:33:08.837259+0000 mon.a (mon.0) 17200 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: cluster 2022-04-22T23:33:09.616502+0000 mgr.y (mgr.34107) 12504 : cluster [DBG] pgmap v12951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: audit 2022-04-22T23:33:10.376093+0000 mon.a (mon.0) 17201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:33:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: audit 2022-04-22T23:33:10.379349+0000 mon.a (mon.0) 17202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:33:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: audit 2022-04-22T23:33:10.381072+0000 mon.a (mon.0) 17203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:33:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: audit 2022-04-22T23:33:10.631088+0000 mon.a (mon.0) 17204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:11 smithi121 bash[36902]: audit 2022-04-22T23:33:10.724634+0000 mon.a (mon.0) 17205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: cluster 2022-04-22T23:33:09.616502+0000 mgr.y (mgr.34107) 12504 : cluster [DBG] pgmap v12951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: audit 2022-04-22T23:33:10.376093+0000 mon.a (mon.0) 17201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: audit 2022-04-22T23:33:10.379349+0000 mon.a (mon.0) 17202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: audit 2022-04-22T23:33:10.381072+0000 mon.a (mon.0) 17203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: audit 2022-04-22T23:33:10.631088+0000 mon.a (mon.0) 17204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[45495]: audit 2022-04-22T23:33:10.724634+0000 mon.a (mon.0) 17205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: cluster 2022-04-22T23:33:09.616502+0000 mgr.y (mgr.34107) 12504 : cluster [DBG] pgmap v12951: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: audit 2022-04-22T23:33:10.376093+0000 mon.a (mon.0) 17201 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: audit 2022-04-22T23:33:10.379349+0000 mon.a (mon.0) 17202 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: audit 2022-04-22T23:33:10.381072+0000 mon.a (mon.0) 17203 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: audit 2022-04-22T23:33:10.631088+0000 mon.a (mon.0) 17204 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:11 smithi115 bash[46934]: audit 2022-04-22T23:33:10.724634+0000 mon.a (mon.0) 17205 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:33:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:13 smithi121 bash[36902]: cluster 2022-04-22T23:33:11.617473+0000 mgr.y (mgr.34107) 12505 : cluster [DBG] pgmap v12952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:13 smithi121 bash[36902]: audit 2022-04-22T23:33:12.192408+0000 mon.a (mon.0) 17206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:13.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:13.063Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:13 smithi115 bash[45495]: cluster 2022-04-22T23:33:11.617473+0000 mgr.y (mgr.34107) 12505 : cluster [DBG] pgmap v12952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:13 smithi115 bash[45495]: audit 2022-04-22T23:33:12.192408+0000 mon.a (mon.0) 17206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:13 smithi115 bash[46934]: cluster 2022-04-22T23:33:11.617473+0000 mgr.y (mgr.34107) 12505 : cluster [DBG] pgmap v12952: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:13 smithi115 bash[46934]: audit 2022-04-22T23:33:12.192408+0000 mon.a (mon.0) 17206 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:14 smithi121 bash[36902]: audit 2022-04-22T23:33:13.840416+0000 mon.a (mon.0) 17207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:14.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:14 smithi115 bash[45495]: audit 2022-04-22T23:33:13.840416+0000 mon.a (mon.0) 17207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:14 smithi115 bash[46934]: audit 2022-04-22T23:33:13.840416+0000 mon.a (mon.0) 17207 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:15 smithi115 bash[45495]: cluster 2022-04-22T23:33:13.618564+0000 mgr.y (mgr.34107) 12506 : cluster [DBG] pgmap v12953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:14.925Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:15 smithi115 bash[46934]: cluster 2022-04-22T23:33:13.618564+0000 mgr.y (mgr.34107) 12506 : cluster [DBG] pgmap v12953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:15 smithi121 bash[36902]: cluster 2022-04-22T23:33:13.618564+0000 mgr.y (mgr.34107) 12506 : cluster [DBG] pgmap v12953: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:16 smithi121 bash[36902]: audit 2022-04-22T23:33:15.634634+0000 mon.a (mon.0) 17208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:16.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:16 smithi115 bash[45495]: audit 2022-04-22T23:33:15.634634+0000 mon.a (mon.0) 17208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:16 smithi115 bash[46934]: audit 2022-04-22T23:33:15.634634+0000 mon.a (mon.0) 17208 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:17 smithi121 bash[36902]: cluster 2022-04-22T23:33:15.619648+0000 mgr.y (mgr.34107) 12507 : cluster [DBG] pgmap v12954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:17.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:17 smithi115 bash[45495]: cluster 2022-04-22T23:33:15.619648+0000 mgr.y (mgr.34107) 12507 : cluster [DBG] pgmap v12954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:17 smithi115 bash[46934]: cluster 2022-04-22T23:33:15.619648+0000 mgr.y (mgr.34107) 12507 : cluster [DBG] pgmap v12954: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:18 smithi121 bash[36902]: audit 2022-04-22T23:33:17.195548+0000 mon.a (mon.0) 17209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:18 smithi121 bash[36902]: audit 2022-04-22T23:33:17.509997+0000 mon.a (mon.0) 17210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:33:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:18 smithi121 bash[36902]: audit 2022-04-22T23:33:17.510731+0000 mon.a (mon.0) 17211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[45495]: audit 2022-04-22T23:33:17.195548+0000 mon.a (mon.0) 17209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[45495]: audit 2022-04-22T23:33:17.509997+0000 mon.a (mon.0) 17210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[45495]: audit 2022-04-22T23:33:17.510731+0000 mon.a (mon.0) 17211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[46934]: audit 2022-04-22T23:33:17.195548+0000 mon.a (mon.0) 17209 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[46934]: audit 2022-04-22T23:33:17.509997+0000 mon.a (mon.0) 17210 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:33:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:18 smithi115 bash[46934]: audit 2022-04-22T23:33:17.510731+0000 mon.a (mon.0) 17211 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:33:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:19 smithi121 bash[36902]: cluster 2022-04-22T23:33:17.620677+0000 mgr.y (mgr.34107) 12508 : cluster [DBG] pgmap v12955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:19 smithi121 bash[36902]: audit 2022-04-22T23:33:18.843494+0000 mon.a (mon.0) 17212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:19 smithi115 bash[45495]: cluster 2022-04-22T23:33:17.620677+0000 mgr.y (mgr.34107) 12508 : cluster [DBG] pgmap v12955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:19 smithi115 bash[45495]: audit 2022-04-22T23:33:18.843494+0000 mon.a (mon.0) 17212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:19 smithi115 bash[46934]: cluster 2022-04-22T23:33:17.620677+0000 mgr.y (mgr.34107) 12508 : cluster [DBG] pgmap v12955: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:19 smithi115 bash[46934]: audit 2022-04-22T23:33:18.843494+0000 mon.a (mon.0) 17212 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:21 smithi121 bash[36902]: cluster 2022-04-22T23:33:19.621726+0000 mgr.y (mgr.34107) 12509 : cluster [DBG] pgmap v12956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:21 smithi121 bash[36902]: audit 2022-04-22T23:33:20.638169+0000 mon.a (mon.0) 17213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:21 smithi115 bash[45495]: cluster 2022-04-22T23:33:19.621726+0000 mgr.y (mgr.34107) 12509 : cluster [DBG] pgmap v12956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:21 smithi115 bash[45495]: audit 2022-04-22T23:33:20.638169+0000 mon.a (mon.0) 17213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:21 smithi115 bash[46934]: cluster 2022-04-22T23:33:19.621726+0000 mgr.y (mgr.34107) 12509 : cluster [DBG] pgmap v12956: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:21.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:21 smithi115 bash[46934]: audit 2022-04-22T23:33:20.638169+0000 mon.a (mon.0) 17213 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:23 smithi121 bash[36902]: cluster 2022-04-22T23:33:21.622828+0000 mgr.y (mgr.34107) 12510 : cluster [DBG] pgmap v12957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:23 smithi121 bash[36902]: audit 2022-04-22T23:33:22.198797+0000 mon.a (mon.0) 17214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:23.540 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:23.064Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:23 smithi115 bash[45495]: cluster 2022-04-22T23:33:21.622828+0000 mgr.y (mgr.34107) 12510 : cluster [DBG] pgmap v12957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:23 smithi115 bash[45495]: audit 2022-04-22T23:33:22.198797+0000 mon.a (mon.0) 17214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:23 smithi115 bash[46934]: cluster 2022-04-22T23:33:21.622828+0000 mgr.y (mgr.34107) 12510 : cluster [DBG] pgmap v12957: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:23 smithi115 bash[46934]: audit 2022-04-22T23:33:22.198797+0000 mon.a (mon.0) 17214 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:24 smithi121 bash[36902]: audit 2022-04-22T23:33:23.846573+0000 mon.a (mon.0) 17215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:24 smithi115 bash[45495]: audit 2022-04-22T23:33:23.846573+0000 mon.a (mon.0) 17215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:24 smithi115 bash[46934]: audit 2022-04-22T23:33:23.846573+0000 mon.a (mon.0) 17215 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:25 smithi115 bash[46934]: cluster 2022-04-22T23:33:23.624043+0000 mgr.y (mgr.34107) 12511 : cluster [DBG] pgmap v12958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:25 smithi115 bash[45495]: cluster 2022-04-22T23:33:23.624043+0000 mgr.y (mgr.34107) 12511 : cluster [DBG] pgmap v12958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:24.925Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:25.349 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:33:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:25 smithi121 bash[36902]: cluster 2022-04-22T23:33:23.624043+0000 mgr.y (mgr.34107) 12511 : cluster [DBG] pgmap v12958: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:25.694 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:33:25.694 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:33:25.694 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 6m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 6m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 5m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 5m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 6m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 5m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:33:25.695 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 5m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 5m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 5m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 5m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 6m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 6m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (5h) 6m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:33:25.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:33:25.697 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:33:26.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:33:26.105 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:33:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:26 smithi121 bash[36902]: audit 2022-04-22T23:33:25.330190+0000 mgr.y (mgr.34107) 12512 : audit [DBG] from='client.48285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:26 smithi121 bash[36902]: audit 2022-04-22T23:33:25.641894+0000 mon.a (mon.0) 17216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:26 smithi121 bash[36902]: audit 2022-04-22T23:33:26.100340+0000 mon.a (mon.0) 17217 : audit [DBG] from='client.? 172.21.15.115:0/2545598325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[45495]: audit 2022-04-22T23:33:25.330190+0000 mgr.y (mgr.34107) 12512 : audit [DBG] from='client.48285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[45495]: audit 2022-04-22T23:33:25.641894+0000 mon.a (mon.0) 17216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:26.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[45495]: audit 2022-04-22T23:33:26.100340+0000 mon.a (mon.0) 17217 : audit [DBG] from='client.? 172.21.15.115:0/2545598325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[46934]: audit 2022-04-22T23:33:25.330190+0000 mgr.y (mgr.34107) 12512 : audit [DBG] from='client.48285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[46934]: audit 2022-04-22T23:33:25.641894+0000 mon.a (mon.0) 17216 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:26.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:26 smithi115 bash[46934]: audit 2022-04-22T23:33:26.100340+0000 mon.a (mon.0) 17217 : audit [DBG] from='client.? 172.21.15.115:0/2545598325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:27 smithi121 bash[36902]: cluster 2022-04-22T23:33:25.625091+0000 mgr.y (mgr.34107) 12513 : cluster [DBG] pgmap v12959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:27 smithi121 bash[36902]: audit 2022-04-22T23:33:25.671902+0000 mgr.y (mgr.34107) 12514 : audit [DBG] from='client.58231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:27 smithi115 bash[45495]: cluster 2022-04-22T23:33:25.625091+0000 mgr.y (mgr.34107) 12513 : cluster [DBG] pgmap v12959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:27.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:27 smithi115 bash[45495]: audit 2022-04-22T23:33:25.671902+0000 mgr.y (mgr.34107) 12514 : audit [DBG] from='client.58231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:27 smithi115 bash[46934]: cluster 2022-04-22T23:33:25.625091+0000 mgr.y (mgr.34107) 12513 : cluster [DBG] pgmap v12959: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:27.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:27 smithi115 bash[46934]: audit 2022-04-22T23:33:25.671902+0000 mgr.y (mgr.34107) 12514 : audit [DBG] from='client.58231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:28 smithi121 bash[36902]: audit 2022-04-22T23:33:27.201937+0000 mon.a (mon.0) 17218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:28 smithi115 bash[45495]: audit 2022-04-22T23:33:27.201937+0000 mon.a (mon.0) 17218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:28 smithi115 bash[46934]: audit 2022-04-22T23:33:27.201937+0000 mon.a (mon.0) 17218 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:29 smithi121 bash[36902]: cluster 2022-04-22T23:33:27.626301+0000 mgr.y (mgr.34107) 12515 : cluster [DBG] pgmap v12960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:29 smithi121 bash[36902]: audit 2022-04-22T23:33:28.849710+0000 mon.a (mon.0) 17219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:29 smithi115 bash[45495]: cluster 2022-04-22T23:33:27.626301+0000 mgr.y (mgr.34107) 12515 : cluster [DBG] pgmap v12960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:29 smithi115 bash[45495]: audit 2022-04-22T23:33:28.849710+0000 mon.a (mon.0) 17219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:29 smithi115 bash[46934]: cluster 2022-04-22T23:33:27.626301+0000 mgr.y (mgr.34107) 12515 : cluster [DBG] pgmap v12960: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:29.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:29 smithi115 bash[46934]: audit 2022-04-22T23:33:28.849710+0000 mon.a (mon.0) 17219 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:31 smithi121 bash[36902]: cluster 2022-04-22T23:33:29.627526+0000 mgr.y (mgr.34107) 12516 : cluster [DBG] pgmap v12961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:31 smithi121 bash[36902]: audit 2022-04-22T23:33:30.645536+0000 mon.a (mon.0) 17220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:31 smithi115 bash[45495]: cluster 2022-04-22T23:33:29.627526+0000 mgr.y (mgr.34107) 12516 : cluster [DBG] pgmap v12961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:31 smithi115 bash[45495]: audit 2022-04-22T23:33:30.645536+0000 mon.a (mon.0) 17220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:31 smithi115 bash[46934]: cluster 2022-04-22T23:33:29.627526+0000 mgr.y (mgr.34107) 12516 : cluster [DBG] pgmap v12961: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:31 smithi115 bash[46934]: audit 2022-04-22T23:33:30.645536+0000 mon.a (mon.0) 17220 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:33 smithi121 bash[36902]: cluster 2022-04-22T23:33:31.628772+0000 mgr.y (mgr.34107) 12517 : cluster [DBG] pgmap v12962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:33 smithi121 bash[36902]: audit 2022-04-22T23:33:32.205088+0000 mon.a (mon.0) 17221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:33.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:33.065Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:33 smithi115 bash[45495]: cluster 2022-04-22T23:33:31.628772+0000 mgr.y (mgr.34107) 12517 : cluster [DBG] pgmap v12962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:33.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:33 smithi115 bash[45495]: audit 2022-04-22T23:33:32.205088+0000 mon.a (mon.0) 17221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:33 smithi115 bash[46934]: cluster 2022-04-22T23:33:31.628772+0000 mgr.y (mgr.34107) 12517 : cluster [DBG] pgmap v12962: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:33 smithi115 bash[46934]: audit 2022-04-22T23:33:32.205088+0000 mon.a (mon.0) 17221 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:34 smithi121 bash[36902]: audit 2022-04-22T23:33:33.852671+0000 mon.a (mon.0) 17222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:34 smithi115 bash[45495]: audit 2022-04-22T23:33:33.852671+0000 mon.a (mon.0) 17222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:34 smithi115 bash[46934]: audit 2022-04-22T23:33:33.852671+0000 mon.a (mon.0) 17222 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:35 smithi115 bash[45495]: cluster 2022-04-22T23:33:33.629946+0000 mgr.y (mgr.34107) 12518 : cluster [DBG] pgmap v12963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:34.928Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:35 smithi115 bash[46934]: cluster 2022-04-22T23:33:33.629946+0000 mgr.y (mgr.34107) 12518 : cluster [DBG] pgmap v12963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:35 smithi121 bash[36902]: cluster 2022-04-22T23:33:33.629946+0000 mgr.y (mgr.34107) 12518 : cluster [DBG] pgmap v12963: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:36 smithi121 bash[36902]: audit 2022-04-22T23:33:35.649050+0000 mon.a (mon.0) 17223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:36 smithi115 bash[45495]: audit 2022-04-22T23:33:35.649050+0000 mon.a (mon.0) 17223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:36.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:36 smithi115 bash[46934]: audit 2022-04-22T23:33:35.649050+0000 mon.a (mon.0) 17223 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:37 smithi121 bash[36902]: cluster 2022-04-22T23:33:35.631039+0000 mgr.y (mgr.34107) 12519 : cluster [DBG] pgmap v12964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:37 smithi115 bash[45495]: cluster 2022-04-22T23:33:35.631039+0000 mgr.y (mgr.34107) 12519 : cluster [DBG] pgmap v12964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:37 smithi115 bash[46934]: cluster 2022-04-22T23:33:35.631039+0000 mgr.y (mgr.34107) 12519 : cluster [DBG] pgmap v12964: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:38 smithi121 bash[36902]: audit 2022-04-22T23:33:37.208076+0000 mon.a (mon.0) 17224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:38 smithi115 bash[45495]: audit 2022-04-22T23:33:37.208076+0000 mon.a (mon.0) 17224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:38.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:38 smithi115 bash[46934]: audit 2022-04-22T23:33:37.208076+0000 mon.a (mon.0) 17224 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:39 smithi121 bash[36902]: cluster 2022-04-22T23:33:37.632134+0000 mgr.y (mgr.34107) 12520 : cluster [DBG] pgmap v12965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:39 smithi121 bash[36902]: audit 2022-04-22T23:33:38.856071+0000 mon.a (mon.0) 17225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:39 smithi115 bash[45495]: cluster 2022-04-22T23:33:37.632134+0000 mgr.y (mgr.34107) 12520 : cluster [DBG] pgmap v12965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:39 smithi115 bash[45495]: audit 2022-04-22T23:33:38.856071+0000 mon.a (mon.0) 17225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:39 smithi115 bash[46934]: cluster 2022-04-22T23:33:37.632134+0000 mgr.y (mgr.34107) 12520 : cluster [DBG] pgmap v12965: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:39 smithi115 bash[46934]: audit 2022-04-22T23:33:38.856071+0000 mon.a (mon.0) 17225 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:41 smithi121 bash[36902]: cluster 2022-04-22T23:33:39.633198+0000 mgr.y (mgr.34107) 12521 : cluster [DBG] pgmap v12966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:41 smithi121 bash[36902]: audit 2022-04-22T23:33:40.652837+0000 mon.a (mon.0) 17226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:41.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:41 smithi115 bash[45495]: cluster 2022-04-22T23:33:39.633198+0000 mgr.y (mgr.34107) 12521 : cluster [DBG] pgmap v12966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:41.583 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:41 smithi115 bash[45495]: audit 2022-04-22T23:33:40.652837+0000 mon.a (mon.0) 17226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:41.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:41 smithi115 bash[46934]: cluster 2022-04-22T23:33:39.633198+0000 mgr.y (mgr.34107) 12521 : cluster [DBG] pgmap v12966: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:41.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:41 smithi115 bash[46934]: audit 2022-04-22T23:33:40.652837+0000 mon.a (mon.0) 17226 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:43 smithi121 bash[36902]: cluster 2022-04-22T23:33:41.634311+0000 mgr.y (mgr.34107) 12522 : cluster [DBG] pgmap v12967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:43 smithi121 bash[36902]: audit 2022-04-22T23:33:42.210895+0000 mon.a (mon.0) 17227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:43.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:43.065Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:43 smithi115 bash[45495]: cluster 2022-04-22T23:33:41.634311+0000 mgr.y (mgr.34107) 12522 : cluster [DBG] pgmap v12967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:43 smithi115 bash[45495]: audit 2022-04-22T23:33:42.210895+0000 mon.a (mon.0) 17227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:43.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:43 smithi115 bash[46934]: cluster 2022-04-22T23:33:41.634311+0000 mgr.y (mgr.34107) 12522 : cluster [DBG] pgmap v12967: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:43.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:43 smithi115 bash[46934]: audit 2022-04-22T23:33:42.210895+0000 mon.a (mon.0) 17227 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:44 smithi121 bash[36902]: audit 2022-04-22T23:33:43.859043+0000 mon.a (mon.0) 17228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:44 smithi115 bash[45495]: audit 2022-04-22T23:33:43.859043+0000 mon.a (mon.0) 17228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:44 smithi115 bash[46934]: audit 2022-04-22T23:33:43.859043+0000 mon.a (mon.0) 17228 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:45.210 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:44.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:45 smithi121 bash[36902]: cluster 2022-04-22T23:33:43.635466+0000 mgr.y (mgr.34107) 12523 : cluster [DBG] pgmap v12968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:45 smithi115 bash[45495]: cluster 2022-04-22T23:33:43.635466+0000 mgr.y (mgr.34107) 12523 : cluster [DBG] pgmap v12968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:45 smithi115 bash[46934]: cluster 2022-04-22T23:33:43.635466+0000 mgr.y (mgr.34107) 12523 : cluster [DBG] pgmap v12968: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:46 smithi121 bash[36902]: audit 2022-04-22T23:33:45.656933+0000 mon.a (mon.0) 17229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:46.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:46 smithi115 bash[45495]: audit 2022-04-22T23:33:45.656933+0000 mon.a (mon.0) 17229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:46 smithi115 bash[46934]: audit 2022-04-22T23:33:45.656933+0000 mon.a (mon.0) 17229 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:47 smithi121 bash[36902]: cluster 2022-04-22T23:33:45.636511+0000 mgr.y (mgr.34107) 12524 : cluster [DBG] pgmap v12969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:47 smithi121 bash[36902]: audit 2022-04-22T23:33:47.213792+0000 mon.a (mon.0) 17230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:47 smithi115 bash[45495]: cluster 2022-04-22T23:33:45.636511+0000 mgr.y (mgr.34107) 12524 : cluster [DBG] pgmap v12969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:47 smithi115 bash[45495]: audit 2022-04-22T23:33:47.213792+0000 mon.a (mon.0) 17230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:47 smithi115 bash[46934]: cluster 2022-04-22T23:33:45.636511+0000 mgr.y (mgr.34107) 12524 : cluster [DBG] pgmap v12969: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:47 smithi115 bash[46934]: audit 2022-04-22T23:33:47.213792+0000 mon.a (mon.0) 17230 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:49 smithi121 bash[36902]: cluster 2022-04-22T23:33:47.637613+0000 mgr.y (mgr.34107) 12525 : cluster [DBG] pgmap v12970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:49 smithi121 bash[36902]: audit 2022-04-22T23:33:48.862244+0000 mon.a (mon.0) 17231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:49 smithi115 bash[45495]: cluster 2022-04-22T23:33:47.637613+0000 mgr.y (mgr.34107) 12525 : cluster [DBG] pgmap v12970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:49 smithi115 bash[45495]: audit 2022-04-22T23:33:48.862244+0000 mon.a (mon.0) 17231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:49 smithi115 bash[46934]: cluster 2022-04-22T23:33:47.637613+0000 mgr.y (mgr.34107) 12525 : cluster [DBG] pgmap v12970: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:49.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:49 smithi115 bash[46934]: audit 2022-04-22T23:33:48.862244+0000 mon.a (mon.0) 17231 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:51 smithi121 bash[36902]: cluster 2022-04-22T23:33:49.638729+0000 mgr.y (mgr.34107) 12526 : cluster [DBG] pgmap v12971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:51.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:51 smithi121 bash[36902]: audit 2022-04-22T23:33:50.660624+0000 mon.a (mon.0) 17232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:51 smithi115 bash[45495]: cluster 2022-04-22T23:33:49.638729+0000 mgr.y (mgr.34107) 12526 : cluster [DBG] pgmap v12971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:51 smithi115 bash[45495]: audit 2022-04-22T23:33:50.660624+0000 mon.a (mon.0) 17232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:51 smithi115 bash[46934]: cluster 2022-04-22T23:33:49.638729+0000 mgr.y (mgr.34107) 12526 : cluster [DBG] pgmap v12971: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:51 smithi115 bash[46934]: audit 2022-04-22T23:33:50.660624+0000 mon.a (mon.0) 17232 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:52.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:52 smithi121 bash[36902]: audit 2022-04-22T23:33:52.216853+0000 mon.a (mon.0) 17233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:52.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:52 smithi115 bash[45495]: audit 2022-04-22T23:33:52.216853+0000 mon.a (mon.0) 17233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:52.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:52 smithi115 bash[46934]: audit 2022-04-22T23:33:52.216853+0000 mon.a (mon.0) 17233 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:53.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:53 smithi121 bash[36902]: cluster 2022-04-22T23:33:51.639783+0000 mgr.y (mgr.34107) 12527 : cluster [DBG] pgmap v12972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:53.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:53.067Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:53 smithi115 bash[45495]: cluster 2022-04-22T23:33:51.639783+0000 mgr.y (mgr.34107) 12527 : cluster [DBG] pgmap v12972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:53.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:53 smithi115 bash[46934]: cluster 2022-04-22T23:33:51.639783+0000 mgr.y (mgr.34107) 12527 : cluster [DBG] pgmap v12972: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:54.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:54 smithi121 bash[36902]: audit 2022-04-22T23:33:53.865429+0000 mon.a (mon.0) 17234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:54.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:54 smithi115 bash[45495]: audit 2022-04-22T23:33:53.865429+0000 mon.a (mon.0) 17234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:54.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:54 smithi115 bash[46934]: audit 2022-04-22T23:33:53.865429+0000 mon.a (mon.0) 17234 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:55.249 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:33:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:33:54.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:33:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:55 smithi121 bash[36902]: cluster 2022-04-22T23:33:53.641014+0000 mgr.y (mgr.34107) 12528 : cluster [DBG] pgmap v12973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:55 smithi115 bash[45495]: cluster 2022-04-22T23:33:53.641014+0000 mgr.y (mgr.34107) 12528 : cluster [DBG] pgmap v12973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:55 smithi115 bash[46934]: cluster 2022-04-22T23:33:53.641014+0000 mgr.y (mgr.34107) 12528 : cluster [DBG] pgmap v12973: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:56.489 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:33:56.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:56 smithi121 bash[36902]: audit 2022-04-22T23:33:55.664341+0000 mon.a (mon.0) 17235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:56.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:56 smithi115 bash[46934]: audit 2022-04-22T23:33:55.664341+0000 mon.a (mon.0) 17235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:56.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:56 smithi115 bash[45495]: audit 2022-04-22T23:33:55.664341+0000 mon.a (mon.0) 17235 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:33:56.834 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:33:56.834 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 5m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:33:56.834 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 6m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:33:56.834 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 5m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 6m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 6m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 5m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 5m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 6m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 5m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 5m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 6m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 5m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:33:56.835 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 5m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 5m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 5m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 6m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 6m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 6m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 6m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:33:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 6m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:33:57.240 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:33:57.240 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:33:57.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:33:57.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:57.241 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:33:57.241 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:33:57.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:57.241 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:33:57.242 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:33:57.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:57.242 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:33:57.242 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:33:57.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:33:57.243 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:33:57.243 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:33:57.243 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:33:57.243 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:33:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[45495]: cluster 2022-04-22T23:33:55.642078+0000 mgr.y (mgr.34107) 12529 : cluster [DBG] pgmap v12974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[45495]: audit 2022-04-22T23:33:56.467251+0000 mgr.y (mgr.34107) 12530 : audit [DBG] from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[45495]: audit 2022-04-22T23:33:57.219882+0000 mon.a (mon.0) 17236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:57.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[45495]: audit 2022-04-22T23:33:57.236292+0000 mon.a (mon.0) 17237 : audit [DBG] from='client.? 172.21.15.115:0/2187211020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:57.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[46934]: cluster 2022-04-22T23:33:55.642078+0000 mgr.y (mgr.34107) 12529 : cluster [DBG] pgmap v12974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[46934]: audit 2022-04-22T23:33:56.467251+0000 mgr.y (mgr.34107) 12530 : audit [DBG] from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[46934]: audit 2022-04-22T23:33:57.219882+0000 mon.a (mon.0) 17236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:57.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:57 smithi115 bash[46934]: audit 2022-04-22T23:33:57.236292+0000 mon.a (mon.0) 17237 : audit [DBG] from='client.? 172.21.15.115:0/2187211020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:57 smithi121 bash[36902]: cluster 2022-04-22T23:33:55.642078+0000 mgr.y (mgr.34107) 12529 : cluster [DBG] pgmap v12974: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:57 smithi121 bash[36902]: audit 2022-04-22T23:33:56.467251+0000 mgr.y (mgr.34107) 12530 : audit [DBG] from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:57 smithi121 bash[36902]: audit 2022-04-22T23:33:57.219882+0000 mon.a (mon.0) 17236 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:33:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:57 smithi121 bash[36902]: audit 2022-04-22T23:33:57.236292+0000 mon.a (mon.0) 17237 : audit [DBG] from='client.? 172.21.15.115:0/2187211020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:33:58.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:58 smithi115 bash[45495]: audit 2022-04-22T23:33:56.816911+0000 mgr.y (mgr.34107) 12531 : audit [DBG] from='client.48309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:58.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:58 smithi115 bash[46934]: audit 2022-04-22T23:33:56.816911+0000 mgr.y (mgr.34107) 12531 : audit [DBG] from='client.48309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:58 smithi121 bash[36902]: audit 2022-04-22T23:33:56.816911+0000 mgr.y (mgr.34107) 12531 : audit [DBG] from='client.48309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:33:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:59 smithi115 bash[46934]: cluster 2022-04-22T23:33:57.643143+0000 mgr.y (mgr.34107) 12532 : cluster [DBG] pgmap v12975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:59.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:33:59 smithi115 bash[46934]: audit 2022-04-22T23:33:58.868550+0000 mon.a (mon.0) 17238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:59.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:59 smithi115 bash[45495]: cluster 2022-04-22T23:33:57.643143+0000 mgr.y (mgr.34107) 12532 : cluster [DBG] pgmap v12975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:59.580 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:33:59 smithi115 bash[45495]: audit 2022-04-22T23:33:58.868550+0000 mon.a (mon.0) 17238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:33:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:59 smithi121 bash[36902]: cluster 2022-04-22T23:33:57.643143+0000 mgr.y (mgr.34107) 12532 : cluster [DBG] pgmap v12975: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:33:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:33:59 smithi121 bash[36902]: audit 2022-04-22T23:33:58.868550+0000 mon.a (mon.0) 17238 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:01 smithi115 bash[45495]: cluster 2022-04-22T23:33:59.644286+0000 mgr.y (mgr.34107) 12533 : cluster [DBG] pgmap v12976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:01.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:01 smithi115 bash[45495]: audit 2022-04-22T23:34:00.668171+0000 mon.a (mon.0) 17239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:01 smithi115 bash[46934]: cluster 2022-04-22T23:33:59.644286+0000 mgr.y (mgr.34107) 12533 : cluster [DBG] pgmap v12976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:01 smithi115 bash[46934]: audit 2022-04-22T23:34:00.668171+0000 mon.a (mon.0) 17239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:01 smithi121 bash[36902]: cluster 2022-04-22T23:33:59.644286+0000 mgr.y (mgr.34107) 12533 : cluster [DBG] pgmap v12976: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:01 smithi121 bash[36902]: audit 2022-04-22T23:34:00.668171+0000 mon.a (mon.0) 17239 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:02.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:02 smithi115 bash[45495]: audit 2022-04-22T23:34:02.222686+0000 mon.a (mon.0) 17240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:02.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:02 smithi115 bash[46934]: audit 2022-04-22T23:34:02.222686+0000 mon.a (mon.0) 17240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:02 smithi121 bash[36902]: audit 2022-04-22T23:34:02.222686+0000 mon.a (mon.0) 17240 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:03 smithi115 bash[45495]: cluster 2022-04-22T23:34:01.645314+0000 mgr.y (mgr.34107) 12534 : cluster [DBG] pgmap v12977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:03.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:03.067Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:03 smithi115 bash[46934]: cluster 2022-04-22T23:34:01.645314+0000 mgr.y (mgr.34107) 12534 : cluster [DBG] pgmap v12977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:03 smithi121 bash[36902]: cluster 2022-04-22T23:34:01.645314+0000 mgr.y (mgr.34107) 12534 : cluster [DBG] pgmap v12977: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:04 smithi121 bash[36902]: audit 2022-04-22T23:34:03.871952+0000 mon.a (mon.0) 17241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:04 smithi115 bash[45495]: audit 2022-04-22T23:34:03.871952+0000 mon.a (mon.0) 17241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:04 smithi115 bash[46934]: audit 2022-04-22T23:34:03.871952+0000 mon.a (mon.0) 17241 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:04.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:05 smithi121 bash[36902]: cluster 2022-04-22T23:34:03.646573+0000 mgr.y (mgr.34107) 12535 : cluster [DBG] pgmap v12978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:05 smithi115 bash[45495]: cluster 2022-04-22T23:34:03.646573+0000 mgr.y (mgr.34107) 12535 : cluster [DBG] pgmap v12978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:05 smithi115 bash[46934]: cluster 2022-04-22T23:34:03.646573+0000 mgr.y (mgr.34107) 12535 : cluster [DBG] pgmap v12978: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:06 smithi121 bash[36902]: audit 2022-04-22T23:34:05.671823+0000 mon.a (mon.0) 17242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:06 smithi115 bash[45495]: audit 2022-04-22T23:34:05.671823+0000 mon.a (mon.0) 17242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:06 smithi115 bash[46934]: audit 2022-04-22T23:34:05.671823+0000 mon.a (mon.0) 17242 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:07 smithi121 bash[36902]: cluster 2022-04-22T23:34:05.647687+0000 mgr.y (mgr.34107) 12536 : cluster [DBG] pgmap v12979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:07 smithi121 bash[36902]: audit 2022-04-22T23:34:07.225737+0000 mon.a (mon.0) 17243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:07 smithi115 bash[45495]: cluster 2022-04-22T23:34:05.647687+0000 mgr.y (mgr.34107) 12536 : cluster [DBG] pgmap v12979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:07 smithi115 bash[45495]: audit 2022-04-22T23:34:07.225737+0000 mon.a (mon.0) 17243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:07 smithi115 bash[46934]: cluster 2022-04-22T23:34:05.647687+0000 mgr.y (mgr.34107) 12536 : cluster [DBG] pgmap v12979: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:07.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:07 smithi115 bash[46934]: audit 2022-04-22T23:34:07.225737+0000 mon.a (mon.0) 17243 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:09 smithi121 bash[36902]: cluster 2022-04-22T23:34:07.648792+0000 mgr.y (mgr.34107) 12537 : cluster [DBG] pgmap v12980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:09 smithi121 bash[36902]: audit 2022-04-22T23:34:08.875134+0000 mon.a (mon.0) 17244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:09 smithi115 bash[45495]: cluster 2022-04-22T23:34:07.648792+0000 mgr.y (mgr.34107) 12537 : cluster [DBG] pgmap v12980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:09 smithi115 bash[45495]: audit 2022-04-22T23:34:08.875134+0000 mon.a (mon.0) 17244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:09 smithi115 bash[46934]: cluster 2022-04-22T23:34:07.648792+0000 mgr.y (mgr.34107) 12537 : cluster [DBG] pgmap v12980: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:09.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:09 smithi115 bash[46934]: audit 2022-04-22T23:34:08.875134+0000 mon.a (mon.0) 17244 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: cluster 2022-04-22T23:34:09.649939+0000 mgr.y (mgr.34107) 12538 : cluster [DBG] pgmap v12981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: audit 2022-04-22T23:34:10.675389+0000 mon.a (mon.0) 17245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:11.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: audit 2022-04-22T23:34:10.732298+0000 mon.a (mon.0) 17246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:34:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: audit 2022-04-22T23:34:10.735223+0000 mon.a (mon.0) 17247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:34:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: audit 2022-04-22T23:34:10.736894+0000 mon.a (mon.0) 17248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:34:11.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:11 smithi121 bash[36902]: audit 2022-04-22T23:34:11.108028+0000 mon.a (mon.0) 17249 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:34:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: cluster 2022-04-22T23:34:09.649939+0000 mgr.y (mgr.34107) 12538 : cluster [DBG] pgmap v12981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:11.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: audit 2022-04-22T23:34:10.675389+0000 mon.a (mon.0) 17245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: audit 2022-04-22T23:34:10.732298+0000 mon.a (mon.0) 17246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:34:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: audit 2022-04-22T23:34:10.735223+0000 mon.a (mon.0) 17247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:34:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: audit 2022-04-22T23:34:10.736894+0000 mon.a (mon.0) 17248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:34:11.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[45495]: audit 2022-04-22T23:34:11.108028+0000 mon.a (mon.0) 17249 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:34:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: cluster 2022-04-22T23:34:09.649939+0000 mgr.y (mgr.34107) 12538 : cluster [DBG] pgmap v12981: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: audit 2022-04-22T23:34:10.675389+0000 mon.a (mon.0) 17245 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: audit 2022-04-22T23:34:10.732298+0000 mon.a (mon.0) 17246 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:34:11.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: audit 2022-04-22T23:34:10.735223+0000 mon.a (mon.0) 17247 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:34:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: audit 2022-04-22T23:34:10.736894+0000 mon.a (mon.0) 17248 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:34:11.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:11 smithi115 bash[46934]: audit 2022-04-22T23:34:11.108028+0000 mon.a (mon.0) 17249 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:34:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:12 smithi121 bash[36902]: audit 2022-04-22T23:34:12.228731+0000 mon.a (mon.0) 17250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:12 smithi115 bash[45495]: audit 2022-04-22T23:34:12.228731+0000 mon.a (mon.0) 17250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:12 smithi115 bash[46934]: audit 2022-04-22T23:34:12.228731+0000 mon.a (mon.0) 17250 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:13.333 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:13.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:13 smithi121 bash[36902]: cluster 2022-04-22T23:34:11.651108+0000 mgr.y (mgr.34107) 12539 : cluster [DBG] pgmap v12982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:13.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:13 smithi115 bash[45495]: cluster 2022-04-22T23:34:11.651108+0000 mgr.y (mgr.34107) 12539 : cluster [DBG] pgmap v12982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:13 smithi115 bash[46934]: cluster 2022-04-22T23:34:11.651108+0000 mgr.y (mgr.34107) 12539 : cluster [DBG] pgmap v12982: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:14 smithi121 bash[36902]: audit 2022-04-22T23:34:13.878043+0000 mon.a (mon.0) 17251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:14 smithi115 bash[45495]: audit 2022-04-22T23:34:13.878043+0000 mon.a (mon.0) 17251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:14 smithi115 bash[46934]: audit 2022-04-22T23:34:13.878043+0000 mon.a (mon.0) 17251 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:14.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:15 smithi121 bash[36902]: cluster 2022-04-22T23:34:13.652253+0000 mgr.y (mgr.34107) 12540 : cluster [DBG] pgmap v12983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:15 smithi115 bash[45495]: cluster 2022-04-22T23:34:13.652253+0000 mgr.y (mgr.34107) 12540 : cluster [DBG] pgmap v12983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:15 smithi115 bash[46934]: cluster 2022-04-22T23:34:13.652253+0000 mgr.y (mgr.34107) 12540 : cluster [DBG] pgmap v12983: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:16 smithi121 bash[36902]: audit 2022-04-22T23:34:15.679101+0000 mon.a (mon.0) 17252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:16 smithi115 bash[45495]: audit 2022-04-22T23:34:15.679101+0000 mon.a (mon.0) 17252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:16 smithi115 bash[46934]: audit 2022-04-22T23:34:15.679101+0000 mon.a (mon.0) 17252 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:17 smithi121 bash[36902]: cluster 2022-04-22T23:34:15.653310+0000 mgr.y (mgr.34107) 12541 : cluster [DBG] pgmap v12984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:17 smithi121 bash[36902]: audit 2022-04-22T23:34:17.231459+0000 mon.a (mon.0) 17253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:17 smithi115 bash[45495]: cluster 2022-04-22T23:34:15.653310+0000 mgr.y (mgr.34107) 12541 : cluster [DBG] pgmap v12984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:17 smithi115 bash[45495]: audit 2022-04-22T23:34:17.231459+0000 mon.a (mon.0) 17253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:17 smithi115 bash[46934]: cluster 2022-04-22T23:34:15.653310+0000 mgr.y (mgr.34107) 12541 : cluster [DBG] pgmap v12984: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:17 smithi115 bash[46934]: audit 2022-04-22T23:34:17.231459+0000 mon.a (mon.0) 17253 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:18 smithi121 bash[36902]: audit 2022-04-22T23:34:17.522059+0000 mon.a (mon.0) 17254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:34:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:18 smithi121 bash[36902]: audit 2022-04-22T23:34:17.522575+0000 mon.a (mon.0) 17255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:34:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:18 smithi115 bash[45495]: audit 2022-04-22T23:34:17.522059+0000 mon.a (mon.0) 17254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:34:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:18 smithi115 bash[45495]: audit 2022-04-22T23:34:17.522575+0000 mon.a (mon.0) 17255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:34:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:18 smithi115 bash[46934]: audit 2022-04-22T23:34:17.522059+0000 mon.a (mon.0) 17254 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:34:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:18 smithi115 bash[46934]: audit 2022-04-22T23:34:17.522575+0000 mon.a (mon.0) 17255 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:34:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:19 smithi121 bash[36902]: cluster 2022-04-22T23:34:17.654379+0000 mgr.y (mgr.34107) 12542 : cluster [DBG] pgmap v12985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:19 smithi121 bash[36902]: audit 2022-04-22T23:34:18.881338+0000 mon.a (mon.0) 17256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:19 smithi115 bash[45495]: cluster 2022-04-22T23:34:17.654379+0000 mgr.y (mgr.34107) 12542 : cluster [DBG] pgmap v12985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:19 smithi115 bash[45495]: audit 2022-04-22T23:34:18.881338+0000 mon.a (mon.0) 17256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:19 smithi115 bash[46934]: cluster 2022-04-22T23:34:17.654379+0000 mgr.y (mgr.34107) 12542 : cluster [DBG] pgmap v12985: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:19 smithi115 bash[46934]: audit 2022-04-22T23:34:18.881338+0000 mon.a (mon.0) 17256 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:21.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:21 smithi121 bash[36902]: cluster 2022-04-22T23:34:19.655600+0000 mgr.y (mgr.34107) 12543 : cluster [DBG] pgmap v12986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:21.769 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:21 smithi121 bash[36902]: audit 2022-04-22T23:34:20.683009+0000 mon.a (mon.0) 17257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:21 smithi115 bash[45495]: cluster 2022-04-22T23:34:19.655600+0000 mgr.y (mgr.34107) 12543 : cluster [DBG] pgmap v12986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:21 smithi115 bash[45495]: audit 2022-04-22T23:34:20.683009+0000 mon.a (mon.0) 17257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:21 smithi115 bash[46934]: cluster 2022-04-22T23:34:19.655600+0000 mgr.y (mgr.34107) 12543 : cluster [DBG] pgmap v12986: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:21 smithi115 bash[46934]: audit 2022-04-22T23:34:20.683009+0000 mon.a (mon.0) 17257 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:22 smithi121 bash[36902]: audit 2022-04-22T23:34:22.234768+0000 mon.a (mon.0) 17258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:22.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:22 smithi115 bash[45495]: audit 2022-04-22T23:34:22.234768+0000 mon.a (mon.0) 17258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:22.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:22 smithi115 bash[46934]: audit 2022-04-22T23:34:22.234768+0000 mon.a (mon.0) 17258 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:23.389 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:23.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:23 smithi121 bash[36902]: cluster 2022-04-22T23:34:21.656680+0000 mgr.y (mgr.34107) 12544 : cluster [DBG] pgmap v12987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:23 smithi115 bash[46934]: cluster 2022-04-22T23:34:21.656680+0000 mgr.y (mgr.34107) 12544 : cluster [DBG] pgmap v12987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:23.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:23 smithi115 bash[45495]: cluster 2022-04-22T23:34:21.656680+0000 mgr.y (mgr.34107) 12544 : cluster [DBG] pgmap v12987: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:24.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:24 smithi121 bash[36902]: audit 2022-04-22T23:34:23.884408+0000 mon.a (mon.0) 17259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:24 smithi115 bash[46934]: audit 2022-04-22T23:34:23.884408+0000 mon.a (mon.0) 17259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:24 smithi115 bash[45495]: audit 2022-04-22T23:34:23.884408+0000 mon.a (mon.0) 17259 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:24.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:25 smithi121 bash[36902]: cluster 2022-04-22T23:34:23.657933+0000 mgr.y (mgr.34107) 12545 : cluster [DBG] pgmap v12988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:25.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:25 smithi115 bash[45495]: cluster 2022-04-22T23:34:23.657933+0000 mgr.y (mgr.34107) 12545 : cluster [DBG] pgmap v12988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:25 smithi115 bash[46934]: cluster 2022-04-22T23:34:23.657933+0000 mgr.y (mgr.34107) 12545 : cluster [DBG] pgmap v12988: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:26 smithi121 bash[36902]: audit 2022-04-22T23:34:25.686846+0000 mon.a (mon.0) 17260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:26.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:26 smithi115 bash[45495]: audit 2022-04-22T23:34:25.686846+0000 mon.a (mon.0) 17260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:26 smithi115 bash[46934]: audit 2022-04-22T23:34:25.686846+0000 mon.a (mon.0) 17260 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:27.618 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:34:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:27 smithi121 bash[36902]: cluster 2022-04-22T23:34:25.659140+0000 mgr.y (mgr.34107) 12546 : cluster [DBG] pgmap v12989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:27 smithi121 bash[36902]: audit 2022-04-22T23:34:27.237916+0000 mon.a (mon.0) 17261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:27 smithi115 bash[46934]: cluster 2022-04-22T23:34:25.659140+0000 mgr.y (mgr.34107) 12546 : cluster [DBG] pgmap v12989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:27 smithi115 bash[46934]: audit 2022-04-22T23:34:27.237916+0000 mon.a (mon.0) 17261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:27 smithi115 bash[45495]: cluster 2022-04-22T23:34:25.659140+0000 mgr.y (mgr.34107) 12546 : cluster [DBG] pgmap v12989: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:27 smithi115 bash[45495]: audit 2022-04-22T23:34:27.237916+0000 mon.a (mon.0) 17261 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:34:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:34:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 7m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (5h) 6m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 7m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 6m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 6m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 7m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 6m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:34:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 6m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 6m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 6m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 6m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 7m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 7m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 7m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:34:27.965 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (5h) 7m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:34:27.966 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:34:28.361 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:34:28.361 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:28.362 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:34:28.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:34:28.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:28.366 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:34:28.366 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:34:28.366 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:34:28.367 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:34:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:28 smithi121 bash[36902]: audit 2022-04-22T23:34:27.595755+0000 mgr.y (mgr.34107) 12547 : audit [DBG] from='client.48321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:28 smithi121 bash[36902]: cluster 2022-04-22T23:34:27.660230+0000 mgr.y (mgr.34107) 12548 : cluster [DBG] pgmap v12990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:28 smithi121 bash[36902]: audit 2022-04-22T23:34:27.940189+0000 mgr.y (mgr.34107) 12549 : audit [DBG] from='client.58267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:28 smithi121 bash[36902]: audit 2022-04-22T23:34:28.357618+0000 mon.c (mon.1) 423 : audit [DBG] from='client.? 172.21.15.115:0/2697870767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:34:29.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:28 smithi121 bash[36902]: audit 2022-04-22T23:34:28.887827+0000 mon.a (mon.0) 17262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[45495]: audit 2022-04-22T23:34:27.595755+0000 mgr.y (mgr.34107) 12547 : audit [DBG] from='client.48321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[45495]: cluster 2022-04-22T23:34:27.660230+0000 mgr.y (mgr.34107) 12548 : cluster [DBG] pgmap v12990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[45495]: audit 2022-04-22T23:34:27.940189+0000 mgr.y (mgr.34107) 12549 : audit [DBG] from='client.58267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[45495]: audit 2022-04-22T23:34:28.357618+0000 mon.c (mon.1) 423 : audit [DBG] from='client.? 172.21.15.115:0/2697870767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:34:29.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[45495]: audit 2022-04-22T23:34:28.887827+0000 mon.a (mon.0) 17262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[46934]: audit 2022-04-22T23:34:27.595755+0000 mgr.y (mgr.34107) 12547 : audit [DBG] from='client.48321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[46934]: cluster 2022-04-22T23:34:27.660230+0000 mgr.y (mgr.34107) 12548 : cluster [DBG] pgmap v12990: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[46934]: audit 2022-04-22T23:34:27.940189+0000 mgr.y (mgr.34107) 12549 : audit [DBG] from='client.58267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:34:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[46934]: audit 2022-04-22T23:34:28.357618+0000 mon.c (mon.1) 423 : audit [DBG] from='client.? 172.21.15.115:0/2697870767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:34:29.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:28 smithi115 bash[46934]: audit 2022-04-22T23:34:28.887827+0000 mon.a (mon.0) 17262 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:30 smithi121 bash[36902]: cluster 2022-04-22T23:34:29.661333+0000 mgr.y (mgr.34107) 12550 : cluster [DBG] pgmap v12991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:31.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:30 smithi121 bash[36902]: audit 2022-04-22T23:34:30.690679+0000 mon.a (mon.0) 17263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:30 smithi115 bash[45495]: cluster 2022-04-22T23:34:29.661333+0000 mgr.y (mgr.34107) 12550 : cluster [DBG] pgmap v12991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:31.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:30 smithi115 bash[45495]: audit 2022-04-22T23:34:30.690679+0000 mon.a (mon.0) 17263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:30 smithi115 bash[46934]: cluster 2022-04-22T23:34:29.661333+0000 mgr.y (mgr.34107) 12550 : cluster [DBG] pgmap v12991: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:31.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:30 smithi115 bash[46934]: audit 2022-04-22T23:34:30.690679+0000 mon.a (mon.0) 17263 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:32 smithi121 bash[36902]: cluster 2022-04-22T23:34:31.662384+0000 mgr.y (mgr.34107) 12551 : cluster [DBG] pgmap v12992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:32 smithi121 bash[36902]: audit 2022-04-22T23:34:32.241177+0000 mon.a (mon.0) 17264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:32 smithi115 bash[46934]: cluster 2022-04-22T23:34:31.662384+0000 mgr.y (mgr.34107) 12551 : cluster [DBG] pgmap v12992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:33.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:32 smithi115 bash[46934]: audit 2022-04-22T23:34:32.241177+0000 mon.a (mon.0) 17264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:32 smithi115 bash[45495]: cluster 2022-04-22T23:34:31.662384+0000 mgr.y (mgr.34107) 12551 : cluster [DBG] pgmap v12992: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:33.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:32 smithi115 bash[45495]: audit 2022-04-22T23:34:32.241177+0000 mon.a (mon.0) 17264 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:33.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:33.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:34.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:33 smithi121 bash[36902]: audit 2022-04-22T23:34:33.890875+0000 mon.a (mon.0) 17265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:34.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:33 smithi115 bash[45495]: audit 2022-04-22T23:34:33.890875+0000 mon.a (mon.0) 17265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:34.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:33 smithi115 bash[46934]: audit 2022-04-22T23:34:33.890875+0000 mon.a (mon.0) 17265 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:35.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:34 smithi121 bash[36902]: cluster 2022-04-22T23:34:33.663536+0000 mgr.y (mgr.34107) 12552 : cluster [DBG] pgmap v12993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:34.929Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:35.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:34 smithi115 bash[45495]: cluster 2022-04-22T23:34:33.663536+0000 mgr.y (mgr.34107) 12552 : cluster [DBG] pgmap v12993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:35.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:34 smithi115 bash[46934]: cluster 2022-04-22T23:34:33.663536+0000 mgr.y (mgr.34107) 12552 : cluster [DBG] pgmap v12993: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:36.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:35 smithi121 bash[36902]: audit 2022-04-22T23:34:35.694310+0000 mon.a (mon.0) 17266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:36.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:35 smithi115 bash[45495]: audit 2022-04-22T23:34:35.694310+0000 mon.a (mon.0) 17266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:36.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:35 smithi115 bash[46934]: audit 2022-04-22T23:34:35.694310+0000 mon.a (mon.0) 17266 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:37.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:36 smithi121 bash[36902]: cluster 2022-04-22T23:34:35.664539+0000 mgr.y (mgr.34107) 12553 : cluster [DBG] pgmap v12994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:37.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:36 smithi115 bash[46934]: cluster 2022-04-22T23:34:35.664539+0000 mgr.y (mgr.34107) 12553 : cluster [DBG] pgmap v12994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:37.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:36 smithi115 bash[45495]: cluster 2022-04-22T23:34:35.664539+0000 mgr.y (mgr.34107) 12553 : cluster [DBG] pgmap v12994: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:38.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:37 smithi121 bash[36902]: audit 2022-04-22T23:34:37.244219+0000 mon.a (mon.0) 17267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:38.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:37 smithi115 bash[45495]: audit 2022-04-22T23:34:37.244219+0000 mon.a (mon.0) 17267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:38.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:37 smithi115 bash[46934]: audit 2022-04-22T23:34:37.244219+0000 mon.a (mon.0) 17267 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:38 smithi121 bash[36902]: cluster 2022-04-22T23:34:37.665654+0000 mgr.y (mgr.34107) 12554 : cluster [DBG] pgmap v12995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:39.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:38 smithi121 bash[36902]: audit 2022-04-22T23:34:38.894112+0000 mon.a (mon.0) 17268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:38 smithi115 bash[45495]: cluster 2022-04-22T23:34:37.665654+0000 mgr.y (mgr.34107) 12554 : cluster [DBG] pgmap v12995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:39.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:38 smithi115 bash[45495]: audit 2022-04-22T23:34:38.894112+0000 mon.a (mon.0) 17268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:38 smithi115 bash[46934]: cluster 2022-04-22T23:34:37.665654+0000 mgr.y (mgr.34107) 12554 : cluster [DBG] pgmap v12995: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:39.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:38 smithi115 bash[46934]: audit 2022-04-22T23:34:38.894112+0000 mon.a (mon.0) 17268 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:40 smithi121 bash[36902]: cluster 2022-04-22T23:34:39.666703+0000 mgr.y (mgr.34107) 12555 : cluster [DBG] pgmap v12996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:41.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:40 smithi121 bash[36902]: audit 2022-04-22T23:34:40.698032+0000 mon.a (mon.0) 17269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:40 smithi115 bash[45495]: cluster 2022-04-22T23:34:39.666703+0000 mgr.y (mgr.34107) 12555 : cluster [DBG] pgmap v12996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:41.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:40 smithi115 bash[45495]: audit 2022-04-22T23:34:40.698032+0000 mon.a (mon.0) 17269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:40 smithi115 bash[46934]: cluster 2022-04-22T23:34:39.666703+0000 mgr.y (mgr.34107) 12555 : cluster [DBG] pgmap v12996: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:41.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:40 smithi115 bash[46934]: audit 2022-04-22T23:34:40.698032+0000 mon.a (mon.0) 17269 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:42 smithi121 bash[36902]: cluster 2022-04-22T23:34:41.667833+0000 mgr.y (mgr.34107) 12556 : cluster [DBG] pgmap v12997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:43.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:42 smithi121 bash[36902]: audit 2022-04-22T23:34:42.247362+0000 mon.a (mon.0) 17270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:43.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:43.068Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:42 smithi115 bash[45495]: cluster 2022-04-22T23:34:41.667833+0000 mgr.y (mgr.34107) 12556 : cluster [DBG] pgmap v12997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:43.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:42 smithi115 bash[45495]: audit 2022-04-22T23:34:42.247362+0000 mon.a (mon.0) 17270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:42 smithi115 bash[46934]: cluster 2022-04-22T23:34:41.667833+0000 mgr.y (mgr.34107) 12556 : cluster [DBG] pgmap v12997: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:42 smithi115 bash[46934]: audit 2022-04-22T23:34:42.247362+0000 mon.a (mon.0) 17270 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:44.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:43 smithi121 bash[36902]: audit 2022-04-22T23:34:43.897225+0000 mon.a (mon.0) 17271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:44.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:43 smithi115 bash[45495]: audit 2022-04-22T23:34:43.897225+0000 mon.a (mon.0) 17271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:44.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:43 smithi115 bash[46934]: audit 2022-04-22T23:34:43.897225+0000 mon.a (mon.0) 17271 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:45.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:44 smithi121 bash[36902]: cluster 2022-04-22T23:34:43.668952+0000 mgr.y (mgr.34107) 12557 : cluster [DBG] pgmap v12998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:45.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:44 smithi115 bash[45495]: cluster 2022-04-22T23:34:43.668952+0000 mgr.y (mgr.34107) 12557 : cluster [DBG] pgmap v12998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:45.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:44 smithi115 bash[46934]: cluster 2022-04-22T23:34:43.668952+0000 mgr.y (mgr.34107) 12557 : cluster [DBG] pgmap v12998: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:45.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:44.930Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:46.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:45 smithi121 bash[36902]: audit 2022-04-22T23:34:45.701547+0000 mon.a (mon.0) 17272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:46.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:45 smithi115 bash[45495]: audit 2022-04-22T23:34:45.701547+0000 mon.a (mon.0) 17272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:45 smithi115 bash[46934]: audit 2022-04-22T23:34:45.701547+0000 mon.a (mon.0) 17272 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:47.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:46 smithi121 bash[36902]: cluster 2022-04-22T23:34:45.670149+0000 mgr.y (mgr.34107) 12558 : cluster [DBG] pgmap v12999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:47.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:46 smithi115 bash[45495]: cluster 2022-04-22T23:34:45.670149+0000 mgr.y (mgr.34107) 12558 : cluster [DBG] pgmap v12999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:47.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:46 smithi115 bash[46934]: cluster 2022-04-22T23:34:45.670149+0000 mgr.y (mgr.34107) 12558 : cluster [DBG] pgmap v12999: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:48.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:47 smithi121 bash[36902]: audit 2022-04-22T23:34:47.250482+0000 mon.a (mon.0) 17273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:48.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:47 smithi115 bash[45495]: audit 2022-04-22T23:34:47.250482+0000 mon.a (mon.0) 17273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:48.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:47 smithi115 bash[46934]: audit 2022-04-22T23:34:47.250482+0000 mon.a (mon.0) 17273 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:48 smithi121 bash[36902]: cluster 2022-04-22T23:34:47.671274+0000 mgr.y (mgr.34107) 12559 : cluster [DBG] pgmap v13000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:49.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:48 smithi121 bash[36902]: audit 2022-04-22T23:34:48.900519+0000 mon.a (mon.0) 17274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:48 smithi115 bash[46934]: cluster 2022-04-22T23:34:47.671274+0000 mgr.y (mgr.34107) 12559 : cluster [DBG] pgmap v13000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:49.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:48 smithi115 bash[46934]: audit 2022-04-22T23:34:48.900519+0000 mon.a (mon.0) 17274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:48 smithi115 bash[45495]: cluster 2022-04-22T23:34:47.671274+0000 mgr.y (mgr.34107) 12559 : cluster [DBG] pgmap v13000: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:49.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:48 smithi115 bash[45495]: audit 2022-04-22T23:34:48.900519+0000 mon.a (mon.0) 17274 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:50 smithi121 bash[36902]: cluster 2022-04-22T23:34:49.672358+0000 mgr.y (mgr.34107) 12560 : cluster [DBG] pgmap v13001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:51.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:50 smithi121 bash[36902]: audit 2022-04-22T23:34:50.705252+0000 mon.a (mon.0) 17275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:50 smithi115 bash[45495]: cluster 2022-04-22T23:34:49.672358+0000 mgr.y (mgr.34107) 12560 : cluster [DBG] pgmap v13001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:51.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:50 smithi115 bash[45495]: audit 2022-04-22T23:34:50.705252+0000 mon.a (mon.0) 17275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:50 smithi115 bash[46934]: cluster 2022-04-22T23:34:49.672358+0000 mgr.y (mgr.34107) 12560 : cluster [DBG] pgmap v13001: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:51.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:50 smithi115 bash[46934]: audit 2022-04-22T23:34:50.705252+0000 mon.a (mon.0) 17275 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:52 smithi121 bash[36902]: cluster 2022-04-22T23:34:51.673537+0000 mgr.y (mgr.34107) 12561 : cluster [DBG] pgmap v13002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:53.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:52 smithi121 bash[36902]: audit 2022-04-22T23:34:52.253768+0000 mon.a (mon.0) 17276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:52 smithi115 bash[45495]: cluster 2022-04-22T23:34:51.673537+0000 mgr.y (mgr.34107) 12561 : cluster [DBG] pgmap v13002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:53.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:52 smithi115 bash[45495]: audit 2022-04-22T23:34:52.253768+0000 mon.a (mon.0) 17276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:52 smithi115 bash[46934]: cluster 2022-04-22T23:34:51.673537+0000 mgr.y (mgr.34107) 12561 : cluster [DBG] pgmap v13002: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:53.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:52 smithi115 bash[46934]: audit 2022-04-22T23:34:52.253768+0000 mon.a (mon.0) 17276 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:53.292 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:53.069Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:54.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:53 smithi121 bash[36902]: audit 2022-04-22T23:34:53.903742+0000 mon.a (mon.0) 17277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:54.290 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:53 smithi115 bash[45495]: audit 2022-04-22T23:34:53.903742+0000 mon.a (mon.0) 17277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:54.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:53 smithi115 bash[46934]: audit 2022-04-22T23:34:53.903742+0000 mon.a (mon.0) 17277 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:55.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:55 smithi121 bash[36902]: cluster 2022-04-22T23:34:53.674715+0000 mgr.y (mgr.34107) 12562 : cluster [DBG] pgmap v13003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:55.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:34:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:34:54.931Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:34:55.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:54 smithi115 bash[45495]: cluster 2022-04-22T23:34:53.674715+0000 mgr.y (mgr.34107) 12562 : cluster [DBG] pgmap v13003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:55.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:54 smithi115 bash[46934]: cluster 2022-04-22T23:34:53.674715+0000 mgr.y (mgr.34107) 12562 : cluster [DBG] pgmap v13003: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:56.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:56 smithi121 bash[36902]: audit 2022-04-22T23:34:55.709296+0000 mon.a (mon.0) 17278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:56.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:56 smithi115 bash[45495]: audit 2022-04-22T23:34:55.709296+0000 mon.a (mon.0) 17278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:56.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:56 smithi115 bash[46934]: audit 2022-04-22T23:34:55.709296+0000 mon.a (mon.0) 17278 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:34:57.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:57 smithi115 bash[45495]: cluster 2022-04-22T23:34:55.675842+0000 mgr.y (mgr.34107) 12563 : cluster [DBG] pgmap v13004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:57.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:57 smithi115 bash[46934]: cluster 2022-04-22T23:34:55.675842+0000 mgr.y (mgr.34107) 12563 : cluster [DBG] pgmap v13004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:57.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:57 smithi121 bash[36902]: cluster 2022-04-22T23:34:55.675842+0000 mgr.y (mgr.34107) 12563 : cluster [DBG] pgmap v13004: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:58.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:58 smithi115 bash[45495]: audit 2022-04-22T23:34:57.256814+0000 mon.a (mon.0) 17279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:58.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:58 smithi115 bash[46934]: audit 2022-04-22T23:34:57.256814+0000 mon.a (mon.0) 17279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:58.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:58 smithi121 bash[36902]: audit 2022-04-22T23:34:57.256814+0000 mon.a (mon.0) 17279 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:34:58.731 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 6m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 7m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6h) 6m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (5h) 7m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:34:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 7m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 6m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 6m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 7m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 6m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 6m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 7m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 6m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 6m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 6m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:34:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 6m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:34:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 7m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:34:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 7m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:34:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 7m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:34:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6h) 7m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:34:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 7m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:34:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:59 smithi115 bash[46934]: cluster 2022-04-22T23:34:57.676982+0000 mgr.y (mgr.34107) 12564 : cluster [DBG] pgmap v13005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:59.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:34:59 smithi115 bash[46934]: audit 2022-04-22T23:34:58.907050+0000 mon.a (mon.0) 17280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:59 smithi115 bash[45495]: cluster 2022-04-22T23:34:57.676982+0000 mgr.y (mgr.34107) 12564 : cluster [DBG] pgmap v13005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:59.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:34:59 smithi115 bash[45495]: audit 2022-04-22T23:34:58.907050+0000 mon.a (mon.0) 17280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:34:59.499 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:34:59.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:34:59.501 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:34:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:59 smithi121 bash[36902]: cluster 2022-04-22T23:34:57.676982+0000 mgr.y (mgr.34107) 12564 : cluster [DBG] pgmap v13005: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:34:59.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:34:59 smithi121 bash[36902]: audit 2022-04-22T23:34:58.907050+0000 mon.a (mon.0) 17280 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[45495]: audit 2022-04-22T23:34:58.712420+0000 mgr.y (mgr.34107) 12565 : audit [DBG] from='client.48339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[45495]: audit 2022-04-22T23:34:59.064089+0000 mgr.y (mgr.34107) 12566 : audit [DBG] from='client.48345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[45495]: audit 2022-04-22T23:34:59.496213+0000 mon.c (mon.1) 424 : audit [DBG] from='client.? 172.21.15.115:0/3619902945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[46934]: audit 2022-04-22T23:34:58.712420+0000 mgr.y (mgr.34107) 12565 : audit [DBG] from='client.48339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[46934]: audit 2022-04-22T23:34:59.064089+0000 mgr.y (mgr.34107) 12566 : audit [DBG] from='client.48345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:00 smithi115 bash[46934]: audit 2022-04-22T23:34:59.496213+0000 mon.c (mon.1) 424 : audit [DBG] from='client.? 172.21.15.115:0/3619902945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:00 smithi121 bash[36902]: audit 2022-04-22T23:34:58.712420+0000 mgr.y (mgr.34107) 12565 : audit [DBG] from='client.48339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:00 smithi121 bash[36902]: audit 2022-04-22T23:34:59.064089+0000 mgr.y (mgr.34107) 12566 : audit [DBG] from='client.48345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:00.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:00 smithi121 bash[36902]: audit 2022-04-22T23:34:59.496213+0000 mon.c (mon.1) 424 : audit [DBG] from='client.? 172.21.15.115:0/3619902945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:01.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:01 smithi115 bash[45495]: cluster 2022-04-22T23:34:59.678110+0000 mgr.y (mgr.34107) 12567 : cluster [DBG] pgmap v13006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:01.343 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:01 smithi115 bash[45495]: audit 2022-04-22T23:35:00.713020+0000 mon.a (mon.0) 17281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:01.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:01 smithi115 bash[46934]: cluster 2022-04-22T23:34:59.678110+0000 mgr.y (mgr.34107) 12567 : cluster [DBG] pgmap v13006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:01.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:01 smithi115 bash[46934]: audit 2022-04-22T23:35:00.713020+0000 mon.a (mon.0) 17281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:01 smithi121 bash[36902]: cluster 2022-04-22T23:34:59.678110+0000 mgr.y (mgr.34107) 12567 : cluster [DBG] pgmap v13006: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:01.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:01 smithi121 bash[36902]: audit 2022-04-22T23:35:00.713020+0000 mon.a (mon.0) 17281 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:03 smithi121 bash[36902]: cluster 2022-04-22T23:35:01.679193+0000 mgr.y (mgr.34107) 12568 : cluster [DBG] pgmap v13007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:03.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:03 smithi121 bash[36902]: audit 2022-04-22T23:35:02.260265+0000 mon.a (mon.0) 17282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:03.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:03.070Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:03 smithi115 bash[45495]: cluster 2022-04-22T23:35:01.679193+0000 mgr.y (mgr.34107) 12568 : cluster [DBG] pgmap v13007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:03 smithi115 bash[45495]: audit 2022-04-22T23:35:02.260265+0000 mon.a (mon.0) 17282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:03 smithi115 bash[46934]: cluster 2022-04-22T23:35:01.679193+0000 mgr.y (mgr.34107) 12568 : cluster [DBG] pgmap v13007: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:03.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:03 smithi115 bash[46934]: audit 2022-04-22T23:35:02.260265+0000 mon.a (mon.0) 17282 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:04.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:04 smithi121 bash[36902]: audit 2022-04-22T23:35:03.910123+0000 mon.a (mon.0) 17283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:04.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:04 smithi115 bash[45495]: audit 2022-04-22T23:35:03.910123+0000 mon.a (mon.0) 17283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:04.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:04 smithi115 bash[46934]: audit 2022-04-22T23:35:03.910123+0000 mon.a (mon.0) 17283 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:05.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:05 smithi115 bash[45495]: cluster 2022-04-22T23:35:03.680310+0000 mgr.y (mgr.34107) 12569 : cluster [DBG] pgmap v13008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:04.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:05.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:05 smithi115 bash[46934]: cluster 2022-04-22T23:35:03.680310+0000 mgr.y (mgr.34107) 12569 : cluster [DBG] pgmap v13008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:05.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:05 smithi121 bash[36902]: cluster 2022-04-22T23:35:03.680310+0000 mgr.y (mgr.34107) 12569 : cluster [DBG] pgmap v13008: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:06.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:06 smithi121 bash[36902]: audit 2022-04-22T23:35:05.716748+0000 mon.a (mon.0) 17284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:06.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:06 smithi115 bash[45495]: audit 2022-04-22T23:35:05.716748+0000 mon.a (mon.0) 17284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:06.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:06 smithi115 bash[46934]: audit 2022-04-22T23:35:05.716748+0000 mon.a (mon.0) 17284 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:07.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:07 smithi121 bash[36902]: cluster 2022-04-22T23:35:05.681316+0000 mgr.y (mgr.34107) 12570 : cluster [DBG] pgmap v13009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:07.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:07 smithi115 bash[45495]: cluster 2022-04-22T23:35:05.681316+0000 mgr.y (mgr.34107) 12570 : cluster [DBG] pgmap v13009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:07.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:07 smithi115 bash[46934]: cluster 2022-04-22T23:35:05.681316+0000 mgr.y (mgr.34107) 12570 : cluster [DBG] pgmap v13009: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:08.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:08 smithi121 bash[36902]: audit 2022-04-22T23:35:07.263135+0000 mon.a (mon.0) 17285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:08.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:08 smithi115 bash[45495]: audit 2022-04-22T23:35:07.263135+0000 mon.a (mon.0) 17285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:08.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:08 smithi115 bash[46934]: audit 2022-04-22T23:35:07.263135+0000 mon.a (mon.0) 17285 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:09 smithi121 bash[36902]: cluster 2022-04-22T23:35:07.682443+0000 mgr.y (mgr.34107) 12571 : cluster [DBG] pgmap v13010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:09.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:09 smithi121 bash[36902]: audit 2022-04-22T23:35:08.912985+0000 mon.a (mon.0) 17286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:09.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:09 smithi115 bash[45495]: cluster 2022-04-22T23:35:07.682443+0000 mgr.y (mgr.34107) 12571 : cluster [DBG] pgmap v13010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:09.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:09 smithi115 bash[45495]: audit 2022-04-22T23:35:08.912985+0000 mon.a (mon.0) 17286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:09 smithi115 bash[46934]: cluster 2022-04-22T23:35:07.682443+0000 mgr.y (mgr.34107) 12571 : cluster [DBG] pgmap v13010: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:09.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:09 smithi115 bash[46934]: audit 2022-04-22T23:35:08.912985+0000 mon.a (mon.0) 17286 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:11.477 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:11 smithi115 bash[45495]: cluster 2022-04-22T23:35:09.683559+0000 mgr.y (mgr.34107) 12572 : cluster [DBG] pgmap v13011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:11.478 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:11 smithi115 bash[45495]: audit 2022-04-22T23:35:10.720641+0000 mon.a (mon.0) 17287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:11.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:11 smithi115 bash[46934]: cluster 2022-04-22T23:35:09.683559+0000 mgr.y (mgr.34107) 12572 : cluster [DBG] pgmap v13011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:11.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:11 smithi115 bash[46934]: audit 2022-04-22T23:35:10.720641+0000 mon.a (mon.0) 17287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:11.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:11 smithi121 bash[36902]: cluster 2022-04-22T23:35:09.683559+0000 mgr.y (mgr.34107) 12572 : cluster [DBG] pgmap v13011: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:11.487 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:11 smithi121 bash[36902]: audit 2022-04-22T23:35:10.720641+0000 mon.a (mon.0) 17287 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:12 smithi121 bash[36902]: audit 2022-04-22T23:35:11.120140+0000 mon.a (mon.0) 17288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:35:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:12 smithi121 bash[36902]: audit 2022-04-22T23:35:11.123156+0000 mon.a (mon.0) 17289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:35:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:12 smithi121 bash[36902]: audit 2022-04-22T23:35:11.124926+0000 mon.a (mon.0) 17290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:35:12.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:12 smithi121 bash[36902]: audit 2022-04-22T23:35:11.495810+0000 mon.a (mon.0) 17291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[45495]: audit 2022-04-22T23:35:11.120140+0000 mon.a (mon.0) 17288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[45495]: audit 2022-04-22T23:35:11.123156+0000 mon.a (mon.0) 17289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[45495]: audit 2022-04-22T23:35:11.124926+0000 mon.a (mon.0) 17290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[45495]: audit 2022-04-22T23:35:11.495810+0000 mon.a (mon.0) 17291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[46934]: audit 2022-04-22T23:35:11.120140+0000 mon.a (mon.0) 17288 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[46934]: audit 2022-04-22T23:35:11.123156+0000 mon.a (mon.0) 17289 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:35:12.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[46934]: audit 2022-04-22T23:35:11.124926+0000 mon.a (mon.0) 17290 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:35:12.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:12 smithi115 bash[46934]: audit 2022-04-22T23:35:11.495810+0000 mon.a (mon.0) 17291 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:35:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:13 smithi121 bash[36902]: cluster 2022-04-22T23:35:11.684586+0000 mgr.y (mgr.34107) 12573 : cluster [DBG] pgmap v13012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:13.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:13 smithi121 bash[36902]: audit 2022-04-22T23:35:12.265707+0000 mon.a (mon.0) 17292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:13.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:13.071Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:13 smithi115 bash[45495]: cluster 2022-04-22T23:35:11.684586+0000 mgr.y (mgr.34107) 12573 : cluster [DBG] pgmap v13012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:13.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:13 smithi115 bash[45495]: audit 2022-04-22T23:35:12.265707+0000 mon.a (mon.0) 17292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:13.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:13 smithi115 bash[46934]: cluster 2022-04-22T23:35:11.684586+0000 mgr.y (mgr.34107) 12573 : cluster [DBG] pgmap v13012: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:13.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:13 smithi115 bash[46934]: audit 2022-04-22T23:35:12.265707+0000 mon.a (mon.0) 17292 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:14.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:14 smithi121 bash[36902]: audit 2022-04-22T23:35:13.916278+0000 mon.a (mon.0) 17293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:14.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:14 smithi115 bash[45495]: audit 2022-04-22T23:35:13.916278+0000 mon.a (mon.0) 17293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:14 smithi115 bash[46934]: audit 2022-04-22T23:35:13.916278+0000 mon.a (mon.0) 17293 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:14.932Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:15.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:15 smithi115 bash[45495]: cluster 2022-04-22T23:35:13.685711+0000 mgr.y (mgr.34107) 12574 : cluster [DBG] pgmap v13013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:15.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:15 smithi115 bash[46934]: cluster 2022-04-22T23:35:13.685711+0000 mgr.y (mgr.34107) 12574 : cluster [DBG] pgmap v13013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:15.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:15 smithi121 bash[36902]: cluster 2022-04-22T23:35:13.685711+0000 mgr.y (mgr.34107) 12574 : cluster [DBG] pgmap v13013: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:16.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:16 smithi121 bash[36902]: audit 2022-04-22T23:35:15.724426+0000 mon.a (mon.0) 17294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:16.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:16 smithi115 bash[45495]: audit 2022-04-22T23:35:15.724426+0000 mon.a (mon.0) 17294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:16.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:16 smithi115 bash[46934]: audit 2022-04-22T23:35:15.724426+0000 mon.a (mon.0) 17294 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:17.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:17 smithi121 bash[36902]: cluster 2022-04-22T23:35:15.686763+0000 mgr.y (mgr.34107) 12575 : cluster [DBG] pgmap v13014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:17.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:17 smithi115 bash[45495]: cluster 2022-04-22T23:35:15.686763+0000 mgr.y (mgr.34107) 12575 : cluster [DBG] pgmap v13014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:17.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:17 smithi115 bash[46934]: cluster 2022-04-22T23:35:15.686763+0000 mgr.y (mgr.34107) 12575 : cluster [DBG] pgmap v13014: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:18 smithi121 bash[36902]: audit 2022-04-22T23:35:17.268644+0000 mon.a (mon.0) 17295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:18 smithi121 bash[36902]: audit 2022-04-22T23:35:17.534777+0000 mon.a (mon.0) 17296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:35:18.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:18 smithi121 bash[36902]: audit 2022-04-22T23:35:17.535458+0000 mon.a (mon.0) 17297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[46934]: audit 2022-04-22T23:35:17.268644+0000 mon.a (mon.0) 17295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[46934]: audit 2022-04-22T23:35:17.534777+0000 mon.a (mon.0) 17296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[46934]: audit 2022-04-22T23:35:17.535458+0000 mon.a (mon.0) 17297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[45495]: audit 2022-04-22T23:35:17.268644+0000 mon.a (mon.0) 17295 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[45495]: audit 2022-04-22T23:35:17.534777+0000 mon.a (mon.0) 17296 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:35:18.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:18 smithi115 bash[45495]: audit 2022-04-22T23:35:17.535458+0000 mon.a (mon.0) 17297 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:19 smithi121 bash[36902]: cluster 2022-04-22T23:35:17.687924+0000 mgr.y (mgr.34107) 12576 : cluster [DBG] pgmap v13015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:19.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:19 smithi121 bash[36902]: audit 2022-04-22T23:35:18.919330+0000 mon.a (mon.0) 17298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:19 smithi115 bash[45495]: cluster 2022-04-22T23:35:17.687924+0000 mgr.y (mgr.34107) 12576 : cluster [DBG] pgmap v13015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:19.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:19 smithi115 bash[45495]: audit 2022-04-22T23:35:18.919330+0000 mon.a (mon.0) 17298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:19 smithi115 bash[46934]: cluster 2022-04-22T23:35:17.687924+0000 mgr.y (mgr.34107) 12576 : cluster [DBG] pgmap v13015: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:19.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:19 smithi115 bash[46934]: audit 2022-04-22T23:35:18.919330+0000 mon.a (mon.0) 17298 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:21 smithi121 bash[36902]: cluster 2022-04-22T23:35:19.689006+0000 mgr.y (mgr.34107) 12577 : cluster [DBG] pgmap v13016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:21.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:21 smithi121 bash[36902]: audit 2022-04-22T23:35:20.728144+0000 mon.a (mon.0) 17299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:21 smithi115 bash[45495]: cluster 2022-04-22T23:35:19.689006+0000 mgr.y (mgr.34107) 12577 : cluster [DBG] pgmap v13016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:21.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:21 smithi115 bash[45495]: audit 2022-04-22T23:35:20.728144+0000 mon.a (mon.0) 17299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:21 smithi115 bash[46934]: cluster 2022-04-22T23:35:19.689006+0000 mgr.y (mgr.34107) 12577 : cluster [DBG] pgmap v13016: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:21.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:21 smithi115 bash[46934]: audit 2022-04-22T23:35:20.728144+0000 mon.a (mon.0) 17299 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:23 smithi121 bash[36902]: cluster 2022-04-22T23:35:21.690113+0000 mgr.y (mgr.34107) 12578 : cluster [DBG] pgmap v13017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:23.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:23 smithi121 bash[36902]: audit 2022-04-22T23:35:22.271656+0000 mon.a (mon.0) 17300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:23.540 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:23.072Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:23 smithi115 bash[45495]: cluster 2022-04-22T23:35:21.690113+0000 mgr.y (mgr.34107) 12578 : cluster [DBG] pgmap v13017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:23.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:23 smithi115 bash[45495]: audit 2022-04-22T23:35:22.271656+0000 mon.a (mon.0) 17300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:23 smithi115 bash[46934]: cluster 2022-04-22T23:35:21.690113+0000 mgr.y (mgr.34107) 12578 : cluster [DBG] pgmap v13017: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:23.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:23 smithi115 bash[46934]: audit 2022-04-22T23:35:22.271656+0000 mon.a (mon.0) 17300 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:24.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:24 smithi121 bash[36902]: audit 2022-04-22T23:35:23.922493+0000 mon.a (mon.0) 17301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:24.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:24 smithi115 bash[45495]: audit 2022-04-22T23:35:23.922493+0000 mon.a (mon.0) 17301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:24 smithi115 bash[46934]: audit 2022-04-22T23:35:23.922493+0000 mon.a (mon.0) 17301 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:25.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:25 smithi115 bash[45495]: cluster 2022-04-22T23:35:23.691261+0000 mgr.y (mgr.34107) 12579 : cluster [DBG] pgmap v13018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:24.933Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:25.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:25 smithi115 bash[46934]: cluster 2022-04-22T23:35:23.691261+0000 mgr.y (mgr.34107) 12579 : cluster [DBG] pgmap v13018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:25.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:25 smithi121 bash[36902]: cluster 2022-04-22T23:35:23.691261+0000 mgr.y (mgr.34107) 12579 : cluster [DBG] pgmap v13018: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:26.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:26 smithi121 bash[36902]: audit 2022-04-22T23:35:25.731769+0000 mon.a (mon.0) 17302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:26.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:26 smithi115 bash[45495]: audit 2022-04-22T23:35:25.731769+0000 mon.a (mon.0) 17302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:26.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:26 smithi115 bash[46934]: audit 2022-04-22T23:35:25.731769+0000 mon.a (mon.0) 17302 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:27.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:27 smithi121 bash[36902]: cluster 2022-04-22T23:35:25.692379+0000 mgr.y (mgr.34107) 12580 : cluster [DBG] pgmap v13019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:27.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:27 smithi115 bash[45495]: cluster 2022-04-22T23:35:25.692379+0000 mgr.y (mgr.34107) 12580 : cluster [DBG] pgmap v13019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:27.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:27 smithi115 bash[46934]: cluster 2022-04-22T23:35:25.692379+0000 mgr.y (mgr.34107) 12580 : cluster [DBG] pgmap v13019: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:28.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:28 smithi121 bash[36902]: audit 2022-04-22T23:35:27.274499+0000 mon.a (mon.0) 17303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:28.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:28 smithi115 bash[45495]: audit 2022-04-22T23:35:27.274499+0000 mon.a (mon.0) 17303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:28.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:28 smithi115 bash[46934]: audit 2022-04-22T23:35:27.274499+0000 mon.a (mon.0) 17303 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:29 smithi121 bash[36902]: cluster 2022-04-22T23:35:27.693507+0000 mgr.y (mgr.34107) 12581 : cluster [DBG] pgmap v13020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:29.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:29 smithi121 bash[36902]: audit 2022-04-22T23:35:28.925943+0000 mon.a (mon.0) 17304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:29.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:29 smithi115 bash[46934]: cluster 2022-04-22T23:35:27.693507+0000 mgr.y (mgr.34107) 12581 : cluster [DBG] pgmap v13020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:29 smithi115 bash[46934]: audit 2022-04-22T23:35:28.925943+0000 mon.a (mon.0) 17304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:29 smithi115 bash[45495]: cluster 2022-04-22T23:35:27.693507+0000 mgr.y (mgr.34107) 12581 : cluster [DBG] pgmap v13020: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:29.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:29 smithi115 bash[45495]: audit 2022-04-22T23:35:28.925943+0000 mon.a (mon.0) 17304 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:29.882 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 8m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6h) 7m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6h) 8m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:35:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 8m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:35:30.236 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 7m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:35:30.236 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 7m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:35:30.236 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 8m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:35:30.236 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 7m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:35:30.236 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5h) 7m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (5h) 8m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 7m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 7m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 7m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 7m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 8m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 8m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 8m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6h) 8m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:35:30.237 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:35:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:35:30.638 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:35:30.639 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:31 smithi121 bash[36902]: cluster 2022-04-22T23:35:29.694603+0000 mgr.y (mgr.34107) 12582 : cluster [DBG] pgmap v13021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:31 smithi121 bash[36902]: audit 2022-04-22T23:35:29.862524+0000 mgr.y (mgr.34107) 12583 : audit [DBG] from='client.48357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:31 smithi121 bash[36902]: audit 2022-04-22T23:35:30.211681+0000 mgr.y (mgr.34107) 12584 : audit [DBG] from='client.58303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:31 smithi121 bash[36902]: audit 2022-04-22T23:35:30.633412+0000 mon.a (mon.0) 17305 : audit [DBG] from='client.? 172.21.15.115:0/2955503388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:31.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:31 smithi121 bash[36902]: audit 2022-04-22T23:35:30.735688+0000 mon.a (mon.0) 17306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[45495]: cluster 2022-04-22T23:35:29.694603+0000 mgr.y (mgr.34107) 12582 : cluster [DBG] pgmap v13021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[45495]: audit 2022-04-22T23:35:29.862524+0000 mgr.y (mgr.34107) 12583 : audit [DBG] from='client.48357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[45495]: audit 2022-04-22T23:35:30.211681+0000 mgr.y (mgr.34107) 12584 : audit [DBG] from='client.58303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[45495]: audit 2022-04-22T23:35:30.633412+0000 mon.a (mon.0) 17305 : audit [DBG] from='client.? 172.21.15.115:0/2955503388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:31.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[45495]: audit 2022-04-22T23:35:30.735688+0000 mon.a (mon.0) 17306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[46934]: cluster 2022-04-22T23:35:29.694603+0000 mgr.y (mgr.34107) 12582 : cluster [DBG] pgmap v13021: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[46934]: audit 2022-04-22T23:35:29.862524+0000 mgr.y (mgr.34107) 12583 : audit [DBG] from='client.48357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[46934]: audit 2022-04-22T23:35:30.211681+0000 mgr.y (mgr.34107) 12584 : audit [DBG] from='client.58303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:35:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[46934]: audit 2022-04-22T23:35:30.633412+0000 mon.a (mon.0) 17305 : audit [DBG] from='client.? 172.21.15.115:0/2955503388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:35:31.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:31 smithi115 bash[46934]: audit 2022-04-22T23:35:30.735688+0000 mon.a (mon.0) 17306 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:33 smithi121 bash[36902]: cluster 2022-04-22T23:35:31.695788+0000 mgr.y (mgr.34107) 12585 : cluster [DBG] pgmap v13022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:33.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:33 smithi121 bash[36902]: audit 2022-04-22T23:35:32.277713+0000 mon.a (mon.0) 17307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:33.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:33.072Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:33 smithi115 bash[46934]: cluster 2022-04-22T23:35:31.695788+0000 mgr.y (mgr.34107) 12585 : cluster [DBG] pgmap v13022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:33.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:33 smithi115 bash[46934]: audit 2022-04-22T23:35:32.277713+0000 mon.a (mon.0) 17307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:33 smithi115 bash[45495]: cluster 2022-04-22T23:35:31.695788+0000 mgr.y (mgr.34107) 12585 : cluster [DBG] pgmap v13022: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:33.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:33 smithi115 bash[45495]: audit 2022-04-22T23:35:32.277713+0000 mon.a (mon.0) 17307 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:34.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:34 smithi121 bash[36902]: audit 2022-04-22T23:35:33.928985+0000 mon.a (mon.0) 17308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:34.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:34 smithi115 bash[45495]: audit 2022-04-22T23:35:33.928985+0000 mon.a (mon.0) 17308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:34.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:34 smithi115 bash[46934]: audit 2022-04-22T23:35:33.928985+0000 mon.a (mon.0) 17308 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:35.195 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:34.934Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:35.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:35 smithi121 bash[36902]: cluster 2022-04-22T23:35:33.696916+0000 mgr.y (mgr.34107) 12586 : cluster [DBG] pgmap v13023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:35.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:35 smithi115 bash[45495]: cluster 2022-04-22T23:35:33.696916+0000 mgr.y (mgr.34107) 12586 : cluster [DBG] pgmap v13023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:35.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:35 smithi115 bash[46934]: cluster 2022-04-22T23:35:33.696916+0000 mgr.y (mgr.34107) 12586 : cluster [DBG] pgmap v13023: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:36.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:36 smithi121 bash[36902]: audit 2022-04-22T23:35:35.739423+0000 mon.a (mon.0) 17309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:36.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:36 smithi115 bash[46934]: audit 2022-04-22T23:35:35.739423+0000 mon.a (mon.0) 17309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:36.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:36 smithi115 bash[45495]: audit 2022-04-22T23:35:35.739423+0000 mon.a (mon.0) 17309 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:37.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:37 smithi121 bash[36902]: cluster 2022-04-22T23:35:35.698008+0000 mgr.y (mgr.34107) 12587 : cluster [DBG] pgmap v13024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:37 smithi115 bash[46934]: cluster 2022-04-22T23:35:35.698008+0000 mgr.y (mgr.34107) 12587 : cluster [DBG] pgmap v13024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:37.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:37 smithi115 bash[45495]: cluster 2022-04-22T23:35:35.698008+0000 mgr.y (mgr.34107) 12587 : cluster [DBG] pgmap v13024: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:38.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:38 smithi121 bash[36902]: audit 2022-04-22T23:35:37.280857+0000 mon.a (mon.0) 17310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:38.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:38 smithi115 bash[46934]: audit 2022-04-22T23:35:37.280857+0000 mon.a (mon.0) 17310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:38.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:38 smithi115 bash[45495]: audit 2022-04-22T23:35:37.280857+0000 mon.a (mon.0) 17310 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:39 smithi121 bash[36902]: cluster 2022-04-22T23:35:37.699107+0000 mgr.y (mgr.34107) 12588 : cluster [DBG] pgmap v13025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:39.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:39 smithi121 bash[36902]: audit 2022-04-22T23:35:38.931721+0000 mon.a (mon.0) 17311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:39 smithi115 bash[45495]: cluster 2022-04-22T23:35:37.699107+0000 mgr.y (mgr.34107) 12588 : cluster [DBG] pgmap v13025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:39.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:39 smithi115 bash[45495]: audit 2022-04-22T23:35:38.931721+0000 mon.a (mon.0) 17311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:39 smithi115 bash[46934]: cluster 2022-04-22T23:35:37.699107+0000 mgr.y (mgr.34107) 12588 : cluster [DBG] pgmap v13025: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:39.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:39 smithi115 bash[46934]: audit 2022-04-22T23:35:38.931721+0000 mon.a (mon.0) 17311 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:41 smithi121 bash[36902]: cluster 2022-04-22T23:35:39.700241+0000 mgr.y (mgr.34107) 12589 : cluster [DBG] pgmap v13026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:41.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:41 smithi121 bash[36902]: audit 2022-04-22T23:35:40.743310+0000 mon.a (mon.0) 17312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:41.544 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:41 smithi115 bash[45495]: cluster 2022-04-22T23:35:39.700241+0000 mgr.y (mgr.34107) 12589 : cluster [DBG] pgmap v13026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:41.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:41 smithi115 bash[45495]: audit 2022-04-22T23:35:40.743310+0000 mon.a (mon.0) 17312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:41.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:41 smithi115 bash[46934]: cluster 2022-04-22T23:35:39.700241+0000 mgr.y (mgr.34107) 12589 : cluster [DBG] pgmap v13026: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:41.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:41 smithi115 bash[46934]: audit 2022-04-22T23:35:40.743310+0000 mon.a (mon.0) 17312 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:43 smithi121 bash[36902]: cluster 2022-04-22T23:35:41.701331+0000 mgr.y (mgr.34107) 12590 : cluster [DBG] pgmap v13027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:43.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:43 smithi121 bash[36902]: audit 2022-04-22T23:35:42.283964+0000 mon.a (mon.0) 17313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:43.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:43 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:43.073Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:43 smithi115 bash[45495]: cluster 2022-04-22T23:35:41.701331+0000 mgr.y (mgr.34107) 12590 : cluster [DBG] pgmap v13027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:43.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:43 smithi115 bash[45495]: audit 2022-04-22T23:35:42.283964+0000 mon.a (mon.0) 17313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:43 smithi115 bash[46934]: cluster 2022-04-22T23:35:41.701331+0000 mgr.y (mgr.34107) 12590 : cluster [DBG] pgmap v13027: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:43.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:43 smithi115 bash[46934]: audit 2022-04-22T23:35:42.283964+0000 mon.a (mon.0) 17313 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:44.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:44 smithi121 bash[36902]: audit 2022-04-22T23:35:43.934687+0000 mon.a (mon.0) 17314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:44.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:44 smithi115 bash[46934]: audit 2022-04-22T23:35:43.934687+0000 mon.a (mon.0) 17314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:44.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:44 smithi115 bash[45495]: audit 2022-04-22T23:35:43.934687+0000 mon.a (mon.0) 17314 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:45.237 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:44 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:44.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:45.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:45 smithi121 bash[36902]: cluster 2022-04-22T23:35:43.702467+0000 mgr.y (mgr.34107) 12591 : cluster [DBG] pgmap v13028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:45.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:45 smithi115 bash[45495]: cluster 2022-04-22T23:35:43.702467+0000 mgr.y (mgr.34107) 12591 : cluster [DBG] pgmap v13028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:45.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:45 smithi115 bash[46934]: cluster 2022-04-22T23:35:43.702467+0000 mgr.y (mgr.34107) 12591 : cluster [DBG] pgmap v13028: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:46.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:46 smithi121 bash[36902]: audit 2022-04-22T23:35:45.747116+0000 mon.a (mon.0) 17315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:46.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:46 smithi115 bash[45495]: audit 2022-04-22T23:35:45.747116+0000 mon.a (mon.0) 17315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:46.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:46 smithi115 bash[46934]: audit 2022-04-22T23:35:45.747116+0000 mon.a (mon.0) 17315 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:47.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:47 smithi121 bash[36902]: cluster 2022-04-22T23:35:45.703630+0000 mgr.y (mgr.34107) 12592 : cluster [DBG] pgmap v13029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:47.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:47 smithi115 bash[46934]: cluster 2022-04-22T23:35:45.703630+0000 mgr.y (mgr.34107) 12592 : cluster [DBG] pgmap v13029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:47.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:47 smithi115 bash[45495]: cluster 2022-04-22T23:35:45.703630+0000 mgr.y (mgr.34107) 12592 : cluster [DBG] pgmap v13029: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:48.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:48 smithi121 bash[36902]: audit 2022-04-22T23:35:47.286858+0000 mon.a (mon.0) 17316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:48.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:48 smithi115 bash[45495]: audit 2022-04-22T23:35:47.286858+0000 mon.a (mon.0) 17316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:48.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:48 smithi115 bash[46934]: audit 2022-04-22T23:35:47.286858+0000 mon.a (mon.0) 17316 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:49 smithi121 bash[36902]: cluster 2022-04-22T23:35:47.704775+0000 mgr.y (mgr.34107) 12593 : cluster [DBG] pgmap v13030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:49.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:49 smithi121 bash[36902]: audit 2022-04-22T23:35:48.937724+0000 mon.a (mon.0) 17317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:49.540 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:49 smithi115 bash[45495]: cluster 2022-04-22T23:35:47.704775+0000 mgr.y (mgr.34107) 12593 : cluster [DBG] pgmap v13030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:49.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:49 smithi115 bash[45495]: audit 2022-04-22T23:35:48.937724+0000 mon.a (mon.0) 17317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:49 smithi115 bash[46934]: cluster 2022-04-22T23:35:47.704775+0000 mgr.y (mgr.34107) 12593 : cluster [DBG] pgmap v13030: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:49.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:49 smithi115 bash[46934]: audit 2022-04-22T23:35:48.937724+0000 mon.a (mon.0) 17317 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:51 smithi115 bash[45495]: cluster 2022-04-22T23:35:49.705884+0000 mgr.y (mgr.34107) 12594 : cluster [DBG] pgmap v13031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:51.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:51 smithi115 bash[45495]: audit 2022-04-22T23:35:50.750872+0000 mon.a (mon.0) 17318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:51 smithi115 bash[46934]: cluster 2022-04-22T23:35:49.705884+0000 mgr.y (mgr.34107) 12594 : cluster [DBG] pgmap v13031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:51.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:51 smithi115 bash[46934]: audit 2022-04-22T23:35:50.750872+0000 mon.a (mon.0) 17318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:51 smithi121 bash[36902]: cluster 2022-04-22T23:35:49.705884+0000 mgr.y (mgr.34107) 12594 : cluster [DBG] pgmap v13031: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:51.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:51 smithi121 bash[36902]: audit 2022-04-22T23:35:50.750872+0000 mon.a (mon.0) 17318 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:53.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:53 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:53.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:53 smithi115 bash[45495]: cluster 2022-04-22T23:35:51.707017+0000 mgr.y (mgr.34107) 12595 : cluster [DBG] pgmap v13032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:53.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:53 smithi115 bash[45495]: audit 2022-04-22T23:35:52.290078+0000 mon.a (mon.0) 17319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:53 smithi115 bash[46934]: cluster 2022-04-22T23:35:51.707017+0000 mgr.y (mgr.34107) 12595 : cluster [DBG] pgmap v13032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:53.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:53 smithi115 bash[46934]: audit 2022-04-22T23:35:52.290078+0000 mon.a (mon.0) 17319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:53 smithi121 bash[36902]: cluster 2022-04-22T23:35:51.707017+0000 mgr.y (mgr.34107) 12595 : cluster [DBG] pgmap v13032: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:53 smithi121 bash[36902]: audit 2022-04-22T23:35:52.290078+0000 mon.a (mon.0) 17319 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:54.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:54 smithi115 bash[45495]: audit 2022-04-22T23:35:53.940802+0000 mon.a (mon.0) 17320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:54.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:54 smithi115 bash[46934]: audit 2022-04-22T23:35:53.940802+0000 mon.a (mon.0) 17320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:54.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:54 smithi121 bash[36902]: audit 2022-04-22T23:35:53.940802+0000 mon.a (mon.0) 17320 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:55.280 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:35:54 smithi115 bash[60466]: level=error ts=2022-04-22T23:35:54.935Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:35:55.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:55 smithi115 bash[45495]: cluster 2022-04-22T23:35:53.708314+0000 mgr.y (mgr.34107) 12596 : cluster [DBG] pgmap v13033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:55.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:55 smithi115 bash[46934]: cluster 2022-04-22T23:35:53.708314+0000 mgr.y (mgr.34107) 12596 : cluster [DBG] pgmap v13033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:55.767 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:55 smithi121 bash[36902]: cluster 2022-04-22T23:35:53.708314+0000 mgr.y (mgr.34107) 12596 : cluster [DBG] pgmap v13033: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:56.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:56 smithi115 bash[45495]: audit 2022-04-22T23:35:55.754677+0000 mon.a (mon.0) 17321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:56.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:56 smithi115 bash[46934]: audit 2022-04-22T23:35:55.754677+0000 mon.a (mon.0) 17321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:56.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:56 smithi121 bash[36902]: audit 2022-04-22T23:35:55.754677+0000 mon.a (mon.0) 17321 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:35:57.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:57 smithi115 bash[45495]: cluster 2022-04-22T23:35:55.709466+0000 mgr.y (mgr.34107) 12597 : cluster [DBG] pgmap v13034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:57.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:57 smithi115 bash[46934]: cluster 2022-04-22T23:35:55.709466+0000 mgr.y (mgr.34107) 12597 : cluster [DBG] pgmap v13034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:57.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:57 smithi121 bash[36902]: cluster 2022-04-22T23:35:55.709466+0000 mgr.y (mgr.34107) 12597 : cluster [DBG] pgmap v13034: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:58.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:58 smithi121 bash[36902]: audit 2022-04-22T23:35:57.294292+0000 mon.a (mon.0) 17322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:58.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:58 smithi115 bash[45495]: audit 2022-04-22T23:35:57.294292+0000 mon.a (mon.0) 17322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:58.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:58 smithi115 bash[46934]: audit 2022-04-22T23:35:57.294292+0000 mon.a (mon.0) 17322 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:35:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:59 smithi121 bash[36902]: cluster 2022-04-22T23:35:57.710535+0000 mgr.y (mgr.34107) 12598 : cluster [DBG] pgmap v13035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:59.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:35:59 smithi121 bash[36902]: audit 2022-04-22T23:35:58.943701+0000 mon.a (mon.0) 17323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:59.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:59 smithi115 bash[45495]: cluster 2022-04-22T23:35:57.710535+0000 mgr.y (mgr.34107) 12598 : cluster [DBG] pgmap v13035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:59.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:35:59 smithi115 bash[45495]: audit 2022-04-22T23:35:58.943701+0000 mon.a (mon.0) 17323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:35:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:59 smithi115 bash[46934]: cluster 2022-04-22T23:35:57.710535+0000 mgr.y (mgr.34107) 12598 : cluster [DBG] pgmap v13035: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:35:59.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:35:59 smithi115 bash[46934]: audit 2022-04-22T23:35:58.943701+0000 mon.a (mon.0) 17323 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:01.010 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 7m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 8m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6h) 7m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6h) 8m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:36:01.359 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 8m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 7m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 7m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 8m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 7m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6h) 7m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6h) 8m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 7m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 7m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 7m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:36:01.360 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 7m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:36:01.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 8m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:36:01.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 8m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:36:01.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 8m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:36:01.361 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6h) 8m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:36:01.361 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (5h) 8m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:36:01.761 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:36:01.762 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:36:01.763 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:36:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:01 smithi121 bash[36902]: cluster 2022-04-22T23:35:59.711660+0000 mgr.y (mgr.34107) 12599 : cluster [DBG] pgmap v13036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:01.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:01 smithi121 bash[36902]: audit 2022-04-22T23:36:00.758431+0000 mon.a (mon.0) 17324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:01.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:01 smithi115 bash[45495]: cluster 2022-04-22T23:35:59.711660+0000 mgr.y (mgr.34107) 12599 : cluster [DBG] pgmap v13036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:01.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:01 smithi115 bash[45495]: audit 2022-04-22T23:36:00.758431+0000 mon.a (mon.0) 17324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:01 smithi115 bash[46934]: cluster 2022-04-22T23:35:59.711660+0000 mgr.y (mgr.34107) 12599 : cluster [DBG] pgmap v13036: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:01.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:01 smithi115 bash[46934]: audit 2022-04-22T23:36:00.758431+0000 mon.a (mon.0) 17324 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:02 smithi121 bash[36902]: audit 2022-04-22T23:36:00.991833+0000 mgr.y (mgr.34107) 12600 : audit [DBG] from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:02 smithi121 bash[36902]: audit 2022-04-22T23:36:01.336287+0000 mgr.y (mgr.34107) 12601 : audit [DBG] from='client.48381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:02 smithi121 bash[36902]: audit 2022-04-22T23:36:01.757742+0000 mon.a (mon.0) 17325 : audit [DBG] from='client.? 172.21.15.115:0/2535335114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:02.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:02 smithi121 bash[36902]: audit 2022-04-22T23:36:02.296390+0000 mon.a (mon.0) 17326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[45495]: audit 2022-04-22T23:36:00.991833+0000 mgr.y (mgr.34107) 12600 : audit [DBG] from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[45495]: audit 2022-04-22T23:36:01.336287+0000 mgr.y (mgr.34107) 12601 : audit [DBG] from='client.48381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[45495]: audit 2022-04-22T23:36:01.757742+0000 mon.a (mon.0) 17325 : audit [DBG] from='client.? 172.21.15.115:0/2535335114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[45495]: audit 2022-04-22T23:36:02.296390+0000 mon.a (mon.0) 17326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[46934]: audit 2022-04-22T23:36:00.991833+0000 mgr.y (mgr.34107) 12600 : audit [DBG] from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[46934]: audit 2022-04-22T23:36:01.336287+0000 mgr.y (mgr.34107) 12601 : audit [DBG] from='client.48381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[46934]: audit 2022-04-22T23:36:01.757742+0000 mon.a (mon.0) 17325 : audit [DBG] from='client.? 172.21.15.115:0/2535335114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:02.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:02 smithi115 bash[46934]: audit 2022-04-22T23:36:02.296390+0000 mon.a (mon.0) 17326 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:03.541 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:03 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:03.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:03.541 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:03 smithi115 bash[45495]: cluster 2022-04-22T23:36:01.712696+0000 mgr.y (mgr.34107) 12602 : cluster [DBG] pgmap v13037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:03.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:03 smithi115 bash[46934]: cluster 2022-04-22T23:36:01.712696+0000 mgr.y (mgr.34107) 12602 : cluster [DBG] pgmap v13037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:03.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:03 smithi121 bash[36902]: cluster 2022-04-22T23:36:01.712696+0000 mgr.y (mgr.34107) 12602 : cluster [DBG] pgmap v13037: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:04.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:04 smithi121 bash[36902]: audit 2022-04-22T23:36:03.946800+0000 mon.a (mon.0) 17327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:04 smithi115 bash[45495]: audit 2022-04-22T23:36:03.946800+0000 mon.a (mon.0) 17327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:04 smithi115 bash[46934]: audit 2022-04-22T23:36:03.946800+0000 mon.a (mon.0) 17327 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:05.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:04 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:04.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:05.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:05 smithi121 bash[36902]: cluster 2022-04-22T23:36:03.713847+0000 mgr.y (mgr.34107) 12603 : cluster [DBG] pgmap v13038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:05.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:05 smithi115 bash[45495]: cluster 2022-04-22T23:36:03.713847+0000 mgr.y (mgr.34107) 12603 : cluster [DBG] pgmap v13038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:05.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:05 smithi115 bash[46934]: cluster 2022-04-22T23:36:03.713847+0000 mgr.y (mgr.34107) 12603 : cluster [DBG] pgmap v13038: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:06.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:06 smithi121 bash[36902]: audit 2022-04-22T23:36:05.762157+0000 mon.a (mon.0) 17328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:06.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:06 smithi115 bash[45495]: audit 2022-04-22T23:36:05.762157+0000 mon.a (mon.0) 17328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:06.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:06 smithi115 bash[46934]: audit 2022-04-22T23:36:05.762157+0000 mon.a (mon.0) 17328 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:07 smithi121 bash[36902]: cluster 2022-04-22T23:36:05.714935+0000 mgr.y (mgr.34107) 12604 : cluster [DBG] pgmap v13039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:07 smithi121 bash[36902]: audit 2022-04-22T23:36:07.299394+0000 mon.a (mon.0) 17329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:07 smithi115 bash[45495]: cluster 2022-04-22T23:36:05.714935+0000 mgr.y (mgr.34107) 12604 : cluster [DBG] pgmap v13039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:07 smithi115 bash[45495]: audit 2022-04-22T23:36:07.299394+0000 mon.a (mon.0) 17329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:07 smithi115 bash[46934]: cluster 2022-04-22T23:36:05.714935+0000 mgr.y (mgr.34107) 12604 : cluster [DBG] pgmap v13039: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:07.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:07 smithi115 bash[46934]: audit 2022-04-22T23:36:07.299394+0000 mon.a (mon.0) 17329 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:09 smithi121 bash[36902]: cluster 2022-04-22T23:36:07.716074+0000 mgr.y (mgr.34107) 12605 : cluster [DBG] pgmap v13040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:09.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:09 smithi121 bash[36902]: audit 2022-04-22T23:36:08.949928+0000 mon.a (mon.0) 17330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:09 smithi115 bash[45495]: cluster 2022-04-22T23:36:07.716074+0000 mgr.y (mgr.34107) 12605 : cluster [DBG] pgmap v13040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:09.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:09 smithi115 bash[45495]: audit 2022-04-22T23:36:08.949928+0000 mon.a (mon.0) 17330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:09 smithi115 bash[46934]: cluster 2022-04-22T23:36:07.716074+0000 mgr.y (mgr.34107) 12605 : cluster [DBG] pgmap v13040: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:09.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:09 smithi115 bash[46934]: audit 2022-04-22T23:36:08.949928+0000 mon.a (mon.0) 17330 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:11.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:11 smithi115 bash[46934]: cluster 2022-04-22T23:36:09.717195+0000 mgr.y (mgr.34107) 12606 : cluster [DBG] pgmap v13041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:11.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:11 smithi115 bash[46934]: audit 2022-04-22T23:36:10.766038+0000 mon.a (mon.0) 17331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:11.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:11 smithi115 bash[45495]: cluster 2022-04-22T23:36:09.717195+0000 mgr.y (mgr.34107) 12606 : cluster [DBG] pgmap v13041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:11.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:11 smithi115 bash[45495]: audit 2022-04-22T23:36:10.766038+0000 mon.a (mon.0) 17331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:11.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:11 smithi121 bash[36902]: cluster 2022-04-22T23:36:09.717195+0000 mgr.y (mgr.34107) 12606 : cluster [DBG] pgmap v13041: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:11.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:11 smithi121 bash[36902]: audit 2022-04-22T23:36:10.766038+0000 mon.a (mon.0) 17331 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:12 smithi121 bash[36902]: audit 2022-04-22T23:36:11.503235+0000 mon.a (mon.0) 17332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:36:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:12 smithi121 bash[36902]: audit 2022-04-22T23:36:11.506103+0000 mon.a (mon.0) 17333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:36:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:12 smithi121 bash[36902]: audit 2022-04-22T23:36:11.507812+0000 mon.a (mon.0) 17334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:36:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:12 smithi121 bash[36902]: audit 2022-04-22T23:36:11.880384+0000 mon.a (mon.0) 17335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:36:12.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:12 smithi121 bash[36902]: audit 2022-04-22T23:36:12.302272+0000 mon.a (mon.0) 17336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[45495]: audit 2022-04-22T23:36:11.503235+0000 mon.a (mon.0) 17332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:36:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[45495]: audit 2022-04-22T23:36:11.506103+0000 mon.a (mon.0) 17333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:36:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[45495]: audit 2022-04-22T23:36:11.507812+0000 mon.a (mon.0) 17334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:36:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[45495]: audit 2022-04-22T23:36:11.880384+0000 mon.a (mon.0) 17335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:36:12.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[45495]: audit 2022-04-22T23:36:12.302272+0000 mon.a (mon.0) 17336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[46934]: audit 2022-04-22T23:36:11.503235+0000 mon.a (mon.0) 17332 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-04-22T23:36:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[46934]: audit 2022-04-22T23:36:11.506103+0000 mon.a (mon.0) 17333 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-04-22T23:36:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[46934]: audit 2022-04-22T23:36:11.507812+0000 mon.a (mon.0) 17334 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-04-22T23:36:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[46934]: audit 2022-04-22T23:36:11.880384+0000 mon.a (mon.0) 17335 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' 2022-04-22T23:36:12.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:12 smithi115 bash[46934]: audit 2022-04-22T23:36:12.302272+0000 mon.a (mon.0) 17336 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:13.356 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:13 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:13.073Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:13.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:13 smithi121 bash[36902]: cluster 2022-04-22T23:36:11.717966+0000 mgr.y (mgr.34107) 12607 : cluster [DBG] pgmap v13042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:13.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:13 smithi115 bash[45495]: cluster 2022-04-22T23:36:11.717966+0000 mgr.y (mgr.34107) 12607 : cluster [DBG] pgmap v13042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:13.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:13 smithi115 bash[46934]: cluster 2022-04-22T23:36:11.717966+0000 mgr.y (mgr.34107) 12607 : cluster [DBG] pgmap v13042: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:14.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:14 smithi121 bash[36902]: audit 2022-04-22T23:36:13.953090+0000 mon.a (mon.0) 17337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:14.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:14 smithi115 bash[45495]: audit 2022-04-22T23:36:13.953090+0000 mon.a (mon.0) 17337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:14.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:14 smithi115 bash[46934]: audit 2022-04-22T23:36:13.953090+0000 mon.a (mon.0) 17337 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:15.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:14 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:14.938Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:15.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:15 smithi121 bash[36902]: cluster 2022-04-22T23:36:13.719059+0000 mgr.y (mgr.34107) 12608 : cluster [DBG] pgmap v13043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:15.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:15 smithi115 bash[45495]: cluster 2022-04-22T23:36:13.719059+0000 mgr.y (mgr.34107) 12608 : cluster [DBG] pgmap v13043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:15 smithi115 bash[46934]: cluster 2022-04-22T23:36:13.719059+0000 mgr.y (mgr.34107) 12608 : cluster [DBG] pgmap v13043: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:16.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:16 smithi121 bash[36902]: audit 2022-04-22T23:36:15.769469+0000 mon.a (mon.0) 17338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:16.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:16 smithi115 bash[45495]: audit 2022-04-22T23:36:15.769469+0000 mon.a (mon.0) 17338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:16.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:16 smithi115 bash[46934]: audit 2022-04-22T23:36:15.769469+0000 mon.a (mon.0) 17338 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:17 smithi121 bash[36902]: cluster 2022-04-22T23:36:15.720212+0000 mgr.y (mgr.34107) 12609 : cluster [DBG] pgmap v13044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:17.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:17 smithi121 bash[36902]: audit 2022-04-22T23:36:17.305247+0000 mon.a (mon.0) 17339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:17 smithi115 bash[45495]: cluster 2022-04-22T23:36:15.720212+0000 mgr.y (mgr.34107) 12609 : cluster [DBG] pgmap v13044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:17.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:17 smithi115 bash[45495]: audit 2022-04-22T23:36:17.305247+0000 mon.a (mon.0) 17339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:17 smithi115 bash[46934]: cluster 2022-04-22T23:36:15.720212+0000 mgr.y (mgr.34107) 12609 : cluster [DBG] pgmap v13044: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:17.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:17 smithi115 bash[46934]: audit 2022-04-22T23:36:17.305247+0000 mon.a (mon.0) 17339 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:18 smithi121 bash[36902]: audit 2022-04-22T23:36:17.545076+0000 mon.a (mon.0) 17340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:36:18.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:18 smithi121 bash[36902]: audit 2022-04-22T23:36:17.545451+0000 mon.a (mon.0) 17341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:36:18.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:18 smithi115 bash[45495]: audit 2022-04-22T23:36:17.545076+0000 mon.a (mon.0) 17340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:36:18.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:18 smithi115 bash[45495]: audit 2022-04-22T23:36:17.545451+0000 mon.a (mon.0) 17341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:36:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:18 smithi115 bash[46934]: audit 2022-04-22T23:36:17.545076+0000 mon.a (mon.0) 17340 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-04-22T23:36:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:18 smithi115 bash[46934]: audit 2022-04-22T23:36:17.545451+0000 mon.a (mon.0) 17341 : audit [INF] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-04-22T23:36:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:19 smithi121 bash[36902]: cluster 2022-04-22T23:36:17.721264+0000 mgr.y (mgr.34107) 12610 : cluster [DBG] pgmap v13045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:19.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:19 smithi121 bash[36902]: audit 2022-04-22T23:36:18.956250+0000 mon.a (mon.0) 17342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:19.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:19 smithi115 bash[45495]: cluster 2022-04-22T23:36:17.721264+0000 mgr.y (mgr.34107) 12610 : cluster [DBG] pgmap v13045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:19.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:19 smithi115 bash[45495]: audit 2022-04-22T23:36:18.956250+0000 mon.a (mon.0) 17342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:19 smithi115 bash[46934]: cluster 2022-04-22T23:36:17.721264+0000 mgr.y (mgr.34107) 12610 : cluster [DBG] pgmap v13045: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:19.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:19 smithi115 bash[46934]: audit 2022-04-22T23:36:18.956250+0000 mon.a (mon.0) 17342 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:21 smithi121 bash[36902]: cluster 2022-04-22T23:36:19.722272+0000 mgr.y (mgr.34107) 12611 : cluster [DBG] pgmap v13046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:21.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:21 smithi121 bash[36902]: audit 2022-04-22T23:36:20.772984+0000 mon.a (mon.0) 17343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:21.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:21 smithi115 bash[45495]: cluster 2022-04-22T23:36:19.722272+0000 mgr.y (mgr.34107) 12611 : cluster [DBG] pgmap v13046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:21.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:21 smithi115 bash[45495]: audit 2022-04-22T23:36:20.772984+0000 mon.a (mon.0) 17343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:21 smithi115 bash[46934]: cluster 2022-04-22T23:36:19.722272+0000 mgr.y (mgr.34107) 12611 : cluster [DBG] pgmap v13046: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:21.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:21 smithi115 bash[46934]: audit 2022-04-22T23:36:20.772984+0000 mon.a (mon.0) 17343 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:22.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:22 smithi121 bash[36902]: audit 2022-04-22T23:36:22.308283+0000 mon.a (mon.0) 17344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:22.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:22 smithi115 bash[45495]: audit 2022-04-22T23:36:22.308283+0000 mon.a (mon.0) 17344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:22.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:22 smithi115 bash[46934]: audit 2022-04-22T23:36:22.308283+0000 mon.a (mon.0) 17344 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:23.410 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:23 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:23.075Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:23.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:23 smithi121 bash[36902]: cluster 2022-04-22T23:36:21.723441+0000 mgr.y (mgr.34107) 12612 : cluster [DBG] pgmap v13047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:23.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:23 smithi115 bash[45495]: cluster 2022-04-22T23:36:21.723441+0000 mgr.y (mgr.34107) 12612 : cluster [DBG] pgmap v13047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:23.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:23 smithi115 bash[46934]: cluster 2022-04-22T23:36:21.723441+0000 mgr.y (mgr.34107) 12612 : cluster [DBG] pgmap v13047: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:24.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:24 smithi121 bash[36902]: audit 2022-04-22T23:36:23.959321+0000 mon.a (mon.0) 17345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:24.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:24 smithi115 bash[45495]: audit 2022-04-22T23:36:23.959321+0000 mon.a (mon.0) 17345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:24 smithi115 bash[46934]: audit 2022-04-22T23:36:23.959321+0000 mon.a (mon.0) 17345 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:25.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:24 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:24.940Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:25.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:25 smithi121 bash[36902]: cluster 2022-04-22T23:36:23.724735+0000 mgr.y (mgr.34107) 12613 : cluster [DBG] pgmap v13048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:25.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:25 smithi115 bash[45495]: cluster 2022-04-22T23:36:23.724735+0000 mgr.y (mgr.34107) 12613 : cluster [DBG] pgmap v13048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:25.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:25 smithi115 bash[46934]: cluster 2022-04-22T23:36:23.724735+0000 mgr.y (mgr.34107) 12613 : cluster [DBG] pgmap v13048: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:26.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:26 smithi121 bash[36902]: audit 2022-04-22T23:36:25.776834+0000 mon.a (mon.0) 17346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:26.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:26 smithi115 bash[45495]: audit 2022-04-22T23:36:25.776834+0000 mon.a (mon.0) 17346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:26.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:26 smithi115 bash[46934]: audit 2022-04-22T23:36:25.776834+0000 mon.a (mon.0) 17346 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:27 smithi121 bash[36902]: cluster 2022-04-22T23:36:25.725926+0000 mgr.y (mgr.34107) 12614 : cluster [DBG] pgmap v13049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:27.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:27 smithi121 bash[36902]: audit 2022-04-22T23:36:27.311258+0000 mon.a (mon.0) 17347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:27 smithi115 bash[45495]: cluster 2022-04-22T23:36:25.725926+0000 mgr.y (mgr.34107) 12614 : cluster [DBG] pgmap v13049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:27.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:27 smithi115 bash[45495]: audit 2022-04-22T23:36:27.311258+0000 mon.a (mon.0) 17347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:27 smithi115 bash[46934]: cluster 2022-04-22T23:36:25.725926+0000 mgr.y (mgr.34107) 12614 : cluster [DBG] pgmap v13049: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:27 smithi115 bash[46934]: audit 2022-04-22T23:36:27.311258+0000 mon.a (mon.0) 17347 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:29 smithi121 bash[36902]: cluster 2022-04-22T23:36:27.727071+0000 mgr.y (mgr.34107) 12615 : cluster [DBG] pgmap v13050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:29.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:29 smithi121 bash[36902]: audit 2022-04-22T23:36:28.962258+0000 mon.a (mon.0) 17348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:29 smithi115 bash[45495]: cluster 2022-04-22T23:36:27.727071+0000 mgr.y (mgr.34107) 12615 : cluster [DBG] pgmap v13050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:29.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:29 smithi115 bash[45495]: audit 2022-04-22T23:36:28.962258+0000 mon.a (mon.0) 17348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:29 smithi115 bash[46934]: cluster 2022-04-22T23:36:27.727071+0000 mgr.y (mgr.34107) 12615 : cluster [DBG] pgmap v13050: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:29.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:29 smithi115 bash[46934]: audit 2022-04-22T23:36:28.962258+0000 mon.a (mon.0) 17348 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:31 smithi121 bash[36902]: cluster 2022-04-22T23:36:29.728210+0000 mgr.y (mgr.34107) 12616 : cluster [DBG] pgmap v13051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:31.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:31 smithi121 bash[36902]: audit 2022-04-22T23:36:30.780507+0000 mon.a (mon.0) 17349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:31 smithi115 bash[45495]: cluster 2022-04-22T23:36:29.728210+0000 mgr.y (mgr.34107) 12616 : cluster [DBG] pgmap v13051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:31.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:31 smithi115 bash[45495]: audit 2022-04-22T23:36:30.780507+0000 mon.a (mon.0) 17349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:31 smithi115 bash[46934]: cluster 2022-04-22T23:36:29.728210+0000 mgr.y (mgr.34107) 12616 : cluster [DBG] pgmap v13051: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:31.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:31 smithi115 bash[46934]: audit 2022-04-22T23:36:30.780507+0000 mon.a (mon.0) 17349 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:32.129 INFO:teuthology.orchestra.run.smithi115.stdout:true 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5h) 8m ago 6h 15.9M - ba2b418f427c 9255ba08f72f 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi121 *:3000 running (6h) 9m ago 6h 26.1M - 6.7.4 557c83e11646 520d113f1a88 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (6h) 8m ago 6h 23.8M - 17.0.0-11772-g72067459 6c95b0749680 2d1a2f3efb8a 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi121 running (6h) 9m ago 6h 22.0M - 17.0.0-11772-g72067459 6c95b0749680 c9a981d9bf5f 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi121 *:8443 running (6h) 9m ago 6h 414M - 17.0.0-11772-g72067459 6c95b0749680 376eff784ed3 2022-04-22T23:36:32.475 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443 running (6h) 8m ago 6h 495M - 17.0.0-11772-g72067459 6c95b0749680 cf8a86ef247f 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6h) 8m ago 6h 477M 2048M 17.0.0-11772-g72067459 6c95b0749680 ecb8193ee4ba 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi121 running (6h) 9m ago 6h 321M 2048M 17.0.0-11772-g72067459 6c95b0749680 17a9a507cedc 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6h) 8m ago 6h 329M 2048M 17.0.0-11772-g72067459 6c95b0749680 af10e87c1814 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6h) 8m ago 6h 10.5M - 1dbe0e931976 3b7b10b76eee 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi121 *:9100 running (6h) 9m ago 6h 10.3M - 1dbe0e931976 f4b044642a96 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6h) 8m ago 6h 722M 2526M 17.0.0-11772-g72067459 6c95b0749680 47e1b418ab08 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6h) 8m ago 6h 803M 2526M 17.0.0-11772-g72067459 6c95b0749680 abd08003b59f 2022-04-22T23:36:32.476 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6h) 8m ago 6h 544M 2526M 17.0.0-11772-g72067459 6c95b0749680 71e3e090c8ff 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6h) 8m ago 6h 507M 2526M 17.0.0-11772-g72067459 6c95b0749680 a3725c5cc863 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi121 running (6h) 9m ago 6h 591M 2526M 17.0.0-11772-g72067459 6c95b0749680 78d622c6e355 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi121 running (6h) 9m ago 6h 725M 2526M 17.0.0-11772-g72067459 6c95b0749680 d180f41db62d 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi121 running (6h) 9m ago 6h 510M 2526M 17.0.0-11772-g72067459 6c95b0749680 0d04f70082c8 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi121 running (6h) 9m ago 6h 655M 2526M 17.0.0-11772-g72067459 6c95b0749680 224045e11699 2022-04-22T23:36:32.477 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi121 *:9095 running (6h) 9m ago 6h 67.2M - 514e6a882f6e 184051ba2cf2 2022-04-22T23:36:32.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:32 smithi121 bash[36902]: audit 2022-04-22T23:36:32.314084+0000 mon.a (mon.0) 17350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:32.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:32 smithi115 bash[45495]: audit 2022-04-22T23:36:32.314084+0000 mon.a (mon.0) 17350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:32.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:32 smithi115 bash[46934]: audit 2022-04-22T23:36:32.314084+0000 mon.a (mon.0) 17350 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:32.883 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-04-22T23:36:32.883 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2022-04-22T23:36:32.883 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 3 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 8 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 2 2022-04-22T23:36:32.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2022-04-22T23:36:32.885 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2022-04-22T23:36:32.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.0.0-11772-g72067459 (720674591667bc34cb7086618bbed8722a1f429d) quincy (dev)": 15 2022-04-22T23:36:32.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2022-04-22T23:36:32.885 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-04-22T23:36:33.454 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:33.074Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:33 smithi121 bash[36902]: cluster 2022-04-22T23:36:31.729354+0000 mgr.y (mgr.34107) 12617 : cluster [DBG] pgmap v13052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:33 smithi121 bash[36902]: audit 2022-04-22T23:36:32.109276+0000 mgr.y (mgr.34107) 12618 : audit [DBG] from='client.58333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:33 smithi121 bash[36902]: audit 2022-04-22T23:36:32.455771+0000 mgr.y (mgr.34107) 12619 : audit [DBG] from='client.48399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:33 smithi121 bash[36902]: audit 2022-04-22T23:36:32.879736+0000 mon.a (mon.0) 17351 : audit [DBG] from='client.? 172.21.15.115:0/541474177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[45495]: cluster 2022-04-22T23:36:31.729354+0000 mgr.y (mgr.34107) 12617 : cluster [DBG] pgmap v13052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[45495]: audit 2022-04-22T23:36:32.109276+0000 mgr.y (mgr.34107) 12618 : audit [DBG] from='client.58333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[45495]: audit 2022-04-22T23:36:32.455771+0000 mgr.y (mgr.34107) 12619 : audit [DBG] from='client.48399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[45495]: audit 2022-04-22T23:36:32.879736+0000 mon.a (mon.0) 17351 : audit [DBG] from='client.? 172.21.15.115:0/541474177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[46934]: cluster 2022-04-22T23:36:31.729354+0000 mgr.y (mgr.34107) 12617 : cluster [DBG] pgmap v13052: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[46934]: audit 2022-04-22T23:36:32.109276+0000 mgr.y (mgr.34107) 12618 : audit [DBG] from='client.58333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[46934]: audit 2022-04-22T23:36:32.455771+0000 mgr.y (mgr.34107) 12619 : audit [DBG] from='client.48399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-04-22T23:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:33 smithi115 bash[46934]: audit 2022-04-22T23:36:32.879736+0000 mon.a (mon.0) 17351 : audit [DBG] from='client.? 172.21.15.115:0/541474177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-04-22T23:36:34.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:34 smithi121 bash[36902]: audit 2022-04-22T23:36:33.965529+0000 mon.a (mon.0) 17352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:34.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:34 smithi115 bash[45495]: audit 2022-04-22T23:36:33.965529+0000 mon.a (mon.0) 17352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:34.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:34 smithi115 bash[46934]: audit 2022-04-22T23:36:33.965529+0000 mon.a (mon.0) 17352 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:35.291 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Apr 22 23:36:34 smithi115 bash[60466]: level=error ts=2022-04-22T23:36:34.939Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://172.21.15.115:8443//api/prometheus_receiver" 2022-04-22T23:36:35.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:35 smithi121 bash[36902]: cluster 2022-04-22T23:36:33.730533+0000 mgr.y (mgr.34107) 12620 : cluster [DBG] pgmap v13053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:35.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:35 smithi115 bash[45495]: cluster 2022-04-22T23:36:33.730533+0000 mgr.y (mgr.34107) 12620 : cluster [DBG] pgmap v13053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:35.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:35 smithi115 bash[46934]: cluster 2022-04-22T23:36:33.730533+0000 mgr.y (mgr.34107) 12620 : cluster [DBG] pgmap v13053: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:36.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:36 smithi121 bash[36902]: audit 2022-04-22T23:36:35.784098+0000 mon.a (mon.0) 17353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:36.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:36 smithi115 bash[45495]: audit 2022-04-22T23:36:35.784098+0000 mon.a (mon.0) 17353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:36.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:36 smithi115 bash[46934]: audit 2022-04-22T23:36:35.784098+0000 mon.a (mon.0) 17353 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-04-22T23:36:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:37 smithi121 bash[36902]: cluster 2022-04-22T23:36:35.731648+0000 mgr.y (mgr.34107) 12621 : cluster [DBG] pgmap v13054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:37.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:37 smithi121 bash[36902]: audit 2022-04-22T23:36:37.316688+0000 mon.a (mon.0) 17354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:37 smithi115 bash[45495]: cluster 2022-04-22T23:36:35.731648+0000 mgr.y (mgr.34107) 12621 : cluster [DBG] pgmap v13054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:37.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:37 smithi115 bash[45495]: audit 2022-04-22T23:36:37.316688+0000 mon.a (mon.0) 17354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:37 smithi115 bash[46934]: cluster 2022-04-22T23:36:35.731648+0000 mgr.y (mgr.34107) 12621 : cluster [DBG] pgmap v13054: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:37.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:37 smithi115 bash[46934]: audit 2022-04-22T23:36:37.316688+0000 mon.a (mon.0) 17354 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-04-22T23:36:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:39 smithi121 bash[36902]: cluster 2022-04-22T23:36:37.732720+0000 mgr.y (mgr.34107) 12622 : cluster [DBG] pgmap v13055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:39.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 22 23:36:39 smithi121 bash[36902]: audit 2022-04-22T23:36:38.968690+0000 mon.a (mon.0) 17355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:39 smithi115 bash[45495]: cluster 2022-04-22T23:36:37.732720+0000 mgr.y (mgr.34107) 12622 : cluster [DBG] pgmap v13055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:39.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Apr 22 23:36:39 smithi115 bash[45495]: audit 2022-04-22T23:36:38.968690+0000 mon.a (mon.0) 17355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:39 smithi115 bash[46934]: cluster 2022-04-22T23:36:37.732720+0000 mgr.y (mgr.34107) 12622 : cluster [DBG] pgmap v13055: 105 pgs: 105 active+clean; 4.6 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2022-04-22T23:36:39.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Apr 22 23:36:39 smithi115 bash[46934]: audit 2022-04-22T23:36:38.968690+0000 mon.a (mon.0) 17355 : audit [DBG] from='mgr.34107 172.21.15.115:0/2801271726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-04-22T23:36:41.314 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-22T23:36:41.315 DEBUG:teuthology.task.console_log:Killing console logger for smithi115 2022-04-22T23:36:41.315 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2022-04-22T23:36:41.316 DEBUG:teuthology.exit:Finished running handlers